2023-07-28T23:22:29.684 INFO:root:teuthology version: 0.0.1.dev172+g6114016 2023-07-28T23:22:29.685 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2023-07-28_23:12:07-rgw-reef-release-distro-default-smithi --owner scheduled_yuriw@teuthology --description rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_latest} tasks/test_multi} --archive /home/teuthworker/archive/yuriw-2023-07-28_23:12:07-rgw-reef-release-distro-default-smithi/7355843 --verbose -- /home/teuthworker/archive/yuriw-2023-07-28_23:12:07-rgw-reef-release-distro-default-smithi/7355843/orig.config.yaml 2023-07-28T23:22:29.720 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-28T23:22:29.807 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-07-28_23:12:07-rgw-reef-release-distro-default-smithi/7355843 branch: reef-release description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_latest} tasks/test_multi} email: yweinste@redhat.com first_in_suite: false job_id: '7355843' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-07-28_23:12:07-rgw-reef-release-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: reef-release ceph: conf: client: debug rgw: 20 debug rgw sync: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync data inject err probability: 0.1 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph client.0: rgw data notify interval msec: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) - \(POOL_APP_NOT_ENABLED\) sha1: 0484097b0572e96f42cf6402cbe0ac8dcb046577 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 0484097b0572e96f42cf6402cbe0ac8dcb046577 rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - enabled_features: - resharding endpoints: - c1.client.0 is_default: true is_master: true name: a zones: - endpoints: - c1.client.0 is_default: true is_master: true name: a1 - endpoints: - c1.client.1 name: a2 - enabled_features: - resharding endpoints: - c2.client.0 is_default: true name: b zones: - endpoints: - c2.client.0 is_default: true is_master: true name: b1 - endpoints: - c2.client.1 name: b2 rgw-multisite-tests: args: - tests.py workunit: branch: reef-release sha1: 0484097b0572e96f42cf6402cbe0ac8dcb046577 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 seed: 2147 sha1: 0484097b0572e96f42cf6402cbe0ac8dcb046577 sleep_before_teardown: 0 subset: 111/12000 suite: rgw suite_branch: reef-release suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_0484097b0572e96f42cf6402cbe0ac8dcb046577/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 0484097b0572e96f42cf6402cbe0ac8dcb046577 targets: smithi101.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDciPeiA+F8OU2n9arsBRdxjittPCt9x6Dl/s+Hfxtf1oRgJdXHBnQWafNP5NjVmYg7hnf5zZnMl1HsH2pMMc+8= smithi139.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGcp7KnNK6sxlDspWEyXozyB+lgk0NC9jK/uC8FVE4JLB64hFISTyFr+uS0NN4bYfyCUHn21kkjENMNOmGNcFI4= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 61140164d2579c30ffff3970615e0f82e40c6470 timestamp: 2023-07-28_23:12:07 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23459 2023-07-28T23:22:29.807 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_0484097b0572e96f42cf6402cbe0ac8dcb046577/qa; will attempt to use it 2023-07-28T23:22:29.892 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_0484097b0572e96f42cf6402cbe0ac8dcb046577/qa/tasks 2023-07-28T23:22:29.892 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-07-28T23:22:29.894 INFO:teuthology.task.internal:Checking packages... 2023-07-28T23:22:29.924 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '0484097b0572e96f42cf6402cbe0ac8dcb046577' 2023-07-28T23:22:29.924 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-07-28T23:22:29.925 INFO:teuthology.packaging:ref: None 2023-07-28T23:22:29.925 INFO:teuthology.packaging:tag: None 2023-07-28T23:22:29.925 INFO:teuthology.packaging:branch: reef-release 2023-07-28T23:22:29.925 INFO:teuthology.packaging:sha1: 0484097b0572e96f42cf6402cbe0ac8dcb046577 2023-07-28T23:22:29.926 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef-release 2023-07-28T23:22:32.243 INFO:teuthology.task.internal:Found packages for ceph version 18.1.3-11.g0484097b 2023-07-28T23:22:32.244 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-07-28T23:22:32.253 INFO:teuthology.task.internal:no buildpackages task found 2023-07-28T23:22:32.254 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-07-28T23:22:32.267 INFO:teuthology.task.internal:Saving configuration 2023-07-28T23:22:32.281 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-07-28T23:22:32.291 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-28T23:22:40.772 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi101.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-28_23:12:07-rgw-reef-release-distro-default-smithi/7355843', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-28 23:17:01.281838', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDciPeiA+F8OU2n9arsBRdxjittPCt9x6Dl/s+Hfxtf1oRgJdXHBnQWafNP5NjVmYg7hnf5zZnMl1HsH2pMMc+8='} 2023-07-28T23:22:40.801 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi139.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-28_23:12:07-rgw-reef-release-distro-default-smithi/7355843', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-28 23:17:01.280354', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGcp7KnNK6sxlDspWEyXozyB+lgk0NC9jK/uC8FVE4JLB64hFISTyFr+uS0NN4bYfyCUHn21kkjENMNOmGNcFI4='} 2023-07-28T23:22:40.802 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-07-28T23:22:40.811 INFO:teuthology.task.internal:roles: ubuntu@smithi101.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2023-07-28T23:22:40.811 INFO:teuthology.task.internal:roles: ubuntu@smithi139.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2023-07-28T23:22:40.811 INFO:teuthology.run_tasks:Running task console_log... 2023-07-28T23:22:40.952 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9e27f7fe50>, signals=[15]) 2023-07-28T23:22:40.953 INFO:teuthology.run_tasks:Running task internal.connect... 2023-07-28T23:22:40.962 INFO:teuthology.task.internal:Opening connections... 2023-07-28T23:22:40.962 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi101.front.sepia.ceph.com 2023-07-28T23:22:40.964 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi101.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-28T23:22:41.050 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi139.front.sepia.ceph.com 2023-07-28T23:22:41.051 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-28T23:22:41.136 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-07-28T23:22:41.144 DEBUG:teuthology.orchestra.run.smithi101:> uname -m 2023-07-28T23:22:41.177 INFO:teuthology.orchestra.run.smithi101.stdout:x86_64 2023-07-28T23:22:41.178 DEBUG:teuthology.orchestra.run.smithi101:> cat /etc/os-release 2023-07-28T23:22:41.231 INFO:teuthology.orchestra.run.smithi101.stdout:NAME="CentOS Stream" 2023-07-28T23:22:41.232 INFO:teuthology.orchestra.run.smithi101.stdout:VERSION="9" 2023-07-28T23:22:41.232 INFO:teuthology.orchestra.run.smithi101.stdout:ID="centos" 2023-07-28T23:22:41.232 INFO:teuthology.orchestra.run.smithi101.stdout:ID_LIKE="rhel fedora" 2023-07-28T23:22:41.232 INFO:teuthology.orchestra.run.smithi101.stdout:VERSION_ID="9" 2023-07-28T23:22:41.233 INFO:teuthology.orchestra.run.smithi101.stdout:PLATFORM_ID="platform:el9" 2023-07-28T23:22:41.233 INFO:teuthology.orchestra.run.smithi101.stdout:PRETTY_NAME="CentOS Stream 9" 2023-07-28T23:22:41.233 INFO:teuthology.orchestra.run.smithi101.stdout:ANSI_COLOR="0;31" 2023-07-28T23:22:41.233 INFO:teuthology.orchestra.run.smithi101.stdout:LOGO="fedora-logo-icon" 2023-07-28T23:22:41.233 INFO:teuthology.orchestra.run.smithi101.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-07-28T23:22:41.233 INFO:teuthology.orchestra.run.smithi101.stdout:HOME_URL="https://centos.org/" 2023-07-28T23:22:41.233 INFO:teuthology.orchestra.run.smithi101.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-28T23:22:41.233 INFO:teuthology.orchestra.run.smithi101.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-07-28T23:22:41.234 INFO:teuthology.orchestra.run.smithi101.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-07-28T23:22:41.234 INFO:teuthology.lock.ops:Updating smithi101.front.sepia.ceph.com on lock server 2023-07-28T23:22:41.258 DEBUG:teuthology.orchestra.run.smithi139:> uname -m 2023-07-28T23:22:41.272 INFO:teuthology.orchestra.run.smithi139.stdout:x86_64 2023-07-28T23:22:41.273 DEBUG:teuthology.orchestra.run.smithi139:> cat /etc/os-release 2023-07-28T23:22:41.327 INFO:teuthology.orchestra.run.smithi139.stdout:NAME="CentOS Stream" 2023-07-28T23:22:41.328 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION="9" 2023-07-28T23:22:41.328 INFO:teuthology.orchestra.run.smithi139.stdout:ID="centos" 2023-07-28T23:22:41.328 INFO:teuthology.orchestra.run.smithi139.stdout:ID_LIKE="rhel fedora" 2023-07-28T23:22:41.328 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION_ID="9" 2023-07-28T23:22:41.328 INFO:teuthology.orchestra.run.smithi139.stdout:PLATFORM_ID="platform:el9" 2023-07-28T23:22:41.329 INFO:teuthology.orchestra.run.smithi139.stdout:PRETTY_NAME="CentOS Stream 9" 2023-07-28T23:22:41.329 INFO:teuthology.orchestra.run.smithi139.stdout:ANSI_COLOR="0;31" 2023-07-28T23:22:41.329 INFO:teuthology.orchestra.run.smithi139.stdout:LOGO="fedora-logo-icon" 2023-07-28T23:22:41.329 INFO:teuthology.orchestra.run.smithi139.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-07-28T23:22:41.329 INFO:teuthology.orchestra.run.smithi139.stdout:HOME_URL="https://centos.org/" 2023-07-28T23:22:41.329 INFO:teuthology.orchestra.run.smithi139.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-28T23:22:41.329 INFO:teuthology.orchestra.run.smithi139.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-07-28T23:22:41.330 INFO:teuthology.orchestra.run.smithi139.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-07-28T23:22:41.330 INFO:teuthology.lock.ops:Updating smithi139.front.sepia.ceph.com on lock server 2023-07-28T23:22:41.357 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-07-28T23:22:41.367 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-07-28T23:22:41.440 INFO:teuthology.task.internal:Checking for old test directory... 2023-07-28T23:22:41.441 DEBUG:teuthology.orchestra.run.smithi101:> test '!' -e /home/ubuntu/cephtest 2023-07-28T23:22:41.443 DEBUG:teuthology.orchestra.run.smithi139:> test '!' -e /home/ubuntu/cephtest 2023-07-28T23:22:41.457 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-07-28T23:22:41.465 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-07-28T23:22:41.465 DEBUG:teuthology.orchestra.run.smithi101:> test -z $(ls -A /var/lib/ceph) 2023-07-28T23:22:41.497 DEBUG:teuthology.orchestra.run.smithi139:> test -z $(ls -A /var/lib/ceph) 2023-07-28T23:22:41.523 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-07-28T23:22:41.615 INFO:teuthology.run_tasks:Running task kernel... 2023-07-28T23:22:41.632 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-07-28T23:22:41.633 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-07-28T23:22:41.633 DEBUG:teuthology.orchestra.run.smithi101:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-28T23:22:41.634 DEBUG:teuthology.orchestra.run.smithi139:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-28T23:22:41.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-28T23:22:41.648 DEBUG:teuthology.orchestra.run.smithi139:> uname -r 2023-07-28T23:22:41.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-28T23:22:41.649 DEBUG:teuthology.orchestra.run.smithi101:> uname -r 2023-07-28T23:22:41.702 INFO:teuthology.orchestra.run.smithi139.stdout:5.14.0-239.el9.x86_64 2023-07-28T23:22:41.703 INFO:teuthology.task.kernel:Running kernel on smithi139: 5.14.0-239.el9.x86_64 2023-07-28T23:22:41.703 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum install -y kernel 2023-07-28T23:22:41.704 INFO:teuthology.orchestra.run.smithi101.stdout:5.14.0-239.el9.x86_64 2023-07-28T23:22:41.704 INFO:teuthology.task.kernel:Running kernel on smithi101: 5.14.0-239.el9.x86_64 2023-07-28T23:22:41.705 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum install -y kernel 2023-07-28T23:22:44.463 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS-9 - AppStream 5.0 MB/s | 6.3 MB 00:01 2023-07-28T23:22:44.541 INFO:teuthology.orchestra.run.smithi101.stdout:CentOS-9 - AppStream 4.4 MB/s | 6.3 MB 00:01 2023-07-28T23:22:46.263 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS-9 - BaseOS 2.5 MB/s | 2.0 MB 00:00 2023-07-28T23:22:46.975 INFO:teuthology.orchestra.run.smithi101.stdout:CentOS-9 - BaseOS 1.4 MB/s | 2.0 MB 00:01 2023-07-28T23:22:47.770 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 9 - BaseOS 5.8 MB/s | 6.9 MB 00:01 2023-07-28T23:22:48.332 INFO:teuthology.orchestra.run.smithi101.stdout:CentOS Stream 9 - BaseOS 6.6 MB/s | 6.9 MB 00:01 2023-07-28T23:22:51.147 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 9 - AppStream 7.4 MB/s | 17 MB 00:02 2023-07-28T23:22:51.836 INFO:teuthology.orchestra.run.smithi101.stdout:CentOS Stream 9 - AppStream 7.0 MB/s | 17 MB 00:02 2023-07-28T23:22:55.581 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 9 - Extras packages 11 kB/s | 13 kB 00:01 2023-07-28T23:22:56.084 INFO:teuthology.orchestra.run.smithi101.stdout:CentOS Stream 9 - Extras packages 13 kB/s | 13 kB 00:01 2023-07-28T23:22:57.831 INFO:teuthology.orchestra.run.smithi139.stdout:Extra Packages for Enterprise Linux 8.6 MB/s | 18 MB 00:02 2023-07-28T23:22:58.076 INFO:teuthology.orchestra.run.smithi101.stdout:Extra Packages for Enterprise Linux 9.8 MB/s | 18 MB 00:01 2023-07-28T23:23:04.153 INFO:teuthology.orchestra.run.smithi139.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-07-28T23:23:04.173 INFO:teuthology.orchestra.run.smithi101.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-07-28T23:23:04.245 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2023-07-28T23:23:04.249 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-07-28T23:23:04.257 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-07-28T23:23:04.257 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2023-07-28T23:23:04.257 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-07-28T23:23:04.257 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2023-07-28T23:23:04.258 INFO:teuthology.orchestra.run.smithi139.stdout: kernel x86_64 5.14.0-344.el9 CentOS-BaseOS 4.8 M 2023-07-28T23:23:04.258 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2023-07-28T23:23:04.258 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-core x86_64 5.14.0-344.el9 CentOS-BaseOS 19 M 2023-07-28T23:23:04.258 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-modules x86_64 5.14.0-344.el9 CentOS-BaseOS 37 M 2023-07-28T23:23:04.258 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-modules-core x86_64 5.14.0-344.el9 CentOS-BaseOS 31 M 2023-07-28T23:23:04.258 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:23:04.258 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2023-07-28T23:23:04.259 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-07-28T23:23:04.259 INFO:teuthology.orchestra.run.smithi139.stdout:Install 4 Packages 2023-07-28T23:23:04.259 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:23:04.260 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 93 M 2023-07-28T23:23:04.260 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 120 M 2023-07-28T23:23:04.260 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2023-07-28T23:23:04.260 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-07-28T23:23:04.260 INFO:teuthology.orchestra.run.smithi101.stdout: Package Arch Version Repository Size 2023-07-28T23:23:04.261 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-07-28T23:23:04.261 INFO:teuthology.orchestra.run.smithi101.stdout:Installing: 2023-07-28T23:23:04.261 INFO:teuthology.orchestra.run.smithi101.stdout: kernel x86_64 5.14.0-344.el9 CentOS-BaseOS 4.8 M 2023-07-28T23:23:04.261 INFO:teuthology.orchestra.run.smithi101.stdout:Installing dependencies: 2023-07-28T23:23:04.261 INFO:teuthology.orchestra.run.smithi101.stdout: kernel-core x86_64 5.14.0-344.el9 CentOS-BaseOS 19 M 2023-07-28T23:23:04.261 INFO:teuthology.orchestra.run.smithi101.stdout: kernel-modules x86_64 5.14.0-344.el9 CentOS-BaseOS 37 M 2023-07-28T23:23:04.261 INFO:teuthology.orchestra.run.smithi101.stdout: kernel-modules-core x86_64 5.14.0-344.el9 CentOS-BaseOS 31 M 2023-07-28T23:23:04.261 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:23:04.262 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2023-07-28T23:23:04.262 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-07-28T23:23:04.262 INFO:teuthology.orchestra.run.smithi101.stdout:Install 4 Packages 2023-07-28T23:23:04.262 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:23:04.262 INFO:teuthology.orchestra.run.smithi101.stdout:Total download size: 93 M 2023-07-28T23:23:04.262 INFO:teuthology.orchestra.run.smithi101.stdout:Installed size: 120 M 2023-07-28T23:23:04.262 INFO:teuthology.orchestra.run.smithi101.stdout:Downloading Packages: 2023-07-28T23:23:05.011 INFO:teuthology.orchestra.run.smithi101.stdout:(1/4): kernel-5.14.0-344.el9.x86_64.rpm 6.4 MB/s | 4.8 MB 00:00 2023-07-28T23:23:05.032 INFO:teuthology.orchestra.run.smithi139.stdout:(1/4): kernel-5.14.0-344.el9.x86_64.rpm 6.3 MB/s | 4.8 MB 00:00 2023-07-28T23:23:06.165 INFO:teuthology.orchestra.run.smithi139.stdout:(2/4): kernel-core-5.14.0-344.el9.x86_64.rpm 10 MB/s | 19 MB 00:01 2023-07-28T23:23:07.844 INFO:teuthology.orchestra.run.smithi101.stdout:(2/4): kernel-core-5.14.0-344.el9.x86_64.rpm 5.4 MB/s | 19 MB 00:03 2023-07-28T23:23:08.424 INFO:teuthology.orchestra.run.smithi139.stdout:(3/4): kernel-modules-core-5.14.0-344.el9.x86_6 9.3 MB/s | 31 MB 00:03 2023-07-28T23:23:09.115 INFO:teuthology.orchestra.run.smithi139.stdout:(4/4): kernel-modules-5.14.0-344.el9.x86_64.rpm 7.6 MB/s | 37 MB 00:04 2023-07-28T23:23:09.118 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2023-07-28T23:23:09.119 INFO:teuthology.orchestra.run.smithi139.stdout:Total 19 MB/s | 93 MB 00:04 2023-07-28T23:23:09.119 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2023-07-28T23:23:09.463 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2023-07-28T23:23:09.463 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2023-07-28T23:23:10.270 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2023-07-28T23:23:10.270 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2023-07-28T23:23:10.402 INFO:teuthology.orchestra.run.smithi101.stdout:(3/4): kernel-modules-core-5.14.0-344.el9.x86_6 5.8 MB/s | 31 MB 00:05 2023-07-28T23:23:11.277 INFO:teuthology.orchestra.run.smithi101.stdout:(4/4): kernel-modules-5.14.0-344.el9.x86_64.rpm 5.3 MB/s | 37 MB 00:07 2023-07-28T23:23:11.281 INFO:teuthology.orchestra.run.smithi101.stdout:-------------------------------------------------------------------------------- 2023-07-28T23:23:11.281 INFO:teuthology.orchestra.run.smithi101.stdout:Total 13 MB/s | 93 MB 00:07 2023-07-28T23:23:11.282 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction check 2023-07-28T23:23:11.636 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction check succeeded. 2023-07-28T23:23:11.636 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction test 2023-07-28T23:23:12.431 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction test succeeded. 2023-07-28T23:23:12.431 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction 2023-07-28T23:23:12.474 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2023-07-28T23:23:13.122 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : kernel-modules-core-5.14.0-344.el9.x86_64 1/4 2023-07-28T23:23:13.256 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : kernel-core-5.14.0-344.el9.x86_64 2/4 2023-07-28T23:23:14.595 INFO:teuthology.orchestra.run.smithi101.stdout: Preparing : 1/1 2023-07-28T23:23:15.333 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : kernel-modules-core-5.14.0-344.el9.x86_64 1/4 2023-07-28T23:23:15.452 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : kernel-core-5.14.0-344.el9.x86_64 2/4 2023-07-28T23:23:15.489 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-core-5.14.0-344.el9.x86_64 2/4 2023-07-28T23:23:15.652 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : kernel-modules-5.14.0-344.el9.x86_64 3/4 2023-07-28T23:23:17.685 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: kernel-core-5.14.0-344.el9.x86_64 2/4 2023-07-28T23:23:17.857 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : kernel-modules-5.14.0-344.el9.x86_64 3/4 2023-07-28T23:23:20.199 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-modules-5.14.0-344.el9.x86_64 3/4 2023-07-28T23:23:20.326 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : kernel-5.14.0-344.el9.x86_64 4/4 2023-07-28T23:23:22.423 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: kernel-modules-5.14.0-344.el9.x86_64 3/4 2023-07-28T23:23:22.715 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : kernel-5.14.0-344.el9.x86_64 4/4 2023-07-28T23:23:24.920 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-modules-core-5.14.0-344.el9.x86_64 4/4 2023-07-28T23:23:27.291 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: kernel-modules-core-5.14.0-344.el9.x86_64 4/4 2023-07-28T23:23:53.898 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-core-5.14.0-344.el9.x86_64 4/4 2023-07-28T23:23:53.967 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-modules-5.14.0-344.el9.x86_64 4/4 2023-07-28T23:23:54.518 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-5.14.0-344.el9.x86_64 4/4 2023-07-28T23:23:54.519 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : kernel-5.14.0-344.el9.x86_64 1/4 2023-07-28T23:23:54.519 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : kernel-core-5.14.0-344.el9.x86_64 2/4 2023-07-28T23:23:54.519 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : kernel-modules-5.14.0-344.el9.x86_64 3/4 2023-07-28T23:23:54.983 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : kernel-modules-core-5.14.0-344.el9.x86_64 4/4 2023-07-28T23:23:54.983 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:23:54.983 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2023-07-28T23:23:54.984 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-5.14.0-344.el9.x86_64 2023-07-28T23:23:54.984 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-core-5.14.0-344.el9.x86_64 2023-07-28T23:23:54.984 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-modules-5.14.0-344.el9.x86_64 2023-07-28T23:23:54.984 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-modules-core-5.14.0-344.el9.x86_64 2023-07-28T23:23:54.984 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:23:54.984 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2023-07-28T23:23:55.585 DEBUG:teuthology.orchestra.run.smithi139:> rpm -q kernel | sort -rV | head -n 1 2023-07-28T23:23:55.669 INFO:teuthology.orchestra.run.smithi139.stdout:kernel-5.14.0-344.el9.x86_64 2023-07-28T23:23:55.670 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-344.el9.x86_64 2023-07-28T23:23:55.670 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-239.el9.x86_64 Expected: 5.14.0-344.el9.x86_64 2023-07-28T23:23:55.670 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-07-28T23:23:55.671 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-07-28T23:23:55.671 INFO:teuthology.task.kernel:Installing distro kernel on c2.mon.a... 2023-07-28T23:23:55.671 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi139.front.sepia.ceph.com, path=None, version=distro) 2023-07-28T23:23:55.671 DEBUG:teuthology.orchestra.run.smithi139:> rpm -q kernel | sort -rV | head -n 1 2023-07-28T23:23:55.739 INFO:teuthology.orchestra.run.smithi139.stdout:kernel-5.14.0-344.el9.x86_64 2023-07-28T23:23:55.739 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-344.el9.x86_64 2023-07-28T23:23:55.739 DEBUG:teuthology.orchestra.run.smithi139:> sudo rpm -qi grub2-tools 2023-07-28T23:23:55.773 INFO:teuthology.orchestra.run.smithi139.stdout:Name : grub2-tools 2023-07-28T23:23:55.773 INFO:teuthology.orchestra.run.smithi139.stdout:Epoch : 1 2023-07-28T23:23:55.773 INFO:teuthology.orchestra.run.smithi139.stdout:Version : 2.06 2023-07-28T23:23:55.773 INFO:teuthology.orchestra.run.smithi139.stdout:Release : 53.el9 2023-07-28T23:23:55.774 INFO:teuthology.orchestra.run.smithi139.stdout:Architecture: x86_64 2023-07-28T23:23:55.774 INFO:teuthology.orchestra.run.smithi139.stdout:Install Date: Mon 30 Jan 2023 10:38:40 AM UTC 2023-07-28T23:23:55.774 INFO:teuthology.orchestra.run.smithi139.stdout:Group : Unspecified 2023-07-28T23:23:55.774 INFO:teuthology.orchestra.run.smithi139.stdout:Size : 8551923 2023-07-28T23:23:55.774 INFO:teuthology.orchestra.run.smithi139.stdout:License : GPLv3+ 2023-07-28T23:23:55.774 INFO:teuthology.orchestra.run.smithi139.stdout:Signature : RSA/SHA256, Wed 18 Jan 2023 04:40:07 PM UTC, Key ID 05b555b38483c65d 2023-07-28T23:23:55.774 INFO:teuthology.orchestra.run.smithi139.stdout:Source RPM : grub2-2.06-53.el9.src.rpm 2023-07-28T23:23:55.775 INFO:teuthology.orchestra.run.smithi139.stdout:Build Date : Fri 13 Jan 2023 09:07:28 PM UTC 2023-07-28T23:23:55.775 INFO:teuthology.orchestra.run.smithi139.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-07-28T23:23:55.775 INFO:teuthology.orchestra.run.smithi139.stdout:Packager : builder@centos.org 2023-07-28T23:23:55.775 INFO:teuthology.orchestra.run.smithi139.stdout:Vendor : CentOS 2023-07-28T23:23:55.775 INFO:teuthology.orchestra.run.smithi139.stdout:URL : http://www.gnu.org/software/grub/ 2023-07-28T23:23:55.775 INFO:teuthology.orchestra.run.smithi139.stdout:Summary : Support tools for GRUB. 2023-07-28T23:23:55.775 INFO:teuthology.orchestra.run.smithi139.stdout:Description : 2023-07-28T23:23:55.775 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:23:55.776 INFO:teuthology.orchestra.run.smithi139.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-07-28T23:23:55.776 INFO:teuthology.orchestra.run.smithi139.stdout:customizable bootloader with modular architecture. It supports a rich 2023-07-28T23:23:55.776 INFO:teuthology.orchestra.run.smithi139.stdout:variety of kernel formats, file systems, computer architectures and 2023-07-28T23:23:55.776 INFO:teuthology.orchestra.run.smithi139.stdout:hardware devices. 2023-07-28T23:23:55.776 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:23:55.776 INFO:teuthology.orchestra.run.smithi139.stdout:This subpackage provides tools for support of all platforms. 2023-07-28T23:23:55.777 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-07-28T23:23:55.777 INFO:teuthology.task.kernel:Updating grub on smithi139 to boot 5.14.0-344.el9.x86_64 2023-07-28T23:23:55.777 DEBUG:teuthology.orchestra.run.smithi139:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-07-28T23:23:56.168 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: kernel-core-5.14.0-344.el9.x86_64 4/4 2023-07-28T23:23:56.209 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: kernel-modules-5.14.0-344.el9.x86_64 4/4 2023-07-28T23:23:56.518 INFO:teuthology.orchestra.run.smithi139.stderr:Generating grub configuration file ... 2023-07-28T23:23:56.875 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: kernel-5.14.0-344.el9.x86_64 4/4 2023-07-28T23:23:56.876 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : kernel-5.14.0-344.el9.x86_64 1/4 2023-07-28T23:23:56.876 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : kernel-core-5.14.0-344.el9.x86_64 2/4 2023-07-28T23:23:56.876 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : kernel-modules-5.14.0-344.el9.x86_64 3/4 2023-07-28T23:23:57.344 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : kernel-modules-core-5.14.0-344.el9.x86_64 4/4 2023-07-28T23:23:57.345 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:23:57.345 INFO:teuthology.orchestra.run.smithi101.stdout:Installed: 2023-07-28T23:23:57.345 INFO:teuthology.orchestra.run.smithi101.stdout: kernel-5.14.0-344.el9.x86_64 2023-07-28T23:23:57.345 INFO:teuthology.orchestra.run.smithi101.stdout: kernel-core-5.14.0-344.el9.x86_64 2023-07-28T23:23:57.345 INFO:teuthology.orchestra.run.smithi101.stdout: kernel-modules-5.14.0-344.el9.x86_64 2023-07-28T23:23:57.345 INFO:teuthology.orchestra.run.smithi101.stdout: kernel-modules-core-5.14.0-344.el9.x86_64 2023-07-28T23:23:57.345 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:23:57.346 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-07-28T23:23:57.828 DEBUG:teuthology.orchestra.run.smithi101:> rpm -q kernel | sort -rV | head -n 1 2023-07-28T23:23:57.906 INFO:teuthology.orchestra.run.smithi139.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-07-28T23:23:57.913 INFO:teuthology.orchestra.run.smithi101.stdout:kernel-5.14.0-344.el9.x86_64 2023-07-28T23:23:57.913 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-344.el9.x86_64 2023-07-28T23:23:57.914 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-239.el9.x86_64 Expected: 5.14.0-344.el9.x86_64 2023-07-28T23:23:57.914 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-07-28T23:23:57.914 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-07-28T23:23:57.914 INFO:teuthology.task.kernel:Installing distro kernel on c1.mon.a... 2023-07-28T23:23:57.914 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi101.front.sepia.ceph.com, path=None, version=distro) 2023-07-28T23:23:57.914 DEBUG:teuthology.orchestra.run.smithi101:> rpm -q kernel | sort -rV | head -n 1 2023-07-28T23:23:57.942 INFO:teuthology.orchestra.run.smithi139.stderr:done 2023-07-28T23:23:57.953 DEBUG:teuthology.orchestra.run.smithi139:> mktemp 2023-07-28T23:23:57.982 INFO:teuthology.orchestra.run.smithi101.stdout:kernel-5.14.0-344.el9.x86_64 2023-07-28T23:23:57.982 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-344.el9.x86_64 2023-07-28T23:23:57.982 DEBUG:teuthology.orchestra.run.smithi101:> sudo rpm -qi grub2-tools 2023-07-28T23:23:58.008 INFO:teuthology.orchestra.run.smithi139.stdout:/tmp/tmp.wVekQdVjpd 2023-07-28T23:23:58.008 DEBUG:teuthology.orchestra.run.smithi139:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.wVekQdVjpd 2023-07-28T23:23:58.026 INFO:teuthology.orchestra.run.smithi101.stdout:Name : grub2-tools 2023-07-28T23:23:58.026 INFO:teuthology.orchestra.run.smithi101.stdout:Epoch : 1 2023-07-28T23:23:58.026 INFO:teuthology.orchestra.run.smithi101.stdout:Version : 2.06 2023-07-28T23:23:58.026 INFO:teuthology.orchestra.run.smithi101.stdout:Release : 53.el9 2023-07-28T23:23:58.026 INFO:teuthology.orchestra.run.smithi101.stdout:Architecture: x86_64 2023-07-28T23:23:58.026 INFO:teuthology.orchestra.run.smithi101.stdout:Install Date: Mon 30 Jan 2023 10:38:40 AM UTC 2023-07-28T23:23:58.027 INFO:teuthology.orchestra.run.smithi101.stdout:Group : Unspecified 2023-07-28T23:23:58.027 INFO:teuthology.orchestra.run.smithi101.stdout:Size : 8551923 2023-07-28T23:23:58.027 INFO:teuthology.orchestra.run.smithi101.stdout:License : GPLv3+ 2023-07-28T23:23:58.027 INFO:teuthology.orchestra.run.smithi101.stdout:Signature : RSA/SHA256, Wed 18 Jan 2023 04:40:07 PM UTC, Key ID 05b555b38483c65d 2023-07-28T23:23:58.027 INFO:teuthology.orchestra.run.smithi101.stdout:Source RPM : grub2-2.06-53.el9.src.rpm 2023-07-28T23:23:58.027 INFO:teuthology.orchestra.run.smithi101.stdout:Build Date : Fri 13 Jan 2023 09:07:28 PM UTC 2023-07-28T23:23:58.027 INFO:teuthology.orchestra.run.smithi101.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-07-28T23:23:58.028 INFO:teuthology.orchestra.run.smithi101.stdout:Packager : builder@centos.org 2023-07-28T23:23:58.028 INFO:teuthology.orchestra.run.smithi101.stdout:Vendor : CentOS 2023-07-28T23:23:58.028 INFO:teuthology.orchestra.run.smithi101.stdout:URL : http://www.gnu.org/software/grub/ 2023-07-28T23:23:58.028 INFO:teuthology.orchestra.run.smithi101.stdout:Summary : Support tools for GRUB. 2023-07-28T23:23:58.028 INFO:teuthology.orchestra.run.smithi101.stdout:Description : 2023-07-28T23:23:58.028 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:23:58.028 INFO:teuthology.orchestra.run.smithi101.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-07-28T23:23:58.028 INFO:teuthology.orchestra.run.smithi101.stdout:customizable bootloader with modular architecture. It supports a rich 2023-07-28T23:23:58.029 INFO:teuthology.orchestra.run.smithi101.stdout:variety of kernel formats, file systems, computer architectures and 2023-07-28T23:23:58.029 INFO:teuthology.orchestra.run.smithi101.stdout:hardware devices. 2023-07-28T23:23:58.029 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:23:58.029 INFO:teuthology.orchestra.run.smithi101.stdout:This subpackage provides tools for support of all platforms. 2023-07-28T23:23:58.030 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-07-28T23:23:58.030 INFO:teuthology.task.kernel:Updating grub on smithi101 to boot 5.14.0-344.el9.x86_64 2023-07-28T23:23:58.030 DEBUG:teuthology.orchestra.run.smithi101:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-07-28T23:23:58.036 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod 0666 /tmp/tmp.wVekQdVjpd 2023-07-28T23:23:58.181 DEBUG:teuthology.orchestra.remote:smithi139:/tmp/tmp.wVekQdVjpd is 6KB 2023-07-28T23:23:58.199 DEBUG:teuthology.orchestra.run.smithi139:> rm -fr /tmp/tmp.wVekQdVjpd 2023-07-28T23:23:58.212 DEBUG:teuthology.orchestra.run.smithi139:> sudo /bin/ls /boot/loader/entries || true 2023-07-28T23:23:58.279 INFO:teuthology.orchestra.run.smithi139.stdout:e13cba1d25f14c51a1d04da8db823a4e-0-rescue.conf 2023-07-28T23:23:58.280 INFO:teuthology.orchestra.run.smithi139.stdout:e13cba1d25f14c51a1d04da8db823a4e-5.14.0-239.el9.x86_64.conf 2023-07-28T23:23:58.280 INFO:teuthology.orchestra.run.smithi139.stdout:e13cba1d25f14c51a1d04da8db823a4e-5.14.0-344.el9.x86_64.conf 2023-07-28T23:23:58.281 DEBUG:teuthology.orchestra.run.smithi139:> sudo grub2-set-default e13cba1d25f14c51a1d04da8db823a4e-5.14.0-344.el9.x86_64 2023-07-28T23:23:58.460 DEBUG:teuthology.orchestra.run.smithi139:> sudo shutdown -r now 2023-07-28T23:23:58.755 INFO:teuthology.orchestra.run.smithi101.stderr:Generating grub configuration file ... 2023-07-28T23:24:00.165 INFO:teuthology.orchestra.run.smithi101.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-07-28T23:24:00.201 INFO:teuthology.orchestra.run.smithi101.stderr:done 2023-07-28T23:24:00.203 DEBUG:teuthology.orchestra.run.smithi101:> mktemp 2023-07-28T23:24:00.218 INFO:teuthology.orchestra.run.smithi101.stdout:/tmp/tmp.zeAT3spXuq 2023-07-28T23:24:00.218 DEBUG:teuthology.orchestra.run.smithi101:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.zeAT3spXuq 2023-07-28T23:24:00.286 DEBUG:teuthology.orchestra.run.smithi101:> sudo chmod 0666 /tmp/tmp.zeAT3spXuq 2023-07-28T23:24:00.424 DEBUG:teuthology.orchestra.remote:smithi101:/tmp/tmp.zeAT3spXuq is 6KB 2023-07-28T23:24:00.441 DEBUG:teuthology.orchestra.run.smithi101:> rm -fr /tmp/tmp.zeAT3spXuq 2023-07-28T23:24:00.455 DEBUG:teuthology.orchestra.run.smithi101:> sudo /bin/ls /boot/loader/entries || true 2023-07-28T23:24:00.523 INFO:teuthology.orchestra.run.smithi101.stdout:e13cba1d25f14c51a1d04da8db823a4e-0-rescue.conf 2023-07-28T23:24:00.523 INFO:teuthology.orchestra.run.smithi101.stdout:e13cba1d25f14c51a1d04da8db823a4e-5.14.0-239.el9.x86_64.conf 2023-07-28T23:24:00.523 INFO:teuthology.orchestra.run.smithi101.stdout:e13cba1d25f14c51a1d04da8db823a4e-5.14.0-344.el9.x86_64.conf 2023-07-28T23:24:00.524 DEBUG:teuthology.orchestra.run.smithi101:> sudo grub2-set-default e13cba1d25f14c51a1d04da8db823a4e-5.14.0-344.el9.x86_64 2023-07-28T23:24:00.683 DEBUG:teuthology.orchestra.run.smithi101:> sudo shutdown -r now 2023-07-28T23:24:28.488 INFO:teuthology.task.kernel:Checking client c2.mon.a for new kernel version... 2023-07-28T23:24:28.489 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com' 2023-07-28T23:24:28.489 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-28T23:24:33.495 INFO:teuthology.task.kernel:Checking client c1.mon.a for new kernel version... 2023-07-28T23:24:33.496 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi101.front.sepia.ceph.com' 2023-07-28T23:24:33.496 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi101.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-28T23:25:33.552 DEBUG:teuthology.orchestra.remote:timed out 2023-07-28T23:25:33.554 DEBUG:teuthology.orchestra.remote:timed out 2023-07-28T23:25:42.560 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com' 2023-07-28T23:25:42.561 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-28T23:25:47.564 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi101.front.sepia.ceph.com' 2023-07-28T23:25:47.565 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi101.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-28T23:25:57.822 DEBUG:teuthology.orchestra.run.smithi101:> true 2023-07-28T23:25:57.898 DEBUG:teuthology.orchestra.run.smithi139:> true 2023-07-28T23:25:58.170 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi101.front.sepia.ceph.com' 2023-07-28T23:25:58.170 INFO:teuthology.task.kernel:Checking kernel version of c1.mon.a, want "5.14.0-344.el9.x86_64"... 2023-07-28T23:25:58.171 DEBUG:teuthology.orchestra.run.smithi101:> uname -r 2023-07-28T23:25:58.185 INFO:teuthology.orchestra.run.smithi101.stdout:5.14.0-344.el9.x86_64 2023-07-28T23:25:58.185 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-344.el9.x86_64 vs 5.14.0-344.el9.x86_64 2023-07-28T23:25:58.186 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-07-28T23:25:58.186 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-07-28T23:25:58.254 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi139.front.sepia.ceph.com' 2023-07-28T23:25:58.254 INFO:teuthology.task.kernel:Checking kernel version of c2.mon.a, want "5.14.0-344.el9.x86_64"... 2023-07-28T23:25:58.255 DEBUG:teuthology.orchestra.run.smithi139:> uname -r 2023-07-28T23:25:58.269 INFO:teuthology.orchestra.run.smithi139.stdout:5.14.0-344.el9.x86_64 2023-07-28T23:25:58.269 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-344.el9.x86_64 vs 5.14.0-344.el9.x86_64 2023-07-28T23:25:58.270 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-07-28T23:25:58.270 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-07-28T23:25:59.186 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2023-07-28T23:25:59.187 DEBUG:teuthology.orchestra.run.smithi101:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-07-28T23:25:59.269 INFO:teuthology.orchestra.run.smithi101.stdout:ttyS1 2023-07-28T23:25:59.270 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2023-07-28T23:25:59.271 DEBUG:teuthology.orchestra.run.smithi139:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-07-28T23:25:59.302 DEBUG:teuthology.parallel:result is None 2023-07-28T23:25:59.355 INFO:teuthology.orchestra.run.smithi139.stdout:ttyS1 2023-07-28T23:25:59.384 DEBUG:teuthology.parallel:result is None 2023-07-28T23:25:59.384 INFO:teuthology.run_tasks:Running task internal.base... 2023-07-28T23:25:59.398 INFO:teuthology.task.internal:Creating test directory... 2023-07-28T23:25:59.398 DEBUG:teuthology.orchestra.run.smithi101:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-28T23:25:59.401 DEBUG:teuthology.orchestra.run.smithi139:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-28T23:25:59.418 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-07-28T23:25:59.425 INFO:teuthology.run_tasks:Running task internal.archive... 2023-07-28T23:25:59.435 INFO:teuthology.task.internal:Creating archive directory... 2023-07-28T23:25:59.435 DEBUG:teuthology.orchestra.run.smithi101:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-28T23:25:59.463 DEBUG:teuthology.orchestra.run.smithi139:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-28T23:25:59.493 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-07-28T23:25:59.503 INFO:teuthology.task.internal:Enabling coredump saving... 2023-07-28T23:25:59.504 DEBUG:teuthology.orchestra.run.smithi101:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-28T23:25:59.525 DEBUG:teuthology.orchestra.run.smithi139:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-28T23:25:59.559 INFO:teuthology.orchestra.run.smithi101.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-28T23:25:59.572 INFO:teuthology.orchestra.run.smithi139.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-28T23:25:59.572 INFO:teuthology.orchestra.run.smithi101.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-28T23:25:59.585 INFO:teuthology.orchestra.run.smithi139.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-28T23:25:59.587 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-07-28T23:25:59.605 INFO:teuthology.task.internal:Configuring sudo... 2023-07-28T23:25:59.606 DEBUG:teuthology.orchestra.run.smithi101:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-28T23:25:59.617 DEBUG:teuthology.orchestra.run.smithi139:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-28T23:25:59.658 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-07-28T23:25:59.672 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-07-28T23:25:59.673 DEBUG:teuthology.orchestra.run.smithi101:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-28T23:25:59.686 DEBUG:teuthology.orchestra.run.smithi139:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-28T23:25:59.713 DEBUG:teuthology.orchestra.run.smithi101:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-28T23:26:04.793 DEBUG:teuthology.orchestra.run.smithi101:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-28T23:26:04.838 DEBUG:teuthology.orchestra.run.smithi101:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-28T23:26:09.932 DEBUG:teuthology.orchestra.run.smithi101:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-28T23:26:09.964 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-07-28T23:26:09.964 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-28T23:26:10.035 DEBUG:teuthology.orchestra.run.smithi139:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-28T23:26:15.086 DEBUG:teuthology.orchestra.run.smithi139:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-28T23:26:15.194 DEBUG:teuthology.orchestra.run.smithi139:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-28T23:26:20.243 DEBUG:teuthology.orchestra.run.smithi139:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-28T23:26:20.276 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2023-07-28T23:26:20.276 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-28T23:26:20.345 DEBUG:teuthology.orchestra.run.smithi101:> sudo service rsyslog restart 2023-07-28T23:26:20.348 DEBUG:teuthology.orchestra.run.smithi139:> sudo service rsyslog restart 2023-07-28T23:26:20.430 INFO:teuthology.orchestra.run.smithi139.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-28T23:26:20.611 INFO:teuthology.orchestra.run.smithi101.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-28T23:26:20.714 INFO:teuthology.run_tasks:Running task internal.timer... 2023-07-28T23:26:20.725 INFO:teuthology.task.internal:Starting timer... 2023-07-28T23:26:20.725 INFO:teuthology.run_tasks:Running task pcp... 2023-07-28T23:26:20.736 INFO:teuthology.run_tasks:Running task selinux... 2023-07-28T23:26:20.747 DEBUG:teuthology.orchestra.run.smithi101:> sudo service auditd rotate 2023-07-28T23:26:20.802 INFO:teuthology.orchestra.run.smithi101.stdout:Rotating logs: 2023-07-28T23:26:20.804 DEBUG:teuthology.orchestra.run.smithi139:> sudo service auditd rotate 2023-07-28T23:26:20.848 INFO:teuthology.orchestra.run.smithi139.stdout:Rotating logs: 2023-07-28T23:26:20.849 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-07-28T23:26:20.850 DEBUG:teuthology.orchestra.run.smithi101:> /usr/sbin/getenforce 2023-07-28T23:26:20.877 INFO:teuthology.orchestra.run.smithi101.stdout:Permissive 2023-07-28T23:26:20.877 DEBUG:teuthology.orchestra.run.smithi139:> /usr/sbin/getenforce 2023-07-28T23:26:20.911 INFO:teuthology.orchestra.run.smithi139.stdout:Permissive 2023-07-28T23:26:20.912 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi101.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi139.front.sepia.ceph.com': 'permissive'} 2023-07-28T23:26:20.912 DEBUG:teuthology.orchestra.run.smithi101:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-28T23:26:20.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-28T23:26:20.947 DEBUG:teuthology.orchestra.run.smithi139:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-28T23:26:20.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-28T23:26:20.978 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-07-28T23:26:20.979 DEBUG:teuthology.orchestra.run.smithi101:> sudo /usr/sbin/setenforce permissive 2023-07-28T23:26:21.021 DEBUG:teuthology.orchestra.run.smithi139:> sudo /usr/sbin/setenforce permissive 2023-07-28T23:26:21.048 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-07-28T23:26:21.057 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-07-28T23:26:21.058 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-07-28T23:26:21.078 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-07-28T23:26:21.080 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi101.front.sepia.ceph.com,smithi139.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-07-28T23:33:10.373 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi101.front.sepia.ceph.com'), Remote(name='ubuntu@smithi139.front.sepia.ceph.com')] 2023-07-28T23:33:10.375 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi101.front.sepia.ceph.com' 2023-07-28T23:33:10.376 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi101.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-28T23:33:10.459 DEBUG:teuthology.orchestra.run.smithi101:> true 2023-07-28T23:33:10.483 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi101.front.sepia.ceph.com' 2023-07-28T23:33:10.484 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com' 2023-07-28T23:33:10.485 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-28T23:33:10.568 DEBUG:teuthology.orchestra.run.smithi139:> true 2023-07-28T23:33:10.639 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi139.front.sepia.ceph.com' 2023-07-28T23:33:10.639 INFO:teuthology.run_tasks:Running task clock... 2023-07-28T23:33:10.650 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-07-28T23:33:10.650 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-28T23:33:10.651 DEBUG:teuthology.orchestra.run.smithi101:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-28T23:33:10.653 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-28T23:33:10.653 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-28T23:33:10.682 INFO:teuthology.orchestra.run.smithi101.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-28T23:33:10.699 INFO:teuthology.orchestra.run.smithi101.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-28T23:33:10.711 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-28T23:33:10.727 INFO:teuthology.orchestra.run.smithi101.stderr:sudo: ntpd: command not found 2023-07-28T23:33:10.728 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-28T23:33:10.741 INFO:teuthology.orchestra.run.smithi101.stdout:506 Cannot talk to daemon 2023-07-28T23:33:10.757 INFO:teuthology.orchestra.run.smithi101.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-28T23:33:10.757 INFO:teuthology.orchestra.run.smithi139.stderr:sudo: ntpd: command not found 2023-07-28T23:33:10.771 INFO:teuthology.orchestra.run.smithi139.stdout:506 Cannot talk to daemon 2023-07-28T23:33:10.773 INFO:teuthology.orchestra.run.smithi101.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-28T23:33:10.787 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-28T23:33:10.803 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-28T23:33:10.817 INFO:teuthology.orchestra.run.smithi101.stderr:bash: line 1: ntpq: command not found 2023-07-28T23:33:10.820 INFO:teuthology.orchestra.run.smithi101.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-28T23:33:10.820 INFO:teuthology.orchestra.run.smithi101.stdout:=============================================================================== 2023-07-28T23:33:10.820 INFO:teuthology.orchestra.run.smithi101.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-28T23:33:10.820 INFO:teuthology.orchestra.run.smithi101.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-28T23:33:10.821 INFO:teuthology.orchestra.run.smithi101.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-28T23:33:10.821 INFO:teuthology.orchestra.run.smithi101.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-28T23:33:10.847 INFO:teuthology.orchestra.run.smithi139.stderr:bash: line 1: ntpq: command not found 2023-07-28T23:33:10.850 INFO:teuthology.orchestra.run.smithi139.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-28T23:33:10.850 INFO:teuthology.orchestra.run.smithi139.stdout:=============================================================================== 2023-07-28T23:33:10.850 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-28T23:33:10.850 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-28T23:33:10.851 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-28T23:33:10.851 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-28T23:33:10.851 INFO:teuthology.run_tasks:Running task install... 2023-07-28T23:33:10.860 DEBUG:teuthology.task.install:project ceph 2023-07-28T23:33:10.860 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '0484097b0572e96f42cf6402cbe0ac8dcb046577'}} 2023-07-28T23:33:10.861 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '0484097b0572e96f42cf6402cbe0ac8dcb046577'} 2023-07-28T23:33:10.861 INFO:teuthology.task.install:Using flavor: default 2023-07-28T23:33:10.901 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-07-28T23:33:10.902 INFO:teuthology.task.install:extra packages: [] 2023-07-28T23:33:10.902 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0484097b0572e96f42cf6402cbe0ac8dcb046577', 'tag': None, 'wait_for_package': False} 2023-07-28T23:33:10.903 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=0484097b0572e96f42cf6402cbe0ac8dcb046577 2023-07-28T23:33:10.905 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0484097b0572e96f42cf6402cbe0ac8dcb046577', 'tag': None, 'wait_for_package': False} 2023-07-28T23:33:10.906 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=0484097b0572e96f42cf6402cbe0ac8dcb046577 2023-07-28T23:33:11.204 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/reef-release/0484097b0572e96f42cf6402cbe0ac8dcb046577/centos/9/flavors/default/ 2023-07-28T23:33:11.205 INFO:teuthology.task.install.rpm:Package version is 18.1.3-11.g0484097b 2023-07-28T23:33:11.254 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/reef-release/0484097b0572e96f42cf6402cbe0ac8dcb046577/centos/9/flavors/default/ 2023-07-28T23:33:11.254 INFO:teuthology.task.install.rpm:Package version is 18.1.3-11.g0484097b 2023-07-28T23:33:16.748 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/reef-release/0484097b0572e96f42cf6402cbe0ac8dcb046577/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/reef-release/0484097b0572e96f42cf6402cbe0ac8dcb046577/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/reef-release/0484097b0572e96f42cf6402cbe0ac8dcb046577/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-28T23:33:16.748 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2023-07-28T23:33:16.749 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-28T23:33:16.781 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-07-28T23:33:16.782 DEBUG:teuthology.orchestra.run.smithi139:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/0484097b0572e96f42cf6402cbe0ac8dcb046577/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-28T23:33:16.785 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/reef-release/0484097b0572e96f42cf6402cbe0ac8dcb046577/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/reef-release/0484097b0572e96f42cf6402cbe0ac8dcb046577/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/reef-release/0484097b0572e96f42cf6402cbe0ac8dcb046577/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-28T23:33:16.786 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-07-28T23:33:16.786 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-28T23:33:16.818 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-07-28T23:33:16.819 DEBUG:teuthology.orchestra.run.smithi101:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/0484097b0572e96f42cf6402cbe0ac8dcb046577/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-28T23:33:16.853 DEBUG:teuthology.orchestra.run.smithi139:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-28T23:33:16.888 DEBUG:teuthology.orchestra.run.smithi101:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-28T23:33:16.936 DEBUG:teuthology.orchestra.run.smithi139:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-28T23:33:16.970 DEBUG:teuthology.orchestra.run.smithi101:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-28T23:33:17.003 INFO:teuthology.orchestra.run.smithi139.stdout:check_obsoletes = 1 2023-07-28T23:33:17.005 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum clean all 2023-07-28T23:33:17.038 INFO:teuthology.orchestra.run.smithi101.stdout:check_obsoletes = 1 2023-07-28T23:33:17.039 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum clean all 2023-07-28T23:33:17.239 INFO:teuthology.orchestra.run.smithi139.stdout:71 files removed 2023-07-28T23:33:17.266 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-radosgw 2023-07-28T23:33:17.272 INFO:teuthology.orchestra.run.smithi101.stdout:71 files removed 2023-07-28T23:33:17.299 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install ceph-radosgw 2023-07-28T23:33:18.428 INFO:teuthology.orchestra.run.smithi101.stdout:ceph packages for x86_64 91 kB/s | 77 kB 00:00 2023-07-28T23:33:18.486 INFO:teuthology.orchestra.run.smithi139.stdout:ceph packages for x86_64 82 kB/s | 77 kB 00:00 2023-07-28T23:33:19.203 INFO:teuthology.orchestra.run.smithi101.stdout:ceph noarch packages 13 kB/s | 9.4 kB 00:00 2023-07-28T23:33:19.236 INFO:teuthology.orchestra.run.smithi139.stdout:ceph noarch packages 13 kB/s | 9.4 kB 00:00 2023-07-28T23:33:19.912 INFO:teuthology.orchestra.run.smithi139.stdout:ceph source packages 2.9 kB/s | 1.9 kB 00:00 2023-07-28T23:33:19.921 INFO:teuthology.orchestra.run.smithi101.stdout:ceph source packages 2.7 kB/s | 1.9 kB 00:00 2023-07-28T23:33:20.858 INFO:teuthology.orchestra.run.smithi101.stdout:CentOS-9 - AppStream 6.9 MB/s | 6.3 MB 00:00 2023-07-28T23:33:20.889 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS-9 - AppStream 6.6 MB/s | 6.3 MB 00:00 2023-07-28T23:33:22.450 INFO:teuthology.orchestra.run.smithi101.stdout:CentOS-9 - BaseOS 3.4 MB/s | 2.0 MB 00:00 2023-07-28T23:33:22.473 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS-9 - BaseOS 3.6 MB/s | 2.0 MB 00:00 2023-07-28T23:33:23.782 INFO:teuthology.orchestra.run.smithi101.stdout:CentOS Stream 9 - BaseOS 6.7 MB/s | 6.9 MB 00:01 2023-07-28T23:33:24.958 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 9 - BaseOS 3.1 MB/s | 6.9 MB 00:02 2023-07-28T23:33:26.995 INFO:teuthology.orchestra.run.smithi101.stdout:CentOS Stream 9 - AppStream 7.8 MB/s | 17 MB 00:02 2023-07-28T23:33:28.442 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 9 - AppStream 7.1 MB/s | 17 MB 00:02 2023-07-28T23:33:31.810 INFO:teuthology.orchestra.run.smithi101.stdout:CentOS Stream 9 - CRB 3.6 MB/s | 5.6 MB 00:01 2023-07-28T23:33:33.260 INFO:teuthology.orchestra.run.smithi101.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 13 kB 00:00 2023-07-28T23:33:33.423 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 5.6 MB 00:01 2023-07-28T23:33:35.024 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 9 - Extras packages 25 kB/s | 13 kB 00:00 2023-07-28T23:33:36.342 INFO:teuthology.orchestra.run.smithi139.stdout:Extra Packages for Enterprise Linux 15 MB/s | 18 MB 00:01 2023-07-28T23:33:37.737 INFO:teuthology.orchestra.run.smithi101.stdout:Extra Packages for Enterprise Linux 4.2 MB/s | 18 MB 00:04 2023-07-28T23:33:43.024 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2023-07-28T23:33:43.038 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-07-28T23:33:43.038 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2023-07-28T23:33:43.038 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-07-28T23:33:43.039 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2023-07-28T23:33:43.039 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-radosgw x86_64 2:18.1.3-11.g0484097b.el9 ceph 7.6 M 2023-07-28T23:33:43.039 INFO:teuthology.orchestra.run.smithi139.stdout:Upgrading: 2023-07-28T23:33:43.039 INFO:teuthology.orchestra.run.smithi139.stdout: librados2 x86_64 2:18.1.3-11.g0484097b.el9 ceph 3.3 M 2023-07-28T23:33:43.039 INFO:teuthology.orchestra.run.smithi139.stdout: librbd1 x86_64 2:18.1.3-11.g0484097b.el9 ceph 3.0 M 2023-07-28T23:33:43.039 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2023-07-28T23:33:43.039 INFO:teuthology.orchestra.run.smithi139.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-07-28T23:33:43.040 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-base x86_64 2:18.1.3-11.g0484097b.el9 ceph 5.2 M 2023-07-28T23:33:43.040 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-common x86_64 2:18.1.3-11.g0484097b.el9 ceph 18 M 2023-07-28T23:33:43.040 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-selinux x86_64 2:18.1.3-11.g0484097b.el9 ceph 24 k 2023-07-28T23:33:43.040 INFO:teuthology.orchestra.run.smithi139.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2023-07-28T23:33:43.040 INFO:teuthology.orchestra.run.smithi139.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-07-28T23:33:43.040 INFO:teuthology.orchestra.run.smithi139.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-07-28T23:33:43.040 INFO:teuthology.orchestra.run.smithi139.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-07-28T23:33:43.041 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs2 x86_64 2:18.1.3-11.g0484097b.el9 ceph 653 k 2023-07-28T23:33:43.041 INFO:teuthology.orchestra.run.smithi139.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-07-28T23:33:43.041 INFO:teuthology.orchestra.run.smithi139.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-07-28T23:33:43.041 INFO:teuthology.orchestra.run.smithi139.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-07-28T23:33:43.041 INFO:teuthology.orchestra.run.smithi139.stdout: libradosstriper1 x86_64 2:18.1.3-11.g0484097b.el9 ceph 474 k 2023-07-28T23:33:43.041 INFO:teuthology.orchestra.run.smithi139.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-07-28T23:33:43.041 INFO:teuthology.orchestra.run.smithi139.stdout: librgw2 x86_64 2:18.1.3-11.g0484097b.el9 ceph 4.4 M 2023-07-28T23:33:43.041 INFO:teuthology.orchestra.run.smithi139.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-07-28T23:33:43.042 INFO:teuthology.orchestra.run.smithi139.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-07-28T23:33:43.042 INFO:teuthology.orchestra.run.smithi139.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-07-28T23:33:43.042 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-argparse x86_64 2:18.1.3-11.g0484097b.el9 ceph 45 k 2023-07-28T23:33:43.042 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-common x86_64 2:18.1.3-11.g0484097b.el9 ceph 120 k 2023-07-28T23:33:43.042 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cephfs x86_64 2:18.1.3-11.g0484097b.el9 ceph 156 k 2023-07-28T23:33:43.042 INFO:teuthology.orchestra.run.smithi139.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-07-28T23:33:43.042 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rados x86_64 2:18.1.3-11.g0484097b.el9 ceph 320 k 2023-07-28T23:33:43.043 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rbd x86_64 2:18.1.3-11.g0484097b.el9 ceph 297 k 2023-07-28T23:33:43.043 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rgw x86_64 2:18.1.3-11.g0484097b.el9 ceph 99 k 2023-07-28T23:33:43.043 INFO:teuthology.orchestra.run.smithi139.stdout: re2 x86_64 1:20211101-2.el9 epel 192 k 2023-07-28T23:33:43.043 INFO:teuthology.orchestra.run.smithi139.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-07-28T23:33:43.043 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:33:43.043 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2023-07-28T23:33:43.043 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-07-28T23:33:43.043 INFO:teuthology.orchestra.run.smithi139.stdout:Install 28 Packages 2023-07-28T23:33:43.044 INFO:teuthology.orchestra.run.smithi139.stdout:Upgrade 2 Packages 2023-07-28T23:33:43.044 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:33:43.044 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 53 M 2023-07-28T23:33:43.044 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2023-07-28T23:33:44.410 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-07-28T23:33:44.424 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-07-28T23:33:44.424 INFO:teuthology.orchestra.run.smithi101.stdout: Package Arch Version Repository Size 2023-07-28T23:33:44.425 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-07-28T23:33:44.425 INFO:teuthology.orchestra.run.smithi101.stdout:Installing: 2023-07-28T23:33:44.425 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-radosgw x86_64 2:18.1.3-11.g0484097b.el9 ceph 7.6 M 2023-07-28T23:33:44.425 INFO:teuthology.orchestra.run.smithi101.stdout:Upgrading: 2023-07-28T23:33:44.425 INFO:teuthology.orchestra.run.smithi101.stdout: librados2 x86_64 2:18.1.3-11.g0484097b.el9 ceph 3.3 M 2023-07-28T23:33:44.426 INFO:teuthology.orchestra.run.smithi101.stdout: librbd1 x86_64 2:18.1.3-11.g0484097b.el9 ceph 3.0 M 2023-07-28T23:33:44.426 INFO:teuthology.orchestra.run.smithi101.stdout:Installing dependencies: 2023-07-28T23:33:44.426 INFO:teuthology.orchestra.run.smithi101.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-07-28T23:33:44.426 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-base x86_64 2:18.1.3-11.g0484097b.el9 ceph 5.2 M 2023-07-28T23:33:44.427 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-common x86_64 2:18.1.3-11.g0484097b.el9 ceph 18 M 2023-07-28T23:33:44.427 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-selinux x86_64 2:18.1.3-11.g0484097b.el9 ceph 24 k 2023-07-28T23:33:44.427 INFO:teuthology.orchestra.run.smithi101.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2023-07-28T23:33:44.427 INFO:teuthology.orchestra.run.smithi101.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-07-28T23:33:44.428 INFO:teuthology.orchestra.run.smithi101.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-07-28T23:33:44.428 INFO:teuthology.orchestra.run.smithi101.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-07-28T23:33:44.428 INFO:teuthology.orchestra.run.smithi101.stdout: libcephfs2 x86_64 2:18.1.3-11.g0484097b.el9 ceph 653 k 2023-07-28T23:33:44.428 INFO:teuthology.orchestra.run.smithi101.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-07-28T23:33:44.429 INFO:teuthology.orchestra.run.smithi101.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-07-28T23:33:44.429 INFO:teuthology.orchestra.run.smithi101.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-07-28T23:33:44.429 INFO:teuthology.orchestra.run.smithi101.stdout: libradosstriper1 x86_64 2:18.1.3-11.g0484097b.el9 ceph 474 k 2023-07-28T23:33:44.430 INFO:teuthology.orchestra.run.smithi101.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-07-28T23:33:44.430 INFO:teuthology.orchestra.run.smithi101.stdout: librgw2 x86_64 2:18.1.3-11.g0484097b.el9 ceph 4.4 M 2023-07-28T23:33:44.430 INFO:teuthology.orchestra.run.smithi101.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-07-28T23:33:44.430 INFO:teuthology.orchestra.run.smithi101.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-07-28T23:33:44.431 INFO:teuthology.orchestra.run.smithi101.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-07-28T23:33:44.431 INFO:teuthology.orchestra.run.smithi101.stdout: python3-ceph-argparse x86_64 2:18.1.3-11.g0484097b.el9 ceph 45 k 2023-07-28T23:33:44.431 INFO:teuthology.orchestra.run.smithi101.stdout: python3-ceph-common x86_64 2:18.1.3-11.g0484097b.el9 ceph 120 k 2023-07-28T23:33:44.431 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cephfs x86_64 2:18.1.3-11.g0484097b.el9 ceph 156 k 2023-07-28T23:33:44.432 INFO:teuthology.orchestra.run.smithi101.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-07-28T23:33:44.432 INFO:teuthology.orchestra.run.smithi101.stdout: python3-rados x86_64 2:18.1.3-11.g0484097b.el9 ceph 320 k 2023-07-28T23:33:44.432 INFO:teuthology.orchestra.run.smithi101.stdout: python3-rbd x86_64 2:18.1.3-11.g0484097b.el9 ceph 297 k 2023-07-28T23:33:44.433 INFO:teuthology.orchestra.run.smithi101.stdout: python3-rgw x86_64 2:18.1.3-11.g0484097b.el9 ceph 99 k 2023-07-28T23:33:44.433 INFO:teuthology.orchestra.run.smithi101.stdout: re2 x86_64 1:20211101-2.el9 epel 192 k 2023-07-28T23:33:44.433 INFO:teuthology.orchestra.run.smithi101.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-07-28T23:33:44.433 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:33:44.434 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2023-07-28T23:33:44.434 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-07-28T23:33:44.434 INFO:teuthology.orchestra.run.smithi101.stdout:Install 28 Packages 2023-07-28T23:33:44.434 INFO:teuthology.orchestra.run.smithi101.stdout:Upgrade 2 Packages 2023-07-28T23:33:44.435 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:33:44.436 INFO:teuthology.orchestra.run.smithi101.stdout:Total download size: 53 M 2023-07-28T23:33:44.436 INFO:teuthology.orchestra.run.smithi101.stdout:Downloading Packages: 2023-07-28T23:33:48.692 INFO:teuthology.orchestra.run.smithi139.stdout:(1/30): ceph-base-18.1.3-11.g0484097b.el9.x86_6 977 kB/s | 5.2 MB 00:05 2023-07-28T23:33:48.818 INFO:teuthology.orchestra.run.smithi139.stdout:(2/30): ceph-selinux-18.1.3-11.g0484097b.el9.x8 193 kB/s | 24 kB 00:00 2023-07-28T23:33:49.184 INFO:teuthology.orchestra.run.smithi139.stdout:(3/30): libcephfs2-18.1.3-11.g0484097b.el9.x86_ 1.7 MB/s | 653 kB 00:00 2023-07-28T23:33:49.385 INFO:teuthology.orchestra.run.smithi139.stdout:(4/30): ceph-radosgw-18.1.3-11.g0484097b.el9.x8 1.2 MB/s | 7.6 MB 00:06 2023-07-28T23:33:49.510 INFO:teuthology.orchestra.run.smithi139.stdout:(5/30): libradosstriper1-18.1.3-11.g0484097b.el 1.4 MB/s | 474 kB 00:00 2023-07-28T23:33:49.677 INFO:teuthology.orchestra.run.smithi139.stdout:(6/30): python3-ceph-argparse-18.1.3-11.g048409 269 kB/s | 45 kB 00:00 2023-07-28T23:33:49.868 INFO:teuthology.orchestra.run.smithi139.stdout:(7/30): python3-ceph-common-18.1.3-11.g0484097b 625 kB/s | 120 kB 00:00 2023-07-28T23:33:50.044 INFO:teuthology.orchestra.run.smithi139.stdout:(8/30): python3-cephfs-18.1.3-11.g0484097b.el9. 891 kB/s | 156 kB 00:00 2023-07-28T23:33:50.236 INFO:teuthology.orchestra.run.smithi139.stdout:(9/30): python3-rados-18.1.3-11.g0484097b.el9.x 1.6 MB/s | 320 kB 00:00 2023-07-28T23:33:50.769 INFO:teuthology.orchestra.run.smithi139.stdout:(10/30): python3-rbd-18.1.3-11.g0484097b.el9.x8 558 kB/s | 297 kB 00:00 2023-07-28T23:33:51.545 INFO:teuthology.orchestra.run.smithi139.stdout:(11/30): python3-rgw-18.1.3-11.g0484097b.el9.x8 128 kB/s | 99 kB 00:00 2023-07-28T23:33:51.962 INFO:teuthology.orchestra.run.smithi139.stdout:(12/30): librgw2-18.1.3-11.g0484097b.el9.x86_64 1.7 MB/s | 4.4 MB 00:02 2023-07-28T23:33:52.020 INFO:teuthology.orchestra.run.smithi139.stdout:(13/30): boost-program-options-1.75.0-8.el9.x86 222 kB/s | 106 kB 00:00 2023-07-28T23:33:52.254 INFO:teuthology.orchestra.run.smithi139.stdout:(14/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 545 kB/s | 159 kB 00:00 2023-07-28T23:33:52.271 INFO:teuthology.orchestra.run.smithi139.stdout:(15/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 177 kB/s | 44 kB 00:00 2023-07-28T23:33:52.446 INFO:teuthology.orchestra.run.smithi101.stdout:(1/30): ceph-base-18.1.3-11.g0484097b.el9.x86_6 667 kB/s | 5.2 MB 00:07 2023-07-28T23:33:52.529 INFO:teuthology.orchestra.run.smithi139.stdout:(16/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.1 MB/s | 282 kB 00:00 2023-07-28T23:33:52.571 INFO:teuthology.orchestra.run.smithi139.stdout:(17/30): librdkafka-1.6.1-102.el9.x86_64.rpm 2.2 MB/s | 661 kB 00:00 2023-07-28T23:33:52.813 INFO:teuthology.orchestra.run.smithi139.stdout:(18/30): python3-prettytable-0.7.2-27.el9.noarc 145 kB/s | 41 kB 00:00 2023-07-28T23:33:53.030 INFO:teuthology.orchestra.run.smithi139.stdout:(19/30): fmt-8.1.1-5.el9.x86_64.rpm 241 kB/s | 111 kB 00:00 2023-07-28T23:33:53.079 INFO:teuthology.orchestra.run.smithi101.stdout:(2/30): ceph-selinux-18.1.3-11.g0484097b.el9.x8 38 kB/s | 24 kB 00:00 2023-07-28T23:33:53.447 INFO:teuthology.orchestra.run.smithi139.stdout:(20/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 487 kB/s | 309 kB 00:00 2023-07-28T23:33:53.589 INFO:teuthology.orchestra.run.smithi139.stdout:(21/30): libarrow-9.0.0-9.el9.x86_64.rpm 7.9 MB/s | 4.4 MB 00:00 2023-07-28T23:33:53.615 INFO:teuthology.orchestra.run.smithi139.stdout:(22/30): libarrow-doc-9.0.0-9.el9.noarch.rpm 146 kB/s | 24 kB 00:00 2023-07-28T23:33:53.665 INFO:teuthology.orchestra.run.smithi139.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 647 kB/s | 49 kB 00:00 2023-07-28T23:33:53.698 INFO:teuthology.orchestra.run.smithi139.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 808 kB/s | 67 kB 00:00 2023-07-28T23:33:53.766 INFO:teuthology.orchestra.run.smithi139.stdout:(25/30): parquet-libs-9.0.0-9.el9.x86_64.rpm 8.1 MB/s | 837 kB 00:00 2023-07-28T23:33:53.857 INFO:teuthology.orchestra.run.smithi139.stdout:(26/30): re2-20211101-2.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2023-07-28T23:33:53.908 INFO:teuthology.orchestra.run.smithi139.stdout:(27/30): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2023-07-28T23:33:54.421 INFO:teuthology.orchestra.run.smithi101.stdout:(3/30): libcephfs2-18.1.3-11.g0484097b.el9.x86_ 487 kB/s | 653 kB 00:01 2023-07-28T23:33:55.054 INFO:teuthology.orchestra.run.smithi101.stdout:(4/30): libradosstriper1-18.1.3-11.g0484097b.el 749 kB/s | 474 kB 00:00 2023-07-28T23:33:55.841 INFO:teuthology.orchestra.run.smithi139.stdout:(28/30): librbd1-18.1.3-11.g0484097b.el9.x86_64 1.6 MB/s | 3.0 MB 00:01 2023-07-28T23:33:56.075 INFO:teuthology.orchestra.run.smithi139.stdout:(29/30): librados2-18.1.3-11.g0484097b.el9.x86_ 1.5 MB/s | 3.3 MB 00:02 2023-07-28T23:33:56.708 INFO:teuthology.orchestra.run.smithi139.stdout:(30/30): ceph-common-18.1.3-11.g0484097b.el9.x8 1.4 MB/s | 18 MB 00:13 2023-07-28T23:33:56.712 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2023-07-28T23:33:56.712 INFO:teuthology.orchestra.run.smithi139.stdout:Total 3.9 MB/s | 53 MB 00:13 2023-07-28T23:33:56.713 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2023-07-28T23:33:56.750 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2023-07-28T23:33:56.750 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2023-07-28T23:33:56.763 INFO:teuthology.orchestra.run.smithi101.stdout:(5/30): ceph-radosgw-18.1.3-11.g0484097b.el9.x8 637 kB/s | 7.6 MB 00:12 2023-07-28T23:33:56.863 INFO:teuthology.orchestra.run.smithi101.stdout:(6/30): python3-ceph-argparse-18.1.3-11.g048409 450 kB/s | 45 kB 00:00 2023-07-28T23:33:57.014 INFO:teuthology.orchestra.run.smithi101.stdout:(7/30): python3-ceph-common-18.1.3-11.g0484097b 796 kB/s | 120 kB 00:00 2023-07-28T23:33:57.181 INFO:teuthology.orchestra.run.smithi101.stdout:(8/30): python3-cephfs-18.1.3-11.g0484097b.el9. 937 kB/s | 156 kB 00:00 2023-07-28T23:33:57.193 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2023-07-28T23:33:57.193 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2023-07-28T23:33:57.472 INFO:teuthology.orchestra.run.smithi101.stdout:(9/30): python3-rados-18.1.3-11.g0484097b.el9.x 1.1 MB/s | 320 kB 00:00 2023-07-28T23:33:57.706 INFO:teuthology.orchestra.run.smithi101.stdout:(10/30): python3-rbd-18.1.3-11.g0484097b.el9.x8 1.2 MB/s | 297 kB 00:00 2023-07-28T23:33:57.923 INFO:teuthology.orchestra.run.smithi101.stdout:(11/30): python3-rgw-18.1.3-11.g0484097b.el9.x8 460 kB/s | 99 kB 00:00 2023-07-28T23:33:58.065 INFO:teuthology.orchestra.run.smithi101.stdout:(12/30): boost-program-options-1.75.0-8.el9.x86 745 kB/s | 106 kB 00:00 2023-07-28T23:33:58.124 INFO:teuthology.orchestra.run.smithi101.stdout:(13/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 159 kB 00:00 2023-07-28T23:33:58.166 INFO:teuthology.orchestra.run.smithi101.stdout:(14/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2023-07-28T23:33:58.308 INFO:teuthology.orchestra.run.smithi101.stdout:(15/30): librdkafka-1.6.1-102.el9.x86_64.rpm 4.6 MB/s | 661 kB 00:00 2023-07-28T23:33:58.391 INFO:teuthology.orchestra.run.smithi101.stdout:(16/30): librgw2-18.1.3-11.g0484097b.el9.x86_64 1.3 MB/s | 4.4 MB 00:03 2023-07-28T23:33:58.433 INFO:teuthology.orchestra.run.smithi101.stdout:(17/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.2 MB/s | 282 kB 00:00 2023-07-28T23:33:58.450 INFO:teuthology.orchestra.run.smithi101.stdout:(18/30): python3-prettytable-0.7.2-27.el9.noarc 711 kB/s | 41 kB 00:00 2023-07-28T23:33:58.571 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2023-07-28T23:33:58.671 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2023-07-28T23:33:58.713 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2023-07-28T23:33:58.772 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2023-07-28T23:33:59.098 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2023-07-28T23:33:59.170 INFO:teuthology.orchestra.run.smithi139.stdout: Upgrading : librados2-2:18.1.3-11.g0484097b.el9.x86_64 5/32 2023-07-28T23:33:59.262 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librados2-2:18.1.3-11.g0484097b.el9.x86_64 5/32 2023-07-28T23:33:59.349 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rados-2:18.1.3-11.g0484097b.el9.x86_64 6/32 2023-07-28T23:33:59.367 INFO:teuthology.orchestra.run.smithi101.stdout:(19/30): fmt-8.1.1-5.el9.x86_64.rpm 119 kB/s | 111 kB 00:00 2023-07-28T23:33:59.380 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libcephfs2-2:18.1.3-11.g0484097b.el9.x86_64 7/32 2023-07-28T23:33:59.442 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libcephfs2-2:18.1.3-11.g0484097b.el9.x86_64 7/32 2023-07-28T23:33:59.480 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2023-07-28T23:33:59.531 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-ceph-argparse-2:18.1.3-11.g0484097b.el9.x8 9/32 2023-07-28T23:33:59.593 INFO:teuthology.orchestra.run.smithi101.stdout:(20/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 270 kB/s | 309 kB 00:01 2023-07-28T23:33:59.605 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cephfs-2:18.1.3-11.g0484097b.el9.x86_64 10/32 2023-07-28T23:33:59.632 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libradosstriper1-2:18.1.3-11.g0484097b.el9.x86_64 11/32 2023-07-28T23:33:59.698 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libradosstriper1-2:18.1.3-11.g0484097b.el9.x86_64 11/32 2023-07-28T23:33:59.734 INFO:teuthology.orchestra.run.smithi101.stdout:(21/30): libarrow-doc-9.0.0-9.el9.noarch.rpm 172 kB/s | 24 kB 00:00 2023-07-28T23:33:59.813 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : re2-1:20211101-2.el9.x86_64 12/32 2023-07-28T23:33:59.856 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/32 2023-07-28T23:33:59.885 INFO:teuthology.orchestra.run.smithi101.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 324 kB/s | 49 kB 00:00 2023-07-28T23:33:59.905 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2023-07-28T23:33:59.954 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2023-07-28T23:33:59.989 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/32 2023-07-28T23:34:00.025 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2023-07-28T23:34:00.052 INFO:teuthology.orchestra.run.smithi101.stdout:(23/30): libunwind-1.6.2-1.el9.x86_64.rpm 404 kB/s | 67 kB 00:00 2023-07-28T23:34:00.066 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-07-28T23:34:00.373 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2023-07-28T23:34:00.413 INFO:teuthology.orchestra.run.smithi139.stdout: Upgrading : librbd1-2:18.1.3-11.g0484097b.el9.x86_64 20/32 2023-07-28T23:34:00.528 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librbd1-2:18.1.3-11.g0484097b.el9.x86_64 20/32 2023-07-28T23:34:00.535 INFO:teuthology.orchestra.run.smithi101.stdout:(24/30): parquet-libs-9.0.0-9.el9.x86_64.rpm 1.7 MB/s | 837 kB 00:00 2023-07-28T23:34:00.567 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rbd-2:18.1.3-11.g0484097b.el9.x86_64 21/32 2023-07-28T23:34:00.612 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2023-07-28T23:34:00.710 INFO:teuthology.orchestra.run.smithi101.stdout:(25/30): re2-20211101-2.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2023-07-28T23:34:01.014 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 23/32 2023-07-28T23:34:01.049 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librgw2-2:18.1.3-11.g0484097b.el9.x86_64 24/32 2023-07-28T23:34:01.060 INFO:teuthology.orchestra.run.smithi101.stdout:(26/30): thrift-0.15.0-2.el9.x86_64.rpm 4.5 MB/s | 1.6 MB 00:00 2023-07-28T23:34:01.120 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librgw2-2:18.1.3-11.g0484097b.el9.x86_64 24/32 2023-07-28T23:34:01.195 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rgw-2:18.1.3-11.g0484097b.el9.x86_64 25/32 2023-07-28T23:34:01.234 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-ceph-common-2:18.1.3-11.g0484097b.el9.x86_ 26/32 2023-07-28T23:34:01.278 INFO:teuthology.orchestra.run.smithi101.stdout:(27/30): libarrow-9.0.0-9.el9.x86_64.rpm 2.3 MB/s | 4.4 MB 00:01 2023-07-28T23:34:03.311 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-common-2:18.1.3-11.g0484097b.el9.x86_64 27/32 2023-07-28T23:34:03.356 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-common-2:18.1.3-11.g0484097b.el9.x86_64 27/32 2023-07-28T23:34:03.908 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-common-2:18.1.3-11.g0484097b.el9.x86_64 27/32 2023-07-28T23:34:03.959 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-base-2:18.1.3-11.g0484097b.el9.x86_64 28/32 2023-07-28T23:34:04.258 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-base-2:18.1.3-11.g0484097b.el9.x86_64 28/32 2023-07-28T23:34:04.258 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-07-28T23:34:04.258 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-07-28T23:34:04.258 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:34:04.302 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-selinux-2:18.1.3-11.g0484097b.el9.x86_64 29/32 2023-07-28T23:34:04.486 INFO:teuthology.orchestra.run.smithi101.stdout:(28/30): librados2-18.1.3-11.g0484097b.el9.x86_ 978 kB/s | 3.3 MB 00:03 2023-07-28T23:34:04.928 INFO:teuthology.orchestra.run.smithi101.stdout:(29/30): librbd1-18.1.3-11.g0484097b.el9.x86_64 842 kB/s | 3.0 MB 00:03 2023-07-28T23:34:06.837 INFO:teuthology.orchestra.run.smithi101.stdout:(30/30): ceph-common-18.1.3-11.g0484097b.el9.x8 836 kB/s | 18 MB 00:22 2023-07-28T23:34:06.842 INFO:teuthology.orchestra.run.smithi101.stdout:-------------------------------------------------------------------------------- 2023-07-28T23:34:06.843 INFO:teuthology.orchestra.run.smithi101.stdout:Total 2.4 MB/s | 53 MB 00:22 2023-07-28T23:34:06.843 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction check 2023-07-28T23:34:06.880 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction check succeeded. 2023-07-28T23:34:06.880 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction test 2023-07-28T23:34:07.309 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction test succeeded. 2023-07-28T23:34:07.310 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction 2023-07-28T23:34:08.641 INFO:teuthology.orchestra.run.smithi101.stdout: Preparing : 1/1 2023-07-28T23:34:08.738 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2023-07-28T23:34:08.783 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2023-07-28T23:34:08.842 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2023-07-28T23:34:09.177 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2023-07-28T23:34:09.266 INFO:teuthology.orchestra.run.smithi101.stdout: Upgrading : librados2-2:18.1.3-11.g0484097b.el9.x86_64 5/32 2023-07-28T23:34:09.356 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: librados2-2:18.1.3-11.g0484097b.el9.x86_64 5/32 2023-07-28T23:34:09.445 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-rados-2:18.1.3-11.g0484097b.el9.x86_64 6/32 2023-07-28T23:34:09.492 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libcephfs2-2:18.1.3-11.g0484097b.el9.x86_64 7/32 2023-07-28T23:34:09.554 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: libcephfs2-2:18.1.3-11.g0484097b.el9.x86_64 7/32 2023-07-28T23:34:09.591 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2023-07-28T23:34:09.642 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-ceph-argparse-2:18.1.3-11.g0484097b.el9.x8 9/32 2023-07-28T23:34:09.717 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-cephfs-2:18.1.3-11.g0484097b.el9.x86_64 10/32 2023-07-28T23:34:09.752 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libradosstriper1-2:18.1.3-11.g0484097b.el9.x86_64 11/32 2023-07-28T23:34:09.801 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: libradosstriper1-2:18.1.3-11.g0484097b.el9.x86_64 11/32 2023-07-28T23:34:09.924 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : re2-1:20211101-2.el9.x86_64 12/32 2023-07-28T23:34:09.968 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/32 2023-07-28T23:34:10.073 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2023-07-28T23:34:10.125 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2023-07-28T23:34:10.168 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/32 2023-07-28T23:34:10.212 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2023-07-28T23:34:10.262 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-07-28T23:34:10.570 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2023-07-28T23:34:10.600 INFO:teuthology.orchestra.run.smithi101.stdout: Upgrading : librbd1-2:18.1.3-11.g0484097b.el9.x86_64 20/32 2023-07-28T23:34:10.690 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: librbd1-2:18.1.3-11.g0484097b.el9.x86_64 20/32 2023-07-28T23:34:10.730 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-rbd-2:18.1.3-11.g0484097b.el9.x86_64 21/32 2023-07-28T23:34:10.774 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2023-07-28T23:34:11.176 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 23/32 2023-07-28T23:34:11.219 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : librgw2-2:18.1.3-11.g0484097b.el9.x86_64 24/32 2023-07-28T23:34:11.273 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: librgw2-2:18.1.3-11.g0484097b.el9.x86_64 24/32 2023-07-28T23:34:11.357 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-rgw-2:18.1.3-11.g0484097b.el9.x86_64 25/32 2023-07-28T23:34:11.407 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-ceph-common-2:18.1.3-11.g0484097b.el9.x86_ 26/32 2023-07-28T23:34:13.451 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-common-2:18.1.3-11.g0484097b.el9.x86_64 27/32 2023-07-28T23:34:13.495 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-common-2:18.1.3-11.g0484097b.el9.x86_64 27/32 2023-07-28T23:34:13.694 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-selinux-2:18.1.3-11.g0484097b.el9.x86_64 29/32 2023-07-28T23:34:13.694 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /sys 2023-07-28T23:34:13.694 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /proc 2023-07-28T23:34:13.694 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /mnt 2023-07-28T23:34:13.695 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /var/tmp 2023-07-28T23:34:13.695 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /home 2023-07-28T23:34:13.695 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /root 2023-07-28T23:34:13.695 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /tmp 2023-07-28T23:34:13.695 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:34:14.053 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-common-2:18.1.3-11.g0484097b.el9.x86_64 27/32 2023-07-28T23:34:14.114 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-base-2:18.1.3-11.g0484097b.el9.x86_64 28/32 2023-07-28T23:34:14.194 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-base-2:18.1.3-11.g0484097b.el9.x86_64 28/32 2023-07-28T23:34:14.194 INFO:teuthology.orchestra.run.smithi101.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-07-28T23:34:14.194 INFO:teuthology.orchestra.run.smithi101.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-07-28T23:34:14.195 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:34:14.432 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-radosgw-2:18.1.3-11.g0484097b.el9.x86_64 30/32 2023-07-28T23:34:14.432 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-selinux-2:18.1.3-11.g0484097b.el9.x86_64 29/32 2023-07-28T23:34:14.456 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-radosgw-2:18.1.3-11.g0484097b.el9.x86_64 30/32 2023-07-28T23:34:14.457 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-07-28T23:34:14.457 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-07-28T23:34:14.457 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-07-28T23:34:14.457 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-07-28T23:34:14.457 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:34:14.459 INFO:teuthology.orchestra.run.smithi139.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-07-28T23:34:14.542 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-07-28T23:34:14.543 INFO:teuthology.orchestra.run.smithi139.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2023-07-28T23:34:15.752 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2023-07-28T23:34:15.752 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-base-2:18.1.3-11.g0484097b.el9.x86_64 1/32 2023-07-28T23:34:15.752 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-common-2:18.1.3-11.g0484097b.el9.x86_64 2/32 2023-07-28T23:34:15.753 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-radosgw-2:18.1.3-11.g0484097b.el9.x86_64 3/32 2023-07-28T23:34:15.753 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-selinux-2:18.1.3-11.g0484097b.el9.x86_64 4/32 2023-07-28T23:34:15.753 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libcephfs2-2:18.1.3-11.g0484097b.el9.x86_64 5/32 2023-07-28T23:34:15.753 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libradosstriper1-2:18.1.3-11.g0484097b.el9.x86_64 6/32 2023-07-28T23:34:15.753 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librgw2-2:18.1.3-11.g0484097b.el9.x86_64 7/32 2023-07-28T23:34:15.753 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-ceph-argparse-2:18.1.3-11.g0484097b.el9.x8 8/32 2023-07-28T23:34:15.753 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-ceph-common-2:18.1.3-11.g0484097b.el9.x86_ 9/32 2023-07-28T23:34:15.754 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cephfs-2:18.1.3-11.g0484097b.el9.x86_64 10/32 2023-07-28T23:34:15.754 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rados-2:18.1.3-11.g0484097b.el9.x86_64 11/32 2023-07-28T23:34:15.754 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rbd-2:18.1.3-11.g0484097b.el9.x86_64 12/32 2023-07-28T23:34:15.754 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rgw-2:18.1.3-11.g0484097b.el9.x86_64 13/32 2023-07-28T23:34:15.754 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2023-07-28T23:34:15.754 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2023-07-28T23:34:15.754 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2023-07-28T23:34:15.754 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2023-07-28T23:34:15.755 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2023-07-28T23:34:15.755 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2023-07-28T23:34:15.755 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2023-07-28T23:34:15.755 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2023-07-28T23:34:15.755 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 22/32 2023-07-28T23:34:15.755 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 23/32 2023-07-28T23:34:15.755 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2023-07-28T23:34:15.756 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2023-07-28T23:34:15.756 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 26/32 2023-07-28T23:34:15.757 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : re2-1:20211101-2.el9.x86_64 27/32 2023-07-28T23:34:15.757 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2023-07-28T23:34:15.757 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librados2-2:18.1.3-11.g0484097b.el9.x86_64 29/32 2023-07-28T23:34:15.757 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2023-07-28T23:34:15.757 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librbd1-2:18.1.3-11.g0484097b.el9.x86_64 31/32 2023-07-28T23:34:16.574 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2023-07-28T23:34:16.575 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:34:16.575 INFO:teuthology.orchestra.run.smithi139.stdout:Upgraded: 2023-07-28T23:34:16.575 INFO:teuthology.orchestra.run.smithi139.stdout: librados2-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:34:16.575 INFO:teuthology.orchestra.run.smithi139.stdout: librbd1-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:34:16.575 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2023-07-28T23:34:16.575 INFO:teuthology.orchestra.run.smithi139.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-07-28T23:34:16.576 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-base-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:34:16.576 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-common-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:34:16.576 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-radosgw-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:34:16.576 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-selinux-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:34:16.576 INFO:teuthology.orchestra.run.smithi139.stdout: fmt-8.1.1-5.el9.x86_64 2023-07-28T23:34:16.576 INFO:teuthology.orchestra.run.smithi139.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-07-28T23:34:16.576 INFO:teuthology.orchestra.run.smithi139.stdout: libarrow-9.0.0-9.el9.x86_64 2023-07-28T23:34:16.577 INFO:teuthology.orchestra.run.smithi139.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-07-28T23:34:16.577 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs2-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:34:16.577 INFO:teuthology.orchestra.run.smithi139.stdout: liboath-2.6.7-2.el9.x86_64 2023-07-28T23:34:16.577 INFO:teuthology.orchestra.run.smithi139.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-07-28T23:34:16.577 INFO:teuthology.orchestra.run.smithi139.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-07-28T23:34:16.577 INFO:teuthology.orchestra.run.smithi139.stdout: libradosstriper1-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:34:16.577 INFO:teuthology.orchestra.run.smithi139.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-07-28T23:34:16.578 INFO:teuthology.orchestra.run.smithi139.stdout: librgw2-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:34:16.578 INFO:teuthology.orchestra.run.smithi139.stdout: libunwind-1.6.2-1.el9.x86_64 2023-07-28T23:34:16.578 INFO:teuthology.orchestra.run.smithi139.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-07-28T23:34:16.578 INFO:teuthology.orchestra.run.smithi139.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-07-28T23:34:16.578 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-argparse-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:34:16.578 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-common-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:34:16.578 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cephfs-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:34:16.578 INFO:teuthology.orchestra.run.smithi139.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-07-28T23:34:16.579 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rados-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:34:16.579 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rbd-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:34:16.579 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rgw-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:34:16.579 INFO:teuthology.orchestra.run.smithi139.stdout: re2-1:20211101-2.el9.x86_64 2023-07-28T23:34:16.579 INFO:teuthology.orchestra.run.smithi139.stdout: thrift-0.15.0-2.el9.x86_64 2023-07-28T23:34:16.579 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:34:16.579 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2023-07-28T23:34:16.953 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-test 2023-07-28T23:34:17.487 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:00:42 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:34:17.617 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2023-07-28T23:34:17.618 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-07-28T23:34:17.618 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2023-07-28T23:34:17.618 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-07-28T23:34:17.619 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2023-07-28T23:34:17.619 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-test x86_64 2:18.1.3-11.g0484097b.el9 ceph 40 M 2023-07-28T23:34:17.619 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2023-07-28T23:34:17.619 INFO:teuthology.orchestra.run.smithi139.stdout: jq x86_64 1.6-15.el9 CentOS-AppStream 186 k 2023-07-28T23:34:17.619 INFO:teuthology.orchestra.run.smithi139.stdout: libcephsqlite x86_64 2:18.1.3-11.g0484097b.el9 ceph 161 k 2023-07-28T23:34:17.619 INFO:teuthology.orchestra.run.smithi139.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-07-28T23:34:17.619 INFO:teuthology.orchestra.run.smithi139.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-07-28T23:34:17.620 INFO:teuthology.orchestra.run.smithi139.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-07-28T23:34:17.620 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:34:17.620 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2023-07-28T23:34:17.620 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-07-28T23:34:17.620 INFO:teuthology.orchestra.run.smithi139.stdout:Install 6 Packages 2023-07-28T23:34:17.620 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:34:17.621 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 40 M 2023-07-28T23:34:17.621 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 167 M 2023-07-28T23:34:17.621 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2023-07-28T23:34:17.897 INFO:teuthology.orchestra.run.smithi139.stdout:(1/6): jq-1.6-15.el9.x86_64.rpm 672 kB/s | 186 kB 00:00 2023-07-28T23:34:18.014 INFO:teuthology.orchestra.run.smithi139.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.8 MB/s | 217 kB 00:00 2023-07-28T23:34:18.123 INFO:teuthology.orchestra.run.smithi139.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 2.7 MB/s | 300 kB 00:00 2023-07-28T23:34:18.181 INFO:teuthology.orchestra.run.smithi139.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.0 MB/s | 63 kB 00:00 2023-07-28T23:34:18.373 INFO:teuthology.orchestra.run.smithi139.stdout:(5/6): libcephsqlite-18.1.3-11.g0484097b.el9.x8 214 kB/s | 161 kB 00:00 2023-07-28T23:34:23.492 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-selinux-2:18.1.3-11.g0484097b.el9.x86_64 29/32 2023-07-28T23:34:23.492 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /sys 2023-07-28T23:34:23.492 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /proc 2023-07-28T23:34:23.492 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /mnt 2023-07-28T23:34:23.493 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /var/tmp 2023-07-28T23:34:23.493 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /home 2023-07-28T23:34:23.493 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /root 2023-07-28T23:34:23.493 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /tmp 2023-07-28T23:34:23.493 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:34:24.213 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-radosgw-2:18.1.3-11.g0484097b.el9.x86_64 30/32 2023-07-28T23:34:24.238 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-radosgw-2:18.1.3-11.g0484097b.el9.x86_64 30/32 2023-07-28T23:34:24.238 INFO:teuthology.orchestra.run.smithi101.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-07-28T23:34:24.238 INFO:teuthology.orchestra.run.smithi101.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-07-28T23:34:24.238 INFO:teuthology.orchestra.run.smithi101.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-07-28T23:34:24.238 INFO:teuthology.orchestra.run.smithi101.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-07-28T23:34:24.239 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:34:24.240 INFO:teuthology.orchestra.run.smithi101.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-07-28T23:34:24.315 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-07-28T23:34:24.316 INFO:teuthology.orchestra.run.smithi101.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2023-07-28T23:34:25.555 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2023-07-28T23:34:25.556 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-base-2:18.1.3-11.g0484097b.el9.x86_64 1/32 2023-07-28T23:34:25.556 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-common-2:18.1.3-11.g0484097b.el9.x86_64 2/32 2023-07-28T23:34:25.556 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-radosgw-2:18.1.3-11.g0484097b.el9.x86_64 3/32 2023-07-28T23:34:25.556 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-selinux-2:18.1.3-11.g0484097b.el9.x86_64 4/32 2023-07-28T23:34:25.556 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libcephfs2-2:18.1.3-11.g0484097b.el9.x86_64 5/32 2023-07-28T23:34:25.556 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libradosstriper1-2:18.1.3-11.g0484097b.el9.x86_64 6/32 2023-07-28T23:34:25.557 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : librgw2-2:18.1.3-11.g0484097b.el9.x86_64 7/32 2023-07-28T23:34:25.557 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-ceph-argparse-2:18.1.3-11.g0484097b.el9.x8 8/32 2023-07-28T23:34:25.557 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-ceph-common-2:18.1.3-11.g0484097b.el9.x86_ 9/32 2023-07-28T23:34:25.557 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-cephfs-2:18.1.3-11.g0484097b.el9.x86_64 10/32 2023-07-28T23:34:25.557 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-rados-2:18.1.3-11.g0484097b.el9.x86_64 11/32 2023-07-28T23:34:25.557 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-rbd-2:18.1.3-11.g0484097b.el9.x86_64 12/32 2023-07-28T23:34:25.558 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-rgw-2:18.1.3-11.g0484097b.el9.x86_64 13/32 2023-07-28T23:34:25.559 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2023-07-28T23:34:25.559 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2023-07-28T23:34:25.559 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2023-07-28T23:34:25.559 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2023-07-28T23:34:25.559 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2023-07-28T23:34:25.559 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2023-07-28T23:34:25.560 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2023-07-28T23:34:25.560 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2023-07-28T23:34:25.560 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 22/32 2023-07-28T23:34:25.560 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 23/32 2023-07-28T23:34:25.560 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2023-07-28T23:34:25.560 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2023-07-28T23:34:25.560 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 26/32 2023-07-28T23:34:25.561 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : re2-1:20211101-2.el9.x86_64 27/32 2023-07-28T23:34:25.561 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2023-07-28T23:34:25.561 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : librados2-2:18.1.3-11.g0484097b.el9.x86_64 29/32 2023-07-28T23:34:25.562 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2023-07-28T23:34:25.562 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : librbd1-2:18.1.3-11.g0484097b.el9.x86_64 31/32 2023-07-28T23:34:26.281 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2023-07-28T23:34:26.281 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:34:26.281 INFO:teuthology.orchestra.run.smithi101.stdout:Upgraded: 2023-07-28T23:34:26.281 INFO:teuthology.orchestra.run.smithi101.stdout: librados2-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:34:26.282 INFO:teuthology.orchestra.run.smithi101.stdout: librbd1-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:34:26.282 INFO:teuthology.orchestra.run.smithi101.stdout:Installed: 2023-07-28T23:34:26.282 INFO:teuthology.orchestra.run.smithi101.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-07-28T23:34:26.282 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-base-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:34:26.282 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-common-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:34:26.282 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-radosgw-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:34:26.282 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-selinux-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:34:26.283 INFO:teuthology.orchestra.run.smithi101.stdout: fmt-8.1.1-5.el9.x86_64 2023-07-28T23:34:26.283 INFO:teuthology.orchestra.run.smithi101.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-07-28T23:34:26.283 INFO:teuthology.orchestra.run.smithi101.stdout: libarrow-9.0.0-9.el9.x86_64 2023-07-28T23:34:26.283 INFO:teuthology.orchestra.run.smithi101.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-07-28T23:34:26.283 INFO:teuthology.orchestra.run.smithi101.stdout: libcephfs2-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:34:26.283 INFO:teuthology.orchestra.run.smithi101.stdout: liboath-2.6.7-2.el9.x86_64 2023-07-28T23:34:26.283 INFO:teuthology.orchestra.run.smithi101.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-07-28T23:34:26.283 INFO:teuthology.orchestra.run.smithi101.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-07-28T23:34:26.284 INFO:teuthology.orchestra.run.smithi101.stdout: libradosstriper1-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:34:26.284 INFO:teuthology.orchestra.run.smithi101.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-07-28T23:34:26.284 INFO:teuthology.orchestra.run.smithi101.stdout: librgw2-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:34:26.284 INFO:teuthology.orchestra.run.smithi101.stdout: libunwind-1.6.2-1.el9.x86_64 2023-07-28T23:34:26.284 INFO:teuthology.orchestra.run.smithi101.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-07-28T23:34:26.284 INFO:teuthology.orchestra.run.smithi101.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-07-28T23:34:26.284 INFO:teuthology.orchestra.run.smithi101.stdout: python3-ceph-argparse-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:34:26.285 INFO:teuthology.orchestra.run.smithi101.stdout: python3-ceph-common-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:34:26.285 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cephfs-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:34:26.285 INFO:teuthology.orchestra.run.smithi101.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-07-28T23:34:26.285 INFO:teuthology.orchestra.run.smithi101.stdout: python3-rados-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:34:26.285 INFO:teuthology.orchestra.run.smithi101.stdout: python3-rbd-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:34:26.285 INFO:teuthology.orchestra.run.smithi101.stdout: python3-rgw-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:34:26.285 INFO:teuthology.orchestra.run.smithi101.stdout: re2-1:20211101-2.el9.x86_64 2023-07-28T23:34:26.285 INFO:teuthology.orchestra.run.smithi101.stdout: thrift-0.15.0-2.el9.x86_64 2023-07-28T23:34:26.286 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:34:26.286 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-07-28T23:34:26.684 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install ceph-test 2023-07-28T23:34:27.221 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:00:52 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:34:27.349 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-07-28T23:34:27.350 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-07-28T23:34:27.350 INFO:teuthology.orchestra.run.smithi101.stdout: Package Arch Version Repository Size 2023-07-28T23:34:27.350 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-07-28T23:34:27.351 INFO:teuthology.orchestra.run.smithi101.stdout:Installing: 2023-07-28T23:34:27.351 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-test x86_64 2:18.1.3-11.g0484097b.el9 ceph 40 M 2023-07-28T23:34:27.351 INFO:teuthology.orchestra.run.smithi101.stdout:Installing dependencies: 2023-07-28T23:34:27.351 INFO:teuthology.orchestra.run.smithi101.stdout: jq x86_64 1.6-15.el9 CentOS-AppStream 186 k 2023-07-28T23:34:27.351 INFO:teuthology.orchestra.run.smithi101.stdout: libcephsqlite x86_64 2:18.1.3-11.g0484097b.el9 ceph 161 k 2023-07-28T23:34:27.351 INFO:teuthology.orchestra.run.smithi101.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-07-28T23:34:27.351 INFO:teuthology.orchestra.run.smithi101.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-07-28T23:34:27.352 INFO:teuthology.orchestra.run.smithi101.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-07-28T23:34:27.352 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:34:27.352 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2023-07-28T23:34:27.352 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-07-28T23:34:27.352 INFO:teuthology.orchestra.run.smithi101.stdout:Install 6 Packages 2023-07-28T23:34:27.352 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:34:27.353 INFO:teuthology.orchestra.run.smithi101.stdout:Total download size: 40 M 2023-07-28T23:34:27.353 INFO:teuthology.orchestra.run.smithi101.stdout:Installed size: 167 M 2023-07-28T23:34:27.353 INFO:teuthology.orchestra.run.smithi101.stdout:Downloading Packages: 2023-07-28T23:34:27.561 INFO:teuthology.orchestra.run.smithi101.stdout:(1/6): jq-1.6-15.el9.x86_64.rpm 894 kB/s | 186 kB 00:00 2023-07-28T23:34:27.611 INFO:teuthology.orchestra.run.smithi101.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2023-07-28T23:34:27.654 INFO:teuthology.orchestra.run.smithi101.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2023-07-28T23:34:27.687 INFO:teuthology.orchestra.run.smithi101.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2023-07-28T23:34:27.737 INFO:teuthology.orchestra.run.smithi101.stdout:(5/6): libcephsqlite-18.1.3-11.g0484097b.el9.x8 420 kB/s | 161 kB 00:00 2023-07-28T23:34:55.652 INFO:teuthology.orchestra.run.smithi139.stdout:(6/6): ceph-test-18.1.3-11.g0484097b.el9.x86_64 1.0 MB/s | 40 MB 00:38 2023-07-28T23:34:55.655 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2023-07-28T23:34:55.655 INFO:teuthology.orchestra.run.smithi139.stdout:Total 1.1 MB/s | 40 MB 00:38 2023-07-28T23:34:55.656 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2023-07-28T23:34:55.677 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2023-07-28T23:34:55.677 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2023-07-28T23:34:56.010 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2023-07-28T23:34:56.011 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2023-07-28T23:34:56.553 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2023-07-28T23:34:56.645 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-07-28T23:34:56.691 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-07-28T23:34:56.735 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-07-28T23:34:56.785 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : jq-1.6-15.el9.x86_64 4/6 2023-07-28T23:34:56.822 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libcephsqlite-2:18.1.3-11.g0484097b.el9.x86_64 5/6 2023-07-28T23:35:00.569 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libcephsqlite-2:18.1.3-11.g0484097b.el9.x86_64 5/6 2023-07-28T23:35:00.653 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-test-2:18.1.3-11.g0484097b.el9.x86_64 6/6 2023-07-28T23:35:02.886 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-test-2:18.1.3-11.g0484097b.el9.x86_64 6/6 2023-07-28T23:35:02.887 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-test-2:18.1.3-11.g0484097b.el9.x86_64 1/6 2023-07-28T23:35:02.887 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libcephsqlite-2:18.1.3-11.g0484097b.el9.x86_64 2/6 2023-07-28T23:35:02.887 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : jq-1.6-15.el9.x86_64 3/6 2023-07-28T23:35:02.887 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2023-07-28T23:35:02.887 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2023-07-28T23:35:03.249 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2023-07-28T23:35:03.250 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:35:03.250 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2023-07-28T23:35:03.250 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-test-2:18.1.3-11.g0484097b.el9.x86_64 jq-1.6-15.el9.x86_64 2023-07-28T23:35:03.250 INFO:teuthology.orchestra.run.smithi139.stdout: libcephsqlite-2:18.1.3-11.g0484097b.el9.x86_64 oniguruma-6.9.6-1.el9.5.x86_64 2023-07-28T23:35:03.251 INFO:teuthology.orchestra.run.smithi139.stdout: socat-1.7.4.1-5.el9.x86_64 xmlstarlet-1.6.1-20.el9.x86_64 2023-07-28T23:35:03.251 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:35:03.251 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2023-07-28T23:35:03.566 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph 2023-07-28T23:35:04.110 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:29 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:35:04.240 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2023-07-28T23:35:04.243 INFO:teuthology.orchestra.run.smithi139.stdout:======================================================================================= 2023-07-28T23:35:04.243 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2023-07-28T23:35:04.243 INFO:teuthology.orchestra.run.smithi139.stdout:======================================================================================= 2023-07-28T23:35:04.243 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2023-07-28T23:35:04.243 INFO:teuthology.orchestra.run.smithi139.stdout: ceph x86_64 2:18.1.3-11.g0484097b.el9 ceph 6.5 k 2023-07-28T23:35:04.244 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2023-07-28T23:35:04.244 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mds x86_64 2:18.1.3-11.g0484097b.el9 ceph 2.1 M 2023-07-28T23:35:04.244 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr x86_64 2:18.1.3-11.g0484097b.el9 ceph 1.4 M 2023-07-28T23:35:04.244 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-modules-core noarch 2:18.1.3-11.g0484097b.el9 ceph-noarch 240 k 2023-07-28T23:35:04.244 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mon x86_64 2:18.1.3-11.g0484097b.el9 ceph 4.4 M 2023-07-28T23:35:04.245 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-osd x86_64 2:18.1.3-11.g0484097b.el9 ceph 18 M 2023-07-28T23:35:04.245 INFO:teuthology.orchestra.run.smithi139.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-07-28T23:35:04.245 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-07-28T23:35:04.245 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-07-28T23:35:04.245 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-07-28T23:35:04.245 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-07-28T23:35:04.245 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-07-28T23:35:04.245 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-07-28T23:35:04.246 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-07-28T23:35:04.246 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-07-28T23:35:04.246 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-07-28T23:35:04.246 INFO:teuthology.orchestra.run.smithi139.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-07-28T23:35:04.246 INFO:teuthology.orchestra.run.smithi139.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-07-28T23:35:04.246 INFO:teuthology.orchestra.run.smithi139.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-07-28T23:35:04.247 INFO:teuthology.orchestra.run.smithi139.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-07-28T23:35:04.247 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-07-28T23:35:04.247 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-07-28T23:35:04.247 INFO:teuthology.orchestra.run.smithi139.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-07-28T23:35:04.247 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-07-28T23:35:04.247 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-07-28T23:35:04.247 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-07-28T23:35:04.247 INFO:teuthology.orchestra.run.smithi139.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-07-28T23:35:04.248 INFO:teuthology.orchestra.run.smithi139.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-07-28T23:35:04.248 INFO:teuthology.orchestra.run.smithi139.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-07-28T23:35:04.248 INFO:teuthology.orchestra.run.smithi139.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-07-28T23:35:04.248 INFO:teuthology.orchestra.run.smithi139.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-07-28T23:35:04.248 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:35:04.248 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2023-07-28T23:35:04.248 INFO:teuthology.orchestra.run.smithi139.stdout:======================================================================================= 2023-07-28T23:35:04.249 INFO:teuthology.orchestra.run.smithi139.stdout:Install 31 Packages 2023-07-28T23:35:04.249 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:35:04.249 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 29 M 2023-07-28T23:35:04.250 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 108 M 2023-07-28T23:35:04.250 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2023-07-28T23:35:05.356 INFO:teuthology.orchestra.run.smithi139.stdout:(1/31): ceph-18.1.3-11.g0484097b.el9.x86_64.rpm 14 kB/s | 6.5 kB 00:00 2023-07-28T23:35:07.240 INFO:teuthology.orchestra.run.smithi139.stdout:(2/31): ceph-mgr-18.1.3-11.g0484097b.el9.x86_64 626 kB/s | 1.4 MB 00:02 2023-07-28T23:35:07.723 INFO:teuthology.orchestra.run.smithi139.stdout:(3/31): ceph-mds-18.1.3-11.g0484097b.el9.x86_64 755 kB/s | 2.1 MB 00:02 2023-07-28T23:35:08.015 INFO:teuthology.orchestra.run.smithi139.stdout:(4/31): ceph-mgr-modules-core-18.1.3-11.g048409 825 kB/s | 240 kB 00:00 2023-07-28T23:35:08.565 INFO:teuthology.orchestra.run.smithi139.stdout:(5/31): python3-mako-1.1.4-6.el9.noarch.rpm 262 kB/s | 144 kB 00:00 2023-07-28T23:35:08.691 INFO:teuthology.orchestra.run.smithi139.stdout:(6/31): python3-markupsafe-1.1.1-12.el9.x86_64. 252 kB/s | 32 kB 00:00 2023-07-28T23:35:08.832 INFO:teuthology.orchestra.run.smithi139.stdout:(7/31): python3-pytz-2021.1-5.el9.noarch.rpm 334 kB/s | 47 kB 00:00 2023-07-28T23:35:08.975 INFO:teuthology.orchestra.run.smithi139.stdout:(8/31): python3-toml-0.10.2-6.el9.noarch.rpm 272 kB/s | 39 kB 00:00 2023-07-28T23:35:09.075 INFO:teuthology.orchestra.run.smithi139.stdout:(9/31): ceph-mon-18.1.3-11.g0484097b.el9.x86_64 1.2 MB/s | 4.4 MB 00:03 2023-07-28T23:35:09.223 INFO:teuthology.orchestra.run.smithi139.stdout:(10/31): python3-cffi-1.14.5-5.el9.x86_64.rpm 973 kB/s | 241 kB 00:00 2023-07-28T23:35:09.467 INFO:teuthology.orchestra.run.smithi139.stdout:(11/31): python3-cryptography-36.0.1-4.el9.x86_ 2.9 MB/s | 1.1 MB 00:00 2023-07-28T23:35:09.501 INFO:teuthology.orchestra.run.smithi139.stdout:(12/31): python3-ply-3.11-14.el9.noarch.rpm 371 kB/s | 103 kB 00:00 2023-07-28T23:35:09.609 INFO:teuthology.orchestra.run.smithi139.stdout:(13/31): python3-pycparser-2.20-6.el9.noarch.rp 876 kB/s | 124 kB 00:00 2023-07-28T23:35:10.051 INFO:teuthology.orchestra.run.smithi139.stdout:(14/31): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 79 kB/s | 43 kB 00:00 2023-07-28T23:35:10.268 INFO:teuthology.orchestra.run.smithi139.stdout:(15/31): python3-cheroot-8.6.0-4.el9.noarch.rpm 261 kB/s | 172 kB 00:00 2023-07-28T23:35:10.435 INFO:teuthology.orchestra.run.smithi139.stdout:(16/31): python3-jaraco-8.2.1-3.el9.noarch.rpm 64 kB/s | 11 kB 00:00 2023-07-28T23:35:10.469 INFO:teuthology.orchestra.run.smithi139.stdout:(17/31): python3-cherrypy-18.6.1-2.el9.noarch.r 859 kB/s | 358 kB 00:00 2023-07-28T23:35:10.602 INFO:teuthology.orchestra.run.smithi139.stdout:(18/31): python3-jaraco-classes-3.2.1-5.el9.noa 106 kB/s | 18 kB 00:00 2023-07-28T23:35:10.628 INFO:teuthology.orchestra.run.smithi139.stdout:(19/31): python3-jaraco-collections-3.0.0-8.el9 146 kB/s | 23 kB 00:00 2023-07-28T23:35:10.761 INFO:teuthology.orchestra.run.smithi139.stdout:(20/31): python3-jaraco-functools-3.5.0-2.el9.n 123 kB/s | 19 kB 00:00 2023-07-28T23:35:10.795 INFO:teuthology.orchestra.run.smithi139.stdout:(21/31): python3-jaraco-text-3.2.0-6.el9.noarch 118 kB/s | 20 kB 00:00 2023-07-28T23:35:10.978 INFO:teuthology.orchestra.run.smithi139.stdout:(22/31): python3-logutils-0.3.5-21.el9.noarch.r 213 kB/s | 46 kB 00:00 2023-07-28T23:35:11.337 INFO:teuthology.orchestra.run.smithi139.stdout:(23/31): python3-pecan-1.4.2-3.el9.noarch.rpm 760 kB/s | 272 kB 00:00 2023-07-28T23:35:11.371 INFO:teuthology.orchestra.run.smithi139.stdout:(24/31): python3-more-itertools-8.12.0-2.el9.no 137 kB/s | 79 kB 00:00 2023-07-28T23:35:11.529 INFO:teuthology.orchestra.run.smithi139.stdout:(25/31): python3-portend-3.1.0-2.el9.noarch.rpm 85 kB/s | 16 kB 00:00 2023-07-28T23:35:11.563 INFO:teuthology.orchestra.run.smithi139.stdout:(26/31): python3-pyOpenSSL-21.0.0-1.el9.noarch. 469 kB/s | 90 kB 00:00 2023-07-28T23:35:11.697 INFO:teuthology.orchestra.run.smithi139.stdout:(27/31): python3-tempora-5.0.0-2.el9.noarch.rpm 214 kB/s | 36 kB 00:00 2023-07-28T23:35:11.763 INFO:teuthology.orchestra.run.smithi139.stdout:(28/31): python3-webob-1.8.7-6.el9.noarch.rpm 1.1 MB/s | 230 kB 00:00 2023-07-28T23:35:11.939 INFO:teuthology.orchestra.run.smithi139.stdout:(29/31): python3-zc-lockfile-2.0-10.el9.noarch. 114 kB/s | 20 kB 00:00 2023-07-28T23:35:12.139 INFO:teuthology.orchestra.run.smithi139.stdout:(30/31): python3-werkzeug-2.0.3-3.el9.1.noarch. 967 kB/s | 427 kB 00:00 2023-07-28T23:35:13.326 INFO:teuthology.orchestra.run.smithi101.stdout:(6/6): ceph-test-18.1.3-11.g0484097b.el9.x86_64 881 kB/s | 40 MB 00:45 2023-07-28T23:35:13.328 INFO:teuthology.orchestra.run.smithi101.stdout:-------------------------------------------------------------------------------- 2023-07-28T23:35:13.328 INFO:teuthology.orchestra.run.smithi101.stdout:Total 901 kB/s | 40 MB 00:45 2023-07-28T23:35:13.329 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction check 2023-07-28T23:35:13.347 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction check succeeded. 2023-07-28T23:35:13.347 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction test 2023-07-28T23:35:13.692 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction test succeeded. 2023-07-28T23:35:13.692 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction 2023-07-28T23:35:13.706 INFO:teuthology.orchestra.run.smithi139.stdout:(31/31): ceph-osd-18.1.3-11.g0484097b.el9.x86_6 2.7 MB/s | 18 MB 00:06 2023-07-28T23:35:13.710 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2023-07-28T23:35:13.710 INFO:teuthology.orchestra.run.smithi139.stdout:Total 3.1 MB/s | 29 MB 00:09 2023-07-28T23:35:13.710 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2023-07-28T23:35:13.747 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2023-07-28T23:35:13.748 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2023-07-28T23:35:14.100 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2023-07-28T23:35:14.100 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2023-07-28T23:35:14.223 INFO:teuthology.orchestra.run.smithi101.stdout: Preparing : 1/1 2023-07-28T23:35:14.293 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-07-28T23:35:14.330 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-07-28T23:35:14.366 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-07-28T23:35:14.409 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : jq-1.6-15.el9.x86_64 4/6 2023-07-28T23:35:14.437 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libcephsqlite-2:18.1.3-11.g0484097b.el9.x86_64 5/6 2023-07-28T23:35:15.212 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2023-07-28T23:35:15.292 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/31 2023-07-28T23:35:15.341 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/31 2023-07-28T23:35:15.382 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/31 2023-07-28T23:35:15.482 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/31 2023-07-28T23:35:15.568 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/31 2023-07-28T23:35:15.618 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/31 2023-07-28T23:35:15.672 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/31 2023-07-28T23:35:15.729 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-ply-3.11-14.el9.noarch 8/31 2023-07-28T23:35:15.801 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 9/31 2023-07-28T23:35:16.006 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 10/31 2023-07-28T23:35:16.068 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 11/31 2023-07-28T23:35:16.115 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 12/31 2023-07-28T23:35:16.158 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 13/31 2023-07-28T23:35:16.196 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 14/31 2023-07-28T23:35:16.280 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 15/31 2023-07-28T23:35:16.323 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 16/31 2023-07-28T23:35:16.365 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 17/31 2023-07-28T23:35:16.414 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 18/31 2023-07-28T23:35:16.482 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 19/31 2023-07-28T23:35:16.517 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 20/31 2023-07-28T23:35:16.613 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 21/31 2023-07-28T23:35:16.655 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 22/31 2023-07-28T23:35:16.735 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/31 2023-07-28T23:35:16.882 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 24/31 2023-07-28T23:35:17.005 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 25/31 2023-07-28T23:35:17.160 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-modules-core-2:18.1.3-11.g0484097b.el9.no 26/31 2023-07-28T23:35:17.194 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-2:18.1.3-11.g0484097b.el9.x86_64 27/31 2023-07-28T23:35:17.219 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-2:18.1.3-11.g0484097b.el9.x86_64 27/31 2023-07-28T23:35:17.219 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-07-28T23:35:17.219 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-07-28T23:35:17.219 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-07-28T23:35:17.220 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-07-28T23:35:17.220 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:35:18.248 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: libcephsqlite-2:18.1.3-11.g0484097b.el9.x86_64 5/6 2023-07-28T23:35:18.325 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-test-2:18.1.3-11.g0484097b.el9.x86_64 6/6 2023-07-28T23:35:18.737 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-osd-2:18.1.3-11.g0484097b.el9.x86_64 28/31 2023-07-28T23:35:18.763 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-osd-2:18.1.3-11.g0484097b.el9.x86_64 28/31 2023-07-28T23:35:18.764 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-07-28T23:35:18.764 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-07-28T23:35:18.764 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-07-28T23:35:18.764 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-07-28T23:35:18.764 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:35:19.220 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mon-2:18.1.3-11.g0484097b.el9.x86_64 29/31 2023-07-28T23:35:19.243 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mon-2:18.1.3-11.g0484097b.el9.x86_64 29/31 2023-07-28T23:35:19.244 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-07-28T23:35:19.244 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-07-28T23:35:19.244 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-07-28T23:35:19.244 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-07-28T23:35:19.244 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:35:19.471 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mds-2:18.1.3-11.g0484097b.el9.x86_64 30/31 2023-07-28T23:35:19.494 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mds-2:18.1.3-11.g0484097b.el9.x86_64 30/31 2023-07-28T23:35:19.494 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-07-28T23:35:19.494 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-07-28T23:35:19.495 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-07-28T23:35:19.495 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-07-28T23:35:19.495 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:35:19.567 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-2:18.1.3-11.g0484097b.el9.x86_64 31/31 2023-07-28T23:35:20.259 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-test-2:18.1.3-11.g0484097b.el9.x86_64 6/6 2023-07-28T23:35:20.260 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-test-2:18.1.3-11.g0484097b.el9.x86_64 1/6 2023-07-28T23:35:20.260 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libcephsqlite-2:18.1.3-11.g0484097b.el9.x86_64 2/6 2023-07-28T23:35:20.260 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : jq-1.6-15.el9.x86_64 3/6 2023-07-28T23:35:20.260 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2023-07-28T23:35:20.260 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2023-07-28T23:35:20.598 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2023-07-28T23:35:20.598 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:35:20.598 INFO:teuthology.orchestra.run.smithi101.stdout:Installed: 2023-07-28T23:35:20.599 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-test-2:18.1.3-11.g0484097b.el9.x86_64 jq-1.6-15.el9.x86_64 2023-07-28T23:35:20.599 INFO:teuthology.orchestra.run.smithi101.stdout: libcephsqlite-2:18.1.3-11.g0484097b.el9.x86_64 oniguruma-6.9.6-1.el9.5.x86_64 2023-07-28T23:35:20.599 INFO:teuthology.orchestra.run.smithi101.stdout: socat-1.7.4.1-5.el9.x86_64 xmlstarlet-1.6.1-20.el9.x86_64 2023-07-28T23:35:20.599 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:35:20.599 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-07-28T23:35:20.899 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install ceph 2023-07-28T23:35:21.434 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:01:46 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:35:21.489 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-2:18.1.3-11.g0484097b.el9.x86_64 31/31 2023-07-28T23:35:21.489 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-2:18.1.3-11.g0484097b.el9.x86_64 1/31 2023-07-28T23:35:21.489 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mds-2:18.1.3-11.g0484097b.el9.x86_64 2/31 2023-07-28T23:35:21.490 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-2:18.1.3-11.g0484097b.el9.x86_64 3/31 2023-07-28T23:35:21.490 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mon-2:18.1.3-11.g0484097b.el9.x86_64 4/31 2023-07-28T23:35:21.490 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-osd-2:18.1.3-11.g0484097b.el9.x86_64 5/31 2023-07-28T23:35:21.490 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-modules-core-2:18.1.3-11.g0484097b.el9.no 6/31 2023-07-28T23:35:21.490 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/31 2023-07-28T23:35:21.490 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/31 2023-07-28T23:35:21.490 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/31 2023-07-28T23:35:21.491 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/31 2023-07-28T23:35:21.491 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/31 2023-07-28T23:35:21.491 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/31 2023-07-28T23:35:21.491 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/31 2023-07-28T23:35:21.491 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/31 2023-07-28T23:35:21.491 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 15/31 2023-07-28T23:35:21.491 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 16/31 2023-07-28T23:35:21.491 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 17/31 2023-07-28T23:35:21.492 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 18/31 2023-07-28T23:35:21.492 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 19/31 2023-07-28T23:35:21.492 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 20/31 2023-07-28T23:35:21.492 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 21/31 2023-07-28T23:35:21.492 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 22/31 2023-07-28T23:35:21.492 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 23/31 2023-07-28T23:35:21.492 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 24/31 2023-07-28T23:35:21.493 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 25/31 2023-07-28T23:35:21.493 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 26/31 2023-07-28T23:35:21.493 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/31 2023-07-28T23:35:21.493 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 28/31 2023-07-28T23:35:21.493 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 29/31 2023-07-28T23:35:21.494 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 30/31 2023-07-28T23:35:21.565 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-07-28T23:35:21.568 INFO:teuthology.orchestra.run.smithi101.stdout:======================================================================================= 2023-07-28T23:35:21.568 INFO:teuthology.orchestra.run.smithi101.stdout: Package Arch Version Repository Size 2023-07-28T23:35:21.568 INFO:teuthology.orchestra.run.smithi101.stdout:======================================================================================= 2023-07-28T23:35:21.568 INFO:teuthology.orchestra.run.smithi101.stdout:Installing: 2023-07-28T23:35:21.568 INFO:teuthology.orchestra.run.smithi101.stdout: ceph x86_64 2:18.1.3-11.g0484097b.el9 ceph 6.5 k 2023-07-28T23:35:21.568 INFO:teuthology.orchestra.run.smithi101.stdout:Installing dependencies: 2023-07-28T23:35:21.569 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mds x86_64 2:18.1.3-11.g0484097b.el9 ceph 2.1 M 2023-07-28T23:35:21.569 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr x86_64 2:18.1.3-11.g0484097b.el9 ceph 1.4 M 2023-07-28T23:35:21.569 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-modules-core noarch 2:18.1.3-11.g0484097b.el9 ceph-noarch 240 k 2023-07-28T23:35:21.569 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mon x86_64 2:18.1.3-11.g0484097b.el9 ceph 4.4 M 2023-07-28T23:35:21.569 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-osd x86_64 2:18.1.3-11.g0484097b.el9 ceph 18 M 2023-07-28T23:35:21.569 INFO:teuthology.orchestra.run.smithi101.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-07-28T23:35:21.569 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-07-28T23:35:21.569 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-07-28T23:35:21.570 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-07-28T23:35:21.570 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-07-28T23:35:21.570 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-07-28T23:35:21.570 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-07-28T23:35:21.570 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-07-28T23:35:21.570 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-07-28T23:35:21.570 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-07-28T23:35:21.571 INFO:teuthology.orchestra.run.smithi101.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-07-28T23:35:21.571 INFO:teuthology.orchestra.run.smithi101.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-07-28T23:35:21.571 INFO:teuthology.orchestra.run.smithi101.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-07-28T23:35:21.571 INFO:teuthology.orchestra.run.smithi101.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-07-28T23:35:21.571 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-07-28T23:35:21.571 INFO:teuthology.orchestra.run.smithi101.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-07-28T23:35:21.571 INFO:teuthology.orchestra.run.smithi101.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-07-28T23:35:21.571 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-07-28T23:35:21.572 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-07-28T23:35:21.572 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-07-28T23:35:21.572 INFO:teuthology.orchestra.run.smithi101.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-07-28T23:35:21.572 INFO:teuthology.orchestra.run.smithi101.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-07-28T23:35:21.572 INFO:teuthology.orchestra.run.smithi101.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-07-28T23:35:21.572 INFO:teuthology.orchestra.run.smithi101.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-07-28T23:35:21.572 INFO:teuthology.orchestra.run.smithi101.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-07-28T23:35:21.572 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:35:21.573 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2023-07-28T23:35:21.573 INFO:teuthology.orchestra.run.smithi101.stdout:======================================================================================= 2023-07-28T23:35:21.573 INFO:teuthology.orchestra.run.smithi101.stdout:Install 31 Packages 2023-07-28T23:35:21.573 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:35:21.574 INFO:teuthology.orchestra.run.smithi101.stdout:Total download size: 29 M 2023-07-28T23:35:21.574 INFO:teuthology.orchestra.run.smithi101.stdout:Installed size: 108 M 2023-07-28T23:35:21.574 INFO:teuthology.orchestra.run.smithi101.stdout:Downloading Packages: 2023-07-28T23:35:22.577 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 31/31 2023-07-28T23:35:22.577 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:35:22.578 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2023-07-28T23:35:22.578 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:35:22.578 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mds-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:35:22.578 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:35:22.578 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-modules-core-2:18.1.3-11.g0484097b.el9.noarch 2023-07-28T23:35:22.578 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mon-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:35:22.578 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-osd-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:35:22.579 INFO:teuthology.orchestra.run.smithi139.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-07-28T23:35:22.579 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-07-28T23:35:22.579 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-07-28T23:35:22.579 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-07-28T23:35:22.579 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-07-28T23:35:22.579 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-07-28T23:35:22.579 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-07-28T23:35:22.580 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-07-28T23:35:22.580 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-07-28T23:35:22.580 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-07-28T23:35:22.580 INFO:teuthology.orchestra.run.smithi139.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-07-28T23:35:22.580 INFO:teuthology.orchestra.run.smithi139.stdout: python3-mako-1.1.4-6.el9.noarch 2023-07-28T23:35:22.580 INFO:teuthology.orchestra.run.smithi139.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-07-28T23:35:22.580 INFO:teuthology.orchestra.run.smithi139.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-07-28T23:35:22.581 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-07-28T23:35:22.581 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ply-3.11-14.el9.noarch 2023-07-28T23:35:22.581 INFO:teuthology.orchestra.run.smithi139.stdout: python3-portend-3.1.0-2.el9.noarch 2023-07-28T23:35:22.581 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-07-28T23:35:22.581 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pycparser-2.20-6.el9.noarch 2023-07-28T23:35:22.581 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pytz-2021.1-5.el9.noarch 2023-07-28T23:35:22.581 INFO:teuthology.orchestra.run.smithi139.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-07-28T23:35:22.581 INFO:teuthology.orchestra.run.smithi139.stdout: python3-toml-0.10.2-6.el9.noarch 2023-07-28T23:35:22.582 INFO:teuthology.orchestra.run.smithi139.stdout: python3-webob-1.8.7-6.el9.noarch 2023-07-28T23:35:22.582 INFO:teuthology.orchestra.run.smithi139.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-07-28T23:35:22.582 INFO:teuthology.orchestra.run.smithi139.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-07-28T23:35:22.582 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:35:22.582 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2023-07-28T23:35:22.953 INFO:teuthology.orchestra.run.smithi101.stdout:(1/31): ceph-18.1.3-11.g0484097b.el9.x86_64.rpm 15 kB/s | 6.5 kB 00:00 2023-07-28T23:35:22.995 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-base 2023-07-28T23:35:23.526 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:48 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:35:23.607 INFO:teuthology.orchestra.run.smithi139.stdout:Package ceph-base-2:18.1.3-11.g0484097b.el9.x86_64 is already installed. 2023-07-28T23:35:23.657 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2023-07-28T23:35:23.658 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2023-07-28T23:35:23.658 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2023-07-28T23:35:23.735 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install cephadm 2023-07-28T23:35:24.264 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:49 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:35:24.394 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2023-07-28T23:35:24.395 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-07-28T23:35:24.395 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2023-07-28T23:35:24.395 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-07-28T23:35:24.395 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2023-07-28T23:35:24.396 INFO:teuthology.orchestra.run.smithi139.stdout: cephadm noarch 2:18.1.3-11.g0484097b.el9 ceph-noarch 209 k 2023-07-28T23:35:24.396 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:35:24.396 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2023-07-28T23:35:24.396 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-07-28T23:35:24.396 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2023-07-28T23:35:24.396 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:35:24.396 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 209 k 2023-07-28T23:35:24.397 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 200 k 2023-07-28T23:35:24.397 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2023-07-28T23:35:24.411 INFO:teuthology.orchestra.run.smithi101.stdout:(2/31): ceph-mgr-18.1.3-11.g0484097b.el9.x86_64 780 kB/s | 1.4 MB 00:01 2023-07-28T23:35:25.686 INFO:teuthology.orchestra.run.smithi101.stdout:(3/31): ceph-mon-18.1.3-11.g0484097b.el9.x86_64 1.6 MB/s | 4.4 MB 00:02 2023-07-28T23:35:25.953 INFO:teuthology.orchestra.run.smithi101.stdout:(4/31): ceph-mds-18.1.3-11.g0484097b.el9.x86_64 625 kB/s | 2.1 MB 00:03 2023-07-28T23:35:25.978 INFO:teuthology.orchestra.run.smithi101.stdout:(5/31): ceph-mgr-modules-core-18.1.3-11.g048409 824 kB/s | 240 kB 00:00 2023-07-28T23:35:26.312 INFO:teuthology.orchestra.run.smithi101.stdout:(6/31): python3-markupsafe-1.1.1-12.el9.x86_64. 95 kB/s | 32 kB 00:00 2023-07-28T23:35:26.462 INFO:teuthology.orchestra.run.smithi101.stdout:(7/31): python3-mako-1.1.4-6.el9.noarch.rpm 283 kB/s | 144 kB 00:00 2023-07-28T23:35:26.488 INFO:teuthology.orchestra.run.smithi101.stdout:(8/31): python3-pytz-2021.1-5.el9.noarch.rpm 270 kB/s | 47 kB 00:00 2023-07-28T23:35:26.580 INFO:teuthology.orchestra.run.smithi101.stdout:(9/31): python3-toml-0.10.2-6.el9.noarch.rpm 330 kB/s | 39 kB 00:00 2023-07-28T23:35:26.694 INFO:teuthology.orchestra.run.smithi139.stdout:cephadm-18.1.3-11.g0484097b.el9.noarch.rpm 91 kB/s | 209 kB 00:02 2023-07-28T23:35:26.695 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2023-07-28T23:35:26.695 INFO:teuthology.orchestra.run.smithi139.stdout:Total 91 kB/s | 209 kB 00:02 2023-07-28T23:35:26.696 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2023-07-28T23:35:26.696 INFO:teuthology.orchestra.run.smithi101.stdout:(10/31): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.1 MB/s | 241 kB 00:00 2023-07-28T23:35:26.701 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2023-07-28T23:35:26.702 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2023-07-28T23:35:26.704 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2023-07-28T23:35:26.705 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2023-07-28T23:35:26.776 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2023-07-28T23:35:26.927 INFO:teuthology.orchestra.run.smithi101.stdout:(11/31): python3-cryptography-36.0.1-4.el9.x86_ 3.3 MB/s | 1.1 MB 00:00 2023-07-28T23:35:26.955 INFO:teuthology.orchestra.run.smithi101.stdout:(12/31): python3-ply-3.11-14.el9.noarch.rpm 398 kB/s | 103 kB 00:00 2023-07-28T23:35:27.039 INFO:teuthology.orchestra.run.smithi101.stdout:(13/31): python3-pycparser-2.20-6.el9.noarch.rp 1.1 MB/s | 124 kB 00:00 2023-07-28T23:35:27.141 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: cephadm-2:18.1.3-11.g0484097b.el9.noarch 1/1 2023-07-28T23:35:27.224 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : cephadm-2:18.1.3-11.g0484097b.el9.noarch 1/1 2023-07-28T23:35:27.608 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: cephadm-2:18.1.3-11.g0484097b.el9.noarch 1/1 2023-07-28T23:35:27.925 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : cephadm-2:18.1.3-11.g0484097b.el9.noarch 1/1 2023-07-28T23:35:27.925 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:35:27.926 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2023-07-28T23:35:27.926 INFO:teuthology.orchestra.run.smithi139.stdout: cephadm-2:18.1.3-11.g0484097b.el9.noarch 2023-07-28T23:35:27.926 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:35:27.926 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2023-07-28T23:35:28.031 INFO:teuthology.orchestra.run.smithi101.stdout:(14/31): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 40 kB/s | 43 kB 00:01 2023-07-28T23:35:28.136 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-immutable-object-cache 2023-07-28T23:35:28.464 INFO:teuthology.orchestra.run.smithi101.stdout:(15/31): python3-cheroot-8.6.0-4.el9.noarch.rpm 120 kB/s | 172 kB 00:01 2023-07-28T23:35:28.678 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:53 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:35:28.698 INFO:teuthology.orchestra.run.smithi101.stdout:(16/31): python3-jaraco-8.2.1-3.el9.noarch.rpm 46 kB/s | 11 kB 00:00 2023-07-28T23:35:28.811 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2023-07-28T23:35:28.812 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-07-28T23:35:28.813 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repo Size 2023-07-28T23:35:28.813 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-07-28T23:35:28.813 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2023-07-28T23:35:28.813 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-immutable-object-cache x86_64 2:18.1.3-11.g0484097b.el9 ceph 142 k 2023-07-28T23:35:28.813 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:35:28.813 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2023-07-28T23:35:28.813 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-07-28T23:35:28.814 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2023-07-28T23:35:28.814 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:35:28.814 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 142 k 2023-07-28T23:35:28.814 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 400 k 2023-07-28T23:35:28.814 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2023-07-28T23:35:28.907 INFO:teuthology.orchestra.run.smithi101.stdout:(17/31): python3-jaraco-classes-3.2.1-5.el9.noa 85 kB/s | 18 kB 00:00 2023-07-28T23:35:29.115 INFO:teuthology.orchestra.run.smithi101.stdout:(18/31): python3-cherrypy-18.6.1-2.el9.noarch.r 331 kB/s | 358 kB 00:01 2023-07-28T23:35:29.132 INFO:teuthology.orchestra.run.smithi101.stdout:(19/31): python3-jaraco-collections-3.0.0-8.el9 103 kB/s | 23 kB 00:00 2023-07-28T23:35:29.324 INFO:teuthology.orchestra.run.smithi101.stdout:(20/31): python3-jaraco-functools-3.5.0-2.el9.n 93 kB/s | 19 kB 00:00 2023-07-28T23:35:29.341 INFO:teuthology.orchestra.run.smithi101.stdout:(21/31): python3-jaraco-text-3.2.0-6.el9.noarch 94 kB/s | 20 kB 00:00 2023-07-28T23:35:29.550 INFO:teuthology.orchestra.run.smithi101.stdout:(22/31): python3-logutils-0.3.5-21.el9.noarch.r 205 kB/s | 46 kB 00:00 2023-07-28T23:35:29.575 INFO:teuthology.orchestra.run.smithi101.stdout:(23/31): python3-more-itertools-8.12.0-2.el9.no 337 kB/s | 79 kB 00:00 2023-07-28T23:35:29.742 INFO:teuthology.orchestra.run.smithi101.stdout:(24/31): python3-portend-3.1.0-2.el9.noarch.rpm 99 kB/s | 16 kB 00:00 2023-07-28T23:35:29.909 INFO:teuthology.orchestra.run.smithi101.stdout:(25/31): python3-pyOpenSSL-21.0.0-1.el9.noarch. 540 kB/s | 90 kB 00:00 2023-07-28T23:35:30.101 INFO:teuthology.orchestra.run.smithi101.stdout:(26/31): python3-tempora-5.0.0-2.el9.noarch.rpm 186 kB/s | 36 kB 00:00 2023-07-28T23:35:30.159 INFO:teuthology.orchestra.run.smithi139.stdout:ceph-immutable-object-cache-18.1.3-11.g0484097b 106 kB/s | 142 kB 00:01 2023-07-28T23:35:30.159 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2023-07-28T23:35:30.160 INFO:teuthology.orchestra.run.smithi139.stdout:Total 106 kB/s | 142 kB 00:01 2023-07-28T23:35:30.160 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2023-07-28T23:35:30.169 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2023-07-28T23:35:30.169 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2023-07-28T23:35:30.228 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2023-07-28T23:35:30.228 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2023-07-28T23:35:30.268 INFO:teuthology.orchestra.run.smithi101.stdout:(27/31): python3-pecan-1.4.2-3.el9.noarch.rpm 379 kB/s | 272 kB 00:00 2023-07-28T23:35:30.370 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2023-07-28T23:35:30.434 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-immutable-object-cache-2:18.1.3-11.g0484097b.el 1/1 2023-07-28T23:35:30.459 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.1.3-11.g0484097b.el 1/1 2023-07-28T23:35:30.460 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-07-28T23:35:30.460 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-07-28T23:35:30.460 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:35:31.310 INFO:teuthology.orchestra.run.smithi101.stdout:(28/31): python3-webob-1.8.7-6.el9.noarch.rpm 190 kB/s | 230 kB 00:01 2023-07-28T23:35:31.485 INFO:teuthology.orchestra.run.smithi101.stdout:(29/31): python3-zc-lockfile-2.0-10.el9.noarch. 114 kB/s | 20 kB 00:00 2023-07-28T23:35:31.535 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-immutable-object-cache-2:18.1.3-11.g0484097b.el 1/1 2023-07-28T23:35:31.535 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:35:31.536 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2023-07-28T23:35:31.536 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-immutable-object-cache-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:35:31.536 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:35:31.536 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2023-07-28T23:35:31.762 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-mgr 2023-07-28T23:35:32.300 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:57 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:35:32.380 INFO:teuthology.orchestra.run.smithi139.stdout:Package ceph-mgr-2:18.1.3-11.g0484097b.el9.x86_64 is already installed. 2023-07-28T23:35:32.429 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2023-07-28T23:35:32.430 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2023-07-28T23:35:32.430 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2023-07-28T23:35:32.507 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-mgr-dashboard 2023-07-28T23:35:32.552 INFO:teuthology.orchestra.run.smithi101.stdout:(30/31): python3-werkzeug-2.0.3-3.el9.1.noarch. 187 kB/s | 427 kB 00:02 2023-07-28T23:35:33.038 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:58 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:35:33.169 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2023-07-28T23:35:33.170 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-07-28T23:35:33.170 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2023-07-28T23:35:33.170 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-07-28T23:35:33.171 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2023-07-28T23:35:33.171 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-dashboard noarch 2:18.1.3-11.g0484097b.el9 ceph-noarch 1.7 M 2023-07-28T23:35:33.171 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2023-07-28T23:35:33.171 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-grafana-dashboards noarch 2:18.1.3-11.g0484097b.el9 ceph-noarch 23 k 2023-07-28T23:35:33.171 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-prometheus-alerts noarch 2:18.1.3-11.g0484097b.el9 ceph-noarch 15 k 2023-07-28T23:35:33.171 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-07-28T23:35:33.171 INFO:teuthology.orchestra.run.smithi139.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-07-28T23:35:33.171 INFO:teuthology.orchestra.run.smithi139.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-07-28T23:35:33.172 INFO:teuthology.orchestra.run.smithi139.stdout:Installing weak dependencies: 2023-07-28T23:35:33.172 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-07-28T23:35:33.172 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:35:33.172 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2023-07-28T23:35:33.172 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-07-28T23:35:33.172 INFO:teuthology.orchestra.run.smithi139.stdout:Install 7 Packages 2023-07-28T23:35:33.172 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:35:33.173 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 2.0 M 2023-07-28T23:35:33.173 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 8.1 M 2023-07-28T23:35:33.173 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2023-07-28T23:35:34.020 INFO:teuthology.orchestra.run.smithi139.stdout:(1/7): ceph-prometheus-alerts-18.1.3-11.g048409 38 kB/s | 15 kB 00:00 2023-07-28T23:35:34.046 INFO:teuthology.orchestra.run.smithi139.stdout:(2/7): ceph-grafana-dashboards-18.1.3-11.g04840 56 kB/s | 23 kB 00:00 2023-07-28T23:35:34.379 INFO:teuthology.orchestra.run.smithi139.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 25 kB/s | 9.0 kB 00:00 2023-07-28T23:35:34.496 INFO:teuthology.orchestra.run.smithi139.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 91 kB/s | 41 kB 00:00 2023-07-28T23:35:34.596 INFO:teuthology.orchestra.run.smithi139.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 142 kB/s | 31 kB 00:00 2023-07-28T23:35:34.713 INFO:teuthology.orchestra.run.smithi139.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 869 kB/s | 188 kB 00:00 2023-07-28T23:35:38.805 INFO:teuthology.orchestra.run.smithi139.stdout:(7/7): ceph-mgr-dashboard-18.1.3-11.g0484097b.e 336 kB/s | 1.7 MB 00:05 2023-07-28T23:35:38.809 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2023-07-28T23:35:38.809 INFO:teuthology.orchestra.run.smithi139.stdout:Total 362 kB/s | 2.0 MB 00:05 2023-07-28T23:35:38.810 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2023-07-28T23:35:38.818 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2023-07-28T23:35:38.818 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2023-07-28T23:35:38.877 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2023-07-28T23:35:38.878 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2023-07-28T23:35:39.238 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2023-07-28T23:35:39.314 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-07-28T23:35:39.356 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-07-28T23:35:39.402 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-07-28T23:35:39.443 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-07-28T23:35:39.480 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-prometheus-alerts-2:18.1.3-11.g0484097b.el9.noa 5/7 2023-07-28T23:35:39.715 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-grafana-dashboards-2:18.1.3-11.g0484097b.el9.no 6/7 2023-07-28T23:35:39.750 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-dashboard-2:18.1.3-11.g0484097b.el9.noarch 7/7 2023-07-28T23:35:40.306 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.1.3-11.g0484097b.el9.noarch 7/7 2023-07-28T23:35:40.306 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-grafana-dashboards-2:18.1.3-11.g0484097b.el9.no 1/7 2023-07-28T23:35:40.307 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-dashboard-2:18.1.3-11.g0484097b.el9.noarch 2/7 2023-07-28T23:35:40.307 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-prometheus-alerts-2:18.1.3-11.g0484097b.el9.noa 3/7 2023-07-28T23:35:40.307 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-07-28T23:35:40.307 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-07-28T23:35:40.307 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-07-28T23:35:40.601 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-07-28T23:35:40.602 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:35:40.602 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2023-07-28T23:35:40.602 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-grafana-dashboards-2:18.1.3-11.g0484097b.el9.noarch 2023-07-28T23:35:40.602 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-dashboard-2:18.1.3-11.g0484097b.el9.noarch 2023-07-28T23:35:40.603 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-prometheus-alerts-2:18.1.3-11.g0484097b.el9.noarch 2023-07-28T23:35:40.603 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-07-28T23:35:40.603 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-07-28T23:35:40.604 INFO:teuthology.orchestra.run.smithi139.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-07-28T23:35:40.604 INFO:teuthology.orchestra.run.smithi139.stdout: python3-routes-2.5.1-5.el9.noarch 2023-07-28T23:35:40.604 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:35:40.604 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2023-07-28T23:35:40.839 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-mgr-diskprediction-local 2023-07-28T23:35:41.370 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:02:06 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:35:41.498 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2023-07-28T23:35:41.499 INFO:teuthology.orchestra.run.smithi139.stdout:========================================================================================== 2023-07-28T23:35:41.499 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2023-07-28T23:35:41.499 INFO:teuthology.orchestra.run.smithi139.stdout:========================================================================================== 2023-07-28T23:35:41.500 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2023-07-28T23:35:41.500 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-diskprediction-local noarch 2:18.1.3-11.g0484097b.el9 ceph-noarch 7.4 M 2023-07-28T23:35:41.500 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2023-07-28T23:35:41.500 INFO:teuthology.orchestra.run.smithi139.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-07-28T23:35:41.500 INFO:teuthology.orchestra.run.smithi139.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-07-28T23:35:41.500 INFO:teuthology.orchestra.run.smithi139.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-07-28T23:35:41.501 INFO:teuthology.orchestra.run.smithi139.stdout: libgfortran x86_64 11.4.1-2.1.el9 CentOS-BaseOS 807 k 2023-07-28T23:35:41.501 INFO:teuthology.orchestra.run.smithi139.stdout: libquadmath x86_64 11.4.1-2.1.el9 CentOS-BaseOS 197 k 2023-07-28T23:35:41.501 INFO:teuthology.orchestra.run.smithi139.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-07-28T23:35:41.501 INFO:teuthology.orchestra.run.smithi139.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-07-28T23:35:41.501 INFO:teuthology.orchestra.run.smithi139.stdout: python3-devel x86_64 3.9.16-1.el9 appstream 247 k 2023-07-28T23:35:41.501 INFO:teuthology.orchestra.run.smithi139.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-07-28T23:35:41.501 INFO:teuthology.orchestra.run.smithi139.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-07-28T23:35:41.502 INFO:teuthology.orchestra.run.smithi139.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-07-28T23:35:41.502 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:35:41.502 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2023-07-28T23:35:41.502 INFO:teuthology.orchestra.run.smithi139.stdout:========================================================================================== 2023-07-28T23:35:41.502 INFO:teuthology.orchestra.run.smithi139.stdout:Install 12 Packages 2023-07-28T23:35:41.502 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:35:41.503 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 37 M 2023-07-28T23:35:41.503 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 209 M 2023-07-28T23:35:41.503 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2023-07-28T23:35:42.012 INFO:teuthology.orchestra.run.smithi139.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 212 kB/s | 32 kB 00:00 2023-07-28T23:35:42.054 INFO:teuthology.orchestra.run.smithi139.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 422 kB/s | 17 kB 00:00 2023-07-28T23:35:42.104 INFO:teuthology.orchestra.run.smithi139.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 686 kB/s | 34 kB 00:00 2023-07-28T23:35:42.221 INFO:teuthology.orchestra.run.smithi139.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.3 MB/s | 3.0 MB 00:00 2023-07-28T23:35:42.438 INFO:teuthology.orchestra.run.smithi139.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 14 MB/s | 4.7 MB 00:00 2023-07-28T23:35:42.522 INFO:teuthology.orchestra.run.smithi139.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 4.8 MB 00:00 2023-07-28T23:35:42.555 INFO:teuthology.orchestra.run.smithi139.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.8 MB/s | 217 kB 00:00 2023-07-28T23:35:42.672 INFO:teuthology.orchestra.run.smithi139.stdout:(8/12): libgfortran-11.4.1-2.1.el9.x86_64.rpm 6.8 MB/s | 807 kB 00:00 2023-07-28T23:35:42.873 INFO:teuthology.orchestra.run.smithi139.stdout:(9/12): libquadmath-11.4.1-2.1.el9.x86_64.rpm 995 kB/s | 197 kB 00:00 2023-07-28T23:35:43.073 INFO:teuthology.orchestra.run.smithi139.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 28 MB/s | 16 MB 00:00 2023-07-28T23:35:43.890 INFO:teuthology.orchestra.run.smithi139.stdout:(11/12): python3-devel-3.9.16-1.el9.x86_64.rpm 243 kB/s | 247 kB 00:01 2023-07-28T23:35:50.301 INFO:teuthology.orchestra.run.smithi101.stdout:(31/31): ceph-osd-18.1.3-11.g0484097b.el9.x86_6 695 kB/s | 18 MB 00:25 2023-07-28T23:35:50.307 INFO:teuthology.orchestra.run.smithi101.stdout:-------------------------------------------------------------------------------- 2023-07-28T23:35:50.308 INFO:teuthology.orchestra.run.smithi101.stdout:Total 1.0 MB/s | 29 MB 00:28 2023-07-28T23:35:50.308 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction check 2023-07-28T23:35:50.344 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction check succeeded. 2023-07-28T23:35:50.344 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction test 2023-07-28T23:35:50.674 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction test succeeded. 2023-07-28T23:35:50.674 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction 2023-07-28T23:35:51.928 INFO:teuthology.orchestra.run.smithi101.stdout: Preparing : 1/1 2023-07-28T23:35:51.994 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/31 2023-07-28T23:35:52.026 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/31 2023-07-28T23:35:52.059 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/31 2023-07-28T23:35:52.150 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/31 2023-07-28T23:35:52.236 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/31 2023-07-28T23:35:52.278 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/31 2023-07-28T23:35:52.315 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/31 2023-07-28T23:35:52.364 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-ply-3.11-14.el9.noarch 8/31 2023-07-28T23:35:52.428 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 9/31 2023-07-28T23:35:52.611 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 10/31 2023-07-28T23:35:52.662 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 11/31 2023-07-28T23:35:52.693 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 12/31 2023-07-28T23:35:52.727 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 13/31 2023-07-28T23:35:52.757 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 14/31 2023-07-28T23:35:52.831 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 15/31 2023-07-28T23:35:52.867 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 16/31 2023-07-28T23:35:52.917 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 17/31 2023-07-28T23:35:52.939 INFO:teuthology.orchestra.run.smithi139.stdout:(12/12): ceph-mgr-diskprediction-local-18.1.3-1 685 kB/s | 7.4 MB 00:11 2023-07-28T23:35:52.943 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2023-07-28T23:35:52.943 INFO:teuthology.orchestra.run.smithi139.stdout:Total 3.2 MB/s | 37 MB 00:11 2023-07-28T23:35:52.962 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2023-07-28T23:35:52.974 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 18/31 2023-07-28T23:35:52.989 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2023-07-28T23:35:52.989 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2023-07-28T23:35:53.009 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 19/31 2023-07-28T23:35:53.044 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 20/31 2023-07-28T23:35:53.131 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 21/31 2023-07-28T23:35:53.166 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 22/31 2023-07-28T23:35:53.237 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/31 2023-07-28T23:35:53.372 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 24/31 2023-07-28T23:35:53.427 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2023-07-28T23:35:53.427 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2023-07-28T23:35:53.505 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 25/31 2023-07-28T23:35:53.670 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-mgr-modules-core-2:18.1.3-11.g0484097b.el9.no 26/31 2023-07-28T23:35:53.704 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-mgr-2:18.1.3-11.g0484097b.el9.x86_64 27/31 2023-07-28T23:35:53.729 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-mgr-2:18.1.3-11.g0484097b.el9.x86_64 27/31 2023-07-28T23:35:53.738 INFO:teuthology.orchestra.run.smithi101.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-07-28T23:35:53.739 INFO:teuthology.orchestra.run.smithi101.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-07-28T23:35:53.739 INFO:teuthology.orchestra.run.smithi101.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-07-28T23:35:53.739 INFO:teuthology.orchestra.run.smithi101.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-07-28T23:35:53.739 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:35:54.128 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2023-07-28T23:35:54.222 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libquadmath-11.4.1-2.1.el9.x86_64 1/12 2023-07-28T23:35:54.259 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libgfortran-11.4.1-2.1.el9.x86_64 2/12 2023-07-28T23:35:54.361 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-07-28T23:35:54.405 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-devel-3.9.16-1.el9.x86_64 4/12 2023-07-28T23:35:54.581 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-07-28T23:35:54.620 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-07-28T23:35:54.710 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-07-28T23:35:55.230 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-osd-2:18.1.3-11.g0484097b.el9.x86_64 28/31 2023-07-28T23:35:55.257 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-osd-2:18.1.3-11.g0484097b.el9.x86_64 28/31 2023-07-28T23:35:55.257 INFO:teuthology.orchestra.run.smithi101.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-07-28T23:35:55.257 INFO:teuthology.orchestra.run.smithi101.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-07-28T23:35:55.257 INFO:teuthology.orchestra.run.smithi101.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-07-28T23:35:55.257 INFO:teuthology.orchestra.run.smithi101.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-07-28T23:35:55.257 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:35:55.365 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-07-28T23:35:55.491 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-07-28T23:35:55.698 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-mon-2:18.1.3-11.g0484097b.el9.x86_64 29/31 2023-07-28T23:35:55.721 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-mon-2:18.1.3-11.g0484097b.el9.x86_64 29/31 2023-07-28T23:35:55.721 INFO:teuthology.orchestra.run.smithi101.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-07-28T23:35:55.721 INFO:teuthology.orchestra.run.smithi101.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-07-28T23:35:55.721 INFO:teuthology.orchestra.run.smithi101.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-07-28T23:35:55.721 INFO:teuthology.orchestra.run.smithi101.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-07-28T23:35:55.722 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:35:55.932 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-mds-2:18.1.3-11.g0484097b.el9.x86_64 30/31 2023-07-28T23:35:55.954 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-mds-2:18.1.3-11.g0484097b.el9.x86_64 30/31 2023-07-28T23:35:55.955 INFO:teuthology.orchestra.run.smithi101.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-07-28T23:35:55.955 INFO:teuthology.orchestra.run.smithi101.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-07-28T23:35:55.955 INFO:teuthology.orchestra.run.smithi101.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-07-28T23:35:55.955 INFO:teuthology.orchestra.run.smithi101.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-07-28T23:35:55.955 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:35:56.018 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-2:18.1.3-11.g0484097b.el9.x86_64 31/31 2023-07-28T23:35:56.730 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-07-28T23:35:57.759 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-07-28T23:35:57.802 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-diskprediction-local-2:18.1.3-11.g0484097 12/12 2023-07-28T23:35:57.872 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-2:18.1.3-11.g0484097b.el9.x86_64 31/31 2023-07-28T23:35:57.873 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-2:18.1.3-11.g0484097b.el9.x86_64 1/31 2023-07-28T23:35:57.873 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-mds-2:18.1.3-11.g0484097b.el9.x86_64 2/31 2023-07-28T23:35:57.873 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-mgr-2:18.1.3-11.g0484097b.el9.x86_64 3/31 2023-07-28T23:35:57.873 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-mon-2:18.1.3-11.g0484097b.el9.x86_64 4/31 2023-07-28T23:35:57.873 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-osd-2:18.1.3-11.g0484097b.el9.x86_64 5/31 2023-07-28T23:35:57.873 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-mgr-modules-core-2:18.1.3-11.g0484097b.el9.no 6/31 2023-07-28T23:35:57.873 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/31 2023-07-28T23:35:57.874 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/31 2023-07-28T23:35:57.874 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/31 2023-07-28T23:35:57.874 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/31 2023-07-28T23:35:57.874 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/31 2023-07-28T23:35:57.874 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/31 2023-07-28T23:35:57.874 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/31 2023-07-28T23:35:57.874 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/31 2023-07-28T23:35:57.874 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 15/31 2023-07-28T23:35:57.875 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 16/31 2023-07-28T23:35:57.875 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 17/31 2023-07-28T23:35:57.875 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 18/31 2023-07-28T23:35:57.875 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 19/31 2023-07-28T23:35:57.875 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 20/31 2023-07-28T23:35:57.875 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 21/31 2023-07-28T23:35:57.875 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 22/31 2023-07-28T23:35:57.875 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 23/31 2023-07-28T23:35:57.876 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 24/31 2023-07-28T23:35:57.876 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 25/31 2023-07-28T23:35:57.876 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 26/31 2023-07-28T23:35:57.877 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/31 2023-07-28T23:35:57.877 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 28/31 2023-07-28T23:35:57.877 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 29/31 2023-07-28T23:35:57.877 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 30/31 2023-07-28T23:35:58.945 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 31/31 2023-07-28T23:35:58.945 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:35:58.945 INFO:teuthology.orchestra.run.smithi101.stdout:Installed: 2023-07-28T23:35:58.946 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:35:58.946 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mds-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:35:58.946 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:35:58.946 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-modules-core-2:18.1.3-11.g0484097b.el9.noarch 2023-07-28T23:35:58.946 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mon-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:35:58.946 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-osd-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:35:58.946 INFO:teuthology.orchestra.run.smithi101.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-07-28T23:35:58.947 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-07-28T23:35:58.947 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-07-28T23:35:58.947 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-07-28T23:35:58.947 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-07-28T23:35:58.947 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-07-28T23:35:58.947 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-07-28T23:35:58.947 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-07-28T23:35:58.948 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-07-28T23:35:58.948 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-07-28T23:35:58.948 INFO:teuthology.orchestra.run.smithi101.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-07-28T23:35:58.948 INFO:teuthology.orchestra.run.smithi101.stdout: python3-mako-1.1.4-6.el9.noarch 2023-07-28T23:35:58.948 INFO:teuthology.orchestra.run.smithi101.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-07-28T23:35:58.948 INFO:teuthology.orchestra.run.smithi101.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-07-28T23:35:58.948 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-07-28T23:35:58.948 INFO:teuthology.orchestra.run.smithi101.stdout: python3-ply-3.11-14.el9.noarch 2023-07-28T23:35:58.949 INFO:teuthology.orchestra.run.smithi101.stdout: python3-portend-3.1.0-2.el9.noarch 2023-07-28T23:35:58.949 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-07-28T23:35:58.949 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pycparser-2.20-6.el9.noarch 2023-07-28T23:35:58.949 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pytz-2021.1-5.el9.noarch 2023-07-28T23:35:58.949 INFO:teuthology.orchestra.run.smithi101.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-07-28T23:35:58.949 INFO:teuthology.orchestra.run.smithi101.stdout: python3-toml-0.10.2-6.el9.noarch 2023-07-28T23:35:58.949 INFO:teuthology.orchestra.run.smithi101.stdout: python3-webob-1.8.7-6.el9.noarch 2023-07-28T23:35:58.950 INFO:teuthology.orchestra.run.smithi101.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-07-28T23:35:58.950 INFO:teuthology.orchestra.run.smithi101.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-07-28T23:35:58.950 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:35:58.950 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-07-28T23:35:59.353 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install ceph-base 2023-07-28T23:35:59.887 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:02:24 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:35:59.967 INFO:teuthology.orchestra.run.smithi101.stdout:Package ceph-base-2:18.1.3-11.g0484097b.el9.x86_64 is already installed. 2023-07-28T23:36:00.016 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-07-28T23:36:00.017 INFO:teuthology.orchestra.run.smithi101.stdout:Nothing to do. 2023-07-28T23:36:00.017 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-07-28T23:36:00.076 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install cephadm 2023-07-28T23:36:00.420 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.1.3-11.g0484097 12/12 2023-07-28T23:36:00.421 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.1.3-11.g0484097 1/12 2023-07-28T23:36:00.421 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-07-28T23:36:00.421 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-07-28T23:36:00.421 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-07-28T23:36:00.421 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-07-28T23:36:00.421 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-07-28T23:36:00.421 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-07-28T23:36:00.422 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-07-28T23:36:00.422 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-07-28T23:36:00.422 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libgfortran-11.4.1-2.1.el9.x86_64 10/12 2023-07-28T23:36:00.422 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libquadmath-11.4.1-2.1.el9.x86_64 11/12 2023-07-28T23:36:00.602 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:02:25 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:36:00.730 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-07-28T23:36:00.731 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-07-28T23:36:00.731 INFO:teuthology.orchestra.run.smithi101.stdout: Package Arch Version Repository Size 2023-07-28T23:36:00.731 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-07-28T23:36:00.731 INFO:teuthology.orchestra.run.smithi101.stdout:Installing: 2023-07-28T23:36:00.731 INFO:teuthology.orchestra.run.smithi101.stdout: cephadm noarch 2:18.1.3-11.g0484097b.el9 ceph-noarch 209 k 2023-07-28T23:36:00.731 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:36:00.732 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2023-07-28T23:36:00.732 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-07-28T23:36:00.732 INFO:teuthology.orchestra.run.smithi101.stdout:Install 1 Package 2023-07-28T23:36:00.732 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:36:00.732 INFO:teuthology.orchestra.run.smithi101.stdout:Total download size: 209 k 2023-07-28T23:36:00.732 INFO:teuthology.orchestra.run.smithi101.stdout:Installed size: 200 k 2023-07-28T23:36:00.732 INFO:teuthology.orchestra.run.smithi101.stdout:Downloading Packages: 2023-07-28T23:36:00.916 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-devel-3.9.16-1.el9.x86_64 12/12 2023-07-28T23:36:00.916 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:36:00.916 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2023-07-28T23:36:00.917 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-diskprediction-local-2:18.1.3-11.g0484097b.el9.noarch 2023-07-28T23:36:00.917 INFO:teuthology.orchestra.run.smithi139.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-07-28T23:36:00.917 INFO:teuthology.orchestra.run.smithi139.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-07-28T23:36:00.917 INFO:teuthology.orchestra.run.smithi139.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-07-28T23:36:00.918 INFO:teuthology.orchestra.run.smithi139.stdout: libgfortran-11.4.1-2.1.el9.x86_64 2023-07-28T23:36:00.918 INFO:teuthology.orchestra.run.smithi139.stdout: libquadmath-11.4.1-2.1.el9.x86_64 2023-07-28T23:36:00.918 INFO:teuthology.orchestra.run.smithi139.stdout: openblas-0.3.21-2.el9.x86_64 2023-07-28T23:36:00.918 INFO:teuthology.orchestra.run.smithi139.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-07-28T23:36:00.919 INFO:teuthology.orchestra.run.smithi139.stdout: python3-devel-3.9.16-1.el9.x86_64 2023-07-28T23:36:00.919 INFO:teuthology.orchestra.run.smithi139.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-07-28T23:36:00.919 INFO:teuthology.orchestra.run.smithi139.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-07-28T23:36:00.919 INFO:teuthology.orchestra.run.smithi139.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-07-28T23:36:00.920 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:36:00.920 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2023-07-28T23:36:01.321 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-mgr-rook 2023-07-28T23:36:01.722 INFO:teuthology.orchestra.run.smithi101.stdout:cephadm-18.1.3-11.g0484097b.el9.noarch.rpm 211 kB/s | 209 kB 00:00 2023-07-28T23:36:01.723 INFO:teuthology.orchestra.run.smithi101.stdout:-------------------------------------------------------------------------------- 2023-07-28T23:36:01.723 INFO:teuthology.orchestra.run.smithi101.stdout:Total 210 kB/s | 209 kB 00:00 2023-07-28T23:36:01.724 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction check 2023-07-28T23:36:01.729 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction check succeeded. 2023-07-28T23:36:01.729 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction test 2023-07-28T23:36:01.732 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction test succeeded. 2023-07-28T23:36:01.732 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction 2023-07-28T23:36:01.800 INFO:teuthology.orchestra.run.smithi101.stdout: Preparing : 1/1 2023-07-28T23:36:01.870 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:02:26 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:36:01.999 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2023-07-28T23:36:02.000 INFO:teuthology.orchestra.run.smithi139.stdout:====================================================================================== 2023-07-28T23:36:02.001 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2023-07-28T23:36:02.001 INFO:teuthology.orchestra.run.smithi139.stdout:====================================================================================== 2023-07-28T23:36:02.001 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2023-07-28T23:36:02.001 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-rook noarch 2:18.1.3-11.g0484097b.el9 ceph-noarch 47 k 2023-07-28T23:36:02.001 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2023-07-28T23:36:02.001 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-07-28T23:36:02.002 INFO:teuthology.orchestra.run.smithi139.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-07-28T23:36:02.002 INFO:teuthology.orchestra.run.smithi139.stdout: python3-google-auth noarch 1:2.22.0-1.el9 epel 197 k 2023-07-28T23:36:02.002 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-07-28T23:36:02.002 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-07-28T23:36:02.002 INFO:teuthology.orchestra.run.smithi139.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-07-28T23:36:02.002 INFO:teuthology.orchestra.run.smithi139.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-07-28T23:36:02.002 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-07-28T23:36:02.003 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-07-28T23:36:02.003 INFO:teuthology.orchestra.run.smithi139.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-07-28T23:36:02.003 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-07-28T23:36:02.003 INFO:teuthology.orchestra.run.smithi139.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-07-28T23:36:02.003 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:36:02.003 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2023-07-28T23:36:02.003 INFO:teuthology.orchestra.run.smithi139.stdout:====================================================================================== 2023-07-28T23:36:02.004 INFO:teuthology.orchestra.run.smithi139.stdout:Install 13 Packages 2023-07-28T23:36:02.004 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:36:02.004 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 2.0 M 2023-07-28T23:36:02.005 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 26 M 2023-07-28T23:36:02.005 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2023-07-28T23:36:02.242 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: cephadm-2:18.1.3-11.g0484097b.el9.noarch 1/1 2023-07-28T23:36:02.343 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : cephadm-2:18.1.3-11.g0484097b.el9.noarch 1/1 2023-07-28T23:36:02.570 INFO:teuthology.orchestra.run.smithi139.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 87 kB/s | 24 kB 00:00 2023-07-28T23:36:02.595 INFO:teuthology.orchestra.run.smithi139.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 59 kB/s | 18 kB 00:00 2023-07-28T23:36:02.722 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: cephadm-2:18.1.3-11.g0484097b.el9.noarch 1/1 2023-07-28T23:36:02.845 INFO:teuthology.orchestra.run.smithi139.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 635 kB/s | 174 kB 00:00 2023-07-28T23:36:02.904 INFO:teuthology.orchestra.run.smithi139.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 428 kB/s | 132 kB 00:00 2023-07-28T23:36:02.979 INFO:teuthology.orchestra.run.smithi139.stdout:(5/13): ceph-mgr-rook-18.1.3-11.g0484097b.el9.n 69 kB/s | 47 kB 00:00 2023-07-28T23:36:03.005 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : cephadm-2:18.1.3-11.g0484097b.el9.noarch 1/1 2023-07-28T23:36:03.006 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:36:03.006 INFO:teuthology.orchestra.run.smithi101.stdout:Installed: 2023-07-28T23:36:03.006 INFO:teuthology.orchestra.run.smithi101.stdout: cephadm-2:18.1.3-11.g0484097b.el9.noarch 2023-07-28T23:36:03.006 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:36:03.006 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-07-28T23:36:03.013 INFO:teuthology.orchestra.run.smithi139.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.2 MB/s | 211 kB 00:00 2023-07-28T23:36:03.046 INFO:teuthology.orchestra.run.smithi139.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 301 kB/s | 43 kB 00:00 2023-07-28T23:36:03.222 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install ceph-immutable-object-cache 2023-07-28T23:36:03.438 INFO:teuthology.orchestra.run.smithi139.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 35 kB/s | 15 kB 00:00 2023-07-28T23:36:03.472 INFO:teuthology.orchestra.run.smithi139.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 65 kB/s | 32 kB 00:00 2023-07-28T23:36:03.766 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:02:28 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:36:03.772 INFO:teuthology.orchestra.run.smithi139.stdout:(10/13): python3-google-auth-2.22.0-1.el9.noarc 271 kB/s | 197 kB 00:00 2023-07-28T23:36:03.898 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-07-28T23:36:03.898 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-07-28T23:36:03.899 INFO:teuthology.orchestra.run.smithi101.stdout: Package Arch Version Repo Size 2023-07-28T23:36:03.899 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-07-28T23:36:03.899 INFO:teuthology.orchestra.run.smithi101.stdout:Installing: 2023-07-28T23:36:03.899 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-immutable-object-cache x86_64 2:18.1.3-11.g0484097b.el9 ceph 142 k 2023-07-28T23:36:03.899 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:36:03.899 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2023-07-28T23:36:03.899 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-07-28T23:36:03.900 INFO:teuthology.orchestra.run.smithi101.stdout:Install 1 Package 2023-07-28T23:36:03.900 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:36:03.900 INFO:teuthology.orchestra.run.smithi101.stdout:Total download size: 142 k 2023-07-28T23:36:03.900 INFO:teuthology.orchestra.run.smithi101.stdout:Installed size: 400 k 2023-07-28T23:36:03.900 INFO:teuthology.orchestra.run.smithi101.stdout:Downloading Packages: 2023-07-28T23:36:04.231 INFO:teuthology.orchestra.run.smithi139.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 78 kB/s | 59 kB 00:00 2023-07-28T23:36:04.423 INFO:teuthology.orchestra.run.smithi139.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 138 kB/s | 90 kB 00:00 2023-07-28T23:36:04.719 INFO:teuthology.orchestra.run.smithi101.stdout:ceph-immutable-object-cache-18.1.3-11.g0484097b 174 kB/s | 142 kB 00:00 2023-07-28T23:36:04.720 INFO:teuthology.orchestra.run.smithi101.stdout:-------------------------------------------------------------------------------- 2023-07-28T23:36:04.720 INFO:teuthology.orchestra.run.smithi101.stdout:Total 173 kB/s | 142 kB 00:00 2023-07-28T23:36:04.720 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction check 2023-07-28T23:36:04.728 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction check succeeded. 2023-07-28T23:36:04.729 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction test 2023-07-28T23:36:04.785 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction test succeeded. 2023-07-28T23:36:04.786 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction 2023-07-28T23:36:04.856 INFO:teuthology.orchestra.run.smithi139.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 737 kB/s | 1.0 MB 00:01 2023-07-28T23:36:04.861 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2023-07-28T23:36:04.862 INFO:teuthology.orchestra.run.smithi139.stdout:Total 730 kB/s | 2.0 MB 00:02 2023-07-28T23:36:04.862 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2023-07-28T23:36:04.877 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2023-07-28T23:36:04.877 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2023-07-28T23:36:04.928 INFO:teuthology.orchestra.run.smithi101.stdout: Preparing : 1/1 2023-07-28T23:36:04.970 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2023-07-28T23:36:04.970 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2023-07-28T23:36:05.012 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-immutable-object-cache-2:18.1.3-11.g0484097b.el 1/1 2023-07-28T23:36:05.036 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.1.3-11.g0484097b.el 1/1 2023-07-28T23:36:05.036 INFO:teuthology.orchestra.run.smithi101.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-07-28T23:36:05.036 INFO:teuthology.orchestra.run.smithi101.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-07-28T23:36:05.037 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:36:05.485 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2023-07-28T23:36:05.652 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-07-28T23:36:05.715 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-07-28T23:36:05.779 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-07-28T23:36:05.816 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-07-28T23:36:05.866 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-07-28T23:36:05.967 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-07-28T23:36:06.035 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-immutable-object-cache-2:18.1.3-11.g0484097b.el 1/1 2023-07-28T23:36:06.036 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:36:06.036 INFO:teuthology.orchestra.run.smithi101.stdout:Installed: 2023-07-28T23:36:06.036 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-immutable-object-cache-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:36:06.036 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:36:06.036 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-07-28T23:36:06.085 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-google-auth-1:2.22.0-1.el9.noarch 7/13 2023-07-28T23:36:06.143 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-07-28T23:36:06.295 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install ceph-mgr 2023-07-28T23:36:06.831 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:02:31 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:36:06.867 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-07-28T23:36:06.911 INFO:teuthology.orchestra.run.smithi101.stdout:Package ceph-mgr-2:18.1.3-11.g0484097b.el9.x86_64 is already installed. 2023-07-28T23:36:06.931 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-07-28T23:36:06.960 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-07-28T23:36:06.961 INFO:teuthology.orchestra.run.smithi101.stdout:Nothing to do. 2023-07-28T23:36:06.961 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-07-28T23:36:06.973 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-07-28T23:36:07.044 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-07-28T23:36:07.056 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install ceph-mgr-dashboard 2023-07-28T23:36:07.077 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-rook-2:18.1.3-11.g0484097b.el9.noarch 13/13 2023-07-28T23:36:07.587 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:02:32 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:36:07.718 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-07-28T23:36:07.719 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-07-28T23:36:07.719 INFO:teuthology.orchestra.run.smithi101.stdout: Package Arch Version Repository Size 2023-07-28T23:36:07.719 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-07-28T23:36:07.719 INFO:teuthology.orchestra.run.smithi101.stdout:Installing: 2023-07-28T23:36:07.719 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-dashboard noarch 2:18.1.3-11.g0484097b.el9 ceph-noarch 1.7 M 2023-07-28T23:36:07.720 INFO:teuthology.orchestra.run.smithi101.stdout:Installing dependencies: 2023-07-28T23:36:07.720 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-grafana-dashboards noarch 2:18.1.3-11.g0484097b.el9 ceph-noarch 23 k 2023-07-28T23:36:07.720 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-prometheus-alerts noarch 2:18.1.3-11.g0484097b.el9 ceph-noarch 15 k 2023-07-28T23:36:07.720 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-07-28T23:36:07.720 INFO:teuthology.orchestra.run.smithi101.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-07-28T23:36:07.720 INFO:teuthology.orchestra.run.smithi101.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-07-28T23:36:07.720 INFO:teuthology.orchestra.run.smithi101.stdout:Installing weak dependencies: 2023-07-28T23:36:07.721 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-07-28T23:36:07.721 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:36:07.721 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2023-07-28T23:36:07.721 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-07-28T23:36:07.721 INFO:teuthology.orchestra.run.smithi101.stdout:Install 7 Packages 2023-07-28T23:36:07.721 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:36:07.722 INFO:teuthology.orchestra.run.smithi101.stdout:Total download size: 2.0 M 2023-07-28T23:36:07.722 INFO:teuthology.orchestra.run.smithi101.stdout:Installed size: 8.1 M 2023-07-28T23:36:07.722 INFO:teuthology.orchestra.run.smithi101.stdout:Downloading Packages: 2023-07-28T23:36:07.892 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-rook-2:18.1.3-11.g0484097b.el9.noarch 13/13 2023-07-28T23:36:07.892 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-rook-2:18.1.3-11.g0484097b.el9.noarch 1/13 2023-07-28T23:36:07.893 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-07-28T23:36:07.893 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-07-28T23:36:07.893 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-07-28T23:36:07.893 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-07-28T23:36:07.893 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-07-28T23:36:07.893 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-07-28T23:36:07.893 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-07-28T23:36:07.893 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-07-28T23:36:07.894 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-google-auth-1:2.22.0-1.el9.noarch 10/13 2023-07-28T23:36:07.894 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-07-28T23:36:07.894 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-07-28T23:36:08.248 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-07-28T23:36:08.248 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:36:08.249 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2023-07-28T23:36:08.249 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-rook-2:18.1.3-11.g0484097b.el9.noarch 2023-07-28T23:36:08.249 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-07-28T23:36:08.249 INFO:teuthology.orchestra.run.smithi139.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-07-28T23:36:08.249 INFO:teuthology.orchestra.run.smithi139.stdout: python3-google-auth-1:2.22.0-1.el9.noarch 2023-07-28T23:36:08.249 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-07-28T23:36:08.249 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-07-28T23:36:08.250 INFO:teuthology.orchestra.run.smithi139.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-07-28T23:36:08.250 INFO:teuthology.orchestra.run.smithi139.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-07-28T23:36:08.250 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-07-28T23:36:08.250 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-07-28T23:36:08.250 INFO:teuthology.orchestra.run.smithi139.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-07-28T23:36:08.250 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rsa-4.9-2.el9.noarch 2023-07-28T23:36:08.250 INFO:teuthology.orchestra.run.smithi139.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-07-28T23:36:08.250 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:36:08.251 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2023-07-28T23:36:08.519 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-mgr-cephadm 2023-07-28T23:36:08.873 INFO:teuthology.orchestra.run.smithi101.stdout:(1/7): ceph-grafana-dashboards-18.1.3-11.g04840 39 kB/s | 23 kB 00:00 2023-07-28T23:36:09.061 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:02:34 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:36:09.204 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2023-07-28T23:36:09.204 INFO:teuthology.orchestra.run.smithi139.stdout:====================================================================================== 2023-07-28T23:36:09.205 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2023-07-28T23:36:09.205 INFO:teuthology.orchestra.run.smithi139.stdout:====================================================================================== 2023-07-28T23:36:09.205 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2023-07-28T23:36:09.205 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-cephadm noarch 2:18.1.3-11.g0484097b.el9 ceph-noarch 127 k 2023-07-28T23:36:09.205 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2023-07-28T23:36:09.205 INFO:teuthology.orchestra.run.smithi139.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-07-28T23:36:09.206 INFO:teuthology.orchestra.run.smithi139.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-07-28T23:36:09.206 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-07-28T23:36:09.206 INFO:teuthology.orchestra.run.smithi139.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-07-28T23:36:09.206 INFO:teuthology.orchestra.run.smithi139.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-07-28T23:36:09.206 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:36:09.206 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2023-07-28T23:36:09.206 INFO:teuthology.orchestra.run.smithi139.stdout:====================================================================================== 2023-07-28T23:36:09.207 INFO:teuthology.orchestra.run.smithi139.stdout:Install 6 Packages 2023-07-28T23:36:09.207 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:36:09.207 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 6.8 M 2023-07-28T23:36:09.207 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 33 M 2023-07-28T23:36:09.208 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2023-07-28T23:36:09.349 INFO:teuthology.orchestra.run.smithi101.stdout:(2/7): ceph-prometheus-alerts-18.1.3-11.g048409 14 kB/s | 15 kB 00:01 2023-07-28T23:36:09.541 INFO:teuthology.orchestra.run.smithi101.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 14 kB/s | 9.0 kB 00:00 2023-07-28T23:36:09.782 INFO:teuthology.orchestra.run.smithi101.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 127 kB/s | 31 kB 00:00 2023-07-28T23:36:09.808 INFO:teuthology.orchestra.run.smithi101.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 89 kB/s | 41 kB 00:00 2023-07-28T23:36:10.623 INFO:teuthology.orchestra.run.smithi139.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 442 kB/s | 228 kB 00:00 2023-07-28T23:36:10.816 INFO:teuthology.orchestra.run.smithi139.stdout:(2/6): ceph-mgr-cephadm-18.1.3-11.g0484097b.el9 179 kB/s | 127 kB 00:00 2023-07-28T23:36:10.900 INFO:teuthology.orchestra.run.smithi101.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 169 kB/s | 188 kB 00:01 2023-07-28T23:36:10.975 INFO:teuthology.orchestra.run.smithi139.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 6.7 MB/s | 5.8 MB 00:00 2023-07-28T23:36:11.317 INFO:teuthology.orchestra.run.smithi139.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 116 kB/s | 58 kB 00:00 2023-07-28T23:36:11.384 INFO:teuthology.orchestra.run.smithi139.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 125 kB/s | 51 kB 00:00 2023-07-28T23:36:11.459 INFO:teuthology.orchestra.run.smithi139.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 656 kB/s | 548 kB 00:00 2023-07-28T23:36:11.461 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2023-07-28T23:36:11.461 INFO:teuthology.orchestra.run.smithi139.stdout:Total 3.0 MB/s | 6.8 MB 00:02 2023-07-28T23:36:11.462 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2023-07-28T23:36:11.472 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2023-07-28T23:36:11.472 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2023-07-28T23:36:11.561 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2023-07-28T23:36:11.567 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2023-07-28T23:36:11.851 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2023-07-28T23:36:12.031 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-07-28T23:36:12.094 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-07-28T23:36:12.629 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-07-28T23:36:12.721 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-07-28T23:36:12.788 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-07-28T23:36:12.823 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-cephadm-2:18.1.3-11.g0484097b.el9.noarch 6/6 2023-07-28T23:36:13.654 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.1.3-11.g0484097b.el9.noarch 6/6 2023-07-28T23:36:13.654 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-cephadm-2:18.1.3-11.g0484097b.el9.noarch 1/6 2023-07-28T23:36:13.655 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-07-28T23:36:13.655 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-07-28T23:36:13.655 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-07-28T23:36:13.655 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-07-28T23:36:13.970 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-07-28T23:36:13.971 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:36:13.971 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2023-07-28T23:36:13.971 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-cephadm-2:18.1.3-11.g0484097b.el9.noarch 2023-07-28T23:36:13.971 INFO:teuthology.orchestra.run.smithi139.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-07-28T23:36:13.971 INFO:teuthology.orchestra.run.smithi139.stdout: python3-babel-2.9.1-2.el9.noarch 2023-07-28T23:36:13.971 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-07-28T23:36:13.972 INFO:teuthology.orchestra.run.smithi139.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-07-28T23:36:13.972 INFO:teuthology.orchestra.run.smithi139.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-07-28T23:36:13.972 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:36:13.972 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2023-07-28T23:36:14.234 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-fuse 2023-07-28T23:36:14.773 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:02:39 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:36:14.902 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2023-07-28T23:36:14.903 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-07-28T23:36:14.903 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2023-07-28T23:36:14.904 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-07-28T23:36:14.904 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2023-07-28T23:36:14.904 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-fuse x86_64 2:18.1.3-11.g0484097b.el9 ceph 835 k 2023-07-28T23:36:14.904 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2023-07-28T23:36:14.904 INFO:teuthology.orchestra.run.smithi139.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-07-28T23:36:14.904 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:36:14.904 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2023-07-28T23:36:14.905 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-07-28T23:36:14.905 INFO:teuthology.orchestra.run.smithi139.stdout:Install 2 Packages 2023-07-28T23:36:14.905 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:36:14.905 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 914 k 2023-07-28T23:36:14.905 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 2.7 M 2023-07-28T23:36:14.905 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2023-07-28T23:36:15.403 INFO:teuthology.orchestra.run.smithi139.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 158 kB/s | 79 kB 00:00 2023-07-28T23:36:15.616 INFO:teuthology.orchestra.run.smithi101.stdout:(7/7): ceph-mgr-dashboard-18.1.3-11.g0484097b.e 237 kB/s | 1.7 MB 00:07 2023-07-28T23:36:15.618 INFO:teuthology.orchestra.run.smithi101.stdout:-------------------------------------------------------------------------------- 2023-07-28T23:36:15.619 INFO:teuthology.orchestra.run.smithi101.stdout:Total 259 kB/s | 2.0 MB 00:07 2023-07-28T23:36:15.619 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction check 2023-07-28T23:36:15.628 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction check succeeded. 2023-07-28T23:36:15.628 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction test 2023-07-28T23:36:15.687 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction test succeeded. 2023-07-28T23:36:15.688 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction 2023-07-28T23:36:16.032 INFO:teuthology.orchestra.run.smithi101.stdout: Preparing : 1/1 2023-07-28T23:36:16.084 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-07-28T23:36:16.118 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-07-28T23:36:16.164 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-07-28T23:36:16.196 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-07-28T23:36:16.225 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-prometheus-alerts-2:18.1.3-11.g0484097b.el9.noa 5/7 2023-07-28T23:36:16.370 INFO:teuthology.orchestra.run.smithi139.stdout:(2/2): ceph-fuse-18.1.3-11.g0484097b.el9.x86_64 570 kB/s | 835 kB 00:01 2023-07-28T23:36:16.371 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2023-07-28T23:36:16.372 INFO:teuthology.orchestra.run.smithi139.stdout:Total 622 kB/s | 914 kB 00:01 2023-07-28T23:36:16.372 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2023-07-28T23:36:16.396 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2023-07-28T23:36:16.396 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2023-07-28T23:36:16.455 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-grafana-dashboards-2:18.1.3-11.g0484097b.el9.no 6/7 2023-07-28T23:36:16.474 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2023-07-28T23:36:16.475 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2023-07-28T23:36:16.486 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-mgr-dashboard-2:18.1.3-11.g0484097b.el9.noarch 7/7 2023-07-28T23:36:16.664 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2023-07-28T23:36:16.803 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-07-28T23:36:16.876 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-fuse-2:18.1.3-11.g0484097b.el9.x86_64 2/2 2023-07-28T23:36:17.037 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.1.3-11.g0484097b.el9.noarch 7/7 2023-07-28T23:36:17.037 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-grafana-dashboards-2:18.1.3-11.g0484097b.el9.no 1/7 2023-07-28T23:36:17.037 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-mgr-dashboard-2:18.1.3-11.g0484097b.el9.noarch 2/7 2023-07-28T23:36:17.037 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-prometheus-alerts-2:18.1.3-11.g0484097b.el9.noa 3/7 2023-07-28T23:36:17.037 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-07-28T23:36:17.038 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-07-28T23:36:17.038 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-07-28T23:36:17.334 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-07-28T23:36:17.334 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:36:17.334 INFO:teuthology.orchestra.run.smithi101.stdout:Installed: 2023-07-28T23:36:17.334 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-grafana-dashboards-2:18.1.3-11.g0484097b.el9.noarch 2023-07-28T23:36:17.335 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-dashboard-2:18.1.3-11.g0484097b.el9.noarch 2023-07-28T23:36:17.335 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-prometheus-alerts-2:18.1.3-11.g0484097b.el9.noarch 2023-07-28T23:36:17.335 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-07-28T23:36:17.335 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-07-28T23:36:17.335 INFO:teuthology.orchestra.run.smithi101.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-07-28T23:36:17.335 INFO:teuthology.orchestra.run.smithi101.stdout: python3-routes-2.5.1-5.el9.noarch 2023-07-28T23:36:17.335 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:36:17.336 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-07-28T23:36:17.540 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install ceph-mgr-diskprediction-local 2023-07-28T23:36:17.601 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-fuse-2:18.1.3-11.g0484097b.el9.x86_64 2/2 2023-07-28T23:36:17.601 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-fuse-2:18.1.3-11.g0484097b.el9.x86_64 1/2 2023-07-28T23:36:17.919 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-07-28T23:36:17.920 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:36:17.920 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2023-07-28T23:36:17.920 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-fuse-2:18.1.3-11.g0484097b.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-07-28T23:36:17.920 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:36:17.920 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2023-07-28T23:36:18.074 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:02:43 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:36:18.202 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-07-28T23:36:18.204 INFO:teuthology.orchestra.run.smithi101.stdout:========================================================================================== 2023-07-28T23:36:18.204 INFO:teuthology.orchestra.run.smithi101.stdout: Package Arch Version Repository Size 2023-07-28T23:36:18.204 INFO:teuthology.orchestra.run.smithi101.stdout:========================================================================================== 2023-07-28T23:36:18.204 INFO:teuthology.orchestra.run.smithi101.stdout:Installing: 2023-07-28T23:36:18.204 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-diskprediction-local noarch 2:18.1.3-11.g0484097b.el9 ceph-noarch 7.4 M 2023-07-28T23:36:18.204 INFO:teuthology.orchestra.run.smithi101.stdout:Installing dependencies: 2023-07-28T23:36:18.204 INFO:teuthology.orchestra.run.smithi101.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-07-28T23:36:18.205 INFO:teuthology.orchestra.run.smithi101.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-07-28T23:36:18.205 INFO:teuthology.orchestra.run.smithi101.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-07-28T23:36:18.205 INFO:teuthology.orchestra.run.smithi101.stdout: libgfortran x86_64 11.4.1-2.1.el9 CentOS-BaseOS 807 k 2023-07-28T23:36:18.205 INFO:teuthology.orchestra.run.smithi101.stdout: libquadmath x86_64 11.4.1-2.1.el9 CentOS-BaseOS 197 k 2023-07-28T23:36:18.205 INFO:teuthology.orchestra.run.smithi101.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-07-28T23:36:18.205 INFO:teuthology.orchestra.run.smithi101.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-07-28T23:36:18.205 INFO:teuthology.orchestra.run.smithi101.stdout: python3-devel x86_64 3.9.16-1.el9 appstream 247 k 2023-07-28T23:36:18.206 INFO:teuthology.orchestra.run.smithi101.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-07-28T23:36:18.206 INFO:teuthology.orchestra.run.smithi101.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-07-28T23:36:18.206 INFO:teuthology.orchestra.run.smithi101.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-07-28T23:36:18.206 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:36:18.206 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2023-07-28T23:36:18.206 INFO:teuthology.orchestra.run.smithi101.stdout:========================================================================================== 2023-07-28T23:36:18.206 INFO:teuthology.orchestra.run.smithi101.stdout:Install 12 Packages 2023-07-28T23:36:18.206 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:36:18.207 INFO:teuthology.orchestra.run.smithi101.stdout:Total download size: 37 M 2023-07-28T23:36:18.207 INFO:teuthology.orchestra.run.smithi101.stdout:Installed size: 209 M 2023-07-28T23:36:18.207 INFO:teuthology.orchestra.run.smithi101.stdout:Downloading Packages: 2023-07-28T23:36:18.209 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-volume 2023-07-28T23:36:18.753 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:02:43 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:36:18.886 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2023-07-28T23:36:18.887 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-07-28T23:36:18.887 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2023-07-28T23:36:18.887 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-07-28T23:36:18.887 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2023-07-28T23:36:18.887 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-volume noarch 2:18.1.3-11.g0484097b.el9 ceph-noarch 259 k 2023-07-28T23:36:18.887 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:36:18.888 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2023-07-28T23:36:18.888 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-07-28T23:36:18.888 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2023-07-28T23:36:18.888 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:36:18.888 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 259 k 2023-07-28T23:36:18.889 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 1.2 M 2023-07-28T23:36:18.889 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2023-07-28T23:36:19.047 INFO:teuthology.orchestra.run.smithi101.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 71 kB/s | 32 kB 00:00 2023-07-28T23:36:19.173 INFO:teuthology.orchestra.run.smithi101.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 137 kB/s | 17 kB 00:00 2023-07-28T23:36:19.314 INFO:teuthology.orchestra.run.smithi101.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 241 kB/s | 34 kB 00:00 2023-07-28T23:36:19.715 INFO:teuthology.orchestra.run.smithi101.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.7 MB/s | 3.0 MB 00:01 2023-07-28T23:36:20.048 INFO:teuthology.orchestra.run.smithi101.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 6.5 MB/s | 4.7 MB 00:00 2023-07-28T23:36:20.190 INFO:teuthology.orchestra.run.smithi101.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 10 MB/s | 4.8 MB 00:00 2023-07-28T23:36:20.215 INFO:teuthology.orchestra.run.smithi101.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.3 MB/s | 217 kB 00:00 2023-07-28T23:36:20.441 INFO:teuthology.orchestra.run.smithi101.stdout:(8/12): libgfortran-11.4.1-2.1.el9.x86_64.rpm 3.5 MB/s | 807 kB 00:00 2023-07-28T23:36:20.591 INFO:teuthology.orchestra.run.smithi139.stdout:ceph-volume-18.1.3-11.g0484097b.el9.noarch.rpm 152 kB/s | 259 kB 00:01 2023-07-28T23:36:20.592 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2023-07-28T23:36:20.592 INFO:teuthology.orchestra.run.smithi139.stdout:Total 152 kB/s | 259 kB 00:01 2023-07-28T23:36:20.592 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2023-07-28T23:36:20.601 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2023-07-28T23:36:20.602 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2023-07-28T23:36:20.636 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2023-07-28T23:36:20.637 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2023-07-28T23:36:20.844 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2023-07-28T23:36:20.925 INFO:teuthology.orchestra.run.smithi101.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 21 MB/s | 16 MB 00:00 2023-07-28T23:36:20.935 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-volume-2:18.1.3-11.g0484097b.el9.noarch 1/1 2023-07-28T23:36:20.949 INFO:teuthology.orchestra.run.smithi101.stdout:(10/12): libquadmath-11.4.1-2.1.el9.x86_64.rpm 389 kB/s | 197 kB 00:00 2023-07-28T23:36:20.950 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-volume-2:18.1.3-11.g0484097b.el9.noarch 1/1 2023-07-28T23:36:20.950 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-07-28T23:36:20.950 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-07-28T23:36:20.950 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:36:22.007 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-volume-2:18.1.3-11.g0484097b.el9.noarch 1/1 2023-07-28T23:36:22.008 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:36:22.008 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2023-07-28T23:36:22.008 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-volume-2:18.1.3-11.g0484097b.el9.noarch 2023-07-28T23:36:22.008 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:36:22.008 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2023-07-28T23:36:22.083 INFO:teuthology.orchestra.run.smithi101.stdout:(11/12): python3-devel-3.9.16-1.el9.x86_64.rpm 213 kB/s | 247 kB 00:01 2023-07-28T23:36:22.304 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install librados-devel 2023-07-28T23:36:22.852 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:02:47 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:36:22.983 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2023-07-28T23:36:22.984 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-07-28T23:36:22.984 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repo Size 2023-07-28T23:36:22.984 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-07-28T23:36:22.984 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2023-07-28T23:36:22.985 INFO:teuthology.orchestra.run.smithi139.stdout: librados-devel x86_64 2:18.1.3-11.g0484097b.el9 ceph 126 k 2023-07-28T23:36:22.985 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:36:22.985 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2023-07-28T23:36:22.985 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-07-28T23:36:22.985 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2023-07-28T23:36:22.985 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:36:22.985 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 126 k 2023-07-28T23:36:22.985 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 456 k 2023-07-28T23:36:22.986 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2023-07-28T23:36:24.289 INFO:teuthology.orchestra.run.smithi139.stdout:librados-devel-18.1.3-11.g0484097b.el9.x86_64.r 97 kB/s | 126 kB 00:01 2023-07-28T23:36:24.290 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2023-07-28T23:36:24.290 INFO:teuthology.orchestra.run.smithi139.stdout:Total 97 kB/s | 126 kB 00:01 2023-07-28T23:36:24.291 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2023-07-28T23:36:24.297 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2023-07-28T23:36:24.297 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2023-07-28T23:36:24.360 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2023-07-28T23:36:24.360 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2023-07-28T23:36:24.515 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2023-07-28T23:36:24.644 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librados-devel-2:18.1.3-11.g0484097b.el9.x86_64 1/1 2023-07-28T23:36:25.082 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librados-devel-2:18.1.3-11.g0484097b.el9.x86_64 1/1 2023-07-28T23:36:25.382 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librados-devel-2:18.1.3-11.g0484097b.el9.x86_64 1/1 2023-07-28T23:36:25.382 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:36:25.382 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2023-07-28T23:36:25.382 INFO:teuthology.orchestra.run.smithi139.stdout: librados-devel-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:36:25.383 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:36:25.383 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2023-07-28T23:36:25.610 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install libcephfs2 2023-07-28T23:36:26.154 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:02:51 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:36:26.235 INFO:teuthology.orchestra.run.smithi139.stdout:Package libcephfs2-2:18.1.3-11.g0484097b.el9.x86_64 is already installed. 2023-07-28T23:36:26.284 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2023-07-28T23:36:26.285 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2023-07-28T23:36:26.286 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2023-07-28T23:36:26.368 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install libcephfs-devel 2023-07-28T23:36:26.904 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:02:51 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:36:27.032 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2023-07-28T23:36:27.033 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-07-28T23:36:27.033 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repo Size 2023-07-28T23:36:27.033 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-07-28T23:36:27.034 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2023-07-28T23:36:27.034 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs-devel x86_64 2:18.1.3-11.g0484097b.el9 ceph 31 k 2023-07-28T23:36:27.034 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:36:27.034 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2023-07-28T23:36:27.034 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-07-28T23:36:27.034 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2023-07-28T23:36:27.034 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:36:27.035 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 31 k 2023-07-28T23:36:27.035 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 137 k 2023-07-28T23:36:27.035 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2023-07-28T23:36:27.902 INFO:teuthology.orchestra.run.smithi139.stdout:libcephfs-devel-18.1.3-11.g0484097b.el9.x86_64. 35 kB/s | 31 kB 00:00 2023-07-28T23:36:27.902 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2023-07-28T23:36:27.903 INFO:teuthology.orchestra.run.smithi139.stdout:Total 35 kB/s | 31 kB 00:00 2023-07-28T23:36:27.903 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2023-07-28T23:36:27.907 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2023-07-28T23:36:27.907 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2023-07-28T23:36:27.928 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2023-07-28T23:36:27.928 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2023-07-28T23:36:28.009 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2023-07-28T23:36:28.131 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libcephfs-devel-2:18.1.3-11.g0484097b.el9.x86_64 1/1 2023-07-28T23:36:28.554 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libcephfs-devel-2:18.1.3-11.g0484097b.el9.x86_64 1/1 2023-07-28T23:36:28.852 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libcephfs-devel-2:18.1.3-11.g0484097b.el9.x86_64 1/1 2023-07-28T23:36:28.853 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:36:28.853 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2023-07-28T23:36:28.853 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs-devel-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:36:28.853 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:36:28.853 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2023-07-28T23:36:29.068 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install librados2 2023-07-28T23:36:29.607 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:02:54 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:36:29.687 INFO:teuthology.orchestra.run.smithi139.stdout:Package librados2-2:18.1.3-11.g0484097b.el9.x86_64 is already installed. 2023-07-28T23:36:29.735 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2023-07-28T23:36:29.736 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2023-07-28T23:36:29.736 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2023-07-28T23:36:29.815 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install librbd1 2023-07-28T23:36:30.353 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:02:55 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:36:30.433 INFO:teuthology.orchestra.run.smithi139.stdout:Package librbd1-2:18.1.3-11.g0484097b.el9.x86_64 is already installed. 2023-07-28T23:36:30.482 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2023-07-28T23:36:30.483 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2023-07-28T23:36:30.483 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2023-07-28T23:36:30.554 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install python3-rados 2023-07-28T23:36:31.089 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:02:56 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:36:31.169 INFO:teuthology.orchestra.run.smithi139.stdout:Package python3-rados-2:18.1.3-11.g0484097b.el9.x86_64 is already installed. 2023-07-28T23:36:31.217 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2023-07-28T23:36:31.218 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2023-07-28T23:36:31.218 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2023-07-28T23:36:31.297 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install python3-rgw 2023-07-28T23:36:31.831 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:02:56 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:36:31.911 INFO:teuthology.orchestra.run.smithi139.stdout:Package python3-rgw-2:18.1.3-11.g0484097b.el9.x86_64 is already installed. 2023-07-28T23:36:31.959 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2023-07-28T23:36:31.960 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2023-07-28T23:36:31.960 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2023-07-28T23:36:32.033 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install python3-cephfs 2023-07-28T23:36:32.570 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:02:57 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:36:32.649 INFO:teuthology.orchestra.run.smithi139.stdout:Package python3-cephfs-2:18.1.3-11.g0484097b.el9.x86_64 is already installed. 2023-07-28T23:36:32.698 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2023-07-28T23:36:32.699 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2023-07-28T23:36:32.699 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2023-07-28T23:36:32.770 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install python3-rbd 2023-07-28T23:36:33.304 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:02:58 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:36:33.385 INFO:teuthology.orchestra.run.smithi139.stdout:Package python3-rbd-2:18.1.3-11.g0484097b.el9.x86_64 is already installed. 2023-07-28T23:36:33.434 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2023-07-28T23:36:33.435 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2023-07-28T23:36:33.435 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2023-07-28T23:36:33.504 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install rbd-fuse 2023-07-28T23:36:33.616 INFO:teuthology.orchestra.run.smithi101.stdout:(12/12): ceph-mgr-diskprediction-local-18.1.3-1 506 kB/s | 7.4 MB 00:15 2023-07-28T23:36:33.620 INFO:teuthology.orchestra.run.smithi101.stdout:-------------------------------------------------------------------------------- 2023-07-28T23:36:33.620 INFO:teuthology.orchestra.run.smithi101.stdout:Total 2.4 MB/s | 37 MB 00:15 2023-07-28T23:36:33.641 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction check 2023-07-28T23:36:33.668 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction check succeeded. 2023-07-28T23:36:33.669 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction test 2023-07-28T23:36:34.033 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:02:59 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:36:34.076 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction test succeeded. 2023-07-28T23:36:34.076 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction 2023-07-28T23:36:34.161 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2023-07-28T23:36:34.162 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-07-28T23:36:34.162 INFO:teuthology.orchestra.run.smithi139.stdout: Package Architecture Version Repository Size 2023-07-28T23:36:34.162 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-07-28T23:36:34.162 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2023-07-28T23:36:34.162 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-fuse x86_64 2:18.1.3-11.g0484097b.el9 ceph 86 k 2023-07-28T23:36:34.162 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:36:34.162 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2023-07-28T23:36:34.163 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-07-28T23:36:34.163 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2023-07-28T23:36:34.163 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:36:34.163 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 86 k 2023-07-28T23:36:34.163 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 231 k 2023-07-28T23:36:34.163 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2023-07-28T23:36:34.776 INFO:teuthology.orchestra.run.smithi101.stdout: Preparing : 1/1 2023-07-28T23:36:34.885 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libquadmath-11.4.1-2.1.el9.x86_64 1/12 2023-07-28T23:36:34.928 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libgfortran-11.4.1-2.1.el9.x86_64 2/12 2023-07-28T23:36:35.020 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-07-28T23:36:35.056 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-devel-3.9.16-1.el9.x86_64 4/12 2023-07-28T23:36:35.234 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-07-28T23:36:35.245 INFO:teuthology.orchestra.run.smithi139.stdout:rbd-fuse-18.1.3-11.g0484097b.el9.x86_64.rpm 79 kB/s | 86 kB 00:01 2023-07-28T23:36:35.246 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2023-07-28T23:36:35.246 INFO:teuthology.orchestra.run.smithi139.stdout:Total 79 kB/s | 86 kB 00:01 2023-07-28T23:36:35.247 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2023-07-28T23:36:35.255 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2023-07-28T23:36:35.255 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2023-07-28T23:36:35.272 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-07-28T23:36:35.317 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2023-07-28T23:36:35.318 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2023-07-28T23:36:35.349 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-07-28T23:36:35.454 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2023-07-28T23:36:35.549 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : rbd-fuse-2:18.1.3-11.g0484097b.el9.x86_64 1/1 2023-07-28T23:36:35.896 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: rbd-fuse-2:18.1.3-11.g0484097b.el9.x86_64 1/1 2023-07-28T23:36:35.995 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-07-28T23:36:36.118 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-07-28T23:36:36.232 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : rbd-fuse-2:18.1.3-11.g0484097b.el9.x86_64 1/1 2023-07-28T23:36:36.232 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:36:36.232 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2023-07-28T23:36:36.232 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-fuse-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:36:36.232 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:36:36.233 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2023-07-28T23:36:36.469 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install rbd-mirror 2023-07-28T23:36:37.014 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:03:02 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:36:37.145 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2023-07-28T23:36:37.146 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-07-28T23:36:37.146 INFO:teuthology.orchestra.run.smithi139.stdout: Package Architecture Version Repository Size 2023-07-28T23:36:37.146 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-07-28T23:36:37.146 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2023-07-28T23:36:37.146 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-mirror x86_64 2:18.1.3-11.g0484097b.el9 ceph 3.0 M 2023-07-28T23:36:37.146 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:36:37.146 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2023-07-28T23:36:37.147 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-07-28T23:36:37.147 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2023-07-28T23:36:37.147 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:36:37.147 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 3.0 M 2023-07-28T23:36:37.148 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 12 M 2023-07-28T23:36:37.148 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2023-07-28T23:36:37.362 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-07-28T23:36:38.402 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-07-28T23:36:38.438 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-mgr-diskprediction-local-2:18.1.3-11.g0484097 12/12 2023-07-28T23:36:41.142 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.1.3-11.g0484097 12/12 2023-07-28T23:36:41.142 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.1.3-11.g0484097 1/12 2023-07-28T23:36:41.142 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-07-28T23:36:41.142 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-07-28T23:36:41.142 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-07-28T23:36:41.143 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-07-28T23:36:41.143 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-07-28T23:36:41.143 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-07-28T23:36:41.143 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-07-28T23:36:41.143 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-07-28T23:36:41.143 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libgfortran-11.4.1-2.1.el9.x86_64 10/12 2023-07-28T23:36:41.143 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libquadmath-11.4.1-2.1.el9.x86_64 11/12 2023-07-28T23:36:41.631 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-devel-3.9.16-1.el9.x86_64 12/12 2023-07-28T23:36:41.631 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:36:41.631 INFO:teuthology.orchestra.run.smithi101.stdout:Installed: 2023-07-28T23:36:41.632 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-diskprediction-local-2:18.1.3-11.g0484097b.el9.noarch 2023-07-28T23:36:41.632 INFO:teuthology.orchestra.run.smithi101.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-07-28T23:36:41.632 INFO:teuthology.orchestra.run.smithi101.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-07-28T23:36:41.632 INFO:teuthology.orchestra.run.smithi101.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-07-28T23:36:41.632 INFO:teuthology.orchestra.run.smithi101.stdout: libgfortran-11.4.1-2.1.el9.x86_64 2023-07-28T23:36:41.632 INFO:teuthology.orchestra.run.smithi101.stdout: libquadmath-11.4.1-2.1.el9.x86_64 2023-07-28T23:36:41.633 INFO:teuthology.orchestra.run.smithi101.stdout: openblas-0.3.21-2.el9.x86_64 2023-07-28T23:36:41.633 INFO:teuthology.orchestra.run.smithi101.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-07-28T23:36:41.633 INFO:teuthology.orchestra.run.smithi101.stdout: python3-devel-3.9.16-1.el9.x86_64 2023-07-28T23:36:41.633 INFO:teuthology.orchestra.run.smithi101.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-07-28T23:36:41.633 INFO:teuthology.orchestra.run.smithi101.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-07-28T23:36:41.633 INFO:teuthology.orchestra.run.smithi101.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-07-28T23:36:41.633 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:36:41.634 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-07-28T23:36:42.050 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install ceph-mgr-rook 2023-07-28T23:36:42.597 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:03:07 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:36:42.665 INFO:teuthology.orchestra.run.smithi139.stdout:rbd-mirror-18.1.3-11.g0484097b.el9.x86_64.rpm 554 kB/s | 3.0 MB 00:05 2023-07-28T23:36:42.666 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2023-07-28T23:36:42.667 INFO:teuthology.orchestra.run.smithi139.stdout:Total 553 kB/s | 3.0 MB 00:05 2023-07-28T23:36:42.667 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2023-07-28T23:36:42.678 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2023-07-28T23:36:42.678 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2023-07-28T23:36:42.728 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-07-28T23:36:42.729 INFO:teuthology.orchestra.run.smithi101.stdout:====================================================================================== 2023-07-28T23:36:42.729 INFO:teuthology.orchestra.run.smithi101.stdout: Package Arch Version Repository Size 2023-07-28T23:36:42.730 INFO:teuthology.orchestra.run.smithi101.stdout:====================================================================================== 2023-07-28T23:36:42.730 INFO:teuthology.orchestra.run.smithi101.stdout:Installing: 2023-07-28T23:36:42.730 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-rook noarch 2:18.1.3-11.g0484097b.el9 ceph-noarch 47 k 2023-07-28T23:36:42.730 INFO:teuthology.orchestra.run.smithi101.stdout:Installing dependencies: 2023-07-28T23:36:42.730 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-07-28T23:36:42.730 INFO:teuthology.orchestra.run.smithi101.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-07-28T23:36:42.731 INFO:teuthology.orchestra.run.smithi101.stdout: python3-google-auth noarch 1:2.22.0-1.el9 epel 197 k 2023-07-28T23:36:42.731 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-07-28T23:36:42.731 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-07-28T23:36:42.731 INFO:teuthology.orchestra.run.smithi101.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-07-28T23:36:42.731 INFO:teuthology.orchestra.run.smithi101.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-07-28T23:36:42.731 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-07-28T23:36:42.731 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-07-28T23:36:42.731 INFO:teuthology.orchestra.run.smithi101.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-07-28T23:36:42.732 INFO:teuthology.orchestra.run.smithi101.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-07-28T23:36:42.732 INFO:teuthology.orchestra.run.smithi101.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-07-28T23:36:42.732 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:36:42.732 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2023-07-28T23:36:42.732 INFO:teuthology.orchestra.run.smithi101.stdout:====================================================================================== 2023-07-28T23:36:42.732 INFO:teuthology.orchestra.run.smithi101.stdout:Install 13 Packages 2023-07-28T23:36:42.732 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:36:42.733 INFO:teuthology.orchestra.run.smithi101.stdout:Total download size: 2.0 M 2023-07-28T23:36:42.733 INFO:teuthology.orchestra.run.smithi101.stdout:Installed size: 26 M 2023-07-28T23:36:42.733 INFO:teuthology.orchestra.run.smithi101.stdout:Downloading Packages: 2023-07-28T23:36:42.763 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2023-07-28T23:36:42.763 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2023-07-28T23:36:43.178 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2023-07-28T23:36:43.258 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : rbd-mirror-2:18.1.3-11.g0484097b.el9.x86_64 1/1 2023-07-28T23:36:43.283 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: rbd-mirror-2:18.1.3-11.g0484097b.el9.x86_64 1/1 2023-07-28T23:36:43.283 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-07-28T23:36:43.283 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-07-28T23:36:43.283 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-07-28T23:36:43.283 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-07-28T23:36:43.284 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:36:43.297 INFO:teuthology.orchestra.run.smithi101.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 50 kB/s | 24 kB 00:00 2023-07-28T23:36:43.314 INFO:teuthology.orchestra.run.smithi101.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 35 kB/s | 18 kB 00:00 2023-07-28T23:36:43.565 INFO:teuthology.orchestra.run.smithi101.stdout:(3/13): ceph-mgr-rook-18.1.3-11.g0484097b.el9.n 63 kB/s | 47 kB 00:00 2023-07-28T23:36:43.607 INFO:teuthology.orchestra.run.smithi101.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 565 kB/s | 174 kB 00:00 2023-07-28T23:36:43.674 INFO:teuthology.orchestra.run.smithi101.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 368 kB/s | 132 kB 00:00 2023-07-28T23:36:43.749 INFO:teuthology.orchestra.run.smithi101.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.1 MB/s | 211 kB 00:00 2023-07-28T23:36:43.774 INFO:teuthology.orchestra.run.smithi101.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 256 kB/s | 43 kB 00:00 2023-07-28T23:36:44.108 INFO:teuthology.orchestra.run.smithi101.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 74 kB/s | 32 kB 00:00 2023-07-28T23:36:44.133 INFO:teuthology.orchestra.run.smithi101.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 39 kB/s | 15 kB 00:00 2023-07-28T23:36:44.467 INFO:teuthology.orchestra.run.smithi101.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 177 kB/s | 59 kB 00:00 2023-07-28T23:36:44.468 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : rbd-mirror-2:18.1.3-11.g0484097b.el9.x86_64 1/1 2023-07-28T23:36:44.468 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:36:44.468 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2023-07-28T23:36:44.468 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-mirror-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:36:44.468 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:36:44.469 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2023-07-28T23:36:44.517 INFO:teuthology.orchestra.run.smithi101.stdout:(11/13): python3-google-auth-2.22.0-1.el9.noarc 265 kB/s | 197 kB 00:00 2023-07-28T23:36:44.742 INFO:teuthology.orchestra.run.smithi101.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 326 kB/s | 90 kB 00:00 2023-07-28T23:36:44.743 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install rbd-nbd 2023-07-28T23:36:44.859 INFO:teuthology.orchestra.run.smithi101.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.4 MB/s | 1.0 MB 00:00 2023-07-28T23:36:44.861 INFO:teuthology.orchestra.run.smithi101.stdout:-------------------------------------------------------------------------------- 2023-07-28T23:36:44.862 INFO:teuthology.orchestra.run.smithi101.stdout:Total 980 kB/s | 2.0 MB 00:02 2023-07-28T23:36:44.862 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction check 2023-07-28T23:36:44.877 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction check succeeded. 2023-07-28T23:36:44.877 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction test 2023-07-28T23:36:44.967 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction test succeeded. 2023-07-28T23:36:44.968 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction 2023-07-28T23:36:45.299 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:03:10 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:36:45.427 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2023-07-28T23:36:45.428 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-07-28T23:36:45.428 INFO:teuthology.orchestra.run.smithi139.stdout: Package Architecture Version Repository Size 2023-07-28T23:36:45.428 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-07-28T23:36:45.429 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2023-07-28T23:36:45.429 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-nbd x86_64 2:18.1.3-11.g0484097b.el9 ceph 170 k 2023-07-28T23:36:45.429 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:36:45.429 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2023-07-28T23:36:45.429 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-07-28T23:36:45.429 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2023-07-28T23:36:45.429 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:36:45.430 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 170 k 2023-07-28T23:36:45.430 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 490 k 2023-07-28T23:36:45.430 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2023-07-28T23:36:45.500 INFO:teuthology.orchestra.run.smithi101.stdout: Preparing : 1/1 2023-07-28T23:36:45.682 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-07-28T23:36:45.752 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-07-28T23:36:45.824 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-07-28T23:36:45.870 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-07-28T23:36:45.919 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-07-28T23:36:46.020 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-07-28T23:36:46.138 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-google-auth-1:2.22.0-1.el9.noarch 7/13 2023-07-28T23:36:46.195 INFO:teuthology.orchestra.run.smithi139.stdout:rbd-nbd-18.1.3-11.g0484097b.el9.x86_64.rpm 221 kB/s | 170 kB 00:00 2023-07-28T23:36:46.196 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-07-28T23:36:46.196 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2023-07-28T23:36:46.197 INFO:teuthology.orchestra.run.smithi139.stdout:Total 221 kB/s | 170 kB 00:00 2023-07-28T23:36:46.197 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2023-07-28T23:36:46.204 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2023-07-28T23:36:46.205 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2023-07-28T23:36:46.264 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2023-07-28T23:36:46.265 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2023-07-28T23:36:46.432 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2023-07-28T23:36:46.533 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : rbd-nbd-2:18.1.3-11.g0484097b.el9.x86_64 1/1 2023-07-28T23:36:46.937 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-07-28T23:36:46.957 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: rbd-nbd-2:18.1.3-11.g0484097b.el9.x86_64 1/1 2023-07-28T23:36:47.001 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-07-28T23:36:47.051 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-07-28T23:36:47.106 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-07-28T23:36:47.138 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-mgr-rook-2:18.1.3-11.g0484097b.el9.noarch 13/13 2023-07-28T23:36:47.270 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : rbd-nbd-2:18.1.3-11.g0484097b.el9.x86_64 1/1 2023-07-28T23:36:47.270 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:36:47.270 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2023-07-28T23:36:47.270 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-nbd-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:36:47.271 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:36:47.271 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2023-07-28T23:36:47.537 DEBUG:teuthology.parallel:result is None 2023-07-28T23:36:47.971 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-mgr-rook-2:18.1.3-11.g0484097b.el9.noarch 13/13 2023-07-28T23:36:47.971 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-mgr-rook-2:18.1.3-11.g0484097b.el9.noarch 1/13 2023-07-28T23:36:47.972 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-07-28T23:36:47.972 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-07-28T23:36:47.972 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-07-28T23:36:47.972 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-07-28T23:36:47.972 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-07-28T23:36:47.972 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-07-28T23:36:47.972 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-07-28T23:36:47.972 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-07-28T23:36:47.973 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-google-auth-1:2.22.0-1.el9.noarch 10/13 2023-07-28T23:36:47.973 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-07-28T23:36:47.973 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-07-28T23:36:48.332 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-07-28T23:36:48.333 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:36:48.333 INFO:teuthology.orchestra.run.smithi101.stdout:Installed: 2023-07-28T23:36:48.333 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-rook-2:18.1.3-11.g0484097b.el9.noarch 2023-07-28T23:36:48.333 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-07-28T23:36:48.333 INFO:teuthology.orchestra.run.smithi101.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-07-28T23:36:48.333 INFO:teuthology.orchestra.run.smithi101.stdout: python3-google-auth-1:2.22.0-1.el9.noarch 2023-07-28T23:36:48.333 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-07-28T23:36:48.334 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-07-28T23:36:48.334 INFO:teuthology.orchestra.run.smithi101.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-07-28T23:36:48.334 INFO:teuthology.orchestra.run.smithi101.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-07-28T23:36:48.334 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-07-28T23:36:48.334 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-07-28T23:36:48.334 INFO:teuthology.orchestra.run.smithi101.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-07-28T23:36:48.334 INFO:teuthology.orchestra.run.smithi101.stdout: python3-rsa-4.9-2.el9.noarch 2023-07-28T23:36:48.335 INFO:teuthology.orchestra.run.smithi101.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-07-28T23:36:48.335 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:36:48.335 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-07-28T23:36:48.607 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install ceph-mgr-cephadm 2023-07-28T23:36:49.141 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:03:14 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:36:49.269 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-07-28T23:36:49.270 INFO:teuthology.orchestra.run.smithi101.stdout:====================================================================================== 2023-07-28T23:36:49.270 INFO:teuthology.orchestra.run.smithi101.stdout: Package Arch Version Repository Size 2023-07-28T23:36:49.270 INFO:teuthology.orchestra.run.smithi101.stdout:====================================================================================== 2023-07-28T23:36:49.270 INFO:teuthology.orchestra.run.smithi101.stdout:Installing: 2023-07-28T23:36:49.271 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-cephadm noarch 2:18.1.3-11.g0484097b.el9 ceph-noarch 127 k 2023-07-28T23:36:49.271 INFO:teuthology.orchestra.run.smithi101.stdout:Installing dependencies: 2023-07-28T23:36:49.271 INFO:teuthology.orchestra.run.smithi101.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-07-28T23:36:49.271 INFO:teuthology.orchestra.run.smithi101.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-07-28T23:36:49.271 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-07-28T23:36:49.271 INFO:teuthology.orchestra.run.smithi101.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-07-28T23:36:49.272 INFO:teuthology.orchestra.run.smithi101.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-07-28T23:36:49.272 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:36:49.272 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2023-07-28T23:36:49.272 INFO:teuthology.orchestra.run.smithi101.stdout:====================================================================================== 2023-07-28T23:36:49.272 INFO:teuthology.orchestra.run.smithi101.stdout:Install 6 Packages 2023-07-28T23:36:49.272 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:36:49.273 INFO:teuthology.orchestra.run.smithi101.stdout:Total download size: 6.8 M 2023-07-28T23:36:49.273 INFO:teuthology.orchestra.run.smithi101.stdout:Installed size: 33 M 2023-07-28T23:36:49.273 INFO:teuthology.orchestra.run.smithi101.stdout:Downloading Packages: 2023-07-28T23:36:49.978 INFO:teuthology.orchestra.run.smithi101.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 350 kB/s | 228 kB 00:00 2023-07-28T23:36:50.270 INFO:teuthology.orchestra.run.smithi101.stdout:(2/6): ceph-mgr-cephadm-18.1.3-11.g0484097b.el9 135 kB/s | 127 kB 00:00 2023-07-28T23:36:50.612 INFO:teuthology.orchestra.run.smithi101.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.5 MB/s | 5.8 MB 00:01 2023-07-28T23:36:51.154 INFO:teuthology.orchestra.run.smithi101.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 65 kB/s | 58 kB 00:00 2023-07-28T23:36:51.212 INFO:teuthology.orchestra.run.smithi101.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 85 kB/s | 51 kB 00:00 2023-07-28T23:36:51.429 INFO:teuthology.orchestra.run.smithi101.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 378 kB/s | 548 kB 00:01 2023-07-28T23:36:51.434 INFO:teuthology.orchestra.run.smithi101.stdout:-------------------------------------------------------------------------------- 2023-07-28T23:36:51.434 INFO:teuthology.orchestra.run.smithi101.stdout:Total 3.1 MB/s | 6.8 MB 00:02 2023-07-28T23:36:51.434 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction check 2023-07-28T23:36:51.444 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction check succeeded. 2023-07-28T23:36:51.444 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction test 2023-07-28T23:36:51.532 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction test succeeded. 2023-07-28T23:36:51.533 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction 2023-07-28T23:36:51.831 INFO:teuthology.orchestra.run.smithi101.stdout: Preparing : 1/1 2023-07-28T23:36:52.014 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-07-28T23:36:52.072 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-07-28T23:36:52.605 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-07-28T23:36:52.708 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-07-28T23:36:52.784 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-07-28T23:36:52.826 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-mgr-cephadm-2:18.1.3-11.g0484097b.el9.noarch 6/6 2023-07-28T23:36:53.702 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.1.3-11.g0484097b.el9.noarch 6/6 2023-07-28T23:36:53.702 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-mgr-cephadm-2:18.1.3-11.g0484097b.el9.noarch 1/6 2023-07-28T23:36:53.702 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-07-28T23:36:53.702 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-07-28T23:36:53.702 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-07-28T23:36:53.702 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-07-28T23:36:53.998 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-07-28T23:36:53.998 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:36:53.998 INFO:teuthology.orchestra.run.smithi101.stdout:Installed: 2023-07-28T23:36:53.998 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-cephadm-2:18.1.3-11.g0484097b.el9.noarch 2023-07-28T23:36:53.998 INFO:teuthology.orchestra.run.smithi101.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-07-28T23:36:53.998 INFO:teuthology.orchestra.run.smithi101.stdout: python3-babel-2.9.1-2.el9.noarch 2023-07-28T23:36:53.999 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-07-28T23:36:53.999 INFO:teuthology.orchestra.run.smithi101.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-07-28T23:36:53.999 INFO:teuthology.orchestra.run.smithi101.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-07-28T23:36:53.999 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:36:53.999 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-07-28T23:36:54.275 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install ceph-fuse 2023-07-28T23:36:54.808 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:03:19 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:36:54.936 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-07-28T23:36:54.937 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-07-28T23:36:54.937 INFO:teuthology.orchestra.run.smithi101.stdout: Package Arch Version Repository Size 2023-07-28T23:36:54.937 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-07-28T23:36:54.937 INFO:teuthology.orchestra.run.smithi101.stdout:Installing: 2023-07-28T23:36:54.938 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-fuse x86_64 2:18.1.3-11.g0484097b.el9 ceph 835 k 2023-07-28T23:36:54.938 INFO:teuthology.orchestra.run.smithi101.stdout:Installing dependencies: 2023-07-28T23:36:54.938 INFO:teuthology.orchestra.run.smithi101.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-07-28T23:36:54.938 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:36:54.938 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2023-07-28T23:36:54.938 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-07-28T23:36:54.938 INFO:teuthology.orchestra.run.smithi101.stdout:Install 2 Packages 2023-07-28T23:36:54.938 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:36:54.939 INFO:teuthology.orchestra.run.smithi101.stdout:Total download size: 914 k 2023-07-28T23:36:54.939 INFO:teuthology.orchestra.run.smithi101.stdout:Installed size: 2.7 M 2023-07-28T23:36:54.939 INFO:teuthology.orchestra.run.smithi101.stdout:Downloading Packages: 2023-07-28T23:36:55.340 INFO:teuthology.orchestra.run.smithi101.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 196 kB/s | 79 kB 00:00 2023-07-28T23:36:56.515 INFO:teuthology.orchestra.run.smithi101.stdout:(2/2): ceph-fuse-18.1.3-11.g0484097b.el9.x86_64 530 kB/s | 835 kB 00:01 2023-07-28T23:36:56.517 INFO:teuthology.orchestra.run.smithi101.stdout:-------------------------------------------------------------------------------- 2023-07-28T23:36:56.517 INFO:teuthology.orchestra.run.smithi101.stdout:Total 579 kB/s | 914 kB 00:01 2023-07-28T23:36:56.517 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction check 2023-07-28T23:36:56.540 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction check succeeded. 2023-07-28T23:36:56.540 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction test 2023-07-28T23:36:56.617 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction test succeeded. 2023-07-28T23:36:56.618 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction 2023-07-28T23:36:56.818 INFO:teuthology.orchestra.run.smithi101.stdout: Preparing : 1/1 2023-07-28T23:36:56.966 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-07-28T23:36:57.046 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-fuse-2:18.1.3-11.g0484097b.el9.x86_64 2/2 2023-07-28T23:36:57.751 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-fuse-2:18.1.3-11.g0484097b.el9.x86_64 2/2 2023-07-28T23:36:57.751 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-fuse-2:18.1.3-11.g0484097b.el9.x86_64 1/2 2023-07-28T23:36:58.041 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-07-28T23:36:58.042 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:36:58.042 INFO:teuthology.orchestra.run.smithi101.stdout:Installed: 2023-07-28T23:36:58.043 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-fuse-2:18.1.3-11.g0484097b.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-07-28T23:36:58.043 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:36:58.043 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-07-28T23:36:58.334 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install ceph-volume 2023-07-28T23:36:58.878 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:03:23 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:36:59.007 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-07-28T23:36:59.008 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-07-28T23:36:59.008 INFO:teuthology.orchestra.run.smithi101.stdout: Package Arch Version Repository Size 2023-07-28T23:36:59.008 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-07-28T23:36:59.008 INFO:teuthology.orchestra.run.smithi101.stdout:Installing: 2023-07-28T23:36:59.008 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-volume noarch 2:18.1.3-11.g0484097b.el9 ceph-noarch 259 k 2023-07-28T23:36:59.008 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:36:59.009 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2023-07-28T23:36:59.009 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-07-28T23:36:59.009 INFO:teuthology.orchestra.run.smithi101.stdout:Install 1 Package 2023-07-28T23:36:59.009 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:36:59.009 INFO:teuthology.orchestra.run.smithi101.stdout:Total download size: 259 k 2023-07-28T23:36:59.009 INFO:teuthology.orchestra.run.smithi101.stdout:Installed size: 1.2 M 2023-07-28T23:36:59.009 INFO:teuthology.orchestra.run.smithi101.stdout:Downloading Packages: 2023-07-28T23:37:00.180 INFO:teuthology.orchestra.run.smithi101.stdout:ceph-volume-18.1.3-11.g0484097b.el9.noarch.rpm 221 kB/s | 259 kB 00:01 2023-07-28T23:37:00.180 INFO:teuthology.orchestra.run.smithi101.stdout:-------------------------------------------------------------------------------- 2023-07-28T23:37:00.181 INFO:teuthology.orchestra.run.smithi101.stdout:Total 221 kB/s | 259 kB 00:01 2023-07-28T23:37:00.181 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction check 2023-07-28T23:37:00.191 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction check succeeded. 2023-07-28T23:37:00.191 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction test 2023-07-28T23:37:00.226 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction test succeeded. 2023-07-28T23:37:00.226 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction 2023-07-28T23:37:00.438 INFO:teuthology.orchestra.run.smithi101.stdout: Preparing : 1/1 2023-07-28T23:37:00.557 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-volume-2:18.1.3-11.g0484097b.el9.noarch 1/1 2023-07-28T23:37:00.572 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-volume-2:18.1.3-11.g0484097b.el9.noarch 1/1 2023-07-28T23:37:00.573 INFO:teuthology.orchestra.run.smithi101.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-07-28T23:37:00.573 INFO:teuthology.orchestra.run.smithi101.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-07-28T23:37:00.573 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:01.687 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-volume-2:18.1.3-11.g0484097b.el9.noarch 1/1 2023-07-28T23:37:01.688 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:01.688 INFO:teuthology.orchestra.run.smithi101.stdout:Installed: 2023-07-28T23:37:01.689 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-volume-2:18.1.3-11.g0484097b.el9.noarch 2023-07-28T23:37:01.689 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:01.689 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-07-28T23:37:01.992 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install librados-devel 2023-07-28T23:37:02.529 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:03:27 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:37:02.659 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-07-28T23:37:02.659 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-07-28T23:37:02.660 INFO:teuthology.orchestra.run.smithi101.stdout: Package Arch Version Repo Size 2023-07-28T23:37:02.660 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-07-28T23:37:02.660 INFO:teuthology.orchestra.run.smithi101.stdout:Installing: 2023-07-28T23:37:02.660 INFO:teuthology.orchestra.run.smithi101.stdout: librados-devel x86_64 2:18.1.3-11.g0484097b.el9 ceph 126 k 2023-07-28T23:37:02.660 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:02.660 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2023-07-28T23:37:02.661 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-07-28T23:37:02.661 INFO:teuthology.orchestra.run.smithi101.stdout:Install 1 Package 2023-07-28T23:37:02.661 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:02.661 INFO:teuthology.orchestra.run.smithi101.stdout:Total download size: 126 k 2023-07-28T23:37:02.661 INFO:teuthology.orchestra.run.smithi101.stdout:Installed size: 456 k 2023-07-28T23:37:02.661 INFO:teuthology.orchestra.run.smithi101.stdout:Downloading Packages: 2023-07-28T23:37:03.447 INFO:teuthology.orchestra.run.smithi101.stdout:librados-devel-18.1.3-11.g0484097b.el9.x86_64.r 161 kB/s | 126 kB 00:00 2023-07-28T23:37:03.448 INFO:teuthology.orchestra.run.smithi101.stdout:-------------------------------------------------------------------------------- 2023-07-28T23:37:03.449 INFO:teuthology.orchestra.run.smithi101.stdout:Total 160 kB/s | 126 kB 00:00 2023-07-28T23:37:03.449 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction check 2023-07-28T23:37:03.454 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction check succeeded. 2023-07-28T23:37:03.455 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction test 2023-07-28T23:37:03.512 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction test succeeded. 2023-07-28T23:37:03.513 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction 2023-07-28T23:37:03.679 INFO:teuthology.orchestra.run.smithi101.stdout: Preparing : 1/1 2023-07-28T23:37:03.819 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : librados-devel-2:18.1.3-11.g0484097b.el9.x86_64 1/1 2023-07-28T23:37:04.290 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: librados-devel-2:18.1.3-11.g0484097b.el9.x86_64 1/1 2023-07-28T23:37:04.602 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : librados-devel-2:18.1.3-11.g0484097b.el9.x86_64 1/1 2023-07-28T23:37:04.603 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:04.603 INFO:teuthology.orchestra.run.smithi101.stdout:Installed: 2023-07-28T23:37:04.603 INFO:teuthology.orchestra.run.smithi101.stdout: librados-devel-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:37:04.603 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:04.603 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-07-28T23:37:04.859 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install libcephfs2 2023-07-28T23:37:05.398 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:03:30 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:37:05.479 INFO:teuthology.orchestra.run.smithi101.stdout:Package libcephfs2-2:18.1.3-11.g0484097b.el9.x86_64 is already installed. 2023-07-28T23:37:05.529 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-07-28T23:37:05.530 INFO:teuthology.orchestra.run.smithi101.stdout:Nothing to do. 2023-07-28T23:37:05.530 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-07-28T23:37:05.624 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install libcephfs-devel 2023-07-28T23:37:06.159 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:03:31 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:37:06.287 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-07-28T23:37:06.288 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-07-28T23:37:06.288 INFO:teuthology.orchestra.run.smithi101.stdout: Package Arch Version Repo Size 2023-07-28T23:37:06.288 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-07-28T23:37:06.289 INFO:teuthology.orchestra.run.smithi101.stdout:Installing: 2023-07-28T23:37:06.289 INFO:teuthology.orchestra.run.smithi101.stdout: libcephfs-devel x86_64 2:18.1.3-11.g0484097b.el9 ceph 31 k 2023-07-28T23:37:06.289 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:06.289 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2023-07-28T23:37:06.289 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-07-28T23:37:06.289 INFO:teuthology.orchestra.run.smithi101.stdout:Install 1 Package 2023-07-28T23:37:06.289 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:06.290 INFO:teuthology.orchestra.run.smithi101.stdout:Total download size: 31 k 2023-07-28T23:37:06.290 INFO:teuthology.orchestra.run.smithi101.stdout:Installed size: 137 k 2023-07-28T23:37:06.290 INFO:teuthology.orchestra.run.smithi101.stdout:Downloading Packages: 2023-07-28T23:37:06.948 INFO:teuthology.orchestra.run.smithi101.stdout:libcephfs-devel-18.1.3-11.g0484097b.el9.x86_64. 46 kB/s | 31 kB 00:00 2023-07-28T23:37:06.949 INFO:teuthology.orchestra.run.smithi101.stdout:-------------------------------------------------------------------------------- 2023-07-28T23:37:06.949 INFO:teuthology.orchestra.run.smithi101.stdout:Total 46 kB/s | 31 kB 00:00 2023-07-28T23:37:06.949 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction check 2023-07-28T23:37:06.953 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction check succeeded. 2023-07-28T23:37:06.953 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction test 2023-07-28T23:37:06.972 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction test succeeded. 2023-07-28T23:37:06.972 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction 2023-07-28T23:37:07.051 INFO:teuthology.orchestra.run.smithi101.stdout: Preparing : 1/1 2023-07-28T23:37:07.170 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libcephfs-devel-2:18.1.3-11.g0484097b.el9.x86_64 1/1 2023-07-28T23:37:07.610 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: libcephfs-devel-2:18.1.3-11.g0484097b.el9.x86_64 1/1 2023-07-28T23:37:07.873 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libcephfs-devel-2:18.1.3-11.g0484097b.el9.x86_64 1/1 2023-07-28T23:37:07.873 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:07.873 INFO:teuthology.orchestra.run.smithi101.stdout:Installed: 2023-07-28T23:37:07.873 INFO:teuthology.orchestra.run.smithi101.stdout: libcephfs-devel-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:37:07.873 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:07.874 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-07-28T23:37:08.073 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install librados2 2023-07-28T23:37:08.605 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:03:33 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:37:08.684 INFO:teuthology.orchestra.run.smithi101.stdout:Package librados2-2:18.1.3-11.g0484097b.el9.x86_64 is already installed. 2023-07-28T23:37:08.732 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-07-28T23:37:08.733 INFO:teuthology.orchestra.run.smithi101.stdout:Nothing to do. 2023-07-28T23:37:08.733 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-07-28T23:37:08.790 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install librbd1 2023-07-28T23:37:09.325 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:03:34 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:37:09.404 INFO:teuthology.orchestra.run.smithi101.stdout:Package librbd1-2:18.1.3-11.g0484097b.el9.x86_64 is already installed. 2023-07-28T23:37:09.451 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-07-28T23:37:09.452 INFO:teuthology.orchestra.run.smithi101.stdout:Nothing to do. 2023-07-28T23:37:09.453 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-07-28T23:37:09.518 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install python3-rados 2023-07-28T23:37:10.055 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:03:35 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:37:10.133 INFO:teuthology.orchestra.run.smithi101.stdout:Package python3-rados-2:18.1.3-11.g0484097b.el9.x86_64 is already installed. 2023-07-28T23:37:10.181 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-07-28T23:37:10.182 INFO:teuthology.orchestra.run.smithi101.stdout:Nothing to do. 2023-07-28T23:37:10.182 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-07-28T23:37:10.252 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install python3-rgw 2023-07-28T23:37:10.788 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:03:35 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:37:10.868 INFO:teuthology.orchestra.run.smithi101.stdout:Package python3-rgw-2:18.1.3-11.g0484097b.el9.x86_64 is already installed. 2023-07-28T23:37:10.918 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-07-28T23:37:10.918 INFO:teuthology.orchestra.run.smithi101.stdout:Nothing to do. 2023-07-28T23:37:10.919 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-07-28T23:37:10.977 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install python3-cephfs 2023-07-28T23:37:11.503 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:03:36 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:37:11.582 INFO:teuthology.orchestra.run.smithi101.stdout:Package python3-cephfs-2:18.1.3-11.g0484097b.el9.x86_64 is already installed. 2023-07-28T23:37:11.631 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-07-28T23:37:11.632 INFO:teuthology.orchestra.run.smithi101.stdout:Nothing to do. 2023-07-28T23:37:11.632 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-07-28T23:37:11.701 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install python3-rbd 2023-07-28T23:37:12.233 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:03:37 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:37:12.313 INFO:teuthology.orchestra.run.smithi101.stdout:Package python3-rbd-2:18.1.3-11.g0484097b.el9.x86_64 is already installed. 2023-07-28T23:37:12.362 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-07-28T23:37:12.363 INFO:teuthology.orchestra.run.smithi101.stdout:Nothing to do. 2023-07-28T23:37:12.363 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-07-28T23:37:12.424 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install rbd-fuse 2023-07-28T23:37:12.953 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:03:37 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:37:13.080 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-07-28T23:37:13.081 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-07-28T23:37:13.081 INFO:teuthology.orchestra.run.smithi101.stdout: Package Architecture Version Repository Size 2023-07-28T23:37:13.082 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-07-28T23:37:13.082 INFO:teuthology.orchestra.run.smithi101.stdout:Installing: 2023-07-28T23:37:13.082 INFO:teuthology.orchestra.run.smithi101.stdout: rbd-fuse x86_64 2:18.1.3-11.g0484097b.el9 ceph 86 k 2023-07-28T23:37:13.082 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:13.082 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2023-07-28T23:37:13.082 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-07-28T23:37:13.082 INFO:teuthology.orchestra.run.smithi101.stdout:Install 1 Package 2023-07-28T23:37:13.082 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:13.083 INFO:teuthology.orchestra.run.smithi101.stdout:Total download size: 86 k 2023-07-28T23:37:13.083 INFO:teuthology.orchestra.run.smithi101.stdout:Installed size: 231 k 2023-07-28T23:37:13.083 INFO:teuthology.orchestra.run.smithi101.stdout:Downloading Packages: 2023-07-28T23:37:13.918 INFO:teuthology.orchestra.run.smithi101.stdout:rbd-fuse-18.1.3-11.g0484097b.el9.x86_64.rpm 103 kB/s | 86 kB 00:00 2023-07-28T23:37:13.919 INFO:teuthology.orchestra.run.smithi101.stdout:-------------------------------------------------------------------------------- 2023-07-28T23:37:13.919 INFO:teuthology.orchestra.run.smithi101.stdout:Total 103 kB/s | 86 kB 00:00 2023-07-28T23:37:13.920 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction check 2023-07-28T23:37:13.927 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction check succeeded. 2023-07-28T23:37:13.927 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction test 2023-07-28T23:37:13.985 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction test succeeded. 2023-07-28T23:37:13.986 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction 2023-07-28T23:37:14.131 INFO:teuthology.orchestra.run.smithi101.stdout: Preparing : 1/1 2023-07-28T23:37:14.248 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : rbd-fuse-2:18.1.3-11.g0484097b.el9.x86_64 1/1 2023-07-28T23:37:14.647 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: rbd-fuse-2:18.1.3-11.g0484097b.el9.x86_64 1/1 2023-07-28T23:37:14.944 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : rbd-fuse-2:18.1.3-11.g0484097b.el9.x86_64 1/1 2023-07-28T23:37:14.944 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:14.944 INFO:teuthology.orchestra.run.smithi101.stdout:Installed: 2023-07-28T23:37:14.944 INFO:teuthology.orchestra.run.smithi101.stdout: rbd-fuse-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:37:14.944 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:14.945 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-07-28T23:37:15.225 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install rbd-mirror 2023-07-28T23:37:15.767 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:03:40 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:37:15.897 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-07-28T23:37:15.898 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-07-28T23:37:15.898 INFO:teuthology.orchestra.run.smithi101.stdout: Package Architecture Version Repository Size 2023-07-28T23:37:15.898 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-07-28T23:37:15.898 INFO:teuthology.orchestra.run.smithi101.stdout:Installing: 2023-07-28T23:37:15.899 INFO:teuthology.orchestra.run.smithi101.stdout: rbd-mirror x86_64 2:18.1.3-11.g0484097b.el9 ceph 3.0 M 2023-07-28T23:37:15.899 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:15.899 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2023-07-28T23:37:15.899 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-07-28T23:37:15.899 INFO:teuthology.orchestra.run.smithi101.stdout:Install 1 Package 2023-07-28T23:37:15.899 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:15.900 INFO:teuthology.orchestra.run.smithi101.stdout:Total download size: 3.0 M 2023-07-28T23:37:15.900 INFO:teuthology.orchestra.run.smithi101.stdout:Installed size: 12 M 2023-07-28T23:37:15.900 INFO:teuthology.orchestra.run.smithi101.stdout:Downloading Packages: 2023-07-28T23:37:19.305 INFO:teuthology.orchestra.run.smithi101.stdout:rbd-mirror-18.1.3-11.g0484097b.el9.x86_64.rpm 897 kB/s | 3.0 MB 00:03 2023-07-28T23:37:19.306 INFO:teuthology.orchestra.run.smithi101.stdout:-------------------------------------------------------------------------------- 2023-07-28T23:37:19.307 INFO:teuthology.orchestra.run.smithi101.stdout:Total 897 kB/s | 3.0 MB 00:03 2023-07-28T23:37:19.307 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction check 2023-07-28T23:37:19.318 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction check succeeded. 2023-07-28T23:37:19.318 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction test 2023-07-28T23:37:19.402 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction test succeeded. 2023-07-28T23:37:19.402 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction 2023-07-28T23:37:19.824 INFO:teuthology.orchestra.run.smithi101.stdout: Preparing : 1/1 2023-07-28T23:37:19.881 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : rbd-mirror-2:18.1.3-11.g0484097b.el9.x86_64 1/1 2023-07-28T23:37:19.905 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: rbd-mirror-2:18.1.3-11.g0484097b.el9.x86_64 1/1 2023-07-28T23:37:19.905 INFO:teuthology.orchestra.run.smithi101.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-07-28T23:37:19.906 INFO:teuthology.orchestra.run.smithi101.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-07-28T23:37:19.906 INFO:teuthology.orchestra.run.smithi101.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-07-28T23:37:19.906 INFO:teuthology.orchestra.run.smithi101.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-07-28T23:37:19.906 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:21.163 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : rbd-mirror-2:18.1.3-11.g0484097b.el9.x86_64 1/1 2023-07-28T23:37:21.163 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:21.163 INFO:teuthology.orchestra.run.smithi101.stdout:Installed: 2023-07-28T23:37:21.163 INFO:teuthology.orchestra.run.smithi101.stdout: rbd-mirror-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:37:21.163 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:21.163 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-07-28T23:37:21.428 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install rbd-nbd 2023-07-28T23:37:21.980 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:03:46 ago on Fri 28 Jul 2023 11:33:35 PM UTC. 2023-07-28T23:37:22.114 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-07-28T23:37:22.114 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-07-28T23:37:22.115 INFO:teuthology.orchestra.run.smithi101.stdout: Package Architecture Version Repository Size 2023-07-28T23:37:22.115 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-07-28T23:37:22.115 INFO:teuthology.orchestra.run.smithi101.stdout:Installing: 2023-07-28T23:37:22.115 INFO:teuthology.orchestra.run.smithi101.stdout: rbd-nbd x86_64 2:18.1.3-11.g0484097b.el9 ceph 170 k 2023-07-28T23:37:22.115 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:22.115 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2023-07-28T23:37:22.115 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-07-28T23:37:22.116 INFO:teuthology.orchestra.run.smithi101.stdout:Install 1 Package 2023-07-28T23:37:22.116 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:22.116 INFO:teuthology.orchestra.run.smithi101.stdout:Total download size: 170 k 2023-07-28T23:37:22.116 INFO:teuthology.orchestra.run.smithi101.stdout:Installed size: 490 k 2023-07-28T23:37:22.116 INFO:teuthology.orchestra.run.smithi101.stdout:Downloading Packages: 2023-07-28T23:37:22.986 INFO:teuthology.orchestra.run.smithi101.stdout:rbd-nbd-18.1.3-11.g0484097b.el9.x86_64.rpm 195 kB/s | 170 kB 00:00 2023-07-28T23:37:22.986 INFO:teuthology.orchestra.run.smithi101.stdout:-------------------------------------------------------------------------------- 2023-07-28T23:37:22.987 INFO:teuthology.orchestra.run.smithi101.stdout:Total 194 kB/s | 170 kB 00:00 2023-07-28T23:37:22.987 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction check 2023-07-28T23:37:22.996 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction check succeeded. 2023-07-28T23:37:22.996 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction test 2023-07-28T23:37:23.056 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction test succeeded. 2023-07-28T23:37:23.057 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction 2023-07-28T23:37:23.271 INFO:teuthology.orchestra.run.smithi101.stdout: Preparing : 1/1 2023-07-28T23:37:23.391 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : rbd-nbd-2:18.1.3-11.g0484097b.el9.x86_64 1/1 2023-07-28T23:37:23.886 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: rbd-nbd-2:18.1.3-11.g0484097b.el9.x86_64 1/1 2023-07-28T23:37:24.168 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : rbd-nbd-2:18.1.3-11.g0484097b.el9.x86_64 1/1 2023-07-28T23:37:24.169 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:24.169 INFO:teuthology.orchestra.run.smithi101.stdout:Installed: 2023-07-28T23:37:24.169 INFO:teuthology.orchestra.run.smithi101.stdout: rbd-nbd-2:18.1.3-11.g0484097b.el9.x86_64 2023-07-28T23:37:24.169 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:24.169 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-07-28T23:37:24.400 DEBUG:teuthology.parallel:result is None 2023-07-28T23:37:24.401 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=0484097b0572e96f42cf6402cbe0ac8dcb046577 2023-07-28T23:37:30.106 DEBUG:teuthology.orchestra.run.smithi101:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-28T23:37:30.130 INFO:teuthology.orchestra.run.smithi101.stdout:18.1.3-11.g0484097b.el9 2023-07-28T23:37:30.130 INFO:teuthology.packaging:The installed version of ceph is 18.1.3-11.g0484097b.el9 2023-07-28T23:37:30.131 INFO:teuthology.task.install:The correct ceph version 18.1.3-11.g0484097b is installed. 2023-07-28T23:37:30.131 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=0484097b0572e96f42cf6402cbe0ac8dcb046577 2023-07-28T23:37:35.780 DEBUG:teuthology.orchestra.run.smithi139:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-28T23:37:35.804 INFO:teuthology.orchestra.run.smithi139.stdout:18.1.3-11.g0484097b.el9 2023-07-28T23:37:35.805 INFO:teuthology.packaging:The installed version of ceph is 18.1.3-11.g0484097b.el9 2023-07-28T23:37:35.805 INFO:teuthology.task.install:The correct ceph version 18.1.3-11.g0484097b is installed. 2023-07-28T23:37:35.805 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-07-28T23:37:35.834 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-07-28T23:37:35.834 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-28T23:37:35.867 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2023-07-28T23:37:35.867 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-28T23:37:35.899 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-07-28T23:37:35.899 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-07-28T23:37:35.899 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd of=/usr/bin/daemon-helper 2023-07-28T23:37:35.935 DEBUG:teuthology.orchestra.run.smithi101:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-28T23:37:36.003 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2023-07-28T23:37:36.004 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/usr/bin/daemon-helper 2023-07-28T23:37:36.035 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-28T23:37:36.105 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-07-28T23:37:36.106 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-07-28T23:37:36.106 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-28T23:37:36.134 DEBUG:teuthology.orchestra.run.smithi101:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-28T23:37:36.201 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2023-07-28T23:37:36.201 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-28T23:37:36.232 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-28T23:37:36.302 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-07-28T23:37:36.302 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-07-28T23:37:36.303 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd of=/usr/bin/stdin-killer 2023-07-28T23:37:36.329 DEBUG:teuthology.orchestra.run.smithi101:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-28T23:37:36.395 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2023-07-28T23:37:36.396 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/usr/bin/stdin-killer 2023-07-28T23:37:36.424 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-28T23:37:36.492 INFO:teuthology.run_tasks:Running task ceph... 2023-07-28T23:37:36.879 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-07-28T23:37:36.880 DEBUG:teuthology.orchestra.run.smithi101:> sudo chmod 777 /var/log/ceph 2023-07-28T23:37:36.882 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod 777 /var/log/ceph 2023-07-28T23:37:36.911 INFO:tasks.ceph:Disabling ceph logrotate... 2023-07-28T23:37:36.911 DEBUG:teuthology.orchestra.run.smithi101:> sudo rm -f -- /etc/logrotate.d/ceph 2023-07-28T23:37:36.951 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f -- /etc/logrotate.d/ceph 2023-07-28T23:37:36.979 INFO:tasks.ceph:Creating extra log directories... 2023-07-28T23:37:36.980 DEBUG:teuthology.orchestra.run.smithi101:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-07-28T23:37:37.018 DEBUG:teuthology.orchestra.run.smithi139:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-07-28T23:37:37.056 INFO:tasks.ceph:Creating ceph cluster c1... 2023-07-28T23:37:37.056 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-07-28T23:37:37.056 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-07-28_23:12:07-rgw-reef-release-distro-default-smithi/7355843', 'branch': 'reef-release', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_latest} tasks/test_multi}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7355843', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-07-28_23:12:07-rgw-reef-release-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'reef-release'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '0484097b0572e96f42cf6402cbe0ac8dcb046577', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '0484097b0572e96f42cf6402cbe0ac8dcb046577'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a1'}, {'endpoints': ['c1.client.1'], 'name': 'a2'}]}, {'enabled_features': ['resharding'], 'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'b', 'zones': [{'endpoints': ['c2.client.0'], 'is_default': True, 'is_master': True, 'name': 'b1'}, {'endpoints': ['c2.client.1'], 'name': 'b2'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'workunit': {'branch': 'reef-release', 'sha1': '0484097b0572e96f42cf6402cbe0ac8dcb046577'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 2147, 'sha1': '0484097b0572e96f42cf6402cbe0ac8dcb046577', 'sleep_before_teardown': 0, 'subset': '111/12000', 'suite': 'rgw', 'suite_branch': 'reef-release', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_0484097b0572e96f42cf6402cbe0ac8dcb046577/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '0484097b0572e96f42cf6402cbe0ac8dcb046577', 'targets': {'smithi101.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDciPeiA+F8OU2n9arsBRdxjittPCt9x6Dl/s+Hfxtf1oRgJdXHBnQWafNP5NjVmYg7hnf5zZnMl1HsH2pMMc+8=', 'smithi139.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGcp7KnNK6sxlDspWEyXozyB+lgk0NC9jK/uC8FVE4JLB64hFISTyFr+uS0NN4bYfyCUHn21kkjENMNOmGNcFI4='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '0484097b0572e96f42cf6402cbe0ac8dcb046577', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '61140164d2579c30ffff3970615e0f82e40c6470', 'timestamp': '2023-07-28_23:12:07', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.23459'} 2023-07-28T23:37:37.057 DEBUG:teuthology.orchestra.run.smithi101:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2023-07-28T23:37:37.086 DEBUG:teuthology.orchestra.run.smithi139:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2023-07-28T23:37:37.101 DEBUG:teuthology.orchestra.run.smithi101:> sudo install -d -m0777 -- /var/run/ceph 2023-07-28T23:37:37.141 DEBUG:teuthology.orchestra.run.smithi139:> sudo install -d -m0777 -- /var/run/ceph 2023-07-28T23:37:37.168 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-07-28T23:37:37.168 DEBUG:teuthology.orchestra.run.smithi101:> dd if=/scratch_devs of=/dev/stdout 2023-07-28T23:37:37.220 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-28T23:37:37.220 DEBUG:teuthology.orchestra.run.smithi101:> stat /dev/vg_nvme/lv_1 2023-07-28T23:37:37.275 INFO:teuthology.orchestra.run.smithi101.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-28T23:37:37.275 INFO:teuthology.orchestra.run.smithi101.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-28T23:37:37.275 INFO:teuthology.orchestra.run.smithi101.stdout:Device: 5h/5d Inode: 655 Links: 1 2023-07-28T23:37:37.275 INFO:teuthology.orchestra.run.smithi101.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-28T23:37:37.275 INFO:teuthology.orchestra.run.smithi101.stdout:Context: system_u:object_r:device_t:s0 2023-07-28T23:37:37.275 INFO:teuthology.orchestra.run.smithi101.stdout:Access: 2023-07-28 23:37:20.158712408 +0000 2023-07-28T23:37:37.275 INFO:teuthology.orchestra.run.smithi101.stdout:Modify: 2023-07-28 23:32:42.128480575 +0000 2023-07-28T23:37:37.276 INFO:teuthology.orchestra.run.smithi101.stdout:Change: 2023-07-28 23:32:42.128480575 +0000 2023-07-28T23:37:37.276 INFO:teuthology.orchestra.run.smithi101.stdout: Birth: 2023-07-28 23:32:42.128480575 +0000 2023-07-28T23:37:37.276 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-28T23:37:37.339 INFO:teuthology.orchestra.run.smithi101.stderr:1+0 records in 2023-07-28T23:37:37.339 INFO:teuthology.orchestra.run.smithi101.stderr:1+0 records out 2023-07-28T23:37:37.339 INFO:teuthology.orchestra.run.smithi101.stderr:512 bytes copied, 0.000201981 s, 2.5 MB/s 2023-07-28T23:37:37.340 DEBUG:teuthology.orchestra.run.smithi101:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-28T23:37:37.395 DEBUG:teuthology.orchestra.run.smithi101:> stat /dev/vg_nvme/lv_2 2023-07-28T23:37:37.449 INFO:teuthology.orchestra.run.smithi101.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-28T23:37:37.449 INFO:teuthology.orchestra.run.smithi101.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-28T23:37:37.449 INFO:teuthology.orchestra.run.smithi101.stdout:Device: 5h/5d Inode: 671 Links: 1 2023-07-28T23:37:37.450 INFO:teuthology.orchestra.run.smithi101.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-28T23:37:37.450 INFO:teuthology.orchestra.run.smithi101.stdout:Context: system_u:object_r:device_t:s0 2023-07-28T23:37:37.450 INFO:teuthology.orchestra.run.smithi101.stdout:Access: 2023-07-28 23:37:20.158712408 +0000 2023-07-28T23:37:37.450 INFO:teuthology.orchestra.run.smithi101.stdout:Modify: 2023-07-28 23:32:42.621468789 +0000 2023-07-28T23:37:37.450 INFO:teuthology.orchestra.run.smithi101.stdout:Change: 2023-07-28 23:32:42.621468789 +0000 2023-07-28T23:37:37.450 INFO:teuthology.orchestra.run.smithi101.stdout: Birth: 2023-07-28 23:32:42.621468789 +0000 2023-07-28T23:37:37.451 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-28T23:37:37.514 INFO:teuthology.orchestra.run.smithi101.stderr:1+0 records in 2023-07-28T23:37:37.514 INFO:teuthology.orchestra.run.smithi101.stderr:1+0 records out 2023-07-28T23:37:37.514 INFO:teuthology.orchestra.run.smithi101.stderr:512 bytes copied, 0.000196078 s, 2.6 MB/s 2023-07-28T23:37:37.515 DEBUG:teuthology.orchestra.run.smithi101:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-28T23:37:37.570 DEBUG:teuthology.orchestra.run.smithi101:> stat /dev/vg_nvme/lv_3 2023-07-28T23:37:37.624 INFO:teuthology.orchestra.run.smithi101.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-28T23:37:37.624 INFO:teuthology.orchestra.run.smithi101.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-28T23:37:37.624 INFO:teuthology.orchestra.run.smithi101.stdout:Device: 5h/5d Inode: 689 Links: 1 2023-07-28T23:37:37.624 INFO:teuthology.orchestra.run.smithi101.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-28T23:37:37.625 INFO:teuthology.orchestra.run.smithi101.stdout:Context: system_u:object_r:device_t:s0 2023-07-28T23:37:37.625 INFO:teuthology.orchestra.run.smithi101.stdout:Access: 2023-07-28 23:37:20.159712384 +0000 2023-07-28T23:37:37.625 INFO:teuthology.orchestra.run.smithi101.stdout:Modify: 2023-07-28 23:32:43.120456859 +0000 2023-07-28T23:37:37.625 INFO:teuthology.orchestra.run.smithi101.stdout:Change: 2023-07-28 23:32:43.120456859 +0000 2023-07-28T23:37:37.625 INFO:teuthology.orchestra.run.smithi101.stdout: Birth: 2023-07-28 23:32:43.120456859 +0000 2023-07-28T23:37:37.625 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-28T23:37:37.689 INFO:teuthology.orchestra.run.smithi101.stderr:1+0 records in 2023-07-28T23:37:37.689 INFO:teuthology.orchestra.run.smithi101.stderr:1+0 records out 2023-07-28T23:37:37.689 INFO:teuthology.orchestra.run.smithi101.stderr:512 bytes copied, 0.000207924 s, 2.5 MB/s 2023-07-28T23:37:37.690 DEBUG:teuthology.orchestra.run.smithi101:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-28T23:37:37.745 DEBUG:teuthology.orchestra.run.smithi101:> stat /dev/vg_nvme/lv_4 2023-07-28T23:37:37.799 INFO:teuthology.orchestra.run.smithi101.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-28T23:37:37.799 INFO:teuthology.orchestra.run.smithi101.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-28T23:37:37.799 INFO:teuthology.orchestra.run.smithi101.stdout:Device: 5h/5d Inode: 708 Links: 1 2023-07-28T23:37:37.800 INFO:teuthology.orchestra.run.smithi101.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-28T23:37:37.800 INFO:teuthology.orchestra.run.smithi101.stdout:Context: system_u:object_r:device_t:s0 2023-07-28T23:37:37.800 INFO:teuthology.orchestra.run.smithi101.stdout:Access: 2023-07-28 23:37:20.159712384 +0000 2023-07-28T23:37:37.800 INFO:teuthology.orchestra.run.smithi101.stdout:Modify: 2023-07-28 23:32:43.633444594 +0000 2023-07-28T23:37:37.800 INFO:teuthology.orchestra.run.smithi101.stdout:Change: 2023-07-28 23:32:43.633444594 +0000 2023-07-28T23:37:37.800 INFO:teuthology.orchestra.run.smithi101.stdout: Birth: 2023-07-28 23:32:43.633444594 +0000 2023-07-28T23:37:37.801 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-28T23:37:37.864 INFO:teuthology.orchestra.run.smithi101.stderr:1+0 records in 2023-07-28T23:37:37.864 INFO:teuthology.orchestra.run.smithi101.stderr:1+0 records out 2023-07-28T23:37:37.864 INFO:teuthology.orchestra.run.smithi101.stderr:512 bytes copied, 0.000219835 s, 2.3 MB/s 2023-07-28T23:37:37.865 DEBUG:teuthology.orchestra.run.smithi101:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-28T23:37:37.920 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-07-28T23:37:37.920 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi101.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-07-28T23:37:37.921 INFO:tasks.ceph:Generating config... 2023-07-28T23:37:37.923 INFO:tasks.ceph:[client] debug rgw = 20 2023-07-28T23:37:37.924 INFO:tasks.ceph:[client] debug rgw sync = 20 2023-07-28T23:37:37.924 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2023-07-28T23:37:37.924 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2023-07-28T23:37:37.924 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-07-28T23:37:37.924 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2023-07-28T23:37:37.925 INFO:tasks.ceph:[client] rgw data log num shards = 4 2023-07-28T23:37:37.925 INFO:tasks.ceph:[client] rgw md log max shards = 4 2023-07-28T23:37:37.925 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2023-07-28T23:37:37.925 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2023-07-28T23:37:37.925 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2023-07-28T23:37:37.926 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2023-07-28T23:37:37.926 INFO:tasks.ceph:[client] setgroup = ceph 2023-07-28T23:37:37.926 INFO:tasks.ceph:[client] setuser = ceph 2023-07-28T23:37:37.926 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2023-07-28T23:37:37.926 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-07-28T23:37:37.927 INFO:tasks.ceph:[mgr] debug ms = 1 2023-07-28T23:37:37.927 INFO:tasks.ceph:[mon] debug mon = 20 2023-07-28T23:37:37.927 INFO:tasks.ceph:[mon] debug ms = 1 2023-07-28T23:37:37.927 INFO:tasks.ceph:[mon] debug paxos = 20 2023-07-28T23:37:37.927 INFO:tasks.ceph:[osd] debug ms = 1 2023-07-28T23:37:37.927 INFO:tasks.ceph:[osd] debug osd = 20 2023-07-28T23:37:37.928 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2023-07-28T23:37:37.928 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2023-07-28T23:37:37.928 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2023-07-28T23:37:37.928 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2023-07-28T23:37:37.929 INFO:tasks.ceph:Setting up c1.mon.a... 2023-07-28T23:37:37.929 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2023-07-28T23:37:38.009 INFO:teuthology.orchestra.run.smithi101.stdout:creating /etc/ceph/c1.keyring 2023-07-28T23:37:38.012 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2023-07-28T23:37:38.055 DEBUG:teuthology.orchestra.run.smithi101:> sudo chmod 0644 /etc/ceph/c1.keyring 2023-07-28T23:37:38.120 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.101')] 2023-07-28T23:37:38.121 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.101'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2023-07-28T23:37:38.121 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-07-28T23:37:38.122 DEBUG:teuthology.orchestra.run.smithi101:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-07-28T23:37:38.174 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.101 --print /home/ubuntu/cephtest/c1.monmap 2023-07-28T23:37:38.252 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:37:38.252 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:37:38.253 INFO:teuthology.orchestra.run.smithi101.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2023-07-28T23:37:38.253 INFO:teuthology.orchestra.run.smithi101.stdout:monmaptool: generated fsid 859e430b-4e7d-4232-8462-93980f0a2aba 2023-07-28T23:37:38.253 INFO:teuthology.orchestra.run.smithi101.stdout:setting min_mon_release = pacific 2023-07-28T23:37:38.253 INFO:teuthology.orchestra.run.smithi101.stdout:epoch 0 2023-07-28T23:37:38.253 INFO:teuthology.orchestra.run.smithi101.stdout:fsid 859e430b-4e7d-4232-8462-93980f0a2aba 2023-07-28T23:37:38.254 INFO:teuthology.orchestra.run.smithi101.stdout:last_changed 2023-07-28T23:37:38.256448+0000 2023-07-28T23:37:38.254 INFO:teuthology.orchestra.run.smithi101.stdout:created 2023-07-28T23:37:38.256448+0000 2023-07-28T23:37:38.254 INFO:teuthology.orchestra.run.smithi101.stdout:min_mon_release 16 (pacific) 2023-07-28T23:37:38.254 INFO:teuthology.orchestra.run.smithi101.stdout:election_strategy: 1 2023-07-28T23:37:38.254 INFO:teuthology.orchestra.run.smithi101.stdout:0: [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] mon.a 2023-07-28T23:37:38.254 INFO:teuthology.orchestra.run.smithi101.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2023-07-28T23:37:38.255 DEBUG:teuthology.orchestra.run.smithi101:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-07-28T23:37:38.309 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID 859e430b-4e7d-4232-8462-93980f0a2aba... 2023-07-28T23:37:38.310 DEBUG:teuthology.orchestra.run.smithi101:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2023-07-28T23:37:38.351 DEBUG:teuthology.orchestra.run.smithi139:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2023-07-28T23:37:38.399 INFO:teuthology.orchestra.run.smithi101.stdout:[global] 2023-07-28T23:37:38.399 INFO:teuthology.orchestra.run.smithi101.stdout: chdir = "" 2023-07-28T23:37:38.399 INFO:teuthology.orchestra.run.smithi101.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-07-28T23:37:38.399 INFO:teuthology.orchestra.run.smithi101.stdout: auth supported = cephx 2023-07-28T23:37:38.399 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:38.399 INFO:teuthology.orchestra.run.smithi101.stdout: filestore xattr use omap = true 2023-07-28T23:37:38.400 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:38.400 INFO:teuthology.orchestra.run.smithi101.stdout: mon clock drift allowed = 1.000 2023-07-28T23:37:38.400 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:38.400 INFO:teuthology.orchestra.run.smithi101.stdout: osd crush chooseleaf type = 0 2023-07-28T23:37:38.400 INFO:teuthology.orchestra.run.smithi101.stdout: auth debug = true 2023-07-28T23:37:38.400 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:38.400 INFO:teuthology.orchestra.run.smithi101.stdout: ms die on old message = true 2023-07-28T23:37:38.401 INFO:teuthology.orchestra.run.smithi101.stdout: ms die on bug = true 2023-07-28T23:37:38.401 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:38.401 INFO:teuthology.orchestra.run.smithi101.stdout: mon max pg per osd = 10000 # >= luminous 2023-07-28T23:37:38.401 INFO:teuthology.orchestra.run.smithi101.stdout: mon pg warn max object skew = 0 2023-07-28T23:37:38.401 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:38.401 INFO:teuthology.orchestra.run.smithi101.stdout: # disable pg_autoscaler by default for new pools 2023-07-28T23:37:38.401 INFO:teuthology.orchestra.run.smithi101.stdout: osd_pool_default_pg_autoscale_mode = off 2023-07-28T23:37:38.401 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:38.402 INFO:teuthology.orchestra.run.smithi101.stdout: osd pool default size = 2 2023-07-28T23:37:38.402 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:38.402 INFO:teuthology.orchestra.run.smithi101.stdout: mon osd allow primary affinity = true 2023-07-28T23:37:38.402 INFO:teuthology.orchestra.run.smithi101.stdout: mon osd allow pg remap = true 2023-07-28T23:37:38.402 INFO:teuthology.orchestra.run.smithi101.stdout: mon warn on legacy crush tunables = false 2023-07-28T23:37:38.402 INFO:teuthology.orchestra.run.smithi101.stdout: mon warn on crush straw calc version zero = false 2023-07-28T23:37:38.402 INFO:teuthology.orchestra.run.smithi101.stdout: mon warn on no sortbitwise = false 2023-07-28T23:37:38.402 INFO:teuthology.orchestra.run.smithi101.stdout: mon warn on osd down out interval zero = false 2023-07-28T23:37:38.403 INFO:teuthology.orchestra.run.smithi101.stdout: mon warn on too few osds = false 2023-07-28T23:37:38.403 INFO:teuthology.orchestra.run.smithi101.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-07-28T23:37:38.403 INFO:teuthology.orchestra.run.smithi101.stdout: mon_warn_on_pool_no_redundancy = false 2023-07-28T23:37:38.403 INFO:teuthology.orchestra.run.smithi101.stdout: mon_allow_pool_size_one = true 2023-07-28T23:37:38.403 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:38.403 INFO:teuthology.orchestra.run.smithi101.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-07-28T23:37:38.404 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:38.404 INFO:teuthology.orchestra.run.smithi101.stdout: osd default data pool replay window = 5 2023-07-28T23:37:38.404 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:38.404 INFO:teuthology.orchestra.run.smithi101.stdout: mon allow pool delete = true 2023-07-28T23:37:38.404 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:38.404 INFO:teuthology.orchestra.run.smithi101.stdout: mon cluster log file level = debug 2023-07-28T23:37:38.405 INFO:teuthology.orchestra.run.smithi101.stdout: debug asserts on shutdown = true 2023-07-28T23:37:38.405 INFO:teuthology.orchestra.run.smithi101.stdout: mon health detail to clog = false 2023-07-28T23:37:38.405 INFO:teuthology.orchestra.run.smithi101.stdout: mon host = 172.21.15.101 2023-07-28T23:37:38.405 INFO:teuthology.orchestra.run.smithi101.stdout: fsid = 859e430b-4e7d-4232-8462-93980f0a2aba 2023-07-28T23:37:38.405 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:38.405 INFO:teuthology.orchestra.run.smithi101.stdout:[osd] 2023-07-28T23:37:38.405 INFO:teuthology.orchestra.run.smithi101.stdout: osd journal size = 100 2023-07-28T23:37:38.405 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:38.406 INFO:teuthology.orchestra.run.smithi101.stdout: osd scrub load threshold = 5.0 2023-07-28T23:37:38.406 INFO:teuthology.orchestra.run.smithi101.stdout: osd scrub max interval = 600 2023-07-28T23:37:38.406 INFO:teuthology.orchestra.run.smithi101.stdout: osd mclock profile = high_recovery_ops 2023-07-28T23:37:38.406 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:38.406 INFO:teuthology.orchestra.run.smithi101.stdout: osd recover clone overlap = true 2023-07-28T23:37:38.406 INFO:teuthology.orchestra.run.smithi101.stdout: osd recovery max chunk = 1048576 2023-07-28T23:37:38.406 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:38.406 INFO:teuthology.orchestra.run.smithi101.stdout: osd debug shutdown = true 2023-07-28T23:37:38.407 INFO:teuthology.orchestra.run.smithi101.stdout: osd debug op order = true 2023-07-28T23:37:38.407 INFO:teuthology.orchestra.run.smithi101.stdout: osd debug verify stray on activate = true 2023-07-28T23:37:38.407 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:38.407 INFO:teuthology.orchestra.run.smithi101.stdout: osd open classes on start = true 2023-07-28T23:37:38.407 INFO:teuthology.orchestra.run.smithi101.stdout: osd debug pg log writeout = true 2023-07-28T23:37:38.407 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:38.407 INFO:teuthology.orchestra.run.smithi101.stdout: osd deep scrub update digest min age = 30 2023-07-28T23:37:38.407 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:38.408 INFO:teuthology.orchestra.run.smithi101.stdout: osd map max advance = 10 2023-07-28T23:37:38.408 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:38.408 INFO:teuthology.orchestra.run.smithi101.stdout: journal zero on create = true 2023-07-28T23:37:38.408 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:38.408 INFO:teuthology.orchestra.run.smithi101.stdout: filestore ondisk finisher threads = 3 2023-07-28T23:37:38.408 INFO:teuthology.orchestra.run.smithi101.stdout: filestore apply finisher threads = 3 2023-07-28T23:37:38.408 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:38.409 INFO:teuthology.orchestra.run.smithi101.stdout: bdev debug aio = true 2023-07-28T23:37:38.409 INFO:teuthology.orchestra.run.smithi101.stdout: osd debug misdirected ops = true 2023-07-28T23:37:38.409 INFO:teuthology.orchestra.run.smithi101.stdout: debug ms = 1 2023-07-28T23:37:38.409 INFO:teuthology.orchestra.run.smithi101.stdout: debug osd = 20 2023-07-28T23:37:38.409 INFO:teuthology.orchestra.run.smithi101.stdout: rocksdb delete range threshold = 0 2023-07-28T23:37:38.409 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:38.409 INFO:teuthology.orchestra.run.smithi101.stdout:[mgr] 2023-07-28T23:37:38.409 INFO:teuthology.orchestra.run.smithi101.stdout: debug ms = 1 2023-07-28T23:37:38.410 INFO:teuthology.orchestra.run.smithi101.stdout: debug mgr = 20 2023-07-28T23:37:38.410 INFO:teuthology.orchestra.run.smithi101.stdout: debug mon = 20 2023-07-28T23:37:38.410 INFO:teuthology.orchestra.run.smithi101.stdout: debug auth = 20 2023-07-28T23:37:38.410 INFO:teuthology.orchestra.run.smithi101.stdout: mon reweight min pgs per osd = 4 2023-07-28T23:37:38.410 INFO:teuthology.orchestra.run.smithi101.stdout: mon reweight min bytes per osd = 10 2023-07-28T23:37:38.410 INFO:teuthology.orchestra.run.smithi101.stdout: mgr/telemetry/nag = false 2023-07-28T23:37:38.410 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:38.410 INFO:teuthology.orchestra.run.smithi101.stdout:[mon] 2023-07-28T23:37:38.411 INFO:teuthology.orchestra.run.smithi101.stdout: debug ms = 1 2023-07-28T23:37:38.411 INFO:teuthology.orchestra.run.smithi101.stdout: debug mon = 20 2023-07-28T23:37:38.411 INFO:teuthology.orchestra.run.smithi101.stdout: debug paxos = 20 2023-07-28T23:37:38.411 INFO:teuthology.orchestra.run.smithi101.stdout: debug auth = 20 2023-07-28T23:37:38.411 INFO:teuthology.orchestra.run.smithi101.stdout: mon data avail warn = 5 2023-07-28T23:37:38.411 INFO:teuthology.orchestra.run.smithi101.stdout: mon mgr mkfs grace = 240 2023-07-28T23:37:38.411 INFO:teuthology.orchestra.run.smithi101.stdout: mon reweight min pgs per osd = 4 2023-07-28T23:37:38.411 INFO:teuthology.orchestra.run.smithi101.stdout: mon osd reporter subtree level = osd 2023-07-28T23:37:38.412 INFO:teuthology.orchestra.run.smithi101.stdout: mon osd prime pg temp = true 2023-07-28T23:37:38.412 INFO:teuthology.orchestra.run.smithi101.stdout: mon reweight min bytes per osd = 10 2023-07-28T23:37:38.412 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:38.412 INFO:teuthology.orchestra.run.smithi101.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-07-28T23:37:38.412 INFO:teuthology.orchestra.run.smithi101.stdout: auth mon ticket ttl = 660 # 11m 2023-07-28T23:37:38.412 INFO:teuthology.orchestra.run.smithi101.stdout: auth service ticket ttl = 240 # 4m 2023-07-28T23:37:38.412 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:38.413 INFO:teuthology.orchestra.run.smithi101.stdout: # don't complain about insecure global_id in the test suite 2023-07-28T23:37:38.413 INFO:teuthology.orchestra.run.smithi101.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-07-28T23:37:38.413 INFO:teuthology.orchestra.run.smithi101.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-07-28T23:37:38.413 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:38.413 INFO:teuthology.orchestra.run.smithi101.stdout: # 1m isn't quite enough 2023-07-28T23:37:38.413 INFO:teuthology.orchestra.run.smithi101.stdout: mon_down_mkfs_grace = 2m 2023-07-28T23:37:38.413 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:38.413 INFO:teuthology.orchestra.run.smithi101.stdout: mon_warn_on_filestore_osds = false 2023-07-28T23:37:38.414 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:38.414 INFO:teuthology.orchestra.run.smithi101.stdout:[client] 2023-07-28T23:37:38.414 INFO:teuthology.orchestra.run.smithi101.stdout: rgw cache enabled = true 2023-07-28T23:37:38.414 INFO:teuthology.orchestra.run.smithi101.stdout: rgw enable ops log = true 2023-07-28T23:37:38.414 INFO:teuthology.orchestra.run.smithi101.stdout: rgw enable usage log = true 2023-07-28T23:37:38.414 INFO:teuthology.orchestra.run.smithi101.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-07-28T23:37:38.414 INFO:teuthology.orchestra.run.smithi101.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-07-28T23:37:38.414 INFO:teuthology.orchestra.run.smithi101.stdout: debug rgw = 20 2023-07-28T23:37:38.415 INFO:teuthology.orchestra.run.smithi101.stdout: debug rgw sync = 20 2023-07-28T23:37:38.415 INFO:teuthology.orchestra.run.smithi101.stdout: rgw crypt require ssl = False 2023-07-28T23:37:38.415 INFO:teuthology.orchestra.run.smithi101.stdout: rgw crypt s3 kms backend = testing 2023-07-28T23:37:38.415 INFO:teuthology.orchestra.run.smithi101.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-07-28T23:37:38.415 INFO:teuthology.orchestra.run.smithi101.stdout: rgw curl low speed time = 300 2023-07-28T23:37:38.415 INFO:teuthology.orchestra.run.smithi101.stdout: rgw data log num shards = 4 2023-07-28T23:37:38.415 INFO:teuthology.orchestra.run.smithi101.stdout: rgw md log max shards = 4 2023-07-28T23:37:38.415 INFO:teuthology.orchestra.run.smithi101.stdout: rgw sync data inject err probability = 0.1 2023-07-28T23:37:38.416 INFO:teuthology.orchestra.run.smithi101.stdout: rgw sync log trim interval = 0 2023-07-28T23:37:38.416 INFO:teuthology.orchestra.run.smithi101.stdout: rgw sync meta inject err probability = 0.1 2023-07-28T23:37:38.416 INFO:teuthology.orchestra.run.smithi101.stdout: rgw sync obj etag verify = True 2023-07-28T23:37:38.416 INFO:teuthology.orchestra.run.smithi101.stdout: setgroup = ceph 2023-07-28T23:37:38.416 INFO:teuthology.orchestra.run.smithi101.stdout: setuser = ceph 2023-07-28T23:37:38.416 INFO:teuthology.orchestra.run.smithi101.stdout:[mon.a] 2023-07-28T23:37:38.416 INFO:teuthology.orchestra.run.smithi101.stdout:[client.0] 2023-07-28T23:37:38.417 INFO:teuthology.orchestra.run.smithi101.stdout: rgw data notify interval msec = 0 2023-07-28T23:37:38.417 INFO:teuthology.orchestra.run.smithi101.stdout:[osd.0] 2023-07-28T23:37:38.417 INFO:teuthology.orchestra.run.smithi101.stdout: osd_max_omap_entries_per_request = 10 2023-07-28T23:37:38.417 INFO:teuthology.orchestra.run.smithi101.stdout:[osd.1] 2023-07-28T23:37:38.417 INFO:teuthology.orchestra.run.smithi101.stdout: osd_max_omap_entries_per_request = 1000 2023-07-28T23:37:38.417 INFO:teuthology.orchestra.run.smithi101.stdout:[osd.2] 2023-07-28T23:37:38.417 INFO:teuthology.orchestra.run.smithi101.stdout: osd_max_omap_entries_per_request = 10000 2023-07-28T23:37:38.418 INFO:teuthology.orchestra.run.smithi139.stdout:[global] 2023-07-28T23:37:38.418 INFO:teuthology.orchestra.run.smithi139.stdout: chdir = "" 2023-07-28T23:37:38.418 INFO:teuthology.orchestra.run.smithi139.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-07-28T23:37:38.418 INFO:teuthology.orchestra.run.smithi139.stdout: auth supported = cephx 2023-07-28T23:37:38.418 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:37:38.419 INFO:teuthology.orchestra.run.smithi139.stdout: filestore xattr use omap = true 2023-07-28T23:37:38.419 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:37:38.419 INFO:teuthology.orchestra.run.smithi139.stdout: mon clock drift allowed = 1.000 2023-07-28T23:37:38.419 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:37:38.419 INFO:teuthology.orchestra.run.smithi139.stdout: osd crush chooseleaf type = 0 2023-07-28T23:37:38.419 INFO:teuthology.orchestra.run.smithi139.stdout: auth debug = true 2023-07-28T23:37:38.419 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:37:38.419 INFO:teuthology.orchestra.run.smithi139.stdout: ms die on old message = true 2023-07-28T23:37:38.420 INFO:teuthology.orchestra.run.smithi139.stdout: ms die on bug = true 2023-07-28T23:37:38.420 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:37:38.420 INFO:teuthology.orchestra.run.smithi139.stdout: mon max pg per osd = 10000 # >= luminous 2023-07-28T23:37:38.420 INFO:teuthology.orchestra.run.smithi139.stdout: mon pg warn max object skew = 0 2023-07-28T23:37:38.420 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:37:38.420 INFO:teuthology.orchestra.run.smithi139.stdout: # disable pg_autoscaler by default for new pools 2023-07-28T23:37:38.420 INFO:teuthology.orchestra.run.smithi139.stdout: osd_pool_default_pg_autoscale_mode = off 2023-07-28T23:37:38.420 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:37:38.421 INFO:teuthology.orchestra.run.smithi139.stdout: osd pool default size = 2 2023-07-28T23:37:38.421 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:37:38.421 INFO:teuthology.orchestra.run.smithi139.stdout: mon osd allow primary affinity = true 2023-07-28T23:37:38.421 INFO:teuthology.orchestra.run.smithi139.stdout: mon osd allow pg remap = true 2023-07-28T23:37:38.421 INFO:teuthology.orchestra.run.smithi139.stdout: mon warn on legacy crush tunables = false 2023-07-28T23:37:38.421 INFO:teuthology.orchestra.run.smithi139.stdout: mon warn on crush straw calc version zero = false 2023-07-28T23:37:38.421 INFO:teuthology.orchestra.run.smithi139.stdout: mon warn on no sortbitwise = false 2023-07-28T23:37:38.422 INFO:teuthology.orchestra.run.smithi139.stdout: mon warn on osd down out interval zero = false 2023-07-28T23:37:38.422 INFO:teuthology.orchestra.run.smithi139.stdout: mon warn on too few osds = false 2023-07-28T23:37:38.422 INFO:teuthology.orchestra.run.smithi139.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-07-28T23:37:38.422 INFO:teuthology.orchestra.run.smithi139.stdout: mon_warn_on_pool_no_redundancy = false 2023-07-28T23:37:38.422 INFO:teuthology.orchestra.run.smithi139.stdout: mon_allow_pool_size_one = true 2023-07-28T23:37:38.422 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:37:38.422 INFO:teuthology.orchestra.run.smithi139.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-07-28T23:37:38.422 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:37:38.423 INFO:teuthology.orchestra.run.smithi139.stdout: osd default data pool replay window = 5 2023-07-28T23:37:38.423 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:37:38.423 INFO:teuthology.orchestra.run.smithi139.stdout: mon allow pool delete = true 2023-07-28T23:37:38.423 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:37:38.423 INFO:teuthology.orchestra.run.smithi139.stdout: mon cluster log file level = debug 2023-07-28T23:37:38.423 INFO:teuthology.orchestra.run.smithi139.stdout: debug asserts on shutdown = true 2023-07-28T23:37:38.423 INFO:teuthology.orchestra.run.smithi139.stdout: mon health detail to clog = false 2023-07-28T23:37:38.423 INFO:teuthology.orchestra.run.smithi139.stdout: mon host = 172.21.15.101 2023-07-28T23:37:38.424 INFO:teuthology.orchestra.run.smithi139.stdout: fsid = 859e430b-4e7d-4232-8462-93980f0a2aba 2023-07-28T23:37:38.424 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:37:38.424 INFO:teuthology.orchestra.run.smithi139.stdout:[osd] 2023-07-28T23:37:38.424 INFO:teuthology.orchestra.run.smithi139.stdout: osd journal size = 100 2023-07-28T23:37:38.424 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:37:38.424 INFO:teuthology.orchestra.run.smithi139.stdout: osd scrub load threshold = 5.0 2023-07-28T23:37:38.425 INFO:teuthology.orchestra.run.smithi139.stdout: osd scrub max interval = 600 2023-07-28T23:37:38.425 INFO:teuthology.orchestra.run.smithi139.stdout: osd mclock profile = high_recovery_ops 2023-07-28T23:37:38.425 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:37:38.425 INFO:teuthology.orchestra.run.smithi139.stdout: osd recover clone overlap = true 2023-07-28T23:37:38.425 INFO:teuthology.orchestra.run.smithi139.stdout: osd recovery max chunk = 1048576 2023-07-28T23:37:38.425 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:37:38.425 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug shutdown = true 2023-07-28T23:37:38.425 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug op order = true 2023-07-28T23:37:38.426 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug verify stray on activate = true 2023-07-28T23:37:38.426 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:37:38.426 INFO:teuthology.orchestra.run.smithi139.stdout: osd open classes on start = true 2023-07-28T23:37:38.426 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug pg log writeout = true 2023-07-28T23:37:38.426 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:37:38.426 INFO:teuthology.orchestra.run.smithi139.stdout: osd deep scrub update digest min age = 30 2023-07-28T23:37:38.426 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:37:38.426 INFO:teuthology.orchestra.run.smithi139.stdout: osd map max advance = 10 2023-07-28T23:37:38.427 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:37:38.427 INFO:teuthology.orchestra.run.smithi139.stdout: journal zero on create = true 2023-07-28T23:37:38.427 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:37:38.427 INFO:teuthology.orchestra.run.smithi139.stdout: filestore ondisk finisher threads = 3 2023-07-28T23:37:38.427 INFO:teuthology.orchestra.run.smithi139.stdout: filestore apply finisher threads = 3 2023-07-28T23:37:38.427 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:37:38.427 INFO:teuthology.orchestra.run.smithi139.stdout: bdev debug aio = true 2023-07-28T23:37:38.427 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug misdirected ops = true 2023-07-28T23:37:38.428 INFO:teuthology.orchestra.run.smithi139.stdout: debug ms = 1 2023-07-28T23:37:38.428 INFO:teuthology.orchestra.run.smithi139.stdout: debug osd = 20 2023-07-28T23:37:38.428 INFO:teuthology.orchestra.run.smithi139.stdout: rocksdb delete range threshold = 0 2023-07-28T23:37:38.428 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:37:38.428 INFO:teuthology.orchestra.run.smithi139.stdout:[mgr] 2023-07-28T23:37:38.428 INFO:teuthology.orchestra.run.smithi139.stdout: debug ms = 1 2023-07-28T23:37:38.429 INFO:teuthology.orchestra.run.smithi139.stdout: debug mgr = 20 2023-07-28T23:37:38.429 INFO:teuthology.orchestra.run.smithi139.stdout: debug mon = 20 2023-07-28T23:37:38.429 INFO:teuthology.orchestra.run.smithi139.stdout: debug auth = 20 2023-07-28T23:37:38.429 INFO:teuthology.orchestra.run.smithi139.stdout: mon reweight min pgs per osd = 4 2023-07-28T23:37:38.429 INFO:teuthology.orchestra.run.smithi139.stdout: mon reweight min bytes per osd = 10 2023-07-28T23:37:38.429 INFO:teuthology.orchestra.run.smithi139.stdout: mgr/telemetry/nag = false 2023-07-28T23:37:38.429 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:37:38.429 INFO:teuthology.orchestra.run.smithi139.stdout:[mon] 2023-07-28T23:37:38.430 INFO:teuthology.orchestra.run.smithi139.stdout: debug ms = 1 2023-07-28T23:37:38.430 INFO:teuthology.orchestra.run.smithi139.stdout: debug mon = 20 2023-07-28T23:37:38.430 INFO:teuthology.orchestra.run.smithi139.stdout: debug paxos = 20 2023-07-28T23:37:38.430 INFO:teuthology.orchestra.run.smithi139.stdout: debug auth = 20 2023-07-28T23:37:38.430 INFO:teuthology.orchestra.run.smithi139.stdout: mon data avail warn = 5 2023-07-28T23:37:38.430 INFO:teuthology.orchestra.run.smithi139.stdout: mon mgr mkfs grace = 240 2023-07-28T23:37:38.430 INFO:teuthology.orchestra.run.smithi139.stdout: mon reweight min pgs per osd = 4 2023-07-28T23:37:38.430 INFO:teuthology.orchestra.run.smithi139.stdout: mon osd reporter subtree level = osd 2023-07-28T23:37:38.431 INFO:teuthology.orchestra.run.smithi139.stdout: mon osd prime pg temp = true 2023-07-28T23:37:38.431 INFO:teuthology.orchestra.run.smithi139.stdout: mon reweight min bytes per osd = 10 2023-07-28T23:37:38.431 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:37:38.431 INFO:teuthology.orchestra.run.smithi139.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-07-28T23:37:38.431 INFO:teuthology.orchestra.run.smithi139.stdout: auth mon ticket ttl = 660 # 11m 2023-07-28T23:37:38.431 INFO:teuthology.orchestra.run.smithi139.stdout: auth service ticket ttl = 240 # 4m 2023-07-28T23:37:38.431 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:37:38.432 INFO:teuthology.orchestra.run.smithi139.stdout: # don't complain about insecure global_id in the test suite 2023-07-28T23:37:38.432 INFO:teuthology.orchestra.run.smithi139.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-07-28T23:37:38.432 INFO:teuthology.orchestra.run.smithi139.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-07-28T23:37:38.432 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:37:38.432 INFO:teuthology.orchestra.run.smithi139.stdout: # 1m isn't quite enough 2023-07-28T23:37:38.432 INFO:teuthology.orchestra.run.smithi139.stdout: mon_down_mkfs_grace = 2m 2023-07-28T23:37:38.432 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:37:38.432 INFO:teuthology.orchestra.run.smithi139.stdout: mon_warn_on_filestore_osds = false 2023-07-28T23:37:38.433 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:37:38.433 INFO:teuthology.orchestra.run.smithi139.stdout:[client] 2023-07-28T23:37:38.433 INFO:teuthology.orchestra.run.smithi139.stdout: rgw cache enabled = true 2023-07-28T23:37:38.433 INFO:teuthology.orchestra.run.smithi139.stdout: rgw enable ops log = true 2023-07-28T23:37:38.433 INFO:teuthology.orchestra.run.smithi139.stdout: rgw enable usage log = true 2023-07-28T23:37:38.433 INFO:teuthology.orchestra.run.smithi139.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-07-28T23:37:38.433 INFO:teuthology.orchestra.run.smithi139.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-07-28T23:37:38.433 INFO:teuthology.orchestra.run.smithi139.stdout: debug rgw = 20 2023-07-28T23:37:38.434 INFO:teuthology.orchestra.run.smithi139.stdout: debug rgw sync = 20 2023-07-28T23:37:38.434 INFO:teuthology.orchestra.run.smithi139.stdout: rgw crypt require ssl = False 2023-07-28T23:37:38.434 INFO:teuthology.orchestra.run.smithi139.stdout: rgw crypt s3 kms backend = testing 2023-07-28T23:37:38.434 INFO:teuthology.orchestra.run.smithi139.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-07-28T23:37:38.434 INFO:teuthology.orchestra.run.smithi139.stdout: rgw curl low speed time = 300 2023-07-28T23:37:38.434 INFO:teuthology.orchestra.run.smithi139.stdout: rgw data log num shards = 4 2023-07-28T23:37:38.434 INFO:teuthology.orchestra.run.smithi139.stdout: rgw md log max shards = 4 2023-07-28T23:37:38.435 INFO:teuthology.orchestra.run.smithi139.stdout: rgw sync data inject err probability = 0.1 2023-07-28T23:37:38.435 INFO:teuthology.orchestra.run.smithi139.stdout: rgw sync log trim interval = 0 2023-07-28T23:37:38.435 INFO:teuthology.orchestra.run.smithi139.stdout: rgw sync meta inject err probability = 0.1 2023-07-28T23:37:38.435 INFO:teuthology.orchestra.run.smithi139.stdout: rgw sync obj etag verify = True 2023-07-28T23:37:38.435 INFO:teuthology.orchestra.run.smithi139.stdout: setgroup = ceph 2023-07-28T23:37:38.435 INFO:teuthology.orchestra.run.smithi139.stdout: setuser = ceph 2023-07-28T23:37:38.435 INFO:teuthology.orchestra.run.smithi139.stdout:[mon.a] 2023-07-28T23:37:38.435 INFO:teuthology.orchestra.run.smithi139.stdout:[client.0] 2023-07-28T23:37:38.436 INFO:teuthology.orchestra.run.smithi139.stdout: rgw data notify interval msec = 0 2023-07-28T23:37:38.436 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.0] 2023-07-28T23:37:38.436 INFO:teuthology.orchestra.run.smithi139.stdout: osd_max_omap_entries_per_request = 10 2023-07-28T23:37:38.436 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.1] 2023-07-28T23:37:38.436 INFO:teuthology.orchestra.run.smithi139.stdout: osd_max_omap_entries_per_request = 1000 2023-07-28T23:37:38.436 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.2] 2023-07-28T23:37:38.436 INFO:teuthology.orchestra.run.smithi139.stdout: osd_max_omap_entries_per_request = 10000 2023-07-28T23:37:38.437 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2023-07-28T23:37:38.438 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2023-07-28T23:37:38.533 INFO:tasks.ceph:Copying monmap to all nodes... 2023-07-28T23:37:38.533 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-07-28T23:37:38.534 DEBUG:teuthology.orchestra.run.smithi101:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2023-07-28T23:37:38.548 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-07-28T23:37:38.548 DEBUG:teuthology.orchestra.run.smithi101:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2023-07-28T23:37:38.601 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi101.front.sepia.ceph.com 2023-07-28T23:37:38.601 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-07-28T23:37:38.602 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd of=/etc/ceph/c1.keyring 2023-07-28T23:37:38.602 DEBUG:teuthology.orchestra.run.smithi101:> sudo chmod 0644 /etc/ceph/c1.keyring 2023-07-28T23:37:38.679 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-07-28T23:37:38.680 DEBUG:teuthology.orchestra.run.smithi101:> dd of=/home/ubuntu/cephtest/c1.monmap 2023-07-28T23:37:38.733 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi139.front.sepia.ceph.com 2023-07-28T23:37:38.734 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2023-07-28T23:37:38.734 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/ceph/c1.keyring 2023-07-28T23:37:38.734 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod 0644 /etc/ceph/c1.keyring 2023-07-28T23:37:38.777 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2023-07-28T23:37:38.777 DEBUG:teuthology.orchestra.run.smithi139:> dd of=/home/ubuntu/cephtest/c1.monmap 2023-07-28T23:37:38.833 INFO:tasks.ceph:Setting up mon nodes... 2023-07-28T23:37:38.833 INFO:tasks.ceph:Setting up mgr nodes... 2023-07-28T23:37:38.833 DEBUG:teuthology.orchestra.run.smithi101:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2023-07-28T23:37:38.887 INFO:teuthology.orchestra.run.smithi101.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2023-07-28T23:37:38.889 INFO:tasks.ceph:Setting up mds nodes... 2023-07-28T23:37:38.889 INFO:tasks.ceph_client:Setting up client nodes... 2023-07-28T23:37:38.890 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2023-07-28T23:37:38.930 INFO:teuthology.orchestra.run.smithi101.stdout:creating /etc/ceph/c1.client.0.keyring 2023-07-28T23:37:38.946 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2023-07-28T23:37:39.027 INFO:teuthology.orchestra.run.smithi101.stdout:creating /etc/ceph/c1.client.1.keyring 2023-07-28T23:37:39.042 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-07-28T23:37:39.043 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi101.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-07-28T23:37:39.043 DEBUG:teuthology.orchestra.run.smithi101:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2023-07-28T23:37:39.108 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-07-28T23:37:39.108 INFO:tasks.ceph:role: c1.osd.0 2023-07-28T23:37:39.109 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi101.front.sepia.ceph.com 2023-07-28T23:37:39.109 DEBUG:teuthology.orchestra.run.smithi101:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-07-28T23:37:39.181 INFO:teuthology.orchestra.run.smithi101.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-07-28T23:37:39.182 INFO:teuthology.orchestra.run.smithi101.stdout: = sectsz=512 attr=2, projid32bit=1 2023-07-28T23:37:39.182 INFO:teuthology.orchestra.run.smithi101.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-07-28T23:37:39.182 INFO:teuthology.orchestra.run.smithi101.stdout: = reflink=1 bigtime=1 inobtcount=1 2023-07-28T23:37:39.182 INFO:teuthology.orchestra.run.smithi101.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-07-28T23:37:39.182 INFO:teuthology.orchestra.run.smithi101.stdout: = sunit=0 swidth=0 blks 2023-07-28T23:37:39.182 INFO:teuthology.orchestra.run.smithi101.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-07-28T23:37:39.182 INFO:teuthology.orchestra.run.smithi101.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-07-28T23:37:39.183 INFO:teuthology.orchestra.run.smithi101.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-07-28T23:37:39.183 INFO:teuthology.orchestra.run.smithi101.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-07-28T23:37:39.660 INFO:teuthology.orchestra.run.smithi101.stdout:Discarding blocks...Done. 2023-07-28T23:37:39.662 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi101.front.sepia.ceph.com -o noatime 2023-07-28T23:37:39.662 DEBUG:teuthology.orchestra.run.smithi101:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2023-07-28T23:37:39.806 DEBUG:teuthology.orchestra.run.smithi101:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2023-07-28T23:37:39.839 DEBUG:teuthology.orchestra.run.smithi101:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2023-07-28T23:37:39.907 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-07-28T23:37:39.907 INFO:tasks.ceph:role: c1.osd.1 2023-07-28T23:37:39.908 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi101.front.sepia.ceph.com 2023-07-28T23:37:39.908 DEBUG:teuthology.orchestra.run.smithi101:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-07-28T23:37:39.981 INFO:teuthology.orchestra.run.smithi101.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-07-28T23:37:39.981 INFO:teuthology.orchestra.run.smithi101.stdout: = sectsz=512 attr=2, projid32bit=1 2023-07-28T23:37:39.982 INFO:teuthology.orchestra.run.smithi101.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-07-28T23:37:39.982 INFO:teuthology.orchestra.run.smithi101.stdout: = reflink=1 bigtime=1 inobtcount=1 2023-07-28T23:37:39.982 INFO:teuthology.orchestra.run.smithi101.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-07-28T23:37:39.982 INFO:teuthology.orchestra.run.smithi101.stdout: = sunit=0 swidth=0 blks 2023-07-28T23:37:39.982 INFO:teuthology.orchestra.run.smithi101.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-07-28T23:37:39.982 INFO:teuthology.orchestra.run.smithi101.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-07-28T23:37:39.982 INFO:teuthology.orchestra.run.smithi101.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-07-28T23:37:39.983 INFO:teuthology.orchestra.run.smithi101.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-07-28T23:37:40.419 INFO:teuthology.orchestra.run.smithi101.stdout:Discarding blocks...Done. 2023-07-28T23:37:40.421 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi101.front.sepia.ceph.com -o noatime 2023-07-28T23:37:40.422 DEBUG:teuthology.orchestra.run.smithi101:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2023-07-28T23:37:40.521 DEBUG:teuthology.orchestra.run.smithi101:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2023-07-28T23:37:40.553 DEBUG:teuthology.orchestra.run.smithi101:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2023-07-28T23:37:40.620 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-07-28T23:37:40.620 INFO:tasks.ceph:role: c1.osd.2 2023-07-28T23:37:40.621 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi101.front.sepia.ceph.com 2023-07-28T23:37:40.621 DEBUG:teuthology.orchestra.run.smithi101:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-07-28T23:37:40.694 INFO:teuthology.orchestra.run.smithi101.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-07-28T23:37:40.694 INFO:teuthology.orchestra.run.smithi101.stdout: = sectsz=512 attr=2, projid32bit=1 2023-07-28T23:37:40.695 INFO:teuthology.orchestra.run.smithi101.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-07-28T23:37:40.695 INFO:teuthology.orchestra.run.smithi101.stdout: = reflink=1 bigtime=1 inobtcount=1 2023-07-28T23:37:40.695 INFO:teuthology.orchestra.run.smithi101.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-07-28T23:37:40.695 INFO:teuthology.orchestra.run.smithi101.stdout: = sunit=0 swidth=0 blks 2023-07-28T23:37:40.695 INFO:teuthology.orchestra.run.smithi101.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-07-28T23:37:40.695 INFO:teuthology.orchestra.run.smithi101.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-07-28T23:37:40.695 INFO:teuthology.orchestra.run.smithi101.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-07-28T23:37:40.696 INFO:teuthology.orchestra.run.smithi101.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-07-28T23:37:41.159 INFO:teuthology.orchestra.run.smithi101.stdout:Discarding blocks...Done. 2023-07-28T23:37:41.161 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi101.front.sepia.ceph.com -o noatime 2023-07-28T23:37:41.161 DEBUG:teuthology.orchestra.run.smithi101:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2023-07-28T23:37:41.269 DEBUG:teuthology.orchestra.run.smithi101:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2023-07-28T23:37:41.300 DEBUG:teuthology.orchestra.run.smithi101:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2023-07-28T23:37:41.394 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:37:41.397+0000 7f69aac15500 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2023-07-28T23:37:41.396 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:37:41.399+0000 7f69aac15500 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2023-07-28T23:37:41.396 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:37:41.399+0000 7f69aac15500 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2023-07-28T23:37:41.397 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:37:41.399+0000 7f69aac15500 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2023-07-28T23:37:41.397 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:37:41.399+0000 7f69aac15500 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2023-07-28T23:37:41.403 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:37:41.405+0000 7f69aac15500 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2023-07-28T23:37:43.484 DEBUG:teuthology.orchestra.run.smithi101:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2023-07-28T23:37:43.514 DEBUG:teuthology.orchestra.run.smithi101:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2023-07-28T23:37:43.604 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:37:43.606+0000 7f8ae0811500 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2023-07-28T23:37:43.604 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:37:43.607+0000 7f8ae0811500 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2023-07-28T23:37:43.605 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:37:43.607+0000 7f8ae0811500 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2023-07-28T23:37:43.605 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:37:43.607+0000 7f8ae0811500 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2023-07-28T23:37:43.605 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:37:43.607+0000 7f8ae0811500 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2023-07-28T23:37:43.605 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:37:43.607+0000 7f8ae0811500 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2023-07-28T23:37:45.687 DEBUG:teuthology.orchestra.run.smithi101:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2023-07-28T23:37:45.719 DEBUG:teuthology.orchestra.run.smithi101:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2023-07-28T23:37:45.812 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:37:45.814+0000 7f9ee0e15500 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2023-07-28T23:37:45.812 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:37:45.814+0000 7f9ee0e15500 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2023-07-28T23:37:45.813 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:37:45.814+0000 7f9ee0e15500 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2023-07-28T23:37:45.813 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:37:45.814+0000 7f9ee0e15500 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2023-07-28T23:37:45.813 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:37:45.814+0000 7f9ee0e15500 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2023-07-28T23:37:45.813 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:37:45.815+0000 7f9ee0e15500 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2023-07-28T23:37:47.900 DEBUG:teuthology.orchestra.run.smithi101:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2023-07-28T23:37:47.930 INFO:tasks.ceph:Reading keys from all nodes... 2023-07-28T23:37:47.931 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-07-28T23:37:47.931 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2023-07-28T23:37:48.000 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-07-28T23:37:48.000 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2023-07-28T23:37:48.070 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-07-28T23:37:48.070 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2023-07-28T23:37:48.138 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-07-28T23:37:48.138 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2023-07-28T23:37:48.206 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-07-28T23:37:48.206 DEBUG:teuthology.orchestra.run.smithi101:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2023-07-28T23:37:48.261 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-07-28T23:37:48.261 DEBUG:teuthology.orchestra.run.smithi101:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2023-07-28T23:37:48.315 INFO:tasks.ceph:Adding keys to all mons... 2023-07-28T23:37:48.316 DEBUG:teuthology.orchestra.run.smithi101:> sudo tee -a /etc/ceph/c1.keyring 2023-07-28T23:37:48.381 INFO:teuthology.orchestra.run.smithi101.stdout:[mgr.x] 2023-07-28T23:37:48.381 INFO:teuthology.orchestra.run.smithi101.stdout: key = AQBCUcRkD4sVNRAATxXwetioSpmNRMYUPHjeXQ== 2023-07-28T23:37:48.382 INFO:teuthology.orchestra.run.smithi101.stdout:[osd.0] 2023-07-28T23:37:48.382 INFO:teuthology.orchestra.run.smithi101.stdout: key = AQBFUcRk3Se+FxAAG0lNg0bmO+l6D3PpNSemlQ== 2023-07-28T23:37:48.382 INFO:teuthology.orchestra.run.smithi101.stdout:[osd.1] 2023-07-28T23:37:48.382 INFO:teuthology.orchestra.run.smithi101.stdout: key = AQBHUcRkTFA6JBAAPBem66P9csWh8J9yFsJ13Q== 2023-07-28T23:37:48.382 INFO:teuthology.orchestra.run.smithi101.stdout:[osd.2] 2023-07-28T23:37:48.382 INFO:teuthology.orchestra.run.smithi101.stdout: key = AQBJUcRktXidMBAArWrG3mO0xncaMMTG3G9rrg== 2023-07-28T23:37:48.382 INFO:teuthology.orchestra.run.smithi101.stdout:[client.0] 2023-07-28T23:37:48.383 INFO:teuthology.orchestra.run.smithi101.stdout: key = AQBCUcRkmdOyNxAAS9iZQXdpHPROQxJvFpCBfQ== 2023-07-28T23:37:48.383 INFO:teuthology.orchestra.run.smithi101.stdout:[client.1] 2023-07-28T23:37:48.383 INFO:teuthology.orchestra.run.smithi101.stdout: key = AQBDUcRkSCHhARAAxIm03QXZold93/pW0jEuTQ== 2023-07-28T23:37:48.384 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-07-28T23:37:48.476 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-07-28T23:37:48.528 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-07-28T23:37:48.579 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-07-28T23:37:48.631 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-07-28T23:37:48.685 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-07-28T23:37:48.738 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-07-28T23:37:48.739 DEBUG:teuthology.orchestra.run.smithi101:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2023-07-28T23:37:48.765 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2023-07-28T23:37:48.861 DEBUG:teuthology.orchestra.run.smithi101:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2023-07-28T23:37:48.887 DEBUG:teuthology.orchestra.run.smithi101:> rm -- /home/ubuntu/cephtest/c1.monmap 2023-07-28T23:37:48.941 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2023-07-28T23:37:48.942 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2023-07-28T23:37:48.942 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2023-07-28T23:37:48.983 INFO:tasks.ceph.c1.mon.a:Started 2023-07-28T23:37:48.984 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2023-07-28T23:37:48.984 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2023-07-28T23:37:48.984 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2023-07-28T23:37:48.986 INFO:tasks.ceph.c1.mgr.x:Started 2023-07-28T23:37:48.986 INFO:tasks.ceph:Setting crush tunables to default 2023-07-28T23:37:48.987 DEBUG:teuthology.orchestra.run.smithi101:> sudo ceph --cluster c1 osd crush tunables default 2023-07-28T23:37:49.180 INFO:teuthology.orchestra.run.smithi101.stderr:adjusted tunables profile to default 2023-07-28T23:37:49.191 INFO:tasks.ceph:check_enable_crimson: False 2023-07-28T23:37:49.192 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2023-07-28T23:37:49.192 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-07-28T23:37:49.192 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2023-07-28T23:37:49.218 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-07-28T23:37:49.219 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2023-07-28T23:37:49.237 INFO:tasks.ceph.c1.mgr.x.smithi101.stderr:2023-07-28T23:37:49.239+0000 7fc75fa3f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-28T23:37:49.263 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-07-28T23:37:49.263 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2023-07-28T23:37:49.328 DEBUG:teuthology.orchestra.run.smithi101:> sudo ceph --cluster c1 osd new 4b45755c-58f9-4137-ad26-9e9af928d2b8 0 2023-07-28T23:37:49.547 INFO:teuthology.orchestra.run.smithi101.stdout:0 2023-07-28T23:37:49.558 DEBUG:teuthology.orchestra.run.smithi101:> sudo ceph --cluster c1 osd new fb572c8b-f0ec-48a4-a58b-16c2484fcb38 1 2023-07-28T23:37:49.736 INFO:teuthology.orchestra.run.smithi101.stdout:1 2023-07-28T23:37:49.747 DEBUG:teuthology.orchestra.run.smithi101:> sudo ceph --cluster c1 osd new 84e94a77-89d7-4ef9-b157-1eda2c7302a2 2 2023-07-28T23:37:49.927 INFO:teuthology.orchestra.run.smithi101.stdout:2 2023-07-28T23:37:49.939 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2023-07-28T23:37:49.939 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2023-07-28T23:37:49.941 INFO:tasks.ceph.c1.osd.0:Started 2023-07-28T23:37:49.941 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2023-07-28T23:37:49.942 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2023-07-28T23:37:49.943 INFO:tasks.ceph.c1.osd.1:Started 2023-07-28T23:37:49.944 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2023-07-28T23:37:49.944 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2023-07-28T23:37:49.945 INFO:tasks.ceph.c1.osd.2:Started 2023-07-28T23:37:49.946 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2023-07-28T23:37:50.079 INFO:tasks.ceph.c1.mgr.x.smithi101.stderr:2023-07-28T23:37:50.081+0000 7fc75fa3f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-28T23:37:50.133 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:37:50.133 INFO:teuthology.orchestra.run.smithi101.stdout:{"epoch":5,"fsid":"859e430b-4e7d-4232-8462-93980f0a2aba","created":"2023-07-28T23:37:49.094504+0000","modified":"2023-07-28T23:37:49.929599+0000","last_up_change":"0.000000","last_in_change":"2023-07-28T23:37:49.929599+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4b45755c-58f9-4137-ad26-9e9af928d2b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"fb572c8b-f0ec-48a4-a58b-16c2484fcb38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"84e94a77-89d7-4ef9-b157-1eda2c7302a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-28T23:37:50.144 INFO:tasks.ceph.ceph_manager.c1:[] 2023-07-28T23:37:50.144 INFO:tasks.ceph:Waiting for OSDs to come up 2023-07-28T23:37:50.221 INFO:tasks.ceph.c1.mgr.x.smithi101.stderr:2023-07-28T23:37:50.223+0000 7fc75fa3f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-28T23:37:50.270 INFO:tasks.ceph.c1.mgr.x.smithi101.stderr:2023-07-28T23:37:50.272+0000 7fc75fa3f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-28T23:37:50.315 INFO:tasks.ceph.c1.mgr.x.smithi101.stderr:2023-07-28T23:37:50.317+0000 7fc75fa3f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-28T23:37:50.447 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2023-07-28T23:37:50.551 INFO:tasks.ceph.c1.osd.2.smithi101.stderr:2023-07-28T23:37:50.553+0000 7fe8b9534500 -1 Falling back to public interface 2023-07-28T23:37:50.559 INFO:tasks.ceph.c1.osd.1.smithi101.stderr:2023-07-28T23:37:50.561+0000 7f278d40d500 -1 Falling back to public interface 2023-07-28T23:37:50.574 INFO:tasks.ceph.c1.osd.0.smithi101.stderr:2023-07-28T23:37:50.576+0000 7fc37aa1f500 -1 Falling back to public interface 2023-07-28T23:37:50.603 INFO:teuthology.misc.health.smithi101.stdout: 2023-07-28T23:37:50.603 INFO:teuthology.misc.health.smithi101.stdout:{"epoch":5,"fsid":"859e430b-4e7d-4232-8462-93980f0a2aba","created":"2023-07-28T23:37:49.094504+0000","modified":"2023-07-28T23:37:49.929599+0000","last_up_change":"0.000000","last_in_change":"2023-07-28T23:37:49.929599+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4b45755c-58f9-4137-ad26-9e9af928d2b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"fb572c8b-f0ec-48a4-a58b-16c2484fcb38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"84e94a77-89d7-4ef9-b157-1eda2c7302a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-28T23:37:50.612 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-07-28T23:37:51.053 INFO:tasks.ceph.c1.mgr.x.smithi101.stderr:2023-07-28T23:37:51.055+0000 7fc75fa3f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-28T23:37:51.107 INFO:tasks.ceph.c1.mgr.x.smithi101.stderr:2023-07-28T23:37:51.109+0000 7fc75fa3f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-28T23:37:51.156 INFO:tasks.ceph.c1.mgr.x.smithi101.stderr:2023-07-28T23:37:51.156+0000 7fc75fa3f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-28T23:37:51.261 INFO:tasks.ceph.c1.mgr.x.smithi101.stderr:2023-07-28T23:37:51.263+0000 7fc75fa3f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-28T23:37:51.400 INFO:tasks.ceph.c1.mgr.x.smithi101.stderr:2023-07-28T23:37:51.402+0000 7fc75fa3f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-28T23:37:51.494 INFO:tasks.ceph.c1.mgr.x.smithi101.stderr:2023-07-28T23:37:51.497+0000 7fc75fa3f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-28T23:37:51.536 INFO:tasks.ceph.c1.mgr.x.smithi101.stderr:2023-07-28T23:37:51.538+0000 7fc75fa3f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-28T23:37:51.672 INFO:tasks.ceph.c1.mgr.x.smithi101.stderr:2023-07-28T23:37:51.675+0000 7fc75fa3f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-28T23:37:52.047 INFO:tasks.ceph.c1.mgr.x.smithi101.stderr:2023-07-28T23:37:52.050+0000 7fc75fa3f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-28T23:37:52.240 INFO:tasks.ceph.c1.mgr.x.smithi101.stderr:2023-07-28T23:37:52.243+0000 7fc75fa3f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-28T23:37:52.292 INFO:tasks.ceph.c1.mgr.x.smithi101.stderr:2023-07-28T23:37:52.294+0000 7fc75fa3f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-28T23:37:52.336 INFO:tasks.ceph.c1.mgr.x.smithi101.stderr:2023-07-28T23:37:52.339+0000 7fc75fa3f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-28T23:37:52.388 INFO:tasks.ceph.c1.mgr.x.smithi101.stderr:2023-07-28T23:37:52.390+0000 7fc75fa3f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-28T23:37:52.621 INFO:tasks.ceph.c1.mgr.x.smithi101.stderr:2023-07-28T23:37:52.623+0000 7fc75fa3f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-28T23:37:52.672 INFO:tasks.ceph.c1.mgr.x.smithi101.stderr:2023-07-28T23:37:52.674+0000 7fc75fa3f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-28T23:37:52.734 INFO:tasks.ceph.c1.mgr.x.smithi101.stderr:2023-07-28T23:37:52.737+0000 7fc75fa3f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-28T23:37:52.874 INFO:tasks.ceph.c1.mgr.x.smithi101.stderr:2023-07-28T23:37:52.876+0000 7fc75fa3f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-28T23:37:52.915 INFO:tasks.ceph.c1.mgr.x.smithi101.stderr:2023-07-28T23:37:52.918+0000 7fc75fa3f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-28T23:37:52.966 INFO:tasks.ceph.c1.mgr.x.smithi101.stderr:2023-07-28T23:37:52.968+0000 7fc75fa3f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-28T23:37:53.013 INFO:tasks.ceph.c1.mgr.x.smithi101.stderr:2023-07-28T23:37:53.015+0000 7fc75fa3f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-28T23:37:53.267 INFO:tasks.ceph.c1.osd.2.smithi101.stderr:2023-07-28T23:37:53.269+0000 7fe8b9534500 -1 osd.2 0 log_to_monitors true 2023-07-28T23:37:53.272 INFO:tasks.ceph.c1.osd.1.smithi101.stderr:2023-07-28T23:37:53.274+0000 7f278d40d500 -1 osd.1 0 log_to_monitors true 2023-07-28T23:37:53.302 INFO:tasks.ceph.c1.osd.0.smithi101.stderr:2023-07-28T23:37:53.304+0000 7fc37aa1f500 -1 osd.0 0 log_to_monitors true 2023-07-28T23:37:54.033 INFO:tasks.ceph.c1.mgr.x.smithi101.stderr:2023-07-28T23:37:54.035+0000 7fc70105f640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-07-28T23:37:56.302 INFO:tasks.ceph.c1.osd.0.smithi101.stderr:2023-07-28T23:37:56.305+0000 7fc376c6d640 -1 osd.0 0 waiting for initial osdmap 2023-07-28T23:37:56.314 INFO:tasks.ceph.c1.osd.0.smithi101.stderr:2023-07-28T23:37:56.317+0000 7fc36e275640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-28T23:37:56.355 INFO:tasks.ceph.c1.osd.1.smithi101.stderr:2023-07-28T23:37:56.357+0000 7f2788dfb640 -1 osd.1 0 waiting for initial osdmap 2023-07-28T23:37:56.369 INFO:tasks.ceph.c1.osd.1.smithi101.stderr:2023-07-28T23:37:56.372+0000 7f2780c1a640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-28T23:37:56.375 INFO:tasks.ceph.c1.osd.2.smithi101.stderr:2023-07-28T23:37:56.377+0000 7fe8b54bf640 -1 osd.2 0 waiting for initial osdmap 2023-07-28T23:37:56.389 INFO:tasks.ceph.c1.osd.2.smithi101.stderr:2023-07-28T23:37:56.392+0000 7fe8ad2de640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-28T23:37:56.915 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2023-07-28T23:37:57.184 INFO:teuthology.misc.health.smithi101.stdout: 2023-07-28T23:37:57.184 INFO:teuthology.misc.health.smithi101.stdout:{"epoch":7,"fsid":"859e430b-4e7d-4232-8462-93980f0a2aba","created":"2023-07-28T23:37:49.094504+0000","modified":"2023-07-28T23:37:55.087477+0000","last_up_change":"0.000000","last_in_change":"2023-07-28T23:37:49.929599+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4b45755c-58f9-4137-ad26-9e9af928d2b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"fb572c8b-f0ec-48a4-a58b-16c2484fcb38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"84e94a77-89d7-4ef9-b157-1eda2c7302a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-28T23:37:57.194 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-07-28T23:38:03.497 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2023-07-28T23:38:03.772 INFO:teuthology.misc.health.smithi101.stdout: 2023-07-28T23:38:03.772 INFO:teuthology.misc.health.smithi101.stdout:{"epoch":11,"fsid":"859e430b-4e7d-4232-8462-93980f0a2aba","created":"2023-07-28T23:37:49.094504+0000","modified":"2023-07-28T23:38:01.132014+0000","last_up_change":"2023-07-28T23:37:57.307212+0000","last_in_change":"2023-07-28T23:37:49.929599+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-28T23:37:59.073054+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b45755c-58f9-4137-ad26-9e9af928d2b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6816","nonce":81942},{"type":"v1","addr":"172.21.15.101:6817","nonce":81942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6818","nonce":81942},{"type":"v1","addr":"172.21.15.101:6819","nonce":81942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6822","nonce":81942},{"type":"v1","addr":"172.21.15.101:6823","nonce":81942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6820","nonce":81942},{"type":"v1","addr":"172.21.15.101:6821","nonce":81942}]},"public_addr":"172.21.15.101:6817/81942","cluster_addr":"172.21.15.101:6819/81942","heartbeat_back_addr":"172.21.15.101:6823/81942","heartbeat_front_addr":"172.21.15.101:6821/81942","state":["exists","up"]},{"osd":1,"uuid":"fb572c8b-f0ec-48a4-a58b-16c2484fcb38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6808","nonce":81941},{"type":"v1","addr":"172.21.15.101:6809","nonce":81941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6810","nonce":81941},{"type":"v1","addr":"172.21.15.101:6811","nonce":81941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6814","nonce":81941},{"type":"v1","addr":"172.21.15.101:6815","nonce":81941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6812","nonce":81941},{"type":"v1","addr":"172.21.15.101:6813","nonce":81941}]},"public_addr":"172.21.15.101:6809/81941","cluster_addr":"172.21.15.101:6811/81941","heartbeat_back_addr":"172.21.15.101:6815/81941","heartbeat_front_addr":"172.21.15.101:6813/81941","state":["exists","up"]},{"osd":2,"uuid":"84e94a77-89d7-4ef9-b157-1eda2c7302a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6800","nonce":81940},{"type":"v1","addr":"172.21.15.101:6801","nonce":81940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6802","nonce":81940},{"type":"v1","addr":"172.21.15.101:6803","nonce":81940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6806","nonce":81940},{"type":"v1","addr":"172.21.15.101:6807","nonce":81940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6804","nonce":81940},{"type":"v1","addr":"172.21.15.101:6805","nonce":81940}]},"public_addr":"172.21.15.101:6801/81940","cluster_addr":"172.21.15.101:6803/81940","heartbeat_back_addr":"172.21.15.101:6807/81940","heartbeat_front_addr":"172.21.15.101:6805/81940","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-28T23:37:54.323555+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-28T23:37:54.316960+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-28T23:37:54.266721+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-28T23:38:03.783 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-07-28T23:38:03.783 INFO:tasks.ceph:Creating RBD pool 2023-07-28T23:38:03.783 DEBUG:teuthology.orchestra.run.smithi101:> sudo ceph --cluster c1 osd pool create rbd 8 2023-07-28T23:38:04.142 INFO:teuthology.orchestra.run.smithi101.stderr:pool 'rbd' created 2023-07-28T23:38:04.156 DEBUG:teuthology.orchestra.run.smithi101:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-07-28T23:38:05.146 INFO:teuthology.orchestra.run.smithi101.stderr:enabled application 'rbd' on pool 'rbd' 2023-07-28T23:38:05.163 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2023-07-28T23:38:05.163 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2023-07-28T23:38:05.164 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2023-07-28T23:38:05.164 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2023-07-28T23:38:05.164 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-07-28T23:38:05.485 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:38:05.496 INFO:teuthology.orchestra.run.smithi101.stdout:{"epoch":5,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6824","nonce":81604},{"type":"v1","addr":"172.21.15.101:6825","nonce":81604}]},"active_addr":"172.21.15.101:6825/81604","active_change":"2023-07-28T23:37:53.029912+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph:v18","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.101:0","nonce":298500629}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.101:0","nonce":3868012129}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.101:0","nonce":138455563}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.101:0","nonce":3106216944}]}]} 2023-07-28T23:38:05.498 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2023-07-28T23:38:05.499 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2023-07-28T23:38:05.499 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2023-07-28T23:38:05.774 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:38:05.774 INFO:teuthology.orchestra.run.smithi101.stdout:{"epoch":13,"fsid":"859e430b-4e7d-4232-8462-93980f0a2aba","created":"2023-07-28T23:37:49.094504+0000","modified":"2023-07-28T23:38:05.144259+0000","last_up_change":"2023-07-28T23:37:57.307212+0000","last_in_change":"2023-07-28T23:37:49.929599+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-28T23:37:59.073054+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-28T23:38:04.075878+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b45755c-58f9-4137-ad26-9e9af928d2b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6816","nonce":81942},{"type":"v1","addr":"172.21.15.101:6817","nonce":81942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6818","nonce":81942},{"type":"v1","addr":"172.21.15.101:6819","nonce":81942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6822","nonce":81942},{"type":"v1","addr":"172.21.15.101:6823","nonce":81942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6820","nonce":81942},{"type":"v1","addr":"172.21.15.101:6821","nonce":81942}]},"public_addr":"172.21.15.101:6817/81942","cluster_addr":"172.21.15.101:6819/81942","heartbeat_back_addr":"172.21.15.101:6823/81942","heartbeat_front_addr":"172.21.15.101:6821/81942","state":["exists","up"]},{"osd":1,"uuid":"fb572c8b-f0ec-48a4-a58b-16c2484fcb38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6808","nonce":81941},{"type":"v1","addr":"172.21.15.101:6809","nonce":81941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6810","nonce":81941},{"type":"v1","addr":"172.21.15.101:6811","nonce":81941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6814","nonce":81941},{"type":"v1","addr":"172.21.15.101:6815","nonce":81941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6812","nonce":81941},{"type":"v1","addr":"172.21.15.101:6813","nonce":81941}]},"public_addr":"172.21.15.101:6809/81941","cluster_addr":"172.21.15.101:6811/81941","heartbeat_back_addr":"172.21.15.101:6815/81941","heartbeat_front_addr":"172.21.15.101:6813/81941","state":["exists","up"]},{"osd":2,"uuid":"84e94a77-89d7-4ef9-b157-1eda2c7302a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6800","nonce":81940},{"type":"v1","addr":"172.21.15.101:6801","nonce":81940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6802","nonce":81940},{"type":"v1","addr":"172.21.15.101:6803","nonce":81940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6806","nonce":81940},{"type":"v1","addr":"172.21.15.101:6807","nonce":81940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6804","nonce":81940},{"type":"v1","addr":"172.21.15.101:6805","nonce":81940}]},"public_addr":"172.21.15.101:6801/81940","cluster_addr":"172.21.15.101:6803/81940","heartbeat_back_addr":"172.21.15.101:6807/81940","heartbeat_front_addr":"172.21.15.101:6805/81940","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-28T23:37:54.323555+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-28T23:37:54.316960+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-28T23:37:54.266721+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-28T23:38:05.786 INFO:tasks.ceph.ceph_manager.c1:all up! 2023-07-28T23:38:05.786 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2023-07-28T23:38:06.058 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:38:06.058 INFO:teuthology.orchestra.run.smithi101.stdout:{"epoch":13,"fsid":"859e430b-4e7d-4232-8462-93980f0a2aba","created":"2023-07-28T23:37:49.094504+0000","modified":"2023-07-28T23:38:05.144259+0000","last_up_change":"2023-07-28T23:37:57.307212+0000","last_in_change":"2023-07-28T23:37:49.929599+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-28T23:37:59.073054+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-28T23:38:04.075878+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b45755c-58f9-4137-ad26-9e9af928d2b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6816","nonce":81942},{"type":"v1","addr":"172.21.15.101:6817","nonce":81942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6818","nonce":81942},{"type":"v1","addr":"172.21.15.101:6819","nonce":81942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6822","nonce":81942},{"type":"v1","addr":"172.21.15.101:6823","nonce":81942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6820","nonce":81942},{"type":"v1","addr":"172.21.15.101:6821","nonce":81942}]},"public_addr":"172.21.15.101:6817/81942","cluster_addr":"172.21.15.101:6819/81942","heartbeat_back_addr":"172.21.15.101:6823/81942","heartbeat_front_addr":"172.21.15.101:6821/81942","state":["exists","up"]},{"osd":1,"uuid":"fb572c8b-f0ec-48a4-a58b-16c2484fcb38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6808","nonce":81941},{"type":"v1","addr":"172.21.15.101:6809","nonce":81941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6810","nonce":81941},{"type":"v1","addr":"172.21.15.101:6811","nonce":81941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6814","nonce":81941},{"type":"v1","addr":"172.21.15.101:6815","nonce":81941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6812","nonce":81941},{"type":"v1","addr":"172.21.15.101:6813","nonce":81941}]},"public_addr":"172.21.15.101:6809/81941","cluster_addr":"172.21.15.101:6811/81941","heartbeat_back_addr":"172.21.15.101:6815/81941","heartbeat_front_addr":"172.21.15.101:6813/81941","state":["exists","up"]},{"osd":2,"uuid":"84e94a77-89d7-4ef9-b157-1eda2c7302a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6800","nonce":81940},{"type":"v1","addr":"172.21.15.101:6801","nonce":81940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6802","nonce":81940},{"type":"v1","addr":"172.21.15.101:6803","nonce":81940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6806","nonce":81940},{"type":"v1","addr":"172.21.15.101:6807","nonce":81940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6804","nonce":81940},{"type":"v1","addr":"172.21.15.101:6805","nonce":81940}]},"public_addr":"172.21.15.101:6801/81940","cluster_addr":"172.21.15.101:6803/81940","heartbeat_back_addr":"172.21.15.101:6807/81940","heartbeat_front_addr":"172.21.15.101:6805/81940","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-28T23:37:54.323555+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-28T23:37:54.316960+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-28T23:37:54.266721+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-28T23:38:06.070 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2023-07-28T23:38:06.070 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2023-07-28T23:38:06.071 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2023-07-28T23:38:06.207 INFO:teuthology.orchestra.run.smithi101.stdout:34359738372 2023-07-28T23:38:06.207 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2023-07-28T23:38:06.226 INFO:teuthology.orchestra.run.smithi101.stdout:34359738372 2023-07-28T23:38:06.226 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2023-07-28T23:38:06.227 INFO:teuthology.orchestra.run.smithi101.stdout:34359738373 2023-07-28T23:38:06.227 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2023-07-28T23:38:06.508 INFO:teuthology.orchestra.run.smithi101.stdout:34359738371 2023-07-28T23:38:06.521 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738371 for osd.1 2023-07-28T23:38:06.556 INFO:teuthology.orchestra.run.smithi101.stdout:34359738371 2023-07-28T23:38:06.565 INFO:teuthology.orchestra.run.smithi101.stdout:34359738372 2023-07-28T23:38:06.569 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738371 for osd.0 2023-07-28T23:38:06.577 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738373 got 34359738372 for osd.2 2023-07-28T23:38:07.521 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2023-07-28T23:38:07.569 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2023-07-28T23:38:07.578 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2023-07-28T23:38:07.873 INFO:teuthology.orchestra.run.smithi101.stdout:34359738372 2023-07-28T23:38:07.886 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738372 for osd.1 2023-07-28T23:38:07.886 DEBUG:teuthology.parallel:result is None 2023-07-28T23:38:07.897 INFO:teuthology.orchestra.run.smithi101.stdout:34359738373 2023-07-28T23:38:07.909 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738373 got 34359738373 for osd.2 2023-07-28T23:38:07.909 DEBUG:teuthology.parallel:result is None 2023-07-28T23:38:07.917 INFO:teuthology.orchestra.run.smithi101.stdout:34359738372 2023-07-28T23:38:07.929 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738372 for osd.0 2023-07-28T23:38:07.930 DEBUG:teuthology.parallel:result is None 2023-07-28T23:38:07.930 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2023-07-28T23:38:07.930 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2023-07-28T23:38:08.240 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:38:08.240 INFO:teuthology.orchestra.run.smithi101.stderr:dumped all 2023-07-28T23:38:08.252 INFO:teuthology.orchestra.run.smithi101.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-07-28T23:38:07.035510+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82352,"kb_used_data":1496,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490448,"statfs":{"total":322122547200,"available":322038218752,"internally_reserved":0,"allocated":1531904,"data_stored":1274693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.890713"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-07-28T23:38:06.152285+0000","last_change":"2023-07-28T23:38:05.153649+0000","last_active":"2023-07-28T23:38:06.152285+0000","last_peered":"2023-07-28T23:38:06.152285+0000","last_clean":"2023-07-28T23:38:06.152285+0000","last_became_active":"2023-07-28T23:38:05.153417+0000","last_became_peered":"2023-07-28T23:38:05.153417+0000","last_unstale":"2023-07-28T23:38:06.152285+0000","last_undegraded":"2023-07-28T23:38:06.152285+0000","last_fullsized":"2023-07-28T23:38:06.152285+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-28T23:38:04.140439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-28T23:38:04.140439+0000","last_clean_scrub_stamp":"2023-07-28T23:38:04.140439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T03:41:07.146038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-07-28T23:38:06.152277+0000","last_change":"2023-07-28T23:38:05.154621+0000","last_active":"2023-07-28T23:38:06.152277+0000","last_peered":"2023-07-28T23:38:06.152277+0000","last_clean":"2023-07-28T23:38:06.152277+0000","last_became_active":"2023-07-28T23:38:05.154344+0000","last_became_peered":"2023-07-28T23:38:05.154344+0000","last_unstale":"2023-07-28T23:38:06.152277+0000","last_undegraded":"2023-07-28T23:38:06.152277+0000","last_fullsized":"2023-07-28T23:38:06.152277+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-28T23:38:04.140439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-28T23:38:04.140439+0000","last_clean_scrub_stamp":"2023-07-28T23:38:04.140439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T10:37:11.264576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-07-28T23:38:06.152242+0000","last_change":"2023-07-28T23:38:05.155068+0000","last_active":"2023-07-28T23:38:06.152242+0000","last_peered":"2023-07-28T23:38:06.152242+0000","last_clean":"2023-07-28T23:38:06.152242+0000","last_became_active":"2023-07-28T23:38:05.154823+0000","last_became_peered":"2023-07-28T23:38:05.154823+0000","last_unstale":"2023-07-28T23:38:06.152242+0000","last_undegraded":"2023-07-28T23:38:06.152242+0000","last_fullsized":"2023-07-28T23:38:06.152242+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-28T23:38:04.140439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-28T23:38:04.140439+0000","last_clean_scrub_stamp":"2023-07-28T23:38:04.140439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T07:24:16.940305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-07-28T23:38:06.152177+0000","last_change":"2023-07-28T23:38:05.154651+0000","last_active":"2023-07-28T23:38:06.152177+0000","last_peered":"2023-07-28T23:38:06.152177+0000","last_clean":"2023-07-28T23:38:06.152177+0000","last_became_active":"2023-07-28T23:38:05.154418+0000","last_became_peered":"2023-07-28T23:38:05.154418+0000","last_unstale":"2023-07-28T23:38:06.152177+0000","last_undegraded":"2023-07-28T23:38:06.152177+0000","last_fullsized":"2023-07-28T23:38:06.152177+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-28T23:38:04.140439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-28T23:38:04.140439+0000","last_clean_scrub_stamp":"2023-07-28T23:38:04.140439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T07:40:30.624383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-07-28T23:38:05.155195+0000","last_change":"2023-07-28T23:38:05.154348+0000","last_active":"2023-07-28T23:38:05.155195+0000","last_peered":"2023-07-28T23:38:05.155195+0000","last_clean":"2023-07-28T23:38:05.155195+0000","last_became_active":"2023-07-28T23:38:05.154104+0000","last_became_peered":"2023-07-28T23:38:05.154104+0000","last_unstale":"2023-07-28T23:38:05.155195+0000","last_undegraded":"2023-07-28T23:38:05.155195+0000","last_fullsized":"2023-07-28T23:38:05.155195+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-28T23:38:04.140439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-28T23:38:04.140439+0000","last_clean_scrub_stamp":"2023-07-28T23:38:04.140439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T08:45:19.242661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-07-28T23:38:05.154278+0000","last_change":"2023-07-28T23:38:05.152805+0000","last_active":"2023-07-28T23:38:05.154278+0000","last_peered":"2023-07-28T23:38:05.154278+0000","last_clean":"2023-07-28T23:38:05.154278+0000","last_became_active":"2023-07-28T23:38:05.152452+0000","last_became_peered":"2023-07-28T23:38:05.152452+0000","last_unstale":"2023-07-28T23:38:05.154278+0000","last_undegraded":"2023-07-28T23:38:05.154278+0000","last_fullsized":"2023-07-28T23:38:05.154278+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-28T23:38:04.140439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-28T23:38:04.140439+0000","last_clean_scrub_stamp":"2023-07-28T23:38:04.140439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T11:25:59.627430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-07-28T23:38:05.154241+0000","last_change":"2023-07-28T23:38:05.153042+0000","last_active":"2023-07-28T23:38:05.154241+0000","last_peered":"2023-07-28T23:38:05.154241+0000","last_clean":"2023-07-28T23:38:05.154241+0000","last_became_active":"2023-07-28T23:38:05.152769+0000","last_became_peered":"2023-07-28T23:38:05.152769+0000","last_unstale":"2023-07-28T23:38:05.154241+0000","last_undegraded":"2023-07-28T23:38:05.154241+0000","last_fullsized":"2023-07-28T23:38:05.154241+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-28T23:38:04.140439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-28T23:38:04.140439+0000","last_clean_scrub_stamp":"2023-07-28T23:38:04.140439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T01:23:21.052105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-07-28T23:38:06.152167+0000","last_change":"2023-07-28T23:38:05.152523+0000","last_active":"2023-07-28T23:38:06.152167+0000","last_peered":"2023-07-28T23:38:06.152167+0000","last_clean":"2023-07-28T23:38:06.152167+0000","last_became_active":"2023-07-28T23:38:05.152286+0000","last_became_peered":"2023-07-28T23:38:05.152286+0000","last_unstale":"2023-07-28T23:38:06.152167+0000","last_undegraded":"2023-07-28T23:38:06.152167+0000","last_fullsized":"2023-07-28T23:38:06.152167+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-28T23:38:04.140439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-28T23:38:04.140439+0000","last_clean_scrub_stamp":"2023-07-28T23:38:04.140439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T03:52:23.312658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":140,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-07-28T23:38:05.148931+0000","last_change":"2023-07-28T23:38:00.138049+0000","last_active":"2023-07-28T23:38:05.148931+0000","last_peered":"2023-07-28T23:38:05.148931+0000","last_clean":"2023-07-28T23:38:05.148931+0000","last_became_active":"2023-07-28T23:38:00.137733+0000","last_became_peered":"2023-07-28T23:38:00.137733+0000","last_unstale":"2023-07-28T23:38:05.148931+0000","last_undegraded":"2023-07-28T23:38:05.148931+0000","last_fullsized":"2023-07-28T23:38:05.148931+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-28T23:37:59.125090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-28T23:37:59.125090+0000","last_clean_scrub_stamp":"2023-07-28T23:37:59.125090+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T03:28:10.735033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27064,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830536,"statfs":{"total":107374182400,"available":107346468864,"internally_reserved":0,"allocated":114688,"data_stored":31319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.252}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27644,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829956,"statfs":{"total":107374182400,"available":107345874944,"internally_reserved":0,"allocated":708608,"data_stored":621687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27644,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829956,"statfs":{"total":107374182400,"available":107345874944,"internally_reserved":0,"allocated":708608,"data_stored":621687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-28T23:38:08.253 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2023-07-28T23:38:08.521 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:38:08.522 INFO:teuthology.orchestra.run.smithi101.stderr:dumped all 2023-07-28T23:38:08.533 INFO:teuthology.orchestra.run.smithi101.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-07-28T23:38:07.035510+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82352,"kb_used_data":1496,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490448,"statfs":{"total":322122547200,"available":322038218752,"internally_reserved":0,"allocated":1531904,"data_stored":1274693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.890713"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-07-28T23:38:06.152285+0000","last_change":"2023-07-28T23:38:05.153649+0000","last_active":"2023-07-28T23:38:06.152285+0000","last_peered":"2023-07-28T23:38:06.152285+0000","last_clean":"2023-07-28T23:38:06.152285+0000","last_became_active":"2023-07-28T23:38:05.153417+0000","last_became_peered":"2023-07-28T23:38:05.153417+0000","last_unstale":"2023-07-28T23:38:06.152285+0000","last_undegraded":"2023-07-28T23:38:06.152285+0000","last_fullsized":"2023-07-28T23:38:06.152285+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-28T23:38:04.140439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-28T23:38:04.140439+0000","last_clean_scrub_stamp":"2023-07-28T23:38:04.140439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T03:41:07.146038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-07-28T23:38:06.152277+0000","last_change":"2023-07-28T23:38:05.154621+0000","last_active":"2023-07-28T23:38:06.152277+0000","last_peered":"2023-07-28T23:38:06.152277+0000","last_clean":"2023-07-28T23:38:06.152277+0000","last_became_active":"2023-07-28T23:38:05.154344+0000","last_became_peered":"2023-07-28T23:38:05.154344+0000","last_unstale":"2023-07-28T23:38:06.152277+0000","last_undegraded":"2023-07-28T23:38:06.152277+0000","last_fullsized":"2023-07-28T23:38:06.152277+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-28T23:38:04.140439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-28T23:38:04.140439+0000","last_clean_scrub_stamp":"2023-07-28T23:38:04.140439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T10:37:11.264576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-07-28T23:38:06.152242+0000","last_change":"2023-07-28T23:38:05.155068+0000","last_active":"2023-07-28T23:38:06.152242+0000","last_peered":"2023-07-28T23:38:06.152242+0000","last_clean":"2023-07-28T23:38:06.152242+0000","last_became_active":"2023-07-28T23:38:05.154823+0000","last_became_peered":"2023-07-28T23:38:05.154823+0000","last_unstale":"2023-07-28T23:38:06.152242+0000","last_undegraded":"2023-07-28T23:38:06.152242+0000","last_fullsized":"2023-07-28T23:38:06.152242+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-28T23:38:04.140439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-28T23:38:04.140439+0000","last_clean_scrub_stamp":"2023-07-28T23:38:04.140439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T07:24:16.940305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-07-28T23:38:06.152177+0000","last_change":"2023-07-28T23:38:05.154651+0000","last_active":"2023-07-28T23:38:06.152177+0000","last_peered":"2023-07-28T23:38:06.152177+0000","last_clean":"2023-07-28T23:38:06.152177+0000","last_became_active":"2023-07-28T23:38:05.154418+0000","last_became_peered":"2023-07-28T23:38:05.154418+0000","last_unstale":"2023-07-28T23:38:06.152177+0000","last_undegraded":"2023-07-28T23:38:06.152177+0000","last_fullsized":"2023-07-28T23:38:06.152177+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-28T23:38:04.140439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-28T23:38:04.140439+0000","last_clean_scrub_stamp":"2023-07-28T23:38:04.140439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T07:40:30.624383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-07-28T23:38:05.155195+0000","last_change":"2023-07-28T23:38:05.154348+0000","last_active":"2023-07-28T23:38:05.155195+0000","last_peered":"2023-07-28T23:38:05.155195+0000","last_clean":"2023-07-28T23:38:05.155195+0000","last_became_active":"2023-07-28T23:38:05.154104+0000","last_became_peered":"2023-07-28T23:38:05.154104+0000","last_unstale":"2023-07-28T23:38:05.155195+0000","last_undegraded":"2023-07-28T23:38:05.155195+0000","last_fullsized":"2023-07-28T23:38:05.155195+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-28T23:38:04.140439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-28T23:38:04.140439+0000","last_clean_scrub_stamp":"2023-07-28T23:38:04.140439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T08:45:19.242661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-07-28T23:38:05.154278+0000","last_change":"2023-07-28T23:38:05.152805+0000","last_active":"2023-07-28T23:38:05.154278+0000","last_peered":"2023-07-28T23:38:05.154278+0000","last_clean":"2023-07-28T23:38:05.154278+0000","last_became_active":"2023-07-28T23:38:05.152452+0000","last_became_peered":"2023-07-28T23:38:05.152452+0000","last_unstale":"2023-07-28T23:38:05.154278+0000","last_undegraded":"2023-07-28T23:38:05.154278+0000","last_fullsized":"2023-07-28T23:38:05.154278+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-28T23:38:04.140439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-28T23:38:04.140439+0000","last_clean_scrub_stamp":"2023-07-28T23:38:04.140439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T11:25:59.627430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-07-28T23:38:05.154241+0000","last_change":"2023-07-28T23:38:05.153042+0000","last_active":"2023-07-28T23:38:05.154241+0000","last_peered":"2023-07-28T23:38:05.154241+0000","last_clean":"2023-07-28T23:38:05.154241+0000","last_became_active":"2023-07-28T23:38:05.152769+0000","last_became_peered":"2023-07-28T23:38:05.152769+0000","last_unstale":"2023-07-28T23:38:05.154241+0000","last_undegraded":"2023-07-28T23:38:05.154241+0000","last_fullsized":"2023-07-28T23:38:05.154241+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-28T23:38:04.140439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-28T23:38:04.140439+0000","last_clean_scrub_stamp":"2023-07-28T23:38:04.140439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T01:23:21.052105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-07-28T23:38:06.152167+0000","last_change":"2023-07-28T23:38:05.152523+0000","last_active":"2023-07-28T23:38:06.152167+0000","last_peered":"2023-07-28T23:38:06.152167+0000","last_clean":"2023-07-28T23:38:06.152167+0000","last_became_active":"2023-07-28T23:38:05.152286+0000","last_became_peered":"2023-07-28T23:38:05.152286+0000","last_unstale":"2023-07-28T23:38:06.152167+0000","last_undegraded":"2023-07-28T23:38:06.152167+0000","last_fullsized":"2023-07-28T23:38:06.152167+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-28T23:38:04.140439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-28T23:38:04.140439+0000","last_clean_scrub_stamp":"2023-07-28T23:38:04.140439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T03:52:23.312658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":140,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-07-28T23:38:05.148931+0000","last_change":"2023-07-28T23:38:00.138049+0000","last_active":"2023-07-28T23:38:05.148931+0000","last_peered":"2023-07-28T23:38:05.148931+0000","last_clean":"2023-07-28T23:38:05.148931+0000","last_became_active":"2023-07-28T23:38:00.137733+0000","last_became_peered":"2023-07-28T23:38:00.137733+0000","last_unstale":"2023-07-28T23:38:05.148931+0000","last_undegraded":"2023-07-28T23:38:05.148931+0000","last_fullsized":"2023-07-28T23:38:05.148931+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-28T23:37:59.125090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-28T23:37:59.125090+0000","last_clean_scrub_stamp":"2023-07-28T23:37:59.125090+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T03:28:10.735033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27064,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830536,"statfs":{"total":107374182400,"available":107346468864,"internally_reserved":0,"allocated":114688,"data_stored":31319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.252}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27644,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829956,"statfs":{"total":107374182400,"available":107345874944,"internally_reserved":0,"allocated":708608,"data_stored":621687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27644,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829956,"statfs":{"total":107374182400,"available":107345874944,"internally_reserved":0,"allocated":708608,"data_stored":621687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-28T23:38:08.534 INFO:tasks.ceph.ceph_manager.c1:clean! 2023-07-28T23:38:08.534 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2023-07-28T23:38:08.535 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2023-07-28T23:38:08.535 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2023-07-28T23:38:08.839 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:38:08.839 INFO:teuthology.orchestra.run.smithi101.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-07-28T23:38:08.851 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2023-07-28T23:38:08.851 INFO:teuthology.run_tasks:Running task ceph... 2023-07-28T23:38:08.860 INFO:tasks.ceph:Creating ceph cluster c2... 2023-07-28T23:38:08.860 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-07-28T23:38:08.860 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-07-28_23:12:07-rgw-reef-release-distro-default-smithi/7355843', 'branch': 'reef-release', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_latest} tasks/test_multi}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7355843', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-07-28_23:12:07-rgw-reef-release-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'reef-release'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '0484097b0572e96f42cf6402cbe0ac8dcb046577', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '0484097b0572e96f42cf6402cbe0ac8dcb046577'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a1'}, {'endpoints': ['c1.client.1'], 'name': 'a2'}]}, {'enabled_features': ['resharding'], 'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'b', 'zones': [{'endpoints': ['c2.client.0'], 'is_default': True, 'is_master': True, 'name': 'b1'}, {'endpoints': ['c2.client.1'], 'name': 'b2'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'workunit': {'branch': 'reef-release', 'sha1': '0484097b0572e96f42cf6402cbe0ac8dcb046577'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 2147, 'sha1': '0484097b0572e96f42cf6402cbe0ac8dcb046577', 'sleep_before_teardown': 0, 'subset': '111/12000', 'suite': 'rgw', 'suite_branch': 'reef-release', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_0484097b0572e96f42cf6402cbe0ac8dcb046577/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '0484097b0572e96f42cf6402cbe0ac8dcb046577', 'targets': {'smithi101.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDciPeiA+F8OU2n9arsBRdxjittPCt9x6Dl/s+Hfxtf1oRgJdXHBnQWafNP5NjVmYg7hnf5zZnMl1HsH2pMMc+8=', 'smithi139.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGcp7KnNK6sxlDspWEyXozyB+lgk0NC9jK/uC8FVE4JLB64hFISTyFr+uS0NN4bYfyCUHn21kkjENMNOmGNcFI4='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '0484097b0572e96f42cf6402cbe0ac8dcb046577', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '0484097b0572e96f42cf6402cbe0ac8dcb046577', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '61140164d2579c30ffff3970615e0f82e40c6470', 'timestamp': '2023-07-28_23:12:07', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.23459'} 2023-07-28T23:38:08.861 DEBUG:teuthology.orchestra.run.smithi101:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2023-07-28T23:38:08.863 DEBUG:teuthology.orchestra.run.smithi139:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2023-07-28T23:38:08.881 DEBUG:teuthology.orchestra.run.smithi101:> sudo install -d -m0777 -- /var/run/ceph 2023-07-28T23:38:08.918 DEBUG:teuthology.orchestra.run.smithi139:> sudo install -d -m0777 -- /var/run/ceph 2023-07-28T23:38:08.952 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2023-07-28T23:38:08.952 DEBUG:teuthology.orchestra.run.smithi139:> dd if=/scratch_devs of=/dev/stdout 2023-07-28T23:38:09.010 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-28T23:38:09.010 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_1 2023-07-28T23:38:09.066 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-28T23:38:09.067 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-28T23:38:09.067 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 5h/5d Inode: 653 Links: 1 2023-07-28T23:38:09.067 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-28T23:38:09.067 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2023-07-28T23:38:09.067 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2023-07-28 23:36:43.560323233 +0000 2023-07-28T23:38:09.068 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2023-07-28 23:32:41.275089520 +0000 2023-07-28T23:38:09.068 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2023-07-28 23:32:41.275089520 +0000 2023-07-28T23:38:09.068 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: 2023-07-28 23:32:41.275089520 +0000 2023-07-28T23:38:09.068 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-28T23:38:09.134 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2023-07-28T23:38:09.134 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2023-07-28T23:38:09.135 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000217577 s, 2.4 MB/s 2023-07-28T23:38:09.135 DEBUG:teuthology.orchestra.run.smithi139:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-28T23:38:09.193 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_2 2023-07-28T23:38:09.248 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-28T23:38:09.248 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-28T23:38:09.249 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 5h/5d Inode: 672 Links: 1 2023-07-28T23:38:09.249 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-28T23:38:09.249 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2023-07-28T23:38:09.249 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2023-07-28 23:36:43.561323212 +0000 2023-07-28T23:38:09.249 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2023-07-28 23:32:41.770079045 +0000 2023-07-28T23:38:09.249 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2023-07-28 23:32:41.770079045 +0000 2023-07-28T23:38:09.249 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: 2023-07-28 23:32:41.770079045 +0000 2023-07-28T23:38:09.250 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-28T23:38:09.314 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2023-07-28T23:38:09.315 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2023-07-28T23:38:09.315 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000185594 s, 2.8 MB/s 2023-07-28T23:38:09.316 DEBUG:teuthology.orchestra.run.smithi139:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-28T23:38:09.370 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_3 2023-07-28T23:38:09.425 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-28T23:38:09.425 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-28T23:38:09.425 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 5h/5d Inode: 691 Links: 1 2023-07-28T23:38:09.425 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-28T23:38:09.426 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2023-07-28T23:38:09.426 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2023-07-28 23:36:43.561323212 +0000 2023-07-28T23:38:09.426 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2023-07-28 23:32:42.280068254 +0000 2023-07-28T23:38:09.426 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2023-07-28 23:32:42.280068254 +0000 2023-07-28T23:38:09.426 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: 2023-07-28 23:32:42.280068254 +0000 2023-07-28T23:38:09.426 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-28T23:38:09.490 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2023-07-28T23:38:09.491 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2023-07-28T23:38:09.491 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000226725 s, 2.3 MB/s 2023-07-28T23:38:09.492 DEBUG:teuthology.orchestra.run.smithi139:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-28T23:38:09.546 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_4 2023-07-28T23:38:09.600 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-28T23:38:09.601 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-28T23:38:09.601 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 5h/5d Inode: 710 Links: 1 2023-07-28T23:38:09.601 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-28T23:38:09.601 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2023-07-28T23:38:09.601 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2023-07-28 23:36:43.562323191 +0000 2023-07-28T23:38:09.601 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2023-07-28 23:32:42.765057991 +0000 2023-07-28T23:38:09.601 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2023-07-28 23:32:42.765057991 +0000 2023-07-28T23:38:09.602 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: 2023-07-28 23:32:42.765057991 +0000 2023-07-28T23:38:09.602 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-28T23:38:09.665 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2023-07-28T23:38:09.665 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2023-07-28T23:38:09.665 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000191103 s, 2.7 MB/s 2023-07-28T23:38:09.666 DEBUG:teuthology.orchestra.run.smithi139:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-28T23:38:09.721 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-07-28T23:38:09.722 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi139.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-07-28T23:38:09.722 INFO:tasks.ceph:Generating config... 2023-07-28T23:38:09.723 INFO:tasks.ceph:[client] debug rgw = 20 2023-07-28T23:38:09.723 INFO:tasks.ceph:[client] debug rgw sync = 20 2023-07-28T23:38:09.723 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2023-07-28T23:38:09.724 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2023-07-28T23:38:09.724 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-07-28T23:38:09.724 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2023-07-28T23:38:09.724 INFO:tasks.ceph:[client] rgw data log num shards = 4 2023-07-28T23:38:09.724 INFO:tasks.ceph:[client] rgw md log max shards = 4 2023-07-28T23:38:09.725 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2023-07-28T23:38:09.725 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2023-07-28T23:38:09.725 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2023-07-28T23:38:09.725 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2023-07-28T23:38:09.725 INFO:tasks.ceph:[client] setgroup = ceph 2023-07-28T23:38:09.725 INFO:tasks.ceph:[client] setuser = ceph 2023-07-28T23:38:09.726 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2023-07-28T23:38:09.726 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-07-28T23:38:09.726 INFO:tasks.ceph:[mgr] debug ms = 1 2023-07-28T23:38:09.726 INFO:tasks.ceph:[mon] debug mon = 20 2023-07-28T23:38:09.726 INFO:tasks.ceph:[mon] debug ms = 1 2023-07-28T23:38:09.727 INFO:tasks.ceph:[mon] debug paxos = 20 2023-07-28T23:38:09.727 INFO:tasks.ceph:[osd] debug ms = 1 2023-07-28T23:38:09.727 INFO:tasks.ceph:[osd] debug osd = 20 2023-07-28T23:38:09.727 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2023-07-28T23:38:09.727 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2023-07-28T23:38:09.728 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2023-07-28T23:38:09.728 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2023-07-28T23:38:09.728 INFO:tasks.ceph:Setting up c2.mon.a... 2023-07-28T23:38:09.728 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2023-07-28T23:38:09.819 INFO:teuthology.orchestra.run.smithi139.stdout:creating /etc/ceph/c2.keyring 2023-07-28T23:38:09.821 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2023-07-28T23:38:09.864 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod 0644 /etc/ceph/c2.keyring 2023-07-28T23:38:09.929 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.139')] 2023-07-28T23:38:09.930 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.139'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2023-07-28T23:38:09.931 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2023-07-28T23:38:09.931 DEBUG:teuthology.orchestra.run.smithi139:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-07-28T23:38:09.983 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.139 --print /home/ubuntu/cephtest/c2.monmap 2023-07-28T23:38:10.061 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:38:10.061 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:38:10.062 INFO:teuthology.orchestra.run.smithi139.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2023-07-28T23:38:10.062 INFO:teuthology.orchestra.run.smithi139.stdout:monmaptool: generated fsid 5f79b2fe-1924-457c-a2b5-ff6dcac41f5f 2023-07-28T23:38:10.062 INFO:teuthology.orchestra.run.smithi139.stdout:setting min_mon_release = pacific 2023-07-28T23:38:10.062 INFO:teuthology.orchestra.run.smithi139.stdout:epoch 0 2023-07-28T23:38:10.062 INFO:teuthology.orchestra.run.smithi139.stdout:fsid 5f79b2fe-1924-457c-a2b5-ff6dcac41f5f 2023-07-28T23:38:10.063 INFO:teuthology.orchestra.run.smithi139.stdout:last_changed 2023-07-28T23:38:10.064985+0000 2023-07-28T23:38:10.063 INFO:teuthology.orchestra.run.smithi139.stdout:created 2023-07-28T23:38:10.064985+0000 2023-07-28T23:38:10.063 INFO:teuthology.orchestra.run.smithi139.stdout:min_mon_release 16 (pacific) 2023-07-28T23:38:10.063 INFO:teuthology.orchestra.run.smithi139.stdout:election_strategy: 1 2023-07-28T23:38:10.063 INFO:teuthology.orchestra.run.smithi139.stdout:0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.a 2023-07-28T23:38:10.063 INFO:teuthology.orchestra.run.smithi139.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2023-07-28T23:38:10.064 DEBUG:teuthology.orchestra.run.smithi139:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-07-28T23:38:10.118 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 5f79b2fe-1924-457c-a2b5-ff6dcac41f5f... 2023-07-28T23:38:10.119 DEBUG:teuthology.orchestra.run.smithi101:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2023-07-28T23:38:10.122 DEBUG:teuthology.orchestra.run.smithi139:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2023-07-28T23:38:10.174 INFO:teuthology.orchestra.run.smithi101.stdout:[global] 2023-07-28T23:38:10.175 INFO:teuthology.orchestra.run.smithi101.stdout: chdir = "" 2023-07-28T23:38:10.175 INFO:teuthology.orchestra.run.smithi101.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-07-28T23:38:10.175 INFO:teuthology.orchestra.run.smithi101.stdout: auth supported = cephx 2023-07-28T23:38:10.175 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:38:10.175 INFO:teuthology.orchestra.run.smithi101.stdout: filestore xattr use omap = true 2023-07-28T23:38:10.175 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:38:10.176 INFO:teuthology.orchestra.run.smithi101.stdout: mon clock drift allowed = 1.000 2023-07-28T23:38:10.176 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:38:10.176 INFO:teuthology.orchestra.run.smithi101.stdout: osd crush chooseleaf type = 0 2023-07-28T23:38:10.176 INFO:teuthology.orchestra.run.smithi101.stdout: auth debug = true 2023-07-28T23:38:10.176 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:38:10.176 INFO:teuthology.orchestra.run.smithi101.stdout: ms die on old message = true 2023-07-28T23:38:10.176 INFO:teuthology.orchestra.run.smithi101.stdout: ms die on bug = true 2023-07-28T23:38:10.176 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:38:10.177 INFO:teuthology.orchestra.run.smithi101.stdout: mon max pg per osd = 10000 # >= luminous 2023-07-28T23:38:10.177 INFO:teuthology.orchestra.run.smithi101.stdout: mon pg warn max object skew = 0 2023-07-28T23:38:10.177 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:38:10.177 INFO:teuthology.orchestra.run.smithi101.stdout: # disable pg_autoscaler by default for new pools 2023-07-28T23:38:10.177 INFO:teuthology.orchestra.run.smithi101.stdout: osd_pool_default_pg_autoscale_mode = off 2023-07-28T23:38:10.177 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:38:10.177 INFO:teuthology.orchestra.run.smithi101.stdout: osd pool default size = 2 2023-07-28T23:38:10.177 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:38:10.178 INFO:teuthology.orchestra.run.smithi101.stdout: mon osd allow primary affinity = true 2023-07-28T23:38:10.178 INFO:teuthology.orchestra.run.smithi101.stdout: mon osd allow pg remap = true 2023-07-28T23:38:10.178 INFO:teuthology.orchestra.run.smithi101.stdout: mon warn on legacy crush tunables = false 2023-07-28T23:38:10.178 INFO:teuthology.orchestra.run.smithi101.stdout: mon warn on crush straw calc version zero = false 2023-07-28T23:38:10.178 INFO:teuthology.orchestra.run.smithi101.stdout: mon warn on no sortbitwise = false 2023-07-28T23:38:10.178 INFO:teuthology.orchestra.run.smithi101.stdout: mon warn on osd down out interval zero = false 2023-07-28T23:38:10.178 INFO:teuthology.orchestra.run.smithi101.stdout: mon warn on too few osds = false 2023-07-28T23:38:10.179 INFO:teuthology.orchestra.run.smithi101.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-07-28T23:38:10.179 INFO:teuthology.orchestra.run.smithi101.stdout: mon_warn_on_pool_no_redundancy = false 2023-07-28T23:38:10.179 INFO:teuthology.orchestra.run.smithi101.stdout: mon_allow_pool_size_one = true 2023-07-28T23:38:10.179 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:38:10.179 INFO:teuthology.orchestra.run.smithi101.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-07-28T23:38:10.179 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:38:10.179 INFO:teuthology.orchestra.run.smithi101.stdout: osd default data pool replay window = 5 2023-07-28T23:38:10.179 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:38:10.180 INFO:teuthology.orchestra.run.smithi101.stdout: mon allow pool delete = true 2023-07-28T23:38:10.180 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:38:10.180 INFO:teuthology.orchestra.run.smithi101.stdout: mon cluster log file level = debug 2023-07-28T23:38:10.180 INFO:teuthology.orchestra.run.smithi101.stdout: debug asserts on shutdown = true 2023-07-28T23:38:10.180 INFO:teuthology.orchestra.run.smithi101.stdout: mon health detail to clog = false 2023-07-28T23:38:10.180 INFO:teuthology.orchestra.run.smithi101.stdout: mon host = 172.21.15.139 2023-07-28T23:38:10.180 INFO:teuthology.orchestra.run.smithi101.stdout: fsid = 5f79b2fe-1924-457c-a2b5-ff6dcac41f5f 2023-07-28T23:38:10.180 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:38:10.181 INFO:teuthology.orchestra.run.smithi101.stdout:[osd] 2023-07-28T23:38:10.181 INFO:teuthology.orchestra.run.smithi101.stdout: osd journal size = 100 2023-07-28T23:38:10.181 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:38:10.181 INFO:teuthology.orchestra.run.smithi101.stdout: osd scrub load threshold = 5.0 2023-07-28T23:38:10.181 INFO:teuthology.orchestra.run.smithi101.stdout: osd scrub max interval = 600 2023-07-28T23:38:10.181 INFO:teuthology.orchestra.run.smithi101.stdout: osd mclock profile = high_recovery_ops 2023-07-28T23:38:10.181 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:38:10.181 INFO:teuthology.orchestra.run.smithi101.stdout: osd recover clone overlap = true 2023-07-28T23:38:10.182 INFO:teuthology.orchestra.run.smithi101.stdout: osd recovery max chunk = 1048576 2023-07-28T23:38:10.182 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:38:10.182 INFO:teuthology.orchestra.run.smithi101.stdout: osd debug shutdown = true 2023-07-28T23:38:10.182 INFO:teuthology.orchestra.run.smithi101.stdout: osd debug op order = true 2023-07-28T23:38:10.182 INFO:teuthology.orchestra.run.smithi101.stdout: osd debug verify stray on activate = true 2023-07-28T23:38:10.182 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:38:10.182 INFO:teuthology.orchestra.run.smithi101.stdout: osd open classes on start = true 2023-07-28T23:38:10.182 INFO:teuthology.orchestra.run.smithi101.stdout: osd debug pg log writeout = true 2023-07-28T23:38:10.183 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:38:10.183 INFO:teuthology.orchestra.run.smithi101.stdout: osd deep scrub update digest min age = 30 2023-07-28T23:38:10.183 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:38:10.183 INFO:teuthology.orchestra.run.smithi101.stdout: osd map max advance = 10 2023-07-28T23:38:10.183 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:38:10.183 INFO:teuthology.orchestra.run.smithi101.stdout: journal zero on create = true 2023-07-28T23:38:10.183 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:38:10.183 INFO:teuthology.orchestra.run.smithi101.stdout: filestore ondisk finisher threads = 3 2023-07-28T23:38:10.184 INFO:teuthology.orchestra.run.smithi101.stdout: filestore apply finisher threads = 3 2023-07-28T23:38:10.184 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:38:10.184 INFO:teuthology.orchestra.run.smithi101.stdout: bdev debug aio = true 2023-07-28T23:38:10.184 INFO:teuthology.orchestra.run.smithi101.stdout: osd debug misdirected ops = true 2023-07-28T23:38:10.184 INFO:teuthology.orchestra.run.smithi101.stdout: debug ms = 1 2023-07-28T23:38:10.184 INFO:teuthology.orchestra.run.smithi101.stdout: debug osd = 20 2023-07-28T23:38:10.184 INFO:teuthology.orchestra.run.smithi101.stdout: rocksdb delete range threshold = 0 2023-07-28T23:38:10.184 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:38:10.185 INFO:teuthology.orchestra.run.smithi101.stdout:[mgr] 2023-07-28T23:38:10.185 INFO:teuthology.orchestra.run.smithi101.stdout: debug ms = 1 2023-07-28T23:38:10.185 INFO:teuthology.orchestra.run.smithi101.stdout: debug mgr = 20 2023-07-28T23:38:10.185 INFO:teuthology.orchestra.run.smithi101.stdout: debug mon = 20 2023-07-28T23:38:10.185 INFO:teuthology.orchestra.run.smithi101.stdout: debug auth = 20 2023-07-28T23:38:10.185 INFO:teuthology.orchestra.run.smithi101.stdout: mon reweight min pgs per osd = 4 2023-07-28T23:38:10.185 INFO:teuthology.orchestra.run.smithi101.stdout: mon reweight min bytes per osd = 10 2023-07-28T23:38:10.185 INFO:teuthology.orchestra.run.smithi101.stdout: mgr/telemetry/nag = false 2023-07-28T23:38:10.186 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:38:10.186 INFO:teuthology.orchestra.run.smithi101.stdout:[mon] 2023-07-28T23:38:10.186 INFO:teuthology.orchestra.run.smithi101.stdout: debug ms = 1 2023-07-28T23:38:10.186 INFO:teuthology.orchestra.run.smithi101.stdout: debug mon = 20 2023-07-28T23:38:10.186 INFO:teuthology.orchestra.run.smithi101.stdout: debug paxos = 20 2023-07-28T23:38:10.186 INFO:teuthology.orchestra.run.smithi101.stdout: debug auth = 20 2023-07-28T23:38:10.186 INFO:teuthology.orchestra.run.smithi101.stdout: mon data avail warn = 5 2023-07-28T23:38:10.186 INFO:teuthology.orchestra.run.smithi101.stdout: mon mgr mkfs grace = 240 2023-07-28T23:38:10.187 INFO:teuthology.orchestra.run.smithi101.stdout: mon reweight min pgs per osd = 4 2023-07-28T23:38:10.187 INFO:teuthology.orchestra.run.smithi101.stdout: mon osd reporter subtree level = osd 2023-07-28T23:38:10.187 INFO:teuthology.orchestra.run.smithi101.stdout: mon osd prime pg temp = true 2023-07-28T23:38:10.187 INFO:teuthology.orchestra.run.smithi101.stdout: mon reweight min bytes per osd = 10 2023-07-28T23:38:10.187 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:38:10.187 INFO:teuthology.orchestra.run.smithi101.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-07-28T23:38:10.187 INFO:teuthology.orchestra.run.smithi101.stdout: auth mon ticket ttl = 660 # 11m 2023-07-28T23:38:10.187 INFO:teuthology.orchestra.run.smithi101.stdout: auth service ticket ttl = 240 # 4m 2023-07-28T23:38:10.188 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:38:10.188 INFO:teuthology.orchestra.run.smithi101.stdout: # don't complain about insecure global_id in the test suite 2023-07-28T23:38:10.188 INFO:teuthology.orchestra.run.smithi101.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-07-28T23:38:10.188 INFO:teuthology.orchestra.run.smithi101.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-07-28T23:38:10.188 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:38:10.188 INFO:teuthology.orchestra.run.smithi101.stdout: # 1m isn't quite enough 2023-07-28T23:38:10.188 INFO:teuthology.orchestra.run.smithi101.stdout: mon_down_mkfs_grace = 2m 2023-07-28T23:38:10.189 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:38:10.189 INFO:teuthology.orchestra.run.smithi101.stdout: mon_warn_on_filestore_osds = false 2023-07-28T23:38:10.189 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:38:10.189 INFO:teuthology.orchestra.run.smithi101.stdout:[client] 2023-07-28T23:38:10.189 INFO:teuthology.orchestra.run.smithi101.stdout: rgw cache enabled = true 2023-07-28T23:38:10.189 INFO:teuthology.orchestra.run.smithi101.stdout: rgw enable ops log = true 2023-07-28T23:38:10.189 INFO:teuthology.orchestra.run.smithi101.stdout: rgw enable usage log = true 2023-07-28T23:38:10.189 INFO:teuthology.orchestra.run.smithi101.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-07-28T23:38:10.190 INFO:teuthology.orchestra.run.smithi101.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-07-28T23:38:10.190 INFO:teuthology.orchestra.run.smithi101.stdout: debug rgw = 20 2023-07-28T23:38:10.190 INFO:teuthology.orchestra.run.smithi101.stdout: debug rgw sync = 20 2023-07-28T23:38:10.190 INFO:teuthology.orchestra.run.smithi101.stdout: rgw crypt require ssl = False 2023-07-28T23:38:10.190 INFO:teuthology.orchestra.run.smithi101.stdout: rgw crypt s3 kms backend = testing 2023-07-28T23:38:10.190 INFO:teuthology.orchestra.run.smithi101.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-07-28T23:38:10.190 INFO:teuthology.orchestra.run.smithi101.stdout: rgw curl low speed time = 300 2023-07-28T23:38:10.190 INFO:teuthology.orchestra.run.smithi101.stdout: rgw data log num shards = 4 2023-07-28T23:38:10.191 INFO:teuthology.orchestra.run.smithi101.stdout: rgw md log max shards = 4 2023-07-28T23:38:10.191 INFO:teuthology.orchestra.run.smithi101.stdout: rgw sync data inject err probability = 0.1 2023-07-28T23:38:10.191 INFO:teuthology.orchestra.run.smithi101.stdout: rgw sync log trim interval = 0 2023-07-28T23:38:10.191 INFO:teuthology.orchestra.run.smithi101.stdout: rgw sync meta inject err probability = 0.1 2023-07-28T23:38:10.191 INFO:teuthology.orchestra.run.smithi101.stdout: rgw sync obj etag verify = True 2023-07-28T23:38:10.191 INFO:teuthology.orchestra.run.smithi101.stdout: setgroup = ceph 2023-07-28T23:38:10.191 INFO:teuthology.orchestra.run.smithi101.stdout: setuser = ceph 2023-07-28T23:38:10.191 INFO:teuthology.orchestra.run.smithi101.stdout:[mon.a] 2023-07-28T23:38:10.191 INFO:teuthology.orchestra.run.smithi101.stdout:[client.0] 2023-07-28T23:38:10.192 INFO:teuthology.orchestra.run.smithi101.stdout: rgw data notify interval msec = 0 2023-07-28T23:38:10.192 INFO:teuthology.orchestra.run.smithi101.stdout:[osd.0] 2023-07-28T23:38:10.192 INFO:teuthology.orchestra.run.smithi101.stdout: osd_max_omap_entries_per_request = 10 2023-07-28T23:38:10.192 INFO:teuthology.orchestra.run.smithi101.stdout:[osd.1] 2023-07-28T23:38:10.192 INFO:teuthology.orchestra.run.smithi101.stdout: osd_max_omap_entries_per_request = 1000 2023-07-28T23:38:10.192 INFO:teuthology.orchestra.run.smithi101.stdout:[osd.2] 2023-07-28T23:38:10.192 INFO:teuthology.orchestra.run.smithi101.stdout: osd_max_omap_entries_per_request = 10000 2023-07-28T23:38:10.207 INFO:teuthology.orchestra.run.smithi139.stdout:[global] 2023-07-28T23:38:10.207 INFO:teuthology.orchestra.run.smithi139.stdout: chdir = "" 2023-07-28T23:38:10.208 INFO:teuthology.orchestra.run.smithi139.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-07-28T23:38:10.208 INFO:teuthology.orchestra.run.smithi139.stdout: auth supported = cephx 2023-07-28T23:38:10.208 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:38:10.208 INFO:teuthology.orchestra.run.smithi139.stdout: filestore xattr use omap = true 2023-07-28T23:38:10.208 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:38:10.208 INFO:teuthology.orchestra.run.smithi139.stdout: mon clock drift allowed = 1.000 2023-07-28T23:38:10.208 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:38:10.209 INFO:teuthology.orchestra.run.smithi139.stdout: osd crush chooseleaf type = 0 2023-07-28T23:38:10.209 INFO:teuthology.orchestra.run.smithi139.stdout: auth debug = true 2023-07-28T23:38:10.209 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:38:10.209 INFO:teuthology.orchestra.run.smithi139.stdout: ms die on old message = true 2023-07-28T23:38:10.209 INFO:teuthology.orchestra.run.smithi139.stdout: ms die on bug = true 2023-07-28T23:38:10.209 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:38:10.209 INFO:teuthology.orchestra.run.smithi139.stdout: mon max pg per osd = 10000 # >= luminous 2023-07-28T23:38:10.209 INFO:teuthology.orchestra.run.smithi139.stdout: mon pg warn max object skew = 0 2023-07-28T23:38:10.210 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:38:10.210 INFO:teuthology.orchestra.run.smithi139.stdout: # disable pg_autoscaler by default for new pools 2023-07-28T23:38:10.210 INFO:teuthology.orchestra.run.smithi139.stdout: osd_pool_default_pg_autoscale_mode = off 2023-07-28T23:38:10.210 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:38:10.210 INFO:teuthology.orchestra.run.smithi139.stdout: osd pool default size = 2 2023-07-28T23:38:10.210 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:38:10.210 INFO:teuthology.orchestra.run.smithi139.stdout: mon osd allow primary affinity = true 2023-07-28T23:38:10.210 INFO:teuthology.orchestra.run.smithi139.stdout: mon osd allow pg remap = true 2023-07-28T23:38:10.211 INFO:teuthology.orchestra.run.smithi139.stdout: mon warn on legacy crush tunables = false 2023-07-28T23:38:10.211 INFO:teuthology.orchestra.run.smithi139.stdout: mon warn on crush straw calc version zero = false 2023-07-28T23:38:10.211 INFO:teuthology.orchestra.run.smithi139.stdout: mon warn on no sortbitwise = false 2023-07-28T23:38:10.211 INFO:teuthology.orchestra.run.smithi139.stdout: mon warn on osd down out interval zero = false 2023-07-28T23:38:10.211 INFO:teuthology.orchestra.run.smithi139.stdout: mon warn on too few osds = false 2023-07-28T23:38:10.211 INFO:teuthology.orchestra.run.smithi139.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-07-28T23:38:10.211 INFO:teuthology.orchestra.run.smithi139.stdout: mon_warn_on_pool_no_redundancy = false 2023-07-28T23:38:10.211 INFO:teuthology.orchestra.run.smithi139.stdout: mon_allow_pool_size_one = true 2023-07-28T23:38:10.212 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:38:10.212 INFO:teuthology.orchestra.run.smithi139.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-07-28T23:38:10.212 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:38:10.212 INFO:teuthology.orchestra.run.smithi139.stdout: osd default data pool replay window = 5 2023-07-28T23:38:10.212 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:38:10.212 INFO:teuthology.orchestra.run.smithi139.stdout: mon allow pool delete = true 2023-07-28T23:38:10.212 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:38:10.213 INFO:teuthology.orchestra.run.smithi139.stdout: mon cluster log file level = debug 2023-07-28T23:38:10.213 INFO:teuthology.orchestra.run.smithi139.stdout: debug asserts on shutdown = true 2023-07-28T23:38:10.213 INFO:teuthology.orchestra.run.smithi139.stdout: mon health detail to clog = false 2023-07-28T23:38:10.213 INFO:teuthology.orchestra.run.smithi139.stdout: mon host = 172.21.15.139 2023-07-28T23:38:10.213 INFO:teuthology.orchestra.run.smithi139.stdout: fsid = 5f79b2fe-1924-457c-a2b5-ff6dcac41f5f 2023-07-28T23:38:10.213 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:38:10.213 INFO:teuthology.orchestra.run.smithi139.stdout:[osd] 2023-07-28T23:38:10.213 INFO:teuthology.orchestra.run.smithi139.stdout: osd journal size = 100 2023-07-28T23:38:10.214 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:38:10.214 INFO:teuthology.orchestra.run.smithi139.stdout: osd scrub load threshold = 5.0 2023-07-28T23:38:10.214 INFO:teuthology.orchestra.run.smithi139.stdout: osd scrub max interval = 600 2023-07-28T23:38:10.214 INFO:teuthology.orchestra.run.smithi139.stdout: osd mclock profile = high_recovery_ops 2023-07-28T23:38:10.214 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:38:10.214 INFO:teuthology.orchestra.run.smithi139.stdout: osd recover clone overlap = true 2023-07-28T23:38:10.214 INFO:teuthology.orchestra.run.smithi139.stdout: osd recovery max chunk = 1048576 2023-07-28T23:38:10.214 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:38:10.215 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug shutdown = true 2023-07-28T23:38:10.215 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug op order = true 2023-07-28T23:38:10.215 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug verify stray on activate = true 2023-07-28T23:38:10.215 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:38:10.215 INFO:teuthology.orchestra.run.smithi139.stdout: osd open classes on start = true 2023-07-28T23:38:10.215 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug pg log writeout = true 2023-07-28T23:38:10.215 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:38:10.215 INFO:teuthology.orchestra.run.smithi139.stdout: osd deep scrub update digest min age = 30 2023-07-28T23:38:10.216 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:38:10.216 INFO:teuthology.orchestra.run.smithi139.stdout: osd map max advance = 10 2023-07-28T23:38:10.216 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:38:10.216 INFO:teuthology.orchestra.run.smithi139.stdout: journal zero on create = true 2023-07-28T23:38:10.216 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:38:10.216 INFO:teuthology.orchestra.run.smithi139.stdout: filestore ondisk finisher threads = 3 2023-07-28T23:38:10.216 INFO:teuthology.orchestra.run.smithi139.stdout: filestore apply finisher threads = 3 2023-07-28T23:38:10.216 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:38:10.217 INFO:teuthology.orchestra.run.smithi139.stdout: bdev debug aio = true 2023-07-28T23:38:10.217 INFO:teuthology.orchestra.run.smithi139.stdout: osd debug misdirected ops = true 2023-07-28T23:38:10.217 INFO:teuthology.orchestra.run.smithi139.stdout: debug ms = 1 2023-07-28T23:38:10.217 INFO:teuthology.orchestra.run.smithi139.stdout: debug osd = 20 2023-07-28T23:38:10.217 INFO:teuthology.orchestra.run.smithi139.stdout: rocksdb delete range threshold = 0 2023-07-28T23:38:10.217 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:38:10.217 INFO:teuthology.orchestra.run.smithi139.stdout:[mgr] 2023-07-28T23:38:10.217 INFO:teuthology.orchestra.run.smithi139.stdout: debug ms = 1 2023-07-28T23:38:10.218 INFO:teuthology.orchestra.run.smithi139.stdout: debug mgr = 20 2023-07-28T23:38:10.218 INFO:teuthology.orchestra.run.smithi139.stdout: debug mon = 20 2023-07-28T23:38:10.218 INFO:teuthology.orchestra.run.smithi139.stdout: debug auth = 20 2023-07-28T23:38:10.218 INFO:teuthology.orchestra.run.smithi139.stdout: mon reweight min pgs per osd = 4 2023-07-28T23:38:10.218 INFO:teuthology.orchestra.run.smithi139.stdout: mon reweight min bytes per osd = 10 2023-07-28T23:38:10.218 INFO:teuthology.orchestra.run.smithi139.stdout: mgr/telemetry/nag = false 2023-07-28T23:38:10.218 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:38:10.218 INFO:teuthology.orchestra.run.smithi139.stdout:[mon] 2023-07-28T23:38:10.219 INFO:teuthology.orchestra.run.smithi139.stdout: debug ms = 1 2023-07-28T23:38:10.219 INFO:teuthology.orchestra.run.smithi139.stdout: debug mon = 20 2023-07-28T23:38:10.219 INFO:teuthology.orchestra.run.smithi139.stdout: debug paxos = 20 2023-07-28T23:38:10.219 INFO:teuthology.orchestra.run.smithi139.stdout: debug auth = 20 2023-07-28T23:38:10.219 INFO:teuthology.orchestra.run.smithi139.stdout: mon data avail warn = 5 2023-07-28T23:38:10.219 INFO:teuthology.orchestra.run.smithi139.stdout: mon mgr mkfs grace = 240 2023-07-28T23:38:10.219 INFO:teuthology.orchestra.run.smithi139.stdout: mon reweight min pgs per osd = 4 2023-07-28T23:38:10.219 INFO:teuthology.orchestra.run.smithi139.stdout: mon osd reporter subtree level = osd 2023-07-28T23:38:10.220 INFO:teuthology.orchestra.run.smithi139.stdout: mon osd prime pg temp = true 2023-07-28T23:38:10.220 INFO:teuthology.orchestra.run.smithi139.stdout: mon reweight min bytes per osd = 10 2023-07-28T23:38:10.220 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:38:10.220 INFO:teuthology.orchestra.run.smithi139.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-07-28T23:38:10.220 INFO:teuthology.orchestra.run.smithi139.stdout: auth mon ticket ttl = 660 # 11m 2023-07-28T23:38:10.220 INFO:teuthology.orchestra.run.smithi139.stdout: auth service ticket ttl = 240 # 4m 2023-07-28T23:38:10.220 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:38:10.220 INFO:teuthology.orchestra.run.smithi139.stdout: # don't complain about insecure global_id in the test suite 2023-07-28T23:38:10.221 INFO:teuthology.orchestra.run.smithi139.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-07-28T23:38:10.221 INFO:teuthology.orchestra.run.smithi139.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-07-28T23:38:10.221 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:38:10.221 INFO:teuthology.orchestra.run.smithi139.stdout: # 1m isn't quite enough 2023-07-28T23:38:10.221 INFO:teuthology.orchestra.run.smithi139.stdout: mon_down_mkfs_grace = 2m 2023-07-28T23:38:10.221 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:38:10.221 INFO:teuthology.orchestra.run.smithi139.stdout: mon_warn_on_filestore_osds = false 2023-07-28T23:38:10.221 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:38:10.222 INFO:teuthology.orchestra.run.smithi139.stdout:[client] 2023-07-28T23:38:10.222 INFO:teuthology.orchestra.run.smithi139.stdout: rgw cache enabled = true 2023-07-28T23:38:10.222 INFO:teuthology.orchestra.run.smithi139.stdout: rgw enable ops log = true 2023-07-28T23:38:10.222 INFO:teuthology.orchestra.run.smithi139.stdout: rgw enable usage log = true 2023-07-28T23:38:10.222 INFO:teuthology.orchestra.run.smithi139.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-07-28T23:38:10.222 INFO:teuthology.orchestra.run.smithi139.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-07-28T23:38:10.222 INFO:teuthology.orchestra.run.smithi139.stdout: debug rgw = 20 2023-07-28T23:38:10.222 INFO:teuthology.orchestra.run.smithi139.stdout: debug rgw sync = 20 2023-07-28T23:38:10.223 INFO:teuthology.orchestra.run.smithi139.stdout: rgw crypt require ssl = False 2023-07-28T23:38:10.223 INFO:teuthology.orchestra.run.smithi139.stdout: rgw crypt s3 kms backend = testing 2023-07-28T23:38:10.223 INFO:teuthology.orchestra.run.smithi139.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-07-28T23:38:10.223 INFO:teuthology.orchestra.run.smithi139.stdout: rgw curl low speed time = 300 2023-07-28T23:38:10.223 INFO:teuthology.orchestra.run.smithi139.stdout: rgw data log num shards = 4 2023-07-28T23:38:10.223 INFO:teuthology.orchestra.run.smithi139.stdout: rgw md log max shards = 4 2023-07-28T23:38:10.223 INFO:teuthology.orchestra.run.smithi139.stdout: rgw sync data inject err probability = 0.1 2023-07-28T23:38:10.223 INFO:teuthology.orchestra.run.smithi139.stdout: rgw sync log trim interval = 0 2023-07-28T23:38:10.224 INFO:teuthology.orchestra.run.smithi139.stdout: rgw sync meta inject err probability = 0.1 2023-07-28T23:38:10.224 INFO:teuthology.orchestra.run.smithi139.stdout: rgw sync obj etag verify = True 2023-07-28T23:38:10.224 INFO:teuthology.orchestra.run.smithi139.stdout: setgroup = ceph 2023-07-28T23:38:10.224 INFO:teuthology.orchestra.run.smithi139.stdout: setuser = ceph 2023-07-28T23:38:10.224 INFO:teuthology.orchestra.run.smithi139.stdout:[mon.a] 2023-07-28T23:38:10.224 INFO:teuthology.orchestra.run.smithi139.stdout:[client.0] 2023-07-28T23:38:10.224 INFO:teuthology.orchestra.run.smithi139.stdout: rgw data notify interval msec = 0 2023-07-28T23:38:10.225 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.0] 2023-07-28T23:38:10.225 INFO:teuthology.orchestra.run.smithi139.stdout: osd_max_omap_entries_per_request = 10 2023-07-28T23:38:10.225 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.1] 2023-07-28T23:38:10.225 INFO:teuthology.orchestra.run.smithi139.stdout: osd_max_omap_entries_per_request = 1000 2023-07-28T23:38:10.225 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.2] 2023-07-28T23:38:10.225 INFO:teuthology.orchestra.run.smithi139.stdout: osd_max_omap_entries_per_request = 10000 2023-07-28T23:38:10.226 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2023-07-28T23:38:10.226 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2023-07-28T23:38:10.280 INFO:tasks.ceph:Copying monmap to all nodes... 2023-07-28T23:38:10.280 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2023-07-28T23:38:10.281 DEBUG:teuthology.orchestra.run.smithi139:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2023-07-28T23:38:10.295 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2023-07-28T23:38:10.295 DEBUG:teuthology.orchestra.run.smithi139:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2023-07-28T23:38:10.349 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi101.front.sepia.ceph.com 2023-07-28T23:38:10.349 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-07-28T23:38:10.349 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd of=/etc/ceph/c2.keyring 2023-07-28T23:38:10.350 DEBUG:teuthology.orchestra.run.smithi101:> sudo chmod 0644 /etc/ceph/c2.keyring 2023-07-28T23:38:10.391 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-07-28T23:38:10.391 DEBUG:teuthology.orchestra.run.smithi101:> dd of=/home/ubuntu/cephtest/c2.monmap 2023-07-28T23:38:10.445 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi139.front.sepia.ceph.com 2023-07-28T23:38:10.446 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2023-07-28T23:38:10.446 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/ceph/c2.keyring 2023-07-28T23:38:10.446 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod 0644 /etc/ceph/c2.keyring 2023-07-28T23:38:10.484 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2023-07-28T23:38:10.485 DEBUG:teuthology.orchestra.run.smithi139:> dd of=/home/ubuntu/cephtest/c2.monmap 2023-07-28T23:38:10.538 INFO:tasks.ceph:Setting up mon nodes... 2023-07-28T23:38:10.539 INFO:tasks.ceph:Setting up mgr nodes... 2023-07-28T23:38:10.539 DEBUG:teuthology.orchestra.run.smithi139:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2023-07-28T23:38:10.632 INFO:teuthology.orchestra.run.smithi139.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2023-07-28T23:38:10.634 INFO:tasks.ceph:Setting up mds nodes... 2023-07-28T23:38:10.635 INFO:tasks.ceph_client:Setting up client nodes... 2023-07-28T23:38:10.635 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2023-07-28T23:38:10.676 INFO:teuthology.orchestra.run.smithi139.stdout:creating /etc/ceph/c2.client.0.keyring 2023-07-28T23:38:10.691 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2023-07-28T23:38:10.772 INFO:teuthology.orchestra.run.smithi139.stdout:creating /etc/ceph/c2.client.1.keyring 2023-07-28T23:38:10.787 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-07-28T23:38:10.787 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi101.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi139.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-07-28T23:38:10.788 DEBUG:teuthology.orchestra.run.smithi139:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2023-07-28T23:38:10.853 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-07-28T23:38:10.853 INFO:tasks.ceph:role: c2.osd.0 2023-07-28T23:38:10.854 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi139.front.sepia.ceph.com 2023-07-28T23:38:10.854 DEBUG:teuthology.orchestra.run.smithi139:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-07-28T23:38:10.926 INFO:teuthology.orchestra.run.smithi139.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-07-28T23:38:10.927 INFO:teuthology.orchestra.run.smithi139.stdout: = sectsz=512 attr=2, projid32bit=1 2023-07-28T23:38:10.927 INFO:teuthology.orchestra.run.smithi139.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-07-28T23:38:10.927 INFO:teuthology.orchestra.run.smithi139.stdout: = reflink=1 bigtime=1 inobtcount=1 2023-07-28T23:38:10.927 INFO:teuthology.orchestra.run.smithi139.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-07-28T23:38:10.927 INFO:teuthology.orchestra.run.smithi139.stdout: = sunit=0 swidth=0 blks 2023-07-28T23:38:10.927 INFO:teuthology.orchestra.run.smithi139.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-07-28T23:38:10.927 INFO:teuthology.orchestra.run.smithi139.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-07-28T23:38:10.928 INFO:teuthology.orchestra.run.smithi139.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-07-28T23:38:10.928 INFO:teuthology.orchestra.run.smithi139.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-07-28T23:38:11.419 INFO:teuthology.orchestra.run.smithi139.stdout:Discarding blocks...Done. 2023-07-28T23:38:11.421 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi139.front.sepia.ceph.com -o noatime 2023-07-28T23:38:11.422 DEBUG:teuthology.orchestra.run.smithi139:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2023-07-28T23:38:11.582 DEBUG:teuthology.orchestra.run.smithi139:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2023-07-28T23:38:11.617 DEBUG:teuthology.orchestra.run.smithi139:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2023-07-28T23:38:11.685 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-07-28T23:38:11.686 INFO:tasks.ceph:role: c2.osd.1 2023-07-28T23:38:11.686 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi139.front.sepia.ceph.com 2023-07-28T23:38:11.686 DEBUG:teuthology.orchestra.run.smithi139:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-07-28T23:38:11.760 INFO:teuthology.orchestra.run.smithi139.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-07-28T23:38:11.760 INFO:teuthology.orchestra.run.smithi139.stdout: = sectsz=512 attr=2, projid32bit=1 2023-07-28T23:38:11.760 INFO:teuthology.orchestra.run.smithi139.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-07-28T23:38:11.761 INFO:teuthology.orchestra.run.smithi139.stdout: = reflink=1 bigtime=1 inobtcount=1 2023-07-28T23:38:11.761 INFO:teuthology.orchestra.run.smithi139.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-07-28T23:38:11.761 INFO:teuthology.orchestra.run.smithi139.stdout: = sunit=0 swidth=0 blks 2023-07-28T23:38:11.761 INFO:teuthology.orchestra.run.smithi139.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-07-28T23:38:11.761 INFO:teuthology.orchestra.run.smithi139.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-07-28T23:38:11.761 INFO:teuthology.orchestra.run.smithi139.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-07-28T23:38:11.761 INFO:teuthology.orchestra.run.smithi139.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-07-28T23:38:12.173 INFO:teuthology.orchestra.run.smithi139.stdout:Discarding blocks...Done. 2023-07-28T23:38:12.175 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi139.front.sepia.ceph.com -o noatime 2023-07-28T23:38:12.176 DEBUG:teuthology.orchestra.run.smithi139:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2023-07-28T23:38:12.278 DEBUG:teuthology.orchestra.run.smithi139:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2023-07-28T23:38:12.310 DEBUG:teuthology.orchestra.run.smithi139:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2023-07-28T23:38:12.378 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-07-28T23:38:12.378 INFO:tasks.ceph:role: c2.osd.2 2023-07-28T23:38:12.378 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi139.front.sepia.ceph.com 2023-07-28T23:38:12.379 DEBUG:teuthology.orchestra.run.smithi139:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-07-28T23:38:12.454 INFO:teuthology.orchestra.run.smithi139.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-07-28T23:38:12.454 INFO:teuthology.orchestra.run.smithi139.stdout: = sectsz=512 attr=2, projid32bit=1 2023-07-28T23:38:12.454 INFO:teuthology.orchestra.run.smithi139.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-07-28T23:38:12.455 INFO:teuthology.orchestra.run.smithi139.stdout: = reflink=1 bigtime=1 inobtcount=1 2023-07-28T23:38:12.455 INFO:teuthology.orchestra.run.smithi139.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-07-28T23:38:12.455 INFO:teuthology.orchestra.run.smithi139.stdout: = sunit=0 swidth=0 blks 2023-07-28T23:38:12.455 INFO:teuthology.orchestra.run.smithi139.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-07-28T23:38:12.455 INFO:teuthology.orchestra.run.smithi139.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-07-28T23:38:12.455 INFO:teuthology.orchestra.run.smithi139.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-07-28T23:38:12.455 INFO:teuthology.orchestra.run.smithi139.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-07-28T23:38:12.924 INFO:teuthology.orchestra.run.smithi139.stdout:Discarding blocks...Done. 2023-07-28T23:38:12.926 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi139.front.sepia.ceph.com -o noatime 2023-07-28T23:38:12.927 DEBUG:teuthology.orchestra.run.smithi139:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2023-07-28T23:38:13.084 DEBUG:teuthology.orchestra.run.smithi139:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2023-07-28T23:38:13.120 DEBUG:teuthology.orchestra.run.smithi139:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2023-07-28T23:38:13.222 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:38:13.224+0000 7f3125aa9500 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2023-07-28T23:38:13.227 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:38:13.229+0000 7f3125aa9500 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2023-07-28T23:38:13.227 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:38:13.229+0000 7f3125aa9500 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2023-07-28T23:38:13.227 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:38:13.229+0000 7f3125aa9500 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2023-07-28T23:38:13.227 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:38:13.229+0000 7f3125aa9500 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2023-07-28T23:38:13.235 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:38:13.237+0000 7f3125aa9500 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2023-07-28T23:38:15.313 DEBUG:teuthology.orchestra.run.smithi139:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2023-07-28T23:38:15.344 DEBUG:teuthology.orchestra.run.smithi139:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2023-07-28T23:38:15.434 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:38:15.436+0000 7f7ba5f45500 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2023-07-28T23:38:15.434 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:38:15.436+0000 7f7ba5f45500 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2023-07-28T23:38:15.434 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:38:15.436+0000 7f7ba5f45500 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2023-07-28T23:38:15.434 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:38:15.436+0000 7f7ba5f45500 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2023-07-28T23:38:15.434 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:38:15.436+0000 7f7ba5f45500 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2023-07-28T23:38:15.435 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:38:15.437+0000 7f7ba5f45500 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2023-07-28T23:38:17.510 DEBUG:teuthology.orchestra.run.smithi139:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2023-07-28T23:38:17.541 DEBUG:teuthology.orchestra.run.smithi139:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2023-07-28T23:38:17.632 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:38:17.634+0000 7fb4ffdbe500 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2023-07-28T23:38:17.633 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:38:17.635+0000 7fb4ffdbe500 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2023-07-28T23:38:17.633 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:38:17.635+0000 7fb4ffdbe500 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2023-07-28T23:38:17.633 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:38:17.635+0000 7fb4ffdbe500 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2023-07-28T23:38:17.633 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:38:17.635+0000 7fb4ffdbe500 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2023-07-28T23:38:17.634 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:38:17.635+0000 7fb4ffdbe500 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2023-07-28T23:38:19.717 DEBUG:teuthology.orchestra.run.smithi139:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2023-07-28T23:38:19.749 INFO:tasks.ceph:Reading keys from all nodes... 2023-07-28T23:38:19.749 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2023-07-28T23:38:19.749 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2023-07-28T23:38:19.819 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2023-07-28T23:38:19.819 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2023-07-28T23:38:19.887 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2023-07-28T23:38:19.887 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2023-07-28T23:38:19.954 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2023-07-28T23:38:19.955 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2023-07-28T23:38:20.021 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2023-07-28T23:38:20.022 DEBUG:teuthology.orchestra.run.smithi139:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2023-07-28T23:38:20.076 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2023-07-28T23:38:20.076 DEBUG:teuthology.orchestra.run.smithi139:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2023-07-28T23:38:20.131 INFO:tasks.ceph:Adding keys to all mons... 2023-07-28T23:38:20.131 DEBUG:teuthology.orchestra.run.smithi139:> sudo tee -a /etc/ceph/c2.keyring 2023-07-28T23:38:20.195 INFO:teuthology.orchestra.run.smithi139.stdout:[mgr.x] 2023-07-28T23:38:20.195 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQBiUcRkbrnkJRAAtfjp6sYnF7eXLAzHu7wJeg== 2023-07-28T23:38:20.195 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.0] 2023-07-28T23:38:20.196 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQBlUcRkto5rDRAAD59twXG9SGjsH3qGhzPCdQ== 2023-07-28T23:38:20.196 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.1] 2023-07-28T23:38:20.196 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQBnUcRkAv4JGhAA6IcdgaKtBHj6y3e5NOYE2A== 2023-07-28T23:38:20.196 INFO:teuthology.orchestra.run.smithi139.stdout:[osd.2] 2023-07-28T23:38:20.196 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQBpUcRk4crfJRAAZfo4nRgZFDIDFvU25/zR/g== 2023-07-28T23:38:20.196 INFO:teuthology.orchestra.run.smithi139.stdout:[client.0] 2023-07-28T23:38:20.196 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQBiUcRkz0CFKBAAA8YzaTIjThC3pYFXGMvoJA== 2023-07-28T23:38:20.197 INFO:teuthology.orchestra.run.smithi139.stdout:[client.1] 2023-07-28T23:38:20.197 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQBiUcRkUTI9LhAALTIuxMlxwCwYA9xDs60FJA== 2023-07-28T23:38:20.197 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-07-28T23:38:20.289 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-07-28T23:38:20.341 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-07-28T23:38:20.392 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-07-28T23:38:20.444 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-07-28T23:38:20.498 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-07-28T23:38:20.555 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-07-28T23:38:20.555 DEBUG:teuthology.orchestra.run.smithi139:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2023-07-28T23:38:20.581 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2023-07-28T23:38:20.679 DEBUG:teuthology.orchestra.run.smithi139:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2023-07-28T23:38:20.705 DEBUG:teuthology.orchestra.run.smithi139:> rm -- /home/ubuntu/cephtest/c2.monmap 2023-07-28T23:38:20.759 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2023-07-28T23:38:20.760 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2023-07-28T23:38:20.760 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2023-07-28T23:38:20.801 INFO:tasks.ceph.c2.mon.a:Started 2023-07-28T23:38:20.802 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2023-07-28T23:38:20.802 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2023-07-28T23:38:20.802 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2023-07-28T23:38:20.804 INFO:tasks.ceph.c2.mgr.x:Started 2023-07-28T23:38:20.805 INFO:tasks.ceph:Setting crush tunables to default 2023-07-28T23:38:20.805 DEBUG:teuthology.orchestra.run.smithi139:> sudo ceph --cluster c2 osd crush tunables default 2023-07-28T23:38:20.989 INFO:teuthology.orchestra.run.smithi139.stderr:adjusted tunables profile to default 2023-07-28T23:38:21.000 INFO:tasks.ceph:check_enable_crimson: False 2023-07-28T23:38:21.000 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2023-07-28T23:38:21.001 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2023-07-28T23:38:21.001 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2023-07-28T23:38:21.027 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2023-07-28T23:38:21.027 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2023-07-28T23:38:21.094 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2023-07-28T23:38:21.095 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2023-07-28T23:38:21.161 DEBUG:teuthology.orchestra.run.smithi139:> sudo ceph --cluster c2 osd new d871fec6-9c45-4792-8086-a94c0916c2ca 0 2023-07-28T23:38:21.214 INFO:tasks.ceph.c2.mgr.x.smithi139.stderr:2023-07-28T23:38:21.217+0000 7f3192e16040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-28T23:38:21.381 INFO:teuthology.orchestra.run.smithi139.stdout:0 2023-07-28T23:38:21.392 DEBUG:teuthology.orchestra.run.smithi139:> sudo ceph --cluster c2 osd new 367dcee6-86cb-4589-879a-d61a553e3369 1 2023-07-28T23:38:21.612 INFO:teuthology.orchestra.run.smithi139.stdout:1 2023-07-28T23:38:21.622 DEBUG:teuthology.orchestra.run.smithi139:> sudo ceph --cluster c2 osd new 47016cf2-1bfc-42f2-b645-1556adf25996 2 2023-07-28T23:38:21.801 INFO:teuthology.orchestra.run.smithi139.stdout:2 2023-07-28T23:38:21.812 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2023-07-28T23:38:21.813 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2023-07-28T23:38:21.815 INFO:tasks.ceph.c2.osd.0:Started 2023-07-28T23:38:21.815 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2023-07-28T23:38:21.815 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2023-07-28T23:38:21.817 INFO:tasks.ceph.c2.osd.1:Started 2023-07-28T23:38:21.817 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2023-07-28T23:38:21.818 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2023-07-28T23:38:21.820 INFO:tasks.ceph.c2.osd.2:Started 2023-07-28T23:38:21.820 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2023-07-28T23:38:22.002 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:38:22.002 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":5,"fsid":"5f79b2fe-1924-457c-a2b5-ff6dcac41f5f","created":"2023-07-28T23:38:20.914874+0000","modified":"2023-07-28T23:38:21.802978+0000","last_up_change":"0.000000","last_in_change":"2023-07-28T23:38:21.802978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d871fec6-9c45-4792-8086-a94c0916c2ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"367dcee6-86cb-4589-879a-d61a553e3369","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47016cf2-1bfc-42f2-b645-1556adf25996","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-28T23:38:22.012 INFO:tasks.ceph.ceph_manager.c2:[] 2023-07-28T23:38:22.013 INFO:tasks.ceph:Waiting for OSDs to come up 2023-07-28T23:38:22.060 INFO:tasks.ceph.c2.mgr.x.smithi139.stderr:2023-07-28T23:38:22.062+0000 7f3192e16040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-28T23:38:22.198 INFO:tasks.ceph.c2.mgr.x.smithi139.stderr:2023-07-28T23:38:22.200+0000 7f3192e16040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-28T23:38:22.246 INFO:tasks.ceph.c2.mgr.x.smithi139.stderr:2023-07-28T23:38:22.249+0000 7f3192e16040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-28T23:38:22.290 INFO:tasks.ceph.c2.mgr.x.smithi139.stderr:2023-07-28T23:38:22.293+0000 7f3192e16040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-28T23:38:22.336 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2023-07-28T23:38:22.414 INFO:tasks.ceph.c2.osd.0.smithi139.stderr:2023-07-28T23:38:22.416+0000 7fc6d4636500 -1 Falling back to public interface 2023-07-28T23:38:22.415 INFO:tasks.ceph.c2.osd.2.smithi139.stderr:2023-07-28T23:38:22.416+0000 7f62ce206500 -1 Falling back to public interface 2023-07-28T23:38:22.415 INFO:tasks.ceph.c2.osd.0.smithi139.stderr:2023-07-28T23:38:22.416+0000 7fc6d35dd640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6800/0: (98) Address already in use 2023-07-28T23:38:22.415 INFO:tasks.ceph.c2.osd.0.smithi139.stderr:2023-07-28T23:38:22.416+0000 7fc6d35dd640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6801/0: (98) Address already in use 2023-07-28T23:38:22.426 INFO:tasks.ceph.c2.osd.1.smithi139.stderr:2023-07-28T23:38:22.428+0000 7f861d7d5500 -1 Falling back to public interface 2023-07-28T23:38:22.496 INFO:teuthology.misc.health.smithi139.stdout: 2023-07-28T23:38:22.496 INFO:teuthology.misc.health.smithi139.stdout:{"epoch":5,"fsid":"5f79b2fe-1924-457c-a2b5-ff6dcac41f5f","created":"2023-07-28T23:38:20.914874+0000","modified":"2023-07-28T23:38:21.802978+0000","last_up_change":"0.000000","last_in_change":"2023-07-28T23:38:21.802978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d871fec6-9c45-4792-8086-a94c0916c2ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"367dcee6-86cb-4589-879a-d61a553e3369","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47016cf2-1bfc-42f2-b645-1556adf25996","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-28T23:38:22.505 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-07-28T23:38:23.001 INFO:tasks.ceph.c2.mgr.x.smithi139.stderr:2023-07-28T23:38:23.003+0000 7f3192e16040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-28T23:38:23.053 INFO:tasks.ceph.c2.mgr.x.smithi139.stderr:2023-07-28T23:38:23.055+0000 7f3192e16040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-28T23:38:23.098 INFO:tasks.ceph.c2.mgr.x.smithi139.stderr:2023-07-28T23:38:23.101+0000 7f3192e16040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-28T23:38:23.206 INFO:tasks.ceph.c2.mgr.x.smithi139.stderr:2023-07-28T23:38:23.209+0000 7f3192e16040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-28T23:38:23.344 INFO:tasks.ceph.c2.mgr.x.smithi139.stderr:2023-07-28T23:38:23.347+0000 7f3192e16040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-28T23:38:23.438 INFO:tasks.ceph.c2.mgr.x.smithi139.stderr:2023-07-28T23:38:23.440+0000 7f3192e16040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-28T23:38:23.480 INFO:tasks.ceph.c2.mgr.x.smithi139.stderr:2023-07-28T23:38:23.482+0000 7f3192e16040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-28T23:38:23.613 INFO:tasks.ceph.c2.mgr.x.smithi139.stderr:2023-07-28T23:38:23.616+0000 7f3192e16040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-28T23:38:23.985 INFO:tasks.ceph.c2.mgr.x.smithi139.stderr:2023-07-28T23:38:23.987+0000 7f3192e16040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-28T23:38:24.177 INFO:tasks.ceph.c2.mgr.x.smithi139.stderr:2023-07-28T23:38:24.179+0000 7f3192e16040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-28T23:38:24.229 INFO:tasks.ceph.c2.mgr.x.smithi139.stderr:2023-07-28T23:38:24.231+0000 7f3192e16040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-28T23:38:24.273 INFO:tasks.ceph.c2.mgr.x.smithi139.stderr:2023-07-28T23:38:24.276+0000 7f3192e16040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-28T23:38:24.325 INFO:tasks.ceph.c2.mgr.x.smithi139.stderr:2023-07-28T23:38:24.327+0000 7f3192e16040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-28T23:38:24.557 INFO:tasks.ceph.c2.mgr.x.smithi139.stderr:2023-07-28T23:38:24.559+0000 7f3192e16040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-28T23:38:24.609 INFO:tasks.ceph.c2.mgr.x.smithi139.stderr:2023-07-28T23:38:24.611+0000 7f3192e16040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-28T23:38:24.656 INFO:tasks.ceph.c2.mgr.x.smithi139.stderr:2023-07-28T23:38:24.659+0000 7f3192e16040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-28T23:38:24.795 INFO:tasks.ceph.c2.mgr.x.smithi139.stderr:2023-07-28T23:38:24.797+0000 7f3192e16040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-28T23:38:24.836 INFO:tasks.ceph.c2.mgr.x.smithi139.stderr:2023-07-28T23:38:24.838+0000 7f3192e16040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-28T23:38:24.887 INFO:tasks.ceph.c2.mgr.x.smithi139.stderr:2023-07-28T23:38:24.890+0000 7f3192e16040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-28T23:38:24.937 INFO:tasks.ceph.c2.mgr.x.smithi139.stderr:2023-07-28T23:38:24.940+0000 7f3192e16040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-28T23:38:25.179 INFO:tasks.ceph.c2.osd.2.smithi139.stderr:2023-07-28T23:38:25.181+0000 7f62ce206500 -1 osd.2 0 log_to_monitors true 2023-07-28T23:38:25.185 INFO:tasks.ceph.c2.osd.1.smithi139.stderr:2023-07-28T23:38:25.187+0000 7f861d7d5500 -1 osd.1 0 log_to_monitors true 2023-07-28T23:38:25.191 INFO:tasks.ceph.c2.osd.0.smithi139.stderr:2023-07-28T23:38:25.193+0000 7fc6d4636500 -1 osd.0 0 log_to_monitors true 2023-07-28T23:38:25.957 INFO:tasks.ceph.c2.mgr.x.smithi139.stderr:2023-07-28T23:38:25.960+0000 7f3134192640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-07-28T23:38:28.234 INFO:tasks.ceph.c2.osd.0.smithi139.stderr:2023-07-28T23:38:28.237+0000 7fc6d05c1640 -1 osd.0 0 waiting for initial osdmap 2023-07-28T23:38:28.247 INFO:tasks.ceph.c2.osd.0.smithi139.stderr:2023-07-28T23:38:28.250+0000 7fc6c83e0640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-28T23:38:28.274 INFO:tasks.ceph.c2.osd.2.smithi139.stderr:2023-07-28T23:38:28.277+0000 7f62ca193640 -1 osd.2 0 waiting for initial osdmap 2023-07-28T23:38:28.293 INFO:tasks.ceph.c2.osd.2.smithi139.stderr:2023-07-28T23:38:28.296+0000 7f62c1fb2640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-28T23:38:28.311 INFO:tasks.ceph.c2.osd.1.smithi139.stderr:2023-07-28T23:38:28.313+0000 7f8619760640 -1 osd.1 0 waiting for initial osdmap 2023-07-28T23:38:28.320 INFO:tasks.ceph.c2.osd.1.smithi139.stderr:2023-07-28T23:38:28.323+0000 7f861157f640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-28T23:38:28.808 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2023-07-28T23:38:29.073 INFO:teuthology.misc.health.smithi139.stdout: 2023-07-28T23:38:29.074 INFO:teuthology.misc.health.smithi139.stdout:{"epoch":7,"fsid":"5f79b2fe-1924-457c-a2b5-ff6dcac41f5f","created":"2023-07-28T23:38:20.914874+0000","modified":"2023-07-28T23:38:27.012040+0000","last_up_change":"0.000000","last_in_change":"2023-07-28T23:38:21.802978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d871fec6-9c45-4792-8086-a94c0916c2ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"367dcee6-86cb-4589-879a-d61a553e3369","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47016cf2-1bfc-42f2-b645-1556adf25996","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-28T23:38:29.085 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-07-28T23:38:35.388 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2023-07-28T23:38:35.661 INFO:teuthology.misc.health.smithi139.stdout: 2023-07-28T23:38:35.661 INFO:teuthology.misc.health.smithi139.stdout:{"epoch":11,"fsid":"5f79b2fe-1924-457c-a2b5-ff6dcac41f5f","created":"2023-07-28T23:38:20.914874+0000","modified":"2023-07-28T23:38:33.051291+0000","last_up_change":"2023-07-28T23:38:29.239010+0000","last_in_change":"2023-07-28T23:38:21.802978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-28T23:38:30.997597+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d871fec6-9c45-4792-8086-a94c0916c2ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":82192},{"type":"v1","addr":"172.21.15.139:6804","nonce":82192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":82192},{"type":"v1","addr":"172.21.15.139:6808","nonce":82192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":82192},{"type":"v1","addr":"172.21.15.139:6815","nonce":82192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":82192},{"type":"v1","addr":"172.21.15.139:6812","nonce":82192}]},"public_addr":"172.21.15.139:6804/82192","cluster_addr":"172.21.15.139:6808/82192","heartbeat_back_addr":"172.21.15.139:6815/82192","heartbeat_front_addr":"172.21.15.139:6812/82192","state":["exists","up"]},{"osd":1,"uuid":"367dcee6-86cb-4589-879a-d61a553e3369","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":82194},{"type":"v1","addr":"172.21.15.139:6817","nonce":82194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":82194},{"type":"v1","addr":"172.21.15.139:6819","nonce":82194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":82194},{"type":"v1","addr":"172.21.15.139:6823","nonce":82194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":82194},{"type":"v1","addr":"172.21.15.139:6821","nonce":82194}]},"public_addr":"172.21.15.139:6817/82194","cluster_addr":"172.21.15.139:6819/82194","heartbeat_back_addr":"172.21.15.139:6823/82194","heartbeat_front_addr":"172.21.15.139:6821/82194","state":["exists","up"]},{"osd":2,"uuid":"47016cf2-1bfc-42f2-b645-1556adf25996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":82193},{"type":"v1","addr":"172.21.15.139:6801","nonce":82193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6803","nonce":82193},{"type":"v1","addr":"172.21.15.139:6805","nonce":82193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6811","nonce":82193},{"type":"v1","addr":"172.21.15.139:6813","nonce":82193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6807","nonce":82193},{"type":"v1","addr":"172.21.15.139:6809","nonce":82193}]},"public_addr":"172.21.15.139:6801/82193","cluster_addr":"172.21.15.139:6805/82193","heartbeat_back_addr":"172.21.15.139:6813/82193","heartbeat_front_addr":"172.21.15.139:6809/82193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-28T23:38:26.147504+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-28T23:38:26.212465+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-28T23:38:26.173521+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-28T23:38:35.672 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-07-28T23:38:35.673 INFO:tasks.ceph:Creating RBD pool 2023-07-28T23:38:35.673 DEBUG:teuthology.orchestra.run.smithi139:> sudo ceph --cluster c2 osd pool create rbd 8 2023-07-28T23:38:36.889 INFO:teuthology.orchestra.run.smithi139.stderr:pool 'rbd' created 2023-07-28T23:38:36.905 DEBUG:teuthology.orchestra.run.smithi139:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-07-28T23:38:37.891 INFO:teuthology.orchestra.run.smithi139.stderr:enabled application 'rbd' on pool 'rbd' 2023-07-28T23:38:37.909 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2023-07-28T23:38:37.909 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2023-07-28T23:38:37.910 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2023-07-28T23:38:37.910 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2023-07-28T23:38:37.910 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-07-28T23:38:38.237 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:38:38.248 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":5,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":81856},{"type":"v1","addr":"172.21.15.139:6825","nonce":81856}]},"active_addr":"172.21.15.139:6825/81856","active_change":"2023-07-28T23:38:24.953805+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph:v18","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.139:0","nonce":1990914162}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.139:0","nonce":3799113566}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.139:0","nonce":1451747894}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.139:0","nonce":2609638692}]}]} 2023-07-28T23:38:38.251 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2023-07-28T23:38:38.251 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2023-07-28T23:38:38.251 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2023-07-28T23:38:38.526 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:38:38.526 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":13,"fsid":"5f79b2fe-1924-457c-a2b5-ff6dcac41f5f","created":"2023-07-28T23:38:20.914874+0000","modified":"2023-07-28T23:38:37.888137+0000","last_up_change":"2023-07-28T23:38:29.239010+0000","last_in_change":"2023-07-28T23:38:21.802978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-28T23:38:30.997597+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-28T23:38:35.966808+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d871fec6-9c45-4792-8086-a94c0916c2ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":82192},{"type":"v1","addr":"172.21.15.139:6804","nonce":82192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":82192},{"type":"v1","addr":"172.21.15.139:6808","nonce":82192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":82192},{"type":"v1","addr":"172.21.15.139:6815","nonce":82192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":82192},{"type":"v1","addr":"172.21.15.139:6812","nonce":82192}]},"public_addr":"172.21.15.139:6804/82192","cluster_addr":"172.21.15.139:6808/82192","heartbeat_back_addr":"172.21.15.139:6815/82192","heartbeat_front_addr":"172.21.15.139:6812/82192","state":["exists","up"]},{"osd":1,"uuid":"367dcee6-86cb-4589-879a-d61a553e3369","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":82194},{"type":"v1","addr":"172.21.15.139:6817","nonce":82194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":82194},{"type":"v1","addr":"172.21.15.139:6819","nonce":82194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":82194},{"type":"v1","addr":"172.21.15.139:6823","nonce":82194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":82194},{"type":"v1","addr":"172.21.15.139:6821","nonce":82194}]},"public_addr":"172.21.15.139:6817/82194","cluster_addr":"172.21.15.139:6819/82194","heartbeat_back_addr":"172.21.15.139:6823/82194","heartbeat_front_addr":"172.21.15.139:6821/82194","state":["exists","up"]},{"osd":2,"uuid":"47016cf2-1bfc-42f2-b645-1556adf25996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":82193},{"type":"v1","addr":"172.21.15.139:6801","nonce":82193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6803","nonce":82193},{"type":"v1","addr":"172.21.15.139:6805","nonce":82193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6811","nonce":82193},{"type":"v1","addr":"172.21.15.139:6813","nonce":82193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6807","nonce":82193},{"type":"v1","addr":"172.21.15.139:6809","nonce":82193}]},"public_addr":"172.21.15.139:6801/82193","cluster_addr":"172.21.15.139:6805/82193","heartbeat_back_addr":"172.21.15.139:6813/82193","heartbeat_front_addr":"172.21.15.139:6809/82193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-28T23:38:26.147504+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-28T23:38:26.212465+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-28T23:38:26.173521+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-28T23:38:38.538 INFO:tasks.ceph.ceph_manager.c2:all up! 2023-07-28T23:38:38.538 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2023-07-28T23:38:38.812 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:38:38.812 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":13,"fsid":"5f79b2fe-1924-457c-a2b5-ff6dcac41f5f","created":"2023-07-28T23:38:20.914874+0000","modified":"2023-07-28T23:38:37.888137+0000","last_up_change":"2023-07-28T23:38:29.239010+0000","last_in_change":"2023-07-28T23:38:21.802978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-28T23:38:30.997597+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-28T23:38:35.966808+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d871fec6-9c45-4792-8086-a94c0916c2ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":82192},{"type":"v1","addr":"172.21.15.139:6804","nonce":82192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":82192},{"type":"v1","addr":"172.21.15.139:6808","nonce":82192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":82192},{"type":"v1","addr":"172.21.15.139:6815","nonce":82192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":82192},{"type":"v1","addr":"172.21.15.139:6812","nonce":82192}]},"public_addr":"172.21.15.139:6804/82192","cluster_addr":"172.21.15.139:6808/82192","heartbeat_back_addr":"172.21.15.139:6815/82192","heartbeat_front_addr":"172.21.15.139:6812/82192","state":["exists","up"]},{"osd":1,"uuid":"367dcee6-86cb-4589-879a-d61a553e3369","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":82194},{"type":"v1","addr":"172.21.15.139:6817","nonce":82194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":82194},{"type":"v1","addr":"172.21.15.139:6819","nonce":82194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":82194},{"type":"v1","addr":"172.21.15.139:6823","nonce":82194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":82194},{"type":"v1","addr":"172.21.15.139:6821","nonce":82194}]},"public_addr":"172.21.15.139:6817/82194","cluster_addr":"172.21.15.139:6819/82194","heartbeat_back_addr":"172.21.15.139:6823/82194","heartbeat_front_addr":"172.21.15.139:6821/82194","state":["exists","up"]},{"osd":2,"uuid":"47016cf2-1bfc-42f2-b645-1556adf25996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":82193},{"type":"v1","addr":"172.21.15.139:6801","nonce":82193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6803","nonce":82193},{"type":"v1","addr":"172.21.15.139:6805","nonce":82193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6811","nonce":82193},{"type":"v1","addr":"172.21.15.139:6813","nonce":82193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6807","nonce":82193},{"type":"v1","addr":"172.21.15.139:6809","nonce":82193}]},"public_addr":"172.21.15.139:6801/82193","cluster_addr":"172.21.15.139:6805/82193","heartbeat_back_addr":"172.21.15.139:6813/82193","heartbeat_front_addr":"172.21.15.139:6809/82193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-28T23:38:26.147504+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-28T23:38:26.212465+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-28T23:38:26.173521+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-28T23:38:38.824 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2023-07-28T23:38:38.825 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2023-07-28T23:38:38.825 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2023-07-28T23:38:38.973 INFO:teuthology.orchestra.run.smithi139.stdout:34359738373 2023-07-28T23:38:38.973 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2023-07-28T23:38:38.979 INFO:teuthology.orchestra.run.smithi139.stdout:34359738374 2023-07-28T23:38:38.979 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2023-07-28T23:38:38.984 INFO:teuthology.orchestra.run.smithi139.stdout:34359738374 2023-07-28T23:38:38.984 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2023-07-28T23:38:39.264 INFO:teuthology.orchestra.run.smithi139.stdout:34359738372 2023-07-28T23:38:39.277 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738373 got 34359738372 for osd.1 2023-07-28T23:38:39.315 INFO:teuthology.orchestra.run.smithi139.stdout:34359738373 2023-07-28T23:38:39.316 INFO:teuthology.orchestra.run.smithi139.stdout:34359738373 2023-07-28T23:38:39.328 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738374 got 34359738373 for osd.0 2023-07-28T23:38:39.329 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738374 got 34359738373 for osd.2 2023-07-28T23:38:40.278 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2023-07-28T23:38:40.329 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2023-07-28T23:38:40.329 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2023-07-28T23:38:40.580 INFO:teuthology.orchestra.run.smithi139.stdout:34359738372 2023-07-28T23:38:40.592 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738373 got 34359738372 for osd.1 2023-07-28T23:38:40.623 INFO:teuthology.orchestra.run.smithi139.stdout:34359738373 2023-07-28T23:38:40.627 INFO:teuthology.orchestra.run.smithi139.stdout:34359738373 2023-07-28T23:38:40.637 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738374 got 34359738373 for osd.2 2023-07-28T23:38:40.640 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738374 got 34359738373 for osd.0 2023-07-28T23:38:41.593 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2023-07-28T23:38:41.637 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2023-07-28T23:38:41.641 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2023-07-28T23:38:41.933 INFO:teuthology.orchestra.run.smithi139.stdout:34359738374 2023-07-28T23:38:41.945 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738373 got 34359738374 for osd.1 2023-07-28T23:38:41.946 DEBUG:teuthology.parallel:result is None 2023-07-28T23:38:41.969 INFO:teuthology.orchestra.run.smithi139.stdout:34359738375 2023-07-28T23:38:41.971 INFO:teuthology.orchestra.run.smithi139.stdout:34359738375 2023-07-28T23:38:41.985 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738374 got 34359738375 for osd.0 2023-07-28T23:38:41.985 DEBUG:teuthology.parallel:result is None 2023-07-28T23:38:41.987 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738374 got 34359738375 for osd.2 2023-07-28T23:38:41.987 DEBUG:teuthology.parallel:result is None 2023-07-28T23:38:41.988 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2023-07-28T23:38:41.988 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2023-07-28T23:38:42.300 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:38:42.300 INFO:teuthology.orchestra.run.smithi139.stderr:dumped all 2023-07-28T23:38:42.312 INFO:teuthology.orchestra.run.smithi139.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2023-07-28T23:38:40.959991+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82368,"kb_used_data":1512,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490432,"statfs":{"total":322122547200,"available":322038202368,"internally_reserved":0,"allocated":1548288,"data_stored":1281649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.069423"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-07-28T23:38:38.896356+0000","last_change":"2023-07-28T23:38:37.897536+0000","last_active":"2023-07-28T23:38:38.896356+0000","last_peered":"2023-07-28T23:38:38.896356+0000","last_clean":"2023-07-28T23:38:38.896356+0000","last_became_active":"2023-07-28T23:38:37.897242+0000","last_became_peered":"2023-07-28T23:38:37.897242+0000","last_unstale":"2023-07-28T23:38:38.896356+0000","last_undegraded":"2023-07-28T23:38:38.896356+0000","last_fullsized":"2023-07-28T23:38:38.896356+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-28T23:38:36.887044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-28T23:38:36.887044+0000","last_clean_scrub_stamp":"2023-07-28T23:38:36.887044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-29T23:42:04.863366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-07-28T23:38:38.896505+0000","last_change":"2023-07-28T23:38:37.898521+0000","last_active":"2023-07-28T23:38:38.896505+0000","last_peered":"2023-07-28T23:38:38.896505+0000","last_clean":"2023-07-28T23:38:38.896505+0000","last_became_active":"2023-07-28T23:38:37.898272+0000","last_became_peered":"2023-07-28T23:38:37.898272+0000","last_unstale":"2023-07-28T23:38:38.896505+0000","last_undegraded":"2023-07-28T23:38:38.896505+0000","last_fullsized":"2023-07-28T23:38:38.896505+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-28T23:38:36.887044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-28T23:38:36.887044+0000","last_clean_scrub_stamp":"2023-07-28T23:38:36.887044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T08:11:07.968483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-07-28T23:38:38.896467+0000","last_change":"2023-07-28T23:38:37.897459+0000","last_active":"2023-07-28T23:38:38.896467+0000","last_peered":"2023-07-28T23:38:38.896467+0000","last_clean":"2023-07-28T23:38:38.896467+0000","last_became_active":"2023-07-28T23:38:37.897228+0000","last_became_peered":"2023-07-28T23:38:37.897228+0000","last_unstale":"2023-07-28T23:38:38.896467+0000","last_undegraded":"2023-07-28T23:38:38.896467+0000","last_fullsized":"2023-07-28T23:38:38.896467+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-28T23:38:36.887044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-28T23:38:36.887044+0000","last_clean_scrub_stamp":"2023-07-28T23:38:36.887044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T10:10:22.509382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-07-28T23:38:38.896476+0000","last_change":"2023-07-28T23:38:37.897492+0000","last_active":"2023-07-28T23:38:38.896476+0000","last_peered":"2023-07-28T23:38:38.896476+0000","last_clean":"2023-07-28T23:38:38.896476+0000","last_became_active":"2023-07-28T23:38:37.897204+0000","last_became_peered":"2023-07-28T23:38:37.897204+0000","last_unstale":"2023-07-28T23:38:38.896476+0000","last_undegraded":"2023-07-28T23:38:38.896476+0000","last_fullsized":"2023-07-28T23:38:38.896476+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-28T23:38:36.887044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-28T23:38:36.887044+0000","last_clean_scrub_stamp":"2023-07-28T23:38:36.887044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T08:13:28.525757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-07-28T23:38:38.975286+0000","last_change":"2023-07-28T23:38:37.898959+0000","last_active":"2023-07-28T23:38:38.975286+0000","last_peered":"2023-07-28T23:38:38.975286+0000","last_clean":"2023-07-28T23:38:38.975286+0000","last_became_active":"2023-07-28T23:38:37.898556+0000","last_became_peered":"2023-07-28T23:38:37.898556+0000","last_unstale":"2023-07-28T23:38:38.975286+0000","last_undegraded":"2023-07-28T23:38:38.975286+0000","last_fullsized":"2023-07-28T23:38:38.975286+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-28T23:38:36.887044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-28T23:38:36.887044+0000","last_clean_scrub_stamp":"2023-07-28T23:38:36.887044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T10:22:15.677632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-07-28T23:38:38.967593+0000","last_change":"2023-07-28T23:38:37.898898+0000","last_active":"2023-07-28T23:38:38.967593+0000","last_peered":"2023-07-28T23:38:38.967593+0000","last_clean":"2023-07-28T23:38:38.967593+0000","last_became_active":"2023-07-28T23:38:37.898629+0000","last_became_peered":"2023-07-28T23:38:37.898629+0000","last_unstale":"2023-07-28T23:38:38.967593+0000","last_undegraded":"2023-07-28T23:38:38.967593+0000","last_fullsized":"2023-07-28T23:38:38.967593+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-28T23:38:36.887044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-28T23:38:36.887044+0000","last_clean_scrub_stamp":"2023-07-28T23:38:36.887044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T07:31:48.584176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-07-28T23:38:38.967567+0000","last_change":"2023-07-28T23:38:37.899608+0000","last_active":"2023-07-28T23:38:38.967567+0000","last_peered":"2023-07-28T23:38:38.967567+0000","last_clean":"2023-07-28T23:38:38.967567+0000","last_became_active":"2023-07-28T23:38:37.899240+0000","last_became_peered":"2023-07-28T23:38:37.899240+0000","last_unstale":"2023-07-28T23:38:38.967567+0000","last_undegraded":"2023-07-28T23:38:38.967567+0000","last_fullsized":"2023-07-28T23:38:38.967567+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-28T23:38:36.887044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-28T23:38:36.887044+0000","last_clean_scrub_stamp":"2023-07-28T23:38:36.887044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T02:43:07.978921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-07-28T23:38:38.896408+0000","last_change":"2023-07-28T23:38:37.896513+0000","last_active":"2023-07-28T23:38:38.896408+0000","last_peered":"2023-07-28T23:38:38.896408+0000","last_clean":"2023-07-28T23:38:38.896408+0000","last_became_active":"2023-07-28T23:38:37.896249+0000","last_became_peered":"2023-07-28T23:38:37.896249+0000","last_unstale":"2023-07-28T23:38:38.896408+0000","last_undegraded":"2023-07-28T23:38:38.896408+0000","last_fullsized":"2023-07-28T23:38:38.896408+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-28T23:38:36.887044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-28T23:38:36.887044+0000","last_clean_scrub_stamp":"2023-07-28T23:38:36.887044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T00:44:19.211549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":140,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-07-28T23:38:37.892976+0000","last_change":"2023-07-28T23:38:32.057572+0000","last_active":"2023-07-28T23:38:37.892976+0000","last_peered":"2023-07-28T23:38:37.892976+0000","last_clean":"2023-07-28T23:38:37.892976+0000","last_became_active":"2023-07-28T23:38:32.056590+0000","last_became_peered":"2023-07-28T23:38:32.056590+0000","last_unstale":"2023-07-28T23:38:37.892976+0000","last_undegraded":"2023-07-28T23:38:37.892976+0000","last_fullsized":"2023-07-28T23:38:37.892976+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-28T23:38:31.045113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-28T23:38:31.045113+0000","last_clean_scrub_stamp":"2023-07-28T23:38:31.045113+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T08:09:51.563425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738375,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27064,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830536,"statfs":{"total":107374182400,"available":107346468864,"internally_reserved":0,"allocated":114688,"data_stored":31319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27652,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829948,"statfs":{"total":107374182400,"available":107345866752,"internally_reserved":0,"allocated":716800,"data_stored":625165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.311}]}]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27652,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829948,"statfs":{"total":107374182400,"available":107345866752,"internally_reserved":0,"allocated":716800,"data_stored":625165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.251}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-28T23:38:42.313 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2023-07-28T23:38:42.584 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:38:42.585 INFO:teuthology.orchestra.run.smithi139.stderr:dumped all 2023-07-28T23:38:42.596 INFO:teuthology.orchestra.run.smithi139.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2023-07-28T23:38:40.959991+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82368,"kb_used_data":1512,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490432,"statfs":{"total":322122547200,"available":322038202368,"internally_reserved":0,"allocated":1548288,"data_stored":1281649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.069423"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-07-28T23:38:38.896356+0000","last_change":"2023-07-28T23:38:37.897536+0000","last_active":"2023-07-28T23:38:38.896356+0000","last_peered":"2023-07-28T23:38:38.896356+0000","last_clean":"2023-07-28T23:38:38.896356+0000","last_became_active":"2023-07-28T23:38:37.897242+0000","last_became_peered":"2023-07-28T23:38:37.897242+0000","last_unstale":"2023-07-28T23:38:38.896356+0000","last_undegraded":"2023-07-28T23:38:38.896356+0000","last_fullsized":"2023-07-28T23:38:38.896356+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-28T23:38:36.887044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-28T23:38:36.887044+0000","last_clean_scrub_stamp":"2023-07-28T23:38:36.887044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-29T23:42:04.863366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-07-28T23:38:38.896505+0000","last_change":"2023-07-28T23:38:37.898521+0000","last_active":"2023-07-28T23:38:38.896505+0000","last_peered":"2023-07-28T23:38:38.896505+0000","last_clean":"2023-07-28T23:38:38.896505+0000","last_became_active":"2023-07-28T23:38:37.898272+0000","last_became_peered":"2023-07-28T23:38:37.898272+0000","last_unstale":"2023-07-28T23:38:38.896505+0000","last_undegraded":"2023-07-28T23:38:38.896505+0000","last_fullsized":"2023-07-28T23:38:38.896505+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-28T23:38:36.887044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-28T23:38:36.887044+0000","last_clean_scrub_stamp":"2023-07-28T23:38:36.887044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T08:11:07.968483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-07-28T23:38:38.896467+0000","last_change":"2023-07-28T23:38:37.897459+0000","last_active":"2023-07-28T23:38:38.896467+0000","last_peered":"2023-07-28T23:38:38.896467+0000","last_clean":"2023-07-28T23:38:38.896467+0000","last_became_active":"2023-07-28T23:38:37.897228+0000","last_became_peered":"2023-07-28T23:38:37.897228+0000","last_unstale":"2023-07-28T23:38:38.896467+0000","last_undegraded":"2023-07-28T23:38:38.896467+0000","last_fullsized":"2023-07-28T23:38:38.896467+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-28T23:38:36.887044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-28T23:38:36.887044+0000","last_clean_scrub_stamp":"2023-07-28T23:38:36.887044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T10:10:22.509382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-07-28T23:38:38.896476+0000","last_change":"2023-07-28T23:38:37.897492+0000","last_active":"2023-07-28T23:38:38.896476+0000","last_peered":"2023-07-28T23:38:38.896476+0000","last_clean":"2023-07-28T23:38:38.896476+0000","last_became_active":"2023-07-28T23:38:37.897204+0000","last_became_peered":"2023-07-28T23:38:37.897204+0000","last_unstale":"2023-07-28T23:38:38.896476+0000","last_undegraded":"2023-07-28T23:38:38.896476+0000","last_fullsized":"2023-07-28T23:38:38.896476+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-28T23:38:36.887044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-28T23:38:36.887044+0000","last_clean_scrub_stamp":"2023-07-28T23:38:36.887044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T08:13:28.525757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-07-28T23:38:38.975286+0000","last_change":"2023-07-28T23:38:37.898959+0000","last_active":"2023-07-28T23:38:38.975286+0000","last_peered":"2023-07-28T23:38:38.975286+0000","last_clean":"2023-07-28T23:38:38.975286+0000","last_became_active":"2023-07-28T23:38:37.898556+0000","last_became_peered":"2023-07-28T23:38:37.898556+0000","last_unstale":"2023-07-28T23:38:38.975286+0000","last_undegraded":"2023-07-28T23:38:38.975286+0000","last_fullsized":"2023-07-28T23:38:38.975286+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-28T23:38:36.887044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-28T23:38:36.887044+0000","last_clean_scrub_stamp":"2023-07-28T23:38:36.887044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T10:22:15.677632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-07-28T23:38:38.967593+0000","last_change":"2023-07-28T23:38:37.898898+0000","last_active":"2023-07-28T23:38:38.967593+0000","last_peered":"2023-07-28T23:38:38.967593+0000","last_clean":"2023-07-28T23:38:38.967593+0000","last_became_active":"2023-07-28T23:38:37.898629+0000","last_became_peered":"2023-07-28T23:38:37.898629+0000","last_unstale":"2023-07-28T23:38:38.967593+0000","last_undegraded":"2023-07-28T23:38:38.967593+0000","last_fullsized":"2023-07-28T23:38:38.967593+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-28T23:38:36.887044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-28T23:38:36.887044+0000","last_clean_scrub_stamp":"2023-07-28T23:38:36.887044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T07:31:48.584176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-07-28T23:38:38.967567+0000","last_change":"2023-07-28T23:38:37.899608+0000","last_active":"2023-07-28T23:38:38.967567+0000","last_peered":"2023-07-28T23:38:38.967567+0000","last_clean":"2023-07-28T23:38:38.967567+0000","last_became_active":"2023-07-28T23:38:37.899240+0000","last_became_peered":"2023-07-28T23:38:37.899240+0000","last_unstale":"2023-07-28T23:38:38.967567+0000","last_undegraded":"2023-07-28T23:38:38.967567+0000","last_fullsized":"2023-07-28T23:38:38.967567+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-28T23:38:36.887044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-28T23:38:36.887044+0000","last_clean_scrub_stamp":"2023-07-28T23:38:36.887044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T02:43:07.978921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-07-28T23:38:38.896408+0000","last_change":"2023-07-28T23:38:37.896513+0000","last_active":"2023-07-28T23:38:38.896408+0000","last_peered":"2023-07-28T23:38:38.896408+0000","last_clean":"2023-07-28T23:38:38.896408+0000","last_became_active":"2023-07-28T23:38:37.896249+0000","last_became_peered":"2023-07-28T23:38:37.896249+0000","last_unstale":"2023-07-28T23:38:38.896408+0000","last_undegraded":"2023-07-28T23:38:38.896408+0000","last_fullsized":"2023-07-28T23:38:38.896408+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-28T23:38:36.887044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-28T23:38:36.887044+0000","last_clean_scrub_stamp":"2023-07-28T23:38:36.887044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T00:44:19.211549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":140,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-07-28T23:38:37.892976+0000","last_change":"2023-07-28T23:38:32.057572+0000","last_active":"2023-07-28T23:38:37.892976+0000","last_peered":"2023-07-28T23:38:37.892976+0000","last_clean":"2023-07-28T23:38:37.892976+0000","last_became_active":"2023-07-28T23:38:32.056590+0000","last_became_peered":"2023-07-28T23:38:32.056590+0000","last_unstale":"2023-07-28T23:38:37.892976+0000","last_undegraded":"2023-07-28T23:38:37.892976+0000","last_fullsized":"2023-07-28T23:38:37.892976+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-28T23:38:31.045113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-28T23:38:31.045113+0000","last_clean_scrub_stamp":"2023-07-28T23:38:31.045113+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T08:09:51.563425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738375,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27064,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830536,"statfs":{"total":107374182400,"available":107346468864,"internally_reserved":0,"allocated":114688,"data_stored":31319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27652,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829948,"statfs":{"total":107374182400,"available":107345866752,"internally_reserved":0,"allocated":716800,"data_stored":625165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.311}]}]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27652,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829948,"statfs":{"total":107374182400,"available":107345866752,"internally_reserved":0,"allocated":716800,"data_stored":625165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.251}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-28T23:38:42.597 INFO:tasks.ceph.ceph_manager.c2:clean! 2023-07-28T23:38:42.597 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2023-07-28T23:38:42.597 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2023-07-28T23:38:42.598 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2023-07-28T23:38:42.902 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-28T23:38:42.902 INFO:teuthology.orchestra.run.smithi139.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-07-28T23:38:42.914 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2023-07-28T23:38:42.915 INFO:teuthology.run_tasks:Running task rgw... 2023-07-28T23:38:42.965 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2023-07-28T23:38:42.965 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2023-07-28T23:38:42.966 INFO:tasks.rgw:Creating data pools 2023-07-28T23:38:42.966 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2023-07-28T23:38:42.967 DEBUG:teuthology.orchestra.run.smithi101:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2023-07-28T23:38:44.170 INFO:teuthology.orchestra.run.smithi101.stderr:pool 'default.rgw.buckets.data' created 2023-07-28T23:38:44.190 DEBUG:teuthology.orchestra.run.smithi101:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2023-07-28T23:38:45.176 INFO:teuthology.orchestra.run.smithi101.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-07-28T23:38:45.205 DEBUG:teuthology.orchestra.run.smithi101:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2023-07-28T23:38:46.184 INFO:teuthology.orchestra.run.smithi101.stderr:pool 'default.rgw.buckets.index' created 2023-07-28T23:38:46.207 DEBUG:teuthology.orchestra.run.smithi101:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2023-07-28T23:38:47.186 INFO:teuthology.orchestra.run.smithi101.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-07-28T23:38:47.217 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2023-07-28T23:38:47.217 DEBUG:teuthology.orchestra.run.smithi101:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2023-07-28T23:38:47.494 INFO:teuthology.orchestra.run.smithi101.stderr:pool 'default.rgw.buckets.data' already exists 2023-07-28T23:38:47.506 DEBUG:teuthology.orchestra.run.smithi101:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2023-07-28T23:38:48.185 INFO:teuthology.orchestra.run.smithi101.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-07-28T23:38:48.198 DEBUG:teuthology.orchestra.run.smithi101:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2023-07-28T23:38:48.472 INFO:teuthology.orchestra.run.smithi101.stderr:pool 'default.rgw.buckets.index' already exists 2023-07-28T23:38:48.485 DEBUG:teuthology.orchestra.run.smithi101:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2023-07-28T23:38:49.189 INFO:teuthology.orchestra.run.smithi101.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-07-28T23:38:49.203 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2023-07-28T23:38:49.203 DEBUG:teuthology.orchestra.run.smithi139:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2023-07-28T23:38:50.025 INFO:teuthology.orchestra.run.smithi139.stderr:pool 'default.rgw.buckets.data' created 2023-07-28T23:38:50.048 DEBUG:teuthology.orchestra.run.smithi139:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2023-07-28T23:38:51.033 INFO:teuthology.orchestra.run.smithi139.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-07-28T23:38:51.067 DEBUG:teuthology.orchestra.run.smithi139:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2023-07-28T23:38:52.032 INFO:teuthology.orchestra.run.smithi139.stderr:pool 'default.rgw.buckets.index' created 2023-07-28T23:38:52.057 DEBUG:teuthology.orchestra.run.smithi139:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2023-07-28T23:38:53.041 INFO:teuthology.orchestra.run.smithi139.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-07-28T23:38:53.073 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2023-07-28T23:38:53.073 DEBUG:teuthology.orchestra.run.smithi139:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2023-07-28T23:38:53.352 INFO:teuthology.orchestra.run.smithi139.stderr:pool 'default.rgw.buckets.data' already exists 2023-07-28T23:38:53.364 DEBUG:teuthology.orchestra.run.smithi139:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2023-07-28T23:38:54.040 INFO:teuthology.orchestra.run.smithi139.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-07-28T23:38:54.053 DEBUG:teuthology.orchestra.run.smithi139:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2023-07-28T23:38:54.325 INFO:teuthology.orchestra.run.smithi139.stderr:pool 'default.rgw.buckets.index' already exists 2023-07-28T23:38:54.337 DEBUG:teuthology.orchestra.run.smithi139:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2023-07-28T23:38:55.043 INFO:teuthology.orchestra.run.smithi139.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-07-28T23:38:55.057 DEBUG:tasks.rgw:Pools created 2023-07-28T23:38:55.058 INFO:tasks.rgw:Configuring compression type = random 2023-07-28T23:38:55.058 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2023-07-28T23:38:55.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2023-07-28T23:38:55.059 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2023-07-28T23:38:55.152 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:38:55.152 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:38:55.163 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:55.164+0000 7f705f80f880 20 rados->read ofs=0 len=0 2023-07-28T23:38:57.137 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:57.139+0000 7f705f80f880 20 rados_obj.operate() r=-2 bl.length=0 2023-07-28T23:38:57.138 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:57.139+0000 7f705f80f880 20 realm 2023-07-28T23:38:57.138 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:57.139+0000 7f705f80f880 20 rados->read ofs=0 len=0 2023-07-28T23:38:57.138 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:57.139+0000 7f705f80f880 20 rados_obj.operate() r=-2 bl.length=0 2023-07-28T23:38:57.138 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:57.139+0000 7f705f80f880 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-07-28T23:38:57.138 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:57.139+0000 7f705f80f880 20 rados->read ofs=0 len=0 2023-07-28T23:38:57.139 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:57.140+0000 7f705f80f880 20 rados_obj.operate() r=-2 bl.length=0 2023-07-28T23:38:57.139 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:57.140+0000 7f705f80f880 20 rados->read ofs=0 len=0 2023-07-28T23:38:57.142 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:57.144+0000 7f705f80f880 20 rados_obj.operate() r=-2 bl.length=0 2023-07-28T23:38:57.142 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:57.144+0000 7f705f80f880 20 rados->read ofs=0 len=0 2023-07-28T23:38:57.143 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:57.144+0000 7f705f80f880 20 rados_obj.operate() r=-2 bl.length=0 2023-07-28T23:38:57.143 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:57.144+0000 7f705f80f880 20 rados->read ofs=0 len=0 2023-07-28T23:38:57.143 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:57.145+0000 7f705f80f880 20 rados_obj.operate() r=-2 bl.length=0 2023-07-28T23:38:57.144 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:57.145+0000 7f705f80f880 10 Creating default zonegroup 2023-07-28T23:38:57.144 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:57.145+0000 7f705f80f880 10 couldn't find old data placement pools config, setting up new ones for the zone 2023-07-28T23:38:57.153 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:57.154+0000 7f705f80f880 10 WARNING: driver->list_zones() returned r=-2 2023-07-28T23:38:57.153 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:57.154+0000 7f705f80f880 20 rados->read ofs=0 len=0 2023-07-28T23:38:57.153 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:57.154+0000 7f705f80f880 20 rados_obj.operate() r=-2 bl.length=0 2023-07-28T23:38:57.157 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:57.159+0000 7f705f80f880 20 rados->read ofs=0 len=0 2023-07-28T23:38:57.157 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:57.159+0000 7f705f80f880 20 rados_obj.operate() r=-2 bl.length=0 2023-07-28T23:38:57.157 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:57.159+0000 7f705f80f880 10 could not read realm id: (2) No such file or directory 2023-07-28T23:38:57.158 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:57.159+0000 7f705f80f880 10 WARNING: failed to set zone as default, r=-22 2023-07-28T23:38:57.158 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:57.159+0000 7f705f80f880 20 rados->read ofs=0 len=0 2023-07-28T23:38:57.158 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:57.159+0000 7f705f80f880 20 rados_obj.operate() r=-2 bl.length=0 2023-07-28T23:38:57.159 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:57.161+0000 7f705f80f880 20 rados->read ofs=0 len=0 2023-07-28T23:38:57.160 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:57.161+0000 7f705f80f880 20 rados_obj.operate() r=0 bl.length=874 2023-07-28T23:38:57.160 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:57.161+0000 7f705f80f880 20 rados->read ofs=0 len=0 2023-07-28T23:38:57.162 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:57.164+0000 7f705f80f880 20 rados_obj.operate() r=0 bl.length=420 2023-07-28T23:38:57.162 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:57.164+0000 7f705f80f880 10 Using default name default 2023-07-28T23:38:57.162 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:57.164+0000 7f705f80f880 20 rados->read ofs=0 len=0 2023-07-28T23:38:57.163 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:57.164+0000 7f705f80f880 20 rados_obj.operate() r=0 bl.length=46 2023-07-28T23:38:57.163 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:57.164+0000 7f705f80f880 20 rados->read ofs=0 len=0 2023-07-28T23:38:57.163 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:57.164+0000 7f705f80f880 20 rados_obj.operate() r=0 bl.length=874 2023-07-28T23:38:57.163 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:57.164+0000 7f705f80f880 20 zone default found 2023-07-28T23:38:57.164 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:57.164+0000 7f705f80f880 4 Realm: () 2023-07-28T23:38:57.164 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:57.164+0000 7f705f80f880 4 ZoneGroup: default (79116a35-506a-46f0-9b7d-a6466efb23c9) 2023-07-28T23:38:57.164 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:57.164+0000 7f705f80f880 4 Zone: default (2ef2953a-3e98-4529-8f07-1a1fcae427b8) 2023-07-28T23:38:57.164 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:57.164+0000 7f705f80f880 10 cannot find current period zonegroup using local zonegroup configuration 2023-07-28T23:38:57.164 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:57.164+0000 7f705f80f880 20 zonegroup default 2023-07-28T23:38:57.164 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:57.164+0000 7f705f80f880 20 rados->read ofs=0 len=0 2023-07-28T23:38:57.165 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:57.164+0000 7f705f80f880 20 rados_obj.operate() r=-2 bl.length=0 2023-07-28T23:38:57.165 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:57.165+0000 7f705f80f880 20 rados->read ofs=0 len=0 2023-07-28T23:38:59.153 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:59.155+0000 7f705f80f880 20 rados_obj.operate() r=-2 bl.length=0 2023-07-28T23:38:59.154 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:59.155+0000 7f705f80f880 20 rados->read ofs=0 len=0 2023-07-28T23:38:59.154 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:59.155+0000 7f705f80f880 20 rados_obj.operate() r=-2 bl.length=0 2023-07-28T23:38:59.154 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:59.155+0000 7f705f80f880 20 started sync module instance, tier type = 2023-07-28T23:38:59.154 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:38:59.155+0000 7f705f80f880 20 started zone id=2ef2953a-3e98-4529-8f07-1a1fcae427b8 (name=default) with tier type = 2023-07-28T23:39:01.194 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:01.196+0000 7f705f80f880 20 add_watcher() i=0 2023-07-28T23:39:01.195 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:01.196+0000 7f705f80f880 20 add_watcher() i=1 2023-07-28T23:39:01.195 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:01.196+0000 7f705f80f880 20 add_watcher() i=2 2023-07-28T23:39:01.195 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:01.196+0000 7f705f80f880 20 add_watcher() i=3 2023-07-28T23:39:01.195 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:01.196+0000 7f705f80f880 20 add_watcher() i=4 2023-07-28T23:39:01.196 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:01.196+0000 7f705f80f880 20 add_watcher() i=5 2023-07-28T23:39:01.196 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:01.196+0000 7f705f80f880 20 add_watcher() i=6 2023-07-28T23:39:01.197 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:01.197+0000 7f705f80f880 20 add_watcher() i=7 2023-07-28T23:39:01.197 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:01.197+0000 7f705f80f880 2 all 8 watchers are set, enabling cache 2023-07-28T23:39:01.197 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:01.197+0000 7f705f80f880 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2023-07-28T23:39:01.197 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:01.197+0000 7f705f80f880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2023-07-28T23:39:01.197 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:01.197+0000 7f705f80f880 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-07-28T23:39:01.197 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:01.198+0000 7f705f80f880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2023-07-28T23:39:01.198 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:01.198+0000 7f705f80f880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2023-07-28T23:39:01.198 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:01.198+0000 7f705f80f880 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-07-28T23:39:01.198 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:01.198+0000 7f705f80f880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2023-07-28T23:39:01.198 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:01.198+0000 7f705f80f880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2023-07-28T23:39:01.198 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:01.198+0000 7f705f80f880 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-07-28T23:39:01.198 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:01.198+0000 7f705f80f880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2023-07-28T23:39:01.198 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:01.198+0000 7f705f80f880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2023-07-28T23:39:01.199 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:01.198+0000 7f705f80f880 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-07-28T23:39:01.199 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:01.198+0000 7f705f80f880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2023-07-28T23:39:01.199 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:01.198+0000 7f705f80f880 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2023-07-28T23:39:01.199 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:01.200+0000 7f705f80f880 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-07-28T23:39:01.203 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:01.205+0000 7f7042c72640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-07-28T23:39:01.203 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:01.205+0000 7f705f80f880 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-07-28T23:39:01.203 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:01.205+0000 7f705f80f880 20 check_secure_mon_conn(): mode 1 is insecure 2023-07-28T23:39:04.157 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.158+0000 7f705f80f880 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2023-07-28T23:39:04.157 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.158+0000 7f705f80f880 5 note: GC not initialized 2023-07-28T23:39:04.158 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.159+0000 7f703e469640 20 reqs_thread_entry: start 2023-07-28T23:39:04.158 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.159+0000 7f705f80f880 20 init_complete bucket index max shards: 11 2023-07-28T23:39:04.158 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.159+0000 7f705f80f880 10 rgw notify: Started notification manager with: 1 workers 2023-07-28T23:39:04.158 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.159+0000 7f703c465640 20 rgw notify: INFO: next queues processing will happen at: Fri Jul 28 23:39:34 2023 2023-07-28T23:39:04.158 INFO:teuthology.orchestra.run.smithi101.stderr: 2023-07-28T23:39:04.159 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.160+0000 7f703bc43640 20 reqs_thread_entry: start 2023-07-28T23:39:04.179 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.180+0000 7f7042c72640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-07-28T23:39:04.181 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.183+0000 7f705f80f880 20 remove_watcher() i=0 2023-07-28T23:39:04.181 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.183+0000 7f705f80f880 2 removed watcher, disabling cache 2023-07-28T23:39:04.182 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.184+0000 7f705f80f880 20 remove_watcher() i=1 2023-07-28T23:39:04.183 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.185+0000 7f705f80f880 20 remove_watcher() i=2 2023-07-28T23:39:04.186 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.187+0000 7f705f80f880 20 remove_watcher() i=3 2023-07-28T23:39:04.186 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.188+0000 7f705f80f880 20 remove_watcher() i=4 2023-07-28T23:39:04.188 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.189+0000 7f705f80f880 20 remove_watcher() i=5 2023-07-28T23:39:04.188 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.190+0000 7f705f80f880 20 remove_watcher() i=6 2023-07-28T23:39:04.190 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.191+0000 7f705f80f880 20 remove_watcher() i=7 2023-07-28T23:39:04.207 INFO:teuthology.orchestra.run.smithi101.stdout:[] 2023-07-28T23:39:04.210 DEBUG:tasks.util.rgw: json result: [] 2023-07-28T23:39:04.210 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-07-28T23:39:04.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-07-28T23:39:04.211 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-07-28T23:39:04.261 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:39:04.261 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:39:04.271 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.273+0000 7ff5be528880 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-07-28T23:39:04.272 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.273+0000 7ff5be528880 20 check_secure_mon_conn(): mode 1 is insecure 2023-07-28T23:39:04.274 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.276+0000 7ff59e660640 20 reqs_thread_entry: start 2023-07-28T23:39:04.281 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:39:04.281 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "2ef2953a-3e98-4529-8f07-1a1fcae427b8", 2023-07-28T23:39:04.281 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default", 2023-07-28T23:39:04.281 INFO:teuthology.orchestra.run.smithi101.stdout: "domain_root": "default.rgw.meta:root", 2023-07-28T23:39:04.281 INFO:teuthology.orchestra.run.smithi101.stdout: "control_pool": "default.rgw.control", 2023-07-28T23:39:04.281 INFO:teuthology.orchestra.run.smithi101.stdout: "gc_pool": "default.rgw.log:gc", 2023-07-28T23:39:04.282 INFO:teuthology.orchestra.run.smithi101.stdout: "lc_pool": "default.rgw.log:lc", 2023-07-28T23:39:04.282 INFO:teuthology.orchestra.run.smithi101.stdout: "log_pool": "default.rgw.log", 2023-07-28T23:39:04.282 INFO:teuthology.orchestra.run.smithi101.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-07-28T23:39:04.282 INFO:teuthology.orchestra.run.smithi101.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-07-28T23:39:04.282 INFO:teuthology.orchestra.run.smithi101.stdout: "roles_pool": "default.rgw.meta:roles", 2023-07-28T23:39:04.282 INFO:teuthology.orchestra.run.smithi101.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-07-28T23:39:04.282 INFO:teuthology.orchestra.run.smithi101.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-07-28T23:39:04.283 INFO:teuthology.orchestra.run.smithi101.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-07-28T23:39:04.283 INFO:teuthology.orchestra.run.smithi101.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-07-28T23:39:04.283 INFO:teuthology.orchestra.run.smithi101.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-07-28T23:39:04.283 INFO:teuthology.orchestra.run.smithi101.stdout: "otp_pool": "default.rgw.otp", 2023-07-28T23:39:04.283 INFO:teuthology.orchestra.run.smithi101.stdout: "system_key": { 2023-07-28T23:39:04.283 INFO:teuthology.orchestra.run.smithi101.stdout: "access_key": "", 2023-07-28T23:39:04.283 INFO:teuthology.orchestra.run.smithi101.stdout: "secret_key": "" 2023-07-28T23:39:04.283 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:39:04.284 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_pools": [ 2023-07-28T23:39:04.284 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:39:04.284 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "default-placement", 2023-07-28T23:39:04.284 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:39:04.284 INFO:teuthology.orchestra.run.smithi101.stdout: "index_pool": "default.rgw.buckets.index", 2023-07-28T23:39:04.284 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": { 2023-07-28T23:39:04.284 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD": { 2023-07-28T23:39:04.285 INFO:teuthology.orchestra.run.smithi101.stdout: "data_pool": "default.rgw.buckets.data", 2023-07-28T23:39:04.285 INFO:teuthology.orchestra.run.smithi101.stdout: "compression_type": "random" 2023-07-28T23:39:04.285 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:39:04.285 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:39:04.285 INFO:teuthology.orchestra.run.smithi101.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-07-28T23:39:04.285 INFO:teuthology.orchestra.run.smithi101.stdout: "index_type": 0, 2023-07-28T23:39:04.285 INFO:teuthology.orchestra.run.smithi101.stdout: "inline_data": true 2023-07-28T23:39:04.285 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:39:04.286 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:39:04.286 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:39:04.286 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "", 2023-07-28T23:39:04.286 INFO:teuthology.orchestra.run.smithi101.stdout: "notif_pool": "default.rgw.log:notif" 2023-07-28T23:39:04.286 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:39:04.287 DEBUG:tasks.util.rgw: json result: {'id': '2ef2953a-3e98-4529-8f07-1a1fcae427b8', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-07-28T23:39:04.287 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2023-07-28T23:39:04.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2023-07-28T23:39:04.288 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2023-07-28T23:39:04.329 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:39:04.329 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:39:04.339 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.340+0000 7f9a6f526880 20 rados->read ofs=0 len=0 2023-07-28T23:39:04.339 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.341+0000 7f9a6f526880 20 rados_obj.operate() r=-2 bl.length=0 2023-07-28T23:39:04.340 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.341+0000 7f9a6f526880 20 realm 2023-07-28T23:39:04.340 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.341+0000 7f9a6f526880 20 rados->read ofs=0 len=0 2023-07-28T23:39:04.340 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.341+0000 7f9a6f526880 20 rados_obj.operate() r=-2 bl.length=0 2023-07-28T23:39:04.340 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.341+0000 7f9a6f526880 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-07-28T23:39:04.340 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.341+0000 7f9a6f526880 20 rados->read ofs=0 len=0 2023-07-28T23:39:04.340 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.341+0000 7f9a6f526880 20 rados_obj.operate() r=-2 bl.length=0 2023-07-28T23:39:04.341 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.341+0000 7f9a6f526880 20 rados->read ofs=0 len=0 2023-07-28T23:39:04.341 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.342+0000 7f9a6f526880 20 rados_obj.operate() r=0 bl.length=46 2023-07-28T23:39:04.341 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.342+0000 7f9a6f526880 20 rados->read ofs=0 len=0 2023-07-28T23:39:04.342 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.342+0000 7f9a6f526880 20 rados_obj.operate() r=0 bl.length=890 2023-07-28T23:39:04.342 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.342+0000 7f9a6f526880 20 searching for the correct realm 2023-07-28T23:39:04.347 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.349+0000 7f9a6f526880 20 RGWRados::pool_iterate: got zone_info.2ef2953a-3e98-4529-8f07-1a1fcae427b8 2023-07-28T23:39:04.348 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.349+0000 7f9a6f526880 20 RGWRados::pool_iterate: got zonegroup_info.79116a35-506a-46f0-9b7d-a6466efb23c9 2023-07-28T23:39:04.348 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.349+0000 7f9a6f526880 20 RGWRados::pool_iterate: got zone_names.default 2023-07-28T23:39:04.348 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.349+0000 7f9a6f526880 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-07-28T23:39:04.348 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.349+0000 7f9a6f526880 20 rados->read ofs=0 len=0 2023-07-28T23:39:04.348 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.349+0000 7f9a6f526880 20 rados_obj.operate() r=-2 bl.length=0 2023-07-28T23:39:04.348 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.349+0000 7f9a6f526880 20 rados->read ofs=0 len=0 2023-07-28T23:39:04.348 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.349+0000 7f9a6f526880 20 rados_obj.operate() r=0 bl.length=46 2023-07-28T23:39:04.349 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.349+0000 7f9a6f526880 20 rados->read ofs=0 len=0 2023-07-28T23:39:04.349 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.349+0000 7f9a6f526880 20 rados_obj.operate() r=0 bl.length=420 2023-07-28T23:39:04.350 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.349+0000 7f9a6f526880 20 zone default found 2023-07-28T23:39:04.350 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.349+0000 7f9a6f526880 4 Realm: () 2023-07-28T23:39:04.350 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.349+0000 7f9a6f526880 4 ZoneGroup: default (79116a35-506a-46f0-9b7d-a6466efb23c9) 2023-07-28T23:39:04.350 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.349+0000 7f9a6f526880 4 Zone: default (2ef2953a-3e98-4529-8f07-1a1fcae427b8) 2023-07-28T23:39:04.350 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.349+0000 7f9a6f526880 10 cannot find current period zonegroup using local zonegroup configuration 2023-07-28T23:39:04.350 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.349+0000 7f9a6f526880 20 zonegroup default 2023-07-28T23:39:04.350 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.349+0000 7f9a6f526880 20 rados->read ofs=0 len=0 2023-07-28T23:39:04.351 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.350+0000 7f9a6f526880 20 rados_obj.operate() r=-2 bl.length=0 2023-07-28T23:39:04.351 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.350+0000 7f9a6f526880 20 rados->read ofs=0 len=0 2023-07-28T23:39:04.351 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.350+0000 7f9a6f526880 20 rados_obj.operate() r=-2 bl.length=0 2023-07-28T23:39:04.351 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.350+0000 7f9a6f526880 20 rados->read ofs=0 len=0 2023-07-28T23:39:04.351 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.350+0000 7f9a6f526880 20 rados_obj.operate() r=-2 bl.length=0 2023-07-28T23:39:04.351 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.350+0000 7f9a6f526880 20 started sync module instance, tier type = 2023-07-28T23:39:04.351 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.350+0000 7f9a6f526880 20 started zone id=2ef2953a-3e98-4529-8f07-1a1fcae427b8 (name=default) with tier type = 2023-07-28T23:39:04.357 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.359+0000 7f9a6f526880 20 add_watcher() i=0 2023-07-28T23:39:04.357 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.359+0000 7f9a6f526880 20 add_watcher() i=1 2023-07-28T23:39:04.357 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.359+0000 7f9a6f526880 20 add_watcher() i=2 2023-07-28T23:39:04.357 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.359+0000 7f9a6f526880 20 add_watcher() i=3 2023-07-28T23:39:04.358 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.359+0000 7f9a6f526880 20 add_watcher() i=4 2023-07-28T23:39:04.358 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.359+0000 7f9a6f526880 20 add_watcher() i=5 2023-07-28T23:39:04.358 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.359+0000 7f9a6f526880 20 add_watcher() i=6 2023-07-28T23:39:04.358 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.360+0000 7f9a6f526880 20 add_watcher() i=7 2023-07-28T23:39:04.358 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.360+0000 7f9a6f526880 2 all 8 watchers are set, enabling cache 2023-07-28T23:39:04.359 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.361+0000 7f9a52d83640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-07-28T23:39:04.359 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.361+0000 7f9a6f526880 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-07-28T23:39:04.359 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.361+0000 7f9a6f526880 20 check_secure_mon_conn(): mode 1 is insecure 2023-07-28T23:39:04.363 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.364+0000 7f9a6f526880 5 note: GC not initialized 2023-07-28T23:39:04.363 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.365+0000 7f9a4e57a640 20 reqs_thread_entry: start 2023-07-28T23:39:04.363 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.365+0000 7f9a6f526880 20 init_complete bucket index max shards: 11 2023-07-28T23:39:04.363 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.365+0000 7f9a6f526880 10 rgw notify: Started notification manager with: 1 workers 2023-07-28T23:39:04.364 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.365+0000 7f9a4c576640 20 rgw notify: INFO: next queues processing will happen at: Fri Jul 28 23:39:34 2023 2023-07-28T23:39:04.364 INFO:teuthology.orchestra.run.smithi101.stderr: 2023-07-28T23:39:04.364 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.365+0000 7f9a4bd54640 20 reqs_thread_entry: start 2023-07-28T23:39:04.370 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.371+0000 7f9a52d83640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-07-28T23:39:04.371 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.373+0000 7f9a6f526880 20 remove_watcher() i=0 2023-07-28T23:39:04.371 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.373+0000 7f9a6f526880 2 removed watcher, disabling cache 2023-07-28T23:39:04.372 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.374+0000 7f9a6f526880 20 remove_watcher() i=1 2023-07-28T23:39:04.373 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.375+0000 7f9a6f526880 20 remove_watcher() i=2 2023-07-28T23:39:04.374 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.376+0000 7f9a6f526880 20 remove_watcher() i=3 2023-07-28T23:39:04.375 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.377+0000 7f9a6f526880 20 remove_watcher() i=4 2023-07-28T23:39:04.376 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.378+0000 7f9a6f526880 20 remove_watcher() i=5 2023-07-28T23:39:04.376 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.378+0000 7f9a6f526880 20 remove_watcher() i=6 2023-07-28T23:39:04.377 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.379+0000 7f9a6f526880 20 remove_watcher() i=7 2023-07-28T23:39:04.380 INFO:teuthology.orchestra.run.smithi101.stdout:[] 2023-07-28T23:39:04.381 DEBUG:tasks.util.rgw: json result: [] 2023-07-28T23:39:04.382 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-07-28T23:39:04.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-07-28T23:39:04.382 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-07-28T23:39:04.468 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:39:04.468 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:39:04.478 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.479+0000 7f9601211880 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-07-28T23:39:04.478 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.479+0000 7f9601211880 20 check_secure_mon_conn(): mode 1 is insecure 2023-07-28T23:39:04.480 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:04.482+0000 7f95e0f51640 20 reqs_thread_entry: start 2023-07-28T23:39:04.486 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:39:04.486 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "2ef2953a-3e98-4529-8f07-1a1fcae427b8", 2023-07-28T23:39:04.486 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default", 2023-07-28T23:39:04.487 INFO:teuthology.orchestra.run.smithi101.stdout: "domain_root": "default.rgw.meta:root", 2023-07-28T23:39:04.487 INFO:teuthology.orchestra.run.smithi101.stdout: "control_pool": "default.rgw.control", 2023-07-28T23:39:04.487 INFO:teuthology.orchestra.run.smithi101.stdout: "gc_pool": "default.rgw.log:gc", 2023-07-28T23:39:04.487 INFO:teuthology.orchestra.run.smithi101.stdout: "lc_pool": "default.rgw.log:lc", 2023-07-28T23:39:04.487 INFO:teuthology.orchestra.run.smithi101.stdout: "log_pool": "default.rgw.log", 2023-07-28T23:39:04.487 INFO:teuthology.orchestra.run.smithi101.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-07-28T23:39:04.487 INFO:teuthology.orchestra.run.smithi101.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-07-28T23:39:04.488 INFO:teuthology.orchestra.run.smithi101.stdout: "roles_pool": "default.rgw.meta:roles", 2023-07-28T23:39:04.488 INFO:teuthology.orchestra.run.smithi101.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-07-28T23:39:04.488 INFO:teuthology.orchestra.run.smithi101.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-07-28T23:39:04.488 INFO:teuthology.orchestra.run.smithi101.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-07-28T23:39:04.488 INFO:teuthology.orchestra.run.smithi101.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-07-28T23:39:04.488 INFO:teuthology.orchestra.run.smithi101.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-07-28T23:39:04.488 INFO:teuthology.orchestra.run.smithi101.stdout: "otp_pool": "default.rgw.otp", 2023-07-28T23:39:04.488 INFO:teuthology.orchestra.run.smithi101.stdout: "system_key": { 2023-07-28T23:39:04.489 INFO:teuthology.orchestra.run.smithi101.stdout: "access_key": "", 2023-07-28T23:39:04.489 INFO:teuthology.orchestra.run.smithi101.stdout: "secret_key": "" 2023-07-28T23:39:04.489 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:39:04.489 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_pools": [ 2023-07-28T23:39:04.489 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:39:04.489 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "default-placement", 2023-07-28T23:39:04.489 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:39:04.489 INFO:teuthology.orchestra.run.smithi101.stdout: "index_pool": "default.rgw.buckets.index", 2023-07-28T23:39:04.490 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": { 2023-07-28T23:39:04.490 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD": { 2023-07-28T23:39:04.490 INFO:teuthology.orchestra.run.smithi101.stdout: "data_pool": "default.rgw.buckets.data", 2023-07-28T23:39:04.490 INFO:teuthology.orchestra.run.smithi101.stdout: "compression_type": "random" 2023-07-28T23:39:04.490 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:39:04.491 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:39:04.491 INFO:teuthology.orchestra.run.smithi101.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-07-28T23:39:04.491 INFO:teuthology.orchestra.run.smithi101.stdout: "index_type": 0, 2023-07-28T23:39:04.491 INFO:teuthology.orchestra.run.smithi101.stdout: "inline_data": true 2023-07-28T23:39:04.491 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:39:04.491 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:39:04.491 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:39:04.491 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "", 2023-07-28T23:39:04.492 INFO:teuthology.orchestra.run.smithi101.stdout: "notif_pool": "default.rgw.log:notif" 2023-07-28T23:39:04.492 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:39:04.492 DEBUG:tasks.util.rgw: json result: {'id': '2ef2953a-3e98-4529-8f07-1a1fcae427b8', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-07-28T23:39:04.493 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2023-07-28T23:39:04.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2023-07-28T23:39:04.493 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2023-07-28T23:39:04.600 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:39:04.600 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:39:04.612 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:04.614+0000 7f5bc3b36880 20 rados->read ofs=0 len=0 2023-07-28T23:39:06.038 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:06.040+0000 7f5bc3b36880 20 rados_obj.operate() r=-2 bl.length=0 2023-07-28T23:39:06.038 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:06.040+0000 7f5bc3b36880 20 realm 2023-07-28T23:39:06.038 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:06.040+0000 7f5bc3b36880 20 rados->read ofs=0 len=0 2023-07-28T23:39:06.040 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:06.042+0000 7f5bc3b36880 20 rados_obj.operate() r=-2 bl.length=0 2023-07-28T23:39:06.041 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:06.043+0000 7f5bc3b36880 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-07-28T23:39:06.041 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:06.043+0000 7f5bc3b36880 20 rados->read ofs=0 len=0 2023-07-28T23:39:06.041 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:06.043+0000 7f5bc3b36880 20 rados_obj.operate() r=-2 bl.length=0 2023-07-28T23:39:06.042 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:06.043+0000 7f5bc3b36880 20 rados->read ofs=0 len=0 2023-07-28T23:39:06.042 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:06.043+0000 7f5bc3b36880 20 rados_obj.operate() r=-2 bl.length=0 2023-07-28T23:39:06.042 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:06.043+0000 7f5bc3b36880 20 rados->read ofs=0 len=0 2023-07-28T23:39:06.042 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:06.043+0000 7f5bc3b36880 20 rados_obj.operate() r=-2 bl.length=0 2023-07-28T23:39:06.042 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:06.043+0000 7f5bc3b36880 20 rados->read ofs=0 len=0 2023-07-28T23:39:06.042 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:06.044+0000 7f5bc3b36880 20 rados_obj.operate() r=-2 bl.length=0 2023-07-28T23:39:06.042 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:06.044+0000 7f5bc3b36880 10 Creating default zonegroup 2023-07-28T23:39:06.043 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:06.044+0000 7f5bc3b36880 10 couldn't find old data placement pools config, setting up new ones for the zone 2023-07-28T23:39:06.051 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:06.053+0000 7f5bc3b36880 10 WARNING: driver->list_zones() returned r=-2 2023-07-28T23:39:06.051 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:06.053+0000 7f5bc3b36880 20 rados->read ofs=0 len=0 2023-07-28T23:39:06.051 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:06.053+0000 7f5bc3b36880 20 rados_obj.operate() r=-2 bl.length=0 2023-07-28T23:39:06.053 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:06.056+0000 7f5bc3b36880 20 rados->read ofs=0 len=0 2023-07-28T23:39:06.054 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:06.056+0000 7f5bc3b36880 20 rados_obj.operate() r=-2 bl.length=0 2023-07-28T23:39:06.054 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:06.056+0000 7f5bc3b36880 10 could not read realm id: (2) No such file or directory 2023-07-28T23:39:06.054 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:06.056+0000 7f5bc3b36880 10 WARNING: failed to set zone as default, r=-22 2023-07-28T23:39:06.054 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:06.056+0000 7f5bc3b36880 20 rados->read ofs=0 len=0 2023-07-28T23:39:06.055 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:06.056+0000 7f5bc3b36880 20 rados_obj.operate() r=-2 bl.length=0 2023-07-28T23:39:06.056 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:06.058+0000 7f5bc3b36880 20 rados->read ofs=0 len=0 2023-07-28T23:39:06.057 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:06.058+0000 7f5bc3b36880 20 rados_obj.operate() r=0 bl.length=874 2023-07-28T23:39:06.057 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:06.058+0000 7f5bc3b36880 20 rados->read ofs=0 len=0 2023-07-28T23:39:06.057 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:06.059+0000 7f5bc3b36880 20 rados_obj.operate() r=0 bl.length=420 2023-07-28T23:39:06.057 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:06.059+0000 7f5bc3b36880 10 Using default name default 2023-07-28T23:39:06.057 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:06.059+0000 7f5bc3b36880 20 rados->read ofs=0 len=0 2023-07-28T23:39:06.058 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:06.059+0000 7f5bc3b36880 20 rados_obj.operate() r=0 bl.length=46 2023-07-28T23:39:06.058 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:06.059+0000 7f5bc3b36880 20 rados->read ofs=0 len=0 2023-07-28T23:39:06.058 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:06.060+0000 7f5bc3b36880 20 rados_obj.operate() r=0 bl.length=874 2023-07-28T23:39:06.059 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:06.060+0000 7f5bc3b36880 20 zone default found 2023-07-28T23:39:06.059 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:06.060+0000 7f5bc3b36880 4 Realm: () 2023-07-28T23:39:06.059 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:06.060+0000 7f5bc3b36880 4 ZoneGroup: default (b11ff385-f70b-4818-9ab6-be0c007be727) 2023-07-28T23:39:06.059 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:06.060+0000 7f5bc3b36880 4 Zone: default (64242a37-6dec-48b1-aee1-91483c8d11be) 2023-07-28T23:39:06.059 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:06.060+0000 7f5bc3b36880 10 cannot find current period zonegroup using local zonegroup configuration 2023-07-28T23:39:06.059 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:06.060+0000 7f5bc3b36880 20 zonegroup default 2023-07-28T23:39:06.059 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:06.060+0000 7f5bc3b36880 20 rados->read ofs=0 len=0 2023-07-28T23:39:06.060 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:06.060+0000 7f5bc3b36880 20 rados_obj.operate() r=-2 bl.length=0 2023-07-28T23:39:06.060 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:06.060+0000 7f5bc3b36880 20 rados->read ofs=0 len=0 2023-07-28T23:39:08.047 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:08.049+0000 7f5bc3b36880 20 rados_obj.operate() r=-2 bl.length=0 2023-07-28T23:39:08.047 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:08.049+0000 7f5bc3b36880 20 rados->read ofs=0 len=0 2023-07-28T23:39:08.048 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:08.050+0000 7f5bc3b36880 20 rados_obj.operate() r=-2 bl.length=0 2023-07-28T23:39:08.048 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:08.050+0000 7f5bc3b36880 20 started sync module instance, tier type = 2023-07-28T23:39:08.048 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:08.050+0000 7f5bc3b36880 20 started zone id=64242a37-6dec-48b1-aee1-91483c8d11be (name=default) with tier type = 2023-07-28T23:39:10.095 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:10.097+0000 7f5bc3b36880 20 add_watcher() i=0 2023-07-28T23:39:10.095 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:10.097+0000 7f5bc3b36880 20 add_watcher() i=1 2023-07-28T23:39:10.095 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:10.097+0000 7f5bc3b36880 20 add_watcher() i=2 2023-07-28T23:39:10.095 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:10.097+0000 7f5bc3b36880 20 add_watcher() i=3 2023-07-28T23:39:10.096 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:10.097+0000 7f5bc3b36880 20 add_watcher() i=4 2023-07-28T23:39:10.096 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:10.097+0000 7f5bc3b36880 20 add_watcher() i=5 2023-07-28T23:39:10.096 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:10.097+0000 7f5bc3b36880 20 add_watcher() i=6 2023-07-28T23:39:10.097 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:10.098+0000 7f5bc3b36880 20 add_watcher() i=7 2023-07-28T23:39:10.097 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:10.098+0000 7f5bc3b36880 2 all 8 watchers are set, enabling cache 2023-07-28T23:39:10.097 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:10.098+0000 7f5bc3b36880 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2023-07-28T23:39:10.097 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:10.098+0000 7f5bc3b36880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2023-07-28T23:39:10.098 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:10.098+0000 7f5bc3b36880 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-07-28T23:39:10.098 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:10.098+0000 7f5bc3b36880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2023-07-28T23:39:10.098 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:10.098+0000 7f5bc3b36880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2023-07-28T23:39:10.098 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:10.098+0000 7f5bc3b36880 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-07-28T23:39:10.098 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:10.098+0000 7f5bc3b36880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2023-07-28T23:39:10.098 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:10.098+0000 7f5bc3b36880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2023-07-28T23:39:10.098 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:10.098+0000 7f5bc3b36880 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-07-28T23:39:10.099 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:10.099+0000 7f5bc3b36880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2023-07-28T23:39:10.099 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:10.099+0000 7f5bc3b36880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2023-07-28T23:39:10.099 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:10.099+0000 7f5bc3b36880 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-07-28T23:39:10.099 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:10.099+0000 7f5bc3b36880 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2023-07-28T23:39:10.099 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:10.099+0000 7f5bc3b36880 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2023-07-28T23:39:10.100 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:10.100+0000 7f5bc3b36880 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-07-28T23:39:10.100 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:10.102+0000 7f5ba742e640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-07-28T23:39:10.100 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:10.102+0000 7f5bc3b36880 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-07-28T23:39:10.100 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:10.102+0000 7f5bc3b36880 20 check_secure_mon_conn(): mode 1 is insecure 2023-07-28T23:39:13.067 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.069+0000 7f5bc3b36880 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2023-07-28T23:39:13.068 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.069+0000 7f5bc3b36880 5 note: GC not initialized 2023-07-28T23:39:13.068 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.069+0000 7f5ba2c25640 20 reqs_thread_entry: start 2023-07-28T23:39:13.068 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.069+0000 7f5bc3b36880 20 init_complete bucket index max shards: 11 2023-07-28T23:39:13.068 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.070+0000 7f5bc3b36880 10 rgw notify: Started notification manager with: 1 workers 2023-07-28T23:39:13.069 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.070+0000 7f5ba0420640 20 reqs_thread_entry: start 2023-07-28T23:39:13.069 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.070+0000 7f5ba0c21640 20 rgw notify: INFO: next queues processing will happen at: Fri Jul 28 23:39:43 2023 2023-07-28T23:39:13.069 INFO:teuthology.orchestra.run.smithi139.stderr: 2023-07-28T23:39:13.088 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.090+0000 7f5ba742e640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-07-28T23:39:13.093 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.095+0000 7f5bc3b36880 20 remove_watcher() i=0 2023-07-28T23:39:13.093 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.095+0000 7f5bc3b36880 2 removed watcher, disabling cache 2023-07-28T23:39:13.095 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.097+0000 7f5bc3b36880 20 remove_watcher() i=1 2023-07-28T23:39:13.096 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.098+0000 7f5bc3b36880 20 remove_watcher() i=2 2023-07-28T23:39:13.099 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.101+0000 7f5bc3b36880 20 remove_watcher() i=3 2023-07-28T23:39:13.102 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.104+0000 7f5bc3b36880 20 remove_watcher() i=4 2023-07-28T23:39:13.103 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.105+0000 7f5bc3b36880 20 remove_watcher() i=5 2023-07-28T23:39:13.104 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.106+0000 7f5bc3b36880 20 remove_watcher() i=6 2023-07-28T23:39:13.107 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.109+0000 7f5bc3b36880 20 remove_watcher() i=7 2023-07-28T23:39:13.111 INFO:teuthology.orchestra.run.smithi139.stdout:[] 2023-07-28T23:39:13.113 DEBUG:tasks.util.rgw: json result: [] 2023-07-28T23:39:13.114 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-07-28T23:39:13.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-07-28T23:39:13.114 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-07-28T23:39:13.164 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:39:13.164 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:39:13.174 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.176+0000 7fcbedb26880 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-07-28T23:39:13.174 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.176+0000 7fcbedb26880 20 check_secure_mon_conn(): mode 1 is insecure 2023-07-28T23:39:13.177 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.179+0000 7fcbcdcf1640 20 reqs_thread_entry: start 2023-07-28T23:39:13.183 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2023-07-28T23:39:13.183 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "64242a37-6dec-48b1-aee1-91483c8d11be", 2023-07-28T23:39:13.183 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "default", 2023-07-28T23:39:13.183 INFO:teuthology.orchestra.run.smithi139.stdout: "domain_root": "default.rgw.meta:root", 2023-07-28T23:39:13.183 INFO:teuthology.orchestra.run.smithi139.stdout: "control_pool": "default.rgw.control", 2023-07-28T23:39:13.184 INFO:teuthology.orchestra.run.smithi139.stdout: "gc_pool": "default.rgw.log:gc", 2023-07-28T23:39:13.184 INFO:teuthology.orchestra.run.smithi139.stdout: "lc_pool": "default.rgw.log:lc", 2023-07-28T23:39:13.184 INFO:teuthology.orchestra.run.smithi139.stdout: "log_pool": "default.rgw.log", 2023-07-28T23:39:13.184 INFO:teuthology.orchestra.run.smithi139.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-07-28T23:39:13.184 INFO:teuthology.orchestra.run.smithi139.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-07-28T23:39:13.184 INFO:teuthology.orchestra.run.smithi139.stdout: "roles_pool": "default.rgw.meta:roles", 2023-07-28T23:39:13.185 INFO:teuthology.orchestra.run.smithi139.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-07-28T23:39:13.185 INFO:teuthology.orchestra.run.smithi139.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-07-28T23:39:13.185 INFO:teuthology.orchestra.run.smithi139.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-07-28T23:39:13.185 INFO:teuthology.orchestra.run.smithi139.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-07-28T23:39:13.185 INFO:teuthology.orchestra.run.smithi139.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-07-28T23:39:13.185 INFO:teuthology.orchestra.run.smithi139.stdout: "otp_pool": "default.rgw.otp", 2023-07-28T23:39:13.185 INFO:teuthology.orchestra.run.smithi139.stdout: "system_key": { 2023-07-28T23:39:13.185 INFO:teuthology.orchestra.run.smithi139.stdout: "access_key": "", 2023-07-28T23:39:13.186 INFO:teuthology.orchestra.run.smithi139.stdout: "secret_key": "" 2023-07-28T23:39:13.186 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:39:13.186 INFO:teuthology.orchestra.run.smithi139.stdout: "placement_pools": [ 2023-07-28T23:39:13.186 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:39:13.186 INFO:teuthology.orchestra.run.smithi139.stdout: "key": "default-placement", 2023-07-28T23:39:13.186 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2023-07-28T23:39:13.186 INFO:teuthology.orchestra.run.smithi139.stdout: "index_pool": "default.rgw.buckets.index", 2023-07-28T23:39:13.186 INFO:teuthology.orchestra.run.smithi139.stdout: "storage_classes": { 2023-07-28T23:39:13.187 INFO:teuthology.orchestra.run.smithi139.stdout: "STANDARD": { 2023-07-28T23:39:13.187 INFO:teuthology.orchestra.run.smithi139.stdout: "data_pool": "default.rgw.buckets.data", 2023-07-28T23:39:13.187 INFO:teuthology.orchestra.run.smithi139.stdout: "compression_type": "random" 2023-07-28T23:39:13.187 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:39:13.187 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:39:13.187 INFO:teuthology.orchestra.run.smithi139.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-07-28T23:39:13.187 INFO:teuthology.orchestra.run.smithi139.stdout: "index_type": 0, 2023-07-28T23:39:13.188 INFO:teuthology.orchestra.run.smithi139.stdout: "inline_data": true 2023-07-28T23:39:13.188 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:39:13.188 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:39:13.188 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2023-07-28T23:39:13.188 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_id": "", 2023-07-28T23:39:13.188 INFO:teuthology.orchestra.run.smithi139.stdout: "notif_pool": "default.rgw.log:notif" 2023-07-28T23:39:13.188 INFO:teuthology.orchestra.run.smithi139.stdout:} 2023-07-28T23:39:13.189 DEBUG:tasks.util.rgw: json result: {'id': '64242a37-6dec-48b1-aee1-91483c8d11be', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-07-28T23:39:13.189 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2023-07-28T23:39:13.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2023-07-28T23:39:13.190 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2023-07-28T23:39:13.231 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:39:13.231 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:39:13.241 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.243+0000 7f1769f11880 20 rados->read ofs=0 len=0 2023-07-28T23:39:13.242 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.243+0000 7f1769f11880 20 rados_obj.operate() r=-2 bl.length=0 2023-07-28T23:39:13.242 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.243+0000 7f1769f11880 20 realm 2023-07-28T23:39:13.242 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.243+0000 7f1769f11880 20 rados->read ofs=0 len=0 2023-07-28T23:39:13.242 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.244+0000 7f1769f11880 20 rados_obj.operate() r=-2 bl.length=0 2023-07-28T23:39:13.242 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.244+0000 7f1769f11880 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-07-28T23:39:13.242 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.244+0000 7f1769f11880 20 rados->read ofs=0 len=0 2023-07-28T23:39:13.242 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.244+0000 7f1769f11880 20 rados_obj.operate() r=-2 bl.length=0 2023-07-28T23:39:13.243 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.244+0000 7f1769f11880 20 rados->read ofs=0 len=0 2023-07-28T23:39:13.243 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.244+0000 7f1769f11880 20 rados_obj.operate() r=0 bl.length=46 2023-07-28T23:39:13.243 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.244+0000 7f1769f11880 20 rados->read ofs=0 len=0 2023-07-28T23:39:13.244 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.244+0000 7f1769f11880 20 rados_obj.operate() r=0 bl.length=890 2023-07-28T23:39:13.244 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.244+0000 7f1769f11880 20 searching for the correct realm 2023-07-28T23:39:13.249 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.251+0000 7f1769f11880 20 RGWRados::pool_iterate: got zone_info.64242a37-6dec-48b1-aee1-91483c8d11be 2023-07-28T23:39:13.249 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.251+0000 7f1769f11880 20 RGWRados::pool_iterate: got zonegroup_info.b11ff385-f70b-4818-9ab6-be0c007be727 2023-07-28T23:39:13.250 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.251+0000 7f1769f11880 20 RGWRados::pool_iterate: got zone_names.default 2023-07-28T23:39:13.250 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.251+0000 7f1769f11880 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-07-28T23:39:13.250 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.251+0000 7f1769f11880 20 rados->read ofs=0 len=0 2023-07-28T23:39:13.250 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.251+0000 7f1769f11880 20 rados_obj.operate() r=-2 bl.length=0 2023-07-28T23:39:13.250 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.251+0000 7f1769f11880 20 rados->read ofs=0 len=0 2023-07-28T23:39:13.250 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.251+0000 7f1769f11880 20 rados_obj.operate() r=0 bl.length=46 2023-07-28T23:39:13.250 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.251+0000 7f1769f11880 20 rados->read ofs=0 len=0 2023-07-28T23:39:13.251 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.252+0000 7f1769f11880 20 rados_obj.operate() r=0 bl.length=420 2023-07-28T23:39:13.252 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.252+0000 7f1769f11880 20 zone default found 2023-07-28T23:39:13.252 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.252+0000 7f1769f11880 4 Realm: () 2023-07-28T23:39:13.252 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.252+0000 7f1769f11880 4 ZoneGroup: default (b11ff385-f70b-4818-9ab6-be0c007be727) 2023-07-28T23:39:13.252 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.252+0000 7f1769f11880 4 Zone: default (64242a37-6dec-48b1-aee1-91483c8d11be) 2023-07-28T23:39:13.252 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.252+0000 7f1769f11880 10 cannot find current period zonegroup using local zonegroup configuration 2023-07-28T23:39:13.252 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.252+0000 7f1769f11880 20 zonegroup default 2023-07-28T23:39:13.252 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.252+0000 7f1769f11880 20 rados->read ofs=0 len=0 2023-07-28T23:39:13.253 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.252+0000 7f1769f11880 20 rados_obj.operate() r=-2 bl.length=0 2023-07-28T23:39:13.253 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.252+0000 7f1769f11880 20 rados->read ofs=0 len=0 2023-07-28T23:39:13.253 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.252+0000 7f1769f11880 20 rados_obj.operate() r=-2 bl.length=0 2023-07-28T23:39:13.253 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.252+0000 7f1769f11880 20 rados->read ofs=0 len=0 2023-07-28T23:39:13.253 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.252+0000 7f1769f11880 20 rados_obj.operate() r=-2 bl.length=0 2023-07-28T23:39:13.253 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.252+0000 7f1769f11880 20 started sync module instance, tier type = 2023-07-28T23:39:13.253 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.252+0000 7f1769f11880 20 started zone id=64242a37-6dec-48b1-aee1-91483c8d11be (name=default) with tier type = 2023-07-28T23:39:13.271 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.272+0000 7f1769f11880 20 add_watcher() i=0 2023-07-28T23:39:13.271 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.273+0000 7f1769f11880 20 add_watcher() i=1 2023-07-28T23:39:13.271 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.273+0000 7f1769f11880 20 add_watcher() i=2 2023-07-28T23:39:13.271 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.273+0000 7f1769f11880 20 add_watcher() i=3 2023-07-28T23:39:13.272 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.273+0000 7f1769f11880 20 add_watcher() i=4 2023-07-28T23:39:13.272 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.273+0000 7f1769f11880 20 add_watcher() i=5 2023-07-28T23:39:13.272 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.273+0000 7f1769f11880 20 add_watcher() i=6 2023-07-28T23:39:13.273 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.275+0000 7f1769f11880 20 add_watcher() i=7 2023-07-28T23:39:13.273 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.275+0000 7f1769f11880 2 all 8 watchers are set, enabling cache 2023-07-28T23:39:13.275 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.277+0000 7f174f684640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-07-28T23:39:13.275 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.277+0000 7f1769f11880 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-07-28T23:39:13.275 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.277+0000 7f1769f11880 20 check_secure_mon_conn(): mode 1 is insecure 2023-07-28T23:39:13.278 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.280+0000 7f1769f11880 5 note: GC not initialized 2023-07-28T23:39:13.279 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.281+0000 7f174ae7b640 20 reqs_thread_entry: start 2023-07-28T23:39:13.279 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.281+0000 7f1769f11880 20 init_complete bucket index max shards: 11 2023-07-28T23:39:13.279 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.281+0000 7f1769f11880 10 rgw notify: Started notification manager with: 1 workers 2023-07-28T23:39:13.280 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.281+0000 7f1748e77640 20 rgw notify: INFO: next queues processing will happen at: Fri Jul 28 23:39:43 2023 2023-07-28T23:39:13.280 INFO:teuthology.orchestra.run.smithi139.stderr: 2023-07-28T23:39:13.280 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.281+0000 7f1748655640 20 reqs_thread_entry: start 2023-07-28T23:39:13.286 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.288+0000 7f174f684640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-07-28T23:39:13.289 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.292+0000 7f1769f11880 20 remove_watcher() i=0 2023-07-28T23:39:13.290 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.292+0000 7f1769f11880 2 removed watcher, disabling cache 2023-07-28T23:39:13.290 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.293+0000 7f1769f11880 20 remove_watcher() i=1 2023-07-28T23:39:13.291 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.293+0000 7f1769f11880 20 remove_watcher() i=2 2023-07-28T23:39:13.292 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.294+0000 7f1769f11880 20 remove_watcher() i=3 2023-07-28T23:39:13.293 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.295+0000 7f1769f11880 20 remove_watcher() i=4 2023-07-28T23:39:13.294 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.296+0000 7f1769f11880 20 remove_watcher() i=5 2023-07-28T23:39:13.295 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.297+0000 7f1769f11880 20 remove_watcher() i=6 2023-07-28T23:39:13.296 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.299+0000 7f1769f11880 20 remove_watcher() i=7 2023-07-28T23:39:13.299 INFO:teuthology.orchestra.run.smithi139.stdout:[] 2023-07-28T23:39:13.301 DEBUG:tasks.util.rgw: json result: [] 2023-07-28T23:39:13.301 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-07-28T23:39:13.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-07-28T23:39:13.301 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-07-28T23:39:13.389 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:39:13.389 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:39:13.398 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.400+0000 7f04ec612880 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-07-28T23:39:13.398 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.400+0000 7f04ec612880 20 check_secure_mon_conn(): mode 1 is insecure 2023-07-28T23:39:13.400 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:13.403+0000 7f04cc3a7640 20 reqs_thread_entry: start 2023-07-28T23:39:13.406 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2023-07-28T23:39:13.406 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "64242a37-6dec-48b1-aee1-91483c8d11be", 2023-07-28T23:39:13.407 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "default", 2023-07-28T23:39:13.407 INFO:teuthology.orchestra.run.smithi139.stdout: "domain_root": "default.rgw.meta:root", 2023-07-28T23:39:13.407 INFO:teuthology.orchestra.run.smithi139.stdout: "control_pool": "default.rgw.control", 2023-07-28T23:39:13.407 INFO:teuthology.orchestra.run.smithi139.stdout: "gc_pool": "default.rgw.log:gc", 2023-07-28T23:39:13.407 INFO:teuthology.orchestra.run.smithi139.stdout: "lc_pool": "default.rgw.log:lc", 2023-07-28T23:39:13.407 INFO:teuthology.orchestra.run.smithi139.stdout: "log_pool": "default.rgw.log", 2023-07-28T23:39:13.407 INFO:teuthology.orchestra.run.smithi139.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-07-28T23:39:13.408 INFO:teuthology.orchestra.run.smithi139.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-07-28T23:39:13.408 INFO:teuthology.orchestra.run.smithi139.stdout: "roles_pool": "default.rgw.meta:roles", 2023-07-28T23:39:13.408 INFO:teuthology.orchestra.run.smithi139.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-07-28T23:39:13.408 INFO:teuthology.orchestra.run.smithi139.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-07-28T23:39:13.408 INFO:teuthology.orchestra.run.smithi139.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-07-28T23:39:13.408 INFO:teuthology.orchestra.run.smithi139.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-07-28T23:39:13.408 INFO:teuthology.orchestra.run.smithi139.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-07-28T23:39:13.409 INFO:teuthology.orchestra.run.smithi139.stdout: "otp_pool": "default.rgw.otp", 2023-07-28T23:39:13.409 INFO:teuthology.orchestra.run.smithi139.stdout: "system_key": { 2023-07-28T23:39:13.409 INFO:teuthology.orchestra.run.smithi139.stdout: "access_key": "", 2023-07-28T23:39:13.409 INFO:teuthology.orchestra.run.smithi139.stdout: "secret_key": "" 2023-07-28T23:39:13.409 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:39:13.409 INFO:teuthology.orchestra.run.smithi139.stdout: "placement_pools": [ 2023-07-28T23:39:13.409 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:39:13.410 INFO:teuthology.orchestra.run.smithi139.stdout: "key": "default-placement", 2023-07-28T23:39:13.410 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2023-07-28T23:39:13.410 INFO:teuthology.orchestra.run.smithi139.stdout: "index_pool": "default.rgw.buckets.index", 2023-07-28T23:39:13.410 INFO:teuthology.orchestra.run.smithi139.stdout: "storage_classes": { 2023-07-28T23:39:13.410 INFO:teuthology.orchestra.run.smithi139.stdout: "STANDARD": { 2023-07-28T23:39:13.410 INFO:teuthology.orchestra.run.smithi139.stdout: "data_pool": "default.rgw.buckets.data", 2023-07-28T23:39:13.410 INFO:teuthology.orchestra.run.smithi139.stdout: "compression_type": "random" 2023-07-28T23:39:13.410 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:39:13.411 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:39:13.411 INFO:teuthology.orchestra.run.smithi139.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-07-28T23:39:13.411 INFO:teuthology.orchestra.run.smithi139.stdout: "index_type": 0, 2023-07-28T23:39:13.411 INFO:teuthology.orchestra.run.smithi139.stdout: "inline_data": true 2023-07-28T23:39:13.411 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:39:13.411 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:39:13.411 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2023-07-28T23:39:13.412 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_id": "", 2023-07-28T23:39:13.412 INFO:teuthology.orchestra.run.smithi139.stdout: "notif_pool": "default.rgw.log:notif" 2023-07-28T23:39:13.412 INFO:teuthology.orchestra.run.smithi139.stdout:} 2023-07-28T23:39:13.413 DEBUG:tasks.util.rgw: json result: {'id': '64242a37-6dec-48b1-aee1-91483c8d11be', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-07-28T23:39:13.413 INFO:tasks.rgw:Starting rgw... 2023-07-28T23:39:13.413 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2023-07-28T23:39:13.414 INFO:tasks.rgw:Using beast as radosgw frontend 2023-07-28T23:39:13.414 INFO:tasks.rgw.c1.client.0:Restarting daemon 2023-07-28T23:39:13.414 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2023-07-28T23:39:13.417 INFO:tasks.rgw.c1.client.0:Started 2023-07-28T23:39:13.417 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2023-07-28T23:39:13.418 INFO:tasks.rgw:Using beast as radosgw frontend 2023-07-28T23:39:13.418 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-07-28T23:39:13.418 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-07-28T23:39:13.420 INFO:tasks.rgw.c1.client.1:Started 2023-07-28T23:39:13.421 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2023-07-28T23:39:13.421 INFO:tasks.rgw:Using beast as radosgw frontend 2023-07-28T23:39:13.421 INFO:tasks.rgw.c2.client.0:Restarting daemon 2023-07-28T23:39:13.422 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2023-07-28T23:39:13.424 INFO:tasks.rgw.c2.client.0:Started 2023-07-28T23:39:13.424 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2023-07-28T23:39:13.425 INFO:tasks.rgw:Using beast as radosgw frontend 2023-07-28T23:39:13.425 INFO:tasks.rgw.c2.client.1:Restarting daemon 2023-07-28T23:39:13.425 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2023-07-28T23:39:13.427 INFO:tasks.rgw.c2.client.1:Started 2023-07-28T23:39:13.427 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi101.front.sepia.ceph.com:8000/ 2023-07-28T23:39:13.427 DEBUG:teuthology.orchestra.run.smithi101:> curl http://smithi101.front.sepia.ceph.com:8000/ 2023-07-28T23:39:13.478 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-07-28T23:39:13.479 INFO:teuthology.orchestra.run.smithi101.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-07-28T23:39:13.479 INFO:teuthology.orchestra.run.smithi101.stderr: Dload Upload Total Spent Left Speed 2023-07-28T23:39:13.479 INFO:teuthology.orchestra.run.smithi101.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-07-28T23:39:13.480 INFO:teuthology.orchestra.run.smithi101.stderr:curl: (7) Failed to connect to smithi101.front.sepia.ceph.com port 8000: Connection refused 2023-07-28T23:39:14.480 DEBUG:teuthology.orchestra.run.smithi101:> curl http://smithi101.front.sepia.ceph.com:8000/ 2023-07-28T23:39:14.499 INFO:teuthology.orchestra.run.smithi101.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-07-28T23:39:14.499 INFO:teuthology.orchestra.run.smithi101.stderr: Dload Upload Total Spent Left Speed 2023-07-28T23:39:14.500 INFO:teuthology.orchestra.run.smithi101.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-07-28T23:39:14.501 INFO:teuthology.orchestra.run.smithi101.stdout:anonymous 2023-07-28T23:39:14.501 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi101.front.sepia.ceph.com:8001/ 2023-07-28T23:39:14.501 DEBUG:teuthology.orchestra.run.smithi101:> curl http://smithi101.front.sepia.ceph.com:8001/ 2023-07-28T23:39:14.559 INFO:teuthology.orchestra.run.smithi101.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-07-28T23:39:14.559 INFO:teuthology.orchestra.run.smithi101.stderr: Dload Upload Total Spent Left Speed 2023-07-28T23:39:14.559 INFO:teuthology.orchestra.run.smithi101.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-07-28T23:39:14.560 INFO:teuthology.orchestra.run.smithi101.stdout:anonymous 2023-07-28T23:39:14.561 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi139.front.sepia.ceph.com:8000/ 2023-07-28T23:39:14.561 DEBUG:teuthology.orchestra.run.smithi139:> curl http://smithi139.front.sepia.ceph.com:8000/ 2023-07-28T23:39:14.587 INFO:teuthology.orchestra.run.smithi139.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-07-28T23:39:14.587 INFO:teuthology.orchestra.run.smithi139.stderr: Dload Upload Total Spent Left Speed 2023-07-28T23:39:14.588 INFO:teuthology.orchestra.run.smithi139.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-07-28T23:39:14.589 INFO:teuthology.orchestra.run.smithi139.stdout:anonymous 2023-07-28T23:39:14.589 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi139.front.sepia.ceph.com:8001/ 2023-07-28T23:39:14.589 DEBUG:teuthology.orchestra.run.smithi139:> curl http://smithi139.front.sepia.ceph.com:8001/ 2023-07-28T23:39:14.647 INFO:teuthology.orchestra.run.smithi139.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-07-28T23:39:14.647 INFO:teuthology.orchestra.run.smithi139.stderr: Dload Upload Total Spent Left Speed 2023-07-28T23:39:14.648 INFO:teuthology.orchestra.run.smithi139.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-07-28T23:39:14.648 INFO:teuthology.orchestra.run.smithi139.stdout:anonymous 2023-07-28T23:39:14.649 INFO:teuthology.run_tasks:Running task rgw-multisite... 2023-07-28T23:39:14.937 INFO:tasks.rgw_multisite:creating realm.. 2023-07-28T23:39:14.938 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:39:14.938 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:39:14.939 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-28T23:39:14.982 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:39:14.982 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:39:15.005 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:39:15.005 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-28T23:39:15.006 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "test-realm", 2023-07-28T23:39:15.006 INFO:teuthology.orchestra.run.smithi101.stdout: "current_period": "a0987484-ce37-4520-a564-54b7ea8c5251", 2023-07-28T23:39:15.006 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 1 2023-07-28T23:39:15.006 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:39:15.007 INFO:tasks.rgw_multisite:creating master zonegroup.. 2023-07-28T23:39:15.007 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:39:15.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:39:15.008 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup a --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-28T23:39:15.090 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:39:15.090 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:39:15.104 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:15.105+0000 7f520c611880 -1 old format 2023-07-28T23:39:15.111 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:39:15.111 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "a", 2023-07-28T23:39:15.111 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a", 2023-07-28T23:39:15.111 INFO:teuthology.orchestra.run.smithi101.stdout: "api_name": "", 2023-07-28T23:39:15.111 INFO:teuthology.orchestra.run.smithi101.stdout: "is_master": true, 2023-07-28T23:39:15.112 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-28T23:39:15.112 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-28T23:39:15.112 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:39:15.112 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames": [], 2023-07-28T23:39:15.112 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames_s3website": [], 2023-07-28T23:39:15.112 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "", 2023-07-28T23:39:15.112 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [], 2023-07-28T23:39:15.112 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_targets": [ 2023-07-28T23:39:15.113 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:39:15.113 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default-placement", 2023-07-28T23:39:15.113 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [], 2023-07-28T23:39:15.113 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": [] 2023-07-28T23:39:15.113 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:39:15.113 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:39:15.113 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "default-placement", 2023-07-28T23:39:15.114 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-28T23:39:15.114 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_policy": { 2023-07-28T23:39:15.114 INFO:teuthology.orchestra.run.smithi101.stdout: "groups": [] 2023-07-28T23:39:15.114 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:39:15.114 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled_features": [ 2023-07-28T23:39:15.114 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-28T23:39:15.114 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:39:15.114 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:39:15.115 INFO:tasks.rgw_multisite:creating master zone.. 2023-07-28T23:39:15.116 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'RX82IHH7C5RUJ54D', '--secret', 'InTLvCQsePIbOVhlgLo1AkTNflWbmSsQ', '--endpoints', 'http://smithi101.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:39:15.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'RX82IHH7C5RUJ54D', '--secret', 'InTLvCQsePIbOVhlgLo1AkTNflWbmSsQ', '--endpoints', 'http://smithi101.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:39:15.116 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --default --master --access-key RX82IHH7C5RUJ54D --secret InTLvCQsePIbOVhlgLo1AkTNflWbmSsQ --endpoints http://smithi101.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-28T23:39:15.206 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:39:15.206 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:39:15.220 INFO:teuthology.orchestra.run.smithi101.stderr:NOTICE: set zone's realm_id=72b68b73-6728-4b14-854a-54780b6610bf 2023-07-28T23:39:15.235 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:39:15.235 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-28T23:39:15.235 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a1", 2023-07-28T23:39:15.235 INFO:teuthology.orchestra.run.smithi101.stdout: "domain_root": "a1.rgw.meta:root", 2023-07-28T23:39:15.235 INFO:teuthology.orchestra.run.smithi101.stdout: "control_pool": "a1.rgw.control", 2023-07-28T23:39:15.235 INFO:teuthology.orchestra.run.smithi101.stdout: "gc_pool": "a1.rgw.log:gc", 2023-07-28T23:39:15.235 INFO:teuthology.orchestra.run.smithi101.stdout: "lc_pool": "a1.rgw.log:lc", 2023-07-28T23:39:15.236 INFO:teuthology.orchestra.run.smithi101.stdout: "log_pool": "a1.rgw.log", 2023-07-28T23:39:15.236 INFO:teuthology.orchestra.run.smithi101.stdout: "intent_log_pool": "a1.rgw.log:intent", 2023-07-28T23:39:15.236 INFO:teuthology.orchestra.run.smithi101.stdout: "usage_log_pool": "a1.rgw.log:usage", 2023-07-28T23:39:15.236 INFO:teuthology.orchestra.run.smithi101.stdout: "roles_pool": "a1.rgw.meta:roles", 2023-07-28T23:39:15.236 INFO:teuthology.orchestra.run.smithi101.stdout: "reshard_pool": "a1.rgw.log:reshard", 2023-07-28T23:39:15.236 INFO:teuthology.orchestra.run.smithi101.stdout: "user_keys_pool": "a1.rgw.meta:users.keys", 2023-07-28T23:39:15.236 INFO:teuthology.orchestra.run.smithi101.stdout: "user_email_pool": "a1.rgw.meta:users.email", 2023-07-28T23:39:15.237 INFO:teuthology.orchestra.run.smithi101.stdout: "user_swift_pool": "a1.rgw.meta:users.swift", 2023-07-28T23:39:15.237 INFO:teuthology.orchestra.run.smithi101.stdout: "user_uid_pool": "a1.rgw.meta:users.uid", 2023-07-28T23:39:15.237 INFO:teuthology.orchestra.run.smithi101.stdout: "otp_pool": "a1.rgw.otp", 2023-07-28T23:39:15.237 INFO:teuthology.orchestra.run.smithi101.stdout: "system_key": { 2023-07-28T23:39:15.237 INFO:teuthology.orchestra.run.smithi101.stdout: "access_key": "RX82IHH7C5RUJ54D", 2023-07-28T23:39:15.237 INFO:teuthology.orchestra.run.smithi101.stdout: "secret_key": "InTLvCQsePIbOVhlgLo1AkTNflWbmSsQ" 2023-07-28T23:39:15.237 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:39:15.238 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_pools": [ 2023-07-28T23:39:15.238 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:39:15.238 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "default-placement", 2023-07-28T23:39:15.238 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:39:15.238 INFO:teuthology.orchestra.run.smithi101.stdout: "index_pool": "a1.rgw.buckets.index", 2023-07-28T23:39:15.238 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": { 2023-07-28T23:39:15.238 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD": { 2023-07-28T23:39:15.239 INFO:teuthology.orchestra.run.smithi101.stdout: "data_pool": "a1.rgw.buckets.data" 2023-07-28T23:39:15.239 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:39:15.239 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:39:15.239 INFO:teuthology.orchestra.run.smithi101.stdout: "data_extra_pool": "a1.rgw.buckets.non-ec", 2023-07-28T23:39:15.239 INFO:teuthology.orchestra.run.smithi101.stdout: "index_type": 0, 2023-07-28T23:39:15.239 INFO:teuthology.orchestra.run.smithi101.stdout: "inline_data": true 2023-07-28T23:39:15.239 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:39:15.239 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:39:15.240 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:39:15.240 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-28T23:39:15.240 INFO:teuthology.orchestra.run.smithi101.stdout: "notif_pool": "a1.rgw.log:notif" 2023-07-28T23:39:15.240 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:39:15.241 DEBUG:teuthology.orchestra.run.smithi101:> sudo ceph osd pool create a1.rgw.buckets.data 64 64 --cluster c1 2023-07-28T23:39:16.113 INFO:teuthology.orchestra.run.smithi101.stderr:pool 'a1.rgw.buckets.data' created 2023-07-28T23:39:16.134 DEBUG:teuthology.orchestra.run.smithi101:> sudo ceph osd pool application enable a1.rgw.buckets.data rgw --cluster c1 2023-07-28T23:39:17.138 INFO:teuthology.orchestra.run.smithi101.stderr:enabled application 'rgw' on pool 'a1.rgw.buckets.data' 2023-07-28T23:39:17.165 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:39:17.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:39:17.166 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-28T23:39:17.217 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:39:17.218 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:39:17.236 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:39:17.237 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-28T23:39:17.237 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a1", 2023-07-28T23:39:17.237 INFO:teuthology.orchestra.run.smithi101.stdout: "domain_root": "a1.rgw.meta:root", 2023-07-28T23:39:17.237 INFO:teuthology.orchestra.run.smithi101.stdout: "control_pool": "a1.rgw.control", 2023-07-28T23:39:17.237 INFO:teuthology.orchestra.run.smithi101.stdout: "gc_pool": "a1.rgw.log:gc", 2023-07-28T23:39:17.237 INFO:teuthology.orchestra.run.smithi101.stdout: "lc_pool": "a1.rgw.log:lc", 2023-07-28T23:39:17.237 INFO:teuthology.orchestra.run.smithi101.stdout: "log_pool": "a1.rgw.log", 2023-07-28T23:39:17.238 INFO:teuthology.orchestra.run.smithi101.stdout: "intent_log_pool": "a1.rgw.log:intent", 2023-07-28T23:39:17.238 INFO:teuthology.orchestra.run.smithi101.stdout: "usage_log_pool": "a1.rgw.log:usage", 2023-07-28T23:39:17.238 INFO:teuthology.orchestra.run.smithi101.stdout: "roles_pool": "a1.rgw.meta:roles", 2023-07-28T23:39:17.238 INFO:teuthology.orchestra.run.smithi101.stdout: "reshard_pool": "a1.rgw.log:reshard", 2023-07-28T23:39:17.238 INFO:teuthology.orchestra.run.smithi101.stdout: "user_keys_pool": "a1.rgw.meta:users.keys", 2023-07-28T23:39:17.238 INFO:teuthology.orchestra.run.smithi101.stdout: "user_email_pool": "a1.rgw.meta:users.email", 2023-07-28T23:39:17.238 INFO:teuthology.orchestra.run.smithi101.stdout: "user_swift_pool": "a1.rgw.meta:users.swift", 2023-07-28T23:39:17.238 INFO:teuthology.orchestra.run.smithi101.stdout: "user_uid_pool": "a1.rgw.meta:users.uid", 2023-07-28T23:39:17.239 INFO:teuthology.orchestra.run.smithi101.stdout: "otp_pool": "a1.rgw.otp", 2023-07-28T23:39:17.239 INFO:teuthology.orchestra.run.smithi101.stdout: "system_key": { 2023-07-28T23:39:17.239 INFO:teuthology.orchestra.run.smithi101.stdout: "access_key": "RX82IHH7C5RUJ54D", 2023-07-28T23:39:17.239 INFO:teuthology.orchestra.run.smithi101.stdout: "secret_key": "InTLvCQsePIbOVhlgLo1AkTNflWbmSsQ" 2023-07-28T23:39:17.239 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:39:17.239 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_pools": [ 2023-07-28T23:39:17.239 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:39:17.239 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "default-placement", 2023-07-28T23:39:17.240 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:39:17.240 INFO:teuthology.orchestra.run.smithi101.stdout: "index_pool": "a1.rgw.buckets.index", 2023-07-28T23:39:17.240 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": { 2023-07-28T23:39:17.240 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD": { 2023-07-28T23:39:17.240 INFO:teuthology.orchestra.run.smithi101.stdout: "data_pool": "a1.rgw.buckets.data", 2023-07-28T23:39:17.241 INFO:teuthology.orchestra.run.smithi101.stdout: "compression_type": "random" 2023-07-28T23:39:17.241 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:39:17.241 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:39:17.241 INFO:teuthology.orchestra.run.smithi101.stdout: "data_extra_pool": "a1.rgw.buckets.non-ec", 2023-07-28T23:39:17.242 INFO:teuthology.orchestra.run.smithi101.stdout: "index_type": 0, 2023-07-28T23:39:17.242 INFO:teuthology.orchestra.run.smithi101.stdout: "inline_data": true 2023-07-28T23:39:17.242 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:39:17.242 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:39:17.243 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:39:17.243 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-28T23:39:17.243 INFO:teuthology.orchestra.run.smithi101.stdout: "notif_pool": "a1.rgw.log:notif" 2023-07-28T23:39:17.243 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:39:17.244 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:39:17.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:39:17.245 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-28T23:39:17.290 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:39:17.290 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:39:17.302 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:17.304+0000 7f899d92a880 0 period (a0987484-ce37-4520-a564-54b7ea8c5251 does not have zone 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433 configured 2023-07-28T23:39:24.217 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:39:24.218 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:39:24.218 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 1, 2023-07-28T23:39:24.218 INFO:teuthology.orchestra.run.smithi101.stdout: "predecessor_uuid": "a0987484-ce37-4520-a564-54b7ea8c5251", 2023-07-28T23:39:24.218 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": [], 2023-07-28T23:39:24.218 INFO:teuthology.orchestra.run.smithi101.stdout: "period_map": { 2023-07-28T23:39:24.218 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:39:24.218 INFO:teuthology.orchestra.run.smithi101.stdout: "zonegroups": [ 2023-07-28T23:39:24.219 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:39:24.219 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "a", 2023-07-28T23:39:24.219 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a", 2023-07-28T23:39:24.219 INFO:teuthology.orchestra.run.smithi101.stdout: "api_name": "", 2023-07-28T23:39:24.219 INFO:teuthology.orchestra.run.smithi101.stdout: "is_master": true, 2023-07-28T23:39:24.219 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-28T23:39:24.219 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-28T23:39:24.219 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:39:24.220 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames": [], 2023-07-28T23:39:24.220 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames_s3website": [], 2023-07-28T23:39:24.220 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-28T23:39:24.220 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-28T23:39:24.220 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:39:24.220 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-28T23:39:24.220 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a1", 2023-07-28T23:39:24.221 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-28T23:39:24.221 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-28T23:39:24.221 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:39:24.221 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-28T23:39:24.221 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": false, 2023-07-28T23:39:24.221 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-28T23:39:24.221 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-28T23:39:24.221 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-28T23:39:24.222 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-28T23:39:24.222 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-28T23:39:24.222 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-28T23:39:24.222 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-28T23:39:24.222 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-28T23:39:24.222 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-28T23:39:24.222 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:39:24.222 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:39:24.223 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:39:24.223 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_targets": [ 2023-07-28T23:39:24.223 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:39:24.223 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default-placement", 2023-07-28T23:39:24.223 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [], 2023-07-28T23:39:24.223 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": [ 2023-07-28T23:39:24.223 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD" 2023-07-28T23:39:24.224 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:39:24.224 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:39:24.224 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:39:24.224 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "default-placement", 2023-07-28T23:39:24.224 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-28T23:39:24.224 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_policy": { 2023-07-28T23:39:24.224 INFO:teuthology.orchestra.run.smithi101.stdout: "groups": [] 2023-07-28T23:39:24.224 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:39:24.225 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled_features": [ 2023-07-28T23:39:24.225 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-28T23:39:24.225 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:39:24.225 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:39:24.225 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:39:24.225 INFO:teuthology.orchestra.run.smithi101.stdout: "short_zone_ids": [ 2023-07-28T23:39:24.225 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:39:24.225 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-28T23:39:24.226 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 2896287231 2023-07-28T23:39:24.226 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:39:24.226 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:39:24.226 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:39:24.226 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zonegroup": "a", 2023-07-28T23:39:24.226 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-28T23:39:24.226 INFO:teuthology.orchestra.run.smithi101.stdout: "period_config": { 2023-07-28T23:39:24.227 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_quota": { 2023-07-28T23:39:24.227 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-28T23:39:24.227 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-28T23:39:24.227 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-28T23:39:24.227 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-28T23:39:24.227 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-28T23:39:24.227 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:39:24.227 INFO:teuthology.orchestra.run.smithi101.stdout: "user_quota": { 2023-07-28T23:39:24.228 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-28T23:39:24.228 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-28T23:39:24.228 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-28T23:39:24.228 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-28T23:39:24.228 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-28T23:39:24.228 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:39:24.228 INFO:teuthology.orchestra.run.smithi101.stdout: "user_ratelimit": { 2023-07-28T23:39:24.228 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-28T23:39:24.229 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-28T23:39:24.229 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-28T23:39:24.229 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-28T23:39:24.229 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-28T23:39:24.229 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:39:24.229 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_ratelimit": { 2023-07-28T23:39:24.229 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-28T23:39:24.230 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-28T23:39:24.230 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-28T23:39:24.230 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-28T23:39:24.230 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-28T23:39:24.230 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:39:24.230 INFO:teuthology.orchestra.run.smithi101.stdout: "anonymous_ratelimit": { 2023-07-28T23:39:24.230 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-28T23:39:24.230 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-28T23:39:24.231 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-28T23:39:24.231 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-28T23:39:24.231 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-28T23:39:24.231 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:39:24.231 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:39:24.231 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-28T23:39:24.231 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_name": "test-realm", 2023-07-28T23:39:24.231 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:39:24.232 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:39:24.233 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2023-07-28T23:39:24.233 INFO:teuthology.orchestra.run:waiting for 300 2023-07-28T23:39:24.234 INFO:tasks.rgw.c1.client.0.smithi101.stdout:2023-07-28T23:39:24.235+0000 7f26a204a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 86998) UID: 0 2023-07-28T23:39:24.234 INFO:tasks.rgw.c1.client.0.smithi101.stdout:2023-07-28T23:39:24.235+0000 7f26a58a9880 -1 shutting down 2023-07-28T23:39:28.725 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2023-07-28T23:39:30.335 INFO:tasks.rgw.c1.client.0:Stopped 2023-07-28T23:39:30.335 INFO:tasks.rgw.c1.client.0:Restarting daemon 2023-07-28T23:39:30.336 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2023-07-28T23:39:30.339 INFO:tasks.rgw.c1.client.0:Started 2023-07-28T23:39:30.339 DEBUG:teuthology.orchestra.run.smithi101:> curl http://smithi101.front.sepia.ceph.com:8000 2023-07-28T23:39:30.360 INFO:teuthology.orchestra.run.smithi101.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-07-28T23:39:30.360 INFO:teuthology.orchestra.run.smithi101.stderr: Dload Upload Total Spent Left Speed 2023-07-28T23:39:30.360 INFO:teuthology.orchestra.run.smithi101.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-07-28T23:39:30.361 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-07-28T23:39:30.361 INFO:teuthology.orchestra.run.smithi101.stderr:curl: (7) Failed to connect to smithi101.front.sepia.ceph.com port 8000: Connection refused 2023-07-28T23:39:31.363 DEBUG:teuthology.orchestra.run.smithi101:> curl http://smithi101.front.sepia.ceph.com:8000 2023-07-28T23:39:31.382 INFO:teuthology.orchestra.run.smithi101.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-07-28T23:39:31.382 INFO:teuthology.orchestra.run.smithi101.stderr: Dload Upload Total Spent Left Speed 2023-07-28T23:39:31.383 INFO:teuthology.orchestra.run.smithi101.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-07-28T23:39:31.384 INFO:teuthology.orchestra.run.smithi101.stdout:anonymous 2023-07-28T23:39:31.384 INFO:tasks.rgw_multisite:creating admin user.. 2023-07-28T23:39:31.385 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'RX82IHH7C5RUJ54D', '--secret', 'InTLvCQsePIbOVhlgLo1AkTNflWbmSsQ', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:39:31.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'RX82IHH7C5RUJ54D', '--secret', 'InTLvCQsePIbOVhlgLo1AkTNflWbmSsQ', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:39:31.386 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key RX82IHH7C5RUJ54D --secret InTLvCQsePIbOVhlgLo1AkTNflWbmSsQ --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-28T23:39:31.479 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:39:31.479 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:39:31.508 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:39:31.509 INFO:teuthology.orchestra.run.smithi101.stdout: "user_id": "realm-admin", 2023-07-28T23:39:31.509 INFO:teuthology.orchestra.run.smithi101.stdout: "display_name": "Realm Admin", 2023-07-28T23:39:31.509 INFO:teuthology.orchestra.run.smithi101.stdout: "email": "", 2023-07-28T23:39:31.509 INFO:teuthology.orchestra.run.smithi101.stdout: "suspended": 0, 2023-07-28T23:39:31.509 INFO:teuthology.orchestra.run.smithi101.stdout: "max_buckets": 1000, 2023-07-28T23:39:31.509 INFO:teuthology.orchestra.run.smithi101.stdout: "subusers": [], 2023-07-28T23:39:31.510 INFO:teuthology.orchestra.run.smithi101.stdout: "keys": [ 2023-07-28T23:39:31.510 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:39:31.510 INFO:teuthology.orchestra.run.smithi101.stdout: "user": "realm-admin", 2023-07-28T23:39:31.510 INFO:teuthology.orchestra.run.smithi101.stdout: "access_key": "RX82IHH7C5RUJ54D", 2023-07-28T23:39:31.510 INFO:teuthology.orchestra.run.smithi101.stdout: "secret_key": "InTLvCQsePIbOVhlgLo1AkTNflWbmSsQ" 2023-07-28T23:39:31.510 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:39:31.510 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:39:31.510 INFO:teuthology.orchestra.run.smithi101.stdout: "swift_keys": [], 2023-07-28T23:39:31.511 INFO:teuthology.orchestra.run.smithi101.stdout: "caps": [], 2023-07-28T23:39:31.511 INFO:teuthology.orchestra.run.smithi101.stdout: "op_mask": "read, write, delete", 2023-07-28T23:39:31.511 INFO:teuthology.orchestra.run.smithi101.stdout: "system": true, 2023-07-28T23:39:31.511 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "", 2023-07-28T23:39:31.511 INFO:teuthology.orchestra.run.smithi101.stdout: "default_storage_class": "", 2023-07-28T23:39:31.511 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_tags": [], 2023-07-28T23:39:31.511 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_quota": { 2023-07-28T23:39:31.512 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-28T23:39:31.512 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-28T23:39:31.512 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-28T23:39:31.512 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-28T23:39:31.512 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-28T23:39:31.512 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:39:31.513 INFO:teuthology.orchestra.run.smithi101.stdout: "user_quota": { 2023-07-28T23:39:31.513 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-28T23:39:31.513 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-28T23:39:31.513 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-28T23:39:31.513 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-28T23:39:31.513 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-28T23:39:31.513 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:39:31.513 INFO:teuthology.orchestra.run.smithi101.stdout: "temp_url_keys": [], 2023-07-28T23:39:31.514 INFO:teuthology.orchestra.run.smithi101.stdout: "type": "rgw", 2023-07-28T23:39:31.514 INFO:teuthology.orchestra.run.smithi101.stdout: "mfa_ids": [] 2023-07-28T23:39:31.514 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:39:31.514 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:39:31.520 INFO:tasks.rgw_multisite:creating zone.. 2023-07-28T23:39:31.520 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'RX82IHH7C5RUJ54D', '--secret', 'InTLvCQsePIbOVhlgLo1AkTNflWbmSsQ', '--endpoints', 'http://smithi101.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:39:31.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'RX82IHH7C5RUJ54D', '--secret', 'InTLvCQsePIbOVhlgLo1AkTNflWbmSsQ', '--endpoints', 'http://smithi101.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:39:31.521 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --access-key RX82IHH7C5RUJ54D --secret InTLvCQsePIbOVhlgLo1AkTNflWbmSsQ --endpoints http://smithi101.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-28T23:39:31.571 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:39:31.571 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:39:31.585 INFO:teuthology.orchestra.run.smithi101.stderr:NOTICE: set zone's realm_id=72b68b73-6728-4b14-854a-54780b6610bf 2023-07-28T23:39:31.600 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:39:31.600 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-28T23:39:31.600 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a2", 2023-07-28T23:39:31.601 INFO:teuthology.orchestra.run.smithi101.stdout: "domain_root": "a2.rgw.meta:root", 2023-07-28T23:39:31.601 INFO:teuthology.orchestra.run.smithi101.stdout: "control_pool": "a2.rgw.control", 2023-07-28T23:39:31.601 INFO:teuthology.orchestra.run.smithi101.stdout: "gc_pool": "a2.rgw.log:gc", 2023-07-28T23:39:31.601 INFO:teuthology.orchestra.run.smithi101.stdout: "lc_pool": "a2.rgw.log:lc", 2023-07-28T23:39:31.601 INFO:teuthology.orchestra.run.smithi101.stdout: "log_pool": "a2.rgw.log", 2023-07-28T23:39:31.601 INFO:teuthology.orchestra.run.smithi101.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-07-28T23:39:31.601 INFO:teuthology.orchestra.run.smithi101.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-07-28T23:39:31.602 INFO:teuthology.orchestra.run.smithi101.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-07-28T23:39:31.602 INFO:teuthology.orchestra.run.smithi101.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-07-28T23:39:31.602 INFO:teuthology.orchestra.run.smithi101.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-07-28T23:39:31.602 INFO:teuthology.orchestra.run.smithi101.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-07-28T23:39:31.602 INFO:teuthology.orchestra.run.smithi101.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-07-28T23:39:31.602 INFO:teuthology.orchestra.run.smithi101.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-07-28T23:39:31.602 INFO:teuthology.orchestra.run.smithi101.stdout: "otp_pool": "a2.rgw.otp", 2023-07-28T23:39:31.602 INFO:teuthology.orchestra.run.smithi101.stdout: "system_key": { 2023-07-28T23:39:31.603 INFO:teuthology.orchestra.run.smithi101.stdout: "access_key": "RX82IHH7C5RUJ54D", 2023-07-28T23:39:31.603 INFO:teuthology.orchestra.run.smithi101.stdout: "secret_key": "InTLvCQsePIbOVhlgLo1AkTNflWbmSsQ" 2023-07-28T23:39:31.603 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:39:31.603 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_pools": [ 2023-07-28T23:39:31.603 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:39:31.603 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "default-placement", 2023-07-28T23:39:31.603 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:39:31.604 INFO:teuthology.orchestra.run.smithi101.stdout: "index_pool": "a2.rgw.buckets.index", 2023-07-28T23:39:31.604 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": { 2023-07-28T23:39:31.604 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD": { 2023-07-28T23:39:31.604 INFO:teuthology.orchestra.run.smithi101.stdout: "data_pool": "a2.rgw.buckets.data" 2023-07-28T23:39:31.604 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:39:31.604 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:39:31.604 INFO:teuthology.orchestra.run.smithi101.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-07-28T23:39:31.605 INFO:teuthology.orchestra.run.smithi101.stdout: "index_type": 0, 2023-07-28T23:39:31.605 INFO:teuthology.orchestra.run.smithi101.stdout: "inline_data": true 2023-07-28T23:39:31.605 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:39:31.605 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:39:31.605 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:39:31.605 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-28T23:39:31.605 INFO:teuthology.orchestra.run.smithi101.stdout: "notif_pool": "a2.rgw.log:notif" 2023-07-28T23:39:31.605 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:39:31.606 DEBUG:teuthology.orchestra.run.smithi101:> sudo ceph osd pool create a2.rgw.buckets.data 64 64 --cluster c1 2023-07-28T23:39:32.115 INFO:teuthology.orchestra.run.smithi101.stderr:pool 'a2.rgw.buckets.data' created 2023-07-28T23:39:32.137 DEBUG:teuthology.orchestra.run.smithi101:> sudo ceph osd pool application enable a2.rgw.buckets.data rgw --cluster c1 2023-07-28T23:39:33.120 INFO:teuthology.orchestra.run.smithi101.stderr:enabled application 'rgw' on pool 'a2.rgw.buckets.data' 2023-07-28T23:39:33.165 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:39:33.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:39:33.166 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-28T23:39:33.217 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:39:33.218 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:39:33.235 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:39:33.236 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-28T23:39:33.236 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a2", 2023-07-28T23:39:33.236 INFO:teuthology.orchestra.run.smithi101.stdout: "domain_root": "a2.rgw.meta:root", 2023-07-28T23:39:33.236 INFO:teuthology.orchestra.run.smithi101.stdout: "control_pool": "a2.rgw.control", 2023-07-28T23:39:33.236 INFO:teuthology.orchestra.run.smithi101.stdout: "gc_pool": "a2.rgw.log:gc", 2023-07-28T23:39:33.236 INFO:teuthology.orchestra.run.smithi101.stdout: "lc_pool": "a2.rgw.log:lc", 2023-07-28T23:39:33.237 INFO:teuthology.orchestra.run.smithi101.stdout: "log_pool": "a2.rgw.log", 2023-07-28T23:39:33.237 INFO:teuthology.orchestra.run.smithi101.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-07-28T23:39:33.237 INFO:teuthology.orchestra.run.smithi101.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-07-28T23:39:33.237 INFO:teuthology.orchestra.run.smithi101.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-07-28T23:39:33.237 INFO:teuthology.orchestra.run.smithi101.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-07-28T23:39:33.237 INFO:teuthology.orchestra.run.smithi101.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-07-28T23:39:33.237 INFO:teuthology.orchestra.run.smithi101.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-07-28T23:39:33.237 INFO:teuthology.orchestra.run.smithi101.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-07-28T23:39:33.238 INFO:teuthology.orchestra.run.smithi101.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-07-28T23:39:33.238 INFO:teuthology.orchestra.run.smithi101.stdout: "otp_pool": "a2.rgw.otp", 2023-07-28T23:39:33.238 INFO:teuthology.orchestra.run.smithi101.stdout: "system_key": { 2023-07-28T23:39:33.238 INFO:teuthology.orchestra.run.smithi101.stdout: "access_key": "RX82IHH7C5RUJ54D", 2023-07-28T23:39:33.238 INFO:teuthology.orchestra.run.smithi101.stdout: "secret_key": "InTLvCQsePIbOVhlgLo1AkTNflWbmSsQ" 2023-07-28T23:39:33.238 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:39:33.238 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_pools": [ 2023-07-28T23:39:33.239 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:39:33.239 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "default-placement", 2023-07-28T23:39:33.239 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:39:33.239 INFO:teuthology.orchestra.run.smithi101.stdout: "index_pool": "a2.rgw.buckets.index", 2023-07-28T23:39:33.239 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": { 2023-07-28T23:39:33.239 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD": { 2023-07-28T23:39:33.239 INFO:teuthology.orchestra.run.smithi101.stdout: "data_pool": "a2.rgw.buckets.data", 2023-07-28T23:39:33.239 INFO:teuthology.orchestra.run.smithi101.stdout: "compression_type": "random" 2023-07-28T23:39:33.240 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:39:33.240 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:39:33.240 INFO:teuthology.orchestra.run.smithi101.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-07-28T23:39:33.240 INFO:teuthology.orchestra.run.smithi101.stdout: "index_type": 0, 2023-07-28T23:39:33.240 INFO:teuthology.orchestra.run.smithi101.stdout: "inline_data": true 2023-07-28T23:39:33.240 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:39:33.240 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:39:33.240 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:39:33.241 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-28T23:39:33.241 INFO:teuthology.orchestra.run.smithi101.stdout: "notif_pool": "a2.rgw.log:notif" 2023-07-28T23:39:33.241 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:39:33.242 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:39:33.242 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:39:33.242 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-28T23:39:33.287 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:39:33.288 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:39:33.300 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:39:33.302+0000 7fb75cd2f880 0 period (2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 does not have zone 59d1bb30-3b1c-476f-9111-052c1ea0b711 configured 2023-07-28T23:39:34.431 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2023-07-28T23:39:40.183 INFO:teuthology.orchestra.run.smithi101.stderr:Sending period to new master zone 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433 2023-07-28T23:39:40.210 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:39:40.210 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:39:40.210 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 2, 2023-07-28T23:39:40.210 INFO:teuthology.orchestra.run.smithi101.stdout: "predecessor_uuid": "a0987484-ce37-4520-a564-54b7ea8c5251", 2023-07-28T23:39:40.211 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": [], 2023-07-28T23:39:40.211 INFO:teuthology.orchestra.run.smithi101.stdout: "period_map": { 2023-07-28T23:39:40.211 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:39:40.211 INFO:teuthology.orchestra.run.smithi101.stdout: "zonegroups": [ 2023-07-28T23:39:40.211 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:39:40.211 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "a", 2023-07-28T23:39:40.211 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a", 2023-07-28T23:39:40.212 INFO:teuthology.orchestra.run.smithi101.stdout: "api_name": "", 2023-07-28T23:39:40.212 INFO:teuthology.orchestra.run.smithi101.stdout: "is_master": true, 2023-07-28T23:39:40.212 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-28T23:39:40.212 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-28T23:39:40.212 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:39:40.212 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames": [], 2023-07-28T23:39:40.212 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames_s3website": [], 2023-07-28T23:39:40.213 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-28T23:39:40.213 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-28T23:39:40.213 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:39:40.213 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-28T23:39:40.213 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a1", 2023-07-28T23:39:40.213 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-28T23:39:40.213 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-28T23:39:40.213 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:39:40.214 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-28T23:39:40.214 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-28T23:39:40.214 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-28T23:39:40.214 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-28T23:39:40.214 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-28T23:39:40.214 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-28T23:39:40.214 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-28T23:39:40.214 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-28T23:39:40.215 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-28T23:39:40.215 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-28T23:39:40.215 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-28T23:39:40.215 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:39:40.215 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:39:40.215 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:39:40.215 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-28T23:39:40.215 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a2", 2023-07-28T23:39:40.216 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-28T23:39:40.216 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8001" 2023-07-28T23:39:40.216 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:39:40.216 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-28T23:39:40.216 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-28T23:39:40.216 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-28T23:39:40.216 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-28T23:39:40.217 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-28T23:39:40.217 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-28T23:39:40.217 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-28T23:39:40.217 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-28T23:39:40.217 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-28T23:39:40.217 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-28T23:39:40.217 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-28T23:39:40.217 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:39:40.218 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:39:40.218 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:39:40.218 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_targets": [ 2023-07-28T23:39:40.218 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:39:40.218 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default-placement", 2023-07-28T23:39:40.218 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [], 2023-07-28T23:39:40.218 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": [ 2023-07-28T23:39:40.219 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD" 2023-07-28T23:39:40.219 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:39:40.219 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:39:40.219 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:39:40.219 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "default-placement", 2023-07-28T23:39:40.219 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-28T23:39:40.219 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_policy": { 2023-07-28T23:39:40.219 INFO:teuthology.orchestra.run.smithi101.stdout: "groups": [] 2023-07-28T23:39:40.220 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:39:40.220 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled_features": [ 2023-07-28T23:39:40.220 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-28T23:39:40.220 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:39:40.220 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:39:40.220 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:39:40.220 INFO:teuthology.orchestra.run.smithi101.stdout: "short_zone_ids": [ 2023-07-28T23:39:40.220 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:39:40.221 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-28T23:39:40.221 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 2896287231 2023-07-28T23:39:40.221 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:39:40.221 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:39:40.221 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-28T23:39:40.221 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 815606810 2023-07-28T23:39:40.221 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:39:40.221 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:39:40.222 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:39:40.222 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zonegroup": "a", 2023-07-28T23:39:40.222 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-28T23:39:40.222 INFO:teuthology.orchestra.run.smithi101.stdout: "period_config": { 2023-07-28T23:39:40.222 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_quota": { 2023-07-28T23:39:40.222 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-28T23:39:40.222 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-28T23:39:40.223 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-28T23:39:40.223 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-28T23:39:40.223 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-28T23:39:40.223 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:39:40.223 INFO:teuthology.orchestra.run.smithi101.stdout: "user_quota": { 2023-07-28T23:39:40.223 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-28T23:39:40.223 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-28T23:39:40.223 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-28T23:39:40.224 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-28T23:39:40.224 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-28T23:39:40.224 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:39:40.224 INFO:teuthology.orchestra.run.smithi101.stdout: "user_ratelimit": { 2023-07-28T23:39:40.224 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-28T23:39:40.224 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-28T23:39:40.224 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-28T23:39:40.224 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-28T23:39:40.225 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-28T23:39:40.225 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:39:40.225 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_ratelimit": { 2023-07-28T23:39:40.225 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-28T23:39:40.225 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-28T23:39:40.225 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-28T23:39:40.225 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-28T23:39:40.226 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-28T23:39:40.226 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:39:40.227 INFO:teuthology.orchestra.run.smithi101.stdout: "anonymous_ratelimit": { 2023-07-28T23:39:40.227 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-28T23:39:40.227 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-28T23:39:40.227 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-28T23:39:40.227 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-28T23:39:40.227 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-28T23:39:40.227 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:39:40.228 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:39:40.228 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-28T23:39:40.228 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_name": "test-realm", 2023-07-28T23:39:40.228 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:39:40.228 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:39:40.229 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-07-28T23:39:40.229 INFO:teuthology.orchestra.run:waiting for 300 2023-07-28T23:39:40.229 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-28T23:39:40.231+0000 7f8ba5d2b640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 86999) UID: 0 2023-07-28T23:39:40.230 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-28T23:39:40.231+0000 7f8babe34880 -1 shutting down 2023-07-28T23:39:45.743 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-07-28T23:39:46.331 INFO:tasks.rgw.c1.client.1:Stopped 2023-07-28T23:39:46.331 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-07-28T23:39:46.332 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-07-28T23:39:46.334 INFO:tasks.rgw.c1.client.1:Started 2023-07-28T23:39:46.335 DEBUG:teuthology.orchestra.run.smithi101:> curl http://smithi101.front.sepia.ceph.com:8001 2023-07-28T23:39:46.355 INFO:teuthology.orchestra.run.smithi101.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-07-28T23:39:46.355 INFO:teuthology.orchestra.run.smithi101.stderr: Dload Upload Total Spent Left Speed 2023-07-28T23:39:46.355 INFO:teuthology.orchestra.run.smithi101.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-07-28T23:39:46.356 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-07-28T23:39:46.357 INFO:teuthology.orchestra.run.smithi101.stderr:curl: (7) Failed to connect to smithi101.front.sepia.ceph.com port 8001: Connection refused 2023-07-28T23:39:47.358 DEBUG:teuthology.orchestra.run.smithi101:> curl http://smithi101.front.sepia.ceph.com:8001 2023-07-28T23:39:47.376 INFO:teuthology.orchestra.run.smithi101.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-07-28T23:39:47.377 INFO:teuthology.orchestra.run.smithi101.stderr: Dload Upload Total Spent Left Speed 2023-07-28T23:39:47.377 INFO:teuthology.orchestra.run.smithi101.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-07-28T23:39:47.378 INFO:teuthology.orchestra.run.smithi101.stdout:anonymous 2023-07-28T23:39:47.378 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2023-07-28T23:39:47.379 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi101.front.sepia.ceph.com:8000', '--access-key', 'RX82IHH7C5RUJ54D', '--secret', 'InTLvCQsePIbOVhlgLo1AkTNflWbmSsQ', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:39:47.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi101.front.sepia.ceph.com:8000', '--access-key', 'RX82IHH7C5RUJ54D', '--secret', 'InTLvCQsePIbOVhlgLo1AkTNflWbmSsQ', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:39:47.379 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi101.front.sepia.ceph.com:8000 --access-key RX82IHH7C5RUJ54D --secret InTLvCQsePIbOVhlgLo1AkTNflWbmSsQ --cluster c2 --debug-rgw 0 --debug-ms 0 2023-07-28T23:39:47.434 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:39:47.434 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:39:47.469 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2023-07-28T23:39:47.470 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-28T23:39:47.470 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "test-realm", 2023-07-28T23:39:47.470 INFO:teuthology.orchestra.run.smithi139.stdout: "current_period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:39:47.470 INFO:teuthology.orchestra.run.smithi139.stdout: "epoch": 2 2023-07-28T23:39:47.470 INFO:teuthology.orchestra.run.smithi139.stdout:} 2023-07-28T23:39:47.471 INFO:tasks.rgw_multisite:creating zonegroup.. 2023-07-28T23:39:47.471 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:39:47.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zonegroup', 'set', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:39:47.472 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zonegroup set --rgw-zonegroup b --rgw-realm test-realm --default --cluster c2 --debug-rgw 0 --debug-ms 0 2023-07-28T23:39:47.517 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:39:47.518 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:39:47.531 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:47.533+0000 7f0e4ac8b880 -1 old format 2023-07-28T23:39:47.541 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2023-07-28T23:39:47.541 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "b", 2023-07-28T23:39:47.541 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "b", 2023-07-28T23:39:47.542 INFO:teuthology.orchestra.run.smithi139.stdout: "api_name": "", 2023-07-28T23:39:47.542 INFO:teuthology.orchestra.run.smithi139.stdout: "is_master": false, 2023-07-28T23:39:47.542 INFO:teuthology.orchestra.run.smithi139.stdout: "endpoints": [ 2023-07-28T23:39:47.542 INFO:teuthology.orchestra.run.smithi139.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-28T23:39:47.542 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2023-07-28T23:39:47.542 INFO:teuthology.orchestra.run.smithi139.stdout: "hostnames": [], 2023-07-28T23:39:47.542 INFO:teuthology.orchestra.run.smithi139.stdout: "hostnames_s3website": [], 2023-07-28T23:39:47.543 INFO:teuthology.orchestra.run.smithi139.stdout: "master_zone": "", 2023-07-28T23:39:47.543 INFO:teuthology.orchestra.run.smithi139.stdout: "zones": [], 2023-07-28T23:39:47.543 INFO:teuthology.orchestra.run.smithi139.stdout: "placement_targets": [ 2023-07-28T23:39:47.543 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:39:47.543 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "default-placement", 2023-07-28T23:39:47.543 INFO:teuthology.orchestra.run.smithi139.stdout: "tags": [], 2023-07-28T23:39:47.543 INFO:teuthology.orchestra.run.smithi139.stdout: "storage_classes": [] 2023-07-28T23:39:47.544 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:39:47.544 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2023-07-28T23:39:47.544 INFO:teuthology.orchestra.run.smithi139.stdout: "default_placement": "default-placement", 2023-07-28T23:39:47.544 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-28T23:39:47.544 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_policy": { 2023-07-28T23:39:47.544 INFO:teuthology.orchestra.run.smithi139.stdout: "groups": [] 2023-07-28T23:39:47.545 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:39:47.545 INFO:teuthology.orchestra.run.smithi139.stdout: "enabled_features": [ 2023-07-28T23:39:47.545 INFO:teuthology.orchestra.run.smithi139.stdout: "resharding" 2023-07-28T23:39:47.545 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2023-07-28T23:39:47.545 INFO:teuthology.orchestra.run.smithi139.stdout:} 2023-07-28T23:39:47.546 INFO:tasks.rgw_multisite:creating zone.. 2023-07-28T23:39:47.546 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'RX82IHH7C5RUJ54D', '--secret', 'InTLvCQsePIbOVhlgLo1AkTNflWbmSsQ', '--endpoints', 'http://smithi139.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:39:47.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'RX82IHH7C5RUJ54D', '--secret', 'InTLvCQsePIbOVhlgLo1AkTNflWbmSsQ', '--endpoints', 'http://smithi139.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:39:47.547 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --default --master --access-key RX82IHH7C5RUJ54D --secret InTLvCQsePIbOVhlgLo1AkTNflWbmSsQ --endpoints http://smithi139.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2023-07-28T23:39:47.638 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:39:47.638 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:39:47.652 INFO:teuthology.orchestra.run.smithi139.stderr:NOTICE: set zone's realm_id=72b68b73-6728-4b14-854a-54780b6610bf 2023-07-28T23:39:47.670 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2023-07-28T23:39:47.670 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-28T23:39:47.670 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "b1", 2023-07-28T23:39:47.670 INFO:teuthology.orchestra.run.smithi139.stdout: "domain_root": "b1.rgw.meta:root", 2023-07-28T23:39:47.670 INFO:teuthology.orchestra.run.smithi139.stdout: "control_pool": "b1.rgw.control", 2023-07-28T23:39:47.671 INFO:teuthology.orchestra.run.smithi139.stdout: "gc_pool": "b1.rgw.log:gc", 2023-07-28T23:39:47.671 INFO:teuthology.orchestra.run.smithi139.stdout: "lc_pool": "b1.rgw.log:lc", 2023-07-28T23:39:47.671 INFO:teuthology.orchestra.run.smithi139.stdout: "log_pool": "b1.rgw.log", 2023-07-28T23:39:47.671 INFO:teuthology.orchestra.run.smithi139.stdout: "intent_log_pool": "b1.rgw.log:intent", 2023-07-28T23:39:47.671 INFO:teuthology.orchestra.run.smithi139.stdout: "usage_log_pool": "b1.rgw.log:usage", 2023-07-28T23:39:47.671 INFO:teuthology.orchestra.run.smithi139.stdout: "roles_pool": "b1.rgw.meta:roles", 2023-07-28T23:39:47.671 INFO:teuthology.orchestra.run.smithi139.stdout: "reshard_pool": "b1.rgw.log:reshard", 2023-07-28T23:39:47.672 INFO:teuthology.orchestra.run.smithi139.stdout: "user_keys_pool": "b1.rgw.meta:users.keys", 2023-07-28T23:39:47.672 INFO:teuthology.orchestra.run.smithi139.stdout: "user_email_pool": "b1.rgw.meta:users.email", 2023-07-28T23:39:47.672 INFO:teuthology.orchestra.run.smithi139.stdout: "user_swift_pool": "b1.rgw.meta:users.swift", 2023-07-28T23:39:47.672 INFO:teuthology.orchestra.run.smithi139.stdout: "user_uid_pool": "b1.rgw.meta:users.uid", 2023-07-28T23:39:47.672 INFO:teuthology.orchestra.run.smithi139.stdout: "otp_pool": "b1.rgw.otp", 2023-07-28T23:39:47.672 INFO:teuthology.orchestra.run.smithi139.stdout: "system_key": { 2023-07-28T23:39:47.672 INFO:teuthology.orchestra.run.smithi139.stdout: "access_key": "RX82IHH7C5RUJ54D", 2023-07-28T23:39:47.673 INFO:teuthology.orchestra.run.smithi139.stdout: "secret_key": "InTLvCQsePIbOVhlgLo1AkTNflWbmSsQ" 2023-07-28T23:39:47.673 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:39:47.673 INFO:teuthology.orchestra.run.smithi139.stdout: "placement_pools": [ 2023-07-28T23:39:47.673 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:39:47.673 INFO:teuthology.orchestra.run.smithi139.stdout: "key": "default-placement", 2023-07-28T23:39:47.673 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2023-07-28T23:39:47.673 INFO:teuthology.orchestra.run.smithi139.stdout: "index_pool": "b1.rgw.buckets.index", 2023-07-28T23:39:47.674 INFO:teuthology.orchestra.run.smithi139.stdout: "storage_classes": { 2023-07-28T23:39:47.674 INFO:teuthology.orchestra.run.smithi139.stdout: "STANDARD": { 2023-07-28T23:39:47.674 INFO:teuthology.orchestra.run.smithi139.stdout: "data_pool": "b1.rgw.buckets.data" 2023-07-28T23:39:47.674 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:39:47.674 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:39:47.674 INFO:teuthology.orchestra.run.smithi139.stdout: "data_extra_pool": "b1.rgw.buckets.non-ec", 2023-07-28T23:39:47.674 INFO:teuthology.orchestra.run.smithi139.stdout: "index_type": 0, 2023-07-28T23:39:47.675 INFO:teuthology.orchestra.run.smithi139.stdout: "inline_data": true 2023-07-28T23:39:47.675 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:39:47.675 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:39:47.675 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2023-07-28T23:39:47.675 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-28T23:39:47.675 INFO:teuthology.orchestra.run.smithi139.stdout: "notif_pool": "b1.rgw.log:notif" 2023-07-28T23:39:47.675 INFO:teuthology.orchestra.run.smithi139.stdout:} 2023-07-28T23:39:47.676 DEBUG:teuthology.orchestra.run.smithi139:> sudo ceph osd pool create b1.rgw.buckets.data 64 64 --cluster c2 2023-07-28T23:39:48.034 INFO:teuthology.orchestra.run.smithi139.stderr:pool 'b1.rgw.buckets.data' created 2023-07-28T23:39:48.060 DEBUG:teuthology.orchestra.run.smithi139:> sudo ceph osd pool application enable b1.rgw.buckets.data rgw --cluster c2 2023-07-28T23:39:49.039 INFO:teuthology.orchestra.run.smithi139.stderr:enabled application 'rgw' on pool 'b1.rgw.buckets.data' 2023-07-28T23:39:49.080 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:39:49.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:39:49.081 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2023-07-28T23:39:49.131 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:39:49.131 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:39:49.149 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2023-07-28T23:39:49.150 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-28T23:39:49.150 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "b1", 2023-07-28T23:39:49.150 INFO:teuthology.orchestra.run.smithi139.stdout: "domain_root": "b1.rgw.meta:root", 2023-07-28T23:39:49.150 INFO:teuthology.orchestra.run.smithi139.stdout: "control_pool": "b1.rgw.control", 2023-07-28T23:39:49.150 INFO:teuthology.orchestra.run.smithi139.stdout: "gc_pool": "b1.rgw.log:gc", 2023-07-28T23:39:49.150 INFO:teuthology.orchestra.run.smithi139.stdout: "lc_pool": "b1.rgw.log:lc", 2023-07-28T23:39:49.151 INFO:teuthology.orchestra.run.smithi139.stdout: "log_pool": "b1.rgw.log", 2023-07-28T23:39:49.151 INFO:teuthology.orchestra.run.smithi139.stdout: "intent_log_pool": "b1.rgw.log:intent", 2023-07-28T23:39:49.151 INFO:teuthology.orchestra.run.smithi139.stdout: "usage_log_pool": "b1.rgw.log:usage", 2023-07-28T23:39:49.151 INFO:teuthology.orchestra.run.smithi139.stdout: "roles_pool": "b1.rgw.meta:roles", 2023-07-28T23:39:49.151 INFO:teuthology.orchestra.run.smithi139.stdout: "reshard_pool": "b1.rgw.log:reshard", 2023-07-28T23:39:49.151 INFO:teuthology.orchestra.run.smithi139.stdout: "user_keys_pool": "b1.rgw.meta:users.keys", 2023-07-28T23:39:49.151 INFO:teuthology.orchestra.run.smithi139.stdout: "user_email_pool": "b1.rgw.meta:users.email", 2023-07-28T23:39:49.152 INFO:teuthology.orchestra.run.smithi139.stdout: "user_swift_pool": "b1.rgw.meta:users.swift", 2023-07-28T23:39:49.152 INFO:teuthology.orchestra.run.smithi139.stdout: "user_uid_pool": "b1.rgw.meta:users.uid", 2023-07-28T23:39:49.152 INFO:teuthology.orchestra.run.smithi139.stdout: "otp_pool": "b1.rgw.otp", 2023-07-28T23:39:49.152 INFO:teuthology.orchestra.run.smithi139.stdout: "system_key": { 2023-07-28T23:39:49.152 INFO:teuthology.orchestra.run.smithi139.stdout: "access_key": "RX82IHH7C5RUJ54D", 2023-07-28T23:39:49.152 INFO:teuthology.orchestra.run.smithi139.stdout: "secret_key": "InTLvCQsePIbOVhlgLo1AkTNflWbmSsQ" 2023-07-28T23:39:49.152 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:39:49.153 INFO:teuthology.orchestra.run.smithi139.stdout: "placement_pools": [ 2023-07-28T23:39:49.153 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:39:49.153 INFO:teuthology.orchestra.run.smithi139.stdout: "key": "default-placement", 2023-07-28T23:39:49.153 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2023-07-28T23:39:49.153 INFO:teuthology.orchestra.run.smithi139.stdout: "index_pool": "b1.rgw.buckets.index", 2023-07-28T23:39:49.153 INFO:teuthology.orchestra.run.smithi139.stdout: "storage_classes": { 2023-07-28T23:39:49.153 INFO:teuthology.orchestra.run.smithi139.stdout: "STANDARD": { 2023-07-28T23:39:49.153 INFO:teuthology.orchestra.run.smithi139.stdout: "data_pool": "b1.rgw.buckets.data", 2023-07-28T23:39:49.154 INFO:teuthology.orchestra.run.smithi139.stdout: "compression_type": "random" 2023-07-28T23:39:49.154 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:39:49.154 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:39:49.154 INFO:teuthology.orchestra.run.smithi139.stdout: "data_extra_pool": "b1.rgw.buckets.non-ec", 2023-07-28T23:39:49.154 INFO:teuthology.orchestra.run.smithi139.stdout: "index_type": 0, 2023-07-28T23:39:49.154 INFO:teuthology.orchestra.run.smithi139.stdout: "inline_data": true 2023-07-28T23:39:49.154 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:39:49.155 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:39:49.155 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2023-07-28T23:39:49.155 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-28T23:39:49.155 INFO:teuthology.orchestra.run.smithi139.stdout: "notif_pool": "b1.rgw.log:notif" 2023-07-28T23:39:49.155 INFO:teuthology.orchestra.run.smithi139.stdout:} 2023-07-28T23:39:49.156 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:39:49.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:39:49.157 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2023-07-28T23:39:49.202 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:39:49.202 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:39:49.214 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:39:49.216+0000 7f2990122880 0 period (2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 does not have zone 61176ab8-057a-43ac-9dd0-9e7d5d38f6e5 configured 2023-07-28T23:39:51.449 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-07-28T23:39:56.093 INFO:teuthology.orchestra.run.smithi139.stderr:Sending period to new master zone 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433 2023-07-28T23:39:56.127 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2023-07-28T23:39:56.127 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:39:56.127 INFO:teuthology.orchestra.run.smithi139.stdout: "epoch": 3, 2023-07-28T23:39:56.127 INFO:teuthology.orchestra.run.smithi139.stdout: "predecessor_uuid": "a0987484-ce37-4520-a564-54b7ea8c5251", 2023-07-28T23:39:56.127 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": [], 2023-07-28T23:39:56.127 INFO:teuthology.orchestra.run.smithi139.stdout: "period_map": { 2023-07-28T23:39:56.128 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:39:56.128 INFO:teuthology.orchestra.run.smithi139.stdout: "zonegroups": [ 2023-07-28T23:39:56.128 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:39:56.128 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "a", 2023-07-28T23:39:56.128 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "a", 2023-07-28T23:39:56.128 INFO:teuthology.orchestra.run.smithi139.stdout: "api_name": "", 2023-07-28T23:39:56.129 INFO:teuthology.orchestra.run.smithi139.stdout: "is_master": true, 2023-07-28T23:39:56.129 INFO:teuthology.orchestra.run.smithi139.stdout: "endpoints": [ 2023-07-28T23:39:56.129 INFO:teuthology.orchestra.run.smithi139.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-28T23:39:56.130 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2023-07-28T23:39:56.130 INFO:teuthology.orchestra.run.smithi139.stdout: "hostnames": [], 2023-07-28T23:39:56.130 INFO:teuthology.orchestra.run.smithi139.stdout: "hostnames_s3website": [], 2023-07-28T23:39:56.130 INFO:teuthology.orchestra.run.smithi139.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-28T23:39:56.131 INFO:teuthology.orchestra.run.smithi139.stdout: "zones": [ 2023-07-28T23:39:56.131 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:39:56.131 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-28T23:39:56.132 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "a1", 2023-07-28T23:39:56.132 INFO:teuthology.orchestra.run.smithi139.stdout: "endpoints": [ 2023-07-28T23:39:56.132 INFO:teuthology.orchestra.run.smithi139.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-28T23:39:56.133 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2023-07-28T23:39:56.133 INFO:teuthology.orchestra.run.smithi139.stdout: "log_meta": false, 2023-07-28T23:39:56.133 INFO:teuthology.orchestra.run.smithi139.stdout: "log_data": true, 2023-07-28T23:39:56.134 INFO:teuthology.orchestra.run.smithi139.stdout: "bucket_index_max_shards": 11, 2023-07-28T23:39:56.134 INFO:teuthology.orchestra.run.smithi139.stdout: "read_only": false, 2023-07-28T23:39:56.134 INFO:teuthology.orchestra.run.smithi139.stdout: "tier_type": "", 2023-07-28T23:39:56.135 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_from_all": true, 2023-07-28T23:39:56.135 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_from": [], 2023-07-28T23:39:56.135 INFO:teuthology.orchestra.run.smithi139.stdout: "redirect_zone": "", 2023-07-28T23:39:56.135 INFO:teuthology.orchestra.run.smithi139.stdout: "supported_features": [ 2023-07-28T23:39:56.136 INFO:teuthology.orchestra.run.smithi139.stdout: "compress-encrypted", 2023-07-28T23:39:56.136 INFO:teuthology.orchestra.run.smithi139.stdout: "resharding" 2023-07-28T23:39:56.136 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2023-07-28T23:39:56.137 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:39:56.137 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:39:56.137 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-28T23:39:56.137 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "a2", 2023-07-28T23:39:56.138 INFO:teuthology.orchestra.run.smithi139.stdout: "endpoints": [ 2023-07-28T23:39:56.138 INFO:teuthology.orchestra.run.smithi139.stdout: "http://smithi101.front.sepia.ceph.com:8001" 2023-07-28T23:39:56.138 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2023-07-28T23:39:56.138 INFO:teuthology.orchestra.run.smithi139.stdout: "log_meta": false, 2023-07-28T23:39:56.139 INFO:teuthology.orchestra.run.smithi139.stdout: "log_data": true, 2023-07-28T23:39:56.139 INFO:teuthology.orchestra.run.smithi139.stdout: "bucket_index_max_shards": 11, 2023-07-28T23:39:56.140 INFO:teuthology.orchestra.run.smithi139.stdout: "read_only": false, 2023-07-28T23:39:56.140 INFO:teuthology.orchestra.run.smithi139.stdout: "tier_type": "", 2023-07-28T23:39:56.140 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_from_all": true, 2023-07-28T23:39:56.140 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_from": [], 2023-07-28T23:39:56.141 INFO:teuthology.orchestra.run.smithi139.stdout: "redirect_zone": "", 2023-07-28T23:39:56.141 INFO:teuthology.orchestra.run.smithi139.stdout: "supported_features": [ 2023-07-28T23:39:56.141 INFO:teuthology.orchestra.run.smithi139.stdout: "compress-encrypted", 2023-07-28T23:39:56.141 INFO:teuthology.orchestra.run.smithi139.stdout: "resharding" 2023-07-28T23:39:56.141 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2023-07-28T23:39:56.141 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:39:56.141 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2023-07-28T23:39:56.142 INFO:teuthology.orchestra.run.smithi139.stdout: "placement_targets": [ 2023-07-28T23:39:56.142 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:39:56.142 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "default-placement", 2023-07-28T23:39:56.142 INFO:teuthology.orchestra.run.smithi139.stdout: "tags": [], 2023-07-28T23:39:56.142 INFO:teuthology.orchestra.run.smithi139.stdout: "storage_classes": [ 2023-07-28T23:39:56.142 INFO:teuthology.orchestra.run.smithi139.stdout: "STANDARD" 2023-07-28T23:39:56.142 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2023-07-28T23:39:56.143 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:39:56.143 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2023-07-28T23:39:56.143 INFO:teuthology.orchestra.run.smithi139.stdout: "default_placement": "default-placement", 2023-07-28T23:39:56.143 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-28T23:39:56.143 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_policy": { 2023-07-28T23:39:56.143 INFO:teuthology.orchestra.run.smithi139.stdout: "groups": [] 2023-07-28T23:39:56.143 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:39:56.144 INFO:teuthology.orchestra.run.smithi139.stdout: "enabled_features": [ 2023-07-28T23:39:56.144 INFO:teuthology.orchestra.run.smithi139.stdout: "resharding" 2023-07-28T23:39:56.144 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2023-07-28T23:39:56.144 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:39:56.144 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:39:56.144 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "b", 2023-07-28T23:39:56.144 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "b", 2023-07-28T23:39:56.145 INFO:teuthology.orchestra.run.smithi139.stdout: "api_name": "", 2023-07-28T23:39:56.145 INFO:teuthology.orchestra.run.smithi139.stdout: "is_master": false, 2023-07-28T23:39:56.145 INFO:teuthology.orchestra.run.smithi139.stdout: "endpoints": [ 2023-07-28T23:39:56.145 INFO:teuthology.orchestra.run.smithi139.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-28T23:39:56.145 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2023-07-28T23:39:56.145 INFO:teuthology.orchestra.run.smithi139.stdout: "hostnames": [], 2023-07-28T23:39:56.145 INFO:teuthology.orchestra.run.smithi139.stdout: "hostnames_s3website": [], 2023-07-28T23:39:56.146 INFO:teuthology.orchestra.run.smithi139.stdout: "master_zone": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-28T23:39:56.146 INFO:teuthology.orchestra.run.smithi139.stdout: "zones": [ 2023-07-28T23:39:56.146 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:39:56.146 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-28T23:39:56.146 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "b1", 2023-07-28T23:39:56.146 INFO:teuthology.orchestra.run.smithi139.stdout: "endpoints": [ 2023-07-28T23:39:56.147 INFO:teuthology.orchestra.run.smithi139.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-28T23:39:56.147 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2023-07-28T23:39:56.147 INFO:teuthology.orchestra.run.smithi139.stdout: "log_meta": false, 2023-07-28T23:39:56.148 INFO:teuthology.orchestra.run.smithi139.stdout: "log_data": false, 2023-07-28T23:39:56.148 INFO:teuthology.orchestra.run.smithi139.stdout: "bucket_index_max_shards": 11, 2023-07-28T23:39:56.148 INFO:teuthology.orchestra.run.smithi139.stdout: "read_only": false, 2023-07-28T23:39:56.148 INFO:teuthology.orchestra.run.smithi139.stdout: "tier_type": "", 2023-07-28T23:39:56.148 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_from_all": true, 2023-07-28T23:39:56.148 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_from": [], 2023-07-28T23:39:56.148 INFO:teuthology.orchestra.run.smithi139.stdout: "redirect_zone": "", 2023-07-28T23:39:56.149 INFO:teuthology.orchestra.run.smithi139.stdout: "supported_features": [ 2023-07-28T23:39:56.149 INFO:teuthology.orchestra.run.smithi139.stdout: "compress-encrypted", 2023-07-28T23:39:56.151 INFO:teuthology.orchestra.run.smithi139.stdout: "resharding" 2023-07-28T23:39:56.152 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2023-07-28T23:39:56.152 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:39:56.152 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2023-07-28T23:39:56.152 INFO:teuthology.orchestra.run.smithi139.stdout: "placement_targets": [ 2023-07-28T23:39:56.153 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:39:56.153 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "default-placement", 2023-07-28T23:39:56.153 INFO:teuthology.orchestra.run.smithi139.stdout: "tags": [], 2023-07-28T23:39:56.153 INFO:teuthology.orchestra.run.smithi139.stdout: "storage_classes": [ 2023-07-28T23:39:56.153 INFO:teuthology.orchestra.run.smithi139.stdout: "STANDARD" 2023-07-28T23:39:56.153 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2023-07-28T23:39:56.153 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:39:56.154 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2023-07-28T23:39:56.154 INFO:teuthology.orchestra.run.smithi139.stdout: "default_placement": "default-placement", 2023-07-28T23:39:56.154 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-28T23:39:56.154 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_policy": { 2023-07-28T23:39:56.154 INFO:teuthology.orchestra.run.smithi139.stdout: "groups": [] 2023-07-28T23:39:56.154 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:39:56.154 INFO:teuthology.orchestra.run.smithi139.stdout: "enabled_features": [ 2023-07-28T23:39:56.155 INFO:teuthology.orchestra.run.smithi139.stdout: "resharding" 2023-07-28T23:39:56.155 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2023-07-28T23:39:56.155 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:39:56.155 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2023-07-28T23:39:56.155 INFO:teuthology.orchestra.run.smithi139.stdout: "short_zone_ids": [ 2023-07-28T23:39:56.155 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:39:56.155 INFO:teuthology.orchestra.run.smithi139.stdout: "key": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-28T23:39:56.156 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 2896287231 2023-07-28T23:39:56.156 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:39:56.156 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:39:56.156 INFO:teuthology.orchestra.run.smithi139.stdout: "key": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-28T23:39:56.156 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 815606810 2023-07-28T23:39:56.157 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:39:56.157 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:39:56.157 INFO:teuthology.orchestra.run.smithi139.stdout: "key": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-28T23:39:56.157 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 2009171363 2023-07-28T23:39:56.157 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:39:56.157 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2023-07-28T23:39:56.157 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:39:56.158 INFO:teuthology.orchestra.run.smithi139.stdout: "master_zonegroup": "a", 2023-07-28T23:39:56.158 INFO:teuthology.orchestra.run.smithi139.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-28T23:39:56.158 INFO:teuthology.orchestra.run.smithi139.stdout: "period_config": { 2023-07-28T23:39:56.158 INFO:teuthology.orchestra.run.smithi139.stdout: "bucket_quota": { 2023-07-28T23:39:56.158 INFO:teuthology.orchestra.run.smithi139.stdout: "enabled": false, 2023-07-28T23:39:56.158 INFO:teuthology.orchestra.run.smithi139.stdout: "check_on_raw": false, 2023-07-28T23:39:56.158 INFO:teuthology.orchestra.run.smithi139.stdout: "max_size": -1, 2023-07-28T23:39:56.159 INFO:teuthology.orchestra.run.smithi139.stdout: "max_size_kb": 0, 2023-07-28T23:39:56.159 INFO:teuthology.orchestra.run.smithi139.stdout: "max_objects": -1 2023-07-28T23:39:56.159 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:39:56.159 INFO:teuthology.orchestra.run.smithi139.stdout: "user_quota": { 2023-07-28T23:39:56.159 INFO:teuthology.orchestra.run.smithi139.stdout: "enabled": false, 2023-07-28T23:39:56.159 INFO:teuthology.orchestra.run.smithi139.stdout: "check_on_raw": false, 2023-07-28T23:39:56.159 INFO:teuthology.orchestra.run.smithi139.stdout: "max_size": -1, 2023-07-28T23:39:56.160 INFO:teuthology.orchestra.run.smithi139.stdout: "max_size_kb": 0, 2023-07-28T23:39:56.160 INFO:teuthology.orchestra.run.smithi139.stdout: "max_objects": -1 2023-07-28T23:39:56.160 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:39:56.160 INFO:teuthology.orchestra.run.smithi139.stdout: "user_ratelimit": { 2023-07-28T23:39:56.160 INFO:teuthology.orchestra.run.smithi139.stdout: "max_read_ops": 0, 2023-07-28T23:39:56.160 INFO:teuthology.orchestra.run.smithi139.stdout: "max_write_ops": 0, 2023-07-28T23:39:56.160 INFO:teuthology.orchestra.run.smithi139.stdout: "max_read_bytes": 0, 2023-07-28T23:39:56.161 INFO:teuthology.orchestra.run.smithi139.stdout: "max_write_bytes": 0, 2023-07-28T23:39:56.161 INFO:teuthology.orchestra.run.smithi139.stdout: "enabled": false 2023-07-28T23:39:56.161 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:39:56.161 INFO:teuthology.orchestra.run.smithi139.stdout: "bucket_ratelimit": { 2023-07-28T23:39:56.161 INFO:teuthology.orchestra.run.smithi139.stdout: "max_read_ops": 0, 2023-07-28T23:39:56.161 INFO:teuthology.orchestra.run.smithi139.stdout: "max_write_ops": 0, 2023-07-28T23:39:56.162 INFO:teuthology.orchestra.run.smithi139.stdout: "max_read_bytes": 0, 2023-07-28T23:39:56.162 INFO:teuthology.orchestra.run.smithi139.stdout: "max_write_bytes": 0, 2023-07-28T23:39:56.162 INFO:teuthology.orchestra.run.smithi139.stdout: "enabled": false 2023-07-28T23:39:56.162 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:39:56.162 INFO:teuthology.orchestra.run.smithi139.stdout: "anonymous_ratelimit": { 2023-07-28T23:39:56.162 INFO:teuthology.orchestra.run.smithi139.stdout: "max_read_ops": 0, 2023-07-28T23:39:56.162 INFO:teuthology.orchestra.run.smithi139.stdout: "max_write_ops": 0, 2023-07-28T23:39:56.163 INFO:teuthology.orchestra.run.smithi139.stdout: "max_read_bytes": 0, 2023-07-28T23:39:56.163 INFO:teuthology.orchestra.run.smithi139.stdout: "max_write_bytes": 0, 2023-07-28T23:39:56.163 INFO:teuthology.orchestra.run.smithi139.stdout: "enabled": false 2023-07-28T23:39:56.163 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:39:56.163 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:39:56.163 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-28T23:39:56.164 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_name": "test-realm", 2023-07-28T23:39:56.164 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2023-07-28T23:39:56.164 INFO:teuthology.orchestra.run.smithi139.stdout:} 2023-07-28T23:39:56.165 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2023-07-28T23:39:56.165 INFO:teuthology.orchestra.run:waiting for 300 2023-07-28T23:39:56.166 INFO:tasks.rgw.c2.client.0.smithi139.stdout:2023-07-28T23:39:56.167+0000 7f60433c2640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 86890) UID: 0 2023-07-28T23:39:56.166 INFO:tasks.rgw.c2.client.0.smithi139.stdout:2023-07-28T23:39:56.167+0000 7f6049434880 -1 shutting down 2023-07-28T23:39:56.981 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2023-07-28T23:40:02.267 INFO:tasks.rgw.c2.client.0:Stopped 2023-07-28T23:40:02.267 INFO:tasks.rgw.c2.client.0:Restarting daemon 2023-07-28T23:40:02.267 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2023-07-28T23:40:02.270 INFO:tasks.rgw.c2.client.0:Started 2023-07-28T23:40:02.270 DEBUG:teuthology.orchestra.run.smithi139:> curl http://smithi139.front.sepia.ceph.com:8000 2023-07-28T23:40:02.290 INFO:teuthology.orchestra.run.smithi139.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-07-28T23:40:02.291 INFO:teuthology.orchestra.run.smithi139.stderr: Dload Upload Total Spent Left Speed 2023-07-28T23:40:02.291 INFO:teuthology.orchestra.run.smithi139.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-07-28T23:40:02.291 INFO:teuthology.orchestra.run.smithi139.stderr:curl: (7) Failed to connect to smithi139.front.sepia.ceph.com port 8000: Connection refused 2023-07-28T23:40:02.292 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-07-28T23:40:02.687 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2023-07-28T23:40:03.293 DEBUG:teuthology.orchestra.run.smithi139:> curl http://smithi139.front.sepia.ceph.com:8000 2023-07-28T23:40:03.312 INFO:teuthology.orchestra.run.smithi139.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-07-28T23:40:03.313 INFO:teuthology.orchestra.run.smithi139.stderr: Dload Upload Total Spent Left Speed 2023-07-28T23:40:03.313 INFO:teuthology.orchestra.run.smithi139.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-07-28T23:40:03.314 INFO:teuthology.orchestra.run.smithi139.stdout:anonymous 2023-07-28T23:40:03.314 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2023-07-28T23:40:03.315 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi101.front.sepia.ceph.com:8000', '--access-key', 'RX82IHH7C5RUJ54D', '--secret', 'InTLvCQsePIbOVhlgLo1AkTNflWbmSsQ', '--url', 'http://smithi101.front.sepia.ceph.com:8000', '--access-key', 'RX82IHH7C5RUJ54D', '--secret', 'InTLvCQsePIbOVhlgLo1AkTNflWbmSsQ', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:40:03.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi101.front.sepia.ceph.com:8000', '--access-key', 'RX82IHH7C5RUJ54D', '--secret', 'InTLvCQsePIbOVhlgLo1AkTNflWbmSsQ', '--url', 'http://smithi101.front.sepia.ceph.com:8000', '--access-key', 'RX82IHH7C5RUJ54D', '--secret', 'InTLvCQsePIbOVhlgLo1AkTNflWbmSsQ', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:40:03.315 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi101.front.sepia.ceph.com:8000 --access-key RX82IHH7C5RUJ54D --secret InTLvCQsePIbOVhlgLo1AkTNflWbmSsQ --url http://smithi101.front.sepia.ceph.com:8000 --access-key RX82IHH7C5RUJ54D --secret InTLvCQsePIbOVhlgLo1AkTNflWbmSsQ --cluster c2 --debug-rgw 0 --debug-ms 0 2023-07-28T23:40:03.408 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:40:03.408 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:40:03.443 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2023-07-28T23:40:03.444 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-28T23:40:03.444 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "test-realm", 2023-07-28T23:40:03.444 INFO:teuthology.orchestra.run.smithi139.stdout: "current_period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:40:03.444 INFO:teuthology.orchestra.run.smithi139.stdout: "epoch": 2 2023-07-28T23:40:03.444 INFO:teuthology.orchestra.run.smithi139.stdout:} 2023-07-28T23:40:03.445 INFO:tasks.rgw_multisite:creating zone.. 2023-07-28T23:40:03.445 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--access-key', 'RX82IHH7C5RUJ54D', '--secret', 'InTLvCQsePIbOVhlgLo1AkTNflWbmSsQ', '--endpoints', 'http://smithi139.front.sepia.ceph.com:8001', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:40:03.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--access-key', 'RX82IHH7C5RUJ54D', '--secret', 'InTLvCQsePIbOVhlgLo1AkTNflWbmSsQ', '--endpoints', 'http://smithi139.front.sepia.ceph.com:8001', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:40:03.446 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --access-key RX82IHH7C5RUJ54D --secret InTLvCQsePIbOVhlgLo1AkTNflWbmSsQ --endpoints http://smithi139.front.sepia.ceph.com:8001 --cluster c2 --debug-rgw 0 --debug-ms 0 2023-07-28T23:40:03.497 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:40:03.498 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:40:03.511 INFO:teuthology.orchestra.run.smithi139.stderr:NOTICE: set zone's realm_id=72b68b73-6728-4b14-854a-54780b6610bf 2023-07-28T23:40:03.525 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2023-07-28T23:40:03.526 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-28T23:40:03.526 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "b2", 2023-07-28T23:40:03.526 INFO:teuthology.orchestra.run.smithi139.stdout: "domain_root": "b2.rgw.meta:root", 2023-07-28T23:40:03.526 INFO:teuthology.orchestra.run.smithi139.stdout: "control_pool": "b2.rgw.control", 2023-07-28T23:40:03.527 INFO:teuthology.orchestra.run.smithi139.stdout: "gc_pool": "b2.rgw.log:gc", 2023-07-28T23:40:03.527 INFO:teuthology.orchestra.run.smithi139.stdout: "lc_pool": "b2.rgw.log:lc", 2023-07-28T23:40:03.527 INFO:teuthology.orchestra.run.smithi139.stdout: "log_pool": "b2.rgw.log", 2023-07-28T23:40:03.527 INFO:teuthology.orchestra.run.smithi139.stdout: "intent_log_pool": "b2.rgw.log:intent", 2023-07-28T23:40:03.527 INFO:teuthology.orchestra.run.smithi139.stdout: "usage_log_pool": "b2.rgw.log:usage", 2023-07-28T23:40:03.527 INFO:teuthology.orchestra.run.smithi139.stdout: "roles_pool": "b2.rgw.meta:roles", 2023-07-28T23:40:03.528 INFO:teuthology.orchestra.run.smithi139.stdout: "reshard_pool": "b2.rgw.log:reshard", 2023-07-28T23:40:03.528 INFO:teuthology.orchestra.run.smithi139.stdout: "user_keys_pool": "b2.rgw.meta:users.keys", 2023-07-28T23:40:03.528 INFO:teuthology.orchestra.run.smithi139.stdout: "user_email_pool": "b2.rgw.meta:users.email", 2023-07-28T23:40:03.528 INFO:teuthology.orchestra.run.smithi139.stdout: "user_swift_pool": "b2.rgw.meta:users.swift", 2023-07-28T23:40:03.528 INFO:teuthology.orchestra.run.smithi139.stdout: "user_uid_pool": "b2.rgw.meta:users.uid", 2023-07-28T23:40:03.529 INFO:teuthology.orchestra.run.smithi139.stdout: "otp_pool": "b2.rgw.otp", 2023-07-28T23:40:03.529 INFO:teuthology.orchestra.run.smithi139.stdout: "system_key": { 2023-07-28T23:40:03.529 INFO:teuthology.orchestra.run.smithi139.stdout: "access_key": "RX82IHH7C5RUJ54D", 2023-07-28T23:40:03.529 INFO:teuthology.orchestra.run.smithi139.stdout: "secret_key": "InTLvCQsePIbOVhlgLo1AkTNflWbmSsQ" 2023-07-28T23:40:03.529 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:40:03.530 INFO:teuthology.orchestra.run.smithi139.stdout: "placement_pools": [ 2023-07-28T23:40:03.530 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:40:03.530 INFO:teuthology.orchestra.run.smithi139.stdout: "key": "default-placement", 2023-07-28T23:40:03.530 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2023-07-28T23:40:03.530 INFO:teuthology.orchestra.run.smithi139.stdout: "index_pool": "b2.rgw.buckets.index", 2023-07-28T23:40:03.530 INFO:teuthology.orchestra.run.smithi139.stdout: "storage_classes": { 2023-07-28T23:40:03.531 INFO:teuthology.orchestra.run.smithi139.stdout: "STANDARD": { 2023-07-28T23:40:03.531 INFO:teuthology.orchestra.run.smithi139.stdout: "data_pool": "b2.rgw.buckets.data" 2023-07-28T23:40:03.531 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:40:03.531 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:40:03.531 INFO:teuthology.orchestra.run.smithi139.stdout: "data_extra_pool": "b2.rgw.buckets.non-ec", 2023-07-28T23:40:03.531 INFO:teuthology.orchestra.run.smithi139.stdout: "index_type": 0, 2023-07-28T23:40:03.531 INFO:teuthology.orchestra.run.smithi139.stdout: "inline_data": true 2023-07-28T23:40:03.532 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:40:03.532 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:40:03.532 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2023-07-28T23:40:03.532 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-28T23:40:03.532 INFO:teuthology.orchestra.run.smithi139.stdout: "notif_pool": "b2.rgw.log:notif" 2023-07-28T23:40:03.532 INFO:teuthology.orchestra.run.smithi139.stdout:} 2023-07-28T23:40:03.533 DEBUG:teuthology.orchestra.run.smithi139:> sudo ceph osd pool create b2.rgw.buckets.data 64 64 --cluster c2 2023-07-28T23:40:04.043 INFO:teuthology.orchestra.run.smithi139.stderr:pool 'b2.rgw.buckets.data' created 2023-07-28T23:40:04.074 DEBUG:teuthology.orchestra.run.smithi139:> sudo ceph osd pool application enable b2.rgw.buckets.data rgw --cluster c2 2023-07-28T23:40:05.045 INFO:teuthology.orchestra.run.smithi139.stderr:enabled application 'rgw' on pool 'b2.rgw.buckets.data' 2023-07-28T23:40:05.076 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:40:05.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:40:05.077 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2023-07-28T23:40:05.135 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:40:05.135 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:40:05.153 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2023-07-28T23:40:05.153 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-28T23:40:05.153 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "b2", 2023-07-28T23:40:05.154 INFO:teuthology.orchestra.run.smithi139.stdout: "domain_root": "b2.rgw.meta:root", 2023-07-28T23:40:05.154 INFO:teuthology.orchestra.run.smithi139.stdout: "control_pool": "b2.rgw.control", 2023-07-28T23:40:05.154 INFO:teuthology.orchestra.run.smithi139.stdout: "gc_pool": "b2.rgw.log:gc", 2023-07-28T23:40:05.154 INFO:teuthology.orchestra.run.smithi139.stdout: "lc_pool": "b2.rgw.log:lc", 2023-07-28T23:40:05.154 INFO:teuthology.orchestra.run.smithi139.stdout: "log_pool": "b2.rgw.log", 2023-07-28T23:40:05.154 INFO:teuthology.orchestra.run.smithi139.stdout: "intent_log_pool": "b2.rgw.log:intent", 2023-07-28T23:40:05.154 INFO:teuthology.orchestra.run.smithi139.stdout: "usage_log_pool": "b2.rgw.log:usage", 2023-07-28T23:40:05.155 INFO:teuthology.orchestra.run.smithi139.stdout: "roles_pool": "b2.rgw.meta:roles", 2023-07-28T23:40:05.155 INFO:teuthology.orchestra.run.smithi139.stdout: "reshard_pool": "b2.rgw.log:reshard", 2023-07-28T23:40:05.155 INFO:teuthology.orchestra.run.smithi139.stdout: "user_keys_pool": "b2.rgw.meta:users.keys", 2023-07-28T23:40:05.155 INFO:teuthology.orchestra.run.smithi139.stdout: "user_email_pool": "b2.rgw.meta:users.email", 2023-07-28T23:40:05.155 INFO:teuthology.orchestra.run.smithi139.stdout: "user_swift_pool": "b2.rgw.meta:users.swift", 2023-07-28T23:40:05.155 INFO:teuthology.orchestra.run.smithi139.stdout: "user_uid_pool": "b2.rgw.meta:users.uid", 2023-07-28T23:40:05.155 INFO:teuthology.orchestra.run.smithi139.stdout: "otp_pool": "b2.rgw.otp", 2023-07-28T23:40:05.156 INFO:teuthology.orchestra.run.smithi139.stdout: "system_key": { 2023-07-28T23:40:05.156 INFO:teuthology.orchestra.run.smithi139.stdout: "access_key": "RX82IHH7C5RUJ54D", 2023-07-28T23:40:05.156 INFO:teuthology.orchestra.run.smithi139.stdout: "secret_key": "InTLvCQsePIbOVhlgLo1AkTNflWbmSsQ" 2023-07-28T23:40:05.156 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:40:05.156 INFO:teuthology.orchestra.run.smithi139.stdout: "placement_pools": [ 2023-07-28T23:40:05.156 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:40:05.156 INFO:teuthology.orchestra.run.smithi139.stdout: "key": "default-placement", 2023-07-28T23:40:05.157 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2023-07-28T23:40:05.157 INFO:teuthology.orchestra.run.smithi139.stdout: "index_pool": "b2.rgw.buckets.index", 2023-07-28T23:40:05.157 INFO:teuthology.orchestra.run.smithi139.stdout: "storage_classes": { 2023-07-28T23:40:05.157 INFO:teuthology.orchestra.run.smithi139.stdout: "STANDARD": { 2023-07-28T23:40:05.157 INFO:teuthology.orchestra.run.smithi139.stdout: "data_pool": "b2.rgw.buckets.data", 2023-07-28T23:40:05.157 INFO:teuthology.orchestra.run.smithi139.stdout: "compression_type": "random" 2023-07-28T23:40:05.157 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:40:05.157 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:40:05.158 INFO:teuthology.orchestra.run.smithi139.stdout: "data_extra_pool": "b2.rgw.buckets.non-ec", 2023-07-28T23:40:05.158 INFO:teuthology.orchestra.run.smithi139.stdout: "index_type": 0, 2023-07-28T23:40:05.158 INFO:teuthology.orchestra.run.smithi139.stdout: "inline_data": true 2023-07-28T23:40:05.158 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:40:05.158 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:40:05.158 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2023-07-28T23:40:05.158 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-28T23:40:05.159 INFO:teuthology.orchestra.run.smithi139.stdout: "notif_pool": "b2.rgw.log:notif" 2023-07-28T23:40:05.159 INFO:teuthology.orchestra.run.smithi139.stdout:} 2023-07-28T23:40:05.160 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:40:05.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:40:05.160 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2023-07-28T23:40:05.205 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:40:05.205 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:40:05.219 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:40:05.220+0000 7f7d71126880 0 period (2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 does not have zone 9d60ade7-31f2-415d-9bf3-633ac867407d configured 2023-07-28T23:40:12.124 INFO:teuthology.orchestra.run.smithi139.stderr:Sending period to new master zone 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433 2023-07-28T23:40:12.165 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2023-07-28T23:40:12.166 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:40:12.166 INFO:teuthology.orchestra.run.smithi139.stdout: "epoch": 4, 2023-07-28T23:40:12.166 INFO:teuthology.orchestra.run.smithi139.stdout: "predecessor_uuid": "a0987484-ce37-4520-a564-54b7ea8c5251", 2023-07-28T23:40:12.166 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": [], 2023-07-28T23:40:12.166 INFO:teuthology.orchestra.run.smithi139.stdout: "period_map": { 2023-07-28T23:40:12.166 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:40:12.167 INFO:teuthology.orchestra.run.smithi139.stdout: "zonegroups": [ 2023-07-28T23:40:12.167 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:40:12.167 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "a", 2023-07-28T23:40:12.167 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "a", 2023-07-28T23:40:12.167 INFO:teuthology.orchestra.run.smithi139.stdout: "api_name": "", 2023-07-28T23:40:12.167 INFO:teuthology.orchestra.run.smithi139.stdout: "is_master": true, 2023-07-28T23:40:12.167 INFO:teuthology.orchestra.run.smithi139.stdout: "endpoints": [ 2023-07-28T23:40:12.168 INFO:teuthology.orchestra.run.smithi139.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-28T23:40:12.168 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2023-07-28T23:40:12.168 INFO:teuthology.orchestra.run.smithi139.stdout: "hostnames": [], 2023-07-28T23:40:12.168 INFO:teuthology.orchestra.run.smithi139.stdout: "hostnames_s3website": [], 2023-07-28T23:40:12.168 INFO:teuthology.orchestra.run.smithi139.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-28T23:40:12.168 INFO:teuthology.orchestra.run.smithi139.stdout: "zones": [ 2023-07-28T23:40:12.168 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:40:12.169 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-28T23:40:12.169 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "a1", 2023-07-28T23:40:12.169 INFO:teuthology.orchestra.run.smithi139.stdout: "endpoints": [ 2023-07-28T23:40:12.169 INFO:teuthology.orchestra.run.smithi139.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-28T23:40:12.169 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2023-07-28T23:40:12.169 INFO:teuthology.orchestra.run.smithi139.stdout: "log_meta": false, 2023-07-28T23:40:12.169 INFO:teuthology.orchestra.run.smithi139.stdout: "log_data": true, 2023-07-28T23:40:12.169 INFO:teuthology.orchestra.run.smithi139.stdout: "bucket_index_max_shards": 11, 2023-07-28T23:40:12.170 INFO:teuthology.orchestra.run.smithi139.stdout: "read_only": false, 2023-07-28T23:40:12.170 INFO:teuthology.orchestra.run.smithi139.stdout: "tier_type": "", 2023-07-28T23:40:12.170 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_from_all": true, 2023-07-28T23:40:12.170 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_from": [], 2023-07-28T23:40:12.170 INFO:teuthology.orchestra.run.smithi139.stdout: "redirect_zone": "", 2023-07-28T23:40:12.170 INFO:teuthology.orchestra.run.smithi139.stdout: "supported_features": [ 2023-07-28T23:40:12.170 INFO:teuthology.orchestra.run.smithi139.stdout: "compress-encrypted", 2023-07-28T23:40:12.171 INFO:teuthology.orchestra.run.smithi139.stdout: "resharding" 2023-07-28T23:40:12.171 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2023-07-28T23:40:12.171 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:40:12.171 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:40:12.171 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-28T23:40:12.171 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "a2", 2023-07-28T23:40:12.171 INFO:teuthology.orchestra.run.smithi139.stdout: "endpoints": [ 2023-07-28T23:40:12.172 INFO:teuthology.orchestra.run.smithi139.stdout: "http://smithi101.front.sepia.ceph.com:8001" 2023-07-28T23:40:12.172 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2023-07-28T23:40:12.172 INFO:teuthology.orchestra.run.smithi139.stdout: "log_meta": false, 2023-07-28T23:40:12.172 INFO:teuthology.orchestra.run.smithi139.stdout: "log_data": true, 2023-07-28T23:40:12.172 INFO:teuthology.orchestra.run.smithi139.stdout: "bucket_index_max_shards": 11, 2023-07-28T23:40:12.172 INFO:teuthology.orchestra.run.smithi139.stdout: "read_only": false, 2023-07-28T23:40:12.172 INFO:teuthology.orchestra.run.smithi139.stdout: "tier_type": "", 2023-07-28T23:40:12.172 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_from_all": true, 2023-07-28T23:40:12.173 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_from": [], 2023-07-28T23:40:12.173 INFO:teuthology.orchestra.run.smithi139.stdout: "redirect_zone": "", 2023-07-28T23:40:12.173 INFO:teuthology.orchestra.run.smithi139.stdout: "supported_features": [ 2023-07-28T23:40:12.173 INFO:teuthology.orchestra.run.smithi139.stdout: "compress-encrypted", 2023-07-28T23:40:12.173 INFO:teuthology.orchestra.run.smithi139.stdout: "resharding" 2023-07-28T23:40:12.173 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2023-07-28T23:40:12.173 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:40:12.174 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2023-07-28T23:40:12.174 INFO:teuthology.orchestra.run.smithi139.stdout: "placement_targets": [ 2023-07-28T23:40:12.174 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:40:12.174 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "default-placement", 2023-07-28T23:40:12.174 INFO:teuthology.orchestra.run.smithi139.stdout: "tags": [], 2023-07-28T23:40:12.174 INFO:teuthology.orchestra.run.smithi139.stdout: "storage_classes": [ 2023-07-28T23:40:12.174 INFO:teuthology.orchestra.run.smithi139.stdout: "STANDARD" 2023-07-28T23:40:12.174 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2023-07-28T23:40:12.175 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:40:12.175 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2023-07-28T23:40:12.175 INFO:teuthology.orchestra.run.smithi139.stdout: "default_placement": "default-placement", 2023-07-28T23:40:12.175 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-28T23:40:12.175 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_policy": { 2023-07-28T23:40:12.175 INFO:teuthology.orchestra.run.smithi139.stdout: "groups": [] 2023-07-28T23:40:12.175 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:40:12.176 INFO:teuthology.orchestra.run.smithi139.stdout: "enabled_features": [ 2023-07-28T23:40:12.176 INFO:teuthology.orchestra.run.smithi139.stdout: "resharding" 2023-07-28T23:40:12.176 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2023-07-28T23:40:12.176 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:40:12.176 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:40:12.176 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "b", 2023-07-28T23:40:12.177 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "b", 2023-07-28T23:40:12.177 INFO:teuthology.orchestra.run.smithi139.stdout: "api_name": "", 2023-07-28T23:40:12.177 INFO:teuthology.orchestra.run.smithi139.stdout: "is_master": false, 2023-07-28T23:40:12.177 INFO:teuthology.orchestra.run.smithi139.stdout: "endpoints": [ 2023-07-28T23:40:12.178 INFO:teuthology.orchestra.run.smithi139.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-28T23:40:12.178 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2023-07-28T23:40:12.178 INFO:teuthology.orchestra.run.smithi139.stdout: "hostnames": [], 2023-07-28T23:40:12.178 INFO:teuthology.orchestra.run.smithi139.stdout: "hostnames_s3website": [], 2023-07-28T23:40:12.179 INFO:teuthology.orchestra.run.smithi139.stdout: "master_zone": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-28T23:40:12.179 INFO:teuthology.orchestra.run.smithi139.stdout: "zones": [ 2023-07-28T23:40:12.179 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:40:12.179 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-28T23:40:12.180 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "b1", 2023-07-28T23:40:12.180 INFO:teuthology.orchestra.run.smithi139.stdout: "endpoints": [ 2023-07-28T23:40:12.180 INFO:teuthology.orchestra.run.smithi139.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-28T23:40:12.180 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2023-07-28T23:40:12.181 INFO:teuthology.orchestra.run.smithi139.stdout: "log_meta": false, 2023-07-28T23:40:12.181 INFO:teuthology.orchestra.run.smithi139.stdout: "log_data": true, 2023-07-28T23:40:12.181 INFO:teuthology.orchestra.run.smithi139.stdout: "bucket_index_max_shards": 11, 2023-07-28T23:40:12.181 INFO:teuthology.orchestra.run.smithi139.stdout: "read_only": false, 2023-07-28T23:40:12.181 INFO:teuthology.orchestra.run.smithi139.stdout: "tier_type": "", 2023-07-28T23:40:12.181 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_from_all": true, 2023-07-28T23:40:12.182 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_from": [], 2023-07-28T23:40:12.182 INFO:teuthology.orchestra.run.smithi139.stdout: "redirect_zone": "", 2023-07-28T23:40:12.182 INFO:teuthology.orchestra.run.smithi139.stdout: "supported_features": [ 2023-07-28T23:40:12.182 INFO:teuthology.orchestra.run.smithi139.stdout: "compress-encrypted", 2023-07-28T23:40:12.183 INFO:teuthology.orchestra.run.smithi139.stdout: "resharding" 2023-07-28T23:40:12.183 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2023-07-28T23:40:12.183 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:40:12.183 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:40:12.183 INFO:teuthology.orchestra.run.smithi139.stdout: "id": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-28T23:40:12.184 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "b2", 2023-07-28T23:40:12.184 INFO:teuthology.orchestra.run.smithi139.stdout: "endpoints": [ 2023-07-28T23:40:12.184 INFO:teuthology.orchestra.run.smithi139.stdout: "http://smithi139.front.sepia.ceph.com:8001" 2023-07-28T23:40:12.184 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2023-07-28T23:40:12.184 INFO:teuthology.orchestra.run.smithi139.stdout: "log_meta": false, 2023-07-28T23:40:12.184 INFO:teuthology.orchestra.run.smithi139.stdout: "log_data": true, 2023-07-28T23:40:12.185 INFO:teuthology.orchestra.run.smithi139.stdout: "bucket_index_max_shards": 11, 2023-07-28T23:40:12.185 INFO:teuthology.orchestra.run.smithi139.stdout: "read_only": false, 2023-07-28T23:40:12.185 INFO:teuthology.orchestra.run.smithi139.stdout: "tier_type": "", 2023-07-28T23:40:12.185 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_from_all": true, 2023-07-28T23:40:12.185 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_from": [], 2023-07-28T23:40:12.185 INFO:teuthology.orchestra.run.smithi139.stdout: "redirect_zone": "", 2023-07-28T23:40:12.186 INFO:teuthology.orchestra.run.smithi139.stdout: "supported_features": [ 2023-07-28T23:40:12.186 INFO:teuthology.orchestra.run.smithi139.stdout: "compress-encrypted", 2023-07-28T23:40:12.186 INFO:teuthology.orchestra.run.smithi139.stdout: "resharding" 2023-07-28T23:40:12.186 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2023-07-28T23:40:12.186 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:40:12.186 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2023-07-28T23:40:12.186 INFO:teuthology.orchestra.run.smithi139.stdout: "placement_targets": [ 2023-07-28T23:40:12.187 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:40:12.187 INFO:teuthology.orchestra.run.smithi139.stdout: "name": "default-placement", 2023-07-28T23:40:12.187 INFO:teuthology.orchestra.run.smithi139.stdout: "tags": [], 2023-07-28T23:40:12.187 INFO:teuthology.orchestra.run.smithi139.stdout: "storage_classes": [ 2023-07-28T23:40:12.187 INFO:teuthology.orchestra.run.smithi139.stdout: "STANDARD" 2023-07-28T23:40:12.187 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2023-07-28T23:40:12.187 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:40:12.188 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2023-07-28T23:40:12.188 INFO:teuthology.orchestra.run.smithi139.stdout: "default_placement": "default-placement", 2023-07-28T23:40:12.188 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-28T23:40:12.188 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_policy": { 2023-07-28T23:40:12.188 INFO:teuthology.orchestra.run.smithi139.stdout: "groups": [] 2023-07-28T23:40:12.188 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:40:12.189 INFO:teuthology.orchestra.run.smithi139.stdout: "enabled_features": [ 2023-07-28T23:40:12.189 INFO:teuthology.orchestra.run.smithi139.stdout: "resharding" 2023-07-28T23:40:12.189 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2023-07-28T23:40:12.189 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:40:12.189 INFO:teuthology.orchestra.run.smithi139.stdout: ], 2023-07-28T23:40:12.189 INFO:teuthology.orchestra.run.smithi139.stdout: "short_zone_ids": [ 2023-07-28T23:40:12.189 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:40:12.190 INFO:teuthology.orchestra.run.smithi139.stdout: "key": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-28T23:40:12.190 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 2896287231 2023-07-28T23:40:12.190 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:40:12.190 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:40:12.190 INFO:teuthology.orchestra.run.smithi139.stdout: "key": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-28T23:40:12.190 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 815606810 2023-07-28T23:40:12.191 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:40:12.191 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:40:12.191 INFO:teuthology.orchestra.run.smithi139.stdout: "key": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-28T23:40:12.191 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 2009171363 2023-07-28T23:40:12.191 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:40:12.191 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:40:12.191 INFO:teuthology.orchestra.run.smithi139.stdout: "key": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-28T23:40:12.192 INFO:teuthology.orchestra.run.smithi139.stdout: "val": 3105190295 2023-07-28T23:40:12.192 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:40:12.192 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2023-07-28T23:40:12.192 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:40:12.192 INFO:teuthology.orchestra.run.smithi139.stdout: "master_zonegroup": "a", 2023-07-28T23:40:12.192 INFO:teuthology.orchestra.run.smithi139.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-28T23:40:12.193 INFO:teuthology.orchestra.run.smithi139.stdout: "period_config": { 2023-07-28T23:40:12.193 INFO:teuthology.orchestra.run.smithi139.stdout: "bucket_quota": { 2023-07-28T23:40:12.193 INFO:teuthology.orchestra.run.smithi139.stdout: "enabled": false, 2023-07-28T23:40:12.193 INFO:teuthology.orchestra.run.smithi139.stdout: "check_on_raw": false, 2023-07-28T23:40:12.193 INFO:teuthology.orchestra.run.smithi139.stdout: "max_size": -1, 2023-07-28T23:40:12.193 INFO:teuthology.orchestra.run.smithi139.stdout: "max_size_kb": 0, 2023-07-28T23:40:12.193 INFO:teuthology.orchestra.run.smithi139.stdout: "max_objects": -1 2023-07-28T23:40:12.194 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:40:12.194 INFO:teuthology.orchestra.run.smithi139.stdout: "user_quota": { 2023-07-28T23:40:12.194 INFO:teuthology.orchestra.run.smithi139.stdout: "enabled": false, 2023-07-28T23:40:12.194 INFO:teuthology.orchestra.run.smithi139.stdout: "check_on_raw": false, 2023-07-28T23:40:12.194 INFO:teuthology.orchestra.run.smithi139.stdout: "max_size": -1, 2023-07-28T23:40:12.194 INFO:teuthology.orchestra.run.smithi139.stdout: "max_size_kb": 0, 2023-07-28T23:40:12.195 INFO:teuthology.orchestra.run.smithi139.stdout: "max_objects": -1 2023-07-28T23:40:12.195 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:40:12.195 INFO:teuthology.orchestra.run.smithi139.stdout: "user_ratelimit": { 2023-07-28T23:40:12.195 INFO:teuthology.orchestra.run.smithi139.stdout: "max_read_ops": 0, 2023-07-28T23:40:12.195 INFO:teuthology.orchestra.run.smithi139.stdout: "max_write_ops": 0, 2023-07-28T23:40:12.195 INFO:teuthology.orchestra.run.smithi139.stdout: "max_read_bytes": 0, 2023-07-28T23:40:12.195 INFO:teuthology.orchestra.run.smithi139.stdout: "max_write_bytes": 0, 2023-07-28T23:40:12.196 INFO:teuthology.orchestra.run.smithi139.stdout: "enabled": false 2023-07-28T23:40:12.196 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:40:12.196 INFO:teuthology.orchestra.run.smithi139.stdout: "bucket_ratelimit": { 2023-07-28T23:40:12.197 INFO:teuthology.orchestra.run.smithi139.stdout: "max_read_ops": 0, 2023-07-28T23:40:12.197 INFO:teuthology.orchestra.run.smithi139.stdout: "max_write_ops": 0, 2023-07-28T23:40:12.197 INFO:teuthology.orchestra.run.smithi139.stdout: "max_read_bytes": 0, 2023-07-28T23:40:12.197 INFO:teuthology.orchestra.run.smithi139.stdout: "max_write_bytes": 0, 2023-07-28T23:40:12.198 INFO:teuthology.orchestra.run.smithi139.stdout: "enabled": false 2023-07-28T23:40:12.198 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:40:12.198 INFO:teuthology.orchestra.run.smithi139.stdout: "anonymous_ratelimit": { 2023-07-28T23:40:12.198 INFO:teuthology.orchestra.run.smithi139.stdout: "max_read_ops": 0, 2023-07-28T23:40:12.198 INFO:teuthology.orchestra.run.smithi139.stdout: "max_write_ops": 0, 2023-07-28T23:40:12.198 INFO:teuthology.orchestra.run.smithi139.stdout: "max_read_bytes": 0, 2023-07-28T23:40:12.198 INFO:teuthology.orchestra.run.smithi139.stdout: "max_write_bytes": 0, 2023-07-28T23:40:12.199 INFO:teuthology.orchestra.run.smithi139.stdout: "enabled": false 2023-07-28T23:40:12.199 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:40:12.199 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:40:12.199 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-28T23:40:12.199 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_name": "test-realm", 2023-07-28T23:40:12.199 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2023-07-28T23:40:12.199 INFO:teuthology.orchestra.run.smithi139.stdout:} 2023-07-28T23:40:12.201 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2023-07-28T23:40:12.201 INFO:teuthology.orchestra.run:waiting for 300 2023-07-28T23:40:12.202 INFO:tasks.rgw.c2.client.1.smithi139.stdout:2023-07-28T23:40:12.202+0000 7faac213d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 86893) UID: 0 2023-07-28T23:40:12.202 INFO:tasks.rgw.c2.client.1.smithi139.stdout:2023-07-28T23:40:12.202+0000 7faac8234880 -1 shutting down 2023-07-28T23:40:13.999 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2023-07-28T23:40:18.303 INFO:tasks.rgw.c2.client.1:Stopped 2023-07-28T23:40:18.303 INFO:tasks.rgw.c2.client.1:Restarting daemon 2023-07-28T23:40:18.304 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2023-07-28T23:40:18.306 INFO:tasks.rgw.c2.client.1:Started 2023-07-28T23:40:18.307 DEBUG:teuthology.orchestra.run.smithi139:> curl http://smithi139.front.sepia.ceph.com:8001 2023-07-28T23:40:18.327 INFO:teuthology.orchestra.run.smithi139.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-07-28T23:40:18.328 INFO:teuthology.orchestra.run.smithi139.stderr: Dload Upload Total Spent Left Speed 2023-07-28T23:40:18.328 INFO:teuthology.orchestra.run.smithi139.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-07-28T23:40:18.328 INFO:teuthology.orchestra.run.smithi139.stderr:curl: (7) Failed to connect to smithi139.front.sepia.ceph.com port 8001: Connection refused 2023-07-28T23:40:18.328 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-07-28T23:40:19.330 DEBUG:teuthology.orchestra.run.smithi139:> curl http://smithi139.front.sepia.ceph.com:8001 2023-07-28T23:40:19.350 INFO:teuthology.orchestra.run.smithi139.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-07-28T23:40:19.351 INFO:teuthology.orchestra.run.smithi139.stderr: Dload Upload Total Spent Left Speed 2023-07-28T23:40:19.352 INFO:teuthology.orchestra.run.smithi139.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-07-28T23:40:19.353 INFO:teuthology.orchestra.run.smithi139.stdout:anonymous 2023-07-28T23:40:19.354 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2023-07-28T23:40:19.354 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2023-07-28T23:40:19.368 INFO:tasks.rgw_multisite_tests:importing tests from /home/teuthworker/src/github.com_ceph_ceph-c_0484097b0572e96f42cf6402cbe0ac8dcb046577/qa/../src/test/rgw/rgw_multi 2023-07-28T23:40:19.454 INFO:tasks.rgw_multisite_tests:creating test user.. 2023-07-28T23:40:19.454 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:40:19.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:40:19.455 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --caps 'roles=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-28T23:40:19.514 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:40:19.515 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:40:19.550 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:40:19.551 INFO:teuthology.orchestra.run.smithi101.stdout: "user_id": "rgw-multisite-test-user", 2023-07-28T23:40:19.551 INFO:teuthology.orchestra.run.smithi101.stdout: "display_name": "Multisite Test User", 2023-07-28T23:40:19.551 INFO:teuthology.orchestra.run.smithi101.stdout: "email": "", 2023-07-28T23:40:19.551 INFO:teuthology.orchestra.run.smithi101.stdout: "suspended": 0, 2023-07-28T23:40:19.551 INFO:teuthology.orchestra.run.smithi101.stdout: "max_buckets": 1000, 2023-07-28T23:40:19.551 INFO:teuthology.orchestra.run.smithi101.stdout: "subusers": [], 2023-07-28T23:40:19.551 INFO:teuthology.orchestra.run.smithi101.stdout: "keys": [ 2023-07-28T23:40:19.552 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:40:19.552 INFO:teuthology.orchestra.run.smithi101.stdout: "user": "rgw-multisite-test-user", 2023-07-28T23:40:19.552 INFO:teuthology.orchestra.run.smithi101.stdout: "access_key": "S7W2J6K8BFBCG65TLASH", 2023-07-28T23:40:19.552 INFO:teuthology.orchestra.run.smithi101.stdout: "secret_key": "0nbMJF1b5FqgWlYCyAuWsSaGDTm72w1yFTiIvJZK" 2023-07-28T23:40:19.552 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:19.552 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:40:19.552 INFO:teuthology.orchestra.run.smithi101.stdout: "swift_keys": [], 2023-07-28T23:40:19.553 INFO:teuthology.orchestra.run.smithi101.stdout: "caps": [ 2023-07-28T23:40:19.553 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:40:19.553 INFO:teuthology.orchestra.run.smithi101.stdout: "type": "roles", 2023-07-28T23:40:19.553 INFO:teuthology.orchestra.run.smithi101.stdout: "perm": "*" 2023-07-28T23:40:19.553 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:19.553 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:40:19.553 INFO:teuthology.orchestra.run.smithi101.stdout: "op_mask": "read, write, delete", 2023-07-28T23:40:19.554 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "", 2023-07-28T23:40:19.554 INFO:teuthology.orchestra.run.smithi101.stdout: "default_storage_class": "", 2023-07-28T23:40:19.554 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_tags": [], 2023-07-28T23:40:19.554 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_quota": { 2023-07-28T23:40:19.554 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-28T23:40:19.554 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-28T23:40:19.554 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-28T23:40:19.554 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-28T23:40:19.555 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-28T23:40:19.555 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:19.555 INFO:teuthology.orchestra.run.smithi101.stdout: "user_quota": { 2023-07-28T23:40:19.555 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-28T23:40:19.555 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-28T23:40:19.555 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-28T23:40:19.555 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-28T23:40:19.556 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-28T23:40:19.556 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:19.556 INFO:teuthology.orchestra.run.smithi101.stdout: "temp_url_keys": [], 2023-07-28T23:40:19.556 INFO:teuthology.orchestra.run.smithi101.stdout: "type": "rgw", 2023-07-28T23:40:19.556 INFO:teuthology.orchestra.run.smithi101.stdout: "mfa_ids": [] 2023-07-28T23:40:19.556 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:40:19.556 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-28T23:40:19.563 INFO:rgw_multi.tests:meta checkpoint 2023-07-28T23:40:19.564 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:40:19.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:40:19.564 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:40:19.611 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:40:19.612 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:40:19.644 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-28T23:40:19.656 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-28T23:40:19.657 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:40:19.657 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587619.550468_53.1", 2023-07-28T23:40:19.657 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:40:19.550468Z" 2023-07-28T23:40:19.657 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:19.657 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:40:19.657 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:40:19.658 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:40:19.658 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:19.658 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:40:19.658 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:40:19.658 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:40:19.658 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:19.658 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:40:19.659 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:40:19.659 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:40:19.659 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:19.659 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-28T23:40:19.660 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-28T23:40:19.660 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:40:19.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:40:19.661 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:40:19.706 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:40:19.706 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:40:19.732 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:40:19.734+0000 7f47d6529880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:40:19.733 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:40:19.734+0000 7f47d6529880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:40:19.744 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:40:19.744 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:40:19.744 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:40:19.744 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:40:19.745 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:40:19.745 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:40:19.745 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:40:19.745 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:19.745 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:40:19.745 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:40:19.746 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:40:19.746 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:40:19.746 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:40:19.746 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:40:19.746 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:40:19.746 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:40:19.747 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:40:19.747 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:40:19.747 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:40:19.747 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:19.747 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:19.747 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:40:19.747 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:40:19.747 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:40:19.748 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:40:19.748 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:40:19.748 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:40:19.748 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:40:19.748 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:40:19.749 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:40:19.749 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:40:19.749 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:19.749 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:19.749 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:40:19.749 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:40:19.749 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:40:19.749 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:40:19.750 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:40:19.750 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:40:19.750 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:40:19.750 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:40:19.750 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:40:19.750 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:40:19.750 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:19.750 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:19.751 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:40:19.751 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:40:19.751 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:40:19.751 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:40:19.751 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:40:19.751 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:40:19.751 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:40:19.751 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:40:19.752 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:40:19.752 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:40:19.752 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:19.752 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:19.752 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:40:19.752 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:19.753 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:40:19.753 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:40:19.753 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:40:19.753 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:19.753 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:40:19Z" 2023-07-28T23:40:19.753 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:40:19.754 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1690587619.550468_53.1 target= 2023-07-28T23:40:19.760 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2023-07-28T23:40:24.755 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:40:24.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:40:24.756 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:40:24.810 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:40:24.810 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:40:24.841 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:40:24.841+0000 7fbccde96880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:40:24.841 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:40:24.842+0000 7fbccde96880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:40:24.853 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:40:24.854 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:40:24.854 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:40:24.854 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:40:24.854 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:40:24.854 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:40:24.855 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:40:24.855 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:24.855 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:40:24.855 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:40:24.855 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:40:24.856 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:40:24.856 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:40:24.856 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:40:24.856 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:40:24.856 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:40:24.856 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:40:24.856 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:40:24.857 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:40:24.857 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:24.857 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:24.857 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:40:24.857 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:40:24.857 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:40:24.857 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:40:24.858 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:40:24.858 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:40:24.858 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:40:24.858 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:40:24.858 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:40:24.858 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:40:24.858 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:24.858 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:24.859 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:40:24.859 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:40:24.859 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:40:24.859 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:40:24.859 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:40:24.860 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:40:24.860 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:40:24.860 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:40:24.860 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:40:24.860 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:40:24.860 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:24.861 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:24.861 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:40:24.861 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:40:24.861 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:40:24.861 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:40:24.862 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:40:24.862 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:40:24.862 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:40:24.862 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:40:24.862 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:40:24.862 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:40:24.862 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:24.863 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:24.863 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:40:24.863 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:24.863 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:40:24.863 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:40:24.863 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:40:24.863 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:24.864 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:40:24Z" 2023-07-28T23:40:24.864 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:40:24.865 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1690587619.550468_53.1 target= 2023-07-28T23:40:29.866 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:40:29.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:40:29.867 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:40:29.920 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:40:29.920 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:40:29.951 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:40:29.952+0000 7f89fa28b880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:40:29.952 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:40:29.953+0000 7f89fa28b880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:40:29.964 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:40:29.964 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:40:29.964 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:40:29.964 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:40:29.964 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:40:29.965 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:40:29.965 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:40:29.965 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:29.965 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:40:29.965 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:40:29.965 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:40:29.966 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:40:29.966 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:40:29.966 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:40:29.966 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:40:29.966 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:40:29.966 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:40:29.966 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:40:29.966 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:40:29.967 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:29.967 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:29.967 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:40:29.967 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:40:29.967 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:40:29.967 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:40:29.967 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:40:29.968 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:40:29.968 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:40:29.968 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:40:29.968 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:40:29.968 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:40:29.968 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:29.968 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:29.969 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:40:29.969 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:40:29.969 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:40:29.969 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:40:29.969 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:40:29.969 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:40:29.969 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:40:29.970 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:40:29.970 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:40:29.970 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:40:29.970 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:29.970 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:29.970 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:40:29.971 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:40:29.971 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:40:29.971 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:40:29.971 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:40:29.971 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:40:29.971 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:40:29.971 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:40:29.972 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:40:29.972 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:40:29.972 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:29.972 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:29.972 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:40:29.972 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:29.973 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:40:29.973 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:40:29.973 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:40:29.973 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:29.973 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:40:29Z" 2023-07-28T23:40:29.973 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:40:29.974 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1690587619.550468_53.1 target= 2023-07-28T23:40:34.975 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:40:34.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:40:34.976 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:40:35.029 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:40:35.029 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:40:35.059 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:40:35.060+0000 7fc30f711880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:40:35.060 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:40:35.061+0000 7fc30f711880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:40:35.072 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:40:35.072 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:40:35.072 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:40:35.072 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:40:35.073 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:40:35.073 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:40:35.073 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:40:35.073 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:35.073 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:40:35.073 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:40:35.073 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:40:35.074 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:40:35.074 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:40:35.074 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:40:35.074 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:40:35.074 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:40:35.074 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:40:35.074 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:40:35.074 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:40:35.075 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:35.075 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:35.075 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:40:35.075 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:40:35.075 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:40:35.075 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:40:35.075 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:40:35.076 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:40:35.076 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:40:35.076 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:40:35.076 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:40:35.076 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:40:35.076 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:35.076 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:35.077 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:40:35.077 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:40:35.077 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:40:35.077 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:40:35.077 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:40:35.077 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:40:35.077 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:40:35.077 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:40:35.078 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:40:35.078 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:40:35.078 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:35.078 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:35.078 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:40:35.078 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:40:35.078 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:40:35.079 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:40:35.079 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:40:35.079 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:40:35.079 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:40:35.079 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:40:35.079 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:40:35.079 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:40:35.079 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:35.080 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:35.080 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:40:35.080 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:35.080 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:40:35.080 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:40:35.080 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:40:35.080 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:35.081 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:40:35Z" 2023-07-28T23:40:35.081 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:40:35.082 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1690587619.550468_53.1 target= 2023-07-28T23:40:40.083 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:40:40.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:40:40.084 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:40:40.137 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:40:40.138 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:40:40.168 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:40:40.169+0000 7f2b62e10880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:40:40.168 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:40:40.170+0000 7f2b62e10880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:40:40.180 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:40:40.181 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:40:40.181 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:40:40.181 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:40:40.181 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:40:40.181 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:40:40.181 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:40:40.181 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:40.181 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:40:40.182 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:40:40.182 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:40:40.182 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:40:40.182 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:40:40.182 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:40:40.182 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:40:40.183 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:40:40.183 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:40:40.183 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:40:40.183 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:40:40.183 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:40.183 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:40.183 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:40:40.183 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:40:40.184 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:40:40.184 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:40:40.184 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:40:40.184 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:40:40.184 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:40:40.184 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:40:40.184 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:40:40.185 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:40:40.185 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:40.185 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:40.185 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:40:40.185 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:40:40.185 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:40:40.185 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:40:40.185 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:40:40.186 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:40:40.186 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:40:40.186 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:40:40.186 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:40:40.186 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:40:40.186 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:40.186 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:40.186 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:40:40.187 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:40:40.187 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:40:40.187 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:40:40.187 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:40:40.187 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:40:40.187 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:40:40.187 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:40:40.188 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:40:40.188 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:40:40.188 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:40.188 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:40.188 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:40:40.188 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:40.188 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:40:40.188 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:40:40.189 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:40:40.189 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:40.189 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:40:40Z" 2023-07-28T23:40:40.189 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:40:40.190 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1690587619.550468_53.1 target= 2023-07-28T23:40:45.191 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:40:45.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:40:45.192 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:40:45.243 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:40:45.243 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:40:45.275 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:40:45.276+0000 7f1a7df35880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:40:45.275 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:40:45.277+0000 7f1a7df35880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:40:45.287 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:40:45.287 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:40:45.287 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:40:45.287 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:40:45.287 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:40:45.287 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:40:45.288 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:40:45.288 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:45.288 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:40:45.288 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:40:45.288 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:40:45.288 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:40:45.288 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:40:45.289 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:40:45.289 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:40:45.289 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:40:45.289 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:40:45.289 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:40:45.289 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:40:45.289 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:45.289 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:45.290 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:40:45.290 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:40:45.290 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:40:45.290 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:40:45.290 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:40:45.290 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:40:45.290 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:40:45.290 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:40:45.291 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:40:45.291 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:40:45.291 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:45.291 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:45.291 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:40:45.291 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:40:45.291 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:40:45.292 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:40:45.292 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:40:45.292 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:40:45.292 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:40:45.292 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:40:45.292 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:40:45.292 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:40:45.292 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:45.293 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:45.293 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:40:45.293 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:40:45.293 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:40:45.293 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:40:45.293 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:40:45.293 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:40:45.293 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:40:45.294 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:40:45.294 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:40:45.294 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:40:45.294 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:45.294 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:45.294 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:40:45.294 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:45.294 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:40:45.295 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:40:45.295 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:40:45.295 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:45.295 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:40:45Z" 2023-07-28T23:40:45.295 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:40:45.296 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1690587619.550468_53.1 target= 2023-07-28T23:40:50.297 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:40:50.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:40:50.298 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:40:50.351 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:40:50.351 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:40:50.381 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:40:50.382+0000 7f1dc9128880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:40:50.382 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:40:50.383+0000 7f1dc9128880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:40:50.393 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:40:50.394 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:40:50.394 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:40:50.394 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:40:50.394 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:40:50.394 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:40:50.395 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:40:50.395 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:50.395 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:40:50.395 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:40:50.395 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:40:50.395 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:40:50.396 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:40:50.396 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:40:50.396 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:40:50.396 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:40:50.396 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:40:50.396 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:40:50.397 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:40:50.397 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:50.397 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:50.397 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:40:50.397 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:40:50.397 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:40:50.397 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:40:50.397 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:40:50.398 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:40:50.398 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:40:50.398 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:40:50.398 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:40:50.398 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:40:50.398 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:50.398 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:50.399 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:40:50.399 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:40:50.399 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:40:50.399 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:40:50.399 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:40:50.399 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:40:50.400 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:40:50.400 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:40:50.400 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:40:50.400 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:40:50.400 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:50.400 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:50.401 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:40:50.401 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:40:50.401 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:40:50.401 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:40:50.401 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:40:50.401 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:40:50.401 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:40:50.401 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:40:50.402 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:40:50.402 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:40:50.402 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:50.402 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:50.402 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:40:50.402 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:50.402 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:40:50.402 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:40:50.403 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:40:50.403 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:50.403 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:40:50Z" 2023-07-28T23:40:50.403 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:40:50.404 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1690587619.550468_53.1 target= 2023-07-28T23:40:55.406 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:40:55.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:40:55.407 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:40:55.459 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:40:55.460 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:40:55.490 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:40:55.491+0000 7fc2a220f880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:40:55.491 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:40:55.492+0000 7fc2a220f880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:40:55.502 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:40:55.503 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:40:55.503 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:40:55.503 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:40:55.503 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:40:55.503 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:40:55.503 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:40:55.504 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:55.504 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:40:55.504 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:40:55.504 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:40:55.504 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:40:55.504 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:40:55.504 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:40:55.505 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:40:55.505 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:40:55.505 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:40:55.505 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:40:55.505 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:40:55.505 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:55.505 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:55.506 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:40:55.506 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:40:55.506 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:40:55.506 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:40:55.506 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:40:55.506 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:40:55.507 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:40:55.507 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:40:55.507 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:40:55.507 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:40:55.507 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:55.507 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:55.507 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:40:55.508 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:40:55.508 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:40:55.508 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:40:55.508 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:40:55.508 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:40:55.508 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:40:55.509 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:40:55.509 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:40:55.509 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:40:55.509 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:55.509 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:55.509 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:40:55.509 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:40:55.510 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:40:55.510 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:40:55.510 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:40:55.510 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:40:55.510 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:40:55.510 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:40:55.510 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:40:55.511 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:40:55.511 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:55.511 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:40:55.511 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:40:55.511 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:55.511 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:40:55.511 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:40:55.511 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:40:55.512 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:40:55.512 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:40:55Z" 2023-07-28T23:40:55.512 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:40:55.513 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1690587619.550468_53.1 target= 2023-07-28T23:41:00.514 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:41:00.515 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:41:00.515 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:41:00.568 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:41:00.568 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:41:00.598 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:41:00.598+0000 7f1e12f35880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:41:00.598 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:41:00.599+0000 7f1e12f35880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:41:00.610 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:41:00.610 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:41:00.610 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:41:00.611 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:41:00.611 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:41:00.611 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:41:00.611 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:00.611 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:00.611 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:41:00.611 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:00.612 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:41:00.612 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:00.612 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:00.612 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:00.612 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:00.612 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:41:00.612 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:00.613 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:00.613 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:00.613 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:00.613 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:00.613 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:00.613 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:41:00.613 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:00.613 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:00.614 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:00.614 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:00.614 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:41:00.614 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:00.614 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:00.614 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:00.614 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:00.614 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:00.615 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:00.615 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:41:00.615 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:00.615 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:00.615 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:00.615 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:00.615 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:41:00.616 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:00.616 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:00.616 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:00.616 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:00.616 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:00.616 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:00.616 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:41:00.616 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:00.617 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:00.617 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:00.617 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:00.617 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:41:00.617 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:00.617 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:00.617 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:00.617 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:00.618 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:00.618 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:41:00.618 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:00.618 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:41:00.618 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:41:00.618 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:41:00.618 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:00.618 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:41:00Z" 2023-07-28T23:41:00.619 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:41:00.620 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1690587619.550468_53.1 target= 2023-07-28T23:41:05.621 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:41:05.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:41:05.622 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:41:05.677 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:41:05.677 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:41:05.709 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:41:05.710+0000 7f1c6a736880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:41:05.710 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:41:05.711+0000 7f1c6a736880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:41:05.722 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:41:05.722 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:41:05.722 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:41:05.722 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:41:05.723 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:41:05.723 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:41:05.723 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:05.723 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:05.723 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:41:05.723 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:05.723 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:41:05.724 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:05.724 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:05.724 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:05.724 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:05.724 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:41:05.724 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:05.724 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:05.725 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:05.725 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:05.725 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:05.725 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:05.725 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:41:05.725 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:05.725 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:05.725 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:05.726 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:05.726 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:41:05.726 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:05.726 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:05.726 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:05.726 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:05.726 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:05.727 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:05.727 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:41:05.727 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:05.727 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:05.727 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:05.727 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:05.727 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:41:05.727 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:05.728 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:05.728 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:05.728 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:05.728 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:05.728 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:05.728 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:41:05.728 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:05.728 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:05.729 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:05.729 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:05.729 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:41:05.729 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:05.729 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:05.729 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:05.729 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:05.729 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:05.730 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:41:05.730 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:05.730 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:41:05.730 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:41:05.730 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:41:05.730 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:05.730 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:41:05Z" 2023-07-28T23:41:05.731 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:41:05.732 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1690587619.550468_53.1 target= 2023-07-28T23:41:10.733 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:41:10.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:41:10.734 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:41:10.787 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:41:10.787 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:41:10.819 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:41:10.819+0000 7efda1012880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:41:10.819 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:41:10.820+0000 7efda1012880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:41:10.832 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:41:10.832 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:41:10.832 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:41:10.832 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:41:10.832 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:41:10.832 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:41:10.833 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:10.833 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:10.833 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:41:10.833 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:10.833 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:41:10.833 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:10.833 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:10.834 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:10.834 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:10.834 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:41:10.834 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:10.834 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:10.834 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:10.834 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:10.834 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:10.835 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:10.835 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:41:10.835 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:10.835 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:10.835 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:10.835 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:10.835 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:41:10.835 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:10.836 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:10.836 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:10.836 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:10.836 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:10.836 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:10.836 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:41:10.836 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:10.837 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:10.837 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:10.837 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:10.837 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:41:10.837 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:10.837 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:10.837 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:10.838 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:10.838 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:10.838 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:10.838 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:41:10.838 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:10.838 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:10.838 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:10.838 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:10.839 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:41:10.839 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:10.839 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:10.839 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:10.839 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:10.839 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:10.839 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:41:10.839 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:10.840 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:41:10.840 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:41:10.840 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:41:10.840 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:10.840 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:41:10Z" 2023-07-28T23:41:10.840 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:41:10.841 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1690587619.550468_53.1 target= 2023-07-28T23:41:15.843 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:41:15.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:41:15.844 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:41:15.896 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:41:15.896 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:41:15.927 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:41:15.927+0000 7fbde6a14880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:41:15.927 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:41:15.928+0000 7fbde6a14880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:41:15.939 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:41:15.940 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:41:15.940 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:41:15.941 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:41:15.941 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:41:15.941 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:41:15.941 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:15.942 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:15.942 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:41:15.942 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:15.942 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:41:15.942 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:15.942 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:15.942 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:15.943 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:15.943 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:41:15.943 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:15.943 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:15.943 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:15.943 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:15.943 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:15.943 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:15.944 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:41:15.944 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:15.944 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:15.944 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:15.944 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:15.944 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:41:15.944 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:15.945 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:15.945 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:15.945 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:15.945 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:15.945 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:15.945 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:41:15.945 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:15.945 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:15.946 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:15.946 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:15.946 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:41:15.946 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:15.946 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:15.946 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:15.946 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:15.946 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:15.947 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:15.947 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:41:15.947 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:15.947 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:15.947 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:15.947 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:15.947 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:41:15.948 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:15.948 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:15.948 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:15.948 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:15.948 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:15.948 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:41:15.948 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:15.949 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:41:15.949 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:41:15.949 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:41:15.949 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:15.949 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:41:15Z" 2023-07-28T23:41:15.949 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:41:15.950 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1690587619.550468_53.1 target= 2023-07-28T23:41:20.951 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:41:20.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:41:20.952 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:41:21.006 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:41:21.006 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:41:21.036 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:41:21.037+0000 7fafdd40f880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:41:21.036 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:41:21.037+0000 7fafdd40f880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:41:21.049 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:41:21.049 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:41:21.050 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:41:21.050 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:41:21.050 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:41:21.050 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:41:21.050 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:21.050 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:21.050 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:41:21.051 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:21.051 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:41:21.051 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:21.051 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:21.051 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:21.051 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:21.051 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:41:21.052 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:21.052 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:21.052 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:21.052 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:21.052 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:21.052 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:21.052 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:41:21.052 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:21.053 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:21.053 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:21.053 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:21.053 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:41:21.053 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:21.053 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:21.053 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:21.054 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:21.054 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:21.054 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:21.054 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:41:21.054 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:21.054 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:21.054 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:21.054 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:21.055 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:41:21.055 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:21.055 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:21.055 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:21.056 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:21.056 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:21.056 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:21.056 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:41:21.056 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:21.056 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:21.057 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:21.057 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:21.057 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:41:21.057 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:21.057 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:21.057 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:21.057 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:21.058 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:21.058 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:41:21.058 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:21.058 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:41:21.058 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:41:21.058 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:41:21.058 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:21.059 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:41:21Z" 2023-07-28T23:41:21.059 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:41:21.060 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1690587619.550468_53.1 target= 2023-07-28T23:41:26.061 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:41:26.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:41:26.062 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:41:26.115 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:41:26.115 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:41:26.146 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:41:26.147+0000 7ff330f2e880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:41:26.147 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:41:26.148+0000 7ff330f2e880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:41:26.159 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:41:26.159 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:41:26.159 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:41:26.159 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:41:26.159 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:41:26.159 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:41:26.160 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:26.160 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:26.160 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:41:26.160 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:26.160 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:41:26.160 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:26.160 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:26.161 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:26.161 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:26.161 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:41:26.161 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:26.161 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:26.161 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:26.161 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:26.162 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:26.162 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:26.162 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:41:26.162 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:26.162 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:26.162 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:26.163 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:26.163 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:41:26.163 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:26.163 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:26.163 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:26.163 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:26.163 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:26.163 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:26.164 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:41:26.164 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:26.164 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:26.164 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:26.164 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:26.164 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:41:26.164 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:26.165 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:26.165 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:26.165 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:26.165 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:26.165 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:26.165 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:41:26.165 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:26.165 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:26.166 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:26.166 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:26.166 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:41:26.166 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:26.166 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:26.166 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:26.167 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:26.167 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:26.167 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:41:26.167 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:26.167 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:41:26.167 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:41:26.167 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:41:26.167 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:26.168 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:41:26Z" 2023-07-28T23:41:26.168 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:41:26.169 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1690587619.550468_53.1 target= 2023-07-28T23:41:31.169 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:41:31.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:41:31.170 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:41:31.222 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:41:31.222 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:41:31.251 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:41:31.251+0000 7fdc59096880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:41:31.251 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:41:31.252+0000 7fdc59096880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:41:31.263 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:41:31.263 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:41:31.264 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:41:31.264 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:41:31.264 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:41:31.264 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:41:31.264 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:31.264 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:31.264 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:41:31.265 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:31.265 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:41:31.265 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:31.265 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:31.265 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:31.265 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:31.265 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:41:31.266 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:31.266 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:31.266 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:31.266 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:31.266 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:31.266 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:31.266 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:41:31.266 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:31.267 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:31.267 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:31.267 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:31.267 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:41:31.267 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:31.267 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:31.267 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:31.268 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:31.268 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:31.268 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:31.268 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:41:31.268 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:31.269 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:31.269 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:31.269 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:31.269 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:41:31.269 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:31.269 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:31.269 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:31.269 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:31.270 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:31.270 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:31.270 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:41:31.270 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:31.270 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:31.271 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:31.271 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:31.271 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:41:31.271 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:31.271 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:31.271 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:31.272 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:31.272 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:31.272 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:41:31.272 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:31.272 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:41:31.272 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:41:31.272 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:41:31.273 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:31.273 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:41:31Z" 2023-07-28T23:41:31.273 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:41:31.274 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1690587619.550468_53.1 target= 2023-07-28T23:41:36.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:41:36.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:41:36.276 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:41:36.329 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:41:36.329 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:41:36.358 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:41:36.359+0000 7fdf8ad2b880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:41:36.359 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:41:36.360+0000 7fdf8ad2b880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:41:36.371 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:41:36.372 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:41:36.372 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:41:36.372 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:41:36.372 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:41:36.372 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:41:36.372 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:36.373 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:36.373 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:41:36.373 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:36.373 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:41:36.373 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:36.373 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:36.373 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:36.373 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:36.374 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:41:36.374 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:36.374 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:36.374 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:36.374 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:36.374 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:36.374 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:36.375 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:41:36.375 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:36.375 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:36.375 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:36.375 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:36.375 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:41:36.375 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:36.375 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:36.376 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:36.376 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:36.376 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:36.376 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:36.376 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:41:36.376 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:36.376 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:36.376 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:36.377 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:36.377 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:41:36.377 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:36.377 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:36.377 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:36.377 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:36.377 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:36.377 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:36.378 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:41:36.378 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:36.378 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:36.378 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:36.378 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:36.378 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:41:36.378 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:36.379 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:36.379 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:36.379 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:36.379 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:36.379 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:41:36.379 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:36.379 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:41:36.379 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:41:36.380 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:41:36.380 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:36.380 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:41:36Z" 2023-07-28T23:41:36.380 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:41:36.381 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1690587619.550468_53.1 target= 2023-07-28T23:41:41.382 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:41:41.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:41:41.383 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:41:41.436 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:41:41.436 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:41:41.466 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:41:41.467+0000 7f2908c0f880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:41:41.467 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:41:41.468+0000 7f2908c0f880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:41:41.479 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:41:41.479 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:41:41.480 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:41:41.480 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:41:41.480 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:41:41.480 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:41:41.480 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:41.480 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:41.481 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:41:41.481 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:41.481 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:41:41.481 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:41.481 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:41.481 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:41.481 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:41.482 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:41:41.482 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:41.482 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:41.482 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:41.482 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:41.482 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:41.482 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:41.482 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:41:41.483 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:41.483 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:41.483 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:41.483 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:41.483 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:41:41.483 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:41.483 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:41.483 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:41.484 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:41.484 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:41.484 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:41.484 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:41:41.484 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:41.484 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:41.484 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:41.485 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:41.485 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:41:41.485 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:41.485 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:41.485 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:41.485 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:41.485 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:41.485 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:41.486 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:41:41.486 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:41.486 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:41.486 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:41.486 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:41.486 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:41:41.486 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:41.487 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:41.487 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:41.487 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:41.487 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:41.487 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:41:41.487 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:41.487 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:41:41.488 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:41:41.488 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:41:41.488 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:41.488 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:41:41Z" 2023-07-28T23:41:41.488 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:41:41.489 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1690587619.550468_53.1 target= 2023-07-28T23:41:46.490 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:41:46.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:41:46.491 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:41:46.544 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:41:46.544 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:41:46.574 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:41:46.575+0000 7f56aa111880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:41:46.575 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:41:46.576+0000 7f56aa111880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:41:46.587 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:41:46.588 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:41:46.588 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:41:46.588 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:41:46.588 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:41:46.588 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:41:46.589 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:46.589 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:46.589 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:41:46.589 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:46.589 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:41:46.589 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:46.589 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:46.589 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:46.590 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:46.590 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:41:46.590 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:46.590 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:46.590 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:46.590 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:46.590 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:46.591 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:46.591 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:41:46.591 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:46.591 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:46.591 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:46.591 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:46.591 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:41:46.591 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:46.592 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:46.592 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:46.592 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:46.592 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:46.592 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:46.592 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:41:46.592 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:46.593 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:46.593 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:46.593 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:46.593 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:41:46.593 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:46.593 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:46.593 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:46.593 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:46.594 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:46.594 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:46.594 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:41:46.594 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:46.594 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:46.594 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:46.594 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:46.595 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:41:46.595 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:46.595 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:46.595 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:46.595 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:46.595 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:46.596 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:41:46.596 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:46.596 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:41:46.596 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:41:46.596 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:41:46.596 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:46.596 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:41:46Z" 2023-07-28T23:41:46.597 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:41:46.598 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1690587619.550468_53.1 target= 2023-07-28T23:41:51.598 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:41:51.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:41:51.599 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:41:51.652 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:41:51.653 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:41:51.683 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:41:51.684+0000 7f6becb22880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:41:51.684 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:41:51.685+0000 7f6becb22880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:41:51.696 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:41:51.696 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:41:51.696 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:41:51.696 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:41:51.697 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:41:51.697 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:41:51.697 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:51.697 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:51.697 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:41:51.697 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:51.697 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:41:51.698 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:51.698 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:51.698 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:51.698 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:51.698 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:41:51.698 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:51.698 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:51.698 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:51.699 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:51.699 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:51.699 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:51.699 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:41:51.699 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:51.699 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:51.700 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:51.700 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:51.700 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:41:51.700 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:51.700 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:51.700 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:51.701 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:51.701 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:51.701 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:51.701 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:41:51.701 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:51.701 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:51.702 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:51.702 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:51.702 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:41:51.702 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:51.702 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:51.702 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:51.702 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:51.702 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:51.703 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:51.703 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:41:51.703 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:51.703 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:51.703 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:51.703 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:51.703 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:41:51.703 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:51.704 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:51.704 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:51.704 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:51.704 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:51.704 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:41:51.704 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:51.704 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:41:51.705 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:41:51.705 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:41:51.705 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:51.705 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:41:51Z" 2023-07-28T23:41:51.705 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:41:51.706 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1690587619.550468_53.1 target= 2023-07-28T23:41:56.707 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:41:56.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:41:56.708 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:41:56.761 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:41:56.761 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:41:56.792 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:41:56.792+0000 7f21d8210880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:41:56.793 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:41:56.793+0000 7f21d8210880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:41:56.804 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:41:56.805 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:41:56.805 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:41:56.805 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:41:56.805 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:41:56.805 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:41:56.805 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:56.806 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:56.806 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:41:56.806 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:56.806 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:41:56.806 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:56.806 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:56.806 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:56.806 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:56.807 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:41:56.807 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:56.807 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:56.807 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:56.807 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:56.807 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:56.807 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:56.808 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:41:56.808 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:56.808 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:56.808 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:56.808 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:56.808 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:41:56.808 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:56.808 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:56.809 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:56.809 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:56.809 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:56.809 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:56.809 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:41:56.809 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:56.809 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:56.809 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:56.810 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:56.810 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:41:56.810 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:56.810 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:56.810 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:56.810 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:56.810 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:56.811 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:41:56.811 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:41:56.811 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:41:56.811 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:41:56.811 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:41:56.811 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:41:56.811 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:41:56.811 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:41:56.812 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:41:56.812 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:41:56.812 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:56.812 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:41:56.812 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:41:56.812 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:56.812 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:41:56.812 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:41:56.813 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:41:56.813 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:41:56.813 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:41:56Z" 2023-07-28T23:41:56.813 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:41:56.814 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1690587619.550468_53.1 target= 2023-07-28T23:42:01.815 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:01.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:01.816 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:42:01.869 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:42:01.870 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:42:01.899 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:42:01.899+0000 7fbc07135880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:42:01.900 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:42:01.900+0000 7fbc07135880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:42:01.911 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:42:01.912 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:42:01.912 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:42:01.912 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:42:01.913 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:42:01.913 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:42:01.913 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:01.913 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:01.914 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:42:01.914 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:01.914 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:42:01.915 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:01.915 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:01.915 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:01.915 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:01.915 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:42:01.915 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:01.916 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:42:01.916 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:01.916 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:01.916 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:01.916 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:01.916 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:42:01.916 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:01.917 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:01.917 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:01.917 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:01.917 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:42:01.917 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:01.917 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:42:01.917 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:01.917 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:01.918 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:01.918 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:01.918 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:42:01.918 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:01.918 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:01.918 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:01.918 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:01.918 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:42:01.919 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:01.919 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:42:01.919 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:01.919 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:01.919 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:01.919 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:01.919 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:42:01.920 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:01.920 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:01.920 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:01.920 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:01.920 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:42:01.920 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:01.920 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:42:01.920 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:01.921 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:01.921 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:01.921 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:42:01.921 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:01.921 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:42:01.921 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:42:01.921 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:42:01.922 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:01.922 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:42:01Z" 2023-07-28T23:42:01.922 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:42:01.923 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1690587619.550468_53.1 target= 2023-07-28T23:42:06.924 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:06.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:06.925 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:42:06.979 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:42:06.979 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:42:07.012 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:42:07.013+0000 7f9ab4326880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:42:07.013 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:42:07.014+0000 7f9ab4326880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:42:07.025 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:42:07.025 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:42:07.025 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:42:07.026 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:42:07.026 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:42:07.026 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:42:07.026 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:07.026 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:07.026 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:42:07.027 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:07.027 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:42:07.027 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:07.027 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:07.027 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:07.027 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:07.027 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:42:07.028 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:07.028 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:42:07.028 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:07.028 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:07.028 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:07.028 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:07.028 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:42:07.029 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:07.029 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:07.029 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:07.029 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:07.029 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:42:07.029 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:07.029 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:42:07.029 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:07.030 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:07.030 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:07.030 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:07.030 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:42:07.030 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:07.030 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:07.030 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:07.030 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:07.031 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:42:07.031 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:07.031 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:42:07.031 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:07.031 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:07.031 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:07.031 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:07.031 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:42:07.032 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:07.032 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:07.032 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:07.032 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:07.032 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:42:07.032 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:07.032 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:42:07.033 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:07.033 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:07.033 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:07.033 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:42:07.033 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:07.033 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:42:07.033 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:42:07.033 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:42:07.034 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:07.034 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:42:07Z" 2023-07-28T23:42:07.034 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:42:07.035 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1690587619.550468_53.1 target= 2023-07-28T23:42:12.036 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:12.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:12.038 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:42:12.091 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:42:12.091 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:42:12.123 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:42:12.122+0000 7f70c1332880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:42:12.124 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:42:12.123+0000 7f70c1332880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:42:12.134 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:42:12.135 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:42:12.135 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:42:12.135 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:42:12.136 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:42:12.136 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:42:12.136 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:12.136 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:12.137 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:42:12.137 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:12.137 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:42:12.137 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:12.138 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:12.138 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:12.138 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:12.139 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:42:12.139 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:12.140 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:42:12.140 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:12.140 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:12.140 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:12.141 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:12.141 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:42:12.141 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:12.141 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:12.141 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:12.142 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:12.142 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:42:12.142 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:12.143 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:42:12.143 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:12.143 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:12.144 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:12.144 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:12.145 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:42:12.145 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:12.145 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:12.146 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:12.146 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:12.146 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:42:12.146 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:12.147 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:42:12.147 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:12.147 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:12.147 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:12.148 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:12.148 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:42:12.148 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:12.148 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:12.148 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:12.148 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:12.148 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:42:12.149 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:12.149 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:42:12.149 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:12.149 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:12.149 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:12.149 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:42:12.149 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:12.150 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:42:12.150 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:42:12.150 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:42:12.150 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:12.150 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:42:12Z" 2023-07-28T23:42:12.150 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:42:12.151 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1690587619.550468_53.1 target= 2023-07-28T23:42:17.152 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:17.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:17.153 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:42:17.208 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:42:17.209 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:42:17.239 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:42:17.240+0000 7f58d9322880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:42:17.240 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:42:17.241+0000 7f58d9322880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:42:17.252 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:42:17.252 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:42:17.252 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:42:17.253 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:42:17.253 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:42:17.253 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:42:17.253 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:17.253 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:17.253 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:42:17.253 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:17.253 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:42:17.254 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:17.254 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:17.254 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587619.550468_53.1", 2023-07-28T23:42:17.254 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:17.254 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:42:17.254 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:17.254 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:40:19.550468Z", 2023-07-28T23:42:17.255 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:17.255 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:17.255 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:17.255 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:17.255 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:42:17.255 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:17.255 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:17.255 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:17.256 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:17.256 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:42:17.256 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:17.256 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:42:17.256 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:17.256 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:17.256 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:17.256 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:17.257 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:42:17.257 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:17.257 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:17.257 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:17.257 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:17.257 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:42:17.257 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:17.258 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:42:17.258 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:17.258 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:17.258 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:17.258 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:17.258 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:42:17.258 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:17.258 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:17.259 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:17.259 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:17.259 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:42:17.259 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:17.259 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:42:17.259 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:17.259 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:17.259 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:17.260 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:42:17.260 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:17.260 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:42:17.260 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:42:17.260 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:42:17.260 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:17.260 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:42:17Z" 2023-07-28T23:42:17.261 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:42:17.261 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-28T23:42:17.262 INFO:rgw_multi.tests:starting meta checkpoint for zone=b1 2023-07-28T23:42:17.262 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:17.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:17.263 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:42:17.316 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:42:17.316 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:42:17.352 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:42:17.352+0000 7fe19f612880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:42:17.352 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:42:17.353+0000 7fe19f612880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:42:17.364 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2023-07-28T23:42:17.365 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2023-07-28T23:42:17.365 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2023-07-28T23:42:17.365 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2023-07-28T23:42:17.365 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2023-07-28T23:42:17.365 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:42:17.365 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2023-07-28T23:42:17.366 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:42:17.366 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2023-07-28T23:42:17.366 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:42:17.366 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2023-07-28T23:42:17.366 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2023-07-28T23:42:17.366 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2023-07-28T23:42:17.366 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1690587619.550468_53.1", 2023-07-28T23:42:17.367 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2023-07-28T23:42:17.367 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2023-07-28T23:42:17.367 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2023-07-28T23:42:17.367 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2023-07-28T23:40:19.550468Z", 2023-07-28T23:42:17.367 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2023-07-28T23:42:17.367 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:42:17.367 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:42:17.367 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:42:17.368 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2023-07-28T23:42:17.368 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2023-07-28T23:42:17.368 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2023-07-28T23:42:17.368 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2023-07-28T23:42:17.368 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2023-07-28T23:42:17.368 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2023-07-28T23:42:17.368 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2023-07-28T23:42:17.369 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2023-07-28T23:42:17.369 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2023-07-28T23:42:17.369 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:42:17.369 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:42:17.369 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:42:17.369 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2023-07-28T23:42:17.369 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2023-07-28T23:42:17.369 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2023-07-28T23:42:17.370 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2023-07-28T23:42:17.370 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2023-07-28T23:42:17.370 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2023-07-28T23:42:17.370 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2023-07-28T23:42:17.370 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2023-07-28T23:42:17.370 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2023-07-28T23:42:17.370 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:42:17.370 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:42:17.371 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:42:17.371 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2023-07-28T23:42:17.371 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2023-07-28T23:42:17.371 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2023-07-28T23:42:17.371 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2023-07-28T23:42:17.371 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2023-07-28T23:42:17.371 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2023-07-28T23:42:17.371 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2023-07-28T23:42:17.372 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2023-07-28T23:42:17.372 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2023-07-28T23:42:17.372 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:42:17.372 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:42:17.372 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2023-07-28T23:42:17.372 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:42:17.372 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2023-07-28T23:42:17.373 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2023-07-28T23:42:17.373 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2023-07-28T23:42:17.373 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:42:17.373 INFO:teuthology.orchestra.run.smithi139.stdout: "current_time": "2023-07-28T23:42:17Z" 2023-07-28T23:42:17.373 INFO:teuthology.orchestra.run.smithi139.stdout:} 2023-07-28T23:42:17.374 INFO:rgw_multi.tests:finish meta checkpoint for zone=b1 2023-07-28T23:42:17.374 INFO:rgw_multi.tests:starting meta checkpoint for zone=b2 2023-07-28T23:42:17.375 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:17.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:17.375 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:42:17.422 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:42:17.422 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:42:17.452 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:42:17.452+0000 7f8eb520f880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:42:17.452 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:42:17.453+0000 7f8eb520f880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:42:17.463 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2023-07-28T23:42:17.463 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2023-07-28T23:42:17.464 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2023-07-28T23:42:17.464 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2023-07-28T23:42:17.464 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2023-07-28T23:42:17.464 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:42:17.464 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2023-07-28T23:42:17.464 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:42:17.464 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2023-07-28T23:42:17.465 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:42:17.465 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2023-07-28T23:42:17.465 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2023-07-28T23:42:17.465 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2023-07-28T23:42:17.465 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1690587619.550468_53.1", 2023-07-28T23:42:17.465 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2023-07-28T23:42:17.465 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2023-07-28T23:42:17.465 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2023-07-28T23:42:17.466 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2023-07-28T23:40:19.550468Z", 2023-07-28T23:42:17.466 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2023-07-28T23:42:17.466 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:42:17.466 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:42:17.466 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:42:17.466 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2023-07-28T23:42:17.466 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2023-07-28T23:42:17.467 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2023-07-28T23:42:17.467 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2023-07-28T23:42:17.467 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2023-07-28T23:42:17.467 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2023-07-28T23:42:17.467 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2023-07-28T23:42:17.467 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2023-07-28T23:42:17.467 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2023-07-28T23:42:17.467 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:42:17.468 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:42:17.468 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:42:17.468 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2023-07-28T23:42:17.468 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2023-07-28T23:42:17.468 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2023-07-28T23:42:17.468 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2023-07-28T23:42:17.468 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2023-07-28T23:42:17.469 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2023-07-28T23:42:17.469 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2023-07-28T23:42:17.469 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2023-07-28T23:42:17.469 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2023-07-28T23:42:17.469 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:42:17.469 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:42:17.469 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:42:17.469 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2023-07-28T23:42:17.470 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2023-07-28T23:42:17.470 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2023-07-28T23:42:17.470 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2023-07-28T23:42:17.470 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2023-07-28T23:42:17.470 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2023-07-28T23:42:17.470 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2023-07-28T23:42:17.470 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2023-07-28T23:42:17.470 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2023-07-28T23:42:17.471 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:42:17.471 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:42:17.471 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2023-07-28T23:42:17.471 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:42:17.471 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2023-07-28T23:42:17.471 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2023-07-28T23:42:17.471 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2023-07-28T23:42:17.472 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:42:17.472 INFO:teuthology.orchestra.run.smithi139.stdout: "current_time": "2023-07-28T23:42:17Z" 2023-07-28T23:42:17.472 INFO:teuthology.orchestra.run.smithi139.stdout:} 2023-07-28T23:42:17.473 INFO:rgw_multi.tests:finish meta checkpoint for zone=b2 2023-07-28T23:42:17.473 INFO:rgw_multi.tests:meta checkpoint 2023-07-28T23:42:17.473 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:17.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:17.474 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:42:17.521 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:42:17.521 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:42:17.550 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-28T23:42:17.561 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-28T23:42:17.562 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:17.562 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587619.550468_53.1", 2023-07-28T23:42:17.562 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:40:19.550468Z" 2023-07-28T23:42:17.562 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:17.562 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:17.562 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:17.563 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:42:17.563 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:17.563 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:17.563 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:17.563 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:42:17.563 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:17.563 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:17.563 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:17.564 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:42:17.564 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:17.564 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-28T23:42:17.565 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-28T23:42:17.565 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:17.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:17.566 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:42:17.611 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:42:17.612 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:42:17.639 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:42:17.639+0000 7f9294526880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:42:17.639 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:42:17.640+0000 7f9294526880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:42:17.650 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:42:17.651 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:42:17.651 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:42:17.651 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:42:17.651 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:42:17.651 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:42:17.651 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:17.652 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:17.652 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:42:17.652 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:17.652 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:42:17.652 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:17.652 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:17.652 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587619.550468_53.1", 2023-07-28T23:42:17.653 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:17.653 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:42:17.653 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:17.653 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:40:19.550468Z", 2023-07-28T23:42:17.653 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:17.653 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:17.653 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:17.653 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:17.654 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:42:17.654 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:17.654 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:17.654 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:17.654 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:17.654 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:42:17.654 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:17.655 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:42:17.655 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:17.655 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:17.655 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:17.655 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:17.655 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:42:17.655 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:17.655 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:17.656 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:17.656 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:17.656 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:42:17.656 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:17.656 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:42:17.656 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:17.656 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:17.657 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:17.657 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:17.657 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:42:17.657 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:17.657 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:17.657 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:17.657 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:17.657 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:42:17.658 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:17.658 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:42:17.658 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:17.658 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:17.658 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:17.658 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:42:17.658 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:17.659 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:42:17.659 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:42:17.659 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:42:17.659 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:17.659 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:42:17Z" 2023-07-28T23:42:17.659 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:42:17.660 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-28T23:42:17.660 INFO:rgw_multi.tests:starting meta checkpoint for zone=b1 2023-07-28T23:42:17.661 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:17.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:17.661 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:42:17.709 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:42:17.709 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:42:17.737 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:42:17.738+0000 7fd875210880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:42:17.737 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:42:17.738+0000 7fd875210880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:42:17.748 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2023-07-28T23:42:17.749 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2023-07-28T23:42:17.749 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2023-07-28T23:42:17.749 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2023-07-28T23:42:17.749 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2023-07-28T23:42:17.749 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:42:17.749 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2023-07-28T23:42:17.749 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:42:17.750 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2023-07-28T23:42:17.750 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:42:17.750 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2023-07-28T23:42:17.750 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2023-07-28T23:42:17.750 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2023-07-28T23:42:17.750 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1690587619.550468_53.1", 2023-07-28T23:42:17.750 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2023-07-28T23:42:17.750 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2023-07-28T23:42:17.751 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2023-07-28T23:42:17.751 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2023-07-28T23:40:19.550468Z", 2023-07-28T23:42:17.751 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2023-07-28T23:42:17.751 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:42:17.751 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:42:17.751 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:42:17.751 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2023-07-28T23:42:17.752 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2023-07-28T23:42:17.752 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2023-07-28T23:42:17.752 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2023-07-28T23:42:17.752 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2023-07-28T23:42:17.752 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2023-07-28T23:42:17.752 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2023-07-28T23:42:17.753 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2023-07-28T23:42:17.753 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2023-07-28T23:42:17.753 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:42:17.753 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:42:17.753 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:42:17.753 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2023-07-28T23:42:17.753 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2023-07-28T23:42:17.753 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2023-07-28T23:42:17.754 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2023-07-28T23:42:17.754 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2023-07-28T23:42:17.754 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2023-07-28T23:42:17.754 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2023-07-28T23:42:17.754 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2023-07-28T23:42:17.754 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2023-07-28T23:42:17.754 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:42:17.754 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:42:17.755 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:42:17.755 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2023-07-28T23:42:17.755 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2023-07-28T23:42:17.755 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2023-07-28T23:42:17.755 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2023-07-28T23:42:17.755 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2023-07-28T23:42:17.755 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2023-07-28T23:42:17.756 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2023-07-28T23:42:17.756 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2023-07-28T23:42:17.756 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2023-07-28T23:42:17.756 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:42:17.756 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:42:17.756 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2023-07-28T23:42:17.756 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:42:17.757 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2023-07-28T23:42:17.757 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2023-07-28T23:42:17.757 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2023-07-28T23:42:17.757 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:42:17.757 INFO:teuthology.orchestra.run.smithi139.stdout: "current_time": "2023-07-28T23:42:17Z" 2023-07-28T23:42:17.757 INFO:teuthology.orchestra.run.smithi139.stdout:} 2023-07-28T23:42:17.758 INFO:rgw_multi.tests:finish meta checkpoint for zone=b1 2023-07-28T23:42:17.758 INFO:rgw_multi.tests:starting meta checkpoint for zone=b2 2023-07-28T23:42:17.759 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:17.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:17.759 DEBUG:teuthology.orchestra.run.smithi139:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:42:17.805 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:42:17.805 INFO:teuthology.orchestra.run.smithi139.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:42:17.831 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:42:17.832+0000 7fd1e0526880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:42:17.832 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-28T23:42:17.833+0000 7fd1e0526880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:42:17.843 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2023-07-28T23:42:17.843 INFO:teuthology.orchestra.run.smithi139.stdout: "sync_status": { 2023-07-28T23:42:17.843 INFO:teuthology.orchestra.run.smithi139.stdout: "info": { 2023-07-28T23:42:17.844 INFO:teuthology.orchestra.run.smithi139.stdout: "status": "sync", 2023-07-28T23:42:17.844 INFO:teuthology.orchestra.run.smithi139.stdout: "num_shards": 4, 2023-07-28T23:42:17.844 INFO:teuthology.orchestra.run.smithi139.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:42:17.844 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2023-07-28T23:42:17.844 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:42:17.844 INFO:teuthology.orchestra.run.smithi139.stdout: "markers": [ 2023-07-28T23:42:17.844 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:42:17.845 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 0, 2023-07-28T23:42:17.845 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2023-07-28T23:42:17.845 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2023-07-28T23:42:17.845 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "1_1690587619.550468_53.1", 2023-07-28T23:42:17.845 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2023-07-28T23:42:17.845 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 1, 2023-07-28T23:42:17.845 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2023-07-28T23:42:17.845 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "2023-07-28T23:40:19.550468Z", 2023-07-28T23:42:17.846 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2023-07-28T23:42:17.846 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:42:17.846 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:42:17.846 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:42:17.846 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 1, 2023-07-28T23:42:17.847 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2023-07-28T23:42:17.847 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2023-07-28T23:42:17.847 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2023-07-28T23:42:17.847 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2023-07-28T23:42:17.847 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2023-07-28T23:42:17.847 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2023-07-28T23:42:17.847 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2023-07-28T23:42:17.848 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2023-07-28T23:42:17.848 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:42:17.848 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:42:17.848 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:42:17.848 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 2, 2023-07-28T23:42:17.848 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2023-07-28T23:42:17.848 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2023-07-28T23:42:17.848 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2023-07-28T23:42:17.849 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2023-07-28T23:42:17.849 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2023-07-28T23:42:17.849 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2023-07-28T23:42:17.849 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2023-07-28T23:42:17.849 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2023-07-28T23:42:17.849 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:42:17.849 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:42:17.850 INFO:teuthology.orchestra.run.smithi139.stdout: { 2023-07-28T23:42:17.850 INFO:teuthology.orchestra.run.smithi139.stdout: "key": 3, 2023-07-28T23:42:17.850 INFO:teuthology.orchestra.run.smithi139.stdout: "val": { 2023-07-28T23:42:17.850 INFO:teuthology.orchestra.run.smithi139.stdout: "state": 1, 2023-07-28T23:42:17.850 INFO:teuthology.orchestra.run.smithi139.stdout: "marker": "", 2023-07-28T23:42:17.850 INFO:teuthology.orchestra.run.smithi139.stdout: "next_step_marker": "", 2023-07-28T23:42:17.850 INFO:teuthology.orchestra.run.smithi139.stdout: "total_entries": 0, 2023-07-28T23:42:17.850 INFO:teuthology.orchestra.run.smithi139.stdout: "pos": 0, 2023-07-28T23:42:17.851 INFO:teuthology.orchestra.run.smithi139.stdout: "timestamp": "0.000000", 2023-07-28T23:42:17.851 INFO:teuthology.orchestra.run.smithi139.stdout: "realm_epoch": 2 2023-07-28T23:42:17.851 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:42:17.851 INFO:teuthology.orchestra.run.smithi139.stdout: } 2023-07-28T23:42:17.851 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2023-07-28T23:42:17.851 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:42:17.851 INFO:teuthology.orchestra.run.smithi139.stdout: "full_sync": { 2023-07-28T23:42:17.852 INFO:teuthology.orchestra.run.smithi139.stdout: "total": 1, 2023-07-28T23:42:17.852 INFO:teuthology.orchestra.run.smithi139.stdout: "complete": 1 2023-07-28T23:42:17.852 INFO:teuthology.orchestra.run.smithi139.stdout: }, 2023-07-28T23:42:17.852 INFO:teuthology.orchestra.run.smithi139.stdout: "current_time": "2023-07-28T23:42:17Z" 2023-07-28T23:42:17.852 INFO:teuthology.orchestra.run.smithi139.stdout:} 2023-07-28T23:42:17.853 INFO:rgw_multi.tests:finish meta checkpoint for zone=b2 2023-07-28T23:42:17.853 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on '/home/teuthworker/src/github.com_ceph_ceph-c_0484097b0572e96f42cf6402cbe0ac8dcb046577/qa/../src/test/rgw/rgw_multi' with args=['tests.py'] 2023-07-28T23:42:18.021 DEBUG:boto:Using access key provided by client. 2023-07-28T23:42:18.023 DEBUG:boto:Using secret key provided by client. 2023-07-28T23:42:18.024 DEBUG:boto:Using access key provided by client. 2023-07-28T23:42:18.026 DEBUG:boto:Using secret key provided by client. 2023-07-28T23:42:18.027 DEBUG:boto:Using access key provided by client. 2023-07-28T23:42:18.027 DEBUG:boto:Using secret key provided by client. 2023-07-28T23:42:18.028 DEBUG:boto:Using access key provided by client. 2023-07-28T23:42:18.029 DEBUG:boto:Using secret key provided by client. 2023-07-28T23:42:18.030 INFO:rgw_multi.tests:create bucket zone=a1 name=hiiqos-1 2023-07-28T23:42:18.031 DEBUG:boto:path=/hiiqos-1/ 2023-07-28T23:42:18.031 DEBUG:boto:auth_path=/hiiqos-1/ 2023-07-28T23:42:18.031 DEBUG:boto:Method: PUT 2023-07-28T23:42:18.032 DEBUG:boto:Path: /hiiqos-1/ 2023-07-28T23:42:18.032 DEBUG:boto:Data: 2023-07-28T23:42:18.032 DEBUG:boto:Headers: {} 2023-07-28T23:42:18.033 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:42:18.033 DEBUG:boto:Port: 8000 2023-07-28T23:42:18.033 DEBUG:boto:Params: {} 2023-07-28T23:42:18.034 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-28T23:42:18.034 DEBUG:boto:Token: None 2023-07-28T23:42:18.035 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:42:18 GMT /hiiqos-1/ 2023-07-28T23:42:18.035 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lbkfYfbNPrxS6qE+H5+UXiR7oqE= 2023-07-28T23:42:18.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lbkfYfbNPrxS6qE+H5+UXiR7oqE=', 'Content-Length': '0'} 2023-07-28T23:42:25.162 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000098467c0495d88e89-0064c4525f-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:42:25 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:25.162 INFO:rgw_multi.tests:create bucket zone=a2 name=hiiqos-2 2023-07-28T23:42:25.163 DEBUG:boto:path=/hiiqos-2/ 2023-07-28T23:42:25.163 DEBUG:boto:auth_path=/hiiqos-2/ 2023-07-28T23:42:25.164 DEBUG:boto:Method: PUT 2023-07-28T23:42:25.164 DEBUG:boto:Path: /hiiqos-2/ 2023-07-28T23:42:25.164 DEBUG:boto:Data: 2023-07-28T23:42:25.165 DEBUG:boto:Headers: {} 2023-07-28T23:42:25.165 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:42:25.165 DEBUG:boto:Port: 8001 2023-07-28T23:42:25.165 DEBUG:boto:Params: {} 2023-07-28T23:42:25.166 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-28T23:42:25.166 DEBUG:boto:Token: None 2023-07-28T23:42:25.167 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:42:25 GMT /hiiqos-2/ 2023-07-28T23:42:25.167 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yVYeh6qqXiH/Dq4Vra+OdVITxi0= 2023-07-28T23:42:25.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:25 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yVYeh6qqXiH/Dq4Vra+OdVITxi0=', 'Content-Length': '0'} 2023-07-28T23:42:30.206 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f800ac8b88811358-0064c45266-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:42:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:30.207 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:30.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:30.209 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:42:30.263 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:42:30.263 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:42:30.294 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-28T23:42:30.306 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-28T23:42:30.306 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:30.306 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587750.192787_61.1", 2023-07-28T23:42:30.307 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:42:30.192787Z" 2023-07-28T23:42:30.307 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:30.307 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:30.307 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:30.307 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:42:30.307 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:30.307 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:30.308 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:30.308 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:42:30.308 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:30.308 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:30.308 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:30.309 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:42:30.309 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:30.309 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-28T23:42:30.310 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690587750.192787_61.1', 1: '', 2: '', 3: ''} 2023-07-28T23:42:30.311 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-28T23:42:30.311 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:30.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:30.312 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:42:30.358 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:42:30.359 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:42:30.386 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:42:30.386+0000 7fe936216880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:42:30.387 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:42:30.387+0000 7fe936216880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:42:30.397 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:42:30.397 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:42:30.398 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:42:30.398 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:42:30.398 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:42:30.398 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:42:30.398 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:30.398 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:30.399 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:42:30.399 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:30.399 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:42:30.399 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:30.399 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:30.399 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587745.149133_54.1", 2023-07-28T23:42:30.399 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:30.400 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:42:30.400 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:30.400 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:42:25.149133Z", 2023-07-28T23:42:30.400 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:30.400 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:30.400 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:30.400 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:30.401 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:42:30.401 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:30.401 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:30.401 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:30.401 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:30.401 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:42:30.402 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:30.402 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:42:30.402 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:30.402 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:30.402 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:30.402 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:30.402 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:42:30.403 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:30.403 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:30.403 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:30.403 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:30.403 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:42:30.403 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:30.403 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:42:30.404 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:30.404 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:30.404 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:30.404 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:30.404 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:42:30.404 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:30.405 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:30.405 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:30.405 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:30.405 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:42:30.405 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:30.405 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:42:30.405 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:30.406 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:30.406 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:30.406 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:42:30.406 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:30.406 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:42:30.406 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:42:30.406 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:42:30.407 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:30.407 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:42:30Z" 2023-07-28T23:42:30.407 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:42:30.408 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587745.149133_54.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:42:25.149133Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-28T23:42:30Z" } 2023-07-28T23:42:30.408 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587745.149133_54.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:42:25.149133Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-28T23:42:30.409 DEBUG:rgw_multi.tests:log_status={0: '1_1690587750.192787_61.1', 1: '', 2: '', 3: ''} 2023-07-28T23:42:30.409 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587745.149133_54.1', 1: '', 2: '', 3: ''} 2023-07-28T23:42:30.409 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1690587750.192787_61.1 target=1_1690587745.149133_54.1 2023-07-28T23:42:35.411 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:35.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:35.413 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:42:35.466 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:42:35.467 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:42:35.497 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:42:35.497+0000 7f74ca036880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:42:35.498 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:42:35.498+0000 7f74ca036880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:42:35.510 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:42:35.510 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:42:35.511 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:42:35.511 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:42:35.511 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:42:35.511 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:42:35.511 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:35.511 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:35.511 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:42:35.512 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:35.512 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:42:35.512 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:35.512 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:35.512 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587750.192787_61.1", 2023-07-28T23:42:35.512 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:35.513 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:42:35.513 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:35.513 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:42:30.192787Z", 2023-07-28T23:42:35.513 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:35.513 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:35.513 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:35.514 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:35.514 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:42:35.514 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:35.514 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:35.514 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:35.514 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:35.514 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:42:35.515 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:35.515 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:42:35.515 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:35.515 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:35.515 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:35.515 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:35.515 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:42:35.516 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:35.516 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:35.516 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:35.516 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:35.516 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:42:35.516 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:35.516 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:42:35.518 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:35.518 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:35.518 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:35.518 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:35.519 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:42:35.519 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:35.519 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:35.519 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:35.519 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:35.519 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:42:35.519 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:35.520 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:42:35.520 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:35.520 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:35.520 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:35.520 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:42:35.520 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:35.521 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:42:35.521 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:42:35.521 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:42:35.521 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:35.521 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:42:35Z" 2023-07-28T23:42:35.521 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:42:35.522 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587750.192787_61.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:42:30.192787Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-28T23:42:35Z" } 2023-07-28T23:42:35.523 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587750.192787_61.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:42:30.192787Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-28T23:42:35.523 DEBUG:rgw_multi.tests:log_status={0: '1_1690587750.192787_61.1', 1: '', 2: '', 3: ''} 2023-07-28T23:42:35.524 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587750.192787_61.1', 1: '', 2: '', 3: ''} 2023-07-28T23:42:35.524 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-28T23:42:35.524 DEBUG:boto:path=/hiiqos-1/ 2023-07-28T23:42:35.525 DEBUG:boto:auth_path=/hiiqos-1/ 2023-07-28T23:42:35.525 DEBUG:boto:Method: HEAD 2023-07-28T23:42:35.525 DEBUG:boto:Path: /hiiqos-1/ 2023-07-28T23:42:35.526 DEBUG:boto:Data: 2023-07-28T23:42:35.526 DEBUG:boto:Headers: {} 2023-07-28T23:42:35.526 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:42:35.527 DEBUG:boto:Port: 8000 2023-07-28T23:42:35.527 DEBUG:boto:Params: {} 2023-07-28T23:42:35.527 DEBUG:boto:Token: None 2023-07-28T23:42:35.528 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:42:35 GMT /hiiqos-1/ 2023-07-28T23:42:35.528 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mjRW7nocCLGQIPk1A2yTUOqgcwg= 2023-07-28T23:42:35.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mjRW7nocCLGQIPk1A2yTUOqgcwg=', 'Content-Length': '0'} 2023-07-28T23:42:35.531 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006b9705ab61901cad-0064c4526b-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:42:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:35.531 DEBUG:boto:path=/hiiqos-2/ 2023-07-28T23:42:35.532 DEBUG:boto:auth_path=/hiiqos-2/ 2023-07-28T23:42:35.532 DEBUG:boto:Method: HEAD 2023-07-28T23:42:35.532 DEBUG:boto:Path: /hiiqos-2/ 2023-07-28T23:42:35.533 DEBUG:boto:Data: 2023-07-28T23:42:35.533 DEBUG:boto:Headers: {} 2023-07-28T23:42:35.533 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:42:35.534 DEBUG:boto:Port: 8000 2023-07-28T23:42:35.534 DEBUG:boto:Params: {} 2023-07-28T23:42:35.534 DEBUG:boto:Token: None 2023-07-28T23:42:35.535 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:42:35 GMT /hiiqos-2/ 2023-07-28T23:42:35.535 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Dtbchc3XCw2ourLkStsdlzAyhl0= 2023-07-28T23:42:35.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Dtbchc3XCw2ourLkStsdlzAyhl0=', 'Content-Length': '0'} 2023-07-28T23:42:35.538 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004214a6362b06f7fc-0064c4526b-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:42:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:35.538 DEBUG:boto:path=/hiiqos-1/ 2023-07-28T23:42:35.539 DEBUG:boto:auth_path=/hiiqos-1/ 2023-07-28T23:42:35.539 DEBUG:boto:Method: HEAD 2023-07-28T23:42:35.539 DEBUG:boto:Path: /hiiqos-1/ 2023-07-28T23:42:35.540 DEBUG:boto:Data: 2023-07-28T23:42:35.540 DEBUG:boto:Headers: {} 2023-07-28T23:42:35.540 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:42:35.541 DEBUG:boto:Port: 8001 2023-07-28T23:42:35.541 DEBUG:boto:Params: {} 2023-07-28T23:42:35.541 DEBUG:boto:Token: None 2023-07-28T23:42:35.542 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:42:35 GMT /hiiqos-1/ 2023-07-28T23:42:35.542 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mjRW7nocCLGQIPk1A2yTUOqgcwg= 2023-07-28T23:42:35.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mjRW7nocCLGQIPk1A2yTUOqgcwg=', 'Content-Length': '0'} 2023-07-28T23:42:35.545 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000006b40fd594271cbd-0064c4526b-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:42:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:35.545 DEBUG:boto:path=/hiiqos-2/ 2023-07-28T23:42:35.546 DEBUG:boto:auth_path=/hiiqos-2/ 2023-07-28T23:42:35.546 DEBUG:boto:Method: HEAD 2023-07-28T23:42:35.547 DEBUG:boto:Path: /hiiqos-2/ 2023-07-28T23:42:35.547 DEBUG:boto:Data: 2023-07-28T23:42:35.547 DEBUG:boto:Headers: {} 2023-07-28T23:42:35.547 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:42:35.548 DEBUG:boto:Port: 8001 2023-07-28T23:42:35.548 DEBUG:boto:Params: {} 2023-07-28T23:42:35.549 DEBUG:boto:Token: None 2023-07-28T23:42:35.549 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:42:35 GMT /hiiqos-2/ 2023-07-28T23:42:35.549 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Dtbchc3XCw2ourLkStsdlzAyhl0= 2023-07-28T23:42:35.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Dtbchc3XCw2ourLkStsdlzAyhl0=', 'Content-Length': '0'} 2023-07-28T23:42:35.552 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc7a33bc83d03f19-0064c4526b-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:42:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:35.553 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_create ... ok 2023-07-28T23:42:35.554 INFO:rgw_multi.tests:create bucket zone=a1 name=hiiqos-3 2023-07-28T23:42:35.554 DEBUG:boto:path=/hiiqos-3/ 2023-07-28T23:42:35.555 DEBUG:boto:auth_path=/hiiqos-3/ 2023-07-28T23:42:35.555 DEBUG:boto:Method: PUT 2023-07-28T23:42:35.555 DEBUG:boto:Path: /hiiqos-3/ 2023-07-28T23:42:35.556 DEBUG:boto:Data: 2023-07-28T23:42:35.556 DEBUG:boto:Headers: {} 2023-07-28T23:42:35.557 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:42:35.557 DEBUG:boto:Port: 8000 2023-07-28T23:42:35.557 DEBUG:boto:Params: {} 2023-07-28T23:42:35.558 DEBUG:boto:Token: None 2023-07-28T23:42:35.558 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:42:35 GMT /hiiqos-3/ 2023-07-28T23:42:35.559 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LQXHrUCcXTlqN6VtKlXFw+71YB8= 2023-07-28T23:42:35.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LQXHrUCcXTlqN6VtKlXFw+71YB8=', 'Content-Length': '0'} 2023-07-28T23:42:35.574 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000059e7e907a949d721-0064c4526b-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:42:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:35.575 INFO:rgw_multi.tests:create bucket zone=a2 name=hiiqos-4 2023-07-28T23:42:35.575 DEBUG:boto:path=/hiiqos-4/ 2023-07-28T23:42:35.576 DEBUG:boto:auth_path=/hiiqos-4/ 2023-07-28T23:42:35.577 DEBUG:boto:Method: PUT 2023-07-28T23:42:35.577 DEBUG:boto:Path: /hiiqos-4/ 2023-07-28T23:42:35.578 DEBUG:boto:Data: 2023-07-28T23:42:35.579 DEBUG:boto:Headers: {} 2023-07-28T23:42:35.579 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:42:35.580 DEBUG:boto:Port: 8001 2023-07-28T23:42:35.580 DEBUG:boto:Params: {} 2023-07-28T23:42:35.581 DEBUG:boto:Token: None 2023-07-28T23:42:35.581 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:42:35 GMT /hiiqos-4/ 2023-07-28T23:42:35.581 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9lZULXBX9EOGcE6l6Bjq2xjX4BY= 2023-07-28T23:42:35.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9lZULXBX9EOGcE6l6Bjq2xjX4BY=', 'Content-Length': '0'} 2023-07-28T23:42:35.602 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000af160f286624dbfd-0064c4526b-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:42:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:35.603 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:35.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:35.605 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:42:35.654 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:42:35.654 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:42:35.685 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-28T23:42:35.696 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-28T23:42:35.697 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:35.697 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587755.595725_69.1", 2023-07-28T23:42:35.697 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:42:35.595725Z" 2023-07-28T23:42:35.698 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:35.698 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:35.698 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:35.699 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:42:35.699 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:35.699 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:35.700 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:35.700 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:42:35.700 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:35.700 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:35.700 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:35.700 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:42:35.701 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:35.701 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-28T23:42:35.702 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690587755.595725_69.1', 1: '', 2: '', 3: ''} 2023-07-28T23:42:35.702 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-28T23:42:35.703 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:35.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:35.703 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:42:35.749 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:42:35.749 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:42:35.775 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:42:35.776+0000 7f4476d22880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:42:35.776 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:42:35.777+0000 7f4476d22880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:42:35.787 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:42:35.787 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:42:35.788 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:42:35.788 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:42:35.788 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:42:35.789 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:42:35.789 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:35.789 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:35.789 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:42:35.790 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:35.790 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:42:35.790 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:35.790 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:35.791 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587755.595725_69.1", 2023-07-28T23:42:35.791 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:35.791 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:42:35.791 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:35.791 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:42:35.595725Z", 2023-07-28T23:42:35.792 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:35.792 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:35.792 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:35.792 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:35.793 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:42:35.793 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:35.793 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:35.793 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:35.793 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:35.793 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:42:35.793 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:35.794 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:42:35.794 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:35.794 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:35.795 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:35.795 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:35.795 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:42:35.795 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:35.795 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:35.795 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:35.795 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:35.796 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:42:35.796 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:35.796 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:42:35.796 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:35.796 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:35.796 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:35.797 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:35.797 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:42:35.797 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:35.797 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:35.797 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:35.797 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:35.797 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:42:35.798 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:35.798 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:42:35.798 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:35.799 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:35.799 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:35.799 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:42:35.799 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:35.799 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:42:35.799 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:42:35.800 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:42:35.800 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:35.800 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:42:35Z" 2023-07-28T23:42:35.800 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:42:35.801 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587755.595725_69.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:42:35.595725Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-28T23:42:35Z" } 2023-07-28T23:42:35.801 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587755.595725_69.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:42:35.595725Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-28T23:42:35.802 DEBUG:rgw_multi.tests:log_status={0: '1_1690587755.595725_69.1', 1: '', 2: '', 3: ''} 2023-07-28T23:42:35.802 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587755.595725_69.1', 1: '', 2: '', 3: ''} 2023-07-28T23:42:35.803 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-28T23:42:35.803 DEBUG:boto:path=/hiiqos-3/ 2023-07-28T23:42:35.804 DEBUG:boto:auth_path=/hiiqos-3/ 2023-07-28T23:42:35.804 DEBUG:boto:Method: HEAD 2023-07-28T23:42:35.804 DEBUG:boto:Path: /hiiqos-3/ 2023-07-28T23:42:35.805 DEBUG:boto:Data: 2023-07-28T23:42:35.805 DEBUG:boto:Headers: {} 2023-07-28T23:42:35.806 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:42:35.806 DEBUG:boto:Port: 8000 2023-07-28T23:42:35.806 DEBUG:boto:Params: {} 2023-07-28T23:42:35.807 DEBUG:boto:Token: None 2023-07-28T23:42:35.807 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:42:35 GMT /hiiqos-3/ 2023-07-28T23:42:35.808 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KaD5j6o7LwOppqRLeNOiqdTPRpk= 2023-07-28T23:42:35.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KaD5j6o7LwOppqRLeNOiqdTPRpk=', 'Content-Length': '0'} 2023-07-28T23:42:35.810 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ea7e03774436a80f-0064c4526b-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:42:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:35.811 DEBUG:boto:path=/hiiqos-4/ 2023-07-28T23:42:35.811 DEBUG:boto:auth_path=/hiiqos-4/ 2023-07-28T23:42:35.811 DEBUG:boto:Method: HEAD 2023-07-28T23:42:35.812 DEBUG:boto:Path: /hiiqos-4/ 2023-07-28T23:42:35.812 DEBUG:boto:Data: 2023-07-28T23:42:35.813 DEBUG:boto:Headers: {} 2023-07-28T23:42:35.813 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:42:35.814 DEBUG:boto:Port: 8000 2023-07-28T23:42:35.814 DEBUG:boto:Params: {} 2023-07-28T23:42:35.815 DEBUG:boto:Token: None 2023-07-28T23:42:35.815 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:42:35 GMT /hiiqos-4/ 2023-07-28T23:42:35.816 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cO93XmPsmP5lNwAqvKFOWtBnR2o= 2023-07-28T23:42:35.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cO93XmPsmP5lNwAqvKFOWtBnR2o=', 'Content-Length': '0'} 2023-07-28T23:42:35.818 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b3f27ce79a05572f-0064c4526b-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:42:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:35.819 DEBUG:boto:path=/hiiqos-3/ 2023-07-28T23:42:35.819 DEBUG:boto:auth_path=/hiiqos-3/ 2023-07-28T23:42:35.819 DEBUG:boto:Method: HEAD 2023-07-28T23:42:35.820 DEBUG:boto:Path: /hiiqos-3/ 2023-07-28T23:42:35.820 DEBUG:boto:Data: 2023-07-28T23:42:35.820 DEBUG:boto:Headers: {} 2023-07-28T23:42:35.821 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:42:35.821 DEBUG:boto:Port: 8001 2023-07-28T23:42:35.821 DEBUG:boto:Params: {} 2023-07-28T23:42:35.822 DEBUG:boto:Token: None 2023-07-28T23:42:35.822 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:42:35 GMT /hiiqos-3/ 2023-07-28T23:42:35.823 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KaD5j6o7LwOppqRLeNOiqdTPRpk= 2023-07-28T23:42:35.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KaD5j6o7LwOppqRLeNOiqdTPRpk=', 'Content-Length': '0'} 2023-07-28T23:42:35.825 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000caf16b43b3afecaa-0064c4526b-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:42:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:35.826 DEBUG:boto:path=/hiiqos-4/ 2023-07-28T23:42:35.826 DEBUG:boto:auth_path=/hiiqos-4/ 2023-07-28T23:42:35.826 DEBUG:boto:Method: HEAD 2023-07-28T23:42:35.827 DEBUG:boto:Path: /hiiqos-4/ 2023-07-28T23:42:35.827 DEBUG:boto:Data: 2023-07-28T23:42:35.827 DEBUG:boto:Headers: {} 2023-07-28T23:42:35.828 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:42:35.828 DEBUG:boto:Port: 8001 2023-07-28T23:42:35.828 DEBUG:boto:Params: {} 2023-07-28T23:42:35.829 DEBUG:boto:Token: None 2023-07-28T23:42:35.829 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:42:35 GMT /hiiqos-4/ 2023-07-28T23:42:35.829 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cO93XmPsmP5lNwAqvKFOWtBnR2o= 2023-07-28T23:42:35.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cO93XmPsmP5lNwAqvKFOWtBnR2o=', 'Content-Length': '0'} 2023-07-28T23:42:35.832 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b7fa5535c7fa0d83-0064c4526b-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:42:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:35.832 DEBUG:boto:path=/hiiqos-3/ 2023-07-28T23:42:35.832 DEBUG:boto:auth_path=/hiiqos-3/ 2023-07-28T23:42:35.833 DEBUG:boto:Method: PUT 2023-07-28T23:42:35.833 DEBUG:boto:Path: /hiiqos-3/ 2023-07-28T23:42:35.833 DEBUG:boto:Data: 2023-07-28T23:42:35.834 DEBUG:boto:Headers: {} 2023-07-28T23:42:35.834 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:42:35.834 DEBUG:boto:Port: 8000 2023-07-28T23:42:35.835 DEBUG:boto:Params: {} 2023-07-28T23:42:35.835 DEBUG:boto:Token: None 2023-07-28T23:42:35.835 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:42:35 GMT /hiiqos-3/ 2023-07-28T23:42:35.836 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LQXHrUCcXTlqN6VtKlXFw+71YB8= 2023-07-28T23:42:35.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LQXHrUCcXTlqN6VtKlXFw+71YB8=', 'Content-Length': '0'} 2023-07-28T23:42:35.839 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000009646e049b546615-0064c4526b-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:42:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:35.839 DEBUG:boto:path=/hiiqos-4/ 2023-07-28T23:42:35.840 DEBUG:boto:auth_path=/hiiqos-4/ 2023-07-28T23:42:35.840 DEBUG:boto:Method: PUT 2023-07-28T23:42:35.840 DEBUG:boto:Path: /hiiqos-4/ 2023-07-28T23:42:35.841 DEBUG:boto:Data: 2023-07-28T23:42:35.841 DEBUG:boto:Headers: {} 2023-07-28T23:42:35.841 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:42:35.842 DEBUG:boto:Port: 8000 2023-07-28T23:42:35.842 DEBUG:boto:Params: {} 2023-07-28T23:42:35.842 DEBUG:boto:Token: None 2023-07-28T23:42:35.843 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:42:35 GMT /hiiqos-4/ 2023-07-28T23:42:35.843 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9lZULXBX9EOGcE6l6Bjq2xjX4BY= 2023-07-28T23:42:35.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9lZULXBX9EOGcE6l6Bjq2xjX4BY=', 'Content-Length': '0'} 2023-07-28T23:42:35.845 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004ee5a14ccfec5ec8-0064c4526b-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:42:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:35.846 DEBUG:boto:path=/hiiqos-3/ 2023-07-28T23:42:35.846 DEBUG:boto:auth_path=/hiiqos-3/ 2023-07-28T23:42:35.847 DEBUG:boto:Method: PUT 2023-07-28T23:42:35.847 DEBUG:boto:Path: /hiiqos-3/ 2023-07-28T23:42:35.847 DEBUG:boto:Data: 2023-07-28T23:42:35.848 DEBUG:boto:Headers: {} 2023-07-28T23:42:35.848 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:42:35.848 DEBUG:boto:Port: 8001 2023-07-28T23:42:35.849 DEBUG:boto:Params: {} 2023-07-28T23:42:35.849 DEBUG:boto:Token: None 2023-07-28T23:42:35.849 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:42:35 GMT /hiiqos-3/ 2023-07-28T23:42:35.850 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LQXHrUCcXTlqN6VtKlXFw+71YB8= 2023-07-28T23:42:35.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LQXHrUCcXTlqN6VtKlXFw+71YB8=', 'Content-Length': '0'} 2023-07-28T23:42:35.854 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006ed390a8cc3005c6-0064c4526b-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:42:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:35.855 DEBUG:boto:path=/hiiqos-4/ 2023-07-28T23:42:35.855 DEBUG:boto:auth_path=/hiiqos-4/ 2023-07-28T23:42:35.856 DEBUG:boto:Method: PUT 2023-07-28T23:42:35.856 DEBUG:boto:Path: /hiiqos-4/ 2023-07-28T23:42:35.856 DEBUG:boto:Data: 2023-07-28T23:42:35.857 DEBUG:boto:Headers: {} 2023-07-28T23:42:35.857 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:42:35.857 DEBUG:boto:Port: 8001 2023-07-28T23:42:35.858 DEBUG:boto:Params: {} 2023-07-28T23:42:35.858 DEBUG:boto:Token: None 2023-07-28T23:42:35.858 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:42:35 GMT /hiiqos-4/ 2023-07-28T23:42:35.859 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9lZULXBX9EOGcE6l6Bjq2xjX4BY= 2023-07-28T23:42:35.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9lZULXBX9EOGcE6l6Bjq2xjX4BY=', 'Content-Length': '0'} 2023-07-28T23:42:35.904 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c1c45cebfe52f776-0064c4526b-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:42:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:35.905 DEBUG:boto:path=/hiiqos-3/ 2023-07-28T23:42:35.905 DEBUG:boto:auth_path=/hiiqos-3/ 2023-07-28T23:42:35.906 DEBUG:boto:Method: HEAD 2023-07-28T23:42:35.906 DEBUG:boto:Path: /hiiqos-3/ 2023-07-28T23:42:35.906 DEBUG:boto:Data: 2023-07-28T23:42:35.907 DEBUG:boto:Headers: {} 2023-07-28T23:42:35.907 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:42:35.907 DEBUG:boto:Port: 8000 2023-07-28T23:42:35.908 DEBUG:boto:Params: {} 2023-07-28T23:42:35.908 DEBUG:boto:Token: None 2023-07-28T23:42:35.908 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:42:35 GMT /hiiqos-3/ 2023-07-28T23:42:35.909 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KaD5j6o7LwOppqRLeNOiqdTPRpk= 2023-07-28T23:42:35.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KaD5j6o7LwOppqRLeNOiqdTPRpk=', 'Content-Length': '0'} 2023-07-28T23:42:35.912 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001990256da6d277ab-0064c4526b-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:42:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:35.912 DEBUG:boto:path=/hiiqos-4/ 2023-07-28T23:42:35.912 DEBUG:boto:auth_path=/hiiqos-4/ 2023-07-28T23:42:35.913 DEBUG:boto:Method: HEAD 2023-07-28T23:42:35.913 DEBUG:boto:Path: /hiiqos-4/ 2023-07-28T23:42:35.914 DEBUG:boto:Data: 2023-07-28T23:42:35.914 DEBUG:boto:Headers: {} 2023-07-28T23:42:35.915 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:42:35.915 DEBUG:boto:Port: 8000 2023-07-28T23:42:35.915 DEBUG:boto:Params: {} 2023-07-28T23:42:35.916 DEBUG:boto:Token: None 2023-07-28T23:42:35.916 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:42:35 GMT /hiiqos-4/ 2023-07-28T23:42:35.916 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cO93XmPsmP5lNwAqvKFOWtBnR2o= 2023-07-28T23:42:35.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cO93XmPsmP5lNwAqvKFOWtBnR2o=', 'Content-Length': '0'} 2023-07-28T23:42:35.919 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000142a52f07cad615a-0064c4526b-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:42:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:35.919 DEBUG:boto:path=/hiiqos-3/ 2023-07-28T23:42:35.919 DEBUG:boto:auth_path=/hiiqos-3/ 2023-07-28T23:42:35.920 DEBUG:boto:Method: HEAD 2023-07-28T23:42:35.920 DEBUG:boto:Path: /hiiqos-3/ 2023-07-28T23:42:35.920 DEBUG:boto:Data: 2023-07-28T23:42:35.921 DEBUG:boto:Headers: {} 2023-07-28T23:42:35.921 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:42:35.921 DEBUG:boto:Port: 8001 2023-07-28T23:42:35.922 DEBUG:boto:Params: {} 2023-07-28T23:42:35.922 DEBUG:boto:Token: None 2023-07-28T23:42:35.922 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:42:35 GMT /hiiqos-3/ 2023-07-28T23:42:35.923 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KaD5j6o7LwOppqRLeNOiqdTPRpk= 2023-07-28T23:42:35.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KaD5j6o7LwOppqRLeNOiqdTPRpk=', 'Content-Length': '0'} 2023-07-28T23:42:35.925 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002938b781f04d3f28-0064c4526b-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:42:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:35.926 DEBUG:boto:path=/hiiqos-4/ 2023-07-28T23:42:35.926 DEBUG:boto:auth_path=/hiiqos-4/ 2023-07-28T23:42:35.927 DEBUG:boto:Method: HEAD 2023-07-28T23:42:35.927 DEBUG:boto:Path: /hiiqos-4/ 2023-07-28T23:42:35.927 DEBUG:boto:Data: 2023-07-28T23:42:35.928 DEBUG:boto:Headers: {} 2023-07-28T23:42:35.928 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:42:35.928 DEBUG:boto:Port: 8001 2023-07-28T23:42:35.929 DEBUG:boto:Params: {} 2023-07-28T23:42:35.929 DEBUG:boto:Token: None 2023-07-28T23:42:35.929 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:42:35 GMT /hiiqos-4/ 2023-07-28T23:42:35.930 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cO93XmPsmP5lNwAqvKFOWtBnR2o= 2023-07-28T23:42:35.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cO93XmPsmP5lNwAqvKFOWtBnR2o=', 'Content-Length': '0'} 2023-07-28T23:42:35.932 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bef6bfad776c01b8-0064c4526b-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:42:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:35.933 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:35.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:35.934 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:42:35.979 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:42:35.979 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:42:36.007 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-28T23:42:36.018 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-28T23:42:36.018 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:36.019 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587755.595725_69.1", 2023-07-28T23:42:36.019 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:42:35.595725Z" 2023-07-28T23:42:36.019 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:36.019 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:36.019 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:36.019 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:42:36.020 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:36.020 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:36.020 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:36.020 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:42:36.020 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:36.020 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:36.020 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:36.021 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:42:36.021 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:36.021 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-28T23:42:36.022 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690587755.595725_69.1', 1: '', 2: '', 3: ''} 2023-07-28T23:42:36.022 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-28T23:42:36.023 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:36.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:36.024 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:42:36.069 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:42:36.070 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:42:36.096 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:42:36.097+0000 7f29bbd35880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:42:36.097 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:42:36.097+0000 7f29bbd35880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:42:36.108 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:42:36.108 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:42:36.108 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:42:36.108 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:42:36.109 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:42:36.109 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:42:36.109 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:36.109 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:36.109 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:42:36.109 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:36.110 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:42:36.110 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:36.110 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:36.110 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587755.595725_69.1", 2023-07-28T23:42:36.110 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:36.110 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:42:36.110 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:36.111 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:42:35.595725Z", 2023-07-28T23:42:36.111 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:36.111 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:36.111 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:36.111 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:36.111 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:42:36.111 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:36.112 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:36.112 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:36.112 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:36.113 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:42:36.113 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:36.114 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:42:36.114 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:36.114 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:36.114 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:36.115 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:36.115 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:42:36.115 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:36.115 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:36.115 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:36.115 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:36.116 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:42:36.116 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:36.116 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:42:36.116 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:36.116 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:36.116 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:36.117 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:36.117 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:42:36.117 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:36.117 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:36.117 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:36.117 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:36.118 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:42:36.118 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:36.118 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:42:36.118 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:36.118 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:36.118 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:36.119 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:42:36.119 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:36.119 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:42:36.119 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:42:36.119 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:42:36.119 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:36.119 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:42:36Z" 2023-07-28T23:42:36.120 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:42:36.121 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587755.595725_69.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:42:35.595725Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-28T23:42:36Z" } 2023-07-28T23:42:36.121 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587755.595725_69.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:42:35.595725Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-28T23:42:36.122 DEBUG:rgw_multi.tests:log_status={0: '1_1690587755.595725_69.1', 1: '', 2: '', 3: ''} 2023-07-28T23:42:36.122 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587755.595725_69.1', 1: '', 2: '', 3: ''} 2023-07-28T23:42:36.123 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-28T23:42:36.123 DEBUG:boto:path=/hiiqos-3/ 2023-07-28T23:42:36.123 DEBUG:boto:auth_path=/hiiqos-3/ 2023-07-28T23:42:36.124 DEBUG:boto:Method: HEAD 2023-07-28T23:42:36.124 DEBUG:boto:Path: /hiiqos-3/ 2023-07-28T23:42:36.125 DEBUG:boto:Data: 2023-07-28T23:42:36.126 DEBUG:boto:Headers: {} 2023-07-28T23:42:36.126 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:42:36.126 DEBUG:boto:Port: 8000 2023-07-28T23:42:36.127 DEBUG:boto:Params: {} 2023-07-28T23:42:36.127 DEBUG:boto:Token: None 2023-07-28T23:42:36.128 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:42:36 GMT /hiiqos-3/ 2023-07-28T23:42:36.128 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O8kmQMLR/DxnGpUAMY35+KfP2Xk= 2023-07-28T23:42:36.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O8kmQMLR/DxnGpUAMY35+KfP2Xk=', 'Content-Length': '0'} 2023-07-28T23:42:36.131 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005eecb312100fa2d0-0064c4526c-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:36.131 DEBUG:boto:path=/hiiqos-4/ 2023-07-28T23:42:36.132 DEBUG:boto:auth_path=/hiiqos-4/ 2023-07-28T23:42:36.132 DEBUG:boto:Method: HEAD 2023-07-28T23:42:36.132 DEBUG:boto:Path: /hiiqos-4/ 2023-07-28T23:42:36.133 DEBUG:boto:Data: 2023-07-28T23:42:36.133 DEBUG:boto:Headers: {} 2023-07-28T23:42:36.133 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:42:36.134 DEBUG:boto:Port: 8000 2023-07-28T23:42:36.134 DEBUG:boto:Params: {} 2023-07-28T23:42:36.134 DEBUG:boto:Token: None 2023-07-28T23:42:36.135 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:42:36 GMT /hiiqos-4/ 2023-07-28T23:42:36.135 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KzBZYE1aYacGhUwpkVmwVHxvh9Y= 2023-07-28T23:42:36.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KzBZYE1aYacGhUwpkVmwVHxvh9Y=', 'Content-Length': '0'} 2023-07-28T23:42:36.137 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bfa8ee3458bc8a75-0064c4526c-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:36.138 DEBUG:boto:path=/hiiqos-3/ 2023-07-28T23:42:36.138 DEBUG:boto:auth_path=/hiiqos-3/ 2023-07-28T23:42:36.139 DEBUG:boto:Method: HEAD 2023-07-28T23:42:36.139 DEBUG:boto:Path: /hiiqos-3/ 2023-07-28T23:42:36.139 DEBUG:boto:Data: 2023-07-28T23:42:36.140 DEBUG:boto:Headers: {} 2023-07-28T23:42:36.140 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:42:36.140 DEBUG:boto:Port: 8001 2023-07-28T23:42:36.141 DEBUG:boto:Params: {} 2023-07-28T23:42:36.141 DEBUG:boto:Token: None 2023-07-28T23:42:36.141 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:42:36 GMT /hiiqos-3/ 2023-07-28T23:42:36.142 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O8kmQMLR/DxnGpUAMY35+KfP2Xk= 2023-07-28T23:42:36.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O8kmQMLR/DxnGpUAMY35+KfP2Xk=', 'Content-Length': '0'} 2023-07-28T23:42:36.148 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c0f5609f5762a15f-0064c4526c-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:36.149 DEBUG:boto:path=/hiiqos-4/ 2023-07-28T23:42:36.149 DEBUG:boto:auth_path=/hiiqos-4/ 2023-07-28T23:42:36.150 DEBUG:boto:Method: HEAD 2023-07-28T23:42:36.150 DEBUG:boto:Path: /hiiqos-4/ 2023-07-28T23:42:36.150 DEBUG:boto:Data: 2023-07-28T23:42:36.152 DEBUG:boto:Headers: {} 2023-07-28T23:42:36.152 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:42:36.153 DEBUG:boto:Port: 8001 2023-07-28T23:42:36.153 DEBUG:boto:Params: {} 2023-07-28T23:42:36.153 DEBUG:boto:Token: None 2023-07-28T23:42:36.154 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:42:36 GMT /hiiqos-4/ 2023-07-28T23:42:36.154 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KzBZYE1aYacGhUwpkVmwVHxvh9Y= 2023-07-28T23:42:36.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KzBZYE1aYacGhUwpkVmwVHxvh9Y=', 'Content-Length': '0'} 2023-07-28T23:42:36.157 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e540fd3c8662e647-0064c4526c-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:36.157 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_recreate ... ok 2023-07-28T23:42:36.158 INFO:rgw_multi.tests:create bucket zone=a1 name=hiiqos-5 2023-07-28T23:42:36.159 DEBUG:boto:path=/hiiqos-5/ 2023-07-28T23:42:36.159 DEBUG:boto:auth_path=/hiiqos-5/ 2023-07-28T23:42:36.160 DEBUG:boto:Method: PUT 2023-07-28T23:42:36.160 DEBUG:boto:Path: /hiiqos-5/ 2023-07-28T23:42:36.160 DEBUG:boto:Data: 2023-07-28T23:42:36.161 DEBUG:boto:Headers: {} 2023-07-28T23:42:36.161 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:42:36.162 DEBUG:boto:Port: 8000 2023-07-28T23:42:36.162 DEBUG:boto:Params: {} 2023-07-28T23:42:36.163 DEBUG:boto:Token: None 2023-07-28T23:42:36.163 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:42:36 GMT /hiiqos-5/ 2023-07-28T23:42:36.164 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tgz3PF6cx1mjXHxUF24NmuOJH+A= 2023-07-28T23:42:36.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tgz3PF6cx1mjXHxUF24NmuOJH+A=', 'Content-Length': '0'} 2023-07-28T23:42:36.181 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f8334fac809e1e16-0064c4526c-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:36.182 INFO:rgw_multi.tests:create bucket zone=a2 name=hiiqos-6 2023-07-28T23:42:36.182 DEBUG:boto:path=/hiiqos-6/ 2023-07-28T23:42:36.183 DEBUG:boto:auth_path=/hiiqos-6/ 2023-07-28T23:42:36.183 DEBUG:boto:Method: PUT 2023-07-28T23:42:36.183 DEBUG:boto:Path: /hiiqos-6/ 2023-07-28T23:42:36.184 DEBUG:boto:Data: 2023-07-28T23:42:36.184 DEBUG:boto:Headers: {} 2023-07-28T23:42:36.184 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:42:36.185 DEBUG:boto:Port: 8001 2023-07-28T23:42:36.185 DEBUG:boto:Params: {} 2023-07-28T23:42:36.186 DEBUG:boto:Token: None 2023-07-28T23:42:36.186 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:42:36 GMT /hiiqos-6/ 2023-07-28T23:42:36.186 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:L23+fIXCoKfesWWM+cJ7PlSf8k4= 2023-07-28T23:42:36.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:L23+fIXCoKfesWWM+cJ7PlSf8k4=', 'Content-Length': '0'} 2023-07-28T23:42:36.214 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009b610c2c134ab549-0064c4526c-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:36.215 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:36.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:36.216 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:42:36.264 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:42:36.265 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:42:36.293 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-28T23:42:36.304 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-28T23:42:36.305 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:36.305 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587756.204014_77.1", 2023-07-28T23:42:36.305 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:42:36.204014Z" 2023-07-28T23:42:36.305 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:36.305 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:36.305 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:36.306 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:42:36.306 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:36.306 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:36.306 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:36.306 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:42:36.306 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:36.306 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:36.307 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:36.307 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:42:36.307 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:36.307 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-28T23:42:36.308 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690587756.204014_77.1', 1: '', 2: '', 3: ''} 2023-07-28T23:42:36.309 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-28T23:42:36.309 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:36.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:36.310 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:42:36.355 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:42:36.355 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:42:36.383 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:42:36.383+0000 7f203fb24880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:42:36.384 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:42:36.384+0000 7f203fb24880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:42:36.395 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:42:36.395 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:42:36.395 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:42:36.396 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:42:36.396 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:42:36.396 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:42:36.396 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:36.397 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:36.397 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:42:36.397 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:36.397 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:42:36.397 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:36.397 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:36.397 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587756.204014_77.1", 2023-07-28T23:42:36.398 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:36.398 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:42:36.398 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:36.398 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:42:36.204014Z", 2023-07-28T23:42:36.398 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:36.398 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:36.399 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:36.399 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:36.399 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:42:36.399 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:36.399 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:36.399 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:36.400 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:36.400 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:42:36.400 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:36.400 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:42:36.400 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:36.400 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:36.400 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:36.401 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:36.401 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:42:36.401 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:36.401 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:36.401 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:36.401 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:36.401 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:42:36.402 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:36.402 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:42:36.402 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:36.402 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:36.402 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:36.403 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:36.403 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:42:36.403 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:36.403 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:36.404 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:36.404 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:36.404 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:42:36.404 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:36.404 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:42:36.405 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:36.405 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:36.405 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:36.405 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:42:36.405 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:36.405 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:42:36.405 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:42:36.406 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:42:36.406 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:36.407 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:42:36Z" 2023-07-28T23:42:36.407 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:42:36.408 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587756.204014_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:42:36.204014Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-28T23:42:36Z" } 2023-07-28T23:42:36.408 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587756.204014_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:42:36.204014Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-28T23:42:36.409 DEBUG:rgw_multi.tests:log_status={0: '1_1690587756.204014_77.1', 1: '', 2: '', 3: ''} 2023-07-28T23:42:36.409 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587756.204014_77.1', 1: '', 2: '', 3: ''} 2023-07-28T23:42:36.409 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-28T23:42:36.410 DEBUG:boto:path=/hiiqos-5/ 2023-07-28T23:42:36.410 DEBUG:boto:auth_path=/hiiqos-5/ 2023-07-28T23:42:36.410 DEBUG:boto:Method: HEAD 2023-07-28T23:42:36.411 DEBUG:boto:Path: /hiiqos-5/ 2023-07-28T23:42:36.411 DEBUG:boto:Data: 2023-07-28T23:42:36.411 DEBUG:boto:Headers: {} 2023-07-28T23:42:36.412 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:42:36.412 DEBUG:boto:Port: 8000 2023-07-28T23:42:36.412 DEBUG:boto:Params: {} 2023-07-28T23:42:36.414 DEBUG:boto:Token: None 2023-07-28T23:42:36.414 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:42:36 GMT /hiiqos-5/ 2023-07-28T23:42:36.415 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7TWfeKbRYyblu7jfuYtTxp5lBR0= 2023-07-28T23:42:36.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7TWfeKbRYyblu7jfuYtTxp5lBR0=', 'Content-Length': '0'} 2023-07-28T23:42:36.417 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000befe77036c78444-0064c4526c-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:36.418 DEBUG:boto:path=/hiiqos-6/ 2023-07-28T23:42:36.418 DEBUG:boto:auth_path=/hiiqos-6/ 2023-07-28T23:42:36.419 DEBUG:boto:Method: HEAD 2023-07-28T23:42:36.419 DEBUG:boto:Path: /hiiqos-6/ 2023-07-28T23:42:36.419 DEBUG:boto:Data: 2023-07-28T23:42:36.420 DEBUG:boto:Headers: {} 2023-07-28T23:42:36.420 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:42:36.420 DEBUG:boto:Port: 8000 2023-07-28T23:42:36.421 DEBUG:boto:Params: {} 2023-07-28T23:42:36.421 DEBUG:boto:Token: None 2023-07-28T23:42:36.422 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:42:36 GMT /hiiqos-6/ 2023-07-28T23:42:36.422 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YaUiiJXt+QjKYo6KxYwF/APPZw8= 2023-07-28T23:42:36.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YaUiiJXt+QjKYo6KxYwF/APPZw8=', 'Content-Length': '0'} 2023-07-28T23:42:36.425 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000006974525501c341a-0064c4526c-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:36.425 DEBUG:boto:path=/hiiqos-5/ 2023-07-28T23:42:36.426 DEBUG:boto:auth_path=/hiiqos-5/ 2023-07-28T23:42:36.426 DEBUG:boto:Method: HEAD 2023-07-28T23:42:36.426 DEBUG:boto:Path: /hiiqos-5/ 2023-07-28T23:42:36.427 DEBUG:boto:Data: 2023-07-28T23:42:36.427 DEBUG:boto:Headers: {} 2023-07-28T23:42:36.427 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:42:36.428 DEBUG:boto:Port: 8001 2023-07-28T23:42:36.428 DEBUG:boto:Params: {} 2023-07-28T23:42:36.428 DEBUG:boto:Token: None 2023-07-28T23:42:36.429 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:42:36 GMT /hiiqos-5/ 2023-07-28T23:42:36.429 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7TWfeKbRYyblu7jfuYtTxp5lBR0= 2023-07-28T23:42:36.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7TWfeKbRYyblu7jfuYtTxp5lBR0=', 'Content-Length': '0'} 2023-07-28T23:42:36.432 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000088ec1ce3e70b5de5-0064c4526c-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:36.433 DEBUG:boto:path=/hiiqos-6/ 2023-07-28T23:42:36.433 DEBUG:boto:auth_path=/hiiqos-6/ 2023-07-28T23:42:36.433 DEBUG:boto:Method: HEAD 2023-07-28T23:42:36.434 DEBUG:boto:Path: /hiiqos-6/ 2023-07-28T23:42:36.434 DEBUG:boto:Data: 2023-07-28T23:42:36.434 DEBUG:boto:Headers: {} 2023-07-28T23:42:36.435 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:42:36.435 DEBUG:boto:Port: 8001 2023-07-28T23:42:36.436 DEBUG:boto:Params: {} 2023-07-28T23:42:36.436 DEBUG:boto:Token: None 2023-07-28T23:42:36.436 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:42:36 GMT /hiiqos-6/ 2023-07-28T23:42:36.437 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YaUiiJXt+QjKYo6KxYwF/APPZw8= 2023-07-28T23:42:36.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YaUiiJXt+QjKYo6KxYwF/APPZw8=', 'Content-Length': '0'} 2023-07-28T23:42:36.439 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ffc4063dbddf6068-0064c4526c-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:36.440 DEBUG:boto:path=/hiiqos-5/ 2023-07-28T23:42:36.440 DEBUG:boto:auth_path=/hiiqos-5/ 2023-07-28T23:42:36.441 DEBUG:boto:Method: DELETE 2023-07-28T23:42:36.441 DEBUG:boto:Path: /hiiqos-5/ 2023-07-28T23:42:36.441 DEBUG:boto:Data: 2023-07-28T23:42:36.442 DEBUG:boto:Headers: {} 2023-07-28T23:42:36.442 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:42:36.443 DEBUG:boto:Port: 8000 2023-07-28T23:42:36.443 DEBUG:boto:Params: {} 2023-07-28T23:42:36.443 DEBUG:boto:Token: None 2023-07-28T23:42:36.444 DEBUG:boto:StringToSign: DELETE Fri, 28 Jul 2023 23:42:36 GMT /hiiqos-5/ 2023-07-28T23:42:36.444 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:f7MA3oegMw1mShSHQW+KC/b1tRo= 2023-07-28T23:42:36.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:f7MA3oegMw1mShSHQW+KC/b1tRo=', 'Content-Length': '0'} 2023-07-28T23:42:36.470 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000aacd601e779020b4-0064c4526c-1117-a1'), ('Date', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:36.470 DEBUG:boto:path=/hiiqos-6/ 2023-07-28T23:42:36.471 DEBUG:boto:auth_path=/hiiqos-6/ 2023-07-28T23:42:36.471 DEBUG:boto:Method: DELETE 2023-07-28T23:42:36.471 DEBUG:boto:Path: /hiiqos-6/ 2023-07-28T23:42:36.472 DEBUG:boto:Data: 2023-07-28T23:42:36.472 DEBUG:boto:Headers: {} 2023-07-28T23:42:36.472 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:42:36.473 DEBUG:boto:Port: 8001 2023-07-28T23:42:36.473 DEBUG:boto:Params: {} 2023-07-28T23:42:36.474 DEBUG:boto:Token: None 2023-07-28T23:42:36.474 DEBUG:boto:StringToSign: DELETE Fri, 28 Jul 2023 23:42:36 GMT /hiiqos-6/ 2023-07-28T23:42:36.475 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:r/7RPcmIsPo+Da1+jzt0/kXGOAY= 2023-07-28T23:42:36.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:r/7RPcmIsPo+Da1+jzt0/kXGOAY=', 'Content-Length': '0'} 2023-07-28T23:42:36.523 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000062894fc93f86986a-0064c4526c-1113-a2'), ('Date', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:36.524 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:36.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:36.525 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:42:36.570 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:42:36.571 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:42:36.598 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-28T23:42:36.609 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-28T23:42:36.610 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:36.610 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587756.502719_85.1", 2023-07-28T23:42:36.610 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:42:36.502719Z" 2023-07-28T23:42:36.611 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:36.611 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:36.611 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:36.611 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:42:36.612 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:36.612 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:36.612 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:36.613 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:42:36.613 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:36.613 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:36.613 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:36.614 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:42:36.614 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:36.614 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-28T23:42:36.615 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690587756.502719_85.1', 1: '', 2: '', 3: ''} 2023-07-28T23:42:36.616 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-28T23:42:36.617 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:36.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:36.618 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:42:36.668 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:42:36.668 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:42:36.701 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:42:36.701+0000 7fec0409d880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:42:36.701 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:42:36.702+0000 7fec0409d880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:42:36.712 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:42:36.713 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:42:36.713 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:42:36.713 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:42:36.714 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:42:36.714 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:42:36.714 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:36.714 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:36.714 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:42:36.714 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:36.715 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:42:36.715 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:36.715 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:36.715 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587756.502719_85.1", 2023-07-28T23:42:36.715 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:36.716 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:42:36.716 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:36.716 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:42:36.502719Z", 2023-07-28T23:42:36.716 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:36.716 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:36.716 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:36.717 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:36.717 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:42:36.717 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:36.717 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:36.717 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:36.717 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:36.718 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:42:36.718 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:36.718 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:42:36.718 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:36.718 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:36.718 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:36.718 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:36.719 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:42:36.719 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:36.719 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:36.719 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:36.719 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:36.720 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:42:36.720 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:36.720 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:42:36.720 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:36.720 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:36.720 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:36.721 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:36.721 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:42:36.721 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:36.721 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:36.721 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:36.721 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:36.721 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:42:36.722 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:36.722 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:42:36.722 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:36.722 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:36.722 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:36.722 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:42:36.722 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:36.723 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:42:36.723 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:42:36.723 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:42:36.723 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:36.723 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:42:36Z" 2023-07-28T23:42:36.724 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:42:36.725 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587756.502719_85.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:42:36.502719Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-28T23:42:36Z" } 2023-07-28T23:42:36.725 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587756.502719_85.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:42:36.502719Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-28T23:42:36.726 DEBUG:rgw_multi.tests:log_status={0: '1_1690587756.502719_85.1', 1: '', 2: '', 3: ''} 2023-07-28T23:42:36.726 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587756.502719_85.1', 1: '', 2: '', 3: ''} 2023-07-28T23:42:36.726 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-28T23:42:36.727 DEBUG:boto:path=/hiiqos-5/ 2023-07-28T23:42:36.727 DEBUG:boto:auth_path=/hiiqos-5/ 2023-07-28T23:42:36.728 DEBUG:boto:Method: HEAD 2023-07-28T23:42:36.728 DEBUG:boto:Path: /hiiqos-5/ 2023-07-28T23:42:36.728 DEBUG:boto:Data: 2023-07-28T23:42:36.729 DEBUG:boto:Headers: {} 2023-07-28T23:42:36.729 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:42:36.730 DEBUG:boto:Port: 8000 2023-07-28T23:42:36.730 DEBUG:boto:Params: {} 2023-07-28T23:42:36.731 DEBUG:boto:Token: None 2023-07-28T23:42:36.731 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:42:36 GMT /hiiqos-5/ 2023-07-28T23:42:36.732 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7TWfeKbRYyblu7jfuYtTxp5lBR0= 2023-07-28T23:42:36.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7TWfeKbRYyblu7jfuYtTxp5lBR0=', 'Content-Length': '0'} 2023-07-28T23:42:36.734 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx00000073955a36d65b4d2-0064c4526c-1117-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:36.735 DEBUG:boto:path=/hiiqos-6/ 2023-07-28T23:42:36.736 DEBUG:boto:auth_path=/hiiqos-6/ 2023-07-28T23:42:36.736 DEBUG:boto:Method: HEAD 2023-07-28T23:42:36.736 DEBUG:boto:Path: /hiiqos-6/ 2023-07-28T23:42:36.737 DEBUG:boto:Data: 2023-07-28T23:42:36.737 DEBUG:boto:Headers: {} 2023-07-28T23:42:36.737 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:42:36.738 DEBUG:boto:Port: 8000 2023-07-28T23:42:36.738 DEBUG:boto:Params: {} 2023-07-28T23:42:36.739 DEBUG:boto:Token: None 2023-07-28T23:42:36.739 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:42:36 GMT /hiiqos-6/ 2023-07-28T23:42:36.740 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YaUiiJXt+QjKYo6KxYwF/APPZw8= 2023-07-28T23:42:36.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YaUiiJXt+QjKYo6KxYwF/APPZw8=', 'Content-Length': '0'} 2023-07-28T23:42:36.741 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx00000af9a7bbfcffdc6d0-0064c4526c-1117-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:36.742 DEBUG:boto:path=/hiiqos-5/ 2023-07-28T23:42:36.742 DEBUG:boto:auth_path=/hiiqos-5/ 2023-07-28T23:42:36.742 DEBUG:boto:Method: HEAD 2023-07-28T23:42:36.743 DEBUG:boto:Path: /hiiqos-5/ 2023-07-28T23:42:36.743 DEBUG:boto:Data: 2023-07-28T23:42:36.744 DEBUG:boto:Headers: {} 2023-07-28T23:42:36.744 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:42:36.744 DEBUG:boto:Port: 8001 2023-07-28T23:42:36.745 DEBUG:boto:Params: {} 2023-07-28T23:42:36.745 DEBUG:boto:Token: None 2023-07-28T23:42:36.745 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:42:36 GMT /hiiqos-5/ 2023-07-28T23:42:36.746 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7TWfeKbRYyblu7jfuYtTxp5lBR0= 2023-07-28T23:42:36.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7TWfeKbRYyblu7jfuYtTxp5lBR0=', 'Content-Length': '0'} 2023-07-28T23:42:36.749 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx000001f2457b019610c1d-0064c4526c-1113-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:36.749 DEBUG:boto:path=/hiiqos-6/ 2023-07-28T23:42:36.749 DEBUG:boto:auth_path=/hiiqos-6/ 2023-07-28T23:42:36.750 DEBUG:boto:Method: HEAD 2023-07-28T23:42:36.751 DEBUG:boto:Path: /hiiqos-6/ 2023-07-28T23:42:36.751 DEBUG:boto:Data: 2023-07-28T23:42:36.751 DEBUG:boto:Headers: {} 2023-07-28T23:42:36.752 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:42:36.752 DEBUG:boto:Port: 8001 2023-07-28T23:42:36.752 DEBUG:boto:Params: {} 2023-07-28T23:42:36.753 DEBUG:boto:Token: None 2023-07-28T23:42:36.753 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:42:36 GMT /hiiqos-6/ 2023-07-28T23:42:36.754 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YaUiiJXt+QjKYo6KxYwF/APPZw8= 2023-07-28T23:42:36.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YaUiiJXt+QjKYo6KxYwF/APPZw8=', 'Content-Length': '0'} 2023-07-28T23:42:36.755 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx0000042dbf43fe1926407-0064c4526c-1113-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:36.756 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_remove ... ok 2023-07-28T23:42:36.756 INFO:rgw_multi.tests:create bucket zone=a1 name=hiiqos-7 2023-07-28T23:42:36.757 DEBUG:boto:path=/hiiqos-7/ 2023-07-28T23:42:36.757 DEBUG:boto:auth_path=/hiiqos-7/ 2023-07-28T23:42:36.757 DEBUG:boto:Method: PUT 2023-07-28T23:42:36.758 DEBUG:boto:Path: /hiiqos-7/ 2023-07-28T23:42:36.758 DEBUG:boto:Data: 2023-07-28T23:42:36.758 DEBUG:boto:Headers: {} 2023-07-28T23:42:36.759 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:42:36.759 DEBUG:boto:Port: 8000 2023-07-28T23:42:36.760 DEBUG:boto:Params: {} 2023-07-28T23:42:36.760 DEBUG:boto:Token: None 2023-07-28T23:42:36.760 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:42:36 GMT /hiiqos-7/ 2023-07-28T23:42:36.761 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:j1NhqhSrrJImukmr2yNtCNN64Rc= 2023-07-28T23:42:36.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:j1NhqhSrrJImukmr2yNtCNN64Rc=', 'Content-Length': '0'} 2023-07-28T23:42:36.773 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000058ad3ed1493abe9d-0064c4526c-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:36.773 INFO:rgw_multi.tests:create bucket zone=a2 name=hiiqos-8 2023-07-28T23:42:36.774 DEBUG:boto:path=/hiiqos-8/ 2023-07-28T23:42:36.774 DEBUG:boto:auth_path=/hiiqos-8/ 2023-07-28T23:42:36.775 DEBUG:boto:Method: PUT 2023-07-28T23:42:36.775 DEBUG:boto:Path: /hiiqos-8/ 2023-07-28T23:42:36.775 DEBUG:boto:Data: 2023-07-28T23:42:36.776 DEBUG:boto:Headers: {} 2023-07-28T23:42:36.776 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:42:36.776 DEBUG:boto:Port: 8001 2023-07-28T23:42:36.777 DEBUG:boto:Params: {} 2023-07-28T23:42:36.777 DEBUG:boto:Token: None 2023-07-28T23:42:36.777 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:42:36 GMT /hiiqos-8/ 2023-07-28T23:42:36.778 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AJruTHUwhqf5OzuiQ69lNSyfm/s= 2023-07-28T23:42:36.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AJruTHUwhqf5OzuiQ69lNSyfm/s=', 'Content-Length': '0'} 2023-07-28T23:42:36.797 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e4363810b660627d-0064c4526c-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:36.798 DEBUG:boto:path=/hiiqos-7/ 2023-07-28T23:42:36.799 DEBUG:boto:auth_path=/hiiqos-7/ 2023-07-28T23:42:36.799 DEBUG:boto:Method: HEAD 2023-07-28T23:42:36.799 DEBUG:boto:Path: /hiiqos-7/ 2023-07-28T23:42:36.800 DEBUG:boto:Data: 2023-07-28T23:42:36.800 DEBUG:boto:Headers: {} 2023-07-28T23:42:36.800 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:42:36.801 DEBUG:boto:Port: 8000 2023-07-28T23:42:36.801 DEBUG:boto:Params: {} 2023-07-28T23:42:36.802 DEBUG:boto:Token: None 2023-07-28T23:42:36.802 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:42:36 GMT /hiiqos-7/ 2023-07-28T23:42:36.803 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kk03H9P4QuvJtkt1CkhMl0BxYV0= 2023-07-28T23:42:36.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kk03H9P4QuvJtkt1CkhMl0BxYV0=', 'Content-Length': '0'} 2023-07-28T23:42:36.805 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000378cd2871e70ac4c-0064c4526c-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:36.806 DEBUG:boto:path=/hiiqos-7/myobj 2023-07-28T23:42:36.808 DEBUG:boto:auth_path=/hiiqos-7/myobj 2023-07-28T23:42:36.808 DEBUG:boto:Method: PUT 2023-07-28T23:42:36.809 DEBUG:boto:Path: /hiiqos-7/myobj 2023-07-28T23:42:36.810 DEBUG:boto:Data: 2023-07-28T23:42:36.811 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-28T23:42:36.812 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:42:36.813 DEBUG:boto:Port: 8000 2023-07-28T23:42:36.813 DEBUG:boto:Params: {} 2023-07-28T23:42:36.814 DEBUG:boto:Token: None 2023-07-28T23:42:36.815 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 28 Jul 2023 23:42:36 GMT /hiiqos-7/myobj 2023-07-28T23:42:36.816 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:p/oxfJ+hzuXq1bzV7u1g6CosEqU= 2023-07-28T23:42:36.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:42:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:p/oxfJ+hzuXq1bzV7u1g6CosEqU='} 2023-07-28T23:42:36.860 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003b3e1c44686bf95d-0064c4526c-1117-a1'), ('Date', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:36.861 DEBUG:boto:path=/hiiqos-7/ 2023-07-28T23:42:36.861 DEBUG:boto:auth_path=/hiiqos-7/ 2023-07-28T23:42:36.862 DEBUG:boto:Method: HEAD 2023-07-28T23:42:36.862 DEBUG:boto:Path: /hiiqos-7/ 2023-07-28T23:42:36.862 DEBUG:boto:Data: 2023-07-28T23:42:36.863 DEBUG:boto:Headers: {} 2023-07-28T23:42:36.863 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:42:36.863 DEBUG:boto:Port: 8000 2023-07-28T23:42:36.864 DEBUG:boto:Params: {} 2023-07-28T23:42:36.864 DEBUG:boto:Token: None 2023-07-28T23:42:36.865 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:42:36 GMT /hiiqos-7/ 2023-07-28T23:42:36.865 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kk03H9P4QuvJtkt1CkhMl0BxYV0= 2023-07-28T23:42:36.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kk03H9P4QuvJtkt1CkhMl0BxYV0=', 'Content-Length': '0'} 2023-07-28T23:42:36.867 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000040fd25945369e8a-0064c4526c-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:36.868 DEBUG:boto:path=/hiiqos-7/_myobj 2023-07-28T23:42:36.869 DEBUG:boto:auth_path=/hiiqos-7/_myobj 2023-07-28T23:42:36.869 DEBUG:boto:Method: PUT 2023-07-28T23:42:36.869 DEBUG:boto:Path: /hiiqos-7/_myobj 2023-07-28T23:42:36.870 DEBUG:boto:Data: 2023-07-28T23:42:36.870 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-28T23:42:36.870 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:42:36.871 DEBUG:boto:Port: 8000 2023-07-28T23:42:36.871 DEBUG:boto:Params: {} 2023-07-28T23:42:36.871 DEBUG:boto:Token: None 2023-07-28T23:42:36.872 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 28 Jul 2023 23:42:36 GMT /hiiqos-7/_myobj 2023-07-28T23:42:36.872 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:W3L7eoRdoI6zYlKGkDIdrak9KOA= 2023-07-28T23:42:36.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:42:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:W3L7eoRdoI6zYlKGkDIdrak9KOA='} 2023-07-28T23:42:36.881 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000027c64ef6f9bbe47e-0064c4526c-1117-a1'), ('Date', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:36.882 DEBUG:boto:path=/hiiqos-7/ 2023-07-28T23:42:36.882 DEBUG:boto:auth_path=/hiiqos-7/ 2023-07-28T23:42:36.882 DEBUG:boto:Method: HEAD 2023-07-28T23:42:36.883 DEBUG:boto:Path: /hiiqos-7/ 2023-07-28T23:42:36.883 DEBUG:boto:Data: 2023-07-28T23:42:36.883 DEBUG:boto:Headers: {} 2023-07-28T23:42:36.884 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:42:36.884 DEBUG:boto:Port: 8000 2023-07-28T23:42:36.884 DEBUG:boto:Params: {} 2023-07-28T23:42:36.885 DEBUG:boto:Token: None 2023-07-28T23:42:36.885 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:42:36 GMT /hiiqos-7/ 2023-07-28T23:42:36.885 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kk03H9P4QuvJtkt1CkhMl0BxYV0= 2023-07-28T23:42:36.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kk03H9P4QuvJtkt1CkhMl0BxYV0=', 'Content-Length': '0'} 2023-07-28T23:42:36.888 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b58eb6d102d092d5-0064c4526c-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:36.889 DEBUG:boto:path=/hiiqos-7/%3A 2023-07-28T23:42:36.889 DEBUG:boto:auth_path=/hiiqos-7/%3A 2023-07-28T23:42:36.889 DEBUG:boto:Method: PUT 2023-07-28T23:42:36.890 DEBUG:boto:Path: /hiiqos-7/%3A 2023-07-28T23:42:36.890 DEBUG:boto:Data: 2023-07-28T23:42:36.890 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-28T23:42:36.891 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:42:36.891 DEBUG:boto:Port: 8000 2023-07-28T23:42:36.891 DEBUG:boto:Params: {} 2023-07-28T23:42:36.892 DEBUG:boto:Token: None 2023-07-28T23:42:36.892 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 28 Jul 2023 23:42:36 GMT /hiiqos-7/%3A 2023-07-28T23:42:36.892 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xwMfDWd6fAnPjG0PfgH/bGtf9hM= 2023-07-28T23:42:36.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:42:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xwMfDWd6fAnPjG0PfgH/bGtf9hM='} 2023-07-28T23:42:36.906 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001e2bb1179440d989-0064c4526c-1117-a1'), ('Date', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:36.907 DEBUG:boto:path=/hiiqos-7/ 2023-07-28T23:42:36.908 DEBUG:boto:auth_path=/hiiqos-7/ 2023-07-28T23:42:36.908 DEBUG:boto:Method: HEAD 2023-07-28T23:42:36.908 DEBUG:boto:Path: /hiiqos-7/ 2023-07-28T23:42:36.909 DEBUG:boto:Data: 2023-07-28T23:42:36.909 DEBUG:boto:Headers: {} 2023-07-28T23:42:36.909 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:42:36.909 DEBUG:boto:Port: 8000 2023-07-28T23:42:36.910 DEBUG:boto:Params: {} 2023-07-28T23:42:36.910 DEBUG:boto:Token: None 2023-07-28T23:42:36.911 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:42:36 GMT /hiiqos-7/ 2023-07-28T23:42:36.911 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kk03H9P4QuvJtkt1CkhMl0BxYV0= 2023-07-28T23:42:36.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kk03H9P4QuvJtkt1CkhMl0BxYV0=', 'Content-Length': '0'} 2023-07-28T23:42:36.913 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000099be3fa0cb9030c8-0064c4526c-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:36.914 DEBUG:boto:path=/hiiqos-7/%26 2023-07-28T23:42:36.915 DEBUG:boto:auth_path=/hiiqos-7/%26 2023-07-28T23:42:36.915 DEBUG:boto:Method: PUT 2023-07-28T23:42:36.915 DEBUG:boto:Path: /hiiqos-7/%26 2023-07-28T23:42:36.916 DEBUG:boto:Data: 2023-07-28T23:42:36.916 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-28T23:42:36.916 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:42:36.917 DEBUG:boto:Port: 8000 2023-07-28T23:42:36.917 DEBUG:boto:Params: {} 2023-07-28T23:42:36.917 DEBUG:boto:Token: None 2023-07-28T23:42:36.918 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 28 Jul 2023 23:42:36 GMT /hiiqos-7/%26 2023-07-28T23:42:36.918 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZO5SqZ2vhH2tT30sC2iAdSCHdoI= 2023-07-28T23:42:36.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:42:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZO5SqZ2vhH2tT30sC2iAdSCHdoI='} 2023-07-28T23:42:36.964 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000435723108e22dff0-0064c4526c-1117-a1'), ('Date', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:36.965 DEBUG:boto:path=/hiiqos-8/ 2023-07-28T23:42:36.965 DEBUG:boto:auth_path=/hiiqos-8/ 2023-07-28T23:42:36.966 DEBUG:boto:Method: HEAD 2023-07-28T23:42:36.966 DEBUG:boto:Path: /hiiqos-8/ 2023-07-28T23:42:36.966 DEBUG:boto:Data: 2023-07-28T23:42:36.967 DEBUG:boto:Headers: {} 2023-07-28T23:42:36.967 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:42:36.967 DEBUG:boto:Port: 8001 2023-07-28T23:42:36.968 DEBUG:boto:Params: {} 2023-07-28T23:42:36.968 DEBUG:boto:Token: None 2023-07-28T23:42:36.968 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:42:36 GMT /hiiqos-8/ 2023-07-28T23:42:36.969 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3VQg0wypX9utTJnaddJ8uGANJ4g= 2023-07-28T23:42:36.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3VQg0wypX9utTJnaddJ8uGANJ4g=', 'Content-Length': '0'} 2023-07-28T23:42:36.971 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a96e002ec3e693eb-0064c4526c-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:36.972 DEBUG:boto:path=/hiiqos-8/myobj 2023-07-28T23:42:36.972 DEBUG:boto:auth_path=/hiiqos-8/myobj 2023-07-28T23:42:36.973 DEBUG:boto:Method: PUT 2023-07-28T23:42:36.973 DEBUG:boto:Path: /hiiqos-8/myobj 2023-07-28T23:42:36.973 DEBUG:boto:Data: 2023-07-28T23:42:36.974 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-28T23:42:36.974 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:42:36.974 DEBUG:boto:Port: 8001 2023-07-28T23:42:36.975 DEBUG:boto:Params: {} 2023-07-28T23:42:36.975 DEBUG:boto:Token: None 2023-07-28T23:42:36.976 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 28 Jul 2023 23:42:36 GMT /hiiqos-8/myobj 2023-07-28T23:42:36.976 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RElbypBXdbpjQppMPpn/GmBrYSM= 2023-07-28T23:42:36.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:42:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RElbypBXdbpjQppMPpn/GmBrYSM='} 2023-07-28T23:42:36.987 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cefe9e2e208ea81b-0064c4526c-1113-a2'), ('Date', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:36.988 DEBUG:boto:path=/hiiqos-8/ 2023-07-28T23:42:36.989 DEBUG:boto:auth_path=/hiiqos-8/ 2023-07-28T23:42:36.989 DEBUG:boto:Method: HEAD 2023-07-28T23:42:36.989 DEBUG:boto:Path: /hiiqos-8/ 2023-07-28T23:42:36.990 DEBUG:boto:Data: 2023-07-28T23:42:36.990 DEBUG:boto:Headers: {} 2023-07-28T23:42:36.991 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:42:36.991 DEBUG:boto:Port: 8001 2023-07-28T23:42:36.992 DEBUG:boto:Params: {} 2023-07-28T23:42:36.992 DEBUG:boto:Token: None 2023-07-28T23:42:36.992 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:42:36 GMT /hiiqos-8/ 2023-07-28T23:42:36.993 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3VQg0wypX9utTJnaddJ8uGANJ4g= 2023-07-28T23:42:36.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3VQg0wypX9utTJnaddJ8uGANJ4g=', 'Content-Length': '0'} 2023-07-28T23:42:36.995 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ccf51fde2e2d32e4-0064c4526c-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:36.996 DEBUG:boto:path=/hiiqos-8/_myobj 2023-07-28T23:42:36.996 DEBUG:boto:auth_path=/hiiqos-8/_myobj 2023-07-28T23:42:36.996 DEBUG:boto:Method: PUT 2023-07-28T23:42:36.997 DEBUG:boto:Path: /hiiqos-8/_myobj 2023-07-28T23:42:36.997 DEBUG:boto:Data: 2023-07-28T23:42:36.997 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-28T23:42:36.998 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:42:36.998 DEBUG:boto:Port: 8001 2023-07-28T23:42:36.998 DEBUG:boto:Params: {} 2023-07-28T23:42:36.999 DEBUG:boto:Token: None 2023-07-28T23:42:36.999 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 28 Jul 2023 23:42:36 GMT /hiiqos-8/_myobj 2023-07-28T23:42:36.999 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IaZz9KMJlPWsguDFpP7xVs7+988= 2023-07-28T23:42:37.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:42:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IaZz9KMJlPWsguDFpP7xVs7+988='} 2023-07-28T23:42:37.009 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cdbcb3a2087b3659-0064c4526d-1113-a2'), ('Date', 'Fri, 28 Jul 2023 23:42:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:37.010 DEBUG:boto:path=/hiiqos-8/ 2023-07-28T23:42:37.011 DEBUG:boto:auth_path=/hiiqos-8/ 2023-07-28T23:42:37.012 DEBUG:boto:Method: HEAD 2023-07-28T23:42:37.012 DEBUG:boto:Path: /hiiqos-8/ 2023-07-28T23:42:37.012 DEBUG:boto:Data: 2023-07-28T23:42:37.013 DEBUG:boto:Headers: {} 2023-07-28T23:42:37.013 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:42:37.013 DEBUG:boto:Port: 8001 2023-07-28T23:42:37.014 DEBUG:boto:Params: {} 2023-07-28T23:42:37.014 DEBUG:boto:Token: None 2023-07-28T23:42:37.014 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:42:37 GMT /hiiqos-8/ 2023-07-28T23:42:37.015 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:t1xPeI1HT1axq2tb9Vi9w24v6tQ= 2023-07-28T23:42:37.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t1xPeI1HT1axq2tb9Vi9w24v6tQ=', 'Content-Length': '0'} 2023-07-28T23:42:37.017 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f8461cf8251b5a22-0064c4526d-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:42:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:37.018 DEBUG:boto:path=/hiiqos-8/%3A 2023-07-28T23:42:37.019 DEBUG:boto:auth_path=/hiiqos-8/%3A 2023-07-28T23:42:37.019 DEBUG:boto:Method: PUT 2023-07-28T23:42:37.019 DEBUG:boto:Path: /hiiqos-8/%3A 2023-07-28T23:42:37.020 DEBUG:boto:Data: 2023-07-28T23:42:37.020 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-28T23:42:37.021 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:42:37.021 DEBUG:boto:Port: 8001 2023-07-28T23:42:37.021 DEBUG:boto:Params: {} 2023-07-28T23:42:37.022 DEBUG:boto:Token: None 2023-07-28T23:42:37.022 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 28 Jul 2023 23:42:37 GMT /hiiqos-8/%3A 2023-07-28T23:42:37.022 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LBo9x87eTfL2Ol2POabDAWk9mII= 2023-07-28T23:42:37.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:42:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LBo9x87eTfL2Ol2POabDAWk9mII='} 2023-07-28T23:42:37.064 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f3d5b4c4d74adc31-0064c4526d-1113-a2'), ('Date', 'Fri, 28 Jul 2023 23:42:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:37.065 DEBUG:boto:path=/hiiqos-8/ 2023-07-28T23:42:37.065 DEBUG:boto:auth_path=/hiiqos-8/ 2023-07-28T23:42:37.066 DEBUG:boto:Method: HEAD 2023-07-28T23:42:37.066 DEBUG:boto:Path: /hiiqos-8/ 2023-07-28T23:42:37.066 DEBUG:boto:Data: 2023-07-28T23:42:37.067 DEBUG:boto:Headers: {} 2023-07-28T23:42:37.067 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:42:37.067 DEBUG:boto:Port: 8001 2023-07-28T23:42:37.068 DEBUG:boto:Params: {} 2023-07-28T23:42:37.068 DEBUG:boto:Token: None 2023-07-28T23:42:37.069 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:42:37 GMT /hiiqos-8/ 2023-07-28T23:42:37.069 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:t1xPeI1HT1axq2tb9Vi9w24v6tQ= 2023-07-28T23:42:37.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:42:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t1xPeI1HT1axq2tb9Vi9w24v6tQ=', 'Content-Length': '0'} 2023-07-28T23:42:37.071 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef264a2f390b4606-0064c4526d-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:42:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:37.072 DEBUG:boto:path=/hiiqos-8/%26 2023-07-28T23:42:37.072 DEBUG:boto:auth_path=/hiiqos-8/%26 2023-07-28T23:42:37.073 DEBUG:boto:Method: PUT 2023-07-28T23:42:37.073 DEBUG:boto:Path: /hiiqos-8/%26 2023-07-28T23:42:37.073 DEBUG:boto:Data: 2023-07-28T23:42:37.074 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-28T23:42:37.074 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:42:37.074 DEBUG:boto:Port: 8001 2023-07-28T23:42:37.075 DEBUG:boto:Params: {} 2023-07-28T23:42:37.075 DEBUG:boto:Token: None 2023-07-28T23:42:37.075 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 28 Jul 2023 23:42:37 GMT /hiiqos-8/%26 2023-07-28T23:42:37.076 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w8BRqtv4ayzezRpGOaTIeln6QM8= 2023-07-28T23:42:37.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:42:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w8BRqtv4ayzezRpGOaTIeln6QM8='} 2023-07-28T23:42:37.081 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a6235f2a9feef963-0064c4526d-1113-a2'), ('Date', 'Fri, 28 Jul 2023 23:42:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:42:37.081 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:37.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:37.082 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:42:37.126 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:42:37.127 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:42:37.156 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-28T23:42:37.171 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-28T23:42:37.171 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:37.171 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587756.791052_93.1", 2023-07-28T23:42:37.171 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:42:36.791052Z" 2023-07-28T23:42:37.171 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:37.172 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:37.172 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:37.172 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:42:37.172 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:37.172 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:37.172 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:37.173 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:42:37.173 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:37.173 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:37.173 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:37.173 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:42:37.173 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:37.173 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-28T23:42:37.174 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690587756.791052_93.1', 1: '', 2: '', 3: ''} 2023-07-28T23:42:37.175 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-28T23:42:37.175 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:37.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:42:37.176 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:42:37.223 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:42:37.224 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:42:37.252 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:42:37.253+0000 7f193189d880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:42:37.253 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:42:37.254+0000 7f193189d880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:42:37.264 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:42:37.264 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:42:37.265 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:42:37.265 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:42:37.265 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:42:37.265 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:42:37.265 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:37.265 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:37.266 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:42:37.266 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:37.266 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:42:37.266 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:37.266 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:37.266 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587756.791052_93.1", 2023-07-28T23:42:37.266 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:37.267 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:42:37.267 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:37.267 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:42:36.791052Z", 2023-07-28T23:42:37.267 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:37.267 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:37.267 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:37.267 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:37.268 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:42:37.268 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:37.268 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:37.268 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:37.268 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:37.268 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:42:37.269 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:37.269 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:42:37.269 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:37.269 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:37.269 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:37.269 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:37.270 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:42:37.270 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:37.270 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:37.270 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:37.270 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:37.270 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:42:37.270 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:37.271 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:42:37.271 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:37.271 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:37.271 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:37.271 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:42:37.271 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:42:37.271 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:42:37.272 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:42:37.272 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:42:37.272 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:42:37.272 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:42:37.272 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:42:37.272 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:42:37.272 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:42:37.273 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:37.273 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:42:37.273 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:42:37.273 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:37.273 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:42:37.273 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:42:37.273 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:42:37.274 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:42:37.274 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:42:37Z" 2023-07-28T23:42:37.274 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:42:37.275 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587756.791052_93.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:42:36.791052Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-28T23:42:37Z" } 2023-07-28T23:42:37.275 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587756.791052_93.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:42:36.791052Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-28T23:42:37.276 DEBUG:rgw_multi.tests:log_status={0: '1_1690587756.791052_93.1', 1: '', 2: '', 3: ''} 2023-07-28T23:42:37.276 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587756.791052_93.1', 1: '', 2: '', 3: ''} 2023-07-28T23:42:37.277 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-28T23:42:37.277 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-7', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:42:37.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-7', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:42:37.278 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-7 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-28T23:42:37.330 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:42:37.331 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:42:37.362 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:42:37.363+0000 7fe990267880 1 waiting to reach incremental sync.. 2023-07-28T23:42:42.363 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:42:42.364+0000 7fe990267880 1 waiting to reach incremental sync.. 2023-07-28T23:42:47.364 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:42:47.364+0000 7fe990267880 1 waiting to reach incremental sync.. 2023-07-28T23:42:52.364 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:42:52.365+0000 7fe990267880 1 waiting to reach incremental sync.. 2023-07-28T23:42:57.365 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:42:57.365+0000 7fe990267880 1 waiting to reach incremental sync.. 2023-07-28T23:43:02.366 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:43:02.366+0000 7fe990267880 1 waiting to reach incremental sync.. 2023-07-28T23:43:07.366 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:43:07.366+0000 7fe990267880 1 waiting to reach incremental sync.. 2023-07-28T23:43:12.367 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:43:12.367+0000 7fe990267880 1 waiting to reach incremental sync.. 2023-07-28T23:43:17.367 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:43:17.368+0000 7fe990267880 1 waiting to reach incremental sync.. 2023-07-28T23:43:22.368 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:43:22.368+0000 7fe990267880 1 waiting to reach incremental sync.. 2023-07-28T23:43:27.369 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:43:27.369+0000 7fe990267880 1 waiting to reach incremental sync.. 2023-07-28T23:43:32.369 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:43:32.370+0000 7fe990267880 1 waiting to reach incremental sync.. 2023-07-28T23:43:37.371 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:43:37.371+0000 7fe990267880 1 bucket sync caught up with source: 2023-07-28T23:43:37.371 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000001.8.6, , 00000000001.5.6, , , , 00000000001.5.6, 00000000001.4.6, , ] 2023-07-28T23:43:37.371 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000001.8.6, , 00000000001.5.6, , , , 00000000001.5.6, 00000000001.4.6, , ] 2023-07-28T23:43:37.372 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:43:37.371+0000 7fe990267880 0 bucket checkpoint complete 2023-07-28T23:43:37.387 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-7 zones={a2, a1} 2023-07-28T23:43:37.387 DEBUG:boto:path=/hiiqos-7/ 2023-07-28T23:43:37.388 DEBUG:boto:auth_path=/hiiqos-7/ 2023-07-28T23:43:37.388 DEBUG:boto:Method: HEAD 2023-07-28T23:43:37.388 DEBUG:boto:Path: /hiiqos-7/ 2023-07-28T23:43:37.389 DEBUG:boto:Data: 2023-07-28T23:43:37.389 DEBUG:boto:Headers: {} 2023-07-28T23:43:37.389 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:43:37.390 DEBUG:boto:Port: 8001 2023-07-28T23:43:37.390 DEBUG:boto:Params: {} 2023-07-28T23:43:37.391 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-28T23:43:37.391 DEBUG:boto:Token: None 2023-07-28T23:43:37.392 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:43:37 GMT /hiiqos-7/ 2023-07-28T23:43:37.392 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IiSnemqPMBRSCTCvktSdpPkKtXc= 2023-07-28T23:43:37.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IiSnemqPMBRSCTCvktSdpPkKtXc=', 'Content-Length': '0'} 2023-07-28T23:43:42.397 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000919f774a3225ba4d-0064c452ae-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:43:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:42.398 DEBUG:boto:path=/hiiqos-7/ 2023-07-28T23:43:42.399 DEBUG:boto:auth_path=/hiiqos-7/ 2023-07-28T23:43:42.399 DEBUG:boto:Method: HEAD 2023-07-28T23:43:42.400 DEBUG:boto:Path: /hiiqos-7/ 2023-07-28T23:43:42.400 DEBUG:boto:Data: 2023-07-28T23:43:42.401 DEBUG:boto:Headers: {} 2023-07-28T23:43:42.401 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:43:42.402 DEBUG:boto:Port: 8000 2023-07-28T23:43:42.402 DEBUG:boto:Params: {} 2023-07-28T23:43:42.403 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-28T23:43:42.404 DEBUG:boto:Token: None 2023-07-28T23:43:42.404 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:43:42 GMT /hiiqos-7/ 2023-07-28T23:43:42.405 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Zgzh+/eiw2ZERy24kuZaBcrZxLc= 2023-07-28T23:43:42.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Zgzh+/eiw2ZERy24kuZaBcrZxLc=', 'Content-Length': '0'} 2023-07-28T23:43:47.410 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000050d7ebd95c1bd2e4-0064c452b3-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.411 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-28T23:43:47.412 DEBUG:boto:path=/hiiqos-7/ 2023-07-28T23:43:47.413 DEBUG:boto:auth_path=/hiiqos-7/ 2023-07-28T23:43:47.413 DEBUG:boto:path=/hiiqos-7/?versions&max-keys=999 2023-07-28T23:43:47.414 DEBUG:boto:auth_path=/hiiqos-7/?versions&max-keys=999 2023-07-28T23:43:47.414 DEBUG:boto:Method: GET 2023-07-28T23:43:47.415 DEBUG:boto:Path: /hiiqos-7/?versions&max-keys=999 2023-07-28T23:43:47.415 DEBUG:boto:Data: 2023-07-28T23:43:47.416 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.416 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:43:47.417 DEBUG:boto:Port: 8001 2023-07-28T23:43:47.417 DEBUG:boto:Params: {} 2023-07-28T23:43:47.417 DEBUG:boto:Token: None 2023-07-28T23:43:47.418 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/?versions 2023-07-28T23:43:47.418 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rjfsIHCXpgn5vMTvmdVM/CNogEc= 2023-07-28T23:43:47.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rjfsIHCXpgn5vMTvmdVM/CNogEc=', 'Content-Length': '0'} 2023-07-28T23:43:47.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000436301442445755f-0064c452b3-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.422 DEBUG:boto:b'hiiqos-7999false&nulltrue2023-07-28T23:42:36.922Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-07-28T23:42:36.896Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-07-28T23:42:36.876Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-07-28T23:42:36.820Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:43:47.423 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-07-28T23:43:47.424 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-07-28T23:43:47.424 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-07-28T23:43:47.425 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-07-28T23:43:47.425 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-28T23:43:47.425 DEBUG:boto:path=/hiiqos-7/ 2023-07-28T23:43:47.426 DEBUG:boto:auth_path=/hiiqos-7/ 2023-07-28T23:43:47.427 DEBUG:boto:path=/hiiqos-7/?versions&max-keys=999 2023-07-28T23:43:47.427 DEBUG:boto:auth_path=/hiiqos-7/?versions&max-keys=999 2023-07-28T23:43:47.428 DEBUG:boto:Method: GET 2023-07-28T23:43:47.429 DEBUG:boto:Path: /hiiqos-7/?versions&max-keys=999 2023-07-28T23:43:47.429 DEBUG:boto:Data: 2023-07-28T23:43:47.430 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.430 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:43:47.430 DEBUG:boto:Port: 8000 2023-07-28T23:43:47.431 DEBUG:boto:Params: {} 2023-07-28T23:43:47.431 DEBUG:boto:Token: None 2023-07-28T23:43:47.431 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/?versions 2023-07-28T23:43:47.432 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rjfsIHCXpgn5vMTvmdVM/CNogEc= 2023-07-28T23:43:47.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rjfsIHCXpgn5vMTvmdVM/CNogEc=', 'Content-Length': '0'} 2023-07-28T23:43:47.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089ef737fe7543c07-0064c452b3-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.435 DEBUG:boto:b'hiiqos-7999false&nulltrue2023-07-28T23:42:36.922Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-07-28T23:42:36.896Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-07-28T23:42:36.876Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-07-28T23:42:36.820Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:43:47.436 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-07-28T23:43:47.437 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-07-28T23:43:47.437 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-07-28T23:43:47.437 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-07-28T23:43:47.437 DEBUG:boto:path=/hiiqos-7/ 2023-07-28T23:43:47.438 DEBUG:boto:auth_path=/hiiqos-7/ 2023-07-28T23:43:47.438 DEBUG:boto:path=/hiiqos-7/?versions&max-keys=999 2023-07-28T23:43:47.439 DEBUG:boto:auth_path=/hiiqos-7/?versions&max-keys=999 2023-07-28T23:43:47.439 DEBUG:boto:Method: GET 2023-07-28T23:43:47.439 DEBUG:boto:Path: /hiiqos-7/?versions&max-keys=999 2023-07-28T23:43:47.440 DEBUG:boto:Data: 2023-07-28T23:43:47.440 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.440 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:43:47.441 DEBUG:boto:Port: 8001 2023-07-28T23:43:47.441 DEBUG:boto:Params: {} 2023-07-28T23:43:47.441 DEBUG:boto:Token: None 2023-07-28T23:43:47.442 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/?versions 2023-07-28T23:43:47.442 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rjfsIHCXpgn5vMTvmdVM/CNogEc= 2023-07-28T23:43:47.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rjfsIHCXpgn5vMTvmdVM/CNogEc=', 'Content-Length': '0'} 2023-07-28T23:43:47.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008c7db3bcef51c5a-0064c452b3-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.445 DEBUG:boto:b'hiiqos-7999false&nulltrue2023-07-28T23:42:36.922Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-07-28T23:42:36.896Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-07-28T23:42:36.876Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-07-28T23:42:36.820Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:43:47.446 DEBUG:boto:path=/hiiqos-7/ 2023-07-28T23:43:47.447 DEBUG:boto:auth_path=/hiiqos-7/ 2023-07-28T23:43:47.447 DEBUG:boto:path=/hiiqos-7/?versions&max-keys=999 2023-07-28T23:43:47.447 DEBUG:boto:auth_path=/hiiqos-7/?versions&max-keys=999 2023-07-28T23:43:47.448 DEBUG:boto:Method: GET 2023-07-28T23:43:47.448 DEBUG:boto:Path: /hiiqos-7/?versions&max-keys=999 2023-07-28T23:43:47.448 DEBUG:boto:Data: 2023-07-28T23:43:47.449 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.449 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:43:47.449 DEBUG:boto:Port: 8000 2023-07-28T23:43:47.450 DEBUG:boto:Params: {} 2023-07-28T23:43:47.450 DEBUG:boto:Token: None 2023-07-28T23:43:47.450 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/?versions 2023-07-28T23:43:47.451 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rjfsIHCXpgn5vMTvmdVM/CNogEc= 2023-07-28T23:43:47.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rjfsIHCXpgn5vMTvmdVM/CNogEc=', 'Content-Length': '0'} 2023-07-28T23:43:47.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a6dbb5c6a992d0e-0064c452b3-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.454 DEBUG:boto:b'hiiqos-7999false&nulltrue2023-07-28T23:42:36.922Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-07-28T23:42:36.896Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-07-28T23:42:36.876Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-07-28T23:42:36.820Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:43:47.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-07-28T23:43:47.455 DEBUG:boto:path=/hiiqos-7/%26 2023-07-28T23:43:47.456 DEBUG:boto:auth_path=/hiiqos-7/%26 2023-07-28T23:43:47.456 DEBUG:boto:path=/hiiqos-7/%26?versionId=null 2023-07-28T23:43:47.456 DEBUG:boto:auth_path=/hiiqos-7/%26?versionId=null 2023-07-28T23:43:47.457 DEBUG:boto:Method: GET 2023-07-28T23:43:47.457 DEBUG:boto:Path: /hiiqos-7/%26?versionId=null 2023-07-28T23:43:47.458 DEBUG:boto:Data: 2023-07-28T23:43:47.458 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.458 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:43:47.459 DEBUG:boto:Port: 8001 2023-07-28T23:43:47.459 DEBUG:boto:Params: {} 2023-07-28T23:43:47.459 DEBUG:boto:Token: None 2023-07-28T23:43:47.460 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/%26?versionId=null 2023-07-28T23:43:47.460 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2WERWSSD3m9uFvdcSAghIEAo3rk= 2023-07-28T23:43:47.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2WERWSSD3m9uFvdcSAghIEAo3rk=', 'Content-Length': '0'} 2023-07-28T23:43:47.463 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dfe043b1797adbef-0064c452b3-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.463 DEBUG:boto:path=/hiiqos-7/%26 2023-07-28T23:43:47.464 DEBUG:boto:auth_path=/hiiqos-7/%26 2023-07-28T23:43:47.464 DEBUG:boto:path=/hiiqos-7/%26?versionId=null 2023-07-28T23:43:47.464 DEBUG:boto:auth_path=/hiiqos-7/%26?versionId=null 2023-07-28T23:43:47.465 DEBUG:boto:Method: GET 2023-07-28T23:43:47.465 DEBUG:boto:Path: /hiiqos-7/%26?versionId=null 2023-07-28T23:43:47.465 DEBUG:boto:Data: 2023-07-28T23:43:47.466 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.466 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:43:47.466 DEBUG:boto:Port: 8000 2023-07-28T23:43:47.467 DEBUG:boto:Params: {} 2023-07-28T23:43:47.467 DEBUG:boto:Token: None 2023-07-28T23:43:47.468 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/%26?versionId=null 2023-07-28T23:43:47.468 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2WERWSSD3m9uFvdcSAghIEAo3rk= 2023-07-28T23:43:47.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2WERWSSD3m9uFvdcSAghIEAo3rk=', 'Content-Length': '0'} 2023-07-28T23:43:47.470 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006d2f799e5af6fb8e-0064c452b3-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.471 DEBUG:boto:path=/hiiqos-7/%26 2023-07-28T23:43:47.471 DEBUG:boto:auth_path=/hiiqos-7/%26 2023-07-28T23:43:47.472 DEBUG:boto:path=/hiiqos-7/%26?versionId=null 2023-07-28T23:43:47.472 DEBUG:boto:auth_path=/hiiqos-7/%26?versionId=null 2023-07-28T23:43:47.472 DEBUG:boto:Method: HEAD 2023-07-28T23:43:47.473 DEBUG:boto:Path: /hiiqos-7/%26?versionId=null 2023-07-28T23:43:47.473 DEBUG:boto:Data: 2023-07-28T23:43:47.473 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.474 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:43:47.474 DEBUG:boto:Port: 8001 2023-07-28T23:43:47.474 DEBUG:boto:Params: {} 2023-07-28T23:43:47.475 DEBUG:boto:Token: None 2023-07-28T23:43:47.475 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/%26?versionId=null 2023-07-28T23:43:47.475 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Dz+zuOeA3XlH1vsf204Pfv/oRZc= 2023-07-28T23:43:47.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Dz+zuOeA3XlH1vsf204Pfv/oRZc=', 'Content-Length': '0'} 2023-07-28T23:43:47.478 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000047a2196527db5644-0064c452b3-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.478 DEBUG:boto:path=/hiiqos-7/%26 2023-07-28T23:43:47.479 DEBUG:boto:auth_path=/hiiqos-7/%26 2023-07-28T23:43:47.479 DEBUG:boto:path=/hiiqos-7/%26?versionId=null 2023-07-28T23:43:47.479 DEBUG:boto:auth_path=/hiiqos-7/%26?versionId=null 2023-07-28T23:43:47.480 DEBUG:boto:Method: HEAD 2023-07-28T23:43:47.480 DEBUG:boto:Path: /hiiqos-7/%26?versionId=null 2023-07-28T23:43:47.480 DEBUG:boto:Data: 2023-07-28T23:43:47.481 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.481 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:43:47.481 DEBUG:boto:Port: 8000 2023-07-28T23:43:47.482 DEBUG:boto:Params: {} 2023-07-28T23:43:47.482 DEBUG:boto:Token: None 2023-07-28T23:43:47.482 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/%26?versionId=null 2023-07-28T23:43:47.483 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Dz+zuOeA3XlH1vsf204Pfv/oRZc= 2023-07-28T23:43:47.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Dz+zuOeA3XlH1vsf204Pfv/oRZc=', 'Content-Length': '0'} 2023-07-28T23:43:47.485 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009b23e905b8a1d869-0064c452b3-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-07-28T23:43:47.486 DEBUG:boto:path=/hiiqos-7/%26 2023-07-28T23:43:47.486 DEBUG:boto:auth_path=/hiiqos-7/%26 2023-07-28T23:43:47.487 DEBUG:boto:path=/hiiqos-7/%26?versionId=null 2023-07-28T23:43:47.487 DEBUG:boto:auth_path=/hiiqos-7/%26?versionId=null 2023-07-28T23:43:47.487 DEBUG:boto:Method: GET 2023-07-28T23:43:47.488 DEBUG:boto:Path: /hiiqos-7/%26?versionId=null 2023-07-28T23:43:47.488 DEBUG:boto:Data: 2023-07-28T23:43:47.488 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.489 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:43:47.489 DEBUG:boto:Port: 8001 2023-07-28T23:43:47.489 DEBUG:boto:Params: {} 2023-07-28T23:43:47.490 DEBUG:boto:Token: None 2023-07-28T23:43:47.490 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/%26?versionId=null 2023-07-28T23:43:47.491 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2WERWSSD3m9uFvdcSAghIEAo3rk= 2023-07-28T23:43:47.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2WERWSSD3m9uFvdcSAghIEAo3rk=', 'Content-Length': '0'} 2023-07-28T23:43:47.493 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f0a9c378701cf353-0064c452b3-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.494 DEBUG:boto:path=/hiiqos-7/%26 2023-07-28T23:43:47.495 DEBUG:boto:auth_path=/hiiqos-7/%26 2023-07-28T23:43:47.495 DEBUG:boto:path=/hiiqos-7/%26?versionId=null 2023-07-28T23:43:47.495 DEBUG:boto:auth_path=/hiiqos-7/%26?versionId=null 2023-07-28T23:43:47.496 DEBUG:boto:Method: GET 2023-07-28T23:43:47.496 DEBUG:boto:Path: /hiiqos-7/%26?versionId=null 2023-07-28T23:43:47.497 DEBUG:boto:Data: 2023-07-28T23:43:47.497 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.497 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:43:47.498 DEBUG:boto:Port: 8000 2023-07-28T23:43:47.498 DEBUG:boto:Params: {} 2023-07-28T23:43:47.498 DEBUG:boto:Token: None 2023-07-28T23:43:47.499 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/%26?versionId=null 2023-07-28T23:43:47.499 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2WERWSSD3m9uFvdcSAghIEAo3rk= 2023-07-28T23:43:47.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2WERWSSD3m9uFvdcSAghIEAo3rk=', 'Content-Length': '0'} 2023-07-28T23:43:47.501 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001c7af9e0296a7b42-0064c452b3-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.502 DEBUG:boto:path=/hiiqos-7/ 2023-07-28T23:43:47.502 DEBUG:boto:auth_path=/hiiqos-7/ 2023-07-28T23:43:47.503 DEBUG:boto:path=/hiiqos-7/?prefix=%26 2023-07-28T23:43:47.503 DEBUG:boto:auth_path=/hiiqos-7/?prefix=%26 2023-07-28T23:43:47.503 DEBUG:boto:Method: GET 2023-07-28T23:43:47.504 DEBUG:boto:Path: /hiiqos-7/?prefix=%26 2023-07-28T23:43:47.504 DEBUG:boto:Data: 2023-07-28T23:43:47.504 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.505 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:43:47.505 DEBUG:boto:Port: 8001 2023-07-28T23:43:47.506 DEBUG:boto:Params: {} 2023-07-28T23:43:47.506 DEBUG:boto:Token: None 2023-07-28T23:43:47.506 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/ 2023-07-28T23:43:47.507 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Z78YuLqN3rP/OYpHG0yaosE1X+k= 2023-07-28T23:43:47.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Z78YuLqN3rP/OYpHG0yaosE1X+k=', 'Content-Length': '0'} 2023-07-28T23:43:47.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ce1318cf573c89e-0064c452b3-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.552 DEBUG:boto:b'hiiqos-7&1000false&2023-07-28T23:42:36.922Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:43:47.553 DEBUG:boto:path=/hiiqos-7/ 2023-07-28T23:43:47.553 DEBUG:boto:auth_path=/hiiqos-7/ 2023-07-28T23:43:47.554 DEBUG:boto:path=/hiiqos-7/?prefix=%26 2023-07-28T23:43:47.554 DEBUG:boto:auth_path=/hiiqos-7/?prefix=%26 2023-07-28T23:43:47.554 DEBUG:boto:Method: GET 2023-07-28T23:43:47.555 DEBUG:boto:Path: /hiiqos-7/?prefix=%26 2023-07-28T23:43:47.555 DEBUG:boto:Data: 2023-07-28T23:43:47.555 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.556 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:43:47.556 DEBUG:boto:Port: 8000 2023-07-28T23:43:47.556 DEBUG:boto:Params: {} 2023-07-28T23:43:47.557 DEBUG:boto:Token: None 2023-07-28T23:43:47.557 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/ 2023-07-28T23:43:47.558 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Z78YuLqN3rP/OYpHG0yaosE1X+k= 2023-07-28T23:43:47.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Z78YuLqN3rP/OYpHG0yaosE1X+k=', 'Content-Length': '0'} 2023-07-28T23:43:47.560 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0d2ca283a644e3f-0064c452b3-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.561 DEBUG:boto:b'hiiqos-7&1000false&2023-07-28T23:42:36.922Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:43:47.562 DEBUG:boto:path=/hiiqos-7/%26 2023-07-28T23:43:47.562 DEBUG:boto:auth_path=/hiiqos-7/%26 2023-07-28T23:43:47.562 DEBUG:boto:Method: HEAD 2023-07-28T23:43:47.563 DEBUG:boto:Path: /hiiqos-7/%26 2023-07-28T23:43:47.563 DEBUG:boto:Data: 2023-07-28T23:43:47.563 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.564 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:43:47.564 DEBUG:boto:Port: 8001 2023-07-28T23:43:47.564 DEBUG:boto:Params: {} 2023-07-28T23:43:47.565 DEBUG:boto:Token: None 2023-07-28T23:43:47.565 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/%26 2023-07-28T23:43:47.565 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4gxcdkJE9b534IHNv2SERJMx7ME= 2023-07-28T23:43:47.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4gxcdkJE9b534IHNv2SERJMx7ME=', 'Content-Length': '0'} 2023-07-28T23:43:47.568 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009665dea39a626b92-0064c452b3-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.568 DEBUG:boto:path=/hiiqos-7/%26 2023-07-28T23:43:47.569 DEBUG:boto:auth_path=/hiiqos-7/%26 2023-07-28T23:43:47.569 DEBUG:boto:Method: HEAD 2023-07-28T23:43:47.569 DEBUG:boto:Path: /hiiqos-7/%26 2023-07-28T23:43:47.570 DEBUG:boto:Data: 2023-07-28T23:43:47.570 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.570 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:43:47.571 DEBUG:boto:Port: 8000 2023-07-28T23:43:47.571 DEBUG:boto:Params: {} 2023-07-28T23:43:47.571 DEBUG:boto:Token: None 2023-07-28T23:43:47.572 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/%26 2023-07-28T23:43:47.572 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4gxcdkJE9b534IHNv2SERJMx7ME= 2023-07-28T23:43:47.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4gxcdkJE9b534IHNv2SERJMx7ME=', 'Content-Length': '0'} 2023-07-28T23:43:47.574 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000016cc85b7bf8c2839-0064c452b3-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-07-28T23:43:47.575 DEBUG:boto:path=/hiiqos-7/%26 2023-07-28T23:43:47.576 DEBUG:boto:auth_path=/hiiqos-7/%26 2023-07-28T23:43:47.576 DEBUG:boto:Method: GET 2023-07-28T23:43:47.576 DEBUG:boto:Path: /hiiqos-7/%26 2023-07-28T23:43:47.577 DEBUG:boto:Data: 2023-07-28T23:43:47.577 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.577 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:43:47.578 DEBUG:boto:Port: 8001 2023-07-28T23:43:47.578 DEBUG:boto:Params: {} 2023-07-28T23:43:47.578 DEBUG:boto:Token: None 2023-07-28T23:43:47.579 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/%26 2023-07-28T23:43:47.579 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4/fj6hiRnkZw8/dsuOdrOge7QCM= 2023-07-28T23:43:47.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4/fj6hiRnkZw8/dsuOdrOge7QCM=', 'Content-Length': '0'} 2023-07-28T23:43:47.582 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000046870a7b7d678923-0064c452b3-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.582 DEBUG:boto:path=/hiiqos-7/%26 2023-07-28T23:43:47.583 DEBUG:boto:auth_path=/hiiqos-7/%26 2023-07-28T23:43:47.583 DEBUG:boto:Method: GET 2023-07-28T23:43:47.583 DEBUG:boto:Path: /hiiqos-7/%26 2023-07-28T23:43:47.584 DEBUG:boto:Data: 2023-07-28T23:43:47.584 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.585 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:43:47.585 DEBUG:boto:Port: 8000 2023-07-28T23:43:47.585 DEBUG:boto:Params: {} 2023-07-28T23:43:47.586 DEBUG:boto:Token: None 2023-07-28T23:43:47.586 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/%26 2023-07-28T23:43:47.586 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4/fj6hiRnkZw8/dsuOdrOge7QCM= 2023-07-28T23:43:47.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4/fj6hiRnkZw8/dsuOdrOge7QCM=', 'Content-Length': '0'} 2023-07-28T23:43:47.589 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001804407d9e3b370f-0064c452b3-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.632 DEBUG:boto:path=/hiiqos-7/ 2023-07-28T23:43:47.633 DEBUG:boto:auth_path=/hiiqos-7/ 2023-07-28T23:43:47.633 DEBUG:boto:path=/hiiqos-7/?prefix=%26 2023-07-28T23:43:47.633 DEBUG:boto:auth_path=/hiiqos-7/?prefix=%26 2023-07-28T23:43:47.634 DEBUG:boto:Method: GET 2023-07-28T23:43:47.634 DEBUG:boto:Path: /hiiqos-7/?prefix=%26 2023-07-28T23:43:47.634 DEBUG:boto:Data: 2023-07-28T23:43:47.635 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.635 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:43:47.635 DEBUG:boto:Port: 8001 2023-07-28T23:43:47.636 DEBUG:boto:Params: {} 2023-07-28T23:43:47.636 DEBUG:boto:Token: None 2023-07-28T23:43:47.637 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/ 2023-07-28T23:43:47.637 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Z78YuLqN3rP/OYpHG0yaosE1X+k= 2023-07-28T23:43:47.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Z78YuLqN3rP/OYpHG0yaosE1X+k=', 'Content-Length': '0'} 2023-07-28T23:43:47.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000460ef570e7ed141f-0064c452b3-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.640 DEBUG:boto:b'hiiqos-7&1000false&2023-07-28T23:42:36.922Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:43:47.641 DEBUG:boto:path=/hiiqos-7/ 2023-07-28T23:43:47.641 DEBUG:boto:auth_path=/hiiqos-7/ 2023-07-28T23:43:47.642 DEBUG:boto:path=/hiiqos-7/?prefix=%26 2023-07-28T23:43:47.642 DEBUG:boto:auth_path=/hiiqos-7/?prefix=%26 2023-07-28T23:43:47.642 DEBUG:boto:Method: GET 2023-07-28T23:43:47.643 DEBUG:boto:Path: /hiiqos-7/?prefix=%26 2023-07-28T23:43:47.643 DEBUG:boto:Data: 2023-07-28T23:43:47.643 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.644 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:43:47.644 DEBUG:boto:Port: 8000 2023-07-28T23:43:47.645 DEBUG:boto:Params: {} 2023-07-28T23:43:47.645 DEBUG:boto:Token: None 2023-07-28T23:43:47.645 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/ 2023-07-28T23:43:47.646 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Z78YuLqN3rP/OYpHG0yaosE1X+k= 2023-07-28T23:43:47.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Z78YuLqN3rP/OYpHG0yaosE1X+k=', 'Content-Length': '0'} 2023-07-28T23:43:47.649 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1be5c06aa20a079-0064c452b3-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.650 DEBUG:boto:b'hiiqos-7&1000false&2023-07-28T23:42:36.922Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:43:47.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-07-28T23:43:47.651 DEBUG:boto:path=/hiiqos-7/%3A 2023-07-28T23:43:47.651 DEBUG:boto:auth_path=/hiiqos-7/%3A 2023-07-28T23:43:47.652 DEBUG:boto:path=/hiiqos-7/%3A?versionId=null 2023-07-28T23:43:47.652 DEBUG:boto:auth_path=/hiiqos-7/%3A?versionId=null 2023-07-28T23:43:47.652 DEBUG:boto:Method: GET 2023-07-28T23:43:47.653 DEBUG:boto:Path: /hiiqos-7/%3A?versionId=null 2023-07-28T23:43:47.653 DEBUG:boto:Data: 2023-07-28T23:43:47.653 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.654 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:43:47.654 DEBUG:boto:Port: 8001 2023-07-28T23:43:47.654 DEBUG:boto:Params: {} 2023-07-28T23:43:47.655 DEBUG:boto:Token: None 2023-07-28T23:43:47.655 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/%3A?versionId=null 2023-07-28T23:43:47.656 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6OGbn/KkCUlLHe9E/ZyOkfEjsJg= 2023-07-28T23:43:47.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6OGbn/KkCUlLHe9E/ZyOkfEjsJg=', 'Content-Length': '0'} 2023-07-28T23:43:47.658 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030f767be419b2e17-0064c452b3-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.658 DEBUG:boto:path=/hiiqos-7/%3A 2023-07-28T23:43:47.659 DEBUG:boto:auth_path=/hiiqos-7/%3A 2023-07-28T23:43:47.659 DEBUG:boto:path=/hiiqos-7/%3A?versionId=null 2023-07-28T23:43:47.660 DEBUG:boto:auth_path=/hiiqos-7/%3A?versionId=null 2023-07-28T23:43:47.660 DEBUG:boto:Method: GET 2023-07-28T23:43:47.660 DEBUG:boto:Path: /hiiqos-7/%3A?versionId=null 2023-07-28T23:43:47.661 DEBUG:boto:Data: 2023-07-28T23:43:47.661 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.661 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:43:47.662 DEBUG:boto:Port: 8000 2023-07-28T23:43:47.662 DEBUG:boto:Params: {} 2023-07-28T23:43:47.662 DEBUG:boto:Token: None 2023-07-28T23:43:47.663 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/%3A?versionId=null 2023-07-28T23:43:47.663 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6OGbn/KkCUlLHe9E/ZyOkfEjsJg= 2023-07-28T23:43:47.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6OGbn/KkCUlLHe9E/ZyOkfEjsJg=', 'Content-Length': '0'} 2023-07-28T23:43:47.666 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c931e350afc145a1-0064c452b3-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.666 DEBUG:boto:path=/hiiqos-7/%3A 2023-07-28T23:43:47.667 DEBUG:boto:auth_path=/hiiqos-7/%3A 2023-07-28T23:43:47.667 DEBUG:boto:path=/hiiqos-7/%3A?versionId=null 2023-07-28T23:43:47.667 DEBUG:boto:auth_path=/hiiqos-7/%3A?versionId=null 2023-07-28T23:43:47.668 DEBUG:boto:Method: HEAD 2023-07-28T23:43:47.668 DEBUG:boto:Path: /hiiqos-7/%3A?versionId=null 2023-07-28T23:43:47.668 DEBUG:boto:Data: 2023-07-28T23:43:47.669 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.669 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:43:47.669 DEBUG:boto:Port: 8001 2023-07-28T23:43:47.670 DEBUG:boto:Params: {} 2023-07-28T23:43:47.671 DEBUG:boto:Token: None 2023-07-28T23:43:47.671 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/%3A?versionId=null 2023-07-28T23:43:47.672 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MnSCaA8n+BEPXoFwCi0y3/5N7IA= 2023-07-28T23:43:47.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MnSCaA8n+BEPXoFwCi0y3/5N7IA=', 'Content-Length': '0'} 2023-07-28T23:43:47.674 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ccad15563bf68f4a-0064c452b3-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.675 DEBUG:boto:path=/hiiqos-7/%3A 2023-07-28T23:43:47.675 DEBUG:boto:auth_path=/hiiqos-7/%3A 2023-07-28T23:43:47.675 DEBUG:boto:path=/hiiqos-7/%3A?versionId=null 2023-07-28T23:43:47.676 DEBUG:boto:auth_path=/hiiqos-7/%3A?versionId=null 2023-07-28T23:43:47.676 DEBUG:boto:Method: HEAD 2023-07-28T23:43:47.676 DEBUG:boto:Path: /hiiqos-7/%3A?versionId=null 2023-07-28T23:43:47.677 DEBUG:boto:Data: 2023-07-28T23:43:47.677 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.677 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:43:47.678 DEBUG:boto:Port: 8000 2023-07-28T23:43:47.678 DEBUG:boto:Params: {} 2023-07-28T23:43:47.678 DEBUG:boto:Token: None 2023-07-28T23:43:47.679 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/%3A?versionId=null 2023-07-28T23:43:47.679 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MnSCaA8n+BEPXoFwCi0y3/5N7IA= 2023-07-28T23:43:47.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MnSCaA8n+BEPXoFwCi0y3/5N7IA=', 'Content-Length': '0'} 2023-07-28T23:43:47.681 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000376a7343fdc1d9cb-0064c452b3-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-07-28T23:43:47.683 DEBUG:boto:path=/hiiqos-7/%3A 2023-07-28T23:43:47.683 DEBUG:boto:auth_path=/hiiqos-7/%3A 2023-07-28T23:43:47.684 DEBUG:boto:path=/hiiqos-7/%3A?versionId=null 2023-07-28T23:43:47.685 DEBUG:boto:auth_path=/hiiqos-7/%3A?versionId=null 2023-07-28T23:43:47.685 DEBUG:boto:Method: GET 2023-07-28T23:43:47.686 DEBUG:boto:Path: /hiiqos-7/%3A?versionId=null 2023-07-28T23:43:47.687 DEBUG:boto:Data: 2023-07-28T23:43:47.687 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.688 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:43:47.689 DEBUG:boto:Port: 8001 2023-07-28T23:43:47.689 DEBUG:boto:Params: {} 2023-07-28T23:43:47.690 DEBUG:boto:Token: None 2023-07-28T23:43:47.691 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/%3A?versionId=null 2023-07-28T23:43:47.691 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6OGbn/KkCUlLHe9E/ZyOkfEjsJg= 2023-07-28T23:43:47.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6OGbn/KkCUlLHe9E/ZyOkfEjsJg=', 'Content-Length': '0'} 2023-07-28T23:43:47.694 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fba1955c6c27567b-0064c452b3-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.695 DEBUG:boto:path=/hiiqos-7/%3A 2023-07-28T23:43:47.695 DEBUG:boto:auth_path=/hiiqos-7/%3A 2023-07-28T23:43:47.695 DEBUG:boto:path=/hiiqos-7/%3A?versionId=null 2023-07-28T23:43:47.696 DEBUG:boto:auth_path=/hiiqos-7/%3A?versionId=null 2023-07-28T23:43:47.696 DEBUG:boto:Method: GET 2023-07-28T23:43:47.697 DEBUG:boto:Path: /hiiqos-7/%3A?versionId=null 2023-07-28T23:43:47.697 DEBUG:boto:Data: 2023-07-28T23:43:47.697 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.698 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:43:47.698 DEBUG:boto:Port: 8000 2023-07-28T23:43:47.698 DEBUG:boto:Params: {} 2023-07-28T23:43:47.699 DEBUG:boto:Token: None 2023-07-28T23:43:47.699 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/%3A?versionId=null 2023-07-28T23:43:47.700 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6OGbn/KkCUlLHe9E/ZyOkfEjsJg= 2023-07-28T23:43:47.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6OGbn/KkCUlLHe9E/ZyOkfEjsJg=', 'Content-Length': '0'} 2023-07-28T23:43:47.702 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008e90c23b493ef57a-0064c452b3-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.703 DEBUG:boto:path=/hiiqos-7/ 2023-07-28T23:43:47.703 DEBUG:boto:auth_path=/hiiqos-7/ 2023-07-28T23:43:47.703 DEBUG:boto:path=/hiiqos-7/?prefix=%3A 2023-07-28T23:43:47.704 DEBUG:boto:auth_path=/hiiqos-7/?prefix=%3A 2023-07-28T23:43:47.704 DEBUG:boto:Method: GET 2023-07-28T23:43:47.704 DEBUG:boto:Path: /hiiqos-7/?prefix=%3A 2023-07-28T23:43:47.705 DEBUG:boto:Data: 2023-07-28T23:43:47.705 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.706 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:43:47.706 DEBUG:boto:Port: 8001 2023-07-28T23:43:47.706 DEBUG:boto:Params: {} 2023-07-28T23:43:47.707 DEBUG:boto:Token: None 2023-07-28T23:43:47.707 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/ 2023-07-28T23:43:47.708 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Z78YuLqN3rP/OYpHG0yaosE1X+k= 2023-07-28T23:43:47.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Z78YuLqN3rP/OYpHG0yaosE1X+k=', 'Content-Length': '0'} 2023-07-28T23:43:47.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4ed0cb69a09d706-0064c452b3-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.711 DEBUG:boto:b'hiiqos-7:1000false:2023-07-28T23:42:36.896Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:43:47.712 DEBUG:boto:path=/hiiqos-7/ 2023-07-28T23:43:47.712 DEBUG:boto:auth_path=/hiiqos-7/ 2023-07-28T23:43:47.713 DEBUG:boto:path=/hiiqos-7/?prefix=%3A 2023-07-28T23:43:47.713 DEBUG:boto:auth_path=/hiiqos-7/?prefix=%3A 2023-07-28T23:43:47.714 DEBUG:boto:Method: GET 2023-07-28T23:43:47.714 DEBUG:boto:Path: /hiiqos-7/?prefix=%3A 2023-07-28T23:43:47.714 DEBUG:boto:Data: 2023-07-28T23:43:47.715 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.715 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:43:47.715 DEBUG:boto:Port: 8000 2023-07-28T23:43:47.716 DEBUG:boto:Params: {} 2023-07-28T23:43:47.716 DEBUG:boto:Token: None 2023-07-28T23:43:47.717 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/ 2023-07-28T23:43:47.717 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Z78YuLqN3rP/OYpHG0yaosE1X+k= 2023-07-28T23:43:47.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Z78YuLqN3rP/OYpHG0yaosE1X+k=', 'Content-Length': '0'} 2023-07-28T23:43:47.720 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b10b42ef7084aa85-0064c452b3-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.720 DEBUG:boto:b'hiiqos-7:1000false:2023-07-28T23:42:36.896Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:43:47.721 DEBUG:boto:path=/hiiqos-7/%3A 2023-07-28T23:43:47.721 DEBUG:boto:auth_path=/hiiqos-7/%3A 2023-07-28T23:43:47.722 DEBUG:boto:Method: HEAD 2023-07-28T23:43:47.722 DEBUG:boto:Path: /hiiqos-7/%3A 2023-07-28T23:43:47.722 DEBUG:boto:Data: 2023-07-28T23:43:47.723 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.723 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:43:47.723 DEBUG:boto:Port: 8001 2023-07-28T23:43:47.724 DEBUG:boto:Params: {} 2023-07-28T23:43:47.724 DEBUG:boto:Token: None 2023-07-28T23:43:47.725 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/%3A 2023-07-28T23:43:47.725 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FGunT6ZLDNlqdqLYG+Wx/AiMehc= 2023-07-28T23:43:47.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FGunT6ZLDNlqdqLYG+Wx/AiMehc=', 'Content-Length': '0'} 2023-07-28T23:43:47.727 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006b6e649f0b682966-0064c452b3-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.728 DEBUG:boto:path=/hiiqos-7/%3A 2023-07-28T23:43:47.728 DEBUG:boto:auth_path=/hiiqos-7/%3A 2023-07-28T23:43:47.729 DEBUG:boto:Method: HEAD 2023-07-28T23:43:47.729 DEBUG:boto:Path: /hiiqos-7/%3A 2023-07-28T23:43:47.729 DEBUG:boto:Data: 2023-07-28T23:43:47.730 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.730 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:43:47.731 DEBUG:boto:Port: 8000 2023-07-28T23:43:47.731 DEBUG:boto:Params: {} 2023-07-28T23:43:47.731 DEBUG:boto:Token: None 2023-07-28T23:43:47.732 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/%3A 2023-07-28T23:43:47.732 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FGunT6ZLDNlqdqLYG+Wx/AiMehc= 2023-07-28T23:43:47.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FGunT6ZLDNlqdqLYG+Wx/AiMehc=', 'Content-Length': '0'} 2023-07-28T23:43:47.735 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004eb2d9e74535cabd-0064c452b3-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-07-28T23:43:47.736 DEBUG:boto:path=/hiiqos-7/%3A 2023-07-28T23:43:47.737 DEBUG:boto:auth_path=/hiiqos-7/%3A 2023-07-28T23:43:47.737 DEBUG:boto:Method: GET 2023-07-28T23:43:47.738 DEBUG:boto:Path: /hiiqos-7/%3A 2023-07-28T23:43:47.738 DEBUG:boto:Data: 2023-07-28T23:43:47.738 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.739 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:43:47.739 DEBUG:boto:Port: 8001 2023-07-28T23:43:47.740 DEBUG:boto:Params: {} 2023-07-28T23:43:47.740 DEBUG:boto:Token: None 2023-07-28T23:43:47.741 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/%3A 2023-07-28T23:43:47.741 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xzSIxj4OzbRhokCHo8cTdYvLAhU= 2023-07-28T23:43:47.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xzSIxj4OzbRhokCHo8cTdYvLAhU=', 'Content-Length': '0'} 2023-07-28T23:43:47.744 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000096661c693a1098cf-0064c452b3-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.745 DEBUG:boto:path=/hiiqos-7/%3A 2023-07-28T23:43:47.745 DEBUG:boto:auth_path=/hiiqos-7/%3A 2023-07-28T23:43:47.746 DEBUG:boto:Method: GET 2023-07-28T23:43:47.746 DEBUG:boto:Path: /hiiqos-7/%3A 2023-07-28T23:43:47.746 DEBUG:boto:Data: 2023-07-28T23:43:47.747 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.747 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:43:47.748 DEBUG:boto:Port: 8000 2023-07-28T23:43:47.748 DEBUG:boto:Params: {} 2023-07-28T23:43:47.748 DEBUG:boto:Token: None 2023-07-28T23:43:47.749 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/%3A 2023-07-28T23:43:47.749 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xzSIxj4OzbRhokCHo8cTdYvLAhU= 2023-07-28T23:43:47.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xzSIxj4OzbRhokCHo8cTdYvLAhU=', 'Content-Length': '0'} 2023-07-28T23:43:47.752 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ce0021031cf177a8-0064c452b3-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.752 DEBUG:boto:path=/hiiqos-7/ 2023-07-28T23:43:47.753 DEBUG:boto:auth_path=/hiiqos-7/ 2023-07-28T23:43:47.753 DEBUG:boto:path=/hiiqos-7/?prefix=%3A 2023-07-28T23:43:47.754 DEBUG:boto:auth_path=/hiiqos-7/?prefix=%3A 2023-07-28T23:43:47.754 DEBUG:boto:Method: GET 2023-07-28T23:43:47.754 DEBUG:boto:Path: /hiiqos-7/?prefix=%3A 2023-07-28T23:43:47.755 DEBUG:boto:Data: 2023-07-28T23:43:47.755 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.755 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:43:47.756 DEBUG:boto:Port: 8001 2023-07-28T23:43:47.756 DEBUG:boto:Params: {} 2023-07-28T23:43:47.757 DEBUG:boto:Token: None 2023-07-28T23:43:47.757 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/ 2023-07-28T23:43:47.758 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Z78YuLqN3rP/OYpHG0yaosE1X+k= 2023-07-28T23:43:47.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Z78YuLqN3rP/OYpHG0yaosE1X+k=', 'Content-Length': '0'} 2023-07-28T23:43:47.760 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000280aa0b4034f3c66-0064c452b3-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.761 DEBUG:boto:b'hiiqos-7:1000false:2023-07-28T23:42:36.896Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:43:47.762 DEBUG:boto:path=/hiiqos-7/ 2023-07-28T23:43:47.762 DEBUG:boto:auth_path=/hiiqos-7/ 2023-07-28T23:43:47.762 DEBUG:boto:path=/hiiqos-7/?prefix=%3A 2023-07-28T23:43:47.763 DEBUG:boto:auth_path=/hiiqos-7/?prefix=%3A 2023-07-28T23:43:47.763 DEBUG:boto:Method: GET 2023-07-28T23:43:47.763 DEBUG:boto:Path: /hiiqos-7/?prefix=%3A 2023-07-28T23:43:47.764 DEBUG:boto:Data: 2023-07-28T23:43:47.764 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.765 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:43:47.765 DEBUG:boto:Port: 8000 2023-07-28T23:43:47.765 DEBUG:boto:Params: {} 2023-07-28T23:43:47.766 DEBUG:boto:Token: None 2023-07-28T23:43:47.766 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/ 2023-07-28T23:43:47.767 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Z78YuLqN3rP/OYpHG0yaosE1X+k= 2023-07-28T23:43:47.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Z78YuLqN3rP/OYpHG0yaosE1X+k=', 'Content-Length': '0'} 2023-07-28T23:43:47.770 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f9505c7e30b730b-0064c452b3-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.770 DEBUG:boto:b'hiiqos-7:1000false:2023-07-28T23:42:36.896Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:43:47.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-07-28T23:43:47.771 DEBUG:boto:path=/hiiqos-7/_myobj 2023-07-28T23:43:47.772 DEBUG:boto:auth_path=/hiiqos-7/_myobj 2023-07-28T23:43:47.772 DEBUG:boto:path=/hiiqos-7/_myobj?versionId=null 2023-07-28T23:43:47.773 DEBUG:boto:auth_path=/hiiqos-7/_myobj?versionId=null 2023-07-28T23:43:47.773 DEBUG:boto:Method: GET 2023-07-28T23:43:47.773 DEBUG:boto:Path: /hiiqos-7/_myobj?versionId=null 2023-07-28T23:43:47.774 DEBUG:boto:Data: 2023-07-28T23:43:47.774 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.775 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:43:47.775 DEBUG:boto:Port: 8001 2023-07-28T23:43:47.775 DEBUG:boto:Params: {} 2023-07-28T23:43:47.776 DEBUG:boto:Token: None 2023-07-28T23:43:47.776 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/_myobj?versionId=null 2023-07-28T23:43:47.777 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0qYf1HlnkEu20m2B7C64b4LC0gA= 2023-07-28T23:43:47.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0qYf1HlnkEu20m2B7C64b4LC0gA=', 'Content-Length': '0'} 2023-07-28T23:43:47.779 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007ebe1f68ed099999-0064c452b3-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.820 DEBUG:boto:path=/hiiqos-7/_myobj 2023-07-28T23:43:47.821 DEBUG:boto:auth_path=/hiiqos-7/_myobj 2023-07-28T23:43:47.821 DEBUG:boto:path=/hiiqos-7/_myobj?versionId=null 2023-07-28T23:43:47.822 DEBUG:boto:auth_path=/hiiqos-7/_myobj?versionId=null 2023-07-28T23:43:47.822 DEBUG:boto:Method: GET 2023-07-28T23:43:47.822 DEBUG:boto:Path: /hiiqos-7/_myobj?versionId=null 2023-07-28T23:43:47.823 DEBUG:boto:Data: 2023-07-28T23:43:47.823 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.824 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:43:47.824 DEBUG:boto:Port: 8000 2023-07-28T23:43:47.824 DEBUG:boto:Params: {} 2023-07-28T23:43:47.825 DEBUG:boto:Token: None 2023-07-28T23:43:47.825 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/_myobj?versionId=null 2023-07-28T23:43:47.826 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0qYf1HlnkEu20m2B7C64b4LC0gA= 2023-07-28T23:43:47.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0qYf1HlnkEu20m2B7C64b4LC0gA=', 'Content-Length': '0'} 2023-07-28T23:43:47.828 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d5fe2866ea27cec3-0064c452b3-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.829 DEBUG:boto:path=/hiiqos-7/_myobj 2023-07-28T23:43:47.830 DEBUG:boto:auth_path=/hiiqos-7/_myobj 2023-07-28T23:43:47.830 DEBUG:boto:path=/hiiqos-7/_myobj?versionId=null 2023-07-28T23:43:47.830 DEBUG:boto:auth_path=/hiiqos-7/_myobj?versionId=null 2023-07-28T23:43:47.831 DEBUG:boto:Method: HEAD 2023-07-28T23:43:47.831 DEBUG:boto:Path: /hiiqos-7/_myobj?versionId=null 2023-07-28T23:43:47.831 DEBUG:boto:Data: 2023-07-28T23:43:47.832 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.832 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:43:47.833 DEBUG:boto:Port: 8001 2023-07-28T23:43:47.833 DEBUG:boto:Params: {} 2023-07-28T23:43:47.833 DEBUG:boto:Token: None 2023-07-28T23:43:47.834 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/_myobj?versionId=null 2023-07-28T23:43:47.834 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SGQYjfNILizsOHzel4SsTHE1F4Q= 2023-07-28T23:43:47.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SGQYjfNILizsOHzel4SsTHE1F4Q=', 'Content-Length': '0'} 2023-07-28T23:43:47.836 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000875d7976f62f57ca-0064c452b3-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.837 DEBUG:boto:path=/hiiqos-7/_myobj 2023-07-28T23:43:47.837 DEBUG:boto:auth_path=/hiiqos-7/_myobj 2023-07-28T23:43:47.838 DEBUG:boto:path=/hiiqos-7/_myobj?versionId=null 2023-07-28T23:43:47.838 DEBUG:boto:auth_path=/hiiqos-7/_myobj?versionId=null 2023-07-28T23:43:47.839 DEBUG:boto:Method: HEAD 2023-07-28T23:43:47.839 DEBUG:boto:Path: /hiiqos-7/_myobj?versionId=null 2023-07-28T23:43:47.839 DEBUG:boto:Data: 2023-07-28T23:43:47.840 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.840 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:43:47.840 DEBUG:boto:Port: 8000 2023-07-28T23:43:47.841 DEBUG:boto:Params: {} 2023-07-28T23:43:47.841 DEBUG:boto:Token: None 2023-07-28T23:43:47.842 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/_myobj?versionId=null 2023-07-28T23:43:47.842 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SGQYjfNILizsOHzel4SsTHE1F4Q= 2023-07-28T23:43:47.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SGQYjfNILizsOHzel4SsTHE1F4Q=', 'Content-Length': '0'} 2023-07-28T23:43:47.844 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000023b3b74c48bf87e2-0064c452b3-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-07-28T23:43:47.845 DEBUG:boto:path=/hiiqos-7/_myobj 2023-07-28T23:43:47.846 DEBUG:boto:auth_path=/hiiqos-7/_myobj 2023-07-28T23:43:47.846 DEBUG:boto:path=/hiiqos-7/_myobj?versionId=null 2023-07-28T23:43:47.846 DEBUG:boto:auth_path=/hiiqos-7/_myobj?versionId=null 2023-07-28T23:43:47.847 DEBUG:boto:Method: GET 2023-07-28T23:43:47.847 DEBUG:boto:Path: /hiiqos-7/_myobj?versionId=null 2023-07-28T23:43:47.848 DEBUG:boto:Data: 2023-07-28T23:43:47.848 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.848 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:43:47.849 DEBUG:boto:Port: 8001 2023-07-28T23:43:47.849 DEBUG:boto:Params: {} 2023-07-28T23:43:47.850 DEBUG:boto:Token: None 2023-07-28T23:43:47.850 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/_myobj?versionId=null 2023-07-28T23:43:47.850 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0qYf1HlnkEu20m2B7C64b4LC0gA= 2023-07-28T23:43:47.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0qYf1HlnkEu20m2B7C64b4LC0gA=', 'Content-Length': '0'} 2023-07-28T23:43:47.853 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000039c1d263e0b0bb72-0064c452b3-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.854 DEBUG:boto:path=/hiiqos-7/_myobj 2023-07-28T23:43:47.854 DEBUG:boto:auth_path=/hiiqos-7/_myobj 2023-07-28T23:43:47.855 DEBUG:boto:path=/hiiqos-7/_myobj?versionId=null 2023-07-28T23:43:47.855 DEBUG:boto:auth_path=/hiiqos-7/_myobj?versionId=null 2023-07-28T23:43:47.855 DEBUG:boto:Method: GET 2023-07-28T23:43:47.856 DEBUG:boto:Path: /hiiqos-7/_myobj?versionId=null 2023-07-28T23:43:47.856 DEBUG:boto:Data: 2023-07-28T23:43:47.857 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.857 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:43:47.857 DEBUG:boto:Port: 8000 2023-07-28T23:43:47.858 DEBUG:boto:Params: {} 2023-07-28T23:43:47.858 DEBUG:boto:Token: None 2023-07-28T23:43:47.858 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/_myobj?versionId=null 2023-07-28T23:43:47.859 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0qYf1HlnkEu20m2B7C64b4LC0gA= 2023-07-28T23:43:47.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0qYf1HlnkEu20m2B7C64b4LC0gA=', 'Content-Length': '0'} 2023-07-28T23:43:47.861 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000047e7bab8dd89b32e-0064c452b3-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.904 DEBUG:boto:path=/hiiqos-7/ 2023-07-28T23:43:47.905 DEBUG:boto:auth_path=/hiiqos-7/ 2023-07-28T23:43:47.905 DEBUG:boto:path=/hiiqos-7/?prefix=_myobj 2023-07-28T23:43:47.905 DEBUG:boto:auth_path=/hiiqos-7/?prefix=_myobj 2023-07-28T23:43:47.906 DEBUG:boto:Method: GET 2023-07-28T23:43:47.906 DEBUG:boto:Path: /hiiqos-7/?prefix=_myobj 2023-07-28T23:43:47.906 DEBUG:boto:Data: 2023-07-28T23:43:47.907 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.907 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:43:47.907 DEBUG:boto:Port: 8001 2023-07-28T23:43:47.908 DEBUG:boto:Params: {} 2023-07-28T23:43:47.908 DEBUG:boto:Token: None 2023-07-28T23:43:47.909 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/ 2023-07-28T23:43:47.909 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Z78YuLqN3rP/OYpHG0yaosE1X+k= 2023-07-28T23:43:47.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Z78YuLqN3rP/OYpHG0yaosE1X+k=', 'Content-Length': '0'} 2023-07-28T23:43:47.912 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050c9a4cd099fe3b8-0064c452b3-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.913 DEBUG:boto:b'hiiqos-7_myobj1000false_myobj2023-07-28T23:42:36.876Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:43:47.913 DEBUG:boto:path=/hiiqos-7/ 2023-07-28T23:43:47.914 DEBUG:boto:auth_path=/hiiqos-7/ 2023-07-28T23:43:47.914 DEBUG:boto:path=/hiiqos-7/?prefix=_myobj 2023-07-28T23:43:47.914 DEBUG:boto:auth_path=/hiiqos-7/?prefix=_myobj 2023-07-28T23:43:47.915 DEBUG:boto:Method: GET 2023-07-28T23:43:47.915 DEBUG:boto:Path: /hiiqos-7/?prefix=_myobj 2023-07-28T23:43:47.915 DEBUG:boto:Data: 2023-07-28T23:43:47.916 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.916 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:43:47.917 DEBUG:boto:Port: 8000 2023-07-28T23:43:47.917 DEBUG:boto:Params: {} 2023-07-28T23:43:47.917 DEBUG:boto:Token: None 2023-07-28T23:43:47.918 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/ 2023-07-28T23:43:47.918 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Z78YuLqN3rP/OYpHG0yaosE1X+k= 2023-07-28T23:43:47.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Z78YuLqN3rP/OYpHG0yaosE1X+k=', 'Content-Length': '0'} 2023-07-28T23:43:47.921 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e6b861970b58960-0064c452b3-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.921 DEBUG:boto:b'hiiqos-7_myobj1000false_myobj2023-07-28T23:42:36.876Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:43:47.922 DEBUG:boto:path=/hiiqos-7/_myobj 2023-07-28T23:43:47.922 DEBUG:boto:auth_path=/hiiqos-7/_myobj 2023-07-28T23:43:47.923 DEBUG:boto:Method: HEAD 2023-07-28T23:43:47.923 DEBUG:boto:Path: /hiiqos-7/_myobj 2023-07-28T23:43:47.923 DEBUG:boto:Data: 2023-07-28T23:43:47.924 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.924 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:43:47.924 DEBUG:boto:Port: 8001 2023-07-28T23:43:47.925 DEBUG:boto:Params: {} 2023-07-28T23:43:47.925 DEBUG:boto:Token: None 2023-07-28T23:43:47.925 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/_myobj 2023-07-28T23:43:47.926 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:z9PVurQyk2FAGJQVkJXM/n65Gtk= 2023-07-28T23:43:47.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:z9PVurQyk2FAGJQVkJXM/n65Gtk=', 'Content-Length': '0'} 2023-07-28T23:43:47.928 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000051695a6ff1bcce7a-0064c452b3-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.929 DEBUG:boto:path=/hiiqos-7/_myobj 2023-07-28T23:43:47.929 DEBUG:boto:auth_path=/hiiqos-7/_myobj 2023-07-28T23:43:47.929 DEBUG:boto:Method: HEAD 2023-07-28T23:43:47.930 DEBUG:boto:Path: /hiiqos-7/_myobj 2023-07-28T23:43:47.930 DEBUG:boto:Data: 2023-07-28T23:43:47.930 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.931 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:43:47.931 DEBUG:boto:Port: 8000 2023-07-28T23:43:47.931 DEBUG:boto:Params: {} 2023-07-28T23:43:47.932 DEBUG:boto:Token: None 2023-07-28T23:43:47.932 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/_myobj 2023-07-28T23:43:47.932 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:z9PVurQyk2FAGJQVkJXM/n65Gtk= 2023-07-28T23:43:47.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:z9PVurQyk2FAGJQVkJXM/n65Gtk=', 'Content-Length': '0'} 2023-07-28T23:43:47.935 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009991def48943f93d-0064c452b3-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-07-28T23:43:47.936 DEBUG:boto:path=/hiiqos-7/_myobj 2023-07-28T23:43:47.936 DEBUG:boto:auth_path=/hiiqos-7/_myobj 2023-07-28T23:43:47.936 DEBUG:boto:Method: GET 2023-07-28T23:43:47.937 DEBUG:boto:Path: /hiiqos-7/_myobj 2023-07-28T23:43:47.937 DEBUG:boto:Data: 2023-07-28T23:43:47.937 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.938 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:43:47.938 DEBUG:boto:Port: 8001 2023-07-28T23:43:47.939 DEBUG:boto:Params: {} 2023-07-28T23:43:47.939 DEBUG:boto:Token: None 2023-07-28T23:43:47.939 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/_myobj 2023-07-28T23:43:47.940 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Y/8u/m0Lr5qkVjAfZwd7wqXO0vk= 2023-07-28T23:43:47.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Y/8u/m0Lr5qkVjAfZwd7wqXO0vk=', 'Content-Length': '0'} 2023-07-28T23:43:47.942 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000880d539b25a4d2e6-0064c452b3-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.984 DEBUG:boto:path=/hiiqos-7/_myobj 2023-07-28T23:43:47.985 DEBUG:boto:auth_path=/hiiqos-7/_myobj 2023-07-28T23:43:47.985 DEBUG:boto:Method: GET 2023-07-28T23:43:47.986 DEBUG:boto:Path: /hiiqos-7/_myobj 2023-07-28T23:43:47.988 DEBUG:boto:Data: 2023-07-28T23:43:47.988 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.988 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:43:47.989 DEBUG:boto:Port: 8000 2023-07-28T23:43:47.989 DEBUG:boto:Params: {} 2023-07-28T23:43:47.990 DEBUG:boto:Token: None 2023-07-28T23:43:47.990 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/_myobj 2023-07-28T23:43:47.990 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Y/8u/m0Lr5qkVjAfZwd7wqXO0vk= 2023-07-28T23:43:47.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Y/8u/m0Lr5qkVjAfZwd7wqXO0vk=', 'Content-Length': '0'} 2023-07-28T23:43:47.993 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fbcdc1e413b5cdf9-0064c452b3-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:47.993 DEBUG:boto:path=/hiiqos-7/ 2023-07-28T23:43:47.994 DEBUG:boto:auth_path=/hiiqos-7/ 2023-07-28T23:43:47.994 DEBUG:boto:path=/hiiqos-7/?prefix=_myobj 2023-07-28T23:43:47.995 DEBUG:boto:auth_path=/hiiqos-7/?prefix=_myobj 2023-07-28T23:43:47.995 DEBUG:boto:Method: GET 2023-07-28T23:43:47.995 DEBUG:boto:Path: /hiiqos-7/?prefix=_myobj 2023-07-28T23:43:47.996 DEBUG:boto:Data: 2023-07-28T23:43:47.996 DEBUG:boto:Headers: {} 2023-07-28T23:43:47.996 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:43:47.997 DEBUG:boto:Port: 8001 2023-07-28T23:43:47.997 DEBUG:boto:Params: {} 2023-07-28T23:43:47.997 DEBUG:boto:Token: None 2023-07-28T23:43:47.998 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:47 GMT /hiiqos-7/ 2023-07-28T23:43:47.998 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Z78YuLqN3rP/OYpHG0yaosE1X+k= 2023-07-28T23:43:47.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Z78YuLqN3rP/OYpHG0yaosE1X+k=', 'Content-Length': '0'} 2023-07-28T23:43:48.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028a3f1aa240e861f-0064c452b4-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:43:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:48.002 DEBUG:boto:b'hiiqos-7_myobj1000false_myobj2023-07-28T23:42:36.876Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:43:48.003 DEBUG:boto:path=/hiiqos-7/ 2023-07-28T23:43:48.003 DEBUG:boto:auth_path=/hiiqos-7/ 2023-07-28T23:43:48.004 DEBUG:boto:path=/hiiqos-7/?prefix=_myobj 2023-07-28T23:43:48.004 DEBUG:boto:auth_path=/hiiqos-7/?prefix=_myobj 2023-07-28T23:43:48.004 DEBUG:boto:Method: GET 2023-07-28T23:43:48.005 DEBUG:boto:Path: /hiiqos-7/?prefix=_myobj 2023-07-28T23:43:48.005 DEBUG:boto:Data: 2023-07-28T23:43:48.005 DEBUG:boto:Headers: {} 2023-07-28T23:43:48.006 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:43:48.006 DEBUG:boto:Port: 8000 2023-07-28T23:43:48.007 DEBUG:boto:Params: {} 2023-07-28T23:43:48.007 DEBUG:boto:Token: None 2023-07-28T23:43:48.007 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:48 GMT /hiiqos-7/ 2023-07-28T23:43:48.008 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kcdvLvInQS7iHk6TVsRf113GCY4= 2023-07-28T23:43:48.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kcdvLvInQS7iHk6TVsRf113GCY4=', 'Content-Length': '0'} 2023-07-28T23:43:48.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fb2a57b745259f3-0064c452b4-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:43:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:48.052 DEBUG:boto:b'hiiqos-7_myobj1000false_myobj2023-07-28T23:42:36.876Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:43:48.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-07-28T23:43:48.053 DEBUG:boto:path=/hiiqos-7/myobj 2023-07-28T23:43:48.054 DEBUG:boto:auth_path=/hiiqos-7/myobj 2023-07-28T23:43:48.054 DEBUG:boto:path=/hiiqos-7/myobj?versionId=null 2023-07-28T23:43:48.055 DEBUG:boto:auth_path=/hiiqos-7/myobj?versionId=null 2023-07-28T23:43:48.055 DEBUG:boto:Method: GET 2023-07-28T23:43:48.055 DEBUG:boto:Path: /hiiqos-7/myobj?versionId=null 2023-07-28T23:43:48.056 DEBUG:boto:Data: 2023-07-28T23:43:48.056 DEBUG:boto:Headers: {} 2023-07-28T23:43:48.056 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:43:48.057 DEBUG:boto:Port: 8001 2023-07-28T23:43:48.057 DEBUG:boto:Params: {} 2023-07-28T23:43:48.057 DEBUG:boto:Token: None 2023-07-28T23:43:48.058 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:48 GMT /hiiqos-7/myobj?versionId=null 2023-07-28T23:43:48.058 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:c8owMCmgXDzni0eLcXTXTZJakiY= 2023-07-28T23:43:48.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c8owMCmgXDzni0eLcXTXTZJakiY=', 'Content-Length': '0'} 2023-07-28T23:43:48.061 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007984d28a9823806c-0064c452b4-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:43:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:48.062 DEBUG:boto:path=/hiiqos-7/myobj 2023-07-28T23:43:48.062 DEBUG:boto:auth_path=/hiiqos-7/myobj 2023-07-28T23:43:48.063 DEBUG:boto:path=/hiiqos-7/myobj?versionId=null 2023-07-28T23:43:48.063 DEBUG:boto:auth_path=/hiiqos-7/myobj?versionId=null 2023-07-28T23:43:48.063 DEBUG:boto:Method: GET 2023-07-28T23:43:48.064 DEBUG:boto:Path: /hiiqos-7/myobj?versionId=null 2023-07-28T23:43:48.064 DEBUG:boto:Data: 2023-07-28T23:43:48.064 DEBUG:boto:Headers: {} 2023-07-28T23:43:48.065 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:43:48.065 DEBUG:boto:Port: 8000 2023-07-28T23:43:48.066 DEBUG:boto:Params: {} 2023-07-28T23:43:48.066 DEBUG:boto:Token: None 2023-07-28T23:43:48.066 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:48 GMT /hiiqos-7/myobj?versionId=null 2023-07-28T23:43:48.067 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:c8owMCmgXDzni0eLcXTXTZJakiY= 2023-07-28T23:43:48.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c8owMCmgXDzni0eLcXTXTZJakiY=', 'Content-Length': '0'} 2023-07-28T23:43:48.069 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000088531c59ef8a9764-0064c452b4-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:43:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:48.070 DEBUG:boto:path=/hiiqos-7/myobj 2023-07-28T23:43:48.070 DEBUG:boto:auth_path=/hiiqos-7/myobj 2023-07-28T23:43:48.071 DEBUG:boto:path=/hiiqos-7/myobj?versionId=null 2023-07-28T23:43:48.071 DEBUG:boto:auth_path=/hiiqos-7/myobj?versionId=null 2023-07-28T23:43:48.071 DEBUG:boto:Method: HEAD 2023-07-28T23:43:48.072 DEBUG:boto:Path: /hiiqos-7/myobj?versionId=null 2023-07-28T23:43:48.072 DEBUG:boto:Data: 2023-07-28T23:43:48.072 DEBUG:boto:Headers: {} 2023-07-28T23:43:48.073 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:43:48.073 DEBUG:boto:Port: 8001 2023-07-28T23:43:48.073 DEBUG:boto:Params: {} 2023-07-28T23:43:48.074 DEBUG:boto:Token: None 2023-07-28T23:43:48.074 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:43:48 GMT /hiiqos-7/myobj?versionId=null 2023-07-28T23:43:48.074 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I7mKAMMlqtbG9ELYa4hBGDBYoYU= 2023-07-28T23:43:48.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I7mKAMMlqtbG9ELYa4hBGDBYoYU=', 'Content-Length': '0'} 2023-07-28T23:43:48.077 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c8b172eed11ea2c9-0064c452b4-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:43:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:48.077 DEBUG:boto:path=/hiiqos-7/myobj 2023-07-28T23:43:48.078 DEBUG:boto:auth_path=/hiiqos-7/myobj 2023-07-28T23:43:48.078 DEBUG:boto:path=/hiiqos-7/myobj?versionId=null 2023-07-28T23:43:48.078 DEBUG:boto:auth_path=/hiiqos-7/myobj?versionId=null 2023-07-28T23:43:48.079 DEBUG:boto:Method: HEAD 2023-07-28T23:43:48.079 DEBUG:boto:Path: /hiiqos-7/myobj?versionId=null 2023-07-28T23:43:48.079 DEBUG:boto:Data: 2023-07-28T23:43:48.080 DEBUG:boto:Headers: {} 2023-07-28T23:43:48.080 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:43:48.080 DEBUG:boto:Port: 8000 2023-07-28T23:43:48.081 DEBUG:boto:Params: {} 2023-07-28T23:43:48.081 DEBUG:boto:Token: None 2023-07-28T23:43:48.081 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:43:48 GMT /hiiqos-7/myobj?versionId=null 2023-07-28T23:43:48.082 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I7mKAMMlqtbG9ELYa4hBGDBYoYU= 2023-07-28T23:43:48.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I7mKAMMlqtbG9ELYa4hBGDBYoYU=', 'Content-Length': '0'} 2023-07-28T23:43:48.084 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a516c0e55d4ba7b8-0064c452b4-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:43:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:48.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-07-28T23:43:48.085 DEBUG:boto:path=/hiiqos-7/myobj 2023-07-28T23:43:48.085 DEBUG:boto:auth_path=/hiiqos-7/myobj 2023-07-28T23:43:48.086 DEBUG:boto:path=/hiiqos-7/myobj?versionId=null 2023-07-28T23:43:48.086 DEBUG:boto:auth_path=/hiiqos-7/myobj?versionId=null 2023-07-28T23:43:48.087 DEBUG:boto:Method: GET 2023-07-28T23:43:48.087 DEBUG:boto:Path: /hiiqos-7/myobj?versionId=null 2023-07-28T23:43:48.087 DEBUG:boto:Data: 2023-07-28T23:43:48.088 DEBUG:boto:Headers: {} 2023-07-28T23:43:48.088 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:43:48.088 DEBUG:boto:Port: 8001 2023-07-28T23:43:48.089 DEBUG:boto:Params: {} 2023-07-28T23:43:48.089 DEBUG:boto:Token: None 2023-07-28T23:43:48.090 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:48 GMT /hiiqos-7/myobj?versionId=null 2023-07-28T23:43:48.090 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:c8owMCmgXDzni0eLcXTXTZJakiY= 2023-07-28T23:43:48.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c8owMCmgXDzni0eLcXTXTZJakiY=', 'Content-Length': '0'} 2023-07-28T23:43:48.092 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d124a453b4b1a1a6-0064c452b4-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:43:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:48.136 DEBUG:boto:path=/hiiqos-7/myobj 2023-07-28T23:43:48.137 DEBUG:boto:auth_path=/hiiqos-7/myobj 2023-07-28T23:43:48.137 DEBUG:boto:path=/hiiqos-7/myobj?versionId=null 2023-07-28T23:43:48.138 DEBUG:boto:auth_path=/hiiqos-7/myobj?versionId=null 2023-07-28T23:43:48.138 DEBUG:boto:Method: GET 2023-07-28T23:43:48.139 DEBUG:boto:Path: /hiiqos-7/myobj?versionId=null 2023-07-28T23:43:48.139 DEBUG:boto:Data: 2023-07-28T23:43:48.139 DEBUG:boto:Headers: {} 2023-07-28T23:43:48.140 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:43:48.140 DEBUG:boto:Port: 8000 2023-07-28T23:43:48.140 DEBUG:boto:Params: {} 2023-07-28T23:43:48.141 DEBUG:boto:Token: None 2023-07-28T23:43:48.141 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:48 GMT /hiiqos-7/myobj?versionId=null 2023-07-28T23:43:48.142 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:c8owMCmgXDzni0eLcXTXTZJakiY= 2023-07-28T23:43:48.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c8owMCmgXDzni0eLcXTXTZJakiY=', 'Content-Length': '0'} 2023-07-28T23:43:48.144 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ee8dffbf9f00a61-0064c452b4-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:43:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:48.144 DEBUG:boto:path=/hiiqos-7/ 2023-07-28T23:43:48.145 DEBUG:boto:auth_path=/hiiqos-7/ 2023-07-28T23:43:48.145 DEBUG:boto:path=/hiiqos-7/?prefix=myobj 2023-07-28T23:43:48.146 DEBUG:boto:auth_path=/hiiqos-7/?prefix=myobj 2023-07-28T23:43:48.146 DEBUG:boto:Method: GET 2023-07-28T23:43:48.146 DEBUG:boto:Path: /hiiqos-7/?prefix=myobj 2023-07-28T23:43:48.147 DEBUG:boto:Data: 2023-07-28T23:43:48.147 DEBUG:boto:Headers: {} 2023-07-28T23:43:48.147 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:43:48.148 DEBUG:boto:Port: 8001 2023-07-28T23:43:48.148 DEBUG:boto:Params: {} 2023-07-28T23:43:48.148 DEBUG:boto:Token: None 2023-07-28T23:43:48.149 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:48 GMT /hiiqos-7/ 2023-07-28T23:43:48.149 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kcdvLvInQS7iHk6TVsRf113GCY4= 2023-07-28T23:43:48.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kcdvLvInQS7iHk6TVsRf113GCY4=', 'Content-Length': '0'} 2023-07-28T23:43:48.152 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000399525f4f78a162e-0064c452b4-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:43:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:48.153 DEBUG:boto:b'hiiqos-7myobj1000falsemyobj2023-07-28T23:42:36.820Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:43:48.153 DEBUG:boto:path=/hiiqos-7/ 2023-07-28T23:43:48.154 DEBUG:boto:auth_path=/hiiqos-7/ 2023-07-28T23:43:48.154 DEBUG:boto:path=/hiiqos-7/?prefix=myobj 2023-07-28T23:43:48.154 DEBUG:boto:auth_path=/hiiqos-7/?prefix=myobj 2023-07-28T23:43:48.155 DEBUG:boto:Method: GET 2023-07-28T23:43:48.155 DEBUG:boto:Path: /hiiqos-7/?prefix=myobj 2023-07-28T23:43:48.155 DEBUG:boto:Data: 2023-07-28T23:43:48.156 DEBUG:boto:Headers: {} 2023-07-28T23:43:48.156 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:43:48.156 DEBUG:boto:Port: 8000 2023-07-28T23:43:48.157 DEBUG:boto:Params: {} 2023-07-28T23:43:48.157 DEBUG:boto:Token: None 2023-07-28T23:43:48.158 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:48 GMT /hiiqos-7/ 2023-07-28T23:43:48.158 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kcdvLvInQS7iHk6TVsRf113GCY4= 2023-07-28T23:43:48.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kcdvLvInQS7iHk6TVsRf113GCY4=', 'Content-Length': '0'} 2023-07-28T23:43:48.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092b34a3ba6d249b7-0064c452b4-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:43:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:48.161 DEBUG:boto:b'hiiqos-7myobj1000falsemyobj2023-07-28T23:42:36.820Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:43:48.162 DEBUG:boto:path=/hiiqos-7/myobj 2023-07-28T23:43:48.162 DEBUG:boto:auth_path=/hiiqos-7/myobj 2023-07-28T23:43:48.163 DEBUG:boto:Method: HEAD 2023-07-28T23:43:48.163 DEBUG:boto:Path: /hiiqos-7/myobj 2023-07-28T23:43:48.163 DEBUG:boto:Data: 2023-07-28T23:43:48.164 DEBUG:boto:Headers: {} 2023-07-28T23:43:48.164 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:43:48.164 DEBUG:boto:Port: 8001 2023-07-28T23:43:48.165 DEBUG:boto:Params: {} 2023-07-28T23:43:48.165 DEBUG:boto:Token: None 2023-07-28T23:43:48.165 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:43:48 GMT /hiiqos-7/myobj 2023-07-28T23:43:48.166 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:a/YvzKULMzbKQ1Be7khNRbdsaK4= 2023-07-28T23:43:48.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:a/YvzKULMzbKQ1Be7khNRbdsaK4=', 'Content-Length': '0'} 2023-07-28T23:43:48.168 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5af072bf5475473-0064c452b4-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:43:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:48.169 DEBUG:boto:path=/hiiqos-7/myobj 2023-07-28T23:43:48.169 DEBUG:boto:auth_path=/hiiqos-7/myobj 2023-07-28T23:43:48.169 DEBUG:boto:Method: HEAD 2023-07-28T23:43:48.170 DEBUG:boto:Path: /hiiqos-7/myobj 2023-07-28T23:43:48.170 DEBUG:boto:Data: 2023-07-28T23:43:48.170 DEBUG:boto:Headers: {} 2023-07-28T23:43:48.171 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:43:48.171 DEBUG:boto:Port: 8000 2023-07-28T23:43:48.171 DEBUG:boto:Params: {} 2023-07-28T23:43:48.172 DEBUG:boto:Token: None 2023-07-28T23:43:48.172 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:43:48 GMT /hiiqos-7/myobj 2023-07-28T23:43:48.173 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:a/YvzKULMzbKQ1Be7khNRbdsaK4= 2023-07-28T23:43:48.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:a/YvzKULMzbKQ1Be7khNRbdsaK4=', 'Content-Length': '0'} 2023-07-28T23:43:48.175 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c5206cfb93278de3-0064c452b4-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:43:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:48.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-07-28T23:43:48.176 DEBUG:boto:path=/hiiqos-7/myobj 2023-07-28T23:43:48.176 DEBUG:boto:auth_path=/hiiqos-7/myobj 2023-07-28T23:43:48.176 DEBUG:boto:Method: GET 2023-07-28T23:43:48.177 DEBUG:boto:Path: /hiiqos-7/myobj 2023-07-28T23:43:48.177 DEBUG:boto:Data: 2023-07-28T23:43:48.177 DEBUG:boto:Headers: {} 2023-07-28T23:43:48.178 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:43:48.178 DEBUG:boto:Port: 8001 2023-07-28T23:43:48.179 DEBUG:boto:Params: {} 2023-07-28T23:43:48.179 DEBUG:boto:Token: None 2023-07-28T23:43:48.179 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:48 GMT /hiiqos-7/myobj 2023-07-28T23:43:48.180 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dpMMzRDByXCvz0/0BoqYpXDvKps= 2023-07-28T23:43:48.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dpMMzRDByXCvz0/0BoqYpXDvKps=', 'Content-Length': '0'} 2023-07-28T23:43:48.182 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001c679e733c0946ac-0064c452b4-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:43:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:48.183 DEBUG:boto:path=/hiiqos-7/myobj 2023-07-28T23:43:48.183 DEBUG:boto:auth_path=/hiiqos-7/myobj 2023-07-28T23:43:48.184 DEBUG:boto:Method: GET 2023-07-28T23:43:48.184 DEBUG:boto:Path: /hiiqos-7/myobj 2023-07-28T23:43:48.184 DEBUG:boto:Data: 2023-07-28T23:43:48.185 DEBUG:boto:Headers: {} 2023-07-28T23:43:48.185 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:43:48.185 DEBUG:boto:Port: 8000 2023-07-28T23:43:48.186 DEBUG:boto:Params: {} 2023-07-28T23:43:48.186 DEBUG:boto:Token: None 2023-07-28T23:43:48.186 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:48 GMT /hiiqos-7/myobj 2023-07-28T23:43:48.187 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dpMMzRDByXCvz0/0BoqYpXDvKps= 2023-07-28T23:43:48.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dpMMzRDByXCvz0/0BoqYpXDvKps=', 'Content-Length': '0'} 2023-07-28T23:43:48.189 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004cf308b91bb88600-0064c452b4-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:43:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:48.232 DEBUG:boto:path=/hiiqos-7/ 2023-07-28T23:43:48.233 DEBUG:boto:auth_path=/hiiqos-7/ 2023-07-28T23:43:48.233 DEBUG:boto:path=/hiiqos-7/?prefix=myobj 2023-07-28T23:43:48.233 DEBUG:boto:auth_path=/hiiqos-7/?prefix=myobj 2023-07-28T23:43:48.234 DEBUG:boto:Method: GET 2023-07-28T23:43:48.234 DEBUG:boto:Path: /hiiqos-7/?prefix=myobj 2023-07-28T23:43:48.235 DEBUG:boto:Data: 2023-07-28T23:43:48.235 DEBUG:boto:Headers: {} 2023-07-28T23:43:48.235 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:43:48.236 DEBUG:boto:Port: 8001 2023-07-28T23:43:48.236 DEBUG:boto:Params: {} 2023-07-28T23:43:48.236 DEBUG:boto:Token: None 2023-07-28T23:43:48.237 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:48 GMT /hiiqos-7/ 2023-07-28T23:43:48.237 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kcdvLvInQS7iHk6TVsRf113GCY4= 2023-07-28T23:43:48.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kcdvLvInQS7iHk6TVsRf113GCY4=', 'Content-Length': '0'} 2023-07-28T23:43:48.240 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc9e20c7707f656c-0064c452b4-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:43:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:48.241 DEBUG:boto:b'hiiqos-7myobj1000falsemyobj2023-07-28T23:42:36.820Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:43:48.242 DEBUG:boto:path=/hiiqos-7/ 2023-07-28T23:43:48.242 DEBUG:boto:auth_path=/hiiqos-7/ 2023-07-28T23:43:48.242 DEBUG:boto:path=/hiiqos-7/?prefix=myobj 2023-07-28T23:43:48.243 DEBUG:boto:auth_path=/hiiqos-7/?prefix=myobj 2023-07-28T23:43:48.243 DEBUG:boto:Method: GET 2023-07-28T23:43:48.243 DEBUG:boto:Path: /hiiqos-7/?prefix=myobj 2023-07-28T23:43:48.244 DEBUG:boto:Data: 2023-07-28T23:43:48.244 DEBUG:boto:Headers: {} 2023-07-28T23:43:48.245 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:43:48.245 DEBUG:boto:Port: 8000 2023-07-28T23:43:48.245 DEBUG:boto:Params: {} 2023-07-28T23:43:48.246 DEBUG:boto:Token: None 2023-07-28T23:43:48.246 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:43:48 GMT /hiiqos-7/ 2023-07-28T23:43:48.247 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kcdvLvInQS7iHk6TVsRf113GCY4= 2023-07-28T23:43:48.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:43:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kcdvLvInQS7iHk6TVsRf113GCY4=', 'Content-Length': '0'} 2023-07-28T23:43:48.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d290e92d0538d67b-0064c452b4-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:43:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:43:48.250 DEBUG:boto:b'hiiqos-7myobj1000falsemyobj2023-07-28T23:42:36.820Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:43:48.251 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-7 zones={a2, a1} 2023-07-28T23:43:48.251 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-8', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:43:48.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-8', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:43:48.252 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-8 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-28T23:43:48.311 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:43:48.311 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:43:48.345 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:43:48.346+0000 7f68bd122880 1 waiting to reach incremental sync.. 2023-07-28T23:43:53.346 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:43:53.346+0000 7f68bd122880 1 waiting to reach incremental sync.. 2023-07-28T23:43:58.347 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:43:58.347+0000 7f68bd122880 1 waiting to reach incremental sync.. 2023-07-28T23:44:03.348 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:44:03.348+0000 7f68bd122880 1 waiting to reach incremental sync.. 2023-07-28T23:44:08.348 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:44:08.349+0000 7f68bd122880 1 waiting to reach incremental sync.. 2023-07-28T23:44:13.349 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:44:13.349+0000 7f68bd122880 1 waiting to reach incremental sync.. 2023-07-28T23:44:18.350 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:44:18.350+0000 7f68bd122880 1 waiting to reach incremental sync.. 2023-07-28T23:44:23.350 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:44:23.350+0000 7f68bd122880 1 waiting to reach incremental sync.. 2023-07-28T23:44:28.351 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:44:28.351+0000 7f68bd122880 1 waiting to reach incremental sync.. 2023-07-28T23:44:33.352 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:44:33.351+0000 7f68bd122880 1 waiting to reach incremental sync.. 2023-07-28T23:44:38.352 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:44:38.352+0000 7f68bd122880 1 waiting to reach incremental sync.. 2023-07-28T23:44:43.353 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:44:43.353+0000 7f68bd122880 1 waiting to reach incremental sync.. 2023-07-28T23:44:48.354 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:44:48.354+0000 7f68bd122880 1 waiting to reach incremental sync.. 2023-07-28T23:44:53.354 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:44:53.354+0000 7f68bd122880 1 waiting to reach incremental sync.. 2023-07-28T23:44:58.355 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:44:58.355+0000 7f68bd122880 1 waiting to reach incremental sync.. 2023-07-28T23:45:03.356 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:45:03.356+0000 7f68bd122880 1 waiting to reach incremental sync.. 2023-07-28T23:45:08.357 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:45:08.356+0000 7f68bd122880 1 waiting to reach incremental sync.. 2023-07-28T23:45:13.358 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:45:13.357+0000 7f68bd122880 1 waiting to reach incremental sync.. 2023-07-28T23:45:18.358 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:45:18.358+0000 7f68bd122880 1 waiting to reach incremental sync.. 2023-07-28T23:45:23.359 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:45:23.359+0000 7f68bd122880 1 waiting to reach incremental sync.. 2023-07-28T23:45:28.360 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:45:28.359+0000 7f68bd122880 1 waiting to reach incremental sync.. 2023-07-28T23:45:33.360 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:45:33.360+0000 7f68bd122880 1 waiting to reach incremental sync.. 2023-07-28T23:45:38.361 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:45:38.361+0000 7f68bd122880 1 waiting to reach incremental sync.. 2023-07-28T23:45:43.362 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:45:43.361+0000 7f68bd122880 1 waiting to reach incremental sync.. 2023-07-28T23:45:48.362 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:45:48.362+0000 7f68bd122880 1 waiting to reach incremental sync.. 2023-07-28T23:45:53.363 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:45:53.362+0000 7f68bd122880 1 waiting to reach incremental sync.. 2023-07-28T23:45:58.365 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:45:58.364+0000 7f68bd122880 1 bucket sync caught up with source: 2023-07-28T23:45:58.365 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000001.17.6, , 00000000001.3.6, , , , 00000000001.6.6, 00000000001.17.6, , ] 2023-07-28T23:45:58.365 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000001.17.6, , 00000000001.3.6, , , , 00000000001.6.6, 00000000001.17.6, , ] 2023-07-28T23:45:58.365 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:45:58.364+0000 7f68bd122880 0 bucket checkpoint complete 2023-07-28T23:45:58.381 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-8 zones={a1, a2} 2023-07-28T23:45:58.382 DEBUG:boto:path=/hiiqos-8/ 2023-07-28T23:45:58.383 DEBUG:boto:auth_path=/hiiqos-8/ 2023-07-28T23:45:58.384 DEBUG:boto:Method: HEAD 2023-07-28T23:45:58.385 DEBUG:boto:Path: /hiiqos-8/ 2023-07-28T23:45:58.386 DEBUG:boto:Data: 2023-07-28T23:45:58.387 DEBUG:boto:Headers: {} 2023-07-28T23:45:58.387 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:45:58.388 DEBUG:boto:Port: 8000 2023-07-28T23:45:58.389 DEBUG:boto:Params: {} 2023-07-28T23:45:58.389 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-28T23:45:58.390 DEBUG:boto:Token: None 2023-07-28T23:45:58.391 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:45:58 GMT /hiiqos-8/ 2023-07-28T23:45:58.392 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TNAeKr+Ny1uW4UymrSEsCyUK1sc= 2023-07-28T23:45:58.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:45:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TNAeKr+Ny1uW4UymrSEsCyUK1sc=', 'Content-Length': '0'} 2023-07-28T23:46:03.398 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f49fe506886e546f-0064c4533b-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:46:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:03.398 DEBUG:boto:path=/hiiqos-8/ 2023-07-28T23:46:03.399 DEBUG:boto:auth_path=/hiiqos-8/ 2023-07-28T23:46:03.400 DEBUG:boto:Method: HEAD 2023-07-28T23:46:03.400 DEBUG:boto:Path: /hiiqos-8/ 2023-07-28T23:46:03.400 DEBUG:boto:Data: 2023-07-28T23:46:03.401 DEBUG:boto:Headers: {} 2023-07-28T23:46:03.401 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:03.401 DEBUG:boto:Port: 8001 2023-07-28T23:46:03.402 DEBUG:boto:Params: {} 2023-07-28T23:46:03.403 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-28T23:46:03.403 DEBUG:boto:Token: None 2023-07-28T23:46:03.403 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:46:03 GMT /hiiqos-8/ 2023-07-28T23:46:03.404 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Jv647E0A4lyu1I8OuJ3akhQtj6A= 2023-07-28T23:46:03.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Jv647E0A4lyu1I8OuJ3akhQtj6A=', 'Content-Length': '0'} 2023-07-28T23:46:08.409 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c8ac9a7754f2e9b9-0064c45340-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.410 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-28T23:46:08.410 DEBUG:boto:path=/hiiqos-8/ 2023-07-28T23:46:08.411 DEBUG:boto:auth_path=/hiiqos-8/ 2023-07-28T23:46:08.411 DEBUG:boto:path=/hiiqos-8/?versions&max-keys=999 2023-07-28T23:46:08.412 DEBUG:boto:auth_path=/hiiqos-8/?versions&max-keys=999 2023-07-28T23:46:08.412 DEBUG:boto:Method: GET 2023-07-28T23:46:08.412 DEBUG:boto:Path: /hiiqos-8/?versions&max-keys=999 2023-07-28T23:46:08.413 DEBUG:boto:Data: 2023-07-28T23:46:08.413 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.413 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:08.414 DEBUG:boto:Port: 8000 2023-07-28T23:46:08.414 DEBUG:boto:Params: {} 2023-07-28T23:46:08.414 DEBUG:boto:Token: None 2023-07-28T23:46:08.415 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/?versions 2023-07-28T23:46:08.415 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gblBqX/+7hJXjzsbzCIPtB/JA7Y= 2023-07-28T23:46:08.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gblBqX/+7hJXjzsbzCIPtB/JA7Y=', 'Content-Length': '0'} 2023-07-28T23:46:08.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e683d0f3960abf78-0064c45340-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.419 DEBUG:boto:b'hiiqos-8999false&nulltrue2023-07-28T23:42:37.079Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-07-28T23:42:37.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-07-28T23:42:37.003Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-07-28T23:42:36.980Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:46:08.420 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-07-28T23:46:08.420 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-07-28T23:46:08.421 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-07-28T23:46:08.421 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-07-28T23:46:08.421 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-28T23:46:08.422 DEBUG:boto:path=/hiiqos-8/ 2023-07-28T23:46:08.422 DEBUG:boto:auth_path=/hiiqos-8/ 2023-07-28T23:46:08.422 DEBUG:boto:path=/hiiqos-8/?versions&max-keys=999 2023-07-28T23:46:08.423 DEBUG:boto:auth_path=/hiiqos-8/?versions&max-keys=999 2023-07-28T23:46:08.423 DEBUG:boto:Method: GET 2023-07-28T23:46:08.424 DEBUG:boto:Path: /hiiqos-8/?versions&max-keys=999 2023-07-28T23:46:08.424 DEBUG:boto:Data: 2023-07-28T23:46:08.424 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.425 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:08.425 DEBUG:boto:Port: 8001 2023-07-28T23:46:08.425 DEBUG:boto:Params: {} 2023-07-28T23:46:08.426 DEBUG:boto:Token: None 2023-07-28T23:46:08.426 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/?versions 2023-07-28T23:46:08.426 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gblBqX/+7hJXjzsbzCIPtB/JA7Y= 2023-07-28T23:46:08.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gblBqX/+7hJXjzsbzCIPtB/JA7Y=', 'Content-Length': '0'} 2023-07-28T23:46:08.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3b87b4c9a6f3345-0064c45340-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.436 DEBUG:boto:b'hiiqos-8999false&nulltrue2023-07-28T23:42:37.079Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-07-28T23:42:37.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-07-28T23:42:37.003Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-07-28T23:42:36.980Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:46:08.437 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-07-28T23:46:08.437 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-07-28T23:46:08.438 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-07-28T23:46:08.438 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-07-28T23:46:08.439 DEBUG:boto:path=/hiiqos-8/ 2023-07-28T23:46:08.439 DEBUG:boto:auth_path=/hiiqos-8/ 2023-07-28T23:46:08.440 DEBUG:boto:path=/hiiqos-8/?versions&max-keys=999 2023-07-28T23:46:08.440 DEBUG:boto:auth_path=/hiiqos-8/?versions&max-keys=999 2023-07-28T23:46:08.441 DEBUG:boto:Method: GET 2023-07-28T23:46:08.441 DEBUG:boto:Path: /hiiqos-8/?versions&max-keys=999 2023-07-28T23:46:08.442 DEBUG:boto:Data: 2023-07-28T23:46:08.443 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.444 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:08.444 DEBUG:boto:Port: 8000 2023-07-28T23:46:08.445 DEBUG:boto:Params: {} 2023-07-28T23:46:08.446 DEBUG:boto:Token: None 2023-07-28T23:46:08.446 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/?versions 2023-07-28T23:46:08.447 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gblBqX/+7hJXjzsbzCIPtB/JA7Y= 2023-07-28T23:46:08.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gblBqX/+7hJXjzsbzCIPtB/JA7Y=', 'Content-Length': '0'} 2023-07-28T23:46:08.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbccac7f25723b3e-0064c45340-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.451 DEBUG:boto:b'hiiqos-8999false&nulltrue2023-07-28T23:42:37.079Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-07-28T23:42:37.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-07-28T23:42:37.003Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-07-28T23:42:36.980Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:46:08.452 DEBUG:boto:path=/hiiqos-8/ 2023-07-28T23:46:08.452 DEBUG:boto:auth_path=/hiiqos-8/ 2023-07-28T23:46:08.453 DEBUG:boto:path=/hiiqos-8/?versions&max-keys=999 2023-07-28T23:46:08.453 DEBUG:boto:auth_path=/hiiqos-8/?versions&max-keys=999 2023-07-28T23:46:08.454 DEBUG:boto:Method: GET 2023-07-28T23:46:08.454 DEBUG:boto:Path: /hiiqos-8/?versions&max-keys=999 2023-07-28T23:46:08.454 DEBUG:boto:Data: 2023-07-28T23:46:08.455 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.455 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:08.455 DEBUG:boto:Port: 8001 2023-07-28T23:46:08.456 DEBUG:boto:Params: {} 2023-07-28T23:46:08.456 DEBUG:boto:Token: None 2023-07-28T23:46:08.457 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/?versions 2023-07-28T23:46:08.457 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gblBqX/+7hJXjzsbzCIPtB/JA7Y= 2023-07-28T23:46:08.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gblBqX/+7hJXjzsbzCIPtB/JA7Y=', 'Content-Length': '0'} 2023-07-28T23:46:08.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bde7efc214bea5b5-0064c45340-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.461 DEBUG:boto:b'hiiqos-8999false&nulltrue2023-07-28T23:42:37.079Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-07-28T23:42:37.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-07-28T23:42:37.003Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-07-28T23:42:36.980Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:46:08.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-07-28T23:46:08.462 DEBUG:boto:path=/hiiqos-8/%26 2023-07-28T23:46:08.463 DEBUG:boto:auth_path=/hiiqos-8/%26 2023-07-28T23:46:08.464 DEBUG:boto:path=/hiiqos-8/%26?versionId=null 2023-07-28T23:46:08.464 DEBUG:boto:auth_path=/hiiqos-8/%26?versionId=null 2023-07-28T23:46:08.464 DEBUG:boto:Method: GET 2023-07-28T23:46:08.465 DEBUG:boto:Path: /hiiqos-8/%26?versionId=null 2023-07-28T23:46:08.465 DEBUG:boto:Data: 2023-07-28T23:46:08.465 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.466 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:08.466 DEBUG:boto:Port: 8000 2023-07-28T23:46:08.466 DEBUG:boto:Params: {} 2023-07-28T23:46:08.467 DEBUG:boto:Token: None 2023-07-28T23:46:08.467 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/%26?versionId=null 2023-07-28T23:46:08.468 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ltA1yzAOouYWtpJCKzJiU8smYoY= 2023-07-28T23:46:08.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ltA1yzAOouYWtpJCKzJiU8smYoY=', 'Content-Length': '0'} 2023-07-28T23:46:08.470 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e87b6f62ddb15a92-0064c45340-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.471 DEBUG:boto:path=/hiiqos-8/%26 2023-07-28T23:46:08.472 DEBUG:boto:auth_path=/hiiqos-8/%26 2023-07-28T23:46:08.472 DEBUG:boto:path=/hiiqos-8/%26?versionId=null 2023-07-28T23:46:08.472 DEBUG:boto:auth_path=/hiiqos-8/%26?versionId=null 2023-07-28T23:46:08.473 DEBUG:boto:Method: GET 2023-07-28T23:46:08.473 DEBUG:boto:Path: /hiiqos-8/%26?versionId=null 2023-07-28T23:46:08.474 DEBUG:boto:Data: 2023-07-28T23:46:08.474 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.474 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:08.475 DEBUG:boto:Port: 8001 2023-07-28T23:46:08.475 DEBUG:boto:Params: {} 2023-07-28T23:46:08.476 DEBUG:boto:Token: None 2023-07-28T23:46:08.476 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/%26?versionId=null 2023-07-28T23:46:08.477 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ltA1yzAOouYWtpJCKzJiU8smYoY= 2023-07-28T23:46:08.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ltA1yzAOouYWtpJCKzJiU8smYoY=', 'Content-Length': '0'} 2023-07-28T23:46:08.479 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c12db11238d4bbd7-0064c45340-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.520 DEBUG:boto:path=/hiiqos-8/%26 2023-07-28T23:46:08.521 DEBUG:boto:auth_path=/hiiqos-8/%26 2023-07-28T23:46:08.521 DEBUG:boto:path=/hiiqos-8/%26?versionId=null 2023-07-28T23:46:08.522 DEBUG:boto:auth_path=/hiiqos-8/%26?versionId=null 2023-07-28T23:46:08.522 DEBUG:boto:Method: HEAD 2023-07-28T23:46:08.522 DEBUG:boto:Path: /hiiqos-8/%26?versionId=null 2023-07-28T23:46:08.523 DEBUG:boto:Data: 2023-07-28T23:46:08.523 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.523 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:08.524 DEBUG:boto:Port: 8000 2023-07-28T23:46:08.524 DEBUG:boto:Params: {} 2023-07-28T23:46:08.524 DEBUG:boto:Token: None 2023-07-28T23:46:08.525 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/%26?versionId=null 2023-07-28T23:46:08.525 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:725R0FXoHCToPLmReeK1C1U76d4= 2023-07-28T23:46:08.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:725R0FXoHCToPLmReeK1C1U76d4=', 'Content-Length': '0'} 2023-07-28T23:46:08.528 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ff30577900c00bde-0064c45340-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.528 DEBUG:boto:path=/hiiqos-8/%26 2023-07-28T23:46:08.529 DEBUG:boto:auth_path=/hiiqos-8/%26 2023-07-28T23:46:08.529 DEBUG:boto:path=/hiiqos-8/%26?versionId=null 2023-07-28T23:46:08.529 DEBUG:boto:auth_path=/hiiqos-8/%26?versionId=null 2023-07-28T23:46:08.530 DEBUG:boto:Method: HEAD 2023-07-28T23:46:08.530 DEBUG:boto:Path: /hiiqos-8/%26?versionId=null 2023-07-28T23:46:08.530 DEBUG:boto:Data: 2023-07-28T23:46:08.531 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.531 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:08.531 DEBUG:boto:Port: 8001 2023-07-28T23:46:08.532 DEBUG:boto:Params: {} 2023-07-28T23:46:08.532 DEBUG:boto:Token: None 2023-07-28T23:46:08.532 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/%26?versionId=null 2023-07-28T23:46:08.533 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:725R0FXoHCToPLmReeK1C1U76d4= 2023-07-28T23:46:08.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:725R0FXoHCToPLmReeK1C1U76d4=', 'Content-Length': '0'} 2023-07-28T23:46:08.535 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a8121ed3db2c12d-0064c45340-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-07-28T23:46:08.536 DEBUG:boto:path=/hiiqos-8/%26 2023-07-28T23:46:08.537 DEBUG:boto:auth_path=/hiiqos-8/%26 2023-07-28T23:46:08.537 DEBUG:boto:path=/hiiqos-8/%26?versionId=null 2023-07-28T23:46:08.537 DEBUG:boto:auth_path=/hiiqos-8/%26?versionId=null 2023-07-28T23:46:08.538 DEBUG:boto:Method: GET 2023-07-28T23:46:08.538 DEBUG:boto:Path: /hiiqos-8/%26?versionId=null 2023-07-28T23:46:08.538 DEBUG:boto:Data: 2023-07-28T23:46:08.539 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.539 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:08.539 DEBUG:boto:Port: 8000 2023-07-28T23:46:08.540 DEBUG:boto:Params: {} 2023-07-28T23:46:08.540 DEBUG:boto:Token: None 2023-07-28T23:46:08.541 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/%26?versionId=null 2023-07-28T23:46:08.541 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ltA1yzAOouYWtpJCKzJiU8smYoY= 2023-07-28T23:46:08.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ltA1yzAOouYWtpJCKzJiU8smYoY=', 'Content-Length': '0'} 2023-07-28T23:46:08.543 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb825fee0ac9e608-0064c45340-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.544 DEBUG:boto:path=/hiiqos-8/%26 2023-07-28T23:46:08.544 DEBUG:boto:auth_path=/hiiqos-8/%26 2023-07-28T23:46:08.545 DEBUG:boto:path=/hiiqos-8/%26?versionId=null 2023-07-28T23:46:08.545 DEBUG:boto:auth_path=/hiiqos-8/%26?versionId=null 2023-07-28T23:46:08.546 DEBUG:boto:Method: GET 2023-07-28T23:46:08.546 DEBUG:boto:Path: /hiiqos-8/%26?versionId=null 2023-07-28T23:46:08.546 DEBUG:boto:Data: 2023-07-28T23:46:08.547 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.547 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:08.547 DEBUG:boto:Port: 8001 2023-07-28T23:46:08.548 DEBUG:boto:Params: {} 2023-07-28T23:46:08.548 DEBUG:boto:Token: None 2023-07-28T23:46:08.548 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/%26?versionId=null 2023-07-28T23:46:08.549 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ltA1yzAOouYWtpJCKzJiU8smYoY= 2023-07-28T23:46:08.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ltA1yzAOouYWtpJCKzJiU8smYoY=', 'Content-Length': '0'} 2023-07-28T23:46:08.551 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000024397558dfa8dd88-0064c45340-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.552 DEBUG:boto:path=/hiiqos-8/ 2023-07-28T23:46:08.552 DEBUG:boto:auth_path=/hiiqos-8/ 2023-07-28T23:46:08.553 DEBUG:boto:path=/hiiqos-8/?prefix=%26 2023-07-28T23:46:08.553 DEBUG:boto:auth_path=/hiiqos-8/?prefix=%26 2023-07-28T23:46:08.553 DEBUG:boto:Method: GET 2023-07-28T23:46:08.554 DEBUG:boto:Path: /hiiqos-8/?prefix=%26 2023-07-28T23:46:08.554 DEBUG:boto:Data: 2023-07-28T23:46:08.554 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.555 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:08.555 DEBUG:boto:Port: 8000 2023-07-28T23:46:08.555 DEBUG:boto:Params: {} 2023-07-28T23:46:08.556 DEBUG:boto:Token: None 2023-07-28T23:46:08.556 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/ 2023-07-28T23:46:08.557 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/xj8ftb+Rbjoj6ijDWM7RpGYdX0= 2023-07-28T23:46:08.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/xj8ftb+Rbjoj6ijDWM7RpGYdX0=', 'Content-Length': '0'} 2023-07-28T23:46:08.560 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006170774f37919606-0064c45340-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.560 DEBUG:boto:b'hiiqos-8&1000false&2023-07-28T23:42:37.079Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:46:08.561 DEBUG:boto:path=/hiiqos-8/ 2023-07-28T23:46:08.562 DEBUG:boto:auth_path=/hiiqos-8/ 2023-07-28T23:46:08.562 DEBUG:boto:path=/hiiqos-8/?prefix=%26 2023-07-28T23:46:08.563 DEBUG:boto:auth_path=/hiiqos-8/?prefix=%26 2023-07-28T23:46:08.564 DEBUG:boto:Method: GET 2023-07-28T23:46:08.565 DEBUG:boto:Path: /hiiqos-8/?prefix=%26 2023-07-28T23:46:08.566 DEBUG:boto:Data: 2023-07-28T23:46:08.567 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.568 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:08.569 DEBUG:boto:Port: 8001 2023-07-28T23:46:08.569 DEBUG:boto:Params: {} 2023-07-28T23:46:08.570 DEBUG:boto:Token: None 2023-07-28T23:46:08.571 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/ 2023-07-28T23:46:08.572 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/xj8ftb+Rbjoj6ijDWM7RpGYdX0= 2023-07-28T23:46:08.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/xj8ftb+Rbjoj6ijDWM7RpGYdX0=', 'Content-Length': '0'} 2023-07-28T23:46:08.575 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dffd1792423c3e17-0064c45340-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.576 DEBUG:boto:b'hiiqos-8&1000false&2023-07-28T23:42:37.079Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:46:08.576 DEBUG:boto:path=/hiiqos-8/%26 2023-07-28T23:46:08.577 DEBUG:boto:auth_path=/hiiqos-8/%26 2023-07-28T23:46:08.577 DEBUG:boto:Method: HEAD 2023-07-28T23:46:08.578 DEBUG:boto:Path: /hiiqos-8/%26 2023-07-28T23:46:08.578 DEBUG:boto:Data: 2023-07-28T23:46:08.578 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.579 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:08.579 DEBUG:boto:Port: 8000 2023-07-28T23:46:08.579 DEBUG:boto:Params: {} 2023-07-28T23:46:08.580 DEBUG:boto:Token: None 2023-07-28T23:46:08.580 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/%26 2023-07-28T23:46:08.581 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xFPTduIfCZOoerPWCw0/B7/OiD4= 2023-07-28T23:46:08.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xFPTduIfCZOoerPWCw0/B7/OiD4=', 'Content-Length': '0'} 2023-07-28T23:46:08.583 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bfdc8cf351e42ae0-0064c45340-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.583 DEBUG:boto:path=/hiiqos-8/%26 2023-07-28T23:46:08.584 DEBUG:boto:auth_path=/hiiqos-8/%26 2023-07-28T23:46:08.584 DEBUG:boto:Method: HEAD 2023-07-28T23:46:08.585 DEBUG:boto:Path: /hiiqos-8/%26 2023-07-28T23:46:08.585 DEBUG:boto:Data: 2023-07-28T23:46:08.585 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.586 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:08.586 DEBUG:boto:Port: 8001 2023-07-28T23:46:08.586 DEBUG:boto:Params: {} 2023-07-28T23:46:08.587 DEBUG:boto:Token: None 2023-07-28T23:46:08.587 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/%26 2023-07-28T23:46:08.588 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xFPTduIfCZOoerPWCw0/B7/OiD4= 2023-07-28T23:46:08.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xFPTduIfCZOoerPWCw0/B7/OiD4=', 'Content-Length': '0'} 2023-07-28T23:46:08.590 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008345d8d5b0284068-0064c45340-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-07-28T23:46:08.591 DEBUG:boto:path=/hiiqos-8/%26 2023-07-28T23:46:08.591 DEBUG:boto:auth_path=/hiiqos-8/%26 2023-07-28T23:46:08.592 DEBUG:boto:Method: GET 2023-07-28T23:46:08.592 DEBUG:boto:Path: /hiiqos-8/%26 2023-07-28T23:46:08.592 DEBUG:boto:Data: 2023-07-28T23:46:08.593 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.593 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:08.594 DEBUG:boto:Port: 8000 2023-07-28T23:46:08.595 DEBUG:boto:Params: {} 2023-07-28T23:46:08.595 DEBUG:boto:Token: None 2023-07-28T23:46:08.596 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/%26 2023-07-28T23:46:08.596 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LV27bEnezVs57LB0GbOgFqaVMro= 2023-07-28T23:46:08.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LV27bEnezVs57LB0GbOgFqaVMro=', 'Content-Length': '0'} 2023-07-28T23:46:08.599 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000254e4d58da374ea4-0064c45340-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.600 DEBUG:boto:path=/hiiqos-8/%26 2023-07-28T23:46:08.601 DEBUG:boto:auth_path=/hiiqos-8/%26 2023-07-28T23:46:08.602 DEBUG:boto:Method: GET 2023-07-28T23:46:08.603 DEBUG:boto:Path: /hiiqos-8/%26 2023-07-28T23:46:08.604 DEBUG:boto:Data: 2023-07-28T23:46:08.605 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.606 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:08.606 DEBUG:boto:Port: 8001 2023-07-28T23:46:08.607 DEBUG:boto:Params: {} 2023-07-28T23:46:08.608 DEBUG:boto:Token: None 2023-07-28T23:46:08.609 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/%26 2023-07-28T23:46:08.609 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LV27bEnezVs57LB0GbOgFqaVMro= 2023-07-28T23:46:08.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LV27bEnezVs57LB0GbOgFqaVMro=', 'Content-Length': '0'} 2023-07-28T23:46:08.612 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000712ca5ba8fb813bd-0064c45340-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.613 DEBUG:boto:path=/hiiqos-8/ 2023-07-28T23:46:08.613 DEBUG:boto:auth_path=/hiiqos-8/ 2023-07-28T23:46:08.613 DEBUG:boto:path=/hiiqos-8/?prefix=%26 2023-07-28T23:46:08.614 DEBUG:boto:auth_path=/hiiqos-8/?prefix=%26 2023-07-28T23:46:08.614 DEBUG:boto:Method: GET 2023-07-28T23:46:08.615 DEBUG:boto:Path: /hiiqos-8/?prefix=%26 2023-07-28T23:46:08.615 DEBUG:boto:Data: 2023-07-28T23:46:08.615 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.616 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:08.616 DEBUG:boto:Port: 8000 2023-07-28T23:46:08.616 DEBUG:boto:Params: {} 2023-07-28T23:46:08.617 DEBUG:boto:Token: None 2023-07-28T23:46:08.617 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/ 2023-07-28T23:46:08.618 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/xj8ftb+Rbjoj6ijDWM7RpGYdX0= 2023-07-28T23:46:08.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/xj8ftb+Rbjoj6ijDWM7RpGYdX0=', 'Content-Length': '0'} 2023-07-28T23:46:08.621 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ff6a2e8d2aab190-0064c45340-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.621 DEBUG:boto:b'hiiqos-8&1000false&2023-07-28T23:42:37.079Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:46:08.622 DEBUG:boto:path=/hiiqos-8/ 2023-07-28T23:46:08.622 DEBUG:boto:auth_path=/hiiqos-8/ 2023-07-28T23:46:08.623 DEBUG:boto:path=/hiiqos-8/?prefix=%26 2023-07-28T23:46:08.623 DEBUG:boto:auth_path=/hiiqos-8/?prefix=%26 2023-07-28T23:46:08.623 DEBUG:boto:Method: GET 2023-07-28T23:46:08.624 DEBUG:boto:Path: /hiiqos-8/?prefix=%26 2023-07-28T23:46:08.624 DEBUG:boto:Data: 2023-07-28T23:46:08.625 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.625 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:08.625 DEBUG:boto:Port: 8001 2023-07-28T23:46:08.626 DEBUG:boto:Params: {} 2023-07-28T23:46:08.626 DEBUG:boto:Token: None 2023-07-28T23:46:08.627 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/ 2023-07-28T23:46:08.627 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/xj8ftb+Rbjoj6ijDWM7RpGYdX0= 2023-07-28T23:46:08.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/xj8ftb+Rbjoj6ijDWM7RpGYdX0=', 'Content-Length': '0'} 2023-07-28T23:46:08.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de64e648eb0560b5-0064c45340-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.631 DEBUG:boto:b'hiiqos-8&1000false&2023-07-28T23:42:37.079Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:46:08.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-07-28T23:46:08.632 DEBUG:boto:path=/hiiqos-8/%3A 2023-07-28T23:46:08.632 DEBUG:boto:auth_path=/hiiqos-8/%3A 2023-07-28T23:46:08.633 DEBUG:boto:path=/hiiqos-8/%3A?versionId=null 2023-07-28T23:46:08.633 DEBUG:boto:auth_path=/hiiqos-8/%3A?versionId=null 2023-07-28T23:46:08.634 DEBUG:boto:Method: GET 2023-07-28T23:46:08.634 DEBUG:boto:Path: /hiiqos-8/%3A?versionId=null 2023-07-28T23:46:08.635 DEBUG:boto:Data: 2023-07-28T23:46:08.635 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.636 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:08.636 DEBUG:boto:Port: 8000 2023-07-28T23:46:08.636 DEBUG:boto:Params: {} 2023-07-28T23:46:08.637 DEBUG:boto:Token: None 2023-07-28T23:46:08.637 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/%3A?versionId=null 2023-07-28T23:46:08.638 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cvGxMZrsMP8mJJnJKSyDH0HYf4U= 2023-07-28T23:46:08.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cvGxMZrsMP8mJJnJKSyDH0HYf4U=', 'Content-Length': '0'} 2023-07-28T23:46:08.640 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000063e63b893105ff50-0064c45340-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.642 DEBUG:boto:path=/hiiqos-8/%3A 2023-07-28T23:46:08.643 DEBUG:boto:auth_path=/hiiqos-8/%3A 2023-07-28T23:46:08.644 DEBUG:boto:path=/hiiqos-8/%3A?versionId=null 2023-07-28T23:46:08.645 DEBUG:boto:auth_path=/hiiqos-8/%3A?versionId=null 2023-07-28T23:46:08.646 DEBUG:boto:Method: GET 2023-07-28T23:46:08.646 DEBUG:boto:Path: /hiiqos-8/%3A?versionId=null 2023-07-28T23:46:08.646 DEBUG:boto:Data: 2023-07-28T23:46:08.647 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.647 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:08.648 DEBUG:boto:Port: 8001 2023-07-28T23:46:08.648 DEBUG:boto:Params: {} 2023-07-28T23:46:08.649 DEBUG:boto:Token: None 2023-07-28T23:46:08.649 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/%3A?versionId=null 2023-07-28T23:46:08.649 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cvGxMZrsMP8mJJnJKSyDH0HYf4U= 2023-07-28T23:46:08.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cvGxMZrsMP8mJJnJKSyDH0HYf4U=', 'Content-Length': '0'} 2023-07-28T23:46:08.652 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000987a2a8497147fbe-0064c45340-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.653 DEBUG:boto:path=/hiiqos-8/%3A 2023-07-28T23:46:08.653 DEBUG:boto:auth_path=/hiiqos-8/%3A 2023-07-28T23:46:08.653 DEBUG:boto:path=/hiiqos-8/%3A?versionId=null 2023-07-28T23:46:08.654 DEBUG:boto:auth_path=/hiiqos-8/%3A?versionId=null 2023-07-28T23:46:08.654 DEBUG:boto:Method: HEAD 2023-07-28T23:46:08.655 DEBUG:boto:Path: /hiiqos-8/%3A?versionId=null 2023-07-28T23:46:08.655 DEBUG:boto:Data: 2023-07-28T23:46:08.655 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.656 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:08.656 DEBUG:boto:Port: 8000 2023-07-28T23:46:08.656 DEBUG:boto:Params: {} 2023-07-28T23:46:08.657 DEBUG:boto:Token: None 2023-07-28T23:46:08.657 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/%3A?versionId=null 2023-07-28T23:46:08.658 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Tl5gnDknQgKKf/o1mqdF/OFMg3M= 2023-07-28T23:46:08.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Tl5gnDknQgKKf/o1mqdF/OFMg3M=', 'Content-Length': '0'} 2023-07-28T23:46:08.660 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002bb40ffc463b42f7-0064c45340-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.661 DEBUG:boto:path=/hiiqos-8/%3A 2023-07-28T23:46:08.661 DEBUG:boto:auth_path=/hiiqos-8/%3A 2023-07-28T23:46:08.661 DEBUG:boto:path=/hiiqos-8/%3A?versionId=null 2023-07-28T23:46:08.662 DEBUG:boto:auth_path=/hiiqos-8/%3A?versionId=null 2023-07-28T23:46:08.662 DEBUG:boto:Method: HEAD 2023-07-28T23:46:08.662 DEBUG:boto:Path: /hiiqos-8/%3A?versionId=null 2023-07-28T23:46:08.663 DEBUG:boto:Data: 2023-07-28T23:46:08.663 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.663 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:08.664 DEBUG:boto:Port: 8001 2023-07-28T23:46:08.664 DEBUG:boto:Params: {} 2023-07-28T23:46:08.665 DEBUG:boto:Token: None 2023-07-28T23:46:08.665 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/%3A?versionId=null 2023-07-28T23:46:08.665 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Tl5gnDknQgKKf/o1mqdF/OFMg3M= 2023-07-28T23:46:08.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Tl5gnDknQgKKf/o1mqdF/OFMg3M=', 'Content-Length': '0'} 2023-07-28T23:46:08.668 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f7f09028c552774f-0064c45340-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-07-28T23:46:08.669 DEBUG:boto:path=/hiiqos-8/%3A 2023-07-28T23:46:08.669 DEBUG:boto:auth_path=/hiiqos-8/%3A 2023-07-28T23:46:08.669 DEBUG:boto:path=/hiiqos-8/%3A?versionId=null 2023-07-28T23:46:08.670 DEBUG:boto:auth_path=/hiiqos-8/%3A?versionId=null 2023-07-28T23:46:08.670 DEBUG:boto:Method: GET 2023-07-28T23:46:08.670 DEBUG:boto:Path: /hiiqos-8/%3A?versionId=null 2023-07-28T23:46:08.671 DEBUG:boto:Data: 2023-07-28T23:46:08.671 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.672 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:08.672 DEBUG:boto:Port: 8000 2023-07-28T23:46:08.673 DEBUG:boto:Params: {} 2023-07-28T23:46:08.673 DEBUG:boto:Token: None 2023-07-28T23:46:08.674 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/%3A?versionId=null 2023-07-28T23:46:08.675 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cvGxMZrsMP8mJJnJKSyDH0HYf4U= 2023-07-28T23:46:08.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cvGxMZrsMP8mJJnJKSyDH0HYf4U=', 'Content-Length': '0'} 2023-07-28T23:46:08.678 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000a315b09b3da7959-0064c45340-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.679 DEBUG:boto:path=/hiiqos-8/%3A 2023-07-28T23:46:08.680 DEBUG:boto:auth_path=/hiiqos-8/%3A 2023-07-28T23:46:08.680 DEBUG:boto:path=/hiiqos-8/%3A?versionId=null 2023-07-28T23:46:08.681 DEBUG:boto:auth_path=/hiiqos-8/%3A?versionId=null 2023-07-28T23:46:08.682 DEBUG:boto:Method: GET 2023-07-28T23:46:08.682 DEBUG:boto:Path: /hiiqos-8/%3A?versionId=null 2023-07-28T23:46:08.683 DEBUG:boto:Data: 2023-07-28T23:46:08.683 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.684 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:08.685 DEBUG:boto:Port: 8001 2023-07-28T23:46:08.685 DEBUG:boto:Params: {} 2023-07-28T23:46:08.686 DEBUG:boto:Token: None 2023-07-28T23:46:08.687 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/%3A?versionId=null 2023-07-28T23:46:08.688 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cvGxMZrsMP8mJJnJKSyDH0HYf4U= 2023-07-28T23:46:08.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cvGxMZrsMP8mJJnJKSyDH0HYf4U=', 'Content-Length': '0'} 2023-07-28T23:46:08.691 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000142d035d5dbc27df-0064c45340-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.692 DEBUG:boto:path=/hiiqos-8/ 2023-07-28T23:46:08.693 DEBUG:boto:auth_path=/hiiqos-8/ 2023-07-28T23:46:08.693 DEBUG:boto:path=/hiiqos-8/?prefix=%3A 2023-07-28T23:46:08.693 DEBUG:boto:auth_path=/hiiqos-8/?prefix=%3A 2023-07-28T23:46:08.694 DEBUG:boto:Method: GET 2023-07-28T23:46:08.694 DEBUG:boto:Path: /hiiqos-8/?prefix=%3A 2023-07-28T23:46:08.694 DEBUG:boto:Data: 2023-07-28T23:46:08.695 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.695 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:08.696 DEBUG:boto:Port: 8000 2023-07-28T23:46:08.696 DEBUG:boto:Params: {} 2023-07-28T23:46:08.696 DEBUG:boto:Token: None 2023-07-28T23:46:08.697 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/ 2023-07-28T23:46:08.697 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/xj8ftb+Rbjoj6ijDWM7RpGYdX0= 2023-07-28T23:46:08.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/xj8ftb+Rbjoj6ijDWM7RpGYdX0=', 'Content-Length': '0'} 2023-07-28T23:46:08.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000681d146ff9deeb69-0064c45340-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.701 DEBUG:boto:b'hiiqos-8:1000false:2023-07-28T23:42:37.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:46:08.701 DEBUG:boto:path=/hiiqos-8/ 2023-07-28T23:46:08.702 DEBUG:boto:auth_path=/hiiqos-8/ 2023-07-28T23:46:08.702 DEBUG:boto:path=/hiiqos-8/?prefix=%3A 2023-07-28T23:46:08.702 DEBUG:boto:auth_path=/hiiqos-8/?prefix=%3A 2023-07-28T23:46:08.703 DEBUG:boto:Method: GET 2023-07-28T23:46:08.703 DEBUG:boto:Path: /hiiqos-8/?prefix=%3A 2023-07-28T23:46:08.704 DEBUG:boto:Data: 2023-07-28T23:46:08.704 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.704 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:08.705 DEBUG:boto:Port: 8001 2023-07-28T23:46:08.705 DEBUG:boto:Params: {} 2023-07-28T23:46:08.706 DEBUG:boto:Token: None 2023-07-28T23:46:08.706 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/ 2023-07-28T23:46:08.706 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/xj8ftb+Rbjoj6ijDWM7RpGYdX0= 2023-07-28T23:46:08.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/xj8ftb+Rbjoj6ijDWM7RpGYdX0=', 'Content-Length': '0'} 2023-07-28T23:46:08.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b43e4a6ffa3e7aa-0064c45340-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.710 DEBUG:boto:b'hiiqos-8:1000false:2023-07-28T23:42:37.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:46:08.710 DEBUG:boto:path=/hiiqos-8/%3A 2023-07-28T23:46:08.711 DEBUG:boto:auth_path=/hiiqos-8/%3A 2023-07-28T23:46:08.711 DEBUG:boto:Method: HEAD 2023-07-28T23:46:08.711 DEBUG:boto:Path: /hiiqos-8/%3A 2023-07-28T23:46:08.712 DEBUG:boto:Data: 2023-07-28T23:46:08.712 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.713 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:08.713 DEBUG:boto:Port: 8000 2023-07-28T23:46:08.713 DEBUG:boto:Params: {} 2023-07-28T23:46:08.714 DEBUG:boto:Token: None 2023-07-28T23:46:08.714 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/%3A 2023-07-28T23:46:08.715 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cUbea4uguBkVP7v+2ySpZBpejyw= 2023-07-28T23:46:08.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cUbea4uguBkVP7v+2ySpZBpejyw=', 'Content-Length': '0'} 2023-07-28T23:46:08.717 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e76ca69f7eaf881f-0064c45340-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.717 DEBUG:boto:path=/hiiqos-8/%3A 2023-07-28T23:46:08.718 DEBUG:boto:auth_path=/hiiqos-8/%3A 2023-07-28T23:46:08.718 DEBUG:boto:Method: HEAD 2023-07-28T23:46:08.719 DEBUG:boto:Path: /hiiqos-8/%3A 2023-07-28T23:46:08.719 DEBUG:boto:Data: 2023-07-28T23:46:08.719 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.720 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:08.720 DEBUG:boto:Port: 8001 2023-07-28T23:46:08.721 DEBUG:boto:Params: {} 2023-07-28T23:46:08.721 DEBUG:boto:Token: None 2023-07-28T23:46:08.721 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/%3A 2023-07-28T23:46:08.722 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cUbea4uguBkVP7v+2ySpZBpejyw= 2023-07-28T23:46:08.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cUbea4uguBkVP7v+2ySpZBpejyw=', 'Content-Length': '0'} 2023-07-28T23:46:08.724 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ef7b0d8f3185e01d-0064c45340-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-07-28T23:46:08.725 DEBUG:boto:path=/hiiqos-8/%3A 2023-07-28T23:46:08.725 DEBUG:boto:auth_path=/hiiqos-8/%3A 2023-07-28T23:46:08.726 DEBUG:boto:Method: GET 2023-07-28T23:46:08.726 DEBUG:boto:Path: /hiiqos-8/%3A 2023-07-28T23:46:08.727 DEBUG:boto:Data: 2023-07-28T23:46:08.727 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.727 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:08.728 DEBUG:boto:Port: 8000 2023-07-28T23:46:08.728 DEBUG:boto:Params: {} 2023-07-28T23:46:08.729 DEBUG:boto:Token: None 2023-07-28T23:46:08.729 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/%3A 2023-07-28T23:46:08.730 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6QFxl+3JbAk4w17yOEYDYob1e/A= 2023-07-28T23:46:08.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6QFxl+3JbAk4w17yOEYDYob1e/A=', 'Content-Length': '0'} 2023-07-28T23:46:08.732 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000881c715ce63db30a-0064c45340-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.772 DEBUG:boto:path=/hiiqos-8/%3A 2023-07-28T23:46:08.773 DEBUG:boto:auth_path=/hiiqos-8/%3A 2023-07-28T23:46:08.773 DEBUG:boto:Method: GET 2023-07-28T23:46:08.774 DEBUG:boto:Path: /hiiqos-8/%3A 2023-07-28T23:46:08.774 DEBUG:boto:Data: 2023-07-28T23:46:08.774 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.775 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:08.775 DEBUG:boto:Port: 8001 2023-07-28T23:46:08.776 DEBUG:boto:Params: {} 2023-07-28T23:46:08.776 DEBUG:boto:Token: None 2023-07-28T23:46:08.777 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/%3A 2023-07-28T23:46:08.777 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6QFxl+3JbAk4w17yOEYDYob1e/A= 2023-07-28T23:46:08.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6QFxl+3JbAk4w17yOEYDYob1e/A=', 'Content-Length': '0'} 2023-07-28T23:46:08.780 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f92190ac30bfe90b-0064c45340-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.780 DEBUG:boto:path=/hiiqos-8/ 2023-07-28T23:46:08.781 DEBUG:boto:auth_path=/hiiqos-8/ 2023-07-28T23:46:08.781 DEBUG:boto:path=/hiiqos-8/?prefix=%3A 2023-07-28T23:46:08.781 DEBUG:boto:auth_path=/hiiqos-8/?prefix=%3A 2023-07-28T23:46:08.782 DEBUG:boto:Method: GET 2023-07-28T23:46:08.782 DEBUG:boto:Path: /hiiqos-8/?prefix=%3A 2023-07-28T23:46:08.783 DEBUG:boto:Data: 2023-07-28T23:46:08.783 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.783 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:08.784 DEBUG:boto:Port: 8000 2023-07-28T23:46:08.784 DEBUG:boto:Params: {} 2023-07-28T23:46:08.785 DEBUG:boto:Token: None 2023-07-28T23:46:08.785 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/ 2023-07-28T23:46:08.786 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/xj8ftb+Rbjoj6ijDWM7RpGYdX0= 2023-07-28T23:46:08.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/xj8ftb+Rbjoj6ijDWM7RpGYdX0=', 'Content-Length': '0'} 2023-07-28T23:46:08.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053738195bab4938e-0064c45340-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.789 DEBUG:boto:b'hiiqos-8:1000false:2023-07-28T23:42:37.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:46:08.790 DEBUG:boto:path=/hiiqos-8/ 2023-07-28T23:46:08.790 DEBUG:boto:auth_path=/hiiqos-8/ 2023-07-28T23:46:08.790 DEBUG:boto:path=/hiiqos-8/?prefix=%3A 2023-07-28T23:46:08.791 DEBUG:boto:auth_path=/hiiqos-8/?prefix=%3A 2023-07-28T23:46:08.791 DEBUG:boto:Method: GET 2023-07-28T23:46:08.792 DEBUG:boto:Path: /hiiqos-8/?prefix=%3A 2023-07-28T23:46:08.792 DEBUG:boto:Data: 2023-07-28T23:46:08.792 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.793 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:08.793 DEBUG:boto:Port: 8001 2023-07-28T23:46:08.794 DEBUG:boto:Params: {} 2023-07-28T23:46:08.794 DEBUG:boto:Token: None 2023-07-28T23:46:08.794 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/ 2023-07-28T23:46:08.795 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/xj8ftb+Rbjoj6ijDWM7RpGYdX0= 2023-07-28T23:46:08.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/xj8ftb+Rbjoj6ijDWM7RpGYdX0=', 'Content-Length': '0'} 2023-07-28T23:46:08.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000109506638d5915eb-0064c45340-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.798 DEBUG:boto:b'hiiqos-8:1000false:2023-07-28T23:42:37.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:46:08.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-07-28T23:46:08.799 DEBUG:boto:path=/hiiqos-8/_myobj 2023-07-28T23:46:08.800 DEBUG:boto:auth_path=/hiiqos-8/_myobj 2023-07-28T23:46:08.800 DEBUG:boto:path=/hiiqos-8/_myobj?versionId=null 2023-07-28T23:46:08.800 DEBUG:boto:auth_path=/hiiqos-8/_myobj?versionId=null 2023-07-28T23:46:08.801 DEBUG:boto:Method: GET 2023-07-28T23:46:08.801 DEBUG:boto:Path: /hiiqos-8/_myobj?versionId=null 2023-07-28T23:46:08.801 DEBUG:boto:Data: 2023-07-28T23:46:08.802 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.802 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:08.803 DEBUG:boto:Port: 8000 2023-07-28T23:46:08.803 DEBUG:boto:Params: {} 2023-07-28T23:46:08.803 DEBUG:boto:Token: None 2023-07-28T23:46:08.804 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/_myobj?versionId=null 2023-07-28T23:46:08.804 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZcbZ0MMiwpBbl91Io4WsRitJ4UI= 2023-07-28T23:46:08.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZcbZ0MMiwpBbl91Io4WsRitJ4UI=', 'Content-Length': '0'} 2023-07-28T23:46:08.807 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000795dd8191ab7489b-0064c45340-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.808 DEBUG:boto:path=/hiiqos-8/_myobj 2023-07-28T23:46:08.808 DEBUG:boto:auth_path=/hiiqos-8/_myobj 2023-07-28T23:46:08.809 DEBUG:boto:path=/hiiqos-8/_myobj?versionId=null 2023-07-28T23:46:08.809 DEBUG:boto:auth_path=/hiiqos-8/_myobj?versionId=null 2023-07-28T23:46:08.810 DEBUG:boto:Method: GET 2023-07-28T23:46:08.810 DEBUG:boto:Path: /hiiqos-8/_myobj?versionId=null 2023-07-28T23:46:08.810 DEBUG:boto:Data: 2023-07-28T23:46:08.811 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.811 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:08.811 DEBUG:boto:Port: 8001 2023-07-28T23:46:08.812 DEBUG:boto:Params: {} 2023-07-28T23:46:08.812 DEBUG:boto:Token: None 2023-07-28T23:46:08.813 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/_myobj?versionId=null 2023-07-28T23:46:08.813 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZcbZ0MMiwpBbl91Io4WsRitJ4UI= 2023-07-28T23:46:08.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZcbZ0MMiwpBbl91Io4WsRitJ4UI=', 'Content-Length': '0'} 2023-07-28T23:46:08.815 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004a7632f3823ea7dc-0064c45340-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.860 DEBUG:boto:path=/hiiqos-8/_myobj 2023-07-28T23:46:08.861 DEBUG:boto:auth_path=/hiiqos-8/_myobj 2023-07-28T23:46:08.861 DEBUG:boto:path=/hiiqos-8/_myobj?versionId=null 2023-07-28T23:46:08.862 DEBUG:boto:auth_path=/hiiqos-8/_myobj?versionId=null 2023-07-28T23:46:08.862 DEBUG:boto:Method: HEAD 2023-07-28T23:46:08.862 DEBUG:boto:Path: /hiiqos-8/_myobj?versionId=null 2023-07-28T23:46:08.863 DEBUG:boto:Data: 2023-07-28T23:46:08.863 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.863 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:08.864 DEBUG:boto:Port: 8000 2023-07-28T23:46:08.864 DEBUG:boto:Params: {} 2023-07-28T23:46:08.864 DEBUG:boto:Token: None 2023-07-28T23:46:08.865 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/_myobj?versionId=null 2023-07-28T23:46:08.865 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sxXFTaAuYBi4sHd6R4LkSuPBm9k= 2023-07-28T23:46:08.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sxXFTaAuYBi4sHd6R4LkSuPBm9k=', 'Content-Length': '0'} 2023-07-28T23:46:08.868 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f20780594cdcb0c4-0064c45340-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.868 DEBUG:boto:path=/hiiqos-8/_myobj 2023-07-28T23:46:08.869 DEBUG:boto:auth_path=/hiiqos-8/_myobj 2023-07-28T23:46:08.869 DEBUG:boto:path=/hiiqos-8/_myobj?versionId=null 2023-07-28T23:46:08.870 DEBUG:boto:auth_path=/hiiqos-8/_myobj?versionId=null 2023-07-28T23:46:08.870 DEBUG:boto:Method: HEAD 2023-07-28T23:46:08.870 DEBUG:boto:Path: /hiiqos-8/_myobj?versionId=null 2023-07-28T23:46:08.871 DEBUG:boto:Data: 2023-07-28T23:46:08.871 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.871 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:08.872 DEBUG:boto:Port: 8001 2023-07-28T23:46:08.872 DEBUG:boto:Params: {} 2023-07-28T23:46:08.872 DEBUG:boto:Token: None 2023-07-28T23:46:08.873 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/_myobj?versionId=null 2023-07-28T23:46:08.873 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sxXFTaAuYBi4sHd6R4LkSuPBm9k= 2023-07-28T23:46:08.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sxXFTaAuYBi4sHd6R4LkSuPBm9k=', 'Content-Length': '0'} 2023-07-28T23:46:08.876 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007fbc0fd22877159d-0064c45340-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-07-28T23:46:08.877 DEBUG:boto:path=/hiiqos-8/_myobj 2023-07-28T23:46:08.877 DEBUG:boto:auth_path=/hiiqos-8/_myobj 2023-07-28T23:46:08.877 DEBUG:boto:path=/hiiqos-8/_myobj?versionId=null 2023-07-28T23:46:08.878 DEBUG:boto:auth_path=/hiiqos-8/_myobj?versionId=null 2023-07-28T23:46:08.878 DEBUG:boto:Method: GET 2023-07-28T23:46:08.879 DEBUG:boto:Path: /hiiqos-8/_myobj?versionId=null 2023-07-28T23:46:08.879 DEBUG:boto:Data: 2023-07-28T23:46:08.879 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.880 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:08.880 DEBUG:boto:Port: 8000 2023-07-28T23:46:08.880 DEBUG:boto:Params: {} 2023-07-28T23:46:08.881 DEBUG:boto:Token: None 2023-07-28T23:46:08.881 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/_myobj?versionId=null 2023-07-28T23:46:08.882 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZcbZ0MMiwpBbl91Io4WsRitJ4UI= 2023-07-28T23:46:08.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZcbZ0MMiwpBbl91Io4WsRitJ4UI=', 'Content-Length': '0'} 2023-07-28T23:46:08.884 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e9327317b97e0ca0-0064c45340-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.885 DEBUG:boto:path=/hiiqos-8/_myobj 2023-07-28T23:46:08.885 DEBUG:boto:auth_path=/hiiqos-8/_myobj 2023-07-28T23:46:08.885 DEBUG:boto:path=/hiiqos-8/_myobj?versionId=null 2023-07-28T23:46:08.886 DEBUG:boto:auth_path=/hiiqos-8/_myobj?versionId=null 2023-07-28T23:46:08.886 DEBUG:boto:Method: GET 2023-07-28T23:46:08.886 DEBUG:boto:Path: /hiiqos-8/_myobj?versionId=null 2023-07-28T23:46:08.887 DEBUG:boto:Data: 2023-07-28T23:46:08.887 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.888 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:08.888 DEBUG:boto:Port: 8001 2023-07-28T23:46:08.888 DEBUG:boto:Params: {} 2023-07-28T23:46:08.889 DEBUG:boto:Token: None 2023-07-28T23:46:08.889 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/_myobj?versionId=null 2023-07-28T23:46:08.890 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZcbZ0MMiwpBbl91Io4WsRitJ4UI= 2023-07-28T23:46:08.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZcbZ0MMiwpBbl91Io4WsRitJ4UI=', 'Content-Length': '0'} 2023-07-28T23:46:08.892 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:37 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a53fee437b80f36-0064c45340-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.893 DEBUG:boto:path=/hiiqos-8/ 2023-07-28T23:46:08.894 DEBUG:boto:auth_path=/hiiqos-8/ 2023-07-28T23:46:08.894 DEBUG:boto:path=/hiiqos-8/?prefix=_myobj 2023-07-28T23:46:08.894 DEBUG:boto:auth_path=/hiiqos-8/?prefix=_myobj 2023-07-28T23:46:08.895 DEBUG:boto:Method: GET 2023-07-28T23:46:08.895 DEBUG:boto:Path: /hiiqos-8/?prefix=_myobj 2023-07-28T23:46:08.895 DEBUG:boto:Data: 2023-07-28T23:46:08.896 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.896 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:08.897 DEBUG:boto:Port: 8000 2023-07-28T23:46:08.897 DEBUG:boto:Params: {} 2023-07-28T23:46:08.897 DEBUG:boto:Token: None 2023-07-28T23:46:08.898 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/ 2023-07-28T23:46:08.898 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/xj8ftb+Rbjoj6ijDWM7RpGYdX0= 2023-07-28T23:46:08.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/xj8ftb+Rbjoj6ijDWM7RpGYdX0=', 'Content-Length': '0'} 2023-07-28T23:46:08.901 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007eaf3984c91779a2-0064c45340-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.901 DEBUG:boto:b'hiiqos-8_myobj1000false_myobj2023-07-28T23:42:37.003Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:46:08.902 DEBUG:boto:path=/hiiqos-8/ 2023-07-28T23:46:08.903 DEBUG:boto:auth_path=/hiiqos-8/ 2023-07-28T23:46:08.903 DEBUG:boto:path=/hiiqos-8/?prefix=_myobj 2023-07-28T23:46:08.903 DEBUG:boto:auth_path=/hiiqos-8/?prefix=_myobj 2023-07-28T23:46:08.904 DEBUG:boto:Method: GET 2023-07-28T23:46:08.904 DEBUG:boto:Path: /hiiqos-8/?prefix=_myobj 2023-07-28T23:46:08.905 DEBUG:boto:Data: 2023-07-28T23:46:08.905 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.905 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:08.906 DEBUG:boto:Port: 8001 2023-07-28T23:46:08.906 DEBUG:boto:Params: {} 2023-07-28T23:46:08.906 DEBUG:boto:Token: None 2023-07-28T23:46:08.907 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/ 2023-07-28T23:46:08.907 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/xj8ftb+Rbjoj6ijDWM7RpGYdX0= 2023-07-28T23:46:08.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/xj8ftb+Rbjoj6ijDWM7RpGYdX0=', 'Content-Length': '0'} 2023-07-28T23:46:08.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d525a2395ee7bc3e-0064c45340-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.910 DEBUG:boto:b'hiiqos-8_myobj1000false_myobj2023-07-28T23:42:37.003Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:46:08.911 DEBUG:boto:path=/hiiqos-8/_myobj 2023-07-28T23:46:08.911 DEBUG:boto:auth_path=/hiiqos-8/_myobj 2023-07-28T23:46:08.912 DEBUG:boto:Method: HEAD 2023-07-28T23:46:08.912 DEBUG:boto:Path: /hiiqos-8/_myobj 2023-07-28T23:46:08.913 DEBUG:boto:Data: 2023-07-28T23:46:08.913 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.913 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:08.914 DEBUG:boto:Port: 8000 2023-07-28T23:46:08.914 DEBUG:boto:Params: {} 2023-07-28T23:46:08.914 DEBUG:boto:Token: None 2023-07-28T23:46:08.915 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/_myobj 2023-07-28T23:46:08.915 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kqBuhYftfEAl1NLSiHaXvMzPXxo= 2023-07-28T23:46:08.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kqBuhYftfEAl1NLSiHaXvMzPXxo=', 'Content-Length': '0'} 2023-07-28T23:46:08.917 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c2dc3ac505bf244f-0064c45340-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.918 DEBUG:boto:path=/hiiqos-8/_myobj 2023-07-28T23:46:08.918 DEBUG:boto:auth_path=/hiiqos-8/_myobj 2023-07-28T23:46:08.919 DEBUG:boto:Method: HEAD 2023-07-28T23:46:08.919 DEBUG:boto:Path: /hiiqos-8/_myobj 2023-07-28T23:46:08.919 DEBUG:boto:Data: 2023-07-28T23:46:08.920 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.920 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:08.921 DEBUG:boto:Port: 8001 2023-07-28T23:46:08.921 DEBUG:boto:Params: {} 2023-07-28T23:46:08.922 DEBUG:boto:Token: None 2023-07-28T23:46:08.922 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/_myobj 2023-07-28T23:46:08.922 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kqBuhYftfEAl1NLSiHaXvMzPXxo= 2023-07-28T23:46:08.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kqBuhYftfEAl1NLSiHaXvMzPXxo=', 'Content-Length': '0'} 2023-07-28T23:46:08.924 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9e7275905f6b69a-0064c45340-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-07-28T23:46:08.925 DEBUG:boto:path=/hiiqos-8/_myobj 2023-07-28T23:46:08.926 DEBUG:boto:auth_path=/hiiqos-8/_myobj 2023-07-28T23:46:08.926 DEBUG:boto:Method: GET 2023-07-28T23:46:08.927 DEBUG:boto:Path: /hiiqos-8/_myobj 2023-07-28T23:46:08.927 DEBUG:boto:Data: 2023-07-28T23:46:08.927 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.928 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:08.928 DEBUG:boto:Port: 8000 2023-07-28T23:46:08.928 DEBUG:boto:Params: {} 2023-07-28T23:46:08.929 DEBUG:boto:Token: None 2023-07-28T23:46:08.929 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/_myobj 2023-07-28T23:46:08.930 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+AIMMiudMjb/b/4xf9oZtvAjkz4= 2023-07-28T23:46:08.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+AIMMiudMjb/b/4xf9oZtvAjkz4=', 'Content-Length': '0'} 2023-07-28T23:46:08.932 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d166c9a42108ec6-0064c45340-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.976 DEBUG:boto:path=/hiiqos-8/_myobj 2023-07-28T23:46:08.977 DEBUG:boto:auth_path=/hiiqos-8/_myobj 2023-07-28T23:46:08.978 DEBUG:boto:Method: GET 2023-07-28T23:46:08.978 DEBUG:boto:Path: /hiiqos-8/_myobj 2023-07-28T23:46:08.979 DEBUG:boto:Data: 2023-07-28T23:46:08.979 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.980 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:08.981 DEBUG:boto:Port: 8001 2023-07-28T23:46:08.982 DEBUG:boto:Params: {} 2023-07-28T23:46:08.982 DEBUG:boto:Token: None 2023-07-28T23:46:08.983 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/_myobj 2023-07-28T23:46:08.984 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+AIMMiudMjb/b/4xf9oZtvAjkz4= 2023-07-28T23:46:08.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+AIMMiudMjb/b/4xf9oZtvAjkz4=', 'Content-Length': '0'} 2023-07-28T23:46:08.988 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:37 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eb1601ba37427109-0064c45340-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.988 DEBUG:boto:path=/hiiqos-8/ 2023-07-28T23:46:08.989 DEBUG:boto:auth_path=/hiiqos-8/ 2023-07-28T23:46:08.989 DEBUG:boto:path=/hiiqos-8/?prefix=_myobj 2023-07-28T23:46:08.989 DEBUG:boto:auth_path=/hiiqos-8/?prefix=_myobj 2023-07-28T23:46:08.990 DEBUG:boto:Method: GET 2023-07-28T23:46:08.990 DEBUG:boto:Path: /hiiqos-8/?prefix=_myobj 2023-07-28T23:46:08.991 DEBUG:boto:Data: 2023-07-28T23:46:08.991 DEBUG:boto:Headers: {} 2023-07-28T23:46:08.991 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:08.992 DEBUG:boto:Port: 8000 2023-07-28T23:46:08.992 DEBUG:boto:Params: {} 2023-07-28T23:46:08.992 DEBUG:boto:Token: None 2023-07-28T23:46:08.993 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:08 GMT /hiiqos-8/ 2023-07-28T23:46:08.994 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/xj8ftb+Rbjoj6ijDWM7RpGYdX0= 2023-07-28T23:46:08.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/xj8ftb+Rbjoj6ijDWM7RpGYdX0=', 'Content-Length': '0'} 2023-07-28T23:46:08.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008daee838756666b4-0064c45340-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:08.997 DEBUG:boto:b'hiiqos-8_myobj1000false_myobj2023-07-28T23:42:37.003Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:46:08.998 DEBUG:boto:path=/hiiqos-8/ 2023-07-28T23:46:08.998 DEBUG:boto:auth_path=/hiiqos-8/ 2023-07-28T23:46:08.999 DEBUG:boto:path=/hiiqos-8/?prefix=_myobj 2023-07-28T23:46:08.999 DEBUG:boto:auth_path=/hiiqos-8/?prefix=_myobj 2023-07-28T23:46:08.999 DEBUG:boto:Method: GET 2023-07-28T23:46:09.000 DEBUG:boto:Path: /hiiqos-8/?prefix=_myobj 2023-07-28T23:46:09.000 DEBUG:boto:Data: 2023-07-28T23:46:09.000 DEBUG:boto:Headers: {} 2023-07-28T23:46:09.001 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:09.001 DEBUG:boto:Port: 8001 2023-07-28T23:46:09.002 DEBUG:boto:Params: {} 2023-07-28T23:46:09.002 DEBUG:boto:Token: None 2023-07-28T23:46:09.002 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:09 GMT /hiiqos-8/ 2023-07-28T23:46:09.003 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yqTWppJINh673TgdifBMbrGzFyc= 2023-07-28T23:46:09.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yqTWppJINh673TgdifBMbrGzFyc=', 'Content-Length': '0'} 2023-07-28T23:46:09.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086bac93786f7e648-0064c45341-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:09.007 DEBUG:boto:b'hiiqos-8_myobj1000false_myobj2023-07-28T23:42:37.003Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:46:09.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-07-28T23:46:09.008 DEBUG:boto:path=/hiiqos-8/myobj 2023-07-28T23:46:09.009 DEBUG:boto:auth_path=/hiiqos-8/myobj 2023-07-28T23:46:09.009 DEBUG:boto:path=/hiiqos-8/myobj?versionId=null 2023-07-28T23:46:09.009 DEBUG:boto:auth_path=/hiiqos-8/myobj?versionId=null 2023-07-28T23:46:09.010 DEBUG:boto:Method: GET 2023-07-28T23:46:09.010 DEBUG:boto:Path: /hiiqos-8/myobj?versionId=null 2023-07-28T23:46:09.011 DEBUG:boto:Data: 2023-07-28T23:46:09.011 DEBUG:boto:Headers: {} 2023-07-28T23:46:09.011 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:09.012 DEBUG:boto:Port: 8000 2023-07-28T23:46:09.012 DEBUG:boto:Params: {} 2023-07-28T23:46:09.013 DEBUG:boto:Token: None 2023-07-28T23:46:09.014 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:09 GMT /hiiqos-8/myobj?versionId=null 2023-07-28T23:46:09.014 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ufTVhXWJUxDDRd80T+5WcLiQBTU= 2023-07-28T23:46:09.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ufTVhXWJUxDDRd80T+5WcLiQBTU=', 'Content-Length': '0'} 2023-07-28T23:46:09.016 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000428af2d9d670ca38-0064c45341-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:09.017 DEBUG:boto:path=/hiiqos-8/myobj 2023-07-28T23:46:09.018 DEBUG:boto:auth_path=/hiiqos-8/myobj 2023-07-28T23:46:09.018 DEBUG:boto:path=/hiiqos-8/myobj?versionId=null 2023-07-28T23:46:09.018 DEBUG:boto:auth_path=/hiiqos-8/myobj?versionId=null 2023-07-28T23:46:09.019 DEBUG:boto:Method: GET 2023-07-28T23:46:09.019 DEBUG:boto:Path: /hiiqos-8/myobj?versionId=null 2023-07-28T23:46:09.019 DEBUG:boto:Data: 2023-07-28T23:46:09.020 DEBUG:boto:Headers: {} 2023-07-28T23:46:09.020 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:09.020 DEBUG:boto:Port: 8001 2023-07-28T23:46:09.021 DEBUG:boto:Params: {} 2023-07-28T23:46:09.021 DEBUG:boto:Token: None 2023-07-28T23:46:09.022 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:09 GMT /hiiqos-8/myobj?versionId=null 2023-07-28T23:46:09.022 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ufTVhXWJUxDDRd80T+5WcLiQBTU= 2023-07-28T23:46:09.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ufTVhXWJUxDDRd80T+5WcLiQBTU=', 'Content-Length': '0'} 2023-07-28T23:46:09.024 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d694fee8c58656f2-0064c45341-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:09.025 DEBUG:boto:path=/hiiqos-8/myobj 2023-07-28T23:46:09.025 DEBUG:boto:auth_path=/hiiqos-8/myobj 2023-07-28T23:46:09.026 DEBUG:boto:path=/hiiqos-8/myobj?versionId=null 2023-07-28T23:46:09.026 DEBUG:boto:auth_path=/hiiqos-8/myobj?versionId=null 2023-07-28T23:46:09.026 DEBUG:boto:Method: HEAD 2023-07-28T23:46:09.027 DEBUG:boto:Path: /hiiqos-8/myobj?versionId=null 2023-07-28T23:46:09.027 DEBUG:boto:Data: 2023-07-28T23:46:09.027 DEBUG:boto:Headers: {} 2023-07-28T23:46:09.028 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:09.028 DEBUG:boto:Port: 8000 2023-07-28T23:46:09.028 DEBUG:boto:Params: {} 2023-07-28T23:46:09.029 DEBUG:boto:Token: None 2023-07-28T23:46:09.029 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:46:09 GMT /hiiqos-8/myobj?versionId=null 2023-07-28T23:46:09.030 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:NEg3MX5mxhe84vZW/iQfIQAS8M4= 2023-07-28T23:46:09.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:NEg3MX5mxhe84vZW/iQfIQAS8M4=', 'Content-Length': '0'} 2023-07-28T23:46:09.032 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000085e9986e3e88aeb2-0064c45341-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:09.033 DEBUG:boto:path=/hiiqos-8/myobj 2023-07-28T23:46:09.033 DEBUG:boto:auth_path=/hiiqos-8/myobj 2023-07-28T23:46:09.033 DEBUG:boto:path=/hiiqos-8/myobj?versionId=null 2023-07-28T23:46:09.034 DEBUG:boto:auth_path=/hiiqos-8/myobj?versionId=null 2023-07-28T23:46:09.034 DEBUG:boto:Method: HEAD 2023-07-28T23:46:09.034 DEBUG:boto:Path: /hiiqos-8/myobj?versionId=null 2023-07-28T23:46:09.035 DEBUG:boto:Data: 2023-07-28T23:46:09.035 DEBUG:boto:Headers: {} 2023-07-28T23:46:09.036 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:09.036 DEBUG:boto:Port: 8001 2023-07-28T23:46:09.037 DEBUG:boto:Params: {} 2023-07-28T23:46:09.037 DEBUG:boto:Token: None 2023-07-28T23:46:09.037 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:46:09 GMT /hiiqos-8/myobj?versionId=null 2023-07-28T23:46:09.038 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:NEg3MX5mxhe84vZW/iQfIQAS8M4= 2023-07-28T23:46:09.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:NEg3MX5mxhe84vZW/iQfIQAS8M4=', 'Content-Length': '0'} 2023-07-28T23:46:09.040 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000061e84721a98c60da-0064c45341-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:09.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-07-28T23:46:09.041 DEBUG:boto:path=/hiiqos-8/myobj 2023-07-28T23:46:09.042 DEBUG:boto:auth_path=/hiiqos-8/myobj 2023-07-28T23:46:09.042 DEBUG:boto:path=/hiiqos-8/myobj?versionId=null 2023-07-28T23:46:09.042 DEBUG:boto:auth_path=/hiiqos-8/myobj?versionId=null 2023-07-28T23:46:09.043 DEBUG:boto:Method: GET 2023-07-28T23:46:09.043 DEBUG:boto:Path: /hiiqos-8/myobj?versionId=null 2023-07-28T23:46:09.043 DEBUG:boto:Data: 2023-07-28T23:46:09.044 DEBUG:boto:Headers: {} 2023-07-28T23:46:09.044 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:09.044 DEBUG:boto:Port: 8000 2023-07-28T23:46:09.045 DEBUG:boto:Params: {} 2023-07-28T23:46:09.045 DEBUG:boto:Token: None 2023-07-28T23:46:09.046 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:09 GMT /hiiqos-8/myobj?versionId=null 2023-07-28T23:46:09.046 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ufTVhXWJUxDDRd80T+5WcLiQBTU= 2023-07-28T23:46:09.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ufTVhXWJUxDDRd80T+5WcLiQBTU=', 'Content-Length': '0'} 2023-07-28T23:46:09.049 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f98a28b2d4c72ed7-0064c45341-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:09.092 DEBUG:boto:path=/hiiqos-8/myobj 2023-07-28T23:46:09.093 DEBUG:boto:auth_path=/hiiqos-8/myobj 2023-07-28T23:46:09.093 DEBUG:boto:path=/hiiqos-8/myobj?versionId=null 2023-07-28T23:46:09.094 DEBUG:boto:auth_path=/hiiqos-8/myobj?versionId=null 2023-07-28T23:46:09.094 DEBUG:boto:Method: GET 2023-07-28T23:46:09.094 DEBUG:boto:Path: /hiiqos-8/myobj?versionId=null 2023-07-28T23:46:09.095 DEBUG:boto:Data: 2023-07-28T23:46:09.095 DEBUG:boto:Headers: {} 2023-07-28T23:46:09.095 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:09.096 DEBUG:boto:Port: 8001 2023-07-28T23:46:09.096 DEBUG:boto:Params: {} 2023-07-28T23:46:09.097 DEBUG:boto:Token: None 2023-07-28T23:46:09.097 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:09 GMT /hiiqos-8/myobj?versionId=null 2023-07-28T23:46:09.098 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ufTVhXWJUxDDRd80T+5WcLiQBTU= 2023-07-28T23:46:09.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ufTVhXWJUxDDRd80T+5WcLiQBTU=', 'Content-Length': '0'} 2023-07-28T23:46:09.100 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008bbec374832b780f-0064c45341-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:09.101 DEBUG:boto:path=/hiiqos-8/ 2023-07-28T23:46:09.101 DEBUG:boto:auth_path=/hiiqos-8/ 2023-07-28T23:46:09.102 DEBUG:boto:path=/hiiqos-8/?prefix=myobj 2023-07-28T23:46:09.102 DEBUG:boto:auth_path=/hiiqos-8/?prefix=myobj 2023-07-28T23:46:09.103 DEBUG:boto:Method: GET 2023-07-28T23:46:09.103 DEBUG:boto:Path: /hiiqos-8/?prefix=myobj 2023-07-28T23:46:09.103 DEBUG:boto:Data: 2023-07-28T23:46:09.104 DEBUG:boto:Headers: {} 2023-07-28T23:46:09.104 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:09.104 DEBUG:boto:Port: 8000 2023-07-28T23:46:09.105 DEBUG:boto:Params: {} 2023-07-28T23:46:09.105 DEBUG:boto:Token: None 2023-07-28T23:46:09.106 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:09 GMT /hiiqos-8/ 2023-07-28T23:46:09.106 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yqTWppJINh673TgdifBMbrGzFyc= 2023-07-28T23:46:09.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yqTWppJINh673TgdifBMbrGzFyc=', 'Content-Length': '0'} 2023-07-28T23:46:09.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e901041a5f68804c-0064c45341-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:09.109 DEBUG:boto:b'hiiqos-8myobj1000falsemyobj2023-07-28T23:42:36.980Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:46:09.110 DEBUG:boto:path=/hiiqos-8/ 2023-07-28T23:46:09.110 DEBUG:boto:auth_path=/hiiqos-8/ 2023-07-28T23:46:09.111 DEBUG:boto:path=/hiiqos-8/?prefix=myobj 2023-07-28T23:46:09.111 DEBUG:boto:auth_path=/hiiqos-8/?prefix=myobj 2023-07-28T23:46:09.111 DEBUG:boto:Method: GET 2023-07-28T23:46:09.112 DEBUG:boto:Path: /hiiqos-8/?prefix=myobj 2023-07-28T23:46:09.112 DEBUG:boto:Data: 2023-07-28T23:46:09.113 DEBUG:boto:Headers: {} 2023-07-28T23:46:09.113 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:09.113 DEBUG:boto:Port: 8001 2023-07-28T23:46:09.114 DEBUG:boto:Params: {} 2023-07-28T23:46:09.114 DEBUG:boto:Token: None 2023-07-28T23:46:09.114 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:09 GMT /hiiqos-8/ 2023-07-28T23:46:09.115 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yqTWppJINh673TgdifBMbrGzFyc= 2023-07-28T23:46:09.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yqTWppJINh673TgdifBMbrGzFyc=', 'Content-Length': '0'} 2023-07-28T23:46:09.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a4fecf4815e0600-0064c45341-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:09.160 DEBUG:boto:b'hiiqos-8myobj1000falsemyobj2023-07-28T23:42:36.980Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:46:09.162 DEBUG:boto:path=/hiiqos-8/myobj 2023-07-28T23:46:09.163 DEBUG:boto:auth_path=/hiiqos-8/myobj 2023-07-28T23:46:09.163 DEBUG:boto:Method: HEAD 2023-07-28T23:46:09.164 DEBUG:boto:Path: /hiiqos-8/myobj 2023-07-28T23:46:09.164 DEBUG:boto:Data: 2023-07-28T23:46:09.165 DEBUG:boto:Headers: {} 2023-07-28T23:46:09.166 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:09.166 DEBUG:boto:Port: 8000 2023-07-28T23:46:09.167 DEBUG:boto:Params: {} 2023-07-28T23:46:09.167 DEBUG:boto:Token: None 2023-07-28T23:46:09.168 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:46:09 GMT /hiiqos-8/myobj 2023-07-28T23:46:09.168 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:y5GpaZeO3ZdT4STU7/uzI3FZrNU= 2023-07-28T23:46:09.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:y5GpaZeO3ZdT4STU7/uzI3FZrNU=', 'Content-Length': '0'} 2023-07-28T23:46:09.171 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c81d1313b92ddbe3-0064c45341-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:09.171 DEBUG:boto:path=/hiiqos-8/myobj 2023-07-28T23:46:09.172 DEBUG:boto:auth_path=/hiiqos-8/myobj 2023-07-28T23:46:09.172 DEBUG:boto:Method: HEAD 2023-07-28T23:46:09.172 DEBUG:boto:Path: /hiiqos-8/myobj 2023-07-28T23:46:09.173 DEBUG:boto:Data: 2023-07-28T23:46:09.173 DEBUG:boto:Headers: {} 2023-07-28T23:46:09.173 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:09.174 DEBUG:boto:Port: 8001 2023-07-28T23:46:09.174 DEBUG:boto:Params: {} 2023-07-28T23:46:09.174 DEBUG:boto:Token: None 2023-07-28T23:46:09.175 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:46:09 GMT /hiiqos-8/myobj 2023-07-28T23:46:09.175 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:y5GpaZeO3ZdT4STU7/uzI3FZrNU= 2023-07-28T23:46:09.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:y5GpaZeO3ZdT4STU7/uzI3FZrNU=', 'Content-Length': '0'} 2023-07-28T23:46:09.177 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f7f6e99d07135a9-0064c45341-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:09.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-07-28T23:46:09.178 DEBUG:boto:path=/hiiqos-8/myobj 2023-07-28T23:46:09.179 DEBUG:boto:auth_path=/hiiqos-8/myobj 2023-07-28T23:46:09.179 DEBUG:boto:Method: GET 2023-07-28T23:46:09.180 DEBUG:boto:Path: /hiiqos-8/myobj 2023-07-28T23:46:09.180 DEBUG:boto:Data: 2023-07-28T23:46:09.180 DEBUG:boto:Headers: {} 2023-07-28T23:46:09.181 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:09.181 DEBUG:boto:Port: 8000 2023-07-28T23:46:09.182 DEBUG:boto:Params: {} 2023-07-28T23:46:09.182 DEBUG:boto:Token: None 2023-07-28T23:46:09.182 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:09 GMT /hiiqos-8/myobj 2023-07-28T23:46:09.183 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IDh2vLXrqXjP9mpxjCGfZTXumcE= 2023-07-28T23:46:09.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IDh2vLXrqXjP9mpxjCGfZTXumcE=', 'Content-Length': '0'} 2023-07-28T23:46:09.186 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000751d3b0fcfae02ff-0064c45341-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:09.186 DEBUG:boto:path=/hiiqos-8/myobj 2023-07-28T23:46:09.187 DEBUG:boto:auth_path=/hiiqos-8/myobj 2023-07-28T23:46:09.187 DEBUG:boto:Method: GET 2023-07-28T23:46:09.187 DEBUG:boto:Path: /hiiqos-8/myobj 2023-07-28T23:46:09.188 DEBUG:boto:Data: 2023-07-28T23:46:09.188 DEBUG:boto:Headers: {} 2023-07-28T23:46:09.189 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:09.189 DEBUG:boto:Port: 8001 2023-07-28T23:46:09.190 DEBUG:boto:Params: {} 2023-07-28T23:46:09.190 DEBUG:boto:Token: None 2023-07-28T23:46:09.191 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:09 GMT /hiiqos-8/myobj 2023-07-28T23:46:09.191 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IDh2vLXrqXjP9mpxjCGfZTXumcE= 2023-07-28T23:46:09.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IDh2vLXrqXjP9mpxjCGfZTXumcE=', 'Content-Length': '0'} 2023-07-28T23:46:09.194 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:42:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005bd7d3a07dafb2b2-0064c45341-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:09.194 DEBUG:boto:path=/hiiqos-8/ 2023-07-28T23:46:09.195 DEBUG:boto:auth_path=/hiiqos-8/ 2023-07-28T23:46:09.195 DEBUG:boto:path=/hiiqos-8/?prefix=myobj 2023-07-28T23:46:09.195 DEBUG:boto:auth_path=/hiiqos-8/?prefix=myobj 2023-07-28T23:46:09.196 DEBUG:boto:Method: GET 2023-07-28T23:46:09.196 DEBUG:boto:Path: /hiiqos-8/?prefix=myobj 2023-07-28T23:46:09.196 DEBUG:boto:Data: 2023-07-28T23:46:09.197 DEBUG:boto:Headers: {} 2023-07-28T23:46:09.197 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:09.198 DEBUG:boto:Port: 8000 2023-07-28T23:46:09.198 DEBUG:boto:Params: {} 2023-07-28T23:46:09.198 DEBUG:boto:Token: None 2023-07-28T23:46:09.199 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:09 GMT /hiiqos-8/ 2023-07-28T23:46:09.199 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yqTWppJINh673TgdifBMbrGzFyc= 2023-07-28T23:46:09.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yqTWppJINh673TgdifBMbrGzFyc=', 'Content-Length': '0'} 2023-07-28T23:46:09.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000301096056190e2b0-0064c45341-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:09.203 DEBUG:boto:b'hiiqos-8myobj1000falsemyobj2023-07-28T23:42:36.980Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:46:09.203 DEBUG:boto:path=/hiiqos-8/ 2023-07-28T23:46:09.204 DEBUG:boto:auth_path=/hiiqos-8/ 2023-07-28T23:46:09.204 DEBUG:boto:path=/hiiqos-8/?prefix=myobj 2023-07-28T23:46:09.205 DEBUG:boto:auth_path=/hiiqos-8/?prefix=myobj 2023-07-28T23:46:09.205 DEBUG:boto:Method: GET 2023-07-28T23:46:09.205 DEBUG:boto:Path: /hiiqos-8/?prefix=myobj 2023-07-28T23:46:09.206 DEBUG:boto:Data: 2023-07-28T23:46:09.206 DEBUG:boto:Headers: {} 2023-07-28T23:46:09.206 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:09.207 DEBUG:boto:Port: 8001 2023-07-28T23:46:09.207 DEBUG:boto:Params: {} 2023-07-28T23:46:09.207 DEBUG:boto:Token: None 2023-07-28T23:46:09.208 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:09 GMT /hiiqos-8/ 2023-07-28T23:46:09.208 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yqTWppJINh673TgdifBMbrGzFyc= 2023-07-28T23:46:09.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yqTWppJINh673TgdifBMbrGzFyc=', 'Content-Length': '0'} 2023-07-28T23:46:09.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa6aaf9f5ab81e99-0064c45341-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:09.212 DEBUG:boto:b'hiiqos-8myobj1000falsemyobj2023-07-28T23:42:36.980Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:46:09.212 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-8 zones={a1, a2} 2023-07-28T23:46:09.213 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_sync ... ok 2023-07-28T23:46:09.214 INFO:rgw_multi.tests:create bucket zone=a1 name=hiiqos-9 2023-07-28T23:46:09.214 DEBUG:boto:path=/hiiqos-9/ 2023-07-28T23:46:09.215 DEBUG:boto:auth_path=/hiiqos-9/ 2023-07-28T23:46:09.215 DEBUG:boto:Method: PUT 2023-07-28T23:46:09.215 DEBUG:boto:Path: /hiiqos-9/ 2023-07-28T23:46:09.216 DEBUG:boto:Data: 2023-07-28T23:46:09.216 DEBUG:boto:Headers: {} 2023-07-28T23:46:09.216 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:09.217 DEBUG:boto:Port: 8000 2023-07-28T23:46:09.217 DEBUG:boto:Params: {} 2023-07-28T23:46:09.217 DEBUG:boto:Token: None 2023-07-28T23:46:09.218 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:46:09 GMT /hiiqos-9/ 2023-07-28T23:46:09.218 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eTOlGBIIWYAjQi/qIFqSt7skr4o= 2023-07-28T23:46:09.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eTOlGBIIWYAjQi/qIFqSt7skr4o=', 'Content-Length': '0'} 2023-07-28T23:46:09.235 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000097ba3ef7aa1182dd-0064c45341-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:09.236 INFO:rgw_multi.tests:create bucket zone=a2 name=hiiqos-10 2023-07-28T23:46:09.236 DEBUG:boto:path=/hiiqos-10/ 2023-07-28T23:46:09.237 DEBUG:boto:auth_path=/hiiqos-10/ 2023-07-28T23:46:09.237 DEBUG:boto:Method: PUT 2023-07-28T23:46:09.237 DEBUG:boto:Path: /hiiqos-10/ 2023-07-28T23:46:09.238 DEBUG:boto:Data: 2023-07-28T23:46:09.238 DEBUG:boto:Headers: {} 2023-07-28T23:46:09.238 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:09.239 DEBUG:boto:Port: 8001 2023-07-28T23:46:09.239 DEBUG:boto:Params: {} 2023-07-28T23:46:09.240 DEBUG:boto:Token: None 2023-07-28T23:46:09.240 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:46:09 GMT /hiiqos-10/ 2023-07-28T23:46:09.240 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ADm5jty5FY6YH/gF3XzwXo0HxZA= 2023-07-28T23:46:09.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ADm5jty5FY6YH/gF3XzwXo0HxZA=', 'Content-Length': '0'} 2023-07-28T23:46:09.273 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d0054dd1585e18ec-0064c45341-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:09.274 DEBUG:boto:path=/hiiqos-9/ 2023-07-28T23:46:09.274 DEBUG:boto:auth_path=/hiiqos-9/ 2023-07-28T23:46:09.275 DEBUG:boto:Method: HEAD 2023-07-28T23:46:09.275 DEBUG:boto:Path: /hiiqos-9/ 2023-07-28T23:46:09.275 DEBUG:boto:Data: 2023-07-28T23:46:09.276 DEBUG:boto:Headers: {} 2023-07-28T23:46:09.276 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:09.277 DEBUG:boto:Port: 8000 2023-07-28T23:46:09.277 DEBUG:boto:Params: {} 2023-07-28T23:46:09.277 DEBUG:boto:Token: None 2023-07-28T23:46:09.278 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:46:09 GMT /hiiqos-9/ 2023-07-28T23:46:09.278 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:czlPMxMm43BLK0G1XRZYyHZOMto= 2023-07-28T23:46:09.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:czlPMxMm43BLK0G1XRZYyHZOMto=', 'Content-Length': '0'} 2023-07-28T23:46:09.281 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000093a014c416e8af9f-0064c45341-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:09.281 DEBUG:boto:path=/hiiqos-9/myobj 2023-07-28T23:46:09.282 DEBUG:boto:auth_path=/hiiqos-9/myobj 2023-07-28T23:46:09.282 DEBUG:boto:Method: PUT 2023-07-28T23:46:09.282 DEBUG:boto:Path: /hiiqos-9/myobj 2023-07-28T23:46:09.283 DEBUG:boto:Data: 2023-07-28T23:46:09.283 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-28T23:46:09.283 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:09.284 DEBUG:boto:Port: 8000 2023-07-28T23:46:09.284 DEBUG:boto:Params: {} 2023-07-28T23:46:09.285 DEBUG:boto:Token: None 2023-07-28T23:46:09.285 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 28 Jul 2023 23:46:09 GMT /hiiqos-9/myobj 2023-07-28T23:46:09.286 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xZJ7f9mCyu8NNd8E2k+xYXHkeGE= 2023-07-28T23:46:09.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:46:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xZJ7f9mCyu8NNd8E2k+xYXHkeGE='} 2023-07-28T23:46:09.328 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fc1332203036f398-0064c45341-1117-a1'), ('Date', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:09.329 DEBUG:boto:path=/hiiqos-10/ 2023-07-28T23:46:09.329 DEBUG:boto:auth_path=/hiiqos-10/ 2023-07-28T23:46:09.330 DEBUG:boto:Method: HEAD 2023-07-28T23:46:09.330 DEBUG:boto:Path: /hiiqos-10/ 2023-07-28T23:46:09.330 DEBUG:boto:Data: 2023-07-28T23:46:09.331 DEBUG:boto:Headers: {} 2023-07-28T23:46:09.331 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:09.331 DEBUG:boto:Port: 8001 2023-07-28T23:46:09.332 DEBUG:boto:Params: {} 2023-07-28T23:46:09.332 DEBUG:boto:Token: None 2023-07-28T23:46:09.332 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:46:09 GMT /hiiqos-10/ 2023-07-28T23:46:09.333 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Yq177gjpzTzkN5qKYK3LpI4R+PI= 2023-07-28T23:46:09.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Yq177gjpzTzkN5qKYK3LpI4R+PI=', 'Content-Length': '0'} 2023-07-28T23:46:09.335 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b8b4205c90a37ae2-0064c45341-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:09.336 DEBUG:boto:path=/hiiqos-10/myobj 2023-07-28T23:46:09.337 DEBUG:boto:auth_path=/hiiqos-10/myobj 2023-07-28T23:46:09.337 DEBUG:boto:Method: PUT 2023-07-28T23:46:09.337 DEBUG:boto:Path: /hiiqos-10/myobj 2023-07-28T23:46:09.338 DEBUG:boto:Data: 2023-07-28T23:46:09.338 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-28T23:46:09.338 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:09.339 DEBUG:boto:Port: 8001 2023-07-28T23:46:09.339 DEBUG:boto:Params: {} 2023-07-28T23:46:09.339 DEBUG:boto:Token: None 2023-07-28T23:46:09.340 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 28 Jul 2023 23:46:09 GMT /hiiqos-10/myobj 2023-07-28T23:46:09.340 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QypZyuYQrJc3u9n5DEfnKH7xKiI= 2023-07-28T23:46:09.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:46:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QypZyuYQrJc3u9n5DEfnKH7xKiI='} 2023-07-28T23:46:09.351 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007781e8b485a36480-0064c45341-1113-a2'), ('Date', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:09.352 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:46:09.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:46:09.353 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:46:09.401 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:46:09.402 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:46:09.430 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-28T23:46:09.441 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-28T23:46:09.442 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:46:09.442 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-28T23:46:09.442 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:46:09.233742Z" 2023-07-28T23:46:09.442 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:46:09.442 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:46:09.442 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.262338_126.1", 2023-07-28T23:46:09.443 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:46:09.262338Z" 2023-07-28T23:46:09.443 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:46:09.443 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:46:09.443 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:46:09.443 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:46:09.443 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:46:09.443 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:46:09.444 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:46:09.444 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:46:09.444 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:46:09.444 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-28T23:46:09.445 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690587969.233742_99.1', 1: '1_1690587969.262338_126.1', 2: '', 3: ''} 2023-07-28T23:46:09.446 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-28T23:46:09.446 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:46:09.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:46:09.447 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:46:09.492 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:46:09.492 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:46:09.519 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:46:09.519+0000 7f2d6e29d880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:46:09.520 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:46:09.520+0000 7f2d6e29d880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:46:09.532 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:46:09.532 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:46:09.532 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:46:09.533 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:46:09.533 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:46:09.533 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:46:09.533 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:46:09.533 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:46:09.533 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:46:09.533 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:46:09.534 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:46:09.534 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:46:09.534 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:46:09.535 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-28T23:46:09.535 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:46:09.535 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:46:09.535 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:46:09.535 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-28T23:46:09.535 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:46:09.535 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:46:09.536 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:46:09.536 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:46:09.536 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:46:09.536 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:46:09.536 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:46:09.536 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.262338_126.1", 2023-07-28T23:46:09.537 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:46:09.537 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:46:09.537 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:46:09.537 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.262338Z", 2023-07-28T23:46:09.537 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:46:09.537 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:46:09.537 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:46:09.538 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:46:09.538 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:46:09.538 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:46:09.538 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:46:09.538 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:46:09.538 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:46:09.539 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:46:09.539 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:46:09.539 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:46:09.539 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:46:09.539 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:46:09.539 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:46:09.539 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:46:09.540 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:46:09.540 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:46:09.540 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:46:09.540 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:46:09.541 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:46:09.541 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:46:09.541 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:46:09.541 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:46:09.542 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:46:09.542 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:46:09.542 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:46:09.542 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:46:09.542 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:46:09.543 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:46:09.543 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:46:09.543 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:46:09.543 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:46:09.543 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:46:09Z" 2023-07-28T23:46:09.543 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:46:09.544 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690587969.262338_126.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:46:09.262338Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-28T23:46:09Z" } 2023-07-28T23:46:09.545 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690587969.262338_126.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.262338Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-28T23:46:09.545 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690587969.262338_126.1', 2: '', 3: ''} 2023-07-28T23:46:09.546 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690587969.262338_126.1', 2: '', 3: ''} 2023-07-28T23:46:09.546 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-28T23:46:09.546 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:46:09.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:46:09.547 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-9 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-28T23:46:09.600 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:46:09.600 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:46:09.631 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:46:09.631+0000 7fae72d4b880 1 waiting to reach incremental sync.. 2023-07-28T23:46:14.633 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:46:14.633+0000 7fae72d4b880 1 bucket sync caught up with source: 2023-07-28T23:46:14.634 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000001.4.6, , , , , , , , , ] 2023-07-28T23:46:14.634 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000001.4.6, , , , , , , , , ] 2023-07-28T23:46:14.634 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:46:14.633+0000 7fae72d4b880 0 bucket checkpoint complete 2023-07-28T23:46:14.649 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-9 zones={a2, a1} 2023-07-28T23:46:14.649 DEBUG:boto:path=/hiiqos-9/ 2023-07-28T23:46:14.649 DEBUG:boto:auth_path=/hiiqos-9/ 2023-07-28T23:46:14.650 DEBUG:boto:Method: HEAD 2023-07-28T23:46:14.650 DEBUG:boto:Path: /hiiqos-9/ 2023-07-28T23:46:14.651 DEBUG:boto:Data: 2023-07-28T23:46:14.651 DEBUG:boto:Headers: {} 2023-07-28T23:46:14.651 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:14.651 DEBUG:boto:Port: 8001 2023-07-28T23:46:14.652 DEBUG:boto:Params: {} 2023-07-28T23:46:14.652 DEBUG:boto:Token: None 2023-07-28T23:46:14.653 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:46:14 GMT /hiiqos-9/ 2023-07-28T23:46:14.653 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8wDoMhHlRUqfs6QN3cTKmxALTCQ= 2023-07-28T23:46:14.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8wDoMhHlRUqfs6QN3cTKmxALTCQ=', 'Content-Length': '0'} 2023-07-28T23:46:14.656 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000017250677a3035e6e-0064c45346-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:46:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:14.656 DEBUG:boto:path=/hiiqos-9/ 2023-07-28T23:46:14.657 DEBUG:boto:auth_path=/hiiqos-9/ 2023-07-28T23:46:14.657 DEBUG:boto:Method: HEAD 2023-07-28T23:46:14.657 DEBUG:boto:Path: /hiiqos-9/ 2023-07-28T23:46:14.658 DEBUG:boto:Data: 2023-07-28T23:46:14.658 DEBUG:boto:Headers: {} 2023-07-28T23:46:14.658 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:14.659 DEBUG:boto:Port: 8000 2023-07-28T23:46:14.659 DEBUG:boto:Params: {} 2023-07-28T23:46:14.660 DEBUG:boto:Token: None 2023-07-28T23:46:14.660 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:46:14 GMT /hiiqos-9/ 2023-07-28T23:46:14.660 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8wDoMhHlRUqfs6QN3cTKmxALTCQ= 2023-07-28T23:46:14.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8wDoMhHlRUqfs6QN3cTKmxALTCQ=', 'Content-Length': '0'} 2023-07-28T23:46:14.663 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000621e3211505ad8cc-0064c45346-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:46:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:14.664 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-28T23:46:14.664 DEBUG:boto:path=/hiiqos-9/ 2023-07-28T23:46:14.664 DEBUG:boto:auth_path=/hiiqos-9/ 2023-07-28T23:46:14.665 DEBUG:boto:path=/hiiqos-9/?versions&max-keys=999 2023-07-28T23:46:14.665 DEBUG:boto:auth_path=/hiiqos-9/?versions&max-keys=999 2023-07-28T23:46:14.665 DEBUG:boto:Method: GET 2023-07-28T23:46:14.666 DEBUG:boto:Path: /hiiqos-9/?versions&max-keys=999 2023-07-28T23:46:14.666 DEBUG:boto:Data: 2023-07-28T23:46:14.666 DEBUG:boto:Headers: {} 2023-07-28T23:46:14.667 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:14.667 DEBUG:boto:Port: 8001 2023-07-28T23:46:14.667 DEBUG:boto:Params: {} 2023-07-28T23:46:14.668 DEBUG:boto:Token: None 2023-07-28T23:46:14.668 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:14 GMT /hiiqos-9/?versions 2023-07-28T23:46:14.669 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tGW6AB6VIFQ8Hf1BC+wjqbgflRs= 2023-07-28T23:46:14.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tGW6AB6VIFQ8Hf1BC+wjqbgflRs=', 'Content-Length': '0'} 2023-07-28T23:46:14.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000729f5cc42e8ada78-0064c45346-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:14.672 DEBUG:boto:b'hiiqos-9999falsemyobjnulltrue2023-07-28T23:46:09.289Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:46:14.672 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-07-28T23:46:14.673 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-28T23:46:14.673 DEBUG:boto:path=/hiiqos-9/ 2023-07-28T23:46:14.673 DEBUG:boto:auth_path=/hiiqos-9/ 2023-07-28T23:46:14.674 DEBUG:boto:path=/hiiqos-9/?versions&max-keys=999 2023-07-28T23:46:14.674 DEBUG:boto:auth_path=/hiiqos-9/?versions&max-keys=999 2023-07-28T23:46:14.674 DEBUG:boto:Method: GET 2023-07-28T23:46:14.675 DEBUG:boto:Path: /hiiqos-9/?versions&max-keys=999 2023-07-28T23:46:14.675 DEBUG:boto:Data: 2023-07-28T23:46:14.675 DEBUG:boto:Headers: {} 2023-07-28T23:46:14.676 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:14.676 DEBUG:boto:Port: 8000 2023-07-28T23:46:14.676 DEBUG:boto:Params: {} 2023-07-28T23:46:14.677 DEBUG:boto:Token: None 2023-07-28T23:46:14.677 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:14 GMT /hiiqos-9/?versions 2023-07-28T23:46:14.678 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tGW6AB6VIFQ8Hf1BC+wjqbgflRs= 2023-07-28T23:46:14.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tGW6AB6VIFQ8Hf1BC+wjqbgflRs=', 'Content-Length': '0'} 2023-07-28T23:46:14.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b143da30f646e82e-0064c45346-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:14.681 DEBUG:boto:b'hiiqos-9999falsemyobjnulltrue2023-07-28T23:46:09.289Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:46:14.681 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-07-28T23:46:14.682 DEBUG:boto:path=/hiiqos-9/ 2023-07-28T23:46:14.682 DEBUG:boto:auth_path=/hiiqos-9/ 2023-07-28T23:46:14.682 DEBUG:boto:path=/hiiqos-9/?versions&max-keys=999 2023-07-28T23:46:14.683 DEBUG:boto:auth_path=/hiiqos-9/?versions&max-keys=999 2023-07-28T23:46:14.683 DEBUG:boto:Method: GET 2023-07-28T23:46:14.683 DEBUG:boto:Path: /hiiqos-9/?versions&max-keys=999 2023-07-28T23:46:14.684 DEBUG:boto:Data: 2023-07-28T23:46:14.684 DEBUG:boto:Headers: {} 2023-07-28T23:46:14.684 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:14.685 DEBUG:boto:Port: 8001 2023-07-28T23:46:14.685 DEBUG:boto:Params: {} 2023-07-28T23:46:14.685 DEBUG:boto:Token: None 2023-07-28T23:46:14.686 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:14 GMT /hiiqos-9/?versions 2023-07-28T23:46:14.686 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tGW6AB6VIFQ8Hf1BC+wjqbgflRs= 2023-07-28T23:46:14.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tGW6AB6VIFQ8Hf1BC+wjqbgflRs=', 'Content-Length': '0'} 2023-07-28T23:46:14.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef4561ccab1db22d-0064c45346-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:14.689 DEBUG:boto:b'hiiqos-9999falsemyobjnulltrue2023-07-28T23:46:09.289Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:46:14.690 DEBUG:boto:path=/hiiqos-9/ 2023-07-28T23:46:14.690 DEBUG:boto:auth_path=/hiiqos-9/ 2023-07-28T23:46:14.690 DEBUG:boto:path=/hiiqos-9/?versions&max-keys=999 2023-07-28T23:46:14.691 DEBUG:boto:auth_path=/hiiqos-9/?versions&max-keys=999 2023-07-28T23:46:14.691 DEBUG:boto:Method: GET 2023-07-28T23:46:14.691 DEBUG:boto:Path: /hiiqos-9/?versions&max-keys=999 2023-07-28T23:46:14.692 DEBUG:boto:Data: 2023-07-28T23:46:14.692 DEBUG:boto:Headers: {} 2023-07-28T23:46:14.692 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:14.693 DEBUG:boto:Port: 8000 2023-07-28T23:46:14.693 DEBUG:boto:Params: {} 2023-07-28T23:46:14.694 DEBUG:boto:Token: None 2023-07-28T23:46:14.694 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:14 GMT /hiiqos-9/?versions 2023-07-28T23:46:14.694 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tGW6AB6VIFQ8Hf1BC+wjqbgflRs= 2023-07-28T23:46:14.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tGW6AB6VIFQ8Hf1BC+wjqbgflRs=', 'Content-Length': '0'} 2023-07-28T23:46:14.697 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006685934f7cfff4e2-0064c45346-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:14.697 DEBUG:boto:b'hiiqos-9999falsemyobjnulltrue2023-07-28T23:46:09.289Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:46:14.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-07-28T23:46:14.698 DEBUG:boto:path=/hiiqos-9/myobj 2023-07-28T23:46:14.699 DEBUG:boto:auth_path=/hiiqos-9/myobj 2023-07-28T23:46:14.699 DEBUG:boto:path=/hiiqos-9/myobj?versionId=null 2023-07-28T23:46:14.699 DEBUG:boto:auth_path=/hiiqos-9/myobj?versionId=null 2023-07-28T23:46:14.700 DEBUG:boto:Method: GET 2023-07-28T23:46:14.700 DEBUG:boto:Path: /hiiqos-9/myobj?versionId=null 2023-07-28T23:46:14.701 DEBUG:boto:Data: 2023-07-28T23:46:14.701 DEBUG:boto:Headers: {} 2023-07-28T23:46:14.701 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:14.702 DEBUG:boto:Port: 8001 2023-07-28T23:46:14.702 DEBUG:boto:Params: {} 2023-07-28T23:46:14.702 DEBUG:boto:Token: None 2023-07-28T23:46:14.703 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:14 GMT /hiiqos-9/myobj?versionId=null 2023-07-28T23:46:14.703 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/VUit8+5yHZhlWFABhHNceuHjcs= 2023-07-28T23:46:14.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/VUit8+5yHZhlWFABhHNceuHjcs=', 'Content-Length': '0'} 2023-07-28T23:46:14.706 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-9:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000080a106ca9507e584-0064c45346-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:14.706 DEBUG:boto:path=/hiiqos-9/myobj 2023-07-28T23:46:14.707 DEBUG:boto:auth_path=/hiiqos-9/myobj 2023-07-28T23:46:14.707 DEBUG:boto:path=/hiiqos-9/myobj?versionId=null 2023-07-28T23:46:14.707 DEBUG:boto:auth_path=/hiiqos-9/myobj?versionId=null 2023-07-28T23:46:14.708 DEBUG:boto:Method: GET 2023-07-28T23:46:14.708 DEBUG:boto:Path: /hiiqos-9/myobj?versionId=null 2023-07-28T23:46:14.708 DEBUG:boto:Data: 2023-07-28T23:46:14.709 DEBUG:boto:Headers: {} 2023-07-28T23:46:14.709 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:14.709 DEBUG:boto:Port: 8000 2023-07-28T23:46:14.710 DEBUG:boto:Params: {} 2023-07-28T23:46:14.710 DEBUG:boto:Token: None 2023-07-28T23:46:14.711 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:14 GMT /hiiqos-9/myobj?versionId=null 2023-07-28T23:46:14.711 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/VUit8+5yHZhlWFABhHNceuHjcs= 2023-07-28T23:46:14.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/VUit8+5yHZhlWFABhHNceuHjcs=', 'Content-Length': '0'} 2023-07-28T23:46:14.713 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ebefcd706975b82a-0064c45346-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:14.714 DEBUG:boto:path=/hiiqos-9/myobj 2023-07-28T23:46:14.714 DEBUG:boto:auth_path=/hiiqos-9/myobj 2023-07-28T23:46:14.714 DEBUG:boto:path=/hiiqos-9/myobj?versionId=null 2023-07-28T23:46:14.715 DEBUG:boto:auth_path=/hiiqos-9/myobj?versionId=null 2023-07-28T23:46:14.715 DEBUG:boto:Method: HEAD 2023-07-28T23:46:14.715 DEBUG:boto:Path: /hiiqos-9/myobj?versionId=null 2023-07-28T23:46:14.716 DEBUG:boto:Data: 2023-07-28T23:46:14.716 DEBUG:boto:Headers: {} 2023-07-28T23:46:14.716 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:14.717 DEBUG:boto:Port: 8001 2023-07-28T23:46:14.717 DEBUG:boto:Params: {} 2023-07-28T23:46:14.717 DEBUG:boto:Token: None 2023-07-28T23:46:14.718 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:46:14 GMT /hiiqos-9/myobj?versionId=null 2023-07-28T23:46:14.718 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qK8dIe29NafpTb5q7AbHnfxo/iI= 2023-07-28T23:46:14.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qK8dIe29NafpTb5q7AbHnfxo/iI=', 'Content-Length': '0'} 2023-07-28T23:46:14.720 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-9:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000024f6511abab4aa5e-0064c45346-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:14.721 DEBUG:boto:path=/hiiqos-9/myobj 2023-07-28T23:46:14.721 DEBUG:boto:auth_path=/hiiqos-9/myobj 2023-07-28T23:46:14.721 DEBUG:boto:path=/hiiqos-9/myobj?versionId=null 2023-07-28T23:46:14.722 DEBUG:boto:auth_path=/hiiqos-9/myobj?versionId=null 2023-07-28T23:46:14.722 DEBUG:boto:Method: HEAD 2023-07-28T23:46:14.722 DEBUG:boto:Path: /hiiqos-9/myobj?versionId=null 2023-07-28T23:46:14.723 DEBUG:boto:Data: 2023-07-28T23:46:14.723 DEBUG:boto:Headers: {} 2023-07-28T23:46:14.723 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:14.724 DEBUG:boto:Port: 8000 2023-07-28T23:46:14.724 DEBUG:boto:Params: {} 2023-07-28T23:46:14.724 DEBUG:boto:Token: None 2023-07-28T23:46:14.725 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:46:14 GMT /hiiqos-9/myobj?versionId=null 2023-07-28T23:46:14.725 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qK8dIe29NafpTb5q7AbHnfxo/iI= 2023-07-28T23:46:14.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qK8dIe29NafpTb5q7AbHnfxo/iI=', 'Content-Length': '0'} 2023-07-28T23:46:14.727 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e5ead86ba2a9ed3-0064c45346-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:14.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-07-28T23:46:14.728 DEBUG:boto:path=/hiiqos-9/myobj 2023-07-28T23:46:14.729 DEBUG:boto:auth_path=/hiiqos-9/myobj 2023-07-28T23:46:14.729 DEBUG:boto:path=/hiiqos-9/myobj?versionId=null 2023-07-28T23:46:14.729 DEBUG:boto:auth_path=/hiiqos-9/myobj?versionId=null 2023-07-28T23:46:14.730 DEBUG:boto:Method: GET 2023-07-28T23:46:14.730 DEBUG:boto:Path: /hiiqos-9/myobj?versionId=null 2023-07-28T23:46:14.730 DEBUG:boto:Data: 2023-07-28T23:46:14.731 DEBUG:boto:Headers: {} 2023-07-28T23:46:14.731 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:14.731 DEBUG:boto:Port: 8001 2023-07-28T23:46:14.732 DEBUG:boto:Params: {} 2023-07-28T23:46:14.732 DEBUG:boto:Token: None 2023-07-28T23:46:14.733 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:14 GMT /hiiqos-9/myobj?versionId=null 2023-07-28T23:46:14.733 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/VUit8+5yHZhlWFABhHNceuHjcs= 2023-07-28T23:46:14.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/VUit8+5yHZhlWFABhHNceuHjcs=', 'Content-Length': '0'} 2023-07-28T23:46:14.736 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-9:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b3695c3fc6ac2fb8-0064c45346-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:14.776 DEBUG:boto:path=/hiiqos-9/myobj 2023-07-28T23:46:14.777 DEBUG:boto:auth_path=/hiiqos-9/myobj 2023-07-28T23:46:14.777 DEBUG:boto:path=/hiiqos-9/myobj?versionId=null 2023-07-28T23:46:14.778 DEBUG:boto:auth_path=/hiiqos-9/myobj?versionId=null 2023-07-28T23:46:14.778 DEBUG:boto:Method: GET 2023-07-28T23:46:14.778 DEBUG:boto:Path: /hiiqos-9/myobj?versionId=null 2023-07-28T23:46:14.779 DEBUG:boto:Data: 2023-07-28T23:46:14.779 DEBUG:boto:Headers: {} 2023-07-28T23:46:14.779 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:14.780 DEBUG:boto:Port: 8000 2023-07-28T23:46:14.780 DEBUG:boto:Params: {} 2023-07-28T23:46:14.780 DEBUG:boto:Token: None 2023-07-28T23:46:14.781 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:14 GMT /hiiqos-9/myobj?versionId=null 2023-07-28T23:46:14.781 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/VUit8+5yHZhlWFABhHNceuHjcs= 2023-07-28T23:46:14.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/VUit8+5yHZhlWFABhHNceuHjcs=', 'Content-Length': '0'} 2023-07-28T23:46:14.784 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006628d9b3ef37bb10-0064c45346-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:14.784 DEBUG:boto:path=/hiiqos-9/ 2023-07-28T23:46:14.785 DEBUG:boto:auth_path=/hiiqos-9/ 2023-07-28T23:46:14.785 DEBUG:boto:path=/hiiqos-9/?prefix=myobj 2023-07-28T23:46:14.785 DEBUG:boto:auth_path=/hiiqos-9/?prefix=myobj 2023-07-28T23:46:14.786 DEBUG:boto:Method: GET 2023-07-28T23:46:14.786 DEBUG:boto:Path: /hiiqos-9/?prefix=myobj 2023-07-28T23:46:14.786 DEBUG:boto:Data: 2023-07-28T23:46:14.787 DEBUG:boto:Headers: {} 2023-07-28T23:46:14.787 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:14.787 DEBUG:boto:Port: 8001 2023-07-28T23:46:14.788 DEBUG:boto:Params: {} 2023-07-28T23:46:14.788 DEBUG:boto:Token: None 2023-07-28T23:46:14.789 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:14 GMT /hiiqos-9/ 2023-07-28T23:46:14.789 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:g4zI0pHjp2a/ScDj1TCTtYizqvw= 2023-07-28T23:46:14.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:g4zI0pHjp2a/ScDj1TCTtYizqvw=', 'Content-Length': '0'} 2023-07-28T23:46:14.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0824d2e932bccc5-0064c45346-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:14.792 DEBUG:boto:b'hiiqos-9myobj1000falsemyobj2023-07-28T23:46:09.289Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:46:14.793 DEBUG:boto:path=/hiiqos-9/ 2023-07-28T23:46:14.794 DEBUG:boto:auth_path=/hiiqos-9/ 2023-07-28T23:46:14.794 DEBUG:boto:path=/hiiqos-9/?prefix=myobj 2023-07-28T23:46:14.794 DEBUG:boto:auth_path=/hiiqos-9/?prefix=myobj 2023-07-28T23:46:14.795 DEBUG:boto:Method: GET 2023-07-28T23:46:14.795 DEBUG:boto:Path: /hiiqos-9/?prefix=myobj 2023-07-28T23:46:14.795 DEBUG:boto:Data: 2023-07-28T23:46:14.796 DEBUG:boto:Headers: {} 2023-07-28T23:46:14.796 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:14.796 DEBUG:boto:Port: 8000 2023-07-28T23:46:14.797 DEBUG:boto:Params: {} 2023-07-28T23:46:14.797 DEBUG:boto:Token: None 2023-07-28T23:46:14.798 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:14 GMT /hiiqos-9/ 2023-07-28T23:46:14.798 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:g4zI0pHjp2a/ScDj1TCTtYizqvw= 2023-07-28T23:46:14.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:g4zI0pHjp2a/ScDj1TCTtYizqvw=', 'Content-Length': '0'} 2023-07-28T23:46:14.800 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba463409c4eef526-0064c45346-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:14.801 DEBUG:boto:b'hiiqos-9myobj1000falsemyobj2023-07-28T23:46:09.289Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:46:14.802 DEBUG:boto:path=/hiiqos-9/myobj 2023-07-28T23:46:14.802 DEBUG:boto:auth_path=/hiiqos-9/myobj 2023-07-28T23:46:14.802 DEBUG:boto:Method: HEAD 2023-07-28T23:46:14.803 DEBUG:boto:Path: /hiiqos-9/myobj 2023-07-28T23:46:14.803 DEBUG:boto:Data: 2023-07-28T23:46:14.803 DEBUG:boto:Headers: {} 2023-07-28T23:46:14.804 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:14.804 DEBUG:boto:Port: 8001 2023-07-28T23:46:14.804 DEBUG:boto:Params: {} 2023-07-28T23:46:14.805 DEBUG:boto:Token: None 2023-07-28T23:46:14.805 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:46:14 GMT /hiiqos-9/myobj 2023-07-28T23:46:14.805 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Mu42qea4PMSlGiMDny8Q8JNXMJM= 2023-07-28T23:46:14.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Mu42qea4PMSlGiMDny8Q8JNXMJM=', 'Content-Length': '0'} 2023-07-28T23:46:14.808 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-9:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a4989426a7ed3ca6-0064c45346-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:14.808 DEBUG:boto:path=/hiiqos-9/myobj 2023-07-28T23:46:14.809 DEBUG:boto:auth_path=/hiiqos-9/myobj 2023-07-28T23:46:14.809 DEBUG:boto:Method: HEAD 2023-07-28T23:46:14.809 DEBUG:boto:Path: /hiiqos-9/myobj 2023-07-28T23:46:14.810 DEBUG:boto:Data: 2023-07-28T23:46:14.810 DEBUG:boto:Headers: {} 2023-07-28T23:46:14.810 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:14.811 DEBUG:boto:Port: 8000 2023-07-28T23:46:14.811 DEBUG:boto:Params: {} 2023-07-28T23:46:14.811 DEBUG:boto:Token: None 2023-07-28T23:46:14.812 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:46:14 GMT /hiiqos-9/myobj 2023-07-28T23:46:14.812 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Mu42qea4PMSlGiMDny8Q8JNXMJM= 2023-07-28T23:46:14.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Mu42qea4PMSlGiMDny8Q8JNXMJM=', 'Content-Length': '0'} 2023-07-28T23:46:14.814 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b02568660f42352-0064c45346-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:14.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-07-28T23:46:14.815 DEBUG:boto:path=/hiiqos-9/myobj 2023-07-28T23:46:14.816 DEBUG:boto:auth_path=/hiiqos-9/myobj 2023-07-28T23:46:14.816 DEBUG:boto:Method: GET 2023-07-28T23:46:14.816 DEBUG:boto:Path: /hiiqos-9/myobj 2023-07-28T23:46:14.817 DEBUG:boto:Data: 2023-07-28T23:46:14.817 DEBUG:boto:Headers: {} 2023-07-28T23:46:14.817 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:14.818 DEBUG:boto:Port: 8001 2023-07-28T23:46:14.818 DEBUG:boto:Params: {} 2023-07-28T23:46:14.818 DEBUG:boto:Token: None 2023-07-28T23:46:14.819 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:14 GMT /hiiqos-9/myobj 2023-07-28T23:46:14.819 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5usg8N8jCBEZ/OGEz8tQnnayUs0= 2023-07-28T23:46:14.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5usg8N8jCBEZ/OGEz8tQnnayUs0=', 'Content-Length': '0'} 2023-07-28T23:46:14.821 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-9:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e194305f899a93a0-0064c45346-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:14.822 DEBUG:boto:path=/hiiqos-9/myobj 2023-07-28T23:46:14.822 DEBUG:boto:auth_path=/hiiqos-9/myobj 2023-07-28T23:46:14.823 DEBUG:boto:Method: GET 2023-07-28T23:46:14.823 DEBUG:boto:Path: /hiiqos-9/myobj 2023-07-28T23:46:14.824 DEBUG:boto:Data: 2023-07-28T23:46:14.824 DEBUG:boto:Headers: {} 2023-07-28T23:46:14.824 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:14.825 DEBUG:boto:Port: 8000 2023-07-28T23:46:14.825 DEBUG:boto:Params: {} 2023-07-28T23:46:14.825 DEBUG:boto:Token: None 2023-07-28T23:46:14.826 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:14 GMT /hiiqos-9/myobj 2023-07-28T23:46:14.826 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5usg8N8jCBEZ/OGEz8tQnnayUs0= 2023-07-28T23:46:14.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5usg8N8jCBEZ/OGEz8tQnnayUs0=', 'Content-Length': '0'} 2023-07-28T23:46:14.828 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000063c055554144d821-0064c45346-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:14.872 DEBUG:boto:path=/hiiqos-9/ 2023-07-28T23:46:14.873 DEBUG:boto:auth_path=/hiiqos-9/ 2023-07-28T23:46:14.873 DEBUG:boto:path=/hiiqos-9/?prefix=myobj 2023-07-28T23:46:14.873 DEBUG:boto:auth_path=/hiiqos-9/?prefix=myobj 2023-07-28T23:46:14.874 DEBUG:boto:Method: GET 2023-07-28T23:46:14.874 DEBUG:boto:Path: /hiiqos-9/?prefix=myobj 2023-07-28T23:46:14.875 DEBUG:boto:Data: 2023-07-28T23:46:14.875 DEBUG:boto:Headers: {} 2023-07-28T23:46:14.875 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:14.876 DEBUG:boto:Port: 8001 2023-07-28T23:46:14.876 DEBUG:boto:Params: {} 2023-07-28T23:46:14.876 DEBUG:boto:Token: None 2023-07-28T23:46:14.877 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:14 GMT /hiiqos-9/ 2023-07-28T23:46:14.877 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:g4zI0pHjp2a/ScDj1TCTtYizqvw= 2023-07-28T23:46:14.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:g4zI0pHjp2a/ScDj1TCTtYizqvw=', 'Content-Length': '0'} 2023-07-28T23:46:14.880 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000937e70a55633a32c-0064c45346-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:14.880 DEBUG:boto:b'hiiqos-9myobj1000falsemyobj2023-07-28T23:46:09.289Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:46:14.881 DEBUG:boto:path=/hiiqos-9/ 2023-07-28T23:46:14.881 DEBUG:boto:auth_path=/hiiqos-9/ 2023-07-28T23:46:14.882 DEBUG:boto:path=/hiiqos-9/?prefix=myobj 2023-07-28T23:46:14.882 DEBUG:boto:auth_path=/hiiqos-9/?prefix=myobj 2023-07-28T23:46:14.882 DEBUG:boto:Method: GET 2023-07-28T23:46:14.883 DEBUG:boto:Path: /hiiqos-9/?prefix=myobj 2023-07-28T23:46:14.883 DEBUG:boto:Data: 2023-07-28T23:46:14.883 DEBUG:boto:Headers: {} 2023-07-28T23:46:14.884 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:14.884 DEBUG:boto:Port: 8000 2023-07-28T23:46:14.885 DEBUG:boto:Params: {} 2023-07-28T23:46:14.885 DEBUG:boto:Token: None 2023-07-28T23:46:14.885 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:14 GMT /hiiqos-9/ 2023-07-28T23:46:14.886 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:g4zI0pHjp2a/ScDj1TCTtYizqvw= 2023-07-28T23:46:14.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:g4zI0pHjp2a/ScDj1TCTtYizqvw=', 'Content-Length': '0'} 2023-07-28T23:46:14.888 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019ad9cec437cbdda-0064c45346-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:14.889 DEBUG:boto:b'hiiqos-9myobj1000falsemyobj2023-07-28T23:46:09.289Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:46:14.890 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-9 zones={a2, a1} 2023-07-28T23:46:14.890 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:46:14.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:46:14.891 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-10 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-28T23:46:14.950 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:46:14.951 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:46:14.986 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:46:14.985+0000 7fd9cce96880 1 bucket sync caught up with source: 2023-07-28T23:46:14.986 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000001.7.6, , , , , , , , , ] 2023-07-28T23:46:14.987 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000001.7.6, , , , , , , , , ] 2023-07-28T23:46:14.987 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:46:14.985+0000 7fd9cce96880 0 bucket checkpoint complete 2023-07-28T23:46:14.998 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-10 zones={a1, a2} 2023-07-28T23:46:14.999 DEBUG:boto:path=/hiiqos-10/ 2023-07-28T23:46:14.999 DEBUG:boto:auth_path=/hiiqos-10/ 2023-07-28T23:46:14.999 DEBUG:boto:Method: HEAD 2023-07-28T23:46:15.000 DEBUG:boto:Path: /hiiqos-10/ 2023-07-28T23:46:15.000 DEBUG:boto:Data: 2023-07-28T23:46:15.000 DEBUG:boto:Headers: {} 2023-07-28T23:46:15.001 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:15.001 DEBUG:boto:Port: 8000 2023-07-28T23:46:15.001 DEBUG:boto:Params: {} 2023-07-28T23:46:15.002 DEBUG:boto:Token: None 2023-07-28T23:46:15.002 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:46:15 GMT /hiiqos-10/ 2023-07-28T23:46:15.003 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3G/Yro4wOVUsnolqI6MpoJZISyU= 2023-07-28T23:46:15.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3G/Yro4wOVUsnolqI6MpoJZISyU=', 'Content-Length': '0'} 2023-07-28T23:46:15.005 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c7804768cfa27ad4-0064c45347-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:15.006 DEBUG:boto:path=/hiiqos-10/ 2023-07-28T23:46:15.006 DEBUG:boto:auth_path=/hiiqos-10/ 2023-07-28T23:46:15.007 DEBUG:boto:Method: HEAD 2023-07-28T23:46:15.007 DEBUG:boto:Path: /hiiqos-10/ 2023-07-28T23:46:15.007 DEBUG:boto:Data: 2023-07-28T23:46:15.008 DEBUG:boto:Headers: {} 2023-07-28T23:46:15.008 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:15.009 DEBUG:boto:Port: 8001 2023-07-28T23:46:15.009 DEBUG:boto:Params: {} 2023-07-28T23:46:15.009 DEBUG:boto:Token: None 2023-07-28T23:46:15.010 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:46:15 GMT /hiiqos-10/ 2023-07-28T23:46:15.010 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3G/Yro4wOVUsnolqI6MpoJZISyU= 2023-07-28T23:46:15.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3G/Yro4wOVUsnolqI6MpoJZISyU=', 'Content-Length': '0'} 2023-07-28T23:46:15.013 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b9a3c8b879113a70-0064c45347-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:15.013 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-28T23:46:15.014 DEBUG:boto:path=/hiiqos-10/ 2023-07-28T23:46:15.014 DEBUG:boto:auth_path=/hiiqos-10/ 2023-07-28T23:46:15.015 DEBUG:boto:path=/hiiqos-10/?versions&max-keys=999 2023-07-28T23:46:15.015 DEBUG:boto:auth_path=/hiiqos-10/?versions&max-keys=999 2023-07-28T23:46:15.015 DEBUG:boto:Method: GET 2023-07-28T23:46:15.016 DEBUG:boto:Path: /hiiqos-10/?versions&max-keys=999 2023-07-28T23:46:15.016 DEBUG:boto:Data: 2023-07-28T23:46:15.016 DEBUG:boto:Headers: {} 2023-07-28T23:46:15.017 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:15.017 DEBUG:boto:Port: 8000 2023-07-28T23:46:15.017 DEBUG:boto:Params: {} 2023-07-28T23:46:15.018 DEBUG:boto:Token: None 2023-07-28T23:46:15.018 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:15 GMT /hiiqos-10/?versions 2023-07-28T23:46:15.019 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5CKMwhdrrA9zj9hYmQXDDMeUtwI= 2023-07-28T23:46:15.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5CKMwhdrrA9zj9hYmQXDDMeUtwI=', 'Content-Length': '0'} 2023-07-28T23:46:15.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098d52b07c2a5f09f-0064c45347-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:15.022 DEBUG:boto:b'hiiqos-10999falsemyobjnulltrue2023-07-28T23:46:09.345Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:46:15.023 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-07-28T23:46:15.023 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-28T23:46:15.023 DEBUG:boto:path=/hiiqos-10/ 2023-07-28T23:46:15.024 DEBUG:boto:auth_path=/hiiqos-10/ 2023-07-28T23:46:15.024 DEBUG:boto:path=/hiiqos-10/?versions&max-keys=999 2023-07-28T23:46:15.024 DEBUG:boto:auth_path=/hiiqos-10/?versions&max-keys=999 2023-07-28T23:46:15.025 DEBUG:boto:Method: GET 2023-07-28T23:46:15.025 DEBUG:boto:Path: /hiiqos-10/?versions&max-keys=999 2023-07-28T23:46:15.026 DEBUG:boto:Data: 2023-07-28T23:46:15.026 DEBUG:boto:Headers: {} 2023-07-28T23:46:15.026 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:15.027 DEBUG:boto:Port: 8001 2023-07-28T23:46:15.027 DEBUG:boto:Params: {} 2023-07-28T23:46:15.027 DEBUG:boto:Token: None 2023-07-28T23:46:15.028 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:15 GMT /hiiqos-10/?versions 2023-07-28T23:46:15.028 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5CKMwhdrrA9zj9hYmQXDDMeUtwI= 2023-07-28T23:46:15.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5CKMwhdrrA9zj9hYmQXDDMeUtwI=', 'Content-Length': '0'} 2023-07-28T23:46:15.031 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e832f78ab36b2dd8-0064c45347-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:15.076 DEBUG:boto:b'hiiqos-10999falsemyobjnulltrue2023-07-28T23:46:09.345Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:46:15.077 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-07-28T23:46:15.077 DEBUG:boto:path=/hiiqos-10/ 2023-07-28T23:46:15.078 DEBUG:boto:auth_path=/hiiqos-10/ 2023-07-28T23:46:15.078 DEBUG:boto:path=/hiiqos-10/?versions&max-keys=999 2023-07-28T23:46:15.078 DEBUG:boto:auth_path=/hiiqos-10/?versions&max-keys=999 2023-07-28T23:46:15.079 DEBUG:boto:Method: GET 2023-07-28T23:46:15.079 DEBUG:boto:Path: /hiiqos-10/?versions&max-keys=999 2023-07-28T23:46:15.080 DEBUG:boto:Data: 2023-07-28T23:46:15.080 DEBUG:boto:Headers: {} 2023-07-28T23:46:15.080 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:15.081 DEBUG:boto:Port: 8000 2023-07-28T23:46:15.081 DEBUG:boto:Params: {} 2023-07-28T23:46:15.082 DEBUG:boto:Token: None 2023-07-28T23:46:15.082 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:15 GMT /hiiqos-10/?versions 2023-07-28T23:46:15.082 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5CKMwhdrrA9zj9hYmQXDDMeUtwI= 2023-07-28T23:46:15.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5CKMwhdrrA9zj9hYmQXDDMeUtwI=', 'Content-Length': '0'} 2023-07-28T23:46:15.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019636b67b6b39fc1-0064c45347-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:15.085 DEBUG:boto:b'hiiqos-10999falsemyobjnulltrue2023-07-28T23:46:09.345Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:46:15.086 DEBUG:boto:path=/hiiqos-10/ 2023-07-28T23:46:15.087 DEBUG:boto:auth_path=/hiiqos-10/ 2023-07-28T23:46:15.087 DEBUG:boto:path=/hiiqos-10/?versions&max-keys=999 2023-07-28T23:46:15.087 DEBUG:boto:auth_path=/hiiqos-10/?versions&max-keys=999 2023-07-28T23:46:15.088 DEBUG:boto:Method: GET 2023-07-28T23:46:15.088 DEBUG:boto:Path: /hiiqos-10/?versions&max-keys=999 2023-07-28T23:46:15.088 DEBUG:boto:Data: 2023-07-28T23:46:15.089 DEBUG:boto:Headers: {} 2023-07-28T23:46:15.089 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:15.089 DEBUG:boto:Port: 8001 2023-07-28T23:46:15.090 DEBUG:boto:Params: {} 2023-07-28T23:46:15.090 DEBUG:boto:Token: None 2023-07-28T23:46:15.090 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:15 GMT /hiiqos-10/?versions 2023-07-28T23:46:15.091 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5CKMwhdrrA9zj9hYmQXDDMeUtwI= 2023-07-28T23:46:15.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5CKMwhdrrA9zj9hYmQXDDMeUtwI=', 'Content-Length': '0'} 2023-07-28T23:46:15.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009348b167c5cde2ae-0064c45347-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:15.094 DEBUG:boto:b'hiiqos-10999falsemyobjnulltrue2023-07-28T23:46:09.345Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:46:15.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-07-28T23:46:15.095 DEBUG:boto:path=/hiiqos-10/myobj 2023-07-28T23:46:15.095 DEBUG:boto:auth_path=/hiiqos-10/myobj 2023-07-28T23:46:15.096 DEBUG:boto:path=/hiiqos-10/myobj?versionId=null 2023-07-28T23:46:15.096 DEBUG:boto:auth_path=/hiiqos-10/myobj?versionId=null 2023-07-28T23:46:15.097 DEBUG:boto:Method: GET 2023-07-28T23:46:15.097 DEBUG:boto:Path: /hiiqos-10/myobj?versionId=null 2023-07-28T23:46:15.097 DEBUG:boto:Data: 2023-07-28T23:46:15.098 DEBUG:boto:Headers: {} 2023-07-28T23:46:15.098 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:15.098 DEBUG:boto:Port: 8000 2023-07-28T23:46:15.099 DEBUG:boto:Params: {} 2023-07-28T23:46:15.099 DEBUG:boto:Token: None 2023-07-28T23:46:15.100 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:15 GMT /hiiqos-10/myobj?versionId=null 2023-07-28T23:46:15.100 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RrIPVaBdY7TFtsr0km6TogST+j8= 2023-07-28T23:46:15.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RrIPVaBdY7TFtsr0km6TogST+j8=', 'Content-Length': '0'} 2023-07-28T23:46:15.102 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-10:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000039ce89f5664e050-0064c45347-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:15.144 DEBUG:boto:path=/hiiqos-10/myobj 2023-07-28T23:46:15.145 DEBUG:boto:auth_path=/hiiqos-10/myobj 2023-07-28T23:46:15.145 DEBUG:boto:path=/hiiqos-10/myobj?versionId=null 2023-07-28T23:46:15.146 DEBUG:boto:auth_path=/hiiqos-10/myobj?versionId=null 2023-07-28T23:46:15.146 DEBUG:boto:Method: GET 2023-07-28T23:46:15.146 DEBUG:boto:Path: /hiiqos-10/myobj?versionId=null 2023-07-28T23:46:15.147 DEBUG:boto:Data: 2023-07-28T23:46:15.147 DEBUG:boto:Headers: {} 2023-07-28T23:46:15.147 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:15.148 DEBUG:boto:Port: 8001 2023-07-28T23:46:15.148 DEBUG:boto:Params: {} 2023-07-28T23:46:15.148 DEBUG:boto:Token: None 2023-07-28T23:46:15.149 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:15 GMT /hiiqos-10/myobj?versionId=null 2023-07-28T23:46:15.149 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RrIPVaBdY7TFtsr0km6TogST+j8= 2023-07-28T23:46:15.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RrIPVaBdY7TFtsr0km6TogST+j8=', 'Content-Length': '0'} 2023-07-28T23:46:15.152 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fdb9004b82a48eb6-0064c45347-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:15.153 DEBUG:boto:path=/hiiqos-10/myobj 2023-07-28T23:46:15.153 DEBUG:boto:auth_path=/hiiqos-10/myobj 2023-07-28T23:46:15.153 DEBUG:boto:path=/hiiqos-10/myobj?versionId=null 2023-07-28T23:46:15.154 DEBUG:boto:auth_path=/hiiqos-10/myobj?versionId=null 2023-07-28T23:46:15.154 DEBUG:boto:Method: HEAD 2023-07-28T23:46:15.155 DEBUG:boto:Path: /hiiqos-10/myobj?versionId=null 2023-07-28T23:46:15.155 DEBUG:boto:Data: 2023-07-28T23:46:15.155 DEBUG:boto:Headers: {} 2023-07-28T23:46:15.156 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:15.156 DEBUG:boto:Port: 8000 2023-07-28T23:46:15.156 DEBUG:boto:Params: {} 2023-07-28T23:46:15.157 DEBUG:boto:Token: None 2023-07-28T23:46:15.157 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:46:15 GMT /hiiqos-10/myobj?versionId=null 2023-07-28T23:46:15.158 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7yTbEHH4vO2lP+0EsrumtNUVqh4= 2023-07-28T23:46:15.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7yTbEHH4vO2lP+0EsrumtNUVqh4=', 'Content-Length': '0'} 2023-07-28T23:46:15.160 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-10:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001efb96eb32f48401-0064c45347-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:15.160 DEBUG:boto:path=/hiiqos-10/myobj 2023-07-28T23:46:15.160 DEBUG:boto:auth_path=/hiiqos-10/myobj 2023-07-28T23:46:15.161 DEBUG:boto:path=/hiiqos-10/myobj?versionId=null 2023-07-28T23:46:15.161 DEBUG:boto:auth_path=/hiiqos-10/myobj?versionId=null 2023-07-28T23:46:15.161 DEBUG:boto:Method: HEAD 2023-07-28T23:46:15.162 DEBUG:boto:Path: /hiiqos-10/myobj?versionId=null 2023-07-28T23:46:15.162 DEBUG:boto:Data: 2023-07-28T23:46:15.162 DEBUG:boto:Headers: {} 2023-07-28T23:46:15.163 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:15.163 DEBUG:boto:Port: 8001 2023-07-28T23:46:15.163 DEBUG:boto:Params: {} 2023-07-28T23:46:15.164 DEBUG:boto:Token: None 2023-07-28T23:46:15.164 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:46:15 GMT /hiiqos-10/myobj?versionId=null 2023-07-28T23:46:15.165 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7yTbEHH4vO2lP+0EsrumtNUVqh4= 2023-07-28T23:46:15.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7yTbEHH4vO2lP+0EsrumtNUVqh4=', 'Content-Length': '0'} 2023-07-28T23:46:15.167 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003c1f19b54b260fee-0064c45347-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:15.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-07-28T23:46:15.168 DEBUG:boto:path=/hiiqos-10/myobj 2023-07-28T23:46:15.168 DEBUG:boto:auth_path=/hiiqos-10/myobj 2023-07-28T23:46:15.169 DEBUG:boto:path=/hiiqos-10/myobj?versionId=null 2023-07-28T23:46:15.169 DEBUG:boto:auth_path=/hiiqos-10/myobj?versionId=null 2023-07-28T23:46:15.169 DEBUG:boto:Method: GET 2023-07-28T23:46:15.170 DEBUG:boto:Path: /hiiqos-10/myobj?versionId=null 2023-07-28T23:46:15.170 DEBUG:boto:Data: 2023-07-28T23:46:15.170 DEBUG:boto:Headers: {} 2023-07-28T23:46:15.171 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:15.171 DEBUG:boto:Port: 8000 2023-07-28T23:46:15.171 DEBUG:boto:Params: {} 2023-07-28T23:46:15.172 DEBUG:boto:Token: None 2023-07-28T23:46:15.172 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:15 GMT /hiiqos-10/myobj?versionId=null 2023-07-28T23:46:15.173 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RrIPVaBdY7TFtsr0km6TogST+j8= 2023-07-28T23:46:15.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RrIPVaBdY7TFtsr0km6TogST+j8=', 'Content-Length': '0'} 2023-07-28T23:46:15.175 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-10:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e618dd033c71dc5-0064c45347-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:15.175 DEBUG:boto:path=/hiiqos-10/myobj 2023-07-28T23:46:15.176 DEBUG:boto:auth_path=/hiiqos-10/myobj 2023-07-28T23:46:15.176 DEBUG:boto:path=/hiiqos-10/myobj?versionId=null 2023-07-28T23:46:15.177 DEBUG:boto:auth_path=/hiiqos-10/myobj?versionId=null 2023-07-28T23:46:15.177 DEBUG:boto:Method: GET 2023-07-28T23:46:15.177 DEBUG:boto:Path: /hiiqos-10/myobj?versionId=null 2023-07-28T23:46:15.178 DEBUG:boto:Data: 2023-07-28T23:46:15.178 DEBUG:boto:Headers: {} 2023-07-28T23:46:15.178 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:15.179 DEBUG:boto:Port: 8001 2023-07-28T23:46:15.179 DEBUG:boto:Params: {} 2023-07-28T23:46:15.179 DEBUG:boto:Token: None 2023-07-28T23:46:15.180 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:15 GMT /hiiqos-10/myobj?versionId=null 2023-07-28T23:46:15.180 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RrIPVaBdY7TFtsr0km6TogST+j8= 2023-07-28T23:46:15.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RrIPVaBdY7TFtsr0km6TogST+j8=', 'Content-Length': '0'} 2023-07-28T23:46:15.182 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009bd6f6810c49afae-0064c45347-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:15.183 DEBUG:boto:path=/hiiqos-10/ 2023-07-28T23:46:15.183 DEBUG:boto:auth_path=/hiiqos-10/ 2023-07-28T23:46:15.184 DEBUG:boto:path=/hiiqos-10/?prefix=myobj 2023-07-28T23:46:15.184 DEBUG:boto:auth_path=/hiiqos-10/?prefix=myobj 2023-07-28T23:46:15.184 DEBUG:boto:Method: GET 2023-07-28T23:46:15.185 DEBUG:boto:Path: /hiiqos-10/?prefix=myobj 2023-07-28T23:46:15.185 DEBUG:boto:Data: 2023-07-28T23:46:15.187 DEBUG:boto:Headers: {} 2023-07-28T23:46:15.188 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:15.188 DEBUG:boto:Port: 8000 2023-07-28T23:46:15.189 DEBUG:boto:Params: {} 2023-07-28T23:46:15.189 DEBUG:boto:Token: None 2023-07-28T23:46:15.189 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:15 GMT /hiiqos-10/ 2023-07-28T23:46:15.190 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:t+o7eIPLC6/tVVulC+qVVIsboC8= 2023-07-28T23:46:15.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t+o7eIPLC6/tVVulC+qVVIsboC8=', 'Content-Length': '0'} 2023-07-28T23:46:15.192 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9253ffa94847902-0064c45347-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:15.193 DEBUG:boto:b'hiiqos-10myobj1000falsemyobj2023-07-28T23:46:09.345Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:46:15.193 DEBUG:boto:path=/hiiqos-10/ 2023-07-28T23:46:15.194 DEBUG:boto:auth_path=/hiiqos-10/ 2023-07-28T23:46:15.194 DEBUG:boto:path=/hiiqos-10/?prefix=myobj 2023-07-28T23:46:15.195 DEBUG:boto:auth_path=/hiiqos-10/?prefix=myobj 2023-07-28T23:46:15.195 DEBUG:boto:Method: GET 2023-07-28T23:46:15.195 DEBUG:boto:Path: /hiiqos-10/?prefix=myobj 2023-07-28T23:46:15.196 DEBUG:boto:Data: 2023-07-28T23:46:15.196 DEBUG:boto:Headers: {} 2023-07-28T23:46:15.196 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:15.197 DEBUG:boto:Port: 8001 2023-07-28T23:46:15.197 DEBUG:boto:Params: {} 2023-07-28T23:46:15.198 DEBUG:boto:Token: None 2023-07-28T23:46:15.198 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:15 GMT /hiiqos-10/ 2023-07-28T23:46:15.198 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:t+o7eIPLC6/tVVulC+qVVIsboC8= 2023-07-28T23:46:15.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t+o7eIPLC6/tVVulC+qVVIsboC8=', 'Content-Length': '0'} 2023-07-28T23:46:15.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cd5b66c92208fad-0064c45347-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:15.201 DEBUG:boto:b'hiiqos-10myobj1000falsemyobj2023-07-28T23:46:09.345Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:46:15.202 DEBUG:boto:path=/hiiqos-10/myobj 2023-07-28T23:46:15.202 DEBUG:boto:auth_path=/hiiqos-10/myobj 2023-07-28T23:46:15.203 DEBUG:boto:Method: HEAD 2023-07-28T23:46:15.203 DEBUG:boto:Path: /hiiqos-10/myobj 2023-07-28T23:46:15.203 DEBUG:boto:Data: 2023-07-28T23:46:15.204 DEBUG:boto:Headers: {} 2023-07-28T23:46:15.204 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:15.204 DEBUG:boto:Port: 8000 2023-07-28T23:46:15.205 DEBUG:boto:Params: {} 2023-07-28T23:46:15.205 DEBUG:boto:Token: None 2023-07-28T23:46:15.205 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:46:15 GMT /hiiqos-10/myobj 2023-07-28T23:46:15.206 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+GLyK2DIdmfXoqFrNKf80kDNND0= 2023-07-28T23:46:15.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+GLyK2DIdmfXoqFrNKf80kDNND0=', 'Content-Length': '0'} 2023-07-28T23:46:15.208 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-10:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000953fedd35b305e45-0064c45347-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:15.209 DEBUG:boto:path=/hiiqos-10/myobj 2023-07-28T23:46:15.209 DEBUG:boto:auth_path=/hiiqos-10/myobj 2023-07-28T23:46:15.209 DEBUG:boto:Method: HEAD 2023-07-28T23:46:15.210 DEBUG:boto:Path: /hiiqos-10/myobj 2023-07-28T23:46:15.210 DEBUG:boto:Data: 2023-07-28T23:46:15.210 DEBUG:boto:Headers: {} 2023-07-28T23:46:15.211 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:15.211 DEBUG:boto:Port: 8001 2023-07-28T23:46:15.211 DEBUG:boto:Params: {} 2023-07-28T23:46:15.212 DEBUG:boto:Token: None 2023-07-28T23:46:15.212 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:46:15 GMT /hiiqos-10/myobj 2023-07-28T23:46:15.212 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+GLyK2DIdmfXoqFrNKf80kDNND0= 2023-07-28T23:46:15.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+GLyK2DIdmfXoqFrNKf80kDNND0=', 'Content-Length': '0'} 2023-07-28T23:46:15.214 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bde91a41add42d11-0064c45347-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:15.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-07-28T23:46:15.215 DEBUG:boto:path=/hiiqos-10/myobj 2023-07-28T23:46:15.216 DEBUG:boto:auth_path=/hiiqos-10/myobj 2023-07-28T23:46:15.216 DEBUG:boto:Method: GET 2023-07-28T23:46:15.216 DEBUG:boto:Path: /hiiqos-10/myobj 2023-07-28T23:46:15.217 DEBUG:boto:Data: 2023-07-28T23:46:15.217 DEBUG:boto:Headers: {} 2023-07-28T23:46:15.217 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:15.218 DEBUG:boto:Port: 8000 2023-07-28T23:46:15.218 DEBUG:boto:Params: {} 2023-07-28T23:46:15.219 DEBUG:boto:Token: None 2023-07-28T23:46:15.219 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:15 GMT /hiiqos-10/myobj 2023-07-28T23:46:15.220 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mINk4j3MINLJXXzMTy9waJgbvXs= 2023-07-28T23:46:15.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mINk4j3MINLJXXzMTy9waJgbvXs=', 'Content-Length': '0'} 2023-07-28T23:46:15.222 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-10:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c2678045a891fc7c-0064c45347-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:15.222 DEBUG:boto:path=/hiiqos-10/myobj 2023-07-28T23:46:15.223 DEBUG:boto:auth_path=/hiiqos-10/myobj 2023-07-28T23:46:15.223 DEBUG:boto:Method: GET 2023-07-28T23:46:15.223 DEBUG:boto:Path: /hiiqos-10/myobj 2023-07-28T23:46:15.224 DEBUG:boto:Data: 2023-07-28T23:46:15.224 DEBUG:boto:Headers: {} 2023-07-28T23:46:15.224 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:15.225 DEBUG:boto:Port: 8001 2023-07-28T23:46:15.225 DEBUG:boto:Params: {} 2023-07-28T23:46:15.225 DEBUG:boto:Token: None 2023-07-28T23:46:15.226 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:15 GMT /hiiqos-10/myobj 2023-07-28T23:46:15.226 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mINk4j3MINLJXXzMTy9waJgbvXs= 2023-07-28T23:46:15.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mINk4j3MINLJXXzMTy9waJgbvXs=', 'Content-Length': '0'} 2023-07-28T23:46:15.228 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009ac342c927a0d509-0064c45347-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:15.229 DEBUG:boto:path=/hiiqos-10/ 2023-07-28T23:46:15.230 DEBUG:boto:auth_path=/hiiqos-10/ 2023-07-28T23:46:15.230 DEBUG:boto:path=/hiiqos-10/?prefix=myobj 2023-07-28T23:46:15.230 DEBUG:boto:auth_path=/hiiqos-10/?prefix=myobj 2023-07-28T23:46:15.231 DEBUG:boto:Method: GET 2023-07-28T23:46:15.231 DEBUG:boto:Path: /hiiqos-10/?prefix=myobj 2023-07-28T23:46:15.231 DEBUG:boto:Data: 2023-07-28T23:46:15.232 DEBUG:boto:Headers: {} 2023-07-28T23:46:15.232 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:15.232 DEBUG:boto:Port: 8000 2023-07-28T23:46:15.233 DEBUG:boto:Params: {} 2023-07-28T23:46:15.233 DEBUG:boto:Token: None 2023-07-28T23:46:15.234 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:15 GMT /hiiqos-10/ 2023-07-28T23:46:15.234 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:t+o7eIPLC6/tVVulC+qVVIsboC8= 2023-07-28T23:46:15.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t+o7eIPLC6/tVVulC+qVVIsboC8=', 'Content-Length': '0'} 2023-07-28T23:46:15.236 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a7c175887d25c14-0064c45347-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:15.237 DEBUG:boto:b'hiiqos-10myobj1000falsemyobj2023-07-28T23:46:09.345Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:46:15.238 DEBUG:boto:path=/hiiqos-10/ 2023-07-28T23:46:15.238 DEBUG:boto:auth_path=/hiiqos-10/ 2023-07-28T23:46:15.238 DEBUG:boto:path=/hiiqos-10/?prefix=myobj 2023-07-28T23:46:15.239 DEBUG:boto:auth_path=/hiiqos-10/?prefix=myobj 2023-07-28T23:46:15.239 DEBUG:boto:Method: GET 2023-07-28T23:46:15.240 DEBUG:boto:Path: /hiiqos-10/?prefix=myobj 2023-07-28T23:46:15.240 DEBUG:boto:Data: 2023-07-28T23:46:15.240 DEBUG:boto:Headers: {} 2023-07-28T23:46:15.241 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:15.241 DEBUG:boto:Port: 8001 2023-07-28T23:46:15.241 DEBUG:boto:Params: {} 2023-07-28T23:46:15.242 DEBUG:boto:Token: None 2023-07-28T23:46:15.242 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:15 GMT /hiiqos-10/ 2023-07-28T23:46:15.243 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:t+o7eIPLC6/tVVulC+qVVIsboC8= 2023-07-28T23:46:15.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t+o7eIPLC6/tVVulC+qVVIsboC8=', 'Content-Length': '0'} 2023-07-28T23:46:15.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e114488bf10013d4-0064c45347-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:15.292 DEBUG:boto:b'hiiqos-10myobj1000falsemyobj2023-07-28T23:46:09.345Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:46:15.294 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-10 zones={a1, a2} 2023-07-28T23:46:15.294 DEBUG:boto:path=/hiiqos-9/ 2023-07-28T23:46:15.295 DEBUG:boto:auth_path=/hiiqos-9/ 2023-07-28T23:46:15.295 DEBUG:boto:Method: HEAD 2023-07-28T23:46:15.295 DEBUG:boto:Path: /hiiqos-9/ 2023-07-28T23:46:15.296 DEBUG:boto:Data: 2023-07-28T23:46:15.296 DEBUG:boto:Headers: {} 2023-07-28T23:46:15.296 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:15.297 DEBUG:boto:Port: 8000 2023-07-28T23:46:15.297 DEBUG:boto:Params: {} 2023-07-28T23:46:15.297 DEBUG:boto:Token: None 2023-07-28T23:46:15.298 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:46:15 GMT /hiiqos-9/ 2023-07-28T23:46:15.298 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lxHYMpZo74hN4fQwWvtoJWYlU/k= 2023-07-28T23:46:15.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lxHYMpZo74hN4fQwWvtoJWYlU/k=', 'Content-Length': '0'} 2023-07-28T23:46:15.301 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000012e7b6680952471a-0064c45347-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:15.301 DEBUG:boto:path=/hiiqos-9/myobj 2023-07-28T23:46:15.302 DEBUG:boto:auth_path=/hiiqos-9/myobj 2023-07-28T23:46:15.302 DEBUG:boto:Method: HEAD 2023-07-28T23:46:15.302 DEBUG:boto:Path: /hiiqos-9/myobj 2023-07-28T23:46:15.303 DEBUG:boto:Data: 2023-07-28T23:46:15.303 DEBUG:boto:Headers: {} 2023-07-28T23:46:15.304 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:15.304 DEBUG:boto:Port: 8000 2023-07-28T23:46:15.304 DEBUG:boto:Params: {} 2023-07-28T23:46:15.305 DEBUG:boto:Token: None 2023-07-28T23:46:15.305 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:46:15 GMT /hiiqos-9/myobj 2023-07-28T23:46:15.305 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZnNrrDejj7gmYNI62f7S+Y6i4vY= 2023-07-28T23:46:15.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZnNrrDejj7gmYNI62f7S+Y6i4vY=', 'Content-Length': '0'} 2023-07-28T23:46:15.307 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000185b315eb4a03d18-0064c45347-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:15.308 DEBUG:boto:path=/hiiqos-9/myobj 2023-07-28T23:46:15.308 DEBUG:boto:auth_path=/hiiqos-9/myobj 2023-07-28T23:46:15.309 DEBUG:boto:Method: DELETE 2023-07-28T23:46:15.309 DEBUG:boto:Path: /hiiqos-9/myobj 2023-07-28T23:46:15.309 DEBUG:boto:Data: 2023-07-28T23:46:15.310 DEBUG:boto:Headers: {} 2023-07-28T23:46:15.310 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:15.310 DEBUG:boto:Port: 8000 2023-07-28T23:46:15.311 DEBUG:boto:Params: {} 2023-07-28T23:46:15.311 DEBUG:boto:Token: None 2023-07-28T23:46:15.312 DEBUG:boto:StringToSign: DELETE Fri, 28 Jul 2023 23:46:15 GMT /hiiqos-9/myobj 2023-07-28T23:46:15.312 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Xe9svt5Sa1BFWNz5N8LND9ngMx0= 2023-07-28T23:46:15.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Xe9svt5Sa1BFWNz5N8LND9ngMx0=', 'Content-Length': '0'} 2023-07-28T23:46:15.317 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000036f1e396296288a9-0064c45347-1117-a1'), ('Date', 'Fri, 28 Jul 2023 23:46:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:15.317 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:46:15.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:46:15.318 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-9 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-28T23:46:15.371 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:46:15.371 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:46:15.404 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:46:15.403+0000 7ff947726880 1 waiting for incremental sync to catch up: 2023-07-28T23:46:15.404 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000001.4.6, , , , , , , , , ] 2023-07-28T23:46:15.404 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000002.6.6, , , , , , , , , ] 2023-07-28T23:46:20.405 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:46:20.404+0000 7ff947726880 1 waiting for incremental sync to catch up: 2023-07-28T23:46:20.406 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000001.4.6, , , , , , , , , ] 2023-07-28T23:46:20.406 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000002.6.6, , , , , , , , , ] 2023-07-28T23:46:25.406 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:46:25.406+0000 7ff947726880 1 waiting for incremental sync to catch up: 2023-07-28T23:46:25.407 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000001.4.6, , , , , , , , , ] 2023-07-28T23:46:25.407 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000002.6.6, , , , , , , , , ] 2023-07-28T23:46:30.408 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:46:30.407+0000 7ff947726880 1 waiting for incremental sync to catch up: 2023-07-28T23:46:30.408 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000001.4.6, , , , , , , , , ] 2023-07-28T23:46:30.409 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000002.6.6, , , , , , , , , ] 2023-07-28T23:46:35.409 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:46:35.408+0000 7ff947726880 1 bucket sync caught up with source: 2023-07-28T23:46:35.410 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.6.6, , , , , , , , , ] 2023-07-28T23:46:35.410 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000002.6.6, , , , , , , , , ] 2023-07-28T23:46:35.410 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:46:35.408+0000 7ff947726880 0 bucket checkpoint complete 2023-07-28T23:46:35.425 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-9 zones={a2, a1} 2023-07-28T23:46:35.425 DEBUG:boto:path=/hiiqos-9/ 2023-07-28T23:46:35.426 DEBUG:boto:auth_path=/hiiqos-9/ 2023-07-28T23:46:35.427 DEBUG:boto:Method: HEAD 2023-07-28T23:46:35.428 DEBUG:boto:Path: /hiiqos-9/ 2023-07-28T23:46:35.429 DEBUG:boto:Data: 2023-07-28T23:46:35.430 DEBUG:boto:Headers: {} 2023-07-28T23:46:35.430 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:35.431 DEBUG:boto:Port: 8001 2023-07-28T23:46:35.432 DEBUG:boto:Params: {} 2023-07-28T23:46:35.433 DEBUG:boto:Token: None 2023-07-28T23:46:35.435 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:46:35 GMT /hiiqos-9/ 2023-07-28T23:46:35.435 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:iHnPocKXGxdO70lr/XPI+fcaCwA= 2023-07-28T23:46:35.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:iHnPocKXGxdO70lr/XPI+fcaCwA=', 'Content-Length': '0'} 2023-07-28T23:46:35.440 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001ad7de97b94e7d1f-0064c4535b-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:46:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:35.441 DEBUG:boto:path=/hiiqos-9/ 2023-07-28T23:46:35.441 DEBUG:boto:auth_path=/hiiqos-9/ 2023-07-28T23:46:35.442 DEBUG:boto:Method: HEAD 2023-07-28T23:46:35.442 DEBUG:boto:Path: /hiiqos-9/ 2023-07-28T23:46:35.443 DEBUG:boto:Data: 2023-07-28T23:46:35.444 DEBUG:boto:Headers: {} 2023-07-28T23:46:35.444 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:35.445 DEBUG:boto:Port: 8000 2023-07-28T23:46:35.446 DEBUG:boto:Params: {} 2023-07-28T23:46:35.446 DEBUG:boto:Token: None 2023-07-28T23:46:35.447 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:46:35 GMT /hiiqos-9/ 2023-07-28T23:46:35.447 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:iHnPocKXGxdO70lr/XPI+fcaCwA= 2023-07-28T23:46:35.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:iHnPocKXGxdO70lr/XPI+fcaCwA=', 'Content-Length': '0'} 2023-07-28T23:46:35.450 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000133d7ad1859f0a71-0064c4535b-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:46:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:35.450 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-28T23:46:35.451 DEBUG:boto:path=/hiiqos-9/ 2023-07-28T23:46:35.451 DEBUG:boto:auth_path=/hiiqos-9/ 2023-07-28T23:46:35.451 DEBUG:boto:path=/hiiqos-9/?versions&max-keys=999 2023-07-28T23:46:35.452 DEBUG:boto:auth_path=/hiiqos-9/?versions&max-keys=999 2023-07-28T23:46:35.452 DEBUG:boto:Method: GET 2023-07-28T23:46:35.453 DEBUG:boto:Path: /hiiqos-9/?versions&max-keys=999 2023-07-28T23:46:35.453 DEBUG:boto:Data: 2023-07-28T23:46:35.453 DEBUG:boto:Headers: {} 2023-07-28T23:46:35.454 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:35.454 DEBUG:boto:Port: 8001 2023-07-28T23:46:35.454 DEBUG:boto:Params: {} 2023-07-28T23:46:35.455 DEBUG:boto:Token: None 2023-07-28T23:46:35.455 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:35 GMT /hiiqos-9/?versions 2023-07-28T23:46:35.456 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SKpuX18uUEMMOKaVVIU1Ieg/IWw= 2023-07-28T23:46:35.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SKpuX18uUEMMOKaVVIU1Ieg/IWw=', 'Content-Length': '0'} 2023-07-28T23:46:35.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec718802f16e9e88-0064c4535b-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:35.460 DEBUG:boto:b'hiiqos-9999false' 2023-07-28T23:46:35.461 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-28T23:46:35.462 DEBUG:boto:path=/hiiqos-9/ 2023-07-28T23:46:35.462 DEBUG:boto:auth_path=/hiiqos-9/ 2023-07-28T23:46:35.462 DEBUG:boto:path=/hiiqos-9/?versions&max-keys=999 2023-07-28T23:46:35.463 DEBUG:boto:auth_path=/hiiqos-9/?versions&max-keys=999 2023-07-28T23:46:35.463 DEBUG:boto:Method: GET 2023-07-28T23:46:35.464 DEBUG:boto:Path: /hiiqos-9/?versions&max-keys=999 2023-07-28T23:46:35.464 DEBUG:boto:Data: 2023-07-28T23:46:35.464 DEBUG:boto:Headers: {} 2023-07-28T23:46:35.465 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:35.465 DEBUG:boto:Port: 8000 2023-07-28T23:46:35.465 DEBUG:boto:Params: {} 2023-07-28T23:46:35.466 DEBUG:boto:Token: None 2023-07-28T23:46:35.466 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:35 GMT /hiiqos-9/?versions 2023-07-28T23:46:35.467 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SKpuX18uUEMMOKaVVIU1Ieg/IWw= 2023-07-28T23:46:35.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SKpuX18uUEMMOKaVVIU1Ieg/IWw=', 'Content-Length': '0'} 2023-07-28T23:46:35.469 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090e04c7927fa661f-0064c4535b-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:35.470 DEBUG:boto:b'hiiqos-9999false' 2023-07-28T23:46:35.470 DEBUG:boto:path=/hiiqos-9/ 2023-07-28T23:46:35.471 DEBUG:boto:auth_path=/hiiqos-9/ 2023-07-28T23:46:35.471 DEBUG:boto:path=/hiiqos-9/?versions&max-keys=999 2023-07-28T23:46:35.471 DEBUG:boto:auth_path=/hiiqos-9/?versions&max-keys=999 2023-07-28T23:46:35.472 DEBUG:boto:Method: GET 2023-07-28T23:46:35.472 DEBUG:boto:Path: /hiiqos-9/?versions&max-keys=999 2023-07-28T23:46:35.472 DEBUG:boto:Data: 2023-07-28T23:46:35.473 DEBUG:boto:Headers: {} 2023-07-28T23:46:35.473 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:35.474 DEBUG:boto:Port: 8001 2023-07-28T23:46:35.474 DEBUG:boto:Params: {} 2023-07-28T23:46:35.475 DEBUG:boto:Token: None 2023-07-28T23:46:35.475 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:35 GMT /hiiqos-9/?versions 2023-07-28T23:46:35.475 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SKpuX18uUEMMOKaVVIU1Ieg/IWw= 2023-07-28T23:46:35.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SKpuX18uUEMMOKaVVIU1Ieg/IWw=', 'Content-Length': '0'} 2023-07-28T23:46:35.478 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ce585f98de86a07-0064c4535b-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:35.479 DEBUG:boto:b'hiiqos-9999false' 2023-07-28T23:46:35.480 DEBUG:boto:path=/hiiqos-9/ 2023-07-28T23:46:35.480 DEBUG:boto:auth_path=/hiiqos-9/ 2023-07-28T23:46:35.480 DEBUG:boto:path=/hiiqos-9/?versions&max-keys=999 2023-07-28T23:46:35.481 DEBUG:boto:auth_path=/hiiqos-9/?versions&max-keys=999 2023-07-28T23:46:35.481 DEBUG:boto:Method: GET 2023-07-28T23:46:35.481 DEBUG:boto:Path: /hiiqos-9/?versions&max-keys=999 2023-07-28T23:46:35.482 DEBUG:boto:Data: 2023-07-28T23:46:35.482 DEBUG:boto:Headers: {} 2023-07-28T23:46:35.483 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:35.483 DEBUG:boto:Port: 8000 2023-07-28T23:46:35.484 DEBUG:boto:Params: {} 2023-07-28T23:46:35.484 DEBUG:boto:Token: None 2023-07-28T23:46:35.484 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:35 GMT /hiiqos-9/?versions 2023-07-28T23:46:35.485 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SKpuX18uUEMMOKaVVIU1Ieg/IWw= 2023-07-28T23:46:35.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SKpuX18uUEMMOKaVVIU1Ieg/IWw=', 'Content-Length': '0'} 2023-07-28T23:46:35.488 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c0e01b486eda32a-0064c4535b-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:35.489 DEBUG:boto:b'hiiqos-9999false' 2023-07-28T23:46:35.489 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-9 zones={a2, a1} 2023-07-28T23:46:35.490 DEBUG:boto:path=/hiiqos-10/ 2023-07-28T23:46:35.490 DEBUG:boto:auth_path=/hiiqos-10/ 2023-07-28T23:46:35.490 DEBUG:boto:Method: HEAD 2023-07-28T23:46:35.491 DEBUG:boto:Path: /hiiqos-10/ 2023-07-28T23:46:35.491 DEBUG:boto:Data: 2023-07-28T23:46:35.491 DEBUG:boto:Headers: {} 2023-07-28T23:46:35.492 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:35.492 DEBUG:boto:Port: 8001 2023-07-28T23:46:35.492 DEBUG:boto:Params: {} 2023-07-28T23:46:35.493 DEBUG:boto:Token: None 2023-07-28T23:46:35.493 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:46:35 GMT /hiiqos-10/ 2023-07-28T23:46:35.493 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jYAoLUmNPlDI5+Nc1nRqsd1Okkc= 2023-07-28T23:46:35.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jYAoLUmNPlDI5+Nc1nRqsd1Okkc=', 'Content-Length': '0'} 2023-07-28T23:46:35.496 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d8012743077637f-0064c4535b-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:46:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:35.497 DEBUG:boto:path=/hiiqos-10/myobj 2023-07-28T23:46:35.497 DEBUG:boto:auth_path=/hiiqos-10/myobj 2023-07-28T23:46:35.498 DEBUG:boto:Method: HEAD 2023-07-28T23:46:35.498 DEBUG:boto:Path: /hiiqos-10/myobj 2023-07-28T23:46:35.498 DEBUG:boto:Data: 2023-07-28T23:46:35.499 DEBUG:boto:Headers: {} 2023-07-28T23:46:35.499 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:35.499 DEBUG:boto:Port: 8001 2023-07-28T23:46:35.500 DEBUG:boto:Params: {} 2023-07-28T23:46:35.500 DEBUG:boto:Token: None 2023-07-28T23:46:35.500 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:46:35 GMT /hiiqos-10/myobj 2023-07-28T23:46:35.501 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Hz79rGyXV1odArxHFsK3xy4+FFw= 2023-07-28T23:46:35.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Hz79rGyXV1odArxHFsK3xy4+FFw=', 'Content-Length': '0'} 2023-07-28T23:46:35.503 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:46:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000082a6c9841d571db6-0064c4535b-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:46:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:35.503 DEBUG:boto:path=/hiiqos-10/myobj 2023-07-28T23:46:35.504 DEBUG:boto:auth_path=/hiiqos-10/myobj 2023-07-28T23:46:35.504 DEBUG:boto:Method: DELETE 2023-07-28T23:46:35.505 DEBUG:boto:Path: /hiiqos-10/myobj 2023-07-28T23:46:35.506 DEBUG:boto:Data: 2023-07-28T23:46:35.506 DEBUG:boto:Headers: {} 2023-07-28T23:46:35.507 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:35.508 DEBUG:boto:Port: 8001 2023-07-28T23:46:35.508 DEBUG:boto:Params: {} 2023-07-28T23:46:35.509 DEBUG:boto:Token: None 2023-07-28T23:46:35.510 DEBUG:boto:StringToSign: DELETE Fri, 28 Jul 2023 23:46:35 GMT /hiiqos-10/myobj 2023-07-28T23:46:35.511 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bUtjqsh8O67XCzFPeRnDDOlp4kc= 2023-07-28T23:46:35.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bUtjqsh8O67XCzFPeRnDDOlp4kc=', 'Content-Length': '0'} 2023-07-28T23:46:35.517 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d76b477daa9b91e0-0064c4535b-1113-a2'), ('Date', 'Fri, 28 Jul 2023 23:46:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:35.517 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:46:35.518 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:46:35.518 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-10 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-28T23:46:35.577 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:46:35.577 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:46:35.612 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:46:35.611+0000 7fa5e7fa2880 1 waiting for incremental sync to catch up: 2023-07-28T23:46:35.612 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000001.7.6, , , , , , , , , ] 2023-07-28T23:46:35.612 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000002.11.6, , , , , , , , , ] 2023-07-28T23:46:40.613 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:46:40.612+0000 7fa5e7fa2880 1 waiting for incremental sync to catch up: 2023-07-28T23:46:40.614 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000001.7.6, , , , , , , , , ] 2023-07-28T23:46:40.614 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000002.11.6, , , , , , , , , ] 2023-07-28T23:46:45.614 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:46:45.614+0000 7fa5e7fa2880 1 waiting for incremental sync to catch up: 2023-07-28T23:46:45.615 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000001.7.6, , , , , , , , , ] 2023-07-28T23:46:45.615 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000002.11.6, , , , , , , , , ] 2023-07-28T23:46:50.615 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:46:50.615+0000 7fa5e7fa2880 1 waiting for incremental sync to catch up: 2023-07-28T23:46:50.616 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000001.7.6, , , , , , , , , ] 2023-07-28T23:46:50.616 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000002.11.6, , , , , , , , , ] 2023-07-28T23:46:55.617 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:46:55.616+0000 7fa5e7fa2880 1 bucket sync caught up with source: 2023-07-28T23:46:55.617 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.11.6, , , , , , , , , ] 2023-07-28T23:46:55.617 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000002.11.6, , , , , , , , , ] 2023-07-28T23:46:55.618 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:46:55.616+0000 7fa5e7fa2880 0 bucket checkpoint complete 2023-07-28T23:46:55.634 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-10 zones={a1, a2} 2023-07-28T23:46:55.635 DEBUG:boto:path=/hiiqos-10/ 2023-07-28T23:46:55.635 DEBUG:boto:auth_path=/hiiqos-10/ 2023-07-28T23:46:55.636 DEBUG:boto:Method: HEAD 2023-07-28T23:46:55.636 DEBUG:boto:Path: /hiiqos-10/ 2023-07-28T23:46:55.636 DEBUG:boto:Data: 2023-07-28T23:46:55.637 DEBUG:boto:Headers: {} 2023-07-28T23:46:55.637 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:55.637 DEBUG:boto:Port: 8000 2023-07-28T23:46:55.638 DEBUG:boto:Params: {} 2023-07-28T23:46:55.638 DEBUG:boto:Token: None 2023-07-28T23:46:55.638 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:46:55 GMT /hiiqos-10/ 2023-07-28T23:46:55.639 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aT9WR+vKvjveFTmhzfpKfuCWxCY= 2023-07-28T23:46:55.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aT9WR+vKvjveFTmhzfpKfuCWxCY=', 'Content-Length': '0'} 2023-07-28T23:46:55.641 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000011ac2f4d37fa6708-0064c4536f-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:46:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:55.642 DEBUG:boto:path=/hiiqos-10/ 2023-07-28T23:46:55.642 DEBUG:boto:auth_path=/hiiqos-10/ 2023-07-28T23:46:55.643 DEBUG:boto:Method: HEAD 2023-07-28T23:46:55.643 DEBUG:boto:Path: /hiiqos-10/ 2023-07-28T23:46:55.643 DEBUG:boto:Data: 2023-07-28T23:46:55.644 DEBUG:boto:Headers: {} 2023-07-28T23:46:55.644 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:55.644 DEBUG:boto:Port: 8001 2023-07-28T23:46:55.645 DEBUG:boto:Params: {} 2023-07-28T23:46:55.645 DEBUG:boto:Token: None 2023-07-28T23:46:55.646 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:46:55 GMT /hiiqos-10/ 2023-07-28T23:46:55.646 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aT9WR+vKvjveFTmhzfpKfuCWxCY= 2023-07-28T23:46:55.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aT9WR+vKvjveFTmhzfpKfuCWxCY=', 'Content-Length': '0'} 2023-07-28T23:46:55.649 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000379e2b7483f5e985-0064c4536f-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:46:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:55.649 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-28T23:46:55.650 DEBUG:boto:path=/hiiqos-10/ 2023-07-28T23:46:55.650 DEBUG:boto:auth_path=/hiiqos-10/ 2023-07-28T23:46:55.650 DEBUG:boto:path=/hiiqos-10/?versions&max-keys=999 2023-07-28T23:46:55.651 DEBUG:boto:auth_path=/hiiqos-10/?versions&max-keys=999 2023-07-28T23:46:55.651 DEBUG:boto:Method: GET 2023-07-28T23:46:55.652 DEBUG:boto:Path: /hiiqos-10/?versions&max-keys=999 2023-07-28T23:46:55.652 DEBUG:boto:Data: 2023-07-28T23:46:55.653 DEBUG:boto:Headers: {} 2023-07-28T23:46:55.653 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:55.654 DEBUG:boto:Port: 8000 2023-07-28T23:46:55.654 DEBUG:boto:Params: {} 2023-07-28T23:46:55.654 DEBUG:boto:Token: None 2023-07-28T23:46:55.655 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:55 GMT /hiiqos-10/?versions 2023-07-28T23:46:55.655 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HaoIa6a/IxX0q5DivNj+qhuUP/s= 2023-07-28T23:46:55.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HaoIa6a/IxX0q5DivNj+qhuUP/s=', 'Content-Length': '0'} 2023-07-28T23:46:55.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0cbbf931df10026-0064c4536f-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:55.659 DEBUG:boto:b'hiiqos-10999false' 2023-07-28T23:46:55.659 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-28T23:46:55.660 DEBUG:boto:path=/hiiqos-10/ 2023-07-28T23:46:55.660 DEBUG:boto:auth_path=/hiiqos-10/ 2023-07-28T23:46:55.661 DEBUG:boto:path=/hiiqos-10/?versions&max-keys=999 2023-07-28T23:46:55.661 DEBUG:boto:auth_path=/hiiqos-10/?versions&max-keys=999 2023-07-28T23:46:55.661 DEBUG:boto:Method: GET 2023-07-28T23:46:55.662 DEBUG:boto:Path: /hiiqos-10/?versions&max-keys=999 2023-07-28T23:46:55.662 DEBUG:boto:Data: 2023-07-28T23:46:55.662 DEBUG:boto:Headers: {} 2023-07-28T23:46:55.663 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:55.663 DEBUG:boto:Port: 8001 2023-07-28T23:46:55.663 DEBUG:boto:Params: {} 2023-07-28T23:46:55.664 DEBUG:boto:Token: None 2023-07-28T23:46:55.664 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:55 GMT /hiiqos-10/?versions 2023-07-28T23:46:55.665 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HaoIa6a/IxX0q5DivNj+qhuUP/s= 2023-07-28T23:46:55.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HaoIa6a/IxX0q5DivNj+qhuUP/s=', 'Content-Length': '0'} 2023-07-28T23:46:55.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d0510e1bb1c4fcf-0064c4536f-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:55.668 DEBUG:boto:b'hiiqos-10999false' 2023-07-28T23:46:55.669 DEBUG:boto:path=/hiiqos-10/ 2023-07-28T23:46:55.669 DEBUG:boto:auth_path=/hiiqos-10/ 2023-07-28T23:46:55.669 DEBUG:boto:path=/hiiqos-10/?versions&max-keys=999 2023-07-28T23:46:55.670 DEBUG:boto:auth_path=/hiiqos-10/?versions&max-keys=999 2023-07-28T23:46:55.670 DEBUG:boto:Method: GET 2023-07-28T23:46:55.670 DEBUG:boto:Path: /hiiqos-10/?versions&max-keys=999 2023-07-28T23:46:55.671 DEBUG:boto:Data: 2023-07-28T23:46:55.671 DEBUG:boto:Headers: {} 2023-07-28T23:46:55.671 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:55.672 DEBUG:boto:Port: 8000 2023-07-28T23:46:55.672 DEBUG:boto:Params: {} 2023-07-28T23:46:55.673 DEBUG:boto:Token: None 2023-07-28T23:46:55.673 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:55 GMT /hiiqos-10/?versions 2023-07-28T23:46:55.673 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HaoIa6a/IxX0q5DivNj+qhuUP/s= 2023-07-28T23:46:55.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HaoIa6a/IxX0q5DivNj+qhuUP/s=', 'Content-Length': '0'} 2023-07-28T23:46:55.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d74e11b02fa1dbdb-0064c4536f-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:55.677 DEBUG:boto:b'hiiqos-10999false' 2023-07-28T23:46:55.677 DEBUG:boto:path=/hiiqos-10/ 2023-07-28T23:46:55.678 DEBUG:boto:auth_path=/hiiqos-10/ 2023-07-28T23:46:55.678 DEBUG:boto:path=/hiiqos-10/?versions&max-keys=999 2023-07-28T23:46:55.678 DEBUG:boto:auth_path=/hiiqos-10/?versions&max-keys=999 2023-07-28T23:46:55.679 DEBUG:boto:Method: GET 2023-07-28T23:46:55.679 DEBUG:boto:Path: /hiiqos-10/?versions&max-keys=999 2023-07-28T23:46:55.679 DEBUG:boto:Data: 2023-07-28T23:46:55.680 DEBUG:boto:Headers: {} 2023-07-28T23:46:55.680 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:55.680 DEBUG:boto:Port: 8001 2023-07-28T23:46:55.681 DEBUG:boto:Params: {} 2023-07-28T23:46:55.681 DEBUG:boto:Token: None 2023-07-28T23:46:55.682 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:46:55 GMT /hiiqos-10/?versions 2023-07-28T23:46:55.682 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HaoIa6a/IxX0q5DivNj+qhuUP/s= 2023-07-28T23:46:55.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HaoIa6a/IxX0q5DivNj+qhuUP/s=', 'Content-Length': '0'} 2023-07-28T23:46:55.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e94f5c7b0d476e01-0064c4536f-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:46:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:55.685 DEBUG:boto:b'hiiqos-10999false' 2023-07-28T23:46:55.686 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-10 zones={a1, a2} 2023-07-28T23:46:55.686 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_delete ... ok 2023-07-28T23:46:55.687 INFO:rgw_multi.tests:create bucket zone=a1 name=hiiqos-11 2023-07-28T23:46:55.687 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:46:55.688 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:46:55.688 DEBUG:boto:Method: PUT 2023-07-28T23:46:55.688 DEBUG:boto:Path: /hiiqos-11/ 2023-07-28T23:46:55.689 DEBUG:boto:Data: 2023-07-28T23:46:55.689 DEBUG:boto:Headers: {} 2023-07-28T23:46:55.689 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:55.690 DEBUG:boto:Port: 8000 2023-07-28T23:46:55.690 DEBUG:boto:Params: {} 2023-07-28T23:46:55.690 DEBUG:boto:Token: None 2023-07-28T23:46:55.691 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:46:55 GMT /hiiqos-11/ 2023-07-28T23:46:55.691 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JSTdqdPTggBB8/7J3CIX7r4uHWA= 2023-07-28T23:46:55.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JSTdqdPTggBB8/7J3CIX7r4uHWA=', 'Content-Length': '0'} 2023-07-28T23:46:55.705 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000002e6e96d00328522-0064c4536f-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:46:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:55.706 INFO:rgw_multi.tests:create bucket zone=a2 name=hiiqos-12 2023-07-28T23:46:55.706 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:46:55.707 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:46:55.707 DEBUG:boto:Method: PUT 2023-07-28T23:46:55.707 DEBUG:boto:Path: /hiiqos-12/ 2023-07-28T23:46:55.708 DEBUG:boto:Data: 2023-07-28T23:46:55.708 DEBUG:boto:Headers: {} 2023-07-28T23:46:55.709 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:55.709 DEBUG:boto:Port: 8001 2023-07-28T23:46:55.709 DEBUG:boto:Params: {} 2023-07-28T23:46:55.710 DEBUG:boto:Token: None 2023-07-28T23:46:55.710 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:46:55 GMT /hiiqos-12/ 2023-07-28T23:46:55.710 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gXNtvmVPEV3xAu2lZYSuZgkK4Vg= 2023-07-28T23:46:55.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gXNtvmVPEV3xAu2lZYSuZgkK4Vg=', 'Content-Length': '0'} 2023-07-28T23:46:55.734 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000638c269c7e46ed17-0064c4536f-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:46:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:55.735 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:46:55.735 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:46:55.735 DEBUG:boto:path=/hiiqos-11/?versioning 2023-07-28T23:46:55.736 DEBUG:boto:auth_path=/hiiqos-11/?versioning 2023-07-28T23:46:55.736 DEBUG:boto:Method: PUT 2023-07-28T23:46:55.736 DEBUG:boto:Path: /hiiqos-11/?versioning 2023-07-28T23:46:55.737 DEBUG:boto:Data: Enabled Disabled 2023-07-28T23:46:55.737 DEBUG:boto:Headers: {} 2023-07-28T23:46:55.737 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:46:55.738 DEBUG:boto:Port: 8000 2023-07-28T23:46:55.738 DEBUG:boto:Params: {} 2023-07-28T23:46:55.738 DEBUG:boto:Token: None 2023-07-28T23:46:55.739 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:46:55 GMT /hiiqos-11/?versioning 2023-07-28T23:46:55.739 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Nt2UQP3qbWlBt3IsJzgUey22j2M= 2023-07-28T23:46:55.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Nt2UQP3qbWlBt3IsJzgUey22j2M=', 'Content-Length': '228'} 2023-07-28T23:46:55.744 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005876d41de0e60c02-0064c4536f-1117-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:46:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:55.745 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:46:55.745 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:46:55.745 DEBUG:boto:path=/hiiqos-12/?versioning 2023-07-28T23:46:55.746 DEBUG:boto:auth_path=/hiiqos-12/?versioning 2023-07-28T23:46:55.746 DEBUG:boto:Method: PUT 2023-07-28T23:46:55.746 DEBUG:boto:Path: /hiiqos-12/?versioning 2023-07-28T23:46:55.746 DEBUG:boto:Data: Enabled Disabled 2023-07-28T23:46:55.747 DEBUG:boto:Headers: {} 2023-07-28T23:46:55.747 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:46:55.747 DEBUG:boto:Port: 8001 2023-07-28T23:46:55.748 DEBUG:boto:Params: {} 2023-07-28T23:46:55.748 DEBUG:boto:Token: None 2023-07-28T23:46:55.748 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:46:55 GMT /hiiqos-12/?versioning 2023-07-28T23:46:55.749 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/vJ6C598Un/e19z8ivnM+ISsp2g= 2023-07-28T23:46:55.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:46:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/vJ6C598Un/e19z8ivnM+ISsp2g=', 'Content-Length': '228'} 2023-07-28T23:46:55.757 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000038703875bd55714c-0064c4536f-1113-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:46:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:46:55.757 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:46:55.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:46:55.758 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:46:55.809 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:46:55.809 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:46:55.841 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-28T23:46:55.853 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-28T23:46:55.853 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:46:55.854 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-28T23:46:55.854 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:46:09.233742Z" 2023-07-28T23:46:55.854 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:46:55.854 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:46:55.854 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588015.754899_144.1", 2023-07-28T23:46:55.855 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:46:55.754899Z" 2023-07-28T23:46:55.855 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:46:55.855 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:46:55.855 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:46:55.855 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:46:55.856 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:46:55.856 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:46:55.856 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:46:55.856 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:46:55.856 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:46:55.856 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-28T23:46:55.858 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690587969.233742_99.1', 1: '1_1690588015.754899_144.1', 2: '', 3: ''} 2023-07-28T23:46:55.859 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-28T23:46:55.859 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:46:55.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:46:55.861 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:46:55.907 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:46:55.907 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:46:55.934 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:46:55.933+0000 7fc9fd549880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:46:55.934 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:46:55.934+0000 7fc9fd549880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:46:55.945 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:46:55.946 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:46:55.946 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:46:55.946 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:46:55.946 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:46:55.946 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:46:55.947 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:46:55.947 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:46:55.947 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:46:55.947 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:46:55.947 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:46:55.947 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:46:55.948 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:46:55.948 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-28T23:46:55.948 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:46:55.948 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:46:55.948 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:46:55.948 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-28T23:46:55.948 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:46:55.949 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:46:55.949 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:46:55.949 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:46:55.949 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:46:55.949 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:46:55.949 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:46:55.949 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588015.698104_127.1", 2023-07-28T23:46:55.950 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:46:55.950 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:46:55.950 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:46:55.950 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:55.698104Z", 2023-07-28T23:46:55.950 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:46:55.950 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:46:55.950 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:46:55.951 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:46:55.951 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:46:55.951 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:46:55.951 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:46:55.951 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:46:55.951 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:46:55.951 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:46:55.952 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:46:55.952 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:46:55.952 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:46:55.952 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:46:55.952 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:46:55.952 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:46:55.952 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:46:55.953 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:46:55.953 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:46:55.953 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:46:55.953 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:46:55.953 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:46:55.953 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:46:55.953 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:46:55.954 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:46:55.954 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:46:55.954 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:46:55.954 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:46:55.954 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:46:55.954 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:46:55.954 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:46:55.955 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:46:55.955 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:46:55.955 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:46:55Z" 2023-07-28T23:46:55.955 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:46:55.956 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588015.698104_127.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:46:55.698104Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-28T23:46:55Z" } 2023-07-28T23:46:55.956 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588015.698104_127.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:46:55.698104Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-28T23:46:55.957 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588015.754899_144.1', 2: '', 3: ''} 2023-07-28T23:46:55.957 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588015.698104_127.1', 2: '', 3: ''} 2023-07-28T23:46:55.957 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588015.754899_144.1 target=1_1690588015.698104_127.1 2023-07-28T23:47:00.958 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:47:00.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:47:00.960 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:47:01.014 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:47:01.014 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:47:01.045 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:47:01.044+0000 7f4075324880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:47:01.046 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:47:01.045+0000 7f4075324880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:47:01.059 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:47:01.059 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:47:01.059 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:47:01.059 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:47:01.060 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:47:01.060 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:47:01.060 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:47:01.060 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:47:01.061 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:47:01.061 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:47:01.061 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:47:01.062 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:47:01.065 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:47:01.065 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-28T23:47:01.066 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:47:01.066 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:47:01.066 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:47:01.066 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-28T23:47:01.067 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:47:01.067 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:47:01.067 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:47:01.068 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:47:01.068 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:47:01.068 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:47:01.069 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:47:01.069 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588015.754899_144.1", 2023-07-28T23:47:01.069 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:47:01.069 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:47:01.069 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:47:01.069 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:55.754899Z", 2023-07-28T23:47:01.069 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:47:01.070 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:47:01.070 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:47:01.070 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:47:01.070 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:47:01.070 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:47:01.071 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:47:01.071 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:47:01.071 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:47:01.071 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:47:01.071 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:47:01.071 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:47:01.072 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:47:01.072 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:47:01.072 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:47:01.072 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:47:01.072 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:47:01.072 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:47:01.072 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:47:01.073 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:47:01.073 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:47:01.073 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:47:01.073 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:47:01.073 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:47:01.073 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:47:01.074 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:47:01.074 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:47:01.074 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:47:01.075 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:47:01.075 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:47:01.075 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:47:01.075 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:47:01.076 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:47:01.076 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:47:01Z" 2023-07-28T23:47:01.076 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:47:01.078 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588015.754899_144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:46:55.754899Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-28T23:47:01Z" } 2023-07-28T23:47:01.079 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588015.754899_144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:46:55.754899Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-28T23:47:01.080 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588015.754899_144.1', 2: '', 3: ''} 2023-07-28T23:47:01.080 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588015.754899_144.1', 2: '', 3: ''} 2023-07-28T23:47:01.080 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-28T23:47:01.081 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:47:01.081 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:47:01.081 DEBUG:boto:Method: HEAD 2023-07-28T23:47:01.082 DEBUG:boto:Path: /hiiqos-11/ 2023-07-28T23:47:01.082 DEBUG:boto:Data: 2023-07-28T23:47:01.082 DEBUG:boto:Headers: {} 2023-07-28T23:47:01.083 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:47:01.083 DEBUG:boto:Port: 8000 2023-07-28T23:47:01.084 DEBUG:boto:Params: {} 2023-07-28T23:47:01.084 DEBUG:boto:Token: None 2023-07-28T23:47:01.085 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:47:01 GMT /hiiqos-11/ 2023-07-28T23:47:01.085 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3mmj7ORIeaAQXYQ78eP96khjtv0= 2023-07-28T23:47:01.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:47:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3mmj7ORIeaAQXYQ78eP96khjtv0=', 'Content-Length': '0'} 2023-07-28T23:47:01.089 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000990bb6dbbbf7d81f-0064c45375-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:47:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:47:01.089 DEBUG:boto:path=/hiiqos-11/dummy 2023-07-28T23:47:01.090 DEBUG:boto:auth_path=/hiiqos-11/dummy 2023-07-28T23:47:01.090 DEBUG:boto:Method: PUT 2023-07-28T23:47:01.090 DEBUG:boto:Path: /hiiqos-11/dummy 2023-07-28T23:47:01.091 DEBUG:boto:Data: 2023-07-28T23:47:01.091 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-07-28T23:47:01.091 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:47:01.092 DEBUG:boto:Port: 8000 2023-07-28T23:47:01.092 DEBUG:boto:Params: {} 2023-07-28T23:47:01.093 DEBUG:boto:Token: None 2023-07-28T23:47:01.093 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 28 Jul 2023 23:47:01 GMT /hiiqos-11/dummy 2023-07-28T23:47:01.093 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:albIMrrB23t/barRYBiXj4pSSlI= 2023-07-28T23:47:01.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:47:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:albIMrrB23t/barRYBiXj4pSSlI='} 2023-07-28T23:47:01.104 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN'), ('x-amz-request-id', 'tx00000b1f2a5c91afc8ad1-0064c45375-1117-a1'), ('Date', 'Fri, 28 Jul 2023 23:47:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:47:01.105 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:47:01.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:47:01.105 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-11 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-28T23:47:01.161 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:47:01.162 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:47:01.198 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:47:01.196+0000 7fe9c102d880 1 waiting to reach incremental sync.. 2023-07-28T23:47:06.198 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:47:06.197+0000 7fe9c102d880 1 waiting to reach incremental sync.. 2023-07-28T23:47:11.198 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:47:11.197+0000 7fe9c102d880 1 waiting to reach incremental sync.. 2023-07-28T23:47:16.200 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:47:16.199+0000 7fe9c102d880 1 bucket sync caught up with source: 2023-07-28T23:47:16.201 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000002.13.11, , , , , ] 2023-07-28T23:47:16.201 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000002.13.11, , , , , ] 2023-07-28T23:47:16.201 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:47:16.199+0000 7fe9c102d880 0 bucket checkpoint complete 2023-07-28T23:47:16.216 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:47:16.217 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:47:16.217 DEBUG:boto:Method: HEAD 2023-07-28T23:47:16.218 DEBUG:boto:Path: /hiiqos-12/ 2023-07-28T23:47:16.218 DEBUG:boto:Data: 2023-07-28T23:47:16.219 DEBUG:boto:Headers: {} 2023-07-28T23:47:16.219 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:47:16.220 DEBUG:boto:Port: 8001 2023-07-28T23:47:16.220 DEBUG:boto:Params: {} 2023-07-28T23:47:16.221 DEBUG:boto:Token: None 2023-07-28T23:47:16.222 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:47:16 GMT /hiiqos-12/ 2023-07-28T23:47:16.223 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5sW7sWX4QBu97J/E4lTtlTOlgWQ= 2023-07-28T23:47:16.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:47:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5sW7sWX4QBu97J/E4lTtlTOlgWQ=', 'Content-Length': '0'} 2023-07-28T23:47:16.226 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000012b4887e4a0884e7-0064c45384-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:47:16.227 DEBUG:boto:path=/hiiqos-12/dummy 2023-07-28T23:47:16.228 DEBUG:boto:auth_path=/hiiqos-12/dummy 2023-07-28T23:47:16.228 DEBUG:boto:Method: PUT 2023-07-28T23:47:16.229 DEBUG:boto:Path: /hiiqos-12/dummy 2023-07-28T23:47:16.230 DEBUG:boto:Data: 2023-07-28T23:47:16.231 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-07-28T23:47:16.231 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:47:16.232 DEBUG:boto:Port: 8001 2023-07-28T23:47:16.233 DEBUG:boto:Params: {} 2023-07-28T23:47:16.234 DEBUG:boto:Token: None 2023-07-28T23:47:16.234 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 28 Jul 2023 23:47:16 GMT /hiiqos-12/dummy 2023-07-28T23:47:16.235 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VJK9yzLnobHHLevpIYf4334OnBI= 2023-07-28T23:47:16.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:47:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VJK9yzLnobHHLevpIYf4334OnBI='} 2023-07-28T23:47:16.248 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'cVvFSQn3t1--pfP44jwpm7HeB4ciTyV'), ('x-amz-request-id', 'tx00000c66e7c7265a1aafa-0064c45384-1113-a2'), ('Date', 'Fri, 28 Jul 2023 23:47:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:47:16.249 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:47:16.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:47:16.251 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-12 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-28T23:47:16.309 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:47:16.310 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:47:16.343 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:47:16.342+0000 7f3bcd5af880 1 waiting to reach incremental sync.. 2023-07-28T23:47:21.344 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:47:21.343+0000 7f3bcd5af880 1 waiting to reach incremental sync.. 2023-07-28T23:47:26.344 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:47:26.343+0000 7f3bcd5af880 1 waiting to reach incremental sync.. 2023-07-28T23:47:31.345 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:47:31.344+0000 7f3bcd5af880 1 waiting to reach incremental sync.. 2023-07-28T23:47:36.347 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:47:36.346+0000 7f3bcd5af880 1 bucket sync caught up with source: 2023-07-28T23:47:36.347 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000002.35.11, , , , , ] 2023-07-28T23:47:36.347 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000002.35.11, , , , , ] 2023-07-28T23:47:36.347 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:47:36.346+0000 7f3bcd5af880 0 bucket checkpoint complete 2023-07-28T23:47:36.363 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:47:36.364 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:47:36.364 DEBUG:boto:Method: HEAD 2023-07-28T23:47:36.364 DEBUG:boto:Path: /hiiqos-11/ 2023-07-28T23:47:36.365 DEBUG:boto:Data: 2023-07-28T23:47:36.365 DEBUG:boto:Headers: {} 2023-07-28T23:47:36.365 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:47:36.366 DEBUG:boto:Port: 8000 2023-07-28T23:47:36.366 DEBUG:boto:Params: {} 2023-07-28T23:47:36.367 DEBUG:boto:Token: None 2023-07-28T23:47:36.367 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:47:36 GMT /hiiqos-11/ 2023-07-28T23:47:36.368 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wUG5Hc5YnX+iwoaSqZbTgo2sBUk= 2023-07-28T23:47:36.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:47:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wUG5Hc5YnX+iwoaSqZbTgo2sBUk=', 'Content-Length': '0'} 2023-07-28T23:47:36.371 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b166967fc6adc428-0064c45398-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:47:36.371 DEBUG:boto:path=/hiiqos-11/obj-a1 2023-07-28T23:47:36.372 DEBUG:boto:auth_path=/hiiqos-11/obj-a1 2023-07-28T23:47:36.372 DEBUG:boto:Method: PUT 2023-07-28T23:47:36.372 DEBUG:boto:Path: /hiiqos-11/obj-a1 2023-07-28T23:47:36.373 DEBUG:boto:Data: 2023-07-28T23:47:36.373 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-07-28T23:47:36.373 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:47:36.374 DEBUG:boto:Port: 8000 2023-07-28T23:47:36.374 DEBUG:boto:Params: {} 2023-07-28T23:47:36.374 DEBUG:boto:Token: None 2023-07-28T23:47:36.375 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 28 Jul 2023 23:47:36 GMT /hiiqos-11/obj-a1 2023-07-28T23:47:36.375 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HfZrEQ3KYaqYXXx12j+YLAfH1r0= 2023-07-28T23:47:36.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:47:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HfZrEQ3KYaqYXXx12j+YLAfH1r0='} 2023-07-28T23:47:36.386 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'S8uVclWm2JBYB4C047A9I83Y7ulD0.f'), ('x-amz-request-id', 'tx00000eab1503993525ffd-0064c45398-1117-a1'), ('Date', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:47:36.387 DEBUG:rgw_multi.tests:version1 id=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:47:36.388 DEBUG:boto:path=/hiiqos-11/obj-a1 2023-07-28T23:47:36.388 DEBUG:boto:auth_path=/hiiqos-11/obj-a1 2023-07-28T23:47:36.389 DEBUG:boto:Method: PUT 2023-07-28T23:47:36.389 DEBUG:boto:Path: /hiiqos-11/obj-a1 2023-07-28T23:47:36.389 DEBUG:boto:Data: 2023-07-28T23:47:36.390 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-07-28T23:47:36.390 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:47:36.390 DEBUG:boto:Port: 8000 2023-07-28T23:47:36.391 DEBUG:boto:Params: {} 2023-07-28T23:47:36.391 DEBUG:boto:Token: None 2023-07-28T23:47:36.392 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 28 Jul 2023 23:47:36 GMT /hiiqos-11/obj-a1 2023-07-28T23:47:36.392 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VN7P0vFFC0uoxQFElSAyD7nTSFY= 2023-07-28T23:47:36.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:47:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VN7P0vFFC0uoxQFElSAyD7nTSFY='} 2023-07-28T23:47:36.402 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'iRnd3vmYI7ouq4ZlLx8NWnyOw4QJSi1'), ('x-amz-request-id', 'tx00000debfba467fc160d9-0064c45398-1117-a1'), ('Date', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:47:36.402 DEBUG:rgw_multi.tests:version2 id=iRnd3vmYI7ouq4ZlLx8NWnyOw4QJSi1 2023-07-28T23:47:36.403 DEBUG:boto:path=/hiiqos-11/obj-a1 2023-07-28T23:47:36.403 DEBUG:boto:auth_path=/hiiqos-11/obj-a1 2023-07-28T23:47:36.404 DEBUG:boto:path=/hiiqos-11/obj-a1?versionId=iRnd3vmYI7ouq4ZlLx8NWnyOw4QJSi1 2023-07-28T23:47:36.404 DEBUG:boto:auth_path=/hiiqos-11/obj-a1?versionId=iRnd3vmYI7ouq4ZlLx8NWnyOw4QJSi1 2023-07-28T23:47:36.404 DEBUG:boto:Method: DELETE 2023-07-28T23:47:36.405 DEBUG:boto:Path: /hiiqos-11/obj-a1?versionId=iRnd3vmYI7ouq4ZlLx8NWnyOw4QJSi1 2023-07-28T23:47:36.405 DEBUG:boto:Data: 2023-07-28T23:47:36.405 DEBUG:boto:Headers: {} 2023-07-28T23:47:36.406 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:47:36.406 DEBUG:boto:Port: 8000 2023-07-28T23:47:36.406 DEBUG:boto:Params: {} 2023-07-28T23:47:36.407 DEBUG:boto:Token: None 2023-07-28T23:47:36.407 DEBUG:boto:StringToSign: DELETE Fri, 28 Jul 2023 23:47:36 GMT /hiiqos-11/obj-a1?versionId=iRnd3vmYI7ouq4ZlLx8NWnyOw4QJSi1 2023-07-28T23:47:36.407 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dHEoX6oU68wXhmvgHIml5MJgUEI= 2023-07-28T23:47:36.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:47:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dHEoX6oU68wXhmvgHIml5MJgUEI=', 'Content-Length': '0'} 2023-07-28T23:47:36.418 DEBUG:boto:Response headers: [('x-amz-version-id', 'iRnd3vmYI7ouq4ZlLx8NWnyOw4QJSi1'), ('x-amz-request-id', 'tx000000984d04fb6fd30e4-0064c45398-1117-a1'), ('Date', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:47:36.419 DEBUG:boto:path=/hiiqos-11/obj-a1 2023-07-28T23:47:36.419 DEBUG:boto:auth_path=/hiiqos-11/obj-a1 2023-07-28T23:47:36.419 DEBUG:boto:Method: PUT 2023-07-28T23:47:36.420 DEBUG:boto:Path: /hiiqos-11/obj-a1 2023-07-28T23:47:36.420 DEBUG:boto:Data: 2023-07-28T23:47:36.420 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-07-28T23:47:36.421 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:47:36.421 DEBUG:boto:Port: 8000 2023-07-28T23:47:36.421 DEBUG:boto:Params: {} 2023-07-28T23:47:36.422 DEBUG:boto:Token: None 2023-07-28T23:47:36.422 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 28 Jul 2023 23:47:36 GMT /hiiqos-11/obj-a1 2023-07-28T23:47:36.423 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4hfttv1eMmbuvBEkGJBaxNfCI/k= 2023-07-28T23:47:36.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:47:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4hfttv1eMmbuvBEkGJBaxNfCI/k='} 2023-07-28T23:47:36.468 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'XsgbVNAWD-fwenAeRnUMKAQ4jT8GA8i'), ('x-amz-request-id', 'tx00000003024d4b0c3a4c4-0064c45398-1117-a1'), ('Date', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:47:36.469 DEBUG:rgw_multi.tests:version3 id=XsgbVNAWD-fwenAeRnUMKAQ4jT8GA8i 2023-07-28T23:47:36.470 DEBUG:boto:path=/hiiqos-11/obj-a1 2023-07-28T23:47:36.470 DEBUG:boto:auth_path=/hiiqos-11/obj-a1 2023-07-28T23:47:36.471 DEBUG:boto:path=/hiiqos-11/obj-a1?versionId=XsgbVNAWD-fwenAeRnUMKAQ4jT8GA8i 2023-07-28T23:47:36.471 DEBUG:boto:auth_path=/hiiqos-11/obj-a1?versionId=XsgbVNAWD-fwenAeRnUMKAQ4jT8GA8i 2023-07-28T23:47:36.471 DEBUG:boto:Method: DELETE 2023-07-28T23:47:36.471 DEBUG:boto:Path: /hiiqos-11/obj-a1?versionId=XsgbVNAWD-fwenAeRnUMKAQ4jT8GA8i 2023-07-28T23:47:36.472 DEBUG:boto:Data: 2023-07-28T23:47:36.472 DEBUG:boto:Headers: {} 2023-07-28T23:47:36.473 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:47:36.473 DEBUG:boto:Port: 8000 2023-07-28T23:47:36.473 DEBUG:boto:Params: {} 2023-07-28T23:47:36.474 DEBUG:boto:Token: None 2023-07-28T23:47:36.474 DEBUG:boto:StringToSign: DELETE Fri, 28 Jul 2023 23:47:36 GMT /hiiqos-11/obj-a1?versionId=XsgbVNAWD-fwenAeRnUMKAQ4jT8GA8i 2023-07-28T23:47:36.475 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qBZHPCaMVZ/tNsAjqHXLHVUUHhA= 2023-07-28T23:47:36.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:47:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qBZHPCaMVZ/tNsAjqHXLHVUUHhA=', 'Content-Length': '0'} 2023-07-28T23:47:36.484 DEBUG:boto:Response headers: [('x-amz-version-id', 'XsgbVNAWD-fwenAeRnUMKAQ4jT8GA8i'), ('x-amz-request-id', 'tx00000e94bf39b4da07d3d-0064c45398-1117-a1'), ('Date', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:47:36.485 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:47:36.485 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:47:36.486 DEBUG:boto:Method: HEAD 2023-07-28T23:47:36.486 DEBUG:boto:Path: /hiiqos-11/ 2023-07-28T23:47:36.486 DEBUG:boto:Data: 2023-07-28T23:47:36.487 DEBUG:boto:Headers: {} 2023-07-28T23:47:36.487 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:47:36.487 DEBUG:boto:Port: 8001 2023-07-28T23:47:36.488 DEBUG:boto:Params: {} 2023-07-28T23:47:36.488 DEBUG:boto:Token: None 2023-07-28T23:47:36.489 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:47:36 GMT /hiiqos-11/ 2023-07-28T23:47:36.489 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wUG5Hc5YnX+iwoaSqZbTgo2sBUk= 2023-07-28T23:47:36.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:47:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wUG5Hc5YnX+iwoaSqZbTgo2sBUk=', 'Content-Length': '0'} 2023-07-28T23:47:36.491 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c63d4b05ab1edbeb-0064c45398-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:47:36.492 DEBUG:boto:path=/hiiqos-11/obj-a2 2023-07-28T23:47:36.493 DEBUG:boto:auth_path=/hiiqos-11/obj-a2 2023-07-28T23:47:36.493 DEBUG:boto:Method: PUT 2023-07-28T23:47:36.493 DEBUG:boto:Path: /hiiqos-11/obj-a2 2023-07-28T23:47:36.494 DEBUG:boto:Data: 2023-07-28T23:47:36.494 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-07-28T23:47:36.494 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:47:36.495 DEBUG:boto:Port: 8001 2023-07-28T23:47:36.495 DEBUG:boto:Params: {} 2023-07-28T23:47:36.496 DEBUG:boto:Token: None 2023-07-28T23:47:36.496 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 28 Jul 2023 23:47:36 GMT /hiiqos-11/obj-a2 2023-07-28T23:47:36.496 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:m8q7HJMlz3/JY5EboUoxwMjE7jk= 2023-07-28T23:47:36.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:47:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:m8q7HJMlz3/JY5EboUoxwMjE7jk='} 2023-07-28T23:47:36.509 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda'), ('x-amz-request-id', 'tx00000c07e1174a762ff1c-0064c45398-1113-a2'), ('Date', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:47:36.510 DEBUG:rgw_multi.tests:version1 id=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:47:36.511 DEBUG:boto:path=/hiiqos-11/obj-a2 2023-07-28T23:47:36.511 DEBUG:boto:auth_path=/hiiqos-11/obj-a2 2023-07-28T23:47:36.511 DEBUG:boto:Method: PUT 2023-07-28T23:47:36.512 DEBUG:boto:Path: /hiiqos-11/obj-a2 2023-07-28T23:47:36.512 DEBUG:boto:Data: 2023-07-28T23:47:36.513 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-07-28T23:47:36.513 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:47:36.513 DEBUG:boto:Port: 8001 2023-07-28T23:47:36.514 DEBUG:boto:Params: {} 2023-07-28T23:47:36.514 DEBUG:boto:Token: None 2023-07-28T23:47:36.515 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 28 Jul 2023 23:47:36 GMT /hiiqos-11/obj-a2 2023-07-28T23:47:36.515 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:r5+vNvEmbRNiAdJctnislfyNJX4= 2023-07-28T23:47:36.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:47:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:r5+vNvEmbRNiAdJctnislfyNJX4='} 2023-07-28T23:47:36.526 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ad0ZD2T5n5YidDyy4PU2naLQpVx6lJo'), ('x-amz-request-id', 'tx000001362482965566e07-0064c45398-1113-a2'), ('Date', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:47:36.527 DEBUG:rgw_multi.tests:version2 id=ad0ZD2T5n5YidDyy4PU2naLQpVx6lJo 2023-07-28T23:47:36.527 DEBUG:boto:path=/hiiqos-11/obj-a2 2023-07-28T23:47:36.528 DEBUG:boto:auth_path=/hiiqos-11/obj-a2 2023-07-28T23:47:36.528 DEBUG:boto:path=/hiiqos-11/obj-a2?versionId=ad0ZD2T5n5YidDyy4PU2naLQpVx6lJo 2023-07-28T23:47:36.529 DEBUG:boto:auth_path=/hiiqos-11/obj-a2?versionId=ad0ZD2T5n5YidDyy4PU2naLQpVx6lJo 2023-07-28T23:47:36.529 DEBUG:boto:Method: DELETE 2023-07-28T23:47:36.529 DEBUG:boto:Path: /hiiqos-11/obj-a2?versionId=ad0ZD2T5n5YidDyy4PU2naLQpVx6lJo 2023-07-28T23:47:36.530 DEBUG:boto:Data: 2023-07-28T23:47:36.530 DEBUG:boto:Headers: {} 2023-07-28T23:47:36.531 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:47:36.531 DEBUG:boto:Port: 8001 2023-07-28T23:47:36.532 DEBUG:boto:Params: {} 2023-07-28T23:47:36.532 DEBUG:boto:Token: None 2023-07-28T23:47:36.532 DEBUG:boto:StringToSign: DELETE Fri, 28 Jul 2023 23:47:36 GMT /hiiqos-11/obj-a2?versionId=ad0ZD2T5n5YidDyy4PU2naLQpVx6lJo 2023-07-28T23:47:36.533 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CTmjlnSndOTjXMBkkZ2ZWRJjONc= 2023-07-28T23:47:36.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:47:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CTmjlnSndOTjXMBkkZ2ZWRJjONc=', 'Content-Length': '0'} 2023-07-28T23:47:36.542 DEBUG:boto:Response headers: [('x-amz-version-id', 'ad0ZD2T5n5YidDyy4PU2naLQpVx6lJo'), ('x-amz-request-id', 'tx000007d2dc37a1a43f98e-0064c45398-1113-a2'), ('Date', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:47:36.543 DEBUG:boto:path=/hiiqos-11/obj-a2 2023-07-28T23:47:36.544 DEBUG:boto:auth_path=/hiiqos-11/obj-a2 2023-07-28T23:47:36.544 DEBUG:boto:Method: PUT 2023-07-28T23:47:36.544 DEBUG:boto:Path: /hiiqos-11/obj-a2 2023-07-28T23:47:36.545 DEBUG:boto:Data: 2023-07-28T23:47:36.545 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-07-28T23:47:36.546 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:47:36.546 DEBUG:boto:Port: 8001 2023-07-28T23:47:36.546 DEBUG:boto:Params: {} 2023-07-28T23:47:36.547 DEBUG:boto:Token: None 2023-07-28T23:47:36.547 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 28 Jul 2023 23:47:36 GMT /hiiqos-11/obj-a2 2023-07-28T23:47:36.548 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:r30Pz9WrPXJqmqOpWSq/itH0S5A= 2023-07-28T23:47:36.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:47:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:r30Pz9WrPXJqmqOpWSq/itH0S5A='} 2023-07-28T23:47:36.596 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'tBsyjN.46ECSz.UJ2uUJNicUgRIfiNb'), ('x-amz-request-id', 'tx000007a2bbe1032e927d4-0064c45398-1113-a2'), ('Date', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:47:36.597 DEBUG:rgw_multi.tests:version3 id=tBsyjN.46ECSz.UJ2uUJNicUgRIfiNb 2023-07-28T23:47:36.598 DEBUG:boto:path=/hiiqos-11/obj-a2 2023-07-28T23:47:36.598 DEBUG:boto:auth_path=/hiiqos-11/obj-a2 2023-07-28T23:47:36.598 DEBUG:boto:path=/hiiqos-11/obj-a2?versionId=tBsyjN.46ECSz.UJ2uUJNicUgRIfiNb 2023-07-28T23:47:36.599 DEBUG:boto:auth_path=/hiiqos-11/obj-a2?versionId=tBsyjN.46ECSz.UJ2uUJNicUgRIfiNb 2023-07-28T23:47:36.599 DEBUG:boto:Method: DELETE 2023-07-28T23:47:36.599 DEBUG:boto:Path: /hiiqos-11/obj-a2?versionId=tBsyjN.46ECSz.UJ2uUJNicUgRIfiNb 2023-07-28T23:47:36.600 DEBUG:boto:Data: 2023-07-28T23:47:36.600 DEBUG:boto:Headers: {} 2023-07-28T23:47:36.600 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:47:36.601 DEBUG:boto:Port: 8001 2023-07-28T23:47:36.601 DEBUG:boto:Params: {} 2023-07-28T23:47:36.602 DEBUG:boto:Token: None 2023-07-28T23:47:36.602 DEBUG:boto:StringToSign: DELETE Fri, 28 Jul 2023 23:47:36 GMT /hiiqos-11/obj-a2?versionId=tBsyjN.46ECSz.UJ2uUJNicUgRIfiNb 2023-07-28T23:47:36.603 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gd6W6v/4IBeijfKIVTtHvqi5GvE= 2023-07-28T23:47:36.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:47:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gd6W6v/4IBeijfKIVTtHvqi5GvE=', 'Content-Length': '0'} 2023-07-28T23:47:36.612 DEBUG:boto:Response headers: [('x-amz-version-id', 'tBsyjN.46ECSz.UJ2uUJNicUgRIfiNb'), ('x-amz-request-id', 'tx000003bd555af768453e6-0064c45398-1113-a2'), ('Date', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:47:36.613 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:47:36.613 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:47:36.613 DEBUG:boto:Method: HEAD 2023-07-28T23:47:36.614 DEBUG:boto:Path: /hiiqos-12/ 2023-07-28T23:47:36.614 DEBUG:boto:Data: 2023-07-28T23:47:36.614 DEBUG:boto:Headers: {} 2023-07-28T23:47:36.615 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:47:36.615 DEBUG:boto:Port: 8000 2023-07-28T23:47:36.615 DEBUG:boto:Params: {} 2023-07-28T23:47:36.616 DEBUG:boto:Token: None 2023-07-28T23:47:36.616 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:47:36 GMT /hiiqos-12/ 2023-07-28T23:47:36.616 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ApYMrEwuf4ztYtQqjWDXiZHb7pY= 2023-07-28T23:47:36.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:47:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ApYMrEwuf4ztYtQqjWDXiZHb7pY=', 'Content-Length': '0'} 2023-07-28T23:47:36.619 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000068ae805b0e215eec-0064c45398-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:47:36.620 DEBUG:boto:path=/hiiqos-12/obj-a1 2023-07-28T23:47:36.620 DEBUG:boto:auth_path=/hiiqos-12/obj-a1 2023-07-28T23:47:36.621 DEBUG:boto:Method: PUT 2023-07-28T23:47:36.621 DEBUG:boto:Path: /hiiqos-12/obj-a1 2023-07-28T23:47:36.621 DEBUG:boto:Data: 2023-07-28T23:47:36.622 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-07-28T23:47:36.622 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:47:36.622 DEBUG:boto:Port: 8000 2023-07-28T23:47:36.623 DEBUG:boto:Params: {} 2023-07-28T23:47:36.623 DEBUG:boto:Token: None 2023-07-28T23:47:36.623 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 28 Jul 2023 23:47:36 GMT /hiiqos-12/obj-a1 2023-07-28T23:47:36.624 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:x2C+AEUsktbQr2ugWm0Gvyb0Yww= 2023-07-28T23:47:36.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:47:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:x2C+AEUsktbQr2ugWm0Gvyb0Yww='} 2023-07-28T23:47:36.633 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Zt8NMtOe11K3S282QQxMGXP84LEhPzt'), ('x-amz-request-id', 'tx00000edf0da8a8777d1bc-0064c45398-1117-a1'), ('Date', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:47:36.634 DEBUG:rgw_multi.tests:version1 id=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:47:36.634 DEBUG:boto:path=/hiiqos-12/obj-a1 2023-07-28T23:47:36.635 DEBUG:boto:auth_path=/hiiqos-12/obj-a1 2023-07-28T23:47:36.635 DEBUG:boto:Method: PUT 2023-07-28T23:47:36.635 DEBUG:boto:Path: /hiiqos-12/obj-a1 2023-07-28T23:47:36.636 DEBUG:boto:Data: 2023-07-28T23:47:36.636 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-07-28T23:47:36.637 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:47:36.637 DEBUG:boto:Port: 8000 2023-07-28T23:47:36.637 DEBUG:boto:Params: {} 2023-07-28T23:47:36.638 DEBUG:boto:Token: None 2023-07-28T23:47:36.638 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 28 Jul 2023 23:47:36 GMT /hiiqos-12/obj-a1 2023-07-28T23:47:36.639 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kjOgxcEoQW2zWNiK7qmUf+Qg6ZI= 2023-07-28T23:47:36.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:47:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kjOgxcEoQW2zWNiK7qmUf+Qg6ZI='} 2023-07-28T23:47:36.647 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'CAY0OfY-TR6pPGf6gieof3y4.7ucqvg'), ('x-amz-request-id', 'tx00000c4a93f4b74534889-0064c45398-1117-a1'), ('Date', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:47:36.648 DEBUG:rgw_multi.tests:version2 id=CAY0OfY-TR6pPGf6gieof3y4.7ucqvg 2023-07-28T23:47:36.648 DEBUG:boto:path=/hiiqos-12/obj-a1 2023-07-28T23:47:36.648 DEBUG:boto:auth_path=/hiiqos-12/obj-a1 2023-07-28T23:47:36.649 DEBUG:boto:path=/hiiqos-12/obj-a1?versionId=CAY0OfY-TR6pPGf6gieof3y4.7ucqvg 2023-07-28T23:47:36.649 DEBUG:boto:auth_path=/hiiqos-12/obj-a1?versionId=CAY0OfY-TR6pPGf6gieof3y4.7ucqvg 2023-07-28T23:47:36.649 DEBUG:boto:Method: DELETE 2023-07-28T23:47:36.650 DEBUG:boto:Path: /hiiqos-12/obj-a1?versionId=CAY0OfY-TR6pPGf6gieof3y4.7ucqvg 2023-07-28T23:47:36.650 DEBUG:boto:Data: 2023-07-28T23:47:36.650 DEBUG:boto:Headers: {} 2023-07-28T23:47:36.651 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:47:36.651 DEBUG:boto:Port: 8000 2023-07-28T23:47:36.651 DEBUG:boto:Params: {} 2023-07-28T23:47:36.652 DEBUG:boto:Token: None 2023-07-28T23:47:36.652 DEBUG:boto:StringToSign: DELETE Fri, 28 Jul 2023 23:47:36 GMT /hiiqos-12/obj-a1?versionId=CAY0OfY-TR6pPGf6gieof3y4.7ucqvg 2023-07-28T23:47:36.652 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VNpX9t7Spw0ummDBCz+umaypPEo= 2023-07-28T23:47:36.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:47:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VNpX9t7Spw0ummDBCz+umaypPEo=', 'Content-Length': '0'} 2023-07-28T23:47:36.661 DEBUG:boto:Response headers: [('x-amz-version-id', 'CAY0OfY-TR6pPGf6gieof3y4.7ucqvg'), ('x-amz-request-id', 'tx0000039c8bff7ee442441-0064c45398-1117-a1'), ('Date', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:47:36.662 DEBUG:boto:path=/hiiqos-12/obj-a1 2023-07-28T23:47:36.663 DEBUG:boto:auth_path=/hiiqos-12/obj-a1 2023-07-28T23:47:36.663 DEBUG:boto:Method: PUT 2023-07-28T23:47:36.663 DEBUG:boto:Path: /hiiqos-12/obj-a1 2023-07-28T23:47:36.664 DEBUG:boto:Data: 2023-07-28T23:47:36.664 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-07-28T23:47:36.664 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:47:36.665 DEBUG:boto:Port: 8000 2023-07-28T23:47:36.665 DEBUG:boto:Params: {} 2023-07-28T23:47:36.665 DEBUG:boto:Token: None 2023-07-28T23:47:36.666 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 28 Jul 2023 23:47:36 GMT /hiiqos-12/obj-a1 2023-07-28T23:47:36.667 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LucwncmD8AB+jUEWY/pUMqw2llE= 2023-07-28T23:47:36.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:47:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LucwncmD8AB+jUEWY/pUMqw2llE='} 2023-07-28T23:47:36.717 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'pFtsOjD20ZyMwYnxpYXlan.fuZOFsyz'), ('x-amz-request-id', 'tx00000c1d8163a9bd9b3bb-0064c45398-1117-a1'), ('Date', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:47:36.717 DEBUG:rgw_multi.tests:version3 id=pFtsOjD20ZyMwYnxpYXlan.fuZOFsyz 2023-07-28T23:47:36.718 DEBUG:boto:path=/hiiqos-12/obj-a1 2023-07-28T23:47:36.718 DEBUG:boto:auth_path=/hiiqos-12/obj-a1 2023-07-28T23:47:36.718 DEBUG:boto:path=/hiiqos-12/obj-a1?versionId=pFtsOjD20ZyMwYnxpYXlan.fuZOFsyz 2023-07-28T23:47:36.719 DEBUG:boto:auth_path=/hiiqos-12/obj-a1?versionId=pFtsOjD20ZyMwYnxpYXlan.fuZOFsyz 2023-07-28T23:47:36.719 DEBUG:boto:Method: DELETE 2023-07-28T23:47:36.719 DEBUG:boto:Path: /hiiqos-12/obj-a1?versionId=pFtsOjD20ZyMwYnxpYXlan.fuZOFsyz 2023-07-28T23:47:36.720 DEBUG:boto:Data: 2023-07-28T23:47:36.720 DEBUG:boto:Headers: {} 2023-07-28T23:47:36.720 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:47:36.721 DEBUG:boto:Port: 8000 2023-07-28T23:47:36.721 DEBUG:boto:Params: {} 2023-07-28T23:47:36.721 DEBUG:boto:Token: None 2023-07-28T23:47:36.722 DEBUG:boto:StringToSign: DELETE Fri, 28 Jul 2023 23:47:36 GMT /hiiqos-12/obj-a1?versionId=pFtsOjD20ZyMwYnxpYXlan.fuZOFsyz 2023-07-28T23:47:36.722 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:s/J2BzYXtG3lIM/kVJrO2cKnfcw= 2023-07-28T23:47:36.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:47:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:s/J2BzYXtG3lIM/kVJrO2cKnfcw=', 'Content-Length': '0'} 2023-07-28T23:47:36.731 DEBUG:boto:Response headers: [('x-amz-version-id', 'pFtsOjD20ZyMwYnxpYXlan.fuZOFsyz'), ('x-amz-request-id', 'tx00000791833cd6382e65b-0064c45398-1117-a1'), ('Date', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:47:36.732 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:47:36.733 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:47:36.733 DEBUG:boto:Method: HEAD 2023-07-28T23:47:36.733 DEBUG:boto:Path: /hiiqos-12/ 2023-07-28T23:47:36.734 DEBUG:boto:Data: 2023-07-28T23:47:36.734 DEBUG:boto:Headers: {} 2023-07-28T23:47:36.734 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:47:36.735 DEBUG:boto:Port: 8001 2023-07-28T23:47:36.735 DEBUG:boto:Params: {} 2023-07-28T23:47:36.736 DEBUG:boto:Token: None 2023-07-28T23:47:36.736 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:47:36 GMT /hiiqos-12/ 2023-07-28T23:47:36.736 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ApYMrEwuf4ztYtQqjWDXiZHb7pY= 2023-07-28T23:47:36.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:47:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ApYMrEwuf4ztYtQqjWDXiZHb7pY=', 'Content-Length': '0'} 2023-07-28T23:47:36.743 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000028708f5b49e5ae14-0064c45398-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:47:36.744 DEBUG:boto:path=/hiiqos-12/obj-a2 2023-07-28T23:47:36.745 DEBUG:boto:auth_path=/hiiqos-12/obj-a2 2023-07-28T23:47:36.745 DEBUG:boto:Method: PUT 2023-07-28T23:47:36.745 DEBUG:boto:Path: /hiiqos-12/obj-a2 2023-07-28T23:47:36.746 DEBUG:boto:Data: 2023-07-28T23:47:36.746 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-07-28T23:47:36.746 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:47:36.747 DEBUG:boto:Port: 8001 2023-07-28T23:47:36.747 DEBUG:boto:Params: {} 2023-07-28T23:47:36.747 DEBUG:boto:Token: None 2023-07-28T23:47:36.748 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 28 Jul 2023 23:47:36 GMT /hiiqos-12/obj-a2 2023-07-28T23:47:36.748 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QO00xIYn21SD4jV52hLA3Jlj77k= 2023-07-28T23:47:36.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:47:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QO00xIYn21SD4jV52hLA3Jlj77k='} 2023-07-28T23:47:36.763 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'KycZQE1frBdJnvJd6sV3PeY5cRPnOLF'), ('x-amz-request-id', 'tx0000058d38d76ee8c2b1b-0064c45398-1113-a2'), ('Date', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:47:36.764 DEBUG:rgw_multi.tests:version1 id=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:47:36.764 DEBUG:boto:path=/hiiqos-12/obj-a2 2023-07-28T23:47:36.765 DEBUG:boto:auth_path=/hiiqos-12/obj-a2 2023-07-28T23:47:36.765 DEBUG:boto:Method: PUT 2023-07-28T23:47:36.766 DEBUG:boto:Path: /hiiqos-12/obj-a2 2023-07-28T23:47:36.766 DEBUG:boto:Data: 2023-07-28T23:47:36.767 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-07-28T23:47:36.768 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:47:36.768 DEBUG:boto:Port: 8001 2023-07-28T23:47:36.769 DEBUG:boto:Params: {} 2023-07-28T23:47:36.769 DEBUG:boto:Token: None 2023-07-28T23:47:36.769 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 28 Jul 2023 23:47:36 GMT /hiiqos-12/obj-a2 2023-07-28T23:47:36.770 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mFwvmddAskW1v1+trsERa22DN1Q= 2023-07-28T23:47:36.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:47:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mFwvmddAskW1v1+trsERa22DN1Q='} 2023-07-28T23:47:36.780 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'coaCJC7O3q8Py0YCeE5riEr5UDOZrNc'), ('x-amz-request-id', 'tx00000f0bbee73415bd0f4-0064c45398-1113-a2'), ('Date', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:47:36.781 DEBUG:rgw_multi.tests:version2 id=coaCJC7O3q8Py0YCeE5riEr5UDOZrNc 2023-07-28T23:47:36.781 DEBUG:boto:path=/hiiqos-12/obj-a2 2023-07-28T23:47:36.781 DEBUG:boto:auth_path=/hiiqos-12/obj-a2 2023-07-28T23:47:36.782 DEBUG:boto:path=/hiiqos-12/obj-a2?versionId=coaCJC7O3q8Py0YCeE5riEr5UDOZrNc 2023-07-28T23:47:36.782 DEBUG:boto:auth_path=/hiiqos-12/obj-a2?versionId=coaCJC7O3q8Py0YCeE5riEr5UDOZrNc 2023-07-28T23:47:36.782 DEBUG:boto:Method: DELETE 2023-07-28T23:47:36.783 DEBUG:boto:Path: /hiiqos-12/obj-a2?versionId=coaCJC7O3q8Py0YCeE5riEr5UDOZrNc 2023-07-28T23:47:36.783 DEBUG:boto:Data: 2023-07-28T23:47:36.783 DEBUG:boto:Headers: {} 2023-07-28T23:47:36.784 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:47:36.784 DEBUG:boto:Port: 8001 2023-07-28T23:47:36.785 DEBUG:boto:Params: {} 2023-07-28T23:47:36.785 DEBUG:boto:Token: None 2023-07-28T23:47:36.785 DEBUG:boto:StringToSign: DELETE Fri, 28 Jul 2023 23:47:36 GMT /hiiqos-12/obj-a2?versionId=coaCJC7O3q8Py0YCeE5riEr5UDOZrNc 2023-07-28T23:47:36.786 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Odh0Y6TRC9R1lW6NL+lCEQe6s9s= 2023-07-28T23:47:36.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:47:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Odh0Y6TRC9R1lW6NL+lCEQe6s9s=', 'Content-Length': '0'} 2023-07-28T23:47:36.795 DEBUG:boto:Response headers: [('x-amz-version-id', 'coaCJC7O3q8Py0YCeE5riEr5UDOZrNc'), ('x-amz-request-id', 'tx00000d22e8cd09386cdd9-0064c45398-1113-a2'), ('Date', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:47:36.796 DEBUG:boto:path=/hiiqos-12/obj-a2 2023-07-28T23:47:36.797 DEBUG:boto:auth_path=/hiiqos-12/obj-a2 2023-07-28T23:47:36.797 DEBUG:boto:Method: PUT 2023-07-28T23:47:36.797 DEBUG:boto:Path: /hiiqos-12/obj-a2 2023-07-28T23:47:36.798 DEBUG:boto:Data: 2023-07-28T23:47:36.798 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-07-28T23:47:36.798 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:47:36.799 DEBUG:boto:Port: 8001 2023-07-28T23:47:36.799 DEBUG:boto:Params: {} 2023-07-28T23:47:36.799 DEBUG:boto:Token: None 2023-07-28T23:47:36.800 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 28 Jul 2023 23:47:36 GMT /hiiqos-12/obj-a2 2023-07-28T23:47:36.800 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:61RZ8/wEce27XDSnu1HB/87IkyM= 2023-07-28T23:47:36.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:47:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:61RZ8/wEce27XDSnu1HB/87IkyM='} 2023-07-28T23:47:36.844 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'VYTVJYyaXtCJRnsFDoc0kjyKuMtQA-7'), ('x-amz-request-id', 'tx00000cf303f3d94522549-0064c45398-1113-a2'), ('Date', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:47:36.845 DEBUG:rgw_multi.tests:version3 id=VYTVJYyaXtCJRnsFDoc0kjyKuMtQA-7 2023-07-28T23:47:36.845 DEBUG:boto:path=/hiiqos-12/obj-a2 2023-07-28T23:47:36.846 DEBUG:boto:auth_path=/hiiqos-12/obj-a2 2023-07-28T23:47:36.846 DEBUG:boto:path=/hiiqos-12/obj-a2?versionId=VYTVJYyaXtCJRnsFDoc0kjyKuMtQA-7 2023-07-28T23:47:36.846 DEBUG:boto:auth_path=/hiiqos-12/obj-a2?versionId=VYTVJYyaXtCJRnsFDoc0kjyKuMtQA-7 2023-07-28T23:47:36.847 DEBUG:boto:Method: DELETE 2023-07-28T23:47:36.847 DEBUG:boto:Path: /hiiqos-12/obj-a2?versionId=VYTVJYyaXtCJRnsFDoc0kjyKuMtQA-7 2023-07-28T23:47:36.847 DEBUG:boto:Data: 2023-07-28T23:47:36.848 DEBUG:boto:Headers: {} 2023-07-28T23:47:36.848 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:47:36.848 DEBUG:boto:Port: 8001 2023-07-28T23:47:36.849 DEBUG:boto:Params: {} 2023-07-28T23:47:36.849 DEBUG:boto:Token: None 2023-07-28T23:47:36.849 DEBUG:boto:StringToSign: DELETE Fri, 28 Jul 2023 23:47:36 GMT /hiiqos-12/obj-a2?versionId=VYTVJYyaXtCJRnsFDoc0kjyKuMtQA-7 2023-07-28T23:47:36.850 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qtRD4ZYt7VcOVJ1SOEZOCojNq5I= 2023-07-28T23:47:36.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:47:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qtRD4ZYt7VcOVJ1SOEZOCojNq5I=', 'Content-Length': '0'} 2023-07-28T23:47:36.858 DEBUG:boto:Response headers: [('x-amz-version-id', 'VYTVJYyaXtCJRnsFDoc0kjyKuMtQA-7'), ('x-amz-request-id', 'tx000002903bc9fc49ad1f8-0064c45398-1113-a2'), ('Date', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:47:36.859 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-11 2023-07-28T23:47:36.859 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:47:36.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:47:36.860 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-11 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-28T23:47:36.912 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:47:36.913 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:47:36.944 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:47:36.943+0000 7f08874c0880 1 waiting for incremental sync to catch up: 2023-07-28T23:47:36.944 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000002.13.11, , , , , ] 2023-07-28T23:47:36.945 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000003.19.10, , , , 00000000010.23.6, ] 2023-07-28T23:47:41.946 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:47:41.945+0000 7f08874c0880 1 waiting for incremental sync to catch up: 2023-07-28T23:47:41.946 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000002.13.11, , , , , ] 2023-07-28T23:47:41.947 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000003.19.10, , , , 00000000010.23.6, ] 2023-07-28T23:47:46.947 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:47:46.946+0000 7f08874c0880 1 waiting for incremental sync to catch up: 2023-07-28T23:47:46.947 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000002.13.11, , , , , ] 2023-07-28T23:47:46.947 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000003.19.10, , , , 00000000010.23.6, ] 2023-07-28T23:47:51.948 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:47:51.947+0000 7f08874c0880 1 waiting for incremental sync to catch up: 2023-07-28T23:47:51.948 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000002.13.11, , , , , ] 2023-07-28T23:47:51.948 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000003.19.10, , , , 00000000010.23.6, ] 2023-07-28T23:47:56.949 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:47:56.948+0000 7f08874c0880 1 bucket sync caught up with source: 2023-07-28T23:47:56.949 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000003.19.10, , , 00000000002.7.11, 00000000010.23.6, ] 2023-07-28T23:47:56.950 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000003.19.10, , , , 00000000010.23.6, ] 2023-07-28T23:47:56.950 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:47:56.948+0000 7f08874c0880 0 bucket checkpoint complete 2023-07-28T23:47:56.964 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-11 2023-07-28T23:47:56.965 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:47:56.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:47:56.966 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-11 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-28T23:47:57.024 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:47:57.024 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:47:57.060 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:47:57.059+0000 7f5b61d26880 1 waiting for incremental sync to catch up: 2023-07-28T23:47:57.060 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000002.31.11, , , 00000000010.30.6, , ] 2023-07-28T23:47:57.061 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000002.31.11, , , 00000000010.30.6, 00000000002.13.11, ] 2023-07-28T23:48:02.061 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:48:02.060+0000 7f5b61d26880 1 waiting for incremental sync to catch up: 2023-07-28T23:48:02.062 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000002.31.11, , , 00000000010.30.6, , ] 2023-07-28T23:48:02.062 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000002.31.11, , , 00000000010.30.6, 00000000002.13.11, ] 2023-07-28T23:48:07.063 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:48:07.062+0000 7f5b61d26880 1 waiting for incremental sync to catch up: 2023-07-28T23:48:07.063 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000002.31.11, , , 00000000010.30.6, , ] 2023-07-28T23:48:07.063 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000002.31.11, , , 00000000010.30.6, 00000000002.13.11, ] 2023-07-28T23:48:12.064 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:48:12.063+0000 7f5b61d26880 1 waiting for incremental sync to catch up: 2023-07-28T23:48:12.065 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000002.31.11, , , 00000000010.30.6, , ] 2023-07-28T23:48:12.065 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000002.31.11, , , 00000000010.30.6, 00000000002.13.11, ] 2023-07-28T23:48:17.066 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:48:17.064+0000 7f5b61d26880 1 bucket sync caught up with source: 2023-07-28T23:48:17.066 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000002.31.11, , , 00000000010.30.6, 00000000002.13.11, ] 2023-07-28T23:48:17.067 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000002.31.11, , , 00000000010.30.6, 00000000002.13.11, ] 2023-07-28T23:48:17.067 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:48:17.064+0000 7f5b61d26880 0 bucket checkpoint complete 2023-07-28T23:48:17.084 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-11 zones={a2, a1} 2023-07-28T23:48:17.085 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:48:17.085 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:48:17.086 DEBUG:boto:Method: HEAD 2023-07-28T23:48:17.086 DEBUG:boto:Path: /hiiqos-11/ 2023-07-28T23:48:17.086 DEBUG:boto:Data: 2023-07-28T23:48:17.087 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.087 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:17.087 DEBUG:boto:Port: 8001 2023-07-28T23:48:17.088 DEBUG:boto:Params: {} 2023-07-28T23:48:17.088 DEBUG:boto:Token: None 2023-07-28T23:48:17.089 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/ 2023-07-28T23:48:17.089 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:voHkxCGNkb1RscJi74WD9p4ECFM= 2023-07-28T23:48:17.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:voHkxCGNkb1RscJi74WD9p4ECFM=', 'Content-Length': '0'} 2023-07-28T23:48:17.092 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000096390f3f1f4e0ad7-0064c453c1-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.093 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:48:17.093 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:48:17.094 DEBUG:boto:Method: HEAD 2023-07-28T23:48:17.094 DEBUG:boto:Path: /hiiqos-11/ 2023-07-28T23:48:17.095 DEBUG:boto:Data: 2023-07-28T23:48:17.095 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.095 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:17.096 DEBUG:boto:Port: 8000 2023-07-28T23:48:17.096 DEBUG:boto:Params: {} 2023-07-28T23:48:17.096 DEBUG:boto:Token: None 2023-07-28T23:48:17.097 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/ 2023-07-28T23:48:17.097 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:voHkxCGNkb1RscJi74WD9p4ECFM= 2023-07-28T23:48:17.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:voHkxCGNkb1RscJi74WD9p4ECFM=', 'Content-Length': '0'} 2023-07-28T23:48:17.100 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000529e5f3fec86a3d7-0064c453c1-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.100 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-28T23:48:17.101 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:48:17.101 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:48:17.101 DEBUG:boto:path=/hiiqos-11/?versions&max-keys=999 2023-07-28T23:48:17.102 DEBUG:boto:auth_path=/hiiqos-11/?versions&max-keys=999 2023-07-28T23:48:17.102 DEBUG:boto:Method: GET 2023-07-28T23:48:17.102 DEBUG:boto:Path: /hiiqos-11/?versions&max-keys=999 2023-07-28T23:48:17.103 DEBUG:boto:Data: 2023-07-28T23:48:17.103 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.103 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:17.104 DEBUG:boto:Port: 8001 2023-07-28T23:48:17.104 DEBUG:boto:Params: {} 2023-07-28T23:48:17.105 DEBUG:boto:Token: None 2023-07-28T23:48:17.105 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/?versions 2023-07-28T23:48:17.105 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:iKpbmqhOGH7hPe5Jw2vq7lmTf6A= 2023-07-28T23:48:17.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:iKpbmqhOGH7hPe5Jw2vq7lmTf6A=', 'Content-Length': '0'} 2023-07-28T23:48:17.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000256b041294054e86-0064c453c1-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.108 DEBUG:boto:b'hiiqos-11999falsedummyRUBZW9FJzfoEbFpjsAMCNtq8zLwltmNtrue2023-07-28T23:47:01.096Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1S8uVclWm2JBYB4C047A9I83Y7ulD0.ftrue2023-07-28T23:47:36.378Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYdatrue2023-07-28T23:47:36.499Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:17.109 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-07-28T23:48:17.110 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-07-28T23:48:17.110 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-07-28T23:48:17.110 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-28T23:48:17.111 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:48:17.111 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:48:17.111 DEBUG:boto:path=/hiiqos-11/?versions&max-keys=999 2023-07-28T23:48:17.112 DEBUG:boto:auth_path=/hiiqos-11/?versions&max-keys=999 2023-07-28T23:48:17.112 DEBUG:boto:Method: GET 2023-07-28T23:48:17.112 DEBUG:boto:Path: /hiiqos-11/?versions&max-keys=999 2023-07-28T23:48:17.113 DEBUG:boto:Data: 2023-07-28T23:48:17.113 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.113 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:17.114 DEBUG:boto:Port: 8000 2023-07-28T23:48:17.114 DEBUG:boto:Params: {} 2023-07-28T23:48:17.115 DEBUG:boto:Token: None 2023-07-28T23:48:17.115 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/?versions 2023-07-28T23:48:17.115 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:iKpbmqhOGH7hPe5Jw2vq7lmTf6A= 2023-07-28T23:48:17.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:iKpbmqhOGH7hPe5Jw2vq7lmTf6A=', 'Content-Length': '0'} 2023-07-28T23:48:17.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7a9531b7c7c8626-0064c453c1-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.119 DEBUG:boto:b'hiiqos-11999falsedummyRUBZW9FJzfoEbFpjsAMCNtq8zLwltmNtrue2023-07-28T23:47:01.096Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1S8uVclWm2JBYB4C047A9I83Y7ulD0.ftrue2023-07-28T23:47:36.378Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYdatrue2023-07-28T23:47:36.499Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:17.120 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-07-28T23:48:17.120 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-07-28T23:48:17.120 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-07-28T23:48:17.121 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:48:17.121 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:48:17.121 DEBUG:boto:path=/hiiqos-11/?versions&max-keys=999 2023-07-28T23:48:17.122 DEBUG:boto:auth_path=/hiiqos-11/?versions&max-keys=999 2023-07-28T23:48:17.122 DEBUG:boto:Method: GET 2023-07-28T23:48:17.122 DEBUG:boto:Path: /hiiqos-11/?versions&max-keys=999 2023-07-28T23:48:17.123 DEBUG:boto:Data: 2023-07-28T23:48:17.123 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.123 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:17.124 DEBUG:boto:Port: 8001 2023-07-28T23:48:17.124 DEBUG:boto:Params: {} 2023-07-28T23:48:17.125 DEBUG:boto:Token: None 2023-07-28T23:48:17.125 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/?versions 2023-07-28T23:48:17.125 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:iKpbmqhOGH7hPe5Jw2vq7lmTf6A= 2023-07-28T23:48:17.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:iKpbmqhOGH7hPe5Jw2vq7lmTf6A=', 'Content-Length': '0'} 2023-07-28T23:48:17.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000771eeb0941378fad-0064c453c1-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.129 DEBUG:boto:b'hiiqos-11999falsedummyRUBZW9FJzfoEbFpjsAMCNtq8zLwltmNtrue2023-07-28T23:47:01.096Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1S8uVclWm2JBYB4C047A9I83Y7ulD0.ftrue2023-07-28T23:47:36.378Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYdatrue2023-07-28T23:47:36.499Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:17.129 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:48:17.130 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:48:17.130 DEBUG:boto:path=/hiiqos-11/?versions&max-keys=999 2023-07-28T23:48:17.131 DEBUG:boto:auth_path=/hiiqos-11/?versions&max-keys=999 2023-07-28T23:48:17.131 DEBUG:boto:Method: GET 2023-07-28T23:48:17.131 DEBUG:boto:Path: /hiiqos-11/?versions&max-keys=999 2023-07-28T23:48:17.132 DEBUG:boto:Data: 2023-07-28T23:48:17.132 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.132 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:17.133 DEBUG:boto:Port: 8000 2023-07-28T23:48:17.133 DEBUG:boto:Params: {} 2023-07-28T23:48:17.133 DEBUG:boto:Token: None 2023-07-28T23:48:17.134 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/?versions 2023-07-28T23:48:17.134 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:iKpbmqhOGH7hPe5Jw2vq7lmTf6A= 2023-07-28T23:48:17.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:iKpbmqhOGH7hPe5Jw2vq7lmTf6A=', 'Content-Length': '0'} 2023-07-28T23:48:17.137 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000767954c65575d6a5-0064c453c1-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.137 DEBUG:boto:b'hiiqos-11999falsedummyRUBZW9FJzfoEbFpjsAMCNtq8zLwltmNtrue2023-07-28T23:47:01.096Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1S8uVclWm2JBYB4C047A9I83Y7ulD0.ftrue2023-07-28T23:47:36.378Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYdatrue2023-07-28T23:47:36.499Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:17.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-07-28T23:48:17.139 DEBUG:boto:path=/hiiqos-11/dummy 2023-07-28T23:48:17.139 DEBUG:boto:auth_path=/hiiqos-11/dummy 2023-07-28T23:48:17.140 DEBUG:boto:path=/hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:17.140 DEBUG:boto:auth_path=/hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:17.140 DEBUG:boto:Method: GET 2023-07-28T23:48:17.141 DEBUG:boto:Path: /hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:17.141 DEBUG:boto:Data: 2023-07-28T23:48:17.141 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.142 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:17.142 DEBUG:boto:Port: 8001 2023-07-28T23:48:17.142 DEBUG:boto:Params: {} 2023-07-28T23:48:17.143 DEBUG:boto:Token: None 2023-07-28T23:48:17.143 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:17.144 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AMfVmGwZ7GQe0gmFZPoUN8vvzww= 2023-07-28T23:48:17.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AMfVmGwZ7GQe0gmFZPoUN8vvzww=', 'Content-Length': '0'} 2023-07-28T23:48:17.146 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:01 GMT'), ('x-amz-version-id', 'RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a682b2e58355ded3-0064c453c1-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.147 DEBUG:boto:path=/hiiqos-11/dummy 2023-07-28T23:48:17.147 DEBUG:boto:auth_path=/hiiqos-11/dummy 2023-07-28T23:48:17.148 DEBUG:boto:path=/hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:17.148 DEBUG:boto:auth_path=/hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:17.148 DEBUG:boto:Method: GET 2023-07-28T23:48:17.149 DEBUG:boto:Path: /hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:17.149 DEBUG:boto:Data: 2023-07-28T23:48:17.149 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.150 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:17.150 DEBUG:boto:Port: 8000 2023-07-28T23:48:17.150 DEBUG:boto:Params: {} 2023-07-28T23:48:17.151 DEBUG:boto:Token: None 2023-07-28T23:48:17.151 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:17.152 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AMfVmGwZ7GQe0gmFZPoUN8vvzww= 2023-07-28T23:48:17.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AMfVmGwZ7GQe0gmFZPoUN8vvzww=', 'Content-Length': '0'} 2023-07-28T23:48:17.154 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:01 GMT'), ('x-amz-version-id', 'RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004c8958e8e8344374-0064c453c1-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.155 DEBUG:boto:path=/hiiqos-11/dummy 2023-07-28T23:48:17.155 DEBUG:boto:auth_path=/hiiqos-11/dummy 2023-07-28T23:48:17.155 DEBUG:boto:path=/hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:17.156 DEBUG:boto:auth_path=/hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:17.156 DEBUG:boto:Method: HEAD 2023-07-28T23:48:17.156 DEBUG:boto:Path: /hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:17.157 DEBUG:boto:Data: 2023-07-28T23:48:17.157 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.157 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:17.158 DEBUG:boto:Port: 8001 2023-07-28T23:48:17.158 DEBUG:boto:Params: {} 2023-07-28T23:48:17.158 DEBUG:boto:Token: None 2023-07-28T23:48:17.159 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:17.159 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CXnL0NAb3GG6W4mglTNpoEUorYM= 2023-07-28T23:48:17.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CXnL0NAb3GG6W4mglTNpoEUorYM=', 'Content-Length': '0'} 2023-07-28T23:48:17.161 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:01 GMT'), ('x-amz-version-id', 'RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007765e220a62c0341-0064c453c1-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.162 DEBUG:boto:path=/hiiqos-11/dummy 2023-07-28T23:48:17.162 DEBUG:boto:auth_path=/hiiqos-11/dummy 2023-07-28T23:48:17.162 DEBUG:boto:path=/hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:17.163 DEBUG:boto:auth_path=/hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:17.163 DEBUG:boto:Method: HEAD 2023-07-28T23:48:17.163 DEBUG:boto:Path: /hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:17.164 DEBUG:boto:Data: 2023-07-28T23:48:17.164 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.164 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:17.165 DEBUG:boto:Port: 8000 2023-07-28T23:48:17.165 DEBUG:boto:Params: {} 2023-07-28T23:48:17.165 DEBUG:boto:Token: None 2023-07-28T23:48:17.166 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:17.166 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CXnL0NAb3GG6W4mglTNpoEUorYM= 2023-07-28T23:48:17.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CXnL0NAb3GG6W4mglTNpoEUorYM=', 'Content-Length': '0'} 2023-07-28T23:48:17.168 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:01 GMT'), ('x-amz-version-id', 'RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b488a9031e002a3c-0064c453c1-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-07-28T23:48:17.169 DEBUG:boto:path=/hiiqos-11/dummy 2023-07-28T23:48:17.170 DEBUG:boto:auth_path=/hiiqos-11/dummy 2023-07-28T23:48:17.170 DEBUG:boto:path=/hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:17.170 DEBUG:boto:auth_path=/hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:17.171 DEBUG:boto:Method: GET 2023-07-28T23:48:17.171 DEBUG:boto:Path: /hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:17.171 DEBUG:boto:Data: 2023-07-28T23:48:17.172 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.172 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:17.172 DEBUG:boto:Port: 8001 2023-07-28T23:48:17.173 DEBUG:boto:Params: {} 2023-07-28T23:48:17.173 DEBUG:boto:Token: None 2023-07-28T23:48:17.174 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:17.174 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AMfVmGwZ7GQe0gmFZPoUN8vvzww= 2023-07-28T23:48:17.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AMfVmGwZ7GQe0gmFZPoUN8vvzww=', 'Content-Length': '0'} 2023-07-28T23:48:17.176 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:01 GMT'), ('x-amz-version-id', 'RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003a2a426c703577d2-0064c453c1-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.176 DEBUG:boto:path=/hiiqos-11/dummy 2023-07-28T23:48:17.177 DEBUG:boto:auth_path=/hiiqos-11/dummy 2023-07-28T23:48:17.177 DEBUG:boto:path=/hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:17.178 DEBUG:boto:auth_path=/hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:17.178 DEBUG:boto:Method: GET 2023-07-28T23:48:17.178 DEBUG:boto:Path: /hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:17.179 DEBUG:boto:Data: 2023-07-28T23:48:17.179 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.179 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:17.180 DEBUG:boto:Port: 8000 2023-07-28T23:48:17.180 DEBUG:boto:Params: {} 2023-07-28T23:48:17.180 DEBUG:boto:Token: None 2023-07-28T23:48:17.181 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:17.181 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AMfVmGwZ7GQe0gmFZPoUN8vvzww= 2023-07-28T23:48:17.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AMfVmGwZ7GQe0gmFZPoUN8vvzww=', 'Content-Length': '0'} 2023-07-28T23:48:17.183 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:01 GMT'), ('x-amz-version-id', 'RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000036ad260ce8866223-0064c453c1-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.184 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:48:17.184 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:48:17.185 DEBUG:boto:path=/hiiqos-11/?prefix=dummy 2023-07-28T23:48:17.185 DEBUG:boto:auth_path=/hiiqos-11/?prefix=dummy 2023-07-28T23:48:17.186 DEBUG:boto:Method: GET 2023-07-28T23:48:17.186 DEBUG:boto:Path: /hiiqos-11/?prefix=dummy 2023-07-28T23:48:17.186 DEBUG:boto:Data: 2023-07-28T23:48:17.187 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.187 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:17.187 DEBUG:boto:Port: 8001 2023-07-28T23:48:17.188 DEBUG:boto:Params: {} 2023-07-28T23:48:17.188 DEBUG:boto:Token: None 2023-07-28T23:48:17.188 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/ 2023-07-28T23:48:17.189 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FR4zwVjrSL5xZkGEPFbh4PrQPL0= 2023-07-28T23:48:17.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FR4zwVjrSL5xZkGEPFbh4PrQPL0=', 'Content-Length': '0'} 2023-07-28T23:48:17.192 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5a1fb2f845f4570-0064c453c1-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.192 DEBUG:boto:b'hiiqos-11dummy1000falsedummy2023-07-28T23:47:01.096Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:17.193 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:48:17.193 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:48:17.194 DEBUG:boto:path=/hiiqos-11/?prefix=dummy 2023-07-28T23:48:17.194 DEBUG:boto:auth_path=/hiiqos-11/?prefix=dummy 2023-07-28T23:48:17.195 DEBUG:boto:Method: GET 2023-07-28T23:48:17.195 DEBUG:boto:Path: /hiiqos-11/?prefix=dummy 2023-07-28T23:48:17.195 DEBUG:boto:Data: 2023-07-28T23:48:17.196 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.196 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:17.196 DEBUG:boto:Port: 8000 2023-07-28T23:48:17.197 DEBUG:boto:Params: {} 2023-07-28T23:48:17.197 DEBUG:boto:Token: None 2023-07-28T23:48:17.197 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/ 2023-07-28T23:48:17.198 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FR4zwVjrSL5xZkGEPFbh4PrQPL0= 2023-07-28T23:48:17.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FR4zwVjrSL5xZkGEPFbh4PrQPL0=', 'Content-Length': '0'} 2023-07-28T23:48:17.200 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b751f5a3ee8789a-0064c453c1-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.244 DEBUG:boto:b'hiiqos-11dummy1000falsedummy2023-07-28T23:47:01.096Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:17.245 DEBUG:boto:path=/hiiqos-11/dummy 2023-07-28T23:48:17.246 DEBUG:boto:auth_path=/hiiqos-11/dummy 2023-07-28T23:48:17.246 DEBUG:boto:Method: HEAD 2023-07-28T23:48:17.246 DEBUG:boto:Path: /hiiqos-11/dummy 2023-07-28T23:48:17.247 DEBUG:boto:Data: 2023-07-28T23:48:17.247 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.247 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:17.248 DEBUG:boto:Port: 8001 2023-07-28T23:48:17.248 DEBUG:boto:Params: {} 2023-07-28T23:48:17.249 DEBUG:boto:Token: None 2023-07-28T23:48:17.249 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/dummy 2023-07-28T23:48:17.249 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0++At6GjlA7DgONAgErdJaY6Ob4= 2023-07-28T23:48:17.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0++At6GjlA7DgONAgErdJaY6Ob4=', 'Content-Length': '0'} 2023-07-28T23:48:17.252 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:01 GMT'), ('x-amz-version-id', 'RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002fc7a4c0f9051718-0064c453c1-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.253 DEBUG:boto:path=/hiiqos-11/dummy 2023-07-28T23:48:17.253 DEBUG:boto:auth_path=/hiiqos-11/dummy 2023-07-28T23:48:17.253 DEBUG:boto:Method: HEAD 2023-07-28T23:48:17.254 DEBUG:boto:Path: /hiiqos-11/dummy 2023-07-28T23:48:17.254 DEBUG:boto:Data: 2023-07-28T23:48:17.254 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.255 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:17.255 DEBUG:boto:Port: 8000 2023-07-28T23:48:17.255 DEBUG:boto:Params: {} 2023-07-28T23:48:17.256 DEBUG:boto:Token: None 2023-07-28T23:48:17.256 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/dummy 2023-07-28T23:48:17.257 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0++At6GjlA7DgONAgErdJaY6Ob4= 2023-07-28T23:48:17.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0++At6GjlA7DgONAgErdJaY6Ob4=', 'Content-Length': '0'} 2023-07-28T23:48:17.259 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:01 GMT'), ('x-amz-version-id', 'RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008261c8a739f3c68d-0064c453c1-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-07-28T23:48:17.260 DEBUG:boto:path=/hiiqos-11/dummy 2023-07-28T23:48:17.260 DEBUG:boto:auth_path=/hiiqos-11/dummy 2023-07-28T23:48:17.261 DEBUG:boto:path=/hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:17.261 DEBUG:boto:auth_path=/hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:17.261 DEBUG:boto:Method: GET 2023-07-28T23:48:17.262 DEBUG:boto:Path: /hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:17.262 DEBUG:boto:Data: 2023-07-28T23:48:17.263 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.263 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:17.263 DEBUG:boto:Port: 8001 2023-07-28T23:48:17.264 DEBUG:boto:Params: {} 2023-07-28T23:48:17.264 DEBUG:boto:Token: None 2023-07-28T23:48:17.264 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:17.265 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AMfVmGwZ7GQe0gmFZPoUN8vvzww= 2023-07-28T23:48:17.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AMfVmGwZ7GQe0gmFZPoUN8vvzww=', 'Content-Length': '0'} 2023-07-28T23:48:17.267 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:01 GMT'), ('x-amz-version-id', 'RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000472db22e75649c8c-0064c453c1-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.268 DEBUG:boto:path=/hiiqos-11/dummy 2023-07-28T23:48:17.268 DEBUG:boto:auth_path=/hiiqos-11/dummy 2023-07-28T23:48:17.269 DEBUG:boto:path=/hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:17.269 DEBUG:boto:auth_path=/hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:17.269 DEBUG:boto:Method: GET 2023-07-28T23:48:17.270 DEBUG:boto:Path: /hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:17.270 DEBUG:boto:Data: 2023-07-28T23:48:17.270 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.271 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:17.271 DEBUG:boto:Port: 8000 2023-07-28T23:48:17.271 DEBUG:boto:Params: {} 2023-07-28T23:48:17.272 DEBUG:boto:Token: None 2023-07-28T23:48:17.272 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:17.273 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AMfVmGwZ7GQe0gmFZPoUN8vvzww= 2023-07-28T23:48:17.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AMfVmGwZ7GQe0gmFZPoUN8vvzww=', 'Content-Length': '0'} 2023-07-28T23:48:17.275 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:01 GMT'), ('x-amz-version-id', 'RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000013f2597147d3456b-0064c453c1-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.275 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:48:17.276 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:48:17.276 DEBUG:boto:path=/hiiqos-11/?prefix=dummy 2023-07-28T23:48:17.277 DEBUG:boto:auth_path=/hiiqos-11/?prefix=dummy 2023-07-28T23:48:17.277 DEBUG:boto:Method: GET 2023-07-28T23:48:17.277 DEBUG:boto:Path: /hiiqos-11/?prefix=dummy 2023-07-28T23:48:17.278 DEBUG:boto:Data: 2023-07-28T23:48:17.278 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.278 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:17.279 DEBUG:boto:Port: 8001 2023-07-28T23:48:17.279 DEBUG:boto:Params: {} 2023-07-28T23:48:17.279 DEBUG:boto:Token: None 2023-07-28T23:48:17.280 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/ 2023-07-28T23:48:17.280 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FR4zwVjrSL5xZkGEPFbh4PrQPL0= 2023-07-28T23:48:17.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FR4zwVjrSL5xZkGEPFbh4PrQPL0=', 'Content-Length': '0'} 2023-07-28T23:48:17.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f336f766ee810462-0064c453c1-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.284 DEBUG:boto:b'hiiqos-11dummy1000falsedummy2023-07-28T23:47:01.096Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:17.284 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:48:17.285 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:48:17.285 DEBUG:boto:path=/hiiqos-11/?prefix=dummy 2023-07-28T23:48:17.286 DEBUG:boto:auth_path=/hiiqos-11/?prefix=dummy 2023-07-28T23:48:17.286 DEBUG:boto:Method: GET 2023-07-28T23:48:17.286 DEBUG:boto:Path: /hiiqos-11/?prefix=dummy 2023-07-28T23:48:17.287 DEBUG:boto:Data: 2023-07-28T23:48:17.287 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.287 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:17.288 DEBUG:boto:Port: 8000 2023-07-28T23:48:17.288 DEBUG:boto:Params: {} 2023-07-28T23:48:17.289 DEBUG:boto:Token: None 2023-07-28T23:48:17.289 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/ 2023-07-28T23:48:17.289 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FR4zwVjrSL5xZkGEPFbh4PrQPL0= 2023-07-28T23:48:17.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FR4zwVjrSL5xZkGEPFbh4PrQPL0=', 'Content-Length': '0'} 2023-07-28T23:48:17.292 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000439dd4c4cf815a40-0064c453c1-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.292 DEBUG:boto:b'hiiqos-11dummy1000falsedummy2023-07-28T23:47:01.096Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:17.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-07-28T23:48:17.293 DEBUG:boto:path=/hiiqos-11/obj-a1 2023-07-28T23:48:17.294 DEBUG:boto:auth_path=/hiiqos-11/obj-a1 2023-07-28T23:48:17.294 DEBUG:boto:path=/hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:17.295 DEBUG:boto:auth_path=/hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:17.295 DEBUG:boto:Method: GET 2023-07-28T23:48:17.295 DEBUG:boto:Path: /hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:17.296 DEBUG:boto:Data: 2023-07-28T23:48:17.296 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.296 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:17.297 DEBUG:boto:Port: 8001 2023-07-28T23:48:17.297 DEBUG:boto:Params: {} 2023-07-28T23:48:17.298 DEBUG:boto:Token: None 2023-07-28T23:48:17.298 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:17.298 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gGKfjvDbGAH4P7NFnJQ7pEyl+dk= 2023-07-28T23:48:17.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gGKfjvDbGAH4P7NFnJQ7pEyl+dk=', 'Content-Length': '0'} 2023-07-28T23:48:17.301 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'S8uVclWm2JBYB4C047A9I83Y7ulD0.f'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bfcc617186f452e1-0064c453c1-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.302 DEBUG:boto:path=/hiiqos-11/obj-a1 2023-07-28T23:48:17.302 DEBUG:boto:auth_path=/hiiqos-11/obj-a1 2023-07-28T23:48:17.302 DEBUG:boto:path=/hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:17.303 DEBUG:boto:auth_path=/hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:17.303 DEBUG:boto:Method: GET 2023-07-28T23:48:17.303 DEBUG:boto:Path: /hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:17.304 DEBUG:boto:Data: 2023-07-28T23:48:17.304 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.305 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:17.305 DEBUG:boto:Port: 8000 2023-07-28T23:48:17.305 DEBUG:boto:Params: {} 2023-07-28T23:48:17.306 DEBUG:boto:Token: None 2023-07-28T23:48:17.306 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:17.307 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gGKfjvDbGAH4P7NFnJQ7pEyl+dk= 2023-07-28T23:48:17.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gGKfjvDbGAH4P7NFnJQ7pEyl+dk=', 'Content-Length': '0'} 2023-07-28T23:48:17.309 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'S8uVclWm2JBYB4C047A9I83Y7ulD0.f'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000062f1a9b3c3b3961e-0064c453c1-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.310 DEBUG:boto:path=/hiiqos-11/obj-a1 2023-07-28T23:48:17.310 DEBUG:boto:auth_path=/hiiqos-11/obj-a1 2023-07-28T23:48:17.310 DEBUG:boto:path=/hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:17.311 DEBUG:boto:auth_path=/hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:17.311 DEBUG:boto:Method: HEAD 2023-07-28T23:48:17.311 DEBUG:boto:Path: /hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:17.312 DEBUG:boto:Data: 2023-07-28T23:48:17.312 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.312 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:17.313 DEBUG:boto:Port: 8001 2023-07-28T23:48:17.313 DEBUG:boto:Params: {} 2023-07-28T23:48:17.313 DEBUG:boto:Token: None 2023-07-28T23:48:17.314 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:17.314 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aFB2UrQFXZW2t5gXGZQZdRpVHro= 2023-07-28T23:48:17.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aFB2UrQFXZW2t5gXGZQZdRpVHro=', 'Content-Length': '0'} 2023-07-28T23:48:17.316 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'S8uVclWm2JBYB4C047A9I83Y7ulD0.f'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000054821a5e011b1673-0064c453c1-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.317 DEBUG:boto:path=/hiiqos-11/obj-a1 2023-07-28T23:48:17.317 DEBUG:boto:auth_path=/hiiqos-11/obj-a1 2023-07-28T23:48:17.318 DEBUG:boto:path=/hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:17.318 DEBUG:boto:auth_path=/hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:17.318 DEBUG:boto:Method: HEAD 2023-07-28T23:48:17.319 DEBUG:boto:Path: /hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:17.319 DEBUG:boto:Data: 2023-07-28T23:48:17.319 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.320 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:17.320 DEBUG:boto:Port: 8000 2023-07-28T23:48:17.320 DEBUG:boto:Params: {} 2023-07-28T23:48:17.321 DEBUG:boto:Token: None 2023-07-28T23:48:17.321 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:17.322 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aFB2UrQFXZW2t5gXGZQZdRpVHro= 2023-07-28T23:48:17.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aFB2UrQFXZW2t5gXGZQZdRpVHro=', 'Content-Length': '0'} 2023-07-28T23:48:17.324 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'S8uVclWm2JBYB4C047A9I83Y7ulD0.f'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c7e722730f742bc7-0064c453c1-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-07-28T23:48:17.325 DEBUG:boto:path=/hiiqos-11/obj-a1 2023-07-28T23:48:17.325 DEBUG:boto:auth_path=/hiiqos-11/obj-a1 2023-07-28T23:48:17.325 DEBUG:boto:path=/hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:17.326 DEBUG:boto:auth_path=/hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:17.326 DEBUG:boto:Method: GET 2023-07-28T23:48:17.326 DEBUG:boto:Path: /hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:17.327 DEBUG:boto:Data: 2023-07-28T23:48:17.327 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.328 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:17.328 DEBUG:boto:Port: 8001 2023-07-28T23:48:17.328 DEBUG:boto:Params: {} 2023-07-28T23:48:17.329 DEBUG:boto:Token: None 2023-07-28T23:48:17.329 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:17.330 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gGKfjvDbGAH4P7NFnJQ7pEyl+dk= 2023-07-28T23:48:17.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gGKfjvDbGAH4P7NFnJQ7pEyl+dk=', 'Content-Length': '0'} 2023-07-28T23:48:17.332 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'S8uVclWm2JBYB4C047A9I83Y7ulD0.f'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dac22b34183fcb8e-0064c453c1-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.376 DEBUG:boto:path=/hiiqos-11/obj-a1 2023-07-28T23:48:17.377 DEBUG:boto:auth_path=/hiiqos-11/obj-a1 2023-07-28T23:48:17.378 DEBUG:boto:path=/hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:17.378 DEBUG:boto:auth_path=/hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:17.378 DEBUG:boto:Method: GET 2023-07-28T23:48:17.379 DEBUG:boto:Path: /hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:17.379 DEBUG:boto:Data: 2023-07-28T23:48:17.379 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.380 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:17.380 DEBUG:boto:Port: 8000 2023-07-28T23:48:17.381 DEBUG:boto:Params: {} 2023-07-28T23:48:17.381 DEBUG:boto:Token: None 2023-07-28T23:48:17.381 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:17.382 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gGKfjvDbGAH4P7NFnJQ7pEyl+dk= 2023-07-28T23:48:17.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gGKfjvDbGAH4P7NFnJQ7pEyl+dk=', 'Content-Length': '0'} 2023-07-28T23:48:17.384 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'S8uVclWm2JBYB4C047A9I83Y7ulD0.f'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e53516896be49d7d-0064c453c1-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.385 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:48:17.386 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:48:17.386 DEBUG:boto:path=/hiiqos-11/?prefix=obj-a1 2023-07-28T23:48:17.386 DEBUG:boto:auth_path=/hiiqos-11/?prefix=obj-a1 2023-07-28T23:48:17.387 DEBUG:boto:Method: GET 2023-07-28T23:48:17.387 DEBUG:boto:Path: /hiiqos-11/?prefix=obj-a1 2023-07-28T23:48:17.387 DEBUG:boto:Data: 2023-07-28T23:48:17.388 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.388 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:17.388 DEBUG:boto:Port: 8001 2023-07-28T23:48:17.389 DEBUG:boto:Params: {} 2023-07-28T23:48:17.389 DEBUG:boto:Token: None 2023-07-28T23:48:17.390 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/ 2023-07-28T23:48:17.390 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FR4zwVjrSL5xZkGEPFbh4PrQPL0= 2023-07-28T23:48:17.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FR4zwVjrSL5xZkGEPFbh4PrQPL0=', 'Content-Length': '0'} 2023-07-28T23:48:17.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d79b3588bf46023f-0064c453c1-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.393 DEBUG:boto:b'hiiqos-11obj-a11000falseobj-a12023-07-28T23:47:36.378Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:17.394 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:48:17.395 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:48:17.395 DEBUG:boto:path=/hiiqos-11/?prefix=obj-a1 2023-07-28T23:48:17.395 DEBUG:boto:auth_path=/hiiqos-11/?prefix=obj-a1 2023-07-28T23:48:17.396 DEBUG:boto:Method: GET 2023-07-28T23:48:17.396 DEBUG:boto:Path: /hiiqos-11/?prefix=obj-a1 2023-07-28T23:48:17.396 DEBUG:boto:Data: 2023-07-28T23:48:17.397 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.397 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:17.398 DEBUG:boto:Port: 8000 2023-07-28T23:48:17.398 DEBUG:boto:Params: {} 2023-07-28T23:48:17.398 DEBUG:boto:Token: None 2023-07-28T23:48:17.399 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/ 2023-07-28T23:48:17.399 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FR4zwVjrSL5xZkGEPFbh4PrQPL0= 2023-07-28T23:48:17.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FR4zwVjrSL5xZkGEPFbh4PrQPL0=', 'Content-Length': '0'} 2023-07-28T23:48:17.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095d0e4c6b3765bbd-0064c453c1-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.444 DEBUG:boto:b'hiiqos-11obj-a11000falseobj-a12023-07-28T23:47:36.378Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:17.445 DEBUG:boto:path=/hiiqos-11/obj-a1 2023-07-28T23:48:17.446 DEBUG:boto:auth_path=/hiiqos-11/obj-a1 2023-07-28T23:48:17.446 DEBUG:boto:Method: HEAD 2023-07-28T23:48:17.447 DEBUG:boto:Path: /hiiqos-11/obj-a1 2023-07-28T23:48:17.447 DEBUG:boto:Data: 2023-07-28T23:48:17.447 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.448 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:17.448 DEBUG:boto:Port: 8001 2023-07-28T23:48:17.448 DEBUG:boto:Params: {} 2023-07-28T23:48:17.449 DEBUG:boto:Token: None 2023-07-28T23:48:17.449 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/obj-a1 2023-07-28T23:48:17.450 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IkQsnMonAnvoRdTuseLCFiec2ek= 2023-07-28T23:48:17.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IkQsnMonAnvoRdTuseLCFiec2ek=', 'Content-Length': '0'} 2023-07-28T23:48:17.453 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'S8uVclWm2JBYB4C047A9I83Y7ulD0.f'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007789aa41ffecbfef-0064c453c1-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.453 DEBUG:boto:path=/hiiqos-11/obj-a1 2023-07-28T23:48:17.454 DEBUG:boto:auth_path=/hiiqos-11/obj-a1 2023-07-28T23:48:17.454 DEBUG:boto:Method: HEAD 2023-07-28T23:48:17.454 DEBUG:boto:Path: /hiiqos-11/obj-a1 2023-07-28T23:48:17.455 DEBUG:boto:Data: 2023-07-28T23:48:17.455 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.455 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:17.456 DEBUG:boto:Port: 8000 2023-07-28T23:48:17.456 DEBUG:boto:Params: {} 2023-07-28T23:48:17.456 DEBUG:boto:Token: None 2023-07-28T23:48:17.457 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/obj-a1 2023-07-28T23:48:17.457 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IkQsnMonAnvoRdTuseLCFiec2ek= 2023-07-28T23:48:17.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IkQsnMonAnvoRdTuseLCFiec2ek=', 'Content-Length': '0'} 2023-07-28T23:48:17.460 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'S8uVclWm2JBYB4C047A9I83Y7ulD0.f'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000078f75f8e0d690d5e-0064c453c1-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-07-28T23:48:17.461 DEBUG:boto:path=/hiiqos-11/obj-a1 2023-07-28T23:48:17.461 DEBUG:boto:auth_path=/hiiqos-11/obj-a1 2023-07-28T23:48:17.462 DEBUG:boto:path=/hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:17.462 DEBUG:boto:auth_path=/hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:17.462 DEBUG:boto:Method: GET 2023-07-28T23:48:17.463 DEBUG:boto:Path: /hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:17.463 DEBUG:boto:Data: 2023-07-28T23:48:17.464 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.464 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:17.465 DEBUG:boto:Port: 8001 2023-07-28T23:48:17.465 DEBUG:boto:Params: {} 2023-07-28T23:48:17.465 DEBUG:boto:Token: None 2023-07-28T23:48:17.466 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:17.466 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gGKfjvDbGAH4P7NFnJQ7pEyl+dk= 2023-07-28T23:48:17.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gGKfjvDbGAH4P7NFnJQ7pEyl+dk=', 'Content-Length': '0'} 2023-07-28T23:48:17.469 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'S8uVclWm2JBYB4C047A9I83Y7ulD0.f'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e92764c18df5e5b9-0064c453c1-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.469 DEBUG:boto:path=/hiiqos-11/obj-a1 2023-07-28T23:48:17.470 DEBUG:boto:auth_path=/hiiqos-11/obj-a1 2023-07-28T23:48:17.470 DEBUG:boto:path=/hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:17.470 DEBUG:boto:auth_path=/hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:17.471 DEBUG:boto:Method: GET 2023-07-28T23:48:17.471 DEBUG:boto:Path: /hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:17.472 DEBUG:boto:Data: 2023-07-28T23:48:17.472 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.472 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:17.473 DEBUG:boto:Port: 8000 2023-07-28T23:48:17.473 DEBUG:boto:Params: {} 2023-07-28T23:48:17.473 DEBUG:boto:Token: None 2023-07-28T23:48:17.474 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:17.474 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gGKfjvDbGAH4P7NFnJQ7pEyl+dk= 2023-07-28T23:48:17.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gGKfjvDbGAH4P7NFnJQ7pEyl+dk=', 'Content-Length': '0'} 2023-07-28T23:48:17.477 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'S8uVclWm2JBYB4C047A9I83Y7ulD0.f'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000028ec2ed4b03cf432-0064c453c1-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.477 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:48:17.478 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:48:17.478 DEBUG:boto:path=/hiiqos-11/?prefix=obj-a1 2023-07-28T23:48:17.479 DEBUG:boto:auth_path=/hiiqos-11/?prefix=obj-a1 2023-07-28T23:48:17.479 DEBUG:boto:Method: GET 2023-07-28T23:48:17.479 DEBUG:boto:Path: /hiiqos-11/?prefix=obj-a1 2023-07-28T23:48:17.480 DEBUG:boto:Data: 2023-07-28T23:48:17.480 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.480 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:17.481 DEBUG:boto:Port: 8001 2023-07-28T23:48:17.481 DEBUG:boto:Params: {} 2023-07-28T23:48:17.481 DEBUG:boto:Token: None 2023-07-28T23:48:17.482 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/ 2023-07-28T23:48:17.482 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FR4zwVjrSL5xZkGEPFbh4PrQPL0= 2023-07-28T23:48:17.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FR4zwVjrSL5xZkGEPFbh4PrQPL0=', 'Content-Length': '0'} 2023-07-28T23:48:17.485 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000726e5f508eaf7c63-0064c453c1-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.528 DEBUG:boto:b'hiiqos-11obj-a11000falseobj-a12023-07-28T23:47:36.378Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:17.529 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:48:17.530 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:48:17.530 DEBUG:boto:path=/hiiqos-11/?prefix=obj-a1 2023-07-28T23:48:17.530 DEBUG:boto:auth_path=/hiiqos-11/?prefix=obj-a1 2023-07-28T23:48:17.531 DEBUG:boto:Method: GET 2023-07-28T23:48:17.531 DEBUG:boto:Path: /hiiqos-11/?prefix=obj-a1 2023-07-28T23:48:17.532 DEBUG:boto:Data: 2023-07-28T23:48:17.532 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.532 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:17.533 DEBUG:boto:Port: 8000 2023-07-28T23:48:17.533 DEBUG:boto:Params: {} 2023-07-28T23:48:17.533 DEBUG:boto:Token: None 2023-07-28T23:48:17.534 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/ 2023-07-28T23:48:17.534 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FR4zwVjrSL5xZkGEPFbh4PrQPL0= 2023-07-28T23:48:17.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FR4zwVjrSL5xZkGEPFbh4PrQPL0=', 'Content-Length': '0'} 2023-07-28T23:48:17.537 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004258ed622f2f0efa-0064c453c1-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.538 DEBUG:boto:b'hiiqos-11obj-a11000falseobj-a12023-07-28T23:47:36.378Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:17.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-07-28T23:48:17.539 DEBUG:boto:path=/hiiqos-11/obj-a2 2023-07-28T23:48:17.539 DEBUG:boto:auth_path=/hiiqos-11/obj-a2 2023-07-28T23:48:17.540 DEBUG:boto:path=/hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:17.540 DEBUG:boto:auth_path=/hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:17.540 DEBUG:boto:Method: GET 2023-07-28T23:48:17.541 DEBUG:boto:Path: /hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:17.541 DEBUG:boto:Data: 2023-07-28T23:48:17.541 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.542 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:17.542 DEBUG:boto:Port: 8001 2023-07-28T23:48:17.542 DEBUG:boto:Params: {} 2023-07-28T23:48:17.543 DEBUG:boto:Token: None 2023-07-28T23:48:17.543 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:17.544 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gSO7COUtE44zRG5OrrZCRS1lQTQ= 2023-07-28T23:48:17.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gSO7COUtE44zRG5OrrZCRS1lQTQ=', 'Content-Length': '0'} 2023-07-28T23:48:17.546 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000881056a8b1139d50-0064c453c1-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.547 DEBUG:boto:path=/hiiqos-11/obj-a2 2023-07-28T23:48:17.547 DEBUG:boto:auth_path=/hiiqos-11/obj-a2 2023-07-28T23:48:17.548 DEBUG:boto:path=/hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:17.548 DEBUG:boto:auth_path=/hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:17.548 DEBUG:boto:Method: GET 2023-07-28T23:48:17.549 DEBUG:boto:Path: /hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:17.549 DEBUG:boto:Data: 2023-07-28T23:48:17.549 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.550 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:17.550 DEBUG:boto:Port: 8000 2023-07-28T23:48:17.550 DEBUG:boto:Params: {} 2023-07-28T23:48:17.551 DEBUG:boto:Token: None 2023-07-28T23:48:17.551 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:17.552 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gSO7COUtE44zRG5OrrZCRS1lQTQ= 2023-07-28T23:48:17.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gSO7COUtE44zRG5OrrZCRS1lQTQ=', 'Content-Length': '0'} 2023-07-28T23:48:17.554 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c19576ff711c39c3-0064c453c1-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.555 DEBUG:boto:path=/hiiqos-11/obj-a2 2023-07-28T23:48:17.556 DEBUG:boto:auth_path=/hiiqos-11/obj-a2 2023-07-28T23:48:17.556 DEBUG:boto:path=/hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:17.557 DEBUG:boto:auth_path=/hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:17.557 DEBUG:boto:Method: HEAD 2023-07-28T23:48:17.557 DEBUG:boto:Path: /hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:17.558 DEBUG:boto:Data: 2023-07-28T23:48:17.558 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.558 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:17.559 DEBUG:boto:Port: 8001 2023-07-28T23:48:17.559 DEBUG:boto:Params: {} 2023-07-28T23:48:17.559 DEBUG:boto:Token: None 2023-07-28T23:48:17.560 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:17.560 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mizDT+lsocN4qrQwmdsY5ezc+yg= 2023-07-28T23:48:17.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mizDT+lsocN4qrQwmdsY5ezc+yg=', 'Content-Length': '0'} 2023-07-28T23:48:17.562 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000912bf47f74d44bce-0064c453c1-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.563 DEBUG:boto:path=/hiiqos-11/obj-a2 2023-07-28T23:48:17.563 DEBUG:boto:auth_path=/hiiqos-11/obj-a2 2023-07-28T23:48:17.564 DEBUG:boto:path=/hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:17.564 DEBUG:boto:auth_path=/hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:17.564 DEBUG:boto:Method: HEAD 2023-07-28T23:48:17.565 DEBUG:boto:Path: /hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:17.565 DEBUG:boto:Data: 2023-07-28T23:48:17.565 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.566 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:17.566 DEBUG:boto:Port: 8000 2023-07-28T23:48:17.566 DEBUG:boto:Params: {} 2023-07-28T23:48:17.567 DEBUG:boto:Token: None 2023-07-28T23:48:17.567 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:17.568 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mizDT+lsocN4qrQwmdsY5ezc+yg= 2023-07-28T23:48:17.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mizDT+lsocN4qrQwmdsY5ezc+yg=', 'Content-Length': '0'} 2023-07-28T23:48:17.570 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa451de5b64b2955-0064c453c1-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-07-28T23:48:17.571 DEBUG:boto:path=/hiiqos-11/obj-a2 2023-07-28T23:48:17.571 DEBUG:boto:auth_path=/hiiqos-11/obj-a2 2023-07-28T23:48:17.572 DEBUG:boto:path=/hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:17.572 DEBUG:boto:auth_path=/hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:17.572 DEBUG:boto:Method: GET 2023-07-28T23:48:17.573 DEBUG:boto:Path: /hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:17.573 DEBUG:boto:Data: 2023-07-28T23:48:17.574 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.574 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:17.574 DEBUG:boto:Port: 8001 2023-07-28T23:48:17.575 DEBUG:boto:Params: {} 2023-07-28T23:48:17.575 DEBUG:boto:Token: None 2023-07-28T23:48:17.575 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:17.576 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gSO7COUtE44zRG5OrrZCRS1lQTQ= 2023-07-28T23:48:17.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gSO7COUtE44zRG5OrrZCRS1lQTQ=', 'Content-Length': '0'} 2023-07-28T23:48:17.578 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000008eb76be11d3d7b8-0064c453c1-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.579 DEBUG:boto:path=/hiiqos-11/obj-a2 2023-07-28T23:48:17.579 DEBUG:boto:auth_path=/hiiqos-11/obj-a2 2023-07-28T23:48:17.580 DEBUG:boto:path=/hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:17.580 DEBUG:boto:auth_path=/hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:17.580 DEBUG:boto:Method: GET 2023-07-28T23:48:17.581 DEBUG:boto:Path: /hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:17.581 DEBUG:boto:Data: 2023-07-28T23:48:17.581 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.582 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:17.582 DEBUG:boto:Port: 8000 2023-07-28T23:48:17.582 DEBUG:boto:Params: {} 2023-07-28T23:48:17.583 DEBUG:boto:Token: None 2023-07-28T23:48:17.583 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:17.584 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gSO7COUtE44zRG5OrrZCRS1lQTQ= 2023-07-28T23:48:17.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gSO7COUtE44zRG5OrrZCRS1lQTQ=', 'Content-Length': '0'} 2023-07-28T23:48:17.586 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000013dbce613825e2b4-0064c453c1-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.628 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:48:17.629 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:48:17.629 DEBUG:boto:path=/hiiqos-11/?prefix=obj-a2 2023-07-28T23:48:17.630 DEBUG:boto:auth_path=/hiiqos-11/?prefix=obj-a2 2023-07-28T23:48:17.630 DEBUG:boto:Method: GET 2023-07-28T23:48:17.630 DEBUG:boto:Path: /hiiqos-11/?prefix=obj-a2 2023-07-28T23:48:17.631 DEBUG:boto:Data: 2023-07-28T23:48:17.631 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.631 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:17.632 DEBUG:boto:Port: 8001 2023-07-28T23:48:17.632 DEBUG:boto:Params: {} 2023-07-28T23:48:17.632 DEBUG:boto:Token: None 2023-07-28T23:48:17.633 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/ 2023-07-28T23:48:17.633 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FR4zwVjrSL5xZkGEPFbh4PrQPL0= 2023-07-28T23:48:17.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FR4zwVjrSL5xZkGEPFbh4PrQPL0=', 'Content-Length': '0'} 2023-07-28T23:48:17.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d45df4cf77725ccd-0064c453c1-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.637 DEBUG:boto:b'hiiqos-11obj-a21000falseobj-a22023-07-28T23:47:36.499Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:17.638 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:48:17.638 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:48:17.638 DEBUG:boto:path=/hiiqos-11/?prefix=obj-a2 2023-07-28T23:48:17.639 DEBUG:boto:auth_path=/hiiqos-11/?prefix=obj-a2 2023-07-28T23:48:17.639 DEBUG:boto:Method: GET 2023-07-28T23:48:17.639 DEBUG:boto:Path: /hiiqos-11/?prefix=obj-a2 2023-07-28T23:48:17.640 DEBUG:boto:Data: 2023-07-28T23:48:17.640 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.641 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:17.641 DEBUG:boto:Port: 8000 2023-07-28T23:48:17.641 DEBUG:boto:Params: {} 2023-07-28T23:48:17.642 DEBUG:boto:Token: None 2023-07-28T23:48:17.642 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/ 2023-07-28T23:48:17.643 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FR4zwVjrSL5xZkGEPFbh4PrQPL0= 2023-07-28T23:48:17.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FR4zwVjrSL5xZkGEPFbh4PrQPL0=', 'Content-Length': '0'} 2023-07-28T23:48:17.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eac90669bc6799f5-0064c453c1-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.688 DEBUG:boto:b'hiiqos-11obj-a21000falseobj-a22023-07-28T23:47:36.499Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:17.689 DEBUG:boto:path=/hiiqos-11/obj-a2 2023-07-28T23:48:17.690 DEBUG:boto:auth_path=/hiiqos-11/obj-a2 2023-07-28T23:48:17.690 DEBUG:boto:Method: HEAD 2023-07-28T23:48:17.690 DEBUG:boto:Path: /hiiqos-11/obj-a2 2023-07-28T23:48:17.691 DEBUG:boto:Data: 2023-07-28T23:48:17.691 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.691 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:17.692 DEBUG:boto:Port: 8001 2023-07-28T23:48:17.692 DEBUG:boto:Params: {} 2023-07-28T23:48:17.693 DEBUG:boto:Token: None 2023-07-28T23:48:17.694 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/obj-a2 2023-07-28T23:48:17.694 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zKCb4Hz6J9x379G/1GFl072f1Yo= 2023-07-28T23:48:17.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zKCb4Hz6J9x379G/1GFl072f1Yo=', 'Content-Length': '0'} 2023-07-28T23:48:17.697 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009847a3aaad78382c-0064c453c1-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.698 DEBUG:boto:path=/hiiqos-11/obj-a2 2023-07-28T23:48:17.699 DEBUG:boto:auth_path=/hiiqos-11/obj-a2 2023-07-28T23:48:17.700 DEBUG:boto:Method: HEAD 2023-07-28T23:48:17.700 DEBUG:boto:Path: /hiiqos-11/obj-a2 2023-07-28T23:48:17.701 DEBUG:boto:Data: 2023-07-28T23:48:17.702 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.703 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:17.703 DEBUG:boto:Port: 8000 2023-07-28T23:48:17.704 DEBUG:boto:Params: {} 2023-07-28T23:48:17.704 DEBUG:boto:Token: None 2023-07-28T23:48:17.705 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/obj-a2 2023-07-28T23:48:17.705 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zKCb4Hz6J9x379G/1GFl072f1Yo= 2023-07-28T23:48:17.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zKCb4Hz6J9x379G/1GFl072f1Yo=', 'Content-Length': '0'} 2023-07-28T23:48:17.708 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d04558a7b06cbd09-0064c453c1-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-07-28T23:48:17.709 DEBUG:boto:path=/hiiqos-11/obj-a2 2023-07-28T23:48:17.709 DEBUG:boto:auth_path=/hiiqos-11/obj-a2 2023-07-28T23:48:17.710 DEBUG:boto:path=/hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:17.710 DEBUG:boto:auth_path=/hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:17.710 DEBUG:boto:Method: GET 2023-07-28T23:48:17.711 DEBUG:boto:Path: /hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:17.711 DEBUG:boto:Data: 2023-07-28T23:48:17.711 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.712 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:17.712 DEBUG:boto:Port: 8001 2023-07-28T23:48:17.713 DEBUG:boto:Params: {} 2023-07-28T23:48:17.713 DEBUG:boto:Token: None 2023-07-28T23:48:17.714 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:17.714 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gSO7COUtE44zRG5OrrZCRS1lQTQ= 2023-07-28T23:48:17.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gSO7COUtE44zRG5OrrZCRS1lQTQ=', 'Content-Length': '0'} 2023-07-28T23:48:17.717 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000075f2f1166999f53b-0064c453c1-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.718 DEBUG:boto:path=/hiiqos-11/obj-a2 2023-07-28T23:48:17.718 DEBUG:boto:auth_path=/hiiqos-11/obj-a2 2023-07-28T23:48:17.719 DEBUG:boto:path=/hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:17.719 DEBUG:boto:auth_path=/hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:17.720 DEBUG:boto:Method: GET 2023-07-28T23:48:17.720 DEBUG:boto:Path: /hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:17.721 DEBUG:boto:Data: 2023-07-28T23:48:17.722 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.723 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:17.724 DEBUG:boto:Port: 8000 2023-07-28T23:48:17.725 DEBUG:boto:Params: {} 2023-07-28T23:48:17.726 DEBUG:boto:Token: None 2023-07-28T23:48:17.727 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:17.727 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gSO7COUtE44zRG5OrrZCRS1lQTQ= 2023-07-28T23:48:17.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gSO7COUtE44zRG5OrrZCRS1lQTQ=', 'Content-Length': '0'} 2023-07-28T23:48:17.730 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000094f5ffb670f8b297-0064c453c1-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.731 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:48:17.731 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:48:17.731 DEBUG:boto:path=/hiiqos-11/?prefix=obj-a2 2023-07-28T23:48:17.732 DEBUG:boto:auth_path=/hiiqos-11/?prefix=obj-a2 2023-07-28T23:48:17.732 DEBUG:boto:Method: GET 2023-07-28T23:48:17.732 DEBUG:boto:Path: /hiiqos-11/?prefix=obj-a2 2023-07-28T23:48:17.733 DEBUG:boto:Data: 2023-07-28T23:48:17.733 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.734 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:17.734 DEBUG:boto:Port: 8001 2023-07-28T23:48:17.734 DEBUG:boto:Params: {} 2023-07-28T23:48:17.735 DEBUG:boto:Token: None 2023-07-28T23:48:17.735 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/ 2023-07-28T23:48:17.736 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FR4zwVjrSL5xZkGEPFbh4PrQPL0= 2023-07-28T23:48:17.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FR4zwVjrSL5xZkGEPFbh4PrQPL0=', 'Content-Length': '0'} 2023-07-28T23:48:17.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000627ca301bf729032-0064c453c1-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.784 DEBUG:boto:b'hiiqos-11obj-a21000falseobj-a22023-07-28T23:47:36.499Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:17.786 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:48:17.787 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:48:17.788 DEBUG:boto:path=/hiiqos-11/?prefix=obj-a2 2023-07-28T23:48:17.788 DEBUG:boto:auth_path=/hiiqos-11/?prefix=obj-a2 2023-07-28T23:48:17.789 DEBUG:boto:Method: GET 2023-07-28T23:48:17.790 DEBUG:boto:Path: /hiiqos-11/?prefix=obj-a2 2023-07-28T23:48:17.790 DEBUG:boto:Data: 2023-07-28T23:48:17.791 DEBUG:boto:Headers: {} 2023-07-28T23:48:17.792 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:17.792 DEBUG:boto:Port: 8000 2023-07-28T23:48:17.792 DEBUG:boto:Params: {} 2023-07-28T23:48:17.793 DEBUG:boto:Token: None 2023-07-28T23:48:17.793 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:17 GMT /hiiqos-11/ 2023-07-28T23:48:17.794 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FR4zwVjrSL5xZkGEPFbh4PrQPL0= 2023-07-28T23:48:17.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FR4zwVjrSL5xZkGEPFbh4PrQPL0=', 'Content-Length': '0'} 2023-07-28T23:48:17.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b08fc6d09c6748a-0064c453c1-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:17.797 DEBUG:boto:b'hiiqos-11obj-a21000falseobj-a22023-07-28T23:47:36.499Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:17.798 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-11 zones={a2, a1} 2023-07-28T23:48:17.798 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-12 2023-07-28T23:48:17.799 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:48:17.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:48:17.800 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-12 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-28T23:48:17.859 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:48:17.860 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:48:17.895 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:48:17.894+0000 7fd4c654b880 1 bucket sync caught up with source: 2023-07-28T23:48:17.896 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000002.17.11, , , 00000000002.6.11, 00000000010.26.6, ] 2023-07-28T23:48:17.896 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000002.17.11, , , 00000000002.6.11, 00000000010.26.6, ] 2023-07-28T23:48:17.896 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:48:17.894+0000 7fd4c654b880 0 bucket checkpoint complete 2023-07-28T23:48:17.908 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-12 2023-07-28T23:48:17.909 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:48:17.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:48:17.911 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-12 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-28T23:48:17.965 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:48:17.965 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:48:17.996 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:48:17.995+0000 7fda8bd24880 1 bucket sync caught up with source: 2023-07-28T23:48:17.996 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000003.37.10, , , 00000000010.27.6, 00000000003.26.10, ] 2023-07-28T23:48:17.996 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000003.37.10, , , 00000000010.27.6, 00000000003.26.10, ] 2023-07-28T23:48:17.996 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:48:17.995+0000 7fda8bd24880 0 bucket checkpoint complete 2023-07-28T23:48:18.008 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-12 zones={a2, a1} 2023-07-28T23:48:18.008 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:48:18.009 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:48:18.009 DEBUG:boto:Method: HEAD 2023-07-28T23:48:18.009 DEBUG:boto:Path: /hiiqos-12/ 2023-07-28T23:48:18.010 DEBUG:boto:Data: 2023-07-28T23:48:18.010 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.010 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:18.011 DEBUG:boto:Port: 8001 2023-07-28T23:48:18.011 DEBUG:boto:Params: {} 2023-07-28T23:48:18.011 DEBUG:boto:Token: None 2023-07-28T23:48:18.012 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/ 2023-07-28T23:48:18.012 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yegpm6+K/aPT0lwTiE+iqMLkzLE= 2023-07-28T23:48:18.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yegpm6+K/aPT0lwTiE+iqMLkzLE=', 'Content-Length': '0'} 2023-07-28T23:48:18.015 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fca63db199a3c0be-0064c453c2-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.015 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:48:18.016 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:48:18.016 DEBUG:boto:Method: HEAD 2023-07-28T23:48:18.017 DEBUG:boto:Path: /hiiqos-12/ 2023-07-28T23:48:18.017 DEBUG:boto:Data: 2023-07-28T23:48:18.017 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.018 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:18.018 DEBUG:boto:Port: 8000 2023-07-28T23:48:18.018 DEBUG:boto:Params: {} 2023-07-28T23:48:18.019 DEBUG:boto:Token: None 2023-07-28T23:48:18.019 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/ 2023-07-28T23:48:18.019 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yegpm6+K/aPT0lwTiE+iqMLkzLE= 2023-07-28T23:48:18.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yegpm6+K/aPT0lwTiE+iqMLkzLE=', 'Content-Length': '0'} 2023-07-28T23:48:18.022 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b77d2c3717d1b64c-0064c453c2-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.023 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-28T23:48:18.023 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:48:18.023 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:48:18.024 DEBUG:boto:path=/hiiqos-12/?versions&max-keys=999 2023-07-28T23:48:18.024 DEBUG:boto:auth_path=/hiiqos-12/?versions&max-keys=999 2023-07-28T23:48:18.024 DEBUG:boto:Method: GET 2023-07-28T23:48:18.025 DEBUG:boto:Path: /hiiqos-12/?versions&max-keys=999 2023-07-28T23:48:18.025 DEBUG:boto:Data: 2023-07-28T23:48:18.025 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.026 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:18.026 DEBUG:boto:Port: 8001 2023-07-28T23:48:18.026 DEBUG:boto:Params: {} 2023-07-28T23:48:18.027 DEBUG:boto:Token: None 2023-07-28T23:48:18.027 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/?versions 2023-07-28T23:48:18.028 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BpIJY8qv2FSSz1wFIk+xLj5odpI= 2023-07-28T23:48:18.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BpIJY8qv2FSSz1wFIk+xLj5odpI=', 'Content-Length': '0'} 2023-07-28T23:48:18.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcadf9ec5131b6b6-0064c453c2-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.031 DEBUG:boto:b'hiiqos-12999falsedummycVvFSQn3t1--pfP44jwpm7HeB4ciTyVtrue2023-07-28T23:47:16.239Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1Zt8NMtOe11K3S282QQxMGXP84LEhPzttrue2023-07-28T23:47:36.626Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2KycZQE1frBdJnvJd6sV3PeY5cRPnOLFtrue2023-07-28T23:47:36.751Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:18.032 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-07-28T23:48:18.032 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-07-28T23:48:18.032 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-07-28T23:48:18.033 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-28T23:48:18.033 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:48:18.033 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:48:18.034 DEBUG:boto:path=/hiiqos-12/?versions&max-keys=999 2023-07-28T23:48:18.034 DEBUG:boto:auth_path=/hiiqos-12/?versions&max-keys=999 2023-07-28T23:48:18.034 DEBUG:boto:Method: GET 2023-07-28T23:48:18.035 DEBUG:boto:Path: /hiiqos-12/?versions&max-keys=999 2023-07-28T23:48:18.035 DEBUG:boto:Data: 2023-07-28T23:48:18.035 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.036 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:18.036 DEBUG:boto:Port: 8000 2023-07-28T23:48:18.036 DEBUG:boto:Params: {} 2023-07-28T23:48:18.037 DEBUG:boto:Token: None 2023-07-28T23:48:18.037 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/?versions 2023-07-28T23:48:18.038 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BpIJY8qv2FSSz1wFIk+xLj5odpI= 2023-07-28T23:48:18.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BpIJY8qv2FSSz1wFIk+xLj5odpI=', 'Content-Length': '0'} 2023-07-28T23:48:18.040 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bc2bb29ac10f644-0064c453c2-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.041 DEBUG:boto:b'hiiqos-12999falsedummycVvFSQn3t1--pfP44jwpm7HeB4ciTyVtrue2023-07-28T23:47:16.239Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1Zt8NMtOe11K3S282QQxMGXP84LEhPzttrue2023-07-28T23:47:36.626Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2KycZQE1frBdJnvJd6sV3PeY5cRPnOLFtrue2023-07-28T23:47:36.751Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:18.042 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-07-28T23:48:18.042 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-07-28T23:48:18.042 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-07-28T23:48:18.043 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:48:18.043 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:48:18.043 DEBUG:boto:path=/hiiqos-12/?versions&max-keys=999 2023-07-28T23:48:18.044 DEBUG:boto:auth_path=/hiiqos-12/?versions&max-keys=999 2023-07-28T23:48:18.044 DEBUG:boto:Method: GET 2023-07-28T23:48:18.045 DEBUG:boto:Path: /hiiqos-12/?versions&max-keys=999 2023-07-28T23:48:18.045 DEBUG:boto:Data: 2023-07-28T23:48:18.045 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.046 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:18.046 DEBUG:boto:Port: 8001 2023-07-28T23:48:18.046 DEBUG:boto:Params: {} 2023-07-28T23:48:18.047 DEBUG:boto:Token: None 2023-07-28T23:48:18.047 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/?versions 2023-07-28T23:48:18.047 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BpIJY8qv2FSSz1wFIk+xLj5odpI= 2023-07-28T23:48:18.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BpIJY8qv2FSSz1wFIk+xLj5odpI=', 'Content-Length': '0'} 2023-07-28T23:48:18.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d90a101863289fb2-0064c453c2-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.051 DEBUG:boto:b'hiiqos-12999falsedummycVvFSQn3t1--pfP44jwpm7HeB4ciTyVtrue2023-07-28T23:47:16.239Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1Zt8NMtOe11K3S282QQxMGXP84LEhPzttrue2023-07-28T23:47:36.626Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2KycZQE1frBdJnvJd6sV3PeY5cRPnOLFtrue2023-07-28T23:47:36.751Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:18.051 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:48:18.052 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:48:18.052 DEBUG:boto:path=/hiiqos-12/?versions&max-keys=999 2023-07-28T23:48:18.053 DEBUG:boto:auth_path=/hiiqos-12/?versions&max-keys=999 2023-07-28T23:48:18.053 DEBUG:boto:Method: GET 2023-07-28T23:48:18.053 DEBUG:boto:Path: /hiiqos-12/?versions&max-keys=999 2023-07-28T23:48:18.054 DEBUG:boto:Data: 2023-07-28T23:48:18.054 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.054 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:18.055 DEBUG:boto:Port: 8000 2023-07-28T23:48:18.055 DEBUG:boto:Params: {} 2023-07-28T23:48:18.056 DEBUG:boto:Token: None 2023-07-28T23:48:18.056 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/?versions 2023-07-28T23:48:18.056 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BpIJY8qv2FSSz1wFIk+xLj5odpI= 2023-07-28T23:48:18.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BpIJY8qv2FSSz1wFIk+xLj5odpI=', 'Content-Length': '0'} 2023-07-28T23:48:18.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005233305ff48e76a9-0064c453c2-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.059 DEBUG:boto:b'hiiqos-12999falsedummycVvFSQn3t1--pfP44jwpm7HeB4ciTyVtrue2023-07-28T23:47:16.239Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1Zt8NMtOe11K3S282QQxMGXP84LEhPzttrue2023-07-28T23:47:36.626Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2KycZQE1frBdJnvJd6sV3PeY5cRPnOLFtrue2023-07-28T23:47:36.751Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:18.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-07-28T23:48:18.060 DEBUG:boto:path=/hiiqos-12/dummy 2023-07-28T23:48:18.061 DEBUG:boto:auth_path=/hiiqos-12/dummy 2023-07-28T23:48:18.061 DEBUG:boto:path=/hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:18.062 DEBUG:boto:auth_path=/hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:18.062 DEBUG:boto:Method: GET 2023-07-28T23:48:18.062 DEBUG:boto:Path: /hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:18.063 DEBUG:boto:Data: 2023-07-28T23:48:18.063 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.063 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:18.064 DEBUG:boto:Port: 8001 2023-07-28T23:48:18.064 DEBUG:boto:Params: {} 2023-07-28T23:48:18.065 DEBUG:boto:Token: None 2023-07-28T23:48:18.066 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:18.067 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sM8zuMVTAa+9ZmO7pyR+kqpPfTA= 2023-07-28T23:48:18.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sM8zuMVTAa+9ZmO7pyR+kqpPfTA=', 'Content-Length': '0'} 2023-07-28T23:48:18.070 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:16 GMT'), ('x-amz-version-id', 'cVvFSQn3t1--pfP44jwpm7HeB4ciTyV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ce3c6933874f276b-0064c453c2-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.071 DEBUG:boto:path=/hiiqos-12/dummy 2023-07-28T23:48:18.072 DEBUG:boto:auth_path=/hiiqos-12/dummy 2023-07-28T23:48:18.073 DEBUG:boto:path=/hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:18.073 DEBUG:boto:auth_path=/hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:18.074 DEBUG:boto:Method: GET 2023-07-28T23:48:18.075 DEBUG:boto:Path: /hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:18.076 DEBUG:boto:Data: 2023-07-28T23:48:18.077 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.077 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:18.078 DEBUG:boto:Port: 8000 2023-07-28T23:48:18.079 DEBUG:boto:Params: {} 2023-07-28T23:48:18.080 DEBUG:boto:Token: None 2023-07-28T23:48:18.081 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:18.082 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sM8zuMVTAa+9ZmO7pyR+kqpPfTA= 2023-07-28T23:48:18.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sM8zuMVTAa+9ZmO7pyR+kqpPfTA=', 'Content-Length': '0'} 2023-07-28T23:48:18.085 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:16 GMT'), ('x-amz-version-id', 'cVvFSQn3t1--pfP44jwpm7HeB4ciTyV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009fa9f34e1ea3ef82-0064c453c2-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.086 DEBUG:boto:path=/hiiqos-12/dummy 2023-07-28T23:48:18.087 DEBUG:boto:auth_path=/hiiqos-12/dummy 2023-07-28T23:48:18.087 DEBUG:boto:path=/hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:18.088 DEBUG:boto:auth_path=/hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:18.088 DEBUG:boto:Method: HEAD 2023-07-28T23:48:18.089 DEBUG:boto:Path: /hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:18.090 DEBUG:boto:Data: 2023-07-28T23:48:18.090 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.091 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:18.092 DEBUG:boto:Port: 8001 2023-07-28T23:48:18.092 DEBUG:boto:Params: {} 2023-07-28T23:48:18.093 DEBUG:boto:Token: None 2023-07-28T23:48:18.094 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:18.095 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hrU3rGo8or83cFZmx7xMIfI4lk0= 2023-07-28T23:48:18.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hrU3rGo8or83cFZmx7xMIfI4lk0=', 'Content-Length': '0'} 2023-07-28T23:48:18.098 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:16 GMT'), ('x-amz-version-id', 'cVvFSQn3t1--pfP44jwpm7HeB4ciTyV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004e4b27d5171cd979-0064c453c2-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.098 DEBUG:boto:path=/hiiqos-12/dummy 2023-07-28T23:48:18.099 DEBUG:boto:auth_path=/hiiqos-12/dummy 2023-07-28T23:48:18.099 DEBUG:boto:path=/hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:18.100 DEBUG:boto:auth_path=/hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:18.100 DEBUG:boto:Method: HEAD 2023-07-28T23:48:18.100 DEBUG:boto:Path: /hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:18.101 DEBUG:boto:Data: 2023-07-28T23:48:18.101 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.102 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:18.103 DEBUG:boto:Port: 8000 2023-07-28T23:48:18.104 DEBUG:boto:Params: {} 2023-07-28T23:48:18.105 DEBUG:boto:Token: None 2023-07-28T23:48:18.106 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:18.107 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hrU3rGo8or83cFZmx7xMIfI4lk0= 2023-07-28T23:48:18.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hrU3rGo8or83cFZmx7xMIfI4lk0=', 'Content-Length': '0'} 2023-07-28T23:48:18.109 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:16 GMT'), ('x-amz-version-id', 'cVvFSQn3t1--pfP44jwpm7HeB4ciTyV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005fe6553a1632fb60-0064c453c2-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-07-28T23:48:18.110 DEBUG:boto:path=/hiiqos-12/dummy 2023-07-28T23:48:18.111 DEBUG:boto:auth_path=/hiiqos-12/dummy 2023-07-28T23:48:18.111 DEBUG:boto:path=/hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:18.112 DEBUG:boto:auth_path=/hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:18.112 DEBUG:boto:Method: GET 2023-07-28T23:48:18.112 DEBUG:boto:Path: /hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:18.113 DEBUG:boto:Data: 2023-07-28T23:48:18.113 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.113 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:18.114 DEBUG:boto:Port: 8001 2023-07-28T23:48:18.114 DEBUG:boto:Params: {} 2023-07-28T23:48:18.114 DEBUG:boto:Token: None 2023-07-28T23:48:18.115 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:18.115 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sM8zuMVTAa+9ZmO7pyR+kqpPfTA= 2023-07-28T23:48:18.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sM8zuMVTAa+9ZmO7pyR+kqpPfTA=', 'Content-Length': '0'} 2023-07-28T23:48:18.117 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:16 GMT'), ('x-amz-version-id', 'cVvFSQn3t1--pfP44jwpm7HeB4ciTyV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000017d1c9073c834cc-0064c453c2-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.118 DEBUG:boto:path=/hiiqos-12/dummy 2023-07-28T23:48:18.118 DEBUG:boto:auth_path=/hiiqos-12/dummy 2023-07-28T23:48:18.119 DEBUG:boto:path=/hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:18.119 DEBUG:boto:auth_path=/hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:18.119 DEBUG:boto:Method: GET 2023-07-28T23:48:18.120 DEBUG:boto:Path: /hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:18.120 DEBUG:boto:Data: 2023-07-28T23:48:18.121 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.121 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:18.121 DEBUG:boto:Port: 8000 2023-07-28T23:48:18.122 DEBUG:boto:Params: {} 2023-07-28T23:48:18.122 DEBUG:boto:Token: None 2023-07-28T23:48:18.122 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:18.123 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sM8zuMVTAa+9ZmO7pyR+kqpPfTA= 2023-07-28T23:48:18.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sM8zuMVTAa+9ZmO7pyR+kqpPfTA=', 'Content-Length': '0'} 2023-07-28T23:48:18.125 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:16 GMT'), ('x-amz-version-id', 'cVvFSQn3t1--pfP44jwpm7HeB4ciTyV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000093911c4ce96ff82a-0064c453c2-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.126 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:48:18.126 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:48:18.126 DEBUG:boto:path=/hiiqos-12/?prefix=dummy 2023-07-28T23:48:18.127 DEBUG:boto:auth_path=/hiiqos-12/?prefix=dummy 2023-07-28T23:48:18.127 DEBUG:boto:Method: GET 2023-07-28T23:48:18.127 DEBUG:boto:Path: /hiiqos-12/?prefix=dummy 2023-07-28T23:48:18.128 DEBUG:boto:Data: 2023-07-28T23:48:18.128 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.128 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:18.129 DEBUG:boto:Port: 8001 2023-07-28T23:48:18.129 DEBUG:boto:Params: {} 2023-07-28T23:48:18.130 DEBUG:boto:Token: None 2023-07-28T23:48:18.130 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/ 2023-07-28T23:48:18.130 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gm7hDmp7MwboUuP1lVfpucLTddA= 2023-07-28T23:48:18.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gm7hDmp7MwboUuP1lVfpucLTddA=', 'Content-Length': '0'} 2023-07-28T23:48:18.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d523188891b65c11-0064c453c2-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.133 DEBUG:boto:b'hiiqos-12dummy1000falsedummy2023-07-28T23:47:16.239Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:18.134 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:48:18.134 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:48:18.135 DEBUG:boto:path=/hiiqos-12/?prefix=dummy 2023-07-28T23:48:18.135 DEBUG:boto:auth_path=/hiiqos-12/?prefix=dummy 2023-07-28T23:48:18.135 DEBUG:boto:Method: GET 2023-07-28T23:48:18.136 DEBUG:boto:Path: /hiiqos-12/?prefix=dummy 2023-07-28T23:48:18.136 DEBUG:boto:Data: 2023-07-28T23:48:18.136 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.137 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:18.137 DEBUG:boto:Port: 8000 2023-07-28T23:48:18.137 DEBUG:boto:Params: {} 2023-07-28T23:48:18.138 DEBUG:boto:Token: None 2023-07-28T23:48:18.138 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/ 2023-07-28T23:48:18.139 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gm7hDmp7MwboUuP1lVfpucLTddA= 2023-07-28T23:48:18.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gm7hDmp7MwboUuP1lVfpucLTddA=', 'Content-Length': '0'} 2023-07-28T23:48:18.141 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f7313ab5a55ca2a-0064c453c2-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.188 DEBUG:boto:b'hiiqos-12dummy1000falsedummy2023-07-28T23:47:16.239Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:18.189 DEBUG:boto:path=/hiiqos-12/dummy 2023-07-28T23:48:18.190 DEBUG:boto:auth_path=/hiiqos-12/dummy 2023-07-28T23:48:18.190 DEBUG:boto:Method: HEAD 2023-07-28T23:48:18.190 DEBUG:boto:Path: /hiiqos-12/dummy 2023-07-28T23:48:18.191 DEBUG:boto:Data: 2023-07-28T23:48:18.191 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.191 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:18.192 DEBUG:boto:Port: 8001 2023-07-28T23:48:18.192 DEBUG:boto:Params: {} 2023-07-28T23:48:18.192 DEBUG:boto:Token: None 2023-07-28T23:48:18.193 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/dummy 2023-07-28T23:48:18.193 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GuZh8QE1DVcqNdiLCcYg8PgeMLg= 2023-07-28T23:48:18.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GuZh8QE1DVcqNdiLCcYg8PgeMLg=', 'Content-Length': '0'} 2023-07-28T23:48:18.196 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:16 GMT'), ('x-amz-version-id', 'cVvFSQn3t1--pfP44jwpm7HeB4ciTyV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f70cdc19fe5cbd0f-0064c453c2-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.196 DEBUG:boto:path=/hiiqos-12/dummy 2023-07-28T23:48:18.197 DEBUG:boto:auth_path=/hiiqos-12/dummy 2023-07-28T23:48:18.197 DEBUG:boto:Method: HEAD 2023-07-28T23:48:18.197 DEBUG:boto:Path: /hiiqos-12/dummy 2023-07-28T23:48:18.198 DEBUG:boto:Data: 2023-07-28T23:48:18.198 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.198 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:18.199 DEBUG:boto:Port: 8000 2023-07-28T23:48:18.199 DEBUG:boto:Params: {} 2023-07-28T23:48:18.199 DEBUG:boto:Token: None 2023-07-28T23:48:18.200 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/dummy 2023-07-28T23:48:18.200 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GuZh8QE1DVcqNdiLCcYg8PgeMLg= 2023-07-28T23:48:18.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GuZh8QE1DVcqNdiLCcYg8PgeMLg=', 'Content-Length': '0'} 2023-07-28T23:48:18.203 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:16 GMT'), ('x-amz-version-id', 'cVvFSQn3t1--pfP44jwpm7HeB4ciTyV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ed0a53cd4de2272e-0064c453c2-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-07-28T23:48:18.203 DEBUG:boto:path=/hiiqos-12/dummy 2023-07-28T23:48:18.204 DEBUG:boto:auth_path=/hiiqos-12/dummy 2023-07-28T23:48:18.204 DEBUG:boto:path=/hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:18.205 DEBUG:boto:auth_path=/hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:18.205 DEBUG:boto:Method: GET 2023-07-28T23:48:18.205 DEBUG:boto:Path: /hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:18.206 DEBUG:boto:Data: 2023-07-28T23:48:18.206 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.206 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:18.207 DEBUG:boto:Port: 8001 2023-07-28T23:48:18.207 DEBUG:boto:Params: {} 2023-07-28T23:48:18.207 DEBUG:boto:Token: None 2023-07-28T23:48:18.208 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:18.208 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sM8zuMVTAa+9ZmO7pyR+kqpPfTA= 2023-07-28T23:48:18.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sM8zuMVTAa+9ZmO7pyR+kqpPfTA=', 'Content-Length': '0'} 2023-07-28T23:48:18.210 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:16 GMT'), ('x-amz-version-id', 'cVvFSQn3t1--pfP44jwpm7HeB4ciTyV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000013ec82764920afd7-0064c453c2-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.211 DEBUG:boto:path=/hiiqos-12/dummy 2023-07-28T23:48:18.212 DEBUG:boto:auth_path=/hiiqos-12/dummy 2023-07-28T23:48:18.212 DEBUG:boto:path=/hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:18.212 DEBUG:boto:auth_path=/hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:18.213 DEBUG:boto:Method: GET 2023-07-28T23:48:18.213 DEBUG:boto:Path: /hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:18.213 DEBUG:boto:Data: 2023-07-28T23:48:18.214 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.214 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:18.214 DEBUG:boto:Port: 8000 2023-07-28T23:48:18.215 DEBUG:boto:Params: {} 2023-07-28T23:48:18.215 DEBUG:boto:Token: None 2023-07-28T23:48:18.216 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:18.216 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sM8zuMVTAa+9ZmO7pyR+kqpPfTA= 2023-07-28T23:48:18.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sM8zuMVTAa+9ZmO7pyR+kqpPfTA=', 'Content-Length': '0'} 2023-07-28T23:48:18.218 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:16 GMT'), ('x-amz-version-id', 'cVvFSQn3t1--pfP44jwpm7HeB4ciTyV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009bfb8e7950a1c729-0064c453c2-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.219 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:48:18.219 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:48:18.219 DEBUG:boto:path=/hiiqos-12/?prefix=dummy 2023-07-28T23:48:18.220 DEBUG:boto:auth_path=/hiiqos-12/?prefix=dummy 2023-07-28T23:48:18.220 DEBUG:boto:Method: GET 2023-07-28T23:48:18.220 DEBUG:boto:Path: /hiiqos-12/?prefix=dummy 2023-07-28T23:48:18.221 DEBUG:boto:Data: 2023-07-28T23:48:18.221 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.221 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:18.222 DEBUG:boto:Port: 8001 2023-07-28T23:48:18.222 DEBUG:boto:Params: {} 2023-07-28T23:48:18.223 DEBUG:boto:Token: None 2023-07-28T23:48:18.223 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/ 2023-07-28T23:48:18.223 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gm7hDmp7MwboUuP1lVfpucLTddA= 2023-07-28T23:48:18.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gm7hDmp7MwboUuP1lVfpucLTddA=', 'Content-Length': '0'} 2023-07-28T23:48:18.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056918ed002291784-0064c453c2-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.227 DEBUG:boto:b'hiiqos-12dummy1000falsedummy2023-07-28T23:47:16.239Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:18.227 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:48:18.228 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:48:18.228 DEBUG:boto:path=/hiiqos-12/?prefix=dummy 2023-07-28T23:48:18.228 DEBUG:boto:auth_path=/hiiqos-12/?prefix=dummy 2023-07-28T23:48:18.229 DEBUG:boto:Method: GET 2023-07-28T23:48:18.229 DEBUG:boto:Path: /hiiqos-12/?prefix=dummy 2023-07-28T23:48:18.229 DEBUG:boto:Data: 2023-07-28T23:48:18.230 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.230 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:18.230 DEBUG:boto:Port: 8000 2023-07-28T23:48:18.231 DEBUG:boto:Params: {} 2023-07-28T23:48:18.231 DEBUG:boto:Token: None 2023-07-28T23:48:18.232 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/ 2023-07-28T23:48:18.232 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gm7hDmp7MwboUuP1lVfpucLTddA= 2023-07-28T23:48:18.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gm7hDmp7MwboUuP1lVfpucLTddA=', 'Content-Length': '0'} 2023-07-28T23:48:18.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006291e301a5571048-0064c453c2-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.235 DEBUG:boto:b'hiiqos-12dummy1000falsedummy2023-07-28T23:47:16.239Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:18.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-07-28T23:48:18.236 DEBUG:boto:path=/hiiqos-12/obj-a1 2023-07-28T23:48:18.237 DEBUG:boto:auth_path=/hiiqos-12/obj-a1 2023-07-28T23:48:18.237 DEBUG:boto:path=/hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:18.237 DEBUG:boto:auth_path=/hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:18.238 DEBUG:boto:Method: GET 2023-07-28T23:48:18.238 DEBUG:boto:Path: /hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:18.239 DEBUG:boto:Data: 2023-07-28T23:48:18.239 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.239 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:18.240 DEBUG:boto:Port: 8001 2023-07-28T23:48:18.240 DEBUG:boto:Params: {} 2023-07-28T23:48:18.240 DEBUG:boto:Token: None 2023-07-28T23:48:18.241 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:18.241 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hZhcbbAlRPiYgdk6kgrmvAzmj3E= 2023-07-28T23:48:18.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hZhcbbAlRPiYgdk6kgrmvAzmj3E=', 'Content-Length': '0'} 2023-07-28T23:48:18.243 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'Zt8NMtOe11K3S282QQxMGXP84LEhPzt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d23f649ab6d18c4e-0064c453c2-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.244 DEBUG:boto:path=/hiiqos-12/obj-a1 2023-07-28T23:48:18.245 DEBUG:boto:auth_path=/hiiqos-12/obj-a1 2023-07-28T23:48:18.245 DEBUG:boto:path=/hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:18.245 DEBUG:boto:auth_path=/hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:18.246 DEBUG:boto:Method: GET 2023-07-28T23:48:18.246 DEBUG:boto:Path: /hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:18.246 DEBUG:boto:Data: 2023-07-28T23:48:18.247 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.247 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:18.247 DEBUG:boto:Port: 8000 2023-07-28T23:48:18.248 DEBUG:boto:Params: {} 2023-07-28T23:48:18.248 DEBUG:boto:Token: None 2023-07-28T23:48:18.249 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:18.249 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hZhcbbAlRPiYgdk6kgrmvAzmj3E= 2023-07-28T23:48:18.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hZhcbbAlRPiYgdk6kgrmvAzmj3E=', 'Content-Length': '0'} 2023-07-28T23:48:18.251 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'Zt8NMtOe11K3S282QQxMGXP84LEhPzt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e30671456bd6c7f2-0064c453c2-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.252 DEBUG:boto:path=/hiiqos-12/obj-a1 2023-07-28T23:48:18.252 DEBUG:boto:auth_path=/hiiqos-12/obj-a1 2023-07-28T23:48:18.253 DEBUG:boto:path=/hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:18.253 DEBUG:boto:auth_path=/hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:18.253 DEBUG:boto:Method: HEAD 2023-07-28T23:48:18.254 DEBUG:boto:Path: /hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:18.254 DEBUG:boto:Data: 2023-07-28T23:48:18.254 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.255 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:18.255 DEBUG:boto:Port: 8001 2023-07-28T23:48:18.255 DEBUG:boto:Params: {} 2023-07-28T23:48:18.256 DEBUG:boto:Token: None 2023-07-28T23:48:18.256 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:18.257 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Vmo4ofCFomquciCPbYoDTbyPYUI= 2023-07-28T23:48:18.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Vmo4ofCFomquciCPbYoDTbyPYUI=', 'Content-Length': '0'} 2023-07-28T23:48:18.259 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'Zt8NMtOe11K3S282QQxMGXP84LEhPzt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fb12cb0db15aafb8-0064c453c2-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.259 DEBUG:boto:path=/hiiqos-12/obj-a1 2023-07-28T23:48:18.260 DEBUG:boto:auth_path=/hiiqos-12/obj-a1 2023-07-28T23:48:18.260 DEBUG:boto:path=/hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:18.260 DEBUG:boto:auth_path=/hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:18.261 DEBUG:boto:Method: HEAD 2023-07-28T23:48:18.261 DEBUG:boto:Path: /hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:18.261 DEBUG:boto:Data: 2023-07-28T23:48:18.262 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.262 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:18.262 DEBUG:boto:Port: 8000 2023-07-28T23:48:18.263 DEBUG:boto:Params: {} 2023-07-28T23:48:18.263 DEBUG:boto:Token: None 2023-07-28T23:48:18.263 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:18.264 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Vmo4ofCFomquciCPbYoDTbyPYUI= 2023-07-28T23:48:18.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Vmo4ofCFomquciCPbYoDTbyPYUI=', 'Content-Length': '0'} 2023-07-28T23:48:18.266 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'Zt8NMtOe11K3S282QQxMGXP84LEhPzt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c3d7e3a415e812c7-0064c453c2-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-07-28T23:48:18.267 DEBUG:boto:path=/hiiqos-12/obj-a1 2023-07-28T23:48:18.267 DEBUG:boto:auth_path=/hiiqos-12/obj-a1 2023-07-28T23:48:18.268 DEBUG:boto:path=/hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:18.268 DEBUG:boto:auth_path=/hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:18.269 DEBUG:boto:Method: GET 2023-07-28T23:48:18.269 DEBUG:boto:Path: /hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:18.269 DEBUG:boto:Data: 2023-07-28T23:48:18.270 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.270 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:18.270 DEBUG:boto:Port: 8001 2023-07-28T23:48:18.271 DEBUG:boto:Params: {} 2023-07-28T23:48:18.271 DEBUG:boto:Token: None 2023-07-28T23:48:18.271 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:18.272 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hZhcbbAlRPiYgdk6kgrmvAzmj3E= 2023-07-28T23:48:18.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hZhcbbAlRPiYgdk6kgrmvAzmj3E=', 'Content-Length': '0'} 2023-07-28T23:48:18.274 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'Zt8NMtOe11K3S282QQxMGXP84LEhPzt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000c9dd5f66a1d4b58-0064c453c2-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.275 DEBUG:boto:path=/hiiqos-12/obj-a1 2023-07-28T23:48:18.275 DEBUG:boto:auth_path=/hiiqos-12/obj-a1 2023-07-28T23:48:18.275 DEBUG:boto:path=/hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:18.276 DEBUG:boto:auth_path=/hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:18.276 DEBUG:boto:Method: GET 2023-07-28T23:48:18.277 DEBUG:boto:Path: /hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:18.277 DEBUG:boto:Data: 2023-07-28T23:48:18.277 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.278 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:18.278 DEBUG:boto:Port: 8000 2023-07-28T23:48:18.278 DEBUG:boto:Params: {} 2023-07-28T23:48:18.279 DEBUG:boto:Token: None 2023-07-28T23:48:18.279 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:18.280 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hZhcbbAlRPiYgdk6kgrmvAzmj3E= 2023-07-28T23:48:18.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hZhcbbAlRPiYgdk6kgrmvAzmj3E=', 'Content-Length': '0'} 2023-07-28T23:48:18.281 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'Zt8NMtOe11K3S282QQxMGXP84LEhPzt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000050cfcb28b107ccb1-0064c453c2-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.282 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:48:18.283 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:48:18.283 DEBUG:boto:path=/hiiqos-12/?prefix=obj-a1 2023-07-28T23:48:18.283 DEBUG:boto:auth_path=/hiiqos-12/?prefix=obj-a1 2023-07-28T23:48:18.284 DEBUG:boto:Method: GET 2023-07-28T23:48:18.284 DEBUG:boto:Path: /hiiqos-12/?prefix=obj-a1 2023-07-28T23:48:18.284 DEBUG:boto:Data: 2023-07-28T23:48:18.285 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.285 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:18.285 DEBUG:boto:Port: 8001 2023-07-28T23:48:18.286 DEBUG:boto:Params: {} 2023-07-28T23:48:18.286 DEBUG:boto:Token: None 2023-07-28T23:48:18.287 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/ 2023-07-28T23:48:18.287 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gm7hDmp7MwboUuP1lVfpucLTddA= 2023-07-28T23:48:18.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gm7hDmp7MwboUuP1lVfpucLTddA=', 'Content-Length': '0'} 2023-07-28T23:48:18.289 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a48205a81680644-0064c453c2-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.290 DEBUG:boto:b'hiiqos-12obj-a11000falseobj-a12023-07-28T23:47:36.626Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:18.291 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:48:18.291 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:48:18.291 DEBUG:boto:path=/hiiqos-12/?prefix=obj-a1 2023-07-28T23:48:18.292 DEBUG:boto:auth_path=/hiiqos-12/?prefix=obj-a1 2023-07-28T23:48:18.292 DEBUG:boto:Method: GET 2023-07-28T23:48:18.292 DEBUG:boto:Path: /hiiqos-12/?prefix=obj-a1 2023-07-28T23:48:18.293 DEBUG:boto:Data: 2023-07-28T23:48:18.293 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.293 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:18.294 DEBUG:boto:Port: 8000 2023-07-28T23:48:18.294 DEBUG:boto:Params: {} 2023-07-28T23:48:18.295 DEBUG:boto:Token: None 2023-07-28T23:48:18.295 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/ 2023-07-28T23:48:18.296 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gm7hDmp7MwboUuP1lVfpucLTddA= 2023-07-28T23:48:18.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gm7hDmp7MwboUuP1lVfpucLTddA=', 'Content-Length': '0'} 2023-07-28T23:48:18.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060c8c32db5ea5f2b-0064c453c2-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.299 DEBUG:boto:b'hiiqos-12obj-a11000falseobj-a12023-07-28T23:47:36.626Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:18.299 DEBUG:boto:path=/hiiqos-12/obj-a1 2023-07-28T23:48:18.300 DEBUG:boto:auth_path=/hiiqos-12/obj-a1 2023-07-28T23:48:18.300 DEBUG:boto:Method: HEAD 2023-07-28T23:48:18.300 DEBUG:boto:Path: /hiiqos-12/obj-a1 2023-07-28T23:48:18.301 DEBUG:boto:Data: 2023-07-28T23:48:18.301 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.301 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:18.302 DEBUG:boto:Port: 8001 2023-07-28T23:48:18.302 DEBUG:boto:Params: {} 2023-07-28T23:48:18.302 DEBUG:boto:Token: None 2023-07-28T23:48:18.303 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/obj-a1 2023-07-28T23:48:18.303 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cqxLmO2QjhjQtQPlyqzpawaHw1M= 2023-07-28T23:48:18.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cqxLmO2QjhjQtQPlyqzpawaHw1M=', 'Content-Length': '0'} 2023-07-28T23:48:18.305 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'Zt8NMtOe11K3S282QQxMGXP84LEhPzt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007d2ac23d0fc366e9-0064c453c2-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.306 DEBUG:boto:path=/hiiqos-12/obj-a1 2023-07-28T23:48:18.307 DEBUG:boto:auth_path=/hiiqos-12/obj-a1 2023-07-28T23:48:18.307 DEBUG:boto:Method: HEAD 2023-07-28T23:48:18.307 DEBUG:boto:Path: /hiiqos-12/obj-a1 2023-07-28T23:48:18.308 DEBUG:boto:Data: 2023-07-28T23:48:18.308 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.308 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:18.309 DEBUG:boto:Port: 8000 2023-07-28T23:48:18.309 DEBUG:boto:Params: {} 2023-07-28T23:48:18.310 DEBUG:boto:Token: None 2023-07-28T23:48:18.310 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/obj-a1 2023-07-28T23:48:18.311 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cqxLmO2QjhjQtQPlyqzpawaHw1M= 2023-07-28T23:48:18.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cqxLmO2QjhjQtQPlyqzpawaHw1M=', 'Content-Length': '0'} 2023-07-28T23:48:18.313 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'Zt8NMtOe11K3S282QQxMGXP84LEhPzt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000480fb4f4de2581a-0064c453c2-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-07-28T23:48:18.314 DEBUG:boto:path=/hiiqos-12/obj-a1 2023-07-28T23:48:18.315 DEBUG:boto:auth_path=/hiiqos-12/obj-a1 2023-07-28T23:48:18.315 DEBUG:boto:path=/hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:18.316 DEBUG:boto:auth_path=/hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:18.316 DEBUG:boto:Method: GET 2023-07-28T23:48:18.316 DEBUG:boto:Path: /hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:18.317 DEBUG:boto:Data: 2023-07-28T23:48:18.317 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.318 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:18.318 DEBUG:boto:Port: 8001 2023-07-28T23:48:18.319 DEBUG:boto:Params: {} 2023-07-28T23:48:18.319 DEBUG:boto:Token: None 2023-07-28T23:48:18.320 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:18.321 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hZhcbbAlRPiYgdk6kgrmvAzmj3E= 2023-07-28T23:48:18.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hZhcbbAlRPiYgdk6kgrmvAzmj3E=', 'Content-Length': '0'} 2023-07-28T23:48:18.324 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'Zt8NMtOe11K3S282QQxMGXP84LEhPzt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007418e706cd6d1c5a-0064c453c2-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.325 DEBUG:boto:path=/hiiqos-12/obj-a1 2023-07-28T23:48:18.326 DEBUG:boto:auth_path=/hiiqos-12/obj-a1 2023-07-28T23:48:18.327 DEBUG:boto:path=/hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:18.328 DEBUG:boto:auth_path=/hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:18.329 DEBUG:boto:Method: GET 2023-07-28T23:48:18.330 DEBUG:boto:Path: /hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:18.330 DEBUG:boto:Data: 2023-07-28T23:48:18.331 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.331 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:18.331 DEBUG:boto:Port: 8000 2023-07-28T23:48:18.332 DEBUG:boto:Params: {} 2023-07-28T23:48:18.332 DEBUG:boto:Token: None 2023-07-28T23:48:18.333 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:18.333 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hZhcbbAlRPiYgdk6kgrmvAzmj3E= 2023-07-28T23:48:18.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hZhcbbAlRPiYgdk6kgrmvAzmj3E=', 'Content-Length': '0'} 2023-07-28T23:48:18.336 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'Zt8NMtOe11K3S282QQxMGXP84LEhPzt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000681c47f49101208-0064c453c2-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.337 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:48:18.338 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:48:18.339 DEBUG:boto:path=/hiiqos-12/?prefix=obj-a1 2023-07-28T23:48:18.340 DEBUG:boto:auth_path=/hiiqos-12/?prefix=obj-a1 2023-07-28T23:48:18.341 DEBUG:boto:Method: GET 2023-07-28T23:48:18.342 DEBUG:boto:Path: /hiiqos-12/?prefix=obj-a1 2023-07-28T23:48:18.343 DEBUG:boto:Data: 2023-07-28T23:48:18.343 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.344 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:18.344 DEBUG:boto:Port: 8001 2023-07-28T23:48:18.345 DEBUG:boto:Params: {} 2023-07-28T23:48:18.346 DEBUG:boto:Token: None 2023-07-28T23:48:18.346 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/ 2023-07-28T23:48:18.347 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gm7hDmp7MwboUuP1lVfpucLTddA= 2023-07-28T23:48:18.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gm7hDmp7MwboUuP1lVfpucLTddA=', 'Content-Length': '0'} 2023-07-28T23:48:18.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006775382d63a33804-0064c453c2-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.350 DEBUG:boto:b'hiiqos-12obj-a11000falseobj-a12023-07-28T23:47:36.626Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:18.351 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:48:18.351 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:48:18.351 DEBUG:boto:path=/hiiqos-12/?prefix=obj-a1 2023-07-28T23:48:18.352 DEBUG:boto:auth_path=/hiiqos-12/?prefix=obj-a1 2023-07-28T23:48:18.352 DEBUG:boto:Method: GET 2023-07-28T23:48:18.353 DEBUG:boto:Path: /hiiqos-12/?prefix=obj-a1 2023-07-28T23:48:18.353 DEBUG:boto:Data: 2023-07-28T23:48:18.353 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.354 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:18.354 DEBUG:boto:Port: 8000 2023-07-28T23:48:18.354 DEBUG:boto:Params: {} 2023-07-28T23:48:18.355 DEBUG:boto:Token: None 2023-07-28T23:48:18.355 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/ 2023-07-28T23:48:18.356 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gm7hDmp7MwboUuP1lVfpucLTddA= 2023-07-28T23:48:18.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gm7hDmp7MwboUuP1lVfpucLTddA=', 'Content-Length': '0'} 2023-07-28T23:48:18.360 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c232e0122b6152e3-0064c453c2-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.361 DEBUG:boto:b'hiiqos-12obj-a11000falseobj-a12023-07-28T23:47:36.626Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:18.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-07-28T23:48:18.363 DEBUG:boto:path=/hiiqos-12/obj-a2 2023-07-28T23:48:18.364 DEBUG:boto:auth_path=/hiiqos-12/obj-a2 2023-07-28T23:48:18.364 DEBUG:boto:path=/hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:18.365 DEBUG:boto:auth_path=/hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:18.366 DEBUG:boto:Method: GET 2023-07-28T23:48:18.366 DEBUG:boto:Path: /hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:18.367 DEBUG:boto:Data: 2023-07-28T23:48:18.367 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.368 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:18.368 DEBUG:boto:Port: 8001 2023-07-28T23:48:18.369 DEBUG:boto:Params: {} 2023-07-28T23:48:18.369 DEBUG:boto:Token: None 2023-07-28T23:48:18.370 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:18.370 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lqTnTBLACLF7Iod2L6FbdZlbRKY= 2023-07-28T23:48:18.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lqTnTBLACLF7Iod2L6FbdZlbRKY=', 'Content-Length': '0'} 2023-07-28T23:48:18.373 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'KycZQE1frBdJnvJd6sV3PeY5cRPnOLF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f22df45585542bc3-0064c453c2-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.374 DEBUG:boto:path=/hiiqos-12/obj-a2 2023-07-28T23:48:18.374 DEBUG:boto:auth_path=/hiiqos-12/obj-a2 2023-07-28T23:48:18.375 DEBUG:boto:path=/hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:18.376 DEBUG:boto:auth_path=/hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:18.377 DEBUG:boto:Method: GET 2023-07-28T23:48:18.378 DEBUG:boto:Path: /hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:18.379 DEBUG:boto:Data: 2023-07-28T23:48:18.379 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.380 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:18.381 DEBUG:boto:Port: 8000 2023-07-28T23:48:18.382 DEBUG:boto:Params: {} 2023-07-28T23:48:18.383 DEBUG:boto:Token: None 2023-07-28T23:48:18.384 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:18.384 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lqTnTBLACLF7Iod2L6FbdZlbRKY= 2023-07-28T23:48:18.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lqTnTBLACLF7Iod2L6FbdZlbRKY=', 'Content-Length': '0'} 2023-07-28T23:48:18.388 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'KycZQE1frBdJnvJd6sV3PeY5cRPnOLF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001191423368ddd10e-0064c453c2-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.389 DEBUG:boto:path=/hiiqos-12/obj-a2 2023-07-28T23:48:18.390 DEBUG:boto:auth_path=/hiiqos-12/obj-a2 2023-07-28T23:48:18.390 DEBUG:boto:path=/hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:18.391 DEBUG:boto:auth_path=/hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:18.392 DEBUG:boto:Method: HEAD 2023-07-28T23:48:18.392 DEBUG:boto:Path: /hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:18.393 DEBUG:boto:Data: 2023-07-28T23:48:18.394 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.394 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:18.395 DEBUG:boto:Port: 8001 2023-07-28T23:48:18.396 DEBUG:boto:Params: {} 2023-07-28T23:48:18.397 DEBUG:boto:Token: None 2023-07-28T23:48:18.398 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:18.399 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fC64bhXGnaDWrQefKx/r+A9+iNs= 2023-07-28T23:48:18.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fC64bhXGnaDWrQefKx/r+A9+iNs=', 'Content-Length': '0'} 2023-07-28T23:48:18.401 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'KycZQE1frBdJnvJd6sV3PeY5cRPnOLF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000071cf671c07b8df13-0064c453c2-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.402 DEBUG:boto:path=/hiiqos-12/obj-a2 2023-07-28T23:48:18.402 DEBUG:boto:auth_path=/hiiqos-12/obj-a2 2023-07-28T23:48:18.403 DEBUG:boto:path=/hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:18.403 DEBUG:boto:auth_path=/hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:18.403 DEBUG:boto:Method: HEAD 2023-07-28T23:48:18.404 DEBUG:boto:Path: /hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:18.404 DEBUG:boto:Data: 2023-07-28T23:48:18.404 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.405 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:18.405 DEBUG:boto:Port: 8000 2023-07-28T23:48:18.405 DEBUG:boto:Params: {} 2023-07-28T23:48:18.406 DEBUG:boto:Token: None 2023-07-28T23:48:18.406 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:18.407 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fC64bhXGnaDWrQefKx/r+A9+iNs= 2023-07-28T23:48:18.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fC64bhXGnaDWrQefKx/r+A9+iNs=', 'Content-Length': '0'} 2023-07-28T23:48:18.408 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'KycZQE1frBdJnvJd6sV3PeY5cRPnOLF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000019c43d6ede29eb2a-0064c453c2-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-07-28T23:48:18.409 DEBUG:boto:path=/hiiqos-12/obj-a2 2023-07-28T23:48:18.410 DEBUG:boto:auth_path=/hiiqos-12/obj-a2 2023-07-28T23:48:18.410 DEBUG:boto:path=/hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:18.411 DEBUG:boto:auth_path=/hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:18.411 DEBUG:boto:Method: GET 2023-07-28T23:48:18.411 DEBUG:boto:Path: /hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:18.412 DEBUG:boto:Data: 2023-07-28T23:48:18.412 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.412 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:18.413 DEBUG:boto:Port: 8001 2023-07-28T23:48:18.413 DEBUG:boto:Params: {} 2023-07-28T23:48:18.413 DEBUG:boto:Token: None 2023-07-28T23:48:18.414 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:18.414 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lqTnTBLACLF7Iod2L6FbdZlbRKY= 2023-07-28T23:48:18.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lqTnTBLACLF7Iod2L6FbdZlbRKY=', 'Content-Length': '0'} 2023-07-28T23:48:18.417 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'KycZQE1frBdJnvJd6sV3PeY5cRPnOLF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007569e7f2224fa2a1-0064c453c2-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.417 DEBUG:boto:path=/hiiqos-12/obj-a2 2023-07-28T23:48:18.418 DEBUG:boto:auth_path=/hiiqos-12/obj-a2 2023-07-28T23:48:18.418 DEBUG:boto:path=/hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:18.418 DEBUG:boto:auth_path=/hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:18.419 DEBUG:boto:Method: GET 2023-07-28T23:48:18.419 DEBUG:boto:Path: /hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:18.419 DEBUG:boto:Data: 2023-07-28T23:48:18.420 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.420 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:18.420 DEBUG:boto:Port: 8000 2023-07-28T23:48:18.421 DEBUG:boto:Params: {} 2023-07-28T23:48:18.421 DEBUG:boto:Token: None 2023-07-28T23:48:18.422 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:18.422 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lqTnTBLACLF7Iod2L6FbdZlbRKY= 2023-07-28T23:48:18.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lqTnTBLACLF7Iod2L6FbdZlbRKY=', 'Content-Length': '0'} 2023-07-28T23:48:18.424 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'KycZQE1frBdJnvJd6sV3PeY5cRPnOLF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001a105728e553893c-0064c453c2-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.464 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:48:18.465 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:48:18.465 DEBUG:boto:path=/hiiqos-12/?prefix=obj-a2 2023-07-28T23:48:18.466 DEBUG:boto:auth_path=/hiiqos-12/?prefix=obj-a2 2023-07-28T23:48:18.466 DEBUG:boto:Method: GET 2023-07-28T23:48:18.467 DEBUG:boto:Path: /hiiqos-12/?prefix=obj-a2 2023-07-28T23:48:18.467 DEBUG:boto:Data: 2023-07-28T23:48:18.467 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.468 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:18.468 DEBUG:boto:Port: 8001 2023-07-28T23:48:18.468 DEBUG:boto:Params: {} 2023-07-28T23:48:18.469 DEBUG:boto:Token: None 2023-07-28T23:48:18.469 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/ 2023-07-28T23:48:18.470 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gm7hDmp7MwboUuP1lVfpucLTddA= 2023-07-28T23:48:18.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gm7hDmp7MwboUuP1lVfpucLTddA=', 'Content-Length': '0'} 2023-07-28T23:48:18.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051c1208375780af7-0064c453c2-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.473 DEBUG:boto:b'hiiqos-12obj-a21000falseobj-a22023-07-28T23:47:36.751Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:18.474 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:48:18.475 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:48:18.475 DEBUG:boto:path=/hiiqos-12/?prefix=obj-a2 2023-07-28T23:48:18.475 DEBUG:boto:auth_path=/hiiqos-12/?prefix=obj-a2 2023-07-28T23:48:18.476 DEBUG:boto:Method: GET 2023-07-28T23:48:18.476 DEBUG:boto:Path: /hiiqos-12/?prefix=obj-a2 2023-07-28T23:48:18.477 DEBUG:boto:Data: 2023-07-28T23:48:18.478 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.479 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:18.479 DEBUG:boto:Port: 8000 2023-07-28T23:48:18.480 DEBUG:boto:Params: {} 2023-07-28T23:48:18.480 DEBUG:boto:Token: None 2023-07-28T23:48:18.481 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/ 2023-07-28T23:48:18.481 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gm7hDmp7MwboUuP1lVfpucLTddA= 2023-07-28T23:48:18.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gm7hDmp7MwboUuP1lVfpucLTddA=', 'Content-Length': '0'} 2023-07-28T23:48:18.484 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ae2dee95da74fd1-0064c453c2-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.485 DEBUG:boto:b'hiiqos-12obj-a21000falseobj-a22023-07-28T23:47:36.751Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:18.485 DEBUG:boto:path=/hiiqos-12/obj-a2 2023-07-28T23:48:18.486 DEBUG:boto:auth_path=/hiiqos-12/obj-a2 2023-07-28T23:48:18.486 DEBUG:boto:Method: HEAD 2023-07-28T23:48:18.487 DEBUG:boto:Path: /hiiqos-12/obj-a2 2023-07-28T23:48:18.488 DEBUG:boto:Data: 2023-07-28T23:48:18.488 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.489 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:18.490 DEBUG:boto:Port: 8001 2023-07-28T23:48:18.491 DEBUG:boto:Params: {} 2023-07-28T23:48:18.492 DEBUG:boto:Token: None 2023-07-28T23:48:18.493 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/obj-a2 2023-07-28T23:48:18.493 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9sTEPNTSEIJhijHzD+tsqHhlJtA= 2023-07-28T23:48:18.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9sTEPNTSEIJhijHzD+tsqHhlJtA=', 'Content-Length': '0'} 2023-07-28T23:48:18.496 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'KycZQE1frBdJnvJd6sV3PeY5cRPnOLF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000925965300ca13188-0064c453c2-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.497 DEBUG:boto:path=/hiiqos-12/obj-a2 2023-07-28T23:48:18.497 DEBUG:boto:auth_path=/hiiqos-12/obj-a2 2023-07-28T23:48:18.498 DEBUG:boto:Method: HEAD 2023-07-28T23:48:18.498 DEBUG:boto:Path: /hiiqos-12/obj-a2 2023-07-28T23:48:18.499 DEBUG:boto:Data: 2023-07-28T23:48:18.499 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.499 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:18.500 DEBUG:boto:Port: 8000 2023-07-28T23:48:18.500 DEBUG:boto:Params: {} 2023-07-28T23:48:18.501 DEBUG:boto:Token: None 2023-07-28T23:48:18.501 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/obj-a2 2023-07-28T23:48:18.502 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9sTEPNTSEIJhijHzD+tsqHhlJtA= 2023-07-28T23:48:18.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9sTEPNTSEIJhijHzD+tsqHhlJtA=', 'Content-Length': '0'} 2023-07-28T23:48:18.504 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'KycZQE1frBdJnvJd6sV3PeY5cRPnOLF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bf96862ce82ea1b7-0064c453c2-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-07-28T23:48:18.505 DEBUG:boto:path=/hiiqos-12/obj-a2 2023-07-28T23:48:18.505 DEBUG:boto:auth_path=/hiiqos-12/obj-a2 2023-07-28T23:48:18.506 DEBUG:boto:path=/hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:18.506 DEBUG:boto:auth_path=/hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:18.507 DEBUG:boto:Method: GET 2023-07-28T23:48:18.507 DEBUG:boto:Path: /hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:18.507 DEBUG:boto:Data: 2023-07-28T23:48:18.508 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.508 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:18.509 DEBUG:boto:Port: 8001 2023-07-28T23:48:18.510 DEBUG:boto:Params: {} 2023-07-28T23:48:18.511 DEBUG:boto:Token: None 2023-07-28T23:48:18.512 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:18.513 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lqTnTBLACLF7Iod2L6FbdZlbRKY= 2023-07-28T23:48:18.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lqTnTBLACLF7Iod2L6FbdZlbRKY=', 'Content-Length': '0'} 2023-07-28T23:48:18.516 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'KycZQE1frBdJnvJd6sV3PeY5cRPnOLF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000923d4e0724a88491-0064c453c2-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.556 DEBUG:boto:path=/hiiqos-12/obj-a2 2023-07-28T23:48:18.557 DEBUG:boto:auth_path=/hiiqos-12/obj-a2 2023-07-28T23:48:18.557 DEBUG:boto:path=/hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:18.558 DEBUG:boto:auth_path=/hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:18.558 DEBUG:boto:Method: GET 2023-07-28T23:48:18.558 DEBUG:boto:Path: /hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:18.559 DEBUG:boto:Data: 2023-07-28T23:48:18.559 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.559 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:18.560 DEBUG:boto:Port: 8000 2023-07-28T23:48:18.560 DEBUG:boto:Params: {} 2023-07-28T23:48:18.561 DEBUG:boto:Token: None 2023-07-28T23:48:18.561 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:18.561 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lqTnTBLACLF7Iod2L6FbdZlbRKY= 2023-07-28T23:48:18.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lqTnTBLACLF7Iod2L6FbdZlbRKY=', 'Content-Length': '0'} 2023-07-28T23:48:18.564 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'KycZQE1frBdJnvJd6sV3PeY5cRPnOLF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000092451d48a5f8f208-0064c453c2-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.565 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:48:18.566 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:48:18.566 DEBUG:boto:path=/hiiqos-12/?prefix=obj-a2 2023-07-28T23:48:18.566 DEBUG:boto:auth_path=/hiiqos-12/?prefix=obj-a2 2023-07-28T23:48:18.567 DEBUG:boto:Method: GET 2023-07-28T23:48:18.567 DEBUG:boto:Path: /hiiqos-12/?prefix=obj-a2 2023-07-28T23:48:18.567 DEBUG:boto:Data: 2023-07-28T23:48:18.568 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.568 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:18.569 DEBUG:boto:Port: 8001 2023-07-28T23:48:18.569 DEBUG:boto:Params: {} 2023-07-28T23:48:18.569 DEBUG:boto:Token: None 2023-07-28T23:48:18.570 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/ 2023-07-28T23:48:18.570 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gm7hDmp7MwboUuP1lVfpucLTddA= 2023-07-28T23:48:18.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gm7hDmp7MwboUuP1lVfpucLTddA=', 'Content-Length': '0'} 2023-07-28T23:48:18.573 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe04b702570704ea-0064c453c2-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.574 DEBUG:boto:b'hiiqos-12obj-a21000falseobj-a22023-07-28T23:47:36.751Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:18.575 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:48:18.575 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:48:18.575 DEBUG:boto:path=/hiiqos-12/?prefix=obj-a2 2023-07-28T23:48:18.576 DEBUG:boto:auth_path=/hiiqos-12/?prefix=obj-a2 2023-07-28T23:48:18.576 DEBUG:boto:Method: GET 2023-07-28T23:48:18.577 DEBUG:boto:Path: /hiiqos-12/?prefix=obj-a2 2023-07-28T23:48:18.577 DEBUG:boto:Data: 2023-07-28T23:48:18.577 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.578 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:18.578 DEBUG:boto:Port: 8000 2023-07-28T23:48:18.578 DEBUG:boto:Params: {} 2023-07-28T23:48:18.579 DEBUG:boto:Token: None 2023-07-28T23:48:18.579 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/ 2023-07-28T23:48:18.580 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gm7hDmp7MwboUuP1lVfpucLTddA= 2023-07-28T23:48:18.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gm7hDmp7MwboUuP1lVfpucLTddA=', 'Content-Length': '0'} 2023-07-28T23:48:18.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc1b94defd29f399-0064c453c2-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.583 DEBUG:boto:b'hiiqos-12obj-a21000falseobj-a22023-07-28T23:47:36.751Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:18.585 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-12 zones={a2, a1} 2023-07-28T23:48:18.585 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:48:18.586 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:48:18.587 DEBUG:boto:Method: HEAD 2023-07-28T23:48:18.588 DEBUG:boto:Path: /hiiqos-11/ 2023-07-28T23:48:18.589 DEBUG:boto:Data: 2023-07-28T23:48:18.589 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.590 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:18.591 DEBUG:boto:Port: 8000 2023-07-28T23:48:18.592 DEBUG:boto:Params: {} 2023-07-28T23:48:18.593 DEBUG:boto:Token: None 2023-07-28T23:48:18.593 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-11/ 2023-07-28T23:48:18.594 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:H9QOGe2bZRJHzE+/R8RFdtMUP9o= 2023-07-28T23:48:18.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:H9QOGe2bZRJHzE+/R8RFdtMUP9o=', 'Content-Length': '0'} 2023-07-28T23:48:18.598 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e03c33321f9d1a98-0064c453c2-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.599 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:48:18.600 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:48:18.600 DEBUG:boto:path=/hiiqos-11/?versions&max-keys=999&prefix=obj-a1 2023-07-28T23:48:18.601 DEBUG:boto:auth_path=/hiiqos-11/?versions&max-keys=999&prefix=obj-a1 2023-07-28T23:48:18.601 DEBUG:boto:Method: GET 2023-07-28T23:48:18.602 DEBUG:boto:Path: /hiiqos-11/?versions&max-keys=999&prefix=obj-a1 2023-07-28T23:48:18.602 DEBUG:boto:Data: 2023-07-28T23:48:18.602 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.603 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:18.603 DEBUG:boto:Port: 8000 2023-07-28T23:48:18.603 DEBUG:boto:Params: {} 2023-07-28T23:48:18.604 DEBUG:boto:Token: None 2023-07-28T23:48:18.604 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-11/?versions 2023-07-28T23:48:18.605 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cmuhgDqQOuw094odCe7wEIThqRQ= 2023-07-28T23:48:18.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cmuhgDqQOuw094odCe7wEIThqRQ=', 'Content-Length': '0'} 2023-07-28T23:48:18.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e84c6292bc893642-0064c453c2-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.608 DEBUG:boto:b'hiiqos-11obj-a1999falseobj-a1S8uVclWm2JBYB4C047A9I83Y7ulD0.ftrue2023-07-28T23:47:36.378Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:18.608 DEBUG:boto:path=/hiiqos-11/obj-a1 2023-07-28T23:48:18.609 DEBUG:boto:auth_path=/hiiqos-11/obj-a1 2023-07-28T23:48:18.609 DEBUG:boto:path=/hiiqos-11/obj-a1?acl 2023-07-28T23:48:18.609 DEBUG:boto:auth_path=/hiiqos-11/obj-a1?acl 2023-07-28T23:48:18.610 DEBUG:boto:Method: PUT 2023-07-28T23:48:18.610 DEBUG:boto:Path: /hiiqos-11/obj-a1?acl 2023-07-28T23:48:18.611 DEBUG:boto:Data: 2023-07-28T23:48:18.611 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-07-28T23:48:18.612 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:18.612 DEBUG:boto:Port: 8000 2023-07-28T23:48:18.613 DEBUG:boto:Params: {} 2023-07-28T23:48:18.613 DEBUG:boto:Token: None 2023-07-28T23:48:18.614 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:48:18 GMT x-amz-acl:public-read /hiiqos-11/obj-a1?acl 2023-07-28T23:48:18.614 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oMx8JW4l2o+9xbQVdZPo1MbQMKY= 2023-07-28T23:48:18.614 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oMx8JW4l2o+9xbQVdZPo1MbQMKY=', 'Content-Length': '0'} 2023-07-28T23:48:18.619 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d6696051e6c33a1f-0064c453c2-1117-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.619 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:48:18.620 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:48:18.620 DEBUG:boto:Method: HEAD 2023-07-28T23:48:18.620 DEBUG:boto:Path: /hiiqos-11/ 2023-07-28T23:48:18.621 DEBUG:boto:Data: 2023-07-28T23:48:18.621 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.622 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:18.622 DEBUG:boto:Port: 8001 2023-07-28T23:48:18.623 DEBUG:boto:Params: {} 2023-07-28T23:48:18.623 DEBUG:boto:Token: None 2023-07-28T23:48:18.623 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-11/ 2023-07-28T23:48:18.624 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:H9QOGe2bZRJHzE+/R8RFdtMUP9o= 2023-07-28T23:48:18.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:H9QOGe2bZRJHzE+/R8RFdtMUP9o=', 'Content-Length': '0'} 2023-07-28T23:48:18.626 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007c9b78d713c8fcda-0064c453c2-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.627 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:48:18.628 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:48:18.629 DEBUG:boto:path=/hiiqos-11/?versions&max-keys=999&prefix=obj-a2 2023-07-28T23:48:18.629 DEBUG:boto:auth_path=/hiiqos-11/?versions&max-keys=999&prefix=obj-a2 2023-07-28T23:48:18.629 DEBUG:boto:Method: GET 2023-07-28T23:48:18.630 DEBUG:boto:Path: /hiiqos-11/?versions&max-keys=999&prefix=obj-a2 2023-07-28T23:48:18.630 DEBUG:boto:Data: 2023-07-28T23:48:18.630 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.631 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:18.631 DEBUG:boto:Port: 8001 2023-07-28T23:48:18.632 DEBUG:boto:Params: {} 2023-07-28T23:48:18.633 DEBUG:boto:Token: None 2023-07-28T23:48:18.633 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-11/?versions 2023-07-28T23:48:18.634 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cmuhgDqQOuw094odCe7wEIThqRQ= 2023-07-28T23:48:18.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cmuhgDqQOuw094odCe7wEIThqRQ=', 'Content-Length': '0'} 2023-07-28T23:48:18.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cacbb5275c98f46c-0064c453c2-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.637 DEBUG:boto:b'hiiqos-11obj-a2999falseobj-a2Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYdatrue2023-07-28T23:47:36.499Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:18.637 DEBUG:boto:path=/hiiqos-11/obj-a2 2023-07-28T23:48:18.638 DEBUG:boto:auth_path=/hiiqos-11/obj-a2 2023-07-28T23:48:18.638 DEBUG:boto:path=/hiiqos-11/obj-a2?acl 2023-07-28T23:48:18.638 DEBUG:boto:auth_path=/hiiqos-11/obj-a2?acl 2023-07-28T23:48:18.639 DEBUG:boto:Method: PUT 2023-07-28T23:48:18.639 DEBUG:boto:Path: /hiiqos-11/obj-a2?acl 2023-07-28T23:48:18.640 DEBUG:boto:Data: 2023-07-28T23:48:18.640 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-07-28T23:48:18.640 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:18.641 DEBUG:boto:Port: 8001 2023-07-28T23:48:18.641 DEBUG:boto:Params: {} 2023-07-28T23:48:18.641 DEBUG:boto:Token: None 2023-07-28T23:48:18.642 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:48:18 GMT x-amz-acl:public-read /hiiqos-11/obj-a2?acl 2023-07-28T23:48:18.642 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nSYPn3acNST8BYr8XsvomqHVeLk= 2023-07-28T23:48:18.643 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nSYPn3acNST8BYr8XsvomqHVeLk=', 'Content-Length': '0'} 2023-07-28T23:48:18.647 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ebc034460052b230-0064c453c2-1113-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.648 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:48:18.648 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:48:18.649 DEBUG:boto:Method: HEAD 2023-07-28T23:48:18.649 DEBUG:boto:Path: /hiiqos-12/ 2023-07-28T23:48:18.649 DEBUG:boto:Data: 2023-07-28T23:48:18.650 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.650 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:18.650 DEBUG:boto:Port: 8000 2023-07-28T23:48:18.651 DEBUG:boto:Params: {} 2023-07-28T23:48:18.651 DEBUG:boto:Token: None 2023-07-28T23:48:18.651 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/ 2023-07-28T23:48:18.652 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yegpm6+K/aPT0lwTiE+iqMLkzLE= 2023-07-28T23:48:18.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yegpm6+K/aPT0lwTiE+iqMLkzLE=', 'Content-Length': '0'} 2023-07-28T23:48:18.654 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000218a35216b23e43c-0064c453c2-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.655 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:48:18.655 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:48:18.656 DEBUG:boto:path=/hiiqos-12/?versions&max-keys=999&prefix=obj-a1 2023-07-28T23:48:18.656 DEBUG:boto:auth_path=/hiiqos-12/?versions&max-keys=999&prefix=obj-a1 2023-07-28T23:48:18.657 DEBUG:boto:Method: GET 2023-07-28T23:48:18.657 DEBUG:boto:Path: /hiiqos-12/?versions&max-keys=999&prefix=obj-a1 2023-07-28T23:48:18.657 DEBUG:boto:Data: 2023-07-28T23:48:18.657 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.658 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:18.658 DEBUG:boto:Port: 8000 2023-07-28T23:48:18.658 DEBUG:boto:Params: {} 2023-07-28T23:48:18.659 DEBUG:boto:Token: None 2023-07-28T23:48:18.659 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/?versions 2023-07-28T23:48:18.660 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BpIJY8qv2FSSz1wFIk+xLj5odpI= 2023-07-28T23:48:18.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BpIJY8qv2FSSz1wFIk+xLj5odpI=', 'Content-Length': '0'} 2023-07-28T23:48:18.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f91e8834a2aefbb-0064c453c2-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.663 DEBUG:boto:b'hiiqos-12obj-a1999falseobj-a1Zt8NMtOe11K3S282QQxMGXP84LEhPzttrue2023-07-28T23:47:36.626Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:18.664 DEBUG:boto:path=/hiiqos-12/obj-a1 2023-07-28T23:48:18.664 DEBUG:boto:auth_path=/hiiqos-12/obj-a1 2023-07-28T23:48:18.665 DEBUG:boto:path=/hiiqos-12/obj-a1?acl 2023-07-28T23:48:18.665 DEBUG:boto:auth_path=/hiiqos-12/obj-a1?acl 2023-07-28T23:48:18.665 DEBUG:boto:Method: PUT 2023-07-28T23:48:18.666 DEBUG:boto:Path: /hiiqos-12/obj-a1?acl 2023-07-28T23:48:18.666 DEBUG:boto:Data: 2023-07-28T23:48:18.666 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-07-28T23:48:18.667 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:18.667 DEBUG:boto:Port: 8000 2023-07-28T23:48:18.667 DEBUG:boto:Params: {} 2023-07-28T23:48:18.668 DEBUG:boto:Token: None 2023-07-28T23:48:18.668 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:48:18 GMT x-amz-acl:public-read /hiiqos-12/obj-a1?acl 2023-07-28T23:48:18.669 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dK1xxCllakg8CV321vE8nF8g/vc= 2023-07-28T23:48:18.669 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dK1xxCllakg8CV321vE8nF8g/vc=', 'Content-Length': '0'} 2023-07-28T23:48:18.673 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002d82436dd02979c6-0064c453c2-1117-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.674 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:48:18.674 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:48:18.675 DEBUG:boto:Method: HEAD 2023-07-28T23:48:18.675 DEBUG:boto:Path: /hiiqos-12/ 2023-07-28T23:48:18.675 DEBUG:boto:Data: 2023-07-28T23:48:18.676 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.676 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:18.676 DEBUG:boto:Port: 8001 2023-07-28T23:48:18.677 DEBUG:boto:Params: {} 2023-07-28T23:48:18.677 DEBUG:boto:Token: None 2023-07-28T23:48:18.678 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/ 2023-07-28T23:48:18.678 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yegpm6+K/aPT0lwTiE+iqMLkzLE= 2023-07-28T23:48:18.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yegpm6+K/aPT0lwTiE+iqMLkzLE=', 'Content-Length': '0'} 2023-07-28T23:48:18.681 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ed21d1eeda5b669-0064c453c2-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.681 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:48:18.681 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:48:18.682 DEBUG:boto:path=/hiiqos-12/?versions&max-keys=999&prefix=obj-a2 2023-07-28T23:48:18.682 DEBUG:boto:auth_path=/hiiqos-12/?versions&max-keys=999&prefix=obj-a2 2023-07-28T23:48:18.682 DEBUG:boto:Method: GET 2023-07-28T23:48:18.683 DEBUG:boto:Path: /hiiqos-12/?versions&max-keys=999&prefix=obj-a2 2023-07-28T23:48:18.683 DEBUG:boto:Data: 2023-07-28T23:48:18.684 DEBUG:boto:Headers: {} 2023-07-28T23:48:18.685 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:18.686 DEBUG:boto:Port: 8001 2023-07-28T23:48:18.686 DEBUG:boto:Params: {} 2023-07-28T23:48:18.687 DEBUG:boto:Token: None 2023-07-28T23:48:18.688 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:18 GMT /hiiqos-12/?versions 2023-07-28T23:48:18.689 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BpIJY8qv2FSSz1wFIk+xLj5odpI= 2023-07-28T23:48:18.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BpIJY8qv2FSSz1wFIk+xLj5odpI=', 'Content-Length': '0'} 2023-07-28T23:48:18.692 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c42a0819364e7bed-0064c453c2-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.736 DEBUG:boto:b'hiiqos-12obj-a2999falseobj-a2KycZQE1frBdJnvJd6sV3PeY5cRPnOLFtrue2023-07-28T23:47:36.751Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:18.738 DEBUG:boto:path=/hiiqos-12/obj-a2 2023-07-28T23:48:18.738 DEBUG:boto:auth_path=/hiiqos-12/obj-a2 2023-07-28T23:48:18.739 DEBUG:boto:path=/hiiqos-12/obj-a2?acl 2023-07-28T23:48:18.740 DEBUG:boto:auth_path=/hiiqos-12/obj-a2?acl 2023-07-28T23:48:18.741 DEBUG:boto:Method: PUT 2023-07-28T23:48:18.741 DEBUG:boto:Path: /hiiqos-12/obj-a2?acl 2023-07-28T23:48:18.742 DEBUG:boto:Data: 2023-07-28T23:48:18.742 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-07-28T23:48:18.742 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:18.743 DEBUG:boto:Port: 8001 2023-07-28T23:48:18.744 DEBUG:boto:Params: {} 2023-07-28T23:48:18.744 DEBUG:boto:Token: None 2023-07-28T23:48:18.745 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:48:18 GMT x-amz-acl:public-read /hiiqos-12/obj-a2?acl 2023-07-28T23:48:18.746 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nsudR8ziJE6BupbM6uWOdfj5nw0= 2023-07-28T23:48:18.746 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nsudR8ziJE6BupbM6uWOdfj5nw0=', 'Content-Length': '0'} 2023-07-28T23:48:18.751 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001ba2dfe0d5a53809-0064c453c2-1113-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:48:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:18.752 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-11 2023-07-28T23:48:18.752 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:48:18.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:48:18.753 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-11 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-28T23:48:18.807 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:48:18.807 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:48:18.840 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:48:18.839+0000 7ffa0234b880 1 waiting for incremental sync to catch up: 2023-07-28T23:48:18.840 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000003.19.10, , , 00000000002.7.11, 00000000010.23.6, ] 2023-07-28T23:48:18.840 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000003.19.10, , , 00000000002.7.11, 00000000011.26.6, ] 2023-07-28T23:48:23.841 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:48:23.840+0000 7ffa0234b880 1 waiting for incremental sync to catch up: 2023-07-28T23:48:23.842 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000003.19.10, , , 00000000002.7.11, 00000000010.23.6, ] 2023-07-28T23:48:23.842 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000003.19.10, , , 00000000002.7.11, 00000000011.26.6, ] 2023-07-28T23:48:28.843 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:48:28.841+0000 7ffa0234b880 1 waiting for incremental sync to catch up: 2023-07-28T23:48:28.843 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000003.19.10, , , 00000000002.7.11, 00000000010.23.6, ] 2023-07-28T23:48:28.843 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000003.19.10, , , 00000000002.7.11, 00000000011.26.6, ] 2023-07-28T23:48:33.844 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:48:33.843+0000 7ffa0234b880 1 waiting for incremental sync to catch up: 2023-07-28T23:48:33.844 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000003.19.10, , , 00000000002.7.11, 00000000010.23.6, ] 2023-07-28T23:48:33.845 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000003.19.10, , , 00000000002.7.11, 00000000011.26.6, ] 2023-07-28T23:48:38.845 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:48:38.844+0000 7ffa0234b880 1 waiting for incremental sync to catch up: 2023-07-28T23:48:38.846 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000003.19.10, , , 00000000002.7.11, 00000000010.23.6, ] 2023-07-28T23:48:38.846 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000003.19.10, , , 00000000002.7.11, 00000000011.26.6, ] 2023-07-28T23:48:43.847 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:48:43.846+0000 7ffa0234b880 1 waiting for incremental sync to catch up: 2023-07-28T23:48:43.847 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000003.19.10, , , 00000000002.7.11, 00000000010.23.6, ] 2023-07-28T23:48:43.848 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000003.19.10, , , 00000000002.7.11, 00000000011.26.6, ] 2023-07-28T23:48:48.848 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:48:48.847+0000 7ffa0234b880 1 waiting for incremental sync to catch up: 2023-07-28T23:48:48.848 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000003.19.10, , , 00000000002.7.11, 00000000010.23.6, ] 2023-07-28T23:48:48.849 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000003.19.10, , , 00000000002.7.11, 00000000011.26.6, ] 2023-07-28T23:48:53.849 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:48:53.848+0000 7ffa0234b880 1 bucket sync caught up with source: 2023-07-28T23:48:53.850 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000003.19.10, , , 00000000002.7.11, 00000000011.26.6, ] 2023-07-28T23:48:53.850 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000003.19.10, , , 00000000002.7.11, 00000000011.26.6, ] 2023-07-28T23:48:53.850 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:48:53.848+0000 7ffa0234b880 0 bucket checkpoint complete 2023-07-28T23:48:53.867 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-11 2023-07-28T23:48:53.868 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:48:53.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:48:53.870 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-11 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-28T23:48:53.929 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:48:53.929 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:48:53.965 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:48:53.964+0000 7f9a43321880 1 bucket sync caught up with source: 2023-07-28T23:48:53.965 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000002.31.11, , , 00000000011.33.6, 00000000002.13.11, ] 2023-07-28T23:48:53.965 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000002.31.11, , , 00000000011.33.6, 00000000002.13.11, ] 2023-07-28T23:48:53.966 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:48:53.964+0000 7f9a43321880 0 bucket checkpoint complete 2023-07-28T23:48:53.977 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-11 zones={a2, a1} 2023-07-28T23:48:53.978 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:48:53.978 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:48:53.979 DEBUG:boto:Method: HEAD 2023-07-28T23:48:53.979 DEBUG:boto:Path: /hiiqos-11/ 2023-07-28T23:48:53.980 DEBUG:boto:Data: 2023-07-28T23:48:53.980 DEBUG:boto:Headers: {} 2023-07-28T23:48:53.980 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:53.981 DEBUG:boto:Port: 8001 2023-07-28T23:48:53.981 DEBUG:boto:Params: {} 2023-07-28T23:48:53.982 DEBUG:boto:Token: None 2023-07-28T23:48:53.982 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:53 GMT /hiiqos-11/ 2023-07-28T23:48:53.983 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KPqSm8tyiHIY6YiI52mJ0keuz58= 2023-07-28T23:48:53.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KPqSm8tyiHIY6YiI52mJ0keuz58=', 'Content-Length': '0'} 2023-07-28T23:48:53.985 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000014c7b26432035584-0064c453e5-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:48:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:53.986 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:48:53.986 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:48:53.987 DEBUG:boto:Method: HEAD 2023-07-28T23:48:53.987 DEBUG:boto:Path: /hiiqos-11/ 2023-07-28T23:48:53.987 DEBUG:boto:Data: 2023-07-28T23:48:53.988 DEBUG:boto:Headers: {} 2023-07-28T23:48:53.988 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:53.988 DEBUG:boto:Port: 8000 2023-07-28T23:48:53.989 DEBUG:boto:Params: {} 2023-07-28T23:48:53.989 DEBUG:boto:Token: None 2023-07-28T23:48:53.990 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:53 GMT /hiiqos-11/ 2023-07-28T23:48:53.990 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KPqSm8tyiHIY6YiI52mJ0keuz58= 2023-07-28T23:48:53.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KPqSm8tyiHIY6YiI52mJ0keuz58=', 'Content-Length': '0'} 2023-07-28T23:48:53.993 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b8c1f4faa896838-0064c453e5-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:48:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:53.993 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-28T23:48:53.994 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:48:53.994 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:48:53.994 DEBUG:boto:path=/hiiqos-11/?versions&max-keys=999 2023-07-28T23:48:53.995 DEBUG:boto:auth_path=/hiiqos-11/?versions&max-keys=999 2023-07-28T23:48:53.995 DEBUG:boto:Method: GET 2023-07-28T23:48:53.995 DEBUG:boto:Path: /hiiqos-11/?versions&max-keys=999 2023-07-28T23:48:53.996 DEBUG:boto:Data: 2023-07-28T23:48:53.996 DEBUG:boto:Headers: {} 2023-07-28T23:48:53.997 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:53.997 DEBUG:boto:Port: 8001 2023-07-28T23:48:53.997 DEBUG:boto:Params: {} 2023-07-28T23:48:53.998 DEBUG:boto:Token: None 2023-07-28T23:48:53.998 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:53 GMT /hiiqos-11/?versions 2023-07-28T23:48:53.999 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3AceaxpRbGbsWuI492TvCarmS1A= 2023-07-28T23:48:53.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3AceaxpRbGbsWuI492TvCarmS1A=', 'Content-Length': '0'} 2023-07-28T23:48:54.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4056c9850273a05-0064c453e5-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.002 DEBUG:boto:b'hiiqos-11999falsedummyRUBZW9FJzfoEbFpjsAMCNtq8zLwltmNtrue2023-07-28T23:47:01.096Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1S8uVclWm2JBYB4C047A9I83Y7ulD0.ftrue2023-07-28T23:47:36.378Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYdatrue2023-07-28T23:47:36.499Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:54.003 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-07-28T23:48:54.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-07-28T23:48:54.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-07-28T23:48:54.004 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-28T23:48:54.004 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:48:54.005 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:48:54.005 DEBUG:boto:path=/hiiqos-11/?versions&max-keys=999 2023-07-28T23:48:54.005 DEBUG:boto:auth_path=/hiiqos-11/?versions&max-keys=999 2023-07-28T23:48:54.006 DEBUG:boto:Method: GET 2023-07-28T23:48:54.006 DEBUG:boto:Path: /hiiqos-11/?versions&max-keys=999 2023-07-28T23:48:54.006 DEBUG:boto:Data: 2023-07-28T23:48:54.007 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.007 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:54.008 DEBUG:boto:Port: 8000 2023-07-28T23:48:54.008 DEBUG:boto:Params: {} 2023-07-28T23:48:54.008 DEBUG:boto:Token: None 2023-07-28T23:48:54.009 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/?versions 2023-07-28T23:48:54.009 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gfH+UOeoUVBUKKGGtNcPO/Wurqk= 2023-07-28T23:48:54.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gfH+UOeoUVBUKKGGtNcPO/Wurqk=', 'Content-Length': '0'} 2023-07-28T23:48:54.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000855ccb978903b4a1-0064c453e6-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.012 DEBUG:boto:b'hiiqos-11999falsedummyRUBZW9FJzfoEbFpjsAMCNtq8zLwltmNtrue2023-07-28T23:47:01.096Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1S8uVclWm2JBYB4C047A9I83Y7ulD0.ftrue2023-07-28T23:47:36.378Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYdatrue2023-07-28T23:47:36.499Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:54.013 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-07-28T23:48:54.013 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-07-28T23:48:54.014 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-07-28T23:48:54.015 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:48:54.015 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:48:54.016 DEBUG:boto:path=/hiiqos-11/?versions&max-keys=999 2023-07-28T23:48:54.017 DEBUG:boto:auth_path=/hiiqos-11/?versions&max-keys=999 2023-07-28T23:48:54.017 DEBUG:boto:Method: GET 2023-07-28T23:48:54.017 DEBUG:boto:Path: /hiiqos-11/?versions&max-keys=999 2023-07-28T23:48:54.018 DEBUG:boto:Data: 2023-07-28T23:48:54.018 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.019 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:54.019 DEBUG:boto:Port: 8001 2023-07-28T23:48:54.020 DEBUG:boto:Params: {} 2023-07-28T23:48:54.020 DEBUG:boto:Token: None 2023-07-28T23:48:54.021 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/?versions 2023-07-28T23:48:54.021 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gfH+UOeoUVBUKKGGtNcPO/Wurqk= 2023-07-28T23:48:54.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gfH+UOeoUVBUKKGGtNcPO/Wurqk=', 'Content-Length': '0'} 2023-07-28T23:48:54.024 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf6635edc1278460-0064c453e6-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.025 DEBUG:boto:b'hiiqos-11999falsedummyRUBZW9FJzfoEbFpjsAMCNtq8zLwltmNtrue2023-07-28T23:47:01.096Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1S8uVclWm2JBYB4C047A9I83Y7ulD0.ftrue2023-07-28T23:47:36.378Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYdatrue2023-07-28T23:47:36.499Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:54.026 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:48:54.027 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:48:54.028 DEBUG:boto:path=/hiiqos-11/?versions&max-keys=999 2023-07-28T23:48:54.028 DEBUG:boto:auth_path=/hiiqos-11/?versions&max-keys=999 2023-07-28T23:48:54.029 DEBUG:boto:Method: GET 2023-07-28T23:48:54.029 DEBUG:boto:Path: /hiiqos-11/?versions&max-keys=999 2023-07-28T23:48:54.030 DEBUG:boto:Data: 2023-07-28T23:48:54.030 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.030 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:54.031 DEBUG:boto:Port: 8000 2023-07-28T23:48:54.031 DEBUG:boto:Params: {} 2023-07-28T23:48:54.032 DEBUG:boto:Token: None 2023-07-28T23:48:54.032 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/?versions 2023-07-28T23:48:54.032 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gfH+UOeoUVBUKKGGtNcPO/Wurqk= 2023-07-28T23:48:54.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gfH+UOeoUVBUKKGGtNcPO/Wurqk=', 'Content-Length': '0'} 2023-07-28T23:48:54.035 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c40d22ef64c85de-0064c453e6-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.036 DEBUG:boto:b'hiiqos-11999falsedummyRUBZW9FJzfoEbFpjsAMCNtq8zLwltmNtrue2023-07-28T23:47:01.096Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1S8uVclWm2JBYB4C047A9I83Y7ulD0.ftrue2023-07-28T23:47:36.378Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYdatrue2023-07-28T23:47:36.499Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:54.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-07-28T23:48:54.037 DEBUG:boto:path=/hiiqos-11/dummy 2023-07-28T23:48:54.038 DEBUG:boto:auth_path=/hiiqos-11/dummy 2023-07-28T23:48:54.038 DEBUG:boto:path=/hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:54.038 DEBUG:boto:auth_path=/hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:54.039 DEBUG:boto:Method: GET 2023-07-28T23:48:54.039 DEBUG:boto:Path: /hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:54.039 DEBUG:boto:Data: 2023-07-28T23:48:54.040 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.041 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:54.041 DEBUG:boto:Port: 8001 2023-07-28T23:48:54.041 DEBUG:boto:Params: {} 2023-07-28T23:48:54.042 DEBUG:boto:Token: None 2023-07-28T23:48:54.042 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:54.043 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gLOsbIMiEmxhyctoAt7jVl0ATo8= 2023-07-28T23:48:54.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gLOsbIMiEmxhyctoAt7jVl0ATo8=', 'Content-Length': '0'} 2023-07-28T23:48:54.045 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:01 GMT'), ('x-amz-version-id', 'RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000069d24d2fb44b5304-0064c453e6-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.046 DEBUG:boto:path=/hiiqos-11/dummy 2023-07-28T23:48:54.047 DEBUG:boto:auth_path=/hiiqos-11/dummy 2023-07-28T23:48:54.047 DEBUG:boto:path=/hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:54.048 DEBUG:boto:auth_path=/hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:54.049 DEBUG:boto:Method: GET 2023-07-28T23:48:54.049 DEBUG:boto:Path: /hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:54.050 DEBUG:boto:Data: 2023-07-28T23:48:54.050 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.050 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:54.051 DEBUG:boto:Port: 8000 2023-07-28T23:48:54.051 DEBUG:boto:Params: {} 2023-07-28T23:48:54.052 DEBUG:boto:Token: None 2023-07-28T23:48:54.052 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:54.053 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gLOsbIMiEmxhyctoAt7jVl0ATo8= 2023-07-28T23:48:54.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gLOsbIMiEmxhyctoAt7jVl0ATo8=', 'Content-Length': '0'} 2023-07-28T23:48:54.055 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:01 GMT'), ('x-amz-version-id', 'RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009080f6794369cb63-0064c453e6-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.056 DEBUG:boto:path=/hiiqos-11/dummy 2023-07-28T23:48:54.056 DEBUG:boto:auth_path=/hiiqos-11/dummy 2023-07-28T23:48:54.057 DEBUG:boto:path=/hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:54.057 DEBUG:boto:auth_path=/hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:54.057 DEBUG:boto:Method: HEAD 2023-07-28T23:48:54.058 DEBUG:boto:Path: /hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:54.058 DEBUG:boto:Data: 2023-07-28T23:48:54.059 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.059 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:54.059 DEBUG:boto:Port: 8001 2023-07-28T23:48:54.060 DEBUG:boto:Params: {} 2023-07-28T23:48:54.061 DEBUG:boto:Token: None 2023-07-28T23:48:54.062 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:54.062 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/nl5lIuul2DRAO5SF8t7jSe28mI= 2023-07-28T23:48:54.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/nl5lIuul2DRAO5SF8t7jSe28mI=', 'Content-Length': '0'} 2023-07-28T23:48:54.065 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:01 GMT'), ('x-amz-version-id', 'RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000eb0111197a12c10f-0064c453e6-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.067 DEBUG:boto:path=/hiiqos-11/dummy 2023-07-28T23:48:54.068 DEBUG:boto:auth_path=/hiiqos-11/dummy 2023-07-28T23:48:54.069 DEBUG:boto:path=/hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:54.069 DEBUG:boto:auth_path=/hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:54.071 DEBUG:boto:Method: HEAD 2023-07-28T23:48:54.072 DEBUG:boto:Path: /hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:54.073 DEBUG:boto:Data: 2023-07-28T23:48:54.074 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.075 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:54.076 DEBUG:boto:Port: 8000 2023-07-28T23:48:54.077 DEBUG:boto:Params: {} 2023-07-28T23:48:54.078 DEBUG:boto:Token: None 2023-07-28T23:48:54.079 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:54.080 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/nl5lIuul2DRAO5SF8t7jSe28mI= 2023-07-28T23:48:54.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/nl5lIuul2DRAO5SF8t7jSe28mI=', 'Content-Length': '0'} 2023-07-28T23:48:54.082 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:01 GMT'), ('x-amz-version-id', 'RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000420958655ef08686-0064c453e6-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-07-28T23:48:54.083 DEBUG:boto:path=/hiiqos-11/dummy 2023-07-28T23:48:54.084 DEBUG:boto:auth_path=/hiiqos-11/dummy 2023-07-28T23:48:54.084 DEBUG:boto:path=/hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:54.085 DEBUG:boto:auth_path=/hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:54.085 DEBUG:boto:Method: GET 2023-07-28T23:48:54.085 DEBUG:boto:Path: /hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:54.086 DEBUG:boto:Data: 2023-07-28T23:48:54.086 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.086 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:54.087 DEBUG:boto:Port: 8001 2023-07-28T23:48:54.087 DEBUG:boto:Params: {} 2023-07-28T23:48:54.088 DEBUG:boto:Token: None 2023-07-28T23:48:54.088 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:54.088 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gLOsbIMiEmxhyctoAt7jVl0ATo8= 2023-07-28T23:48:54.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gLOsbIMiEmxhyctoAt7jVl0ATo8=', 'Content-Length': '0'} 2023-07-28T23:48:54.091 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:01 GMT'), ('x-amz-version-id', 'RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b09db11e3f3a90ab-0064c453e6-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.091 DEBUG:boto:path=/hiiqos-11/dummy 2023-07-28T23:48:54.092 DEBUG:boto:auth_path=/hiiqos-11/dummy 2023-07-28T23:48:54.092 DEBUG:boto:path=/hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:54.093 DEBUG:boto:auth_path=/hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:54.093 DEBUG:boto:Method: GET 2023-07-28T23:48:54.094 DEBUG:boto:Path: /hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:54.094 DEBUG:boto:Data: 2023-07-28T23:48:54.094 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.095 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:54.095 DEBUG:boto:Port: 8000 2023-07-28T23:48:54.095 DEBUG:boto:Params: {} 2023-07-28T23:48:54.096 DEBUG:boto:Token: None 2023-07-28T23:48:54.096 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:54.097 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gLOsbIMiEmxhyctoAt7jVl0ATo8= 2023-07-28T23:48:54.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gLOsbIMiEmxhyctoAt7jVl0ATo8=', 'Content-Length': '0'} 2023-07-28T23:48:54.099 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:01 GMT'), ('x-amz-version-id', 'RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000aa6ca8045d659acd-0064c453e6-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.100 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:48:54.100 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:48:54.100 DEBUG:boto:path=/hiiqos-11/?prefix=dummy 2023-07-28T23:48:54.101 DEBUG:boto:auth_path=/hiiqos-11/?prefix=dummy 2023-07-28T23:48:54.101 DEBUG:boto:Method: GET 2023-07-28T23:48:54.102 DEBUG:boto:Path: /hiiqos-11/?prefix=dummy 2023-07-28T23:48:54.102 DEBUG:boto:Data: 2023-07-28T23:48:54.103 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.103 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:54.104 DEBUG:boto:Port: 8001 2023-07-28T23:48:54.104 DEBUG:boto:Params: {} 2023-07-28T23:48:54.104 DEBUG:boto:Token: None 2023-07-28T23:48:54.105 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/ 2023-07-28T23:48:54.105 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vDlXlXJ4zV0QTLCUmc6I9HzmuHE= 2023-07-28T23:48:54.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vDlXlXJ4zV0QTLCUmc6I9HzmuHE=', 'Content-Length': '0'} 2023-07-28T23:48:54.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026fc0948bd9b73bd-0064c453e6-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.152 DEBUG:boto:b'hiiqos-11dummy1000falsedummy2023-07-28T23:47:01.096Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:54.153 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:48:54.154 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:48:54.154 DEBUG:boto:path=/hiiqos-11/?prefix=dummy 2023-07-28T23:48:54.155 DEBUG:boto:auth_path=/hiiqos-11/?prefix=dummy 2023-07-28T23:48:54.155 DEBUG:boto:Method: GET 2023-07-28T23:48:54.155 DEBUG:boto:Path: /hiiqos-11/?prefix=dummy 2023-07-28T23:48:54.156 DEBUG:boto:Data: 2023-07-28T23:48:54.156 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.156 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:54.157 DEBUG:boto:Port: 8000 2023-07-28T23:48:54.157 DEBUG:boto:Params: {} 2023-07-28T23:48:54.158 DEBUG:boto:Token: None 2023-07-28T23:48:54.158 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/ 2023-07-28T23:48:54.158 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vDlXlXJ4zV0QTLCUmc6I9HzmuHE= 2023-07-28T23:48:54.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vDlXlXJ4zV0QTLCUmc6I9HzmuHE=', 'Content-Length': '0'} 2023-07-28T23:48:54.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec3a96f34431bfaf-0064c453e6-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.162 DEBUG:boto:b'hiiqos-11dummy1000falsedummy2023-07-28T23:47:01.096Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:54.163 DEBUG:boto:path=/hiiqos-11/dummy 2023-07-28T23:48:54.163 DEBUG:boto:auth_path=/hiiqos-11/dummy 2023-07-28T23:48:54.163 DEBUG:boto:Method: HEAD 2023-07-28T23:48:54.164 DEBUG:boto:Path: /hiiqos-11/dummy 2023-07-28T23:48:54.164 DEBUG:boto:Data: 2023-07-28T23:48:54.164 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.165 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:54.165 DEBUG:boto:Port: 8001 2023-07-28T23:48:54.165 DEBUG:boto:Params: {} 2023-07-28T23:48:54.166 DEBUG:boto:Token: None 2023-07-28T23:48:54.166 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/dummy 2023-07-28T23:48:54.167 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:h2hybX1bklOe8UiTET7DpT/YPkA= 2023-07-28T23:48:54.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:h2hybX1bklOe8UiTET7DpT/YPkA=', 'Content-Length': '0'} 2023-07-28T23:48:54.169 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:01 GMT'), ('x-amz-version-id', 'RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000eb1fe6a64a43614b-0064c453e6-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.169 DEBUG:boto:path=/hiiqos-11/dummy 2023-07-28T23:48:54.170 DEBUG:boto:auth_path=/hiiqos-11/dummy 2023-07-28T23:48:54.170 DEBUG:boto:Method: HEAD 2023-07-28T23:48:54.171 DEBUG:boto:Path: /hiiqos-11/dummy 2023-07-28T23:48:54.171 DEBUG:boto:Data: 2023-07-28T23:48:54.171 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.172 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:54.172 DEBUG:boto:Port: 8000 2023-07-28T23:48:54.172 DEBUG:boto:Params: {} 2023-07-28T23:48:54.173 DEBUG:boto:Token: None 2023-07-28T23:48:54.173 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/dummy 2023-07-28T23:48:54.173 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:h2hybX1bklOe8UiTET7DpT/YPkA= 2023-07-28T23:48:54.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:h2hybX1bklOe8UiTET7DpT/YPkA=', 'Content-Length': '0'} 2023-07-28T23:48:54.175 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:01 GMT'), ('x-amz-version-id', 'RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000752091b485fb3c0f-0064c453e6-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-07-28T23:48:54.176 DEBUG:boto:path=/hiiqos-11/dummy 2023-07-28T23:48:54.177 DEBUG:boto:auth_path=/hiiqos-11/dummy 2023-07-28T23:48:54.177 DEBUG:boto:path=/hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:54.178 DEBUG:boto:auth_path=/hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:54.178 DEBUG:boto:Method: GET 2023-07-28T23:48:54.178 DEBUG:boto:Path: /hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:54.179 DEBUG:boto:Data: 2023-07-28T23:48:54.179 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.179 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:54.180 DEBUG:boto:Port: 8001 2023-07-28T23:48:54.180 DEBUG:boto:Params: {} 2023-07-28T23:48:54.180 DEBUG:boto:Token: None 2023-07-28T23:48:54.181 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:54.181 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gLOsbIMiEmxhyctoAt7jVl0ATo8= 2023-07-28T23:48:54.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gLOsbIMiEmxhyctoAt7jVl0ATo8=', 'Content-Length': '0'} 2023-07-28T23:48:54.183 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:01 GMT'), ('x-amz-version-id', 'RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000eea9c4c2dcc6704f-0064c453e6-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.184 DEBUG:boto:path=/hiiqos-11/dummy 2023-07-28T23:48:54.184 DEBUG:boto:auth_path=/hiiqos-11/dummy 2023-07-28T23:48:54.185 DEBUG:boto:path=/hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:54.185 DEBUG:boto:auth_path=/hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:54.185 DEBUG:boto:Method: GET 2023-07-28T23:48:54.186 DEBUG:boto:Path: /hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:54.186 DEBUG:boto:Data: 2023-07-28T23:48:54.186 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.187 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:54.187 DEBUG:boto:Port: 8000 2023-07-28T23:48:54.187 DEBUG:boto:Params: {} 2023-07-28T23:48:54.188 DEBUG:boto:Token: None 2023-07-28T23:48:54.188 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/dummy?versionId=RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN 2023-07-28T23:48:54.189 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gLOsbIMiEmxhyctoAt7jVl0ATo8= 2023-07-28T23:48:54.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gLOsbIMiEmxhyctoAt7jVl0ATo8=', 'Content-Length': '0'} 2023-07-28T23:48:54.191 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:01 GMT'), ('x-amz-version-id', 'RUBZW9FJzfoEbFpjsAMCNtq8zLwltmN'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004ad3f20d962233c0-0064c453e6-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.191 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:48:54.192 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:48:54.192 DEBUG:boto:path=/hiiqos-11/?prefix=dummy 2023-07-28T23:48:54.192 DEBUG:boto:auth_path=/hiiqos-11/?prefix=dummy 2023-07-28T23:48:54.193 DEBUG:boto:Method: GET 2023-07-28T23:48:54.193 DEBUG:boto:Path: /hiiqos-11/?prefix=dummy 2023-07-28T23:48:54.193 DEBUG:boto:Data: 2023-07-28T23:48:54.194 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.194 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:54.194 DEBUG:boto:Port: 8001 2023-07-28T23:48:54.195 DEBUG:boto:Params: {} 2023-07-28T23:48:54.195 DEBUG:boto:Token: None 2023-07-28T23:48:54.196 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/ 2023-07-28T23:48:54.196 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vDlXlXJ4zV0QTLCUmc6I9HzmuHE= 2023-07-28T23:48:54.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vDlXlXJ4zV0QTLCUmc6I9HzmuHE=', 'Content-Length': '0'} 2023-07-28T23:48:54.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b54ab869a3fa1224-0064c453e6-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.199 DEBUG:boto:b'hiiqos-11dummy1000falsedummy2023-07-28T23:47:01.096Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:54.200 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:48:54.200 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:48:54.201 DEBUG:boto:path=/hiiqos-11/?prefix=dummy 2023-07-28T23:48:54.201 DEBUG:boto:auth_path=/hiiqos-11/?prefix=dummy 2023-07-28T23:48:54.201 DEBUG:boto:Method: GET 2023-07-28T23:48:54.202 DEBUG:boto:Path: /hiiqos-11/?prefix=dummy 2023-07-28T23:48:54.202 DEBUG:boto:Data: 2023-07-28T23:48:54.202 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.203 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:54.203 DEBUG:boto:Port: 8000 2023-07-28T23:48:54.203 DEBUG:boto:Params: {} 2023-07-28T23:48:54.204 DEBUG:boto:Token: None 2023-07-28T23:48:54.204 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/ 2023-07-28T23:48:54.205 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vDlXlXJ4zV0QTLCUmc6I9HzmuHE= 2023-07-28T23:48:54.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vDlXlXJ4zV0QTLCUmc6I9HzmuHE=', 'Content-Length': '0'} 2023-07-28T23:48:54.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030ae66f8511ba290-0064c453e6-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.208 DEBUG:boto:b'hiiqos-11dummy1000falsedummy2023-07-28T23:47:01.096Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:54.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-07-28T23:48:54.209 DEBUG:boto:path=/hiiqos-11/obj-a1 2023-07-28T23:48:54.210 DEBUG:boto:auth_path=/hiiqos-11/obj-a1 2023-07-28T23:48:54.210 DEBUG:boto:path=/hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:54.210 DEBUG:boto:auth_path=/hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:54.211 DEBUG:boto:Method: GET 2023-07-28T23:48:54.211 DEBUG:boto:Path: /hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:54.212 DEBUG:boto:Data: 2023-07-28T23:48:54.212 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.212 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:54.213 DEBUG:boto:Port: 8001 2023-07-28T23:48:54.213 DEBUG:boto:Params: {} 2023-07-28T23:48:54.213 DEBUG:boto:Token: None 2023-07-28T23:48:54.214 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:54.214 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:D7IO3l/hEnnLTbWhxe2ZXgDjlLw= 2023-07-28T23:48:54.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:D7IO3l/hEnnLTbWhxe2ZXgDjlLw=', 'Content-Length': '0'} 2023-07-28T23:48:54.216 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'S8uVclWm2JBYB4C047A9I83Y7ulD0.f'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ae339b94438a74a7-0064c453e6-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.217 DEBUG:boto:path=/hiiqos-11/obj-a1 2023-07-28T23:48:54.217 DEBUG:boto:auth_path=/hiiqos-11/obj-a1 2023-07-28T23:48:54.218 DEBUG:boto:path=/hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:54.218 DEBUG:boto:auth_path=/hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:54.218 DEBUG:boto:Method: GET 2023-07-28T23:48:54.219 DEBUG:boto:Path: /hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:54.219 DEBUG:boto:Data: 2023-07-28T23:48:54.219 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.220 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:54.220 DEBUG:boto:Port: 8000 2023-07-28T23:48:54.220 DEBUG:boto:Params: {} 2023-07-28T23:48:54.221 DEBUG:boto:Token: None 2023-07-28T23:48:54.221 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:54.222 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:D7IO3l/hEnnLTbWhxe2ZXgDjlLw= 2023-07-28T23:48:54.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:D7IO3l/hEnnLTbWhxe2ZXgDjlLw=', 'Content-Length': '0'} 2023-07-28T23:48:54.224 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'S8uVclWm2JBYB4C047A9I83Y7ulD0.f'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000088f9f8958278b3c2-0064c453e6-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.224 DEBUG:boto:path=/hiiqos-11/obj-a1 2023-07-28T23:48:54.225 DEBUG:boto:auth_path=/hiiqos-11/obj-a1 2023-07-28T23:48:54.225 DEBUG:boto:path=/hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:54.225 DEBUG:boto:auth_path=/hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:54.226 DEBUG:boto:Method: HEAD 2023-07-28T23:48:54.226 DEBUG:boto:Path: /hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:54.226 DEBUG:boto:Data: 2023-07-28T23:48:54.227 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.227 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:54.227 DEBUG:boto:Port: 8001 2023-07-28T23:48:54.228 DEBUG:boto:Params: {} 2023-07-28T23:48:54.228 DEBUG:boto:Token: None 2023-07-28T23:48:54.228 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:54.229 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZCNXiDe2O3QENx4ZPusw4Smx89E= 2023-07-28T23:48:54.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZCNXiDe2O3QENx4ZPusw4Smx89E=', 'Content-Length': '0'} 2023-07-28T23:48:54.231 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'S8uVclWm2JBYB4C047A9I83Y7ulD0.f'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000efc8608cd957d7d9-0064c453e6-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.231 DEBUG:boto:path=/hiiqos-11/obj-a1 2023-07-28T23:48:54.232 DEBUG:boto:auth_path=/hiiqos-11/obj-a1 2023-07-28T23:48:54.232 DEBUG:boto:path=/hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:54.233 DEBUG:boto:auth_path=/hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:54.233 DEBUG:boto:Method: HEAD 2023-07-28T23:48:54.233 DEBUG:boto:Path: /hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:54.234 DEBUG:boto:Data: 2023-07-28T23:48:54.234 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.234 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:54.234 DEBUG:boto:Port: 8000 2023-07-28T23:48:54.235 DEBUG:boto:Params: {} 2023-07-28T23:48:54.235 DEBUG:boto:Token: None 2023-07-28T23:48:54.236 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:54.236 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZCNXiDe2O3QENx4ZPusw4Smx89E= 2023-07-28T23:48:54.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZCNXiDe2O3QENx4ZPusw4Smx89E=', 'Content-Length': '0'} 2023-07-28T23:48:54.238 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'S8uVclWm2JBYB4C047A9I83Y7ulD0.f'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008686d883e3c8360a-0064c453e6-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-07-28T23:48:54.239 DEBUG:boto:path=/hiiqos-11/obj-a1 2023-07-28T23:48:54.239 DEBUG:boto:auth_path=/hiiqos-11/obj-a1 2023-07-28T23:48:54.240 DEBUG:boto:path=/hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:54.240 DEBUG:boto:auth_path=/hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:54.240 DEBUG:boto:Method: GET 2023-07-28T23:48:54.241 DEBUG:boto:Path: /hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:54.241 DEBUG:boto:Data: 2023-07-28T23:48:54.242 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.242 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:54.242 DEBUG:boto:Port: 8001 2023-07-28T23:48:54.243 DEBUG:boto:Params: {} 2023-07-28T23:48:54.243 DEBUG:boto:Token: None 2023-07-28T23:48:54.243 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:54.244 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:D7IO3l/hEnnLTbWhxe2ZXgDjlLw= 2023-07-28T23:48:54.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:D7IO3l/hEnnLTbWhxe2ZXgDjlLw=', 'Content-Length': '0'} 2023-07-28T23:48:54.246 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'S8uVclWm2JBYB4C047A9I83Y7ulD0.f'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004d1530641fe47e91-0064c453e6-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.292 DEBUG:boto:path=/hiiqos-11/obj-a1 2023-07-28T23:48:54.293 DEBUG:boto:auth_path=/hiiqos-11/obj-a1 2023-07-28T23:48:54.293 DEBUG:boto:path=/hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:54.294 DEBUG:boto:auth_path=/hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:54.294 DEBUG:boto:Method: GET 2023-07-28T23:48:54.295 DEBUG:boto:Path: /hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:54.295 DEBUG:boto:Data: 2023-07-28T23:48:54.295 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.296 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:54.296 DEBUG:boto:Port: 8000 2023-07-28T23:48:54.296 DEBUG:boto:Params: {} 2023-07-28T23:48:54.297 DEBUG:boto:Token: None 2023-07-28T23:48:54.297 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:54.298 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:D7IO3l/hEnnLTbWhxe2ZXgDjlLw= 2023-07-28T23:48:54.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:D7IO3l/hEnnLTbWhxe2ZXgDjlLw=', 'Content-Length': '0'} 2023-07-28T23:48:54.300 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'S8uVclWm2JBYB4C047A9I83Y7ulD0.f'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c528c0ed5884663a-0064c453e6-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.301 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:48:54.301 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:48:54.302 DEBUG:boto:path=/hiiqos-11/?prefix=obj-a1 2023-07-28T23:48:54.302 DEBUG:boto:auth_path=/hiiqos-11/?prefix=obj-a1 2023-07-28T23:48:54.302 DEBUG:boto:Method: GET 2023-07-28T23:48:54.303 DEBUG:boto:Path: /hiiqos-11/?prefix=obj-a1 2023-07-28T23:48:54.303 DEBUG:boto:Data: 2023-07-28T23:48:54.303 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.304 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:54.304 DEBUG:boto:Port: 8001 2023-07-28T23:48:54.305 DEBUG:boto:Params: {} 2023-07-28T23:48:54.305 DEBUG:boto:Token: None 2023-07-28T23:48:54.305 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/ 2023-07-28T23:48:54.306 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vDlXlXJ4zV0QTLCUmc6I9HzmuHE= 2023-07-28T23:48:54.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vDlXlXJ4zV0QTLCUmc6I9HzmuHE=', 'Content-Length': '0'} 2023-07-28T23:48:54.308 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035179321691b65f0-0064c453e6-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.309 DEBUG:boto:b'hiiqos-11obj-a11000falseobj-a12023-07-28T23:47:36.378Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:54.310 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:48:54.310 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:48:54.310 DEBUG:boto:path=/hiiqos-11/?prefix=obj-a1 2023-07-28T23:48:54.311 DEBUG:boto:auth_path=/hiiqos-11/?prefix=obj-a1 2023-07-28T23:48:54.311 DEBUG:boto:Method: GET 2023-07-28T23:48:54.312 DEBUG:boto:Path: /hiiqos-11/?prefix=obj-a1 2023-07-28T23:48:54.312 DEBUG:boto:Data: 2023-07-28T23:48:54.312 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.313 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:54.313 DEBUG:boto:Port: 8000 2023-07-28T23:48:54.313 DEBUG:boto:Params: {} 2023-07-28T23:48:54.314 DEBUG:boto:Token: None 2023-07-28T23:48:54.314 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/ 2023-07-28T23:48:54.315 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vDlXlXJ4zV0QTLCUmc6I9HzmuHE= 2023-07-28T23:48:54.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vDlXlXJ4zV0QTLCUmc6I9HzmuHE=', 'Content-Length': '0'} 2023-07-28T23:48:54.317 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000083e73835729b6f8-0064c453e6-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.317 DEBUG:boto:b'hiiqos-11obj-a11000falseobj-a12023-07-28T23:47:36.378Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:54.318 DEBUG:boto:path=/hiiqos-11/obj-a1 2023-07-28T23:48:54.319 DEBUG:boto:auth_path=/hiiqos-11/obj-a1 2023-07-28T23:48:54.319 DEBUG:boto:Method: HEAD 2023-07-28T23:48:54.319 DEBUG:boto:Path: /hiiqos-11/obj-a1 2023-07-28T23:48:54.320 DEBUG:boto:Data: 2023-07-28T23:48:54.320 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.320 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:54.321 DEBUG:boto:Port: 8001 2023-07-28T23:48:54.321 DEBUG:boto:Params: {} 2023-07-28T23:48:54.321 DEBUG:boto:Token: None 2023-07-28T23:48:54.322 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/obj-a1 2023-07-28T23:48:54.322 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SOhp9/Vh6CK6Tg2IS55fK7VuZ1Q= 2023-07-28T23:48:54.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SOhp9/Vh6CK6Tg2IS55fK7VuZ1Q=', 'Content-Length': '0'} 2023-07-28T23:48:54.325 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'S8uVclWm2JBYB4C047A9I83Y7ulD0.f'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000027264dbaead9d482-0064c453e6-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.325 DEBUG:boto:path=/hiiqos-11/obj-a1 2023-07-28T23:48:54.326 DEBUG:boto:auth_path=/hiiqos-11/obj-a1 2023-07-28T23:48:54.326 DEBUG:boto:Method: HEAD 2023-07-28T23:48:54.326 DEBUG:boto:Path: /hiiqos-11/obj-a1 2023-07-28T23:48:54.327 DEBUG:boto:Data: 2023-07-28T23:48:54.327 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.327 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:54.328 DEBUG:boto:Port: 8000 2023-07-28T23:48:54.328 DEBUG:boto:Params: {} 2023-07-28T23:48:54.328 DEBUG:boto:Token: None 2023-07-28T23:48:54.329 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/obj-a1 2023-07-28T23:48:54.329 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SOhp9/Vh6CK6Tg2IS55fK7VuZ1Q= 2023-07-28T23:48:54.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SOhp9/Vh6CK6Tg2IS55fK7VuZ1Q=', 'Content-Length': '0'} 2023-07-28T23:48:54.332 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'S8uVclWm2JBYB4C047A9I83Y7ulD0.f'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000052efb225a1740b01-0064c453e6-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-07-28T23:48:54.333 DEBUG:boto:path=/hiiqos-11/obj-a1 2023-07-28T23:48:54.333 DEBUG:boto:auth_path=/hiiqos-11/obj-a1 2023-07-28T23:48:54.333 DEBUG:boto:path=/hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:54.334 DEBUG:boto:auth_path=/hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:54.334 DEBUG:boto:Method: GET 2023-07-28T23:48:54.335 DEBUG:boto:Path: /hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:54.335 DEBUG:boto:Data: 2023-07-28T23:48:54.335 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.336 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:54.336 DEBUG:boto:Port: 8001 2023-07-28T23:48:54.336 DEBUG:boto:Params: {} 2023-07-28T23:48:54.337 DEBUG:boto:Token: None 2023-07-28T23:48:54.337 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:54.338 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:D7IO3l/hEnnLTbWhxe2ZXgDjlLw= 2023-07-28T23:48:54.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:D7IO3l/hEnnLTbWhxe2ZXgDjlLw=', 'Content-Length': '0'} 2023-07-28T23:48:54.340 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'S8uVclWm2JBYB4C047A9I83Y7ulD0.f'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003dfc115567bc8617-0064c453e6-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.384 DEBUG:boto:path=/hiiqos-11/obj-a1 2023-07-28T23:48:54.385 DEBUG:boto:auth_path=/hiiqos-11/obj-a1 2023-07-28T23:48:54.385 DEBUG:boto:path=/hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:54.386 DEBUG:boto:auth_path=/hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:54.386 DEBUG:boto:Method: GET 2023-07-28T23:48:54.387 DEBUG:boto:Path: /hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:54.387 DEBUG:boto:Data: 2023-07-28T23:48:54.387 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.388 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:54.388 DEBUG:boto:Port: 8000 2023-07-28T23:48:54.388 DEBUG:boto:Params: {} 2023-07-28T23:48:54.389 DEBUG:boto:Token: None 2023-07-28T23:48:54.389 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/obj-a1?versionId=S8uVclWm2JBYB4C047A9I83Y7ulD0.f 2023-07-28T23:48:54.390 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:D7IO3l/hEnnLTbWhxe2ZXgDjlLw= 2023-07-28T23:48:54.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:D7IO3l/hEnnLTbWhxe2ZXgDjlLw=', 'Content-Length': '0'} 2023-07-28T23:48:54.392 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'S8uVclWm2JBYB4C047A9I83Y7ulD0.f'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000058cdb9377140523a-0064c453e6-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.393 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:48:54.393 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:48:54.394 DEBUG:boto:path=/hiiqos-11/?prefix=obj-a1 2023-07-28T23:48:54.394 DEBUG:boto:auth_path=/hiiqos-11/?prefix=obj-a1 2023-07-28T23:48:54.394 DEBUG:boto:Method: GET 2023-07-28T23:48:54.395 DEBUG:boto:Path: /hiiqos-11/?prefix=obj-a1 2023-07-28T23:48:54.395 DEBUG:boto:Data: 2023-07-28T23:48:54.395 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.396 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:54.396 DEBUG:boto:Port: 8001 2023-07-28T23:48:54.396 DEBUG:boto:Params: {} 2023-07-28T23:48:54.397 DEBUG:boto:Token: None 2023-07-28T23:48:54.397 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/ 2023-07-28T23:48:54.398 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vDlXlXJ4zV0QTLCUmc6I9HzmuHE= 2023-07-28T23:48:54.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vDlXlXJ4zV0QTLCUmc6I9HzmuHE=', 'Content-Length': '0'} 2023-07-28T23:48:54.400 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d4e43dca55a2501-0064c453e6-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.401 DEBUG:boto:b'hiiqos-11obj-a11000falseobj-a12023-07-28T23:47:36.378Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:54.402 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:48:54.402 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:48:54.402 DEBUG:boto:path=/hiiqos-11/?prefix=obj-a1 2023-07-28T23:48:54.403 DEBUG:boto:auth_path=/hiiqos-11/?prefix=obj-a1 2023-07-28T23:48:54.403 DEBUG:boto:Method: GET 2023-07-28T23:48:54.403 DEBUG:boto:Path: /hiiqos-11/?prefix=obj-a1 2023-07-28T23:48:54.404 DEBUG:boto:Data: 2023-07-28T23:48:54.404 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.404 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:54.405 DEBUG:boto:Port: 8000 2023-07-28T23:48:54.405 DEBUG:boto:Params: {} 2023-07-28T23:48:54.406 DEBUG:boto:Token: None 2023-07-28T23:48:54.406 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/ 2023-07-28T23:48:54.406 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vDlXlXJ4zV0QTLCUmc6I9HzmuHE= 2023-07-28T23:48:54.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vDlXlXJ4zV0QTLCUmc6I9HzmuHE=', 'Content-Length': '0'} 2023-07-28T23:48:54.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081f03e2bff44cd8b-0064c453e6-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.410 DEBUG:boto:b'hiiqos-11obj-a11000falseobj-a12023-07-28T23:47:36.378Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:54.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-07-28T23:48:54.411 DEBUG:boto:path=/hiiqos-11/obj-a2 2023-07-28T23:48:54.411 DEBUG:boto:auth_path=/hiiqos-11/obj-a2 2023-07-28T23:48:54.411 DEBUG:boto:path=/hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:54.412 DEBUG:boto:auth_path=/hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:54.412 DEBUG:boto:Method: GET 2023-07-28T23:48:54.413 DEBUG:boto:Path: /hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:54.413 DEBUG:boto:Data: 2023-07-28T23:48:54.413 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.414 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:54.414 DEBUG:boto:Port: 8001 2023-07-28T23:48:54.414 DEBUG:boto:Params: {} 2023-07-28T23:48:54.415 DEBUG:boto:Token: None 2023-07-28T23:48:54.415 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:54.416 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QtHqG2FHGdyCS/8wyg6f/GjT7l8= 2023-07-28T23:48:54.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QtHqG2FHGdyCS/8wyg6f/GjT7l8=', 'Content-Length': '0'} 2023-07-28T23:48:54.418 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000022c33360efb7e258-0064c453e6-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.419 DEBUG:boto:path=/hiiqos-11/obj-a2 2023-07-28T23:48:54.419 DEBUG:boto:auth_path=/hiiqos-11/obj-a2 2023-07-28T23:48:54.420 DEBUG:boto:path=/hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:54.420 DEBUG:boto:auth_path=/hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:54.420 DEBUG:boto:Method: GET 2023-07-28T23:48:54.421 DEBUG:boto:Path: /hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:54.421 DEBUG:boto:Data: 2023-07-28T23:48:54.421 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.422 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:54.422 DEBUG:boto:Port: 8000 2023-07-28T23:48:54.422 DEBUG:boto:Params: {} 2023-07-28T23:48:54.423 DEBUG:boto:Token: None 2023-07-28T23:48:54.423 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:54.424 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QtHqG2FHGdyCS/8wyg6f/GjT7l8= 2023-07-28T23:48:54.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QtHqG2FHGdyCS/8wyg6f/GjT7l8=', 'Content-Length': '0'} 2023-07-28T23:48:54.426 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001c344177649b719f-0064c453e6-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.426 DEBUG:boto:path=/hiiqos-11/obj-a2 2023-07-28T23:48:54.427 DEBUG:boto:auth_path=/hiiqos-11/obj-a2 2023-07-28T23:48:54.427 DEBUG:boto:path=/hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:54.427 DEBUG:boto:auth_path=/hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:54.428 DEBUG:boto:Method: HEAD 2023-07-28T23:48:54.428 DEBUG:boto:Path: /hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:54.428 DEBUG:boto:Data: 2023-07-28T23:48:54.429 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.429 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:54.429 DEBUG:boto:Port: 8001 2023-07-28T23:48:54.430 DEBUG:boto:Params: {} 2023-07-28T23:48:54.430 DEBUG:boto:Token: None 2023-07-28T23:48:54.430 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:54.431 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QMlBmralVeN0P2GwzJmPSkIG32w= 2023-07-28T23:48:54.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QMlBmralVeN0P2GwzJmPSkIG32w=', 'Content-Length': '0'} 2023-07-28T23:48:54.433 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f523bf7e2756a201-0064c453e6-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.434 DEBUG:boto:path=/hiiqos-11/obj-a2 2023-07-28T23:48:54.434 DEBUG:boto:auth_path=/hiiqos-11/obj-a2 2023-07-28T23:48:54.434 DEBUG:boto:path=/hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:54.435 DEBUG:boto:auth_path=/hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:54.435 DEBUG:boto:Method: HEAD 2023-07-28T23:48:54.435 DEBUG:boto:Path: /hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:54.436 DEBUG:boto:Data: 2023-07-28T23:48:54.436 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.436 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:54.437 DEBUG:boto:Port: 8000 2023-07-28T23:48:54.437 DEBUG:boto:Params: {} 2023-07-28T23:48:54.437 DEBUG:boto:Token: None 2023-07-28T23:48:54.438 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:54.438 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QMlBmralVeN0P2GwzJmPSkIG32w= 2023-07-28T23:48:54.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QMlBmralVeN0P2GwzJmPSkIG32w=', 'Content-Length': '0'} 2023-07-28T23:48:54.440 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000969405e645d10e20-0064c453e6-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-07-28T23:48:54.441 DEBUG:boto:path=/hiiqos-11/obj-a2 2023-07-28T23:48:54.441 DEBUG:boto:auth_path=/hiiqos-11/obj-a2 2023-07-28T23:48:54.442 DEBUG:boto:path=/hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:54.442 DEBUG:boto:auth_path=/hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:54.442 DEBUG:boto:Method: GET 2023-07-28T23:48:54.443 DEBUG:boto:Path: /hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:54.443 DEBUG:boto:Data: 2023-07-28T23:48:54.443 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.444 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:54.444 DEBUG:boto:Port: 8001 2023-07-28T23:48:54.444 DEBUG:boto:Params: {} 2023-07-28T23:48:54.445 DEBUG:boto:Token: None 2023-07-28T23:48:54.445 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:54.446 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QtHqG2FHGdyCS/8wyg6f/GjT7l8= 2023-07-28T23:48:54.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QtHqG2FHGdyCS/8wyg6f/GjT7l8=', 'Content-Length': '0'} 2023-07-28T23:48:54.447 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dc4119fdfb21efd5-0064c453e6-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.488 DEBUG:boto:path=/hiiqos-11/obj-a2 2023-07-28T23:48:54.489 DEBUG:boto:auth_path=/hiiqos-11/obj-a2 2023-07-28T23:48:54.489 DEBUG:boto:path=/hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:54.490 DEBUG:boto:auth_path=/hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:54.490 DEBUG:boto:Method: GET 2023-07-28T23:48:54.490 DEBUG:boto:Path: /hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:54.491 DEBUG:boto:Data: 2023-07-28T23:48:54.491 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.491 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:54.492 DEBUG:boto:Port: 8000 2023-07-28T23:48:54.492 DEBUG:boto:Params: {} 2023-07-28T23:48:54.493 DEBUG:boto:Token: None 2023-07-28T23:48:54.493 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:54.493 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QtHqG2FHGdyCS/8wyg6f/GjT7l8= 2023-07-28T23:48:54.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QtHqG2FHGdyCS/8wyg6f/GjT7l8=', 'Content-Length': '0'} 2023-07-28T23:48:54.496 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009a9e931ad71fc0f2-0064c453e6-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.496 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:48:54.497 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:48:54.497 DEBUG:boto:path=/hiiqos-11/?prefix=obj-a2 2023-07-28T23:48:54.497 DEBUG:boto:auth_path=/hiiqos-11/?prefix=obj-a2 2023-07-28T23:48:54.498 DEBUG:boto:Method: GET 2023-07-28T23:48:54.498 DEBUG:boto:Path: /hiiqos-11/?prefix=obj-a2 2023-07-28T23:48:54.499 DEBUG:boto:Data: 2023-07-28T23:48:54.499 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.500 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:54.500 DEBUG:boto:Port: 8001 2023-07-28T23:48:54.501 DEBUG:boto:Params: {} 2023-07-28T23:48:54.501 DEBUG:boto:Token: None 2023-07-28T23:48:54.502 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/ 2023-07-28T23:48:54.502 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vDlXlXJ4zV0QTLCUmc6I9HzmuHE= 2023-07-28T23:48:54.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vDlXlXJ4zV0QTLCUmc6I9HzmuHE=', 'Content-Length': '0'} 2023-07-28T23:48:54.505 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de48101d21173585-0064c453e6-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.505 DEBUG:boto:b'hiiqos-11obj-a21000falseobj-a22023-07-28T23:47:36.499Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:54.506 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:48:54.506 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:48:54.507 DEBUG:boto:path=/hiiqos-11/?prefix=obj-a2 2023-07-28T23:48:54.507 DEBUG:boto:auth_path=/hiiqos-11/?prefix=obj-a2 2023-07-28T23:48:54.507 DEBUG:boto:Method: GET 2023-07-28T23:48:54.508 DEBUG:boto:Path: /hiiqos-11/?prefix=obj-a2 2023-07-28T23:48:54.508 DEBUG:boto:Data: 2023-07-28T23:48:54.509 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.509 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:54.510 DEBUG:boto:Port: 8000 2023-07-28T23:48:54.510 DEBUG:boto:Params: {} 2023-07-28T23:48:54.511 DEBUG:boto:Token: None 2023-07-28T23:48:54.511 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/ 2023-07-28T23:48:54.512 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vDlXlXJ4zV0QTLCUmc6I9HzmuHE= 2023-07-28T23:48:54.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vDlXlXJ4zV0QTLCUmc6I9HzmuHE=', 'Content-Length': '0'} 2023-07-28T23:48:54.516 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fff4e7b3cf5feeb6-0064c453e6-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.516 DEBUG:boto:b'hiiqos-11obj-a21000falseobj-a22023-07-28T23:47:36.499Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:54.517 DEBUG:boto:path=/hiiqos-11/obj-a2 2023-07-28T23:48:54.517 DEBUG:boto:auth_path=/hiiqos-11/obj-a2 2023-07-28T23:48:54.518 DEBUG:boto:Method: HEAD 2023-07-28T23:48:54.518 DEBUG:boto:Path: /hiiqos-11/obj-a2 2023-07-28T23:48:54.519 DEBUG:boto:Data: 2023-07-28T23:48:54.519 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.519 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:54.519 DEBUG:boto:Port: 8001 2023-07-28T23:48:54.520 DEBUG:boto:Params: {} 2023-07-28T23:48:54.520 DEBUG:boto:Token: None 2023-07-28T23:48:54.521 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/obj-a2 2023-07-28T23:48:54.521 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lNid9cPmlXmMqV4/KbWLN6HiTxA= 2023-07-28T23:48:54.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lNid9cPmlXmMqV4/KbWLN6HiTxA=', 'Content-Length': '0'} 2023-07-28T23:48:54.523 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f414ec93431fa9a3-0064c453e6-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.524 DEBUG:boto:path=/hiiqos-11/obj-a2 2023-07-28T23:48:54.524 DEBUG:boto:auth_path=/hiiqos-11/obj-a2 2023-07-28T23:48:54.525 DEBUG:boto:Method: HEAD 2023-07-28T23:48:54.526 DEBUG:boto:Path: /hiiqos-11/obj-a2 2023-07-28T23:48:54.527 DEBUG:boto:Data: 2023-07-28T23:48:54.528 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.528 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:54.529 DEBUG:boto:Port: 8000 2023-07-28T23:48:54.530 DEBUG:boto:Params: {} 2023-07-28T23:48:54.530 DEBUG:boto:Token: None 2023-07-28T23:48:54.531 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/obj-a2 2023-07-28T23:48:54.532 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lNid9cPmlXmMqV4/KbWLN6HiTxA= 2023-07-28T23:48:54.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lNid9cPmlXmMqV4/KbWLN6HiTxA=', 'Content-Length': '0'} 2023-07-28T23:48:54.535 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000161fe70a387380e1-0064c453e6-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-07-28T23:48:54.537 DEBUG:boto:path=/hiiqos-11/obj-a2 2023-07-28T23:48:54.538 DEBUG:boto:auth_path=/hiiqos-11/obj-a2 2023-07-28T23:48:54.538 DEBUG:boto:path=/hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:54.539 DEBUG:boto:auth_path=/hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:54.540 DEBUG:boto:Method: GET 2023-07-28T23:48:54.541 DEBUG:boto:Path: /hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:54.542 DEBUG:boto:Data: 2023-07-28T23:48:54.542 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.543 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:54.544 DEBUG:boto:Port: 8001 2023-07-28T23:48:54.544 DEBUG:boto:Params: {} 2023-07-28T23:48:54.545 DEBUG:boto:Token: None 2023-07-28T23:48:54.545 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:54.546 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QtHqG2FHGdyCS/8wyg6f/GjT7l8= 2023-07-28T23:48:54.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QtHqG2FHGdyCS/8wyg6f/GjT7l8=', 'Content-Length': '0'} 2023-07-28T23:48:54.548 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d0d34bfc7cf9932d-0064c453e6-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.549 DEBUG:boto:path=/hiiqos-11/obj-a2 2023-07-28T23:48:54.550 DEBUG:boto:auth_path=/hiiqos-11/obj-a2 2023-07-28T23:48:54.551 DEBUG:boto:path=/hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:54.552 DEBUG:boto:auth_path=/hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:54.553 DEBUG:boto:Method: GET 2023-07-28T23:48:54.554 DEBUG:boto:Path: /hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:54.555 DEBUG:boto:Data: 2023-07-28T23:48:54.555 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.556 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:54.557 DEBUG:boto:Port: 8000 2023-07-28T23:48:54.557 DEBUG:boto:Params: {} 2023-07-28T23:48:54.558 DEBUG:boto:Token: None 2023-07-28T23:48:54.559 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/obj-a2?versionId=Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda 2023-07-28T23:48:54.560 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QtHqG2FHGdyCS/8wyg6f/GjT7l8= 2023-07-28T23:48:54.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QtHqG2FHGdyCS/8wyg6f/GjT7l8=', 'Content-Length': '0'} 2023-07-28T23:48:54.562 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'Q7G8CMCMW4AZgdmQ.MELUR.CsrrFYda'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000006edc5aface43bfc-0064c453e6-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.563 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:48:54.563 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:48:54.564 DEBUG:boto:path=/hiiqos-11/?prefix=obj-a2 2023-07-28T23:48:54.564 DEBUG:boto:auth_path=/hiiqos-11/?prefix=obj-a2 2023-07-28T23:48:54.565 DEBUG:boto:Method: GET 2023-07-28T23:48:54.565 DEBUG:boto:Path: /hiiqos-11/?prefix=obj-a2 2023-07-28T23:48:54.565 DEBUG:boto:Data: 2023-07-28T23:48:54.566 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.566 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:54.567 DEBUG:boto:Port: 8001 2023-07-28T23:48:54.568 DEBUG:boto:Params: {} 2023-07-28T23:48:54.568 DEBUG:boto:Token: None 2023-07-28T23:48:54.569 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/ 2023-07-28T23:48:54.570 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vDlXlXJ4zV0QTLCUmc6I9HzmuHE= 2023-07-28T23:48:54.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vDlXlXJ4zV0QTLCUmc6I9HzmuHE=', 'Content-Length': '0'} 2023-07-28T23:48:54.573 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079d189656fe26d09-0064c453e6-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.574 DEBUG:boto:b'hiiqos-11obj-a21000falseobj-a22023-07-28T23:47:36.499Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:54.575 DEBUG:boto:path=/hiiqos-11/ 2023-07-28T23:48:54.575 DEBUG:boto:auth_path=/hiiqos-11/ 2023-07-28T23:48:54.576 DEBUG:boto:path=/hiiqos-11/?prefix=obj-a2 2023-07-28T23:48:54.577 DEBUG:boto:auth_path=/hiiqos-11/?prefix=obj-a2 2023-07-28T23:48:54.578 DEBUG:boto:Method: GET 2023-07-28T23:48:54.579 DEBUG:boto:Path: /hiiqos-11/?prefix=obj-a2 2023-07-28T23:48:54.579 DEBUG:boto:Data: 2023-07-28T23:48:54.581 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.581 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:54.582 DEBUG:boto:Port: 8000 2023-07-28T23:48:54.583 DEBUG:boto:Params: {} 2023-07-28T23:48:54.584 DEBUG:boto:Token: None 2023-07-28T23:48:54.585 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-11/ 2023-07-28T23:48:54.585 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vDlXlXJ4zV0QTLCUmc6I9HzmuHE= 2023-07-28T23:48:54.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vDlXlXJ4zV0QTLCUmc6I9HzmuHE=', 'Content-Length': '0'} 2023-07-28T23:48:54.588 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005829ad314fadc5d6-0064c453e6-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.588 DEBUG:boto:b'hiiqos-11obj-a21000falseobj-a22023-07-28T23:47:36.499Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:54.589 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-11 zones={a2, a1} 2023-07-28T23:48:54.589 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-12 2023-07-28T23:48:54.590 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:48:54.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:48:54.591 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-12 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-28T23:48:54.644 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:48:54.644 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:48:54.675 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:48:54.674+0000 7f89f1c22880 1 bucket sync caught up with source: 2023-07-28T23:48:54.676 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000002.17.11, , , 00000000002.6.11, 00000000011.29.6, ] 2023-07-28T23:48:54.676 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000002.17.11, , , 00000000002.6.11, 00000000011.29.6, ] 2023-07-28T23:48:54.676 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:48:54.674+0000 7f89f1c22880 0 bucket checkpoint complete 2023-07-28T23:48:54.687 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-12 2023-07-28T23:48:54.688 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:48:54.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:48:54.689 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-12 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-28T23:48:54.741 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:48:54.741 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:48:54.773 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:48:54.772+0000 7fba7752a880 1 bucket sync caught up with source: 2023-07-28T23:48:54.774 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000003.37.10, , , 00000000011.32.6, 00000000003.26.10, ] 2023-07-28T23:48:54.774 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000003.37.10, , , 00000000011.32.6, 00000000003.26.10, ] 2023-07-28T23:48:54.774 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:48:54.772+0000 7fba7752a880 0 bucket checkpoint complete 2023-07-28T23:48:54.785 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-12 zones={a2, a1} 2023-07-28T23:48:54.785 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:48:54.786 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:48:54.786 DEBUG:boto:Method: HEAD 2023-07-28T23:48:54.786 DEBUG:boto:Path: /hiiqos-12/ 2023-07-28T23:48:54.787 DEBUG:boto:Data: 2023-07-28T23:48:54.787 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.787 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:54.788 DEBUG:boto:Port: 8001 2023-07-28T23:48:54.788 DEBUG:boto:Params: {} 2023-07-28T23:48:54.789 DEBUG:boto:Token: None 2023-07-28T23:48:54.789 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-12/ 2023-07-28T23:48:54.789 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4aXwcDJuQysEu0LPutVWBtx6y5w= 2023-07-28T23:48:54.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4aXwcDJuQysEu0LPutVWBtx6y5w=', 'Content-Length': '0'} 2023-07-28T23:48:54.792 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e5f744ac01a19bbd-0064c453e6-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.792 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:48:54.793 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:48:54.793 DEBUG:boto:Method: HEAD 2023-07-28T23:48:54.794 DEBUG:boto:Path: /hiiqos-12/ 2023-07-28T23:48:54.794 DEBUG:boto:Data: 2023-07-28T23:48:54.794 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.795 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:54.795 DEBUG:boto:Port: 8000 2023-07-28T23:48:54.795 DEBUG:boto:Params: {} 2023-07-28T23:48:54.796 DEBUG:boto:Token: None 2023-07-28T23:48:54.796 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-12/ 2023-07-28T23:48:54.796 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4aXwcDJuQysEu0LPutVWBtx6y5w= 2023-07-28T23:48:54.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4aXwcDJuQysEu0LPutVWBtx6y5w=', 'Content-Length': '0'} 2023-07-28T23:48:54.799 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000045c6655db13973b1-0064c453e6-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.799 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-28T23:48:54.800 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:48:54.800 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:48:54.801 DEBUG:boto:path=/hiiqos-12/?versions&max-keys=999 2023-07-28T23:48:54.801 DEBUG:boto:auth_path=/hiiqos-12/?versions&max-keys=999 2023-07-28T23:48:54.801 DEBUG:boto:Method: GET 2023-07-28T23:48:54.802 DEBUG:boto:Path: /hiiqos-12/?versions&max-keys=999 2023-07-28T23:48:54.802 DEBUG:boto:Data: 2023-07-28T23:48:54.802 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.803 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:54.803 DEBUG:boto:Port: 8001 2023-07-28T23:48:54.803 DEBUG:boto:Params: {} 2023-07-28T23:48:54.804 DEBUG:boto:Token: None 2023-07-28T23:48:54.804 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-12/?versions 2023-07-28T23:48:54.804 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BYyTU8Vpq5EP7ulT2cs1/wVBfUE= 2023-07-28T23:48:54.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BYyTU8Vpq5EP7ulT2cs1/wVBfUE=', 'Content-Length': '0'} 2023-07-28T23:48:54.807 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053a090cf568566db-0064c453e6-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.808 DEBUG:boto:b'hiiqos-12999falsedummycVvFSQn3t1--pfP44jwpm7HeB4ciTyVtrue2023-07-28T23:47:16.239Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1Zt8NMtOe11K3S282QQxMGXP84LEhPzttrue2023-07-28T23:47:36.626Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2KycZQE1frBdJnvJd6sV3PeY5cRPnOLFtrue2023-07-28T23:47:36.751Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:54.808 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-07-28T23:48:54.809 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-07-28T23:48:54.809 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-07-28T23:48:54.809 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-28T23:48:54.810 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:48:54.810 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:48:54.810 DEBUG:boto:path=/hiiqos-12/?versions&max-keys=999 2023-07-28T23:48:54.811 DEBUG:boto:auth_path=/hiiqos-12/?versions&max-keys=999 2023-07-28T23:48:54.811 DEBUG:boto:Method: GET 2023-07-28T23:48:54.811 DEBUG:boto:Path: /hiiqos-12/?versions&max-keys=999 2023-07-28T23:48:54.812 DEBUG:boto:Data: 2023-07-28T23:48:54.812 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.812 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:54.813 DEBUG:boto:Port: 8000 2023-07-28T23:48:54.813 DEBUG:boto:Params: {} 2023-07-28T23:48:54.813 DEBUG:boto:Token: None 2023-07-28T23:48:54.814 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-12/?versions 2023-07-28T23:48:54.814 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BYyTU8Vpq5EP7ulT2cs1/wVBfUE= 2023-07-28T23:48:54.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BYyTU8Vpq5EP7ulT2cs1/wVBfUE=', 'Content-Length': '0'} 2023-07-28T23:48:54.817 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c2e54a6dab85904-0064c453e6-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.817 DEBUG:boto:b'hiiqos-12999falsedummycVvFSQn3t1--pfP44jwpm7HeB4ciTyVtrue2023-07-28T23:47:16.239Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1Zt8NMtOe11K3S282QQxMGXP84LEhPzttrue2023-07-28T23:47:36.626Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2KycZQE1frBdJnvJd6sV3PeY5cRPnOLFtrue2023-07-28T23:47:36.751Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:54.818 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-07-28T23:48:54.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-07-28T23:48:54.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-07-28T23:48:54.819 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:48:54.819 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:48:54.820 DEBUG:boto:path=/hiiqos-12/?versions&max-keys=999 2023-07-28T23:48:54.820 DEBUG:boto:auth_path=/hiiqos-12/?versions&max-keys=999 2023-07-28T23:48:54.820 DEBUG:boto:Method: GET 2023-07-28T23:48:54.821 DEBUG:boto:Path: /hiiqos-12/?versions&max-keys=999 2023-07-28T23:48:54.821 DEBUG:boto:Data: 2023-07-28T23:48:54.821 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.822 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:54.822 DEBUG:boto:Port: 8001 2023-07-28T23:48:54.822 DEBUG:boto:Params: {} 2023-07-28T23:48:54.823 DEBUG:boto:Token: None 2023-07-28T23:48:54.823 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-12/?versions 2023-07-28T23:48:54.824 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BYyTU8Vpq5EP7ulT2cs1/wVBfUE= 2023-07-28T23:48:54.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BYyTU8Vpq5EP7ulT2cs1/wVBfUE=', 'Content-Length': '0'} 2023-07-28T23:48:54.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be3023d61c43efc7-0064c453e6-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.827 DEBUG:boto:b'hiiqos-12999falsedummycVvFSQn3t1--pfP44jwpm7HeB4ciTyVtrue2023-07-28T23:47:16.239Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1Zt8NMtOe11K3S282QQxMGXP84LEhPzttrue2023-07-28T23:47:36.626Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2KycZQE1frBdJnvJd6sV3PeY5cRPnOLFtrue2023-07-28T23:47:36.751Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:54.827 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:48:54.828 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:48:54.828 DEBUG:boto:path=/hiiqos-12/?versions&max-keys=999 2023-07-28T23:48:54.828 DEBUG:boto:auth_path=/hiiqos-12/?versions&max-keys=999 2023-07-28T23:48:54.829 DEBUG:boto:Method: GET 2023-07-28T23:48:54.829 DEBUG:boto:Path: /hiiqos-12/?versions&max-keys=999 2023-07-28T23:48:54.830 DEBUG:boto:Data: 2023-07-28T23:48:54.830 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.830 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:54.830 DEBUG:boto:Port: 8000 2023-07-28T23:48:54.831 DEBUG:boto:Params: {} 2023-07-28T23:48:54.831 DEBUG:boto:Token: None 2023-07-28T23:48:54.832 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-12/?versions 2023-07-28T23:48:54.832 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BYyTU8Vpq5EP7ulT2cs1/wVBfUE= 2023-07-28T23:48:54.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BYyTU8Vpq5EP7ulT2cs1/wVBfUE=', 'Content-Length': '0'} 2023-07-28T23:48:54.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000bf781e31d05516-0064c453e6-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.835 DEBUG:boto:b'hiiqos-12999falsedummycVvFSQn3t1--pfP44jwpm7HeB4ciTyVtrue2023-07-28T23:47:16.239Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1Zt8NMtOe11K3S282QQxMGXP84LEhPzttrue2023-07-28T23:47:36.626Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2KycZQE1frBdJnvJd6sV3PeY5cRPnOLFtrue2023-07-28T23:47:36.751Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:54.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-07-28T23:48:54.836 DEBUG:boto:path=/hiiqos-12/dummy 2023-07-28T23:48:54.837 DEBUG:boto:auth_path=/hiiqos-12/dummy 2023-07-28T23:48:54.837 DEBUG:boto:path=/hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:54.837 DEBUG:boto:auth_path=/hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:54.838 DEBUG:boto:Method: GET 2023-07-28T23:48:54.838 DEBUG:boto:Path: /hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:54.838 DEBUG:boto:Data: 2023-07-28T23:48:54.839 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.839 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:54.839 DEBUG:boto:Port: 8001 2023-07-28T23:48:54.840 DEBUG:boto:Params: {} 2023-07-28T23:48:54.840 DEBUG:boto:Token: None 2023-07-28T23:48:54.841 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:54.841 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XnbQ7pj7nWZkNN3Am2QeGiVOdBo= 2023-07-28T23:48:54.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XnbQ7pj7nWZkNN3Am2QeGiVOdBo=', 'Content-Length': '0'} 2023-07-28T23:48:54.843 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:16 GMT'), ('x-amz-version-id', 'cVvFSQn3t1--pfP44jwpm7HeB4ciTyV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d071c1aaf6ffe2ba-0064c453e6-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.844 DEBUG:boto:path=/hiiqos-12/dummy 2023-07-28T23:48:54.844 DEBUG:boto:auth_path=/hiiqos-12/dummy 2023-07-28T23:48:54.845 DEBUG:boto:path=/hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:54.845 DEBUG:boto:auth_path=/hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:54.846 DEBUG:boto:Method: GET 2023-07-28T23:48:54.846 DEBUG:boto:Path: /hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:54.846 DEBUG:boto:Data: 2023-07-28T23:48:54.847 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.847 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:54.847 DEBUG:boto:Port: 8000 2023-07-28T23:48:54.848 DEBUG:boto:Params: {} 2023-07-28T23:48:54.848 DEBUG:boto:Token: None 2023-07-28T23:48:54.848 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:54.849 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XnbQ7pj7nWZkNN3Am2QeGiVOdBo= 2023-07-28T23:48:54.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XnbQ7pj7nWZkNN3Am2QeGiVOdBo=', 'Content-Length': '0'} 2023-07-28T23:48:54.851 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:16 GMT'), ('x-amz-version-id', 'cVvFSQn3t1--pfP44jwpm7HeB4ciTyV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f51965ca5871d994-0064c453e6-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.852 DEBUG:boto:path=/hiiqos-12/dummy 2023-07-28T23:48:54.852 DEBUG:boto:auth_path=/hiiqos-12/dummy 2023-07-28T23:48:54.852 DEBUG:boto:path=/hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:54.853 DEBUG:boto:auth_path=/hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:54.853 DEBUG:boto:Method: HEAD 2023-07-28T23:48:54.853 DEBUG:boto:Path: /hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:54.854 DEBUG:boto:Data: 2023-07-28T23:48:54.854 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.855 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:54.855 DEBUG:boto:Port: 8001 2023-07-28T23:48:54.855 DEBUG:boto:Params: {} 2023-07-28T23:48:54.856 DEBUG:boto:Token: None 2023-07-28T23:48:54.856 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:54.857 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w9W+dot0bm5y01n3+F6xg1LFy4Y= 2023-07-28T23:48:54.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w9W+dot0bm5y01n3+F6xg1LFy4Y=', 'Content-Length': '0'} 2023-07-28T23:48:54.859 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:16 GMT'), ('x-amz-version-id', 'cVvFSQn3t1--pfP44jwpm7HeB4ciTyV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001d2a005d6c545317-0064c453e6-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.859 DEBUG:boto:path=/hiiqos-12/dummy 2023-07-28T23:48:54.860 DEBUG:boto:auth_path=/hiiqos-12/dummy 2023-07-28T23:48:54.860 DEBUG:boto:path=/hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:54.860 DEBUG:boto:auth_path=/hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:54.861 DEBUG:boto:Method: HEAD 2023-07-28T23:48:54.861 DEBUG:boto:Path: /hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:54.861 DEBUG:boto:Data: 2023-07-28T23:48:54.862 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.862 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:54.862 DEBUG:boto:Port: 8000 2023-07-28T23:48:54.863 DEBUG:boto:Params: {} 2023-07-28T23:48:54.863 DEBUG:boto:Token: None 2023-07-28T23:48:54.864 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:54.864 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w9W+dot0bm5y01n3+F6xg1LFy4Y= 2023-07-28T23:48:54.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w9W+dot0bm5y01n3+F6xg1LFy4Y=', 'Content-Length': '0'} 2023-07-28T23:48:54.866 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:16 GMT'), ('x-amz-version-id', 'cVvFSQn3t1--pfP44jwpm7HeB4ciTyV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000734c1ed147c9e8c-0064c453e6-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-07-28T23:48:54.867 DEBUG:boto:path=/hiiqos-12/dummy 2023-07-28T23:48:54.868 DEBUG:boto:auth_path=/hiiqos-12/dummy 2023-07-28T23:48:54.868 DEBUG:boto:path=/hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:54.869 DEBUG:boto:auth_path=/hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:54.869 DEBUG:boto:Method: GET 2023-07-28T23:48:54.869 DEBUG:boto:Path: /hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:54.870 DEBUG:boto:Data: 2023-07-28T23:48:54.870 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.871 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:54.871 DEBUG:boto:Port: 8001 2023-07-28T23:48:54.871 DEBUG:boto:Params: {} 2023-07-28T23:48:54.872 DEBUG:boto:Token: None 2023-07-28T23:48:54.872 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:54.873 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XnbQ7pj7nWZkNN3Am2QeGiVOdBo= 2023-07-28T23:48:54.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XnbQ7pj7nWZkNN3Am2QeGiVOdBo=', 'Content-Length': '0'} 2023-07-28T23:48:54.875 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:16 GMT'), ('x-amz-version-id', 'cVvFSQn3t1--pfP44jwpm7HeB4ciTyV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000024abccda037d6ded-0064c453e6-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.875 DEBUG:boto:path=/hiiqos-12/dummy 2023-07-28T23:48:54.876 DEBUG:boto:auth_path=/hiiqos-12/dummy 2023-07-28T23:48:54.876 DEBUG:boto:path=/hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:54.877 DEBUG:boto:auth_path=/hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:54.877 DEBUG:boto:Method: GET 2023-07-28T23:48:54.877 DEBUG:boto:Path: /hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:54.878 DEBUG:boto:Data: 2023-07-28T23:48:54.878 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.879 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:54.879 DEBUG:boto:Port: 8000 2023-07-28T23:48:54.880 DEBUG:boto:Params: {} 2023-07-28T23:48:54.880 DEBUG:boto:Token: None 2023-07-28T23:48:54.881 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:54.881 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XnbQ7pj7nWZkNN3Am2QeGiVOdBo= 2023-07-28T23:48:54.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XnbQ7pj7nWZkNN3Am2QeGiVOdBo=', 'Content-Length': '0'} 2023-07-28T23:48:54.884 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:16 GMT'), ('x-amz-version-id', 'cVvFSQn3t1--pfP44jwpm7HeB4ciTyV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008b8f990ad2d538e0-0064c453e6-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.885 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:48:54.885 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:48:54.885 DEBUG:boto:path=/hiiqos-12/?prefix=dummy 2023-07-28T23:48:54.886 DEBUG:boto:auth_path=/hiiqos-12/?prefix=dummy 2023-07-28T23:48:54.886 DEBUG:boto:Method: GET 2023-07-28T23:48:54.887 DEBUG:boto:Path: /hiiqos-12/?prefix=dummy 2023-07-28T23:48:54.887 DEBUG:boto:Data: 2023-07-28T23:48:54.888 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.888 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:54.889 DEBUG:boto:Port: 8001 2023-07-28T23:48:54.890 DEBUG:boto:Params: {} 2023-07-28T23:48:54.890 DEBUG:boto:Token: None 2023-07-28T23:48:54.891 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-12/ 2023-07-28T23:48:54.892 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TJXlVcWMw8az4D7tb9XF1hNSvA8= 2023-07-28T23:48:54.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TJXlVcWMw8az4D7tb9XF1hNSvA8=', 'Content-Length': '0'} 2023-07-28T23:48:54.896 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041a41a4a72e4c150-0064c453e6-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.897 DEBUG:boto:b'hiiqos-12dummy1000falsedummy2023-07-28T23:47:16.239Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:54.898 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:48:54.898 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:48:54.899 DEBUG:boto:path=/hiiqos-12/?prefix=dummy 2023-07-28T23:48:54.899 DEBUG:boto:auth_path=/hiiqos-12/?prefix=dummy 2023-07-28T23:48:54.899 DEBUG:boto:Method: GET 2023-07-28T23:48:54.900 DEBUG:boto:Path: /hiiqos-12/?prefix=dummy 2023-07-28T23:48:54.900 DEBUG:boto:Data: 2023-07-28T23:48:54.900 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.901 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:54.901 DEBUG:boto:Port: 8000 2023-07-28T23:48:54.902 DEBUG:boto:Params: {} 2023-07-28T23:48:54.902 DEBUG:boto:Token: None 2023-07-28T23:48:54.902 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-12/ 2023-07-28T23:48:54.903 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TJXlVcWMw8az4D7tb9XF1hNSvA8= 2023-07-28T23:48:54.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TJXlVcWMw8az4D7tb9XF1hNSvA8=', 'Content-Length': '0'} 2023-07-28T23:48:54.905 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1b6e81ff7ab4352-0064c453e6-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.952 DEBUG:boto:b'hiiqos-12dummy1000falsedummy2023-07-28T23:47:16.239Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:54.953 DEBUG:boto:path=/hiiqos-12/dummy 2023-07-28T23:48:54.954 DEBUG:boto:auth_path=/hiiqos-12/dummy 2023-07-28T23:48:54.954 DEBUG:boto:Method: HEAD 2023-07-28T23:48:54.954 DEBUG:boto:Path: /hiiqos-12/dummy 2023-07-28T23:48:54.955 DEBUG:boto:Data: 2023-07-28T23:48:54.955 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.955 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:54.956 DEBUG:boto:Port: 8001 2023-07-28T23:48:54.957 DEBUG:boto:Params: {} 2023-07-28T23:48:54.957 DEBUG:boto:Token: None 2023-07-28T23:48:54.958 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-12/dummy 2023-07-28T23:48:54.958 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LUp3MVK+LordI1PQWzzBPXgZ9f4= 2023-07-28T23:48:54.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LUp3MVK+LordI1PQWzzBPXgZ9f4=', 'Content-Length': '0'} 2023-07-28T23:48:54.961 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:16 GMT'), ('x-amz-version-id', 'cVvFSQn3t1--pfP44jwpm7HeB4ciTyV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004a54703c31d8782e-0064c453e6-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.961 DEBUG:boto:path=/hiiqos-12/dummy 2023-07-28T23:48:54.962 DEBUG:boto:auth_path=/hiiqos-12/dummy 2023-07-28T23:48:54.962 DEBUG:boto:Method: HEAD 2023-07-28T23:48:54.962 DEBUG:boto:Path: /hiiqos-12/dummy 2023-07-28T23:48:54.963 DEBUG:boto:Data: 2023-07-28T23:48:54.963 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.963 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:54.964 DEBUG:boto:Port: 8000 2023-07-28T23:48:54.964 DEBUG:boto:Params: {} 2023-07-28T23:48:54.964 DEBUG:boto:Token: None 2023-07-28T23:48:54.965 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-12/dummy 2023-07-28T23:48:54.965 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LUp3MVK+LordI1PQWzzBPXgZ9f4= 2023-07-28T23:48:54.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LUp3MVK+LordI1PQWzzBPXgZ9f4=', 'Content-Length': '0'} 2023-07-28T23:48:54.967 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:16 GMT'), ('x-amz-version-id', 'cVvFSQn3t1--pfP44jwpm7HeB4ciTyV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000de3501d85b47c67f-0064c453e6-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-07-28T23:48:54.969 DEBUG:boto:path=/hiiqos-12/dummy 2023-07-28T23:48:54.970 DEBUG:boto:auth_path=/hiiqos-12/dummy 2023-07-28T23:48:54.971 DEBUG:boto:path=/hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:54.972 DEBUG:boto:auth_path=/hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:54.973 DEBUG:boto:Method: GET 2023-07-28T23:48:54.973 DEBUG:boto:Path: /hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:54.973 DEBUG:boto:Data: 2023-07-28T23:48:54.974 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.974 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:54.974 DEBUG:boto:Port: 8001 2023-07-28T23:48:54.975 DEBUG:boto:Params: {} 2023-07-28T23:48:54.975 DEBUG:boto:Token: None 2023-07-28T23:48:54.976 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:54.976 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XnbQ7pj7nWZkNN3Am2QeGiVOdBo= 2023-07-28T23:48:54.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XnbQ7pj7nWZkNN3Am2QeGiVOdBo=', 'Content-Length': '0'} 2023-07-28T23:48:54.978 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:16 GMT'), ('x-amz-version-id', 'cVvFSQn3t1--pfP44jwpm7HeB4ciTyV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b5f79890cca714c1-0064c453e6-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.979 DEBUG:boto:path=/hiiqos-12/dummy 2023-07-28T23:48:54.979 DEBUG:boto:auth_path=/hiiqos-12/dummy 2023-07-28T23:48:54.979 DEBUG:boto:path=/hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:54.980 DEBUG:boto:auth_path=/hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:54.980 DEBUG:boto:Method: GET 2023-07-28T23:48:54.981 DEBUG:boto:Path: /hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:54.981 DEBUG:boto:Data: 2023-07-28T23:48:54.981 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.982 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:54.982 DEBUG:boto:Port: 8000 2023-07-28T23:48:54.983 DEBUG:boto:Params: {} 2023-07-28T23:48:54.983 DEBUG:boto:Token: None 2023-07-28T23:48:54.983 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-12/dummy?versionId=cVvFSQn3t1--pfP44jwpm7HeB4ciTyV 2023-07-28T23:48:54.984 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XnbQ7pj7nWZkNN3Am2QeGiVOdBo= 2023-07-28T23:48:54.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XnbQ7pj7nWZkNN3Am2QeGiVOdBo=', 'Content-Length': '0'} 2023-07-28T23:48:54.986 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:16 GMT'), ('x-amz-version-id', 'cVvFSQn3t1--pfP44jwpm7HeB4ciTyV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004b8775dc2678f9a9-0064c453e6-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:54.987 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:48:54.987 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:48:54.987 DEBUG:boto:path=/hiiqos-12/?prefix=dummy 2023-07-28T23:48:54.988 DEBUG:boto:auth_path=/hiiqos-12/?prefix=dummy 2023-07-28T23:48:54.988 DEBUG:boto:Method: GET 2023-07-28T23:48:54.989 DEBUG:boto:Path: /hiiqos-12/?prefix=dummy 2023-07-28T23:48:54.989 DEBUG:boto:Data: 2023-07-28T23:48:54.989 DEBUG:boto:Headers: {} 2023-07-28T23:48:54.990 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:54.990 DEBUG:boto:Port: 8001 2023-07-28T23:48:54.990 DEBUG:boto:Params: {} 2023-07-28T23:48:54.991 DEBUG:boto:Token: None 2023-07-28T23:48:54.991 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:54 GMT /hiiqos-12/ 2023-07-28T23:48:54.992 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TJXlVcWMw8az4D7tb9XF1hNSvA8= 2023-07-28T23:48:54.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TJXlVcWMw8az4D7tb9XF1hNSvA8=', 'Content-Length': '0'} 2023-07-28T23:48:54.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000888532899f5f4fd2-0064c453e6-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:55.036 DEBUG:boto:b'hiiqos-12dummy1000falsedummy2023-07-28T23:47:16.239Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:55.037 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:48:55.037 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:48:55.038 DEBUG:boto:path=/hiiqos-12/?prefix=dummy 2023-07-28T23:48:55.038 DEBUG:boto:auth_path=/hiiqos-12/?prefix=dummy 2023-07-28T23:48:55.038 DEBUG:boto:Method: GET 2023-07-28T23:48:55.039 DEBUG:boto:Path: /hiiqos-12/?prefix=dummy 2023-07-28T23:48:55.039 DEBUG:boto:Data: 2023-07-28T23:48:55.039 DEBUG:boto:Headers: {} 2023-07-28T23:48:55.040 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:55.040 DEBUG:boto:Port: 8000 2023-07-28T23:48:55.041 DEBUG:boto:Params: {} 2023-07-28T23:48:55.041 DEBUG:boto:Token: None 2023-07-28T23:48:55.041 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:55 GMT /hiiqos-12/ 2023-07-28T23:48:55.042 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+t/ZJUrlQR5nYv06tFcatg7Zua0= 2023-07-28T23:48:55.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+t/ZJUrlQR5nYv06tFcatg7Zua0=', 'Content-Length': '0'} 2023-07-28T23:48:55.044 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074dd8537a335a8a8-0064c453e7-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:55.045 DEBUG:boto:b'hiiqos-12dummy1000falsedummy2023-07-28T23:47:16.239Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:55.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-07-28T23:48:55.046 DEBUG:boto:path=/hiiqos-12/obj-a1 2023-07-28T23:48:55.046 DEBUG:boto:auth_path=/hiiqos-12/obj-a1 2023-07-28T23:48:55.047 DEBUG:boto:path=/hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:55.047 DEBUG:boto:auth_path=/hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:55.047 DEBUG:boto:Method: GET 2023-07-28T23:48:55.048 DEBUG:boto:Path: /hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:55.048 DEBUG:boto:Data: 2023-07-28T23:48:55.048 DEBUG:boto:Headers: {} 2023-07-28T23:48:55.049 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:55.049 DEBUG:boto:Port: 8001 2023-07-28T23:48:55.049 DEBUG:boto:Params: {} 2023-07-28T23:48:55.050 DEBUG:boto:Token: None 2023-07-28T23:48:55.050 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:55 GMT /hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:55.051 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1UhzLZx+Romda6nV1BbPwAHtaAU= 2023-07-28T23:48:55.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1UhzLZx+Romda6nV1BbPwAHtaAU=', 'Content-Length': '0'} 2023-07-28T23:48:55.053 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'Zt8NMtOe11K3S282QQxMGXP84LEhPzt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009dd9a9b582c55c6b-0064c453e7-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:55.054 DEBUG:boto:path=/hiiqos-12/obj-a1 2023-07-28T23:48:55.054 DEBUG:boto:auth_path=/hiiqos-12/obj-a1 2023-07-28T23:48:55.054 DEBUG:boto:path=/hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:55.055 DEBUG:boto:auth_path=/hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:55.055 DEBUG:boto:Method: GET 2023-07-28T23:48:55.055 DEBUG:boto:Path: /hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:55.056 DEBUG:boto:Data: 2023-07-28T23:48:55.056 DEBUG:boto:Headers: {} 2023-07-28T23:48:55.057 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:55.057 DEBUG:boto:Port: 8000 2023-07-28T23:48:55.057 DEBUG:boto:Params: {} 2023-07-28T23:48:55.058 DEBUG:boto:Token: None 2023-07-28T23:48:55.058 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:55 GMT /hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:55.059 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1UhzLZx+Romda6nV1BbPwAHtaAU= 2023-07-28T23:48:55.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1UhzLZx+Romda6nV1BbPwAHtaAU=', 'Content-Length': '0'} 2023-07-28T23:48:55.061 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'Zt8NMtOe11K3S282QQxMGXP84LEhPzt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9c32b4b3195441f-0064c453e7-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:55.061 DEBUG:boto:path=/hiiqos-12/obj-a1 2023-07-28T23:48:55.062 DEBUG:boto:auth_path=/hiiqos-12/obj-a1 2023-07-28T23:48:55.062 DEBUG:boto:path=/hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:55.062 DEBUG:boto:auth_path=/hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:55.063 DEBUG:boto:Method: HEAD 2023-07-28T23:48:55.064 DEBUG:boto:Path: /hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:55.065 DEBUG:boto:Data: 2023-07-28T23:48:55.066 DEBUG:boto:Headers: {} 2023-07-28T23:48:55.067 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:55.067 DEBUG:boto:Port: 8001 2023-07-28T23:48:55.068 DEBUG:boto:Params: {} 2023-07-28T23:48:55.069 DEBUG:boto:Token: None 2023-07-28T23:48:55.070 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:55 GMT /hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:55.071 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bvnTFmmzYMPr5tJ81O0ionu855E= 2023-07-28T23:48:55.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bvnTFmmzYMPr5tJ81O0ionu855E=', 'Content-Length': '0'} 2023-07-28T23:48:55.073 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'Zt8NMtOe11K3S282QQxMGXP84LEhPzt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000097cf3df36e9dea25-0064c453e7-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:55.074 DEBUG:boto:path=/hiiqos-12/obj-a1 2023-07-28T23:48:55.074 DEBUG:boto:auth_path=/hiiqos-12/obj-a1 2023-07-28T23:48:55.075 DEBUG:boto:path=/hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:55.075 DEBUG:boto:auth_path=/hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:55.075 DEBUG:boto:Method: HEAD 2023-07-28T23:48:55.076 DEBUG:boto:Path: /hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:55.076 DEBUG:boto:Data: 2023-07-28T23:48:55.076 DEBUG:boto:Headers: {} 2023-07-28T23:48:55.077 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:55.077 DEBUG:boto:Port: 8000 2023-07-28T23:48:55.078 DEBUG:boto:Params: {} 2023-07-28T23:48:55.078 DEBUG:boto:Token: None 2023-07-28T23:48:55.079 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:55 GMT /hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:55.079 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bvnTFmmzYMPr5tJ81O0ionu855E= 2023-07-28T23:48:55.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bvnTFmmzYMPr5tJ81O0ionu855E=', 'Content-Length': '0'} 2023-07-28T23:48:55.081 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'Zt8NMtOe11K3S282QQxMGXP84LEhPzt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000065c9b09a21ad8b2a-0064c453e7-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:55.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-07-28T23:48:55.083 DEBUG:boto:path=/hiiqos-12/obj-a1 2023-07-28T23:48:55.084 DEBUG:boto:auth_path=/hiiqos-12/obj-a1 2023-07-28T23:48:55.085 DEBUG:boto:path=/hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:55.085 DEBUG:boto:auth_path=/hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:55.086 DEBUG:boto:Method: GET 2023-07-28T23:48:55.087 DEBUG:boto:Path: /hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:55.087 DEBUG:boto:Data: 2023-07-28T23:48:55.088 DEBUG:boto:Headers: {} 2023-07-28T23:48:55.089 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:55.089 DEBUG:boto:Port: 8001 2023-07-28T23:48:55.090 DEBUG:boto:Params: {} 2023-07-28T23:48:55.091 DEBUG:boto:Token: None 2023-07-28T23:48:55.092 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:55 GMT /hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:55.092 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1UhzLZx+Romda6nV1BbPwAHtaAU= 2023-07-28T23:48:55.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1UhzLZx+Romda6nV1BbPwAHtaAU=', 'Content-Length': '0'} 2023-07-28T23:48:55.096 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'Zt8NMtOe11K3S282QQxMGXP84LEhPzt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c2c15b28bd8d6b76-0064c453e7-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:55.097 DEBUG:boto:path=/hiiqos-12/obj-a1 2023-07-28T23:48:55.097 DEBUG:boto:auth_path=/hiiqos-12/obj-a1 2023-07-28T23:48:55.098 DEBUG:boto:path=/hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:55.099 DEBUG:boto:auth_path=/hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:55.099 DEBUG:boto:Method: GET 2023-07-28T23:48:55.100 DEBUG:boto:Path: /hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:55.101 DEBUG:boto:Data: 2023-07-28T23:48:55.101 DEBUG:boto:Headers: {} 2023-07-28T23:48:55.102 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:55.103 DEBUG:boto:Port: 8000 2023-07-28T23:48:55.103 DEBUG:boto:Params: {} 2023-07-28T23:48:55.104 DEBUG:boto:Token: None 2023-07-28T23:48:55.105 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:55 GMT /hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:55.106 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1UhzLZx+Romda6nV1BbPwAHtaAU= 2023-07-28T23:48:55.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1UhzLZx+Romda6nV1BbPwAHtaAU=', 'Content-Length': '0'} 2023-07-28T23:48:55.109 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'Zt8NMtOe11K3S282QQxMGXP84LEhPzt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006b0e38de3937c039-0064c453e7-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:55.152 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:48:55.153 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:48:55.153 DEBUG:boto:path=/hiiqos-12/?prefix=obj-a1 2023-07-28T23:48:55.154 DEBUG:boto:auth_path=/hiiqos-12/?prefix=obj-a1 2023-07-28T23:48:55.154 DEBUG:boto:Method: GET 2023-07-28T23:48:55.155 DEBUG:boto:Path: /hiiqos-12/?prefix=obj-a1 2023-07-28T23:48:55.155 DEBUG:boto:Data: 2023-07-28T23:48:55.156 DEBUG:boto:Headers: {} 2023-07-28T23:48:55.156 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:55.157 DEBUG:boto:Port: 8001 2023-07-28T23:48:55.157 DEBUG:boto:Params: {} 2023-07-28T23:48:55.157 DEBUG:boto:Token: None 2023-07-28T23:48:55.158 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:55 GMT /hiiqos-12/ 2023-07-28T23:48:55.159 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+t/ZJUrlQR5nYv06tFcatg7Zua0= 2023-07-28T23:48:55.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+t/ZJUrlQR5nYv06tFcatg7Zua0=', 'Content-Length': '0'} 2023-07-28T23:48:55.163 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3577e879d0db568-0064c453e7-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:55.164 DEBUG:boto:b'hiiqos-12obj-a11000falseobj-a12023-07-28T23:47:36.626Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:55.164 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:48:55.165 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:48:55.165 DEBUG:boto:path=/hiiqos-12/?prefix=obj-a1 2023-07-28T23:48:55.166 DEBUG:boto:auth_path=/hiiqos-12/?prefix=obj-a1 2023-07-28T23:48:55.166 DEBUG:boto:Method: GET 2023-07-28T23:48:55.167 DEBUG:boto:Path: /hiiqos-12/?prefix=obj-a1 2023-07-28T23:48:55.167 DEBUG:boto:Data: 2023-07-28T23:48:55.167 DEBUG:boto:Headers: {} 2023-07-28T23:48:55.168 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:55.168 DEBUG:boto:Port: 8000 2023-07-28T23:48:55.169 DEBUG:boto:Params: {} 2023-07-28T23:48:55.169 DEBUG:boto:Token: None 2023-07-28T23:48:55.169 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:55 GMT /hiiqos-12/ 2023-07-28T23:48:55.170 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+t/ZJUrlQR5nYv06tFcatg7Zua0= 2023-07-28T23:48:55.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+t/ZJUrlQR5nYv06tFcatg7Zua0=', 'Content-Length': '0'} 2023-07-28T23:48:55.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000166c21b9508404c5-0064c453e7-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:55.173 DEBUG:boto:b'hiiqos-12obj-a11000falseobj-a12023-07-28T23:47:36.626Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:55.174 DEBUG:boto:path=/hiiqos-12/obj-a1 2023-07-28T23:48:55.174 DEBUG:boto:auth_path=/hiiqos-12/obj-a1 2023-07-28T23:48:55.175 DEBUG:boto:Method: HEAD 2023-07-28T23:48:55.175 DEBUG:boto:Path: /hiiqos-12/obj-a1 2023-07-28T23:48:55.175 DEBUG:boto:Data: 2023-07-28T23:48:55.176 DEBUG:boto:Headers: {} 2023-07-28T23:48:55.176 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:55.177 DEBUG:boto:Port: 8001 2023-07-28T23:48:55.177 DEBUG:boto:Params: {} 2023-07-28T23:48:55.177 DEBUG:boto:Token: None 2023-07-28T23:48:55.178 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:55 GMT /hiiqos-12/obj-a1 2023-07-28T23:48:55.178 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8lAQsoIIeLIdw89Hw3gN3GHEPHg= 2023-07-28T23:48:55.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8lAQsoIIeLIdw89Hw3gN3GHEPHg=', 'Content-Length': '0'} 2023-07-28T23:48:55.181 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'Zt8NMtOe11K3S282QQxMGXP84LEhPzt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000063059b8afe38c4fe-0064c453e7-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:55.181 DEBUG:boto:path=/hiiqos-12/obj-a1 2023-07-28T23:48:55.182 DEBUG:boto:auth_path=/hiiqos-12/obj-a1 2023-07-28T23:48:55.182 DEBUG:boto:Method: HEAD 2023-07-28T23:48:55.182 DEBUG:boto:Path: /hiiqos-12/obj-a1 2023-07-28T23:48:55.183 DEBUG:boto:Data: 2023-07-28T23:48:55.183 DEBUG:boto:Headers: {} 2023-07-28T23:48:55.183 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:55.184 DEBUG:boto:Port: 8000 2023-07-28T23:48:55.184 DEBUG:boto:Params: {} 2023-07-28T23:48:55.185 DEBUG:boto:Token: None 2023-07-28T23:48:55.185 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:55 GMT /hiiqos-12/obj-a1 2023-07-28T23:48:55.185 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8lAQsoIIeLIdw89Hw3gN3GHEPHg= 2023-07-28T23:48:55.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8lAQsoIIeLIdw89Hw3gN3GHEPHg=', 'Content-Length': '0'} 2023-07-28T23:48:55.188 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'Zt8NMtOe11K3S282QQxMGXP84LEhPzt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004b0bca66b9fc48b3-0064c453e7-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:55.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-07-28T23:48:55.190 DEBUG:boto:path=/hiiqos-12/obj-a1 2023-07-28T23:48:55.191 DEBUG:boto:auth_path=/hiiqos-12/obj-a1 2023-07-28T23:48:55.192 DEBUG:boto:path=/hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:55.193 DEBUG:boto:auth_path=/hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:55.193 DEBUG:boto:Method: GET 2023-07-28T23:48:55.194 DEBUG:boto:Path: /hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:55.194 DEBUG:boto:Data: 2023-07-28T23:48:55.194 DEBUG:boto:Headers: {} 2023-07-28T23:48:55.195 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:55.195 DEBUG:boto:Port: 8001 2023-07-28T23:48:55.195 DEBUG:boto:Params: {} 2023-07-28T23:48:55.196 DEBUG:boto:Token: None 2023-07-28T23:48:55.196 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:55 GMT /hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:55.197 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1UhzLZx+Romda6nV1BbPwAHtaAU= 2023-07-28T23:48:55.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1UhzLZx+Romda6nV1BbPwAHtaAU=', 'Content-Length': '0'} 2023-07-28T23:48:55.199 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'Zt8NMtOe11K3S282QQxMGXP84LEhPzt'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000085bb84009cc2be5c-0064c453e7-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:55.244 DEBUG:boto:path=/hiiqos-12/obj-a1 2023-07-28T23:48:55.245 DEBUG:boto:auth_path=/hiiqos-12/obj-a1 2023-07-28T23:48:55.245 DEBUG:boto:path=/hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:55.246 DEBUG:boto:auth_path=/hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:55.246 DEBUG:boto:Method: GET 2023-07-28T23:48:55.246 DEBUG:boto:Path: /hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:55.247 DEBUG:boto:Data: 2023-07-28T23:48:55.247 DEBUG:boto:Headers: {} 2023-07-28T23:48:55.247 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:55.248 DEBUG:boto:Port: 8000 2023-07-28T23:48:55.248 DEBUG:boto:Params: {} 2023-07-28T23:48:55.249 DEBUG:boto:Token: None 2023-07-28T23:48:55.249 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:55 GMT /hiiqos-12/obj-a1?versionId=Zt8NMtOe11K3S282QQxMGXP84LEhPzt 2023-07-28T23:48:55.250 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1UhzLZx+Romda6nV1BbPwAHtaAU= 2023-07-28T23:48:55.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1UhzLZx+Romda6nV1BbPwAHtaAU=', 'Content-Length': '0'} 2023-07-28T23:48:55.252 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'Zt8NMtOe11K3S282QQxMGXP84LEhPzt'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000088ad9a0f58068d12-0064c453e7-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:55.252 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:48:55.253 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:48:55.253 DEBUG:boto:path=/hiiqos-12/?prefix=obj-a1 2023-07-28T23:48:55.254 DEBUG:boto:auth_path=/hiiqos-12/?prefix=obj-a1 2023-07-28T23:48:55.254 DEBUG:boto:Method: GET 2023-07-28T23:48:55.254 DEBUG:boto:Path: /hiiqos-12/?prefix=obj-a1 2023-07-28T23:48:55.255 DEBUG:boto:Data: 2023-07-28T23:48:55.255 DEBUG:boto:Headers: {} 2023-07-28T23:48:55.255 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:55.256 DEBUG:boto:Port: 8001 2023-07-28T23:48:55.256 DEBUG:boto:Params: {} 2023-07-28T23:48:55.257 DEBUG:boto:Token: None 2023-07-28T23:48:55.257 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:55 GMT /hiiqos-12/ 2023-07-28T23:48:55.257 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+t/ZJUrlQR5nYv06tFcatg7Zua0= 2023-07-28T23:48:55.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+t/ZJUrlQR5nYv06tFcatg7Zua0=', 'Content-Length': '0'} 2023-07-28T23:48:55.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b83d6f8cb1ea162-0064c453e7-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:55.260 DEBUG:boto:b'hiiqos-12obj-a11000falseobj-a12023-07-28T23:47:36.626Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:55.261 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:48:55.262 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:48:55.262 DEBUG:boto:path=/hiiqos-12/?prefix=obj-a1 2023-07-28T23:48:55.262 DEBUG:boto:auth_path=/hiiqos-12/?prefix=obj-a1 2023-07-28T23:48:55.263 DEBUG:boto:Method: GET 2023-07-28T23:48:55.263 DEBUG:boto:Path: /hiiqos-12/?prefix=obj-a1 2023-07-28T23:48:55.263 DEBUG:boto:Data: 2023-07-28T23:48:55.264 DEBUG:boto:Headers: {} 2023-07-28T23:48:55.264 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:55.264 DEBUG:boto:Port: 8000 2023-07-28T23:48:55.265 DEBUG:boto:Params: {} 2023-07-28T23:48:55.265 DEBUG:boto:Token: None 2023-07-28T23:48:55.266 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:55 GMT /hiiqos-12/ 2023-07-28T23:48:55.266 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+t/ZJUrlQR5nYv06tFcatg7Zua0= 2023-07-28T23:48:55.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+t/ZJUrlQR5nYv06tFcatg7Zua0=', 'Content-Length': '0'} 2023-07-28T23:48:55.268 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f2b8a526a5fed9f-0064c453e7-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:55.269 DEBUG:boto:b'hiiqos-12obj-a11000falseobj-a12023-07-28T23:47:36.626Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:55.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-07-28T23:48:55.270 DEBUG:boto:path=/hiiqos-12/obj-a2 2023-07-28T23:48:55.270 DEBUG:boto:auth_path=/hiiqos-12/obj-a2 2023-07-28T23:48:55.271 DEBUG:boto:path=/hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:55.271 DEBUG:boto:auth_path=/hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:55.271 DEBUG:boto:Method: GET 2023-07-28T23:48:55.272 DEBUG:boto:Path: /hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:55.272 DEBUG:boto:Data: 2023-07-28T23:48:55.272 DEBUG:boto:Headers: {} 2023-07-28T23:48:55.273 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:55.273 DEBUG:boto:Port: 8001 2023-07-28T23:48:55.274 DEBUG:boto:Params: {} 2023-07-28T23:48:55.274 DEBUG:boto:Token: None 2023-07-28T23:48:55.274 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:55 GMT /hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:55.275 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:99UGlVlWLbUiPPZxFmoHTErWPCc= 2023-07-28T23:48:55.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:99UGlVlWLbUiPPZxFmoHTErWPCc=', 'Content-Length': '0'} 2023-07-28T23:48:55.277 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'KycZQE1frBdJnvJd6sV3PeY5cRPnOLF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000155f27f55ef48509-0064c453e7-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:55.277 DEBUG:boto:path=/hiiqos-12/obj-a2 2023-07-28T23:48:55.278 DEBUG:boto:auth_path=/hiiqos-12/obj-a2 2023-07-28T23:48:55.278 DEBUG:boto:path=/hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:55.279 DEBUG:boto:auth_path=/hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:55.279 DEBUG:boto:Method: GET 2023-07-28T23:48:55.279 DEBUG:boto:Path: /hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:55.280 DEBUG:boto:Data: 2023-07-28T23:48:55.280 DEBUG:boto:Headers: {} 2023-07-28T23:48:55.280 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:55.281 DEBUG:boto:Port: 8000 2023-07-28T23:48:55.281 DEBUG:boto:Params: {} 2023-07-28T23:48:55.282 DEBUG:boto:Token: None 2023-07-28T23:48:55.282 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:55 GMT /hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:55.282 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:99UGlVlWLbUiPPZxFmoHTErWPCc= 2023-07-28T23:48:55.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:99UGlVlWLbUiPPZxFmoHTErWPCc=', 'Content-Length': '0'} 2023-07-28T23:48:55.284 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'KycZQE1frBdJnvJd6sV3PeY5cRPnOLF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000252c53af6da3a8a8-0064c453e7-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:55.328 DEBUG:boto:path=/hiiqos-12/obj-a2 2023-07-28T23:48:55.329 DEBUG:boto:auth_path=/hiiqos-12/obj-a2 2023-07-28T23:48:55.329 DEBUG:boto:path=/hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:55.329 DEBUG:boto:auth_path=/hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:55.330 DEBUG:boto:Method: HEAD 2023-07-28T23:48:55.330 DEBUG:boto:Path: /hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:55.330 DEBUG:boto:Data: 2023-07-28T23:48:55.331 DEBUG:boto:Headers: {} 2023-07-28T23:48:55.331 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:55.331 DEBUG:boto:Port: 8001 2023-07-28T23:48:55.332 DEBUG:boto:Params: {} 2023-07-28T23:48:55.332 DEBUG:boto:Token: None 2023-07-28T23:48:55.333 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:55 GMT /hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:55.333 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8+L7h+9x1Wrv5OcuD/2F9eGTijI= 2023-07-28T23:48:55.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8+L7h+9x1Wrv5OcuD/2F9eGTijI=', 'Content-Length': '0'} 2023-07-28T23:48:55.335 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'KycZQE1frBdJnvJd6sV3PeY5cRPnOLF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000612156c5a30bd176-0064c453e7-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:55.336 DEBUG:boto:path=/hiiqos-12/obj-a2 2023-07-28T23:48:55.336 DEBUG:boto:auth_path=/hiiqos-12/obj-a2 2023-07-28T23:48:55.337 DEBUG:boto:path=/hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:55.337 DEBUG:boto:auth_path=/hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:55.337 DEBUG:boto:Method: HEAD 2023-07-28T23:48:55.338 DEBUG:boto:Path: /hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:55.338 DEBUG:boto:Data: 2023-07-28T23:48:55.338 DEBUG:boto:Headers: {} 2023-07-28T23:48:55.339 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:55.339 DEBUG:boto:Port: 8000 2023-07-28T23:48:55.339 DEBUG:boto:Params: {} 2023-07-28T23:48:55.340 DEBUG:boto:Token: None 2023-07-28T23:48:55.340 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:55 GMT /hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:55.341 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8+L7h+9x1Wrv5OcuD/2F9eGTijI= 2023-07-28T23:48:55.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8+L7h+9x1Wrv5OcuD/2F9eGTijI=', 'Content-Length': '0'} 2023-07-28T23:48:55.343 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'KycZQE1frBdJnvJd6sV3PeY5cRPnOLF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ffa25ea1fb70c45-0064c453e7-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:55.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-07-28T23:48:55.344 DEBUG:boto:path=/hiiqos-12/obj-a2 2023-07-28T23:48:55.344 DEBUG:boto:auth_path=/hiiqos-12/obj-a2 2023-07-28T23:48:55.345 DEBUG:boto:path=/hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:55.345 DEBUG:boto:auth_path=/hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:55.345 DEBUG:boto:Method: GET 2023-07-28T23:48:55.346 DEBUG:boto:Path: /hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:55.346 DEBUG:boto:Data: 2023-07-28T23:48:55.346 DEBUG:boto:Headers: {} 2023-07-28T23:48:55.347 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:55.347 DEBUG:boto:Port: 8001 2023-07-28T23:48:55.348 DEBUG:boto:Params: {} 2023-07-28T23:48:55.348 DEBUG:boto:Token: None 2023-07-28T23:48:55.348 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:55 GMT /hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:55.349 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:99UGlVlWLbUiPPZxFmoHTErWPCc= 2023-07-28T23:48:55.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:99UGlVlWLbUiPPZxFmoHTErWPCc=', 'Content-Length': '0'} 2023-07-28T23:48:55.351 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'KycZQE1frBdJnvJd6sV3PeY5cRPnOLF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e2bf0210c948c4fc-0064c453e7-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:55.352 DEBUG:boto:path=/hiiqos-12/obj-a2 2023-07-28T23:48:55.352 DEBUG:boto:auth_path=/hiiqos-12/obj-a2 2023-07-28T23:48:55.353 DEBUG:boto:path=/hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:55.353 DEBUG:boto:auth_path=/hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:55.353 DEBUG:boto:Method: GET 2023-07-28T23:48:55.354 DEBUG:boto:Path: /hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:55.354 DEBUG:boto:Data: 2023-07-28T23:48:55.355 DEBUG:boto:Headers: {} 2023-07-28T23:48:55.355 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:55.356 DEBUG:boto:Port: 8000 2023-07-28T23:48:55.356 DEBUG:boto:Params: {} 2023-07-28T23:48:55.357 DEBUG:boto:Token: None 2023-07-28T23:48:55.357 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:55 GMT /hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:55.357 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:99UGlVlWLbUiPPZxFmoHTErWPCc= 2023-07-28T23:48:55.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:99UGlVlWLbUiPPZxFmoHTErWPCc=', 'Content-Length': '0'} 2023-07-28T23:48:55.359 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'KycZQE1frBdJnvJd6sV3PeY5cRPnOLF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000033b64dcebc84232d-0064c453e7-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:55.360 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:48:55.361 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:48:55.361 DEBUG:boto:path=/hiiqos-12/?prefix=obj-a2 2023-07-28T23:48:55.361 DEBUG:boto:auth_path=/hiiqos-12/?prefix=obj-a2 2023-07-28T23:48:55.362 DEBUG:boto:Method: GET 2023-07-28T23:48:55.363 DEBUG:boto:Path: /hiiqos-12/?prefix=obj-a2 2023-07-28T23:48:55.364 DEBUG:boto:Data: 2023-07-28T23:48:55.365 DEBUG:boto:Headers: {} 2023-07-28T23:48:55.365 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:55.366 DEBUG:boto:Port: 8001 2023-07-28T23:48:55.367 DEBUG:boto:Params: {} 2023-07-28T23:48:55.368 DEBUG:boto:Token: None 2023-07-28T23:48:55.368 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:55 GMT /hiiqos-12/ 2023-07-28T23:48:55.369 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+t/ZJUrlQR5nYv06tFcatg7Zua0= 2023-07-28T23:48:55.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+t/ZJUrlQR5nYv06tFcatg7Zua0=', 'Content-Length': '0'} 2023-07-28T23:48:55.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088337b87b71a62ea-0064c453e7-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:55.372 DEBUG:boto:b'hiiqos-12obj-a21000falseobj-a22023-07-28T23:47:36.751Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:55.373 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:48:55.373 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:48:55.374 DEBUG:boto:path=/hiiqos-12/?prefix=obj-a2 2023-07-28T23:48:55.374 DEBUG:boto:auth_path=/hiiqos-12/?prefix=obj-a2 2023-07-28T23:48:55.374 DEBUG:boto:Method: GET 2023-07-28T23:48:55.375 DEBUG:boto:Path: /hiiqos-12/?prefix=obj-a2 2023-07-28T23:48:55.375 DEBUG:boto:Data: 2023-07-28T23:48:55.375 DEBUG:boto:Headers: {} 2023-07-28T23:48:55.376 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:55.376 DEBUG:boto:Port: 8000 2023-07-28T23:48:55.377 DEBUG:boto:Params: {} 2023-07-28T23:48:55.377 DEBUG:boto:Token: None 2023-07-28T23:48:55.377 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:55 GMT /hiiqos-12/ 2023-07-28T23:48:55.378 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+t/ZJUrlQR5nYv06tFcatg7Zua0= 2023-07-28T23:48:55.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+t/ZJUrlQR5nYv06tFcatg7Zua0=', 'Content-Length': '0'} 2023-07-28T23:48:55.381 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000054dffb0726718b-0064c453e7-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:55.381 DEBUG:boto:b'hiiqos-12obj-a21000falseobj-a22023-07-28T23:47:36.751Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:55.382 DEBUG:boto:path=/hiiqos-12/obj-a2 2023-07-28T23:48:55.382 DEBUG:boto:auth_path=/hiiqos-12/obj-a2 2023-07-28T23:48:55.383 DEBUG:boto:Method: HEAD 2023-07-28T23:48:55.383 DEBUG:boto:Path: /hiiqos-12/obj-a2 2023-07-28T23:48:55.383 DEBUG:boto:Data: 2023-07-28T23:48:55.384 DEBUG:boto:Headers: {} 2023-07-28T23:48:55.384 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:55.384 DEBUG:boto:Port: 8001 2023-07-28T23:48:55.385 DEBUG:boto:Params: {} 2023-07-28T23:48:55.385 DEBUG:boto:Token: None 2023-07-28T23:48:55.386 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:55 GMT /hiiqos-12/obj-a2 2023-07-28T23:48:55.386 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mkHVxm3vP0u7olFUvUKbqLHEBNA= 2023-07-28T23:48:55.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mkHVxm3vP0u7olFUvUKbqLHEBNA=', 'Content-Length': '0'} 2023-07-28T23:48:55.389 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'KycZQE1frBdJnvJd6sV3PeY5cRPnOLF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a392a8235471bb6-0064c453e7-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:55.389 DEBUG:boto:path=/hiiqos-12/obj-a2 2023-07-28T23:48:55.390 DEBUG:boto:auth_path=/hiiqos-12/obj-a2 2023-07-28T23:48:55.391 DEBUG:boto:Method: HEAD 2023-07-28T23:48:55.392 DEBUG:boto:Path: /hiiqos-12/obj-a2 2023-07-28T23:48:55.392 DEBUG:boto:Data: 2023-07-28T23:48:55.393 DEBUG:boto:Headers: {} 2023-07-28T23:48:55.394 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:55.394 DEBUG:boto:Port: 8000 2023-07-28T23:48:55.394 DEBUG:boto:Params: {} 2023-07-28T23:48:55.395 DEBUG:boto:Token: None 2023-07-28T23:48:55.395 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:48:55 GMT /hiiqos-12/obj-a2 2023-07-28T23:48:55.396 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mkHVxm3vP0u7olFUvUKbqLHEBNA= 2023-07-28T23:48:55.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mkHVxm3vP0u7olFUvUKbqLHEBNA=', 'Content-Length': '0'} 2023-07-28T23:48:55.398 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'KycZQE1frBdJnvJd6sV3PeY5cRPnOLF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b5404be04b6c9780-0064c453e7-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:55.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-07-28T23:48:55.400 DEBUG:boto:path=/hiiqos-12/obj-a2 2023-07-28T23:48:55.401 DEBUG:boto:auth_path=/hiiqos-12/obj-a2 2023-07-28T23:48:55.402 DEBUG:boto:path=/hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:55.402 DEBUG:boto:auth_path=/hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:55.403 DEBUG:boto:Method: GET 2023-07-28T23:48:55.404 DEBUG:boto:Path: /hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:55.405 DEBUG:boto:Data: 2023-07-28T23:48:55.406 DEBUG:boto:Headers: {} 2023-07-28T23:48:55.407 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:55.407 DEBUG:boto:Port: 8001 2023-07-28T23:48:55.408 DEBUG:boto:Params: {} 2023-07-28T23:48:55.409 DEBUG:boto:Token: None 2023-07-28T23:48:55.410 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:55 GMT /hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:55.411 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:99UGlVlWLbUiPPZxFmoHTErWPCc= 2023-07-28T23:48:55.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:99UGlVlWLbUiPPZxFmoHTErWPCc=', 'Content-Length': '0'} 2023-07-28T23:48:55.414 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'KycZQE1frBdJnvJd6sV3PeY5cRPnOLF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007f58fb01582c1dcd-0064c453e7-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:55.415 DEBUG:boto:path=/hiiqos-12/obj-a2 2023-07-28T23:48:55.416 DEBUG:boto:auth_path=/hiiqos-12/obj-a2 2023-07-28T23:48:55.416 DEBUG:boto:path=/hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:55.417 DEBUG:boto:auth_path=/hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:55.417 DEBUG:boto:Method: GET 2023-07-28T23:48:55.417 DEBUG:boto:Path: /hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:55.418 DEBUG:boto:Data: 2023-07-28T23:48:55.418 DEBUG:boto:Headers: {} 2023-07-28T23:48:55.418 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:55.419 DEBUG:boto:Port: 8000 2023-07-28T23:48:55.419 DEBUG:boto:Params: {} 2023-07-28T23:48:55.420 DEBUG:boto:Token: None 2023-07-28T23:48:55.420 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:55 GMT /hiiqos-12/obj-a2?versionId=KycZQE1frBdJnvJd6sV3PeY5cRPnOLF 2023-07-28T23:48:55.421 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:99UGlVlWLbUiPPZxFmoHTErWPCc= 2023-07-28T23:48:55.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:99UGlVlWLbUiPPZxFmoHTErWPCc=', 'Content-Length': '0'} 2023-07-28T23:48:55.423 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:47:36 GMT'), ('x-amz-version-id', 'KycZQE1frBdJnvJd6sV3PeY5cRPnOLF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f3fca039cd07cb2b-0064c453e7-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:48:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:55.424 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:48:55.425 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:48:55.426 DEBUG:boto:path=/hiiqos-12/?prefix=obj-a2 2023-07-28T23:48:55.427 DEBUG:boto:auth_path=/hiiqos-12/?prefix=obj-a2 2023-07-28T23:48:55.428 DEBUG:boto:Method: GET 2023-07-28T23:48:55.429 DEBUG:boto:Path: /hiiqos-12/?prefix=obj-a2 2023-07-28T23:48:55.429 DEBUG:boto:Data: 2023-07-28T23:48:55.430 DEBUG:boto:Headers: {} 2023-07-28T23:48:55.430 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:48:55.431 DEBUG:boto:Port: 8001 2023-07-28T23:48:55.431 DEBUG:boto:Params: {} 2023-07-28T23:48:55.432 DEBUG:boto:Token: None 2023-07-28T23:48:55.432 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:55 GMT /hiiqos-12/ 2023-07-28T23:48:55.433 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+t/ZJUrlQR5nYv06tFcatg7Zua0= 2023-07-28T23:48:55.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+t/ZJUrlQR5nYv06tFcatg7Zua0=', 'Content-Length': '0'} 2023-07-28T23:48:55.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d23ca8bf4fe62dfe-0064c453e7-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:55.436 DEBUG:boto:b'hiiqos-12obj-a21000falseobj-a22023-07-28T23:47:36.751Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:55.436 DEBUG:boto:path=/hiiqos-12/ 2023-07-28T23:48:55.437 DEBUG:boto:auth_path=/hiiqos-12/ 2023-07-28T23:48:55.438 DEBUG:boto:path=/hiiqos-12/?prefix=obj-a2 2023-07-28T23:48:55.438 DEBUG:boto:auth_path=/hiiqos-12/?prefix=obj-a2 2023-07-28T23:48:55.438 DEBUG:boto:Method: GET 2023-07-28T23:48:55.439 DEBUG:boto:Path: /hiiqos-12/?prefix=obj-a2 2023-07-28T23:48:55.439 DEBUG:boto:Data: 2023-07-28T23:48:55.439 DEBUG:boto:Headers: {} 2023-07-28T23:48:55.440 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:55.440 DEBUG:boto:Port: 8000 2023-07-28T23:48:55.441 DEBUG:boto:Params: {} 2023-07-28T23:48:55.442 DEBUG:boto:Token: None 2023-07-28T23:48:55.442 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:48:55 GMT /hiiqos-12/ 2023-07-28T23:48:55.443 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+t/ZJUrlQR5nYv06tFcatg7Zua0= 2023-07-28T23:48:55.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+t/ZJUrlQR5nYv06tFcatg7Zua0=', 'Content-Length': '0'} 2023-07-28T23:48:55.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b81001f665a47ebe-0064c453e7-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:48:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:55.446 DEBUG:boto:b'hiiqos-12obj-a21000falseobj-a22023-07-28T23:47:36.751Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:48:55.446 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-12 zones={a2, a1} 2023-07-28T23:48:55.447 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2023-07-28T23:48:55.448 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:48:55.448 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:48:55.449 DEBUG:boto:Method: PUT 2023-07-28T23:48:55.449 DEBUG:boto:Path: /hiiqos-13/ 2023-07-28T23:48:55.449 DEBUG:boto:Data: 2023-07-28T23:48:55.450 DEBUG:boto:Headers: {} 2023-07-28T23:48:55.451 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:55.451 DEBUG:boto:Port: 8000 2023-07-28T23:48:55.452 DEBUG:boto:Params: {} 2023-07-28T23:48:55.452 DEBUG:boto:Token: None 2023-07-28T23:48:55.453 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:48:55 GMT /hiiqos-13/ 2023-07-28T23:48:55.453 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KCCcfIOZQzhLYbuLOM57//yXamM= 2023-07-28T23:48:55.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KCCcfIOZQzhLYbuLOM57//yXamM=', 'Content-Length': '0'} 2023-07-28T23:48:55.471 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007033b2e7e3802a65-0064c453e7-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:48:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:55.473 DEBUG:rgw_multi.tests:created bucket=hiiqos-13 2023-07-28T23:48:55.473 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:48:55.474 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:48:55.475 DEBUG:boto:path=/hiiqos-13/?versioning 2023-07-28T23:48:55.475 DEBUG:boto:auth_path=/hiiqos-13/?versioning 2023-07-28T23:48:55.476 DEBUG:boto:Method: PUT 2023-07-28T23:48:55.476 DEBUG:boto:Path: /hiiqos-13/?versioning 2023-07-28T23:48:55.477 DEBUG:boto:Data: Enabled Disabled 2023-07-28T23:48:55.477 DEBUG:boto:Headers: {} 2023-07-28T23:48:55.478 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:48:55.478 DEBUG:boto:Port: 8000 2023-07-28T23:48:55.478 DEBUG:boto:Params: {} 2023-07-28T23:48:55.479 DEBUG:boto:Token: None 2023-07-28T23:48:55.479 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:48:55 GMT /hiiqos-13/?versioning 2023-07-28T23:48:55.480 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:iNEDS/gEUgviOcGztnwMCTqjOD4= 2023-07-28T23:48:55.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:48:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:iNEDS/gEUgviOcGztnwMCTqjOD4=', 'Content-Length': '228'} 2023-07-28T23:48:55.485 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000041d0f810f559705f-0064c453e7-1117-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:48:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:48:55.486 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:48:55.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:48:55.487 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:48:55.532 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:48:55.533 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:48:55.562 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-28T23:48:55.573 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-28T23:48:55.573 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:48:55.573 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-28T23:48:55.574 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:46:09.233742Z" 2023-07-28T23:48:55.574 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:48:55.574 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:48:55.574 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588135.484705_162.1", 2023-07-28T23:48:55.574 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:48:55.484705Z" 2023-07-28T23:48:55.574 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:48:55.575 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:48:55.575 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:48:55.575 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:48:55.575 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:48:55.575 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:48:55.575 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:48:55.575 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:48:55.576 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:48:55.576 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-28T23:48:55.577 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690587969.233742_99.1', 1: '1_1690588135.484705_162.1', 2: '', 3: ''} 2023-07-28T23:48:55.577 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-28T23:48:55.577 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:48:55.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:48:55.578 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:48:55.626 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:48:55.626 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:48:55.655 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:48:55.654+0000 7f2b22320880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:48:55.655 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:48:55.654+0000 7f2b22320880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:48:55.667 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:48:55.668 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:48:55.668 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:48:55.668 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:48:55.668 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:48:55.669 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:48:55.669 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:48:55.669 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:48:55.669 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:48:55.669 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:48:55.669 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:48:55.669 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:48:55.670 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:48:55.670 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-28T23:48:55.670 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:48:55.670 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:48:55.670 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:48:55.670 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-28T23:48:55.671 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:48:55.671 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:48:55.671 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:48:55.671 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:48:55.671 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:48:55.671 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:48:55.672 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:48:55.672 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588135.462831_145.1", 2023-07-28T23:48:55.672 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:48:55.672 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:48:55.672 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:48:55.672 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:48:55.462831Z", 2023-07-28T23:48:55.673 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:48:55.673 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:48:55.673 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:48:55.673 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:48:55.673 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:48:55.673 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:48:55.673 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:48:55.674 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:48:55.674 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:48:55.674 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:48:55.674 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:48:55.674 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:48:55.674 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:48:55.674 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:48:55.675 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:48:55.675 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:48:55.675 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:48:55.675 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:48:55.675 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:48:55.675 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:48:55.675 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:48:55.676 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:48:55.676 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:48:55.676 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:48:55.676 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:48:55.676 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:48:55.676 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:48:55.677 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:48:55.677 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:48:55.677 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:48:55.677 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:48:55.678 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:48:55.678 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:48:55.678 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:48:55Z" 2023-07-28T23:48:55.678 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:48:55.680 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588135.462831_145.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:48:55.462831Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-28T23:48:55Z" } 2023-07-28T23:48:55.681 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588135.462831_145.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:48:55.462831Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-28T23:48:55.681 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588135.484705_162.1', 2: '', 3: ''} 2023-07-28T23:48:55.682 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588135.462831_145.1', 2: '', 3: ''} 2023-07-28T23:48:55.683 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588135.484705_162.1 target=1_1690588135.462831_145.1 2023-07-28T23:49:00.685 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:49:00.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:49:00.686 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:49:00.741 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:49:00.741 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:49:00.772 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:49:00.771+0000 7f4195626880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:49:00.773 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:49:00.772+0000 7f4195626880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:49:00.785 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:49:00.786 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:49:00.786 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:49:00.786 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:49:00.786 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:49:00.787 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:49:00.787 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:49:00.787 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:49:00.787 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:49:00.787 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:49:00.787 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:49:00.787 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:49:00.788 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:49:00.788 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-28T23:49:00.788 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:49:00.788 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:49:00.788 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:49:00.788 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-28T23:49:00.789 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:49:00.789 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:49:00.789 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:49:00.789 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:49:00.789 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:49:00.789 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:49:00.790 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:49:00.790 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588135.484705_162.1", 2023-07-28T23:49:00.790 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:49:00.790 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:49:00.790 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:49:00.790 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:48:55.484705Z", 2023-07-28T23:49:00.790 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:49:00.791 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:49:00.791 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:49:00.791 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:49:00.792 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:49:00.792 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:49:00.792 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:49:00.792 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:49:00.792 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:49:00.793 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:49:00.793 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:49:00.793 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:49:00.793 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:49:00.793 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:49:00.793 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:49:00.793 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:49:00.794 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:49:00.794 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:49:00.794 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:49:00.794 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:49:00.794 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:49:00.794 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:49:00.794 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:49:00.795 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:49:00.795 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:49:00.795 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:49:00.795 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:49:00.795 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:49:00.795 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:49:00.795 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:49:00.796 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:49:00.796 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:49:00.796 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:49:00.796 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:49:00Z" 2023-07-28T23:49:00.796 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:49:00.797 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588135.484705_162.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:48:55.484705Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-28T23:49:00Z" } 2023-07-28T23:49:00.798 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588135.484705_162.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:48:55.484705Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-28T23:49:00.799 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588135.484705_162.1', 2: '', 3: ''} 2023-07-28T23:49:00.799 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588135.484705_162.1', 2: '', 3: ''} 2023-07-28T23:49:00.800 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-28T23:49:00.801 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:49:00.802 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:49:00.803 DEBUG:boto:Method: HEAD 2023-07-28T23:49:00.804 DEBUG:boto:Path: /hiiqos-13/ 2023-07-28T23:49:00.805 DEBUG:boto:Data: 2023-07-28T23:49:00.806 DEBUG:boto:Headers: {} 2023-07-28T23:49:00.806 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:49:00.807 DEBUG:boto:Port: 8000 2023-07-28T23:49:00.807 DEBUG:boto:Params: {} 2023-07-28T23:49:00.808 DEBUG:boto:Token: None 2023-07-28T23:49:00.809 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:49:00 GMT /hiiqos-13/ 2023-07-28T23:49:00.810 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ud7zSYQepIhFDc71GWboxnXQZnI= 2023-07-28T23:49:00.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:49:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ud7zSYQepIhFDc71GWboxnXQZnI=', 'Content-Length': '0'} 2023-07-28T23:49:00.812 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000035baeae1ae3a45bd-0064c453ec-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:49:00.813 DEBUG:boto:path=/hiiqos-13/dummy 2023-07-28T23:49:00.813 DEBUG:boto:auth_path=/hiiqos-13/dummy 2023-07-28T23:49:00.813 DEBUG:boto:Method: PUT 2023-07-28T23:49:00.814 DEBUG:boto:Path: /hiiqos-13/dummy 2023-07-28T23:49:00.814 DEBUG:boto:Data: 2023-07-28T23:49:00.815 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-07-28T23:49:00.815 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:49:00.815 DEBUG:boto:Port: 8000 2023-07-28T23:49:00.816 DEBUG:boto:Params: {} 2023-07-28T23:49:00.817 DEBUG:boto:Token: None 2023-07-28T23:49:00.817 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 28 Jul 2023 23:49:00 GMT /hiiqos-13/dummy 2023-07-28T23:49:00.818 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4aUA4db9X6cs6Y03Wxx2j1QTK2I= 2023-07-28T23:49:00.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:49:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4aUA4db9X6cs6Y03Wxx2j1QTK2I='} 2023-07-28T23:49:00.829 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad'), ('x-amz-request-id', 'tx000005e3548957f8ce2f1-0064c453ec-1117-a1'), ('Date', 'Fri, 28 Jul 2023 23:49:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:49:00.829 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-13 2023-07-28T23:49:00.830 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:49:00.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:49:00.831 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-13 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-28T23:49:00.884 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:49:00.884 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:49:00.915 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:49:00.914+0000 7f029d038880 1 waiting to reach incremental sync.. 2023-07-28T23:49:05.915 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:49:05.914+0000 7f029d038880 1 waiting to reach incremental sync.. 2023-07-28T23:49:10.916 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:49:10.915+0000 7f029d038880 1 waiting to reach incremental sync.. 2023-07-28T23:49:15.917 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:49:15.915+0000 7f029d038880 1 waiting to reach incremental sync.. 2023-07-28T23:49:20.917 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:49:20.916+0000 7f029d038880 1 waiting to reach incremental sync.. 2023-07-28T23:49:25.918 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:49:25.917+0000 7f029d038880 1 waiting to reach incremental sync.. 2023-07-28T23:49:30.919 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:49:30.918+0000 7f029d038880 1 waiting to reach incremental sync.. 2023-07-28T23:49:35.920 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:49:35.918+0000 7f029d038880 1 waiting to reach incremental sync.. 2023-07-28T23:49:40.920 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:49:40.919+0000 7f029d038880 1 waiting to reach incremental sync.. 2023-07-28T23:49:45.921 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:49:45.920+0000 7f029d038880 1 waiting to reach incremental sync.. 2023-07-28T23:49:50.922 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:49:50.920+0000 7f029d038880 1 waiting to reach incremental sync.. 2023-07-28T23:49:55.923 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:49:55.922+0000 7f029d038880 1 bucket sync caught up with source: 2023-07-28T23:49:55.924 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000002.6.11, , , , , ] 2023-07-28T23:49:55.924 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000002.6.11, , , , , ] 2023-07-28T23:49:55.924 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:49:55.922+0000 7f029d038880 0 bucket checkpoint complete 2023-07-28T23:49:55.939 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-13 2023-07-28T23:49:55.940 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:49:55.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:49:55.941 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-13 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-28T23:49:55.999 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:49:55.999 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:49:56.034 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:49:56.032+0000 7f5c12526880 1 waiting to reach incremental sync.. 2023-07-28T23:50:01.034 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:50:01.033+0000 7f5c12526880 1 waiting to reach incremental sync.. 2023-07-28T23:50:06.035 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:50:06.033+0000 7f5c12526880 1 waiting to reach incremental sync.. 2023-07-28T23:50:11.035 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:50:11.034+0000 7f5c12526880 1 waiting to reach incremental sync.. 2023-07-28T23:50:16.037 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:50:16.036+0000 7f5c12526880 1 bucket sync caught up with source: 2023-07-28T23:50:16.037 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000002.14.11, , , , , ] 2023-07-28T23:50:16.038 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000002.14.11, , , , , ] 2023-07-28T23:50:16.038 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:50:16.036+0000 7f5c12526880 0 bucket checkpoint complete 2023-07-28T23:50:16.053 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-13 zones={a2, a1} 2023-07-28T23:50:16.053 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:50:16.054 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:50:16.054 DEBUG:boto:Method: HEAD 2023-07-28T23:50:16.054 DEBUG:boto:Path: /hiiqos-13/ 2023-07-28T23:50:16.055 DEBUG:boto:Data: 2023-07-28T23:50:16.055 DEBUG:boto:Headers: {} 2023-07-28T23:50:16.055 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:50:16.056 DEBUG:boto:Port: 8001 2023-07-28T23:50:16.056 DEBUG:boto:Params: {} 2023-07-28T23:50:16.057 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-28T23:50:16.057 DEBUG:boto:Token: None 2023-07-28T23:50:16.058 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/ 2023-07-28T23:50:16.059 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus= 2023-07-28T23:50:16.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus=', 'Content-Length': '0'} 2023-07-28T23:50:16.063 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000152b54654ab6ad34-0064c45438-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.064 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:50:16.064 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:50:16.065 DEBUG:boto:Method: HEAD 2023-07-28T23:50:16.065 DEBUG:boto:Path: /hiiqos-13/ 2023-07-28T23:50:16.065 DEBUG:boto:Data: 2023-07-28T23:50:16.066 DEBUG:boto:Headers: {} 2023-07-28T23:50:16.066 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:50:16.066 DEBUG:boto:Port: 8000 2023-07-28T23:50:16.067 DEBUG:boto:Params: {} 2023-07-28T23:50:16.067 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-28T23:50:16.067 DEBUG:boto:Token: None 2023-07-28T23:50:16.068 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/ 2023-07-28T23:50:16.068 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus= 2023-07-28T23:50:16.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus=', 'Content-Length': '0'} 2023-07-28T23:50:16.071 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000016030cbb6c65f8ed-0064c45438-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.072 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-28T23:50:16.072 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:50:16.073 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:50:16.073 DEBUG:boto:path=/hiiqos-13/?versions&max-keys=999 2023-07-28T23:50:16.073 DEBUG:boto:auth_path=/hiiqos-13/?versions&max-keys=999 2023-07-28T23:50:16.074 DEBUG:boto:Method: GET 2023-07-28T23:50:16.074 DEBUG:boto:Path: /hiiqos-13/?versions&max-keys=999 2023-07-28T23:50:16.074 DEBUG:boto:Data: 2023-07-28T23:50:16.075 DEBUG:boto:Headers: {} 2023-07-28T23:50:16.075 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:50:16.075 DEBUG:boto:Port: 8001 2023-07-28T23:50:16.076 DEBUG:boto:Params: {} 2023-07-28T23:50:16.076 DEBUG:boto:Token: None 2023-07-28T23:50:16.077 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/?versions 2023-07-28T23:50:16.077 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:m9UjY7MBPIvYNTdVen6DIFK+dIk= 2023-07-28T23:50:16.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:m9UjY7MBPIvYNTdVen6DIFK+dIk=', 'Content-Length': '0'} 2023-07-28T23:50:16.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c89ae882c1c55b7a-0064c45438-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.080 DEBUG:boto:b'hiiqos-13999falsedummy2iCgPpKTi-zSyx2.F.26sM9kQ2T--adtrue2023-07-28T23:49:00.821Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:50:16.081 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-07-28T23:50:16.081 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-28T23:50:16.082 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:50:16.082 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:50:16.082 DEBUG:boto:path=/hiiqos-13/?versions&max-keys=999 2023-07-28T23:50:16.083 DEBUG:boto:auth_path=/hiiqos-13/?versions&max-keys=999 2023-07-28T23:50:16.083 DEBUG:boto:Method: GET 2023-07-28T23:50:16.083 DEBUG:boto:Path: /hiiqos-13/?versions&max-keys=999 2023-07-28T23:50:16.084 DEBUG:boto:Data: 2023-07-28T23:50:16.084 DEBUG:boto:Headers: {} 2023-07-28T23:50:16.084 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:50:16.085 DEBUG:boto:Port: 8000 2023-07-28T23:50:16.085 DEBUG:boto:Params: {} 2023-07-28T23:50:16.085 DEBUG:boto:Token: None 2023-07-28T23:50:16.086 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/?versions 2023-07-28T23:50:16.086 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:m9UjY7MBPIvYNTdVen6DIFK+dIk= 2023-07-28T23:50:16.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:m9UjY7MBPIvYNTdVen6DIFK+dIk=', 'Content-Length': '0'} 2023-07-28T23:50:16.089 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000654deffeaa584b38-0064c45438-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.089 DEBUG:boto:b'hiiqos-13999falsedummy2iCgPpKTi-zSyx2.F.26sM9kQ2T--adtrue2023-07-28T23:49:00.821Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:50:16.090 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-07-28T23:50:16.090 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:50:16.091 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:50:16.091 DEBUG:boto:path=/hiiqos-13/?versions&max-keys=999 2023-07-28T23:50:16.091 DEBUG:boto:auth_path=/hiiqos-13/?versions&max-keys=999 2023-07-28T23:50:16.092 DEBUG:boto:Method: GET 2023-07-28T23:50:16.092 DEBUG:boto:Path: /hiiqos-13/?versions&max-keys=999 2023-07-28T23:50:16.092 DEBUG:boto:Data: 2023-07-28T23:50:16.093 DEBUG:boto:Headers: {} 2023-07-28T23:50:16.093 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:50:16.093 DEBUG:boto:Port: 8001 2023-07-28T23:50:16.094 DEBUG:boto:Params: {} 2023-07-28T23:50:16.094 DEBUG:boto:Token: None 2023-07-28T23:50:16.094 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/?versions 2023-07-28T23:50:16.095 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:m9UjY7MBPIvYNTdVen6DIFK+dIk= 2023-07-28T23:50:16.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:m9UjY7MBPIvYNTdVen6DIFK+dIk=', 'Content-Length': '0'} 2023-07-28T23:50:16.097 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083688284f7021bc7-0064c45438-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.098 DEBUG:boto:b'hiiqos-13999falsedummy2iCgPpKTi-zSyx2.F.26sM9kQ2T--adtrue2023-07-28T23:49:00.821Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:50:16.098 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:50:16.099 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:50:16.099 DEBUG:boto:path=/hiiqos-13/?versions&max-keys=999 2023-07-28T23:50:16.100 DEBUG:boto:auth_path=/hiiqos-13/?versions&max-keys=999 2023-07-28T23:50:16.100 DEBUG:boto:Method: GET 2023-07-28T23:50:16.100 DEBUG:boto:Path: /hiiqos-13/?versions&max-keys=999 2023-07-28T23:50:16.101 DEBUG:boto:Data: 2023-07-28T23:50:16.101 DEBUG:boto:Headers: {} 2023-07-28T23:50:16.101 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:50:16.102 DEBUG:boto:Port: 8000 2023-07-28T23:50:16.102 DEBUG:boto:Params: {} 2023-07-28T23:50:16.102 DEBUG:boto:Token: None 2023-07-28T23:50:16.103 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/?versions 2023-07-28T23:50:16.103 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:m9UjY7MBPIvYNTdVen6DIFK+dIk= 2023-07-28T23:50:16.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:m9UjY7MBPIvYNTdVen6DIFK+dIk=', 'Content-Length': '0'} 2023-07-28T23:50:16.106 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046a84d948c5ab97f-0064c45438-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.106 DEBUG:boto:b'hiiqos-13999falsedummy2iCgPpKTi-zSyx2.F.26sM9kQ2T--adtrue2023-07-28T23:49:00.821Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:50:16.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-07-28T23:50:16.107 DEBUG:boto:path=/hiiqos-13/dummy 2023-07-28T23:50:16.108 DEBUG:boto:auth_path=/hiiqos-13/dummy 2023-07-28T23:50:16.108 DEBUG:boto:path=/hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:50:16.108 DEBUG:boto:auth_path=/hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:50:16.109 DEBUG:boto:Method: GET 2023-07-28T23:50:16.109 DEBUG:boto:Path: /hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:50:16.109 DEBUG:boto:Data: 2023-07-28T23:50:16.110 DEBUG:boto:Headers: {} 2023-07-28T23:50:16.110 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:50:16.110 DEBUG:boto:Port: 8001 2023-07-28T23:50:16.111 DEBUG:boto:Params: {} 2023-07-28T23:50:16.111 DEBUG:boto:Token: None 2023-07-28T23:50:16.112 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:50:16.112 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qt+mIkStEsRHnzOWTiuMZL7UgYk= 2023-07-28T23:50:16.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qt+mIkStEsRHnzOWTiuMZL7UgYk=', 'Content-Length': '0'} 2023-07-28T23:50:16.115 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:49:00 GMT'), ('x-amz-version-id', '2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ca8f9836086901df-0064c45438-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.116 DEBUG:boto:path=/hiiqos-13/dummy 2023-07-28T23:50:16.116 DEBUG:boto:auth_path=/hiiqos-13/dummy 2023-07-28T23:50:16.116 DEBUG:boto:path=/hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:50:16.117 DEBUG:boto:auth_path=/hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:50:16.117 DEBUG:boto:Method: GET 2023-07-28T23:50:16.117 DEBUG:boto:Path: /hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:50:16.118 DEBUG:boto:Data: 2023-07-28T23:50:16.118 DEBUG:boto:Headers: {} 2023-07-28T23:50:16.119 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:50:16.119 DEBUG:boto:Port: 8000 2023-07-28T23:50:16.119 DEBUG:boto:Params: {} 2023-07-28T23:50:16.120 DEBUG:boto:Token: None 2023-07-28T23:50:16.120 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:50:16.120 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qt+mIkStEsRHnzOWTiuMZL7UgYk= 2023-07-28T23:50:16.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qt+mIkStEsRHnzOWTiuMZL7UgYk=', 'Content-Length': '0'} 2023-07-28T23:50:16.123 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:49:00 GMT'), ('x-amz-version-id', '2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000047c1089a8534dc5b-0064c45438-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.123 DEBUG:boto:path=/hiiqos-13/dummy 2023-07-28T23:50:16.124 DEBUG:boto:auth_path=/hiiqos-13/dummy 2023-07-28T23:50:16.124 DEBUG:boto:path=/hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:50:16.125 DEBUG:boto:auth_path=/hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:50:16.125 DEBUG:boto:Method: HEAD 2023-07-28T23:50:16.125 DEBUG:boto:Path: /hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:50:16.126 DEBUG:boto:Data: 2023-07-28T23:50:16.126 DEBUG:boto:Headers: {} 2023-07-28T23:50:16.126 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:50:16.127 DEBUG:boto:Port: 8001 2023-07-28T23:50:16.127 DEBUG:boto:Params: {} 2023-07-28T23:50:16.127 DEBUG:boto:Token: None 2023-07-28T23:50:16.128 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:50:16.128 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YCtdNBLs9bSrf9AtJagrACqTpsw= 2023-07-28T23:50:16.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YCtdNBLs9bSrf9AtJagrACqTpsw=', 'Content-Length': '0'} 2023-07-28T23:50:16.130 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:49:00 GMT'), ('x-amz-version-id', '2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ca14d15c912a3a02-0064c45438-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.131 DEBUG:boto:path=/hiiqos-13/dummy 2023-07-28T23:50:16.132 DEBUG:boto:auth_path=/hiiqos-13/dummy 2023-07-28T23:50:16.132 DEBUG:boto:path=/hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:50:16.132 DEBUG:boto:auth_path=/hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:50:16.133 DEBUG:boto:Method: HEAD 2023-07-28T23:50:16.133 DEBUG:boto:Path: /hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:50:16.133 DEBUG:boto:Data: 2023-07-28T23:50:16.134 DEBUG:boto:Headers: {} 2023-07-28T23:50:16.134 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:50:16.134 DEBUG:boto:Port: 8000 2023-07-28T23:50:16.135 DEBUG:boto:Params: {} 2023-07-28T23:50:16.135 DEBUG:boto:Token: None 2023-07-28T23:50:16.136 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:50:16.136 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YCtdNBLs9bSrf9AtJagrACqTpsw= 2023-07-28T23:50:16.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YCtdNBLs9bSrf9AtJagrACqTpsw=', 'Content-Length': '0'} 2023-07-28T23:50:16.138 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:49:00 GMT'), ('x-amz-version-id', '2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a60999636846f8c3-0064c45438-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-07-28T23:50:16.139 DEBUG:boto:path=/hiiqos-13/dummy 2023-07-28T23:50:16.139 DEBUG:boto:auth_path=/hiiqos-13/dummy 2023-07-28T23:50:16.140 DEBUG:boto:path=/hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:50:16.140 DEBUG:boto:auth_path=/hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:50:16.140 DEBUG:boto:Method: GET 2023-07-28T23:50:16.141 DEBUG:boto:Path: /hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:50:16.141 DEBUG:boto:Data: 2023-07-28T23:50:16.141 DEBUG:boto:Headers: {} 2023-07-28T23:50:16.142 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:50:16.142 DEBUG:boto:Port: 8001 2023-07-28T23:50:16.142 DEBUG:boto:Params: {} 2023-07-28T23:50:16.143 DEBUG:boto:Token: None 2023-07-28T23:50:16.143 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:50:16.144 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qt+mIkStEsRHnzOWTiuMZL7UgYk= 2023-07-28T23:50:16.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qt+mIkStEsRHnzOWTiuMZL7UgYk=', 'Content-Length': '0'} 2023-07-28T23:50:16.146 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:49:00 GMT'), ('x-amz-version-id', '2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000014fe77d9c8e38a33-0064c45438-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.146 DEBUG:boto:path=/hiiqos-13/dummy 2023-07-28T23:50:16.147 DEBUG:boto:auth_path=/hiiqos-13/dummy 2023-07-28T23:50:16.147 DEBUG:boto:path=/hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:50:16.148 DEBUG:boto:auth_path=/hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:50:16.149 DEBUG:boto:Method: GET 2023-07-28T23:50:16.149 DEBUG:boto:Path: /hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:50:16.149 DEBUG:boto:Data: 2023-07-28T23:50:16.150 DEBUG:boto:Headers: {} 2023-07-28T23:50:16.150 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:50:16.150 DEBUG:boto:Port: 8000 2023-07-28T23:50:16.151 DEBUG:boto:Params: {} 2023-07-28T23:50:16.151 DEBUG:boto:Token: None 2023-07-28T23:50:16.152 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:50:16.152 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qt+mIkStEsRHnzOWTiuMZL7UgYk= 2023-07-28T23:50:16.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qt+mIkStEsRHnzOWTiuMZL7UgYk=', 'Content-Length': '0'} 2023-07-28T23:50:16.154 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:49:00 GMT'), ('x-amz-version-id', '2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000073894863a5bb9603-0064c45438-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.155 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:50:16.155 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:50:16.156 DEBUG:boto:path=/hiiqos-13/?prefix=dummy 2023-07-28T23:50:16.156 DEBUG:boto:auth_path=/hiiqos-13/?prefix=dummy 2023-07-28T23:50:16.156 DEBUG:boto:Method: GET 2023-07-28T23:50:16.157 DEBUG:boto:Path: /hiiqos-13/?prefix=dummy 2023-07-28T23:50:16.157 DEBUG:boto:Data: 2023-07-28T23:50:16.157 DEBUG:boto:Headers: {} 2023-07-28T23:50:16.158 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:50:16.158 DEBUG:boto:Port: 8001 2023-07-28T23:50:16.159 DEBUG:boto:Params: {} 2023-07-28T23:50:16.159 DEBUG:boto:Token: None 2023-07-28T23:50:16.159 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/ 2023-07-28T23:50:16.160 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dcV/CkCn0AEOZr+GidOeUgkY4wQ= 2023-07-28T23:50:16.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dcV/CkCn0AEOZr+GidOeUgkY4wQ=', 'Content-Length': '0'} 2023-07-28T23:50:16.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fbaaab4915214a5-0064c45438-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.163 DEBUG:boto:b'hiiqos-13dummy1000falsedummy2023-07-28T23:49:00.821Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:50:16.164 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:50:16.165 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:50:16.165 DEBUG:boto:path=/hiiqos-13/?prefix=dummy 2023-07-28T23:50:16.165 DEBUG:boto:auth_path=/hiiqos-13/?prefix=dummy 2023-07-28T23:50:16.166 DEBUG:boto:Method: GET 2023-07-28T23:50:16.166 DEBUG:boto:Path: /hiiqos-13/?prefix=dummy 2023-07-28T23:50:16.166 DEBUG:boto:Data: 2023-07-28T23:50:16.167 DEBUG:boto:Headers: {} 2023-07-28T23:50:16.167 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:50:16.167 DEBUG:boto:Port: 8000 2023-07-28T23:50:16.168 DEBUG:boto:Params: {} 2023-07-28T23:50:16.168 DEBUG:boto:Token: None 2023-07-28T23:50:16.169 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/ 2023-07-28T23:50:16.169 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dcV/CkCn0AEOZr+GidOeUgkY4wQ= 2023-07-28T23:50:16.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dcV/CkCn0AEOZr+GidOeUgkY4wQ=', 'Content-Length': '0'} 2023-07-28T23:50:16.172 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd515bf0db93b60b-0064c45438-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.172 DEBUG:boto:b'hiiqos-13dummy1000falsedummy2023-07-28T23:49:00.821Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:50:16.173 DEBUG:boto:path=/hiiqos-13/dummy 2023-07-28T23:50:16.173 DEBUG:boto:auth_path=/hiiqos-13/dummy 2023-07-28T23:50:16.174 DEBUG:boto:Method: HEAD 2023-07-28T23:50:16.174 DEBUG:boto:Path: /hiiqos-13/dummy 2023-07-28T23:50:16.174 DEBUG:boto:Data: 2023-07-28T23:50:16.174 DEBUG:boto:Headers: {} 2023-07-28T23:50:16.175 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:50:16.175 DEBUG:boto:Port: 8001 2023-07-28T23:50:16.175 DEBUG:boto:Params: {} 2023-07-28T23:50:16.176 DEBUG:boto:Token: None 2023-07-28T23:50:16.176 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/dummy 2023-07-28T23:50:16.177 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:501Fg1ySJcCLkfudFMk3+g+hKTg= 2023-07-28T23:50:16.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:501Fg1ySJcCLkfudFMk3+g+hKTg=', 'Content-Length': '0'} 2023-07-28T23:50:16.179 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:49:00 GMT'), ('x-amz-version-id', '2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000001cfb0214d2a100b-0064c45438-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.180 DEBUG:boto:path=/hiiqos-13/dummy 2023-07-28T23:50:16.180 DEBUG:boto:auth_path=/hiiqos-13/dummy 2023-07-28T23:50:16.181 DEBUG:boto:Method: HEAD 2023-07-28T23:50:16.181 DEBUG:boto:Path: /hiiqos-13/dummy 2023-07-28T23:50:16.182 DEBUG:boto:Data: 2023-07-28T23:50:16.182 DEBUG:boto:Headers: {} 2023-07-28T23:50:16.183 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:50:16.183 DEBUG:boto:Port: 8000 2023-07-28T23:50:16.184 DEBUG:boto:Params: {} 2023-07-28T23:50:16.184 DEBUG:boto:Token: None 2023-07-28T23:50:16.184 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/dummy 2023-07-28T23:50:16.185 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:501Fg1ySJcCLkfudFMk3+g+hKTg= 2023-07-28T23:50:16.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:501Fg1ySJcCLkfudFMk3+g+hKTg=', 'Content-Length': '0'} 2023-07-28T23:50:16.188 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:49:00 GMT'), ('x-amz-version-id', '2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000005f30f8519351b7-0064c45438-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-07-28T23:50:16.189 DEBUG:boto:path=/hiiqos-13/dummy 2023-07-28T23:50:16.190 DEBUG:boto:auth_path=/hiiqos-13/dummy 2023-07-28T23:50:16.190 DEBUG:boto:path=/hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:50:16.190 DEBUG:boto:auth_path=/hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:50:16.191 DEBUG:boto:Method: GET 2023-07-28T23:50:16.191 DEBUG:boto:Path: /hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:50:16.191 DEBUG:boto:Data: 2023-07-28T23:50:16.192 DEBUG:boto:Headers: {} 2023-07-28T23:50:16.192 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:50:16.192 DEBUG:boto:Port: 8001 2023-07-28T23:50:16.193 DEBUG:boto:Params: {} 2023-07-28T23:50:16.193 DEBUG:boto:Token: None 2023-07-28T23:50:16.193 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:50:16.194 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qt+mIkStEsRHnzOWTiuMZL7UgYk= 2023-07-28T23:50:16.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qt+mIkStEsRHnzOWTiuMZL7UgYk=', 'Content-Length': '0'} 2023-07-28T23:50:16.196 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:49:00 GMT'), ('x-amz-version-id', '2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f7da27bfc76a8dde-0064c45438-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.197 DEBUG:boto:path=/hiiqos-13/dummy 2023-07-28T23:50:16.197 DEBUG:boto:auth_path=/hiiqos-13/dummy 2023-07-28T23:50:16.197 DEBUG:boto:path=/hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:50:16.198 DEBUG:boto:auth_path=/hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:50:16.198 DEBUG:boto:Method: GET 2023-07-28T23:50:16.199 DEBUG:boto:Path: /hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:50:16.199 DEBUG:boto:Data: 2023-07-28T23:50:16.199 DEBUG:boto:Headers: {} 2023-07-28T23:50:16.200 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:50:16.200 DEBUG:boto:Port: 8000 2023-07-28T23:50:16.201 DEBUG:boto:Params: {} 2023-07-28T23:50:16.202 DEBUG:boto:Token: None 2023-07-28T23:50:16.202 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:50:16.203 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qt+mIkStEsRHnzOWTiuMZL7UgYk= 2023-07-28T23:50:16.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qt+mIkStEsRHnzOWTiuMZL7UgYk=', 'Content-Length': '0'} 2023-07-28T23:50:16.205 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:49:00 GMT'), ('x-amz-version-id', '2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e983c7da6f605652-0064c45438-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.206 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:50:16.206 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:50:16.207 DEBUG:boto:path=/hiiqos-13/?prefix=dummy 2023-07-28T23:50:16.207 DEBUG:boto:auth_path=/hiiqos-13/?prefix=dummy 2023-07-28T23:50:16.207 DEBUG:boto:Method: GET 2023-07-28T23:50:16.208 DEBUG:boto:Path: /hiiqos-13/?prefix=dummy 2023-07-28T23:50:16.208 DEBUG:boto:Data: 2023-07-28T23:50:16.208 DEBUG:boto:Headers: {} 2023-07-28T23:50:16.209 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:50:16.209 DEBUG:boto:Port: 8001 2023-07-28T23:50:16.209 DEBUG:boto:Params: {} 2023-07-28T23:50:16.210 DEBUG:boto:Token: None 2023-07-28T23:50:16.210 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/ 2023-07-28T23:50:16.211 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dcV/CkCn0AEOZr+GidOeUgkY4wQ= 2023-07-28T23:50:16.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dcV/CkCn0AEOZr+GidOeUgkY4wQ=', 'Content-Length': '0'} 2023-07-28T23:50:16.213 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f60deb6dbf832652-0064c45438-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.256 DEBUG:boto:b'hiiqos-13dummy1000falsedummy2023-07-28T23:49:00.821Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:50:16.257 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:50:16.258 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:50:16.258 DEBUG:boto:path=/hiiqos-13/?prefix=dummy 2023-07-28T23:50:16.259 DEBUG:boto:auth_path=/hiiqos-13/?prefix=dummy 2023-07-28T23:50:16.259 DEBUG:boto:Method: GET 2023-07-28T23:50:16.259 DEBUG:boto:Path: /hiiqos-13/?prefix=dummy 2023-07-28T23:50:16.260 DEBUG:boto:Data: 2023-07-28T23:50:16.260 DEBUG:boto:Headers: {} 2023-07-28T23:50:16.260 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:50:16.261 DEBUG:boto:Port: 8000 2023-07-28T23:50:16.261 DEBUG:boto:Params: {} 2023-07-28T23:50:16.262 DEBUG:boto:Token: None 2023-07-28T23:50:16.262 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/ 2023-07-28T23:50:16.262 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dcV/CkCn0AEOZr+GidOeUgkY4wQ= 2023-07-28T23:50:16.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dcV/CkCn0AEOZr+GidOeUgkY4wQ=', 'Content-Length': '0'} 2023-07-28T23:50:16.265 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3dabde86c6caaf9-0064c45438-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.266 DEBUG:boto:b'hiiqos-13dummy1000falsedummy2023-07-28T23:49:00.821Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:50:16.267 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-13 zones={a2, a1} 2023-07-28T23:50:16.267 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:50:16.267 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:50:16.268 DEBUG:boto:Method: HEAD 2023-07-28T23:50:16.268 DEBUG:boto:Path: /hiiqos-13/ 2023-07-28T23:50:16.269 DEBUG:boto:Data: 2023-07-28T23:50:16.269 DEBUG:boto:Headers: {} 2023-07-28T23:50:16.269 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:50:16.270 DEBUG:boto:Port: 8000 2023-07-28T23:50:16.270 DEBUG:boto:Params: {} 2023-07-28T23:50:16.270 DEBUG:boto:Token: None 2023-07-28T23:50:16.271 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/ 2023-07-28T23:50:16.271 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus= 2023-07-28T23:50:16.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus=', 'Content-Length': '0'} 2023-07-28T23:50:16.273 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c357877e41c960c-0064c45438-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.274 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:50:16.274 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:50:16.275 DEBUG:boto:Method: PUT 2023-07-28T23:50:16.275 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:50:16.275 DEBUG:boto:Data: 2023-07-28T23:50:16.276 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-07-28T23:50:16.276 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:50:16.276 DEBUG:boto:Port: 8000 2023-07-28T23:50:16.277 DEBUG:boto:Params: {} 2023-07-28T23:50:16.277 DEBUG:boto:Token: None 2023-07-28T23:50:16.277 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/obj 2023-07-28T23:50:16.278 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mKlBOowpz+rxIG8DRDvZM7+EUZU= 2023-07-28T23:50:16.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mKlBOowpz+rxIG8DRDvZM7+EUZU='} 2023-07-28T23:50:16.324 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'nQT3SHUTnGGHoRE.HK4699V4TWQJbCA'), ('x-amz-request-id', 'tx00000d355193b2285e550-0064c45438-1117-a1'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.325 DEBUG:rgw_multi.tests:zone=a1 version=nQT3SHUTnGGHoRE.HK4699V4TWQJbCA 2023-07-28T23:50:16.326 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:50:16.326 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:50:16.327 DEBUG:boto:Method: HEAD 2023-07-28T23:50:16.327 DEBUG:boto:Path: /hiiqos-13/ 2023-07-28T23:50:16.327 DEBUG:boto:Data: 2023-07-28T23:50:16.328 DEBUG:boto:Headers: {} 2023-07-28T23:50:16.328 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:50:16.328 DEBUG:boto:Port: 8001 2023-07-28T23:50:16.329 DEBUG:boto:Params: {} 2023-07-28T23:50:16.329 DEBUG:boto:Token: None 2023-07-28T23:50:16.330 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/ 2023-07-28T23:50:16.330 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus= 2023-07-28T23:50:16.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus=', 'Content-Length': '0'} 2023-07-28T23:50:16.333 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008027ce2bea042a11-0064c45438-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.333 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:50:16.334 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:50:16.334 DEBUG:boto:Method: PUT 2023-07-28T23:50:16.335 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:50:16.335 DEBUG:boto:Data: 2023-07-28T23:50:16.335 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-07-28T23:50:16.336 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:50:16.337 DEBUG:boto:Port: 8001 2023-07-28T23:50:16.338 DEBUG:boto:Params: {} 2023-07-28T23:50:16.339 DEBUG:boto:Token: None 2023-07-28T23:50:16.340 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/obj 2023-07-28T23:50:16.341 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mKlBOowpz+rxIG8DRDvZM7+EUZU= 2023-07-28T23:50:16.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mKlBOowpz+rxIG8DRDvZM7+EUZU='} 2023-07-28T23:50:16.384 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'OYqlT9m6zvLoRrfJK6xkLWsj3aXU2MG'), ('x-amz-request-id', 'tx00000e883421660c65ea3-0064c45438-1113-a2'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.385 DEBUG:rgw_multi.tests:zone=a2 version=OYqlT9m6zvLoRrfJK6xkLWsj3aXU2MG 2023-07-28T23:50:16.385 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:50:16.386 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:50:16.386 DEBUG:boto:Method: HEAD 2023-07-28T23:50:16.386 DEBUG:boto:Path: /hiiqos-13/ 2023-07-28T23:50:16.387 DEBUG:boto:Data: 2023-07-28T23:50:16.387 DEBUG:boto:Headers: {} 2023-07-28T23:50:16.387 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:50:16.388 DEBUG:boto:Port: 8000 2023-07-28T23:50:16.388 DEBUG:boto:Params: {} 2023-07-28T23:50:16.388 DEBUG:boto:Token: None 2023-07-28T23:50:16.389 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/ 2023-07-28T23:50:16.389 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus= 2023-07-28T23:50:16.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus=', 'Content-Length': '0'} 2023-07-28T23:50:16.392 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003edabce199e30319-0064c45438-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.393 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:50:16.393 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:50:16.393 DEBUG:boto:Method: PUT 2023-07-28T23:50:16.394 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:50:16.394 DEBUG:boto:Data: 2023-07-28T23:50:16.394 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-07-28T23:50:16.395 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:50:16.395 DEBUG:boto:Port: 8000 2023-07-28T23:50:16.395 DEBUG:boto:Params: {} 2023-07-28T23:50:16.396 DEBUG:boto:Token: None 2023-07-28T23:50:16.396 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/obj 2023-07-28T23:50:16.397 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mKlBOowpz+rxIG8DRDvZM7+EUZU= 2023-07-28T23:50:16.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mKlBOowpz+rxIG8DRDvZM7+EUZU='} 2023-07-28T23:50:16.406 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'iRoeqa0SUPCZisrafCoZycv1QQITmgh'), ('x-amz-request-id', 'tx000003d064aaadc07d852-0064c45438-1117-a1'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.407 DEBUG:rgw_multi.tests:zone=a1 version=iRoeqa0SUPCZisrafCoZycv1QQITmgh 2023-07-28T23:50:16.407 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:50:16.408 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:50:16.408 DEBUG:boto:Method: HEAD 2023-07-28T23:50:16.408 DEBUG:boto:Path: /hiiqos-13/ 2023-07-28T23:50:16.409 DEBUG:boto:Data: 2023-07-28T23:50:16.409 DEBUG:boto:Headers: {} 2023-07-28T23:50:16.409 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:50:16.410 DEBUG:boto:Port: 8001 2023-07-28T23:50:16.410 DEBUG:boto:Params: {} 2023-07-28T23:50:16.410 DEBUG:boto:Token: None 2023-07-28T23:50:16.411 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/ 2023-07-28T23:50:16.411 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus= 2023-07-28T23:50:16.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus=', 'Content-Length': '0'} 2023-07-28T23:50:16.414 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b4c0a019519e1a77-0064c45438-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.415 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:50:16.415 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:50:16.416 DEBUG:boto:Method: PUT 2023-07-28T23:50:16.417 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:50:16.418 DEBUG:boto:Data: 2023-07-28T23:50:16.419 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-07-28T23:50:16.419 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:50:16.420 DEBUG:boto:Port: 8001 2023-07-28T23:50:16.420 DEBUG:boto:Params: {} 2023-07-28T23:50:16.420 DEBUG:boto:Token: None 2023-07-28T23:50:16.421 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/obj 2023-07-28T23:50:16.421 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mKlBOowpz+rxIG8DRDvZM7+EUZU= 2023-07-28T23:50:16.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mKlBOowpz+rxIG8DRDvZM7+EUZU='} 2023-07-28T23:50:16.432 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '58wMXe18IrbR7RSdRvF4V8KhCCY6CR.'), ('x-amz-request-id', 'tx00000108dd0e35ae31ee4-0064c45438-1113-a2'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.433 DEBUG:rgw_multi.tests:zone=a2 version=58wMXe18IrbR7RSdRvF4V8KhCCY6CR. 2023-07-28T23:50:16.433 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:50:16.433 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:50:16.434 DEBUG:boto:Method: HEAD 2023-07-28T23:50:16.434 DEBUG:boto:Path: /hiiqos-13/ 2023-07-28T23:50:16.434 DEBUG:boto:Data: 2023-07-28T23:50:16.435 DEBUG:boto:Headers: {} 2023-07-28T23:50:16.435 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:50:16.435 DEBUG:boto:Port: 8000 2023-07-28T23:50:16.436 DEBUG:boto:Params: {} 2023-07-28T23:50:16.436 DEBUG:boto:Token: None 2023-07-28T23:50:16.436 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/ 2023-07-28T23:50:16.437 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus= 2023-07-28T23:50:16.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus=', 'Content-Length': '0'} 2023-07-28T23:50:16.440 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000713d2a9d087e64c7-0064c45438-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.440 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:50:16.441 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:50:16.441 DEBUG:boto:Method: PUT 2023-07-28T23:50:16.441 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:50:16.442 DEBUG:boto:Data: 2023-07-28T23:50:16.442 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-07-28T23:50:16.442 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:50:16.443 DEBUG:boto:Port: 8000 2023-07-28T23:50:16.443 DEBUG:boto:Params: {} 2023-07-28T23:50:16.443 DEBUG:boto:Token: None 2023-07-28T23:50:16.444 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/obj 2023-07-28T23:50:16.444 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mKlBOowpz+rxIG8DRDvZM7+EUZU= 2023-07-28T23:50:16.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mKlBOowpz+rxIG8DRDvZM7+EUZU='} 2023-07-28T23:50:16.455 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'KF9pysOWS.qePO4W3-Ux03VJiRC9ph7'), ('x-amz-request-id', 'tx000000be4607c4d63a5b0-0064c45438-1117-a1'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.456 DEBUG:rgw_multi.tests:zone=a1 version=KF9pysOWS.qePO4W3-Ux03VJiRC9ph7 2023-07-28T23:50:16.456 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:50:16.456 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:50:16.457 DEBUG:boto:Method: HEAD 2023-07-28T23:50:16.457 DEBUG:boto:Path: /hiiqos-13/ 2023-07-28T23:50:16.457 DEBUG:boto:Data: 2023-07-28T23:50:16.458 DEBUG:boto:Headers: {} 2023-07-28T23:50:16.458 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:50:16.458 DEBUG:boto:Port: 8001 2023-07-28T23:50:16.459 DEBUG:boto:Params: {} 2023-07-28T23:50:16.459 DEBUG:boto:Token: None 2023-07-28T23:50:16.459 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/ 2023-07-28T23:50:16.460 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus= 2023-07-28T23:50:16.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus=', 'Content-Length': '0'} 2023-07-28T23:50:16.463 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000091fa998bef0567bf-0064c45438-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.463 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:50:16.464 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:50:16.464 DEBUG:boto:Method: PUT 2023-07-28T23:50:16.464 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:50:16.465 DEBUG:boto:Data: 2023-07-28T23:50:16.465 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-07-28T23:50:16.466 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:50:16.466 DEBUG:boto:Port: 8001 2023-07-28T23:50:16.467 DEBUG:boto:Params: {} 2023-07-28T23:50:16.467 DEBUG:boto:Token: None 2023-07-28T23:50:16.467 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/obj 2023-07-28T23:50:16.468 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mKlBOowpz+rxIG8DRDvZM7+EUZU= 2023-07-28T23:50:16.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mKlBOowpz+rxIG8DRDvZM7+EUZU='} 2023-07-28T23:50:16.512 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '22KG5jfO2TftvgPgeJvtwXX52mGQhV1'), ('x-amz-request-id', 'tx00000bb36bac69f689fbf-0064c45438-1113-a2'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.513 DEBUG:rgw_multi.tests:zone=a2 version=22KG5jfO2TftvgPgeJvtwXX52mGQhV1 2023-07-28T23:50:16.514 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:50:16.514 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:50:16.515 DEBUG:boto:Method: HEAD 2023-07-28T23:50:16.516 DEBUG:boto:Path: /hiiqos-13/ 2023-07-28T23:50:16.517 DEBUG:boto:Data: 2023-07-28T23:50:16.518 DEBUG:boto:Headers: {} 2023-07-28T23:50:16.518 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:50:16.519 DEBUG:boto:Port: 8000 2023-07-28T23:50:16.519 DEBUG:boto:Params: {} 2023-07-28T23:50:16.520 DEBUG:boto:Token: None 2023-07-28T23:50:16.520 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/ 2023-07-28T23:50:16.521 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus= 2023-07-28T23:50:16.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus=', 'Content-Length': '0'} 2023-07-28T23:50:16.523 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bede251dd937e5a4-0064c45438-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.524 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:50:16.524 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:50:16.525 DEBUG:boto:Method: PUT 2023-07-28T23:50:16.525 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:50:16.525 DEBUG:boto:Data: 2023-07-28T23:50:16.526 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-07-28T23:50:16.526 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:50:16.527 DEBUG:boto:Port: 8000 2023-07-28T23:50:16.527 DEBUG:boto:Params: {} 2023-07-28T23:50:16.527 DEBUG:boto:Token: None 2023-07-28T23:50:16.528 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/obj 2023-07-28T23:50:16.529 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mKlBOowpz+rxIG8DRDvZM7+EUZU= 2023-07-28T23:50:16.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mKlBOowpz+rxIG8DRDvZM7+EUZU='} 2023-07-28T23:50:16.552 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'fxsCLuGqt0tjCmqlJgclHqGWnHEcxLA'), ('x-amz-request-id', 'tx000001193e71c4404b76b-0064c45438-1117-a1'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.553 DEBUG:rgw_multi.tests:zone=a1 version=fxsCLuGqt0tjCmqlJgclHqGWnHEcxLA 2023-07-28T23:50:16.553 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:50:16.554 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:50:16.554 DEBUG:boto:Method: HEAD 2023-07-28T23:50:16.555 DEBUG:boto:Path: /hiiqos-13/ 2023-07-28T23:50:16.555 DEBUG:boto:Data: 2023-07-28T23:50:16.556 DEBUG:boto:Headers: {} 2023-07-28T23:50:16.556 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:50:16.557 DEBUG:boto:Port: 8001 2023-07-28T23:50:16.558 DEBUG:boto:Params: {} 2023-07-28T23:50:16.558 DEBUG:boto:Token: None 2023-07-28T23:50:16.559 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/ 2023-07-28T23:50:16.560 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus= 2023-07-28T23:50:16.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus=', 'Content-Length': '0'} 2023-07-28T23:50:16.563 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000584ec755d6c80209-0064c45438-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.564 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:50:16.564 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:50:16.565 DEBUG:boto:Method: PUT 2023-07-28T23:50:16.565 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:50:16.566 DEBUG:boto:Data: 2023-07-28T23:50:16.566 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-07-28T23:50:16.566 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:50:16.567 DEBUG:boto:Port: 8001 2023-07-28T23:50:16.567 DEBUG:boto:Params: {} 2023-07-28T23:50:16.567 DEBUG:boto:Token: None 2023-07-28T23:50:16.568 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/obj 2023-07-28T23:50:16.568 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mKlBOowpz+rxIG8DRDvZM7+EUZU= 2023-07-28T23:50:16.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mKlBOowpz+rxIG8DRDvZM7+EUZU='} 2023-07-28T23:50:16.580 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'MMa4nLQC3W82FeUky1Sq9XW9uFjpW6Q'), ('x-amz-request-id', 'tx000003f08aaf7a7ff17a4-0064c45438-1113-a2'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.581 DEBUG:rgw_multi.tests:zone=a2 version=MMa4nLQC3W82FeUky1Sq9XW9uFjpW6Q 2023-07-28T23:50:16.581 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:50:16.581 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:50:16.582 DEBUG:boto:Method: HEAD 2023-07-28T23:50:16.582 DEBUG:boto:Path: /hiiqos-13/ 2023-07-28T23:50:16.582 DEBUG:boto:Data: 2023-07-28T23:50:16.583 DEBUG:boto:Headers: {} 2023-07-28T23:50:16.583 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:50:16.583 DEBUG:boto:Port: 8000 2023-07-28T23:50:16.584 DEBUG:boto:Params: {} 2023-07-28T23:50:16.584 DEBUG:boto:Token: None 2023-07-28T23:50:16.584 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/ 2023-07-28T23:50:16.585 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus= 2023-07-28T23:50:16.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus=', 'Content-Length': '0'} 2023-07-28T23:50:16.588 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b214250048c4b77f-0064c45438-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.588 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:50:16.589 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:50:16.589 DEBUG:boto:Method: PUT 2023-07-28T23:50:16.589 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:50:16.590 DEBUG:boto:Data: 2023-07-28T23:50:16.590 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-07-28T23:50:16.590 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:50:16.591 DEBUG:boto:Port: 8000 2023-07-28T23:50:16.591 DEBUG:boto:Params: {} 2023-07-28T23:50:16.591 DEBUG:boto:Token: None 2023-07-28T23:50:16.592 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/obj 2023-07-28T23:50:16.592 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mKlBOowpz+rxIG8DRDvZM7+EUZU= 2023-07-28T23:50:16.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mKlBOowpz+rxIG8DRDvZM7+EUZU='} 2023-07-28T23:50:16.603 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'l2NN5Bw32G2ViK4cHi-XORehcdjCcxD'), ('x-amz-request-id', 'tx00000d997b427070d77f4-0064c45438-1117-a1'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.603 DEBUG:rgw_multi.tests:zone=a1 version=l2NN5Bw32G2ViK4cHi-XORehcdjCcxD 2023-07-28T23:50:16.604 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:50:16.604 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:50:16.605 DEBUG:boto:Method: HEAD 2023-07-28T23:50:16.605 DEBUG:boto:Path: /hiiqos-13/ 2023-07-28T23:50:16.605 DEBUG:boto:Data: 2023-07-28T23:50:16.606 DEBUG:boto:Headers: {} 2023-07-28T23:50:16.606 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:50:16.606 DEBUG:boto:Port: 8001 2023-07-28T23:50:16.607 DEBUG:boto:Params: {} 2023-07-28T23:50:16.608 DEBUG:boto:Token: None 2023-07-28T23:50:16.608 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/ 2023-07-28T23:50:16.609 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus= 2023-07-28T23:50:16.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus=', 'Content-Length': '0'} 2023-07-28T23:50:16.611 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000564b13365c89d05d-0064c45438-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.612 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:50:16.612 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:50:16.612 DEBUG:boto:Method: PUT 2023-07-28T23:50:16.613 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:50:16.613 DEBUG:boto:Data: 2023-07-28T23:50:16.613 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-07-28T23:50:16.614 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:50:16.614 DEBUG:boto:Port: 8001 2023-07-28T23:50:16.614 DEBUG:boto:Params: {} 2023-07-28T23:50:16.615 DEBUG:boto:Token: None 2023-07-28T23:50:16.615 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/obj 2023-07-28T23:50:16.615 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mKlBOowpz+rxIG8DRDvZM7+EUZU= 2023-07-28T23:50:16.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mKlBOowpz+rxIG8DRDvZM7+EUZU='} 2023-07-28T23:50:16.625 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'fqZHgkQ0VnkgGOiotHJVpzmOgEM1PNX'), ('x-amz-request-id', 'tx000008e096b4a60a28045-0064c45438-1113-a2'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.625 DEBUG:rgw_multi.tests:zone=a2 version=fqZHgkQ0VnkgGOiotHJVpzmOgEM1PNX 2023-07-28T23:50:16.626 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:50:16.626 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:50:16.627 DEBUG:boto:Method: HEAD 2023-07-28T23:50:16.627 DEBUG:boto:Path: /hiiqos-13/ 2023-07-28T23:50:16.628 DEBUG:boto:Data: 2023-07-28T23:50:16.628 DEBUG:boto:Headers: {} 2023-07-28T23:50:16.629 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:50:16.629 DEBUG:boto:Port: 8000 2023-07-28T23:50:16.629 DEBUG:boto:Params: {} 2023-07-28T23:50:16.630 DEBUG:boto:Token: None 2023-07-28T23:50:16.630 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/ 2023-07-28T23:50:16.630 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus= 2023-07-28T23:50:16.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus=', 'Content-Length': '0'} 2023-07-28T23:50:16.633 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a7adddb770b9b6ba-0064c45438-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.634 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:50:16.635 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:50:16.636 DEBUG:boto:Method: PUT 2023-07-28T23:50:16.637 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:50:16.637 DEBUG:boto:Data: 2023-07-28T23:50:16.638 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-07-28T23:50:16.639 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:50:16.639 DEBUG:boto:Port: 8000 2023-07-28T23:50:16.640 DEBUG:boto:Params: {} 2023-07-28T23:50:16.641 DEBUG:boto:Token: None 2023-07-28T23:50:16.642 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/obj 2023-07-28T23:50:16.643 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mKlBOowpz+rxIG8DRDvZM7+EUZU= 2023-07-28T23:50:16.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mKlBOowpz+rxIG8DRDvZM7+EUZU='} 2023-07-28T23:50:16.689 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '9hfXRNs96rGpisQpVYKBPxZpFPAeL47'), ('x-amz-request-id', 'tx000002261be6074acce4f-0064c45438-1117-a1'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.690 DEBUG:rgw_multi.tests:zone=a1 version=9hfXRNs96rGpisQpVYKBPxZpFPAeL47 2023-07-28T23:50:16.690 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:50:16.691 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:50:16.692 DEBUG:boto:Method: HEAD 2023-07-28T23:50:16.693 DEBUG:boto:Path: /hiiqos-13/ 2023-07-28T23:50:16.694 DEBUG:boto:Data: 2023-07-28T23:50:16.695 DEBUG:boto:Headers: {} 2023-07-28T23:50:16.695 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:50:16.695 DEBUG:boto:Port: 8001 2023-07-28T23:50:16.696 DEBUG:boto:Params: {} 2023-07-28T23:50:16.696 DEBUG:boto:Token: None 2023-07-28T23:50:16.697 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/ 2023-07-28T23:50:16.697 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus= 2023-07-28T23:50:16.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus=', 'Content-Length': '0'} 2023-07-28T23:50:16.700 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000030b7de6aabf77f8d-0064c45438-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.701 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:50:16.701 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:50:16.702 DEBUG:boto:Method: PUT 2023-07-28T23:50:16.702 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:50:16.702 DEBUG:boto:Data: 2023-07-28T23:50:16.703 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-07-28T23:50:16.703 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:50:16.703 DEBUG:boto:Port: 8001 2023-07-28T23:50:16.704 DEBUG:boto:Params: {} 2023-07-28T23:50:16.704 DEBUG:boto:Token: None 2023-07-28T23:50:16.705 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/obj 2023-07-28T23:50:16.705 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mKlBOowpz+rxIG8DRDvZM7+EUZU= 2023-07-28T23:50:16.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mKlBOowpz+rxIG8DRDvZM7+EUZU='} 2023-07-28T23:50:16.715 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'KW4erdPcpVPNf4hlrOZysjOAxTTzNpZ'), ('x-amz-request-id', 'tx000005b64744f528fb1d2-0064c45438-1113-a2'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.715 DEBUG:rgw_multi.tests:zone=a2 version=KW4erdPcpVPNf4hlrOZysjOAxTTzNpZ 2023-07-28T23:50:16.716 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:50:16.716 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:50:16.717 DEBUG:boto:Method: HEAD 2023-07-28T23:50:16.717 DEBUG:boto:Path: /hiiqos-13/ 2023-07-28T23:50:16.718 DEBUG:boto:Data: 2023-07-28T23:50:16.718 DEBUG:boto:Headers: {} 2023-07-28T23:50:16.718 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:50:16.719 DEBUG:boto:Port: 8000 2023-07-28T23:50:16.719 DEBUG:boto:Params: {} 2023-07-28T23:50:16.719 DEBUG:boto:Token: None 2023-07-28T23:50:16.720 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/ 2023-07-28T23:50:16.720 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus= 2023-07-28T23:50:16.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus=', 'Content-Length': '0'} 2023-07-28T23:50:16.722 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000502aeaa3227a94f1-0064c45438-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.723 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:50:16.723 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:50:16.724 DEBUG:boto:Method: PUT 2023-07-28T23:50:16.724 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:50:16.725 DEBUG:boto:Data: 2023-07-28T23:50:16.725 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-07-28T23:50:16.725 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:50:16.726 DEBUG:boto:Port: 8000 2023-07-28T23:50:16.726 DEBUG:boto:Params: {} 2023-07-28T23:50:16.727 DEBUG:boto:Token: None 2023-07-28T23:50:16.728 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/obj 2023-07-28T23:50:16.728 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mKlBOowpz+rxIG8DRDvZM7+EUZU= 2023-07-28T23:50:16.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mKlBOowpz+rxIG8DRDvZM7+EUZU='} 2023-07-28T23:50:16.738 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '2N3xN1haVyocj8z79kCS3UT0nbjXj6Z'), ('x-amz-request-id', 'tx000000949f6410864bcf7-0064c45438-1117-a1'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.739 DEBUG:rgw_multi.tests:zone=a1 version=2N3xN1haVyocj8z79kCS3UT0nbjXj6Z 2023-07-28T23:50:16.739 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:50:16.740 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:50:16.740 DEBUG:boto:Method: HEAD 2023-07-28T23:50:16.741 DEBUG:boto:Path: /hiiqos-13/ 2023-07-28T23:50:16.742 DEBUG:boto:Data: 2023-07-28T23:50:16.742 DEBUG:boto:Headers: {} 2023-07-28T23:50:16.743 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:50:16.743 DEBUG:boto:Port: 8001 2023-07-28T23:50:16.744 DEBUG:boto:Params: {} 2023-07-28T23:50:16.744 DEBUG:boto:Token: None 2023-07-28T23:50:16.745 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/ 2023-07-28T23:50:16.745 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus= 2023-07-28T23:50:16.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus=', 'Content-Length': '0'} 2023-07-28T23:50:16.748 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a16368e1856a686a-0064c45438-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.748 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:50:16.749 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:50:16.749 DEBUG:boto:Method: PUT 2023-07-28T23:50:16.750 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:50:16.750 DEBUG:boto:Data: 2023-07-28T23:50:16.750 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-07-28T23:50:16.751 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:50:16.751 DEBUG:boto:Port: 8001 2023-07-28T23:50:16.751 DEBUG:boto:Params: {} 2023-07-28T23:50:16.752 DEBUG:boto:Token: None 2023-07-28T23:50:16.752 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/obj 2023-07-28T23:50:16.752 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mKlBOowpz+rxIG8DRDvZM7+EUZU= 2023-07-28T23:50:16.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mKlBOowpz+rxIG8DRDvZM7+EUZU='} 2023-07-28T23:50:16.762 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'eu89nyTHw35ZG459SndVR8LnRvJs7VP'), ('x-amz-request-id', 'tx00000994904732c64cca9-0064c45438-1113-a2'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.762 DEBUG:rgw_multi.tests:zone=a2 version=eu89nyTHw35ZG459SndVR8LnRvJs7VP 2023-07-28T23:50:16.763 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:50:16.763 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:50:16.763 DEBUG:boto:Method: HEAD 2023-07-28T23:50:16.764 DEBUG:boto:Path: /hiiqos-13/ 2023-07-28T23:50:16.764 DEBUG:boto:Data: 2023-07-28T23:50:16.764 DEBUG:boto:Headers: {} 2023-07-28T23:50:16.765 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:50:16.765 DEBUG:boto:Port: 8000 2023-07-28T23:50:16.765 DEBUG:boto:Params: {} 2023-07-28T23:50:16.766 DEBUG:boto:Token: None 2023-07-28T23:50:16.766 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/ 2023-07-28T23:50:16.766 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus= 2023-07-28T23:50:16.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus=', 'Content-Length': '0'} 2023-07-28T23:50:16.769 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007be0b0ef497f7b0a-0064c45438-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.769 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:50:16.770 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:50:16.770 DEBUG:boto:Method: PUT 2023-07-28T23:50:16.771 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:50:16.771 DEBUG:boto:Data: 2023-07-28T23:50:16.772 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-07-28T23:50:16.772 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:50:16.772 DEBUG:boto:Port: 8000 2023-07-28T23:50:16.773 DEBUG:boto:Params: {} 2023-07-28T23:50:16.773 DEBUG:boto:Token: None 2023-07-28T23:50:16.774 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/obj 2023-07-28T23:50:16.774 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mKlBOowpz+rxIG8DRDvZM7+EUZU= 2023-07-28T23:50:16.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mKlBOowpz+rxIG8DRDvZM7+EUZU='} 2023-07-28T23:50:16.816 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'WxYcV1Cw2jkGBm-hoDmeDENoWMoiniX'), ('x-amz-request-id', 'tx000004a6e6b2404bfb759-0064c45438-1117-a1'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.817 DEBUG:rgw_multi.tests:zone=a1 version=WxYcV1Cw2jkGBm-hoDmeDENoWMoiniX 2023-07-28T23:50:16.817 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:50:16.818 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:50:16.818 DEBUG:boto:Method: HEAD 2023-07-28T23:50:16.818 DEBUG:boto:Path: /hiiqos-13/ 2023-07-28T23:50:16.819 DEBUG:boto:Data: 2023-07-28T23:50:16.819 DEBUG:boto:Headers: {} 2023-07-28T23:50:16.819 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:50:16.820 DEBUG:boto:Port: 8001 2023-07-28T23:50:16.820 DEBUG:boto:Params: {} 2023-07-28T23:50:16.820 DEBUG:boto:Token: None 2023-07-28T23:50:16.821 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/ 2023-07-28T23:50:16.821 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus= 2023-07-28T23:50:16.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus=', 'Content-Length': '0'} 2023-07-28T23:50:16.824 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000992f8679198f32a5-0064c45438-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.824 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:50:16.825 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:50:16.825 DEBUG:boto:Method: PUT 2023-07-28T23:50:16.825 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:50:16.826 DEBUG:boto:Data: 2023-07-28T23:50:16.826 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-07-28T23:50:16.826 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:50:16.827 DEBUG:boto:Port: 8001 2023-07-28T23:50:16.827 DEBUG:boto:Params: {} 2023-07-28T23:50:16.827 DEBUG:boto:Token: None 2023-07-28T23:50:16.828 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/obj 2023-07-28T23:50:16.828 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mKlBOowpz+rxIG8DRDvZM7+EUZU= 2023-07-28T23:50:16.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mKlBOowpz+rxIG8DRDvZM7+EUZU='} 2023-07-28T23:50:16.837 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'KZjcyesGbGSQE5pZSDDLRJ6rT-Mqm8x'), ('x-amz-request-id', 'tx00000922784b7f858e6a5-0064c45438-1113-a2'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.838 DEBUG:rgw_multi.tests:zone=a2 version=KZjcyesGbGSQE5pZSDDLRJ6rT-Mqm8x 2023-07-28T23:50:16.838 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:50:16.838 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:50:16.839 DEBUG:boto:Method: HEAD 2023-07-28T23:50:16.839 DEBUG:boto:Path: /hiiqos-13/ 2023-07-28T23:50:16.839 DEBUG:boto:Data: 2023-07-28T23:50:16.839 DEBUG:boto:Headers: {} 2023-07-28T23:50:16.840 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:50:16.840 DEBUG:boto:Port: 8000 2023-07-28T23:50:16.841 DEBUG:boto:Params: {} 2023-07-28T23:50:16.841 DEBUG:boto:Token: None 2023-07-28T23:50:16.841 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/ 2023-07-28T23:50:16.842 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus= 2023-07-28T23:50:16.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus=', 'Content-Length': '0'} 2023-07-28T23:50:16.844 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e0670bcd98d9dab-0064c45438-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.845 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:50:16.845 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:50:16.846 DEBUG:boto:Method: PUT 2023-07-28T23:50:16.846 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:50:16.846 DEBUG:boto:Data: 2023-07-28T23:50:16.847 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-07-28T23:50:16.847 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:50:16.847 DEBUG:boto:Port: 8000 2023-07-28T23:50:16.848 DEBUG:boto:Params: {} 2023-07-28T23:50:16.848 DEBUG:boto:Token: None 2023-07-28T23:50:16.848 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/obj 2023-07-28T23:50:16.849 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mKlBOowpz+rxIG8DRDvZM7+EUZU= 2023-07-28T23:50:16.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mKlBOowpz+rxIG8DRDvZM7+EUZU='} 2023-07-28T23:50:16.857 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Fg82BDw2qebn-TlTZEmuWfNDUSVLkvC'), ('x-amz-request-id', 'tx00000c557db27008da92f-0064c45438-1117-a1'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.858 DEBUG:rgw_multi.tests:zone=a1 version=Fg82BDw2qebn-TlTZEmuWfNDUSVLkvC 2023-07-28T23:50:16.858 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:50:16.859 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:50:16.859 DEBUG:boto:Method: HEAD 2023-07-28T23:50:16.859 DEBUG:boto:Path: /hiiqos-13/ 2023-07-28T23:50:16.860 DEBUG:boto:Data: 2023-07-28T23:50:16.860 DEBUG:boto:Headers: {} 2023-07-28T23:50:16.860 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:50:16.861 DEBUG:boto:Port: 8001 2023-07-28T23:50:16.861 DEBUG:boto:Params: {} 2023-07-28T23:50:16.861 DEBUG:boto:Token: None 2023-07-28T23:50:16.862 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/ 2023-07-28T23:50:16.862 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus= 2023-07-28T23:50:16.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus=', 'Content-Length': '0'} 2023-07-28T23:50:16.865 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e62f8168d8f79ff3-0064c45438-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.865 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:50:16.866 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:50:16.866 DEBUG:boto:Method: PUT 2023-07-28T23:50:16.866 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:50:16.867 DEBUG:boto:Data: 2023-07-28T23:50:16.867 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-07-28T23:50:16.868 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:50:16.869 DEBUG:boto:Port: 8001 2023-07-28T23:50:16.869 DEBUG:boto:Params: {} 2023-07-28T23:50:16.870 DEBUG:boto:Token: None 2023-07-28T23:50:16.871 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/obj 2023-07-28T23:50:16.872 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mKlBOowpz+rxIG8DRDvZM7+EUZU= 2023-07-28T23:50:16.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mKlBOowpz+rxIG8DRDvZM7+EUZU='} 2023-07-28T23:50:16.920 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'z7auNlW8oH1FJvaHlG2v6quhJD.17R1'), ('x-amz-request-id', 'tx00000b40a7a12e781c85b-0064c45438-1113-a2'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.921 DEBUG:rgw_multi.tests:zone=a2 version=z7auNlW8oH1FJvaHlG2v6quhJD.17R1 2023-07-28T23:50:16.921 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:50:16.922 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:50:16.922 DEBUG:boto:Method: HEAD 2023-07-28T23:50:16.922 DEBUG:boto:Path: /hiiqos-13/ 2023-07-28T23:50:16.923 DEBUG:boto:Data: 2023-07-28T23:50:16.923 DEBUG:boto:Headers: {} 2023-07-28T23:50:16.923 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:50:16.924 DEBUG:boto:Port: 8000 2023-07-28T23:50:16.924 DEBUG:boto:Params: {} 2023-07-28T23:50:16.924 DEBUG:boto:Token: None 2023-07-28T23:50:16.925 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/ 2023-07-28T23:50:16.925 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus= 2023-07-28T23:50:16.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus=', 'Content-Length': '0'} 2023-07-28T23:50:16.928 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000075eb707d90a07df9-0064c45438-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.928 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:50:16.929 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:50:16.929 DEBUG:boto:Method: PUT 2023-07-28T23:50:16.929 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:50:16.930 DEBUG:boto:Data: 2023-07-28T23:50:16.930 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-07-28T23:50:16.930 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:50:16.931 DEBUG:boto:Port: 8000 2023-07-28T23:50:16.931 DEBUG:boto:Params: {} 2023-07-28T23:50:16.931 DEBUG:boto:Token: None 2023-07-28T23:50:16.932 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/obj 2023-07-28T23:50:16.932 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mKlBOowpz+rxIG8DRDvZM7+EUZU= 2023-07-28T23:50:16.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mKlBOowpz+rxIG8DRDvZM7+EUZU='} 2023-07-28T23:50:16.943 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-amz-request-id', 'tx0000099e3286c395b9e41-0064c45438-1117-a1'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.944 DEBUG:rgw_multi.tests:zone=a1 version=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:50:16.944 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:50:16.944 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:50:16.945 DEBUG:boto:Method: HEAD 2023-07-28T23:50:16.945 DEBUG:boto:Path: /hiiqos-13/ 2023-07-28T23:50:16.945 DEBUG:boto:Data: 2023-07-28T23:50:16.946 DEBUG:boto:Headers: {} 2023-07-28T23:50:16.946 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:50:16.946 DEBUG:boto:Port: 8001 2023-07-28T23:50:16.947 DEBUG:boto:Params: {} 2023-07-28T23:50:16.947 DEBUG:boto:Token: None 2023-07-28T23:50:16.947 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/ 2023-07-28T23:50:16.948 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus= 2023-07-28T23:50:16.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1P9ZRmrLPkpPI4yxDEETS2dVZus=', 'Content-Length': '0'} 2023-07-28T23:50:16.950 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e582b9283c8cbb1d-0064c45438-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.951 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:50:16.951 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:50:16.952 DEBUG:boto:Method: PUT 2023-07-28T23:50:16.952 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:50:16.952 DEBUG:boto:Data: 2023-07-28T23:50:16.953 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-07-28T23:50:16.953 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:50:16.953 DEBUG:boto:Port: 8001 2023-07-28T23:50:16.954 DEBUG:boto:Params: {} 2023-07-28T23:50:16.954 DEBUG:boto:Token: None 2023-07-28T23:50:16.955 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 28 Jul 2023 23:50:16 GMT /hiiqos-13/obj 2023-07-28T23:50:16.955 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mKlBOowpz+rxIG8DRDvZM7+EUZU= 2023-07-28T23:50:16.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:50:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mKlBOowpz+rxIG8DRDvZM7+EUZU='} 2023-07-28T23:50:16.965 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'wcK7VKcRTxILi71arrN1zUpJKg8KpNP'), ('x-amz-request-id', 'tx00000df6b939921a3ac88-0064c45438-1113-a2'), ('Date', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:50:16.966 DEBUG:rgw_multi.tests:zone=a2 version=wcK7VKcRTxILi71arrN1zUpJKg8KpNP 2023-07-28T23:50:16.966 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-13 2023-07-28T23:50:16.967 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:50:16.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:50:16.968 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-13 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-28T23:50:17.021 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:50:17.021 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:50:17.054 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:50:17.053+0000 7fbe02baf880 1 waiting for incremental sync to catch up: 2023-07-28T23:50:17.054 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000003.8.10, , , , , ] 2023-07-28T23:50:17.055 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000003.8.10, , , 00000000020.43.12, , ] 2023-07-28T23:50:22.055 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:50:22.054+0000 7fbe02baf880 1 waiting for incremental sync to catch up: 2023-07-28T23:50:22.056 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000003.8.10, , , , , ] 2023-07-28T23:50:22.056 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000003.8.10, , , 00000000020.43.12, , ] 2023-07-28T23:50:27.057 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:50:27.055+0000 7fbe02baf880 1 waiting for incremental sync to catch up: 2023-07-28T23:50:27.057 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000003.8.10, , , , , ] 2023-07-28T23:50:27.057 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000003.8.10, , , 00000000020.43.12, , ] 2023-07-28T23:50:32.058 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:50:32.057+0000 7fbe02baf880 1 waiting for incremental sync to catch up: 2023-07-28T23:50:32.058 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000003.8.10, , , , , ] 2023-07-28T23:50:32.059 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000003.8.10, , , 00000000020.43.12, , ] 2023-07-28T23:50:37.060 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:50:37.058+0000 7fbe02baf880 1 waiting for incremental sync to catch up: 2023-07-28T23:50:37.060 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000003.8.10, , , 00000000013.30.6, , ] 2023-07-28T23:50:37.060 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000003.8.10, , , 00000000020.43.12, , ] 2023-07-28T23:50:42.061 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:50:42.059+0000 7fbe02baf880 1 waiting for incremental sync to catch up: 2023-07-28T23:50:42.062 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000003.8.10, , , 00000000013.30.6, , ] 2023-07-28T23:50:42.062 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000003.8.10, , , 00000000020.43.12, , ] 2023-07-28T23:50:47.062 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:50:47.061+0000 7fbe02baf880 1 waiting for incremental sync to catch up: 2023-07-28T23:50:47.062 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000003.8.10, , , 00000000013.30.6, , ] 2023-07-28T23:50:47.063 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000003.8.10, , , 00000000020.43.12, , ] 2023-07-28T23:50:52.063 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:50:52.062+0000 7fbe02baf880 1 waiting for incremental sync to catch up: 2023-07-28T23:50:52.064 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000003.8.10, , , 00000000013.30.6, , ] 2023-07-28T23:50:52.064 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000003.8.10, , , 00000000020.43.12, , ] 2023-07-28T23:50:57.065 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:50:57.063+0000 7fbe02baf880 1 bucket sync caught up with source: 2023-07-28T23:50:57.065 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000003.8.10, , , 00000000031.74.9, , ] 2023-07-28T23:50:57.066 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000003.8.10, , , 00000000020.43.12, , ] 2023-07-28T23:50:57.066 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:50:57.063+0000 7fbe02baf880 0 bucket checkpoint complete 2023-07-28T23:50:57.080 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-13 2023-07-28T23:50:57.081 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:50:57.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:50:57.082 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-13 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-28T23:50:57.140 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:50:57.140 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:50:57.176 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:50:57.175+0000 7f2ab8e22880 1 waiting for incremental sync to catch up: 2023-07-28T23:50:57.177 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000002.14.11, , , 00000000026.67.6, , ] 2023-07-28T23:50:57.177 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000002.14.11, , , 00000000031.80.12, , ] 2023-07-28T23:51:02.177 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:51:02.176+0000 7f2ab8e22880 1 waiting for incremental sync to catch up: 2023-07-28T23:51:02.178 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000002.14.11, , , 00000000026.67.6, , ] 2023-07-28T23:51:02.178 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000002.14.11, , , 00000000031.80.12, , ] 2023-07-28T23:51:07.178 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:51:07.177+0000 7f2ab8e22880 1 waiting for incremental sync to catch up: 2023-07-28T23:51:07.179 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000002.14.11, , , 00000000026.67.6, , ] 2023-07-28T23:51:07.179 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000002.14.11, , , 00000000031.80.12, , ] 2023-07-28T23:51:12.180 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:51:12.178+0000 7f2ab8e22880 1 waiting for incremental sync to catch up: 2023-07-28T23:51:12.180 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000002.14.11, , , 00000000026.67.6, , ] 2023-07-28T23:51:12.180 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000002.14.11, , , 00000000031.80.12, , ] 2023-07-28T23:51:17.181 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:51:17.179+0000 7f2ab8e22880 1 bucket sync caught up with source: 2023-07-28T23:51:17.181 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000002.14.11, , , 00000000031.80.12, , ] 2023-07-28T23:51:17.181 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000002.14.11, , , 00000000031.80.12, , ] 2023-07-28T23:51:17.182 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:51:17.179+0000 7f2ab8e22880 0 bucket checkpoint complete 2023-07-28T23:51:17.196 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-13 zones={a2, a1} 2023-07-28T23:51:17.196 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:17.197 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:17.197 DEBUG:boto:Method: HEAD 2023-07-28T23:51:17.197 DEBUG:boto:Path: /hiiqos-13/ 2023-07-28T23:51:17.198 DEBUG:boto:Data: 2023-07-28T23:51:17.198 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.198 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:17.199 DEBUG:boto:Port: 8001 2023-07-28T23:51:17.199 DEBUG:boto:Params: {} 2023-07-28T23:51:17.200 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-28T23:51:17.200 DEBUG:boto:Token: None 2023-07-28T23:51:17.201 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/ 2023-07-28T23:51:17.201 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+TpZ28+wen1mommAdWvwFaW9GPQ= 2023-07-28T23:51:17.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+TpZ28+wen1mommAdWvwFaW9GPQ=', 'Content-Length': '0'} 2023-07-28T23:51:17.205 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '160'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000071c655646108d97f-0064c45475-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.205 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:17.206 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:17.206 DEBUG:boto:Method: HEAD 2023-07-28T23:51:17.207 DEBUG:boto:Path: /hiiqos-13/ 2023-07-28T23:51:17.207 DEBUG:boto:Data: 2023-07-28T23:51:17.207 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.208 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:17.208 DEBUG:boto:Port: 8000 2023-07-28T23:51:17.208 DEBUG:boto:Params: {} 2023-07-28T23:51:17.209 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-28T23:51:17.209 DEBUG:boto:Token: None 2023-07-28T23:51:17.209 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/ 2023-07-28T23:51:17.210 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+TpZ28+wen1mommAdWvwFaW9GPQ= 2023-07-28T23:51:17.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+TpZ28+wen1mommAdWvwFaW9GPQ=', 'Content-Length': '0'} 2023-07-28T23:51:17.213 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '160'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006932a3db527e2273-0064c45475-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.214 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-28T23:51:17.214 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:17.215 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:17.215 DEBUG:boto:path=/hiiqos-13/?versions&max-keys=999 2023-07-28T23:51:17.215 DEBUG:boto:auth_path=/hiiqos-13/?versions&max-keys=999 2023-07-28T23:51:17.216 DEBUG:boto:Method: GET 2023-07-28T23:51:17.216 DEBUG:boto:Path: /hiiqos-13/?versions&max-keys=999 2023-07-28T23:51:17.216 DEBUG:boto:Data: 2023-07-28T23:51:17.217 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.217 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:17.217 DEBUG:boto:Port: 8001 2023-07-28T23:51:17.218 DEBUG:boto:Params: {} 2023-07-28T23:51:17.218 DEBUG:boto:Token: None 2023-07-28T23:51:17.218 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/?versions 2023-07-28T23:51:17.219 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/mN0AA8KKE+ICXmxBOVRBM3oCEg= 2023-07-28T23:51:17.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/mN0AA8KKE+ICXmxBOVRBM3oCEg=', 'Content-Length': '0'} 2023-07-28T23:51:17.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5e1ba92b5bc414f-0064c45475-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.223 DEBUG:boto:b'hiiqos-13999falsedummy2iCgPpKTi-zSyx2.F.26sM9kQ2T--adtrue2023-07-28T23:49:00.821Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjvbCzaR0huShRhVgNSTpuzXOsOhW3EDEtrue2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjwcK7VKcRTxILi71arrN1zUpJKg8KpNPfalse2023-07-28T23:50:16.957Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFg82BDw2qebn-TlTZEmuWfNDUSVLkvCfalse2023-07-28T23:50:16.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjz7auNlW8oH1FJvaHlG2v6quhJD.17R1false2023-07-28T23:50:16.874Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKZjcyesGbGSQE5pZSDDLRJ6rT-Mqm8xfalse2023-07-28T23:50:16.830Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWxYcV1Cw2jkGBm-hoDmeDENoWMoiniXfalse2023-07-28T23:50:16.776Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2N3xN1haVyocj8z79kCS3UT0nbjXj6Zfalse2023-07-28T23:50:16.730Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjeu89nyTHw35ZG459SndVR8LnRvJs7VPfalse2023-07-28T23:50:16.754Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9hfXRNs96rGpisQpVYKBPxZpFPAeL47false2023-07-28T23:50:16.645Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKW4erdPcpVPNf4hlrOZysjOAxTTzNpZfalse2023-07-28T23:50:16.707Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjfqZHgkQ0VnkgGOiotHJVpzmOgEM1PNXfalse2023-07-28T23:50:16.618Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjl2NN5Bw32G2ViK4cHi-XORehcdjCcxDfalse2023-07-28T23:50:16.595Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMMa4nLQC3W82FeUky1Sq9XW9uFjpW6Qfalse2023-07-28T23:50:16.570Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjfxsCLuGqt0tjCmqlJgclHqGWnHEcxLAfalse2023-07-28T23:50:16.530Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22KG5jfO2TftvgPgeJvtwXX52mGQhV1false2023-07-28T23:50:16.470Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKF9pysOWS.qePO4W3-Ux03VJiRC9ph7false2023-07-28T23:50:16.446Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58wMXe18IrbR7RSdRvF4V8KhCCY6CR.false2023-07-28T23:50:16.423Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiRoeqa0SUPCZisrafCoZycv1QQITmghfalse2023-07-28T23:50:16.398Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjOYqlT9m6zvLoRrfJK6xkLWsj3aXU2MGfalse2023-07-28T23:50:16.343Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnQT3SHUTnGGHoRE.HK4699V4TWQJbCAfalse2023-07-28T23:50:16.280Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:17.224 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-07-28T23:51:17.225 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:51:17.225 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:51:17.225 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:51:17.226 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:51:17.226 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:51:17.226 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:51:17.226 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:51:17.227 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:51:17.227 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:51:17.227 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:51:17.228 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:51:17.228 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:51:17.228 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:51:17.229 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:51:17.229 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:51:17.229 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:51:17.229 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:51:17.230 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:51:17.230 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:51:17.230 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:51:17.231 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-28T23:51:17.231 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:17.231 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:17.232 DEBUG:boto:path=/hiiqos-13/?versions&max-keys=999 2023-07-28T23:51:17.232 DEBUG:boto:auth_path=/hiiqos-13/?versions&max-keys=999 2023-07-28T23:51:17.232 DEBUG:boto:Method: GET 2023-07-28T23:51:17.233 DEBUG:boto:Path: /hiiqos-13/?versions&max-keys=999 2023-07-28T23:51:17.233 DEBUG:boto:Data: 2023-07-28T23:51:17.233 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.234 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:17.234 DEBUG:boto:Port: 8000 2023-07-28T23:51:17.234 DEBUG:boto:Params: {} 2023-07-28T23:51:17.235 DEBUG:boto:Token: None 2023-07-28T23:51:17.235 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/?versions 2023-07-28T23:51:17.235 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/mN0AA8KKE+ICXmxBOVRBM3oCEg= 2023-07-28T23:51:17.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/mN0AA8KKE+ICXmxBOVRBM3oCEg=', 'Content-Length': '0'} 2023-07-28T23:51:17.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef9f012da1946793-0064c45475-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.239 DEBUG:boto:b'hiiqos-13999falsedummy2iCgPpKTi-zSyx2.F.26sM9kQ2T--adtrue2023-07-28T23:49:00.821Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjvbCzaR0huShRhVgNSTpuzXOsOhW3EDEtrue2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjwcK7VKcRTxILi71arrN1zUpJKg8KpNPfalse2023-07-28T23:50:16.957Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFg82BDw2qebn-TlTZEmuWfNDUSVLkvCfalse2023-07-28T23:50:16.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjz7auNlW8oH1FJvaHlG2v6quhJD.17R1false2023-07-28T23:50:16.874Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKZjcyesGbGSQE5pZSDDLRJ6rT-Mqm8xfalse2023-07-28T23:50:16.830Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWxYcV1Cw2jkGBm-hoDmeDENoWMoiniXfalse2023-07-28T23:50:16.776Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2N3xN1haVyocj8z79kCS3UT0nbjXj6Zfalse2023-07-28T23:50:16.730Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjeu89nyTHw35ZG459SndVR8LnRvJs7VPfalse2023-07-28T23:50:16.754Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9hfXRNs96rGpisQpVYKBPxZpFPAeL47false2023-07-28T23:50:16.645Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKW4erdPcpVPNf4hlrOZysjOAxTTzNpZfalse2023-07-28T23:50:16.707Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjfqZHgkQ0VnkgGOiotHJVpzmOgEM1PNXfalse2023-07-28T23:50:16.618Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjl2NN5Bw32G2ViK4cHi-XORehcdjCcxDfalse2023-07-28T23:50:16.595Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMMa4nLQC3W82FeUky1Sq9XW9uFjpW6Qfalse2023-07-28T23:50:16.570Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjfxsCLuGqt0tjCmqlJgclHqGWnHEcxLAfalse2023-07-28T23:50:16.530Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22KG5jfO2TftvgPgeJvtwXX52mGQhV1false2023-07-28T23:50:16.470Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKF9pysOWS.qePO4W3-Ux03VJiRC9ph7false2023-07-28T23:50:16.446Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58wMXe18IrbR7RSdRvF4V8KhCCY6CR.false2023-07-28T23:50:16.423Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiRoeqa0SUPCZisrafCoZycv1QQITmghfalse2023-07-28T23:50:16.398Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjOYqlT9m6zvLoRrfJK6xkLWsj3aXU2MGfalse2023-07-28T23:50:16.343Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnQT3SHUTnGGHoRE.HK4699V4TWQJbCAfalse2023-07-28T23:50:16.280Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:17.240 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-07-28T23:51:17.241 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:51:17.241 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:51:17.241 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:51:17.242 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:51:17.242 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:51:17.242 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:51:17.242 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:51:17.243 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:51:17.243 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:51:17.243 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:51:17.244 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:51:17.244 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:51:17.244 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:51:17.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:51:17.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:51:17.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:51:17.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:51:17.246 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:51:17.246 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:51:17.246 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:51:17.247 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:17.247 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:17.247 DEBUG:boto:path=/hiiqos-13/?versions&max-keys=999 2023-07-28T23:51:17.248 DEBUG:boto:auth_path=/hiiqos-13/?versions&max-keys=999 2023-07-28T23:51:17.248 DEBUG:boto:Method: GET 2023-07-28T23:51:17.248 DEBUG:boto:Path: /hiiqos-13/?versions&max-keys=999 2023-07-28T23:51:17.249 DEBUG:boto:Data: 2023-07-28T23:51:17.249 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.249 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:17.250 DEBUG:boto:Port: 8001 2023-07-28T23:51:17.250 DEBUG:boto:Params: {} 2023-07-28T23:51:17.250 DEBUG:boto:Token: None 2023-07-28T23:51:17.251 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/?versions 2023-07-28T23:51:17.251 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/mN0AA8KKE+ICXmxBOVRBM3oCEg= 2023-07-28T23:51:17.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/mN0AA8KKE+ICXmxBOVRBM3oCEg=', 'Content-Length': '0'} 2023-07-28T23:51:17.254 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bb9ba25cdf2018b-0064c45475-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.255 DEBUG:boto:b'hiiqos-13999falsedummy2iCgPpKTi-zSyx2.F.26sM9kQ2T--adtrue2023-07-28T23:49:00.821Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjvbCzaR0huShRhVgNSTpuzXOsOhW3EDEtrue2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjwcK7VKcRTxILi71arrN1zUpJKg8KpNPfalse2023-07-28T23:50:16.957Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFg82BDw2qebn-TlTZEmuWfNDUSVLkvCfalse2023-07-28T23:50:16.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjz7auNlW8oH1FJvaHlG2v6quhJD.17R1false2023-07-28T23:50:16.874Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKZjcyesGbGSQE5pZSDDLRJ6rT-Mqm8xfalse2023-07-28T23:50:16.830Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWxYcV1Cw2jkGBm-hoDmeDENoWMoiniXfalse2023-07-28T23:50:16.776Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2N3xN1haVyocj8z79kCS3UT0nbjXj6Zfalse2023-07-28T23:50:16.730Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjeu89nyTHw35ZG459SndVR8LnRvJs7VPfalse2023-07-28T23:50:16.754Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9hfXRNs96rGpisQpVYKBPxZpFPAeL47false2023-07-28T23:50:16.645Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKW4erdPcpVPNf4hlrOZysjOAxTTzNpZfalse2023-07-28T23:50:16.707Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjfqZHgkQ0VnkgGOiotHJVpzmOgEM1PNXfalse2023-07-28T23:50:16.618Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjl2NN5Bw32G2ViK4cHi-XORehcdjCcxDfalse2023-07-28T23:50:16.595Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMMa4nLQC3W82FeUky1Sq9XW9uFjpW6Qfalse2023-07-28T23:50:16.570Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjfxsCLuGqt0tjCmqlJgclHqGWnHEcxLAfalse2023-07-28T23:50:16.530Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22KG5jfO2TftvgPgeJvtwXX52mGQhV1false2023-07-28T23:50:16.470Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKF9pysOWS.qePO4W3-Ux03VJiRC9ph7false2023-07-28T23:50:16.446Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58wMXe18IrbR7RSdRvF4V8KhCCY6CR.false2023-07-28T23:50:16.423Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiRoeqa0SUPCZisrafCoZycv1QQITmghfalse2023-07-28T23:50:16.398Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjOYqlT9m6zvLoRrfJK6xkLWsj3aXU2MGfalse2023-07-28T23:50:16.343Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnQT3SHUTnGGHoRE.HK4699V4TWQJbCAfalse2023-07-28T23:50:16.280Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:17.256 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:17.257 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:17.257 DEBUG:boto:path=/hiiqos-13/?versions&max-keys=999 2023-07-28T23:51:17.257 DEBUG:boto:auth_path=/hiiqos-13/?versions&max-keys=999 2023-07-28T23:51:17.258 DEBUG:boto:Method: GET 2023-07-28T23:51:17.258 DEBUG:boto:Path: /hiiqos-13/?versions&max-keys=999 2023-07-28T23:51:17.258 DEBUG:boto:Data: 2023-07-28T23:51:17.259 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.259 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:17.259 DEBUG:boto:Port: 8000 2023-07-28T23:51:17.260 DEBUG:boto:Params: {} 2023-07-28T23:51:17.260 DEBUG:boto:Token: None 2023-07-28T23:51:17.260 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/?versions 2023-07-28T23:51:17.261 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/mN0AA8KKE+ICXmxBOVRBM3oCEg= 2023-07-28T23:51:17.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/mN0AA8KKE+ICXmxBOVRBM3oCEg=', 'Content-Length': '0'} 2023-07-28T23:51:17.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000272d729b94c95eee-0064c45475-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.264 DEBUG:boto:b'hiiqos-13999falsedummy2iCgPpKTi-zSyx2.F.26sM9kQ2T--adtrue2023-07-28T23:49:00.821Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjvbCzaR0huShRhVgNSTpuzXOsOhW3EDEtrue2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjwcK7VKcRTxILi71arrN1zUpJKg8KpNPfalse2023-07-28T23:50:16.957Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFg82BDw2qebn-TlTZEmuWfNDUSVLkvCfalse2023-07-28T23:50:16.850Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjz7auNlW8oH1FJvaHlG2v6quhJD.17R1false2023-07-28T23:50:16.874Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKZjcyesGbGSQE5pZSDDLRJ6rT-Mqm8xfalse2023-07-28T23:50:16.830Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWxYcV1Cw2jkGBm-hoDmeDENoWMoiniXfalse2023-07-28T23:50:16.776Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2N3xN1haVyocj8z79kCS3UT0nbjXj6Zfalse2023-07-28T23:50:16.730Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjeu89nyTHw35ZG459SndVR8LnRvJs7VPfalse2023-07-28T23:50:16.754Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9hfXRNs96rGpisQpVYKBPxZpFPAeL47false2023-07-28T23:50:16.645Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKW4erdPcpVPNf4hlrOZysjOAxTTzNpZfalse2023-07-28T23:50:16.707Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjfqZHgkQ0VnkgGOiotHJVpzmOgEM1PNXfalse2023-07-28T23:50:16.618Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjl2NN5Bw32G2ViK4cHi-XORehcdjCcxDfalse2023-07-28T23:50:16.595Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjMMa4nLQC3W82FeUky1Sq9XW9uFjpW6Qfalse2023-07-28T23:50:16.570Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjfxsCLuGqt0tjCmqlJgclHqGWnHEcxLAfalse2023-07-28T23:50:16.530Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22KG5jfO2TftvgPgeJvtwXX52mGQhV1false2023-07-28T23:50:16.470Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKF9pysOWS.qePO4W3-Ux03VJiRC9ph7false2023-07-28T23:50:16.446Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58wMXe18IrbR7RSdRvF4V8KhCCY6CR.false2023-07-28T23:50:16.423Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiRoeqa0SUPCZisrafCoZycv1QQITmghfalse2023-07-28T23:50:16.398Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjOYqlT9m6zvLoRrfJK6xkLWsj3aXU2MGfalse2023-07-28T23:50:16.343Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnQT3SHUTnGGHoRE.HK4699V4TWQJbCAfalse2023-07-28T23:50:16.280Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:17.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-07-28T23:51:17.266 DEBUG:boto:path=/hiiqos-13/dummy 2023-07-28T23:51:17.267 DEBUG:boto:auth_path=/hiiqos-13/dummy 2023-07-28T23:51:17.267 DEBUG:boto:path=/hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:51:17.267 DEBUG:boto:auth_path=/hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:51:17.268 DEBUG:boto:Method: GET 2023-07-28T23:51:17.268 DEBUG:boto:Path: /hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:51:17.268 DEBUG:boto:Data: 2023-07-28T23:51:17.269 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.269 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:17.269 DEBUG:boto:Port: 8001 2023-07-28T23:51:17.270 DEBUG:boto:Params: {} 2023-07-28T23:51:17.270 DEBUG:boto:Token: None 2023-07-28T23:51:17.271 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:51:17.271 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xKjk+aYL3RORd5t+OR8C3m1DIek= 2023-07-28T23:51:17.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xKjk+aYL3RORd5t+OR8C3m1DIek=', 'Content-Length': '0'} 2023-07-28T23:51:17.273 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:49:00 GMT'), ('x-amz-version-id', '2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ad524604339f0820-0064c45475-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.274 DEBUG:boto:path=/hiiqos-13/dummy 2023-07-28T23:51:17.274 DEBUG:boto:auth_path=/hiiqos-13/dummy 2023-07-28T23:51:17.275 DEBUG:boto:path=/hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:51:17.275 DEBUG:boto:auth_path=/hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:51:17.275 DEBUG:boto:Method: GET 2023-07-28T23:51:17.276 DEBUG:boto:Path: /hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:51:17.276 DEBUG:boto:Data: 2023-07-28T23:51:17.276 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.277 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:17.277 DEBUG:boto:Port: 8000 2023-07-28T23:51:17.277 DEBUG:boto:Params: {} 2023-07-28T23:51:17.278 DEBUG:boto:Token: None 2023-07-28T23:51:17.278 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:51:17.279 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xKjk+aYL3RORd5t+OR8C3m1DIek= 2023-07-28T23:51:17.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xKjk+aYL3RORd5t+OR8C3m1DIek=', 'Content-Length': '0'} 2023-07-28T23:51:17.281 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:49:00 GMT'), ('x-amz-version-id', '2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003eb0d1c31a74709d-0064c45475-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.281 DEBUG:boto:path=/hiiqos-13/dummy 2023-07-28T23:51:17.282 DEBUG:boto:auth_path=/hiiqos-13/dummy 2023-07-28T23:51:17.282 DEBUG:boto:path=/hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:51:17.282 DEBUG:boto:auth_path=/hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:51:17.283 DEBUG:boto:Method: HEAD 2023-07-28T23:51:17.283 DEBUG:boto:Path: /hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:51:17.283 DEBUG:boto:Data: 2023-07-28T23:51:17.284 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.284 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:17.284 DEBUG:boto:Port: 8001 2023-07-28T23:51:17.285 DEBUG:boto:Params: {} 2023-07-28T23:51:17.285 DEBUG:boto:Token: None 2023-07-28T23:51:17.285 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:51:17.286 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BdX/kJM4a5xYuJEgipEQjQtmld4= 2023-07-28T23:51:17.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BdX/kJM4a5xYuJEgipEQjQtmld4=', 'Content-Length': '0'} 2023-07-28T23:51:17.288 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:49:00 GMT'), ('x-amz-version-id', '2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c651dfd23936849d-0064c45475-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.288 DEBUG:boto:path=/hiiqos-13/dummy 2023-07-28T23:51:17.289 DEBUG:boto:auth_path=/hiiqos-13/dummy 2023-07-28T23:51:17.289 DEBUG:boto:path=/hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:51:17.289 DEBUG:boto:auth_path=/hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:51:17.290 DEBUG:boto:Method: HEAD 2023-07-28T23:51:17.290 DEBUG:boto:Path: /hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:51:17.290 DEBUG:boto:Data: 2023-07-28T23:51:17.291 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.291 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:17.291 DEBUG:boto:Port: 8000 2023-07-28T23:51:17.292 DEBUG:boto:Params: {} 2023-07-28T23:51:17.292 DEBUG:boto:Token: None 2023-07-28T23:51:17.292 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:51:17.293 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BdX/kJM4a5xYuJEgipEQjQtmld4= 2023-07-28T23:51:17.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BdX/kJM4a5xYuJEgipEQjQtmld4=', 'Content-Length': '0'} 2023-07-28T23:51:17.295 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:49:00 GMT'), ('x-amz-version-id', '2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000065e96ebbe925e71f-0064c45475-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-07-28T23:51:17.296 DEBUG:boto:path=/hiiqos-13/dummy 2023-07-28T23:51:17.296 DEBUG:boto:auth_path=/hiiqos-13/dummy 2023-07-28T23:51:17.296 DEBUG:boto:path=/hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:51:17.297 DEBUG:boto:auth_path=/hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:51:17.297 DEBUG:boto:Method: GET 2023-07-28T23:51:17.297 DEBUG:boto:Path: /hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:51:17.298 DEBUG:boto:Data: 2023-07-28T23:51:17.298 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.298 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:17.299 DEBUG:boto:Port: 8001 2023-07-28T23:51:17.299 DEBUG:boto:Params: {} 2023-07-28T23:51:17.299 DEBUG:boto:Token: None 2023-07-28T23:51:17.300 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:51:17.300 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xKjk+aYL3RORd5t+OR8C3m1DIek= 2023-07-28T23:51:17.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xKjk+aYL3RORd5t+OR8C3m1DIek=', 'Content-Length': '0'} 2023-07-28T23:51:17.303 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:49:00 GMT'), ('x-amz-version-id', '2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e1ec2afba846d912-0064c45475-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.303 DEBUG:boto:path=/hiiqos-13/dummy 2023-07-28T23:51:17.304 DEBUG:boto:auth_path=/hiiqos-13/dummy 2023-07-28T23:51:17.304 DEBUG:boto:path=/hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:51:17.304 DEBUG:boto:auth_path=/hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:51:17.305 DEBUG:boto:Method: GET 2023-07-28T23:51:17.305 DEBUG:boto:Path: /hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:51:17.305 DEBUG:boto:Data: 2023-07-28T23:51:17.306 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.306 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:17.306 DEBUG:boto:Port: 8000 2023-07-28T23:51:17.307 DEBUG:boto:Params: {} 2023-07-28T23:51:17.307 DEBUG:boto:Token: None 2023-07-28T23:51:17.308 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:51:17.308 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xKjk+aYL3RORd5t+OR8C3m1DIek= 2023-07-28T23:51:17.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xKjk+aYL3RORd5t+OR8C3m1DIek=', 'Content-Length': '0'} 2023-07-28T23:51:17.310 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:49:00 GMT'), ('x-amz-version-id', '2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e7c25e35f4689d78-0064c45475-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.311 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:17.311 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:17.312 DEBUG:boto:path=/hiiqos-13/?prefix=dummy 2023-07-28T23:51:17.312 DEBUG:boto:auth_path=/hiiqos-13/?prefix=dummy 2023-07-28T23:51:17.312 DEBUG:boto:Method: GET 2023-07-28T23:51:17.313 DEBUG:boto:Path: /hiiqos-13/?prefix=dummy 2023-07-28T23:51:17.313 DEBUG:boto:Data: 2023-07-28T23:51:17.313 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.314 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:17.314 DEBUG:boto:Port: 8001 2023-07-28T23:51:17.314 DEBUG:boto:Params: {} 2023-07-28T23:51:17.315 DEBUG:boto:Token: None 2023-07-28T23:51:17.315 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/ 2023-07-28T23:51:17.316 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TqnQ5G24uPNdnM79HmlBTN/5oHM= 2023-07-28T23:51:17.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TqnQ5G24uPNdnM79HmlBTN/5oHM=', 'Content-Length': '0'} 2023-07-28T23:51:17.318 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004063d771b41ca3f3-0064c45475-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.319 DEBUG:boto:b'hiiqos-13dummy1000falsedummy2023-07-28T23:49:00.821Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:17.320 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:17.320 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:17.320 DEBUG:boto:path=/hiiqos-13/?prefix=dummy 2023-07-28T23:51:17.321 DEBUG:boto:auth_path=/hiiqos-13/?prefix=dummy 2023-07-28T23:51:17.321 DEBUG:boto:Method: GET 2023-07-28T23:51:17.321 DEBUG:boto:Path: /hiiqos-13/?prefix=dummy 2023-07-28T23:51:17.322 DEBUG:boto:Data: 2023-07-28T23:51:17.322 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.322 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:17.323 DEBUG:boto:Port: 8000 2023-07-28T23:51:17.323 DEBUG:boto:Params: {} 2023-07-28T23:51:17.323 DEBUG:boto:Token: None 2023-07-28T23:51:17.324 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/ 2023-07-28T23:51:17.324 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TqnQ5G24uPNdnM79HmlBTN/5oHM= 2023-07-28T23:51:17.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TqnQ5G24uPNdnM79HmlBTN/5oHM=', 'Content-Length': '0'} 2023-07-28T23:51:17.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005aaca5fd0ed9d5bc-0064c45475-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.328 DEBUG:boto:b'hiiqos-13dummy1000falsedummy2023-07-28T23:49:00.821Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:17.328 DEBUG:boto:path=/hiiqos-13/dummy 2023-07-28T23:51:17.329 DEBUG:boto:auth_path=/hiiqos-13/dummy 2023-07-28T23:51:17.329 DEBUG:boto:Method: HEAD 2023-07-28T23:51:17.329 DEBUG:boto:Path: /hiiqos-13/dummy 2023-07-28T23:51:17.330 DEBUG:boto:Data: 2023-07-28T23:51:17.330 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.330 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:17.331 DEBUG:boto:Port: 8001 2023-07-28T23:51:17.331 DEBUG:boto:Params: {} 2023-07-28T23:51:17.331 DEBUG:boto:Token: None 2023-07-28T23:51:17.332 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/dummy 2023-07-28T23:51:17.332 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bFa7R9kseRs1VKttM+xBRYVtYzA= 2023-07-28T23:51:17.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bFa7R9kseRs1VKttM+xBRYVtYzA=', 'Content-Length': '0'} 2023-07-28T23:51:17.334 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:49:00 GMT'), ('x-amz-version-id', '2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a28ff903aa0d7972-0064c45475-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.335 DEBUG:boto:path=/hiiqos-13/dummy 2023-07-28T23:51:17.335 DEBUG:boto:auth_path=/hiiqos-13/dummy 2023-07-28T23:51:17.336 DEBUG:boto:Method: HEAD 2023-07-28T23:51:17.336 DEBUG:boto:Path: /hiiqos-13/dummy 2023-07-28T23:51:17.336 DEBUG:boto:Data: 2023-07-28T23:51:17.337 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.337 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:17.337 DEBUG:boto:Port: 8000 2023-07-28T23:51:17.338 DEBUG:boto:Params: {} 2023-07-28T23:51:17.338 DEBUG:boto:Token: None 2023-07-28T23:51:17.338 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/dummy 2023-07-28T23:51:17.339 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bFa7R9kseRs1VKttM+xBRYVtYzA= 2023-07-28T23:51:17.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bFa7R9kseRs1VKttM+xBRYVtYzA=', 'Content-Length': '0'} 2023-07-28T23:51:17.341 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:49:00 GMT'), ('x-amz-version-id', '2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001e5741a3ebd0462d-0064c45475-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-07-28T23:51:17.342 DEBUG:boto:path=/hiiqos-13/dummy 2023-07-28T23:51:17.343 DEBUG:boto:auth_path=/hiiqos-13/dummy 2023-07-28T23:51:17.343 DEBUG:boto:path=/hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:51:17.343 DEBUG:boto:auth_path=/hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:51:17.344 DEBUG:boto:Method: GET 2023-07-28T23:51:17.344 DEBUG:boto:Path: /hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:51:17.344 DEBUG:boto:Data: 2023-07-28T23:51:17.345 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.345 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:17.345 DEBUG:boto:Port: 8001 2023-07-28T23:51:17.346 DEBUG:boto:Params: {} 2023-07-28T23:51:17.346 DEBUG:boto:Token: None 2023-07-28T23:51:17.346 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:51:17.347 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xKjk+aYL3RORd5t+OR8C3m1DIek= 2023-07-28T23:51:17.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xKjk+aYL3RORd5t+OR8C3m1DIek=', 'Content-Length': '0'} 2023-07-28T23:51:17.349 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:49:00 GMT'), ('x-amz-version-id', '2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001a25557ba9898650-0064c45475-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.350 DEBUG:boto:path=/hiiqos-13/dummy 2023-07-28T23:51:17.350 DEBUG:boto:auth_path=/hiiqos-13/dummy 2023-07-28T23:51:17.350 DEBUG:boto:path=/hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:51:17.351 DEBUG:boto:auth_path=/hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:51:17.351 DEBUG:boto:Method: GET 2023-07-28T23:51:17.351 DEBUG:boto:Path: /hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:51:17.352 DEBUG:boto:Data: 2023-07-28T23:51:17.352 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.353 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:17.353 DEBUG:boto:Port: 8000 2023-07-28T23:51:17.353 DEBUG:boto:Params: {} 2023-07-28T23:51:17.354 DEBUG:boto:Token: None 2023-07-28T23:51:17.354 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/dummy?versionId=2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad 2023-07-28T23:51:17.354 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xKjk+aYL3RORd5t+OR8C3m1DIek= 2023-07-28T23:51:17.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xKjk+aYL3RORd5t+OR8C3m1DIek=', 'Content-Length': '0'} 2023-07-28T23:51:17.356 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:49:00 GMT'), ('x-amz-version-id', '2iCgPpKTi-zSyx2.F.26sM9kQ2T--ad'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000766427bd794b36f5-0064c45475-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.357 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:17.357 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:17.358 DEBUG:boto:path=/hiiqos-13/?prefix=dummy 2023-07-28T23:51:17.358 DEBUG:boto:auth_path=/hiiqos-13/?prefix=dummy 2023-07-28T23:51:17.358 DEBUG:boto:Method: GET 2023-07-28T23:51:17.359 DEBUG:boto:Path: /hiiqos-13/?prefix=dummy 2023-07-28T23:51:17.359 DEBUG:boto:Data: 2023-07-28T23:51:17.360 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.360 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:17.360 DEBUG:boto:Port: 8001 2023-07-28T23:51:17.361 DEBUG:boto:Params: {} 2023-07-28T23:51:17.361 DEBUG:boto:Token: None 2023-07-28T23:51:17.361 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/ 2023-07-28T23:51:17.362 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TqnQ5G24uPNdnM79HmlBTN/5oHM= 2023-07-28T23:51:17.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TqnQ5G24uPNdnM79HmlBTN/5oHM=', 'Content-Length': '0'} 2023-07-28T23:51:17.365 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6c7773907adfddc-0064c45475-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.408 DEBUG:boto:b'hiiqos-13dummy1000falsedummy2023-07-28T23:49:00.821Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:17.409 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:17.409 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:17.410 DEBUG:boto:path=/hiiqos-13/?prefix=dummy 2023-07-28T23:51:17.410 DEBUG:boto:auth_path=/hiiqos-13/?prefix=dummy 2023-07-28T23:51:17.410 DEBUG:boto:Method: GET 2023-07-28T23:51:17.411 DEBUG:boto:Path: /hiiqos-13/?prefix=dummy 2023-07-28T23:51:17.411 DEBUG:boto:Data: 2023-07-28T23:51:17.412 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.412 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:17.412 DEBUG:boto:Port: 8000 2023-07-28T23:51:17.413 DEBUG:boto:Params: {} 2023-07-28T23:51:17.413 DEBUG:boto:Token: None 2023-07-28T23:51:17.414 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/ 2023-07-28T23:51:17.414 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TqnQ5G24uPNdnM79HmlBTN/5oHM= 2023-07-28T23:51:17.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TqnQ5G24uPNdnM79HmlBTN/5oHM=', 'Content-Length': '0'} 2023-07-28T23:51:17.417 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008385c49cf8497ed5-0064c45475-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.417 DEBUG:boto:b'hiiqos-13dummy1000falsedummy2023-07-28T23:49:00.821Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:17.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:17.418 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:17.419 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:17.419 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.419 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.420 DEBUG:boto:Method: GET 2023-07-28T23:51:17.420 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.421 DEBUG:boto:Data: 2023-07-28T23:51:17.421 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.421 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:17.422 DEBUG:boto:Port: 8001 2023-07-28T23:51:17.422 DEBUG:boto:Params: {} 2023-07-28T23:51:17.422 DEBUG:boto:Token: None 2023-07-28T23:51:17.423 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.423 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oHSMk/nDzhuivy21MXwOs1ZXhwg= 2023-07-28T23:51:17.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oHSMk/nDzhuivy21MXwOs1ZXhwg=', 'Content-Length': '0'} 2023-07-28T23:51:17.425 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000018398557efdcedef-0064c45475-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.426 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:17.426 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:17.427 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.427 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.428 DEBUG:boto:Method: GET 2023-07-28T23:51:17.428 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.428 DEBUG:boto:Data: 2023-07-28T23:51:17.429 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.429 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:17.429 DEBUG:boto:Port: 8000 2023-07-28T23:51:17.430 DEBUG:boto:Params: {} 2023-07-28T23:51:17.430 DEBUG:boto:Token: None 2023-07-28T23:51:17.430 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.431 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oHSMk/nDzhuivy21MXwOs1ZXhwg= 2023-07-28T23:51:17.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oHSMk/nDzhuivy21MXwOs1ZXhwg=', 'Content-Length': '0'} 2023-07-28T23:51:17.433 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000997d1563e01cc3e3-0064c45475-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.434 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:17.434 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:17.434 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.435 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.435 DEBUG:boto:Method: HEAD 2023-07-28T23:51:17.435 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.436 DEBUG:boto:Data: 2023-07-28T23:51:17.436 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.436 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:17.437 DEBUG:boto:Port: 8001 2023-07-28T23:51:17.437 DEBUG:boto:Params: {} 2023-07-28T23:51:17.437 DEBUG:boto:Token: None 2023-07-28T23:51:17.438 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.438 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:stcUY76Il/D94vRkS+NtXfQetcQ= 2023-07-28T23:51:17.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:stcUY76Il/D94vRkS+NtXfQetcQ=', 'Content-Length': '0'} 2023-07-28T23:51:17.440 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000204d6226ef98b70e-0064c45475-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.441 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:17.441 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:17.442 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.442 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.442 DEBUG:boto:Method: HEAD 2023-07-28T23:51:17.443 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.443 DEBUG:boto:Data: 2023-07-28T23:51:17.443 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.444 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:17.444 DEBUG:boto:Port: 8000 2023-07-28T23:51:17.444 DEBUG:boto:Params: {} 2023-07-28T23:51:17.445 DEBUG:boto:Token: None 2023-07-28T23:51:17.445 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.445 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:stcUY76Il/D94vRkS+NtXfQetcQ= 2023-07-28T23:51:17.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:stcUY76Il/D94vRkS+NtXfQetcQ=', 'Content-Length': '0'} 2023-07-28T23:51:17.447 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad219fc8c07e883b-0064c45475-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.448 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:17.448 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:17.449 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:17.449 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.449 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.450 DEBUG:boto:Method: GET 2023-07-28T23:51:17.450 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.451 DEBUG:boto:Data: 2023-07-28T23:51:17.451 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.451 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:17.452 DEBUG:boto:Port: 8001 2023-07-28T23:51:17.452 DEBUG:boto:Params: {} 2023-07-28T23:51:17.452 DEBUG:boto:Token: None 2023-07-28T23:51:17.453 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.453 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oHSMk/nDzhuivy21MXwOs1ZXhwg= 2023-07-28T23:51:17.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oHSMk/nDzhuivy21MXwOs1ZXhwg=', 'Content-Length': '0'} 2023-07-28T23:51:17.455 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001f70931bb1d23bfe-0064c45475-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.456 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:17.456 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:17.457 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.457 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.457 DEBUG:boto:Method: GET 2023-07-28T23:51:17.458 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.458 DEBUG:boto:Data: 2023-07-28T23:51:17.458 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.459 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:17.459 DEBUG:boto:Port: 8000 2023-07-28T23:51:17.459 DEBUG:boto:Params: {} 2023-07-28T23:51:17.460 DEBUG:boto:Token: None 2023-07-28T23:51:17.460 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.461 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oHSMk/nDzhuivy21MXwOs1ZXhwg= 2023-07-28T23:51:17.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oHSMk/nDzhuivy21MXwOs1ZXhwg=', 'Content-Length': '0'} 2023-07-28T23:51:17.463 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e0ebfea5c78a052f-0064c45475-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.463 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:17.464 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:17.464 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:17.464 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:17.465 DEBUG:boto:Method: GET 2023-07-28T23:51:17.465 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:17.466 DEBUG:boto:Data: 2023-07-28T23:51:17.466 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.466 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:17.467 DEBUG:boto:Port: 8001 2023-07-28T23:51:17.467 DEBUG:boto:Params: {} 2023-07-28T23:51:17.467 DEBUG:boto:Token: None 2023-07-28T23:51:17.468 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/ 2023-07-28T23:51:17.468 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TqnQ5G24uPNdnM79HmlBTN/5oHM= 2023-07-28T23:51:17.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TqnQ5G24uPNdnM79HmlBTN/5oHM=', 'Content-Length': '0'} 2023-07-28T23:51:17.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000984786989e8f5952-0064c45475-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.471 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:17.472 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:17.472 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:17.473 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:17.473 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:17.473 DEBUG:boto:Method: GET 2023-07-28T23:51:17.474 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:17.474 DEBUG:boto:Data: 2023-07-28T23:51:17.474 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.475 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:17.475 DEBUG:boto:Port: 8000 2023-07-28T23:51:17.475 DEBUG:boto:Params: {} 2023-07-28T23:51:17.476 DEBUG:boto:Token: None 2023-07-28T23:51:17.476 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/ 2023-07-28T23:51:17.476 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TqnQ5G24uPNdnM79HmlBTN/5oHM= 2023-07-28T23:51:17.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TqnQ5G24uPNdnM79HmlBTN/5oHM=', 'Content-Length': '0'} 2023-07-28T23:51:17.479 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b9f902d62e8bcb3-0064c45475-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.480 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:17.480 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:17.481 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:17.481 DEBUG:boto:Method: HEAD 2023-07-28T23:51:17.481 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:51:17.482 DEBUG:boto:Data: 2023-07-28T23:51:17.482 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.482 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:17.483 DEBUG:boto:Port: 8001 2023-07-28T23:51:17.483 DEBUG:boto:Params: {} 2023-07-28T23:51:17.483 DEBUG:boto:Token: None 2023-07-28T23:51:17.484 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/obj 2023-07-28T23:51:17.484 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9LkqZndmQrAsjh22SZmOf2JHjmk= 2023-07-28T23:51:17.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9LkqZndmQrAsjh22SZmOf2JHjmk=', 'Content-Length': '0'} 2023-07-28T23:51:17.487 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001bacea27fbce8599-0064c45475-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.488 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:17.488 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:17.489 DEBUG:boto:Method: HEAD 2023-07-28T23:51:17.489 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:51:17.489 DEBUG:boto:Data: 2023-07-28T23:51:17.490 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.490 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:17.490 DEBUG:boto:Port: 8000 2023-07-28T23:51:17.491 DEBUG:boto:Params: {} 2023-07-28T23:51:17.491 DEBUG:boto:Token: None 2023-07-28T23:51:17.491 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/obj 2023-07-28T23:51:17.492 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9LkqZndmQrAsjh22SZmOf2JHjmk= 2023-07-28T23:51:17.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9LkqZndmQrAsjh22SZmOf2JHjmk=', 'Content-Length': '0'} 2023-07-28T23:51:17.495 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006a6f568dbd7b22d1-0064c45475-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:17.495 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:17.496 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:17.496 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.497 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.497 DEBUG:boto:Method: GET 2023-07-28T23:51:17.497 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.498 DEBUG:boto:Data: 2023-07-28T23:51:17.498 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.498 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:17.499 DEBUG:boto:Port: 8001 2023-07-28T23:51:17.499 DEBUG:boto:Params: {} 2023-07-28T23:51:17.499 DEBUG:boto:Token: None 2023-07-28T23:51:17.500 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.500 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oHSMk/nDzhuivy21MXwOs1ZXhwg= 2023-07-28T23:51:17.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oHSMk/nDzhuivy21MXwOs1ZXhwg=', 'Content-Length': '0'} 2023-07-28T23:51:17.502 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b9df813019b9f55e-0064c45475-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.503 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:17.503 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:17.504 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.504 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.504 DEBUG:boto:Method: GET 2023-07-28T23:51:17.505 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.505 DEBUG:boto:Data: 2023-07-28T23:51:17.505 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.506 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:17.506 DEBUG:boto:Port: 8000 2023-07-28T23:51:17.507 DEBUG:boto:Params: {} 2023-07-28T23:51:17.507 DEBUG:boto:Token: None 2023-07-28T23:51:17.507 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.508 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oHSMk/nDzhuivy21MXwOs1ZXhwg= 2023-07-28T23:51:17.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oHSMk/nDzhuivy21MXwOs1ZXhwg=', 'Content-Length': '0'} 2023-07-28T23:51:17.510 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001c19385f072346ee-0064c45475-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.511 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:17.512 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:17.512 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:17.513 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:17.513 DEBUG:boto:Method: GET 2023-07-28T23:51:17.514 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:17.514 DEBUG:boto:Data: 2023-07-28T23:51:17.515 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.515 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:17.515 DEBUG:boto:Port: 8001 2023-07-28T23:51:17.516 DEBUG:boto:Params: {} 2023-07-28T23:51:17.516 DEBUG:boto:Token: None 2023-07-28T23:51:17.517 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/ 2023-07-28T23:51:17.517 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TqnQ5G24uPNdnM79HmlBTN/5oHM= 2023-07-28T23:51:17.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TqnQ5G24uPNdnM79HmlBTN/5oHM=', 'Content-Length': '0'} 2023-07-28T23:51:17.520 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000638f142ffeb2143d-0064c45475-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.521 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:17.521 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:17.522 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:17.522 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:17.522 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:17.523 DEBUG:boto:Method: GET 2023-07-28T23:51:17.523 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:17.524 DEBUG:boto:Data: 2023-07-28T23:51:17.524 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.524 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:17.525 DEBUG:boto:Port: 8000 2023-07-28T23:51:17.525 DEBUG:boto:Params: {} 2023-07-28T23:51:17.525 DEBUG:boto:Token: None 2023-07-28T23:51:17.526 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/ 2023-07-28T23:51:17.526 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TqnQ5G24uPNdnM79HmlBTN/5oHM= 2023-07-28T23:51:17.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TqnQ5G24uPNdnM79HmlBTN/5oHM=', 'Content-Length': '0'} 2023-07-28T23:51:17.529 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d08f8d65bef071e-0064c45475-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.529 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:17.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:17.530 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:17.530 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:17.531 DEBUG:boto:path=/hiiqos-13/obj?versionId=wcK7VKcRTxILi71arrN1zUpJKg8KpNP 2023-07-28T23:51:17.531 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=wcK7VKcRTxILi71arrN1zUpJKg8KpNP 2023-07-28T23:51:17.531 DEBUG:boto:Method: GET 2023-07-28T23:51:17.532 DEBUG:boto:Path: /hiiqos-13/obj?versionId=wcK7VKcRTxILi71arrN1zUpJKg8KpNP 2023-07-28T23:51:17.532 DEBUG:boto:Data: 2023-07-28T23:51:17.533 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.533 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:17.533 DEBUG:boto:Port: 8001 2023-07-28T23:51:17.534 DEBUG:boto:Params: {} 2023-07-28T23:51:17.534 DEBUG:boto:Token: None 2023-07-28T23:51:17.534 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/obj?versionId=wcK7VKcRTxILi71arrN1zUpJKg8KpNP 2023-07-28T23:51:17.535 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CmxQgAwXhDOvHlqa1YEFlPGTpTo= 2023-07-28T23:51:17.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CmxQgAwXhDOvHlqa1YEFlPGTpTo=', 'Content-Length': '0'} 2023-07-28T23:51:17.537 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'wcK7VKcRTxILi71arrN1zUpJKg8KpNP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ba93d73f95f73bc-0064c45475-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.580 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:17.581 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:17.581 DEBUG:boto:path=/hiiqos-13/obj?versionId=wcK7VKcRTxILi71arrN1zUpJKg8KpNP 2023-07-28T23:51:17.582 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=wcK7VKcRTxILi71arrN1zUpJKg8KpNP 2023-07-28T23:51:17.582 DEBUG:boto:Method: GET 2023-07-28T23:51:17.582 DEBUG:boto:Path: /hiiqos-13/obj?versionId=wcK7VKcRTxILi71arrN1zUpJKg8KpNP 2023-07-28T23:51:17.583 DEBUG:boto:Data: 2023-07-28T23:51:17.583 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.583 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:17.584 DEBUG:boto:Port: 8000 2023-07-28T23:51:17.584 DEBUG:boto:Params: {} 2023-07-28T23:51:17.590 DEBUG:boto:Token: None 2023-07-28T23:51:17.591 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/obj?versionId=wcK7VKcRTxILi71arrN1zUpJKg8KpNP 2023-07-28T23:51:17.591 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CmxQgAwXhDOvHlqa1YEFlPGTpTo= 2023-07-28T23:51:17.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CmxQgAwXhDOvHlqa1YEFlPGTpTo=', 'Content-Length': '0'} 2023-07-28T23:51:17.594 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'wcK7VKcRTxILi71arrN1zUpJKg8KpNP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b18502f060ffa7f-0064c45475-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.594 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:17.595 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:17.595 DEBUG:boto:path=/hiiqos-13/obj?versionId=wcK7VKcRTxILi71arrN1zUpJKg8KpNP 2023-07-28T23:51:17.595 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=wcK7VKcRTxILi71arrN1zUpJKg8KpNP 2023-07-28T23:51:17.596 DEBUG:boto:Method: HEAD 2023-07-28T23:51:17.596 DEBUG:boto:Path: /hiiqos-13/obj?versionId=wcK7VKcRTxILi71arrN1zUpJKg8KpNP 2023-07-28T23:51:17.596 DEBUG:boto:Data: 2023-07-28T23:51:17.597 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.597 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:17.597 DEBUG:boto:Port: 8001 2023-07-28T23:51:17.598 DEBUG:boto:Params: {} 2023-07-28T23:51:17.598 DEBUG:boto:Token: None 2023-07-28T23:51:17.598 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/obj?versionId=wcK7VKcRTxILi71arrN1zUpJKg8KpNP 2023-07-28T23:51:17.599 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sjkoxPaUYdjT9ZsW2LzKD11g17w= 2023-07-28T23:51:17.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sjkoxPaUYdjT9ZsW2LzKD11g17w=', 'Content-Length': '0'} 2023-07-28T23:51:17.601 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'wcK7VKcRTxILi71arrN1zUpJKg8KpNP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005fca79efccee783a-0064c45475-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.601 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:17.602 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:17.602 DEBUG:boto:path=/hiiqos-13/obj?versionId=wcK7VKcRTxILi71arrN1zUpJKg8KpNP 2023-07-28T23:51:17.602 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=wcK7VKcRTxILi71arrN1zUpJKg8KpNP 2023-07-28T23:51:17.603 DEBUG:boto:Method: HEAD 2023-07-28T23:51:17.603 DEBUG:boto:Path: /hiiqos-13/obj?versionId=wcK7VKcRTxILi71arrN1zUpJKg8KpNP 2023-07-28T23:51:17.603 DEBUG:boto:Data: 2023-07-28T23:51:17.604 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.604 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:17.604 DEBUG:boto:Port: 8000 2023-07-28T23:51:17.605 DEBUG:boto:Params: {} 2023-07-28T23:51:17.605 DEBUG:boto:Token: None 2023-07-28T23:51:17.605 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/obj?versionId=wcK7VKcRTxILi71arrN1zUpJKg8KpNP 2023-07-28T23:51:17.606 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sjkoxPaUYdjT9ZsW2LzKD11g17w= 2023-07-28T23:51:17.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sjkoxPaUYdjT9ZsW2LzKD11g17w=', 'Content-Length': '0'} 2023-07-28T23:51:17.608 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'wcK7VKcRTxILi71arrN1zUpJKg8KpNP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007393830864b2f8c0-0064c45475-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:17.609 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:17.609 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:17.610 DEBUG:boto:path=/hiiqos-13/obj?versionId=wcK7VKcRTxILi71arrN1zUpJKg8KpNP 2023-07-28T23:51:17.610 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=wcK7VKcRTxILi71arrN1zUpJKg8KpNP 2023-07-28T23:51:17.610 DEBUG:boto:Method: GET 2023-07-28T23:51:17.611 DEBUG:boto:Path: /hiiqos-13/obj?versionId=wcK7VKcRTxILi71arrN1zUpJKg8KpNP 2023-07-28T23:51:17.611 DEBUG:boto:Data: 2023-07-28T23:51:17.611 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.612 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:17.612 DEBUG:boto:Port: 8001 2023-07-28T23:51:17.612 DEBUG:boto:Params: {} 2023-07-28T23:51:17.613 DEBUG:boto:Token: None 2023-07-28T23:51:17.613 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/obj?versionId=wcK7VKcRTxILi71arrN1zUpJKg8KpNP 2023-07-28T23:51:17.614 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CmxQgAwXhDOvHlqa1YEFlPGTpTo= 2023-07-28T23:51:17.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CmxQgAwXhDOvHlqa1YEFlPGTpTo=', 'Content-Length': '0'} 2023-07-28T23:51:17.616 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'wcK7VKcRTxILi71arrN1zUpJKg8KpNP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001213ac624ae6902a-0064c45475-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.616 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:17.617 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:17.617 DEBUG:boto:path=/hiiqos-13/obj?versionId=wcK7VKcRTxILi71arrN1zUpJKg8KpNP 2023-07-28T23:51:17.617 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=wcK7VKcRTxILi71arrN1zUpJKg8KpNP 2023-07-28T23:51:17.618 DEBUG:boto:Method: GET 2023-07-28T23:51:17.618 DEBUG:boto:Path: /hiiqos-13/obj?versionId=wcK7VKcRTxILi71arrN1zUpJKg8KpNP 2023-07-28T23:51:17.618 DEBUG:boto:Data: 2023-07-28T23:51:17.619 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.619 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:17.619 DEBUG:boto:Port: 8000 2023-07-28T23:51:17.620 DEBUG:boto:Params: {} 2023-07-28T23:51:17.620 DEBUG:boto:Token: None 2023-07-28T23:51:17.621 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/obj?versionId=wcK7VKcRTxILi71arrN1zUpJKg8KpNP 2023-07-28T23:51:17.621 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CmxQgAwXhDOvHlqa1YEFlPGTpTo= 2023-07-28T23:51:17.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CmxQgAwXhDOvHlqa1YEFlPGTpTo=', 'Content-Length': '0'} 2023-07-28T23:51:17.623 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'wcK7VKcRTxILi71arrN1zUpJKg8KpNP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e53fc29de508974f-0064c45475-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.664 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:17.665 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:17.665 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:17.665 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:17.666 DEBUG:boto:Method: GET 2023-07-28T23:51:17.666 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:17.667 DEBUG:boto:Data: 2023-07-28T23:51:17.667 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.667 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:17.668 DEBUG:boto:Port: 8001 2023-07-28T23:51:17.668 DEBUG:boto:Params: {} 2023-07-28T23:51:17.668 DEBUG:boto:Token: None 2023-07-28T23:51:17.669 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/ 2023-07-28T23:51:17.669 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TqnQ5G24uPNdnM79HmlBTN/5oHM= 2023-07-28T23:51:17.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TqnQ5G24uPNdnM79HmlBTN/5oHM=', 'Content-Length': '0'} 2023-07-28T23:51:17.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000712ed1ea2cf527fa-0064c45475-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.672 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:17.673 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:17.673 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:17.674 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:17.674 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:17.675 DEBUG:boto:Method: GET 2023-07-28T23:51:17.675 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:17.675 DEBUG:boto:Data: 2023-07-28T23:51:17.676 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.676 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:17.676 DEBUG:boto:Port: 8000 2023-07-28T23:51:17.677 DEBUG:boto:Params: {} 2023-07-28T23:51:17.677 DEBUG:boto:Token: None 2023-07-28T23:51:17.678 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/ 2023-07-28T23:51:17.678 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TqnQ5G24uPNdnM79HmlBTN/5oHM= 2023-07-28T23:51:17.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TqnQ5G24uPNdnM79HmlBTN/5oHM=', 'Content-Length': '0'} 2023-07-28T23:51:17.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027e4f8e6338e50b3-0064c45475-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.724 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:17.725 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:17.725 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:17.726 DEBUG:boto:Method: HEAD 2023-07-28T23:51:17.726 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:51:17.726 DEBUG:boto:Data: 2023-07-28T23:51:17.727 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.727 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:17.727 DEBUG:boto:Port: 8001 2023-07-28T23:51:17.728 DEBUG:boto:Params: {} 2023-07-28T23:51:17.728 DEBUG:boto:Token: None 2023-07-28T23:51:17.728 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/obj 2023-07-28T23:51:17.729 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9LkqZndmQrAsjh22SZmOf2JHjmk= 2023-07-28T23:51:17.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9LkqZndmQrAsjh22SZmOf2JHjmk=', 'Content-Length': '0'} 2023-07-28T23:51:17.732 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e92e873a36c058c-0064c45475-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.733 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:17.733 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:17.733 DEBUG:boto:Method: HEAD 2023-07-28T23:51:17.734 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:51:17.734 DEBUG:boto:Data: 2023-07-28T23:51:17.734 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.735 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:17.735 DEBUG:boto:Port: 8000 2023-07-28T23:51:17.735 DEBUG:boto:Params: {} 2023-07-28T23:51:17.736 DEBUG:boto:Token: None 2023-07-28T23:51:17.736 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/obj 2023-07-28T23:51:17.736 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9LkqZndmQrAsjh22SZmOf2JHjmk= 2023-07-28T23:51:17.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9LkqZndmQrAsjh22SZmOf2JHjmk=', 'Content-Length': '0'} 2023-07-28T23:51:17.740 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e94ecd98c427339-0064c45475-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:17.741 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:17.741 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:17.741 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.742 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.742 DEBUG:boto:Method: GET 2023-07-28T23:51:17.743 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.743 DEBUG:boto:Data: 2023-07-28T23:51:17.743 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.744 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:17.744 DEBUG:boto:Port: 8001 2023-07-28T23:51:17.744 DEBUG:boto:Params: {} 2023-07-28T23:51:17.745 DEBUG:boto:Token: None 2023-07-28T23:51:17.745 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.746 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oHSMk/nDzhuivy21MXwOs1ZXhwg= 2023-07-28T23:51:17.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oHSMk/nDzhuivy21MXwOs1ZXhwg=', 'Content-Length': '0'} 2023-07-28T23:51:17.748 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008322ef24487ade75-0064c45475-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.748 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:17.749 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:17.749 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.749 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.750 DEBUG:boto:Method: GET 2023-07-28T23:51:17.750 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.750 DEBUG:boto:Data: 2023-07-28T23:51:17.751 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.751 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:17.752 DEBUG:boto:Port: 8000 2023-07-28T23:51:17.752 DEBUG:boto:Params: {} 2023-07-28T23:51:17.752 DEBUG:boto:Token: None 2023-07-28T23:51:17.753 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.753 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oHSMk/nDzhuivy21MXwOs1ZXhwg= 2023-07-28T23:51:17.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oHSMk/nDzhuivy21MXwOs1ZXhwg=', 'Content-Length': '0'} 2023-07-28T23:51:17.755 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a6bdd227d56bae23-0064c45475-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.756 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:17.756 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:17.757 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:17.757 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:17.757 DEBUG:boto:Method: GET 2023-07-28T23:51:17.758 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:17.758 DEBUG:boto:Data: 2023-07-28T23:51:17.758 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.759 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:17.759 DEBUG:boto:Port: 8001 2023-07-28T23:51:17.759 DEBUG:boto:Params: {} 2023-07-28T23:51:17.760 DEBUG:boto:Token: None 2023-07-28T23:51:17.760 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/ 2023-07-28T23:51:17.761 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TqnQ5G24uPNdnM79HmlBTN/5oHM= 2023-07-28T23:51:17.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TqnQ5G24uPNdnM79HmlBTN/5oHM=', 'Content-Length': '0'} 2023-07-28T23:51:17.763 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021b8872dac840189-0064c45475-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.764 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:17.764 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:17.765 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:17.765 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:17.765 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:17.766 DEBUG:boto:Method: GET 2023-07-28T23:51:17.766 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:17.766 DEBUG:boto:Data: 2023-07-28T23:51:17.767 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.767 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:17.767 DEBUG:boto:Port: 8000 2023-07-28T23:51:17.768 DEBUG:boto:Params: {} 2023-07-28T23:51:17.768 DEBUG:boto:Token: None 2023-07-28T23:51:17.769 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/ 2023-07-28T23:51:17.769 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TqnQ5G24uPNdnM79HmlBTN/5oHM= 2023-07-28T23:51:17.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TqnQ5G24uPNdnM79HmlBTN/5oHM=', 'Content-Length': '0'} 2023-07-28T23:51:17.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000084d4911f2be718-0064c45475-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.772 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:17.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:17.773 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:17.774 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:17.774 DEBUG:boto:path=/hiiqos-13/obj?versionId=Fg82BDw2qebn-TlTZEmuWfNDUSVLkvC 2023-07-28T23:51:17.774 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=Fg82BDw2qebn-TlTZEmuWfNDUSVLkvC 2023-07-28T23:51:17.775 DEBUG:boto:Method: GET 2023-07-28T23:51:17.775 DEBUG:boto:Path: /hiiqos-13/obj?versionId=Fg82BDw2qebn-TlTZEmuWfNDUSVLkvC 2023-07-28T23:51:17.775 DEBUG:boto:Data: 2023-07-28T23:51:17.776 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.776 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:17.776 DEBUG:boto:Port: 8001 2023-07-28T23:51:17.777 DEBUG:boto:Params: {} 2023-07-28T23:51:17.777 DEBUG:boto:Token: None 2023-07-28T23:51:17.777 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/obj?versionId=Fg82BDw2qebn-TlTZEmuWfNDUSVLkvC 2023-07-28T23:51:17.778 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qvY3LdaSNbkYW4k0KNMq+vUAuDE= 2023-07-28T23:51:17.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qvY3LdaSNbkYW4k0KNMq+vUAuDE=', 'Content-Length': '0'} 2023-07-28T23:51:17.780 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'Fg82BDw2qebn-TlTZEmuWfNDUSVLkvC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e00be4a5d9748049-0064c45475-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.781 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:17.781 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:17.781 DEBUG:boto:path=/hiiqos-13/obj?versionId=Fg82BDw2qebn-TlTZEmuWfNDUSVLkvC 2023-07-28T23:51:17.782 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=Fg82BDw2qebn-TlTZEmuWfNDUSVLkvC 2023-07-28T23:51:17.782 DEBUG:boto:Method: GET 2023-07-28T23:51:17.782 DEBUG:boto:Path: /hiiqos-13/obj?versionId=Fg82BDw2qebn-TlTZEmuWfNDUSVLkvC 2023-07-28T23:51:17.783 DEBUG:boto:Data: 2023-07-28T23:51:17.783 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.783 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:17.784 DEBUG:boto:Port: 8000 2023-07-28T23:51:17.784 DEBUG:boto:Params: {} 2023-07-28T23:51:17.785 DEBUG:boto:Token: None 2023-07-28T23:51:17.785 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/obj?versionId=Fg82BDw2qebn-TlTZEmuWfNDUSVLkvC 2023-07-28T23:51:17.785 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qvY3LdaSNbkYW4k0KNMq+vUAuDE= 2023-07-28T23:51:17.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qvY3LdaSNbkYW4k0KNMq+vUAuDE=', 'Content-Length': '0'} 2023-07-28T23:51:17.788 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'Fg82BDw2qebn-TlTZEmuWfNDUSVLkvC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000295ba261d8958266-0064c45475-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.788 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:17.789 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:17.789 DEBUG:boto:path=/hiiqos-13/obj?versionId=Fg82BDw2qebn-TlTZEmuWfNDUSVLkvC 2023-07-28T23:51:17.789 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=Fg82BDw2qebn-TlTZEmuWfNDUSVLkvC 2023-07-28T23:51:17.790 DEBUG:boto:Method: HEAD 2023-07-28T23:51:17.790 DEBUG:boto:Path: /hiiqos-13/obj?versionId=Fg82BDw2qebn-TlTZEmuWfNDUSVLkvC 2023-07-28T23:51:17.790 DEBUG:boto:Data: 2023-07-28T23:51:17.790 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.791 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:17.791 DEBUG:boto:Port: 8001 2023-07-28T23:51:17.791 DEBUG:boto:Params: {} 2023-07-28T23:51:17.792 DEBUG:boto:Token: None 2023-07-28T23:51:17.792 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/obj?versionId=Fg82BDw2qebn-TlTZEmuWfNDUSVLkvC 2023-07-28T23:51:17.793 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ob3A1Qh9ydA3yjhE3cvCVVkfgh8= 2023-07-28T23:51:17.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ob3A1Qh9ydA3yjhE3cvCVVkfgh8=', 'Content-Length': '0'} 2023-07-28T23:51:17.794 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'Fg82BDw2qebn-TlTZEmuWfNDUSVLkvC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000064eeb5e2d4152abe-0064c45475-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.795 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:17.795 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:17.796 DEBUG:boto:path=/hiiqos-13/obj?versionId=Fg82BDw2qebn-TlTZEmuWfNDUSVLkvC 2023-07-28T23:51:17.796 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=Fg82BDw2qebn-TlTZEmuWfNDUSVLkvC 2023-07-28T23:51:17.796 DEBUG:boto:Method: HEAD 2023-07-28T23:51:17.797 DEBUG:boto:Path: /hiiqos-13/obj?versionId=Fg82BDw2qebn-TlTZEmuWfNDUSVLkvC 2023-07-28T23:51:17.797 DEBUG:boto:Data: 2023-07-28T23:51:17.797 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.798 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:17.798 DEBUG:boto:Port: 8000 2023-07-28T23:51:17.798 DEBUG:boto:Params: {} 2023-07-28T23:51:17.799 DEBUG:boto:Token: None 2023-07-28T23:51:17.799 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/obj?versionId=Fg82BDw2qebn-TlTZEmuWfNDUSVLkvC 2023-07-28T23:51:17.799 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ob3A1Qh9ydA3yjhE3cvCVVkfgh8= 2023-07-28T23:51:17.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ob3A1Qh9ydA3yjhE3cvCVVkfgh8=', 'Content-Length': '0'} 2023-07-28T23:51:17.801 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'Fg82BDw2qebn-TlTZEmuWfNDUSVLkvC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d4ba2ffa4462789a-0064c45475-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:17.802 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:17.803 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:17.803 DEBUG:boto:path=/hiiqos-13/obj?versionId=Fg82BDw2qebn-TlTZEmuWfNDUSVLkvC 2023-07-28T23:51:17.803 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=Fg82BDw2qebn-TlTZEmuWfNDUSVLkvC 2023-07-28T23:51:17.804 DEBUG:boto:Method: GET 2023-07-28T23:51:17.804 DEBUG:boto:Path: /hiiqos-13/obj?versionId=Fg82BDw2qebn-TlTZEmuWfNDUSVLkvC 2023-07-28T23:51:17.804 DEBUG:boto:Data: 2023-07-28T23:51:17.805 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.805 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:17.805 DEBUG:boto:Port: 8001 2023-07-28T23:51:17.806 DEBUG:boto:Params: {} 2023-07-28T23:51:17.806 DEBUG:boto:Token: None 2023-07-28T23:51:17.807 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/obj?versionId=Fg82BDw2qebn-TlTZEmuWfNDUSVLkvC 2023-07-28T23:51:17.807 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qvY3LdaSNbkYW4k0KNMq+vUAuDE= 2023-07-28T23:51:17.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qvY3LdaSNbkYW4k0KNMq+vUAuDE=', 'Content-Length': '0'} 2023-07-28T23:51:17.809 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'Fg82BDw2qebn-TlTZEmuWfNDUSVLkvC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006ad3c86940c75f2b-0064c45475-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.852 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:17.853 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:17.853 DEBUG:boto:path=/hiiqos-13/obj?versionId=Fg82BDw2qebn-TlTZEmuWfNDUSVLkvC 2023-07-28T23:51:17.854 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=Fg82BDw2qebn-TlTZEmuWfNDUSVLkvC 2023-07-28T23:51:17.854 DEBUG:boto:Method: GET 2023-07-28T23:51:17.854 DEBUG:boto:Path: /hiiqos-13/obj?versionId=Fg82BDw2qebn-TlTZEmuWfNDUSVLkvC 2023-07-28T23:51:17.855 DEBUG:boto:Data: 2023-07-28T23:51:17.855 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.855 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:17.856 DEBUG:boto:Port: 8000 2023-07-28T23:51:17.856 DEBUG:boto:Params: {} 2023-07-28T23:51:17.856 DEBUG:boto:Token: None 2023-07-28T23:51:17.857 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/obj?versionId=Fg82BDw2qebn-TlTZEmuWfNDUSVLkvC 2023-07-28T23:51:17.857 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qvY3LdaSNbkYW4k0KNMq+vUAuDE= 2023-07-28T23:51:17.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qvY3LdaSNbkYW4k0KNMq+vUAuDE=', 'Content-Length': '0'} 2023-07-28T23:51:17.860 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'Fg82BDw2qebn-TlTZEmuWfNDUSVLkvC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000068962cb5944b807c-0064c45475-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.860 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:17.861 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:17.861 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:17.861 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:17.862 DEBUG:boto:Method: GET 2023-07-28T23:51:17.862 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:17.862 DEBUG:boto:Data: 2023-07-28T23:51:17.863 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.863 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:17.863 DEBUG:boto:Port: 8001 2023-07-28T23:51:17.864 DEBUG:boto:Params: {} 2023-07-28T23:51:17.864 DEBUG:boto:Token: None 2023-07-28T23:51:17.865 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/ 2023-07-28T23:51:17.865 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TqnQ5G24uPNdnM79HmlBTN/5oHM= 2023-07-28T23:51:17.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TqnQ5G24uPNdnM79HmlBTN/5oHM=', 'Content-Length': '0'} 2023-07-28T23:51:17.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001664fd4e26d0bcd0-0064c45475-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.868 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:17.869 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:17.870 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:17.870 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:17.870 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:17.871 DEBUG:boto:Method: GET 2023-07-28T23:51:17.871 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:17.871 DEBUG:boto:Data: 2023-07-28T23:51:17.872 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.872 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:17.872 DEBUG:boto:Port: 8000 2023-07-28T23:51:17.873 DEBUG:boto:Params: {} 2023-07-28T23:51:17.873 DEBUG:boto:Token: None 2023-07-28T23:51:17.874 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/ 2023-07-28T23:51:17.874 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TqnQ5G24uPNdnM79HmlBTN/5oHM= 2023-07-28T23:51:17.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TqnQ5G24uPNdnM79HmlBTN/5oHM=', 'Content-Length': '0'} 2023-07-28T23:51:17.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a71fe0881f1f37ed-0064c45475-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.877 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:17.878 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:17.878 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:17.879 DEBUG:boto:Method: HEAD 2023-07-28T23:51:17.879 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:51:17.879 DEBUG:boto:Data: 2023-07-28T23:51:17.880 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.880 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:17.880 DEBUG:boto:Port: 8001 2023-07-28T23:51:17.881 DEBUG:boto:Params: {} 2023-07-28T23:51:17.881 DEBUG:boto:Token: None 2023-07-28T23:51:17.881 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/obj 2023-07-28T23:51:17.882 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9LkqZndmQrAsjh22SZmOf2JHjmk= 2023-07-28T23:51:17.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9LkqZndmQrAsjh22SZmOf2JHjmk=', 'Content-Length': '0'} 2023-07-28T23:51:17.885 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004f08f7806f3fa44f-0064c45475-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.885 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:17.886 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:17.886 DEBUG:boto:Method: HEAD 2023-07-28T23:51:17.886 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:51:17.887 DEBUG:boto:Data: 2023-07-28T23:51:17.887 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.887 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:17.888 DEBUG:boto:Port: 8000 2023-07-28T23:51:17.888 DEBUG:boto:Params: {} 2023-07-28T23:51:17.888 DEBUG:boto:Token: None 2023-07-28T23:51:17.889 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/obj 2023-07-28T23:51:17.889 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9LkqZndmQrAsjh22SZmOf2JHjmk= 2023-07-28T23:51:17.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9LkqZndmQrAsjh22SZmOf2JHjmk=', 'Content-Length': '0'} 2023-07-28T23:51:17.893 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b71763ba64ce0c67-0064c45475-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:17.893 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:17.894 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:17.894 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.895 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.895 DEBUG:boto:Method: GET 2023-07-28T23:51:17.895 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.896 DEBUG:boto:Data: 2023-07-28T23:51:17.896 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.896 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:17.897 DEBUG:boto:Port: 8001 2023-07-28T23:51:17.897 DEBUG:boto:Params: {} 2023-07-28T23:51:17.897 DEBUG:boto:Token: None 2023-07-28T23:51:17.898 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.898 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oHSMk/nDzhuivy21MXwOs1ZXhwg= 2023-07-28T23:51:17.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oHSMk/nDzhuivy21MXwOs1ZXhwg=', 'Content-Length': '0'} 2023-07-28T23:51:17.901 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a4ac47fbc2a578f-0064c45475-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.901 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:17.902 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:17.902 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.902 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.903 DEBUG:boto:Method: GET 2023-07-28T23:51:17.903 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.903 DEBUG:boto:Data: 2023-07-28T23:51:17.904 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.904 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:17.905 DEBUG:boto:Port: 8000 2023-07-28T23:51:17.905 DEBUG:boto:Params: {} 2023-07-28T23:51:17.905 DEBUG:boto:Token: None 2023-07-28T23:51:17.906 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:17.906 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oHSMk/nDzhuivy21MXwOs1ZXhwg= 2023-07-28T23:51:17.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oHSMk/nDzhuivy21MXwOs1ZXhwg=', 'Content-Length': '0'} 2023-07-28T23:51:17.908 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000057f43e2e686d25c1-0064c45475-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.952 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:17.953 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:17.953 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:17.954 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:17.954 DEBUG:boto:Method: GET 2023-07-28T23:51:17.954 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:17.955 DEBUG:boto:Data: 2023-07-28T23:51:17.955 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.955 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:17.956 DEBUG:boto:Port: 8001 2023-07-28T23:51:17.956 DEBUG:boto:Params: {} 2023-07-28T23:51:17.957 DEBUG:boto:Token: None 2023-07-28T23:51:17.957 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/ 2023-07-28T23:51:17.957 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TqnQ5G24uPNdnM79HmlBTN/5oHM= 2023-07-28T23:51:17.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TqnQ5G24uPNdnM79HmlBTN/5oHM=', 'Content-Length': '0'} 2023-07-28T23:51:17.960 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efcbb6ba8f680983-0064c45475-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.961 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:17.962 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:17.962 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:17.962 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:17.963 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:17.963 DEBUG:boto:Method: GET 2023-07-28T23:51:17.963 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:17.964 DEBUG:boto:Data: 2023-07-28T23:51:17.964 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.965 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:17.965 DEBUG:boto:Port: 8000 2023-07-28T23:51:17.965 DEBUG:boto:Params: {} 2023-07-28T23:51:17.966 DEBUG:boto:Token: None 2023-07-28T23:51:17.966 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/ 2023-07-28T23:51:17.966 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TqnQ5G24uPNdnM79HmlBTN/5oHM= 2023-07-28T23:51:17.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TqnQ5G24uPNdnM79HmlBTN/5oHM=', 'Content-Length': '0'} 2023-07-28T23:51:17.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000859d3c25153ca64a-0064c45475-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:17.970 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:17.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:17.971 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:17.971 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:17.971 DEBUG:boto:path=/hiiqos-13/obj?versionId=z7auNlW8oH1FJvaHlG2v6quhJD.17R1 2023-07-28T23:51:17.972 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=z7auNlW8oH1FJvaHlG2v6quhJD.17R1 2023-07-28T23:51:17.972 DEBUG:boto:Method: GET 2023-07-28T23:51:17.972 DEBUG:boto:Path: /hiiqos-13/obj?versionId=z7auNlW8oH1FJvaHlG2v6quhJD.17R1 2023-07-28T23:51:17.973 DEBUG:boto:Data: 2023-07-28T23:51:17.973 DEBUG:boto:Headers: {} 2023-07-28T23:51:17.974 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:17.974 DEBUG:boto:Port: 8001 2023-07-28T23:51:17.974 DEBUG:boto:Params: {} 2023-07-28T23:51:17.975 DEBUG:boto:Token: None 2023-07-28T23:51:17.975 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:17 GMT /hiiqos-13/obj?versionId=z7auNlW8oH1FJvaHlG2v6quhJD.17R1 2023-07-28T23:51:17.975 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MQjAh7zKWdzT6FRzvBr+BCYZ1T0= 2023-07-28T23:51:17.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MQjAh7zKWdzT6FRzvBr+BCYZ1T0=', 'Content-Length': '0'} 2023-07-28T23:51:17.978 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'z7auNlW8oH1FJvaHlG2v6quhJD.17R1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000de2a5b886518c354-0064c45475-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.020 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:18.021 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:18.021 DEBUG:boto:path=/hiiqos-13/obj?versionId=z7auNlW8oH1FJvaHlG2v6quhJD.17R1 2023-07-28T23:51:18.021 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=z7auNlW8oH1FJvaHlG2v6quhJD.17R1 2023-07-28T23:51:18.022 DEBUG:boto:Method: GET 2023-07-28T23:51:18.022 DEBUG:boto:Path: /hiiqos-13/obj?versionId=z7auNlW8oH1FJvaHlG2v6quhJD.17R1 2023-07-28T23:51:18.022 DEBUG:boto:Data: 2023-07-28T23:51:18.023 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.023 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:18.023 DEBUG:boto:Port: 8000 2023-07-28T23:51:18.024 DEBUG:boto:Params: {} 2023-07-28T23:51:18.024 DEBUG:boto:Token: None 2023-07-28T23:51:18.025 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/obj?versionId=z7auNlW8oH1FJvaHlG2v6quhJD.17R1 2023-07-28T23:51:18.025 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:V3GObquBaJ2jfFg6t/lJHNvog+E= 2023-07-28T23:51:18.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:V3GObquBaJ2jfFg6t/lJHNvog+E=', 'Content-Length': '0'} 2023-07-28T23:51:18.028 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'z7auNlW8oH1FJvaHlG2v6quhJD.17R1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000022f7dea5729571db-0064c45476-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.028 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:18.029 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:18.029 DEBUG:boto:path=/hiiqos-13/obj?versionId=z7auNlW8oH1FJvaHlG2v6quhJD.17R1 2023-07-28T23:51:18.029 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=z7auNlW8oH1FJvaHlG2v6quhJD.17R1 2023-07-28T23:51:18.030 DEBUG:boto:Method: HEAD 2023-07-28T23:51:18.030 DEBUG:boto:Path: /hiiqos-13/obj?versionId=z7auNlW8oH1FJvaHlG2v6quhJD.17R1 2023-07-28T23:51:18.030 DEBUG:boto:Data: 2023-07-28T23:51:18.031 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.031 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:18.031 DEBUG:boto:Port: 8001 2023-07-28T23:51:18.032 DEBUG:boto:Params: {} 2023-07-28T23:51:18.032 DEBUG:boto:Token: None 2023-07-28T23:51:18.032 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/obj?versionId=z7auNlW8oH1FJvaHlG2v6quhJD.17R1 2023-07-28T23:51:18.033 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:to3AcIax0/XlyQqsPZ9/0VRUoOk= 2023-07-28T23:51:18.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:to3AcIax0/XlyQqsPZ9/0VRUoOk=', 'Content-Length': '0'} 2023-07-28T23:51:18.035 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'z7auNlW8oH1FJvaHlG2v6quhJD.17R1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000516550cadfa295c4-0064c45476-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.035 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:18.036 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:18.036 DEBUG:boto:path=/hiiqos-13/obj?versionId=z7auNlW8oH1FJvaHlG2v6quhJD.17R1 2023-07-28T23:51:18.036 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=z7auNlW8oH1FJvaHlG2v6quhJD.17R1 2023-07-28T23:51:18.037 DEBUG:boto:Method: HEAD 2023-07-28T23:51:18.037 DEBUG:boto:Path: /hiiqos-13/obj?versionId=z7auNlW8oH1FJvaHlG2v6quhJD.17R1 2023-07-28T23:51:18.037 DEBUG:boto:Data: 2023-07-28T23:51:18.038 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.038 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:18.038 DEBUG:boto:Port: 8000 2023-07-28T23:51:18.039 DEBUG:boto:Params: {} 2023-07-28T23:51:18.039 DEBUG:boto:Token: None 2023-07-28T23:51:18.040 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/obj?versionId=z7auNlW8oH1FJvaHlG2v6quhJD.17R1 2023-07-28T23:51:18.040 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:to3AcIax0/XlyQqsPZ9/0VRUoOk= 2023-07-28T23:51:18.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:to3AcIax0/XlyQqsPZ9/0VRUoOk=', 'Content-Length': '0'} 2023-07-28T23:51:18.042 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'z7auNlW8oH1FJvaHlG2v6quhJD.17R1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000046594dc7089e9f47-0064c45476-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:18.043 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:18.044 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:18.044 DEBUG:boto:path=/hiiqos-13/obj?versionId=z7auNlW8oH1FJvaHlG2v6quhJD.17R1 2023-07-28T23:51:18.044 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=z7auNlW8oH1FJvaHlG2v6quhJD.17R1 2023-07-28T23:51:18.045 DEBUG:boto:Method: GET 2023-07-28T23:51:18.045 DEBUG:boto:Path: /hiiqos-13/obj?versionId=z7auNlW8oH1FJvaHlG2v6quhJD.17R1 2023-07-28T23:51:18.045 DEBUG:boto:Data: 2023-07-28T23:51:18.046 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.046 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:18.046 DEBUG:boto:Port: 8001 2023-07-28T23:51:18.047 DEBUG:boto:Params: {} 2023-07-28T23:51:18.047 DEBUG:boto:Token: None 2023-07-28T23:51:18.047 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/obj?versionId=z7auNlW8oH1FJvaHlG2v6quhJD.17R1 2023-07-28T23:51:18.048 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:V3GObquBaJ2jfFg6t/lJHNvog+E= 2023-07-28T23:51:18.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:V3GObquBaJ2jfFg6t/lJHNvog+E=', 'Content-Length': '0'} 2023-07-28T23:51:18.050 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'z7auNlW8oH1FJvaHlG2v6quhJD.17R1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000064f2963050fba9a9-0064c45476-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.051 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:18.051 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:18.052 DEBUG:boto:path=/hiiqos-13/obj?versionId=z7auNlW8oH1FJvaHlG2v6quhJD.17R1 2023-07-28T23:51:18.052 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=z7auNlW8oH1FJvaHlG2v6quhJD.17R1 2023-07-28T23:51:18.053 DEBUG:boto:Method: GET 2023-07-28T23:51:18.053 DEBUG:boto:Path: /hiiqos-13/obj?versionId=z7auNlW8oH1FJvaHlG2v6quhJD.17R1 2023-07-28T23:51:18.053 DEBUG:boto:Data: 2023-07-28T23:51:18.054 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.054 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:18.055 DEBUG:boto:Port: 8000 2023-07-28T23:51:18.055 DEBUG:boto:Params: {} 2023-07-28T23:51:18.055 DEBUG:boto:Token: None 2023-07-28T23:51:18.056 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/obj?versionId=z7auNlW8oH1FJvaHlG2v6quhJD.17R1 2023-07-28T23:51:18.056 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:V3GObquBaJ2jfFg6t/lJHNvog+E= 2023-07-28T23:51:18.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:V3GObquBaJ2jfFg6t/lJHNvog+E=', 'Content-Length': '0'} 2023-07-28T23:51:18.058 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'z7auNlW8oH1FJvaHlG2v6quhJD.17R1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000968efa3faf657c7a-0064c45476-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.059 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:18.060 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:18.060 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:18.060 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:18.061 DEBUG:boto:Method: GET 2023-07-28T23:51:18.061 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:18.062 DEBUG:boto:Data: 2023-07-28T23:51:18.062 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.062 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:18.063 DEBUG:boto:Port: 8001 2023-07-28T23:51:18.063 DEBUG:boto:Params: {} 2023-07-28T23:51:18.063 DEBUG:boto:Token: None 2023-07-28T23:51:18.064 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/ 2023-07-28T23:51:18.064 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2RFaksKPMg2BSPPcI2oCyjSYO08= 2023-07-28T23:51:18.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2RFaksKPMg2BSPPcI2oCyjSYO08=', 'Content-Length': '0'} 2023-07-28T23:51:18.067 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc64ac58994e7d39-0064c45476-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.068 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:18.068 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:18.069 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:18.069 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:18.070 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:18.070 DEBUG:boto:Method: GET 2023-07-28T23:51:18.070 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:18.071 DEBUG:boto:Data: 2023-07-28T23:51:18.071 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.072 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:18.072 DEBUG:boto:Port: 8000 2023-07-28T23:51:18.072 DEBUG:boto:Params: {} 2023-07-28T23:51:18.073 DEBUG:boto:Token: None 2023-07-28T23:51:18.073 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/ 2023-07-28T23:51:18.074 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2RFaksKPMg2BSPPcI2oCyjSYO08= 2023-07-28T23:51:18.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2RFaksKPMg2BSPPcI2oCyjSYO08=', 'Content-Length': '0'} 2023-07-28T23:51:18.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064fae2061230178e-0064c45476-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.077 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:18.078 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:18.078 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:18.079 DEBUG:boto:Method: HEAD 2023-07-28T23:51:18.079 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:51:18.079 DEBUG:boto:Data: 2023-07-28T23:51:18.080 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.080 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:18.080 DEBUG:boto:Port: 8001 2023-07-28T23:51:18.081 DEBUG:boto:Params: {} 2023-07-28T23:51:18.081 DEBUG:boto:Token: None 2023-07-28T23:51:18.082 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/obj 2023-07-28T23:51:18.082 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CmmMNgWv5R3yb8SOimYLsP7AvSQ= 2023-07-28T23:51:18.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CmmMNgWv5R3yb8SOimYLsP7AvSQ=', 'Content-Length': '0'} 2023-07-28T23:51:18.085 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d5593a00547c70f9-0064c45476-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.086 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:18.086 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:18.087 DEBUG:boto:Method: HEAD 2023-07-28T23:51:18.087 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:51:18.088 DEBUG:boto:Data: 2023-07-28T23:51:18.088 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.088 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:18.089 DEBUG:boto:Port: 8000 2023-07-28T23:51:18.089 DEBUG:boto:Params: {} 2023-07-28T23:51:18.089 DEBUG:boto:Token: None 2023-07-28T23:51:18.090 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/obj 2023-07-28T23:51:18.090 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CmmMNgWv5R3yb8SOimYLsP7AvSQ= 2023-07-28T23:51:18.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CmmMNgWv5R3yb8SOimYLsP7AvSQ=', 'Content-Length': '0'} 2023-07-28T23:51:18.093 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000079de0185d4f5c7fb-0064c45476-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:18.094 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:18.095 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:18.095 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:18.096 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:18.096 DEBUG:boto:Method: GET 2023-07-28T23:51:18.096 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:18.097 DEBUG:boto:Data: 2023-07-28T23:51:18.097 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.098 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:18.098 DEBUG:boto:Port: 8001 2023-07-28T23:51:18.098 DEBUG:boto:Params: {} 2023-07-28T23:51:18.099 DEBUG:boto:Token: None 2023-07-28T23:51:18.099 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:18.100 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mMMie89j0sSjmqoZKWUAzL2baP4= 2023-07-28T23:51:18.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mMMie89j0sSjmqoZKWUAzL2baP4=', 'Content-Length': '0'} 2023-07-28T23:51:18.102 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000017390ef0e923fe22-0064c45476-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.144 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:18.145 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:18.145 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:18.145 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:18.146 DEBUG:boto:Method: GET 2023-07-28T23:51:18.146 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:18.147 DEBUG:boto:Data: 2023-07-28T23:51:18.147 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.147 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:18.148 DEBUG:boto:Port: 8000 2023-07-28T23:51:18.148 DEBUG:boto:Params: {} 2023-07-28T23:51:18.148 DEBUG:boto:Token: None 2023-07-28T23:51:18.149 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:18.149 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mMMie89j0sSjmqoZKWUAzL2baP4= 2023-07-28T23:51:18.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mMMie89j0sSjmqoZKWUAzL2baP4=', 'Content-Length': '0'} 2023-07-28T23:51:18.152 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000057fec434ce0d372d-0064c45476-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.152 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:18.153 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:18.153 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:18.153 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:18.154 DEBUG:boto:Method: GET 2023-07-28T23:51:18.154 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:18.155 DEBUG:boto:Data: 2023-07-28T23:51:18.155 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.155 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:18.156 DEBUG:boto:Port: 8001 2023-07-28T23:51:18.156 DEBUG:boto:Params: {} 2023-07-28T23:51:18.156 DEBUG:boto:Token: None 2023-07-28T23:51:18.157 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/ 2023-07-28T23:51:18.157 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2RFaksKPMg2BSPPcI2oCyjSYO08= 2023-07-28T23:51:18.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2RFaksKPMg2BSPPcI2oCyjSYO08=', 'Content-Length': '0'} 2023-07-28T23:51:18.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2ecc37b007bcafc-0064c45476-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.200 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:18.201 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:18.201 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:18.202 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:18.202 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:18.203 DEBUG:boto:Method: GET 2023-07-28T23:51:18.203 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:18.203 DEBUG:boto:Data: 2023-07-28T23:51:18.204 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.204 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:18.204 DEBUG:boto:Port: 8000 2023-07-28T23:51:18.205 DEBUG:boto:Params: {} 2023-07-28T23:51:18.205 DEBUG:boto:Token: None 2023-07-28T23:51:18.206 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/ 2023-07-28T23:51:18.206 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2RFaksKPMg2BSPPcI2oCyjSYO08= 2023-07-28T23:51:18.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2RFaksKPMg2BSPPcI2oCyjSYO08=', 'Content-Length': '0'} 2023-07-28T23:51:18.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000187d2a33a0c729cd-0064c45476-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.210 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:18.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:18.211 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:18.211 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:18.211 DEBUG:boto:path=/hiiqos-13/obj?versionId=KZjcyesGbGSQE5pZSDDLRJ6rT-Mqm8x 2023-07-28T23:51:18.212 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=KZjcyesGbGSQE5pZSDDLRJ6rT-Mqm8x 2023-07-28T23:51:18.212 DEBUG:boto:Method: GET 2023-07-28T23:51:18.212 DEBUG:boto:Path: /hiiqos-13/obj?versionId=KZjcyesGbGSQE5pZSDDLRJ6rT-Mqm8x 2023-07-28T23:51:18.213 DEBUG:boto:Data: 2023-07-28T23:51:18.213 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.214 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:18.214 DEBUG:boto:Port: 8001 2023-07-28T23:51:18.214 DEBUG:boto:Params: {} 2023-07-28T23:51:18.215 DEBUG:boto:Token: None 2023-07-28T23:51:18.215 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/obj?versionId=KZjcyesGbGSQE5pZSDDLRJ6rT-Mqm8x 2023-07-28T23:51:18.215 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7V/WsHwoIxcUYqpNnmozrhFcJmU= 2023-07-28T23:51:18.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7V/WsHwoIxcUYqpNnmozrhFcJmU=', 'Content-Length': '0'} 2023-07-28T23:51:18.218 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'KZjcyesGbGSQE5pZSDDLRJ6rT-Mqm8x'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000abc994a8cf59ef9a-0064c45476-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.218 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:18.219 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:18.219 DEBUG:boto:path=/hiiqos-13/obj?versionId=KZjcyesGbGSQE5pZSDDLRJ6rT-Mqm8x 2023-07-28T23:51:18.220 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=KZjcyesGbGSQE5pZSDDLRJ6rT-Mqm8x 2023-07-28T23:51:18.220 DEBUG:boto:Method: GET 2023-07-28T23:51:18.220 DEBUG:boto:Path: /hiiqos-13/obj?versionId=KZjcyesGbGSQE5pZSDDLRJ6rT-Mqm8x 2023-07-28T23:51:18.221 DEBUG:boto:Data: 2023-07-28T23:51:18.221 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.221 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:18.222 DEBUG:boto:Port: 8000 2023-07-28T23:51:18.222 DEBUG:boto:Params: {} 2023-07-28T23:51:18.222 DEBUG:boto:Token: None 2023-07-28T23:51:18.223 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/obj?versionId=KZjcyesGbGSQE5pZSDDLRJ6rT-Mqm8x 2023-07-28T23:51:18.223 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7V/WsHwoIxcUYqpNnmozrhFcJmU= 2023-07-28T23:51:18.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7V/WsHwoIxcUYqpNnmozrhFcJmU=', 'Content-Length': '0'} 2023-07-28T23:51:18.226 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'KZjcyesGbGSQE5pZSDDLRJ6rT-Mqm8x'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000068e652f02f067d37-0064c45476-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.268 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:18.269 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:18.269 DEBUG:boto:path=/hiiqos-13/obj?versionId=KZjcyesGbGSQE5pZSDDLRJ6rT-Mqm8x 2023-07-28T23:51:18.269 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=KZjcyesGbGSQE5pZSDDLRJ6rT-Mqm8x 2023-07-28T23:51:18.270 DEBUG:boto:Method: HEAD 2023-07-28T23:51:18.270 DEBUG:boto:Path: /hiiqos-13/obj?versionId=KZjcyesGbGSQE5pZSDDLRJ6rT-Mqm8x 2023-07-28T23:51:18.270 DEBUG:boto:Data: 2023-07-28T23:51:18.271 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.271 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:18.271 DEBUG:boto:Port: 8001 2023-07-28T23:51:18.272 DEBUG:boto:Params: {} 2023-07-28T23:51:18.272 DEBUG:boto:Token: None 2023-07-28T23:51:18.272 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/obj?versionId=KZjcyesGbGSQE5pZSDDLRJ6rT-Mqm8x 2023-07-28T23:51:18.273 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:f5MsxRdfc8OCiSpPtLMdDzpRRbs= 2023-07-28T23:51:18.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:f5MsxRdfc8OCiSpPtLMdDzpRRbs=', 'Content-Length': '0'} 2023-07-28T23:51:18.275 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'KZjcyesGbGSQE5pZSDDLRJ6rT-Mqm8x'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a893984e419caaa6-0064c45476-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.276 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:18.276 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:18.277 DEBUG:boto:path=/hiiqos-13/obj?versionId=KZjcyesGbGSQE5pZSDDLRJ6rT-Mqm8x 2023-07-28T23:51:18.277 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=KZjcyesGbGSQE5pZSDDLRJ6rT-Mqm8x 2023-07-28T23:51:18.277 DEBUG:boto:Method: HEAD 2023-07-28T23:51:18.278 DEBUG:boto:Path: /hiiqos-13/obj?versionId=KZjcyesGbGSQE5pZSDDLRJ6rT-Mqm8x 2023-07-28T23:51:18.278 DEBUG:boto:Data: 2023-07-28T23:51:18.278 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.279 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:18.279 DEBUG:boto:Port: 8000 2023-07-28T23:51:18.279 DEBUG:boto:Params: {} 2023-07-28T23:51:18.280 DEBUG:boto:Token: None 2023-07-28T23:51:18.280 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/obj?versionId=KZjcyesGbGSQE5pZSDDLRJ6rT-Mqm8x 2023-07-28T23:51:18.280 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:f5MsxRdfc8OCiSpPtLMdDzpRRbs= 2023-07-28T23:51:18.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:f5MsxRdfc8OCiSpPtLMdDzpRRbs=', 'Content-Length': '0'} 2023-07-28T23:51:18.283 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'KZjcyesGbGSQE5pZSDDLRJ6rT-Mqm8x'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008ca5bc44f74216c1-0064c45476-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:18.284 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:18.284 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:18.284 DEBUG:boto:path=/hiiqos-13/obj?versionId=KZjcyesGbGSQE5pZSDDLRJ6rT-Mqm8x 2023-07-28T23:51:18.285 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=KZjcyesGbGSQE5pZSDDLRJ6rT-Mqm8x 2023-07-28T23:51:18.285 DEBUG:boto:Method: GET 2023-07-28T23:51:18.286 DEBUG:boto:Path: /hiiqos-13/obj?versionId=KZjcyesGbGSQE5pZSDDLRJ6rT-Mqm8x 2023-07-28T23:51:18.286 DEBUG:boto:Data: 2023-07-28T23:51:18.286 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.287 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:18.287 DEBUG:boto:Port: 8001 2023-07-28T23:51:18.287 DEBUG:boto:Params: {} 2023-07-28T23:51:18.288 DEBUG:boto:Token: None 2023-07-28T23:51:18.288 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/obj?versionId=KZjcyesGbGSQE5pZSDDLRJ6rT-Mqm8x 2023-07-28T23:51:18.288 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7V/WsHwoIxcUYqpNnmozrhFcJmU= 2023-07-28T23:51:18.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7V/WsHwoIxcUYqpNnmozrhFcJmU=', 'Content-Length': '0'} 2023-07-28T23:51:18.291 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'KZjcyesGbGSQE5pZSDDLRJ6rT-Mqm8x'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f8570e49d57dd042-0064c45476-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.291 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:18.292 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:18.292 DEBUG:boto:path=/hiiqos-13/obj?versionId=KZjcyesGbGSQE5pZSDDLRJ6rT-Mqm8x 2023-07-28T23:51:18.292 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=KZjcyesGbGSQE5pZSDDLRJ6rT-Mqm8x 2023-07-28T23:51:18.293 DEBUG:boto:Method: GET 2023-07-28T23:51:18.293 DEBUG:boto:Path: /hiiqos-13/obj?versionId=KZjcyesGbGSQE5pZSDDLRJ6rT-Mqm8x 2023-07-28T23:51:18.293 DEBUG:boto:Data: 2023-07-28T23:51:18.294 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.294 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:18.294 DEBUG:boto:Port: 8000 2023-07-28T23:51:18.295 DEBUG:boto:Params: {} 2023-07-28T23:51:18.295 DEBUG:boto:Token: None 2023-07-28T23:51:18.296 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/obj?versionId=KZjcyesGbGSQE5pZSDDLRJ6rT-Mqm8x 2023-07-28T23:51:18.296 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7V/WsHwoIxcUYqpNnmozrhFcJmU= 2023-07-28T23:51:18.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7V/WsHwoIxcUYqpNnmozrhFcJmU=', 'Content-Length': '0'} 2023-07-28T23:51:18.298 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'KZjcyesGbGSQE5pZSDDLRJ6rT-Mqm8x'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002f9231168f616818-0064c45476-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.299 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:18.299 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:18.300 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:18.300 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:18.300 DEBUG:boto:Method: GET 2023-07-28T23:51:18.301 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:18.301 DEBUG:boto:Data: 2023-07-28T23:51:18.301 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.302 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:18.302 DEBUG:boto:Port: 8001 2023-07-28T23:51:18.302 DEBUG:boto:Params: {} 2023-07-28T23:51:18.303 DEBUG:boto:Token: None 2023-07-28T23:51:18.303 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/ 2023-07-28T23:51:18.304 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2RFaksKPMg2BSPPcI2oCyjSYO08= 2023-07-28T23:51:18.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2RFaksKPMg2BSPPcI2oCyjSYO08=', 'Content-Length': '0'} 2023-07-28T23:51:18.307 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034968c48bd4ea157-0064c45476-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.307 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:18.308 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:18.308 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:18.308 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:18.309 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:18.309 DEBUG:boto:Method: GET 2023-07-28T23:51:18.309 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:18.310 DEBUG:boto:Data: 2023-07-28T23:51:18.310 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.311 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:18.311 DEBUG:boto:Port: 8000 2023-07-28T23:51:18.311 DEBUG:boto:Params: {} 2023-07-28T23:51:18.312 DEBUG:boto:Token: None 2023-07-28T23:51:18.312 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/ 2023-07-28T23:51:18.312 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2RFaksKPMg2BSPPcI2oCyjSYO08= 2023-07-28T23:51:18.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2RFaksKPMg2BSPPcI2oCyjSYO08=', 'Content-Length': '0'} 2023-07-28T23:51:18.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e378004f28dc3f4b-0064c45476-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.316 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:18.316 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:18.317 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:18.317 DEBUG:boto:Method: HEAD 2023-07-28T23:51:18.317 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:51:18.318 DEBUG:boto:Data: 2023-07-28T23:51:18.318 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.318 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:18.319 DEBUG:boto:Port: 8001 2023-07-28T23:51:18.319 DEBUG:boto:Params: {} 2023-07-28T23:51:18.319 DEBUG:boto:Token: None 2023-07-28T23:51:18.320 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/obj 2023-07-28T23:51:18.320 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CmmMNgWv5R3yb8SOimYLsP7AvSQ= 2023-07-28T23:51:18.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CmmMNgWv5R3yb8SOimYLsP7AvSQ=', 'Content-Length': '0'} 2023-07-28T23:51:18.324 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000008a92ea693c6fa29-0064c45476-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.324 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:18.325 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:18.325 DEBUG:boto:Method: HEAD 2023-07-28T23:51:18.325 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:51:18.326 DEBUG:boto:Data: 2023-07-28T23:51:18.326 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.326 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:18.327 DEBUG:boto:Port: 8000 2023-07-28T23:51:18.327 DEBUG:boto:Params: {} 2023-07-28T23:51:18.327 DEBUG:boto:Token: None 2023-07-28T23:51:18.328 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/obj 2023-07-28T23:51:18.328 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CmmMNgWv5R3yb8SOimYLsP7AvSQ= 2023-07-28T23:51:18.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CmmMNgWv5R3yb8SOimYLsP7AvSQ=', 'Content-Length': '0'} 2023-07-28T23:51:18.332 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000028b88bd9b4e9666f-0064c45476-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:18.333 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:18.333 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:18.333 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:18.334 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:18.334 DEBUG:boto:Method: GET 2023-07-28T23:51:18.334 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:18.335 DEBUG:boto:Data: 2023-07-28T23:51:18.335 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.335 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:18.336 DEBUG:boto:Port: 8001 2023-07-28T23:51:18.336 DEBUG:boto:Params: {} 2023-07-28T23:51:18.337 DEBUG:boto:Token: None 2023-07-28T23:51:18.337 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:18.337 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mMMie89j0sSjmqoZKWUAzL2baP4= 2023-07-28T23:51:18.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mMMie89j0sSjmqoZKWUAzL2baP4=', 'Content-Length': '0'} 2023-07-28T23:51:18.340 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000053e868c02eccf7f-0064c45476-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.380 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:18.381 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:18.381 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:18.381 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:18.382 DEBUG:boto:Method: GET 2023-07-28T23:51:18.382 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:18.382 DEBUG:boto:Data: 2023-07-28T23:51:18.383 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.383 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:18.384 DEBUG:boto:Port: 8000 2023-07-28T23:51:18.384 DEBUG:boto:Params: {} 2023-07-28T23:51:18.384 DEBUG:boto:Token: None 2023-07-28T23:51:18.385 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:18.385 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mMMie89j0sSjmqoZKWUAzL2baP4= 2023-07-28T23:51:18.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mMMie89j0sSjmqoZKWUAzL2baP4=', 'Content-Length': '0'} 2023-07-28T23:51:18.387 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001161c8b18ff72b49-0064c45476-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.388 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:18.389 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:18.389 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:18.389 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:18.390 DEBUG:boto:Method: GET 2023-07-28T23:51:18.390 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:18.390 DEBUG:boto:Data: 2023-07-28T23:51:18.391 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.391 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:18.391 DEBUG:boto:Port: 8001 2023-07-28T23:51:18.392 DEBUG:boto:Params: {} 2023-07-28T23:51:18.392 DEBUG:boto:Token: None 2023-07-28T23:51:18.393 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/ 2023-07-28T23:51:18.393 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2RFaksKPMg2BSPPcI2oCyjSYO08= 2023-07-28T23:51:18.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2RFaksKPMg2BSPPcI2oCyjSYO08=', 'Content-Length': '0'} 2023-07-28T23:51:18.396 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000115cefe3139ed413-0064c45476-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.397 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:18.397 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:18.398 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:18.398 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:18.398 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:18.399 DEBUG:boto:Method: GET 2023-07-28T23:51:18.399 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:18.399 DEBUG:boto:Data: 2023-07-28T23:51:18.400 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.400 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:18.400 DEBUG:boto:Port: 8000 2023-07-28T23:51:18.401 DEBUG:boto:Params: {} 2023-07-28T23:51:18.401 DEBUG:boto:Token: None 2023-07-28T23:51:18.402 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/ 2023-07-28T23:51:18.402 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2RFaksKPMg2BSPPcI2oCyjSYO08= 2023-07-28T23:51:18.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2RFaksKPMg2BSPPcI2oCyjSYO08=', 'Content-Length': '0'} 2023-07-28T23:51:18.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e7946c9de493be4-0064c45476-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.405 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:18.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:18.406 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:18.407 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:18.407 DEBUG:boto:path=/hiiqos-13/obj?versionId=WxYcV1Cw2jkGBm-hoDmeDENoWMoiniX 2023-07-28T23:51:18.407 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=WxYcV1Cw2jkGBm-hoDmeDENoWMoiniX 2023-07-28T23:51:18.408 DEBUG:boto:Method: GET 2023-07-28T23:51:18.408 DEBUG:boto:Path: /hiiqos-13/obj?versionId=WxYcV1Cw2jkGBm-hoDmeDENoWMoiniX 2023-07-28T23:51:18.408 DEBUG:boto:Data: 2023-07-28T23:51:18.409 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.409 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:18.409 DEBUG:boto:Port: 8001 2023-07-28T23:51:18.410 DEBUG:boto:Params: {} 2023-07-28T23:51:18.410 DEBUG:boto:Token: None 2023-07-28T23:51:18.411 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/obj?versionId=WxYcV1Cw2jkGBm-hoDmeDENoWMoiniX 2023-07-28T23:51:18.411 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5LNZIbPdTIt4naTnhNE+WyOiLz4= 2023-07-28T23:51:18.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5LNZIbPdTIt4naTnhNE+WyOiLz4=', 'Content-Length': '0'} 2023-07-28T23:51:18.414 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'WxYcV1Cw2jkGBm-hoDmeDENoWMoiniX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000edfa84c2fc5def9b-0064c45476-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.456 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:18.457 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:18.457 DEBUG:boto:path=/hiiqos-13/obj?versionId=WxYcV1Cw2jkGBm-hoDmeDENoWMoiniX 2023-07-28T23:51:18.457 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=WxYcV1Cw2jkGBm-hoDmeDENoWMoiniX 2023-07-28T23:51:18.458 DEBUG:boto:Method: GET 2023-07-28T23:51:18.458 DEBUG:boto:Path: /hiiqos-13/obj?versionId=WxYcV1Cw2jkGBm-hoDmeDENoWMoiniX 2023-07-28T23:51:18.459 DEBUG:boto:Data: 2023-07-28T23:51:18.459 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.459 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:18.460 DEBUG:boto:Port: 8000 2023-07-28T23:51:18.460 DEBUG:boto:Params: {} 2023-07-28T23:51:18.460 DEBUG:boto:Token: None 2023-07-28T23:51:18.461 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/obj?versionId=WxYcV1Cw2jkGBm-hoDmeDENoWMoiniX 2023-07-28T23:51:18.461 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5LNZIbPdTIt4naTnhNE+WyOiLz4= 2023-07-28T23:51:18.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5LNZIbPdTIt4naTnhNE+WyOiLz4=', 'Content-Length': '0'} 2023-07-28T23:51:18.464 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'WxYcV1Cw2jkGBm-hoDmeDENoWMoiniX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004deafaa3c42f7447-0064c45476-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.465 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:18.465 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:18.465 DEBUG:boto:path=/hiiqos-13/obj?versionId=WxYcV1Cw2jkGBm-hoDmeDENoWMoiniX 2023-07-28T23:51:18.466 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=WxYcV1Cw2jkGBm-hoDmeDENoWMoiniX 2023-07-28T23:51:18.466 DEBUG:boto:Method: HEAD 2023-07-28T23:51:18.466 DEBUG:boto:Path: /hiiqos-13/obj?versionId=WxYcV1Cw2jkGBm-hoDmeDENoWMoiniX 2023-07-28T23:51:18.467 DEBUG:boto:Data: 2023-07-28T23:51:18.467 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.467 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:18.468 DEBUG:boto:Port: 8001 2023-07-28T23:51:18.468 DEBUG:boto:Params: {} 2023-07-28T23:51:18.468 DEBUG:boto:Token: None 2023-07-28T23:51:18.469 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/obj?versionId=WxYcV1Cw2jkGBm-hoDmeDENoWMoiniX 2023-07-28T23:51:18.469 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FG5oKA+c2BUQkYoApvzHuaXxseQ= 2023-07-28T23:51:18.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FG5oKA+c2BUQkYoApvzHuaXxseQ=', 'Content-Length': '0'} 2023-07-28T23:51:18.471 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'WxYcV1Cw2jkGBm-hoDmeDENoWMoiniX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000446ad2f870d7721b-0064c45476-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.472 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:18.472 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:18.473 DEBUG:boto:path=/hiiqos-13/obj?versionId=WxYcV1Cw2jkGBm-hoDmeDENoWMoiniX 2023-07-28T23:51:18.473 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=WxYcV1Cw2jkGBm-hoDmeDENoWMoiniX 2023-07-28T23:51:18.473 DEBUG:boto:Method: HEAD 2023-07-28T23:51:18.474 DEBUG:boto:Path: /hiiqos-13/obj?versionId=WxYcV1Cw2jkGBm-hoDmeDENoWMoiniX 2023-07-28T23:51:18.474 DEBUG:boto:Data: 2023-07-28T23:51:18.474 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.475 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:18.475 DEBUG:boto:Port: 8000 2023-07-28T23:51:18.475 DEBUG:boto:Params: {} 2023-07-28T23:51:18.476 DEBUG:boto:Token: None 2023-07-28T23:51:18.476 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/obj?versionId=WxYcV1Cw2jkGBm-hoDmeDENoWMoiniX 2023-07-28T23:51:18.477 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FG5oKA+c2BUQkYoApvzHuaXxseQ= 2023-07-28T23:51:18.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FG5oKA+c2BUQkYoApvzHuaXxseQ=', 'Content-Length': '0'} 2023-07-28T23:51:18.479 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'WxYcV1Cw2jkGBm-hoDmeDENoWMoiniX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000192c5bba79c34ee1-0064c45476-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:18.480 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:18.480 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:18.481 DEBUG:boto:path=/hiiqos-13/obj?versionId=WxYcV1Cw2jkGBm-hoDmeDENoWMoiniX 2023-07-28T23:51:18.481 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=WxYcV1Cw2jkGBm-hoDmeDENoWMoiniX 2023-07-28T23:51:18.481 DEBUG:boto:Method: GET 2023-07-28T23:51:18.482 DEBUG:boto:Path: /hiiqos-13/obj?versionId=WxYcV1Cw2jkGBm-hoDmeDENoWMoiniX 2023-07-28T23:51:18.482 DEBUG:boto:Data: 2023-07-28T23:51:18.482 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.483 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:18.483 DEBUG:boto:Port: 8001 2023-07-28T23:51:18.483 DEBUG:boto:Params: {} 2023-07-28T23:51:18.484 DEBUG:boto:Token: None 2023-07-28T23:51:18.484 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/obj?versionId=WxYcV1Cw2jkGBm-hoDmeDENoWMoiniX 2023-07-28T23:51:18.485 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5LNZIbPdTIt4naTnhNE+WyOiLz4= 2023-07-28T23:51:18.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5LNZIbPdTIt4naTnhNE+WyOiLz4=', 'Content-Length': '0'} 2023-07-28T23:51:18.487 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'WxYcV1Cw2jkGBm-hoDmeDENoWMoiniX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004f1f80fe7e1edcd2-0064c45476-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.528 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:18.529 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:18.530 DEBUG:boto:path=/hiiqos-13/obj?versionId=WxYcV1Cw2jkGBm-hoDmeDENoWMoiniX 2023-07-28T23:51:18.530 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=WxYcV1Cw2jkGBm-hoDmeDENoWMoiniX 2023-07-28T23:51:18.530 DEBUG:boto:Method: GET 2023-07-28T23:51:18.531 DEBUG:boto:Path: /hiiqos-13/obj?versionId=WxYcV1Cw2jkGBm-hoDmeDENoWMoiniX 2023-07-28T23:51:18.531 DEBUG:boto:Data: 2023-07-28T23:51:18.531 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.532 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:18.533 DEBUG:boto:Port: 8000 2023-07-28T23:51:18.533 DEBUG:boto:Params: {} 2023-07-28T23:51:18.534 DEBUG:boto:Token: None 2023-07-28T23:51:18.535 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/obj?versionId=WxYcV1Cw2jkGBm-hoDmeDENoWMoiniX 2023-07-28T23:51:18.536 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5LNZIbPdTIt4naTnhNE+WyOiLz4= 2023-07-28T23:51:18.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5LNZIbPdTIt4naTnhNE+WyOiLz4=', 'Content-Length': '0'} 2023-07-28T23:51:18.539 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'WxYcV1Cw2jkGBm-hoDmeDENoWMoiniX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d9b4a757dfddd73c-0064c45476-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.540 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:18.541 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:18.541 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:18.541 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:18.542 DEBUG:boto:Method: GET 2023-07-28T23:51:18.542 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:18.542 DEBUG:boto:Data: 2023-07-28T23:51:18.543 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.543 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:18.543 DEBUG:boto:Port: 8001 2023-07-28T23:51:18.544 DEBUG:boto:Params: {} 2023-07-28T23:51:18.544 DEBUG:boto:Token: None 2023-07-28T23:51:18.545 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/ 2023-07-28T23:51:18.545 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2RFaksKPMg2BSPPcI2oCyjSYO08= 2023-07-28T23:51:18.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2RFaksKPMg2BSPPcI2oCyjSYO08=', 'Content-Length': '0'} 2023-07-28T23:51:18.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5ec2eb4722fac10-0064c45476-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.549 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:18.549 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:18.550 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:18.550 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:18.550 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:18.551 DEBUG:boto:Method: GET 2023-07-28T23:51:18.551 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:18.552 DEBUG:boto:Data: 2023-07-28T23:51:18.552 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.552 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:18.553 DEBUG:boto:Port: 8000 2023-07-28T23:51:18.553 DEBUG:boto:Params: {} 2023-07-28T23:51:18.554 DEBUG:boto:Token: None 2023-07-28T23:51:18.554 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/ 2023-07-28T23:51:18.554 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2RFaksKPMg2BSPPcI2oCyjSYO08= 2023-07-28T23:51:18.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2RFaksKPMg2BSPPcI2oCyjSYO08=', 'Content-Length': '0'} 2023-07-28T23:51:18.557 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000998968d73260fdd3-0064c45476-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.558 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:18.558 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:18.559 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:18.559 DEBUG:boto:Method: HEAD 2023-07-28T23:51:18.559 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:51:18.560 DEBUG:boto:Data: 2023-07-28T23:51:18.560 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.561 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:18.561 DEBUG:boto:Port: 8001 2023-07-28T23:51:18.561 DEBUG:boto:Params: {} 2023-07-28T23:51:18.562 DEBUG:boto:Token: None 2023-07-28T23:51:18.562 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/obj 2023-07-28T23:51:18.562 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CmmMNgWv5R3yb8SOimYLsP7AvSQ= 2023-07-28T23:51:18.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CmmMNgWv5R3yb8SOimYLsP7AvSQ=', 'Content-Length': '0'} 2023-07-28T23:51:18.566 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000023bb19fe92636703-0064c45476-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.566 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:18.567 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:18.567 DEBUG:boto:Method: HEAD 2023-07-28T23:51:18.567 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:51:18.568 DEBUG:boto:Data: 2023-07-28T23:51:18.568 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.568 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:18.569 DEBUG:boto:Port: 8000 2023-07-28T23:51:18.569 DEBUG:boto:Params: {} 2023-07-28T23:51:18.570 DEBUG:boto:Token: None 2023-07-28T23:51:18.570 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/obj 2023-07-28T23:51:18.570 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CmmMNgWv5R3yb8SOimYLsP7AvSQ= 2023-07-28T23:51:18.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CmmMNgWv5R3yb8SOimYLsP7AvSQ=', 'Content-Length': '0'} 2023-07-28T23:51:18.574 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd659256d9423552-0064c45476-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:18.575 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:18.575 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:18.575 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:18.576 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:18.576 DEBUG:boto:Method: GET 2023-07-28T23:51:18.577 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:18.577 DEBUG:boto:Data: 2023-07-28T23:51:18.577 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.578 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:18.578 DEBUG:boto:Port: 8001 2023-07-28T23:51:18.578 DEBUG:boto:Params: {} 2023-07-28T23:51:18.579 DEBUG:boto:Token: None 2023-07-28T23:51:18.579 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:18.580 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mMMie89j0sSjmqoZKWUAzL2baP4= 2023-07-28T23:51:18.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mMMie89j0sSjmqoZKWUAzL2baP4=', 'Content-Length': '0'} 2023-07-28T23:51:18.582 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c82b871ba2bb7e42-0064c45476-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.624 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:18.625 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:18.625 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:18.625 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:18.626 DEBUG:boto:Method: GET 2023-07-28T23:51:18.626 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:18.626 DEBUG:boto:Data: 2023-07-28T23:51:18.627 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.627 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:18.628 DEBUG:boto:Port: 8000 2023-07-28T23:51:18.628 DEBUG:boto:Params: {} 2023-07-28T23:51:18.628 DEBUG:boto:Token: None 2023-07-28T23:51:18.629 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:18.629 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mMMie89j0sSjmqoZKWUAzL2baP4= 2023-07-28T23:51:18.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mMMie89j0sSjmqoZKWUAzL2baP4=', 'Content-Length': '0'} 2023-07-28T23:51:18.632 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e96c3296d72d6491-0064c45476-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.632 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:18.633 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:18.633 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:18.633 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:18.634 DEBUG:boto:Method: GET 2023-07-28T23:51:18.634 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:18.635 DEBUG:boto:Data: 2023-07-28T23:51:18.635 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.635 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:18.636 DEBUG:boto:Port: 8001 2023-07-28T23:51:18.636 DEBUG:boto:Params: {} 2023-07-28T23:51:18.636 DEBUG:boto:Token: None 2023-07-28T23:51:18.637 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/ 2023-07-28T23:51:18.637 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2RFaksKPMg2BSPPcI2oCyjSYO08= 2023-07-28T23:51:18.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2RFaksKPMg2BSPPcI2oCyjSYO08=', 'Content-Length': '0'} 2023-07-28T23:51:18.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071fbe0c8d6d79243-0064c45476-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.641 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:18.641 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:18.642 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:18.642 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:18.643 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:18.643 DEBUG:boto:Method: GET 2023-07-28T23:51:18.643 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:18.644 DEBUG:boto:Data: 2023-07-28T23:51:18.644 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.644 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:18.645 DEBUG:boto:Port: 8000 2023-07-28T23:51:18.645 DEBUG:boto:Params: {} 2023-07-28T23:51:18.645 DEBUG:boto:Token: None 2023-07-28T23:51:18.646 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/ 2023-07-28T23:51:18.646 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2RFaksKPMg2BSPPcI2oCyjSYO08= 2023-07-28T23:51:18.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2RFaksKPMg2BSPPcI2oCyjSYO08=', 'Content-Length': '0'} 2023-07-28T23:51:18.649 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec180c2b0d3149bc-0064c45476-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.650 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:18.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:18.651 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:18.651 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:18.652 DEBUG:boto:path=/hiiqos-13/obj?versionId=2N3xN1haVyocj8z79kCS3UT0nbjXj6Z 2023-07-28T23:51:18.652 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=2N3xN1haVyocj8z79kCS3UT0nbjXj6Z 2023-07-28T23:51:18.652 DEBUG:boto:Method: GET 2023-07-28T23:51:18.653 DEBUG:boto:Path: /hiiqos-13/obj?versionId=2N3xN1haVyocj8z79kCS3UT0nbjXj6Z 2023-07-28T23:51:18.653 DEBUG:boto:Data: 2023-07-28T23:51:18.654 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.654 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:18.654 DEBUG:boto:Port: 8001 2023-07-28T23:51:18.655 DEBUG:boto:Params: {} 2023-07-28T23:51:18.655 DEBUG:boto:Token: None 2023-07-28T23:51:18.655 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/obj?versionId=2N3xN1haVyocj8z79kCS3UT0nbjXj6Z 2023-07-28T23:51:18.656 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CxzXZGf8LYpaPad4K596Y1yn3VA= 2023-07-28T23:51:18.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CxzXZGf8LYpaPad4K596Y1yn3VA=', 'Content-Length': '0'} 2023-07-28T23:51:18.658 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', '2N3xN1haVyocj8z79kCS3UT0nbjXj6Z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000040a52667326517ad-0064c45476-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.659 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:18.659 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:18.660 DEBUG:boto:path=/hiiqos-13/obj?versionId=2N3xN1haVyocj8z79kCS3UT0nbjXj6Z 2023-07-28T23:51:18.660 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=2N3xN1haVyocj8z79kCS3UT0nbjXj6Z 2023-07-28T23:51:18.660 DEBUG:boto:Method: GET 2023-07-28T23:51:18.661 DEBUG:boto:Path: /hiiqos-13/obj?versionId=2N3xN1haVyocj8z79kCS3UT0nbjXj6Z 2023-07-28T23:51:18.661 DEBUG:boto:Data: 2023-07-28T23:51:18.662 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.662 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:18.662 DEBUG:boto:Port: 8000 2023-07-28T23:51:18.663 DEBUG:boto:Params: {} 2023-07-28T23:51:18.663 DEBUG:boto:Token: None 2023-07-28T23:51:18.664 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/obj?versionId=2N3xN1haVyocj8z79kCS3UT0nbjXj6Z 2023-07-28T23:51:18.664 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CxzXZGf8LYpaPad4K596Y1yn3VA= 2023-07-28T23:51:18.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CxzXZGf8LYpaPad4K596Y1yn3VA=', 'Content-Length': '0'} 2023-07-28T23:51:18.666 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', '2N3xN1haVyocj8z79kCS3UT0nbjXj6Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d5f84cc2e9116555-0064c45476-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.708 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:18.709 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:18.709 DEBUG:boto:path=/hiiqos-13/obj?versionId=2N3xN1haVyocj8z79kCS3UT0nbjXj6Z 2023-07-28T23:51:18.710 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=2N3xN1haVyocj8z79kCS3UT0nbjXj6Z 2023-07-28T23:51:18.710 DEBUG:boto:Method: HEAD 2023-07-28T23:51:18.710 DEBUG:boto:Path: /hiiqos-13/obj?versionId=2N3xN1haVyocj8z79kCS3UT0nbjXj6Z 2023-07-28T23:51:18.711 DEBUG:boto:Data: 2023-07-28T23:51:18.711 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.711 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:18.712 DEBUG:boto:Port: 8001 2023-07-28T23:51:18.712 DEBUG:boto:Params: {} 2023-07-28T23:51:18.712 DEBUG:boto:Token: None 2023-07-28T23:51:18.713 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/obj?versionId=2N3xN1haVyocj8z79kCS3UT0nbjXj6Z 2023-07-28T23:51:18.713 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ou9nMnOmStaq9/7igZKSLUtbhO0= 2023-07-28T23:51:18.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ou9nMnOmStaq9/7igZKSLUtbhO0=', 'Content-Length': '0'} 2023-07-28T23:51:18.716 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', '2N3xN1haVyocj8z79kCS3UT0nbjXj6Z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bcca8e0d3940c73e-0064c45476-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.716 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:18.717 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:18.717 DEBUG:boto:path=/hiiqos-13/obj?versionId=2N3xN1haVyocj8z79kCS3UT0nbjXj6Z 2023-07-28T23:51:18.717 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=2N3xN1haVyocj8z79kCS3UT0nbjXj6Z 2023-07-28T23:51:18.718 DEBUG:boto:Method: HEAD 2023-07-28T23:51:18.718 DEBUG:boto:Path: /hiiqos-13/obj?versionId=2N3xN1haVyocj8z79kCS3UT0nbjXj6Z 2023-07-28T23:51:18.718 DEBUG:boto:Data: 2023-07-28T23:51:18.718 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.719 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:18.719 DEBUG:boto:Port: 8000 2023-07-28T23:51:18.719 DEBUG:boto:Params: {} 2023-07-28T23:51:18.720 DEBUG:boto:Token: None 2023-07-28T23:51:18.720 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/obj?versionId=2N3xN1haVyocj8z79kCS3UT0nbjXj6Z 2023-07-28T23:51:18.721 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ou9nMnOmStaq9/7igZKSLUtbhO0= 2023-07-28T23:51:18.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ou9nMnOmStaq9/7igZKSLUtbhO0=', 'Content-Length': '0'} 2023-07-28T23:51:18.723 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', '2N3xN1haVyocj8z79kCS3UT0nbjXj6Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad5df5dbb1fb22f7-0064c45476-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:18.724 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:18.724 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:18.725 DEBUG:boto:path=/hiiqos-13/obj?versionId=2N3xN1haVyocj8z79kCS3UT0nbjXj6Z 2023-07-28T23:51:18.725 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=2N3xN1haVyocj8z79kCS3UT0nbjXj6Z 2023-07-28T23:51:18.726 DEBUG:boto:Method: GET 2023-07-28T23:51:18.726 DEBUG:boto:Path: /hiiqos-13/obj?versionId=2N3xN1haVyocj8z79kCS3UT0nbjXj6Z 2023-07-28T23:51:18.726 DEBUG:boto:Data: 2023-07-28T23:51:18.727 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.727 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:18.727 DEBUG:boto:Port: 8001 2023-07-28T23:51:18.728 DEBUG:boto:Params: {} 2023-07-28T23:51:18.728 DEBUG:boto:Token: None 2023-07-28T23:51:18.729 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/obj?versionId=2N3xN1haVyocj8z79kCS3UT0nbjXj6Z 2023-07-28T23:51:18.729 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CxzXZGf8LYpaPad4K596Y1yn3VA= 2023-07-28T23:51:18.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CxzXZGf8LYpaPad4K596Y1yn3VA=', 'Content-Length': '0'} 2023-07-28T23:51:18.731 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', '2N3xN1haVyocj8z79kCS3UT0nbjXj6Z'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fd2b6b3ee6721308-0064c45476-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.772 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:18.773 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:18.773 DEBUG:boto:path=/hiiqos-13/obj?versionId=2N3xN1haVyocj8z79kCS3UT0nbjXj6Z 2023-07-28T23:51:18.774 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=2N3xN1haVyocj8z79kCS3UT0nbjXj6Z 2023-07-28T23:51:18.774 DEBUG:boto:Method: GET 2023-07-28T23:51:18.774 DEBUG:boto:Path: /hiiqos-13/obj?versionId=2N3xN1haVyocj8z79kCS3UT0nbjXj6Z 2023-07-28T23:51:18.775 DEBUG:boto:Data: 2023-07-28T23:51:18.775 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.776 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:18.776 DEBUG:boto:Port: 8000 2023-07-28T23:51:18.776 DEBUG:boto:Params: {} 2023-07-28T23:51:18.777 DEBUG:boto:Token: None 2023-07-28T23:51:18.777 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/obj?versionId=2N3xN1haVyocj8z79kCS3UT0nbjXj6Z 2023-07-28T23:51:18.778 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CxzXZGf8LYpaPad4K596Y1yn3VA= 2023-07-28T23:51:18.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CxzXZGf8LYpaPad4K596Y1yn3VA=', 'Content-Length': '0'} 2023-07-28T23:51:18.780 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', '2N3xN1haVyocj8z79kCS3UT0nbjXj6Z'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000790d6b41dbc58b7d-0064c45476-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.781 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:18.781 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:18.781 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:18.782 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:18.782 DEBUG:boto:Method: GET 2023-07-28T23:51:18.782 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:18.783 DEBUG:boto:Data: 2023-07-28T23:51:18.783 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.783 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:18.784 DEBUG:boto:Port: 8001 2023-07-28T23:51:18.784 DEBUG:boto:Params: {} 2023-07-28T23:51:18.785 DEBUG:boto:Token: None 2023-07-28T23:51:18.785 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/ 2023-07-28T23:51:18.786 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2RFaksKPMg2BSPPcI2oCyjSYO08= 2023-07-28T23:51:18.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2RFaksKPMg2BSPPcI2oCyjSYO08=', 'Content-Length': '0'} 2023-07-28T23:51:18.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000351cfca92fac7f46-0064c45476-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.832 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:18.833 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:18.834 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:18.834 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:18.834 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:18.835 DEBUG:boto:Method: GET 2023-07-28T23:51:18.835 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:18.835 DEBUG:boto:Data: 2023-07-28T23:51:18.836 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.836 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:18.836 DEBUG:boto:Port: 8000 2023-07-28T23:51:18.837 DEBUG:boto:Params: {} 2023-07-28T23:51:18.837 DEBUG:boto:Token: None 2023-07-28T23:51:18.838 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/ 2023-07-28T23:51:18.838 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2RFaksKPMg2BSPPcI2oCyjSYO08= 2023-07-28T23:51:18.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2RFaksKPMg2BSPPcI2oCyjSYO08=', 'Content-Length': '0'} 2023-07-28T23:51:18.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000348fc3d642c618b7-0064c45476-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.841 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:18.842 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:18.842 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:18.843 DEBUG:boto:Method: HEAD 2023-07-28T23:51:18.843 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:51:18.843 DEBUG:boto:Data: 2023-07-28T23:51:18.844 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.844 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:18.844 DEBUG:boto:Port: 8001 2023-07-28T23:51:18.845 DEBUG:boto:Params: {} 2023-07-28T23:51:18.845 DEBUG:boto:Token: None 2023-07-28T23:51:18.846 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/obj 2023-07-28T23:51:18.846 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CmmMNgWv5R3yb8SOimYLsP7AvSQ= 2023-07-28T23:51:18.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CmmMNgWv5R3yb8SOimYLsP7AvSQ=', 'Content-Length': '0'} 2023-07-28T23:51:18.850 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b7983841f6ccd296-0064c45476-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.850 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:18.851 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:18.851 DEBUG:boto:Method: HEAD 2023-07-28T23:51:18.851 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:51:18.852 DEBUG:boto:Data: 2023-07-28T23:51:18.852 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.852 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:18.853 DEBUG:boto:Port: 8000 2023-07-28T23:51:18.853 DEBUG:boto:Params: {} 2023-07-28T23:51:18.854 DEBUG:boto:Token: None 2023-07-28T23:51:18.854 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/obj 2023-07-28T23:51:18.854 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CmmMNgWv5R3yb8SOimYLsP7AvSQ= 2023-07-28T23:51:18.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CmmMNgWv5R3yb8SOimYLsP7AvSQ=', 'Content-Length': '0'} 2023-07-28T23:51:18.857 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006be022e5e0a77f61-0064c45476-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:18.859 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:18.859 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:18.860 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:18.860 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:18.860 DEBUG:boto:Method: GET 2023-07-28T23:51:18.861 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:18.861 DEBUG:boto:Data: 2023-07-28T23:51:18.861 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.862 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:18.862 DEBUG:boto:Port: 8001 2023-07-28T23:51:18.863 DEBUG:boto:Params: {} 2023-07-28T23:51:18.863 DEBUG:boto:Token: None 2023-07-28T23:51:18.863 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:18.864 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mMMie89j0sSjmqoZKWUAzL2baP4= 2023-07-28T23:51:18.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mMMie89j0sSjmqoZKWUAzL2baP4=', 'Content-Length': '0'} 2023-07-28T23:51:18.866 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000798e0647d0466dbd-0064c45476-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.912 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:18.913 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:18.913 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:18.914 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:18.914 DEBUG:boto:Method: GET 2023-07-28T23:51:18.914 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:18.915 DEBUG:boto:Data: 2023-07-28T23:51:18.915 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.915 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:18.916 DEBUG:boto:Port: 8000 2023-07-28T23:51:18.916 DEBUG:boto:Params: {} 2023-07-28T23:51:18.917 DEBUG:boto:Token: None 2023-07-28T23:51:18.917 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:18.918 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mMMie89j0sSjmqoZKWUAzL2baP4= 2023-07-28T23:51:18.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mMMie89j0sSjmqoZKWUAzL2baP4=', 'Content-Length': '0'} 2023-07-28T23:51:18.920 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000070ec3de587f810e9-0064c45476-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.921 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:18.921 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:18.922 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:18.922 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:18.922 DEBUG:boto:Method: GET 2023-07-28T23:51:18.923 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:18.923 DEBUG:boto:Data: 2023-07-28T23:51:18.923 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.924 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:18.924 DEBUG:boto:Port: 8001 2023-07-28T23:51:18.924 DEBUG:boto:Params: {} 2023-07-28T23:51:18.925 DEBUG:boto:Token: None 2023-07-28T23:51:18.925 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/ 2023-07-28T23:51:18.926 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2RFaksKPMg2BSPPcI2oCyjSYO08= 2023-07-28T23:51:18.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2RFaksKPMg2BSPPcI2oCyjSYO08=', 'Content-Length': '0'} 2023-07-28T23:51:18.929 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a3721920a1e2936-0064c45476-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.976 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:18.977 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:18.978 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:18.978 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:18.978 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:18.979 DEBUG:boto:Method: GET 2023-07-28T23:51:18.979 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:18.979 DEBUG:boto:Data: 2023-07-28T23:51:18.980 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.980 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:18.981 DEBUG:boto:Port: 8000 2023-07-28T23:51:18.981 DEBUG:boto:Params: {} 2023-07-28T23:51:18.981 DEBUG:boto:Token: None 2023-07-28T23:51:18.982 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/ 2023-07-28T23:51:18.982 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2RFaksKPMg2BSPPcI2oCyjSYO08= 2023-07-28T23:51:18.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2RFaksKPMg2BSPPcI2oCyjSYO08=', 'Content-Length': '0'} 2023-07-28T23:51:18.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003238c30310339ead-0064c45476-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.986 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:18.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:18.987 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:18.987 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:18.988 DEBUG:boto:path=/hiiqos-13/obj?versionId=eu89nyTHw35ZG459SndVR8LnRvJs7VP 2023-07-28T23:51:18.988 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=eu89nyTHw35ZG459SndVR8LnRvJs7VP 2023-07-28T23:51:18.989 DEBUG:boto:Method: GET 2023-07-28T23:51:18.989 DEBUG:boto:Path: /hiiqos-13/obj?versionId=eu89nyTHw35ZG459SndVR8LnRvJs7VP 2023-07-28T23:51:18.989 DEBUG:boto:Data: 2023-07-28T23:51:18.990 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.990 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:18.990 DEBUG:boto:Port: 8001 2023-07-28T23:51:18.991 DEBUG:boto:Params: {} 2023-07-28T23:51:18.991 DEBUG:boto:Token: None 2023-07-28T23:51:18.991 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:18 GMT /hiiqos-13/obj?versionId=eu89nyTHw35ZG459SndVR8LnRvJs7VP 2023-07-28T23:51:18.992 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gRpw3P0hKUBsFoa/BU2Kkl91DnY= 2023-07-28T23:51:18.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gRpw3P0hKUBsFoa/BU2Kkl91DnY=', 'Content-Length': '0'} 2023-07-28T23:51:18.994 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'eu89nyTHw35ZG459SndVR8LnRvJs7VP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b2b2eab935d78a9c-0064c45476-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:18.995 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:18.996 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:18.996 DEBUG:boto:path=/hiiqos-13/obj?versionId=eu89nyTHw35ZG459SndVR8LnRvJs7VP 2023-07-28T23:51:18.996 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=eu89nyTHw35ZG459SndVR8LnRvJs7VP 2023-07-28T23:51:18.997 DEBUG:boto:Method: GET 2023-07-28T23:51:18.997 DEBUG:boto:Path: /hiiqos-13/obj?versionId=eu89nyTHw35ZG459SndVR8LnRvJs7VP 2023-07-28T23:51:18.997 DEBUG:boto:Data: 2023-07-28T23:51:18.998 DEBUG:boto:Headers: {} 2023-07-28T23:51:18.998 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:18.999 DEBUG:boto:Port: 8000 2023-07-28T23:51:18.999 DEBUG:boto:Params: {} 2023-07-28T23:51:19.000 DEBUG:boto:Token: None 2023-07-28T23:51:19.000 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/obj?versionId=eu89nyTHw35ZG459SndVR8LnRvJs7VP 2023-07-28T23:51:19.001 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oHh/OfKfRj7qv74Rl6uiPeEkXm0= 2023-07-28T23:51:19.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oHh/OfKfRj7qv74Rl6uiPeEkXm0=', 'Content-Length': '0'} 2023-07-28T23:51:19.003 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'eu89nyTHw35ZG459SndVR8LnRvJs7VP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000dd0e34c62b2296f-0064c45477-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.044 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:19.045 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:19.045 DEBUG:boto:path=/hiiqos-13/obj?versionId=eu89nyTHw35ZG459SndVR8LnRvJs7VP 2023-07-28T23:51:19.045 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=eu89nyTHw35ZG459SndVR8LnRvJs7VP 2023-07-28T23:51:19.046 DEBUG:boto:Method: HEAD 2023-07-28T23:51:19.046 DEBUG:boto:Path: /hiiqos-13/obj?versionId=eu89nyTHw35ZG459SndVR8LnRvJs7VP 2023-07-28T23:51:19.047 DEBUG:boto:Data: 2023-07-28T23:51:19.047 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.047 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:19.047 DEBUG:boto:Port: 8001 2023-07-28T23:51:19.048 DEBUG:boto:Params: {} 2023-07-28T23:51:19.048 DEBUG:boto:Token: None 2023-07-28T23:51:19.049 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/obj?versionId=eu89nyTHw35ZG459SndVR8LnRvJs7VP 2023-07-28T23:51:19.049 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:l6PNS53OdxU0HFEUo49e51H93Sw= 2023-07-28T23:51:19.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:l6PNS53OdxU0HFEUo49e51H93Sw=', 'Content-Length': '0'} 2023-07-28T23:51:19.052 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'eu89nyTHw35ZG459SndVR8LnRvJs7VP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009181b97e716dbadc-0064c45477-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.052 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:19.053 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:19.053 DEBUG:boto:path=/hiiqos-13/obj?versionId=eu89nyTHw35ZG459SndVR8LnRvJs7VP 2023-07-28T23:51:19.053 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=eu89nyTHw35ZG459SndVR8LnRvJs7VP 2023-07-28T23:51:19.054 DEBUG:boto:Method: HEAD 2023-07-28T23:51:19.054 DEBUG:boto:Path: /hiiqos-13/obj?versionId=eu89nyTHw35ZG459SndVR8LnRvJs7VP 2023-07-28T23:51:19.055 DEBUG:boto:Data: 2023-07-28T23:51:19.055 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.055 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:19.056 DEBUG:boto:Port: 8000 2023-07-28T23:51:19.056 DEBUG:boto:Params: {} 2023-07-28T23:51:19.056 DEBUG:boto:Token: None 2023-07-28T23:51:19.057 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/obj?versionId=eu89nyTHw35ZG459SndVR8LnRvJs7VP 2023-07-28T23:51:19.057 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:l6PNS53OdxU0HFEUo49e51H93Sw= 2023-07-28T23:51:19.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:l6PNS53OdxU0HFEUo49e51H93Sw=', 'Content-Length': '0'} 2023-07-28T23:51:19.059 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'eu89nyTHw35ZG459SndVR8LnRvJs7VP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c5aba95b7b4c2232-0064c45477-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:19.060 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:19.061 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:19.061 DEBUG:boto:path=/hiiqos-13/obj?versionId=eu89nyTHw35ZG459SndVR8LnRvJs7VP 2023-07-28T23:51:19.062 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=eu89nyTHw35ZG459SndVR8LnRvJs7VP 2023-07-28T23:51:19.062 DEBUG:boto:Method: GET 2023-07-28T23:51:19.062 DEBUG:boto:Path: /hiiqos-13/obj?versionId=eu89nyTHw35ZG459SndVR8LnRvJs7VP 2023-07-28T23:51:19.063 DEBUG:boto:Data: 2023-07-28T23:51:19.063 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.063 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:19.064 DEBUG:boto:Port: 8001 2023-07-28T23:51:19.064 DEBUG:boto:Params: {} 2023-07-28T23:51:19.065 DEBUG:boto:Token: None 2023-07-28T23:51:19.065 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/obj?versionId=eu89nyTHw35ZG459SndVR8LnRvJs7VP 2023-07-28T23:51:19.066 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oHh/OfKfRj7qv74Rl6uiPeEkXm0= 2023-07-28T23:51:19.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oHh/OfKfRj7qv74Rl6uiPeEkXm0=', 'Content-Length': '0'} 2023-07-28T23:51:19.068 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'eu89nyTHw35ZG459SndVR8LnRvJs7VP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000095bdf44cfcee6a65-0064c45477-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.068 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:19.069 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:19.069 DEBUG:boto:path=/hiiqos-13/obj?versionId=eu89nyTHw35ZG459SndVR8LnRvJs7VP 2023-07-28T23:51:19.070 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=eu89nyTHw35ZG459SndVR8LnRvJs7VP 2023-07-28T23:51:19.070 DEBUG:boto:Method: GET 2023-07-28T23:51:19.070 DEBUG:boto:Path: /hiiqos-13/obj?versionId=eu89nyTHw35ZG459SndVR8LnRvJs7VP 2023-07-28T23:51:19.071 DEBUG:boto:Data: 2023-07-28T23:51:19.071 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.072 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:19.072 DEBUG:boto:Port: 8000 2023-07-28T23:51:19.073 DEBUG:boto:Params: {} 2023-07-28T23:51:19.073 DEBUG:boto:Token: None 2023-07-28T23:51:19.074 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/obj?versionId=eu89nyTHw35ZG459SndVR8LnRvJs7VP 2023-07-28T23:51:19.074 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oHh/OfKfRj7qv74Rl6uiPeEkXm0= 2023-07-28T23:51:19.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oHh/OfKfRj7qv74Rl6uiPeEkXm0=', 'Content-Length': '0'} 2023-07-28T23:51:19.077 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'eu89nyTHw35ZG459SndVR8LnRvJs7VP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eb4acb26c997301c-0064c45477-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.077 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:19.078 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:19.078 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:19.079 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:19.079 DEBUG:boto:Method: GET 2023-07-28T23:51:19.079 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:19.080 DEBUG:boto:Data: 2023-07-28T23:51:19.080 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.080 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:19.081 DEBUG:boto:Port: 8001 2023-07-28T23:51:19.081 DEBUG:boto:Params: {} 2023-07-28T23:51:19.082 DEBUG:boto:Token: None 2023-07-28T23:51:19.082 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/ 2023-07-28T23:51:19.082 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DlyNiTjyCdClD3M1DZidenjgwug= 2023-07-28T23:51:19.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DlyNiTjyCdClD3M1DZidenjgwug=', 'Content-Length': '0'} 2023-07-28T23:51:19.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c50c23662ba5589-0064c45477-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.086 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:19.087 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:19.087 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:19.087 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:19.088 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:19.088 DEBUG:boto:Method: GET 2023-07-28T23:51:19.088 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:19.089 DEBUG:boto:Data: 2023-07-28T23:51:19.089 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.090 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:19.090 DEBUG:boto:Port: 8000 2023-07-28T23:51:19.090 DEBUG:boto:Params: {} 2023-07-28T23:51:19.091 DEBUG:boto:Token: None 2023-07-28T23:51:19.091 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/ 2023-07-28T23:51:19.091 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DlyNiTjyCdClD3M1DZidenjgwug= 2023-07-28T23:51:19.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DlyNiTjyCdClD3M1DZidenjgwug=', 'Content-Length': '0'} 2023-07-28T23:51:19.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046c6a4c31bbcb88f-0064c45477-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.095 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:19.096 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:19.096 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:19.096 DEBUG:boto:Method: HEAD 2023-07-28T23:51:19.097 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:51:19.097 DEBUG:boto:Data: 2023-07-28T23:51:19.097 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.098 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:19.098 DEBUG:boto:Port: 8001 2023-07-28T23:51:19.099 DEBUG:boto:Params: {} 2023-07-28T23:51:19.099 DEBUG:boto:Token: None 2023-07-28T23:51:19.099 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/obj 2023-07-28T23:51:19.100 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xzPhjsKFbd/IZXdX1u3vbOcG4xE= 2023-07-28T23:51:19.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xzPhjsKFbd/IZXdX1u3vbOcG4xE=', 'Content-Length': '0'} 2023-07-28T23:51:19.103 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ae8fd9b912015ab8-0064c45477-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.104 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:19.105 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:19.105 DEBUG:boto:Method: HEAD 2023-07-28T23:51:19.105 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:51:19.106 DEBUG:boto:Data: 2023-07-28T23:51:19.106 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.107 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:19.107 DEBUG:boto:Port: 8000 2023-07-28T23:51:19.107 DEBUG:boto:Params: {} 2023-07-28T23:51:19.108 DEBUG:boto:Token: None 2023-07-28T23:51:19.108 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/obj 2023-07-28T23:51:19.108 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xzPhjsKFbd/IZXdX1u3vbOcG4xE= 2023-07-28T23:51:19.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xzPhjsKFbd/IZXdX1u3vbOcG4xE=', 'Content-Length': '0'} 2023-07-28T23:51:19.112 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000020936a0dd9e2351c-0064c45477-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:19.113 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:19.114 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:19.115 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:19.116 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:19.116 DEBUG:boto:Method: GET 2023-07-28T23:51:19.117 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:19.118 DEBUG:boto:Data: 2023-07-28T23:51:19.118 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.119 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:19.120 DEBUG:boto:Port: 8001 2023-07-28T23:51:19.121 DEBUG:boto:Params: {} 2023-07-28T23:51:19.121 DEBUG:boto:Token: None 2023-07-28T23:51:19.122 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:19.123 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pFndM9SC0VYXa055LhLiBG1r5d8= 2023-07-28T23:51:19.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pFndM9SC0VYXa055LhLiBG1r5d8=', 'Content-Length': '0'} 2023-07-28T23:51:19.126 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009cee5c46d6216e09-0064c45477-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.168 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:19.169 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:19.169 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:19.170 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:19.170 DEBUG:boto:Method: GET 2023-07-28T23:51:19.170 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:19.171 DEBUG:boto:Data: 2023-07-28T23:51:19.171 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.171 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:19.172 DEBUG:boto:Port: 8000 2023-07-28T23:51:19.172 DEBUG:boto:Params: {} 2023-07-28T23:51:19.173 DEBUG:boto:Token: None 2023-07-28T23:51:19.173 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:19.174 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pFndM9SC0VYXa055LhLiBG1r5d8= 2023-07-28T23:51:19.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pFndM9SC0VYXa055LhLiBG1r5d8=', 'Content-Length': '0'} 2023-07-28T23:51:19.176 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe705bff0b922d7b-0064c45477-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.177 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:19.177 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:19.178 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:19.178 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:19.179 DEBUG:boto:Method: GET 2023-07-28T23:51:19.179 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:19.179 DEBUG:boto:Data: 2023-07-28T23:51:19.180 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.180 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:19.180 DEBUG:boto:Port: 8001 2023-07-28T23:51:19.181 DEBUG:boto:Params: {} 2023-07-28T23:51:19.181 DEBUG:boto:Token: None 2023-07-28T23:51:19.182 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/ 2023-07-28T23:51:19.182 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DlyNiTjyCdClD3M1DZidenjgwug= 2023-07-28T23:51:19.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DlyNiTjyCdClD3M1DZidenjgwug=', 'Content-Length': '0'} 2023-07-28T23:51:19.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bc2ec0fc524cac0-0064c45477-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.228 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:19.229 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:19.230 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:19.230 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:19.230 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:19.231 DEBUG:boto:Method: GET 2023-07-28T23:51:19.231 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:19.231 DEBUG:boto:Data: 2023-07-28T23:51:19.232 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.232 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:19.232 DEBUG:boto:Port: 8000 2023-07-28T23:51:19.233 DEBUG:boto:Params: {} 2023-07-28T23:51:19.233 DEBUG:boto:Token: None 2023-07-28T23:51:19.234 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/ 2023-07-28T23:51:19.234 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DlyNiTjyCdClD3M1DZidenjgwug= 2023-07-28T23:51:19.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DlyNiTjyCdClD3M1DZidenjgwug=', 'Content-Length': '0'} 2023-07-28T23:51:19.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085591b76a4d1e813-0064c45477-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.238 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:19.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:19.239 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:19.239 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:19.239 DEBUG:boto:path=/hiiqos-13/obj?versionId=9hfXRNs96rGpisQpVYKBPxZpFPAeL47 2023-07-28T23:51:19.240 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=9hfXRNs96rGpisQpVYKBPxZpFPAeL47 2023-07-28T23:51:19.240 DEBUG:boto:Method: GET 2023-07-28T23:51:19.241 DEBUG:boto:Path: /hiiqos-13/obj?versionId=9hfXRNs96rGpisQpVYKBPxZpFPAeL47 2023-07-28T23:51:19.241 DEBUG:boto:Data: 2023-07-28T23:51:19.241 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.242 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:19.242 DEBUG:boto:Port: 8001 2023-07-28T23:51:19.242 DEBUG:boto:Params: {} 2023-07-28T23:51:19.243 DEBUG:boto:Token: None 2023-07-28T23:51:19.243 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/obj?versionId=9hfXRNs96rGpisQpVYKBPxZpFPAeL47 2023-07-28T23:51:19.244 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/vI/BJLwkHXDVLBIPw9TI2e1bdU= 2023-07-28T23:51:19.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/vI/BJLwkHXDVLBIPw9TI2e1bdU=', 'Content-Length': '0'} 2023-07-28T23:51:19.246 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', '9hfXRNs96rGpisQpVYKBPxZpFPAeL47'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009015786ba2d630e8-0064c45477-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.247 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:19.247 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:19.248 DEBUG:boto:path=/hiiqos-13/obj?versionId=9hfXRNs96rGpisQpVYKBPxZpFPAeL47 2023-07-28T23:51:19.248 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=9hfXRNs96rGpisQpVYKBPxZpFPAeL47 2023-07-28T23:51:19.249 DEBUG:boto:Method: GET 2023-07-28T23:51:19.249 DEBUG:boto:Path: /hiiqos-13/obj?versionId=9hfXRNs96rGpisQpVYKBPxZpFPAeL47 2023-07-28T23:51:19.249 DEBUG:boto:Data: 2023-07-28T23:51:19.250 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.250 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:19.250 DEBUG:boto:Port: 8000 2023-07-28T23:51:19.251 DEBUG:boto:Params: {} 2023-07-28T23:51:19.251 DEBUG:boto:Token: None 2023-07-28T23:51:19.252 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/obj?versionId=9hfXRNs96rGpisQpVYKBPxZpFPAeL47 2023-07-28T23:51:19.252 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/vI/BJLwkHXDVLBIPw9TI2e1bdU= 2023-07-28T23:51:19.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/vI/BJLwkHXDVLBIPw9TI2e1bdU=', 'Content-Length': '0'} 2023-07-28T23:51:19.254 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', '9hfXRNs96rGpisQpVYKBPxZpFPAeL47'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006bc54b9d9cbb2ad0-0064c45477-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.296 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:19.297 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:19.297 DEBUG:boto:path=/hiiqos-13/obj?versionId=9hfXRNs96rGpisQpVYKBPxZpFPAeL47 2023-07-28T23:51:19.297 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=9hfXRNs96rGpisQpVYKBPxZpFPAeL47 2023-07-28T23:51:19.298 DEBUG:boto:Method: HEAD 2023-07-28T23:51:19.298 DEBUG:boto:Path: /hiiqos-13/obj?versionId=9hfXRNs96rGpisQpVYKBPxZpFPAeL47 2023-07-28T23:51:19.299 DEBUG:boto:Data: 2023-07-28T23:51:19.299 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.299 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:19.300 DEBUG:boto:Port: 8001 2023-07-28T23:51:19.300 DEBUG:boto:Params: {} 2023-07-28T23:51:19.300 DEBUG:boto:Token: None 2023-07-28T23:51:19.301 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/obj?versionId=9hfXRNs96rGpisQpVYKBPxZpFPAeL47 2023-07-28T23:51:19.301 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Fpan/c8ge5kvurN7AO3U98CH3MQ= 2023-07-28T23:51:19.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Fpan/c8ge5kvurN7AO3U98CH3MQ=', 'Content-Length': '0'} 2023-07-28T23:51:19.304 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', '9hfXRNs96rGpisQpVYKBPxZpFPAeL47'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001abc5d4b1e30f766-0064c45477-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.304 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:19.305 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:19.305 DEBUG:boto:path=/hiiqos-13/obj?versionId=9hfXRNs96rGpisQpVYKBPxZpFPAeL47 2023-07-28T23:51:19.305 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=9hfXRNs96rGpisQpVYKBPxZpFPAeL47 2023-07-28T23:51:19.306 DEBUG:boto:Method: HEAD 2023-07-28T23:51:19.306 DEBUG:boto:Path: /hiiqos-13/obj?versionId=9hfXRNs96rGpisQpVYKBPxZpFPAeL47 2023-07-28T23:51:19.306 DEBUG:boto:Data: 2023-07-28T23:51:19.307 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.307 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:19.307 DEBUG:boto:Port: 8000 2023-07-28T23:51:19.308 DEBUG:boto:Params: {} 2023-07-28T23:51:19.308 DEBUG:boto:Token: None 2023-07-28T23:51:19.308 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/obj?versionId=9hfXRNs96rGpisQpVYKBPxZpFPAeL47 2023-07-28T23:51:19.309 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Fpan/c8ge5kvurN7AO3U98CH3MQ= 2023-07-28T23:51:19.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Fpan/c8ge5kvurN7AO3U98CH3MQ=', 'Content-Length': '0'} 2023-07-28T23:51:19.311 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', '9hfXRNs96rGpisQpVYKBPxZpFPAeL47'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e55ca980b78a84a-0064c45477-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:19.312 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:19.313 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:19.313 DEBUG:boto:path=/hiiqos-13/obj?versionId=9hfXRNs96rGpisQpVYKBPxZpFPAeL47 2023-07-28T23:51:19.313 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=9hfXRNs96rGpisQpVYKBPxZpFPAeL47 2023-07-28T23:51:19.314 DEBUG:boto:Method: GET 2023-07-28T23:51:19.314 DEBUG:boto:Path: /hiiqos-13/obj?versionId=9hfXRNs96rGpisQpVYKBPxZpFPAeL47 2023-07-28T23:51:19.314 DEBUG:boto:Data: 2023-07-28T23:51:19.315 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.315 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:19.315 DEBUG:boto:Port: 8001 2023-07-28T23:51:19.316 DEBUG:boto:Params: {} 2023-07-28T23:51:19.316 DEBUG:boto:Token: None 2023-07-28T23:51:19.317 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/obj?versionId=9hfXRNs96rGpisQpVYKBPxZpFPAeL47 2023-07-28T23:51:19.317 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/vI/BJLwkHXDVLBIPw9TI2e1bdU= 2023-07-28T23:51:19.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/vI/BJLwkHXDVLBIPw9TI2e1bdU=', 'Content-Length': '0'} 2023-07-28T23:51:19.319 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', '9hfXRNs96rGpisQpVYKBPxZpFPAeL47'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000093f606dcaaa3348c-0064c45477-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.360 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:19.361 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:19.361 DEBUG:boto:path=/hiiqos-13/obj?versionId=9hfXRNs96rGpisQpVYKBPxZpFPAeL47 2023-07-28T23:51:19.362 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=9hfXRNs96rGpisQpVYKBPxZpFPAeL47 2023-07-28T23:51:19.362 DEBUG:boto:Method: GET 2023-07-28T23:51:19.363 DEBUG:boto:Path: /hiiqos-13/obj?versionId=9hfXRNs96rGpisQpVYKBPxZpFPAeL47 2023-07-28T23:51:19.363 DEBUG:boto:Data: 2023-07-28T23:51:19.363 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.364 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:19.364 DEBUG:boto:Port: 8000 2023-07-28T23:51:19.365 DEBUG:boto:Params: {} 2023-07-28T23:51:19.365 DEBUG:boto:Token: None 2023-07-28T23:51:19.365 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/obj?versionId=9hfXRNs96rGpisQpVYKBPxZpFPAeL47 2023-07-28T23:51:19.366 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/vI/BJLwkHXDVLBIPw9TI2e1bdU= 2023-07-28T23:51:19.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/vI/BJLwkHXDVLBIPw9TI2e1bdU=', 'Content-Length': '0'} 2023-07-28T23:51:19.368 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', '9hfXRNs96rGpisQpVYKBPxZpFPAeL47'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000035ec1ffdeaa0f407-0064c45477-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.369 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:19.369 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:19.370 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:19.370 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:19.370 DEBUG:boto:Method: GET 2023-07-28T23:51:19.371 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:19.371 DEBUG:boto:Data: 2023-07-28T23:51:19.372 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.372 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:19.372 DEBUG:boto:Port: 8001 2023-07-28T23:51:19.373 DEBUG:boto:Params: {} 2023-07-28T23:51:19.373 DEBUG:boto:Token: None 2023-07-28T23:51:19.374 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/ 2023-07-28T23:51:19.374 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DlyNiTjyCdClD3M1DZidenjgwug= 2023-07-28T23:51:19.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DlyNiTjyCdClD3M1DZidenjgwug=', 'Content-Length': '0'} 2023-07-28T23:51:19.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c53856f956cde9c2-0064c45477-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.378 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:19.379 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:19.379 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:19.379 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:19.380 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:19.380 DEBUG:boto:Method: GET 2023-07-28T23:51:19.380 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:19.381 DEBUG:boto:Data: 2023-07-28T23:51:19.381 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.382 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:19.382 DEBUG:boto:Port: 8000 2023-07-28T23:51:19.383 DEBUG:boto:Params: {} 2023-07-28T23:51:19.383 DEBUG:boto:Token: None 2023-07-28T23:51:19.383 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/ 2023-07-28T23:51:19.384 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DlyNiTjyCdClD3M1DZidenjgwug= 2023-07-28T23:51:19.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DlyNiTjyCdClD3M1DZidenjgwug=', 'Content-Length': '0'} 2023-07-28T23:51:19.387 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086fb5c51a4e8ba0e-0064c45477-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.388 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:19.388 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:19.389 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:19.389 DEBUG:boto:Method: HEAD 2023-07-28T23:51:19.389 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:51:19.390 DEBUG:boto:Data: 2023-07-28T23:51:19.390 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.390 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:19.391 DEBUG:boto:Port: 8001 2023-07-28T23:51:19.391 DEBUG:boto:Params: {} 2023-07-28T23:51:19.391 DEBUG:boto:Token: None 2023-07-28T23:51:19.392 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/obj 2023-07-28T23:51:19.392 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xzPhjsKFbd/IZXdX1u3vbOcG4xE= 2023-07-28T23:51:19.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xzPhjsKFbd/IZXdX1u3vbOcG4xE=', 'Content-Length': '0'} 2023-07-28T23:51:19.396 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000055fcac55a73af002-0064c45477-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.397 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:19.397 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:19.397 DEBUG:boto:Method: HEAD 2023-07-28T23:51:19.398 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:51:19.398 DEBUG:boto:Data: 2023-07-28T23:51:19.398 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.399 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:19.399 DEBUG:boto:Port: 8000 2023-07-28T23:51:19.399 DEBUG:boto:Params: {} 2023-07-28T23:51:19.400 DEBUG:boto:Token: None 2023-07-28T23:51:19.400 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/obj 2023-07-28T23:51:19.401 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xzPhjsKFbd/IZXdX1u3vbOcG4xE= 2023-07-28T23:51:19.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xzPhjsKFbd/IZXdX1u3vbOcG4xE=', 'Content-Length': '0'} 2023-07-28T23:51:19.404 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000017681ce10ae32b2-0064c45477-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:19.405 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:19.406 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:19.406 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:19.407 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:19.407 DEBUG:boto:Method: GET 2023-07-28T23:51:19.407 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:19.408 DEBUG:boto:Data: 2023-07-28T23:51:19.409 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.409 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:19.410 DEBUG:boto:Port: 8001 2023-07-28T23:51:19.411 DEBUG:boto:Params: {} 2023-07-28T23:51:19.412 DEBUG:boto:Token: None 2023-07-28T23:51:19.413 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:19.414 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pFndM9SC0VYXa055LhLiBG1r5d8= 2023-07-28T23:51:19.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pFndM9SC0VYXa055LhLiBG1r5d8=', 'Content-Length': '0'} 2023-07-28T23:51:19.417 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000773ba3dc7c18f5c8-0064c45477-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.464 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:19.465 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:19.465 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:19.466 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:19.466 DEBUG:boto:Method: GET 2023-07-28T23:51:19.466 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:19.467 DEBUG:boto:Data: 2023-07-28T23:51:19.467 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.467 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:19.468 DEBUG:boto:Port: 8000 2023-07-28T23:51:19.468 DEBUG:boto:Params: {} 2023-07-28T23:51:19.469 DEBUG:boto:Token: None 2023-07-28T23:51:19.469 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:19.469 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pFndM9SC0VYXa055LhLiBG1r5d8= 2023-07-28T23:51:19.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pFndM9SC0VYXa055LhLiBG1r5d8=', 'Content-Length': '0'} 2023-07-28T23:51:19.472 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000016396986cd98ee78-0064c45477-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.472 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:19.473 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:19.473 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:19.474 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:19.474 DEBUG:boto:Method: GET 2023-07-28T23:51:19.474 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:19.475 DEBUG:boto:Data: 2023-07-28T23:51:19.475 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.476 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:19.476 DEBUG:boto:Port: 8001 2023-07-28T23:51:19.476 DEBUG:boto:Params: {} 2023-07-28T23:51:19.477 DEBUG:boto:Token: None 2023-07-28T23:51:19.477 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/ 2023-07-28T23:51:19.478 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DlyNiTjyCdClD3M1DZidenjgwug= 2023-07-28T23:51:19.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DlyNiTjyCdClD3M1DZidenjgwug=', 'Content-Length': '0'} 2023-07-28T23:51:19.480 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abbe449b821e24cd-0064c45477-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.481 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:19.482 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:19.482 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:19.482 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:19.483 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:19.483 DEBUG:boto:Method: GET 2023-07-28T23:51:19.484 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:19.484 DEBUG:boto:Data: 2023-07-28T23:51:19.484 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.485 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:19.485 DEBUG:boto:Port: 8000 2023-07-28T23:51:19.485 DEBUG:boto:Params: {} 2023-07-28T23:51:19.486 DEBUG:boto:Token: None 2023-07-28T23:51:19.486 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/ 2023-07-28T23:51:19.487 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DlyNiTjyCdClD3M1DZidenjgwug= 2023-07-28T23:51:19.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DlyNiTjyCdClD3M1DZidenjgwug=', 'Content-Length': '0'} 2023-07-28T23:51:19.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027a687b8ad9750cc-0064c45477-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.490 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:19.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:19.491 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:19.491 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:19.492 DEBUG:boto:path=/hiiqos-13/obj?versionId=KW4erdPcpVPNf4hlrOZysjOAxTTzNpZ 2023-07-28T23:51:19.492 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=KW4erdPcpVPNf4hlrOZysjOAxTTzNpZ 2023-07-28T23:51:19.493 DEBUG:boto:Method: GET 2023-07-28T23:51:19.493 DEBUG:boto:Path: /hiiqos-13/obj?versionId=KW4erdPcpVPNf4hlrOZysjOAxTTzNpZ 2023-07-28T23:51:19.493 DEBUG:boto:Data: 2023-07-28T23:51:19.494 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.494 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:19.494 DEBUG:boto:Port: 8001 2023-07-28T23:51:19.495 DEBUG:boto:Params: {} 2023-07-28T23:51:19.495 DEBUG:boto:Token: None 2023-07-28T23:51:19.495 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/obj?versionId=KW4erdPcpVPNf4hlrOZysjOAxTTzNpZ 2023-07-28T23:51:19.496 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LjKo0XD8bAUSoSxORHDhUbRRU20= 2023-07-28T23:51:19.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LjKo0XD8bAUSoSxORHDhUbRRU20=', 'Content-Length': '0'} 2023-07-28T23:51:19.498 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'KW4erdPcpVPNf4hlrOZysjOAxTTzNpZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000176616de8136d177-0064c45477-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.499 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:19.499 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:19.499 DEBUG:boto:path=/hiiqos-13/obj?versionId=KW4erdPcpVPNf4hlrOZysjOAxTTzNpZ 2023-07-28T23:51:19.500 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=KW4erdPcpVPNf4hlrOZysjOAxTTzNpZ 2023-07-28T23:51:19.500 DEBUG:boto:Method: GET 2023-07-28T23:51:19.500 DEBUG:boto:Path: /hiiqos-13/obj?versionId=KW4erdPcpVPNf4hlrOZysjOAxTTzNpZ 2023-07-28T23:51:19.501 DEBUG:boto:Data: 2023-07-28T23:51:19.501 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.502 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:19.502 DEBUG:boto:Port: 8000 2023-07-28T23:51:19.503 DEBUG:boto:Params: {} 2023-07-28T23:51:19.504 DEBUG:boto:Token: None 2023-07-28T23:51:19.504 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/obj?versionId=KW4erdPcpVPNf4hlrOZysjOAxTTzNpZ 2023-07-28T23:51:19.505 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LjKo0XD8bAUSoSxORHDhUbRRU20= 2023-07-28T23:51:19.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LjKo0XD8bAUSoSxORHDhUbRRU20=', 'Content-Length': '0'} 2023-07-28T23:51:19.508 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'KW4erdPcpVPNf4hlrOZysjOAxTTzNpZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000942519bbcee707d8-0064c45477-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.552 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:19.553 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:19.553 DEBUG:boto:path=/hiiqos-13/obj?versionId=KW4erdPcpVPNf4hlrOZysjOAxTTzNpZ 2023-07-28T23:51:19.554 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=KW4erdPcpVPNf4hlrOZysjOAxTTzNpZ 2023-07-28T23:51:19.554 DEBUG:boto:Method: HEAD 2023-07-28T23:51:19.554 DEBUG:boto:Path: /hiiqos-13/obj?versionId=KW4erdPcpVPNf4hlrOZysjOAxTTzNpZ 2023-07-28T23:51:19.555 DEBUG:boto:Data: 2023-07-28T23:51:19.555 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.555 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:19.556 DEBUG:boto:Port: 8001 2023-07-28T23:51:19.556 DEBUG:boto:Params: {} 2023-07-28T23:51:19.556 DEBUG:boto:Token: None 2023-07-28T23:51:19.557 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/obj?versionId=KW4erdPcpVPNf4hlrOZysjOAxTTzNpZ 2023-07-28T23:51:19.557 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IQbL2xiZ3fIWW0BbI2rHug3Y2wk= 2023-07-28T23:51:19.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IQbL2xiZ3fIWW0BbI2rHug3Y2wk=', 'Content-Length': '0'} 2023-07-28T23:51:19.560 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'KW4erdPcpVPNf4hlrOZysjOAxTTzNpZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000068f72746508be024-0064c45477-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.561 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:19.561 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:19.561 DEBUG:boto:path=/hiiqos-13/obj?versionId=KW4erdPcpVPNf4hlrOZysjOAxTTzNpZ 2023-07-28T23:51:19.562 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=KW4erdPcpVPNf4hlrOZysjOAxTTzNpZ 2023-07-28T23:51:19.562 DEBUG:boto:Method: HEAD 2023-07-28T23:51:19.562 DEBUG:boto:Path: /hiiqos-13/obj?versionId=KW4erdPcpVPNf4hlrOZysjOAxTTzNpZ 2023-07-28T23:51:19.563 DEBUG:boto:Data: 2023-07-28T23:51:19.563 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.563 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:19.564 DEBUG:boto:Port: 8000 2023-07-28T23:51:19.564 DEBUG:boto:Params: {} 2023-07-28T23:51:19.564 DEBUG:boto:Token: None 2023-07-28T23:51:19.565 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/obj?versionId=KW4erdPcpVPNf4hlrOZysjOAxTTzNpZ 2023-07-28T23:51:19.565 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IQbL2xiZ3fIWW0BbI2rHug3Y2wk= 2023-07-28T23:51:19.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IQbL2xiZ3fIWW0BbI2rHug3Y2wk=', 'Content-Length': '0'} 2023-07-28T23:51:19.567 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'KW4erdPcpVPNf4hlrOZysjOAxTTzNpZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6a4d5d625151a64-0064c45477-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:19.568 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:19.569 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:19.569 DEBUG:boto:path=/hiiqos-13/obj?versionId=KW4erdPcpVPNf4hlrOZysjOAxTTzNpZ 2023-07-28T23:51:19.570 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=KW4erdPcpVPNf4hlrOZysjOAxTTzNpZ 2023-07-28T23:51:19.570 DEBUG:boto:Method: GET 2023-07-28T23:51:19.570 DEBUG:boto:Path: /hiiqos-13/obj?versionId=KW4erdPcpVPNf4hlrOZysjOAxTTzNpZ 2023-07-28T23:51:19.571 DEBUG:boto:Data: 2023-07-28T23:51:19.571 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.572 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:19.573 DEBUG:boto:Port: 8001 2023-07-28T23:51:19.574 DEBUG:boto:Params: {} 2023-07-28T23:51:19.575 DEBUG:boto:Token: None 2023-07-28T23:51:19.575 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/obj?versionId=KW4erdPcpVPNf4hlrOZysjOAxTTzNpZ 2023-07-28T23:51:19.577 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LjKo0XD8bAUSoSxORHDhUbRRU20= 2023-07-28T23:51:19.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LjKo0XD8bAUSoSxORHDhUbRRU20=', 'Content-Length': '0'} 2023-07-28T23:51:19.580 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'KW4erdPcpVPNf4hlrOZysjOAxTTzNpZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d33fe6a1c380fa01-0064c45477-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.581 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:19.581 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:19.582 DEBUG:boto:path=/hiiqos-13/obj?versionId=KW4erdPcpVPNf4hlrOZysjOAxTTzNpZ 2023-07-28T23:51:19.582 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=KW4erdPcpVPNf4hlrOZysjOAxTTzNpZ 2023-07-28T23:51:19.582 DEBUG:boto:Method: GET 2023-07-28T23:51:19.583 DEBUG:boto:Path: /hiiqos-13/obj?versionId=KW4erdPcpVPNf4hlrOZysjOAxTTzNpZ 2023-07-28T23:51:19.583 DEBUG:boto:Data: 2023-07-28T23:51:19.583 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.584 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:19.584 DEBUG:boto:Port: 8000 2023-07-28T23:51:19.585 DEBUG:boto:Params: {} 2023-07-28T23:51:19.585 DEBUG:boto:Token: None 2023-07-28T23:51:19.585 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/obj?versionId=KW4erdPcpVPNf4hlrOZysjOAxTTzNpZ 2023-07-28T23:51:19.586 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LjKo0XD8bAUSoSxORHDhUbRRU20= 2023-07-28T23:51:19.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LjKo0XD8bAUSoSxORHDhUbRRU20=', 'Content-Length': '0'} 2023-07-28T23:51:19.588 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'KW4erdPcpVPNf4hlrOZysjOAxTTzNpZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1677ece3dd1165a-0064c45477-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.589 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:19.589 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:19.590 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:19.590 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:19.590 DEBUG:boto:Method: GET 2023-07-28T23:51:19.591 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:19.591 DEBUG:boto:Data: 2023-07-28T23:51:19.591 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.592 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:19.592 DEBUG:boto:Port: 8001 2023-07-28T23:51:19.592 DEBUG:boto:Params: {} 2023-07-28T23:51:19.593 DEBUG:boto:Token: None 2023-07-28T23:51:19.593 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/ 2023-07-28T23:51:19.594 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DlyNiTjyCdClD3M1DZidenjgwug= 2023-07-28T23:51:19.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DlyNiTjyCdClD3M1DZidenjgwug=', 'Content-Length': '0'} 2023-07-28T23:51:19.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008bb014eb7b9d6cb-0064c45477-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.640 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:19.641 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:19.642 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:19.642 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:19.643 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:19.643 DEBUG:boto:Method: GET 2023-07-28T23:51:19.643 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:19.644 DEBUG:boto:Data: 2023-07-28T23:51:19.644 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.645 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:19.645 DEBUG:boto:Port: 8000 2023-07-28T23:51:19.645 DEBUG:boto:Params: {} 2023-07-28T23:51:19.646 DEBUG:boto:Token: None 2023-07-28T23:51:19.646 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/ 2023-07-28T23:51:19.647 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DlyNiTjyCdClD3M1DZidenjgwug= 2023-07-28T23:51:19.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DlyNiTjyCdClD3M1DZidenjgwug=', 'Content-Length': '0'} 2023-07-28T23:51:19.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000344e9251d396faa7-0064c45477-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.650 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:19.651 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:19.651 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:19.652 DEBUG:boto:Method: HEAD 2023-07-28T23:51:19.652 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:51:19.652 DEBUG:boto:Data: 2023-07-28T23:51:19.653 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.653 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:19.654 DEBUG:boto:Port: 8001 2023-07-28T23:51:19.654 DEBUG:boto:Params: {} 2023-07-28T23:51:19.654 DEBUG:boto:Token: None 2023-07-28T23:51:19.655 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/obj 2023-07-28T23:51:19.655 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xzPhjsKFbd/IZXdX1u3vbOcG4xE= 2023-07-28T23:51:19.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xzPhjsKFbd/IZXdX1u3vbOcG4xE=', 'Content-Length': '0'} 2023-07-28T23:51:19.659 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003148e04209d34eed-0064c45477-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.659 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:19.660 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:19.660 DEBUG:boto:Method: HEAD 2023-07-28T23:51:19.661 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:51:19.661 DEBUG:boto:Data: 2023-07-28T23:51:19.661 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.662 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:19.662 DEBUG:boto:Port: 8000 2023-07-28T23:51:19.662 DEBUG:boto:Params: {} 2023-07-28T23:51:19.663 DEBUG:boto:Token: None 2023-07-28T23:51:19.663 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/obj 2023-07-28T23:51:19.664 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xzPhjsKFbd/IZXdX1u3vbOcG4xE= 2023-07-28T23:51:19.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xzPhjsKFbd/IZXdX1u3vbOcG4xE=', 'Content-Length': '0'} 2023-07-28T23:51:19.667 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000071f77e2197175753-0064c45477-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:19.668 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:19.669 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:19.669 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:19.669 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:19.670 DEBUG:boto:Method: GET 2023-07-28T23:51:19.670 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:19.670 DEBUG:boto:Data: 2023-07-28T23:51:19.671 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.671 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:19.671 DEBUG:boto:Port: 8001 2023-07-28T23:51:19.672 DEBUG:boto:Params: {} 2023-07-28T23:51:19.672 DEBUG:boto:Token: None 2023-07-28T23:51:19.673 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:19.673 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pFndM9SC0VYXa055LhLiBG1r5d8= 2023-07-28T23:51:19.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pFndM9SC0VYXa055LhLiBG1r5d8=', 'Content-Length': '0'} 2023-07-28T23:51:19.676 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c492909fd262c6b9-0064c45477-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.716 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:19.717 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:19.717 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:19.718 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:19.718 DEBUG:boto:Method: GET 2023-07-28T23:51:19.718 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:19.719 DEBUG:boto:Data: 2023-07-28T23:51:19.719 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.719 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:19.720 DEBUG:boto:Port: 8000 2023-07-28T23:51:19.720 DEBUG:boto:Params: {} 2023-07-28T23:51:19.721 DEBUG:boto:Token: None 2023-07-28T23:51:19.721 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:19.722 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pFndM9SC0VYXa055LhLiBG1r5d8= 2023-07-28T23:51:19.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pFndM9SC0VYXa055LhLiBG1r5d8=', 'Content-Length': '0'} 2023-07-28T23:51:19.724 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce2fd7ba153bedbf-0064c45477-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.725 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:19.725 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:19.726 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:19.726 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:19.726 DEBUG:boto:Method: GET 2023-07-28T23:51:19.727 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:19.727 DEBUG:boto:Data: 2023-07-28T23:51:19.727 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.728 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:19.728 DEBUG:boto:Port: 8001 2023-07-28T23:51:19.729 DEBUG:boto:Params: {} 2023-07-28T23:51:19.729 DEBUG:boto:Token: None 2023-07-28T23:51:19.729 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/ 2023-07-28T23:51:19.730 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DlyNiTjyCdClD3M1DZidenjgwug= 2023-07-28T23:51:19.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DlyNiTjyCdClD3M1DZidenjgwug=', 'Content-Length': '0'} 2023-07-28T23:51:19.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eac0d9780b0b079e-0064c45477-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.733 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:19.734 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:19.735 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:19.735 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:19.736 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:19.736 DEBUG:boto:Method: GET 2023-07-28T23:51:19.736 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:19.737 DEBUG:boto:Data: 2023-07-28T23:51:19.737 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.737 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:19.738 DEBUG:boto:Port: 8000 2023-07-28T23:51:19.738 DEBUG:boto:Params: {} 2023-07-28T23:51:19.738 DEBUG:boto:Token: None 2023-07-28T23:51:19.739 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/ 2023-07-28T23:51:19.739 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DlyNiTjyCdClD3M1DZidenjgwug= 2023-07-28T23:51:19.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DlyNiTjyCdClD3M1DZidenjgwug=', 'Content-Length': '0'} 2023-07-28T23:51:19.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e84d182319a6a388-0064c45477-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.784 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:19.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:19.786 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:19.786 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:19.786 DEBUG:boto:path=/hiiqos-13/obj?versionId=fqZHgkQ0VnkgGOiotHJVpzmOgEM1PNX 2023-07-28T23:51:19.787 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=fqZHgkQ0VnkgGOiotHJVpzmOgEM1PNX 2023-07-28T23:51:19.787 DEBUG:boto:Method: GET 2023-07-28T23:51:19.787 DEBUG:boto:Path: /hiiqos-13/obj?versionId=fqZHgkQ0VnkgGOiotHJVpzmOgEM1PNX 2023-07-28T23:51:19.788 DEBUG:boto:Data: 2023-07-28T23:51:19.788 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.789 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:19.789 DEBUG:boto:Port: 8001 2023-07-28T23:51:19.789 DEBUG:boto:Params: {} 2023-07-28T23:51:19.790 DEBUG:boto:Token: None 2023-07-28T23:51:19.790 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/obj?versionId=fqZHgkQ0VnkgGOiotHJVpzmOgEM1PNX 2023-07-28T23:51:19.791 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dQEGJ/pPDe0VHl2HMSgOyRMAR0Y= 2023-07-28T23:51:19.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dQEGJ/pPDe0VHl2HMSgOyRMAR0Y=', 'Content-Length': '0'} 2023-07-28T23:51:19.793 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'fqZHgkQ0VnkgGOiotHJVpzmOgEM1PNX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e7d02f148a48d957-0064c45477-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.836 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:19.837 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:19.837 DEBUG:boto:path=/hiiqos-13/obj?versionId=fqZHgkQ0VnkgGOiotHJVpzmOgEM1PNX 2023-07-28T23:51:19.838 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=fqZHgkQ0VnkgGOiotHJVpzmOgEM1PNX 2023-07-28T23:51:19.838 DEBUG:boto:Method: GET 2023-07-28T23:51:19.838 DEBUG:boto:Path: /hiiqos-13/obj?versionId=fqZHgkQ0VnkgGOiotHJVpzmOgEM1PNX 2023-07-28T23:51:19.839 DEBUG:boto:Data: 2023-07-28T23:51:19.839 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.839 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:19.840 DEBUG:boto:Port: 8000 2023-07-28T23:51:19.840 DEBUG:boto:Params: {} 2023-07-28T23:51:19.841 DEBUG:boto:Token: None 2023-07-28T23:51:19.841 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/obj?versionId=fqZHgkQ0VnkgGOiotHJVpzmOgEM1PNX 2023-07-28T23:51:19.841 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dQEGJ/pPDe0VHl2HMSgOyRMAR0Y= 2023-07-28T23:51:19.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dQEGJ/pPDe0VHl2HMSgOyRMAR0Y=', 'Content-Length': '0'} 2023-07-28T23:51:19.844 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'fqZHgkQ0VnkgGOiotHJVpzmOgEM1PNX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008dc1733df7e666de-0064c45477-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.845 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:19.845 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:19.846 DEBUG:boto:path=/hiiqos-13/obj?versionId=fqZHgkQ0VnkgGOiotHJVpzmOgEM1PNX 2023-07-28T23:51:19.846 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=fqZHgkQ0VnkgGOiotHJVpzmOgEM1PNX 2023-07-28T23:51:19.846 DEBUG:boto:Method: HEAD 2023-07-28T23:51:19.847 DEBUG:boto:Path: /hiiqos-13/obj?versionId=fqZHgkQ0VnkgGOiotHJVpzmOgEM1PNX 2023-07-28T23:51:19.847 DEBUG:boto:Data: 2023-07-28T23:51:19.847 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.848 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:19.848 DEBUG:boto:Port: 8001 2023-07-28T23:51:19.848 DEBUG:boto:Params: {} 2023-07-28T23:51:19.849 DEBUG:boto:Token: None 2023-07-28T23:51:19.849 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/obj?versionId=fqZHgkQ0VnkgGOiotHJVpzmOgEM1PNX 2023-07-28T23:51:19.849 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MOyC43GitH27TkrpitrzvsXIWBA= 2023-07-28T23:51:19.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MOyC43GitH27TkrpitrzvsXIWBA=', 'Content-Length': '0'} 2023-07-28T23:51:19.852 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'fqZHgkQ0VnkgGOiotHJVpzmOgEM1PNX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009cdad01c4fc12a2c-0064c45477-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.852 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:19.853 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:19.853 DEBUG:boto:path=/hiiqos-13/obj?versionId=fqZHgkQ0VnkgGOiotHJVpzmOgEM1PNX 2023-07-28T23:51:19.853 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=fqZHgkQ0VnkgGOiotHJVpzmOgEM1PNX 2023-07-28T23:51:19.854 DEBUG:boto:Method: HEAD 2023-07-28T23:51:19.854 DEBUG:boto:Path: /hiiqos-13/obj?versionId=fqZHgkQ0VnkgGOiotHJVpzmOgEM1PNX 2023-07-28T23:51:19.854 DEBUG:boto:Data: 2023-07-28T23:51:19.855 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.855 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:19.855 DEBUG:boto:Port: 8000 2023-07-28T23:51:19.856 DEBUG:boto:Params: {} 2023-07-28T23:51:19.856 DEBUG:boto:Token: None 2023-07-28T23:51:19.856 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/obj?versionId=fqZHgkQ0VnkgGOiotHJVpzmOgEM1PNX 2023-07-28T23:51:19.857 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MOyC43GitH27TkrpitrzvsXIWBA= 2023-07-28T23:51:19.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MOyC43GitH27TkrpitrzvsXIWBA=', 'Content-Length': '0'} 2023-07-28T23:51:19.859 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'fqZHgkQ0VnkgGOiotHJVpzmOgEM1PNX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000019e0fd051c835049-0064c45477-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:19.860 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:19.860 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:19.861 DEBUG:boto:path=/hiiqos-13/obj?versionId=fqZHgkQ0VnkgGOiotHJVpzmOgEM1PNX 2023-07-28T23:51:19.861 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=fqZHgkQ0VnkgGOiotHJVpzmOgEM1PNX 2023-07-28T23:51:19.861 DEBUG:boto:Method: GET 2023-07-28T23:51:19.862 DEBUG:boto:Path: /hiiqos-13/obj?versionId=fqZHgkQ0VnkgGOiotHJVpzmOgEM1PNX 2023-07-28T23:51:19.862 DEBUG:boto:Data: 2023-07-28T23:51:19.862 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.863 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:19.863 DEBUG:boto:Port: 8001 2023-07-28T23:51:19.863 DEBUG:boto:Params: {} 2023-07-28T23:51:19.864 DEBUG:boto:Token: None 2023-07-28T23:51:19.864 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/obj?versionId=fqZHgkQ0VnkgGOiotHJVpzmOgEM1PNX 2023-07-28T23:51:19.865 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dQEGJ/pPDe0VHl2HMSgOyRMAR0Y= 2023-07-28T23:51:19.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dQEGJ/pPDe0VHl2HMSgOyRMAR0Y=', 'Content-Length': '0'} 2023-07-28T23:51:19.867 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'fqZHgkQ0VnkgGOiotHJVpzmOgEM1PNX'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a3749795e9480690-0064c45477-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.867 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:19.868 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:19.868 DEBUG:boto:path=/hiiqos-13/obj?versionId=fqZHgkQ0VnkgGOiotHJVpzmOgEM1PNX 2023-07-28T23:51:19.868 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=fqZHgkQ0VnkgGOiotHJVpzmOgEM1PNX 2023-07-28T23:51:19.869 DEBUG:boto:Method: GET 2023-07-28T23:51:19.869 DEBUG:boto:Path: /hiiqos-13/obj?versionId=fqZHgkQ0VnkgGOiotHJVpzmOgEM1PNX 2023-07-28T23:51:19.870 DEBUG:boto:Data: 2023-07-28T23:51:19.870 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.870 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:19.871 DEBUG:boto:Port: 8000 2023-07-28T23:51:19.871 DEBUG:boto:Params: {} 2023-07-28T23:51:19.871 DEBUG:boto:Token: None 2023-07-28T23:51:19.872 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/obj?versionId=fqZHgkQ0VnkgGOiotHJVpzmOgEM1PNX 2023-07-28T23:51:19.872 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dQEGJ/pPDe0VHl2HMSgOyRMAR0Y= 2023-07-28T23:51:19.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dQEGJ/pPDe0VHl2HMSgOyRMAR0Y=', 'Content-Length': '0'} 2023-07-28T23:51:19.875 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'fqZHgkQ0VnkgGOiotHJVpzmOgEM1PNX'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000389ec9621ef45b59-0064c45477-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.875 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:19.876 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:19.876 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:19.877 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:19.877 DEBUG:boto:Method: GET 2023-07-28T23:51:19.877 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:19.878 DEBUG:boto:Data: 2023-07-28T23:51:19.878 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.879 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:19.879 DEBUG:boto:Port: 8001 2023-07-28T23:51:19.879 DEBUG:boto:Params: {} 2023-07-28T23:51:19.880 DEBUG:boto:Token: None 2023-07-28T23:51:19.880 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/ 2023-07-28T23:51:19.881 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DlyNiTjyCdClD3M1DZidenjgwug= 2023-07-28T23:51:19.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DlyNiTjyCdClD3M1DZidenjgwug=', 'Content-Length': '0'} 2023-07-28T23:51:19.884 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4a6b530a6eb8bb8-0064c45477-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.884 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:19.885 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:19.885 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:19.886 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:19.886 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:19.886 DEBUG:boto:Method: GET 2023-07-28T23:51:19.887 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:19.887 DEBUG:boto:Data: 2023-07-28T23:51:19.888 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.888 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:19.888 DEBUG:boto:Port: 8000 2023-07-28T23:51:19.889 DEBUG:boto:Params: {} 2023-07-28T23:51:19.889 DEBUG:boto:Token: None 2023-07-28T23:51:19.889 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/ 2023-07-28T23:51:19.890 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DlyNiTjyCdClD3M1DZidenjgwug= 2023-07-28T23:51:19.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DlyNiTjyCdClD3M1DZidenjgwug=', 'Content-Length': '0'} 2023-07-28T23:51:19.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000aeaea685ddb888e-0064c45477-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.893 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:19.894 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:19.894 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:19.895 DEBUG:boto:Method: HEAD 2023-07-28T23:51:19.895 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:51:19.895 DEBUG:boto:Data: 2023-07-28T23:51:19.896 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.896 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:19.896 DEBUG:boto:Port: 8001 2023-07-28T23:51:19.897 DEBUG:boto:Params: {} 2023-07-28T23:51:19.897 DEBUG:boto:Token: None 2023-07-28T23:51:19.897 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/obj 2023-07-28T23:51:19.898 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xzPhjsKFbd/IZXdX1u3vbOcG4xE= 2023-07-28T23:51:19.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xzPhjsKFbd/IZXdX1u3vbOcG4xE=', 'Content-Length': '0'} 2023-07-28T23:51:19.901 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001c354bce6d32f3c6-0064c45477-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.902 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:19.902 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:19.902 DEBUG:boto:Method: HEAD 2023-07-28T23:51:19.903 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:51:19.903 DEBUG:boto:Data: 2023-07-28T23:51:19.903 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.904 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:19.904 DEBUG:boto:Port: 8000 2023-07-28T23:51:19.904 DEBUG:boto:Params: {} 2023-07-28T23:51:19.905 DEBUG:boto:Token: None 2023-07-28T23:51:19.905 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/obj 2023-07-28T23:51:19.905 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xzPhjsKFbd/IZXdX1u3vbOcG4xE= 2023-07-28T23:51:19.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xzPhjsKFbd/IZXdX1u3vbOcG4xE=', 'Content-Length': '0'} 2023-07-28T23:51:19.909 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b36e153132b4a9d-0064c45477-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:19.910 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:19.910 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:19.910 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:19.911 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:19.911 DEBUG:boto:Method: GET 2023-07-28T23:51:19.911 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:19.912 DEBUG:boto:Data: 2023-07-28T23:51:19.912 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.913 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:19.913 DEBUG:boto:Port: 8001 2023-07-28T23:51:19.913 DEBUG:boto:Params: {} 2023-07-28T23:51:19.914 DEBUG:boto:Token: None 2023-07-28T23:51:19.914 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:19.914 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pFndM9SC0VYXa055LhLiBG1r5d8= 2023-07-28T23:51:19.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pFndM9SC0VYXa055LhLiBG1r5d8=', 'Content-Length': '0'} 2023-07-28T23:51:19.917 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ded516f7746ef42-0064c45477-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.964 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:19.965 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:19.965 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:19.966 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:19.966 DEBUG:boto:Method: GET 2023-07-28T23:51:19.966 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:19.967 DEBUG:boto:Data: 2023-07-28T23:51:19.967 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.967 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:19.968 DEBUG:boto:Port: 8000 2023-07-28T23:51:19.968 DEBUG:boto:Params: {} 2023-07-28T23:51:19.969 DEBUG:boto:Token: None 2023-07-28T23:51:19.969 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:19.969 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pFndM9SC0VYXa055LhLiBG1r5d8= 2023-07-28T23:51:19.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pFndM9SC0VYXa055LhLiBG1r5d8=', 'Content-Length': '0'} 2023-07-28T23:51:19.972 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f99582eca73af1a7-0064c45477-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.973 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:19.973 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:19.973 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:19.974 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:19.974 DEBUG:boto:Method: GET 2023-07-28T23:51:19.975 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:19.975 DEBUG:boto:Data: 2023-07-28T23:51:19.975 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.976 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:19.976 DEBUG:boto:Port: 8001 2023-07-28T23:51:19.976 DEBUG:boto:Params: {} 2023-07-28T23:51:19.977 DEBUG:boto:Token: None 2023-07-28T23:51:19.977 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/ 2023-07-28T23:51:19.978 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DlyNiTjyCdClD3M1DZidenjgwug= 2023-07-28T23:51:19.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DlyNiTjyCdClD3M1DZidenjgwug=', 'Content-Length': '0'} 2023-07-28T23:51:19.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000207e72811925c0ea-0064c45477-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.981 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:19.982 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:19.982 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:19.983 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:19.983 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:19.984 DEBUG:boto:Method: GET 2023-07-28T23:51:19.985 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:19.986 DEBUG:boto:Data: 2023-07-28T23:51:19.986 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.987 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:19.988 DEBUG:boto:Port: 8000 2023-07-28T23:51:19.988 DEBUG:boto:Params: {} 2023-07-28T23:51:19.989 DEBUG:boto:Token: None 2023-07-28T23:51:19.990 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/ 2023-07-28T23:51:19.991 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DlyNiTjyCdClD3M1DZidenjgwug= 2023-07-28T23:51:19.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DlyNiTjyCdClD3M1DZidenjgwug=', 'Content-Length': '0'} 2023-07-28T23:51:19.993 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007004b83dcba6b711-0064c45477-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:19.994 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:19.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:19.995 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:19.995 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:19.995 DEBUG:boto:path=/hiiqos-13/obj?versionId=l2NN5Bw32G2ViK4cHi-XORehcdjCcxD 2023-07-28T23:51:19.996 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=l2NN5Bw32G2ViK4cHi-XORehcdjCcxD 2023-07-28T23:51:19.996 DEBUG:boto:Method: GET 2023-07-28T23:51:19.997 DEBUG:boto:Path: /hiiqos-13/obj?versionId=l2NN5Bw32G2ViK4cHi-XORehcdjCcxD 2023-07-28T23:51:19.997 DEBUG:boto:Data: 2023-07-28T23:51:19.997 DEBUG:boto:Headers: {} 2023-07-28T23:51:19.998 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:19.998 DEBUG:boto:Port: 8001 2023-07-28T23:51:19.998 DEBUG:boto:Params: {} 2023-07-28T23:51:19.999 DEBUG:boto:Token: None 2023-07-28T23:51:19.999 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:19 GMT /hiiqos-13/obj?versionId=l2NN5Bw32G2ViK4cHi-XORehcdjCcxD 2023-07-28T23:51:20.000 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lgDumKUXgfCzVm4hCNkqnCeoJ3k= 2023-07-28T23:51:20.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lgDumKUXgfCzVm4hCNkqnCeoJ3k=', 'Content-Length': '0'} 2023-07-28T23:51:20.002 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'l2NN5Bw32G2ViK4cHi-XORehcdjCcxD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a64e561828cf878-0064c45478-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.003 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.003 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.004 DEBUG:boto:path=/hiiqos-13/obj?versionId=l2NN5Bw32G2ViK4cHi-XORehcdjCcxD 2023-07-28T23:51:20.004 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=l2NN5Bw32G2ViK4cHi-XORehcdjCcxD 2023-07-28T23:51:20.004 DEBUG:boto:Method: GET 2023-07-28T23:51:20.005 DEBUG:boto:Path: /hiiqos-13/obj?versionId=l2NN5Bw32G2ViK4cHi-XORehcdjCcxD 2023-07-28T23:51:20.005 DEBUG:boto:Data: 2023-07-28T23:51:20.006 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.006 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:20.007 DEBUG:boto:Port: 8000 2023-07-28T23:51:20.007 DEBUG:boto:Params: {} 2023-07-28T23:51:20.008 DEBUG:boto:Token: None 2023-07-28T23:51:20.008 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj?versionId=l2NN5Bw32G2ViK4cHi-XORehcdjCcxD 2023-07-28T23:51:20.008 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OXabkelqeHxJq0YCWrJYGYuFm0c= 2023-07-28T23:51:20.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OXabkelqeHxJq0YCWrJYGYuFm0c=', 'Content-Length': '0'} 2023-07-28T23:51:20.011 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'l2NN5Bw32G2ViK4cHi-XORehcdjCcxD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000035c570eb93dbe106-0064c45478-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.056 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.057 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.057 DEBUG:boto:path=/hiiqos-13/obj?versionId=l2NN5Bw32G2ViK4cHi-XORehcdjCcxD 2023-07-28T23:51:20.058 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=l2NN5Bw32G2ViK4cHi-XORehcdjCcxD 2023-07-28T23:51:20.058 DEBUG:boto:Method: HEAD 2023-07-28T23:51:20.058 DEBUG:boto:Path: /hiiqos-13/obj?versionId=l2NN5Bw32G2ViK4cHi-XORehcdjCcxD 2023-07-28T23:51:20.059 DEBUG:boto:Data: 2023-07-28T23:51:20.059 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.059 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:20.060 DEBUG:boto:Port: 8001 2023-07-28T23:51:20.060 DEBUG:boto:Params: {} 2023-07-28T23:51:20.060 DEBUG:boto:Token: None 2023-07-28T23:51:20.061 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj?versionId=l2NN5Bw32G2ViK4cHi-XORehcdjCcxD 2023-07-28T23:51:20.061 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yohVE+7lKKJPVm1k6n7cp4odnWs= 2023-07-28T23:51:20.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yohVE+7lKKJPVm1k6n7cp4odnWs=', 'Content-Length': '0'} 2023-07-28T23:51:20.063 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'l2NN5Bw32G2ViK4cHi-XORehcdjCcxD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c4ac69b193efbc08-0064c45478-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.064 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.064 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.065 DEBUG:boto:path=/hiiqos-13/obj?versionId=l2NN5Bw32G2ViK4cHi-XORehcdjCcxD 2023-07-28T23:51:20.065 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=l2NN5Bw32G2ViK4cHi-XORehcdjCcxD 2023-07-28T23:51:20.066 DEBUG:boto:Method: HEAD 2023-07-28T23:51:20.066 DEBUG:boto:Path: /hiiqos-13/obj?versionId=l2NN5Bw32G2ViK4cHi-XORehcdjCcxD 2023-07-28T23:51:20.066 DEBUG:boto:Data: 2023-07-28T23:51:20.067 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.067 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:20.067 DEBUG:boto:Port: 8000 2023-07-28T23:51:20.068 DEBUG:boto:Params: {} 2023-07-28T23:51:20.068 DEBUG:boto:Token: None 2023-07-28T23:51:20.068 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj?versionId=l2NN5Bw32G2ViK4cHi-XORehcdjCcxD 2023-07-28T23:51:20.069 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yohVE+7lKKJPVm1k6n7cp4odnWs= 2023-07-28T23:51:20.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yohVE+7lKKJPVm1k6n7cp4odnWs=', 'Content-Length': '0'} 2023-07-28T23:51:20.071 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'l2NN5Bw32G2ViK4cHi-XORehcdjCcxD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000beee0bea00eb00ac-0064c45478-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:20.072 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.072 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.073 DEBUG:boto:path=/hiiqos-13/obj?versionId=l2NN5Bw32G2ViK4cHi-XORehcdjCcxD 2023-07-28T23:51:20.073 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=l2NN5Bw32G2ViK4cHi-XORehcdjCcxD 2023-07-28T23:51:20.073 DEBUG:boto:Method: GET 2023-07-28T23:51:20.074 DEBUG:boto:Path: /hiiqos-13/obj?versionId=l2NN5Bw32G2ViK4cHi-XORehcdjCcxD 2023-07-28T23:51:20.074 DEBUG:boto:Data: 2023-07-28T23:51:20.075 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.075 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:20.075 DEBUG:boto:Port: 8001 2023-07-28T23:51:20.076 DEBUG:boto:Params: {} 2023-07-28T23:51:20.076 DEBUG:boto:Token: None 2023-07-28T23:51:20.077 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj?versionId=l2NN5Bw32G2ViK4cHi-XORehcdjCcxD 2023-07-28T23:51:20.077 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OXabkelqeHxJq0YCWrJYGYuFm0c= 2023-07-28T23:51:20.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OXabkelqeHxJq0YCWrJYGYuFm0c=', 'Content-Length': '0'} 2023-07-28T23:51:20.080 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'l2NN5Bw32G2ViK4cHi-XORehcdjCcxD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006f8083fcfeb258bf-0064c45478-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.120 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.121 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.121 DEBUG:boto:path=/hiiqos-13/obj?versionId=l2NN5Bw32G2ViK4cHi-XORehcdjCcxD 2023-07-28T23:51:20.122 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=l2NN5Bw32G2ViK4cHi-XORehcdjCcxD 2023-07-28T23:51:20.122 DEBUG:boto:Method: GET 2023-07-28T23:51:20.122 DEBUG:boto:Path: /hiiqos-13/obj?versionId=l2NN5Bw32G2ViK4cHi-XORehcdjCcxD 2023-07-28T23:51:20.123 DEBUG:boto:Data: 2023-07-28T23:51:20.123 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.123 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:20.124 DEBUG:boto:Port: 8000 2023-07-28T23:51:20.124 DEBUG:boto:Params: {} 2023-07-28T23:51:20.125 DEBUG:boto:Token: None 2023-07-28T23:51:20.125 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj?versionId=l2NN5Bw32G2ViK4cHi-XORehcdjCcxD 2023-07-28T23:51:20.125 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OXabkelqeHxJq0YCWrJYGYuFm0c= 2023-07-28T23:51:20.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OXabkelqeHxJq0YCWrJYGYuFm0c=', 'Content-Length': '0'} 2023-07-28T23:51:20.128 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'l2NN5Bw32G2ViK4cHi-XORehcdjCcxD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000be0f397bbad3e1fc-0064c45478-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.129 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:20.129 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:20.130 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:20.130 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:20.131 DEBUG:boto:Method: GET 2023-07-28T23:51:20.131 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:20.132 DEBUG:boto:Data: 2023-07-28T23:51:20.132 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.132 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:20.133 DEBUG:boto:Port: 8001 2023-07-28T23:51:20.133 DEBUG:boto:Params: {} 2023-07-28T23:51:20.134 DEBUG:boto:Token: None 2023-07-28T23:51:20.134 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/ 2023-07-28T23:51:20.134 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Nci8g8Z2joYrIfL1ye83nnXBwgA= 2023-07-28T23:51:20.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Nci8g8Z2joYrIfL1ye83nnXBwgA=', 'Content-Length': '0'} 2023-07-28T23:51:20.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083b10ea7abb0458a-0064c45478-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.180 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:20.181 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:20.182 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:20.182 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:20.182 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:20.183 DEBUG:boto:Method: GET 2023-07-28T23:51:20.183 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:20.183 DEBUG:boto:Data: 2023-07-28T23:51:20.184 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.184 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:20.185 DEBUG:boto:Port: 8000 2023-07-28T23:51:20.185 DEBUG:boto:Params: {} 2023-07-28T23:51:20.185 DEBUG:boto:Token: None 2023-07-28T23:51:20.186 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/ 2023-07-28T23:51:20.186 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Nci8g8Z2joYrIfL1ye83nnXBwgA= 2023-07-28T23:51:20.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Nci8g8Z2joYrIfL1ye83nnXBwgA=', 'Content-Length': '0'} 2023-07-28T23:51:20.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2858611ee37c21a-0064c45478-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.190 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:20.191 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.191 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.191 DEBUG:boto:Method: HEAD 2023-07-28T23:51:20.192 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:51:20.192 DEBUG:boto:Data: 2023-07-28T23:51:20.192 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.193 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:20.193 DEBUG:boto:Port: 8001 2023-07-28T23:51:20.193 DEBUG:boto:Params: {} 2023-07-28T23:51:20.194 DEBUG:boto:Token: None 2023-07-28T23:51:20.194 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj 2023-07-28T23:51:20.195 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zmRLUZcIvpKNWfo9V9Gfj2LeSXw= 2023-07-28T23:51:20.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zmRLUZcIvpKNWfo9V9Gfj2LeSXw=', 'Content-Length': '0'} 2023-07-28T23:51:20.198 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008b2ac26a9bdb0c93-0064c45478-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.199 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.199 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.199 DEBUG:boto:Method: HEAD 2023-07-28T23:51:20.200 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:51:20.200 DEBUG:boto:Data: 2023-07-28T23:51:20.200 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.201 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:20.201 DEBUG:boto:Port: 8000 2023-07-28T23:51:20.202 DEBUG:boto:Params: {} 2023-07-28T23:51:20.202 DEBUG:boto:Token: None 2023-07-28T23:51:20.202 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj 2023-07-28T23:51:20.203 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zmRLUZcIvpKNWfo9V9Gfj2LeSXw= 2023-07-28T23:51:20.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zmRLUZcIvpKNWfo9V9Gfj2LeSXw=', 'Content-Length': '0'} 2023-07-28T23:51:20.206 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000130a6f1d57e08c35-0064c45478-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:20.207 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.207 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.208 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:20.208 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:20.209 DEBUG:boto:Method: GET 2023-07-28T23:51:20.209 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:20.209 DEBUG:boto:Data: 2023-07-28T23:51:20.210 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.210 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:20.210 DEBUG:boto:Port: 8001 2023-07-28T23:51:20.211 DEBUG:boto:Params: {} 2023-07-28T23:51:20.211 DEBUG:boto:Token: None 2023-07-28T23:51:20.212 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:20.212 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1/QXa/Q6NZxV1Qe0Rm/dsjDBaGI= 2023-07-28T23:51:20.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1/QXa/Q6NZxV1Qe0Rm/dsjDBaGI=', 'Content-Length': '0'} 2023-07-28T23:51:20.214 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f1588f0694cd7512-0064c45478-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.260 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.261 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.261 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:20.262 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:20.262 DEBUG:boto:Method: GET 2023-07-28T23:51:20.262 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:20.263 DEBUG:boto:Data: 2023-07-28T23:51:20.263 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.263 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:20.264 DEBUG:boto:Port: 8000 2023-07-28T23:51:20.264 DEBUG:boto:Params: {} 2023-07-28T23:51:20.265 DEBUG:boto:Token: None 2023-07-28T23:51:20.265 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:20.266 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1/QXa/Q6NZxV1Qe0Rm/dsjDBaGI= 2023-07-28T23:51:20.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1/QXa/Q6NZxV1Qe0Rm/dsjDBaGI=', 'Content-Length': '0'} 2023-07-28T23:51:20.268 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d568310a015b0c9-0064c45478-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.269 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:20.269 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:20.270 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:20.270 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:20.270 DEBUG:boto:Method: GET 2023-07-28T23:51:20.271 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:20.271 DEBUG:boto:Data: 2023-07-28T23:51:20.271 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.272 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:20.272 DEBUG:boto:Port: 8001 2023-07-28T23:51:20.272 DEBUG:boto:Params: {} 2023-07-28T23:51:20.273 DEBUG:boto:Token: None 2023-07-28T23:51:20.273 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/ 2023-07-28T23:51:20.274 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Nci8g8Z2joYrIfL1ye83nnXBwgA= 2023-07-28T23:51:20.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Nci8g8Z2joYrIfL1ye83nnXBwgA=', 'Content-Length': '0'} 2023-07-28T23:51:20.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067a7a5243ff8daf8-0064c45478-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.320 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:20.321 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:20.322 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:20.322 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:20.322 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:20.323 DEBUG:boto:Method: GET 2023-07-28T23:51:20.323 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:20.323 DEBUG:boto:Data: 2023-07-28T23:51:20.324 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.324 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:20.325 DEBUG:boto:Port: 8000 2023-07-28T23:51:20.325 DEBUG:boto:Params: {} 2023-07-28T23:51:20.325 DEBUG:boto:Token: None 2023-07-28T23:51:20.326 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/ 2023-07-28T23:51:20.326 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Nci8g8Z2joYrIfL1ye83nnXBwgA= 2023-07-28T23:51:20.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Nci8g8Z2joYrIfL1ye83nnXBwgA=', 'Content-Length': '0'} 2023-07-28T23:51:20.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d363c2a327889fe-0064c45478-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.330 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:20.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:20.331 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.331 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.332 DEBUG:boto:path=/hiiqos-13/obj?versionId=MMa4nLQC3W82FeUky1Sq9XW9uFjpW6Q 2023-07-28T23:51:20.332 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=MMa4nLQC3W82FeUky1Sq9XW9uFjpW6Q 2023-07-28T23:51:20.333 DEBUG:boto:Method: GET 2023-07-28T23:51:20.333 DEBUG:boto:Path: /hiiqos-13/obj?versionId=MMa4nLQC3W82FeUky1Sq9XW9uFjpW6Q 2023-07-28T23:51:20.333 DEBUG:boto:Data: 2023-07-28T23:51:20.334 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.334 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:20.334 DEBUG:boto:Port: 8001 2023-07-28T23:51:20.335 DEBUG:boto:Params: {} 2023-07-28T23:51:20.335 DEBUG:boto:Token: None 2023-07-28T23:51:20.336 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj?versionId=MMa4nLQC3W82FeUky1Sq9XW9uFjpW6Q 2023-07-28T23:51:20.336 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+G7WlZPy4CREzLJUy5ex7dxR4VM= 2023-07-28T23:51:20.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+G7WlZPy4CREzLJUy5ex7dxR4VM=', 'Content-Length': '0'} 2023-07-28T23:51:20.339 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'MMa4nLQC3W82FeUky1Sq9XW9uFjpW6Q'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000185facba810bb80f-0064c45478-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.339 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.340 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.340 DEBUG:boto:path=/hiiqos-13/obj?versionId=MMa4nLQC3W82FeUky1Sq9XW9uFjpW6Q 2023-07-28T23:51:20.341 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=MMa4nLQC3W82FeUky1Sq9XW9uFjpW6Q 2023-07-28T23:51:20.341 DEBUG:boto:Method: GET 2023-07-28T23:51:20.341 DEBUG:boto:Path: /hiiqos-13/obj?versionId=MMa4nLQC3W82FeUky1Sq9XW9uFjpW6Q 2023-07-28T23:51:20.342 DEBUG:boto:Data: 2023-07-28T23:51:20.342 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.342 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:20.343 DEBUG:boto:Port: 8000 2023-07-28T23:51:20.343 DEBUG:boto:Params: {} 2023-07-28T23:51:20.344 DEBUG:boto:Token: None 2023-07-28T23:51:20.344 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj?versionId=MMa4nLQC3W82FeUky1Sq9XW9uFjpW6Q 2023-07-28T23:51:20.344 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+G7WlZPy4CREzLJUy5ex7dxR4VM= 2023-07-28T23:51:20.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+G7WlZPy4CREzLJUy5ex7dxR4VM=', 'Content-Length': '0'} 2023-07-28T23:51:20.347 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'MMa4nLQC3W82FeUky1Sq9XW9uFjpW6Q'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000086a162e28042d29c-0064c45478-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.388 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.389 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.389 DEBUG:boto:path=/hiiqos-13/obj?versionId=MMa4nLQC3W82FeUky1Sq9XW9uFjpW6Q 2023-07-28T23:51:20.390 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=MMa4nLQC3W82FeUky1Sq9XW9uFjpW6Q 2023-07-28T23:51:20.390 DEBUG:boto:Method: HEAD 2023-07-28T23:51:20.391 DEBUG:boto:Path: /hiiqos-13/obj?versionId=MMa4nLQC3W82FeUky1Sq9XW9uFjpW6Q 2023-07-28T23:51:20.391 DEBUG:boto:Data: 2023-07-28T23:51:20.391 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.392 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:20.392 DEBUG:boto:Port: 8001 2023-07-28T23:51:20.392 DEBUG:boto:Params: {} 2023-07-28T23:51:20.393 DEBUG:boto:Token: None 2023-07-28T23:51:20.393 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj?versionId=MMa4nLQC3W82FeUky1Sq9XW9uFjpW6Q 2023-07-28T23:51:20.394 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MqH4coVCt+tSNUWgloTfy9jtEYk= 2023-07-28T23:51:20.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MqH4coVCt+tSNUWgloTfy9jtEYk=', 'Content-Length': '0'} 2023-07-28T23:51:20.397 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'MMa4nLQC3W82FeUky1Sq9XW9uFjpW6Q'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000007a7419b4ceb914d-0064c45478-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.398 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.399 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.399 DEBUG:boto:path=/hiiqos-13/obj?versionId=MMa4nLQC3W82FeUky1Sq9XW9uFjpW6Q 2023-07-28T23:51:20.400 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=MMa4nLQC3W82FeUky1Sq9XW9uFjpW6Q 2023-07-28T23:51:20.401 DEBUG:boto:Method: HEAD 2023-07-28T23:51:20.401 DEBUG:boto:Path: /hiiqos-13/obj?versionId=MMa4nLQC3W82FeUky1Sq9XW9uFjpW6Q 2023-07-28T23:51:20.402 DEBUG:boto:Data: 2023-07-28T23:51:20.403 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.403 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:20.404 DEBUG:boto:Port: 8000 2023-07-28T23:51:20.404 DEBUG:boto:Params: {} 2023-07-28T23:51:20.405 DEBUG:boto:Token: None 2023-07-28T23:51:20.406 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj?versionId=MMa4nLQC3W82FeUky1Sq9XW9uFjpW6Q 2023-07-28T23:51:20.406 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MqH4coVCt+tSNUWgloTfy9jtEYk= 2023-07-28T23:51:20.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MqH4coVCt+tSNUWgloTfy9jtEYk=', 'Content-Length': '0'} 2023-07-28T23:51:20.409 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'MMa4nLQC3W82FeUky1Sq9XW9uFjpW6Q'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a7e66b59f18ccf6d-0064c45478-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:20.410 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.410 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.411 DEBUG:boto:path=/hiiqos-13/obj?versionId=MMa4nLQC3W82FeUky1Sq9XW9uFjpW6Q 2023-07-28T23:51:20.411 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=MMa4nLQC3W82FeUky1Sq9XW9uFjpW6Q 2023-07-28T23:51:20.412 DEBUG:boto:Method: GET 2023-07-28T23:51:20.412 DEBUG:boto:Path: /hiiqos-13/obj?versionId=MMa4nLQC3W82FeUky1Sq9XW9uFjpW6Q 2023-07-28T23:51:20.413 DEBUG:boto:Data: 2023-07-28T23:51:20.413 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.413 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:20.414 DEBUG:boto:Port: 8001 2023-07-28T23:51:20.414 DEBUG:boto:Params: {} 2023-07-28T23:51:20.415 DEBUG:boto:Token: None 2023-07-28T23:51:20.415 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj?versionId=MMa4nLQC3W82FeUky1Sq9XW9uFjpW6Q 2023-07-28T23:51:20.416 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+G7WlZPy4CREzLJUy5ex7dxR4VM= 2023-07-28T23:51:20.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+G7WlZPy4CREzLJUy5ex7dxR4VM=', 'Content-Length': '0'} 2023-07-28T23:51:20.418 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'MMa4nLQC3W82FeUky1Sq9XW9uFjpW6Q'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000db61a082df5ee062-0064c45478-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.419 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.419 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.420 DEBUG:boto:path=/hiiqos-13/obj?versionId=MMa4nLQC3W82FeUky1Sq9XW9uFjpW6Q 2023-07-28T23:51:20.420 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=MMa4nLQC3W82FeUky1Sq9XW9uFjpW6Q 2023-07-28T23:51:20.420 DEBUG:boto:Method: GET 2023-07-28T23:51:20.421 DEBUG:boto:Path: /hiiqos-13/obj?versionId=MMa4nLQC3W82FeUky1Sq9XW9uFjpW6Q 2023-07-28T23:51:20.421 DEBUG:boto:Data: 2023-07-28T23:51:20.422 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.422 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:20.422 DEBUG:boto:Port: 8000 2023-07-28T23:51:20.423 DEBUG:boto:Params: {} 2023-07-28T23:51:20.423 DEBUG:boto:Token: None 2023-07-28T23:51:20.423 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj?versionId=MMa4nLQC3W82FeUky1Sq9XW9uFjpW6Q 2023-07-28T23:51:20.424 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+G7WlZPy4CREzLJUy5ex7dxR4VM= 2023-07-28T23:51:20.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+G7WlZPy4CREzLJUy5ex7dxR4VM=', 'Content-Length': '0'} 2023-07-28T23:51:20.426 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'MMa4nLQC3W82FeUky1Sq9XW9uFjpW6Q'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002610e44c25b58c4f-0064c45478-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.427 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:20.427 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:20.428 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:20.428 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:20.428 DEBUG:boto:Method: GET 2023-07-28T23:51:20.429 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:20.429 DEBUG:boto:Data: 2023-07-28T23:51:20.430 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.430 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:20.430 DEBUG:boto:Port: 8001 2023-07-28T23:51:20.431 DEBUG:boto:Params: {} 2023-07-28T23:51:20.431 DEBUG:boto:Token: None 2023-07-28T23:51:20.432 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/ 2023-07-28T23:51:20.432 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Nci8g8Z2joYrIfL1ye83nnXBwgA= 2023-07-28T23:51:20.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Nci8g8Z2joYrIfL1ye83nnXBwgA=', 'Content-Length': '0'} 2023-07-28T23:51:20.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d134cc007db3171-0064c45478-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.436 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:20.436 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:20.437 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:20.437 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:20.437 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:20.438 DEBUG:boto:Method: GET 2023-07-28T23:51:20.438 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:20.438 DEBUG:boto:Data: 2023-07-28T23:51:20.439 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.439 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:20.440 DEBUG:boto:Port: 8000 2023-07-28T23:51:20.440 DEBUG:boto:Params: {} 2023-07-28T23:51:20.440 DEBUG:boto:Token: None 2023-07-28T23:51:20.441 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/ 2023-07-28T23:51:20.441 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Nci8g8Z2joYrIfL1ye83nnXBwgA= 2023-07-28T23:51:20.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Nci8g8Z2joYrIfL1ye83nnXBwgA=', 'Content-Length': '0'} 2023-07-28T23:51:20.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099020fa3489b9840-0064c45478-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.445 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:20.446 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.446 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.446 DEBUG:boto:Method: HEAD 2023-07-28T23:51:20.447 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:51:20.447 DEBUG:boto:Data: 2023-07-28T23:51:20.447 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.448 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:20.448 DEBUG:boto:Port: 8001 2023-07-28T23:51:20.449 DEBUG:boto:Params: {} 2023-07-28T23:51:20.449 DEBUG:boto:Token: None 2023-07-28T23:51:20.449 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj 2023-07-28T23:51:20.450 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zmRLUZcIvpKNWfo9V9Gfj2LeSXw= 2023-07-28T23:51:20.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zmRLUZcIvpKNWfo9V9Gfj2LeSXw=', 'Content-Length': '0'} 2023-07-28T23:51:20.453 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a8209a8dc885cf0b-0064c45478-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.454 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.454 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.455 DEBUG:boto:Method: HEAD 2023-07-28T23:51:20.455 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:51:20.455 DEBUG:boto:Data: 2023-07-28T23:51:20.456 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.456 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:20.456 DEBUG:boto:Port: 8000 2023-07-28T23:51:20.457 DEBUG:boto:Params: {} 2023-07-28T23:51:20.457 DEBUG:boto:Token: None 2023-07-28T23:51:20.458 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj 2023-07-28T23:51:20.458 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zmRLUZcIvpKNWfo9V9Gfj2LeSXw= 2023-07-28T23:51:20.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zmRLUZcIvpKNWfo9V9Gfj2LeSXw=', 'Content-Length': '0'} 2023-07-28T23:51:20.461 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a6b4f796cb071f5-0064c45478-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:20.463 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.463 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.463 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:20.464 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:20.464 DEBUG:boto:Method: GET 2023-07-28T23:51:20.464 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:20.465 DEBUG:boto:Data: 2023-07-28T23:51:20.465 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.465 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:20.466 DEBUG:boto:Port: 8001 2023-07-28T23:51:20.466 DEBUG:boto:Params: {} 2023-07-28T23:51:20.467 DEBUG:boto:Token: None 2023-07-28T23:51:20.467 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:20.468 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1/QXa/Q6NZxV1Qe0Rm/dsjDBaGI= 2023-07-28T23:51:20.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1/QXa/Q6NZxV1Qe0Rm/dsjDBaGI=', 'Content-Length': '0'} 2023-07-28T23:51:20.470 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000059a17112058847bc-0064c45478-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.512 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.513 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.513 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:20.514 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:20.514 DEBUG:boto:Method: GET 2023-07-28T23:51:20.515 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:20.515 DEBUG:boto:Data: 2023-07-28T23:51:20.515 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.516 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:20.516 DEBUG:boto:Port: 8000 2023-07-28T23:51:20.516 DEBUG:boto:Params: {} 2023-07-28T23:51:20.517 DEBUG:boto:Token: None 2023-07-28T23:51:20.517 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:20.518 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1/QXa/Q6NZxV1Qe0Rm/dsjDBaGI= 2023-07-28T23:51:20.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1/QXa/Q6NZxV1Qe0Rm/dsjDBaGI=', 'Content-Length': '0'} 2023-07-28T23:51:20.520 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000001c2dc02b8e923ad-0064c45478-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.521 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:20.521 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:20.521 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:20.522 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:20.522 DEBUG:boto:Method: GET 2023-07-28T23:51:20.523 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:20.523 DEBUG:boto:Data: 2023-07-28T23:51:20.524 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.524 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:20.524 DEBUG:boto:Port: 8001 2023-07-28T23:51:20.525 DEBUG:boto:Params: {} 2023-07-28T23:51:20.525 DEBUG:boto:Token: None 2023-07-28T23:51:20.525 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/ 2023-07-28T23:51:20.526 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Nci8g8Z2joYrIfL1ye83nnXBwgA= 2023-07-28T23:51:20.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Nci8g8Z2joYrIfL1ye83nnXBwgA=', 'Content-Length': '0'} 2023-07-28T23:51:20.529 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2cd895b1cb6ae05-0064c45478-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.530 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:20.530 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:20.531 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:20.531 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:20.532 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:20.532 DEBUG:boto:Method: GET 2023-07-28T23:51:20.532 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:20.533 DEBUG:boto:Data: 2023-07-28T23:51:20.533 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.534 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:20.534 DEBUG:boto:Port: 8000 2023-07-28T23:51:20.534 DEBUG:boto:Params: {} 2023-07-28T23:51:20.535 DEBUG:boto:Token: None 2023-07-28T23:51:20.535 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/ 2023-07-28T23:51:20.536 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Nci8g8Z2joYrIfL1ye83nnXBwgA= 2023-07-28T23:51:20.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Nci8g8Z2joYrIfL1ye83nnXBwgA=', 'Content-Length': '0'} 2023-07-28T23:51:20.539 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000644b04d8612b9d8d-0064c45478-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.540 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:20.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:20.541 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.541 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.542 DEBUG:boto:path=/hiiqos-13/obj?versionId=fxsCLuGqt0tjCmqlJgclHqGWnHEcxLA 2023-07-28T23:51:20.542 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=fxsCLuGqt0tjCmqlJgclHqGWnHEcxLA 2023-07-28T23:51:20.542 DEBUG:boto:Method: GET 2023-07-28T23:51:20.543 DEBUG:boto:Path: /hiiqos-13/obj?versionId=fxsCLuGqt0tjCmqlJgclHqGWnHEcxLA 2023-07-28T23:51:20.543 DEBUG:boto:Data: 2023-07-28T23:51:20.544 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.544 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:20.544 DEBUG:boto:Port: 8001 2023-07-28T23:51:20.545 DEBUG:boto:Params: {} 2023-07-28T23:51:20.545 DEBUG:boto:Token: None 2023-07-28T23:51:20.546 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj?versionId=fxsCLuGqt0tjCmqlJgclHqGWnHEcxLA 2023-07-28T23:51:20.546 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gmoIbHL3Kfrb1uvmRFdukcl/mbY= 2023-07-28T23:51:20.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gmoIbHL3Kfrb1uvmRFdukcl/mbY=', 'Content-Length': '0'} 2023-07-28T23:51:20.549 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'fxsCLuGqt0tjCmqlJgclHqGWnHEcxLA'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000040ae5f27ed1a00db-0064c45478-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.549 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.550 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.550 DEBUG:boto:path=/hiiqos-13/obj?versionId=fxsCLuGqt0tjCmqlJgclHqGWnHEcxLA 2023-07-28T23:51:20.550 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=fxsCLuGqt0tjCmqlJgclHqGWnHEcxLA 2023-07-28T23:51:20.551 DEBUG:boto:Method: GET 2023-07-28T23:51:20.551 DEBUG:boto:Path: /hiiqos-13/obj?versionId=fxsCLuGqt0tjCmqlJgclHqGWnHEcxLA 2023-07-28T23:51:20.552 DEBUG:boto:Data: 2023-07-28T23:51:20.552 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.552 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:20.553 DEBUG:boto:Port: 8000 2023-07-28T23:51:20.553 DEBUG:boto:Params: {} 2023-07-28T23:51:20.553 DEBUG:boto:Token: None 2023-07-28T23:51:20.554 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj?versionId=fxsCLuGqt0tjCmqlJgclHqGWnHEcxLA 2023-07-28T23:51:20.554 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gmoIbHL3Kfrb1uvmRFdukcl/mbY= 2023-07-28T23:51:20.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gmoIbHL3Kfrb1uvmRFdukcl/mbY=', 'Content-Length': '0'} 2023-07-28T23:51:20.556 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'fxsCLuGqt0tjCmqlJgclHqGWnHEcxLA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003ec115acf1e88eac-0064c45478-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.557 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.557 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.558 DEBUG:boto:path=/hiiqos-13/obj?versionId=fxsCLuGqt0tjCmqlJgclHqGWnHEcxLA 2023-07-28T23:51:20.558 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=fxsCLuGqt0tjCmqlJgclHqGWnHEcxLA 2023-07-28T23:51:20.558 DEBUG:boto:Method: HEAD 2023-07-28T23:51:20.559 DEBUG:boto:Path: /hiiqos-13/obj?versionId=fxsCLuGqt0tjCmqlJgclHqGWnHEcxLA 2023-07-28T23:51:20.559 DEBUG:boto:Data: 2023-07-28T23:51:20.560 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.560 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:20.561 DEBUG:boto:Port: 8001 2023-07-28T23:51:20.561 DEBUG:boto:Params: {} 2023-07-28T23:51:20.561 DEBUG:boto:Token: None 2023-07-28T23:51:20.562 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj?versionId=fxsCLuGqt0tjCmqlJgclHqGWnHEcxLA 2023-07-28T23:51:20.562 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TQ9o9L86snXpDj93tiSx/wG4Pfs= 2023-07-28T23:51:20.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TQ9o9L86snXpDj93tiSx/wG4Pfs=', 'Content-Length': '0'} 2023-07-28T23:51:20.564 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'fxsCLuGqt0tjCmqlJgclHqGWnHEcxLA'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000041d147971e2240c1-0064c45478-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.565 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.565 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.566 DEBUG:boto:path=/hiiqos-13/obj?versionId=fxsCLuGqt0tjCmqlJgclHqGWnHEcxLA 2023-07-28T23:51:20.566 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=fxsCLuGqt0tjCmqlJgclHqGWnHEcxLA 2023-07-28T23:51:20.566 DEBUG:boto:Method: HEAD 2023-07-28T23:51:20.567 DEBUG:boto:Path: /hiiqos-13/obj?versionId=fxsCLuGqt0tjCmqlJgclHqGWnHEcxLA 2023-07-28T23:51:20.567 DEBUG:boto:Data: 2023-07-28T23:51:20.567 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.568 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:20.568 DEBUG:boto:Port: 8000 2023-07-28T23:51:20.568 DEBUG:boto:Params: {} 2023-07-28T23:51:20.569 DEBUG:boto:Token: None 2023-07-28T23:51:20.569 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj?versionId=fxsCLuGqt0tjCmqlJgclHqGWnHEcxLA 2023-07-28T23:51:20.570 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TQ9o9L86snXpDj93tiSx/wG4Pfs= 2023-07-28T23:51:20.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TQ9o9L86snXpDj93tiSx/wG4Pfs=', 'Content-Length': '0'} 2023-07-28T23:51:20.571 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'fxsCLuGqt0tjCmqlJgclHqGWnHEcxLA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a610854274fa4440-0064c45478-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:20.572 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.573 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.573 DEBUG:boto:path=/hiiqos-13/obj?versionId=fxsCLuGqt0tjCmqlJgclHqGWnHEcxLA 2023-07-28T23:51:20.574 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=fxsCLuGqt0tjCmqlJgclHqGWnHEcxLA 2023-07-28T23:51:20.574 DEBUG:boto:Method: GET 2023-07-28T23:51:20.574 DEBUG:boto:Path: /hiiqos-13/obj?versionId=fxsCLuGqt0tjCmqlJgclHqGWnHEcxLA 2023-07-28T23:51:20.575 DEBUG:boto:Data: 2023-07-28T23:51:20.575 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.575 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:20.576 DEBUG:boto:Port: 8001 2023-07-28T23:51:20.576 DEBUG:boto:Params: {} 2023-07-28T23:51:20.577 DEBUG:boto:Token: None 2023-07-28T23:51:20.577 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj?versionId=fxsCLuGqt0tjCmqlJgclHqGWnHEcxLA 2023-07-28T23:51:20.578 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gmoIbHL3Kfrb1uvmRFdukcl/mbY= 2023-07-28T23:51:20.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gmoIbHL3Kfrb1uvmRFdukcl/mbY=', 'Content-Length': '0'} 2023-07-28T23:51:20.580 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'fxsCLuGqt0tjCmqlJgclHqGWnHEcxLA'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a81c0760efe37eaa-0064c45478-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.620 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.621 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.621 DEBUG:boto:path=/hiiqos-13/obj?versionId=fxsCLuGqt0tjCmqlJgclHqGWnHEcxLA 2023-07-28T23:51:20.622 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=fxsCLuGqt0tjCmqlJgclHqGWnHEcxLA 2023-07-28T23:51:20.622 DEBUG:boto:Method: GET 2023-07-28T23:51:20.622 DEBUG:boto:Path: /hiiqos-13/obj?versionId=fxsCLuGqt0tjCmqlJgclHqGWnHEcxLA 2023-07-28T23:51:20.623 DEBUG:boto:Data: 2023-07-28T23:51:20.623 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.623 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:20.624 DEBUG:boto:Port: 8000 2023-07-28T23:51:20.624 DEBUG:boto:Params: {} 2023-07-28T23:51:20.625 DEBUG:boto:Token: None 2023-07-28T23:51:20.625 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj?versionId=fxsCLuGqt0tjCmqlJgclHqGWnHEcxLA 2023-07-28T23:51:20.625 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gmoIbHL3Kfrb1uvmRFdukcl/mbY= 2023-07-28T23:51:20.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gmoIbHL3Kfrb1uvmRFdukcl/mbY=', 'Content-Length': '0'} 2023-07-28T23:51:20.628 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'fxsCLuGqt0tjCmqlJgclHqGWnHEcxLA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001383baa50c96b85c-0064c45478-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.628 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:20.629 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:20.629 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:20.630 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:20.630 DEBUG:boto:Method: GET 2023-07-28T23:51:20.630 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:20.631 DEBUG:boto:Data: 2023-07-28T23:51:20.631 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.631 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:20.632 DEBUG:boto:Port: 8001 2023-07-28T23:51:20.632 DEBUG:boto:Params: {} 2023-07-28T23:51:20.633 DEBUG:boto:Token: None 2023-07-28T23:51:20.633 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/ 2023-07-28T23:51:20.633 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Nci8g8Z2joYrIfL1ye83nnXBwgA= 2023-07-28T23:51:20.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Nci8g8Z2joYrIfL1ye83nnXBwgA=', 'Content-Length': '0'} 2023-07-28T23:51:20.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009cd029e6599f7c85-0064c45478-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.637 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:20.638 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:20.638 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:20.638 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:20.639 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:20.639 DEBUG:boto:Method: GET 2023-07-28T23:51:20.639 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:20.640 DEBUG:boto:Data: 2023-07-28T23:51:20.640 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.641 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:20.641 DEBUG:boto:Port: 8000 2023-07-28T23:51:20.641 DEBUG:boto:Params: {} 2023-07-28T23:51:20.642 DEBUG:boto:Token: None 2023-07-28T23:51:20.642 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/ 2023-07-28T23:51:20.642 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Nci8g8Z2joYrIfL1ye83nnXBwgA= 2023-07-28T23:51:20.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Nci8g8Z2joYrIfL1ye83nnXBwgA=', 'Content-Length': '0'} 2023-07-28T23:51:20.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdd4d526489c2721-0064c45478-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.646 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:20.647 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.647 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.647 DEBUG:boto:Method: HEAD 2023-07-28T23:51:20.648 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:51:20.648 DEBUG:boto:Data: 2023-07-28T23:51:20.648 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.649 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:20.649 DEBUG:boto:Port: 8001 2023-07-28T23:51:20.649 DEBUG:boto:Params: {} 2023-07-28T23:51:20.650 DEBUG:boto:Token: None 2023-07-28T23:51:20.650 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj 2023-07-28T23:51:20.651 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zmRLUZcIvpKNWfo9V9Gfj2LeSXw= 2023-07-28T23:51:20.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zmRLUZcIvpKNWfo9V9Gfj2LeSXw=', 'Content-Length': '0'} 2023-07-28T23:51:20.654 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005741680bb1279791-0064c45478-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.655 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.655 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.655 DEBUG:boto:Method: HEAD 2023-07-28T23:51:20.656 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:51:20.656 DEBUG:boto:Data: 2023-07-28T23:51:20.656 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.657 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:20.657 DEBUG:boto:Port: 8000 2023-07-28T23:51:20.657 DEBUG:boto:Params: {} 2023-07-28T23:51:20.658 DEBUG:boto:Token: None 2023-07-28T23:51:20.658 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj 2023-07-28T23:51:20.658 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zmRLUZcIvpKNWfo9V9Gfj2LeSXw= 2023-07-28T23:51:20.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zmRLUZcIvpKNWfo9V9Gfj2LeSXw=', 'Content-Length': '0'} 2023-07-28T23:51:20.662 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000174dbfaf859090c8-0064c45478-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:20.663 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.664 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.665 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:20.665 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:20.666 DEBUG:boto:Method: GET 2023-07-28T23:51:20.667 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:20.668 DEBUG:boto:Data: 2023-07-28T23:51:20.669 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.670 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:20.670 DEBUG:boto:Port: 8001 2023-07-28T23:51:20.670 DEBUG:boto:Params: {} 2023-07-28T23:51:20.671 DEBUG:boto:Token: None 2023-07-28T23:51:20.671 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:20.672 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1/QXa/Q6NZxV1Qe0Rm/dsjDBaGI= 2023-07-28T23:51:20.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1/QXa/Q6NZxV1Qe0Rm/dsjDBaGI=', 'Content-Length': '0'} 2023-07-28T23:51:20.674 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000089000c0ea0313fc2-0064c45478-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.716 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.717 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.717 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:20.718 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:20.718 DEBUG:boto:Method: GET 2023-07-28T23:51:20.718 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:20.719 DEBUG:boto:Data: 2023-07-28T23:51:20.719 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.719 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:20.720 DEBUG:boto:Port: 8000 2023-07-28T23:51:20.720 DEBUG:boto:Params: {} 2023-07-28T23:51:20.721 DEBUG:boto:Token: None 2023-07-28T23:51:20.721 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:20.721 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1/QXa/Q6NZxV1Qe0Rm/dsjDBaGI= 2023-07-28T23:51:20.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1/QXa/Q6NZxV1Qe0Rm/dsjDBaGI=', 'Content-Length': '0'} 2023-07-28T23:51:20.724 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ae066ce797ab697-0064c45478-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.724 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:20.725 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:20.725 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:20.726 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:20.726 DEBUG:boto:Method: GET 2023-07-28T23:51:20.726 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:20.727 DEBUG:boto:Data: 2023-07-28T23:51:20.727 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.727 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:20.728 DEBUG:boto:Port: 8001 2023-07-28T23:51:20.728 DEBUG:boto:Params: {} 2023-07-28T23:51:20.729 DEBUG:boto:Token: None 2023-07-28T23:51:20.729 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/ 2023-07-28T23:51:20.729 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Nci8g8Z2joYrIfL1ye83nnXBwgA= 2023-07-28T23:51:20.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Nci8g8Z2joYrIfL1ye83nnXBwgA=', 'Content-Length': '0'} 2023-07-28T23:51:20.732 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7b9ffce29ad14c7-0064c45478-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.733 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:20.734 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:20.734 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:20.734 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:20.735 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:20.735 DEBUG:boto:Method: GET 2023-07-28T23:51:20.735 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:20.736 DEBUG:boto:Data: 2023-07-28T23:51:20.736 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.737 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:20.737 DEBUG:boto:Port: 8000 2023-07-28T23:51:20.737 DEBUG:boto:Params: {} 2023-07-28T23:51:20.738 DEBUG:boto:Token: None 2023-07-28T23:51:20.738 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/ 2023-07-28T23:51:20.738 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Nci8g8Z2joYrIfL1ye83nnXBwgA= 2023-07-28T23:51:20.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Nci8g8Z2joYrIfL1ye83nnXBwgA=', 'Content-Length': '0'} 2023-07-28T23:51:20.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efb64b8b935e00d7-0064c45478-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.742 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:20.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:20.743 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.743 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.743 DEBUG:boto:path=/hiiqos-13/obj?versionId=22KG5jfO2TftvgPgeJvtwXX52mGQhV1 2023-07-28T23:51:20.744 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=22KG5jfO2TftvgPgeJvtwXX52mGQhV1 2023-07-28T23:51:20.744 DEBUG:boto:Method: GET 2023-07-28T23:51:20.745 DEBUG:boto:Path: /hiiqos-13/obj?versionId=22KG5jfO2TftvgPgeJvtwXX52mGQhV1 2023-07-28T23:51:20.745 DEBUG:boto:Data: 2023-07-28T23:51:20.745 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.746 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:20.746 DEBUG:boto:Port: 8001 2023-07-28T23:51:20.747 DEBUG:boto:Params: {} 2023-07-28T23:51:20.747 DEBUG:boto:Token: None 2023-07-28T23:51:20.747 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj?versionId=22KG5jfO2TftvgPgeJvtwXX52mGQhV1 2023-07-28T23:51:20.748 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GNm5hhVA/ml1qEd5obzxscpZNnI= 2023-07-28T23:51:20.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GNm5hhVA/ml1qEd5obzxscpZNnI=', 'Content-Length': '0'} 2023-07-28T23:51:20.750 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', '22KG5jfO2TftvgPgeJvtwXX52mGQhV1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ac12d3e55fe26e88-0064c45478-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.751 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.751 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.751 DEBUG:boto:path=/hiiqos-13/obj?versionId=22KG5jfO2TftvgPgeJvtwXX52mGQhV1 2023-07-28T23:51:20.752 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=22KG5jfO2TftvgPgeJvtwXX52mGQhV1 2023-07-28T23:51:20.752 DEBUG:boto:Method: GET 2023-07-28T23:51:20.752 DEBUG:boto:Path: /hiiqos-13/obj?versionId=22KG5jfO2TftvgPgeJvtwXX52mGQhV1 2023-07-28T23:51:20.753 DEBUG:boto:Data: 2023-07-28T23:51:20.753 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.754 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:20.754 DEBUG:boto:Port: 8000 2023-07-28T23:51:20.754 DEBUG:boto:Params: {} 2023-07-28T23:51:20.755 DEBUG:boto:Token: None 2023-07-28T23:51:20.755 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj?versionId=22KG5jfO2TftvgPgeJvtwXX52mGQhV1 2023-07-28T23:51:20.756 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GNm5hhVA/ml1qEd5obzxscpZNnI= 2023-07-28T23:51:20.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GNm5hhVA/ml1qEd5obzxscpZNnI=', 'Content-Length': '0'} 2023-07-28T23:51:20.758 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', '22KG5jfO2TftvgPgeJvtwXX52mGQhV1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000027b29014d958cf70-0064c45478-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.800 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.801 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.801 DEBUG:boto:path=/hiiqos-13/obj?versionId=22KG5jfO2TftvgPgeJvtwXX52mGQhV1 2023-07-28T23:51:20.802 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=22KG5jfO2TftvgPgeJvtwXX52mGQhV1 2023-07-28T23:51:20.802 DEBUG:boto:Method: HEAD 2023-07-28T23:51:20.802 DEBUG:boto:Path: /hiiqos-13/obj?versionId=22KG5jfO2TftvgPgeJvtwXX52mGQhV1 2023-07-28T23:51:20.803 DEBUG:boto:Data: 2023-07-28T23:51:20.803 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.803 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:20.804 DEBUG:boto:Port: 8001 2023-07-28T23:51:20.804 DEBUG:boto:Params: {} 2023-07-28T23:51:20.804 DEBUG:boto:Token: None 2023-07-28T23:51:20.805 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj?versionId=22KG5jfO2TftvgPgeJvtwXX52mGQhV1 2023-07-28T23:51:20.805 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:irRWZfgsc1VwZPJKeSMeos6lqRs= 2023-07-28T23:51:20.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:irRWZfgsc1VwZPJKeSMeos6lqRs=', 'Content-Length': '0'} 2023-07-28T23:51:20.807 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', '22KG5jfO2TftvgPgeJvtwXX52mGQhV1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000af6348ca80f56b20-0064c45478-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.808 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.808 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.809 DEBUG:boto:path=/hiiqos-13/obj?versionId=22KG5jfO2TftvgPgeJvtwXX52mGQhV1 2023-07-28T23:51:20.809 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=22KG5jfO2TftvgPgeJvtwXX52mGQhV1 2023-07-28T23:51:20.809 DEBUG:boto:Method: HEAD 2023-07-28T23:51:20.810 DEBUG:boto:Path: /hiiqos-13/obj?versionId=22KG5jfO2TftvgPgeJvtwXX52mGQhV1 2023-07-28T23:51:20.810 DEBUG:boto:Data: 2023-07-28T23:51:20.810 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.811 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:20.811 DEBUG:boto:Port: 8000 2023-07-28T23:51:20.811 DEBUG:boto:Params: {} 2023-07-28T23:51:20.812 DEBUG:boto:Token: None 2023-07-28T23:51:20.812 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj?versionId=22KG5jfO2TftvgPgeJvtwXX52mGQhV1 2023-07-28T23:51:20.813 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:irRWZfgsc1VwZPJKeSMeos6lqRs= 2023-07-28T23:51:20.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:irRWZfgsc1VwZPJKeSMeos6lqRs=', 'Content-Length': '0'} 2023-07-28T23:51:20.815 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', '22KG5jfO2TftvgPgeJvtwXX52mGQhV1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000651fd2c904117912-0064c45478-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:20.816 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.816 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.817 DEBUG:boto:path=/hiiqos-13/obj?versionId=22KG5jfO2TftvgPgeJvtwXX52mGQhV1 2023-07-28T23:51:20.817 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=22KG5jfO2TftvgPgeJvtwXX52mGQhV1 2023-07-28T23:51:20.817 DEBUG:boto:Method: GET 2023-07-28T23:51:20.818 DEBUG:boto:Path: /hiiqos-13/obj?versionId=22KG5jfO2TftvgPgeJvtwXX52mGQhV1 2023-07-28T23:51:20.818 DEBUG:boto:Data: 2023-07-28T23:51:20.818 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.819 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:20.819 DEBUG:boto:Port: 8001 2023-07-28T23:51:20.820 DEBUG:boto:Params: {} 2023-07-28T23:51:20.820 DEBUG:boto:Token: None 2023-07-28T23:51:20.820 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj?versionId=22KG5jfO2TftvgPgeJvtwXX52mGQhV1 2023-07-28T23:51:20.821 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GNm5hhVA/ml1qEd5obzxscpZNnI= 2023-07-28T23:51:20.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GNm5hhVA/ml1qEd5obzxscpZNnI=', 'Content-Length': '0'} 2023-07-28T23:51:20.823 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', '22KG5jfO2TftvgPgeJvtwXX52mGQhV1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005bd05b99a91bad4a-0064c45478-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.824 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.824 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.824 DEBUG:boto:path=/hiiqos-13/obj?versionId=22KG5jfO2TftvgPgeJvtwXX52mGQhV1 2023-07-28T23:51:20.825 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=22KG5jfO2TftvgPgeJvtwXX52mGQhV1 2023-07-28T23:51:20.825 DEBUG:boto:Method: GET 2023-07-28T23:51:20.825 DEBUG:boto:Path: /hiiqos-13/obj?versionId=22KG5jfO2TftvgPgeJvtwXX52mGQhV1 2023-07-28T23:51:20.826 DEBUG:boto:Data: 2023-07-28T23:51:20.826 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.826 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:20.827 DEBUG:boto:Port: 8000 2023-07-28T23:51:20.827 DEBUG:boto:Params: {} 2023-07-28T23:51:20.828 DEBUG:boto:Token: None 2023-07-28T23:51:20.828 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj?versionId=22KG5jfO2TftvgPgeJvtwXX52mGQhV1 2023-07-28T23:51:20.828 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GNm5hhVA/ml1qEd5obzxscpZNnI= 2023-07-28T23:51:20.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GNm5hhVA/ml1qEd5obzxscpZNnI=', 'Content-Length': '0'} 2023-07-28T23:51:20.831 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', '22KG5jfO2TftvgPgeJvtwXX52mGQhV1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d92cdd0b3c183714-0064c45478-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.832 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:20.832 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:20.832 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:20.833 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:20.833 DEBUG:boto:Method: GET 2023-07-28T23:51:20.834 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:20.834 DEBUG:boto:Data: 2023-07-28T23:51:20.834 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.835 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:20.835 DEBUG:boto:Port: 8001 2023-07-28T23:51:20.835 DEBUG:boto:Params: {} 2023-07-28T23:51:20.836 DEBUG:boto:Token: None 2023-07-28T23:51:20.836 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/ 2023-07-28T23:51:20.837 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Nci8g8Z2joYrIfL1ye83nnXBwgA= 2023-07-28T23:51:20.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Nci8g8Z2joYrIfL1ye83nnXBwgA=', 'Content-Length': '0'} 2023-07-28T23:51:20.840 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000340ad35d154a688f-0064c45478-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.840 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:20.841 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:20.841 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:20.842 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:20.842 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:20.842 DEBUG:boto:Method: GET 2023-07-28T23:51:20.843 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:20.843 DEBUG:boto:Data: 2023-07-28T23:51:20.843 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.844 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:20.844 DEBUG:boto:Port: 8000 2023-07-28T23:51:20.845 DEBUG:boto:Params: {} 2023-07-28T23:51:20.845 DEBUG:boto:Token: None 2023-07-28T23:51:20.845 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/ 2023-07-28T23:51:20.846 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Nci8g8Z2joYrIfL1ye83nnXBwgA= 2023-07-28T23:51:20.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Nci8g8Z2joYrIfL1ye83nnXBwgA=', 'Content-Length': '0'} 2023-07-28T23:51:20.848 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008517cb5925f1a6c9-0064c45478-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.849 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:20.850 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.850 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.850 DEBUG:boto:Method: HEAD 2023-07-28T23:51:20.851 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:51:20.851 DEBUG:boto:Data: 2023-07-28T23:51:20.851 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.852 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:20.852 DEBUG:boto:Port: 8001 2023-07-28T23:51:20.852 DEBUG:boto:Params: {} 2023-07-28T23:51:20.853 DEBUG:boto:Token: None 2023-07-28T23:51:20.853 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj 2023-07-28T23:51:20.854 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zmRLUZcIvpKNWfo9V9Gfj2LeSXw= 2023-07-28T23:51:20.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zmRLUZcIvpKNWfo9V9Gfj2LeSXw=', 'Content-Length': '0'} 2023-07-28T23:51:20.857 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000955b61cd6f78d2a5-0064c45478-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.858 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.858 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.858 DEBUG:boto:Method: HEAD 2023-07-28T23:51:20.859 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:51:20.859 DEBUG:boto:Data: 2023-07-28T23:51:20.859 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.860 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:20.860 DEBUG:boto:Port: 8000 2023-07-28T23:51:20.860 DEBUG:boto:Params: {} 2023-07-28T23:51:20.861 DEBUG:boto:Token: None 2023-07-28T23:51:20.861 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj 2023-07-28T23:51:20.862 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zmRLUZcIvpKNWfo9V9Gfj2LeSXw= 2023-07-28T23:51:20.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zmRLUZcIvpKNWfo9V9Gfj2LeSXw=', 'Content-Length': '0'} 2023-07-28T23:51:20.865 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000697d1b1b830a9ce5-0064c45478-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:20.866 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.866 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.867 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:20.867 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:20.867 DEBUG:boto:Method: GET 2023-07-28T23:51:20.868 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:20.868 DEBUG:boto:Data: 2023-07-28T23:51:20.869 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.869 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:20.869 DEBUG:boto:Port: 8001 2023-07-28T23:51:20.870 DEBUG:boto:Params: {} 2023-07-28T23:51:20.870 DEBUG:boto:Token: None 2023-07-28T23:51:20.870 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:20.871 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1/QXa/Q6NZxV1Qe0Rm/dsjDBaGI= 2023-07-28T23:51:20.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1/QXa/Q6NZxV1Qe0Rm/dsjDBaGI=', 'Content-Length': '0'} 2023-07-28T23:51:20.873 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c7930234eaa31f85-0064c45478-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.916 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.917 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.917 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:20.918 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:20.918 DEBUG:boto:Method: GET 2023-07-28T23:51:20.918 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:20.919 DEBUG:boto:Data: 2023-07-28T23:51:20.919 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.919 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:20.920 DEBUG:boto:Port: 8000 2023-07-28T23:51:20.920 DEBUG:boto:Params: {} 2023-07-28T23:51:20.921 DEBUG:boto:Token: None 2023-07-28T23:51:20.921 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:20.921 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1/QXa/Q6NZxV1Qe0Rm/dsjDBaGI= 2023-07-28T23:51:20.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1/QXa/Q6NZxV1Qe0Rm/dsjDBaGI=', 'Content-Length': '0'} 2023-07-28T23:51:20.924 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004d6a5bc3cdb528a9-0064c45478-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.924 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:20.925 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:20.925 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:20.926 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:20.926 DEBUG:boto:Method: GET 2023-07-28T23:51:20.926 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:20.927 DEBUG:boto:Data: 2023-07-28T23:51:20.927 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.927 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:20.928 DEBUG:boto:Port: 8001 2023-07-28T23:51:20.929 DEBUG:boto:Params: {} 2023-07-28T23:51:20.929 DEBUG:boto:Token: None 2023-07-28T23:51:20.930 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/ 2023-07-28T23:51:20.930 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Nci8g8Z2joYrIfL1ye83nnXBwgA= 2023-07-28T23:51:20.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Nci8g8Z2joYrIfL1ye83nnXBwgA=', 'Content-Length': '0'} 2023-07-28T23:51:20.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001290791c4bbcec6e-0064c45478-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.934 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:20.935 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:20.935 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:20.936 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:20.936 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:20.937 DEBUG:boto:Method: GET 2023-07-28T23:51:20.937 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:20.937 DEBUG:boto:Data: 2023-07-28T23:51:20.938 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.938 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:20.938 DEBUG:boto:Port: 8000 2023-07-28T23:51:20.939 DEBUG:boto:Params: {} 2023-07-28T23:51:20.939 DEBUG:boto:Token: None 2023-07-28T23:51:20.939 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/ 2023-07-28T23:51:20.940 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Nci8g8Z2joYrIfL1ye83nnXBwgA= 2023-07-28T23:51:20.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Nci8g8Z2joYrIfL1ye83nnXBwgA=', 'Content-Length': '0'} 2023-07-28T23:51:20.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aee03b964b51fd92-0064c45478-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.944 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:20.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:20.945 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.945 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.945 DEBUG:boto:path=/hiiqos-13/obj?versionId=KF9pysOWS.qePO4W3-Ux03VJiRC9ph7 2023-07-28T23:51:20.946 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=KF9pysOWS.qePO4W3-Ux03VJiRC9ph7 2023-07-28T23:51:20.946 DEBUG:boto:Method: GET 2023-07-28T23:51:20.946 DEBUG:boto:Path: /hiiqos-13/obj?versionId=KF9pysOWS.qePO4W3-Ux03VJiRC9ph7 2023-07-28T23:51:20.947 DEBUG:boto:Data: 2023-07-28T23:51:20.947 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.947 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:20.948 DEBUG:boto:Port: 8001 2023-07-28T23:51:20.948 DEBUG:boto:Params: {} 2023-07-28T23:51:20.949 DEBUG:boto:Token: None 2023-07-28T23:51:20.949 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj?versionId=KF9pysOWS.qePO4W3-Ux03VJiRC9ph7 2023-07-28T23:51:20.950 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ag7Yht6/esGmhu7x8QCDYwqFXV4= 2023-07-28T23:51:20.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ag7Yht6/esGmhu7x8QCDYwqFXV4=', 'Content-Length': '0'} 2023-07-28T23:51:20.952 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'KF9pysOWS.qePO4W3-Ux03VJiRC9ph7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001ead0341e6ae1831-0064c45478-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.953 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.953 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.953 DEBUG:boto:path=/hiiqos-13/obj?versionId=KF9pysOWS.qePO4W3-Ux03VJiRC9ph7 2023-07-28T23:51:20.954 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=KF9pysOWS.qePO4W3-Ux03VJiRC9ph7 2023-07-28T23:51:20.954 DEBUG:boto:Method: GET 2023-07-28T23:51:20.954 DEBUG:boto:Path: /hiiqos-13/obj?versionId=KF9pysOWS.qePO4W3-Ux03VJiRC9ph7 2023-07-28T23:51:20.955 DEBUG:boto:Data: 2023-07-28T23:51:20.955 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.955 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:20.956 DEBUG:boto:Port: 8000 2023-07-28T23:51:20.956 DEBUG:boto:Params: {} 2023-07-28T23:51:20.957 DEBUG:boto:Token: None 2023-07-28T23:51:20.957 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj?versionId=KF9pysOWS.qePO4W3-Ux03VJiRC9ph7 2023-07-28T23:51:20.957 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ag7Yht6/esGmhu7x8QCDYwqFXV4= 2023-07-28T23:51:20.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ag7Yht6/esGmhu7x8QCDYwqFXV4=', 'Content-Length': '0'} 2023-07-28T23:51:20.960 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'KF9pysOWS.qePO4W3-Ux03VJiRC9ph7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000af93a3134d22df6c-0064c45478-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.961 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.961 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.961 DEBUG:boto:path=/hiiqos-13/obj?versionId=KF9pysOWS.qePO4W3-Ux03VJiRC9ph7 2023-07-28T23:51:20.962 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=KF9pysOWS.qePO4W3-Ux03VJiRC9ph7 2023-07-28T23:51:20.962 DEBUG:boto:Method: HEAD 2023-07-28T23:51:20.962 DEBUG:boto:Path: /hiiqos-13/obj?versionId=KF9pysOWS.qePO4W3-Ux03VJiRC9ph7 2023-07-28T23:51:20.963 DEBUG:boto:Data: 2023-07-28T23:51:20.963 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.964 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:20.964 DEBUG:boto:Port: 8001 2023-07-28T23:51:20.964 DEBUG:boto:Params: {} 2023-07-28T23:51:20.965 DEBUG:boto:Token: None 2023-07-28T23:51:20.965 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj?versionId=KF9pysOWS.qePO4W3-Ux03VJiRC9ph7 2023-07-28T23:51:20.966 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:NIUNsj5GH1YFzagTKxjkLQt5v+Y= 2023-07-28T23:51:20.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:NIUNsj5GH1YFzagTKxjkLQt5v+Y=', 'Content-Length': '0'} 2023-07-28T23:51:20.970 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'KF9pysOWS.qePO4W3-Ux03VJiRC9ph7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bd39652b915f9201-0064c45478-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.971 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.972 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.973 DEBUG:boto:path=/hiiqos-13/obj?versionId=KF9pysOWS.qePO4W3-Ux03VJiRC9ph7 2023-07-28T23:51:20.973 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=KF9pysOWS.qePO4W3-Ux03VJiRC9ph7 2023-07-28T23:51:20.974 DEBUG:boto:Method: HEAD 2023-07-28T23:51:20.974 DEBUG:boto:Path: /hiiqos-13/obj?versionId=KF9pysOWS.qePO4W3-Ux03VJiRC9ph7 2023-07-28T23:51:20.975 DEBUG:boto:Data: 2023-07-28T23:51:20.976 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.976 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:20.976 DEBUG:boto:Port: 8000 2023-07-28T23:51:20.977 DEBUG:boto:Params: {} 2023-07-28T23:51:20.977 DEBUG:boto:Token: None 2023-07-28T23:51:20.977 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj?versionId=KF9pysOWS.qePO4W3-Ux03VJiRC9ph7 2023-07-28T23:51:20.978 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:NIUNsj5GH1YFzagTKxjkLQt5v+Y= 2023-07-28T23:51:20.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:NIUNsj5GH1YFzagTKxjkLQt5v+Y=', 'Content-Length': '0'} 2023-07-28T23:51:20.980 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'KF9pysOWS.qePO4W3-Ux03VJiRC9ph7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a10543b53d2de3ac-0064c45478-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:20.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:20.981 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:20.982 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:20.982 DEBUG:boto:path=/hiiqos-13/obj?versionId=KF9pysOWS.qePO4W3-Ux03VJiRC9ph7 2023-07-28T23:51:20.983 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=KF9pysOWS.qePO4W3-Ux03VJiRC9ph7 2023-07-28T23:51:20.983 DEBUG:boto:Method: GET 2023-07-28T23:51:20.983 DEBUG:boto:Path: /hiiqos-13/obj?versionId=KF9pysOWS.qePO4W3-Ux03VJiRC9ph7 2023-07-28T23:51:20.984 DEBUG:boto:Data: 2023-07-28T23:51:20.984 DEBUG:boto:Headers: {} 2023-07-28T23:51:20.985 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:20.985 DEBUG:boto:Port: 8001 2023-07-28T23:51:20.985 DEBUG:boto:Params: {} 2023-07-28T23:51:20.986 DEBUG:boto:Token: None 2023-07-28T23:51:20.986 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:20 GMT /hiiqos-13/obj?versionId=KF9pysOWS.qePO4W3-Ux03VJiRC9ph7 2023-07-28T23:51:20.987 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ag7Yht6/esGmhu7x8QCDYwqFXV4= 2023-07-28T23:51:20.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ag7Yht6/esGmhu7x8QCDYwqFXV4=', 'Content-Length': '0'} 2023-07-28T23:51:20.989 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'KF9pysOWS.qePO4W3-Ux03VJiRC9ph7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000066c3a78bd34d891-0064c45478-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.032 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:21.033 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:21.033 DEBUG:boto:path=/hiiqos-13/obj?versionId=KF9pysOWS.qePO4W3-Ux03VJiRC9ph7 2023-07-28T23:51:21.034 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=KF9pysOWS.qePO4W3-Ux03VJiRC9ph7 2023-07-28T23:51:21.034 DEBUG:boto:Method: GET 2023-07-28T23:51:21.034 DEBUG:boto:Path: /hiiqos-13/obj?versionId=KF9pysOWS.qePO4W3-Ux03VJiRC9ph7 2023-07-28T23:51:21.035 DEBUG:boto:Data: 2023-07-28T23:51:21.035 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.035 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:21.036 DEBUG:boto:Port: 8000 2023-07-28T23:51:21.036 DEBUG:boto:Params: {} 2023-07-28T23:51:21.037 DEBUG:boto:Token: None 2023-07-28T23:51:21.037 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/obj?versionId=KF9pysOWS.qePO4W3-Ux03VJiRC9ph7 2023-07-28T23:51:21.038 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:l7i0QoiXYLQW3fHJdsNxiPJTjfE= 2023-07-28T23:51:21.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:l7i0QoiXYLQW3fHJdsNxiPJTjfE=', 'Content-Length': '0'} 2023-07-28T23:51:21.040 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'KF9pysOWS.qePO4W3-Ux03VJiRC9ph7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fbd394038fea67f4-0064c45479-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.041 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:21.041 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:21.041 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:21.042 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:21.042 DEBUG:boto:Method: GET 2023-07-28T23:51:21.042 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:21.043 DEBUG:boto:Data: 2023-07-28T23:51:21.043 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.044 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:21.044 DEBUG:boto:Port: 8001 2023-07-28T23:51:21.044 DEBUG:boto:Params: {} 2023-07-28T23:51:21.045 DEBUG:boto:Token: None 2023-07-28T23:51:21.045 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/ 2023-07-28T23:51:21.046 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Y7RST2RtNvOKk7ej6I0tPcg16Go= 2023-07-28T23:51:21.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Y7RST2RtNvOKk7ej6I0tPcg16Go=', 'Content-Length': '0'} 2023-07-28T23:51:21.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f16401ee73e9642e-0064c45479-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.092 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:21.093 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:21.094 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:21.094 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:21.094 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:21.095 DEBUG:boto:Method: GET 2023-07-28T23:51:21.095 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:21.095 DEBUG:boto:Data: 2023-07-28T23:51:21.096 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.097 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:21.097 DEBUG:boto:Port: 8000 2023-07-28T23:51:21.097 DEBUG:boto:Params: {} 2023-07-28T23:51:21.098 DEBUG:boto:Token: None 2023-07-28T23:51:21.098 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/ 2023-07-28T23:51:21.099 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Y7RST2RtNvOKk7ej6I0tPcg16Go= 2023-07-28T23:51:21.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Y7RST2RtNvOKk7ej6I0tPcg16Go=', 'Content-Length': '0'} 2023-07-28T23:51:21.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4d6632afabb78b4-0064c45479-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.102 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:21.103 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:21.103 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:21.103 DEBUG:boto:Method: HEAD 2023-07-28T23:51:21.104 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:51:21.104 DEBUG:boto:Data: 2023-07-28T23:51:21.105 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.105 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:21.105 DEBUG:boto:Port: 8001 2023-07-28T23:51:21.106 DEBUG:boto:Params: {} 2023-07-28T23:51:21.106 DEBUG:boto:Token: None 2023-07-28T23:51:21.106 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/obj 2023-07-28T23:51:21.107 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:N3vHS58SnycCA65dm4bKRRqZu2w= 2023-07-28T23:51:21.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:N3vHS58SnycCA65dm4bKRRqZu2w=', 'Content-Length': '0'} 2023-07-28T23:51:21.110 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004cb3abdc1ce296dd-0064c45479-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.111 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:21.111 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:21.112 DEBUG:boto:Method: HEAD 2023-07-28T23:51:21.112 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:51:21.112 DEBUG:boto:Data: 2023-07-28T23:51:21.113 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.113 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:21.113 DEBUG:boto:Port: 8000 2023-07-28T23:51:21.114 DEBUG:boto:Params: {} 2023-07-28T23:51:21.114 DEBUG:boto:Token: None 2023-07-28T23:51:21.114 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/obj 2023-07-28T23:51:21.115 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:N3vHS58SnycCA65dm4bKRRqZu2w= 2023-07-28T23:51:21.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:N3vHS58SnycCA65dm4bKRRqZu2w=', 'Content-Length': '0'} 2023-07-28T23:51:21.118 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000060d6f5887dc97e88-0064c45479-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:21.119 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:21.120 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:21.120 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:21.120 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:21.121 DEBUG:boto:Method: GET 2023-07-28T23:51:21.121 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:21.121 DEBUG:boto:Data: 2023-07-28T23:51:21.122 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.122 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:21.122 DEBUG:boto:Port: 8001 2023-07-28T23:51:21.123 DEBUG:boto:Params: {} 2023-07-28T23:51:21.123 DEBUG:boto:Token: None 2023-07-28T23:51:21.124 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:21.124 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FUKrIW2NPTc3Uj2RtmvPqgLO4gE= 2023-07-28T23:51:21.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FUKrIW2NPTc3Uj2RtmvPqgLO4gE=', 'Content-Length': '0'} 2023-07-28T23:51:21.127 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e1d4ffd2467e227b-0064c45479-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.168 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:21.169 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:21.169 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:21.170 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:21.170 DEBUG:boto:Method: GET 2023-07-28T23:51:21.170 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:21.171 DEBUG:boto:Data: 2023-07-28T23:51:21.171 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.171 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:21.172 DEBUG:boto:Port: 8000 2023-07-28T23:51:21.172 DEBUG:boto:Params: {} 2023-07-28T23:51:21.173 DEBUG:boto:Token: None 2023-07-28T23:51:21.173 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:21.173 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FUKrIW2NPTc3Uj2RtmvPqgLO4gE= 2023-07-28T23:51:21.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FUKrIW2NPTc3Uj2RtmvPqgLO4gE=', 'Content-Length': '0'} 2023-07-28T23:51:21.176 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000392bdbfef841d2a6-0064c45479-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.177 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:21.178 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:21.179 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:21.180 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:21.180 DEBUG:boto:Method: GET 2023-07-28T23:51:21.181 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:21.182 DEBUG:boto:Data: 2023-07-28T23:51:21.183 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.184 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:21.185 DEBUG:boto:Port: 8001 2023-07-28T23:51:21.185 DEBUG:boto:Params: {} 2023-07-28T23:51:21.186 DEBUG:boto:Token: None 2023-07-28T23:51:21.187 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/ 2023-07-28T23:51:21.188 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Y7RST2RtNvOKk7ej6I0tPcg16Go= 2023-07-28T23:51:21.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Y7RST2RtNvOKk7ej6I0tPcg16Go=', 'Content-Length': '0'} 2023-07-28T23:51:21.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000802321e0047fb9ba-0064c45479-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.232 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:21.233 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:21.233 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:21.234 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:21.234 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:21.234 DEBUG:boto:Method: GET 2023-07-28T23:51:21.235 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:21.235 DEBUG:boto:Data: 2023-07-28T23:51:21.236 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.236 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:21.236 DEBUG:boto:Port: 8000 2023-07-28T23:51:21.237 DEBUG:boto:Params: {} 2023-07-28T23:51:21.237 DEBUG:boto:Token: None 2023-07-28T23:51:21.238 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/ 2023-07-28T23:51:21.238 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Y7RST2RtNvOKk7ej6I0tPcg16Go= 2023-07-28T23:51:21.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Y7RST2RtNvOKk7ej6I0tPcg16Go=', 'Content-Length': '0'} 2023-07-28T23:51:21.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a03bfcd583bb07c3-0064c45479-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.241 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:21.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:21.242 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:21.243 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:21.243 DEBUG:boto:path=/hiiqos-13/obj?versionId=58wMXe18IrbR7RSdRvF4V8KhCCY6CR. 2023-07-28T23:51:21.243 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=58wMXe18IrbR7RSdRvF4V8KhCCY6CR. 2023-07-28T23:51:21.244 DEBUG:boto:Method: GET 2023-07-28T23:51:21.244 DEBUG:boto:Path: /hiiqos-13/obj?versionId=58wMXe18IrbR7RSdRvF4V8KhCCY6CR. 2023-07-28T23:51:21.245 DEBUG:boto:Data: 2023-07-28T23:51:21.245 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.246 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:21.247 DEBUG:boto:Port: 8001 2023-07-28T23:51:21.248 DEBUG:boto:Params: {} 2023-07-28T23:51:21.249 DEBUG:boto:Token: None 2023-07-28T23:51:21.250 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/obj?versionId=58wMXe18IrbR7RSdRvF4V8KhCCY6CR. 2023-07-28T23:51:21.251 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cl3w37YllC+4daMF0F/WhyloxLU= 2023-07-28T23:51:21.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cl3w37YllC+4daMF0F/WhyloxLU=', 'Content-Length': '0'} 2023-07-28T23:51:21.255 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', '58wMXe18IrbR7RSdRvF4V8KhCCY6CR.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008b5dec0058ecd424-0064c45479-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.256 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:21.256 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:21.257 DEBUG:boto:path=/hiiqos-13/obj?versionId=58wMXe18IrbR7RSdRvF4V8KhCCY6CR. 2023-07-28T23:51:21.257 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=58wMXe18IrbR7RSdRvF4V8KhCCY6CR. 2023-07-28T23:51:21.257 DEBUG:boto:Method: GET 2023-07-28T23:51:21.258 DEBUG:boto:Path: /hiiqos-13/obj?versionId=58wMXe18IrbR7RSdRvF4V8KhCCY6CR. 2023-07-28T23:51:21.258 DEBUG:boto:Data: 2023-07-28T23:51:21.258 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.259 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:21.259 DEBUG:boto:Port: 8000 2023-07-28T23:51:21.259 DEBUG:boto:Params: {} 2023-07-28T23:51:21.260 DEBUG:boto:Token: None 2023-07-28T23:51:21.261 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/obj?versionId=58wMXe18IrbR7RSdRvF4V8KhCCY6CR. 2023-07-28T23:51:21.261 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cl3w37YllC+4daMF0F/WhyloxLU= 2023-07-28T23:51:21.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cl3w37YllC+4daMF0F/WhyloxLU=', 'Content-Length': '0'} 2023-07-28T23:51:21.264 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', '58wMXe18IrbR7RSdRvF4V8KhCCY6CR.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bc909d68825cc850-0064c45479-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.264 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:21.265 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:21.265 DEBUG:boto:path=/hiiqos-13/obj?versionId=58wMXe18IrbR7RSdRvF4V8KhCCY6CR. 2023-07-28T23:51:21.265 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=58wMXe18IrbR7RSdRvF4V8KhCCY6CR. 2023-07-28T23:51:21.266 DEBUG:boto:Method: HEAD 2023-07-28T23:51:21.266 DEBUG:boto:Path: /hiiqos-13/obj?versionId=58wMXe18IrbR7RSdRvF4V8KhCCY6CR. 2023-07-28T23:51:21.266 DEBUG:boto:Data: 2023-07-28T23:51:21.267 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.267 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:21.267 DEBUG:boto:Port: 8001 2023-07-28T23:51:21.268 DEBUG:boto:Params: {} 2023-07-28T23:51:21.268 DEBUG:boto:Token: None 2023-07-28T23:51:21.269 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/obj?versionId=58wMXe18IrbR7RSdRvF4V8KhCCY6CR. 2023-07-28T23:51:21.269 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kderruWBXwwAPcNHtczVkJqYBSc= 2023-07-28T23:51:21.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kderruWBXwwAPcNHtczVkJqYBSc=', 'Content-Length': '0'} 2023-07-28T23:51:21.272 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', '58wMXe18IrbR7RSdRvF4V8KhCCY6CR.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005bea70af096f8476-0064c45479-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.272 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:21.273 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:21.273 DEBUG:boto:path=/hiiqos-13/obj?versionId=58wMXe18IrbR7RSdRvF4V8KhCCY6CR. 2023-07-28T23:51:21.273 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=58wMXe18IrbR7RSdRvF4V8KhCCY6CR. 2023-07-28T23:51:21.274 DEBUG:boto:Method: HEAD 2023-07-28T23:51:21.274 DEBUG:boto:Path: /hiiqos-13/obj?versionId=58wMXe18IrbR7RSdRvF4V8KhCCY6CR. 2023-07-28T23:51:21.274 DEBUG:boto:Data: 2023-07-28T23:51:21.275 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.275 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:21.275 DEBUG:boto:Port: 8000 2023-07-28T23:51:21.276 DEBUG:boto:Params: {} 2023-07-28T23:51:21.276 DEBUG:boto:Token: None 2023-07-28T23:51:21.276 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/obj?versionId=58wMXe18IrbR7RSdRvF4V8KhCCY6CR. 2023-07-28T23:51:21.277 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kderruWBXwwAPcNHtczVkJqYBSc= 2023-07-28T23:51:21.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kderruWBXwwAPcNHtczVkJqYBSc=', 'Content-Length': '0'} 2023-07-28T23:51:21.279 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', '58wMXe18IrbR7RSdRvF4V8KhCCY6CR.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e236bf8067d5952b-0064c45479-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:21.280 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:21.280 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:21.281 DEBUG:boto:path=/hiiqos-13/obj?versionId=58wMXe18IrbR7RSdRvF4V8KhCCY6CR. 2023-07-28T23:51:21.281 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=58wMXe18IrbR7RSdRvF4V8KhCCY6CR. 2023-07-28T23:51:21.281 DEBUG:boto:Method: GET 2023-07-28T23:51:21.282 DEBUG:boto:Path: /hiiqos-13/obj?versionId=58wMXe18IrbR7RSdRvF4V8KhCCY6CR. 2023-07-28T23:51:21.282 DEBUG:boto:Data: 2023-07-28T23:51:21.282 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.283 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:21.283 DEBUG:boto:Port: 8001 2023-07-28T23:51:21.283 DEBUG:boto:Params: {} 2023-07-28T23:51:21.284 DEBUG:boto:Token: None 2023-07-28T23:51:21.284 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/obj?versionId=58wMXe18IrbR7RSdRvF4V8KhCCY6CR. 2023-07-28T23:51:21.285 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cl3w37YllC+4daMF0F/WhyloxLU= 2023-07-28T23:51:21.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cl3w37YllC+4daMF0F/WhyloxLU=', 'Content-Length': '0'} 2023-07-28T23:51:21.287 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', '58wMXe18IrbR7RSdRvF4V8KhCCY6CR.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000085c144cf10162593-0064c45479-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.288 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:21.288 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:21.289 DEBUG:boto:path=/hiiqos-13/obj?versionId=58wMXe18IrbR7RSdRvF4V8KhCCY6CR. 2023-07-28T23:51:21.289 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=58wMXe18IrbR7RSdRvF4V8KhCCY6CR. 2023-07-28T23:51:21.290 DEBUG:boto:Method: GET 2023-07-28T23:51:21.290 DEBUG:boto:Path: /hiiqos-13/obj?versionId=58wMXe18IrbR7RSdRvF4V8KhCCY6CR. 2023-07-28T23:51:21.290 DEBUG:boto:Data: 2023-07-28T23:51:21.291 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.291 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:21.291 DEBUG:boto:Port: 8000 2023-07-28T23:51:21.292 DEBUG:boto:Params: {} 2023-07-28T23:51:21.292 DEBUG:boto:Token: None 2023-07-28T23:51:21.293 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/obj?versionId=58wMXe18IrbR7RSdRvF4V8KhCCY6CR. 2023-07-28T23:51:21.293 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cl3w37YllC+4daMF0F/WhyloxLU= 2023-07-28T23:51:21.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cl3w37YllC+4daMF0F/WhyloxLU=', 'Content-Length': '0'} 2023-07-28T23:51:21.295 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', '58wMXe18IrbR7RSdRvF4V8KhCCY6CR.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000831b23e9493aee5f-0064c45479-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.296 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:21.296 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:21.297 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:21.297 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:21.297 DEBUG:boto:Method: GET 2023-07-28T23:51:21.298 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:21.298 DEBUG:boto:Data: 2023-07-28T23:51:21.298 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.299 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:21.299 DEBUG:boto:Port: 8001 2023-07-28T23:51:21.299 DEBUG:boto:Params: {} 2023-07-28T23:51:21.300 DEBUG:boto:Token: None 2023-07-28T23:51:21.300 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/ 2023-07-28T23:51:21.301 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Y7RST2RtNvOKk7ej6I0tPcg16Go= 2023-07-28T23:51:21.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Y7RST2RtNvOKk7ej6I0tPcg16Go=', 'Content-Length': '0'} 2023-07-28T23:51:21.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a810230cd6a2269-0064c45479-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.304 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:21.305 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:21.306 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:21.306 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:21.307 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:21.308 DEBUG:boto:Method: GET 2023-07-28T23:51:21.308 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:21.309 DEBUG:boto:Data: 2023-07-28T23:51:21.310 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.311 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:21.312 DEBUG:boto:Port: 8000 2023-07-28T23:51:21.313 DEBUG:boto:Params: {} 2023-07-28T23:51:21.314 DEBUG:boto:Token: None 2023-07-28T23:51:21.314 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/ 2023-07-28T23:51:21.315 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Y7RST2RtNvOKk7ej6I0tPcg16Go= 2023-07-28T23:51:21.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Y7RST2RtNvOKk7ej6I0tPcg16Go=', 'Content-Length': '0'} 2023-07-28T23:51:21.317 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035004097f3212d99-0064c45479-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.318 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:21.319 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:21.319 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:21.319 DEBUG:boto:Method: HEAD 2023-07-28T23:51:21.320 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:51:21.320 DEBUG:boto:Data: 2023-07-28T23:51:21.321 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.321 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:21.321 DEBUG:boto:Port: 8001 2023-07-28T23:51:21.322 DEBUG:boto:Params: {} 2023-07-28T23:51:21.322 DEBUG:boto:Token: None 2023-07-28T23:51:21.322 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/obj 2023-07-28T23:51:21.323 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:N3vHS58SnycCA65dm4bKRRqZu2w= 2023-07-28T23:51:21.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:N3vHS58SnycCA65dm4bKRRqZu2w=', 'Content-Length': '0'} 2023-07-28T23:51:21.326 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c9b20b2ae9319ace-0064c45479-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.326 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:21.327 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:21.327 DEBUG:boto:Method: HEAD 2023-07-28T23:51:21.328 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:51:21.328 DEBUG:boto:Data: 2023-07-28T23:51:21.328 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.329 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:21.329 DEBUG:boto:Port: 8000 2023-07-28T23:51:21.329 DEBUG:boto:Params: {} 2023-07-28T23:51:21.330 DEBUG:boto:Token: None 2023-07-28T23:51:21.330 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/obj 2023-07-28T23:51:21.330 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:N3vHS58SnycCA65dm4bKRRqZu2w= 2023-07-28T23:51:21.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:N3vHS58SnycCA65dm4bKRRqZu2w=', 'Content-Length': '0'} 2023-07-28T23:51:21.334 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aac1144b2934fb04-0064c45479-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:21.335 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:21.335 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:21.335 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:21.336 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:21.336 DEBUG:boto:Method: GET 2023-07-28T23:51:21.337 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:21.337 DEBUG:boto:Data: 2023-07-28T23:51:21.337 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.338 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:21.338 DEBUG:boto:Port: 8001 2023-07-28T23:51:21.338 DEBUG:boto:Params: {} 2023-07-28T23:51:21.339 DEBUG:boto:Token: None 2023-07-28T23:51:21.339 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:21.340 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FUKrIW2NPTc3Uj2RtmvPqgLO4gE= 2023-07-28T23:51:21.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FUKrIW2NPTc3Uj2RtmvPqgLO4gE=', 'Content-Length': '0'} 2023-07-28T23:51:21.342 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001f307a12527b227d-0064c45479-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.388 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:21.389 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:21.390 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:21.391 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:21.392 DEBUG:boto:Method: GET 2023-07-28T23:51:21.393 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:21.393 DEBUG:boto:Data: 2023-07-28T23:51:21.394 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.395 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:21.396 DEBUG:boto:Port: 8000 2023-07-28T23:51:21.396 DEBUG:boto:Params: {} 2023-07-28T23:51:21.397 DEBUG:boto:Token: None 2023-07-28T23:51:21.398 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:21.399 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FUKrIW2NPTc3Uj2RtmvPqgLO4gE= 2023-07-28T23:51:21.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FUKrIW2NPTc3Uj2RtmvPqgLO4gE=', 'Content-Length': '0'} 2023-07-28T23:51:21.402 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e9ca74ba2e44adb5-0064c45479-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.404 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:21.404 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:21.405 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:21.405 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:21.405 DEBUG:boto:Method: GET 2023-07-28T23:51:21.406 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:21.406 DEBUG:boto:Data: 2023-07-28T23:51:21.406 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.407 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:21.407 DEBUG:boto:Port: 8001 2023-07-28T23:51:21.407 DEBUG:boto:Params: {} 2023-07-28T23:51:21.408 DEBUG:boto:Token: None 2023-07-28T23:51:21.408 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/ 2023-07-28T23:51:21.409 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Y7RST2RtNvOKk7ej6I0tPcg16Go= 2023-07-28T23:51:21.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Y7RST2RtNvOKk7ej6I0tPcg16Go=', 'Content-Length': '0'} 2023-07-28T23:51:21.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abd06549ce14a12b-0064c45479-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.452 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:21.453 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:21.454 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:21.454 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:21.454 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:21.455 DEBUG:boto:Method: GET 2023-07-28T23:51:21.455 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:21.455 DEBUG:boto:Data: 2023-07-28T23:51:21.456 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.456 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:21.456 DEBUG:boto:Port: 8000 2023-07-28T23:51:21.457 DEBUG:boto:Params: {} 2023-07-28T23:51:21.457 DEBUG:boto:Token: None 2023-07-28T23:51:21.458 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/ 2023-07-28T23:51:21.458 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Y7RST2RtNvOKk7ej6I0tPcg16Go= 2023-07-28T23:51:21.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Y7RST2RtNvOKk7ej6I0tPcg16Go=', 'Content-Length': '0'} 2023-07-28T23:51:21.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066b8b1f334747ad9-0064c45479-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.462 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:21.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:21.463 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:21.463 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:21.463 DEBUG:boto:path=/hiiqos-13/obj?versionId=iRoeqa0SUPCZisrafCoZycv1QQITmgh 2023-07-28T23:51:21.464 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=iRoeqa0SUPCZisrafCoZycv1QQITmgh 2023-07-28T23:51:21.464 DEBUG:boto:Method: GET 2023-07-28T23:51:21.465 DEBUG:boto:Path: /hiiqos-13/obj?versionId=iRoeqa0SUPCZisrafCoZycv1QQITmgh 2023-07-28T23:51:21.465 DEBUG:boto:Data: 2023-07-28T23:51:21.465 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.466 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:21.466 DEBUG:boto:Port: 8001 2023-07-28T23:51:21.466 DEBUG:boto:Params: {} 2023-07-28T23:51:21.467 DEBUG:boto:Token: None 2023-07-28T23:51:21.467 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/obj?versionId=iRoeqa0SUPCZisrafCoZycv1QQITmgh 2023-07-28T23:51:21.468 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Mz8LBvLba6fTt5g7gWp3dq9O3bs= 2023-07-28T23:51:21.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Mz8LBvLba6fTt5g7gWp3dq9O3bs=', 'Content-Length': '0'} 2023-07-28T23:51:21.471 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'iRoeqa0SUPCZisrafCoZycv1QQITmgh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa0ad461b1d2044a-0064c45479-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.471 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:21.472 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:21.472 DEBUG:boto:path=/hiiqos-13/obj?versionId=iRoeqa0SUPCZisrafCoZycv1QQITmgh 2023-07-28T23:51:21.473 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=iRoeqa0SUPCZisrafCoZycv1QQITmgh 2023-07-28T23:51:21.473 DEBUG:boto:Method: GET 2023-07-28T23:51:21.473 DEBUG:boto:Path: /hiiqos-13/obj?versionId=iRoeqa0SUPCZisrafCoZycv1QQITmgh 2023-07-28T23:51:21.474 DEBUG:boto:Data: 2023-07-28T23:51:21.474 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.474 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:21.475 DEBUG:boto:Port: 8000 2023-07-28T23:51:21.475 DEBUG:boto:Params: {} 2023-07-28T23:51:21.475 DEBUG:boto:Token: None 2023-07-28T23:51:21.476 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/obj?versionId=iRoeqa0SUPCZisrafCoZycv1QQITmgh 2023-07-28T23:51:21.476 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Mz8LBvLba6fTt5g7gWp3dq9O3bs= 2023-07-28T23:51:21.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Mz8LBvLba6fTt5g7gWp3dq9O3bs=', 'Content-Length': '0'} 2023-07-28T23:51:21.479 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'iRoeqa0SUPCZisrafCoZycv1QQITmgh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000974ef7c0a59dc789-0064c45479-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.520 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:21.521 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:21.521 DEBUG:boto:path=/hiiqos-13/obj?versionId=iRoeqa0SUPCZisrafCoZycv1QQITmgh 2023-07-28T23:51:21.522 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=iRoeqa0SUPCZisrafCoZycv1QQITmgh 2023-07-28T23:51:21.523 DEBUG:boto:Method: HEAD 2023-07-28T23:51:21.524 DEBUG:boto:Path: /hiiqos-13/obj?versionId=iRoeqa0SUPCZisrafCoZycv1QQITmgh 2023-07-28T23:51:21.524 DEBUG:boto:Data: 2023-07-28T23:51:21.525 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.525 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:21.525 DEBUG:boto:Port: 8001 2023-07-28T23:51:21.526 DEBUG:boto:Params: {} 2023-07-28T23:51:21.526 DEBUG:boto:Token: None 2023-07-28T23:51:21.526 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/obj?versionId=iRoeqa0SUPCZisrafCoZycv1QQITmgh 2023-07-28T23:51:21.527 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w7u8rYhT6snBXqx1c/k7jJU6tJ4= 2023-07-28T23:51:21.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w7u8rYhT6snBXqx1c/k7jJU6tJ4=', 'Content-Length': '0'} 2023-07-28T23:51:21.529 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'iRoeqa0SUPCZisrafCoZycv1QQITmgh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007290b87fac40511c-0064c45479-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.530 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:21.530 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:21.531 DEBUG:boto:path=/hiiqos-13/obj?versionId=iRoeqa0SUPCZisrafCoZycv1QQITmgh 2023-07-28T23:51:21.531 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=iRoeqa0SUPCZisrafCoZycv1QQITmgh 2023-07-28T23:51:21.531 DEBUG:boto:Method: HEAD 2023-07-28T23:51:21.532 DEBUG:boto:Path: /hiiqos-13/obj?versionId=iRoeqa0SUPCZisrafCoZycv1QQITmgh 2023-07-28T23:51:21.532 DEBUG:boto:Data: 2023-07-28T23:51:21.532 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.533 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:21.533 DEBUG:boto:Port: 8000 2023-07-28T23:51:21.533 DEBUG:boto:Params: {} 2023-07-28T23:51:21.534 DEBUG:boto:Token: None 2023-07-28T23:51:21.534 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/obj?versionId=iRoeqa0SUPCZisrafCoZycv1QQITmgh 2023-07-28T23:51:21.535 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w7u8rYhT6snBXqx1c/k7jJU6tJ4= 2023-07-28T23:51:21.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w7u8rYhT6snBXqx1c/k7jJU6tJ4=', 'Content-Length': '0'} 2023-07-28T23:51:21.537 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'iRoeqa0SUPCZisrafCoZycv1QQITmgh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000001287c7f0d693c8c-0064c45479-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:21.538 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:21.538 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:21.539 DEBUG:boto:path=/hiiqos-13/obj?versionId=iRoeqa0SUPCZisrafCoZycv1QQITmgh 2023-07-28T23:51:21.539 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=iRoeqa0SUPCZisrafCoZycv1QQITmgh 2023-07-28T23:51:21.539 DEBUG:boto:Method: GET 2023-07-28T23:51:21.540 DEBUG:boto:Path: /hiiqos-13/obj?versionId=iRoeqa0SUPCZisrafCoZycv1QQITmgh 2023-07-28T23:51:21.540 DEBUG:boto:Data: 2023-07-28T23:51:21.541 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.541 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:21.541 DEBUG:boto:Port: 8001 2023-07-28T23:51:21.542 DEBUG:boto:Params: {} 2023-07-28T23:51:21.542 DEBUG:boto:Token: None 2023-07-28T23:51:21.543 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/obj?versionId=iRoeqa0SUPCZisrafCoZycv1QQITmgh 2023-07-28T23:51:21.543 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Mz8LBvLba6fTt5g7gWp3dq9O3bs= 2023-07-28T23:51:21.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Mz8LBvLba6fTt5g7gWp3dq9O3bs=', 'Content-Length': '0'} 2023-07-28T23:51:21.546 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'iRoeqa0SUPCZisrafCoZycv1QQITmgh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000055c8c5e4894e8023-0064c45479-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.588 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:21.589 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:21.589 DEBUG:boto:path=/hiiqos-13/obj?versionId=iRoeqa0SUPCZisrafCoZycv1QQITmgh 2023-07-28T23:51:21.590 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=iRoeqa0SUPCZisrafCoZycv1QQITmgh 2023-07-28T23:51:21.590 DEBUG:boto:Method: GET 2023-07-28T23:51:21.590 DEBUG:boto:Path: /hiiqos-13/obj?versionId=iRoeqa0SUPCZisrafCoZycv1QQITmgh 2023-07-28T23:51:21.591 DEBUG:boto:Data: 2023-07-28T23:51:21.591 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.591 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:21.592 DEBUG:boto:Port: 8000 2023-07-28T23:51:21.592 DEBUG:boto:Params: {} 2023-07-28T23:51:21.593 DEBUG:boto:Token: None 2023-07-28T23:51:21.593 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/obj?versionId=iRoeqa0SUPCZisrafCoZycv1QQITmgh 2023-07-28T23:51:21.594 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Mz8LBvLba6fTt5g7gWp3dq9O3bs= 2023-07-28T23:51:21.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Mz8LBvLba6fTt5g7gWp3dq9O3bs=', 'Content-Length': '0'} 2023-07-28T23:51:21.596 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'iRoeqa0SUPCZisrafCoZycv1QQITmgh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003fa5e30a4922c51e-0064c45479-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.597 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:21.597 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:21.597 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:21.598 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:21.598 DEBUG:boto:Method: GET 2023-07-28T23:51:21.599 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:21.599 DEBUG:boto:Data: 2023-07-28T23:51:21.599 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.600 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:21.600 DEBUG:boto:Port: 8001 2023-07-28T23:51:21.600 DEBUG:boto:Params: {} 2023-07-28T23:51:21.601 DEBUG:boto:Token: None 2023-07-28T23:51:21.601 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/ 2023-07-28T23:51:21.602 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Y7RST2RtNvOKk7ej6I0tPcg16Go= 2023-07-28T23:51:21.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Y7RST2RtNvOKk7ej6I0tPcg16Go=', 'Content-Length': '0'} 2023-07-28T23:51:21.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000968ce1789c1c3c4c-0064c45479-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.648 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:21.649 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:21.650 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:21.650 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:21.650 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:21.651 DEBUG:boto:Method: GET 2023-07-28T23:51:21.651 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:21.651 DEBUG:boto:Data: 2023-07-28T23:51:21.652 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.652 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:21.653 DEBUG:boto:Port: 8000 2023-07-28T23:51:21.653 DEBUG:boto:Params: {} 2023-07-28T23:51:21.654 DEBUG:boto:Token: None 2023-07-28T23:51:21.654 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/ 2023-07-28T23:51:21.654 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Y7RST2RtNvOKk7ej6I0tPcg16Go= 2023-07-28T23:51:21.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Y7RST2RtNvOKk7ej6I0tPcg16Go=', 'Content-Length': '0'} 2023-07-28T23:51:21.657 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004775bcfd67bfa9ac-0064c45479-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.658 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:21.659 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:21.659 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:21.659 DEBUG:boto:Method: HEAD 2023-07-28T23:51:21.660 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:51:21.660 DEBUG:boto:Data: 2023-07-28T23:51:21.661 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.661 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:21.661 DEBUG:boto:Port: 8001 2023-07-28T23:51:21.661 DEBUG:boto:Params: {} 2023-07-28T23:51:21.662 DEBUG:boto:Token: None 2023-07-28T23:51:21.662 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/obj 2023-07-28T23:51:21.663 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:N3vHS58SnycCA65dm4bKRRqZu2w= 2023-07-28T23:51:21.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:N3vHS58SnycCA65dm4bKRRqZu2w=', 'Content-Length': '0'} 2023-07-28T23:51:21.667 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006ef57af7b0e3ec19-0064c45479-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.668 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:21.669 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:21.669 DEBUG:boto:Method: HEAD 2023-07-28T23:51:21.670 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:51:21.671 DEBUG:boto:Data: 2023-07-28T23:51:21.671 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.672 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:21.673 DEBUG:boto:Port: 8000 2023-07-28T23:51:21.673 DEBUG:boto:Params: {} 2023-07-28T23:51:21.674 DEBUG:boto:Token: None 2023-07-28T23:51:21.675 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/obj 2023-07-28T23:51:21.676 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:N3vHS58SnycCA65dm4bKRRqZu2w= 2023-07-28T23:51:21.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:N3vHS58SnycCA65dm4bKRRqZu2w=', 'Content-Length': '0'} 2023-07-28T23:51:21.680 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000096d8050a25083ace-0064c45479-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:21.681 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:21.682 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:21.683 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:21.683 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:21.684 DEBUG:boto:Method: GET 2023-07-28T23:51:21.685 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:21.685 DEBUG:boto:Data: 2023-07-28T23:51:21.686 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.687 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:21.688 DEBUG:boto:Port: 8001 2023-07-28T23:51:21.689 DEBUG:boto:Params: {} 2023-07-28T23:51:21.690 DEBUG:boto:Token: None 2023-07-28T23:51:21.690 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:21.691 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FUKrIW2NPTc3Uj2RtmvPqgLO4gE= 2023-07-28T23:51:21.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FUKrIW2NPTc3Uj2RtmvPqgLO4gE=', 'Content-Length': '0'} 2023-07-28T23:51:21.693 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006014baed61a6fa4d-0064c45479-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.736 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:21.737 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:21.737 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:21.738 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:21.738 DEBUG:boto:Method: GET 2023-07-28T23:51:21.738 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:21.739 DEBUG:boto:Data: 2023-07-28T23:51:21.739 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.739 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:21.740 DEBUG:boto:Port: 8000 2023-07-28T23:51:21.740 DEBUG:boto:Params: {} 2023-07-28T23:51:21.741 DEBUG:boto:Token: None 2023-07-28T23:51:21.741 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:21.742 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FUKrIW2NPTc3Uj2RtmvPqgLO4gE= 2023-07-28T23:51:21.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FUKrIW2NPTc3Uj2RtmvPqgLO4gE=', 'Content-Length': '0'} 2023-07-28T23:51:21.744 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007bb9f8f4ee291cf9-0064c45479-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.745 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:21.745 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:21.745 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:21.746 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:21.746 DEBUG:boto:Method: GET 2023-07-28T23:51:21.747 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:21.747 DEBUG:boto:Data: 2023-07-28T23:51:21.747 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.748 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:21.748 DEBUG:boto:Port: 8001 2023-07-28T23:51:21.748 DEBUG:boto:Params: {} 2023-07-28T23:51:21.749 DEBUG:boto:Token: None 2023-07-28T23:51:21.749 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/ 2023-07-28T23:51:21.750 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Y7RST2RtNvOKk7ej6I0tPcg16Go= 2023-07-28T23:51:21.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Y7RST2RtNvOKk7ej6I0tPcg16Go=', 'Content-Length': '0'} 2023-07-28T23:51:21.753 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071233f1c3d7d9008-0064c45479-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.754 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:21.755 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:21.756 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:21.757 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:21.757 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:21.757 DEBUG:boto:Method: GET 2023-07-28T23:51:21.758 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:21.758 DEBUG:boto:Data: 2023-07-28T23:51:21.758 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.759 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:21.759 DEBUG:boto:Port: 8000 2023-07-28T23:51:21.759 DEBUG:boto:Params: {} 2023-07-28T23:51:21.760 DEBUG:boto:Token: None 2023-07-28T23:51:21.760 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/ 2023-07-28T23:51:21.761 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Y7RST2RtNvOKk7ej6I0tPcg16Go= 2023-07-28T23:51:21.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Y7RST2RtNvOKk7ej6I0tPcg16Go=', 'Content-Length': '0'} 2023-07-28T23:51:21.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b6efe829a54b3c9-0064c45479-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.804 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:21.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:21.805 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:21.806 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:21.806 DEBUG:boto:path=/hiiqos-13/obj?versionId=OYqlT9m6zvLoRrfJK6xkLWsj3aXU2MG 2023-07-28T23:51:21.806 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=OYqlT9m6zvLoRrfJK6xkLWsj3aXU2MG 2023-07-28T23:51:21.807 DEBUG:boto:Method: GET 2023-07-28T23:51:21.807 DEBUG:boto:Path: /hiiqos-13/obj?versionId=OYqlT9m6zvLoRrfJK6xkLWsj3aXU2MG 2023-07-28T23:51:21.808 DEBUG:boto:Data: 2023-07-28T23:51:21.808 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.808 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:21.809 DEBUG:boto:Port: 8001 2023-07-28T23:51:21.809 DEBUG:boto:Params: {} 2023-07-28T23:51:21.809 DEBUG:boto:Token: None 2023-07-28T23:51:21.810 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/obj?versionId=OYqlT9m6zvLoRrfJK6xkLWsj3aXU2MG 2023-07-28T23:51:21.810 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+Bl1Zc4uIjiy7M5aesnrs7lWXc4= 2023-07-28T23:51:21.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+Bl1Zc4uIjiy7M5aesnrs7lWXc4=', 'Content-Length': '0'} 2023-07-28T23:51:21.813 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'OYqlT9m6zvLoRrfJK6xkLWsj3aXU2MG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000dd6af4a5bc21ae1-0064c45479-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.856 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:21.857 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:21.858 DEBUG:boto:path=/hiiqos-13/obj?versionId=OYqlT9m6zvLoRrfJK6xkLWsj3aXU2MG 2023-07-28T23:51:21.859 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=OYqlT9m6zvLoRrfJK6xkLWsj3aXU2MG 2023-07-28T23:51:21.859 DEBUG:boto:Method: GET 2023-07-28T23:51:21.860 DEBUG:boto:Path: /hiiqos-13/obj?versionId=OYqlT9m6zvLoRrfJK6xkLWsj3aXU2MG 2023-07-28T23:51:21.860 DEBUG:boto:Data: 2023-07-28T23:51:21.861 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.862 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:21.862 DEBUG:boto:Port: 8000 2023-07-28T23:51:21.863 DEBUG:boto:Params: {} 2023-07-28T23:51:21.864 DEBUG:boto:Token: None 2023-07-28T23:51:21.865 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/obj?versionId=OYqlT9m6zvLoRrfJK6xkLWsj3aXU2MG 2023-07-28T23:51:21.865 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+Bl1Zc4uIjiy7M5aesnrs7lWXc4= 2023-07-28T23:51:21.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+Bl1Zc4uIjiy7M5aesnrs7lWXc4=', 'Content-Length': '0'} 2023-07-28T23:51:21.869 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'OYqlT9m6zvLoRrfJK6xkLWsj3aXU2MG'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000076502bd2b99b6598-0064c45479-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.871 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:21.871 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:21.872 DEBUG:boto:path=/hiiqos-13/obj?versionId=OYqlT9m6zvLoRrfJK6xkLWsj3aXU2MG 2023-07-28T23:51:21.873 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=OYqlT9m6zvLoRrfJK6xkLWsj3aXU2MG 2023-07-28T23:51:21.873 DEBUG:boto:Method: HEAD 2023-07-28T23:51:21.873 DEBUG:boto:Path: /hiiqos-13/obj?versionId=OYqlT9m6zvLoRrfJK6xkLWsj3aXU2MG 2023-07-28T23:51:21.874 DEBUG:boto:Data: 2023-07-28T23:51:21.874 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.875 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:21.875 DEBUG:boto:Port: 8001 2023-07-28T23:51:21.875 DEBUG:boto:Params: {} 2023-07-28T23:51:21.876 DEBUG:boto:Token: None 2023-07-28T23:51:21.876 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/obj?versionId=OYqlT9m6zvLoRrfJK6xkLWsj3aXU2MG 2023-07-28T23:51:21.876 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VcvlLUwtvpuMvVbThXSmMQPI8Ms= 2023-07-28T23:51:21.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VcvlLUwtvpuMvVbThXSmMQPI8Ms=', 'Content-Length': '0'} 2023-07-28T23:51:21.879 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'OYqlT9m6zvLoRrfJK6xkLWsj3aXU2MG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006c14e8c471298d68-0064c45479-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.879 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:21.880 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:21.880 DEBUG:boto:path=/hiiqos-13/obj?versionId=OYqlT9m6zvLoRrfJK6xkLWsj3aXU2MG 2023-07-28T23:51:21.880 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=OYqlT9m6zvLoRrfJK6xkLWsj3aXU2MG 2023-07-28T23:51:21.881 DEBUG:boto:Method: HEAD 2023-07-28T23:51:21.881 DEBUG:boto:Path: /hiiqos-13/obj?versionId=OYqlT9m6zvLoRrfJK6xkLWsj3aXU2MG 2023-07-28T23:51:21.881 DEBUG:boto:Data: 2023-07-28T23:51:21.882 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.882 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:21.882 DEBUG:boto:Port: 8000 2023-07-28T23:51:21.883 DEBUG:boto:Params: {} 2023-07-28T23:51:21.883 DEBUG:boto:Token: None 2023-07-28T23:51:21.883 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/obj?versionId=OYqlT9m6zvLoRrfJK6xkLWsj3aXU2MG 2023-07-28T23:51:21.884 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VcvlLUwtvpuMvVbThXSmMQPI8Ms= 2023-07-28T23:51:21.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VcvlLUwtvpuMvVbThXSmMQPI8Ms=', 'Content-Length': '0'} 2023-07-28T23:51:21.886 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'OYqlT9m6zvLoRrfJK6xkLWsj3aXU2MG'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bdce3c2e666c00d2-0064c45479-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:21.887 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:21.887 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:21.888 DEBUG:boto:path=/hiiqos-13/obj?versionId=OYqlT9m6zvLoRrfJK6xkLWsj3aXU2MG 2023-07-28T23:51:21.888 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=OYqlT9m6zvLoRrfJK6xkLWsj3aXU2MG 2023-07-28T23:51:21.888 DEBUG:boto:Method: GET 2023-07-28T23:51:21.889 DEBUG:boto:Path: /hiiqos-13/obj?versionId=OYqlT9m6zvLoRrfJK6xkLWsj3aXU2MG 2023-07-28T23:51:21.889 DEBUG:boto:Data: 2023-07-28T23:51:21.890 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.890 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:21.890 DEBUG:boto:Port: 8001 2023-07-28T23:51:21.891 DEBUG:boto:Params: {} 2023-07-28T23:51:21.891 DEBUG:boto:Token: None 2023-07-28T23:51:21.892 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/obj?versionId=OYqlT9m6zvLoRrfJK6xkLWsj3aXU2MG 2023-07-28T23:51:21.892 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+Bl1Zc4uIjiy7M5aesnrs7lWXc4= 2023-07-28T23:51:21.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+Bl1Zc4uIjiy7M5aesnrs7lWXc4=', 'Content-Length': '0'} 2023-07-28T23:51:21.895 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'OYqlT9m6zvLoRrfJK6xkLWsj3aXU2MG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d8cc037310c0471b-0064c45479-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.895 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:21.896 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:21.896 DEBUG:boto:path=/hiiqos-13/obj?versionId=OYqlT9m6zvLoRrfJK6xkLWsj3aXU2MG 2023-07-28T23:51:21.896 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=OYqlT9m6zvLoRrfJK6xkLWsj3aXU2MG 2023-07-28T23:51:21.897 DEBUG:boto:Method: GET 2023-07-28T23:51:21.897 DEBUG:boto:Path: /hiiqos-13/obj?versionId=OYqlT9m6zvLoRrfJK6xkLWsj3aXU2MG 2023-07-28T23:51:21.897 DEBUG:boto:Data: 2023-07-28T23:51:21.898 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.898 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:21.898 DEBUG:boto:Port: 8000 2023-07-28T23:51:21.899 DEBUG:boto:Params: {} 2023-07-28T23:51:21.899 DEBUG:boto:Token: None 2023-07-28T23:51:21.900 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/obj?versionId=OYqlT9m6zvLoRrfJK6xkLWsj3aXU2MG 2023-07-28T23:51:21.900 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+Bl1Zc4uIjiy7M5aesnrs7lWXc4= 2023-07-28T23:51:21.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+Bl1Zc4uIjiy7M5aesnrs7lWXc4=', 'Content-Length': '0'} 2023-07-28T23:51:21.904 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'OYqlT9m6zvLoRrfJK6xkLWsj3aXU2MG'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a0c52afcde005495-0064c45479-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.944 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:21.945 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:21.945 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:21.946 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:21.946 DEBUG:boto:Method: GET 2023-07-28T23:51:21.946 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:21.947 DEBUG:boto:Data: 2023-07-28T23:51:21.947 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.948 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:21.948 DEBUG:boto:Port: 8001 2023-07-28T23:51:21.948 DEBUG:boto:Params: {} 2023-07-28T23:51:21.949 DEBUG:boto:Token: None 2023-07-28T23:51:21.949 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/ 2023-07-28T23:51:21.950 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Y7RST2RtNvOKk7ej6I0tPcg16Go= 2023-07-28T23:51:21.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Y7RST2RtNvOKk7ej6I0tPcg16Go=', 'Content-Length': '0'} 2023-07-28T23:51:21.955 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000780c6aa40e2a82c9-0064c45479-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:21.956 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:21.957 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:21.958 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:21.959 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:21.959 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:21.960 DEBUG:boto:Method: GET 2023-07-28T23:51:21.961 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:21.961 DEBUG:boto:Data: 2023-07-28T23:51:21.961 DEBUG:boto:Headers: {} 2023-07-28T23:51:21.962 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:21.962 DEBUG:boto:Port: 8000 2023-07-28T23:51:21.962 DEBUG:boto:Params: {} 2023-07-28T23:51:21.963 DEBUG:boto:Token: None 2023-07-28T23:51:21.963 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:21 GMT /hiiqos-13/ 2023-07-28T23:51:21.964 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Y7RST2RtNvOKk7ej6I0tPcg16Go= 2023-07-28T23:51:21.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Y7RST2RtNvOKk7ej6I0tPcg16Go=', 'Content-Length': '0'} 2023-07-28T23:51:21.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050cd78247fe0c187-0064c45479-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:22.008 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:22.009 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:22.009 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:22.010 DEBUG:boto:Method: HEAD 2023-07-28T23:51:22.010 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:51:22.010 DEBUG:boto:Data: 2023-07-28T23:51:22.011 DEBUG:boto:Headers: {} 2023-07-28T23:51:22.011 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:22.011 DEBUG:boto:Port: 8001 2023-07-28T23:51:22.012 DEBUG:boto:Params: {} 2023-07-28T23:51:22.012 DEBUG:boto:Token: None 2023-07-28T23:51:22.012 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:22 GMT /hiiqos-13/obj 2023-07-28T23:51:22.013 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:v0HaV0vO+lOuIs94TDL4lLQ1Lro= 2023-07-28T23:51:22.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:v0HaV0vO+lOuIs94TDL4lLQ1Lro=', 'Content-Length': '0'} 2023-07-28T23:51:22.017 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001ae3269b026ad708-0064c4547a-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:22.017 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:22.018 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:22.018 DEBUG:boto:Method: HEAD 2023-07-28T23:51:22.018 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:51:22.019 DEBUG:boto:Data: 2023-07-28T23:51:22.019 DEBUG:boto:Headers: {} 2023-07-28T23:51:22.020 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:22.020 DEBUG:boto:Port: 8000 2023-07-28T23:51:22.020 DEBUG:boto:Params: {} 2023-07-28T23:51:22.021 DEBUG:boto:Token: None 2023-07-28T23:51:22.021 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:22 GMT /hiiqos-13/obj 2023-07-28T23:51:22.021 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:v0HaV0vO+lOuIs94TDL4lLQ1Lro= 2023-07-28T23:51:22.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:v0HaV0vO+lOuIs94TDL4lLQ1Lro=', 'Content-Length': '0'} 2023-07-28T23:51:22.025 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006b1b679ffbf48a5b-0064c4547a-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:22.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:22.026 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:22.026 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:22.027 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:22.027 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:22.027 DEBUG:boto:Method: GET 2023-07-28T23:51:22.028 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:22.028 DEBUG:boto:Data: 2023-07-28T23:51:22.029 DEBUG:boto:Headers: {} 2023-07-28T23:51:22.029 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:22.029 DEBUG:boto:Port: 8001 2023-07-28T23:51:22.030 DEBUG:boto:Params: {} 2023-07-28T23:51:22.030 DEBUG:boto:Token: None 2023-07-28T23:51:22.030 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:22 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:22.031 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hzUz6a4UhLVlfwHL8N7Jw9AEjug= 2023-07-28T23:51:22.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hzUz6a4UhLVlfwHL8N7Jw9AEjug=', 'Content-Length': '0'} 2023-07-28T23:51:22.033 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003806eea83e21f2f9-0064c4547a-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:22.080 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:22.081 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:22.081 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:22.082 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:22.082 DEBUG:boto:Method: GET 2023-07-28T23:51:22.082 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:22.083 DEBUG:boto:Data: 2023-07-28T23:51:22.083 DEBUG:boto:Headers: {} 2023-07-28T23:51:22.083 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:22.084 DEBUG:boto:Port: 8000 2023-07-28T23:51:22.084 DEBUG:boto:Params: {} 2023-07-28T23:51:22.085 DEBUG:boto:Token: None 2023-07-28T23:51:22.085 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:22 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:22.085 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hzUz6a4UhLVlfwHL8N7Jw9AEjug= 2023-07-28T23:51:22.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hzUz6a4UhLVlfwHL8N7Jw9AEjug=', 'Content-Length': '0'} 2023-07-28T23:51:22.088 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000092fa6f1b2341511b-0064c4547a-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:22.089 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:22.089 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:22.089 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:22.090 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:22.090 DEBUG:boto:Method: GET 2023-07-28T23:51:22.090 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:22.091 DEBUG:boto:Data: 2023-07-28T23:51:22.091 DEBUG:boto:Headers: {} 2023-07-28T23:51:22.092 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:22.092 DEBUG:boto:Port: 8001 2023-07-28T23:51:22.092 DEBUG:boto:Params: {} 2023-07-28T23:51:22.093 DEBUG:boto:Token: None 2023-07-28T23:51:22.093 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:22 GMT /hiiqos-13/ 2023-07-28T23:51:22.094 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ckMBapogZxM4u620dQwUMBD89xw= 2023-07-28T23:51:22.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ckMBapogZxM4u620dQwUMBD89xw=', 'Content-Length': '0'} 2023-07-28T23:51:22.097 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fee856c0939dbbd-0064c4547a-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:22.140 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:22.141 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:22.141 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:22.142 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:22.142 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:22.143 DEBUG:boto:Method: GET 2023-07-28T23:51:22.143 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:22.143 DEBUG:boto:Data: 2023-07-28T23:51:22.144 DEBUG:boto:Headers: {} 2023-07-28T23:51:22.144 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:22.144 DEBUG:boto:Port: 8000 2023-07-28T23:51:22.145 DEBUG:boto:Params: {} 2023-07-28T23:51:22.145 DEBUG:boto:Token: None 2023-07-28T23:51:22.146 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:22 GMT /hiiqos-13/ 2023-07-28T23:51:22.146 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ckMBapogZxM4u620dQwUMBD89xw= 2023-07-28T23:51:22.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ckMBapogZxM4u620dQwUMBD89xw=', 'Content-Length': '0'} 2023-07-28T23:51:22.149 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009821f5f75ab127a2-0064c4547a-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:22.149 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:22.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:22.150 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:22.151 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:22.151 DEBUG:boto:path=/hiiqos-13/obj?versionId=nQT3SHUTnGGHoRE.HK4699V4TWQJbCA 2023-07-28T23:51:22.151 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=nQT3SHUTnGGHoRE.HK4699V4TWQJbCA 2023-07-28T23:51:22.152 DEBUG:boto:Method: GET 2023-07-28T23:51:22.152 DEBUG:boto:Path: /hiiqos-13/obj?versionId=nQT3SHUTnGGHoRE.HK4699V4TWQJbCA 2023-07-28T23:51:22.153 DEBUG:boto:Data: 2023-07-28T23:51:22.153 DEBUG:boto:Headers: {} 2023-07-28T23:51:22.153 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:22.154 DEBUG:boto:Port: 8001 2023-07-28T23:51:22.154 DEBUG:boto:Params: {} 2023-07-28T23:51:22.154 DEBUG:boto:Token: None 2023-07-28T23:51:22.155 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:22 GMT /hiiqos-13/obj?versionId=nQT3SHUTnGGHoRE.HK4699V4TWQJbCA 2023-07-28T23:51:22.155 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ERP4hjK5wyWKm+x+j3iJqGuv6R8= 2023-07-28T23:51:22.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ERP4hjK5wyWKm+x+j3iJqGuv6R8=', 'Content-Length': '0'} 2023-07-28T23:51:22.158 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'nQT3SHUTnGGHoRE.HK4699V4TWQJbCA'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d4a31ceaeef60787-0064c4547a-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:22.158 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:22.159 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:22.159 DEBUG:boto:path=/hiiqos-13/obj?versionId=nQT3SHUTnGGHoRE.HK4699V4TWQJbCA 2023-07-28T23:51:22.159 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=nQT3SHUTnGGHoRE.HK4699V4TWQJbCA 2023-07-28T23:51:22.160 DEBUG:boto:Method: GET 2023-07-28T23:51:22.160 DEBUG:boto:Path: /hiiqos-13/obj?versionId=nQT3SHUTnGGHoRE.HK4699V4TWQJbCA 2023-07-28T23:51:22.160 DEBUG:boto:Data: 2023-07-28T23:51:22.161 DEBUG:boto:Headers: {} 2023-07-28T23:51:22.161 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:22.161 DEBUG:boto:Port: 8000 2023-07-28T23:51:22.162 DEBUG:boto:Params: {} 2023-07-28T23:51:22.162 DEBUG:boto:Token: None 2023-07-28T23:51:22.163 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:22 GMT /hiiqos-13/obj?versionId=nQT3SHUTnGGHoRE.HK4699V4TWQJbCA 2023-07-28T23:51:22.163 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ERP4hjK5wyWKm+x+j3iJqGuv6R8= 2023-07-28T23:51:22.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ERP4hjK5wyWKm+x+j3iJqGuv6R8=', 'Content-Length': '0'} 2023-07-28T23:51:22.166 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'nQT3SHUTnGGHoRE.HK4699V4TWQJbCA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ecdc42972dc249bd-0064c4547a-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:22.166 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:22.167 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:22.167 DEBUG:boto:path=/hiiqos-13/obj?versionId=nQT3SHUTnGGHoRE.HK4699V4TWQJbCA 2023-07-28T23:51:22.167 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=nQT3SHUTnGGHoRE.HK4699V4TWQJbCA 2023-07-28T23:51:22.168 DEBUG:boto:Method: HEAD 2023-07-28T23:51:22.168 DEBUG:boto:Path: /hiiqos-13/obj?versionId=nQT3SHUTnGGHoRE.HK4699V4TWQJbCA 2023-07-28T23:51:22.168 DEBUG:boto:Data: 2023-07-28T23:51:22.169 DEBUG:boto:Headers: {} 2023-07-28T23:51:22.169 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:22.169 DEBUG:boto:Port: 8001 2023-07-28T23:51:22.170 DEBUG:boto:Params: {} 2023-07-28T23:51:22.170 DEBUG:boto:Token: None 2023-07-28T23:51:22.170 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:22 GMT /hiiqos-13/obj?versionId=nQT3SHUTnGGHoRE.HK4699V4TWQJbCA 2023-07-28T23:51:22.171 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nwstjMk0DSEEGfevKkUsmo3e/+Y= 2023-07-28T23:51:22.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nwstjMk0DSEEGfevKkUsmo3e/+Y=', 'Content-Length': '0'} 2023-07-28T23:51:22.173 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'nQT3SHUTnGGHoRE.HK4699V4TWQJbCA'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004348a42e9f048c8d-0064c4547a-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:22.174 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:22.174 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:22.174 DEBUG:boto:path=/hiiqos-13/obj?versionId=nQT3SHUTnGGHoRE.HK4699V4TWQJbCA 2023-07-28T23:51:22.175 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=nQT3SHUTnGGHoRE.HK4699V4TWQJbCA 2023-07-28T23:51:22.175 DEBUG:boto:Method: HEAD 2023-07-28T23:51:22.175 DEBUG:boto:Path: /hiiqos-13/obj?versionId=nQT3SHUTnGGHoRE.HK4699V4TWQJbCA 2023-07-28T23:51:22.176 DEBUG:boto:Data: 2023-07-28T23:51:22.176 DEBUG:boto:Headers: {} 2023-07-28T23:51:22.176 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:22.177 DEBUG:boto:Port: 8000 2023-07-28T23:51:22.177 DEBUG:boto:Params: {} 2023-07-28T23:51:22.177 DEBUG:boto:Token: None 2023-07-28T23:51:22.178 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:22 GMT /hiiqos-13/obj?versionId=nQT3SHUTnGGHoRE.HK4699V4TWQJbCA 2023-07-28T23:51:22.178 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nwstjMk0DSEEGfevKkUsmo3e/+Y= 2023-07-28T23:51:22.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nwstjMk0DSEEGfevKkUsmo3e/+Y=', 'Content-Length': '0'} 2023-07-28T23:51:22.180 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'nQT3SHUTnGGHoRE.HK4699V4TWQJbCA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e1258351470e217d-0064c4547a-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:22.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:22.181 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:22.182 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:22.182 DEBUG:boto:path=/hiiqos-13/obj?versionId=nQT3SHUTnGGHoRE.HK4699V4TWQJbCA 2023-07-28T23:51:22.182 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=nQT3SHUTnGGHoRE.HK4699V4TWQJbCA 2023-07-28T23:51:22.183 DEBUG:boto:Method: GET 2023-07-28T23:51:22.183 DEBUG:boto:Path: /hiiqos-13/obj?versionId=nQT3SHUTnGGHoRE.HK4699V4TWQJbCA 2023-07-28T23:51:22.183 DEBUG:boto:Data: 2023-07-28T23:51:22.184 DEBUG:boto:Headers: {} 2023-07-28T23:51:22.184 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:22.185 DEBUG:boto:Port: 8001 2023-07-28T23:51:22.185 DEBUG:boto:Params: {} 2023-07-28T23:51:22.185 DEBUG:boto:Token: None 2023-07-28T23:51:22.186 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:22 GMT /hiiqos-13/obj?versionId=nQT3SHUTnGGHoRE.HK4699V4TWQJbCA 2023-07-28T23:51:22.186 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ERP4hjK5wyWKm+x+j3iJqGuv6R8= 2023-07-28T23:51:22.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ERP4hjK5wyWKm+x+j3iJqGuv6R8=', 'Content-Length': '0'} 2023-07-28T23:51:22.188 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'nQT3SHUTnGGHoRE.HK4699V4TWQJbCA'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000567690ba2a56a4b2-0064c4547a-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:22.232 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:22.233 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:22.233 DEBUG:boto:path=/hiiqos-13/obj?versionId=nQT3SHUTnGGHoRE.HK4699V4TWQJbCA 2023-07-28T23:51:22.233 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=nQT3SHUTnGGHoRE.HK4699V4TWQJbCA 2023-07-28T23:51:22.234 DEBUG:boto:Method: GET 2023-07-28T23:51:22.234 DEBUG:boto:Path: /hiiqos-13/obj?versionId=nQT3SHUTnGGHoRE.HK4699V4TWQJbCA 2023-07-28T23:51:22.234 DEBUG:boto:Data: 2023-07-28T23:51:22.235 DEBUG:boto:Headers: {} 2023-07-28T23:51:22.235 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:22.235 DEBUG:boto:Port: 8000 2023-07-28T23:51:22.236 DEBUG:boto:Params: {} 2023-07-28T23:51:22.236 DEBUG:boto:Token: None 2023-07-28T23:51:22.237 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:22 GMT /hiiqos-13/obj?versionId=nQT3SHUTnGGHoRE.HK4699V4TWQJbCA 2023-07-28T23:51:22.237 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ERP4hjK5wyWKm+x+j3iJqGuv6R8= 2023-07-28T23:51:22.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ERP4hjK5wyWKm+x+j3iJqGuv6R8=', 'Content-Length': '0'} 2023-07-28T23:51:22.239 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'nQT3SHUTnGGHoRE.HK4699V4TWQJbCA'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000571db4f1e22f83e8-0064c4547a-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:22.240 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:22.240 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:22.241 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:22.241 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:22.242 DEBUG:boto:Method: GET 2023-07-28T23:51:22.242 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:22.242 DEBUG:boto:Data: 2023-07-28T23:51:22.243 DEBUG:boto:Headers: {} 2023-07-28T23:51:22.243 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:22.243 DEBUG:boto:Port: 8001 2023-07-28T23:51:22.244 DEBUG:boto:Params: {} 2023-07-28T23:51:22.244 DEBUG:boto:Token: None 2023-07-28T23:51:22.245 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:22 GMT /hiiqos-13/ 2023-07-28T23:51:22.245 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ckMBapogZxM4u620dQwUMBD89xw= 2023-07-28T23:51:22.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ckMBapogZxM4u620dQwUMBD89xw=', 'Content-Length': '0'} 2023-07-28T23:51:22.248 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a504aee498dc3201-0064c4547a-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:22.292 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:22.293 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:22.293 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:22.294 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:22.294 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:22.294 DEBUG:boto:Method: GET 2023-07-28T23:51:22.295 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:22.295 DEBUG:boto:Data: 2023-07-28T23:51:22.295 DEBUG:boto:Headers: {} 2023-07-28T23:51:22.296 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:22.296 DEBUG:boto:Port: 8000 2023-07-28T23:51:22.296 DEBUG:boto:Params: {} 2023-07-28T23:51:22.297 DEBUG:boto:Token: None 2023-07-28T23:51:22.297 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:22 GMT /hiiqos-13/ 2023-07-28T23:51:22.298 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ckMBapogZxM4u620dQwUMBD89xw= 2023-07-28T23:51:22.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ckMBapogZxM4u620dQwUMBD89xw=', 'Content-Length': '0'} 2023-07-28T23:51:22.301 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fd1cb656dba0db7-0064c4547a-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:22.301 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:22.302 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:22.302 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:22.302 DEBUG:boto:Method: HEAD 2023-07-28T23:51:22.303 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:51:22.303 DEBUG:boto:Data: 2023-07-28T23:51:22.303 DEBUG:boto:Headers: {} 2023-07-28T23:51:22.304 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:22.304 DEBUG:boto:Port: 8001 2023-07-28T23:51:22.304 DEBUG:boto:Params: {} 2023-07-28T23:51:22.305 DEBUG:boto:Token: None 2023-07-28T23:51:22.305 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:22 GMT /hiiqos-13/obj 2023-07-28T23:51:22.306 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:v0HaV0vO+lOuIs94TDL4lLQ1Lro= 2023-07-28T23:51:22.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:v0HaV0vO+lOuIs94TDL4lLQ1Lro=', 'Content-Length': '0'} 2023-07-28T23:51:22.309 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a17aff9d9e5e2eda-0064c4547a-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:22.310 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:22.310 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:22.310 DEBUG:boto:Method: HEAD 2023-07-28T23:51:22.311 DEBUG:boto:Path: /hiiqos-13/obj 2023-07-28T23:51:22.311 DEBUG:boto:Data: 2023-07-28T23:51:22.311 DEBUG:boto:Headers: {} 2023-07-28T23:51:22.312 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:22.312 DEBUG:boto:Port: 8000 2023-07-28T23:51:22.312 DEBUG:boto:Params: {} 2023-07-28T23:51:22.313 DEBUG:boto:Token: None 2023-07-28T23:51:22.313 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:22 GMT /hiiqos-13/obj 2023-07-28T23:51:22.314 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:v0HaV0vO+lOuIs94TDL4lLQ1Lro= 2023-07-28T23:51:22.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:v0HaV0vO+lOuIs94TDL4lLQ1Lro=', 'Content-Length': '0'} 2023-07-28T23:51:22.317 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007368ea220c24d273-0064c4547a-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:22.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:22.318 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:22.319 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:22.319 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:22.319 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:22.320 DEBUG:boto:Method: GET 2023-07-28T23:51:22.320 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:22.320 DEBUG:boto:Data: 2023-07-28T23:51:22.321 DEBUG:boto:Headers: {} 2023-07-28T23:51:22.321 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:22.321 DEBUG:boto:Port: 8001 2023-07-28T23:51:22.322 DEBUG:boto:Params: {} 2023-07-28T23:51:22.322 DEBUG:boto:Token: None 2023-07-28T23:51:22.323 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:22 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:22.323 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hzUz6a4UhLVlfwHL8N7Jw9AEjug= 2023-07-28T23:51:22.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hzUz6a4UhLVlfwHL8N7Jw9AEjug=', 'Content-Length': '0'} 2023-07-28T23:51:22.326 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000006e46154dbd2a8a4-0064c4547a-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:22.368 DEBUG:boto:path=/hiiqos-13/obj 2023-07-28T23:51:22.369 DEBUG:boto:auth_path=/hiiqos-13/obj 2023-07-28T23:51:22.369 DEBUG:boto:path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:22.370 DEBUG:boto:auth_path=/hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:22.370 DEBUG:boto:Method: GET 2023-07-28T23:51:22.370 DEBUG:boto:Path: /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:22.371 DEBUG:boto:Data: 2023-07-28T23:51:22.371 DEBUG:boto:Headers: {} 2023-07-28T23:51:22.371 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:22.372 DEBUG:boto:Port: 8000 2023-07-28T23:51:22.372 DEBUG:boto:Params: {} 2023-07-28T23:51:22.373 DEBUG:boto:Token: None 2023-07-28T23:51:22.373 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:22 GMT /hiiqos-13/obj?versionId=vbCzaR0huShRhVgNSTpuzXOsOhW3EDE 2023-07-28T23:51:22.373 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hzUz6a4UhLVlfwHL8N7Jw9AEjug= 2023-07-28T23:51:22.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hzUz6a4UhLVlfwHL8N7Jw9AEjug=', 'Content-Length': '0'} 2023-07-28T23:51:22.376 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:50:16 GMT'), ('x-amz-version-id', 'vbCzaR0huShRhVgNSTpuzXOsOhW3EDE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004635dc38db43e071-0064c4547a-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:22.376 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:22.377 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:22.377 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:22.378 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:22.378 DEBUG:boto:Method: GET 2023-07-28T23:51:22.378 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:22.379 DEBUG:boto:Data: 2023-07-28T23:51:22.379 DEBUG:boto:Headers: {} 2023-07-28T23:51:22.379 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:22.380 DEBUG:boto:Port: 8001 2023-07-28T23:51:22.380 DEBUG:boto:Params: {} 2023-07-28T23:51:22.381 DEBUG:boto:Token: None 2023-07-28T23:51:22.381 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:22 GMT /hiiqos-13/ 2023-07-28T23:51:22.382 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ckMBapogZxM4u620dQwUMBD89xw= 2023-07-28T23:51:22.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ckMBapogZxM4u620dQwUMBD89xw=', 'Content-Length': '0'} 2023-07-28T23:51:22.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c8bc24e44d4b51f-0064c4547a-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:22.428 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:22.429 DEBUG:boto:path=/hiiqos-13/ 2023-07-28T23:51:22.430 DEBUG:boto:auth_path=/hiiqos-13/ 2023-07-28T23:51:22.430 DEBUG:boto:path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:22.431 DEBUG:boto:auth_path=/hiiqos-13/?prefix=obj 2023-07-28T23:51:22.431 DEBUG:boto:Method: GET 2023-07-28T23:51:22.432 DEBUG:boto:Path: /hiiqos-13/?prefix=obj 2023-07-28T23:51:22.432 DEBUG:boto:Data: 2023-07-28T23:51:22.433 DEBUG:boto:Headers: {} 2023-07-28T23:51:22.433 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:22.433 DEBUG:boto:Port: 8000 2023-07-28T23:51:22.434 DEBUG:boto:Params: {} 2023-07-28T23:51:22.434 DEBUG:boto:Token: None 2023-07-28T23:51:22.435 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:22 GMT /hiiqos-13/ 2023-07-28T23:51:22.435 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ckMBapogZxM4u620dQwUMBD89xw= 2023-07-28T23:51:22.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ckMBapogZxM4u620dQwUMBD89xw=', 'Content-Length': '0'} 2023-07-28T23:51:22.438 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005280affb3341aac6-0064c4547a-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:22.439 DEBUG:boto:b'hiiqos-13obj1000falseobj2023-07-28T23:50:16.934Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:22.439 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-13 zones={a2, a1} 2023-07-28T23:51:22.440 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-07-28T23:51:22.440 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:51:22.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:51:22.441 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:51:22.493 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:51:22.493 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:51:22.540 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-28T23:51:22.541 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:51:22.541 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000001632", 2023-07-28T23:51:22.541 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:50:50.308534Z" 2023-07-28T23:51:22.541 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:51:22.541 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:51:22.542 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000002482", 2023-07-28T23:51:22.542 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:50:13.440177Z" 2023-07-28T23:51:22.542 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:51:22.542 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:51:22.542 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-07-28T23:51:22.542 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:46:26.300717Z" 2023-07-28T23:51:22.542 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:51:22.543 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:51:22.543 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:51:22.543 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:51:22.543 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:51:22.543 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-28T23:51:22.544 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000001632', 1: '00000000000000000000:00000000000000002482', 2: '00000000000000000000:00000000000000000929', 3: ''} 2023-07-28T23:51:22.545 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-07-28T23:51:22.545 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:51:22.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:51:22.546 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:51:22.594 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:51:22.595 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:51:22.635 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:51:22.635 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:51:22.636 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:51:22.636 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:51:22.636 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:51:22.636 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 16749483048653174838 2023-07-28T23:51:22.636 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:51:22.636 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:51:22.637 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:51:22.637 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:51:22.637 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:51:22.637 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-28T23:51:22.637 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000001632", 2023-07-28T23:51:22.637 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:51:22.638 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:51:22.638 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:51:22.638 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:50:50.308534Z" 2023-07-28T23:51:22.638 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:51:22.639 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:51:22.639 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:51:22.639 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:51:22.640 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:51:22.640 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-28T23:51:22.640 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000002482", 2023-07-28T23:51:22.641 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:51:22.641 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:51:22.641 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:51:22.641 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:50:13.440177Z" 2023-07-28T23:51:22.642 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:51:22.642 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:51:22.642 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:51:22.643 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:51:22.643 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:51:22.643 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-28T23:51:22.643 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-07-28T23:51:22.644 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:51:22.644 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:51:22.644 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:51:22.644 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:26.300717Z" 2023-07-28T23:51:22.645 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:51:22.645 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:51:22.645 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:51:22.645 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:51:22.645 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:51:22.645 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-28T23:51:22.645 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:51:22.646 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:51:22.646 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:51:22.646 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:51:22.646 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000" 2023-07-28T23:51:22.646 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:51:22.646 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:51:22.646 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:51:22.647 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:51:22.647 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:51:22.647 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 0, 2023-07-28T23:51:22.647 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 0 2023-07-28T23:51:22.647 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:51:22.647 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:51:22Z" 2023-07-28T23:51:22.648 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:51:22.649 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16749483048653174838 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001632", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:50:50.308534Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002482", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:50:13.440177Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000929", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:46:26.300717Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-07-28T23:51:22Z" } 2023-07-28T23:51:22.649 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001632', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:50:50.308534Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002482', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:50:13.440177Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000929', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:46:26.300717Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-07-28T23:51:22.650 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001632', 1: '00000000000000000000:00000000000000002482', 2: '00000000000000000000:00000000000000000929', 3: ''} 2023-07-28T23:51:22.650 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000001632', 1: '00000000000000000000:00000000000000002482', 2: '00000000000000000000:00000000000000000929', 3: ''} 2023-07-28T23:51:22.650 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-07-28T23:51:22.651 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-07-28T23:51:22.651 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:51:22.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:51:22.652 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:51:22.696 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:51:22.697 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:51:22.744 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-28T23:51:22.744 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:51:22.745 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000002337", 2023-07-28T23:51:22.745 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:51:12.243929Z" 2023-07-28T23:51:22.745 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:51:22.745 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:51:22.745 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000001918", 2023-07-28T23:51:22.745 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:49:53.525225Z" 2023-07-28T23:51:22.745 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:51:22.746 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:51:22.746 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-07-28T23:51:22.746 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:46:48.232939Z" 2023-07-28T23:51:22.746 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:51:22.746 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:51:22.746 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:51:22.747 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:51:22.747 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:51:22.747 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-28T23:51:22.748 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000002337', 1: '00000000000000000000:00000000000000001918', 2: '00000000000000000000:00000000000000000929', 3: ''} 2023-07-28T23:51:22.748 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-07-28T23:51:22.748 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:51:22.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:51:22.749 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:51:22.796 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:51:22.796 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:51:22.836 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:51:22.836 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:51:22.837 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:51:22.837 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:51:22.837 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:51:22.837 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 17882949198279251152 2023-07-28T23:51:22.837 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:51:22.837 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:51:22.837 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:51:22.838 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:51:22.838 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:51:22.838 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-28T23:51:22.838 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000002337", 2023-07-28T23:51:22.838 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:51:22.838 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:51:22.838 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:51:22.839 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:51:12.243929Z" 2023-07-28T23:51:22.839 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:51:22.839 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:51:22.839 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:51:22.839 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:51:22.839 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:51:22.839 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-28T23:51:22.840 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000001918", 2023-07-28T23:51:22.840 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:51:22.840 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:51:22.840 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:51:22.840 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:49:53.525225Z" 2023-07-28T23:51:22.840 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:51:22.840 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:51:22.841 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:51:22.841 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:51:22.841 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:51:22.841 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-28T23:51:22.841 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-07-28T23:51:22.841 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:51:22.842 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:51:22.842 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:51:22.842 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:48.232939Z" 2023-07-28T23:51:22.842 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:51:22.842 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:51:22.842 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:51:22.842 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:51:22.843 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:51:22.843 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-28T23:51:22.843 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:51:22.843 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:51:22.843 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:51:22.843 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:51:22.843 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000" 2023-07-28T23:51:22.844 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:51:22.844 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:51:22.844 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:51:22.844 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:51:22.844 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:51:22.844 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 0, 2023-07-28T23:51:22.844 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 0 2023-07-28T23:51:22.845 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:51:22.845 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:51:22Z" 2023-07-28T23:51:22.845 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:51:22.846 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17882949198279251152 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002337", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:51:12.243929Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:49:53.525225Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000929", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:46:48.232939Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-07-28T23:51:22Z" } 2023-07-28T23:51:22.846 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002337', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:51:12.243929Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:49:53.525225Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000929', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:46:48.232939Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-07-28T23:51:22.847 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002337', 1: '00000000000000000000:00000000000000001918', 2: '00000000000000000000:00000000000000000929', 3: ''} 2023-07-28T23:51:22.847 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002337', 1: '00000000000000000000:00000000000000001918', 2: '00000000000000000000:00000000000000000929', 3: ''} 2023-07-28T23:51:22.847 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-07-28T23:51:22.848 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2023-07-28T23:51:22.849 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:51:22.849 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:51:22.850 DEBUG:boto:Method: PUT 2023-07-28T23:51:22.850 DEBUG:boto:Path: /hiiqos-14/ 2023-07-28T23:51:22.850 DEBUG:boto:Data: 2023-07-28T23:51:22.851 DEBUG:boto:Headers: {} 2023-07-28T23:51:22.851 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:22.851 DEBUG:boto:Port: 8000 2023-07-28T23:51:22.852 DEBUG:boto:Params: {} 2023-07-28T23:51:22.852 DEBUG:boto:Token: None 2023-07-28T23:51:22.853 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:51:22 GMT /hiiqos-14/ 2023-07-28T23:51:22.853 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8t8yKNYS2k/e4BTGBb883pXyur8= 2023-07-28T23:51:22.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8t8yKNYS2k/e4BTGBb883pXyur8=', 'Content-Length': '0'} 2023-07-28T23:51:22.865 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000014a0c919bd62c653-0064c4547a-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:51:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:22.866 DEBUG:rgw_multi.tests:created bucket=hiiqos-14 2023-07-28T23:51:22.866 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:51:22.866 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:51:22.867 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:51:22.913 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:51:22.913 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:51:22.945 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-28T23:51:22.965 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-28T23:51:22.965 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:51:22.965 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-28T23:51:22.965 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:46:09.233742Z" 2023-07-28T23:51:22.965 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:51:22.966 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:51:22.966 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588282.863575_188.1", 2023-07-28T23:51:22.966 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:51:22.863575Z" 2023-07-28T23:51:22.967 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:51:22.967 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:51:22.967 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:51:22.967 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:51:22.967 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:51:22.967 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:51:22.967 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:51:22.968 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:51:22.968 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:51:22.968 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-28T23:51:22.969 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690587969.233742_99.1', 1: '1_1690588282.863575_188.1', 2: '', 3: ''} 2023-07-28T23:51:22.969 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-28T23:51:22.970 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:51:22.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:51:22.971 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:51:23.017 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:51:23.017 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:51:23.044 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:51:23.043+0000 7f3da4b35880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:51:23.045 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:51:23.044+0000 7f3da4b35880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:51:23.056 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:51:23.057 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:51:23.057 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:51:23.057 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:51:23.057 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:51:23.057 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:51:23.057 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:51:23.058 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:51:23.058 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:51:23.058 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:51:23.059 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:51:23.059 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:51:23.059 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:51:23.059 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-28T23:51:23.059 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:51:23.059 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:51:23.060 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:51:23.060 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-28T23:51:23.060 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:51:23.060 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:51:23.060 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:51:23.060 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:51:23.061 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:51:23.061 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:51:23.061 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:51:23.061 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588282.863575_188.1", 2023-07-28T23:51:23.061 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:51:23.061 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:51:23.061 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:51:23.062 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:51:22.863575Z", 2023-07-28T23:51:23.062 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:51:23.062 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:51:23.062 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:51:23.062 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:51:23.063 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:51:23.063 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:51:23.063 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:51:23.063 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:51:23.063 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:51:23.063 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:51:23.063 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:51:23.064 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:51:23.064 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:51:23.064 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:51:23.064 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:51:23.064 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:51:23.064 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:51:23.065 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:51:23.065 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:51:23.065 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:51:23.065 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:51:23.065 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:51:23.065 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:51:23.065 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:51:23.065 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:51:23.066 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:51:23.066 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:51:23.066 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:51:23.066 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:51:23.066 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:51:23.066 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:51:23.066 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:51:23.067 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:51:23.067 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:51:23Z" 2023-07-28T23:51:23.067 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:51:23.068 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588282.863575_188.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:51:22.863575Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-28T23:51:23Z" } 2023-07-28T23:51:23.068 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588282.863575_188.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:51:22.863575Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-28T23:51:23.069 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588282.863575_188.1', 2: '', 3: ''} 2023-07-28T23:51:23.069 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588282.863575_188.1', 2: '', 3: ''} 2023-07-28T23:51:23.070 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-28T23:51:23.070 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:51:23.070 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:51:23.071 DEBUG:boto:Method: HEAD 2023-07-28T23:51:23.071 DEBUG:boto:Path: /hiiqos-14/ 2023-07-28T23:51:23.071 DEBUG:boto:Data: 2023-07-28T23:51:23.072 DEBUG:boto:Headers: {} 2023-07-28T23:51:23.072 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:23.072 DEBUG:boto:Port: 8000 2023-07-28T23:51:23.073 DEBUG:boto:Params: {} 2023-07-28T23:51:23.073 DEBUG:boto:Token: None 2023-07-28T23:51:23.073 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:23 GMT /hiiqos-14/ 2023-07-28T23:51:23.074 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SGEQM0sfpfBYRTGwRwxI/Mz+epQ= 2023-07-28T23:51:23.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SGEQM0sfpfBYRTGwRwxI/Mz+epQ=', 'Content-Length': '0'} 2023-07-28T23:51:23.076 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b782e5f207841760-0064c4547b-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:51:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:23.077 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:51:23.077 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:51:23.078 DEBUG:boto:Method: PUT 2023-07-28T23:51:23.078 DEBUG:boto:Path: /hiiqos-14/obj 2023-07-28T23:51:23.078 DEBUG:boto:Data: 2023-07-28T23:51:23.079 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-07-28T23:51:23.079 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:23.079 DEBUG:boto:Port: 8000 2023-07-28T23:51:23.080 DEBUG:boto:Params: {} 2023-07-28T23:51:23.080 DEBUG:boto:Token: None 2023-07-28T23:51:23.081 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 28 Jul 2023 23:51:23 GMT /hiiqos-14/obj 2023-07-28T23:51:23.081 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Ujv4R3qReTkXi/NbSGWk4CT9bvo= 2023-07-28T23:51:23.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:51:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Ujv4R3qReTkXi/NbSGWk4CT9bvo='} 2023-07-28T23:51:23.087 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e1fa74cdfc2ee77a-0064c4547b-1117-a1'), ('Date', 'Fri, 28 Jul 2023 23:51:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:23.087 DEBUG:rgw_multi.tests:created initial version id=None 2023-07-28T23:51:23.088 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-14 2023-07-28T23:51:23.088 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:51:23.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:51:23.089 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-28T23:51:23.141 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:51:23.141 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:51:23.173 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:51:23.171+0000 7f336d335880 1 waiting to reach incremental sync.. 2023-07-28T23:51:28.173 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:51:28.172+0000 7f336d335880 1 waiting to reach incremental sync.. 2023-07-28T23:51:33.174 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:51:33.172+0000 7f336d335880 1 waiting to reach incremental sync.. 2023-07-28T23:51:38.175 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:51:38.174+0000 7f336d335880 1 bucket sync caught up with source: 2023-07-28T23:51:38.176 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , 00000000001.29.6, , ] 2023-07-28T23:51:38.176 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000001.29.6, , ] 2023-07-28T23:51:38.176 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:51:38.174+0000 7f336d335880 0 bucket checkpoint complete 2023-07-28T23:51:38.191 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-14 2023-07-28T23:51:38.192 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:51:38.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:51:38.193 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-28T23:51:38.252 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:51:38.252 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:51:38.287 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:51:38.285+0000 7f854da2f880 1 bucket sync caught up with source: 2023-07-28T23:51:38.287 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , 00000000001.17.6, , ] 2023-07-28T23:51:38.287 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000001.17.6, , ] 2023-07-28T23:51:38.287 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:51:38.285+0000 7f854da2f880 0 bucket checkpoint complete 2023-07-28T23:51:38.299 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-14 zones={a2, a1} 2023-07-28T23:51:38.300 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:51:38.300 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:51:38.301 DEBUG:boto:Method: HEAD 2023-07-28T23:51:38.301 DEBUG:boto:Path: /hiiqos-14/ 2023-07-28T23:51:38.301 DEBUG:boto:Data: 2023-07-28T23:51:38.302 DEBUG:boto:Headers: {} 2023-07-28T23:51:38.302 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:38.302 DEBUG:boto:Port: 8001 2023-07-28T23:51:38.303 DEBUG:boto:Params: {} 2023-07-28T23:51:38.303 DEBUG:boto:Token: None 2023-07-28T23:51:38.304 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:38 GMT /hiiqos-14/ 2023-07-28T23:51:38.304 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5CQhA13KlWRiqsYH8S30wVxHXI0= 2023-07-28T23:51:38.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5CQhA13KlWRiqsYH8S30wVxHXI0=', 'Content-Length': '0'} 2023-07-28T23:51:38.307 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000060d1623521c92ddf-0064c4548a-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:38.307 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:51:38.308 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:51:38.308 DEBUG:boto:Method: HEAD 2023-07-28T23:51:38.308 DEBUG:boto:Path: /hiiqos-14/ 2023-07-28T23:51:38.309 DEBUG:boto:Data: 2023-07-28T23:51:38.309 DEBUG:boto:Headers: {} 2023-07-28T23:51:38.309 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:38.310 DEBUG:boto:Port: 8000 2023-07-28T23:51:38.310 DEBUG:boto:Params: {} 2023-07-28T23:51:38.310 DEBUG:boto:Token: None 2023-07-28T23:51:38.311 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:38 GMT /hiiqos-14/ 2023-07-28T23:51:38.311 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5CQhA13KlWRiqsYH8S30wVxHXI0= 2023-07-28T23:51:38.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5CQhA13KlWRiqsYH8S30wVxHXI0=', 'Content-Length': '0'} 2023-07-28T23:51:38.313 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ceb3d8f74806c28e-0064c4548a-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:38.314 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-28T23:51:38.314 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:51:38.315 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:51:38.315 DEBUG:boto:path=/hiiqos-14/?versions&max-keys=999 2023-07-28T23:51:38.315 DEBUG:boto:auth_path=/hiiqos-14/?versions&max-keys=999 2023-07-28T23:51:38.316 DEBUG:boto:Method: GET 2023-07-28T23:51:38.316 DEBUG:boto:Path: /hiiqos-14/?versions&max-keys=999 2023-07-28T23:51:38.316 DEBUG:boto:Data: 2023-07-28T23:51:38.317 DEBUG:boto:Headers: {} 2023-07-28T23:51:38.317 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:38.317 DEBUG:boto:Port: 8001 2023-07-28T23:51:38.318 DEBUG:boto:Params: {} 2023-07-28T23:51:38.318 DEBUG:boto:Token: None 2023-07-28T23:51:38.318 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:38 GMT /hiiqos-14/?versions 2023-07-28T23:51:38.319 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2Bxk/Gb5yiGnuUM7f/ojM9XGxpw= 2023-07-28T23:51:38.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2Bxk/Gb5yiGnuUM7f/ojM9XGxpw=', 'Content-Length': '0'} 2023-07-28T23:51:38.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d34fe311f64d4271-0064c4548a-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:38.322 DEBUG:boto:b'hiiqos-14999falseobjnulltrue2023-07-28T23:51:23.083Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:38.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:51:38.323 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-28T23:51:38.323 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:51:38.324 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:51:38.324 DEBUG:boto:path=/hiiqos-14/?versions&max-keys=999 2023-07-28T23:51:38.324 DEBUG:boto:auth_path=/hiiqos-14/?versions&max-keys=999 2023-07-28T23:51:38.325 DEBUG:boto:Method: GET 2023-07-28T23:51:38.325 DEBUG:boto:Path: /hiiqos-14/?versions&max-keys=999 2023-07-28T23:51:38.325 DEBUG:boto:Data: 2023-07-28T23:51:38.326 DEBUG:boto:Headers: {} 2023-07-28T23:51:38.326 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:38.326 DEBUG:boto:Port: 8000 2023-07-28T23:51:38.327 DEBUG:boto:Params: {} 2023-07-28T23:51:38.327 DEBUG:boto:Token: None 2023-07-28T23:51:38.327 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:38 GMT /hiiqos-14/?versions 2023-07-28T23:51:38.328 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2Bxk/Gb5yiGnuUM7f/ojM9XGxpw= 2023-07-28T23:51:38.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2Bxk/Gb5yiGnuUM7f/ojM9XGxpw=', 'Content-Length': '0'} 2023-07-28T23:51:38.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000511c3bb364b7762e-0064c4548a-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:38.331 DEBUG:boto:b'hiiqos-14999falseobjnulltrue2023-07-28T23:51:23.083Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:38.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:51:38.332 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:51:38.332 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:51:38.332 DEBUG:boto:path=/hiiqos-14/?versions&max-keys=999 2023-07-28T23:51:38.333 DEBUG:boto:auth_path=/hiiqos-14/?versions&max-keys=999 2023-07-28T23:51:38.333 DEBUG:boto:Method: GET 2023-07-28T23:51:38.333 DEBUG:boto:Path: /hiiqos-14/?versions&max-keys=999 2023-07-28T23:51:38.334 DEBUG:boto:Data: 2023-07-28T23:51:38.334 DEBUG:boto:Headers: {} 2023-07-28T23:51:38.334 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:38.335 DEBUG:boto:Port: 8001 2023-07-28T23:51:38.335 DEBUG:boto:Params: {} 2023-07-28T23:51:38.335 DEBUG:boto:Token: None 2023-07-28T23:51:38.336 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:38 GMT /hiiqos-14/?versions 2023-07-28T23:51:38.336 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2Bxk/Gb5yiGnuUM7f/ojM9XGxpw= 2023-07-28T23:51:38.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2Bxk/Gb5yiGnuUM7f/ojM9XGxpw=', 'Content-Length': '0'} 2023-07-28T23:51:38.338 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000467544fae0336ec6-0064c4548a-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:38.339 DEBUG:boto:b'hiiqos-14999falseobjnulltrue2023-07-28T23:51:23.083Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:38.340 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:51:38.340 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:51:38.340 DEBUG:boto:path=/hiiqos-14/?versions&max-keys=999 2023-07-28T23:51:38.341 DEBUG:boto:auth_path=/hiiqos-14/?versions&max-keys=999 2023-07-28T23:51:38.341 DEBUG:boto:Method: GET 2023-07-28T23:51:38.341 DEBUG:boto:Path: /hiiqos-14/?versions&max-keys=999 2023-07-28T23:51:38.342 DEBUG:boto:Data: 2023-07-28T23:51:38.342 DEBUG:boto:Headers: {} 2023-07-28T23:51:38.342 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:38.343 DEBUG:boto:Port: 8000 2023-07-28T23:51:38.343 DEBUG:boto:Params: {} 2023-07-28T23:51:38.343 DEBUG:boto:Token: None 2023-07-28T23:51:38.344 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:38 GMT /hiiqos-14/?versions 2023-07-28T23:51:38.344 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2Bxk/Gb5yiGnuUM7f/ojM9XGxpw= 2023-07-28T23:51:38.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2Bxk/Gb5yiGnuUM7f/ojM9XGxpw=', 'Content-Length': '0'} 2023-07-28T23:51:38.346 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061bf4316a09f9482-0064c4548a-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:38.347 DEBUG:boto:b'hiiqos-14999falseobjnulltrue2023-07-28T23:51:23.083Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:38.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:38.348 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:51:38.348 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:51:38.349 DEBUG:boto:path=/hiiqos-14/obj?versionId=null 2023-07-28T23:51:38.349 DEBUG:boto:auth_path=/hiiqos-14/obj?versionId=null 2023-07-28T23:51:38.349 DEBUG:boto:Method: GET 2023-07-28T23:51:38.350 DEBUG:boto:Path: /hiiqos-14/obj?versionId=null 2023-07-28T23:51:38.350 DEBUG:boto:Data: 2023-07-28T23:51:38.350 DEBUG:boto:Headers: {} 2023-07-28T23:51:38.351 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:38.351 DEBUG:boto:Port: 8001 2023-07-28T23:51:38.351 DEBUG:boto:Params: {} 2023-07-28T23:51:38.352 DEBUG:boto:Token: None 2023-07-28T23:51:38.352 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:38 GMT /hiiqos-14/obj?versionId=null 2023-07-28T23:51:38.352 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RWeGX4hBURJx2uz96V/xvwZCE1A= 2023-07-28T23:51:38.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RWeGX4hBURJx2uz96V/xvwZCE1A=', 'Content-Length': '0'} 2023-07-28T23:51:38.354 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:51:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000123ce15f6ced5876-0064c4548a-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:38.355 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:51:38.355 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:51:38.356 DEBUG:boto:path=/hiiqos-14/obj?versionId=null 2023-07-28T23:51:38.356 DEBUG:boto:auth_path=/hiiqos-14/obj?versionId=null 2023-07-28T23:51:38.357 DEBUG:boto:Method: GET 2023-07-28T23:51:38.357 DEBUG:boto:Path: /hiiqos-14/obj?versionId=null 2023-07-28T23:51:38.357 DEBUG:boto:Data: 2023-07-28T23:51:38.358 DEBUG:boto:Headers: {} 2023-07-28T23:51:38.358 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:38.358 DEBUG:boto:Port: 8000 2023-07-28T23:51:38.359 DEBUG:boto:Params: {} 2023-07-28T23:51:38.359 DEBUG:boto:Token: None 2023-07-28T23:51:38.360 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:38 GMT /hiiqos-14/obj?versionId=null 2023-07-28T23:51:38.360 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RWeGX4hBURJx2uz96V/xvwZCE1A= 2023-07-28T23:51:38.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RWeGX4hBURJx2uz96V/xvwZCE1A=', 'Content-Length': '0'} 2023-07-28T23:51:38.362 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:51:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009cfbfb3fd26acd11-0064c4548a-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:38.363 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:51:38.363 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:51:38.363 DEBUG:boto:path=/hiiqos-14/obj?versionId=null 2023-07-28T23:51:38.364 DEBUG:boto:auth_path=/hiiqos-14/obj?versionId=null 2023-07-28T23:51:38.364 DEBUG:boto:Method: HEAD 2023-07-28T23:51:38.364 DEBUG:boto:Path: /hiiqos-14/obj?versionId=null 2023-07-28T23:51:38.365 DEBUG:boto:Data: 2023-07-28T23:51:38.365 DEBUG:boto:Headers: {} 2023-07-28T23:51:38.365 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:38.366 DEBUG:boto:Port: 8001 2023-07-28T23:51:38.366 DEBUG:boto:Params: {} 2023-07-28T23:51:38.366 DEBUG:boto:Token: None 2023-07-28T23:51:38.367 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:38 GMT /hiiqos-14/obj?versionId=null 2023-07-28T23:51:38.367 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+GKNrsPETikBSmy6Q1r/jGEqiiE= 2023-07-28T23:51:38.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+GKNrsPETikBSmy6Q1r/jGEqiiE=', 'Content-Length': '0'} 2023-07-28T23:51:38.369 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:51:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dde0c528a7e06cee-0064c4548a-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:38.370 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:51:38.370 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:51:38.371 DEBUG:boto:path=/hiiqos-14/obj?versionId=null 2023-07-28T23:51:38.371 DEBUG:boto:auth_path=/hiiqos-14/obj?versionId=null 2023-07-28T23:51:38.371 DEBUG:boto:Method: HEAD 2023-07-28T23:51:38.372 DEBUG:boto:Path: /hiiqos-14/obj?versionId=null 2023-07-28T23:51:38.372 DEBUG:boto:Data: 2023-07-28T23:51:38.372 DEBUG:boto:Headers: {} 2023-07-28T23:51:38.373 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:38.373 DEBUG:boto:Port: 8000 2023-07-28T23:51:38.373 DEBUG:boto:Params: {} 2023-07-28T23:51:38.374 DEBUG:boto:Token: None 2023-07-28T23:51:38.374 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:38 GMT /hiiqos-14/obj?versionId=null 2023-07-28T23:51:38.374 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+GKNrsPETikBSmy6Q1r/jGEqiiE= 2023-07-28T23:51:38.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+GKNrsPETikBSmy6Q1r/jGEqiiE=', 'Content-Length': '0'} 2023-07-28T23:51:38.376 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:51:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000092e5bd77865f4d43-0064c4548a-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:38.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:38.377 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:51:38.378 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:51:38.378 DEBUG:boto:path=/hiiqos-14/obj?versionId=null 2023-07-28T23:51:38.378 DEBUG:boto:auth_path=/hiiqos-14/obj?versionId=null 2023-07-28T23:51:38.379 DEBUG:boto:Method: GET 2023-07-28T23:51:38.379 DEBUG:boto:Path: /hiiqos-14/obj?versionId=null 2023-07-28T23:51:38.379 DEBUG:boto:Data: 2023-07-28T23:51:38.380 DEBUG:boto:Headers: {} 2023-07-28T23:51:38.380 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:38.381 DEBUG:boto:Port: 8001 2023-07-28T23:51:38.381 DEBUG:boto:Params: {} 2023-07-28T23:51:38.381 DEBUG:boto:Token: None 2023-07-28T23:51:38.382 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:38 GMT /hiiqos-14/obj?versionId=null 2023-07-28T23:51:38.382 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RWeGX4hBURJx2uz96V/xvwZCE1A= 2023-07-28T23:51:38.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RWeGX4hBURJx2uz96V/xvwZCE1A=', 'Content-Length': '0'} 2023-07-28T23:51:38.384 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:51:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000032a1e074f8a7780d-0064c4548a-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:38.384 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:51:38.385 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:51:38.385 DEBUG:boto:path=/hiiqos-14/obj?versionId=null 2023-07-28T23:51:38.386 DEBUG:boto:auth_path=/hiiqos-14/obj?versionId=null 2023-07-28T23:51:38.386 DEBUG:boto:Method: GET 2023-07-28T23:51:38.386 DEBUG:boto:Path: /hiiqos-14/obj?versionId=null 2023-07-28T23:51:38.387 DEBUG:boto:Data: 2023-07-28T23:51:38.387 DEBUG:boto:Headers: {} 2023-07-28T23:51:38.387 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:38.388 DEBUG:boto:Port: 8000 2023-07-28T23:51:38.388 DEBUG:boto:Params: {} 2023-07-28T23:51:38.388 DEBUG:boto:Token: None 2023-07-28T23:51:38.389 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:38 GMT /hiiqos-14/obj?versionId=null 2023-07-28T23:51:38.389 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RWeGX4hBURJx2uz96V/xvwZCE1A= 2023-07-28T23:51:38.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RWeGX4hBURJx2uz96V/xvwZCE1A=', 'Content-Length': '0'} 2023-07-28T23:51:38.391 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:51:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000032da843c84d5e672-0064c4548a-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:38.392 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:51:38.392 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:51:38.392 DEBUG:boto:path=/hiiqos-14/?prefix=obj 2023-07-28T23:51:38.393 DEBUG:boto:auth_path=/hiiqos-14/?prefix=obj 2023-07-28T23:51:38.393 DEBUG:boto:Method: GET 2023-07-28T23:51:38.394 DEBUG:boto:Path: /hiiqos-14/?prefix=obj 2023-07-28T23:51:38.394 DEBUG:boto:Data: 2023-07-28T23:51:38.394 DEBUG:boto:Headers: {} 2023-07-28T23:51:38.395 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:38.395 DEBUG:boto:Port: 8001 2023-07-28T23:51:38.395 DEBUG:boto:Params: {} 2023-07-28T23:51:38.396 DEBUG:boto:Token: None 2023-07-28T23:51:38.396 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:38 GMT /hiiqos-14/ 2023-07-28T23:51:38.397 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:t65j/w7ylDFdhVqMgc979nrJWis= 2023-07-28T23:51:38.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t65j/w7ylDFdhVqMgc979nrJWis=', 'Content-Length': '0'} 2023-07-28T23:51:38.399 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000964f13ace9ef9cba-0064c4548a-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:38.399 DEBUG:boto:b'hiiqos-14obj1000falseobj2023-07-28T23:51:23.083Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:38.400 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:51:38.400 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:51:38.401 DEBUG:boto:path=/hiiqos-14/?prefix=obj 2023-07-28T23:51:38.401 DEBUG:boto:auth_path=/hiiqos-14/?prefix=obj 2023-07-28T23:51:38.402 DEBUG:boto:Method: GET 2023-07-28T23:51:38.402 DEBUG:boto:Path: /hiiqos-14/?prefix=obj 2023-07-28T23:51:38.402 DEBUG:boto:Data: 2023-07-28T23:51:38.403 DEBUG:boto:Headers: {} 2023-07-28T23:51:38.403 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:38.403 DEBUG:boto:Port: 8000 2023-07-28T23:51:38.404 DEBUG:boto:Params: {} 2023-07-28T23:51:38.404 DEBUG:boto:Token: None 2023-07-28T23:51:38.405 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:38 GMT /hiiqos-14/ 2023-07-28T23:51:38.405 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:t65j/w7ylDFdhVqMgc979nrJWis= 2023-07-28T23:51:38.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t65j/w7ylDFdhVqMgc979nrJWis=', 'Content-Length': '0'} 2023-07-28T23:51:38.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e172b34e5cec77ee-0064c4548a-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:38.448 DEBUG:boto:b'hiiqos-14obj1000falseobj2023-07-28T23:51:23.083Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:38.449 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:51:38.449 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:51:38.450 DEBUG:boto:Method: HEAD 2023-07-28T23:51:38.450 DEBUG:boto:Path: /hiiqos-14/obj 2023-07-28T23:51:38.450 DEBUG:boto:Data: 2023-07-28T23:51:38.451 DEBUG:boto:Headers: {} 2023-07-28T23:51:38.451 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:38.451 DEBUG:boto:Port: 8001 2023-07-28T23:51:38.451 DEBUG:boto:Params: {} 2023-07-28T23:51:38.452 DEBUG:boto:Token: None 2023-07-28T23:51:38.452 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:38 GMT /hiiqos-14/obj 2023-07-28T23:51:38.453 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/8uHyeWbPIcrYeoWeDwmgkQjpMs= 2023-07-28T23:51:38.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/8uHyeWbPIcrYeoWeDwmgkQjpMs=', 'Content-Length': '0'} 2023-07-28T23:51:38.455 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:51:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c0789f7edc7f2bfa-0064c4548a-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:38.455 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:51:38.456 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:51:38.456 DEBUG:boto:Method: HEAD 2023-07-28T23:51:38.456 DEBUG:boto:Path: /hiiqos-14/obj 2023-07-28T23:51:38.457 DEBUG:boto:Data: 2023-07-28T23:51:38.457 DEBUG:boto:Headers: {} 2023-07-28T23:51:38.457 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:38.458 DEBUG:boto:Port: 8000 2023-07-28T23:51:38.458 DEBUG:boto:Params: {} 2023-07-28T23:51:38.458 DEBUG:boto:Token: None 2023-07-28T23:51:38.459 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:38 GMT /hiiqos-14/obj 2023-07-28T23:51:38.459 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/8uHyeWbPIcrYeoWeDwmgkQjpMs= 2023-07-28T23:51:38.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/8uHyeWbPIcrYeoWeDwmgkQjpMs=', 'Content-Length': '0'} 2023-07-28T23:51:38.461 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:51:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000009419e1ac0df8e4d-0064c4548a-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:38.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:51:38.462 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:51:38.463 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:51:38.463 DEBUG:boto:Method: GET 2023-07-28T23:51:38.463 DEBUG:boto:Path: /hiiqos-14/obj 2023-07-28T23:51:38.464 DEBUG:boto:Data: 2023-07-28T23:51:38.464 DEBUG:boto:Headers: {} 2023-07-28T23:51:38.464 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:38.465 DEBUG:boto:Port: 8001 2023-07-28T23:51:38.465 DEBUG:boto:Params: {} 2023-07-28T23:51:38.465 DEBUG:boto:Token: None 2023-07-28T23:51:38.466 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:38 GMT /hiiqos-14/obj 2023-07-28T23:51:38.466 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I6BYQwfvV2n2hLqLhfOVxPeg3Vc= 2023-07-28T23:51:38.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I6BYQwfvV2n2hLqLhfOVxPeg3Vc=', 'Content-Length': '0'} 2023-07-28T23:51:38.468 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:51:23 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000220096aea147801d-0064c4548a-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:38.469 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:51:38.469 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:51:38.469 DEBUG:boto:Method: GET 2023-07-28T23:51:38.470 DEBUG:boto:Path: /hiiqos-14/obj 2023-07-28T23:51:38.470 DEBUG:boto:Data: 2023-07-28T23:51:38.471 DEBUG:boto:Headers: {} 2023-07-28T23:51:38.471 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:38.471 DEBUG:boto:Port: 8000 2023-07-28T23:51:38.472 DEBUG:boto:Params: {} 2023-07-28T23:51:38.472 DEBUG:boto:Token: None 2023-07-28T23:51:38.472 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:38 GMT /hiiqos-14/obj 2023-07-28T23:51:38.473 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I6BYQwfvV2n2hLqLhfOVxPeg3Vc= 2023-07-28T23:51:38.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I6BYQwfvV2n2hLqLhfOVxPeg3Vc=', 'Content-Length': '0'} 2023-07-28T23:51:38.475 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:51:23 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000db60156094ba428d-0064c4548a-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:38.475 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:51:38.476 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:51:38.476 DEBUG:boto:path=/hiiqos-14/?prefix=obj 2023-07-28T23:51:38.476 DEBUG:boto:auth_path=/hiiqos-14/?prefix=obj 2023-07-28T23:51:38.477 DEBUG:boto:Method: GET 2023-07-28T23:51:38.477 DEBUG:boto:Path: /hiiqos-14/?prefix=obj 2023-07-28T23:51:38.478 DEBUG:boto:Data: 2023-07-28T23:51:38.478 DEBUG:boto:Headers: {} 2023-07-28T23:51:38.478 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:51:38.479 DEBUG:boto:Port: 8001 2023-07-28T23:51:38.479 DEBUG:boto:Params: {} 2023-07-28T23:51:38.479 DEBUG:boto:Token: None 2023-07-28T23:51:38.480 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:38 GMT /hiiqos-14/ 2023-07-28T23:51:38.480 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:t65j/w7ylDFdhVqMgc979nrJWis= 2023-07-28T23:51:38.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t65j/w7ylDFdhVqMgc979nrJWis=', 'Content-Length': '0'} 2023-07-28T23:51:38.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db742710df946f81-0064c4548a-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:38.483 DEBUG:boto:b'hiiqos-14obj1000falseobj2023-07-28T23:51:23.083Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:38.484 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:51:38.484 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:51:38.485 DEBUG:boto:path=/hiiqos-14/?prefix=obj 2023-07-28T23:51:38.485 DEBUG:boto:auth_path=/hiiqos-14/?prefix=obj 2023-07-28T23:51:38.485 DEBUG:boto:Method: GET 2023-07-28T23:51:38.486 DEBUG:boto:Path: /hiiqos-14/?prefix=obj 2023-07-28T23:51:38.486 DEBUG:boto:Data: 2023-07-28T23:51:38.487 DEBUG:boto:Headers: {} 2023-07-28T23:51:38.487 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:38.487 DEBUG:boto:Port: 8000 2023-07-28T23:51:38.488 DEBUG:boto:Params: {} 2023-07-28T23:51:38.488 DEBUG:boto:Token: None 2023-07-28T23:51:38.488 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:51:38 GMT /hiiqos-14/ 2023-07-28T23:51:38.489 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:t65j/w7ylDFdhVqMgc979nrJWis= 2023-07-28T23:51:38.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t65j/w7ylDFdhVqMgc979nrJWis=', 'Content-Length': '0'} 2023-07-28T23:51:38.491 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058af9b377f31a61b-0064c4548a-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:38.492 DEBUG:boto:b'hiiqos-14obj1000falseobj2023-07-28T23:51:23.083Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:51:38.493 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-14 zones={a2, a1} 2023-07-28T23:51:38.493 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:51:38.493 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:51:38.494 DEBUG:boto:path=/hiiqos-14/?versioning 2023-07-28T23:51:38.494 DEBUG:boto:auth_path=/hiiqos-14/?versioning 2023-07-28T23:51:38.494 DEBUG:boto:Method: PUT 2023-07-28T23:51:38.494 DEBUG:boto:Path: /hiiqos-14/?versioning 2023-07-28T23:51:38.495 DEBUG:boto:Data: Enabled Disabled 2023-07-28T23:51:38.495 DEBUG:boto:Headers: {} 2023-07-28T23:51:38.495 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:38.496 DEBUG:boto:Port: 8000 2023-07-28T23:51:38.496 DEBUG:boto:Params: {} 2023-07-28T23:51:38.496 DEBUG:boto:Token: None 2023-07-28T23:51:38.497 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:51:38 GMT /hiiqos-14/?versioning 2023-07-28T23:51:38.497 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:R9tZVH21Y5ZusPSwpgq7iB6OhEA= 2023-07-28T23:51:38.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:R9tZVH21Y5ZusPSwpgq7iB6OhEA=', 'Content-Length': '228'} 2023-07-28T23:51:38.502 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001d0e351fab4b1ab4-0064c4548a-1117-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:38.503 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:51:38.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:51:38.504 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:51:38.549 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:51:38.549 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:51:38.580 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-28T23:51:38.593 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-28T23:51:38.593 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:51:38.593 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-28T23:51:38.593 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:46:09.233742Z" 2023-07-28T23:51:38.593 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:51:38.593 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:51:38.594 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588298.501536_195.1", 2023-07-28T23:51:38.594 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:51:38.501536Z" 2023-07-28T23:51:38.594 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:51:38.594 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:51:38.594 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:51:38.594 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:51:38.594 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:51:38.595 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:51:38.595 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:51:38.595 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:51:38.595 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:51:38.595 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-28T23:51:38.596 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690587969.233742_99.1', 1: '1_1690588298.501536_195.1', 2: '', 3: ''} 2023-07-28T23:51:38.596 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-28T23:51:38.597 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:51:38.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:51:38.598 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:51:38.642 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:51:38.642 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:51:38.670 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:51:38.669+0000 7fe453926880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:51:38.671 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:51:38.670+0000 7fe453926880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:51:38.683 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:51:38.683 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:51:38.683 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:51:38.683 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:51:38.683 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:51:38.683 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:51:38.684 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:51:38.684 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:51:38.684 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:51:38.684 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:51:38.684 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:51:38.684 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:51:38.685 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:51:38.685 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-28T23:51:38.685 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:51:38.685 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:51:38.685 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:51:38.685 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-28T23:51:38.685 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:51:38.686 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:51:38.686 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:51:38.686 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:51:38.686 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:51:38.686 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:51:38.686 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:51:38.687 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588298.501536_195.1", 2023-07-28T23:51:38.687 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:51:38.687 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:51:38.687 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:51:38.687 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:51:38.501536Z", 2023-07-28T23:51:38.687 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:51:38.687 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:51:38.688 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:51:38.688 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:51:38.688 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:51:38.688 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:51:38.688 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:51:38.688 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:51:38.688 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:51:38.689 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:51:38.689 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:51:38.689 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:51:38.689 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:51:38.689 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:51:38.689 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:51:38.689 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:51:38.690 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:51:38.690 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:51:38.690 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:51:38.690 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:51:38.690 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:51:38.690 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:51:38.690 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:51:38.691 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:51:38.691 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:51:38.691 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:51:38.691 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:51:38.691 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:51:38.691 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:51:38.691 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:51:38.691 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:51:38.692 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:51:38.692 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:51:38.692 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:51:38Z" 2023-07-28T23:51:38.692 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:51:38.693 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588298.501536_195.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:51:38.501536Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-28T23:51:38Z" } 2023-07-28T23:51:38.693 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588298.501536_195.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:51:38.501536Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-28T23:51:38.694 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588298.501536_195.1', 2: '', 3: ''} 2023-07-28T23:51:38.694 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588298.501536_195.1', 2: '', 3: ''} 2023-07-28T23:51:38.694 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-28T23:51:38.695 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:51:38.695 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:51:38.695 DEBUG:boto:Method: HEAD 2023-07-28T23:51:38.696 DEBUG:boto:Path: /hiiqos-14/ 2023-07-28T23:51:38.696 DEBUG:boto:Data: 2023-07-28T23:51:38.696 DEBUG:boto:Headers: {} 2023-07-28T23:51:38.697 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:38.697 DEBUG:boto:Port: 8000 2023-07-28T23:51:38.697 DEBUG:boto:Params: {} 2023-07-28T23:51:38.698 DEBUG:boto:Token: None 2023-07-28T23:51:38.698 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:51:38 GMT /hiiqos-14/ 2023-07-28T23:51:38.699 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5CQhA13KlWRiqsYH8S30wVxHXI0= 2023-07-28T23:51:38.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:51:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5CQhA13KlWRiqsYH8S30wVxHXI0=', 'Content-Length': '0'} 2023-07-28T23:51:38.701 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c490d12c6ce954d4-0064c4548a-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:38.702 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:51:38.702 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:51:38.702 DEBUG:boto:Method: PUT 2023-07-28T23:51:38.703 DEBUG:boto:Path: /hiiqos-14/obj 2023-07-28T23:51:38.703 DEBUG:boto:Data: 2023-07-28T23:51:38.703 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-07-28T23:51:38.704 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:51:38.704 DEBUG:boto:Port: 8000 2023-07-28T23:51:38.704 DEBUG:boto:Params: {} 2023-07-28T23:51:38.705 DEBUG:boto:Token: None 2023-07-28T23:51:38.705 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 28 Jul 2023 23:51:38 GMT /hiiqos-14/obj 2023-07-28T23:51:38.705 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IePaLsCWdRF4Zxzx69LZGIUveUM= 2023-07-28T23:51:38.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:51:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IePaLsCWdRF4Zxzx69LZGIUveUM='} 2023-07-28T23:51:38.713 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ISvc8k9pn.km8NePppXJwSK7pRcUJDI'), ('x-amz-request-id', 'tx00000ace819b6c04290cc-0064c4548a-1117-a1'), ('Date', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:51:38.714 DEBUG:rgw_multi.tests:created new version id=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:51:38.714 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-14 2023-07-28T23:51:38.715 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:51:38.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:51:38.715 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-28T23:51:38.769 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:51:38.770 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:51:38.801 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:51:38.800+0000 7f9d34d4b880 1 waiting for incremental sync to catch up: 2023-07-28T23:51:38.802 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , 00000000001.29.6, , ] 2023-07-28T23:51:38.802 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000003.32.13, , ] 2023-07-28T23:51:43.803 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:51:43.801+0000 7f9d34d4b880 1 waiting for incremental sync to catch up: 2023-07-28T23:51:43.803 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , 00000000001.29.6, , ] 2023-07-28T23:51:43.804 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000003.32.13, , ] 2023-07-28T23:51:48.804 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:51:48.803+0000 7f9d34d4b880 1 waiting for incremental sync to catch up: 2023-07-28T23:51:48.805 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , 00000000001.29.6, , ] 2023-07-28T23:51:48.805 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000003.32.13, , ] 2023-07-28T23:51:53.804 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:51:53.803+0000 7f9d34d4b880 1 waiting for incremental sync to catch up: 2023-07-28T23:51:53.805 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , 00000000001.29.6, , ] 2023-07-28T23:51:53.805 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000003.32.13, , ] 2023-07-28T23:51:58.806 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:51:58.804+0000 7f9d34d4b880 1 waiting for incremental sync to catch up: 2023-07-28T23:51:58.806 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , 00000000001.29.6, , ] 2023-07-28T23:51:58.806 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000003.32.13, , ] 2023-07-28T23:52:03.807 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:52:03.805+0000 7f9d34d4b880 1 waiting for incremental sync to catch up: 2023-07-28T23:52:03.808 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , 00000000001.29.6, , ] 2023-07-28T23:52:03.808 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000003.32.13, , ] 2023-07-28T23:52:08.808 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:52:08.807+0000 7f9d34d4b880 1 waiting for incremental sync to catch up: 2023-07-28T23:52:08.809 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , 00000000001.29.6, , ] 2023-07-28T23:52:08.809 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000003.32.13, , ] 2023-07-28T23:52:13.809 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:52:13.808+0000 7f9d34d4b880 1 waiting for incremental sync to catch up: 2023-07-28T23:52:13.810 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , 00000000001.29.6, , ] 2023-07-28T23:52:13.810 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000003.32.13, , ] 2023-07-28T23:52:18.811 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:52:18.809+0000 7f9d34d4b880 1 bucket sync caught up with source: 2023-07-28T23:52:18.811 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , 00000000003.32.13, , ] 2023-07-28T23:52:18.811 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000003.32.13, , ] 2023-07-28T23:52:18.811 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:52:18.809+0000 7f9d34d4b880 0 bucket checkpoint complete 2023-07-28T23:52:18.827 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-14 2023-07-28T23:52:18.828 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:52:18.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:52:18.828 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-28T23:52:18.886 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:52:18.886 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:52:18.923 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:52:18.921+0000 7f57da620880 1 waiting for incremental sync to catch up: 2023-07-28T23:52:18.923 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , 00000000001.17.6, , ] 2023-07-28T23:52:18.923 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000003.21.13, , ] 2023-07-28T23:52:23.924 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:52:23.922+0000 7f57da620880 1 waiting for incremental sync to catch up: 2023-07-28T23:52:23.924 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , 00000000001.17.6, , ] 2023-07-28T23:52:23.924 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000003.21.13, , ] 2023-07-28T23:52:28.925 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:52:28.923+0000 7f57da620880 1 waiting for incremental sync to catch up: 2023-07-28T23:52:28.926 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , 00000000001.17.6, , ] 2023-07-28T23:52:28.926 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000003.21.13, , ] 2023-07-28T23:52:33.926 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:52:33.924+0000 7f57da620880 1 waiting for incremental sync to catch up: 2023-07-28T23:52:33.927 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , 00000000001.17.6, , ] 2023-07-28T23:52:33.927 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000003.21.13, , ] 2023-07-28T23:52:38.928 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:52:38.926+0000 7f57da620880 1 bucket sync caught up with source: 2023-07-28T23:52:38.928 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , 00000000003.21.13, , ] 2023-07-28T23:52:38.928 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000003.21.13, , ] 2023-07-28T23:52:38.928 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:52:38.926+0000 7f57da620880 0 bucket checkpoint complete 2023-07-28T23:52:38.944 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-14 zones={a2, a1} 2023-07-28T23:52:38.944 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:52:38.945 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:52:38.945 DEBUG:boto:Method: HEAD 2023-07-28T23:52:38.945 DEBUG:boto:Path: /hiiqos-14/ 2023-07-28T23:52:38.946 DEBUG:boto:Data: 2023-07-28T23:52:38.946 DEBUG:boto:Headers: {} 2023-07-28T23:52:38.946 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:52:38.947 DEBUG:boto:Port: 8001 2023-07-28T23:52:38.947 DEBUG:boto:Params: {} 2023-07-28T23:52:38.948 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-28T23:52:38.948 DEBUG:boto:Token: None 2023-07-28T23:52:38.949 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:52:38 GMT /hiiqos-14/ 2023-07-28T23:52:38.949 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:t16KiGzjjUF19j+zvaqvc/NWeuY= 2023-07-28T23:52:38.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:52:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t16KiGzjjUF19j+zvaqvc/NWeuY=', 'Content-Length': '0'} 2023-07-28T23:52:38.953 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c60653927ab24e2-0064c454c6-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:52:38.954 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:52:38.954 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:52:38.955 DEBUG:boto:Method: HEAD 2023-07-28T23:52:38.955 DEBUG:boto:Path: /hiiqos-14/ 2023-07-28T23:52:38.955 DEBUG:boto:Data: 2023-07-28T23:52:38.956 DEBUG:boto:Headers: {} 2023-07-28T23:52:38.956 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:52:38.957 DEBUG:boto:Port: 8000 2023-07-28T23:52:38.957 DEBUG:boto:Params: {} 2023-07-28T23:52:38.957 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-28T23:52:38.958 DEBUG:boto:Token: None 2023-07-28T23:52:38.958 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:52:38 GMT /hiiqos-14/ 2023-07-28T23:52:38.959 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:t16KiGzjjUF19j+zvaqvc/NWeuY= 2023-07-28T23:52:38.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:52:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t16KiGzjjUF19j+zvaqvc/NWeuY=', 'Content-Length': '0'} 2023-07-28T23:52:38.962 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e07497248b122a3-0064c454c6-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:52:38.962 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-28T23:52:38.963 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:52:38.963 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:52:38.964 DEBUG:boto:path=/hiiqos-14/?versions&max-keys=999 2023-07-28T23:52:38.964 DEBUG:boto:auth_path=/hiiqos-14/?versions&max-keys=999 2023-07-28T23:52:38.965 DEBUG:boto:Method: GET 2023-07-28T23:52:38.965 DEBUG:boto:Path: /hiiqos-14/?versions&max-keys=999 2023-07-28T23:52:38.965 DEBUG:boto:Data: 2023-07-28T23:52:38.966 DEBUG:boto:Headers: {} 2023-07-28T23:52:38.966 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:52:38.966 DEBUG:boto:Port: 8001 2023-07-28T23:52:38.967 DEBUG:boto:Params: {} 2023-07-28T23:52:38.967 DEBUG:boto:Token: None 2023-07-28T23:52:38.968 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:52:38 GMT /hiiqos-14/?versions 2023-07-28T23:52:38.968 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WES21Etos1yrZoYNAT6S1ILhVbc= 2023-07-28T23:52:38.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:52:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WES21Etos1yrZoYNAT6S1ILhVbc=', 'Content-Length': '0'} 2023-07-28T23:52:38.971 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000143d99a05f6be04e-0064c454c6-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:52:38.972 DEBUG:boto:b'hiiqos-14999falseobjISvc8k9pn.km8NePppXJwSK7pRcUJDItrue2023-07-28T23:51:38.707Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-07-28T23:51:23.083Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:52:38.973 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:52:38.973 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:52:38.974 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-28T23:52:38.974 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:52:38.975 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:52:38.975 DEBUG:boto:path=/hiiqos-14/?versions&max-keys=999 2023-07-28T23:52:38.975 DEBUG:boto:auth_path=/hiiqos-14/?versions&max-keys=999 2023-07-28T23:52:38.976 DEBUG:boto:Method: GET 2023-07-28T23:52:38.976 DEBUG:boto:Path: /hiiqos-14/?versions&max-keys=999 2023-07-28T23:52:38.977 DEBUG:boto:Data: 2023-07-28T23:52:38.977 DEBUG:boto:Headers: {} 2023-07-28T23:52:38.977 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:52:38.978 DEBUG:boto:Port: 8000 2023-07-28T23:52:38.978 DEBUG:boto:Params: {} 2023-07-28T23:52:38.978 DEBUG:boto:Token: None 2023-07-28T23:52:38.979 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:52:38 GMT /hiiqos-14/?versions 2023-07-28T23:52:38.979 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WES21Etos1yrZoYNAT6S1ILhVbc= 2023-07-28T23:52:38.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:52:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WES21Etos1yrZoYNAT6S1ILhVbc=', 'Content-Length': '0'} 2023-07-28T23:52:38.982 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0f8acef0863a083-0064c454c6-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:52:38.982 DEBUG:boto:b'hiiqos-14999falseobjISvc8k9pn.km8NePppXJwSK7pRcUJDItrue2023-07-28T23:51:38.707Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-07-28T23:51:23.083Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:52:38.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:52:38.984 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:52:38.984 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:52:38.985 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:52:38.985 DEBUG:boto:path=/hiiqos-14/?versions&max-keys=999 2023-07-28T23:52:38.986 DEBUG:boto:auth_path=/hiiqos-14/?versions&max-keys=999 2023-07-28T23:52:38.986 DEBUG:boto:Method: GET 2023-07-28T23:52:38.986 DEBUG:boto:Path: /hiiqos-14/?versions&max-keys=999 2023-07-28T23:52:38.987 DEBUG:boto:Data: 2023-07-28T23:52:38.987 DEBUG:boto:Headers: {} 2023-07-28T23:52:38.988 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:52:38.988 DEBUG:boto:Port: 8001 2023-07-28T23:52:38.988 DEBUG:boto:Params: {} 2023-07-28T23:52:38.989 DEBUG:boto:Token: None 2023-07-28T23:52:38.989 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:52:38 GMT /hiiqos-14/?versions 2023-07-28T23:52:38.990 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WES21Etos1yrZoYNAT6S1ILhVbc= 2023-07-28T23:52:38.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:52:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WES21Etos1yrZoYNAT6S1ILhVbc=', 'Content-Length': '0'} 2023-07-28T23:52:38.992 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9fc4f633f91572a-0064c454c6-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:52:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:52:38.993 DEBUG:boto:b'hiiqos-14999falseobjISvc8k9pn.km8NePppXJwSK7pRcUJDItrue2023-07-28T23:51:38.707Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-07-28T23:51:23.083Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:52:38.994 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:52:38.994 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:52:38.994 DEBUG:boto:path=/hiiqos-14/?versions&max-keys=999 2023-07-28T23:52:38.995 DEBUG:boto:auth_path=/hiiqos-14/?versions&max-keys=999 2023-07-28T23:52:38.995 DEBUG:boto:Method: GET 2023-07-28T23:52:38.996 DEBUG:boto:Path: /hiiqos-14/?versions&max-keys=999 2023-07-28T23:52:38.996 DEBUG:boto:Data: 2023-07-28T23:52:38.996 DEBUG:boto:Headers: {} 2023-07-28T23:52:38.997 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:52:38.997 DEBUG:boto:Port: 8000 2023-07-28T23:52:38.998 DEBUG:boto:Params: {} 2023-07-28T23:52:38.998 DEBUG:boto:Token: None 2023-07-28T23:52:38.998 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:52:38 GMT /hiiqos-14/?versions 2023-07-28T23:52:38.999 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WES21Etos1yrZoYNAT6S1ILhVbc= 2023-07-28T23:52:38.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:52:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WES21Etos1yrZoYNAT6S1ILhVbc=', 'Content-Length': '0'} 2023-07-28T23:52:39.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070192b1a4856c063-0064c454c6-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:52:39.002 DEBUG:boto:b'hiiqos-14999falseobjISvc8k9pn.km8NePppXJwSK7pRcUJDItrue2023-07-28T23:51:38.707Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-07-28T23:51:23.083Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:52:39.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:52:39.003 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:52:39.004 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:52:39.004 DEBUG:boto:path=/hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:52:39.005 DEBUG:boto:auth_path=/hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:52:39.005 DEBUG:boto:Method: GET 2023-07-28T23:52:39.006 DEBUG:boto:Path: /hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:52:39.006 DEBUG:boto:Data: 2023-07-28T23:52:39.006 DEBUG:boto:Headers: {} 2023-07-28T23:52:39.007 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:52:39.007 DEBUG:boto:Port: 8001 2023-07-28T23:52:39.007 DEBUG:boto:Params: {} 2023-07-28T23:52:39.008 DEBUG:boto:Token: None 2023-07-28T23:52:39.008 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:52:39 GMT /hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:52:39.009 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XVH/X80NdZl8QbVpmZGyb/eSxNw= 2023-07-28T23:52:39.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:52:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XVH/X80NdZl8QbVpmZGyb/eSxNw=', 'Content-Length': '0'} 2023-07-28T23:52:39.011 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('x-amz-version-id', 'ISvc8k9pn.km8NePppXJwSK7pRcUJDI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002eaa190207f7f3bd-0064c454c7-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:52:39.012 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:52:39.012 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:52:39.012 DEBUG:boto:path=/hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:52:39.013 DEBUG:boto:auth_path=/hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:52:39.013 DEBUG:boto:Method: GET 2023-07-28T23:52:39.014 DEBUG:boto:Path: /hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:52:39.014 DEBUG:boto:Data: 2023-07-28T23:52:39.015 DEBUG:boto:Headers: {} 2023-07-28T23:52:39.015 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:52:39.016 DEBUG:boto:Port: 8000 2023-07-28T23:52:39.016 DEBUG:boto:Params: {} 2023-07-28T23:52:39.016 DEBUG:boto:Token: None 2023-07-28T23:52:39.017 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:52:39 GMT /hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:52:39.017 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XVH/X80NdZl8QbVpmZGyb/eSxNw= 2023-07-28T23:52:39.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:52:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XVH/X80NdZl8QbVpmZGyb/eSxNw=', 'Content-Length': '0'} 2023-07-28T23:52:39.019 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('x-amz-version-id', 'ISvc8k9pn.km8NePppXJwSK7pRcUJDI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008803924fa4e95089-0064c454c7-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:52:39.020 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:52:39.021 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:52:39.021 DEBUG:boto:path=/hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:52:39.022 DEBUG:boto:auth_path=/hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:52:39.022 DEBUG:boto:Method: HEAD 2023-07-28T23:52:39.022 DEBUG:boto:Path: /hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:52:39.023 DEBUG:boto:Data: 2023-07-28T23:52:39.023 DEBUG:boto:Headers: {} 2023-07-28T23:52:39.023 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:52:39.024 DEBUG:boto:Port: 8001 2023-07-28T23:52:39.024 DEBUG:boto:Params: {} 2023-07-28T23:52:39.025 DEBUG:boto:Token: None 2023-07-28T23:52:39.025 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:52:39 GMT /hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:52:39.025 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EBdzuP8bWOlbO1LWscz+yGjdpls= 2023-07-28T23:52:39.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:52:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EBdzuP8bWOlbO1LWscz+yGjdpls=', 'Content-Length': '0'} 2023-07-28T23:52:39.028 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('x-amz-version-id', 'ISvc8k9pn.km8NePppXJwSK7pRcUJDI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004034c43c8facf771-0064c454c7-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:52:39.028 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:52:39.029 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:52:39.029 DEBUG:boto:path=/hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:52:39.029 DEBUG:boto:auth_path=/hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:52:39.030 DEBUG:boto:Method: HEAD 2023-07-28T23:52:39.030 DEBUG:boto:Path: /hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:52:39.030 DEBUG:boto:Data: 2023-07-28T23:52:39.031 DEBUG:boto:Headers: {} 2023-07-28T23:52:39.031 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:52:39.031 DEBUG:boto:Port: 8000 2023-07-28T23:52:39.032 DEBUG:boto:Params: {} 2023-07-28T23:52:39.032 DEBUG:boto:Token: None 2023-07-28T23:52:39.033 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:52:39 GMT /hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:52:39.033 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EBdzuP8bWOlbO1LWscz+yGjdpls= 2023-07-28T23:52:39.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:52:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EBdzuP8bWOlbO1LWscz+yGjdpls=', 'Content-Length': '0'} 2023-07-28T23:52:39.035 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('x-amz-version-id', 'ISvc8k9pn.km8NePppXJwSK7pRcUJDI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e8a132e5bba7e797-0064c454c7-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:52:39.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:52:39.037 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:52:39.037 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:52:39.037 DEBUG:boto:path=/hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:52:39.038 DEBUG:boto:auth_path=/hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:52:39.038 DEBUG:boto:Method: GET 2023-07-28T23:52:39.039 DEBUG:boto:Path: /hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:52:39.039 DEBUG:boto:Data: 2023-07-28T23:52:39.040 DEBUG:boto:Headers: {} 2023-07-28T23:52:39.040 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:52:39.041 DEBUG:boto:Port: 8001 2023-07-28T23:52:39.041 DEBUG:boto:Params: {} 2023-07-28T23:52:39.042 DEBUG:boto:Token: None 2023-07-28T23:52:39.042 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:52:39 GMT /hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:52:39.043 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XVH/X80NdZl8QbVpmZGyb/eSxNw= 2023-07-28T23:52:39.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:52:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XVH/X80NdZl8QbVpmZGyb/eSxNw=', 'Content-Length': '0'} 2023-07-28T23:52:39.045 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('x-amz-version-id', 'ISvc8k9pn.km8NePppXJwSK7pRcUJDI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fe107453dc4c7305-0064c454c7-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:52:39.046 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:52:39.046 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:52:39.046 DEBUG:boto:path=/hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:52:39.047 DEBUG:boto:auth_path=/hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:52:39.047 DEBUG:boto:Method: GET 2023-07-28T23:52:39.047 DEBUG:boto:Path: /hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:52:39.048 DEBUG:boto:Data: 2023-07-28T23:52:39.048 DEBUG:boto:Headers: {} 2023-07-28T23:52:39.049 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:52:39.049 DEBUG:boto:Port: 8000 2023-07-28T23:52:39.050 DEBUG:boto:Params: {} 2023-07-28T23:52:39.050 DEBUG:boto:Token: None 2023-07-28T23:52:39.050 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:52:39 GMT /hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:52:39.051 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XVH/X80NdZl8QbVpmZGyb/eSxNw= 2023-07-28T23:52:39.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:52:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XVH/X80NdZl8QbVpmZGyb/eSxNw=', 'Content-Length': '0'} 2023-07-28T23:52:39.053 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('x-amz-version-id', 'ISvc8k9pn.km8NePppXJwSK7pRcUJDI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ce2fbc4507271eed-0064c454c7-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:52:39.054 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:52:39.054 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:52:39.055 DEBUG:boto:path=/hiiqos-14/?prefix=obj 2023-07-28T23:52:39.055 DEBUG:boto:auth_path=/hiiqos-14/?prefix=obj 2023-07-28T23:52:39.055 DEBUG:boto:Method: GET 2023-07-28T23:52:39.056 DEBUG:boto:Path: /hiiqos-14/?prefix=obj 2023-07-28T23:52:39.056 DEBUG:boto:Data: 2023-07-28T23:52:39.057 DEBUG:boto:Headers: {} 2023-07-28T23:52:39.057 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:52:39.057 DEBUG:boto:Port: 8001 2023-07-28T23:52:39.058 DEBUG:boto:Params: {} 2023-07-28T23:52:39.058 DEBUG:boto:Token: None 2023-07-28T23:52:39.059 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:52:39 GMT /hiiqos-14/ 2023-07-28T23:52:39.059 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Tz/282uwjaP4tGvewXmqLevpn+A= 2023-07-28T23:52:39.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:52:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Tz/282uwjaP4tGvewXmqLevpn+A=', 'Content-Length': '0'} 2023-07-28T23:52:39.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef98c466eead42ac-0064c454c7-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:52:39.104 DEBUG:boto:b'hiiqos-14obj1000falseobj2023-07-28T23:51:38.707Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:52:39.105 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:52:39.105 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:52:39.106 DEBUG:boto:path=/hiiqos-14/?prefix=obj 2023-07-28T23:52:39.106 DEBUG:boto:auth_path=/hiiqos-14/?prefix=obj 2023-07-28T23:52:39.106 DEBUG:boto:Method: GET 2023-07-28T23:52:39.107 DEBUG:boto:Path: /hiiqos-14/?prefix=obj 2023-07-28T23:52:39.107 DEBUG:boto:Data: 2023-07-28T23:52:39.108 DEBUG:boto:Headers: {} 2023-07-28T23:52:39.108 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:52:39.108 DEBUG:boto:Port: 8000 2023-07-28T23:52:39.109 DEBUG:boto:Params: {} 2023-07-28T23:52:39.109 DEBUG:boto:Token: None 2023-07-28T23:52:39.110 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:52:39 GMT /hiiqos-14/ 2023-07-28T23:52:39.110 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Tz/282uwjaP4tGvewXmqLevpn+A= 2023-07-28T23:52:39.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:52:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Tz/282uwjaP4tGvewXmqLevpn+A=', 'Content-Length': '0'} 2023-07-28T23:52:39.113 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb133fd4ed3d3fa6-0064c454c7-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:52:39.113 DEBUG:boto:b'hiiqos-14obj1000falseobj2023-07-28T23:51:38.707Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:52:39.114 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:52:39.115 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:52:39.115 DEBUG:boto:Method: HEAD 2023-07-28T23:52:39.116 DEBUG:boto:Path: /hiiqos-14/obj 2023-07-28T23:52:39.116 DEBUG:boto:Data: 2023-07-28T23:52:39.116 DEBUG:boto:Headers: {} 2023-07-28T23:52:39.117 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:52:39.117 DEBUG:boto:Port: 8001 2023-07-28T23:52:39.118 DEBUG:boto:Params: {} 2023-07-28T23:52:39.118 DEBUG:boto:Token: None 2023-07-28T23:52:39.118 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:52:39 GMT /hiiqos-14/obj 2023-07-28T23:52:39.119 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ADnA79V1Y+1aR+EwKKuFQpaVCR4= 2023-07-28T23:52:39.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:52:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ADnA79V1Y+1aR+EwKKuFQpaVCR4=', 'Content-Length': '0'} 2023-07-28T23:52:39.122 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('x-amz-version-id', 'ISvc8k9pn.km8NePppXJwSK7pRcUJDI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a34ee9881652ce03-0064c454c7-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:52:39.122 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:52:39.123 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:52:39.123 DEBUG:boto:Method: HEAD 2023-07-28T23:52:39.123 DEBUG:boto:Path: /hiiqos-14/obj 2023-07-28T23:52:39.124 DEBUG:boto:Data: 2023-07-28T23:52:39.124 DEBUG:boto:Headers: {} 2023-07-28T23:52:39.125 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:52:39.125 DEBUG:boto:Port: 8000 2023-07-28T23:52:39.125 DEBUG:boto:Params: {} 2023-07-28T23:52:39.126 DEBUG:boto:Token: None 2023-07-28T23:52:39.126 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:52:39 GMT /hiiqos-14/obj 2023-07-28T23:52:39.126 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ADnA79V1Y+1aR+EwKKuFQpaVCR4= 2023-07-28T23:52:39.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:52:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ADnA79V1Y+1aR+EwKKuFQpaVCR4=', 'Content-Length': '0'} 2023-07-28T23:52:39.129 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('x-amz-version-id', 'ISvc8k9pn.km8NePppXJwSK7pRcUJDI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000467551736698f4c1-0064c454c7-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:52:39.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:52:39.130 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:52:39.130 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:52:39.131 DEBUG:boto:path=/hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:52:39.131 DEBUG:boto:auth_path=/hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:52:39.132 DEBUG:boto:Method: GET 2023-07-28T23:52:39.132 DEBUG:boto:Path: /hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:52:39.132 DEBUG:boto:Data: 2023-07-28T23:52:39.133 DEBUG:boto:Headers: {} 2023-07-28T23:52:39.133 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:52:39.134 DEBUG:boto:Port: 8001 2023-07-28T23:52:39.134 DEBUG:boto:Params: {} 2023-07-28T23:52:39.135 DEBUG:boto:Token: None 2023-07-28T23:52:39.135 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:52:39 GMT /hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:52:39.136 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XVH/X80NdZl8QbVpmZGyb/eSxNw= 2023-07-28T23:52:39.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:52:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XVH/X80NdZl8QbVpmZGyb/eSxNw=', 'Content-Length': '0'} 2023-07-28T23:52:39.138 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('x-amz-version-id', 'ISvc8k9pn.km8NePppXJwSK7pRcUJDI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000b79a98792eba516-0064c454c7-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:52:39.139 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:52:39.139 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:52:39.139 DEBUG:boto:path=/hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:52:39.140 DEBUG:boto:auth_path=/hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:52:39.140 DEBUG:boto:Method: GET 2023-07-28T23:52:39.141 DEBUG:boto:Path: /hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:52:39.141 DEBUG:boto:Data: 2023-07-28T23:52:39.142 DEBUG:boto:Headers: {} 2023-07-28T23:52:39.142 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:52:39.143 DEBUG:boto:Port: 8000 2023-07-28T23:52:39.143 DEBUG:boto:Params: {} 2023-07-28T23:52:39.144 DEBUG:boto:Token: None 2023-07-28T23:52:39.144 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:52:39 GMT /hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:52:39.145 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XVH/X80NdZl8QbVpmZGyb/eSxNw= 2023-07-28T23:52:39.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:52:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XVH/X80NdZl8QbVpmZGyb/eSxNw=', 'Content-Length': '0'} 2023-07-28T23:52:39.147 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('x-amz-version-id', 'ISvc8k9pn.km8NePppXJwSK7pRcUJDI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007cef61f11c52acb4-0064c454c7-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:52:39.147 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:52:39.148 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:52:39.148 DEBUG:boto:path=/hiiqos-14/?prefix=obj 2023-07-28T23:52:39.149 DEBUG:boto:auth_path=/hiiqos-14/?prefix=obj 2023-07-28T23:52:39.149 DEBUG:boto:Method: GET 2023-07-28T23:52:39.149 DEBUG:boto:Path: /hiiqos-14/?prefix=obj 2023-07-28T23:52:39.150 DEBUG:boto:Data: 2023-07-28T23:52:39.150 DEBUG:boto:Headers: {} 2023-07-28T23:52:39.151 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:52:39.151 DEBUG:boto:Port: 8001 2023-07-28T23:52:39.152 DEBUG:boto:Params: {} 2023-07-28T23:52:39.152 DEBUG:boto:Token: None 2023-07-28T23:52:39.152 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:52:39 GMT /hiiqos-14/ 2023-07-28T23:52:39.153 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Tz/282uwjaP4tGvewXmqLevpn+A= 2023-07-28T23:52:39.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:52:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Tz/282uwjaP4tGvewXmqLevpn+A=', 'Content-Length': '0'} 2023-07-28T23:52:39.156 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1f3c9d850c537de-0064c454c7-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:52:39.156 DEBUG:boto:b'hiiqos-14obj1000falseobj2023-07-28T23:51:38.707Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:52:39.157 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:52:39.158 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:52:39.158 DEBUG:boto:path=/hiiqos-14/?prefix=obj 2023-07-28T23:52:39.158 DEBUG:boto:auth_path=/hiiqos-14/?prefix=obj 2023-07-28T23:52:39.159 DEBUG:boto:Method: GET 2023-07-28T23:52:39.159 DEBUG:boto:Path: /hiiqos-14/?prefix=obj 2023-07-28T23:52:39.159 DEBUG:boto:Data: 2023-07-28T23:52:39.160 DEBUG:boto:Headers: {} 2023-07-28T23:52:39.160 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:52:39.161 DEBUG:boto:Port: 8000 2023-07-28T23:52:39.161 DEBUG:boto:Params: {} 2023-07-28T23:52:39.161 DEBUG:boto:Token: None 2023-07-28T23:52:39.162 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:52:39 GMT /hiiqos-14/ 2023-07-28T23:52:39.162 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Tz/282uwjaP4tGvewXmqLevpn+A= 2023-07-28T23:52:39.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:52:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Tz/282uwjaP4tGvewXmqLevpn+A=', 'Content-Length': '0'} 2023-07-28T23:52:39.165 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067b34521b633d2cb-0064c454c7-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:52:39.166 DEBUG:boto:b'hiiqos-14obj1000falseobj2023-07-28T23:51:38.707Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:52:39.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:52:39.167 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:52:39.167 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:52:39.167 DEBUG:boto:path=/hiiqos-14/obj?versionId=null 2023-07-28T23:52:39.168 DEBUG:boto:auth_path=/hiiqos-14/obj?versionId=null 2023-07-28T23:52:39.168 DEBUG:boto:Method: GET 2023-07-28T23:52:39.169 DEBUG:boto:Path: /hiiqos-14/obj?versionId=null 2023-07-28T23:52:39.169 DEBUG:boto:Data: 2023-07-28T23:52:39.170 DEBUG:boto:Headers: {} 2023-07-28T23:52:39.170 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:52:39.171 DEBUG:boto:Port: 8001 2023-07-28T23:52:39.171 DEBUG:boto:Params: {} 2023-07-28T23:52:39.171 DEBUG:boto:Token: None 2023-07-28T23:52:39.172 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:52:39 GMT /hiiqos-14/obj?versionId=null 2023-07-28T23:52:39.172 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8MCL/Bc5dm9+pxm7kSxuLUITlIk= 2023-07-28T23:52:39.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:52:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8MCL/Bc5dm9+pxm7kSxuLUITlIk=', 'Content-Length': '0'} 2023-07-28T23:52:39.175 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:51:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002384e91c47b55dd2-0064c454c7-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:52:39.175 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:52:39.176 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:52:39.176 DEBUG:boto:path=/hiiqos-14/obj?versionId=null 2023-07-28T23:52:39.177 DEBUG:boto:auth_path=/hiiqos-14/obj?versionId=null 2023-07-28T23:52:39.177 DEBUG:boto:Method: GET 2023-07-28T23:52:39.177 DEBUG:boto:Path: /hiiqos-14/obj?versionId=null 2023-07-28T23:52:39.178 DEBUG:boto:Data: 2023-07-28T23:52:39.178 DEBUG:boto:Headers: {} 2023-07-28T23:52:39.178 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:52:39.179 DEBUG:boto:Port: 8000 2023-07-28T23:52:39.179 DEBUG:boto:Params: {} 2023-07-28T23:52:39.180 DEBUG:boto:Token: None 2023-07-28T23:52:39.180 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:52:39 GMT /hiiqos-14/obj?versionId=null 2023-07-28T23:52:39.181 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8MCL/Bc5dm9+pxm7kSxuLUITlIk= 2023-07-28T23:52:39.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:52:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8MCL/Bc5dm9+pxm7kSxuLUITlIk=', 'Content-Length': '0'} 2023-07-28T23:52:39.183 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:51:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000a7f4144272ef612-0064c454c7-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:52:39.184 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:52:39.184 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:52:39.185 DEBUG:boto:path=/hiiqos-14/obj?versionId=null 2023-07-28T23:52:39.185 DEBUG:boto:auth_path=/hiiqos-14/obj?versionId=null 2023-07-28T23:52:39.185 DEBUG:boto:Method: HEAD 2023-07-28T23:52:39.186 DEBUG:boto:Path: /hiiqos-14/obj?versionId=null 2023-07-28T23:52:39.186 DEBUG:boto:Data: 2023-07-28T23:52:39.187 DEBUG:boto:Headers: {} 2023-07-28T23:52:39.187 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:52:39.187 DEBUG:boto:Port: 8001 2023-07-28T23:52:39.188 DEBUG:boto:Params: {} 2023-07-28T23:52:39.188 DEBUG:boto:Token: None 2023-07-28T23:52:39.189 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:52:39 GMT /hiiqos-14/obj?versionId=null 2023-07-28T23:52:39.189 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7ytta9pg9/S2j1cLqy/waDKT754= 2023-07-28T23:52:39.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:52:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7ytta9pg9/S2j1cLqy/waDKT754=', 'Content-Length': '0'} 2023-07-28T23:52:39.192 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:51:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003edc93398b1a008b-0064c454c7-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:52:39.193 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:52:39.194 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:52:39.194 DEBUG:boto:path=/hiiqos-14/obj?versionId=null 2023-07-28T23:52:39.195 DEBUG:boto:auth_path=/hiiqos-14/obj?versionId=null 2023-07-28T23:52:39.195 DEBUG:boto:Method: HEAD 2023-07-28T23:52:39.196 DEBUG:boto:Path: /hiiqos-14/obj?versionId=null 2023-07-28T23:52:39.197 DEBUG:boto:Data: 2023-07-28T23:52:39.197 DEBUG:boto:Headers: {} 2023-07-28T23:52:39.198 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:52:39.199 DEBUG:boto:Port: 8000 2023-07-28T23:52:39.199 DEBUG:boto:Params: {} 2023-07-28T23:52:39.200 DEBUG:boto:Token: None 2023-07-28T23:52:39.201 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:52:39 GMT /hiiqos-14/obj?versionId=null 2023-07-28T23:52:39.201 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7ytta9pg9/S2j1cLqy/waDKT754= 2023-07-28T23:52:39.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:52:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7ytta9pg9/S2j1cLqy/waDKT754=', 'Content-Length': '0'} 2023-07-28T23:52:39.204 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:51:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009b200e06be740daf-0064c454c7-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:52:39.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:52:39.205 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:52:39.206 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:52:39.206 DEBUG:boto:path=/hiiqos-14/obj?versionId=null 2023-07-28T23:52:39.206 DEBUG:boto:auth_path=/hiiqos-14/obj?versionId=null 2023-07-28T23:52:39.207 DEBUG:boto:Method: GET 2023-07-28T23:52:39.207 DEBUG:boto:Path: /hiiqos-14/obj?versionId=null 2023-07-28T23:52:39.207 DEBUG:boto:Data: 2023-07-28T23:52:39.208 DEBUG:boto:Headers: {} 2023-07-28T23:52:39.208 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:52:39.209 DEBUG:boto:Port: 8001 2023-07-28T23:52:39.209 DEBUG:boto:Params: {} 2023-07-28T23:52:39.209 DEBUG:boto:Token: None 2023-07-28T23:52:39.210 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:52:39 GMT /hiiqos-14/obj?versionId=null 2023-07-28T23:52:39.210 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8MCL/Bc5dm9+pxm7kSxuLUITlIk= 2023-07-28T23:52:39.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:52:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8MCL/Bc5dm9+pxm7kSxuLUITlIk=', 'Content-Length': '0'} 2023-07-28T23:52:39.212 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:51:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000902155dcb895069e-0064c454c7-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:52:39.213 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:52:39.213 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:52:39.214 DEBUG:boto:path=/hiiqos-14/obj?versionId=null 2023-07-28T23:52:39.214 DEBUG:boto:auth_path=/hiiqos-14/obj?versionId=null 2023-07-28T23:52:39.215 DEBUG:boto:Method: GET 2023-07-28T23:52:39.215 DEBUG:boto:Path: /hiiqos-14/obj?versionId=null 2023-07-28T23:52:39.216 DEBUG:boto:Data: 2023-07-28T23:52:39.216 DEBUG:boto:Headers: {} 2023-07-28T23:52:39.217 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:52:39.217 DEBUG:boto:Port: 8000 2023-07-28T23:52:39.218 DEBUG:boto:Params: {} 2023-07-28T23:52:39.218 DEBUG:boto:Token: None 2023-07-28T23:52:39.218 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:52:39 GMT /hiiqos-14/obj?versionId=null 2023-07-28T23:52:39.219 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8MCL/Bc5dm9+pxm7kSxuLUITlIk= 2023-07-28T23:52:39.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:52:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8MCL/Bc5dm9+pxm7kSxuLUITlIk=', 'Content-Length': '0'} 2023-07-28T23:52:39.221 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:51:23 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001326ad0f53bd4a46-0064c454c7-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:52:39.222 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:52:39.222 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:52:39.223 DEBUG:boto:path=/hiiqos-14/?prefix=obj 2023-07-28T23:52:39.223 DEBUG:boto:auth_path=/hiiqos-14/?prefix=obj 2023-07-28T23:52:39.223 DEBUG:boto:Method: GET 2023-07-28T23:52:39.224 DEBUG:boto:Path: /hiiqos-14/?prefix=obj 2023-07-28T23:52:39.224 DEBUG:boto:Data: 2023-07-28T23:52:39.225 DEBUG:boto:Headers: {} 2023-07-28T23:52:39.225 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:52:39.225 DEBUG:boto:Port: 8001 2023-07-28T23:52:39.226 DEBUG:boto:Params: {} 2023-07-28T23:52:39.226 DEBUG:boto:Token: None 2023-07-28T23:52:39.226 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:52:39 GMT /hiiqos-14/ 2023-07-28T23:52:39.227 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Tz/282uwjaP4tGvewXmqLevpn+A= 2023-07-28T23:52:39.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:52:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Tz/282uwjaP4tGvewXmqLevpn+A=', 'Content-Length': '0'} 2023-07-28T23:52:39.230 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f51e0f1d3c833e9-0064c454c7-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:52:39.230 DEBUG:boto:b'hiiqos-14obj1000falseobj2023-07-28T23:51:38.707Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:52:39.231 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:52:39.231 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:52:39.232 DEBUG:boto:path=/hiiqos-14/?prefix=obj 2023-07-28T23:52:39.232 DEBUG:boto:auth_path=/hiiqos-14/?prefix=obj 2023-07-28T23:52:39.233 DEBUG:boto:Method: GET 2023-07-28T23:52:39.233 DEBUG:boto:Path: /hiiqos-14/?prefix=obj 2023-07-28T23:52:39.233 DEBUG:boto:Data: 2023-07-28T23:52:39.234 DEBUG:boto:Headers: {} 2023-07-28T23:52:39.234 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:52:39.234 DEBUG:boto:Port: 8000 2023-07-28T23:52:39.235 DEBUG:boto:Params: {} 2023-07-28T23:52:39.235 DEBUG:boto:Token: None 2023-07-28T23:52:39.236 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:52:39 GMT /hiiqos-14/ 2023-07-28T23:52:39.236 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Tz/282uwjaP4tGvewXmqLevpn+A= 2023-07-28T23:52:39.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:52:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Tz/282uwjaP4tGvewXmqLevpn+A=', 'Content-Length': '0'} 2023-07-28T23:52:39.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c92138522376475-0064c454c7-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:52:39.284 DEBUG:boto:b'hiiqos-14obj1000falseobj2023-07-28T23:51:38.707Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:52:39.285 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:52:39.285 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:52:39.286 DEBUG:boto:Method: HEAD 2023-07-28T23:52:39.286 DEBUG:boto:Path: /hiiqos-14/obj 2023-07-28T23:52:39.287 DEBUG:boto:Data: 2023-07-28T23:52:39.287 DEBUG:boto:Headers: {} 2023-07-28T23:52:39.287 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:52:39.288 DEBUG:boto:Port: 8001 2023-07-28T23:52:39.288 DEBUG:boto:Params: {} 2023-07-28T23:52:39.288 DEBUG:boto:Token: None 2023-07-28T23:52:39.289 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:52:39 GMT /hiiqos-14/obj 2023-07-28T23:52:39.289 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ADnA79V1Y+1aR+EwKKuFQpaVCR4= 2023-07-28T23:52:39.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:52:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ADnA79V1Y+1aR+EwKKuFQpaVCR4=', 'Content-Length': '0'} 2023-07-28T23:52:39.292 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('x-amz-version-id', 'ISvc8k9pn.km8NePppXJwSK7pRcUJDI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000222eef3965fbf7d5-0064c454c7-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:52:39.293 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:52:39.293 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:52:39.294 DEBUG:boto:Method: HEAD 2023-07-28T23:52:39.294 DEBUG:boto:Path: /hiiqos-14/obj 2023-07-28T23:52:39.294 DEBUG:boto:Data: 2023-07-28T23:52:39.295 DEBUG:boto:Headers: {} 2023-07-28T23:52:39.295 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:52:39.295 DEBUG:boto:Port: 8000 2023-07-28T23:52:39.296 DEBUG:boto:Params: {} 2023-07-28T23:52:39.296 DEBUG:boto:Token: None 2023-07-28T23:52:39.297 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:52:39 GMT /hiiqos-14/obj 2023-07-28T23:52:39.297 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ADnA79V1Y+1aR+EwKKuFQpaVCR4= 2023-07-28T23:52:39.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:52:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ADnA79V1Y+1aR+EwKKuFQpaVCR4=', 'Content-Length': '0'} 2023-07-28T23:52:39.300 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('x-amz-version-id', 'ISvc8k9pn.km8NePppXJwSK7pRcUJDI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dc8e602e9c634d4f-0064c454c7-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:52:39.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:52:39.301 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:52:39.301 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:52:39.302 DEBUG:boto:path=/hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:52:39.302 DEBUG:boto:auth_path=/hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:52:39.303 DEBUG:boto:Method: GET 2023-07-28T23:52:39.303 DEBUG:boto:Path: /hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:52:39.303 DEBUG:boto:Data: 2023-07-28T23:52:39.304 DEBUG:boto:Headers: {} 2023-07-28T23:52:39.304 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:52:39.304 DEBUG:boto:Port: 8001 2023-07-28T23:52:39.305 DEBUG:boto:Params: {} 2023-07-28T23:52:39.305 DEBUG:boto:Token: None 2023-07-28T23:52:39.306 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:52:39 GMT /hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:52:39.306 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XVH/X80NdZl8QbVpmZGyb/eSxNw= 2023-07-28T23:52:39.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:52:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XVH/X80NdZl8QbVpmZGyb/eSxNw=', 'Content-Length': '0'} 2023-07-28T23:52:39.308 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('x-amz-version-id', 'ISvc8k9pn.km8NePppXJwSK7pRcUJDI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d50be74aaccfa7b3-0064c454c7-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:52:39.309 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:52:39.309 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:52:39.310 DEBUG:boto:path=/hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:52:39.310 DEBUG:boto:auth_path=/hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:52:39.311 DEBUG:boto:Method: GET 2023-07-28T23:52:39.311 DEBUG:boto:Path: /hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:52:39.311 DEBUG:boto:Data: 2023-07-28T23:52:39.312 DEBUG:boto:Headers: {} 2023-07-28T23:52:39.312 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:52:39.313 DEBUG:boto:Port: 8000 2023-07-28T23:52:39.313 DEBUG:boto:Params: {} 2023-07-28T23:52:39.313 DEBUG:boto:Token: None 2023-07-28T23:52:39.314 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:52:39 GMT /hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:52:39.314 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XVH/X80NdZl8QbVpmZGyb/eSxNw= 2023-07-28T23:52:39.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:52:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XVH/X80NdZl8QbVpmZGyb/eSxNw=', 'Content-Length': '0'} 2023-07-28T23:52:39.317 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('x-amz-version-id', 'ISvc8k9pn.km8NePppXJwSK7pRcUJDI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000630840d2d990222b-0064c454c7-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:52:39.317 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:52:39.318 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:52:39.318 DEBUG:boto:path=/hiiqos-14/?prefix=obj 2023-07-28T23:52:39.319 DEBUG:boto:auth_path=/hiiqos-14/?prefix=obj 2023-07-28T23:52:39.319 DEBUG:boto:Method: GET 2023-07-28T23:52:39.319 DEBUG:boto:Path: /hiiqos-14/?prefix=obj 2023-07-28T23:52:39.320 DEBUG:boto:Data: 2023-07-28T23:52:39.320 DEBUG:boto:Headers: {} 2023-07-28T23:52:39.320 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:52:39.321 DEBUG:boto:Port: 8001 2023-07-28T23:52:39.321 DEBUG:boto:Params: {} 2023-07-28T23:52:39.322 DEBUG:boto:Token: None 2023-07-28T23:52:39.322 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:52:39 GMT /hiiqos-14/ 2023-07-28T23:52:39.322 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Tz/282uwjaP4tGvewXmqLevpn+A= 2023-07-28T23:52:39.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:52:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Tz/282uwjaP4tGvewXmqLevpn+A=', 'Content-Length': '0'} 2023-07-28T23:52:39.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ea242abe307b7fa-0064c454c7-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:52:39.372 DEBUG:boto:b'hiiqos-14obj1000falseobj2023-07-28T23:51:38.707Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:52:39.373 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:52:39.373 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:52:39.374 DEBUG:boto:path=/hiiqos-14/?prefix=obj 2023-07-28T23:52:39.374 DEBUG:boto:auth_path=/hiiqos-14/?prefix=obj 2023-07-28T23:52:39.375 DEBUG:boto:Method: GET 2023-07-28T23:52:39.375 DEBUG:boto:Path: /hiiqos-14/?prefix=obj 2023-07-28T23:52:39.375 DEBUG:boto:Data: 2023-07-28T23:52:39.376 DEBUG:boto:Headers: {} 2023-07-28T23:52:39.376 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:52:39.376 DEBUG:boto:Port: 8000 2023-07-28T23:52:39.377 DEBUG:boto:Params: {} 2023-07-28T23:52:39.377 DEBUG:boto:Token: None 2023-07-28T23:52:39.378 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:52:39 GMT /hiiqos-14/ 2023-07-28T23:52:39.378 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Tz/282uwjaP4tGvewXmqLevpn+A= 2023-07-28T23:52:39.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:52:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Tz/282uwjaP4tGvewXmqLevpn+A=', 'Content-Length': '0'} 2023-07-28T23:52:39.381 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a7d6009d064444e-0064c454c7-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:52:39.382 DEBUG:boto:b'hiiqos-14obj1000falseobj2023-07-28T23:51:38.707Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:52:39.382 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-14 zones={a2, a1} 2023-07-28T23:52:39.383 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:52:39.383 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:52:39.383 DEBUG:boto:path=/hiiqos-14/?versioning 2023-07-28T23:52:39.384 DEBUG:boto:auth_path=/hiiqos-14/?versioning 2023-07-28T23:52:39.384 DEBUG:boto:Method: PUT 2023-07-28T23:52:39.384 DEBUG:boto:Path: /hiiqos-14/?versioning 2023-07-28T23:52:39.385 DEBUG:boto:Data: Suspended Disabled 2023-07-28T23:52:39.385 DEBUG:boto:Headers: {} 2023-07-28T23:52:39.385 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:52:39.386 DEBUG:boto:Port: 8000 2023-07-28T23:52:39.386 DEBUG:boto:Params: {} 2023-07-28T23:52:39.386 DEBUG:boto:Token: None 2023-07-28T23:52:39.387 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:52:39 GMT /hiiqos-14/?versioning 2023-07-28T23:52:39.387 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UDOtA/xZM9H2hLXzVe4uSjPHk18= 2023-07-28T23:52:39.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:52:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UDOtA/xZM9H2hLXzVe4uSjPHk18=', 'Content-Length': '230'} 2023-07-28T23:52:39.394 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000051b974518622ae7b-0064c454c7-1117-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:52:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:52:39.394 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:52:39.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:52:39.395 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:52:39.447 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:52:39.447 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:52:39.480 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-28T23:52:39.492 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-28T23:52:39.493 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:52:39.493 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-28T23:52:39.493 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:46:09.233742Z" 2023-07-28T23:52:39.493 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:52:39.493 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:52:39.493 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588359.392659_202.1", 2023-07-28T23:52:39.494 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:52:39.392659Z" 2023-07-28T23:52:39.494 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:52:39.494 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:52:39.494 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:52:39.494 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:52:39.494 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:52:39.495 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:52:39.495 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:52:39.495 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:52:39.495 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:52:39.495 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-28T23:52:39.496 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690587969.233742_99.1', 1: '1_1690588359.392659_202.1', 2: '', 3: ''} 2023-07-28T23:52:39.497 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-28T23:52:39.497 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:52:39.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:52:39.498 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:52:39.546 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:52:39.547 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:52:39.576 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:52:39.574+0000 7ff406a24880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:52:39.576 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:52:39.575+0000 7ff406a24880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:52:39.590 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:52:39.591 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:52:39.591 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:52:39.591 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:52:39.591 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:52:39.591 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:52:39.591 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:52:39.592 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:52:39.592 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:52:39.592 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:52:39.592 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:52:39.592 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:52:39.592 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:52:39.593 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-28T23:52:39.593 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:52:39.593 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:52:39.593 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:52:39.593 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-28T23:52:39.593 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:52:39.594 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:52:39.594 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:52:39.594 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:52:39.595 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:52:39.595 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:52:39.595 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:52:39.596 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588359.389038_200.1", 2023-07-28T23:52:39.596 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:52:39.596 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:52:39.597 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:52:39.597 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:52:39.389038Z", 2023-07-28T23:52:39.597 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:52:39.598 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:52:39.598 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:52:39.598 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:52:39.599 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:52:39.599 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:52:39.599 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:52:39.600 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:52:39.600 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:52:39.600 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:52:39.601 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:52:39.601 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:52:39.601 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:52:39.602 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:52:39.602 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:52:39.602 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:52:39.602 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:52:39.602 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:52:39.603 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:52:39.603 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:52:39.603 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:52:39.603 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:52:39.603 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:52:39.603 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:52:39.604 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:52:39.604 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:52:39.604 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:52:39.604 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:52:39.604 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:52:39.604 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:52:39.605 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:52:39.605 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:52:39.605 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:52:39.605 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:52:39Z" 2023-07-28T23:52:39.605 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:52:39.606 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588359.389038_200.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:52:39.389038Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-28T23:52:39Z" } 2023-07-28T23:52:39.607 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588359.389038_200.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:52:39.389038Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-28T23:52:39.608 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588359.392659_202.1', 2: '', 3: ''} 2023-07-28T23:52:39.609 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588359.389038_200.1', 2: '', 3: ''} 2023-07-28T23:52:39.609 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588359.392659_202.1 target=1_1690588359.389038_200.1 2023-07-28T23:52:44.610 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:52:44.611 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:52:44.612 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:52:44.665 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:52:44.665 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:52:44.696 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:52:44.694+0000 7fb703b2a880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:52:44.697 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:52:44.695+0000 7fb703b2a880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:52:44.709 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:52:44.710 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:52:44.710 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:52:44.710 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:52:44.710 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:52:44.710 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:52:44.711 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:52:44.711 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:52:44.711 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:52:44.711 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:52:44.711 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:52:44.711 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:52:44.712 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:52:44.712 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-28T23:52:44.712 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:52:44.712 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:52:44.712 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:52:44.713 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-28T23:52:44.713 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:52:44.713 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:52:44.713 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:52:44.713 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:52:44.713 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:52:44.714 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:52:44.714 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:52:44.714 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588359.392659_202.1", 2023-07-28T23:52:44.714 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:52:44.714 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:52:44.714 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:52:44.714 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:52:39.392659Z", 2023-07-28T23:52:44.715 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:52:44.715 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:52:44.715 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:52:44.715 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:52:44.715 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:52:44.715 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:52:44.716 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:52:44.716 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:52:44.716 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:52:44.716 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:52:44.716 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:52:44.716 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:52:44.717 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:52:44.717 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:52:44.717 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:52:44.717 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:52:44.717 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:52:44.717 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:52:44.718 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:52:44.718 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:52:44.718 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:52:44.718 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:52:44.718 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:52:44.718 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:52:44.718 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:52:44.719 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:52:44.719 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:52:44.719 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:52:44.719 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:52:44.719 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:52:44.719 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:52:44.720 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:52:44.720 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:52:44.720 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:52:44Z" 2023-07-28T23:52:44.720 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:52:44.721 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588359.392659_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:52:39.392659Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-28T23:52:44Z" } 2023-07-28T23:52:44.722 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588359.392659_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:52:39.392659Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-28T23:52:44.722 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588359.392659_202.1', 2: '', 3: ''} 2023-07-28T23:52:44.723 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588359.392659_202.1', 2: '', 3: ''} 2023-07-28T23:52:44.723 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-28T23:52:44.723 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:52:44.724 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:52:44.724 DEBUG:boto:Method: HEAD 2023-07-28T23:52:44.724 DEBUG:boto:Path: /hiiqos-14/ 2023-07-28T23:52:44.725 DEBUG:boto:Data: 2023-07-28T23:52:44.725 DEBUG:boto:Headers: {} 2023-07-28T23:52:44.725 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:52:44.726 DEBUG:boto:Port: 8000 2023-07-28T23:52:44.726 DEBUG:boto:Params: {} 2023-07-28T23:52:44.726 DEBUG:boto:Token: None 2023-07-28T23:52:44.727 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:52:44 GMT /hiiqos-14/ 2023-07-28T23:52:44.727 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MvnwXqSmoET2YB7SwnlbpAjNQQo= 2023-07-28T23:52:44.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:52:44 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MvnwXqSmoET2YB7SwnlbpAjNQQo=', 'Content-Length': '0'} 2023-07-28T23:52:44.730 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c1b8829f6a307627-0064c454cc-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:52:44.731 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:52:44.731 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:52:44.731 DEBUG:boto:Method: PUT 2023-07-28T23:52:44.732 DEBUG:boto:Path: /hiiqos-14/obj 2023-07-28T23:52:44.732 DEBUG:boto:Data: 2023-07-28T23:52:44.732 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-07-28T23:52:44.733 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:52:44.733 DEBUG:boto:Port: 8000 2023-07-28T23:52:44.733 DEBUG:boto:Params: {} 2023-07-28T23:52:44.734 DEBUG:boto:Token: None 2023-07-28T23:52:44.734 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 28 Jul 2023 23:52:44 GMT /hiiqos-14/obj 2023-07-28T23:52:44.735 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:grLRWBlH04z1ZxXRc/2UJD0J76Q= 2023-07-28T23:52:44.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:52:44 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:grLRWBlH04z1ZxXRc/2UJD0J76Q='} 2023-07-28T23:52:44.750 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b44d3a4da32b4908-0064c454cc-1117-a1'), ('Date', 'Fri, 28 Jul 2023 23:52:44 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:52:44.751 DEBUG:rgw_multi.tests:created null version id=None 2023-07-28T23:52:44.751 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-14 2023-07-28T23:52:44.752 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:52:44.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:52:44.752 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-28T23:52:44.806 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:52:44.806 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:52:44.837 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:52:44.835+0000 7f67fe975880 1 waiting for incremental sync to catch up: 2023-07-28T23:52:44.837 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , 00000000003.32.13, , ] 2023-07-28T23:52:44.837 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000005.36.18, , ] 2023-07-28T23:52:49.839 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:52:49.836+0000 7f67fe975880 1 waiting for incremental sync to catch up: 2023-07-28T23:52:49.839 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , 00000000003.32.13, , ] 2023-07-28T23:52:49.839 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000005.36.18, , ] 2023-07-28T23:52:54.839 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:52:54.838+0000 7f67fe975880 1 bucket sync caught up with source: 2023-07-28T23:52:54.840 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , 00000000005.36.18, , ] 2023-07-28T23:52:54.840 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000005.36.18, , ] 2023-07-28T23:52:54.840 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:52:54.838+0000 7f67fe975880 0 bucket checkpoint complete 2023-07-28T23:52:54.855 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-14 2023-07-28T23:52:54.856 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:52:54.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:52:54.857 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-28T23:52:54.915 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:52:54.915 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:52:54.949 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:52:54.947+0000 7f1468611880 1 waiting for incremental sync to catch up: 2023-07-28T23:52:54.949 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , 00000000003.21.13, , ] 2023-07-28T23:52:54.949 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000005.26.18, , ] 2023-07-28T23:52:59.950 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:52:59.948+0000 7f1468611880 1 waiting for incremental sync to catch up: 2023-07-28T23:52:59.951 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , 00000000003.21.13, , ] 2023-07-28T23:52:59.951 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000005.26.18, , ] 2023-07-28T23:53:04.952 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:53:04.950+0000 7f1468611880 1 waiting for incremental sync to catch up: 2023-07-28T23:53:04.952 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , 00000000003.21.13, , ] 2023-07-28T23:53:04.952 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000005.26.18, , ] 2023-07-28T23:53:09.953 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:53:09.951+0000 7f1468611880 1 waiting for incremental sync to catch up: 2023-07-28T23:53:09.953 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , 00000000003.21.13, , ] 2023-07-28T23:53:09.953 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000005.26.18, , ] 2023-07-28T23:53:14.954 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:53:14.952+0000 7f1468611880 1 bucket sync caught up with source: 2023-07-28T23:53:14.955 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , 00000000005.26.18, , ] 2023-07-28T23:53:14.955 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000005.26.18, , ] 2023-07-28T23:53:14.955 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:53:14.952+0000 7f1468611880 0 bucket checkpoint complete 2023-07-28T23:53:14.970 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-14 zones={a2, a1} 2023-07-28T23:53:14.971 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:53:14.971 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:53:14.971 DEBUG:boto:Method: HEAD 2023-07-28T23:53:14.972 DEBUG:boto:Path: /hiiqos-14/ 2023-07-28T23:53:14.972 DEBUG:boto:Data: 2023-07-28T23:53:14.973 DEBUG:boto:Headers: {} 2023-07-28T23:53:14.973 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:14.973 DEBUG:boto:Port: 8001 2023-07-28T23:53:14.974 DEBUG:boto:Params: {} 2023-07-28T23:53:14.974 DEBUG:boto:Token: None 2023-07-28T23:53:14.974 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:53:14 GMT /hiiqos-14/ 2023-07-28T23:53:14.975 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xWkzgf02bfNMWCf1vl9b+NhJ55A= 2023-07-28T23:53:14.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xWkzgf02bfNMWCf1vl9b+NhJ55A=', 'Content-Length': '0'} 2023-07-28T23:53:14.978 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000828fe71560dadc88-0064c454ea-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:53:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:14.978 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:53:14.979 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:53:14.979 DEBUG:boto:Method: HEAD 2023-07-28T23:53:14.979 DEBUG:boto:Path: /hiiqos-14/ 2023-07-28T23:53:14.980 DEBUG:boto:Data: 2023-07-28T23:53:14.980 DEBUG:boto:Headers: {} 2023-07-28T23:53:14.980 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:14.981 DEBUG:boto:Port: 8000 2023-07-28T23:53:14.981 DEBUG:boto:Params: {} 2023-07-28T23:53:14.981 DEBUG:boto:Token: None 2023-07-28T23:53:14.982 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:53:14 GMT /hiiqos-14/ 2023-07-28T23:53:14.982 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xWkzgf02bfNMWCf1vl9b+NhJ55A= 2023-07-28T23:53:14.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xWkzgf02bfNMWCf1vl9b+NhJ55A=', 'Content-Length': '0'} 2023-07-28T23:53:14.985 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c0a55caece3d6c08-0064c454ea-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:53:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:14.985 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-28T23:53:14.986 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:53:14.986 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:53:14.987 DEBUG:boto:path=/hiiqos-14/?versions&max-keys=999 2023-07-28T23:53:14.987 DEBUG:boto:auth_path=/hiiqos-14/?versions&max-keys=999 2023-07-28T23:53:14.987 DEBUG:boto:Method: GET 2023-07-28T23:53:14.988 DEBUG:boto:Path: /hiiqos-14/?versions&max-keys=999 2023-07-28T23:53:14.988 DEBUG:boto:Data: 2023-07-28T23:53:14.989 DEBUG:boto:Headers: {} 2023-07-28T23:53:14.989 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:14.989 DEBUG:boto:Port: 8001 2023-07-28T23:53:14.990 DEBUG:boto:Params: {} 2023-07-28T23:53:14.990 DEBUG:boto:Token: None 2023-07-28T23:53:14.990 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:14 GMT /hiiqos-14/?versions 2023-07-28T23:53:14.991 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kEmop6f5PW/G8EEWYyxZggOP2Z4= 2023-07-28T23:53:14.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kEmop6f5PW/G8EEWYyxZggOP2Z4=', 'Content-Length': '0'} 2023-07-28T23:53:14.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8a9fd3c5dbcfa30-0064c454ea-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:53:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:14.994 DEBUG:boto:b'hiiqos-14999falseobjnulltrue2023-07-28T23:52:44.736Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjISvc8k9pn.km8NePppXJwSK7pRcUJDIfalse2023-07-28T23:51:38.707Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:53:14.995 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:53:14.995 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:53:14.996 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-28T23:53:14.996 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:53:14.996 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:53:14.997 DEBUG:boto:path=/hiiqos-14/?versions&max-keys=999 2023-07-28T23:53:14.997 DEBUG:boto:auth_path=/hiiqos-14/?versions&max-keys=999 2023-07-28T23:53:14.998 DEBUG:boto:Method: GET 2023-07-28T23:53:14.998 DEBUG:boto:Path: /hiiqos-14/?versions&max-keys=999 2023-07-28T23:53:14.998 DEBUG:boto:Data: 2023-07-28T23:53:14.999 DEBUG:boto:Headers: {} 2023-07-28T23:53:14.999 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:15.000 DEBUG:boto:Port: 8000 2023-07-28T23:53:15.000 DEBUG:boto:Params: {} 2023-07-28T23:53:15.001 DEBUG:boto:Token: None 2023-07-28T23:53:15.002 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:15 GMT /hiiqos-14/?versions 2023-07-28T23:53:15.002 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:baRDR+rgDBgMPt5PYhH3SV6cKW0= 2023-07-28T23:53:15.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:baRDR+rgDBgMPt5PYhH3SV6cKW0=', 'Content-Length': '0'} 2023-07-28T23:53:15.005 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000703eb6b089a16c6-0064c454eb-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:53:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:15.048 DEBUG:boto:b'hiiqos-14999falseobjnulltrue2023-07-28T23:52:44.736Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjISvc8k9pn.km8NePppXJwSK7pRcUJDIfalse2023-07-28T23:51:38.707Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:53:15.049 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:53:15.050 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:53:15.050 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:53:15.050 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:53:15.051 DEBUG:boto:path=/hiiqos-14/?versions&max-keys=999 2023-07-28T23:53:15.051 DEBUG:boto:auth_path=/hiiqos-14/?versions&max-keys=999 2023-07-28T23:53:15.051 DEBUG:boto:Method: GET 2023-07-28T23:53:15.052 DEBUG:boto:Path: /hiiqos-14/?versions&max-keys=999 2023-07-28T23:53:15.052 DEBUG:boto:Data: 2023-07-28T23:53:15.053 DEBUG:boto:Headers: {} 2023-07-28T23:53:15.053 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:15.053 DEBUG:boto:Port: 8001 2023-07-28T23:53:15.054 DEBUG:boto:Params: {} 2023-07-28T23:53:15.054 DEBUG:boto:Token: None 2023-07-28T23:53:15.054 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:15 GMT /hiiqos-14/?versions 2023-07-28T23:53:15.055 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:baRDR+rgDBgMPt5PYhH3SV6cKW0= 2023-07-28T23:53:15.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:baRDR+rgDBgMPt5PYhH3SV6cKW0=', 'Content-Length': '0'} 2023-07-28T23:53:15.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071952febaca71952-0064c454eb-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:53:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:15.058 DEBUG:boto:b'hiiqos-14999falseobjnulltrue2023-07-28T23:52:44.736Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjISvc8k9pn.km8NePppXJwSK7pRcUJDIfalse2023-07-28T23:51:38.707Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:53:15.059 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:53:15.059 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:53:15.060 DEBUG:boto:path=/hiiqos-14/?versions&max-keys=999 2023-07-28T23:53:15.060 DEBUG:boto:auth_path=/hiiqos-14/?versions&max-keys=999 2023-07-28T23:53:15.060 DEBUG:boto:Method: GET 2023-07-28T23:53:15.061 DEBUG:boto:Path: /hiiqos-14/?versions&max-keys=999 2023-07-28T23:53:15.061 DEBUG:boto:Data: 2023-07-28T23:53:15.062 DEBUG:boto:Headers: {} 2023-07-28T23:53:15.062 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:15.062 DEBUG:boto:Port: 8000 2023-07-28T23:53:15.063 DEBUG:boto:Params: {} 2023-07-28T23:53:15.063 DEBUG:boto:Token: None 2023-07-28T23:53:15.063 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:15 GMT /hiiqos-14/?versions 2023-07-28T23:53:15.064 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:baRDR+rgDBgMPt5PYhH3SV6cKW0= 2023-07-28T23:53:15.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:baRDR+rgDBgMPt5PYhH3SV6cKW0=', 'Content-Length': '0'} 2023-07-28T23:53:15.067 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b3d338d91129c94-0064c454eb-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:53:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:15.067 DEBUG:boto:b'hiiqos-14999falseobjnulltrue2023-07-28T23:52:44.736Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjISvc8k9pn.km8NePppXJwSK7pRcUJDIfalse2023-07-28T23:51:38.707Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:53:15.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:53:15.068 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:53:15.069 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:53:15.069 DEBUG:boto:path=/hiiqos-14/obj?versionId=null 2023-07-28T23:53:15.069 DEBUG:boto:auth_path=/hiiqos-14/obj?versionId=null 2023-07-28T23:53:15.070 DEBUG:boto:Method: GET 2023-07-28T23:53:15.070 DEBUG:boto:Path: /hiiqos-14/obj?versionId=null 2023-07-28T23:53:15.071 DEBUG:boto:Data: 2023-07-28T23:53:15.071 DEBUG:boto:Headers: {} 2023-07-28T23:53:15.072 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:15.072 DEBUG:boto:Port: 8001 2023-07-28T23:53:15.072 DEBUG:boto:Params: {} 2023-07-28T23:53:15.073 DEBUG:boto:Token: None 2023-07-28T23:53:15.073 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:15 GMT /hiiqos-14/obj?versionId=null 2023-07-28T23:53:15.074 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nwqDs/O+P3eQwGEFSjMt7rX48zA= 2023-07-28T23:53:15.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nwqDs/O+P3eQwGEFSjMt7rX48zA=', 'Content-Length': '0'} 2023-07-28T23:53:15.076 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:52:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007d425b1229ca9393-0064c454eb-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:53:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:15.076 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:53:15.077 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:53:15.077 DEBUG:boto:path=/hiiqos-14/obj?versionId=null 2023-07-28T23:53:15.078 DEBUG:boto:auth_path=/hiiqos-14/obj?versionId=null 2023-07-28T23:53:15.078 DEBUG:boto:Method: GET 2023-07-28T23:53:15.078 DEBUG:boto:Path: /hiiqos-14/obj?versionId=null 2023-07-28T23:53:15.079 DEBUG:boto:Data: 2023-07-28T23:53:15.079 DEBUG:boto:Headers: {} 2023-07-28T23:53:15.079 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:15.080 DEBUG:boto:Port: 8000 2023-07-28T23:53:15.080 DEBUG:boto:Params: {} 2023-07-28T23:53:15.081 DEBUG:boto:Token: None 2023-07-28T23:53:15.081 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:15 GMT /hiiqos-14/obj?versionId=null 2023-07-28T23:53:15.082 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nwqDs/O+P3eQwGEFSjMt7rX48zA= 2023-07-28T23:53:15.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nwqDs/O+P3eQwGEFSjMt7rX48zA=', 'Content-Length': '0'} 2023-07-28T23:53:15.084 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:52:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a1b1a62ba564fe1e-0064c454eb-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:53:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:15.084 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:53:15.085 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:53:15.085 DEBUG:boto:path=/hiiqos-14/obj?versionId=null 2023-07-28T23:53:15.085 DEBUG:boto:auth_path=/hiiqos-14/obj?versionId=null 2023-07-28T23:53:15.086 DEBUG:boto:Method: HEAD 2023-07-28T23:53:15.086 DEBUG:boto:Path: /hiiqos-14/obj?versionId=null 2023-07-28T23:53:15.086 DEBUG:boto:Data: 2023-07-28T23:53:15.087 DEBUG:boto:Headers: {} 2023-07-28T23:53:15.087 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:15.087 DEBUG:boto:Port: 8001 2023-07-28T23:53:15.088 DEBUG:boto:Params: {} 2023-07-28T23:53:15.088 DEBUG:boto:Token: None 2023-07-28T23:53:15.089 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:53:15 GMT /hiiqos-14/obj?versionId=null 2023-07-28T23:53:15.089 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5niXLKrN3CtvoX1+Dky5mSX/TdQ= 2023-07-28T23:53:15.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5niXLKrN3CtvoX1+Dky5mSX/TdQ=', 'Content-Length': '0'} 2023-07-28T23:53:15.091 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:52:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000403500a6aea8ac15-0064c454eb-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:53:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:15.092 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:53:15.092 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:53:15.093 DEBUG:boto:path=/hiiqos-14/obj?versionId=null 2023-07-28T23:53:15.093 DEBUG:boto:auth_path=/hiiqos-14/obj?versionId=null 2023-07-28T23:53:15.093 DEBUG:boto:Method: HEAD 2023-07-28T23:53:15.094 DEBUG:boto:Path: /hiiqos-14/obj?versionId=null 2023-07-28T23:53:15.094 DEBUG:boto:Data: 2023-07-28T23:53:15.094 DEBUG:boto:Headers: {} 2023-07-28T23:53:15.095 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:15.095 DEBUG:boto:Port: 8000 2023-07-28T23:53:15.095 DEBUG:boto:Params: {} 2023-07-28T23:53:15.096 DEBUG:boto:Token: None 2023-07-28T23:53:15.096 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:53:15 GMT /hiiqos-14/obj?versionId=null 2023-07-28T23:53:15.097 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5niXLKrN3CtvoX1+Dky5mSX/TdQ= 2023-07-28T23:53:15.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5niXLKrN3CtvoX1+Dky5mSX/TdQ=', 'Content-Length': '0'} 2023-07-28T23:53:15.099 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:52:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004a5c2246a42d4ebf-0064c454eb-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:53:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:15.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:53:15.099 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:53:15.100 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:53:15.100 DEBUG:boto:path=/hiiqos-14/obj?versionId=null 2023-07-28T23:53:15.101 DEBUG:boto:auth_path=/hiiqos-14/obj?versionId=null 2023-07-28T23:53:15.101 DEBUG:boto:Method: GET 2023-07-28T23:53:15.102 DEBUG:boto:Path: /hiiqos-14/obj?versionId=null 2023-07-28T23:53:15.102 DEBUG:boto:Data: 2023-07-28T23:53:15.102 DEBUG:boto:Headers: {} 2023-07-28T23:53:15.103 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:15.103 DEBUG:boto:Port: 8001 2023-07-28T23:53:15.103 DEBUG:boto:Params: {} 2023-07-28T23:53:15.104 DEBUG:boto:Token: None 2023-07-28T23:53:15.104 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:15 GMT /hiiqos-14/obj?versionId=null 2023-07-28T23:53:15.105 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nwqDs/O+P3eQwGEFSjMt7rX48zA= 2023-07-28T23:53:15.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nwqDs/O+P3eQwGEFSjMt7rX48zA=', 'Content-Length': '0'} 2023-07-28T23:53:15.107 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:52:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000058ecc49bdd0e0ae7-0064c454eb-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:53:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:15.107 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:53:15.108 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:53:15.108 DEBUG:boto:path=/hiiqos-14/obj?versionId=null 2023-07-28T23:53:15.109 DEBUG:boto:auth_path=/hiiqos-14/obj?versionId=null 2023-07-28T23:53:15.109 DEBUG:boto:Method: GET 2023-07-28T23:53:15.109 DEBUG:boto:Path: /hiiqos-14/obj?versionId=null 2023-07-28T23:53:15.110 DEBUG:boto:Data: 2023-07-28T23:53:15.110 DEBUG:boto:Headers: {} 2023-07-28T23:53:15.110 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:15.111 DEBUG:boto:Port: 8000 2023-07-28T23:53:15.111 DEBUG:boto:Params: {} 2023-07-28T23:53:15.112 DEBUG:boto:Token: None 2023-07-28T23:53:15.112 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:15 GMT /hiiqos-14/obj?versionId=null 2023-07-28T23:53:15.113 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nwqDs/O+P3eQwGEFSjMt7rX48zA= 2023-07-28T23:53:15.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nwqDs/O+P3eQwGEFSjMt7rX48zA=', 'Content-Length': '0'} 2023-07-28T23:53:15.115 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:52:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000046a82ab37a778175-0064c454eb-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:53:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:15.115 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:53:15.116 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:53:15.116 DEBUG:boto:path=/hiiqos-14/?prefix=obj 2023-07-28T23:53:15.117 DEBUG:boto:auth_path=/hiiqos-14/?prefix=obj 2023-07-28T23:53:15.117 DEBUG:boto:Method: GET 2023-07-28T23:53:15.117 DEBUG:boto:Path: /hiiqos-14/?prefix=obj 2023-07-28T23:53:15.118 DEBUG:boto:Data: 2023-07-28T23:53:15.118 DEBUG:boto:Headers: {} 2023-07-28T23:53:15.118 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:15.119 DEBUG:boto:Port: 8001 2023-07-28T23:53:15.119 DEBUG:boto:Params: {} 2023-07-28T23:53:15.119 DEBUG:boto:Token: None 2023-07-28T23:53:15.120 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:15 GMT /hiiqos-14/ 2023-07-28T23:53:15.120 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yQJYfCNMPhIScxaoid+ZGeE/fEw= 2023-07-28T23:53:15.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yQJYfCNMPhIScxaoid+ZGeE/fEw=', 'Content-Length': '0'} 2023-07-28T23:53:15.123 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003006709dad7dda95-0064c454eb-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:53:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:15.123 DEBUG:boto:b'hiiqos-14obj1000falseobj2023-07-28T23:52:44.736Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:53:15.124 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:53:15.124 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:53:15.125 DEBUG:boto:path=/hiiqos-14/?prefix=obj 2023-07-28T23:53:15.125 DEBUG:boto:auth_path=/hiiqos-14/?prefix=obj 2023-07-28T23:53:15.126 DEBUG:boto:Method: GET 2023-07-28T23:53:15.126 DEBUG:boto:Path: /hiiqos-14/?prefix=obj 2023-07-28T23:53:15.126 DEBUG:boto:Data: 2023-07-28T23:53:15.127 DEBUG:boto:Headers: {} 2023-07-28T23:53:15.127 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:15.127 DEBUG:boto:Port: 8000 2023-07-28T23:53:15.128 DEBUG:boto:Params: {} 2023-07-28T23:53:15.128 DEBUG:boto:Token: None 2023-07-28T23:53:15.128 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:15 GMT /hiiqos-14/ 2023-07-28T23:53:15.129 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yQJYfCNMPhIScxaoid+ZGeE/fEw= 2023-07-28T23:53:15.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yQJYfCNMPhIScxaoid+ZGeE/fEw=', 'Content-Length': '0'} 2023-07-28T23:53:15.132 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057d155868147ed0d-0064c454eb-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:53:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:15.132 DEBUG:boto:b'hiiqos-14obj1000falseobj2023-07-28T23:52:44.736Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:53:15.133 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:53:15.133 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:53:15.134 DEBUG:boto:Method: HEAD 2023-07-28T23:53:15.134 DEBUG:boto:Path: /hiiqos-14/obj 2023-07-28T23:53:15.134 DEBUG:boto:Data: 2023-07-28T23:53:15.135 DEBUG:boto:Headers: {} 2023-07-28T23:53:15.135 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:15.135 DEBUG:boto:Port: 8001 2023-07-28T23:53:15.136 DEBUG:boto:Params: {} 2023-07-28T23:53:15.136 DEBUG:boto:Token: None 2023-07-28T23:53:15.136 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:53:15 GMT /hiiqos-14/obj 2023-07-28T23:53:15.137 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Ev37HLtGtJJ0gAcOmxvMI3j2NMo= 2023-07-28T23:53:15.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Ev37HLtGtJJ0gAcOmxvMI3j2NMo=', 'Content-Length': '0'} 2023-07-28T23:53:15.139 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:52:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000045609a757cf81439-0064c454eb-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:53:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:15.139 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:53:15.140 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:53:15.140 DEBUG:boto:Method: HEAD 2023-07-28T23:53:15.140 DEBUG:boto:Path: /hiiqos-14/obj 2023-07-28T23:53:15.141 DEBUG:boto:Data: 2023-07-28T23:53:15.141 DEBUG:boto:Headers: {} 2023-07-28T23:53:15.141 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:15.142 DEBUG:boto:Port: 8000 2023-07-28T23:53:15.142 DEBUG:boto:Params: {} 2023-07-28T23:53:15.142 DEBUG:boto:Token: None 2023-07-28T23:53:15.143 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:53:15 GMT /hiiqos-14/obj 2023-07-28T23:53:15.143 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Ev37HLtGtJJ0gAcOmxvMI3j2NMo= 2023-07-28T23:53:15.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Ev37HLtGtJJ0gAcOmxvMI3j2NMo=', 'Content-Length': '0'} 2023-07-28T23:53:15.146 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:52:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000092cf425cfee12803-0064c454eb-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:53:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:15.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:53:15.147 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:53:15.147 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:53:15.147 DEBUG:boto:Method: GET 2023-07-28T23:53:15.148 DEBUG:boto:Path: /hiiqos-14/obj 2023-07-28T23:53:15.148 DEBUG:boto:Data: 2023-07-28T23:53:15.149 DEBUG:boto:Headers: {} 2023-07-28T23:53:15.149 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:15.149 DEBUG:boto:Port: 8001 2023-07-28T23:53:15.150 DEBUG:boto:Params: {} 2023-07-28T23:53:15.150 DEBUG:boto:Token: None 2023-07-28T23:53:15.151 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:15 GMT /hiiqos-14/obj 2023-07-28T23:53:15.151 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:go2wB4T7O0S9KY5ACYvZN58xKqA= 2023-07-28T23:53:15.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:go2wB4T7O0S9KY5ACYvZN58xKqA=', 'Content-Length': '0'} 2023-07-28T23:53:15.153 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:52:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000e639b342136c970-0064c454eb-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:53:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:15.154 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:53:15.154 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:53:15.155 DEBUG:boto:Method: GET 2023-07-28T23:53:15.155 DEBUG:boto:Path: /hiiqos-14/obj 2023-07-28T23:53:15.155 DEBUG:boto:Data: 2023-07-28T23:53:15.156 DEBUG:boto:Headers: {} 2023-07-28T23:53:15.156 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:15.156 DEBUG:boto:Port: 8000 2023-07-28T23:53:15.157 DEBUG:boto:Params: {} 2023-07-28T23:53:15.157 DEBUG:boto:Token: None 2023-07-28T23:53:15.158 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:15 GMT /hiiqos-14/obj 2023-07-28T23:53:15.158 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:go2wB4T7O0S9KY5ACYvZN58xKqA= 2023-07-28T23:53:15.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:go2wB4T7O0S9KY5ACYvZN58xKqA=', 'Content-Length': '0'} 2023-07-28T23:53:15.160 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:52:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e90b73a0b97e4f12-0064c454eb-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:53:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:15.160 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:53:15.161 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:53:15.161 DEBUG:boto:path=/hiiqos-14/?prefix=obj 2023-07-28T23:53:15.162 DEBUG:boto:auth_path=/hiiqos-14/?prefix=obj 2023-07-28T23:53:15.162 DEBUG:boto:Method: GET 2023-07-28T23:53:15.162 DEBUG:boto:Path: /hiiqos-14/?prefix=obj 2023-07-28T23:53:15.163 DEBUG:boto:Data: 2023-07-28T23:53:15.163 DEBUG:boto:Headers: {} 2023-07-28T23:53:15.163 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:15.164 DEBUG:boto:Port: 8001 2023-07-28T23:53:15.164 DEBUG:boto:Params: {} 2023-07-28T23:53:15.165 DEBUG:boto:Token: None 2023-07-28T23:53:15.165 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:15 GMT /hiiqos-14/ 2023-07-28T23:53:15.166 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yQJYfCNMPhIScxaoid+ZGeE/fEw= 2023-07-28T23:53:15.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yQJYfCNMPhIScxaoid+ZGeE/fEw=', 'Content-Length': '0'} 2023-07-28T23:53:15.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092ddb70768c220db-0064c454eb-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:53:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:15.169 DEBUG:boto:b'hiiqos-14obj1000falseobj2023-07-28T23:52:44.736Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:53:15.169 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:53:15.170 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:53:15.170 DEBUG:boto:path=/hiiqos-14/?prefix=obj 2023-07-28T23:53:15.171 DEBUG:boto:auth_path=/hiiqos-14/?prefix=obj 2023-07-28T23:53:15.171 DEBUG:boto:Method: GET 2023-07-28T23:53:15.171 DEBUG:boto:Path: /hiiqos-14/?prefix=obj 2023-07-28T23:53:15.172 DEBUG:boto:Data: 2023-07-28T23:53:15.172 DEBUG:boto:Headers: {} 2023-07-28T23:53:15.172 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:15.173 DEBUG:boto:Port: 8000 2023-07-28T23:53:15.173 DEBUG:boto:Params: {} 2023-07-28T23:53:15.173 DEBUG:boto:Token: None 2023-07-28T23:53:15.174 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:15 GMT /hiiqos-14/ 2023-07-28T23:53:15.175 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yQJYfCNMPhIScxaoid+ZGeE/fEw= 2023-07-28T23:53:15.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yQJYfCNMPhIScxaoid+ZGeE/fEw=', 'Content-Length': '0'} 2023-07-28T23:53:15.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009bca7b75a5bad2e-0064c454eb-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:53:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:15.178 DEBUG:boto:b'hiiqos-14obj1000falseobj2023-07-28T23:52:44.736Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:53:15.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:53:15.178 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:53:15.179 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:53:15.179 DEBUG:boto:path=/hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:53:15.180 DEBUG:boto:auth_path=/hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:53:15.180 DEBUG:boto:Method: GET 2023-07-28T23:53:15.180 DEBUG:boto:Path: /hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:53:15.181 DEBUG:boto:Data: 2023-07-28T23:53:15.181 DEBUG:boto:Headers: {} 2023-07-28T23:53:15.181 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:15.182 DEBUG:boto:Port: 8001 2023-07-28T23:53:15.182 DEBUG:boto:Params: {} 2023-07-28T23:53:15.182 DEBUG:boto:Token: None 2023-07-28T23:53:15.183 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:15 GMT /hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:53:15.183 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZfHHeuo9QwmbBYv9ztv3P7beO5w= 2023-07-28T23:53:15.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZfHHeuo9QwmbBYv9ztv3P7beO5w=', 'Content-Length': '0'} 2023-07-28T23:53:15.185 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('x-amz-version-id', 'ISvc8k9pn.km8NePppXJwSK7pRcUJDI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002f1b01fe6fd482ae-0064c454eb-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:53:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:15.186 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:53:15.186 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:53:15.187 DEBUG:boto:path=/hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:53:15.187 DEBUG:boto:auth_path=/hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:53:15.187 DEBUG:boto:Method: GET 2023-07-28T23:53:15.188 DEBUG:boto:Path: /hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:53:15.188 DEBUG:boto:Data: 2023-07-28T23:53:15.189 DEBUG:boto:Headers: {} 2023-07-28T23:53:15.189 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:15.189 DEBUG:boto:Port: 8000 2023-07-28T23:53:15.190 DEBUG:boto:Params: {} 2023-07-28T23:53:15.190 DEBUG:boto:Token: None 2023-07-28T23:53:15.191 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:15 GMT /hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:53:15.191 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZfHHeuo9QwmbBYv9ztv3P7beO5w= 2023-07-28T23:53:15.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZfHHeuo9QwmbBYv9ztv3P7beO5w=', 'Content-Length': '0'} 2023-07-28T23:53:15.194 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('x-amz-version-id', 'ISvc8k9pn.km8NePppXJwSK7pRcUJDI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dd306ae1b6b3feb3-0064c454eb-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:53:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:15.194 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:53:15.195 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:53:15.195 DEBUG:boto:path=/hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:53:15.195 DEBUG:boto:auth_path=/hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:53:15.196 DEBUG:boto:Method: HEAD 2023-07-28T23:53:15.196 DEBUG:boto:Path: /hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:53:15.196 DEBUG:boto:Data: 2023-07-28T23:53:15.197 DEBUG:boto:Headers: {} 2023-07-28T23:53:15.197 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:15.197 DEBUG:boto:Port: 8001 2023-07-28T23:53:15.198 DEBUG:boto:Params: {} 2023-07-28T23:53:15.198 DEBUG:boto:Token: None 2023-07-28T23:53:15.199 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:53:15 GMT /hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:53:15.199 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:StQtvDv8vxY/rL5MQY2BNZmOgj8= 2023-07-28T23:53:15.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:StQtvDv8vxY/rL5MQY2BNZmOgj8=', 'Content-Length': '0'} 2023-07-28T23:53:15.201 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('x-amz-version-id', 'ISvc8k9pn.km8NePppXJwSK7pRcUJDI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009365e2733e56f29f-0064c454eb-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:53:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:15.201 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:53:15.202 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:53:15.202 DEBUG:boto:path=/hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:53:15.202 DEBUG:boto:auth_path=/hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:53:15.203 DEBUG:boto:Method: HEAD 2023-07-28T23:53:15.203 DEBUG:boto:Path: /hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:53:15.203 DEBUG:boto:Data: 2023-07-28T23:53:15.204 DEBUG:boto:Headers: {} 2023-07-28T23:53:15.204 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:15.204 DEBUG:boto:Port: 8000 2023-07-28T23:53:15.205 DEBUG:boto:Params: {} 2023-07-28T23:53:15.205 DEBUG:boto:Token: None 2023-07-28T23:53:15.206 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:53:15 GMT /hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:53:15.206 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:StQtvDv8vxY/rL5MQY2BNZmOgj8= 2023-07-28T23:53:15.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:StQtvDv8vxY/rL5MQY2BNZmOgj8=', 'Content-Length': '0'} 2023-07-28T23:53:15.208 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('x-amz-version-id', 'ISvc8k9pn.km8NePppXJwSK7pRcUJDI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000635c7dfa8d354812-0064c454eb-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:53:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:15.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:53:15.209 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:53:15.210 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:53:15.210 DEBUG:boto:path=/hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:53:15.210 DEBUG:boto:auth_path=/hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:53:15.211 DEBUG:boto:Method: GET 2023-07-28T23:53:15.211 DEBUG:boto:Path: /hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:53:15.211 DEBUG:boto:Data: 2023-07-28T23:53:15.212 DEBUG:boto:Headers: {} 2023-07-28T23:53:15.212 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:15.213 DEBUG:boto:Port: 8001 2023-07-28T23:53:15.213 DEBUG:boto:Params: {} 2023-07-28T23:53:15.213 DEBUG:boto:Token: None 2023-07-28T23:53:15.214 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:15 GMT /hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:53:15.214 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZfHHeuo9QwmbBYv9ztv3P7beO5w= 2023-07-28T23:53:15.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZfHHeuo9QwmbBYv9ztv3P7beO5w=', 'Content-Length': '0'} 2023-07-28T23:53:15.216 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('x-amz-version-id', 'ISvc8k9pn.km8NePppXJwSK7pRcUJDI'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006ff4fa8b39074842-0064c454eb-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:53:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:15.217 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:53:15.217 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:53:15.218 DEBUG:boto:path=/hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:53:15.218 DEBUG:boto:auth_path=/hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:53:15.218 DEBUG:boto:Method: GET 2023-07-28T23:53:15.219 DEBUG:boto:Path: /hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:53:15.219 DEBUG:boto:Data: 2023-07-28T23:53:15.219 DEBUG:boto:Headers: {} 2023-07-28T23:53:15.220 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:15.220 DEBUG:boto:Port: 8000 2023-07-28T23:53:15.220 DEBUG:boto:Params: {} 2023-07-28T23:53:15.221 DEBUG:boto:Token: None 2023-07-28T23:53:15.221 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:15 GMT /hiiqos-14/obj?versionId=ISvc8k9pn.km8NePppXJwSK7pRcUJDI 2023-07-28T23:53:15.222 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZfHHeuo9QwmbBYv9ztv3P7beO5w= 2023-07-28T23:53:15.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZfHHeuo9QwmbBYv9ztv3P7beO5w=', 'Content-Length': '0'} 2023-07-28T23:53:15.224 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:51:38 GMT'), ('x-amz-version-id', 'ISvc8k9pn.km8NePppXJwSK7pRcUJDI'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006a521b4dbfb64233-0064c454eb-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:53:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:15.224 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:53:15.225 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:53:15.225 DEBUG:boto:path=/hiiqos-14/?prefix=obj 2023-07-28T23:53:15.225 DEBUG:boto:auth_path=/hiiqos-14/?prefix=obj 2023-07-28T23:53:15.226 DEBUG:boto:Method: GET 2023-07-28T23:53:15.226 DEBUG:boto:Path: /hiiqos-14/?prefix=obj 2023-07-28T23:53:15.227 DEBUG:boto:Data: 2023-07-28T23:53:15.227 DEBUG:boto:Headers: {} 2023-07-28T23:53:15.227 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:15.228 DEBUG:boto:Port: 8001 2023-07-28T23:53:15.228 DEBUG:boto:Params: {} 2023-07-28T23:53:15.229 DEBUG:boto:Token: None 2023-07-28T23:53:15.229 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:15 GMT /hiiqos-14/ 2023-07-28T23:53:15.229 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yQJYfCNMPhIScxaoid+ZGeE/fEw= 2023-07-28T23:53:15.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yQJYfCNMPhIScxaoid+ZGeE/fEw=', 'Content-Length': '0'} 2023-07-28T23:53:15.232 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009241b72ae7bc54fe-0064c454eb-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:53:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:15.272 DEBUG:boto:b'hiiqos-14obj1000falseobj2023-07-28T23:52:44.736Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:53:15.273 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:53:15.273 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:53:15.274 DEBUG:boto:path=/hiiqos-14/?prefix=obj 2023-07-28T23:53:15.274 DEBUG:boto:auth_path=/hiiqos-14/?prefix=obj 2023-07-28T23:53:15.274 DEBUG:boto:Method: GET 2023-07-28T23:53:15.275 DEBUG:boto:Path: /hiiqos-14/?prefix=obj 2023-07-28T23:53:15.275 DEBUG:boto:Data: 2023-07-28T23:53:15.275 DEBUG:boto:Headers: {} 2023-07-28T23:53:15.276 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:15.276 DEBUG:boto:Port: 8000 2023-07-28T23:53:15.276 DEBUG:boto:Params: {} 2023-07-28T23:53:15.277 DEBUG:boto:Token: None 2023-07-28T23:53:15.277 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:15 GMT /hiiqos-14/ 2023-07-28T23:53:15.278 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yQJYfCNMPhIScxaoid+ZGeE/fEw= 2023-07-28T23:53:15.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yQJYfCNMPhIScxaoid+ZGeE/fEw=', 'Content-Length': '0'} 2023-07-28T23:53:15.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069e2d70aaf9e3d63-0064c454eb-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:53:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:15.281 DEBUG:boto:b'hiiqos-14obj1000falseobj2023-07-28T23:52:44.736Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:53:15.282 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:53:15.282 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:53:15.283 DEBUG:boto:Method: HEAD 2023-07-28T23:53:15.283 DEBUG:boto:Path: /hiiqos-14/obj 2023-07-28T23:53:15.283 DEBUG:boto:Data: 2023-07-28T23:53:15.284 DEBUG:boto:Headers: {} 2023-07-28T23:53:15.284 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:15.284 DEBUG:boto:Port: 8001 2023-07-28T23:53:15.285 DEBUG:boto:Params: {} 2023-07-28T23:53:15.285 DEBUG:boto:Token: None 2023-07-28T23:53:15.285 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:53:15 GMT /hiiqos-14/obj 2023-07-28T23:53:15.286 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Ev37HLtGtJJ0gAcOmxvMI3j2NMo= 2023-07-28T23:53:15.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Ev37HLtGtJJ0gAcOmxvMI3j2NMo=', 'Content-Length': '0'} 2023-07-28T23:53:15.288 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:52:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000066d5d076237c55d5-0064c454eb-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:53:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:15.289 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:53:15.289 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:53:15.289 DEBUG:boto:Method: HEAD 2023-07-28T23:53:15.290 DEBUG:boto:Path: /hiiqos-14/obj 2023-07-28T23:53:15.290 DEBUG:boto:Data: 2023-07-28T23:53:15.290 DEBUG:boto:Headers: {} 2023-07-28T23:53:15.291 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:15.291 DEBUG:boto:Port: 8000 2023-07-28T23:53:15.291 DEBUG:boto:Params: {} 2023-07-28T23:53:15.292 DEBUG:boto:Token: None 2023-07-28T23:53:15.292 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:53:15 GMT /hiiqos-14/obj 2023-07-28T23:53:15.293 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Ev37HLtGtJJ0gAcOmxvMI3j2NMo= 2023-07-28T23:53:15.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Ev37HLtGtJJ0gAcOmxvMI3j2NMo=', 'Content-Length': '0'} 2023-07-28T23:53:15.295 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:52:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bd92ec5bfb6e363c-0064c454eb-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:53:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:15.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:53:15.296 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:53:15.296 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:53:15.296 DEBUG:boto:Method: GET 2023-07-28T23:53:15.297 DEBUG:boto:Path: /hiiqos-14/obj 2023-07-28T23:53:15.297 DEBUG:boto:Data: 2023-07-28T23:53:15.297 DEBUG:boto:Headers: {} 2023-07-28T23:53:15.298 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:15.298 DEBUG:boto:Port: 8001 2023-07-28T23:53:15.298 DEBUG:boto:Params: {} 2023-07-28T23:53:15.299 DEBUG:boto:Token: None 2023-07-28T23:53:15.299 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:15 GMT /hiiqos-14/obj 2023-07-28T23:53:15.300 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:go2wB4T7O0S9KY5ACYvZN58xKqA= 2023-07-28T23:53:15.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:go2wB4T7O0S9KY5ACYvZN58xKqA=', 'Content-Length': '0'} 2023-07-28T23:53:15.302 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:52:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c7fb7316fa976744-0064c454eb-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:53:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:15.303 DEBUG:boto:path=/hiiqos-14/obj 2023-07-28T23:53:15.303 DEBUG:boto:auth_path=/hiiqos-14/obj 2023-07-28T23:53:15.304 DEBUG:boto:Method: GET 2023-07-28T23:53:15.304 DEBUG:boto:Path: /hiiqos-14/obj 2023-07-28T23:53:15.304 DEBUG:boto:Data: 2023-07-28T23:53:15.305 DEBUG:boto:Headers: {} 2023-07-28T23:53:15.305 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:15.305 DEBUG:boto:Port: 8000 2023-07-28T23:53:15.306 DEBUG:boto:Params: {} 2023-07-28T23:53:15.306 DEBUG:boto:Token: None 2023-07-28T23:53:15.306 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:15 GMT /hiiqos-14/obj 2023-07-28T23:53:15.307 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:go2wB4T7O0S9KY5ACYvZN58xKqA= 2023-07-28T23:53:15.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:go2wB4T7O0S9KY5ACYvZN58xKqA=', 'Content-Length': '0'} 2023-07-28T23:53:15.309 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:52:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000087e6a4ea4f11c094-0064c454eb-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:53:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:15.310 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:53:15.310 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:53:15.311 DEBUG:boto:path=/hiiqos-14/?prefix=obj 2023-07-28T23:53:15.311 DEBUG:boto:auth_path=/hiiqos-14/?prefix=obj 2023-07-28T23:53:15.311 DEBUG:boto:Method: GET 2023-07-28T23:53:15.312 DEBUG:boto:Path: /hiiqos-14/?prefix=obj 2023-07-28T23:53:15.312 DEBUG:boto:Data: 2023-07-28T23:53:15.312 DEBUG:boto:Headers: {} 2023-07-28T23:53:15.313 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:15.313 DEBUG:boto:Port: 8001 2023-07-28T23:53:15.313 DEBUG:boto:Params: {} 2023-07-28T23:53:15.314 DEBUG:boto:Token: None 2023-07-28T23:53:15.314 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:15 GMT /hiiqos-14/ 2023-07-28T23:53:15.315 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yQJYfCNMPhIScxaoid+ZGeE/fEw= 2023-07-28T23:53:15.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yQJYfCNMPhIScxaoid+ZGeE/fEw=', 'Content-Length': '0'} 2023-07-28T23:53:15.317 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a90ad935ea4c2631-0064c454eb-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:53:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:15.319 DEBUG:boto:b'hiiqos-14obj1000falseobj2023-07-28T23:52:44.736Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:53:15.320 DEBUG:boto:path=/hiiqos-14/ 2023-07-28T23:53:15.320 DEBUG:boto:auth_path=/hiiqos-14/ 2023-07-28T23:53:15.321 DEBUG:boto:path=/hiiqos-14/?prefix=obj 2023-07-28T23:53:15.322 DEBUG:boto:auth_path=/hiiqos-14/?prefix=obj 2023-07-28T23:53:15.323 DEBUG:boto:Method: GET 2023-07-28T23:53:15.323 DEBUG:boto:Path: /hiiqos-14/?prefix=obj 2023-07-28T23:53:15.324 DEBUG:boto:Data: 2023-07-28T23:53:15.325 DEBUG:boto:Headers: {} 2023-07-28T23:53:15.325 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:15.326 DEBUG:boto:Port: 8000 2023-07-28T23:53:15.327 DEBUG:boto:Params: {} 2023-07-28T23:53:15.327 DEBUG:boto:Token: None 2023-07-28T23:53:15.328 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:15 GMT /hiiqos-14/ 2023-07-28T23:53:15.329 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yQJYfCNMPhIScxaoid+ZGeE/fEw= 2023-07-28T23:53:15.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yQJYfCNMPhIScxaoid+ZGeE/fEw=', 'Content-Length': '0'} 2023-07-28T23:53:15.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e789de3c7ab19e0-0064c454eb-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:53:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:15.333 DEBUG:boto:b'hiiqos-14obj1000falseobj2023-07-28T23:52:44.736Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:53:15.334 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-14 zones={a2, a1} 2023-07-28T23:53:15.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2023-07-28T23:53:15.336 INFO:rgw_multi.tests:create bucket zone=a1 name=hiiqos-15 2023-07-28T23:53:15.337 DEBUG:boto:path=/hiiqos-15/ 2023-07-28T23:53:15.338 DEBUG:boto:auth_path=/hiiqos-15/ 2023-07-28T23:53:15.338 DEBUG:boto:Method: PUT 2023-07-28T23:53:15.339 DEBUG:boto:Path: /hiiqos-15/ 2023-07-28T23:53:15.340 DEBUG:boto:Data: 2023-07-28T23:53:15.340 DEBUG:boto:Headers: {} 2023-07-28T23:53:15.341 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:15.341 DEBUG:boto:Port: 8000 2023-07-28T23:53:15.342 DEBUG:boto:Params: {} 2023-07-28T23:53:15.343 DEBUG:boto:Token: None 2023-07-28T23:53:15.343 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:53:15 GMT /hiiqos-15/ 2023-07-28T23:53:15.344 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TGFcT4+kgCbtszzayjW8QNpvEHA= 2023-07-28T23:53:15.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TGFcT4+kgCbtszzayjW8QNpvEHA=', 'Content-Length': '0'} 2023-07-28T23:53:15.360 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000991917bd98693ddb-0064c454eb-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:53:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:15.361 INFO:rgw_multi.tests:create bucket zone=a2 name=hiiqos-16 2023-07-28T23:53:15.361 DEBUG:boto:path=/hiiqos-16/ 2023-07-28T23:53:15.362 DEBUG:boto:auth_path=/hiiqos-16/ 2023-07-28T23:53:15.362 DEBUG:boto:Method: PUT 2023-07-28T23:53:15.363 DEBUG:boto:Path: /hiiqos-16/ 2023-07-28T23:53:15.364 DEBUG:boto:Data: 2023-07-28T23:53:15.364 DEBUG:boto:Headers: {} 2023-07-28T23:53:15.365 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:15.366 DEBUG:boto:Port: 8001 2023-07-28T23:53:15.366 DEBUG:boto:Params: {} 2023-07-28T23:53:15.367 DEBUG:boto:Token: None 2023-07-28T23:53:15.368 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:53:15 GMT /hiiqos-16/ 2023-07-28T23:53:15.368 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HkGVywDZj4zIhJEwF0Jv9epdfUU= 2023-07-28T23:53:15.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HkGVywDZj4zIhJEwF0Jv9epdfUU=', 'Content-Length': '0'} 2023-07-28T23:53:15.395 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d3dd69c50b23cbc0-0064c454eb-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:53:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:15.396 DEBUG:boto:path=/hiiqos-15/ 2023-07-28T23:53:15.396 DEBUG:boto:auth_path=/hiiqos-15/ 2023-07-28T23:53:15.397 DEBUG:boto:path=/hiiqos-15/?versioning 2023-07-28T23:53:15.397 DEBUG:boto:auth_path=/hiiqos-15/?versioning 2023-07-28T23:53:15.398 DEBUG:boto:Method: PUT 2023-07-28T23:53:15.399 DEBUG:boto:Path: /hiiqos-15/?versioning 2023-07-28T23:53:15.399 DEBUG:boto:Data: Enabled Disabled 2023-07-28T23:53:15.400 DEBUG:boto:Headers: {} 2023-07-28T23:53:15.400 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:15.401 DEBUG:boto:Port: 8000 2023-07-28T23:53:15.402 DEBUG:boto:Params: {} 2023-07-28T23:53:15.402 DEBUG:boto:Token: None 2023-07-28T23:53:15.403 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:53:15 GMT /hiiqos-15/?versioning 2023-07-28T23:53:15.404 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ig51SKY+PQ9tPKr8d2Jt10sB3Fc= 2023-07-28T23:53:15.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ig51SKY+PQ9tPKr8d2Jt10sB3Fc=', 'Content-Length': '228'} 2023-07-28T23:53:15.410 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005033e49b3cfcbeaa-0064c454eb-1117-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:53:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:15.411 DEBUG:boto:path=/hiiqos-16/ 2023-07-28T23:53:15.411 DEBUG:boto:auth_path=/hiiqos-16/ 2023-07-28T23:53:15.412 DEBUG:boto:path=/hiiqos-16/?versioning 2023-07-28T23:53:15.412 DEBUG:boto:auth_path=/hiiqos-16/?versioning 2023-07-28T23:53:15.413 DEBUG:boto:Method: PUT 2023-07-28T23:53:15.414 DEBUG:boto:Path: /hiiqos-16/?versioning 2023-07-28T23:53:15.414 DEBUG:boto:Data: Enabled Disabled 2023-07-28T23:53:15.415 DEBUG:boto:Headers: {} 2023-07-28T23:53:15.415 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:15.416 DEBUG:boto:Port: 8001 2023-07-28T23:53:15.417 DEBUG:boto:Params: {} 2023-07-28T23:53:15.417 DEBUG:boto:Token: None 2023-07-28T23:53:15.418 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:53:15 GMT /hiiqos-16/?versioning 2023-07-28T23:53:15.419 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:n5rbLyezQQz4UMYChPaeqicKTmk= 2023-07-28T23:53:15.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:n5rbLyezQQz4UMYChPaeqicKTmk=', 'Content-Length': '228'} 2023-07-28T23:53:15.427 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e70a63d39a19d4c2-0064c454eb-1113-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:53:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:15.428 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:53:15.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:53:15.429 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:53:15.477 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:53:15.477 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:53:15.510 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-28T23:53:15.522 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-28T23:53:15.522 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:53:15.522 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-28T23:53:15.522 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:46:09.233742Z" 2023-07-28T23:53:15.523 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:53:15.523 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:53:15.523 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588395.424288_219.1", 2023-07-28T23:53:15.523 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:53:15.424288Z" 2023-07-28T23:53:15.523 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:53:15.523 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:53:15.523 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:53:15.524 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:53:15.524 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:53:15.524 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:53:15.524 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:53:15.524 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:53:15.524 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:53:15.525 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-28T23:53:15.525 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690587969.233742_99.1', 1: '1_1690588395.424288_219.1', 2: '', 3: ''} 2023-07-28T23:53:15.526 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-28T23:53:15.526 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:53:15.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:53:15.527 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:53:15.578 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:53:15.578 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:53:15.608 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:53:15.606+0000 7f024e019880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:53:15.610 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:53:15.608+0000 7f024e019880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:53:15.623 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:53:15.623 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:53:15.623 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:53:15.623 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:53:15.623 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:53:15.623 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:53:15.624 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:53:15.624 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:53:15.624 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:53:15.624 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:53:15.624 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:53:15.624 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:53:15.625 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:53:15.625 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-28T23:53:15.625 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:53:15.625 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:53:15.625 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:53:15.625 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-28T23:53:15.625 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:53:15.626 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:53:15.626 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:53:15.626 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:53:15.626 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:53:15.626 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:53:15.626 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:53:15.626 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588395.350538_207.1", 2023-07-28T23:53:15.627 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:53:15.627 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:53:15.627 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:53:15.627 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:53:15.350538Z", 2023-07-28T23:53:15.627 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:53:15.627 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:53:15.627 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:53:15.628 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:53:15.628 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:53:15.628 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:53:15.628 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:53:15.628 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:53:15.628 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:53:15.629 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:53:15.629 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:53:15.629 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:53:15.629 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:53:15.629 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:53:15.629 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:53:15.629 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:53:15.630 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:53:15.630 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:53:15.630 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:53:15.630 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:53:15.630 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:53:15.630 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:53:15.630 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:53:15.631 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:53:15.631 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:53:15.631 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:53:15.631 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:53:15.631 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:53:15.631 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:53:15.631 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:53:15.632 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:53:15.632 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:53:15.632 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:53:15.632 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:53:15Z" 2023-07-28T23:53:15.632 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:53:15.633 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588395.350538_207.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:53:15.350538Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-28T23:53:15Z" } 2023-07-28T23:53:15.634 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588395.350538_207.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:53:15.350538Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-28T23:53:15.634 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588395.424288_219.1', 2: '', 3: ''} 2023-07-28T23:53:15.634 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588395.350538_207.1', 2: '', 3: ''} 2023-07-28T23:53:15.635 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588395.424288_219.1 target=1_1690588395.350538_207.1 2023-07-28T23:53:20.636 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:53:20.637 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:53:20.637 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:53:20.690 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:53:20.691 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:53:20.723 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:53:20.721+0000 7f8ab1733880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:53:20.724 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:53:20.722+0000 7f8ab1733880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:53:20.736 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:53:20.736 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:53:20.736 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:53:20.737 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:53:20.737 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:53:20.737 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:53:20.737 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:53:20.737 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:53:20.737 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:53:20.738 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:53:20.738 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:53:20.738 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:53:20.738 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:53:20.738 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-28T23:53:20.738 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:53:20.738 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:53:20.739 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:53:20.739 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-28T23:53:20.739 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:53:20.739 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:53:20.739 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:53:20.739 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:53:20.739 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:53:20.740 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:53:20.740 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:53:20.740 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588395.424288_219.1", 2023-07-28T23:53:20.740 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:53:20.740 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:53:20.740 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:53:20.740 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:53:15.424288Z", 2023-07-28T23:53:20.741 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:53:20.741 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:53:20.741 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:53:20.741 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:53:20.741 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:53:20.741 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:53:20.742 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:53:20.742 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:53:20.742 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:53:20.742 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:53:20.742 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:53:20.742 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:53:20.742 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:53:20.743 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:53:20.743 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:53:20.743 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:53:20.743 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:53:20.743 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:53:20.743 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:53:20.743 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:53:20.744 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:53:20.744 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:53:20.744 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:53:20.744 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:53:20.744 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:53:20.744 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:53:20.745 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:53:20.745 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:53:20.745 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:53:20.745 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:53:20.745 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:53:20.745 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:53:20.745 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:53:20.746 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:53:20Z" 2023-07-28T23:53:20.746 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:53:20.747 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588395.424288_219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:53:15.424288Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-28T23:53:20Z" } 2023-07-28T23:53:20.747 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588395.424288_219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:53:15.424288Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-28T23:53:20.748 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588395.424288_219.1', 2: '', 3: ''} 2023-07-28T23:53:20.748 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588395.424288_219.1', 2: '', 3: ''} 2023-07-28T23:53:20.748 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-28T23:53:20.749 DEBUG:boto:path=/hiiqos-15/ 2023-07-28T23:53:20.749 DEBUG:boto:auth_path=/hiiqos-15/ 2023-07-28T23:53:20.749 DEBUG:boto:Method: HEAD 2023-07-28T23:53:20.750 DEBUG:boto:Path: /hiiqos-15/ 2023-07-28T23:53:20.750 DEBUG:boto:Data: 2023-07-28T23:53:20.751 DEBUG:boto:Headers: {} 2023-07-28T23:53:20.751 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:20.751 DEBUG:boto:Port: 8000 2023-07-28T23:53:20.752 DEBUG:boto:Params: {} 2023-07-28T23:53:20.752 DEBUG:boto:Token: None 2023-07-28T23:53:20.753 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:53:20 GMT /hiiqos-15/ 2023-07-28T23:53:20.753 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:byBOeN+mRyheB+1NtqZqGsa4kGI= 2023-07-28T23:53:20.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:byBOeN+mRyheB+1NtqZqGsa4kGI=', 'Content-Length': '0'} 2023-07-28T23:53:20.756 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004ba6b48cac956a4e-0064c454f0-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:53:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:20.756 DEBUG:boto:path=/hiiqos-15/obj 2023-07-28T23:53:20.757 DEBUG:boto:auth_path=/hiiqos-15/obj 2023-07-28T23:53:20.757 DEBUG:boto:Method: PUT 2023-07-28T23:53:20.757 DEBUG:boto:Path: /hiiqos-15/obj 2023-07-28T23:53:20.758 DEBUG:boto:Data: 2023-07-28T23:53:20.758 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-07-28T23:53:20.758 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:20.759 DEBUG:boto:Port: 8000 2023-07-28T23:53:20.759 DEBUG:boto:Params: {} 2023-07-28T23:53:20.759 DEBUG:boto:Token: None 2023-07-28T23:53:20.760 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 28 Jul 2023 23:53:20 GMT /hiiqos-15/obj 2023-07-28T23:53:20.760 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:woVOqq9QXlmza9W3F0Xw3VfzekM= 2023-07-28T23:53:20.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:53:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:woVOqq9QXlmza9W3F0Xw3VfzekM='} 2023-07-28T23:53:20.770 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'XPWszzKL9FTCuO9TQktFmh6a7eyLHpQ'), ('x-amz-request-id', 'tx000008744c35749d85270-0064c454f0-1117-a1'), ('Date', 'Fri, 28 Jul 2023 23:53:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:20.771 DEBUG:boto:path=/hiiqos-15/ 2023-07-28T23:53:20.771 DEBUG:boto:auth_path=/hiiqos-15/ 2023-07-28T23:53:20.771 DEBUG:boto:Method: HEAD 2023-07-28T23:53:20.772 DEBUG:boto:Path: /hiiqos-15/ 2023-07-28T23:53:20.772 DEBUG:boto:Data: 2023-07-28T23:53:20.772 DEBUG:boto:Headers: {} 2023-07-28T23:53:20.773 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:20.773 DEBUG:boto:Port: 8000 2023-07-28T23:53:20.773 DEBUG:boto:Params: {} 2023-07-28T23:53:20.774 DEBUG:boto:Token: None 2023-07-28T23:53:20.774 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:53:20 GMT /hiiqos-15/ 2023-07-28T23:53:20.775 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:byBOeN+mRyheB+1NtqZqGsa4kGI= 2023-07-28T23:53:20.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:byBOeN+mRyheB+1NtqZqGsa4kGI=', 'Content-Length': '0'} 2023-07-28T23:53:20.777 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af441738819360cc-0064c454f0-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:53:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:20.777 DEBUG:boto:path=/hiiqos-15/obj 2023-07-28T23:53:20.778 DEBUG:boto:auth_path=/hiiqos-15/obj 2023-07-28T23:53:20.778 DEBUG:boto:Method: DELETE 2023-07-28T23:53:20.779 DEBUG:boto:Path: /hiiqos-15/obj 2023-07-28T23:53:20.779 DEBUG:boto:Data: 2023-07-28T23:53:20.779 DEBUG:boto:Headers: {} 2023-07-28T23:53:20.780 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:20.780 DEBUG:boto:Port: 8000 2023-07-28T23:53:20.780 DEBUG:boto:Params: {} 2023-07-28T23:53:20.781 DEBUG:boto:Token: None 2023-07-28T23:53:20.781 DEBUG:boto:StringToSign: DELETE Fri, 28 Jul 2023 23:53:20 GMT /hiiqos-15/obj 2023-07-28T23:53:20.781 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kmjVvKqnFUTk23kyGvUQMt4xq14= 2023-07-28T23:53:20.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kmjVvKqnFUTk23kyGvUQMt4xq14=', 'Content-Length': '0'} 2023-07-28T23:53:20.787 DEBUG:boto:Response headers: [('x-amz-version-id', 'F4sFaWRY6lClu7u-wJxpe88WNvFrtjk'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000e1c637e1b1e84567-0064c454f0-1117-a1'), ('Date', 'Fri, 28 Jul 2023 23:53:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:20.788 DEBUG:boto:path=/hiiqos-16/ 2023-07-28T23:53:20.788 DEBUG:boto:auth_path=/hiiqos-16/ 2023-07-28T23:53:20.789 DEBUG:boto:Method: HEAD 2023-07-28T23:53:20.789 DEBUG:boto:Path: /hiiqos-16/ 2023-07-28T23:53:20.789 DEBUG:boto:Data: 2023-07-28T23:53:20.790 DEBUG:boto:Headers: {} 2023-07-28T23:53:20.790 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:20.790 DEBUG:boto:Port: 8001 2023-07-28T23:53:20.791 DEBUG:boto:Params: {} 2023-07-28T23:53:20.791 DEBUG:boto:Token: None 2023-07-28T23:53:20.791 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:53:20 GMT /hiiqos-16/ 2023-07-28T23:53:20.792 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MrqoPPc7kpY2ZmO1ALYksevyT1M= 2023-07-28T23:53:20.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MrqoPPc7kpY2ZmO1ALYksevyT1M=', 'Content-Length': '0'} 2023-07-28T23:53:20.794 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000098664af2e5ee6b1b-0064c454f0-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:53:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:20.795 DEBUG:boto:path=/hiiqos-16/obj 2023-07-28T23:53:20.795 DEBUG:boto:auth_path=/hiiqos-16/obj 2023-07-28T23:53:20.796 DEBUG:boto:Method: PUT 2023-07-28T23:53:20.796 DEBUG:boto:Path: /hiiqos-16/obj 2023-07-28T23:53:20.796 DEBUG:boto:Data: 2023-07-28T23:53:20.797 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-07-28T23:53:20.797 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:20.797 DEBUG:boto:Port: 8001 2023-07-28T23:53:20.798 DEBUG:boto:Params: {} 2023-07-28T23:53:20.798 DEBUG:boto:Token: None 2023-07-28T23:53:20.798 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 28 Jul 2023 23:53:20 GMT /hiiqos-16/obj 2023-07-28T23:53:20.799 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mPdd0zY0cUyAOJ0CMzvFHgbH5R8= 2023-07-28T23:53:20.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:53:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mPdd0zY0cUyAOJ0CMzvFHgbH5R8='} 2023-07-28T23:53:20.809 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'arpVnoFZTsBYgcmcJ6HYUv2zgJJSiqL'), ('x-amz-request-id', 'tx000005bd3d4987106e25e-0064c454f0-1113-a2'), ('Date', 'Fri, 28 Jul 2023 23:53:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:20.809 DEBUG:boto:path=/hiiqos-16/ 2023-07-28T23:53:20.809 DEBUG:boto:auth_path=/hiiqos-16/ 2023-07-28T23:53:20.810 DEBUG:boto:Method: HEAD 2023-07-28T23:53:20.810 DEBUG:boto:Path: /hiiqos-16/ 2023-07-28T23:53:20.810 DEBUG:boto:Data: 2023-07-28T23:53:20.811 DEBUG:boto:Headers: {} 2023-07-28T23:53:20.811 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:20.811 DEBUG:boto:Port: 8001 2023-07-28T23:53:20.812 DEBUG:boto:Params: {} 2023-07-28T23:53:20.812 DEBUG:boto:Token: None 2023-07-28T23:53:20.812 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:53:20 GMT /hiiqos-16/ 2023-07-28T23:53:20.813 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MrqoPPc7kpY2ZmO1ALYksevyT1M= 2023-07-28T23:53:20.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MrqoPPc7kpY2ZmO1ALYksevyT1M=', 'Content-Length': '0'} 2023-07-28T23:53:20.815 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f301a3af153fa8fb-0064c454f0-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:53:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:20.816 DEBUG:boto:path=/hiiqos-16/obj 2023-07-28T23:53:20.816 DEBUG:boto:auth_path=/hiiqos-16/obj 2023-07-28T23:53:20.816 DEBUG:boto:Method: DELETE 2023-07-28T23:53:20.817 DEBUG:boto:Path: /hiiqos-16/obj 2023-07-28T23:53:20.817 DEBUG:boto:Data: 2023-07-28T23:53:20.817 DEBUG:boto:Headers: {} 2023-07-28T23:53:20.818 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:20.818 DEBUG:boto:Port: 8001 2023-07-28T23:53:20.818 DEBUG:boto:Params: {} 2023-07-28T23:53:20.819 DEBUG:boto:Token: None 2023-07-28T23:53:20.819 DEBUG:boto:StringToSign: DELETE Fri, 28 Jul 2023 23:53:20 GMT /hiiqos-16/obj 2023-07-28T23:53:20.820 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/8rzobJJoKdNrw+QYLtkh2UoEEo= 2023-07-28T23:53:20.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/8rzobJJoKdNrw+QYLtkh2UoEEo=', 'Content-Length': '0'} 2023-07-28T23:53:20.825 DEBUG:boto:Response headers: [('x-amz-version-id', 'Fp8Y.s8tSo9pMrSp7xiIjUzSdsmYyCm'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx000008be3bf0cfa9b3bca-0064c454f0-1113-a2'), ('Date', 'Fri, 28 Jul 2023 23:53:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:20.826 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-15 2023-07-28T23:53:20.826 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-15', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:53:20.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-15', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:53:20.827 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-15 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-28T23:53:20.879 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:53:20.880 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:53:20.911 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:53:20.910+0000 7fd347222880 1 waiting to reach incremental sync.. 2023-07-28T23:53:25.912 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:53:25.910+0000 7fd347222880 1 waiting to reach incremental sync.. 2023-07-28T23:53:30.913 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:53:30.911+0000 7fd347222880 1 waiting to reach incremental sync.. 2023-07-28T23:53:35.915 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:53:35.913+0000 7fd347222880 1 bucket sync caught up with source: 2023-07-28T23:53:35.915 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , 00000000003.17.12, , ] 2023-07-28T23:53:35.915 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000003.17.12, , ] 2023-07-28T23:53:35.915 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:53:35.913+0000 7fd347222880 0 bucket checkpoint complete 2023-07-28T23:53:35.931 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-15 2023-07-28T23:53:35.931 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-15', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:53:35.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-15', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:53:35.932 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-15 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-28T23:53:35.990 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:53:35.990 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:53:36.025 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:53:36.022+0000 7faa52738880 1 waiting to reach incremental sync.. 2023-07-28T23:53:41.025 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:53:41.023+0000 7faa52738880 1 waiting to reach incremental sync.. 2023-07-28T23:53:46.025 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:53:46.024+0000 7faa52738880 1 waiting to reach incremental sync.. 2023-07-28T23:53:51.026 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:53:51.024+0000 7faa52738880 1 waiting to reach incremental sync.. 2023-07-28T23:53:56.028 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:53:56.026+0000 7faa52738880 1 bucket sync caught up with source: 2023-07-28T23:53:56.029 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , 00000000002.45.6, , ] 2023-07-28T23:53:56.029 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000002.45.6, , ] 2023-07-28T23:53:56.029 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:53:56.026+0000 7faa52738880 0 bucket checkpoint complete 2023-07-28T23:53:56.044 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-15 zones={a2, a1} 2023-07-28T23:53:56.045 DEBUG:boto:path=/hiiqos-15/ 2023-07-28T23:53:56.045 DEBUG:boto:auth_path=/hiiqos-15/ 2023-07-28T23:53:56.046 DEBUG:boto:Method: HEAD 2023-07-28T23:53:56.046 DEBUG:boto:Path: /hiiqos-15/ 2023-07-28T23:53:56.047 DEBUG:boto:Data: 2023-07-28T23:53:56.048 DEBUG:boto:Headers: {} 2023-07-28T23:53:56.049 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:56.049 DEBUG:boto:Port: 8001 2023-07-28T23:53:56.050 DEBUG:boto:Params: {} 2023-07-28T23:53:56.050 DEBUG:boto:Token: None 2023-07-28T23:53:56.051 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:53:56 GMT /hiiqos-15/ 2023-07-28T23:53:56.051 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Y0Hys2k9uCrWCO2h0Hhtjkw4dlc= 2023-07-28T23:53:56.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Y0Hys2k9uCrWCO2h0Hhtjkw4dlc=', 'Content-Length': '0'} 2023-07-28T23:53:56.054 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cccf433f43560fc7-0064c45514-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:56.054 DEBUG:boto:path=/hiiqos-15/ 2023-07-28T23:53:56.055 DEBUG:boto:auth_path=/hiiqos-15/ 2023-07-28T23:53:56.055 DEBUG:boto:Method: HEAD 2023-07-28T23:53:56.056 DEBUG:boto:Path: /hiiqos-15/ 2023-07-28T23:53:56.056 DEBUG:boto:Data: 2023-07-28T23:53:56.056 DEBUG:boto:Headers: {} 2023-07-28T23:53:56.057 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:56.057 DEBUG:boto:Port: 8000 2023-07-28T23:53:56.057 DEBUG:boto:Params: {} 2023-07-28T23:53:56.058 DEBUG:boto:Token: None 2023-07-28T23:53:56.058 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:53:56 GMT /hiiqos-15/ 2023-07-28T23:53:56.058 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Y0Hys2k9uCrWCO2h0Hhtjkw4dlc= 2023-07-28T23:53:56.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Y0Hys2k9uCrWCO2h0Hhtjkw4dlc=', 'Content-Length': '0'} 2023-07-28T23:53:56.061 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000587b610f36565e56-0064c45514-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:56.062 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-28T23:53:56.062 DEBUG:boto:path=/hiiqos-15/ 2023-07-28T23:53:56.063 DEBUG:boto:auth_path=/hiiqos-15/ 2023-07-28T23:53:56.063 DEBUG:boto:path=/hiiqos-15/?versions&max-keys=999 2023-07-28T23:53:56.064 DEBUG:boto:auth_path=/hiiqos-15/?versions&max-keys=999 2023-07-28T23:53:56.064 DEBUG:boto:Method: GET 2023-07-28T23:53:56.064 DEBUG:boto:Path: /hiiqos-15/?versions&max-keys=999 2023-07-28T23:53:56.065 DEBUG:boto:Data: 2023-07-28T23:53:56.065 DEBUG:boto:Headers: {} 2023-07-28T23:53:56.065 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:56.066 DEBUG:boto:Port: 8001 2023-07-28T23:53:56.066 DEBUG:boto:Params: {} 2023-07-28T23:53:56.067 DEBUG:boto:Token: None 2023-07-28T23:53:56.067 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:56 GMT /hiiqos-15/?versions 2023-07-28T23:53:56.068 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SDo9t1bILBA5YQHFA4SHXfg3WA0= 2023-07-28T23:53:56.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SDo9t1bILBA5YQHFA4SHXfg3WA0=', 'Content-Length': '0'} 2023-07-28T23:53:56.071 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045e2b1eeb495c8b8-0064c45514-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:56.071 DEBUG:boto:b'hiiqos-15999falseobjF4sFaWRY6lClu7u-wJxpe88WNvFrtjktrue2023-07-28T23:53:20.783Zrgw-multisite-test-userMultisite Test UserNormalobjXPWszzKL9FTCuO9TQktFmh6a7eyLHpQfalse2023-07-28T23:53:20.762Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:53:56.072 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:53:56.073 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:53:56.073 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-28T23:53:56.074 DEBUG:boto:path=/hiiqos-15/ 2023-07-28T23:53:56.074 DEBUG:boto:auth_path=/hiiqos-15/ 2023-07-28T23:53:56.074 DEBUG:boto:path=/hiiqos-15/?versions&max-keys=999 2023-07-28T23:53:56.075 DEBUG:boto:auth_path=/hiiqos-15/?versions&max-keys=999 2023-07-28T23:53:56.075 DEBUG:boto:Method: GET 2023-07-28T23:53:56.075 DEBUG:boto:Path: /hiiqos-15/?versions&max-keys=999 2023-07-28T23:53:56.076 DEBUG:boto:Data: 2023-07-28T23:53:56.076 DEBUG:boto:Headers: {} 2023-07-28T23:53:56.076 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:56.077 DEBUG:boto:Port: 8000 2023-07-28T23:53:56.077 DEBUG:boto:Params: {} 2023-07-28T23:53:56.078 DEBUG:boto:Token: None 2023-07-28T23:53:56.078 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:56 GMT /hiiqos-15/?versions 2023-07-28T23:53:56.079 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SDo9t1bILBA5YQHFA4SHXfg3WA0= 2023-07-28T23:53:56.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SDo9t1bILBA5YQHFA4SHXfg3WA0=', 'Content-Length': '0'} 2023-07-28T23:53:56.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c671c57c94f52787-0064c45514-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:56.124 DEBUG:boto:b'hiiqos-15999falseobjF4sFaWRY6lClu7u-wJxpe88WNvFrtjktrue2023-07-28T23:53:20.783Zrgw-multisite-test-userMultisite Test UserNormalobjXPWszzKL9FTCuO9TQktFmh6a7eyLHpQfalse2023-07-28T23:53:20.762Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:53:56.125 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:53:56.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:53:56.126 DEBUG:boto:path=/hiiqos-15/ 2023-07-28T23:53:56.126 DEBUG:boto:auth_path=/hiiqos-15/ 2023-07-28T23:53:56.127 DEBUG:boto:path=/hiiqos-15/?versions&max-keys=999 2023-07-28T23:53:56.127 DEBUG:boto:auth_path=/hiiqos-15/?versions&max-keys=999 2023-07-28T23:53:56.127 DEBUG:boto:Method: GET 2023-07-28T23:53:56.128 DEBUG:boto:Path: /hiiqos-15/?versions&max-keys=999 2023-07-28T23:53:56.128 DEBUG:boto:Data: 2023-07-28T23:53:56.128 DEBUG:boto:Headers: {} 2023-07-28T23:53:56.129 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:56.129 DEBUG:boto:Port: 8001 2023-07-28T23:53:56.129 DEBUG:boto:Params: {} 2023-07-28T23:53:56.130 DEBUG:boto:Token: None 2023-07-28T23:53:56.130 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:56 GMT /hiiqos-15/?versions 2023-07-28T23:53:56.131 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SDo9t1bILBA5YQHFA4SHXfg3WA0= 2023-07-28T23:53:56.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SDo9t1bILBA5YQHFA4SHXfg3WA0=', 'Content-Length': '0'} 2023-07-28T23:53:56.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007085e28f89a4a7a4-0064c45514-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:56.134 DEBUG:boto:b'hiiqos-15999falseobjF4sFaWRY6lClu7u-wJxpe88WNvFrtjktrue2023-07-28T23:53:20.783Zrgw-multisite-test-userMultisite Test UserNormalobjXPWszzKL9FTCuO9TQktFmh6a7eyLHpQfalse2023-07-28T23:53:20.762Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:53:56.135 DEBUG:boto:path=/hiiqos-15/ 2023-07-28T23:53:56.135 DEBUG:boto:auth_path=/hiiqos-15/ 2023-07-28T23:53:56.136 DEBUG:boto:path=/hiiqos-15/?versions&max-keys=999 2023-07-28T23:53:56.136 DEBUG:boto:auth_path=/hiiqos-15/?versions&max-keys=999 2023-07-28T23:53:56.137 DEBUG:boto:Method: GET 2023-07-28T23:53:56.137 DEBUG:boto:Path: /hiiqos-15/?versions&max-keys=999 2023-07-28T23:53:56.137 DEBUG:boto:Data: 2023-07-28T23:53:56.138 DEBUG:boto:Headers: {} 2023-07-28T23:53:56.138 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:56.138 DEBUG:boto:Port: 8000 2023-07-28T23:53:56.139 DEBUG:boto:Params: {} 2023-07-28T23:53:56.139 DEBUG:boto:Token: None 2023-07-28T23:53:56.139 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:56 GMT /hiiqos-15/?versions 2023-07-28T23:53:56.140 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SDo9t1bILBA5YQHFA4SHXfg3WA0= 2023-07-28T23:53:56.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SDo9t1bILBA5YQHFA4SHXfg3WA0=', 'Content-Length': '0'} 2023-07-28T23:53:56.143 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8b36a229330164c-0064c45514-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:56.144 DEBUG:boto:b'hiiqos-15999falseobjF4sFaWRY6lClu7u-wJxpe88WNvFrtjktrue2023-07-28T23:53:20.783Zrgw-multisite-test-userMultisite Test UserNormalobjXPWszzKL9FTCuO9TQktFmh6a7eyLHpQfalse2023-07-28T23:53:20.762Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:53:56.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:53:56.145 DEBUG:boto:path=/hiiqos-15/obj 2023-07-28T23:53:56.145 DEBUG:boto:auth_path=/hiiqos-15/obj 2023-07-28T23:53:56.145 DEBUG:boto:Method: HEAD 2023-07-28T23:53:56.146 DEBUG:boto:Path: /hiiqos-15/obj 2023-07-28T23:53:56.146 DEBUG:boto:Data: 2023-07-28T23:53:56.146 DEBUG:boto:Headers: {} 2023-07-28T23:53:56.147 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:56.147 DEBUG:boto:Port: 8001 2023-07-28T23:53:56.147 DEBUG:boto:Params: {} 2023-07-28T23:53:56.148 DEBUG:boto:Token: None 2023-07-28T23:53:56.148 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:53:56 GMT /hiiqos-15/obj 2023-07-28T23:53:56.148 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MlipHuqxkAR2h1JP3JLTOQpLvQc= 2023-07-28T23:53:56.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MlipHuqxkAR2h1JP3JLTOQpLvQc=', 'Content-Length': '0'} 2023-07-28T23:53:56.151 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000000a75004564f5f0f7-0064c45514-1113-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:56.151 DEBUG:boto:path=/hiiqos-15/obj 2023-07-28T23:53:56.152 DEBUG:boto:auth_path=/hiiqos-15/obj 2023-07-28T23:53:56.152 DEBUG:boto:Method: HEAD 2023-07-28T23:53:56.153 DEBUG:boto:Path: /hiiqos-15/obj 2023-07-28T23:53:56.153 DEBUG:boto:Data: 2023-07-28T23:53:56.153 DEBUG:boto:Headers: {} 2023-07-28T23:53:56.154 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:56.154 DEBUG:boto:Port: 8000 2023-07-28T23:53:56.154 DEBUG:boto:Params: {} 2023-07-28T23:53:56.155 DEBUG:boto:Token: None 2023-07-28T23:53:56.155 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:53:56 GMT /hiiqos-15/obj 2023-07-28T23:53:56.155 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MlipHuqxkAR2h1JP3JLTOQpLvQc= 2023-07-28T23:53:56.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MlipHuqxkAR2h1JP3JLTOQpLvQc=', 'Content-Length': '0'} 2023-07-28T23:53:56.158 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000003846f3921326db39-0064c45514-1117-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:56.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:53:56.159 DEBUG:boto:path=/hiiqos-15/obj 2023-07-28T23:53:56.159 DEBUG:boto:auth_path=/hiiqos-15/obj 2023-07-28T23:53:56.159 DEBUG:boto:path=/hiiqos-15/obj?versionId=XPWszzKL9FTCuO9TQktFmh6a7eyLHpQ 2023-07-28T23:53:56.160 DEBUG:boto:auth_path=/hiiqos-15/obj?versionId=XPWszzKL9FTCuO9TQktFmh6a7eyLHpQ 2023-07-28T23:53:56.160 DEBUG:boto:Method: GET 2023-07-28T23:53:56.160 DEBUG:boto:Path: /hiiqos-15/obj?versionId=XPWszzKL9FTCuO9TQktFmh6a7eyLHpQ 2023-07-28T23:53:56.161 DEBUG:boto:Data: 2023-07-28T23:53:56.161 DEBUG:boto:Headers: {} 2023-07-28T23:53:56.162 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:56.162 DEBUG:boto:Port: 8001 2023-07-28T23:53:56.162 DEBUG:boto:Params: {} 2023-07-28T23:53:56.163 DEBUG:boto:Token: None 2023-07-28T23:53:56.163 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:56 GMT /hiiqos-15/obj?versionId=XPWszzKL9FTCuO9TQktFmh6a7eyLHpQ 2023-07-28T23:53:56.163 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Z7oF2xPI8xSlRbb0qU5RRJ9y0zI= 2023-07-28T23:53:56.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Z7oF2xPI8xSlRbb0qU5RRJ9y0zI=', 'Content-Length': '0'} 2023-07-28T23:53:56.166 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:53:20 GMT'), ('x-amz-version-id', 'XPWszzKL9FTCuO9TQktFmh6a7eyLHpQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-15:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d9dab870e59e939d-0064c45514-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:56.166 DEBUG:boto:path=/hiiqos-15/obj 2023-07-28T23:53:56.167 DEBUG:boto:auth_path=/hiiqos-15/obj 2023-07-28T23:53:56.167 DEBUG:boto:path=/hiiqos-15/obj?versionId=XPWszzKL9FTCuO9TQktFmh6a7eyLHpQ 2023-07-28T23:53:56.167 DEBUG:boto:auth_path=/hiiqos-15/obj?versionId=XPWszzKL9FTCuO9TQktFmh6a7eyLHpQ 2023-07-28T23:53:56.168 DEBUG:boto:Method: GET 2023-07-28T23:53:56.168 DEBUG:boto:Path: /hiiqos-15/obj?versionId=XPWszzKL9FTCuO9TQktFmh6a7eyLHpQ 2023-07-28T23:53:56.168 DEBUG:boto:Data: 2023-07-28T23:53:56.169 DEBUG:boto:Headers: {} 2023-07-28T23:53:56.169 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:56.170 DEBUG:boto:Port: 8000 2023-07-28T23:53:56.170 DEBUG:boto:Params: {} 2023-07-28T23:53:56.170 DEBUG:boto:Token: None 2023-07-28T23:53:56.171 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:56 GMT /hiiqos-15/obj?versionId=XPWszzKL9FTCuO9TQktFmh6a7eyLHpQ 2023-07-28T23:53:56.171 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Z7oF2xPI8xSlRbb0qU5RRJ9y0zI= 2023-07-28T23:53:56.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Z7oF2xPI8xSlRbb0qU5RRJ9y0zI=', 'Content-Length': '0'} 2023-07-28T23:53:56.174 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:53:20 GMT'), ('x-amz-version-id', 'XPWszzKL9FTCuO9TQktFmh6a7eyLHpQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008a19d44ea31de363-0064c45514-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:56.174 DEBUG:boto:path=/hiiqos-15/obj 2023-07-28T23:53:56.175 DEBUG:boto:auth_path=/hiiqos-15/obj 2023-07-28T23:53:56.175 DEBUG:boto:path=/hiiqos-15/obj?versionId=XPWszzKL9FTCuO9TQktFmh6a7eyLHpQ 2023-07-28T23:53:56.176 DEBUG:boto:auth_path=/hiiqos-15/obj?versionId=XPWszzKL9FTCuO9TQktFmh6a7eyLHpQ 2023-07-28T23:53:56.176 DEBUG:boto:Method: HEAD 2023-07-28T23:53:56.176 DEBUG:boto:Path: /hiiqos-15/obj?versionId=XPWszzKL9FTCuO9TQktFmh6a7eyLHpQ 2023-07-28T23:53:56.177 DEBUG:boto:Data: 2023-07-28T23:53:56.177 DEBUG:boto:Headers: {} 2023-07-28T23:53:56.177 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:56.177 DEBUG:boto:Port: 8001 2023-07-28T23:53:56.178 DEBUG:boto:Params: {} 2023-07-28T23:53:56.178 DEBUG:boto:Token: None 2023-07-28T23:53:56.179 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:53:56 GMT /hiiqos-15/obj?versionId=XPWszzKL9FTCuO9TQktFmh6a7eyLHpQ 2023-07-28T23:53:56.179 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CtuQiaGDN8v9HW72ZBQevdsGRsI= 2023-07-28T23:53:56.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CtuQiaGDN8v9HW72ZBQevdsGRsI=', 'Content-Length': '0'} 2023-07-28T23:53:56.181 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:53:20 GMT'), ('x-amz-version-id', 'XPWszzKL9FTCuO9TQktFmh6a7eyLHpQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-15:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000b73abd6805ebce6-0064c45514-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:56.182 DEBUG:boto:path=/hiiqos-15/obj 2023-07-28T23:53:56.182 DEBUG:boto:auth_path=/hiiqos-15/obj 2023-07-28T23:53:56.183 DEBUG:boto:path=/hiiqos-15/obj?versionId=XPWszzKL9FTCuO9TQktFmh6a7eyLHpQ 2023-07-28T23:53:56.183 DEBUG:boto:auth_path=/hiiqos-15/obj?versionId=XPWszzKL9FTCuO9TQktFmh6a7eyLHpQ 2023-07-28T23:53:56.183 DEBUG:boto:Method: HEAD 2023-07-28T23:53:56.183 DEBUG:boto:Path: /hiiqos-15/obj?versionId=XPWszzKL9FTCuO9TQktFmh6a7eyLHpQ 2023-07-28T23:53:56.184 DEBUG:boto:Data: 2023-07-28T23:53:56.184 DEBUG:boto:Headers: {} 2023-07-28T23:53:56.184 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:56.185 DEBUG:boto:Port: 8000 2023-07-28T23:53:56.185 DEBUG:boto:Params: {} 2023-07-28T23:53:56.186 DEBUG:boto:Token: None 2023-07-28T23:53:56.186 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:53:56 GMT /hiiqos-15/obj?versionId=XPWszzKL9FTCuO9TQktFmh6a7eyLHpQ 2023-07-28T23:53:56.186 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CtuQiaGDN8v9HW72ZBQevdsGRsI= 2023-07-28T23:53:56.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CtuQiaGDN8v9HW72ZBQevdsGRsI=', 'Content-Length': '0'} 2023-07-28T23:53:56.188 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:53:20 GMT'), ('x-amz-version-id', 'XPWszzKL9FTCuO9TQktFmh6a7eyLHpQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000b4b8a0e6b7ba0ed-0064c45514-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:56.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:53:56.189 DEBUG:boto:path=/hiiqos-15/obj 2023-07-28T23:53:56.190 DEBUG:boto:auth_path=/hiiqos-15/obj 2023-07-28T23:53:56.190 DEBUG:boto:path=/hiiqos-15/obj?versionId=XPWszzKL9FTCuO9TQktFmh6a7eyLHpQ 2023-07-28T23:53:56.190 DEBUG:boto:auth_path=/hiiqos-15/obj?versionId=XPWszzKL9FTCuO9TQktFmh6a7eyLHpQ 2023-07-28T23:53:56.191 DEBUG:boto:Method: GET 2023-07-28T23:53:56.191 DEBUG:boto:Path: /hiiqos-15/obj?versionId=XPWszzKL9FTCuO9TQktFmh6a7eyLHpQ 2023-07-28T23:53:56.191 DEBUG:boto:Data: 2023-07-28T23:53:56.192 DEBUG:boto:Headers: {} 2023-07-28T23:53:56.192 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:56.193 DEBUG:boto:Port: 8001 2023-07-28T23:53:56.193 DEBUG:boto:Params: {} 2023-07-28T23:53:56.193 DEBUG:boto:Token: None 2023-07-28T23:53:56.194 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:56 GMT /hiiqos-15/obj?versionId=XPWszzKL9FTCuO9TQktFmh6a7eyLHpQ 2023-07-28T23:53:56.194 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Z7oF2xPI8xSlRbb0qU5RRJ9y0zI= 2023-07-28T23:53:56.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Z7oF2xPI8xSlRbb0qU5RRJ9y0zI=', 'Content-Length': '0'} 2023-07-28T23:53:56.196 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:53:20 GMT'), ('x-amz-version-id', 'XPWszzKL9FTCuO9TQktFmh6a7eyLHpQ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-15:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cf6a7fdff43bfbfe-0064c45514-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:56.197 DEBUG:boto:path=/hiiqos-15/obj 2023-07-28T23:53:56.197 DEBUG:boto:auth_path=/hiiqos-15/obj 2023-07-28T23:53:56.198 DEBUG:boto:path=/hiiqos-15/obj?versionId=XPWszzKL9FTCuO9TQktFmh6a7eyLHpQ 2023-07-28T23:53:56.198 DEBUG:boto:auth_path=/hiiqos-15/obj?versionId=XPWszzKL9FTCuO9TQktFmh6a7eyLHpQ 2023-07-28T23:53:56.198 DEBUG:boto:Method: GET 2023-07-28T23:53:56.199 DEBUG:boto:Path: /hiiqos-15/obj?versionId=XPWszzKL9FTCuO9TQktFmh6a7eyLHpQ 2023-07-28T23:53:56.199 DEBUG:boto:Data: 2023-07-28T23:53:56.199 DEBUG:boto:Headers: {} 2023-07-28T23:53:56.200 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:56.200 DEBUG:boto:Port: 8000 2023-07-28T23:53:56.200 DEBUG:boto:Params: {} 2023-07-28T23:53:56.201 DEBUG:boto:Token: None 2023-07-28T23:53:56.201 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:56 GMT /hiiqos-15/obj?versionId=XPWszzKL9FTCuO9TQktFmh6a7eyLHpQ 2023-07-28T23:53:56.202 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Z7oF2xPI8xSlRbb0qU5RRJ9y0zI= 2023-07-28T23:53:56.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Z7oF2xPI8xSlRbb0qU5RRJ9y0zI=', 'Content-Length': '0'} 2023-07-28T23:53:56.204 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:53:20 GMT'), ('x-amz-version-id', 'XPWszzKL9FTCuO9TQktFmh6a7eyLHpQ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b4eb9301d8554b58-0064c45514-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:56.205 DEBUG:boto:path=/hiiqos-15/ 2023-07-28T23:53:56.205 DEBUG:boto:auth_path=/hiiqos-15/ 2023-07-28T23:53:56.205 DEBUG:boto:path=/hiiqos-15/?prefix=obj 2023-07-28T23:53:56.206 DEBUG:boto:auth_path=/hiiqos-15/?prefix=obj 2023-07-28T23:53:56.206 DEBUG:boto:Method: GET 2023-07-28T23:53:56.206 DEBUG:boto:Path: /hiiqos-15/?prefix=obj 2023-07-28T23:53:56.207 DEBUG:boto:Data: 2023-07-28T23:53:56.207 DEBUG:boto:Headers: {} 2023-07-28T23:53:56.207 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:56.208 DEBUG:boto:Port: 8001 2023-07-28T23:53:56.208 DEBUG:boto:Params: {} 2023-07-28T23:53:56.208 DEBUG:boto:Token: None 2023-07-28T23:53:56.209 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:56 GMT /hiiqos-15/ 2023-07-28T23:53:56.209 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rup/XnBRJoMmO7Q4MNiJk3AKzYM= 2023-07-28T23:53:56.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rup/XnBRJoMmO7Q4MNiJk3AKzYM=', 'Content-Length': '0'} 2023-07-28T23:53:56.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b416d8403f464a0f-0064c45514-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:56.256 DEBUG:boto:b'hiiqos-15obj1000false' 2023-07-28T23:53:56.257 DEBUG:boto:path=/hiiqos-15/ 2023-07-28T23:53:56.257 DEBUG:boto:auth_path=/hiiqos-15/ 2023-07-28T23:53:56.258 DEBUG:boto:path=/hiiqos-15/?prefix=obj 2023-07-28T23:53:56.258 DEBUG:boto:auth_path=/hiiqos-15/?prefix=obj 2023-07-28T23:53:56.258 DEBUG:boto:Method: GET 2023-07-28T23:53:56.259 DEBUG:boto:Path: /hiiqos-15/?prefix=obj 2023-07-28T23:53:56.259 DEBUG:boto:Data: 2023-07-28T23:53:56.259 DEBUG:boto:Headers: {} 2023-07-28T23:53:56.260 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:56.260 DEBUG:boto:Port: 8000 2023-07-28T23:53:56.261 DEBUG:boto:Params: {} 2023-07-28T23:53:56.261 DEBUG:boto:Token: None 2023-07-28T23:53:56.261 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:56 GMT /hiiqos-15/ 2023-07-28T23:53:56.262 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rup/XnBRJoMmO7Q4MNiJk3AKzYM= 2023-07-28T23:53:56.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rup/XnBRJoMmO7Q4MNiJk3AKzYM=', 'Content-Length': '0'} 2023-07-28T23:53:56.265 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017b46b3b2d06cbaa-0064c45514-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:56.265 DEBUG:boto:b'hiiqos-15obj1000false' 2023-07-28T23:53:56.266 DEBUG:boto:path=/hiiqos-15/obj 2023-07-28T23:53:56.266 DEBUG:boto:auth_path=/hiiqos-15/obj 2023-07-28T23:53:56.267 DEBUG:boto:Method: HEAD 2023-07-28T23:53:56.267 DEBUG:boto:Path: /hiiqos-15/obj 2023-07-28T23:53:56.267 DEBUG:boto:Data: 2023-07-28T23:53:56.268 DEBUG:boto:Headers: {} 2023-07-28T23:53:56.268 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:56.268 DEBUG:boto:Port: 8001 2023-07-28T23:53:56.269 DEBUG:boto:Params: {} 2023-07-28T23:53:56.269 DEBUG:boto:Token: None 2023-07-28T23:53:56.269 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:53:56 GMT /hiiqos-15/obj 2023-07-28T23:53:56.270 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MlipHuqxkAR2h1JP3JLTOQpLvQc= 2023-07-28T23:53:56.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MlipHuqxkAR2h1JP3JLTOQpLvQc=', 'Content-Length': '0'} 2023-07-28T23:53:56.272 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000ef26b0f6edc153d7-0064c45514-1113-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:56.273 DEBUG:boto:path=/hiiqos-15/obj 2023-07-28T23:53:56.273 DEBUG:boto:auth_path=/hiiqos-15/obj 2023-07-28T23:53:56.273 DEBUG:boto:Method: HEAD 2023-07-28T23:53:56.274 DEBUG:boto:Path: /hiiqos-15/obj 2023-07-28T23:53:56.274 DEBUG:boto:Data: 2023-07-28T23:53:56.274 DEBUG:boto:Headers: {} 2023-07-28T23:53:56.275 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:56.275 DEBUG:boto:Port: 8000 2023-07-28T23:53:56.275 DEBUG:boto:Params: {} 2023-07-28T23:53:56.276 DEBUG:boto:Token: None 2023-07-28T23:53:56.276 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:53:56 GMT /hiiqos-15/obj 2023-07-28T23:53:56.277 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MlipHuqxkAR2h1JP3JLTOQpLvQc= 2023-07-28T23:53:56.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MlipHuqxkAR2h1JP3JLTOQpLvQc=', 'Content-Length': '0'} 2023-07-28T23:53:56.279 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000553e64149140c975-0064c45514-1117-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:56.279 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-15 zones={a2, a1} 2023-07-28T23:53:56.280 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-16 2023-07-28T23:53:56.280 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:53:56.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:53:56.281 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-16 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-28T23:53:56.341 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:53:56.341 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:53:56.376 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:53:56.374+0000 7f90caa67880 1 bucket sync caught up with source: 2023-07-28T23:53:56.377 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , 00000000002.24.6, , ] 2023-07-28T23:53:56.377 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000002.24.6, , ] 2023-07-28T23:53:56.377 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:53:56.374+0000 7f90caa67880 0 bucket checkpoint complete 2023-07-28T23:53:56.389 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-16 2023-07-28T23:53:56.389 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:53:56.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:53:56.390 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-16 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-28T23:53:56.443 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:53:56.444 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:53:56.478 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:53:56.476+0000 7fed9e218880 1 bucket sync caught up with source: 2023-07-28T23:53:56.478 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , 00000000003.46.12, , ] 2023-07-28T23:53:56.479 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000003.46.12, , ] 2023-07-28T23:53:56.479 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:53:56.476+0000 7fed9e218880 0 bucket checkpoint complete 2023-07-28T23:53:56.491 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-16 zones={a2, a1} 2023-07-28T23:53:56.492 DEBUG:boto:path=/hiiqos-16/ 2023-07-28T23:53:56.492 DEBUG:boto:auth_path=/hiiqos-16/ 2023-07-28T23:53:56.492 DEBUG:boto:Method: HEAD 2023-07-28T23:53:56.493 DEBUG:boto:Path: /hiiqos-16/ 2023-07-28T23:53:56.493 DEBUG:boto:Data: 2023-07-28T23:53:56.493 DEBUG:boto:Headers: {} 2023-07-28T23:53:56.494 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:56.494 DEBUG:boto:Port: 8001 2023-07-28T23:53:56.494 DEBUG:boto:Params: {} 2023-07-28T23:53:56.495 DEBUG:boto:Token: None 2023-07-28T23:53:56.495 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:53:56 GMT /hiiqos-16/ 2023-07-28T23:53:56.496 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:upqAD+p8WStEULGDxtVAjdu1DNo= 2023-07-28T23:53:56.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:upqAD+p8WStEULGDxtVAjdu1DNo=', 'Content-Length': '0'} 2023-07-28T23:53:56.498 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa21c1bef3e979e9-0064c45514-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:56.499 DEBUG:boto:path=/hiiqos-16/ 2023-07-28T23:53:56.499 DEBUG:boto:auth_path=/hiiqos-16/ 2023-07-28T23:53:56.499 DEBUG:boto:Method: HEAD 2023-07-28T23:53:56.500 DEBUG:boto:Path: /hiiqos-16/ 2023-07-28T23:53:56.500 DEBUG:boto:Data: 2023-07-28T23:53:56.500 DEBUG:boto:Headers: {} 2023-07-28T23:53:56.501 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:56.501 DEBUG:boto:Port: 8000 2023-07-28T23:53:56.501 DEBUG:boto:Params: {} 2023-07-28T23:53:56.502 DEBUG:boto:Token: None 2023-07-28T23:53:56.502 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:53:56 GMT /hiiqos-16/ 2023-07-28T23:53:56.503 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:upqAD+p8WStEULGDxtVAjdu1DNo= 2023-07-28T23:53:56.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:upqAD+p8WStEULGDxtVAjdu1DNo=', 'Content-Length': '0'} 2023-07-28T23:53:56.505 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a5417c9047c0df0-0064c45514-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:56.506 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-28T23:53:56.506 DEBUG:boto:path=/hiiqos-16/ 2023-07-28T23:53:56.507 DEBUG:boto:auth_path=/hiiqos-16/ 2023-07-28T23:53:56.507 DEBUG:boto:path=/hiiqos-16/?versions&max-keys=999 2023-07-28T23:53:56.507 DEBUG:boto:auth_path=/hiiqos-16/?versions&max-keys=999 2023-07-28T23:53:56.508 DEBUG:boto:Method: GET 2023-07-28T23:53:56.508 DEBUG:boto:Path: /hiiqos-16/?versions&max-keys=999 2023-07-28T23:53:56.508 DEBUG:boto:Data: 2023-07-28T23:53:56.509 DEBUG:boto:Headers: {} 2023-07-28T23:53:56.509 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:56.509 DEBUG:boto:Port: 8001 2023-07-28T23:53:56.510 DEBUG:boto:Params: {} 2023-07-28T23:53:56.510 DEBUG:boto:Token: None 2023-07-28T23:53:56.511 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:56 GMT /hiiqos-16/?versions 2023-07-28T23:53:56.511 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KKBVYKzHpWQRs6/BTsM9VIh0SG4= 2023-07-28T23:53:56.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KKBVYKzHpWQRs6/BTsM9VIh0SG4=', 'Content-Length': '0'} 2023-07-28T23:53:56.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a635ad080311d3fd-0064c45514-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:56.514 DEBUG:boto:b'hiiqos-16999falseobjFp8Y.s8tSo9pMrSp7xiIjUzSdsmYyCmtrue2023-07-28T23:53:20.821Zrgw-multisite-test-userMultisite Test UserNormalobjarpVnoFZTsBYgcmcJ6HYUv2zgJJSiqLfalse2023-07-28T23:53:20.800Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:53:56.514 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:53:56.515 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:53:56.515 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-28T23:53:56.515 DEBUG:boto:path=/hiiqos-16/ 2023-07-28T23:53:56.516 DEBUG:boto:auth_path=/hiiqos-16/ 2023-07-28T23:53:56.516 DEBUG:boto:path=/hiiqos-16/?versions&max-keys=999 2023-07-28T23:53:56.516 DEBUG:boto:auth_path=/hiiqos-16/?versions&max-keys=999 2023-07-28T23:53:56.517 DEBUG:boto:Method: GET 2023-07-28T23:53:56.517 DEBUG:boto:Path: /hiiqos-16/?versions&max-keys=999 2023-07-28T23:53:56.517 DEBUG:boto:Data: 2023-07-28T23:53:56.518 DEBUG:boto:Headers: {} 2023-07-28T23:53:56.518 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:56.518 DEBUG:boto:Port: 8000 2023-07-28T23:53:56.519 DEBUG:boto:Params: {} 2023-07-28T23:53:56.519 DEBUG:boto:Token: None 2023-07-28T23:53:56.520 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:56 GMT /hiiqos-16/?versions 2023-07-28T23:53:56.520 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KKBVYKzHpWQRs6/BTsM9VIh0SG4= 2023-07-28T23:53:56.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KKBVYKzHpWQRs6/BTsM9VIh0SG4=', 'Content-Length': '0'} 2023-07-28T23:53:56.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfdb495cebf141fd-0064c45514-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:56.523 DEBUG:boto:b'hiiqos-16999falseobjFp8Y.s8tSo9pMrSp7xiIjUzSdsmYyCmtrue2023-07-28T23:53:20.821Zrgw-multisite-test-userMultisite Test UserNormalobjarpVnoFZTsBYgcmcJ6HYUv2zgJJSiqLfalse2023-07-28T23:53:20.800Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:53:56.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:53:56.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:53:56.524 DEBUG:boto:path=/hiiqos-16/ 2023-07-28T23:53:56.525 DEBUG:boto:auth_path=/hiiqos-16/ 2023-07-28T23:53:56.525 DEBUG:boto:path=/hiiqos-16/?versions&max-keys=999 2023-07-28T23:53:56.525 DEBUG:boto:auth_path=/hiiqos-16/?versions&max-keys=999 2023-07-28T23:53:56.526 DEBUG:boto:Method: GET 2023-07-28T23:53:56.526 DEBUG:boto:Path: /hiiqos-16/?versions&max-keys=999 2023-07-28T23:53:56.526 DEBUG:boto:Data: 2023-07-28T23:53:56.527 DEBUG:boto:Headers: {} 2023-07-28T23:53:56.527 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:56.527 DEBUG:boto:Port: 8001 2023-07-28T23:53:56.528 DEBUG:boto:Params: {} 2023-07-28T23:53:56.528 DEBUG:boto:Token: None 2023-07-28T23:53:56.529 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:56 GMT /hiiqos-16/?versions 2023-07-28T23:53:56.529 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KKBVYKzHpWQRs6/BTsM9VIh0SG4= 2023-07-28T23:53:56.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KKBVYKzHpWQRs6/BTsM9VIh0SG4=', 'Content-Length': '0'} 2023-07-28T23:53:56.532 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ed62151fcf3ff93-0064c45514-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:56.532 DEBUG:boto:b'hiiqos-16999falseobjFp8Y.s8tSo9pMrSp7xiIjUzSdsmYyCmtrue2023-07-28T23:53:20.821Zrgw-multisite-test-userMultisite Test UserNormalobjarpVnoFZTsBYgcmcJ6HYUv2zgJJSiqLfalse2023-07-28T23:53:20.800Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:53:56.533 DEBUG:boto:path=/hiiqos-16/ 2023-07-28T23:53:56.533 DEBUG:boto:auth_path=/hiiqos-16/ 2023-07-28T23:53:56.534 DEBUG:boto:path=/hiiqos-16/?versions&max-keys=999 2023-07-28T23:53:56.534 DEBUG:boto:auth_path=/hiiqos-16/?versions&max-keys=999 2023-07-28T23:53:56.534 DEBUG:boto:Method: GET 2023-07-28T23:53:56.535 DEBUG:boto:Path: /hiiqos-16/?versions&max-keys=999 2023-07-28T23:53:56.535 DEBUG:boto:Data: 2023-07-28T23:53:56.535 DEBUG:boto:Headers: {} 2023-07-28T23:53:56.536 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:56.536 DEBUG:boto:Port: 8000 2023-07-28T23:53:56.536 DEBUG:boto:Params: {} 2023-07-28T23:53:56.537 DEBUG:boto:Token: None 2023-07-28T23:53:56.537 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:56 GMT /hiiqos-16/?versions 2023-07-28T23:53:56.538 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KKBVYKzHpWQRs6/BTsM9VIh0SG4= 2023-07-28T23:53:56.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KKBVYKzHpWQRs6/BTsM9VIh0SG4=', 'Content-Length': '0'} 2023-07-28T23:53:56.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f242acf425caa529-0064c45514-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:56.540 DEBUG:boto:b'hiiqos-16999falseobjFp8Y.s8tSo9pMrSp7xiIjUzSdsmYyCmtrue2023-07-28T23:53:20.821Zrgw-multisite-test-userMultisite Test UserNormalobjarpVnoFZTsBYgcmcJ6HYUv2zgJJSiqLfalse2023-07-28T23:53:20.800Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:53:56.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:53:56.541 DEBUG:boto:path=/hiiqos-16/obj 2023-07-28T23:53:56.542 DEBUG:boto:auth_path=/hiiqos-16/obj 2023-07-28T23:53:56.542 DEBUG:boto:Method: HEAD 2023-07-28T23:53:56.542 DEBUG:boto:Path: /hiiqos-16/obj 2023-07-28T23:53:56.543 DEBUG:boto:Data: 2023-07-28T23:53:56.543 DEBUG:boto:Headers: {} 2023-07-28T23:53:56.543 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:56.544 DEBUG:boto:Port: 8001 2023-07-28T23:53:56.544 DEBUG:boto:Params: {} 2023-07-28T23:53:56.544 DEBUG:boto:Token: None 2023-07-28T23:53:56.545 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:53:56 GMT /hiiqos-16/obj 2023-07-28T23:53:56.545 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wTY0ivNalj1+FObAcV4LfU1BU3I= 2023-07-28T23:53:56.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wTY0ivNalj1+FObAcV4LfU1BU3I=', 'Content-Length': '0'} 2023-07-28T23:53:56.547 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000068c737eebb36ea28-0064c45514-1113-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:56.548 DEBUG:boto:path=/hiiqos-16/obj 2023-07-28T23:53:56.548 DEBUG:boto:auth_path=/hiiqos-16/obj 2023-07-28T23:53:56.549 DEBUG:boto:Method: HEAD 2023-07-28T23:53:56.549 DEBUG:boto:Path: /hiiqos-16/obj 2023-07-28T23:53:56.549 DEBUG:boto:Data: 2023-07-28T23:53:56.549 DEBUG:boto:Headers: {} 2023-07-28T23:53:56.550 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:56.550 DEBUG:boto:Port: 8000 2023-07-28T23:53:56.550 DEBUG:boto:Params: {} 2023-07-28T23:53:56.551 DEBUG:boto:Token: None 2023-07-28T23:53:56.551 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:53:56 GMT /hiiqos-16/obj 2023-07-28T23:53:56.552 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wTY0ivNalj1+FObAcV4LfU1BU3I= 2023-07-28T23:53:56.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wTY0ivNalj1+FObAcV4LfU1BU3I=', 'Content-Length': '0'} 2023-07-28T23:53:56.554 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000e5543ba9e4e4b8dc-0064c45514-1117-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:56.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:53:56.555 DEBUG:boto:path=/hiiqos-16/obj 2023-07-28T23:53:56.555 DEBUG:boto:auth_path=/hiiqos-16/obj 2023-07-28T23:53:56.556 DEBUG:boto:path=/hiiqos-16/obj?versionId=arpVnoFZTsBYgcmcJ6HYUv2zgJJSiqL 2023-07-28T23:53:56.556 DEBUG:boto:auth_path=/hiiqos-16/obj?versionId=arpVnoFZTsBYgcmcJ6HYUv2zgJJSiqL 2023-07-28T23:53:56.556 DEBUG:boto:Method: GET 2023-07-28T23:53:56.557 DEBUG:boto:Path: /hiiqos-16/obj?versionId=arpVnoFZTsBYgcmcJ6HYUv2zgJJSiqL 2023-07-28T23:53:56.557 DEBUG:boto:Data: 2023-07-28T23:53:56.557 DEBUG:boto:Headers: {} 2023-07-28T23:53:56.558 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:56.558 DEBUG:boto:Port: 8001 2023-07-28T23:53:56.558 DEBUG:boto:Params: {} 2023-07-28T23:53:56.559 DEBUG:boto:Token: None 2023-07-28T23:53:56.559 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:56 GMT /hiiqos-16/obj?versionId=arpVnoFZTsBYgcmcJ6HYUv2zgJJSiqL 2023-07-28T23:53:56.560 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2lQTKudroWdpChArj3uNSWOoN2s= 2023-07-28T23:53:56.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2lQTKudroWdpChArj3uNSWOoN2s=', 'Content-Length': '0'} 2023-07-28T23:53:56.562 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:53:20 GMT'), ('x-amz-version-id', 'arpVnoFZTsBYgcmcJ6HYUv2zgJJSiqL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004ce8d52460609f89-0064c45514-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:56.563 DEBUG:boto:path=/hiiqos-16/obj 2023-07-28T23:53:56.563 DEBUG:boto:auth_path=/hiiqos-16/obj 2023-07-28T23:53:56.563 DEBUG:boto:path=/hiiqos-16/obj?versionId=arpVnoFZTsBYgcmcJ6HYUv2zgJJSiqL 2023-07-28T23:53:56.564 DEBUG:boto:auth_path=/hiiqos-16/obj?versionId=arpVnoFZTsBYgcmcJ6HYUv2zgJJSiqL 2023-07-28T23:53:56.564 DEBUG:boto:Method: GET 2023-07-28T23:53:56.565 DEBUG:boto:Path: /hiiqos-16/obj?versionId=arpVnoFZTsBYgcmcJ6HYUv2zgJJSiqL 2023-07-28T23:53:56.565 DEBUG:boto:Data: 2023-07-28T23:53:56.565 DEBUG:boto:Headers: {} 2023-07-28T23:53:56.566 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:56.566 DEBUG:boto:Port: 8000 2023-07-28T23:53:56.566 DEBUG:boto:Params: {} 2023-07-28T23:53:56.567 DEBUG:boto:Token: None 2023-07-28T23:53:56.567 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:56 GMT /hiiqos-16/obj?versionId=arpVnoFZTsBYgcmcJ6HYUv2zgJJSiqL 2023-07-28T23:53:56.568 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2lQTKudroWdpChArj3uNSWOoN2s= 2023-07-28T23:53:56.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2lQTKudroWdpChArj3uNSWOoN2s=', 'Content-Length': '0'} 2023-07-28T23:53:56.570 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:53:20 GMT'), ('x-amz-version-id', 'arpVnoFZTsBYgcmcJ6HYUv2zgJJSiqL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-16:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008c55243a22288240-0064c45514-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:56.571 DEBUG:boto:path=/hiiqos-16/obj 2023-07-28T23:53:56.571 DEBUG:boto:auth_path=/hiiqos-16/obj 2023-07-28T23:53:56.571 DEBUG:boto:path=/hiiqos-16/obj?versionId=arpVnoFZTsBYgcmcJ6HYUv2zgJJSiqL 2023-07-28T23:53:56.572 DEBUG:boto:auth_path=/hiiqos-16/obj?versionId=arpVnoFZTsBYgcmcJ6HYUv2zgJJSiqL 2023-07-28T23:53:56.572 DEBUG:boto:Method: HEAD 2023-07-28T23:53:56.572 DEBUG:boto:Path: /hiiqos-16/obj?versionId=arpVnoFZTsBYgcmcJ6HYUv2zgJJSiqL 2023-07-28T23:53:56.573 DEBUG:boto:Data: 2023-07-28T23:53:56.573 DEBUG:boto:Headers: {} 2023-07-28T23:53:56.573 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:56.574 DEBUG:boto:Port: 8001 2023-07-28T23:53:56.574 DEBUG:boto:Params: {} 2023-07-28T23:53:56.574 DEBUG:boto:Token: None 2023-07-28T23:53:56.575 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:53:56 GMT /hiiqos-16/obj?versionId=arpVnoFZTsBYgcmcJ6HYUv2zgJJSiqL 2023-07-28T23:53:56.575 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ag6uVv1J31JxjwcLPriWQ7W+3+c= 2023-07-28T23:53:56.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ag6uVv1J31JxjwcLPriWQ7W+3+c=', 'Content-Length': '0'} 2023-07-28T23:53:56.577 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:53:20 GMT'), ('x-amz-version-id', 'arpVnoFZTsBYgcmcJ6HYUv2zgJJSiqL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007d4183bebb5c9aef-0064c45514-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:56.578 DEBUG:boto:path=/hiiqos-16/obj 2023-07-28T23:53:56.578 DEBUG:boto:auth_path=/hiiqos-16/obj 2023-07-28T23:53:56.579 DEBUG:boto:path=/hiiqos-16/obj?versionId=arpVnoFZTsBYgcmcJ6HYUv2zgJJSiqL 2023-07-28T23:53:56.579 DEBUG:boto:auth_path=/hiiqos-16/obj?versionId=arpVnoFZTsBYgcmcJ6HYUv2zgJJSiqL 2023-07-28T23:53:56.579 DEBUG:boto:Method: HEAD 2023-07-28T23:53:56.580 DEBUG:boto:Path: /hiiqos-16/obj?versionId=arpVnoFZTsBYgcmcJ6HYUv2zgJJSiqL 2023-07-28T23:53:56.580 DEBUG:boto:Data: 2023-07-28T23:53:56.580 DEBUG:boto:Headers: {} 2023-07-28T23:53:56.581 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:56.581 DEBUG:boto:Port: 8000 2023-07-28T23:53:56.581 DEBUG:boto:Params: {} 2023-07-28T23:53:56.582 DEBUG:boto:Token: None 2023-07-28T23:53:56.582 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:53:56 GMT /hiiqos-16/obj?versionId=arpVnoFZTsBYgcmcJ6HYUv2zgJJSiqL 2023-07-28T23:53:56.582 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ag6uVv1J31JxjwcLPriWQ7W+3+c= 2023-07-28T23:53:56.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ag6uVv1J31JxjwcLPriWQ7W+3+c=', 'Content-Length': '0'} 2023-07-28T23:53:56.584 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:53:20 GMT'), ('x-amz-version-id', 'arpVnoFZTsBYgcmcJ6HYUv2zgJJSiqL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-16:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003a46958576eb4829-0064c45514-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:56.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:53:56.585 DEBUG:boto:path=/hiiqos-16/obj 2023-07-28T23:53:56.585 DEBUG:boto:auth_path=/hiiqos-16/obj 2023-07-28T23:53:56.586 DEBUG:boto:path=/hiiqos-16/obj?versionId=arpVnoFZTsBYgcmcJ6HYUv2zgJJSiqL 2023-07-28T23:53:56.586 DEBUG:boto:auth_path=/hiiqos-16/obj?versionId=arpVnoFZTsBYgcmcJ6HYUv2zgJJSiqL 2023-07-28T23:53:56.586 DEBUG:boto:Method: GET 2023-07-28T23:53:56.587 DEBUG:boto:Path: /hiiqos-16/obj?versionId=arpVnoFZTsBYgcmcJ6HYUv2zgJJSiqL 2023-07-28T23:53:56.587 DEBUG:boto:Data: 2023-07-28T23:53:56.588 DEBUG:boto:Headers: {} 2023-07-28T23:53:56.588 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:56.588 DEBUG:boto:Port: 8001 2023-07-28T23:53:56.589 DEBUG:boto:Params: {} 2023-07-28T23:53:56.589 DEBUG:boto:Token: None 2023-07-28T23:53:56.589 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:56 GMT /hiiqos-16/obj?versionId=arpVnoFZTsBYgcmcJ6HYUv2zgJJSiqL 2023-07-28T23:53:56.590 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2lQTKudroWdpChArj3uNSWOoN2s= 2023-07-28T23:53:56.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2lQTKudroWdpChArj3uNSWOoN2s=', 'Content-Length': '0'} 2023-07-28T23:53:56.592 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:53:20 GMT'), ('x-amz-version-id', 'arpVnoFZTsBYgcmcJ6HYUv2zgJJSiqL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fd3e3fac8e97c4e8-0064c45514-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:56.593 DEBUG:boto:path=/hiiqos-16/obj 2023-07-28T23:53:56.593 DEBUG:boto:auth_path=/hiiqos-16/obj 2023-07-28T23:53:56.593 DEBUG:boto:path=/hiiqos-16/obj?versionId=arpVnoFZTsBYgcmcJ6HYUv2zgJJSiqL 2023-07-28T23:53:56.594 DEBUG:boto:auth_path=/hiiqos-16/obj?versionId=arpVnoFZTsBYgcmcJ6HYUv2zgJJSiqL 2023-07-28T23:53:56.594 DEBUG:boto:Method: GET 2023-07-28T23:53:56.594 DEBUG:boto:Path: /hiiqos-16/obj?versionId=arpVnoFZTsBYgcmcJ6HYUv2zgJJSiqL 2023-07-28T23:53:56.595 DEBUG:boto:Data: 2023-07-28T23:53:56.595 DEBUG:boto:Headers: {} 2023-07-28T23:53:56.595 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:56.596 DEBUG:boto:Port: 8000 2023-07-28T23:53:56.596 DEBUG:boto:Params: {} 2023-07-28T23:53:56.597 DEBUG:boto:Token: None 2023-07-28T23:53:56.597 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:56 GMT /hiiqos-16/obj?versionId=arpVnoFZTsBYgcmcJ6HYUv2zgJJSiqL 2023-07-28T23:53:56.597 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2lQTKudroWdpChArj3uNSWOoN2s= 2023-07-28T23:53:56.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2lQTKudroWdpChArj3uNSWOoN2s=', 'Content-Length': '0'} 2023-07-28T23:53:56.599 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:53:20 GMT'), ('x-amz-version-id', 'arpVnoFZTsBYgcmcJ6HYUv2zgJJSiqL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-16:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000008fff16212813735-0064c45514-1117-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:56.600 DEBUG:boto:path=/hiiqos-16/ 2023-07-28T23:53:56.600 DEBUG:boto:auth_path=/hiiqos-16/ 2023-07-28T23:53:56.601 DEBUG:boto:path=/hiiqos-16/?prefix=obj 2023-07-28T23:53:56.601 DEBUG:boto:auth_path=/hiiqos-16/?prefix=obj 2023-07-28T23:53:56.601 DEBUG:boto:Method: GET 2023-07-28T23:53:56.602 DEBUG:boto:Path: /hiiqos-16/?prefix=obj 2023-07-28T23:53:56.602 DEBUG:boto:Data: 2023-07-28T23:53:56.602 DEBUG:boto:Headers: {} 2023-07-28T23:53:56.603 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:56.603 DEBUG:boto:Port: 8001 2023-07-28T23:53:56.604 DEBUG:boto:Params: {} 2023-07-28T23:53:56.604 DEBUG:boto:Token: None 2023-07-28T23:53:56.604 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:56 GMT /hiiqos-16/ 2023-07-28T23:53:56.605 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ppojIuHq6LNPGTbjoo6l5MPQBsM= 2023-07-28T23:53:56.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ppojIuHq6LNPGTbjoo6l5MPQBsM=', 'Content-Length': '0'} 2023-07-28T23:53:56.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000003d83c00c339765-0064c45514-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:56.608 DEBUG:boto:b'hiiqos-16obj1000false' 2023-07-28T23:53:56.609 DEBUG:boto:path=/hiiqos-16/ 2023-07-28T23:53:56.609 DEBUG:boto:auth_path=/hiiqos-16/ 2023-07-28T23:53:56.609 DEBUG:boto:path=/hiiqos-16/?prefix=obj 2023-07-28T23:53:56.610 DEBUG:boto:auth_path=/hiiqos-16/?prefix=obj 2023-07-28T23:53:56.610 DEBUG:boto:Method: GET 2023-07-28T23:53:56.610 DEBUG:boto:Path: /hiiqos-16/?prefix=obj 2023-07-28T23:53:56.611 DEBUG:boto:Data: 2023-07-28T23:53:56.611 DEBUG:boto:Headers: {} 2023-07-28T23:53:56.611 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:56.612 DEBUG:boto:Port: 8000 2023-07-28T23:53:56.612 DEBUG:boto:Params: {} 2023-07-28T23:53:56.613 DEBUG:boto:Token: None 2023-07-28T23:53:56.613 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:53:56 GMT /hiiqos-16/ 2023-07-28T23:53:56.613 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ppojIuHq6LNPGTbjoo6l5MPQBsM= 2023-07-28T23:53:56.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ppojIuHq6LNPGTbjoo6l5MPQBsM=', 'Content-Length': '0'} 2023-07-28T23:53:56.616 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064d1d838864f197d-0064c45514-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:56.617 DEBUG:boto:b'hiiqos-16obj1000false' 2023-07-28T23:53:56.617 DEBUG:boto:path=/hiiqos-16/obj 2023-07-28T23:53:56.617 DEBUG:boto:auth_path=/hiiqos-16/obj 2023-07-28T23:53:56.618 DEBUG:boto:Method: HEAD 2023-07-28T23:53:56.618 DEBUG:boto:Path: /hiiqos-16/obj 2023-07-28T23:53:56.619 DEBUG:boto:Data: 2023-07-28T23:53:56.619 DEBUG:boto:Headers: {} 2023-07-28T23:53:56.619 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:56.619 DEBUG:boto:Port: 8001 2023-07-28T23:53:56.620 DEBUG:boto:Params: {} 2023-07-28T23:53:56.620 DEBUG:boto:Token: None 2023-07-28T23:53:56.621 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:53:56 GMT /hiiqos-16/obj 2023-07-28T23:53:56.621 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wTY0ivNalj1+FObAcV4LfU1BU3I= 2023-07-28T23:53:56.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wTY0ivNalj1+FObAcV4LfU1BU3I=', 'Content-Length': '0'} 2023-07-28T23:53:56.623 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000004a1dd54230c1bdb2-0064c45514-1113-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:56.624 DEBUG:boto:path=/hiiqos-16/obj 2023-07-28T23:53:56.624 DEBUG:boto:auth_path=/hiiqos-16/obj 2023-07-28T23:53:56.624 DEBUG:boto:Method: HEAD 2023-07-28T23:53:56.625 DEBUG:boto:Path: /hiiqos-16/obj 2023-07-28T23:53:56.625 DEBUG:boto:Data: 2023-07-28T23:53:56.625 DEBUG:boto:Headers: {} 2023-07-28T23:53:56.626 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:56.626 DEBUG:boto:Port: 8000 2023-07-28T23:53:56.626 DEBUG:boto:Params: {} 2023-07-28T23:53:56.627 DEBUG:boto:Token: None 2023-07-28T23:53:56.627 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:53:56 GMT /hiiqos-16/obj 2023-07-28T23:53:56.628 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wTY0ivNalj1+FObAcV4LfU1BU3I= 2023-07-28T23:53:56.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wTY0ivNalj1+FObAcV4LfU1BU3I=', 'Content-Length': '0'} 2023-07-28T23:53:56.630 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000098a3c077ead7c1a-0064c45514-1117-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:56.630 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-16 zones={a2, a1} 2023-07-28T23:53:56.630 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_delete_marker_full_sync ... ok 2023-07-28T23:53:56.631 INFO:rgw_multi.tests:create bucket zone=a1 name=hiiqos-17 2023-07-28T23:53:56.632 DEBUG:boto:path=/hiiqos-17/ 2023-07-28T23:53:56.632 DEBUG:boto:auth_path=/hiiqos-17/ 2023-07-28T23:53:56.632 DEBUG:boto:Method: PUT 2023-07-28T23:53:56.633 DEBUG:boto:Path: /hiiqos-17/ 2023-07-28T23:53:56.633 DEBUG:boto:Data: 2023-07-28T23:53:56.633 DEBUG:boto:Headers: {} 2023-07-28T23:53:56.634 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:56.634 DEBUG:boto:Port: 8000 2023-07-28T23:53:56.634 DEBUG:boto:Params: {} 2023-07-28T23:53:56.635 DEBUG:boto:Token: None 2023-07-28T23:53:56.635 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:53:56 GMT /hiiqos-17/ 2023-07-28T23:53:56.636 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zmp89idy6QCUX3jiCR/+OXcebWg= 2023-07-28T23:53:56.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zmp89idy6QCUX3jiCR/+OXcebWg=', 'Content-Length': '0'} 2023-07-28T23:53:56.653 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005548ba4fde5e2933-0064c45514-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:56.654 INFO:rgw_multi.tests:create bucket zone=a2 name=hiiqos-18 2023-07-28T23:53:56.654 DEBUG:boto:path=/hiiqos-18/ 2023-07-28T23:53:56.655 DEBUG:boto:auth_path=/hiiqos-18/ 2023-07-28T23:53:56.655 DEBUG:boto:Method: PUT 2023-07-28T23:53:56.655 DEBUG:boto:Path: /hiiqos-18/ 2023-07-28T23:53:56.656 DEBUG:boto:Data: 2023-07-28T23:53:56.656 DEBUG:boto:Headers: {} 2023-07-28T23:53:56.656 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:56.657 DEBUG:boto:Port: 8001 2023-07-28T23:53:56.657 DEBUG:boto:Params: {} 2023-07-28T23:53:56.657 DEBUG:boto:Token: None 2023-07-28T23:53:56.658 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:53:56 GMT /hiiqos-18/ 2023-07-28T23:53:56.658 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sZxvWWdd1Zg//dRTD3rQBX0cfYY= 2023-07-28T23:53:56.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sZxvWWdd1Zg//dRTD3rQBX0cfYY=', 'Content-Length': '0'} 2023-07-28T23:53:56.684 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007e4b605294161ac8-0064c45514-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:56.684 DEBUG:boto:path=/hiiqos-17/ 2023-07-28T23:53:56.685 DEBUG:boto:auth_path=/hiiqos-17/ 2023-07-28T23:53:56.685 DEBUG:boto:path=/hiiqos-17/?versioning 2023-07-28T23:53:56.685 DEBUG:boto:auth_path=/hiiqos-17/?versioning 2023-07-28T23:53:56.686 DEBUG:boto:Method: PUT 2023-07-28T23:53:56.686 DEBUG:boto:Path: /hiiqos-17/?versioning 2023-07-28T23:53:56.686 DEBUG:boto:Data: Enabled Disabled 2023-07-28T23:53:56.687 DEBUG:boto:Headers: {} 2023-07-28T23:53:56.687 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:56.687 DEBUG:boto:Port: 8000 2023-07-28T23:53:56.688 DEBUG:boto:Params: {} 2023-07-28T23:53:56.688 DEBUG:boto:Token: None 2023-07-28T23:53:56.688 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:53:56 GMT /hiiqos-17/?versioning 2023-07-28T23:53:56.689 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xo0kCEmKB9Q886SxgCgYug3PFI8= 2023-07-28T23:53:56.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xo0kCEmKB9Q886SxgCgYug3PFI8=', 'Content-Length': '228'} 2023-07-28T23:53:56.694 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d8298f7ec074e5ed-0064c45514-1117-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:56.695 DEBUG:boto:path=/hiiqos-17/ 2023-07-28T23:53:56.696 DEBUG:boto:auth_path=/hiiqos-17/ 2023-07-28T23:53:56.697 DEBUG:boto:path=/hiiqos-17/?versioning 2023-07-28T23:53:56.697 DEBUG:boto:auth_path=/hiiqos-17/?versioning 2023-07-28T23:53:56.698 DEBUG:boto:Method: PUT 2023-07-28T23:53:56.699 DEBUG:boto:Path: /hiiqos-17/?versioning 2023-07-28T23:53:56.700 DEBUG:boto:Data: Suspended Disabled 2023-07-28T23:53:56.701 DEBUG:boto:Headers: {} 2023-07-28T23:53:56.701 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:53:56.702 DEBUG:boto:Port: 8000 2023-07-28T23:53:56.702 DEBUG:boto:Params: {} 2023-07-28T23:53:56.702 DEBUG:boto:Token: None 2023-07-28T23:53:56.703 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:53:56 GMT /hiiqos-17/?versioning 2023-07-28T23:53:56.703 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xo0kCEmKB9Q886SxgCgYug3PFI8= 2023-07-28T23:53:56.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xo0kCEmKB9Q886SxgCgYug3PFI8=', 'Content-Length': '230'} 2023-07-28T23:53:56.708 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000eebfb27d9bc397ce-0064c45514-1117-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:56.709 DEBUG:boto:path=/hiiqos-18/ 2023-07-28T23:53:56.709 DEBUG:boto:auth_path=/hiiqos-18/ 2023-07-28T23:53:56.710 DEBUG:boto:path=/hiiqos-18/?versioning 2023-07-28T23:53:56.711 DEBUG:boto:auth_path=/hiiqos-18/?versioning 2023-07-28T23:53:56.711 DEBUG:boto:Method: PUT 2023-07-28T23:53:56.712 DEBUG:boto:Path: /hiiqos-18/?versioning 2023-07-28T23:53:56.713 DEBUG:boto:Data: Enabled Disabled 2023-07-28T23:53:56.713 DEBUG:boto:Headers: {} 2023-07-28T23:53:56.714 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:56.715 DEBUG:boto:Port: 8001 2023-07-28T23:53:56.715 DEBUG:boto:Params: {} 2023-07-28T23:53:56.716 DEBUG:boto:Token: None 2023-07-28T23:53:56.717 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:53:56 GMT /hiiqos-18/?versioning 2023-07-28T23:53:56.718 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Q9OA5iCPznFHg1bRgNGdIJD9ABI= 2023-07-28T23:53:56.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q9OA5iCPznFHg1bRgNGdIJD9ABI=', 'Content-Length': '228'} 2023-07-28T23:53:56.725 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b88100dad8bd1abf-0064c45514-1113-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:56.726 DEBUG:boto:path=/hiiqos-18/ 2023-07-28T23:53:56.727 DEBUG:boto:auth_path=/hiiqos-18/ 2023-07-28T23:53:56.727 DEBUG:boto:path=/hiiqos-18/?versioning 2023-07-28T23:53:56.728 DEBUG:boto:auth_path=/hiiqos-18/?versioning 2023-07-28T23:53:56.729 DEBUG:boto:Method: PUT 2023-07-28T23:53:56.730 DEBUG:boto:Path: /hiiqos-18/?versioning 2023-07-28T23:53:56.730 DEBUG:boto:Data: Suspended Disabled 2023-07-28T23:53:56.731 DEBUG:boto:Headers: {} 2023-07-28T23:53:56.732 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:53:56.732 DEBUG:boto:Port: 8001 2023-07-28T23:53:56.733 DEBUG:boto:Params: {} 2023-07-28T23:53:56.733 DEBUG:boto:Token: None 2023-07-28T23:53:56.734 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:53:56 GMT /hiiqos-18/?versioning 2023-07-28T23:53:56.734 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Q9OA5iCPznFHg1bRgNGdIJD9ABI= 2023-07-28T23:53:56.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:53:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q9OA5iCPznFHg1bRgNGdIJD9ABI=', 'Content-Length': '230'} 2023-07-28T23:53:56.741 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008045e13c56771e75-0064c45514-1113-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:53:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:53:56.742 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:53:56.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:53:56.743 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:53:56.791 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:53:56.791 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:53:56.820 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-28T23:53:56.842 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-28T23:53:56.842 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:53:56.843 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-28T23:53:56.843 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:46:09.233742Z" 2023-07-28T23:53:56.843 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:53:56.843 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:53:56.843 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588436.738614_242.1", 2023-07-28T23:53:56.843 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:53:56.738614Z" 2023-07-28T23:53:56.843 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:53:56.844 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:53:56.844 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:53:56.844 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:53:56.844 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:53:56.844 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:53:56.844 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:53:56.845 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:53:56.845 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:53:56.845 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-28T23:53:56.846 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690587969.233742_99.1', 1: '1_1690588436.738614_242.1', 2: '', 3: ''} 2023-07-28T23:53:56.846 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-28T23:53:56.847 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:53:56.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:53:56.847 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:53:56.893 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:53:56.893 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:53:56.920 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:53:56.918+0000 7f2218522880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:53:56.921 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:53:56.919+0000 7f2218522880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:53:56.932 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:53:56.933 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:53:56.933 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:53:56.933 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:53:56.933 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:53:56.933 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:53:56.934 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:53:56.934 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:53:56.934 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:53:56.934 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:53:56.934 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:53:56.934 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:53:56.934 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:53:56.935 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-28T23:53:56.935 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:53:56.935 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:53:56.935 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:53:56.935 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-28T23:53:56.935 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:53:56.935 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:53:56.936 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:53:56.936 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:53:56.936 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:53:56.936 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:53:56.936 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:53:56.936 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588436.645089_224.1", 2023-07-28T23:53:56.937 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:53:56.937 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:53:56.937 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:53:56.937 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:53:56.645089Z", 2023-07-28T23:53:56.937 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:53:56.937 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:53:56.937 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:53:56.938 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:53:56.938 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:53:56.938 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:53:56.938 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:53:56.938 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:53:56.938 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:53:56.938 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:53:56.939 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:53:56.939 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:53:56.939 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:53:56.939 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:53:56.939 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:53:56.939 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:53:56.939 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:53:56.940 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:53:56.940 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:53:56.940 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:53:56.940 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:53:56.940 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:53:56.940 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:53:56.940 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:53:56.941 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:53:56.941 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:53:56.941 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:53:56.941 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:53:56.941 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:53:56.941 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:53:56.941 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:53:56.942 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:53:56.942 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:53:56.942 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:53:56Z" 2023-07-28T23:53:56.942 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:53:56.943 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588436.645089_224.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:53:56.645089Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-28T23:53:56Z" } 2023-07-28T23:53:56.943 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588436.645089_224.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:53:56.645089Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-28T23:53:56.944 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588436.738614_242.1', 2: '', 3: ''} 2023-07-28T23:53:56.944 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588436.645089_224.1', 2: '', 3: ''} 2023-07-28T23:53:56.945 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588436.738614_242.1 target=1_1690588436.645089_224.1 2023-07-28T23:54:01.946 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:54:01.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:54:01.947 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:54:02.000 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:54:02.000 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:54:02.031 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:54:02.029+0000 7f7cc9810880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:54:02.032 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:54:02.030+0000 7f7cc9810880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:54:02.044 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:54:02.044 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:54:02.044 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:54:02.045 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:54:02.045 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:54:02.045 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:54:02.045 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:54:02.045 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:02.045 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:54:02.046 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:02.046 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:54:02.046 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:54:02.046 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:54:02.047 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-28T23:54:02.047 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:54:02.047 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:54:02.047 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:54:02.047 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-28T23:54:02.047 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:54:02.047 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:02.048 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:02.048 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:02.048 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:54:02.048 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:54:02.048 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:54:02.048 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588436.738614_242.1", 2023-07-28T23:54:02.049 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:54:02.049 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:54:02.049 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:54:02.049 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:53:56.738614Z", 2023-07-28T23:54:02.049 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:54:02.049 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:02.049 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:02.050 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:02.050 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:54:02.050 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:54:02.050 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:54:02.050 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:54:02.050 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:54:02.050 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:54:02.051 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:54:02.051 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:54:02.051 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:54:02.051 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:02.051 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:02.051 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:02.051 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:54:02.052 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:54:02.052 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:54:02.052 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:54:02.052 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:54:02.052 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:54:02.052 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:54:02.052 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:54:02.053 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:54:02.053 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:02.053 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:02.053 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:54:02.053 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:02.053 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:54:02.053 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:54:02.054 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:54:02.054 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:02.054 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:54:02Z" 2023-07-28T23:54:02.054 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:54:02.055 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588436.738614_242.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:53:56.738614Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-28T23:54:02Z" } 2023-07-28T23:54:02.055 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588436.738614_242.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:53:56.738614Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-28T23:54:02.056 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588436.738614_242.1', 2: '', 3: ''} 2023-07-28T23:54:02.056 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588436.738614_242.1', 2: '', 3: ''} 2023-07-28T23:54:02.056 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-28T23:54:02.057 DEBUG:boto:path=/hiiqos-17/ 2023-07-28T23:54:02.057 DEBUG:boto:auth_path=/hiiqos-17/ 2023-07-28T23:54:02.057 DEBUG:boto:Method: HEAD 2023-07-28T23:54:02.058 DEBUG:boto:Path: /hiiqos-17/ 2023-07-28T23:54:02.058 DEBUG:boto:Data: 2023-07-28T23:54:02.058 DEBUG:boto:Headers: {} 2023-07-28T23:54:02.059 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:54:02.059 DEBUG:boto:Port: 8000 2023-07-28T23:54:02.059 DEBUG:boto:Params: {} 2023-07-28T23:54:02.060 DEBUG:boto:Token: None 2023-07-28T23:54:02.060 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:54:02 GMT /hiiqos-17/ 2023-07-28T23:54:02.061 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FzKjHT1YoB40z0WpFx1RugA3/Vs= 2023-07-28T23:54:02.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FzKjHT1YoB40z0WpFx1RugA3/Vs=', 'Content-Length': '0'} 2023-07-28T23:54:02.063 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000468f5c9141d043e2-0064c4551a-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:02.064 DEBUG:boto:path=/hiiqos-17/obj 2023-07-28T23:54:02.064 DEBUG:boto:auth_path=/hiiqos-17/obj 2023-07-28T23:54:02.065 DEBUG:boto:Method: PUT 2023-07-28T23:54:02.065 DEBUG:boto:Path: /hiiqos-17/obj 2023-07-28T23:54:02.065 DEBUG:boto:Data: 2023-07-28T23:54:02.066 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-07-28T23:54:02.066 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:54:02.066 DEBUG:boto:Port: 8000 2023-07-28T23:54:02.067 DEBUG:boto:Params: {} 2023-07-28T23:54:02.067 DEBUG:boto:Token: None 2023-07-28T23:54:02.068 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 28 Jul 2023 23:54:02 GMT /hiiqos-17/obj 2023-07-28T23:54:02.068 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7xHDgfl3H/fNyQuFsEt2hAhafAI= 2023-07-28T23:54:02.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:54:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7xHDgfl3H/fNyQuFsEt2hAhafAI='} 2023-07-28T23:54:02.074 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000087e33f36910b802e-0064c4551a-1117-a1'), ('Date', 'Fri, 28 Jul 2023 23:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:02.074 DEBUG:boto:path=/hiiqos-17/ 2023-07-28T23:54:02.075 DEBUG:boto:auth_path=/hiiqos-17/ 2023-07-28T23:54:02.075 DEBUG:boto:Method: HEAD 2023-07-28T23:54:02.075 DEBUG:boto:Path: /hiiqos-17/ 2023-07-28T23:54:02.076 DEBUG:boto:Data: 2023-07-28T23:54:02.076 DEBUG:boto:Headers: {} 2023-07-28T23:54:02.076 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:54:02.077 DEBUG:boto:Port: 8000 2023-07-28T23:54:02.077 DEBUG:boto:Params: {} 2023-07-28T23:54:02.077 DEBUG:boto:Token: None 2023-07-28T23:54:02.078 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:54:02 GMT /hiiqos-17/ 2023-07-28T23:54:02.078 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FzKjHT1YoB40z0WpFx1RugA3/Vs= 2023-07-28T23:54:02.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FzKjHT1YoB40z0WpFx1RugA3/Vs=', 'Content-Length': '0'} 2023-07-28T23:54:02.080 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a9d9ae3efd19569f-0064c4551a-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:02.081 DEBUG:boto:path=/hiiqos-17/obj 2023-07-28T23:54:02.081 DEBUG:boto:auth_path=/hiiqos-17/obj 2023-07-28T23:54:02.082 DEBUG:boto:Method: DELETE 2023-07-28T23:54:02.082 DEBUG:boto:Path: /hiiqos-17/obj 2023-07-28T23:54:02.082 DEBUG:boto:Data: 2023-07-28T23:54:02.083 DEBUG:boto:Headers: {} 2023-07-28T23:54:02.083 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:54:02.083 DEBUG:boto:Port: 8000 2023-07-28T23:54:02.084 DEBUG:boto:Params: {} 2023-07-28T23:54:02.084 DEBUG:boto:Token: None 2023-07-28T23:54:02.084 DEBUG:boto:StringToSign: DELETE Fri, 28 Jul 2023 23:54:02 GMT /hiiqos-17/obj 2023-07-28T23:54:02.085 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eUPvPr+ga7c5vBa8kb3VE4MPq8I= 2023-07-28T23:54:02.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eUPvPr+ga7c5vBa8kb3VE4MPq8I=', 'Content-Length': '0'} 2023-07-28T23:54:02.093 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx000006e44b0a9864386d8-0064c4551a-1117-a1'), ('Date', 'Fri, 28 Jul 2023 23:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:02.093 DEBUG:boto:path=/hiiqos-18/ 2023-07-28T23:54:02.093 DEBUG:boto:auth_path=/hiiqos-18/ 2023-07-28T23:54:02.094 DEBUG:boto:Method: HEAD 2023-07-28T23:54:02.094 DEBUG:boto:Path: /hiiqos-18/ 2023-07-28T23:54:02.094 DEBUG:boto:Data: 2023-07-28T23:54:02.095 DEBUG:boto:Headers: {} 2023-07-28T23:54:02.095 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:54:02.095 DEBUG:boto:Port: 8001 2023-07-28T23:54:02.096 DEBUG:boto:Params: {} 2023-07-28T23:54:02.096 DEBUG:boto:Token: None 2023-07-28T23:54:02.096 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:54:02 GMT /hiiqos-18/ 2023-07-28T23:54:02.097 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KgS92obKRo3z1XYuFLRptc9nkI4= 2023-07-28T23:54:02.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KgS92obKRo3z1XYuFLRptc9nkI4=', 'Content-Length': '0'} 2023-07-28T23:54:02.099 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f4aaa1e80a49320-0064c4551a-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:02.100 DEBUG:boto:path=/hiiqos-18/obj 2023-07-28T23:54:02.100 DEBUG:boto:auth_path=/hiiqos-18/obj 2023-07-28T23:54:02.101 DEBUG:boto:Method: PUT 2023-07-28T23:54:02.101 DEBUG:boto:Path: /hiiqos-18/obj 2023-07-28T23:54:02.101 DEBUG:boto:Data: 2023-07-28T23:54:02.102 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-07-28T23:54:02.102 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:54:02.102 DEBUG:boto:Port: 8001 2023-07-28T23:54:02.102 DEBUG:boto:Params: {} 2023-07-28T23:54:02.103 DEBUG:boto:Token: None 2023-07-28T23:54:02.103 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 28 Jul 2023 23:54:02 GMT /hiiqos-18/obj 2023-07-28T23:54:02.104 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P55JTwAaKv9TGEWdHmMZrlrhFMs= 2023-07-28T23:54:02.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:54:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P55JTwAaKv9TGEWdHmMZrlrhFMs='} 2023-07-28T23:54:02.110 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008b19bdd26195f092-0064c4551a-1113-a2'), ('Date', 'Fri, 28 Jul 2023 23:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:02.110 DEBUG:boto:path=/hiiqos-18/ 2023-07-28T23:54:02.110 DEBUG:boto:auth_path=/hiiqos-18/ 2023-07-28T23:54:02.111 DEBUG:boto:Method: HEAD 2023-07-28T23:54:02.111 DEBUG:boto:Path: /hiiqos-18/ 2023-07-28T23:54:02.111 DEBUG:boto:Data: 2023-07-28T23:54:02.112 DEBUG:boto:Headers: {} 2023-07-28T23:54:02.112 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:54:02.112 DEBUG:boto:Port: 8001 2023-07-28T23:54:02.113 DEBUG:boto:Params: {} 2023-07-28T23:54:02.113 DEBUG:boto:Token: None 2023-07-28T23:54:02.113 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:54:02 GMT /hiiqos-18/ 2023-07-28T23:54:02.114 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KgS92obKRo3z1XYuFLRptc9nkI4= 2023-07-28T23:54:02.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KgS92obKRo3z1XYuFLRptc9nkI4=', 'Content-Length': '0'} 2023-07-28T23:54:02.116 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000084da69db2b4ffe1e-0064c4551a-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:02.116 DEBUG:boto:path=/hiiqos-18/obj 2023-07-28T23:54:02.117 DEBUG:boto:auth_path=/hiiqos-18/obj 2023-07-28T23:54:02.117 DEBUG:boto:Method: DELETE 2023-07-28T23:54:02.117 DEBUG:boto:Path: /hiiqos-18/obj 2023-07-28T23:54:02.118 DEBUG:boto:Data: 2023-07-28T23:54:02.118 DEBUG:boto:Headers: {} 2023-07-28T23:54:02.118 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:54:02.119 DEBUG:boto:Port: 8001 2023-07-28T23:54:02.119 DEBUG:boto:Params: {} 2023-07-28T23:54:02.119 DEBUG:boto:Token: None 2023-07-28T23:54:02.120 DEBUG:boto:StringToSign: DELETE Fri, 28 Jul 2023 23:54:02 GMT /hiiqos-18/obj 2023-07-28T23:54:02.120 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:01Wp/MY3AphCshhHAWXPLq7Rz2Y= 2023-07-28T23:54:02.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:01Wp/MY3AphCshhHAWXPLq7Rz2Y=', 'Content-Length': '0'} 2023-07-28T23:54:02.129 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx0000089c58c4969e6829b-0064c4551a-1113-a2'), ('Date', 'Fri, 28 Jul 2023 23:54:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:02.129 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-17 2023-07-28T23:54:02.129 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-17', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:54:02.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-17', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:54:02.130 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-17 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-28T23:54:02.183 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:54:02.183 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:54:02.214 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:54:02.212+0000 7fe566b28880 1 waiting to reach incremental sync.. 2023-07-28T23:54:07.215 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:54:07.213+0000 7fe566b28880 1 waiting to reach incremental sync.. 2023-07-28T23:54:12.216 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:54:12.214+0000 7fe566b28880 1 waiting to reach incremental sync.. 2023-07-28T23:54:17.217 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:54:17.215+0000 7fe566b28880 1 bucket sync caught up with source: 2023-07-28T23:54:17.218 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , 00000000003.13.7, , ] 2023-07-28T23:54:17.218 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000003.13.7, , ] 2023-07-28T23:54:17.218 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:54:17.215+0000 7fe566b28880 0 bucket checkpoint complete 2023-07-28T23:54:17.234 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-17 2023-07-28T23:54:17.234 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-17', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:54:17.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-17', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:54:17.235 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-17 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-28T23:54:17.293 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:54:17.294 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:54:17.328 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:54:17.326+0000 7f5f22921880 1 waiting to reach incremental sync.. 2023-07-28T23:54:22.329 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:54:22.327+0000 7f5f22921880 1 waiting to reach incremental sync.. 2023-07-28T23:54:27.329 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:54:27.327+0000 7f5f22921880 1 waiting to reach incremental sync.. 2023-07-28T23:54:32.330 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:54:32.328+0000 7f5f22921880 1 waiting to reach incremental sync.. 2023-07-28T23:54:37.332 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:54:37.330+0000 7f5f22921880 1 bucket sync caught up with source: 2023-07-28T23:54:37.332 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , 00000000001.11.13, , ] 2023-07-28T23:54:37.332 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000001.11.13, , ] 2023-07-28T23:54:37.332 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:54:37.330+0000 7f5f22921880 0 bucket checkpoint complete 2023-07-28T23:54:37.347 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-17 zones={a2, a1} 2023-07-28T23:54:37.348 DEBUG:boto:path=/hiiqos-17/ 2023-07-28T23:54:37.348 DEBUG:boto:auth_path=/hiiqos-17/ 2023-07-28T23:54:37.349 DEBUG:boto:Method: HEAD 2023-07-28T23:54:37.349 DEBUG:boto:Path: /hiiqos-17/ 2023-07-28T23:54:37.349 DEBUG:boto:Data: 2023-07-28T23:54:37.350 DEBUG:boto:Headers: {} 2023-07-28T23:54:37.350 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:54:37.350 DEBUG:boto:Port: 8001 2023-07-28T23:54:37.351 DEBUG:boto:Params: {} 2023-07-28T23:54:37.351 DEBUG:boto:Token: None 2023-07-28T23:54:37.351 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:54:37 GMT /hiiqos-17/ 2023-07-28T23:54:37.352 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:b+oVyOBLxfdxF96oqvx+slMdLAs= 2023-07-28T23:54:37.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:b+oVyOBLxfdxF96oqvx+slMdLAs=', 'Content-Length': '0'} 2023-07-28T23:54:37.355 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000745406a61fb29d8c-0064c4553d-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:54:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:37.355 DEBUG:boto:path=/hiiqos-17/ 2023-07-28T23:54:37.356 DEBUG:boto:auth_path=/hiiqos-17/ 2023-07-28T23:54:37.356 DEBUG:boto:Method: HEAD 2023-07-28T23:54:37.356 DEBUG:boto:Path: /hiiqos-17/ 2023-07-28T23:54:37.357 DEBUG:boto:Data: 2023-07-28T23:54:37.357 DEBUG:boto:Headers: {} 2023-07-28T23:54:37.358 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:54:37.358 DEBUG:boto:Port: 8000 2023-07-28T23:54:37.358 DEBUG:boto:Params: {} 2023-07-28T23:54:37.359 DEBUG:boto:Token: None 2023-07-28T23:54:37.359 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:54:37 GMT /hiiqos-17/ 2023-07-28T23:54:37.359 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:b+oVyOBLxfdxF96oqvx+slMdLAs= 2023-07-28T23:54:37.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:b+oVyOBLxfdxF96oqvx+slMdLAs=', 'Content-Length': '0'} 2023-07-28T23:54:37.362 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e28dfb6f24b8359b-0064c4553d-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:54:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:37.362 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-28T23:54:37.363 DEBUG:boto:path=/hiiqos-17/ 2023-07-28T23:54:37.363 DEBUG:boto:auth_path=/hiiqos-17/ 2023-07-28T23:54:37.364 DEBUG:boto:path=/hiiqos-17/?versions&max-keys=999 2023-07-28T23:54:37.364 DEBUG:boto:auth_path=/hiiqos-17/?versions&max-keys=999 2023-07-28T23:54:37.364 DEBUG:boto:Method: GET 2023-07-28T23:54:37.365 DEBUG:boto:Path: /hiiqos-17/?versions&max-keys=999 2023-07-28T23:54:37.365 DEBUG:boto:Data: 2023-07-28T23:54:37.365 DEBUG:boto:Headers: {} 2023-07-28T23:54:37.366 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:54:37.366 DEBUG:boto:Port: 8001 2023-07-28T23:54:37.366 DEBUG:boto:Params: {} 2023-07-28T23:54:37.367 DEBUG:boto:Token: None 2023-07-28T23:54:37.367 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:54:37 GMT /hiiqos-17/?versions 2023-07-28T23:54:37.367 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Fuh2kMj9lNEg1WDg45+AAQMqqrk= 2023-07-28T23:54:37.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Fuh2kMj9lNEg1WDg45+AAQMqqrk=', 'Content-Length': '0'} 2023-07-28T23:54:37.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d69006933902cdb6-0064c4553d-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:54:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:37.371 DEBUG:boto:b'hiiqos-17999falseobjnulltrue2023-07-28T23:54:02.086Zrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:54:37.372 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:54:37.372 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-28T23:54:37.372 DEBUG:boto:path=/hiiqos-17/ 2023-07-28T23:54:37.373 DEBUG:boto:auth_path=/hiiqos-17/ 2023-07-28T23:54:37.373 DEBUG:boto:path=/hiiqos-17/?versions&max-keys=999 2023-07-28T23:54:37.373 DEBUG:boto:auth_path=/hiiqos-17/?versions&max-keys=999 2023-07-28T23:54:37.374 DEBUG:boto:Method: GET 2023-07-28T23:54:37.374 DEBUG:boto:Path: /hiiqos-17/?versions&max-keys=999 2023-07-28T23:54:37.374 DEBUG:boto:Data: 2023-07-28T23:54:37.375 DEBUG:boto:Headers: {} 2023-07-28T23:54:37.375 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:54:37.375 DEBUG:boto:Port: 8000 2023-07-28T23:54:37.376 DEBUG:boto:Params: {} 2023-07-28T23:54:37.376 DEBUG:boto:Token: None 2023-07-28T23:54:37.376 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:54:37 GMT /hiiqos-17/?versions 2023-07-28T23:54:37.377 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Fuh2kMj9lNEg1WDg45+AAQMqqrk= 2023-07-28T23:54:37.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Fuh2kMj9lNEg1WDg45+AAQMqqrk=', 'Content-Length': '0'} 2023-07-28T23:54:37.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc5d3f1aca422f45-0064c4553d-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:54:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:37.380 DEBUG:boto:b'hiiqos-17999falseobjnulltrue2023-07-28T23:54:02.086Zrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:54:37.381 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:54:37.381 DEBUG:boto:path=/hiiqos-17/ 2023-07-28T23:54:37.382 DEBUG:boto:auth_path=/hiiqos-17/ 2023-07-28T23:54:37.382 DEBUG:boto:path=/hiiqos-17/?versions&max-keys=999 2023-07-28T23:54:37.382 DEBUG:boto:auth_path=/hiiqos-17/?versions&max-keys=999 2023-07-28T23:54:37.383 DEBUG:boto:Method: GET 2023-07-28T23:54:37.383 DEBUG:boto:Path: /hiiqos-17/?versions&max-keys=999 2023-07-28T23:54:37.383 DEBUG:boto:Data: 2023-07-28T23:54:37.384 DEBUG:boto:Headers: {} 2023-07-28T23:54:37.384 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:54:37.384 DEBUG:boto:Port: 8001 2023-07-28T23:54:37.385 DEBUG:boto:Params: {} 2023-07-28T23:54:37.385 DEBUG:boto:Token: None 2023-07-28T23:54:37.386 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:54:37 GMT /hiiqos-17/?versions 2023-07-28T23:54:37.386 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Fuh2kMj9lNEg1WDg45+AAQMqqrk= 2023-07-28T23:54:37.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Fuh2kMj9lNEg1WDg45+AAQMqqrk=', 'Content-Length': '0'} 2023-07-28T23:54:37.389 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036c171d7d12d2be6-0064c4553d-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:54:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:37.389 DEBUG:boto:b'hiiqos-17999falseobjnulltrue2023-07-28T23:54:02.086Zrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:54:37.390 DEBUG:boto:path=/hiiqos-17/ 2023-07-28T23:54:37.390 DEBUG:boto:auth_path=/hiiqos-17/ 2023-07-28T23:54:37.391 DEBUG:boto:path=/hiiqos-17/?versions&max-keys=999 2023-07-28T23:54:37.391 DEBUG:boto:auth_path=/hiiqos-17/?versions&max-keys=999 2023-07-28T23:54:37.391 DEBUG:boto:Method: GET 2023-07-28T23:54:37.392 DEBUG:boto:Path: /hiiqos-17/?versions&max-keys=999 2023-07-28T23:54:37.392 DEBUG:boto:Data: 2023-07-28T23:54:37.392 DEBUG:boto:Headers: {} 2023-07-28T23:54:37.393 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:54:37.393 DEBUG:boto:Port: 8000 2023-07-28T23:54:37.393 DEBUG:boto:Params: {} 2023-07-28T23:54:37.394 DEBUG:boto:Token: None 2023-07-28T23:54:37.394 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:54:37 GMT /hiiqos-17/?versions 2023-07-28T23:54:37.395 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Fuh2kMj9lNEg1WDg45+AAQMqqrk= 2023-07-28T23:54:37.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Fuh2kMj9lNEg1WDg45+AAQMqqrk=', 'Content-Length': '0'} 2023-07-28T23:54:37.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000407d660ecf7a346f-0064c4553d-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:54:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:37.398 DEBUG:boto:b'hiiqos-17999falseobjnulltrue2023-07-28T23:54:02.086Zrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:54:37.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:54:37.399 DEBUG:boto:path=/hiiqos-17/obj 2023-07-28T23:54:37.399 DEBUG:boto:auth_path=/hiiqos-17/obj 2023-07-28T23:54:37.400 DEBUG:boto:Method: HEAD 2023-07-28T23:54:37.400 DEBUG:boto:Path: /hiiqos-17/obj 2023-07-28T23:54:37.400 DEBUG:boto:Data: 2023-07-28T23:54:37.401 DEBUG:boto:Headers: {} 2023-07-28T23:54:37.401 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:54:37.401 DEBUG:boto:Port: 8001 2023-07-28T23:54:37.402 DEBUG:boto:Params: {} 2023-07-28T23:54:37.402 DEBUG:boto:Token: None 2023-07-28T23:54:37.402 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:54:37 GMT /hiiqos-17/obj 2023-07-28T23:54:37.403 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CgBRYdqu8+mvLYW2bXesXdmq5WU= 2023-07-28T23:54:37.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CgBRYdqu8+mvLYW2bXesXdmq5WU=', 'Content-Length': '0'} 2023-07-28T23:54:37.405 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000e5db36d69e715b69-0064c4553d-1113-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:54:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:37.405 DEBUG:boto:path=/hiiqos-17/obj 2023-07-28T23:54:37.406 DEBUG:boto:auth_path=/hiiqos-17/obj 2023-07-28T23:54:37.406 DEBUG:boto:Method: HEAD 2023-07-28T23:54:37.406 DEBUG:boto:Path: /hiiqos-17/obj 2023-07-28T23:54:37.407 DEBUG:boto:Data: 2023-07-28T23:54:37.407 DEBUG:boto:Headers: {} 2023-07-28T23:54:37.407 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:54:37.408 DEBUG:boto:Port: 8000 2023-07-28T23:54:37.408 DEBUG:boto:Params: {} 2023-07-28T23:54:37.409 DEBUG:boto:Token: None 2023-07-28T23:54:37.409 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:54:37 GMT /hiiqos-17/obj 2023-07-28T23:54:37.409 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CgBRYdqu8+mvLYW2bXesXdmq5WU= 2023-07-28T23:54:37.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CgBRYdqu8+mvLYW2bXesXdmq5WU=', 'Content-Length': '0'} 2023-07-28T23:54:37.411 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000009ca5e0af29ef2d74-0064c4553d-1117-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:54:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:37.412 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-17 zones={a2, a1} 2023-07-28T23:54:37.412 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-18 2023-07-28T23:54:37.412 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-18', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:54:37.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-18', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:54:37.413 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-18 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-28T23:54:37.472 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:54:37.472 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:54:37.507 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:54:37.505+0000 7fa634f22880 1 bucket sync caught up with source: 2023-07-28T23:54:37.508 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , 00000000001.13.13, , ] 2023-07-28T23:54:37.508 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000001.13.13, , ] 2023-07-28T23:54:37.508 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:54:37.505+0000 7fa634f22880 0 bucket checkpoint complete 2023-07-28T23:54:37.520 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-18 2023-07-28T23:54:37.521 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-18', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:54:37.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-18', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-28T23:54:37.522 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-18 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-28T23:54:37.574 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:54:37.574 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:54:37.606 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:54:37.604+0000 7f67a8d31880 1 bucket sync caught up with source: 2023-07-28T23:54:37.607 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , 00000000003.53.7, , ] 2023-07-28T23:54:37.607 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000003.53.7, , ] 2023-07-28T23:54:37.607 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:54:37.604+0000 7f67a8d31880 0 bucket checkpoint complete 2023-07-28T23:54:37.620 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-18 zones={a2, a1} 2023-07-28T23:54:37.621 DEBUG:boto:path=/hiiqos-18/ 2023-07-28T23:54:37.621 DEBUG:boto:auth_path=/hiiqos-18/ 2023-07-28T23:54:37.621 DEBUG:boto:Method: HEAD 2023-07-28T23:54:37.622 DEBUG:boto:Path: /hiiqos-18/ 2023-07-28T23:54:37.622 DEBUG:boto:Data: 2023-07-28T23:54:37.622 DEBUG:boto:Headers: {} 2023-07-28T23:54:37.623 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:54:37.623 DEBUG:boto:Port: 8001 2023-07-28T23:54:37.623 DEBUG:boto:Params: {} 2023-07-28T23:54:37.624 DEBUG:boto:Token: None 2023-07-28T23:54:37.624 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:54:37 GMT /hiiqos-18/ 2023-07-28T23:54:37.625 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:iWtD3N7wpQRBy5U5dEqeUEbSC4I= 2023-07-28T23:54:37.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:iWtD3N7wpQRBy5U5dEqeUEbSC4I=', 'Content-Length': '0'} 2023-07-28T23:54:37.627 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009d42ff3d3dfcad21-0064c4553d-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:54:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:37.628 DEBUG:boto:path=/hiiqos-18/ 2023-07-28T23:54:37.628 DEBUG:boto:auth_path=/hiiqos-18/ 2023-07-28T23:54:37.629 DEBUG:boto:Method: HEAD 2023-07-28T23:54:37.629 DEBUG:boto:Path: /hiiqos-18/ 2023-07-28T23:54:37.629 DEBUG:boto:Data: 2023-07-28T23:54:37.630 DEBUG:boto:Headers: {} 2023-07-28T23:54:37.630 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:54:37.630 DEBUG:boto:Port: 8000 2023-07-28T23:54:37.631 DEBUG:boto:Params: {} 2023-07-28T23:54:37.631 DEBUG:boto:Token: None 2023-07-28T23:54:37.631 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:54:37 GMT /hiiqos-18/ 2023-07-28T23:54:37.632 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:iWtD3N7wpQRBy5U5dEqeUEbSC4I= 2023-07-28T23:54:37.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:iWtD3N7wpQRBy5U5dEqeUEbSC4I=', 'Content-Length': '0'} 2023-07-28T23:54:37.634 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000063f136a8d2b3b4f6-0064c4553d-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:54:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:37.635 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-28T23:54:37.635 DEBUG:boto:path=/hiiqos-18/ 2023-07-28T23:54:37.635 DEBUG:boto:auth_path=/hiiqos-18/ 2023-07-28T23:54:37.636 DEBUG:boto:path=/hiiqos-18/?versions&max-keys=999 2023-07-28T23:54:37.636 DEBUG:boto:auth_path=/hiiqos-18/?versions&max-keys=999 2023-07-28T23:54:37.636 DEBUG:boto:Method: GET 2023-07-28T23:54:37.637 DEBUG:boto:Path: /hiiqos-18/?versions&max-keys=999 2023-07-28T23:54:37.637 DEBUG:boto:Data: 2023-07-28T23:54:37.637 DEBUG:boto:Headers: {} 2023-07-28T23:54:37.638 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:54:37.638 DEBUG:boto:Port: 8001 2023-07-28T23:54:37.638 DEBUG:boto:Params: {} 2023-07-28T23:54:37.639 DEBUG:boto:Token: None 2023-07-28T23:54:37.639 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:54:37 GMT /hiiqos-18/?versions 2023-07-28T23:54:37.640 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:C88v56C9l3PkU7yZha58p6qhvl4= 2023-07-28T23:54:37.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:C88v56C9l3PkU7yZha58p6qhvl4=', 'Content-Length': '0'} 2023-07-28T23:54:37.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043bb180651614fcd-0064c4553d-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:54:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:37.642 DEBUG:boto:b'hiiqos-18999falseobjnulltrue2023-07-28T23:54:02.122Zrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:54:37.643 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:54:37.643 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-28T23:54:37.644 DEBUG:boto:path=/hiiqos-18/ 2023-07-28T23:54:37.644 DEBUG:boto:auth_path=/hiiqos-18/ 2023-07-28T23:54:37.645 DEBUG:boto:path=/hiiqos-18/?versions&max-keys=999 2023-07-28T23:54:37.645 DEBUG:boto:auth_path=/hiiqos-18/?versions&max-keys=999 2023-07-28T23:54:37.645 DEBUG:boto:Method: GET 2023-07-28T23:54:37.646 DEBUG:boto:Path: /hiiqos-18/?versions&max-keys=999 2023-07-28T23:54:37.646 DEBUG:boto:Data: 2023-07-28T23:54:37.646 DEBUG:boto:Headers: {} 2023-07-28T23:54:37.647 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:54:37.647 DEBUG:boto:Port: 8000 2023-07-28T23:54:37.647 DEBUG:boto:Params: {} 2023-07-28T23:54:37.648 DEBUG:boto:Token: None 2023-07-28T23:54:37.648 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:54:37 GMT /hiiqos-18/?versions 2023-07-28T23:54:37.648 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:C88v56C9l3PkU7yZha58p6qhvl4= 2023-07-28T23:54:37.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:C88v56C9l3PkU7yZha58p6qhvl4=', 'Content-Length': '0'} 2023-07-28T23:54:37.651 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063557e6276649e1b-0064c4553d-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:54:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:37.651 DEBUG:boto:b'hiiqos-18999falseobjnulltrue2023-07-28T23:54:02.122Zrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:54:37.652 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-07-28T23:54:37.652 DEBUG:boto:path=/hiiqos-18/ 2023-07-28T23:54:37.653 DEBUG:boto:auth_path=/hiiqos-18/ 2023-07-28T23:54:37.653 DEBUG:boto:path=/hiiqos-18/?versions&max-keys=999 2023-07-28T23:54:37.653 DEBUG:boto:auth_path=/hiiqos-18/?versions&max-keys=999 2023-07-28T23:54:37.654 DEBUG:boto:Method: GET 2023-07-28T23:54:37.654 DEBUG:boto:Path: /hiiqos-18/?versions&max-keys=999 2023-07-28T23:54:37.654 DEBUG:boto:Data: 2023-07-28T23:54:37.655 DEBUG:boto:Headers: {} 2023-07-28T23:54:37.655 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:54:37.655 DEBUG:boto:Port: 8001 2023-07-28T23:54:37.656 DEBUG:boto:Params: {} 2023-07-28T23:54:37.656 DEBUG:boto:Token: None 2023-07-28T23:54:37.656 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:54:37 GMT /hiiqos-18/?versions 2023-07-28T23:54:37.657 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:C88v56C9l3PkU7yZha58p6qhvl4= 2023-07-28T23:54:37.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:C88v56C9l3PkU7yZha58p6qhvl4=', 'Content-Length': '0'} 2023-07-28T23:54:37.659 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f171ce61078514d5-0064c4553d-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:54:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:37.660 DEBUG:boto:b'hiiqos-18999falseobjnulltrue2023-07-28T23:54:02.122Zrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:54:37.660 DEBUG:boto:path=/hiiqos-18/ 2023-07-28T23:54:37.661 DEBUG:boto:auth_path=/hiiqos-18/ 2023-07-28T23:54:37.661 DEBUG:boto:path=/hiiqos-18/?versions&max-keys=999 2023-07-28T23:54:37.661 DEBUG:boto:auth_path=/hiiqos-18/?versions&max-keys=999 2023-07-28T23:54:37.662 DEBUG:boto:Method: GET 2023-07-28T23:54:37.662 DEBUG:boto:Path: /hiiqos-18/?versions&max-keys=999 2023-07-28T23:54:37.662 DEBUG:boto:Data: 2023-07-28T23:54:37.663 DEBUG:boto:Headers: {} 2023-07-28T23:54:37.663 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:54:37.663 DEBUG:boto:Port: 8000 2023-07-28T23:54:37.664 DEBUG:boto:Params: {} 2023-07-28T23:54:37.664 DEBUG:boto:Token: None 2023-07-28T23:54:37.665 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:54:37 GMT /hiiqos-18/?versions 2023-07-28T23:54:37.665 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:C88v56C9l3PkU7yZha58p6qhvl4= 2023-07-28T23:54:37.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:C88v56C9l3PkU7yZha58p6qhvl4=', 'Content-Length': '0'} 2023-07-28T23:54:37.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c103cd38bd28021-0064c4553d-1117-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:54:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:37.668 DEBUG:boto:b'hiiqos-18999falseobjnulltrue2023-07-28T23:54:02.122Zrgw-multisite-test-userMultisite Test UserNormal' 2023-07-28T23:54:37.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-07-28T23:54:37.669 DEBUG:boto:path=/hiiqos-18/obj 2023-07-28T23:54:37.670 DEBUG:boto:auth_path=/hiiqos-18/obj 2023-07-28T23:54:37.670 DEBUG:boto:Method: HEAD 2023-07-28T23:54:37.670 DEBUG:boto:Path: /hiiqos-18/obj 2023-07-28T23:54:37.671 DEBUG:boto:Data: 2023-07-28T23:54:37.671 DEBUG:boto:Headers: {} 2023-07-28T23:54:37.671 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:54:37.671 DEBUG:boto:Port: 8001 2023-07-28T23:54:37.672 DEBUG:boto:Params: {} 2023-07-28T23:54:37.672 DEBUG:boto:Token: None 2023-07-28T23:54:37.673 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:54:37 GMT /hiiqos-18/obj 2023-07-28T23:54:37.673 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6QqtmT3HNICRGvuNy3OWY/qlVEw= 2023-07-28T23:54:37.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6QqtmT3HNICRGvuNy3OWY/qlVEw=', 'Content-Length': '0'} 2023-07-28T23:54:37.675 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000000ec2162d95740270-0064c4553d-1113-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:54:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:37.675 DEBUG:boto:path=/hiiqos-18/obj 2023-07-28T23:54:37.675 DEBUG:boto:auth_path=/hiiqos-18/obj 2023-07-28T23:54:37.676 DEBUG:boto:Method: HEAD 2023-07-28T23:54:37.676 DEBUG:boto:Path: /hiiqos-18/obj 2023-07-28T23:54:37.677 DEBUG:boto:Data: 2023-07-28T23:54:37.677 DEBUG:boto:Headers: {} 2023-07-28T23:54:37.677 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:54:37.678 DEBUG:boto:Port: 8000 2023-07-28T23:54:37.678 DEBUG:boto:Params: {} 2023-07-28T23:54:37.678 DEBUG:boto:Token: None 2023-07-28T23:54:37.679 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:54:37 GMT /hiiqos-18/obj 2023-07-28T23:54:37.679 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6QqtmT3HNICRGvuNy3OWY/qlVEw= 2023-07-28T23:54:37.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6QqtmT3HNICRGvuNy3OWY/qlVEw=', 'Content-Length': '0'} 2023-07-28T23:54:37.681 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000005b41696a59db5c7c-0064c4553d-1117-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:54:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:37.681 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-18 zones={a2, a1} 2023-07-28T23:54:37.682 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2023-07-28T23:54:37.682 INFO:rgw_multi.tests:create bucket zone=a1 name=hiiqos-19 2023-07-28T23:54:37.683 DEBUG:boto:path=/hiiqos-19/ 2023-07-28T23:54:37.683 DEBUG:boto:auth_path=/hiiqos-19/ 2023-07-28T23:54:37.683 DEBUG:boto:Method: PUT 2023-07-28T23:54:37.684 DEBUG:boto:Path: /hiiqos-19/ 2023-07-28T23:54:37.684 DEBUG:boto:Data: 2023-07-28T23:54:37.685 DEBUG:boto:Headers: {} 2023-07-28T23:54:37.685 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:54:37.685 DEBUG:boto:Port: 8000 2023-07-28T23:54:37.685 DEBUG:boto:Params: {} 2023-07-28T23:54:37.686 DEBUG:boto:Token: None 2023-07-28T23:54:37.686 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:54:37 GMT /hiiqos-19/ 2023-07-28T23:54:37.687 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qggEyRPZa3POL9RSs5nfRAjSdt8= 2023-07-28T23:54:37.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qggEyRPZa3POL9RSs5nfRAjSdt8=', 'Content-Length': '0'} 2023-07-28T23:54:37.703 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004153689610a246f6-0064c4553d-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:54:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:37.704 INFO:rgw_multi.tests:create bucket zone=a2 name=hiiqos-20 2023-07-28T23:54:37.704 DEBUG:boto:path=/hiiqos-20/ 2023-07-28T23:54:37.704 DEBUG:boto:auth_path=/hiiqos-20/ 2023-07-28T23:54:37.705 DEBUG:boto:Method: PUT 2023-07-28T23:54:37.705 DEBUG:boto:Path: /hiiqos-20/ 2023-07-28T23:54:37.705 DEBUG:boto:Data: 2023-07-28T23:54:37.706 DEBUG:boto:Headers: {} 2023-07-28T23:54:37.706 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:54:37.706 DEBUG:boto:Port: 8001 2023-07-28T23:54:37.707 DEBUG:boto:Params: {} 2023-07-28T23:54:37.707 DEBUG:boto:Token: None 2023-07-28T23:54:37.707 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:54:37 GMT /hiiqos-20/ 2023-07-28T23:54:37.708 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kCrbKsLuqvlDmPqb9I4VmfLKAOU= 2023-07-28T23:54:37.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kCrbKsLuqvlDmPqb9I4VmfLKAOU=', 'Content-Length': '0'} 2023-07-28T23:54:37.734 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000009be7358c613ecb9-0064c4553d-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:54:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:37.735 DEBUG:boto:Using access key provided by client. 2023-07-28T23:54:37.736 DEBUG:boto:Using secret key provided by client. 2023-07-28T23:54:37.736 DEBUG:boto:Using access key provided by client. 2023-07-28T23:54:37.737 DEBUG:boto:Using secret key provided by client. 2023-07-28T23:54:37.738 DEBUG:boto:Using access key provided by client. 2023-07-28T23:54:37.738 DEBUG:boto:Using secret key provided by client. 2023-07-28T23:54:37.739 DEBUG:boto:Using access key provided by client. 2023-07-28T23:54:37.739 DEBUG:boto:Using secret key provided by client. 2023-07-28T23:54:37.739 INFO:rgw_multi.tests:create bucket zone=b1 name=hiiqos-21 2023-07-28T23:54:37.740 DEBUG:boto:path=/hiiqos-21/ 2023-07-28T23:54:37.740 DEBUG:boto:auth_path=/hiiqos-21/ 2023-07-28T23:54:37.741 DEBUG:boto:Method: PUT 2023-07-28T23:54:37.741 DEBUG:boto:Path: /hiiqos-21/ 2023-07-28T23:54:37.741 DEBUG:boto:Data: 2023-07-28T23:54:37.742 DEBUG:boto:Headers: {} 2023-07-28T23:54:37.742 DEBUG:boto:Host: smithi139.front.sepia.ceph.com:8000 2023-07-28T23:54:37.742 DEBUG:boto:Port: 8000 2023-07-28T23:54:37.742 DEBUG:boto:Params: {} 2023-07-28T23:54:37.743 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-28T23:54:37.743 DEBUG:boto:Token: None 2023-07-28T23:54:37.744 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:54:37 GMT /hiiqos-21/ 2023-07-28T23:54:37.744 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cj96mmRPDQf454Y4y8nRlsiJ4Xo= 2023-07-28T23:54:37.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cj96mmRPDQf454Y4y8nRlsiJ4Xo=', 'Content-Length': '0'} 2023-07-28T23:54:37.775 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000095e77baec936b2e4-0064c4553d-1109-b1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:54:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:37.776 INFO:rgw_multi.tests:create bucket zone=b2 name=hiiqos-22 2023-07-28T23:54:37.776 DEBUG:boto:path=/hiiqos-22/ 2023-07-28T23:54:37.777 DEBUG:boto:auth_path=/hiiqos-22/ 2023-07-28T23:54:37.777 DEBUG:boto:Method: PUT 2023-07-28T23:54:37.777 DEBUG:boto:Path: /hiiqos-22/ 2023-07-28T23:54:37.778 DEBUG:boto:Data: 2023-07-28T23:54:37.778 DEBUG:boto:Headers: {} 2023-07-28T23:54:37.778 DEBUG:boto:Host: smithi139.front.sepia.ceph.com:8001 2023-07-28T23:54:37.779 DEBUG:boto:Port: 8001 2023-07-28T23:54:37.779 DEBUG:boto:Params: {} 2023-07-28T23:54:37.779 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-28T23:54:37.780 DEBUG:boto:Token: None 2023-07-28T23:54:37.780 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:54:37 GMT /hiiqos-22/ 2023-07-28T23:54:37.780 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QhnwVZFH5aVtL6OeI+PMuNxEuUk= 2023-07-28T23:54:37.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QhnwVZFH5aVtL6OeI+PMuNxEuUk=', 'Content-Length': '0'} 2023-07-28T23:54:37.809 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008453411ada63b46e-0064c4553d-110f-b2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:54:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:37.809 DEBUG:boto:path=/hiiqos-19/ 2023-07-28T23:54:37.810 DEBUG:boto:auth_path=/hiiqos-19/ 2023-07-28T23:54:37.810 DEBUG:boto:path=/hiiqos-19/?versioning 2023-07-28T23:54:37.810 DEBUG:boto:auth_path=/hiiqos-19/?versioning 2023-07-28T23:54:37.811 DEBUG:boto:Method: PUT 2023-07-28T23:54:37.811 DEBUG:boto:Path: /hiiqos-19/?versioning 2023-07-28T23:54:37.811 DEBUG:boto:Data: Enabled Disabled 2023-07-28T23:54:37.812 DEBUG:boto:Headers: {} 2023-07-28T23:54:37.812 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:54:37.812 DEBUG:boto:Port: 8000 2023-07-28T23:54:37.813 DEBUG:boto:Params: {} 2023-07-28T23:54:37.813 DEBUG:boto:Token: None 2023-07-28T23:54:37.813 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:54:37 GMT /hiiqos-19/?versioning 2023-07-28T23:54:37.814 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:V0K9ath6EpiBv9p9237nUxheMmU= 2023-07-28T23:54:37.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:V0K9ath6EpiBv9p9237nUxheMmU=', 'Content-Length': '228'} 2023-07-28T23:54:37.819 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a474b49a4229b864-0064c4553d-1117-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:54:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:37.819 DEBUG:boto:path=/hiiqos-19/ 2023-07-28T23:54:37.820 DEBUG:boto:auth_path=/hiiqos-19/ 2023-07-28T23:54:37.820 DEBUG:boto:path=/hiiqos-19/?versioning 2023-07-28T23:54:37.820 DEBUG:boto:auth_path=/hiiqos-19/?versioning 2023-07-28T23:54:37.821 DEBUG:boto:Method: GET 2023-07-28T23:54:37.821 DEBUG:boto:Path: /hiiqos-19/?versioning 2023-07-28T23:54:37.821 DEBUG:boto:Data: 2023-07-28T23:54:37.822 DEBUG:boto:Headers: {} 2023-07-28T23:54:37.822 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:54:37.822 DEBUG:boto:Port: 8000 2023-07-28T23:54:37.822 DEBUG:boto:Params: {} 2023-07-28T23:54:37.823 DEBUG:boto:Token: None 2023-07-28T23:54:37.823 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:54:37 GMT /hiiqos-19/?versioning 2023-07-28T23:54:37.824 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eJZF65gGn6si1IjHBXksFnWug4c= 2023-07-28T23:54:37.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eJZF65gGn6si1IjHBXksFnWug4c=', 'Content-Length': '0'} 2023-07-28T23:54:37.825 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b3d0cf585478feb7-0064c4553d-1117-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Fri, 28 Jul 2023 23:54:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:37.826 DEBUG:boto:EnabledDisabled 2023-07-28T23:54:37.827 DEBUG:boto:path=/hiiqos-20/ 2023-07-28T23:54:37.827 DEBUG:boto:auth_path=/hiiqos-20/ 2023-07-28T23:54:37.827 DEBUG:boto:path=/hiiqos-20/?versioning 2023-07-28T23:54:37.828 DEBUG:boto:auth_path=/hiiqos-20/?versioning 2023-07-28T23:54:37.828 DEBUG:boto:Method: PUT 2023-07-28T23:54:37.828 DEBUG:boto:Path: /hiiqos-20/?versioning 2023-07-28T23:54:37.829 DEBUG:boto:Data: Enabled Disabled 2023-07-28T23:54:37.829 DEBUG:boto:Headers: {} 2023-07-28T23:54:37.829 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:54:37.830 DEBUG:boto:Port: 8001 2023-07-28T23:54:37.830 DEBUG:boto:Params: {} 2023-07-28T23:54:37.830 DEBUG:boto:Token: None 2023-07-28T23:54:37.831 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:54:37 GMT /hiiqos-20/?versioning 2023-07-28T23:54:37.831 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nBjwyiVPpKD7MQZtnCO3IKgIoaw= 2023-07-28T23:54:37.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nBjwyiVPpKD7MQZtnCO3IKgIoaw=', 'Content-Length': '228'} 2023-07-28T23:54:37.838 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000445255b4e1a97059-0064c4553d-1113-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:54:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:37.838 DEBUG:boto:path=/hiiqos-20/ 2023-07-28T23:54:37.839 DEBUG:boto:auth_path=/hiiqos-20/ 2023-07-28T23:54:37.839 DEBUG:boto:path=/hiiqos-20/?versioning 2023-07-28T23:54:37.839 DEBUG:boto:auth_path=/hiiqos-20/?versioning 2023-07-28T23:54:37.840 DEBUG:boto:Method: GET 2023-07-28T23:54:37.840 DEBUG:boto:Path: /hiiqos-20/?versioning 2023-07-28T23:54:37.840 DEBUG:boto:Data: 2023-07-28T23:54:37.841 DEBUG:boto:Headers: {} 2023-07-28T23:54:37.841 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:54:37.841 DEBUG:boto:Port: 8001 2023-07-28T23:54:37.841 DEBUG:boto:Params: {} 2023-07-28T23:54:37.842 DEBUG:boto:Token: None 2023-07-28T23:54:37.842 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:54:37 GMT /hiiqos-20/?versioning 2023-07-28T23:54:37.843 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PUEy/PATXiUMEMeH7L4IR1yCl4I= 2023-07-28T23:54:37.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PUEy/PATXiUMEMeH7L4IR1yCl4I=', 'Content-Length': '0'} 2023-07-28T23:54:37.844 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f5af7a65d05f61a1-0064c4553d-1113-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Fri, 28 Jul 2023 23:54:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:37.884 DEBUG:boto:EnabledDisabled 2023-07-28T23:54:37.885 DEBUG:boto:path=/hiiqos-21/ 2023-07-28T23:54:37.885 DEBUG:boto:auth_path=/hiiqos-21/ 2023-07-28T23:54:37.885 DEBUG:boto:path=/hiiqos-21/?versioning 2023-07-28T23:54:37.886 DEBUG:boto:auth_path=/hiiqos-21/?versioning 2023-07-28T23:54:37.886 DEBUG:boto:Method: PUT 2023-07-28T23:54:37.886 DEBUG:boto:Path: /hiiqos-21/?versioning 2023-07-28T23:54:37.886 DEBUG:boto:Data: Enabled Disabled 2023-07-28T23:54:37.887 DEBUG:boto:Headers: {} 2023-07-28T23:54:37.887 DEBUG:boto:Host: smithi139.front.sepia.ceph.com:8000 2023-07-28T23:54:37.887 DEBUG:boto:Port: 8000 2023-07-28T23:54:37.888 DEBUG:boto:Params: {} 2023-07-28T23:54:37.888 DEBUG:boto:Token: None 2023-07-28T23:54:37.888 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:54:37 GMT /hiiqos-21/?versioning 2023-07-28T23:54:37.889 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S3vxbHGA2RoIlD9dDrEbRGGfbFU= 2023-07-28T23:54:37.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S3vxbHGA2RoIlD9dDrEbRGGfbFU=', 'Content-Length': '228'} 2023-07-28T23:54:37.898 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ce2fb25d8212cb82-0064c4553d-1109-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:54:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:37.898 DEBUG:boto:path=/hiiqos-21/ 2023-07-28T23:54:37.898 DEBUG:boto:auth_path=/hiiqos-21/ 2023-07-28T23:54:37.899 DEBUG:boto:path=/hiiqos-21/?versioning 2023-07-28T23:54:37.899 DEBUG:boto:auth_path=/hiiqos-21/?versioning 2023-07-28T23:54:37.899 DEBUG:boto:Method: GET 2023-07-28T23:54:37.900 DEBUG:boto:Path: /hiiqos-21/?versioning 2023-07-28T23:54:37.900 DEBUG:boto:Data: 2023-07-28T23:54:37.900 DEBUG:boto:Headers: {} 2023-07-28T23:54:37.901 DEBUG:boto:Host: smithi139.front.sepia.ceph.com:8000 2023-07-28T23:54:37.901 DEBUG:boto:Port: 8000 2023-07-28T23:54:37.901 DEBUG:boto:Params: {} 2023-07-28T23:54:37.902 DEBUG:boto:Token: None 2023-07-28T23:54:37.902 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:54:37 GMT /hiiqos-21/?versioning 2023-07-28T23:54:37.902 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:o/ssUI9B2Mnuu8ABVChTS+69EpI= 2023-07-28T23:54:37.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:o/ssUI9B2Mnuu8ABVChTS+69EpI=', 'Content-Length': '0'} 2023-07-28T23:54:37.904 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a203ddebcfb05bf2-0064c4553d-1109-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Fri, 28 Jul 2023 23:54:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:37.904 DEBUG:boto:EnabledDisabled 2023-07-28T23:54:37.905 DEBUG:boto:path=/hiiqos-22/ 2023-07-28T23:54:37.905 DEBUG:boto:auth_path=/hiiqos-22/ 2023-07-28T23:54:37.905 DEBUG:boto:path=/hiiqos-22/?versioning 2023-07-28T23:54:37.906 DEBUG:boto:auth_path=/hiiqos-22/?versioning 2023-07-28T23:54:37.906 DEBUG:boto:Method: PUT 2023-07-28T23:54:37.906 DEBUG:boto:Path: /hiiqos-22/?versioning 2023-07-28T23:54:37.907 DEBUG:boto:Data: Enabled Disabled 2023-07-28T23:54:37.907 DEBUG:boto:Headers: {} 2023-07-28T23:54:37.907 DEBUG:boto:Host: smithi139.front.sepia.ceph.com:8001 2023-07-28T23:54:37.908 DEBUG:boto:Port: 8001 2023-07-28T23:54:37.908 DEBUG:boto:Params: {} 2023-07-28T23:54:37.908 DEBUG:boto:Token: None 2023-07-28T23:54:37.909 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:54:37 GMT /hiiqos-22/?versioning 2023-07-28T23:54:37.909 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rBrOzvDCW89x7X06llcHJNrksGo= 2023-07-28T23:54:37.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rBrOzvDCW89x7X06llcHJNrksGo=', 'Content-Length': '228'} 2023-07-28T23:54:37.917 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c3671a7ebe5fbcfc-0064c4553d-110f-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:54:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:37.918 DEBUG:boto:path=/hiiqos-22/ 2023-07-28T23:54:37.918 DEBUG:boto:auth_path=/hiiqos-22/ 2023-07-28T23:54:37.918 DEBUG:boto:path=/hiiqos-22/?versioning 2023-07-28T23:54:37.919 DEBUG:boto:auth_path=/hiiqos-22/?versioning 2023-07-28T23:54:37.919 DEBUG:boto:Method: GET 2023-07-28T23:54:37.919 DEBUG:boto:Path: /hiiqos-22/?versioning 2023-07-28T23:54:37.920 DEBUG:boto:Data: 2023-07-28T23:54:37.920 DEBUG:boto:Headers: {} 2023-07-28T23:54:37.920 DEBUG:boto:Host: smithi139.front.sepia.ceph.com:8001 2023-07-28T23:54:37.921 DEBUG:boto:Port: 8001 2023-07-28T23:54:37.921 DEBUG:boto:Params: {} 2023-07-28T23:54:37.921 DEBUG:boto:Token: None 2023-07-28T23:54:37.922 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:54:37 GMT /hiiqos-22/?versioning 2023-07-28T23:54:37.922 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Ou2TapU+b/BMdUMjV1OhfRGnukU= 2023-07-28T23:54:37.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Ou2TapU+b/BMdUMjV1OhfRGnukU=', 'Content-Length': '0'} 2023-07-28T23:54:37.924 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000232f1efbc87a4e95-0064c4553d-110f-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Fri, 28 Jul 2023 23:54:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:37.924 DEBUG:boto:EnabledDisabled 2023-07-28T23:54:37.925 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_versioning ... ok 2023-07-28T23:54:37.925 INFO:rgw_multi.tests:create bucket zone=a1 name=hiiqos-23 2023-07-28T23:54:37.926 DEBUG:boto:path=/hiiqos-23/ 2023-07-28T23:54:37.926 DEBUG:boto:auth_path=/hiiqos-23/ 2023-07-28T23:54:37.926 DEBUG:boto:Method: PUT 2023-07-28T23:54:37.927 DEBUG:boto:Path: /hiiqos-23/ 2023-07-28T23:54:37.927 DEBUG:boto:Data: 2023-07-28T23:54:37.927 DEBUG:boto:Headers: {} 2023-07-28T23:54:37.928 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:54:37.928 DEBUG:boto:Port: 8000 2023-07-28T23:54:37.928 DEBUG:boto:Params: {} 2023-07-28T23:54:37.929 DEBUG:boto:Token: None 2023-07-28T23:54:37.929 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:54:37 GMT /hiiqos-23/ 2023-07-28T23:54:37.929 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2jeau0rFukBuQFafNh7uqCcXB0c= 2023-07-28T23:54:37.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2jeau0rFukBuQFafNh7uqCcXB0c=', 'Content-Length': '0'} 2023-07-28T23:54:37.954 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bae61feec8df54fc-0064c4553d-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:54:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:37.955 INFO:rgw_multi.tests:create bucket zone=a2 name=hiiqos-24 2023-07-28T23:54:37.955 DEBUG:boto:path=/hiiqos-24/ 2023-07-28T23:54:37.955 DEBUG:boto:auth_path=/hiiqos-24/ 2023-07-28T23:54:37.956 DEBUG:boto:Method: PUT 2023-07-28T23:54:37.956 DEBUG:boto:Path: /hiiqos-24/ 2023-07-28T23:54:37.956 DEBUG:boto:Data: 2023-07-28T23:54:37.957 DEBUG:boto:Headers: {} 2023-07-28T23:54:37.957 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:54:37.957 DEBUG:boto:Port: 8001 2023-07-28T23:54:37.958 DEBUG:boto:Params: {} 2023-07-28T23:54:37.958 DEBUG:boto:Token: None 2023-07-28T23:54:37.959 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:54:37 GMT /hiiqos-24/ 2023-07-28T23:54:37.959 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uu6dGEzFkHkoFvqD186uvzA7y98= 2023-07-28T23:54:37.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uu6dGEzFkHkoFvqD186uvzA7y98=', 'Content-Length': '0'} 2023-07-28T23:54:37.992 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000da988bc8bfa3b3d0-0064c4553d-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:54:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:37.993 INFO:rgw_multi.tests:create bucket zone=b1 name=hiiqos-25 2023-07-28T23:54:37.993 DEBUG:boto:path=/hiiqos-25/ 2023-07-28T23:54:37.993 DEBUG:boto:auth_path=/hiiqos-25/ 2023-07-28T23:54:37.994 DEBUG:boto:Method: PUT 2023-07-28T23:54:37.994 DEBUG:boto:Path: /hiiqos-25/ 2023-07-28T23:54:37.994 DEBUG:boto:Data: 2023-07-28T23:54:37.995 DEBUG:boto:Headers: {} 2023-07-28T23:54:37.995 DEBUG:boto:Host: smithi139.front.sepia.ceph.com:8000 2023-07-28T23:54:37.995 DEBUG:boto:Port: 8000 2023-07-28T23:54:37.996 DEBUG:boto:Params: {} 2023-07-28T23:54:37.996 DEBUG:boto:Token: None 2023-07-28T23:54:37.996 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:54:37 GMT /hiiqos-25/ 2023-07-28T23:54:37.997 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fD9M2xGKF/i1uo8/TwLGF+ftXGI= 2023-07-28T23:54:37.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fD9M2xGKF/i1uo8/TwLGF+ftXGI=', 'Content-Length': '0'} 2023-07-28T23:54:38.023 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f92eda4e0144d647-0064c4553d-1109-b1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:54:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:38.024 INFO:rgw_multi.tests:create bucket zone=b2 name=hiiqos-26 2023-07-28T23:54:38.024 DEBUG:boto:path=/hiiqos-26/ 2023-07-28T23:54:38.025 DEBUG:boto:auth_path=/hiiqos-26/ 2023-07-28T23:54:38.025 DEBUG:boto:Method: PUT 2023-07-28T23:54:38.025 DEBUG:boto:Path: /hiiqos-26/ 2023-07-28T23:54:38.026 DEBUG:boto:Data: 2023-07-28T23:54:38.026 DEBUG:boto:Headers: {} 2023-07-28T23:54:38.026 DEBUG:boto:Host: smithi139.front.sepia.ceph.com:8001 2023-07-28T23:54:38.027 DEBUG:boto:Port: 8001 2023-07-28T23:54:38.027 DEBUG:boto:Params: {} 2023-07-28T23:54:38.027 DEBUG:boto:Token: None 2023-07-28T23:54:38.028 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:54:38 GMT /hiiqos-26/ 2023-07-28T23:54:38.028 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wXh4z9h/taTMAnT5hdBON9OUCkA= 2023-07-28T23:54:38.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wXh4z9h/taTMAnT5hdBON9OUCkA=', 'Content-Length': '0'} 2023-07-28T23:54:38.061 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fd42c4335fb6e74d-0064c4553e-110f-b2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:54:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:38.061 DEBUG:boto:path=/hiiqos-23/ 2023-07-28T23:54:38.062 DEBUG:boto:auth_path=/hiiqos-23/ 2023-07-28T23:54:38.062 DEBUG:boto:path=/hiiqos-23/?acl 2023-07-28T23:54:38.062 DEBUG:boto:auth_path=/hiiqos-23/?acl 2023-07-28T23:54:38.063 DEBUG:boto:Method: GET 2023-07-28T23:54:38.063 DEBUG:boto:Path: /hiiqos-23/?acl 2023-07-28T23:54:38.063 DEBUG:boto:Data: 2023-07-28T23:54:38.064 DEBUG:boto:Headers: {} 2023-07-28T23:54:38.064 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:54:38.064 DEBUG:boto:Port: 8000 2023-07-28T23:54:38.065 DEBUG:boto:Params: {} 2023-07-28T23:54:38.065 DEBUG:boto:Token: None 2023-07-28T23:54:38.065 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:54:38 GMT /hiiqos-23/?acl 2023-07-28T23:54:38.066 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yaSDxCf6D94Yj/Daa5Y/niBhy/0= 2023-07-28T23:54:38.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yaSDxCf6D94Yj/Daa5Y/niBhy/0=', 'Content-Length': '0'} 2023-07-28T23:54:38.067 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c7f38dbeedf313cc-0064c4553e-1117-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Fri, 28 Jul 2023 23:54:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:38.068 DEBUG:boto:path=/hiiqos-23/ 2023-07-28T23:54:38.068 DEBUG:boto:auth_path=/hiiqos-23/ 2023-07-28T23:54:38.069 DEBUG:boto:path=/hiiqos-23/?acl 2023-07-28T23:54:38.069 DEBUG:boto:auth_path=/hiiqos-23/?acl 2023-07-28T23:54:38.069 DEBUG:boto:Method: PUT 2023-07-28T23:54:38.070 DEBUG:boto:Path: /hiiqos-23/?acl 2023-07-28T23:54:38.070 DEBUG:boto:Data: 2023-07-28T23:54:38.070 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-07-28T23:54:38.071 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:54:38.071 DEBUG:boto:Port: 8000 2023-07-28T23:54:38.071 DEBUG:boto:Params: {} 2023-07-28T23:54:38.072 DEBUG:boto:Token: None 2023-07-28T23:54:38.072 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:54:38 GMT x-amz-acl:public-read /hiiqos-23/?acl 2023-07-28T23:54:38.072 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TLj42m1z4+8CJZvtqY+7VkzpOuE= 2023-07-28T23:54:38.073 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TLj42m1z4+8CJZvtqY+7VkzpOuE=', 'Content-Length': '0'} 2023-07-28T23:54:38.077 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c2cb0d69b692354a-0064c4553e-1117-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:54:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:38.078 DEBUG:boto:path=/hiiqos-23/ 2023-07-28T23:54:38.078 DEBUG:boto:auth_path=/hiiqos-23/ 2023-07-28T23:54:38.078 DEBUG:boto:path=/hiiqos-23/?acl 2023-07-28T23:54:38.079 DEBUG:boto:auth_path=/hiiqos-23/?acl 2023-07-28T23:54:38.079 DEBUG:boto:Method: GET 2023-07-28T23:54:38.079 DEBUG:boto:Path: /hiiqos-23/?acl 2023-07-28T23:54:38.080 DEBUG:boto:Data: 2023-07-28T23:54:38.080 DEBUG:boto:Headers: {} 2023-07-28T23:54:38.080 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:54:38.081 DEBUG:boto:Port: 8000 2023-07-28T23:54:38.081 DEBUG:boto:Params: {} 2023-07-28T23:54:38.081 DEBUG:boto:Token: None 2023-07-28T23:54:38.082 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:54:38 GMT /hiiqos-23/?acl 2023-07-28T23:54:38.082 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yaSDxCf6D94Yj/Daa5Y/niBhy/0= 2023-07-28T23:54:38.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yaSDxCf6D94Yj/Daa5Y/niBhy/0=', 'Content-Length': '0'} 2023-07-28T23:54:38.083 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bc97d3f4b55c7275-0064c4553e-1117-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Fri, 28 Jul 2023 23:54:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:38.124 DEBUG:boto:path=/hiiqos-24/ 2023-07-28T23:54:38.125 DEBUG:boto:auth_path=/hiiqos-24/ 2023-07-28T23:54:38.125 DEBUG:boto:path=/hiiqos-24/?acl 2023-07-28T23:54:38.125 DEBUG:boto:auth_path=/hiiqos-24/?acl 2023-07-28T23:54:38.126 DEBUG:boto:Method: GET 2023-07-28T23:54:38.126 DEBUG:boto:Path: /hiiqos-24/?acl 2023-07-28T23:54:38.126 DEBUG:boto:Data: 2023-07-28T23:54:38.127 DEBUG:boto:Headers: {} 2023-07-28T23:54:38.127 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:54:38.127 DEBUG:boto:Port: 8001 2023-07-28T23:54:38.128 DEBUG:boto:Params: {} 2023-07-28T23:54:38.128 DEBUG:boto:Token: None 2023-07-28T23:54:38.128 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:54:38 GMT /hiiqos-24/?acl 2023-07-28T23:54:38.129 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:t145pqVzE25EgfIgVoe1jCk/B+I= 2023-07-28T23:54:38.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t145pqVzE25EgfIgVoe1jCk/B+I=', 'Content-Length': '0'} 2023-07-28T23:54:38.130 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b41ae202516f6227-0064c4553e-1113-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Fri, 28 Jul 2023 23:54:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:38.131 DEBUG:boto:path=/hiiqos-24/ 2023-07-28T23:54:38.131 DEBUG:boto:auth_path=/hiiqos-24/ 2023-07-28T23:54:38.132 DEBUG:boto:path=/hiiqos-24/?acl 2023-07-28T23:54:38.132 DEBUG:boto:auth_path=/hiiqos-24/?acl 2023-07-28T23:54:38.132 DEBUG:boto:Method: PUT 2023-07-28T23:54:38.133 DEBUG:boto:Path: /hiiqos-24/?acl 2023-07-28T23:54:38.133 DEBUG:boto:Data: 2023-07-28T23:54:38.133 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-07-28T23:54:38.134 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:54:38.134 DEBUG:boto:Port: 8001 2023-07-28T23:54:38.134 DEBUG:boto:Params: {} 2023-07-28T23:54:38.135 DEBUG:boto:Token: None 2023-07-28T23:54:38.135 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:54:38 GMT x-amz-acl:public-read /hiiqos-24/?acl 2023-07-28T23:54:38.135 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w1LZyT83geIjQjO8ovGxzlTZw48= 2023-07-28T23:54:38.136 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w1LZyT83geIjQjO8ovGxzlTZw48=', 'Content-Length': '0'} 2023-07-28T23:54:38.142 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000098bdc2f0419dd33c-0064c4553e-1113-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:54:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:38.143 DEBUG:boto:path=/hiiqos-24/ 2023-07-28T23:54:38.143 DEBUG:boto:auth_path=/hiiqos-24/ 2023-07-28T23:54:38.144 DEBUG:boto:path=/hiiqos-24/?acl 2023-07-28T23:54:38.144 DEBUG:boto:auth_path=/hiiqos-24/?acl 2023-07-28T23:54:38.144 DEBUG:boto:Method: GET 2023-07-28T23:54:38.145 DEBUG:boto:Path: /hiiqos-24/?acl 2023-07-28T23:54:38.145 DEBUG:boto:Data: 2023-07-28T23:54:38.145 DEBUG:boto:Headers: {} 2023-07-28T23:54:38.145 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:54:38.146 DEBUG:boto:Port: 8001 2023-07-28T23:54:38.146 DEBUG:boto:Params: {} 2023-07-28T23:54:38.146 DEBUG:boto:Token: None 2023-07-28T23:54:38.147 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:54:38 GMT /hiiqos-24/?acl 2023-07-28T23:54:38.147 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:t145pqVzE25EgfIgVoe1jCk/B+I= 2023-07-28T23:54:38.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t145pqVzE25EgfIgVoe1jCk/B+I=', 'Content-Length': '0'} 2023-07-28T23:54:38.149 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007fb3111e8b879b31-0064c4553e-1113-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Fri, 28 Jul 2023 23:54:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:38.149 DEBUG:boto:path=/hiiqos-25/ 2023-07-28T23:54:38.150 DEBUG:boto:auth_path=/hiiqos-25/ 2023-07-28T23:54:38.150 DEBUG:boto:path=/hiiqos-25/?acl 2023-07-28T23:54:38.150 DEBUG:boto:auth_path=/hiiqos-25/?acl 2023-07-28T23:54:38.150 DEBUG:boto:Method: GET 2023-07-28T23:54:38.151 DEBUG:boto:Path: /hiiqos-25/?acl 2023-07-28T23:54:38.151 DEBUG:boto:Data: 2023-07-28T23:54:38.151 DEBUG:boto:Headers: {} 2023-07-28T23:54:38.152 DEBUG:boto:Host: smithi139.front.sepia.ceph.com:8000 2023-07-28T23:54:38.152 DEBUG:boto:Port: 8000 2023-07-28T23:54:38.152 DEBUG:boto:Params: {} 2023-07-28T23:54:38.153 DEBUG:boto:Token: None 2023-07-28T23:54:38.153 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:54:38 GMT /hiiqos-25/?acl 2023-07-28T23:54:38.153 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tRv4iNDiPMsmrpp1RHVlg+HNXjE= 2023-07-28T23:54:38.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tRv4iNDiPMsmrpp1RHVlg+HNXjE=', 'Content-Length': '0'} 2023-07-28T23:54:38.155 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003f8d75ebc23c9eb4-0064c4553e-1109-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Fri, 28 Jul 2023 23:54:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:38.156 DEBUG:boto:path=/hiiqos-25/ 2023-07-28T23:54:38.156 DEBUG:boto:auth_path=/hiiqos-25/ 2023-07-28T23:54:38.156 DEBUG:boto:path=/hiiqos-25/?acl 2023-07-28T23:54:38.157 DEBUG:boto:auth_path=/hiiqos-25/?acl 2023-07-28T23:54:38.157 DEBUG:boto:Method: PUT 2023-07-28T23:54:38.157 DEBUG:boto:Path: /hiiqos-25/?acl 2023-07-28T23:54:38.158 DEBUG:boto:Data: 2023-07-28T23:54:38.158 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-07-28T23:54:38.158 DEBUG:boto:Host: smithi139.front.sepia.ceph.com:8000 2023-07-28T23:54:38.158 DEBUG:boto:Port: 8000 2023-07-28T23:54:38.159 DEBUG:boto:Params: {} 2023-07-28T23:54:38.159 DEBUG:boto:Token: None 2023-07-28T23:54:38.160 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:54:38 GMT x-amz-acl:public-read /hiiqos-25/?acl 2023-07-28T23:54:38.160 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:V2dwl1m73vIe3/N9hkxMFvGgFME= 2023-07-28T23:54:38.160 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:V2dwl1m73vIe3/N9hkxMFvGgFME=', 'Content-Length': '0'} 2023-07-28T23:54:38.169 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ed3c45bceb1fa51c-0064c4553e-1109-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:54:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:38.169 DEBUG:boto:path=/hiiqos-25/ 2023-07-28T23:54:38.170 DEBUG:boto:auth_path=/hiiqos-25/ 2023-07-28T23:54:38.170 DEBUG:boto:path=/hiiqos-25/?acl 2023-07-28T23:54:38.170 DEBUG:boto:auth_path=/hiiqos-25/?acl 2023-07-28T23:54:38.170 DEBUG:boto:Method: GET 2023-07-28T23:54:38.171 DEBUG:boto:Path: /hiiqos-25/?acl 2023-07-28T23:54:38.171 DEBUG:boto:Data: 2023-07-28T23:54:38.171 DEBUG:boto:Headers: {} 2023-07-28T23:54:38.172 DEBUG:boto:Host: smithi139.front.sepia.ceph.com:8000 2023-07-28T23:54:38.172 DEBUG:boto:Port: 8000 2023-07-28T23:54:38.172 DEBUG:boto:Params: {} 2023-07-28T23:54:38.173 DEBUG:boto:Token: None 2023-07-28T23:54:38.173 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:54:38 GMT /hiiqos-25/?acl 2023-07-28T23:54:38.173 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tRv4iNDiPMsmrpp1RHVlg+HNXjE= 2023-07-28T23:54:38.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tRv4iNDiPMsmrpp1RHVlg+HNXjE=', 'Content-Length': '0'} 2023-07-28T23:54:38.175 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000008fda059e25eb2e3-0064c4553e-1109-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Fri, 28 Jul 2023 23:54:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:38.176 DEBUG:boto:path=/hiiqos-26/ 2023-07-28T23:54:38.176 DEBUG:boto:auth_path=/hiiqos-26/ 2023-07-28T23:54:38.176 DEBUG:boto:path=/hiiqos-26/?acl 2023-07-28T23:54:38.177 DEBUG:boto:auth_path=/hiiqos-26/?acl 2023-07-28T23:54:38.177 DEBUG:boto:Method: GET 2023-07-28T23:54:38.177 DEBUG:boto:Path: /hiiqos-26/?acl 2023-07-28T23:54:38.177 DEBUG:boto:Data: 2023-07-28T23:54:38.178 DEBUG:boto:Headers: {} 2023-07-28T23:54:38.178 DEBUG:boto:Host: smithi139.front.sepia.ceph.com:8001 2023-07-28T23:54:38.178 DEBUG:boto:Port: 8001 2023-07-28T23:54:38.179 DEBUG:boto:Params: {} 2023-07-28T23:54:38.179 DEBUG:boto:Token: None 2023-07-28T23:54:38.179 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:54:38 GMT /hiiqos-26/?acl 2023-07-28T23:54:38.180 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QRCq1fbrM2TdmiEzPaO4qFO2eeU= 2023-07-28T23:54:38.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QRCq1fbrM2TdmiEzPaO4qFO2eeU=', 'Content-Length': '0'} 2023-07-28T23:54:38.181 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009cd5427cf6ad9b8b-0064c4553e-110f-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Fri, 28 Jul 2023 23:54:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:38.182 DEBUG:boto:path=/hiiqos-26/ 2023-07-28T23:54:38.182 DEBUG:boto:auth_path=/hiiqos-26/ 2023-07-28T23:54:38.183 DEBUG:boto:path=/hiiqos-26/?acl 2023-07-28T23:54:38.183 DEBUG:boto:auth_path=/hiiqos-26/?acl 2023-07-28T23:54:38.183 DEBUG:boto:Method: PUT 2023-07-28T23:54:38.184 DEBUG:boto:Path: /hiiqos-26/?acl 2023-07-28T23:54:38.184 DEBUG:boto:Data: 2023-07-28T23:54:38.184 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-07-28T23:54:38.185 DEBUG:boto:Host: smithi139.front.sepia.ceph.com:8001 2023-07-28T23:54:38.185 DEBUG:boto:Port: 8001 2023-07-28T23:54:38.185 DEBUG:boto:Params: {} 2023-07-28T23:54:38.186 DEBUG:boto:Token: None 2023-07-28T23:54:38.186 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:54:38 GMT x-amz-acl:public-read /hiiqos-26/?acl 2023-07-28T23:54:38.186 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Gn07tkbyQ80ghSQTNpGvDoMt/EU= 2023-07-28T23:54:38.187 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Gn07tkbyQ80ghSQTNpGvDoMt/EU=', 'Content-Length': '0'} 2023-07-28T23:54:38.195 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003c5c91e25169d42f-0064c4553e-110f-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:54:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:38.195 DEBUG:boto:path=/hiiqos-26/ 2023-07-28T23:54:38.196 DEBUG:boto:auth_path=/hiiqos-26/ 2023-07-28T23:54:38.196 DEBUG:boto:path=/hiiqos-26/?acl 2023-07-28T23:54:38.196 DEBUG:boto:auth_path=/hiiqos-26/?acl 2023-07-28T23:54:38.197 DEBUG:boto:Method: GET 2023-07-28T23:54:38.197 DEBUG:boto:Path: /hiiqos-26/?acl 2023-07-28T23:54:38.197 DEBUG:boto:Data: 2023-07-28T23:54:38.197 DEBUG:boto:Headers: {} 2023-07-28T23:54:38.198 DEBUG:boto:Host: smithi139.front.sepia.ceph.com:8001 2023-07-28T23:54:38.198 DEBUG:boto:Port: 8001 2023-07-28T23:54:38.198 DEBUG:boto:Params: {} 2023-07-28T23:54:38.199 DEBUG:boto:Token: None 2023-07-28T23:54:38.199 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:54:38 GMT /hiiqos-26/?acl 2023-07-28T23:54:38.199 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QRCq1fbrM2TdmiEzPaO4qFO2eeU= 2023-07-28T23:54:38.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QRCq1fbrM2TdmiEzPaO4qFO2eeU=', 'Content-Length': '0'} 2023-07-28T23:54:38.201 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000051b31938c3aecfe0-0064c4553e-110f-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Fri, 28 Jul 2023 23:54:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:38.202 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_acl ... ok 2023-07-28T23:54:38.202 INFO:rgw_multi.tests:create bucket zone=a1 name=hiiqos-27 2023-07-28T23:54:38.203 DEBUG:boto:path=/hiiqos-27/ 2023-07-28T23:54:38.203 DEBUG:boto:auth_path=/hiiqos-27/ 2023-07-28T23:54:38.204 DEBUG:boto:Method: PUT 2023-07-28T23:54:38.204 DEBUG:boto:Path: /hiiqos-27/ 2023-07-28T23:54:38.204 DEBUG:boto:Data: 2023-07-28T23:54:38.205 DEBUG:boto:Headers: {} 2023-07-28T23:54:38.205 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:54:38.205 DEBUG:boto:Port: 8000 2023-07-28T23:54:38.206 DEBUG:boto:Params: {} 2023-07-28T23:54:38.206 DEBUG:boto:Token: None 2023-07-28T23:54:38.206 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:54:38 GMT /hiiqos-27/ 2023-07-28T23:54:38.207 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YGU5apTiWYwGcOJlXQzwVMyFep0= 2023-07-28T23:54:38.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YGU5apTiWYwGcOJlXQzwVMyFep0=', 'Content-Length': '0'} 2023-07-28T23:54:38.222 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000047beb3b95318eea9-0064c4553e-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:54:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:38.222 INFO:rgw_multi.tests:create bucket zone=a2 name=hiiqos-28 2023-07-28T23:54:38.223 DEBUG:boto:path=/hiiqos-28/ 2023-07-28T23:54:38.223 DEBUG:boto:auth_path=/hiiqos-28/ 2023-07-28T23:54:38.223 DEBUG:boto:Method: PUT 2023-07-28T23:54:38.224 DEBUG:boto:Path: /hiiqos-28/ 2023-07-28T23:54:38.224 DEBUG:boto:Data: 2023-07-28T23:54:38.224 DEBUG:boto:Headers: {} 2023-07-28T23:54:38.225 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:54:38.225 DEBUG:boto:Port: 8001 2023-07-28T23:54:38.225 DEBUG:boto:Params: {} 2023-07-28T23:54:38.226 DEBUG:boto:Token: None 2023-07-28T23:54:38.226 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:54:38 GMT /hiiqos-28/ 2023-07-28T23:54:38.227 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kBzw6z+9qjqfJZZMumbu5SRGCng= 2023-07-28T23:54:38.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kBzw6z+9qjqfJZZMumbu5SRGCng=', 'Content-Length': '0'} 2023-07-28T23:54:38.255 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000700016beebac0b56-0064c4553e-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:54:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:38.256 INFO:rgw_multi.tests:create bucket zone=b1 name=hiiqos-29 2023-07-28T23:54:38.256 DEBUG:boto:path=/hiiqos-29/ 2023-07-28T23:54:38.256 DEBUG:boto:auth_path=/hiiqos-29/ 2023-07-28T23:54:38.257 DEBUG:boto:Method: PUT 2023-07-28T23:54:38.257 DEBUG:boto:Path: /hiiqos-29/ 2023-07-28T23:54:38.257 DEBUG:boto:Data: 2023-07-28T23:54:38.258 DEBUG:boto:Headers: {} 2023-07-28T23:54:38.258 DEBUG:boto:Host: smithi139.front.sepia.ceph.com:8000 2023-07-28T23:54:38.258 DEBUG:boto:Port: 8000 2023-07-28T23:54:38.259 DEBUG:boto:Params: {} 2023-07-28T23:54:38.259 DEBUG:boto:Token: None 2023-07-28T23:54:38.259 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:54:38 GMT /hiiqos-29/ 2023-07-28T23:54:38.260 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YSqYGepVkAaKcKfFCj3vODRiNFA= 2023-07-28T23:54:38.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YSqYGepVkAaKcKfFCj3vODRiNFA=', 'Content-Length': '0'} 2023-07-28T23:54:38.293 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c8a89ecbca0367a2-0064c4553e-1109-b1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:54:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:38.294 INFO:rgw_multi.tests:create bucket zone=b2 name=hiiqos-30 2023-07-28T23:54:38.294 DEBUG:boto:path=/hiiqos-30/ 2023-07-28T23:54:38.294 DEBUG:boto:auth_path=/hiiqos-30/ 2023-07-28T23:54:38.295 DEBUG:boto:Method: PUT 2023-07-28T23:54:38.295 DEBUG:boto:Path: /hiiqos-30/ 2023-07-28T23:54:38.295 DEBUG:boto:Data: 2023-07-28T23:54:38.296 DEBUG:boto:Headers: {} 2023-07-28T23:54:38.296 DEBUG:boto:Host: smithi139.front.sepia.ceph.com:8001 2023-07-28T23:54:38.296 DEBUG:boto:Port: 8001 2023-07-28T23:54:38.297 DEBUG:boto:Params: {} 2023-07-28T23:54:38.297 DEBUG:boto:Token: None 2023-07-28T23:54:38.298 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:54:38 GMT /hiiqos-30/ 2023-07-28T23:54:38.298 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jxB+8CY9o05V2Khynjzyi2k1aB0= 2023-07-28T23:54:38.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jxB+8CY9o05V2Khynjzyi2k1aB0=', 'Content-Length': '0'} 2023-07-28T23:54:38.333 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f0f66f4dc234dfd9-0064c4553e-110f-b2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:54:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:38.334 DEBUG:boto:path=/hiiqos-27/ 2023-07-28T23:54:38.334 DEBUG:boto:auth_path=/hiiqos-27/ 2023-07-28T23:54:38.334 DEBUG:boto:path=/hiiqos-27/?cors 2023-07-28T23:54:38.335 DEBUG:boto:auth_path=/hiiqos-27/?cors 2023-07-28T23:54:38.335 DEBUG:boto:Method: PUT 2023-07-28T23:54:38.335 DEBUG:boto:Path: /hiiqos-27/?cors 2023-07-28T23:54:38.336 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-07-28T23:54:38.336 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-07-28T23:54:38.336 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:54:38.337 DEBUG:boto:Port: 8000 2023-07-28T23:54:38.337 DEBUG:boto:Params: {} 2023-07-28T23:54:38.337 DEBUG:boto:Token: None 2023-07-28T23:54:38.338 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Fri, 28 Jul 2023 23:54:38 GMT /hiiqos-27/?cors 2023-07-28T23:54:38.338 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:K3QlBfKXnW5YVZWHe2jx0Gkz/OM= 2023-07-28T23:54:38.339 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K3QlBfKXnW5YVZWHe2jx0Gkz/OM=', 'Content-Length': '218'} 2023-07-28T23:54:38.343 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000029a09272335b58f2-0064c4553e-1117-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:54:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:38.344 DEBUG:boto:path=/hiiqos-27/ 2023-07-28T23:54:38.344 DEBUG:boto:auth_path=/hiiqos-27/ 2023-07-28T23:54:38.345 DEBUG:boto:path=/hiiqos-27/?cors 2023-07-28T23:54:38.345 DEBUG:boto:auth_path=/hiiqos-27/?cors 2023-07-28T23:54:38.345 DEBUG:boto:Method: GET 2023-07-28T23:54:38.346 DEBUG:boto:Path: /hiiqos-27/?cors 2023-07-28T23:54:38.346 DEBUG:boto:Data: 2023-07-28T23:54:38.346 DEBUG:boto:Headers: {} 2023-07-28T23:54:38.347 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:54:38.347 DEBUG:boto:Port: 8000 2023-07-28T23:54:38.347 DEBUG:boto:Params: {} 2023-07-28T23:54:38.347 DEBUG:boto:Token: None 2023-07-28T23:54:38.348 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:54:38 GMT /hiiqos-27/?cors 2023-07-28T23:54:38.348 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BgMFLveS/0fQehSCecc6EXQmb+Y= 2023-07-28T23:54:38.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BgMFLveS/0fQehSCecc6EXQmb+Y=', 'Content-Length': '0'} 2023-07-28T23:54:38.350 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f40b085fb07d5836-0064c4553e-1117-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Fri, 28 Jul 2023 23:54:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:38.351 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-07-28T23:54:38.351 DEBUG:boto:path=/hiiqos-28/ 2023-07-28T23:54:38.352 DEBUG:boto:auth_path=/hiiqos-28/ 2023-07-28T23:54:38.352 DEBUG:boto:path=/hiiqos-28/?cors 2023-07-28T23:54:38.353 DEBUG:boto:auth_path=/hiiqos-28/?cors 2023-07-28T23:54:38.353 DEBUG:boto:Method: PUT 2023-07-28T23:54:38.353 DEBUG:boto:Path: /hiiqos-28/?cors 2023-07-28T23:54:38.353 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-07-28T23:54:38.354 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-07-28T23:54:38.354 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:54:38.354 DEBUG:boto:Port: 8001 2023-07-28T23:54:38.355 DEBUG:boto:Params: {} 2023-07-28T23:54:38.355 DEBUG:boto:Token: None 2023-07-28T23:54:38.355 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Fri, 28 Jul 2023 23:54:38 GMT /hiiqos-28/?cors 2023-07-28T23:54:38.356 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IOMgU6jeX3WxAgq48WwQjJYrmkQ= 2023-07-28T23:54:38.356 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IOMgU6jeX3WxAgq48WwQjJYrmkQ=', 'Content-Length': '218'} 2023-07-28T23:54:39.364 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000088bbe9fe2f10f537-0064c4553e-1113-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:54:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:39.365 DEBUG:boto:path=/hiiqos-28/ 2023-07-28T23:54:39.365 DEBUG:boto:auth_path=/hiiqos-28/ 2023-07-28T23:54:39.366 DEBUG:boto:path=/hiiqos-28/?cors 2023-07-28T23:54:39.366 DEBUG:boto:auth_path=/hiiqos-28/?cors 2023-07-28T23:54:39.366 DEBUG:boto:Method: GET 2023-07-28T23:54:39.367 DEBUG:boto:Path: /hiiqos-28/?cors 2023-07-28T23:54:39.367 DEBUG:boto:Data: 2023-07-28T23:54:39.367 DEBUG:boto:Headers: {} 2023-07-28T23:54:39.368 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:54:39.368 DEBUG:boto:Port: 8001 2023-07-28T23:54:39.368 DEBUG:boto:Params: {} 2023-07-28T23:54:39.369 DEBUG:boto:Token: None 2023-07-28T23:54:39.369 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:54:39 GMT /hiiqos-28/?cors 2023-07-28T23:54:39.369 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7gges8Vcv1ty3LBlQuhfTcN1YUY= 2023-07-28T23:54:39.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7gges8Vcv1ty3LBlQuhfTcN1YUY=', 'Content-Length': '0'} 2023-07-28T23:54:39.371 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002a32474b6d04ece7-0064c4553f-1113-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Fri, 28 Jul 2023 23:54:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:39.372 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-07-28T23:54:39.372 DEBUG:boto:path=/hiiqos-29/ 2023-07-28T23:54:39.373 DEBUG:boto:auth_path=/hiiqos-29/ 2023-07-28T23:54:39.373 DEBUG:boto:path=/hiiqos-29/?cors 2023-07-28T23:54:39.373 DEBUG:boto:auth_path=/hiiqos-29/?cors 2023-07-28T23:54:39.374 DEBUG:boto:Method: PUT 2023-07-28T23:54:39.374 DEBUG:boto:Path: /hiiqos-29/?cors 2023-07-28T23:54:39.374 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-07-28T23:54:39.375 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-07-28T23:54:39.375 DEBUG:boto:Host: smithi139.front.sepia.ceph.com:8000 2023-07-28T23:54:39.375 DEBUG:boto:Port: 8000 2023-07-28T23:54:39.375 DEBUG:boto:Params: {} 2023-07-28T23:54:39.376 DEBUG:boto:Token: None 2023-07-28T23:54:39.376 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Fri, 28 Jul 2023 23:54:39 GMT /hiiqos-29/?cors 2023-07-28T23:54:39.377 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CuM2iDEOtmHAHaWZP7IHVLqTHRY= 2023-07-28T23:54:39.377 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CuM2iDEOtmHAHaWZP7IHVLqTHRY=', 'Content-Length': '218'} 2023-07-28T23:54:40.388 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c08129d33dc065f0-0064c4553f-1109-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:54:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:40.388 DEBUG:boto:path=/hiiqos-29/ 2023-07-28T23:54:40.389 DEBUG:boto:auth_path=/hiiqos-29/ 2023-07-28T23:54:40.389 DEBUG:boto:path=/hiiqos-29/?cors 2023-07-28T23:54:40.389 DEBUG:boto:auth_path=/hiiqos-29/?cors 2023-07-28T23:54:40.390 DEBUG:boto:Method: GET 2023-07-28T23:54:40.390 DEBUG:boto:Path: /hiiqos-29/?cors 2023-07-28T23:54:40.390 DEBUG:boto:Data: 2023-07-28T23:54:40.391 DEBUG:boto:Headers: {} 2023-07-28T23:54:40.391 DEBUG:boto:Host: smithi139.front.sepia.ceph.com:8000 2023-07-28T23:54:40.391 DEBUG:boto:Port: 8000 2023-07-28T23:54:40.392 DEBUG:boto:Params: {} 2023-07-28T23:54:40.392 DEBUG:boto:Token: None 2023-07-28T23:54:40.392 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:54:40 GMT /hiiqos-29/?cors 2023-07-28T23:54:40.393 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nDocWLRwa7ivPWhMCYYoyhBl8dE= 2023-07-28T23:54:40.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nDocWLRwa7ivPWhMCYYoyhBl8dE=', 'Content-Length': '0'} 2023-07-28T23:54:40.394 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000014c4e219409cbed3-0064c45540-1109-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Fri, 28 Jul 2023 23:54:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:40.436 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-07-28T23:54:40.437 DEBUG:boto:path=/hiiqos-30/ 2023-07-28T23:54:40.437 DEBUG:boto:auth_path=/hiiqos-30/ 2023-07-28T23:54:40.438 DEBUG:boto:path=/hiiqos-30/?cors 2023-07-28T23:54:40.438 DEBUG:boto:auth_path=/hiiqos-30/?cors 2023-07-28T23:54:40.438 DEBUG:boto:Method: PUT 2023-07-28T23:54:40.439 DEBUG:boto:Path: /hiiqos-30/?cors 2023-07-28T23:54:40.439 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-07-28T23:54:40.439 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-07-28T23:54:40.440 DEBUG:boto:Host: smithi139.front.sepia.ceph.com:8001 2023-07-28T23:54:40.440 DEBUG:boto:Port: 8001 2023-07-28T23:54:40.440 DEBUG:boto:Params: {} 2023-07-28T23:54:40.441 DEBUG:boto:Token: None 2023-07-28T23:54:40.441 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Fri, 28 Jul 2023 23:54:40 GMT /hiiqos-30/?cors 2023-07-28T23:54:40.441 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TUbVeE19HZRxtNhHGKiZ/moqWms= 2023-07-28T23:54:40.442 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TUbVeE19HZRxtNhHGKiZ/moqWms=', 'Content-Length': '218'} 2023-07-28T23:54:41.453 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d561c27eb4cca113-0064c45540-110f-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:54:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:41.454 DEBUG:boto:path=/hiiqos-30/ 2023-07-28T23:54:41.455 DEBUG:boto:auth_path=/hiiqos-30/ 2023-07-28T23:54:41.455 DEBUG:boto:path=/hiiqos-30/?cors 2023-07-28T23:54:41.455 DEBUG:boto:auth_path=/hiiqos-30/?cors 2023-07-28T23:54:41.456 DEBUG:boto:Method: GET 2023-07-28T23:54:41.456 DEBUG:boto:Path: /hiiqos-30/?cors 2023-07-28T23:54:41.456 DEBUG:boto:Data: 2023-07-28T23:54:41.457 DEBUG:boto:Headers: {} 2023-07-28T23:54:41.457 DEBUG:boto:Host: smithi139.front.sepia.ceph.com:8001 2023-07-28T23:54:41.458 DEBUG:boto:Port: 8001 2023-07-28T23:54:41.459 DEBUG:boto:Params: {} 2023-07-28T23:54:41.460 DEBUG:boto:Token: None 2023-07-28T23:54:41.461 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:54:41 GMT /hiiqos-30/?cors 2023-07-28T23:54:41.462 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:M+nXfH5BJodKLkmLIcmkXJv3NDo= 2023-07-28T23:54:41.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:M+nXfH5BJodKLkmLIcmkXJv3NDo=', 'Content-Length': '0'} 2023-07-28T23:54:41.466 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000abb86a5f81c05b6f-0064c45541-110f-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Fri, 28 Jul 2023 23:54:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:41.509 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-07-28T23:54:41.510 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_cors ... ok 2023-07-28T23:54:41.511 INFO:rgw_multi.tests:create bucket zone=a1 name=hiiqos-31 2023-07-28T23:54:41.512 DEBUG:boto:path=/hiiqos-31/ 2023-07-28T23:54:41.512 DEBUG:boto:auth_path=/hiiqos-31/ 2023-07-28T23:54:41.513 DEBUG:boto:Method: PUT 2023-07-28T23:54:41.513 DEBUG:boto:Path: /hiiqos-31/ 2023-07-28T23:54:41.514 DEBUG:boto:Data: 2023-07-28T23:54:41.514 DEBUG:boto:Headers: {} 2023-07-28T23:54:41.514 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:54:41.515 DEBUG:boto:Port: 8000 2023-07-28T23:54:41.515 DEBUG:boto:Params: {} 2023-07-28T23:54:41.515 DEBUG:boto:Token: None 2023-07-28T23:54:41.516 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:54:41 GMT /hiiqos-31/ 2023-07-28T23:54:41.516 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Ar+84YLiipsllAZ1Diw+2kai3aw= 2023-07-28T23:54:41.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Ar+84YLiipsllAZ1Diw+2kai3aw=', 'Content-Length': '0'} 2023-07-28T23:54:41.533 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000089d4194ae0430999-0064c45541-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:54:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:41.534 INFO:rgw_multi.tests:create bucket zone=a2 name=hiiqos-32 2023-07-28T23:54:41.535 DEBUG:boto:path=/hiiqos-32/ 2023-07-28T23:54:41.536 DEBUG:boto:auth_path=/hiiqos-32/ 2023-07-28T23:54:41.537 DEBUG:boto:Method: PUT 2023-07-28T23:54:41.538 DEBUG:boto:Path: /hiiqos-32/ 2023-07-28T23:54:41.538 DEBUG:boto:Data: 2023-07-28T23:54:41.539 DEBUG:boto:Headers: {} 2023-07-28T23:54:41.540 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:54:41.541 DEBUG:boto:Port: 8001 2023-07-28T23:54:41.542 DEBUG:boto:Params: {} 2023-07-28T23:54:41.543 DEBUG:boto:Token: None 2023-07-28T23:54:41.543 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:54:41 GMT /hiiqos-32/ 2023-07-28T23:54:41.544 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0CFNRFcE30PmoIOKxgSvDXl1SNs= 2023-07-28T23:54:41.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0CFNRFcE30PmoIOKxgSvDXl1SNs=', 'Content-Length': '0'} 2023-07-28T23:54:41.573 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000450f1f16b348cd6f-0064c45541-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:54:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:41.574 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:54:41.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:54:41.575 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:54:41.630 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:54:41.631 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:54:41.671 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-28T23:54:41.683 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-28T23:54:41.684 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:41.684 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-28T23:54:41.684 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:46:09.233742Z" 2023-07-28T23:54:41.685 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:41.685 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:41.685 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588481.561613_329.1", 2023-07-28T23:54:41.686 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:54:41.561613Z" 2023-07-28T23:54:41.686 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:41.686 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:41.686 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:54:41.687 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:54:41.687 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:41.687 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:41.687 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:54:41.688 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:54:41.688 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:41.688 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-28T23:54:41.689 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690587969.233742_99.1', 1: '1_1690588481.561613_329.1', 2: '', 3: ''} 2023-07-28T23:54:41.689 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-28T23:54:41.690 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:54:41.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:54:41.691 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:54:41.736 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:54:41.737 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:54:41.764 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:54:41.761+0000 7f1fd574b880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:54:41.765 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:54:41.762+0000 7f1fd574b880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:54:41.775 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:54:41.776 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:54:41.776 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:54:41.776 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:54:41.776 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:54:41.776 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:54:41.776 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:54:41.777 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:41.777 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:54:41.777 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:41.777 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:54:41.777 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:54:41.777 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:54:41.777 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-28T23:54:41.778 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:54:41.778 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:54:41.778 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:54:41.778 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-28T23:54:41.778 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:54:41.779 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:41.779 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:41.779 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:41.779 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:54:41.779 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:54:41.780 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:54:41.780 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588481.561613_329.1", 2023-07-28T23:54:41.780 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:54:41.780 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:54:41.780 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:54:41.781 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:54:41.561613Z", 2023-07-28T23:54:41.781 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:54:41.781 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:41.781 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:41.781 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:41.781 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:54:41.782 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:54:41.782 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:54:41.782 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:54:41.782 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:54:41.782 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:54:41.782 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:54:41.782 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:54:41.783 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:54:41.783 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:41.783 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:41.784 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:41.784 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:54:41.784 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:54:41.784 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:54:41.784 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:54:41.784 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:54:41.785 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:54:41.785 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:54:41.785 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:54:41.785 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:54:41.785 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:41.785 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:41.785 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:54:41.786 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:41.786 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:54:41.786 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:54:41.786 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:54:41.786 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:41.786 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:54:41Z" 2023-07-28T23:54:41.786 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:54:41.787 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588481.561613_329.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:54:41.561613Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-28T23:54:41Z" } 2023-07-28T23:54:41.788 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588481.561613_329.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:54:41.561613Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-28T23:54:41.788 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588481.561613_329.1', 2: '', 3: ''} 2023-07-28T23:54:41.789 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588481.561613_329.1', 2: '', 3: ''} 2023-07-28T23:54:41.789 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-28T23:54:41.789 DEBUG:boto:path=/hiiqos-31/ 2023-07-28T23:54:41.790 DEBUG:boto:auth_path=/hiiqos-31/ 2023-07-28T23:54:41.790 DEBUG:boto:Method: HEAD 2023-07-28T23:54:41.790 DEBUG:boto:Path: /hiiqos-31/ 2023-07-28T23:54:41.791 DEBUG:boto:Data: 2023-07-28T23:54:41.791 DEBUG:boto:Headers: {} 2023-07-28T23:54:41.791 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:54:41.792 DEBUG:boto:Port: 8000 2023-07-28T23:54:41.792 DEBUG:boto:Params: {} 2023-07-28T23:54:41.792 DEBUG:boto:Token: None 2023-07-28T23:54:41.793 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:54:41 GMT /hiiqos-31/ 2023-07-28T23:54:41.793 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:942HGi8t0ffXsBeYcm0iB4QAAj0= 2023-07-28T23:54:41.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:942HGi8t0ffXsBeYcm0iB4QAAj0=', 'Content-Length': '0'} 2023-07-28T23:54:41.796 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b13affb76beb9486-0064c45541-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:54:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:41.796 DEBUG:boto:path=/hiiqos-31/foo 2023-07-28T23:54:41.797 DEBUG:boto:auth_path=/hiiqos-31/foo 2023-07-28T23:54:41.797 DEBUG:boto:Method: PUT 2023-07-28T23:54:41.797 DEBUG:boto:Path: /hiiqos-31/foo 2023-07-28T23:54:41.798 DEBUG:boto:Data: 2023-07-28T23:54:41.798 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-28T23:54:41.798 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:54:41.799 DEBUG:boto:Port: 8000 2023-07-28T23:54:41.799 DEBUG:boto:Params: {} 2023-07-28T23:54:41.800 DEBUG:boto:Token: None 2023-07-28T23:54:41.800 DEBUG:boto:StringToSign: PUT N7UdGUp1E+RbVvZSTy1R8g== application/octet-stream Fri, 28 Jul 2023 23:54:41 GMT /hiiqos-31/foo 2023-07-28T23:54:41.800 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Mh9iuBtKjWX7JYvIqT2NNgHaV8M= 2023-07-28T23:54:41.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:54:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Mh9iuBtKjWX7JYvIqT2NNgHaV8M='} 2023-07-28T23:54:41.806 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"37b51d194a7513e45b56f6524f2d51f2"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004b441efd1d8dbfee-0064c45541-1117-a1'), ('Date', 'Fri, 28 Jul 2023 23:54:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:41.807 DEBUG:boto:path=/hiiqos-31/ 2023-07-28T23:54:41.807 DEBUG:boto:auth_path=/hiiqos-31/ 2023-07-28T23:54:41.807 DEBUG:boto:Method: DELETE 2023-07-28T23:54:41.808 DEBUG:boto:Path: /hiiqos-31/ 2023-07-28T23:54:41.808 DEBUG:boto:Data: 2023-07-28T23:54:41.809 DEBUG:boto:Headers: {} 2023-07-28T23:54:41.809 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:54:41.810 DEBUG:boto:Port: 8000 2023-07-28T23:54:41.810 DEBUG:boto:Params: {} 2023-07-28T23:54:41.810 DEBUG:boto:Token: None 2023-07-28T23:54:41.811 DEBUG:boto:StringToSign: DELETE Fri, 28 Jul 2023 23:54:41 GMT /hiiqos-31/ 2023-07-28T23:54:41.811 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:o2RKi/75TXVEq8DDTGO29zyo2Ag= 2023-07-28T23:54:41.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:o2RKi/75TXVEq8DDTGO29zyo2Ag=', 'Content-Length': '0'} 2023-07-28T23:54:41.817 DEBUG:boto:Response headers: [('Content-Length', '224'), ('x-amz-request-id', 'tx00000f97655e12a2b09b4-0064c45541-1117-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:54:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:41.818 DEBUG:boto:path=/hiiqos-32/ 2023-07-28T23:54:41.818 DEBUG:boto:auth_path=/hiiqos-32/ 2023-07-28T23:54:41.819 DEBUG:boto:Method: HEAD 2023-07-28T23:54:41.819 DEBUG:boto:Path: /hiiqos-32/ 2023-07-28T23:54:41.819 DEBUG:boto:Data: 2023-07-28T23:54:41.820 DEBUG:boto:Headers: {} 2023-07-28T23:54:41.820 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:54:41.820 DEBUG:boto:Port: 8001 2023-07-28T23:54:41.821 DEBUG:boto:Params: {} 2023-07-28T23:54:41.821 DEBUG:boto:Token: None 2023-07-28T23:54:41.821 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:54:41 GMT /hiiqos-32/ 2023-07-28T23:54:41.822 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:593bvPRvxYUT/mfZpCi2hVSoBUM= 2023-07-28T23:54:41.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:593bvPRvxYUT/mfZpCi2hVSoBUM=', 'Content-Length': '0'} 2023-07-28T23:54:41.824 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007888fce5f104905d-0064c45541-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:54:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:41.825 DEBUG:boto:path=/hiiqos-32/foo 2023-07-28T23:54:41.825 DEBUG:boto:auth_path=/hiiqos-32/foo 2023-07-28T23:54:41.826 DEBUG:boto:Method: PUT 2023-07-28T23:54:41.826 DEBUG:boto:Path: /hiiqos-32/foo 2023-07-28T23:54:41.826 DEBUG:boto:Data: 2023-07-28T23:54:41.827 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-28T23:54:41.827 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:54:41.827 DEBUG:boto:Port: 8001 2023-07-28T23:54:41.828 DEBUG:boto:Params: {} 2023-07-28T23:54:41.828 DEBUG:boto:Token: None 2023-07-28T23:54:41.828 DEBUG:boto:StringToSign: PUT N7UdGUp1E+RbVvZSTy1R8g== application/octet-stream Fri, 28 Jul 2023 23:54:41 GMT /hiiqos-32/foo 2023-07-28T23:54:41.829 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OhSzeIlG302NoAZSXyldTlBN31Q= 2023-07-28T23:54:41.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:54:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OhSzeIlG302NoAZSXyldTlBN31Q='} 2023-07-28T23:54:41.872 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"37b51d194a7513e45b56f6524f2d51f2"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009fa2458a2e8068d2-0064c45541-1113-a2'), ('Date', 'Fri, 28 Jul 2023 23:54:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:41.873 DEBUG:boto:path=/hiiqos-32/ 2023-07-28T23:54:41.874 DEBUG:boto:auth_path=/hiiqos-32/ 2023-07-28T23:54:41.874 DEBUG:boto:Method: DELETE 2023-07-28T23:54:41.874 DEBUG:boto:Path: /hiiqos-32/ 2023-07-28T23:54:41.875 DEBUG:boto:Data: 2023-07-28T23:54:41.875 DEBUG:boto:Headers: {} 2023-07-28T23:54:41.875 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:54:41.876 DEBUG:boto:Port: 8001 2023-07-28T23:54:41.876 DEBUG:boto:Params: {} 2023-07-28T23:54:41.876 DEBUG:boto:Token: None 2023-07-28T23:54:41.877 DEBUG:boto:StringToSign: DELETE Fri, 28 Jul 2023 23:54:41 GMT /hiiqos-32/ 2023-07-28T23:54:41.877 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:h5TGttjROvEKuPWCpRCmS/ppyG0= 2023-07-28T23:54:41.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:h5TGttjROvEKuPWCpRCmS/ppyG0=', 'Content-Length': '0'} 2023-07-28T23:54:41.884 DEBUG:boto:Response headers: [('Content-Length', '224'), ('x-amz-request-id', 'tx00000c5a411e39697f17a-0064c45541-1113-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:54:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:41.885 DEBUG:boto:path=/hiiqos-31/ 2023-07-28T23:54:41.885 DEBUG:boto:auth_path=/hiiqos-31/ 2023-07-28T23:54:41.885 DEBUG:boto:Method: HEAD 2023-07-28T23:54:41.886 DEBUG:boto:Path: /hiiqos-31/ 2023-07-28T23:54:41.886 DEBUG:boto:Data: 2023-07-28T23:54:41.886 DEBUG:boto:Headers: {} 2023-07-28T23:54:41.887 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:54:41.887 DEBUG:boto:Port: 8000 2023-07-28T23:54:41.887 DEBUG:boto:Params: {} 2023-07-28T23:54:41.888 DEBUG:boto:Token: None 2023-07-28T23:54:41.888 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:54:41 GMT /hiiqos-31/ 2023-07-28T23:54:41.889 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:942HGi8t0ffXsBeYcm0iB4QAAj0= 2023-07-28T23:54:41.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:942HGi8t0ffXsBeYcm0iB4QAAj0=', 'Content-Length': '0'} 2023-07-28T23:54:41.891 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000a099a4a50bac355-0064c45541-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:54:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:41.892 DEBUG:boto:path=/hiiqos-32/ 2023-07-28T23:54:41.892 DEBUG:boto:auth_path=/hiiqos-32/ 2023-07-28T23:54:41.892 DEBUG:boto:Method: HEAD 2023-07-28T23:54:41.893 DEBUG:boto:Path: /hiiqos-32/ 2023-07-28T23:54:41.893 DEBUG:boto:Data: 2023-07-28T23:54:41.893 DEBUG:boto:Headers: {} 2023-07-28T23:54:41.894 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:54:41.894 DEBUG:boto:Port: 8000 2023-07-28T23:54:41.894 DEBUG:boto:Params: {} 2023-07-28T23:54:41.895 DEBUG:boto:Token: None 2023-07-28T23:54:41.895 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:54:41 GMT /hiiqos-32/ 2023-07-28T23:54:41.895 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:593bvPRvxYUT/mfZpCi2hVSoBUM= 2023-07-28T23:54:41.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:593bvPRvxYUT/mfZpCi2hVSoBUM=', 'Content-Length': '0'} 2023-07-28T23:54:41.898 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000431b625a6941d4a-0064c45541-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:54:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:41.898 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_delete_notempty ... ok 2023-07-28T23:54:41.899 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_period_incremental_sync ... SKIP: test_multi_period_incremental_sync skipped. Requires 3 or more zones in master zonegroup. 2023-07-28T23:54:41.901 INFO:rgw_multi.tests:create bucket zone=a1 name=hiiqos-33 2023-07-28T23:54:41.901 DEBUG:boto:path=/hiiqos-33/ 2023-07-28T23:54:41.902 DEBUG:boto:auth_path=/hiiqos-33/ 2023-07-28T23:54:41.902 DEBUG:boto:Method: PUT 2023-07-28T23:54:41.903 DEBUG:boto:Path: /hiiqos-33/ 2023-07-28T23:54:41.903 DEBUG:boto:Data: 2023-07-28T23:54:41.904 DEBUG:boto:Headers: {} 2023-07-28T23:54:41.904 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:54:41.904 DEBUG:boto:Port: 8000 2023-07-28T23:54:41.905 DEBUG:boto:Params: {} 2023-07-28T23:54:41.905 DEBUG:boto:Token: None 2023-07-28T23:54:41.906 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:54:41 GMT /hiiqos-33/ 2023-07-28T23:54:41.906 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QmjsNnVbPeTrEWeM9EccgQZVnEc= 2023-07-28T23:54:41.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QmjsNnVbPeTrEWeM9EccgQZVnEc=', 'Content-Length': '0'} 2023-07-28T23:54:41.919 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000091797ec6390057fe-0064c45541-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:54:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:41.920 INFO:rgw_multi.tests:create bucket zone=a2 name=hiiqos-34 2023-07-28T23:54:41.920 DEBUG:boto:path=/hiiqos-34/ 2023-07-28T23:54:41.920 DEBUG:boto:auth_path=/hiiqos-34/ 2023-07-28T23:54:41.921 DEBUG:boto:Method: PUT 2023-07-28T23:54:41.921 DEBUG:boto:Path: /hiiqos-34/ 2023-07-28T23:54:41.922 DEBUG:boto:Data: 2023-07-28T23:54:41.922 DEBUG:boto:Headers: {} 2023-07-28T23:54:41.922 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:54:41.923 DEBUG:boto:Port: 8001 2023-07-28T23:54:41.923 DEBUG:boto:Params: {} 2023-07-28T23:54:41.923 DEBUG:boto:Token: None 2023-07-28T23:54:41.924 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:54:41 GMT /hiiqos-34/ 2023-07-28T23:54:41.924 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oAy3Vpc4ugwOfJE7kkES6hdUP7U= 2023-07-28T23:54:41.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oAy3Vpc4ugwOfJE7kkES6hdUP7U=', 'Content-Length': '0'} 2023-07-28T23:54:41.952 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b3ece0efb9969b1f-0064c45541-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:54:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:41.952 DEBUG:boto:path=/hiiqos-33/ 2023-07-28T23:54:41.953 DEBUG:boto:auth_path=/hiiqos-33/ 2023-07-28T23:54:41.953 DEBUG:boto:Method: HEAD 2023-07-28T23:54:41.954 DEBUG:boto:Path: /hiiqos-33/ 2023-07-28T23:54:41.954 DEBUG:boto:Data: 2023-07-28T23:54:41.954 DEBUG:boto:Headers: {} 2023-07-28T23:54:41.955 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:54:41.955 DEBUG:boto:Port: 8000 2023-07-28T23:54:41.955 DEBUG:boto:Params: {} 2023-07-28T23:54:41.956 DEBUG:boto:Token: None 2023-07-28T23:54:41.956 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:54:41 GMT /hiiqos-33/ 2023-07-28T23:54:41.956 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BNNSK1/V6GzID7JoMgZAMXJjNsY= 2023-07-28T23:54:41.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BNNSK1/V6GzID7JoMgZAMXJjNsY=', 'Content-Length': '0'} 2023-07-28T23:54:41.959 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000093e387582e0bd98-0064c45541-1117-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:54:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:41.960 DEBUG:boto:path=/hiiqos-33/key 2023-07-28T23:54:41.960 DEBUG:boto:auth_path=/hiiqos-33/key 2023-07-28T23:54:41.960 DEBUG:boto:Method: PUT 2023-07-28T23:54:41.961 DEBUG:boto:Path: /hiiqos-33/key 2023-07-28T23:54:41.961 DEBUG:boto:Data: 2023-07-28T23:54:41.962 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2023-07-28T23:54:41.963 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:54:41.963 DEBUG:boto:Port: 8000 2023-07-28T23:54:41.964 DEBUG:boto:Params: {} 2023-07-28T23:54:41.964 DEBUG:boto:Token: None 2023-07-28T23:54:41.964 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Fri, 28 Jul 2023 23:54:41 GMT /hiiqos-33/key 2023-07-28T23:54:41.965 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MQ1+ZG/bpzwDMF3ci/Rc9VxN2gM= 2023-07-28T23:54:41.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:54:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MQ1+ZG/bpzwDMF3ci/Rc9VxN2gM='} 2023-07-28T23:54:41.973 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c85dbbfce24e2c25-0064c45541-1117-a1'), ('Date', 'Fri, 28 Jul 2023 23:54:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:41.974 DEBUG:boto:path=/hiiqos-34/ 2023-07-28T23:54:41.974 DEBUG:boto:auth_path=/hiiqos-34/ 2023-07-28T23:54:41.974 DEBUG:boto:Method: HEAD 2023-07-28T23:54:41.975 DEBUG:boto:Path: /hiiqos-34/ 2023-07-28T23:54:41.975 DEBUG:boto:Data: 2023-07-28T23:54:41.975 DEBUG:boto:Headers: {} 2023-07-28T23:54:41.976 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:54:41.976 DEBUG:boto:Port: 8001 2023-07-28T23:54:41.976 DEBUG:boto:Params: {} 2023-07-28T23:54:41.977 DEBUG:boto:Token: None 2023-07-28T23:54:41.977 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:54:41 GMT /hiiqos-34/ 2023-07-28T23:54:41.978 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:z+hnngNnP7xJD9TnqSKX5EwSxiM= 2023-07-28T23:54:41.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:54:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:z+hnngNnP7xJD9TnqSKX5EwSxiM=', 'Content-Length': '0'} 2023-07-28T23:54:41.981 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000054db053f9b00607d-0064c45541-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:54:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:41.982 DEBUG:boto:path=/hiiqos-34/key 2023-07-28T23:54:41.982 DEBUG:boto:auth_path=/hiiqos-34/key 2023-07-28T23:54:41.983 DEBUG:boto:Method: PUT 2023-07-28T23:54:41.983 DEBUG:boto:Path: /hiiqos-34/key 2023-07-28T23:54:41.983 DEBUG:boto:Data: 2023-07-28T23:54:41.984 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2023-07-28T23:54:41.984 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:54:41.984 DEBUG:boto:Port: 8001 2023-07-28T23:54:41.985 DEBUG:boto:Params: {} 2023-07-28T23:54:41.985 DEBUG:boto:Token: None 2023-07-28T23:54:41.985 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Fri, 28 Jul 2023 23:54:41 GMT /hiiqos-34/key 2023-07-28T23:54:41.986 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0S1aqCeCfMncI6uuL3t7i5LB20w= 2023-07-28T23:54:41.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:54:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0S1aqCeCfMncI6uuL3t7i5LB20w='} 2023-07-28T23:54:42.028 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000076aa793d09530498-0064c45541-1113-a2'), ('Date', 'Fri, 28 Jul 2023 23:54:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:54:42.029 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:54:42.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:54:42.030 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:54:42.077 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:54:42.078 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:54:42.106 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-28T23:54:42.118 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-28T23:54:42.118 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:42.119 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-28T23:54:42.119 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:46:09.233742Z" 2023-07-28T23:54:42.119 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:42.119 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:42.119 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588481.939221_339.1", 2023-07-28T23:54:42.119 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:54:41.939221Z" 2023-07-28T23:54:42.119 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:42.120 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:42.120 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:54:42.120 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:54:42.120 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:42.120 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:42.120 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:54:42.121 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:54:42.121 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:42.121 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-28T23:54:42.122 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690587969.233742_99.1', 1: '1_1690588481.939221_339.1', 2: '', 3: ''} 2023-07-28T23:54:42.122 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-28T23:54:42.123 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:54:42.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:54:42.124 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:54:42.170 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:54:42.170 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:54:42.197 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:54:42.195+0000 7f25d540f880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:54:42.198 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:54:42.196+0000 7f25d540f880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:54:42.210 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:54:42.210 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:54:42.210 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:54:42.210 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:54:42.210 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:54:42.211 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:54:42.211 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:54:42.211 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:42.211 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:54:42.211 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:42.211 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:54:42.211 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:54:42.212 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:54:42.212 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-28T23:54:42.212 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:54:42.212 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:54:42.212 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:54:42.212 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-28T23:54:42.213 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:54:42.213 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:42.213 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:42.213 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:42.213 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:54:42.213 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:54:42.214 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:54:42.214 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588481.911779_330.1", 2023-07-28T23:54:42.214 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:54:42.214 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:54:42.214 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:54:42.214 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:54:41.911779Z", 2023-07-28T23:54:42.215 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:54:42.215 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:42.215 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:42.215 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:42.215 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:54:42.215 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:54:42.215 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:54:42.216 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:54:42.216 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:54:42.216 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:54:42.216 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:54:42.216 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:54:42.216 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:54:42.217 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:42.217 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:42.217 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:42.217 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:54:42.217 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:54:42.217 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:54:42.217 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:54:42.218 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:54:42.218 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:54:42.218 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:54:42.218 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:54:42.218 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:54:42.218 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:42.218 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:42.219 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:54:42.219 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:42.219 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:54:42.219 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:54:42.219 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:54:42.219 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:42.219 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:54:42Z" 2023-07-28T23:54:42.219 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:54:42.220 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588481.911779_330.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:54:41.911779Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-28T23:54:42Z" } 2023-07-28T23:54:42.221 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588481.911779_330.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:54:41.911779Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-28T23:54:42.221 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588481.939221_339.1', 2: '', 3: ''} 2023-07-28T23:54:42.222 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588481.911779_330.1', 2: '', 3: ''} 2023-07-28T23:54:42.222 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588481.939221_339.1 target=1_1690588481.911779_330.1 2023-07-28T23:54:47.223 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:54:47.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:54:47.224 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:54:47.279 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:54:47.279 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:54:47.313 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:54:47.311+0000 7f5b5e936880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:54:47.314 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:54:47.312+0000 7f5b5e936880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:54:47.327 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:54:47.327 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:54:47.328 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:54:47.328 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:54:47.328 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:54:47.328 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:54:47.328 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:54:47.328 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:47.328 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:54:47.329 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:47.329 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:54:47.329 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:54:47.329 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:54:47.329 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-28T23:54:47.329 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:54:47.329 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:54:47.330 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:54:47.330 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-28T23:54:47.330 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:54:47.330 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:47.330 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:47.330 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:47.331 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:54:47.331 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:54:47.331 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:54:47.331 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588481.939221_339.1", 2023-07-28T23:54:47.331 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:54:47.331 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:54:47.331 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:54:47.332 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:54:41.939221Z", 2023-07-28T23:54:47.333 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:54:47.333 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:47.333 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:47.333 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:47.333 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:54:47.333 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:54:47.334 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:54:47.334 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:54:47.334 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:54:47.334 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:54:47.334 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:54:47.334 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:54:47.334 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:54:47.335 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:47.335 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:47.335 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:47.335 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:54:47.335 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:54:47.335 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:54:47.335 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:54:47.336 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:54:47.336 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:54:47.336 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:54:47.336 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:54:47.336 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:54:47.336 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:47.337 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:47.337 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:54:47.337 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:47.337 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:54:47.337 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:54:47.337 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:54:47.337 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:47.338 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:54:47Z" 2023-07-28T23:54:47.338 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:54:47.339 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588481.939221_339.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:54:41.939221Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-28T23:54:47Z" } 2023-07-28T23:54:47.339 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588481.939221_339.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:54:41.939221Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-28T23:54:47.340 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588481.939221_339.1', 2: '', 3: ''} 2023-07-28T23:54:47.340 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588481.939221_339.1', 2: '', 3: ''} 2023-07-28T23:54:47.340 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-28T23:54:47.341 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-07-28T23:54:47.341 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:54:47.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:54:47.342 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:54:47.387 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:54:47.388 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:54:47.435 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-28T23:54:47.436 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:47.436 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000003465", 2023-07-28T23:54:47.436 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:54:41.969483Z" 2023-07-28T23:54:47.436 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:47.436 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:47.437 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000002482", 2023-07-28T23:54:47.437 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:50:13.440177Z" 2023-07-28T23:54:47.437 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:47.437 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:47.437 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-07-28T23:54:47.437 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:54:41.804947Z" 2023-07-28T23:54:47.438 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:47.438 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:47.438 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:54:47.438 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:54:47.438 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:47.438 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-28T23:54:47.439 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000003465', 1: '00000000000000000000:00000000000000002482', 2: '00000000000000000000:00000000000000001068', 3: ''} 2023-07-28T23:54:47.440 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-07-28T23:54:47.440 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:54:47.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:54:47.441 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:54:47.487 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:54:47.488 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:54:47.527 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:54:47.528 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:54:47.528 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:54:47.528 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:54:47.528 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:54:47.528 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 16749483048653174838 2023-07-28T23:54:47.528 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:47.529 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:54:47.529 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:47.529 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:54:47.529 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:54:47.529 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-28T23:54:47.529 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000003324", 2023-07-28T23:54:47.530 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:54:47.530 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:54:47.530 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:54:47.530 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:54:14.538552Z" 2023-07-28T23:54:47.530 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:47.530 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:47.530 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:47.531 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:54:47.531 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:54:47.531 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-28T23:54:47.531 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000002482", 2023-07-28T23:54:47.531 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:54:47.531 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:54:47.532 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:54:47.532 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:50:13.440177Z" 2023-07-28T23:54:47.532 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:47.532 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:47.532 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:47.532 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:54:47.533 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:54:47.533 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-28T23:54:47.533 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-07-28T23:54:47.533 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:54:47.534 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:54:47.534 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:54:47.534 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:26.300717Z" 2023-07-28T23:54:47.534 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:47.534 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:47.534 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:47.535 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:54:47.535 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:54:47.535 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-28T23:54:47.535 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:54:47.535 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:54:47.535 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:54:47.535 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:54:47.536 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000" 2023-07-28T23:54:47.536 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:47.536 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:47.536 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:54:47.536 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:47.536 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:54:47.537 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 0, 2023-07-28T23:54:47.537 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 0 2023-07-28T23:54:47.537 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:47.537 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:54:47Z" 2023-07-28T23:54:47.537 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:54:47.538 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16749483048653174838 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003324", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:54:14.538552Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002482", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:50:13.440177Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000929", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:46:26.300717Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-07-28T23:54:47Z" } 2023-07-28T23:54:47.539 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003324', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:54:14.538552Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002482', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:50:13.440177Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000929', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:46:26.300717Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-07-28T23:54:47.539 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003465', 1: '00000000000000000000:00000000000000002482', 2: '00000000000000000000:00000000000000001068', 3: ''} 2023-07-28T23:54:47.539 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003324', 1: '00000000000000000000:00000000000000002482', 2: '00000000000000000000:00000000000000000929', 3: ''} 2023-07-28T23:54:47.540 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000003465 target=00000000000000000000:00000000000000003324, shard=2 master=00000000000000000000:00000000000000001068 target=00000000000000000000:00000000000000000929 2023-07-28T23:54:52.541 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:54:52.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:54:52.542 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:54:52.597 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:54:52.598 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:54:52.641 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:54:52.642 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:54:52.642 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:54:52.642 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:54:52.642 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:54:52.642 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 16749483048653174838 2023-07-28T23:54:52.643 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:52.643 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:54:52.643 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:52.643 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:54:52.643 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:54:52.644 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-28T23:54:52.644 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000003324", 2023-07-28T23:54:52.644 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:54:52.644 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:54:52.644 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:54:52.644 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:54:14.538552Z" 2023-07-28T23:54:52.645 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:52.645 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:52.645 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:52.645 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:54:52.645 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:54:52.645 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-28T23:54:52.645 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000002482", 2023-07-28T23:54:52.646 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:54:52.646 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:54:52.646 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:54:52.646 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:50:13.440177Z" 2023-07-28T23:54:52.646 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:52.646 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:52.646 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:52.647 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:54:52.647 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:54:52.647 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-28T23:54:52.647 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-07-28T23:54:52.647 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:54:52.647 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:54:52.648 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:54:52.648 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:26.300717Z" 2023-07-28T23:54:52.648 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:52.648 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:52.648 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:52.648 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:54:52.648 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:54:52.649 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-28T23:54:52.649 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:54:52.649 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:54:52.649 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:54:52.649 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:54:52.649 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000" 2023-07-28T23:54:52.649 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:52.650 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:52.650 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:54:52.650 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:52.650 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:54:52.650 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 0, 2023-07-28T23:54:52.650 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 0 2023-07-28T23:54:52.651 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:52.651 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:54:52Z" 2023-07-28T23:54:52.651 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:54:52.652 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16749483048653174838 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003324", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:54:14.538552Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002482", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:50:13.440177Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000929", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:46:26.300717Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-07-28T23:54:52Z" } 2023-07-28T23:54:52.652 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003324', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:54:14.538552Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002482', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:50:13.440177Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000929', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:46:26.300717Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-07-28T23:54:52.653 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003465', 1: '00000000000000000000:00000000000000002482', 2: '00000000000000000000:00000000000000001068', 3: ''} 2023-07-28T23:54:52.653 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003324', 1: '00000000000000000000:00000000000000002482', 2: '00000000000000000000:00000000000000000929', 3: ''} 2023-07-28T23:54:52.653 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000003465 target=00000000000000000000:00000000000000003324, shard=2 master=00000000000000000000:00000000000000001068 target=00000000000000000000:00000000000000000929 2023-07-28T23:54:57.655 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:54:57.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:54:57.656 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:54:57.711 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:54:57.711 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:54:57.755 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:54:57.756 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:54:57.756 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:54:57.757 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:54:57.757 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:54:57.757 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 16749483048653174838 2023-07-28T23:54:57.757 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:57.757 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:54:57.757 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:57.757 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:54:57.758 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:54:57.758 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-28T23:54:57.758 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000003606", 2023-07-28T23:54:57.758 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:54:57.758 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:54:57.758 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:54:57.759 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:54:54.539998Z" 2023-07-28T23:54:57.759 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:57.759 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:57.759 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:57.759 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:54:57.759 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:54:57.759 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-28T23:54:57.760 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000002482", 2023-07-28T23:54:57.760 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:54:57.760 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:54:57.760 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:54:57.760 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:50:13.440177Z" 2023-07-28T23:54:57.760 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:57.760 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:57.761 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:57.761 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:54:57.761 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:54:57.761 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-28T23:54:57.761 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-07-28T23:54:57.761 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:54:57.762 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:54:57.762 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:54:57.762 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:54:41.804947Z" 2023-07-28T23:54:57.762 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:57.762 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:57.762 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:57.763 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:54:57.763 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:54:57.763 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-28T23:54:57.763 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:54:57.763 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:54:57.763 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:54:57.764 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:54:57.764 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000" 2023-07-28T23:54:57.764 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:57.764 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:57.764 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:54:57.764 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:57.764 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:54:57.765 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 0, 2023-07-28T23:54:57.765 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 0 2023-07-28T23:54:57.765 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:57.765 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:54:57Z" 2023-07-28T23:54:57.765 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:54:57.766 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16749483048653174838 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003606", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:54:54.539998Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002482", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:50:13.440177Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001068", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:54:41.804947Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-07-28T23:54:57Z" } 2023-07-28T23:54:57.767 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003606', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:54:54.539998Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002482', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:50:13.440177Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001068', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:54:41.804947Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-07-28T23:54:57.767 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003465', 1: '00000000000000000000:00000000000000002482', 2: '00000000000000000000:00000000000000001068', 3: ''} 2023-07-28T23:54:57.767 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003606', 1: '00000000000000000000:00000000000000002482', 2: '00000000000000000000:00000000000000001068', 3: ''} 2023-07-28T23:54:57.768 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-07-28T23:54:57.768 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-07-28T23:54:57.769 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:54:57.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:54:57.770 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:54:57.815 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:54:57.816 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:54:57.860 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-28T23:54:57.860 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:57.860 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000004170", 2023-07-28T23:54:57.861 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:54:54.786672Z" 2023-07-28T23:54:57.861 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:57.861 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:57.861 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000001918", 2023-07-28T23:54:57.861 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:49:53.525225Z" 2023-07-28T23:54:57.861 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:57.862 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:57.862 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-07-28T23:54:57.862 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:54:53.493673Z" 2023-07-28T23:54:57.862 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:57.862 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:57.862 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:54:57.862 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:54:57.863 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:57.863 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-28T23:54:57.864 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000004170', 1: '00000000000000000000:00000000000000001918', 2: '00000000000000000000:00000000000000001210', 3: ''} 2023-07-28T23:54:57.864 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-07-28T23:54:57.865 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:54:57.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:54:57.866 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:54:57.913 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:54:57.914 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:54:57.953 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:54:57.953 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:54:57.953 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:54:57.953 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:54:57.954 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:54:57.954 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 17882949198279251152 2023-07-28T23:54:57.954 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:57.954 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:54:57.954 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:57.954 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:54:57.955 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:54:57.955 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-28T23:54:57.955 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000004029", 2023-07-28T23:54:57.955 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:54:57.955 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:54:57.955 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:54:57.955 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:54:41.990531Z" 2023-07-28T23:54:57.956 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:57.956 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:57.956 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:57.956 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:54:57.956 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:54:57.956 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-28T23:54:57.956 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000001918", 2023-07-28T23:54:57.957 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:54:57.957 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:54:57.957 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:54:57.957 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:49:53.525225Z" 2023-07-28T23:54:57.958 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:57.958 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:57.958 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:57.958 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:54:57.958 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:54:57.958 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-28T23:54:57.959 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-07-28T23:54:57.959 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:54:57.959 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:54:57.959 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:54:57.959 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:54:53.493673Z" 2023-07-28T23:54:57.960 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:57.960 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:57.960 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:54:57.960 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:54:57.960 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:54:57.960 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-28T23:54:57.961 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:54:57.961 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:54:57.961 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:54:57.961 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:54:57.961 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000" 2023-07-28T23:54:57.961 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:57.961 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:54:57.962 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:54:57.962 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:57.962 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:54:57.962 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 0, 2023-07-28T23:54:57.962 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 0 2023-07-28T23:54:57.962 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:54:57.962 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:54:57Z" 2023-07-28T23:54:57.963 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:54:57.964 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17882949198279251152 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004029", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:54:41.990531Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:49:53.525225Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001210", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:54:53.493673Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-07-28T23:54:57Z" } 2023-07-28T23:54:57.964 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004029', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:54:41.990531Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:49:53.525225Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:54:53.493673Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-07-28T23:54:57.965 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004170', 1: '00000000000000000000:00000000000000001918', 2: '00000000000000000000:00000000000000001210', 3: ''} 2023-07-28T23:54:57.965 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004029', 1: '00000000000000000000:00000000000000001918', 2: '00000000000000000000:00000000000000001210', 3: ''} 2023-07-28T23:54:57.965 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=0 master=00000000000000000000:00000000000000004170 target=00000000000000000000:00000000000000004029 2023-07-28T23:55:02.966 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:55:02.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:55:02.968 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:55:03.026 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:55:03.026 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:55:03.074 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:55:03.074 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:55:03.075 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:55:03.075 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:55:03.075 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:55:03.075 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 17882949198279251152 2023-07-28T23:55:03.075 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:03.075 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:55:03.075 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:55:03.076 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:55:03.076 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:55:03.076 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-28T23:55:03.076 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000004029", 2023-07-28T23:55:03.076 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:55:03.076 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:55:03.077 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:55:03.077 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:54:41.990531Z" 2023-07-28T23:55:03.077 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:55:03.077 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:03.077 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:55:03.078 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:55:03.078 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:55:03.078 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-28T23:55:03.078 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000001918", 2023-07-28T23:55:03.078 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:55:03.078 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:55:03.079 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:55:03.079 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:49:53.525225Z" 2023-07-28T23:55:03.079 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:55:03.079 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:03.079 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:55:03.079 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:55:03.079 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:55:03.080 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-28T23:55:03.080 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-07-28T23:55:03.080 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:55:03.080 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:55:03.080 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:55:03.080 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:54:53.493673Z" 2023-07-28T23:55:03.081 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:55:03.081 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:03.081 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:55:03.081 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:55:03.081 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:55:03.081 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-28T23:55:03.081 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:55:03.082 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:55:03.082 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:55:03.082 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:55:03.082 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000" 2023-07-28T23:55:03.082 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:55:03.082 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:55:03.082 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:55:03.083 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:03.083 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:55:03.083 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 0, 2023-07-28T23:55:03.083 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 0 2023-07-28T23:55:03.083 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:03.083 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:55:03Z" 2023-07-28T23:55:03.083 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:55:03.084 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17882949198279251152 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004029", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:54:41.990531Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:49:53.525225Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001210", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:54:53.493673Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-07-28T23:55:03Z" } 2023-07-28T23:55:03.085 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004029', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:54:41.990531Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:49:53.525225Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:54:53.493673Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-07-28T23:55:03.085 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004170', 1: '00000000000000000000:00000000000000001918', 2: '00000000000000000000:00000000000000001210', 3: ''} 2023-07-28T23:55:03.086 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004029', 1: '00000000000000000000:00000000000000001918', 2: '00000000000000000000:00000000000000001210', 3: ''} 2023-07-28T23:55:03.086 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=0 master=00000000000000000000:00000000000000004170 target=00000000000000000000:00000000000000004029 2023-07-28T23:55:08.087 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:55:08.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:55:08.088 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:55:08.144 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:55:08.144 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:55:08.189 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:55:08.190 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:55:08.190 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:55:08.190 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:55:08.190 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:55:08.191 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 17882949198279251152 2023-07-28T23:55:08.191 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:08.191 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:55:08.191 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:55:08.191 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:55:08.191 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:55:08.191 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-28T23:55:08.192 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000004029", 2023-07-28T23:55:08.192 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:55:08.192 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:55:08.192 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:55:08.192 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:54:41.990531Z" 2023-07-28T23:55:08.193 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:55:08.193 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:08.193 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:55:08.193 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:55:08.193 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:55:08.193 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-28T23:55:08.194 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000001918", 2023-07-28T23:55:08.194 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:55:08.194 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:55:08.194 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:55:08.195 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:49:53.525225Z" 2023-07-28T23:55:08.195 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:55:08.195 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:08.195 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:55:08.195 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:55:08.195 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:55:08.195 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-28T23:55:08.196 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-07-28T23:55:08.196 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:55:08.196 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:55:08.196 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:55:08.196 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:54:53.493673Z" 2023-07-28T23:55:08.197 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:55:08.197 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:08.197 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:55:08.197 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:55:08.197 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:55:08.197 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-28T23:55:08.198 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:55:08.198 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:55:08.198 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:55:08.198 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:55:08.198 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000" 2023-07-28T23:55:08.198 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:55:08.198 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:55:08.199 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:55:08.199 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:08.199 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:55:08.199 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 0, 2023-07-28T23:55:08.199 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 0 2023-07-28T23:55:08.199 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:08.200 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:55:08Z" 2023-07-28T23:55:08.200 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:55:08.201 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17882949198279251152 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004029", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:54:41.990531Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:49:53.525225Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001210", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:54:53.493673Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-07-28T23:55:08Z" } 2023-07-28T23:55:08.202 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004029', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:54:41.990531Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:49:53.525225Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:54:53.493673Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-07-28T23:55:08.202 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004170', 1: '00000000000000000000:00000000000000001918', 2: '00000000000000000000:00000000000000001210', 3: ''} 2023-07-28T23:55:08.202 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004029', 1: '00000000000000000000:00000000000000001918', 2: '00000000000000000000:00000000000000001210', 3: ''} 2023-07-28T23:55:08.203 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=0 master=00000000000000000000:00000000000000004170 target=00000000000000000000:00000000000000004029 2023-07-28T23:55:13.203 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:55:13.204 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:55:13.205 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:55:13.256 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:55:13.256 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:55:13.300 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:55:13.300 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:55:13.300 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:55:13.300 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:55:13.300 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:55:13.301 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 17882949198279251152 2023-07-28T23:55:13.301 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:13.301 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:55:13.301 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:55:13.301 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:55:13.301 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:55:13.302 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-28T23:55:13.302 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000004029", 2023-07-28T23:55:13.302 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:55:13.302 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:55:13.302 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:55:13.302 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:54:41.990531Z" 2023-07-28T23:55:13.302 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:55:13.303 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:13.303 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:55:13.303 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:55:13.303 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:55:13.303 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-28T23:55:13.303 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000001918", 2023-07-28T23:55:13.303 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:55:13.304 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:55:13.304 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:55:13.304 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:49:53.525225Z" 2023-07-28T23:55:13.304 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:55:13.304 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:13.304 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:55:13.305 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:55:13.305 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:55:13.305 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-28T23:55:13.305 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-07-28T23:55:13.305 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:55:13.305 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:55:13.305 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:55:13.306 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:54:53.493673Z" 2023-07-28T23:55:13.306 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:55:13.306 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:13.306 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:55:13.306 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:55:13.306 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:55:13.306 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-28T23:55:13.307 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:55:13.307 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:55:13.307 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:55:13.307 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:55:13.307 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000" 2023-07-28T23:55:13.307 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:55:13.307 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:55:13.308 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:55:13.308 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:13.308 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:55:13.308 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 0, 2023-07-28T23:55:13.308 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 0 2023-07-28T23:55:13.308 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:13.308 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:55:13Z" 2023-07-28T23:55:13.309 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:55:13.310 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17882949198279251152 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004029", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:54:41.990531Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:49:53.525225Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001210", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:54:53.493673Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-07-28T23:55:13Z" } 2023-07-28T23:55:13.310 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004029', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:54:41.990531Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:49:53.525225Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:54:53.493673Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-07-28T23:55:13.310 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004170', 1: '00000000000000000000:00000000000000001918', 2: '00000000000000000000:00000000000000001210', 3: ''} 2023-07-28T23:55:13.311 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004029', 1: '00000000000000000000:00000000000000001918', 2: '00000000000000000000:00000000000000001210', 3: ''} 2023-07-28T23:55:13.311 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=0 master=00000000000000000000:00000000000000004170 target=00000000000000000000:00000000000000004029 2023-07-28T23:55:18.312 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:55:18.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:55:18.313 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:55:18.368 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:55:18.368 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:55:18.413 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:55:18.413 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:55:18.414 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:55:18.414 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:55:18.414 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:55:18.414 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 17882949198279251152 2023-07-28T23:55:18.415 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:18.415 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:55:18.415 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:55:18.415 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:55:18.416 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:55:18.416 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-28T23:55:18.416 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000004170", 2023-07-28T23:55:18.417 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:55:18.417 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:55:18.417 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:55:18.417 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:54:54.786672Z" 2023-07-28T23:55:18.418 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:55:18.418 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:18.418 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:55:18.418 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:55:18.419 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:55:18.419 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-28T23:55:18.419 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000001918", 2023-07-28T23:55:18.420 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:55:18.420 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:55:18.420 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:55:18.421 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:49:53.525225Z" 2023-07-28T23:55:18.421 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:55:18.421 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:18.421 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:55:18.422 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:55:18.422 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:55:18.422 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-28T23:55:18.422 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-07-28T23:55:18.423 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:55:18.423 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:55:18.423 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:55:18.423 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:54:53.493673Z" 2023-07-28T23:55:18.423 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:55:18.423 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:18.424 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:55:18.424 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:55:18.424 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:55:18.424 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-28T23:55:18.424 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:55:18.424 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:55:18.425 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:55:18.425 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:55:18.425 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000" 2023-07-28T23:55:18.425 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:55:18.425 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:55:18.425 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:55:18.426 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:18.426 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:55:18.426 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 0, 2023-07-28T23:55:18.426 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 0 2023-07-28T23:55:18.426 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:18.426 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:55:18Z" 2023-07-28T23:55:18.426 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:55:18.427 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17882949198279251152 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004170", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:54:54.786672Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:49:53.525225Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001210", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:54:53.493673Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-07-28T23:55:18Z" } 2023-07-28T23:55:18.428 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004170', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:54:54.786672Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:49:53.525225Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:54:53.493673Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-07-28T23:55:18.428 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004170', 1: '00000000000000000000:00000000000000001918', 2: '00000000000000000000:00000000000000001210', 3: ''} 2023-07-28T23:55:18.429 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004170', 1: '00000000000000000000:00000000000000001918', 2: '00000000000000000000:00000000000000001210', 3: ''} 2023-07-28T23:55:18.429 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-07-28T23:55:18.429 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:55:18.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:55:18.430 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:55:18.474 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:55:18.474 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:55:18.519 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-28T23:55:18.519 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:55:18.520 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000003606", 2023-07-28T23:55:18.520 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:54:54.539998Z" 2023-07-28T23:55:18.520 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:18.520 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:55:18.520 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000002482", 2023-07-28T23:55:18.521 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:50:13.440177Z" 2023-07-28T23:55:18.521 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:18.521 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:55:18.521 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-07-28T23:55:18.521 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:54:53.570640Z" 2023-07-28T23:55:18.521 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:18.521 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:55:18.522 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:55:18.522 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:55:18.522 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:55:18.522 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-28T23:55:18.523 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:55:18.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:55:18.524 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-28T23:55:18.604 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:55:18.604 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:55:18.653 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:55:18.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:55:18.655 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 0 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:55:18.739 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:55:18.740 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:55:18.781 INFO:teuthology.orchestra.run.smithi101.stdout:[] 2023-07-28T23:55:18.783 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:55:18.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:55:18.785 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 1 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:55:18.831 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:55:18.832 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:55:18.872 INFO:teuthology.orchestra.run.smithi101.stdout:[] 2023-07-28T23:55:18.873 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:55:18.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:55:18.874 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 2 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:55:18.921 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:55:18.922 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:55:18.961 INFO:teuthology.orchestra.run.smithi101.stdout:[] 2023-07-28T23:55:18.963 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:55:18.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:55:18.964 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:55:19.007 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:55:19.008 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:55:19.052 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-28T23:55:19.053 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:55:19.053 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000003606", 2023-07-28T23:55:19.053 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:54:54.539998Z" 2023-07-28T23:55:19.053 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:19.053 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:55:19.053 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000002482", 2023-07-28T23:55:19.054 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:50:13.440177Z" 2023-07-28T23:55:19.054 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:19.054 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:55:19.054 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-07-28T23:55:19.054 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:54:53.570640Z" 2023-07-28T23:55:19.054 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:19.055 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:55:19.055 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:55:19.055 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:55:19.055 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:55:19.055 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-28T23:55:19.056 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:55:19.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:55:19.057 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-28T23:55:19.137 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:55:19.138 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:55:19.185 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:55:19.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:55:19.186 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 0 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:55:19.272 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:55:19.272 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:55:19.312 INFO:teuthology.orchestra.run.smithi101.stdout:[] 2023-07-28T23:55:19.313 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:55:19.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:55:19.315 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 1 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:55:19.361 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:55:19.362 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:55:19.402 INFO:teuthology.orchestra.run.smithi101.stdout:[] 2023-07-28T23:55:19.404 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:55:19.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:55:19.405 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 2 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:55:19.450 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:55:19.450 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:55:19.489 INFO:teuthology.orchestra.run.smithi101.stdout:[] 2023-07-28T23:55:19.491 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_datalog_autotrim ... ok 2023-07-28T23:55:19.492 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:55:19.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:55:19.493 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --sync-from-all=false --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-28T23:55:19.537 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:55:19.537 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:55:19.556 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:55:19.557 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-28T23:55:19.557 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a2", 2023-07-28T23:55:19.557 INFO:teuthology.orchestra.run.smithi101.stdout: "domain_root": "a2.rgw.meta:root", 2023-07-28T23:55:19.557 INFO:teuthology.orchestra.run.smithi101.stdout: "control_pool": "a2.rgw.control", 2023-07-28T23:55:19.557 INFO:teuthology.orchestra.run.smithi101.stdout: "gc_pool": "a2.rgw.log:gc", 2023-07-28T23:55:19.558 INFO:teuthology.orchestra.run.smithi101.stdout: "lc_pool": "a2.rgw.log:lc", 2023-07-28T23:55:19.558 INFO:teuthology.orchestra.run.smithi101.stdout: "log_pool": "a2.rgw.log", 2023-07-28T23:55:19.558 INFO:teuthology.orchestra.run.smithi101.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-07-28T23:55:19.558 INFO:teuthology.orchestra.run.smithi101.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-07-28T23:55:19.558 INFO:teuthology.orchestra.run.smithi101.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-07-28T23:55:19.558 INFO:teuthology.orchestra.run.smithi101.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-07-28T23:55:19.558 INFO:teuthology.orchestra.run.smithi101.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-07-28T23:55:19.559 INFO:teuthology.orchestra.run.smithi101.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-07-28T23:55:19.559 INFO:teuthology.orchestra.run.smithi101.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-07-28T23:55:19.559 INFO:teuthology.orchestra.run.smithi101.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-07-28T23:55:19.559 INFO:teuthology.orchestra.run.smithi101.stdout: "otp_pool": "a2.rgw.otp", 2023-07-28T23:55:19.559 INFO:teuthology.orchestra.run.smithi101.stdout: "system_key": { 2023-07-28T23:55:19.559 INFO:teuthology.orchestra.run.smithi101.stdout: "access_key": "RX82IHH7C5RUJ54D", 2023-07-28T23:55:19.559 INFO:teuthology.orchestra.run.smithi101.stdout: "secret_key": "InTLvCQsePIbOVhlgLo1AkTNflWbmSsQ" 2023-07-28T23:55:19.560 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:19.560 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_pools": [ 2023-07-28T23:55:19.560 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:55:19.560 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "default-placement", 2023-07-28T23:55:19.560 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:55:19.560 INFO:teuthology.orchestra.run.smithi101.stdout: "index_pool": "a2.rgw.buckets.index", 2023-07-28T23:55:19.560 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": { 2023-07-28T23:55:19.561 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD": { 2023-07-28T23:55:19.561 INFO:teuthology.orchestra.run.smithi101.stdout: "data_pool": "a2.rgw.buckets.data", 2023-07-28T23:55:19.561 INFO:teuthology.orchestra.run.smithi101.stdout: "compression_type": "random" 2023-07-28T23:55:19.561 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:55:19.561 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:19.561 INFO:teuthology.orchestra.run.smithi101.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-07-28T23:55:19.561 INFO:teuthology.orchestra.run.smithi101.stdout: "index_type": 0, 2023-07-28T23:55:19.562 INFO:teuthology.orchestra.run.smithi101.stdout: "inline_data": true 2023-07-28T23:55:19.562 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:55:19.562 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:55:19.562 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:55:19.562 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-28T23:55:19.562 INFO:teuthology.orchestra.run.smithi101.stdout: "notif_pool": "a2.rgw.log:notif" 2023-07-28T23:55:19.562 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:55:19.564 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:55:19.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:55:19.565 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-28T23:55:19.611 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:55:19.611 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:55:19.650 INFO:teuthology.orchestra.run.smithi101.stderr:Sending period to new master zone 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433 2023-07-28T23:55:19.678 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:55:19.679 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:55:19.679 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 5, 2023-07-28T23:55:19.680 INFO:teuthology.orchestra.run.smithi101.stdout: "predecessor_uuid": "a0987484-ce37-4520-a564-54b7ea8c5251", 2023-07-28T23:55:19.680 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": [], 2023-07-28T23:55:19.680 INFO:teuthology.orchestra.run.smithi101.stdout: "period_map": { 2023-07-28T23:55:19.680 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:55:19.680 INFO:teuthology.orchestra.run.smithi101.stdout: "zonegroups": [ 2023-07-28T23:55:19.681 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:55:19.681 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "a", 2023-07-28T23:55:19.681 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a", 2023-07-28T23:55:19.681 INFO:teuthology.orchestra.run.smithi101.stdout: "api_name": "", 2023-07-28T23:55:19.681 INFO:teuthology.orchestra.run.smithi101.stdout: "is_master": true, 2023-07-28T23:55:19.681 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-28T23:55:19.681 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-28T23:55:19.682 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:55:19.682 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames": [], 2023-07-28T23:55:19.682 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames_s3website": [], 2023-07-28T23:55:19.682 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-28T23:55:19.682 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-28T23:55:19.682 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:55:19.682 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-28T23:55:19.683 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a1", 2023-07-28T23:55:19.683 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-28T23:55:19.683 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-28T23:55:19.683 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:55:19.683 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-28T23:55:19.683 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-28T23:55:19.683 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-28T23:55:19.684 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-28T23:55:19.684 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-28T23:55:19.684 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-28T23:55:19.684 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-28T23:55:19.684 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-28T23:55:19.684 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-28T23:55:19.685 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-28T23:55:19.685 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-28T23:55:19.685 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:55:19.685 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:19.685 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:55:19.685 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-28T23:55:19.685 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a2", 2023-07-28T23:55:19.686 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-28T23:55:19.686 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8001" 2023-07-28T23:55:19.686 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:55:19.686 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-28T23:55:19.686 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-28T23:55:19.686 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-28T23:55:19.686 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-28T23:55:19.687 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-28T23:55:19.687 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": false, 2023-07-28T23:55:19.687 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-28T23:55:19.688 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-28T23:55:19.688 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-28T23:55:19.688 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-28T23:55:19.688 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-28T23:55:19.688 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:55:19.689 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:55:19.689 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:55:19.689 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_targets": [ 2023-07-28T23:55:19.689 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:55:19.689 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default-placement", 2023-07-28T23:55:19.689 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [], 2023-07-28T23:55:19.690 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": [ 2023-07-28T23:55:19.690 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD" 2023-07-28T23:55:19.690 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:55:19.690 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:55:19.690 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:55:19.690 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "default-placement", 2023-07-28T23:55:19.690 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-28T23:55:19.691 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_policy": { 2023-07-28T23:55:19.691 INFO:teuthology.orchestra.run.smithi101.stdout: "groups": [] 2023-07-28T23:55:19.691 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:19.691 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled_features": [ 2023-07-28T23:55:19.691 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-28T23:55:19.691 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:55:19.691 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:19.692 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:55:19.692 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "b", 2023-07-28T23:55:19.692 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b", 2023-07-28T23:55:19.692 INFO:teuthology.orchestra.run.smithi101.stdout: "api_name": "", 2023-07-28T23:55:19.692 INFO:teuthology.orchestra.run.smithi101.stdout: "is_master": false, 2023-07-28T23:55:19.692 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-28T23:55:19.693 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-28T23:55:19.693 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:55:19.693 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames": [], 2023-07-28T23:55:19.693 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames_s3website": [], 2023-07-28T23:55:19.693 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-28T23:55:19.693 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-28T23:55:19.693 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:55:19.694 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-28T23:55:19.694 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b1", 2023-07-28T23:55:19.694 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-28T23:55:19.694 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-28T23:55:19.694 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:55:19.694 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-28T23:55:19.694 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-28T23:55:19.695 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-28T23:55:19.695 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-28T23:55:19.695 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-28T23:55:19.695 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-28T23:55:19.695 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-28T23:55:19.695 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-28T23:55:19.696 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-28T23:55:19.696 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-28T23:55:19.697 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-28T23:55:19.697 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:55:19.697 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:19.697 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:55:19.697 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-28T23:55:19.697 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b2", 2023-07-28T23:55:19.698 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-28T23:55:19.698 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8001" 2023-07-28T23:55:19.698 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:55:19.698 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-28T23:55:19.698 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-28T23:55:19.699 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-28T23:55:19.699 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-28T23:55:19.699 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-28T23:55:19.699 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-28T23:55:19.699 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-28T23:55:19.699 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-28T23:55:19.700 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-28T23:55:19.700 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-28T23:55:19.700 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-28T23:55:19.700 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:55:19.700 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:55:19.700 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:55:19.700 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_targets": [ 2023-07-28T23:55:19.701 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:55:19.701 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default-placement", 2023-07-28T23:55:19.701 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [], 2023-07-28T23:55:19.701 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": [ 2023-07-28T23:55:19.701 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD" 2023-07-28T23:55:19.701 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:55:19.701 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:55:19.702 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:55:19.702 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "default-placement", 2023-07-28T23:55:19.702 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-28T23:55:19.702 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_policy": { 2023-07-28T23:55:19.702 INFO:teuthology.orchestra.run.smithi101.stdout: "groups": [] 2023-07-28T23:55:19.702 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:19.702 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled_features": [ 2023-07-28T23:55:19.703 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-28T23:55:19.703 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:55:19.703 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:55:19.703 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:55:19.703 INFO:teuthology.orchestra.run.smithi101.stdout: "short_zone_ids": [ 2023-07-28T23:55:19.703 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:55:19.703 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-28T23:55:19.704 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 2896287231 2023-07-28T23:55:19.704 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:19.704 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:55:19.704 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-28T23:55:19.704 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 815606810 2023-07-28T23:55:19.704 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:19.705 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:55:19.705 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-28T23:55:19.705 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 2009171363 2023-07-28T23:55:19.705 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:19.705 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:55:19.705 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-28T23:55:19.705 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 3105190295 2023-07-28T23:55:19.706 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:55:19.706 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:55:19.706 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:19.706 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zonegroup": "a", 2023-07-28T23:55:19.706 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-28T23:55:19.706 INFO:teuthology.orchestra.run.smithi101.stdout: "period_config": { 2023-07-28T23:55:19.706 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_quota": { 2023-07-28T23:55:19.707 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-28T23:55:19.707 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-28T23:55:19.707 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-28T23:55:19.707 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-28T23:55:19.707 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-28T23:55:19.707 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:19.707 INFO:teuthology.orchestra.run.smithi101.stdout: "user_quota": { 2023-07-28T23:55:19.708 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-28T23:55:19.708 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-28T23:55:19.708 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-28T23:55:19.708 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-28T23:55:19.708 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-28T23:55:19.709 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:19.709 INFO:teuthology.orchestra.run.smithi101.stdout: "user_ratelimit": { 2023-07-28T23:55:19.709 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-28T23:55:19.709 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-28T23:55:19.710 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-28T23:55:19.710 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-28T23:55:19.710 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-28T23:55:19.710 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:19.710 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_ratelimit": { 2023-07-28T23:55:19.711 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-28T23:55:19.711 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-28T23:55:19.711 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-28T23:55:19.711 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-28T23:55:19.711 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-28T23:55:19.711 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:19.712 INFO:teuthology.orchestra.run.smithi101.stdout: "anonymous_ratelimit": { 2023-07-28T23:55:19.712 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-28T23:55:19.712 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-28T23:55:19.712 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-28T23:55:19.712 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-28T23:55:19.712 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-28T23:55:19.712 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:55:19.713 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:55:19.713 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-28T23:55:19.713 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_name": "test-realm", 2023-07-28T23:55:19.713 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:55:19.713 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:55:19.714 INFO:rgw_multi.tests:Set sync_from_all flag on zone a2 to false 2023-07-28T23:56:19.715 INFO:rgw_multi.tests:create bucket zone=a1 name=hiiqos-35 2023-07-28T23:56:19.716 DEBUG:boto:path=/hiiqos-35/ 2023-07-28T23:56:19.716 DEBUG:boto:auth_path=/hiiqos-35/ 2023-07-28T23:56:19.716 DEBUG:boto:Method: PUT 2023-07-28T23:56:19.717 DEBUG:boto:Path: /hiiqos-35/ 2023-07-28T23:56:19.717 DEBUG:boto:Data: 2023-07-28T23:56:19.717 DEBUG:boto:Headers: {} 2023-07-28T23:56:19.718 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:56:19.718 DEBUG:boto:Port: 8000 2023-07-28T23:56:19.718 DEBUG:boto:Params: {} 2023-07-28T23:56:19.719 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-28T23:56:19.719 DEBUG:boto:Token: None 2023-07-28T23:56:19.720 DEBUG:boto:StringToSign: PUT Fri, 28 Jul 2023 23:56:19 GMT /hiiqos-35/ 2023-07-28T23:56:19.720 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KrWYK6Zy9ElX1CpL3sOqdruHdOk= 2023-07-28T23:56:19.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:56:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KrWYK6Zy9ElX1CpL3sOqdruHdOk=', 'Content-Length': '0'} 2023-07-28T23:56:19.743 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000870b0a21af62683e-0064c455a3-1515-a1'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:56:19.744 DEBUG:boto:path=/hiiqos-35/testredirect 2023-07-28T23:56:19.745 DEBUG:boto:auth_path=/hiiqos-35/testredirect 2023-07-28T23:56:19.745 DEBUG:boto:Method: PUT 2023-07-28T23:56:19.745 DEBUG:boto:Path: /hiiqos-35/testredirect 2023-07-28T23:56:19.746 DEBUG:boto:Data: 2023-07-28T23:56:19.746 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-07-28T23:56:19.746 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:56:19.747 DEBUG:boto:Port: 8000 2023-07-28T23:56:19.747 DEBUG:boto:Params: {} 2023-07-28T23:56:19.747 DEBUG:boto:Token: None 2023-07-28T23:56:19.748 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Fri, 28 Jul 2023 23:56:19 GMT /hiiqos-35/testredirect 2023-07-28T23:56:19.748 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Vngqgtc6rwT4YsRMPVAZ9gpC0mE= 2023-07-28T23:56:19.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:56:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Vngqgtc6rwT4YsRMPVAZ9gpC0mE='} 2023-07-28T23:56:19.757 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005030f899bd5fc1b8-0064c455a3-1515-a1'), ('Date', 'Fri, 28 Jul 2023 23:56:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:56:19.758 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:56:19.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:56:19.758 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:56:19.809 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:56:19.810 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:56:19.841 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-28T23:56:19.854 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-28T23:56:19.855 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:56:19.855 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-28T23:56:19.855 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:46:09.233742Z" 2023-07-28T23:56:19.855 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:19.855 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:56:19.856 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588579.740560_402.1", 2023-07-28T23:56:19.856 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:56:19.740560Z" 2023-07-28T23:56:19.856 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:19.856 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:56:19.856 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:56:19.856 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:56:19.856 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:19.857 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:56:19.857 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:56:19.857 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-28T23:56:19.857 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:56:19.857 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-28T23:56:19.858 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690587969.233742_99.1', 1: '1_1690588579.740560_402.1', 2: '', 3: ''} 2023-07-28T23:56:19.859 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-28T23:56:19.859 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:56:19.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:56:19.860 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:56:19.905 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:56:19.905 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:56:19.935 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:56:19.933+0000 7f5c5c528880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:56:19.936 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:56:19.934+0000 7f5c5c528880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:56:19.947 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:56:19.948 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:56:19.948 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:56:19.948 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:56:19.948 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:56:19.948 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:56:19.949 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:56:19.949 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:19.949 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:56:19.949 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:56:19.949 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:56:19.949 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:56:19.949 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:56:19.950 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-28T23:56:19.950 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:56:19.950 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:56:19.950 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:56:19.950 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-28T23:56:19.950 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:56:19.951 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:56:19.951 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:19.951 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:56:19.951 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:56:19.951 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:56:19.951 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:56:19.951 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588481.939221_339.1", 2023-07-28T23:56:19.952 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:56:19.952 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:56:19.952 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:56:19.952 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:54:41.939221Z", 2023-07-28T23:56:19.952 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:56:19.952 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:56:19.952 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:19.953 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:56:19.953 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:56:19.953 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:56:19.953 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:56:19.953 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:56:19.953 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:56:19.953 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:56:19.954 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:56:19.954 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:56:19.954 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:56:19.954 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:56:19.954 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:19.954 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:56:19.954 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:56:19.955 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:56:19.955 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:56:19.955 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:56:19.955 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:56:19.955 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:56:19.955 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:56:19.955 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:56:19.956 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:56:19.956 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:56:19.956 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:56:19.956 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:56:19.956 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:19.956 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:56:19.956 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:56:19.957 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:56:19.957 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:19.957 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:56:19Z" 2023-07-28T23:56:19.957 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:56:19.958 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588481.939221_339.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:54:41.939221Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-28T23:56:19Z" } 2023-07-28T23:56:19.958 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588481.939221_339.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:54:41.939221Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-28T23:56:19.959 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588579.740560_402.1', 2: '', 3: ''} 2023-07-28T23:56:19.959 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588481.939221_339.1', 2: '', 3: ''} 2023-07-28T23:56:19.959 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588579.740560_402.1 target=1_1690588481.939221_339.1 2023-07-28T23:56:24.961 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:56:24.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:56:24.962 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:56:25.015 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:56:25.015 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:56:25.046 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:56:25.043+0000 7f90ab696880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:56:25.046 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:56:25.044+0000 7f90ab696880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:56:25.059 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:56:25.059 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:56:25.060 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:56:25.060 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:56:25.060 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:56:25.060 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:56:25.060 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:56:25.060 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:25.061 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:56:25.061 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:56:25.061 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:56:25.061 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:56:25.061 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:56:25.061 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-28T23:56:25.061 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:56:25.062 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:56:25.062 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:56:25.062 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-28T23:56:25.062 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:56:25.062 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:56:25.062 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:25.062 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:56:25.063 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:56:25.063 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:56:25.063 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:56:25.063 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588481.939221_339.1", 2023-07-28T23:56:25.063 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:56:25.063 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:56:25.064 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:56:25.064 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:54:41.939221Z", 2023-07-28T23:56:25.064 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:56:25.064 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:56:25.064 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:25.064 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:56:25.064 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:56:25.065 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:56:25.065 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:56:25.065 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:56:25.065 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:56:25.065 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:56:25.065 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:56:25.065 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:56:25.066 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:56:25.066 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:56:25.066 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:25.066 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:56:25.066 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:56:25.067 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:56:25.067 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:56:25.067 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:56:25.067 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:56:25.067 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:56:25.067 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:56:25.068 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:56:25.068 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:56:25.068 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:56:25.068 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:56:25.068 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:56:25.068 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:25.068 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:56:25.069 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:56:25.069 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:56:25.069 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:25.069 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:56:25Z" 2023-07-28T23:56:25.069 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:56:25.070 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588481.939221_339.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:54:41.939221Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-28T23:56:25Z" } 2023-07-28T23:56:25.071 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588481.939221_339.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:54:41.939221Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-28T23:56:25.071 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588579.740560_402.1', 2: '', 3: ''} 2023-07-28T23:56:25.071 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588481.939221_339.1', 2: '', 3: ''} 2023-07-28T23:56:25.072 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588579.740560_402.1 target=1_1690588481.939221_339.1 2023-07-28T23:56:30.073 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:56:30.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:56:30.074 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:56:30.127 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:56:30.128 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:56:30.162 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:56:30.159+0000 7fc97c124880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:56:30.162 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:56:30.160+0000 7fc97c124880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:56:30.175 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:56:30.176 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:56:30.176 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:56:30.176 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:56:30.176 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:56:30.176 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:56:30.177 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:56:30.177 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:30.177 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:56:30.177 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:56:30.177 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:56:30.177 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:56:30.177 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:56:30.178 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-28T23:56:30.178 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:56:30.178 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:56:30.178 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:56:30.178 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-28T23:56:30.178 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:56:30.178 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:56:30.179 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:30.179 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:56:30.179 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:56:30.179 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:56:30.179 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:56:30.179 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588481.939221_339.1", 2023-07-28T23:56:30.180 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:56:30.180 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:56:30.180 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:56:30.180 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:54:41.939221Z", 2023-07-28T23:56:30.180 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:56:30.180 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:56:30.181 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:30.181 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:56:30.181 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:56:30.181 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:56:30.181 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:56:30.181 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:56:30.181 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:56:30.182 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:56:30.182 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:56:30.182 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:56:30.182 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:56:30.182 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:56:30.182 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:30.183 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:56:30.183 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:56:30.183 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:56:30.183 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:56:30.183 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:56:30.183 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:56:30.183 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:56:30.184 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:56:30.184 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:56:30.184 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:56:30.184 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:56:30.184 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:56:30.184 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:56:30.184 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:30.185 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:56:30.185 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:56:30.185 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:56:30.185 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:30.185 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:56:30Z" 2023-07-28T23:56:30.185 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:56:30.186 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588481.939221_339.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:54:41.939221Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-28T23:56:30Z" } 2023-07-28T23:56:30.187 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588481.939221_339.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:54:41.939221Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-28T23:56:30.187 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588579.740560_402.1', 2: '', 3: ''} 2023-07-28T23:56:30.187 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588481.939221_339.1', 2: '', 3: ''} 2023-07-28T23:56:30.188 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588579.740560_402.1 target=1_1690588481.939221_339.1 2023-07-28T23:56:35.189 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:56:35.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:56:35.190 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:56:35.245 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:56:35.245 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:56:35.277 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:56:35.275+0000 7f530cd26880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:56:35.278 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:56:35.276+0000 7f530cd26880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:56:35.291 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:56:35.291 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:56:35.291 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:56:35.291 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:56:35.291 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:56:35.292 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:56:35.292 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:56:35.292 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:35.292 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:56:35.292 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:56:35.292 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:56:35.293 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:56:35.293 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:56:35.293 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-28T23:56:35.293 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:56:35.293 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:56:35.293 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:56:35.294 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-28T23:56:35.294 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:56:35.294 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:56:35.294 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:35.294 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:56:35.294 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:56:35.295 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:56:35.295 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:56:35.295 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588481.939221_339.1", 2023-07-28T23:56:35.295 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:56:35.295 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:56:35.295 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:56:35.296 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:54:41.939221Z", 2023-07-28T23:56:35.296 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:56:35.296 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:56:35.296 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:35.296 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:56:35.297 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:56:35.297 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:56:35.297 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:56:35.297 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:56:35.297 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:56:35.297 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:56:35.297 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:56:35.298 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:56:35.298 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:56:35.298 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:56:35.298 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:35.298 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:56:35.298 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:56:35.299 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:56:35.299 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:56:35.299 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:56:35.299 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:56:35.299 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:56:35.299 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:56:35.299 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:56:35.300 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:56:35.300 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:56:35.300 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:56:35.300 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:56:35.300 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:35.301 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:56:35.301 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:56:35.301 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:56:35.301 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:35.301 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:56:35Z" 2023-07-28T23:56:35.301 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:56:35.302 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588481.939221_339.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:54:41.939221Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-28T23:56:35Z" } 2023-07-28T23:56:35.303 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588481.939221_339.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:54:41.939221Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-28T23:56:35.303 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588579.740560_402.1', 2: '', 3: ''} 2023-07-28T23:56:35.303 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588481.939221_339.1', 2: '', 3: ''} 2023-07-28T23:56:35.304 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588579.740560_402.1 target=1_1690588481.939221_339.1 2023-07-28T23:56:40.305 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:56:40.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:56:40.306 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:56:40.359 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:56:40.360 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:56:40.391 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:56:40.389+0000 7f73dcb35880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:56:40.392 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:56:40.390+0000 7f73dcb35880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:56:40.405 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:56:40.405 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:56:40.405 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:56:40.405 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:56:40.406 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:56:40.406 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:56:40.406 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:56:40.406 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:40.406 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:56:40.406 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:56:40.406 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:56:40.407 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:56:40.407 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:56:40.407 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-28T23:56:40.407 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:56:40.407 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:56:40.407 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:56:40.407 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-28T23:56:40.408 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:56:40.408 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:56:40.408 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:40.408 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:56:40.408 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:56:40.408 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:56:40.409 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:56:40.409 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588481.939221_339.1", 2023-07-28T23:56:40.409 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:56:40.409 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:56:40.409 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:56:40.409 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:54:41.939221Z", 2023-07-28T23:56:40.409 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:56:40.410 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:56:40.410 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:40.410 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:56:40.410 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:56:40.410 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:56:40.410 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:56:40.410 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:56:40.411 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:56:40.411 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:56:40.411 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:56:40.411 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:56:40.411 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:56:40.411 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:56:40.411 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:40.412 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:56:40.412 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:56:40.412 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:56:40.412 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:56:40.412 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:56:40.412 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:56:40.412 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:56:40.413 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:56:40.413 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:56:40.413 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:56:40.413 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:56:40.413 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:56:40.413 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:56:40.413 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:40.414 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:56:40.414 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:56:40.414 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:56:40.414 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:40.414 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:56:40Z" 2023-07-28T23:56:40.414 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:56:40.415 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588481.939221_339.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:54:41.939221Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-28T23:56:40Z" } 2023-07-28T23:56:40.416 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588481.939221_339.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:54:41.939221Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-28T23:56:40.416 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588579.740560_402.1', 2: '', 3: ''} 2023-07-28T23:56:40.416 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588481.939221_339.1', 2: '', 3: ''} 2023-07-28T23:56:40.417 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588579.740560_402.1 target=1_1690588481.939221_339.1 2023-07-28T23:56:45.418 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:56:45.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:56:45.419 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:56:45.472 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:56:45.472 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:56:45.503 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:56:45.501+0000 7f677322d880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:56:45.504 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:56:45.502+0000 7f677322d880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:56:45.516 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:56:45.517 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:56:45.517 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:56:45.517 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:56:45.517 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:56:45.517 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:56:45.518 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:56:45.518 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:45.518 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:56:45.518 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:56:45.518 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:56:45.518 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:56:45.518 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:56:45.519 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-28T23:56:45.519 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:56:45.519 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:56:45.519 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:56:45.519 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-28T23:56:45.519 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:56:45.519 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:56:45.520 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:45.520 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:56:45.520 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:56:45.520 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:56:45.520 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:56:45.520 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588481.939221_339.1", 2023-07-28T23:56:45.521 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:56:45.521 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:56:45.521 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:56:45.521 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:54:41.939221Z", 2023-07-28T23:56:45.521 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:56:45.521 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:56:45.521 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:45.522 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:56:45.522 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:56:45.522 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:56:45.522 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:56:45.522 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:56:45.522 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:56:45.522 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:56:45.523 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:56:45.523 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:56:45.523 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:56:45.523 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:56:45.523 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:45.523 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:56:45.523 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:56:45.524 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:56:45.524 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:56:45.524 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:56:45.524 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:56:45.524 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:56:45.524 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:56:45.524 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:56:45.525 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:56:45.525 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:56:45.525 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:56:45.525 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:56:45.525 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:45.525 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:56:45.525 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:56:45.526 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:56:45.526 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:45.526 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:56:45Z" 2023-07-28T23:56:45.526 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:56:45.527 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588481.939221_339.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:54:41.939221Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-28T23:56:45Z" } 2023-07-28T23:56:45.527 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588481.939221_339.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:54:41.939221Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-28T23:56:45.528 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588579.740560_402.1', 2: '', 3: ''} 2023-07-28T23:56:45.528 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588481.939221_339.1', 2: '', 3: ''} 2023-07-28T23:56:45.528 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588579.740560_402.1 target=1_1690588481.939221_339.1 2023-07-28T23:56:50.530 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:56:50.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:56:50.531 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:56:50.584 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:56:50.584 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:56:50.615 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:56:50.613+0000 7ff60cb22880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:56:50.616 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:56:50.614+0000 7ff60cb22880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:56:50.629 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:56:50.629 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:56:50.629 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:56:50.629 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:56:50.629 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:56:50.630 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:56:50.630 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:56:50.630 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:50.630 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:56:50.630 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:56:50.630 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:56:50.630 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:56:50.631 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:56:50.631 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-28T23:56:50.631 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:56:50.631 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:56:50.631 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:56:50.631 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-28T23:56:50.631 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:56:50.632 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:56:50.632 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:50.632 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:56:50.632 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:56:50.632 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:56:50.632 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:56:50.633 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588481.939221_339.1", 2023-07-28T23:56:50.633 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:56:50.633 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:56:50.633 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:56:50.633 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:54:41.939221Z", 2023-07-28T23:56:50.633 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:56:50.633 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:56:50.634 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:50.634 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:56:50.634 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:56:50.634 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:56:50.634 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:56:50.634 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:56:50.634 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:56:50.635 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:56:50.635 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:56:50.635 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:56:50.635 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:56:50.635 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:56:50.635 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:50.635 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:56:50.636 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:56:50.636 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:56:50.636 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:56:50.636 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:56:50.636 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:56:50.636 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:56:50.636 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:56:50.637 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:56:50.637 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:56:50.637 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:56:50.637 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:56:50.637 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:56:50.637 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:50.637 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:56:50.638 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:56:50.638 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:56:50.638 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:50.638 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:56:50Z" 2023-07-28T23:56:50.638 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:56:50.639 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588481.939221_339.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:54:41.939221Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-28T23:56:50Z" } 2023-07-28T23:56:50.639 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588481.939221_339.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:54:41.939221Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-28T23:56:50.640 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588579.740560_402.1', 2: '', 3: ''} 2023-07-28T23:56:50.640 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588481.939221_339.1', 2: '', 3: ''} 2023-07-28T23:56:50.641 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588579.740560_402.1 target=1_1690588481.939221_339.1 2023-07-28T23:56:55.642 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:56:55.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:56:55.643 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:56:55.696 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:56:55.697 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:56:55.728 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:56:55.725+0000 7f799a326880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:56:55.728 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:56:55.726+0000 7f799a326880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:56:55.742 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:56:55.742 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:56:55.743 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:56:55.743 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:56:55.743 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:56:55.743 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:56:55.744 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:56:55.744 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:55.744 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:56:55.744 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:56:55.744 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:56:55.745 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:56:55.745 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:56:55.745 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-28T23:56:55.745 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:56:55.745 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:56:55.745 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:56:55.745 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-28T23:56:55.746 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:56:55.746 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:56:55.746 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:55.746 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:56:55.746 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:56:55.746 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:56:55.747 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:56:55.747 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588481.939221_339.1", 2023-07-28T23:56:55.747 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:56:55.747 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:56:55.747 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:56:55.748 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:54:41.939221Z", 2023-07-28T23:56:55.748 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:56:55.748 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:56:55.748 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:55.748 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:56:55.748 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:56:55.749 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:56:55.749 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:56:55.749 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:56:55.749 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:56:55.749 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:56:55.749 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:56:55.750 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:56:55.750 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:56:55.750 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:56:55.750 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:55.750 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:56:55.750 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:56:55.751 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:56:55.751 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:56:55.751 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:56:55.751 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:56:55.751 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:56:55.751 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:56:55.752 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:56:55.752 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:56:55.752 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:56:55.752 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:56:55.752 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:56:55.753 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:55.753 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:56:55.753 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:56:55.754 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:56:55.754 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:56:55.754 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:56:55Z" 2023-07-28T23:56:55.754 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:56:55.755 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588481.939221_339.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:54:41.939221Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-28T23:56:55Z" } 2023-07-28T23:56:55.756 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588481.939221_339.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:54:41.939221Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-28T23:56:55.756 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588579.740560_402.1', 2: '', 3: ''} 2023-07-28T23:56:55.757 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588481.939221_339.1', 2: '', 3: ''} 2023-07-28T23:56:55.757 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588579.740560_402.1 target=1_1690588481.939221_339.1 2023-07-28T23:57:00.759 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:57:00.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:57:00.760 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:57:00.813 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:57:00.814 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:57:00.844 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:57:00.842+0000 7efc3a6c0880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:57:00.845 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:57:00.843+0000 7efc3a6c0880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:57:00.857 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:57:00.858 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:57:00.858 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:57:00.858 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:57:00.858 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:57:00.858 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:57:00.858 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:57:00.859 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:00.859 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:57:00.859 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:57:00.859 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:57:00.859 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:57:00.859 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:57:00.860 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-28T23:57:00.860 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:57:00.860 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:57:00.860 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:57:00.861 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-28T23:57:00.861 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:57:00.861 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:57:00.861 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:00.861 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:57:00.861 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:57:00.861 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:57:00.862 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:57:00.862 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588481.939221_339.1", 2023-07-28T23:57:00.862 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:57:00.862 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:57:00.862 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:57:00.862 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:54:41.939221Z", 2023-07-28T23:57:00.862 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:57:00.863 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:57:00.863 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:00.863 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:57:00.863 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:57:00.863 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:57:00.863 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:57:00.864 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:57:00.864 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:57:00.864 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:57:00.864 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:57:00.864 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:57:00.864 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:57:00.865 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:57:00.865 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:00.865 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:57:00.865 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:57:00.865 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:57:00.865 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:57:00.865 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:57:00.866 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:57:00.866 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:57:00.866 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:57:00.866 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:57:00.866 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:57:00.866 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:57:00.866 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:57:00.867 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:57:00.867 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:00.867 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:57:00.867 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:57:00.867 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:57:00.867 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:00.867 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:57:00Z" 2023-07-28T23:57:00.868 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:57:00.868 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588481.939221_339.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:54:41.939221Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-28T23:57:00Z" } 2023-07-28T23:57:00.869 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588481.939221_339.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:54:41.939221Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-28T23:57:00.869 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588579.740560_402.1', 2: '', 3: ''} 2023-07-28T23:57:00.870 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588481.939221_339.1', 2: '', 3: ''} 2023-07-28T23:57:00.870 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588579.740560_402.1 target=1_1690588481.939221_339.1 2023-07-28T23:57:05.871 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:57:05.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:57:05.872 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:57:05.927 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:57:05.927 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:57:05.958 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:57:05.956+0000 7fbe90c96880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:57:05.959 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:57:05.957+0000 7fbe90c96880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:57:05.971 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:57:05.972 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:57:05.972 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:57:05.972 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:57:05.972 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:57:05.972 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:57:05.973 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:57:05.973 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:05.973 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:57:05.973 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:57:05.973 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:57:05.973 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:57:05.973 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:57:05.974 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-28T23:57:05.974 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:57:05.974 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:57:05.974 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:57:05.974 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-28T23:57:05.974 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:57:05.974 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:57:05.975 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:05.975 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:57:05.975 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:57:05.975 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:57:05.975 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:57:05.975 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588481.939221_339.1", 2023-07-28T23:57:05.975 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:57:05.976 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:57:05.976 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:57:05.976 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:54:41.939221Z", 2023-07-28T23:57:05.976 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:57:05.976 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:57:05.976 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:05.977 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:57:05.977 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:57:05.977 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:57:05.977 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:57:05.977 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:57:05.977 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:57:05.977 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:57:05.978 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:57:05.978 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:57:05.978 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:57:05.978 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:57:05.978 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:05.978 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:57:05.978 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:57:05.979 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:57:05.979 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:57:05.979 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:57:05.979 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:57:05.979 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:57:05.979 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:57:05.979 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:57:05.980 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:57:05.980 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:57:05.980 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:57:05.980 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:57:05.980 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:05.980 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:57:05.980 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:57:05.981 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:57:05.981 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:05.981 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:57:05Z" 2023-07-28T23:57:05.981 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:57:05.982 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588481.939221_339.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:54:41.939221Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-28T23:57:05Z" } 2023-07-28T23:57:05.982 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588481.939221_339.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:54:41.939221Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-28T23:57:05.983 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588579.740560_402.1', 2: '', 3: ''} 2023-07-28T23:57:05.983 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588481.939221_339.1', 2: '', 3: ''} 2023-07-28T23:57:05.983 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588579.740560_402.1 target=1_1690588481.939221_339.1 2023-07-28T23:57:10.984 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:57:10.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:57:10.985 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:57:11.038 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:57:11.039 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:57:11.069 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:57:11.066+0000 7f6055b36880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:57:11.069 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:57:11.067+0000 7f6055b36880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:57:11.082 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:57:11.083 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:57:11.083 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:57:11.083 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:57:11.083 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:57:11.083 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:57:11.083 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:57:11.084 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:11.084 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:57:11.084 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:57:11.084 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:57:11.084 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:57:11.084 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:57:11.085 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-28T23:57:11.085 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:57:11.085 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:57:11.085 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:57:11.085 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-28T23:57:11.085 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:57:11.085 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:57:11.086 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:11.086 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:57:11.086 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:57:11.086 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:57:11.086 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:57:11.087 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588481.939221_339.1", 2023-07-28T23:57:11.087 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:57:11.087 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:57:11.087 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:57:11.087 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:54:41.939221Z", 2023-07-28T23:57:11.087 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:57:11.087 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:57:11.088 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:11.088 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:57:11.088 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:57:11.088 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:57:11.088 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:57:11.088 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:57:11.089 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:57:11.089 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:57:11.089 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:57:11.089 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:57:11.089 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:57:11.089 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:57:11.089 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:11.090 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:57:11.090 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:57:11.090 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:57:11.090 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:57:11.090 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:57:11.090 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:57:11.090 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:57:11.091 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:57:11.091 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:57:11.091 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:57:11.091 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:57:11.091 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:57:11.091 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:57:11.091 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:11.092 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:57:11.092 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:57:11.092 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:57:11.092 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:11.092 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:57:11Z" 2023-07-28T23:57:11.092 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:57:11.093 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588481.939221_339.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:54:41.939221Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-28T23:57:11Z" } 2023-07-28T23:57:11.094 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588481.939221_339.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:54:41.939221Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-28T23:57:11.094 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588579.740560_402.1', 2: '', 3: ''} 2023-07-28T23:57:11.095 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588481.939221_339.1', 2: '', 3: ''} 2023-07-28T23:57:11.095 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588579.740560_402.1 target=1_1690588481.939221_339.1 2023-07-28T23:57:16.096 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:57:16.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:57:16.097 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:57:16.149 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:57:16.150 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:57:16.181 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:57:16.179+0000 7f4e16735880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:57:16.182 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:57:16.180+0000 7f4e16735880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:57:16.195 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:57:16.195 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:57:16.196 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:57:16.196 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:57:16.196 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:57:16.196 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:57:16.196 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:57:16.196 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:16.197 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:57:16.197 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:57:16.197 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:57:16.197 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:57:16.197 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:57:16.197 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-28T23:57:16.197 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:57:16.198 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:57:16.198 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:57:16.198 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-28T23:57:16.198 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:57:16.198 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:57:16.198 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:16.198 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:57:16.199 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:57:16.199 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:57:16.199 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:57:16.199 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588481.939221_339.1", 2023-07-28T23:57:16.199 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:57:16.199 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:57:16.200 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:57:16.200 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:54:41.939221Z", 2023-07-28T23:57:16.200 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:57:16.200 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:57:16.200 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:16.200 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:57:16.200 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:57:16.201 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:57:16.201 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:57:16.201 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:57:16.201 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:57:16.201 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:57:16.201 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:57:16.201 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:57:16.202 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:57:16.202 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:57:16.202 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:16.202 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:57:16.202 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:57:16.202 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:57:16.202 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:57:16.203 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:57:16.203 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:57:16.203 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:57:16.203 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:57:16.203 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:57:16.203 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:57:16.203 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:57:16.204 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:57:16.204 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:57:16.204 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:16.204 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:57:16.204 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:57:16.204 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:57:16.204 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:16.205 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:57:16Z" 2023-07-28T23:57:16.205 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:57:16.206 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588481.939221_339.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:54:41.939221Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-28T23:57:16Z" } 2023-07-28T23:57:16.206 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588481.939221_339.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:54:41.939221Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-28T23:57:16.207 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588579.740560_402.1', 2: '', 3: ''} 2023-07-28T23:57:16.207 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588481.939221_339.1', 2: '', 3: ''} 2023-07-28T23:57:16.207 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588579.740560_402.1 target=1_1690588481.939221_339.1 2023-07-28T23:57:21.208 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:57:21.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-28T23:57:21.209 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-28T23:57:21.262 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:57:21.263 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:57:21.293 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:57:21.291+0000 7f47ee3af880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:57:21.294 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-28T23:57:21.292+0000 7f47ee3af880 20 RGW-SYNC:meta: read sync status 2023-07-28T23:57:21.306 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:57:21.307 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-28T23:57:21.307 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-28T23:57:21.307 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-28T23:57:21.307 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-28T23:57:21.307 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:57:21.307 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:57:21.308 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:21.308 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-28T23:57:21.308 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:57:21.308 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-28T23:57:21.308 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:57:21.308 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:57:21.309 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-28T23:57:21.309 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:57:21.309 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-28T23:57:21.309 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:57:21.309 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-28T23:57:21.309 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:57:21.309 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:57:21.310 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:21.310 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:57:21.310 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-28T23:57:21.310 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:57:21.310 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:57:21.310 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588579.740560_402.1", 2023-07-28T23:57:21.311 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:57:21.311 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:57:21.311 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:57:21.311 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:56:19.740560Z", 2023-07-28T23:57:21.311 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:57:21.311 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:57:21.311 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:21.312 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:57:21.312 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-28T23:57:21.312 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:57:21.312 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:57:21.312 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:57:21.312 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:57:21.312 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:57:21.313 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:57:21.313 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:57:21.313 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:57:21.313 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:57:21.313 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:21.313 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:57:21.313 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-28T23:57:21.314 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:57:21.314 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-28T23:57:21.314 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-28T23:57:21.314 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-28T23:57:21.314 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-28T23:57:21.315 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-28T23:57:21.315 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-28T23:57:21.315 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:57:21.315 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:57:21.315 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:57:21.315 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:57:21.315 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:21.316 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-28T23:57:21.316 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-28T23:57:21.316 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-28T23:57:21.316 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:21.316 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-28T23:57:21Z" 2023-07-28T23:57:21.316 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:57:21.317 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588579.740560_402.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:56:19.740560Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-28T23:57:21Z" } 2023-07-28T23:57:21.318 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588579.740560_402.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:56:19.740560Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-28T23:57:21.318 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588579.740560_402.1', 2: '', 3: ''} 2023-07-28T23:57:21.319 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588579.740560_402.1', 2: '', 3: ''} 2023-07-28T23:57:21.319 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-28T23:57:21.319 DEBUG:boto:path=/hiiqos-35/ 2023-07-28T23:57:21.320 DEBUG:boto:auth_path=/hiiqos-35/ 2023-07-28T23:57:21.320 DEBUG:boto:Method: HEAD 2023-07-28T23:57:21.320 DEBUG:boto:Path: /hiiqos-35/ 2023-07-28T23:57:21.321 DEBUG:boto:Data: 2023-07-28T23:57:21.321 DEBUG:boto:Headers: {} 2023-07-28T23:57:21.321 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:57:21.322 DEBUG:boto:Port: 8001 2023-07-28T23:57:21.322 DEBUG:boto:Params: {} 2023-07-28T23:57:21.323 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-28T23:57:21.323 DEBUG:boto:Token: None 2023-07-28T23:57:21.323 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:57:21 GMT /hiiqos-35/ 2023-07-28T23:57:21.324 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:R6D7iRmIl3EmBE/Z+4QApMH/CqE= 2023-07-28T23:57:21.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:57:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:R6D7iRmIl3EmBE/Z+4QApMH/CqE=', 'Content-Length': '0'} 2023-07-28T23:57:21.330 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ae9d2db2e7d48f0a-0064c455e1-1511-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:57:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:57:21.330 DEBUG:boto:path=/hiiqos-35/testredirect 2023-07-28T23:57:21.331 DEBUG:boto:auth_path=/hiiqos-35/testredirect 2023-07-28T23:57:21.331 DEBUG:boto:Method: HEAD 2023-07-28T23:57:21.331 DEBUG:boto:Path: /hiiqos-35/testredirect 2023-07-28T23:57:21.332 DEBUG:boto:Data: 2023-07-28T23:57:21.332 DEBUG:boto:Headers: {} 2023-07-28T23:57:21.333 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:57:21.333 DEBUG:boto:Port: 8001 2023-07-28T23:57:21.333 DEBUG:boto:Params: {} 2023-07-28T23:57:21.334 DEBUG:boto:Token: None 2023-07-28T23:57:21.334 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:57:21 GMT /hiiqos-35/testredirect 2023-07-28T23:57:21.334 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ltdpOu8MDUxsXf7CBrArcJiIyZQ= 2023-07-28T23:57:21.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:57:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ltdpOu8MDUxsXf7CBrArcJiIyZQ=', 'Content-Length': '0'} 2023-07-28T23:57:21.336 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000271ffe2dae516d14-0064c455e1-1511-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 28 Jul 2023 23:57:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:57:21.337 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:57:21.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:57:21.338 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --redirect-zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-28T23:57:21.385 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:57:21.385 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:57:21.405 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:57:21.405 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-28T23:57:21.405 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a2", 2023-07-28T23:57:21.405 INFO:teuthology.orchestra.run.smithi101.stdout: "domain_root": "a2.rgw.meta:root", 2023-07-28T23:57:21.405 INFO:teuthology.orchestra.run.smithi101.stdout: "control_pool": "a2.rgw.control", 2023-07-28T23:57:21.405 INFO:teuthology.orchestra.run.smithi101.stdout: "gc_pool": "a2.rgw.log:gc", 2023-07-28T23:57:21.406 INFO:teuthology.orchestra.run.smithi101.stdout: "lc_pool": "a2.rgw.log:lc", 2023-07-28T23:57:21.406 INFO:teuthology.orchestra.run.smithi101.stdout: "log_pool": "a2.rgw.log", 2023-07-28T23:57:21.406 INFO:teuthology.orchestra.run.smithi101.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-07-28T23:57:21.406 INFO:teuthology.orchestra.run.smithi101.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-07-28T23:57:21.406 INFO:teuthology.orchestra.run.smithi101.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-07-28T23:57:21.406 INFO:teuthology.orchestra.run.smithi101.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-07-28T23:57:21.407 INFO:teuthology.orchestra.run.smithi101.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-07-28T23:57:21.407 INFO:teuthology.orchestra.run.smithi101.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-07-28T23:57:21.407 INFO:teuthology.orchestra.run.smithi101.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-07-28T23:57:21.407 INFO:teuthology.orchestra.run.smithi101.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-07-28T23:57:21.407 INFO:teuthology.orchestra.run.smithi101.stdout: "otp_pool": "a2.rgw.otp", 2023-07-28T23:57:21.407 INFO:teuthology.orchestra.run.smithi101.stdout: "system_key": { 2023-07-28T23:57:21.408 INFO:teuthology.orchestra.run.smithi101.stdout: "access_key": "RX82IHH7C5RUJ54D", 2023-07-28T23:57:21.408 INFO:teuthology.orchestra.run.smithi101.stdout: "secret_key": "InTLvCQsePIbOVhlgLo1AkTNflWbmSsQ" 2023-07-28T23:57:21.408 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:21.408 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_pools": [ 2023-07-28T23:57:21.408 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:57:21.408 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "default-placement", 2023-07-28T23:57:21.409 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:57:21.409 INFO:teuthology.orchestra.run.smithi101.stdout: "index_pool": "a2.rgw.buckets.index", 2023-07-28T23:57:21.409 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": { 2023-07-28T23:57:21.409 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD": { 2023-07-28T23:57:21.409 INFO:teuthology.orchestra.run.smithi101.stdout: "data_pool": "a2.rgw.buckets.data", 2023-07-28T23:57:21.409 INFO:teuthology.orchestra.run.smithi101.stdout: "compression_type": "random" 2023-07-28T23:57:21.409 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:57:21.410 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:21.410 INFO:teuthology.orchestra.run.smithi101.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-07-28T23:57:21.410 INFO:teuthology.orchestra.run.smithi101.stdout: "index_type": 0, 2023-07-28T23:57:21.410 INFO:teuthology.orchestra.run.smithi101.stdout: "inline_data": true 2023-07-28T23:57:21.410 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:57:21.410 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:57:21.410 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:57:21.411 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-28T23:57:21.411 INFO:teuthology.orchestra.run.smithi101.stdout: "notif_pool": "a2.rgw.log:notif" 2023-07-28T23:57:21.411 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:57:21.412 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:57:21.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:57:21.413 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-28T23:57:21.457 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:57:21.458 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:57:21.498 INFO:teuthology.orchestra.run.smithi101.stderr:Sending period to new master zone 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433 2023-07-28T23:57:21.521 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:57:21.522 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:57:21.522 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 6, 2023-07-28T23:57:21.522 INFO:teuthology.orchestra.run.smithi101.stdout: "predecessor_uuid": "a0987484-ce37-4520-a564-54b7ea8c5251", 2023-07-28T23:57:21.522 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": [], 2023-07-28T23:57:21.522 INFO:teuthology.orchestra.run.smithi101.stdout: "period_map": { 2023-07-28T23:57:21.523 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:57:21.523 INFO:teuthology.orchestra.run.smithi101.stdout: "zonegroups": [ 2023-07-28T23:57:21.523 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:57:21.523 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "a", 2023-07-28T23:57:21.523 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a", 2023-07-28T23:57:21.523 INFO:teuthology.orchestra.run.smithi101.stdout: "api_name": "", 2023-07-28T23:57:21.524 INFO:teuthology.orchestra.run.smithi101.stdout: "is_master": true, 2023-07-28T23:57:21.524 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-28T23:57:21.524 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-28T23:57:21.524 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:57:21.524 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames": [], 2023-07-28T23:57:21.524 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames_s3website": [], 2023-07-28T23:57:21.524 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-28T23:57:21.525 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-28T23:57:21.525 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:57:21.525 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-28T23:57:21.525 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a1", 2023-07-28T23:57:21.525 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-28T23:57:21.525 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-28T23:57:21.525 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:57:21.526 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-28T23:57:21.526 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-28T23:57:21.526 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-28T23:57:21.526 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-28T23:57:21.526 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-28T23:57:21.526 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-28T23:57:21.527 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-28T23:57:21.527 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-28T23:57:21.527 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-28T23:57:21.527 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-28T23:57:21.527 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-28T23:57:21.527 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:57:21.527 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:21.528 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:57:21.528 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-28T23:57:21.528 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a2", 2023-07-28T23:57:21.528 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-28T23:57:21.528 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8001" 2023-07-28T23:57:21.528 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:57:21.528 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-28T23:57:21.529 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-28T23:57:21.529 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-28T23:57:21.529 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-28T23:57:21.529 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-28T23:57:21.529 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": false, 2023-07-28T23:57:21.529 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-28T23:57:21.529 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-28T23:57:21.530 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-28T23:57:21.530 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-28T23:57:21.530 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-28T23:57:21.530 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:57:21.530 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:57:21.531 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:57:21.531 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_targets": [ 2023-07-28T23:57:21.531 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:57:21.531 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default-placement", 2023-07-28T23:57:21.531 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [], 2023-07-28T23:57:21.531 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": [ 2023-07-28T23:57:21.531 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD" 2023-07-28T23:57:21.532 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:57:21.532 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:57:21.532 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:57:21.532 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "default-placement", 2023-07-28T23:57:21.532 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-28T23:57:21.532 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_policy": { 2023-07-28T23:57:21.532 INFO:teuthology.orchestra.run.smithi101.stdout: "groups": [] 2023-07-28T23:57:21.533 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:21.533 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled_features": [ 2023-07-28T23:57:21.533 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-28T23:57:21.533 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:57:21.533 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:21.533 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:57:21.534 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "b", 2023-07-28T23:57:21.534 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b", 2023-07-28T23:57:21.534 INFO:teuthology.orchestra.run.smithi101.stdout: "api_name": "", 2023-07-28T23:57:21.534 INFO:teuthology.orchestra.run.smithi101.stdout: "is_master": false, 2023-07-28T23:57:21.534 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-28T23:57:21.534 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-28T23:57:21.534 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:57:21.535 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames": [], 2023-07-28T23:57:21.535 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames_s3website": [], 2023-07-28T23:57:21.535 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-28T23:57:21.535 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-28T23:57:21.535 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:57:21.536 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-28T23:57:21.536 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b1", 2023-07-28T23:57:21.536 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-28T23:57:21.536 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-28T23:57:21.536 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:57:21.536 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-28T23:57:21.537 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-28T23:57:21.537 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-28T23:57:21.537 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-28T23:57:21.537 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-28T23:57:21.537 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-28T23:57:21.537 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-28T23:57:21.537 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-28T23:57:21.538 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-28T23:57:21.539 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-28T23:57:21.539 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-28T23:57:21.539 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:57:21.539 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:21.539 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:57:21.540 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-28T23:57:21.540 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b2", 2023-07-28T23:57:21.540 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-28T23:57:21.540 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8001" 2023-07-28T23:57:21.540 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:57:21.541 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-28T23:57:21.541 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-28T23:57:21.541 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-28T23:57:21.541 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-28T23:57:21.541 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-28T23:57:21.541 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-28T23:57:21.541 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-28T23:57:21.542 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-28T23:57:21.542 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-28T23:57:21.542 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-28T23:57:21.542 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-28T23:57:21.542 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:57:21.542 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:57:21.543 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:57:21.543 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_targets": [ 2023-07-28T23:57:21.543 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:57:21.543 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default-placement", 2023-07-28T23:57:21.543 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [], 2023-07-28T23:57:21.543 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": [ 2023-07-28T23:57:21.543 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD" 2023-07-28T23:57:21.544 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:57:21.544 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:57:21.544 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:57:21.544 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "default-placement", 2023-07-28T23:57:21.544 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-28T23:57:21.544 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_policy": { 2023-07-28T23:57:21.544 INFO:teuthology.orchestra.run.smithi101.stdout: "groups": [] 2023-07-28T23:57:21.545 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:21.545 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled_features": [ 2023-07-28T23:57:21.545 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-28T23:57:21.545 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:57:21.545 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:57:21.545 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:57:21.545 INFO:teuthology.orchestra.run.smithi101.stdout: "short_zone_ids": [ 2023-07-28T23:57:21.546 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:57:21.546 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-28T23:57:21.546 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 2896287231 2023-07-28T23:57:21.546 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:21.546 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:57:21.546 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-28T23:57:21.547 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 815606810 2023-07-28T23:57:21.547 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:21.547 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:57:21.547 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-28T23:57:21.547 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 2009171363 2023-07-28T23:57:21.547 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:21.547 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:57:21.547 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-28T23:57:21.548 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 3105190295 2023-07-28T23:57:21.548 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:57:21.548 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:57:21.548 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:21.548 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zonegroup": "a", 2023-07-28T23:57:21.548 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-28T23:57:21.549 INFO:teuthology.orchestra.run.smithi101.stdout: "period_config": { 2023-07-28T23:57:21.549 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_quota": { 2023-07-28T23:57:21.549 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-28T23:57:21.549 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-28T23:57:21.549 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-28T23:57:21.549 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-28T23:57:21.549 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-28T23:57:21.550 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:21.550 INFO:teuthology.orchestra.run.smithi101.stdout: "user_quota": { 2023-07-28T23:57:21.550 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-28T23:57:21.550 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-28T23:57:21.550 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-28T23:57:21.550 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-28T23:57:21.551 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-28T23:57:21.551 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:21.551 INFO:teuthology.orchestra.run.smithi101.stdout: "user_ratelimit": { 2023-07-28T23:57:21.551 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-28T23:57:21.551 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-28T23:57:21.551 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-28T23:57:21.552 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-28T23:57:21.552 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-28T23:57:21.552 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:21.552 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_ratelimit": { 2023-07-28T23:57:21.552 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-28T23:57:21.552 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-28T23:57:21.553 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-28T23:57:21.553 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-28T23:57:21.553 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-28T23:57:21.553 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:21.553 INFO:teuthology.orchestra.run.smithi101.stdout: "anonymous_ratelimit": { 2023-07-28T23:57:21.553 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-28T23:57:21.553 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-28T23:57:21.554 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-28T23:57:21.554 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-28T23:57:21.554 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-28T23:57:21.554 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:57:21.554 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:57:21.554 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-28T23:57:21.555 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_name": "test-realm", 2023-07-28T23:57:21.555 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:57:21.555 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:57:21.555 INFO:rgw_multi.tests:Set redirect_zone zone a2 to "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433" 2023-07-28T23:58:21.556 DEBUG:boto:path=/hiiqos-35/testredirect 2023-07-28T23:58:21.557 DEBUG:boto:auth_path=/hiiqos-35/testredirect 2023-07-28T23:58:21.557 DEBUG:boto:Method: HEAD 2023-07-28T23:58:21.558 DEBUG:boto:Path: /hiiqos-35/testredirect 2023-07-28T23:58:21.558 DEBUG:boto:Data: 2023-07-28T23:58:21.558 DEBUG:boto:Headers: {} 2023-07-28T23:58:21.559 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:58:21.559 DEBUG:boto:Port: 8001 2023-07-28T23:58:21.559 DEBUG:boto:Params: {} 2023-07-28T23:58:21.560 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-28T23:58:21.561 DEBUG:boto:Token: None 2023-07-28T23:58:21.561 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:58:21 GMT /hiiqos-35/testredirect 2023-07-28T23:58:21.561 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ODjoqZT6Q5QhGn2p6VJHYXmPvIQ= 2023-07-28T23:58:21.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:58:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ODjoqZT6Q5QhGn2p6VJHYXmPvIQ=', 'Content-Length': '0'} 2023-07-28T23:58:21.567 DEBUG:boto:Response headers: [('Location', 'http://smithi101.front.sepia.ceph.com:8000/hiiqos-35/testredirect?'), ('x-amz-request-id', 'tx00000a81af4be630af430-0064c4561d-159d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:58:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:58:21.568 DEBUG:boto:Redirecting: http://smithi101.front.sepia.ceph.com/hiiqos-35/testredirect 2023-07-28T23:58:21.568 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': '8000'} 2023-07-28T23:58:21.569 DEBUG:boto:Token: None 2023-07-28T23:58:21.569 DEBUG:boto:StringToSign: HEAD Fri, 28 Jul 2023 23:58:21 GMT /hiiqos-35/testredirect 2023-07-28T23:58:21.569 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ODjoqZT6Q5QhGn2p6VJHYXmPvIQ= 2023-07-28T23:58:21.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:58:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ODjoqZT6Q5QhGn2p6VJHYXmPvIQ=', 'Content-Length': '0'} 2023-07-28T23:58:21.574 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:56:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000fdb2dd10dcd77047-0064c4561d-1599-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:58:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:58:21.575 DEBUG:boto:path=/hiiqos-35/testredirect 2023-07-28T23:58:21.575 DEBUG:boto:auth_path=/hiiqos-35/testredirect 2023-07-28T23:58:21.575 DEBUG:boto:Method: GET 2023-07-28T23:58:21.576 DEBUG:boto:Path: /hiiqos-35/testredirect 2023-07-28T23:58:21.576 DEBUG:boto:Data: 2023-07-28T23:58:21.577 DEBUG:boto:Headers: {} 2023-07-28T23:58:21.577 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:58:21.577 DEBUG:boto:Port: 8001 2023-07-28T23:58:21.578 DEBUG:boto:Params: {} 2023-07-28T23:58:21.578 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-28T23:58:21.578 DEBUG:boto:Token: None 2023-07-28T23:58:21.579 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:58:21 GMT /hiiqos-35/testredirect 2023-07-28T23:58:21.579 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:onW2HDfgruNQTHafqw0H626vxvk= 2023-07-28T23:58:21.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:58:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:onW2HDfgruNQTHafqw0H626vxvk=', 'Content-Length': '0'} 2023-07-28T23:58:21.582 DEBUG:boto:Response headers: [('Location', 'http://smithi101.front.sepia.ceph.com:8000/hiiqos-35/testredirect?'), ('x-amz-request-id', 'tx0000010ff2428f941c1ef-0064c4561d-159d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:58:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:58:21.583 DEBUG:boto:Redirecting: http://smithi101.front.sepia.ceph.com/hiiqos-35/testredirect 2023-07-28T23:58:21.583 DEBUG:boto:Token: None 2023-07-28T23:58:21.584 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:58:21 GMT /hiiqos-35/testredirect 2023-07-28T23:58:21.584 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:onW2HDfgruNQTHafqw0H626vxvk= 2023-07-28T23:58:21.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:58:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:onW2HDfgruNQTHafqw0H626vxvk=', 'Content-Length': '0'} 2023-07-28T23:58:21.587 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:56:19 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000775e6640cf9e9479-0064c4561d-1599-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:58:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:58:21.588 DEBUG:boto:path=/hiiqos-35/testredirect 2023-07-28T23:58:21.589 DEBUG:boto:auth_path=/hiiqos-35/testredirect 2023-07-28T23:58:21.590 DEBUG:boto:Method: PUT 2023-07-28T23:58:21.591 DEBUG:boto:Path: /hiiqos-35/testredirect 2023-07-28T23:58:21.591 DEBUG:boto:Data: 2023-07-28T23:58:21.592 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'VpBzljOcorCZvRIkX5Nt3A==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-07-28T23:58:21.593 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:58:21.594 DEBUG:boto:Port: 8000 2023-07-28T23:58:21.595 DEBUG:boto:Params: {} 2023-07-28T23:58:21.596 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-28T23:58:21.597 DEBUG:boto:Token: None 2023-07-28T23:58:21.597 DEBUG:boto:StringToSign: PUT VpBzljOcorCZvRIkX5Nt3A== application/octet-stream Fri, 28 Jul 2023 23:58:21 GMT /hiiqos-35/testredirect 2023-07-28T23:58:21.598 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aNaCpeLp1ZxK3NJlQFn2zU5tlqk= 2023-07-28T23:58:21.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'VpBzljOcorCZvRIkX5Nt3A==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:58:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aNaCpeLp1ZxK3NJlQFn2zU5tlqk='} 2023-07-28T23:58:21.614 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"56907396339ca2b099bd12245f936ddc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000db74fa964c0ee499-0064c4561d-1599-a1'), ('Date', 'Fri, 28 Jul 2023 23:58:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:58:21.615 DEBUG:boto:path=/hiiqos-35/testredirect 2023-07-28T23:58:21.616 DEBUG:boto:auth_path=/hiiqos-35/testredirect 2023-07-28T23:58:21.616 DEBUG:boto:Method: GET 2023-07-28T23:58:21.617 DEBUG:boto:Path: /hiiqos-35/testredirect 2023-07-28T23:58:21.617 DEBUG:boto:Data: 2023-07-28T23:58:21.618 DEBUG:boto:Headers: {} 2023-07-28T23:58:21.619 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:58:21.619 DEBUG:boto:Port: 8001 2023-07-28T23:58:21.620 DEBUG:boto:Params: {} 2023-07-28T23:58:21.620 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-28T23:58:21.621 DEBUG:boto:Token: None 2023-07-28T23:58:21.621 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:58:21 GMT /hiiqos-35/testredirect 2023-07-28T23:58:21.622 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:onW2HDfgruNQTHafqw0H626vxvk= 2023-07-28T23:58:21.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:58:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:onW2HDfgruNQTHafqw0H626vxvk=', 'Content-Length': '0'} 2023-07-28T23:58:26.625 DEBUG:boto:Response headers: [('Location', 'http://smithi101.front.sepia.ceph.com:8000/hiiqos-35/testredirect?'), ('x-amz-request-id', 'tx00000ecf4c4eb3f88f21e-0064c45622-159d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:58:26 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:58:26.626 DEBUG:boto:Redirecting: http://smithi101.front.sepia.ceph.com/hiiqos-35/testredirect 2023-07-28T23:58:26.627 DEBUG:boto:Token: None 2023-07-28T23:58:26.627 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:58:21 GMT /hiiqos-35/testredirect 2023-07-28T23:58:26.628 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:onW2HDfgruNQTHafqw0H626vxvk= 2023-07-28T23:58:26.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:58:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:onW2HDfgruNQTHafqw0H626vxvk=', 'Content-Length': '0'} 2023-07-28T23:58:26.630 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:58:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"56907396339ca2b099bd12245f936ddc"'), ('x-amz-request-id', 'tx00000fbba896b38418feb-0064c45622-1599-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:58:26 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:58:26.631 DEBUG:boto:path=/hiiqos-35/testredirect 2023-07-28T23:58:26.631 DEBUG:boto:auth_path=/hiiqos-35/testredirect 2023-07-28T23:58:26.632 DEBUG:boto:Method: PUT 2023-07-28T23:58:26.632 DEBUG:boto:Path: /hiiqos-35/testredirect 2023-07-28T23:58:26.632 DEBUG:boto:Data: 2023-07-28T23:58:26.633 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'uk9S5NXZfBvPq4jGr+LM5g==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-07-28T23:58:26.633 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:58:26.633 DEBUG:boto:Port: 8000 2023-07-28T23:58:26.634 DEBUG:boto:Params: {} 2023-07-28T23:58:26.634 DEBUG:boto:Token: None 2023-07-28T23:58:26.635 DEBUG:boto:StringToSign: PUT uk9S5NXZfBvPq4jGr+LM5g== application/octet-stream Fri, 28 Jul 2023 23:58:26 GMT /hiiqos-35/testredirect 2023-07-28T23:58:26.635 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wgw9jMhHCpN+dDxfj7QKWxngV5Q= 2023-07-28T23:58:26.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'uk9S5NXZfBvPq4jGr+LM5g==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:58:26 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wgw9jMhHCpN+dDxfj7QKWxngV5Q='} 2023-07-28T23:58:26.641 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"ba4f52e4d5d97c1bcfab88c6afe2cce6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002e9bd195b54ac8d1-0064c45622-1599-a1'), ('Date', 'Fri, 28 Jul 2023 23:58:26 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:58:26.641 DEBUG:boto:path=/hiiqos-35/testredirect 2023-07-28T23:58:26.642 DEBUG:boto:auth_path=/hiiqos-35/testredirect 2023-07-28T23:58:26.642 DEBUG:boto:Method: GET 2023-07-28T23:58:26.643 DEBUG:boto:Path: /hiiqos-35/testredirect 2023-07-28T23:58:26.643 DEBUG:boto:Data: 2023-07-28T23:58:26.643 DEBUG:boto:Headers: {} 2023-07-28T23:58:26.644 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:58:26.644 DEBUG:boto:Port: 8001 2023-07-28T23:58:26.644 DEBUG:boto:Params: {} 2023-07-28T23:58:26.645 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-28T23:58:26.645 DEBUG:boto:Token: None 2023-07-28T23:58:26.646 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:58:26 GMT /hiiqos-35/testredirect 2023-07-28T23:58:26.646 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eYkyGiMDzz1DXbiCDQFitBmVBJY= 2023-07-28T23:58:26.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:58:26 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eYkyGiMDzz1DXbiCDQFitBmVBJY=', 'Content-Length': '0'} 2023-07-28T23:58:31.650 DEBUG:boto:Response headers: [('Location', 'http://smithi101.front.sepia.ceph.com:8000/hiiqos-35/testredirect?'), ('x-amz-request-id', 'tx0000016bf685c85bbd938-0064c45627-159d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:58:31.651 DEBUG:boto:Redirecting: http://smithi101.front.sepia.ceph.com/hiiqos-35/testredirect 2023-07-28T23:58:31.652 DEBUG:boto:Token: None 2023-07-28T23:58:31.652 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:58:26 GMT /hiiqos-35/testredirect 2023-07-28T23:58:31.652 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eYkyGiMDzz1DXbiCDQFitBmVBJY= 2023-07-28T23:58:31.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:58:26 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eYkyGiMDzz1DXbiCDQFitBmVBJY=', 'Content-Length': '0'} 2023-07-28T23:58:31.655 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:58:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"ba4f52e4d5d97c1bcfab88c6afe2cce6"'), ('x-amz-request-id', 'tx0000075e71dc6d90d31b4-0064c45627-1599-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:58:31.656 DEBUG:boto:path=/hiiqos-35/testredirect 2023-07-28T23:58:31.656 DEBUG:boto:auth_path=/hiiqos-35/testredirect 2023-07-28T23:58:31.656 DEBUG:boto:Method: PUT 2023-07-28T23:58:31.657 DEBUG:boto:Path: /hiiqos-35/testredirect 2023-07-28T23:58:31.657 DEBUG:boto:Data: 2023-07-28T23:58:31.658 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '6aKmvaJMgaK7Z2rfDZvVeg==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-07-28T23:58:31.658 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:58:31.658 DEBUG:boto:Port: 8000 2023-07-28T23:58:31.658 DEBUG:boto:Params: {} 2023-07-28T23:58:31.659 DEBUG:boto:Token: None 2023-07-28T23:58:31.659 DEBUG:boto:StringToSign: PUT 6aKmvaJMgaK7Z2rfDZvVeg== application/octet-stream Fri, 28 Jul 2023 23:58:31 GMT /hiiqos-35/testredirect 2023-07-28T23:58:31.660 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:l/2l8wp8gCIpn/C3/DcUS2WkIBA= 2023-07-28T23:58:31.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '6aKmvaJMgaK7Z2rfDZvVeg==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:58:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:l/2l8wp8gCIpn/C3/DcUS2WkIBA='} 2023-07-28T23:58:31.666 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"e9a2a6bda24c81a2bb676adf0d9bd57a"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f7a78b87d2d8bd7a-0064c45627-1599-a1'), ('Date', 'Fri, 28 Jul 2023 23:58:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:58:31.666 DEBUG:boto:path=/hiiqos-35/testredirect 2023-07-28T23:58:31.667 DEBUG:boto:auth_path=/hiiqos-35/testredirect 2023-07-28T23:58:31.667 DEBUG:boto:Method: GET 2023-07-28T23:58:31.667 DEBUG:boto:Path: /hiiqos-35/testredirect 2023-07-28T23:58:31.668 DEBUG:boto:Data: 2023-07-28T23:58:31.668 DEBUG:boto:Headers: {} 2023-07-28T23:58:31.668 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:58:31.669 DEBUG:boto:Port: 8001 2023-07-28T23:58:31.669 DEBUG:boto:Params: {} 2023-07-28T23:58:31.669 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-28T23:58:31.670 DEBUG:boto:Token: None 2023-07-28T23:58:31.670 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:58:31 GMT /hiiqos-35/testredirect 2023-07-28T23:58:31.671 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EZBIr1ETFfMnbWrctPvY3R6htaU= 2023-07-28T23:58:31.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:58:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EZBIr1ETFfMnbWrctPvY3R6htaU=', 'Content-Length': '0'} 2023-07-28T23:58:36.674 DEBUG:boto:Response headers: [('Location', 'http://smithi101.front.sepia.ceph.com:8000/hiiqos-35/testredirect?'), ('x-amz-request-id', 'tx00000371f705885947eeb-0064c4562c-159d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:58:36.675 DEBUG:boto:Redirecting: http://smithi101.front.sepia.ceph.com/hiiqos-35/testredirect 2023-07-28T23:58:36.675 DEBUG:boto:Token: None 2023-07-28T23:58:36.676 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:58:31 GMT /hiiqos-35/testredirect 2023-07-28T23:58:36.677 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EZBIr1ETFfMnbWrctPvY3R6htaU= 2023-07-28T23:58:36.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:58:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EZBIr1ETFfMnbWrctPvY3R6htaU=', 'Content-Length': '0'} 2023-07-28T23:58:36.679 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:58:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"e9a2a6bda24c81a2bb676adf0d9bd57a"'), ('x-amz-request-id', 'tx000002fb456b59280a531-0064c4562c-1599-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:58:36.680 DEBUG:boto:path=/hiiqos-35/testredirect 2023-07-28T23:58:36.680 DEBUG:boto:auth_path=/hiiqos-35/testredirect 2023-07-28T23:58:36.681 DEBUG:boto:Method: PUT 2023-07-28T23:58:36.681 DEBUG:boto:Path: /hiiqos-35/testredirect 2023-07-28T23:58:36.681 DEBUG:boto:Data: 2023-07-28T23:58:36.682 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'GbNgNQ5MAukCtgY+rsT5Xw==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-07-28T23:58:36.682 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-28T23:58:36.682 DEBUG:boto:Port: 8000 2023-07-28T23:58:36.683 DEBUG:boto:Params: {} 2023-07-28T23:58:36.683 DEBUG:boto:Token: None 2023-07-28T23:58:36.683 DEBUG:boto:StringToSign: PUT GbNgNQ5MAukCtgY+rsT5Xw== application/octet-stream Fri, 28 Jul 2023 23:58:36 GMT /hiiqos-35/testredirect 2023-07-28T23:58:36.684 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nUAy3q29CgkxzRzmmD7AHUswYqo= 2023-07-28T23:58:36.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'GbNgNQ5MAukCtgY+rsT5Xw==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 28 Jul 2023 23:58:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nUAy3q29CgkxzRzmmD7AHUswYqo='} 2023-07-28T23:58:36.690 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"19b360350e4c02e902b6063eaec4f95f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008eb4d5784984bc41-0064c4562c-1599-a1'), ('Date', 'Fri, 28 Jul 2023 23:58:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:58:36.691 DEBUG:boto:path=/hiiqos-35/testredirect 2023-07-28T23:58:36.691 DEBUG:boto:auth_path=/hiiqos-35/testredirect 2023-07-28T23:58:36.691 DEBUG:boto:Method: GET 2023-07-28T23:58:36.692 DEBUG:boto:Path: /hiiqos-35/testredirect 2023-07-28T23:58:36.692 DEBUG:boto:Data: 2023-07-28T23:58:36.692 DEBUG:boto:Headers: {} 2023-07-28T23:58:36.693 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-28T23:58:36.693 DEBUG:boto:Port: 8001 2023-07-28T23:58:36.693 DEBUG:boto:Params: {} 2023-07-28T23:58:36.694 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-28T23:58:36.694 DEBUG:boto:Token: None 2023-07-28T23:58:36.695 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:58:36 GMT /hiiqos-35/testredirect 2023-07-28T23:58:36.695 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bnq+6swqWZ/HwRlmj957kj6eQcs= 2023-07-28T23:58:36.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:58:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bnq+6swqWZ/HwRlmj957kj6eQcs=', 'Content-Length': '0'} 2023-07-28T23:58:41.699 DEBUG:boto:Response headers: [('Location', 'http://smithi101.front.sepia.ceph.com:8000/hiiqos-35/testredirect?'), ('x-amz-request-id', 'tx00000afaa44f4e8840f32-0064c45631-159d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 28 Jul 2023 23:58:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:58:41.700 DEBUG:boto:Redirecting: http://smithi101.front.sepia.ceph.com/hiiqos-35/testredirect 2023-07-28T23:58:41.700 DEBUG:boto:Token: None 2023-07-28T23:58:41.701 DEBUG:boto:StringToSign: GET Fri, 28 Jul 2023 23:58:36 GMT /hiiqos-35/testredirect 2023-07-28T23:58:41.701 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bnq+6swqWZ/HwRlmj957kj6eQcs= 2023-07-28T23:58:41.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Fri, 28 Jul 2023 23:58:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bnq+6swqWZ/HwRlmj957kj6eQcs=', 'Content-Length': '0'} 2023-07-28T23:58:41.704 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 28 Jul 2023 23:58:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"19b360350e4c02e902b6063eaec4f95f"'), ('x-amz-request-id', 'tx000003ff3bd3436475392-0064c45631-1599-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 28 Jul 2023 23:58:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-28T23:58:41.705 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:58:41.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:58:41.706 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --sync-from-all=true --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-28T23:58:41.758 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:58:41.758 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:58:41.780 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:58:41.780 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-28T23:58:41.780 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a2", 2023-07-28T23:58:41.781 INFO:teuthology.orchestra.run.smithi101.stdout: "domain_root": "a2.rgw.meta:root", 2023-07-28T23:58:41.781 INFO:teuthology.orchestra.run.smithi101.stdout: "control_pool": "a2.rgw.control", 2023-07-28T23:58:41.781 INFO:teuthology.orchestra.run.smithi101.stdout: "gc_pool": "a2.rgw.log:gc", 2023-07-28T23:58:41.781 INFO:teuthology.orchestra.run.smithi101.stdout: "lc_pool": "a2.rgw.log:lc", 2023-07-28T23:58:41.781 INFO:teuthology.orchestra.run.smithi101.stdout: "log_pool": "a2.rgw.log", 2023-07-28T23:58:41.781 INFO:teuthology.orchestra.run.smithi101.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-07-28T23:58:41.782 INFO:teuthology.orchestra.run.smithi101.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-07-28T23:58:41.782 INFO:teuthology.orchestra.run.smithi101.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-07-28T23:58:41.782 INFO:teuthology.orchestra.run.smithi101.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-07-28T23:58:41.782 INFO:teuthology.orchestra.run.smithi101.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-07-28T23:58:41.782 INFO:teuthology.orchestra.run.smithi101.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-07-28T23:58:41.782 INFO:teuthology.orchestra.run.smithi101.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-07-28T23:58:41.782 INFO:teuthology.orchestra.run.smithi101.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-07-28T23:58:41.783 INFO:teuthology.orchestra.run.smithi101.stdout: "otp_pool": "a2.rgw.otp", 2023-07-28T23:58:41.783 INFO:teuthology.orchestra.run.smithi101.stdout: "system_key": { 2023-07-28T23:58:41.783 INFO:teuthology.orchestra.run.smithi101.stdout: "access_key": "RX82IHH7C5RUJ54D", 2023-07-28T23:58:41.783 INFO:teuthology.orchestra.run.smithi101.stdout: "secret_key": "InTLvCQsePIbOVhlgLo1AkTNflWbmSsQ" 2023-07-28T23:58:41.783 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:58:41.783 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_pools": [ 2023-07-28T23:58:41.783 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:58:41.784 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "default-placement", 2023-07-28T23:58:41.784 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:58:41.784 INFO:teuthology.orchestra.run.smithi101.stdout: "index_pool": "a2.rgw.buckets.index", 2023-07-28T23:58:41.784 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": { 2023-07-28T23:58:41.784 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD": { 2023-07-28T23:58:41.784 INFO:teuthology.orchestra.run.smithi101.stdout: "data_pool": "a2.rgw.buckets.data", 2023-07-28T23:58:41.785 INFO:teuthology.orchestra.run.smithi101.stdout: "compression_type": "random" 2023-07-28T23:58:41.785 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:58:41.785 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:58:41.785 INFO:teuthology.orchestra.run.smithi101.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-07-28T23:58:41.785 INFO:teuthology.orchestra.run.smithi101.stdout: "index_type": 0, 2023-07-28T23:58:41.785 INFO:teuthology.orchestra.run.smithi101.stdout: "inline_data": true 2023-07-28T23:58:41.785 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:58:41.786 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:58:41.786 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:58:41.786 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-28T23:58:41.786 INFO:teuthology.orchestra.run.smithi101.stdout: "notif_pool": "a2.rgw.log:notif" 2023-07-28T23:58:41.786 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:58:41.787 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:58:41.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:58:41.788 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-28T23:58:41.835 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:58:41.835 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:58:41.876 INFO:teuthology.orchestra.run.smithi101.stderr:Sending period to new master zone 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433 2023-07-28T23:58:41.897 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:58:41.898 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:58:41.898 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 7, 2023-07-28T23:58:41.898 INFO:teuthology.orchestra.run.smithi101.stdout: "predecessor_uuid": "a0987484-ce37-4520-a564-54b7ea8c5251", 2023-07-28T23:58:41.898 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": [], 2023-07-28T23:58:41.898 INFO:teuthology.orchestra.run.smithi101.stdout: "period_map": { 2023-07-28T23:58:41.898 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:58:41.899 INFO:teuthology.orchestra.run.smithi101.stdout: "zonegroups": [ 2023-07-28T23:58:41.899 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:58:41.899 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "a", 2023-07-28T23:58:41.899 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a", 2023-07-28T23:58:41.899 INFO:teuthology.orchestra.run.smithi101.stdout: "api_name": "", 2023-07-28T23:58:41.899 INFO:teuthology.orchestra.run.smithi101.stdout: "is_master": true, 2023-07-28T23:58:41.899 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-28T23:58:41.900 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-28T23:58:41.900 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:58:41.900 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames": [], 2023-07-28T23:58:41.900 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames_s3website": [], 2023-07-28T23:58:41.900 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-28T23:58:41.900 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-28T23:58:41.900 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:58:41.901 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-28T23:58:41.901 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a1", 2023-07-28T23:58:41.901 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-28T23:58:41.901 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-28T23:58:41.901 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:58:41.901 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-28T23:58:41.901 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-28T23:58:41.902 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-28T23:58:41.902 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-28T23:58:41.902 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-28T23:58:41.902 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-28T23:58:41.902 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-28T23:58:41.902 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-28T23:58:41.902 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-28T23:58:41.903 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-28T23:58:41.903 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-28T23:58:41.903 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:58:41.903 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:58:41.903 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:58:41.903 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-28T23:58:41.903 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a2", 2023-07-28T23:58:41.904 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-28T23:58:41.904 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8001" 2023-07-28T23:58:41.904 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:58:41.904 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-28T23:58:41.904 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-28T23:58:41.904 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-28T23:58:41.905 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-28T23:58:41.905 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-28T23:58:41.905 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-28T23:58:41.905 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-28T23:58:41.905 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-28T23:58:41.905 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-28T23:58:41.905 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-28T23:58:41.906 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-28T23:58:41.906 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:58:41.906 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:58:41.906 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:58:41.906 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_targets": [ 2023-07-28T23:58:41.906 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:58:41.906 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default-placement", 2023-07-28T23:58:41.907 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [], 2023-07-28T23:58:41.907 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": [ 2023-07-28T23:58:41.907 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD" 2023-07-28T23:58:41.907 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:58:41.907 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:58:41.907 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:58:41.907 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "default-placement", 2023-07-28T23:58:41.908 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-28T23:58:41.908 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_policy": { 2023-07-28T23:58:41.908 INFO:teuthology.orchestra.run.smithi101.stdout: "groups": [] 2023-07-28T23:58:41.908 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:58:41.908 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled_features": [ 2023-07-28T23:58:41.908 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-28T23:58:41.908 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:58:41.909 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:58:41.909 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:58:41.909 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "b", 2023-07-28T23:58:41.909 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b", 2023-07-28T23:58:41.909 INFO:teuthology.orchestra.run.smithi101.stdout: "api_name": "", 2023-07-28T23:58:41.909 INFO:teuthology.orchestra.run.smithi101.stdout: "is_master": false, 2023-07-28T23:58:41.909 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-28T23:58:41.910 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-28T23:58:41.910 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:58:41.910 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames": [], 2023-07-28T23:58:41.910 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames_s3website": [], 2023-07-28T23:58:41.910 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-28T23:58:41.910 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-28T23:58:41.911 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:58:41.911 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-28T23:58:41.911 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b1", 2023-07-28T23:58:41.911 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-28T23:58:41.911 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-28T23:58:41.911 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:58:41.911 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-28T23:58:41.912 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-28T23:58:41.912 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-28T23:58:41.912 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-28T23:58:41.912 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-28T23:58:41.912 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-28T23:58:41.912 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-28T23:58:41.912 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-28T23:58:41.913 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-28T23:58:41.913 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-28T23:58:41.913 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-28T23:58:41.914 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:58:41.914 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:58:41.914 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:58:41.914 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-28T23:58:41.914 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b2", 2023-07-28T23:58:41.914 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-28T23:58:41.914 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8001" 2023-07-28T23:58:41.915 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:58:41.915 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-28T23:58:41.915 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-28T23:58:41.915 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-28T23:58:41.915 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-28T23:58:41.915 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-28T23:58:41.915 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-28T23:58:41.916 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-28T23:58:41.916 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-28T23:58:41.916 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-28T23:58:41.916 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-28T23:58:41.916 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-28T23:58:41.916 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:58:41.916 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:58:41.917 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:58:41.917 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_targets": [ 2023-07-28T23:58:41.917 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:58:41.917 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default-placement", 2023-07-28T23:58:41.917 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [], 2023-07-28T23:58:41.917 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": [ 2023-07-28T23:58:41.917 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD" 2023-07-28T23:58:41.918 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:58:41.918 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:58:41.918 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:58:41.918 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "default-placement", 2023-07-28T23:58:41.918 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-28T23:58:41.918 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_policy": { 2023-07-28T23:58:41.918 INFO:teuthology.orchestra.run.smithi101.stdout: "groups": [] 2023-07-28T23:58:41.919 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:58:41.919 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled_features": [ 2023-07-28T23:58:41.919 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-28T23:58:41.919 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:58:41.919 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:58:41.919 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:58:41.919 INFO:teuthology.orchestra.run.smithi101.stdout: "short_zone_ids": [ 2023-07-28T23:58:41.920 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:58:41.920 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-28T23:58:41.920 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 2896287231 2023-07-28T23:58:41.920 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:58:41.920 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:58:41.920 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-28T23:58:41.920 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 815606810 2023-07-28T23:58:41.921 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:58:41.921 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:58:41.921 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-28T23:58:41.921 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 2009171363 2023-07-28T23:58:41.921 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:58:41.921 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:58:41.921 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-28T23:58:41.922 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 3105190295 2023-07-28T23:58:41.922 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:58:41.922 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:58:41.922 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:58:41.922 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zonegroup": "a", 2023-07-28T23:58:41.922 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-28T23:58:41.922 INFO:teuthology.orchestra.run.smithi101.stdout: "period_config": { 2023-07-28T23:58:41.923 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_quota": { 2023-07-28T23:58:41.923 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-28T23:58:41.923 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-28T23:58:41.923 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-28T23:58:41.923 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-28T23:58:41.923 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-28T23:58:41.923 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:58:41.924 INFO:teuthology.orchestra.run.smithi101.stdout: "user_quota": { 2023-07-28T23:58:41.924 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-28T23:58:41.924 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-28T23:58:41.924 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-28T23:58:41.924 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-28T23:58:41.924 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-28T23:58:41.924 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:58:41.925 INFO:teuthology.orchestra.run.smithi101.stdout: "user_ratelimit": { 2023-07-28T23:58:41.925 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-28T23:58:41.925 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-28T23:58:41.925 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-28T23:58:41.925 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-28T23:58:41.925 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-28T23:58:41.925 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:58:41.926 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_ratelimit": { 2023-07-28T23:58:41.926 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-28T23:58:41.926 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-28T23:58:41.926 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-28T23:58:41.926 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-28T23:58:41.926 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-28T23:58:41.926 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:58:41.927 INFO:teuthology.orchestra.run.smithi101.stdout: "anonymous_ratelimit": { 2023-07-28T23:58:41.927 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-28T23:58:41.927 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-28T23:58:41.927 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-28T23:58:41.927 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-28T23:58:41.927 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-28T23:58:41.927 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:58:41.928 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:58:41.928 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-28T23:58:41.928 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_name": "test-realm", 2023-07-28T23:58:41.928 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:58:41.928 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:58:41.929 INFO:rgw_multi.tests:Set sync_from_all flag on zone a2 to true 2023-07-28T23:59:41.931 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:59:41.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:59:41.932 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --redirect-zone= --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-28T23:59:41.985 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:59:41.985 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:59:42.007 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:59:42.007 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-28T23:59:42.007 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a2", 2023-07-28T23:59:42.007 INFO:teuthology.orchestra.run.smithi101.stdout: "domain_root": "a2.rgw.meta:root", 2023-07-28T23:59:42.008 INFO:teuthology.orchestra.run.smithi101.stdout: "control_pool": "a2.rgw.control", 2023-07-28T23:59:42.008 INFO:teuthology.orchestra.run.smithi101.stdout: "gc_pool": "a2.rgw.log:gc", 2023-07-28T23:59:42.008 INFO:teuthology.orchestra.run.smithi101.stdout: "lc_pool": "a2.rgw.log:lc", 2023-07-28T23:59:42.008 INFO:teuthology.orchestra.run.smithi101.stdout: "log_pool": "a2.rgw.log", 2023-07-28T23:59:42.009 INFO:teuthology.orchestra.run.smithi101.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-07-28T23:59:42.009 INFO:teuthology.orchestra.run.smithi101.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-07-28T23:59:42.009 INFO:teuthology.orchestra.run.smithi101.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-07-28T23:59:42.009 INFO:teuthology.orchestra.run.smithi101.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-07-28T23:59:42.009 INFO:teuthology.orchestra.run.smithi101.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-07-28T23:59:42.009 INFO:teuthology.orchestra.run.smithi101.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-07-28T23:59:42.010 INFO:teuthology.orchestra.run.smithi101.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-07-28T23:59:42.010 INFO:teuthology.orchestra.run.smithi101.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-07-28T23:59:42.010 INFO:teuthology.orchestra.run.smithi101.stdout: "otp_pool": "a2.rgw.otp", 2023-07-28T23:59:42.010 INFO:teuthology.orchestra.run.smithi101.stdout: "system_key": { 2023-07-28T23:59:42.010 INFO:teuthology.orchestra.run.smithi101.stdout: "access_key": "RX82IHH7C5RUJ54D", 2023-07-28T23:59:42.010 INFO:teuthology.orchestra.run.smithi101.stdout: "secret_key": "InTLvCQsePIbOVhlgLo1AkTNflWbmSsQ" 2023-07-28T23:59:42.011 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:59:42.011 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_pools": [ 2023-07-28T23:59:42.011 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:59:42.011 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "default-placement", 2023-07-28T23:59:42.011 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-28T23:59:42.011 INFO:teuthology.orchestra.run.smithi101.stdout: "index_pool": "a2.rgw.buckets.index", 2023-07-28T23:59:42.011 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": { 2023-07-28T23:59:42.012 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD": { 2023-07-28T23:59:42.012 INFO:teuthology.orchestra.run.smithi101.stdout: "data_pool": "a2.rgw.buckets.data", 2023-07-28T23:59:42.012 INFO:teuthology.orchestra.run.smithi101.stdout: "compression_type": "random" 2023-07-28T23:59:42.012 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:59:42.012 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:59:42.013 INFO:teuthology.orchestra.run.smithi101.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-07-28T23:59:42.013 INFO:teuthology.orchestra.run.smithi101.stdout: "index_type": 0, 2023-07-28T23:59:42.013 INFO:teuthology.orchestra.run.smithi101.stdout: "inline_data": true 2023-07-28T23:59:42.013 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:59:42.013 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:59:42.013 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:59:42.014 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-28T23:59:42.014 INFO:teuthology.orchestra.run.smithi101.stdout: "notif_pool": "a2.rgw.log:notif" 2023-07-28T23:59:42.014 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:59:42.015 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:59:42.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-28T23:59:42.017 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-28T23:59:42.064 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-28T23:59:42.065 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-28T23:59:42.107 INFO:teuthology.orchestra.run.smithi101.stderr:Sending period to new master zone 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433 2023-07-28T23:59:42.132 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-28T23:59:42.133 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:59:42.133 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 8, 2023-07-28T23:59:42.133 INFO:teuthology.orchestra.run.smithi101.stdout: "predecessor_uuid": "a0987484-ce37-4520-a564-54b7ea8c5251", 2023-07-28T23:59:42.133 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": [], 2023-07-28T23:59:42.134 INFO:teuthology.orchestra.run.smithi101.stdout: "period_map": { 2023-07-28T23:59:42.134 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-28T23:59:42.134 INFO:teuthology.orchestra.run.smithi101.stdout: "zonegroups": [ 2023-07-28T23:59:42.135 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:59:42.135 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "a", 2023-07-28T23:59:42.135 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a", 2023-07-28T23:59:42.136 INFO:teuthology.orchestra.run.smithi101.stdout: "api_name": "", 2023-07-28T23:59:42.136 INFO:teuthology.orchestra.run.smithi101.stdout: "is_master": true, 2023-07-28T23:59:42.136 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-28T23:59:42.137 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-28T23:59:42.137 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:59:42.137 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames": [], 2023-07-28T23:59:42.138 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames_s3website": [], 2023-07-28T23:59:42.139 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-28T23:59:42.139 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-28T23:59:42.139 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:59:42.140 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-28T23:59:42.140 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a1", 2023-07-28T23:59:42.140 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-28T23:59:42.140 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-28T23:59:42.141 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:59:42.141 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-28T23:59:42.141 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-28T23:59:42.141 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-28T23:59:42.142 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-28T23:59:42.142 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-28T23:59:42.142 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-28T23:59:42.142 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-28T23:59:42.143 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-28T23:59:42.143 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-28T23:59:42.143 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-28T23:59:42.144 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-28T23:59:42.144 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:59:42.144 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:59:42.144 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:59:42.144 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-28T23:59:42.145 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a2", 2023-07-28T23:59:42.145 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-28T23:59:42.145 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8001" 2023-07-28T23:59:42.145 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:59:42.145 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-28T23:59:42.145 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-28T23:59:42.145 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-28T23:59:42.146 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-28T23:59:42.146 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-28T23:59:42.146 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-28T23:59:42.146 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-28T23:59:42.146 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-28T23:59:42.146 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-28T23:59:42.146 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-28T23:59:42.147 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-28T23:59:42.147 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:59:42.147 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:59:42.147 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:59:42.147 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_targets": [ 2023-07-28T23:59:42.147 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:59:42.147 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default-placement", 2023-07-28T23:59:42.148 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [], 2023-07-28T23:59:42.148 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": [ 2023-07-28T23:59:42.148 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD" 2023-07-28T23:59:42.148 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:59:42.149 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:59:42.149 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:59:42.149 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "default-placement", 2023-07-28T23:59:42.149 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-28T23:59:42.149 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_policy": { 2023-07-28T23:59:42.149 INFO:teuthology.orchestra.run.smithi101.stdout: "groups": [] 2023-07-28T23:59:42.150 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:59:42.150 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled_features": [ 2023-07-28T23:59:42.150 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-28T23:59:42.150 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:59:42.150 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:59:42.150 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:59:42.151 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "b", 2023-07-28T23:59:42.151 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b", 2023-07-28T23:59:42.151 INFO:teuthology.orchestra.run.smithi101.stdout: "api_name": "", 2023-07-28T23:59:42.151 INFO:teuthology.orchestra.run.smithi101.stdout: "is_master": false, 2023-07-28T23:59:42.151 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-28T23:59:42.152 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-28T23:59:42.152 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:59:42.152 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames": [], 2023-07-28T23:59:42.152 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames_s3website": [], 2023-07-28T23:59:42.152 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-28T23:59:42.152 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-28T23:59:42.152 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:59:42.153 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-28T23:59:42.153 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b1", 2023-07-28T23:59:42.153 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-28T23:59:42.153 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-28T23:59:42.153 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:59:42.153 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-28T23:59:42.154 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-28T23:59:42.154 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-28T23:59:42.154 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-28T23:59:42.154 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-28T23:59:42.154 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-28T23:59:42.154 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-28T23:59:42.154 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-28T23:59:42.154 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-28T23:59:42.155 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-28T23:59:42.156 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-28T23:59:42.156 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:59:42.156 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:59:42.156 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:59:42.156 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-28T23:59:42.156 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b2", 2023-07-28T23:59:42.157 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-28T23:59:42.157 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8001" 2023-07-28T23:59:42.157 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:59:42.157 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-28T23:59:42.157 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-28T23:59:42.157 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-28T23:59:42.158 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-28T23:59:42.158 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-28T23:59:42.158 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-28T23:59:42.158 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-28T23:59:42.158 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-28T23:59:42.158 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-28T23:59:42.158 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-28T23:59:42.159 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-28T23:59:42.159 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:59:42.159 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:59:42.159 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:59:42.159 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_targets": [ 2023-07-28T23:59:42.160 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:59:42.160 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default-placement", 2023-07-28T23:59:42.160 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [], 2023-07-28T23:59:42.160 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": [ 2023-07-28T23:59:42.160 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD" 2023-07-28T23:59:42.160 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:59:42.161 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:59:42.161 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:59:42.161 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "default-placement", 2023-07-28T23:59:42.161 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-28T23:59:42.162 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_policy": { 2023-07-28T23:59:42.162 INFO:teuthology.orchestra.run.smithi101.stdout: "groups": [] 2023-07-28T23:59:42.162 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:59:42.162 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled_features": [ 2023-07-28T23:59:42.162 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-28T23:59:42.163 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:59:42.163 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:59:42.163 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-28T23:59:42.163 INFO:teuthology.orchestra.run.smithi101.stdout: "short_zone_ids": [ 2023-07-28T23:59:42.163 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:59:42.163 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-28T23:59:42.163 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 2896287231 2023-07-28T23:59:42.164 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:59:42.164 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:59:42.164 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-28T23:59:42.164 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 815606810 2023-07-28T23:59:42.164 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:59:42.164 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:59:42.165 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-28T23:59:42.165 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 2009171363 2023-07-28T23:59:42.165 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:59:42.165 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-28T23:59:42.165 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-28T23:59:42.165 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 3105190295 2023-07-28T23:59:42.166 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:59:42.166 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-28T23:59:42.166 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:59:42.166 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zonegroup": "a", 2023-07-28T23:59:42.166 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-28T23:59:42.166 INFO:teuthology.orchestra.run.smithi101.stdout: "period_config": { 2023-07-28T23:59:42.166 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_quota": { 2023-07-28T23:59:42.167 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-28T23:59:42.167 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-28T23:59:42.167 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-28T23:59:42.167 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-28T23:59:42.167 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-28T23:59:42.167 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:59:42.168 INFO:teuthology.orchestra.run.smithi101.stdout: "user_quota": { 2023-07-28T23:59:42.168 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-28T23:59:42.168 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-28T23:59:42.168 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-28T23:59:42.168 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-28T23:59:42.168 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-28T23:59:42.168 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:59:42.169 INFO:teuthology.orchestra.run.smithi101.stdout: "user_ratelimit": { 2023-07-28T23:59:42.169 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-28T23:59:42.169 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-28T23:59:42.169 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-28T23:59:42.170 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-28T23:59:42.170 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-28T23:59:42.170 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:59:42.170 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_ratelimit": { 2023-07-28T23:59:42.170 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-28T23:59:42.170 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-28T23:59:42.171 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-28T23:59:42.171 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-28T23:59:42.171 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-28T23:59:42.171 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:59:42.171 INFO:teuthology.orchestra.run.smithi101.stdout: "anonymous_ratelimit": { 2023-07-28T23:59:42.171 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-28T23:59:42.171 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-28T23:59:42.172 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-28T23:59:42.172 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-28T23:59:42.172 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-28T23:59:42.172 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-28T23:59:42.172 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-28T23:59:42.173 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-28T23:59:42.173 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_name": "test-realm", 2023-07-28T23:59:42.173 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-28T23:59:42.173 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-28T23:59:42.173 INFO:rgw_multi.tests:Set redirect_zone zone a2 to "" 2023-07-29T00:00:42.174 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_zone_redirect ... ok 2023-07-29T00:00:42.176 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:00:42.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:00:42.177 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:00:42.231 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:00:42.231 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:00:42.264 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T00:00:42.276 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T00:00:42.277 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:42.277 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:00:42.277 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:46:09.233742Z" 2023-07-29T00:00:42.277 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:42.277 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:42.277 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588579.740560_402.1", 2023-07-29T00:00:42.278 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:56:19.740560Z" 2023-07-29T00:00:42.278 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:42.278 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:42.278 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:00:42.279 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T00:00:42.279 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:42.279 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:42.279 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:00:42.279 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T00:00:42.279 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:42.280 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T00:00:42.281 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690587969.233742_99.1', 1: '1_1690588579.740560_402.1', 2: '', 3: ''} 2023-07-29T00:00:42.281 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T00:00:42.281 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:00:42.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:00:42.282 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:00:42.331 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:00:42.331 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:00:42.358 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:00:42.353+0000 7f1b02211880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:00:42.359 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:00:42.354+0000 7f1b02211880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:00:42.370 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:00:42.371 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:00:42.371 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:00:42.371 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:00:42.371 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:00:42.371 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:00:42.371 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:00:42.372 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:42.372 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:00:42.372 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:42.372 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:00:42.372 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:00:42.372 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:00:42.373 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:00:42.373 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:00:42.373 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:00:42.373 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:00:42.373 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:00:42.374 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:00:42.374 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:42.374 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:42.374 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:42.374 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:00:42.374 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:00:42.375 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:00:42.375 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588579.740560_402.1", 2023-07-29T00:00:42.375 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:00:42.375 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:00:42.375 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:00:42.375 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:56:19.740560Z", 2023-07-29T00:00:42.376 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:00:42.376 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:42.376 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:42.376 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:42.376 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:00:42.376 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:00:42.376 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:00:42.377 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:00:42.377 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:00:42.377 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:00:42.377 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:00:42.377 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:00:42.377 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:00:42.377 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:42.378 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:42.378 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:42.378 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:00:42.378 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:00:42.378 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:00:42.378 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:00:42.379 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:00:42.379 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:00:42.379 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:00:42.379 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:00:42.379 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:00:42.379 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:42.379 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:42.380 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:42.380 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:42.380 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:00:42.380 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:00:42.380 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:00:42.380 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:42.380 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:00:42Z" 2023-07-29T00:00:42.381 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:00:42.381 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588579.740560_402.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:56:19.740560Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:00:42Z" } 2023-07-29T00:00:42.382 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588579.740560_402.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:56:19.740560Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:00:42.382 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588579.740560_402.1', 2: '', 3: ''} 2023-07-29T00:00:42.383 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588579.740560_402.1', 2: '', 3: ''} 2023-07-29T00:00:42.383 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T00:00:42.383 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'RX82IHH7C5RUJ54D', '--secret', 'InTLvCQsePIbOVhlgLo1AkTNflWbmSsQ', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:00:42.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'RX82IHH7C5RUJ54D', '--secret', 'InTLvCQsePIbOVhlgLo1AkTNflWbmSsQ', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:00:42.384 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup a --rgw-realm test-realm --access-key RX82IHH7C5RUJ54D --secret InTLvCQsePIbOVhlgLo1AkTNflWbmSsQ --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:00:42.432 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:00:42.432 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:00:42.446 INFO:teuthology.orchestra.run.smithi101.stderr:NOTICE: set zone's realm_id=72b68b73-6728-4b14-854a-54780b6610bf 2023-07-29T00:00:42.461 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:00:42.461 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "22aff295-48b7-4f1b-a473-c9d2abb2a09a", 2023-07-29T00:00:42.461 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "remove", 2023-07-29T00:00:42.461 INFO:teuthology.orchestra.run.smithi101.stdout: "domain_root": "remove.rgw.meta:root", 2023-07-29T00:00:42.461 INFO:teuthology.orchestra.run.smithi101.stdout: "control_pool": "remove.rgw.control", 2023-07-29T00:00:42.462 INFO:teuthology.orchestra.run.smithi101.stdout: "gc_pool": "remove.rgw.log:gc", 2023-07-29T00:00:42.462 INFO:teuthology.orchestra.run.smithi101.stdout: "lc_pool": "remove.rgw.log:lc", 2023-07-29T00:00:42.462 INFO:teuthology.orchestra.run.smithi101.stdout: "log_pool": "remove.rgw.log", 2023-07-29T00:00:42.462 INFO:teuthology.orchestra.run.smithi101.stdout: "intent_log_pool": "remove.rgw.log:intent", 2023-07-29T00:00:42.462 INFO:teuthology.orchestra.run.smithi101.stdout: "usage_log_pool": "remove.rgw.log:usage", 2023-07-29T00:00:42.462 INFO:teuthology.orchestra.run.smithi101.stdout: "roles_pool": "remove.rgw.meta:roles", 2023-07-29T00:00:42.463 INFO:teuthology.orchestra.run.smithi101.stdout: "reshard_pool": "remove.rgw.log:reshard", 2023-07-29T00:00:42.463 INFO:teuthology.orchestra.run.smithi101.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2023-07-29T00:00:42.463 INFO:teuthology.orchestra.run.smithi101.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2023-07-29T00:00:42.463 INFO:teuthology.orchestra.run.smithi101.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2023-07-29T00:00:42.463 INFO:teuthology.orchestra.run.smithi101.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2023-07-29T00:00:42.463 INFO:teuthology.orchestra.run.smithi101.stdout: "otp_pool": "remove.rgw.otp", 2023-07-29T00:00:42.463 INFO:teuthology.orchestra.run.smithi101.stdout: "system_key": { 2023-07-29T00:00:42.464 INFO:teuthology.orchestra.run.smithi101.stdout: "access_key": "RX82IHH7C5RUJ54D", 2023-07-29T00:00:42.464 INFO:teuthology.orchestra.run.smithi101.stdout: "secret_key": "InTLvCQsePIbOVhlgLo1AkTNflWbmSsQ" 2023-07-29T00:00:42.464 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:42.464 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_pools": [ 2023-07-29T00:00:42.464 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:42.464 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "default-placement", 2023-07-29T00:00:42.465 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:00:42.465 INFO:teuthology.orchestra.run.smithi101.stdout: "index_pool": "remove.rgw.buckets.index", 2023-07-29T00:00:42.465 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": { 2023-07-29T00:00:42.465 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD": { 2023-07-29T00:00:42.465 INFO:teuthology.orchestra.run.smithi101.stdout: "data_pool": "remove.rgw.buckets.data" 2023-07-29T00:00:42.465 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:42.465 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:42.466 INFO:teuthology.orchestra.run.smithi101.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2023-07-29T00:00:42.466 INFO:teuthology.orchestra.run.smithi101.stdout: "index_type": 0, 2023-07-29T00:00:42.466 INFO:teuthology.orchestra.run.smithi101.stdout: "inline_data": true 2023-07-29T00:00:42.466 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:42.466 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:42.466 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:42.466 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T00:00:42.467 INFO:teuthology.orchestra.run.smithi101.stdout: "notif_pool": "remove.rgw.log:notif" 2023-07-29T00:00:42.467 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:00:42.468 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:00:42.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:00:42.469 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone remove --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:00:42.513 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:00:42.514 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:00:42.526 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:00:42.520+0000 7fb7c6614880 0 period (2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 does not have zone 22aff295-48b7-4f1b-a473-c9d2abb2a09a configured 2023-07-29T00:00:49.412 INFO:teuthology.orchestra.run.smithi101.stderr:Sending period to new master zone 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433 2023-07-29T00:00:49.437 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:00:49.437 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:00:49.437 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 9, 2023-07-29T00:00:49.437 INFO:teuthology.orchestra.run.smithi101.stdout: "predecessor_uuid": "a0987484-ce37-4520-a564-54b7ea8c5251", 2023-07-29T00:00:49.438 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": [], 2023-07-29T00:00:49.438 INFO:teuthology.orchestra.run.smithi101.stdout: "period_map": { 2023-07-29T00:00:49.438 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:00:49.438 INFO:teuthology.orchestra.run.smithi101.stdout: "zonegroups": [ 2023-07-29T00:00:49.438 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:49.438 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "a", 2023-07-29T00:00:49.438 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a", 2023-07-29T00:00:49.439 INFO:teuthology.orchestra.run.smithi101.stdout: "api_name": "", 2023-07-29T00:00:49.439 INFO:teuthology.orchestra.run.smithi101.stdout: "is_master": true, 2023-07-29T00:00:49.439 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T00:00:49.439 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-29T00:00:49.439 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:49.439 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames": [], 2023-07-29T00:00:49.439 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames_s3website": [], 2023-07-29T00:00:49.440 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T00:00:49.440 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T00:00:49.440 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:49.440 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "22aff295-48b7-4f1b-a473-c9d2abb2a09a", 2023-07-29T00:00:49.440 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "remove", 2023-07-29T00:00:49.440 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [], 2023-07-29T00:00:49.441 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T00:00:49.441 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T00:00:49.441 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T00:00:49.441 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T00:00:49.441 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T00:00:49.441 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T00:00:49.441 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T00:00:49.442 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T00:00:49.442 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T00:00:49.442 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T00:00:49.442 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T00:00:49.442 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:49.442 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:49.443 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:49.443 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T00:00:49.443 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a1", 2023-07-29T00:00:49.443 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T00:00:49.443 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-29T00:00:49.443 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:49.443 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T00:00:49.444 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T00:00:49.444 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T00:00:49.444 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T00:00:49.444 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T00:00:49.444 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T00:00:49.444 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T00:00:49.444 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T00:00:49.445 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T00:00:49.445 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T00:00:49.445 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T00:00:49.445 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:49.445 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:49.445 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:49.446 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-29T00:00:49.446 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a2", 2023-07-29T00:00:49.446 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T00:00:49.446 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8001" 2023-07-29T00:00:49.446 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:49.446 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T00:00:49.446 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T00:00:49.447 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T00:00:49.447 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T00:00:49.447 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T00:00:49.447 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T00:00:49.447 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T00:00:49.447 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T00:00:49.448 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T00:00:49.448 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T00:00:49.448 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T00:00:49.448 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:49.448 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:49.448 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:49.448 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_targets": [ 2023-07-29T00:00:49.449 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:49.449 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default-placement", 2023-07-29T00:00:49.449 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [], 2023-07-29T00:00:49.449 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": [ 2023-07-29T00:00:49.449 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD" 2023-07-29T00:00:49.449 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:49.449 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:49.450 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:49.450 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "default-placement", 2023-07-29T00:00:49.450 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T00:00:49.450 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_policy": { 2023-07-29T00:00:49.450 INFO:teuthology.orchestra.run.smithi101.stdout: "groups": [] 2023-07-29T00:00:49.450 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:49.450 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled_features": [ 2023-07-29T00:00:49.451 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T00:00:49.451 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:49.451 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:49.451 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:49.451 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "b", 2023-07-29T00:00:49.451 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b", 2023-07-29T00:00:49.452 INFO:teuthology.orchestra.run.smithi101.stdout: "api_name": "", 2023-07-29T00:00:49.452 INFO:teuthology.orchestra.run.smithi101.stdout: "is_master": false, 2023-07-29T00:00:49.452 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T00:00:49.452 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-29T00:00:49.452 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:49.452 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames": [], 2023-07-29T00:00:49.452 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames_s3website": [], 2023-07-29T00:00:49.453 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T00:00:49.453 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T00:00:49.453 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:49.453 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T00:00:49.453 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b1", 2023-07-29T00:00:49.453 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T00:00:49.453 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-29T00:00:49.454 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:49.454 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T00:00:49.454 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T00:00:49.454 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T00:00:49.454 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T00:00:49.454 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T00:00:49.454 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T00:00:49.455 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T00:00:49.455 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T00:00:49.455 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T00:00:49.455 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T00:00:49.455 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T00:00:49.455 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:49.455 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:49.456 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:49.456 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-29T00:00:49.456 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b2", 2023-07-29T00:00:49.456 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T00:00:49.456 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8001" 2023-07-29T00:00:49.456 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:49.456 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T00:00:49.457 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T00:00:49.457 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T00:00:49.457 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T00:00:49.457 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T00:00:49.457 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T00:00:49.457 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T00:00:49.457 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T00:00:49.458 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T00:00:49.458 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T00:00:49.458 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T00:00:49.458 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:49.458 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:49.458 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:49.458 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_targets": [ 2023-07-29T00:00:49.459 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:49.459 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default-placement", 2023-07-29T00:00:49.459 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [], 2023-07-29T00:00:49.459 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": [ 2023-07-29T00:00:49.459 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD" 2023-07-29T00:00:49.459 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:49.460 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:49.460 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:49.460 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "default-placement", 2023-07-29T00:00:49.460 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T00:00:49.460 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_policy": { 2023-07-29T00:00:49.460 INFO:teuthology.orchestra.run.smithi101.stdout: "groups": [] 2023-07-29T00:00:49.460 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:49.461 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled_features": [ 2023-07-29T00:00:49.461 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T00:00:49.461 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:49.461 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:49.461 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:49.461 INFO:teuthology.orchestra.run.smithi101.stdout: "short_zone_ids": [ 2023-07-29T00:00:49.461 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:49.462 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "22aff295-48b7-4f1b-a473-c9d2abb2a09a", 2023-07-29T00:00:49.462 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 769444612 2023-07-29T00:00:49.462 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:49.462 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:49.462 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T00:00:49.462 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 2896287231 2023-07-29T00:00:49.462 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:49.463 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:49.463 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-29T00:00:49.463 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 815606810 2023-07-29T00:00:49.463 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:49.463 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:49.463 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T00:00:49.463 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 2009171363 2023-07-29T00:00:49.464 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:49.464 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:49.464 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-29T00:00:49.464 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 3105190295 2023-07-29T00:00:49.464 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:49.464 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:49.465 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:49.465 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zonegroup": "a", 2023-07-29T00:00:49.465 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T00:00:49.465 INFO:teuthology.orchestra.run.smithi101.stdout: "period_config": { 2023-07-29T00:00:49.465 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_quota": { 2023-07-29T00:00:49.465 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-29T00:00:49.465 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-29T00:00:49.466 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-29T00:00:49.466 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-29T00:00:49.466 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-29T00:00:49.466 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:49.466 INFO:teuthology.orchestra.run.smithi101.stdout: "user_quota": { 2023-07-29T00:00:49.466 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-29T00:00:49.467 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-29T00:00:49.467 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-29T00:00:49.467 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-29T00:00:49.467 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-29T00:00:49.467 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:49.467 INFO:teuthology.orchestra.run.smithi101.stdout: "user_ratelimit": { 2023-07-29T00:00:49.467 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-29T00:00:49.468 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-29T00:00:49.468 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-29T00:00:49.468 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-29T00:00:49.468 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-29T00:00:49.468 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:49.468 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_ratelimit": { 2023-07-29T00:00:49.468 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-29T00:00:49.469 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-29T00:00:49.469 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-29T00:00:49.469 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-29T00:00:49.469 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-29T00:00:49.469 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:49.469 INFO:teuthology.orchestra.run.smithi101.stdout: "anonymous_ratelimit": { 2023-07-29T00:00:49.469 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-29T00:00:49.470 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-29T00:00:49.470 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-29T00:00:49.470 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-29T00:00:49.470 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-29T00:00:49.470 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:49.470 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:49.471 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T00:00:49.471 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_name": "test-realm", 2023-07-29T00:00:49.472 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:00:49.472 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:00:49.472 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:00:49.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:00:49.473 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup a --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:00:49.533 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:00:49.534 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:00:49.574 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:00:49.574 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "a", 2023-07-29T00:00:49.574 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a", 2023-07-29T00:00:49.575 INFO:teuthology.orchestra.run.smithi101.stdout: "api_name": "", 2023-07-29T00:00:49.575 INFO:teuthology.orchestra.run.smithi101.stdout: "is_master": true, 2023-07-29T00:00:49.575 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T00:00:49.575 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-29T00:00:49.575 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:49.576 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames": [], 2023-07-29T00:00:49.576 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames_s3website": [], 2023-07-29T00:00:49.576 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T00:00:49.576 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T00:00:49.576 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:49.577 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T00:00:49.577 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a1", 2023-07-29T00:00:49.577 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T00:00:49.578 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-29T00:00:49.578 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:49.578 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T00:00:49.578 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T00:00:49.579 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T00:00:49.579 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T00:00:49.579 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T00:00:49.580 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T00:00:49.580 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T00:00:49.580 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T00:00:49.581 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T00:00:49.581 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T00:00:49.582 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T00:00:49.582 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:49.582 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:49.582 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:49.582 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-29T00:00:49.582 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a2", 2023-07-29T00:00:49.582 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T00:00:49.583 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8001" 2023-07-29T00:00:49.583 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:49.583 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T00:00:49.583 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T00:00:49.583 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T00:00:49.583 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T00:00:49.584 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T00:00:49.584 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T00:00:49.584 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T00:00:49.584 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T00:00:49.584 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T00:00:49.585 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T00:00:49.585 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T00:00:49.585 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:49.586 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:49.586 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:49.586 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_targets": [ 2023-07-29T00:00:49.587 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:49.587 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default-placement", 2023-07-29T00:00:49.587 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [], 2023-07-29T00:00:49.587 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": [ 2023-07-29T00:00:49.588 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD" 2023-07-29T00:00:49.588 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:49.588 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:49.588 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:49.588 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "default-placement", 2023-07-29T00:00:49.589 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T00:00:49.589 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_policy": { 2023-07-29T00:00:49.589 INFO:teuthology.orchestra.run.smithi101.stdout: "groups": [] 2023-07-29T00:00:49.589 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:49.589 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled_features": [ 2023-07-29T00:00:49.589 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T00:00:49.590 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:49.590 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:00:49.591 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:00:49.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:00:49.593 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup a --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:00:49.660 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:00:49.661 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:00:49.702 INFO:teuthology.orchestra.run.smithi101.stderr:zone name remove not found in zonegroup a 2023-07-29T00:00:49.709 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-07-29T00:00:49.710 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:00:49.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:00:49.711 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:00:49.782 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:00:49.782 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:00:49.837 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:00:49.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:00:49.838 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:00:49.898 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:00:49.898 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:00:49.918 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:00:49.912+0000 7faed3936880 0 WARNING: can't generate connection for zone 22aff295-48b7-4f1b-a473-c9d2abb2a09a id remove: no endpoints defined 2023-07-29T00:00:49.965 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:00:49.966 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:00:49.966 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 10, 2023-07-29T00:00:49.966 INFO:teuthology.orchestra.run.smithi101.stdout: "predecessor_uuid": "a0987484-ce37-4520-a564-54b7ea8c5251", 2023-07-29T00:00:49.966 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": [], 2023-07-29T00:00:49.966 INFO:teuthology.orchestra.run.smithi101.stdout: "period_map": { 2023-07-29T00:00:49.967 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:00:49.967 INFO:teuthology.orchestra.run.smithi101.stdout: "zonegroups": [ 2023-07-29T00:00:49.967 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:49.967 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "a", 2023-07-29T00:00:49.967 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a", 2023-07-29T00:00:49.968 INFO:teuthology.orchestra.run.smithi101.stdout: "api_name": "", 2023-07-29T00:00:49.968 INFO:teuthology.orchestra.run.smithi101.stdout: "is_master": true, 2023-07-29T00:00:49.968 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T00:00:49.968 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-29T00:00:49.968 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:49.969 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames": [], 2023-07-29T00:00:49.969 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames_s3website": [], 2023-07-29T00:00:49.969 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T00:00:49.969 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T00:00:49.969 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:49.970 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T00:00:49.970 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a1", 2023-07-29T00:00:49.970 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T00:00:49.970 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-29T00:00:49.970 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:49.970 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T00:00:49.970 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T00:00:49.971 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T00:00:49.971 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T00:00:49.971 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T00:00:49.971 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T00:00:49.971 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T00:00:49.971 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T00:00:49.972 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T00:00:49.972 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T00:00:49.972 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T00:00:49.972 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:49.972 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:49.972 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:49.973 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-29T00:00:49.973 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a2", 2023-07-29T00:00:49.973 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T00:00:49.973 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8001" 2023-07-29T00:00:49.973 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:49.973 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T00:00:49.973 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T00:00:49.974 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T00:00:49.974 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T00:00:49.974 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T00:00:49.974 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T00:00:49.974 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T00:00:49.974 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T00:00:49.974 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T00:00:49.975 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T00:00:49.975 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T00:00:49.975 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:49.975 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:49.975 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:49.975 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_targets": [ 2023-07-29T00:00:49.976 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:49.976 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default-placement", 2023-07-29T00:00:49.976 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [], 2023-07-29T00:00:49.976 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": [ 2023-07-29T00:00:49.976 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD" 2023-07-29T00:00:49.976 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:49.976 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:49.977 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:49.977 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "default-placement", 2023-07-29T00:00:49.977 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T00:00:49.977 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_policy": { 2023-07-29T00:00:49.977 INFO:teuthology.orchestra.run.smithi101.stdout: "groups": [] 2023-07-29T00:00:49.977 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:49.977 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled_features": [ 2023-07-29T00:00:49.978 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T00:00:49.978 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:49.978 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:49.978 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:49.978 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "b", 2023-07-29T00:00:49.978 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b", 2023-07-29T00:00:49.979 INFO:teuthology.orchestra.run.smithi101.stdout: "api_name": "", 2023-07-29T00:00:49.979 INFO:teuthology.orchestra.run.smithi101.stdout: "is_master": false, 2023-07-29T00:00:49.979 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T00:00:49.979 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-29T00:00:49.979 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:49.979 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames": [], 2023-07-29T00:00:49.979 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames_s3website": [], 2023-07-29T00:00:49.980 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T00:00:49.980 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T00:00:49.980 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:49.980 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T00:00:49.980 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b1", 2023-07-29T00:00:49.980 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T00:00:49.981 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-29T00:00:49.981 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:49.981 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T00:00:49.981 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T00:00:49.981 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T00:00:49.981 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T00:00:49.981 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T00:00:49.982 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T00:00:49.982 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T00:00:49.982 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T00:00:49.982 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T00:00:49.982 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T00:00:49.983 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T00:00:49.983 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:49.984 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:49.984 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:49.984 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-29T00:00:49.984 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b2", 2023-07-29T00:00:49.984 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T00:00:49.985 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8001" 2023-07-29T00:00:49.985 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:49.985 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T00:00:49.985 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T00:00:49.985 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T00:00:49.985 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T00:00:49.985 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T00:00:49.986 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T00:00:49.986 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T00:00:49.986 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T00:00:49.986 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T00:00:49.986 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T00:00:49.986 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T00:00:49.987 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:49.987 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:49.987 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:49.987 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_targets": [ 2023-07-29T00:00:49.987 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:49.987 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default-placement", 2023-07-29T00:00:49.987 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [], 2023-07-29T00:00:49.988 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": [ 2023-07-29T00:00:49.988 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD" 2023-07-29T00:00:49.988 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:49.988 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:49.988 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:49.988 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "default-placement", 2023-07-29T00:00:49.989 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T00:00:49.989 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_policy": { 2023-07-29T00:00:49.989 INFO:teuthology.orchestra.run.smithi101.stdout: "groups": [] 2023-07-29T00:00:49.989 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:49.989 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled_features": [ 2023-07-29T00:00:49.989 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T00:00:49.989 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:49.990 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:49.990 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:49.990 INFO:teuthology.orchestra.run.smithi101.stdout: "short_zone_ids": [ 2023-07-29T00:00:49.990 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:49.990 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T00:00:49.990 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 2896287231 2023-07-29T00:00:49.990 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:49.991 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:49.991 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-29T00:00:49.991 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 815606810 2023-07-29T00:00:49.991 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:49.991 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:49.991 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T00:00:49.991 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 2009171363 2023-07-29T00:00:49.992 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:49.992 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:49.992 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-29T00:00:49.992 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 3105190295 2023-07-29T00:00:49.992 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:49.992 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:49.993 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:49.993 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zonegroup": "a", 2023-07-29T00:00:49.993 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T00:00:49.993 INFO:teuthology.orchestra.run.smithi101.stdout: "period_config": { 2023-07-29T00:00:49.993 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_quota": { 2023-07-29T00:00:49.993 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-29T00:00:49.993 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-29T00:00:49.994 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-29T00:00:49.994 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-29T00:00:49.994 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-29T00:00:49.994 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:49.994 INFO:teuthology.orchestra.run.smithi101.stdout: "user_quota": { 2023-07-29T00:00:49.994 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-29T00:00:49.994 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-29T00:00:49.995 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-29T00:00:49.995 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-29T00:00:49.995 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-29T00:00:49.995 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:49.995 INFO:teuthology.orchestra.run.smithi101.stdout: "user_ratelimit": { 2023-07-29T00:00:49.995 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-29T00:00:49.996 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-29T00:00:49.996 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-29T00:00:49.996 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-29T00:00:49.996 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-29T00:00:49.996 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:49.996 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_ratelimit": { 2023-07-29T00:00:49.996 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-29T00:00:49.997 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-29T00:00:49.997 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-29T00:00:49.997 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-29T00:00:49.997 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-29T00:00:49.997 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:49.997 INFO:teuthology.orchestra.run.smithi101.stdout: "anonymous_ratelimit": { 2023-07-29T00:00:49.997 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-29T00:00:49.998 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-29T00:00:49.998 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-29T00:00:49.998 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-29T00:00:49.998 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-29T00:00:49.998 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:49.998 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:49.999 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T00:00:49.999 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_name": "test-realm", 2023-07-29T00:00:49.999 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:00:49.999 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:00:49.999 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zonegroup_remove ... ok 2023-07-29T00:00:50.000 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:00:50.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:00:50.001 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:00:50.046 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:00:50.046 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:00:50.067 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:00:50.068 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "105def2b-98ab-4e12-b71b-c37abb2015e1", 2023-07-29T00:00:50.068 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "remove_zg", 2023-07-29T00:00:50.068 INFO:teuthology.orchestra.run.smithi101.stdout: "api_name": "remove_zg", 2023-07-29T00:00:50.068 INFO:teuthology.orchestra.run.smithi101.stdout: "is_master": false, 2023-07-29T00:00:50.068 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [], 2023-07-29T00:00:50.069 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames": [], 2023-07-29T00:00:50.069 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames_s3website": [], 2023-07-29T00:00:50.069 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "", 2023-07-29T00:00:50.069 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [], 2023-07-29T00:00:50.069 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_targets": [ 2023-07-29T00:00:50.070 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:50.070 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default-placement", 2023-07-29T00:00:50.070 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [], 2023-07-29T00:00:50.070 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": [] 2023-07-29T00:00:50.070 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:50.070 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:50.070 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "default-placement", 2023-07-29T00:00:50.071 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T00:00:50.071 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_policy": { 2023-07-29T00:00:50.071 INFO:teuthology.orchestra.run.smithi101.stdout: "groups": [] 2023-07-29T00:00:50.071 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:50.071 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled_features": [ 2023-07-29T00:00:50.071 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T00:00:50.072 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:50.072 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:00:50.073 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:00:50.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:00:50.074 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:00:50.124 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:00:50.124 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:00:50.189 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:00:50.183+0000 7f700b213880 0 NOTICE: promoted remove as new master_zone of zonegroup remove_zg 2023-07-29T00:00:50.197 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:00:50.197 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "951f39df-ef16-499d-8585-2b2a92ac44b2", 2023-07-29T00:00:50.197 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "remove", 2023-07-29T00:00:50.198 INFO:teuthology.orchestra.run.smithi101.stdout: "domain_root": "remove.rgw.meta:root", 2023-07-29T00:00:50.198 INFO:teuthology.orchestra.run.smithi101.stdout: "control_pool": "remove.rgw.control", 2023-07-29T00:00:50.198 INFO:teuthology.orchestra.run.smithi101.stdout: "gc_pool": "remove.rgw.log:gc", 2023-07-29T00:00:50.198 INFO:teuthology.orchestra.run.smithi101.stdout: "lc_pool": "remove.rgw.log:lc", 2023-07-29T00:00:50.198 INFO:teuthology.orchestra.run.smithi101.stdout: "log_pool": "remove.rgw.log", 2023-07-29T00:00:50.198 INFO:teuthology.orchestra.run.smithi101.stdout: "intent_log_pool": "remove.rgw.log:intent", 2023-07-29T00:00:50.199 INFO:teuthology.orchestra.run.smithi101.stdout: "usage_log_pool": "remove.rgw.log:usage", 2023-07-29T00:00:50.199 INFO:teuthology.orchestra.run.smithi101.stdout: "roles_pool": "remove.rgw.meta:roles", 2023-07-29T00:00:50.199 INFO:teuthology.orchestra.run.smithi101.stdout: "reshard_pool": "remove.rgw.log:reshard", 2023-07-29T00:00:50.199 INFO:teuthology.orchestra.run.smithi101.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2023-07-29T00:00:50.199 INFO:teuthology.orchestra.run.smithi101.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2023-07-29T00:00:50.199 INFO:teuthology.orchestra.run.smithi101.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2023-07-29T00:00:50.200 INFO:teuthology.orchestra.run.smithi101.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2023-07-29T00:00:50.200 INFO:teuthology.orchestra.run.smithi101.stdout: "otp_pool": "remove.rgw.otp", 2023-07-29T00:00:50.200 INFO:teuthology.orchestra.run.smithi101.stdout: "system_key": { 2023-07-29T00:00:50.200 INFO:teuthology.orchestra.run.smithi101.stdout: "access_key": "", 2023-07-29T00:00:50.201 INFO:teuthology.orchestra.run.smithi101.stdout: "secret_key": "" 2023-07-29T00:00:50.201 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:50.201 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_pools": [ 2023-07-29T00:00:50.201 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:50.201 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "default-placement", 2023-07-29T00:00:50.201 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:00:50.202 INFO:teuthology.orchestra.run.smithi101.stdout: "index_pool": "remove.rgw.buckets.index", 2023-07-29T00:00:50.202 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": { 2023-07-29T00:00:50.202 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD": { 2023-07-29T00:00:50.202 INFO:teuthology.orchestra.run.smithi101.stdout: "data_pool": "remove.rgw.buckets.data" 2023-07-29T00:00:50.202 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:50.202 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:50.203 INFO:teuthology.orchestra.run.smithi101.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2023-07-29T00:00:50.203 INFO:teuthology.orchestra.run.smithi101.stdout: "index_type": 0, 2023-07-29T00:00:50.203 INFO:teuthology.orchestra.run.smithi101.stdout: "inline_data": true 2023-07-29T00:00:50.203 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:50.203 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:50.203 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:50.203 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T00:00:50.204 INFO:teuthology.orchestra.run.smithi101.stdout: "notif_pool": "remove.rgw.log:notif" 2023-07-29T00:00:50.204 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:00:50.205 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:00:50.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:00:50.206 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:00:50.280 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:00:50.281 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:00:50.408 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:00:50.409 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:00:50.409 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 11, 2023-07-29T00:00:50.409 INFO:teuthology.orchestra.run.smithi101.stdout: "predecessor_uuid": "a0987484-ce37-4520-a564-54b7ea8c5251", 2023-07-29T00:00:50.410 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": [], 2023-07-29T00:00:50.410 INFO:teuthology.orchestra.run.smithi101.stdout: "period_map": { 2023-07-29T00:00:50.410 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:00:50.410 INFO:teuthology.orchestra.run.smithi101.stdout: "zonegroups": [ 2023-07-29T00:00:50.410 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:50.410 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "105def2b-98ab-4e12-b71b-c37abb2015e1", 2023-07-29T00:00:50.410 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "remove_zg", 2023-07-29T00:00:50.411 INFO:teuthology.orchestra.run.smithi101.stdout: "api_name": "remove_zg", 2023-07-29T00:00:50.411 INFO:teuthology.orchestra.run.smithi101.stdout: "is_master": false, 2023-07-29T00:00:50.411 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [], 2023-07-29T00:00:50.411 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames": [], 2023-07-29T00:00:50.411 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames_s3website": [], 2023-07-29T00:00:50.411 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "951f39df-ef16-499d-8585-2b2a92ac44b2", 2023-07-29T00:00:50.411 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T00:00:50.412 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:50.412 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "951f39df-ef16-499d-8585-2b2a92ac44b2", 2023-07-29T00:00:50.412 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "remove", 2023-07-29T00:00:50.412 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [], 2023-07-29T00:00:50.412 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T00:00:50.412 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": false, 2023-07-29T00:00:50.413 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T00:00:50.413 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T00:00:50.413 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T00:00:50.413 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T00:00:50.413 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T00:00:50.413 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T00:00:50.413 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T00:00:50.414 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T00:00:50.414 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T00:00:50.414 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:50.414 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:50.414 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:50.414 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_targets": [ 2023-07-29T00:00:50.414 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:50.415 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default-placement", 2023-07-29T00:00:50.415 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [], 2023-07-29T00:00:50.415 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": [ 2023-07-29T00:00:50.415 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD" 2023-07-29T00:00:50.415 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:50.415 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:50.415 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:50.416 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "default-placement", 2023-07-29T00:00:50.416 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T00:00:50.416 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_policy": { 2023-07-29T00:00:50.416 INFO:teuthology.orchestra.run.smithi101.stdout: "groups": [] 2023-07-29T00:00:50.416 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:50.416 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled_features": [ 2023-07-29T00:00:50.417 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T00:00:50.417 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:50.417 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:50.417 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:50.417 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "a", 2023-07-29T00:00:50.417 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a", 2023-07-29T00:00:50.417 INFO:teuthology.orchestra.run.smithi101.stdout: "api_name": "", 2023-07-29T00:00:50.418 INFO:teuthology.orchestra.run.smithi101.stdout: "is_master": true, 2023-07-29T00:00:50.418 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T00:00:50.418 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-29T00:00:50.418 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:50.418 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames": [], 2023-07-29T00:00:50.419 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames_s3website": [], 2023-07-29T00:00:50.419 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T00:00:50.419 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T00:00:50.420 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:50.420 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T00:00:50.420 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a1", 2023-07-29T00:00:50.421 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T00:00:50.421 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-29T00:00:50.421 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:50.422 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T00:00:50.422 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T00:00:50.422 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T00:00:50.423 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T00:00:50.423 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T00:00:50.423 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T00:00:50.423 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T00:00:50.423 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T00:00:50.424 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T00:00:50.424 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T00:00:50.424 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T00:00:50.425 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:50.425 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:50.425 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:50.425 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-29T00:00:50.426 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a2", 2023-07-29T00:00:50.426 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T00:00:50.426 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8001" 2023-07-29T00:00:50.426 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:50.426 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T00:00:50.426 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T00:00:50.427 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T00:00:50.427 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T00:00:50.427 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T00:00:50.427 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T00:00:50.427 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T00:00:50.427 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T00:00:50.428 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T00:00:50.428 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T00:00:50.428 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T00:00:50.428 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:50.428 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:50.428 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:50.429 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_targets": [ 2023-07-29T00:00:50.429 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:50.429 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default-placement", 2023-07-29T00:00:50.429 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [], 2023-07-29T00:00:50.429 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": [ 2023-07-29T00:00:50.429 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD" 2023-07-29T00:00:50.429 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:50.430 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:50.430 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:50.430 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "default-placement", 2023-07-29T00:00:50.430 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T00:00:50.430 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_policy": { 2023-07-29T00:00:50.430 INFO:teuthology.orchestra.run.smithi101.stdout: "groups": [] 2023-07-29T00:00:50.430 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:50.431 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled_features": [ 2023-07-29T00:00:50.431 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T00:00:50.431 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:50.432 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:50.432 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:50.432 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "b", 2023-07-29T00:00:50.433 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b", 2023-07-29T00:00:50.433 INFO:teuthology.orchestra.run.smithi101.stdout: "api_name": "", 2023-07-29T00:00:50.433 INFO:teuthology.orchestra.run.smithi101.stdout: "is_master": false, 2023-07-29T00:00:50.434 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T00:00:50.434 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-29T00:00:50.434 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:50.435 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames": [], 2023-07-29T00:00:50.435 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames_s3website": [], 2023-07-29T00:00:50.435 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T00:00:50.435 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T00:00:50.436 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:50.436 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T00:00:50.436 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b1", 2023-07-29T00:00:50.436 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T00:00:50.437 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-29T00:00:50.437 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:50.437 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T00:00:50.438 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T00:00:50.438 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T00:00:50.438 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T00:00:50.438 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T00:00:50.438 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T00:00:50.438 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T00:00:50.438 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T00:00:50.439 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T00:00:50.439 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T00:00:50.439 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T00:00:50.439 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:50.439 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:50.439 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:50.439 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-29T00:00:50.440 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b2", 2023-07-29T00:00:50.440 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T00:00:50.440 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8001" 2023-07-29T00:00:50.440 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:50.440 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T00:00:50.440 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T00:00:50.441 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T00:00:50.441 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T00:00:50.441 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T00:00:50.441 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T00:00:50.441 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T00:00:50.441 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T00:00:50.441 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T00:00:50.442 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T00:00:50.442 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T00:00:50.442 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:50.442 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:50.442 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:50.442 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_targets": [ 2023-07-29T00:00:50.442 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:50.443 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default-placement", 2023-07-29T00:00:50.443 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [], 2023-07-29T00:00:50.443 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": [ 2023-07-29T00:00:50.443 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD" 2023-07-29T00:00:50.443 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:50.443 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:50.443 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:50.444 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "default-placement", 2023-07-29T00:00:50.444 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T00:00:50.444 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_policy": { 2023-07-29T00:00:50.444 INFO:teuthology.orchestra.run.smithi101.stdout: "groups": [] 2023-07-29T00:00:50.444 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:50.444 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled_features": [ 2023-07-29T00:00:50.444 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T00:00:50.445 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:50.445 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:50.445 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:50.445 INFO:teuthology.orchestra.run.smithi101.stdout: "short_zone_ids": [ 2023-07-29T00:00:50.445 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:50.446 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T00:00:50.446 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 2896287231 2023-07-29T00:00:50.446 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:50.446 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:50.446 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-29T00:00:50.446 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 815606810 2023-07-29T00:00:50.446 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:50.447 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:50.447 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T00:00:50.447 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 2009171363 2023-07-29T00:00:50.447 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:50.447 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:50.447 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "951f39df-ef16-499d-8585-2b2a92ac44b2", 2023-07-29T00:00:50.447 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 3540223569 2023-07-29T00:00:50.448 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:50.448 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:50.448 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-29T00:00:50.448 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 3105190295 2023-07-29T00:00:50.448 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:50.448 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:50.449 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:50.449 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zonegroup": "a", 2023-07-29T00:00:50.449 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T00:00:50.449 INFO:teuthology.orchestra.run.smithi101.stdout: "period_config": { 2023-07-29T00:00:50.450 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_quota": { 2023-07-29T00:00:50.450 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-29T00:00:50.450 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-29T00:00:50.450 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-29T00:00:50.451 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-29T00:00:50.451 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-29T00:00:50.451 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:50.451 INFO:teuthology.orchestra.run.smithi101.stdout: "user_quota": { 2023-07-29T00:00:50.451 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-29T00:00:50.451 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-29T00:00:50.452 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-29T00:00:50.452 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-29T00:00:50.452 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-29T00:00:50.452 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:50.452 INFO:teuthology.orchestra.run.smithi101.stdout: "user_ratelimit": { 2023-07-29T00:00:50.452 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-29T00:00:50.452 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-29T00:00:50.453 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-29T00:00:50.453 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-29T00:00:50.453 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-29T00:00:50.453 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:50.453 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_ratelimit": { 2023-07-29T00:00:50.453 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-29T00:00:50.453 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-29T00:00:50.454 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-29T00:00:50.454 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-29T00:00:50.454 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-29T00:00:50.454 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:50.454 INFO:teuthology.orchestra.run.smithi101.stdout: "anonymous_ratelimit": { 2023-07-29T00:00:50.454 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-29T00:00:50.454 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-29T00:00:50.455 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-29T00:00:50.455 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-29T00:00:50.455 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-29T00:00:50.455 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:50.455 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:50.455 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T00:00:50.456 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_name": "test-realm", 2023-07-29T00:00:50.456 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:00:50.456 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:00:50.456 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:00:50.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:00:50.457 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:00:50.513 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:00:50.513 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:00:50.547 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:00:50.541+0000 7f07699a0880 0 NOTICE: removed master_zone of zonegroup remove_zg 2023-07-29T00:00:50.549 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:00:50.543+0000 7f07699a0880 0 Removed zone from zonegroup remove_zg 2023-07-29T00:00:50.559 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:00:50.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:00:50.560 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:00:50.617 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:00:50.618 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:00:50.649 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:00:50.643+0000 7fe059126880 0 ERROR: zonegroup remove_zg has a non existent master zone 2023-07-29T00:00:50.655 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-07-29T00:00:50.656 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:00:50.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:00:50.657 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:00:50.728 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:00:50.729 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:00:50.769 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:00:50.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:00:50.770 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:00:50.853 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:00:50.853 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:00:50.945 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:00:50.945 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:00:50.946 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 12, 2023-07-29T00:00:50.946 INFO:teuthology.orchestra.run.smithi101.stdout: "predecessor_uuid": "a0987484-ce37-4520-a564-54b7ea8c5251", 2023-07-29T00:00:50.946 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": [], 2023-07-29T00:00:50.946 INFO:teuthology.orchestra.run.smithi101.stdout: "period_map": { 2023-07-29T00:00:50.946 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:00:50.946 INFO:teuthology.orchestra.run.smithi101.stdout: "zonegroups": [ 2023-07-29T00:00:50.946 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:50.947 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "a", 2023-07-29T00:00:50.947 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a", 2023-07-29T00:00:50.947 INFO:teuthology.orchestra.run.smithi101.stdout: "api_name": "", 2023-07-29T00:00:50.947 INFO:teuthology.orchestra.run.smithi101.stdout: "is_master": true, 2023-07-29T00:00:50.947 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T00:00:50.947 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-29T00:00:50.947 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:50.948 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames": [], 2023-07-29T00:00:50.948 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames_s3website": [], 2023-07-29T00:00:50.948 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T00:00:50.948 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T00:00:50.948 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:50.948 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T00:00:50.949 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a1", 2023-07-29T00:00:50.949 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T00:00:50.949 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-29T00:00:50.949 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:50.949 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T00:00:50.949 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T00:00:50.949 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T00:00:50.950 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T00:00:50.950 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T00:00:50.950 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T00:00:50.950 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T00:00:50.950 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T00:00:50.950 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T00:00:50.950 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T00:00:50.951 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T00:00:50.951 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:50.951 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:50.951 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:50.951 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-29T00:00:50.951 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a2", 2023-07-29T00:00:50.951 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T00:00:50.952 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8001" 2023-07-29T00:00:50.952 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:50.952 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T00:00:50.952 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T00:00:50.952 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T00:00:50.952 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T00:00:50.953 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T00:00:50.953 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T00:00:50.953 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T00:00:50.953 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T00:00:50.953 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T00:00:50.953 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T00:00:50.953 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T00:00:50.953 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:50.954 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:50.954 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:50.954 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_targets": [ 2023-07-29T00:00:50.954 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:50.954 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default-placement", 2023-07-29T00:00:50.954 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [], 2023-07-29T00:00:50.954 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": [ 2023-07-29T00:00:50.955 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD" 2023-07-29T00:00:50.955 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:50.955 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:50.955 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:50.955 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "default-placement", 2023-07-29T00:00:50.955 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T00:00:50.956 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_policy": { 2023-07-29T00:00:50.956 INFO:teuthology.orchestra.run.smithi101.stdout: "groups": [] 2023-07-29T00:00:50.956 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:50.956 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled_features": [ 2023-07-29T00:00:50.956 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T00:00:50.956 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:50.956 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:50.957 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:50.957 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "b", 2023-07-29T00:00:50.957 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b", 2023-07-29T00:00:50.957 INFO:teuthology.orchestra.run.smithi101.stdout: "api_name": "", 2023-07-29T00:00:50.957 INFO:teuthology.orchestra.run.smithi101.stdout: "is_master": false, 2023-07-29T00:00:50.958 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T00:00:50.958 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-29T00:00:50.958 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:50.958 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames": [], 2023-07-29T00:00:50.958 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames_s3website": [], 2023-07-29T00:00:50.958 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T00:00:50.958 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T00:00:50.959 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:50.959 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T00:00:50.959 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b1", 2023-07-29T00:00:50.959 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T00:00:50.959 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-29T00:00:50.959 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:50.959 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T00:00:50.960 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T00:00:50.960 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T00:00:50.960 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T00:00:50.960 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T00:00:50.960 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T00:00:50.960 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T00:00:50.960 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T00:00:50.961 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T00:00:50.961 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T00:00:50.971 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T00:00:50.971 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:50.971 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:50.971 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:50.971 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-29T00:00:50.972 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b2", 2023-07-29T00:00:50.972 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T00:00:50.972 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8001" 2023-07-29T00:00:50.972 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:50.972 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T00:00:50.972 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T00:00:50.973 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T00:00:50.973 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T00:00:50.973 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T00:00:50.973 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T00:00:50.973 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T00:00:50.973 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T00:00:50.973 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T00:00:50.974 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T00:00:50.974 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T00:00:50.974 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:50.974 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:50.974 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:50.974 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_targets": [ 2023-07-29T00:00:50.974 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:50.975 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default-placement", 2023-07-29T00:00:50.975 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [], 2023-07-29T00:00:50.975 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": [ 2023-07-29T00:00:50.975 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD" 2023-07-29T00:00:50.975 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:50.975 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:50.975 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:50.976 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "default-placement", 2023-07-29T00:00:50.976 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T00:00:50.976 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_policy": { 2023-07-29T00:00:50.976 INFO:teuthology.orchestra.run.smithi101.stdout: "groups": [] 2023-07-29T00:00:50.976 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:50.977 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled_features": [ 2023-07-29T00:00:50.977 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T00:00:50.977 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:50.977 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:50.977 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T00:00:50.977 INFO:teuthology.orchestra.run.smithi101.stdout: "short_zone_ids": [ 2023-07-29T00:00:50.977 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:50.978 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T00:00:50.978 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 2896287231 2023-07-29T00:00:50.978 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:50.978 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:50.978 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-29T00:00:50.978 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 815606810 2023-07-29T00:00:50.979 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:50.979 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:50.979 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T00:00:50.979 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 2009171363 2023-07-29T00:00:50.979 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:50.979 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:00:50.979 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-29T00:00:50.980 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 3105190295 2023-07-29T00:00:50.980 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:50.980 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:00:50.980 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:50.980 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zonegroup": "a", 2023-07-29T00:00:50.980 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T00:00:50.980 INFO:teuthology.orchestra.run.smithi101.stdout: "period_config": { 2023-07-29T00:00:50.981 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_quota": { 2023-07-29T00:00:50.981 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-29T00:00:50.981 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-29T00:00:50.981 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-29T00:00:50.981 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-29T00:00:50.981 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-29T00:00:50.981 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:50.982 INFO:teuthology.orchestra.run.smithi101.stdout: "user_quota": { 2023-07-29T00:00:50.982 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-29T00:00:50.982 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-29T00:00:50.982 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-29T00:00:50.982 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-29T00:00:50.982 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-29T00:00:50.982 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:50.983 INFO:teuthology.orchestra.run.smithi101.stdout: "user_ratelimit": { 2023-07-29T00:00:50.983 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-29T00:00:50.983 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-29T00:00:50.983 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-29T00:00:50.983 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-29T00:00:50.983 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-29T00:00:50.983 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:50.984 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_ratelimit": { 2023-07-29T00:00:50.984 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-29T00:00:50.984 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-29T00:00:50.984 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-29T00:00:50.984 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-29T00:00:50.984 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-29T00:00:50.985 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:50.985 INFO:teuthology.orchestra.run.smithi101.stdout: "anonymous_ratelimit": { 2023-07-29T00:00:50.985 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-29T00:00:50.985 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-29T00:00:50.985 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-29T00:00:50.985 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-29T00:00:50.985 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-29T00:00:50.986 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:00:50.986 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:00:50.986 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T00:00:50.986 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_name": "test-realm", 2023-07-29T00:00:50.986 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:00:50.986 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:00:50.987 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zg_master_zone_delete ... ok 2023-07-29T00:00:50.988 INFO:rgw_multi.tests:create bucket zone=a1 name=hiiqos-36 2023-07-29T00:00:50.989 DEBUG:boto:path=/hiiqos-36/ 2023-07-29T00:00:50.989 DEBUG:boto:auth_path=/hiiqos-36/ 2023-07-29T00:00:50.989 DEBUG:boto:Method: PUT 2023-07-29T00:00:50.990 DEBUG:boto:Path: /hiiqos-36/ 2023-07-29T00:00:50.990 DEBUG:boto:Data: 2023-07-29T00:00:50.990 DEBUG:boto:Headers: {} 2023-07-29T00:00:50.991 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:00:50.991 DEBUG:boto:Port: 8000 2023-07-29T00:00:50.991 DEBUG:boto:Params: {} 2023-07-29T00:00:50.992 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-29T00:00:50.992 DEBUG:boto:Token: None 2023-07-29T00:00:50.993 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 00:00:50 GMT /hiiqos-36/ 2023-07-29T00:00:50.993 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WsCu1WitvMANTnZXNvLw8JWeGbg= 2023-07-29T00:00:50.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:00:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WsCu1WitvMANTnZXNvLw8JWeGbg=', 'Content-Length': '0'} 2023-07-29T00:00:56.012 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007e130d56b15729b6-0064c456b7-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:00:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:00:56.013 INFO:rgw_multi.tests:create bucket zone=a2 name=hiiqos-37 2023-07-29T00:00:56.013 DEBUG:boto:path=/hiiqos-37/ 2023-07-29T00:00:56.014 DEBUG:boto:auth_path=/hiiqos-37/ 2023-07-29T00:00:56.014 DEBUG:boto:Method: PUT 2023-07-29T00:00:56.014 DEBUG:boto:Path: /hiiqos-37/ 2023-07-29T00:00:56.015 DEBUG:boto:Data: 2023-07-29T00:00:56.015 DEBUG:boto:Headers: {} 2023-07-29T00:00:56.015 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:00:56.016 DEBUG:boto:Port: 8001 2023-07-29T00:00:56.016 DEBUG:boto:Params: {} 2023-07-29T00:00:56.016 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-29T00:00:56.017 DEBUG:boto:Token: None 2023-07-29T00:00:56.017 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 00:00:56 GMT /hiiqos-37/ 2023-07-29T00:00:56.018 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KYKDx1Z1R1LtI959jc2BpO9rCIw= 2023-07-29T00:00:56.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:00:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KYKDx1Z1R1LtI959jc2BpO9rCIw=', 'Content-Length': '0'} 2023-07-29T00:01:01.048 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e6d781a281a55f64-0064c456bd-1663-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:01:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:01:01.049 INFO:rgw_multi.tests:create bucket zone=b1 name=hiiqos-38 2023-07-29T00:01:01.049 DEBUG:boto:path=/hiiqos-38/ 2023-07-29T00:01:01.050 DEBUG:boto:auth_path=/hiiqos-38/ 2023-07-29T00:01:01.050 DEBUG:boto:Method: PUT 2023-07-29T00:01:01.051 DEBUG:boto:Path: /hiiqos-38/ 2023-07-29T00:01:01.051 DEBUG:boto:Data: 2023-07-29T00:01:01.051 DEBUG:boto:Headers: {} 2023-07-29T00:01:01.052 DEBUG:boto:Host: smithi139.front.sepia.ceph.com:8000 2023-07-29T00:01:01.052 DEBUG:boto:Port: 8000 2023-07-29T00:01:01.052 DEBUG:boto:Params: {} 2023-07-29T00:01:01.053 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-29T00:01:01.053 DEBUG:boto:Token: None 2023-07-29T00:01:01.054 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 00:01:01 GMT /hiiqos-38/ 2023-07-29T00:01:01.054 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xGVme5HuLzm3m0s+tA/Yq+51M6w= 2023-07-29T00:01:01.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:01:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xGVme5HuLzm3m0s+tA/Yq+51M6w=', 'Content-Length': '0'} 2023-07-29T00:01:06.088 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d296bf52837dedd8-0064c456c2-116d-b1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:01:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:01:06.089 INFO:rgw_multi.tests:create bucket zone=b2 name=hiiqos-39 2023-07-29T00:01:06.089 DEBUG:boto:path=/hiiqos-39/ 2023-07-29T00:01:06.089 DEBUG:boto:auth_path=/hiiqos-39/ 2023-07-29T00:01:06.090 DEBUG:boto:Method: PUT 2023-07-29T00:01:06.090 DEBUG:boto:Path: /hiiqos-39/ 2023-07-29T00:01:06.090 DEBUG:boto:Data: 2023-07-29T00:01:06.091 DEBUG:boto:Headers: {} 2023-07-29T00:01:06.091 DEBUG:boto:Host: smithi139.front.sepia.ceph.com:8001 2023-07-29T00:01:06.091 DEBUG:boto:Port: 8001 2023-07-29T00:01:06.092 DEBUG:boto:Params: {} 2023-07-29T00:01:06.092 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-29T00:01:06.093 DEBUG:boto:Token: None 2023-07-29T00:01:06.093 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 00:01:06 GMT /hiiqos-39/ 2023-07-29T00:01:06.094 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:89sahlaw+/Yu+4jVT+iCM2u0ZiU= 2023-07-29T00:01:06.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:01:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:89sahlaw+/Yu+4jVT+iCM2u0ZiU=', 'Content-Length': '0'} 2023-07-29T00:01:11.125 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b5cff04df616283a-0064c456c7-1171-b2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:01:11.126 DEBUG:boto:path=/hiiqos-36/ 2023-07-29T00:01:11.127 DEBUG:boto:auth_path=/hiiqos-36/ 2023-07-29T00:01:11.127 DEBUG:boto:path=/hiiqos-36/?website 2023-07-29T00:01:11.127 DEBUG:boto:auth_path=/hiiqos-36/?website 2023-07-29T00:01:11.127 DEBUG:boto:Method: PUT 2023-07-29T00:01:11.128 DEBUG:boto:Path: /hiiqos-36/?website 2023-07-29T00:01:11.128 DEBUG:boto:Data: index.htmlerror.html 2023-07-29T00:01:11.129 DEBUG:boto:Headers: {} 2023-07-29T00:01:11.129 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:01:11.129 DEBUG:boto:Port: 8000 2023-07-29T00:01:11.129 DEBUG:boto:Params: {} 2023-07-29T00:01:11.130 DEBUG:boto:Token: None 2023-07-29T00:01:11.130 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 00:01:11 GMT /hiiqos-36/?website 2023-07-29T00:01:11.131 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hdqfpt4NemJEo/opbhfRwHD7ONs= 2023-07-29T00:01:11.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:01:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hdqfpt4NemJEo/opbhfRwHD7ONs=', 'Content-Length': '241'} 2023-07-29T00:01:11.132 DEBUG:boto:Response headers: [('Content-Length', '192'), ('x-amz-request-id', 'tx00000dc011fabab8b9de0-0064c456c7-165f-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:01:11.134 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_website ... SKIP: test_set_bucket_website skipped. Requires rgw_enable_static_website = 1. 2023-07-29T00:01:11.135 INFO:rgw_multi.tests:create bucket zone=a1 name=hiiqos-40 2023-07-29T00:01:11.135 DEBUG:boto:path=/hiiqos-40/ 2023-07-29T00:01:11.136 DEBUG:boto:auth_path=/hiiqos-40/ 2023-07-29T00:01:11.136 DEBUG:boto:Method: PUT 2023-07-29T00:01:11.137 DEBUG:boto:Path: /hiiqos-40/ 2023-07-29T00:01:11.137 DEBUG:boto:Data: 2023-07-29T00:01:11.137 DEBUG:boto:Headers: {} 2023-07-29T00:01:11.138 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:01:11.138 DEBUG:boto:Port: 8000 2023-07-29T00:01:11.138 DEBUG:boto:Params: {} 2023-07-29T00:01:11.139 DEBUG:boto:Token: None 2023-07-29T00:01:11.140 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 00:01:11 GMT /hiiqos-40/ 2023-07-29T00:01:11.140 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mjwFQK+83Btmnuqatsh5+D2rKws= 2023-07-29T00:01:11.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:01:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mjwFQK+83Btmnuqatsh5+D2rKws=', 'Content-Length': '0'} 2023-07-29T00:01:11.158 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d8446bf0070671b2-0064c456c7-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:01:11.158 INFO:rgw_multi.tests:create bucket zone=a2 name=hiiqos-41 2023-07-29T00:01:11.159 DEBUG:boto:path=/hiiqos-41/ 2023-07-29T00:01:11.160 DEBUG:boto:auth_path=/hiiqos-41/ 2023-07-29T00:01:11.160 DEBUG:boto:Method: PUT 2023-07-29T00:01:11.161 DEBUG:boto:Path: /hiiqos-41/ 2023-07-29T00:01:11.161 DEBUG:boto:Data: 2023-07-29T00:01:11.161 DEBUG:boto:Headers: {} 2023-07-29T00:01:11.162 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:01:11.162 DEBUG:boto:Port: 8001 2023-07-29T00:01:11.162 DEBUG:boto:Params: {} 2023-07-29T00:01:11.163 DEBUG:boto:Token: None 2023-07-29T00:01:11.163 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 00:01:11 GMT /hiiqos-41/ 2023-07-29T00:01:11.164 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PUYp8BSI1uQJi4CFlf9tKU19fxw= 2023-07-29T00:01:11.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:01:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PUYp8BSI1uQJi4CFlf9tKU19fxw=', 'Content-Length': '0'} 2023-07-29T00:01:11.188 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e067165a9e407df9-0064c456c7-1663-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:01:11.189 INFO:rgw_multi.tests:create bucket zone=b1 name=hiiqos-42 2023-07-29T00:01:11.189 DEBUG:boto:path=/hiiqos-42/ 2023-07-29T00:01:11.190 DEBUG:boto:auth_path=/hiiqos-42/ 2023-07-29T00:01:11.190 DEBUG:boto:Method: PUT 2023-07-29T00:01:11.190 DEBUG:boto:Path: /hiiqos-42/ 2023-07-29T00:01:11.191 DEBUG:boto:Data: 2023-07-29T00:01:11.191 DEBUG:boto:Headers: {} 2023-07-29T00:01:11.192 DEBUG:boto:Host: smithi139.front.sepia.ceph.com:8000 2023-07-29T00:01:11.192 DEBUG:boto:Port: 8000 2023-07-29T00:01:11.192 DEBUG:boto:Params: {} 2023-07-29T00:01:11.193 DEBUG:boto:Token: None 2023-07-29T00:01:11.193 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 00:01:11 GMT /hiiqos-42/ 2023-07-29T00:01:11.194 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6HK27u9WFKvw5dwZdcxpn2xW3vU= 2023-07-29T00:01:11.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:01:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6HK27u9WFKvw5dwZdcxpn2xW3vU=', 'Content-Length': '0'} 2023-07-29T00:01:11.222 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003bb56d34995470dd-0064c456c7-116d-b1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:01:11.223 INFO:rgw_multi.tests:create bucket zone=b2 name=hiiqos-43 2023-07-29T00:01:11.223 DEBUG:boto:path=/hiiqos-43/ 2023-07-29T00:01:11.223 DEBUG:boto:auth_path=/hiiqos-43/ 2023-07-29T00:01:11.224 DEBUG:boto:Method: PUT 2023-07-29T00:01:11.224 DEBUG:boto:Path: /hiiqos-43/ 2023-07-29T00:01:11.224 DEBUG:boto:Data: 2023-07-29T00:01:11.225 DEBUG:boto:Headers: {} 2023-07-29T00:01:11.225 DEBUG:boto:Host: smithi139.front.sepia.ceph.com:8001 2023-07-29T00:01:11.225 DEBUG:boto:Port: 8001 2023-07-29T00:01:11.226 DEBUG:boto:Params: {} 2023-07-29T00:01:11.226 DEBUG:boto:Token: None 2023-07-29T00:01:11.227 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 00:01:11 GMT /hiiqos-43/ 2023-07-29T00:01:11.227 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:peG/W23ilK0z15hyJvG9/VJn7OQ= 2023-07-29T00:01:11.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:01:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:peG/W23ilK0z15hyJvG9/VJn7OQ=', 'Content-Length': '0'} 2023-07-29T00:01:11.255 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a0156797ba3529af-0064c456c7-1171-b2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:01:11.256 DEBUG:boto:path=/hiiqos-40/ 2023-07-29T00:01:11.257 DEBUG:boto:auth_path=/hiiqos-40/ 2023-07-29T00:01:11.258 DEBUG:boto:path=/hiiqos-40/?policy 2023-07-29T00:01:11.258 DEBUG:boto:auth_path=/hiiqos-40/?policy 2023-07-29T00:01:11.258 DEBUG:boto:Method: PUT 2023-07-29T00:01:11.259 DEBUG:boto:Path: /hiiqos-40/?policy 2023-07-29T00:01:11.259 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-07-29T00:01:11.259 DEBUG:boto:Headers: {} 2023-07-29T00:01:11.260 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:01:11.260 DEBUG:boto:Port: 8000 2023-07-29T00:01:11.261 DEBUG:boto:Params: {} 2023-07-29T00:01:11.261 DEBUG:boto:Token: None 2023-07-29T00:01:11.261 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 00:01:11 GMT /hiiqos-40/?policy 2023-07-29T00:01:11.262 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lOuOpKo1BzG/PooRVE9Tby+Yh0Y= 2023-07-29T00:01:11.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:01:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lOuOpKo1BzG/PooRVE9Tby+Yh0Y=', 'Content-Length': '97'} 2023-07-29T00:01:11.267 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004099e9f3ef3683be-0064c456c7-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:01:11.268 DEBUG:boto:path=/hiiqos-40/ 2023-07-29T00:01:11.269 DEBUG:boto:auth_path=/hiiqos-40/ 2023-07-29T00:01:11.269 DEBUG:boto:path=/hiiqos-40/?policy 2023-07-29T00:01:11.269 DEBUG:boto:auth_path=/hiiqos-40/?policy 2023-07-29T00:01:11.270 DEBUG:boto:Method: GET 2023-07-29T00:01:11.270 DEBUG:boto:Path: /hiiqos-40/?policy 2023-07-29T00:01:11.270 DEBUG:boto:Data: 2023-07-29T00:01:11.271 DEBUG:boto:Headers: {} 2023-07-29T00:01:11.271 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:01:11.271 DEBUG:boto:Port: 8000 2023-07-29T00:01:11.272 DEBUG:boto:Params: {} 2023-07-29T00:01:11.272 DEBUG:boto:Token: None 2023-07-29T00:01:11.272 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:01:11 GMT /hiiqos-40/?policy 2023-07-29T00:01:11.273 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LaRsvqsoK9bX/eMIkErTZtz9Qpo= 2023-07-29T00:01:11.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:01:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LaRsvqsoK9bX/eMIkErTZtz9Qpo=', 'Content-Length': '0'} 2023-07-29T00:01:11.274 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006ab7d9c96d98c674-0064c456c7-165f-a1'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Sat, 29 Jul 2023 00:01:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:01:11.275 DEBUG:boto:path=/hiiqos-41/ 2023-07-29T00:01:11.275 DEBUG:boto:auth_path=/hiiqos-41/ 2023-07-29T00:01:11.276 DEBUG:boto:path=/hiiqos-41/?policy 2023-07-29T00:01:11.276 DEBUG:boto:auth_path=/hiiqos-41/?policy 2023-07-29T00:01:11.277 DEBUG:boto:Method: PUT 2023-07-29T00:01:11.277 DEBUG:boto:Path: /hiiqos-41/?policy 2023-07-29T00:01:11.277 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-07-29T00:01:11.277 DEBUG:boto:Headers: {} 2023-07-29T00:01:11.278 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:01:11.278 DEBUG:boto:Port: 8001 2023-07-29T00:01:11.278 DEBUG:boto:Params: {} 2023-07-29T00:01:11.279 DEBUG:boto:Token: None 2023-07-29T00:01:11.279 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 00:01:11 GMT /hiiqos-41/?policy 2023-07-29T00:01:11.280 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GqRAjzNpbe2hKsSlMSVE3dHLM50= 2023-07-29T00:01:11.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:01:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GqRAjzNpbe2hKsSlMSVE3dHLM50=', 'Content-Length': '97'} 2023-07-29T00:01:12.290 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007ae93c17e12d6d41-0064c456c7-1663-a2'), ('Date', 'Sat, 29 Jul 2023 00:01:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:01:12.291 DEBUG:boto:path=/hiiqos-41/ 2023-07-29T00:01:12.291 DEBUG:boto:auth_path=/hiiqos-41/ 2023-07-29T00:01:12.292 DEBUG:boto:path=/hiiqos-41/?policy 2023-07-29T00:01:12.292 DEBUG:boto:auth_path=/hiiqos-41/?policy 2023-07-29T00:01:12.293 DEBUG:boto:Method: GET 2023-07-29T00:01:12.293 DEBUG:boto:Path: /hiiqos-41/?policy 2023-07-29T00:01:12.293 DEBUG:boto:Data: 2023-07-29T00:01:12.294 DEBUG:boto:Headers: {} 2023-07-29T00:01:12.294 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:01:12.294 DEBUG:boto:Port: 8001 2023-07-29T00:01:12.294 DEBUG:boto:Params: {} 2023-07-29T00:01:12.295 DEBUG:boto:Token: None 2023-07-29T00:01:12.295 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:01:12 GMT /hiiqos-41/?policy 2023-07-29T00:01:12.296 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xJ21F9k0tkLXoC0K2lvYfblROKk= 2023-07-29T00:01:12.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:01:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xJ21F9k0tkLXoC0K2lvYfblROKk=', 'Content-Length': '0'} 2023-07-29T00:01:12.297 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008d3f0bf7975b7877-0064c456c8-1663-a2'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Sat, 29 Jul 2023 00:01:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:01:12.298 DEBUG:boto:path=/hiiqos-42/ 2023-07-29T00:01:12.298 DEBUG:boto:auth_path=/hiiqos-42/ 2023-07-29T00:01:12.299 DEBUG:boto:path=/hiiqos-42/?policy 2023-07-29T00:01:12.299 DEBUG:boto:auth_path=/hiiqos-42/?policy 2023-07-29T00:01:12.299 DEBUG:boto:Method: PUT 2023-07-29T00:01:12.300 DEBUG:boto:Path: /hiiqos-42/?policy 2023-07-29T00:01:12.300 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-07-29T00:01:12.301 DEBUG:boto:Headers: {} 2023-07-29T00:01:12.301 DEBUG:boto:Host: smithi139.front.sepia.ceph.com:8000 2023-07-29T00:01:12.301 DEBUG:boto:Port: 8000 2023-07-29T00:01:12.302 DEBUG:boto:Params: {} 2023-07-29T00:01:12.302 DEBUG:boto:Token: None 2023-07-29T00:01:12.302 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 00:01:12 GMT /hiiqos-42/?policy 2023-07-29T00:01:12.303 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eARJecyk4GAats4xUSOhZsLlJao= 2023-07-29T00:01:12.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:01:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eARJecyk4GAats4xUSOhZsLlJao=', 'Content-Length': '97'} 2023-07-29T00:01:13.314 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f1ae1290efd55af3-0064c456c8-116d-b1'), ('Date', 'Sat, 29 Jul 2023 00:01:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:01:13.315 DEBUG:boto:path=/hiiqos-42/ 2023-07-29T00:01:13.315 DEBUG:boto:auth_path=/hiiqos-42/ 2023-07-29T00:01:13.316 DEBUG:boto:path=/hiiqos-42/?policy 2023-07-29T00:01:13.316 DEBUG:boto:auth_path=/hiiqos-42/?policy 2023-07-29T00:01:13.316 DEBUG:boto:Method: GET 2023-07-29T00:01:13.317 DEBUG:boto:Path: /hiiqos-42/?policy 2023-07-29T00:01:13.317 DEBUG:boto:Data: 2023-07-29T00:01:13.317 DEBUG:boto:Headers: {} 2023-07-29T00:01:13.318 DEBUG:boto:Host: smithi139.front.sepia.ceph.com:8000 2023-07-29T00:01:13.318 DEBUG:boto:Port: 8000 2023-07-29T00:01:13.318 DEBUG:boto:Params: {} 2023-07-29T00:01:13.319 DEBUG:boto:Token: None 2023-07-29T00:01:13.319 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:01:13 GMT /hiiqos-42/?policy 2023-07-29T00:01:13.320 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:B3n1Ao9rLW6CIswL3I1sBKQMI6I= 2023-07-29T00:01:13.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:01:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:B3n1Ao9rLW6CIswL3I1sBKQMI6I=', 'Content-Length': '0'} 2023-07-29T00:01:13.322 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009ec05ae65220a882-0064c456c9-116d-b1'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Sat, 29 Jul 2023 00:01:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:01:13.322 DEBUG:boto:path=/hiiqos-43/ 2023-07-29T00:01:13.323 DEBUG:boto:auth_path=/hiiqos-43/ 2023-07-29T00:01:13.324 DEBUG:boto:path=/hiiqos-43/?policy 2023-07-29T00:01:13.324 DEBUG:boto:auth_path=/hiiqos-43/?policy 2023-07-29T00:01:13.325 DEBUG:boto:Method: PUT 2023-07-29T00:01:13.325 DEBUG:boto:Path: /hiiqos-43/?policy 2023-07-29T00:01:13.326 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-07-29T00:01:13.326 DEBUG:boto:Headers: {} 2023-07-29T00:01:13.327 DEBUG:boto:Host: smithi139.front.sepia.ceph.com:8001 2023-07-29T00:01:13.327 DEBUG:boto:Port: 8001 2023-07-29T00:01:13.328 DEBUG:boto:Params: {} 2023-07-29T00:01:13.328 DEBUG:boto:Token: None 2023-07-29T00:01:13.329 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 00:01:13 GMT /hiiqos-43/?policy 2023-07-29T00:01:13.329 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:g6Ak9MXpWn8/ubUYJXf/I5lZPsY= 2023-07-29T00:01:13.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:01:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:g6Ak9MXpWn8/ubUYJXf/I5lZPsY=', 'Content-Length': '97'} 2023-07-29T00:01:14.341 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000cca397095d96e7a4-0064c456c9-1171-b2'), ('Date', 'Sat, 29 Jul 2023 00:01:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:01:14.342 DEBUG:boto:path=/hiiqos-43/ 2023-07-29T00:01:14.343 DEBUG:boto:auth_path=/hiiqos-43/ 2023-07-29T00:01:14.343 DEBUG:boto:path=/hiiqos-43/?policy 2023-07-29T00:01:14.344 DEBUG:boto:auth_path=/hiiqos-43/?policy 2023-07-29T00:01:14.345 DEBUG:boto:Method: GET 2023-07-29T00:01:14.345 DEBUG:boto:Path: /hiiqos-43/?policy 2023-07-29T00:01:14.345 DEBUG:boto:Data: 2023-07-29T00:01:14.346 DEBUG:boto:Headers: {} 2023-07-29T00:01:14.346 DEBUG:boto:Host: smithi139.front.sepia.ceph.com:8001 2023-07-29T00:01:14.347 DEBUG:boto:Port: 8001 2023-07-29T00:01:14.347 DEBUG:boto:Params: {} 2023-07-29T00:01:14.348 DEBUG:boto:Token: None 2023-07-29T00:01:14.348 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:01:14 GMT /hiiqos-43/?policy 2023-07-29T00:01:14.349 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YD056dDLcY4fJ1wa9gtN0HMDtEQ= 2023-07-29T00:01:14.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:01:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YD056dDLcY4fJ1wa9gtN0HMDtEQ=', 'Content-Length': '0'} 2023-07-29T00:01:14.351 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008ed30b4247c6a6c4-0064c456ca-1171-b2'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Sat, 29 Jul 2023 00:01:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:01:14.352 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_policy ... ok 2023-07-29T00:01:14.353 INFO:rgw_multi.tests:create bucket zone=a1 name=hiiqos-44 2023-07-29T00:01:14.353 DEBUG:boto:path=/hiiqos-44/ 2023-07-29T00:01:14.354 DEBUG:boto:auth_path=/hiiqos-44/ 2023-07-29T00:01:14.354 DEBUG:boto:Method: PUT 2023-07-29T00:01:14.354 DEBUG:boto:Path: /hiiqos-44/ 2023-07-29T00:01:14.355 DEBUG:boto:Data: 2023-07-29T00:01:14.355 DEBUG:boto:Headers: {} 2023-07-29T00:01:14.355 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:01:14.356 DEBUG:boto:Port: 8000 2023-07-29T00:01:14.356 DEBUG:boto:Params: {} 2023-07-29T00:01:14.357 DEBUG:boto:Token: None 2023-07-29T00:01:14.357 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 00:01:14 GMT /hiiqos-44/ 2023-07-29T00:01:14.358 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rONFzhE/ES8VHww0j2FO6J3+OjA= 2023-07-29T00:01:14.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:01:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rONFzhE/ES8VHww0j2FO6J3+OjA=', 'Content-Length': '0'} 2023-07-29T00:01:14.376 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000687941333f6ba525-0064c456ca-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:01:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:01:14.377 INFO:rgw_multi.tests:create bucket zone=a2 name=hiiqos-45 2023-07-29T00:01:14.377 DEBUG:boto:path=/hiiqos-45/ 2023-07-29T00:01:14.378 DEBUG:boto:auth_path=/hiiqos-45/ 2023-07-29T00:01:14.379 DEBUG:boto:Method: PUT 2023-07-29T00:01:14.379 DEBUG:boto:Path: /hiiqos-45/ 2023-07-29T00:01:14.380 DEBUG:boto:Data: 2023-07-29T00:01:14.381 DEBUG:boto:Headers: {} 2023-07-29T00:01:14.381 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:01:14.382 DEBUG:boto:Port: 8001 2023-07-29T00:01:14.383 DEBUG:boto:Params: {} 2023-07-29T00:01:14.383 DEBUG:boto:Token: None 2023-07-29T00:01:14.384 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 00:01:14 GMT /hiiqos-45/ 2023-07-29T00:01:14.385 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2mzMFXrB7+46UfcPLWZbDMlYnio= 2023-07-29T00:01:14.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:01:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2mzMFXrB7+46UfcPLWZbDMlYnio=', 'Content-Length': '0'} 2023-07-29T00:01:14.412 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c2ad1018d54688c1-0064c456ca-1663-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:01:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:01:14.413 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:01:14.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:01:14.414 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:01:14.463 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:01:14.464 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:01:14.494 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T00:01:14.507 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T00:01:14.507 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:01:14.507 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:01:14.507 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:46:09.233742Z" 2023-07-29T00:01:14.508 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:14.508 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:01:14.508 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588874.395997_652.1", 2023-07-29T00:01:14.508 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:01:14.395997Z" 2023-07-29T00:01:14.509 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:14.509 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:01:14.509 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:01:14.509 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T00:01:14.509 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:14.510 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:01:14.510 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:01:14.510 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T00:01:14.510 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:14.510 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T00:01:14.512 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:01:14.512 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T00:01:14.513 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:01:14.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:01:14.514 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:01:14.561 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:01:14.561 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:01:14.588 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:01:14.580+0000 7f208a726880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:01:14.589 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:01:14.581+0000 7f208a726880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:01:14.600 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:01:14.601 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:01:14.601 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:01:14.601 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:01:14.601 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:01:14.602 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:01:14.602 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:14.602 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:14.602 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:01:14.602 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:01:14.602 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:01:14.603 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:01:14.603 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:01:14.603 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:01:14.603 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:01:14.603 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:01:14.603 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:01:14.604 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:01:14.604 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:14.604 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:14.604 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:14.604 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:01:14.604 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:01:14.605 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:01:14.605 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:01:14.605 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588579.740560_402.1", 2023-07-29T00:01:14.605 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:01:14.605 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:01:14.605 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:01:14.606 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:56:19.740560Z", 2023-07-29T00:01:14.606 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:14.606 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:14.606 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:14.606 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:01:14.606 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:01:14.607 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:01:14.607 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:01:14.607 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:01:14.607 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:01:14.607 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:01:14.607 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:01:14.608 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:01:14.608 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:14.608 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:14.608 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:14.608 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:01:14.608 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:01:14.608 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:01:14.609 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:01:14.609 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:01:14.609 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:01:14.609 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:01:14.609 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:01:14.609 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:01:14.610 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:14.610 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:14.610 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:14.610 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:01:14.610 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:14.610 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:01:14.610 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:01:14.611 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:01:14.611 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:14.611 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:01:14Z" 2023-07-29T00:01:14.611 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:01:14.612 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588579.740560_402.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:56:19.740560Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:01:14Z" } 2023-07-29T00:01:14.613 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588579.740560_402.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:56:19.740560Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:01:14.613 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:01:14.614 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588579.740560_402.1', 2: '', 3: ''} 2023-07-29T00:01:14.614 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588579.740560_402.1 2023-07-29T00:01:19.615 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:01:19.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:01:19.616 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:01:19.669 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:01:19.670 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:01:19.701 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:01:19.694+0000 7f7af0c12880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:01:19.702 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:01:19.695+0000 7f7af0c12880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:01:19.715 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:01:19.715 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:01:19.715 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:01:19.715 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:01:19.715 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:01:19.715 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:01:19.716 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:19.716 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:19.716 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:01:19.716 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:01:19.716 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:01:19.716 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:01:19.717 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:01:19.717 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:01:19.717 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:01:19.717 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:01:19.717 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:01:19.717 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:01:19.717 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:19.718 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:19.718 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:19.718 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:01:19.718 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:01:19.718 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:01:19.718 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:01:19.718 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588579.740560_402.1", 2023-07-29T00:01:19.719 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:01:19.719 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:01:19.719 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:01:19.719 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:56:19.740560Z", 2023-07-29T00:01:19.719 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:19.719 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:19.720 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:19.720 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:01:19.720 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:01:19.720 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:01:19.720 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:01:19.720 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:01:19.720 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:01:19.721 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:01:19.721 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:01:19.721 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:01:19.721 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:19.721 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:19.721 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:19.721 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:01:19.722 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:01:19.722 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:01:19.722 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:01:19.722 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:01:19.722 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:01:19.722 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:01:19.722 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:01:19.723 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:01:19.723 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:19.723 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:19.723 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:19.723 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:01:19.723 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:19.723 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:01:19.724 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:01:19.724 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:01:19.724 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:19.724 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:01:19Z" 2023-07-29T00:01:19.724 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:01:19.725 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588579.740560_402.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:56:19.740560Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:01:19Z" } 2023-07-29T00:01:19.725 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588579.740560_402.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:56:19.740560Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:01:19.726 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:01:19.726 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588579.740560_402.1', 2: '', 3: ''} 2023-07-29T00:01:19.726 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588579.740560_402.1 2023-07-29T00:01:24.728 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:01:24.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:01:24.729 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:01:24.783 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:01:24.783 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:01:24.814 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:01:24.806+0000 7f42a0936880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:01:24.815 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:01:24.806+0000 7f42a0936880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:01:24.827 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:01:24.828 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:01:24.828 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:01:24.828 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:01:24.828 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:01:24.829 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:01:24.829 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:24.829 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:24.830 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:01:24.830 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:01:24.830 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:01:24.831 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:01:24.831 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:01:24.831 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:01:24.831 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:01:24.832 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:01:24.832 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:01:24.832 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:01:24.833 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:24.833 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:24.834 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:24.834 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:01:24.834 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:01:24.834 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:01:24.835 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:01:24.835 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588579.740560_402.1", 2023-07-29T00:01:24.835 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:01:24.836 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:01:24.836 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:01:24.836 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:56:19.740560Z", 2023-07-29T00:01:24.837 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:24.837 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:24.837 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:24.837 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:01:24.838 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:01:24.838 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:01:24.838 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:01:24.838 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:01:24.839 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:01:24.839 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:01:24.839 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:01:24.840 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:01:24.840 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:24.840 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:24.840 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:24.841 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:01:24.841 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:01:24.841 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:01:24.841 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:01:24.841 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:01:24.841 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:01:24.841 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:01:24.842 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:01:24.842 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:01:24.842 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:24.842 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:24.843 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:24.843 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:01:24.843 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:24.843 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:01:24.843 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:01:24.843 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:01:24.843 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:24.844 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:01:24Z" 2023-07-29T00:01:24.844 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:01:24.845 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588579.740560_402.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:56:19.740560Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:01:24Z" } 2023-07-29T00:01:24.845 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588579.740560_402.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:56:19.740560Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:01:24.845 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:01:24.846 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588579.740560_402.1', 2: '', 3: ''} 2023-07-29T00:01:24.846 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588579.740560_402.1 2023-07-29T00:01:29.847 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:01:29.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:01:29.848 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:01:29.901 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:01:29.902 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:01:29.932 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:01:29.924+0000 7f68c3d26880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:01:29.934 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:01:29.925+0000 7f68c3d26880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:01:29.946 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:01:29.946 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:01:29.947 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:01:29.947 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:01:29.947 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:01:29.948 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:01:29.948 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:29.948 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:29.949 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:01:29.949 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:01:29.949 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:01:29.950 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:01:29.950 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:01:29.950 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:01:29.950 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:01:29.951 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:01:29.951 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:01:29.951 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:01:29.951 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:29.951 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:29.951 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:29.952 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:01:29.952 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:01:29.952 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:01:29.953 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:01:29.953 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588579.740560_402.1", 2023-07-29T00:01:29.953 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:01:29.953 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:01:29.954 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:01:29.954 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:56:19.740560Z", 2023-07-29T00:01:29.954 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:29.954 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:29.955 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:29.955 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:01:29.955 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:01:29.955 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:01:29.956 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:01:29.956 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:01:29.957 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:01:29.957 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:01:29.957 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:01:29.957 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:01:29.958 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:29.958 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:29.958 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:29.959 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:01:29.959 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:01:29.959 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:01:29.960 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:01:29.960 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:01:29.960 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:01:29.961 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:01:29.961 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:01:29.961 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:01:29.962 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:29.962 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:29.962 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:29.963 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:01:29.963 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:29.963 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:01:29.964 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:01:29.964 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:01:29.964 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:29.965 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:01:29Z" 2023-07-29T00:01:29.965 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:01:29.966 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588579.740560_402.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:56:19.740560Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:01:29Z" } 2023-07-29T00:01:29.967 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588579.740560_402.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:56:19.740560Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:01:29.968 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:01:29.968 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588579.740560_402.1', 2: '', 3: ''} 2023-07-29T00:01:29.969 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588579.740560_402.1 2023-07-29T00:01:34.971 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:01:34.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:01:34.972 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:01:35.028 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:01:35.029 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:01:35.060 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:01:35.051+0000 7f8c7bb26880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:01:35.060 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:01:35.052+0000 7f8c7bb26880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:01:35.073 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:01:35.073 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:01:35.074 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:01:35.074 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:01:35.074 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:01:35.074 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:01:35.074 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:35.074 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:35.075 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:01:35.075 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:01:35.075 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:01:35.075 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:01:35.075 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:01:35.075 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:01:35.076 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:01:35.076 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:01:35.076 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:01:35.076 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:01:35.076 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:35.076 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:35.077 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:35.077 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:01:35.077 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:01:35.077 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:01:35.077 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:01:35.077 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588579.740560_402.1", 2023-07-29T00:01:35.078 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:01:35.078 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:01:35.078 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:01:35.078 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:56:19.740560Z", 2023-07-29T00:01:35.078 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:35.078 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:35.078 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:35.079 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:01:35.079 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:01:35.079 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:01:35.079 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:01:35.079 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:01:35.079 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:01:35.080 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:01:35.080 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:01:35.080 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:01:35.080 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:35.080 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:35.080 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:35.081 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:01:35.081 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:01:35.081 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:01:35.081 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:01:35.081 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:01:35.081 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:01:35.081 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:01:35.082 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:01:35.082 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:01:35.082 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:35.082 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:35.082 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:35.082 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:01:35.082 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:35.083 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:01:35.083 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:01:35.083 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:01:35.083 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:35.083 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:01:35Z" 2023-07-29T00:01:35.083 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:01:35.085 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588579.740560_402.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:56:19.740560Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:01:35Z" } 2023-07-29T00:01:35.085 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588579.740560_402.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:56:19.740560Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:01:35.086 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:01:35.087 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588579.740560_402.1', 2: '', 3: ''} 2023-07-29T00:01:35.087 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588579.740560_402.1 2023-07-29T00:01:40.088 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:01:40.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:01:40.089 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:01:40.142 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:01:40.143 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:01:40.173 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:01:40.164+0000 7f8357322880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:01:40.174 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:01:40.165+0000 7f8357322880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:01:40.187 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:01:40.187 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:01:40.187 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:01:40.187 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:01:40.188 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:01:40.188 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:01:40.188 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:40.188 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:40.188 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:01:40.189 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:01:40.189 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:01:40.189 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:01:40.189 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:01:40.189 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:01:40.189 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:01:40.189 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:01:40.190 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:01:40.190 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:01:40.190 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:40.190 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:40.190 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:40.190 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:01:40.191 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:01:40.191 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:01:40.191 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:01:40.191 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588579.740560_402.1", 2023-07-29T00:01:40.191 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:01:40.191 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:01:40.192 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:01:40.192 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:56:19.740560Z", 2023-07-29T00:01:40.192 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:40.192 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:40.192 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:40.192 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:01:40.192 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:01:40.193 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:01:40.193 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:01:40.193 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:01:40.193 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:01:40.193 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:01:40.193 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:01:40.194 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:01:40.194 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:40.194 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:40.194 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:40.194 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:01:40.194 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:01:40.194 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:01:40.195 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:01:40.195 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:01:40.195 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:01:40.195 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:01:40.195 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:01:40.195 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:01:40.196 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:40.196 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:40.196 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:40.196 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:01:40.196 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:40.196 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:01:40.197 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:01:40.197 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:01:40.197 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:40.197 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:01:40Z" 2023-07-29T00:01:40.197 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:01:40.198 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588579.740560_402.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:56:19.740560Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:01:40Z" } 2023-07-29T00:01:40.198 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588579.740560_402.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:56:19.740560Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:01:40.199 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:01:40.199 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588579.740560_402.1', 2: '', 3: ''} 2023-07-29T00:01:40.200 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588579.740560_402.1 2023-07-29T00:01:45.200 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:01:45.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:01:45.201 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:01:45.255 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:01:45.256 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:01:45.286 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:01:45.277+0000 7f79d4d35880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:01:45.287 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:01:45.278+0000 7f79d4d35880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:01:45.300 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:01:45.300 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:01:45.300 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:01:45.301 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:01:45.301 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:01:45.301 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:01:45.301 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:45.301 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:45.302 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:01:45.302 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:01:45.302 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:01:45.302 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:01:45.303 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:01:45.303 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:01:45.303 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:01:45.303 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:01:45.303 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:01:45.304 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:01:45.304 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:45.304 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:45.304 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:45.304 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:01:45.305 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:01:45.305 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:01:45.305 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:01:45.305 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588579.740560_402.1", 2023-07-29T00:01:45.305 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:01:45.306 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:01:45.306 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:01:45.306 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:56:19.740560Z", 2023-07-29T00:01:45.306 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:45.306 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:45.307 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:45.307 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:01:45.307 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:01:45.307 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:01:45.307 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:01:45.307 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:01:45.308 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:01:45.308 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:01:45.308 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:01:45.308 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:01:45.309 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:45.309 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:45.309 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:45.309 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:01:45.309 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:01:45.310 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:01:45.310 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:01:45.310 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:01:45.310 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:01:45.310 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:01:45.311 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:01:45.311 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:01:45.311 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:45.311 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:45.311 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:45.312 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:01:45.312 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:45.312 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:01:45.312 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:01:45.312 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:01:45.313 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:45.313 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:01:45Z" 2023-07-29T00:01:45.313 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:01:45.314 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588579.740560_402.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:56:19.740560Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:01:45Z" } 2023-07-29T00:01:45.315 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588579.740560_402.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:56:19.740560Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:01:45.315 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:01:45.315 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588579.740560_402.1', 2: '', 3: ''} 2023-07-29T00:01:45.316 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588579.740560_402.1 2023-07-29T00:01:50.317 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:01:50.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:01:50.319 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:01:50.373 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:01:50.373 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:01:50.405 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:01:50.395+0000 7f8c66414880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:01:50.406 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:01:50.396+0000 7f8c66414880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:01:50.418 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:01:50.419 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:01:50.419 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:01:50.419 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:01:50.419 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:01:50.420 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:01:50.420 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:50.420 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:50.420 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:01:50.420 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:01:50.420 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:01:50.420 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:01:50.421 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:01:50.421 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:01:50.421 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:01:50.421 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:01:50.421 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:01:50.422 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:01:50.422 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:50.422 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:50.422 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:50.422 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:01:50.422 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:01:50.422 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:01:50.423 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:01:50.423 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588579.740560_402.1", 2023-07-29T00:01:50.423 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:01:50.423 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:01:50.423 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:01:50.423 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:56:19.740560Z", 2023-07-29T00:01:50.424 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:50.424 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:50.424 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:50.424 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:01:50.424 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:01:50.424 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:01:50.424 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:01:50.425 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:01:50.425 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:01:50.425 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:01:50.425 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:01:50.425 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:01:50.425 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:50.425 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:50.426 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:50.426 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:01:50.426 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:01:50.426 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:01:50.426 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:01:50.426 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:01:50.427 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:01:50.427 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:01:50.427 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:01:50.427 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:01:50.427 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:50.427 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:50.427 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:50.428 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:01:50.428 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:50.428 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:01:50.428 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:01:50.428 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:01:50.428 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:50.428 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:01:50Z" 2023-07-29T00:01:50.429 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:01:50.429 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588579.740560_402.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:56:19.740560Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:01:50Z" } 2023-07-29T00:01:50.430 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588579.740560_402.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:56:19.740560Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:01:50.430 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:01:50.431 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588579.740560_402.1', 2: '', 3: ''} 2023-07-29T00:01:50.431 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588579.740560_402.1 2023-07-29T00:01:55.432 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:01:55.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:01:55.434 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:01:55.487 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:01:55.487 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:01:55.518 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:01:55.507+0000 7f8aa0c22880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:01:55.519 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:01:55.508+0000 7f8aa0c22880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:01:55.531 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:01:55.532 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:01:55.532 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:01:55.532 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:01:55.532 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:01:55.532 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:01:55.533 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:55.533 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:55.533 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:01:55.533 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:01:55.533 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:01:55.533 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:01:55.534 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:01:55.534 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:01:55.534 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:01:55.534 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:01:55.534 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:01:55.535 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:01:55.535 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:55.535 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:55.535 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:55.535 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:01:55.535 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:01:55.535 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:01:55.536 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:01:55.536 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588579.740560_402.1", 2023-07-29T00:01:55.536 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:01:55.536 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:01:55.536 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:01:55.536 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:56:19.740560Z", 2023-07-29T00:01:55.537 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:55.537 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:55.537 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:55.537 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:01:55.537 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:01:55.537 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:01:55.537 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:01:55.538 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:01:55.538 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:01:55.538 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:01:55.538 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:01:55.538 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:01:55.538 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:55.538 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:55.539 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:55.539 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:01:55.539 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:01:55.539 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:01:55.539 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:01:55.539 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:01:55.539 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:01:55.540 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:01:55.540 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:01:55.540 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:01:55.540 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:01:55.540 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:55.540 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:01:55.541 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:01:55.541 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:55.541 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:01:55.541 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:01:55.541 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:01:55.541 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:01:55.541 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:01:55Z" 2023-07-29T00:01:55.542 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:01:55.542 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588579.740560_402.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:56:19.740560Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:01:55Z" } 2023-07-29T00:01:55.543 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588579.740560_402.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:56:19.740560Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:01:55.543 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:01:55.544 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588579.740560_402.1', 2: '', 3: ''} 2023-07-29T00:01:55.544 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588579.740560_402.1 2023-07-29T00:02:00.545 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:02:00.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:02:00.546 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:02:00.599 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:02:00.600 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:02:00.631 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:02:00.621+0000 7fcaccf26880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:02:00.632 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:02:00.622+0000 7fcaccf26880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:02:00.644 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:02:00.644 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:02:00.645 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:02:00.645 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:02:00.645 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:02:00.645 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:02:00.645 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:00.645 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:00.646 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:02:00.646 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:00.646 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:02:00.646 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:00.646 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:00.646 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:02:00.647 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:00.647 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:02:00.647 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:00.647 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:02:00.647 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:00.647 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:00.647 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:00.648 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:00.648 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:02:00.648 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:00.648 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:00.648 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588579.740560_402.1", 2023-07-29T00:02:00.648 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:00.648 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:02:00.649 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:00.649 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:56:19.740560Z", 2023-07-29T00:02:00.649 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:00.649 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:00.649 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:00.649 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:00.649 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:02:00.650 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:00.650 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:00.650 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:02:00.650 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:00.650 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:02:00.650 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:00.650 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:02:00.651 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:00.651 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:00.651 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:00.651 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:00.651 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:02:00.651 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:00.651 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:00.652 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:02:00.652 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:00.652 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:02:00.652 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:00.652 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:02:00.652 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:00.653 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:00.653 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:00.653 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:02:00.653 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:00.653 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:02:00.653 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:02:00.653 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:02:00.654 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:00.654 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:02:00Z" 2023-07-29T00:02:00.654 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:02:00.655 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588579.740560_402.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:56:19.740560Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:02:00Z" } 2023-07-29T00:02:00.655 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588579.740560_402.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:56:19.740560Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:02:00.656 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:02:00.658 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588579.740560_402.1', 2: '', 3: ''} 2023-07-29T00:02:00.658 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588579.740560_402.1 2023-07-29T00:02:05.659 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:02:05.660 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:02:05.660 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:02:05.713 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:02:05.713 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:02:05.744 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:02:05.733+0000 7f1b2de18880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:02:05.744 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:02:05.734+0000 7f1b2de18880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:02:05.757 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:02:05.757 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:02:05.757 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:02:05.758 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:02:05.758 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:02:05.758 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:02:05.758 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:05.758 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:05.759 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:02:05.759 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:05.759 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:02:05.759 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:05.759 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:05.759 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:02:05.760 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:05.760 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:02:05.760 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:05.760 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:02:05.760 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:05.760 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:05.760 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:05.761 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:05.761 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:02:05.761 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:05.761 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:05.761 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588579.740560_402.1", 2023-07-29T00:02:05.761 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:05.761 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:02:05.762 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:05.762 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:56:19.740560Z", 2023-07-29T00:02:05.762 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:05.762 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:05.762 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:05.762 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:05.763 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:02:05.763 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:05.763 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:05.763 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:02:05.763 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:05.763 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:02:05.763 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:05.764 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:02:05.764 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:05.764 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:05.764 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:05.764 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:05.764 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:02:05.765 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:05.765 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:05.765 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:02:05.765 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:05.765 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:02:05.765 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:05.765 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:02:05.766 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:05.766 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:05.766 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:05.766 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:02:05.766 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:05.767 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:02:05.767 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:02:05.767 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:02:05.767 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:05.767 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:02:05Z" 2023-07-29T00:02:05.767 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:02:05.768 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588579.740560_402.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:56:19.740560Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:02:05Z" } 2023-07-29T00:02:05.769 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588579.740560_402.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:56:19.740560Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:02:05.769 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:02:05.769 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588579.740560_402.1', 2: '', 3: ''} 2023-07-29T00:02:05.770 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588579.740560_402.1 2023-07-29T00:02:10.771 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:02:10.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:02:10.772 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:02:10.827 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:02:10.828 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:02:10.859 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:02:10.847+0000 7f412d922880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:02:10.859 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:02:10.848+0000 7f412d922880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:02:10.872 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:02:10.873 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:02:10.873 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:02:10.873 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:02:10.873 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:02:10.873 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:02:10.873 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:10.873 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:10.874 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:02:10.874 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:10.874 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:02:10.874 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:10.874 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:10.874 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:02:10.875 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:10.875 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:02:10.875 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:10.875 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:02:10.875 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:10.875 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:10.875 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:10.876 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:10.876 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:02:10.876 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:10.876 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:10.876 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588579.740560_402.1", 2023-07-29T00:02:10.876 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:10.876 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:02:10.877 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:10.877 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:56:19.740560Z", 2023-07-29T00:02:10.877 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:10.877 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:10.877 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:10.877 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:10.877 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:02:10.878 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:10.878 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:10.878 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:02:10.878 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:10.878 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:02:10.878 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:10.878 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:02:10.879 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:10.879 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:10.879 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:10.879 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:10.879 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:02:10.879 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:10.879 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:10.880 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:02:10.880 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:10.880 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:02:10.880 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:10.880 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:02:10.880 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:10.881 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:10.881 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:10.881 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:02:10.881 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:10.881 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:02:10.881 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:02:10.881 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:02:10.882 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:10.882 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:02:10Z" 2023-07-29T00:02:10.882 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:02:10.883 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588579.740560_402.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:56:19.740560Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:02:10Z" } 2023-07-29T00:02:10.883 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588579.740560_402.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:56:19.740560Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:02:10.884 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:02:10.884 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588579.740560_402.1', 2: '', 3: ''} 2023-07-29T00:02:10.884 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588579.740560_402.1 2023-07-29T00:02:15.885 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:02:15.886 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:02:15.887 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:02:15.940 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:02:15.940 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:02:15.971 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:02:15.959+0000 7f810ea2f880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:02:15.972 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:02:15.960+0000 7f810ea2f880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:02:15.984 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:02:15.984 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:02:15.984 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:02:15.985 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:02:15.985 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:02:15.985 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:02:15.985 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:15.985 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:15.985 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:02:15.986 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:15.986 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:02:15.986 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:15.986 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:15.986 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:02:15.986 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:15.986 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:02:15.987 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:15.987 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:02:15.987 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:15.987 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:15.987 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:15.987 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:15.987 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:02:15.988 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:15.988 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:15.988 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588579.740560_402.1", 2023-07-29T00:02:15.988 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:15.988 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:02:15.988 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:15.988 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:56:19.740560Z", 2023-07-29T00:02:15.989 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:15.989 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:15.989 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:15.989 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:15.989 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:02:15.989 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:15.989 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:15.990 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:02:15.990 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:15.990 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:02:15.990 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:15.990 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:02:15.990 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:15.990 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:15.991 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:15.991 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:15.991 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:02:15.991 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:15.992 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:15.992 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:02:15.992 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:15.992 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:02:15.992 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:15.993 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:02:15.993 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:15.993 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:15.993 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:15.993 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:02:15.993 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:15.994 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:02:15.994 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:02:15.994 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:02:15.994 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:15.994 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:02:15Z" 2023-07-29T00:02:15.994 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:02:15.995 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588579.740560_402.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:56:19.740560Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:02:15Z" } 2023-07-29T00:02:15.996 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588579.740560_402.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:56:19.740560Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:02:15.996 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:02:15.997 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588579.740560_402.1', 2: '', 3: ''} 2023-07-29T00:02:15.997 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588579.740560_402.1 2023-07-29T00:02:20.999 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:02:20.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:02:21.000 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:02:21.053 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:02:21.054 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:02:21.085 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:02:21.073+0000 7f6fb9136880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:02:21.085 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:02:21.074+0000 7f6fb9136880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:02:21.098 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:02:21.098 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:02:21.098 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:02:21.099 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:02:21.099 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:02:21.099 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:02:21.099 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:21.099 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:21.099 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:02:21.100 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:21.100 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:02:21.100 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:21.100 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:21.100 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:02:21.100 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:21.100 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:02:21.101 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:21.101 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:02:21.101 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:21.101 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:21.101 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:21.101 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:21.102 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:02:21.102 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:21.102 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:21.102 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588579.740560_402.1", 2023-07-29T00:02:21.102 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:21.102 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:02:21.102 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:21.103 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:56:19.740560Z", 2023-07-29T00:02:21.103 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:21.103 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:21.103 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:21.103 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:21.103 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:02:21.103 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:21.104 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:21.104 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:02:21.104 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:21.104 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:02:21.104 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:21.104 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:02:21.104 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:21.105 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:21.105 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:21.105 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:21.105 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:02:21.105 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:21.105 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:21.105 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:02:21.106 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:21.106 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:02:21.106 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:21.106 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:02:21.106 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:21.106 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:21.106 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:21.107 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:02:21.107 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:21.107 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:02:21.107 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:02:21.107 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:02:21.107 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:21.108 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:02:21Z" 2023-07-29T00:02:21.108 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:02:21.109 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588579.740560_402.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:56:19.740560Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:02:21Z" } 2023-07-29T00:02:21.109 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588579.740560_402.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:56:19.740560Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:02:21.110 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:02:21.110 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588579.740560_402.1', 2: '', 3: ''} 2023-07-29T00:02:21.110 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588579.740560_402.1 2023-07-29T00:02:26.111 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:02:26.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:02:26.112 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:02:26.166 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:02:26.166 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:02:26.197 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:02:26.185+0000 7f05a5722880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:02:26.198 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:02:26.186+0000 7f05a5722880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:02:26.213 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:02:26.213 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:02:26.213 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:02:26.214 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:02:26.214 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:02:26.214 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:02:26.214 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:26.214 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:26.214 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:02:26.214 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:26.215 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:02:26.215 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:26.215 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:26.215 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:02:26.215 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:26.215 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:02:26.215 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:26.216 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:02:26.216 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:26.217 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:26.217 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:26.218 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:26.218 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:02:26.218 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:26.218 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:26.219 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588579.740560_402.1", 2023-07-29T00:02:26.219 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:26.219 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:02:26.219 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:26.220 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:56:19.740560Z", 2023-07-29T00:02:26.220 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:26.220 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:26.220 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:26.221 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:26.221 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:02:26.221 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:26.221 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:26.222 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:02:26.222 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:26.222 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:02:26.222 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:26.222 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:02:26.223 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:26.223 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:26.223 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:26.223 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:26.224 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:02:26.224 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:26.224 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:26.224 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:02:26.225 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:26.225 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:02:26.225 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:26.226 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:02:26.226 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:26.226 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:26.226 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:26.226 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:02:26.227 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:26.227 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:02:26.227 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:02:26.227 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:02:26.228 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:26.228 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:02:26Z" 2023-07-29T00:02:26.228 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:02:26.229 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588579.740560_402.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:56:19.740560Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:02:26Z" } 2023-07-29T00:02:26.230 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588579.740560_402.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:56:19.740560Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:02:26.230 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:02:26.230 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588579.740560_402.1', 2: '', 3: ''} 2023-07-29T00:02:26.231 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588579.740560_402.1 2023-07-29T00:02:31.232 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:02:31.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:02:31.234 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:02:31.288 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:02:31.288 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:02:31.319 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:02:31.306+0000 7fef2ef24880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:02:31.319 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:02:31.307+0000 7fef2ef24880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:02:31.332 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:02:31.332 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:02:31.332 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:02:31.332 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:02:31.333 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:02:31.333 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:02:31.333 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:31.333 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:31.333 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:02:31.333 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:31.333 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:02:31.334 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:31.334 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:31.334 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:02:31.334 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:31.334 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:02:31.334 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:31.335 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:02:31.335 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:31.335 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:31.335 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:31.335 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:31.335 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:02:31.335 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:31.336 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:31.336 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588579.740560_402.1", 2023-07-29T00:02:31.336 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:31.336 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:02:31.336 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:31.336 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:56:19.740560Z", 2023-07-29T00:02:31.336 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:31.337 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:31.337 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:31.337 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:31.337 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:02:31.337 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:31.337 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:31.337 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:02:31.338 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:31.338 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:02:31.338 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:31.338 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:02:31.338 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:31.338 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:31.338 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:31.339 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:31.339 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:02:31.339 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:31.339 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:31.339 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:02:31.339 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:31.339 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:02:31.340 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:31.340 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:02:31.340 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:31.340 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:31.340 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:31.340 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:02:31.340 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:31.341 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:02:31.341 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:02:31.341 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:02:31.341 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:31.341 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:02:31Z" 2023-07-29T00:02:31.341 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:02:31.342 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588579.740560_402.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:56:19.740560Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:02:31Z" } 2023-07-29T00:02:31.343 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588579.740560_402.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:56:19.740560Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:02:31.343 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:02:31.343 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588579.740560_402.1', 2: '', 3: ''} 2023-07-29T00:02:31.344 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588579.740560_402.1 2023-07-29T00:02:36.345 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:02:36.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:02:36.346 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:02:36.399 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:02:36.400 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:02:36.431 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:02:36.418+0000 7fd1f2726880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:02:36.431 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:02:36.419+0000 7fd1f2726880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:02:36.444 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:02:36.444 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:02:36.445 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:02:36.445 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:02:36.445 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:02:36.445 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:02:36.445 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:36.445 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:36.446 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:02:36.446 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:36.446 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:02:36.446 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:36.446 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:36.446 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:02:36.446 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:36.447 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:02:36.447 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:36.447 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:02:36.447 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:36.447 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:36.447 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:36.447 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:36.448 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:02:36.448 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:36.448 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:36.448 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588579.740560_402.1", 2023-07-29T00:02:36.448 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:36.448 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:02:36.449 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:36.449 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:56:19.740560Z", 2023-07-29T00:02:36.449 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:36.449 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:36.449 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:36.449 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:36.450 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:02:36.450 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:36.450 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:36.450 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:02:36.450 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:36.450 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:02:36.450 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:36.451 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:02:36.451 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:36.451 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:36.451 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:36.451 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:36.451 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:02:36.451 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:36.452 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:36.452 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:02:36.452 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:36.452 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:02:36.452 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:36.453 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:02:36.453 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:36.453 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:36.453 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:36.453 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:02:36.453 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:36.453 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:02:36.454 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:02:36.454 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:02:36.454 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:36.454 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:02:36Z" 2023-07-29T00:02:36.454 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:02:36.455 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588579.740560_402.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:56:19.740560Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:02:36Z" } 2023-07-29T00:02:36.455 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588579.740560_402.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:56:19.740560Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:02:36.456 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:02:36.456 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588579.740560_402.1', 2: '', 3: ''} 2023-07-29T00:02:36.457 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588579.740560_402.1 2023-07-29T00:02:41.458 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:02:41.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:02:41.459 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:02:41.511 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:02:41.511 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:02:41.542 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:02:41.529+0000 7fd929326880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:02:41.543 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:02:41.530+0000 7fd929326880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:02:41.556 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:02:41.556 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:02:41.556 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:02:41.556 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:02:41.556 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:02:41.557 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:02:41.557 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:41.557 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:41.557 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:02:41.557 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:41.557 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:02:41.557 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:41.558 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:41.558 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:02:41.558 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:41.558 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:02:41.558 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:41.558 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:02:41.559 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:41.559 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:41.559 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:41.559 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:41.559 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:02:41.559 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:41.559 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:41.560 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588579.740560_402.1", 2023-07-29T00:02:41.560 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:41.560 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:02:41.560 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:41.560 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:56:19.740560Z", 2023-07-29T00:02:41.560 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:41.560 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:41.561 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:41.561 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:41.561 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:02:41.561 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:41.561 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:41.561 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:02:41.561 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:41.562 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:02:41.562 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:41.562 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:02:41.562 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:41.562 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:41.562 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:41.562 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:41.563 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:02:41.563 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:41.563 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:41.563 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:02:41.563 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:41.563 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:02:41.564 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:41.564 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:02:41.564 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:41.564 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:41.564 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:41.564 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:02:41.564 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:41.565 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:02:41.565 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:02:41.565 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:02:41.565 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:41.565 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:02:41Z" 2023-07-29T00:02:41.565 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:02:41.566 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588579.740560_402.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:56:19.740560Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:02:41Z" } 2023-07-29T00:02:41.567 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588579.740560_402.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:56:19.740560Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:02:41.567 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:02:41.567 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588579.740560_402.1', 2: '', 3: ''} 2023-07-29T00:02:41.568 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588579.740560_402.1 2023-07-29T00:02:46.569 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:02:46.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:02:46.570 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:02:46.646 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:02:46.646 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:02:46.675 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:02:46.661+0000 7f14a9526880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:02:46.675 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:02:46.662+0000 7f14a9526880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:02:46.688 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:02:46.688 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:02:46.688 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:02:46.688 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:02:46.689 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:02:46.689 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:02:46.689 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:46.689 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:46.689 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:02:46.689 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:46.689 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:02:46.690 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:46.690 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:46.690 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:02:46.690 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:46.690 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:02:46.690 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:46.690 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:02:46.691 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:46.691 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:46.691 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:46.691 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:46.691 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:02:46.692 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:46.692 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:46.692 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588866.063571_613.1", 2023-07-29T00:02:46.692 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:46.692 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:02:46.692 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:46.693 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:01:06.063571Z", 2023-07-29T00:02:46.693 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:46.693 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:46.693 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:46.693 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:46.693 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:02:46.693 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:46.694 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:46.694 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:02:46.694 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:46.694 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:02:46.694 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:46.694 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:02:46.694 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:46.695 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:46.695 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:46.695 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:46.695 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:02:46.695 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:46.695 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:46.695 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:02:46.696 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:46.696 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:02:46.696 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:46.696 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:02:46.696 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:46.696 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:46.697 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:46.697 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:02:46.697 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:46.697 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:02:46.697 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:02:46.697 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:02:46.697 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:46.698 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:02:46Z" 2023-07-29T00:02:46.698 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:02:46.699 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588866.063571_613.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:01:06.063571Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:02:46Z" } 2023-07-29T00:02:46.699 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588866.063571_613.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:01:06.063571Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:02:46.699 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:02:46.700 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588866.063571_613.1', 2: '', 3: ''} 2023-07-29T00:02:46.700 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588866.063571_613.1 2023-07-29T00:02:51.701 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:02:51.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:02:51.702 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:02:51.752 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:02:51.752 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:02:51.781 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:02:51.767+0000 7fd74df32880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:02:51.781 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:02:51.768+0000 7fd74df32880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:02:51.793 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:02:51.793 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:02:51.794 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:02:51.794 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:02:51.794 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:02:51.794 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:02:51.794 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:51.794 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:51.794 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:02:51.795 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:51.795 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:02:51.795 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:51.795 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:51.795 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:02:51.795 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:51.796 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:02:51.796 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:51.796 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:02:51.796 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:51.796 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:51.796 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:51.796 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:51.797 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:02:51.797 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:51.797 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:51.797 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588861.032172_612.1", 2023-07-29T00:02:51.797 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:51.797 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:02:51.797 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:51.798 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:01:01.032172Z", 2023-07-29T00:02:51.798 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:51.798 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:51.798 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:51.798 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:51.798 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:02:51.798 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:51.799 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:51.799 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:02:51.799 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:51.799 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:02:51.799 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:51.799 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:02:51.799 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:51.800 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:51.800 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:51.800 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:51.800 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:02:51.800 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:51.800 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:51.800 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:02:51.801 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:51.801 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:02:51.801 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:51.801 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:02:51.801 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:51.801 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:51.801 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:51.802 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:02:51.802 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:51.802 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:02:51.802 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:02:51.802 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:02:51.802 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:51.802 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:02:51Z" 2023-07-29T00:02:51.803 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:02:51.803 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588861.032172_612.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:01:01.032172Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:02:51Z" } 2023-07-29T00:02:51.804 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588861.032172_612.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:01:01.032172Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:02:51.804 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:02:51.805 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588861.032172_612.1', 2: '', 3: ''} 2023-07-29T00:02:51.805 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588861.032172_612.1 2023-07-29T00:02:56.806 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:02:56.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:02:56.808 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:02:56.863 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:02:56.863 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:02:56.893 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:02:56.879+0000 7f939bc14880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:02:56.894 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:02:56.880+0000 7f939bc14880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:02:56.907 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:02:56.907 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:02:56.908 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:02:56.908 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:02:56.908 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:02:56.908 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:02:56.908 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:56.908 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:56.909 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:02:56.909 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:56.909 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:02:56.909 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:56.909 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:56.909 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:02:56.910 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:56.910 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:02:56.910 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:56.910 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:02:56.910 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:56.910 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:56.911 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:56.911 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:56.911 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:02:56.911 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:56.911 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:56.911 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588856.002780_607.1", 2023-07-29T00:02:56.911 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:56.912 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:02:56.912 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:56.912 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:00:56.002780Z", 2023-07-29T00:02:56.912 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:56.912 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:56.912 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:56.912 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:56.913 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:02:56.913 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:56.913 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:56.913 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:02:56.913 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:56.913 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:02:56.913 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:56.914 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:02:56.914 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:56.914 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:56.914 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:56.914 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:02:56.914 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:02:56.914 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:02:56.915 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:02:56.915 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:02:56.915 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:02:56.915 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:02:56.915 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:02:56.915 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:02:56.916 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:02:56.916 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:56.916 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:02:56.916 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:02:56.916 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:56.916 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:02:56.917 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:02:56.917 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:02:56.917 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:02:56.917 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:02:56Z" 2023-07-29T00:02:56.917 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:02:56.918 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588856.002780_607.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:00:56.002780Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:02:56Z" } 2023-07-29T00:02:56.919 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588856.002780_607.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:00:56.002780Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:02:56.919 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:02:56.919 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588856.002780_607.1', 2: '', 3: ''} 2023-07-29T00:02:56.920 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588856.002780_607.1 2023-07-29T00:03:01.921 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:03:01.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:03:01.922 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:03:01.970 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:03:01.971 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:03:01.999 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:03:01.985+0000 7fd04a012880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:03:02.000 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:03:01.986+0000 7fd04a012880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:03:02.013 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:03:02.013 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:03:02.013 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:03:02.013 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:03:02.013 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:03:02.013 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:03:02.014 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:02.014 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:02.014 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:03:02.014 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:02.014 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:03:02.014 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:02.015 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:02.015 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:03:02.015 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:02.015 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:03:02.015 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:02.015 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:03:02.015 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:02.016 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:02.016 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:02.016 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:02.016 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:03:02.016 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:02.016 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:02.016 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588861.032172_612.1", 2023-07-29T00:03:02.017 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:02.017 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:03:02.017 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:02.017 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:01:01.032172Z", 2023-07-29T00:03:02.017 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:02.017 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:02.017 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:02.018 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:02.018 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:03:02.018 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:02.018 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:02.018 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:03:02.018 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:02.018 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:03:02.019 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:02.019 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:03:02.019 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:02.019 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:02.019 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:02.019 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:02.019 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:03:02.020 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:02.020 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:02.020 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:03:02.020 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:02.020 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:03:02.020 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:02.021 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:03:02.021 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:02.021 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:02.021 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:02.021 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:03:02.021 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:02.021 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:03:02.021 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:03:02.022 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:03:02.022 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:02.022 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:03:01Z" 2023-07-29T00:03:02.022 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:03:02.023 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588861.032172_612.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:01:01.032172Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:03:01Z" } 2023-07-29T00:03:02.023 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588861.032172_612.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:01:01.032172Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:03:02.024 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:03:02.024 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588861.032172_612.1', 2: '', 3: ''} 2023-07-29T00:03:02.024 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588861.032172_612.1 2023-07-29T00:03:07.025 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:03:07.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:03:07.027 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:03:07.074 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:03:07.074 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:03:07.102 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:03:07.088+0000 7fcfeb926880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:03:07.103 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:03:07.089+0000 7fcfeb926880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:03:07.115 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:03:07.115 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:03:07.115 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:03:07.115 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:03:07.116 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:03:07.116 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:03:07.116 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:07.116 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:07.116 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:03:07.116 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:07.117 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:03:07.117 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:07.117 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:07.117 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:03:07.117 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:07.117 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:03:07.117 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:07.118 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:03:07.118 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:07.118 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:07.118 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:07.118 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:07.118 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:03:07.118 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:07.119 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:07.119 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588866.067979_615.1", 2023-07-29T00:03:07.119 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:07.119 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:03:07.119 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:07.119 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:01:06.067979Z", 2023-07-29T00:03:07.119 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:07.120 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:07.120 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:07.120 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:07.120 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:03:07.120 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:07.120 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:07.121 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:03:07.121 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:07.121 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:03:07.121 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:07.121 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:03:07.122 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:07.122 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:07.122 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:07.122 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:07.122 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:03:07.122 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:07.122 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:07.123 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:03:07.123 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:07.123 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:03:07.123 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:07.123 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:03:07.123 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:07.123 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:07.123 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:07.124 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:03:07.124 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:07.124 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:03:07.124 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:03:07.124 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:03:07.124 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:07.125 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:03:07Z" 2023-07-29T00:03:07.125 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:03:07.125 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588866.067979_615.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:01:06.067979Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:03:07Z" } 2023-07-29T00:03:07.126 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588866.067979_615.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:01:06.067979Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:03:07.126 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:03:07.127 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588866.067979_615.1', 2: '', 3: ''} 2023-07-29T00:03:07.127 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588866.067979_615.1 2023-07-29T00:03:12.128 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:03:12.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:03:12.130 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:03:12.175 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:03:12.175 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:03:12.205 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:03:12.190+0000 7fcb33f26880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:03:12.206 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:03:12.191+0000 7fcb33f26880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:03:12.218 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:03:12.218 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:03:12.218 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:03:12.218 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:03:12.218 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:03:12.219 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:03:12.219 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:12.219 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:12.219 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:03:12.219 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:12.219 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:03:12.219 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:12.220 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:12.220 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:03:12.220 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:12.220 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:03:12.220 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:12.220 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:03:12.221 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:12.221 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:12.221 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:12.221 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:12.221 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:03:12.221 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:12.221 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:12.222 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588871.100165_617.1", 2023-07-29T00:03:12.222 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:12.222 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:03:12.222 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:12.222 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:01:11.100165Z", 2023-07-29T00:03:12.222 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:12.222 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:12.223 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:12.223 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:12.223 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:03:12.223 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:12.223 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:12.223 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:03:12.223 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:12.224 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:03:12.224 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:12.224 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:03:12.224 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:12.224 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:12.224 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:12.224 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:12.225 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:03:12.225 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:12.225 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:12.225 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:03:12.225 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:12.225 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:03:12.225 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:12.226 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:03:12.226 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:12.226 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:12.226 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:12.226 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:03:12.226 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:12.226 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:03:12.227 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:03:12.227 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:03:12.227 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:12.227 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:03:12Z" 2023-07-29T00:03:12.228 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:03:12.229 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588871.100165_617.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:01:11.100165Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:03:12Z" } 2023-07-29T00:03:12.229 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588871.100165_617.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:01:11.100165Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:03:12.230 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:03:12.230 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588871.100165_617.1', 2: '', 3: ''} 2023-07-29T00:03:12.231 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588871.100165_617.1 2023-07-29T00:03:17.232 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:03:17.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:03:17.233 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:03:17.280 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:03:17.281 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:03:17.309 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:03:17.294+0000 7f2b8e28b880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:03:17.309 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:03:17.294+0000 7f2b8e28b880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:03:17.322 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:03:17.322 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:03:17.322 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:03:17.322 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:03:17.322 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:03:17.323 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:03:17.323 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:17.323 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:17.323 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:03:17.323 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:17.323 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:03:17.323 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:17.324 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:17.324 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:03:17.324 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:17.324 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:03:17.324 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:17.325 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:03:17.325 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:17.325 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:17.325 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:17.325 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:17.325 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:03:17.325 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:17.326 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:17.326 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588856.002780_607.1", 2023-07-29T00:03:17.326 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:17.326 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:03:17.326 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:17.327 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:00:56.002780Z", 2023-07-29T00:03:17.327 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:17.327 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:17.327 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:17.327 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:17.327 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:03:17.327 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:17.328 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:17.328 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:03:17.328 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:17.328 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:03:17.328 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:17.328 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:03:17.329 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:17.329 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:17.329 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:17.329 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:17.329 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:03:17.329 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:17.329 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:17.330 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:03:17.330 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:17.330 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:03:17.330 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:17.330 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:03:17.330 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:17.330 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:17.331 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:17.331 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:03:17.331 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:17.331 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:03:17.331 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:03:17.331 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:03:17.331 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:17.332 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:03:17Z" 2023-07-29T00:03:17.332 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:03:17.333 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588856.002780_607.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:00:56.002780Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:03:17Z" } 2023-07-29T00:03:17.333 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588856.002780_607.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:00:56.002780Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:03:17.333 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:03:17.334 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588856.002780_607.1', 2: '', 3: ''} 2023-07-29T00:03:17.334 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588856.002780_607.1 2023-07-29T00:03:22.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:03:22.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:03:22.337 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:03:22.392 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:03:22.392 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:03:22.423 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:03:22.407+0000 7f948548b880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:03:22.424 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:03:22.408+0000 7f948548b880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:03:22.447 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:03:22.447 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:03:22.448 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:03:22.448 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:03:22.448 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:03:22.448 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:03:22.448 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:22.448 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:22.449 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:03:22.449 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:22.449 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:03:22.449 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:22.449 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:22.449 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:03:22.450 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:22.450 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:03:22.450 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:22.450 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:03:22.450 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:22.450 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:22.450 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:22.451 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:22.451 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:03:22.451 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:22.451 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:22.451 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588861.029063_611.1", 2023-07-29T00:03:22.451 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:22.451 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:03:22.452 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:22.452 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:01:01.029063Z", 2023-07-29T00:03:22.452 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:22.452 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:22.452 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:22.452 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:22.452 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:03:22.453 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:22.453 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:22.453 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:03:22.453 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:22.453 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:03:22.453 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:22.454 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:03:22.454 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:22.454 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:22.454 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:22.454 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:22.454 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:03:22.454 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:22.455 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:22.455 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:03:22.455 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:22.455 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:03:22.455 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:22.455 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:03:22.456 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:22.456 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:22.456 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:22.456 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:03:22.456 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:22.456 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:03:22.457 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:03:22.457 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:03:22.457 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:22.457 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:03:22Z" 2023-07-29T00:03:22.457 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:03:22.458 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588861.029063_611.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:01:01.029063Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:03:22Z" } 2023-07-29T00:03:22.459 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588861.029063_611.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:01:01.029063Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:03:22.459 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:03:22.460 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588861.029063_611.1', 2: '', 3: ''} 2023-07-29T00:03:22.460 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588861.029063_611.1 2023-07-29T00:03:27.461 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:03:27.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:03:27.463 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:03:27.516 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:03:27.516 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:03:27.544 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:03:27.528+0000 7fea81213880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:03:27.545 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:03:27.529+0000 7fea81213880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:03:27.557 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:03:27.557 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:03:27.557 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:03:27.558 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:03:27.558 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:03:27.558 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:03:27.558 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:27.558 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:27.559 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:03:27.559 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:27.559 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:03:27.559 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:27.559 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:27.559 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:03:27.560 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:27.560 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:03:27.560 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:27.560 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:03:27.561 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:27.561 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:27.561 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:27.561 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:27.561 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:03:27.562 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:27.562 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:27.562 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588866.063571_613.1", 2023-07-29T00:03:27.562 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:27.562 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:03:27.562 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:27.562 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:01:06.063571Z", 2023-07-29T00:03:27.563 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:27.563 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:27.563 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:27.563 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:27.563 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:03:27.563 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:27.563 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:27.564 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:03:27.564 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:27.564 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:03:27.564 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:27.564 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:03:27.564 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:27.565 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:27.565 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:27.565 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:27.565 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:03:27.565 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:27.565 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:27.565 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:03:27.566 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:27.566 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:03:27.566 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:27.566 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:03:27.566 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:27.566 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:27.566 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:27.567 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:03:27.567 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:27.567 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:03:27.567 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:03:27.567 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:03:27.567 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:27.567 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:03:27Z" 2023-07-29T00:03:27.568 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:03:27.568 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588866.063571_613.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:01:06.063571Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:03:27Z" } 2023-07-29T00:03:27.569 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588866.063571_613.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:01:06.063571Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:03:27.569 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:03:27.570 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588866.063571_613.1', 2: '', 3: ''} 2023-07-29T00:03:27.570 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588866.063571_613.1 2023-07-29T00:03:32.571 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:03:32.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:03:32.572 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:03:32.622 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:03:32.622 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:03:32.650 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:03:32.634+0000 7f9751d22880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:03:32.651 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:03:32.635+0000 7f9751d22880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:03:32.663 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:03:32.663 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:03:32.663 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:03:32.664 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:03:32.664 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:03:32.664 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:03:32.664 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:32.664 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:32.664 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:03:32.665 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:32.665 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:03:32.665 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:32.665 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:32.665 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:03:32.666 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:32.666 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:03:32.666 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:32.666 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:03:32.666 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:32.666 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:32.667 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:32.667 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:32.667 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:03:32.667 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:32.667 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:32.667 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588856.005386_608.1", 2023-07-29T00:03:32.668 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:32.668 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:03:32.668 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:32.668 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:00:56.005386Z", 2023-07-29T00:03:32.668 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:32.668 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:32.668 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:32.669 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:32.669 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:03:32.669 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:32.669 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:32.669 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:03:32.669 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:32.669 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:03:32.670 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:32.670 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:03:32.670 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:32.670 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:32.670 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:32.671 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:32.671 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:03:32.671 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:32.671 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:32.671 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:03:32.671 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:32.671 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:03:32.672 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:32.672 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:03:32.672 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:32.672 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:32.672 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:32.672 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:03:32.673 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:32.673 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:03:32.673 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:03:32.673 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:03:32.673 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:32.673 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:03:32Z" 2023-07-29T00:03:32.673 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:03:32.674 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588856.005386_608.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:00:56.005386Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:03:32Z" } 2023-07-29T00:03:32.675 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588856.005386_608.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:00:56.005386Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:03:32.675 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:03:32.675 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588856.005386_608.1', 2: '', 3: ''} 2023-07-29T00:03:32.676 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588856.005386_608.1 2023-07-29T00:03:37.676 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:03:37.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:03:37.678 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:03:37.731 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:03:37.731 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:03:37.763 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:03:37.747+0000 7f3bba922880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:03:37.764 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:03:37.748+0000 7f3bba922880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:03:37.777 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:03:37.777 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:03:37.777 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:03:37.778 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:03:37.778 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:03:37.778 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:03:37.778 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:37.778 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:37.778 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:03:37.778 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:37.779 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:03:37.779 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:37.779 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:37.779 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:03:37.779 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:37.779 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:03:37.780 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:37.780 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:03:37.780 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:37.780 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:37.780 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:37.780 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:37.780 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:03:37.781 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:37.781 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:37.781 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588856.002780_607.1", 2023-07-29T00:03:37.781 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:37.781 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:03:37.781 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:37.781 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:00:56.002780Z", 2023-07-29T00:03:37.782 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:37.782 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:37.782 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:37.782 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:37.782 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:03:37.782 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:37.782 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:37.783 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:03:37.783 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:37.783 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:03:37.783 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:37.783 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:03:37.783 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:37.783 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:37.784 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:37.784 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:37.784 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:03:37.784 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:37.784 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:37.784 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:03:37.785 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:37.785 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:03:37.785 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:37.785 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:03:37.785 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:37.785 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:37.785 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:37.786 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:03:37.786 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:37.786 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:03:37.786 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:03:37.786 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:03:37.786 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:37.786 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:03:37Z" 2023-07-29T00:03:37.787 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:03:37.787 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588856.002780_607.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:00:56.002780Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:03:37Z" } 2023-07-29T00:03:37.788 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588856.002780_607.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:00:56.002780Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:03:37.788 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:03:37.789 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588856.002780_607.1', 2: '', 3: ''} 2023-07-29T00:03:37.789 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588856.002780_607.1 2023-07-29T00:03:42.790 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:03:42.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:03:42.791 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:03:42.843 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:03:42.844 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:03:42.874 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:03:42.858+0000 7f14da522880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:03:42.875 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:03:42.859+0000 7f14da522880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:03:42.888 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:03:42.888 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:03:42.888 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:03:42.889 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:03:42.889 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:03:42.889 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:03:42.889 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:42.889 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:42.889 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:03:42.889 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:42.890 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:03:42.890 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:42.890 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:42.890 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:03:42.890 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:42.890 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:03:42.890 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:42.891 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:03:42.891 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:42.891 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:42.891 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:42.891 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:42.891 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:03:42.891 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:42.892 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:42.892 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588861.024534_609.1", 2023-07-29T00:03:42.892 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:42.892 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:03:42.892 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:42.892 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:01:01.024534Z", 2023-07-29T00:03:42.893 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:42.893 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:42.893 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:42.893 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:42.893 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:03:42.893 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:42.893 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:42.894 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:03:42.894 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:42.894 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:03:42.894 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:42.894 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:03:42.894 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:42.894 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:42.895 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:42.895 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:42.895 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:03:42.895 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:42.895 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:42.895 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:03:42.895 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:42.896 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:03:42.896 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:42.896 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:03:42.896 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:42.896 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:42.896 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:42.896 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:03:42.897 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:42.897 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:03:42.897 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:03:42.897 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:03:42.897 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:42.897 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:03:42Z" 2023-07-29T00:03:42.897 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:03:42.898 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588861.024534_609.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:01:01.024534Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:03:42Z" } 2023-07-29T00:03:42.899 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588861.024534_609.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:01:01.024534Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:03:42.899 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:03:42.900 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588861.024534_609.1', 2: '', 3: ''} 2023-07-29T00:03:42.900 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588861.024534_609.1 2023-07-29T00:03:47.901 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:03:47.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:03:47.903 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:03:47.949 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:03:47.949 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:03:47.977 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:03:47.960+0000 7f4a43c11880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:03:47.977 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:03:47.961+0000 7f4a43c11880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:03:47.990 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:03:47.990 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:03:47.990 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:03:47.990 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:03:47.991 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:03:47.991 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:03:47.991 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:47.991 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:47.991 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:03:47.991 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:47.992 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:03:47.992 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:47.992 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:47.992 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:03:47.992 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:47.993 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:03:47.993 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:47.993 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:03:47.993 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:47.993 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:47.994 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:47.994 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:47.994 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:03:47.994 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:47.994 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:47.994 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588871.149758_624.1", 2023-07-29T00:03:47.995 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:47.995 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:03:47.995 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:47.995 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:01:11.149758Z", 2023-07-29T00:03:47.995 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:47.996 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:47.996 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:47.996 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:47.996 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:03:47.996 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:47.997 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:47.997 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:03:47.997 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:47.997 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:03:47.998 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:47.998 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:03:47.998 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:47.999 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:47.999 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:47.999 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:47.999 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:03:48.000 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:48.000 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:48.000 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:03:48.001 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:48.001 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:03:48.001 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:48.002 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:03:48.002 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:48.002 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:48.002 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:48.003 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:03:48.003 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:48.003 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:03:48.004 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:03:48.004 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:03:48.004 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:48.005 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:03:47Z" 2023-07-29T00:03:48.005 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:03:48.006 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588871.149758_624.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:01:11.149758Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:03:47Z" } 2023-07-29T00:03:48.007 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588871.149758_624.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:01:11.149758Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:03:48.008 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:03:48.009 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588871.149758_624.1', 2: '', 3: ''} 2023-07-29T00:03:48.009 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588871.149758_624.1 2023-07-29T00:03:53.010 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:03:53.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:03:53.012 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:03:53.062 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:03:53.062 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:03:53.092 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:03:53.075+0000 7f511d896880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:03:53.093 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:03:53.076+0000 7f511d896880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:03:53.106 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:03:53.106 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:03:53.106 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:03:53.106 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:03:53.107 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:03:53.107 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:03:53.107 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:53.107 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:53.107 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:03:53.107 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:53.108 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:03:53.108 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:53.108 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:53.108 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:03:53.108 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:53.108 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:03:53.109 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:53.109 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:03:53.109 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:53.109 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:53.109 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:53.109 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:53.110 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:03:53.110 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:53.110 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:53.110 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588861.024534_609.1", 2023-07-29T00:03:53.110 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:53.110 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:03:53.110 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:53.111 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:01:01.024534Z", 2023-07-29T00:03:53.111 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:53.111 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:53.111 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:53.111 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:53.111 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:03:53.112 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:53.112 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:53.112 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:03:53.112 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:53.112 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:03:53.113 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:53.113 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:03:53.113 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:53.113 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:53.113 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:53.113 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:53.113 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:03:53.114 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:53.114 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:53.114 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:03:53.114 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:53.114 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:03:53.114 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:53.115 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:03:53.115 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:53.115 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:53.115 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:53.115 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:03:53.115 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:53.115 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:03:53.116 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:03:53.116 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:03:53.116 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:53.116 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:03:53Z" 2023-07-29T00:03:53.116 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:03:53.117 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588861.024534_609.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:01:01.024534Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:03:53Z" } 2023-07-29T00:03:53.118 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588861.024534_609.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:01:01.024534Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:03:53.118 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:03:53.118 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588861.024534_609.1', 2: '', 3: ''} 2023-07-29T00:03:53.119 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588861.024534_609.1 2023-07-29T00:03:58.120 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:03:58.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:03:58.122 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:03:58.175 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:03:58.175 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:03:58.207 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:03:58.189+0000 7f6b2751f880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:03:58.208 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:03:58.190+0000 7f6b2751f880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:03:58.221 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:03:58.221 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:03:58.222 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:03:58.222 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:03:58.222 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:03:58.222 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:03:58.222 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:58.222 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:58.223 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:03:58.223 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:58.223 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:03:58.223 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:58.223 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:58.223 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:03:58.223 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:58.224 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:03:58.224 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:58.224 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:03:58.224 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:58.224 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:58.224 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:58.225 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:58.225 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:03:58.225 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:58.225 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:58.225 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588866.070914_616.1", 2023-07-29T00:03:58.225 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:58.225 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:03:58.226 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:58.226 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:01:06.070914Z", 2023-07-29T00:03:58.226 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:58.226 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:58.226 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:58.226 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:58.226 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:03:58.227 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:58.227 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:58.227 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:03:58.227 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:58.227 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:03:58.227 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:58.227 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:03:58.228 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:58.228 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:58.228 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:58.228 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:03:58.228 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:03:58.228 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:03:58.229 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:03:58.229 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:03:58.229 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:03:58.229 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:03:58.229 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:03:58.229 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:03:58.229 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:03:58.230 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:58.230 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:03:58.230 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:03:58.230 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:58.231 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:03:58.231 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:03:58.232 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:03:58.232 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:03:58.232 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:03:58Z" 2023-07-29T00:03:58.232 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:03:58.233 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588866.070914_616.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:01:06.070914Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:03:58Z" } 2023-07-29T00:03:58.234 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588866.070914_616.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:01:06.070914Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:03:58.234 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:03:58.234 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588866.070914_616.1', 2: '', 3: ''} 2023-07-29T00:03:58.235 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588866.070914_616.1 2023-07-29T00:04:03.235 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:04:03.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:04:03.237 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:04:03.286 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:04:03.286 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:04:03.315 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:04:03.297+0000 7f735aa2d880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:04:03.316 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:04:03.298+0000 7f735aa2d880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:04:03.335 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:04:03.335 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:04:03.336 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:04:03.336 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:04:03.336 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:04:03.336 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:04:03.336 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:03.336 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:03.337 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:04:03.337 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:03.337 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:04:03.337 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:03.337 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:03.337 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:04:03.338 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:03.338 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:04:03.338 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:03.338 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:04:03.338 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:03.338 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:03.339 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:03.339 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:03.339 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:04:03.340 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:03.340 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:03.340 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588866.063571_613.1", 2023-07-29T00:04:03.340 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:03.340 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:04:03.340 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:03.341 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:01:06.063571Z", 2023-07-29T00:04:03.341 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:03.341 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:03.341 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:03.341 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:03.341 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:04:03.342 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:03.342 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:03.342 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:04:03.342 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:03.342 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:04:03.342 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:03.342 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:04:03.343 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:03.343 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:03.343 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:03.343 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:03.343 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:04:03.343 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:03.344 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:03.344 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:04:03.344 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:03.344 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:04:03.345 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:03.345 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:04:03.345 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:03.345 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:03.346 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:03.346 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:04:03.346 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:03.347 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:04:03.347 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:04:03.347 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:04:03.347 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:03.348 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:04:03Z" 2023-07-29T00:04:03.348 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:04:03.350 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588866.063571_613.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:01:06.063571Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:04:03Z" } 2023-07-29T00:04:03.351 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588866.063571_613.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:01:06.063571Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:04:03.352 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:04:03.353 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588866.063571_613.1', 2: '', 3: ''} 2023-07-29T00:04:03.354 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588866.063571_613.1 2023-07-29T00:04:08.355 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:04:08.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:04:08.356 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:04:08.411 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:04:08.411 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:04:08.442 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:04:08.424+0000 7f9617d24880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:04:08.443 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:04:08.425+0000 7f9617d24880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:04:08.456 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:04:08.456 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:04:08.456 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:04:08.456 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:04:08.456 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:04:08.457 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:04:08.457 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:08.457 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:08.457 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:04:08.457 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:08.457 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:04:08.458 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:08.458 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:08.458 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:04:08.458 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:08.458 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:04:08.458 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:08.458 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:04:08.459 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:08.459 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:08.459 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:08.459 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:08.459 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:04:08.459 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:08.459 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:08.460 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588856.005386_608.1", 2023-07-29T00:04:08.460 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:08.460 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:04:08.460 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:08.460 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:00:56.005386Z", 2023-07-29T00:04:08.460 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:08.460 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:08.461 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:08.461 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:08.461 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:04:08.461 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:08.461 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:08.461 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:04:08.461 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:08.462 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:04:08.462 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:08.462 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:04:08.462 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:08.462 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:08.462 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:08.463 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:08.463 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:04:08.463 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:08.463 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:08.463 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:04:08.463 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:08.463 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:04:08.464 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:08.464 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:04:08.464 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:08.464 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:08.464 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:08.464 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:04:08.465 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:08.465 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:04:08.465 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:04:08.465 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:04:08.465 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:08.465 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:04:08Z" 2023-07-29T00:04:08.465 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:04:08.466 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588856.005386_608.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:00:56.005386Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:04:08Z" } 2023-07-29T00:04:08.467 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588856.005386_608.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:00:56.005386Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:04:08.467 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:04:08.468 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588856.005386_608.1', 2: '', 3: ''} 2023-07-29T00:04:08.468 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588856.005386_608.1 2023-07-29T00:04:13.469 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:04:13.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:04:13.471 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:04:13.530 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:04:13.530 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:04:13.558 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:04:13.540+0000 7ff480426880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:04:13.559 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:04:13.541+0000 7ff480426880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:04:13.571 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:04:13.571 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:04:13.571 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:04:13.572 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:04:13.572 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:04:13.572 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:04:13.572 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:13.572 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:13.572 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:04:13.572 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:13.573 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:04:13.573 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:13.573 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:13.573 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:04:13.573 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:13.573 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:04:13.574 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:13.574 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:04:13.574 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:13.574 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:13.574 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:13.575 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:13.575 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:04:13.575 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:13.575 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:13.575 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588861.027762_610.1", 2023-07-29T00:04:13.575 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:13.575 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:04:13.576 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:13.576 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:01:01.027762Z", 2023-07-29T00:04:13.576 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:13.576 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:13.576 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:13.576 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:13.576 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:04:13.577 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:13.577 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:13.577 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:04:13.577 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:13.577 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:04:13.577 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:13.578 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:04:13.578 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:13.578 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:13.578 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:13.578 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:13.578 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:04:13.578 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:13.579 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:13.579 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:04:13.579 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:13.579 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:04:13.579 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:13.579 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:04:13.579 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:13.580 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:13.580 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:13.580 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:04:13.580 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:13.580 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:04:13.580 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:04:13.580 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:04:13.581 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:13.581 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:04:13Z" 2023-07-29T00:04:13.581 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:04:13.582 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588861.027762_610.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:01:01.027762Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:04:13Z" } 2023-07-29T00:04:13.582 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588861.027762_610.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:01:01.027762Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:04:13.583 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:04:13.583 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588861.027762_610.1', 2: '', 3: ''} 2023-07-29T00:04:13.583 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588861.027762_610.1 2023-07-29T00:04:18.584 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:04:18.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:04:18.586 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:04:18.639 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:04:18.639 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:04:18.671 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:04:18.653+0000 7fdd0f324880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:04:18.672 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:04:18.654+0000 7fdd0f324880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:04:18.686 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:04:18.686 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:04:18.686 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:04:18.686 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:04:18.686 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:04:18.687 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:04:18.687 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:18.687 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:18.687 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:04:18.687 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:18.687 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:04:18.687 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:18.688 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:18.688 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:04:18.688 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:18.688 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:04:18.688 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:18.688 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:04:18.689 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:18.689 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:18.689 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:18.689 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:18.689 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:04:18.690 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:18.690 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:18.690 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588861.032172_612.1", 2023-07-29T00:04:18.690 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:18.690 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:04:18.690 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:18.691 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:01:01.032172Z", 2023-07-29T00:04:18.691 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:18.691 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:18.691 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:18.691 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:18.692 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:04:18.692 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:18.692 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:18.692 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:04:18.692 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:18.692 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:04:18.692 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:18.693 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:04:18.693 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:18.693 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:18.693 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:18.693 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:18.693 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:04:18.694 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:18.694 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:18.694 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:04:18.694 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:18.694 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:04:18.694 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:18.694 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:04:18.695 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:18.695 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:18.695 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:18.695 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:04:18.695 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:18.695 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:04:18.695 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:04:18.696 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:04:18.696 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:18.696 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:04:18Z" 2023-07-29T00:04:18.696 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:04:18.697 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588861.032172_612.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:01:01.032172Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:04:18Z" } 2023-07-29T00:04:18.697 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588861.032172_612.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:01:01.032172Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:04:18.698 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:04:18.698 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588861.032172_612.1', 2: '', 3: ''} 2023-07-29T00:04:18.699 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588861.032172_612.1 2023-07-29T00:04:23.700 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:04:23.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:04:23.701 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:04:23.754 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:04:23.754 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:04:23.786 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:04:23.767+0000 7f15eb12a880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:04:23.787 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:04:23.768+0000 7f15eb12a880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:04:23.800 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:04:23.800 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:04:23.800 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:04:23.800 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:04:23.800 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:04:23.801 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:04:23.801 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:23.801 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:23.801 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:04:23.801 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:23.801 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:04:23.802 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:23.802 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:23.802 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:04:23.802 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:23.802 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:04:23.802 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:23.802 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:04:23.803 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:23.803 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:23.803 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:23.803 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:23.803 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:04:23.803 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:23.803 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:23.804 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588866.063571_613.1", 2023-07-29T00:04:23.804 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:23.804 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:04:23.804 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:23.804 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:01:06.063571Z", 2023-07-29T00:04:23.804 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:23.805 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:23.805 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:23.805 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:23.805 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:04:23.805 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:23.805 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:23.806 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:04:23.806 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:23.806 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:04:23.806 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:23.806 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:04:23.806 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:23.806 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:23.807 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:23.807 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:23.807 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:04:23.807 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:23.807 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:23.807 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:04:23.807 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:23.808 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:04:23.808 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:23.808 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:04:23.808 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:23.808 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:23.808 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:23.808 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:04:23.809 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:23.809 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:04:23.809 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:04:23.809 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:04:23.809 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:23.809 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:04:23Z" 2023-07-29T00:04:23.809 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:04:23.810 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588866.063571_613.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:01:06.063571Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:04:23Z" } 2023-07-29T00:04:23.811 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588866.063571_613.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:01:06.063571Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:04:23.811 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:04:23.812 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588866.063571_613.1', 2: '', 3: ''} 2023-07-29T00:04:23.812 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588866.063571_613.1 2023-07-29T00:04:28.813 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:04:28.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:04:28.814 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:04:28.863 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:04:28.864 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:04:28.893 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:04:28.874+0000 7f30a9612880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:04:28.893 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:04:28.875+0000 7f30a9612880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:04:28.906 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:04:28.906 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:04:28.906 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:04:28.906 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:04:28.907 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:04:28.907 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:04:28.907 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:28.907 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:28.907 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:04:28.907 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:28.907 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:04:28.908 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:28.908 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:28.908 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:04:28.908 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:28.908 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:04:28.908 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:28.909 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:04:28.909 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:28.909 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:28.909 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:28.909 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:28.909 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:04:28.909 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:28.910 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:28.910 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588871.104645_619.1", 2023-07-29T00:04:28.910 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:28.910 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:04:28.910 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:28.911 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:01:11.104645Z", 2023-07-29T00:04:28.911 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:28.911 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:28.911 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:28.911 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:28.911 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:04:28.911 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:28.912 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:28.912 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:04:28.912 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:28.912 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:04:28.912 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:28.912 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:04:28.913 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:28.913 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:28.913 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:28.913 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:28.913 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:04:28.913 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:28.913 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:28.914 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:04:28.914 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:28.914 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:04:28.914 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:28.914 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:04:28.914 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:28.914 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:28.915 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:28.915 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:04:28.915 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:28.915 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:04:28.915 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:04:28.915 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:04:28.915 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:28.916 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:04:28Z" 2023-07-29T00:04:28.916 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:04:28.917 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588871.104645_619.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:01:11.104645Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:04:28Z" } 2023-07-29T00:04:28.917 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588871.104645_619.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:01:11.104645Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:04:28.918 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:04:28.918 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588871.104645_619.1', 2: '', 3: ''} 2023-07-29T00:04:28.918 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588871.104645_619.1 2023-07-29T00:04:33.919 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:04:33.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:04:33.921 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:04:33.973 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:04:33.974 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:04:34.005 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:04:33.986+0000 7f68dca9d880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:04:34.006 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:04:33.987+0000 7f68dca9d880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:04:34.019 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:04:34.019 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:04:34.019 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:04:34.020 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:04:34.020 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:04:34.020 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:04:34.020 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:34.020 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:34.020 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:04:34.021 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:34.021 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:04:34.021 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:34.021 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:34.021 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:04:34.021 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:34.021 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:04:34.022 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:34.022 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:04:34.022 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:34.022 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:34.022 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:34.022 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:34.022 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:04:34.023 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:34.023 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:34.023 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588856.005386_608.1", 2023-07-29T00:04:34.023 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:34.023 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:04:34.023 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:34.023 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:00:56.005386Z", 2023-07-29T00:04:34.024 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:34.024 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:34.024 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:34.024 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:34.024 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:04:34.024 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:34.024 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:34.025 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:04:34.025 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:34.025 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:04:34.025 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:34.025 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:04:34.025 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:34.025 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:34.026 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:34.026 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:34.026 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:04:34.026 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:34.026 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:34.026 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:04:34.027 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:34.027 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:04:34.027 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:34.027 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:04:34.027 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:34.027 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:34.027 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:34.027 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:04:34.028 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:34.028 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:04:34.028 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:04:34.028 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:04:34.028 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:34.028 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:04:33Z" 2023-07-29T00:04:34.029 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:04:34.029 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588856.005386_608.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:00:56.005386Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:04:33Z" } 2023-07-29T00:04:34.030 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588856.005386_608.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:00:56.005386Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:04:34.030 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:04:34.031 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588856.005386_608.1', 2: '', 3: ''} 2023-07-29T00:04:34.031 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588856.005386_608.1 2023-07-29T00:04:39.032 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:04:39.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:04:39.033 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:04:39.086 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:04:39.086 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:04:39.117 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:04:39.098+0000 7fecbf624880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:04:39.118 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:04:39.099+0000 7fecbf624880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:04:39.131 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:04:39.132 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:04:39.132 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:04:39.132 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:04:39.132 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:04:39.132 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:04:39.133 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:39.133 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:39.133 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:04:39.133 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:39.133 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:04:39.133 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:39.134 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:39.134 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:04:39.134 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:39.134 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:04:39.134 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:39.134 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:04:39.134 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:39.135 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:39.135 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:39.135 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:39.135 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:04:39.135 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:39.135 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:39.135 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588861.024534_609.1", 2023-07-29T00:04:39.136 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:39.136 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:04:39.136 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:39.136 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:01:01.024534Z", 2023-07-29T00:04:39.136 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:39.136 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:39.137 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:39.137 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:39.137 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:04:39.137 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:39.137 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:39.138 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:04:39.138 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:39.138 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:04:39.138 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:39.139 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:04:39.139 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:39.139 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:39.140 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:39.140 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:39.140 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:04:39.141 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:39.141 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:39.141 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:04:39.141 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:39.142 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:04:39.142 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:39.142 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:04:39.143 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:39.143 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:39.143 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:39.143 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:04:39.144 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:39.144 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:04:39.144 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:04:39.145 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:04:39.145 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:39.145 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:04:39Z" 2023-07-29T00:04:39.145 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:04:39.147 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588861.024534_609.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:01:01.024534Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:04:39Z" } 2023-07-29T00:04:39.148 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588861.024534_609.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:01:01.024534Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:04:39.148 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:04:39.149 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588861.024534_609.1', 2: '', 3: ''} 2023-07-29T00:04:39.150 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588861.024534_609.1 2023-07-29T00:04:44.151 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:04:44.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:04:44.152 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:04:44.202 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:04:44.202 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:04:44.233 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:04:44.214+0000 7f70ed026880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:04:44.234 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:04:44.215+0000 7f70ed026880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:04:44.246 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:04:44.246 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:04:44.247 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:04:44.247 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:04:44.247 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:04:44.248 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:04:44.248 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:44.248 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:44.249 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:04:44.249 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:44.249 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:04:44.249 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:44.250 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:44.250 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:04:44.250 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:44.251 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:04:44.251 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:44.251 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:04:44.252 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:44.252 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:44.252 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:44.252 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:44.253 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:04:44.253 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:44.253 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:44.254 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588866.063571_613.1", 2023-07-29T00:04:44.254 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:44.254 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:04:44.254 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:44.255 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:01:06.063571Z", 2023-07-29T00:04:44.255 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:44.255 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:44.256 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:44.256 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:44.256 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:04:44.256 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:44.257 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:44.257 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:04:44.257 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:44.258 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:04:44.258 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:44.258 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:04:44.258 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:44.259 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:44.259 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:44.259 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:44.260 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:04:44.260 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:44.260 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:44.260 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:04:44.261 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:44.261 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:04:44.261 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:44.262 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:04:44.262 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:44.262 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:44.262 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:44.263 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:04:44.263 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:44.263 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:04:44.264 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:04:44.264 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:04:44.264 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:44.264 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:04:44Z" 2023-07-29T00:04:44.265 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:04:44.266 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588866.063571_613.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:01:06.063571Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:04:44Z" } 2023-07-29T00:04:44.267 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588866.063571_613.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:01:06.063571Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:04:44.268 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:04:44.268 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588866.063571_613.1', 2: '', 3: ''} 2023-07-29T00:04:44.269 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588866.063571_613.1 2023-07-29T00:04:49.270 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:04:49.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:04:49.271 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:04:49.324 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:04:49.324 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:04:49.356 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:04:49.336+0000 7fec5df11880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:04:49.357 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:04:49.337+0000 7fec5df11880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:04:49.370 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:04:49.370 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:04:49.370 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:04:49.370 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:04:49.371 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:04:49.371 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:04:49.371 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:49.371 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:49.371 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:04:49.371 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:49.372 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:04:49.372 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:49.372 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:49.372 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:04:49.372 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:49.373 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:04:49.373 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:49.373 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:04:49.373 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:49.373 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:49.373 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:49.373 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:49.374 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:04:49.374 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:49.374 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:49.374 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588861.024534_609.1", 2023-07-29T00:04:49.374 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:49.374 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:04:49.375 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:49.375 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:01:01.024534Z", 2023-07-29T00:04:49.375 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:49.375 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:49.375 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:49.375 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:49.375 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:04:49.376 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:49.376 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:49.376 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:04:49.376 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:49.376 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:04:49.376 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:49.377 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:04:49.377 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:49.377 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:49.378 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:49.378 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:49.378 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:04:49.379 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:49.379 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:49.379 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:04:49.380 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:49.380 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:04:49.380 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:49.381 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:04:49.381 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:49.381 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:49.381 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:49.382 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:04:49.382 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:49.382 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:04:49.382 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:04:49.383 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:04:49.383 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:49.383 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:04:49Z" 2023-07-29T00:04:49.384 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:04:49.385 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588861.024534_609.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:01:01.024534Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:04:49Z" } 2023-07-29T00:04:49.385 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588861.024534_609.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:01:01.024534Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:04:49.386 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:04:49.386 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588861.024534_609.1', 2: '', 3: ''} 2023-07-29T00:04:49.386 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690588874.395997_652.1 target=1_1690588861.024534_609.1 2023-07-29T00:04:54.387 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:04:54.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:04:54.388 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:04:54.440 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:04:54.440 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:04:54.471 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:04:54.450+0000 7f8e3f532880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:04:54.471 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:04:54.451+0000 7f8e3f532880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:04:54.484 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:04:54.485 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:04:54.485 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:04:54.485 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:04:54.485 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:04:54.485 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:04:54.485 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:54.485 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:54.486 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:04:54.486 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:54.486 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:04:54.486 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:54.486 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:54.486 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:04:54.487 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:54.487 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:04:54.487 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:54.487 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:04:54.487 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:54.487 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:54.487 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:54.488 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:54.488 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:04:54.488 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:54.488 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:54.488 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690588874.395997_652.1", 2023-07-29T00:04:54.488 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:54.488 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:04:54.489 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:54.489 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:01:14.395997Z", 2023-07-29T00:04:54.489 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:54.489 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:54.489 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:54.489 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:54.489 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:04:54.490 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:54.490 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:54.490 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:04:54.490 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:54.490 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:04:54.490 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:54.490 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:04:54.491 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:54.491 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:54.491 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:54.491 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:54.491 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:04:54.491 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:54.491 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:04:54.492 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:04:54.492 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:54.492 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:04:54.492 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:54.492 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:04:54.492 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:04:54.493 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:54.493 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:54.493 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:04:54.493 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:54.493 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:04:54.493 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:04:54.493 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:04:54.494 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:54.494 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:04:54Z" 2023-07-29T00:04:54.494 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:04:54.495 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690588874.395997_652.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:01:14.395997Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:04:54Z" } 2023-07-29T00:04:54.495 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690588874.395997_652.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:01:14.395997Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:04:54.496 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:04:54.496 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690588874.395997_652.1', 2: '', 3: ''} 2023-07-29T00:04:54.496 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T00:04:54.497 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'hiiqos-44', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:04:54.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'hiiqos-44', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:04:54.497 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket hiiqos-44 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:04:54.545 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:04:54.546 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:04:54.610 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'hiiqos-45', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:04:54.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'hiiqos-45', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:04:54.611 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket hiiqos-45 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:04:54.657 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:04:54.657 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:04:54.720 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:04:54.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:04:54.721 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:04:54.764 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:04:54.764 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:04:54.805 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T00:04:54.805 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000078", 2023-07-29T00:04:54.806 INFO:teuthology.orchestra.run.smithi101.stdout: "meta.history", 2023-07-29T00:04:54.806 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14:2", 2023-07-29T00:04:54.806 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7:1", 2023-07-29T00:04:54.806 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14:1", 2023-07-29T00:04:54.806 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-15:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.15:0", 2023-07-29T00:04:54.806 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000070", 2023-07-29T00:04:54.807 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000104", 2023-07-29T00:04:54.807 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-10:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.10:2", 2023-07-29T00:04:54.807 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000026", 2023-07-29T00:04:54.807 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-16:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.16:1", 2023-07-29T00:04:54.807 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-34:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.34:0", 2023-07-29T00:04:54.807 INFO:teuthology.orchestra.run.smithi101.stdout: "sync.error-log.8", 2023-07-29T00:04:54.807 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000028", 2023-07-29T00:04:54.808 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12:2", 2023-07-29T00:04:54.808 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-17:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.17:2", 2023-07-29T00:04:54.808 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-32:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.32:1", 2023-07-29T00:04:54.808 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000040", 2023-07-29T00:04:54.808 INFO:teuthology.orchestra.run.smithi101.stdout: "sync.error-log.6", 2023-07-29T00:04:54.808 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-18:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.18:1", 2023-07-29T00:04:54.809 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8:10", 2023-07-29T00:04:54.809 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000015", 2023-07-29T00:04:54.809 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000069", 2023-07-29T00:04:54.809 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-10:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.10:7", 2023-07-29T00:04:54.809 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000095", 2023-07-29T00:04:54.809 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000003", 2023-07-29T00:04:54.810 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000047", 2023-07-29T00:04:54.810 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-35:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5399.1:0", 2023-07-29T00:04:54.810 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.full-sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-9:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.9", 2023-07-29T00:04:54.810 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000035", 2023-07-29T00:04:54.810 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000037", 2023-07-29T00:04:54.810 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-33:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.33:4", 2023-07-29T00:04:54.811 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7:4", 2023-07-29T00:04:54.811 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-31:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.31:7", 2023-07-29T00:04:54.811 INFO:teuthology.orchestra.run.smithi101.stdout: "datalog.sync-status.shard.59d1bb30-3b1c-476f-9111-052c1ea0b711.0", 2023-07-29T00:04:54.811 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-31:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.31:4", 2023-07-29T00:04:54.811 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-10:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.10:1", 2023-07-29T00:04:54.811 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000024", 2023-07-29T00:04:54.811 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-35:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5399.1:2", 2023-07-29T00:04:54.812 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-16:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.16:3", 2023-07-29T00:04:54.812 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000000", 2023-07-29T00:04:54.812 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000031", 2023-07-29T00:04:54.813 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-9:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.9:9", 2023-07-29T00:04:54.813 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-10:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.10:8", 2023-07-29T00:04:54.813 INFO:teuthology.orchestra.run.smithi101.stdout: "data_log.3", 2023-07-29T00:04:54.813 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12:3", 2023-07-29T00:04:54.814 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-32:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.32:8", 2023-07-29T00:04:54.814 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-31:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.31:0", 2023-07-29T00:04:54.814 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7:0", 2023-07-29T00:04:54.815 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-33:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.33:2", 2023-07-29T00:04:54.815 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.full-sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-34:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.34", 2023-07-29T00:04:54.815 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8:9", 2023-07-29T00:04:54.816 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-33:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.33:8", 2023-07-29T00:04:54.816 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-18:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.18:7", 2023-07-29T00:04:54.816 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-34:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.34:2", 2023-07-29T00:04:54.816 INFO:teuthology.orchestra.run.smithi101.stdout: "data_loggenerations_metadata", 2023-07-29T00:04:54.817 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-15:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.15:8", 2023-07-29T00:04:54.817 INFO:teuthology.orchestra.run.smithi101.stdout: "datalog.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-29T00:04:54.817 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000076", 2023-07-29T00:04:54.818 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000113", 2023-07-29T00:04:54.818 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000005", 2023-07-29T00:04:54.818 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14:8", 2023-07-29T00:04:54.819 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000011", 2023-07-29T00:04:54.819 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000058", 2023-07-29T00:04:54.819 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000052", 2023-07-29T00:04:54.820 INFO:teuthology.orchestra.run.smithi101.stdout: "meta.log.2fd8ad79-2326-4bb2-9f6c-9f77e2acc853.1", 2023-07-29T00:04:54.820 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-17:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.17:1", 2023-07-29T00:04:54.820 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-15:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.15:3", 2023-07-29T00:04:54.821 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14:10", 2023-07-29T00:04:54.821 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-18:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.18:6", 2023-07-29T00:04:54.821 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11:0", 2023-07-29T00:04:54.822 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-10:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.10:5", 2023-07-29T00:04:54.822 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000088", 2023-07-29T00:04:54.822 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000080", 2023-07-29T00:04:54.822 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000090", 2023-07-29T00:04:54.823 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-9:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.9:10", 2023-07-29T00:04:54.823 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000110", 2023-07-29T00:04:54.823 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-15:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.15:1", 2023-07-29T00:04:54.823 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000096", 2023-07-29T00:04:54.823 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000087", 2023-07-29T00:04:54.824 INFO:teuthology.orchestra.run.smithi101.stdout: "data_log.2.rFwBak84tEY.0", 2023-07-29T00:04:54.824 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12:5", 2023-07-29T00:04:54.825 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000008", 2023-07-29T00:04:54.825 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000006", 2023-07-29T00:04:54.825 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000029", 2023-07-29T00:04:54.825 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7:5", 2023-07-29T00:04:54.825 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-33:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.33:7", 2023-07-29T00:04:54.826 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000089", 2023-07-29T00:04:54.826 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000022", 2023-07-29T00:04:54.826 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-16:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.16:9", 2023-07-29T00:04:54.826 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12:10", 2023-07-29T00:04:54.826 INFO:teuthology.orchestra.run.smithi101.stdout: "bilog.trim", 2023-07-29T00:04:54.826 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000016", 2023-07-29T00:04:54.827 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8:6", 2023-07-29T00:04:54.827 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-16:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.16:0", 2023-07-29T00:04:54.827 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000041", 2023-07-29T00:04:54.827 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-15:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.15:7", 2023-07-29T00:04:54.827 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000018", 2023-07-29T00:04:54.827 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000092", 2023-07-29T00:04:54.827 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-33:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.33:0", 2023-07-29T00:04:54.828 INFO:teuthology.orchestra.run.smithi101.stdout: "sync.error-log.4", 2023-07-29T00:04:54.828 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-16:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.16:7", 2023-07-29T00:04:54.828 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-34:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.34:1", 2023-07-29T00:04:54.828 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14:0", 2023-07-29T00:04:54.828 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000014", 2023-07-29T00:04:54.828 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000112", 2023-07-29T00:04:54.829 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000007", 2023-07-29T00:04:54.829 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000021", 2023-07-29T00:04:54.829 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11:1", 2023-07-29T00:04:54.829 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000064", 2023-07-29T00:04:54.829 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000071", 2023-07-29T00:04:54.829 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11:2", 2023-07-29T00:04:54.829 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-31:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.31:3", 2023-07-29T00:04:54.830 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000074", 2023-07-29T00:04:54.830 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11:3", 2023-07-29T00:04:54.830 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-17:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.17:10", 2023-07-29T00:04:54.830 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14:4", 2023-07-29T00:04:54.830 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12:4", 2023-07-29T00:04:54.830 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000081", 2023-07-29T00:04:54.830 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000009", 2023-07-29T00:04:54.831 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8:8", 2023-07-29T00:04:54.831 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000121", 2023-07-29T00:04:54.831 INFO:teuthology.orchestra.run.smithi101.stdout: "data_log.1", 2023-07-29T00:04:54.831 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-10:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.10:10", 2023-07-29T00:04:54.831 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000125", 2023-07-29T00:04:54.831 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000082", 2023-07-29T00:04:54.832 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-35:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5399.1:1", 2023-07-29T00:04:54.832 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.full-sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-32:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.32", 2023-07-29T00:04:54.832 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-17:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.17:6", 2023-07-29T00:04:54.832 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000105", 2023-07-29T00:04:54.832 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000059", 2023-07-29T00:04:54.832 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-35:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5399.1:8", 2023-07-29T00:04:54.832 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11:7", 2023-07-29T00:04:54.833 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-18:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.18:10", 2023-07-29T00:04:54.833 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-9:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.9:6", 2023-07-29T00:04:54.833 INFO:teuthology.orchestra.run.smithi101.stdout: "datalog.sync-status.shard.59d1bb30-3b1c-476f-9111-052c1ea0b711.1", 2023-07-29T00:04:54.833 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.full-sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7", 2023-07-29T00:04:54.833 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11:9", 2023-07-29T00:04:54.833 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000077", 2023-07-29T00:04:54.833 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-33:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.33:1", 2023-07-29T00:04:54.834 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-9:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.9:4", 2023-07-29T00:04:54.834 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000032", 2023-07-29T00:04:54.834 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8:3", 2023-07-29T00:04:54.834 INFO:teuthology.orchestra.run.smithi101.stdout: "datalog.sync-status.shard.59d1bb30-3b1c-476f-9111-052c1ea0b711.3", 2023-07-29T00:04:54.834 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-31:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.31:8", 2023-07-29T00:04:54.834 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000053", 2023-07-29T00:04:54.835 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000091", 2023-07-29T00:04:54.835 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000065", 2023-07-29T00:04:54.835 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000083", 2023-07-29T00:04:54.836 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000010", 2023-07-29T00:04:54.836 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-15:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.15:4", 2023-07-29T00:04:54.836 INFO:teuthology.orchestra.run.smithi101.stdout: "sync.error-log.7", 2023-07-29T00:04:54.837 INFO:teuthology.orchestra.run.smithi101.stdout: "sync.error-log.3", 2023-07-29T00:04:54.837 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-10:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.10:3", 2023-07-29T00:04:54.837 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-16:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.16:10", 2023-07-29T00:04:54.837 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-34:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.34:9", 2023-07-29T00:04:54.838 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-9:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.9:3", 2023-07-29T00:04:54.838 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12:9", 2023-07-29T00:04:54.838 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-15:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.15:6", 2023-07-29T00:04:54.839 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7:10", 2023-07-29T00:04:54.839 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.full-sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-18:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.18", 2023-07-29T00:04:54.840 INFO:teuthology.orchestra.run.smithi101.stdout: "data_log.1.bD+tOfpjUgT.0", 2023-07-29T00:04:54.840 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.full-sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-35:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5399.1", 2023-07-29T00:04:54.840 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000045", 2023-07-29T00:04:54.840 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000002", 2023-07-29T00:04:54.841 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13:4", 2023-07-29T00:04:54.841 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000116", 2023-07-29T00:04:54.841 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11:5", 2023-07-29T00:04:54.842 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-31:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.31:2", 2023-07-29T00:04:54.842 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8:7", 2023-07-29T00:04:54.842 INFO:teuthology.orchestra.run.smithi101.stdout: "datalog.sync-status.shard.59d1bb30-3b1c-476f-9111-052c1ea0b711.2", 2023-07-29T00:04:54.843 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000034", 2023-07-29T00:04:54.843 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000101", 2023-07-29T00:04:54.843 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-18:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.18:4", 2023-07-29T00:04:54.843 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-18:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.18:8", 2023-07-29T00:04:54.843 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-32:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.32:2", 2023-07-29T00:04:54.844 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-34:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.34:6", 2023-07-29T00:04:54.844 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000079", 2023-07-29T00:04:54.845 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000049", 2023-07-29T00:04:54.845 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000117", 2023-07-29T00:04:54.845 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000044", 2023-07-29T00:04:54.845 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-34:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.34:4", 2023-07-29T00:04:54.845 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000066", 2023-07-29T00:04:54.846 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000068", 2023-07-29T00:04:54.846 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000085", 2023-07-29T00:04:54.846 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-33:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.33:6", 2023-07-29T00:04:54.846 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-34:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.34:7", 2023-07-29T00:04:54.846 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000073", 2023-07-29T00:04:54.846 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000038", 2023-07-29T00:04:54.847 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12:6", 2023-07-29T00:04:54.847 INFO:teuthology.orchestra.run.smithi101.stdout: "data_log.3.7zemK5Exu8W.0", 2023-07-29T00:04:54.847 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14:7", 2023-07-29T00:04:54.847 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12:1", 2023-07-29T00:04:54.847 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000118", 2023-07-29T00:04:54.847 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-31:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.31:6", 2023-07-29T00:04:54.847 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-16:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.16:2", 2023-07-29T00:04:54.848 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7:8", 2023-07-29T00:04:54.848 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-32:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.32:5", 2023-07-29T00:04:54.848 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000036", 2023-07-29T00:04:54.848 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-31:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.31:1", 2023-07-29T00:04:54.848 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000103", 2023-07-29T00:04:54.848 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-32:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.32:10", 2023-07-29T00:04:54.849 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000119", 2023-07-29T00:04:54.849 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12:8", 2023-07-29T00:04:54.849 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000098", 2023-07-29T00:04:54.849 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7:2", 2023-07-29T00:04:54.849 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-32:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.32:3", 2023-07-29T00:04:54.849 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-16:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.16:5", 2023-07-29T00:04:54.849 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-31:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.31:10", 2023-07-29T00:04:54.850 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12:0", 2023-07-29T00:04:54.850 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000027", 2023-07-29T00:04:54.850 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-18:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.18:3", 2023-07-29T00:04:54.850 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000019", 2023-07-29T00:04:54.850 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-33:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.33:5", 2023-07-29T00:04:54.850 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000039", 2023-07-29T00:04:54.850 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-9:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.9:1", 2023-07-29T00:04:54.851 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-17:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.17:3", 2023-07-29T00:04:54.851 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13:9", 2023-07-29T00:04:54.851 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000100", 2023-07-29T00:04:54.851 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.full-sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13", 2023-07-29T00:04:54.851 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-18:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.18:2", 2023-07-29T00:04:54.851 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11:8", 2023-07-29T00:04:54.851 INFO:teuthology.orchestra.run.smithi101.stdout: "datalog.sync-status.shard.59d1bb30-3b1c-476f-9111-052c1ea0b711.1.retry", 2023-07-29T00:04:54.852 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-32:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.32:4", 2023-07-29T00:04:54.852 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.full-sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-33:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.33", 2023-07-29T00:04:54.852 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000093", 2023-07-29T00:04:54.852 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000004", 2023-07-29T00:04:54.852 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000063", 2023-07-29T00:04:54.852 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-17:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.17:7", 2023-07-29T00:04:54.852 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000122", 2023-07-29T00:04:54.853 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000057", 2023-07-29T00:04:54.853 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-34:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.34:3", 2023-07-29T00:04:54.853 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000054", 2023-07-29T00:04:54.853 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000114", 2023-07-29T00:04:54.853 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-35:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5399.1:6", 2023-07-29T00:04:54.853 INFO:teuthology.orchestra.run.smithi101.stdout: "datalog.sync-status.shard.59d1bb30-3b1c-476f-9111-052c1ea0b711.2.retry", 2023-07-29T00:04:54.853 INFO:teuthology.orchestra.run.smithi101.stdout: "sync.error-log.2", 2023-07-29T00:04:54.854 INFO:teuthology.orchestra.run.smithi101.stdout: "data_log.0", 2023-07-29T00:04:54.854 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.full-sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12", 2023-07-29T00:04:54.854 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-15:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.15:9", 2023-07-29T00:04:54.854 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-18:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.18:5", 2023-07-29T00:04:54.854 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000012", 2023-07-29T00:04:54.855 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13:5", 2023-07-29T00:04:54.855 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000084", 2023-07-29T00:04:54.855 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000043", 2023-07-29T00:04:54.855 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-9:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.9:7", 2023-07-29T00:04:54.855 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11:4", 2023-07-29T00:04:54.855 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14:9", 2023-07-29T00:04:54.855 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.full-sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14", 2023-07-29T00:04:54.856 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-32:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.32:6", 2023-07-29T00:04:54.856 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.full-sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-16:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.16", 2023-07-29T00:04:54.856 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-10:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.10:4", 2023-07-29T00:04:54.856 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13:1", 2023-07-29T00:04:54.856 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-35:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5399.1:5", 2023-07-29T00:04:54.856 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000111", 2023-07-29T00:04:54.857 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000048", 2023-07-29T00:04:54.857 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11:10", 2023-07-29T00:04:54.857 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000020", 2023-07-29T00:04:54.857 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-16:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.16:8", 2023-07-29T00:04:54.857 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000099", 2023-07-29T00:04:54.857 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000056", 2023-07-29T00:04:54.857 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-9:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.9:0", 2023-07-29T00:04:54.858 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-32:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.32:7", 2023-07-29T00:04:54.858 INFO:teuthology.orchestra.run.smithi101.stdout: "data_log.0.tJ4fTjvcvZF.0", 2023-07-29T00:04:54.858 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000072", 2023-07-29T00:04:54.858 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000062", 2023-07-29T00:04:54.858 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000115", 2023-07-29T00:04:54.858 INFO:teuthology.orchestra.run.smithi101.stdout: "sync.error-log.1", 2023-07-29T00:04:54.858 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14:6", 2023-07-29T00:04:54.859 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000033", 2023-07-29T00:04:54.859 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13:6", 2023-07-29T00:04:54.859 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13:10", 2023-07-29T00:04:54.859 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.full-sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-31:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.31", 2023-07-29T00:04:54.859 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000025", 2023-07-29T00:04:54.859 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-31:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.31:5", 2023-07-29T00:04:54.859 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000123", 2023-07-29T00:04:54.860 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-35:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5399.1:7", 2023-07-29T00:04:54.860 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-31:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.31:9", 2023-07-29T00:04:54.860 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000108", 2023-07-29T00:04:54.860 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000094", 2023-07-29T00:04:54.860 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-35:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5399.1:10", 2023-07-29T00:04:54.860 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-32:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.32:0", 2023-07-29T00:04:54.861 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000060", 2023-07-29T00:04:54.861 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-34:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.34:5", 2023-07-29T00:04:54.861 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-17:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.17:9", 2023-07-29T00:04:54.861 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8:2", 2023-07-29T00:04:54.861 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000109", 2023-07-29T00:04:54.861 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8:0", 2023-07-29T00:04:54.861 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-16:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.16:6", 2023-07-29T00:04:54.862 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13:8", 2023-07-29T00:04:54.862 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000013", 2023-07-29T00:04:54.862 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-9:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.9:8", 2023-07-29T00:04:54.862 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-35:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5399.1:4", 2023-07-29T00:04:54.862 INFO:teuthology.orchestra.run.smithi101.stdout: "datalog.sync-status.shard.59d1bb30-3b1c-476f-9111-052c1ea0b711.0.retry", 2023-07-29T00:04:54.862 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13:0", 2023-07-29T00:04:54.863 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-18:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.18:9", 2023-07-29T00:04:54.863 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.full-sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8", 2023-07-29T00:04:54.863 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000051", 2023-07-29T00:04:54.863 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8:5", 2023-07-29T00:04:54.863 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000106", 2023-07-29T00:04:54.864 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-10:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.10:6", 2023-07-29T00:04:54.864 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12:7", 2023-07-29T00:04:54.864 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-16:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.16:4", 2023-07-29T00:04:54.864 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000017", 2023-07-29T00:04:54.865 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-32:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.32:9", 2023-07-29T00:04:54.865 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-17:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.17:0", 2023-07-29T00:04:54.865 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8:1", 2023-07-29T00:04:54.865 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-17:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.17:4", 2023-07-29T00:04:54.865 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000023", 2023-07-29T00:04:54.865 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.full-sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-10:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.10", 2023-07-29T00:04:54.865 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000107", 2023-07-29T00:04:54.866 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000046", 2023-07-29T00:04:54.866 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.full-sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11", 2023-07-29T00:04:54.866 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13:7", 2023-07-29T00:04:54.866 INFO:teuthology.orchestra.run.smithi101.stdout: "data_log.2", 2023-07-29T00:04:54.866 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13:2", 2023-07-29T00:04:54.866 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-35:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5399.1:3", 2023-07-29T00:04:54.866 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000067", 2023-07-29T00:04:54.867 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7:7", 2023-07-29T00:04:54.867 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-35:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5399.1:9", 2023-07-29T00:04:54.867 INFO:teuthology.orchestra.run.smithi101.stdout: "sync.error-log.9", 2023-07-29T00:04:54.867 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000086", 2023-07-29T00:04:54.867 INFO:teuthology.orchestra.run.smithi101.stdout: "sync.error-log.5", 2023-07-29T00:04:54.867 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.full-sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-17:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.17", 2023-07-29T00:04:54.867 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-33:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.33:3", 2023-07-29T00:04:54.868 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-9:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.9:5", 2023-07-29T00:04:54.868 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-9:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.9:2", 2023-07-29T00:04:54.868 INFO:teuthology.orchestra.run.smithi101.stdout: "meta.log.2fd8ad79-2326-4bb2-9f6c-9f77e2acc853.0", 2023-07-29T00:04:54.868 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.full-sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-15:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.15", 2023-07-29T00:04:54.868 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-15:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.15:10", 2023-07-29T00:04:54.868 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-10:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.10:0", 2023-07-29T00:04:54.869 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-17:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.17:8", 2023-07-29T00:04:54.869 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000050", 2023-07-29T00:04:54.869 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14:3", 2023-07-29T00:04:54.869 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000102", 2023-07-29T00:04:54.869 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14:5", 2023-07-29T00:04:54.869 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000124", 2023-07-29T00:04:54.869 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-17:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.17:5", 2023-07-29T00:04:54.870 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-33:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.33:10", 2023-07-29T00:04:54.870 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-34:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.34:10", 2023-07-29T00:04:54.870 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000001", 2023-07-29T00:04:54.870 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8:4", 2023-07-29T00:04:54.870 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000126", 2023-07-29T00:04:54.870 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13:3", 2023-07-29T00:04:54.870 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000055", 2023-07-29T00:04:54.871 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-15:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.15:2", 2023-07-29T00:04:54.871 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000061", 2023-07-29T00:04:54.871 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000097", 2023-07-29T00:04:54.871 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7:9", 2023-07-29T00:04:54.871 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7:6", 2023-07-29T00:04:54.871 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-10:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.10:9", 2023-07-29T00:04:54.871 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11:6", 2023-07-29T00:04:54.872 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000042", 2023-07-29T00:04:54.872 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-33:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.33:9", 2023-07-29T00:04:54.872 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-34:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.34:8", 2023-07-29T00:04:54.872 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-15:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.15:5", 2023-07-29T00:04:54.872 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-18:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.18:0", 2023-07-29T00:04:54.872 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000075", 2023-07-29T00:04:54.873 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7:3", 2023-07-29T00:04:54.873 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000120", 2023-07-29T00:04:54.873 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000030" 2023-07-29T00:04:54.873 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T00:04:54.873 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-29T00:04:54.874 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:04:54.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:04:54.875 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:04:54.957 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:04:54.957 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:04:54.998 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T00:04:54.998 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000078", 2023-07-29T00:04:54.999 INFO:teuthology.orchestra.run.smithi101.stdout: "meta.history", 2023-07-29T00:04:54.999 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-34:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.34:10", 2023-07-29T00:04:54.999 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-16:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.16:2", 2023-07-29T00:04:54.999 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-33:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.33:3", 2023-07-29T00:04:54.999 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000070", 2023-07-29T00:04:54.999 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14:9", 2023-07-29T00:04:55.000 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000104", 2023-07-29T00:04:55.000 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000026", 2023-07-29T00:04:55.000 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8:4", 2023-07-29T00:04:55.000 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-32:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.32:6", 2023-07-29T00:04:55.000 INFO:teuthology.orchestra.run.smithi101.stdout: "sync.error-log.8", 2023-07-29T00:04:55.000 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-15:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.15:5", 2023-07-29T00:04:55.001 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000028", 2023-07-29T00:04:55.001 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8:6", 2023-07-29T00:04:55.001 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-33:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.33:7", 2023-07-29T00:04:55.001 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-10:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.10:8", 2023-07-29T00:04:55.001 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8:3", 2023-07-29T00:04:55.001 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-17:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.17:7", 2023-07-29T00:04:55.002 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-33:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.33:5", 2023-07-29T00:04:55.002 INFO:teuthology.orchestra.run.smithi101.stdout: "datalog.sync-status.shard.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.1", 2023-07-29T00:04:55.002 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000040", 2023-07-29T00:04:55.002 INFO:teuthology.orchestra.run.smithi101.stdout: "sync.error-log.6", 2023-07-29T00:04:55.002 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11:8", 2023-07-29T00:04:55.002 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-33:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.33:9", 2023-07-29T00:04:55.003 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-17:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.17:2", 2023-07-29T00:04:55.003 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-15:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.15:3", 2023-07-29T00:04:55.003 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000015", 2023-07-29T00:04:55.003 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000069", 2023-07-29T00:04:55.003 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000095", 2023-07-29T00:04:55.003 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-18:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.18:0", 2023-07-29T00:04:55.004 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000003", 2023-07-29T00:04:55.004 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000047", 2023-07-29T00:04:55.004 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-9:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.9:7", 2023-07-29T00:04:55.004 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12:1", 2023-07-29T00:04:55.004 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000035", 2023-07-29T00:04:55.004 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000037", 2023-07-29T00:04:55.005 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-35:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5399.1:3", 2023-07-29T00:04:55.005 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-34:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.34:5", 2023-07-29T00:04:55.005 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-31:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.31:3", 2023-07-29T00:04:55.005 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8:7", 2023-07-29T00:04:55.005 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-32:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.32:0", 2023-07-29T00:04:55.005 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000024", 2023-07-29T00:04:55.005 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-35:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5399.1:5", 2023-07-29T00:04:55.006 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000000", 2023-07-29T00:04:55.006 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000031", 2023-07-29T00:04:55.006 INFO:teuthology.orchestra.run.smithi101.stdout: "data_log.3", 2023-07-29T00:04:55.006 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.full-sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14", 2023-07-29T00:04:55.007 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-10:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.10:9", 2023-07-29T00:04:55.007 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14:1", 2023-07-29T00:04:55.007 INFO:teuthology.orchestra.run.smithi101.stdout: "datalog.sync-status.shard.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.2", 2023-07-29T00:04:55.007 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-10:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.10:2", 2023-07-29T00:04:55.007 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-10:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.10:6", 2023-07-29T00:04:55.007 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12:4", 2023-07-29T00:04:55.008 INFO:teuthology.orchestra.run.smithi101.stdout: "datalog.sync-status.shard.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.1.retry", 2023-07-29T00:04:55.008 INFO:teuthology.orchestra.run.smithi101.stdout: "data_loggenerations_metadata", 2023-07-29T00:04:55.008 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-10:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.10:10", 2023-07-29T00:04:55.008 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11:6", 2023-07-29T00:04:55.008 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-9:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.9:6", 2023-07-29T00:04:55.008 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000076", 2023-07-29T00:04:55.008 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000113", 2023-07-29T00:04:55.009 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000005", 2023-07-29T00:04:55.009 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000011", 2023-07-29T00:04:55.009 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000058", 2023-07-29T00:04:55.009 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000052", 2023-07-29T00:04:55.009 INFO:teuthology.orchestra.run.smithi101.stdout: "meta.log.2fd8ad79-2326-4bb2-9f6c-9f77e2acc853.1", 2023-07-29T00:04:55.009 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8:9", 2023-07-29T00:04:55.009 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13:5", 2023-07-29T00:04:55.010 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12:6", 2023-07-29T00:04:55.010 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000088", 2023-07-29T00:04:55.010 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-32:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.32:8", 2023-07-29T00:04:55.010 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000080", 2023-07-29T00:04:55.010 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000090", 2023-07-29T00:04:55.011 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-17:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.17:9", 2023-07-29T00:04:55.011 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000110", 2023-07-29T00:04:55.011 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000096", 2023-07-29T00:04:55.011 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13:10", 2023-07-29T00:04:55.011 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11:9", 2023-07-29T00:04:55.011 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000087", 2023-07-29T00:04:55.011 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000008", 2023-07-29T00:04:55.012 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.full-sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8", 2023-07-29T00:04:55.012 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.full-sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13", 2023-07-29T00:04:55.012 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000006", 2023-07-29T00:04:55.012 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000029", 2023-07-29T00:04:55.012 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7:10", 2023-07-29T00:04:55.012 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000089", 2023-07-29T00:04:55.012 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000022", 2023-07-29T00:04:55.013 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-15:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.15:7", 2023-07-29T00:04:55.013 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11:2", 2023-07-29T00:04:55.013 INFO:teuthology.orchestra.run.smithi101.stdout: "bilog.trim", 2023-07-29T00:04:55.013 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000016", 2023-07-29T00:04:55.013 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8:5", 2023-07-29T00:04:55.013 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000041", 2023-07-29T00:04:55.013 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-32:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.32:9", 2023-07-29T00:04:55.014 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-32:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.32:2", 2023-07-29T00:04:55.014 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-10:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.10:4", 2023-07-29T00:04:55.014 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000018", 2023-07-29T00:04:55.014 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-35:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5399.1:6", 2023-07-29T00:04:55.014 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000092", 2023-07-29T00:04:55.014 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8:1", 2023-07-29T00:04:55.015 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14:4", 2023-07-29T00:04:55.015 INFO:teuthology.orchestra.run.smithi101.stdout: "sync.error-log.4", 2023-07-29T00:04:55.015 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8:0", 2023-07-29T00:04:55.015 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12:10", 2023-07-29T00:04:55.015 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000014", 2023-07-29T00:04:55.015 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000112", 2023-07-29T00:04:55.015 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000007", 2023-07-29T00:04:55.016 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000021", 2023-07-29T00:04:55.016 INFO:teuthology.orchestra.run.smithi101.stdout: "sync.error-log.16", 2023-07-29T00:04:55.016 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000064", 2023-07-29T00:04:55.016 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-32:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.32:4", 2023-07-29T00:04:55.017 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000071", 2023-07-29T00:04:55.017 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-34:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.34:6", 2023-07-29T00:04:55.017 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000074", 2023-07-29T00:04:55.017 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-35:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5399.1:0", 2023-07-29T00:04:55.017 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14:6", 2023-07-29T00:04:55.017 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-17:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.17:8", 2023-07-29T00:04:55.018 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000081", 2023-07-29T00:04:55.018 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000009", 2023-07-29T00:04:55.018 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000121", 2023-07-29T00:04:55.018 INFO:teuthology.orchestra.run.smithi101.stdout: "data_log.1", 2023-07-29T00:04:55.018 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8:10", 2023-07-29T00:04:55.018 INFO:teuthology.orchestra.run.smithi101.stdout: "sync.error-log.17", 2023-07-29T00:04:55.018 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000125", 2023-07-29T00:04:55.019 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-16:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.16:8", 2023-07-29T00:04:55.019 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-16:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.16:10", 2023-07-29T00:04:55.019 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000082", 2023-07-29T00:04:55.019 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-31:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.31:2", 2023-07-29T00:04:55.019 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11:10", 2023-07-29T00:04:55.019 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-15:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.15:10", 2023-07-29T00:04:55.019 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-15:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.15:0", 2023-07-29T00:04:55.020 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000105", 2023-07-29T00:04:55.020 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000059", 2023-07-29T00:04:55.020 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000077", 2023-07-29T00:04:55.020 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000032", 2023-07-29T00:04:55.020 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12:9", 2023-07-29T00:04:55.020 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13:3", 2023-07-29T00:04:55.021 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7:9", 2023-07-29T00:04:55.021 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-33:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.33:10", 2023-07-29T00:04:55.021 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-9:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.9:10", 2023-07-29T00:04:55.021 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000053", 2023-07-29T00:04:55.021 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-32:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.32:1", 2023-07-29T00:04:55.021 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000091", 2023-07-29T00:04:55.022 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-9:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.9:2", 2023-07-29T00:04:55.022 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000065", 2023-07-29T00:04:55.022 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-17:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.17:6", 2023-07-29T00:04:55.022 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14:5", 2023-07-29T00:04:55.022 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000083", 2023-07-29T00:04:55.022 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000010", 2023-07-29T00:04:55.022 INFO:teuthology.orchestra.run.smithi101.stdout: "sync.error-log.7", 2023-07-29T00:04:55.023 INFO:teuthology.orchestra.run.smithi101.stdout: "sync.error-log.3", 2023-07-29T00:04:55.023 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13:4", 2023-07-29T00:04:55.023 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-18:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.18:5", 2023-07-29T00:04:55.023 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-34:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.34:1", 2023-07-29T00:04:55.023 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11:3", 2023-07-29T00:04:55.023 INFO:teuthology.orchestra.run.smithi101.stdout: "mdlog.sync-status.shard.2", 2023-07-29T00:04:55.023 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-34:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.34:2", 2023-07-29T00:04:55.024 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000045", 2023-07-29T00:04:55.024 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.full-sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-34:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.34", 2023-07-29T00:04:55.024 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000002", 2023-07-29T00:04:55.024 INFO:teuthology.orchestra.run.smithi101.stdout: "sync.error-log.12", 2023-07-29T00:04:55.024 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7:4", 2023-07-29T00:04:55.024 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-9:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.9:0", 2023-07-29T00:04:55.025 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.full-sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-18:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.18", 2023-07-29T00:04:55.025 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000116", 2023-07-29T00:04:55.025 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-17:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.17:4", 2023-07-29T00:04:55.025 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13:2", 2023-07-29T00:04:55.025 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-15:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.15:1", 2023-07-29T00:04:55.025 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000034", 2023-07-29T00:04:55.025 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-34:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.34:9", 2023-07-29T00:04:55.026 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7:0", 2023-07-29T00:04:55.026 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000101", 2023-07-29T00:04:55.026 INFO:teuthology.orchestra.run.smithi101.stdout: "sync.error-log.13", 2023-07-29T00:04:55.026 INFO:teuthology.orchestra.run.smithi101.stdout: "mdlog.sync-status.shard.1", 2023-07-29T00:04:55.026 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-35:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5399.1:2", 2023-07-29T00:04:55.026 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000079", 2023-07-29T00:04:55.026 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000049", 2023-07-29T00:04:55.027 INFO:teuthology.orchestra.run.smithi101.stdout: "datalog.sync-status.shard.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.0", 2023-07-29T00:04:55.027 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000117", 2023-07-29T00:04:55.027 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000044", 2023-07-29T00:04:55.027 INFO:teuthology.orchestra.run.smithi101.stdout: "datalog.sync-status.shard.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.0.retry", 2023-07-29T00:04:55.028 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.full-sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-33:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.33", 2023-07-29T00:04:55.028 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-16:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.16:1", 2023-07-29T00:04:55.028 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-9:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.9:5", 2023-07-29T00:04:55.028 INFO:teuthology.orchestra.run.smithi101.stdout: "sync.error-log.14", 2023-07-29T00:04:55.028 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12:7", 2023-07-29T00:04:55.028 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000066", 2023-07-29T00:04:55.029 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000068", 2023-07-29T00:04:55.029 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000085", 2023-07-29T00:04:55.029 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-33:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.33:1", 2023-07-29T00:04:55.029 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11:5", 2023-07-29T00:04:55.029 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-32:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.32:5", 2023-07-29T00:04:55.029 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12:2", 2023-07-29T00:04:55.029 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12:0", 2023-07-29T00:04:55.030 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000073", 2023-07-29T00:04:55.030 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11:1", 2023-07-29T00:04:55.030 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000038", 2023-07-29T00:04:55.030 INFO:teuthology.orchestra.run.smithi101.stdout: "sync.error-log.10", 2023-07-29T00:04:55.030 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000118", 2023-07-29T00:04:55.031 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-9:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.9:8", 2023-07-29T00:04:55.031 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-18:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.18:2", 2023-07-29T00:04:55.031 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000036", 2023-07-29T00:04:55.031 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13:0", 2023-07-29T00:04:55.031 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000103", 2023-07-29T00:04:55.031 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-9:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.9:3", 2023-07-29T00:04:55.031 INFO:teuthology.orchestra.run.smithi101.stdout: "mdlog.sync-status.shard.3", 2023-07-29T00:04:55.032 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000119", 2023-07-29T00:04:55.032 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14:10", 2023-07-29T00:04:55.032 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000098", 2023-07-29T00:04:55.032 INFO:teuthology.orchestra.run.smithi101.stdout: "datalog.sync-status.shard.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.2.retry", 2023-07-29T00:04:55.032 INFO:teuthology.orchestra.run.smithi101.stdout: "data_log.1.VYYJ9Enr90h.0", 2023-07-29T00:04:55.032 INFO:teuthology.orchestra.run.smithi101.stdout: "mdlog.sync-status.shard.0", 2023-07-29T00:04:55.033 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000027", 2023-07-29T00:04:55.033 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-35:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5399.1:7", 2023-07-29T00:04:55.033 INFO:teuthology.orchestra.run.smithi101.stdout: "mdlog.sync-status", 2023-07-29T00:04:55.033 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000019", 2023-07-29T00:04:55.033 INFO:teuthology.orchestra.run.smithi101.stdout: "datalog.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T00:04:55.034 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7:8", 2023-07-29T00:04:55.034 INFO:teuthology.orchestra.run.smithi101.stdout: "datalog.sync-status.shard.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.3", 2023-07-29T00:04:55.034 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-16:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.16:0", 2023-07-29T00:04:55.034 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000039", 2023-07-29T00:04:55.034 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7:3", 2023-07-29T00:04:55.034 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13:7", 2023-07-29T00:04:55.035 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-18:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.18:7", 2023-07-29T00:04:55.035 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7:5", 2023-07-29T00:04:55.035 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000100", 2023-07-29T00:04:55.035 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-10:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.10:0", 2023-07-29T00:04:55.035 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-18:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.18:4", 2023-07-29T00:04:55.035 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.full-sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-15:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.15", 2023-07-29T00:04:55.035 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000093", 2023-07-29T00:04:55.036 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-35:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5399.1:8", 2023-07-29T00:04:55.036 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-10:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.10:1", 2023-07-29T00:04:55.036 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-33:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.33:0", 2023-07-29T00:04:55.036 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.full-sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-10:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.10", 2023-07-29T00:04:55.036 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000004", 2023-07-29T00:04:55.036 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000063", 2023-07-29T00:04:55.036 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000122", 2023-07-29T00:04:55.037 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000057", 2023-07-29T00:04:55.037 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000054", 2023-07-29T00:04:55.037 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000114", 2023-07-29T00:04:55.037 INFO:teuthology.orchestra.run.smithi101.stdout: "sync.error-log.2", 2023-07-29T00:04:55.037 INFO:teuthology.orchestra.run.smithi101.stdout: "data_log.0", 2023-07-29T00:04:55.037 INFO:teuthology.orchestra.run.smithi101.stdout: "data_log.2.Ib3kTiAVB4g.0", 2023-07-29T00:04:55.037 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-33:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.33:6", 2023-07-29T00:04:55.038 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000012", 2023-07-29T00:04:55.038 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-17:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.17:10", 2023-07-29T00:04:55.038 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-33:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.33:8", 2023-07-29T00:04:55.038 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-10:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.10:5", 2023-07-29T00:04:55.038 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000084", 2023-07-29T00:04:55.038 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000043", 2023-07-29T00:04:55.038 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.full-sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11", 2023-07-29T00:04:55.039 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14:2", 2023-07-29T00:04:55.039 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-17:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.17:5", 2023-07-29T00:04:55.039 INFO:teuthology.orchestra.run.smithi101.stdout: "sync.error-log.11", 2023-07-29T00:04:55.039 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11:4", 2023-07-29T00:04:55.039 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.full-sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12", 2023-07-29T00:04:55.039 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-33:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.33:4", 2023-07-29T00:04:55.039 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7:6", 2023-07-29T00:04:55.040 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-34:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.34:4", 2023-07-29T00:04:55.040 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000111", 2023-07-29T00:04:55.040 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000048", 2023-07-29T00:04:55.040 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-17:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.17:1", 2023-07-29T00:04:55.040 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000020", 2023-07-29T00:04:55.040 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8:2", 2023-07-29T00:04:55.041 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000099", 2023-07-29T00:04:55.041 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000056", 2023-07-29T00:04:55.041 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.full-sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7", 2023-07-29T00:04:55.041 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-32:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.32:7", 2023-07-29T00:04:55.041 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14:3", 2023-07-29T00:04:55.041 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-17:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.17:3", 2023-07-29T00:04:55.041 INFO:teuthology.orchestra.run.smithi101.stdout: "sync.error-log.19", 2023-07-29T00:04:55.042 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-16:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.16:5", 2023-07-29T00:04:55.042 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000072", 2023-07-29T00:04:55.042 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000062", 2023-07-29T00:04:55.042 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-9:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.9:4", 2023-07-29T00:04:55.042 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000115", 2023-07-29T00:04:55.042 INFO:teuthology.orchestra.run.smithi101.stdout: "sync.error-log.1", 2023-07-29T00:04:55.042 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-15:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.15:8", 2023-07-29T00:04:55.043 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000033", 2023-07-29T00:04:55.043 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.full-sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-35:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5399.1", 2023-07-29T00:04:55.043 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-15:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.15:6", 2023-07-29T00:04:55.043 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-35:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5399.1:1", 2023-07-29T00:04:55.043 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-35:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5399.1:9", 2023-07-29T00:04:55.043 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000025", 2023-07-29T00:04:55.043 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000123", 2023-07-29T00:04:55.044 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12:5", 2023-07-29T00:04:55.044 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-17:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.17:0", 2023-07-29T00:04:55.044 INFO:teuthology.orchestra.run.smithi101.stdout: "sync.error-log.18", 2023-07-29T00:04:55.044 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-34:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.34:7", 2023-07-29T00:04:55.044 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000108", 2023-07-29T00:04:55.044 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000094", 2023-07-29T00:04:55.044 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-18:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.18:8", 2023-07-29T00:04:55.045 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7:2", 2023-07-29T00:04:55.045 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-16:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.16:6", 2023-07-29T00:04:55.045 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14:8", 2023-07-29T00:04:55.045 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000060", 2023-07-29T00:04:55.045 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000109", 2023-07-29T00:04:55.045 INFO:teuthology.orchestra.run.smithi101.stdout: "sync.error-log.15", 2023-07-29T00:04:55.045 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-9:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.9:1", 2023-07-29T00:04:55.046 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-32:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.32:3", 2023-07-29T00:04:55.046 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-9:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.9:9", 2023-07-29T00:04:55.046 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.full-sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-32:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.32", 2023-07-29T00:04:55.046 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000013", 2023-07-29T00:04:55.046 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14:0", 2023-07-29T00:04:55.046 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-16:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.16:9", 2023-07-29T00:04:55.046 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-32:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.32:10", 2023-07-29T00:04:55.047 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.full-sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-16:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.16", 2023-07-29T00:04:55.047 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-31:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.31:5", 2023-07-29T00:04:55.047 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-16:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.16:3", 2023-07-29T00:04:55.047 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-31:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.31:9", 2023-07-29T00:04:55.047 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000051", 2023-07-29T00:04:55.047 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000106", 2023-07-29T00:04:55.047 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-31:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.31:1", 2023-07-29T00:04:55.048 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000017", 2023-07-29T00:04:55.048 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-18:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.18:1", 2023-07-29T00:04:55.048 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-31:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.31:4", 2023-07-29T00:04:55.048 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-31:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.31:8", 2023-07-29T00:04:55.048 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13:6", 2023-07-29T00:04:55.048 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000023", 2023-07-29T00:04:55.049 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.full-sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-9:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.9", 2023-07-29T00:04:55.049 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-34:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.34:8", 2023-07-29T00:04:55.049 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-16:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.16:7", 2023-07-29T00:04:55.049 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000107", 2023-07-29T00:04:55.049 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000046", 2023-07-29T00:04:55.049 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.full-sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-17:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.17", 2023-07-29T00:04:55.049 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-15:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.15:9", 2023-07-29T00:04:55.050 INFO:teuthology.orchestra.run.smithi101.stdout: "data_log.2", 2023-07-29T00:04:55.050 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000067", 2023-07-29T00:04:55.050 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-31:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.31:6", 2023-07-29T00:04:55.050 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-8:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.8:8", 2023-07-29T00:04:55.050 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-15:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.15:4", 2023-07-29T00:04:55.050 INFO:teuthology.orchestra.run.smithi101.stdout: "sync.error-log.9", 2023-07-29T00:04:55.050 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000086", 2023-07-29T00:04:55.051 INFO:teuthology.orchestra.run.smithi101.stdout: "sync.error-log.5", 2023-07-29T00:04:55.051 INFO:teuthology.orchestra.run.smithi101.stdout: "data_log.0.34hl+YOX3EJ.0", 2023-07-29T00:04:55.051 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-18:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.18:6", 2023-07-29T00:04:55.051 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.full-sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-31:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.31", 2023-07-29T00:04:55.051 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-34:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.34:0", 2023-07-29T00:04:55.051 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7:7", 2023-07-29T00:04:55.051 INFO:teuthology.orchestra.run.smithi101.stdout: "meta.log.2fd8ad79-2326-4bb2-9f6c-9f77e2acc853.0", 2023-07-29T00:04:55.052 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-33:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.33:2", 2023-07-29T00:04:55.052 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-16:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.16:4", 2023-07-29T00:04:55.052 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-7:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.7:1", 2023-07-29T00:04:55.052 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12:8", 2023-07-29T00:04:55.052 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-34:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.34:3", 2023-07-29T00:04:55.052 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000050", 2023-07-29T00:04:55.052 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-31:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.31:7", 2023-07-29T00:04:55.053 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11:7", 2023-07-29T00:04:55.053 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000102", 2023-07-29T00:04:55.053 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-35:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5399.1:10", 2023-07-29T00:04:55.053 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-18:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.18:3", 2023-07-29T00:04:55.053 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-11:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.11:0", 2023-07-29T00:04:55.053 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-18:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.18:10", 2023-07-29T00:04:55.054 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-31:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.31:10", 2023-07-29T00:04:55.054 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000124", 2023-07-29T00:04:55.054 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000001", 2023-07-29T00:04:55.054 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13:1", 2023-07-29T00:04:55.054 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13:9", 2023-07-29T00:04:55.054 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-10:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.10:3", 2023-07-29T00:04:55.054 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000126", 2023-07-29T00:04:55.055 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-18:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.18:9", 2023-07-29T00:04:55.055 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000055", 2023-07-29T00:04:55.055 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000061", 2023-07-29T00:04:55.055 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-12:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.12:3", 2023-07-29T00:04:55.055 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000097", 2023-07-29T00:04:55.055 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-15:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.15:2", 2023-07-29T00:04:55.055 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-13:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.13:8", 2023-07-29T00:04:55.056 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-14:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.14:7", 2023-07-29T00:04:55.056 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000042", 2023-07-29T00:04:55.056 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-10:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.10:7", 2023-07-29T00:04:55.056 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-31:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.31:0", 2023-07-29T00:04:55.056 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000075", 2023-07-29T00:04:55.056 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000120", 2023-07-29T00:04:55.056 INFO:teuthology.orchestra.run.smithi101.stdout: "obj_delete_at_hint.0000000030", 2023-07-29T00:04:55.057 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-35:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5399.1:4" 2023-07-29T00:04:55.057 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T00:04:55.057 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-29T00:04:55.058 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-07-29T00:04:55.058 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:04:55.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:04:55.059 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:04:55.143 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:04:55.143 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:04:55.187 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T00:04:55.187 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:55.187 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000004449", 2023-07-29T00:04:55.187 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:04:54.682232Z" 2023-07-29T00:04:55.188 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:55.188 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:55.188 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000003885", 2023-07-29T00:04:55.188 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:04:54.683157Z" 2023-07-29T00:04:55.188 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:55.189 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:55.189 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000002055", 2023-07-29T00:04:55.189 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:04:54.684200Z" 2023-07-29T00:04:55.189 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:55.189 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:55.189 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000000933", 2023-07-29T00:04:55.189 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:04:54.681342Z" 2023-07-29T00:04:55.190 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:55.190 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T00:04:55.191 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000004449', 1: '00000000000000000000:00000000000000003885', 2: '00000000000000000000:00000000000000002055', 3: '00000000000000000000:00000000000000000933'} 2023-07-29T00:04:55.191 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-07-29T00:04:55.191 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:04:55.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:04:55.192 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:04:55.242 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:04:55.243 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:04:55.285 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:04:55.286 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:04:55.286 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:04:55.286 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:04:55.286 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:04:55.286 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 16749483048653174838 2023-07-29T00:04:55.287 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:55.287 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:04:55.287 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:55.287 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:04:55.287 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:55.287 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:04:55.288 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000003606", 2023-07-29T00:04:55.288 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:55.288 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:04:55.288 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:55.288 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:54:54.539998Z" 2023-07-29T00:04:55.288 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:55.288 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:55.289 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:55.289 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:04:55.289 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:55.289 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:04:55.289 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000003043", 2023-07-29T00:04:55.289 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:55.289 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:04:55.290 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:55.290 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:58:41.899760Z" 2023-07-29T00:04:55.290 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:55.290 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:55.290 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:55.290 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:04:55.290 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:55.291 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:04:55.291 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-07-29T00:04:55.291 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:55.291 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:04:55.291 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:55.291 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:54:53.570640Z" 2023-07-29T00:04:55.292 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:55.292 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:55.292 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:04:55.292 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:04:55.292 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:04:55.292 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:04:55.292 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:04:55.293 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:04:55.293 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:04:55.293 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:04:55.293 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000" 2023-07-29T00:04:55.293 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:55.293 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:04:55.293 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:04:55.294 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:55.294 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:04:55.294 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 0, 2023-07-29T00:04:55.294 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 0 2023-07-29T00:04:55.294 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:04:55.294 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:04:55Z" 2023-07-29T00:04:55.294 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:04:55.295 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16749483048653174838 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003606", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:54:54.539998Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003043", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:58:41.899760Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001210", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:54:53.570640Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-07-29T00:04:55Z" } 2023-07-29T00:04:55.296 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003606', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:54:54.539998Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003043', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:58:41.899760Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:54:53.570640Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-07-29T00:04:55.296 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004449', 1: '00000000000000000000:00000000000000003885', 2: '00000000000000000000:00000000000000002055', 3: '00000000000000000000:00000000000000000933'} 2023-07-29T00:04:55.297 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003606', 1: '00000000000000000000:00000000000000003043', 2: '00000000000000000000:00000000000000001210', 3: ''} 2023-07-29T00:04:55.297 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000004449 target=00000000000000000000:00000000000000003606, shard=1 master=00000000000000000000:00000000000000003885 target=00000000000000000000:00000000000000003043, shard=2 master=00000000000000000000:00000000000000002055 target=00000000000000000000:00000000000000001210, shard=3 master=00000000000000000000:00000000000000000933 target= 2023-07-29T00:04:58.356 INFO:tasks.rgw.c2.client.0.smithi139.stdout:2023-07-29T00:04:58.334+0000 7f5f2671b640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T00:05:00.298 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:05:00.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:05:00.299 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:05:00.355 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:05:00.355 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:05:00.405 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:05:00.405 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:05:00.405 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:05:00.406 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:05:00.406 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:05:00.406 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 16749483048653174838 2023-07-29T00:05:00.406 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:00.406 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:05:00.406 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:05:00.407 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:05:00.407 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:05:00.407 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:05:00.407 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000003606", 2023-07-29T00:05:00.407 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:05:00.408 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:05:00.408 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:05:00.408 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:54:54.539998Z" 2023-07-29T00:05:00.408 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:05:00.408 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:00.408 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:05:00.408 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:05:00.409 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:05:00.409 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:05:00.409 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000003043", 2023-07-29T00:05:00.409 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:05:00.409 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:05:00.409 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:05:00.410 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:58:41.899760Z" 2023-07-29T00:05:00.410 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:05:00.410 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:00.410 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:05:00.410 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:05:00.410 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:05:00.410 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:05:00.411 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-07-29T00:05:00.411 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:05:00.411 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:05:00.411 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:05:00.411 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:54:53.570640Z" 2023-07-29T00:05:00.411 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:05:00.411 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:00.412 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:05:00.412 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:05:00.412 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:05:00.412 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:05:00.413 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:05:00.413 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:05:00.413 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:05:00.413 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:05:00.413 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000" 2023-07-29T00:05:00.413 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:05:00.414 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:05:00.414 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:05:00.414 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:00.414 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:05:00.414 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 0, 2023-07-29T00:05:00.414 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 0 2023-07-29T00:05:00.414 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:00.415 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:05:00Z" 2023-07-29T00:05:00.415 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:05:00.416 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16749483048653174838 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003606", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:54:54.539998Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003043", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:58:41.899760Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001210", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:54:53.570640Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-07-29T00:05:00Z" } 2023-07-29T00:05:00.416 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003606', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:54:54.539998Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003043', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:58:41.899760Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:54:53.570640Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-07-29T00:05:00.417 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004449', 1: '00000000000000000000:00000000000000003885', 2: '00000000000000000000:00000000000000002055', 3: '00000000000000000000:00000000000000000933'} 2023-07-29T00:05:00.417 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003606', 1: '00000000000000000000:00000000000000003043', 2: '00000000000000000000:00000000000000001210', 3: ''} 2023-07-29T00:05:00.417 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000004449 target=00000000000000000000:00000000000000003606, shard=1 master=00000000000000000000:00000000000000003885 target=00000000000000000000:00000000000000003043, shard=2 master=00000000000000000000:00000000000000002055 target=00000000000000000000:00000000000000001210, shard=3 master=00000000000000000000:00000000000000000933 target= 2023-07-29T00:05:05.419 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:05:05.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:05:05.420 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:05:05.474 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:05:05.475 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:05:05.520 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:05:05.521 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:05:05.521 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:05:05.521 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:05:05.521 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:05:05.521 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 16749483048653174838 2023-07-29T00:05:05.522 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:05.522 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:05:05.522 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:05:05.522 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:05:05.522 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:05:05.522 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:05:05.522 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000003606", 2023-07-29T00:05:05.523 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:05:05.523 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:05:05.523 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:05:05.523 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:54:54.539998Z" 2023-07-29T00:05:05.523 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:05:05.523 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:05.524 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:05:05.524 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:05:05.524 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:05:05.524 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:05:05.524 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000003043", 2023-07-29T00:05:05.525 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:05:05.525 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:05:05.525 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:05:05.525 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:58:41.899760Z" 2023-07-29T00:05:05.525 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:05:05.525 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:05.525 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:05:05.526 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:05:05.526 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:05:05.526 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:05:05.526 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-07-29T00:05:05.526 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:05:05.526 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:05:05.526 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:05:05.527 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:54:53.570640Z" 2023-07-29T00:05:05.527 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:05:05.527 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:05.527 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:05:05.528 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:05:05.528 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:05:05.528 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:05:05.528 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:05:05.528 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:05:05.528 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:05:05.529 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:05:05.529 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000" 2023-07-29T00:05:05.529 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:05:05.529 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:05:05.529 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:05:05.529 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:05.529 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:05:05.530 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 0, 2023-07-29T00:05:05.530 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 0 2023-07-29T00:05:05.530 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:05.530 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:05:05Z" 2023-07-29T00:05:05.530 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:05:05.531 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16749483048653174838 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003606", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:54:54.539998Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003043", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:58:41.899760Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001210", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:54:53.570640Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-07-29T00:05:05Z" } 2023-07-29T00:05:05.532 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003606', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:54:54.539998Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003043', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:58:41.899760Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:54:53.570640Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-07-29T00:05:05.532 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004449', 1: '00000000000000000000:00000000000000003885', 2: '00000000000000000000:00000000000000002055', 3: '00000000000000000000:00000000000000000933'} 2023-07-29T00:05:05.533 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003606', 1: '00000000000000000000:00000000000000003043', 2: '00000000000000000000:00000000000000001210', 3: ''} 2023-07-29T00:05:05.533 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000004449 target=00000000000000000000:00000000000000003606, shard=1 master=00000000000000000000:00000000000000003885 target=00000000000000000000:00000000000000003043, shard=2 master=00000000000000000000:00000000000000002055 target=00000000000000000000:00000000000000001210, shard=3 master=00000000000000000000:00000000000000000933 target= 2023-07-29T00:05:10.535 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:05:10.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:05:10.536 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:05:10.592 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:05:10.592 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:05:10.638 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:05:10.638 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:05:10.638 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:05:10.639 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:05:10.639 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:05:10.639 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 16749483048653174838 2023-07-29T00:05:10.639 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:10.639 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:05:10.639 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:05:10.640 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:05:10.640 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:05:10.640 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:05:10.640 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000003606", 2023-07-29T00:05:10.640 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:05:10.641 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:05:10.641 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:05:10.641 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:54:54.539998Z" 2023-07-29T00:05:10.641 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:05:10.641 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:10.641 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:05:10.641 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:05:10.642 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:05:10.642 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:05:10.642 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000003043", 2023-07-29T00:05:10.642 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:05:10.642 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:05:10.642 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:05:10.643 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:58:41.899760Z" 2023-07-29T00:05:10.643 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:05:10.643 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:10.643 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:05:10.643 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:05:10.643 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:05:10.644 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:05:10.644 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-07-29T00:05:10.644 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:05:10.644 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:05:10.644 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:05:10.644 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:54:53.570640Z" 2023-07-29T00:05:10.645 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:05:10.645 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:10.645 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:05:10.645 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:05:10.645 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:05:10.645 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:05:10.646 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:05:10.646 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:05:10.646 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:05:10.646 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:05:10.646 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000" 2023-07-29T00:05:10.646 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:05:10.646 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:05:10.647 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:05:10.647 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:10.647 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:05:10.647 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 0, 2023-07-29T00:05:10.647 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 0 2023-07-29T00:05:10.648 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:10.648 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:05:10Z" 2023-07-29T00:05:10.648 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:05:10.649 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16749483048653174838 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003606", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:54:54.539998Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003043", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:58:41.899760Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001210", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-28T23:54:53.570640Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-07-29T00:05:10Z" } 2023-07-29T00:05:10.649 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003606', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:54:54.539998Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003043', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:58:41.899760Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-28T23:54:53.570640Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-07-29T00:05:10.650 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004449', 1: '00000000000000000000:00000000000000003885', 2: '00000000000000000000:00000000000000002055', 3: '00000000000000000000:00000000000000000933'} 2023-07-29T00:05:10.650 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003606', 1: '00000000000000000000:00000000000000003043', 2: '00000000000000000000:00000000000000001210', 3: ''} 2023-07-29T00:05:10.651 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000004449 target=00000000000000000000:00000000000000003606, shard=1 master=00000000000000000000:00000000000000003885 target=00000000000000000000:00000000000000003043, shard=2 master=00000000000000000000:00000000000000002055 target=00000000000000000000:00000000000000001210, shard=3 master=00000000000000000000:00000000000000000933 target= 2023-07-29T00:05:12.345 INFO:tasks.rgw.c1.client.0.smithi101.stdout:2023-07-29T00:05:12.323+0000 7f0081a5e640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T00:05:12.358 INFO:tasks.rgw.c2.client.1.smithi139.stdout:2023-07-29T00:05:12.337+0000 7f5731a6a640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T00:05:15.652 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:05:15.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:05:15.653 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:05:15.708 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:05:15.708 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:05:15.755 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:05:15.756 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:05:15.756 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:05:15.756 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:05:15.756 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:05:15.756 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 16749483048653174838 2023-07-29T00:05:15.756 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:15.757 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:05:15.757 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:05:15.757 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:05:15.757 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:05:15.757 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:05:15.757 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000004449", 2023-07-29T00:05:15.757 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:05:15.758 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:05:15.758 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:05:15.758 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:04:54.682232Z" 2023-07-29T00:05:15.758 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:05:15.758 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:15.758 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:05:15.759 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:05:15.759 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:05:15.759 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:05:15.759 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000003885", 2023-07-29T00:05:15.759 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:05:15.759 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:05:15.759 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:05:15.760 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:04:54.683157Z" 2023-07-29T00:05:15.760 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:05:15.760 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:15.760 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:05:15.760 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:05:15.760 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:05:15.761 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:05:15.761 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000002055", 2023-07-29T00:05:15.761 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:05:15.761 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:05:15.761 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:05:15.761 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:04:54.684200Z" 2023-07-29T00:05:15.761 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:05:15.762 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:15.762 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:05:15.762 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:05:15.762 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:05:15.762 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:05:15.762 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000000933", 2023-07-29T00:05:15.763 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:05:15.763 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:05:15.763 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:05:15.763 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:04:54.681342Z" 2023-07-29T00:05:15.763 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:05:15.763 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:05:15.763 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:05:15.764 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:15.764 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:05:15.764 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 0, 2023-07-29T00:05:15.764 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 0 2023-07-29T00:05:15.764 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:15.764 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:05:15Z" 2023-07-29T00:05:15.764 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:05:15.765 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16749483048653174838 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004449", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:04:54.682232Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003885", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:04:54.683157Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002055", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:04:54.684200Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000933", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:04:54.681342Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-07-29T00:05:15Z" } 2023-07-29T00:05:15.766 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004449', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:04:54.682232Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003885', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:04:54.683157Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002055', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:04:54.684200Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000933', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:04:54.681342Z'}}] 2023-07-29T00:05:15.766 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004449', 1: '00000000000000000000:00000000000000003885', 2: '00000000000000000000:00000000000000002055', 3: '00000000000000000000:00000000000000000933'} 2023-07-29T00:05:15.767 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004449', 1: '00000000000000000000:00000000000000003885', 2: '00000000000000000000:00000000000000002055', 3: '00000000000000000000:00000000000000000933'} 2023-07-29T00:05:15.767 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-07-29T00:05:15.767 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-07-29T00:05:15.768 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:05:15.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:05:15.769 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:05:15.814 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:05:15.814 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:05:15.861 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T00:05:15.861 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:05:15.861 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000005014", 2023-07-29T00:05:15.861 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:05:12.145056Z" 2023-07-29T00:05:15.862 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:15.862 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:05:15.862 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000002902", 2023-07-29T00:05:15.862 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:05:12.146079Z" 2023-07-29T00:05:15.862 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:15.862 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:05:15.863 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000002056", 2023-07-29T00:05:15.863 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:05:12.147247Z" 2023-07-29T00:05:15.863 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:15.863 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:05:15.863 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000000934", 2023-07-29T00:05:15.863 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:05:12.144025Z" 2023-07-29T00:05:15.864 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:05:15.864 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T00:05:15.865 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000005014', 1: '00000000000000000000:00000000000000002902', 2: '00000000000000000000:00000000000000002056', 3: '00000000000000000000:00000000000000000934'} 2023-07-29T00:05:15.865 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-07-29T00:05:15.866 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:05:15.866 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:05:15.867 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:05:15.914 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:05:15.914 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:05:15.954 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:05:15.954 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:05:15.955 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:05:15.955 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:05:15.955 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:05:15.955 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 17882949198279251152 2023-07-29T00:05:15.955 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:15.955 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:05:15.956 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:05:15.956 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:05:15.956 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:05:15.956 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:05:15.956 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000005014", 2023-07-29T00:05:15.956 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:05:15.957 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:05:15.957 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:05:15.957 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:05:12.145056Z" 2023-07-29T00:05:15.957 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:05:15.957 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:15.958 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:05:15.958 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:05:15.958 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:05:15.958 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:05:15.958 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000002902", 2023-07-29T00:05:15.958 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:05:15.959 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:05:15.959 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:05:15.959 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:05:12.146079Z" 2023-07-29T00:05:15.959 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:05:15.959 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:15.959 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:05:15.960 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:05:15.960 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:05:15.960 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:05:15.960 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000002056", 2023-07-29T00:05:15.960 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:05:15.960 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:05:15.960 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:05:15.961 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:05:12.147247Z" 2023-07-29T00:05:15.961 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:05:15.961 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:15.962 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:05:15.962 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:05:15.962 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:05:15.962 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:05:15.963 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000000934", 2023-07-29T00:05:15.963 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:05:15.963 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:05:15.964 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:05:15.964 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:05:12.144025Z" 2023-07-29T00:05:15.964 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:05:15.965 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:05:15.965 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:05:15.965 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:15.966 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:05:15.966 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 0, 2023-07-29T00:05:15.966 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 0 2023-07-29T00:05:15.966 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:15.967 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:05:15Z" 2023-07-29T00:05:15.967 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:05:15.968 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17882949198279251152 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005014", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:05:12.145056Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002902", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:05:12.146079Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002056", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:05:12.147247Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000934", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:05:12.144025Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-07-29T00:05:15Z" } 2023-07-29T00:05:15.968 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005014', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:05:12.145056Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002902', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:05:12.146079Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002056', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:05:12.147247Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000934', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:05:12.144025Z'}}] 2023-07-29T00:05:15.969 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000005014', 1: '00000000000000000000:00000000000000002902', 2: '00000000000000000000:00000000000000002056', 3: '00000000000000000000:00000000000000000934'} 2023-07-29T00:05:15.969 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000005014', 1: '00000000000000000000:00000000000000002902', 2: '00000000000000000000:00000000000000002056', 3: '00000000000000000000:00000000000000000934'} 2023-07-29T00:05:15.969 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-07-29T00:05:15.970 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable ... ok 2023-07-29T00:05:15.971 INFO:rgw_multi.tests:create bucket zone=a1 name=hiiqos-46 2023-07-29T00:05:15.971 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:05:15.972 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:05:15.972 DEBUG:boto:Method: PUT 2023-07-29T00:05:15.973 DEBUG:boto:Path: /hiiqos-46/ 2023-07-29T00:05:15.973 DEBUG:boto:Data: 2023-07-29T00:05:15.973 DEBUG:boto:Headers: {} 2023-07-29T00:05:15.974 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:05:15.974 DEBUG:boto:Port: 8000 2023-07-29T00:05:15.974 DEBUG:boto:Params: {} 2023-07-29T00:05:15.975 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-29T00:05:15.976 DEBUG:boto:Token: None 2023-07-29T00:05:15.976 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 00:05:15 GMT /hiiqos-46/ 2023-07-29T00:05:15.977 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MBaBcTSSBF3kDQMrYFvasLNfpPE= 2023-07-29T00:05:15.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MBaBcTSSBF3kDQMrYFvasLNfpPE=', 'Content-Length': '0'} 2023-07-29T00:05:15.992 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bc071a6afc9531fc-0064c457bb-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:15.992 INFO:rgw_multi.tests:create bucket zone=a2 name=hiiqos-47 2023-07-29T00:05:15.993 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:05:15.993 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:05:15.993 DEBUG:boto:Method: PUT 2023-07-29T00:05:15.994 DEBUG:boto:Path: /hiiqos-47/ 2023-07-29T00:05:15.994 DEBUG:boto:Data: 2023-07-29T00:05:15.994 DEBUG:boto:Headers: {} 2023-07-29T00:05:15.995 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:05:15.995 DEBUG:boto:Port: 8001 2023-07-29T00:05:15.995 DEBUG:boto:Params: {} 2023-07-29T00:05:15.996 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-29T00:05:15.996 DEBUG:boto:Token: None 2023-07-29T00:05:15.997 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 00:05:15 GMT /hiiqos-47/ 2023-07-29T00:05:15.997 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:57s3XZgvGFcLoVCDVP22A1ztjvE= 2023-07-29T00:05:15.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:57s3XZgvGFcLoVCDVP22A1ztjvE=', 'Content-Length': '0'} 2023-07-29T00:05:16.020 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005eb06ab02598903e-0064c457bb-1663-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:05:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:16.021 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:05:16.022 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:05:16.023 DEBUG:boto:Method: HEAD 2023-07-29T00:05:16.024 DEBUG:boto:Path: /hiiqos-46/ 2023-07-29T00:05:16.025 DEBUG:boto:Data: 2023-07-29T00:05:16.025 DEBUG:boto:Headers: {} 2023-07-29T00:05:16.026 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:05:16.026 DEBUG:boto:Port: 8000 2023-07-29T00:05:16.026 DEBUG:boto:Params: {} 2023-07-29T00:05:16.027 DEBUG:boto:Token: None 2023-07-29T00:05:16.027 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:05:16 GMT /hiiqos-46/ 2023-07-29T00:05:16.027 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:su/4kocKBkSLWYM6U6X9T8tKZrM= 2023-07-29T00:05:16.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:su/4kocKBkSLWYM6U6X9T8tKZrM=', 'Content-Length': '0'} 2023-07-29T00:05:16.030 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006e632857a6502943-0064c457bc-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:16.030 DEBUG:boto:path=/hiiqos-46/obj1 2023-07-29T00:05:16.031 DEBUG:boto:auth_path=/hiiqos-46/obj1 2023-07-29T00:05:16.031 DEBUG:boto:Method: PUT 2023-07-29T00:05:16.032 DEBUG:boto:Path: /hiiqos-46/obj1 2023-07-29T00:05:16.032 DEBUG:boto:Data: 2023-07-29T00:05:16.032 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T00:05:16.033 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:05:16.033 DEBUG:boto:Port: 8000 2023-07-29T00:05:16.033 DEBUG:boto:Params: {} 2023-07-29T00:05:16.034 DEBUG:boto:Token: None 2023-07-29T00:05:16.034 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 00:05:16 GMT /hiiqos-46/obj1 2023-07-29T00:05:16.034 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DP0d3Vwppjyol15wCgYh2kw/KKk= 2023-07-29T00:05:16.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:05:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DP0d3Vwppjyol15wCgYh2kw/KKk='} 2023-07-29T00:05:16.040 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ade004ebe548efc2-0064c457bc-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:16.041 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:05:16.041 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:05:16.041 DEBUG:boto:Method: HEAD 2023-07-29T00:05:16.042 DEBUG:boto:Path: /hiiqos-46/ 2023-07-29T00:05:16.042 DEBUG:boto:Data: 2023-07-29T00:05:16.042 DEBUG:boto:Headers: {} 2023-07-29T00:05:16.042 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:05:16.043 DEBUG:boto:Port: 8000 2023-07-29T00:05:16.043 DEBUG:boto:Params: {} 2023-07-29T00:05:16.044 DEBUG:boto:Token: None 2023-07-29T00:05:16.044 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:05:16 GMT /hiiqos-46/ 2023-07-29T00:05:16.044 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:su/4kocKBkSLWYM6U6X9T8tKZrM= 2023-07-29T00:05:16.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:su/4kocKBkSLWYM6U6X9T8tKZrM=', 'Content-Length': '0'} 2023-07-29T00:05:16.047 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000077c62b03074cbaf7-0064c457bc-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:16.047 DEBUG:boto:path=/hiiqos-46/obj2 2023-07-29T00:05:16.048 DEBUG:boto:auth_path=/hiiqos-46/obj2 2023-07-29T00:05:16.048 DEBUG:boto:Method: PUT 2023-07-29T00:05:16.049 DEBUG:boto:Path: /hiiqos-46/obj2 2023-07-29T00:05:16.049 DEBUG:boto:Data: 2023-07-29T00:05:16.049 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T00:05:16.050 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:05:16.050 DEBUG:boto:Port: 8000 2023-07-29T00:05:16.050 DEBUG:boto:Params: {} 2023-07-29T00:05:16.051 DEBUG:boto:Token: None 2023-07-29T00:05:16.051 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 00:05:16 GMT /hiiqos-46/obj2 2023-07-29T00:05:16.051 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0LZNNJEfSgmkg4SXxbkV8taiJtQ= 2023-07-29T00:05:16.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:05:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0LZNNJEfSgmkg4SXxbkV8taiJtQ='} 2023-07-29T00:05:16.056 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000001db4f60c30396f2-0064c457bc-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:16.057 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:05:16.057 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:05:16.058 DEBUG:boto:Method: HEAD 2023-07-29T00:05:16.058 DEBUG:boto:Path: /hiiqos-46/ 2023-07-29T00:05:16.059 DEBUG:boto:Data: 2023-07-29T00:05:16.059 DEBUG:boto:Headers: {} 2023-07-29T00:05:16.059 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:05:16.060 DEBUG:boto:Port: 8000 2023-07-29T00:05:16.060 DEBUG:boto:Params: {} 2023-07-29T00:05:16.061 DEBUG:boto:Token: None 2023-07-29T00:05:16.061 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:05:16 GMT /hiiqos-46/ 2023-07-29T00:05:16.062 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:su/4kocKBkSLWYM6U6X9T8tKZrM= 2023-07-29T00:05:16.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:su/4kocKBkSLWYM6U6X9T8tKZrM=', 'Content-Length': '0'} 2023-07-29T00:05:16.065 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b5d12897163dc855-0064c457bc-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:16.065 DEBUG:boto:path=/hiiqos-46/obj3 2023-07-29T00:05:16.066 DEBUG:boto:auth_path=/hiiqos-46/obj3 2023-07-29T00:05:16.066 DEBUG:boto:Method: PUT 2023-07-29T00:05:16.067 DEBUG:boto:Path: /hiiqos-46/obj3 2023-07-29T00:05:16.067 DEBUG:boto:Data: 2023-07-29T00:05:16.067 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T00:05:16.068 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:05:16.068 DEBUG:boto:Port: 8000 2023-07-29T00:05:16.069 DEBUG:boto:Params: {} 2023-07-29T00:05:16.069 DEBUG:boto:Token: None 2023-07-29T00:05:16.070 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 00:05:16 GMT /hiiqos-46/obj3 2023-07-29T00:05:16.070 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Wcw3twidI18gW27KB65kP2WF0YM= 2023-07-29T00:05:16.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:05:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Wcw3twidI18gW27KB65kP2WF0YM='} 2023-07-29T00:05:16.112 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e6f1cc937d985c29-0064c457bc-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:16.113 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:05:16.113 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:05:16.114 DEBUG:boto:Method: HEAD 2023-07-29T00:05:16.114 DEBUG:boto:Path: /hiiqos-46/ 2023-07-29T00:05:16.114 DEBUG:boto:Data: 2023-07-29T00:05:16.115 DEBUG:boto:Headers: {} 2023-07-29T00:05:16.115 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:05:16.115 DEBUG:boto:Port: 8000 2023-07-29T00:05:16.116 DEBUG:boto:Params: {} 2023-07-29T00:05:16.116 DEBUG:boto:Token: None 2023-07-29T00:05:16.117 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:05:16 GMT /hiiqos-46/ 2023-07-29T00:05:16.118 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:su/4kocKBkSLWYM6U6X9T8tKZrM= 2023-07-29T00:05:16.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:su/4kocKBkSLWYM6U6X9T8tKZrM=', 'Content-Length': '0'} 2023-07-29T00:05:16.121 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d95a7a8420f2735-0064c457bc-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:16.122 DEBUG:boto:path=/hiiqos-46/obj4 2023-07-29T00:05:16.123 DEBUG:boto:auth_path=/hiiqos-46/obj4 2023-07-29T00:05:16.124 DEBUG:boto:Method: PUT 2023-07-29T00:05:16.124 DEBUG:boto:Path: /hiiqos-46/obj4 2023-07-29T00:05:16.125 DEBUG:boto:Data: 2023-07-29T00:05:16.126 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T00:05:16.127 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:05:16.128 DEBUG:boto:Port: 8000 2023-07-29T00:05:16.129 DEBUG:boto:Params: {} 2023-07-29T00:05:16.129 DEBUG:boto:Token: None 2023-07-29T00:05:16.130 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 00:05:16 GMT /hiiqos-46/obj4 2023-07-29T00:05:16.130 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eG7RXW43fW5+8SDLskcq2sw8jNg= 2023-07-29T00:05:16.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:05:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eG7RXW43fW5+8SDLskcq2sw8jNg='} 2023-07-29T00:05:16.138 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e7cc650a98cd546f-0064c457bc-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:16.138 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:05:16.139 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:05:16.140 DEBUG:boto:Method: HEAD 2023-07-29T00:05:16.140 DEBUG:boto:Path: /hiiqos-47/ 2023-07-29T00:05:16.141 DEBUG:boto:Data: 2023-07-29T00:05:16.141 DEBUG:boto:Headers: {} 2023-07-29T00:05:16.142 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:05:16.142 DEBUG:boto:Port: 8001 2023-07-29T00:05:16.143 DEBUG:boto:Params: {} 2023-07-29T00:05:16.143 DEBUG:boto:Token: None 2023-07-29T00:05:16.144 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:05:16 GMT /hiiqos-47/ 2023-07-29T00:05:16.145 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ay1jPtVBfjNmIQVTP2v7tGffgS4= 2023-07-29T00:05:16.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ay1jPtVBfjNmIQVTP2v7tGffgS4=', 'Content-Length': '0'} 2023-07-29T00:05:16.148 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005e6c1b249d252f25-0064c457bc-1663-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:16.148 DEBUG:boto:path=/hiiqos-47/obj1 2023-07-29T00:05:16.149 DEBUG:boto:auth_path=/hiiqos-47/obj1 2023-07-29T00:05:16.149 DEBUG:boto:Method: PUT 2023-07-29T00:05:16.150 DEBUG:boto:Path: /hiiqos-47/obj1 2023-07-29T00:05:16.150 DEBUG:boto:Data: 2023-07-29T00:05:16.150 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T00:05:16.151 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:05:16.151 DEBUG:boto:Port: 8001 2023-07-29T00:05:16.151 DEBUG:boto:Params: {} 2023-07-29T00:05:16.152 DEBUG:boto:Token: None 2023-07-29T00:05:16.152 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 00:05:16 GMT /hiiqos-47/obj1 2023-07-29T00:05:16.153 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vgK1+rrYYKrGDqwAs4LhjxBoFeg= 2023-07-29T00:05:16.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:05:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vgK1+rrYYKrGDqwAs4LhjxBoFeg='} 2023-07-29T00:05:16.160 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c77553a831bc83a4-0064c457bc-1663-a2'), ('Date', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:16.161 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:05:16.161 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:05:16.161 DEBUG:boto:Method: HEAD 2023-07-29T00:05:16.162 DEBUG:boto:Path: /hiiqos-47/ 2023-07-29T00:05:16.162 DEBUG:boto:Data: 2023-07-29T00:05:16.162 DEBUG:boto:Headers: {} 2023-07-29T00:05:16.163 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:05:16.163 DEBUG:boto:Port: 8001 2023-07-29T00:05:16.163 DEBUG:boto:Params: {} 2023-07-29T00:05:16.164 DEBUG:boto:Token: None 2023-07-29T00:05:16.164 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:05:16 GMT /hiiqos-47/ 2023-07-29T00:05:16.164 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ay1jPtVBfjNmIQVTP2v7tGffgS4= 2023-07-29T00:05:16.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ay1jPtVBfjNmIQVTP2v7tGffgS4=', 'Content-Length': '0'} 2023-07-29T00:05:16.167 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000084effff6bd5d74a7-0064c457bc-1663-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:16.168 DEBUG:boto:path=/hiiqos-47/obj2 2023-07-29T00:05:16.168 DEBUG:boto:auth_path=/hiiqos-47/obj2 2023-07-29T00:05:16.168 DEBUG:boto:Method: PUT 2023-07-29T00:05:16.169 DEBUG:boto:Path: /hiiqos-47/obj2 2023-07-29T00:05:16.169 DEBUG:boto:Data: 2023-07-29T00:05:16.169 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T00:05:16.170 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:05:16.170 DEBUG:boto:Port: 8001 2023-07-29T00:05:16.170 DEBUG:boto:Params: {} 2023-07-29T00:05:16.171 DEBUG:boto:Token: None 2023-07-29T00:05:16.171 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 00:05:16 GMT /hiiqos-47/obj2 2023-07-29T00:05:16.171 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:070HWVmRrbxGOGHGyb+xqvNEM/o= 2023-07-29T00:05:16.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:05:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:070HWVmRrbxGOGHGyb+xqvNEM/o='} 2023-07-29T00:05:16.177 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001f855a0f59a258f6-0064c457bc-1663-a2'), ('Date', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:16.178 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:05:16.178 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:05:16.179 DEBUG:boto:Method: HEAD 2023-07-29T00:05:16.179 DEBUG:boto:Path: /hiiqos-47/ 2023-07-29T00:05:16.180 DEBUG:boto:Data: 2023-07-29T00:05:16.180 DEBUG:boto:Headers: {} 2023-07-29T00:05:16.181 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:05:16.181 DEBUG:boto:Port: 8001 2023-07-29T00:05:16.182 DEBUG:boto:Params: {} 2023-07-29T00:05:16.183 DEBUG:boto:Token: None 2023-07-29T00:05:16.183 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:05:16 GMT /hiiqos-47/ 2023-07-29T00:05:16.184 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ay1jPtVBfjNmIQVTP2v7tGffgS4= 2023-07-29T00:05:16.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ay1jPtVBfjNmIQVTP2v7tGffgS4=', 'Content-Length': '0'} 2023-07-29T00:05:16.187 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000456210aabce15e8c-0064c457bc-1663-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:16.187 DEBUG:boto:path=/hiiqos-47/obj3 2023-07-29T00:05:16.188 DEBUG:boto:auth_path=/hiiqos-47/obj3 2023-07-29T00:05:16.188 DEBUG:boto:Method: PUT 2023-07-29T00:05:16.189 DEBUG:boto:Path: /hiiqos-47/obj3 2023-07-29T00:05:16.189 DEBUG:boto:Data: 2023-07-29T00:05:16.189 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T00:05:16.190 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:05:16.190 DEBUG:boto:Port: 8001 2023-07-29T00:05:16.190 DEBUG:boto:Params: {} 2023-07-29T00:05:16.191 DEBUG:boto:Token: None 2023-07-29T00:05:16.191 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 00:05:16 GMT /hiiqos-47/obj3 2023-07-29T00:05:16.191 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6A6LOokYpc2bPyT1EqUh5Vs3MUQ= 2023-07-29T00:05:16.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:05:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6A6LOokYpc2bPyT1EqUh5Vs3MUQ='} 2023-07-29T00:05:16.197 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a6af0f18c3569ebe-0064c457bc-1663-a2'), ('Date', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:16.198 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:05:16.198 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:05:16.198 DEBUG:boto:Method: HEAD 2023-07-29T00:05:16.199 DEBUG:boto:Path: /hiiqos-47/ 2023-07-29T00:05:16.199 DEBUG:boto:Data: 2023-07-29T00:05:16.199 DEBUG:boto:Headers: {} 2023-07-29T00:05:16.200 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:05:16.200 DEBUG:boto:Port: 8001 2023-07-29T00:05:16.200 DEBUG:boto:Params: {} 2023-07-29T00:05:16.201 DEBUG:boto:Token: None 2023-07-29T00:05:16.201 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:05:16 GMT /hiiqos-47/ 2023-07-29T00:05:16.202 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ay1jPtVBfjNmIQVTP2v7tGffgS4= 2023-07-29T00:05:16.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ay1jPtVBfjNmIQVTP2v7tGffgS4=', 'Content-Length': '0'} 2023-07-29T00:05:16.204 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003dbc12b115eefe6e-0064c457bc-1663-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:16.205 DEBUG:boto:path=/hiiqos-47/obj4 2023-07-29T00:05:16.205 DEBUG:boto:auth_path=/hiiqos-47/obj4 2023-07-29T00:05:16.206 DEBUG:boto:Method: PUT 2023-07-29T00:05:16.206 DEBUG:boto:Path: /hiiqos-47/obj4 2023-07-29T00:05:16.206 DEBUG:boto:Data: 2023-07-29T00:05:16.207 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T00:05:16.207 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:05:16.207 DEBUG:boto:Port: 8001 2023-07-29T00:05:16.208 DEBUG:boto:Params: {} 2023-07-29T00:05:16.208 DEBUG:boto:Token: None 2023-07-29T00:05:16.208 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 00:05:16 GMT /hiiqos-47/obj4 2023-07-29T00:05:16.209 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Jp9XAzPq/QCuluBIdwlUHZSUlA0= 2023-07-29T00:05:16.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:05:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Jp9XAzPq/QCuluBIdwlUHZSUlA0='} 2023-07-29T00:05:16.252 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006683baee8abdc5d6-0064c457bc-1663-a2'), ('Date', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:16.253 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:05:16.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:05:16.254 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:05:16.298 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:05:16.299 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:05:16.327 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T00:05:16.340 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T00:05:16.340 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:05:16.340 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:05:16.340 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:46:09.233742Z" 2023-07-29T00:05:16.340 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:16.340 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:05:16.341 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690589115.991466_676.1", 2023-07-29T00:05:16.341 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:05:15.991466Z" 2023-07-29T00:05:16.341 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:16.341 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:05:16.341 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:05:16.341 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T00:05:16.342 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:16.342 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:05:16.342 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:05:16.342 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T00:05:16.342 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:05:16.342 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T00:05:16.343 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690587969.233742_99.1', 1: '1_1690589115.991466_676.1', 2: '', 3: ''} 2023-07-29T00:05:16.343 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T00:05:16.344 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:05:16.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:05:16.345 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:05:16.392 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:05:16.392 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:05:16.421 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:05:16.400+0000 7efe9f726880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:05:16.422 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:05:16.401+0000 7efe9f726880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:05:16.434 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:05:16.434 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:05:16.435 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:05:16.435 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:05:16.435 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:05:16.435 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:05:16.435 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:05:16.435 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:16.436 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:05:16.436 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:05:16.436 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:05:16.436 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:05:16.436 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:05:16.436 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:05:16.437 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:05:16.437 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:05:16.437 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:05:16.437 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:05:16.437 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:05:16.437 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:05:16.437 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:16.438 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:05:16.438 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:05:16.438 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:05:16.438 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:05:16.438 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690589094.687543_661.1", 2023-07-29T00:05:16.438 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:05:16.438 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:05:16.439 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:05:16.439 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:04:54.687543Z", 2023-07-29T00:05:16.439 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:05:16.439 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:05:16.439 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:16.439 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:05:16.439 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:05:16.440 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:05:16.440 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:05:16.440 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:05:16.440 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:05:16.440 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:05:16.440 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:05:16.441 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:05:16.441 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:05:16.441 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:05:16.441 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:16.441 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:05:16.441 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:05:16.441 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:05:16.442 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:05:16.442 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:05:16.442 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:05:16.442 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:05:16.442 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:05:16.442 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:05:16.442 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:05:16.443 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:05:16.443 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:05:16.443 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:05:16.443 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:16.443 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:05:16.443 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:05:16.444 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:05:16.444 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:16.444 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:05:16Z" 2023-07-29T00:05:16.444 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:05:16.445 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690589094.687543_661.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:04:54.687543Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:05:16Z" } 2023-07-29T00:05:16.445 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690589094.687543_661.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:04:54.687543Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:05:16.446 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690589115.991466_676.1', 2: '', 3: ''} 2023-07-29T00:05:16.446 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690589094.687543_661.1', 2: '', 3: ''} 2023-07-29T00:05:16.446 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690589115.991466_676.1 target=1_1690589094.687543_661.1 2023-07-29T00:05:21.448 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:05:21.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:05:21.449 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:05:21.503 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:05:21.504 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:05:21.536 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:05:21.514+0000 7fb52f535880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:05:21.537 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:05:21.515+0000 7fb52f535880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:05:21.550 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:05:21.550 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:05:21.550 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:05:21.550 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:05:21.551 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:05:21.551 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:05:21.551 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:05:21.551 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:21.551 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:05:21.551 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:05:21.551 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:05:21.552 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:05:21.552 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:05:21.552 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:05:21.552 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:05:21.552 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:05:21.552 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:05:21.553 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:05:21.553 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:05:21.553 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:05:21.553 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:21.553 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:05:21.553 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:05:21.553 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:05:21.554 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:05:21.554 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690589115.991466_676.1", 2023-07-29T00:05:21.554 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:05:21.554 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:05:21.554 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:05:21.554 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:05:15.991466Z", 2023-07-29T00:05:21.554 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:05:21.555 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:05:21.555 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:21.555 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:05:21.555 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:05:21.555 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:05:21.555 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:05:21.555 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:05:21.556 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:05:21.556 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:05:21.556 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:05:21.556 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:05:21.556 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:05:21.556 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:05:21.557 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:21.557 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:05:21.557 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:05:21.557 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:05:21.557 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:05:21.557 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:05:21.557 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:05:21.558 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:05:21.558 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:05:21.558 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:05:21.558 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:05:21.558 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:05:21.558 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:05:21.558 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:05:21.559 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:21.559 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:05:21.559 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:05:21.559 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:05:21.559 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:05:21.559 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:05:21Z" 2023-07-29T00:05:21.560 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:05:21.561 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690589115.991466_676.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:05:15.991466Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:05:21Z" } 2023-07-29T00:05:21.561 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690589115.991466_676.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:05:15.991466Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:05:21.561 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690589115.991466_676.1', 2: '', 3: ''} 2023-07-29T00:05:21.562 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690589115.991466_676.1', 2: '', 3: ''} 2023-07-29T00:05:21.562 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T00:05:21.562 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-46 2023-07-29T00:05:21.563 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-46', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:05:21.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-46', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:05:21.563 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-46 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:05:21.616 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:05:21.620 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:05:21.649 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:05:21.627+0000 7f5fbec0f880 1 waiting to reach incremental sync.. 2023-07-29T00:05:26.650 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:05:26.628+0000 7f5fbec0f880 1 waiting to reach incremental sync.. 2023-07-29T00:05:31.651 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:05:31.629+0000 7f5fbec0f880 1 waiting to reach incremental sync.. 2023-07-29T00:05:36.653 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:05:36.631+0000 7f5fbec0f880 1 bucket sync caught up with source: 2023-07-29T00:05:36.653 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000001.22.6, , 00000000001.27.6, , 00000000001.26.6, , , 00000000001.22.6, , , ] 2023-07-29T00:05:36.654 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000001.22.6, , 00000000001.27.6, , 00000000001.26.6, , , 00000000001.22.6, , , ] 2023-07-29T00:05:36.654 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:05:36.631+0000 7f5fbec0f880 0 bucket checkpoint complete 2023-07-29T00:05:36.670 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-46 2023-07-29T00:05:36.670 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-46', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:05:36.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-46', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:05:36.671 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-46 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:05:36.729 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:05:36.729 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:05:36.765 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:05:36.742+0000 7ff1a6975880 1 bucket sync caught up with source: 2023-07-29T00:05:36.765 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000001.1166.6, , 00000000001.864.6, , 00000000001.526.6, , , 00000000001.372.6, , , ] 2023-07-29T00:05:36.766 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000001.1166.6, , 00000000001.864.6, , 00000000001.526.6, , , 00000000001.372.6, , , ] 2023-07-29T00:05:36.766 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:05:36.742+0000 7ff1a6975880 0 bucket checkpoint complete 2023-07-29T00:05:36.778 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-46 zones={a2, a1} 2023-07-29T00:05:36.778 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:05:36.779 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:05:36.779 DEBUG:boto:Method: HEAD 2023-07-29T00:05:36.780 DEBUG:boto:Path: /hiiqos-46/ 2023-07-29T00:05:36.780 DEBUG:boto:Data: 2023-07-29T00:05:36.780 DEBUG:boto:Headers: {} 2023-07-29T00:05:36.781 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:05:36.781 DEBUG:boto:Port: 8001 2023-07-29T00:05:36.781 DEBUG:boto:Params: {} 2023-07-29T00:05:36.782 DEBUG:boto:Token: None 2023-07-29T00:05:36.782 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:05:36 GMT /hiiqos-46/ 2023-07-29T00:05:36.783 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nFuesugQ51SrsaziWdJzdfeubeA= 2023-07-29T00:05:36.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nFuesugQ51SrsaziWdJzdfeubeA=', 'Content-Length': '0'} 2023-07-29T00:05:36.786 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000305e1c5423b3e130-0064c457d0-1663-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:36.786 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:05:36.787 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:05:36.787 DEBUG:boto:Method: HEAD 2023-07-29T00:05:36.788 DEBUG:boto:Path: /hiiqos-46/ 2023-07-29T00:05:36.788 DEBUG:boto:Data: 2023-07-29T00:05:36.789 DEBUG:boto:Headers: {} 2023-07-29T00:05:36.789 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:05:36.789 DEBUG:boto:Port: 8000 2023-07-29T00:05:36.790 DEBUG:boto:Params: {} 2023-07-29T00:05:36.791 DEBUG:boto:Token: None 2023-07-29T00:05:36.792 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:05:36 GMT /hiiqos-46/ 2023-07-29T00:05:36.792 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nFuesugQ51SrsaziWdJzdfeubeA= 2023-07-29T00:05:36.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nFuesugQ51SrsaziWdJzdfeubeA=', 'Content-Length': '0'} 2023-07-29T00:05:36.796 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a0ebed88bec23535-0064c457d0-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:36.797 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-29T00:05:36.797 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:05:36.798 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:05:36.799 DEBUG:boto:path=/hiiqos-46/?versions&max-keys=999 2023-07-29T00:05:36.799 DEBUG:boto:auth_path=/hiiqos-46/?versions&max-keys=999 2023-07-29T00:05:36.800 DEBUG:boto:Method: GET 2023-07-29T00:05:36.801 DEBUG:boto:Path: /hiiqos-46/?versions&max-keys=999 2023-07-29T00:05:36.801 DEBUG:boto:Data: 2023-07-29T00:05:36.802 DEBUG:boto:Headers: {} 2023-07-29T00:05:36.803 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:05:36.803 DEBUG:boto:Port: 8001 2023-07-29T00:05:36.804 DEBUG:boto:Params: {} 2023-07-29T00:05:36.805 DEBUG:boto:Token: None 2023-07-29T00:05:36.805 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:36 GMT /hiiqos-46/?versions 2023-07-29T00:05:36.806 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FguIYcYR1MwO1HPkB1H62Pee5w4= 2023-07-29T00:05:36.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FguIYcYR1MwO1HPkB1H62Pee5w4=', 'Content-Length': '0'} 2023-07-29T00:05:36.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033dfe7fa926ef66a-0064c457d0-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:36.810 DEBUG:boto:b'hiiqos-46999falseobj1nulltrue2023-07-29T00:05:16.017Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-07-29T00:05:16.033Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-07-29T00:05:16.052Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-07-29T00:05:16.112Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:05:36.812 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-07-29T00:05:36.812 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-07-29T00:05:36.813 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-07-29T00:05:36.814 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-07-29T00:05:36.814 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-29T00:05:36.815 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:05:36.815 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:05:36.816 DEBUG:boto:path=/hiiqos-46/?versions&max-keys=999 2023-07-29T00:05:36.816 DEBUG:boto:auth_path=/hiiqos-46/?versions&max-keys=999 2023-07-29T00:05:36.817 DEBUG:boto:Method: GET 2023-07-29T00:05:36.817 DEBUG:boto:Path: /hiiqos-46/?versions&max-keys=999 2023-07-29T00:05:36.818 DEBUG:boto:Data: 2023-07-29T00:05:36.818 DEBUG:boto:Headers: {} 2023-07-29T00:05:36.818 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:05:36.819 DEBUG:boto:Port: 8000 2023-07-29T00:05:36.819 DEBUG:boto:Params: {} 2023-07-29T00:05:36.819 DEBUG:boto:Token: None 2023-07-29T00:05:36.820 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:36 GMT /hiiqos-46/?versions 2023-07-29T00:05:36.821 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FguIYcYR1MwO1HPkB1H62Pee5w4= 2023-07-29T00:05:36.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FguIYcYR1MwO1HPkB1H62Pee5w4=', 'Content-Length': '0'} 2023-07-29T00:05:36.824 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088172ce3b2adbb65-0064c457d0-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:36.824 DEBUG:boto:b'hiiqos-46999falseobj1nulltrue2023-07-29T00:05:16.017Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-07-29T00:05:16.033Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-07-29T00:05:16.052Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-07-29T00:05:16.112Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:05:36.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-07-29T00:05:36.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-07-29T00:05:36.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-07-29T00:05:36.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-07-29T00:05:36.827 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:05:36.827 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:05:36.827 DEBUG:boto:path=/hiiqos-46/?versions&max-keys=999 2023-07-29T00:05:36.828 DEBUG:boto:auth_path=/hiiqos-46/?versions&max-keys=999 2023-07-29T00:05:36.828 DEBUG:boto:Method: GET 2023-07-29T00:05:36.829 DEBUG:boto:Path: /hiiqos-46/?versions&max-keys=999 2023-07-29T00:05:36.829 DEBUG:boto:Data: 2023-07-29T00:05:36.829 DEBUG:boto:Headers: {} 2023-07-29T00:05:36.830 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:05:36.830 DEBUG:boto:Port: 8001 2023-07-29T00:05:36.830 DEBUG:boto:Params: {} 2023-07-29T00:05:36.831 DEBUG:boto:Token: None 2023-07-29T00:05:36.831 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:36 GMT /hiiqos-46/?versions 2023-07-29T00:05:36.832 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FguIYcYR1MwO1HPkB1H62Pee5w4= 2023-07-29T00:05:36.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FguIYcYR1MwO1HPkB1H62Pee5w4=', 'Content-Length': '0'} 2023-07-29T00:05:36.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000049f8f6e7c8547d5-0064c457d0-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:36.835 DEBUG:boto:b'hiiqos-46999falseobj1nulltrue2023-07-29T00:05:16.017Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-07-29T00:05:16.033Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-07-29T00:05:16.052Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-07-29T00:05:16.112Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:05:36.836 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:05:36.836 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:05:36.837 DEBUG:boto:path=/hiiqos-46/?versions&max-keys=999 2023-07-29T00:05:36.837 DEBUG:boto:auth_path=/hiiqos-46/?versions&max-keys=999 2023-07-29T00:05:36.837 DEBUG:boto:Method: GET 2023-07-29T00:05:36.838 DEBUG:boto:Path: /hiiqos-46/?versions&max-keys=999 2023-07-29T00:05:36.838 DEBUG:boto:Data: 2023-07-29T00:05:36.838 DEBUG:boto:Headers: {} 2023-07-29T00:05:36.839 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:05:36.839 DEBUG:boto:Port: 8000 2023-07-29T00:05:36.839 DEBUG:boto:Params: {} 2023-07-29T00:05:36.840 DEBUG:boto:Token: None 2023-07-29T00:05:36.840 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:36 GMT /hiiqos-46/?versions 2023-07-29T00:05:36.841 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FguIYcYR1MwO1HPkB1H62Pee5w4= 2023-07-29T00:05:36.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FguIYcYR1MwO1HPkB1H62Pee5w4=', 'Content-Length': '0'} 2023-07-29T00:05:36.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000daf0eead93cb37b7-0064c457d0-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:36.844 DEBUG:boto:b'hiiqos-46999falseobj1nulltrue2023-07-29T00:05:16.017Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-07-29T00:05:16.033Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-07-29T00:05:16.052Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-07-29T00:05:16.112Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:05:36.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-07-29T00:05:36.845 DEBUG:boto:path=/hiiqos-46/obj1 2023-07-29T00:05:36.846 DEBUG:boto:auth_path=/hiiqos-46/obj1 2023-07-29T00:05:36.846 DEBUG:boto:path=/hiiqos-46/obj1?versionId=null 2023-07-29T00:05:36.847 DEBUG:boto:auth_path=/hiiqos-46/obj1?versionId=null 2023-07-29T00:05:36.847 DEBUG:boto:Method: GET 2023-07-29T00:05:36.848 DEBUG:boto:Path: /hiiqos-46/obj1?versionId=null 2023-07-29T00:05:36.848 DEBUG:boto:Data: 2023-07-29T00:05:36.848 DEBUG:boto:Headers: {} 2023-07-29T00:05:36.849 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:05:36.849 DEBUG:boto:Port: 8001 2023-07-29T00:05:36.850 DEBUG:boto:Params: {} 2023-07-29T00:05:36.850 DEBUG:boto:Token: None 2023-07-29T00:05:36.851 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:36 GMT /hiiqos-46/obj1?versionId=null 2023-07-29T00:05:36.851 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fNFTuJeKt2mfdhoAjSsAsf0p0LI= 2023-07-29T00:05:36.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fNFTuJeKt2mfdhoAjSsAsf0p0LI=', 'Content-Length': '0'} 2023-07-29T00:05:36.853 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a688973e88e36b93-0064c457d0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:36.854 DEBUG:boto:path=/hiiqos-46/obj1 2023-07-29T00:05:36.854 DEBUG:boto:auth_path=/hiiqos-46/obj1 2023-07-29T00:05:36.855 DEBUG:boto:path=/hiiqos-46/obj1?versionId=null 2023-07-29T00:05:36.855 DEBUG:boto:auth_path=/hiiqos-46/obj1?versionId=null 2023-07-29T00:05:36.856 DEBUG:boto:Method: GET 2023-07-29T00:05:36.856 DEBUG:boto:Path: /hiiqos-46/obj1?versionId=null 2023-07-29T00:05:36.857 DEBUG:boto:Data: 2023-07-29T00:05:36.857 DEBUG:boto:Headers: {} 2023-07-29T00:05:36.858 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:05:36.858 DEBUG:boto:Port: 8000 2023-07-29T00:05:36.859 DEBUG:boto:Params: {} 2023-07-29T00:05:36.860 DEBUG:boto:Token: None 2023-07-29T00:05:36.860 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:36 GMT /hiiqos-46/obj1?versionId=null 2023-07-29T00:05:36.861 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fNFTuJeKt2mfdhoAjSsAsf0p0LI= 2023-07-29T00:05:36.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fNFTuJeKt2mfdhoAjSsAsf0p0LI=', 'Content-Length': '0'} 2023-07-29T00:05:36.864 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a6e6c0365311e9fe-0064c457d0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:36.865 DEBUG:boto:path=/hiiqos-46/obj1 2023-07-29T00:05:36.865 DEBUG:boto:auth_path=/hiiqos-46/obj1 2023-07-29T00:05:36.866 DEBUG:boto:path=/hiiqos-46/obj1?versionId=null 2023-07-29T00:05:36.866 DEBUG:boto:auth_path=/hiiqos-46/obj1?versionId=null 2023-07-29T00:05:36.867 DEBUG:boto:Method: HEAD 2023-07-29T00:05:36.867 DEBUG:boto:Path: /hiiqos-46/obj1?versionId=null 2023-07-29T00:05:36.868 DEBUG:boto:Data: 2023-07-29T00:05:36.869 DEBUG:boto:Headers: {} 2023-07-29T00:05:36.869 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:05:36.870 DEBUG:boto:Port: 8001 2023-07-29T00:05:36.870 DEBUG:boto:Params: {} 2023-07-29T00:05:36.871 DEBUG:boto:Token: None 2023-07-29T00:05:36.871 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:05:36 GMT /hiiqos-46/obj1?versionId=null 2023-07-29T00:05:36.872 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JDKRosmLeupjKuqPBWRgQse8J2E= 2023-07-29T00:05:36.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JDKRosmLeupjKuqPBWRgQse8J2E=', 'Content-Length': '0'} 2023-07-29T00:05:36.874 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bedd525aed213045-0064c457d0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:36.874 DEBUG:boto:path=/hiiqos-46/obj1 2023-07-29T00:05:36.875 DEBUG:boto:auth_path=/hiiqos-46/obj1 2023-07-29T00:05:36.875 DEBUG:boto:path=/hiiqos-46/obj1?versionId=null 2023-07-29T00:05:36.875 DEBUG:boto:auth_path=/hiiqos-46/obj1?versionId=null 2023-07-29T00:05:36.876 DEBUG:boto:Method: HEAD 2023-07-29T00:05:36.876 DEBUG:boto:Path: /hiiqos-46/obj1?versionId=null 2023-07-29T00:05:36.876 DEBUG:boto:Data: 2023-07-29T00:05:36.877 DEBUG:boto:Headers: {} 2023-07-29T00:05:36.877 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:05:36.878 DEBUG:boto:Port: 8000 2023-07-29T00:05:36.878 DEBUG:boto:Params: {} 2023-07-29T00:05:36.878 DEBUG:boto:Token: None 2023-07-29T00:05:36.879 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:05:36 GMT /hiiqos-46/obj1?versionId=null 2023-07-29T00:05:36.879 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JDKRosmLeupjKuqPBWRgQse8J2E= 2023-07-29T00:05:36.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JDKRosmLeupjKuqPBWRgQse8J2E=', 'Content-Length': '0'} 2023-07-29T00:05:36.881 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000010f48bd70a3fc430-0064c457d0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:36.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-07-29T00:05:36.882 DEBUG:boto:path=/hiiqos-46/obj1 2023-07-29T00:05:36.882 DEBUG:boto:auth_path=/hiiqos-46/obj1 2023-07-29T00:05:36.883 DEBUG:boto:path=/hiiqos-46/obj1?versionId=null 2023-07-29T00:05:36.884 DEBUG:boto:auth_path=/hiiqos-46/obj1?versionId=null 2023-07-29T00:05:36.884 DEBUG:boto:Method: GET 2023-07-29T00:05:36.885 DEBUG:boto:Path: /hiiqos-46/obj1?versionId=null 2023-07-29T00:05:36.885 DEBUG:boto:Data: 2023-07-29T00:05:36.886 DEBUG:boto:Headers: {} 2023-07-29T00:05:36.886 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:05:36.887 DEBUG:boto:Port: 8001 2023-07-29T00:05:36.887 DEBUG:boto:Params: {} 2023-07-29T00:05:36.887 DEBUG:boto:Token: None 2023-07-29T00:05:36.888 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:36 GMT /hiiqos-46/obj1?versionId=null 2023-07-29T00:05:36.888 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fNFTuJeKt2mfdhoAjSsAsf0p0LI= 2023-07-29T00:05:36.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fNFTuJeKt2mfdhoAjSsAsf0p0LI=', 'Content-Length': '0'} 2023-07-29T00:05:36.890 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f45a78f0ee2fad6-0064c457d0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:36.891 DEBUG:boto:path=/hiiqos-46/obj1 2023-07-29T00:05:36.891 DEBUG:boto:auth_path=/hiiqos-46/obj1 2023-07-29T00:05:36.892 DEBUG:boto:path=/hiiqos-46/obj1?versionId=null 2023-07-29T00:05:36.892 DEBUG:boto:auth_path=/hiiqos-46/obj1?versionId=null 2023-07-29T00:05:36.893 DEBUG:boto:Method: GET 2023-07-29T00:05:36.893 DEBUG:boto:Path: /hiiqos-46/obj1?versionId=null 2023-07-29T00:05:36.893 DEBUG:boto:Data: 2023-07-29T00:05:36.894 DEBUG:boto:Headers: {} 2023-07-29T00:05:36.894 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:05:36.894 DEBUG:boto:Port: 8000 2023-07-29T00:05:36.895 DEBUG:boto:Params: {} 2023-07-29T00:05:36.895 DEBUG:boto:Token: None 2023-07-29T00:05:36.896 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:36 GMT /hiiqos-46/obj1?versionId=null 2023-07-29T00:05:36.896 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fNFTuJeKt2mfdhoAjSsAsf0p0LI= 2023-07-29T00:05:36.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fNFTuJeKt2mfdhoAjSsAsf0p0LI=', 'Content-Length': '0'} 2023-07-29T00:05:36.899 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001fc1fd3072a363be-0064c457d0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:36.899 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:05:36.900 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:05:36.900 DEBUG:boto:path=/hiiqos-46/?prefix=obj1 2023-07-29T00:05:36.901 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj1 2023-07-29T00:05:36.901 DEBUG:boto:Method: GET 2023-07-29T00:05:36.901 DEBUG:boto:Path: /hiiqos-46/?prefix=obj1 2023-07-29T00:05:36.902 DEBUG:boto:Data: 2023-07-29T00:05:36.902 DEBUG:boto:Headers: {} 2023-07-29T00:05:36.903 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:05:36.903 DEBUG:boto:Port: 8001 2023-07-29T00:05:36.903 DEBUG:boto:Params: {} 2023-07-29T00:05:36.904 DEBUG:boto:Token: None 2023-07-29T00:05:36.904 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:36 GMT /hiiqos-46/ 2023-07-29T00:05:36.905 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J4c6BdAGgZKhWAGX9OmHe7buenI= 2023-07-29T00:05:36.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J4c6BdAGgZKhWAGX9OmHe7buenI=', 'Content-Length': '0'} 2023-07-29T00:05:36.907 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000453daafd125431c0-0064c457d0-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:36.908 DEBUG:boto:b'hiiqos-46obj11000falseobj12023-07-29T00:05:16.017Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:05:36.909 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:05:36.909 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:05:36.909 DEBUG:boto:path=/hiiqos-46/?prefix=obj1 2023-07-29T00:05:36.910 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj1 2023-07-29T00:05:36.910 DEBUG:boto:Method: GET 2023-07-29T00:05:36.910 DEBUG:boto:Path: /hiiqos-46/?prefix=obj1 2023-07-29T00:05:36.911 DEBUG:boto:Data: 2023-07-29T00:05:36.911 DEBUG:boto:Headers: {} 2023-07-29T00:05:36.912 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:05:36.912 DEBUG:boto:Port: 8000 2023-07-29T00:05:36.912 DEBUG:boto:Params: {} 2023-07-29T00:05:36.913 DEBUG:boto:Token: None 2023-07-29T00:05:36.913 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:36 GMT /hiiqos-46/ 2023-07-29T00:05:36.914 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J4c6BdAGgZKhWAGX9OmHe7buenI= 2023-07-29T00:05:36.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J4c6BdAGgZKhWAGX9OmHe7buenI=', 'Content-Length': '0'} 2023-07-29T00:05:36.916 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091c6a6904aaa7857-0064c457d0-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:36.917 DEBUG:boto:b'hiiqos-46obj11000falseobj12023-07-29T00:05:16.017Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:05:36.917 DEBUG:boto:path=/hiiqos-46/obj1 2023-07-29T00:05:36.918 DEBUG:boto:auth_path=/hiiqos-46/obj1 2023-07-29T00:05:36.918 DEBUG:boto:Method: HEAD 2023-07-29T00:05:36.919 DEBUG:boto:Path: /hiiqos-46/obj1 2023-07-29T00:05:36.919 DEBUG:boto:Data: 2023-07-29T00:05:36.919 DEBUG:boto:Headers: {} 2023-07-29T00:05:36.920 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:05:36.920 DEBUG:boto:Port: 8001 2023-07-29T00:05:36.920 DEBUG:boto:Params: {} 2023-07-29T00:05:36.921 DEBUG:boto:Token: None 2023-07-29T00:05:36.921 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:05:36 GMT /hiiqos-46/obj1 2023-07-29T00:05:36.921 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:647E1oCFGNqYvGwS+O8JDEJ2u4s= 2023-07-29T00:05:36.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:647E1oCFGNqYvGwS+O8JDEJ2u4s=', 'Content-Length': '0'} 2023-07-29T00:05:36.923 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000006c94cc578798a33-0064c457d0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:36.924 DEBUG:boto:path=/hiiqos-46/obj1 2023-07-29T00:05:36.924 DEBUG:boto:auth_path=/hiiqos-46/obj1 2023-07-29T00:05:36.925 DEBUG:boto:Method: HEAD 2023-07-29T00:05:36.925 DEBUG:boto:Path: /hiiqos-46/obj1 2023-07-29T00:05:36.925 DEBUG:boto:Data: 2023-07-29T00:05:36.926 DEBUG:boto:Headers: {} 2023-07-29T00:05:36.926 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:05:36.926 DEBUG:boto:Port: 8000 2023-07-29T00:05:36.927 DEBUG:boto:Params: {} 2023-07-29T00:05:36.927 DEBUG:boto:Token: None 2023-07-29T00:05:36.928 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:05:36 GMT /hiiqos-46/obj1 2023-07-29T00:05:36.928 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:647E1oCFGNqYvGwS+O8JDEJ2u4s= 2023-07-29T00:05:36.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:647E1oCFGNqYvGwS+O8JDEJ2u4s=', 'Content-Length': '0'} 2023-07-29T00:05:36.930 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f42f94c4eee59c8d-0064c457d0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:36.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-07-29T00:05:36.931 DEBUG:boto:path=/hiiqos-46/obj1 2023-07-29T00:05:36.932 DEBUG:boto:auth_path=/hiiqos-46/obj1 2023-07-29T00:05:36.932 DEBUG:boto:Method: GET 2023-07-29T00:05:36.932 DEBUG:boto:Path: /hiiqos-46/obj1 2023-07-29T00:05:36.933 DEBUG:boto:Data: 2023-07-29T00:05:36.933 DEBUG:boto:Headers: {} 2023-07-29T00:05:36.934 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:05:36.934 DEBUG:boto:Port: 8001 2023-07-29T00:05:36.934 DEBUG:boto:Params: {} 2023-07-29T00:05:36.935 DEBUG:boto:Token: None 2023-07-29T00:05:36.935 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:36 GMT /hiiqos-46/obj1 2023-07-29T00:05:36.936 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4IDUy1zWYgkdudyTXcnXp2s3m1Q= 2023-07-29T00:05:36.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4IDUy1zWYgkdudyTXcnXp2s3m1Q=', 'Content-Length': '0'} 2023-07-29T00:05:36.938 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006cdfd629834cdc6a-0064c457d0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:36.938 DEBUG:boto:path=/hiiqos-46/obj1 2023-07-29T00:05:36.939 DEBUG:boto:auth_path=/hiiqos-46/obj1 2023-07-29T00:05:36.939 DEBUG:boto:Method: GET 2023-07-29T00:05:36.940 DEBUG:boto:Path: /hiiqos-46/obj1 2023-07-29T00:05:36.940 DEBUG:boto:Data: 2023-07-29T00:05:36.940 DEBUG:boto:Headers: {} 2023-07-29T00:05:36.941 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:05:36.941 DEBUG:boto:Port: 8000 2023-07-29T00:05:36.941 DEBUG:boto:Params: {} 2023-07-29T00:05:36.942 DEBUG:boto:Token: None 2023-07-29T00:05:36.942 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:36 GMT /hiiqos-46/obj1 2023-07-29T00:05:36.943 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4IDUy1zWYgkdudyTXcnXp2s3m1Q= 2023-07-29T00:05:36.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4IDUy1zWYgkdudyTXcnXp2s3m1Q=', 'Content-Length': '0'} 2023-07-29T00:05:36.945 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a12d4d8ac5c8fc0c-0064c457d0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:36.992 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:05:36.993 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:05:36.994 DEBUG:boto:path=/hiiqos-46/?prefix=obj1 2023-07-29T00:05:36.994 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj1 2023-07-29T00:05:36.995 DEBUG:boto:Method: GET 2023-07-29T00:05:36.995 DEBUG:boto:Path: /hiiqos-46/?prefix=obj1 2023-07-29T00:05:36.996 DEBUG:boto:Data: 2023-07-29T00:05:36.996 DEBUG:boto:Headers: {} 2023-07-29T00:05:36.997 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:05:36.997 DEBUG:boto:Port: 8001 2023-07-29T00:05:36.997 DEBUG:boto:Params: {} 2023-07-29T00:05:36.998 DEBUG:boto:Token: None 2023-07-29T00:05:36.998 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:36 GMT /hiiqos-46/ 2023-07-29T00:05:36.999 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J4c6BdAGgZKhWAGX9OmHe7buenI= 2023-07-29T00:05:36.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J4c6BdAGgZKhWAGX9OmHe7buenI=', 'Content-Length': '0'} 2023-07-29T00:05:37.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008882fed6fe4806b7-0064c457d0-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.002 DEBUG:boto:b'hiiqos-46obj11000falseobj12023-07-29T00:05:16.017Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:05:37.003 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:05:37.004 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:05:37.004 DEBUG:boto:path=/hiiqos-46/?prefix=obj1 2023-07-29T00:05:37.004 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj1 2023-07-29T00:05:37.005 DEBUG:boto:Method: GET 2023-07-29T00:05:37.005 DEBUG:boto:Path: /hiiqos-46/?prefix=obj1 2023-07-29T00:05:37.005 DEBUG:boto:Data: 2023-07-29T00:05:37.006 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.006 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:05:37.007 DEBUG:boto:Port: 8000 2023-07-29T00:05:37.007 DEBUG:boto:Params: {} 2023-07-29T00:05:37.008 DEBUG:boto:Token: None 2023-07-29T00:05:37.008 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/ 2023-07-29T00:05:37.009 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IfseN/msjvmwQlULCXnANA9B94U= 2023-07-29T00:05:37.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IfseN/msjvmwQlULCXnANA9B94U=', 'Content-Length': '0'} 2023-07-29T00:05:37.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e289c405fb382e9-0064c457d0-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.012 DEBUG:boto:b'hiiqos-46obj11000falseobj12023-07-29T00:05:16.017Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:05:37.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-07-29T00:05:37.013 DEBUG:boto:path=/hiiqos-46/obj2 2023-07-29T00:05:37.013 DEBUG:boto:auth_path=/hiiqos-46/obj2 2023-07-29T00:05:37.014 DEBUG:boto:path=/hiiqos-46/obj2?versionId=null 2023-07-29T00:05:37.014 DEBUG:boto:auth_path=/hiiqos-46/obj2?versionId=null 2023-07-29T00:05:37.014 DEBUG:boto:Method: GET 2023-07-29T00:05:37.015 DEBUG:boto:Path: /hiiqos-46/obj2?versionId=null 2023-07-29T00:05:37.015 DEBUG:boto:Data: 2023-07-29T00:05:37.016 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.016 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:05:37.017 DEBUG:boto:Port: 8001 2023-07-29T00:05:37.017 DEBUG:boto:Params: {} 2023-07-29T00:05:37.017 DEBUG:boto:Token: None 2023-07-29T00:05:37.018 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/obj2?versionId=null 2023-07-29T00:05:37.018 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:M0/+5GsdwUbsj7Zad2lMT0Y69YI= 2023-07-29T00:05:37.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:M0/+5GsdwUbsj7Zad2lMT0Y69YI=', 'Content-Length': '0'} 2023-07-29T00:05:37.020 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba0567390964cd79-0064c457d0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:05:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.022 DEBUG:boto:path=/hiiqos-46/obj2 2023-07-29T00:05:37.023 DEBUG:boto:auth_path=/hiiqos-46/obj2 2023-07-29T00:05:37.024 DEBUG:boto:path=/hiiqos-46/obj2?versionId=null 2023-07-29T00:05:37.025 DEBUG:boto:auth_path=/hiiqos-46/obj2?versionId=null 2023-07-29T00:05:37.025 DEBUG:boto:Method: GET 2023-07-29T00:05:37.026 DEBUG:boto:Path: /hiiqos-46/obj2?versionId=null 2023-07-29T00:05:37.027 DEBUG:boto:Data: 2023-07-29T00:05:37.028 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.028 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:05:37.029 DEBUG:boto:Port: 8000 2023-07-29T00:05:37.030 DEBUG:boto:Params: {} 2023-07-29T00:05:37.031 DEBUG:boto:Token: None 2023-07-29T00:05:37.031 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/obj2?versionId=null 2023-07-29T00:05:37.031 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:M0/+5GsdwUbsj7Zad2lMT0Y69YI= 2023-07-29T00:05:37.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:M0/+5GsdwUbsj7Zad2lMT0Y69YI=', 'Content-Length': '0'} 2023-07-29T00:05:37.034 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000066a212091a15831e-0064c457d1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.034 DEBUG:boto:path=/hiiqos-46/obj2 2023-07-29T00:05:37.035 DEBUG:boto:auth_path=/hiiqos-46/obj2 2023-07-29T00:05:37.035 DEBUG:boto:path=/hiiqos-46/obj2?versionId=null 2023-07-29T00:05:37.035 DEBUG:boto:auth_path=/hiiqos-46/obj2?versionId=null 2023-07-29T00:05:37.036 DEBUG:boto:Method: HEAD 2023-07-29T00:05:37.036 DEBUG:boto:Path: /hiiqos-46/obj2?versionId=null 2023-07-29T00:05:37.036 DEBUG:boto:Data: 2023-07-29T00:05:37.037 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.037 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:05:37.038 DEBUG:boto:Port: 8001 2023-07-29T00:05:37.038 DEBUG:boto:Params: {} 2023-07-29T00:05:37.039 DEBUG:boto:Token: None 2023-07-29T00:05:37.039 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/obj2?versionId=null 2023-07-29T00:05:37.039 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FeCIJqOV4WSyPJ+dEIMEdQ5Orvw= 2023-07-29T00:05:37.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FeCIJqOV4WSyPJ+dEIMEdQ5Orvw=', 'Content-Length': '0'} 2023-07-29T00:05:37.041 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005422ae2e11294031-0064c457d1-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.042 DEBUG:boto:path=/hiiqos-46/obj2 2023-07-29T00:05:37.043 DEBUG:boto:auth_path=/hiiqos-46/obj2 2023-07-29T00:05:37.043 DEBUG:boto:path=/hiiqos-46/obj2?versionId=null 2023-07-29T00:05:37.043 DEBUG:boto:auth_path=/hiiqos-46/obj2?versionId=null 2023-07-29T00:05:37.044 DEBUG:boto:Method: HEAD 2023-07-29T00:05:37.044 DEBUG:boto:Path: /hiiqos-46/obj2?versionId=null 2023-07-29T00:05:37.044 DEBUG:boto:Data: 2023-07-29T00:05:37.045 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.045 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:05:37.045 DEBUG:boto:Port: 8000 2023-07-29T00:05:37.046 DEBUG:boto:Params: {} 2023-07-29T00:05:37.046 DEBUG:boto:Token: None 2023-07-29T00:05:37.047 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/obj2?versionId=null 2023-07-29T00:05:37.047 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FeCIJqOV4WSyPJ+dEIMEdQ5Orvw= 2023-07-29T00:05:37.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FeCIJqOV4WSyPJ+dEIMEdQ5Orvw=', 'Content-Length': '0'} 2023-07-29T00:05:37.049 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000080b22bf0dfff1491-0064c457d1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-07-29T00:05:37.050 DEBUG:boto:path=/hiiqos-46/obj2 2023-07-29T00:05:37.050 DEBUG:boto:auth_path=/hiiqos-46/obj2 2023-07-29T00:05:37.051 DEBUG:boto:path=/hiiqos-46/obj2?versionId=null 2023-07-29T00:05:37.051 DEBUG:boto:auth_path=/hiiqos-46/obj2?versionId=null 2023-07-29T00:05:37.051 DEBUG:boto:Method: GET 2023-07-29T00:05:37.052 DEBUG:boto:Path: /hiiqos-46/obj2?versionId=null 2023-07-29T00:05:37.052 DEBUG:boto:Data: 2023-07-29T00:05:37.052 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.053 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:05:37.053 DEBUG:boto:Port: 8001 2023-07-29T00:05:37.054 DEBUG:boto:Params: {} 2023-07-29T00:05:37.054 DEBUG:boto:Token: None 2023-07-29T00:05:37.054 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/obj2?versionId=null 2023-07-29T00:05:37.055 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:M0/+5GsdwUbsj7Zad2lMT0Y69YI= 2023-07-29T00:05:37.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:M0/+5GsdwUbsj7Zad2lMT0Y69YI=', 'Content-Length': '0'} 2023-07-29T00:05:37.057 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000069f46406afd9896-0064c457d1-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.058 DEBUG:boto:path=/hiiqos-46/obj2 2023-07-29T00:05:37.059 DEBUG:boto:auth_path=/hiiqos-46/obj2 2023-07-29T00:05:37.060 DEBUG:boto:path=/hiiqos-46/obj2?versionId=null 2023-07-29T00:05:37.061 DEBUG:boto:auth_path=/hiiqos-46/obj2?versionId=null 2023-07-29T00:05:37.062 DEBUG:boto:Method: GET 2023-07-29T00:05:37.063 DEBUG:boto:Path: /hiiqos-46/obj2?versionId=null 2023-07-29T00:05:37.064 DEBUG:boto:Data: 2023-07-29T00:05:37.064 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.065 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:05:37.066 DEBUG:boto:Port: 8000 2023-07-29T00:05:37.066 DEBUG:boto:Params: {} 2023-07-29T00:05:37.067 DEBUG:boto:Token: None 2023-07-29T00:05:37.068 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/obj2?versionId=null 2023-07-29T00:05:37.069 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:M0/+5GsdwUbsj7Zad2lMT0Y69YI= 2023-07-29T00:05:37.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:M0/+5GsdwUbsj7Zad2lMT0Y69YI=', 'Content-Length': '0'} 2023-07-29T00:05:37.071 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000686c3f8b13d03288-0064c457d1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.072 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:05:37.072 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:05:37.073 DEBUG:boto:path=/hiiqos-46/?prefix=obj2 2023-07-29T00:05:37.073 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj2 2023-07-29T00:05:37.073 DEBUG:boto:Method: GET 2023-07-29T00:05:37.074 DEBUG:boto:Path: /hiiqos-46/?prefix=obj2 2023-07-29T00:05:37.074 DEBUG:boto:Data: 2023-07-29T00:05:37.074 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.075 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:05:37.075 DEBUG:boto:Port: 8001 2023-07-29T00:05:37.075 DEBUG:boto:Params: {} 2023-07-29T00:05:37.076 DEBUG:boto:Token: None 2023-07-29T00:05:37.076 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/ 2023-07-29T00:05:37.077 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IfseN/msjvmwQlULCXnANA9B94U= 2023-07-29T00:05:37.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IfseN/msjvmwQlULCXnANA9B94U=', 'Content-Length': '0'} 2023-07-29T00:05:37.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f67505491fa3070-0064c457d1-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.120 DEBUG:boto:b'hiiqos-46obj21000falseobj22023-07-29T00:05:16.033Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:05:37.121 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:05:37.122 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:05:37.122 DEBUG:boto:path=/hiiqos-46/?prefix=obj2 2023-07-29T00:05:37.123 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj2 2023-07-29T00:05:37.123 DEBUG:boto:Method: GET 2023-07-29T00:05:37.123 DEBUG:boto:Path: /hiiqos-46/?prefix=obj2 2023-07-29T00:05:37.124 DEBUG:boto:Data: 2023-07-29T00:05:37.124 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.125 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:05:37.125 DEBUG:boto:Port: 8000 2023-07-29T00:05:37.125 DEBUG:boto:Params: {} 2023-07-29T00:05:37.126 DEBUG:boto:Token: None 2023-07-29T00:05:37.126 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/ 2023-07-29T00:05:37.127 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IfseN/msjvmwQlULCXnANA9B94U= 2023-07-29T00:05:37.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IfseN/msjvmwQlULCXnANA9B94U=', 'Content-Length': '0'} 2023-07-29T00:05:37.129 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4c996096a0f01e0-0064c457d1-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.130 DEBUG:boto:b'hiiqos-46obj21000falseobj22023-07-29T00:05:16.033Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:05:37.131 DEBUG:boto:path=/hiiqos-46/obj2 2023-07-29T00:05:37.131 DEBUG:boto:auth_path=/hiiqos-46/obj2 2023-07-29T00:05:37.131 DEBUG:boto:Method: HEAD 2023-07-29T00:05:37.132 DEBUG:boto:Path: /hiiqos-46/obj2 2023-07-29T00:05:37.132 DEBUG:boto:Data: 2023-07-29T00:05:37.133 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.133 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:05:37.133 DEBUG:boto:Port: 8001 2023-07-29T00:05:37.134 DEBUG:boto:Params: {} 2023-07-29T00:05:37.134 DEBUG:boto:Token: None 2023-07-29T00:05:37.134 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/obj2 2023-07-29T00:05:37.135 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mxgUC4woKNa2G2izccwMc//6YJg= 2023-07-29T00:05:37.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mxgUC4woKNa2G2izccwMc//6YJg=', 'Content-Length': '0'} 2023-07-29T00:05:37.137 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a62bb9b568ae469a-0064c457d1-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.138 DEBUG:boto:path=/hiiqos-46/obj2 2023-07-29T00:05:37.138 DEBUG:boto:auth_path=/hiiqos-46/obj2 2023-07-29T00:05:37.138 DEBUG:boto:Method: HEAD 2023-07-29T00:05:37.139 DEBUG:boto:Path: /hiiqos-46/obj2 2023-07-29T00:05:37.139 DEBUG:boto:Data: 2023-07-29T00:05:37.139 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.140 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:05:37.140 DEBUG:boto:Port: 8000 2023-07-29T00:05:37.141 DEBUG:boto:Params: {} 2023-07-29T00:05:37.141 DEBUG:boto:Token: None 2023-07-29T00:05:37.141 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/obj2 2023-07-29T00:05:37.142 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mxgUC4woKNa2G2izccwMc//6YJg= 2023-07-29T00:05:37.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mxgUC4woKNa2G2izccwMc//6YJg=', 'Content-Length': '0'} 2023-07-29T00:05:37.144 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a5ea95a680662d01-0064c457d1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-07-29T00:05:37.145 DEBUG:boto:path=/hiiqos-46/obj2 2023-07-29T00:05:37.145 DEBUG:boto:auth_path=/hiiqos-46/obj2 2023-07-29T00:05:37.145 DEBUG:boto:Method: GET 2023-07-29T00:05:37.146 DEBUG:boto:Path: /hiiqos-46/obj2 2023-07-29T00:05:37.146 DEBUG:boto:Data: 2023-07-29T00:05:37.147 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.147 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:05:37.147 DEBUG:boto:Port: 8001 2023-07-29T00:05:37.148 DEBUG:boto:Params: {} 2023-07-29T00:05:37.148 DEBUG:boto:Token: None 2023-07-29T00:05:37.148 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/obj2 2023-07-29T00:05:37.149 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dOcWAqq4dlT3zoqElB3PP7jt+EI= 2023-07-29T00:05:37.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dOcWAqq4dlT3zoqElB3PP7jt+EI=', 'Content-Length': '0'} 2023-07-29T00:05:37.151 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000042eae4f78b122147-0064c457d1-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.152 DEBUG:boto:path=/hiiqos-46/obj2 2023-07-29T00:05:37.152 DEBUG:boto:auth_path=/hiiqos-46/obj2 2023-07-29T00:05:37.152 DEBUG:boto:Method: GET 2023-07-29T00:05:37.153 DEBUG:boto:Path: /hiiqos-46/obj2 2023-07-29T00:05:37.153 DEBUG:boto:Data: 2023-07-29T00:05:37.154 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.154 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:05:37.154 DEBUG:boto:Port: 8000 2023-07-29T00:05:37.155 DEBUG:boto:Params: {} 2023-07-29T00:05:37.155 DEBUG:boto:Token: None 2023-07-29T00:05:37.155 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/obj2 2023-07-29T00:05:37.156 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dOcWAqq4dlT3zoqElB3PP7jt+EI= 2023-07-29T00:05:37.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dOcWAqq4dlT3zoqElB3PP7jt+EI=', 'Content-Length': '0'} 2023-07-29T00:05:37.158 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000611bb6d6409407b5-0064c457d1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.200 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:05:37.201 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:05:37.201 DEBUG:boto:path=/hiiqos-46/?prefix=obj2 2023-07-29T00:05:37.202 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj2 2023-07-29T00:05:37.202 DEBUG:boto:Method: GET 2023-07-29T00:05:37.203 DEBUG:boto:Path: /hiiqos-46/?prefix=obj2 2023-07-29T00:05:37.203 DEBUG:boto:Data: 2023-07-29T00:05:37.204 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.204 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:05:37.204 DEBUG:boto:Port: 8001 2023-07-29T00:05:37.205 DEBUG:boto:Params: {} 2023-07-29T00:05:37.205 DEBUG:boto:Token: None 2023-07-29T00:05:37.206 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/ 2023-07-29T00:05:37.206 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IfseN/msjvmwQlULCXnANA9B94U= 2023-07-29T00:05:37.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IfseN/msjvmwQlULCXnANA9B94U=', 'Content-Length': '0'} 2023-07-29T00:05:37.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d538a97fa129a23-0064c457d1-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.209 DEBUG:boto:b'hiiqos-46obj21000falseobj22023-07-29T00:05:16.033Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:05:37.210 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:05:37.210 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:05:37.211 DEBUG:boto:path=/hiiqos-46/?prefix=obj2 2023-07-29T00:05:37.211 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj2 2023-07-29T00:05:37.211 DEBUG:boto:Method: GET 2023-07-29T00:05:37.212 DEBUG:boto:Path: /hiiqos-46/?prefix=obj2 2023-07-29T00:05:37.212 DEBUG:boto:Data: 2023-07-29T00:05:37.213 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.214 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:05:37.214 DEBUG:boto:Port: 8000 2023-07-29T00:05:37.215 DEBUG:boto:Params: {} 2023-07-29T00:05:37.216 DEBUG:boto:Token: None 2023-07-29T00:05:37.216 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/ 2023-07-29T00:05:37.217 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IfseN/msjvmwQlULCXnANA9B94U= 2023-07-29T00:05:37.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IfseN/msjvmwQlULCXnANA9B94U=', 'Content-Length': '0'} 2023-07-29T00:05:37.219 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000496e9e4a54faf14e-0064c457d1-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.220 DEBUG:boto:b'hiiqos-46obj21000falseobj22023-07-29T00:05:16.033Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:05:37.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-07-29T00:05:37.221 DEBUG:boto:path=/hiiqos-46/obj3 2023-07-29T00:05:37.221 DEBUG:boto:auth_path=/hiiqos-46/obj3 2023-07-29T00:05:37.222 DEBUG:boto:path=/hiiqos-46/obj3?versionId=null 2023-07-29T00:05:37.223 DEBUG:boto:auth_path=/hiiqos-46/obj3?versionId=null 2023-07-29T00:05:37.223 DEBUG:boto:Method: GET 2023-07-29T00:05:37.224 DEBUG:boto:Path: /hiiqos-46/obj3?versionId=null 2023-07-29T00:05:37.224 DEBUG:boto:Data: 2023-07-29T00:05:37.224 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.225 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:05:37.225 DEBUG:boto:Port: 8001 2023-07-29T00:05:37.225 DEBUG:boto:Params: {} 2023-07-29T00:05:37.226 DEBUG:boto:Token: None 2023-07-29T00:05:37.227 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/obj3?versionId=null 2023-07-29T00:05:37.227 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lXJUFPqUxfeiUwy9nmTeUfcexVY= 2023-07-29T00:05:37.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lXJUFPqUxfeiUwy9nmTeUfcexVY=', 'Content-Length': '0'} 2023-07-29T00:05:37.229 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ac21b6bb82438273-0064c457d1-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.272 DEBUG:boto:path=/hiiqos-46/obj3 2023-07-29T00:05:37.273 DEBUG:boto:auth_path=/hiiqos-46/obj3 2023-07-29T00:05:37.273 DEBUG:boto:path=/hiiqos-46/obj3?versionId=null 2023-07-29T00:05:37.273 DEBUG:boto:auth_path=/hiiqos-46/obj3?versionId=null 2023-07-29T00:05:37.274 DEBUG:boto:Method: GET 2023-07-29T00:05:37.274 DEBUG:boto:Path: /hiiqos-46/obj3?versionId=null 2023-07-29T00:05:37.275 DEBUG:boto:Data: 2023-07-29T00:05:37.275 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.276 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:05:37.276 DEBUG:boto:Port: 8000 2023-07-29T00:05:37.277 DEBUG:boto:Params: {} 2023-07-29T00:05:37.277 DEBUG:boto:Token: None 2023-07-29T00:05:37.278 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/obj3?versionId=null 2023-07-29T00:05:37.278 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lXJUFPqUxfeiUwy9nmTeUfcexVY= 2023-07-29T00:05:37.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lXJUFPqUxfeiUwy9nmTeUfcexVY=', 'Content-Length': '0'} 2023-07-29T00:05:37.280 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000025802e3e6eb80354-0064c457d1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.281 DEBUG:boto:path=/hiiqos-46/obj3 2023-07-29T00:05:37.281 DEBUG:boto:auth_path=/hiiqos-46/obj3 2023-07-29T00:05:37.282 DEBUG:boto:path=/hiiqos-46/obj3?versionId=null 2023-07-29T00:05:37.282 DEBUG:boto:auth_path=/hiiqos-46/obj3?versionId=null 2023-07-29T00:05:37.282 DEBUG:boto:Method: HEAD 2023-07-29T00:05:37.283 DEBUG:boto:Path: /hiiqos-46/obj3?versionId=null 2023-07-29T00:05:37.283 DEBUG:boto:Data: 2023-07-29T00:05:37.284 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.284 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:05:37.284 DEBUG:boto:Port: 8001 2023-07-29T00:05:37.285 DEBUG:boto:Params: {} 2023-07-29T00:05:37.285 DEBUG:boto:Token: None 2023-07-29T00:05:37.285 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/obj3?versionId=null 2023-07-29T00:05:37.286 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YP6U8BRiyoj8G5MJOwaTte9M3mw= 2023-07-29T00:05:37.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YP6U8BRiyoj8G5MJOwaTte9M3mw=', 'Content-Length': '0'} 2023-07-29T00:05:37.288 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002935083a8c8afdf9-0064c457d1-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.289 DEBUG:boto:path=/hiiqos-46/obj3 2023-07-29T00:05:37.289 DEBUG:boto:auth_path=/hiiqos-46/obj3 2023-07-29T00:05:37.290 DEBUG:boto:path=/hiiqos-46/obj3?versionId=null 2023-07-29T00:05:37.290 DEBUG:boto:auth_path=/hiiqos-46/obj3?versionId=null 2023-07-29T00:05:37.291 DEBUG:boto:Method: HEAD 2023-07-29T00:05:37.291 DEBUG:boto:Path: /hiiqos-46/obj3?versionId=null 2023-07-29T00:05:37.291 DEBUG:boto:Data: 2023-07-29T00:05:37.292 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.292 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:05:37.292 DEBUG:boto:Port: 8000 2023-07-29T00:05:37.293 DEBUG:boto:Params: {} 2023-07-29T00:05:37.293 DEBUG:boto:Token: None 2023-07-29T00:05:37.294 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/obj3?versionId=null 2023-07-29T00:05:37.294 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YP6U8BRiyoj8G5MJOwaTte9M3mw= 2023-07-29T00:05:37.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YP6U8BRiyoj8G5MJOwaTte9M3mw=', 'Content-Length': '0'} 2023-07-29T00:05:37.296 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000239b81a35e4a89ef-0064c457d1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-07-29T00:05:37.297 DEBUG:boto:path=/hiiqos-46/obj3 2023-07-29T00:05:37.298 DEBUG:boto:auth_path=/hiiqos-46/obj3 2023-07-29T00:05:37.298 DEBUG:boto:path=/hiiqos-46/obj3?versionId=null 2023-07-29T00:05:37.298 DEBUG:boto:auth_path=/hiiqos-46/obj3?versionId=null 2023-07-29T00:05:37.299 DEBUG:boto:Method: GET 2023-07-29T00:05:37.299 DEBUG:boto:Path: /hiiqos-46/obj3?versionId=null 2023-07-29T00:05:37.299 DEBUG:boto:Data: 2023-07-29T00:05:37.300 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.300 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:05:37.301 DEBUG:boto:Port: 8001 2023-07-29T00:05:37.301 DEBUG:boto:Params: {} 2023-07-29T00:05:37.301 DEBUG:boto:Token: None 2023-07-29T00:05:37.302 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/obj3?versionId=null 2023-07-29T00:05:37.302 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lXJUFPqUxfeiUwy9nmTeUfcexVY= 2023-07-29T00:05:37.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lXJUFPqUxfeiUwy9nmTeUfcexVY=', 'Content-Length': '0'} 2023-07-29T00:05:37.304 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001417982e61b21f33-0064c457d1-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.305 DEBUG:boto:path=/hiiqos-46/obj3 2023-07-29T00:05:37.306 DEBUG:boto:auth_path=/hiiqos-46/obj3 2023-07-29T00:05:37.306 DEBUG:boto:path=/hiiqos-46/obj3?versionId=null 2023-07-29T00:05:37.306 DEBUG:boto:auth_path=/hiiqos-46/obj3?versionId=null 2023-07-29T00:05:37.307 DEBUG:boto:Method: GET 2023-07-29T00:05:37.307 DEBUG:boto:Path: /hiiqos-46/obj3?versionId=null 2023-07-29T00:05:37.308 DEBUG:boto:Data: 2023-07-29T00:05:37.308 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.309 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:05:37.309 DEBUG:boto:Port: 8000 2023-07-29T00:05:37.309 DEBUG:boto:Params: {} 2023-07-29T00:05:37.310 DEBUG:boto:Token: None 2023-07-29T00:05:37.310 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/obj3?versionId=null 2023-07-29T00:05:37.311 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lXJUFPqUxfeiUwy9nmTeUfcexVY= 2023-07-29T00:05:37.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lXJUFPqUxfeiUwy9nmTeUfcexVY=', 'Content-Length': '0'} 2023-07-29T00:05:37.313 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002da2a2706a233820-0064c457d1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.314 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:05:37.314 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:05:37.314 DEBUG:boto:path=/hiiqos-46/?prefix=obj3 2023-07-29T00:05:37.315 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj3 2023-07-29T00:05:37.315 DEBUG:boto:Method: GET 2023-07-29T00:05:37.315 DEBUG:boto:Path: /hiiqos-46/?prefix=obj3 2023-07-29T00:05:37.316 DEBUG:boto:Data: 2023-07-29T00:05:37.316 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.317 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:05:37.317 DEBUG:boto:Port: 8001 2023-07-29T00:05:37.317 DEBUG:boto:Params: {} 2023-07-29T00:05:37.318 DEBUG:boto:Token: None 2023-07-29T00:05:37.318 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/ 2023-07-29T00:05:37.319 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IfseN/msjvmwQlULCXnANA9B94U= 2023-07-29T00:05:37.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IfseN/msjvmwQlULCXnANA9B94U=', 'Content-Length': '0'} 2023-07-29T00:05:37.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd4bc5a07e0b1140-0064c457d1-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.364 DEBUG:boto:b'hiiqos-46obj31000falseobj32023-07-29T00:05:16.052Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:05:37.366 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:05:37.367 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:05:37.368 DEBUG:boto:path=/hiiqos-46/?prefix=obj3 2023-07-29T00:05:37.369 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj3 2023-07-29T00:05:37.369 DEBUG:boto:Method: GET 2023-07-29T00:05:37.370 DEBUG:boto:Path: /hiiqos-46/?prefix=obj3 2023-07-29T00:05:37.370 DEBUG:boto:Data: 2023-07-29T00:05:37.370 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.371 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:05:37.371 DEBUG:boto:Port: 8000 2023-07-29T00:05:37.371 DEBUG:boto:Params: {} 2023-07-29T00:05:37.372 DEBUG:boto:Token: None 2023-07-29T00:05:37.372 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/ 2023-07-29T00:05:37.373 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IfseN/msjvmwQlULCXnANA9B94U= 2023-07-29T00:05:37.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IfseN/msjvmwQlULCXnANA9B94U=', 'Content-Length': '0'} 2023-07-29T00:05:37.376 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e067a475e723f34-0064c457d1-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.376 DEBUG:boto:b'hiiqos-46obj31000falseobj32023-07-29T00:05:16.052Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:05:37.377 DEBUG:boto:path=/hiiqos-46/obj3 2023-07-29T00:05:37.378 DEBUG:boto:auth_path=/hiiqos-46/obj3 2023-07-29T00:05:37.378 DEBUG:boto:Method: HEAD 2023-07-29T00:05:37.378 DEBUG:boto:Path: /hiiqos-46/obj3 2023-07-29T00:05:37.379 DEBUG:boto:Data: 2023-07-29T00:05:37.379 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.380 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:05:37.380 DEBUG:boto:Port: 8001 2023-07-29T00:05:37.380 DEBUG:boto:Params: {} 2023-07-29T00:05:37.381 DEBUG:boto:Token: None 2023-07-29T00:05:37.381 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/obj3 2023-07-29T00:05:37.382 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ODvKVMKokVIUFkJc/YHJy7O98tM= 2023-07-29T00:05:37.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ODvKVMKokVIUFkJc/YHJy7O98tM=', 'Content-Length': '0'} 2023-07-29T00:05:37.384 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f4b7dfc954ae99e6-0064c457d1-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.385 DEBUG:boto:path=/hiiqos-46/obj3 2023-07-29T00:05:37.385 DEBUG:boto:auth_path=/hiiqos-46/obj3 2023-07-29T00:05:37.386 DEBUG:boto:Method: HEAD 2023-07-29T00:05:37.386 DEBUG:boto:Path: /hiiqos-46/obj3 2023-07-29T00:05:37.386 DEBUG:boto:Data: 2023-07-29T00:05:37.387 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.387 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:05:37.387 DEBUG:boto:Port: 8000 2023-07-29T00:05:37.388 DEBUG:boto:Params: {} 2023-07-29T00:05:37.388 DEBUG:boto:Token: None 2023-07-29T00:05:37.389 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/obj3 2023-07-29T00:05:37.389 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ODvKVMKokVIUFkJc/YHJy7O98tM= 2023-07-29T00:05:37.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ODvKVMKokVIUFkJc/YHJy7O98tM=', 'Content-Length': '0'} 2023-07-29T00:05:37.391 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000023d9d40954788dab-0064c457d1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-07-29T00:05:37.393 DEBUG:boto:path=/hiiqos-46/obj3 2023-07-29T00:05:37.394 DEBUG:boto:auth_path=/hiiqos-46/obj3 2023-07-29T00:05:37.395 DEBUG:boto:Method: GET 2023-07-29T00:05:37.396 DEBUG:boto:Path: /hiiqos-46/obj3 2023-07-29T00:05:37.397 DEBUG:boto:Data: 2023-07-29T00:05:37.397 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.398 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:05:37.398 DEBUG:boto:Port: 8001 2023-07-29T00:05:37.398 DEBUG:boto:Params: {} 2023-07-29T00:05:37.399 DEBUG:boto:Token: None 2023-07-29T00:05:37.399 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/obj3 2023-07-29T00:05:37.400 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:D4eE3ZOElDtZdGETzpkauYsyqAY= 2023-07-29T00:05:37.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:D4eE3ZOElDtZdGETzpkauYsyqAY=', 'Content-Length': '0'} 2023-07-29T00:05:37.402 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001fdeb587ad4573a7-0064c457d1-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.403 DEBUG:boto:path=/hiiqos-46/obj3 2023-07-29T00:05:37.403 DEBUG:boto:auth_path=/hiiqos-46/obj3 2023-07-29T00:05:37.404 DEBUG:boto:Method: GET 2023-07-29T00:05:37.404 DEBUG:boto:Path: /hiiqos-46/obj3 2023-07-29T00:05:37.404 DEBUG:boto:Data: 2023-07-29T00:05:37.405 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.406 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:05:37.407 DEBUG:boto:Port: 8000 2023-07-29T00:05:37.407 DEBUG:boto:Params: {} 2023-07-29T00:05:37.408 DEBUG:boto:Token: None 2023-07-29T00:05:37.408 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/obj3 2023-07-29T00:05:37.409 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:D4eE3ZOElDtZdGETzpkauYsyqAY= 2023-07-29T00:05:37.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:D4eE3ZOElDtZdGETzpkauYsyqAY=', 'Content-Length': '0'} 2023-07-29T00:05:37.411 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b62c8825728d5364-0064c457d1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.411 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:05:37.412 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:05:37.412 DEBUG:boto:path=/hiiqos-46/?prefix=obj3 2023-07-29T00:05:37.413 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj3 2023-07-29T00:05:37.413 DEBUG:boto:Method: GET 2023-07-29T00:05:37.413 DEBUG:boto:Path: /hiiqos-46/?prefix=obj3 2023-07-29T00:05:37.414 DEBUG:boto:Data: 2023-07-29T00:05:37.414 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.414 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:05:37.415 DEBUG:boto:Port: 8001 2023-07-29T00:05:37.415 DEBUG:boto:Params: {} 2023-07-29T00:05:37.415 DEBUG:boto:Token: None 2023-07-29T00:05:37.416 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/ 2023-07-29T00:05:37.416 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IfseN/msjvmwQlULCXnANA9B94U= 2023-07-29T00:05:37.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IfseN/msjvmwQlULCXnANA9B94U=', 'Content-Length': '0'} 2023-07-29T00:05:37.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005aff9bb59a92888c-0064c457d1-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.419 DEBUG:boto:b'hiiqos-46obj31000falseobj32023-07-29T00:05:16.052Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:05:37.420 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:05:37.421 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:05:37.421 DEBUG:boto:path=/hiiqos-46/?prefix=obj3 2023-07-29T00:05:37.422 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj3 2023-07-29T00:05:37.422 DEBUG:boto:Method: GET 2023-07-29T00:05:37.423 DEBUG:boto:Path: /hiiqos-46/?prefix=obj3 2023-07-29T00:05:37.424 DEBUG:boto:Data: 2023-07-29T00:05:37.424 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.424 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:05:37.425 DEBUG:boto:Port: 8000 2023-07-29T00:05:37.425 DEBUG:boto:Params: {} 2023-07-29T00:05:37.425 DEBUG:boto:Token: None 2023-07-29T00:05:37.426 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/ 2023-07-29T00:05:37.426 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IfseN/msjvmwQlULCXnANA9B94U= 2023-07-29T00:05:37.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IfseN/msjvmwQlULCXnANA9B94U=', 'Content-Length': '0'} 2023-07-29T00:05:37.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8a61bb3e034ef37-0064c457d1-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.430 DEBUG:boto:b'hiiqos-46obj31000falseobj32023-07-29T00:05:16.052Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:05:37.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-07-29T00:05:37.431 DEBUG:boto:path=/hiiqos-46/obj4 2023-07-29T00:05:37.431 DEBUG:boto:auth_path=/hiiqos-46/obj4 2023-07-29T00:05:37.431 DEBUG:boto:path=/hiiqos-46/obj4?versionId=null 2023-07-29T00:05:37.432 DEBUG:boto:auth_path=/hiiqos-46/obj4?versionId=null 2023-07-29T00:05:37.432 DEBUG:boto:Method: GET 2023-07-29T00:05:37.433 DEBUG:boto:Path: /hiiqos-46/obj4?versionId=null 2023-07-29T00:05:37.433 DEBUG:boto:Data: 2023-07-29T00:05:37.433 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.434 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:05:37.434 DEBUG:boto:Port: 8001 2023-07-29T00:05:37.434 DEBUG:boto:Params: {} 2023-07-29T00:05:37.435 DEBUG:boto:Token: None 2023-07-29T00:05:37.435 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/obj4?versionId=null 2023-07-29T00:05:37.436 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uzEy0SQkW5v7I7diULN5iln2pj4= 2023-07-29T00:05:37.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uzEy0SQkW5v7I7diULN5iln2pj4=', 'Content-Length': '0'} 2023-07-29T00:05:37.438 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e3e392dc53f2c37b-0064c457d1-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.438 DEBUG:boto:path=/hiiqos-46/obj4 2023-07-29T00:05:37.439 DEBUG:boto:auth_path=/hiiqos-46/obj4 2023-07-29T00:05:37.439 DEBUG:boto:path=/hiiqos-46/obj4?versionId=null 2023-07-29T00:05:37.440 DEBUG:boto:auth_path=/hiiqos-46/obj4?versionId=null 2023-07-29T00:05:37.440 DEBUG:boto:Method: GET 2023-07-29T00:05:37.441 DEBUG:boto:Path: /hiiqos-46/obj4?versionId=null 2023-07-29T00:05:37.441 DEBUG:boto:Data: 2023-07-29T00:05:37.441 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.442 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:05:37.442 DEBUG:boto:Port: 8000 2023-07-29T00:05:37.442 DEBUG:boto:Params: {} 2023-07-29T00:05:37.443 DEBUG:boto:Token: None 2023-07-29T00:05:37.443 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/obj4?versionId=null 2023-07-29T00:05:37.444 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uzEy0SQkW5v7I7diULN5iln2pj4= 2023-07-29T00:05:37.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uzEy0SQkW5v7I7diULN5iln2pj4=', 'Content-Length': '0'} 2023-07-29T00:05:37.446 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006b392eb0c16d86be-0064c457d1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.488 DEBUG:boto:path=/hiiqos-46/obj4 2023-07-29T00:05:37.489 DEBUG:boto:auth_path=/hiiqos-46/obj4 2023-07-29T00:05:37.489 DEBUG:boto:path=/hiiqos-46/obj4?versionId=null 2023-07-29T00:05:37.489 DEBUG:boto:auth_path=/hiiqos-46/obj4?versionId=null 2023-07-29T00:05:37.490 DEBUG:boto:Method: HEAD 2023-07-29T00:05:37.490 DEBUG:boto:Path: /hiiqos-46/obj4?versionId=null 2023-07-29T00:05:37.491 DEBUG:boto:Data: 2023-07-29T00:05:37.491 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.491 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:05:37.492 DEBUG:boto:Port: 8001 2023-07-29T00:05:37.492 DEBUG:boto:Params: {} 2023-07-29T00:05:37.493 DEBUG:boto:Token: None 2023-07-29T00:05:37.493 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/obj4?versionId=null 2023-07-29T00:05:37.493 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:APnxSDVATflIrjqz09WW5Z9z0fY= 2023-07-29T00:05:37.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:APnxSDVATflIrjqz09WW5Z9z0fY=', 'Content-Length': '0'} 2023-07-29T00:05:37.495 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bf2c051bc9599f60-0064c457d1-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.496 DEBUG:boto:path=/hiiqos-46/obj4 2023-07-29T00:05:37.497 DEBUG:boto:auth_path=/hiiqos-46/obj4 2023-07-29T00:05:37.497 DEBUG:boto:path=/hiiqos-46/obj4?versionId=null 2023-07-29T00:05:37.497 DEBUG:boto:auth_path=/hiiqos-46/obj4?versionId=null 2023-07-29T00:05:37.498 DEBUG:boto:Method: HEAD 2023-07-29T00:05:37.498 DEBUG:boto:Path: /hiiqos-46/obj4?versionId=null 2023-07-29T00:05:37.498 DEBUG:boto:Data: 2023-07-29T00:05:37.499 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.499 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:05:37.499 DEBUG:boto:Port: 8000 2023-07-29T00:05:37.500 DEBUG:boto:Params: {} 2023-07-29T00:05:37.500 DEBUG:boto:Token: None 2023-07-29T00:05:37.501 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/obj4?versionId=null 2023-07-29T00:05:37.501 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:APnxSDVATflIrjqz09WW5Z9z0fY= 2023-07-29T00:05:37.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:APnxSDVATflIrjqz09WW5Z9z0fY=', 'Content-Length': '0'} 2023-07-29T00:05:37.503 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001fc1f1dc6cd42176-0064c457d1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-07-29T00:05:37.504 DEBUG:boto:path=/hiiqos-46/obj4 2023-07-29T00:05:37.505 DEBUG:boto:auth_path=/hiiqos-46/obj4 2023-07-29T00:05:37.505 DEBUG:boto:path=/hiiqos-46/obj4?versionId=null 2023-07-29T00:05:37.505 DEBUG:boto:auth_path=/hiiqos-46/obj4?versionId=null 2023-07-29T00:05:37.506 DEBUG:boto:Method: GET 2023-07-29T00:05:37.506 DEBUG:boto:Path: /hiiqos-46/obj4?versionId=null 2023-07-29T00:05:37.506 DEBUG:boto:Data: 2023-07-29T00:05:37.507 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.507 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:05:37.507 DEBUG:boto:Port: 8001 2023-07-29T00:05:37.508 DEBUG:boto:Params: {} 2023-07-29T00:05:37.508 DEBUG:boto:Token: None 2023-07-29T00:05:37.509 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/obj4?versionId=null 2023-07-29T00:05:37.509 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uzEy0SQkW5v7I7diULN5iln2pj4= 2023-07-29T00:05:37.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uzEy0SQkW5v7I7diULN5iln2pj4=', 'Content-Length': '0'} 2023-07-29T00:05:37.511 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000cff4d6b666ab28e-0064c457d1-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.512 DEBUG:boto:path=/hiiqos-46/obj4 2023-07-29T00:05:37.513 DEBUG:boto:auth_path=/hiiqos-46/obj4 2023-07-29T00:05:37.513 DEBUG:boto:path=/hiiqos-46/obj4?versionId=null 2023-07-29T00:05:37.513 DEBUG:boto:auth_path=/hiiqos-46/obj4?versionId=null 2023-07-29T00:05:37.514 DEBUG:boto:Method: GET 2023-07-29T00:05:37.514 DEBUG:boto:Path: /hiiqos-46/obj4?versionId=null 2023-07-29T00:05:37.515 DEBUG:boto:Data: 2023-07-29T00:05:37.515 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.516 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:05:37.516 DEBUG:boto:Port: 8000 2023-07-29T00:05:37.516 DEBUG:boto:Params: {} 2023-07-29T00:05:37.517 DEBUG:boto:Token: None 2023-07-29T00:05:37.517 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/obj4?versionId=null 2023-07-29T00:05:37.518 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uzEy0SQkW5v7I7diULN5iln2pj4= 2023-07-29T00:05:37.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uzEy0SQkW5v7I7diULN5iln2pj4=', 'Content-Length': '0'} 2023-07-29T00:05:37.520 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008b82785d1750bc86-0064c457d1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.521 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:05:37.521 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:05:37.522 DEBUG:boto:path=/hiiqos-46/?prefix=obj4 2023-07-29T00:05:37.522 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj4 2023-07-29T00:05:37.522 DEBUG:boto:Method: GET 2023-07-29T00:05:37.523 DEBUG:boto:Path: /hiiqos-46/?prefix=obj4 2023-07-29T00:05:37.523 DEBUG:boto:Data: 2023-07-29T00:05:37.524 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.524 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:05:37.525 DEBUG:boto:Port: 8001 2023-07-29T00:05:37.525 DEBUG:boto:Params: {} 2023-07-29T00:05:37.525 DEBUG:boto:Token: None 2023-07-29T00:05:37.526 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/ 2023-07-29T00:05:37.526 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IfseN/msjvmwQlULCXnANA9B94U= 2023-07-29T00:05:37.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IfseN/msjvmwQlULCXnANA9B94U=', 'Content-Length': '0'} 2023-07-29T00:05:37.529 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbba483696d0d93e-0064c457d1-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.529 DEBUG:boto:b'hiiqos-46obj41000falseobj42023-07-29T00:05:16.112Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:05:37.530 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:05:37.530 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:05:37.531 DEBUG:boto:path=/hiiqos-46/?prefix=obj4 2023-07-29T00:05:37.531 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj4 2023-07-29T00:05:37.532 DEBUG:boto:Method: GET 2023-07-29T00:05:37.532 DEBUG:boto:Path: /hiiqos-46/?prefix=obj4 2023-07-29T00:05:37.533 DEBUG:boto:Data: 2023-07-29T00:05:37.533 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.533 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:05:37.534 DEBUG:boto:Port: 8000 2023-07-29T00:05:37.534 DEBUG:boto:Params: {} 2023-07-29T00:05:37.534 DEBUG:boto:Token: None 2023-07-29T00:05:37.535 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/ 2023-07-29T00:05:37.535 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IfseN/msjvmwQlULCXnANA9B94U= 2023-07-29T00:05:37.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IfseN/msjvmwQlULCXnANA9B94U=', 'Content-Length': '0'} 2023-07-29T00:05:37.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f1d2265e708923c-0064c457d1-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.538 DEBUG:boto:b'hiiqos-46obj41000falseobj42023-07-29T00:05:16.112Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:05:37.539 DEBUG:boto:path=/hiiqos-46/obj4 2023-07-29T00:05:37.540 DEBUG:boto:auth_path=/hiiqos-46/obj4 2023-07-29T00:05:37.540 DEBUG:boto:Method: HEAD 2023-07-29T00:05:37.540 DEBUG:boto:Path: /hiiqos-46/obj4 2023-07-29T00:05:37.541 DEBUG:boto:Data: 2023-07-29T00:05:37.541 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.541 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:05:37.542 DEBUG:boto:Port: 8001 2023-07-29T00:05:37.542 DEBUG:boto:Params: {} 2023-07-29T00:05:37.543 DEBUG:boto:Token: None 2023-07-29T00:05:37.544 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/obj4 2023-07-29T00:05:37.545 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4lCrRSYEiQH/ofFLP3ZTAC8xnqg= 2023-07-29T00:05:37.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4lCrRSYEiQH/ofFLP3ZTAC8xnqg=', 'Content-Length': '0'} 2023-07-29T00:05:37.548 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f62c96e1fa32de1-0064c457d1-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.549 DEBUG:boto:path=/hiiqos-46/obj4 2023-07-29T00:05:37.550 DEBUG:boto:auth_path=/hiiqos-46/obj4 2023-07-29T00:05:37.550 DEBUG:boto:Method: HEAD 2023-07-29T00:05:37.550 DEBUG:boto:Path: /hiiqos-46/obj4 2023-07-29T00:05:37.551 DEBUG:boto:Data: 2023-07-29T00:05:37.551 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.551 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:05:37.552 DEBUG:boto:Port: 8000 2023-07-29T00:05:37.552 DEBUG:boto:Params: {} 2023-07-29T00:05:37.552 DEBUG:boto:Token: None 2023-07-29T00:05:37.553 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/obj4 2023-07-29T00:05:37.553 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4lCrRSYEiQH/ofFLP3ZTAC8xnqg= 2023-07-29T00:05:37.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4lCrRSYEiQH/ofFLP3ZTAC8xnqg=', 'Content-Length': '0'} 2023-07-29T00:05:37.555 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd5f506ac3b7737d-0064c457d1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-07-29T00:05:37.556 DEBUG:boto:path=/hiiqos-46/obj4 2023-07-29T00:05:37.557 DEBUG:boto:auth_path=/hiiqos-46/obj4 2023-07-29T00:05:37.557 DEBUG:boto:Method: GET 2023-07-29T00:05:37.557 DEBUG:boto:Path: /hiiqos-46/obj4 2023-07-29T00:05:37.558 DEBUG:boto:Data: 2023-07-29T00:05:37.558 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.559 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:05:37.559 DEBUG:boto:Port: 8001 2023-07-29T00:05:37.559 DEBUG:boto:Params: {} 2023-07-29T00:05:37.560 DEBUG:boto:Token: None 2023-07-29T00:05:37.560 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/obj4 2023-07-29T00:05:37.561 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:W7PFrSsui5nbGZOm86HTg3+ZN1c= 2023-07-29T00:05:37.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:W7PFrSsui5nbGZOm86HTg3+ZN1c=', 'Content-Length': '0'} 2023-07-29T00:05:37.563 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f9d41ad4c10e92f7-0064c457d1-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.564 DEBUG:boto:path=/hiiqos-46/obj4 2023-07-29T00:05:37.564 DEBUG:boto:auth_path=/hiiqos-46/obj4 2023-07-29T00:05:37.565 DEBUG:boto:Method: GET 2023-07-29T00:05:37.565 DEBUG:boto:Path: /hiiqos-46/obj4 2023-07-29T00:05:37.565 DEBUG:boto:Data: 2023-07-29T00:05:37.566 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.566 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:05:37.566 DEBUG:boto:Port: 8000 2023-07-29T00:05:37.567 DEBUG:boto:Params: {} 2023-07-29T00:05:37.567 DEBUG:boto:Token: None 2023-07-29T00:05:37.567 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/obj4 2023-07-29T00:05:37.568 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:W7PFrSsui5nbGZOm86HTg3+ZN1c= 2023-07-29T00:05:37.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:W7PFrSsui5nbGZOm86HTg3+ZN1c=', 'Content-Length': '0'} 2023-07-29T00:05:37.570 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007fd89405c85ba71d-0064c457d1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.571 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:05:37.571 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:05:37.572 DEBUG:boto:path=/hiiqos-46/?prefix=obj4 2023-07-29T00:05:37.572 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj4 2023-07-29T00:05:37.573 DEBUG:boto:Method: GET 2023-07-29T00:05:37.573 DEBUG:boto:Path: /hiiqos-46/?prefix=obj4 2023-07-29T00:05:37.574 DEBUG:boto:Data: 2023-07-29T00:05:37.574 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.575 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:05:37.576 DEBUG:boto:Port: 8001 2023-07-29T00:05:37.576 DEBUG:boto:Params: {} 2023-07-29T00:05:37.577 DEBUG:boto:Token: None 2023-07-29T00:05:37.577 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/ 2023-07-29T00:05:37.578 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IfseN/msjvmwQlULCXnANA9B94U= 2023-07-29T00:05:37.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IfseN/msjvmwQlULCXnANA9B94U=', 'Content-Length': '0'} 2023-07-29T00:05:37.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000339f196a55ff9088-0064c457d1-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.581 DEBUG:boto:b'hiiqos-46obj41000falseobj42023-07-29T00:05:16.112Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:05:37.582 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:05:37.583 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:05:37.583 DEBUG:boto:path=/hiiqos-46/?prefix=obj4 2023-07-29T00:05:37.583 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj4 2023-07-29T00:05:37.584 DEBUG:boto:Method: GET 2023-07-29T00:05:37.584 DEBUG:boto:Path: /hiiqos-46/?prefix=obj4 2023-07-29T00:05:37.585 DEBUG:boto:Data: 2023-07-29T00:05:37.586 DEBUG:boto:Headers: {} 2023-07-29T00:05:37.586 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:05:37.587 DEBUG:boto:Port: 8000 2023-07-29T00:05:37.587 DEBUG:boto:Params: {} 2023-07-29T00:05:37.587 DEBUG:boto:Token: None 2023-07-29T00:05:37.588 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:05:37 GMT /hiiqos-46/ 2023-07-29T00:05:37.588 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IfseN/msjvmwQlULCXnANA9B94U= 2023-07-29T00:05:37.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:05:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IfseN/msjvmwQlULCXnANA9B94U=', 'Content-Length': '0'} 2023-07-29T00:05:37.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028605a367a04a3f7-0064c457d1-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:05:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:05:37.591 DEBUG:boto:b'hiiqos-46obj41000falseobj42023-07-29T00:05:16.112Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:05:37.592 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-46 zones={a2, a1} 2023-07-29T00:05:37.593 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-47 2023-07-29T00:05:37.593 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-47', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:05:37.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-47', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:05:37.594 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-47 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:05:37.647 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:05:37.647 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:05:37.678 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:05:37.656+0000 7fceec296880 1 waiting to reach incremental sync.. 2023-07-29T00:05:42.679 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:05:42.657+0000 7fceec296880 1 waiting to reach incremental sync.. 2023-07-29T00:05:47.680 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:05:47.657+0000 7fceec296880 1 waiting to reach incremental sync.. 2023-07-29T00:05:52.682 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:05:52.659+0000 7fceec296880 1 bucket sync caught up with source: 2023-07-29T00:05:52.683 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000001.24.6, , 00000000001.14.6, , 00000000001.19.6, , , 00000000001.22.6, , , ] 2023-07-29T00:05:52.683 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000001.24.6, , 00000000001.14.6, , 00000000001.19.6, , , 00000000001.22.6, , , ] 2023-07-29T00:05:52.683 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:05:52.659+0000 7fceec296880 0 bucket checkpoint complete 2023-07-29T00:05:52.699 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-47 2023-07-29T00:05:52.699 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-47', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:05:52.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-47', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:05:52.700 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-47 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:05:52.758 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:05:52.759 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:05:52.793 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:05:52.770+0000 7f6b7f726880 1 waiting to reach incremental sync.. 2023-07-29T00:05:57.794 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:05:57.771+0000 7f6b7f726880 1 waiting to reach incremental sync.. 2023-07-29T00:06:02.795 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:06:02.772+0000 7f6b7f726880 1 waiting to reach incremental sync.. 2023-07-29T00:06:07.795 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:06:07.772+0000 7f6b7f726880 1 waiting to reach incremental sync.. 2023-07-29T00:06:12.797 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:06:12.773+0000 7f6b7f726880 1 waiting to reach incremental sync.. 2023-07-29T00:06:17.797 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:06:17.773+0000 7f6b7f726880 1 waiting to reach incremental sync.. 2023-07-29T00:06:22.798 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:06:22.774+0000 7f6b7f726880 1 waiting to reach incremental sync.. 2023-07-29T00:06:27.799 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:06:27.775+0000 7f6b7f726880 1 waiting to reach incremental sync.. 2023-07-29T00:06:32.800 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:06:32.775+0000 7f6b7f726880 1 waiting to reach incremental sync.. 2023-07-29T00:06:37.801 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:06:37.776+0000 7f6b7f726880 1 waiting to reach incremental sync.. 2023-07-29T00:06:42.801 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:06:42.777+0000 7f6b7f726880 1 waiting to reach incremental sync.. 2023-07-29T00:06:47.802 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:06:47.777+0000 7f6b7f726880 1 waiting to reach incremental sync.. 2023-07-29T00:06:52.803 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:06:52.777+0000 7f6b7f726880 1 waiting to reach incremental sync.. 2023-07-29T00:06:57.805 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:06:57.779+0000 7f6b7f726880 1 bucket sync caught up with source: 2023-07-29T00:06:57.805 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000001.683.6, , 00000000001.1152.6, , 00000000001.994.6, , , 00000000001.1655.6, , , ] 2023-07-29T00:06:57.806 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000001.683.6, , 00000000001.1152.6, , 00000000001.994.6, , , 00000000001.1655.6, , , ] 2023-07-29T00:06:57.806 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:06:57.779+0000 7f6b7f726880 0 bucket checkpoint complete 2023-07-29T00:06:57.822 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-47 zones={a2, a1} 2023-07-29T00:06:57.823 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:06:57.824 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:06:57.825 DEBUG:boto:Method: HEAD 2023-07-29T00:06:57.826 DEBUG:boto:Path: /hiiqos-47/ 2023-07-29T00:06:57.827 DEBUG:boto:Data: 2023-07-29T00:06:57.828 DEBUG:boto:Headers: {} 2023-07-29T00:06:57.828 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:06:57.829 DEBUG:boto:Port: 8001 2023-07-29T00:06:57.830 DEBUG:boto:Params: {} 2023-07-29T00:06:57.831 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-29T00:06:57.832 DEBUG:boto:Token: None 2023-07-29T00:06:57.832 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:06:57 GMT /hiiqos-47/ 2023-07-29T00:06:57.833 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:COtlKVyyV4whpQJlwHaIgcGmw4Q= 2023-07-29T00:06:57.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:COtlKVyyV4whpQJlwHaIgcGmw4Q=', 'Content-Length': '0'} 2023-07-29T00:06:57.838 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a11385e18d29ca18-0064c45821-1663-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:06:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:57.839 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:06:57.839 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:06:57.839 DEBUG:boto:Method: HEAD 2023-07-29T00:06:57.840 DEBUG:boto:Path: /hiiqos-47/ 2023-07-29T00:06:57.840 DEBUG:boto:Data: 2023-07-29T00:06:57.841 DEBUG:boto:Headers: {} 2023-07-29T00:06:57.841 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:06:57.841 DEBUG:boto:Port: 8000 2023-07-29T00:06:57.842 DEBUG:boto:Params: {} 2023-07-29T00:06:57.842 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-29T00:06:57.842 DEBUG:boto:Token: None 2023-07-29T00:06:57.843 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:06:57 GMT /hiiqos-47/ 2023-07-29T00:06:57.843 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:COtlKVyyV4whpQJlwHaIgcGmw4Q= 2023-07-29T00:06:57.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:COtlKVyyV4whpQJlwHaIgcGmw4Q=', 'Content-Length': '0'} 2023-07-29T00:06:57.847 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000413a350c23f9c2f8-0064c45821-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:06:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:57.848 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-29T00:06:57.848 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:06:57.849 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:06:57.849 DEBUG:boto:path=/hiiqos-47/?versions&max-keys=999 2023-07-29T00:06:57.849 DEBUG:boto:auth_path=/hiiqos-47/?versions&max-keys=999 2023-07-29T00:06:57.850 DEBUG:boto:Method: GET 2023-07-29T00:06:57.850 DEBUG:boto:Path: /hiiqos-47/?versions&max-keys=999 2023-07-29T00:06:57.851 DEBUG:boto:Data: 2023-07-29T00:06:57.851 DEBUG:boto:Headers: {} 2023-07-29T00:06:57.851 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:06:57.852 DEBUG:boto:Port: 8001 2023-07-29T00:06:57.852 DEBUG:boto:Params: {} 2023-07-29T00:06:57.853 DEBUG:boto:Token: None 2023-07-29T00:06:57.853 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:57 GMT /hiiqos-47/?versions 2023-07-29T00:06:57.854 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1XNAQLEJW/7XPfJyQMVX1SnAUP8= 2023-07-29T00:06:57.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1XNAQLEJW/7XPfJyQMVX1SnAUP8=', 'Content-Length': '0'} 2023-07-29T00:06:57.856 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004515c08619cf863-0064c45821-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:06:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:57.857 DEBUG:boto:b'hiiqos-47999falseobj1nulltrue2023-07-29T00:05:16.136Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-07-29T00:05:16.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-07-29T00:05:16.174Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-07-29T00:05:16.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:06:57.858 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-07-29T00:06:57.858 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-07-29T00:06:57.858 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-07-29T00:06:57.859 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-07-29T00:06:57.859 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-29T00:06:57.859 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:06:57.860 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:06:57.860 DEBUG:boto:path=/hiiqos-47/?versions&max-keys=999 2023-07-29T00:06:57.861 DEBUG:boto:auth_path=/hiiqos-47/?versions&max-keys=999 2023-07-29T00:06:57.861 DEBUG:boto:Method: GET 2023-07-29T00:06:57.861 DEBUG:boto:Path: /hiiqos-47/?versions&max-keys=999 2023-07-29T00:06:57.862 DEBUG:boto:Data: 2023-07-29T00:06:57.862 DEBUG:boto:Headers: {} 2023-07-29T00:06:57.863 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:06:57.863 DEBUG:boto:Port: 8000 2023-07-29T00:06:57.863 DEBUG:boto:Params: {} 2023-07-29T00:06:57.864 DEBUG:boto:Token: None 2023-07-29T00:06:57.864 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:57 GMT /hiiqos-47/?versions 2023-07-29T00:06:57.865 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1XNAQLEJW/7XPfJyQMVX1SnAUP8= 2023-07-29T00:06:57.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1XNAQLEJW/7XPfJyQMVX1SnAUP8=', 'Content-Length': '0'} 2023-07-29T00:06:57.867 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000446e07ed15ddde4b-0064c45821-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:06:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:57.868 DEBUG:boto:b'hiiqos-47999falseobj1nulltrue2023-07-29T00:05:16.136Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-07-29T00:05:16.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-07-29T00:05:16.174Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-07-29T00:05:16.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:06:57.870 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-07-29T00:06:57.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-07-29T00:06:57.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-07-29T00:06:57.872 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-07-29T00:06:57.873 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:06:57.873 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:06:57.874 DEBUG:boto:path=/hiiqos-47/?versions&max-keys=999 2023-07-29T00:06:57.875 DEBUG:boto:auth_path=/hiiqos-47/?versions&max-keys=999 2023-07-29T00:06:57.875 DEBUG:boto:Method: GET 2023-07-29T00:06:57.876 DEBUG:boto:Path: /hiiqos-47/?versions&max-keys=999 2023-07-29T00:06:57.877 DEBUG:boto:Data: 2023-07-29T00:06:57.877 DEBUG:boto:Headers: {} 2023-07-29T00:06:57.877 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:06:57.878 DEBUG:boto:Port: 8001 2023-07-29T00:06:57.878 DEBUG:boto:Params: {} 2023-07-29T00:06:57.878 DEBUG:boto:Token: None 2023-07-29T00:06:57.879 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:57 GMT /hiiqos-47/?versions 2023-07-29T00:06:57.879 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1XNAQLEJW/7XPfJyQMVX1SnAUP8= 2023-07-29T00:06:57.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1XNAQLEJW/7XPfJyQMVX1SnAUP8=', 'Content-Length': '0'} 2023-07-29T00:06:57.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b818349099e7ed4a-0064c45821-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:06:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:57.882 DEBUG:boto:b'hiiqos-47999falseobj1nulltrue2023-07-29T00:05:16.136Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-07-29T00:05:16.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-07-29T00:05:16.174Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-07-29T00:05:16.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:06:57.883 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:06:57.884 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:06:57.884 DEBUG:boto:path=/hiiqos-47/?versions&max-keys=999 2023-07-29T00:06:57.884 DEBUG:boto:auth_path=/hiiqos-47/?versions&max-keys=999 2023-07-29T00:06:57.885 DEBUG:boto:Method: GET 2023-07-29T00:06:57.885 DEBUG:boto:Path: /hiiqos-47/?versions&max-keys=999 2023-07-29T00:06:57.886 DEBUG:boto:Data: 2023-07-29T00:06:57.886 DEBUG:boto:Headers: {} 2023-07-29T00:06:57.886 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:06:57.887 DEBUG:boto:Port: 8000 2023-07-29T00:06:57.887 DEBUG:boto:Params: {} 2023-07-29T00:06:57.887 DEBUG:boto:Token: None 2023-07-29T00:06:57.888 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:57 GMT /hiiqos-47/?versions 2023-07-29T00:06:57.888 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1XNAQLEJW/7XPfJyQMVX1SnAUP8= 2023-07-29T00:06:57.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1XNAQLEJW/7XPfJyQMVX1SnAUP8=', 'Content-Length': '0'} 2023-07-29T00:06:57.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ada5cf4d8b055435-0064c45821-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:06:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:57.892 DEBUG:boto:b'hiiqos-47999falseobj1nulltrue2023-07-29T00:05:16.136Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-07-29T00:05:16.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-07-29T00:05:16.174Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-07-29T00:05:16.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:06:57.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-07-29T00:06:57.893 DEBUG:boto:path=/hiiqos-47/obj1 2023-07-29T00:06:57.894 DEBUG:boto:auth_path=/hiiqos-47/obj1 2023-07-29T00:06:57.894 DEBUG:boto:path=/hiiqos-47/obj1?versionId=null 2023-07-29T00:06:57.894 DEBUG:boto:auth_path=/hiiqos-47/obj1?versionId=null 2023-07-29T00:06:57.895 DEBUG:boto:Method: GET 2023-07-29T00:06:57.895 DEBUG:boto:Path: /hiiqos-47/obj1?versionId=null 2023-07-29T00:06:57.895 DEBUG:boto:Data: 2023-07-29T00:06:57.896 DEBUG:boto:Headers: {} 2023-07-29T00:06:57.896 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:06:57.897 DEBUG:boto:Port: 8001 2023-07-29T00:06:57.897 DEBUG:boto:Params: {} 2023-07-29T00:06:57.897 DEBUG:boto:Token: None 2023-07-29T00:06:57.898 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:57 GMT /hiiqos-47/obj1?versionId=null 2023-07-29T00:06:57.898 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KCCJpduEgl30EEMBGRVHoo98gGY= 2023-07-29T00:06:57.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KCCJpduEgl30EEMBGRVHoo98gGY=', 'Content-Length': '0'} 2023-07-29T00:06:57.901 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007bd7bf2d333352ed-0064c45821-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:06:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:57.902 DEBUG:boto:path=/hiiqos-47/obj1 2023-07-29T00:06:57.903 DEBUG:boto:auth_path=/hiiqos-47/obj1 2023-07-29T00:06:57.904 DEBUG:boto:path=/hiiqos-47/obj1?versionId=null 2023-07-29T00:06:57.904 DEBUG:boto:auth_path=/hiiqos-47/obj1?versionId=null 2023-07-29T00:06:57.905 DEBUG:boto:Method: GET 2023-07-29T00:06:57.906 DEBUG:boto:Path: /hiiqos-47/obj1?versionId=null 2023-07-29T00:06:57.907 DEBUG:boto:Data: 2023-07-29T00:06:57.908 DEBUG:boto:Headers: {} 2023-07-29T00:06:57.908 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:06:57.909 DEBUG:boto:Port: 8000 2023-07-29T00:06:57.909 DEBUG:boto:Params: {} 2023-07-29T00:06:57.910 DEBUG:boto:Token: None 2023-07-29T00:06:57.910 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:57 GMT /hiiqos-47/obj1?versionId=null 2023-07-29T00:06:57.911 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KCCJpduEgl30EEMBGRVHoo98gGY= 2023-07-29T00:06:57.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KCCJpduEgl30EEMBGRVHoo98gGY=', 'Content-Length': '0'} 2023-07-29T00:06:57.913 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e87882a2ddd89f0-0064c45821-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:06:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:57.914 DEBUG:boto:path=/hiiqos-47/obj1 2023-07-29T00:06:57.914 DEBUG:boto:auth_path=/hiiqos-47/obj1 2023-07-29T00:06:57.914 DEBUG:boto:path=/hiiqos-47/obj1?versionId=null 2023-07-29T00:06:57.915 DEBUG:boto:auth_path=/hiiqos-47/obj1?versionId=null 2023-07-29T00:06:57.915 DEBUG:boto:Method: HEAD 2023-07-29T00:06:57.915 DEBUG:boto:Path: /hiiqos-47/obj1?versionId=null 2023-07-29T00:06:57.916 DEBUG:boto:Data: 2023-07-29T00:06:57.916 DEBUG:boto:Headers: {} 2023-07-29T00:06:57.917 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:06:57.917 DEBUG:boto:Port: 8001 2023-07-29T00:06:57.917 DEBUG:boto:Params: {} 2023-07-29T00:06:57.918 DEBUG:boto:Token: None 2023-07-29T00:06:57.918 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:06:57 GMT /hiiqos-47/obj1?versionId=null 2023-07-29T00:06:57.919 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:q8xMOMTF35MdxU5YqktAQZOaw5g= 2023-07-29T00:06:57.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:q8xMOMTF35MdxU5YqktAQZOaw5g=', 'Content-Length': '0'} 2023-07-29T00:06:57.921 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000886923153b5a3a88-0064c45821-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:06:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:57.921 DEBUG:boto:path=/hiiqos-47/obj1 2023-07-29T00:06:57.922 DEBUG:boto:auth_path=/hiiqos-47/obj1 2023-07-29T00:06:57.922 DEBUG:boto:path=/hiiqos-47/obj1?versionId=null 2023-07-29T00:06:57.922 DEBUG:boto:auth_path=/hiiqos-47/obj1?versionId=null 2023-07-29T00:06:57.923 DEBUG:boto:Method: HEAD 2023-07-29T00:06:57.923 DEBUG:boto:Path: /hiiqos-47/obj1?versionId=null 2023-07-29T00:06:57.924 DEBUG:boto:Data: 2023-07-29T00:06:57.924 DEBUG:boto:Headers: {} 2023-07-29T00:06:57.924 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:06:57.925 DEBUG:boto:Port: 8000 2023-07-29T00:06:57.925 DEBUG:boto:Params: {} 2023-07-29T00:06:57.926 DEBUG:boto:Token: None 2023-07-29T00:06:57.926 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:06:57 GMT /hiiqos-47/obj1?versionId=null 2023-07-29T00:06:57.926 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:q8xMOMTF35MdxU5YqktAQZOaw5g= 2023-07-29T00:06:57.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:q8xMOMTF35MdxU5YqktAQZOaw5g=', 'Content-Length': '0'} 2023-07-29T00:06:57.929 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f96f96c4348fdb1-0064c45821-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:06:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:57.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-07-29T00:06:57.930 DEBUG:boto:path=/hiiqos-47/obj1 2023-07-29T00:06:57.931 DEBUG:boto:auth_path=/hiiqos-47/obj1 2023-07-29T00:06:57.931 DEBUG:boto:path=/hiiqos-47/obj1?versionId=null 2023-07-29T00:06:57.932 DEBUG:boto:auth_path=/hiiqos-47/obj1?versionId=null 2023-07-29T00:06:57.932 DEBUG:boto:Method: GET 2023-07-29T00:06:57.932 DEBUG:boto:Path: /hiiqos-47/obj1?versionId=null 2023-07-29T00:06:57.933 DEBUG:boto:Data: 2023-07-29T00:06:57.934 DEBUG:boto:Headers: {} 2023-07-29T00:06:57.934 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:06:57.935 DEBUG:boto:Port: 8001 2023-07-29T00:06:57.935 DEBUG:boto:Params: {} 2023-07-29T00:06:57.935 DEBUG:boto:Token: None 2023-07-29T00:06:57.936 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:57 GMT /hiiqos-47/obj1?versionId=null 2023-07-29T00:06:57.936 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KCCJpduEgl30EEMBGRVHoo98gGY= 2023-07-29T00:06:57.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KCCJpduEgl30EEMBGRVHoo98gGY=', 'Content-Length': '0'} 2023-07-29T00:06:57.939 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004366d8cd9cad1fa2-0064c45821-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:06:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:57.980 DEBUG:boto:path=/hiiqos-47/obj1 2023-07-29T00:06:57.981 DEBUG:boto:auth_path=/hiiqos-47/obj1 2023-07-29T00:06:57.981 DEBUG:boto:path=/hiiqos-47/obj1?versionId=null 2023-07-29T00:06:57.982 DEBUG:boto:auth_path=/hiiqos-47/obj1?versionId=null 2023-07-29T00:06:57.982 DEBUG:boto:Method: GET 2023-07-29T00:06:57.983 DEBUG:boto:Path: /hiiqos-47/obj1?versionId=null 2023-07-29T00:06:57.983 DEBUG:boto:Data: 2023-07-29T00:06:57.984 DEBUG:boto:Headers: {} 2023-07-29T00:06:57.984 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:06:57.985 DEBUG:boto:Port: 8000 2023-07-29T00:06:57.985 DEBUG:boto:Params: {} 2023-07-29T00:06:57.986 DEBUG:boto:Token: None 2023-07-29T00:06:57.986 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:57 GMT /hiiqos-47/obj1?versionId=null 2023-07-29T00:06:57.987 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KCCJpduEgl30EEMBGRVHoo98gGY= 2023-07-29T00:06:57.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KCCJpduEgl30EEMBGRVHoo98gGY=', 'Content-Length': '0'} 2023-07-29T00:06:57.990 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000082067799e5146fc8-0064c45821-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:06:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:57.990 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:06:57.991 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:06:57.991 DEBUG:boto:path=/hiiqos-47/?prefix=obj1 2023-07-29T00:06:57.992 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj1 2023-07-29T00:06:57.992 DEBUG:boto:Method: GET 2023-07-29T00:06:57.993 DEBUG:boto:Path: /hiiqos-47/?prefix=obj1 2023-07-29T00:06:57.994 DEBUG:boto:Data: 2023-07-29T00:06:57.995 DEBUG:boto:Headers: {} 2023-07-29T00:06:57.996 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:06:57.997 DEBUG:boto:Port: 8001 2023-07-29T00:06:57.997 DEBUG:boto:Params: {} 2023-07-29T00:06:57.998 DEBUG:boto:Token: None 2023-07-29T00:06:57.999 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:57 GMT /hiiqos-47/ 2023-07-29T00:06:57.999 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ev0X1N2wPDsz2D1u3X1KYPgmQLA= 2023-07-29T00:06:58.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ev0X1N2wPDsz2D1u3X1KYPgmQLA=', 'Content-Length': '0'} 2023-07-29T00:06:58.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074db5abe86d976ac-0064c45821-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:06:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.003 DEBUG:boto:b'hiiqos-47obj11000falseobj12023-07-29T00:05:16.136Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:06:58.004 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:06:58.004 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:06:58.005 DEBUG:boto:path=/hiiqos-47/?prefix=obj1 2023-07-29T00:06:58.005 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj1 2023-07-29T00:06:58.005 DEBUG:boto:Method: GET 2023-07-29T00:06:58.006 DEBUG:boto:Path: /hiiqos-47/?prefix=obj1 2023-07-29T00:06:58.006 DEBUG:boto:Data: 2023-07-29T00:06:58.006 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.007 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:06:58.007 DEBUG:boto:Port: 8000 2023-07-29T00:06:58.007 DEBUG:boto:Params: {} 2023-07-29T00:06:58.008 DEBUG:boto:Token: None 2023-07-29T00:06:58.008 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/ 2023-07-29T00:06:58.009 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Er1nKg0TJRi24zQ4hUNK7zA+JV4= 2023-07-29T00:06:58.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Er1nKg0TJRi24zQ4hUNK7zA+JV4=', 'Content-Length': '0'} 2023-07-29T00:06:58.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000202b888d1376440d-0064c45821-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:06:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.013 DEBUG:boto:b'hiiqos-47obj11000falseobj12023-07-29T00:05:16.136Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:06:58.014 DEBUG:boto:path=/hiiqos-47/obj1 2023-07-29T00:06:58.014 DEBUG:boto:auth_path=/hiiqos-47/obj1 2023-07-29T00:06:58.015 DEBUG:boto:Method: HEAD 2023-07-29T00:06:58.015 DEBUG:boto:Path: /hiiqos-47/obj1 2023-07-29T00:06:58.016 DEBUG:boto:Data: 2023-07-29T00:06:58.016 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.017 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:06:58.017 DEBUG:boto:Port: 8001 2023-07-29T00:06:58.017 DEBUG:boto:Params: {} 2023-07-29T00:06:58.018 DEBUG:boto:Token: None 2023-07-29T00:06:58.018 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/obj1 2023-07-29T00:06:58.019 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GsPDvIU8IFpde4K+1jiDGkP4riM= 2023-07-29T00:06:58.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GsPDvIU8IFpde4K+1jiDGkP4riM=', 'Content-Length': '0'} 2023-07-29T00:06:58.021 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000023b547a2960f8465-0064c45821-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:06:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.022 DEBUG:boto:path=/hiiqos-47/obj1 2023-07-29T00:06:58.022 DEBUG:boto:auth_path=/hiiqos-47/obj1 2023-07-29T00:06:58.023 DEBUG:boto:Method: HEAD 2023-07-29T00:06:58.023 DEBUG:boto:Path: /hiiqos-47/obj1 2023-07-29T00:06:58.024 DEBUG:boto:Data: 2023-07-29T00:06:58.024 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.025 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:06:58.025 DEBUG:boto:Port: 8000 2023-07-29T00:06:58.026 DEBUG:boto:Params: {} 2023-07-29T00:06:58.026 DEBUG:boto:Token: None 2023-07-29T00:06:58.027 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/obj1 2023-07-29T00:06:58.027 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GsPDvIU8IFpde4K+1jiDGkP4riM= 2023-07-29T00:06:58.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GsPDvIU8IFpde4K+1jiDGkP4riM=', 'Content-Length': '0'} 2023-07-29T00:06:58.030 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000d36f81972d91313-0064c45822-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-07-29T00:06:58.031 DEBUG:boto:path=/hiiqos-47/obj1 2023-07-29T00:06:58.032 DEBUG:boto:auth_path=/hiiqos-47/obj1 2023-07-29T00:06:58.032 DEBUG:boto:Method: GET 2023-07-29T00:06:58.033 DEBUG:boto:Path: /hiiqos-47/obj1 2023-07-29T00:06:58.033 DEBUG:boto:Data: 2023-07-29T00:06:58.033 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.034 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:06:58.034 DEBUG:boto:Port: 8001 2023-07-29T00:06:58.035 DEBUG:boto:Params: {} 2023-07-29T00:06:58.035 DEBUG:boto:Token: None 2023-07-29T00:06:58.035 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/obj1 2023-07-29T00:06:58.036 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:isHHP6Vw3ygqW/bGeUw111R/9o8= 2023-07-29T00:06:58.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:isHHP6Vw3ygqW/bGeUw111R/9o8=', 'Content-Length': '0'} 2023-07-29T00:06:58.038 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001a5360b746983832-0064c45822-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.039 DEBUG:boto:path=/hiiqos-47/obj1 2023-07-29T00:06:58.039 DEBUG:boto:auth_path=/hiiqos-47/obj1 2023-07-29T00:06:58.040 DEBUG:boto:Method: GET 2023-07-29T00:06:58.040 DEBUG:boto:Path: /hiiqos-47/obj1 2023-07-29T00:06:58.041 DEBUG:boto:Data: 2023-07-29T00:06:58.041 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.042 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:06:58.042 DEBUG:boto:Port: 8000 2023-07-29T00:06:58.043 DEBUG:boto:Params: {} 2023-07-29T00:06:58.043 DEBUG:boto:Token: None 2023-07-29T00:06:58.044 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/obj1 2023-07-29T00:06:58.044 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:isHHP6Vw3ygqW/bGeUw111R/9o8= 2023-07-29T00:06:58.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:isHHP6Vw3ygqW/bGeUw111R/9o8=', 'Content-Length': '0'} 2023-07-29T00:06:58.047 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005ac7a33a663d821f-0064c45822-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.047 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:06:58.048 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:06:58.048 DEBUG:boto:path=/hiiqos-47/?prefix=obj1 2023-07-29T00:06:58.048 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj1 2023-07-29T00:06:58.049 DEBUG:boto:Method: GET 2023-07-29T00:06:58.049 DEBUG:boto:Path: /hiiqos-47/?prefix=obj1 2023-07-29T00:06:58.050 DEBUG:boto:Data: 2023-07-29T00:06:58.050 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.051 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:06:58.051 DEBUG:boto:Port: 8001 2023-07-29T00:06:58.052 DEBUG:boto:Params: {} 2023-07-29T00:06:58.052 DEBUG:boto:Token: None 2023-07-29T00:06:58.053 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/ 2023-07-29T00:06:58.054 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Er1nKg0TJRi24zQ4hUNK7zA+JV4= 2023-07-29T00:06:58.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Er1nKg0TJRi24zQ4hUNK7zA+JV4=', 'Content-Length': '0'} 2023-07-29T00:06:58.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000641f39106b3b5d90-0064c45822-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.057 DEBUG:boto:b'hiiqos-47obj11000falseobj12023-07-29T00:05:16.136Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:06:58.058 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:06:58.058 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:06:58.059 DEBUG:boto:path=/hiiqos-47/?prefix=obj1 2023-07-29T00:06:58.059 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj1 2023-07-29T00:06:58.059 DEBUG:boto:Method: GET 2023-07-29T00:06:58.060 DEBUG:boto:Path: /hiiqos-47/?prefix=obj1 2023-07-29T00:06:58.060 DEBUG:boto:Data: 2023-07-29T00:06:58.061 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.061 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:06:58.062 DEBUG:boto:Port: 8000 2023-07-29T00:06:58.062 DEBUG:boto:Params: {} 2023-07-29T00:06:58.062 DEBUG:boto:Token: None 2023-07-29T00:06:58.063 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/ 2023-07-29T00:06:58.063 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Er1nKg0TJRi24zQ4hUNK7zA+JV4= 2023-07-29T00:06:58.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Er1nKg0TJRi24zQ4hUNK7zA+JV4=', 'Content-Length': '0'} 2023-07-29T00:06:58.067 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001108e0f6f7afbf85-0064c45822-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.067 DEBUG:boto:b'hiiqos-47obj11000falseobj12023-07-29T00:05:16.136Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:06:58.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-07-29T00:06:58.068 DEBUG:boto:path=/hiiqos-47/obj2 2023-07-29T00:06:58.069 DEBUG:boto:auth_path=/hiiqos-47/obj2 2023-07-29T00:06:58.069 DEBUG:boto:path=/hiiqos-47/obj2?versionId=null 2023-07-29T00:06:58.069 DEBUG:boto:auth_path=/hiiqos-47/obj2?versionId=null 2023-07-29T00:06:58.070 DEBUG:boto:Method: GET 2023-07-29T00:06:58.070 DEBUG:boto:Path: /hiiqos-47/obj2?versionId=null 2023-07-29T00:06:58.071 DEBUG:boto:Data: 2023-07-29T00:06:58.071 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.072 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:06:58.072 DEBUG:boto:Port: 8001 2023-07-29T00:06:58.073 DEBUG:boto:Params: {} 2023-07-29T00:06:58.073 DEBUG:boto:Token: None 2023-07-29T00:06:58.074 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/obj2?versionId=null 2023-07-29T00:06:58.074 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VEEL09ab+p4DJRfv4GsdEh0es8A= 2023-07-29T00:06:58.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VEEL09ab+p4DJRfv4GsdEh0es8A=', 'Content-Length': '0'} 2023-07-29T00:06:58.076 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c14e772687aa7b66-0064c45822-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.077 DEBUG:boto:path=/hiiqos-47/obj2 2023-07-29T00:06:58.078 DEBUG:boto:auth_path=/hiiqos-47/obj2 2023-07-29T00:06:58.078 DEBUG:boto:path=/hiiqos-47/obj2?versionId=null 2023-07-29T00:06:58.078 DEBUG:boto:auth_path=/hiiqos-47/obj2?versionId=null 2023-07-29T00:06:58.079 DEBUG:boto:Method: GET 2023-07-29T00:06:58.079 DEBUG:boto:Path: /hiiqos-47/obj2?versionId=null 2023-07-29T00:06:58.080 DEBUG:boto:Data: 2023-07-29T00:06:58.080 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.080 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:06:58.081 DEBUG:boto:Port: 8000 2023-07-29T00:06:58.081 DEBUG:boto:Params: {} 2023-07-29T00:06:58.082 DEBUG:boto:Token: None 2023-07-29T00:06:58.082 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/obj2?versionId=null 2023-07-29T00:06:58.082 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VEEL09ab+p4DJRfv4GsdEh0es8A= 2023-07-29T00:06:58.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VEEL09ab+p4DJRfv4GsdEh0es8A=', 'Content-Length': '0'} 2023-07-29T00:06:58.085 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000251d61e9c399735c-0064c45822-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.086 DEBUG:boto:path=/hiiqos-47/obj2 2023-07-29T00:06:58.086 DEBUG:boto:auth_path=/hiiqos-47/obj2 2023-07-29T00:06:58.086 DEBUG:boto:path=/hiiqos-47/obj2?versionId=null 2023-07-29T00:06:58.087 DEBUG:boto:auth_path=/hiiqos-47/obj2?versionId=null 2023-07-29T00:06:58.087 DEBUG:boto:Method: HEAD 2023-07-29T00:06:58.088 DEBUG:boto:Path: /hiiqos-47/obj2?versionId=null 2023-07-29T00:06:58.088 DEBUG:boto:Data: 2023-07-29T00:06:58.089 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.089 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:06:58.089 DEBUG:boto:Port: 8001 2023-07-29T00:06:58.090 DEBUG:boto:Params: {} 2023-07-29T00:06:58.090 DEBUG:boto:Token: None 2023-07-29T00:06:58.090 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/obj2?versionId=null 2023-07-29T00:06:58.091 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wPE543JP7BhQ5RgfRcpaZrc9RIg= 2023-07-29T00:06:58.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wPE543JP7BhQ5RgfRcpaZrc9RIg=', 'Content-Length': '0'} 2023-07-29T00:06:58.093 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e4ef047a7c8babd-0064c45822-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.094 DEBUG:boto:path=/hiiqos-47/obj2 2023-07-29T00:06:58.094 DEBUG:boto:auth_path=/hiiqos-47/obj2 2023-07-29T00:06:58.094 DEBUG:boto:path=/hiiqos-47/obj2?versionId=null 2023-07-29T00:06:58.095 DEBUG:boto:auth_path=/hiiqos-47/obj2?versionId=null 2023-07-29T00:06:58.095 DEBUG:boto:Method: HEAD 2023-07-29T00:06:58.095 DEBUG:boto:Path: /hiiqos-47/obj2?versionId=null 2023-07-29T00:06:58.096 DEBUG:boto:Data: 2023-07-29T00:06:58.096 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.096 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:06:58.097 DEBUG:boto:Port: 8000 2023-07-29T00:06:58.097 DEBUG:boto:Params: {} 2023-07-29T00:06:58.098 DEBUG:boto:Token: None 2023-07-29T00:06:58.098 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/obj2?versionId=null 2023-07-29T00:06:58.098 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wPE543JP7BhQ5RgfRcpaZrc9RIg= 2023-07-29T00:06:58.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wPE543JP7BhQ5RgfRcpaZrc9RIg=', 'Content-Length': '0'} 2023-07-29T00:06:58.101 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d21873f93e6d9a9d-0064c45822-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-07-29T00:06:58.102 DEBUG:boto:path=/hiiqos-47/obj2 2023-07-29T00:06:58.102 DEBUG:boto:auth_path=/hiiqos-47/obj2 2023-07-29T00:06:58.102 DEBUG:boto:path=/hiiqos-47/obj2?versionId=null 2023-07-29T00:06:58.103 DEBUG:boto:auth_path=/hiiqos-47/obj2?versionId=null 2023-07-29T00:06:58.103 DEBUG:boto:Method: GET 2023-07-29T00:06:58.103 DEBUG:boto:Path: /hiiqos-47/obj2?versionId=null 2023-07-29T00:06:58.104 DEBUG:boto:Data: 2023-07-29T00:06:58.104 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.104 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:06:58.105 DEBUG:boto:Port: 8001 2023-07-29T00:06:58.105 DEBUG:boto:Params: {} 2023-07-29T00:06:58.106 DEBUG:boto:Token: None 2023-07-29T00:06:58.106 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/obj2?versionId=null 2023-07-29T00:06:58.106 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VEEL09ab+p4DJRfv4GsdEh0es8A= 2023-07-29T00:06:58.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VEEL09ab+p4DJRfv4GsdEh0es8A=', 'Content-Length': '0'} 2023-07-29T00:06:58.108 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004689b85fdd29790d-0064c45822-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.156 DEBUG:boto:path=/hiiqos-47/obj2 2023-07-29T00:06:58.157 DEBUG:boto:auth_path=/hiiqos-47/obj2 2023-07-29T00:06:58.157 DEBUG:boto:path=/hiiqos-47/obj2?versionId=null 2023-07-29T00:06:58.158 DEBUG:boto:auth_path=/hiiqos-47/obj2?versionId=null 2023-07-29T00:06:58.158 DEBUG:boto:Method: GET 2023-07-29T00:06:58.158 DEBUG:boto:Path: /hiiqos-47/obj2?versionId=null 2023-07-29T00:06:58.159 DEBUG:boto:Data: 2023-07-29T00:06:58.159 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.159 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:06:58.160 DEBUG:boto:Port: 8000 2023-07-29T00:06:58.160 DEBUG:boto:Params: {} 2023-07-29T00:06:58.161 DEBUG:boto:Token: None 2023-07-29T00:06:58.161 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/obj2?versionId=null 2023-07-29T00:06:58.161 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VEEL09ab+p4DJRfv4GsdEh0es8A= 2023-07-29T00:06:58.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VEEL09ab+p4DJRfv4GsdEh0es8A=', 'Content-Length': '0'} 2023-07-29T00:06:58.164 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000086c25c3769616450-0064c45822-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.165 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:06:58.165 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:06:58.165 DEBUG:boto:path=/hiiqos-47/?prefix=obj2 2023-07-29T00:06:58.166 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj2 2023-07-29T00:06:58.166 DEBUG:boto:Method: GET 2023-07-29T00:06:58.166 DEBUG:boto:Path: /hiiqos-47/?prefix=obj2 2023-07-29T00:06:58.167 DEBUG:boto:Data: 2023-07-29T00:06:58.167 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.167 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:06:58.168 DEBUG:boto:Port: 8001 2023-07-29T00:06:58.168 DEBUG:boto:Params: {} 2023-07-29T00:06:58.169 DEBUG:boto:Token: None 2023-07-29T00:06:58.169 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/ 2023-07-29T00:06:58.169 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Er1nKg0TJRi24zQ4hUNK7zA+JV4= 2023-07-29T00:06:58.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Er1nKg0TJRi24zQ4hUNK7zA+JV4=', 'Content-Length': '0'} 2023-07-29T00:06:58.172 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000943483e4b6aeb2c2-0064c45822-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.173 DEBUG:boto:b'hiiqos-47obj21000falseobj22023-07-29T00:05:16.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:06:58.173 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:06:58.174 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:06:58.174 DEBUG:boto:path=/hiiqos-47/?prefix=obj2 2023-07-29T00:06:58.174 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj2 2023-07-29T00:06:58.175 DEBUG:boto:Method: GET 2023-07-29T00:06:58.175 DEBUG:boto:Path: /hiiqos-47/?prefix=obj2 2023-07-29T00:06:58.175 DEBUG:boto:Data: 2023-07-29T00:06:58.176 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.176 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:06:58.177 DEBUG:boto:Port: 8000 2023-07-29T00:06:58.177 DEBUG:boto:Params: {} 2023-07-29T00:06:58.177 DEBUG:boto:Token: None 2023-07-29T00:06:58.178 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/ 2023-07-29T00:06:58.178 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Er1nKg0TJRi24zQ4hUNK7zA+JV4= 2023-07-29T00:06:58.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Er1nKg0TJRi24zQ4hUNK7zA+JV4=', 'Content-Length': '0'} 2023-07-29T00:06:58.181 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f986415ed446548b-0064c45822-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.182 DEBUG:boto:b'hiiqos-47obj21000falseobj22023-07-29T00:05:16.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:06:58.182 DEBUG:boto:path=/hiiqos-47/obj2 2023-07-29T00:06:58.183 DEBUG:boto:auth_path=/hiiqos-47/obj2 2023-07-29T00:06:58.183 DEBUG:boto:Method: HEAD 2023-07-29T00:06:58.183 DEBUG:boto:Path: /hiiqos-47/obj2 2023-07-29T00:06:58.184 DEBUG:boto:Data: 2023-07-29T00:06:58.184 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.184 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:06:58.185 DEBUG:boto:Port: 8001 2023-07-29T00:06:58.185 DEBUG:boto:Params: {} 2023-07-29T00:06:58.185 DEBUG:boto:Token: None 2023-07-29T00:06:58.186 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/obj2 2023-07-29T00:06:58.186 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:NNbZDqEeUeCnQiu/PuV46myihZM= 2023-07-29T00:06:58.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:NNbZDqEeUeCnQiu/PuV46myihZM=', 'Content-Length': '0'} 2023-07-29T00:06:58.188 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f35585a71a20e301-0064c45822-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.189 DEBUG:boto:path=/hiiqos-47/obj2 2023-07-29T00:06:58.189 DEBUG:boto:auth_path=/hiiqos-47/obj2 2023-07-29T00:06:58.190 DEBUG:boto:Method: HEAD 2023-07-29T00:06:58.190 DEBUG:boto:Path: /hiiqos-47/obj2 2023-07-29T00:06:58.190 DEBUG:boto:Data: 2023-07-29T00:06:58.191 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.191 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:06:58.191 DEBUG:boto:Port: 8000 2023-07-29T00:06:58.192 DEBUG:boto:Params: {} 2023-07-29T00:06:58.192 DEBUG:boto:Token: None 2023-07-29T00:06:58.192 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/obj2 2023-07-29T00:06:58.193 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:NNbZDqEeUeCnQiu/PuV46myihZM= 2023-07-29T00:06:58.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:NNbZDqEeUeCnQiu/PuV46myihZM=', 'Content-Length': '0'} 2023-07-29T00:06:58.195 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009ca34675d4432c52-0064c45822-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-07-29T00:06:58.196 DEBUG:boto:path=/hiiqos-47/obj2 2023-07-29T00:06:58.196 DEBUG:boto:auth_path=/hiiqos-47/obj2 2023-07-29T00:06:58.197 DEBUG:boto:Method: GET 2023-07-29T00:06:58.197 DEBUG:boto:Path: /hiiqos-47/obj2 2023-07-29T00:06:58.197 DEBUG:boto:Data: 2023-07-29T00:06:58.198 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.198 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:06:58.199 DEBUG:boto:Port: 8001 2023-07-29T00:06:58.199 DEBUG:boto:Params: {} 2023-07-29T00:06:58.199 DEBUG:boto:Token: None 2023-07-29T00:06:58.200 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/obj2 2023-07-29T00:06:58.200 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ARfKYMnTCQHgncZm6jUiUQ6XEFE= 2023-07-29T00:06:58.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ARfKYMnTCQHgncZm6jUiUQ6XEFE=', 'Content-Length': '0'} 2023-07-29T00:06:58.202 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d94f41b2df8b0622-0064c45822-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.244 DEBUG:boto:path=/hiiqos-47/obj2 2023-07-29T00:06:58.245 DEBUG:boto:auth_path=/hiiqos-47/obj2 2023-07-29T00:06:58.245 DEBUG:boto:Method: GET 2023-07-29T00:06:58.246 DEBUG:boto:Path: /hiiqos-47/obj2 2023-07-29T00:06:58.246 DEBUG:boto:Data: 2023-07-29T00:06:58.246 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.247 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:06:58.247 DEBUG:boto:Port: 8000 2023-07-29T00:06:58.247 DEBUG:boto:Params: {} 2023-07-29T00:06:58.248 DEBUG:boto:Token: None 2023-07-29T00:06:58.248 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/obj2 2023-07-29T00:06:58.249 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ARfKYMnTCQHgncZm6jUiUQ6XEFE= 2023-07-29T00:06:58.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ARfKYMnTCQHgncZm6jUiUQ6XEFE=', 'Content-Length': '0'} 2023-07-29T00:06:58.251 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000037d5906f6ca58b4c-0064c45822-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.252 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:06:58.252 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:06:58.253 DEBUG:boto:path=/hiiqos-47/?prefix=obj2 2023-07-29T00:06:58.253 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj2 2023-07-29T00:06:58.253 DEBUG:boto:Method: GET 2023-07-29T00:06:58.254 DEBUG:boto:Path: /hiiqos-47/?prefix=obj2 2023-07-29T00:06:58.254 DEBUG:boto:Data: 2023-07-29T00:06:58.254 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.255 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:06:58.255 DEBUG:boto:Port: 8001 2023-07-29T00:06:58.256 DEBUG:boto:Params: {} 2023-07-29T00:06:58.256 DEBUG:boto:Token: None 2023-07-29T00:06:58.256 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/ 2023-07-29T00:06:58.257 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Er1nKg0TJRi24zQ4hUNK7zA+JV4= 2023-07-29T00:06:58.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Er1nKg0TJRi24zQ4hUNK7zA+JV4=', 'Content-Length': '0'} 2023-07-29T00:06:58.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f71719e63efb7b0-0064c45822-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.260 DEBUG:boto:b'hiiqos-47obj21000falseobj22023-07-29T00:05:16.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:06:58.261 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:06:58.261 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:06:58.262 DEBUG:boto:path=/hiiqos-47/?prefix=obj2 2023-07-29T00:06:58.262 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj2 2023-07-29T00:06:58.262 DEBUG:boto:Method: GET 2023-07-29T00:06:58.263 DEBUG:boto:Path: /hiiqos-47/?prefix=obj2 2023-07-29T00:06:58.263 DEBUG:boto:Data: 2023-07-29T00:06:58.263 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.264 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:06:58.264 DEBUG:boto:Port: 8000 2023-07-29T00:06:58.264 DEBUG:boto:Params: {} 2023-07-29T00:06:58.265 DEBUG:boto:Token: None 2023-07-29T00:06:58.265 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/ 2023-07-29T00:06:58.266 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Er1nKg0TJRi24zQ4hUNK7zA+JV4= 2023-07-29T00:06:58.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Er1nKg0TJRi24zQ4hUNK7zA+JV4=', 'Content-Length': '0'} 2023-07-29T00:06:58.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000269c9e094e24f480-0064c45822-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.269 DEBUG:boto:b'hiiqos-47obj21000falseobj22023-07-29T00:05:16.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:06:58.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-07-29T00:06:58.270 DEBUG:boto:path=/hiiqos-47/obj3 2023-07-29T00:06:58.271 DEBUG:boto:auth_path=/hiiqos-47/obj3 2023-07-29T00:06:58.271 DEBUG:boto:path=/hiiqos-47/obj3?versionId=null 2023-07-29T00:06:58.271 DEBUG:boto:auth_path=/hiiqos-47/obj3?versionId=null 2023-07-29T00:06:58.272 DEBUG:boto:Method: GET 2023-07-29T00:06:58.272 DEBUG:boto:Path: /hiiqos-47/obj3?versionId=null 2023-07-29T00:06:58.272 DEBUG:boto:Data: 2023-07-29T00:06:58.273 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.273 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:06:58.274 DEBUG:boto:Port: 8001 2023-07-29T00:06:58.274 DEBUG:boto:Params: {} 2023-07-29T00:06:58.274 DEBUG:boto:Token: None 2023-07-29T00:06:58.275 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/obj3?versionId=null 2023-07-29T00:06:58.275 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lv18xIAdKdAwKEM3SBKFzA3sJBk= 2023-07-29T00:06:58.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lv18xIAdKdAwKEM3SBKFzA3sJBk=', 'Content-Length': '0'} 2023-07-29T00:06:58.278 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007591af4808d043a6-0064c45822-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.278 DEBUG:boto:path=/hiiqos-47/obj3 2023-07-29T00:06:58.279 DEBUG:boto:auth_path=/hiiqos-47/obj3 2023-07-29T00:06:58.279 DEBUG:boto:path=/hiiqos-47/obj3?versionId=null 2023-07-29T00:06:58.279 DEBUG:boto:auth_path=/hiiqos-47/obj3?versionId=null 2023-07-29T00:06:58.280 DEBUG:boto:Method: GET 2023-07-29T00:06:58.280 DEBUG:boto:Path: /hiiqos-47/obj3?versionId=null 2023-07-29T00:06:58.280 DEBUG:boto:Data: 2023-07-29T00:06:58.281 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.281 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:06:58.281 DEBUG:boto:Port: 8000 2023-07-29T00:06:58.282 DEBUG:boto:Params: {} 2023-07-29T00:06:58.282 DEBUG:boto:Token: None 2023-07-29T00:06:58.283 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/obj3?versionId=null 2023-07-29T00:06:58.283 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lv18xIAdKdAwKEM3SBKFzA3sJBk= 2023-07-29T00:06:58.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lv18xIAdKdAwKEM3SBKFzA3sJBk=', 'Content-Length': '0'} 2023-07-29T00:06:58.285 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a959d153ea3d2d2-0064c45822-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.286 DEBUG:boto:path=/hiiqos-47/obj3 2023-07-29T00:06:58.286 DEBUG:boto:auth_path=/hiiqos-47/obj3 2023-07-29T00:06:58.287 DEBUG:boto:path=/hiiqos-47/obj3?versionId=null 2023-07-29T00:06:58.287 DEBUG:boto:auth_path=/hiiqos-47/obj3?versionId=null 2023-07-29T00:06:58.287 DEBUG:boto:Method: HEAD 2023-07-29T00:06:58.288 DEBUG:boto:Path: /hiiqos-47/obj3?versionId=null 2023-07-29T00:06:58.288 DEBUG:boto:Data: 2023-07-29T00:06:58.288 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.289 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:06:58.289 DEBUG:boto:Port: 8001 2023-07-29T00:06:58.289 DEBUG:boto:Params: {} 2023-07-29T00:06:58.290 DEBUG:boto:Token: None 2023-07-29T00:06:58.290 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/obj3?versionId=null 2023-07-29T00:06:58.291 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ywuOpJK9lNV1kPjNZZ+GVxgwN4k= 2023-07-29T00:06:58.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ywuOpJK9lNV1kPjNZZ+GVxgwN4k=', 'Content-Length': '0'} 2023-07-29T00:06:58.293 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ead8d54157d1ec76-0064c45822-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.293 DEBUG:boto:path=/hiiqos-47/obj3 2023-07-29T00:06:58.294 DEBUG:boto:auth_path=/hiiqos-47/obj3 2023-07-29T00:06:58.294 DEBUG:boto:path=/hiiqos-47/obj3?versionId=null 2023-07-29T00:06:58.294 DEBUG:boto:auth_path=/hiiqos-47/obj3?versionId=null 2023-07-29T00:06:58.295 DEBUG:boto:Method: HEAD 2023-07-29T00:06:58.295 DEBUG:boto:Path: /hiiqos-47/obj3?versionId=null 2023-07-29T00:06:58.295 DEBUG:boto:Data: 2023-07-29T00:06:58.296 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.296 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:06:58.296 DEBUG:boto:Port: 8000 2023-07-29T00:06:58.297 DEBUG:boto:Params: {} 2023-07-29T00:06:58.297 DEBUG:boto:Token: None 2023-07-29T00:06:58.297 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/obj3?versionId=null 2023-07-29T00:06:58.298 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ywuOpJK9lNV1kPjNZZ+GVxgwN4k= 2023-07-29T00:06:58.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ywuOpJK9lNV1kPjNZZ+GVxgwN4k=', 'Content-Length': '0'} 2023-07-29T00:06:58.300 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000085a8aad972310403-0064c45822-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-07-29T00:06:58.301 DEBUG:boto:path=/hiiqos-47/obj3 2023-07-29T00:06:58.301 DEBUG:boto:auth_path=/hiiqos-47/obj3 2023-07-29T00:06:58.302 DEBUG:boto:path=/hiiqos-47/obj3?versionId=null 2023-07-29T00:06:58.302 DEBUG:boto:auth_path=/hiiqos-47/obj3?versionId=null 2023-07-29T00:06:58.302 DEBUG:boto:Method: GET 2023-07-29T00:06:58.303 DEBUG:boto:Path: /hiiqos-47/obj3?versionId=null 2023-07-29T00:06:58.303 DEBUG:boto:Data: 2023-07-29T00:06:58.303 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.304 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:06:58.304 DEBUG:boto:Port: 8001 2023-07-29T00:06:58.304 DEBUG:boto:Params: {} 2023-07-29T00:06:58.305 DEBUG:boto:Token: None 2023-07-29T00:06:58.305 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/obj3?versionId=null 2023-07-29T00:06:58.306 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lv18xIAdKdAwKEM3SBKFzA3sJBk= 2023-07-29T00:06:58.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lv18xIAdKdAwKEM3SBKFzA3sJBk=', 'Content-Length': '0'} 2023-07-29T00:06:58.308 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008bcba7679f8ab55a-0064c45822-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.309 DEBUG:boto:path=/hiiqos-47/obj3 2023-07-29T00:06:58.309 DEBUG:boto:auth_path=/hiiqos-47/obj3 2023-07-29T00:06:58.310 DEBUG:boto:path=/hiiqos-47/obj3?versionId=null 2023-07-29T00:06:58.310 DEBUG:boto:auth_path=/hiiqos-47/obj3?versionId=null 2023-07-29T00:06:58.310 DEBUG:boto:Method: GET 2023-07-29T00:06:58.311 DEBUG:boto:Path: /hiiqos-47/obj3?versionId=null 2023-07-29T00:06:58.311 DEBUG:boto:Data: 2023-07-29T00:06:58.311 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.312 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:06:58.312 DEBUG:boto:Port: 8000 2023-07-29T00:06:58.312 DEBUG:boto:Params: {} 2023-07-29T00:06:58.313 DEBUG:boto:Token: None 2023-07-29T00:06:58.313 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/obj3?versionId=null 2023-07-29T00:06:58.314 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lv18xIAdKdAwKEM3SBKFzA3sJBk= 2023-07-29T00:06:58.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lv18xIAdKdAwKEM3SBKFzA3sJBk=', 'Content-Length': '0'} 2023-07-29T00:06:58.316 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001f2b74d3bf58a2e7-0064c45822-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.316 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:06:58.317 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:06:58.317 DEBUG:boto:path=/hiiqos-47/?prefix=obj3 2023-07-29T00:06:58.318 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj3 2023-07-29T00:06:58.318 DEBUG:boto:Method: GET 2023-07-29T00:06:58.318 DEBUG:boto:Path: /hiiqos-47/?prefix=obj3 2023-07-29T00:06:58.319 DEBUG:boto:Data: 2023-07-29T00:06:58.319 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.319 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:06:58.320 DEBUG:boto:Port: 8001 2023-07-29T00:06:58.320 DEBUG:boto:Params: {} 2023-07-29T00:06:58.320 DEBUG:boto:Token: None 2023-07-29T00:06:58.321 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/ 2023-07-29T00:06:58.321 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Er1nKg0TJRi24zQ4hUNK7zA+JV4= 2023-07-29T00:06:58.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Er1nKg0TJRi24zQ4hUNK7zA+JV4=', 'Content-Length': '0'} 2023-07-29T00:06:58.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a805649a540c9c0c-0064c45822-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.364 DEBUG:boto:b'hiiqos-47obj31000falseobj32023-07-29T00:05:16.174Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:06:58.365 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:06:58.365 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:06:58.366 DEBUG:boto:path=/hiiqos-47/?prefix=obj3 2023-07-29T00:06:58.366 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj3 2023-07-29T00:06:58.367 DEBUG:boto:Method: GET 2023-07-29T00:06:58.367 DEBUG:boto:Path: /hiiqos-47/?prefix=obj3 2023-07-29T00:06:58.367 DEBUG:boto:Data: 2023-07-29T00:06:58.368 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.368 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:06:58.368 DEBUG:boto:Port: 8000 2023-07-29T00:06:58.369 DEBUG:boto:Params: {} 2023-07-29T00:06:58.369 DEBUG:boto:Token: None 2023-07-29T00:06:58.370 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/ 2023-07-29T00:06:58.370 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Er1nKg0TJRi24zQ4hUNK7zA+JV4= 2023-07-29T00:06:58.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Er1nKg0TJRi24zQ4hUNK7zA+JV4=', 'Content-Length': '0'} 2023-07-29T00:06:58.373 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b049f0c64b97eee5-0064c45822-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.374 DEBUG:boto:b'hiiqos-47obj31000falseobj32023-07-29T00:05:16.174Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:06:58.374 DEBUG:boto:path=/hiiqos-47/obj3 2023-07-29T00:06:58.375 DEBUG:boto:auth_path=/hiiqos-47/obj3 2023-07-29T00:06:58.375 DEBUG:boto:Method: HEAD 2023-07-29T00:06:58.375 DEBUG:boto:Path: /hiiqos-47/obj3 2023-07-29T00:06:58.376 DEBUG:boto:Data: 2023-07-29T00:06:58.376 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.376 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:06:58.377 DEBUG:boto:Port: 8001 2023-07-29T00:06:58.377 DEBUG:boto:Params: {} 2023-07-29T00:06:58.377 DEBUG:boto:Token: None 2023-07-29T00:06:58.378 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/obj3 2023-07-29T00:06:58.378 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EjrW1AQV9bMjOlUuCYPtSSMuOuw= 2023-07-29T00:06:58.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EjrW1AQV9bMjOlUuCYPtSSMuOuw=', 'Content-Length': '0'} 2023-07-29T00:06:58.380 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a67bb342ec7fd253-0064c45822-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.381 DEBUG:boto:path=/hiiqos-47/obj3 2023-07-29T00:06:58.381 DEBUG:boto:auth_path=/hiiqos-47/obj3 2023-07-29T00:06:58.382 DEBUG:boto:Method: HEAD 2023-07-29T00:06:58.382 DEBUG:boto:Path: /hiiqos-47/obj3 2023-07-29T00:06:58.382 DEBUG:boto:Data: 2023-07-29T00:06:58.383 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.383 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:06:58.383 DEBUG:boto:Port: 8000 2023-07-29T00:06:58.384 DEBUG:boto:Params: {} 2023-07-29T00:06:58.384 DEBUG:boto:Token: None 2023-07-29T00:06:58.384 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/obj3 2023-07-29T00:06:58.385 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EjrW1AQV9bMjOlUuCYPtSSMuOuw= 2023-07-29T00:06:58.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EjrW1AQV9bMjOlUuCYPtSSMuOuw=', 'Content-Length': '0'} 2023-07-29T00:06:58.387 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008b82393176eb3773-0064c45822-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-07-29T00:06:58.388 DEBUG:boto:path=/hiiqos-47/obj3 2023-07-29T00:06:58.388 DEBUG:boto:auth_path=/hiiqos-47/obj3 2023-07-29T00:06:58.389 DEBUG:boto:Method: GET 2023-07-29T00:06:58.389 DEBUG:boto:Path: /hiiqos-47/obj3 2023-07-29T00:06:58.389 DEBUG:boto:Data: 2023-07-29T00:06:58.390 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.390 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:06:58.390 DEBUG:boto:Port: 8001 2023-07-29T00:06:58.391 DEBUG:boto:Params: {} 2023-07-29T00:06:58.391 DEBUG:boto:Token: None 2023-07-29T00:06:58.392 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/obj3 2023-07-29T00:06:58.392 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:N/vcTbF6xF3sLPzLkylY7/jWL8M= 2023-07-29T00:06:58.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:N/vcTbF6xF3sLPzLkylY7/jWL8M=', 'Content-Length': '0'} 2023-07-29T00:06:58.394 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d02951a550bcb059-0064c45822-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.395 DEBUG:boto:path=/hiiqos-47/obj3 2023-07-29T00:06:58.395 DEBUG:boto:auth_path=/hiiqos-47/obj3 2023-07-29T00:06:58.396 DEBUG:boto:Method: GET 2023-07-29T00:06:58.396 DEBUG:boto:Path: /hiiqos-47/obj3 2023-07-29T00:06:58.396 DEBUG:boto:Data: 2023-07-29T00:06:58.397 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.397 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:06:58.397 DEBUG:boto:Port: 8000 2023-07-29T00:06:58.398 DEBUG:boto:Params: {} 2023-07-29T00:06:58.398 DEBUG:boto:Token: None 2023-07-29T00:06:58.398 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/obj3 2023-07-29T00:06:58.399 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:N/vcTbF6xF3sLPzLkylY7/jWL8M= 2023-07-29T00:06:58.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:N/vcTbF6xF3sLPzLkylY7/jWL8M=', 'Content-Length': '0'} 2023-07-29T00:06:58.401 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006f9434d317e0f8af-0064c45822-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.402 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:06:58.402 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:06:58.403 DEBUG:boto:path=/hiiqos-47/?prefix=obj3 2023-07-29T00:06:58.403 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj3 2023-07-29T00:06:58.403 DEBUG:boto:Method: GET 2023-07-29T00:06:58.404 DEBUG:boto:Path: /hiiqos-47/?prefix=obj3 2023-07-29T00:06:58.404 DEBUG:boto:Data: 2023-07-29T00:06:58.404 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.405 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:06:58.405 DEBUG:boto:Port: 8001 2023-07-29T00:06:58.405 DEBUG:boto:Params: {} 2023-07-29T00:06:58.406 DEBUG:boto:Token: None 2023-07-29T00:06:58.406 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/ 2023-07-29T00:06:58.407 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Er1nKg0TJRi24zQ4hUNK7zA+JV4= 2023-07-29T00:06:58.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Er1nKg0TJRi24zQ4hUNK7zA+JV4=', 'Content-Length': '0'} 2023-07-29T00:06:58.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddb7154ec6d709bf-0064c45822-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.410 DEBUG:boto:b'hiiqos-47obj31000falseobj32023-07-29T00:05:16.174Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:06:58.411 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:06:58.411 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:06:58.411 DEBUG:boto:path=/hiiqos-47/?prefix=obj3 2023-07-29T00:06:58.412 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj3 2023-07-29T00:06:58.412 DEBUG:boto:Method: GET 2023-07-29T00:06:58.412 DEBUG:boto:Path: /hiiqos-47/?prefix=obj3 2023-07-29T00:06:58.413 DEBUG:boto:Data: 2023-07-29T00:06:58.413 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.414 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:06:58.414 DEBUG:boto:Port: 8000 2023-07-29T00:06:58.414 DEBUG:boto:Params: {} 2023-07-29T00:06:58.415 DEBUG:boto:Token: None 2023-07-29T00:06:58.415 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/ 2023-07-29T00:06:58.415 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Er1nKg0TJRi24zQ4hUNK7zA+JV4= 2023-07-29T00:06:58.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Er1nKg0TJRi24zQ4hUNK7zA+JV4=', 'Content-Length': '0'} 2023-07-29T00:06:58.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a9c397d8296307f-0064c45822-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.460 DEBUG:boto:b'hiiqos-47obj31000falseobj32023-07-29T00:05:16.174Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:06:58.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-07-29T00:06:58.462 DEBUG:boto:path=/hiiqos-47/obj4 2023-07-29T00:06:58.462 DEBUG:boto:auth_path=/hiiqos-47/obj4 2023-07-29T00:06:58.462 DEBUG:boto:path=/hiiqos-47/obj4?versionId=null 2023-07-29T00:06:58.463 DEBUG:boto:auth_path=/hiiqos-47/obj4?versionId=null 2023-07-29T00:06:58.463 DEBUG:boto:Method: GET 2023-07-29T00:06:58.463 DEBUG:boto:Path: /hiiqos-47/obj4?versionId=null 2023-07-29T00:06:58.464 DEBUG:boto:Data: 2023-07-29T00:06:58.464 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.464 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:06:58.465 DEBUG:boto:Port: 8001 2023-07-29T00:06:58.465 DEBUG:boto:Params: {} 2023-07-29T00:06:58.466 DEBUG:boto:Token: None 2023-07-29T00:06:58.466 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/obj4?versionId=null 2023-07-29T00:06:58.466 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:k5sFs+fThcdVoH7rrsdbranRLv0= 2023-07-29T00:06:58.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:k5sFs+fThcdVoH7rrsdbranRLv0=', 'Content-Length': '0'} 2023-07-29T00:06:58.469 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009566426e8390243a-0064c45822-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.469 DEBUG:boto:path=/hiiqos-47/obj4 2023-07-29T00:06:58.470 DEBUG:boto:auth_path=/hiiqos-47/obj4 2023-07-29T00:06:58.470 DEBUG:boto:path=/hiiqos-47/obj4?versionId=null 2023-07-29T00:06:58.471 DEBUG:boto:auth_path=/hiiqos-47/obj4?versionId=null 2023-07-29T00:06:58.471 DEBUG:boto:Method: GET 2023-07-29T00:06:58.471 DEBUG:boto:Path: /hiiqos-47/obj4?versionId=null 2023-07-29T00:06:58.472 DEBUG:boto:Data: 2023-07-29T00:06:58.472 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.472 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:06:58.473 DEBUG:boto:Port: 8000 2023-07-29T00:06:58.473 DEBUG:boto:Params: {} 2023-07-29T00:06:58.474 DEBUG:boto:Token: None 2023-07-29T00:06:58.474 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/obj4?versionId=null 2023-07-29T00:06:58.474 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:k5sFs+fThcdVoH7rrsdbranRLv0= 2023-07-29T00:06:58.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:k5sFs+fThcdVoH7rrsdbranRLv0=', 'Content-Length': '0'} 2023-07-29T00:06:58.477 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d3437bc899fee5e-0064c45822-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.478 DEBUG:boto:path=/hiiqos-47/obj4 2023-07-29T00:06:58.478 DEBUG:boto:auth_path=/hiiqos-47/obj4 2023-07-29T00:06:58.478 DEBUG:boto:path=/hiiqos-47/obj4?versionId=null 2023-07-29T00:06:58.479 DEBUG:boto:auth_path=/hiiqos-47/obj4?versionId=null 2023-07-29T00:06:58.479 DEBUG:boto:Method: HEAD 2023-07-29T00:06:58.479 DEBUG:boto:Path: /hiiqos-47/obj4?versionId=null 2023-07-29T00:06:58.480 DEBUG:boto:Data: 2023-07-29T00:06:58.480 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.480 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:06:58.481 DEBUG:boto:Port: 8001 2023-07-29T00:06:58.481 DEBUG:boto:Params: {} 2023-07-29T00:06:58.481 DEBUG:boto:Token: None 2023-07-29T00:06:58.482 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/obj4?versionId=null 2023-07-29T00:06:58.482 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JrklFSMq2GQ3dMFTESW6x2Z+E6A= 2023-07-29T00:06:58.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JrklFSMq2GQ3dMFTESW6x2Z+E6A=', 'Content-Length': '0'} 2023-07-29T00:06:58.484 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000052957db8de8d5dc8-0064c45822-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.485 DEBUG:boto:path=/hiiqos-47/obj4 2023-07-29T00:06:58.485 DEBUG:boto:auth_path=/hiiqos-47/obj4 2023-07-29T00:06:58.486 DEBUG:boto:path=/hiiqos-47/obj4?versionId=null 2023-07-29T00:06:58.486 DEBUG:boto:auth_path=/hiiqos-47/obj4?versionId=null 2023-07-29T00:06:58.486 DEBUG:boto:Method: HEAD 2023-07-29T00:06:58.487 DEBUG:boto:Path: /hiiqos-47/obj4?versionId=null 2023-07-29T00:06:58.487 DEBUG:boto:Data: 2023-07-29T00:06:58.487 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.488 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:06:58.488 DEBUG:boto:Port: 8000 2023-07-29T00:06:58.488 DEBUG:boto:Params: {} 2023-07-29T00:06:58.489 DEBUG:boto:Token: None 2023-07-29T00:06:58.489 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/obj4?versionId=null 2023-07-29T00:06:58.489 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JrklFSMq2GQ3dMFTESW6x2Z+E6A= 2023-07-29T00:06:58.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JrklFSMq2GQ3dMFTESW6x2Z+E6A=', 'Content-Length': '0'} 2023-07-29T00:06:58.492 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000044b675fc3b1f5beb-0064c45822-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-07-29T00:06:58.493 DEBUG:boto:path=/hiiqos-47/obj4 2023-07-29T00:06:58.493 DEBUG:boto:auth_path=/hiiqos-47/obj4 2023-07-29T00:06:58.493 DEBUG:boto:path=/hiiqos-47/obj4?versionId=null 2023-07-29T00:06:58.494 DEBUG:boto:auth_path=/hiiqos-47/obj4?versionId=null 2023-07-29T00:06:58.494 DEBUG:boto:Method: GET 2023-07-29T00:06:58.495 DEBUG:boto:Path: /hiiqos-47/obj4?versionId=null 2023-07-29T00:06:58.495 DEBUG:boto:Data: 2023-07-29T00:06:58.495 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.496 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:06:58.496 DEBUG:boto:Port: 8001 2023-07-29T00:06:58.496 DEBUG:boto:Params: {} 2023-07-29T00:06:58.497 DEBUG:boto:Token: None 2023-07-29T00:06:58.497 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/obj4?versionId=null 2023-07-29T00:06:58.498 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:k5sFs+fThcdVoH7rrsdbranRLv0= 2023-07-29T00:06:58.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:k5sFs+fThcdVoH7rrsdbranRLv0=', 'Content-Length': '0'} 2023-07-29T00:06:58.500 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000801e1d25f1a0ec4b-0064c45822-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.540 DEBUG:boto:path=/hiiqos-47/obj4 2023-07-29T00:06:58.541 DEBUG:boto:auth_path=/hiiqos-47/obj4 2023-07-29T00:06:58.541 DEBUG:boto:path=/hiiqos-47/obj4?versionId=null 2023-07-29T00:06:58.542 DEBUG:boto:auth_path=/hiiqos-47/obj4?versionId=null 2023-07-29T00:06:58.542 DEBUG:boto:Method: GET 2023-07-29T00:06:58.542 DEBUG:boto:Path: /hiiqos-47/obj4?versionId=null 2023-07-29T00:06:58.543 DEBUG:boto:Data: 2023-07-29T00:06:58.543 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.543 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:06:58.544 DEBUG:boto:Port: 8000 2023-07-29T00:06:58.544 DEBUG:boto:Params: {} 2023-07-29T00:06:58.545 DEBUG:boto:Token: None 2023-07-29T00:06:58.545 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/obj4?versionId=null 2023-07-29T00:06:58.546 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:k5sFs+fThcdVoH7rrsdbranRLv0= 2023-07-29T00:06:58.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:k5sFs+fThcdVoH7rrsdbranRLv0=', 'Content-Length': '0'} 2023-07-29T00:06:58.548 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f610faf4515776b6-0064c45822-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.549 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:06:58.549 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:06:58.549 DEBUG:boto:path=/hiiqos-47/?prefix=obj4 2023-07-29T00:06:58.550 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj4 2023-07-29T00:06:58.550 DEBUG:boto:Method: GET 2023-07-29T00:06:58.551 DEBUG:boto:Path: /hiiqos-47/?prefix=obj4 2023-07-29T00:06:58.551 DEBUG:boto:Data: 2023-07-29T00:06:58.551 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.552 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:06:58.552 DEBUG:boto:Port: 8001 2023-07-29T00:06:58.552 DEBUG:boto:Params: {} 2023-07-29T00:06:58.553 DEBUG:boto:Token: None 2023-07-29T00:06:58.553 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/ 2023-07-29T00:06:58.554 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Er1nKg0TJRi24zQ4hUNK7zA+JV4= 2023-07-29T00:06:58.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Er1nKg0TJRi24zQ4hUNK7zA+JV4=', 'Content-Length': '0'} 2023-07-29T00:06:58.556 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095bde3cd7478670a-0064c45822-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.557 DEBUG:boto:b'hiiqos-47obj41000falseobj42023-07-29T00:05:16.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:06:58.557 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:06:58.558 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:06:58.558 DEBUG:boto:path=/hiiqos-47/?prefix=obj4 2023-07-29T00:06:58.559 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj4 2023-07-29T00:06:58.559 DEBUG:boto:Method: GET 2023-07-29T00:06:58.559 DEBUG:boto:Path: /hiiqos-47/?prefix=obj4 2023-07-29T00:06:58.560 DEBUG:boto:Data: 2023-07-29T00:06:58.560 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.560 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:06:58.561 DEBUG:boto:Port: 8000 2023-07-29T00:06:58.561 DEBUG:boto:Params: {} 2023-07-29T00:06:58.561 DEBUG:boto:Token: None 2023-07-29T00:06:58.562 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/ 2023-07-29T00:06:58.562 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Er1nKg0TJRi24zQ4hUNK7zA+JV4= 2023-07-29T00:06:58.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Er1nKg0TJRi24zQ4hUNK7zA+JV4=', 'Content-Length': '0'} 2023-07-29T00:06:58.565 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096d31d5d210f7a21-0064c45822-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.566 DEBUG:boto:b'hiiqos-47obj41000falseobj42023-07-29T00:05:16.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:06:58.566 DEBUG:boto:path=/hiiqos-47/obj4 2023-07-29T00:06:58.567 DEBUG:boto:auth_path=/hiiqos-47/obj4 2023-07-29T00:06:58.567 DEBUG:boto:Method: HEAD 2023-07-29T00:06:58.568 DEBUG:boto:Path: /hiiqos-47/obj4 2023-07-29T00:06:58.568 DEBUG:boto:Data: 2023-07-29T00:06:58.568 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.569 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:06:58.569 DEBUG:boto:Port: 8001 2023-07-29T00:06:58.569 DEBUG:boto:Params: {} 2023-07-29T00:06:58.570 DEBUG:boto:Token: None 2023-07-29T00:06:58.570 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/obj4 2023-07-29T00:06:58.570 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DTSbrqRmr1/NGwuDLPiMW4n9fqY= 2023-07-29T00:06:58.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DTSbrqRmr1/NGwuDLPiMW4n9fqY=', 'Content-Length': '0'} 2023-07-29T00:06:58.572 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000089e557e8384bc8e-0064c45822-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.573 DEBUG:boto:path=/hiiqos-47/obj4 2023-07-29T00:06:58.573 DEBUG:boto:auth_path=/hiiqos-47/obj4 2023-07-29T00:06:58.574 DEBUG:boto:Method: HEAD 2023-07-29T00:06:58.574 DEBUG:boto:Path: /hiiqos-47/obj4 2023-07-29T00:06:58.574 DEBUG:boto:Data: 2023-07-29T00:06:58.575 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.575 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:06:58.575 DEBUG:boto:Port: 8000 2023-07-29T00:06:58.576 DEBUG:boto:Params: {} 2023-07-29T00:06:58.576 DEBUG:boto:Token: None 2023-07-29T00:06:58.577 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/obj4 2023-07-29T00:06:58.577 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DTSbrqRmr1/NGwuDLPiMW4n9fqY= 2023-07-29T00:06:58.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DTSbrqRmr1/NGwuDLPiMW4n9fqY=', 'Content-Length': '0'} 2023-07-29T00:06:58.579 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bcab6aac4125ea14-0064c45822-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-07-29T00:06:58.580 DEBUG:boto:path=/hiiqos-47/obj4 2023-07-29T00:06:58.581 DEBUG:boto:auth_path=/hiiqos-47/obj4 2023-07-29T00:06:58.581 DEBUG:boto:Method: GET 2023-07-29T00:06:58.581 DEBUG:boto:Path: /hiiqos-47/obj4 2023-07-29T00:06:58.582 DEBUG:boto:Data: 2023-07-29T00:06:58.582 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.582 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:06:58.583 DEBUG:boto:Port: 8001 2023-07-29T00:06:58.583 DEBUG:boto:Params: {} 2023-07-29T00:06:58.584 DEBUG:boto:Token: None 2023-07-29T00:06:58.584 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/obj4 2023-07-29T00:06:58.584 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LouhlLl61PXd+rtWtcjR424K1nU= 2023-07-29T00:06:58.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LouhlLl61PXd+rtWtcjR424K1nU=', 'Content-Length': '0'} 2023-07-29T00:06:58.587 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b959f51c1c713b0d-0064c45822-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.628 DEBUG:boto:path=/hiiqos-47/obj4 2023-07-29T00:06:58.629 DEBUG:boto:auth_path=/hiiqos-47/obj4 2023-07-29T00:06:58.630 DEBUG:boto:Method: GET 2023-07-29T00:06:58.630 DEBUG:boto:Path: /hiiqos-47/obj4 2023-07-29T00:06:58.631 DEBUG:boto:Data: 2023-07-29T00:06:58.631 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.631 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:06:58.632 DEBUG:boto:Port: 8000 2023-07-29T00:06:58.632 DEBUG:boto:Params: {} 2023-07-29T00:06:58.633 DEBUG:boto:Token: None 2023-07-29T00:06:58.633 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/obj4 2023-07-29T00:06:58.634 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LouhlLl61PXd+rtWtcjR424K1nU= 2023-07-29T00:06:58.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LouhlLl61PXd+rtWtcjR424K1nU=', 'Content-Length': '0'} 2023-07-29T00:06:58.636 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c0c7c505f3a755f-0064c45822-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.637 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:06:58.637 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:06:58.638 DEBUG:boto:path=/hiiqos-47/?prefix=obj4 2023-07-29T00:06:58.638 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj4 2023-07-29T00:06:58.639 DEBUG:boto:Method: GET 2023-07-29T00:06:58.639 DEBUG:boto:Path: /hiiqos-47/?prefix=obj4 2023-07-29T00:06:58.639 DEBUG:boto:Data: 2023-07-29T00:06:58.640 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.640 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:06:58.640 DEBUG:boto:Port: 8001 2023-07-29T00:06:58.641 DEBUG:boto:Params: {} 2023-07-29T00:06:58.641 DEBUG:boto:Token: None 2023-07-29T00:06:58.642 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/ 2023-07-29T00:06:58.642 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Er1nKg0TJRi24zQ4hUNK7zA+JV4= 2023-07-29T00:06:58.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Er1nKg0TJRi24zQ4hUNK7zA+JV4=', 'Content-Length': '0'} 2023-07-29T00:06:58.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c8e21127c036c61-0064c45822-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.646 DEBUG:boto:b'hiiqos-47obj41000falseobj42023-07-29T00:05:16.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:06:58.647 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:06:58.647 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:06:58.647 DEBUG:boto:path=/hiiqos-47/?prefix=obj4 2023-07-29T00:06:58.648 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj4 2023-07-29T00:06:58.648 DEBUG:boto:Method: GET 2023-07-29T00:06:58.649 DEBUG:boto:Path: /hiiqos-47/?prefix=obj4 2023-07-29T00:06:58.649 DEBUG:boto:Data: 2023-07-29T00:06:58.649 DEBUG:boto:Headers: {} 2023-07-29T00:06:58.650 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:06:58.650 DEBUG:boto:Port: 8000 2023-07-29T00:06:58.651 DEBUG:boto:Params: {} 2023-07-29T00:06:58.651 DEBUG:boto:Token: None 2023-07-29T00:06:58.652 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:06:58 GMT /hiiqos-47/ 2023-07-29T00:06:58.652 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Er1nKg0TJRi24zQ4hUNK7zA+JV4= 2023-07-29T00:06:58.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Er1nKg0TJRi24zQ4hUNK7zA+JV4=', 'Content-Length': '0'} 2023-07-29T00:06:58.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d634c70eed21a38b-0064c45822-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:06:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:58.656 DEBUG:boto:b'hiiqos-47obj41000falseobj42023-07-29T00:05:16.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:06:58.657 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-47 zones={a2, a1} 2023-07-29T00:06:58.658 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'hiiqos-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:06:58.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'hiiqos-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:06:58.659 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket hiiqos-46 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:06:58.714 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:06:58.715 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:06:58.782 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'hiiqos-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:06:58.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'hiiqos-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:06:58.783 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket hiiqos-46 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:06:58.830 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:06:58.831 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:06:58.894 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'hiiqos-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:06:58.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'hiiqos-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:06:58.895 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket hiiqos-47 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:06:58.941 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:06:58.941 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:06:59.003 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'hiiqos-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:06:59.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'hiiqos-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:06:59.004 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket hiiqos-47 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:06:59.051 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:06:59.051 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:06:59.114 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:06:59.115 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:06:59.115 DEBUG:boto:Method: HEAD 2023-07-29T00:06:59.116 DEBUG:boto:Path: /hiiqos-46/ 2023-07-29T00:06:59.117 DEBUG:boto:Data: 2023-07-29T00:06:59.118 DEBUG:boto:Headers: {} 2023-07-29T00:06:59.118 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:06:59.119 DEBUG:boto:Port: 8000 2023-07-29T00:06:59.120 DEBUG:boto:Params: {} 2023-07-29T00:06:59.121 DEBUG:boto:Token: None 2023-07-29T00:06:59.122 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:06:59 GMT /hiiqos-46/ 2023-07-29T00:06:59.123 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:E3NTWyaBU47KanS9YzWlQHK8Yf4= 2023-07-29T00:06:59.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:E3NTWyaBU47KanS9YzWlQHK8Yf4=', 'Content-Length': '0'} 2023-07-29T00:06:59.126 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f926e86e6e338d24-0064c45823-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:59.127 DEBUG:boto:path=/hiiqos-46/obj5 2023-07-29T00:06:59.127 DEBUG:boto:auth_path=/hiiqos-46/obj5 2023-07-29T00:06:59.127 DEBUG:boto:Method: PUT 2023-07-29T00:06:59.128 DEBUG:boto:Path: /hiiqos-46/obj5 2023-07-29T00:06:59.128 DEBUG:boto:Data: 2023-07-29T00:06:59.129 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T00:06:59.130 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:06:59.130 DEBUG:boto:Port: 8000 2023-07-29T00:06:59.130 DEBUG:boto:Params: {} 2023-07-29T00:06:59.131 DEBUG:boto:Token: None 2023-07-29T00:06:59.131 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 00:06:59 GMT /hiiqos-46/obj5 2023-07-29T00:06:59.132 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:euyr8XX9WgBvAuhiUuB1NlE4ZaY= 2023-07-29T00:06:59.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:06:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:euyr8XX9WgBvAuhiUuB1NlE4ZaY='} 2023-07-29T00:06:59.137 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007f47694ad5879b7a-0064c45823-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:59.137 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:06:59.138 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:06:59.138 DEBUG:boto:Method: HEAD 2023-07-29T00:06:59.139 DEBUG:boto:Path: /hiiqos-46/ 2023-07-29T00:06:59.139 DEBUG:boto:Data: 2023-07-29T00:06:59.140 DEBUG:boto:Headers: {} 2023-07-29T00:06:59.140 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:06:59.140 DEBUG:boto:Port: 8000 2023-07-29T00:06:59.141 DEBUG:boto:Params: {} 2023-07-29T00:06:59.141 DEBUG:boto:Token: None 2023-07-29T00:06:59.142 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:06:59 GMT /hiiqos-46/ 2023-07-29T00:06:59.142 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:E3NTWyaBU47KanS9YzWlQHK8Yf4= 2023-07-29T00:06:59.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:E3NTWyaBU47KanS9YzWlQHK8Yf4=', 'Content-Length': '0'} 2023-07-29T00:06:59.145 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000100d1921d7a5fbf1-0064c45823-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:59.146 DEBUG:boto:path=/hiiqos-46/obj6 2023-07-29T00:06:59.146 DEBUG:boto:auth_path=/hiiqos-46/obj6 2023-07-29T00:06:59.146 DEBUG:boto:Method: PUT 2023-07-29T00:06:59.147 DEBUG:boto:Path: /hiiqos-46/obj6 2023-07-29T00:06:59.147 DEBUG:boto:Data: 2023-07-29T00:06:59.148 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T00:06:59.148 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:06:59.148 DEBUG:boto:Port: 8000 2023-07-29T00:06:59.149 DEBUG:boto:Params: {} 2023-07-29T00:06:59.149 DEBUG:boto:Token: None 2023-07-29T00:06:59.150 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 00:06:59 GMT /hiiqos-46/obj6 2023-07-29T00:06:59.151 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:blVRlNXqUHf8busedZDHWkq5eXQ= 2023-07-29T00:06:59.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:06:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:blVRlNXqUHf8busedZDHWkq5eXQ='} 2023-07-29T00:06:59.156 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c09717b27cd0457c-0064c45823-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:59.156 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:06:59.157 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:06:59.157 DEBUG:boto:Method: HEAD 2023-07-29T00:06:59.157 DEBUG:boto:Path: /hiiqos-46/ 2023-07-29T00:06:59.158 DEBUG:boto:Data: 2023-07-29T00:06:59.158 DEBUG:boto:Headers: {} 2023-07-29T00:06:59.158 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:06:59.159 DEBUG:boto:Port: 8000 2023-07-29T00:06:59.159 DEBUG:boto:Params: {} 2023-07-29T00:06:59.160 DEBUG:boto:Token: None 2023-07-29T00:06:59.160 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:06:59 GMT /hiiqos-46/ 2023-07-29T00:06:59.161 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:E3NTWyaBU47KanS9YzWlQHK8Yf4= 2023-07-29T00:06:59.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:E3NTWyaBU47KanS9YzWlQHK8Yf4=', 'Content-Length': '0'} 2023-07-29T00:06:59.163 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007997a27c3dfc7274-0064c45823-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:59.164 DEBUG:boto:path=/hiiqos-46/obj7 2023-07-29T00:06:59.164 DEBUG:boto:auth_path=/hiiqos-46/obj7 2023-07-29T00:06:59.165 DEBUG:boto:Method: PUT 2023-07-29T00:06:59.165 DEBUG:boto:Path: /hiiqos-46/obj7 2023-07-29T00:06:59.166 DEBUG:boto:Data: 2023-07-29T00:06:59.166 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T00:06:59.166 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:06:59.167 DEBUG:boto:Port: 8000 2023-07-29T00:06:59.167 DEBUG:boto:Params: {} 2023-07-29T00:06:59.167 DEBUG:boto:Token: None 2023-07-29T00:06:59.168 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 00:06:59 GMT /hiiqos-46/obj7 2023-07-29T00:06:59.168 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dnoTxmCqnYOgZS8yFdPTQivXLKY= 2023-07-29T00:06:59.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:06:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dnoTxmCqnYOgZS8yFdPTQivXLKY='} 2023-07-29T00:06:59.212 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007e86d28490b4b379-0064c45823-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:59.213 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:06:59.213 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:06:59.214 DEBUG:boto:Method: HEAD 2023-07-29T00:06:59.214 DEBUG:boto:Path: /hiiqos-46/ 2023-07-29T00:06:59.215 DEBUG:boto:Data: 2023-07-29T00:06:59.215 DEBUG:boto:Headers: {} 2023-07-29T00:06:59.215 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:06:59.216 DEBUG:boto:Port: 8000 2023-07-29T00:06:59.216 DEBUG:boto:Params: {} 2023-07-29T00:06:59.216 DEBUG:boto:Token: None 2023-07-29T00:06:59.217 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:06:59 GMT /hiiqos-46/ 2023-07-29T00:06:59.217 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:E3NTWyaBU47KanS9YzWlQHK8Yf4= 2023-07-29T00:06:59.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:E3NTWyaBU47KanS9YzWlQHK8Yf4=', 'Content-Length': '0'} 2023-07-29T00:06:59.220 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e6d1b00c05b6c65d-0064c45823-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:59.220 DEBUG:boto:path=/hiiqos-46/obj8 2023-07-29T00:06:59.221 DEBUG:boto:auth_path=/hiiqos-46/obj8 2023-07-29T00:06:59.221 DEBUG:boto:Method: PUT 2023-07-29T00:06:59.222 DEBUG:boto:Path: /hiiqos-46/obj8 2023-07-29T00:06:59.222 DEBUG:boto:Data: 2023-07-29T00:06:59.222 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T00:06:59.223 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:06:59.223 DEBUG:boto:Port: 8000 2023-07-29T00:06:59.223 DEBUG:boto:Params: {} 2023-07-29T00:06:59.224 DEBUG:boto:Token: None 2023-07-29T00:06:59.224 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 00:06:59 GMT /hiiqos-46/obj8 2023-07-29T00:06:59.225 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:h9cX0Y1/n5SJzaUyo6x7RjopauQ= 2023-07-29T00:06:59.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:06:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:h9cX0Y1/n5SJzaUyo6x7RjopauQ='} 2023-07-29T00:06:59.230 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cc4cbda4d33da2be-0064c45823-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:59.231 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:06:59.232 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:06:59.232 DEBUG:boto:Method: HEAD 2023-07-29T00:06:59.233 DEBUG:boto:Path: /hiiqos-47/ 2023-07-29T00:06:59.233 DEBUG:boto:Data: 2023-07-29T00:06:59.234 DEBUG:boto:Headers: {} 2023-07-29T00:06:59.234 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:06:59.234 DEBUG:boto:Port: 8001 2023-07-29T00:06:59.235 DEBUG:boto:Params: {} 2023-07-29T00:06:59.235 DEBUG:boto:Token: None 2023-07-29T00:06:59.236 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:06:59 GMT /hiiqos-47/ 2023-07-29T00:06:59.236 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5wjcC/JY4PmqZJkgzxOK0nFRGz4= 2023-07-29T00:06:59.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5wjcC/JY4PmqZJkgzxOK0nFRGz4=', 'Content-Length': '0'} 2023-07-29T00:06:59.239 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000824735c8191f83d-0064c45823-1663-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:59.239 DEBUG:boto:path=/hiiqos-47/obj5 2023-07-29T00:06:59.240 DEBUG:boto:auth_path=/hiiqos-47/obj5 2023-07-29T00:06:59.240 DEBUG:boto:Method: PUT 2023-07-29T00:06:59.240 DEBUG:boto:Path: /hiiqos-47/obj5 2023-07-29T00:06:59.241 DEBUG:boto:Data: 2023-07-29T00:06:59.242 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T00:06:59.242 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:06:59.242 DEBUG:boto:Port: 8001 2023-07-29T00:06:59.243 DEBUG:boto:Params: {} 2023-07-29T00:06:59.243 DEBUG:boto:Token: None 2023-07-29T00:06:59.244 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 00:06:59 GMT /hiiqos-47/obj5 2023-07-29T00:06:59.244 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J2GcOwkNBf5ixyt0u6DKmi+SkZo= 2023-07-29T00:06:59.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:06:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J2GcOwkNBf5ixyt0u6DKmi+SkZo='} 2023-07-29T00:06:59.251 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001d9465787cd25e52-0064c45823-1663-a2'), ('Date', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:59.252 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:06:59.252 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:06:59.253 DEBUG:boto:Method: HEAD 2023-07-29T00:06:59.254 DEBUG:boto:Path: /hiiqos-47/ 2023-07-29T00:06:59.254 DEBUG:boto:Data: 2023-07-29T00:06:59.255 DEBUG:boto:Headers: {} 2023-07-29T00:06:59.255 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:06:59.256 DEBUG:boto:Port: 8001 2023-07-29T00:06:59.256 DEBUG:boto:Params: {} 2023-07-29T00:06:59.256 DEBUG:boto:Token: None 2023-07-29T00:06:59.257 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:06:59 GMT /hiiqos-47/ 2023-07-29T00:06:59.257 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5wjcC/JY4PmqZJkgzxOK0nFRGz4= 2023-07-29T00:06:59.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5wjcC/JY4PmqZJkgzxOK0nFRGz4=', 'Content-Length': '0'} 2023-07-29T00:06:59.259 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000060f23e297bd69e88-0064c45823-1663-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:59.260 DEBUG:boto:path=/hiiqos-47/obj6 2023-07-29T00:06:59.261 DEBUG:boto:auth_path=/hiiqos-47/obj6 2023-07-29T00:06:59.261 DEBUG:boto:Method: PUT 2023-07-29T00:06:59.262 DEBUG:boto:Path: /hiiqos-47/obj6 2023-07-29T00:06:59.262 DEBUG:boto:Data: 2023-07-29T00:06:59.262 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T00:06:59.263 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:06:59.263 DEBUG:boto:Port: 8001 2023-07-29T00:06:59.263 DEBUG:boto:Params: {} 2023-07-29T00:06:59.264 DEBUG:boto:Token: None 2023-07-29T00:06:59.264 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 00:06:59 GMT /hiiqos-47/obj6 2023-07-29T00:06:59.265 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eN3au1CNOgQL7692TiefxexK1IE= 2023-07-29T00:06:59.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:06:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eN3au1CNOgQL7692TiefxexK1IE='} 2023-07-29T00:06:59.270 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000442f25fbd611f9bb-0064c45823-1663-a2'), ('Date', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:59.270 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:06:59.271 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:06:59.271 DEBUG:boto:Method: HEAD 2023-07-29T00:06:59.272 DEBUG:boto:Path: /hiiqos-47/ 2023-07-29T00:06:59.272 DEBUG:boto:Data: 2023-07-29T00:06:59.272 DEBUG:boto:Headers: {} 2023-07-29T00:06:59.273 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:06:59.273 DEBUG:boto:Port: 8001 2023-07-29T00:06:59.273 DEBUG:boto:Params: {} 2023-07-29T00:06:59.274 DEBUG:boto:Token: None 2023-07-29T00:06:59.274 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:06:59 GMT /hiiqos-47/ 2023-07-29T00:06:59.275 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5wjcC/JY4PmqZJkgzxOK0nFRGz4= 2023-07-29T00:06:59.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5wjcC/JY4PmqZJkgzxOK0nFRGz4=', 'Content-Length': '0'} 2023-07-29T00:06:59.277 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c47926f549f7fbd6-0064c45823-1663-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:59.277 DEBUG:boto:path=/hiiqos-47/obj7 2023-07-29T00:06:59.278 DEBUG:boto:auth_path=/hiiqos-47/obj7 2023-07-29T00:06:59.279 DEBUG:boto:Method: PUT 2023-07-29T00:06:59.279 DEBUG:boto:Path: /hiiqos-47/obj7 2023-07-29T00:06:59.279 DEBUG:boto:Data: 2023-07-29T00:06:59.280 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T00:06:59.280 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:06:59.280 DEBUG:boto:Port: 8001 2023-07-29T00:06:59.281 DEBUG:boto:Params: {} 2023-07-29T00:06:59.281 DEBUG:boto:Token: None 2023-07-29T00:06:59.281 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 00:06:59 GMT /hiiqos-47/obj7 2023-07-29T00:06:59.282 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+jKmCyS2joWH1k+Yum7tMzLUdg4= 2023-07-29T00:06:59.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:06:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+jKmCyS2joWH1k+Yum7tMzLUdg4='} 2023-07-29T00:06:59.328 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000064768ff1b5efdf99-0064c45823-1663-a2'), ('Date', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:59.329 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:06:59.330 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:06:59.330 DEBUG:boto:Method: HEAD 2023-07-29T00:06:59.330 DEBUG:boto:Path: /hiiqos-47/ 2023-07-29T00:06:59.331 DEBUG:boto:Data: 2023-07-29T00:06:59.331 DEBUG:boto:Headers: {} 2023-07-29T00:06:59.332 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:06:59.332 DEBUG:boto:Port: 8001 2023-07-29T00:06:59.332 DEBUG:boto:Params: {} 2023-07-29T00:06:59.333 DEBUG:boto:Token: None 2023-07-29T00:06:59.333 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:06:59 GMT /hiiqos-47/ 2023-07-29T00:06:59.333 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5wjcC/JY4PmqZJkgzxOK0nFRGz4= 2023-07-29T00:06:59.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:06:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5wjcC/JY4PmqZJkgzxOK0nFRGz4=', 'Content-Length': '0'} 2023-07-29T00:06:59.336 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000491b7b0db1eacbfc-0064c45823-1663-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:59.337 DEBUG:boto:path=/hiiqos-47/obj8 2023-07-29T00:06:59.337 DEBUG:boto:auth_path=/hiiqos-47/obj8 2023-07-29T00:06:59.337 DEBUG:boto:Method: PUT 2023-07-29T00:06:59.338 DEBUG:boto:Path: /hiiqos-47/obj8 2023-07-29T00:06:59.338 DEBUG:boto:Data: 2023-07-29T00:06:59.338 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T00:06:59.339 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:06:59.339 DEBUG:boto:Port: 8001 2023-07-29T00:06:59.340 DEBUG:boto:Params: {} 2023-07-29T00:06:59.340 DEBUG:boto:Token: None 2023-07-29T00:06:59.341 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 00:06:59 GMT /hiiqos-47/obj8 2023-07-29T00:06:59.341 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HzX8iyw+9JAEZjJOI+BfEKcyE2U= 2023-07-29T00:06:59.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:06:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HzX8iyw+9JAEZjJOI+BfEKcyE2U='} 2023-07-29T00:06:59.346 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008823decb35629edd-0064c45823-1663-a2'), ('Date', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:06:59.348 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-46 2023-07-29T00:06:59.348 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-46', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:06:59.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-46', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:06:59.349 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-46 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:06:59.402 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:06:59.403 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:06:59.434 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:06:59.409+0000 7f304d410880 1 waiting for incremental sync to catch up: 2023-07-29T00:06:59.435 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000001.22.6, , 00000000001.27.6, , 00000000001.26.6, , , 00000000001.22.6, , , ] 2023-07-29T00:06:59.435 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000004.30.6, 00000000002.33.1, 00000000004.31.6, 00000000002.26.1, 00000000004.30.6, 00000000002.20.1, 00000000002.19.1, 00000000004.26.6, 00000000002.17.1, 00000000002.15.1, 00000000002.40.1] 2023-07-29T00:07:04.436 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:07:04.410+0000 7f304d410880 1 waiting for incremental sync to catch up: 2023-07-29T00:07:04.436 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000001.22.6, , 00000000001.27.6, , 00000000001.26.6, , , 00000000001.22.6, , , ] 2023-07-29T00:07:04.437 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000004.30.6, 00000000002.33.1, 00000000004.31.6, 00000000002.26.1, 00000000004.30.6, 00000000002.20.1, 00000000002.19.1, 00000000004.26.6, 00000000002.17.1, 00000000002.15.1, 00000000002.40.1] 2023-07-29T00:07:09.437 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:07:09.411+0000 7f304d410880 1 waiting for incremental sync to catch up: 2023-07-29T00:07:09.438 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000001.22.6, , 00000000001.27.6, , 00000000001.26.6, , , 00000000001.22.6, , , ] 2023-07-29T00:07:09.438 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000004.30.6, 00000000002.33.1, 00000000004.31.6, 00000000002.26.1, 00000000004.30.6, 00000000002.20.1, 00000000002.19.1, 00000000004.26.6, 00000000002.17.1, 00000000002.15.1, 00000000002.40.1] 2023-07-29T00:07:14.439 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:07:14.412+0000 7f304d410880 1 bucket sync caught up with source: 2023-07-29T00:07:14.439 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000004.30.6, 00000000002.33.1, 00000000004.31.6, 00000000002.26.1, 00000000004.30.6, 00000000002.20.1, 00000000002.19.1, 00000000004.26.6, 00000000002.17.1, 00000000002.15.1, 00000000002.40.1] 2023-07-29T00:07:14.439 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000004.30.6, 00000000002.33.1, 00000000004.31.6, 00000000002.26.1, 00000000004.30.6, 00000000002.20.1, 00000000002.19.1, 00000000004.26.6, 00000000002.17.1, 00000000002.15.1, 00000000002.40.1] 2023-07-29T00:07:14.440 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:07:14.412+0000 7f304d410880 0 bucket checkpoint complete 2023-07-29T00:07:14.455 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-46 2023-07-29T00:07:14.456 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-46', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:07:14.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-46', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:07:14.457 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-46 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:07:14.516 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:07:14.516 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:07:14.551 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:07:14.525+0000 7f44e900f880 1 waiting for incremental sync to catch up: 2023-07-29T00:07:14.552 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000001.1166.6, , 00000000001.864.6, , 00000000001.526.6, , , 00000000001.372.6, , , ] 2023-07-29T00:07:14.552 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000002.1168.6, , 00000000002.866.6, , 00000000002.528.6, , , 00000000002.374.6, , , ] 2023-07-29T00:07:19.553 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:07:19.526+0000 7f44e900f880 1 bucket sync caught up with source: 2023-07-29T00:07:19.554 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000002.1168.6, , 00000000002.866.6, , 00000000002.528.6, , , 00000000002.374.6, , , ] 2023-07-29T00:07:19.554 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000002.1168.6, , 00000000002.866.6, , 00000000002.528.6, , , 00000000002.374.6, , , ] 2023-07-29T00:07:19.554 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:07:19.526+0000 7f44e900f880 0 bucket checkpoint complete 2023-07-29T00:07:19.569 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-46 zones={a2, a1} 2023-07-29T00:07:19.570 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:07:19.570 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:07:19.571 DEBUG:boto:Method: HEAD 2023-07-29T00:07:19.571 DEBUG:boto:Path: /hiiqos-46/ 2023-07-29T00:07:19.571 DEBUG:boto:Data: 2023-07-29T00:07:19.572 DEBUG:boto:Headers: {} 2023-07-29T00:07:19.572 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:19.572 DEBUG:boto:Port: 8001 2023-07-29T00:07:19.573 DEBUG:boto:Params: {} 2023-07-29T00:07:19.573 DEBUG:boto:Token: None 2023-07-29T00:07:19.574 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:07:19 GMT /hiiqos-46/ 2023-07-29T00:07:19.574 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:W3cRjNPfqBPHQDPOTd5V1/enhCc= 2023-07-29T00:07:19.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:W3cRjNPfqBPHQDPOTd5V1/enhCc=', 'Content-Length': '0'} 2023-07-29T00:07:19.577 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008ff6a49c538a09f3-0064c45837-1663-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:07:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:19.578 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:07:19.578 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:07:19.578 DEBUG:boto:Method: HEAD 2023-07-29T00:07:19.579 DEBUG:boto:Path: /hiiqos-46/ 2023-07-29T00:07:19.579 DEBUG:boto:Data: 2023-07-29T00:07:19.579 DEBUG:boto:Headers: {} 2023-07-29T00:07:19.580 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:19.580 DEBUG:boto:Port: 8000 2023-07-29T00:07:19.580 DEBUG:boto:Params: {} 2023-07-29T00:07:19.581 DEBUG:boto:Token: None 2023-07-29T00:07:19.581 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:07:19 GMT /hiiqos-46/ 2023-07-29T00:07:19.582 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:W3cRjNPfqBPHQDPOTd5V1/enhCc= 2023-07-29T00:07:19.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:W3cRjNPfqBPHQDPOTd5V1/enhCc=', 'Content-Length': '0'} 2023-07-29T00:07:19.585 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006ecddb49deb9fb4a-0064c45837-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:07:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:19.586 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-29T00:07:19.587 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:07:19.588 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:07:19.589 DEBUG:boto:path=/hiiqos-46/?versions&max-keys=999 2023-07-29T00:07:19.589 DEBUG:boto:auth_path=/hiiqos-46/?versions&max-keys=999 2023-07-29T00:07:19.590 DEBUG:boto:Method: GET 2023-07-29T00:07:19.590 DEBUG:boto:Path: /hiiqos-46/?versions&max-keys=999 2023-07-29T00:07:19.591 DEBUG:boto:Data: 2023-07-29T00:07:19.592 DEBUG:boto:Headers: {} 2023-07-29T00:07:19.592 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:19.592 DEBUG:boto:Port: 8001 2023-07-29T00:07:19.593 DEBUG:boto:Params: {} 2023-07-29T00:07:19.593 DEBUG:boto:Token: None 2023-07-29T00:07:19.594 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:19 GMT /hiiqos-46/?versions 2023-07-29T00:07:19.594 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3I/7CXz8aDuftJGvryYrYlMre8I= 2023-07-29T00:07:19.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3I/7CXz8aDuftJGvryYrYlMre8I=', 'Content-Length': '0'} 2023-07-29T00:07:19.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0f5edae5fb97096-0064c45837-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:07:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:19.597 DEBUG:boto:b'hiiqos-46999falseobj1nulltrue2023-07-29T00:05:16.017Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-07-29T00:05:16.033Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-07-29T00:05:16.052Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-07-29T00:05:16.112Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-07-29T00:06:59.109Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-07-29T00:06:59.128Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-07-29T00:06:59.145Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-07-29T00:06:59.202Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:07:19.599 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-07-29T00:07:19.599 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-07-29T00:07:19.599 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-07-29T00:07:19.600 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-07-29T00:07:19.600 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-07-29T00:07:19.601 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-07-29T00:07:19.601 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-07-29T00:07:19.601 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-07-29T00:07:19.602 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-29T00:07:19.603 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:07:19.604 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:07:19.604 DEBUG:boto:path=/hiiqos-46/?versions&max-keys=999 2023-07-29T00:07:19.605 DEBUG:boto:auth_path=/hiiqos-46/?versions&max-keys=999 2023-07-29T00:07:19.606 DEBUG:boto:Method: GET 2023-07-29T00:07:19.607 DEBUG:boto:Path: /hiiqos-46/?versions&max-keys=999 2023-07-29T00:07:19.608 DEBUG:boto:Data: 2023-07-29T00:07:19.609 DEBUG:boto:Headers: {} 2023-07-29T00:07:19.609 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:19.610 DEBUG:boto:Port: 8000 2023-07-29T00:07:19.610 DEBUG:boto:Params: {} 2023-07-29T00:07:19.611 DEBUG:boto:Token: None 2023-07-29T00:07:19.612 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:19 GMT /hiiqos-46/?versions 2023-07-29T00:07:19.612 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3I/7CXz8aDuftJGvryYrYlMre8I= 2023-07-29T00:07:19.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3I/7CXz8aDuftJGvryYrYlMre8I=', 'Content-Length': '0'} 2023-07-29T00:07:19.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000243dc87917728672-0064c45837-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:07:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:19.616 DEBUG:boto:b'hiiqos-46999falseobj1nulltrue2023-07-29T00:05:16.017Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-07-29T00:05:16.033Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-07-29T00:05:16.052Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-07-29T00:05:16.112Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-07-29T00:06:59.109Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-07-29T00:06:59.128Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-07-29T00:06:59.145Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-07-29T00:06:59.202Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:07:19.617 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-07-29T00:07:19.617 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-07-29T00:07:19.618 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-07-29T00:07:19.618 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-07-29T00:07:19.618 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-07-29T00:07:19.618 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-07-29T00:07:19.619 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-07-29T00:07:19.619 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-07-29T00:07:19.619 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:07:19.620 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:07:19.620 DEBUG:boto:path=/hiiqos-46/?versions&max-keys=999 2023-07-29T00:07:19.621 DEBUG:boto:auth_path=/hiiqos-46/?versions&max-keys=999 2023-07-29T00:07:19.621 DEBUG:boto:Method: GET 2023-07-29T00:07:19.621 DEBUG:boto:Path: /hiiqos-46/?versions&max-keys=999 2023-07-29T00:07:19.622 DEBUG:boto:Data: 2023-07-29T00:07:19.622 DEBUG:boto:Headers: {} 2023-07-29T00:07:19.622 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:19.623 DEBUG:boto:Port: 8001 2023-07-29T00:07:19.623 DEBUG:boto:Params: {} 2023-07-29T00:07:19.623 DEBUG:boto:Token: None 2023-07-29T00:07:19.624 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:19 GMT /hiiqos-46/?versions 2023-07-29T00:07:19.624 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3I/7CXz8aDuftJGvryYrYlMre8I= 2023-07-29T00:07:19.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3I/7CXz8aDuftJGvryYrYlMre8I=', 'Content-Length': '0'} 2023-07-29T00:07:19.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ad790e8f1776b1f-0064c45837-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:07:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:19.628 DEBUG:boto:b'hiiqos-46999falseobj1nulltrue2023-07-29T00:05:16.017Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-07-29T00:05:16.033Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-07-29T00:05:16.052Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-07-29T00:05:16.112Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-07-29T00:06:59.109Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-07-29T00:06:59.128Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-07-29T00:06:59.145Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-07-29T00:06:59.202Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:07:19.629 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:07:19.629 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:07:19.629 DEBUG:boto:path=/hiiqos-46/?versions&max-keys=999 2023-07-29T00:07:19.630 DEBUG:boto:auth_path=/hiiqos-46/?versions&max-keys=999 2023-07-29T00:07:19.630 DEBUG:boto:Method: GET 2023-07-29T00:07:19.630 DEBUG:boto:Path: /hiiqos-46/?versions&max-keys=999 2023-07-29T00:07:19.631 DEBUG:boto:Data: 2023-07-29T00:07:19.631 DEBUG:boto:Headers: {} 2023-07-29T00:07:19.631 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:19.632 DEBUG:boto:Port: 8000 2023-07-29T00:07:19.632 DEBUG:boto:Params: {} 2023-07-29T00:07:19.632 DEBUG:boto:Token: None 2023-07-29T00:07:19.633 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:19 GMT /hiiqos-46/?versions 2023-07-29T00:07:19.633 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3I/7CXz8aDuftJGvryYrYlMre8I= 2023-07-29T00:07:19.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3I/7CXz8aDuftJGvryYrYlMre8I=', 'Content-Length': '0'} 2023-07-29T00:07:19.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a14bb7dedefcb473-0064c45837-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:07:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:19.637 DEBUG:boto:b'hiiqos-46999falseobj1nulltrue2023-07-29T00:05:16.017Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-07-29T00:05:16.033Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-07-29T00:05:16.052Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-07-29T00:05:16.112Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-07-29T00:06:59.109Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-07-29T00:06:59.128Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-07-29T00:06:59.145Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-07-29T00:06:59.202Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:07:19.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-07-29T00:07:19.638 DEBUG:boto:path=/hiiqos-46/obj1 2023-07-29T00:07:19.638 DEBUG:boto:auth_path=/hiiqos-46/obj1 2023-07-29T00:07:19.639 DEBUG:boto:path=/hiiqos-46/obj1?versionId=null 2023-07-29T00:07:19.639 DEBUG:boto:auth_path=/hiiqos-46/obj1?versionId=null 2023-07-29T00:07:19.639 DEBUG:boto:Method: GET 2023-07-29T00:07:19.640 DEBUG:boto:Path: /hiiqos-46/obj1?versionId=null 2023-07-29T00:07:19.640 DEBUG:boto:Data: 2023-07-29T00:07:19.641 DEBUG:boto:Headers: {} 2023-07-29T00:07:19.641 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:19.641 DEBUG:boto:Port: 8001 2023-07-29T00:07:19.642 DEBUG:boto:Params: {} 2023-07-29T00:07:19.642 DEBUG:boto:Token: None 2023-07-29T00:07:19.642 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:19 GMT /hiiqos-46/obj1?versionId=null 2023-07-29T00:07:19.643 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DPj07N1tzMrClLLs5kKUKPmXxak= 2023-07-29T00:07:19.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DPj07N1tzMrClLLs5kKUKPmXxak=', 'Content-Length': '0'} 2023-07-29T00:07:19.645 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000080db574b7d152e0a-0064c45837-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:19.688 DEBUG:boto:path=/hiiqos-46/obj1 2023-07-29T00:07:19.689 DEBUG:boto:auth_path=/hiiqos-46/obj1 2023-07-29T00:07:19.689 DEBUG:boto:path=/hiiqos-46/obj1?versionId=null 2023-07-29T00:07:19.690 DEBUG:boto:auth_path=/hiiqos-46/obj1?versionId=null 2023-07-29T00:07:19.690 DEBUG:boto:Method: GET 2023-07-29T00:07:19.690 DEBUG:boto:Path: /hiiqos-46/obj1?versionId=null 2023-07-29T00:07:19.691 DEBUG:boto:Data: 2023-07-29T00:07:19.691 DEBUG:boto:Headers: {} 2023-07-29T00:07:19.691 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:19.692 DEBUG:boto:Port: 8000 2023-07-29T00:07:19.692 DEBUG:boto:Params: {} 2023-07-29T00:07:19.692 DEBUG:boto:Token: None 2023-07-29T00:07:19.693 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:19 GMT /hiiqos-46/obj1?versionId=null 2023-07-29T00:07:19.693 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DPj07N1tzMrClLLs5kKUKPmXxak= 2023-07-29T00:07:19.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DPj07N1tzMrClLLs5kKUKPmXxak=', 'Content-Length': '0'} 2023-07-29T00:07:19.696 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006d9e47d22e82f162-0064c45837-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:19.696 DEBUG:boto:path=/hiiqos-46/obj1 2023-07-29T00:07:19.697 DEBUG:boto:auth_path=/hiiqos-46/obj1 2023-07-29T00:07:19.697 DEBUG:boto:path=/hiiqos-46/obj1?versionId=null 2023-07-29T00:07:19.697 DEBUG:boto:auth_path=/hiiqos-46/obj1?versionId=null 2023-07-29T00:07:19.698 DEBUG:boto:Method: HEAD 2023-07-29T00:07:19.698 DEBUG:boto:Path: /hiiqos-46/obj1?versionId=null 2023-07-29T00:07:19.698 DEBUG:boto:Data: 2023-07-29T00:07:19.699 DEBUG:boto:Headers: {} 2023-07-29T00:07:19.699 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:19.699 DEBUG:boto:Port: 8001 2023-07-29T00:07:19.700 DEBUG:boto:Params: {} 2023-07-29T00:07:19.700 DEBUG:boto:Token: None 2023-07-29T00:07:19.701 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:07:19 GMT /hiiqos-46/obj1?versionId=null 2023-07-29T00:07:19.701 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nodJO+6lrbWJm6+5Lrdae/zQw1s= 2023-07-29T00:07:19.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nodJO+6lrbWJm6+5Lrdae/zQw1s=', 'Content-Length': '0'} 2023-07-29T00:07:19.703 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000095968ad54691c732-0064c45837-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:19.704 DEBUG:boto:path=/hiiqos-46/obj1 2023-07-29T00:07:19.704 DEBUG:boto:auth_path=/hiiqos-46/obj1 2023-07-29T00:07:19.705 DEBUG:boto:path=/hiiqos-46/obj1?versionId=null 2023-07-29T00:07:19.705 DEBUG:boto:auth_path=/hiiqos-46/obj1?versionId=null 2023-07-29T00:07:19.705 DEBUG:boto:Method: HEAD 2023-07-29T00:07:19.706 DEBUG:boto:Path: /hiiqos-46/obj1?versionId=null 2023-07-29T00:07:19.706 DEBUG:boto:Data: 2023-07-29T00:07:19.706 DEBUG:boto:Headers: {} 2023-07-29T00:07:19.707 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:19.707 DEBUG:boto:Port: 8000 2023-07-29T00:07:19.707 DEBUG:boto:Params: {} 2023-07-29T00:07:19.708 DEBUG:boto:Token: None 2023-07-29T00:07:19.708 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:07:19 GMT /hiiqos-46/obj1?versionId=null 2023-07-29T00:07:19.709 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nodJO+6lrbWJm6+5Lrdae/zQw1s= 2023-07-29T00:07:19.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nodJO+6lrbWJm6+5Lrdae/zQw1s=', 'Content-Length': '0'} 2023-07-29T00:07:19.711 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9c25d0aa7ce51e9-0064c45837-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:19.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-07-29T00:07:19.711 DEBUG:boto:path=/hiiqos-46/obj1 2023-07-29T00:07:19.712 DEBUG:boto:auth_path=/hiiqos-46/obj1 2023-07-29T00:07:19.712 DEBUG:boto:path=/hiiqos-46/obj1?versionId=null 2023-07-29T00:07:19.713 DEBUG:boto:auth_path=/hiiqos-46/obj1?versionId=null 2023-07-29T00:07:19.713 DEBUG:boto:Method: GET 2023-07-29T00:07:19.713 DEBUG:boto:Path: /hiiqos-46/obj1?versionId=null 2023-07-29T00:07:19.714 DEBUG:boto:Data: 2023-07-29T00:07:19.714 DEBUG:boto:Headers: {} 2023-07-29T00:07:19.714 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:19.715 DEBUG:boto:Port: 8001 2023-07-29T00:07:19.715 DEBUG:boto:Params: {} 2023-07-29T00:07:19.715 DEBUG:boto:Token: None 2023-07-29T00:07:19.716 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:19 GMT /hiiqos-46/obj1?versionId=null 2023-07-29T00:07:19.716 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DPj07N1tzMrClLLs5kKUKPmXxak= 2023-07-29T00:07:19.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DPj07N1tzMrClLLs5kKUKPmXxak=', 'Content-Length': '0'} 2023-07-29T00:07:19.718 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000d884505a4b6139e-0064c45837-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:19.719 DEBUG:boto:path=/hiiqos-46/obj1 2023-07-29T00:07:19.719 DEBUG:boto:auth_path=/hiiqos-46/obj1 2023-07-29T00:07:19.720 DEBUG:boto:path=/hiiqos-46/obj1?versionId=null 2023-07-29T00:07:19.720 DEBUG:boto:auth_path=/hiiqos-46/obj1?versionId=null 2023-07-29T00:07:19.720 DEBUG:boto:Method: GET 2023-07-29T00:07:19.721 DEBUG:boto:Path: /hiiqos-46/obj1?versionId=null 2023-07-29T00:07:19.721 DEBUG:boto:Data: 2023-07-29T00:07:19.722 DEBUG:boto:Headers: {} 2023-07-29T00:07:19.722 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:19.722 DEBUG:boto:Port: 8000 2023-07-29T00:07:19.723 DEBUG:boto:Params: {} 2023-07-29T00:07:19.723 DEBUG:boto:Token: None 2023-07-29T00:07:19.724 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:19 GMT /hiiqos-46/obj1?versionId=null 2023-07-29T00:07:19.725 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DPj07N1tzMrClLLs5kKUKPmXxak= 2023-07-29T00:07:19.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DPj07N1tzMrClLLs5kKUKPmXxak=', 'Content-Length': '0'} 2023-07-29T00:07:19.728 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007208e8c505b1d528-0064c45837-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:19.729 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:07:19.730 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:07:19.731 DEBUG:boto:path=/hiiqos-46/?prefix=obj1 2023-07-29T00:07:19.731 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj1 2023-07-29T00:07:19.732 DEBUG:boto:Method: GET 2023-07-29T00:07:19.733 DEBUG:boto:Path: /hiiqos-46/?prefix=obj1 2023-07-29T00:07:19.734 DEBUG:boto:Data: 2023-07-29T00:07:19.734 DEBUG:boto:Headers: {} 2023-07-29T00:07:19.734 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:19.735 DEBUG:boto:Port: 8001 2023-07-29T00:07:19.735 DEBUG:boto:Params: {} 2023-07-29T00:07:19.736 DEBUG:boto:Token: None 2023-07-29T00:07:19.736 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:19 GMT /hiiqos-46/ 2023-07-29T00:07:19.736 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:y0R1Mry7X5jPv7WXFSuJymalqyc= 2023-07-29T00:07:19.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:y0R1Mry7X5jPv7WXFSuJymalqyc=', 'Content-Length': '0'} 2023-07-29T00:07:19.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c10dc803db58f205-0064c45837-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:07:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:19.740 DEBUG:boto:b'hiiqos-46obj11000falseobj12023-07-29T00:05:16.017Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:07:19.741 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:07:19.741 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:07:19.742 DEBUG:boto:path=/hiiqos-46/?prefix=obj1 2023-07-29T00:07:19.742 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj1 2023-07-29T00:07:19.743 DEBUG:boto:Method: GET 2023-07-29T00:07:19.743 DEBUG:boto:Path: /hiiqos-46/?prefix=obj1 2023-07-29T00:07:19.743 DEBUG:boto:Data: 2023-07-29T00:07:19.744 DEBUG:boto:Headers: {} 2023-07-29T00:07:19.744 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:19.744 DEBUG:boto:Port: 8000 2023-07-29T00:07:19.745 DEBUG:boto:Params: {} 2023-07-29T00:07:19.745 DEBUG:boto:Token: None 2023-07-29T00:07:19.746 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:19 GMT /hiiqos-46/ 2023-07-29T00:07:19.746 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:y0R1Mry7X5jPv7WXFSuJymalqyc= 2023-07-29T00:07:19.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:y0R1Mry7X5jPv7WXFSuJymalqyc=', 'Content-Length': '0'} 2023-07-29T00:07:19.749 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3cc1fc9ad48b67b-0064c45837-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:07:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:19.796 DEBUG:boto:b'hiiqos-46obj11000falseobj12023-07-29T00:05:16.017Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:07:19.797 DEBUG:boto:path=/hiiqos-46/obj1 2023-07-29T00:07:19.798 DEBUG:boto:auth_path=/hiiqos-46/obj1 2023-07-29T00:07:19.798 DEBUG:boto:Method: HEAD 2023-07-29T00:07:19.798 DEBUG:boto:Path: /hiiqos-46/obj1 2023-07-29T00:07:19.799 DEBUG:boto:Data: 2023-07-29T00:07:19.799 DEBUG:boto:Headers: {} 2023-07-29T00:07:19.799 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:19.800 DEBUG:boto:Port: 8001 2023-07-29T00:07:19.800 DEBUG:boto:Params: {} 2023-07-29T00:07:19.800 DEBUG:boto:Token: None 2023-07-29T00:07:19.801 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:07:19 GMT /hiiqos-46/obj1 2023-07-29T00:07:19.801 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4Xu0drO9nYYWZVe8ppBdpyqaIBM= 2023-07-29T00:07:19.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4Xu0drO9nYYWZVe8ppBdpyqaIBM=', 'Content-Length': '0'} 2023-07-29T00:07:19.804 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000048b3fe679f37d196-0064c45837-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:19.804 DEBUG:boto:path=/hiiqos-46/obj1 2023-07-29T00:07:19.805 DEBUG:boto:auth_path=/hiiqos-46/obj1 2023-07-29T00:07:19.805 DEBUG:boto:Method: HEAD 2023-07-29T00:07:19.806 DEBUG:boto:Path: /hiiqos-46/obj1 2023-07-29T00:07:19.806 DEBUG:boto:Data: 2023-07-29T00:07:19.806 DEBUG:boto:Headers: {} 2023-07-29T00:07:19.807 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:19.808 DEBUG:boto:Port: 8000 2023-07-29T00:07:19.808 DEBUG:boto:Params: {} 2023-07-29T00:07:19.809 DEBUG:boto:Token: None 2023-07-29T00:07:19.809 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:07:19 GMT /hiiqos-46/obj1 2023-07-29T00:07:19.810 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4Xu0drO9nYYWZVe8ppBdpyqaIBM= 2023-07-29T00:07:19.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4Xu0drO9nYYWZVe8ppBdpyqaIBM=', 'Content-Length': '0'} 2023-07-29T00:07:19.812 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c57c2b7f2029eac4-0064c45837-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:19.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-07-29T00:07:19.814 DEBUG:boto:path=/hiiqos-46/obj1 2023-07-29T00:07:19.815 DEBUG:boto:auth_path=/hiiqos-46/obj1 2023-07-29T00:07:19.816 DEBUG:boto:Method: GET 2023-07-29T00:07:19.816 DEBUG:boto:Path: /hiiqos-46/obj1 2023-07-29T00:07:19.817 DEBUG:boto:Data: 2023-07-29T00:07:19.818 DEBUG:boto:Headers: {} 2023-07-29T00:07:19.818 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:19.819 DEBUG:boto:Port: 8001 2023-07-29T00:07:19.819 DEBUG:boto:Params: {} 2023-07-29T00:07:19.820 DEBUG:boto:Token: None 2023-07-29T00:07:19.820 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:19 GMT /hiiqos-46/obj1 2023-07-29T00:07:19.820 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tzuqdzDjGJPi7QYhVcIwIjAHUSA= 2023-07-29T00:07:19.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tzuqdzDjGJPi7QYhVcIwIjAHUSA=', 'Content-Length': '0'} 2023-07-29T00:07:19.823 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006512e3e56215bf1c-0064c45837-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:19.823 DEBUG:boto:path=/hiiqos-46/obj1 2023-07-29T00:07:19.824 DEBUG:boto:auth_path=/hiiqos-46/obj1 2023-07-29T00:07:19.824 DEBUG:boto:Method: GET 2023-07-29T00:07:19.825 DEBUG:boto:Path: /hiiqos-46/obj1 2023-07-29T00:07:19.825 DEBUG:boto:Data: 2023-07-29T00:07:19.825 DEBUG:boto:Headers: {} 2023-07-29T00:07:19.826 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:19.826 DEBUG:boto:Port: 8000 2023-07-29T00:07:19.826 DEBUG:boto:Params: {} 2023-07-29T00:07:19.827 DEBUG:boto:Token: None 2023-07-29T00:07:19.827 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:19 GMT /hiiqos-46/obj1 2023-07-29T00:07:19.828 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tzuqdzDjGJPi7QYhVcIwIjAHUSA= 2023-07-29T00:07:19.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tzuqdzDjGJPi7QYhVcIwIjAHUSA=', 'Content-Length': '0'} 2023-07-29T00:07:19.830 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000060bd347dc88f103f-0064c45837-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:19.831 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:07:19.831 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:07:19.831 DEBUG:boto:path=/hiiqos-46/?prefix=obj1 2023-07-29T00:07:19.832 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj1 2023-07-29T00:07:19.832 DEBUG:boto:Method: GET 2023-07-29T00:07:19.833 DEBUG:boto:Path: /hiiqos-46/?prefix=obj1 2023-07-29T00:07:19.833 DEBUG:boto:Data: 2023-07-29T00:07:19.833 DEBUG:boto:Headers: {} 2023-07-29T00:07:19.834 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:19.834 DEBUG:boto:Port: 8001 2023-07-29T00:07:19.834 DEBUG:boto:Params: {} 2023-07-29T00:07:19.835 DEBUG:boto:Token: None 2023-07-29T00:07:19.835 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:19 GMT /hiiqos-46/ 2023-07-29T00:07:19.836 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:y0R1Mry7X5jPv7WXFSuJymalqyc= 2023-07-29T00:07:19.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:y0R1Mry7X5jPv7WXFSuJymalqyc=', 'Content-Length': '0'} 2023-07-29T00:07:19.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000541f1d553d029819-0064c45837-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:07:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:19.839 DEBUG:boto:b'hiiqos-46obj11000falseobj12023-07-29T00:05:16.017Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:07:19.840 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:07:19.840 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:07:19.841 DEBUG:boto:path=/hiiqos-46/?prefix=obj1 2023-07-29T00:07:19.842 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj1 2023-07-29T00:07:19.843 DEBUG:boto:Method: GET 2023-07-29T00:07:19.844 DEBUG:boto:Path: /hiiqos-46/?prefix=obj1 2023-07-29T00:07:19.845 DEBUG:boto:Data: 2023-07-29T00:07:19.845 DEBUG:boto:Headers: {} 2023-07-29T00:07:19.846 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:19.846 DEBUG:boto:Port: 8000 2023-07-29T00:07:19.847 DEBUG:boto:Params: {} 2023-07-29T00:07:19.847 DEBUG:boto:Token: None 2023-07-29T00:07:19.848 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:19 GMT /hiiqos-46/ 2023-07-29T00:07:19.848 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:y0R1Mry7X5jPv7WXFSuJymalqyc= 2023-07-29T00:07:19.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:y0R1Mry7X5jPv7WXFSuJymalqyc=', 'Content-Length': '0'} 2023-07-29T00:07:19.851 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000faa59172ad4db77a-0064c45837-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:07:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:19.852 DEBUG:boto:b'hiiqos-46obj11000falseobj12023-07-29T00:05:16.017Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:07:19.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-07-29T00:07:19.853 DEBUG:boto:path=/hiiqos-46/obj2 2023-07-29T00:07:19.853 DEBUG:boto:auth_path=/hiiqos-46/obj2 2023-07-29T00:07:19.853 DEBUG:boto:path=/hiiqos-46/obj2?versionId=null 2023-07-29T00:07:19.854 DEBUG:boto:auth_path=/hiiqos-46/obj2?versionId=null 2023-07-29T00:07:19.854 DEBUG:boto:Method: GET 2023-07-29T00:07:19.855 DEBUG:boto:Path: /hiiqos-46/obj2?versionId=null 2023-07-29T00:07:19.855 DEBUG:boto:Data: 2023-07-29T00:07:19.855 DEBUG:boto:Headers: {} 2023-07-29T00:07:19.856 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:19.856 DEBUG:boto:Port: 8001 2023-07-29T00:07:19.856 DEBUG:boto:Params: {} 2023-07-29T00:07:19.857 DEBUG:boto:Token: None 2023-07-29T00:07:19.857 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:19 GMT /hiiqos-46/obj2?versionId=null 2023-07-29T00:07:19.858 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EtoqLmZZjuRjRutbUYttQQs1OPw= 2023-07-29T00:07:19.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EtoqLmZZjuRjRutbUYttQQs1OPw=', 'Content-Length': '0'} 2023-07-29T00:07:19.860 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030f5ee087fe5dd79-0064c45837-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:19.861 DEBUG:boto:path=/hiiqos-46/obj2 2023-07-29T00:07:19.861 DEBUG:boto:auth_path=/hiiqos-46/obj2 2023-07-29T00:07:19.861 DEBUG:boto:path=/hiiqos-46/obj2?versionId=null 2023-07-29T00:07:19.862 DEBUG:boto:auth_path=/hiiqos-46/obj2?versionId=null 2023-07-29T00:07:19.862 DEBUG:boto:Method: GET 2023-07-29T00:07:19.863 DEBUG:boto:Path: /hiiqos-46/obj2?versionId=null 2023-07-29T00:07:19.863 DEBUG:boto:Data: 2023-07-29T00:07:19.863 DEBUG:boto:Headers: {} 2023-07-29T00:07:19.864 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:19.864 DEBUG:boto:Port: 8000 2023-07-29T00:07:19.864 DEBUG:boto:Params: {} 2023-07-29T00:07:19.865 DEBUG:boto:Token: None 2023-07-29T00:07:19.865 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:19 GMT /hiiqos-46/obj2?versionId=null 2023-07-29T00:07:19.866 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EtoqLmZZjuRjRutbUYttQQs1OPw= 2023-07-29T00:07:19.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EtoqLmZZjuRjRutbUYttQQs1OPw=', 'Content-Length': '0'} 2023-07-29T00:07:19.868 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e64ade06e14d747-0064c45837-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:19.869 DEBUG:boto:path=/hiiqos-46/obj2 2023-07-29T00:07:19.869 DEBUG:boto:auth_path=/hiiqos-46/obj2 2023-07-29T00:07:19.870 DEBUG:boto:path=/hiiqos-46/obj2?versionId=null 2023-07-29T00:07:19.870 DEBUG:boto:auth_path=/hiiqos-46/obj2?versionId=null 2023-07-29T00:07:19.870 DEBUG:boto:Method: HEAD 2023-07-29T00:07:19.871 DEBUG:boto:Path: /hiiqos-46/obj2?versionId=null 2023-07-29T00:07:19.871 DEBUG:boto:Data: 2023-07-29T00:07:19.871 DEBUG:boto:Headers: {} 2023-07-29T00:07:19.872 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:19.872 DEBUG:boto:Port: 8001 2023-07-29T00:07:19.873 DEBUG:boto:Params: {} 2023-07-29T00:07:19.873 DEBUG:boto:Token: None 2023-07-29T00:07:19.874 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:07:19 GMT /hiiqos-46/obj2?versionId=null 2023-07-29T00:07:19.874 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3cvPNKhonz7zUI0iqh0ruhU9HbE= 2023-07-29T00:07:19.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3cvPNKhonz7zUI0iqh0ruhU9HbE=', 'Content-Length': '0'} 2023-07-29T00:07:19.876 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c6607424b373aa1-0064c45837-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:19.877 DEBUG:boto:path=/hiiqos-46/obj2 2023-07-29T00:07:19.877 DEBUG:boto:auth_path=/hiiqos-46/obj2 2023-07-29T00:07:19.878 DEBUG:boto:path=/hiiqos-46/obj2?versionId=null 2023-07-29T00:07:19.878 DEBUG:boto:auth_path=/hiiqos-46/obj2?versionId=null 2023-07-29T00:07:19.879 DEBUG:boto:Method: HEAD 2023-07-29T00:07:19.879 DEBUG:boto:Path: /hiiqos-46/obj2?versionId=null 2023-07-29T00:07:19.880 DEBUG:boto:Data: 2023-07-29T00:07:19.880 DEBUG:boto:Headers: {} 2023-07-29T00:07:19.880 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:19.881 DEBUG:boto:Port: 8000 2023-07-29T00:07:19.881 DEBUG:boto:Params: {} 2023-07-29T00:07:19.881 DEBUG:boto:Token: None 2023-07-29T00:07:19.882 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:07:19 GMT /hiiqos-46/obj2?versionId=null 2023-07-29T00:07:19.882 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3cvPNKhonz7zUI0iqh0ruhU9HbE= 2023-07-29T00:07:19.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3cvPNKhonz7zUI0iqh0ruhU9HbE=', 'Content-Length': '0'} 2023-07-29T00:07:19.884 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ad25d1e46769803-0064c45837-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:19.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-07-29T00:07:19.885 DEBUG:boto:path=/hiiqos-46/obj2 2023-07-29T00:07:19.886 DEBUG:boto:auth_path=/hiiqos-46/obj2 2023-07-29T00:07:19.886 DEBUG:boto:path=/hiiqos-46/obj2?versionId=null 2023-07-29T00:07:19.886 DEBUG:boto:auth_path=/hiiqos-46/obj2?versionId=null 2023-07-29T00:07:19.887 DEBUG:boto:Method: GET 2023-07-29T00:07:19.887 DEBUG:boto:Path: /hiiqos-46/obj2?versionId=null 2023-07-29T00:07:19.887 DEBUG:boto:Data: 2023-07-29T00:07:19.888 DEBUG:boto:Headers: {} 2023-07-29T00:07:19.889 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:19.889 DEBUG:boto:Port: 8001 2023-07-29T00:07:19.890 DEBUG:boto:Params: {} 2023-07-29T00:07:19.891 DEBUG:boto:Token: None 2023-07-29T00:07:19.892 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:19 GMT /hiiqos-46/obj2?versionId=null 2023-07-29T00:07:19.893 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EtoqLmZZjuRjRutbUYttQQs1OPw= 2023-07-29T00:07:19.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EtoqLmZZjuRjRutbUYttQQs1OPw=', 'Content-Length': '0'} 2023-07-29T00:07:19.896 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d587f8f5cdb744a0-0064c45837-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:19.936 DEBUG:boto:path=/hiiqos-46/obj2 2023-07-29T00:07:19.937 DEBUG:boto:auth_path=/hiiqos-46/obj2 2023-07-29T00:07:19.937 DEBUG:boto:path=/hiiqos-46/obj2?versionId=null 2023-07-29T00:07:19.938 DEBUG:boto:auth_path=/hiiqos-46/obj2?versionId=null 2023-07-29T00:07:19.938 DEBUG:boto:Method: GET 2023-07-29T00:07:19.939 DEBUG:boto:Path: /hiiqos-46/obj2?versionId=null 2023-07-29T00:07:19.939 DEBUG:boto:Data: 2023-07-29T00:07:19.940 DEBUG:boto:Headers: {} 2023-07-29T00:07:19.940 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:19.940 DEBUG:boto:Port: 8000 2023-07-29T00:07:19.941 DEBUG:boto:Params: {} 2023-07-29T00:07:19.941 DEBUG:boto:Token: None 2023-07-29T00:07:19.942 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:19 GMT /hiiqos-46/obj2?versionId=null 2023-07-29T00:07:19.942 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EtoqLmZZjuRjRutbUYttQQs1OPw= 2023-07-29T00:07:19.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EtoqLmZZjuRjRutbUYttQQs1OPw=', 'Content-Length': '0'} 2023-07-29T00:07:19.944 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007a77f738ca316757-0064c45837-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:19.945 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:07:19.946 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:07:19.946 DEBUG:boto:path=/hiiqos-46/?prefix=obj2 2023-07-29T00:07:19.946 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj2 2023-07-29T00:07:19.947 DEBUG:boto:Method: GET 2023-07-29T00:07:19.947 DEBUG:boto:Path: /hiiqos-46/?prefix=obj2 2023-07-29T00:07:19.947 DEBUG:boto:Data: 2023-07-29T00:07:19.948 DEBUG:boto:Headers: {} 2023-07-29T00:07:19.948 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:19.948 DEBUG:boto:Port: 8001 2023-07-29T00:07:19.949 DEBUG:boto:Params: {} 2023-07-29T00:07:19.949 DEBUG:boto:Token: None 2023-07-29T00:07:19.950 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:19 GMT /hiiqos-46/ 2023-07-29T00:07:19.951 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:y0R1Mry7X5jPv7WXFSuJymalqyc= 2023-07-29T00:07:19.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:y0R1Mry7X5jPv7WXFSuJymalqyc=', 'Content-Length': '0'} 2023-07-29T00:07:19.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070c3e20dd54e3df5-0064c45837-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:07:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:19.955 DEBUG:boto:b'hiiqos-46obj21000falseobj22023-07-29T00:05:16.033Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:07:19.955 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:07:19.956 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:07:19.956 DEBUG:boto:path=/hiiqos-46/?prefix=obj2 2023-07-29T00:07:19.956 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj2 2023-07-29T00:07:19.957 DEBUG:boto:Method: GET 2023-07-29T00:07:19.957 DEBUG:boto:Path: /hiiqos-46/?prefix=obj2 2023-07-29T00:07:19.957 DEBUG:boto:Data: 2023-07-29T00:07:19.958 DEBUG:boto:Headers: {} 2023-07-29T00:07:19.958 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:19.958 DEBUG:boto:Port: 8000 2023-07-29T00:07:19.959 DEBUG:boto:Params: {} 2023-07-29T00:07:19.959 DEBUG:boto:Token: None 2023-07-29T00:07:19.960 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:19 GMT /hiiqos-46/ 2023-07-29T00:07:19.960 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:y0R1Mry7X5jPv7WXFSuJymalqyc= 2023-07-29T00:07:19.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:19 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:y0R1Mry7X5jPv7WXFSuJymalqyc=', 'Content-Length': '0'} 2023-07-29T00:07:19.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc496e6c80bf45c7-0064c45837-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:07:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.004 DEBUG:boto:b'hiiqos-46obj21000falseobj22023-07-29T00:05:16.033Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:07:20.005 DEBUG:boto:path=/hiiqos-46/obj2 2023-07-29T00:07:20.005 DEBUG:boto:auth_path=/hiiqos-46/obj2 2023-07-29T00:07:20.006 DEBUG:boto:Method: HEAD 2023-07-29T00:07:20.006 DEBUG:boto:Path: /hiiqos-46/obj2 2023-07-29T00:07:20.006 DEBUG:boto:Data: 2023-07-29T00:07:20.007 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.007 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:20.007 DEBUG:boto:Port: 8001 2023-07-29T00:07:20.008 DEBUG:boto:Params: {} 2023-07-29T00:07:20.008 DEBUG:boto:Token: None 2023-07-29T00:07:20.008 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj2 2023-07-29T00:07:20.009 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:biBxlBPMjIYn7AK8f/kLhP/7gXw= 2023-07-29T00:07:20.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:biBxlBPMjIYn7AK8f/kLhP/7gXw=', 'Content-Length': '0'} 2023-07-29T00:07:20.011 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b8ea44dfd4f9539d-0064c45837-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.012 DEBUG:boto:path=/hiiqos-46/obj2 2023-07-29T00:07:20.012 DEBUG:boto:auth_path=/hiiqos-46/obj2 2023-07-29T00:07:20.012 DEBUG:boto:Method: HEAD 2023-07-29T00:07:20.013 DEBUG:boto:Path: /hiiqos-46/obj2 2023-07-29T00:07:20.013 DEBUG:boto:Data: 2023-07-29T00:07:20.013 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.014 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:20.014 DEBUG:boto:Port: 8000 2023-07-29T00:07:20.014 DEBUG:boto:Params: {} 2023-07-29T00:07:20.015 DEBUG:boto:Token: None 2023-07-29T00:07:20.015 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj2 2023-07-29T00:07:20.015 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:biBxlBPMjIYn7AK8f/kLhP/7gXw= 2023-07-29T00:07:20.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:biBxlBPMjIYn7AK8f/kLhP/7gXw=', 'Content-Length': '0'} 2023-07-29T00:07:20.018 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c7d9ee48c9f157f0-0064c45837-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-07-29T00:07:20.018 DEBUG:boto:path=/hiiqos-46/obj2 2023-07-29T00:07:20.019 DEBUG:boto:auth_path=/hiiqos-46/obj2 2023-07-29T00:07:20.019 DEBUG:boto:Method: GET 2023-07-29T00:07:20.020 DEBUG:boto:Path: /hiiqos-46/obj2 2023-07-29T00:07:20.020 DEBUG:boto:Data: 2023-07-29T00:07:20.020 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.021 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:20.021 DEBUG:boto:Port: 8001 2023-07-29T00:07:20.021 DEBUG:boto:Params: {} 2023-07-29T00:07:20.022 DEBUG:boto:Token: None 2023-07-29T00:07:20.022 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj2 2023-07-29T00:07:20.023 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0Z3cfzW2JmyNVe6OFHPQ7BP5010= 2023-07-29T00:07:20.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0Z3cfzW2JmyNVe6OFHPQ7BP5010=', 'Content-Length': '0'} 2023-07-29T00:07:20.025 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f79f229b02b921ef-0064c45837-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:19 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.025 DEBUG:boto:path=/hiiqos-46/obj2 2023-07-29T00:07:20.026 DEBUG:boto:auth_path=/hiiqos-46/obj2 2023-07-29T00:07:20.026 DEBUG:boto:Method: GET 2023-07-29T00:07:20.027 DEBUG:boto:Path: /hiiqos-46/obj2 2023-07-29T00:07:20.027 DEBUG:boto:Data: 2023-07-29T00:07:20.027 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.028 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:20.028 DEBUG:boto:Port: 8000 2023-07-29T00:07:20.028 DEBUG:boto:Params: {} 2023-07-29T00:07:20.029 DEBUG:boto:Token: None 2023-07-29T00:07:20.029 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj2 2023-07-29T00:07:20.030 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0Z3cfzW2JmyNVe6OFHPQ7BP5010= 2023-07-29T00:07:20.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0Z3cfzW2JmyNVe6OFHPQ7BP5010=', 'Content-Length': '0'} 2023-07-29T00:07:20.032 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000076bae842f6e32e65-0064c45838-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.033 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:07:20.033 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:07:20.033 DEBUG:boto:path=/hiiqos-46/?prefix=obj2 2023-07-29T00:07:20.034 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj2 2023-07-29T00:07:20.034 DEBUG:boto:Method: GET 2023-07-29T00:07:20.034 DEBUG:boto:Path: /hiiqos-46/?prefix=obj2 2023-07-29T00:07:20.035 DEBUG:boto:Data: 2023-07-29T00:07:20.035 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.035 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:20.036 DEBUG:boto:Port: 8001 2023-07-29T00:07:20.036 DEBUG:boto:Params: {} 2023-07-29T00:07:20.037 DEBUG:boto:Token: None 2023-07-29T00:07:20.037 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/ 2023-07-29T00:07:20.037 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:l6TjkUNta5lXZRduHIrSC6nFgck= 2023-07-29T00:07:20.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:l6TjkUNta5lXZRduHIrSC6nFgck=', 'Content-Length': '0'} 2023-07-29T00:07:20.040 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b31729831ef2cdc1-0064c45838-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.041 DEBUG:boto:b'hiiqos-46obj21000falseobj22023-07-29T00:05:16.033Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:07:20.042 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:07:20.042 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:07:20.042 DEBUG:boto:path=/hiiqos-46/?prefix=obj2 2023-07-29T00:07:20.043 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj2 2023-07-29T00:07:20.043 DEBUG:boto:Method: GET 2023-07-29T00:07:20.043 DEBUG:boto:Path: /hiiqos-46/?prefix=obj2 2023-07-29T00:07:20.044 DEBUG:boto:Data: 2023-07-29T00:07:20.044 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.045 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:20.045 DEBUG:boto:Port: 8000 2023-07-29T00:07:20.045 DEBUG:boto:Params: {} 2023-07-29T00:07:20.046 DEBUG:boto:Token: None 2023-07-29T00:07:20.046 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/ 2023-07-29T00:07:20.046 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:l6TjkUNta5lXZRduHIrSC6nFgck= 2023-07-29T00:07:20.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:l6TjkUNta5lXZRduHIrSC6nFgck=', 'Content-Length': '0'} 2023-07-29T00:07:20.049 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e41fdea24026fcf5-0064c45838-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.050 DEBUG:boto:b'hiiqos-46obj21000falseobj22023-07-29T00:05:16.033Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:07:20.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-07-29T00:07:20.051 DEBUG:boto:path=/hiiqos-46/obj3 2023-07-29T00:07:20.051 DEBUG:boto:auth_path=/hiiqos-46/obj3 2023-07-29T00:07:20.051 DEBUG:boto:path=/hiiqos-46/obj3?versionId=null 2023-07-29T00:07:20.052 DEBUG:boto:auth_path=/hiiqos-46/obj3?versionId=null 2023-07-29T00:07:20.052 DEBUG:boto:Method: GET 2023-07-29T00:07:20.053 DEBUG:boto:Path: /hiiqos-46/obj3?versionId=null 2023-07-29T00:07:20.053 DEBUG:boto:Data: 2023-07-29T00:07:20.053 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.054 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:20.054 DEBUG:boto:Port: 8001 2023-07-29T00:07:20.054 DEBUG:boto:Params: {} 2023-07-29T00:07:20.055 DEBUG:boto:Token: None 2023-07-29T00:07:20.055 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj3?versionId=null 2023-07-29T00:07:20.056 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ouiVYu1QRa4Bsrx84DGpU571pMM= 2023-07-29T00:07:20.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ouiVYu1QRa4Bsrx84DGpU571pMM=', 'Content-Length': '0'} 2023-07-29T00:07:20.058 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aea4566b93bcbde6-0064c45838-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.100 DEBUG:boto:path=/hiiqos-46/obj3 2023-07-29T00:07:20.101 DEBUG:boto:auth_path=/hiiqos-46/obj3 2023-07-29T00:07:20.102 DEBUG:boto:path=/hiiqos-46/obj3?versionId=null 2023-07-29T00:07:20.103 DEBUG:boto:auth_path=/hiiqos-46/obj3?versionId=null 2023-07-29T00:07:20.104 DEBUG:boto:Method: GET 2023-07-29T00:07:20.105 DEBUG:boto:Path: /hiiqos-46/obj3?versionId=null 2023-07-29T00:07:20.105 DEBUG:boto:Data: 2023-07-29T00:07:20.105 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.106 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:20.106 DEBUG:boto:Port: 8000 2023-07-29T00:07:20.106 DEBUG:boto:Params: {} 2023-07-29T00:07:20.107 DEBUG:boto:Token: None 2023-07-29T00:07:20.107 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj3?versionId=null 2023-07-29T00:07:20.108 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ouiVYu1QRa4Bsrx84DGpU571pMM= 2023-07-29T00:07:20.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ouiVYu1QRa4Bsrx84DGpU571pMM=', 'Content-Length': '0'} 2023-07-29T00:07:20.111 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bd86842e1e88d089-0064c45838-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.111 DEBUG:boto:path=/hiiqos-46/obj3 2023-07-29T00:07:20.112 DEBUG:boto:auth_path=/hiiqos-46/obj3 2023-07-29T00:07:20.112 DEBUG:boto:path=/hiiqos-46/obj3?versionId=null 2023-07-29T00:07:20.112 DEBUG:boto:auth_path=/hiiqos-46/obj3?versionId=null 2023-07-29T00:07:20.113 DEBUG:boto:Method: HEAD 2023-07-29T00:07:20.113 DEBUG:boto:Path: /hiiqos-46/obj3?versionId=null 2023-07-29T00:07:20.113 DEBUG:boto:Data: 2023-07-29T00:07:20.114 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.114 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:20.114 DEBUG:boto:Port: 8001 2023-07-29T00:07:20.115 DEBUG:boto:Params: {} 2023-07-29T00:07:20.115 DEBUG:boto:Token: None 2023-07-29T00:07:20.115 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj3?versionId=null 2023-07-29T00:07:20.116 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QQx8jNZvkHaylZ2r7uZqBkTIJBc= 2023-07-29T00:07:20.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QQx8jNZvkHaylZ2r7uZqBkTIJBc=', 'Content-Length': '0'} 2023-07-29T00:07:20.118 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bdaa33676b83a227-0064c45838-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.119 DEBUG:boto:path=/hiiqos-46/obj3 2023-07-29T00:07:20.119 DEBUG:boto:auth_path=/hiiqos-46/obj3 2023-07-29T00:07:20.119 DEBUG:boto:path=/hiiqos-46/obj3?versionId=null 2023-07-29T00:07:20.120 DEBUG:boto:auth_path=/hiiqos-46/obj3?versionId=null 2023-07-29T00:07:20.120 DEBUG:boto:Method: HEAD 2023-07-29T00:07:20.120 DEBUG:boto:Path: /hiiqos-46/obj3?versionId=null 2023-07-29T00:07:20.121 DEBUG:boto:Data: 2023-07-29T00:07:20.121 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.121 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:20.122 DEBUG:boto:Port: 8000 2023-07-29T00:07:20.122 DEBUG:boto:Params: {} 2023-07-29T00:07:20.122 DEBUG:boto:Token: None 2023-07-29T00:07:20.123 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj3?versionId=null 2023-07-29T00:07:20.123 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QQx8jNZvkHaylZ2r7uZqBkTIJBc= 2023-07-29T00:07:20.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QQx8jNZvkHaylZ2r7uZqBkTIJBc=', 'Content-Length': '0'} 2023-07-29T00:07:20.125 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a100b263657eebf3-0064c45838-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-07-29T00:07:20.126 DEBUG:boto:path=/hiiqos-46/obj3 2023-07-29T00:07:20.127 DEBUG:boto:auth_path=/hiiqos-46/obj3 2023-07-29T00:07:20.127 DEBUG:boto:path=/hiiqos-46/obj3?versionId=null 2023-07-29T00:07:20.128 DEBUG:boto:auth_path=/hiiqos-46/obj3?versionId=null 2023-07-29T00:07:20.128 DEBUG:boto:Method: GET 2023-07-29T00:07:20.129 DEBUG:boto:Path: /hiiqos-46/obj3?versionId=null 2023-07-29T00:07:20.129 DEBUG:boto:Data: 2023-07-29T00:07:20.129 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.130 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:20.130 DEBUG:boto:Port: 8001 2023-07-29T00:07:20.130 DEBUG:boto:Params: {} 2023-07-29T00:07:20.131 DEBUG:boto:Token: None 2023-07-29T00:07:20.131 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj3?versionId=null 2023-07-29T00:07:20.132 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ouiVYu1QRa4Bsrx84DGpU571pMM= 2023-07-29T00:07:20.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ouiVYu1QRa4Bsrx84DGpU571pMM=', 'Content-Length': '0'} 2023-07-29T00:07:20.134 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fc4d0c24db8046a0-0064c45838-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.135 DEBUG:boto:path=/hiiqos-46/obj3 2023-07-29T00:07:20.135 DEBUG:boto:auth_path=/hiiqos-46/obj3 2023-07-29T00:07:20.135 DEBUG:boto:path=/hiiqos-46/obj3?versionId=null 2023-07-29T00:07:20.136 DEBUG:boto:auth_path=/hiiqos-46/obj3?versionId=null 2023-07-29T00:07:20.136 DEBUG:boto:Method: GET 2023-07-29T00:07:20.137 DEBUG:boto:Path: /hiiqos-46/obj3?versionId=null 2023-07-29T00:07:20.137 DEBUG:boto:Data: 2023-07-29T00:07:20.137 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.138 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:20.138 DEBUG:boto:Port: 8000 2023-07-29T00:07:20.138 DEBUG:boto:Params: {} 2023-07-29T00:07:20.139 DEBUG:boto:Token: None 2023-07-29T00:07:20.139 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj3?versionId=null 2023-07-29T00:07:20.140 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ouiVYu1QRa4Bsrx84DGpU571pMM= 2023-07-29T00:07:20.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ouiVYu1QRa4Bsrx84DGpU571pMM=', 'Content-Length': '0'} 2023-07-29T00:07:20.142 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d7ffad8f45bc0807-0064c45838-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.142 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:07:20.143 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:07:20.143 DEBUG:boto:path=/hiiqos-46/?prefix=obj3 2023-07-29T00:07:20.143 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj3 2023-07-29T00:07:20.144 DEBUG:boto:Method: GET 2023-07-29T00:07:20.144 DEBUG:boto:Path: /hiiqos-46/?prefix=obj3 2023-07-29T00:07:20.145 DEBUG:boto:Data: 2023-07-29T00:07:20.145 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.145 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:20.146 DEBUG:boto:Port: 8001 2023-07-29T00:07:20.146 DEBUG:boto:Params: {} 2023-07-29T00:07:20.147 DEBUG:boto:Token: None 2023-07-29T00:07:20.147 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/ 2023-07-29T00:07:20.147 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:l6TjkUNta5lXZRduHIrSC6nFgck= 2023-07-29T00:07:20.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:l6TjkUNta5lXZRduHIrSC6nFgck=', 'Content-Length': '0'} 2023-07-29T00:07:20.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047abb0981300240c-0064c45838-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.151 DEBUG:boto:b'hiiqos-46obj31000falseobj32023-07-29T00:05:16.052Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:07:20.151 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:07:20.152 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:07:20.152 DEBUG:boto:path=/hiiqos-46/?prefix=obj3 2023-07-29T00:07:20.153 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj3 2023-07-29T00:07:20.153 DEBUG:boto:Method: GET 2023-07-29T00:07:20.153 DEBUG:boto:Path: /hiiqos-46/?prefix=obj3 2023-07-29T00:07:20.154 DEBUG:boto:Data: 2023-07-29T00:07:20.154 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.154 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:20.155 DEBUG:boto:Port: 8000 2023-07-29T00:07:20.155 DEBUG:boto:Params: {} 2023-07-29T00:07:20.155 DEBUG:boto:Token: None 2023-07-29T00:07:20.156 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/ 2023-07-29T00:07:20.156 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:l6TjkUNta5lXZRduHIrSC6nFgck= 2023-07-29T00:07:20.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:l6TjkUNta5lXZRduHIrSC6nFgck=', 'Content-Length': '0'} 2023-07-29T00:07:20.159 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dad2139540c29aa8-0064c45838-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.159 DEBUG:boto:b'hiiqos-46obj31000falseobj32023-07-29T00:05:16.052Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:07:20.160 DEBUG:boto:path=/hiiqos-46/obj3 2023-07-29T00:07:20.161 DEBUG:boto:auth_path=/hiiqos-46/obj3 2023-07-29T00:07:20.161 DEBUG:boto:Method: HEAD 2023-07-29T00:07:20.161 DEBUG:boto:Path: /hiiqos-46/obj3 2023-07-29T00:07:20.162 DEBUG:boto:Data: 2023-07-29T00:07:20.162 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.162 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:20.163 DEBUG:boto:Port: 8001 2023-07-29T00:07:20.163 DEBUG:boto:Params: {} 2023-07-29T00:07:20.163 DEBUG:boto:Token: None 2023-07-29T00:07:20.164 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj3 2023-07-29T00:07:20.164 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9HU6Riv16htVvhDX19dyPPU6UQY= 2023-07-29T00:07:20.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9HU6Riv16htVvhDX19dyPPU6UQY=', 'Content-Length': '0'} 2023-07-29T00:07:20.166 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e8269634da0808ef-0064c45838-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.167 DEBUG:boto:path=/hiiqos-46/obj3 2023-07-29T00:07:20.167 DEBUG:boto:auth_path=/hiiqos-46/obj3 2023-07-29T00:07:20.167 DEBUG:boto:Method: HEAD 2023-07-29T00:07:20.168 DEBUG:boto:Path: /hiiqos-46/obj3 2023-07-29T00:07:20.168 DEBUG:boto:Data: 2023-07-29T00:07:20.169 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.169 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:20.169 DEBUG:boto:Port: 8000 2023-07-29T00:07:20.169 DEBUG:boto:Params: {} 2023-07-29T00:07:20.170 DEBUG:boto:Token: None 2023-07-29T00:07:20.170 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj3 2023-07-29T00:07:20.171 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9HU6Riv16htVvhDX19dyPPU6UQY= 2023-07-29T00:07:20.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9HU6Riv16htVvhDX19dyPPU6UQY=', 'Content-Length': '0'} 2023-07-29T00:07:20.173 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bbb39adeaa2daf2f-0064c45838-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.173 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-07-29T00:07:20.174 DEBUG:boto:path=/hiiqos-46/obj3 2023-07-29T00:07:20.174 DEBUG:boto:auth_path=/hiiqos-46/obj3 2023-07-29T00:07:20.174 DEBUG:boto:Method: GET 2023-07-29T00:07:20.175 DEBUG:boto:Path: /hiiqos-46/obj3 2023-07-29T00:07:20.175 DEBUG:boto:Data: 2023-07-29T00:07:20.175 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.176 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:20.176 DEBUG:boto:Port: 8001 2023-07-29T00:07:20.176 DEBUG:boto:Params: {} 2023-07-29T00:07:20.177 DEBUG:boto:Token: None 2023-07-29T00:07:20.177 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj3 2023-07-29T00:07:20.178 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XFuqbHdnYmFp4lkmMWEyJQftZFw= 2023-07-29T00:07:20.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XFuqbHdnYmFp4lkmMWEyJQftZFw=', 'Content-Length': '0'} 2023-07-29T00:07:20.180 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b44ff18687b66a85-0064c45838-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.180 DEBUG:boto:path=/hiiqos-46/obj3 2023-07-29T00:07:20.181 DEBUG:boto:auth_path=/hiiqos-46/obj3 2023-07-29T00:07:20.181 DEBUG:boto:Method: GET 2023-07-29T00:07:20.182 DEBUG:boto:Path: /hiiqos-46/obj3 2023-07-29T00:07:20.182 DEBUG:boto:Data: 2023-07-29T00:07:20.182 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.183 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:20.183 DEBUG:boto:Port: 8000 2023-07-29T00:07:20.183 DEBUG:boto:Params: {} 2023-07-29T00:07:20.184 DEBUG:boto:Token: None 2023-07-29T00:07:20.184 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj3 2023-07-29T00:07:20.185 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XFuqbHdnYmFp4lkmMWEyJQftZFw= 2023-07-29T00:07:20.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XFuqbHdnYmFp4lkmMWEyJQftZFw=', 'Content-Length': '0'} 2023-07-29T00:07:20.187 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005a99a551d004233b-0064c45838-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.232 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:07:20.233 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:07:20.233 DEBUG:boto:path=/hiiqos-46/?prefix=obj3 2023-07-29T00:07:20.234 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj3 2023-07-29T00:07:20.234 DEBUG:boto:Method: GET 2023-07-29T00:07:20.234 DEBUG:boto:Path: /hiiqos-46/?prefix=obj3 2023-07-29T00:07:20.235 DEBUG:boto:Data: 2023-07-29T00:07:20.235 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.235 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:20.236 DEBUG:boto:Port: 8001 2023-07-29T00:07:20.236 DEBUG:boto:Params: {} 2023-07-29T00:07:20.237 DEBUG:boto:Token: None 2023-07-29T00:07:20.237 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/ 2023-07-29T00:07:20.238 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:l6TjkUNta5lXZRduHIrSC6nFgck= 2023-07-29T00:07:20.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:l6TjkUNta5lXZRduHIrSC6nFgck=', 'Content-Length': '0'} 2023-07-29T00:07:20.240 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f3ed86a90e95f92-0064c45838-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.241 DEBUG:boto:b'hiiqos-46obj31000falseobj32023-07-29T00:05:16.052Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:07:20.242 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:07:20.242 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:07:20.243 DEBUG:boto:path=/hiiqos-46/?prefix=obj3 2023-07-29T00:07:20.243 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj3 2023-07-29T00:07:20.244 DEBUG:boto:Method: GET 2023-07-29T00:07:20.244 DEBUG:boto:Path: /hiiqos-46/?prefix=obj3 2023-07-29T00:07:20.244 DEBUG:boto:Data: 2023-07-29T00:07:20.245 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.246 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:20.246 DEBUG:boto:Port: 8000 2023-07-29T00:07:20.246 DEBUG:boto:Params: {} 2023-07-29T00:07:20.247 DEBUG:boto:Token: None 2023-07-29T00:07:20.247 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/ 2023-07-29T00:07:20.248 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:l6TjkUNta5lXZRduHIrSC6nFgck= 2023-07-29T00:07:20.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:l6TjkUNta5lXZRduHIrSC6nFgck=', 'Content-Length': '0'} 2023-07-29T00:07:20.251 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5f29c09fdfda4d5-0064c45838-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.252 DEBUG:boto:b'hiiqos-46obj31000falseobj32023-07-29T00:05:16.052Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:07:20.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-07-29T00:07:20.253 DEBUG:boto:path=/hiiqos-46/obj4 2023-07-29T00:07:20.254 DEBUG:boto:auth_path=/hiiqos-46/obj4 2023-07-29T00:07:20.255 DEBUG:boto:path=/hiiqos-46/obj4?versionId=null 2023-07-29T00:07:20.256 DEBUG:boto:auth_path=/hiiqos-46/obj4?versionId=null 2023-07-29T00:07:20.256 DEBUG:boto:Method: GET 2023-07-29T00:07:20.257 DEBUG:boto:Path: /hiiqos-46/obj4?versionId=null 2023-07-29T00:07:20.257 DEBUG:boto:Data: 2023-07-29T00:07:20.257 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.258 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:20.258 DEBUG:boto:Port: 8001 2023-07-29T00:07:20.258 DEBUG:boto:Params: {} 2023-07-29T00:07:20.259 DEBUG:boto:Token: None 2023-07-29T00:07:20.259 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj4?versionId=null 2023-07-29T00:07:20.260 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vcpJEjHyu3u2Mw214nNAnA8xU14= 2023-07-29T00:07:20.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vcpJEjHyu3u2Mw214nNAnA8xU14=', 'Content-Length': '0'} 2023-07-29T00:07:20.262 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000086306a2790533b1a-0064c45838-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.263 DEBUG:boto:path=/hiiqos-46/obj4 2023-07-29T00:07:20.263 DEBUG:boto:auth_path=/hiiqos-46/obj4 2023-07-29T00:07:20.264 DEBUG:boto:path=/hiiqos-46/obj4?versionId=null 2023-07-29T00:07:20.264 DEBUG:boto:auth_path=/hiiqos-46/obj4?versionId=null 2023-07-29T00:07:20.264 DEBUG:boto:Method: GET 2023-07-29T00:07:20.265 DEBUG:boto:Path: /hiiqos-46/obj4?versionId=null 2023-07-29T00:07:20.265 DEBUG:boto:Data: 2023-07-29T00:07:20.265 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.266 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:20.266 DEBUG:boto:Port: 8000 2023-07-29T00:07:20.266 DEBUG:boto:Params: {} 2023-07-29T00:07:20.267 DEBUG:boto:Token: None 2023-07-29T00:07:20.267 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj4?versionId=null 2023-07-29T00:07:20.268 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vcpJEjHyu3u2Mw214nNAnA8xU14= 2023-07-29T00:07:20.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vcpJEjHyu3u2Mw214nNAnA8xU14=', 'Content-Length': '0'} 2023-07-29T00:07:20.270 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ebc648549e5c044-0064c45838-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.271 DEBUG:boto:path=/hiiqos-46/obj4 2023-07-29T00:07:20.271 DEBUG:boto:auth_path=/hiiqos-46/obj4 2023-07-29T00:07:20.272 DEBUG:boto:path=/hiiqos-46/obj4?versionId=null 2023-07-29T00:07:20.272 DEBUG:boto:auth_path=/hiiqos-46/obj4?versionId=null 2023-07-29T00:07:20.273 DEBUG:boto:Method: HEAD 2023-07-29T00:07:20.273 DEBUG:boto:Path: /hiiqos-46/obj4?versionId=null 2023-07-29T00:07:20.274 DEBUG:boto:Data: 2023-07-29T00:07:20.274 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.275 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:20.276 DEBUG:boto:Port: 8001 2023-07-29T00:07:20.276 DEBUG:boto:Params: {} 2023-07-29T00:07:20.277 DEBUG:boto:Token: None 2023-07-29T00:07:20.277 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj4?versionId=null 2023-07-29T00:07:20.278 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nSCBpYvCtoVlDm+dbDEduN3pw1U= 2023-07-29T00:07:20.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nSCBpYvCtoVlDm+dbDEduN3pw1U=', 'Content-Length': '0'} 2023-07-29T00:07:20.281 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000079e61a3cfbf291e9-0064c45838-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.282 DEBUG:boto:path=/hiiqos-46/obj4 2023-07-29T00:07:20.282 DEBUG:boto:auth_path=/hiiqos-46/obj4 2023-07-29T00:07:20.283 DEBUG:boto:path=/hiiqos-46/obj4?versionId=null 2023-07-29T00:07:20.283 DEBUG:boto:auth_path=/hiiqos-46/obj4?versionId=null 2023-07-29T00:07:20.284 DEBUG:boto:Method: HEAD 2023-07-29T00:07:20.285 DEBUG:boto:Path: /hiiqos-46/obj4?versionId=null 2023-07-29T00:07:20.285 DEBUG:boto:Data: 2023-07-29T00:07:20.286 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.286 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:20.286 DEBUG:boto:Port: 8000 2023-07-29T00:07:20.287 DEBUG:boto:Params: {} 2023-07-29T00:07:20.287 DEBUG:boto:Token: None 2023-07-29T00:07:20.288 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj4?versionId=null 2023-07-29T00:07:20.288 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nSCBpYvCtoVlDm+dbDEduN3pw1U= 2023-07-29T00:07:20.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nSCBpYvCtoVlDm+dbDEduN3pw1U=', 'Content-Length': '0'} 2023-07-29T00:07:20.290 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000d1066b3ab4bd96f-0064c45838-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-07-29T00:07:20.291 DEBUG:boto:path=/hiiqos-46/obj4 2023-07-29T00:07:20.292 DEBUG:boto:auth_path=/hiiqos-46/obj4 2023-07-29T00:07:20.292 DEBUG:boto:path=/hiiqos-46/obj4?versionId=null 2023-07-29T00:07:20.293 DEBUG:boto:auth_path=/hiiqos-46/obj4?versionId=null 2023-07-29T00:07:20.293 DEBUG:boto:Method: GET 2023-07-29T00:07:20.293 DEBUG:boto:Path: /hiiqos-46/obj4?versionId=null 2023-07-29T00:07:20.294 DEBUG:boto:Data: 2023-07-29T00:07:20.294 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.294 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:20.295 DEBUG:boto:Port: 8001 2023-07-29T00:07:20.295 DEBUG:boto:Params: {} 2023-07-29T00:07:20.295 DEBUG:boto:Token: None 2023-07-29T00:07:20.296 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj4?versionId=null 2023-07-29T00:07:20.296 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vcpJEjHyu3u2Mw214nNAnA8xU14= 2023-07-29T00:07:20.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vcpJEjHyu3u2Mw214nNAnA8xU14=', 'Content-Length': '0'} 2023-07-29T00:07:20.299 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe9f229908b8642f-0064c45838-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.340 DEBUG:boto:path=/hiiqos-46/obj4 2023-07-29T00:07:20.341 DEBUG:boto:auth_path=/hiiqos-46/obj4 2023-07-29T00:07:20.341 DEBUG:boto:path=/hiiqos-46/obj4?versionId=null 2023-07-29T00:07:20.342 DEBUG:boto:auth_path=/hiiqos-46/obj4?versionId=null 2023-07-29T00:07:20.342 DEBUG:boto:Method: GET 2023-07-29T00:07:20.342 DEBUG:boto:Path: /hiiqos-46/obj4?versionId=null 2023-07-29T00:07:20.343 DEBUG:boto:Data: 2023-07-29T00:07:20.343 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.343 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:20.344 DEBUG:boto:Port: 8000 2023-07-29T00:07:20.344 DEBUG:boto:Params: {} 2023-07-29T00:07:20.345 DEBUG:boto:Token: None 2023-07-29T00:07:20.345 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj4?versionId=null 2023-07-29T00:07:20.346 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vcpJEjHyu3u2Mw214nNAnA8xU14= 2023-07-29T00:07:20.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vcpJEjHyu3u2Mw214nNAnA8xU14=', 'Content-Length': '0'} 2023-07-29T00:07:20.348 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f075af91bae080a-0064c45838-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.349 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:07:20.349 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:07:20.349 DEBUG:boto:path=/hiiqos-46/?prefix=obj4 2023-07-29T00:07:20.350 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj4 2023-07-29T00:07:20.350 DEBUG:boto:Method: GET 2023-07-29T00:07:20.350 DEBUG:boto:Path: /hiiqos-46/?prefix=obj4 2023-07-29T00:07:20.351 DEBUG:boto:Data: 2023-07-29T00:07:20.351 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.351 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:20.352 DEBUG:boto:Port: 8001 2023-07-29T00:07:20.352 DEBUG:boto:Params: {} 2023-07-29T00:07:20.353 DEBUG:boto:Token: None 2023-07-29T00:07:20.353 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/ 2023-07-29T00:07:20.353 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:l6TjkUNta5lXZRduHIrSC6nFgck= 2023-07-29T00:07:20.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:l6TjkUNta5lXZRduHIrSC6nFgck=', 'Content-Length': '0'} 2023-07-29T00:07:20.356 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6d15379f63d5e92-0064c45838-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.357 DEBUG:boto:b'hiiqos-46obj41000falseobj42023-07-29T00:05:16.112Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:07:20.358 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:07:20.358 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:07:20.358 DEBUG:boto:path=/hiiqos-46/?prefix=obj4 2023-07-29T00:07:20.359 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj4 2023-07-29T00:07:20.359 DEBUG:boto:Method: GET 2023-07-29T00:07:20.359 DEBUG:boto:Path: /hiiqos-46/?prefix=obj4 2023-07-29T00:07:20.360 DEBUG:boto:Data: 2023-07-29T00:07:20.360 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.361 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:20.361 DEBUG:boto:Port: 8000 2023-07-29T00:07:20.361 DEBUG:boto:Params: {} 2023-07-29T00:07:20.362 DEBUG:boto:Token: None 2023-07-29T00:07:20.362 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/ 2023-07-29T00:07:20.363 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:l6TjkUNta5lXZRduHIrSC6nFgck= 2023-07-29T00:07:20.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:l6TjkUNta5lXZRduHIrSC6nFgck=', 'Content-Length': '0'} 2023-07-29T00:07:20.365 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077162b9f44bcfd00-0064c45838-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.366 DEBUG:boto:b'hiiqos-46obj41000falseobj42023-07-29T00:05:16.112Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:07:20.366 DEBUG:boto:path=/hiiqos-46/obj4 2023-07-29T00:07:20.367 DEBUG:boto:auth_path=/hiiqos-46/obj4 2023-07-29T00:07:20.367 DEBUG:boto:Method: HEAD 2023-07-29T00:07:20.368 DEBUG:boto:Path: /hiiqos-46/obj4 2023-07-29T00:07:20.368 DEBUG:boto:Data: 2023-07-29T00:07:20.368 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.369 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:20.369 DEBUG:boto:Port: 8001 2023-07-29T00:07:20.369 DEBUG:boto:Params: {} 2023-07-29T00:07:20.370 DEBUG:boto:Token: None 2023-07-29T00:07:20.370 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj4 2023-07-29T00:07:20.370 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dFIzKRdYFd6V7P4UIeUrJMPlStQ= 2023-07-29T00:07:20.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dFIzKRdYFd6V7P4UIeUrJMPlStQ=', 'Content-Length': '0'} 2023-07-29T00:07:20.372 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000165066b017ca1769-0064c45838-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.373 DEBUG:boto:path=/hiiqos-46/obj4 2023-07-29T00:07:20.373 DEBUG:boto:auth_path=/hiiqos-46/obj4 2023-07-29T00:07:20.374 DEBUG:boto:Method: HEAD 2023-07-29T00:07:20.374 DEBUG:boto:Path: /hiiqos-46/obj4 2023-07-29T00:07:20.374 DEBUG:boto:Data: 2023-07-29T00:07:20.375 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.375 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:20.375 DEBUG:boto:Port: 8000 2023-07-29T00:07:20.376 DEBUG:boto:Params: {} 2023-07-29T00:07:20.376 DEBUG:boto:Token: None 2023-07-29T00:07:20.376 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj4 2023-07-29T00:07:20.377 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dFIzKRdYFd6V7P4UIeUrJMPlStQ= 2023-07-29T00:07:20.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dFIzKRdYFd6V7P4UIeUrJMPlStQ=', 'Content-Length': '0'} 2023-07-29T00:07:20.379 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ce5fc20ba7fd5dff-0064c45838-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-07-29T00:07:20.380 DEBUG:boto:path=/hiiqos-46/obj4 2023-07-29T00:07:20.380 DEBUG:boto:auth_path=/hiiqos-46/obj4 2023-07-29T00:07:20.381 DEBUG:boto:Method: GET 2023-07-29T00:07:20.381 DEBUG:boto:Path: /hiiqos-46/obj4 2023-07-29T00:07:20.381 DEBUG:boto:Data: 2023-07-29T00:07:20.382 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.382 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:20.382 DEBUG:boto:Port: 8001 2023-07-29T00:07:20.383 DEBUG:boto:Params: {} 2023-07-29T00:07:20.383 DEBUG:boto:Token: None 2023-07-29T00:07:20.383 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj4 2023-07-29T00:07:20.384 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3aKAu/amWqxLnWlzxHTHMCyuoOs= 2023-07-29T00:07:20.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3aKAu/amWqxLnWlzxHTHMCyuoOs=', 'Content-Length': '0'} 2023-07-29T00:07:20.386 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000ad7fc9ba8b2e438-0064c45838-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.428 DEBUG:boto:path=/hiiqos-46/obj4 2023-07-29T00:07:20.429 DEBUG:boto:auth_path=/hiiqos-46/obj4 2023-07-29T00:07:20.429 DEBUG:boto:Method: GET 2023-07-29T00:07:20.430 DEBUG:boto:Path: /hiiqos-46/obj4 2023-07-29T00:07:20.430 DEBUG:boto:Data: 2023-07-29T00:07:20.431 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.431 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:20.431 DEBUG:boto:Port: 8000 2023-07-29T00:07:20.432 DEBUG:boto:Params: {} 2023-07-29T00:07:20.432 DEBUG:boto:Token: None 2023-07-29T00:07:20.432 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj4 2023-07-29T00:07:20.433 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3aKAu/amWqxLnWlzxHTHMCyuoOs= 2023-07-29T00:07:20.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3aKAu/amWqxLnWlzxHTHMCyuoOs=', 'Content-Length': '0'} 2023-07-29T00:07:20.435 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1c05173cb49b1a4-0064c45838-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.436 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:07:20.437 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:07:20.438 DEBUG:boto:path=/hiiqos-46/?prefix=obj4 2023-07-29T00:07:20.438 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj4 2023-07-29T00:07:20.439 DEBUG:boto:Method: GET 2023-07-29T00:07:20.439 DEBUG:boto:Path: /hiiqos-46/?prefix=obj4 2023-07-29T00:07:20.440 DEBUG:boto:Data: 2023-07-29T00:07:20.440 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.441 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:20.441 DEBUG:boto:Port: 8001 2023-07-29T00:07:20.441 DEBUG:boto:Params: {} 2023-07-29T00:07:20.442 DEBUG:boto:Token: None 2023-07-29T00:07:20.442 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/ 2023-07-29T00:07:20.443 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:l6TjkUNta5lXZRduHIrSC6nFgck= 2023-07-29T00:07:20.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:l6TjkUNta5lXZRduHIrSC6nFgck=', 'Content-Length': '0'} 2023-07-29T00:07:20.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040d6bf0ba1d764ed-0064c45838-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.447 DEBUG:boto:b'hiiqos-46obj41000falseobj42023-07-29T00:05:16.112Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:07:20.447 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:07:20.448 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:07:20.448 DEBUG:boto:path=/hiiqos-46/?prefix=obj4 2023-07-29T00:07:20.449 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj4 2023-07-29T00:07:20.449 DEBUG:boto:Method: GET 2023-07-29T00:07:20.449 DEBUG:boto:Path: /hiiqos-46/?prefix=obj4 2023-07-29T00:07:20.450 DEBUG:boto:Data: 2023-07-29T00:07:20.450 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.450 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:20.451 DEBUG:boto:Port: 8000 2023-07-29T00:07:20.451 DEBUG:boto:Params: {} 2023-07-29T00:07:20.451 DEBUG:boto:Token: None 2023-07-29T00:07:20.452 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/ 2023-07-29T00:07:20.452 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:l6TjkUNta5lXZRduHIrSC6nFgck= 2023-07-29T00:07:20.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:l6TjkUNta5lXZRduHIrSC6nFgck=', 'Content-Length': '0'} 2023-07-29T00:07:20.455 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d25ac79642631d43-0064c45838-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.456 DEBUG:boto:b'hiiqos-46obj41000falseobj42023-07-29T00:05:16.112Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:07:20.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-07-29T00:07:20.457 DEBUG:boto:path=/hiiqos-46/obj5 2023-07-29T00:07:20.457 DEBUG:boto:auth_path=/hiiqos-46/obj5 2023-07-29T00:07:20.458 DEBUG:boto:path=/hiiqos-46/obj5?versionId=null 2023-07-29T00:07:20.458 DEBUG:boto:auth_path=/hiiqos-46/obj5?versionId=null 2023-07-29T00:07:20.458 DEBUG:boto:Method: GET 2023-07-29T00:07:20.459 DEBUG:boto:Path: /hiiqos-46/obj5?versionId=null 2023-07-29T00:07:20.459 DEBUG:boto:Data: 2023-07-29T00:07:20.459 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.460 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:20.460 DEBUG:boto:Port: 8001 2023-07-29T00:07:20.460 DEBUG:boto:Params: {} 2023-07-29T00:07:20.461 DEBUG:boto:Token: None 2023-07-29T00:07:20.461 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj5?versionId=null 2023-07-29T00:07:20.462 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7KAiCfgh9Qgxt+raUwnmGcQsxHg= 2023-07-29T00:07:20.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7KAiCfgh9Qgxt+raUwnmGcQsxHg=', 'Content-Length': '0'} 2023-07-29T00:07:20.464 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000052e69bc3785f6ca4-0064c45838-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.465 DEBUG:boto:path=/hiiqos-46/obj5 2023-07-29T00:07:20.465 DEBUG:boto:auth_path=/hiiqos-46/obj5 2023-07-29T00:07:20.465 DEBUG:boto:path=/hiiqos-46/obj5?versionId=null 2023-07-29T00:07:20.466 DEBUG:boto:auth_path=/hiiqos-46/obj5?versionId=null 2023-07-29T00:07:20.466 DEBUG:boto:Method: GET 2023-07-29T00:07:20.467 DEBUG:boto:Path: /hiiqos-46/obj5?versionId=null 2023-07-29T00:07:20.467 DEBUG:boto:Data: 2023-07-29T00:07:20.467 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.468 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:20.468 DEBUG:boto:Port: 8000 2023-07-29T00:07:20.468 DEBUG:boto:Params: {} 2023-07-29T00:07:20.469 DEBUG:boto:Token: None 2023-07-29T00:07:20.470 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj5?versionId=null 2023-07-29T00:07:20.470 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7KAiCfgh9Qgxt+raUwnmGcQsxHg= 2023-07-29T00:07:20.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7KAiCfgh9Qgxt+raUwnmGcQsxHg=', 'Content-Length': '0'} 2023-07-29T00:07:20.472 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005dafd8a890b4828c-0064c45838-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.473 DEBUG:boto:path=/hiiqos-46/obj5 2023-07-29T00:07:20.474 DEBUG:boto:auth_path=/hiiqos-46/obj5 2023-07-29T00:07:20.474 DEBUG:boto:path=/hiiqos-46/obj5?versionId=null 2023-07-29T00:07:20.474 DEBUG:boto:auth_path=/hiiqos-46/obj5?versionId=null 2023-07-29T00:07:20.475 DEBUG:boto:Method: HEAD 2023-07-29T00:07:20.475 DEBUG:boto:Path: /hiiqos-46/obj5?versionId=null 2023-07-29T00:07:20.475 DEBUG:boto:Data: 2023-07-29T00:07:20.476 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.476 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:20.477 DEBUG:boto:Port: 8001 2023-07-29T00:07:20.477 DEBUG:boto:Params: {} 2023-07-29T00:07:20.477 DEBUG:boto:Token: None 2023-07-29T00:07:20.478 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj5?versionId=null 2023-07-29T00:07:20.478 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PUZgTiXUC8r3AO83xe8CuKHN9aA= 2023-07-29T00:07:20.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PUZgTiXUC8r3AO83xe8CuKHN9aA=', 'Content-Length': '0'} 2023-07-29T00:07:20.480 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e0ba8525e81065fc-0064c45838-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.481 DEBUG:boto:path=/hiiqos-46/obj5 2023-07-29T00:07:20.481 DEBUG:boto:auth_path=/hiiqos-46/obj5 2023-07-29T00:07:20.482 DEBUG:boto:path=/hiiqos-46/obj5?versionId=null 2023-07-29T00:07:20.482 DEBUG:boto:auth_path=/hiiqos-46/obj5?versionId=null 2023-07-29T00:07:20.482 DEBUG:boto:Method: HEAD 2023-07-29T00:07:20.483 DEBUG:boto:Path: /hiiqos-46/obj5?versionId=null 2023-07-29T00:07:20.483 DEBUG:boto:Data: 2023-07-29T00:07:20.483 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.484 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:20.484 DEBUG:boto:Port: 8000 2023-07-29T00:07:20.485 DEBUG:boto:Params: {} 2023-07-29T00:07:20.485 DEBUG:boto:Token: None 2023-07-29T00:07:20.486 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj5?versionId=null 2023-07-29T00:07:20.486 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PUZgTiXUC8r3AO83xe8CuKHN9aA= 2023-07-29T00:07:20.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PUZgTiXUC8r3AO83xe8CuKHN9aA=', 'Content-Length': '0'} 2023-07-29T00:07:20.488 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000072ac6003daf78b06-0064c45838-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-07-29T00:07:20.489 DEBUG:boto:path=/hiiqos-46/obj5 2023-07-29T00:07:20.490 DEBUG:boto:auth_path=/hiiqos-46/obj5 2023-07-29T00:07:20.490 DEBUG:boto:path=/hiiqos-46/obj5?versionId=null 2023-07-29T00:07:20.490 DEBUG:boto:auth_path=/hiiqos-46/obj5?versionId=null 2023-07-29T00:07:20.491 DEBUG:boto:Method: GET 2023-07-29T00:07:20.491 DEBUG:boto:Path: /hiiqos-46/obj5?versionId=null 2023-07-29T00:07:20.491 DEBUG:boto:Data: 2023-07-29T00:07:20.492 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.492 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:20.493 DEBUG:boto:Port: 8001 2023-07-29T00:07:20.493 DEBUG:boto:Params: {} 2023-07-29T00:07:20.493 DEBUG:boto:Token: None 2023-07-29T00:07:20.494 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj5?versionId=null 2023-07-29T00:07:20.494 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7KAiCfgh9Qgxt+raUwnmGcQsxHg= 2023-07-29T00:07:20.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7KAiCfgh9Qgxt+raUwnmGcQsxHg=', 'Content-Length': '0'} 2023-07-29T00:07:20.497 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b2c2ff78f51b8887-0064c45838-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.540 DEBUG:boto:path=/hiiqos-46/obj5 2023-07-29T00:07:20.541 DEBUG:boto:auth_path=/hiiqos-46/obj5 2023-07-29T00:07:20.541 DEBUG:boto:path=/hiiqos-46/obj5?versionId=null 2023-07-29T00:07:20.542 DEBUG:boto:auth_path=/hiiqos-46/obj5?versionId=null 2023-07-29T00:07:20.542 DEBUG:boto:Method: GET 2023-07-29T00:07:20.542 DEBUG:boto:Path: /hiiqos-46/obj5?versionId=null 2023-07-29T00:07:20.543 DEBUG:boto:Data: 2023-07-29T00:07:20.543 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.544 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:20.545 DEBUG:boto:Port: 8000 2023-07-29T00:07:20.545 DEBUG:boto:Params: {} 2023-07-29T00:07:20.545 DEBUG:boto:Token: None 2023-07-29T00:07:20.546 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj5?versionId=null 2023-07-29T00:07:20.546 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7KAiCfgh9Qgxt+raUwnmGcQsxHg= 2023-07-29T00:07:20.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7KAiCfgh9Qgxt+raUwnmGcQsxHg=', 'Content-Length': '0'} 2023-07-29T00:07:20.549 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003bb3e00b4da5665c-0064c45838-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.549 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:07:20.550 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:07:20.550 DEBUG:boto:path=/hiiqos-46/?prefix=obj5 2023-07-29T00:07:20.550 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj5 2023-07-29T00:07:20.551 DEBUG:boto:Method: GET 2023-07-29T00:07:20.551 DEBUG:boto:Path: /hiiqos-46/?prefix=obj5 2023-07-29T00:07:20.551 DEBUG:boto:Data: 2023-07-29T00:07:20.552 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.552 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:20.553 DEBUG:boto:Port: 8001 2023-07-29T00:07:20.553 DEBUG:boto:Params: {} 2023-07-29T00:07:20.553 DEBUG:boto:Token: None 2023-07-29T00:07:20.554 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/ 2023-07-29T00:07:20.554 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:l6TjkUNta5lXZRduHIrSC6nFgck= 2023-07-29T00:07:20.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:l6TjkUNta5lXZRduHIrSC6nFgck=', 'Content-Length': '0'} 2023-07-29T00:07:20.557 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4aa0317b8bcf371-0064c45838-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.558 DEBUG:boto:b'hiiqos-46obj51000falseobj52023-07-29T00:06:59.109Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:07:20.558 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:07:20.559 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:07:20.559 DEBUG:boto:path=/hiiqos-46/?prefix=obj5 2023-07-29T00:07:20.559 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj5 2023-07-29T00:07:20.560 DEBUG:boto:Method: GET 2023-07-29T00:07:20.560 DEBUG:boto:Path: /hiiqos-46/?prefix=obj5 2023-07-29T00:07:20.561 DEBUG:boto:Data: 2023-07-29T00:07:20.561 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.561 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:20.562 DEBUG:boto:Port: 8000 2023-07-29T00:07:20.562 DEBUG:boto:Params: {} 2023-07-29T00:07:20.562 DEBUG:boto:Token: None 2023-07-29T00:07:20.563 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/ 2023-07-29T00:07:20.563 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:l6TjkUNta5lXZRduHIrSC6nFgck= 2023-07-29T00:07:20.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:l6TjkUNta5lXZRduHIrSC6nFgck=', 'Content-Length': '0'} 2023-07-29T00:07:20.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf212ff8bb271a37-0064c45838-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.567 DEBUG:boto:b'hiiqos-46obj51000falseobj52023-07-29T00:06:59.109Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:07:20.567 DEBUG:boto:path=/hiiqos-46/obj5 2023-07-29T00:07:20.568 DEBUG:boto:auth_path=/hiiqos-46/obj5 2023-07-29T00:07:20.568 DEBUG:boto:Method: HEAD 2023-07-29T00:07:20.568 DEBUG:boto:Path: /hiiqos-46/obj5 2023-07-29T00:07:20.569 DEBUG:boto:Data: 2023-07-29T00:07:20.569 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.569 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:20.570 DEBUG:boto:Port: 8001 2023-07-29T00:07:20.570 DEBUG:boto:Params: {} 2023-07-29T00:07:20.570 DEBUG:boto:Token: None 2023-07-29T00:07:20.571 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj5 2023-07-29T00:07:20.571 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gFsun2kNa40bwzZTq1/l75GPExE= 2023-07-29T00:07:20.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gFsun2kNa40bwzZTq1/l75GPExE=', 'Content-Length': '0'} 2023-07-29T00:07:20.573 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a1427d1fa34a729-0064c45838-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.574 DEBUG:boto:path=/hiiqos-46/obj5 2023-07-29T00:07:20.574 DEBUG:boto:auth_path=/hiiqos-46/obj5 2023-07-29T00:07:20.575 DEBUG:boto:Method: HEAD 2023-07-29T00:07:20.575 DEBUG:boto:Path: /hiiqos-46/obj5 2023-07-29T00:07:20.575 DEBUG:boto:Data: 2023-07-29T00:07:20.576 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.576 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:20.576 DEBUG:boto:Port: 8000 2023-07-29T00:07:20.577 DEBUG:boto:Params: {} 2023-07-29T00:07:20.577 DEBUG:boto:Token: None 2023-07-29T00:07:20.577 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj5 2023-07-29T00:07:20.578 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gFsun2kNa40bwzZTq1/l75GPExE= 2023-07-29T00:07:20.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gFsun2kNa40bwzZTq1/l75GPExE=', 'Content-Length': '0'} 2023-07-29T00:07:20.580 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000074b1861912f14e2a-0064c45838-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-07-29T00:07:20.581 DEBUG:boto:path=/hiiqos-46/obj5 2023-07-29T00:07:20.581 DEBUG:boto:auth_path=/hiiqos-46/obj5 2023-07-29T00:07:20.582 DEBUG:boto:Method: GET 2023-07-29T00:07:20.582 DEBUG:boto:Path: /hiiqos-46/obj5 2023-07-29T00:07:20.583 DEBUG:boto:Data: 2023-07-29T00:07:20.583 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.583 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:20.584 DEBUG:boto:Port: 8001 2023-07-29T00:07:20.584 DEBUG:boto:Params: {} 2023-07-29T00:07:20.585 DEBUG:boto:Token: None 2023-07-29T00:07:20.585 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj5 2023-07-29T00:07:20.586 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+YYp9QHBtJsFuo1ohE4SXmhRl6o= 2023-07-29T00:07:20.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+YYp9QHBtJsFuo1ohE4SXmhRl6o=', 'Content-Length': '0'} 2023-07-29T00:07:20.589 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c4ea5fd3dacc481b-0064c45838-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.590 DEBUG:boto:path=/hiiqos-46/obj5 2023-07-29T00:07:20.591 DEBUG:boto:auth_path=/hiiqos-46/obj5 2023-07-29T00:07:20.592 DEBUG:boto:Method: GET 2023-07-29T00:07:20.592 DEBUG:boto:Path: /hiiqos-46/obj5 2023-07-29T00:07:20.593 DEBUG:boto:Data: 2023-07-29T00:07:20.593 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.594 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:20.594 DEBUG:boto:Port: 8000 2023-07-29T00:07:20.594 DEBUG:boto:Params: {} 2023-07-29T00:07:20.595 DEBUG:boto:Token: None 2023-07-29T00:07:20.595 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj5 2023-07-29T00:07:20.596 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+YYp9QHBtJsFuo1ohE4SXmhRl6o= 2023-07-29T00:07:20.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+YYp9QHBtJsFuo1ohE4SXmhRl6o=', 'Content-Length': '0'} 2023-07-29T00:07:20.598 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000225a8bd8c3bcdc9d-0064c45838-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.599 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:07:20.599 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:07:20.600 DEBUG:boto:path=/hiiqos-46/?prefix=obj5 2023-07-29T00:07:20.600 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj5 2023-07-29T00:07:20.600 DEBUG:boto:Method: GET 2023-07-29T00:07:20.601 DEBUG:boto:Path: /hiiqos-46/?prefix=obj5 2023-07-29T00:07:20.601 DEBUG:boto:Data: 2023-07-29T00:07:20.601 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.602 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:20.602 DEBUG:boto:Port: 8001 2023-07-29T00:07:20.603 DEBUG:boto:Params: {} 2023-07-29T00:07:20.603 DEBUG:boto:Token: None 2023-07-29T00:07:20.603 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/ 2023-07-29T00:07:20.604 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:l6TjkUNta5lXZRduHIrSC6nFgck= 2023-07-29T00:07:20.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:l6TjkUNta5lXZRduHIrSC6nFgck=', 'Content-Length': '0'} 2023-07-29T00:07:20.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009621b2cfa4404404-0064c45838-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.648 DEBUG:boto:b'hiiqos-46obj51000falseobj52023-07-29T00:06:59.109Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:07:20.649 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:07:20.650 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:07:20.650 DEBUG:boto:path=/hiiqos-46/?prefix=obj5 2023-07-29T00:07:20.650 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj5 2023-07-29T00:07:20.651 DEBUG:boto:Method: GET 2023-07-29T00:07:20.651 DEBUG:boto:Path: /hiiqos-46/?prefix=obj5 2023-07-29T00:07:20.651 DEBUG:boto:Data: 2023-07-29T00:07:20.652 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.652 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:20.653 DEBUG:boto:Port: 8000 2023-07-29T00:07:20.653 DEBUG:boto:Params: {} 2023-07-29T00:07:20.653 DEBUG:boto:Token: None 2023-07-29T00:07:20.654 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/ 2023-07-29T00:07:20.654 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:l6TjkUNta5lXZRduHIrSC6nFgck= 2023-07-29T00:07:20.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:l6TjkUNta5lXZRduHIrSC6nFgck=', 'Content-Length': '0'} 2023-07-29T00:07:20.657 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040e5deb5636a487b-0064c45838-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.657 DEBUG:boto:b'hiiqos-46obj51000falseobj52023-07-29T00:06:59.109Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:07:20.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-07-29T00:07:20.660 DEBUG:boto:path=/hiiqos-46/obj6 2023-07-29T00:07:20.661 DEBUG:boto:auth_path=/hiiqos-46/obj6 2023-07-29T00:07:20.661 DEBUG:boto:path=/hiiqos-46/obj6?versionId=null 2023-07-29T00:07:20.662 DEBUG:boto:auth_path=/hiiqos-46/obj6?versionId=null 2023-07-29T00:07:20.663 DEBUG:boto:Method: GET 2023-07-29T00:07:20.664 DEBUG:boto:Path: /hiiqos-46/obj6?versionId=null 2023-07-29T00:07:20.664 DEBUG:boto:Data: 2023-07-29T00:07:20.664 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.665 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:20.665 DEBUG:boto:Port: 8001 2023-07-29T00:07:20.665 DEBUG:boto:Params: {} 2023-07-29T00:07:20.666 DEBUG:boto:Token: None 2023-07-29T00:07:20.666 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj6?versionId=null 2023-07-29T00:07:20.667 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:inX8DFaSZRoZlbFvP4Pur0jhEWQ= 2023-07-29T00:07:20.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:inX8DFaSZRoZlbFvP4Pur0jhEWQ=', 'Content-Length': '0'} 2023-07-29T00:07:20.669 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000595a70ac9bce7b42-0064c45838-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.670 DEBUG:boto:path=/hiiqos-46/obj6 2023-07-29T00:07:20.670 DEBUG:boto:auth_path=/hiiqos-46/obj6 2023-07-29T00:07:20.671 DEBUG:boto:path=/hiiqos-46/obj6?versionId=null 2023-07-29T00:07:20.671 DEBUG:boto:auth_path=/hiiqos-46/obj6?versionId=null 2023-07-29T00:07:20.671 DEBUG:boto:Method: GET 2023-07-29T00:07:20.672 DEBUG:boto:Path: /hiiqos-46/obj6?versionId=null 2023-07-29T00:07:20.672 DEBUG:boto:Data: 2023-07-29T00:07:20.672 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.673 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:20.673 DEBUG:boto:Port: 8000 2023-07-29T00:07:20.674 DEBUG:boto:Params: {} 2023-07-29T00:07:20.674 DEBUG:boto:Token: None 2023-07-29T00:07:20.674 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj6?versionId=null 2023-07-29T00:07:20.675 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:inX8DFaSZRoZlbFvP4Pur0jhEWQ= 2023-07-29T00:07:20.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:inX8DFaSZRoZlbFvP4Pur0jhEWQ=', 'Content-Length': '0'} 2023-07-29T00:07:20.677 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000166961aae7711511-0064c45838-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.724 DEBUG:boto:path=/hiiqos-46/obj6 2023-07-29T00:07:20.725 DEBUG:boto:auth_path=/hiiqos-46/obj6 2023-07-29T00:07:20.725 DEBUG:boto:path=/hiiqos-46/obj6?versionId=null 2023-07-29T00:07:20.726 DEBUG:boto:auth_path=/hiiqos-46/obj6?versionId=null 2023-07-29T00:07:20.726 DEBUG:boto:Method: HEAD 2023-07-29T00:07:20.726 DEBUG:boto:Path: /hiiqos-46/obj6?versionId=null 2023-07-29T00:07:20.727 DEBUG:boto:Data: 2023-07-29T00:07:20.727 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.727 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:20.728 DEBUG:boto:Port: 8001 2023-07-29T00:07:20.728 DEBUG:boto:Params: {} 2023-07-29T00:07:20.728 DEBUG:boto:Token: None 2023-07-29T00:07:20.729 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj6?versionId=null 2023-07-29T00:07:20.729 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4b18/Ecbul91NrWvZvkxvQZmu0w= 2023-07-29T00:07:20.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4b18/Ecbul91NrWvZvkxvQZmu0w=', 'Content-Length': '0'} 2023-07-29T00:07:20.732 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000080dafe8d4f920b78-0064c45838-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.732 DEBUG:boto:path=/hiiqos-46/obj6 2023-07-29T00:07:20.733 DEBUG:boto:auth_path=/hiiqos-46/obj6 2023-07-29T00:07:20.733 DEBUG:boto:path=/hiiqos-46/obj6?versionId=null 2023-07-29T00:07:20.733 DEBUG:boto:auth_path=/hiiqos-46/obj6?versionId=null 2023-07-29T00:07:20.734 DEBUG:boto:Method: HEAD 2023-07-29T00:07:20.734 DEBUG:boto:Path: /hiiqos-46/obj6?versionId=null 2023-07-29T00:07:20.734 DEBUG:boto:Data: 2023-07-29T00:07:20.735 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.735 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:20.735 DEBUG:boto:Port: 8000 2023-07-29T00:07:20.736 DEBUG:boto:Params: {} 2023-07-29T00:07:20.736 DEBUG:boto:Token: None 2023-07-29T00:07:20.736 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj6?versionId=null 2023-07-29T00:07:20.737 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4b18/Ecbul91NrWvZvkxvQZmu0w= 2023-07-29T00:07:20.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4b18/Ecbul91NrWvZvkxvQZmu0w=', 'Content-Length': '0'} 2023-07-29T00:07:20.739 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000004d9fc8df33ff1ac-0064c45838-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-07-29T00:07:20.740 DEBUG:boto:path=/hiiqos-46/obj6 2023-07-29T00:07:20.741 DEBUG:boto:auth_path=/hiiqos-46/obj6 2023-07-29T00:07:20.741 DEBUG:boto:path=/hiiqos-46/obj6?versionId=null 2023-07-29T00:07:20.741 DEBUG:boto:auth_path=/hiiqos-46/obj6?versionId=null 2023-07-29T00:07:20.742 DEBUG:boto:Method: GET 2023-07-29T00:07:20.742 DEBUG:boto:Path: /hiiqos-46/obj6?versionId=null 2023-07-29T00:07:20.743 DEBUG:boto:Data: 2023-07-29T00:07:20.743 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.743 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:20.744 DEBUG:boto:Port: 8001 2023-07-29T00:07:20.744 DEBUG:boto:Params: {} 2023-07-29T00:07:20.744 DEBUG:boto:Token: None 2023-07-29T00:07:20.745 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj6?versionId=null 2023-07-29T00:07:20.745 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:inX8DFaSZRoZlbFvP4Pur0jhEWQ= 2023-07-29T00:07:20.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:inX8DFaSZRoZlbFvP4Pur0jhEWQ=', 'Content-Length': '0'} 2023-07-29T00:07:20.747 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d2d105b066585f40-0064c45838-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.748 DEBUG:boto:path=/hiiqos-46/obj6 2023-07-29T00:07:20.748 DEBUG:boto:auth_path=/hiiqos-46/obj6 2023-07-29T00:07:20.749 DEBUG:boto:path=/hiiqos-46/obj6?versionId=null 2023-07-29T00:07:20.749 DEBUG:boto:auth_path=/hiiqos-46/obj6?versionId=null 2023-07-29T00:07:20.750 DEBUG:boto:Method: GET 2023-07-29T00:07:20.750 DEBUG:boto:Path: /hiiqos-46/obj6?versionId=null 2023-07-29T00:07:20.750 DEBUG:boto:Data: 2023-07-29T00:07:20.751 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.751 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:20.751 DEBUG:boto:Port: 8000 2023-07-29T00:07:20.752 DEBUG:boto:Params: {} 2023-07-29T00:07:20.752 DEBUG:boto:Token: None 2023-07-29T00:07:20.753 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj6?versionId=null 2023-07-29T00:07:20.753 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:inX8DFaSZRoZlbFvP4Pur0jhEWQ= 2023-07-29T00:07:20.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:inX8DFaSZRoZlbFvP4Pur0jhEWQ=', 'Content-Length': '0'} 2023-07-29T00:07:20.755 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d3e4939b379ea749-0064c45838-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.756 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:07:20.756 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:07:20.757 DEBUG:boto:path=/hiiqos-46/?prefix=obj6 2023-07-29T00:07:20.757 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj6 2023-07-29T00:07:20.758 DEBUG:boto:Method: GET 2023-07-29T00:07:20.758 DEBUG:boto:Path: /hiiqos-46/?prefix=obj6 2023-07-29T00:07:20.758 DEBUG:boto:Data: 2023-07-29T00:07:20.759 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.759 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:20.759 DEBUG:boto:Port: 8001 2023-07-29T00:07:20.760 DEBUG:boto:Params: {} 2023-07-29T00:07:20.760 DEBUG:boto:Token: None 2023-07-29T00:07:20.761 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/ 2023-07-29T00:07:20.761 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:l6TjkUNta5lXZRduHIrSC6nFgck= 2023-07-29T00:07:20.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:l6TjkUNta5lXZRduHIrSC6nFgck=', 'Content-Length': '0'} 2023-07-29T00:07:20.763 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe0ff36d1a313ce4-0064c45838-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.764 DEBUG:boto:b'hiiqos-46obj61000falseobj62023-07-29T00:06:59.128Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:07:20.765 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:07:20.765 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:07:20.766 DEBUG:boto:path=/hiiqos-46/?prefix=obj6 2023-07-29T00:07:20.766 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj6 2023-07-29T00:07:20.766 DEBUG:boto:Method: GET 2023-07-29T00:07:20.767 DEBUG:boto:Path: /hiiqos-46/?prefix=obj6 2023-07-29T00:07:20.767 DEBUG:boto:Data: 2023-07-29T00:07:20.767 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.768 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:20.768 DEBUG:boto:Port: 8000 2023-07-29T00:07:20.768 DEBUG:boto:Params: {} 2023-07-29T00:07:20.769 DEBUG:boto:Token: None 2023-07-29T00:07:20.769 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/ 2023-07-29T00:07:20.770 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:l6TjkUNta5lXZRduHIrSC6nFgck= 2023-07-29T00:07:20.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:l6TjkUNta5lXZRduHIrSC6nFgck=', 'Content-Length': '0'} 2023-07-29T00:07:20.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095b4b5afbf37f9fa-0064c45838-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.773 DEBUG:boto:b'hiiqos-46obj61000falseobj62023-07-29T00:06:59.128Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:07:20.773 DEBUG:boto:path=/hiiqos-46/obj6 2023-07-29T00:07:20.774 DEBUG:boto:auth_path=/hiiqos-46/obj6 2023-07-29T00:07:20.774 DEBUG:boto:Method: HEAD 2023-07-29T00:07:20.774 DEBUG:boto:Path: /hiiqos-46/obj6 2023-07-29T00:07:20.775 DEBUG:boto:Data: 2023-07-29T00:07:20.775 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.775 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:20.776 DEBUG:boto:Port: 8001 2023-07-29T00:07:20.776 DEBUG:boto:Params: {} 2023-07-29T00:07:20.777 DEBUG:boto:Token: None 2023-07-29T00:07:20.777 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj6 2023-07-29T00:07:20.777 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/H9/6Lb6Th+omhYDRYofgWKvAR0= 2023-07-29T00:07:20.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/H9/6Lb6Th+omhYDRYofgWKvAR0=', 'Content-Length': '0'} 2023-07-29T00:07:20.779 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003947df2455f6ec1a-0064c45838-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.780 DEBUG:boto:path=/hiiqos-46/obj6 2023-07-29T00:07:20.780 DEBUG:boto:auth_path=/hiiqos-46/obj6 2023-07-29T00:07:20.781 DEBUG:boto:Method: HEAD 2023-07-29T00:07:20.781 DEBUG:boto:Path: /hiiqos-46/obj6 2023-07-29T00:07:20.781 DEBUG:boto:Data: 2023-07-29T00:07:20.782 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.782 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:20.782 DEBUG:boto:Port: 8000 2023-07-29T00:07:20.783 DEBUG:boto:Params: {} 2023-07-29T00:07:20.783 DEBUG:boto:Token: None 2023-07-29T00:07:20.784 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj6 2023-07-29T00:07:20.784 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/H9/6Lb6Th+omhYDRYofgWKvAR0= 2023-07-29T00:07:20.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/H9/6Lb6Th+omhYDRYofgWKvAR0=', 'Content-Length': '0'} 2023-07-29T00:07:20.786 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ddfc39dc3c88036d-0064c45838-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-07-29T00:07:20.787 DEBUG:boto:path=/hiiqos-46/obj6 2023-07-29T00:07:20.788 DEBUG:boto:auth_path=/hiiqos-46/obj6 2023-07-29T00:07:20.788 DEBUG:boto:Method: GET 2023-07-29T00:07:20.788 DEBUG:boto:Path: /hiiqos-46/obj6 2023-07-29T00:07:20.789 DEBUG:boto:Data: 2023-07-29T00:07:20.789 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.789 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:20.790 DEBUG:boto:Port: 8001 2023-07-29T00:07:20.790 DEBUG:boto:Params: {} 2023-07-29T00:07:20.790 DEBUG:boto:Token: None 2023-07-29T00:07:20.791 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj6 2023-07-29T00:07:20.791 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AGitQVxyapID3Y2rUjrv3s4JxEQ= 2023-07-29T00:07:20.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AGitQVxyapID3Y2rUjrv3s4JxEQ=', 'Content-Length': '0'} 2023-07-29T00:07:20.794 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e4faae69f7ba65a6-0064c45838-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.794 DEBUG:boto:path=/hiiqos-46/obj6 2023-07-29T00:07:20.795 DEBUG:boto:auth_path=/hiiqos-46/obj6 2023-07-29T00:07:20.795 DEBUG:boto:Method: GET 2023-07-29T00:07:20.796 DEBUG:boto:Path: /hiiqos-46/obj6 2023-07-29T00:07:20.796 DEBUG:boto:Data: 2023-07-29T00:07:20.796 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.797 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:20.797 DEBUG:boto:Port: 8000 2023-07-29T00:07:20.797 DEBUG:boto:Params: {} 2023-07-29T00:07:20.798 DEBUG:boto:Token: None 2023-07-29T00:07:20.798 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj6 2023-07-29T00:07:20.799 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AGitQVxyapID3Y2rUjrv3s4JxEQ= 2023-07-29T00:07:20.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AGitQVxyapID3Y2rUjrv3s4JxEQ=', 'Content-Length': '0'} 2023-07-29T00:07:20.801 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000088520340df618b29-0064c45838-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.802 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:07:20.802 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:07:20.803 DEBUG:boto:path=/hiiqos-46/?prefix=obj6 2023-07-29T00:07:20.804 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj6 2023-07-29T00:07:20.804 DEBUG:boto:Method: GET 2023-07-29T00:07:20.805 DEBUG:boto:Path: /hiiqos-46/?prefix=obj6 2023-07-29T00:07:20.805 DEBUG:boto:Data: 2023-07-29T00:07:20.805 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.806 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:20.806 DEBUG:boto:Port: 8001 2023-07-29T00:07:20.807 DEBUG:boto:Params: {} 2023-07-29T00:07:20.807 DEBUG:boto:Token: None 2023-07-29T00:07:20.807 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/ 2023-07-29T00:07:20.808 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:l6TjkUNta5lXZRduHIrSC6nFgck= 2023-07-29T00:07:20.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:l6TjkUNta5lXZRduHIrSC6nFgck=', 'Content-Length': '0'} 2023-07-29T00:07:20.811 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000416c2aa14315ff3d-0064c45838-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.811 DEBUG:boto:b'hiiqos-46obj61000falseobj62023-07-29T00:06:59.128Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:07:20.812 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:07:20.813 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:07:20.813 DEBUG:boto:path=/hiiqos-46/?prefix=obj6 2023-07-29T00:07:20.813 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj6 2023-07-29T00:07:20.814 DEBUG:boto:Method: GET 2023-07-29T00:07:20.814 DEBUG:boto:Path: /hiiqos-46/?prefix=obj6 2023-07-29T00:07:20.814 DEBUG:boto:Data: 2023-07-29T00:07:20.815 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.815 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:20.815 DEBUG:boto:Port: 8000 2023-07-29T00:07:20.816 DEBUG:boto:Params: {} 2023-07-29T00:07:20.816 DEBUG:boto:Token: None 2023-07-29T00:07:20.817 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/ 2023-07-29T00:07:20.817 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:l6TjkUNta5lXZRduHIrSC6nFgck= 2023-07-29T00:07:20.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:l6TjkUNta5lXZRduHIrSC6nFgck=', 'Content-Length': '0'} 2023-07-29T00:07:20.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032b6187dd72c45bc-0064c45838-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.860 DEBUG:boto:b'hiiqos-46obj61000falseobj62023-07-29T00:06:59.128Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:07:20.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-07-29T00:07:20.862 DEBUG:boto:path=/hiiqos-46/obj7 2023-07-29T00:07:20.862 DEBUG:boto:auth_path=/hiiqos-46/obj7 2023-07-29T00:07:20.863 DEBUG:boto:path=/hiiqos-46/obj7?versionId=null 2023-07-29T00:07:20.863 DEBUG:boto:auth_path=/hiiqos-46/obj7?versionId=null 2023-07-29T00:07:20.864 DEBUG:boto:Method: GET 2023-07-29T00:07:20.864 DEBUG:boto:Path: /hiiqos-46/obj7?versionId=null 2023-07-29T00:07:20.864 DEBUG:boto:Data: 2023-07-29T00:07:20.865 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.865 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:20.865 DEBUG:boto:Port: 8001 2023-07-29T00:07:20.866 DEBUG:boto:Params: {} 2023-07-29T00:07:20.866 DEBUG:boto:Token: None 2023-07-29T00:07:20.867 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj7?versionId=null 2023-07-29T00:07:20.867 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+H5VVyq5IiU9P6B80RIkDRYKjl0= 2023-07-29T00:07:20.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+H5VVyq5IiU9P6B80RIkDRYKjl0=', 'Content-Length': '0'} 2023-07-29T00:07:20.870 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bdbfec5d8f713cff-0064c45838-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.871 DEBUG:boto:path=/hiiqos-46/obj7 2023-07-29T00:07:20.871 DEBUG:boto:auth_path=/hiiqos-46/obj7 2023-07-29T00:07:20.872 DEBUG:boto:path=/hiiqos-46/obj7?versionId=null 2023-07-29T00:07:20.872 DEBUG:boto:auth_path=/hiiqos-46/obj7?versionId=null 2023-07-29T00:07:20.873 DEBUG:boto:Method: GET 2023-07-29T00:07:20.873 DEBUG:boto:Path: /hiiqos-46/obj7?versionId=null 2023-07-29T00:07:20.874 DEBUG:boto:Data: 2023-07-29T00:07:20.875 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.875 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:20.876 DEBUG:boto:Port: 8000 2023-07-29T00:07:20.876 DEBUG:boto:Params: {} 2023-07-29T00:07:20.877 DEBUG:boto:Token: None 2023-07-29T00:07:20.878 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj7?versionId=null 2023-07-29T00:07:20.878 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+H5VVyq5IiU9P6B80RIkDRYKjl0= 2023-07-29T00:07:20.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+H5VVyq5IiU9P6B80RIkDRYKjl0=', 'Content-Length': '0'} 2023-07-29T00:07:20.881 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006c55502af8b0e4cb-0064c45838-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.882 DEBUG:boto:path=/hiiqos-46/obj7 2023-07-29T00:07:20.882 DEBUG:boto:auth_path=/hiiqos-46/obj7 2023-07-29T00:07:20.883 DEBUG:boto:path=/hiiqos-46/obj7?versionId=null 2023-07-29T00:07:20.883 DEBUG:boto:auth_path=/hiiqos-46/obj7?versionId=null 2023-07-29T00:07:20.883 DEBUG:boto:Method: HEAD 2023-07-29T00:07:20.884 DEBUG:boto:Path: /hiiqos-46/obj7?versionId=null 2023-07-29T00:07:20.884 DEBUG:boto:Data: 2023-07-29T00:07:20.884 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.885 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:20.885 DEBUG:boto:Port: 8001 2023-07-29T00:07:20.885 DEBUG:boto:Params: {} 2023-07-29T00:07:20.886 DEBUG:boto:Token: None 2023-07-29T00:07:20.886 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj7?versionId=null 2023-07-29T00:07:20.887 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3wmbYxIHap/RQ44ou5LDzi/CAYk= 2023-07-29T00:07:20.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3wmbYxIHap/RQ44ou5LDzi/CAYk=', 'Content-Length': '0'} 2023-07-29T00:07:20.889 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e08fe27f0a822e30-0064c45838-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.889 DEBUG:boto:path=/hiiqos-46/obj7 2023-07-29T00:07:20.890 DEBUG:boto:auth_path=/hiiqos-46/obj7 2023-07-29T00:07:20.890 DEBUG:boto:path=/hiiqos-46/obj7?versionId=null 2023-07-29T00:07:20.890 DEBUG:boto:auth_path=/hiiqos-46/obj7?versionId=null 2023-07-29T00:07:20.891 DEBUG:boto:Method: HEAD 2023-07-29T00:07:20.891 DEBUG:boto:Path: /hiiqos-46/obj7?versionId=null 2023-07-29T00:07:20.891 DEBUG:boto:Data: 2023-07-29T00:07:20.892 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.892 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:20.892 DEBUG:boto:Port: 8000 2023-07-29T00:07:20.893 DEBUG:boto:Params: {} 2023-07-29T00:07:20.893 DEBUG:boto:Token: None 2023-07-29T00:07:20.893 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj7?versionId=null 2023-07-29T00:07:20.894 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3wmbYxIHap/RQ44ou5LDzi/CAYk= 2023-07-29T00:07:20.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3wmbYxIHap/RQ44ou5LDzi/CAYk=', 'Content-Length': '0'} 2023-07-29T00:07:20.896 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006abdf23d0ddef25d-0064c45838-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-07-29T00:07:20.897 DEBUG:boto:path=/hiiqos-46/obj7 2023-07-29T00:07:20.898 DEBUG:boto:auth_path=/hiiqos-46/obj7 2023-07-29T00:07:20.898 DEBUG:boto:path=/hiiqos-46/obj7?versionId=null 2023-07-29T00:07:20.898 DEBUG:boto:auth_path=/hiiqos-46/obj7?versionId=null 2023-07-29T00:07:20.899 DEBUG:boto:Method: GET 2023-07-29T00:07:20.899 DEBUG:boto:Path: /hiiqos-46/obj7?versionId=null 2023-07-29T00:07:20.899 DEBUG:boto:Data: 2023-07-29T00:07:20.900 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.900 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:20.900 DEBUG:boto:Port: 8001 2023-07-29T00:07:20.901 DEBUG:boto:Params: {} 2023-07-29T00:07:20.901 DEBUG:boto:Token: None 2023-07-29T00:07:20.902 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj7?versionId=null 2023-07-29T00:07:20.902 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+H5VVyq5IiU9P6B80RIkDRYKjl0= 2023-07-29T00:07:20.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+H5VVyq5IiU9P6B80RIkDRYKjl0=', 'Content-Length': '0'} 2023-07-29T00:07:20.905 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f379cdddb59436e-0064c45838-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.905 DEBUG:boto:path=/hiiqos-46/obj7 2023-07-29T00:07:20.906 DEBUG:boto:auth_path=/hiiqos-46/obj7 2023-07-29T00:07:20.906 DEBUG:boto:path=/hiiqos-46/obj7?versionId=null 2023-07-29T00:07:20.906 DEBUG:boto:auth_path=/hiiqos-46/obj7?versionId=null 2023-07-29T00:07:20.907 DEBUG:boto:Method: GET 2023-07-29T00:07:20.908 DEBUG:boto:Path: /hiiqos-46/obj7?versionId=null 2023-07-29T00:07:20.908 DEBUG:boto:Data: 2023-07-29T00:07:20.909 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.909 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:20.909 DEBUG:boto:Port: 8000 2023-07-29T00:07:20.910 DEBUG:boto:Params: {} 2023-07-29T00:07:20.910 DEBUG:boto:Token: None 2023-07-29T00:07:20.910 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj7?versionId=null 2023-07-29T00:07:20.911 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+H5VVyq5IiU9P6B80RIkDRYKjl0= 2023-07-29T00:07:20.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+H5VVyq5IiU9P6B80RIkDRYKjl0=', 'Content-Length': '0'} 2023-07-29T00:07:20.913 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a6eaef75f8755283-0064c45838-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.956 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:07:20.957 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:07:20.957 DEBUG:boto:path=/hiiqos-46/?prefix=obj7 2023-07-29T00:07:20.958 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj7 2023-07-29T00:07:20.958 DEBUG:boto:Method: GET 2023-07-29T00:07:20.958 DEBUG:boto:Path: /hiiqos-46/?prefix=obj7 2023-07-29T00:07:20.959 DEBUG:boto:Data: 2023-07-29T00:07:20.959 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.959 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:20.960 DEBUG:boto:Port: 8001 2023-07-29T00:07:20.960 DEBUG:boto:Params: {} 2023-07-29T00:07:20.961 DEBUG:boto:Token: None 2023-07-29T00:07:20.961 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/ 2023-07-29T00:07:20.962 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:l6TjkUNta5lXZRduHIrSC6nFgck= 2023-07-29T00:07:20.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:l6TjkUNta5lXZRduHIrSC6nFgck=', 'Content-Length': '0'} 2023-07-29T00:07:20.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026884d5562f98463-0064c45838-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.965 DEBUG:boto:b'hiiqos-46obj71000falseobj72023-07-29T00:06:59.145Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:07:20.965 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:07:20.966 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:07:20.966 DEBUG:boto:path=/hiiqos-46/?prefix=obj7 2023-07-29T00:07:20.967 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj7 2023-07-29T00:07:20.967 DEBUG:boto:Method: GET 2023-07-29T00:07:20.967 DEBUG:boto:Path: /hiiqos-46/?prefix=obj7 2023-07-29T00:07:20.968 DEBUG:boto:Data: 2023-07-29T00:07:20.968 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.968 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:20.969 DEBUG:boto:Port: 8000 2023-07-29T00:07:20.969 DEBUG:boto:Params: {} 2023-07-29T00:07:20.969 DEBUG:boto:Token: None 2023-07-29T00:07:20.970 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/ 2023-07-29T00:07:20.970 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:l6TjkUNta5lXZRduHIrSC6nFgck= 2023-07-29T00:07:20.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:l6TjkUNta5lXZRduHIrSC6nFgck=', 'Content-Length': '0'} 2023-07-29T00:07:20.973 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e261788d50e9b3c-0064c45838-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.974 DEBUG:boto:b'hiiqos-46obj71000falseobj72023-07-29T00:06:59.145Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:07:20.975 DEBUG:boto:path=/hiiqos-46/obj7 2023-07-29T00:07:20.975 DEBUG:boto:auth_path=/hiiqos-46/obj7 2023-07-29T00:07:20.976 DEBUG:boto:Method: HEAD 2023-07-29T00:07:20.977 DEBUG:boto:Path: /hiiqos-46/obj7 2023-07-29T00:07:20.978 DEBUG:boto:Data: 2023-07-29T00:07:20.978 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.979 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:20.980 DEBUG:boto:Port: 8001 2023-07-29T00:07:20.981 DEBUG:boto:Params: {} 2023-07-29T00:07:20.982 DEBUG:boto:Token: None 2023-07-29T00:07:20.982 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj7 2023-07-29T00:07:20.983 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yh0dChX864s+ym/fxoSI/VJMoXU= 2023-07-29T00:07:20.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yh0dChX864s+ym/fxoSI/VJMoXU=', 'Content-Length': '0'} 2023-07-29T00:07:20.985 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000052e3b5db7be33f91-0064c45838-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.986 DEBUG:boto:path=/hiiqos-46/obj7 2023-07-29T00:07:20.986 DEBUG:boto:auth_path=/hiiqos-46/obj7 2023-07-29T00:07:20.986 DEBUG:boto:Method: HEAD 2023-07-29T00:07:20.987 DEBUG:boto:Path: /hiiqos-46/obj7 2023-07-29T00:07:20.987 DEBUG:boto:Data: 2023-07-29T00:07:20.987 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.988 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:20.988 DEBUG:boto:Port: 8000 2023-07-29T00:07:20.988 DEBUG:boto:Params: {} 2023-07-29T00:07:20.989 DEBUG:boto:Token: None 2023-07-29T00:07:20.989 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj7 2023-07-29T00:07:20.990 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yh0dChX864s+ym/fxoSI/VJMoXU= 2023-07-29T00:07:20.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yh0dChX864s+ym/fxoSI/VJMoXU=', 'Content-Length': '0'} 2023-07-29T00:07:20.992 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000005c8b2065f1ccf50-0064c45838-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:20.992 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-07-29T00:07:20.993 DEBUG:boto:path=/hiiqos-46/obj7 2023-07-29T00:07:20.993 DEBUG:boto:auth_path=/hiiqos-46/obj7 2023-07-29T00:07:20.994 DEBUG:boto:Method: GET 2023-07-29T00:07:20.994 DEBUG:boto:Path: /hiiqos-46/obj7 2023-07-29T00:07:20.994 DEBUG:boto:Data: 2023-07-29T00:07:20.995 DEBUG:boto:Headers: {} 2023-07-29T00:07:20.995 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:20.995 DEBUG:boto:Port: 8001 2023-07-29T00:07:20.996 DEBUG:boto:Params: {} 2023-07-29T00:07:20.997 DEBUG:boto:Token: None 2023-07-29T00:07:20.997 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:20 GMT /hiiqos-46/obj7 2023-07-29T00:07:20.998 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:h6F9iDJj9V6lKmbpmC6O0y4T9es= 2023-07-29T00:07:20.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:h6F9iDJj9V6lKmbpmC6O0y4T9es=', 'Content-Length': '0'} 2023-07-29T00:07:21.000 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005669b46d91e38f62-0064c45838-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:21.044 DEBUG:boto:path=/hiiqos-46/obj7 2023-07-29T00:07:21.045 DEBUG:boto:auth_path=/hiiqos-46/obj7 2023-07-29T00:07:21.046 DEBUG:boto:Method: GET 2023-07-29T00:07:21.047 DEBUG:boto:Path: /hiiqos-46/obj7 2023-07-29T00:07:21.048 DEBUG:boto:Data: 2023-07-29T00:07:21.049 DEBUG:boto:Headers: {} 2023-07-29T00:07:21.050 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:21.050 DEBUG:boto:Port: 8000 2023-07-29T00:07:21.051 DEBUG:boto:Params: {} 2023-07-29T00:07:21.051 DEBUG:boto:Token: None 2023-07-29T00:07:21.052 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:21 GMT /hiiqos-46/obj7 2023-07-29T00:07:21.052 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0/FjbOwZp1f7Na2wrg09EqMAHRE= 2023-07-29T00:07:21.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0/FjbOwZp1f7Na2wrg09EqMAHRE=', 'Content-Length': '0'} 2023-07-29T00:07:21.055 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f115656fa322db9-0064c45839-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:21.056 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:07:21.056 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:07:21.057 DEBUG:boto:path=/hiiqos-46/?prefix=obj7 2023-07-29T00:07:21.057 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj7 2023-07-29T00:07:21.057 DEBUG:boto:Method: GET 2023-07-29T00:07:21.058 DEBUG:boto:Path: /hiiqos-46/?prefix=obj7 2023-07-29T00:07:21.058 DEBUG:boto:Data: 2023-07-29T00:07:21.058 DEBUG:boto:Headers: {} 2023-07-29T00:07:21.059 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:21.059 DEBUG:boto:Port: 8001 2023-07-29T00:07:21.059 DEBUG:boto:Params: {} 2023-07-29T00:07:21.060 DEBUG:boto:Token: None 2023-07-29T00:07:21.061 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:21 GMT /hiiqos-46/ 2023-07-29T00:07:21.061 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6Udmp0iiAVR87H38Pm2zk+gmeuM= 2023-07-29T00:07:21.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6Udmp0iiAVR87H38Pm2zk+gmeuM=', 'Content-Length': '0'} 2023-07-29T00:07:21.064 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdc7cef39aac84fb-0064c45839-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:07:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:21.064 DEBUG:boto:b'hiiqos-46obj71000falseobj72023-07-29T00:06:59.145Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:07:21.065 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:07:21.065 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:07:21.066 DEBUG:boto:path=/hiiqos-46/?prefix=obj7 2023-07-29T00:07:21.066 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj7 2023-07-29T00:07:21.066 DEBUG:boto:Method: GET 2023-07-29T00:07:21.067 DEBUG:boto:Path: /hiiqos-46/?prefix=obj7 2023-07-29T00:07:21.067 DEBUG:boto:Data: 2023-07-29T00:07:21.067 DEBUG:boto:Headers: {} 2023-07-29T00:07:21.068 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:21.068 DEBUG:boto:Port: 8000 2023-07-29T00:07:21.069 DEBUG:boto:Params: {} 2023-07-29T00:07:21.069 DEBUG:boto:Token: None 2023-07-29T00:07:21.069 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:21 GMT /hiiqos-46/ 2023-07-29T00:07:21.070 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6Udmp0iiAVR87H38Pm2zk+gmeuM= 2023-07-29T00:07:21.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6Udmp0iiAVR87H38Pm2zk+gmeuM=', 'Content-Length': '0'} 2023-07-29T00:07:21.073 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098332bc9ae356e0d-0064c45839-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:07:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:21.073 DEBUG:boto:b'hiiqos-46obj71000falseobj72023-07-29T00:06:59.145Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:07:21.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-07-29T00:07:21.074 DEBUG:boto:path=/hiiqos-46/obj8 2023-07-29T00:07:21.075 DEBUG:boto:auth_path=/hiiqos-46/obj8 2023-07-29T00:07:21.075 DEBUG:boto:path=/hiiqos-46/obj8?versionId=null 2023-07-29T00:07:21.075 DEBUG:boto:auth_path=/hiiqos-46/obj8?versionId=null 2023-07-29T00:07:21.076 DEBUG:boto:Method: GET 2023-07-29T00:07:21.076 DEBUG:boto:Path: /hiiqos-46/obj8?versionId=null 2023-07-29T00:07:21.076 DEBUG:boto:Data: 2023-07-29T00:07:21.077 DEBUG:boto:Headers: {} 2023-07-29T00:07:21.077 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:21.077 DEBUG:boto:Port: 8001 2023-07-29T00:07:21.078 DEBUG:boto:Params: {} 2023-07-29T00:07:21.078 DEBUG:boto:Token: None 2023-07-29T00:07:21.079 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:21 GMT /hiiqos-46/obj8?versionId=null 2023-07-29T00:07:21.079 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kq2mlW9/mHuCT0UUF86llRseRRM= 2023-07-29T00:07:21.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kq2mlW9/mHuCT0UUF86llRseRRM=', 'Content-Length': '0'} 2023-07-29T00:07:21.082 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000033db5bbe3d5bb90e-0064c45839-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:21.082 DEBUG:boto:path=/hiiqos-46/obj8 2023-07-29T00:07:21.083 DEBUG:boto:auth_path=/hiiqos-46/obj8 2023-07-29T00:07:21.083 DEBUG:boto:path=/hiiqos-46/obj8?versionId=null 2023-07-29T00:07:21.084 DEBUG:boto:auth_path=/hiiqos-46/obj8?versionId=null 2023-07-29T00:07:21.084 DEBUG:boto:Method: GET 2023-07-29T00:07:21.084 DEBUG:boto:Path: /hiiqos-46/obj8?versionId=null 2023-07-29T00:07:21.085 DEBUG:boto:Data: 2023-07-29T00:07:21.085 DEBUG:boto:Headers: {} 2023-07-29T00:07:21.086 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:21.086 DEBUG:boto:Port: 8000 2023-07-29T00:07:21.086 DEBUG:boto:Params: {} 2023-07-29T00:07:21.087 DEBUG:boto:Token: None 2023-07-29T00:07:21.087 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:21 GMT /hiiqos-46/obj8?versionId=null 2023-07-29T00:07:21.088 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kq2mlW9/mHuCT0UUF86llRseRRM= 2023-07-29T00:07:21.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kq2mlW9/mHuCT0UUF86llRseRRM=', 'Content-Length': '0'} 2023-07-29T00:07:21.090 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe3970dee160945d-0064c45839-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:21.132 DEBUG:boto:path=/hiiqos-46/obj8 2023-07-29T00:07:21.133 DEBUG:boto:auth_path=/hiiqos-46/obj8 2023-07-29T00:07:21.133 DEBUG:boto:path=/hiiqos-46/obj8?versionId=null 2023-07-29T00:07:21.134 DEBUG:boto:auth_path=/hiiqos-46/obj8?versionId=null 2023-07-29T00:07:21.134 DEBUG:boto:Method: HEAD 2023-07-29T00:07:21.134 DEBUG:boto:Path: /hiiqos-46/obj8?versionId=null 2023-07-29T00:07:21.135 DEBUG:boto:Data: 2023-07-29T00:07:21.135 DEBUG:boto:Headers: {} 2023-07-29T00:07:21.135 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:21.136 DEBUG:boto:Port: 8001 2023-07-29T00:07:21.136 DEBUG:boto:Params: {} 2023-07-29T00:07:21.136 DEBUG:boto:Token: None 2023-07-29T00:07:21.137 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:07:21 GMT /hiiqos-46/obj8?versionId=null 2023-07-29T00:07:21.137 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hn3Ux3gZFEhaCDVll9tHoJ41lss= 2023-07-29T00:07:21.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hn3Ux3gZFEhaCDVll9tHoJ41lss=', 'Content-Length': '0'} 2023-07-29T00:07:21.140 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f9aab27884cbcb2-0064c45839-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:21.140 DEBUG:boto:path=/hiiqos-46/obj8 2023-07-29T00:07:21.141 DEBUG:boto:auth_path=/hiiqos-46/obj8 2023-07-29T00:07:21.141 DEBUG:boto:path=/hiiqos-46/obj8?versionId=null 2023-07-29T00:07:21.141 DEBUG:boto:auth_path=/hiiqos-46/obj8?versionId=null 2023-07-29T00:07:21.142 DEBUG:boto:Method: HEAD 2023-07-29T00:07:21.142 DEBUG:boto:Path: /hiiqos-46/obj8?versionId=null 2023-07-29T00:07:21.142 DEBUG:boto:Data: 2023-07-29T00:07:21.143 DEBUG:boto:Headers: {} 2023-07-29T00:07:21.143 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:21.143 DEBUG:boto:Port: 8000 2023-07-29T00:07:21.144 DEBUG:boto:Params: {} 2023-07-29T00:07:21.144 DEBUG:boto:Token: None 2023-07-29T00:07:21.144 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:07:21 GMT /hiiqos-46/obj8?versionId=null 2023-07-29T00:07:21.145 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hn3Ux3gZFEhaCDVll9tHoJ41lss= 2023-07-29T00:07:21.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hn3Ux3gZFEhaCDVll9tHoJ41lss=', 'Content-Length': '0'} 2023-07-29T00:07:21.147 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000da82a40c83b941bc-0064c45839-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:21.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-07-29T00:07:21.149 DEBUG:boto:path=/hiiqos-46/obj8 2023-07-29T00:07:21.150 DEBUG:boto:auth_path=/hiiqos-46/obj8 2023-07-29T00:07:21.150 DEBUG:boto:path=/hiiqos-46/obj8?versionId=null 2023-07-29T00:07:21.151 DEBUG:boto:auth_path=/hiiqos-46/obj8?versionId=null 2023-07-29T00:07:21.152 DEBUG:boto:Method: GET 2023-07-29T00:07:21.153 DEBUG:boto:Path: /hiiqos-46/obj8?versionId=null 2023-07-29T00:07:21.153 DEBUG:boto:Data: 2023-07-29T00:07:21.154 DEBUG:boto:Headers: {} 2023-07-29T00:07:21.155 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:21.155 DEBUG:boto:Port: 8001 2023-07-29T00:07:21.156 DEBUG:boto:Params: {} 2023-07-29T00:07:21.157 DEBUG:boto:Token: None 2023-07-29T00:07:21.157 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:21 GMT /hiiqos-46/obj8?versionId=null 2023-07-29T00:07:21.158 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kq2mlW9/mHuCT0UUF86llRseRRM= 2023-07-29T00:07:21.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kq2mlW9/mHuCT0UUF86llRseRRM=', 'Content-Length': '0'} 2023-07-29T00:07:21.161 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005ecb3b63e624bebc-0064c45839-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:21.162 DEBUG:boto:path=/hiiqos-46/obj8 2023-07-29T00:07:21.163 DEBUG:boto:auth_path=/hiiqos-46/obj8 2023-07-29T00:07:21.163 DEBUG:boto:path=/hiiqos-46/obj8?versionId=null 2023-07-29T00:07:21.164 DEBUG:boto:auth_path=/hiiqos-46/obj8?versionId=null 2023-07-29T00:07:21.165 DEBUG:boto:Method: GET 2023-07-29T00:07:21.165 DEBUG:boto:Path: /hiiqos-46/obj8?versionId=null 2023-07-29T00:07:21.166 DEBUG:boto:Data: 2023-07-29T00:07:21.167 DEBUG:boto:Headers: {} 2023-07-29T00:07:21.167 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:21.168 DEBUG:boto:Port: 8000 2023-07-29T00:07:21.169 DEBUG:boto:Params: {} 2023-07-29T00:07:21.169 DEBUG:boto:Token: None 2023-07-29T00:07:21.170 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:21 GMT /hiiqos-46/obj8?versionId=null 2023-07-29T00:07:21.171 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kq2mlW9/mHuCT0UUF86llRseRRM= 2023-07-29T00:07:21.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kq2mlW9/mHuCT0UUF86llRseRRM=', 'Content-Length': '0'} 2023-07-29T00:07:21.174 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000aba8be38f8226ea-0064c45839-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:21.175 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:07:21.176 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:07:21.177 DEBUG:boto:path=/hiiqos-46/?prefix=obj8 2023-07-29T00:07:21.177 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj8 2023-07-29T00:07:21.178 DEBUG:boto:Method: GET 2023-07-29T00:07:21.179 DEBUG:boto:Path: /hiiqos-46/?prefix=obj8 2023-07-29T00:07:21.179 DEBUG:boto:Data: 2023-07-29T00:07:21.180 DEBUG:boto:Headers: {} 2023-07-29T00:07:21.180 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:21.181 DEBUG:boto:Port: 8001 2023-07-29T00:07:21.182 DEBUG:boto:Params: {} 2023-07-29T00:07:21.183 DEBUG:boto:Token: None 2023-07-29T00:07:21.183 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:21 GMT /hiiqos-46/ 2023-07-29T00:07:21.184 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6Udmp0iiAVR87H38Pm2zk+gmeuM= 2023-07-29T00:07:21.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6Udmp0iiAVR87H38Pm2zk+gmeuM=', 'Content-Length': '0'} 2023-07-29T00:07:21.188 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f76f54a3fe823358-0064c45839-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:07:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:21.188 DEBUG:boto:b'hiiqos-46obj81000falseobj82023-07-29T00:06:59.202Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:07:21.190 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:07:21.190 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:07:21.191 DEBUG:boto:path=/hiiqos-46/?prefix=obj8 2023-07-29T00:07:21.191 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj8 2023-07-29T00:07:21.192 DEBUG:boto:Method: GET 2023-07-29T00:07:21.193 DEBUG:boto:Path: /hiiqos-46/?prefix=obj8 2023-07-29T00:07:21.193 DEBUG:boto:Data: 2023-07-29T00:07:21.194 DEBUG:boto:Headers: {} 2023-07-29T00:07:21.195 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:21.195 DEBUG:boto:Port: 8000 2023-07-29T00:07:21.196 DEBUG:boto:Params: {} 2023-07-29T00:07:21.196 DEBUG:boto:Token: None 2023-07-29T00:07:21.197 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:21 GMT /hiiqos-46/ 2023-07-29T00:07:21.197 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6Udmp0iiAVR87H38Pm2zk+gmeuM= 2023-07-29T00:07:21.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6Udmp0iiAVR87H38Pm2zk+gmeuM=', 'Content-Length': '0'} 2023-07-29T00:07:21.200 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d3a6bfb7826a17b-0064c45839-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:07:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:21.201 DEBUG:boto:b'hiiqos-46obj81000falseobj82023-07-29T00:06:59.202Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:07:21.201 DEBUG:boto:path=/hiiqos-46/obj8 2023-07-29T00:07:21.202 DEBUG:boto:auth_path=/hiiqos-46/obj8 2023-07-29T00:07:21.202 DEBUG:boto:Method: HEAD 2023-07-29T00:07:21.202 DEBUG:boto:Path: /hiiqos-46/obj8 2023-07-29T00:07:21.203 DEBUG:boto:Data: 2023-07-29T00:07:21.203 DEBUG:boto:Headers: {} 2023-07-29T00:07:21.203 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:21.204 DEBUG:boto:Port: 8001 2023-07-29T00:07:21.204 DEBUG:boto:Params: {} 2023-07-29T00:07:21.204 DEBUG:boto:Token: None 2023-07-29T00:07:21.205 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:07:21 GMT /hiiqos-46/obj8 2023-07-29T00:07:21.205 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+vZhc6BHtzfjfVLtF1J3Wh0Ajbs= 2023-07-29T00:07:21.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+vZhc6BHtzfjfVLtF1J3Wh0Ajbs=', 'Content-Length': '0'} 2023-07-29T00:07:21.207 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006bd94cf35f328fc1-0064c45839-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:21.208 DEBUG:boto:path=/hiiqos-46/obj8 2023-07-29T00:07:21.208 DEBUG:boto:auth_path=/hiiqos-46/obj8 2023-07-29T00:07:21.209 DEBUG:boto:Method: HEAD 2023-07-29T00:07:21.209 DEBUG:boto:Path: /hiiqos-46/obj8 2023-07-29T00:07:21.209 DEBUG:boto:Data: 2023-07-29T00:07:21.210 DEBUG:boto:Headers: {} 2023-07-29T00:07:21.210 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:21.210 DEBUG:boto:Port: 8000 2023-07-29T00:07:21.211 DEBUG:boto:Params: {} 2023-07-29T00:07:21.211 DEBUG:boto:Token: None 2023-07-29T00:07:21.211 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:07:21 GMT /hiiqos-46/obj8 2023-07-29T00:07:21.212 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+vZhc6BHtzfjfVLtF1J3Wh0Ajbs= 2023-07-29T00:07:21.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+vZhc6BHtzfjfVLtF1J3Wh0Ajbs=', 'Content-Length': '0'} 2023-07-29T00:07:21.214 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aa07458ddeb480a4-0064c45839-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:21.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-07-29T00:07:21.215 DEBUG:boto:path=/hiiqos-46/obj8 2023-07-29T00:07:21.215 DEBUG:boto:auth_path=/hiiqos-46/obj8 2023-07-29T00:07:21.216 DEBUG:boto:Method: GET 2023-07-29T00:07:21.216 DEBUG:boto:Path: /hiiqos-46/obj8 2023-07-29T00:07:21.216 DEBUG:boto:Data: 2023-07-29T00:07:21.217 DEBUG:boto:Headers: {} 2023-07-29T00:07:21.217 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:21.217 DEBUG:boto:Port: 8001 2023-07-29T00:07:21.218 DEBUG:boto:Params: {} 2023-07-29T00:07:21.218 DEBUG:boto:Token: None 2023-07-29T00:07:21.219 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:21 GMT /hiiqos-46/obj8 2023-07-29T00:07:21.219 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:E2WuUpLJBH1Yf8di+7IeWM9rn/E= 2023-07-29T00:07:21.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:E2WuUpLJBH1Yf8di+7IeWM9rn/E=', 'Content-Length': '0'} 2023-07-29T00:07:21.221 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-46:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b7e19c779a4b2ad8-0064c45839-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:21.222 DEBUG:boto:path=/hiiqos-46/obj8 2023-07-29T00:07:21.222 DEBUG:boto:auth_path=/hiiqos-46/obj8 2023-07-29T00:07:21.223 DEBUG:boto:Method: GET 2023-07-29T00:07:21.223 DEBUG:boto:Path: /hiiqos-46/obj8 2023-07-29T00:07:21.223 DEBUG:boto:Data: 2023-07-29T00:07:21.224 DEBUG:boto:Headers: {} 2023-07-29T00:07:21.224 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:21.224 DEBUG:boto:Port: 8000 2023-07-29T00:07:21.225 DEBUG:boto:Params: {} 2023-07-29T00:07:21.225 DEBUG:boto:Token: None 2023-07-29T00:07:21.226 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:21 GMT /hiiqos-46/obj8 2023-07-29T00:07:21.226 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:E2WuUpLJBH1Yf8di+7IeWM9rn/E= 2023-07-29T00:07:21.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:E2WuUpLJBH1Yf8di+7IeWM9rn/E=', 'Content-Length': '0'} 2023-07-29T00:07:21.228 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000fad832a5246fb84-0064c45839-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:07:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:21.229 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:07:21.229 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:07:21.230 DEBUG:boto:path=/hiiqos-46/?prefix=obj8 2023-07-29T00:07:21.230 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj8 2023-07-29T00:07:21.230 DEBUG:boto:Method: GET 2023-07-29T00:07:21.231 DEBUG:boto:Path: /hiiqos-46/?prefix=obj8 2023-07-29T00:07:21.231 DEBUG:boto:Data: 2023-07-29T00:07:21.231 DEBUG:boto:Headers: {} 2023-07-29T00:07:21.232 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:07:21.232 DEBUG:boto:Port: 8001 2023-07-29T00:07:21.232 DEBUG:boto:Params: {} 2023-07-29T00:07:21.236 DEBUG:boto:Token: None 2023-07-29T00:07:21.237 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:21 GMT /hiiqos-46/ 2023-07-29T00:07:21.237 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6Udmp0iiAVR87H38Pm2zk+gmeuM= 2023-07-29T00:07:21.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6Udmp0iiAVR87H38Pm2zk+gmeuM=', 'Content-Length': '0'} 2023-07-29T00:07:21.240 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000354efe3a54ee2663-0064c45839-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:07:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:21.240 DEBUG:boto:b'hiiqos-46obj81000falseobj82023-07-29T00:06:59.202Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:07:21.241 DEBUG:boto:path=/hiiqos-46/ 2023-07-29T00:07:21.241 DEBUG:boto:auth_path=/hiiqos-46/ 2023-07-29T00:07:21.242 DEBUG:boto:path=/hiiqos-46/?prefix=obj8 2023-07-29T00:07:21.242 DEBUG:boto:auth_path=/hiiqos-46/?prefix=obj8 2023-07-29T00:07:21.242 DEBUG:boto:Method: GET 2023-07-29T00:07:21.243 DEBUG:boto:Path: /hiiqos-46/?prefix=obj8 2023-07-29T00:07:21.243 DEBUG:boto:Data: 2023-07-29T00:07:21.244 DEBUG:boto:Headers: {} 2023-07-29T00:07:21.244 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:07:21.244 DEBUG:boto:Port: 8000 2023-07-29T00:07:21.245 DEBUG:boto:Params: {} 2023-07-29T00:07:21.245 DEBUG:boto:Token: None 2023-07-29T00:07:21.245 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:07:21 GMT /hiiqos-46/ 2023-07-29T00:07:21.246 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6Udmp0iiAVR87H38Pm2zk+gmeuM= 2023-07-29T00:07:21.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:07:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6Udmp0iiAVR87H38Pm2zk+gmeuM=', 'Content-Length': '0'} 2023-07-29T00:07:21.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5c61013257dca41-0064c45839-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:07:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:07:21.292 DEBUG:boto:b'hiiqos-46obj81000falseobj82023-07-29T00:06:59.202Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:07:21.293 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-46 zones={a2, a1} 2023-07-29T00:07:21.294 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-47 2023-07-29T00:07:21.295 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-47', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:07:21.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-47', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:07:21.297 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-47 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:07:21.359 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:07:21.359 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:07:21.395 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:07:21.368+0000 7f9977324880 1 waiting for incremental sync to catch up: 2023-07-29T00:07:21.395 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000003.26.1, 00000000002.26.1, 00000000003.16.1, 00000000002.13.1, 00000000003.21.1, 00000000002.16.1, 00000000002.26.1, 00000000003.24.1, 00000000002.27.1, 00000000002.23.1, 00000000002.28.1] 2023-07-29T00:07:21.396 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000004.28.6, 00000000002.26.1, 00000000004.18.6, 00000000002.13.1, 00000000004.23.6, 00000000002.16.1, 00000000002.26.1, 00000000003.24.1, 00000000002.27.1, 00000000002.23.1, 00000000002.28.1] 2023-07-29T00:07:26.396 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:07:26.369+0000 7f9977324880 1 waiting for incremental sync to catch up: 2023-07-29T00:07:26.424 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000003.26.1, 00000000002.26.1, 00000000003.16.1, 00000000002.13.1, 00000000003.21.1, 00000000002.16.1, 00000000002.26.1, 00000000003.24.1, 00000000002.27.1, 00000000002.23.1, 00000000002.28.1] 2023-07-29T00:07:26.425 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000004.28.6, 00000000002.26.1, 00000000004.18.6, 00000000002.13.1, 00000000004.23.6, 00000000002.16.1, 00000000002.26.1, 00000000003.24.1, 00000000002.27.1, 00000000002.23.1, 00000000002.28.1] 2023-07-29T00:07:31.398 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:07:31.371+0000 7f9977324880 1 waiting for incremental sync to catch up: 2023-07-29T00:07:31.399 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000003.26.1, 00000000002.26.1, 00000000003.16.1, 00000000002.13.1, 00000000003.21.1, 00000000002.16.1, 00000000002.26.1, 00000000003.24.1, 00000000002.27.1, 00000000002.23.1, 00000000002.28.1] 2023-07-29T00:07:31.399 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000004.28.6, 00000000002.26.1, 00000000004.18.6, 00000000002.13.1, 00000000004.23.6, 00000000002.16.1, 00000000002.26.1, 00000000003.24.1, 00000000002.27.1, 00000000002.23.1, 00000000002.28.1] 2023-07-29T00:07:36.400 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:07:36.373+0000 7f9977324880 1 bucket sync caught up with source: 2023-07-29T00:07:36.401 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000004.28.6, 00000000002.26.1, 00000000004.18.6, 00000000002.13.1, 00000000004.23.6, 00000000002.16.1, 00000000002.26.1, 00000000003.24.1, 00000000002.27.1, 00000000002.23.1, 00000000002.28.1] 2023-07-29T00:07:36.401 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000004.28.6, 00000000002.26.1, 00000000004.18.6, 00000000002.13.1, 00000000004.23.6, 00000000002.16.1, 00000000002.26.1, 00000000003.24.1, 00000000002.27.1, 00000000002.23.1, 00000000002.28.1] 2023-07-29T00:07:36.401 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:07:36.373+0000 7f9977324880 0 bucket checkpoint complete 2023-07-29T00:07:36.418 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-47 2023-07-29T00:07:36.418 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-47', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:07:36.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-47', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:07:36.419 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-47 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:07:36.477 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:07:36.477 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:07:36.512 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:07:36.485+0000 7ff3f8067880 1 waiting for incremental sync to catch up: 2023-07-29T00:07:36.513 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000002.687.6, , 00000000002.1156.6, , 00000000002.998.6, , , 00000000001.1655.6, , , ] 2023-07-29T00:07:36.513 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000002.687.6, , 00000000002.1156.6, , 00000000002.998.6, , , 00000000002.1659.6, , , ] 2023-07-29T00:07:41.513 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:07:41.486+0000 7ff3f8067880 1 waiting for incremental sync to catch up: 2023-07-29T00:07:41.514 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000002.687.6, , 00000000002.1156.6, , 00000000002.998.6, , , 00000000001.1655.6, , , ] 2023-07-29T00:07:41.514 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000002.687.6, , 00000000002.1156.6, , 00000000002.998.6, , , 00000000002.1659.6, , , ] 2023-07-29T00:07:46.516 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:07:46.487+0000 7ff3f8067880 1 waiting for incremental sync to catch up: 2023-07-29T00:07:46.516 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000002.687.6, , 00000000002.1156.6, , 00000000002.998.6, , , 00000000001.1655.6, , , ] 2023-07-29T00:07:46.516 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000002.687.6, , 00000000002.1156.6, , 00000000002.998.6, , , 00000000002.1659.6, , , ] 2023-07-29T00:07:51.516 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:07:51.488+0000 7ff3f8067880 1 waiting for incremental sync to catch up: 2023-07-29T00:07:51.517 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000002.687.6, , 00000000002.1156.6, , 00000000002.998.6, , , 00000000001.1655.6, , , ] 2023-07-29T00:07:51.517 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000002.687.6, , 00000000002.1156.6, , 00000000002.998.6, , , 00000000002.1659.6, , , ] 2023-07-29T00:07:56.518 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:07:56.490+0000 7ff3f8067880 1 waiting for incremental sync to catch up: 2023-07-29T00:07:56.518 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000002.687.6, , 00000000002.1156.6, , 00000000002.998.6, , , 00000000001.1655.6, , , ] 2023-07-29T00:07:56.518 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000002.687.6, , 00000000002.1156.6, , 00000000002.998.6, , , 00000000002.1659.6, , , ] 2023-07-29T00:08:01.519 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:08:01.491+0000 7ff3f8067880 1 waiting for incremental sync to catch up: 2023-07-29T00:08:01.519 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000002.687.6, , 00000000002.1156.6, , 00000000002.998.6, , , 00000000001.1655.6, , , ] 2023-07-29T00:08:01.520 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000002.687.6, , 00000000002.1156.6, , 00000000002.998.6, , , 00000000002.1659.6, , , ] 2023-07-29T00:08:06.521 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:08:06.492+0000 7ff3f8067880 1 waiting for incremental sync to catch up: 2023-07-29T00:08:06.521 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000002.687.6, , 00000000002.1156.6, , 00000000002.998.6, , , 00000000001.1655.6, , , ] 2023-07-29T00:08:06.521 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000002.687.6, , 00000000002.1156.6, , 00000000002.998.6, , , 00000000002.1659.6, , , ] 2023-07-29T00:08:11.522 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:08:11.493+0000 7ff3f8067880 1 waiting for incremental sync to catch up: 2023-07-29T00:08:11.522 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000002.687.6, , 00000000002.1156.6, , 00000000002.998.6, , , 00000000001.1655.6, , , ] 2023-07-29T00:08:11.523 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000002.687.6, , 00000000002.1156.6, , 00000000002.998.6, , , 00000000002.1659.6, , , ] 2023-07-29T00:08:16.524 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:08:16.495+0000 7ff3f8067880 1 bucket sync caught up with source: 2023-07-29T00:08:16.524 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000002.687.6, , 00000000002.1156.6, , 00000000002.998.6, , , 00000000002.1659.6, , , ] 2023-07-29T00:08:16.524 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000002.687.6, , 00000000002.1156.6, , 00000000002.998.6, , , 00000000002.1659.6, , , ] 2023-07-29T00:08:16.524 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:08:16.495+0000 7ff3f8067880 0 bucket checkpoint complete 2023-07-29T00:08:16.541 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-47 zones={a2, a1} 2023-07-29T00:08:16.541 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:08:16.542 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:08:16.542 DEBUG:boto:Method: HEAD 2023-07-29T00:08:16.542 DEBUG:boto:Path: /hiiqos-47/ 2023-07-29T00:08:16.543 DEBUG:boto:Data: 2023-07-29T00:08:16.543 DEBUG:boto:Headers: {} 2023-07-29T00:08:16.543 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:16.544 DEBUG:boto:Port: 8001 2023-07-29T00:08:16.544 DEBUG:boto:Params: {} 2023-07-29T00:08:16.545 DEBUG:boto:Token: None 2023-07-29T00:08:16.545 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:16 GMT /hiiqos-47/ 2023-07-29T00:08:16.546 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jvn9WPr7PfEIK9gygX7AmktrR1Q= 2023-07-29T00:08:16.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jvn9WPr7PfEIK9gygX7AmktrR1Q=', 'Content-Length': '0'} 2023-07-29T00:08:16.549 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008fdbb8c98514e5d9-0064c45870-1663-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:08:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:16.549 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:08:16.550 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:08:16.550 DEBUG:boto:Method: HEAD 2023-07-29T00:08:16.551 DEBUG:boto:Path: /hiiqos-47/ 2023-07-29T00:08:16.551 DEBUG:boto:Data: 2023-07-29T00:08:16.551 DEBUG:boto:Headers: {} 2023-07-29T00:08:16.552 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:16.552 DEBUG:boto:Port: 8000 2023-07-29T00:08:16.553 DEBUG:boto:Params: {} 2023-07-29T00:08:16.553 DEBUG:boto:Token: None 2023-07-29T00:08:16.553 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:16 GMT /hiiqos-47/ 2023-07-29T00:08:16.554 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jvn9WPr7PfEIK9gygX7AmktrR1Q= 2023-07-29T00:08:16.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jvn9WPr7PfEIK9gygX7AmktrR1Q=', 'Content-Length': '0'} 2023-07-29T00:08:16.557 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f00443fd1009e676-0064c45870-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:08:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:16.557 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-29T00:08:16.558 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:08:16.558 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:08:16.558 DEBUG:boto:path=/hiiqos-47/?versions&max-keys=999 2023-07-29T00:08:16.559 DEBUG:boto:auth_path=/hiiqos-47/?versions&max-keys=999 2023-07-29T00:08:16.559 DEBUG:boto:Method: GET 2023-07-29T00:08:16.560 DEBUG:boto:Path: /hiiqos-47/?versions&max-keys=999 2023-07-29T00:08:16.560 DEBUG:boto:Data: 2023-07-29T00:08:16.560 DEBUG:boto:Headers: {} 2023-07-29T00:08:16.561 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:16.561 DEBUG:boto:Port: 8001 2023-07-29T00:08:16.562 DEBUG:boto:Params: {} 2023-07-29T00:08:16.562 DEBUG:boto:Token: None 2023-07-29T00:08:16.562 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:16 GMT /hiiqos-47/?versions 2023-07-29T00:08:16.563 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0UdoCZR1aFXlglZWXakiqUXR5JI= 2023-07-29T00:08:16.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0UdoCZR1aFXlglZWXakiqUXR5JI=', 'Content-Length': '0'} 2023-07-29T00:08:16.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ef526de65c38668-0064c45870-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:16.566 DEBUG:boto:b'hiiqos-47999falseobj1nulltrue2023-07-29T00:05:16.136Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-07-29T00:05:16.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-07-29T00:05:16.174Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-07-29T00:05:16.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-07-29T00:06:59.221Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-07-29T00:06:59.242Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-07-29T00:06:59.259Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-07-29T00:06:59.319Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:08:16.567 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-07-29T00:08:16.568 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-07-29T00:08:16.568 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-07-29T00:08:16.569 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-07-29T00:08:16.569 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-07-29T00:08:16.569 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-07-29T00:08:16.569 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-07-29T00:08:16.570 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-07-29T00:08:16.570 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-29T00:08:16.571 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:08:16.571 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:08:16.572 DEBUG:boto:path=/hiiqos-47/?versions&max-keys=999 2023-07-29T00:08:16.572 DEBUG:boto:auth_path=/hiiqos-47/?versions&max-keys=999 2023-07-29T00:08:16.572 DEBUG:boto:Method: GET 2023-07-29T00:08:16.573 DEBUG:boto:Path: /hiiqos-47/?versions&max-keys=999 2023-07-29T00:08:16.573 DEBUG:boto:Data: 2023-07-29T00:08:16.574 DEBUG:boto:Headers: {} 2023-07-29T00:08:16.574 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:16.575 DEBUG:boto:Port: 8000 2023-07-29T00:08:16.575 DEBUG:boto:Params: {} 2023-07-29T00:08:16.576 DEBUG:boto:Token: None 2023-07-29T00:08:16.576 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:16 GMT /hiiqos-47/?versions 2023-07-29T00:08:16.577 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0UdoCZR1aFXlglZWXakiqUXR5JI= 2023-07-29T00:08:16.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0UdoCZR1aFXlglZWXakiqUXR5JI=', 'Content-Length': '0'} 2023-07-29T00:08:16.580 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078cefb6f568080e4-0064c45870-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:16.580 DEBUG:boto:b'hiiqos-47999falseobj1nulltrue2023-07-29T00:05:16.136Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-07-29T00:05:16.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-07-29T00:05:16.174Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-07-29T00:05:16.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-07-29T00:06:59.221Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-07-29T00:06:59.242Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-07-29T00:06:59.259Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-07-29T00:06:59.319Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:08:16.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-07-29T00:08:16.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-07-29T00:08:16.582 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-07-29T00:08:16.582 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-07-29T00:08:16.582 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-07-29T00:08:16.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-07-29T00:08:16.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-07-29T00:08:16.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-07-29T00:08:16.584 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:08:16.584 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:08:16.584 DEBUG:boto:path=/hiiqos-47/?versions&max-keys=999 2023-07-29T00:08:16.585 DEBUG:boto:auth_path=/hiiqos-47/?versions&max-keys=999 2023-07-29T00:08:16.585 DEBUG:boto:Method: GET 2023-07-29T00:08:16.586 DEBUG:boto:Path: /hiiqos-47/?versions&max-keys=999 2023-07-29T00:08:16.586 DEBUG:boto:Data: 2023-07-29T00:08:16.586 DEBUG:boto:Headers: {} 2023-07-29T00:08:16.587 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:16.587 DEBUG:boto:Port: 8001 2023-07-29T00:08:16.587 DEBUG:boto:Params: {} 2023-07-29T00:08:16.588 DEBUG:boto:Token: None 2023-07-29T00:08:16.588 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:16 GMT /hiiqos-47/?versions 2023-07-29T00:08:16.589 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0UdoCZR1aFXlglZWXakiqUXR5JI= 2023-07-29T00:08:16.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0UdoCZR1aFXlglZWXakiqUXR5JI=', 'Content-Length': '0'} 2023-07-29T00:08:16.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5bcfa719ccc3718-0064c45870-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:16.592 DEBUG:boto:b'hiiqos-47999falseobj1nulltrue2023-07-29T00:05:16.136Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-07-29T00:05:16.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-07-29T00:05:16.174Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-07-29T00:05:16.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-07-29T00:06:59.221Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-07-29T00:06:59.242Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-07-29T00:06:59.259Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-07-29T00:06:59.319Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:08:16.593 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:08:16.593 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:08:16.594 DEBUG:boto:path=/hiiqos-47/?versions&max-keys=999 2023-07-29T00:08:16.594 DEBUG:boto:auth_path=/hiiqos-47/?versions&max-keys=999 2023-07-29T00:08:16.594 DEBUG:boto:Method: GET 2023-07-29T00:08:16.595 DEBUG:boto:Path: /hiiqos-47/?versions&max-keys=999 2023-07-29T00:08:16.595 DEBUG:boto:Data: 2023-07-29T00:08:16.595 DEBUG:boto:Headers: {} 2023-07-29T00:08:16.596 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:16.596 DEBUG:boto:Port: 8000 2023-07-29T00:08:16.596 DEBUG:boto:Params: {} 2023-07-29T00:08:16.597 DEBUG:boto:Token: None 2023-07-29T00:08:16.597 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:16 GMT /hiiqos-47/?versions 2023-07-29T00:08:16.598 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0UdoCZR1aFXlglZWXakiqUXR5JI= 2023-07-29T00:08:16.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0UdoCZR1aFXlglZWXakiqUXR5JI=', 'Content-Length': '0'} 2023-07-29T00:08:16.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b73fb116100a3b83-0064c45870-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:16.601 DEBUG:boto:b'hiiqos-47999falseobj1nulltrue2023-07-29T00:05:16.136Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-07-29T00:05:16.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-07-29T00:05:16.174Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-07-29T00:05:16.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-07-29T00:06:59.221Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-07-29T00:06:59.242Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-07-29T00:06:59.259Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-07-29T00:06:59.319Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:08:16.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-07-29T00:08:16.602 DEBUG:boto:path=/hiiqos-47/obj1 2023-07-29T00:08:16.603 DEBUG:boto:auth_path=/hiiqos-47/obj1 2023-07-29T00:08:16.603 DEBUG:boto:path=/hiiqos-47/obj1?versionId=null 2023-07-29T00:08:16.604 DEBUG:boto:auth_path=/hiiqos-47/obj1?versionId=null 2023-07-29T00:08:16.604 DEBUG:boto:Method: GET 2023-07-29T00:08:16.604 DEBUG:boto:Path: /hiiqos-47/obj1?versionId=null 2023-07-29T00:08:16.605 DEBUG:boto:Data: 2023-07-29T00:08:16.605 DEBUG:boto:Headers: {} 2023-07-29T00:08:16.605 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:16.606 DEBUG:boto:Port: 8001 2023-07-29T00:08:16.606 DEBUG:boto:Params: {} 2023-07-29T00:08:16.606 DEBUG:boto:Token: None 2023-07-29T00:08:16.607 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:16 GMT /hiiqos-47/obj1?versionId=null 2023-07-29T00:08:16.607 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Lsb/MYqkwAl0/ICc3sgj4I6ByTE= 2023-07-29T00:08:16.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Lsb/MYqkwAl0/ICc3sgj4I6ByTE=', 'Content-Length': '0'} 2023-07-29T00:08:16.610 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009d84a6c5e9b1f459-0064c45870-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:16.652 DEBUG:boto:path=/hiiqos-47/obj1 2023-07-29T00:08:16.653 DEBUG:boto:auth_path=/hiiqos-47/obj1 2023-07-29T00:08:16.653 DEBUG:boto:path=/hiiqos-47/obj1?versionId=null 2023-07-29T00:08:16.654 DEBUG:boto:auth_path=/hiiqos-47/obj1?versionId=null 2023-07-29T00:08:16.654 DEBUG:boto:Method: GET 2023-07-29T00:08:16.654 DEBUG:boto:Path: /hiiqos-47/obj1?versionId=null 2023-07-29T00:08:16.655 DEBUG:boto:Data: 2023-07-29T00:08:16.655 DEBUG:boto:Headers: {} 2023-07-29T00:08:16.656 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:16.656 DEBUG:boto:Port: 8000 2023-07-29T00:08:16.656 DEBUG:boto:Params: {} 2023-07-29T00:08:16.657 DEBUG:boto:Token: None 2023-07-29T00:08:16.657 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:16 GMT /hiiqos-47/obj1?versionId=null 2023-07-29T00:08:16.658 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Lsb/MYqkwAl0/ICc3sgj4I6ByTE= 2023-07-29T00:08:16.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Lsb/MYqkwAl0/ICc3sgj4I6ByTE=', 'Content-Length': '0'} 2023-07-29T00:08:16.660 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000093513b55a6da49dd-0064c45870-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:16.661 DEBUG:boto:path=/hiiqos-47/obj1 2023-07-29T00:08:16.661 DEBUG:boto:auth_path=/hiiqos-47/obj1 2023-07-29T00:08:16.661 DEBUG:boto:path=/hiiqos-47/obj1?versionId=null 2023-07-29T00:08:16.662 DEBUG:boto:auth_path=/hiiqos-47/obj1?versionId=null 2023-07-29T00:08:16.662 DEBUG:boto:Method: HEAD 2023-07-29T00:08:16.662 DEBUG:boto:Path: /hiiqos-47/obj1?versionId=null 2023-07-29T00:08:16.663 DEBUG:boto:Data: 2023-07-29T00:08:16.663 DEBUG:boto:Headers: {} 2023-07-29T00:08:16.663 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:16.664 DEBUG:boto:Port: 8001 2023-07-29T00:08:16.664 DEBUG:boto:Params: {} 2023-07-29T00:08:16.664 DEBUG:boto:Token: None 2023-07-29T00:08:16.665 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:16 GMT /hiiqos-47/obj1?versionId=null 2023-07-29T00:08:16.665 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1ELxU/X3HzYm1HmbmKATEQVpftI= 2023-07-29T00:08:16.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1ELxU/X3HzYm1HmbmKATEQVpftI=', 'Content-Length': '0'} 2023-07-29T00:08:16.667 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b0de0194bafdf36-0064c45870-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:16.668 DEBUG:boto:path=/hiiqos-47/obj1 2023-07-29T00:08:16.668 DEBUG:boto:auth_path=/hiiqos-47/obj1 2023-07-29T00:08:16.669 DEBUG:boto:path=/hiiqos-47/obj1?versionId=null 2023-07-29T00:08:16.669 DEBUG:boto:auth_path=/hiiqos-47/obj1?versionId=null 2023-07-29T00:08:16.669 DEBUG:boto:Method: HEAD 2023-07-29T00:08:16.670 DEBUG:boto:Path: /hiiqos-47/obj1?versionId=null 2023-07-29T00:08:16.670 DEBUG:boto:Data: 2023-07-29T00:08:16.670 DEBUG:boto:Headers: {} 2023-07-29T00:08:16.671 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:16.671 DEBUG:boto:Port: 8000 2023-07-29T00:08:16.671 DEBUG:boto:Params: {} 2023-07-29T00:08:16.672 DEBUG:boto:Token: None 2023-07-29T00:08:16.672 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:16 GMT /hiiqos-47/obj1?versionId=null 2023-07-29T00:08:16.673 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1ELxU/X3HzYm1HmbmKATEQVpftI= 2023-07-29T00:08:16.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1ELxU/X3HzYm1HmbmKATEQVpftI=', 'Content-Length': '0'} 2023-07-29T00:08:16.675 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea0b84900c2dc3d4-0064c45870-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:16.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-07-29T00:08:16.676 DEBUG:boto:path=/hiiqos-47/obj1 2023-07-29T00:08:16.676 DEBUG:boto:auth_path=/hiiqos-47/obj1 2023-07-29T00:08:16.677 DEBUG:boto:path=/hiiqos-47/obj1?versionId=null 2023-07-29T00:08:16.677 DEBUG:boto:auth_path=/hiiqos-47/obj1?versionId=null 2023-07-29T00:08:16.677 DEBUG:boto:Method: GET 2023-07-29T00:08:16.678 DEBUG:boto:Path: /hiiqos-47/obj1?versionId=null 2023-07-29T00:08:16.678 DEBUG:boto:Data: 2023-07-29T00:08:16.679 DEBUG:boto:Headers: {} 2023-07-29T00:08:16.679 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:16.679 DEBUG:boto:Port: 8001 2023-07-29T00:08:16.680 DEBUG:boto:Params: {} 2023-07-29T00:08:16.680 DEBUG:boto:Token: None 2023-07-29T00:08:16.680 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:16 GMT /hiiqos-47/obj1?versionId=null 2023-07-29T00:08:16.681 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Lsb/MYqkwAl0/ICc3sgj4I6ByTE= 2023-07-29T00:08:16.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Lsb/MYqkwAl0/ICc3sgj4I6ByTE=', 'Content-Length': '0'} 2023-07-29T00:08:16.683 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eae48ceeda32cd29-0064c45870-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:16.684 DEBUG:boto:path=/hiiqos-47/obj1 2023-07-29T00:08:16.684 DEBUG:boto:auth_path=/hiiqos-47/obj1 2023-07-29T00:08:16.684 DEBUG:boto:path=/hiiqos-47/obj1?versionId=null 2023-07-29T00:08:16.685 DEBUG:boto:auth_path=/hiiqos-47/obj1?versionId=null 2023-07-29T00:08:16.685 DEBUG:boto:Method: GET 2023-07-29T00:08:16.686 DEBUG:boto:Path: /hiiqos-47/obj1?versionId=null 2023-07-29T00:08:16.686 DEBUG:boto:Data: 2023-07-29T00:08:16.687 DEBUG:boto:Headers: {} 2023-07-29T00:08:16.687 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:16.687 DEBUG:boto:Port: 8000 2023-07-29T00:08:16.688 DEBUG:boto:Params: {} 2023-07-29T00:08:16.688 DEBUG:boto:Token: None 2023-07-29T00:08:16.688 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:16 GMT /hiiqos-47/obj1?versionId=null 2023-07-29T00:08:16.689 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Lsb/MYqkwAl0/ICc3sgj4I6ByTE= 2023-07-29T00:08:16.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Lsb/MYqkwAl0/ICc3sgj4I6ByTE=', 'Content-Length': '0'} 2023-07-29T00:08:16.691 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009979b504a4d24b6a-0064c45870-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:16.692 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:08:16.692 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:08:16.693 DEBUG:boto:path=/hiiqos-47/?prefix=obj1 2023-07-29T00:08:16.693 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj1 2023-07-29T00:08:16.694 DEBUG:boto:Method: GET 2023-07-29T00:08:16.694 DEBUG:boto:Path: /hiiqos-47/?prefix=obj1 2023-07-29T00:08:16.694 DEBUG:boto:Data: 2023-07-29T00:08:16.695 DEBUG:boto:Headers: {} 2023-07-29T00:08:16.695 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:16.695 DEBUG:boto:Port: 8001 2023-07-29T00:08:16.696 DEBUG:boto:Params: {} 2023-07-29T00:08:16.696 DEBUG:boto:Token: None 2023-07-29T00:08:16.696 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:16 GMT /hiiqos-47/ 2023-07-29T00:08:16.697 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xIphdg5s+tMVEsp2itCiVlMEeRc= 2023-07-29T00:08:16.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xIphdg5s+tMVEsp2itCiVlMEeRc=', 'Content-Length': '0'} 2023-07-29T00:08:16.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ca6ace0f2a67142-0064c45870-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:16.700 DEBUG:boto:b'hiiqos-47obj11000falseobj12023-07-29T00:05:16.136Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:08:16.701 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:08:16.702 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:08:16.702 DEBUG:boto:path=/hiiqos-47/?prefix=obj1 2023-07-29T00:08:16.702 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj1 2023-07-29T00:08:16.703 DEBUG:boto:Method: GET 2023-07-29T00:08:16.703 DEBUG:boto:Path: /hiiqos-47/?prefix=obj1 2023-07-29T00:08:16.703 DEBUG:boto:Data: 2023-07-29T00:08:16.704 DEBUG:boto:Headers: {} 2023-07-29T00:08:16.704 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:16.705 DEBUG:boto:Port: 8000 2023-07-29T00:08:16.705 DEBUG:boto:Params: {} 2023-07-29T00:08:16.705 DEBUG:boto:Token: None 2023-07-29T00:08:16.706 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:16 GMT /hiiqos-47/ 2023-07-29T00:08:16.707 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xIphdg5s+tMVEsp2itCiVlMEeRc= 2023-07-29T00:08:16.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xIphdg5s+tMVEsp2itCiVlMEeRc=', 'Content-Length': '0'} 2023-07-29T00:08:16.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009d58c309e77a341-0064c45870-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:16.710 DEBUG:boto:b'hiiqos-47obj11000falseobj12023-07-29T00:05:16.136Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:08:16.711 DEBUG:boto:path=/hiiqos-47/obj1 2023-07-29T00:08:16.711 DEBUG:boto:auth_path=/hiiqos-47/obj1 2023-07-29T00:08:16.712 DEBUG:boto:Method: HEAD 2023-07-29T00:08:16.712 DEBUG:boto:Path: /hiiqos-47/obj1 2023-07-29T00:08:16.712 DEBUG:boto:Data: 2023-07-29T00:08:16.713 DEBUG:boto:Headers: {} 2023-07-29T00:08:16.713 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:16.713 DEBUG:boto:Port: 8001 2023-07-29T00:08:16.714 DEBUG:boto:Params: {} 2023-07-29T00:08:16.714 DEBUG:boto:Token: None 2023-07-29T00:08:16.714 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:16 GMT /hiiqos-47/obj1 2023-07-29T00:08:16.715 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xl1Jj9EG7eKSvfjRCY8QESfkQt0= 2023-07-29T00:08:16.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xl1Jj9EG7eKSvfjRCY8QESfkQt0=', 'Content-Length': '0'} 2023-07-29T00:08:16.717 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b01645fb4c42af02-0064c45870-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:16.718 DEBUG:boto:path=/hiiqos-47/obj1 2023-07-29T00:08:16.718 DEBUG:boto:auth_path=/hiiqos-47/obj1 2023-07-29T00:08:16.718 DEBUG:boto:Method: HEAD 2023-07-29T00:08:16.719 DEBUG:boto:Path: /hiiqos-47/obj1 2023-07-29T00:08:16.719 DEBUG:boto:Data: 2023-07-29T00:08:16.719 DEBUG:boto:Headers: {} 2023-07-29T00:08:16.720 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:16.720 DEBUG:boto:Port: 8000 2023-07-29T00:08:16.720 DEBUG:boto:Params: {} 2023-07-29T00:08:16.721 DEBUG:boto:Token: None 2023-07-29T00:08:16.721 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:16 GMT /hiiqos-47/obj1 2023-07-29T00:08:16.721 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xl1Jj9EG7eKSvfjRCY8QESfkQt0= 2023-07-29T00:08:16.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xl1Jj9EG7eKSvfjRCY8QESfkQt0=', 'Content-Length': '0'} 2023-07-29T00:08:16.724 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003438e81d373ae9e4-0064c45870-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:16.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-07-29T00:08:16.725 DEBUG:boto:path=/hiiqos-47/obj1 2023-07-29T00:08:16.725 DEBUG:boto:auth_path=/hiiqos-47/obj1 2023-07-29T00:08:16.725 DEBUG:boto:Method: GET 2023-07-29T00:08:16.726 DEBUG:boto:Path: /hiiqos-47/obj1 2023-07-29T00:08:16.726 DEBUG:boto:Data: 2023-07-29T00:08:16.726 DEBUG:boto:Headers: {} 2023-07-29T00:08:16.727 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:16.727 DEBUG:boto:Port: 8001 2023-07-29T00:08:16.728 DEBUG:boto:Params: {} 2023-07-29T00:08:16.728 DEBUG:boto:Token: None 2023-07-29T00:08:16.729 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:16 GMT /hiiqos-47/obj1 2023-07-29T00:08:16.729 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EKyYol3C7YmzmIiAG+1+Pmz+fOw= 2023-07-29T00:08:16.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EKyYol3C7YmzmIiAG+1+Pmz+fOw=', 'Content-Length': '0'} 2023-07-29T00:08:16.733 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000031fc24cf79b9890c-0064c45870-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:16.734 DEBUG:boto:path=/hiiqos-47/obj1 2023-07-29T00:08:16.735 DEBUG:boto:auth_path=/hiiqos-47/obj1 2023-07-29T00:08:16.736 DEBUG:boto:Method: GET 2023-07-29T00:08:16.737 DEBUG:boto:Path: /hiiqos-47/obj1 2023-07-29T00:08:16.737 DEBUG:boto:Data: 2023-07-29T00:08:16.738 DEBUG:boto:Headers: {} 2023-07-29T00:08:16.738 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:16.739 DEBUG:boto:Port: 8000 2023-07-29T00:08:16.739 DEBUG:boto:Params: {} 2023-07-29T00:08:16.740 DEBUG:boto:Token: None 2023-07-29T00:08:16.740 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:16 GMT /hiiqos-47/obj1 2023-07-29T00:08:16.741 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EKyYol3C7YmzmIiAG+1+Pmz+fOw= 2023-07-29T00:08:16.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EKyYol3C7YmzmIiAG+1+Pmz+fOw=', 'Content-Length': '0'} 2023-07-29T00:08:16.743 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f0f99db363f9686-0064c45870-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:16.744 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:08:16.744 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:08:16.745 DEBUG:boto:path=/hiiqos-47/?prefix=obj1 2023-07-29T00:08:16.745 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj1 2023-07-29T00:08:16.745 DEBUG:boto:Method: GET 2023-07-29T00:08:16.746 DEBUG:boto:Path: /hiiqos-47/?prefix=obj1 2023-07-29T00:08:16.746 DEBUG:boto:Data: 2023-07-29T00:08:16.747 DEBUG:boto:Headers: {} 2023-07-29T00:08:16.747 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:16.747 DEBUG:boto:Port: 8001 2023-07-29T00:08:16.748 DEBUG:boto:Params: {} 2023-07-29T00:08:16.748 DEBUG:boto:Token: None 2023-07-29T00:08:16.749 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:16 GMT /hiiqos-47/ 2023-07-29T00:08:16.749 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xIphdg5s+tMVEsp2itCiVlMEeRc= 2023-07-29T00:08:16.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xIphdg5s+tMVEsp2itCiVlMEeRc=', 'Content-Length': '0'} 2023-07-29T00:08:16.752 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004d3a843b449749c-0064c45870-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:16.796 DEBUG:boto:b'hiiqos-47obj11000falseobj12023-07-29T00:05:16.136Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:08:16.797 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:08:16.798 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:08:16.798 DEBUG:boto:path=/hiiqos-47/?prefix=obj1 2023-07-29T00:08:16.798 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj1 2023-07-29T00:08:16.799 DEBUG:boto:Method: GET 2023-07-29T00:08:16.799 DEBUG:boto:Path: /hiiqos-47/?prefix=obj1 2023-07-29T00:08:16.799 DEBUG:boto:Data: 2023-07-29T00:08:16.800 DEBUG:boto:Headers: {} 2023-07-29T00:08:16.800 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:16.801 DEBUG:boto:Port: 8000 2023-07-29T00:08:16.801 DEBUG:boto:Params: {} 2023-07-29T00:08:16.801 DEBUG:boto:Token: None 2023-07-29T00:08:16.802 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:16 GMT /hiiqos-47/ 2023-07-29T00:08:16.802 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xIphdg5s+tMVEsp2itCiVlMEeRc= 2023-07-29T00:08:16.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xIphdg5s+tMVEsp2itCiVlMEeRc=', 'Content-Length': '0'} 2023-07-29T00:08:16.805 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a943330650af735-0064c45870-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:16.806 DEBUG:boto:b'hiiqos-47obj11000falseobj12023-07-29T00:05:16.136Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:08:16.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-07-29T00:08:16.807 DEBUG:boto:path=/hiiqos-47/obj2 2023-07-29T00:08:16.807 DEBUG:boto:auth_path=/hiiqos-47/obj2 2023-07-29T00:08:16.808 DEBUG:boto:path=/hiiqos-47/obj2?versionId=null 2023-07-29T00:08:16.808 DEBUG:boto:auth_path=/hiiqos-47/obj2?versionId=null 2023-07-29T00:08:16.808 DEBUG:boto:Method: GET 2023-07-29T00:08:16.809 DEBUG:boto:Path: /hiiqos-47/obj2?versionId=null 2023-07-29T00:08:16.809 DEBUG:boto:Data: 2023-07-29T00:08:16.809 DEBUG:boto:Headers: {} 2023-07-29T00:08:16.810 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:16.810 DEBUG:boto:Port: 8001 2023-07-29T00:08:16.811 DEBUG:boto:Params: {} 2023-07-29T00:08:16.811 DEBUG:boto:Token: None 2023-07-29T00:08:16.811 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:16 GMT /hiiqos-47/obj2?versionId=null 2023-07-29T00:08:16.812 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3k6Dn8I0uwuMMj6Ot5DR2BFr9rM= 2023-07-29T00:08:16.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3k6Dn8I0uwuMMj6Ot5DR2BFr9rM=', 'Content-Length': '0'} 2023-07-29T00:08:16.815 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000009917359b2126fd5-0064c45870-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:16.815 DEBUG:boto:path=/hiiqos-47/obj2 2023-07-29T00:08:16.816 DEBUG:boto:auth_path=/hiiqos-47/obj2 2023-07-29T00:08:16.816 DEBUG:boto:path=/hiiqos-47/obj2?versionId=null 2023-07-29T00:08:16.816 DEBUG:boto:auth_path=/hiiqos-47/obj2?versionId=null 2023-07-29T00:08:16.817 DEBUG:boto:Method: GET 2023-07-29T00:08:16.817 DEBUG:boto:Path: /hiiqos-47/obj2?versionId=null 2023-07-29T00:08:16.818 DEBUG:boto:Data: 2023-07-29T00:08:16.818 DEBUG:boto:Headers: {} 2023-07-29T00:08:16.818 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:16.819 DEBUG:boto:Port: 8000 2023-07-29T00:08:16.819 DEBUG:boto:Params: {} 2023-07-29T00:08:16.819 DEBUG:boto:Token: None 2023-07-29T00:08:16.820 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:16 GMT /hiiqos-47/obj2?versionId=null 2023-07-29T00:08:16.821 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3k6Dn8I0uwuMMj6Ot5DR2BFr9rM= 2023-07-29T00:08:16.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3k6Dn8I0uwuMMj6Ot5DR2BFr9rM=', 'Content-Length': '0'} 2023-07-29T00:08:16.825 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a82831612cfdbda-0064c45870-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:16.868 DEBUG:boto:path=/hiiqos-47/obj2 2023-07-29T00:08:16.869 DEBUG:boto:auth_path=/hiiqos-47/obj2 2023-07-29T00:08:16.869 DEBUG:boto:path=/hiiqos-47/obj2?versionId=null 2023-07-29T00:08:16.869 DEBUG:boto:auth_path=/hiiqos-47/obj2?versionId=null 2023-07-29T00:08:16.870 DEBUG:boto:Method: HEAD 2023-07-29T00:08:16.870 DEBUG:boto:Path: /hiiqos-47/obj2?versionId=null 2023-07-29T00:08:16.870 DEBUG:boto:Data: 2023-07-29T00:08:16.871 DEBUG:boto:Headers: {} 2023-07-29T00:08:16.871 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:16.871 DEBUG:boto:Port: 8001 2023-07-29T00:08:16.872 DEBUG:boto:Params: {} 2023-07-29T00:08:16.872 DEBUG:boto:Token: None 2023-07-29T00:08:16.873 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:16 GMT /hiiqos-47/obj2?versionId=null 2023-07-29T00:08:16.873 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uAVEsx2cBpnHXIK+4REf/SN+yuI= 2023-07-29T00:08:16.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uAVEsx2cBpnHXIK+4REf/SN+yuI=', 'Content-Length': '0'} 2023-07-29T00:08:16.875 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e8e4071f0192acfb-0064c45870-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:16.876 DEBUG:boto:path=/hiiqos-47/obj2 2023-07-29T00:08:16.876 DEBUG:boto:auth_path=/hiiqos-47/obj2 2023-07-29T00:08:16.877 DEBUG:boto:path=/hiiqos-47/obj2?versionId=null 2023-07-29T00:08:16.877 DEBUG:boto:auth_path=/hiiqos-47/obj2?versionId=null 2023-07-29T00:08:16.877 DEBUG:boto:Method: HEAD 2023-07-29T00:08:16.878 DEBUG:boto:Path: /hiiqos-47/obj2?versionId=null 2023-07-29T00:08:16.878 DEBUG:boto:Data: 2023-07-29T00:08:16.878 DEBUG:boto:Headers: {} 2023-07-29T00:08:16.879 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:16.879 DEBUG:boto:Port: 8000 2023-07-29T00:08:16.880 DEBUG:boto:Params: {} 2023-07-29T00:08:16.880 DEBUG:boto:Token: None 2023-07-29T00:08:16.880 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:16 GMT /hiiqos-47/obj2?versionId=null 2023-07-29T00:08:16.881 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uAVEsx2cBpnHXIK+4REf/SN+yuI= 2023-07-29T00:08:16.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uAVEsx2cBpnHXIK+4REf/SN+yuI=', 'Content-Length': '0'} 2023-07-29T00:08:16.883 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c6b5097d94335316-0064c45870-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:16.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-07-29T00:08:16.884 DEBUG:boto:path=/hiiqos-47/obj2 2023-07-29T00:08:16.884 DEBUG:boto:auth_path=/hiiqos-47/obj2 2023-07-29T00:08:16.885 DEBUG:boto:path=/hiiqos-47/obj2?versionId=null 2023-07-29T00:08:16.885 DEBUG:boto:auth_path=/hiiqos-47/obj2?versionId=null 2023-07-29T00:08:16.885 DEBUG:boto:Method: GET 2023-07-29T00:08:16.886 DEBUG:boto:Path: /hiiqos-47/obj2?versionId=null 2023-07-29T00:08:16.886 DEBUG:boto:Data: 2023-07-29T00:08:16.886 DEBUG:boto:Headers: {} 2023-07-29T00:08:16.887 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:16.887 DEBUG:boto:Port: 8001 2023-07-29T00:08:16.888 DEBUG:boto:Params: {} 2023-07-29T00:08:16.888 DEBUG:boto:Token: None 2023-07-29T00:08:16.888 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:16 GMT /hiiqos-47/obj2?versionId=null 2023-07-29T00:08:16.889 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3k6Dn8I0uwuMMj6Ot5DR2BFr9rM= 2023-07-29T00:08:16.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3k6Dn8I0uwuMMj6Ot5DR2BFr9rM=', 'Content-Length': '0'} 2023-07-29T00:08:16.891 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d57ec9f918043874-0064c45870-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:16.892 DEBUG:boto:path=/hiiqos-47/obj2 2023-07-29T00:08:16.892 DEBUG:boto:auth_path=/hiiqos-47/obj2 2023-07-29T00:08:16.893 DEBUG:boto:path=/hiiqos-47/obj2?versionId=null 2023-07-29T00:08:16.893 DEBUG:boto:auth_path=/hiiqos-47/obj2?versionId=null 2023-07-29T00:08:16.893 DEBUG:boto:Method: GET 2023-07-29T00:08:16.894 DEBUG:boto:Path: /hiiqos-47/obj2?versionId=null 2023-07-29T00:08:16.894 DEBUG:boto:Data: 2023-07-29T00:08:16.894 DEBUG:boto:Headers: {} 2023-07-29T00:08:16.895 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:16.895 DEBUG:boto:Port: 8000 2023-07-29T00:08:16.895 DEBUG:boto:Params: {} 2023-07-29T00:08:16.896 DEBUG:boto:Token: None 2023-07-29T00:08:16.896 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:16 GMT /hiiqos-47/obj2?versionId=null 2023-07-29T00:08:16.897 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3k6Dn8I0uwuMMj6Ot5DR2BFr9rM= 2023-07-29T00:08:16.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3k6Dn8I0uwuMMj6Ot5DR2BFr9rM=', 'Content-Length': '0'} 2023-07-29T00:08:16.899 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a5cbe691c3efd1cf-0064c45870-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:16.900 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:08:16.900 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:08:16.900 DEBUG:boto:path=/hiiqos-47/?prefix=obj2 2023-07-29T00:08:16.901 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj2 2023-07-29T00:08:16.901 DEBUG:boto:Method: GET 2023-07-29T00:08:16.902 DEBUG:boto:Path: /hiiqos-47/?prefix=obj2 2023-07-29T00:08:16.902 DEBUG:boto:Data: 2023-07-29T00:08:16.902 DEBUG:boto:Headers: {} 2023-07-29T00:08:16.903 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:16.903 DEBUG:boto:Port: 8001 2023-07-29T00:08:16.903 DEBUG:boto:Params: {} 2023-07-29T00:08:16.904 DEBUG:boto:Token: None 2023-07-29T00:08:16.904 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:16 GMT /hiiqos-47/ 2023-07-29T00:08:16.905 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xIphdg5s+tMVEsp2itCiVlMEeRc= 2023-07-29T00:08:16.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xIphdg5s+tMVEsp2itCiVlMEeRc=', 'Content-Length': '0'} 2023-07-29T00:08:16.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000437657a98f125f4a-0064c45870-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:16.908 DEBUG:boto:b'hiiqos-47obj21000falseobj22023-07-29T00:05:16.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:08:16.909 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:08:16.909 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:08:16.910 DEBUG:boto:path=/hiiqos-47/?prefix=obj2 2023-07-29T00:08:16.910 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj2 2023-07-29T00:08:16.910 DEBUG:boto:Method: GET 2023-07-29T00:08:16.911 DEBUG:boto:Path: /hiiqos-47/?prefix=obj2 2023-07-29T00:08:16.911 DEBUG:boto:Data: 2023-07-29T00:08:16.911 DEBUG:boto:Headers: {} 2023-07-29T00:08:16.912 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:16.912 DEBUG:boto:Port: 8000 2023-07-29T00:08:16.912 DEBUG:boto:Params: {} 2023-07-29T00:08:16.913 DEBUG:boto:Token: None 2023-07-29T00:08:16.913 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:16 GMT /hiiqos-47/ 2023-07-29T00:08:16.914 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xIphdg5s+tMVEsp2itCiVlMEeRc= 2023-07-29T00:08:16.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xIphdg5s+tMVEsp2itCiVlMEeRc=', 'Content-Length': '0'} 2023-07-29T00:08:16.916 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc93e53f3112917a-0064c45870-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:16.917 DEBUG:boto:b'hiiqos-47obj21000falseobj22023-07-29T00:05:16.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:08:16.918 DEBUG:boto:path=/hiiqos-47/obj2 2023-07-29T00:08:16.918 DEBUG:boto:auth_path=/hiiqos-47/obj2 2023-07-29T00:08:16.918 DEBUG:boto:Method: HEAD 2023-07-29T00:08:16.919 DEBUG:boto:Path: /hiiqos-47/obj2 2023-07-29T00:08:16.919 DEBUG:boto:Data: 2023-07-29T00:08:16.919 DEBUG:boto:Headers: {} 2023-07-29T00:08:16.920 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:16.920 DEBUG:boto:Port: 8001 2023-07-29T00:08:16.920 DEBUG:boto:Params: {} 2023-07-29T00:08:16.921 DEBUG:boto:Token: None 2023-07-29T00:08:16.921 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:16 GMT /hiiqos-47/obj2 2023-07-29T00:08:16.921 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZYgpNVjMgmTXoUr+exTPdRW6Fn0= 2023-07-29T00:08:16.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZYgpNVjMgmTXoUr+exTPdRW6Fn0=', 'Content-Length': '0'} 2023-07-29T00:08:16.924 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe5af6f692af9f70-0064c45870-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:16.924 DEBUG:boto:path=/hiiqos-47/obj2 2023-07-29T00:08:16.925 DEBUG:boto:auth_path=/hiiqos-47/obj2 2023-07-29T00:08:16.925 DEBUG:boto:Method: HEAD 2023-07-29T00:08:16.925 DEBUG:boto:Path: /hiiqos-47/obj2 2023-07-29T00:08:16.926 DEBUG:boto:Data: 2023-07-29T00:08:16.926 DEBUG:boto:Headers: {} 2023-07-29T00:08:16.926 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:16.927 DEBUG:boto:Port: 8000 2023-07-29T00:08:16.927 DEBUG:boto:Params: {} 2023-07-29T00:08:16.927 DEBUG:boto:Token: None 2023-07-29T00:08:16.928 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:16 GMT /hiiqos-47/obj2 2023-07-29T00:08:16.928 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZYgpNVjMgmTXoUr+exTPdRW6Fn0= 2023-07-29T00:08:16.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZYgpNVjMgmTXoUr+exTPdRW6Fn0=', 'Content-Length': '0'} 2023-07-29T00:08:16.930 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aa7cdf46ae5c59f1-0064c45870-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:16.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-07-29T00:08:16.931 DEBUG:boto:path=/hiiqos-47/obj2 2023-07-29T00:08:16.932 DEBUG:boto:auth_path=/hiiqos-47/obj2 2023-07-29T00:08:16.932 DEBUG:boto:Method: GET 2023-07-29T00:08:16.933 DEBUG:boto:Path: /hiiqos-47/obj2 2023-07-29T00:08:16.933 DEBUG:boto:Data: 2023-07-29T00:08:16.933 DEBUG:boto:Headers: {} 2023-07-29T00:08:16.934 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:16.934 DEBUG:boto:Port: 8001 2023-07-29T00:08:16.934 DEBUG:boto:Params: {} 2023-07-29T00:08:16.935 DEBUG:boto:Token: None 2023-07-29T00:08:16.935 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:16 GMT /hiiqos-47/obj2 2023-07-29T00:08:16.936 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:i4jDbkRctxFAbrbYhlsm5iw5lXo= 2023-07-29T00:08:16.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:i4jDbkRctxFAbrbYhlsm5iw5lXo=', 'Content-Length': '0'} 2023-07-29T00:08:16.938 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b61094f41d06b278-0064c45870-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:16.938 DEBUG:boto:path=/hiiqos-47/obj2 2023-07-29T00:08:16.939 DEBUG:boto:auth_path=/hiiqos-47/obj2 2023-07-29T00:08:16.939 DEBUG:boto:Method: GET 2023-07-29T00:08:16.940 DEBUG:boto:Path: /hiiqos-47/obj2 2023-07-29T00:08:16.940 DEBUG:boto:Data: 2023-07-29T00:08:16.940 DEBUG:boto:Headers: {} 2023-07-29T00:08:16.941 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:16.941 DEBUG:boto:Port: 8000 2023-07-29T00:08:16.941 DEBUG:boto:Params: {} 2023-07-29T00:08:16.942 DEBUG:boto:Token: None 2023-07-29T00:08:16.942 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:16 GMT /hiiqos-47/obj2 2023-07-29T00:08:16.943 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:i4jDbkRctxFAbrbYhlsm5iw5lXo= 2023-07-29T00:08:16.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:i4jDbkRctxFAbrbYhlsm5iw5lXo=', 'Content-Length': '0'} 2023-07-29T00:08:16.945 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e8a63fd452d17c2-0064c45870-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:16.946 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:08:16.947 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:08:16.947 DEBUG:boto:path=/hiiqos-47/?prefix=obj2 2023-07-29T00:08:16.947 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj2 2023-07-29T00:08:16.948 DEBUG:boto:Method: GET 2023-07-29T00:08:16.948 DEBUG:boto:Path: /hiiqos-47/?prefix=obj2 2023-07-29T00:08:16.948 DEBUG:boto:Data: 2023-07-29T00:08:16.949 DEBUG:boto:Headers: {} 2023-07-29T00:08:16.949 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:16.949 DEBUG:boto:Port: 8001 2023-07-29T00:08:16.950 DEBUG:boto:Params: {} 2023-07-29T00:08:16.950 DEBUG:boto:Token: None 2023-07-29T00:08:16.951 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:16 GMT /hiiqos-47/ 2023-07-29T00:08:16.951 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xIphdg5s+tMVEsp2itCiVlMEeRc= 2023-07-29T00:08:16.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xIphdg5s+tMVEsp2itCiVlMEeRc=', 'Content-Length': '0'} 2023-07-29T00:08:16.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ac6f9d02f5b5be4-0064c45870-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:16.954 DEBUG:boto:b'hiiqos-47obj21000falseobj22023-07-29T00:05:16.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:08:16.955 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:08:16.955 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:08:16.956 DEBUG:boto:path=/hiiqos-47/?prefix=obj2 2023-07-29T00:08:16.956 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj2 2023-07-29T00:08:16.956 DEBUG:boto:Method: GET 2023-07-29T00:08:16.957 DEBUG:boto:Path: /hiiqos-47/?prefix=obj2 2023-07-29T00:08:16.957 DEBUG:boto:Data: 2023-07-29T00:08:16.958 DEBUG:boto:Headers: {} 2023-07-29T00:08:16.958 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:16.958 DEBUG:boto:Port: 8000 2023-07-29T00:08:16.959 DEBUG:boto:Params: {} 2023-07-29T00:08:16.959 DEBUG:boto:Token: None 2023-07-29T00:08:16.960 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:16 GMT /hiiqos-47/ 2023-07-29T00:08:16.960 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xIphdg5s+tMVEsp2itCiVlMEeRc= 2023-07-29T00:08:16.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xIphdg5s+tMVEsp2itCiVlMEeRc=', 'Content-Length': '0'} 2023-07-29T00:08:16.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e1927e312b05961-0064c45870-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.004 DEBUG:boto:b'hiiqos-47obj21000falseobj22023-07-29T00:05:16.153Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:08:17.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-07-29T00:08:17.005 DEBUG:boto:path=/hiiqos-47/obj3 2023-07-29T00:08:17.006 DEBUG:boto:auth_path=/hiiqos-47/obj3 2023-07-29T00:08:17.006 DEBUG:boto:path=/hiiqos-47/obj3?versionId=null 2023-07-29T00:08:17.006 DEBUG:boto:auth_path=/hiiqos-47/obj3?versionId=null 2023-07-29T00:08:17.007 DEBUG:boto:Method: GET 2023-07-29T00:08:17.007 DEBUG:boto:Path: /hiiqos-47/obj3?versionId=null 2023-07-29T00:08:17.008 DEBUG:boto:Data: 2023-07-29T00:08:17.008 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.008 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:17.009 DEBUG:boto:Port: 8001 2023-07-29T00:08:17.009 DEBUG:boto:Params: {} 2023-07-29T00:08:17.009 DEBUG:boto:Token: None 2023-07-29T00:08:17.010 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj3?versionId=null 2023-07-29T00:08:17.010 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Y+zpFbu3X5Bl7BZ14NcfLIwP51o= 2023-07-29T00:08:17.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Y+zpFbu3X5Bl7BZ14NcfLIwP51o=', 'Content-Length': '0'} 2023-07-29T00:08:17.013 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000037ff8b3c741ef1b9-0064c45870-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.013 DEBUG:boto:path=/hiiqos-47/obj3 2023-07-29T00:08:17.014 DEBUG:boto:auth_path=/hiiqos-47/obj3 2023-07-29T00:08:17.014 DEBUG:boto:path=/hiiqos-47/obj3?versionId=null 2023-07-29T00:08:17.015 DEBUG:boto:auth_path=/hiiqos-47/obj3?versionId=null 2023-07-29T00:08:17.015 DEBUG:boto:Method: GET 2023-07-29T00:08:17.015 DEBUG:boto:Path: /hiiqos-47/obj3?versionId=null 2023-07-29T00:08:17.016 DEBUG:boto:Data: 2023-07-29T00:08:17.016 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.016 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:17.017 DEBUG:boto:Port: 8000 2023-07-29T00:08:17.017 DEBUG:boto:Params: {} 2023-07-29T00:08:17.018 DEBUG:boto:Token: None 2023-07-29T00:08:17.018 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj3?versionId=null 2023-07-29T00:08:17.018 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Y+zpFbu3X5Bl7BZ14NcfLIwP51o= 2023-07-29T00:08:17.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Y+zpFbu3X5Bl7BZ14NcfLIwP51o=', 'Content-Length': '0'} 2023-07-29T00:08:17.021 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004179c114bb00972e-0064c45870-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.021 DEBUG:boto:path=/hiiqos-47/obj3 2023-07-29T00:08:17.022 DEBUG:boto:auth_path=/hiiqos-47/obj3 2023-07-29T00:08:17.022 DEBUG:boto:path=/hiiqos-47/obj3?versionId=null 2023-07-29T00:08:17.022 DEBUG:boto:auth_path=/hiiqos-47/obj3?versionId=null 2023-07-29T00:08:17.023 DEBUG:boto:Method: HEAD 2023-07-29T00:08:17.023 DEBUG:boto:Path: /hiiqos-47/obj3?versionId=null 2023-07-29T00:08:17.023 DEBUG:boto:Data: 2023-07-29T00:08:17.024 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.024 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:17.024 DEBUG:boto:Port: 8001 2023-07-29T00:08:17.025 DEBUG:boto:Params: {} 2023-07-29T00:08:17.025 DEBUG:boto:Token: None 2023-07-29T00:08:17.026 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj3?versionId=null 2023-07-29T00:08:17.026 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bfThKMAWu4tpL2N8KHCIHHefa0g= 2023-07-29T00:08:17.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bfThKMAWu4tpL2N8KHCIHHefa0g=', 'Content-Length': '0'} 2023-07-29T00:08:17.028 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cceaffb47ccaaa5b-0064c45870-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.029 DEBUG:boto:path=/hiiqos-47/obj3 2023-07-29T00:08:17.029 DEBUG:boto:auth_path=/hiiqos-47/obj3 2023-07-29T00:08:17.029 DEBUG:boto:path=/hiiqos-47/obj3?versionId=null 2023-07-29T00:08:17.030 DEBUG:boto:auth_path=/hiiqos-47/obj3?versionId=null 2023-07-29T00:08:17.030 DEBUG:boto:Method: HEAD 2023-07-29T00:08:17.030 DEBUG:boto:Path: /hiiqos-47/obj3?versionId=null 2023-07-29T00:08:17.031 DEBUG:boto:Data: 2023-07-29T00:08:17.031 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.031 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:17.032 DEBUG:boto:Port: 8000 2023-07-29T00:08:17.032 DEBUG:boto:Params: {} 2023-07-29T00:08:17.032 DEBUG:boto:Token: None 2023-07-29T00:08:17.033 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj3?versionId=null 2023-07-29T00:08:17.033 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bfThKMAWu4tpL2N8KHCIHHefa0g= 2023-07-29T00:08:17.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bfThKMAWu4tpL2N8KHCIHHefa0g=', 'Content-Length': '0'} 2023-07-29T00:08:17.035 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a311332af050f72-0064c45871-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-07-29T00:08:17.036 DEBUG:boto:path=/hiiqos-47/obj3 2023-07-29T00:08:17.037 DEBUG:boto:auth_path=/hiiqos-47/obj3 2023-07-29T00:08:17.037 DEBUG:boto:path=/hiiqos-47/obj3?versionId=null 2023-07-29T00:08:17.037 DEBUG:boto:auth_path=/hiiqos-47/obj3?versionId=null 2023-07-29T00:08:17.038 DEBUG:boto:Method: GET 2023-07-29T00:08:17.038 DEBUG:boto:Path: /hiiqos-47/obj3?versionId=null 2023-07-29T00:08:17.038 DEBUG:boto:Data: 2023-07-29T00:08:17.039 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.039 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:17.040 DEBUG:boto:Port: 8001 2023-07-29T00:08:17.040 DEBUG:boto:Params: {} 2023-07-29T00:08:17.040 DEBUG:boto:Token: None 2023-07-29T00:08:17.041 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj3?versionId=null 2023-07-29T00:08:17.041 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Y+zpFbu3X5Bl7BZ14NcfLIwP51o= 2023-07-29T00:08:17.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Y+zpFbu3X5Bl7BZ14NcfLIwP51o=', 'Content-Length': '0'} 2023-07-29T00:08:17.044 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000112e34a41c4b76ad-0064c45871-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.044 DEBUG:boto:path=/hiiqos-47/obj3 2023-07-29T00:08:17.045 DEBUG:boto:auth_path=/hiiqos-47/obj3 2023-07-29T00:08:17.045 DEBUG:boto:path=/hiiqos-47/obj3?versionId=null 2023-07-29T00:08:17.045 DEBUG:boto:auth_path=/hiiqos-47/obj3?versionId=null 2023-07-29T00:08:17.046 DEBUG:boto:Method: GET 2023-07-29T00:08:17.046 DEBUG:boto:Path: /hiiqos-47/obj3?versionId=null 2023-07-29T00:08:17.046 DEBUG:boto:Data: 2023-07-29T00:08:17.047 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.047 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:17.048 DEBUG:boto:Port: 8000 2023-07-29T00:08:17.048 DEBUG:boto:Params: {} 2023-07-29T00:08:17.048 DEBUG:boto:Token: None 2023-07-29T00:08:17.049 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj3?versionId=null 2023-07-29T00:08:17.049 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Y+zpFbu3X5Bl7BZ14NcfLIwP51o= 2023-07-29T00:08:17.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Y+zpFbu3X5Bl7BZ14NcfLIwP51o=', 'Content-Length': '0'} 2023-07-29T00:08:17.052 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000a3c419a4afd10a9-0064c45871-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.052 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:08:17.053 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:08:17.053 DEBUG:boto:path=/hiiqos-47/?prefix=obj3 2023-07-29T00:08:17.054 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj3 2023-07-29T00:08:17.054 DEBUG:boto:Method: GET 2023-07-29T00:08:17.054 DEBUG:boto:Path: /hiiqos-47/?prefix=obj3 2023-07-29T00:08:17.055 DEBUG:boto:Data: 2023-07-29T00:08:17.055 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.055 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:17.056 DEBUG:boto:Port: 8001 2023-07-29T00:08:17.056 DEBUG:boto:Params: {} 2023-07-29T00:08:17.057 DEBUG:boto:Token: None 2023-07-29T00:08:17.057 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/ 2023-07-29T00:08:17.058 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mEkmS6vwmcrs+wZGyCWeP8Ranz4= 2023-07-29T00:08:17.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mEkmS6vwmcrs+wZGyCWeP8Ranz4=', 'Content-Length': '0'} 2023-07-29T00:08:17.061 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002de08f4dac3ec741-0064c45871-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.061 DEBUG:boto:b'hiiqos-47obj31000falseobj32023-07-29T00:05:16.174Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:08:17.062 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:08:17.063 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:08:17.063 DEBUG:boto:path=/hiiqos-47/?prefix=obj3 2023-07-29T00:08:17.064 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj3 2023-07-29T00:08:17.064 DEBUG:boto:Method: GET 2023-07-29T00:08:17.064 DEBUG:boto:Path: /hiiqos-47/?prefix=obj3 2023-07-29T00:08:17.065 DEBUG:boto:Data: 2023-07-29T00:08:17.065 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.066 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:17.066 DEBUG:boto:Port: 8000 2023-07-29T00:08:17.066 DEBUG:boto:Params: {} 2023-07-29T00:08:17.067 DEBUG:boto:Token: None 2023-07-29T00:08:17.067 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/ 2023-07-29T00:08:17.067 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mEkmS6vwmcrs+wZGyCWeP8Ranz4= 2023-07-29T00:08:17.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mEkmS6vwmcrs+wZGyCWeP8Ranz4=', 'Content-Length': '0'} 2023-07-29T00:08:17.071 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdf3ccd0aa4af772-0064c45871-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.072 DEBUG:boto:b'hiiqos-47obj31000falseobj32023-07-29T00:05:16.174Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:08:17.073 DEBUG:boto:path=/hiiqos-47/obj3 2023-07-29T00:08:17.074 DEBUG:boto:auth_path=/hiiqos-47/obj3 2023-07-29T00:08:17.075 DEBUG:boto:Method: HEAD 2023-07-29T00:08:17.075 DEBUG:boto:Path: /hiiqos-47/obj3 2023-07-29T00:08:17.076 DEBUG:boto:Data: 2023-07-29T00:08:17.076 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.076 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:17.077 DEBUG:boto:Port: 8001 2023-07-29T00:08:17.077 DEBUG:boto:Params: {} 2023-07-29T00:08:17.077 DEBUG:boto:Token: None 2023-07-29T00:08:17.078 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj3 2023-07-29T00:08:17.078 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BmX85IOggGGHahmPd3tCmh4zi/Y= 2023-07-29T00:08:17.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BmX85IOggGGHahmPd3tCmh4zi/Y=', 'Content-Length': '0'} 2023-07-29T00:08:17.080 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ad42a8c931a66402-0064c45871-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.081 DEBUG:boto:path=/hiiqos-47/obj3 2023-07-29T00:08:17.081 DEBUG:boto:auth_path=/hiiqos-47/obj3 2023-07-29T00:08:17.082 DEBUG:boto:Method: HEAD 2023-07-29T00:08:17.082 DEBUG:boto:Path: /hiiqos-47/obj3 2023-07-29T00:08:17.083 DEBUG:boto:Data: 2023-07-29T00:08:17.083 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.083 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:17.084 DEBUG:boto:Port: 8000 2023-07-29T00:08:17.084 DEBUG:boto:Params: {} 2023-07-29T00:08:17.084 DEBUG:boto:Token: None 2023-07-29T00:08:17.085 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj3 2023-07-29T00:08:17.085 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BmX85IOggGGHahmPd3tCmh4zi/Y= 2023-07-29T00:08:17.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BmX85IOggGGHahmPd3tCmh4zi/Y=', 'Content-Length': '0'} 2023-07-29T00:08:17.088 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc0552a9d2da5550-0064c45871-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-07-29T00:08:17.089 DEBUG:boto:path=/hiiqos-47/obj3 2023-07-29T00:08:17.089 DEBUG:boto:auth_path=/hiiqos-47/obj3 2023-07-29T00:08:17.089 DEBUG:boto:Method: GET 2023-07-29T00:08:17.090 DEBUG:boto:Path: /hiiqos-47/obj3 2023-07-29T00:08:17.090 DEBUG:boto:Data: 2023-07-29T00:08:17.091 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.091 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:17.091 DEBUG:boto:Port: 8001 2023-07-29T00:08:17.092 DEBUG:boto:Params: {} 2023-07-29T00:08:17.092 DEBUG:boto:Token: None 2023-07-29T00:08:17.093 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj3 2023-07-29T00:08:17.093 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bUrXEJf514iolJXT9jcnsbuse2Y= 2023-07-29T00:08:17.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bUrXEJf514iolJXT9jcnsbuse2Y=', 'Content-Length': '0'} 2023-07-29T00:08:17.096 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000020955b2158cedf41-0064c45871-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.136 DEBUG:boto:path=/hiiqos-47/obj3 2023-07-29T00:08:17.137 DEBUG:boto:auth_path=/hiiqos-47/obj3 2023-07-29T00:08:17.137 DEBUG:boto:Method: GET 2023-07-29T00:08:17.138 DEBUG:boto:Path: /hiiqos-47/obj3 2023-07-29T00:08:17.138 DEBUG:boto:Data: 2023-07-29T00:08:17.138 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.139 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:17.139 DEBUG:boto:Port: 8000 2023-07-29T00:08:17.139 DEBUG:boto:Params: {} 2023-07-29T00:08:17.140 DEBUG:boto:Token: None 2023-07-29T00:08:17.140 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj3 2023-07-29T00:08:17.141 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bUrXEJf514iolJXT9jcnsbuse2Y= 2023-07-29T00:08:17.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bUrXEJf514iolJXT9jcnsbuse2Y=', 'Content-Length': '0'} 2023-07-29T00:08:17.143 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c7b18ac1937815b-0064c45871-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.144 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:08:17.144 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:08:17.145 DEBUG:boto:path=/hiiqos-47/?prefix=obj3 2023-07-29T00:08:17.145 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj3 2023-07-29T00:08:17.145 DEBUG:boto:Method: GET 2023-07-29T00:08:17.146 DEBUG:boto:Path: /hiiqos-47/?prefix=obj3 2023-07-29T00:08:17.146 DEBUG:boto:Data: 2023-07-29T00:08:17.146 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.147 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:17.147 DEBUG:boto:Port: 8001 2023-07-29T00:08:17.148 DEBUG:boto:Params: {} 2023-07-29T00:08:17.148 DEBUG:boto:Token: None 2023-07-29T00:08:17.149 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/ 2023-07-29T00:08:17.149 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mEkmS6vwmcrs+wZGyCWeP8Ranz4= 2023-07-29T00:08:17.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mEkmS6vwmcrs+wZGyCWeP8Ranz4=', 'Content-Length': '0'} 2023-07-29T00:08:17.152 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058e360868c201c26-0064c45871-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.153 DEBUG:boto:b'hiiqos-47obj31000falseobj32023-07-29T00:05:16.174Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:08:17.153 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:08:17.154 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:08:17.154 DEBUG:boto:path=/hiiqos-47/?prefix=obj3 2023-07-29T00:08:17.154 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj3 2023-07-29T00:08:17.155 DEBUG:boto:Method: GET 2023-07-29T00:08:17.155 DEBUG:boto:Path: /hiiqos-47/?prefix=obj3 2023-07-29T00:08:17.156 DEBUG:boto:Data: 2023-07-29T00:08:17.156 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.156 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:17.157 DEBUG:boto:Port: 8000 2023-07-29T00:08:17.157 DEBUG:boto:Params: {} 2023-07-29T00:08:17.157 DEBUG:boto:Token: None 2023-07-29T00:08:17.158 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/ 2023-07-29T00:08:17.158 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mEkmS6vwmcrs+wZGyCWeP8Ranz4= 2023-07-29T00:08:17.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mEkmS6vwmcrs+wZGyCWeP8Ranz4=', 'Content-Length': '0'} 2023-07-29T00:08:17.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0a3b09372af9993-0064c45871-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.162 DEBUG:boto:b'hiiqos-47obj31000falseobj32023-07-29T00:05:16.174Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:08:17.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-07-29T00:08:17.163 DEBUG:boto:path=/hiiqos-47/obj4 2023-07-29T00:08:17.163 DEBUG:boto:auth_path=/hiiqos-47/obj4 2023-07-29T00:08:17.164 DEBUG:boto:path=/hiiqos-47/obj4?versionId=null 2023-07-29T00:08:17.164 DEBUG:boto:auth_path=/hiiqos-47/obj4?versionId=null 2023-07-29T00:08:17.165 DEBUG:boto:Method: GET 2023-07-29T00:08:17.165 DEBUG:boto:Path: /hiiqos-47/obj4?versionId=null 2023-07-29T00:08:17.165 DEBUG:boto:Data: 2023-07-29T00:08:17.166 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.166 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:17.166 DEBUG:boto:Port: 8001 2023-07-29T00:08:17.167 DEBUG:boto:Params: {} 2023-07-29T00:08:17.167 DEBUG:boto:Token: None 2023-07-29T00:08:17.168 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj4?versionId=null 2023-07-29T00:08:17.168 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:t+tiC72JOR5zHBQr3y04nKkmnSg= 2023-07-29T00:08:17.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t+tiC72JOR5zHBQr3y04nKkmnSg=', 'Content-Length': '0'} 2023-07-29T00:08:17.170 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b399deb72774cb4f-0064c45871-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.171 DEBUG:boto:path=/hiiqos-47/obj4 2023-07-29T00:08:17.171 DEBUG:boto:auth_path=/hiiqos-47/obj4 2023-07-29T00:08:17.172 DEBUG:boto:path=/hiiqos-47/obj4?versionId=null 2023-07-29T00:08:17.172 DEBUG:boto:auth_path=/hiiqos-47/obj4?versionId=null 2023-07-29T00:08:17.172 DEBUG:boto:Method: GET 2023-07-29T00:08:17.173 DEBUG:boto:Path: /hiiqos-47/obj4?versionId=null 2023-07-29T00:08:17.173 DEBUG:boto:Data: 2023-07-29T00:08:17.173 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.174 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:17.174 DEBUG:boto:Port: 8000 2023-07-29T00:08:17.174 DEBUG:boto:Params: {} 2023-07-29T00:08:17.175 DEBUG:boto:Token: None 2023-07-29T00:08:17.175 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj4?versionId=null 2023-07-29T00:08:17.176 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:t+tiC72JOR5zHBQr3y04nKkmnSg= 2023-07-29T00:08:17.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t+tiC72JOR5zHBQr3y04nKkmnSg=', 'Content-Length': '0'} 2023-07-29T00:08:17.178 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe861778b0c98b30-0064c45871-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.220 DEBUG:boto:path=/hiiqos-47/obj4 2023-07-29T00:08:17.221 DEBUG:boto:auth_path=/hiiqos-47/obj4 2023-07-29T00:08:17.221 DEBUG:boto:path=/hiiqos-47/obj4?versionId=null 2023-07-29T00:08:17.221 DEBUG:boto:auth_path=/hiiqos-47/obj4?versionId=null 2023-07-29T00:08:17.222 DEBUG:boto:Method: HEAD 2023-07-29T00:08:17.222 DEBUG:boto:Path: /hiiqos-47/obj4?versionId=null 2023-07-29T00:08:17.222 DEBUG:boto:Data: 2023-07-29T00:08:17.223 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.223 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:17.223 DEBUG:boto:Port: 8001 2023-07-29T00:08:17.224 DEBUG:boto:Params: {} 2023-07-29T00:08:17.224 DEBUG:boto:Token: None 2023-07-29T00:08:17.225 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj4?versionId=null 2023-07-29T00:08:17.225 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CgEYkGWfiwk6YDf+qJkP19/xhSc= 2023-07-29T00:08:17.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CgEYkGWfiwk6YDf+qJkP19/xhSc=', 'Content-Length': '0'} 2023-07-29T00:08:17.227 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000727f4577b9583a60-0064c45871-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.228 DEBUG:boto:path=/hiiqos-47/obj4 2023-07-29T00:08:17.228 DEBUG:boto:auth_path=/hiiqos-47/obj4 2023-07-29T00:08:17.229 DEBUG:boto:path=/hiiqos-47/obj4?versionId=null 2023-07-29T00:08:17.229 DEBUG:boto:auth_path=/hiiqos-47/obj4?versionId=null 2023-07-29T00:08:17.229 DEBUG:boto:Method: HEAD 2023-07-29T00:08:17.230 DEBUG:boto:Path: /hiiqos-47/obj4?versionId=null 2023-07-29T00:08:17.230 DEBUG:boto:Data: 2023-07-29T00:08:17.230 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.231 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:17.231 DEBUG:boto:Port: 8000 2023-07-29T00:08:17.231 DEBUG:boto:Params: {} 2023-07-29T00:08:17.232 DEBUG:boto:Token: None 2023-07-29T00:08:17.232 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj4?versionId=null 2023-07-29T00:08:17.233 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CgEYkGWfiwk6YDf+qJkP19/xhSc= 2023-07-29T00:08:17.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CgEYkGWfiwk6YDf+qJkP19/xhSc=', 'Content-Length': '0'} 2023-07-29T00:08:17.236 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000083e1fc68c37ec286-0064c45871-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-07-29T00:08:17.237 DEBUG:boto:path=/hiiqos-47/obj4 2023-07-29T00:08:17.237 DEBUG:boto:auth_path=/hiiqos-47/obj4 2023-07-29T00:08:17.237 DEBUG:boto:path=/hiiqos-47/obj4?versionId=null 2023-07-29T00:08:17.238 DEBUG:boto:auth_path=/hiiqos-47/obj4?versionId=null 2023-07-29T00:08:17.238 DEBUG:boto:Method: GET 2023-07-29T00:08:17.238 DEBUG:boto:Path: /hiiqos-47/obj4?versionId=null 2023-07-29T00:08:17.239 DEBUG:boto:Data: 2023-07-29T00:08:17.239 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.239 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:17.240 DEBUG:boto:Port: 8001 2023-07-29T00:08:17.240 DEBUG:boto:Params: {} 2023-07-29T00:08:17.241 DEBUG:boto:Token: None 2023-07-29T00:08:17.241 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj4?versionId=null 2023-07-29T00:08:17.241 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:t+tiC72JOR5zHBQr3y04nKkmnSg= 2023-07-29T00:08:17.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t+tiC72JOR5zHBQr3y04nKkmnSg=', 'Content-Length': '0'} 2023-07-29T00:08:17.244 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a9a30e6b799cbb7-0064c45871-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.244 DEBUG:boto:path=/hiiqos-47/obj4 2023-07-29T00:08:17.245 DEBUG:boto:auth_path=/hiiqos-47/obj4 2023-07-29T00:08:17.245 DEBUG:boto:path=/hiiqos-47/obj4?versionId=null 2023-07-29T00:08:17.245 DEBUG:boto:auth_path=/hiiqos-47/obj4?versionId=null 2023-07-29T00:08:17.246 DEBUG:boto:Method: GET 2023-07-29T00:08:17.246 DEBUG:boto:Path: /hiiqos-47/obj4?versionId=null 2023-07-29T00:08:17.247 DEBUG:boto:Data: 2023-07-29T00:08:17.247 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.248 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:17.249 DEBUG:boto:Port: 8000 2023-07-29T00:08:17.250 DEBUG:boto:Params: {} 2023-07-29T00:08:17.251 DEBUG:boto:Token: None 2023-07-29T00:08:17.252 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj4?versionId=null 2023-07-29T00:08:17.253 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:t+tiC72JOR5zHBQr3y04nKkmnSg= 2023-07-29T00:08:17.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t+tiC72JOR5zHBQr3y04nKkmnSg=', 'Content-Length': '0'} 2023-07-29T00:08:17.255 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e6a75615cd6fa170-0064c45871-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.256 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:08:17.256 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:08:17.257 DEBUG:boto:path=/hiiqos-47/?prefix=obj4 2023-07-29T00:08:17.257 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj4 2023-07-29T00:08:17.257 DEBUG:boto:Method: GET 2023-07-29T00:08:17.258 DEBUG:boto:Path: /hiiqos-47/?prefix=obj4 2023-07-29T00:08:17.258 DEBUG:boto:Data: 2023-07-29T00:08:17.258 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.259 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:17.259 DEBUG:boto:Port: 8001 2023-07-29T00:08:17.260 DEBUG:boto:Params: {} 2023-07-29T00:08:17.260 DEBUG:boto:Token: None 2023-07-29T00:08:17.261 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/ 2023-07-29T00:08:17.261 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mEkmS6vwmcrs+wZGyCWeP8Ranz4= 2023-07-29T00:08:17.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mEkmS6vwmcrs+wZGyCWeP8Ranz4=', 'Content-Length': '0'} 2023-07-29T00:08:17.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000108f2b080c67daf2-0064c45871-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.265 DEBUG:boto:b'hiiqos-47obj41000falseobj42023-07-29T00:05:16.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:08:17.265 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:08:17.266 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:08:17.266 DEBUG:boto:path=/hiiqos-47/?prefix=obj4 2023-07-29T00:08:17.267 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj4 2023-07-29T00:08:17.267 DEBUG:boto:Method: GET 2023-07-29T00:08:17.268 DEBUG:boto:Path: /hiiqos-47/?prefix=obj4 2023-07-29T00:08:17.269 DEBUG:boto:Data: 2023-07-29T00:08:17.269 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.270 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:17.270 DEBUG:boto:Port: 8000 2023-07-29T00:08:17.271 DEBUG:boto:Params: {} 2023-07-29T00:08:17.272 DEBUG:boto:Token: None 2023-07-29T00:08:17.273 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/ 2023-07-29T00:08:17.274 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mEkmS6vwmcrs+wZGyCWeP8Ranz4= 2023-07-29T00:08:17.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mEkmS6vwmcrs+wZGyCWeP8Ranz4=', 'Content-Length': '0'} 2023-07-29T00:08:17.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8f32057dfd832ee-0064c45871-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.279 DEBUG:boto:b'hiiqos-47obj41000falseobj42023-07-29T00:05:16.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:08:17.279 DEBUG:boto:path=/hiiqos-47/obj4 2023-07-29T00:08:17.280 DEBUG:boto:auth_path=/hiiqos-47/obj4 2023-07-29T00:08:17.280 DEBUG:boto:Method: HEAD 2023-07-29T00:08:17.280 DEBUG:boto:Path: /hiiqos-47/obj4 2023-07-29T00:08:17.281 DEBUG:boto:Data: 2023-07-29T00:08:17.281 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.281 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:17.282 DEBUG:boto:Port: 8001 2023-07-29T00:08:17.282 DEBUG:boto:Params: {} 2023-07-29T00:08:17.282 DEBUG:boto:Token: None 2023-07-29T00:08:17.283 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj4 2023-07-29T00:08:17.283 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:H3v613q4ZXCws7yf2XbszWmMfXU= 2023-07-29T00:08:17.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:H3v613q4ZXCws7yf2XbszWmMfXU=', 'Content-Length': '0'} 2023-07-29T00:08:17.285 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004cf0e3c88232c6f9-0064c45871-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.286 DEBUG:boto:path=/hiiqos-47/obj4 2023-07-29T00:08:17.286 DEBUG:boto:auth_path=/hiiqos-47/obj4 2023-07-29T00:08:17.287 DEBUG:boto:Method: HEAD 2023-07-29T00:08:17.287 DEBUG:boto:Path: /hiiqos-47/obj4 2023-07-29T00:08:17.287 DEBUG:boto:Data: 2023-07-29T00:08:17.288 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.288 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:17.288 DEBUG:boto:Port: 8000 2023-07-29T00:08:17.289 DEBUG:boto:Params: {} 2023-07-29T00:08:17.289 DEBUG:boto:Token: None 2023-07-29T00:08:17.290 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj4 2023-07-29T00:08:17.290 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:H3v613q4ZXCws7yf2XbszWmMfXU= 2023-07-29T00:08:17.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:H3v613q4ZXCws7yf2XbszWmMfXU=', 'Content-Length': '0'} 2023-07-29T00:08:17.292 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cdbe29e2c86a667f-0064c45871-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-07-29T00:08:17.293 DEBUG:boto:path=/hiiqos-47/obj4 2023-07-29T00:08:17.293 DEBUG:boto:auth_path=/hiiqos-47/obj4 2023-07-29T00:08:17.294 DEBUG:boto:Method: GET 2023-07-29T00:08:17.294 DEBUG:boto:Path: /hiiqos-47/obj4 2023-07-29T00:08:17.295 DEBUG:boto:Data: 2023-07-29T00:08:17.295 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.295 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:17.296 DEBUG:boto:Port: 8001 2023-07-29T00:08:17.296 DEBUG:boto:Params: {} 2023-07-29T00:08:17.297 DEBUG:boto:Token: None 2023-07-29T00:08:17.297 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj4 2023-07-29T00:08:17.297 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S7WNvDqr9tKkBk7BEHeyQsTWyjA= 2023-07-29T00:08:17.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S7WNvDqr9tKkBk7BEHeyQsTWyjA=', 'Content-Length': '0'} 2023-07-29T00:08:17.299 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b7a17e74b866b31-0064c45871-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.340 DEBUG:boto:path=/hiiqos-47/obj4 2023-07-29T00:08:17.341 DEBUG:boto:auth_path=/hiiqos-47/obj4 2023-07-29T00:08:17.342 DEBUG:boto:Method: GET 2023-07-29T00:08:17.343 DEBUG:boto:Path: /hiiqos-47/obj4 2023-07-29T00:08:17.344 DEBUG:boto:Data: 2023-07-29T00:08:17.344 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.344 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:17.345 DEBUG:boto:Port: 8000 2023-07-29T00:08:17.345 DEBUG:boto:Params: {} 2023-07-29T00:08:17.345 DEBUG:boto:Token: None 2023-07-29T00:08:17.346 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj4 2023-07-29T00:08:17.346 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S7WNvDqr9tKkBk7BEHeyQsTWyjA= 2023-07-29T00:08:17.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S7WNvDqr9tKkBk7BEHeyQsTWyjA=', 'Content-Length': '0'} 2023-07-29T00:08:17.349 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:05:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f19474f4ccc6f7c1-0064c45871-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.349 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:08:17.350 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:08:17.350 DEBUG:boto:path=/hiiqos-47/?prefix=obj4 2023-07-29T00:08:17.351 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj4 2023-07-29T00:08:17.351 DEBUG:boto:Method: GET 2023-07-29T00:08:17.351 DEBUG:boto:Path: /hiiqos-47/?prefix=obj4 2023-07-29T00:08:17.352 DEBUG:boto:Data: 2023-07-29T00:08:17.352 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.352 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:17.353 DEBUG:boto:Port: 8001 2023-07-29T00:08:17.353 DEBUG:boto:Params: {} 2023-07-29T00:08:17.353 DEBUG:boto:Token: None 2023-07-29T00:08:17.354 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/ 2023-07-29T00:08:17.354 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mEkmS6vwmcrs+wZGyCWeP8Ranz4= 2023-07-29T00:08:17.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mEkmS6vwmcrs+wZGyCWeP8Ranz4=', 'Content-Length': '0'} 2023-07-29T00:08:17.357 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fa03c52c3309d24-0064c45871-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.358 DEBUG:boto:b'hiiqos-47obj41000falseobj42023-07-29T00:05:16.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:08:17.358 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:08:17.359 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:08:17.359 DEBUG:boto:path=/hiiqos-47/?prefix=obj4 2023-07-29T00:08:17.359 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj4 2023-07-29T00:08:17.360 DEBUG:boto:Method: GET 2023-07-29T00:08:17.360 DEBUG:boto:Path: /hiiqos-47/?prefix=obj4 2023-07-29T00:08:17.361 DEBUG:boto:Data: 2023-07-29T00:08:17.361 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.361 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:17.362 DEBUG:boto:Port: 8000 2023-07-29T00:08:17.362 DEBUG:boto:Params: {} 2023-07-29T00:08:17.362 DEBUG:boto:Token: None 2023-07-29T00:08:17.363 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/ 2023-07-29T00:08:17.363 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mEkmS6vwmcrs+wZGyCWeP8Ranz4= 2023-07-29T00:08:17.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mEkmS6vwmcrs+wZGyCWeP8Ranz4=', 'Content-Length': '0'} 2023-07-29T00:08:17.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb49066c0168f31b-0064c45871-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.367 DEBUG:boto:b'hiiqos-47obj41000falseobj42023-07-29T00:05:16.190Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:08:17.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-07-29T00:08:17.369 DEBUG:boto:path=/hiiqos-47/obj5 2023-07-29T00:08:17.370 DEBUG:boto:auth_path=/hiiqos-47/obj5 2023-07-29T00:08:17.371 DEBUG:boto:path=/hiiqos-47/obj5?versionId=null 2023-07-29T00:08:17.372 DEBUG:boto:auth_path=/hiiqos-47/obj5?versionId=null 2023-07-29T00:08:17.372 DEBUG:boto:Method: GET 2023-07-29T00:08:17.372 DEBUG:boto:Path: /hiiqos-47/obj5?versionId=null 2023-07-29T00:08:17.373 DEBUG:boto:Data: 2023-07-29T00:08:17.373 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.374 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:17.374 DEBUG:boto:Port: 8001 2023-07-29T00:08:17.374 DEBUG:boto:Params: {} 2023-07-29T00:08:17.375 DEBUG:boto:Token: None 2023-07-29T00:08:17.375 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj5?versionId=null 2023-07-29T00:08:17.375 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:N7jY1HI4bR//QdMiLbHTnrImUBE= 2023-07-29T00:08:17.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:N7jY1HI4bR//QdMiLbHTnrImUBE=', 'Content-Length': '0'} 2023-07-29T00:08:17.378 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000074447c18274de670-0064c45871-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.378 DEBUG:boto:path=/hiiqos-47/obj5 2023-07-29T00:08:17.379 DEBUG:boto:auth_path=/hiiqos-47/obj5 2023-07-29T00:08:17.379 DEBUG:boto:path=/hiiqos-47/obj5?versionId=null 2023-07-29T00:08:17.379 DEBUG:boto:auth_path=/hiiqos-47/obj5?versionId=null 2023-07-29T00:08:17.380 DEBUG:boto:Method: GET 2023-07-29T00:08:17.380 DEBUG:boto:Path: /hiiqos-47/obj5?versionId=null 2023-07-29T00:08:17.381 DEBUG:boto:Data: 2023-07-29T00:08:17.381 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.381 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:17.382 DEBUG:boto:Port: 8000 2023-07-29T00:08:17.382 DEBUG:boto:Params: {} 2023-07-29T00:08:17.382 DEBUG:boto:Token: None 2023-07-29T00:08:17.383 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj5?versionId=null 2023-07-29T00:08:17.383 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:N7jY1HI4bR//QdMiLbHTnrImUBE= 2023-07-29T00:08:17.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:N7jY1HI4bR//QdMiLbHTnrImUBE=', 'Content-Length': '0'} 2023-07-29T00:08:17.386 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004057b0e878e123a7-0064c45871-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.386 DEBUG:boto:path=/hiiqos-47/obj5 2023-07-29T00:08:17.387 DEBUG:boto:auth_path=/hiiqos-47/obj5 2023-07-29T00:08:17.387 DEBUG:boto:path=/hiiqos-47/obj5?versionId=null 2023-07-29T00:08:17.387 DEBUG:boto:auth_path=/hiiqos-47/obj5?versionId=null 2023-07-29T00:08:17.388 DEBUG:boto:Method: HEAD 2023-07-29T00:08:17.388 DEBUG:boto:Path: /hiiqos-47/obj5?versionId=null 2023-07-29T00:08:17.388 DEBUG:boto:Data: 2023-07-29T00:08:17.389 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.389 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:17.389 DEBUG:boto:Port: 8001 2023-07-29T00:08:17.390 DEBUG:boto:Params: {} 2023-07-29T00:08:17.390 DEBUG:boto:Token: None 2023-07-29T00:08:17.391 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj5?versionId=null 2023-07-29T00:08:17.391 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TgVqX4lEH6GDhWl1lrdRzwsOGWQ= 2023-07-29T00:08:17.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TgVqX4lEH6GDhWl1lrdRzwsOGWQ=', 'Content-Length': '0'} 2023-07-29T00:08:17.393 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007ae60298b9d2d188-0064c45871-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.394 DEBUG:boto:path=/hiiqos-47/obj5 2023-07-29T00:08:17.394 DEBUG:boto:auth_path=/hiiqos-47/obj5 2023-07-29T00:08:17.394 DEBUG:boto:path=/hiiqos-47/obj5?versionId=null 2023-07-29T00:08:17.395 DEBUG:boto:auth_path=/hiiqos-47/obj5?versionId=null 2023-07-29T00:08:17.395 DEBUG:boto:Method: HEAD 2023-07-29T00:08:17.395 DEBUG:boto:Path: /hiiqos-47/obj5?versionId=null 2023-07-29T00:08:17.396 DEBUG:boto:Data: 2023-07-29T00:08:17.396 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.396 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:17.397 DEBUG:boto:Port: 8000 2023-07-29T00:08:17.397 DEBUG:boto:Params: {} 2023-07-29T00:08:17.397 DEBUG:boto:Token: None 2023-07-29T00:08:17.398 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj5?versionId=null 2023-07-29T00:08:17.398 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TgVqX4lEH6GDhWl1lrdRzwsOGWQ= 2023-07-29T00:08:17.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TgVqX4lEH6GDhWl1lrdRzwsOGWQ=', 'Content-Length': '0'} 2023-07-29T00:08:17.400 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f0f7ff0c8f4a65b-0064c45871-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-07-29T00:08:17.401 DEBUG:boto:path=/hiiqos-47/obj5 2023-07-29T00:08:17.402 DEBUG:boto:auth_path=/hiiqos-47/obj5 2023-07-29T00:08:17.402 DEBUG:boto:path=/hiiqos-47/obj5?versionId=null 2023-07-29T00:08:17.402 DEBUG:boto:auth_path=/hiiqos-47/obj5?versionId=null 2023-07-29T00:08:17.403 DEBUG:boto:Method: GET 2023-07-29T00:08:17.403 DEBUG:boto:Path: /hiiqos-47/obj5?versionId=null 2023-07-29T00:08:17.404 DEBUG:boto:Data: 2023-07-29T00:08:17.404 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.405 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:17.405 DEBUG:boto:Port: 8001 2023-07-29T00:08:17.405 DEBUG:boto:Params: {} 2023-07-29T00:08:17.406 DEBUG:boto:Token: None 2023-07-29T00:08:17.406 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj5?versionId=null 2023-07-29T00:08:17.407 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:N7jY1HI4bR//QdMiLbHTnrImUBE= 2023-07-29T00:08:17.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:N7jY1HI4bR//QdMiLbHTnrImUBE=', 'Content-Length': '0'} 2023-07-29T00:08:17.409 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000070ce645b39054f41-0064c45871-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.452 DEBUG:boto:path=/hiiqos-47/obj5 2023-07-29T00:08:17.453 DEBUG:boto:auth_path=/hiiqos-47/obj5 2023-07-29T00:08:17.453 DEBUG:boto:path=/hiiqos-47/obj5?versionId=null 2023-07-29T00:08:17.454 DEBUG:boto:auth_path=/hiiqos-47/obj5?versionId=null 2023-07-29T00:08:17.454 DEBUG:boto:Method: GET 2023-07-29T00:08:17.454 DEBUG:boto:Path: /hiiqos-47/obj5?versionId=null 2023-07-29T00:08:17.455 DEBUG:boto:Data: 2023-07-29T00:08:17.455 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.455 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:17.456 DEBUG:boto:Port: 8000 2023-07-29T00:08:17.456 DEBUG:boto:Params: {} 2023-07-29T00:08:17.457 DEBUG:boto:Token: None 2023-07-29T00:08:17.457 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj5?versionId=null 2023-07-29T00:08:17.457 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:N7jY1HI4bR//QdMiLbHTnrImUBE= 2023-07-29T00:08:17.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:N7jY1HI4bR//QdMiLbHTnrImUBE=', 'Content-Length': '0'} 2023-07-29T00:08:17.460 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f01cf2fa38c37cd-0064c45871-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.460 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:08:17.461 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:08:17.461 DEBUG:boto:path=/hiiqos-47/?prefix=obj5 2023-07-29T00:08:17.461 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj5 2023-07-29T00:08:17.462 DEBUG:boto:Method: GET 2023-07-29T00:08:17.462 DEBUG:boto:Path: /hiiqos-47/?prefix=obj5 2023-07-29T00:08:17.462 DEBUG:boto:Data: 2023-07-29T00:08:17.463 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.463 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:17.464 DEBUG:boto:Port: 8001 2023-07-29T00:08:17.464 DEBUG:boto:Params: {} 2023-07-29T00:08:17.465 DEBUG:boto:Token: None 2023-07-29T00:08:17.465 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/ 2023-07-29T00:08:17.465 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mEkmS6vwmcrs+wZGyCWeP8Ranz4= 2023-07-29T00:08:17.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mEkmS6vwmcrs+wZGyCWeP8Ranz4=', 'Content-Length': '0'} 2023-07-29T00:08:17.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5af596cdf75ba1f-0064c45871-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.469 DEBUG:boto:b'hiiqos-47obj51000falseobj52023-07-29T00:06:59.221Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:08:17.469 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:08:17.470 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:08:17.470 DEBUG:boto:path=/hiiqos-47/?prefix=obj5 2023-07-29T00:08:17.471 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj5 2023-07-29T00:08:17.471 DEBUG:boto:Method: GET 2023-07-29T00:08:17.471 DEBUG:boto:Path: /hiiqos-47/?prefix=obj5 2023-07-29T00:08:17.472 DEBUG:boto:Data: 2023-07-29T00:08:17.472 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.472 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:17.473 DEBUG:boto:Port: 8000 2023-07-29T00:08:17.473 DEBUG:boto:Params: {} 2023-07-29T00:08:17.473 DEBUG:boto:Token: None 2023-07-29T00:08:17.474 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/ 2023-07-29T00:08:17.474 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mEkmS6vwmcrs+wZGyCWeP8Ranz4= 2023-07-29T00:08:17.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mEkmS6vwmcrs+wZGyCWeP8Ranz4=', 'Content-Length': '0'} 2023-07-29T00:08:17.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f0189eac0058062-0064c45871-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.478 DEBUG:boto:b'hiiqos-47obj51000falseobj52023-07-29T00:06:59.221Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:08:17.478 DEBUG:boto:path=/hiiqos-47/obj5 2023-07-29T00:08:17.479 DEBUG:boto:auth_path=/hiiqos-47/obj5 2023-07-29T00:08:17.479 DEBUG:boto:Method: HEAD 2023-07-29T00:08:17.479 DEBUG:boto:Path: /hiiqos-47/obj5 2023-07-29T00:08:17.480 DEBUG:boto:Data: 2023-07-29T00:08:17.480 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.480 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:17.481 DEBUG:boto:Port: 8001 2023-07-29T00:08:17.481 DEBUG:boto:Params: {} 2023-07-29T00:08:17.481 DEBUG:boto:Token: None 2023-07-29T00:08:17.482 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj5 2023-07-29T00:08:17.482 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P/CLk8QTQ74v6IcWe+o0iYE9Lf0= 2023-07-29T00:08:17.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P/CLk8QTQ74v6IcWe+o0iYE9Lf0=', 'Content-Length': '0'} 2023-07-29T00:08:17.484 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001afe8dca9023f95f-0064c45871-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.485 DEBUG:boto:path=/hiiqos-47/obj5 2023-07-29T00:08:17.485 DEBUG:boto:auth_path=/hiiqos-47/obj5 2023-07-29T00:08:17.485 DEBUG:boto:Method: HEAD 2023-07-29T00:08:17.486 DEBUG:boto:Path: /hiiqos-47/obj5 2023-07-29T00:08:17.486 DEBUG:boto:Data: 2023-07-29T00:08:17.486 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.487 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:17.487 DEBUG:boto:Port: 8000 2023-07-29T00:08:17.487 DEBUG:boto:Params: {} 2023-07-29T00:08:17.488 DEBUG:boto:Token: None 2023-07-29T00:08:17.488 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj5 2023-07-29T00:08:17.489 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P/CLk8QTQ74v6IcWe+o0iYE9Lf0= 2023-07-29T00:08:17.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P/CLk8QTQ74v6IcWe+o0iYE9Lf0=', 'Content-Length': '0'} 2023-07-29T00:08:17.491 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000072d9ee61b7fccc61-0064c45871-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-07-29T00:08:17.492 DEBUG:boto:path=/hiiqos-47/obj5 2023-07-29T00:08:17.492 DEBUG:boto:auth_path=/hiiqos-47/obj5 2023-07-29T00:08:17.493 DEBUG:boto:Method: GET 2023-07-29T00:08:17.493 DEBUG:boto:Path: /hiiqos-47/obj5 2023-07-29T00:08:17.493 DEBUG:boto:Data: 2023-07-29T00:08:17.494 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.494 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:17.494 DEBUG:boto:Port: 8001 2023-07-29T00:08:17.495 DEBUG:boto:Params: {} 2023-07-29T00:08:17.495 DEBUG:boto:Token: None 2023-07-29T00:08:17.496 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj5 2023-07-29T00:08:17.496 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+s3OFPlb+nsG3AbnynRiQ5bz05c= 2023-07-29T00:08:17.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+s3OFPlb+nsG3AbnynRiQ5bz05c=', 'Content-Length': '0'} 2023-07-29T00:08:17.498 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005de62dee6e0bb95a-0064c45871-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.540 DEBUG:boto:path=/hiiqos-47/obj5 2023-07-29T00:08:17.602 DEBUG:boto:auth_path=/hiiqos-47/obj5 2023-07-29T00:08:17.603 DEBUG:boto:Method: GET 2023-07-29T00:08:17.603 DEBUG:boto:Path: /hiiqos-47/obj5 2023-07-29T00:08:17.603 DEBUG:boto:Data: 2023-07-29T00:08:17.604 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.604 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:17.605 DEBUG:boto:Port: 8000 2023-07-29T00:08:17.606 DEBUG:boto:Params: {} 2023-07-29T00:08:17.606 DEBUG:boto:Token: None 2023-07-29T00:08:17.607 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj5 2023-07-29T00:08:17.608 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+s3OFPlb+nsG3AbnynRiQ5bz05c= 2023-07-29T00:08:17.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+s3OFPlb+nsG3AbnynRiQ5bz05c=', 'Content-Length': '0'} 2023-07-29T00:08:17.611 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a0b7d4c9870950ca-0064c45871-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.611 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:08:17.612 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:08:17.612 DEBUG:boto:path=/hiiqos-47/?prefix=obj5 2023-07-29T00:08:17.612 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj5 2023-07-29T00:08:17.613 DEBUG:boto:Method: GET 2023-07-29T00:08:17.613 DEBUG:boto:Path: /hiiqos-47/?prefix=obj5 2023-07-29T00:08:17.613 DEBUG:boto:Data: 2023-07-29T00:08:17.614 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.614 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:17.615 DEBUG:boto:Port: 8001 2023-07-29T00:08:17.615 DEBUG:boto:Params: {} 2023-07-29T00:08:17.616 DEBUG:boto:Token: None 2023-07-29T00:08:17.616 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/ 2023-07-29T00:08:17.617 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mEkmS6vwmcrs+wZGyCWeP8Ranz4= 2023-07-29T00:08:17.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mEkmS6vwmcrs+wZGyCWeP8Ranz4=', 'Content-Length': '0'} 2023-07-29T00:08:17.619 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000123da4f2513a19f5-0064c45871-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.620 DEBUG:boto:b'hiiqos-47obj51000falseobj52023-07-29T00:06:59.221Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:08:17.621 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:08:17.621 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:08:17.621 DEBUG:boto:path=/hiiqos-47/?prefix=obj5 2023-07-29T00:08:17.622 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj5 2023-07-29T00:08:17.622 DEBUG:boto:Method: GET 2023-07-29T00:08:17.622 DEBUG:boto:Path: /hiiqos-47/?prefix=obj5 2023-07-29T00:08:17.623 DEBUG:boto:Data: 2023-07-29T00:08:17.623 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.623 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:17.624 DEBUG:boto:Port: 8000 2023-07-29T00:08:17.624 DEBUG:boto:Params: {} 2023-07-29T00:08:17.625 DEBUG:boto:Token: None 2023-07-29T00:08:17.625 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/ 2023-07-29T00:08:17.626 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mEkmS6vwmcrs+wZGyCWeP8Ranz4= 2023-07-29T00:08:17.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mEkmS6vwmcrs+wZGyCWeP8Ranz4=', 'Content-Length': '0'} 2023-07-29T00:08:17.628 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ede9f5fefe865ee2-0064c45871-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.629 DEBUG:boto:b'hiiqos-47obj51000falseobj52023-07-29T00:06:59.221Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:08:17.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-07-29T00:08:17.630 DEBUG:boto:path=/hiiqos-47/obj6 2023-07-29T00:08:17.630 DEBUG:boto:auth_path=/hiiqos-47/obj6 2023-07-29T00:08:17.631 DEBUG:boto:path=/hiiqos-47/obj6?versionId=null 2023-07-29T00:08:17.631 DEBUG:boto:auth_path=/hiiqos-47/obj6?versionId=null 2023-07-29T00:08:17.631 DEBUG:boto:Method: GET 2023-07-29T00:08:17.632 DEBUG:boto:Path: /hiiqos-47/obj6?versionId=null 2023-07-29T00:08:17.632 DEBUG:boto:Data: 2023-07-29T00:08:17.633 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.633 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:17.633 DEBUG:boto:Port: 8001 2023-07-29T00:08:17.634 DEBUG:boto:Params: {} 2023-07-29T00:08:17.634 DEBUG:boto:Token: None 2023-07-29T00:08:17.634 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj6?versionId=null 2023-07-29T00:08:17.635 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LY0WWNhAnCk+GCUeMIF0i3gB5Ew= 2023-07-29T00:08:17.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LY0WWNhAnCk+GCUeMIF0i3gB5Ew=', 'Content-Length': '0'} 2023-07-29T00:08:17.637 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002034e617e1092d9f-0064c45871-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.638 DEBUG:boto:path=/hiiqos-47/obj6 2023-07-29T00:08:17.638 DEBUG:boto:auth_path=/hiiqos-47/obj6 2023-07-29T00:08:17.639 DEBUG:boto:path=/hiiqos-47/obj6?versionId=null 2023-07-29T00:08:17.639 DEBUG:boto:auth_path=/hiiqos-47/obj6?versionId=null 2023-07-29T00:08:17.639 DEBUG:boto:Method: GET 2023-07-29T00:08:17.640 DEBUG:boto:Path: /hiiqos-47/obj6?versionId=null 2023-07-29T00:08:17.640 DEBUG:boto:Data: 2023-07-29T00:08:17.640 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.641 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:17.641 DEBUG:boto:Port: 8000 2023-07-29T00:08:17.641 DEBUG:boto:Params: {} 2023-07-29T00:08:17.642 DEBUG:boto:Token: None 2023-07-29T00:08:17.642 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj6?versionId=null 2023-07-29T00:08:17.643 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LY0WWNhAnCk+GCUeMIF0i3gB5Ew= 2023-07-29T00:08:17.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LY0WWNhAnCk+GCUeMIF0i3gB5Ew=', 'Content-Length': '0'} 2023-07-29T00:08:17.645 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000008db06b2a3c0225-0064c45871-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.646 DEBUG:boto:path=/hiiqos-47/obj6 2023-07-29T00:08:17.646 DEBUG:boto:auth_path=/hiiqos-47/obj6 2023-07-29T00:08:17.647 DEBUG:boto:path=/hiiqos-47/obj6?versionId=null 2023-07-29T00:08:17.647 DEBUG:boto:auth_path=/hiiqos-47/obj6?versionId=null 2023-07-29T00:08:17.647 DEBUG:boto:Method: HEAD 2023-07-29T00:08:17.648 DEBUG:boto:Path: /hiiqos-47/obj6?versionId=null 2023-07-29T00:08:17.648 DEBUG:boto:Data: 2023-07-29T00:08:17.648 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.649 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:17.649 DEBUG:boto:Port: 8001 2023-07-29T00:08:17.649 DEBUG:boto:Params: {} 2023-07-29T00:08:17.650 DEBUG:boto:Token: None 2023-07-29T00:08:17.650 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj6?versionId=null 2023-07-29T00:08:17.651 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cq8WoyZ8bjvDkA8K3mGXexrM7CE= 2023-07-29T00:08:17.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cq8WoyZ8bjvDkA8K3mGXexrM7CE=', 'Content-Length': '0'} 2023-07-29T00:08:17.653 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009869f2b2c5d4289f-0064c45871-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.653 DEBUG:boto:path=/hiiqos-47/obj6 2023-07-29T00:08:17.654 DEBUG:boto:auth_path=/hiiqos-47/obj6 2023-07-29T00:08:17.654 DEBUG:boto:path=/hiiqos-47/obj6?versionId=null 2023-07-29T00:08:17.654 DEBUG:boto:auth_path=/hiiqos-47/obj6?versionId=null 2023-07-29T00:08:17.655 DEBUG:boto:Method: HEAD 2023-07-29T00:08:17.655 DEBUG:boto:Path: /hiiqos-47/obj6?versionId=null 2023-07-29T00:08:17.655 DEBUG:boto:Data: 2023-07-29T00:08:17.656 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.656 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:17.656 DEBUG:boto:Port: 8000 2023-07-29T00:08:17.657 DEBUG:boto:Params: {} 2023-07-29T00:08:17.657 DEBUG:boto:Token: None 2023-07-29T00:08:17.657 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj6?versionId=null 2023-07-29T00:08:17.658 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cq8WoyZ8bjvDkA8K3mGXexrM7CE= 2023-07-29T00:08:17.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cq8WoyZ8bjvDkA8K3mGXexrM7CE=', 'Content-Length': '0'} 2023-07-29T00:08:17.660 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ef70809fc872af4-0064c45871-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-07-29T00:08:17.661 DEBUG:boto:path=/hiiqos-47/obj6 2023-07-29T00:08:17.661 DEBUG:boto:auth_path=/hiiqos-47/obj6 2023-07-29T00:08:17.662 DEBUG:boto:path=/hiiqos-47/obj6?versionId=null 2023-07-29T00:08:17.662 DEBUG:boto:auth_path=/hiiqos-47/obj6?versionId=null 2023-07-29T00:08:17.663 DEBUG:boto:Method: GET 2023-07-29T00:08:17.663 DEBUG:boto:Path: /hiiqos-47/obj6?versionId=null 2023-07-29T00:08:17.664 DEBUG:boto:Data: 2023-07-29T00:08:17.664 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.664 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:17.665 DEBUG:boto:Port: 8001 2023-07-29T00:08:17.665 DEBUG:boto:Params: {} 2023-07-29T00:08:17.666 DEBUG:boto:Token: None 2023-07-29T00:08:17.666 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj6?versionId=null 2023-07-29T00:08:17.667 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LY0WWNhAnCk+GCUeMIF0i3gB5Ew= 2023-07-29T00:08:17.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LY0WWNhAnCk+GCUeMIF0i3gB5Ew=', 'Content-Length': '0'} 2023-07-29T00:08:17.669 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f84278e64718b6ea-0064c45871-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.670 DEBUG:boto:path=/hiiqos-47/obj6 2023-07-29T00:08:17.670 DEBUG:boto:auth_path=/hiiqos-47/obj6 2023-07-29T00:08:17.670 DEBUG:boto:path=/hiiqos-47/obj6?versionId=null 2023-07-29T00:08:17.671 DEBUG:boto:auth_path=/hiiqos-47/obj6?versionId=null 2023-07-29T00:08:17.671 DEBUG:boto:Method: GET 2023-07-29T00:08:17.672 DEBUG:boto:Path: /hiiqos-47/obj6?versionId=null 2023-07-29T00:08:17.672 DEBUG:boto:Data: 2023-07-29T00:08:17.673 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.673 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:17.673 DEBUG:boto:Port: 8000 2023-07-29T00:08:17.674 DEBUG:boto:Params: {} 2023-07-29T00:08:17.674 DEBUG:boto:Token: None 2023-07-29T00:08:17.675 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj6?versionId=null 2023-07-29T00:08:17.675 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LY0WWNhAnCk+GCUeMIF0i3gB5Ew= 2023-07-29T00:08:17.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LY0WWNhAnCk+GCUeMIF0i3gB5Ew=', 'Content-Length': '0'} 2023-07-29T00:08:17.678 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003074f4dcdb1c93e2-0064c45871-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.720 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:08:17.721 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:08:17.721 DEBUG:boto:path=/hiiqos-47/?prefix=obj6 2023-07-29T00:08:17.722 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj6 2023-07-29T00:08:17.722 DEBUG:boto:Method: GET 2023-07-29T00:08:17.722 DEBUG:boto:Path: /hiiqos-47/?prefix=obj6 2023-07-29T00:08:17.723 DEBUG:boto:Data: 2023-07-29T00:08:17.723 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.723 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:17.724 DEBUG:boto:Port: 8001 2023-07-29T00:08:17.724 DEBUG:boto:Params: {} 2023-07-29T00:08:17.725 DEBUG:boto:Token: None 2023-07-29T00:08:17.725 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/ 2023-07-29T00:08:17.725 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mEkmS6vwmcrs+wZGyCWeP8Ranz4= 2023-07-29T00:08:17.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mEkmS6vwmcrs+wZGyCWeP8Ranz4=', 'Content-Length': '0'} 2023-07-29T00:08:17.728 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a46df5f64ca8b0f5-0064c45871-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.729 DEBUG:boto:b'hiiqos-47obj61000falseobj62023-07-29T00:06:59.242Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:08:17.729 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:08:17.730 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:08:17.730 DEBUG:boto:path=/hiiqos-47/?prefix=obj6 2023-07-29T00:08:17.731 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj6 2023-07-29T00:08:17.731 DEBUG:boto:Method: GET 2023-07-29T00:08:17.731 DEBUG:boto:Path: /hiiqos-47/?prefix=obj6 2023-07-29T00:08:17.732 DEBUG:boto:Data: 2023-07-29T00:08:17.732 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.732 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:17.733 DEBUG:boto:Port: 8000 2023-07-29T00:08:17.733 DEBUG:boto:Params: {} 2023-07-29T00:08:17.733 DEBUG:boto:Token: None 2023-07-29T00:08:17.734 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/ 2023-07-29T00:08:17.734 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mEkmS6vwmcrs+wZGyCWeP8Ranz4= 2023-07-29T00:08:17.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mEkmS6vwmcrs+wZGyCWeP8Ranz4=', 'Content-Length': '0'} 2023-07-29T00:08:17.737 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb232b8b97770e35-0064c45871-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.738 DEBUG:boto:b'hiiqos-47obj61000falseobj62023-07-29T00:06:59.242Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:08:17.738 DEBUG:boto:path=/hiiqos-47/obj6 2023-07-29T00:08:17.739 DEBUG:boto:auth_path=/hiiqos-47/obj6 2023-07-29T00:08:17.739 DEBUG:boto:Method: HEAD 2023-07-29T00:08:17.740 DEBUG:boto:Path: /hiiqos-47/obj6 2023-07-29T00:08:17.740 DEBUG:boto:Data: 2023-07-29T00:08:17.740 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.741 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:17.741 DEBUG:boto:Port: 8001 2023-07-29T00:08:17.741 DEBUG:boto:Params: {} 2023-07-29T00:08:17.742 DEBUG:boto:Token: None 2023-07-29T00:08:17.742 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj6 2023-07-29T00:08:17.742 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yA1ITKx/3uS9UD9ymGcWz3txFAE= 2023-07-29T00:08:17.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yA1ITKx/3uS9UD9ymGcWz3txFAE=', 'Content-Length': '0'} 2023-07-29T00:08:17.744 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008979ae44015b8436-0064c45871-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.745 DEBUG:boto:path=/hiiqos-47/obj6 2023-07-29T00:08:17.745 DEBUG:boto:auth_path=/hiiqos-47/obj6 2023-07-29T00:08:17.746 DEBUG:boto:Method: HEAD 2023-07-29T00:08:17.746 DEBUG:boto:Path: /hiiqos-47/obj6 2023-07-29T00:08:17.746 DEBUG:boto:Data: 2023-07-29T00:08:17.747 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.747 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:17.747 DEBUG:boto:Port: 8000 2023-07-29T00:08:17.748 DEBUG:boto:Params: {} 2023-07-29T00:08:17.748 DEBUG:boto:Token: None 2023-07-29T00:08:17.749 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj6 2023-07-29T00:08:17.749 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yA1ITKx/3uS9UD9ymGcWz3txFAE= 2023-07-29T00:08:17.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yA1ITKx/3uS9UD9ymGcWz3txFAE=', 'Content-Length': '0'} 2023-07-29T00:08:17.751 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d65fc4747939d38e-0064c45871-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-07-29T00:08:17.752 DEBUG:boto:path=/hiiqos-47/obj6 2023-07-29T00:08:17.753 DEBUG:boto:auth_path=/hiiqos-47/obj6 2023-07-29T00:08:17.753 DEBUG:boto:Method: GET 2023-07-29T00:08:17.754 DEBUG:boto:Path: /hiiqos-47/obj6 2023-07-29T00:08:17.754 DEBUG:boto:Data: 2023-07-29T00:08:17.754 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.755 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:17.755 DEBUG:boto:Port: 8001 2023-07-29T00:08:17.755 DEBUG:boto:Params: {} 2023-07-29T00:08:17.756 DEBUG:boto:Token: None 2023-07-29T00:08:17.756 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj6 2023-07-29T00:08:17.757 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Se1LovwsALBtHnj30ZHkgbHZpN0= 2023-07-29T00:08:17.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Se1LovwsALBtHnj30ZHkgbHZpN0=', 'Content-Length': '0'} 2023-07-29T00:08:17.759 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000039c419038b7cc40f-0064c45871-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.800 DEBUG:boto:path=/hiiqos-47/obj6 2023-07-29T00:08:17.801 DEBUG:boto:auth_path=/hiiqos-47/obj6 2023-07-29T00:08:17.802 DEBUG:boto:Method: GET 2023-07-29T00:08:17.802 DEBUG:boto:Path: /hiiqos-47/obj6 2023-07-29T00:08:17.802 DEBUG:boto:Data: 2023-07-29T00:08:17.803 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.803 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:17.804 DEBUG:boto:Port: 8000 2023-07-29T00:08:17.804 DEBUG:boto:Params: {} 2023-07-29T00:08:17.805 DEBUG:boto:Token: None 2023-07-29T00:08:17.805 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj6 2023-07-29T00:08:17.806 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Se1LovwsALBtHnj30ZHkgbHZpN0= 2023-07-29T00:08:17.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Se1LovwsALBtHnj30ZHkgbHZpN0=', 'Content-Length': '0'} 2023-07-29T00:08:17.808 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f3639401cf899a5e-0064c45871-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.809 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:08:17.809 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:08:17.810 DEBUG:boto:path=/hiiqos-47/?prefix=obj6 2023-07-29T00:08:17.810 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj6 2023-07-29T00:08:17.811 DEBUG:boto:Method: GET 2023-07-29T00:08:17.811 DEBUG:boto:Path: /hiiqos-47/?prefix=obj6 2023-07-29T00:08:17.811 DEBUG:boto:Data: 2023-07-29T00:08:17.812 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.812 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:17.813 DEBUG:boto:Port: 8001 2023-07-29T00:08:17.813 DEBUG:boto:Params: {} 2023-07-29T00:08:17.813 DEBUG:boto:Token: None 2023-07-29T00:08:17.814 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/ 2023-07-29T00:08:17.814 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mEkmS6vwmcrs+wZGyCWeP8Ranz4= 2023-07-29T00:08:17.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mEkmS6vwmcrs+wZGyCWeP8Ranz4=', 'Content-Length': '0'} 2023-07-29T00:08:17.817 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002f2d8331ed4dc6a-0064c45871-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.818 DEBUG:boto:b'hiiqos-47obj61000falseobj62023-07-29T00:06:59.242Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:08:17.819 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:08:17.819 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:08:17.820 DEBUG:boto:path=/hiiqos-47/?prefix=obj6 2023-07-29T00:08:17.820 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj6 2023-07-29T00:08:17.820 DEBUG:boto:Method: GET 2023-07-29T00:08:17.821 DEBUG:boto:Path: /hiiqos-47/?prefix=obj6 2023-07-29T00:08:17.821 DEBUG:boto:Data: 2023-07-29T00:08:17.822 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.822 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:17.822 DEBUG:boto:Port: 8000 2023-07-29T00:08:17.823 DEBUG:boto:Params: {} 2023-07-29T00:08:17.823 DEBUG:boto:Token: None 2023-07-29T00:08:17.824 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/ 2023-07-29T00:08:17.824 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mEkmS6vwmcrs+wZGyCWeP8Ranz4= 2023-07-29T00:08:17.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mEkmS6vwmcrs+wZGyCWeP8Ranz4=', 'Content-Length': '0'} 2023-07-29T00:08:17.827 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ced985e51a6b5ba-0064c45871-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.868 DEBUG:boto:b'hiiqos-47obj61000falseobj62023-07-29T00:06:59.242Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:08:17.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-07-29T00:08:17.870 DEBUG:boto:path=/hiiqos-47/obj7 2023-07-29T00:08:17.870 DEBUG:boto:auth_path=/hiiqos-47/obj7 2023-07-29T00:08:17.871 DEBUG:boto:path=/hiiqos-47/obj7?versionId=null 2023-07-29T00:08:17.871 DEBUG:boto:auth_path=/hiiqos-47/obj7?versionId=null 2023-07-29T00:08:17.871 DEBUG:boto:Method: GET 2023-07-29T00:08:17.872 DEBUG:boto:Path: /hiiqos-47/obj7?versionId=null 2023-07-29T00:08:17.872 DEBUG:boto:Data: 2023-07-29T00:08:17.873 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.873 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:17.873 DEBUG:boto:Port: 8001 2023-07-29T00:08:17.874 DEBUG:boto:Params: {} 2023-07-29T00:08:17.874 DEBUG:boto:Token: None 2023-07-29T00:08:17.875 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj7?versionId=null 2023-07-29T00:08:17.875 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TXxHnzHVpWL4W9zit40bpXNhxi8= 2023-07-29T00:08:17.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TXxHnzHVpWL4W9zit40bpXNhxi8=', 'Content-Length': '0'} 2023-07-29T00:08:17.878 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001fba3418747bfac6-0064c45871-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.879 DEBUG:boto:path=/hiiqos-47/obj7 2023-07-29T00:08:17.879 DEBUG:boto:auth_path=/hiiqos-47/obj7 2023-07-29T00:08:17.879 DEBUG:boto:path=/hiiqos-47/obj7?versionId=null 2023-07-29T00:08:17.880 DEBUG:boto:auth_path=/hiiqos-47/obj7?versionId=null 2023-07-29T00:08:17.880 DEBUG:boto:Method: GET 2023-07-29T00:08:17.881 DEBUG:boto:Path: /hiiqos-47/obj7?versionId=null 2023-07-29T00:08:17.881 DEBUG:boto:Data: 2023-07-29T00:08:17.882 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.882 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:17.882 DEBUG:boto:Port: 8000 2023-07-29T00:08:17.883 DEBUG:boto:Params: {} 2023-07-29T00:08:17.883 DEBUG:boto:Token: None 2023-07-29T00:08:17.884 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj7?versionId=null 2023-07-29T00:08:17.884 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TXxHnzHVpWL4W9zit40bpXNhxi8= 2023-07-29T00:08:17.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TXxHnzHVpWL4W9zit40bpXNhxi8=', 'Content-Length': '0'} 2023-07-29T00:08:17.887 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004e3655c1e99866e6-0064c45871-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.888 DEBUG:boto:path=/hiiqos-47/obj7 2023-07-29T00:08:17.888 DEBUG:boto:auth_path=/hiiqos-47/obj7 2023-07-29T00:08:17.889 DEBUG:boto:path=/hiiqos-47/obj7?versionId=null 2023-07-29T00:08:17.889 DEBUG:boto:auth_path=/hiiqos-47/obj7?versionId=null 2023-07-29T00:08:17.889 DEBUG:boto:Method: HEAD 2023-07-29T00:08:17.890 DEBUG:boto:Path: /hiiqos-47/obj7?versionId=null 2023-07-29T00:08:17.890 DEBUG:boto:Data: 2023-07-29T00:08:17.890 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.891 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:17.891 DEBUG:boto:Port: 8001 2023-07-29T00:08:17.892 DEBUG:boto:Params: {} 2023-07-29T00:08:17.892 DEBUG:boto:Token: None 2023-07-29T00:08:17.892 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj7?versionId=null 2023-07-29T00:08:17.893 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IGYPgQ31Dwz1YUNIRnC8S7uAAy0= 2023-07-29T00:08:17.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IGYPgQ31Dwz1YUNIRnC8S7uAAy0=', 'Content-Length': '0'} 2023-07-29T00:08:17.895 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002431f4bc86addc69-0064c45871-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.896 DEBUG:boto:path=/hiiqos-47/obj7 2023-07-29T00:08:17.897 DEBUG:boto:auth_path=/hiiqos-47/obj7 2023-07-29T00:08:17.897 DEBUG:boto:path=/hiiqos-47/obj7?versionId=null 2023-07-29T00:08:17.898 DEBUG:boto:auth_path=/hiiqos-47/obj7?versionId=null 2023-07-29T00:08:17.899 DEBUG:boto:Method: HEAD 2023-07-29T00:08:17.899 DEBUG:boto:Path: /hiiqos-47/obj7?versionId=null 2023-07-29T00:08:17.900 DEBUG:boto:Data: 2023-07-29T00:08:17.901 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.902 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:17.903 DEBUG:boto:Port: 8000 2023-07-29T00:08:17.903 DEBUG:boto:Params: {} 2023-07-29T00:08:17.904 DEBUG:boto:Token: None 2023-07-29T00:08:17.905 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj7?versionId=null 2023-07-29T00:08:17.906 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IGYPgQ31Dwz1YUNIRnC8S7uAAy0= 2023-07-29T00:08:17.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IGYPgQ31Dwz1YUNIRnC8S7uAAy0=', 'Content-Length': '0'} 2023-07-29T00:08:17.908 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b8938dd25c3c80dd-0064c45871-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-07-29T00:08:17.909 DEBUG:boto:path=/hiiqos-47/obj7 2023-07-29T00:08:17.910 DEBUG:boto:auth_path=/hiiqos-47/obj7 2023-07-29T00:08:17.910 DEBUG:boto:path=/hiiqos-47/obj7?versionId=null 2023-07-29T00:08:17.910 DEBUG:boto:auth_path=/hiiqos-47/obj7?versionId=null 2023-07-29T00:08:17.911 DEBUG:boto:Method: GET 2023-07-29T00:08:17.911 DEBUG:boto:Path: /hiiqos-47/obj7?versionId=null 2023-07-29T00:08:17.912 DEBUG:boto:Data: 2023-07-29T00:08:17.912 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.913 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:17.913 DEBUG:boto:Port: 8001 2023-07-29T00:08:17.913 DEBUG:boto:Params: {} 2023-07-29T00:08:17.914 DEBUG:boto:Token: None 2023-07-29T00:08:17.914 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj7?versionId=null 2023-07-29T00:08:17.915 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TXxHnzHVpWL4W9zit40bpXNhxi8= 2023-07-29T00:08:17.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TXxHnzHVpWL4W9zit40bpXNhxi8=', 'Content-Length': '0'} 2023-07-29T00:08:17.918 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000088c12d8c622867ee-0064c45871-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.918 DEBUG:boto:path=/hiiqos-47/obj7 2023-07-29T00:08:17.919 DEBUG:boto:auth_path=/hiiqos-47/obj7 2023-07-29T00:08:17.919 DEBUG:boto:path=/hiiqos-47/obj7?versionId=null 2023-07-29T00:08:17.919 DEBUG:boto:auth_path=/hiiqos-47/obj7?versionId=null 2023-07-29T00:08:17.920 DEBUG:boto:Method: GET 2023-07-29T00:08:17.920 DEBUG:boto:Path: /hiiqos-47/obj7?versionId=null 2023-07-29T00:08:17.921 DEBUG:boto:Data: 2023-07-29T00:08:17.921 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.921 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:17.922 DEBUG:boto:Port: 8000 2023-07-29T00:08:17.922 DEBUG:boto:Params: {} 2023-07-29T00:08:17.923 DEBUG:boto:Token: None 2023-07-29T00:08:17.923 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj7?versionId=null 2023-07-29T00:08:17.924 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TXxHnzHVpWL4W9zit40bpXNhxi8= 2023-07-29T00:08:17.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TXxHnzHVpWL4W9zit40bpXNhxi8=', 'Content-Length': '0'} 2023-07-29T00:08:17.926 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000068883ef56f9b86ee-0064c45871-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.927 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:08:17.927 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:08:17.928 DEBUG:boto:path=/hiiqos-47/?prefix=obj7 2023-07-29T00:08:17.928 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj7 2023-07-29T00:08:17.929 DEBUG:boto:Method: GET 2023-07-29T00:08:17.929 DEBUG:boto:Path: /hiiqos-47/?prefix=obj7 2023-07-29T00:08:17.929 DEBUG:boto:Data: 2023-07-29T00:08:17.930 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.930 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:17.931 DEBUG:boto:Port: 8001 2023-07-29T00:08:17.931 DEBUG:boto:Params: {} 2023-07-29T00:08:17.932 DEBUG:boto:Token: None 2023-07-29T00:08:17.932 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/ 2023-07-29T00:08:17.933 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mEkmS6vwmcrs+wZGyCWeP8Ranz4= 2023-07-29T00:08:17.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mEkmS6vwmcrs+wZGyCWeP8Ranz4=', 'Content-Length': '0'} 2023-07-29T00:08:17.936 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048b6a93a7d672b97-0064c45871-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.936 DEBUG:boto:b'hiiqos-47obj71000falseobj72023-07-29T00:06:59.259Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:08:17.937 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:08:17.938 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:08:17.938 DEBUG:boto:path=/hiiqos-47/?prefix=obj7 2023-07-29T00:08:17.938 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj7 2023-07-29T00:08:17.939 DEBUG:boto:Method: GET 2023-07-29T00:08:17.939 DEBUG:boto:Path: /hiiqos-47/?prefix=obj7 2023-07-29T00:08:17.940 DEBUG:boto:Data: 2023-07-29T00:08:17.940 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.941 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:17.941 DEBUG:boto:Port: 8000 2023-07-29T00:08:17.942 DEBUG:boto:Params: {} 2023-07-29T00:08:17.942 DEBUG:boto:Token: None 2023-07-29T00:08:17.943 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/ 2023-07-29T00:08:17.943 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mEkmS6vwmcrs+wZGyCWeP8Ranz4= 2023-07-29T00:08:17.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mEkmS6vwmcrs+wZGyCWeP8Ranz4=', 'Content-Length': '0'} 2023-07-29T00:08:17.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c65916c9ec6d5055-0064c45871-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.947 DEBUG:boto:b'hiiqos-47obj71000falseobj72023-07-29T00:06:59.259Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:08:17.948 DEBUG:boto:path=/hiiqos-47/obj7 2023-07-29T00:08:17.948 DEBUG:boto:auth_path=/hiiqos-47/obj7 2023-07-29T00:08:17.949 DEBUG:boto:Method: HEAD 2023-07-29T00:08:17.949 DEBUG:boto:Path: /hiiqos-47/obj7 2023-07-29T00:08:17.949 DEBUG:boto:Data: 2023-07-29T00:08:17.950 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.950 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:17.950 DEBUG:boto:Port: 8001 2023-07-29T00:08:17.951 DEBUG:boto:Params: {} 2023-07-29T00:08:17.951 DEBUG:boto:Token: None 2023-07-29T00:08:17.951 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj7 2023-07-29T00:08:17.952 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zPMrc5IfyszD3/a1DWM/rgKzhLg= 2023-07-29T00:08:17.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zPMrc5IfyszD3/a1DWM/rgKzhLg=', 'Content-Length': '0'} 2023-07-29T00:08:17.955 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008b13c801d312ef21-0064c45871-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.955 DEBUG:boto:path=/hiiqos-47/obj7 2023-07-29T00:08:17.956 DEBUG:boto:auth_path=/hiiqos-47/obj7 2023-07-29T00:08:17.956 DEBUG:boto:Method: HEAD 2023-07-29T00:08:17.956 DEBUG:boto:Path: /hiiqos-47/obj7 2023-07-29T00:08:17.957 DEBUG:boto:Data: 2023-07-29T00:08:17.957 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.957 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:17.958 DEBUG:boto:Port: 8000 2023-07-29T00:08:17.958 DEBUG:boto:Params: {} 2023-07-29T00:08:17.959 DEBUG:boto:Token: None 2023-07-29T00:08:17.959 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj7 2023-07-29T00:08:17.959 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zPMrc5IfyszD3/a1DWM/rgKzhLg= 2023-07-29T00:08:17.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zPMrc5IfyszD3/a1DWM/rgKzhLg=', 'Content-Length': '0'} 2023-07-29T00:08:17.962 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b5174fef46a770ac-0064c45871-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-07-29T00:08:17.963 DEBUG:boto:path=/hiiqos-47/obj7 2023-07-29T00:08:17.963 DEBUG:boto:auth_path=/hiiqos-47/obj7 2023-07-29T00:08:17.963 DEBUG:boto:Method: GET 2023-07-29T00:08:17.964 DEBUG:boto:Path: /hiiqos-47/obj7 2023-07-29T00:08:17.964 DEBUG:boto:Data: 2023-07-29T00:08:17.965 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.965 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:17.966 DEBUG:boto:Port: 8001 2023-07-29T00:08:17.966 DEBUG:boto:Params: {} 2023-07-29T00:08:17.966 DEBUG:boto:Token: None 2023-07-29T00:08:17.967 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj7 2023-07-29T00:08:17.967 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7tuol5U2N1TCKm0cdrK+ihyQsvU= 2023-07-29T00:08:17.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7tuol5U2N1TCKm0cdrK+ihyQsvU=', 'Content-Length': '0'} 2023-07-29T00:08:17.970 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e50bac512e96e56-0064c45871-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.970 DEBUG:boto:path=/hiiqos-47/obj7 2023-07-29T00:08:17.971 DEBUG:boto:auth_path=/hiiqos-47/obj7 2023-07-29T00:08:17.971 DEBUG:boto:Method: GET 2023-07-29T00:08:17.971 DEBUG:boto:Path: /hiiqos-47/obj7 2023-07-29T00:08:17.972 DEBUG:boto:Data: 2023-07-29T00:08:17.972 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.973 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:17.973 DEBUG:boto:Port: 8000 2023-07-29T00:08:17.973 DEBUG:boto:Params: {} 2023-07-29T00:08:17.974 DEBUG:boto:Token: None 2023-07-29T00:08:17.974 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/obj7 2023-07-29T00:08:17.975 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7tuol5U2N1TCKm0cdrK+ihyQsvU= 2023-07-29T00:08:17.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7tuol5U2N1TCKm0cdrK+ihyQsvU=', 'Content-Length': '0'} 2023-07-29T00:08:17.977 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a406353097298c1-0064c45871-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.978 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:08:17.979 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:08:17.979 DEBUG:boto:path=/hiiqos-47/?prefix=obj7 2023-07-29T00:08:17.979 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj7 2023-07-29T00:08:17.980 DEBUG:boto:Method: GET 2023-07-29T00:08:17.980 DEBUG:boto:Path: /hiiqos-47/?prefix=obj7 2023-07-29T00:08:17.980 DEBUG:boto:Data: 2023-07-29T00:08:17.981 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.981 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:17.981 DEBUG:boto:Port: 8001 2023-07-29T00:08:17.982 DEBUG:boto:Params: {} 2023-07-29T00:08:17.982 DEBUG:boto:Token: None 2023-07-29T00:08:17.983 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/ 2023-07-29T00:08:17.983 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mEkmS6vwmcrs+wZGyCWeP8Ranz4= 2023-07-29T00:08:17.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mEkmS6vwmcrs+wZGyCWeP8Ranz4=', 'Content-Length': '0'} 2023-07-29T00:08:17.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a7e44c7e5b2e8b2-0064c45871-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.987 DEBUG:boto:b'hiiqos-47obj71000falseobj72023-07-29T00:06:59.259Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:08:17.988 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:08:17.988 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:08:17.989 DEBUG:boto:path=/hiiqos-47/?prefix=obj7 2023-07-29T00:08:17.989 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj7 2023-07-29T00:08:17.989 DEBUG:boto:Method: GET 2023-07-29T00:08:17.990 DEBUG:boto:Path: /hiiqos-47/?prefix=obj7 2023-07-29T00:08:17.990 DEBUG:boto:Data: 2023-07-29T00:08:17.991 DEBUG:boto:Headers: {} 2023-07-29T00:08:17.991 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:17.991 DEBUG:boto:Port: 8000 2023-07-29T00:08:17.992 DEBUG:boto:Params: {} 2023-07-29T00:08:17.992 DEBUG:boto:Token: None 2023-07-29T00:08:17.993 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:17 GMT /hiiqos-47/ 2023-07-29T00:08:17.993 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mEkmS6vwmcrs+wZGyCWeP8Ranz4= 2023-07-29T00:08:17.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mEkmS6vwmcrs+wZGyCWeP8Ranz4=', 'Content-Length': '0'} 2023-07-29T00:08:17.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c266914aae13221-0064c45871-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:17.997 DEBUG:boto:b'hiiqos-47obj71000falseobj72023-07-29T00:06:59.259Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:08:17.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-07-29T00:08:17.998 DEBUG:boto:path=/hiiqos-47/obj8 2023-07-29T00:08:17.998 DEBUG:boto:auth_path=/hiiqos-47/obj8 2023-07-29T00:08:17.999 DEBUG:boto:path=/hiiqos-47/obj8?versionId=null 2023-07-29T00:08:17.999 DEBUG:boto:auth_path=/hiiqos-47/obj8?versionId=null 2023-07-29T00:08:17.999 DEBUG:boto:Method: GET 2023-07-29T00:08:18.000 DEBUG:boto:Path: /hiiqos-47/obj8?versionId=null 2023-07-29T00:08:18.000 DEBUG:boto:Data: 2023-07-29T00:08:18.000 DEBUG:boto:Headers: {} 2023-07-29T00:08:18.001 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:18.001 DEBUG:boto:Port: 8001 2023-07-29T00:08:18.002 DEBUG:boto:Params: {} 2023-07-29T00:08:18.002 DEBUG:boto:Token: None 2023-07-29T00:08:18.002 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:18 GMT /hiiqos-47/obj8?versionId=null 2023-07-29T00:08:18.003 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+gUgRHN5Xo90sHkIwCYgo4IIo7k= 2023-07-29T00:08:18.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+gUgRHN5Xo90sHkIwCYgo4IIo7k=', 'Content-Length': '0'} 2023-07-29T00:08:18.005 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d3781f3298b3457f-0064c45871-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:18.006 DEBUG:boto:path=/hiiqos-47/obj8 2023-07-29T00:08:18.006 DEBUG:boto:auth_path=/hiiqos-47/obj8 2023-07-29T00:08:18.007 DEBUG:boto:path=/hiiqos-47/obj8?versionId=null 2023-07-29T00:08:18.008 DEBUG:boto:auth_path=/hiiqos-47/obj8?versionId=null 2023-07-29T00:08:18.009 DEBUG:boto:Method: GET 2023-07-29T00:08:18.010 DEBUG:boto:Path: /hiiqos-47/obj8?versionId=null 2023-07-29T00:08:18.011 DEBUG:boto:Data: 2023-07-29T00:08:18.012 DEBUG:boto:Headers: {} 2023-07-29T00:08:18.013 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:18.014 DEBUG:boto:Port: 8000 2023-07-29T00:08:18.014 DEBUG:boto:Params: {} 2023-07-29T00:08:18.015 DEBUG:boto:Token: None 2023-07-29T00:08:18.016 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:18 GMT /hiiqos-47/obj8?versionId=null 2023-07-29T00:08:18.017 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+gUgRHN5Xo90sHkIwCYgo4IIo7k= 2023-07-29T00:08:18.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+gUgRHN5Xo90sHkIwCYgo4IIo7k=', 'Content-Length': '0'} 2023-07-29T00:08:18.020 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005a9dc6326e0885a4-0064c45871-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:18.021 DEBUG:boto:path=/hiiqos-47/obj8 2023-07-29T00:08:18.022 DEBUG:boto:auth_path=/hiiqos-47/obj8 2023-07-29T00:08:18.022 DEBUG:boto:path=/hiiqos-47/obj8?versionId=null 2023-07-29T00:08:18.022 DEBUG:boto:auth_path=/hiiqos-47/obj8?versionId=null 2023-07-29T00:08:18.023 DEBUG:boto:Method: HEAD 2023-07-29T00:08:18.023 DEBUG:boto:Path: /hiiqos-47/obj8?versionId=null 2023-07-29T00:08:18.023 DEBUG:boto:Data: 2023-07-29T00:08:18.024 DEBUG:boto:Headers: {} 2023-07-29T00:08:18.024 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:18.025 DEBUG:boto:Port: 8001 2023-07-29T00:08:18.025 DEBUG:boto:Params: {} 2023-07-29T00:08:18.026 DEBUG:boto:Token: None 2023-07-29T00:08:18.026 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:18 GMT /hiiqos-47/obj8?versionId=null 2023-07-29T00:08:18.027 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S2oNryH/WPxu90gvoK+p8BIwBew= 2023-07-29T00:08:18.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S2oNryH/WPxu90gvoK+p8BIwBew=', 'Content-Length': '0'} 2023-07-29T00:08:18.029 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000540dfea5f55beacc-0064c45872-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:18.030 DEBUG:boto:path=/hiiqos-47/obj8 2023-07-29T00:08:18.031 DEBUG:boto:auth_path=/hiiqos-47/obj8 2023-07-29T00:08:18.032 DEBUG:boto:path=/hiiqos-47/obj8?versionId=null 2023-07-29T00:08:18.033 DEBUG:boto:auth_path=/hiiqos-47/obj8?versionId=null 2023-07-29T00:08:18.034 DEBUG:boto:Method: HEAD 2023-07-29T00:08:18.035 DEBUG:boto:Path: /hiiqos-47/obj8?versionId=null 2023-07-29T00:08:18.035 DEBUG:boto:Data: 2023-07-29T00:08:18.036 DEBUG:boto:Headers: {} 2023-07-29T00:08:18.037 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:18.038 DEBUG:boto:Port: 8000 2023-07-29T00:08:18.038 DEBUG:boto:Params: {} 2023-07-29T00:08:18.039 DEBUG:boto:Token: None 2023-07-29T00:08:18.040 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:18 GMT /hiiqos-47/obj8?versionId=null 2023-07-29T00:08:18.040 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S2oNryH/WPxu90gvoK+p8BIwBew= 2023-07-29T00:08:18.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S2oNryH/WPxu90gvoK+p8BIwBew=', 'Content-Length': '0'} 2023-07-29T00:08:18.043 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a8df4d8a153cc24c-0064c45872-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:18.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-07-29T00:08:18.044 DEBUG:boto:path=/hiiqos-47/obj8 2023-07-29T00:08:18.044 DEBUG:boto:auth_path=/hiiqos-47/obj8 2023-07-29T00:08:18.045 DEBUG:boto:path=/hiiqos-47/obj8?versionId=null 2023-07-29T00:08:18.045 DEBUG:boto:auth_path=/hiiqos-47/obj8?versionId=null 2023-07-29T00:08:18.046 DEBUG:boto:Method: GET 2023-07-29T00:08:18.046 DEBUG:boto:Path: /hiiqos-47/obj8?versionId=null 2023-07-29T00:08:18.046 DEBUG:boto:Data: 2023-07-29T00:08:18.047 DEBUG:boto:Headers: {} 2023-07-29T00:08:18.047 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:18.047 DEBUG:boto:Port: 8001 2023-07-29T00:08:18.048 DEBUG:boto:Params: {} 2023-07-29T00:08:18.048 DEBUG:boto:Token: None 2023-07-29T00:08:18.049 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:18 GMT /hiiqos-47/obj8?versionId=null 2023-07-29T00:08:18.049 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+gUgRHN5Xo90sHkIwCYgo4IIo7k= 2023-07-29T00:08:18.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+gUgRHN5Xo90sHkIwCYgo4IIo7k=', 'Content-Length': '0'} 2023-07-29T00:08:18.052 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003469dcf152aa2b5e-0064c45872-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:18.052 DEBUG:boto:path=/hiiqos-47/obj8 2023-07-29T00:08:18.053 DEBUG:boto:auth_path=/hiiqos-47/obj8 2023-07-29T00:08:18.053 DEBUG:boto:path=/hiiqos-47/obj8?versionId=null 2023-07-29T00:08:18.054 DEBUG:boto:auth_path=/hiiqos-47/obj8?versionId=null 2023-07-29T00:08:18.054 DEBUG:boto:Method: GET 2023-07-29T00:08:18.054 DEBUG:boto:Path: /hiiqos-47/obj8?versionId=null 2023-07-29T00:08:18.055 DEBUG:boto:Data: 2023-07-29T00:08:18.055 DEBUG:boto:Headers: {} 2023-07-29T00:08:18.056 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:18.056 DEBUG:boto:Port: 8000 2023-07-29T00:08:18.056 DEBUG:boto:Params: {} 2023-07-29T00:08:18.057 DEBUG:boto:Token: None 2023-07-29T00:08:18.057 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:18 GMT /hiiqos-47/obj8?versionId=null 2023-07-29T00:08:18.058 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+gUgRHN5Xo90sHkIwCYgo4IIo7k= 2023-07-29T00:08:18.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+gUgRHN5Xo90sHkIwCYgo4IIo7k=', 'Content-Length': '0'} 2023-07-29T00:08:18.060 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c0091535a87dd76e-0064c45872-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:18.100 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:08:18.101 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:08:18.101 DEBUG:boto:path=/hiiqos-47/?prefix=obj8 2023-07-29T00:08:18.102 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj8 2023-07-29T00:08:18.102 DEBUG:boto:Method: GET 2023-07-29T00:08:18.102 DEBUG:boto:Path: /hiiqos-47/?prefix=obj8 2023-07-29T00:08:18.103 DEBUG:boto:Data: 2023-07-29T00:08:18.103 DEBUG:boto:Headers: {} 2023-07-29T00:08:18.103 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:18.104 DEBUG:boto:Port: 8001 2023-07-29T00:08:18.104 DEBUG:boto:Params: {} 2023-07-29T00:08:18.105 DEBUG:boto:Token: None 2023-07-29T00:08:18.105 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:18 GMT /hiiqos-47/ 2023-07-29T00:08:18.106 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PzFGvtNojyIzxFz9JH3WeZfyIh4= 2023-07-29T00:08:18.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PzFGvtNojyIzxFz9JH3WeZfyIh4=', 'Content-Length': '0'} 2023-07-29T00:08:18.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c0a43095b459ba8-0064c45872-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:18.109 DEBUG:boto:b'hiiqos-47obj81000falseobj82023-07-29T00:06:59.319Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:08:18.110 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:08:18.111 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:08:18.111 DEBUG:boto:path=/hiiqos-47/?prefix=obj8 2023-07-29T00:08:18.112 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj8 2023-07-29T00:08:18.113 DEBUG:boto:Method: GET 2023-07-29T00:08:18.114 DEBUG:boto:Path: /hiiqos-47/?prefix=obj8 2023-07-29T00:08:18.114 DEBUG:boto:Data: 2023-07-29T00:08:18.115 DEBUG:boto:Headers: {} 2023-07-29T00:08:18.116 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:18.116 DEBUG:boto:Port: 8000 2023-07-29T00:08:18.117 DEBUG:boto:Params: {} 2023-07-29T00:08:18.118 DEBUG:boto:Token: None 2023-07-29T00:08:18.119 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:18 GMT /hiiqos-47/ 2023-07-29T00:08:18.120 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PzFGvtNojyIzxFz9JH3WeZfyIh4= 2023-07-29T00:08:18.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PzFGvtNojyIzxFz9JH3WeZfyIh4=', 'Content-Length': '0'} 2023-07-29T00:08:18.123 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c1acb37b5bd7933-0064c45872-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:18.124 DEBUG:boto:b'hiiqos-47obj81000falseobj82023-07-29T00:06:59.319Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:08:18.125 DEBUG:boto:path=/hiiqos-47/obj8 2023-07-29T00:08:18.126 DEBUG:boto:auth_path=/hiiqos-47/obj8 2023-07-29T00:08:18.127 DEBUG:boto:Method: HEAD 2023-07-29T00:08:18.127 DEBUG:boto:Path: /hiiqos-47/obj8 2023-07-29T00:08:18.128 DEBUG:boto:Data: 2023-07-29T00:08:18.129 DEBUG:boto:Headers: {} 2023-07-29T00:08:18.130 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:18.130 DEBUG:boto:Port: 8001 2023-07-29T00:08:18.131 DEBUG:boto:Params: {} 2023-07-29T00:08:18.132 DEBUG:boto:Token: None 2023-07-29T00:08:18.132 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:18 GMT /hiiqos-47/obj8 2023-07-29T00:08:18.133 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2HBe+9Ywu8bXZbWOvN9vE5pbANE= 2023-07-29T00:08:18.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2HBe+9Ywu8bXZbWOvN9vE5pbANE=', 'Content-Length': '0'} 2023-07-29T00:08:18.136 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008e66d12bc6c45854-0064c45872-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:18.137 DEBUG:boto:path=/hiiqos-47/obj8 2023-07-29T00:08:18.138 DEBUG:boto:auth_path=/hiiqos-47/obj8 2023-07-29T00:08:18.138 DEBUG:boto:Method: HEAD 2023-07-29T00:08:18.138 DEBUG:boto:Path: /hiiqos-47/obj8 2023-07-29T00:08:18.139 DEBUG:boto:Data: 2023-07-29T00:08:18.139 DEBUG:boto:Headers: {} 2023-07-29T00:08:18.140 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:18.140 DEBUG:boto:Port: 8000 2023-07-29T00:08:18.141 DEBUG:boto:Params: {} 2023-07-29T00:08:18.141 DEBUG:boto:Token: None 2023-07-29T00:08:18.141 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:18 GMT /hiiqos-47/obj8 2023-07-29T00:08:18.142 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2HBe+9Ywu8bXZbWOvN9vE5pbANE= 2023-07-29T00:08:18.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2HBe+9Ywu8bXZbWOvN9vE5pbANE=', 'Content-Length': '0'} 2023-07-29T00:08:18.144 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006faa6c02fc8a7ca6-0064c45872-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:18.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-07-29T00:08:18.145 DEBUG:boto:path=/hiiqos-47/obj8 2023-07-29T00:08:18.146 DEBUG:boto:auth_path=/hiiqos-47/obj8 2023-07-29T00:08:18.146 DEBUG:boto:Method: GET 2023-07-29T00:08:18.146 DEBUG:boto:Path: /hiiqos-47/obj8 2023-07-29T00:08:18.147 DEBUG:boto:Data: 2023-07-29T00:08:18.147 DEBUG:boto:Headers: {} 2023-07-29T00:08:18.147 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:18.148 DEBUG:boto:Port: 8001 2023-07-29T00:08:18.148 DEBUG:boto:Params: {} 2023-07-29T00:08:18.149 DEBUG:boto:Token: None 2023-07-29T00:08:18.149 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:18 GMT /hiiqos-47/obj8 2023-07-29T00:08:18.150 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8LRG4YwiK4N+xxoqvYxoe9XGS5g= 2023-07-29T00:08:18.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8LRG4YwiK4N+xxoqvYxoe9XGS5g=', 'Content-Length': '0'} 2023-07-29T00:08:18.152 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000031916233394731e1-0064c45872-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:18.192 DEBUG:boto:path=/hiiqos-47/obj8 2023-07-29T00:08:18.193 DEBUG:boto:auth_path=/hiiqos-47/obj8 2023-07-29T00:08:18.193 DEBUG:boto:Method: GET 2023-07-29T00:08:18.194 DEBUG:boto:Path: /hiiqos-47/obj8 2023-07-29T00:08:18.194 DEBUG:boto:Data: 2023-07-29T00:08:18.195 DEBUG:boto:Headers: {} 2023-07-29T00:08:18.195 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:18.195 DEBUG:boto:Port: 8000 2023-07-29T00:08:18.196 DEBUG:boto:Params: {} 2023-07-29T00:08:18.196 DEBUG:boto:Token: None 2023-07-29T00:08:18.197 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:18 GMT /hiiqos-47/obj8 2023-07-29T00:08:18.197 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8LRG4YwiK4N+xxoqvYxoe9XGS5g= 2023-07-29T00:08:18.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8LRG4YwiK4N+xxoqvYxoe9XGS5g=', 'Content-Length': '0'} 2023-07-29T00:08:18.199 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:06:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-47:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000053435e348aaf74ad-0064c45872-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:18.200 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:08:18.201 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:08:18.201 DEBUG:boto:path=/hiiqos-47/?prefix=obj8 2023-07-29T00:08:18.201 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj8 2023-07-29T00:08:18.202 DEBUG:boto:Method: GET 2023-07-29T00:08:18.202 DEBUG:boto:Path: /hiiqos-47/?prefix=obj8 2023-07-29T00:08:18.202 DEBUG:boto:Data: 2023-07-29T00:08:18.203 DEBUG:boto:Headers: {} 2023-07-29T00:08:18.203 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:18.204 DEBUG:boto:Port: 8001 2023-07-29T00:08:18.204 DEBUG:boto:Params: {} 2023-07-29T00:08:18.204 DEBUG:boto:Token: None 2023-07-29T00:08:18.205 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:18 GMT /hiiqos-47/ 2023-07-29T00:08:18.205 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PzFGvtNojyIzxFz9JH3WeZfyIh4= 2023-07-29T00:08:18.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PzFGvtNojyIzxFz9JH3WeZfyIh4=', 'Content-Length': '0'} 2023-07-29T00:08:18.208 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e8c895f5eec4726-0064c45872-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:18.209 DEBUG:boto:b'hiiqos-47obj81000falseobj82023-07-29T00:06:59.319Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:08:18.209 DEBUG:boto:path=/hiiqos-47/ 2023-07-29T00:08:18.210 DEBUG:boto:auth_path=/hiiqos-47/ 2023-07-29T00:08:18.210 DEBUG:boto:path=/hiiqos-47/?prefix=obj8 2023-07-29T00:08:18.210 DEBUG:boto:auth_path=/hiiqos-47/?prefix=obj8 2023-07-29T00:08:18.211 DEBUG:boto:Method: GET 2023-07-29T00:08:18.211 DEBUG:boto:Path: /hiiqos-47/?prefix=obj8 2023-07-29T00:08:18.212 DEBUG:boto:Data: 2023-07-29T00:08:18.212 DEBUG:boto:Headers: {} 2023-07-29T00:08:18.213 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:18.213 DEBUG:boto:Port: 8000 2023-07-29T00:08:18.213 DEBUG:boto:Params: {} 2023-07-29T00:08:18.214 DEBUG:boto:Token: None 2023-07-29T00:08:18.214 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:18 GMT /hiiqos-47/ 2023-07-29T00:08:18.215 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PzFGvtNojyIzxFz9JH3WeZfyIh4= 2023-07-29T00:08:18.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:18 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PzFGvtNojyIzxFz9JH3WeZfyIh4=', 'Content-Length': '0'} 2023-07-29T00:08:18.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052710e18aee5f7bd-0064c45872-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:18 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:18.218 DEBUG:boto:b'hiiqos-47obj81000falseobj82023-07-29T00:06:59.319Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:08:18.219 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-47 zones={a2, a1} 2023-07-29T00:08:18.219 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-07-29T00:08:18.219 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:08:18.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:08:18.220 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:08:18.269 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:08:18.269 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:08:18.317 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T00:08:18.317 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:08:18.318 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000007269", 2023-07-29T00:08:18.318 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:07:14.724915Z" 2023-07-29T00:08:18.318 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:18.318 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:08:18.318 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000005577", 2023-07-29T00:08:18.318 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:06:59.071617Z" 2023-07-29T00:08:18.319 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:18.319 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:08:18.319 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000004316", 2023-07-29T00:08:18.319 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:07:14.581970Z" 2023-07-29T00:08:18.319 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:18.319 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:08:18.320 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000002625", 2023-07-29T00:08:18.320 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:08:15.591176Z" 2023-07-29T00:08:18.320 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:08:18.320 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T00:08:18.321 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000007269', 1: '00000000000000000000:00000000000000005577', 2: '00000000000000000000:00000000000000004316', 3: '00000000000000000000:00000000000000002625'} 2023-07-29T00:08:18.322 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-07-29T00:08:18.322 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:08:18.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:08:18.323 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:08:18.371 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:08:18.371 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:08:18.413 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:08:18.414 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:08:18.414 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:08:18.414 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:08:18.414 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:08:18.414 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 16749483048653174838 2023-07-29T00:08:18.415 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:18.415 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:08:18.415 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:08:18.415 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:08:18.415 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:08:18.415 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:08:18.416 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000007269", 2023-07-29T00:08:18.416 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:08:18.416 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:08:18.416 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:08:18.416 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:07:14.724915Z" 2023-07-29T00:08:18.416 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:08:18.417 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:18.417 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:08:18.417 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:08:18.417 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:08:18.417 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:08:18.417 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000005577", 2023-07-29T00:08:18.418 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:08:18.418 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:08:18.418 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:08:18.418 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:06:59.071617Z" 2023-07-29T00:08:18.418 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:08:18.418 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:18.419 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:08:18.419 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:08:18.419 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:08:18.419 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:08:18.419 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000004316", 2023-07-29T00:08:18.419 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:08:18.420 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:08:18.420 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:08:18.420 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:07:14.581970Z" 2023-07-29T00:08:18.420 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:08:18.420 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:18.420 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:08:18.420 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:08:18.421 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:08:18.421 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:08:18.421 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000002484", 2023-07-29T00:08:18.421 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:08:18.421 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:08:18.421 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:08:18.422 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:06:59.205412Z" 2023-07-29T00:08:18.422 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:08:18.422 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:08:18.422 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:08:18.422 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:18.422 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:08:18.423 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 0, 2023-07-29T00:08:18.423 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 0 2023-07-29T00:08:18.423 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:18.423 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:08:18Z" 2023-07-29T00:08:18.423 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:08:18.424 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16749483048653174838 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007269", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:07:14.724915Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005577", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:06:59.071617Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004316", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:07:14.581970Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002484", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:06:59.205412Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-07-29T00:08:18Z" } 2023-07-29T00:08:18.425 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007269', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:07:14.724915Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005577', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:06:59.071617Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004316', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:07:14.581970Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002484', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:06:59.205412Z'}}] 2023-07-29T00:08:18.425 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000007269', 1: '00000000000000000000:00000000000000005577', 2: '00000000000000000000:00000000000000004316', 3: '00000000000000000000:00000000000000002625'} 2023-07-29T00:08:18.425 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000007269', 1: '00000000000000000000:00000000000000005577', 2: '00000000000000000000:00000000000000004316', 3: '00000000000000000000:00000000000000002484'} 2023-07-29T00:08:18.426 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=3 master=00000000000000000000:00000000000000002625 target=00000000000000000000:00000000000000002484 2023-07-29T00:08:23.427 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:08:23.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:08:23.429 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:08:23.483 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:08:23.484 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:08:23.530 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:08:23.531 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:08:23.531 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:08:23.531 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:08:23.531 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:08:23.532 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 16749483048653174838 2023-07-29T00:08:23.532 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:23.533 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:08:23.533 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:08:23.533 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:08:23.534 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:08:23.534 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:08:23.534 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000007269", 2023-07-29T00:08:23.535 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:08:23.535 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:08:23.535 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:08:23.536 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:07:14.724915Z" 2023-07-29T00:08:23.536 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:08:23.536 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:23.536 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:08:23.537 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:08:23.537 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:08:23.537 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:08:23.538 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000005577", 2023-07-29T00:08:23.538 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:08:23.538 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:08:23.539 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:08:23.539 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:06:59.071617Z" 2023-07-29T00:08:23.539 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:08:23.540 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:23.540 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:08:23.540 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:08:23.541 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:08:23.541 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:08:23.541 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000004316", 2023-07-29T00:08:23.541 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:08:23.542 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:08:23.542 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:08:23.542 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:07:14.581970Z" 2023-07-29T00:08:23.543 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:08:23.543 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:23.544 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:08:23.544 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:08:23.544 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:08:23.545 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:08:23.545 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000002484", 2023-07-29T00:08:23.545 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:08:23.546 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:08:23.546 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:08:23.546 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:06:59.205412Z" 2023-07-29T00:08:23.546 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:08:23.546 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:08:23.547 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:08:23.547 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:23.547 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:08:23.547 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 0, 2023-07-29T00:08:23.547 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 0 2023-07-29T00:08:23.547 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:23.548 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:08:23Z" 2023-07-29T00:08:23.548 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:08:23.549 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16749483048653174838 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007269", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:07:14.724915Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005577", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:06:59.071617Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004316", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:07:14.581970Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002484", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:06:59.205412Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-07-29T00:08:23Z" } 2023-07-29T00:08:23.549 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007269', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:07:14.724915Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005577', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:06:59.071617Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004316', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:07:14.581970Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002484', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:06:59.205412Z'}}] 2023-07-29T00:08:23.550 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000007269', 1: '00000000000000000000:00000000000000005577', 2: '00000000000000000000:00000000000000004316', 3: '00000000000000000000:00000000000000002625'} 2023-07-29T00:08:23.550 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000007269', 1: '00000000000000000000:00000000000000005577', 2: '00000000000000000000:00000000000000004316', 3: '00000000000000000000:00000000000000002484'} 2023-07-29T00:08:23.550 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=3 master=00000000000000000000:00000000000000002625 target=00000000000000000000:00000000000000002484 2023-07-29T00:08:28.552 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:08:28.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:08:28.553 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:08:28.607 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:08:28.608 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:08:28.655 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:08:28.655 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:08:28.655 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:08:28.655 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:08:28.656 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:08:28.656 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 16749483048653174838 2023-07-29T00:08:28.656 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:28.656 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:08:28.657 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:08:28.657 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:08:28.657 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:08:28.657 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:08:28.657 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000007269", 2023-07-29T00:08:28.657 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:08:28.658 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:08:28.658 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:08:28.658 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:07:14.724915Z" 2023-07-29T00:08:28.658 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:08:28.658 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:28.658 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:08:28.659 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:08:28.659 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:08:28.659 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:08:28.659 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000005577", 2023-07-29T00:08:28.659 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:08:28.659 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:08:28.659 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:08:28.660 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:06:59.071617Z" 2023-07-29T00:08:28.660 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:08:28.660 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:28.660 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:08:28.660 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:08:28.660 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:08:28.660 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:08:28.661 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000004316", 2023-07-29T00:08:28.661 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:08:28.661 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:08:28.661 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:08:28.661 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:07:14.581970Z" 2023-07-29T00:08:28.661 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:08:28.662 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:28.662 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:08:28.662 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:08:28.662 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:08:28.662 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:08:28.662 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000002484", 2023-07-29T00:08:28.663 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:08:28.663 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:08:28.663 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:08:28.663 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:06:59.205412Z" 2023-07-29T00:08:28.663 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:08:28.663 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:08:28.663 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:08:28.664 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:28.664 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:08:28.664 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 0, 2023-07-29T00:08:28.664 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 0 2023-07-29T00:08:28.664 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:28.664 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:08:28Z" 2023-07-29T00:08:28.665 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:08:28.665 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16749483048653174838 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007269", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:07:14.724915Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005577", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:06:59.071617Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004316", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:07:14.581970Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002484", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:06:59.205412Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-07-29T00:08:28Z" } 2023-07-29T00:08:28.666 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007269', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:07:14.724915Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005577', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:06:59.071617Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004316', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:07:14.581970Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002484', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:06:59.205412Z'}}] 2023-07-29T00:08:28.667 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000007269', 1: '00000000000000000000:00000000000000005577', 2: '00000000000000000000:00000000000000004316', 3: '00000000000000000000:00000000000000002625'} 2023-07-29T00:08:28.667 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000007269', 1: '00000000000000000000:00000000000000005577', 2: '00000000000000000000:00000000000000004316', 3: '00000000000000000000:00000000000000002484'} 2023-07-29T00:08:28.667 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=3 master=00000000000000000000:00000000000000002625 target=00000000000000000000:00000000000000002484 2023-07-29T00:08:33.669 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:08:33.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:08:33.670 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:08:33.724 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:08:33.724 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:08:33.770 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:08:33.770 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:08:33.770 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:08:33.770 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:08:33.770 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:08:33.771 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 16749483048653174838 2023-07-29T00:08:33.771 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:33.771 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:08:33.771 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:08:33.771 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:08:33.771 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:08:33.771 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:08:33.772 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000007269", 2023-07-29T00:08:33.772 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:08:33.772 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:08:33.772 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:08:33.772 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:07:14.724915Z" 2023-07-29T00:08:33.772 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:08:33.773 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:33.773 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:08:33.773 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:08:33.773 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:08:33.773 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:08:33.773 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000005577", 2023-07-29T00:08:33.773 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:08:33.774 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:08:33.774 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:08:33.774 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:06:59.071617Z" 2023-07-29T00:08:33.774 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:08:33.774 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:33.774 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:08:33.774 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:08:33.775 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:08:33.775 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:08:33.775 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000004316", 2023-07-29T00:08:33.775 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:08:33.775 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:08:33.775 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:08:33.775 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:07:14.581970Z" 2023-07-29T00:08:33.776 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:08:33.776 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:33.776 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:08:33.776 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:08:33.776 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:08:33.776 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:08:33.776 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000002625", 2023-07-29T00:08:33.777 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:08:33.777 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:08:33.777 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:08:33.777 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:08:15.591176Z" 2023-07-29T00:08:33.777 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:08:33.777 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:08:33.777 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:08:33.778 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:33.778 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:08:33.778 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 0, 2023-07-29T00:08:33.778 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 0 2023-07-29T00:08:33.778 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:33.778 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:08:33Z" 2023-07-29T00:08:33.778 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:08:33.779 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16749483048653174838 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007269", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:07:14.724915Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005577", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:06:59.071617Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004316", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:07:14.581970Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002625", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:08:15.591176Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-07-29T00:08:33Z" } 2023-07-29T00:08:33.780 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007269', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:07:14.724915Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005577', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:06:59.071617Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004316', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:07:14.581970Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002625', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:08:15.591176Z'}}] 2023-07-29T00:08:33.780 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000007269', 1: '00000000000000000000:00000000000000005577', 2: '00000000000000000000:00000000000000004316', 3: '00000000000000000000:00000000000000002625'} 2023-07-29T00:08:33.781 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000007269', 1: '00000000000000000000:00000000000000005577', 2: '00000000000000000000:00000000000000004316', 3: '00000000000000000000:00000000000000002625'} 2023-07-29T00:08:33.781 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-07-29T00:08:33.781 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-07-29T00:08:33.781 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:08:33.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:08:33.782 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:08:33.828 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:08:33.828 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:08:33.874 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T00:08:33.875 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:08:33.875 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000006141", 2023-07-29T00:08:33.875 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:07:12.472931Z" 2023-07-29T00:08:33.875 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:33.875 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:08:33.875 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000002902", 2023-07-29T00:08:33.876 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:05:12.146079Z" 2023-07-29T00:08:33.876 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:33.876 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:08:33.876 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000002620", 2023-07-29T00:08:33.876 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:07:12.545283Z" 2023-07-29T00:08:33.876 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:33.877 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:08:33.877 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000001497", 2023-07-29T00:08:33.877 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:07:12.472619Z" 2023-07-29T00:08:33.877 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:08:33.878 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T00:08:33.878 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000006141', 1: '00000000000000000000:00000000000000002902', 2: '00000000000000000000:00000000000000002620', 3: '00000000000000000000:00000000000000001497'} 2023-07-29T00:08:33.879 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-07-29T00:08:33.879 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:08:33.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:08:33.880 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:08:33.927 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:08:33.928 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:08:33.969 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:08:33.969 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:08:33.969 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:08:33.969 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:08:33.970 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:08:33.970 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 17882949198279251152 2023-07-29T00:08:33.970 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:33.970 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:08:33.970 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:08:33.970 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:08:33.971 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:08:33.971 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:08:33.971 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000006141", 2023-07-29T00:08:33.971 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:08:33.971 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:08:33.971 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:08:33.971 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:07:12.472931Z" 2023-07-29T00:08:33.972 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:08:33.972 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:33.972 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:08:33.972 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:08:33.972 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:08:33.972 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:08:33.972 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000002902", 2023-07-29T00:08:33.973 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:08:33.973 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:08:33.973 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:08:33.973 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:05:12.146079Z" 2023-07-29T00:08:33.973 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:08:33.973 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:33.974 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:08:33.974 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:08:33.974 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:08:33.974 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:08:33.974 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000002620", 2023-07-29T00:08:33.974 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:08:33.974 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:08:33.975 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:08:33.975 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:07:12.545283Z" 2023-07-29T00:08:33.975 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:08:33.975 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:33.975 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:08:33.975 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:08:33.975 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:08:33.976 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:08:33.976 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000001497", 2023-07-29T00:08:33.976 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:08:33.976 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:08:33.976 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:08:33.976 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:07:12.472619Z" 2023-07-29T00:08:33.976 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:08:33.977 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:08:33.977 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:08:33.977 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:33.977 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:08:33.977 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 0, 2023-07-29T00:08:33.977 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 0 2023-07-29T00:08:33.978 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:33.978 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:08:33Z" 2023-07-29T00:08:33.978 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:08:33.979 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17882949198279251152 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006141", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:07:12.472931Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002902", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:05:12.146079Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002620", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:07:12.545283Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001497", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:07:12.472619Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-07-29T00:08:33Z" } 2023-07-29T00:08:33.979 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006141', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:07:12.472931Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002902', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:05:12.146079Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002620', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:07:12.545283Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001497', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:07:12.472619Z'}}] 2023-07-29T00:08:33.980 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000006141', 1: '00000000000000000000:00000000000000002902', 2: '00000000000000000000:00000000000000002620', 3: '00000000000000000000:00000000000000001497'} 2023-07-29T00:08:33.980 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000006141', 1: '00000000000000000000:00000000000000002902', 2: '00000000000000000000:00000000000000002620', 3: '00000000000000000000:00000000000000001497'} 2023-07-29T00:08:33.980 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-07-29T00:08:33.981 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... ok 2023-07-29T00:08:33.983 INFO:rgw_multi.tests:create bucket zone=a1 name=hiiqos-48 2023-07-29T00:08:33.984 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:08:33.985 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:08:33.985 DEBUG:boto:Method: PUT 2023-07-29T00:08:33.986 DEBUG:boto:Path: /hiiqos-48/ 2023-07-29T00:08:33.987 DEBUG:boto:Data: 2023-07-29T00:08:33.988 DEBUG:boto:Headers: {} 2023-07-29T00:08:33.988 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:33.989 DEBUG:boto:Port: 8000 2023-07-29T00:08:33.989 DEBUG:boto:Params: {} 2023-07-29T00:08:33.990 DEBUG:boto:Token: None 2023-07-29T00:08:33.991 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 00:08:33 GMT /hiiqos-48/ 2023-07-29T00:08:33.991 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GHnloIoOVkp5Hb4Kvfd3bXxj4eE= 2023-07-29T00:08:33.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GHnloIoOVkp5Hb4Kvfd3bXxj4eE=', 'Content-Length': '0'} 2023-07-29T00:08:34.005 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000052be567c514810f5-0064c45881-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:08:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:34.006 INFO:rgw_multi.tests:create bucket zone=a2 name=hiiqos-49 2023-07-29T00:08:34.006 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:08:34.006 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:08:34.007 DEBUG:boto:Method: PUT 2023-07-29T00:08:34.007 DEBUG:boto:Path: /hiiqos-49/ 2023-07-29T00:08:34.007 DEBUG:boto:Data: 2023-07-29T00:08:34.008 DEBUG:boto:Headers: {} 2023-07-29T00:08:34.008 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:34.008 DEBUG:boto:Port: 8001 2023-07-29T00:08:34.009 DEBUG:boto:Params: {} 2023-07-29T00:08:34.009 DEBUG:boto:Token: None 2023-07-29T00:08:34.010 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 00:08:34 GMT /hiiqos-49/ 2023-07-29T00:08:34.010 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vnk+iBGNNApdZ2+JrxwQIFSpeBI= 2023-07-29T00:08:34.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vnk+iBGNNApdZ2+JrxwQIFSpeBI=', 'Content-Length': '0'} 2023-07-29T00:08:34.033 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000012180f5a0acfd239-0064c45881-1663-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:34.034 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:08:34.034 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:08:34.034 DEBUG:boto:Method: HEAD 2023-07-29T00:08:34.035 DEBUG:boto:Path: /hiiqos-48/ 2023-07-29T00:08:34.035 DEBUG:boto:Data: 2023-07-29T00:08:34.035 DEBUG:boto:Headers: {} 2023-07-29T00:08:34.036 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:34.036 DEBUG:boto:Port: 8000 2023-07-29T00:08:34.036 DEBUG:boto:Params: {} 2023-07-29T00:08:34.037 DEBUG:boto:Token: None 2023-07-29T00:08:34.037 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:34 GMT /hiiqos-48/ 2023-07-29T00:08:34.038 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rtduGNuhhf8KPjxuHBDDc3f0WA8= 2023-07-29T00:08:34.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rtduGNuhhf8KPjxuHBDDc3f0WA8=', 'Content-Length': '0'} 2023-07-29T00:08:34.040 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a89aea8b02cfa750-0064c45882-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:34.041 DEBUG:boto:path=/hiiqos-48/obj1 2023-07-29T00:08:34.041 DEBUG:boto:auth_path=/hiiqos-48/obj1 2023-07-29T00:08:34.041 DEBUG:boto:Method: PUT 2023-07-29T00:08:34.042 DEBUG:boto:Path: /hiiqos-48/obj1 2023-07-29T00:08:34.042 DEBUG:boto:Data: 2023-07-29T00:08:34.042 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T00:08:34.043 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:34.043 DEBUG:boto:Port: 8000 2023-07-29T00:08:34.043 DEBUG:boto:Params: {} 2023-07-29T00:08:34.044 DEBUG:boto:Token: None 2023-07-29T00:08:34.044 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 00:08:34 GMT /hiiqos-48/obj1 2023-07-29T00:08:34.045 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Xt2PrMaCDKX4IWcG8AY7o+zxVI0= 2023-07-29T00:08:34.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:08:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Xt2PrMaCDKX4IWcG8AY7o+zxVI0='} 2023-07-29T00:08:34.092 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f5c08a43906c5039-0064c45882-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:34.093 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:08:34.093 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:08:34.094 DEBUG:boto:Method: HEAD 2023-07-29T00:08:34.094 DEBUG:boto:Path: /hiiqos-48/ 2023-07-29T00:08:34.094 DEBUG:boto:Data: 2023-07-29T00:08:34.095 DEBUG:boto:Headers: {} 2023-07-29T00:08:34.095 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:34.095 DEBUG:boto:Port: 8000 2023-07-29T00:08:34.096 DEBUG:boto:Params: {} 2023-07-29T00:08:34.096 DEBUG:boto:Token: None 2023-07-29T00:08:34.096 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:34 GMT /hiiqos-48/ 2023-07-29T00:08:34.097 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rtduGNuhhf8KPjxuHBDDc3f0WA8= 2023-07-29T00:08:34.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rtduGNuhhf8KPjxuHBDDc3f0WA8=', 'Content-Length': '0'} 2023-07-29T00:08:34.100 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ba618fd84cd22a73-0064c45882-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:34.100 DEBUG:boto:path=/hiiqos-48/obj2 2023-07-29T00:08:34.101 DEBUG:boto:auth_path=/hiiqos-48/obj2 2023-07-29T00:08:34.101 DEBUG:boto:Method: PUT 2023-07-29T00:08:34.101 DEBUG:boto:Path: /hiiqos-48/obj2 2023-07-29T00:08:34.102 DEBUG:boto:Data: 2023-07-29T00:08:34.102 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T00:08:34.102 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:34.103 DEBUG:boto:Port: 8000 2023-07-29T00:08:34.103 DEBUG:boto:Params: {} 2023-07-29T00:08:34.103 DEBUG:boto:Token: None 2023-07-29T00:08:34.104 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 00:08:34 GMT /hiiqos-48/obj2 2023-07-29T00:08:34.104 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fCZ5phKo6Wscx0FkpDck+0RxmFA= 2023-07-29T00:08:34.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:08:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fCZ5phKo6Wscx0FkpDck+0RxmFA='} 2023-07-29T00:08:34.109 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000037764bbd90f49976-0064c45882-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:34.110 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:08:34.110 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:08:34.110 DEBUG:boto:Method: HEAD 2023-07-29T00:08:34.111 DEBUG:boto:Path: /hiiqos-48/ 2023-07-29T00:08:34.111 DEBUG:boto:Data: 2023-07-29T00:08:34.111 DEBUG:boto:Headers: {} 2023-07-29T00:08:34.112 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:34.112 DEBUG:boto:Port: 8000 2023-07-29T00:08:34.112 DEBUG:boto:Params: {} 2023-07-29T00:08:34.113 DEBUG:boto:Token: None 2023-07-29T00:08:34.113 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:34 GMT /hiiqos-48/ 2023-07-29T00:08:34.113 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rtduGNuhhf8KPjxuHBDDc3f0WA8= 2023-07-29T00:08:34.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rtduGNuhhf8KPjxuHBDDc3f0WA8=', 'Content-Length': '0'} 2023-07-29T00:08:34.116 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000edfa1ee2c81edcdb-0064c45882-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:34.116 DEBUG:boto:path=/hiiqos-48/obj3 2023-07-29T00:08:34.117 DEBUG:boto:auth_path=/hiiqos-48/obj3 2023-07-29T00:08:34.117 DEBUG:boto:Method: PUT 2023-07-29T00:08:34.117 DEBUG:boto:Path: /hiiqos-48/obj3 2023-07-29T00:08:34.118 DEBUG:boto:Data: 2023-07-29T00:08:34.118 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T00:08:34.118 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:34.119 DEBUG:boto:Port: 8000 2023-07-29T00:08:34.119 DEBUG:boto:Params: {} 2023-07-29T00:08:34.119 DEBUG:boto:Token: None 2023-07-29T00:08:34.120 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 00:08:34 GMT /hiiqos-48/obj3 2023-07-29T00:08:34.120 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QkCB1RHNi6DvDfkQHqmzt2XSZdI= 2023-07-29T00:08:34.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:08:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QkCB1RHNi6DvDfkQHqmzt2XSZdI='} 2023-07-29T00:08:34.125 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000095fc491b74591d2-0064c45882-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:34.125 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:08:34.126 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:08:34.126 DEBUG:boto:Method: HEAD 2023-07-29T00:08:34.126 DEBUG:boto:Path: /hiiqos-48/ 2023-07-29T00:08:34.127 DEBUG:boto:Data: 2023-07-29T00:08:34.127 DEBUG:boto:Headers: {} 2023-07-29T00:08:34.127 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:34.128 DEBUG:boto:Port: 8000 2023-07-29T00:08:34.128 DEBUG:boto:Params: {} 2023-07-29T00:08:34.128 DEBUG:boto:Token: None 2023-07-29T00:08:34.129 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:34 GMT /hiiqos-48/ 2023-07-29T00:08:34.129 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rtduGNuhhf8KPjxuHBDDc3f0WA8= 2023-07-29T00:08:34.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rtduGNuhhf8KPjxuHBDDc3f0WA8=', 'Content-Length': '0'} 2023-07-29T00:08:34.131 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b17b42af60c3d878-0064c45882-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:34.132 DEBUG:boto:path=/hiiqos-48/obj4 2023-07-29T00:08:34.132 DEBUG:boto:auth_path=/hiiqos-48/obj4 2023-07-29T00:08:34.133 DEBUG:boto:Method: PUT 2023-07-29T00:08:34.133 DEBUG:boto:Path: /hiiqos-48/obj4 2023-07-29T00:08:34.133 DEBUG:boto:Data: 2023-07-29T00:08:34.134 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T00:08:34.134 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:34.134 DEBUG:boto:Port: 8000 2023-07-29T00:08:34.135 DEBUG:boto:Params: {} 2023-07-29T00:08:34.135 DEBUG:boto:Token: None 2023-07-29T00:08:34.135 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 00:08:34 GMT /hiiqos-48/obj4 2023-07-29T00:08:34.136 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6bKip/55I16PfEn2XMyFTi6vO3Y= 2023-07-29T00:08:34.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:08:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6bKip/55I16PfEn2XMyFTi6vO3Y='} 2023-07-29T00:08:34.180 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001bea70988962f3dd-0064c45882-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:34.182 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:08:34.183 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:08:34.183 DEBUG:boto:Method: HEAD 2023-07-29T00:08:34.184 DEBUG:boto:Path: /hiiqos-49/ 2023-07-29T00:08:34.185 DEBUG:boto:Data: 2023-07-29T00:08:34.186 DEBUG:boto:Headers: {} 2023-07-29T00:08:34.186 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:34.187 DEBUG:boto:Port: 8001 2023-07-29T00:08:34.188 DEBUG:boto:Params: {} 2023-07-29T00:08:34.188 DEBUG:boto:Token: None 2023-07-29T00:08:34.189 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:34 GMT /hiiqos-49/ 2023-07-29T00:08:34.190 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2dB2L/CwRyB4/vX86SaDetnq9hc= 2023-07-29T00:08:34.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2dB2L/CwRyB4/vX86SaDetnq9hc=', 'Content-Length': '0'} 2023-07-29T00:08:34.193 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007c4a0dead2890ea9-0064c45882-1663-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:34.193 DEBUG:boto:path=/hiiqos-49/obj1 2023-07-29T00:08:34.194 DEBUG:boto:auth_path=/hiiqos-49/obj1 2023-07-29T00:08:34.194 DEBUG:boto:Method: PUT 2023-07-29T00:08:34.194 DEBUG:boto:Path: /hiiqos-49/obj1 2023-07-29T00:08:34.195 DEBUG:boto:Data: 2023-07-29T00:08:34.195 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T00:08:34.195 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:34.196 DEBUG:boto:Port: 8001 2023-07-29T00:08:34.196 DEBUG:boto:Params: {} 2023-07-29T00:08:34.197 DEBUG:boto:Token: None 2023-07-29T00:08:34.197 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 00:08:34 GMT /hiiqos-49/obj1 2023-07-29T00:08:34.197 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xI5ziZK85jYhdTUs+iPu6Y/XJns= 2023-07-29T00:08:34.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:08:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xI5ziZK85jYhdTUs+iPu6Y/XJns='} 2023-07-29T00:08:34.203 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000858f776285923e83-0064c45882-1663-a2'), ('Date', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:34.204 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:08:34.204 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:08:34.205 DEBUG:boto:Method: HEAD 2023-07-29T00:08:34.205 DEBUG:boto:Path: /hiiqos-49/ 2023-07-29T00:08:34.205 DEBUG:boto:Data: 2023-07-29T00:08:34.206 DEBUG:boto:Headers: {} 2023-07-29T00:08:34.206 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:34.206 DEBUG:boto:Port: 8001 2023-07-29T00:08:34.207 DEBUG:boto:Params: {} 2023-07-29T00:08:34.207 DEBUG:boto:Token: None 2023-07-29T00:08:34.207 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:34 GMT /hiiqos-49/ 2023-07-29T00:08:34.208 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2dB2L/CwRyB4/vX86SaDetnq9hc= 2023-07-29T00:08:34.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2dB2L/CwRyB4/vX86SaDetnq9hc=', 'Content-Length': '0'} 2023-07-29T00:08:34.211 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000075c1ddb03e7d278d-0064c45882-1663-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:34.212 DEBUG:boto:path=/hiiqos-49/obj2 2023-07-29T00:08:34.213 DEBUG:boto:auth_path=/hiiqos-49/obj2 2023-07-29T00:08:34.214 DEBUG:boto:Method: PUT 2023-07-29T00:08:34.214 DEBUG:boto:Path: /hiiqos-49/obj2 2023-07-29T00:08:34.215 DEBUG:boto:Data: 2023-07-29T00:08:34.215 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T00:08:34.215 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:34.216 DEBUG:boto:Port: 8001 2023-07-29T00:08:34.216 DEBUG:boto:Params: {} 2023-07-29T00:08:34.216 DEBUG:boto:Token: None 2023-07-29T00:08:34.217 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 00:08:34 GMT /hiiqos-49/obj2 2023-07-29T00:08:34.217 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:d6JFEQM4TlhwaPR7smwGTI2UdMI= 2023-07-29T00:08:34.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:08:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:d6JFEQM4TlhwaPR7smwGTI2UdMI='} 2023-07-29T00:08:34.260 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000048e31d87c9c43106-0064c45882-1663-a2'), ('Date', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:34.261 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:08:34.261 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:08:34.262 DEBUG:boto:Method: HEAD 2023-07-29T00:08:34.262 DEBUG:boto:Path: /hiiqos-49/ 2023-07-29T00:08:34.262 DEBUG:boto:Data: 2023-07-29T00:08:34.263 DEBUG:boto:Headers: {} 2023-07-29T00:08:34.263 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:34.263 DEBUG:boto:Port: 8001 2023-07-29T00:08:34.264 DEBUG:boto:Params: {} 2023-07-29T00:08:34.264 DEBUG:boto:Token: None 2023-07-29T00:08:34.265 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:34 GMT /hiiqos-49/ 2023-07-29T00:08:34.265 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2dB2L/CwRyB4/vX86SaDetnq9hc= 2023-07-29T00:08:34.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2dB2L/CwRyB4/vX86SaDetnq9hc=', 'Content-Length': '0'} 2023-07-29T00:08:34.268 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c6837e092b8465b6-0064c45882-1663-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:34.268 DEBUG:boto:path=/hiiqos-49/obj3 2023-07-29T00:08:34.269 DEBUG:boto:auth_path=/hiiqos-49/obj3 2023-07-29T00:08:34.269 DEBUG:boto:Method: PUT 2023-07-29T00:08:34.269 DEBUG:boto:Path: /hiiqos-49/obj3 2023-07-29T00:08:34.270 DEBUG:boto:Data: 2023-07-29T00:08:34.270 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T00:08:34.270 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:34.271 DEBUG:boto:Port: 8001 2023-07-29T00:08:34.271 DEBUG:boto:Params: {} 2023-07-29T00:08:34.271 DEBUG:boto:Token: None 2023-07-29T00:08:34.272 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 00:08:34 GMT /hiiqos-49/obj3 2023-07-29T00:08:34.272 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+E1zWgidtV+NzXTc98REdsE+4mY= 2023-07-29T00:08:34.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:08:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+E1zWgidtV+NzXTc98REdsE+4mY='} 2023-07-29T00:08:34.278 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000066e1d25d4bee17a7-0064c45882-1663-a2'), ('Date', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:34.278 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:08:34.278 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:08:34.279 DEBUG:boto:Method: HEAD 2023-07-29T00:08:34.279 DEBUG:boto:Path: /hiiqos-49/ 2023-07-29T00:08:34.279 DEBUG:boto:Data: 2023-07-29T00:08:34.280 DEBUG:boto:Headers: {} 2023-07-29T00:08:34.280 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:34.280 DEBUG:boto:Port: 8001 2023-07-29T00:08:34.281 DEBUG:boto:Params: {} 2023-07-29T00:08:34.281 DEBUG:boto:Token: None 2023-07-29T00:08:34.282 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:34 GMT /hiiqos-49/ 2023-07-29T00:08:34.282 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2dB2L/CwRyB4/vX86SaDetnq9hc= 2023-07-29T00:08:34.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2dB2L/CwRyB4/vX86SaDetnq9hc=', 'Content-Length': '0'} 2023-07-29T00:08:34.285 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003cfe5de9cb96a580-0064c45882-1663-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:34.285 DEBUG:boto:path=/hiiqos-49/obj4 2023-07-29T00:08:34.286 DEBUG:boto:auth_path=/hiiqos-49/obj4 2023-07-29T00:08:34.287 DEBUG:boto:Method: PUT 2023-07-29T00:08:34.288 DEBUG:boto:Path: /hiiqos-49/obj4 2023-07-29T00:08:34.289 DEBUG:boto:Data: 2023-07-29T00:08:34.290 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T00:08:34.290 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:34.291 DEBUG:boto:Port: 8001 2023-07-29T00:08:34.291 DEBUG:boto:Params: {} 2023-07-29T00:08:34.291 DEBUG:boto:Token: None 2023-07-29T00:08:34.292 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 00:08:34 GMT /hiiqos-49/obj4 2023-07-29T00:08:34.292 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yy6i8GpPYtmGXINDPizEhwqncVU= 2023-07-29T00:08:34.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:08:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yy6i8GpPYtmGXINDPizEhwqncVU='} 2023-07-29T00:08:34.336 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000755c4386233ad75b-0064c45882-1663-a2'), ('Date', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:34.337 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:08:34.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:08:34.338 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:08:34.383 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:08:34.383 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:08:34.412 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T00:08:34.424 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T00:08:34.424 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:08:34.424 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:08:34.425 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:46:09.233742Z" 2023-07-29T00:08:34.425 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:34.425 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:08:34.425 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690589313.995996_717.1", 2023-07-29T00:08:34.425 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:08:33.995996Z" 2023-07-29T00:08:34.425 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:34.425 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:08:34.426 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:08:34.426 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T00:08:34.426 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:34.426 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:08:34.426 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:08:34.427 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T00:08:34.427 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:08:34.427 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T00:08:34.428 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690587969.233742_99.1', 1: '1_1690589313.995996_717.1', 2: '', 3: ''} 2023-07-29T00:08:34.429 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T00:08:34.429 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:08:34.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:08:34.430 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:08:34.475 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:08:34.475 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:08:34.503 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:08:34.474+0000 7f13e7922880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:08:34.504 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:08:34.475+0000 7f13e7922880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:08:34.516 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:08:34.516 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:08:34.516 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:08:34.517 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:08:34.517 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:08:34.517 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:08:34.517 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:08:34.517 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:34.517 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:08:34.517 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:08:34.518 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:08:34.518 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:08:34.518 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:08:34.518 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:08:34.518 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:08:34.518 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:08:34.518 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:08:34.519 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:08:34.519 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:08:34.519 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:08:34.519 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:34.519 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:08:34.519 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:08:34.519 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:08:34.520 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:08:34.520 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690589219.075786_700.1", 2023-07-29T00:08:34.520 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:08:34.520 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:08:34.520 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:08:34.520 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:06:59.075786Z", 2023-07-29T00:08:34.521 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:08:34.521 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:08:34.521 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:34.521 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:08:34.521 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:08:34.521 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:08:34.521 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:08:34.522 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:08:34.522 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:08:34.522 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:08:34.522 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:08:34.522 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:08:34.522 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:08:34.522 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:08:34.523 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:34.523 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:08:34.523 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:08:34.523 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:08:34.523 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:08:34.523 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:08:34.523 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:08:34.524 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:08:34.524 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:08:34.524 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:08:34.524 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:08:34.524 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:08:34.524 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:08:34.524 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:08:34.525 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:34.525 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:08:34.525 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:08:34.525 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:08:34.525 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:34.526 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:08:34Z" 2023-07-29T00:08:34.526 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:08:34.527 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690589219.075786_700.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:06:59.075786Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:08:34Z" } 2023-07-29T00:08:34.528 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690589219.075786_700.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:06:59.075786Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:08:34.529 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690589313.995996_717.1', 2: '', 3: ''} 2023-07-29T00:08:34.530 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690589219.075786_700.1', 2: '', 3: ''} 2023-07-29T00:08:34.530 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690589313.995996_717.1 target=1_1690589219.075786_700.1 2023-07-29T00:08:39.531 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:08:39.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:08:39.532 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:08:39.585 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:08:39.585 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:08:39.616 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:08:39.587+0000 7fc30c74b880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:08:39.617 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:08:39.588+0000 7fc30c74b880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:08:39.631 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:08:39.631 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:08:39.632 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:08:39.632 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:08:39.632 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:08:39.632 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:08:39.632 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:08:39.633 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:39.633 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:08:39.633 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:08:39.633 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:08:39.633 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:08:39.633 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:08:39.634 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:08:39.634 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:08:39.634 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:08:39.634 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:08:39.634 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:08:39.634 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:08:39.634 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:08:39.635 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:39.635 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:08:39.635 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:08:39.635 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:08:39.635 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:08:39.635 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690589313.995996_717.1", 2023-07-29T00:08:39.636 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:08:39.636 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:08:39.636 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:08:39.636 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:08:33.995996Z", 2023-07-29T00:08:39.636 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:08:39.637 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:08:39.637 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:39.637 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:08:39.637 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:08:39.637 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:08:39.638 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:08:39.638 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:08:39.638 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:08:39.638 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:08:39.638 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:08:39.638 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:08:39.639 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:08:39.639 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:08:39.639 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:39.639 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:08:39.639 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:08:39.639 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:08:39.640 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:08:39.640 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:08:39.640 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:08:39.640 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:08:39.640 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:08:39.640 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:08:39.641 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:08:39.641 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:08:39.641 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:08:39.641 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:08:39.641 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:39.642 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:08:39.642 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:08:39.642 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:08:39.642 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:08:39.642 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:08:39Z" 2023-07-29T00:08:39.642 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:08:39.643 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690589313.995996_717.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:08:33.995996Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:08:39Z" } 2023-07-29T00:08:39.644 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690589313.995996_717.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:08:33.995996Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:08:39.644 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690589313.995996_717.1', 2: '', 3: ''} 2023-07-29T00:08:39.645 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690589313.995996_717.1', 2: '', 3: ''} 2023-07-29T00:08:39.645 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T00:08:39.645 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-48 2023-07-29T00:08:39.646 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:08:39.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:08:39.646 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-48 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:08:39.701 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:08:39.701 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:08:39.733 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:08:39.704+0000 7fc5f5818880 1 waiting to reach incremental sync.. 2023-07-29T00:08:44.734 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:08:44.705+0000 7fc5f5818880 1 waiting to reach incremental sync.. 2023-07-29T00:08:49.735 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:08:49.706+0000 7fc5f5818880 1 waiting to reach incremental sync.. 2023-07-29T00:08:54.737 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:08:54.707+0000 7fc5f5818880 1 bucket sync caught up with source: 2023-07-29T00:08:54.737 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000001.30.6, , 00000000001.22.6, , 00000000001.20.6, , , 00000000001.45.6, , , ] 2023-07-29T00:08:54.738 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000001.30.6, , 00000000001.22.6, , 00000000001.20.6, , , 00000000001.45.6, , , ] 2023-07-29T00:08:54.738 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:08:54.707+0000 7fc5f5818880 0 bucket checkpoint complete 2023-07-29T00:08:54.755 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-48 2023-07-29T00:08:54.756 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:08:54.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:08:54.758 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-48 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:08:54.817 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:08:54.817 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:08:54.852 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:08:54.822+0000 7fc00072c880 1 waiting to reach incremental sync.. 2023-07-29T00:08:59.854 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:08:59.824+0000 7fc00072c880 1 bucket sync caught up with source: 2023-07-29T00:08:59.855 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000001.1336.6, , 00000000001.202.6, , 00000000001.201.6, , , 00000000001.389.6, , , ] 2023-07-29T00:08:59.855 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000001.1336.6, , 00000000001.202.6, , 00000000001.201.6, , , 00000000001.389.6, , , ] 2023-07-29T00:08:59.855 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:08:59.824+0000 7fc00072c880 0 bucket checkpoint complete 2023-07-29T00:08:59.872 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-48 zones={a2, a1} 2023-07-29T00:08:59.872 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:08:59.872 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:08:59.873 DEBUG:boto:Method: HEAD 2023-07-29T00:08:59.873 DEBUG:boto:Path: /hiiqos-48/ 2023-07-29T00:08:59.874 DEBUG:boto:Data: 2023-07-29T00:08:59.874 DEBUG:boto:Headers: {} 2023-07-29T00:08:59.874 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:59.875 DEBUG:boto:Port: 8001 2023-07-29T00:08:59.875 DEBUG:boto:Params: {} 2023-07-29T00:08:59.875 DEBUG:boto:Token: None 2023-07-29T00:08:59.876 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:59 GMT /hiiqos-48/ 2023-07-29T00:08:59.876 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MXZJdSrsljQWETQloWh5aDzxTvM= 2023-07-29T00:08:59.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MXZJdSrsljQWETQloWh5aDzxTvM=', 'Content-Length': '0'} 2023-07-29T00:08:59.879 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005b4285336b653f97-0064c4589b-1663-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:08:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:59.879 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:08:59.880 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:08:59.881 DEBUG:boto:Method: HEAD 2023-07-29T00:08:59.881 DEBUG:boto:Path: /hiiqos-48/ 2023-07-29T00:08:59.882 DEBUG:boto:Data: 2023-07-29T00:08:59.882 DEBUG:boto:Headers: {} 2023-07-29T00:08:59.882 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:59.883 DEBUG:boto:Port: 8000 2023-07-29T00:08:59.883 DEBUG:boto:Params: {} 2023-07-29T00:08:59.883 DEBUG:boto:Token: None 2023-07-29T00:08:59.884 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:59 GMT /hiiqos-48/ 2023-07-29T00:08:59.884 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MXZJdSrsljQWETQloWh5aDzxTvM= 2023-07-29T00:08:59.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MXZJdSrsljQWETQloWh5aDzxTvM=', 'Content-Length': '0'} 2023-07-29T00:08:59.887 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005386d949a62cc804-0064c4589b-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:08:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:59.887 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-29T00:08:59.888 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:08:59.888 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:08:59.888 DEBUG:boto:path=/hiiqos-48/?versions&max-keys=999 2023-07-29T00:08:59.889 DEBUG:boto:auth_path=/hiiqos-48/?versions&max-keys=999 2023-07-29T00:08:59.889 DEBUG:boto:Method: GET 2023-07-29T00:08:59.889 DEBUG:boto:Path: /hiiqos-48/?versions&max-keys=999 2023-07-29T00:08:59.890 DEBUG:boto:Data: 2023-07-29T00:08:59.890 DEBUG:boto:Headers: {} 2023-07-29T00:08:59.890 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:59.891 DEBUG:boto:Port: 8001 2023-07-29T00:08:59.891 DEBUG:boto:Params: {} 2023-07-29T00:08:59.891 DEBUG:boto:Token: None 2023-07-29T00:08:59.892 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:59 GMT /hiiqos-48/?versions 2023-07-29T00:08:59.892 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:C1fl65zW0omgEnos8B7KB7P5C30= 2023-07-29T00:08:59.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:C1fl65zW0omgEnos8B7KB7P5C30=', 'Content-Length': '0'} 2023-07-29T00:08:59.895 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6a276b4edb8be70-0064c4589b-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:59.895 DEBUG:boto:b'hiiqos-48999falseobj1nulltrue2023-07-29T00:08:34.018Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-07-29T00:08:34.078Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-07-29T00:08:34.094Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-07-29T00:08:34.109Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:08:59.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-07-29T00:08:59.897 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-07-29T00:08:59.897 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-07-29T00:08:59.897 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-07-29T00:08:59.898 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-29T00:08:59.898 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:08:59.898 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:08:59.899 DEBUG:boto:path=/hiiqos-48/?versions&max-keys=999 2023-07-29T00:08:59.899 DEBUG:boto:auth_path=/hiiqos-48/?versions&max-keys=999 2023-07-29T00:08:59.899 DEBUG:boto:Method: GET 2023-07-29T00:08:59.900 DEBUG:boto:Path: /hiiqos-48/?versions&max-keys=999 2023-07-29T00:08:59.900 DEBUG:boto:Data: 2023-07-29T00:08:59.900 DEBUG:boto:Headers: {} 2023-07-29T00:08:59.901 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:59.901 DEBUG:boto:Port: 8000 2023-07-29T00:08:59.901 DEBUG:boto:Params: {} 2023-07-29T00:08:59.902 DEBUG:boto:Token: None 2023-07-29T00:08:59.902 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:59 GMT /hiiqos-48/?versions 2023-07-29T00:08:59.903 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:C1fl65zW0omgEnos8B7KB7P5C30= 2023-07-29T00:08:59.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:C1fl65zW0omgEnos8B7KB7P5C30=', 'Content-Length': '0'} 2023-07-29T00:08:59.905 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f63d60f6965ff977-0064c4589b-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:59.906 DEBUG:boto:b'hiiqos-48999falseobj1nulltrue2023-07-29T00:08:34.018Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-07-29T00:08:34.078Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-07-29T00:08:34.094Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-07-29T00:08:34.109Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:08:59.907 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-07-29T00:08:59.907 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-07-29T00:08:59.907 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-07-29T00:08:59.907 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-07-29T00:08:59.908 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:08:59.908 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:08:59.909 DEBUG:boto:path=/hiiqos-48/?versions&max-keys=999 2023-07-29T00:08:59.909 DEBUG:boto:auth_path=/hiiqos-48/?versions&max-keys=999 2023-07-29T00:08:59.909 DEBUG:boto:Method: GET 2023-07-29T00:08:59.910 DEBUG:boto:Path: /hiiqos-48/?versions&max-keys=999 2023-07-29T00:08:59.910 DEBUG:boto:Data: 2023-07-29T00:08:59.910 DEBUG:boto:Headers: {} 2023-07-29T00:08:59.911 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:59.911 DEBUG:boto:Port: 8001 2023-07-29T00:08:59.911 DEBUG:boto:Params: {} 2023-07-29T00:08:59.912 DEBUG:boto:Token: None 2023-07-29T00:08:59.912 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:59 GMT /hiiqos-48/?versions 2023-07-29T00:08:59.913 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:C1fl65zW0omgEnos8B7KB7P5C30= 2023-07-29T00:08:59.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:C1fl65zW0omgEnos8B7KB7P5C30=', 'Content-Length': '0'} 2023-07-29T00:08:59.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0077f16f836e785-0064c4589b-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:59.916 DEBUG:boto:b'hiiqos-48999falseobj1nulltrue2023-07-29T00:08:34.018Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-07-29T00:08:34.078Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-07-29T00:08:34.094Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-07-29T00:08:34.109Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:08:59.916 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:08:59.917 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:08:59.917 DEBUG:boto:path=/hiiqos-48/?versions&max-keys=999 2023-07-29T00:08:59.917 DEBUG:boto:auth_path=/hiiqos-48/?versions&max-keys=999 2023-07-29T00:08:59.918 DEBUG:boto:Method: GET 2023-07-29T00:08:59.918 DEBUG:boto:Path: /hiiqos-48/?versions&max-keys=999 2023-07-29T00:08:59.919 DEBUG:boto:Data: 2023-07-29T00:08:59.919 DEBUG:boto:Headers: {} 2023-07-29T00:08:59.919 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:59.920 DEBUG:boto:Port: 8000 2023-07-29T00:08:59.920 DEBUG:boto:Params: {} 2023-07-29T00:08:59.920 DEBUG:boto:Token: None 2023-07-29T00:08:59.921 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:59 GMT /hiiqos-48/?versions 2023-07-29T00:08:59.921 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:C1fl65zW0omgEnos8B7KB7P5C30= 2023-07-29T00:08:59.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:C1fl65zW0omgEnos8B7KB7P5C30=', 'Content-Length': '0'} 2023-07-29T00:08:59.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006520a4f4efdabdcb-0064c4589b-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:59.924 DEBUG:boto:b'hiiqos-48999falseobj1nulltrue2023-07-29T00:08:34.018Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-07-29T00:08:34.078Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-07-29T00:08:34.094Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-07-29T00:08:34.109Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:08:59.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-07-29T00:08:59.925 DEBUG:boto:path=/hiiqos-48/obj1 2023-07-29T00:08:59.926 DEBUG:boto:auth_path=/hiiqos-48/obj1 2023-07-29T00:08:59.926 DEBUG:boto:path=/hiiqos-48/obj1?versionId=null 2023-07-29T00:08:59.926 DEBUG:boto:auth_path=/hiiqos-48/obj1?versionId=null 2023-07-29T00:08:59.927 DEBUG:boto:Method: GET 2023-07-29T00:08:59.927 DEBUG:boto:Path: /hiiqos-48/obj1?versionId=null 2023-07-29T00:08:59.927 DEBUG:boto:Data: 2023-07-29T00:08:59.928 DEBUG:boto:Headers: {} 2023-07-29T00:08:59.928 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:59.928 DEBUG:boto:Port: 8001 2023-07-29T00:08:59.929 DEBUG:boto:Params: {} 2023-07-29T00:08:59.929 DEBUG:boto:Token: None 2023-07-29T00:08:59.930 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:59 GMT /hiiqos-48/obj1?versionId=null 2023-07-29T00:08:59.930 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cpMhzcWgMUBd1CIT+eSdSH0mhyE= 2023-07-29T00:08:59.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cpMhzcWgMUBd1CIT+eSdSH0mhyE=', 'Content-Length': '0'} 2023-07-29T00:08:59.932 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000166611e3b4a93088-0064c4589b-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:59.933 DEBUG:boto:path=/hiiqos-48/obj1 2023-07-29T00:08:59.933 DEBUG:boto:auth_path=/hiiqos-48/obj1 2023-07-29T00:08:59.934 DEBUG:boto:path=/hiiqos-48/obj1?versionId=null 2023-07-29T00:08:59.934 DEBUG:boto:auth_path=/hiiqos-48/obj1?versionId=null 2023-07-29T00:08:59.934 DEBUG:boto:Method: GET 2023-07-29T00:08:59.935 DEBUG:boto:Path: /hiiqos-48/obj1?versionId=null 2023-07-29T00:08:59.935 DEBUG:boto:Data: 2023-07-29T00:08:59.935 DEBUG:boto:Headers: {} 2023-07-29T00:08:59.936 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:59.936 DEBUG:boto:Port: 8000 2023-07-29T00:08:59.936 DEBUG:boto:Params: {} 2023-07-29T00:08:59.937 DEBUG:boto:Token: None 2023-07-29T00:08:59.937 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:59 GMT /hiiqos-48/obj1?versionId=null 2023-07-29T00:08:59.938 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cpMhzcWgMUBd1CIT+eSdSH0mhyE= 2023-07-29T00:08:59.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cpMhzcWgMUBd1CIT+eSdSH0mhyE=', 'Content-Length': '0'} 2023-07-29T00:08:59.940 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ab11e2751a10117b-0064c4589b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:59.940 DEBUG:boto:path=/hiiqos-48/obj1 2023-07-29T00:08:59.941 DEBUG:boto:auth_path=/hiiqos-48/obj1 2023-07-29T00:08:59.941 DEBUG:boto:path=/hiiqos-48/obj1?versionId=null 2023-07-29T00:08:59.942 DEBUG:boto:auth_path=/hiiqos-48/obj1?versionId=null 2023-07-29T00:08:59.942 DEBUG:boto:Method: HEAD 2023-07-29T00:08:59.942 DEBUG:boto:Path: /hiiqos-48/obj1?versionId=null 2023-07-29T00:08:59.943 DEBUG:boto:Data: 2023-07-29T00:08:59.943 DEBUG:boto:Headers: {} 2023-07-29T00:08:59.943 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:59.944 DEBUG:boto:Port: 8001 2023-07-29T00:08:59.944 DEBUG:boto:Params: {} 2023-07-29T00:08:59.944 DEBUG:boto:Token: None 2023-07-29T00:08:59.945 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:59 GMT /hiiqos-48/obj1?versionId=null 2023-07-29T00:08:59.945 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4jaV/6lDzrKb6ILJbMruZAcBmz8= 2023-07-29T00:08:59.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4jaV/6lDzrKb6ILJbMruZAcBmz8=', 'Content-Length': '0'} 2023-07-29T00:08:59.947 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a176f8130d58214d-0064c4589b-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:59.948 DEBUG:boto:path=/hiiqos-48/obj1 2023-07-29T00:08:59.948 DEBUG:boto:auth_path=/hiiqos-48/obj1 2023-07-29T00:08:59.948 DEBUG:boto:path=/hiiqos-48/obj1?versionId=null 2023-07-29T00:08:59.949 DEBUG:boto:auth_path=/hiiqos-48/obj1?versionId=null 2023-07-29T00:08:59.949 DEBUG:boto:Method: HEAD 2023-07-29T00:08:59.949 DEBUG:boto:Path: /hiiqos-48/obj1?versionId=null 2023-07-29T00:08:59.950 DEBUG:boto:Data: 2023-07-29T00:08:59.950 DEBUG:boto:Headers: {} 2023-07-29T00:08:59.950 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:59.951 DEBUG:boto:Port: 8000 2023-07-29T00:08:59.951 DEBUG:boto:Params: {} 2023-07-29T00:08:59.951 DEBUG:boto:Token: None 2023-07-29T00:08:59.952 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:59 GMT /hiiqos-48/obj1?versionId=null 2023-07-29T00:08:59.952 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4jaV/6lDzrKb6ILJbMruZAcBmz8= 2023-07-29T00:08:59.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4jaV/6lDzrKb6ILJbMruZAcBmz8=', 'Content-Length': '0'} 2023-07-29T00:08:59.954 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e491d382b16bc8c-0064c4589b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:59.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-07-29T00:08:59.955 DEBUG:boto:path=/hiiqos-48/obj1 2023-07-29T00:08:59.955 DEBUG:boto:auth_path=/hiiqos-48/obj1 2023-07-29T00:08:59.956 DEBUG:boto:path=/hiiqos-48/obj1?versionId=null 2023-07-29T00:08:59.956 DEBUG:boto:auth_path=/hiiqos-48/obj1?versionId=null 2023-07-29T00:08:59.956 DEBUG:boto:Method: GET 2023-07-29T00:08:59.957 DEBUG:boto:Path: /hiiqos-48/obj1?versionId=null 2023-07-29T00:08:59.957 DEBUG:boto:Data: 2023-07-29T00:08:59.958 DEBUG:boto:Headers: {} 2023-07-29T00:08:59.958 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:59.958 DEBUG:boto:Port: 8001 2023-07-29T00:08:59.959 DEBUG:boto:Params: {} 2023-07-29T00:08:59.959 DEBUG:boto:Token: None 2023-07-29T00:08:59.959 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:59 GMT /hiiqos-48/obj1?versionId=null 2023-07-29T00:08:59.960 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cpMhzcWgMUBd1CIT+eSdSH0mhyE= 2023-07-29T00:08:59.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cpMhzcWgMUBd1CIT+eSdSH0mhyE=', 'Content-Length': '0'} 2023-07-29T00:08:59.962 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000e412f71fa1f5d39-0064c4589b-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:59.963 DEBUG:boto:path=/hiiqos-48/obj1 2023-07-29T00:08:59.963 DEBUG:boto:auth_path=/hiiqos-48/obj1 2023-07-29T00:08:59.963 DEBUG:boto:path=/hiiqos-48/obj1?versionId=null 2023-07-29T00:08:59.964 DEBUG:boto:auth_path=/hiiqos-48/obj1?versionId=null 2023-07-29T00:08:59.964 DEBUG:boto:Method: GET 2023-07-29T00:08:59.965 DEBUG:boto:Path: /hiiqos-48/obj1?versionId=null 2023-07-29T00:08:59.965 DEBUG:boto:Data: 2023-07-29T00:08:59.965 DEBUG:boto:Headers: {} 2023-07-29T00:08:59.966 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:59.966 DEBUG:boto:Port: 8000 2023-07-29T00:08:59.966 DEBUG:boto:Params: {} 2023-07-29T00:08:59.967 DEBUG:boto:Token: None 2023-07-29T00:08:59.967 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:59 GMT /hiiqos-48/obj1?versionId=null 2023-07-29T00:08:59.967 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cpMhzcWgMUBd1CIT+eSdSH0mhyE= 2023-07-29T00:08:59.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cpMhzcWgMUBd1CIT+eSdSH0mhyE=', 'Content-Length': '0'} 2023-07-29T00:08:59.969 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000064d0e8dc8b8dd4e4-0064c4589b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:59.970 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:08:59.970 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:08:59.971 DEBUG:boto:path=/hiiqos-48/?prefix=obj1 2023-07-29T00:08:59.971 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj1 2023-07-29T00:08:59.972 DEBUG:boto:Method: GET 2023-07-29T00:08:59.972 DEBUG:boto:Path: /hiiqos-48/?prefix=obj1 2023-07-29T00:08:59.972 DEBUG:boto:Data: 2023-07-29T00:08:59.973 DEBUG:boto:Headers: {} 2023-07-29T00:08:59.973 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:59.973 DEBUG:boto:Port: 8001 2023-07-29T00:08:59.974 DEBUG:boto:Params: {} 2023-07-29T00:08:59.974 DEBUG:boto:Token: None 2023-07-29T00:08:59.974 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:59 GMT /hiiqos-48/ 2023-07-29T00:08:59.975 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:avPcNS4Tz/1+rlqSxk5LsdMCD7w= 2023-07-29T00:08:59.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:avPcNS4Tz/1+rlqSxk5LsdMCD7w=', 'Content-Length': '0'} 2023-07-29T00:08:59.978 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083002411f081b378-0064c4589b-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:59.978 DEBUG:boto:b'hiiqos-48obj11000falseobj12023-07-29T00:08:34.018Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:08:59.979 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:08:59.979 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:08:59.980 DEBUG:boto:path=/hiiqos-48/?prefix=obj1 2023-07-29T00:08:59.980 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj1 2023-07-29T00:08:59.980 DEBUG:boto:Method: GET 2023-07-29T00:08:59.981 DEBUG:boto:Path: /hiiqos-48/?prefix=obj1 2023-07-29T00:08:59.981 DEBUG:boto:Data: 2023-07-29T00:08:59.981 DEBUG:boto:Headers: {} 2023-07-29T00:08:59.982 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:59.982 DEBUG:boto:Port: 8000 2023-07-29T00:08:59.982 DEBUG:boto:Params: {} 2023-07-29T00:08:59.983 DEBUG:boto:Token: None 2023-07-29T00:08:59.983 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:08:59 GMT /hiiqos-48/ 2023-07-29T00:08:59.984 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:avPcNS4Tz/1+rlqSxk5LsdMCD7w= 2023-07-29T00:08:59.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:avPcNS4Tz/1+rlqSxk5LsdMCD7w=', 'Content-Length': '0'} 2023-07-29T00:08:59.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4d0cbc89ca64eb5-0064c4589b-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:59.987 DEBUG:boto:b'hiiqos-48obj11000falseobj12023-07-29T00:08:34.018Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:08:59.987 DEBUG:boto:path=/hiiqos-48/obj1 2023-07-29T00:08:59.988 DEBUG:boto:auth_path=/hiiqos-48/obj1 2023-07-29T00:08:59.988 DEBUG:boto:Method: HEAD 2023-07-29T00:08:59.989 DEBUG:boto:Path: /hiiqos-48/obj1 2023-07-29T00:08:59.989 DEBUG:boto:Data: 2023-07-29T00:08:59.989 DEBUG:boto:Headers: {} 2023-07-29T00:08:59.990 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:08:59.990 DEBUG:boto:Port: 8001 2023-07-29T00:08:59.990 DEBUG:boto:Params: {} 2023-07-29T00:08:59.991 DEBUG:boto:Token: None 2023-07-29T00:08:59.991 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:59 GMT /hiiqos-48/obj1 2023-07-29T00:08:59.991 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:NXUAJb2QV/FaVPfZZPJTbn7dRdU= 2023-07-29T00:08:59.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:NXUAJb2QV/FaVPfZZPJTbn7dRdU=', 'Content-Length': '0'} 2023-07-29T00:08:59.993 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b526fb124fc1d1da-0064c4589b-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:08:59.994 DEBUG:boto:path=/hiiqos-48/obj1 2023-07-29T00:08:59.994 DEBUG:boto:auth_path=/hiiqos-48/obj1 2023-07-29T00:08:59.995 DEBUG:boto:Method: HEAD 2023-07-29T00:08:59.995 DEBUG:boto:Path: /hiiqos-48/obj1 2023-07-29T00:08:59.995 DEBUG:boto:Data: 2023-07-29T00:08:59.996 DEBUG:boto:Headers: {} 2023-07-29T00:08:59.996 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:08:59.996 DEBUG:boto:Port: 8000 2023-07-29T00:08:59.997 DEBUG:boto:Params: {} 2023-07-29T00:08:59.997 DEBUG:boto:Token: None 2023-07-29T00:08:59.997 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:08:59 GMT /hiiqos-48/obj1 2023-07-29T00:08:59.998 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:NXUAJb2QV/FaVPfZZPJTbn7dRdU= 2023-07-29T00:08:59.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:08:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:NXUAJb2QV/FaVPfZZPJTbn7dRdU=', 'Content-Length': '0'} 2023-07-29T00:09:00.000 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000084e51792a54b1bb4-0064c4589b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-07-29T00:09:00.001 DEBUG:boto:path=/hiiqos-48/obj1 2023-07-29T00:09:00.001 DEBUG:boto:auth_path=/hiiqos-48/obj1 2023-07-29T00:09:00.001 DEBUG:boto:Method: GET 2023-07-29T00:09:00.002 DEBUG:boto:Path: /hiiqos-48/obj1 2023-07-29T00:09:00.002 DEBUG:boto:Data: 2023-07-29T00:09:00.003 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.003 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:00.003 DEBUG:boto:Port: 8001 2023-07-29T00:09:00.004 DEBUG:boto:Params: {} 2023-07-29T00:09:00.004 DEBUG:boto:Token: None 2023-07-29T00:09:00.004 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/obj1 2023-07-29T00:09:00.005 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KTJ8/8buE4KanArnDOzylI+Q5NA= 2023-07-29T00:09:00.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KTJ8/8buE4KanArnDOzylI+Q5NA=', 'Content-Length': '0'} 2023-07-29T00:09:00.007 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b3f3eac255a6297f-0064c4589b-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.007 DEBUG:boto:path=/hiiqos-48/obj1 2023-07-29T00:09:00.008 DEBUG:boto:auth_path=/hiiqos-48/obj1 2023-07-29T00:09:00.008 DEBUG:boto:Method: GET 2023-07-29T00:09:00.009 DEBUG:boto:Path: /hiiqos-48/obj1 2023-07-29T00:09:00.009 DEBUG:boto:Data: 2023-07-29T00:09:00.009 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.010 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:00.010 DEBUG:boto:Port: 8000 2023-07-29T00:09:00.010 DEBUG:boto:Params: {} 2023-07-29T00:09:00.011 DEBUG:boto:Token: None 2023-07-29T00:09:00.011 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/obj1 2023-07-29T00:09:00.012 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KTJ8/8buE4KanArnDOzylI+Q5NA= 2023-07-29T00:09:00.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KTJ8/8buE4KanArnDOzylI+Q5NA=', 'Content-Length': '0'} 2023-07-29T00:09:00.014 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c0eac1c0e504d983-0064c4589b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:08:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.014 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:00.015 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:00.015 DEBUG:boto:path=/hiiqos-48/?prefix=obj1 2023-07-29T00:09:00.015 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj1 2023-07-29T00:09:00.016 DEBUG:boto:Method: GET 2023-07-29T00:09:00.016 DEBUG:boto:Path: /hiiqos-48/?prefix=obj1 2023-07-29T00:09:00.019 DEBUG:boto:Data: 2023-07-29T00:09:00.020 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.020 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:00.021 DEBUG:boto:Port: 8001 2023-07-29T00:09:00.021 DEBUG:boto:Params: {} 2023-07-29T00:09:00.021 DEBUG:boto:Token: None 2023-07-29T00:09:00.022 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/ 2023-07-29T00:09:00.022 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eRChp5Tsa7c/OC+TTBj0T4/dDKo= 2023-07-29T00:09:00.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eRChp5Tsa7c/OC+TTBj0T4/dDKo=', 'Content-Length': '0'} 2023-07-29T00:09:00.024 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e18a990f9e66e676-0064c4589b-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:08:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.025 DEBUG:boto:b'hiiqos-48obj11000falseobj12023-07-29T00:08:34.018Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:00.026 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:00.026 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:00.026 DEBUG:boto:path=/hiiqos-48/?prefix=obj1 2023-07-29T00:09:00.027 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj1 2023-07-29T00:09:00.027 DEBUG:boto:Method: GET 2023-07-29T00:09:00.027 DEBUG:boto:Path: /hiiqos-48/?prefix=obj1 2023-07-29T00:09:00.028 DEBUG:boto:Data: 2023-07-29T00:09:00.028 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.028 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:00.029 DEBUG:boto:Port: 8000 2023-07-29T00:09:00.029 DEBUG:boto:Params: {} 2023-07-29T00:09:00.030 DEBUG:boto:Token: None 2023-07-29T00:09:00.030 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/ 2023-07-29T00:09:00.030 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eRChp5Tsa7c/OC+TTBj0T4/dDKo= 2023-07-29T00:09:00.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eRChp5Tsa7c/OC+TTBj0T4/dDKo=', 'Content-Length': '0'} 2023-07-29T00:09:00.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf9e24f6f0a479d9-0064c4589c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.033 DEBUG:boto:b'hiiqos-48obj11000falseobj12023-07-29T00:08:34.018Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:00.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-07-29T00:09:00.034 DEBUG:boto:path=/hiiqos-48/obj2 2023-07-29T00:09:00.035 DEBUG:boto:auth_path=/hiiqos-48/obj2 2023-07-29T00:09:00.035 DEBUG:boto:path=/hiiqos-48/obj2?versionId=null 2023-07-29T00:09:00.036 DEBUG:boto:auth_path=/hiiqos-48/obj2?versionId=null 2023-07-29T00:09:00.036 DEBUG:boto:Method: GET 2023-07-29T00:09:00.036 DEBUG:boto:Path: /hiiqos-48/obj2?versionId=null 2023-07-29T00:09:00.037 DEBUG:boto:Data: 2023-07-29T00:09:00.037 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.037 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:00.038 DEBUG:boto:Port: 8001 2023-07-29T00:09:00.038 DEBUG:boto:Params: {} 2023-07-29T00:09:00.038 DEBUG:boto:Token: None 2023-07-29T00:09:00.039 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/obj2?versionId=null 2023-07-29T00:09:00.039 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:NglpIHXQe+oq8tBq6KKgVUjpRGs= 2023-07-29T00:09:00.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:NglpIHXQe+oq8tBq6KKgVUjpRGs=', 'Content-Length': '0'} 2023-07-29T00:09:00.041 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d438efe07b4b5ae-0064c4589c-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.084 DEBUG:boto:path=/hiiqos-48/obj2 2023-07-29T00:09:00.085 DEBUG:boto:auth_path=/hiiqos-48/obj2 2023-07-29T00:09:00.085 DEBUG:boto:path=/hiiqos-48/obj2?versionId=null 2023-07-29T00:09:00.085 DEBUG:boto:auth_path=/hiiqos-48/obj2?versionId=null 2023-07-29T00:09:00.086 DEBUG:boto:Method: GET 2023-07-29T00:09:00.086 DEBUG:boto:Path: /hiiqos-48/obj2?versionId=null 2023-07-29T00:09:00.086 DEBUG:boto:Data: 2023-07-29T00:09:00.087 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.087 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:00.088 DEBUG:boto:Port: 8000 2023-07-29T00:09:00.088 DEBUG:boto:Params: {} 2023-07-29T00:09:00.088 DEBUG:boto:Token: None 2023-07-29T00:09:00.089 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/obj2?versionId=null 2023-07-29T00:09:00.089 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:NglpIHXQe+oq8tBq6KKgVUjpRGs= 2023-07-29T00:09:00.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:NglpIHXQe+oq8tBq6KKgVUjpRGs=', 'Content-Length': '0'} 2023-07-29T00:09:00.091 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bd860ff1e2ec418b-0064c4589c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.092 DEBUG:boto:path=/hiiqos-48/obj2 2023-07-29T00:09:00.092 DEBUG:boto:auth_path=/hiiqos-48/obj2 2023-07-29T00:09:00.093 DEBUG:boto:path=/hiiqos-48/obj2?versionId=null 2023-07-29T00:09:00.093 DEBUG:boto:auth_path=/hiiqos-48/obj2?versionId=null 2023-07-29T00:09:00.093 DEBUG:boto:Method: HEAD 2023-07-29T00:09:00.094 DEBUG:boto:Path: /hiiqos-48/obj2?versionId=null 2023-07-29T00:09:00.094 DEBUG:boto:Data: 2023-07-29T00:09:00.094 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.095 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:00.095 DEBUG:boto:Port: 8001 2023-07-29T00:09:00.095 DEBUG:boto:Params: {} 2023-07-29T00:09:00.096 DEBUG:boto:Token: None 2023-07-29T00:09:00.096 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/obj2?versionId=null 2023-07-29T00:09:00.097 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QUjAX+4zM+t2ekaYVv6LzlWMnh0= 2023-07-29T00:09:00.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QUjAX+4zM+t2ekaYVv6LzlWMnh0=', 'Content-Length': '0'} 2023-07-29T00:09:00.099 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c20a4b705b9775e5-0064c4589c-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.099 DEBUG:boto:path=/hiiqos-48/obj2 2023-07-29T00:09:00.100 DEBUG:boto:auth_path=/hiiqos-48/obj2 2023-07-29T00:09:00.100 DEBUG:boto:path=/hiiqos-48/obj2?versionId=null 2023-07-29T00:09:00.100 DEBUG:boto:auth_path=/hiiqos-48/obj2?versionId=null 2023-07-29T00:09:00.101 DEBUG:boto:Method: HEAD 2023-07-29T00:09:00.101 DEBUG:boto:Path: /hiiqos-48/obj2?versionId=null 2023-07-29T00:09:00.101 DEBUG:boto:Data: 2023-07-29T00:09:00.102 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.102 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:00.102 DEBUG:boto:Port: 8000 2023-07-29T00:09:00.103 DEBUG:boto:Params: {} 2023-07-29T00:09:00.103 DEBUG:boto:Token: None 2023-07-29T00:09:00.103 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/obj2?versionId=null 2023-07-29T00:09:00.104 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QUjAX+4zM+t2ekaYVv6LzlWMnh0= 2023-07-29T00:09:00.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QUjAX+4zM+t2ekaYVv6LzlWMnh0=', 'Content-Length': '0'} 2023-07-29T00:09:00.106 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003cd7cfe685c5cdd5-0064c4589c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-07-29T00:09:00.107 DEBUG:boto:path=/hiiqos-48/obj2 2023-07-29T00:09:00.107 DEBUG:boto:auth_path=/hiiqos-48/obj2 2023-07-29T00:09:00.107 DEBUG:boto:path=/hiiqos-48/obj2?versionId=null 2023-07-29T00:09:00.108 DEBUG:boto:auth_path=/hiiqos-48/obj2?versionId=null 2023-07-29T00:09:00.108 DEBUG:boto:Method: GET 2023-07-29T00:09:00.109 DEBUG:boto:Path: /hiiqos-48/obj2?versionId=null 2023-07-29T00:09:00.109 DEBUG:boto:Data: 2023-07-29T00:09:00.109 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.110 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:00.110 DEBUG:boto:Port: 8001 2023-07-29T00:09:00.110 DEBUG:boto:Params: {} 2023-07-29T00:09:00.111 DEBUG:boto:Token: None 2023-07-29T00:09:00.111 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/obj2?versionId=null 2023-07-29T00:09:00.111 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:NglpIHXQe+oq8tBq6KKgVUjpRGs= 2023-07-29T00:09:00.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:NglpIHXQe+oq8tBq6KKgVUjpRGs=', 'Content-Length': '0'} 2023-07-29T00:09:00.114 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000091da246b49b3eab9-0064c4589c-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.114 DEBUG:boto:path=/hiiqos-48/obj2 2023-07-29T00:09:00.115 DEBUG:boto:auth_path=/hiiqos-48/obj2 2023-07-29T00:09:00.115 DEBUG:boto:path=/hiiqos-48/obj2?versionId=null 2023-07-29T00:09:00.115 DEBUG:boto:auth_path=/hiiqos-48/obj2?versionId=null 2023-07-29T00:09:00.116 DEBUG:boto:Method: GET 2023-07-29T00:09:00.116 DEBUG:boto:Path: /hiiqos-48/obj2?versionId=null 2023-07-29T00:09:00.116 DEBUG:boto:Data: 2023-07-29T00:09:00.117 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.117 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:00.117 DEBUG:boto:Port: 8000 2023-07-29T00:09:00.118 DEBUG:boto:Params: {} 2023-07-29T00:09:00.118 DEBUG:boto:Token: None 2023-07-29T00:09:00.119 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/obj2?versionId=null 2023-07-29T00:09:00.119 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:NglpIHXQe+oq8tBq6KKgVUjpRGs= 2023-07-29T00:09:00.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:NglpIHXQe+oq8tBq6KKgVUjpRGs=', 'Content-Length': '0'} 2023-07-29T00:09:00.121 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000927e876717df3a94-0064c4589c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.122 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:00.122 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:00.122 DEBUG:boto:path=/hiiqos-48/?prefix=obj2 2023-07-29T00:09:00.123 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj2 2023-07-29T00:09:00.123 DEBUG:boto:Method: GET 2023-07-29T00:09:00.123 DEBUG:boto:Path: /hiiqos-48/?prefix=obj2 2023-07-29T00:09:00.124 DEBUG:boto:Data: 2023-07-29T00:09:00.124 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.124 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:00.125 DEBUG:boto:Port: 8001 2023-07-29T00:09:00.125 DEBUG:boto:Params: {} 2023-07-29T00:09:00.126 DEBUG:boto:Token: None 2023-07-29T00:09:00.126 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/ 2023-07-29T00:09:00.126 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eRChp5Tsa7c/OC+TTBj0T4/dDKo= 2023-07-29T00:09:00.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eRChp5Tsa7c/OC+TTBj0T4/dDKo=', 'Content-Length': '0'} 2023-07-29T00:09:00.129 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da61b4731f98b461-0064c4589c-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.129 DEBUG:boto:b'hiiqos-48obj21000falseobj22023-07-29T00:08:34.078Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:00.130 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:00.130 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:00.131 DEBUG:boto:path=/hiiqos-48/?prefix=obj2 2023-07-29T00:09:00.131 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj2 2023-07-29T00:09:00.131 DEBUG:boto:Method: GET 2023-07-29T00:09:00.132 DEBUG:boto:Path: /hiiqos-48/?prefix=obj2 2023-07-29T00:09:00.132 DEBUG:boto:Data: 2023-07-29T00:09:00.133 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.133 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:00.133 DEBUG:boto:Port: 8000 2023-07-29T00:09:00.134 DEBUG:boto:Params: {} 2023-07-29T00:09:00.134 DEBUG:boto:Token: None 2023-07-29T00:09:00.134 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/ 2023-07-29T00:09:00.135 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eRChp5Tsa7c/OC+TTBj0T4/dDKo= 2023-07-29T00:09:00.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eRChp5Tsa7c/OC+TTBj0T4/dDKo=', 'Content-Length': '0'} 2023-07-29T00:09:00.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc4296cabe6cea19-0064c4589c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.138 DEBUG:boto:b'hiiqos-48obj21000falseobj22023-07-29T00:08:34.078Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:00.139 DEBUG:boto:path=/hiiqos-48/obj2 2023-07-29T00:09:00.139 DEBUG:boto:auth_path=/hiiqos-48/obj2 2023-07-29T00:09:00.139 DEBUG:boto:Method: HEAD 2023-07-29T00:09:00.140 DEBUG:boto:Path: /hiiqos-48/obj2 2023-07-29T00:09:00.140 DEBUG:boto:Data: 2023-07-29T00:09:00.140 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.141 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:00.141 DEBUG:boto:Port: 8001 2023-07-29T00:09:00.141 DEBUG:boto:Params: {} 2023-07-29T00:09:00.142 DEBUG:boto:Token: None 2023-07-29T00:09:00.142 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/obj2 2023-07-29T00:09:00.142 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xsse4xCb5pViLIxdeYC42rPbflI= 2023-07-29T00:09:00.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xsse4xCb5pViLIxdeYC42rPbflI=', 'Content-Length': '0'} 2023-07-29T00:09:00.144 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001cab9e951bc2b6c6-0064c4589c-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.145 DEBUG:boto:path=/hiiqos-48/obj2 2023-07-29T00:09:00.145 DEBUG:boto:auth_path=/hiiqos-48/obj2 2023-07-29T00:09:00.146 DEBUG:boto:Method: HEAD 2023-07-29T00:09:00.146 DEBUG:boto:Path: /hiiqos-48/obj2 2023-07-29T00:09:00.146 DEBUG:boto:Data: 2023-07-29T00:09:00.147 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.147 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:00.147 DEBUG:boto:Port: 8000 2023-07-29T00:09:00.148 DEBUG:boto:Params: {} 2023-07-29T00:09:00.148 DEBUG:boto:Token: None 2023-07-29T00:09:00.148 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/obj2 2023-07-29T00:09:00.149 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xsse4xCb5pViLIxdeYC42rPbflI= 2023-07-29T00:09:00.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xsse4xCb5pViLIxdeYC42rPbflI=', 'Content-Length': '0'} 2023-07-29T00:09:00.151 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e3a5e2bbb102dc9-0064c4589c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-07-29T00:09:00.152 DEBUG:boto:path=/hiiqos-48/obj2 2023-07-29T00:09:00.152 DEBUG:boto:auth_path=/hiiqos-48/obj2 2023-07-29T00:09:00.153 DEBUG:boto:Method: GET 2023-07-29T00:09:00.153 DEBUG:boto:Path: /hiiqos-48/obj2 2023-07-29T00:09:00.153 DEBUG:boto:Data: 2023-07-29T00:09:00.154 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.154 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:00.154 DEBUG:boto:Port: 8001 2023-07-29T00:09:00.155 DEBUG:boto:Params: {} 2023-07-29T00:09:00.155 DEBUG:boto:Token: None 2023-07-29T00:09:00.155 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/obj2 2023-07-29T00:09:00.156 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GDYqipu7bO8zeen0DHRfecc/iZY= 2023-07-29T00:09:00.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GDYqipu7bO8zeen0DHRfecc/iZY=', 'Content-Length': '0'} 2023-07-29T00:09:00.158 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f99052d2fe392c7-0064c4589c-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.200 DEBUG:boto:path=/hiiqos-48/obj2 2023-07-29T00:09:00.201 DEBUG:boto:auth_path=/hiiqos-48/obj2 2023-07-29T00:09:00.201 DEBUG:boto:Method: GET 2023-07-29T00:09:00.201 DEBUG:boto:Path: /hiiqos-48/obj2 2023-07-29T00:09:00.202 DEBUG:boto:Data: 2023-07-29T00:09:00.202 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.202 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:00.203 DEBUG:boto:Port: 8000 2023-07-29T00:09:00.203 DEBUG:boto:Params: {} 2023-07-29T00:09:00.204 DEBUG:boto:Token: None 2023-07-29T00:09:00.204 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/obj2 2023-07-29T00:09:00.204 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GDYqipu7bO8zeen0DHRfecc/iZY= 2023-07-29T00:09:00.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GDYqipu7bO8zeen0DHRfecc/iZY=', 'Content-Length': '0'} 2023-07-29T00:09:00.206 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cc1762927636d7ae-0064c4589c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.207 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:00.207 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:00.208 DEBUG:boto:path=/hiiqos-48/?prefix=obj2 2023-07-29T00:09:00.208 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj2 2023-07-29T00:09:00.209 DEBUG:boto:Method: GET 2023-07-29T00:09:00.209 DEBUG:boto:Path: /hiiqos-48/?prefix=obj2 2023-07-29T00:09:00.209 DEBUG:boto:Data: 2023-07-29T00:09:00.210 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.210 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:00.210 DEBUG:boto:Port: 8001 2023-07-29T00:09:00.211 DEBUG:boto:Params: {} 2023-07-29T00:09:00.211 DEBUG:boto:Token: None 2023-07-29T00:09:00.211 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/ 2023-07-29T00:09:00.212 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eRChp5Tsa7c/OC+TTBj0T4/dDKo= 2023-07-29T00:09:00.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eRChp5Tsa7c/OC+TTBj0T4/dDKo=', 'Content-Length': '0'} 2023-07-29T00:09:00.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d9b022c80ed1f08-0064c4589c-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.215 DEBUG:boto:b'hiiqos-48obj21000falseobj22023-07-29T00:08:34.078Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:00.216 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:00.216 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:00.217 DEBUG:boto:path=/hiiqos-48/?prefix=obj2 2023-07-29T00:09:00.217 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj2 2023-07-29T00:09:00.217 DEBUG:boto:Method: GET 2023-07-29T00:09:00.218 DEBUG:boto:Path: /hiiqos-48/?prefix=obj2 2023-07-29T00:09:00.218 DEBUG:boto:Data: 2023-07-29T00:09:00.218 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.219 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:00.219 DEBUG:boto:Port: 8000 2023-07-29T00:09:00.219 DEBUG:boto:Params: {} 2023-07-29T00:09:00.220 DEBUG:boto:Token: None 2023-07-29T00:09:00.220 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/ 2023-07-29T00:09:00.221 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eRChp5Tsa7c/OC+TTBj0T4/dDKo= 2023-07-29T00:09:00.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eRChp5Tsa7c/OC+TTBj0T4/dDKo=', 'Content-Length': '0'} 2023-07-29T00:09:00.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a20849cb142ae912-0064c4589c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.224 DEBUG:boto:b'hiiqos-48obj21000falseobj22023-07-29T00:08:34.078Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:00.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-07-29T00:09:00.225 DEBUG:boto:path=/hiiqos-48/obj3 2023-07-29T00:09:00.225 DEBUG:boto:auth_path=/hiiqos-48/obj3 2023-07-29T00:09:00.226 DEBUG:boto:path=/hiiqos-48/obj3?versionId=null 2023-07-29T00:09:00.226 DEBUG:boto:auth_path=/hiiqos-48/obj3?versionId=null 2023-07-29T00:09:00.227 DEBUG:boto:Method: GET 2023-07-29T00:09:00.227 DEBUG:boto:Path: /hiiqos-48/obj3?versionId=null 2023-07-29T00:09:00.227 DEBUG:boto:Data: 2023-07-29T00:09:00.228 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.228 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:00.228 DEBUG:boto:Port: 8001 2023-07-29T00:09:00.229 DEBUG:boto:Params: {} 2023-07-29T00:09:00.229 DEBUG:boto:Token: None 2023-07-29T00:09:00.229 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/obj3?versionId=null 2023-07-29T00:09:00.230 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OjPTNydwoZipyc2b9aW+V4VrxCY= 2023-07-29T00:09:00.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OjPTNydwoZipyc2b9aW+V4VrxCY=', 'Content-Length': '0'} 2023-07-29T00:09:00.232 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000084bdcbad7e18fc5a-0064c4589c-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.233 DEBUG:boto:path=/hiiqos-48/obj3 2023-07-29T00:09:00.233 DEBUG:boto:auth_path=/hiiqos-48/obj3 2023-07-29T00:09:00.234 DEBUG:boto:path=/hiiqos-48/obj3?versionId=null 2023-07-29T00:09:00.234 DEBUG:boto:auth_path=/hiiqos-48/obj3?versionId=null 2023-07-29T00:09:00.234 DEBUG:boto:Method: GET 2023-07-29T00:09:00.235 DEBUG:boto:Path: /hiiqos-48/obj3?versionId=null 2023-07-29T00:09:00.235 DEBUG:boto:Data: 2023-07-29T00:09:00.235 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.236 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:00.236 DEBUG:boto:Port: 8000 2023-07-29T00:09:00.236 DEBUG:boto:Params: {} 2023-07-29T00:09:00.237 DEBUG:boto:Token: None 2023-07-29T00:09:00.237 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/obj3?versionId=null 2023-07-29T00:09:00.238 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OjPTNydwoZipyc2b9aW+V4VrxCY= 2023-07-29T00:09:00.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OjPTNydwoZipyc2b9aW+V4VrxCY=', 'Content-Length': '0'} 2023-07-29T00:09:00.240 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000047ae6a29f8bf7e43-0064c4589c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.241 DEBUG:boto:path=/hiiqos-48/obj3 2023-07-29T00:09:00.241 DEBUG:boto:auth_path=/hiiqos-48/obj3 2023-07-29T00:09:00.241 DEBUG:boto:path=/hiiqos-48/obj3?versionId=null 2023-07-29T00:09:00.242 DEBUG:boto:auth_path=/hiiqos-48/obj3?versionId=null 2023-07-29T00:09:00.242 DEBUG:boto:Method: HEAD 2023-07-29T00:09:00.242 DEBUG:boto:Path: /hiiqos-48/obj3?versionId=null 2023-07-29T00:09:00.243 DEBUG:boto:Data: 2023-07-29T00:09:00.243 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.243 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:00.244 DEBUG:boto:Port: 8001 2023-07-29T00:09:00.244 DEBUG:boto:Params: {} 2023-07-29T00:09:00.244 DEBUG:boto:Token: None 2023-07-29T00:09:00.245 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/obj3?versionId=null 2023-07-29T00:09:00.245 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:q3PutBCoOan/6g935tCBziTNQUU= 2023-07-29T00:09:00.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:q3PutBCoOan/6g935tCBziTNQUU=', 'Content-Length': '0'} 2023-07-29T00:09:00.247 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1b6572fad564c4c-0064c4589c-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.248 DEBUG:boto:path=/hiiqos-48/obj3 2023-07-29T00:09:00.248 DEBUG:boto:auth_path=/hiiqos-48/obj3 2023-07-29T00:09:00.249 DEBUG:boto:path=/hiiqos-48/obj3?versionId=null 2023-07-29T00:09:00.249 DEBUG:boto:auth_path=/hiiqos-48/obj3?versionId=null 2023-07-29T00:09:00.249 DEBUG:boto:Method: HEAD 2023-07-29T00:09:00.250 DEBUG:boto:Path: /hiiqos-48/obj3?versionId=null 2023-07-29T00:09:00.250 DEBUG:boto:Data: 2023-07-29T00:09:00.250 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.251 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:00.251 DEBUG:boto:Port: 8000 2023-07-29T00:09:00.251 DEBUG:boto:Params: {} 2023-07-29T00:09:00.252 DEBUG:boto:Token: None 2023-07-29T00:09:00.252 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/obj3?versionId=null 2023-07-29T00:09:00.253 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:q3PutBCoOan/6g935tCBziTNQUU= 2023-07-29T00:09:00.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:q3PutBCoOan/6g935tCBziTNQUU=', 'Content-Length': '0'} 2023-07-29T00:09:00.254 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000585d11e32b0ff77f-0064c4589c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-07-29T00:09:00.255 DEBUG:boto:path=/hiiqos-48/obj3 2023-07-29T00:09:00.256 DEBUG:boto:auth_path=/hiiqos-48/obj3 2023-07-29T00:09:00.256 DEBUG:boto:path=/hiiqos-48/obj3?versionId=null 2023-07-29T00:09:00.256 DEBUG:boto:auth_path=/hiiqos-48/obj3?versionId=null 2023-07-29T00:09:00.257 DEBUG:boto:Method: GET 2023-07-29T00:09:00.257 DEBUG:boto:Path: /hiiqos-48/obj3?versionId=null 2023-07-29T00:09:00.258 DEBUG:boto:Data: 2023-07-29T00:09:00.258 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.258 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:00.259 DEBUG:boto:Port: 8001 2023-07-29T00:09:00.259 DEBUG:boto:Params: {} 2023-07-29T00:09:00.259 DEBUG:boto:Token: None 2023-07-29T00:09:00.260 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/obj3?versionId=null 2023-07-29T00:09:00.260 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OjPTNydwoZipyc2b9aW+V4VrxCY= 2023-07-29T00:09:00.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OjPTNydwoZipyc2b9aW+V4VrxCY=', 'Content-Length': '0'} 2023-07-29T00:09:00.262 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a872538031c95cf8-0064c4589c-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.263 DEBUG:boto:path=/hiiqos-48/obj3 2023-07-29T00:09:00.263 DEBUG:boto:auth_path=/hiiqos-48/obj3 2023-07-29T00:09:00.264 DEBUG:boto:path=/hiiqos-48/obj3?versionId=null 2023-07-29T00:09:00.264 DEBUG:boto:auth_path=/hiiqos-48/obj3?versionId=null 2023-07-29T00:09:00.264 DEBUG:boto:Method: GET 2023-07-29T00:09:00.265 DEBUG:boto:Path: /hiiqos-48/obj3?versionId=null 2023-07-29T00:09:00.265 DEBUG:boto:Data: 2023-07-29T00:09:00.265 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.266 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:00.266 DEBUG:boto:Port: 8000 2023-07-29T00:09:00.266 DEBUG:boto:Params: {} 2023-07-29T00:09:00.267 DEBUG:boto:Token: None 2023-07-29T00:09:00.267 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/obj3?versionId=null 2023-07-29T00:09:00.268 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OjPTNydwoZipyc2b9aW+V4VrxCY= 2023-07-29T00:09:00.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OjPTNydwoZipyc2b9aW+V4VrxCY=', 'Content-Length': '0'} 2023-07-29T00:09:00.270 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008b798a7582fe8112-0064c4589c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.271 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:00.271 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:00.271 DEBUG:boto:path=/hiiqos-48/?prefix=obj3 2023-07-29T00:09:00.272 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj3 2023-07-29T00:09:00.272 DEBUG:boto:Method: GET 2023-07-29T00:09:00.273 DEBUG:boto:Path: /hiiqos-48/?prefix=obj3 2023-07-29T00:09:00.273 DEBUG:boto:Data: 2023-07-29T00:09:00.273 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.274 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:00.274 DEBUG:boto:Port: 8001 2023-07-29T00:09:00.274 DEBUG:boto:Params: {} 2023-07-29T00:09:00.275 DEBUG:boto:Token: None 2023-07-29T00:09:00.275 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/ 2023-07-29T00:09:00.275 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eRChp5Tsa7c/OC+TTBj0T4/dDKo= 2023-07-29T00:09:00.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eRChp5Tsa7c/OC+TTBj0T4/dDKo=', 'Content-Length': '0'} 2023-07-29T00:09:00.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000570c869d1282a044-0064c4589c-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.279 DEBUG:boto:b'hiiqos-48obj31000falseobj32023-07-29T00:08:34.094Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:00.279 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:00.280 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:00.280 DEBUG:boto:path=/hiiqos-48/?prefix=obj3 2023-07-29T00:09:00.280 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj3 2023-07-29T00:09:00.281 DEBUG:boto:Method: GET 2023-07-29T00:09:00.281 DEBUG:boto:Path: /hiiqos-48/?prefix=obj3 2023-07-29T00:09:00.281 DEBUG:boto:Data: 2023-07-29T00:09:00.282 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.282 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:00.282 DEBUG:boto:Port: 8000 2023-07-29T00:09:00.283 DEBUG:boto:Params: {} 2023-07-29T00:09:00.283 DEBUG:boto:Token: None 2023-07-29T00:09:00.284 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/ 2023-07-29T00:09:00.284 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eRChp5Tsa7c/OC+TTBj0T4/dDKo= 2023-07-29T00:09:00.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eRChp5Tsa7c/OC+TTBj0T4/dDKo=', 'Content-Length': '0'} 2023-07-29T00:09:00.287 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010d6868c24dcdb63-0064c4589c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.287 DEBUG:boto:b'hiiqos-48obj31000falseobj32023-07-29T00:08:34.094Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:00.288 DEBUG:boto:path=/hiiqos-48/obj3 2023-07-29T00:09:00.288 DEBUG:boto:auth_path=/hiiqos-48/obj3 2023-07-29T00:09:00.289 DEBUG:boto:Method: HEAD 2023-07-29T00:09:00.289 DEBUG:boto:Path: /hiiqos-48/obj3 2023-07-29T00:09:00.289 DEBUG:boto:Data: 2023-07-29T00:09:00.290 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.290 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:00.290 DEBUG:boto:Port: 8001 2023-07-29T00:09:00.291 DEBUG:boto:Params: {} 2023-07-29T00:09:00.291 DEBUG:boto:Token: None 2023-07-29T00:09:00.291 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/obj3 2023-07-29T00:09:00.292 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TOkTu98z/B+OW/9dvBGzKQbzOaI= 2023-07-29T00:09:00.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TOkTu98z/B+OW/9dvBGzKQbzOaI=', 'Content-Length': '0'} 2023-07-29T00:09:00.294 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b12952115b00013-0064c4589c-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.294 DEBUG:boto:path=/hiiqos-48/obj3 2023-07-29T00:09:00.295 DEBUG:boto:auth_path=/hiiqos-48/obj3 2023-07-29T00:09:00.295 DEBUG:boto:Method: HEAD 2023-07-29T00:09:00.295 DEBUG:boto:Path: /hiiqos-48/obj3 2023-07-29T00:09:00.296 DEBUG:boto:Data: 2023-07-29T00:09:00.296 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.296 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:00.297 DEBUG:boto:Port: 8000 2023-07-29T00:09:00.297 DEBUG:boto:Params: {} 2023-07-29T00:09:00.297 DEBUG:boto:Token: None 2023-07-29T00:09:00.298 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/obj3 2023-07-29T00:09:00.298 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TOkTu98z/B+OW/9dvBGzKQbzOaI= 2023-07-29T00:09:00.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TOkTu98z/B+OW/9dvBGzKQbzOaI=', 'Content-Length': '0'} 2023-07-29T00:09:00.300 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f401b5236832cfe-0064c4589c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-07-29T00:09:00.301 DEBUG:boto:path=/hiiqos-48/obj3 2023-07-29T00:09:00.301 DEBUG:boto:auth_path=/hiiqos-48/obj3 2023-07-29T00:09:00.302 DEBUG:boto:Method: GET 2023-07-29T00:09:00.302 DEBUG:boto:Path: /hiiqos-48/obj3 2023-07-29T00:09:00.303 DEBUG:boto:Data: 2023-07-29T00:09:00.303 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.303 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:00.304 DEBUG:boto:Port: 8001 2023-07-29T00:09:00.304 DEBUG:boto:Params: {} 2023-07-29T00:09:00.304 DEBUG:boto:Token: None 2023-07-29T00:09:00.305 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/obj3 2023-07-29T00:09:00.305 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8HpIv+M0GQCNjyJnCHvRDbLJhiI= 2023-07-29T00:09:00.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8HpIv+M0GQCNjyJnCHvRDbLJhiI=', 'Content-Length': '0'} 2023-07-29T00:09:00.307 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000021ca6b5ad972fd9d-0064c4589c-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.308 DEBUG:boto:path=/hiiqos-48/obj3 2023-07-29T00:09:00.308 DEBUG:boto:auth_path=/hiiqos-48/obj3 2023-07-29T00:09:00.309 DEBUG:boto:Method: GET 2023-07-29T00:09:00.309 DEBUG:boto:Path: /hiiqos-48/obj3 2023-07-29T00:09:00.309 DEBUG:boto:Data: 2023-07-29T00:09:00.310 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.310 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:00.310 DEBUG:boto:Port: 8000 2023-07-29T00:09:00.311 DEBUG:boto:Params: {} 2023-07-29T00:09:00.311 DEBUG:boto:Token: None 2023-07-29T00:09:00.311 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/obj3 2023-07-29T00:09:00.312 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8HpIv+M0GQCNjyJnCHvRDbLJhiI= 2023-07-29T00:09:00.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8HpIv+M0GQCNjyJnCHvRDbLJhiI=', 'Content-Length': '0'} 2023-07-29T00:09:00.314 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008bf16d4a7eb9121f-0064c4589c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.315 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:00.315 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:00.315 DEBUG:boto:path=/hiiqos-48/?prefix=obj3 2023-07-29T00:09:00.316 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj3 2023-07-29T00:09:00.316 DEBUG:boto:Method: GET 2023-07-29T00:09:00.316 DEBUG:boto:Path: /hiiqos-48/?prefix=obj3 2023-07-29T00:09:00.317 DEBUG:boto:Data: 2023-07-29T00:09:00.317 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.317 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:00.318 DEBUG:boto:Port: 8001 2023-07-29T00:09:00.318 DEBUG:boto:Params: {} 2023-07-29T00:09:00.319 DEBUG:boto:Token: None 2023-07-29T00:09:00.319 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/ 2023-07-29T00:09:00.319 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eRChp5Tsa7c/OC+TTBj0T4/dDKo= 2023-07-29T00:09:00.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eRChp5Tsa7c/OC+TTBj0T4/dDKo=', 'Content-Length': '0'} 2023-07-29T00:09:00.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000551a90d0b1ca2d4e-0064c4589c-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.322 DEBUG:boto:b'hiiqos-48obj31000falseobj32023-07-29T00:08:34.094Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:00.323 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:00.323 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:00.324 DEBUG:boto:path=/hiiqos-48/?prefix=obj3 2023-07-29T00:09:00.324 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj3 2023-07-29T00:09:00.324 DEBUG:boto:Method: GET 2023-07-29T00:09:00.325 DEBUG:boto:Path: /hiiqos-48/?prefix=obj3 2023-07-29T00:09:00.325 DEBUG:boto:Data: 2023-07-29T00:09:00.326 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.326 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:00.326 DEBUG:boto:Port: 8000 2023-07-29T00:09:00.327 DEBUG:boto:Params: {} 2023-07-29T00:09:00.327 DEBUG:boto:Token: None 2023-07-29T00:09:00.327 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/ 2023-07-29T00:09:00.328 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eRChp5Tsa7c/OC+TTBj0T4/dDKo= 2023-07-29T00:09:00.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eRChp5Tsa7c/OC+TTBj0T4/dDKo=', 'Content-Length': '0'} 2023-07-29T00:09:00.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093a401891a33ef95-0064c4589c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.331 DEBUG:boto:b'hiiqos-48obj31000falseobj32023-07-29T00:08:34.094Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:00.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-07-29T00:09:00.332 DEBUG:boto:path=/hiiqos-48/obj4 2023-07-29T00:09:00.332 DEBUG:boto:auth_path=/hiiqos-48/obj4 2023-07-29T00:09:00.333 DEBUG:boto:path=/hiiqos-48/obj4?versionId=null 2023-07-29T00:09:00.333 DEBUG:boto:auth_path=/hiiqos-48/obj4?versionId=null 2023-07-29T00:09:00.333 DEBUG:boto:Method: GET 2023-07-29T00:09:00.334 DEBUG:boto:Path: /hiiqos-48/obj4?versionId=null 2023-07-29T00:09:00.334 DEBUG:boto:Data: 2023-07-29T00:09:00.334 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.335 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:00.335 DEBUG:boto:Port: 8001 2023-07-29T00:09:00.335 DEBUG:boto:Params: {} 2023-07-29T00:09:00.336 DEBUG:boto:Token: None 2023-07-29T00:09:00.336 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/obj4?versionId=null 2023-07-29T00:09:00.337 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0H9bfXHGfQQNmWhKqG6VcnB6jPY= 2023-07-29T00:09:00.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0H9bfXHGfQQNmWhKqG6VcnB6jPY=', 'Content-Length': '0'} 2023-07-29T00:09:00.339 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006688cd9424bfda32-0064c4589c-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.339 DEBUG:boto:path=/hiiqos-48/obj4 2023-07-29T00:09:00.340 DEBUG:boto:auth_path=/hiiqos-48/obj4 2023-07-29T00:09:00.340 DEBUG:boto:path=/hiiqos-48/obj4?versionId=null 2023-07-29T00:09:00.340 DEBUG:boto:auth_path=/hiiqos-48/obj4?versionId=null 2023-07-29T00:09:00.341 DEBUG:boto:Method: GET 2023-07-29T00:09:00.341 DEBUG:boto:Path: /hiiqos-48/obj4?versionId=null 2023-07-29T00:09:00.341 DEBUG:boto:Data: 2023-07-29T00:09:00.342 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.342 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:00.343 DEBUG:boto:Port: 8000 2023-07-29T00:09:00.343 DEBUG:boto:Params: {} 2023-07-29T00:09:00.343 DEBUG:boto:Token: None 2023-07-29T00:09:00.344 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/obj4?versionId=null 2023-07-29T00:09:00.344 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0H9bfXHGfQQNmWhKqG6VcnB6jPY= 2023-07-29T00:09:00.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0H9bfXHGfQQNmWhKqG6VcnB6jPY=', 'Content-Length': '0'} 2023-07-29T00:09:00.346 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000283c349970526d70-0064c4589c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.347 DEBUG:boto:path=/hiiqos-48/obj4 2023-07-29T00:09:00.347 DEBUG:boto:auth_path=/hiiqos-48/obj4 2023-07-29T00:09:00.348 DEBUG:boto:path=/hiiqos-48/obj4?versionId=null 2023-07-29T00:09:00.348 DEBUG:boto:auth_path=/hiiqos-48/obj4?versionId=null 2023-07-29T00:09:00.348 DEBUG:boto:Method: HEAD 2023-07-29T00:09:00.349 DEBUG:boto:Path: /hiiqos-48/obj4?versionId=null 2023-07-29T00:09:00.349 DEBUG:boto:Data: 2023-07-29T00:09:00.349 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.350 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:00.350 DEBUG:boto:Port: 8001 2023-07-29T00:09:00.350 DEBUG:boto:Params: {} 2023-07-29T00:09:00.351 DEBUG:boto:Token: None 2023-07-29T00:09:00.351 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/obj4?versionId=null 2023-07-29T00:09:00.351 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vE5oMYAQsJYq/+sUABJkJKse99U= 2023-07-29T00:09:00.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vE5oMYAQsJYq/+sUABJkJKse99U=', 'Content-Length': '0'} 2023-07-29T00:09:00.354 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000798d3bca686dacf5-0064c4589c-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.354 DEBUG:boto:path=/hiiqos-48/obj4 2023-07-29T00:09:00.355 DEBUG:boto:auth_path=/hiiqos-48/obj4 2023-07-29T00:09:00.355 DEBUG:boto:path=/hiiqos-48/obj4?versionId=null 2023-07-29T00:09:00.355 DEBUG:boto:auth_path=/hiiqos-48/obj4?versionId=null 2023-07-29T00:09:00.356 DEBUG:boto:Method: HEAD 2023-07-29T00:09:00.356 DEBUG:boto:Path: /hiiqos-48/obj4?versionId=null 2023-07-29T00:09:00.356 DEBUG:boto:Data: 2023-07-29T00:09:00.357 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.357 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:00.357 DEBUG:boto:Port: 8000 2023-07-29T00:09:00.358 DEBUG:boto:Params: {} 2023-07-29T00:09:00.358 DEBUG:boto:Token: None 2023-07-29T00:09:00.358 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/obj4?versionId=null 2023-07-29T00:09:00.359 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vE5oMYAQsJYq/+sUABJkJKse99U= 2023-07-29T00:09:00.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vE5oMYAQsJYq/+sUABJkJKse99U=', 'Content-Length': '0'} 2023-07-29T00:09:00.361 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000542449f7e458a697-0064c4589c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-07-29T00:09:00.362 DEBUG:boto:path=/hiiqos-48/obj4 2023-07-29T00:09:00.362 DEBUG:boto:auth_path=/hiiqos-48/obj4 2023-07-29T00:09:00.362 DEBUG:boto:path=/hiiqos-48/obj4?versionId=null 2023-07-29T00:09:00.363 DEBUG:boto:auth_path=/hiiqos-48/obj4?versionId=null 2023-07-29T00:09:00.363 DEBUG:boto:Method: GET 2023-07-29T00:09:00.363 DEBUG:boto:Path: /hiiqos-48/obj4?versionId=null 2023-07-29T00:09:00.364 DEBUG:boto:Data: 2023-07-29T00:09:00.364 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.365 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:00.365 DEBUG:boto:Port: 8001 2023-07-29T00:09:00.365 DEBUG:boto:Params: {} 2023-07-29T00:09:00.366 DEBUG:boto:Token: None 2023-07-29T00:09:00.366 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/obj4?versionId=null 2023-07-29T00:09:00.366 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0H9bfXHGfQQNmWhKqG6VcnB6jPY= 2023-07-29T00:09:00.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0H9bfXHGfQQNmWhKqG6VcnB6jPY=', 'Content-Length': '0'} 2023-07-29T00:09:00.368 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d01ddba4f00a52a8-0064c4589c-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.369 DEBUG:boto:path=/hiiqos-48/obj4 2023-07-29T00:09:00.370 DEBUG:boto:auth_path=/hiiqos-48/obj4 2023-07-29T00:09:00.370 DEBUG:boto:path=/hiiqos-48/obj4?versionId=null 2023-07-29T00:09:00.370 DEBUG:boto:auth_path=/hiiqos-48/obj4?versionId=null 2023-07-29T00:09:00.371 DEBUG:boto:Method: GET 2023-07-29T00:09:00.371 DEBUG:boto:Path: /hiiqos-48/obj4?versionId=null 2023-07-29T00:09:00.371 DEBUG:boto:Data: 2023-07-29T00:09:00.372 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.372 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:00.372 DEBUG:boto:Port: 8000 2023-07-29T00:09:00.373 DEBUG:boto:Params: {} 2023-07-29T00:09:00.373 DEBUG:boto:Token: None 2023-07-29T00:09:00.373 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/obj4?versionId=null 2023-07-29T00:09:00.374 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0H9bfXHGfQQNmWhKqG6VcnB6jPY= 2023-07-29T00:09:00.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0H9bfXHGfQQNmWhKqG6VcnB6jPY=', 'Content-Length': '0'} 2023-07-29T00:09:00.376 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000092e082b0d7c8bc5d-0064c4589c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.377 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:00.377 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:00.377 DEBUG:boto:path=/hiiqos-48/?prefix=obj4 2023-07-29T00:09:00.378 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj4 2023-07-29T00:09:00.378 DEBUG:boto:Method: GET 2023-07-29T00:09:00.379 DEBUG:boto:Path: /hiiqos-48/?prefix=obj4 2023-07-29T00:09:00.379 DEBUG:boto:Data: 2023-07-29T00:09:00.379 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.380 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:00.380 DEBUG:boto:Port: 8001 2023-07-29T00:09:00.380 DEBUG:boto:Params: {} 2023-07-29T00:09:00.381 DEBUG:boto:Token: None 2023-07-29T00:09:00.381 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/ 2023-07-29T00:09:00.382 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eRChp5Tsa7c/OC+TTBj0T4/dDKo= 2023-07-29T00:09:00.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eRChp5Tsa7c/OC+TTBj0T4/dDKo=', 'Content-Length': '0'} 2023-07-29T00:09:00.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040b7d50ec72b4aab-0064c4589c-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.432 DEBUG:boto:b'hiiqos-48obj41000falseobj42023-07-29T00:08:34.109Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:00.433 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:00.433 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:00.434 DEBUG:boto:path=/hiiqos-48/?prefix=obj4 2023-07-29T00:09:00.434 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj4 2023-07-29T00:09:00.434 DEBUG:boto:Method: GET 2023-07-29T00:09:00.435 DEBUG:boto:Path: /hiiqos-48/?prefix=obj4 2023-07-29T00:09:00.435 DEBUG:boto:Data: 2023-07-29T00:09:00.436 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.436 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:00.436 DEBUG:boto:Port: 8000 2023-07-29T00:09:00.437 DEBUG:boto:Params: {} 2023-07-29T00:09:00.437 DEBUG:boto:Token: None 2023-07-29T00:09:00.437 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/ 2023-07-29T00:09:00.438 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eRChp5Tsa7c/OC+TTBj0T4/dDKo= 2023-07-29T00:09:00.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eRChp5Tsa7c/OC+TTBj0T4/dDKo=', 'Content-Length': '0'} 2023-07-29T00:09:00.441 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008176f5d54c311d4-0064c4589c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.441 DEBUG:boto:b'hiiqos-48obj41000falseobj42023-07-29T00:08:34.109Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:00.442 DEBUG:boto:path=/hiiqos-48/obj4 2023-07-29T00:09:00.442 DEBUG:boto:auth_path=/hiiqos-48/obj4 2023-07-29T00:09:00.443 DEBUG:boto:Method: HEAD 2023-07-29T00:09:00.443 DEBUG:boto:Path: /hiiqos-48/obj4 2023-07-29T00:09:00.443 DEBUG:boto:Data: 2023-07-29T00:09:00.444 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.444 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:00.444 DEBUG:boto:Port: 8001 2023-07-29T00:09:00.445 DEBUG:boto:Params: {} 2023-07-29T00:09:00.445 DEBUG:boto:Token: None 2023-07-29T00:09:00.445 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/obj4 2023-07-29T00:09:00.446 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8aBPyI0gSatlT8o/1bJgTQN3qEc= 2023-07-29T00:09:00.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8aBPyI0gSatlT8o/1bJgTQN3qEc=', 'Content-Length': '0'} 2023-07-29T00:09:00.448 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c6cfb255c04ea9a1-0064c4589c-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.448 DEBUG:boto:path=/hiiqos-48/obj4 2023-07-29T00:09:00.449 DEBUG:boto:auth_path=/hiiqos-48/obj4 2023-07-29T00:09:00.449 DEBUG:boto:Method: HEAD 2023-07-29T00:09:00.450 DEBUG:boto:Path: /hiiqos-48/obj4 2023-07-29T00:09:00.450 DEBUG:boto:Data: 2023-07-29T00:09:00.450 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.451 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:00.451 DEBUG:boto:Port: 8000 2023-07-29T00:09:00.451 DEBUG:boto:Params: {} 2023-07-29T00:09:00.452 DEBUG:boto:Token: None 2023-07-29T00:09:00.452 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/obj4 2023-07-29T00:09:00.452 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8aBPyI0gSatlT8o/1bJgTQN3qEc= 2023-07-29T00:09:00.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8aBPyI0gSatlT8o/1bJgTQN3qEc=', 'Content-Length': '0'} 2023-07-29T00:09:00.454 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b8e36e77d3482186-0064c4589c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-07-29T00:09:00.455 DEBUG:boto:path=/hiiqos-48/obj4 2023-07-29T00:09:00.456 DEBUG:boto:auth_path=/hiiqos-48/obj4 2023-07-29T00:09:00.456 DEBUG:boto:Method: GET 2023-07-29T00:09:00.456 DEBUG:boto:Path: /hiiqos-48/obj4 2023-07-29T00:09:00.457 DEBUG:boto:Data: 2023-07-29T00:09:00.457 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.458 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:00.458 DEBUG:boto:Port: 8001 2023-07-29T00:09:00.458 DEBUG:boto:Params: {} 2023-07-29T00:09:00.459 DEBUG:boto:Token: None 2023-07-29T00:09:00.459 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/obj4 2023-07-29T00:09:00.459 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:D3HEsv8Ds9KmuK63pn38V+KnoF8= 2023-07-29T00:09:00.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:D3HEsv8Ds9KmuK63pn38V+KnoF8=', 'Content-Length': '0'} 2023-07-29T00:09:00.462 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000117392a02f7095cc-0064c4589c-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.462 DEBUG:boto:path=/hiiqos-48/obj4 2023-07-29T00:09:00.463 DEBUG:boto:auth_path=/hiiqos-48/obj4 2023-07-29T00:09:00.463 DEBUG:boto:Method: GET 2023-07-29T00:09:00.463 DEBUG:boto:Path: /hiiqos-48/obj4 2023-07-29T00:09:00.464 DEBUG:boto:Data: 2023-07-29T00:09:00.464 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.464 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:00.465 DEBUG:boto:Port: 8000 2023-07-29T00:09:00.465 DEBUG:boto:Params: {} 2023-07-29T00:09:00.465 DEBUG:boto:Token: None 2023-07-29T00:09:00.466 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/obj4 2023-07-29T00:09:00.466 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:D3HEsv8Ds9KmuK63pn38V+KnoF8= 2023-07-29T00:09:00.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:D3HEsv8Ds9KmuK63pn38V+KnoF8=', 'Content-Length': '0'} 2023-07-29T00:09:00.469 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c2605fbf80f038c-0064c4589c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.512 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:00.513 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:00.513 DEBUG:boto:path=/hiiqos-48/?prefix=obj4 2023-07-29T00:09:00.513 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj4 2023-07-29T00:09:00.514 DEBUG:boto:Method: GET 2023-07-29T00:09:00.514 DEBUG:boto:Path: /hiiqos-48/?prefix=obj4 2023-07-29T00:09:00.514 DEBUG:boto:Data: 2023-07-29T00:09:00.515 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.515 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:00.515 DEBUG:boto:Port: 8001 2023-07-29T00:09:00.516 DEBUG:boto:Params: {} 2023-07-29T00:09:00.516 DEBUG:boto:Token: None 2023-07-29T00:09:00.517 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/ 2023-07-29T00:09:00.517 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eRChp5Tsa7c/OC+TTBj0T4/dDKo= 2023-07-29T00:09:00.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eRChp5Tsa7c/OC+TTBj0T4/dDKo=', 'Content-Length': '0'} 2023-07-29T00:09:00.520 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000196c2ad1b331049-0064c4589c-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.520 DEBUG:boto:b'hiiqos-48obj41000falseobj42023-07-29T00:08:34.109Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:00.521 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:00.521 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:00.522 DEBUG:boto:path=/hiiqos-48/?prefix=obj4 2023-07-29T00:09:00.522 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj4 2023-07-29T00:09:00.523 DEBUG:boto:Method: GET 2023-07-29T00:09:00.523 DEBUG:boto:Path: /hiiqos-48/?prefix=obj4 2023-07-29T00:09:00.523 DEBUG:boto:Data: 2023-07-29T00:09:00.524 DEBUG:boto:Headers: {} 2023-07-29T00:09:00.524 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:00.524 DEBUG:boto:Port: 8000 2023-07-29T00:09:00.525 DEBUG:boto:Params: {} 2023-07-29T00:09:00.525 DEBUG:boto:Token: None 2023-07-29T00:09:00.525 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:00 GMT /hiiqos-48/ 2023-07-29T00:09:00.526 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eRChp5Tsa7c/OC+TTBj0T4/dDKo= 2023-07-29T00:09:00.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eRChp5Tsa7c/OC+TTBj0T4/dDKo=', 'Content-Length': '0'} 2023-07-29T00:09:00.529 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddc4e867dddbda07-0064c4589c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:00.529 DEBUG:boto:b'hiiqos-48obj41000falseobj42023-07-29T00:08:34.109Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:00.530 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-48 zones={a2, a1} 2023-07-29T00:09:00.530 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-49 2023-07-29T00:09:00.531 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:09:00.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:09:00.532 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-49 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:09:00.591 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:09:00.591 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:09:00.627 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:09:00.597+0000 7fc00bd4b880 1 waiting to reach incremental sync.. 2023-07-29T00:09:05.628 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:09:05.598+0000 7fc00bd4b880 1 waiting to reach incremental sync.. 2023-07-29T00:09:10.629 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:09:10.598+0000 7fc00bd4b880 1 waiting to reach incremental sync.. 2023-07-29T00:09:15.631 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:09:15.600+0000 7fc00bd4b880 1 bucket sync caught up with source: 2023-07-29T00:09:15.632 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000001.34.6, , 00000000001.47.6, , 00000000001.29.6, , , 00000000001.17.6, , , ] 2023-07-29T00:09:15.632 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000001.34.6, , 00000000001.47.6, , 00000000001.29.6, , , 00000000001.17.6, , , ] 2023-07-29T00:09:15.633 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:09:15.600+0000 7fc00bd4b880 0 bucket checkpoint complete 2023-07-29T00:09:15.649 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-49 2023-07-29T00:09:15.650 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:09:15.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:09:15.651 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-49 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:09:15.709 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:09:15.709 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:09:15.745 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:09:15.714+0000 7f1f0a724880 1 bucket sync caught up with source: 2023-07-29T00:09:15.745 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000001.536.6, , 00000000001.384.6, , 00000000001.1667.6, , , 00000000001.675.6, , , ] 2023-07-29T00:09:15.746 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000001.536.6, , 00000000001.384.6, , 00000000001.1667.6, , , 00000000001.675.6, , , ] 2023-07-29T00:09:15.746 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:09:15.714+0000 7f1f0a724880 0 bucket checkpoint complete 2023-07-29T00:09:15.759 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-49 zones={a2, a1} 2023-07-29T00:09:15.760 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:09:15.761 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:09:15.761 DEBUG:boto:Method: HEAD 2023-07-29T00:09:15.762 DEBUG:boto:Path: /hiiqos-49/ 2023-07-29T00:09:15.763 DEBUG:boto:Data: 2023-07-29T00:09:15.764 DEBUG:boto:Headers: {} 2023-07-29T00:09:15.765 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:15.766 DEBUG:boto:Port: 8001 2023-07-29T00:09:15.766 DEBUG:boto:Params: {} 2023-07-29T00:09:15.767 DEBUG:boto:Token: None 2023-07-29T00:09:15.768 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:15 GMT /hiiqos-49/ 2023-07-29T00:09:15.768 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sJc0iZapMUPMy1HuHVbAh+Q80/I= 2023-07-29T00:09:15.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sJc0iZapMUPMy1HuHVbAh+Q80/I=', 'Content-Length': '0'} 2023-07-29T00:09:15.771 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000047f348d8ca9b287a-0064c458ab-1663-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:09:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:15.772 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:09:15.772 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:09:15.773 DEBUG:boto:Method: HEAD 2023-07-29T00:09:15.773 DEBUG:boto:Path: /hiiqos-49/ 2023-07-29T00:09:15.773 DEBUG:boto:Data: 2023-07-29T00:09:15.774 DEBUG:boto:Headers: {} 2023-07-29T00:09:15.774 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:15.775 DEBUG:boto:Port: 8000 2023-07-29T00:09:15.775 DEBUG:boto:Params: {} 2023-07-29T00:09:15.775 DEBUG:boto:Token: None 2023-07-29T00:09:15.776 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:15 GMT /hiiqos-49/ 2023-07-29T00:09:15.777 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sJc0iZapMUPMy1HuHVbAh+Q80/I= 2023-07-29T00:09:15.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sJc0iZapMUPMy1HuHVbAh+Q80/I=', 'Content-Length': '0'} 2023-07-29T00:09:15.779 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000094efdf0ecc7d8ba6-0064c458ab-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:09:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:15.780 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-29T00:09:15.780 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:09:15.780 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:09:15.781 DEBUG:boto:path=/hiiqos-49/?versions&max-keys=999 2023-07-29T00:09:15.781 DEBUG:boto:auth_path=/hiiqos-49/?versions&max-keys=999 2023-07-29T00:09:15.782 DEBUG:boto:Method: GET 2023-07-29T00:09:15.782 DEBUG:boto:Path: /hiiqos-49/?versions&max-keys=999 2023-07-29T00:09:15.782 DEBUG:boto:Data: 2023-07-29T00:09:15.783 DEBUG:boto:Headers: {} 2023-07-29T00:09:15.783 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:15.784 DEBUG:boto:Port: 8001 2023-07-29T00:09:15.784 DEBUG:boto:Params: {} 2023-07-29T00:09:15.785 DEBUG:boto:Token: None 2023-07-29T00:09:15.785 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:15 GMT /hiiqos-49/?versions 2023-07-29T00:09:15.786 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JeSpWgLY3AJEbklhjrX7r2oQga0= 2023-07-29T00:09:15.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JeSpWgLY3AJEbklhjrX7r2oQga0=', 'Content-Length': '0'} 2023-07-29T00:09:15.788 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e690c83a3c534d66-0064c458ab-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:15.789 DEBUG:boto:b'hiiqos-49999falseobj1nulltrue2023-07-29T00:08:34.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-07-29T00:08:34.191Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-07-29T00:08:34.246Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-07-29T00:08:34.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:15.790 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-07-29T00:09:15.790 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-07-29T00:09:15.791 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-07-29T00:09:15.791 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-07-29T00:09:15.791 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-29T00:09:15.792 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:09:15.792 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:09:15.793 DEBUG:boto:path=/hiiqos-49/?versions&max-keys=999 2023-07-29T00:09:15.793 DEBUG:boto:auth_path=/hiiqos-49/?versions&max-keys=999 2023-07-29T00:09:15.794 DEBUG:boto:Method: GET 2023-07-29T00:09:15.794 DEBUG:boto:Path: /hiiqos-49/?versions&max-keys=999 2023-07-29T00:09:15.794 DEBUG:boto:Data: 2023-07-29T00:09:15.795 DEBUG:boto:Headers: {} 2023-07-29T00:09:15.795 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:15.795 DEBUG:boto:Port: 8000 2023-07-29T00:09:15.796 DEBUG:boto:Params: {} 2023-07-29T00:09:15.796 DEBUG:boto:Token: None 2023-07-29T00:09:15.797 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:15 GMT /hiiqos-49/?versions 2023-07-29T00:09:15.797 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JeSpWgLY3AJEbklhjrX7r2oQga0= 2023-07-29T00:09:15.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JeSpWgLY3AJEbklhjrX7r2oQga0=', 'Content-Length': '0'} 2023-07-29T00:09:15.800 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3b54b12d7d58414-0064c458ab-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:15.801 DEBUG:boto:b'hiiqos-49999falseobj1nulltrue2023-07-29T00:08:34.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-07-29T00:08:34.191Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-07-29T00:08:34.246Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-07-29T00:08:34.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:15.803 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-07-29T00:09:15.804 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-07-29T00:09:15.804 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-07-29T00:09:15.805 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-07-29T00:09:15.805 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:09:15.806 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:09:15.806 DEBUG:boto:path=/hiiqos-49/?versions&max-keys=999 2023-07-29T00:09:15.806 DEBUG:boto:auth_path=/hiiqos-49/?versions&max-keys=999 2023-07-29T00:09:15.807 DEBUG:boto:Method: GET 2023-07-29T00:09:15.807 DEBUG:boto:Path: /hiiqos-49/?versions&max-keys=999 2023-07-29T00:09:15.807 DEBUG:boto:Data: 2023-07-29T00:09:15.808 DEBUG:boto:Headers: {} 2023-07-29T00:09:15.809 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:15.809 DEBUG:boto:Port: 8001 2023-07-29T00:09:15.809 DEBUG:boto:Params: {} 2023-07-29T00:09:15.810 DEBUG:boto:Token: None 2023-07-29T00:09:15.810 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:15 GMT /hiiqos-49/?versions 2023-07-29T00:09:15.811 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JeSpWgLY3AJEbklhjrX7r2oQga0= 2023-07-29T00:09:15.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JeSpWgLY3AJEbklhjrX7r2oQga0=', 'Content-Length': '0'} 2023-07-29T00:09:15.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c363e8d59b09e71f-0064c458ab-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:15.814 DEBUG:boto:b'hiiqos-49999falseobj1nulltrue2023-07-29T00:08:34.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-07-29T00:08:34.191Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-07-29T00:08:34.246Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-07-29T00:08:34.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:15.815 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:09:15.815 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:09:15.816 DEBUG:boto:path=/hiiqos-49/?versions&max-keys=999 2023-07-29T00:09:15.816 DEBUG:boto:auth_path=/hiiqos-49/?versions&max-keys=999 2023-07-29T00:09:15.816 DEBUG:boto:Method: GET 2023-07-29T00:09:15.817 DEBUG:boto:Path: /hiiqos-49/?versions&max-keys=999 2023-07-29T00:09:15.817 DEBUG:boto:Data: 2023-07-29T00:09:15.818 DEBUG:boto:Headers: {} 2023-07-29T00:09:15.818 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:15.818 DEBUG:boto:Port: 8000 2023-07-29T00:09:15.819 DEBUG:boto:Params: {} 2023-07-29T00:09:15.819 DEBUG:boto:Token: None 2023-07-29T00:09:15.820 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:15 GMT /hiiqos-49/?versions 2023-07-29T00:09:15.820 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JeSpWgLY3AJEbklhjrX7r2oQga0= 2023-07-29T00:09:15.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JeSpWgLY3AJEbklhjrX7r2oQga0=', 'Content-Length': '0'} 2023-07-29T00:09:15.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000235514503ffd76a9-0064c458ab-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:15.823 DEBUG:boto:b'hiiqos-49999falseobj1nulltrue2023-07-29T00:08:34.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-07-29T00:08:34.191Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-07-29T00:08:34.246Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-07-29T00:08:34.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:15.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-07-29T00:09:15.825 DEBUG:boto:path=/hiiqos-49/obj1 2023-07-29T00:09:15.825 DEBUG:boto:auth_path=/hiiqos-49/obj1 2023-07-29T00:09:15.825 DEBUG:boto:path=/hiiqos-49/obj1?versionId=null 2023-07-29T00:09:15.826 DEBUG:boto:auth_path=/hiiqos-49/obj1?versionId=null 2023-07-29T00:09:15.826 DEBUG:boto:Method: GET 2023-07-29T00:09:15.826 DEBUG:boto:Path: /hiiqos-49/obj1?versionId=null 2023-07-29T00:09:15.827 DEBUG:boto:Data: 2023-07-29T00:09:15.827 DEBUG:boto:Headers: {} 2023-07-29T00:09:15.828 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:15.828 DEBUG:boto:Port: 8001 2023-07-29T00:09:15.828 DEBUG:boto:Params: {} 2023-07-29T00:09:15.829 DEBUG:boto:Token: None 2023-07-29T00:09:15.829 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:15 GMT /hiiqos-49/obj1?versionId=null 2023-07-29T00:09:15.830 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Q/FTXNUK8Cg6ShGMzaqX6jfXUMA= 2023-07-29T00:09:15.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q/FTXNUK8Cg6ShGMzaqX6jfXUMA=', 'Content-Length': '0'} 2023-07-29T00:09:15.832 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014539439a5029b9e-0064c458ab-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:15.872 DEBUG:boto:path=/hiiqos-49/obj1 2023-07-29T00:09:15.873 DEBUG:boto:auth_path=/hiiqos-49/obj1 2023-07-29T00:09:15.873 DEBUG:boto:path=/hiiqos-49/obj1?versionId=null 2023-07-29T00:09:15.874 DEBUG:boto:auth_path=/hiiqos-49/obj1?versionId=null 2023-07-29T00:09:15.874 DEBUG:boto:Method: GET 2023-07-29T00:09:15.875 DEBUG:boto:Path: /hiiqos-49/obj1?versionId=null 2023-07-29T00:09:15.875 DEBUG:boto:Data: 2023-07-29T00:09:15.875 DEBUG:boto:Headers: {} 2023-07-29T00:09:15.876 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:15.876 DEBUG:boto:Port: 8000 2023-07-29T00:09:15.877 DEBUG:boto:Params: {} 2023-07-29T00:09:15.878 DEBUG:boto:Token: None 2023-07-29T00:09:15.879 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:15 GMT /hiiqos-49/obj1?versionId=null 2023-07-29T00:09:15.880 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Q/FTXNUK8Cg6ShGMzaqX6jfXUMA= 2023-07-29T00:09:15.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q/FTXNUK8Cg6ShGMzaqX6jfXUMA=', 'Content-Length': '0'} 2023-07-29T00:09:15.883 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d5c0a1e05c881823-0064c458ab-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:15.884 DEBUG:boto:path=/hiiqos-49/obj1 2023-07-29T00:09:15.884 DEBUG:boto:auth_path=/hiiqos-49/obj1 2023-07-29T00:09:15.884 DEBUG:boto:path=/hiiqos-49/obj1?versionId=null 2023-07-29T00:09:15.885 DEBUG:boto:auth_path=/hiiqos-49/obj1?versionId=null 2023-07-29T00:09:15.885 DEBUG:boto:Method: HEAD 2023-07-29T00:09:15.886 DEBUG:boto:Path: /hiiqos-49/obj1?versionId=null 2023-07-29T00:09:15.886 DEBUG:boto:Data: 2023-07-29T00:09:15.886 DEBUG:boto:Headers: {} 2023-07-29T00:09:15.887 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:15.887 DEBUG:boto:Port: 8001 2023-07-29T00:09:15.888 DEBUG:boto:Params: {} 2023-07-29T00:09:15.888 DEBUG:boto:Token: None 2023-07-29T00:09:15.889 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:15 GMT /hiiqos-49/obj1?versionId=null 2023-07-29T00:09:15.889 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JNAqdl69ksr5UDH1DlXgqXl4ShY= 2023-07-29T00:09:15.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JNAqdl69ksr5UDH1DlXgqXl4ShY=', 'Content-Length': '0'} 2023-07-29T00:09:15.891 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000094cf0540f38a2bf8-0064c458ab-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:15.892 DEBUG:boto:path=/hiiqos-49/obj1 2023-07-29T00:09:15.892 DEBUG:boto:auth_path=/hiiqos-49/obj1 2023-07-29T00:09:15.893 DEBUG:boto:path=/hiiqos-49/obj1?versionId=null 2023-07-29T00:09:15.893 DEBUG:boto:auth_path=/hiiqos-49/obj1?versionId=null 2023-07-29T00:09:15.893 DEBUG:boto:Method: HEAD 2023-07-29T00:09:15.894 DEBUG:boto:Path: /hiiqos-49/obj1?versionId=null 2023-07-29T00:09:15.894 DEBUG:boto:Data: 2023-07-29T00:09:15.895 DEBUG:boto:Headers: {} 2023-07-29T00:09:15.895 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:15.895 DEBUG:boto:Port: 8000 2023-07-29T00:09:15.896 DEBUG:boto:Params: {} 2023-07-29T00:09:15.896 DEBUG:boto:Token: None 2023-07-29T00:09:15.897 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:15 GMT /hiiqos-49/obj1?versionId=null 2023-07-29T00:09:15.897 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JNAqdl69ksr5UDH1DlXgqXl4ShY= 2023-07-29T00:09:15.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JNAqdl69ksr5UDH1DlXgqXl4ShY=', 'Content-Length': '0'} 2023-07-29T00:09:15.899 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d297efd89faa8695-0064c458ab-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:15.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-07-29T00:09:15.900 DEBUG:boto:path=/hiiqos-49/obj1 2023-07-29T00:09:15.900 DEBUG:boto:auth_path=/hiiqos-49/obj1 2023-07-29T00:09:15.901 DEBUG:boto:path=/hiiqos-49/obj1?versionId=null 2023-07-29T00:09:15.901 DEBUG:boto:auth_path=/hiiqos-49/obj1?versionId=null 2023-07-29T00:09:15.902 DEBUG:boto:Method: GET 2023-07-29T00:09:15.902 DEBUG:boto:Path: /hiiqos-49/obj1?versionId=null 2023-07-29T00:09:15.902 DEBUG:boto:Data: 2023-07-29T00:09:15.903 DEBUG:boto:Headers: {} 2023-07-29T00:09:15.903 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:15.904 DEBUG:boto:Port: 8001 2023-07-29T00:09:15.904 DEBUG:boto:Params: {} 2023-07-29T00:09:15.905 DEBUG:boto:Token: None 2023-07-29T00:09:15.905 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:15 GMT /hiiqos-49/obj1?versionId=null 2023-07-29T00:09:15.906 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Q/FTXNUK8Cg6ShGMzaqX6jfXUMA= 2023-07-29T00:09:15.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q/FTXNUK8Cg6ShGMzaqX6jfXUMA=', 'Content-Length': '0'} 2023-07-29T00:09:15.909 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000de2762485eb0f6d1-0064c458ab-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:15.910 DEBUG:boto:path=/hiiqos-49/obj1 2023-07-29T00:09:15.910 DEBUG:boto:auth_path=/hiiqos-49/obj1 2023-07-29T00:09:15.910 DEBUG:boto:path=/hiiqos-49/obj1?versionId=null 2023-07-29T00:09:15.911 DEBUG:boto:auth_path=/hiiqos-49/obj1?versionId=null 2023-07-29T00:09:15.911 DEBUG:boto:Method: GET 2023-07-29T00:09:15.911 DEBUG:boto:Path: /hiiqos-49/obj1?versionId=null 2023-07-29T00:09:15.912 DEBUG:boto:Data: 2023-07-29T00:09:15.912 DEBUG:boto:Headers: {} 2023-07-29T00:09:15.913 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:15.913 DEBUG:boto:Port: 8000 2023-07-29T00:09:15.913 DEBUG:boto:Params: {} 2023-07-29T00:09:15.914 DEBUG:boto:Token: None 2023-07-29T00:09:15.914 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:15 GMT /hiiqos-49/obj1?versionId=null 2023-07-29T00:09:15.915 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Q/FTXNUK8Cg6ShGMzaqX6jfXUMA= 2023-07-29T00:09:15.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q/FTXNUK8Cg6ShGMzaqX6jfXUMA=', 'Content-Length': '0'} 2023-07-29T00:09:15.917 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000e304115cb416290-0064c458ab-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:15.917 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:09:15.918 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:09:15.918 DEBUG:boto:path=/hiiqos-49/?prefix=obj1 2023-07-29T00:09:15.919 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj1 2023-07-29T00:09:15.920 DEBUG:boto:Method: GET 2023-07-29T00:09:15.921 DEBUG:boto:Path: /hiiqos-49/?prefix=obj1 2023-07-29T00:09:15.922 DEBUG:boto:Data: 2023-07-29T00:09:15.922 DEBUG:boto:Headers: {} 2023-07-29T00:09:15.923 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:15.924 DEBUG:boto:Port: 8001 2023-07-29T00:09:15.925 DEBUG:boto:Params: {} 2023-07-29T00:09:15.926 DEBUG:boto:Token: None 2023-07-29T00:09:15.926 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:15 GMT /hiiqos-49/ 2023-07-29T00:09:15.927 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BDLHZjNnjPP2MMxwUCAkbpAcWwo= 2023-07-29T00:09:15.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BDLHZjNnjPP2MMxwUCAkbpAcWwo=', 'Content-Length': '0'} 2023-07-29T00:09:15.929 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ffd1c79e4be6461-0064c458ab-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:15.930 DEBUG:boto:b'hiiqos-49obj11000falseobj12023-07-29T00:08:34.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:15.931 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:09:15.931 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:09:15.931 DEBUG:boto:path=/hiiqos-49/?prefix=obj1 2023-07-29T00:09:15.932 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj1 2023-07-29T00:09:15.932 DEBUG:boto:Method: GET 2023-07-29T00:09:15.933 DEBUG:boto:Path: /hiiqos-49/?prefix=obj1 2023-07-29T00:09:15.933 DEBUG:boto:Data: 2023-07-29T00:09:15.934 DEBUG:boto:Headers: {} 2023-07-29T00:09:15.934 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:15.934 DEBUG:boto:Port: 8000 2023-07-29T00:09:15.935 DEBUG:boto:Params: {} 2023-07-29T00:09:15.935 DEBUG:boto:Token: None 2023-07-29T00:09:15.936 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:15 GMT /hiiqos-49/ 2023-07-29T00:09:15.936 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BDLHZjNnjPP2MMxwUCAkbpAcWwo= 2023-07-29T00:09:15.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BDLHZjNnjPP2MMxwUCAkbpAcWwo=', 'Content-Length': '0'} 2023-07-29T00:09:15.939 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f4b6a41a32de56c-0064c458ab-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:15.980 DEBUG:boto:b'hiiqos-49obj11000falseobj12023-07-29T00:08:34.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:15.981 DEBUG:boto:path=/hiiqos-49/obj1 2023-07-29T00:09:15.981 DEBUG:boto:auth_path=/hiiqos-49/obj1 2023-07-29T00:09:15.982 DEBUG:boto:Method: HEAD 2023-07-29T00:09:15.982 DEBUG:boto:Path: /hiiqos-49/obj1 2023-07-29T00:09:15.983 DEBUG:boto:Data: 2023-07-29T00:09:15.983 DEBUG:boto:Headers: {} 2023-07-29T00:09:15.983 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:15.984 DEBUG:boto:Port: 8001 2023-07-29T00:09:15.984 DEBUG:boto:Params: {} 2023-07-29T00:09:15.984 DEBUG:boto:Token: None 2023-07-29T00:09:15.985 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:15 GMT /hiiqos-49/obj1 2023-07-29T00:09:15.985 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/rtMqyojoQnQo4cUuNXWJ7TbXsc= 2023-07-29T00:09:15.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/rtMqyojoQnQo4cUuNXWJ7TbXsc=', 'Content-Length': '0'} 2023-07-29T00:09:15.987 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bd6d9ae4d7f4c4b0-0064c458ab-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:15.988 DEBUG:boto:path=/hiiqos-49/obj1 2023-07-29T00:09:15.989 DEBUG:boto:auth_path=/hiiqos-49/obj1 2023-07-29T00:09:15.989 DEBUG:boto:Method: HEAD 2023-07-29T00:09:15.989 DEBUG:boto:Path: /hiiqos-49/obj1 2023-07-29T00:09:15.990 DEBUG:boto:Data: 2023-07-29T00:09:15.990 DEBUG:boto:Headers: {} 2023-07-29T00:09:15.991 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:15.991 DEBUG:boto:Port: 8000 2023-07-29T00:09:15.991 DEBUG:boto:Params: {} 2023-07-29T00:09:15.992 DEBUG:boto:Token: None 2023-07-29T00:09:15.992 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:15 GMT /hiiqos-49/obj1 2023-07-29T00:09:15.993 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/rtMqyojoQnQo4cUuNXWJ7TbXsc= 2023-07-29T00:09:15.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/rtMqyojoQnQo4cUuNXWJ7TbXsc=', 'Content-Length': '0'} 2023-07-29T00:09:15.995 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c303e80e5ee39691-0064c458ab-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:15.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-07-29T00:09:15.997 DEBUG:boto:path=/hiiqos-49/obj1 2023-07-29T00:09:15.998 DEBUG:boto:auth_path=/hiiqos-49/obj1 2023-07-29T00:09:15.998 DEBUG:boto:Method: GET 2023-07-29T00:09:15.999 DEBUG:boto:Path: /hiiqos-49/obj1 2023-07-29T00:09:16.000 DEBUG:boto:Data: 2023-07-29T00:09:16.000 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.001 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:16.001 DEBUG:boto:Port: 8001 2023-07-29T00:09:16.002 DEBUG:boto:Params: {} 2023-07-29T00:09:16.003 DEBUG:boto:Token: None 2023-07-29T00:09:16.003 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/obj1 2023-07-29T00:09:16.004 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XE+Nt3Ig94v2NOmX3ZjM+AlPDQs= 2023-07-29T00:09:16.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XE+Nt3Ig94v2NOmX3ZjM+AlPDQs=', 'Content-Length': '0'} 2023-07-29T00:09:16.007 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000715bb331b135dd53-0064c458ab-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.008 DEBUG:boto:path=/hiiqos-49/obj1 2023-07-29T00:09:16.008 DEBUG:boto:auth_path=/hiiqos-49/obj1 2023-07-29T00:09:16.009 DEBUG:boto:Method: GET 2023-07-29T00:09:16.009 DEBUG:boto:Path: /hiiqos-49/obj1 2023-07-29T00:09:16.010 DEBUG:boto:Data: 2023-07-29T00:09:16.010 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.010 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:16.011 DEBUG:boto:Port: 8000 2023-07-29T00:09:16.011 DEBUG:boto:Params: {} 2023-07-29T00:09:16.012 DEBUG:boto:Token: None 2023-07-29T00:09:16.013 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/obj1 2023-07-29T00:09:16.013 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XE+Nt3Ig94v2NOmX3ZjM+AlPDQs= 2023-07-29T00:09:16.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XE+Nt3Ig94v2NOmX3ZjM+AlPDQs=', 'Content-Length': '0'} 2023-07-29T00:09:16.018 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f898486f4919cfdc-0064c458ab-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.019 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:09:16.019 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:09:16.020 DEBUG:boto:path=/hiiqos-49/?prefix=obj1 2023-07-29T00:09:16.020 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj1 2023-07-29T00:09:16.021 DEBUG:boto:Method: GET 2023-07-29T00:09:16.021 DEBUG:boto:Path: /hiiqos-49/?prefix=obj1 2023-07-29T00:09:16.021 DEBUG:boto:Data: 2023-07-29T00:09:16.022 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.022 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:16.023 DEBUG:boto:Port: 8001 2023-07-29T00:09:16.023 DEBUG:boto:Params: {} 2023-07-29T00:09:16.024 DEBUG:boto:Token: None 2023-07-29T00:09:16.024 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/ 2023-07-29T00:09:16.025 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JtQLrZI22HavlZk26k+ijabLkVE= 2023-07-29T00:09:16.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JtQLrZI22HavlZk26k+ijabLkVE=', 'Content-Length': '0'} 2023-07-29T00:09:16.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ebfe9a2780a0e70-0064c458ab-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.028 DEBUG:boto:b'hiiqos-49obj11000falseobj12023-07-29T00:08:34.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:16.028 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:09:16.029 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:09:16.029 DEBUG:boto:path=/hiiqos-49/?prefix=obj1 2023-07-29T00:09:16.029 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj1 2023-07-29T00:09:16.030 DEBUG:boto:Method: GET 2023-07-29T00:09:16.030 DEBUG:boto:Path: /hiiqos-49/?prefix=obj1 2023-07-29T00:09:16.030 DEBUG:boto:Data: 2023-07-29T00:09:16.031 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.031 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:16.032 DEBUG:boto:Port: 8000 2023-07-29T00:09:16.032 DEBUG:boto:Params: {} 2023-07-29T00:09:16.032 DEBUG:boto:Token: None 2023-07-29T00:09:16.033 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/ 2023-07-29T00:09:16.033 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JtQLrZI22HavlZk26k+ijabLkVE= 2023-07-29T00:09:16.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JtQLrZI22HavlZk26k+ijabLkVE=', 'Content-Length': '0'} 2023-07-29T00:09:16.036 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7d7adfd22641f9b-0064c458ac-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.036 DEBUG:boto:b'hiiqos-49obj11000falseobj12023-07-29T00:08:34.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:16.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-07-29T00:09:16.037 DEBUG:boto:path=/hiiqos-49/obj2 2023-07-29T00:09:16.038 DEBUG:boto:auth_path=/hiiqos-49/obj2 2023-07-29T00:09:16.038 DEBUG:boto:path=/hiiqos-49/obj2?versionId=null 2023-07-29T00:09:16.038 DEBUG:boto:auth_path=/hiiqos-49/obj2?versionId=null 2023-07-29T00:09:16.039 DEBUG:boto:Method: GET 2023-07-29T00:09:16.039 DEBUG:boto:Path: /hiiqos-49/obj2?versionId=null 2023-07-29T00:09:16.039 DEBUG:boto:Data: 2023-07-29T00:09:16.040 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.040 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:16.041 DEBUG:boto:Port: 8001 2023-07-29T00:09:16.041 DEBUG:boto:Params: {} 2023-07-29T00:09:16.042 DEBUG:boto:Token: None 2023-07-29T00:09:16.042 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/obj2?versionId=null 2023-07-29T00:09:16.042 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WNN/s+TyQYBoymG2tsgQZg7Jvos= 2023-07-29T00:09:16.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WNN/s+TyQYBoymG2tsgQZg7Jvos=', 'Content-Length': '0'} 2023-07-29T00:09:16.044 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000059d49c1cc3595a1f-0064c458ac-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.045 DEBUG:boto:path=/hiiqos-49/obj2 2023-07-29T00:09:16.046 DEBUG:boto:auth_path=/hiiqos-49/obj2 2023-07-29T00:09:16.046 DEBUG:boto:path=/hiiqos-49/obj2?versionId=null 2023-07-29T00:09:16.046 DEBUG:boto:auth_path=/hiiqos-49/obj2?versionId=null 2023-07-29T00:09:16.047 DEBUG:boto:Method: GET 2023-07-29T00:09:16.047 DEBUG:boto:Path: /hiiqos-49/obj2?versionId=null 2023-07-29T00:09:16.047 DEBUG:boto:Data: 2023-07-29T00:09:16.048 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.048 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:16.048 DEBUG:boto:Port: 8000 2023-07-29T00:09:16.049 DEBUG:boto:Params: {} 2023-07-29T00:09:16.049 DEBUG:boto:Token: None 2023-07-29T00:09:16.050 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/obj2?versionId=null 2023-07-29T00:09:16.050 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WNN/s+TyQYBoymG2tsgQZg7Jvos= 2023-07-29T00:09:16.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WNN/s+TyQYBoymG2tsgQZg7Jvos=', 'Content-Length': '0'} 2023-07-29T00:09:16.052 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dcb4dba3d8afdc48-0064c458ac-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.053 DEBUG:boto:path=/hiiqos-49/obj2 2023-07-29T00:09:16.053 DEBUG:boto:auth_path=/hiiqos-49/obj2 2023-07-29T00:09:16.054 DEBUG:boto:path=/hiiqos-49/obj2?versionId=null 2023-07-29T00:09:16.054 DEBUG:boto:auth_path=/hiiqos-49/obj2?versionId=null 2023-07-29T00:09:16.054 DEBUG:boto:Method: HEAD 2023-07-29T00:09:16.055 DEBUG:boto:Path: /hiiqos-49/obj2?versionId=null 2023-07-29T00:09:16.055 DEBUG:boto:Data: 2023-07-29T00:09:16.055 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.056 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:16.056 DEBUG:boto:Port: 8001 2023-07-29T00:09:16.057 DEBUG:boto:Params: {} 2023-07-29T00:09:16.057 DEBUG:boto:Token: None 2023-07-29T00:09:16.057 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/obj2?versionId=null 2023-07-29T00:09:16.058 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9uOnhFyfsF1A0RRobD7ncaKoWUY= 2023-07-29T00:09:16.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9uOnhFyfsF1A0RRobD7ncaKoWUY=', 'Content-Length': '0'} 2023-07-29T00:09:16.060 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f7bdb8c63268cd22-0064c458ac-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.060 DEBUG:boto:path=/hiiqos-49/obj2 2023-07-29T00:09:16.061 DEBUG:boto:auth_path=/hiiqos-49/obj2 2023-07-29T00:09:16.061 DEBUG:boto:path=/hiiqos-49/obj2?versionId=null 2023-07-29T00:09:16.061 DEBUG:boto:auth_path=/hiiqos-49/obj2?versionId=null 2023-07-29T00:09:16.062 DEBUG:boto:Method: HEAD 2023-07-29T00:09:16.062 DEBUG:boto:Path: /hiiqos-49/obj2?versionId=null 2023-07-29T00:09:16.062 DEBUG:boto:Data: 2023-07-29T00:09:16.063 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.063 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:16.063 DEBUG:boto:Port: 8000 2023-07-29T00:09:16.064 DEBUG:boto:Params: {} 2023-07-29T00:09:16.064 DEBUG:boto:Token: None 2023-07-29T00:09:16.065 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/obj2?versionId=null 2023-07-29T00:09:16.065 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9uOnhFyfsF1A0RRobD7ncaKoWUY= 2023-07-29T00:09:16.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9uOnhFyfsF1A0RRobD7ncaKoWUY=', 'Content-Length': '0'} 2023-07-29T00:09:16.067 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000722bde557375c291-0064c458ac-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-07-29T00:09:16.068 DEBUG:boto:path=/hiiqos-49/obj2 2023-07-29T00:09:16.068 DEBUG:boto:auth_path=/hiiqos-49/obj2 2023-07-29T00:09:16.069 DEBUG:boto:path=/hiiqos-49/obj2?versionId=null 2023-07-29T00:09:16.069 DEBUG:boto:auth_path=/hiiqos-49/obj2?versionId=null 2023-07-29T00:09:16.070 DEBUG:boto:Method: GET 2023-07-29T00:09:16.070 DEBUG:boto:Path: /hiiqos-49/obj2?versionId=null 2023-07-29T00:09:16.070 DEBUG:boto:Data: 2023-07-29T00:09:16.071 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.071 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:16.071 DEBUG:boto:Port: 8001 2023-07-29T00:09:16.072 DEBUG:boto:Params: {} 2023-07-29T00:09:16.072 DEBUG:boto:Token: None 2023-07-29T00:09:16.073 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/obj2?versionId=null 2023-07-29T00:09:16.073 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WNN/s+TyQYBoymG2tsgQZg7Jvos= 2023-07-29T00:09:16.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WNN/s+TyQYBoymG2tsgQZg7Jvos=', 'Content-Length': '0'} 2023-07-29T00:09:16.075 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000840c0d4ad64b4a05-0064c458ac-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.076 DEBUG:boto:path=/hiiqos-49/obj2 2023-07-29T00:09:16.076 DEBUG:boto:auth_path=/hiiqos-49/obj2 2023-07-29T00:09:16.077 DEBUG:boto:path=/hiiqos-49/obj2?versionId=null 2023-07-29T00:09:16.077 DEBUG:boto:auth_path=/hiiqos-49/obj2?versionId=null 2023-07-29T00:09:16.077 DEBUG:boto:Method: GET 2023-07-29T00:09:16.078 DEBUG:boto:Path: /hiiqos-49/obj2?versionId=null 2023-07-29T00:09:16.078 DEBUG:boto:Data: 2023-07-29T00:09:16.078 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.079 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:16.079 DEBUG:boto:Port: 8000 2023-07-29T00:09:16.080 DEBUG:boto:Params: {} 2023-07-29T00:09:16.080 DEBUG:boto:Token: None 2023-07-29T00:09:16.080 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/obj2?versionId=null 2023-07-29T00:09:16.081 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WNN/s+TyQYBoymG2tsgQZg7Jvos= 2023-07-29T00:09:16.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WNN/s+TyQYBoymG2tsgQZg7Jvos=', 'Content-Length': '0'} 2023-07-29T00:09:16.083 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000047dbb015309cf55f-0064c458ac-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.084 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:09:16.084 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:09:16.085 DEBUG:boto:path=/hiiqos-49/?prefix=obj2 2023-07-29T00:09:16.085 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj2 2023-07-29T00:09:16.085 DEBUG:boto:Method: GET 2023-07-29T00:09:16.086 DEBUG:boto:Path: /hiiqos-49/?prefix=obj2 2023-07-29T00:09:16.086 DEBUG:boto:Data: 2023-07-29T00:09:16.086 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.087 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:16.087 DEBUG:boto:Port: 8001 2023-07-29T00:09:16.088 DEBUG:boto:Params: {} 2023-07-29T00:09:16.088 DEBUG:boto:Token: None 2023-07-29T00:09:16.088 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/ 2023-07-29T00:09:16.089 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JtQLrZI22HavlZk26k+ijabLkVE= 2023-07-29T00:09:16.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JtQLrZI22HavlZk26k+ijabLkVE=', 'Content-Length': '0'} 2023-07-29T00:09:16.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e86628ebf47c58bc-0064c458ac-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.092 DEBUG:boto:b'hiiqos-49obj21000falseobj22023-07-29T00:08:34.191Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:16.093 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:09:16.093 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:09:16.093 DEBUG:boto:path=/hiiqos-49/?prefix=obj2 2023-07-29T00:09:16.094 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj2 2023-07-29T00:09:16.094 DEBUG:boto:Method: GET 2023-07-29T00:09:16.095 DEBUG:boto:Path: /hiiqos-49/?prefix=obj2 2023-07-29T00:09:16.095 DEBUG:boto:Data: 2023-07-29T00:09:16.095 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.096 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:16.096 DEBUG:boto:Port: 8000 2023-07-29T00:09:16.096 DEBUG:boto:Params: {} 2023-07-29T00:09:16.097 DEBUG:boto:Token: None 2023-07-29T00:09:16.097 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/ 2023-07-29T00:09:16.098 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JtQLrZI22HavlZk26k+ijabLkVE= 2023-07-29T00:09:16.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JtQLrZI22HavlZk26k+ijabLkVE=', 'Content-Length': '0'} 2023-07-29T00:09:16.100 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bad640d4251e47c3-0064c458ac-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.101 DEBUG:boto:b'hiiqos-49obj21000falseobj22023-07-29T00:08:34.191Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:16.101 DEBUG:boto:path=/hiiqos-49/obj2 2023-07-29T00:09:16.102 DEBUG:boto:auth_path=/hiiqos-49/obj2 2023-07-29T00:09:16.102 DEBUG:boto:Method: HEAD 2023-07-29T00:09:16.102 DEBUG:boto:Path: /hiiqos-49/obj2 2023-07-29T00:09:16.103 DEBUG:boto:Data: 2023-07-29T00:09:16.103 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.103 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:16.104 DEBUG:boto:Port: 8001 2023-07-29T00:09:16.104 DEBUG:boto:Params: {} 2023-07-29T00:09:16.104 DEBUG:boto:Token: None 2023-07-29T00:09:16.105 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/obj2 2023-07-29T00:09:16.105 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JlNSdqMoYZdFxFl76vdTxynGfdE= 2023-07-29T00:09:16.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JlNSdqMoYZdFxFl76vdTxynGfdE=', 'Content-Length': '0'} 2023-07-29T00:09:16.108 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f044d7072df2859c-0064c458ac-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.108 DEBUG:boto:path=/hiiqos-49/obj2 2023-07-29T00:09:16.109 DEBUG:boto:auth_path=/hiiqos-49/obj2 2023-07-29T00:09:16.109 DEBUG:boto:Method: HEAD 2023-07-29T00:09:16.109 DEBUG:boto:Path: /hiiqos-49/obj2 2023-07-29T00:09:16.110 DEBUG:boto:Data: 2023-07-29T00:09:16.110 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.110 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:16.111 DEBUG:boto:Port: 8000 2023-07-29T00:09:16.111 DEBUG:boto:Params: {} 2023-07-29T00:09:16.111 DEBUG:boto:Token: None 2023-07-29T00:09:16.112 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/obj2 2023-07-29T00:09:16.112 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JlNSdqMoYZdFxFl76vdTxynGfdE= 2023-07-29T00:09:16.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JlNSdqMoYZdFxFl76vdTxynGfdE=', 'Content-Length': '0'} 2023-07-29T00:09:16.114 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ee181bca29342cbf-0064c458ac-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-07-29T00:09:16.115 DEBUG:boto:path=/hiiqos-49/obj2 2023-07-29T00:09:16.116 DEBUG:boto:auth_path=/hiiqos-49/obj2 2023-07-29T00:09:16.116 DEBUG:boto:Method: GET 2023-07-29T00:09:16.116 DEBUG:boto:Path: /hiiqos-49/obj2 2023-07-29T00:09:16.117 DEBUG:boto:Data: 2023-07-29T00:09:16.117 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.118 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:16.118 DEBUG:boto:Port: 8001 2023-07-29T00:09:16.118 DEBUG:boto:Params: {} 2023-07-29T00:09:16.119 DEBUG:boto:Token: None 2023-07-29T00:09:16.119 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/obj2 2023-07-29T00:09:16.119 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KIviCH+mre6qyrjgLpGHlJdX7x0= 2023-07-29T00:09:16.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KIviCH+mre6qyrjgLpGHlJdX7x0=', 'Content-Length': '0'} 2023-07-29T00:09:16.121 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007464bc51317ff0e-0064c458ac-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.122 DEBUG:boto:path=/hiiqos-49/obj2 2023-07-29T00:09:16.122 DEBUG:boto:auth_path=/hiiqos-49/obj2 2023-07-29T00:09:16.123 DEBUG:boto:Method: GET 2023-07-29T00:09:16.123 DEBUG:boto:Path: /hiiqos-49/obj2 2023-07-29T00:09:16.124 DEBUG:boto:Data: 2023-07-29T00:09:16.124 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.124 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:16.125 DEBUG:boto:Port: 8000 2023-07-29T00:09:16.125 DEBUG:boto:Params: {} 2023-07-29T00:09:16.125 DEBUG:boto:Token: None 2023-07-29T00:09:16.126 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/obj2 2023-07-29T00:09:16.126 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KIviCH+mre6qyrjgLpGHlJdX7x0= 2023-07-29T00:09:16.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KIviCH+mre6qyrjgLpGHlJdX7x0=', 'Content-Length': '0'} 2023-07-29T00:09:16.128 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d582742bf92b8436-0064c458ac-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.172 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:09:16.173 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:09:16.173 DEBUG:boto:path=/hiiqos-49/?prefix=obj2 2023-07-29T00:09:16.174 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj2 2023-07-29T00:09:16.174 DEBUG:boto:Method: GET 2023-07-29T00:09:16.174 DEBUG:boto:Path: /hiiqos-49/?prefix=obj2 2023-07-29T00:09:16.175 DEBUG:boto:Data: 2023-07-29T00:09:16.175 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.176 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:16.176 DEBUG:boto:Port: 8001 2023-07-29T00:09:16.176 DEBUG:boto:Params: {} 2023-07-29T00:09:16.177 DEBUG:boto:Token: None 2023-07-29T00:09:16.177 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/ 2023-07-29T00:09:16.178 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JtQLrZI22HavlZk26k+ijabLkVE= 2023-07-29T00:09:16.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JtQLrZI22HavlZk26k+ijabLkVE=', 'Content-Length': '0'} 2023-07-29T00:09:16.180 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071e48cb06010b996-0064c458ac-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.181 DEBUG:boto:b'hiiqos-49obj21000falseobj22023-07-29T00:08:34.191Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:16.182 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:09:16.182 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:09:16.183 DEBUG:boto:path=/hiiqos-49/?prefix=obj2 2023-07-29T00:09:16.183 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj2 2023-07-29T00:09:16.183 DEBUG:boto:Method: GET 2023-07-29T00:09:16.184 DEBUG:boto:Path: /hiiqos-49/?prefix=obj2 2023-07-29T00:09:16.184 DEBUG:boto:Data: 2023-07-29T00:09:16.184 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.185 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:16.185 DEBUG:boto:Port: 8000 2023-07-29T00:09:16.186 DEBUG:boto:Params: {} 2023-07-29T00:09:16.186 DEBUG:boto:Token: None 2023-07-29T00:09:16.186 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/ 2023-07-29T00:09:16.187 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JtQLrZI22HavlZk26k+ijabLkVE= 2023-07-29T00:09:16.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JtQLrZI22HavlZk26k+ijabLkVE=', 'Content-Length': '0'} 2023-07-29T00:09:16.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5114d9351b6001e-0064c458ac-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.190 DEBUG:boto:b'hiiqos-49obj21000falseobj22023-07-29T00:08:34.191Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:16.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-07-29T00:09:16.191 DEBUG:boto:path=/hiiqos-49/obj3 2023-07-29T00:09:16.192 DEBUG:boto:auth_path=/hiiqos-49/obj3 2023-07-29T00:09:16.192 DEBUG:boto:path=/hiiqos-49/obj3?versionId=null 2023-07-29T00:09:16.192 DEBUG:boto:auth_path=/hiiqos-49/obj3?versionId=null 2023-07-29T00:09:16.193 DEBUG:boto:Method: GET 2023-07-29T00:09:16.193 DEBUG:boto:Path: /hiiqos-49/obj3?versionId=null 2023-07-29T00:09:16.194 DEBUG:boto:Data: 2023-07-29T00:09:16.194 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.194 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:16.195 DEBUG:boto:Port: 8001 2023-07-29T00:09:16.195 DEBUG:boto:Params: {} 2023-07-29T00:09:16.195 DEBUG:boto:Token: None 2023-07-29T00:09:16.196 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/obj3?versionId=null 2023-07-29T00:09:16.196 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1rzSwWtUSrDxpP8RtA9gVqulE64= 2023-07-29T00:09:16.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1rzSwWtUSrDxpP8RtA9gVqulE64=', 'Content-Length': '0'} 2023-07-29T00:09:16.199 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f75e0a0ba05699b1-0064c458ac-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.240 DEBUG:boto:path=/hiiqos-49/obj3 2023-07-29T00:09:16.241 DEBUG:boto:auth_path=/hiiqos-49/obj3 2023-07-29T00:09:16.242 DEBUG:boto:path=/hiiqos-49/obj3?versionId=null 2023-07-29T00:09:16.243 DEBUG:boto:auth_path=/hiiqos-49/obj3?versionId=null 2023-07-29T00:09:16.243 DEBUG:boto:Method: GET 2023-07-29T00:09:16.244 DEBUG:boto:Path: /hiiqos-49/obj3?versionId=null 2023-07-29T00:09:16.245 DEBUG:boto:Data: 2023-07-29T00:09:16.245 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.246 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:16.246 DEBUG:boto:Port: 8000 2023-07-29T00:09:16.246 DEBUG:boto:Params: {} 2023-07-29T00:09:16.247 DEBUG:boto:Token: None 2023-07-29T00:09:16.247 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/obj3?versionId=null 2023-07-29T00:09:16.248 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1rzSwWtUSrDxpP8RtA9gVqulE64= 2023-07-29T00:09:16.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1rzSwWtUSrDxpP8RtA9gVqulE64=', 'Content-Length': '0'} 2023-07-29T00:09:16.250 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000084f1a7ae16ae2ba-0064c458ac-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.251 DEBUG:boto:path=/hiiqos-49/obj3 2023-07-29T00:09:16.251 DEBUG:boto:auth_path=/hiiqos-49/obj3 2023-07-29T00:09:16.251 DEBUG:boto:path=/hiiqos-49/obj3?versionId=null 2023-07-29T00:09:16.252 DEBUG:boto:auth_path=/hiiqos-49/obj3?versionId=null 2023-07-29T00:09:16.252 DEBUG:boto:Method: HEAD 2023-07-29T00:09:16.253 DEBUG:boto:Path: /hiiqos-49/obj3?versionId=null 2023-07-29T00:09:16.253 DEBUG:boto:Data: 2023-07-29T00:09:16.254 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.254 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:16.255 DEBUG:boto:Port: 8001 2023-07-29T00:09:16.256 DEBUG:boto:Params: {} 2023-07-29T00:09:16.256 DEBUG:boto:Token: None 2023-07-29T00:09:16.257 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/obj3?versionId=null 2023-07-29T00:09:16.258 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zuIzzYGNLuyLUQX4TzhVeI83czY= 2023-07-29T00:09:16.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zuIzzYGNLuyLUQX4TzhVeI83czY=', 'Content-Length': '0'} 2023-07-29T00:09:16.261 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000096cc5e762b6c57fa-0064c458ac-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.262 DEBUG:boto:path=/hiiqos-49/obj3 2023-07-29T00:09:16.263 DEBUG:boto:auth_path=/hiiqos-49/obj3 2023-07-29T00:09:16.263 DEBUG:boto:path=/hiiqos-49/obj3?versionId=null 2023-07-29T00:09:16.263 DEBUG:boto:auth_path=/hiiqos-49/obj3?versionId=null 2023-07-29T00:09:16.264 DEBUG:boto:Method: HEAD 2023-07-29T00:09:16.264 DEBUG:boto:Path: /hiiqos-49/obj3?versionId=null 2023-07-29T00:09:16.265 DEBUG:boto:Data: 2023-07-29T00:09:16.266 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.266 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:16.267 DEBUG:boto:Port: 8000 2023-07-29T00:09:16.268 DEBUG:boto:Params: {} 2023-07-29T00:09:16.268 DEBUG:boto:Token: None 2023-07-29T00:09:16.269 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/obj3?versionId=null 2023-07-29T00:09:16.270 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zuIzzYGNLuyLUQX4TzhVeI83czY= 2023-07-29T00:09:16.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zuIzzYGNLuyLUQX4TzhVeI83czY=', 'Content-Length': '0'} 2023-07-29T00:09:16.273 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f7a721791c746823-0064c458ac-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-07-29T00:09:16.275 DEBUG:boto:path=/hiiqos-49/obj3 2023-07-29T00:09:16.275 DEBUG:boto:auth_path=/hiiqos-49/obj3 2023-07-29T00:09:16.276 DEBUG:boto:path=/hiiqos-49/obj3?versionId=null 2023-07-29T00:09:16.277 DEBUG:boto:auth_path=/hiiqos-49/obj3?versionId=null 2023-07-29T00:09:16.278 DEBUG:boto:Method: GET 2023-07-29T00:09:16.279 DEBUG:boto:Path: /hiiqos-49/obj3?versionId=null 2023-07-29T00:09:16.280 DEBUG:boto:Data: 2023-07-29T00:09:16.280 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.281 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:16.281 DEBUG:boto:Port: 8001 2023-07-29T00:09:16.282 DEBUG:boto:Params: {} 2023-07-29T00:09:16.283 DEBUG:boto:Token: None 2023-07-29T00:09:16.284 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/obj3?versionId=null 2023-07-29T00:09:16.285 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1rzSwWtUSrDxpP8RtA9gVqulE64= 2023-07-29T00:09:16.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1rzSwWtUSrDxpP8RtA9gVqulE64=', 'Content-Length': '0'} 2023-07-29T00:09:16.288 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000206a2121ada25cba-0064c458ac-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.289 DEBUG:boto:path=/hiiqos-49/obj3 2023-07-29T00:09:16.290 DEBUG:boto:auth_path=/hiiqos-49/obj3 2023-07-29T00:09:16.290 DEBUG:boto:path=/hiiqos-49/obj3?versionId=null 2023-07-29T00:09:16.290 DEBUG:boto:auth_path=/hiiqos-49/obj3?versionId=null 2023-07-29T00:09:16.291 DEBUG:boto:Method: GET 2023-07-29T00:09:16.291 DEBUG:boto:Path: /hiiqos-49/obj3?versionId=null 2023-07-29T00:09:16.292 DEBUG:boto:Data: 2023-07-29T00:09:16.292 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.293 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:16.293 DEBUG:boto:Port: 8000 2023-07-29T00:09:16.294 DEBUG:boto:Params: {} 2023-07-29T00:09:16.294 DEBUG:boto:Token: None 2023-07-29T00:09:16.294 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/obj3?versionId=null 2023-07-29T00:09:16.295 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1rzSwWtUSrDxpP8RtA9gVqulE64= 2023-07-29T00:09:16.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1rzSwWtUSrDxpP8RtA9gVqulE64=', 'Content-Length': '0'} 2023-07-29T00:09:16.297 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f45566ca54534a5-0064c458ac-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.298 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:09:16.299 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:09:16.300 DEBUG:boto:path=/hiiqos-49/?prefix=obj3 2023-07-29T00:09:16.301 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj3 2023-07-29T00:09:16.302 DEBUG:boto:Method: GET 2023-07-29T00:09:16.303 DEBUG:boto:Path: /hiiqos-49/?prefix=obj3 2023-07-29T00:09:16.304 DEBUG:boto:Data: 2023-07-29T00:09:16.304 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.305 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:16.306 DEBUG:boto:Port: 8001 2023-07-29T00:09:16.306 DEBUG:boto:Params: {} 2023-07-29T00:09:16.307 DEBUG:boto:Token: None 2023-07-29T00:09:16.308 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/ 2023-07-29T00:09:16.308 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JtQLrZI22HavlZk26k+ijabLkVE= 2023-07-29T00:09:16.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JtQLrZI22HavlZk26k+ijabLkVE=', 'Content-Length': '0'} 2023-07-29T00:09:16.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c13eb51c72d104cc-0064c458ac-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.352 DEBUG:boto:b'hiiqos-49obj31000falseobj32023-07-29T00:08:34.246Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:16.353 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:09:16.354 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:09:16.355 DEBUG:boto:path=/hiiqos-49/?prefix=obj3 2023-07-29T00:09:16.355 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj3 2023-07-29T00:09:16.356 DEBUG:boto:Method: GET 2023-07-29T00:09:16.357 DEBUG:boto:Path: /hiiqos-49/?prefix=obj3 2023-07-29T00:09:16.358 DEBUG:boto:Data: 2023-07-29T00:09:16.359 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.359 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:16.360 DEBUG:boto:Port: 8000 2023-07-29T00:09:16.360 DEBUG:boto:Params: {} 2023-07-29T00:09:16.361 DEBUG:boto:Token: None 2023-07-29T00:09:16.361 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/ 2023-07-29T00:09:16.361 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JtQLrZI22HavlZk26k+ijabLkVE= 2023-07-29T00:09:16.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JtQLrZI22HavlZk26k+ijabLkVE=', 'Content-Length': '0'} 2023-07-29T00:09:16.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b6ee8354ee2ee2a-0064c458ac-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.365 DEBUG:boto:b'hiiqos-49obj31000falseobj32023-07-29T00:08:34.246Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:16.366 DEBUG:boto:path=/hiiqos-49/obj3 2023-07-29T00:09:16.366 DEBUG:boto:auth_path=/hiiqos-49/obj3 2023-07-29T00:09:16.366 DEBUG:boto:Method: HEAD 2023-07-29T00:09:16.367 DEBUG:boto:Path: /hiiqos-49/obj3 2023-07-29T00:09:16.367 DEBUG:boto:Data: 2023-07-29T00:09:16.367 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.368 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:16.368 DEBUG:boto:Port: 8001 2023-07-29T00:09:16.368 DEBUG:boto:Params: {} 2023-07-29T00:09:16.369 DEBUG:boto:Token: None 2023-07-29T00:09:16.369 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/obj3 2023-07-29T00:09:16.370 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:C/mcYX2DthjEuUOO8ct/x/J+8bc= 2023-07-29T00:09:16.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:C/mcYX2DthjEuUOO8ct/x/J+8bc=', 'Content-Length': '0'} 2023-07-29T00:09:16.372 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d6562df70e721abd-0064c458ac-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.372 DEBUG:boto:path=/hiiqos-49/obj3 2023-07-29T00:09:16.373 DEBUG:boto:auth_path=/hiiqos-49/obj3 2023-07-29T00:09:16.373 DEBUG:boto:Method: HEAD 2023-07-29T00:09:16.374 DEBUG:boto:Path: /hiiqos-49/obj3 2023-07-29T00:09:16.374 DEBUG:boto:Data: 2023-07-29T00:09:16.374 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.375 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:16.375 DEBUG:boto:Port: 8000 2023-07-29T00:09:16.375 DEBUG:boto:Params: {} 2023-07-29T00:09:16.376 DEBUG:boto:Token: None 2023-07-29T00:09:16.376 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/obj3 2023-07-29T00:09:16.377 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:C/mcYX2DthjEuUOO8ct/x/J+8bc= 2023-07-29T00:09:16.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:C/mcYX2DthjEuUOO8ct/x/J+8bc=', 'Content-Length': '0'} 2023-07-29T00:09:16.379 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000231710c87241d919-0064c458ac-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-07-29T00:09:16.380 DEBUG:boto:path=/hiiqos-49/obj3 2023-07-29T00:09:16.380 DEBUG:boto:auth_path=/hiiqos-49/obj3 2023-07-29T00:09:16.381 DEBUG:boto:Method: GET 2023-07-29T00:09:16.381 DEBUG:boto:Path: /hiiqos-49/obj3 2023-07-29T00:09:16.381 DEBUG:boto:Data: 2023-07-29T00:09:16.382 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.382 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:16.382 DEBUG:boto:Port: 8001 2023-07-29T00:09:16.383 DEBUG:boto:Params: {} 2023-07-29T00:09:16.383 DEBUG:boto:Token: None 2023-07-29T00:09:16.384 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/obj3 2023-07-29T00:09:16.384 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aJSj3o7ADJ/dd4cR49VaCt6hty0= 2023-07-29T00:09:16.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aJSj3o7ADJ/dd4cR49VaCt6hty0=', 'Content-Length': '0'} 2023-07-29T00:09:16.386 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c32e8a493c5c866-0064c458ac-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.387 DEBUG:boto:path=/hiiqos-49/obj3 2023-07-29T00:09:16.387 DEBUG:boto:auth_path=/hiiqos-49/obj3 2023-07-29T00:09:16.387 DEBUG:boto:Method: GET 2023-07-29T00:09:16.388 DEBUG:boto:Path: /hiiqos-49/obj3 2023-07-29T00:09:16.388 DEBUG:boto:Data: 2023-07-29T00:09:16.389 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.389 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:16.389 DEBUG:boto:Port: 8000 2023-07-29T00:09:16.390 DEBUG:boto:Params: {} 2023-07-29T00:09:16.390 DEBUG:boto:Token: None 2023-07-29T00:09:16.390 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/obj3 2023-07-29T00:09:16.391 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aJSj3o7ADJ/dd4cR49VaCt6hty0= 2023-07-29T00:09:16.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aJSj3o7ADJ/dd4cR49VaCt6hty0=', 'Content-Length': '0'} 2023-07-29T00:09:16.393 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000025e63c7fc5836586-0064c458ac-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.436 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:09:16.437 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:09:16.437 DEBUG:boto:path=/hiiqos-49/?prefix=obj3 2023-07-29T00:09:16.438 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj3 2023-07-29T00:09:16.438 DEBUG:boto:Method: GET 2023-07-29T00:09:16.438 DEBUG:boto:Path: /hiiqos-49/?prefix=obj3 2023-07-29T00:09:16.439 DEBUG:boto:Data: 2023-07-29T00:09:16.439 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.439 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:16.440 DEBUG:boto:Port: 8001 2023-07-29T00:09:16.440 DEBUG:boto:Params: {} 2023-07-29T00:09:16.441 DEBUG:boto:Token: None 2023-07-29T00:09:16.441 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/ 2023-07-29T00:09:16.442 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JtQLrZI22HavlZk26k+ijabLkVE= 2023-07-29T00:09:16.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JtQLrZI22HavlZk26k+ijabLkVE=', 'Content-Length': '0'} 2023-07-29T00:09:16.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c1aca79155c7b2a-0064c458ac-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.445 DEBUG:boto:b'hiiqos-49obj31000falseobj32023-07-29T00:08:34.246Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:16.446 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:09:16.446 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:09:16.447 DEBUG:boto:path=/hiiqos-49/?prefix=obj3 2023-07-29T00:09:16.447 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj3 2023-07-29T00:09:16.447 DEBUG:boto:Method: GET 2023-07-29T00:09:16.448 DEBUG:boto:Path: /hiiqos-49/?prefix=obj3 2023-07-29T00:09:16.448 DEBUG:boto:Data: 2023-07-29T00:09:16.449 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.449 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:16.450 DEBUG:boto:Port: 8000 2023-07-29T00:09:16.450 DEBUG:boto:Params: {} 2023-07-29T00:09:16.450 DEBUG:boto:Token: None 2023-07-29T00:09:16.451 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/ 2023-07-29T00:09:16.451 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JtQLrZI22HavlZk26k+ijabLkVE= 2023-07-29T00:09:16.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JtQLrZI22HavlZk26k+ijabLkVE=', 'Content-Length': '0'} 2023-07-29T00:09:16.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2cde0ad297ed150-0064c458ac-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.455 DEBUG:boto:b'hiiqos-49obj31000falseobj32023-07-29T00:08:34.246Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:16.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-07-29T00:09:16.457 DEBUG:boto:path=/hiiqos-49/obj4 2023-07-29T00:09:16.457 DEBUG:boto:auth_path=/hiiqos-49/obj4 2023-07-29T00:09:16.458 DEBUG:boto:path=/hiiqos-49/obj4?versionId=null 2023-07-29T00:09:16.459 DEBUG:boto:auth_path=/hiiqos-49/obj4?versionId=null 2023-07-29T00:09:16.460 DEBUG:boto:Method: GET 2023-07-29T00:09:16.460 DEBUG:boto:Path: /hiiqos-49/obj4?versionId=null 2023-07-29T00:09:16.460 DEBUG:boto:Data: 2023-07-29T00:09:16.461 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.461 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:16.462 DEBUG:boto:Port: 8001 2023-07-29T00:09:16.462 DEBUG:boto:Params: {} 2023-07-29T00:09:16.462 DEBUG:boto:Token: None 2023-07-29T00:09:16.463 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/obj4?versionId=null 2023-07-29T00:09:16.463 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JpHndMhDew1uaCJW3BMzhuwOuDo= 2023-07-29T00:09:16.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JpHndMhDew1uaCJW3BMzhuwOuDo=', 'Content-Length': '0'} 2023-07-29T00:09:16.466 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b1cc5025f215608c-0064c458ac-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.466 DEBUG:boto:path=/hiiqos-49/obj4 2023-07-29T00:09:16.467 DEBUG:boto:auth_path=/hiiqos-49/obj4 2023-07-29T00:09:16.467 DEBUG:boto:path=/hiiqos-49/obj4?versionId=null 2023-07-29T00:09:16.468 DEBUG:boto:auth_path=/hiiqos-49/obj4?versionId=null 2023-07-29T00:09:16.468 DEBUG:boto:Method: GET 2023-07-29T00:09:16.468 DEBUG:boto:Path: /hiiqos-49/obj4?versionId=null 2023-07-29T00:09:16.469 DEBUG:boto:Data: 2023-07-29T00:09:16.469 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.469 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:16.470 DEBUG:boto:Port: 8000 2023-07-29T00:09:16.470 DEBUG:boto:Params: {} 2023-07-29T00:09:16.471 DEBUG:boto:Token: None 2023-07-29T00:09:16.471 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/obj4?versionId=null 2023-07-29T00:09:16.471 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JpHndMhDew1uaCJW3BMzhuwOuDo= 2023-07-29T00:09:16.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JpHndMhDew1uaCJW3BMzhuwOuDo=', 'Content-Length': '0'} 2023-07-29T00:09:16.474 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000010077160f12bbb21-0064c458ac-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.474 DEBUG:boto:path=/hiiqos-49/obj4 2023-07-29T00:09:16.475 DEBUG:boto:auth_path=/hiiqos-49/obj4 2023-07-29T00:09:16.475 DEBUG:boto:path=/hiiqos-49/obj4?versionId=null 2023-07-29T00:09:16.475 DEBUG:boto:auth_path=/hiiqos-49/obj4?versionId=null 2023-07-29T00:09:16.476 DEBUG:boto:Method: HEAD 2023-07-29T00:09:16.476 DEBUG:boto:Path: /hiiqos-49/obj4?versionId=null 2023-07-29T00:09:16.476 DEBUG:boto:Data: 2023-07-29T00:09:16.477 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.477 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:16.477 DEBUG:boto:Port: 8001 2023-07-29T00:09:16.478 DEBUG:boto:Params: {} 2023-07-29T00:09:16.478 DEBUG:boto:Token: None 2023-07-29T00:09:16.478 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/obj4?versionId=null 2023-07-29T00:09:16.479 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LT3jh2T54UKKCG1JdPGj7l3XLQg= 2023-07-29T00:09:16.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LT3jh2T54UKKCG1JdPGj7l3XLQg=', 'Content-Length': '0'} 2023-07-29T00:09:16.481 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b2286462ab0e2fe7-0064c458ac-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.481 DEBUG:boto:path=/hiiqos-49/obj4 2023-07-29T00:09:16.482 DEBUG:boto:auth_path=/hiiqos-49/obj4 2023-07-29T00:09:16.482 DEBUG:boto:path=/hiiqos-49/obj4?versionId=null 2023-07-29T00:09:16.482 DEBUG:boto:auth_path=/hiiqos-49/obj4?versionId=null 2023-07-29T00:09:16.483 DEBUG:boto:Method: HEAD 2023-07-29T00:09:16.483 DEBUG:boto:Path: /hiiqos-49/obj4?versionId=null 2023-07-29T00:09:16.483 DEBUG:boto:Data: 2023-07-29T00:09:16.484 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.484 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:16.484 DEBUG:boto:Port: 8000 2023-07-29T00:09:16.485 DEBUG:boto:Params: {} 2023-07-29T00:09:16.485 DEBUG:boto:Token: None 2023-07-29T00:09:16.485 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/obj4?versionId=null 2023-07-29T00:09:16.486 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LT3jh2T54UKKCG1JdPGj7l3XLQg= 2023-07-29T00:09:16.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LT3jh2T54UKKCG1JdPGj7l3XLQg=', 'Content-Length': '0'} 2023-07-29T00:09:16.488 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b89f04f3bd831518-0064c458ac-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-07-29T00:09:16.489 DEBUG:boto:path=/hiiqos-49/obj4 2023-07-29T00:09:16.489 DEBUG:boto:auth_path=/hiiqos-49/obj4 2023-07-29T00:09:16.490 DEBUG:boto:path=/hiiqos-49/obj4?versionId=null 2023-07-29T00:09:16.490 DEBUG:boto:auth_path=/hiiqos-49/obj4?versionId=null 2023-07-29T00:09:16.490 DEBUG:boto:Method: GET 2023-07-29T00:09:16.491 DEBUG:boto:Path: /hiiqos-49/obj4?versionId=null 2023-07-29T00:09:16.491 DEBUG:boto:Data: 2023-07-29T00:09:16.491 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.492 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:16.492 DEBUG:boto:Port: 8001 2023-07-29T00:09:16.493 DEBUG:boto:Params: {} 2023-07-29T00:09:16.494 DEBUG:boto:Token: None 2023-07-29T00:09:16.495 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/obj4?versionId=null 2023-07-29T00:09:16.496 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JpHndMhDew1uaCJW3BMzhuwOuDo= 2023-07-29T00:09:16.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JpHndMhDew1uaCJW3BMzhuwOuDo=', 'Content-Length': '0'} 2023-07-29T00:09:16.499 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000047beb59cd410e4ec-0064c458ac-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.544 DEBUG:boto:path=/hiiqos-49/obj4 2023-07-29T00:09:16.545 DEBUG:boto:auth_path=/hiiqos-49/obj4 2023-07-29T00:09:16.546 DEBUG:boto:path=/hiiqos-49/obj4?versionId=null 2023-07-29T00:09:16.546 DEBUG:boto:auth_path=/hiiqos-49/obj4?versionId=null 2023-07-29T00:09:16.547 DEBUG:boto:Method: GET 2023-07-29T00:09:16.547 DEBUG:boto:Path: /hiiqos-49/obj4?versionId=null 2023-07-29T00:09:16.548 DEBUG:boto:Data: 2023-07-29T00:09:16.548 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.548 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:16.549 DEBUG:boto:Port: 8000 2023-07-29T00:09:16.549 DEBUG:boto:Params: {} 2023-07-29T00:09:16.550 DEBUG:boto:Token: None 2023-07-29T00:09:16.550 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/obj4?versionId=null 2023-07-29T00:09:16.551 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JpHndMhDew1uaCJW3BMzhuwOuDo= 2023-07-29T00:09:16.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JpHndMhDew1uaCJW3BMzhuwOuDo=', 'Content-Length': '0'} 2023-07-29T00:09:16.553 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e2f62c4213d24c04-0064c458ac-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.554 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:09:16.554 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:09:16.555 DEBUG:boto:path=/hiiqos-49/?prefix=obj4 2023-07-29T00:09:16.555 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj4 2023-07-29T00:09:16.556 DEBUG:boto:Method: GET 2023-07-29T00:09:16.556 DEBUG:boto:Path: /hiiqos-49/?prefix=obj4 2023-07-29T00:09:16.556 DEBUG:boto:Data: 2023-07-29T00:09:16.557 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.557 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:16.557 DEBUG:boto:Port: 8001 2023-07-29T00:09:16.558 DEBUG:boto:Params: {} 2023-07-29T00:09:16.558 DEBUG:boto:Token: None 2023-07-29T00:09:16.559 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/ 2023-07-29T00:09:16.559 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JtQLrZI22HavlZk26k+ijabLkVE= 2023-07-29T00:09:16.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JtQLrZI22HavlZk26k+ijabLkVE=', 'Content-Length': '0'} 2023-07-29T00:09:16.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008c4a635a9e5b69b-0064c458ac-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.562 DEBUG:boto:b'hiiqos-49obj41000falseobj42023-07-29T00:08:34.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:16.563 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:09:16.564 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:09:16.564 DEBUG:boto:path=/hiiqos-49/?prefix=obj4 2023-07-29T00:09:16.565 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj4 2023-07-29T00:09:16.565 DEBUG:boto:Method: GET 2023-07-29T00:09:16.565 DEBUG:boto:Path: /hiiqos-49/?prefix=obj4 2023-07-29T00:09:16.566 DEBUG:boto:Data: 2023-07-29T00:09:16.566 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.566 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:16.567 DEBUG:boto:Port: 8000 2023-07-29T00:09:16.567 DEBUG:boto:Params: {} 2023-07-29T00:09:16.568 DEBUG:boto:Token: None 2023-07-29T00:09:16.568 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/ 2023-07-29T00:09:16.568 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JtQLrZI22HavlZk26k+ijabLkVE= 2023-07-29T00:09:16.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JtQLrZI22HavlZk26k+ijabLkVE=', 'Content-Length': '0'} 2023-07-29T00:09:16.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae1b797ce31ed4e9-0064c458ac-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.572 DEBUG:boto:b'hiiqos-49obj41000falseobj42023-07-29T00:08:34.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:16.572 DEBUG:boto:path=/hiiqos-49/obj4 2023-07-29T00:09:16.573 DEBUG:boto:auth_path=/hiiqos-49/obj4 2023-07-29T00:09:16.573 DEBUG:boto:Method: HEAD 2023-07-29T00:09:16.574 DEBUG:boto:Path: /hiiqos-49/obj4 2023-07-29T00:09:16.574 DEBUG:boto:Data: 2023-07-29T00:09:16.574 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.575 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:16.575 DEBUG:boto:Port: 8001 2023-07-29T00:09:16.575 DEBUG:boto:Params: {} 2023-07-29T00:09:16.576 DEBUG:boto:Token: None 2023-07-29T00:09:16.576 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/obj4 2023-07-29T00:09:16.576 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bZMQM7rVpMPuMfCUoQpntDuozw0= 2023-07-29T00:09:16.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bZMQM7rVpMPuMfCUoQpntDuozw0=', 'Content-Length': '0'} 2023-07-29T00:09:16.578 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000066e60210f18cfdb4-0064c458ac-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.579 DEBUG:boto:path=/hiiqos-49/obj4 2023-07-29T00:09:16.579 DEBUG:boto:auth_path=/hiiqos-49/obj4 2023-07-29T00:09:16.580 DEBUG:boto:Method: HEAD 2023-07-29T00:09:16.580 DEBUG:boto:Path: /hiiqos-49/obj4 2023-07-29T00:09:16.581 DEBUG:boto:Data: 2023-07-29T00:09:16.581 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.581 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:16.582 DEBUG:boto:Port: 8000 2023-07-29T00:09:16.582 DEBUG:boto:Params: {} 2023-07-29T00:09:16.582 DEBUG:boto:Token: None 2023-07-29T00:09:16.583 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/obj4 2023-07-29T00:09:16.583 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bZMQM7rVpMPuMfCUoQpntDuozw0= 2023-07-29T00:09:16.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bZMQM7rVpMPuMfCUoQpntDuozw0=', 'Content-Length': '0'} 2023-07-29T00:09:16.586 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000acaf53fc22aaf929-0064c458ac-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-07-29T00:09:16.587 DEBUG:boto:path=/hiiqos-49/obj4 2023-07-29T00:09:16.588 DEBUG:boto:auth_path=/hiiqos-49/obj4 2023-07-29T00:09:16.589 DEBUG:boto:Method: GET 2023-07-29T00:09:16.590 DEBUG:boto:Path: /hiiqos-49/obj4 2023-07-29T00:09:16.590 DEBUG:boto:Data: 2023-07-29T00:09:16.591 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.592 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:16.592 DEBUG:boto:Port: 8001 2023-07-29T00:09:16.593 DEBUG:boto:Params: {} 2023-07-29T00:09:16.594 DEBUG:boto:Token: None 2023-07-29T00:09:16.595 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/obj4 2023-07-29T00:09:16.595 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Jj8De4uqLNRhm/xrAHmIxMABsKc= 2023-07-29T00:09:16.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Jj8De4uqLNRhm/xrAHmIxMABsKc=', 'Content-Length': '0'} 2023-07-29T00:09:16.598 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d38689eafd9b5b6-0064c458ac-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.599 DEBUG:boto:path=/hiiqos-49/obj4 2023-07-29T00:09:16.600 DEBUG:boto:auth_path=/hiiqos-49/obj4 2023-07-29T00:09:16.601 DEBUG:boto:Method: GET 2023-07-29T00:09:16.601 DEBUG:boto:Path: /hiiqos-49/obj4 2023-07-29T00:09:16.602 DEBUG:boto:Data: 2023-07-29T00:09:16.603 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.604 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:16.604 DEBUG:boto:Port: 8000 2023-07-29T00:09:16.605 DEBUG:boto:Params: {} 2023-07-29T00:09:16.606 DEBUG:boto:Token: None 2023-07-29T00:09:16.607 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/obj4 2023-07-29T00:09:16.608 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Jj8De4uqLNRhm/xrAHmIxMABsKc= 2023-07-29T00:09:16.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Jj8De4uqLNRhm/xrAHmIxMABsKc=', 'Content-Length': '0'} 2023-07-29T00:09:16.611 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa69333737a20789-0064c458ac-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.612 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:09:16.613 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:09:16.613 DEBUG:boto:path=/hiiqos-49/?prefix=obj4 2023-07-29T00:09:16.613 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj4 2023-07-29T00:09:16.614 DEBUG:boto:Method: GET 2023-07-29T00:09:16.614 DEBUG:boto:Path: /hiiqos-49/?prefix=obj4 2023-07-29T00:09:16.615 DEBUG:boto:Data: 2023-07-29T00:09:16.615 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.616 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:16.616 DEBUG:boto:Port: 8001 2023-07-29T00:09:16.616 DEBUG:boto:Params: {} 2023-07-29T00:09:16.617 DEBUG:boto:Token: None 2023-07-29T00:09:16.618 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/ 2023-07-29T00:09:16.619 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JtQLrZI22HavlZk26k+ijabLkVE= 2023-07-29T00:09:16.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JtQLrZI22HavlZk26k+ijabLkVE=', 'Content-Length': '0'} 2023-07-29T00:09:16.623 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e80d30d3d3d4b57-0064c458ac-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.623 DEBUG:boto:b'hiiqos-49obj41000falseobj42023-07-29T00:08:34.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:16.624 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:09:16.625 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:09:16.625 DEBUG:boto:path=/hiiqos-49/?prefix=obj4 2023-07-29T00:09:16.626 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj4 2023-07-29T00:09:16.626 DEBUG:boto:Method: GET 2023-07-29T00:09:16.626 DEBUG:boto:Path: /hiiqos-49/?prefix=obj4 2023-07-29T00:09:16.627 DEBUG:boto:Data: 2023-07-29T00:09:16.627 DEBUG:boto:Headers: {} 2023-07-29T00:09:16.627 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:16.628 DEBUG:boto:Port: 8000 2023-07-29T00:09:16.628 DEBUG:boto:Params: {} 2023-07-29T00:09:16.628 DEBUG:boto:Token: None 2023-07-29T00:09:16.629 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:16 GMT /hiiqos-49/ 2023-07-29T00:09:16.629 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JtQLrZI22HavlZk26k+ijabLkVE= 2023-07-29T00:09:16.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JtQLrZI22HavlZk26k+ijabLkVE=', 'Content-Length': '0'} 2023-07-29T00:09:16.632 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c0ef38e42f4b11f-0064c458ac-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:16.632 DEBUG:boto:b'hiiqos-49obj41000falseobj42023-07-29T00:08:34.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:16.633 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-49 zones={a2, a1} 2023-07-29T00:09:16.633 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'hiiqos-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:09:16.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'hiiqos-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:09:16.634 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket hiiqos-48 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:09:16.682 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:09:16.682 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:09:16.743 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'hiiqos-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:09:16.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'hiiqos-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:09:16.745 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket hiiqos-49 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:09:16.791 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:09:16.792 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:09:16.853 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:09:16.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:09:16.854 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:09:16.899 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:09:16.900 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:09:16.929 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T00:09:16.942 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T00:09:16.942 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:09:16.942 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:09:16.942 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:46:09.233742Z" 2023-07-29T00:09:16.943 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:09:16.943 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:09:16.943 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690589356.810494_732.1", 2023-07-29T00:09:16.943 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:09:16.810494Z" 2023-07-29T00:09:16.943 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:09:16.943 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:09:16.944 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:09:16.944 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T00:09:16.944 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:09:16.944 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:09:16.944 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:09:16.945 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T00:09:16.945 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:09:16.945 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T00:09:16.946 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690587969.233742_99.1', 1: '1_1690589356.810494_732.1', 2: '', 3: ''} 2023-07-29T00:09:16.946 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T00:09:16.947 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:09:16.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:09:16.948 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:09:16.994 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:09:16.994 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:09:17.022 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:09:16.991+0000 7fbcd9936880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:09:17.024 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:09:16.992+0000 7fbcd9936880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:09:17.035 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:09:17.036 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:09:17.036 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:09:17.036 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:09:17.036 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:09:17.037 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:09:17.037 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:09:17.037 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:09:17.037 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:09:17.037 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:09:17.038 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:09:17.038 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:09:17.038 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:09:17.038 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:09:17.039 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:09:17.039 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:09:17.039 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:09:17.039 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:09:17.039 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:09:17.039 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:09:17.039 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:09:17.040 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:09:17.040 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:09:17.040 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:09:17.040 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:09:17.040 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690589313.995996_717.1", 2023-07-29T00:09:17.040 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:09:17.041 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:09:17.041 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:09:17.041 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:08:33.995996Z", 2023-07-29T00:09:17.041 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:09:17.041 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:09:17.041 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:09:17.041 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:09:17.042 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:09:17.042 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:09:17.042 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:09:17.042 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:09:17.042 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:09:17.042 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:09:17.042 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:09:17.043 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:09:17.043 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:09:17.043 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:09:17.043 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:09:17.043 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:09:17.043 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:09:17.043 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:09:17.044 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:09:17.044 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:09:17.044 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:09:17.044 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:09:17.044 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:09:17.044 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:09:17.044 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:09:17.045 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:09:17.045 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:09:17.045 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:09:17.045 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:09:17.045 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:09:17.045 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:09:17.046 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:09:17.046 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:09:17.046 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:09:16Z" 2023-07-29T00:09:17.046 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:09:17.047 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690589313.995996_717.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:08:33.995996Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:09:16Z" } 2023-07-29T00:09:17.048 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690589313.995996_717.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:08:33.995996Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:09:17.048 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690589356.810494_732.1', 2: '', 3: ''} 2023-07-29T00:09:17.048 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690589313.995996_717.1', 2: '', 3: ''} 2023-07-29T00:09:17.049 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690589356.810494_732.1 target=1_1690589313.995996_717.1 2023-07-29T00:09:22.055 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:09:22.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:09:22.057 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:09:22.110 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:09:22.111 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:09:22.143 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:09:22.112+0000 7fdc26926880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:09:22.143 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:09:22.113+0000 7fdc26926880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:09:22.156 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:09:22.157 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:09:22.157 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:09:22.157 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:09:22.157 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:09:22.157 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:09:22.158 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:09:22.158 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:09:22.158 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:09:22.158 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:09:22.158 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:09:22.158 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:09:22.159 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:09:22.159 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:09:22.159 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:09:22.159 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:09:22.159 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:09:22.159 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:09:22.159 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:09:22.160 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:09:22.160 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:09:22.160 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:09:22.160 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:09:22.160 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:09:22.160 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:09:22.161 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690589356.810494_732.1", 2023-07-29T00:09:22.161 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:09:22.161 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:09:22.161 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:09:22.161 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:09:16.810494Z", 2023-07-29T00:09:22.161 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:09:22.161 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:09:22.162 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:09:22.162 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:09:22.162 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:09:22.162 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:09:22.162 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:09:22.162 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:09:22.163 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:09:22.163 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:09:22.163 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:09:22.163 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:09:22.163 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:09:22.163 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:09:22.163 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:09:22.164 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:09:22.164 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:09:22.164 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:09:22.164 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:09:22.164 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:09:22.164 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:09:22.164 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:09:22.165 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:09:22.165 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:09:22.165 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:09:22.165 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:09:22.165 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:09:22.165 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:09:22.165 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:09:22.166 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:09:22.166 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:09:22.166 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:09:22.166 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:09:22.166 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:09:22Z" 2023-07-29T00:09:22.166 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:09:22.167 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690589356.810494_732.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:09:16.810494Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:09:22Z" } 2023-07-29T00:09:22.168 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690589356.810494_732.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:09:16.810494Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:09:22.168 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690589356.810494_732.1', 2: '', 3: ''} 2023-07-29T00:09:22.168 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690589356.810494_732.1', 2: '', 3: ''} 2023-07-29T00:09:22.169 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T00:09:22.169 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:22.170 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:22.170 DEBUG:boto:Method: HEAD 2023-07-29T00:09:22.171 DEBUG:boto:Path: /hiiqos-48/ 2023-07-29T00:09:22.171 DEBUG:boto:Data: 2023-07-29T00:09:22.171 DEBUG:boto:Headers: {} 2023-07-29T00:09:22.172 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:22.172 DEBUG:boto:Port: 8000 2023-07-29T00:09:22.173 DEBUG:boto:Params: {} 2023-07-29T00:09:22.173 DEBUG:boto:Token: None 2023-07-29T00:09:22.174 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:22 GMT /hiiqos-48/ 2023-07-29T00:09:22.175 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2uqcfC3tXxwnjNS0gUHCL3nQp+A= 2023-07-29T00:09:22.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2uqcfC3tXxwnjNS0gUHCL3nQp+A=', 'Content-Length': '0'} 2023-07-29T00:09:22.178 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ead50fe368bbc546-0064c458b2-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:22.178 DEBUG:boto:path=/hiiqos-48/obj5 2023-07-29T00:09:22.179 DEBUG:boto:auth_path=/hiiqos-48/obj5 2023-07-29T00:09:22.179 DEBUG:boto:Method: PUT 2023-07-29T00:09:22.180 DEBUG:boto:Path: /hiiqos-48/obj5 2023-07-29T00:09:22.180 DEBUG:boto:Data: 2023-07-29T00:09:22.180 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T00:09:22.181 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:22.181 DEBUG:boto:Port: 8000 2023-07-29T00:09:22.181 DEBUG:boto:Params: {} 2023-07-29T00:09:22.182 DEBUG:boto:Token: None 2023-07-29T00:09:22.182 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 00:09:22 GMT /hiiqos-48/obj5 2023-07-29T00:09:22.183 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+9kQDG5LYsuPVYpjR2yuCNrVrck= 2023-07-29T00:09:22.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:09:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+9kQDG5LYsuPVYpjR2yuCNrVrck='} 2023-07-29T00:09:22.188 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f7f4f21f84b7ec14-0064c458b2-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:22.189 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:22.189 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:22.190 DEBUG:boto:Method: HEAD 2023-07-29T00:09:22.191 DEBUG:boto:Path: /hiiqos-48/ 2023-07-29T00:09:22.191 DEBUG:boto:Data: 2023-07-29T00:09:22.192 DEBUG:boto:Headers: {} 2023-07-29T00:09:22.192 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:22.193 DEBUG:boto:Port: 8000 2023-07-29T00:09:22.196 DEBUG:boto:Params: {} 2023-07-29T00:09:22.196 DEBUG:boto:Token: None 2023-07-29T00:09:22.197 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:22 GMT /hiiqos-48/ 2023-07-29T00:09:22.197 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2uqcfC3tXxwnjNS0gUHCL3nQp+A= 2023-07-29T00:09:22.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2uqcfC3tXxwnjNS0gUHCL3nQp+A=', 'Content-Length': '0'} 2023-07-29T00:09:22.200 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005041ffa2f8e5a709-0064c458b2-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:22.200 DEBUG:boto:path=/hiiqos-48/obj6 2023-07-29T00:09:22.201 DEBUG:boto:auth_path=/hiiqos-48/obj6 2023-07-29T00:09:22.201 DEBUG:boto:Method: PUT 2023-07-29T00:09:22.201 DEBUG:boto:Path: /hiiqos-48/obj6 2023-07-29T00:09:22.202 DEBUG:boto:Data: 2023-07-29T00:09:22.202 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T00:09:22.203 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:22.204 DEBUG:boto:Port: 8000 2023-07-29T00:09:22.204 DEBUG:boto:Params: {} 2023-07-29T00:09:22.205 DEBUG:boto:Token: None 2023-07-29T00:09:22.205 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 00:09:22 GMT /hiiqos-48/obj6 2023-07-29T00:09:22.206 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VnJpzzE+1CEeQd+oAVc5u7YfhK4= 2023-07-29T00:09:22.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:09:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VnJpzzE+1CEeQd+oAVc5u7YfhK4='} 2023-07-29T00:09:22.252 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fa5875bba1d50f45-0064c458b2-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:22.253 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:22.253 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:22.254 DEBUG:boto:Method: HEAD 2023-07-29T00:09:22.254 DEBUG:boto:Path: /hiiqos-48/ 2023-07-29T00:09:22.255 DEBUG:boto:Data: 2023-07-29T00:09:22.255 DEBUG:boto:Headers: {} 2023-07-29T00:09:22.255 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:22.256 DEBUG:boto:Port: 8000 2023-07-29T00:09:22.256 DEBUG:boto:Params: {} 2023-07-29T00:09:22.256 DEBUG:boto:Token: None 2023-07-29T00:09:22.257 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:22 GMT /hiiqos-48/ 2023-07-29T00:09:22.258 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2uqcfC3tXxwnjNS0gUHCL3nQp+A= 2023-07-29T00:09:22.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2uqcfC3tXxwnjNS0gUHCL3nQp+A=', 'Content-Length': '0'} 2023-07-29T00:09:22.260 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dcf0d81e9c39c914-0064c458b2-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:22.261 DEBUG:boto:path=/hiiqos-48/obj7 2023-07-29T00:09:22.261 DEBUG:boto:auth_path=/hiiqos-48/obj7 2023-07-29T00:09:22.262 DEBUG:boto:Method: PUT 2023-07-29T00:09:22.262 DEBUG:boto:Path: /hiiqos-48/obj7 2023-07-29T00:09:22.262 DEBUG:boto:Data: 2023-07-29T00:09:22.263 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T00:09:22.263 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:22.264 DEBUG:boto:Port: 8000 2023-07-29T00:09:22.264 DEBUG:boto:Params: {} 2023-07-29T00:09:22.264 DEBUG:boto:Token: None 2023-07-29T00:09:22.265 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 00:09:22 GMT /hiiqos-48/obj7 2023-07-29T00:09:22.265 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WugM1gJWv7DNn+OjygDVa3IAbfA= 2023-07-29T00:09:22.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:09:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WugM1gJWv7DNn+OjygDVa3IAbfA='} 2023-07-29T00:09:22.270 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000edde86c2214c0034-0064c458b2-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:22.271 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:22.272 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:22.273 DEBUG:boto:Method: HEAD 2023-07-29T00:09:22.273 DEBUG:boto:Path: /hiiqos-48/ 2023-07-29T00:09:22.274 DEBUG:boto:Data: 2023-07-29T00:09:22.274 DEBUG:boto:Headers: {} 2023-07-29T00:09:22.274 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:22.275 DEBUG:boto:Port: 8000 2023-07-29T00:09:22.275 DEBUG:boto:Params: {} 2023-07-29T00:09:22.275 DEBUG:boto:Token: None 2023-07-29T00:09:22.276 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:22 GMT /hiiqos-48/ 2023-07-29T00:09:22.276 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2uqcfC3tXxwnjNS0gUHCL3nQp+A= 2023-07-29T00:09:22.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2uqcfC3tXxwnjNS0gUHCL3nQp+A=', 'Content-Length': '0'} 2023-07-29T00:09:22.279 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000313df6ee7146223f-0064c458b2-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:22.279 DEBUG:boto:path=/hiiqos-48/obj8 2023-07-29T00:09:22.280 DEBUG:boto:auth_path=/hiiqos-48/obj8 2023-07-29T00:09:22.280 DEBUG:boto:Method: PUT 2023-07-29T00:09:22.281 DEBUG:boto:Path: /hiiqos-48/obj8 2023-07-29T00:09:22.281 DEBUG:boto:Data: 2023-07-29T00:09:22.281 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T00:09:22.282 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:22.282 DEBUG:boto:Port: 8000 2023-07-29T00:09:22.283 DEBUG:boto:Params: {} 2023-07-29T00:09:22.284 DEBUG:boto:Token: None 2023-07-29T00:09:22.284 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 00:09:22 GMT /hiiqos-48/obj8 2023-07-29T00:09:22.285 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pisRyTn5HNJvdwDfD+ChbEaFUVY= 2023-07-29T00:09:22.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:09:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pisRyTn5HNJvdwDfD+ChbEaFUVY='} 2023-07-29T00:09:22.328 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ddbda69ea518fae6-0064c458b2-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:22.329 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:09:22.330 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:09:22.330 DEBUG:boto:Method: HEAD 2023-07-29T00:09:22.330 DEBUG:boto:Path: /hiiqos-49/ 2023-07-29T00:09:22.331 DEBUG:boto:Data: 2023-07-29T00:09:22.331 DEBUG:boto:Headers: {} 2023-07-29T00:09:22.331 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:22.332 DEBUG:boto:Port: 8001 2023-07-29T00:09:22.332 DEBUG:boto:Params: {} 2023-07-29T00:09:22.333 DEBUG:boto:Token: None 2023-07-29T00:09:22.333 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:22 GMT /hiiqos-49/ 2023-07-29T00:09:22.333 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/aZnBO648HIRw/GBuqShDMoVL+U= 2023-07-29T00:09:22.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/aZnBO648HIRw/GBuqShDMoVL+U=', 'Content-Length': '0'} 2023-07-29T00:09:22.336 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a1c8c5fe015ce88f-0064c458b2-1663-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:22.337 DEBUG:boto:path=/hiiqos-49/obj5 2023-07-29T00:09:22.337 DEBUG:boto:auth_path=/hiiqos-49/obj5 2023-07-29T00:09:22.337 DEBUG:boto:Method: PUT 2023-07-29T00:09:22.338 DEBUG:boto:Path: /hiiqos-49/obj5 2023-07-29T00:09:22.338 DEBUG:boto:Data: 2023-07-29T00:09:22.338 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T00:09:22.339 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:22.339 DEBUG:boto:Port: 8001 2023-07-29T00:09:22.339 DEBUG:boto:Params: {} 2023-07-29T00:09:22.340 DEBUG:boto:Token: None 2023-07-29T00:09:22.340 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 00:09:22 GMT /hiiqos-49/obj5 2023-07-29T00:09:22.341 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KOLvQYdSSdrSrE2YDjBfK2EDv4I= 2023-07-29T00:09:22.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:09:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KOLvQYdSSdrSrE2YDjBfK2EDv4I='} 2023-07-29T00:09:22.345 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fb63a93a2a6bae68-0064c458b2-1663-a2'), ('Date', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:22.346 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:09:22.346 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:09:22.347 DEBUG:boto:Method: HEAD 2023-07-29T00:09:22.347 DEBUG:boto:Path: /hiiqos-49/ 2023-07-29T00:09:22.348 DEBUG:boto:Data: 2023-07-29T00:09:22.348 DEBUG:boto:Headers: {} 2023-07-29T00:09:22.348 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:22.349 DEBUG:boto:Port: 8001 2023-07-29T00:09:22.349 DEBUG:boto:Params: {} 2023-07-29T00:09:22.349 DEBUG:boto:Token: None 2023-07-29T00:09:22.350 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:22 GMT /hiiqos-49/ 2023-07-29T00:09:22.350 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/aZnBO648HIRw/GBuqShDMoVL+U= 2023-07-29T00:09:22.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/aZnBO648HIRw/GBuqShDMoVL+U=', 'Content-Length': '0'} 2023-07-29T00:09:22.353 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d294523e44f92a0e-0064c458b2-1663-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:22.354 DEBUG:boto:path=/hiiqos-49/obj6 2023-07-29T00:09:22.354 DEBUG:boto:auth_path=/hiiqos-49/obj6 2023-07-29T00:09:22.355 DEBUG:boto:Method: PUT 2023-07-29T00:09:22.355 DEBUG:boto:Path: /hiiqos-49/obj6 2023-07-29T00:09:22.355 DEBUG:boto:Data: 2023-07-29T00:09:22.356 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T00:09:22.356 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:22.356 DEBUG:boto:Port: 8001 2023-07-29T00:09:22.357 DEBUG:boto:Params: {} 2023-07-29T00:09:22.357 DEBUG:boto:Token: None 2023-07-29T00:09:22.358 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 00:09:22 GMT /hiiqos-49/obj6 2023-07-29T00:09:22.358 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+C6A/4SmKI/quT1uRyXaNZU0XUk= 2023-07-29T00:09:22.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:09:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+C6A/4SmKI/quT1uRyXaNZU0XUk='} 2023-07-29T00:09:22.400 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002848c5ad62319945-0064c458b2-1663-a2'), ('Date', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:22.402 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:09:22.403 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:09:22.403 DEBUG:boto:Method: HEAD 2023-07-29T00:09:22.404 DEBUG:boto:Path: /hiiqos-49/ 2023-07-29T00:09:22.405 DEBUG:boto:Data: 2023-07-29T00:09:22.406 DEBUG:boto:Headers: {} 2023-07-29T00:09:22.406 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:22.406 DEBUG:boto:Port: 8001 2023-07-29T00:09:22.407 DEBUG:boto:Params: {} 2023-07-29T00:09:22.407 DEBUG:boto:Token: None 2023-07-29T00:09:22.408 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:22 GMT /hiiqos-49/ 2023-07-29T00:09:22.408 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/aZnBO648HIRw/GBuqShDMoVL+U= 2023-07-29T00:09:22.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/aZnBO648HIRw/GBuqShDMoVL+U=', 'Content-Length': '0'} 2023-07-29T00:09:22.411 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000df194c906f818899-0064c458b2-1663-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:22.412 DEBUG:boto:path=/hiiqos-49/obj7 2023-07-29T00:09:22.412 DEBUG:boto:auth_path=/hiiqos-49/obj7 2023-07-29T00:09:22.413 DEBUG:boto:Method: PUT 2023-07-29T00:09:22.413 DEBUG:boto:Path: /hiiqos-49/obj7 2023-07-29T00:09:22.413 DEBUG:boto:Data: 2023-07-29T00:09:22.414 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T00:09:22.414 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:22.414 DEBUG:boto:Port: 8001 2023-07-29T00:09:22.415 DEBUG:boto:Params: {} 2023-07-29T00:09:22.415 DEBUG:boto:Token: None 2023-07-29T00:09:22.415 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 00:09:22 GMT /hiiqos-49/obj7 2023-07-29T00:09:22.416 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LpIYciJ1zYpQCG2fGl3dbS60NFg= 2023-07-29T00:09:22.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:09:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LpIYciJ1zYpQCG2fGl3dbS60NFg='} 2023-07-29T00:09:22.421 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a027fbd5416f7c09-0064c458b2-1663-a2'), ('Date', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:22.422 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:09:22.422 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:09:22.422 DEBUG:boto:Method: HEAD 2023-07-29T00:09:22.423 DEBUG:boto:Path: /hiiqos-49/ 2023-07-29T00:09:22.423 DEBUG:boto:Data: 2023-07-29T00:09:22.423 DEBUG:boto:Headers: {} 2023-07-29T00:09:22.424 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:22.424 DEBUG:boto:Port: 8001 2023-07-29T00:09:22.424 DEBUG:boto:Params: {} 2023-07-29T00:09:22.425 DEBUG:boto:Token: None 2023-07-29T00:09:22.425 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:22 GMT /hiiqos-49/ 2023-07-29T00:09:22.426 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/aZnBO648HIRw/GBuqShDMoVL+U= 2023-07-29T00:09:22.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/aZnBO648HIRw/GBuqShDMoVL+U=', 'Content-Length': '0'} 2023-07-29T00:09:22.428 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000522085aee1d08c6c-0064c458b2-1663-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:22.429 DEBUG:boto:path=/hiiqos-49/obj8 2023-07-29T00:09:22.429 DEBUG:boto:auth_path=/hiiqos-49/obj8 2023-07-29T00:09:22.430 DEBUG:boto:Method: PUT 2023-07-29T00:09:22.430 DEBUG:boto:Path: /hiiqos-49/obj8 2023-07-29T00:09:22.431 DEBUG:boto:Data: 2023-07-29T00:09:22.431 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T00:09:22.431 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:22.432 DEBUG:boto:Port: 8001 2023-07-29T00:09:22.432 DEBUG:boto:Params: {} 2023-07-29T00:09:22.433 DEBUG:boto:Token: None 2023-07-29T00:09:22.433 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 00:09:22 GMT /hiiqos-49/obj8 2023-07-29T00:09:22.433 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vtu8qJCXiIOyTIbMc2md0VaTq60= 2023-07-29T00:09:22.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:09:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vtu8qJCXiIOyTIbMc2md0VaTq60='} 2023-07-29T00:09:22.480 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007685de25c3616acd-0064c458b2-1663-a2'), ('Date', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:22.481 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'hiiqos-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:09:22.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'hiiqos-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:09:22.482 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket hiiqos-48 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:09:22.531 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:09:22.532 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:09:22.595 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'hiiqos-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:09:22.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'hiiqos-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:09:22.597 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket hiiqos-49 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:09:22.643 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:09:22.644 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:09:22.706 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-48 2023-07-29T00:09:22.706 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:09:22.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:09:22.707 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-48 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:09:22.759 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:09:22.760 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:09:22.793 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:09:22.762+0000 7f3118c13880 1 waiting for incremental sync to catch up: 2023-07-29T00:09:22.793 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000001.30.6, , 00000000001.22.6, , 00000000001.20.6, , , 00000000001.45.6, , , ] 2023-07-29T00:09:22.793 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000004.34.1, 00000000002.20.1, 00000000004.29.1, 00000000002.22.1, 00000000004.30.1, 00000000002.58.1, 00000000002.37.1, 00000000004.53.1, 00000000002.54.1, 00000000002.12.1, 00000000002.26.1] 2023-07-29T00:09:27.794 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:09:27.763+0000 7f3118c13880 1 waiting for incremental sync to catch up: 2023-07-29T00:09:27.794 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000001.30.6, , 00000000001.22.6, , 00000000001.20.6, , , 00000000001.45.6, , , ] 2023-07-29T00:09:27.795 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000004.34.1, 00000000002.20.1, 00000000004.29.1, 00000000002.22.1, 00000000004.30.1, 00000000002.58.1, 00000000002.37.1, 00000000004.53.1, 00000000002.54.1, 00000000002.12.1, 00000000002.26.1] 2023-07-29T00:09:32.796 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:09:32.764+0000 7f3118c13880 1 waiting for incremental sync to catch up: 2023-07-29T00:09:32.796 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000001.30.6, , 00000000001.22.6, , 00000000001.20.6, , , 00000000001.45.6, , , ] 2023-07-29T00:09:32.796 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000004.34.1, 00000000002.20.1, 00000000004.29.1, 00000000002.22.1, 00000000004.30.1, 00000000002.58.1, 00000000002.37.1, 00000000004.53.1, 00000000002.54.1, 00000000002.12.1, 00000000002.26.1] 2023-07-29T00:09:37.797 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:09:37.765+0000 7f3118c13880 1 bucket sync caught up with source: 2023-07-29T00:09:37.797 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000004.34.1, 00000000002.20.1, 00000000004.29.1, 00000000002.22.1, 00000000004.30.1, 00000000002.58.1, 00000000002.37.1, 00000000004.53.1, 00000000002.54.1, 00000000002.12.1, 00000000002.26.1] 2023-07-29T00:09:37.797 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000004.34.1, 00000000002.20.1, 00000000004.29.1, 00000000002.22.1, 00000000004.30.1, 00000000002.58.1, 00000000002.37.1, 00000000004.53.1, 00000000002.54.1, 00000000002.12.1, 00000000002.26.1] 2023-07-29T00:09:37.797 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:09:37.765+0000 7f3118c13880 0 bucket checkpoint complete 2023-07-29T00:09:37.813 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-48 2023-07-29T00:09:37.814 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:09:37.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:09:37.815 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-48 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:09:37.871 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:09:37.871 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:09:37.907 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:09:37.875+0000 7fb72d614880 1 bucket sync caught up with source: 2023-07-29T00:09:37.907 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000004.1341.1, 00000000002.527.1, 00000000004.211.1, 00000000002.983.1, 00000000004.212.1, 00000000002.869.1, 00000000002.546.1, 00000000004.401.1, 00000000002.403.1, 00000000002.350.1, 00000000002.1005.1] 2023-07-29T00:09:37.907 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000004.1341.1, 00000000002.527.1, 00000000004.211.1, 00000000002.983.1, 00000000004.212.1, 00000000002.869.1, 00000000002.546.1, 00000000004.401.1, 00000000002.403.1, 00000000002.350.1, 00000000002.1005.1] 2023-07-29T00:09:37.907 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:09:37.875+0000 7fb72d614880 0 bucket checkpoint complete 2023-07-29T00:09:37.920 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-48 zones={a2, a1} 2023-07-29T00:09:37.921 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:37.921 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:37.921 DEBUG:boto:Method: HEAD 2023-07-29T00:09:37.922 DEBUG:boto:Path: /hiiqos-48/ 2023-07-29T00:09:37.922 DEBUG:boto:Data: 2023-07-29T00:09:37.922 DEBUG:boto:Headers: {} 2023-07-29T00:09:37.923 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:37.923 DEBUG:boto:Port: 8001 2023-07-29T00:09:37.923 DEBUG:boto:Params: {} 2023-07-29T00:09:37.924 DEBUG:boto:Token: None 2023-07-29T00:09:37.924 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:37 GMT /hiiqos-48/ 2023-07-29T00:09:37.924 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aewJtry2fa0fx/zMV4uMXqagQgU= 2023-07-29T00:09:37.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aewJtry2fa0fx/zMV4uMXqagQgU=', 'Content-Length': '0'} 2023-07-29T00:09:37.927 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c459b7f5ab15275b-0064c458c1-1663-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:09:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:37.928 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:37.928 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:37.928 DEBUG:boto:Method: HEAD 2023-07-29T00:09:37.929 DEBUG:boto:Path: /hiiqos-48/ 2023-07-29T00:09:37.929 DEBUG:boto:Data: 2023-07-29T00:09:37.929 DEBUG:boto:Headers: {} 2023-07-29T00:09:37.930 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:37.930 DEBUG:boto:Port: 8000 2023-07-29T00:09:37.930 DEBUG:boto:Params: {} 2023-07-29T00:09:37.931 DEBUG:boto:Token: None 2023-07-29T00:09:37.931 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:37 GMT /hiiqos-48/ 2023-07-29T00:09:37.932 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aewJtry2fa0fx/zMV4uMXqagQgU= 2023-07-29T00:09:37.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aewJtry2fa0fx/zMV4uMXqagQgU=', 'Content-Length': '0'} 2023-07-29T00:09:37.934 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000093c67a97c81d9b0f-0064c458c1-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:09:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:37.934 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-29T00:09:37.935 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:37.935 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:37.935 DEBUG:boto:path=/hiiqos-48/?versions&max-keys=999 2023-07-29T00:09:37.936 DEBUG:boto:auth_path=/hiiqos-48/?versions&max-keys=999 2023-07-29T00:09:37.936 DEBUG:boto:Method: GET 2023-07-29T00:09:37.937 DEBUG:boto:Path: /hiiqos-48/?versions&max-keys=999 2023-07-29T00:09:37.937 DEBUG:boto:Data: 2023-07-29T00:09:37.937 DEBUG:boto:Headers: {} 2023-07-29T00:09:37.938 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:37.938 DEBUG:boto:Port: 8001 2023-07-29T00:09:37.938 DEBUG:boto:Params: {} 2023-07-29T00:09:37.939 DEBUG:boto:Token: None 2023-07-29T00:09:37.939 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:37 GMT /hiiqos-48/?versions 2023-07-29T00:09:37.939 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ap6Js0byBpQHgyu6fz4H2UkZkXc= 2023-07-29T00:09:37.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ap6Js0byBpQHgyu6fz4H2UkZkXc=', 'Content-Length': '0'} 2023-07-29T00:09:37.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb137a436b35a546-0064c458c1-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:37.942 DEBUG:boto:b'hiiqos-48999falseobj1nulltrue2023-07-29T00:08:34.018Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-07-29T00:08:34.078Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-07-29T00:08:34.094Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-07-29T00:08:34.109Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-07-29T00:09:22.155Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-07-29T00:09:22.178Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-07-29T00:09:22.237Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-07-29T00:09:22.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:37.943 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-07-29T00:09:37.944 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-07-29T00:09:37.944 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-07-29T00:09:37.944 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-07-29T00:09:37.944 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-07-29T00:09:37.945 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-07-29T00:09:37.945 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-07-29T00:09:37.945 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-07-29T00:09:37.946 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-29T00:09:37.946 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:37.946 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:37.947 DEBUG:boto:path=/hiiqos-48/?versions&max-keys=999 2023-07-29T00:09:37.947 DEBUG:boto:auth_path=/hiiqos-48/?versions&max-keys=999 2023-07-29T00:09:37.947 DEBUG:boto:Method: GET 2023-07-29T00:09:37.948 DEBUG:boto:Path: /hiiqos-48/?versions&max-keys=999 2023-07-29T00:09:37.948 DEBUG:boto:Data: 2023-07-29T00:09:37.948 DEBUG:boto:Headers: {} 2023-07-29T00:09:37.949 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:37.949 DEBUG:boto:Port: 8000 2023-07-29T00:09:37.949 DEBUG:boto:Params: {} 2023-07-29T00:09:37.950 DEBUG:boto:Token: None 2023-07-29T00:09:37.950 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:37 GMT /hiiqos-48/?versions 2023-07-29T00:09:37.951 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ap6Js0byBpQHgyu6fz4H2UkZkXc= 2023-07-29T00:09:37.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ap6Js0byBpQHgyu6fz4H2UkZkXc=', 'Content-Length': '0'} 2023-07-29T00:09:37.953 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005de96331933cb69-0064c458c1-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:37.954 DEBUG:boto:b'hiiqos-48999falseobj1nulltrue2023-07-29T00:08:34.018Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-07-29T00:08:34.078Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-07-29T00:08:34.094Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-07-29T00:08:34.109Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-07-29T00:09:22.155Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-07-29T00:09:22.178Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-07-29T00:09:22.237Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-07-29T00:09:22.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:37.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-07-29T00:09:37.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-07-29T00:09:37.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-07-29T00:09:37.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-07-29T00:09:37.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-07-29T00:09:37.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-07-29T00:09:37.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-07-29T00:09:37.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-07-29T00:09:37.957 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:37.957 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:37.958 DEBUG:boto:path=/hiiqos-48/?versions&max-keys=999 2023-07-29T00:09:37.958 DEBUG:boto:auth_path=/hiiqos-48/?versions&max-keys=999 2023-07-29T00:09:37.958 DEBUG:boto:Method: GET 2023-07-29T00:09:37.959 DEBUG:boto:Path: /hiiqos-48/?versions&max-keys=999 2023-07-29T00:09:37.959 DEBUG:boto:Data: 2023-07-29T00:09:37.959 DEBUG:boto:Headers: {} 2023-07-29T00:09:37.960 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:37.960 DEBUG:boto:Port: 8001 2023-07-29T00:09:37.960 DEBUG:boto:Params: {} 2023-07-29T00:09:37.961 DEBUG:boto:Token: None 2023-07-29T00:09:37.961 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:37 GMT /hiiqos-48/?versions 2023-07-29T00:09:37.961 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ap6Js0byBpQHgyu6fz4H2UkZkXc= 2023-07-29T00:09:37.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ap6Js0byBpQHgyu6fz4H2UkZkXc=', 'Content-Length': '0'} 2023-07-29T00:09:37.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000954ea876e8f3fc17-0064c458c1-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:37.964 DEBUG:boto:b'hiiqos-48999falseobj1nulltrue2023-07-29T00:08:34.018Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-07-29T00:08:34.078Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-07-29T00:08:34.094Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-07-29T00:08:34.109Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-07-29T00:09:22.155Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-07-29T00:09:22.178Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-07-29T00:09:22.237Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-07-29T00:09:22.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:37.965 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:37.966 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:37.966 DEBUG:boto:path=/hiiqos-48/?versions&max-keys=999 2023-07-29T00:09:37.966 DEBUG:boto:auth_path=/hiiqos-48/?versions&max-keys=999 2023-07-29T00:09:37.967 DEBUG:boto:Method: GET 2023-07-29T00:09:37.967 DEBUG:boto:Path: /hiiqos-48/?versions&max-keys=999 2023-07-29T00:09:37.967 DEBUG:boto:Data: 2023-07-29T00:09:37.968 DEBUG:boto:Headers: {} 2023-07-29T00:09:37.968 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:37.968 DEBUG:boto:Port: 8000 2023-07-29T00:09:37.969 DEBUG:boto:Params: {} 2023-07-29T00:09:37.969 DEBUG:boto:Token: None 2023-07-29T00:09:37.969 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:37 GMT /hiiqos-48/?versions 2023-07-29T00:09:37.970 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ap6Js0byBpQHgyu6fz4H2UkZkXc= 2023-07-29T00:09:37.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ap6Js0byBpQHgyu6fz4H2UkZkXc=', 'Content-Length': '0'} 2023-07-29T00:09:37.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca105b4747731746-0064c458c1-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:37.973 DEBUG:boto:b'hiiqos-48999falseobj1nulltrue2023-07-29T00:08:34.018Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-07-29T00:08:34.078Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-07-29T00:08:34.094Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-07-29T00:08:34.109Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-07-29T00:09:22.155Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-07-29T00:09:22.178Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-07-29T00:09:22.237Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-07-29T00:09:22.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:37.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-07-29T00:09:37.974 DEBUG:boto:path=/hiiqos-48/obj1 2023-07-29T00:09:37.974 DEBUG:boto:auth_path=/hiiqos-48/obj1 2023-07-29T00:09:37.975 DEBUG:boto:path=/hiiqos-48/obj1?versionId=null 2023-07-29T00:09:37.975 DEBUG:boto:auth_path=/hiiqos-48/obj1?versionId=null 2023-07-29T00:09:37.975 DEBUG:boto:Method: GET 2023-07-29T00:09:37.976 DEBUG:boto:Path: /hiiqos-48/obj1?versionId=null 2023-07-29T00:09:37.976 DEBUG:boto:Data: 2023-07-29T00:09:37.976 DEBUG:boto:Headers: {} 2023-07-29T00:09:37.977 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:37.977 DEBUG:boto:Port: 8001 2023-07-29T00:09:37.978 DEBUG:boto:Params: {} 2023-07-29T00:09:37.978 DEBUG:boto:Token: None 2023-07-29T00:09:37.978 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:37 GMT /hiiqos-48/obj1?versionId=null 2023-07-29T00:09:37.979 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dH69L4v4kTv+8ftUK/y0Z8iCw0Y= 2023-07-29T00:09:37.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dH69L4v4kTv+8ftUK/y0Z8iCw0Y=', 'Content-Length': '0'} 2023-07-29T00:09:37.981 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c14ada33ad4ec424-0064c458c1-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:37.981 DEBUG:boto:path=/hiiqos-48/obj1 2023-07-29T00:09:37.982 DEBUG:boto:auth_path=/hiiqos-48/obj1 2023-07-29T00:09:37.982 DEBUG:boto:path=/hiiqos-48/obj1?versionId=null 2023-07-29T00:09:37.983 DEBUG:boto:auth_path=/hiiqos-48/obj1?versionId=null 2023-07-29T00:09:37.983 DEBUG:boto:Method: GET 2023-07-29T00:09:37.983 DEBUG:boto:Path: /hiiqos-48/obj1?versionId=null 2023-07-29T00:09:37.984 DEBUG:boto:Data: 2023-07-29T00:09:37.984 DEBUG:boto:Headers: {} 2023-07-29T00:09:37.984 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:37.985 DEBUG:boto:Port: 8000 2023-07-29T00:09:37.985 DEBUG:boto:Params: {} 2023-07-29T00:09:37.985 DEBUG:boto:Token: None 2023-07-29T00:09:37.986 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:37 GMT /hiiqos-48/obj1?versionId=null 2023-07-29T00:09:37.986 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dH69L4v4kTv+8ftUK/y0Z8iCw0Y= 2023-07-29T00:09:37.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dH69L4v4kTv+8ftUK/y0Z8iCw0Y=', 'Content-Length': '0'} 2023-07-29T00:09:37.988 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000009846e2d84a96d10-0064c458c1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:37.989 DEBUG:boto:path=/hiiqos-48/obj1 2023-07-29T00:09:37.989 DEBUG:boto:auth_path=/hiiqos-48/obj1 2023-07-29T00:09:37.990 DEBUG:boto:path=/hiiqos-48/obj1?versionId=null 2023-07-29T00:09:37.990 DEBUG:boto:auth_path=/hiiqos-48/obj1?versionId=null 2023-07-29T00:09:37.990 DEBUG:boto:Method: HEAD 2023-07-29T00:09:37.991 DEBUG:boto:Path: /hiiqos-48/obj1?versionId=null 2023-07-29T00:09:37.991 DEBUG:boto:Data: 2023-07-29T00:09:37.991 DEBUG:boto:Headers: {} 2023-07-29T00:09:37.992 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:37.992 DEBUG:boto:Port: 8001 2023-07-29T00:09:37.992 DEBUG:boto:Params: {} 2023-07-29T00:09:37.993 DEBUG:boto:Token: None 2023-07-29T00:09:37.993 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:37 GMT /hiiqos-48/obj1?versionId=null 2023-07-29T00:09:37.993 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:X9IdxPiKVD9Sf5GwmnFJnnAapv0= 2023-07-29T00:09:37.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:X9IdxPiKVD9Sf5GwmnFJnnAapv0=', 'Content-Length': '0'} 2023-07-29T00:09:37.995 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006c591ba6eaea2aa7-0064c458c1-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:37.996 DEBUG:boto:path=/hiiqos-48/obj1 2023-07-29T00:09:37.996 DEBUG:boto:auth_path=/hiiqos-48/obj1 2023-07-29T00:09:37.996 DEBUG:boto:path=/hiiqos-48/obj1?versionId=null 2023-07-29T00:09:37.997 DEBUG:boto:auth_path=/hiiqos-48/obj1?versionId=null 2023-07-29T00:09:37.997 DEBUG:boto:Method: HEAD 2023-07-29T00:09:37.997 DEBUG:boto:Path: /hiiqos-48/obj1?versionId=null 2023-07-29T00:09:37.998 DEBUG:boto:Data: 2023-07-29T00:09:37.998 DEBUG:boto:Headers: {} 2023-07-29T00:09:37.998 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:37.999 DEBUG:boto:Port: 8000 2023-07-29T00:09:37.999 DEBUG:boto:Params: {} 2023-07-29T00:09:37.999 DEBUG:boto:Token: None 2023-07-29T00:09:38.000 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj1?versionId=null 2023-07-29T00:09:38.000 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HYONKIHtx6l5lftJW1eO5ukJ+ow= 2023-07-29T00:09:38.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HYONKIHtx6l5lftJW1eO5ukJ+ow=', 'Content-Length': '0'} 2023-07-29T00:09:38.002 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ca4c0c81646541ed-0064c458c1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-07-29T00:09:38.003 DEBUG:boto:path=/hiiqos-48/obj1 2023-07-29T00:09:38.003 DEBUG:boto:auth_path=/hiiqos-48/obj1 2023-07-29T00:09:38.004 DEBUG:boto:path=/hiiqos-48/obj1?versionId=null 2023-07-29T00:09:38.004 DEBUG:boto:auth_path=/hiiqos-48/obj1?versionId=null 2023-07-29T00:09:38.004 DEBUG:boto:Method: GET 2023-07-29T00:09:38.005 DEBUG:boto:Path: /hiiqos-48/obj1?versionId=null 2023-07-29T00:09:38.005 DEBUG:boto:Data: 2023-07-29T00:09:38.005 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.006 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.006 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.006 DEBUG:boto:Params: {} 2023-07-29T00:09:38.007 DEBUG:boto:Token: None 2023-07-29T00:09:38.007 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj1?versionId=null 2023-07-29T00:09:38.008 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LuEjj2csAbNnGq0I4QlaMrn7gv8= 2023-07-29T00:09:38.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LuEjj2csAbNnGq0I4QlaMrn7gv8=', 'Content-Length': '0'} 2023-07-29T00:09:38.010 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000033b55746be1547dc-0064c458c1-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.010 DEBUG:boto:path=/hiiqos-48/obj1 2023-07-29T00:09:38.011 DEBUG:boto:auth_path=/hiiqos-48/obj1 2023-07-29T00:09:38.011 DEBUG:boto:path=/hiiqos-48/obj1?versionId=null 2023-07-29T00:09:38.011 DEBUG:boto:auth_path=/hiiqos-48/obj1?versionId=null 2023-07-29T00:09:38.012 DEBUG:boto:Method: GET 2023-07-29T00:09:38.012 DEBUG:boto:Path: /hiiqos-48/obj1?versionId=null 2023-07-29T00:09:38.013 DEBUG:boto:Data: 2023-07-29T00:09:38.013 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.013 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.014 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.014 DEBUG:boto:Params: {} 2023-07-29T00:09:38.014 DEBUG:boto:Token: None 2023-07-29T00:09:38.015 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj1?versionId=null 2023-07-29T00:09:38.015 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LuEjj2csAbNnGq0I4QlaMrn7gv8= 2023-07-29T00:09:38.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LuEjj2csAbNnGq0I4QlaMrn7gv8=', 'Content-Length': '0'} 2023-07-29T00:09:38.017 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009194c192b4104cb7-0064c458c1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.060 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:38.061 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:38.061 DEBUG:boto:path=/hiiqos-48/?prefix=obj1 2023-07-29T00:09:38.062 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj1 2023-07-29T00:09:38.062 DEBUG:boto:Method: GET 2023-07-29T00:09:38.062 DEBUG:boto:Path: /hiiqos-48/?prefix=obj1 2023-07-29T00:09:38.063 DEBUG:boto:Data: 2023-07-29T00:09:38.063 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.063 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.064 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.064 DEBUG:boto:Params: {} 2023-07-29T00:09:38.065 DEBUG:boto:Token: None 2023-07-29T00:09:38.065 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/ 2023-07-29T00:09:38.065 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA= 2023-07-29T00:09:38.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA=', 'Content-Length': '0'} 2023-07-29T00:09:38.068 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000866f5f0ed9d83b20-0064c458c2-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.068 DEBUG:boto:b'hiiqos-48obj11000falseobj12023-07-29T00:08:34.018Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:38.069 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:38.069 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:38.070 DEBUG:boto:path=/hiiqos-48/?prefix=obj1 2023-07-29T00:09:38.070 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj1 2023-07-29T00:09:38.071 DEBUG:boto:Method: GET 2023-07-29T00:09:38.071 DEBUG:boto:Path: /hiiqos-48/?prefix=obj1 2023-07-29T00:09:38.071 DEBUG:boto:Data: 2023-07-29T00:09:38.072 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.072 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.072 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.073 DEBUG:boto:Params: {} 2023-07-29T00:09:38.073 DEBUG:boto:Token: None 2023-07-29T00:09:38.073 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/ 2023-07-29T00:09:38.074 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA= 2023-07-29T00:09:38.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA=', 'Content-Length': '0'} 2023-07-29T00:09:38.076 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f55bd08ce859793-0064c458c2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.077 DEBUG:boto:b'hiiqos-48obj11000falseobj12023-07-29T00:08:34.018Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:38.077 DEBUG:boto:path=/hiiqos-48/obj1 2023-07-29T00:09:38.078 DEBUG:boto:auth_path=/hiiqos-48/obj1 2023-07-29T00:09:38.078 DEBUG:boto:Method: HEAD 2023-07-29T00:09:38.078 DEBUG:boto:Path: /hiiqos-48/obj1 2023-07-29T00:09:38.079 DEBUG:boto:Data: 2023-07-29T00:09:38.079 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.079 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.080 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.080 DEBUG:boto:Params: {} 2023-07-29T00:09:38.081 DEBUG:boto:Token: None 2023-07-29T00:09:38.081 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj1 2023-07-29T00:09:38.081 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:34DzzKnOWo1Gy5NwYT//b4Ox674= 2023-07-29T00:09:38.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:34DzzKnOWo1Gy5NwYT//b4Ox674=', 'Content-Length': '0'} 2023-07-29T00:09:38.083 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ac70dc82197901a5-0064c458c2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.084 DEBUG:boto:path=/hiiqos-48/obj1 2023-07-29T00:09:38.084 DEBUG:boto:auth_path=/hiiqos-48/obj1 2023-07-29T00:09:38.084 DEBUG:boto:Method: HEAD 2023-07-29T00:09:38.085 DEBUG:boto:Path: /hiiqos-48/obj1 2023-07-29T00:09:38.085 DEBUG:boto:Data: 2023-07-29T00:09:38.085 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.086 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.086 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.086 DEBUG:boto:Params: {} 2023-07-29T00:09:38.087 DEBUG:boto:Token: None 2023-07-29T00:09:38.087 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj1 2023-07-29T00:09:38.088 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:34DzzKnOWo1Gy5NwYT//b4Ox674= 2023-07-29T00:09:38.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:34DzzKnOWo1Gy5NwYT//b4Ox674=', 'Content-Length': '0'} 2023-07-29T00:09:38.090 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007e6060cfd96b082-0064c458c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-07-29T00:09:38.091 DEBUG:boto:path=/hiiqos-48/obj1 2023-07-29T00:09:38.091 DEBUG:boto:auth_path=/hiiqos-48/obj1 2023-07-29T00:09:38.091 DEBUG:boto:Method: GET 2023-07-29T00:09:38.092 DEBUG:boto:Path: /hiiqos-48/obj1 2023-07-29T00:09:38.092 DEBUG:boto:Data: 2023-07-29T00:09:38.092 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.093 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.093 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.093 DEBUG:boto:Params: {} 2023-07-29T00:09:38.094 DEBUG:boto:Token: None 2023-07-29T00:09:38.094 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj1 2023-07-29T00:09:38.095 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aXd/T6XZtl1DV6tMyXFhwNPVQeI= 2023-07-29T00:09:38.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aXd/T6XZtl1DV6tMyXFhwNPVQeI=', 'Content-Length': '0'} 2023-07-29T00:09:38.096 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cdec22355bc6a7ad-0064c458c2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.097 DEBUG:boto:path=/hiiqos-48/obj1 2023-07-29T00:09:38.097 DEBUG:boto:auth_path=/hiiqos-48/obj1 2023-07-29T00:09:38.098 DEBUG:boto:Method: GET 2023-07-29T00:09:38.098 DEBUG:boto:Path: /hiiqos-48/obj1 2023-07-29T00:09:38.099 DEBUG:boto:Data: 2023-07-29T00:09:38.099 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.099 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.100 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.100 DEBUG:boto:Params: {} 2023-07-29T00:09:38.100 DEBUG:boto:Token: None 2023-07-29T00:09:38.101 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj1 2023-07-29T00:09:38.101 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aXd/T6XZtl1DV6tMyXFhwNPVQeI= 2023-07-29T00:09:38.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aXd/T6XZtl1DV6tMyXFhwNPVQeI=', 'Content-Length': '0'} 2023-07-29T00:09:38.103 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9ec62a4b4d1be98-0064c458c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.104 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:38.104 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:38.104 DEBUG:boto:path=/hiiqos-48/?prefix=obj1 2023-07-29T00:09:38.105 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj1 2023-07-29T00:09:38.105 DEBUG:boto:Method: GET 2023-07-29T00:09:38.105 DEBUG:boto:Path: /hiiqos-48/?prefix=obj1 2023-07-29T00:09:38.106 DEBUG:boto:Data: 2023-07-29T00:09:38.106 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.107 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.107 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.107 DEBUG:boto:Params: {} 2023-07-29T00:09:38.108 DEBUG:boto:Token: None 2023-07-29T00:09:38.108 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/ 2023-07-29T00:09:38.108 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA= 2023-07-29T00:09:38.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA=', 'Content-Length': '0'} 2023-07-29T00:09:38.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e4ab95cde7582c9-0064c458c2-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.111 DEBUG:boto:b'hiiqos-48obj11000falseobj12023-07-29T00:08:34.018Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:38.112 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:38.112 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:38.113 DEBUG:boto:path=/hiiqos-48/?prefix=obj1 2023-07-29T00:09:38.113 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj1 2023-07-29T00:09:38.113 DEBUG:boto:Method: GET 2023-07-29T00:09:38.114 DEBUG:boto:Path: /hiiqos-48/?prefix=obj1 2023-07-29T00:09:38.114 DEBUG:boto:Data: 2023-07-29T00:09:38.114 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.115 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.115 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.115 DEBUG:boto:Params: {} 2023-07-29T00:09:38.116 DEBUG:boto:Token: None 2023-07-29T00:09:38.116 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/ 2023-07-29T00:09:38.117 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA= 2023-07-29T00:09:38.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA=', 'Content-Length': '0'} 2023-07-29T00:09:38.119 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bddb3e9c1278058-0064c458c2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.119 DEBUG:boto:b'hiiqos-48obj11000falseobj12023-07-29T00:08:34.018Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:38.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-07-29T00:09:38.120 DEBUG:boto:path=/hiiqos-48/obj2 2023-07-29T00:09:38.121 DEBUG:boto:auth_path=/hiiqos-48/obj2 2023-07-29T00:09:38.121 DEBUG:boto:path=/hiiqos-48/obj2?versionId=null 2023-07-29T00:09:38.121 DEBUG:boto:auth_path=/hiiqos-48/obj2?versionId=null 2023-07-29T00:09:38.122 DEBUG:boto:Method: GET 2023-07-29T00:09:38.122 DEBUG:boto:Path: /hiiqos-48/obj2?versionId=null 2023-07-29T00:09:38.122 DEBUG:boto:Data: 2023-07-29T00:09:38.123 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.123 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.123 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.124 DEBUG:boto:Params: {} 2023-07-29T00:09:38.124 DEBUG:boto:Token: None 2023-07-29T00:09:38.125 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj2?versionId=null 2023-07-29T00:09:38.125 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Z+fw5wG3LTSQM9kXLG8THDpHT1M= 2023-07-29T00:09:38.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Z+fw5wG3LTSQM9kXLG8THDpHT1M=', 'Content-Length': '0'} 2023-07-29T00:09:38.127 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000085c476513357fb2a-0064c458c2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.127 DEBUG:boto:path=/hiiqos-48/obj2 2023-07-29T00:09:38.128 DEBUG:boto:auth_path=/hiiqos-48/obj2 2023-07-29T00:09:38.128 DEBUG:boto:path=/hiiqos-48/obj2?versionId=null 2023-07-29T00:09:38.129 DEBUG:boto:auth_path=/hiiqos-48/obj2?versionId=null 2023-07-29T00:09:38.129 DEBUG:boto:Method: GET 2023-07-29T00:09:38.129 DEBUG:boto:Path: /hiiqos-48/obj2?versionId=null 2023-07-29T00:09:38.130 DEBUG:boto:Data: 2023-07-29T00:09:38.130 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.130 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.131 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.131 DEBUG:boto:Params: {} 2023-07-29T00:09:38.131 DEBUG:boto:Token: None 2023-07-29T00:09:38.132 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj2?versionId=null 2023-07-29T00:09:38.132 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Z+fw5wG3LTSQM9kXLG8THDpHT1M= 2023-07-29T00:09:38.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Z+fw5wG3LTSQM9kXLG8THDpHT1M=', 'Content-Length': '0'} 2023-07-29T00:09:38.134 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1b7aca3dc739d7b-0064c458c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.135 DEBUG:boto:path=/hiiqos-48/obj2 2023-07-29T00:09:38.135 DEBUG:boto:auth_path=/hiiqos-48/obj2 2023-07-29T00:09:38.136 DEBUG:boto:path=/hiiqos-48/obj2?versionId=null 2023-07-29T00:09:38.136 DEBUG:boto:auth_path=/hiiqos-48/obj2?versionId=null 2023-07-29T00:09:38.136 DEBUG:boto:Method: HEAD 2023-07-29T00:09:38.137 DEBUG:boto:Path: /hiiqos-48/obj2?versionId=null 2023-07-29T00:09:38.137 DEBUG:boto:Data: 2023-07-29T00:09:38.137 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.138 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.138 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.138 DEBUG:boto:Params: {} 2023-07-29T00:09:38.139 DEBUG:boto:Token: None 2023-07-29T00:09:38.139 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj2?versionId=null 2023-07-29T00:09:38.139 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mdSBxzDcoNKOsxwDDGtDhHC+iXE= 2023-07-29T00:09:38.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mdSBxzDcoNKOsxwDDGtDhHC+iXE=', 'Content-Length': '0'} 2023-07-29T00:09:38.141 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f6545d69e0340d72-0064c458c2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.142 DEBUG:boto:path=/hiiqos-48/obj2 2023-07-29T00:09:38.142 DEBUG:boto:auth_path=/hiiqos-48/obj2 2023-07-29T00:09:38.143 DEBUG:boto:path=/hiiqos-48/obj2?versionId=null 2023-07-29T00:09:38.143 DEBUG:boto:auth_path=/hiiqos-48/obj2?versionId=null 2023-07-29T00:09:38.143 DEBUG:boto:Method: HEAD 2023-07-29T00:09:38.144 DEBUG:boto:Path: /hiiqos-48/obj2?versionId=null 2023-07-29T00:09:38.144 DEBUG:boto:Data: 2023-07-29T00:09:38.144 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.145 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.145 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.145 DEBUG:boto:Params: {} 2023-07-29T00:09:38.146 DEBUG:boto:Token: None 2023-07-29T00:09:38.146 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj2?versionId=null 2023-07-29T00:09:38.146 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mdSBxzDcoNKOsxwDDGtDhHC+iXE= 2023-07-29T00:09:38.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mdSBxzDcoNKOsxwDDGtDhHC+iXE=', 'Content-Length': '0'} 2023-07-29T00:09:38.148 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007b42cf315157ff7c-0064c458c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-07-29T00:09:38.149 DEBUG:boto:path=/hiiqos-48/obj2 2023-07-29T00:09:38.150 DEBUG:boto:auth_path=/hiiqos-48/obj2 2023-07-29T00:09:38.150 DEBUG:boto:path=/hiiqos-48/obj2?versionId=null 2023-07-29T00:09:38.150 DEBUG:boto:auth_path=/hiiqos-48/obj2?versionId=null 2023-07-29T00:09:38.151 DEBUG:boto:Method: GET 2023-07-29T00:09:38.151 DEBUG:boto:Path: /hiiqos-48/obj2?versionId=null 2023-07-29T00:09:38.151 DEBUG:boto:Data: 2023-07-29T00:09:38.152 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.152 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.152 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.153 DEBUG:boto:Params: {} 2023-07-29T00:09:38.153 DEBUG:boto:Token: None 2023-07-29T00:09:38.154 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj2?versionId=null 2023-07-29T00:09:38.154 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Z+fw5wG3LTSQM9kXLG8THDpHT1M= 2023-07-29T00:09:38.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Z+fw5wG3LTSQM9kXLG8THDpHT1M=', 'Content-Length': '0'} 2023-07-29T00:09:38.156 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000061035697c0483a0b-0064c458c2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.157 DEBUG:boto:path=/hiiqos-48/obj2 2023-07-29T00:09:38.157 DEBUG:boto:auth_path=/hiiqos-48/obj2 2023-07-29T00:09:38.158 DEBUG:boto:path=/hiiqos-48/obj2?versionId=null 2023-07-29T00:09:38.158 DEBUG:boto:auth_path=/hiiqos-48/obj2?versionId=null 2023-07-29T00:09:38.158 DEBUG:boto:Method: GET 2023-07-29T00:09:38.159 DEBUG:boto:Path: /hiiqos-48/obj2?versionId=null 2023-07-29T00:09:38.159 DEBUG:boto:Data: 2023-07-29T00:09:38.159 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.160 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.160 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.160 DEBUG:boto:Params: {} 2023-07-29T00:09:38.161 DEBUG:boto:Token: None 2023-07-29T00:09:38.161 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj2?versionId=null 2023-07-29T00:09:38.162 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Z+fw5wG3LTSQM9kXLG8THDpHT1M= 2023-07-29T00:09:38.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Z+fw5wG3LTSQM9kXLG8THDpHT1M=', 'Content-Length': '0'} 2023-07-29T00:09:38.164 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1616a599f985419-0064c458c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.208 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:38.209 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:38.209 DEBUG:boto:path=/hiiqos-48/?prefix=obj2 2023-07-29T00:09:38.210 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj2 2023-07-29T00:09:38.210 DEBUG:boto:Method: GET 2023-07-29T00:09:38.210 DEBUG:boto:Path: /hiiqos-48/?prefix=obj2 2023-07-29T00:09:38.211 DEBUG:boto:Data: 2023-07-29T00:09:38.211 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.211 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.212 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.212 DEBUG:boto:Params: {} 2023-07-29T00:09:38.212 DEBUG:boto:Token: None 2023-07-29T00:09:38.213 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/ 2023-07-29T00:09:38.213 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA= 2023-07-29T00:09:38.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA=', 'Content-Length': '0'} 2023-07-29T00:09:38.216 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008638912cbea61194-0064c458c2-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.216 DEBUG:boto:b'hiiqos-48obj21000falseobj22023-07-29T00:08:34.078Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:38.217 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:38.217 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:38.218 DEBUG:boto:path=/hiiqos-48/?prefix=obj2 2023-07-29T00:09:38.218 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj2 2023-07-29T00:09:38.219 DEBUG:boto:Method: GET 2023-07-29T00:09:38.219 DEBUG:boto:Path: /hiiqos-48/?prefix=obj2 2023-07-29T00:09:38.219 DEBUG:boto:Data: 2023-07-29T00:09:38.220 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.220 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.220 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.221 DEBUG:boto:Params: {} 2023-07-29T00:09:38.221 DEBUG:boto:Token: None 2023-07-29T00:09:38.221 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/ 2023-07-29T00:09:38.222 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA= 2023-07-29T00:09:38.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA=', 'Content-Length': '0'} 2023-07-29T00:09:38.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2efbf1728196a80-0064c458c2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.225 DEBUG:boto:b'hiiqos-48obj21000falseobj22023-07-29T00:08:34.078Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:38.226 DEBUG:boto:path=/hiiqos-48/obj2 2023-07-29T00:09:38.226 DEBUG:boto:auth_path=/hiiqos-48/obj2 2023-07-29T00:09:38.226 DEBUG:boto:Method: HEAD 2023-07-29T00:09:38.227 DEBUG:boto:Path: /hiiqos-48/obj2 2023-07-29T00:09:38.227 DEBUG:boto:Data: 2023-07-29T00:09:38.227 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.228 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.228 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.228 DEBUG:boto:Params: {} 2023-07-29T00:09:38.229 DEBUG:boto:Token: None 2023-07-29T00:09:38.229 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj2 2023-07-29T00:09:38.229 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Xguf9nSlam9ORdBJnMRYxH92Itc= 2023-07-29T00:09:38.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Xguf9nSlam9ORdBJnMRYxH92Itc=', 'Content-Length': '0'} 2023-07-29T00:09:38.231 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006767407939d84048-0064c458c2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.232 DEBUG:boto:path=/hiiqos-48/obj2 2023-07-29T00:09:38.232 DEBUG:boto:auth_path=/hiiqos-48/obj2 2023-07-29T00:09:38.233 DEBUG:boto:Method: HEAD 2023-07-29T00:09:38.233 DEBUG:boto:Path: /hiiqos-48/obj2 2023-07-29T00:09:38.233 DEBUG:boto:Data: 2023-07-29T00:09:38.234 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.234 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.234 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.235 DEBUG:boto:Params: {} 2023-07-29T00:09:38.235 DEBUG:boto:Token: None 2023-07-29T00:09:38.235 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj2 2023-07-29T00:09:38.236 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Xguf9nSlam9ORdBJnMRYxH92Itc= 2023-07-29T00:09:38.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Xguf9nSlam9ORdBJnMRYxH92Itc=', 'Content-Length': '0'} 2023-07-29T00:09:38.238 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c630bed63bc207a3-0064c458c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-07-29T00:09:38.239 DEBUG:boto:path=/hiiqos-48/obj2 2023-07-29T00:09:38.239 DEBUG:boto:auth_path=/hiiqos-48/obj2 2023-07-29T00:09:38.239 DEBUG:boto:Method: GET 2023-07-29T00:09:38.240 DEBUG:boto:Path: /hiiqos-48/obj2 2023-07-29T00:09:38.240 DEBUG:boto:Data: 2023-07-29T00:09:38.241 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.241 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.241 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.242 DEBUG:boto:Params: {} 2023-07-29T00:09:38.242 DEBUG:boto:Token: None 2023-07-29T00:09:38.242 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj2 2023-07-29T00:09:38.243 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rOY827laOVDaCSYfC3nX89hz61A= 2023-07-29T00:09:38.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rOY827laOVDaCSYfC3nX89hz61A=', 'Content-Length': '0'} 2023-07-29T00:09:38.245 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bf9897ab77bb6bca-0064c458c2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.245 DEBUG:boto:path=/hiiqos-48/obj2 2023-07-29T00:09:38.246 DEBUG:boto:auth_path=/hiiqos-48/obj2 2023-07-29T00:09:38.246 DEBUG:boto:Method: GET 2023-07-29T00:09:38.247 DEBUG:boto:Path: /hiiqos-48/obj2 2023-07-29T00:09:38.247 DEBUG:boto:Data: 2023-07-29T00:09:38.247 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.248 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.248 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.248 DEBUG:boto:Params: {} 2023-07-29T00:09:38.249 DEBUG:boto:Token: None 2023-07-29T00:09:38.249 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj2 2023-07-29T00:09:38.250 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rOY827laOVDaCSYfC3nX89hz61A= 2023-07-29T00:09:38.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rOY827laOVDaCSYfC3nX89hz61A=', 'Content-Length': '0'} 2023-07-29T00:09:38.252 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb294768d97cc03b-0064c458c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.252 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:38.253 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:38.253 DEBUG:boto:path=/hiiqos-48/?prefix=obj2 2023-07-29T00:09:38.253 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj2 2023-07-29T00:09:38.254 DEBUG:boto:Method: GET 2023-07-29T00:09:38.254 DEBUG:boto:Path: /hiiqos-48/?prefix=obj2 2023-07-29T00:09:38.254 DEBUG:boto:Data: 2023-07-29T00:09:38.255 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.255 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.255 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.256 DEBUG:boto:Params: {} 2023-07-29T00:09:38.256 DEBUG:boto:Token: None 2023-07-29T00:09:38.257 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/ 2023-07-29T00:09:38.257 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA= 2023-07-29T00:09:38.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA=', 'Content-Length': '0'} 2023-07-29T00:09:38.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb28efcfac8fe15b-0064c458c2-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.300 DEBUG:boto:b'hiiqos-48obj21000falseobj22023-07-29T00:08:34.078Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:38.301 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:38.301 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:38.302 DEBUG:boto:path=/hiiqos-48/?prefix=obj2 2023-07-29T00:09:38.302 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj2 2023-07-29T00:09:38.302 DEBUG:boto:Method: GET 2023-07-29T00:09:38.303 DEBUG:boto:Path: /hiiqos-48/?prefix=obj2 2023-07-29T00:09:38.303 DEBUG:boto:Data: 2023-07-29T00:09:38.303 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.304 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.304 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.304 DEBUG:boto:Params: {} 2023-07-29T00:09:38.305 DEBUG:boto:Token: None 2023-07-29T00:09:38.305 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/ 2023-07-29T00:09:38.306 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA= 2023-07-29T00:09:38.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA=', 'Content-Length': '0'} 2023-07-29T00:09:38.308 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075eabc4034312d57-0064c458c2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.309 DEBUG:boto:b'hiiqos-48obj21000falseobj22023-07-29T00:08:34.078Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:38.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-07-29T00:09:38.310 DEBUG:boto:path=/hiiqos-48/obj3 2023-07-29T00:09:38.310 DEBUG:boto:auth_path=/hiiqos-48/obj3 2023-07-29T00:09:38.310 DEBUG:boto:path=/hiiqos-48/obj3?versionId=null 2023-07-29T00:09:38.311 DEBUG:boto:auth_path=/hiiqos-48/obj3?versionId=null 2023-07-29T00:09:38.311 DEBUG:boto:Method: GET 2023-07-29T00:09:38.311 DEBUG:boto:Path: /hiiqos-48/obj3?versionId=null 2023-07-29T00:09:38.312 DEBUG:boto:Data: 2023-07-29T00:09:38.312 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.312 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.313 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.313 DEBUG:boto:Params: {} 2023-07-29T00:09:38.313 DEBUG:boto:Token: None 2023-07-29T00:09:38.314 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj3?versionId=null 2023-07-29T00:09:38.314 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vQKqiLpxhHRoLxPLmnDP8GxjJUY= 2023-07-29T00:09:38.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vQKqiLpxhHRoLxPLmnDP8GxjJUY=', 'Content-Length': '0'} 2023-07-29T00:09:38.316 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dee603d2a275891e-0064c458c2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.317 DEBUG:boto:path=/hiiqos-48/obj3 2023-07-29T00:09:38.317 DEBUG:boto:auth_path=/hiiqos-48/obj3 2023-07-29T00:09:38.318 DEBUG:boto:path=/hiiqos-48/obj3?versionId=null 2023-07-29T00:09:38.318 DEBUG:boto:auth_path=/hiiqos-48/obj3?versionId=null 2023-07-29T00:09:38.318 DEBUG:boto:Method: GET 2023-07-29T00:09:38.319 DEBUG:boto:Path: /hiiqos-48/obj3?versionId=null 2023-07-29T00:09:38.319 DEBUG:boto:Data: 2023-07-29T00:09:38.319 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.320 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.320 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.324 DEBUG:boto:Params: {} 2023-07-29T00:09:38.325 DEBUG:boto:Token: None 2023-07-29T00:09:38.325 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj3?versionId=null 2023-07-29T00:09:38.326 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vQKqiLpxhHRoLxPLmnDP8GxjJUY= 2023-07-29T00:09:38.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vQKqiLpxhHRoLxPLmnDP8GxjJUY=', 'Content-Length': '0'} 2023-07-29T00:09:38.328 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ccb185080266eec3-0064c458c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.368 DEBUG:boto:path=/hiiqos-48/obj3 2023-07-29T00:09:38.369 DEBUG:boto:auth_path=/hiiqos-48/obj3 2023-07-29T00:09:38.369 DEBUG:boto:path=/hiiqos-48/obj3?versionId=null 2023-07-29T00:09:38.369 DEBUG:boto:auth_path=/hiiqos-48/obj3?versionId=null 2023-07-29T00:09:38.370 DEBUG:boto:Method: HEAD 2023-07-29T00:09:38.370 DEBUG:boto:Path: /hiiqos-48/obj3?versionId=null 2023-07-29T00:09:38.370 DEBUG:boto:Data: 2023-07-29T00:09:38.371 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.371 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.371 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.372 DEBUG:boto:Params: {} 2023-07-29T00:09:38.372 DEBUG:boto:Token: None 2023-07-29T00:09:38.373 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj3?versionId=null 2023-07-29T00:09:38.373 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EaTKSFDalAkUlqRSFpb/7dHi/dI= 2023-07-29T00:09:38.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EaTKSFDalAkUlqRSFpb/7dHi/dI=', 'Content-Length': '0'} 2023-07-29T00:09:38.375 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000111bf886c2433f39-0064c458c2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.376 DEBUG:boto:path=/hiiqos-48/obj3 2023-07-29T00:09:38.376 DEBUG:boto:auth_path=/hiiqos-48/obj3 2023-07-29T00:09:38.377 DEBUG:boto:path=/hiiqos-48/obj3?versionId=null 2023-07-29T00:09:38.377 DEBUG:boto:auth_path=/hiiqos-48/obj3?versionId=null 2023-07-29T00:09:38.377 DEBUG:boto:Method: HEAD 2023-07-29T00:09:38.378 DEBUG:boto:Path: /hiiqos-48/obj3?versionId=null 2023-07-29T00:09:38.378 DEBUG:boto:Data: 2023-07-29T00:09:38.378 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.379 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.379 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.379 DEBUG:boto:Params: {} 2023-07-29T00:09:38.380 DEBUG:boto:Token: None 2023-07-29T00:09:38.380 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj3?versionId=null 2023-07-29T00:09:38.381 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EaTKSFDalAkUlqRSFpb/7dHi/dI= 2023-07-29T00:09:38.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EaTKSFDalAkUlqRSFpb/7dHi/dI=', 'Content-Length': '0'} 2023-07-29T00:09:38.383 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000021eafc8e0d2a5122-0064c458c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-07-29T00:09:38.384 DEBUG:boto:path=/hiiqos-48/obj3 2023-07-29T00:09:38.384 DEBUG:boto:auth_path=/hiiqos-48/obj3 2023-07-29T00:09:38.385 DEBUG:boto:path=/hiiqos-48/obj3?versionId=null 2023-07-29T00:09:38.385 DEBUG:boto:auth_path=/hiiqos-48/obj3?versionId=null 2023-07-29T00:09:38.385 DEBUG:boto:Method: GET 2023-07-29T00:09:38.386 DEBUG:boto:Path: /hiiqos-48/obj3?versionId=null 2023-07-29T00:09:38.386 DEBUG:boto:Data: 2023-07-29T00:09:38.386 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.387 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.387 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.388 DEBUG:boto:Params: {} 2023-07-29T00:09:38.388 DEBUG:boto:Token: None 2023-07-29T00:09:38.388 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj3?versionId=null 2023-07-29T00:09:38.389 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vQKqiLpxhHRoLxPLmnDP8GxjJUY= 2023-07-29T00:09:38.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vQKqiLpxhHRoLxPLmnDP8GxjJUY=', 'Content-Length': '0'} 2023-07-29T00:09:38.391 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a5973d664225a886-0064c458c2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.391 DEBUG:boto:path=/hiiqos-48/obj3 2023-07-29T00:09:38.392 DEBUG:boto:auth_path=/hiiqos-48/obj3 2023-07-29T00:09:38.392 DEBUG:boto:path=/hiiqos-48/obj3?versionId=null 2023-07-29T00:09:38.393 DEBUG:boto:auth_path=/hiiqos-48/obj3?versionId=null 2023-07-29T00:09:38.393 DEBUG:boto:Method: GET 2023-07-29T00:09:38.393 DEBUG:boto:Path: /hiiqos-48/obj3?versionId=null 2023-07-29T00:09:38.394 DEBUG:boto:Data: 2023-07-29T00:09:38.394 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.394 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.395 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.395 DEBUG:boto:Params: {} 2023-07-29T00:09:38.396 DEBUG:boto:Token: None 2023-07-29T00:09:38.396 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj3?versionId=null 2023-07-29T00:09:38.396 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vQKqiLpxhHRoLxPLmnDP8GxjJUY= 2023-07-29T00:09:38.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vQKqiLpxhHRoLxPLmnDP8GxjJUY=', 'Content-Length': '0'} 2023-07-29T00:09:38.398 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000042afff85d6a27816-0064c458c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.399 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:38.399 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:38.400 DEBUG:boto:path=/hiiqos-48/?prefix=obj3 2023-07-29T00:09:38.400 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj3 2023-07-29T00:09:38.401 DEBUG:boto:Method: GET 2023-07-29T00:09:38.401 DEBUG:boto:Path: /hiiqos-48/?prefix=obj3 2023-07-29T00:09:38.401 DEBUG:boto:Data: 2023-07-29T00:09:38.402 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.402 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.402 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.403 DEBUG:boto:Params: {} 2023-07-29T00:09:38.403 DEBUG:boto:Token: None 2023-07-29T00:09:38.404 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/ 2023-07-29T00:09:38.404 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA= 2023-07-29T00:09:38.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA=', 'Content-Length': '0'} 2023-07-29T00:09:38.406 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005784b7f5b6816f7a-0064c458c2-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.407 DEBUG:boto:b'hiiqos-48obj31000falseobj32023-07-29T00:08:34.094Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:38.408 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:38.408 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:38.408 DEBUG:boto:path=/hiiqos-48/?prefix=obj3 2023-07-29T00:09:38.409 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj3 2023-07-29T00:09:38.409 DEBUG:boto:Method: GET 2023-07-29T00:09:38.410 DEBUG:boto:Path: /hiiqos-48/?prefix=obj3 2023-07-29T00:09:38.410 DEBUG:boto:Data: 2023-07-29T00:09:38.410 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.411 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.411 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.411 DEBUG:boto:Params: {} 2023-07-29T00:09:38.412 DEBUG:boto:Token: None 2023-07-29T00:09:38.412 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/ 2023-07-29T00:09:38.413 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA= 2023-07-29T00:09:38.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA=', 'Content-Length': '0'} 2023-07-29T00:09:38.415 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb329d4a63225a6c-0064c458c2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.416 DEBUG:boto:b'hiiqos-48obj31000falseobj32023-07-29T00:08:34.094Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:38.416 DEBUG:boto:path=/hiiqos-48/obj3 2023-07-29T00:09:38.417 DEBUG:boto:auth_path=/hiiqos-48/obj3 2023-07-29T00:09:38.417 DEBUG:boto:Method: HEAD 2023-07-29T00:09:38.417 DEBUG:boto:Path: /hiiqos-48/obj3 2023-07-29T00:09:38.418 DEBUG:boto:Data: 2023-07-29T00:09:38.418 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.418 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.419 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.419 DEBUG:boto:Params: {} 2023-07-29T00:09:38.419 DEBUG:boto:Token: None 2023-07-29T00:09:38.420 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj3 2023-07-29T00:09:38.420 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FM/b5gRsubz4+NtoW/xMXe2Njx8= 2023-07-29T00:09:38.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FM/b5gRsubz4+NtoW/xMXe2Njx8=', 'Content-Length': '0'} 2023-07-29T00:09:38.422 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006980bc0adf71b0a4-0064c458c2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.423 DEBUG:boto:path=/hiiqos-48/obj3 2023-07-29T00:09:38.423 DEBUG:boto:auth_path=/hiiqos-48/obj3 2023-07-29T00:09:38.423 DEBUG:boto:Method: HEAD 2023-07-29T00:09:38.424 DEBUG:boto:Path: /hiiqos-48/obj3 2023-07-29T00:09:38.424 DEBUG:boto:Data: 2023-07-29T00:09:38.425 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.425 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.425 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.426 DEBUG:boto:Params: {} 2023-07-29T00:09:38.426 DEBUG:boto:Token: None 2023-07-29T00:09:38.426 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj3 2023-07-29T00:09:38.427 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FM/b5gRsubz4+NtoW/xMXe2Njx8= 2023-07-29T00:09:38.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FM/b5gRsubz4+NtoW/xMXe2Njx8=', 'Content-Length': '0'} 2023-07-29T00:09:38.429 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ed462a317de8f09-0064c458c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-07-29T00:09:38.430 DEBUG:boto:path=/hiiqos-48/obj3 2023-07-29T00:09:38.430 DEBUG:boto:auth_path=/hiiqos-48/obj3 2023-07-29T00:09:38.430 DEBUG:boto:Method: GET 2023-07-29T00:09:38.431 DEBUG:boto:Path: /hiiqos-48/obj3 2023-07-29T00:09:38.431 DEBUG:boto:Data: 2023-07-29T00:09:38.432 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.432 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.432 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.433 DEBUG:boto:Params: {} 2023-07-29T00:09:38.433 DEBUG:boto:Token: None 2023-07-29T00:09:38.433 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj3 2023-07-29T00:09:38.434 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AC5UvGaPyvj8BNaaXf1DpkcOVzI= 2023-07-29T00:09:38.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AC5UvGaPyvj8BNaaXf1DpkcOVzI=', 'Content-Length': '0'} 2023-07-29T00:09:38.436 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e750feed4fe9b09-0064c458c2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.436 DEBUG:boto:path=/hiiqos-48/obj3 2023-07-29T00:09:38.437 DEBUG:boto:auth_path=/hiiqos-48/obj3 2023-07-29T00:09:38.437 DEBUG:boto:Method: GET 2023-07-29T00:09:38.438 DEBUG:boto:Path: /hiiqos-48/obj3 2023-07-29T00:09:38.438 DEBUG:boto:Data: 2023-07-29T00:09:38.438 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.439 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.439 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.439 DEBUG:boto:Params: {} 2023-07-29T00:09:38.440 DEBUG:boto:Token: None 2023-07-29T00:09:38.440 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj3 2023-07-29T00:09:38.441 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AC5UvGaPyvj8BNaaXf1DpkcOVzI= 2023-07-29T00:09:38.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AC5UvGaPyvj8BNaaXf1DpkcOVzI=', 'Content-Length': '0'} 2023-07-29T00:09:38.443 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000601142829a07ac01-0064c458c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.443 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:38.444 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:38.444 DEBUG:boto:path=/hiiqos-48/?prefix=obj3 2023-07-29T00:09:38.444 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj3 2023-07-29T00:09:38.445 DEBUG:boto:Method: GET 2023-07-29T00:09:38.445 DEBUG:boto:Path: /hiiqos-48/?prefix=obj3 2023-07-29T00:09:38.446 DEBUG:boto:Data: 2023-07-29T00:09:38.446 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.446 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.447 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.447 DEBUG:boto:Params: {} 2023-07-29T00:09:38.447 DEBUG:boto:Token: None 2023-07-29T00:09:38.448 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/ 2023-07-29T00:09:38.448 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA= 2023-07-29T00:09:38.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA=', 'Content-Length': '0'} 2023-07-29T00:09:38.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007080c07ce7b956c6-0064c458c2-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.451 DEBUG:boto:b'hiiqos-48obj31000falseobj32023-07-29T00:08:34.094Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:38.452 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:38.452 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:38.453 DEBUG:boto:path=/hiiqos-48/?prefix=obj3 2023-07-29T00:09:38.453 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj3 2023-07-29T00:09:38.453 DEBUG:boto:Method: GET 2023-07-29T00:09:38.454 DEBUG:boto:Path: /hiiqos-48/?prefix=obj3 2023-07-29T00:09:38.454 DEBUG:boto:Data: 2023-07-29T00:09:38.454 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.455 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.455 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.455 DEBUG:boto:Params: {} 2023-07-29T00:09:38.456 DEBUG:boto:Token: None 2023-07-29T00:09:38.456 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/ 2023-07-29T00:09:38.457 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA= 2023-07-29T00:09:38.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA=', 'Content-Length': '0'} 2023-07-29T00:09:38.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d27995448e8a9bae-0064c458c2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.460 DEBUG:boto:b'hiiqos-48obj31000falseobj32023-07-29T00:08:34.094Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:38.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-07-29T00:09:38.461 DEBUG:boto:path=/hiiqos-48/obj4 2023-07-29T00:09:38.461 DEBUG:boto:auth_path=/hiiqos-48/obj4 2023-07-29T00:09:38.462 DEBUG:boto:path=/hiiqos-48/obj4?versionId=null 2023-07-29T00:09:38.462 DEBUG:boto:auth_path=/hiiqos-48/obj4?versionId=null 2023-07-29T00:09:38.462 DEBUG:boto:Method: GET 2023-07-29T00:09:38.463 DEBUG:boto:Path: /hiiqos-48/obj4?versionId=null 2023-07-29T00:09:38.463 DEBUG:boto:Data: 2023-07-29T00:09:38.464 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.464 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.464 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.465 DEBUG:boto:Params: {} 2023-07-29T00:09:38.465 DEBUG:boto:Token: None 2023-07-29T00:09:38.465 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj4?versionId=null 2023-07-29T00:09:38.466 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6gMVLw0io9G/GgQr61jw0G4AL7U= 2023-07-29T00:09:38.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6gMVLw0io9G/GgQr61jw0G4AL7U=', 'Content-Length': '0'} 2023-07-29T00:09:38.469 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000167e80a8bc680493-0064c458c2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.469 DEBUG:boto:path=/hiiqos-48/obj4 2023-07-29T00:09:38.470 DEBUG:boto:auth_path=/hiiqos-48/obj4 2023-07-29T00:09:38.470 DEBUG:boto:path=/hiiqos-48/obj4?versionId=null 2023-07-29T00:09:38.470 DEBUG:boto:auth_path=/hiiqos-48/obj4?versionId=null 2023-07-29T00:09:38.471 DEBUG:boto:Method: GET 2023-07-29T00:09:38.471 DEBUG:boto:Path: /hiiqos-48/obj4?versionId=null 2023-07-29T00:09:38.471 DEBUG:boto:Data: 2023-07-29T00:09:38.472 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.472 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.473 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.473 DEBUG:boto:Params: {} 2023-07-29T00:09:38.473 DEBUG:boto:Token: None 2023-07-29T00:09:38.474 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj4?versionId=null 2023-07-29T00:09:38.474 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6gMVLw0io9G/GgQr61jw0G4AL7U= 2023-07-29T00:09:38.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6gMVLw0io9G/GgQr61jw0G4AL7U=', 'Content-Length': '0'} 2023-07-29T00:09:38.476 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000213fb3e33577ce04-0064c458c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.477 DEBUG:boto:path=/hiiqos-48/obj4 2023-07-29T00:09:38.477 DEBUG:boto:auth_path=/hiiqos-48/obj4 2023-07-29T00:09:38.478 DEBUG:boto:path=/hiiqos-48/obj4?versionId=null 2023-07-29T00:09:38.478 DEBUG:boto:auth_path=/hiiqos-48/obj4?versionId=null 2023-07-29T00:09:38.478 DEBUG:boto:Method: HEAD 2023-07-29T00:09:38.479 DEBUG:boto:Path: /hiiqos-48/obj4?versionId=null 2023-07-29T00:09:38.479 DEBUG:boto:Data: 2023-07-29T00:09:38.479 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.480 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.480 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.480 DEBUG:boto:Params: {} 2023-07-29T00:09:38.481 DEBUG:boto:Token: None 2023-07-29T00:09:38.481 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj4?versionId=null 2023-07-29T00:09:38.482 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gynt5d/3KsdTf5LTIzWKvpxu9mE= 2023-07-29T00:09:38.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gynt5d/3KsdTf5LTIzWKvpxu9mE=', 'Content-Length': '0'} 2023-07-29T00:09:38.484 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000064b205a9345ccd61-0064c458c2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.484 DEBUG:boto:path=/hiiqos-48/obj4 2023-07-29T00:09:38.485 DEBUG:boto:auth_path=/hiiqos-48/obj4 2023-07-29T00:09:38.485 DEBUG:boto:path=/hiiqos-48/obj4?versionId=null 2023-07-29T00:09:38.485 DEBUG:boto:auth_path=/hiiqos-48/obj4?versionId=null 2023-07-29T00:09:38.486 DEBUG:boto:Method: HEAD 2023-07-29T00:09:38.486 DEBUG:boto:Path: /hiiqos-48/obj4?versionId=null 2023-07-29T00:09:38.486 DEBUG:boto:Data: 2023-07-29T00:09:38.487 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.487 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.487 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.488 DEBUG:boto:Params: {} 2023-07-29T00:09:38.488 DEBUG:boto:Token: None 2023-07-29T00:09:38.489 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj4?versionId=null 2023-07-29T00:09:38.489 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gynt5d/3KsdTf5LTIzWKvpxu9mE= 2023-07-29T00:09:38.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gynt5d/3KsdTf5LTIzWKvpxu9mE=', 'Content-Length': '0'} 2023-07-29T00:09:38.491 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000025a02147d151df0e-0064c458c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-07-29T00:09:38.492 DEBUG:boto:path=/hiiqos-48/obj4 2023-07-29T00:09:38.492 DEBUG:boto:auth_path=/hiiqos-48/obj4 2023-07-29T00:09:38.493 DEBUG:boto:path=/hiiqos-48/obj4?versionId=null 2023-07-29T00:09:38.493 DEBUG:boto:auth_path=/hiiqos-48/obj4?versionId=null 2023-07-29T00:09:38.493 DEBUG:boto:Method: GET 2023-07-29T00:09:38.494 DEBUG:boto:Path: /hiiqos-48/obj4?versionId=null 2023-07-29T00:09:38.494 DEBUG:boto:Data: 2023-07-29T00:09:38.494 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.495 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.495 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.496 DEBUG:boto:Params: {} 2023-07-29T00:09:38.496 DEBUG:boto:Token: None 2023-07-29T00:09:38.496 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj4?versionId=null 2023-07-29T00:09:38.497 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6gMVLw0io9G/GgQr61jw0G4AL7U= 2023-07-29T00:09:38.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6gMVLw0io9G/GgQr61jw0G4AL7U=', 'Content-Length': '0'} 2023-07-29T00:09:38.499 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000071abd08ac7632605-0064c458c2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.500 DEBUG:boto:path=/hiiqos-48/obj4 2023-07-29T00:09:38.500 DEBUG:boto:auth_path=/hiiqos-48/obj4 2023-07-29T00:09:38.500 DEBUG:boto:path=/hiiqos-48/obj4?versionId=null 2023-07-29T00:09:38.501 DEBUG:boto:auth_path=/hiiqos-48/obj4?versionId=null 2023-07-29T00:09:38.501 DEBUG:boto:Method: GET 2023-07-29T00:09:38.502 DEBUG:boto:Path: /hiiqos-48/obj4?versionId=null 2023-07-29T00:09:38.502 DEBUG:boto:Data: 2023-07-29T00:09:38.502 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.503 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.503 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.503 DEBUG:boto:Params: {} 2023-07-29T00:09:38.504 DEBUG:boto:Token: None 2023-07-29T00:09:38.504 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj4?versionId=null 2023-07-29T00:09:38.505 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6gMVLw0io9G/GgQr61jw0G4AL7U= 2023-07-29T00:09:38.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6gMVLw0io9G/GgQr61jw0G4AL7U=', 'Content-Length': '0'} 2023-07-29T00:09:38.507 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002fbfbc636ad41389-0064c458c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.548 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:38.549 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:38.549 DEBUG:boto:path=/hiiqos-48/?prefix=obj4 2023-07-29T00:09:38.550 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj4 2023-07-29T00:09:38.550 DEBUG:boto:Method: GET 2023-07-29T00:09:38.551 DEBUG:boto:Path: /hiiqos-48/?prefix=obj4 2023-07-29T00:09:38.551 DEBUG:boto:Data: 2023-07-29T00:09:38.551 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.552 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.552 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.552 DEBUG:boto:Params: {} 2023-07-29T00:09:38.553 DEBUG:boto:Token: None 2023-07-29T00:09:38.553 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/ 2023-07-29T00:09:38.554 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA= 2023-07-29T00:09:38.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA=', 'Content-Length': '0'} 2023-07-29T00:09:38.557 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d834b129f5e72266-0064c458c2-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.557 DEBUG:boto:b'hiiqos-48obj41000falseobj42023-07-29T00:08:34.109Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:38.558 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:38.558 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:38.559 DEBUG:boto:path=/hiiqos-48/?prefix=obj4 2023-07-29T00:09:38.559 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj4 2023-07-29T00:09:38.559 DEBUG:boto:Method: GET 2023-07-29T00:09:38.560 DEBUG:boto:Path: /hiiqos-48/?prefix=obj4 2023-07-29T00:09:38.560 DEBUG:boto:Data: 2023-07-29T00:09:38.560 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.561 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.561 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.562 DEBUG:boto:Params: {} 2023-07-29T00:09:38.562 DEBUG:boto:Token: None 2023-07-29T00:09:38.562 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/ 2023-07-29T00:09:38.563 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA= 2023-07-29T00:09:38.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA=', 'Content-Length': '0'} 2023-07-29T00:09:38.565 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cb3bdd5a61b7c14-0064c458c2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.566 DEBUG:boto:b'hiiqos-48obj41000falseobj42023-07-29T00:08:34.109Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:38.567 DEBUG:boto:path=/hiiqos-48/obj4 2023-07-29T00:09:38.567 DEBUG:boto:auth_path=/hiiqos-48/obj4 2023-07-29T00:09:38.567 DEBUG:boto:Method: HEAD 2023-07-29T00:09:38.568 DEBUG:boto:Path: /hiiqos-48/obj4 2023-07-29T00:09:38.568 DEBUG:boto:Data: 2023-07-29T00:09:38.568 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.569 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.569 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.569 DEBUG:boto:Params: {} 2023-07-29T00:09:38.570 DEBUG:boto:Token: None 2023-07-29T00:09:38.570 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj4 2023-07-29T00:09:38.571 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FTHGMUytUF+W1l8caM8YaIOWfdc= 2023-07-29T00:09:38.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FTHGMUytUF+W1l8caM8YaIOWfdc=', 'Content-Length': '0'} 2023-07-29T00:09:38.573 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000026f97cb2eeee5004-0064c458c2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.573 DEBUG:boto:path=/hiiqos-48/obj4 2023-07-29T00:09:38.574 DEBUG:boto:auth_path=/hiiqos-48/obj4 2023-07-29T00:09:38.574 DEBUG:boto:Method: HEAD 2023-07-29T00:09:38.575 DEBUG:boto:Path: /hiiqos-48/obj4 2023-07-29T00:09:38.575 DEBUG:boto:Data: 2023-07-29T00:09:38.575 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.576 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.576 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.576 DEBUG:boto:Params: {} 2023-07-29T00:09:38.577 DEBUG:boto:Token: None 2023-07-29T00:09:38.577 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj4 2023-07-29T00:09:38.578 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FTHGMUytUF+W1l8caM8YaIOWfdc= 2023-07-29T00:09:38.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FTHGMUytUF+W1l8caM8YaIOWfdc=', 'Content-Length': '0'} 2023-07-29T00:09:38.579 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b2a14f21ae21c899-0064c458c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-07-29T00:09:38.580 DEBUG:boto:path=/hiiqos-48/obj4 2023-07-29T00:09:38.581 DEBUG:boto:auth_path=/hiiqos-48/obj4 2023-07-29T00:09:38.581 DEBUG:boto:Method: GET 2023-07-29T00:09:38.582 DEBUG:boto:Path: /hiiqos-48/obj4 2023-07-29T00:09:38.582 DEBUG:boto:Data: 2023-07-29T00:09:38.582 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.583 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.583 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.583 DEBUG:boto:Params: {} 2023-07-29T00:09:38.584 DEBUG:boto:Token: None 2023-07-29T00:09:38.584 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj4 2023-07-29T00:09:38.585 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KMqcMJUrUYcEjf0tW2SJrT4rj1Q= 2023-07-29T00:09:38.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KMqcMJUrUYcEjf0tW2SJrT4rj1Q=', 'Content-Length': '0'} 2023-07-29T00:09:38.587 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d0cdd8beb16d893b-0064c458c2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.628 DEBUG:boto:path=/hiiqos-48/obj4 2023-07-29T00:09:38.629 DEBUG:boto:auth_path=/hiiqos-48/obj4 2023-07-29T00:09:38.629 DEBUG:boto:Method: GET 2023-07-29T00:09:38.630 DEBUG:boto:Path: /hiiqos-48/obj4 2023-07-29T00:09:38.630 DEBUG:boto:Data: 2023-07-29T00:09:38.630 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.631 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.631 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.631 DEBUG:boto:Params: {} 2023-07-29T00:09:38.632 DEBUG:boto:Token: None 2023-07-29T00:09:38.632 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj4 2023-07-29T00:09:38.633 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KMqcMJUrUYcEjf0tW2SJrT4rj1Q= 2023-07-29T00:09:38.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KMqcMJUrUYcEjf0tW2SJrT4rj1Q=', 'Content-Length': '0'} 2023-07-29T00:09:38.635 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b21a2014da058e2-0064c458c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.636 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:38.636 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:38.637 DEBUG:boto:path=/hiiqos-48/?prefix=obj4 2023-07-29T00:09:38.637 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj4 2023-07-29T00:09:38.637 DEBUG:boto:Method: GET 2023-07-29T00:09:38.638 DEBUG:boto:Path: /hiiqos-48/?prefix=obj4 2023-07-29T00:09:38.638 DEBUG:boto:Data: 2023-07-29T00:09:38.638 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.639 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.639 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.639 DEBUG:boto:Params: {} 2023-07-29T00:09:38.640 DEBUG:boto:Token: None 2023-07-29T00:09:38.640 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/ 2023-07-29T00:09:38.641 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA= 2023-07-29T00:09:38.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA=', 'Content-Length': '0'} 2023-07-29T00:09:38.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be6636dc57a181ba-0064c458c2-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.644 DEBUG:boto:b'hiiqos-48obj41000falseobj42023-07-29T00:08:34.109Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:38.644 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:38.645 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:38.645 DEBUG:boto:path=/hiiqos-48/?prefix=obj4 2023-07-29T00:09:38.645 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj4 2023-07-29T00:09:38.646 DEBUG:boto:Method: GET 2023-07-29T00:09:38.646 DEBUG:boto:Path: /hiiqos-48/?prefix=obj4 2023-07-29T00:09:38.646 DEBUG:boto:Data: 2023-07-29T00:09:38.647 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.647 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.647 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.648 DEBUG:boto:Params: {} 2023-07-29T00:09:38.648 DEBUG:boto:Token: None 2023-07-29T00:09:38.649 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/ 2023-07-29T00:09:38.649 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA= 2023-07-29T00:09:38.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA=', 'Content-Length': '0'} 2023-07-29T00:09:38.652 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb1f6ac44aba897e-0064c458c2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.652 DEBUG:boto:b'hiiqos-48obj41000falseobj42023-07-29T00:08:34.109Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:38.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-07-29T00:09:38.653 DEBUG:boto:path=/hiiqos-48/obj5 2023-07-29T00:09:38.654 DEBUG:boto:auth_path=/hiiqos-48/obj5 2023-07-29T00:09:38.654 DEBUG:boto:path=/hiiqos-48/obj5?versionId=null 2023-07-29T00:09:38.654 DEBUG:boto:auth_path=/hiiqos-48/obj5?versionId=null 2023-07-29T00:09:38.655 DEBUG:boto:Method: GET 2023-07-29T00:09:38.655 DEBUG:boto:Path: /hiiqos-48/obj5?versionId=null 2023-07-29T00:09:38.655 DEBUG:boto:Data: 2023-07-29T00:09:38.656 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.656 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.656 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.657 DEBUG:boto:Params: {} 2023-07-29T00:09:38.657 DEBUG:boto:Token: None 2023-07-29T00:09:38.658 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj5?versionId=null 2023-07-29T00:09:38.658 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AMydDLtLzGT6abKWq7gQXVfG884= 2023-07-29T00:09:38.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AMydDLtLzGT6abKWq7gQXVfG884=', 'Content-Length': '0'} 2023-07-29T00:09:38.660 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000036bcf84005a93df1-0064c458c2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.661 DEBUG:boto:path=/hiiqos-48/obj5 2023-07-29T00:09:38.661 DEBUG:boto:auth_path=/hiiqos-48/obj5 2023-07-29T00:09:38.661 DEBUG:boto:path=/hiiqos-48/obj5?versionId=null 2023-07-29T00:09:38.662 DEBUG:boto:auth_path=/hiiqos-48/obj5?versionId=null 2023-07-29T00:09:38.662 DEBUG:boto:Method: GET 2023-07-29T00:09:38.662 DEBUG:boto:Path: /hiiqos-48/obj5?versionId=null 2023-07-29T00:09:38.663 DEBUG:boto:Data: 2023-07-29T00:09:38.663 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.663 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.664 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.664 DEBUG:boto:Params: {} 2023-07-29T00:09:38.664 DEBUG:boto:Token: None 2023-07-29T00:09:38.665 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj5?versionId=null 2023-07-29T00:09:38.665 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AMydDLtLzGT6abKWq7gQXVfG884= 2023-07-29T00:09:38.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AMydDLtLzGT6abKWq7gQXVfG884=', 'Content-Length': '0'} 2023-07-29T00:09:38.667 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b62e12a4bcde9ac1-0064c458c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.668 DEBUG:boto:path=/hiiqos-48/obj5 2023-07-29T00:09:38.668 DEBUG:boto:auth_path=/hiiqos-48/obj5 2023-07-29T00:09:38.669 DEBUG:boto:path=/hiiqos-48/obj5?versionId=null 2023-07-29T00:09:38.669 DEBUG:boto:auth_path=/hiiqos-48/obj5?versionId=null 2023-07-29T00:09:38.669 DEBUG:boto:Method: HEAD 2023-07-29T00:09:38.670 DEBUG:boto:Path: /hiiqos-48/obj5?versionId=null 2023-07-29T00:09:38.670 DEBUG:boto:Data: 2023-07-29T00:09:38.670 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.671 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.671 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.671 DEBUG:boto:Params: {} 2023-07-29T00:09:38.672 DEBUG:boto:Token: None 2023-07-29T00:09:38.672 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj5?versionId=null 2023-07-29T00:09:38.673 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uPnyk8YcgS/0n/fp2LfsL+9aiNo= 2023-07-29T00:09:38.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uPnyk8YcgS/0n/fp2LfsL+9aiNo=', 'Content-Length': '0'} 2023-07-29T00:09:38.674 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d3e288442b39997b-0064c458c2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.675 DEBUG:boto:path=/hiiqos-48/obj5 2023-07-29T00:09:38.675 DEBUG:boto:auth_path=/hiiqos-48/obj5 2023-07-29T00:09:38.676 DEBUG:boto:path=/hiiqos-48/obj5?versionId=null 2023-07-29T00:09:38.676 DEBUG:boto:auth_path=/hiiqos-48/obj5?versionId=null 2023-07-29T00:09:38.676 DEBUG:boto:Method: HEAD 2023-07-29T00:09:38.677 DEBUG:boto:Path: /hiiqos-48/obj5?versionId=null 2023-07-29T00:09:38.677 DEBUG:boto:Data: 2023-07-29T00:09:38.677 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.678 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.678 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.678 DEBUG:boto:Params: {} 2023-07-29T00:09:38.679 DEBUG:boto:Token: None 2023-07-29T00:09:38.679 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj5?versionId=null 2023-07-29T00:09:38.680 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uPnyk8YcgS/0n/fp2LfsL+9aiNo= 2023-07-29T00:09:38.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uPnyk8YcgS/0n/fp2LfsL+9aiNo=', 'Content-Length': '0'} 2023-07-29T00:09:38.682 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004003264f9d8509bc-0064c458c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-07-29T00:09:38.683 DEBUG:boto:path=/hiiqos-48/obj5 2023-07-29T00:09:38.683 DEBUG:boto:auth_path=/hiiqos-48/obj5 2023-07-29T00:09:38.683 DEBUG:boto:path=/hiiqos-48/obj5?versionId=null 2023-07-29T00:09:38.684 DEBUG:boto:auth_path=/hiiqos-48/obj5?versionId=null 2023-07-29T00:09:38.684 DEBUG:boto:Method: GET 2023-07-29T00:09:38.684 DEBUG:boto:Path: /hiiqos-48/obj5?versionId=null 2023-07-29T00:09:38.685 DEBUG:boto:Data: 2023-07-29T00:09:38.685 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.685 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.686 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.686 DEBUG:boto:Params: {} 2023-07-29T00:09:38.687 DEBUG:boto:Token: None 2023-07-29T00:09:38.687 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj5?versionId=null 2023-07-29T00:09:38.687 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AMydDLtLzGT6abKWq7gQXVfG884= 2023-07-29T00:09:38.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AMydDLtLzGT6abKWq7gQXVfG884=', 'Content-Length': '0'} 2023-07-29T00:09:38.689 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f17536a2c6f66e4-0064c458c2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.690 DEBUG:boto:path=/hiiqos-48/obj5 2023-07-29T00:09:38.690 DEBUG:boto:auth_path=/hiiqos-48/obj5 2023-07-29T00:09:38.691 DEBUG:boto:path=/hiiqos-48/obj5?versionId=null 2023-07-29T00:09:38.691 DEBUG:boto:auth_path=/hiiqos-48/obj5?versionId=null 2023-07-29T00:09:38.691 DEBUG:boto:Method: GET 2023-07-29T00:09:38.692 DEBUG:boto:Path: /hiiqos-48/obj5?versionId=null 2023-07-29T00:09:38.692 DEBUG:boto:Data: 2023-07-29T00:09:38.693 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.693 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.693 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.694 DEBUG:boto:Params: {} 2023-07-29T00:09:38.694 DEBUG:boto:Token: None 2023-07-29T00:09:38.694 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj5?versionId=null 2023-07-29T00:09:38.695 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AMydDLtLzGT6abKWq7gQXVfG884= 2023-07-29T00:09:38.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AMydDLtLzGT6abKWq7gQXVfG884=', 'Content-Length': '0'} 2023-07-29T00:09:38.697 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c94bf71c0f6b9dc-0064c458c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.697 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:38.698 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:38.698 DEBUG:boto:path=/hiiqos-48/?prefix=obj5 2023-07-29T00:09:38.699 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj5 2023-07-29T00:09:38.699 DEBUG:boto:Method: GET 2023-07-29T00:09:38.699 DEBUG:boto:Path: /hiiqos-48/?prefix=obj5 2023-07-29T00:09:38.700 DEBUG:boto:Data: 2023-07-29T00:09:38.700 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.700 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.701 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.701 DEBUG:boto:Params: {} 2023-07-29T00:09:38.702 DEBUG:boto:Token: None 2023-07-29T00:09:38.702 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/ 2023-07-29T00:09:38.702 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA= 2023-07-29T00:09:38.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA=', 'Content-Length': '0'} 2023-07-29T00:09:38.705 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f42aeff68d384d57-0064c458c2-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.705 DEBUG:boto:b'hiiqos-48obj51000falseobj52023-07-29T00:09:22.155Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:38.706 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:38.706 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:38.707 DEBUG:boto:path=/hiiqos-48/?prefix=obj5 2023-07-29T00:09:38.707 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj5 2023-07-29T00:09:38.707 DEBUG:boto:Method: GET 2023-07-29T00:09:38.708 DEBUG:boto:Path: /hiiqos-48/?prefix=obj5 2023-07-29T00:09:38.708 DEBUG:boto:Data: 2023-07-29T00:09:38.709 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.709 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.709 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.710 DEBUG:boto:Params: {} 2023-07-29T00:09:38.710 DEBUG:boto:Token: None 2023-07-29T00:09:38.710 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/ 2023-07-29T00:09:38.711 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA= 2023-07-29T00:09:38.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA=', 'Content-Length': '0'} 2023-07-29T00:09:38.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db40f2339cba106b-0064c458c2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.756 DEBUG:boto:b'hiiqos-48obj51000falseobj52023-07-29T00:09:22.155Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:38.757 DEBUG:boto:path=/hiiqos-48/obj5 2023-07-29T00:09:38.757 DEBUG:boto:auth_path=/hiiqos-48/obj5 2023-07-29T00:09:38.758 DEBUG:boto:Method: HEAD 2023-07-29T00:09:38.758 DEBUG:boto:Path: /hiiqos-48/obj5 2023-07-29T00:09:38.759 DEBUG:boto:Data: 2023-07-29T00:09:38.759 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.759 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.760 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.760 DEBUG:boto:Params: {} 2023-07-29T00:09:38.760 DEBUG:boto:Token: None 2023-07-29T00:09:38.761 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj5 2023-07-29T00:09:38.761 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3JahIzlb5JpFomIklLIua/reFEo= 2023-07-29T00:09:38.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3JahIzlb5JpFomIklLIua/reFEo=', 'Content-Length': '0'} 2023-07-29T00:09:38.763 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ed7805558f7010ee-0064c458c2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.764 DEBUG:boto:path=/hiiqos-48/obj5 2023-07-29T00:09:38.764 DEBUG:boto:auth_path=/hiiqos-48/obj5 2023-07-29T00:09:38.764 DEBUG:boto:Method: HEAD 2023-07-29T00:09:38.765 DEBUG:boto:Path: /hiiqos-48/obj5 2023-07-29T00:09:38.765 DEBUG:boto:Data: 2023-07-29T00:09:38.765 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.766 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.766 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.766 DEBUG:boto:Params: {} 2023-07-29T00:09:38.767 DEBUG:boto:Token: None 2023-07-29T00:09:38.767 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj5 2023-07-29T00:09:38.768 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3JahIzlb5JpFomIklLIua/reFEo= 2023-07-29T00:09:38.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3JahIzlb5JpFomIklLIua/reFEo=', 'Content-Length': '0'} 2023-07-29T00:09:38.770 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b5cd0c1afa7a1879-0064c458c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-07-29T00:09:38.771 DEBUG:boto:path=/hiiqos-48/obj5 2023-07-29T00:09:38.771 DEBUG:boto:auth_path=/hiiqos-48/obj5 2023-07-29T00:09:38.771 DEBUG:boto:Method: GET 2023-07-29T00:09:38.772 DEBUG:boto:Path: /hiiqos-48/obj5 2023-07-29T00:09:38.772 DEBUG:boto:Data: 2023-07-29T00:09:38.773 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.773 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.773 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.774 DEBUG:boto:Params: {} 2023-07-29T00:09:38.774 DEBUG:boto:Token: None 2023-07-29T00:09:38.774 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj5 2023-07-29T00:09:38.775 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GdpElx9iAR0jE2mTwA8OWmNKeTY= 2023-07-29T00:09:38.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GdpElx9iAR0jE2mTwA8OWmNKeTY=', 'Content-Length': '0'} 2023-07-29T00:09:38.777 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bdbaf0878121ad78-0064c458c2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.777 DEBUG:boto:path=/hiiqos-48/obj5 2023-07-29T00:09:38.778 DEBUG:boto:auth_path=/hiiqos-48/obj5 2023-07-29T00:09:38.778 DEBUG:boto:Method: GET 2023-07-29T00:09:38.778 DEBUG:boto:Path: /hiiqos-48/obj5 2023-07-29T00:09:38.779 DEBUG:boto:Data: 2023-07-29T00:09:38.779 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.779 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.780 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.780 DEBUG:boto:Params: {} 2023-07-29T00:09:38.781 DEBUG:boto:Token: None 2023-07-29T00:09:38.781 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj5 2023-07-29T00:09:38.781 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GdpElx9iAR0jE2mTwA8OWmNKeTY= 2023-07-29T00:09:38.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GdpElx9iAR0jE2mTwA8OWmNKeTY=', 'Content-Length': '0'} 2023-07-29T00:09:38.783 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f94686aaa5504042-0064c458c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.784 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:38.784 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:38.785 DEBUG:boto:path=/hiiqos-48/?prefix=obj5 2023-07-29T00:09:38.785 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj5 2023-07-29T00:09:38.785 DEBUG:boto:Method: GET 2023-07-29T00:09:38.786 DEBUG:boto:Path: /hiiqos-48/?prefix=obj5 2023-07-29T00:09:38.786 DEBUG:boto:Data: 2023-07-29T00:09:38.786 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.787 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.787 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.787 DEBUG:boto:Params: {} 2023-07-29T00:09:38.788 DEBUG:boto:Token: None 2023-07-29T00:09:38.788 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/ 2023-07-29T00:09:38.789 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA= 2023-07-29T00:09:38.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA=', 'Content-Length': '0'} 2023-07-29T00:09:38.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f6dc9efc4129143-0064c458c2-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.792 DEBUG:boto:b'hiiqos-48obj51000falseobj52023-07-29T00:09:22.155Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:38.792 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:38.793 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:38.793 DEBUG:boto:path=/hiiqos-48/?prefix=obj5 2023-07-29T00:09:38.793 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj5 2023-07-29T00:09:38.794 DEBUG:boto:Method: GET 2023-07-29T00:09:38.794 DEBUG:boto:Path: /hiiqos-48/?prefix=obj5 2023-07-29T00:09:38.794 DEBUG:boto:Data: 2023-07-29T00:09:38.795 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.795 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.796 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.796 DEBUG:boto:Params: {} 2023-07-29T00:09:38.796 DEBUG:boto:Token: None 2023-07-29T00:09:38.797 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/ 2023-07-29T00:09:38.797 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA= 2023-07-29T00:09:38.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA=', 'Content-Length': '0'} 2023-07-29T00:09:38.800 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006752b851323c6ca3-0064c458c2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.800 DEBUG:boto:b'hiiqos-48obj51000falseobj52023-07-29T00:09:22.155Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:38.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-07-29T00:09:38.801 DEBUG:boto:path=/hiiqos-48/obj6 2023-07-29T00:09:38.802 DEBUG:boto:auth_path=/hiiqos-48/obj6 2023-07-29T00:09:38.802 DEBUG:boto:path=/hiiqos-48/obj6?versionId=null 2023-07-29T00:09:38.802 DEBUG:boto:auth_path=/hiiqos-48/obj6?versionId=null 2023-07-29T00:09:38.803 DEBUG:boto:Method: GET 2023-07-29T00:09:38.803 DEBUG:boto:Path: /hiiqos-48/obj6?versionId=null 2023-07-29T00:09:38.803 DEBUG:boto:Data: 2023-07-29T00:09:38.804 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.804 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.804 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.805 DEBUG:boto:Params: {} 2023-07-29T00:09:38.805 DEBUG:boto:Token: None 2023-07-29T00:09:38.806 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj6?versionId=null 2023-07-29T00:09:38.806 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zv7i3Wcv8CDBirY5C3CUzOMxQSM= 2023-07-29T00:09:38.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zv7i3Wcv8CDBirY5C3CUzOMxQSM=', 'Content-Length': '0'} 2023-07-29T00:09:38.808 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000044f724e6e753eb76-0064c458c2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.809 DEBUG:boto:path=/hiiqos-48/obj6 2023-07-29T00:09:38.809 DEBUG:boto:auth_path=/hiiqos-48/obj6 2023-07-29T00:09:38.809 DEBUG:boto:path=/hiiqos-48/obj6?versionId=null 2023-07-29T00:09:38.810 DEBUG:boto:auth_path=/hiiqos-48/obj6?versionId=null 2023-07-29T00:09:38.810 DEBUG:boto:Method: GET 2023-07-29T00:09:38.810 DEBUG:boto:Path: /hiiqos-48/obj6?versionId=null 2023-07-29T00:09:38.811 DEBUG:boto:Data: 2023-07-29T00:09:38.811 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.811 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.812 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.812 DEBUG:boto:Params: {} 2023-07-29T00:09:38.813 DEBUG:boto:Token: None 2023-07-29T00:09:38.813 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj6?versionId=null 2023-07-29T00:09:38.813 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zv7i3Wcv8CDBirY5C3CUzOMxQSM= 2023-07-29T00:09:38.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zv7i3Wcv8CDBirY5C3CUzOMxQSM=', 'Content-Length': '0'} 2023-07-29T00:09:38.815 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000341c91822eb84e53-0064c458c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.816 DEBUG:boto:path=/hiiqos-48/obj6 2023-07-29T00:09:38.816 DEBUG:boto:auth_path=/hiiqos-48/obj6 2023-07-29T00:09:38.817 DEBUG:boto:path=/hiiqos-48/obj6?versionId=null 2023-07-29T00:09:38.817 DEBUG:boto:auth_path=/hiiqos-48/obj6?versionId=null 2023-07-29T00:09:38.817 DEBUG:boto:Method: HEAD 2023-07-29T00:09:38.818 DEBUG:boto:Path: /hiiqos-48/obj6?versionId=null 2023-07-29T00:09:38.818 DEBUG:boto:Data: 2023-07-29T00:09:38.818 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.819 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.819 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.819 DEBUG:boto:Params: {} 2023-07-29T00:09:38.820 DEBUG:boto:Token: None 2023-07-29T00:09:38.820 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj6?versionId=null 2023-07-29T00:09:38.820 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3iuvusJ6+Ht7kkSGHzGuWZcnRSA= 2023-07-29T00:09:38.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3iuvusJ6+Ht7kkSGHzGuWZcnRSA=', 'Content-Length': '0'} 2023-07-29T00:09:38.822 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aebdebe932a12cc7-0064c458c2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.823 DEBUG:boto:path=/hiiqos-48/obj6 2023-07-29T00:09:38.823 DEBUG:boto:auth_path=/hiiqos-48/obj6 2023-07-29T00:09:38.824 DEBUG:boto:path=/hiiqos-48/obj6?versionId=null 2023-07-29T00:09:38.824 DEBUG:boto:auth_path=/hiiqos-48/obj6?versionId=null 2023-07-29T00:09:38.824 DEBUG:boto:Method: HEAD 2023-07-29T00:09:38.825 DEBUG:boto:Path: /hiiqos-48/obj6?versionId=null 2023-07-29T00:09:38.825 DEBUG:boto:Data: 2023-07-29T00:09:38.825 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.826 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.826 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.826 DEBUG:boto:Params: {} 2023-07-29T00:09:38.827 DEBUG:boto:Token: None 2023-07-29T00:09:38.827 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj6?versionId=null 2023-07-29T00:09:38.827 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3iuvusJ6+Ht7kkSGHzGuWZcnRSA= 2023-07-29T00:09:38.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3iuvusJ6+Ht7kkSGHzGuWZcnRSA=', 'Content-Length': '0'} 2023-07-29T00:09:38.829 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c5ff45595c49f24e-0064c458c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-07-29T00:09:38.830 DEBUG:boto:path=/hiiqos-48/obj6 2023-07-29T00:09:38.831 DEBUG:boto:auth_path=/hiiqos-48/obj6 2023-07-29T00:09:38.831 DEBUG:boto:path=/hiiqos-48/obj6?versionId=null 2023-07-29T00:09:38.831 DEBUG:boto:auth_path=/hiiqos-48/obj6?versionId=null 2023-07-29T00:09:38.832 DEBUG:boto:Method: GET 2023-07-29T00:09:38.832 DEBUG:boto:Path: /hiiqos-48/obj6?versionId=null 2023-07-29T00:09:38.832 DEBUG:boto:Data: 2023-07-29T00:09:38.833 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.833 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.833 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.834 DEBUG:boto:Params: {} 2023-07-29T00:09:38.834 DEBUG:boto:Token: None 2023-07-29T00:09:38.835 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj6?versionId=null 2023-07-29T00:09:38.835 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zv7i3Wcv8CDBirY5C3CUzOMxQSM= 2023-07-29T00:09:38.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zv7i3Wcv8CDBirY5C3CUzOMxQSM=', 'Content-Length': '0'} 2023-07-29T00:09:38.837 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e9e88ed79e57e35-0064c458c2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.838 DEBUG:boto:path=/hiiqos-48/obj6 2023-07-29T00:09:38.838 DEBUG:boto:auth_path=/hiiqos-48/obj6 2023-07-29T00:09:38.838 DEBUG:boto:path=/hiiqos-48/obj6?versionId=null 2023-07-29T00:09:38.839 DEBUG:boto:auth_path=/hiiqos-48/obj6?versionId=null 2023-07-29T00:09:38.839 DEBUG:boto:Method: GET 2023-07-29T00:09:38.839 DEBUG:boto:Path: /hiiqos-48/obj6?versionId=null 2023-07-29T00:09:38.840 DEBUG:boto:Data: 2023-07-29T00:09:38.840 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.840 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.841 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.841 DEBUG:boto:Params: {} 2023-07-29T00:09:38.841 DEBUG:boto:Token: None 2023-07-29T00:09:38.842 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj6?versionId=null 2023-07-29T00:09:38.842 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zv7i3Wcv8CDBirY5C3CUzOMxQSM= 2023-07-29T00:09:38.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zv7i3Wcv8CDBirY5C3CUzOMxQSM=', 'Content-Length': '0'} 2023-07-29T00:09:38.844 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000047eefbdc15f7b1f9-0064c458c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.845 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:38.845 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:38.846 DEBUG:boto:path=/hiiqos-48/?prefix=obj6 2023-07-29T00:09:38.846 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj6 2023-07-29T00:09:38.846 DEBUG:boto:Method: GET 2023-07-29T00:09:38.847 DEBUG:boto:Path: /hiiqos-48/?prefix=obj6 2023-07-29T00:09:38.847 DEBUG:boto:Data: 2023-07-29T00:09:38.847 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.848 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.848 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.848 DEBUG:boto:Params: {} 2023-07-29T00:09:38.849 DEBUG:boto:Token: None 2023-07-29T00:09:38.849 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/ 2023-07-29T00:09:38.850 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA= 2023-07-29T00:09:38.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA=', 'Content-Length': '0'} 2023-07-29T00:09:38.852 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2f547a41e9255bf-0064c458c2-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.852 DEBUG:boto:b'hiiqos-48obj61000falseobj62023-07-29T00:09:22.178Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:38.853 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:38.853 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:38.854 DEBUG:boto:path=/hiiqos-48/?prefix=obj6 2023-07-29T00:09:38.854 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj6 2023-07-29T00:09:38.854 DEBUG:boto:Method: GET 2023-07-29T00:09:38.855 DEBUG:boto:Path: /hiiqos-48/?prefix=obj6 2023-07-29T00:09:38.855 DEBUG:boto:Data: 2023-07-29T00:09:38.855 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.856 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.856 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.857 DEBUG:boto:Params: {} 2023-07-29T00:09:38.857 DEBUG:boto:Token: None 2023-07-29T00:09:38.857 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/ 2023-07-29T00:09:38.858 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA= 2023-07-29T00:09:38.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA=', 'Content-Length': '0'} 2023-07-29T00:09:38.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002545e1c588bbfd1e-0064c458c2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.860 DEBUG:boto:b'hiiqos-48obj61000falseobj62023-07-29T00:09:22.178Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:38.861 DEBUG:boto:path=/hiiqos-48/obj6 2023-07-29T00:09:38.861 DEBUG:boto:auth_path=/hiiqos-48/obj6 2023-07-29T00:09:38.862 DEBUG:boto:Method: HEAD 2023-07-29T00:09:38.862 DEBUG:boto:Path: /hiiqos-48/obj6 2023-07-29T00:09:38.862 DEBUG:boto:Data: 2023-07-29T00:09:38.863 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.863 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.863 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.864 DEBUG:boto:Params: {} 2023-07-29T00:09:38.864 DEBUG:boto:Token: None 2023-07-29T00:09:38.865 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj6 2023-07-29T00:09:38.865 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EqyucJ7G16IfZGegGSbisFuwEII= 2023-07-29T00:09:38.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EqyucJ7G16IfZGegGSbisFuwEII=', 'Content-Length': '0'} 2023-07-29T00:09:38.867 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005b0aa5ae43dd93ec-0064c458c2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.867 DEBUG:boto:path=/hiiqos-48/obj6 2023-07-29T00:09:38.868 DEBUG:boto:auth_path=/hiiqos-48/obj6 2023-07-29T00:09:38.868 DEBUG:boto:Method: HEAD 2023-07-29T00:09:38.868 DEBUG:boto:Path: /hiiqos-48/obj6 2023-07-29T00:09:38.869 DEBUG:boto:Data: 2023-07-29T00:09:38.869 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.869 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.870 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.870 DEBUG:boto:Params: {} 2023-07-29T00:09:38.870 DEBUG:boto:Token: None 2023-07-29T00:09:38.871 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj6 2023-07-29T00:09:38.871 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EqyucJ7G16IfZGegGSbisFuwEII= 2023-07-29T00:09:38.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EqyucJ7G16IfZGegGSbisFuwEII=', 'Content-Length': '0'} 2023-07-29T00:09:38.873 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b0db2284c204cc0b-0064c458c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-07-29T00:09:38.874 DEBUG:boto:path=/hiiqos-48/obj6 2023-07-29T00:09:38.874 DEBUG:boto:auth_path=/hiiqos-48/obj6 2023-07-29T00:09:38.874 DEBUG:boto:Method: GET 2023-07-29T00:09:38.875 DEBUG:boto:Path: /hiiqos-48/obj6 2023-07-29T00:09:38.875 DEBUG:boto:Data: 2023-07-29T00:09:38.876 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.876 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.876 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.877 DEBUG:boto:Params: {} 2023-07-29T00:09:38.877 DEBUG:boto:Token: None 2023-07-29T00:09:38.877 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj6 2023-07-29T00:09:38.878 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Iae3tQPOqpzcPTo3mCFeKw2jyxo= 2023-07-29T00:09:38.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Iae3tQPOqpzcPTo3mCFeKw2jyxo=', 'Content-Length': '0'} 2023-07-29T00:09:38.880 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001a603fd2e07bc8b2-0064c458c2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.880 DEBUG:boto:path=/hiiqos-48/obj6 2023-07-29T00:09:38.881 DEBUG:boto:auth_path=/hiiqos-48/obj6 2023-07-29T00:09:38.881 DEBUG:boto:Method: GET 2023-07-29T00:09:38.881 DEBUG:boto:Path: /hiiqos-48/obj6 2023-07-29T00:09:38.882 DEBUG:boto:Data: 2023-07-29T00:09:38.882 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.882 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.883 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.883 DEBUG:boto:Params: {} 2023-07-29T00:09:38.883 DEBUG:boto:Token: None 2023-07-29T00:09:38.884 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj6 2023-07-29T00:09:38.884 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Iae3tQPOqpzcPTo3mCFeKw2jyxo= 2023-07-29T00:09:38.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Iae3tQPOqpzcPTo3mCFeKw2jyxo=', 'Content-Length': '0'} 2023-07-29T00:09:38.887 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000083d5e4568fae9746-0064c458c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.928 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:38.929 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:38.929 DEBUG:boto:path=/hiiqos-48/?prefix=obj6 2023-07-29T00:09:38.929 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj6 2023-07-29T00:09:38.930 DEBUG:boto:Method: GET 2023-07-29T00:09:38.930 DEBUG:boto:Path: /hiiqos-48/?prefix=obj6 2023-07-29T00:09:38.930 DEBUG:boto:Data: 2023-07-29T00:09:38.931 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.931 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.931 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.932 DEBUG:boto:Params: {} 2023-07-29T00:09:38.932 DEBUG:boto:Token: None 2023-07-29T00:09:38.933 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/ 2023-07-29T00:09:38.933 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA= 2023-07-29T00:09:38.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA=', 'Content-Length': '0'} 2023-07-29T00:09:38.935 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c5879ae17d7e714-0064c458c2-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.936 DEBUG:boto:b'hiiqos-48obj61000falseobj62023-07-29T00:09:22.178Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:38.937 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:38.937 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:38.937 DEBUG:boto:path=/hiiqos-48/?prefix=obj6 2023-07-29T00:09:38.938 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj6 2023-07-29T00:09:38.938 DEBUG:boto:Method: GET 2023-07-29T00:09:38.938 DEBUG:boto:Path: /hiiqos-48/?prefix=obj6 2023-07-29T00:09:38.939 DEBUG:boto:Data: 2023-07-29T00:09:38.939 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.939 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.940 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.940 DEBUG:boto:Params: {} 2023-07-29T00:09:38.940 DEBUG:boto:Token: None 2023-07-29T00:09:38.941 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/ 2023-07-29T00:09:38.941 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA= 2023-07-29T00:09:38.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA=', 'Content-Length': '0'} 2023-07-29T00:09:38.944 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ff7a40b368812bc-0064c458c2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.944 DEBUG:boto:b'hiiqos-48obj61000falseobj62023-07-29T00:09:22.178Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:38.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-07-29T00:09:38.945 DEBUG:boto:path=/hiiqos-48/obj7 2023-07-29T00:09:38.945 DEBUG:boto:auth_path=/hiiqos-48/obj7 2023-07-29T00:09:38.946 DEBUG:boto:path=/hiiqos-48/obj7?versionId=null 2023-07-29T00:09:38.946 DEBUG:boto:auth_path=/hiiqos-48/obj7?versionId=null 2023-07-29T00:09:38.946 DEBUG:boto:Method: GET 2023-07-29T00:09:38.947 DEBUG:boto:Path: /hiiqos-48/obj7?versionId=null 2023-07-29T00:09:38.947 DEBUG:boto:Data: 2023-07-29T00:09:38.947 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.948 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.948 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.949 DEBUG:boto:Params: {} 2023-07-29T00:09:38.949 DEBUG:boto:Token: None 2023-07-29T00:09:38.949 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj7?versionId=null 2023-07-29T00:09:38.950 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DLtcTvdpt9wdA/TPO7cj2/Hh/BU= 2023-07-29T00:09:38.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DLtcTvdpt9wdA/TPO7cj2/Hh/BU=', 'Content-Length': '0'} 2023-07-29T00:09:38.952 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000029269408a2e7d176-0064c458c2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.952 DEBUG:boto:path=/hiiqos-48/obj7 2023-07-29T00:09:38.953 DEBUG:boto:auth_path=/hiiqos-48/obj7 2023-07-29T00:09:38.953 DEBUG:boto:path=/hiiqos-48/obj7?versionId=null 2023-07-29T00:09:38.954 DEBUG:boto:auth_path=/hiiqos-48/obj7?versionId=null 2023-07-29T00:09:38.954 DEBUG:boto:Method: GET 2023-07-29T00:09:38.954 DEBUG:boto:Path: /hiiqos-48/obj7?versionId=null 2023-07-29T00:09:38.955 DEBUG:boto:Data: 2023-07-29T00:09:38.955 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.955 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.956 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.956 DEBUG:boto:Params: {} 2023-07-29T00:09:38.956 DEBUG:boto:Token: None 2023-07-29T00:09:38.957 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj7?versionId=null 2023-07-29T00:09:38.957 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DLtcTvdpt9wdA/TPO7cj2/Hh/BU= 2023-07-29T00:09:38.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DLtcTvdpt9wdA/TPO7cj2/Hh/BU=', 'Content-Length': '0'} 2023-07-29T00:09:38.959 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d55b47c962224815-0064c458c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.960 DEBUG:boto:path=/hiiqos-48/obj7 2023-07-29T00:09:38.960 DEBUG:boto:auth_path=/hiiqos-48/obj7 2023-07-29T00:09:38.960 DEBUG:boto:path=/hiiqos-48/obj7?versionId=null 2023-07-29T00:09:38.961 DEBUG:boto:auth_path=/hiiqos-48/obj7?versionId=null 2023-07-29T00:09:38.961 DEBUG:boto:Method: HEAD 2023-07-29T00:09:38.961 DEBUG:boto:Path: /hiiqos-48/obj7?versionId=null 2023-07-29T00:09:38.962 DEBUG:boto:Data: 2023-07-29T00:09:38.962 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.962 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.963 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.963 DEBUG:boto:Params: {} 2023-07-29T00:09:38.963 DEBUG:boto:Token: None 2023-07-29T00:09:38.964 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj7?versionId=null 2023-07-29T00:09:38.964 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LCdbSRpCC6jGBhki6JkHyIcyj7w= 2023-07-29T00:09:38.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LCdbSRpCC6jGBhki6JkHyIcyj7w=', 'Content-Length': '0'} 2023-07-29T00:09:38.966 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030b6924b01a35ff1-0064c458c2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.967 DEBUG:boto:path=/hiiqos-48/obj7 2023-07-29T00:09:38.967 DEBUG:boto:auth_path=/hiiqos-48/obj7 2023-07-29T00:09:38.968 DEBUG:boto:path=/hiiqos-48/obj7?versionId=null 2023-07-29T00:09:38.968 DEBUG:boto:auth_path=/hiiqos-48/obj7?versionId=null 2023-07-29T00:09:38.968 DEBUG:boto:Method: HEAD 2023-07-29T00:09:38.969 DEBUG:boto:Path: /hiiqos-48/obj7?versionId=null 2023-07-29T00:09:38.969 DEBUG:boto:Data: 2023-07-29T00:09:38.969 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.969 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.970 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.970 DEBUG:boto:Params: {} 2023-07-29T00:09:38.970 DEBUG:boto:Token: None 2023-07-29T00:09:38.971 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj7?versionId=null 2023-07-29T00:09:38.971 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LCdbSRpCC6jGBhki6JkHyIcyj7w= 2023-07-29T00:09:38.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LCdbSRpCC6jGBhki6JkHyIcyj7w=', 'Content-Length': '0'} 2023-07-29T00:09:38.973 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf4502c03a5e7097-0064c458c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-07-29T00:09:38.974 DEBUG:boto:path=/hiiqos-48/obj7 2023-07-29T00:09:38.975 DEBUG:boto:auth_path=/hiiqos-48/obj7 2023-07-29T00:09:38.975 DEBUG:boto:path=/hiiqos-48/obj7?versionId=null 2023-07-29T00:09:38.975 DEBUG:boto:auth_path=/hiiqos-48/obj7?versionId=null 2023-07-29T00:09:38.976 DEBUG:boto:Method: GET 2023-07-29T00:09:38.976 DEBUG:boto:Path: /hiiqos-48/obj7?versionId=null 2023-07-29T00:09:38.976 DEBUG:boto:Data: 2023-07-29T00:09:38.977 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.977 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.977 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.978 DEBUG:boto:Params: {} 2023-07-29T00:09:38.978 DEBUG:boto:Token: None 2023-07-29T00:09:38.979 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj7?versionId=null 2023-07-29T00:09:38.979 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DLtcTvdpt9wdA/TPO7cj2/Hh/BU= 2023-07-29T00:09:38.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DLtcTvdpt9wdA/TPO7cj2/Hh/BU=', 'Content-Length': '0'} 2023-07-29T00:09:38.981 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b15bc7f1bbdb6511-0064c458c2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.982 DEBUG:boto:path=/hiiqos-48/obj7 2023-07-29T00:09:38.982 DEBUG:boto:auth_path=/hiiqos-48/obj7 2023-07-29T00:09:38.982 DEBUG:boto:path=/hiiqos-48/obj7?versionId=null 2023-07-29T00:09:38.983 DEBUG:boto:auth_path=/hiiqos-48/obj7?versionId=null 2023-07-29T00:09:38.983 DEBUG:boto:Method: GET 2023-07-29T00:09:38.984 DEBUG:boto:Path: /hiiqos-48/obj7?versionId=null 2023-07-29T00:09:38.984 DEBUG:boto:Data: 2023-07-29T00:09:38.985 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.985 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:38.985 DEBUG:boto:Port: 8000 2023-07-29T00:09:38.986 DEBUG:boto:Params: {} 2023-07-29T00:09:38.986 DEBUG:boto:Token: None 2023-07-29T00:09:38.986 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/obj7?versionId=null 2023-07-29T00:09:38.987 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DLtcTvdpt9wdA/TPO7cj2/Hh/BU= 2023-07-29T00:09:38.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DLtcTvdpt9wdA/TPO7cj2/Hh/BU=', 'Content-Length': '0'} 2023-07-29T00:09:38.989 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb0dbcf099aabdb9-0064c458c2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:38.990 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:38.990 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:38.990 DEBUG:boto:path=/hiiqos-48/?prefix=obj7 2023-07-29T00:09:38.991 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj7 2023-07-29T00:09:38.991 DEBUG:boto:Method: GET 2023-07-29T00:09:38.991 DEBUG:boto:Path: /hiiqos-48/?prefix=obj7 2023-07-29T00:09:38.992 DEBUG:boto:Data: 2023-07-29T00:09:38.992 DEBUG:boto:Headers: {} 2023-07-29T00:09:38.992 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:38.993 DEBUG:boto:Port: 8001 2023-07-29T00:09:38.993 DEBUG:boto:Params: {} 2023-07-29T00:09:38.993 DEBUG:boto:Token: None 2023-07-29T00:09:38.994 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:38 GMT /hiiqos-48/ 2023-07-29T00:09:38.994 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA= 2023-07-29T00:09:38.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LQeVyTUHDhv0+XhuPuUmzrZlrWA=', 'Content-Length': '0'} 2023-07-29T00:09:38.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000896f9131d18ea552-0064c458c2-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:39.040 DEBUG:boto:b'hiiqos-48obj71000falseobj72023-07-29T00:09:22.237Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:39.041 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:39.041 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:39.042 DEBUG:boto:path=/hiiqos-48/?prefix=obj7 2023-07-29T00:09:39.042 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj7 2023-07-29T00:09:39.042 DEBUG:boto:Method: GET 2023-07-29T00:09:39.043 DEBUG:boto:Path: /hiiqos-48/?prefix=obj7 2023-07-29T00:09:39.043 DEBUG:boto:Data: 2023-07-29T00:09:39.043 DEBUG:boto:Headers: {} 2023-07-29T00:09:39.044 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:39.044 DEBUG:boto:Port: 8000 2023-07-29T00:09:39.044 DEBUG:boto:Params: {} 2023-07-29T00:09:39.045 DEBUG:boto:Token: None 2023-07-29T00:09:39.045 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:39 GMT /hiiqos-48/ 2023-07-29T00:09:39.046 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:NRP+TyC7CS+77/i24L/cNl0FsHo= 2023-07-29T00:09:39.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:NRP+TyC7CS+77/i24L/cNl0FsHo=', 'Content-Length': '0'} 2023-07-29T00:09:39.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cfc8b04dc2586e4-0064c458c3-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:39.048 DEBUG:boto:b'hiiqos-48obj71000falseobj72023-07-29T00:09:22.237Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:39.049 DEBUG:boto:path=/hiiqos-48/obj7 2023-07-29T00:09:39.049 DEBUG:boto:auth_path=/hiiqos-48/obj7 2023-07-29T00:09:39.050 DEBUG:boto:Method: HEAD 2023-07-29T00:09:39.050 DEBUG:boto:Path: /hiiqos-48/obj7 2023-07-29T00:09:39.050 DEBUG:boto:Data: 2023-07-29T00:09:39.051 DEBUG:boto:Headers: {} 2023-07-29T00:09:39.051 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:39.051 DEBUG:boto:Port: 8001 2023-07-29T00:09:39.052 DEBUG:boto:Params: {} 2023-07-29T00:09:39.052 DEBUG:boto:Token: None 2023-07-29T00:09:39.052 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:39 GMT /hiiqos-48/obj7 2023-07-29T00:09:39.053 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qJSX6Nw0pgReXuVNFaMh5px8YRE= 2023-07-29T00:09:39.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qJSX6Nw0pgReXuVNFaMh5px8YRE=', 'Content-Length': '0'} 2023-07-29T00:09:39.055 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e34920fb9f07f678-0064c458c3-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:39.055 DEBUG:boto:path=/hiiqos-48/obj7 2023-07-29T00:09:39.056 DEBUG:boto:auth_path=/hiiqos-48/obj7 2023-07-29T00:09:39.056 DEBUG:boto:Method: HEAD 2023-07-29T00:09:39.056 DEBUG:boto:Path: /hiiqos-48/obj7 2023-07-29T00:09:39.057 DEBUG:boto:Data: 2023-07-29T00:09:39.057 DEBUG:boto:Headers: {} 2023-07-29T00:09:39.057 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:39.058 DEBUG:boto:Port: 8000 2023-07-29T00:09:39.058 DEBUG:boto:Params: {} 2023-07-29T00:09:39.058 DEBUG:boto:Token: None 2023-07-29T00:09:39.059 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:39 GMT /hiiqos-48/obj7 2023-07-29T00:09:39.059 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qJSX6Nw0pgReXuVNFaMh5px8YRE= 2023-07-29T00:09:39.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qJSX6Nw0pgReXuVNFaMh5px8YRE=', 'Content-Length': '0'} 2023-07-29T00:09:39.061 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cbc6358354b12405-0064c458c3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:39.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-07-29T00:09:39.062 DEBUG:boto:path=/hiiqos-48/obj7 2023-07-29T00:09:39.062 DEBUG:boto:auth_path=/hiiqos-48/obj7 2023-07-29T00:09:39.062 DEBUG:boto:Method: GET 2023-07-29T00:09:39.063 DEBUG:boto:Path: /hiiqos-48/obj7 2023-07-29T00:09:39.063 DEBUG:boto:Data: 2023-07-29T00:09:39.063 DEBUG:boto:Headers: {} 2023-07-29T00:09:39.064 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:39.064 DEBUG:boto:Port: 8001 2023-07-29T00:09:39.064 DEBUG:boto:Params: {} 2023-07-29T00:09:39.065 DEBUG:boto:Token: None 2023-07-29T00:09:39.065 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:39 GMT /hiiqos-48/obj7 2023-07-29T00:09:39.066 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KQGZDzX/3bdYOWTMOboFQez3Lds= 2023-07-29T00:09:39.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KQGZDzX/3bdYOWTMOboFQez3Lds=', 'Content-Length': '0'} 2023-07-29T00:09:39.068 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000900bc4fb2935c0d9-0064c458c3-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:39.068 DEBUG:boto:path=/hiiqos-48/obj7 2023-07-29T00:09:39.069 DEBUG:boto:auth_path=/hiiqos-48/obj7 2023-07-29T00:09:39.069 DEBUG:boto:Method: GET 2023-07-29T00:09:39.069 DEBUG:boto:Path: /hiiqos-48/obj7 2023-07-29T00:09:39.070 DEBUG:boto:Data: 2023-07-29T00:09:39.070 DEBUG:boto:Headers: {} 2023-07-29T00:09:39.070 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:39.071 DEBUG:boto:Port: 8000 2023-07-29T00:09:39.071 DEBUG:boto:Params: {} 2023-07-29T00:09:39.071 DEBUG:boto:Token: None 2023-07-29T00:09:39.072 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:39 GMT /hiiqos-48/obj7 2023-07-29T00:09:39.072 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KQGZDzX/3bdYOWTMOboFQez3Lds= 2023-07-29T00:09:39.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KQGZDzX/3bdYOWTMOboFQez3Lds=', 'Content-Length': '0'} 2023-07-29T00:09:39.074 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000188a99ad84b48f79-0064c458c3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:39.075 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:39.075 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:39.075 DEBUG:boto:path=/hiiqos-48/?prefix=obj7 2023-07-29T00:09:39.076 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj7 2023-07-29T00:09:39.076 DEBUG:boto:Method: GET 2023-07-29T00:09:39.076 DEBUG:boto:Path: /hiiqos-48/?prefix=obj7 2023-07-29T00:09:39.077 DEBUG:boto:Data: 2023-07-29T00:09:39.077 DEBUG:boto:Headers: {} 2023-07-29T00:09:39.077 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:39.078 DEBUG:boto:Port: 8001 2023-07-29T00:09:39.078 DEBUG:boto:Params: {} 2023-07-29T00:09:39.079 DEBUG:boto:Token: None 2023-07-29T00:09:39.079 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:39 GMT /hiiqos-48/ 2023-07-29T00:09:39.079 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:NRP+TyC7CS+77/i24L/cNl0FsHo= 2023-07-29T00:09:39.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:NRP+TyC7CS+77/i24L/cNl0FsHo=', 'Content-Length': '0'} 2023-07-29T00:09:39.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ed1c477002d79b9-0064c458c3-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:39.082 DEBUG:boto:b'hiiqos-48obj71000falseobj72023-07-29T00:09:22.237Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:39.083 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:39.083 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:39.083 DEBUG:boto:path=/hiiqos-48/?prefix=obj7 2023-07-29T00:09:39.084 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj7 2023-07-29T00:09:39.084 DEBUG:boto:Method: GET 2023-07-29T00:09:39.084 DEBUG:boto:Path: /hiiqos-48/?prefix=obj7 2023-07-29T00:09:39.085 DEBUG:boto:Data: 2023-07-29T00:09:39.085 DEBUG:boto:Headers: {} 2023-07-29T00:09:39.085 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:39.086 DEBUG:boto:Port: 8000 2023-07-29T00:09:39.086 DEBUG:boto:Params: {} 2023-07-29T00:09:39.086 DEBUG:boto:Token: None 2023-07-29T00:09:39.087 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:39 GMT /hiiqos-48/ 2023-07-29T00:09:39.087 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:NRP+TyC7CS+77/i24L/cNl0FsHo= 2023-07-29T00:09:39.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:NRP+TyC7CS+77/i24L/cNl0FsHo=', 'Content-Length': '0'} 2023-07-29T00:09:39.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2f4f09251ebeab9-0064c458c3-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:39.090 DEBUG:boto:b'hiiqos-48obj71000falseobj72023-07-29T00:09:22.237Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:39.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-07-29T00:09:39.091 DEBUG:boto:path=/hiiqos-48/obj8 2023-07-29T00:09:39.092 DEBUG:boto:auth_path=/hiiqos-48/obj8 2023-07-29T00:09:39.092 DEBUG:boto:path=/hiiqos-48/obj8?versionId=null 2023-07-29T00:09:39.092 DEBUG:boto:auth_path=/hiiqos-48/obj8?versionId=null 2023-07-29T00:09:39.093 DEBUG:boto:Method: GET 2023-07-29T00:09:39.093 DEBUG:boto:Path: /hiiqos-48/obj8?versionId=null 2023-07-29T00:09:39.093 DEBUG:boto:Data: 2023-07-29T00:09:39.094 DEBUG:boto:Headers: {} 2023-07-29T00:09:39.094 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:39.094 DEBUG:boto:Port: 8001 2023-07-29T00:09:39.095 DEBUG:boto:Params: {} 2023-07-29T00:09:39.095 DEBUG:boto:Token: None 2023-07-29T00:09:39.095 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:39 GMT /hiiqos-48/obj8?versionId=null 2023-07-29T00:09:39.096 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5MTO8CTgxQLzND5NZwb1IMbL5qI= 2023-07-29T00:09:39.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5MTO8CTgxQLzND5NZwb1IMbL5qI=', 'Content-Length': '0'} 2023-07-29T00:09:39.098 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b97f1091a439524-0064c458c3-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:39.098 DEBUG:boto:path=/hiiqos-48/obj8 2023-07-29T00:09:39.099 DEBUG:boto:auth_path=/hiiqos-48/obj8 2023-07-29T00:09:39.099 DEBUG:boto:path=/hiiqos-48/obj8?versionId=null 2023-07-29T00:09:39.099 DEBUG:boto:auth_path=/hiiqos-48/obj8?versionId=null 2023-07-29T00:09:39.100 DEBUG:boto:Method: GET 2023-07-29T00:09:39.100 DEBUG:boto:Path: /hiiqos-48/obj8?versionId=null 2023-07-29T00:09:39.100 DEBUG:boto:Data: 2023-07-29T00:09:39.101 DEBUG:boto:Headers: {} 2023-07-29T00:09:39.101 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:39.102 DEBUG:boto:Port: 8000 2023-07-29T00:09:39.102 DEBUG:boto:Params: {} 2023-07-29T00:09:39.102 DEBUG:boto:Token: None 2023-07-29T00:09:39.103 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:39 GMT /hiiqos-48/obj8?versionId=null 2023-07-29T00:09:39.103 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5MTO8CTgxQLzND5NZwb1IMbL5qI= 2023-07-29T00:09:39.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5MTO8CTgxQLzND5NZwb1IMbL5qI=', 'Content-Length': '0'} 2023-07-29T00:09:39.105 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a44c463105fa12e1-0064c458c3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:39.106 DEBUG:boto:path=/hiiqos-48/obj8 2023-07-29T00:09:39.106 DEBUG:boto:auth_path=/hiiqos-48/obj8 2023-07-29T00:09:39.106 DEBUG:boto:path=/hiiqos-48/obj8?versionId=null 2023-07-29T00:09:39.107 DEBUG:boto:auth_path=/hiiqos-48/obj8?versionId=null 2023-07-29T00:09:39.107 DEBUG:boto:Method: HEAD 2023-07-29T00:09:39.107 DEBUG:boto:Path: /hiiqos-48/obj8?versionId=null 2023-07-29T00:09:39.108 DEBUG:boto:Data: 2023-07-29T00:09:39.108 DEBUG:boto:Headers: {} 2023-07-29T00:09:39.108 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:39.109 DEBUG:boto:Port: 8001 2023-07-29T00:09:39.109 DEBUG:boto:Params: {} 2023-07-29T00:09:39.109 DEBUG:boto:Token: None 2023-07-29T00:09:39.110 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:39 GMT /hiiqos-48/obj8?versionId=null 2023-07-29T00:09:39.110 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DqGPoVTa8du/8h6DDayhAZ5Rdrk= 2023-07-29T00:09:39.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DqGPoVTa8du/8h6DDayhAZ5Rdrk=', 'Content-Length': '0'} 2023-07-29T00:09:39.112 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000028ffc39d3b088fbb-0064c458c3-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:39.112 DEBUG:boto:path=/hiiqos-48/obj8 2023-07-29T00:09:39.113 DEBUG:boto:auth_path=/hiiqos-48/obj8 2023-07-29T00:09:39.113 DEBUG:boto:path=/hiiqos-48/obj8?versionId=null 2023-07-29T00:09:39.113 DEBUG:boto:auth_path=/hiiqos-48/obj8?versionId=null 2023-07-29T00:09:39.114 DEBUG:boto:Method: HEAD 2023-07-29T00:09:39.114 DEBUG:boto:Path: /hiiqos-48/obj8?versionId=null 2023-07-29T00:09:39.114 DEBUG:boto:Data: 2023-07-29T00:09:39.115 DEBUG:boto:Headers: {} 2023-07-29T00:09:39.115 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:39.115 DEBUG:boto:Port: 8000 2023-07-29T00:09:39.116 DEBUG:boto:Params: {} 2023-07-29T00:09:39.116 DEBUG:boto:Token: None 2023-07-29T00:09:39.116 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:39 GMT /hiiqos-48/obj8?versionId=null 2023-07-29T00:09:39.117 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DqGPoVTa8du/8h6DDayhAZ5Rdrk= 2023-07-29T00:09:39.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DqGPoVTa8du/8h6DDayhAZ5Rdrk=', 'Content-Length': '0'} 2023-07-29T00:09:39.118 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005472ba811e9b4981-0064c458c3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:39.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-07-29T00:09:39.119 DEBUG:boto:path=/hiiqos-48/obj8 2023-07-29T00:09:39.120 DEBUG:boto:auth_path=/hiiqos-48/obj8 2023-07-29T00:09:39.120 DEBUG:boto:path=/hiiqos-48/obj8?versionId=null 2023-07-29T00:09:39.120 DEBUG:boto:auth_path=/hiiqos-48/obj8?versionId=null 2023-07-29T00:09:39.121 DEBUG:boto:Method: GET 2023-07-29T00:09:39.121 DEBUG:boto:Path: /hiiqos-48/obj8?versionId=null 2023-07-29T00:09:39.121 DEBUG:boto:Data: 2023-07-29T00:09:39.122 DEBUG:boto:Headers: {} 2023-07-29T00:09:39.122 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:39.123 DEBUG:boto:Port: 8001 2023-07-29T00:09:39.123 DEBUG:boto:Params: {} 2023-07-29T00:09:39.123 DEBUG:boto:Token: None 2023-07-29T00:09:39.124 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:39 GMT /hiiqos-48/obj8?versionId=null 2023-07-29T00:09:39.124 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5MTO8CTgxQLzND5NZwb1IMbL5qI= 2023-07-29T00:09:39.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5MTO8CTgxQLzND5NZwb1IMbL5qI=', 'Content-Length': '0'} 2023-07-29T00:09:39.126 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eb67973d8e59f3e0-0064c458c3-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:39.126 DEBUG:boto:path=/hiiqos-48/obj8 2023-07-29T00:09:39.127 DEBUG:boto:auth_path=/hiiqos-48/obj8 2023-07-29T00:09:39.127 DEBUG:boto:path=/hiiqos-48/obj8?versionId=null 2023-07-29T00:09:39.128 DEBUG:boto:auth_path=/hiiqos-48/obj8?versionId=null 2023-07-29T00:09:39.128 DEBUG:boto:Method: GET 2023-07-29T00:09:39.128 DEBUG:boto:Path: /hiiqos-48/obj8?versionId=null 2023-07-29T00:09:39.129 DEBUG:boto:Data: 2023-07-29T00:09:39.129 DEBUG:boto:Headers: {} 2023-07-29T00:09:39.129 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:39.130 DEBUG:boto:Port: 8000 2023-07-29T00:09:39.130 DEBUG:boto:Params: {} 2023-07-29T00:09:39.130 DEBUG:boto:Token: None 2023-07-29T00:09:39.131 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:39 GMT /hiiqos-48/obj8?versionId=null 2023-07-29T00:09:39.131 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5MTO8CTgxQLzND5NZwb1IMbL5qI= 2023-07-29T00:09:39.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5MTO8CTgxQLzND5NZwb1IMbL5qI=', 'Content-Length': '0'} 2023-07-29T00:09:39.133 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000511eb344a89d391a-0064c458c3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:39.134 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:39.134 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:39.135 DEBUG:boto:path=/hiiqos-48/?prefix=obj8 2023-07-29T00:09:39.135 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj8 2023-07-29T00:09:39.135 DEBUG:boto:Method: GET 2023-07-29T00:09:39.136 DEBUG:boto:Path: /hiiqos-48/?prefix=obj8 2023-07-29T00:09:39.136 DEBUG:boto:Data: 2023-07-29T00:09:39.136 DEBUG:boto:Headers: {} 2023-07-29T00:09:39.137 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:39.137 DEBUG:boto:Port: 8001 2023-07-29T00:09:39.137 DEBUG:boto:Params: {} 2023-07-29T00:09:39.138 DEBUG:boto:Token: None 2023-07-29T00:09:39.138 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:39 GMT /hiiqos-48/ 2023-07-29T00:09:39.138 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:NRP+TyC7CS+77/i24L/cNl0FsHo= 2023-07-29T00:09:39.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:NRP+TyC7CS+77/i24L/cNl0FsHo=', 'Content-Length': '0'} 2023-07-29T00:09:39.141 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff090306d52b4cc2-0064c458c3-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:39.141 DEBUG:boto:b'hiiqos-48obj81000falseobj82023-07-29T00:09:22.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:39.142 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:39.142 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:39.143 DEBUG:boto:path=/hiiqos-48/?prefix=obj8 2023-07-29T00:09:39.143 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj8 2023-07-29T00:09:39.143 DEBUG:boto:Method: GET 2023-07-29T00:09:39.144 DEBUG:boto:Path: /hiiqos-48/?prefix=obj8 2023-07-29T00:09:39.144 DEBUG:boto:Data: 2023-07-29T00:09:39.144 DEBUG:boto:Headers: {} 2023-07-29T00:09:39.145 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:39.145 DEBUG:boto:Port: 8000 2023-07-29T00:09:39.145 DEBUG:boto:Params: {} 2023-07-29T00:09:39.146 DEBUG:boto:Token: None 2023-07-29T00:09:39.146 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:39 GMT /hiiqos-48/ 2023-07-29T00:09:39.147 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:NRP+TyC7CS+77/i24L/cNl0FsHo= 2023-07-29T00:09:39.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:NRP+TyC7CS+77/i24L/cNl0FsHo=', 'Content-Length': '0'} 2023-07-29T00:09:39.149 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000065d0bf6f4052293-0064c458c3-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:39.150 DEBUG:boto:b'hiiqos-48obj81000falseobj82023-07-29T00:09:22.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:39.150 DEBUG:boto:path=/hiiqos-48/obj8 2023-07-29T00:09:39.151 DEBUG:boto:auth_path=/hiiqos-48/obj8 2023-07-29T00:09:39.151 DEBUG:boto:Method: HEAD 2023-07-29T00:09:39.151 DEBUG:boto:Path: /hiiqos-48/obj8 2023-07-29T00:09:39.152 DEBUG:boto:Data: 2023-07-29T00:09:39.152 DEBUG:boto:Headers: {} 2023-07-29T00:09:39.152 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:39.153 DEBUG:boto:Port: 8001 2023-07-29T00:09:39.153 DEBUG:boto:Params: {} 2023-07-29T00:09:39.153 DEBUG:boto:Token: None 2023-07-29T00:09:39.154 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:39 GMT /hiiqos-48/obj8 2023-07-29T00:09:39.154 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zSh38D6qZMUK6HVG7IQZihTOThE= 2023-07-29T00:09:39.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zSh38D6qZMUK6HVG7IQZihTOThE=', 'Content-Length': '0'} 2023-07-29T00:09:39.156 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aeef070d61b6b307-0064c458c3-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:39.157 DEBUG:boto:path=/hiiqos-48/obj8 2023-07-29T00:09:39.157 DEBUG:boto:auth_path=/hiiqos-48/obj8 2023-07-29T00:09:39.157 DEBUG:boto:Method: HEAD 2023-07-29T00:09:39.158 DEBUG:boto:Path: /hiiqos-48/obj8 2023-07-29T00:09:39.158 DEBUG:boto:Data: 2023-07-29T00:09:39.158 DEBUG:boto:Headers: {} 2023-07-29T00:09:39.159 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:39.159 DEBUG:boto:Port: 8000 2023-07-29T00:09:39.159 DEBUG:boto:Params: {} 2023-07-29T00:09:39.160 DEBUG:boto:Token: None 2023-07-29T00:09:39.160 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:09:39 GMT /hiiqos-48/obj8 2023-07-29T00:09:39.160 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zSh38D6qZMUK6HVG7IQZihTOThE= 2023-07-29T00:09:39.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zSh38D6qZMUK6HVG7IQZihTOThE=', 'Content-Length': '0'} 2023-07-29T00:09:39.162 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1c3d2c39778c122-0064c458c3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:39.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-07-29T00:09:39.163 DEBUG:boto:path=/hiiqos-48/obj8 2023-07-29T00:09:39.164 DEBUG:boto:auth_path=/hiiqos-48/obj8 2023-07-29T00:09:39.164 DEBUG:boto:Method: GET 2023-07-29T00:09:39.164 DEBUG:boto:Path: /hiiqos-48/obj8 2023-07-29T00:09:39.165 DEBUG:boto:Data: 2023-07-29T00:09:39.165 DEBUG:boto:Headers: {} 2023-07-29T00:09:39.165 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:39.166 DEBUG:boto:Port: 8001 2023-07-29T00:09:39.166 DEBUG:boto:Params: {} 2023-07-29T00:09:39.166 DEBUG:boto:Token: None 2023-07-29T00:09:39.167 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:39 GMT /hiiqos-48/obj8 2023-07-29T00:09:39.167 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:avamYCWwIxMHJoff0NqkUfspDGE= 2023-07-29T00:09:39.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:avamYCWwIxMHJoff0NqkUfspDGE=', 'Content-Length': '0'} 2023-07-29T00:09:39.169 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-48:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000906a89e843603662-0064c458c3-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:39.212 DEBUG:boto:path=/hiiqos-48/obj8 2023-07-29T00:09:39.213 DEBUG:boto:auth_path=/hiiqos-48/obj8 2023-07-29T00:09:39.213 DEBUG:boto:Method: GET 2023-07-29T00:09:39.214 DEBUG:boto:Path: /hiiqos-48/obj8 2023-07-29T00:09:39.214 DEBUG:boto:Data: 2023-07-29T00:09:39.214 DEBUG:boto:Headers: {} 2023-07-29T00:09:39.215 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:39.215 DEBUG:boto:Port: 8000 2023-07-29T00:09:39.215 DEBUG:boto:Params: {} 2023-07-29T00:09:39.216 DEBUG:boto:Token: None 2023-07-29T00:09:39.216 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:39 GMT /hiiqos-48/obj8 2023-07-29T00:09:39.217 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:avamYCWwIxMHJoff0NqkUfspDGE= 2023-07-29T00:09:39.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:avamYCWwIxMHJoff0NqkUfspDGE=', 'Content-Length': '0'} 2023-07-29T00:09:39.219 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008dc080ec7ba0a6ce-0064c458c3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:09:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:39.219 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:39.220 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:39.220 DEBUG:boto:path=/hiiqos-48/?prefix=obj8 2023-07-29T00:09:39.221 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj8 2023-07-29T00:09:39.221 DEBUG:boto:Method: GET 2023-07-29T00:09:39.221 DEBUG:boto:Path: /hiiqos-48/?prefix=obj8 2023-07-29T00:09:39.222 DEBUG:boto:Data: 2023-07-29T00:09:39.222 DEBUG:boto:Headers: {} 2023-07-29T00:09:39.222 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:09:39.223 DEBUG:boto:Port: 8001 2023-07-29T00:09:39.223 DEBUG:boto:Params: {} 2023-07-29T00:09:39.223 DEBUG:boto:Token: None 2023-07-29T00:09:39.224 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:39 GMT /hiiqos-48/ 2023-07-29T00:09:39.224 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:NRP+TyC7CS+77/i24L/cNl0FsHo= 2023-07-29T00:09:39.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:NRP+TyC7CS+77/i24L/cNl0FsHo=', 'Content-Length': '0'} 2023-07-29T00:09:39.227 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b15cfb665988e23f-0064c458c3-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:39.227 DEBUG:boto:b'hiiqos-48obj81000falseobj82023-07-29T00:09:22.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:39.228 DEBUG:boto:path=/hiiqos-48/ 2023-07-29T00:09:39.228 DEBUG:boto:auth_path=/hiiqos-48/ 2023-07-29T00:09:39.229 DEBUG:boto:path=/hiiqos-48/?prefix=obj8 2023-07-29T00:09:39.229 DEBUG:boto:auth_path=/hiiqos-48/?prefix=obj8 2023-07-29T00:09:39.229 DEBUG:boto:Method: GET 2023-07-29T00:09:39.230 DEBUG:boto:Path: /hiiqos-48/?prefix=obj8 2023-07-29T00:09:39.230 DEBUG:boto:Data: 2023-07-29T00:09:39.230 DEBUG:boto:Headers: {} 2023-07-29T00:09:39.231 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:09:39.231 DEBUG:boto:Port: 8000 2023-07-29T00:09:39.231 DEBUG:boto:Params: {} 2023-07-29T00:09:39.232 DEBUG:boto:Token: None 2023-07-29T00:09:39.232 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:09:39 GMT /hiiqos-48/ 2023-07-29T00:09:39.233 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:NRP+TyC7CS+77/i24L/cNl0FsHo= 2023-07-29T00:09:39.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:09:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:NRP+TyC7CS+77/i24L/cNl0FsHo=', 'Content-Length': '0'} 2023-07-29T00:09:39.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009970c6f8106cd49e-0064c458c3-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:09:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:09:39.235 DEBUG:boto:b'hiiqos-48obj81000falseobj82023-07-29T00:09:22.258Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:09:39.236 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-48 zones={a2, a1} 2023-07-29T00:09:39.236 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-49 2023-07-29T00:09:39.237 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:09:39.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:09:39.238 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-49 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:09:39.293 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:09:39.293 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:09:39.326 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:09:39.294+0000 7fbd74b26880 1 waiting to reach incremental sync.. 2023-07-29T00:09:44.326 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:09:44.295+0000 7fbd74b26880 1 waiting to reach incremental sync.. 2023-07-29T00:09:49.327 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:09:49.295+0000 7fbd74b26880 1 waiting to reach incremental sync.. 2023-07-29T00:09:54.328 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:09:54.296+0000 7fbd74b26880 1 waiting to reach incremental sync.. 2023-07-29T00:09:59.329 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:09:59.297+0000 7fbd74b26880 1 waiting to reach incremental sync.. 2023-07-29T00:10:04.330 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:10:04.297+0000 7fbd74b26880 1 waiting to reach incremental sync.. 2023-07-29T00:10:09.331 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:10:09.298+0000 7fbd74b26880 1 waiting to reach incremental sync.. 2023-07-29T00:10:14.331 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:10:14.298+0000 7fbd74b26880 1 waiting to reach incremental sync.. 2023-07-29T00:10:16.601 INFO:tasks.rgw.c2.client.1.smithi139.stdout:2023-07-29T00:10:16.568+0000 7f5731a6a640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T00:10:19.332 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:10:19.299+0000 7fbd74b26880 1 waiting to reach incremental sync.. 2023-07-29T00:10:24.333 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:10:24.300+0000 7fbd74b26880 1 waiting to reach incremental sync.. 2023-07-29T00:10:29.333 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:10:29.300+0000 7fbd74b26880 1 waiting to reach incremental sync.. 2023-07-29T00:10:34.334 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:10:34.301+0000 7fbd74b26880 1 waiting to reach incremental sync.. 2023-07-29T00:10:39.336 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:10:39.303+0000 7fbd74b26880 1 bucket sync caught up with source: 2023-07-29T00:10:39.337 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000004.40.6, 00000000002.23.1, 00000000004.57.6, 00000000002.11.1, 00000000004.33.6, 00000000002.13.1, 00000000002.21.1, 00000000004.21.6, 00000000002.31.1, 00000000002.59.1, 00000000002.34.1] 2023-07-29T00:10:39.337 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000004.40.6, 00000000002.23.1, 00000000004.57.6, 00000000002.11.1, 00000000004.33.6, 00000000002.13.1, 00000000002.21.1, 00000000004.21.6, 00000000002.31.1, 00000000002.59.1, 00000000002.34.1] 2023-07-29T00:10:39.337 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:10:39.303+0000 7fbd74b26880 0 bucket checkpoint complete 2023-07-29T00:10:39.354 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-49 2023-07-29T00:10:39.354 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:10:39.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:10:39.355 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-49 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:10:39.414 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:10:39.415 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:10:39.451 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:10:39.417+0000 7fdfb6824880 1 bucket sync caught up with source: 2023-07-29T00:10:39.451 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000004.545.1, 00000000002.2417.1, 00000000004.400.1, 00000000002.506.1, 00000000004.1673.1, 00000000002.348.1, 00000000002.525.1, 00000000004.681.1, 00000000002.696.1, 00000000002.867.1, 00000000002.875.1] 2023-07-29T00:10:39.452 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000004.545.1, 00000000002.2417.1, 00000000004.400.1, 00000000002.506.1, 00000000004.1673.1, 00000000002.348.1, 00000000002.525.1, 00000000004.681.1, 00000000002.696.1, 00000000002.867.1, 00000000002.875.1] 2023-07-29T00:10:39.452 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:10:39.417+0000 7fdfb6824880 0 bucket checkpoint complete 2023-07-29T00:10:39.464 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-49 zones={a2, a1} 2023-07-29T00:10:39.465 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:10:39.465 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:10:39.466 DEBUG:boto:Method: HEAD 2023-07-29T00:10:39.466 DEBUG:boto:Path: /hiiqos-49/ 2023-07-29T00:10:39.466 DEBUG:boto:Data: 2023-07-29T00:10:39.467 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.467 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:39.467 DEBUG:boto:Port: 8001 2023-07-29T00:10:39.468 DEBUG:boto:Params: {} 2023-07-29T00:10:39.468 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-29T00:10:39.469 DEBUG:boto:Token: None 2023-07-29T00:10:39.469 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/ 2023-07-29T00:10:39.470 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dvY8u+z9kPXnbRPruEcJ1GtLpBg= 2023-07-29T00:10:39.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dvY8u+z9kPXnbRPruEcJ1GtLpBg=', 'Content-Length': '0'} 2023-07-29T00:10:39.474 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000126c3bf435f78513-0064c458ff-1663-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.475 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:10:39.475 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:10:39.476 DEBUG:boto:Method: HEAD 2023-07-29T00:10:39.476 DEBUG:boto:Path: /hiiqos-49/ 2023-07-29T00:10:39.476 DEBUG:boto:Data: 2023-07-29T00:10:39.477 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.477 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:39.477 DEBUG:boto:Port: 8000 2023-07-29T00:10:39.478 DEBUG:boto:Params: {} 2023-07-29T00:10:39.478 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-29T00:10:39.479 DEBUG:boto:Token: None 2023-07-29T00:10:39.479 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/ 2023-07-29T00:10:39.479 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dvY8u+z9kPXnbRPruEcJ1GtLpBg= 2023-07-29T00:10:39.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dvY8u+z9kPXnbRPruEcJ1GtLpBg=', 'Content-Length': '0'} 2023-07-29T00:10:39.483 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000de0ee174cdaf7f15-0064c458ff-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.483 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-29T00:10:39.484 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:10:39.484 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:10:39.484 DEBUG:boto:path=/hiiqos-49/?versions&max-keys=999 2023-07-29T00:10:39.485 DEBUG:boto:auth_path=/hiiqos-49/?versions&max-keys=999 2023-07-29T00:10:39.485 DEBUG:boto:Method: GET 2023-07-29T00:10:39.486 DEBUG:boto:Path: /hiiqos-49/?versions&max-keys=999 2023-07-29T00:10:39.486 DEBUG:boto:Data: 2023-07-29T00:10:39.486 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.487 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:39.487 DEBUG:boto:Port: 8001 2023-07-29T00:10:39.487 DEBUG:boto:Params: {} 2023-07-29T00:10:39.488 DEBUG:boto:Token: None 2023-07-29T00:10:39.488 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/?versions 2023-07-29T00:10:39.489 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qMWao6m5pFcgi+XxQAIvXgxUAPs= 2023-07-29T00:10:39.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qMWao6m5pFcgi+XxQAIvXgxUAPs=', 'Content-Length': '0'} 2023-07-29T00:10:39.492 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f96358416ce4d960-0064c458ff-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.493 DEBUG:boto:b'hiiqos-49999falseobj1nulltrue2023-07-29T00:08:34.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-07-29T00:08:34.191Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-07-29T00:08:34.246Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-07-29T00:08:34.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-07-29T00:09:22.313Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-07-29T00:09:22.330Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-07-29T00:09:22.388Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-07-29T00:09:22.406Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:10:39.495 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-07-29T00:10:39.495 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-07-29T00:10:39.496 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-07-29T00:10:39.496 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-07-29T00:10:39.496 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-07-29T00:10:39.497 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-07-29T00:10:39.497 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-07-29T00:10:39.497 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-07-29T00:10:39.498 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-29T00:10:39.498 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:10:39.499 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:10:39.499 DEBUG:boto:path=/hiiqos-49/?versions&max-keys=999 2023-07-29T00:10:39.499 DEBUG:boto:auth_path=/hiiqos-49/?versions&max-keys=999 2023-07-29T00:10:39.500 DEBUG:boto:Method: GET 2023-07-29T00:10:39.500 DEBUG:boto:Path: /hiiqos-49/?versions&max-keys=999 2023-07-29T00:10:39.500 DEBUG:boto:Data: 2023-07-29T00:10:39.501 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.501 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:39.502 DEBUG:boto:Port: 8000 2023-07-29T00:10:39.502 DEBUG:boto:Params: {} 2023-07-29T00:10:39.502 DEBUG:boto:Token: None 2023-07-29T00:10:39.503 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/?versions 2023-07-29T00:10:39.503 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qMWao6m5pFcgi+XxQAIvXgxUAPs= 2023-07-29T00:10:39.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qMWao6m5pFcgi+XxQAIvXgxUAPs=', 'Content-Length': '0'} 2023-07-29T00:10:39.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075b1d95387a04d9a-0064c458ff-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.507 DEBUG:boto:b'hiiqos-49999falseobj1nulltrue2023-07-29T00:08:34.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-07-29T00:08:34.191Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-07-29T00:08:34.246Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-07-29T00:08:34.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-07-29T00:09:22.313Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-07-29T00:09:22.330Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-07-29T00:09:22.388Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-07-29T00:09:22.406Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:10:39.508 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-07-29T00:10:39.508 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-07-29T00:10:39.509 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-07-29T00:10:39.509 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-07-29T00:10:39.509 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-07-29T00:10:39.510 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-07-29T00:10:39.510 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-07-29T00:10:39.510 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-07-29T00:10:39.511 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:10:39.511 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:10:39.512 DEBUG:boto:path=/hiiqos-49/?versions&max-keys=999 2023-07-29T00:10:39.512 DEBUG:boto:auth_path=/hiiqos-49/?versions&max-keys=999 2023-07-29T00:10:39.513 DEBUG:boto:Method: GET 2023-07-29T00:10:39.513 DEBUG:boto:Path: /hiiqos-49/?versions&max-keys=999 2023-07-29T00:10:39.513 DEBUG:boto:Data: 2023-07-29T00:10:39.514 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.514 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:39.514 DEBUG:boto:Port: 8001 2023-07-29T00:10:39.515 DEBUG:boto:Params: {} 2023-07-29T00:10:39.515 DEBUG:boto:Token: None 2023-07-29T00:10:39.515 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/?versions 2023-07-29T00:10:39.516 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qMWao6m5pFcgi+XxQAIvXgxUAPs= 2023-07-29T00:10:39.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qMWao6m5pFcgi+XxQAIvXgxUAPs=', 'Content-Length': '0'} 2023-07-29T00:10:39.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006894fb99f7d23c08-0064c458ff-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.519 DEBUG:boto:b'hiiqos-49999falseobj1nulltrue2023-07-29T00:08:34.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-07-29T00:08:34.191Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-07-29T00:08:34.246Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-07-29T00:08:34.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-07-29T00:09:22.313Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-07-29T00:09:22.330Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-07-29T00:09:22.388Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-07-29T00:09:22.406Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:10:39.520 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:10:39.521 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:10:39.521 DEBUG:boto:path=/hiiqos-49/?versions&max-keys=999 2023-07-29T00:10:39.522 DEBUG:boto:auth_path=/hiiqos-49/?versions&max-keys=999 2023-07-29T00:10:39.522 DEBUG:boto:Method: GET 2023-07-29T00:10:39.522 DEBUG:boto:Path: /hiiqos-49/?versions&max-keys=999 2023-07-29T00:10:39.523 DEBUG:boto:Data: 2023-07-29T00:10:39.523 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.523 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:39.524 DEBUG:boto:Port: 8000 2023-07-29T00:10:39.524 DEBUG:boto:Params: {} 2023-07-29T00:10:39.525 DEBUG:boto:Token: None 2023-07-29T00:10:39.525 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/?versions 2023-07-29T00:10:39.526 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qMWao6m5pFcgi+XxQAIvXgxUAPs= 2023-07-29T00:10:39.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qMWao6m5pFcgi+XxQAIvXgxUAPs=', 'Content-Length': '0'} 2023-07-29T00:10:39.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011a4cd1cee997521-0064c458ff-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.529 DEBUG:boto:b'hiiqos-49999falseobj1nulltrue2023-07-29T00:08:34.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-07-29T00:08:34.191Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-07-29T00:08:34.246Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-07-29T00:08:34.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-07-29T00:09:22.313Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-07-29T00:09:22.330Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-07-29T00:09:22.388Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-07-29T00:09:22.406Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:10:39.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-07-29T00:10:39.530 DEBUG:boto:path=/hiiqos-49/obj1 2023-07-29T00:10:39.530 DEBUG:boto:auth_path=/hiiqos-49/obj1 2023-07-29T00:10:39.531 DEBUG:boto:path=/hiiqos-49/obj1?versionId=null 2023-07-29T00:10:39.531 DEBUG:boto:auth_path=/hiiqos-49/obj1?versionId=null 2023-07-29T00:10:39.532 DEBUG:boto:Method: GET 2023-07-29T00:10:39.532 DEBUG:boto:Path: /hiiqos-49/obj1?versionId=null 2023-07-29T00:10:39.532 DEBUG:boto:Data: 2023-07-29T00:10:39.533 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.533 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:39.533 DEBUG:boto:Port: 8001 2023-07-29T00:10:39.534 DEBUG:boto:Params: {} 2023-07-29T00:10:39.534 DEBUG:boto:Token: None 2023-07-29T00:10:39.535 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/obj1?versionId=null 2023-07-29T00:10:39.535 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HRVUMSKwzMZ8sNCKGagQIVJKyGg= 2023-07-29T00:10:39.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HRVUMSKwzMZ8sNCKGagQIVJKyGg=', 'Content-Length': '0'} 2023-07-29T00:10:39.537 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e2fe6f0dd06c756-0064c458ff-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.538 DEBUG:boto:path=/hiiqos-49/obj1 2023-07-29T00:10:39.538 DEBUG:boto:auth_path=/hiiqos-49/obj1 2023-07-29T00:10:39.539 DEBUG:boto:path=/hiiqos-49/obj1?versionId=null 2023-07-29T00:10:39.539 DEBUG:boto:auth_path=/hiiqos-49/obj1?versionId=null 2023-07-29T00:10:39.539 DEBUG:boto:Method: GET 2023-07-29T00:10:39.540 DEBUG:boto:Path: /hiiqos-49/obj1?versionId=null 2023-07-29T00:10:39.540 DEBUG:boto:Data: 2023-07-29T00:10:39.540 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.541 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:39.541 DEBUG:boto:Port: 8000 2023-07-29T00:10:39.542 DEBUG:boto:Params: {} 2023-07-29T00:10:39.542 DEBUG:boto:Token: None 2023-07-29T00:10:39.543 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/obj1?versionId=null 2023-07-29T00:10:39.544 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HRVUMSKwzMZ8sNCKGagQIVJKyGg= 2023-07-29T00:10:39.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HRVUMSKwzMZ8sNCKGagQIVJKyGg=', 'Content-Length': '0'} 2023-07-29T00:10:39.548 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000041f434fb07b50bcd-0064c458ff-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.549 DEBUG:boto:path=/hiiqos-49/obj1 2023-07-29T00:10:39.550 DEBUG:boto:auth_path=/hiiqos-49/obj1 2023-07-29T00:10:39.550 DEBUG:boto:path=/hiiqos-49/obj1?versionId=null 2023-07-29T00:10:39.551 DEBUG:boto:auth_path=/hiiqos-49/obj1?versionId=null 2023-07-29T00:10:39.551 DEBUG:boto:Method: HEAD 2023-07-29T00:10:39.552 DEBUG:boto:Path: /hiiqos-49/obj1?versionId=null 2023-07-29T00:10:39.553 DEBUG:boto:Data: 2023-07-29T00:10:39.553 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.553 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:39.554 DEBUG:boto:Port: 8001 2023-07-29T00:10:39.554 DEBUG:boto:Params: {} 2023-07-29T00:10:39.554 DEBUG:boto:Token: None 2023-07-29T00:10:39.555 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/obj1?versionId=null 2023-07-29T00:10:39.555 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8T+4g9blnkmKFXL6VCYaRn7XOz8= 2023-07-29T00:10:39.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8T+4g9blnkmKFXL6VCYaRn7XOz8=', 'Content-Length': '0'} 2023-07-29T00:10:39.558 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000090f9cc443bbe37e7-0064c458ff-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.558 DEBUG:boto:path=/hiiqos-49/obj1 2023-07-29T00:10:39.559 DEBUG:boto:auth_path=/hiiqos-49/obj1 2023-07-29T00:10:39.559 DEBUG:boto:path=/hiiqos-49/obj1?versionId=null 2023-07-29T00:10:39.559 DEBUG:boto:auth_path=/hiiqos-49/obj1?versionId=null 2023-07-29T00:10:39.560 DEBUG:boto:Method: HEAD 2023-07-29T00:10:39.561 DEBUG:boto:Path: /hiiqos-49/obj1?versionId=null 2023-07-29T00:10:39.561 DEBUG:boto:Data: 2023-07-29T00:10:39.561 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.562 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:39.562 DEBUG:boto:Port: 8000 2023-07-29T00:10:39.563 DEBUG:boto:Params: {} 2023-07-29T00:10:39.563 DEBUG:boto:Token: None 2023-07-29T00:10:39.564 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/obj1?versionId=null 2023-07-29T00:10:39.564 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8T+4g9blnkmKFXL6VCYaRn7XOz8= 2023-07-29T00:10:39.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8T+4g9blnkmKFXL6VCYaRn7XOz8=', 'Content-Length': '0'} 2023-07-29T00:10:39.568 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f5a96b6d34f5e446-0064c458ff-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-07-29T00:10:39.570 DEBUG:boto:path=/hiiqos-49/obj1 2023-07-29T00:10:39.570 DEBUG:boto:auth_path=/hiiqos-49/obj1 2023-07-29T00:10:39.571 DEBUG:boto:path=/hiiqos-49/obj1?versionId=null 2023-07-29T00:10:39.572 DEBUG:boto:auth_path=/hiiqos-49/obj1?versionId=null 2023-07-29T00:10:39.572 DEBUG:boto:Method: GET 2023-07-29T00:10:39.573 DEBUG:boto:Path: /hiiqos-49/obj1?versionId=null 2023-07-29T00:10:39.574 DEBUG:boto:Data: 2023-07-29T00:10:39.574 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.574 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:39.575 DEBUG:boto:Port: 8001 2023-07-29T00:10:39.575 DEBUG:boto:Params: {} 2023-07-29T00:10:39.576 DEBUG:boto:Token: None 2023-07-29T00:10:39.576 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/obj1?versionId=null 2023-07-29T00:10:39.577 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HRVUMSKwzMZ8sNCKGagQIVJKyGg= 2023-07-29T00:10:39.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HRVUMSKwzMZ8sNCKGagQIVJKyGg=', 'Content-Length': '0'} 2023-07-29T00:10:39.579 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002506e8d83d267fb6-0064c458ff-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.579 DEBUG:boto:path=/hiiqos-49/obj1 2023-07-29T00:10:39.580 DEBUG:boto:auth_path=/hiiqos-49/obj1 2023-07-29T00:10:39.580 DEBUG:boto:path=/hiiqos-49/obj1?versionId=null 2023-07-29T00:10:39.581 DEBUG:boto:auth_path=/hiiqos-49/obj1?versionId=null 2023-07-29T00:10:39.581 DEBUG:boto:Method: GET 2023-07-29T00:10:39.581 DEBUG:boto:Path: /hiiqos-49/obj1?versionId=null 2023-07-29T00:10:39.582 DEBUG:boto:Data: 2023-07-29T00:10:39.582 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.582 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:39.583 DEBUG:boto:Port: 8000 2023-07-29T00:10:39.583 DEBUG:boto:Params: {} 2023-07-29T00:10:39.583 DEBUG:boto:Token: None 2023-07-29T00:10:39.584 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/obj1?versionId=null 2023-07-29T00:10:39.584 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HRVUMSKwzMZ8sNCKGagQIVJKyGg= 2023-07-29T00:10:39.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HRVUMSKwzMZ8sNCKGagQIVJKyGg=', 'Content-Length': '0'} 2023-07-29T00:10:39.587 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000edb4347b6cf62428-0064c458ff-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.587 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:10:39.588 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:10:39.588 DEBUG:boto:path=/hiiqos-49/?prefix=obj1 2023-07-29T00:10:39.589 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj1 2023-07-29T00:10:39.589 DEBUG:boto:Method: GET 2023-07-29T00:10:39.590 DEBUG:boto:Path: /hiiqos-49/?prefix=obj1 2023-07-29T00:10:39.590 DEBUG:boto:Data: 2023-07-29T00:10:39.590 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.591 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:39.591 DEBUG:boto:Port: 8001 2023-07-29T00:10:39.591 DEBUG:boto:Params: {} 2023-07-29T00:10:39.592 DEBUG:boto:Token: None 2023-07-29T00:10:39.592 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/ 2023-07-29T00:10:39.593 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LsG1Uend1b5x+h5hPwHfXmQpEAM= 2023-07-29T00:10:39.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LsG1Uend1b5x+h5hPwHfXmQpEAM=', 'Content-Length': '0'} 2023-07-29T00:10:39.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c20ff8ab44e0b9f9-0064c458ff-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.596 DEBUG:boto:b'hiiqos-49obj11000falseobj12023-07-29T00:08:34.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:10:39.596 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:10:39.597 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:10:39.597 DEBUG:boto:path=/hiiqos-49/?prefix=obj1 2023-07-29T00:10:39.598 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj1 2023-07-29T00:10:39.598 DEBUG:boto:Method: GET 2023-07-29T00:10:39.598 DEBUG:boto:Path: /hiiqos-49/?prefix=obj1 2023-07-29T00:10:39.599 DEBUG:boto:Data: 2023-07-29T00:10:39.599 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.599 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:39.600 DEBUG:boto:Port: 8000 2023-07-29T00:10:39.600 DEBUG:boto:Params: {} 2023-07-29T00:10:39.601 DEBUG:boto:Token: None 2023-07-29T00:10:39.601 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/ 2023-07-29T00:10:39.601 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LsG1Uend1b5x+h5hPwHfXmQpEAM= 2023-07-29T00:10:39.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LsG1Uend1b5x+h5hPwHfXmQpEAM=', 'Content-Length': '0'} 2023-07-29T00:10:39.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012f31cf9e21aa35a-0064c458ff-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.605 DEBUG:boto:b'hiiqos-49obj11000falseobj12023-07-29T00:08:34.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:10:39.605 DEBUG:boto:path=/hiiqos-49/obj1 2023-07-29T00:10:39.606 DEBUG:boto:auth_path=/hiiqos-49/obj1 2023-07-29T00:10:39.606 DEBUG:boto:Method: HEAD 2023-07-29T00:10:39.607 DEBUG:boto:Path: /hiiqos-49/obj1 2023-07-29T00:10:39.607 DEBUG:boto:Data: 2023-07-29T00:10:39.607 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.608 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:39.608 DEBUG:boto:Port: 8001 2023-07-29T00:10:39.608 DEBUG:boto:Params: {} 2023-07-29T00:10:39.609 DEBUG:boto:Token: None 2023-07-29T00:10:39.609 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/obj1 2023-07-29T00:10:39.609 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2K356Hk/9T8mEOKVyKIJpfXr2sQ= 2023-07-29T00:10:39.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2K356Hk/9T8mEOKVyKIJpfXr2sQ=', 'Content-Length': '0'} 2023-07-29T00:10:39.612 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb57e6f4c29b413d-0064c458ff-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.612 DEBUG:boto:path=/hiiqos-49/obj1 2023-07-29T00:10:39.613 DEBUG:boto:auth_path=/hiiqos-49/obj1 2023-07-29T00:10:39.613 DEBUG:boto:Method: HEAD 2023-07-29T00:10:39.613 DEBUG:boto:Path: /hiiqos-49/obj1 2023-07-29T00:10:39.614 DEBUG:boto:Data: 2023-07-29T00:10:39.614 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.614 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:39.615 DEBUG:boto:Port: 8000 2023-07-29T00:10:39.615 DEBUG:boto:Params: {} 2023-07-29T00:10:39.616 DEBUG:boto:Token: None 2023-07-29T00:10:39.616 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/obj1 2023-07-29T00:10:39.616 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2K356Hk/9T8mEOKVyKIJpfXr2sQ= 2023-07-29T00:10:39.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2K356Hk/9T8mEOKVyKIJpfXr2sQ=', 'Content-Length': '0'} 2023-07-29T00:10:39.618 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000020a71baaf376e392-0064c458ff-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-07-29T00:10:39.619 DEBUG:boto:path=/hiiqos-49/obj1 2023-07-29T00:10:39.620 DEBUG:boto:auth_path=/hiiqos-49/obj1 2023-07-29T00:10:39.620 DEBUG:boto:Method: GET 2023-07-29T00:10:39.621 DEBUG:boto:Path: /hiiqos-49/obj1 2023-07-29T00:10:39.621 DEBUG:boto:Data: 2023-07-29T00:10:39.621 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.622 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:39.622 DEBUG:boto:Port: 8001 2023-07-29T00:10:39.622 DEBUG:boto:Params: {} 2023-07-29T00:10:39.623 DEBUG:boto:Token: None 2023-07-29T00:10:39.623 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/obj1 2023-07-29T00:10:39.624 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5VxXEE21713wALBembiQnwuH0BE= 2023-07-29T00:10:39.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5VxXEE21713wALBembiQnwuH0BE=', 'Content-Length': '0'} 2023-07-29T00:10:39.626 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cbf15c38f5f93528-0064c458ff-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.626 DEBUG:boto:path=/hiiqos-49/obj1 2023-07-29T00:10:39.627 DEBUG:boto:auth_path=/hiiqos-49/obj1 2023-07-29T00:10:39.627 DEBUG:boto:Method: GET 2023-07-29T00:10:39.628 DEBUG:boto:Path: /hiiqos-49/obj1 2023-07-29T00:10:39.628 DEBUG:boto:Data: 2023-07-29T00:10:39.628 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.629 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:39.629 DEBUG:boto:Port: 8000 2023-07-29T00:10:39.629 DEBUG:boto:Params: {} 2023-07-29T00:10:39.630 DEBUG:boto:Token: None 2023-07-29T00:10:39.630 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/obj1 2023-07-29T00:10:39.631 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5VxXEE21713wALBembiQnwuH0BE= 2023-07-29T00:10:39.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5VxXEE21713wALBembiQnwuH0BE=', 'Content-Length': '0'} 2023-07-29T00:10:39.632 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006cce88ca23b4a03d-0064c458ff-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.633 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:10:39.634 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:10:39.634 DEBUG:boto:path=/hiiqos-49/?prefix=obj1 2023-07-29T00:10:39.634 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj1 2023-07-29T00:10:39.635 DEBUG:boto:Method: GET 2023-07-29T00:10:39.635 DEBUG:boto:Path: /hiiqos-49/?prefix=obj1 2023-07-29T00:10:39.635 DEBUG:boto:Data: 2023-07-29T00:10:39.636 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.636 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:39.636 DEBUG:boto:Port: 8001 2023-07-29T00:10:39.637 DEBUG:boto:Params: {} 2023-07-29T00:10:39.637 DEBUG:boto:Token: None 2023-07-29T00:10:39.637 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/ 2023-07-29T00:10:39.638 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LsG1Uend1b5x+h5hPwHfXmQpEAM= 2023-07-29T00:10:39.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LsG1Uend1b5x+h5hPwHfXmQpEAM=', 'Content-Length': '0'} 2023-07-29T00:10:39.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b97538ba17ec5f2f-0064c458ff-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.641 DEBUG:boto:b'hiiqos-49obj11000falseobj12023-07-29T00:08:34.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:10:39.641 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:10:39.642 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:10:39.642 DEBUG:boto:path=/hiiqos-49/?prefix=obj1 2023-07-29T00:10:39.642 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj1 2023-07-29T00:10:39.643 DEBUG:boto:Method: GET 2023-07-29T00:10:39.643 DEBUG:boto:Path: /hiiqos-49/?prefix=obj1 2023-07-29T00:10:39.644 DEBUG:boto:Data: 2023-07-29T00:10:39.644 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.644 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:39.645 DEBUG:boto:Port: 8000 2023-07-29T00:10:39.645 DEBUG:boto:Params: {} 2023-07-29T00:10:39.645 DEBUG:boto:Token: None 2023-07-29T00:10:39.646 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/ 2023-07-29T00:10:39.646 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LsG1Uend1b5x+h5hPwHfXmQpEAM= 2023-07-29T00:10:39.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LsG1Uend1b5x+h5hPwHfXmQpEAM=', 'Content-Length': '0'} 2023-07-29T00:10:39.649 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017c6fce8b2f0b064-0064c458ff-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.692 DEBUG:boto:b'hiiqos-49obj11000falseobj12023-07-29T00:08:34.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:10:39.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-07-29T00:10:39.694 DEBUG:boto:path=/hiiqos-49/obj2 2023-07-29T00:10:39.694 DEBUG:boto:auth_path=/hiiqos-49/obj2 2023-07-29T00:10:39.694 DEBUG:boto:path=/hiiqos-49/obj2?versionId=null 2023-07-29T00:10:39.695 DEBUG:boto:auth_path=/hiiqos-49/obj2?versionId=null 2023-07-29T00:10:39.695 DEBUG:boto:Method: GET 2023-07-29T00:10:39.695 DEBUG:boto:Path: /hiiqos-49/obj2?versionId=null 2023-07-29T00:10:39.696 DEBUG:boto:Data: 2023-07-29T00:10:39.696 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.696 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:39.697 DEBUG:boto:Port: 8001 2023-07-29T00:10:39.697 DEBUG:boto:Params: {} 2023-07-29T00:10:39.698 DEBUG:boto:Token: None 2023-07-29T00:10:39.698 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/obj2?versionId=null 2023-07-29T00:10:39.699 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KKNAdERks6AUM24Bui6wMVreAcg= 2023-07-29T00:10:39.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KKNAdERks6AUM24Bui6wMVreAcg=', 'Content-Length': '0'} 2023-07-29T00:10:39.701 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000090edefd8a6857e48-0064c458ff-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.702 DEBUG:boto:path=/hiiqos-49/obj2 2023-07-29T00:10:39.702 DEBUG:boto:auth_path=/hiiqos-49/obj2 2023-07-29T00:10:39.703 DEBUG:boto:path=/hiiqos-49/obj2?versionId=null 2023-07-29T00:10:39.703 DEBUG:boto:auth_path=/hiiqos-49/obj2?versionId=null 2023-07-29T00:10:39.703 DEBUG:boto:Method: GET 2023-07-29T00:10:39.704 DEBUG:boto:Path: /hiiqos-49/obj2?versionId=null 2023-07-29T00:10:39.704 DEBUG:boto:Data: 2023-07-29T00:10:39.705 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.705 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:39.705 DEBUG:boto:Port: 8000 2023-07-29T00:10:39.706 DEBUG:boto:Params: {} 2023-07-29T00:10:39.706 DEBUG:boto:Token: None 2023-07-29T00:10:39.707 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/obj2?versionId=null 2023-07-29T00:10:39.707 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KKNAdERks6AUM24Bui6wMVreAcg= 2023-07-29T00:10:39.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KKNAdERks6AUM24Bui6wMVreAcg=', 'Content-Length': '0'} 2023-07-29T00:10:39.709 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f444da9fd7dd974-0064c458ff-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.710 DEBUG:boto:path=/hiiqos-49/obj2 2023-07-29T00:10:39.711 DEBUG:boto:auth_path=/hiiqos-49/obj2 2023-07-29T00:10:39.712 DEBUG:boto:path=/hiiqos-49/obj2?versionId=null 2023-07-29T00:10:39.712 DEBUG:boto:auth_path=/hiiqos-49/obj2?versionId=null 2023-07-29T00:10:39.713 DEBUG:boto:Method: HEAD 2023-07-29T00:10:39.713 DEBUG:boto:Path: /hiiqos-49/obj2?versionId=null 2023-07-29T00:10:39.714 DEBUG:boto:Data: 2023-07-29T00:10:39.714 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.715 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:39.715 DEBUG:boto:Port: 8001 2023-07-29T00:10:39.716 DEBUG:boto:Params: {} 2023-07-29T00:10:39.716 DEBUG:boto:Token: None 2023-07-29T00:10:39.716 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/obj2?versionId=null 2023-07-29T00:10:39.717 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HjeG4SkcAdoK2CLzvSVE+5P5YVk= 2023-07-29T00:10:39.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HjeG4SkcAdoK2CLzvSVE+5P5YVk=', 'Content-Length': '0'} 2023-07-29T00:10:39.719 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006033e6b55e10fa3d-0064c458ff-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.720 DEBUG:boto:path=/hiiqos-49/obj2 2023-07-29T00:10:39.720 DEBUG:boto:auth_path=/hiiqos-49/obj2 2023-07-29T00:10:39.721 DEBUG:boto:path=/hiiqos-49/obj2?versionId=null 2023-07-29T00:10:39.721 DEBUG:boto:auth_path=/hiiqos-49/obj2?versionId=null 2023-07-29T00:10:39.722 DEBUG:boto:Method: HEAD 2023-07-29T00:10:39.722 DEBUG:boto:Path: /hiiqos-49/obj2?versionId=null 2023-07-29T00:10:39.723 DEBUG:boto:Data: 2023-07-29T00:10:39.723 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.723 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:39.724 DEBUG:boto:Port: 8000 2023-07-29T00:10:39.724 DEBUG:boto:Params: {} 2023-07-29T00:10:39.724 DEBUG:boto:Token: None 2023-07-29T00:10:39.725 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/obj2?versionId=null 2023-07-29T00:10:39.725 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HjeG4SkcAdoK2CLzvSVE+5P5YVk= 2023-07-29T00:10:39.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HjeG4SkcAdoK2CLzvSVE+5P5YVk=', 'Content-Length': '0'} 2023-07-29T00:10:39.727 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bab4151a1edd75dc-0064c458ff-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-07-29T00:10:39.728 DEBUG:boto:path=/hiiqos-49/obj2 2023-07-29T00:10:39.729 DEBUG:boto:auth_path=/hiiqos-49/obj2 2023-07-29T00:10:39.729 DEBUG:boto:path=/hiiqos-49/obj2?versionId=null 2023-07-29T00:10:39.729 DEBUG:boto:auth_path=/hiiqos-49/obj2?versionId=null 2023-07-29T00:10:39.730 DEBUG:boto:Method: GET 2023-07-29T00:10:39.730 DEBUG:boto:Path: /hiiqos-49/obj2?versionId=null 2023-07-29T00:10:39.731 DEBUG:boto:Data: 2023-07-29T00:10:39.731 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.731 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:39.732 DEBUG:boto:Port: 8001 2023-07-29T00:10:39.732 DEBUG:boto:Params: {} 2023-07-29T00:10:39.732 DEBUG:boto:Token: None 2023-07-29T00:10:39.733 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/obj2?versionId=null 2023-07-29T00:10:39.733 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KKNAdERks6AUM24Bui6wMVreAcg= 2023-07-29T00:10:39.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KKNAdERks6AUM24Bui6wMVreAcg=', 'Content-Length': '0'} 2023-07-29T00:10:39.735 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009046f71584f5cc3f-0064c458ff-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.736 DEBUG:boto:path=/hiiqos-49/obj2 2023-07-29T00:10:39.737 DEBUG:boto:auth_path=/hiiqos-49/obj2 2023-07-29T00:10:39.737 DEBUG:boto:path=/hiiqos-49/obj2?versionId=null 2023-07-29T00:10:39.737 DEBUG:boto:auth_path=/hiiqos-49/obj2?versionId=null 2023-07-29T00:10:39.738 DEBUG:boto:Method: GET 2023-07-29T00:10:39.738 DEBUG:boto:Path: /hiiqos-49/obj2?versionId=null 2023-07-29T00:10:39.738 DEBUG:boto:Data: 2023-07-29T00:10:39.739 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.739 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:39.739 DEBUG:boto:Port: 8000 2023-07-29T00:10:39.740 DEBUG:boto:Params: {} 2023-07-29T00:10:39.740 DEBUG:boto:Token: None 2023-07-29T00:10:39.741 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/obj2?versionId=null 2023-07-29T00:10:39.741 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KKNAdERks6AUM24Bui6wMVreAcg= 2023-07-29T00:10:39.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KKNAdERks6AUM24Bui6wMVreAcg=', 'Content-Length': '0'} 2023-07-29T00:10:39.743 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000646e0f015981d76-0064c458ff-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.744 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:10:39.745 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:10:39.745 DEBUG:boto:path=/hiiqos-49/?prefix=obj2 2023-07-29T00:10:39.746 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj2 2023-07-29T00:10:39.746 DEBUG:boto:Method: GET 2023-07-29T00:10:39.746 DEBUG:boto:Path: /hiiqos-49/?prefix=obj2 2023-07-29T00:10:39.747 DEBUG:boto:Data: 2023-07-29T00:10:39.747 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.748 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:39.748 DEBUG:boto:Port: 8001 2023-07-29T00:10:39.748 DEBUG:boto:Params: {} 2023-07-29T00:10:39.749 DEBUG:boto:Token: None 2023-07-29T00:10:39.750 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/ 2023-07-29T00:10:39.751 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LsG1Uend1b5x+h5hPwHfXmQpEAM= 2023-07-29T00:10:39.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LsG1Uend1b5x+h5hPwHfXmQpEAM=', 'Content-Length': '0'} 2023-07-29T00:10:39.755 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3e37630653ec7d2-0064c458ff-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.756 DEBUG:boto:b'hiiqos-49obj21000falseobj22023-07-29T00:08:34.191Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:10:39.757 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:10:39.758 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:10:39.758 DEBUG:boto:path=/hiiqos-49/?prefix=obj2 2023-07-29T00:10:39.759 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj2 2023-07-29T00:10:39.760 DEBUG:boto:Method: GET 2023-07-29T00:10:39.760 DEBUG:boto:Path: /hiiqos-49/?prefix=obj2 2023-07-29T00:10:39.761 DEBUG:boto:Data: 2023-07-29T00:10:39.761 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.761 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:39.762 DEBUG:boto:Port: 8000 2023-07-29T00:10:39.762 DEBUG:boto:Params: {} 2023-07-29T00:10:39.763 DEBUG:boto:Token: None 2023-07-29T00:10:39.763 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/ 2023-07-29T00:10:39.764 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LsG1Uend1b5x+h5hPwHfXmQpEAM= 2023-07-29T00:10:39.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LsG1Uend1b5x+h5hPwHfXmQpEAM=', 'Content-Length': '0'} 2023-07-29T00:10:39.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000415aa1c444fb5849-0064c458ff-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.767 DEBUG:boto:b'hiiqos-49obj21000falseobj22023-07-29T00:08:34.191Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:10:39.768 DEBUG:boto:path=/hiiqos-49/obj2 2023-07-29T00:10:39.768 DEBUG:boto:auth_path=/hiiqos-49/obj2 2023-07-29T00:10:39.768 DEBUG:boto:Method: HEAD 2023-07-29T00:10:39.769 DEBUG:boto:Path: /hiiqos-49/obj2 2023-07-29T00:10:39.769 DEBUG:boto:Data: 2023-07-29T00:10:39.770 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.770 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:39.770 DEBUG:boto:Port: 8001 2023-07-29T00:10:39.770 DEBUG:boto:Params: {} 2023-07-29T00:10:39.771 DEBUG:boto:Token: None 2023-07-29T00:10:39.771 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/obj2 2023-07-29T00:10:39.772 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:iACbASrBuZdNxtPy5QxyqpNsjx4= 2023-07-29T00:10:39.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:iACbASrBuZdNxtPy5QxyqpNsjx4=', 'Content-Length': '0'} 2023-07-29T00:10:39.774 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bdae2e6bf61d7ef8-0064c458ff-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.774 DEBUG:boto:path=/hiiqos-49/obj2 2023-07-29T00:10:39.775 DEBUG:boto:auth_path=/hiiqos-49/obj2 2023-07-29T00:10:39.775 DEBUG:boto:Method: HEAD 2023-07-29T00:10:39.775 DEBUG:boto:Path: /hiiqos-49/obj2 2023-07-29T00:10:39.776 DEBUG:boto:Data: 2023-07-29T00:10:39.776 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.776 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:39.777 DEBUG:boto:Port: 8000 2023-07-29T00:10:39.777 DEBUG:boto:Params: {} 2023-07-29T00:10:39.777 DEBUG:boto:Token: None 2023-07-29T00:10:39.778 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/obj2 2023-07-29T00:10:39.778 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:iACbASrBuZdNxtPy5QxyqpNsjx4= 2023-07-29T00:10:39.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:iACbASrBuZdNxtPy5QxyqpNsjx4=', 'Content-Length': '0'} 2023-07-29T00:10:39.780 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f57cd4a4d6e9b4a3-0064c458ff-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-07-29T00:10:39.781 DEBUG:boto:path=/hiiqos-49/obj2 2023-07-29T00:10:39.782 DEBUG:boto:auth_path=/hiiqos-49/obj2 2023-07-29T00:10:39.782 DEBUG:boto:Method: GET 2023-07-29T00:10:39.782 DEBUG:boto:Path: /hiiqos-49/obj2 2023-07-29T00:10:39.783 DEBUG:boto:Data: 2023-07-29T00:10:39.783 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.783 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:39.784 DEBUG:boto:Port: 8001 2023-07-29T00:10:39.784 DEBUG:boto:Params: {} 2023-07-29T00:10:39.785 DEBUG:boto:Token: None 2023-07-29T00:10:39.785 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/obj2 2023-07-29T00:10:39.785 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:39wkD893OzEWpzJB/IwyP50I64s= 2023-07-29T00:10:39.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:39wkD893OzEWpzJB/IwyP50I64s=', 'Content-Length': '0'} 2023-07-29T00:10:39.788 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e3c9d7f621f7eac-0064c458ff-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.789 DEBUG:boto:path=/hiiqos-49/obj2 2023-07-29T00:10:39.789 DEBUG:boto:auth_path=/hiiqos-49/obj2 2023-07-29T00:10:39.789 DEBUG:boto:Method: GET 2023-07-29T00:10:39.790 DEBUG:boto:Path: /hiiqos-49/obj2 2023-07-29T00:10:39.790 DEBUG:boto:Data: 2023-07-29T00:10:39.791 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.791 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:39.791 DEBUG:boto:Port: 8000 2023-07-29T00:10:39.792 DEBUG:boto:Params: {} 2023-07-29T00:10:39.792 DEBUG:boto:Token: None 2023-07-29T00:10:39.793 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/obj2 2023-07-29T00:10:39.793 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:39wkD893OzEWpzJB/IwyP50I64s= 2023-07-29T00:10:39.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:39wkD893OzEWpzJB/IwyP50I64s=', 'Content-Length': '0'} 2023-07-29T00:10:39.795 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008118baa478aa2f5d-0064c458ff-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.796 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:10:39.796 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:10:39.796 DEBUG:boto:path=/hiiqos-49/?prefix=obj2 2023-07-29T00:10:39.797 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj2 2023-07-29T00:10:39.797 DEBUG:boto:Method: GET 2023-07-29T00:10:39.798 DEBUG:boto:Path: /hiiqos-49/?prefix=obj2 2023-07-29T00:10:39.798 DEBUG:boto:Data: 2023-07-29T00:10:39.798 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.799 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:39.799 DEBUG:boto:Port: 8001 2023-07-29T00:10:39.799 DEBUG:boto:Params: {} 2023-07-29T00:10:39.800 DEBUG:boto:Token: None 2023-07-29T00:10:39.800 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/ 2023-07-29T00:10:39.801 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LsG1Uend1b5x+h5hPwHfXmQpEAM= 2023-07-29T00:10:39.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LsG1Uend1b5x+h5hPwHfXmQpEAM=', 'Content-Length': '0'} 2023-07-29T00:10:39.803 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005738d9e4cdf08b42-0064c458ff-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.804 DEBUG:boto:b'hiiqos-49obj21000falseobj22023-07-29T00:08:34.191Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:10:39.804 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:10:39.805 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:10:39.805 DEBUG:boto:path=/hiiqos-49/?prefix=obj2 2023-07-29T00:10:39.805 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj2 2023-07-29T00:10:39.806 DEBUG:boto:Method: GET 2023-07-29T00:10:39.806 DEBUG:boto:Path: /hiiqos-49/?prefix=obj2 2023-07-29T00:10:39.806 DEBUG:boto:Data: 2023-07-29T00:10:39.807 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.807 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:39.808 DEBUG:boto:Port: 8000 2023-07-29T00:10:39.808 DEBUG:boto:Params: {} 2023-07-29T00:10:39.808 DEBUG:boto:Token: None 2023-07-29T00:10:39.809 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/ 2023-07-29T00:10:39.809 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LsG1Uend1b5x+h5hPwHfXmQpEAM= 2023-07-29T00:10:39.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LsG1Uend1b5x+h5hPwHfXmQpEAM=', 'Content-Length': '0'} 2023-07-29T00:10:39.812 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000953ebaba3db1ce0a-0064c458ff-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.812 DEBUG:boto:b'hiiqos-49obj21000falseobj22023-07-29T00:08:34.191Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:10:39.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-07-29T00:10:39.813 DEBUG:boto:path=/hiiqos-49/obj3 2023-07-29T00:10:39.814 DEBUG:boto:auth_path=/hiiqos-49/obj3 2023-07-29T00:10:39.814 DEBUG:boto:path=/hiiqos-49/obj3?versionId=null 2023-07-29T00:10:39.814 DEBUG:boto:auth_path=/hiiqos-49/obj3?versionId=null 2023-07-29T00:10:39.815 DEBUG:boto:Method: GET 2023-07-29T00:10:39.815 DEBUG:boto:Path: /hiiqos-49/obj3?versionId=null 2023-07-29T00:10:39.815 DEBUG:boto:Data: 2023-07-29T00:10:39.816 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.816 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:39.816 DEBUG:boto:Port: 8001 2023-07-29T00:10:39.817 DEBUG:boto:Params: {} 2023-07-29T00:10:39.817 DEBUG:boto:Token: None 2023-07-29T00:10:39.818 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/obj3?versionId=null 2023-07-29T00:10:39.818 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PdHTPAkr5jB/r8XB0ydefJmH8tg= 2023-07-29T00:10:39.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PdHTPAkr5jB/r8XB0ydefJmH8tg=', 'Content-Length': '0'} 2023-07-29T00:10:39.820 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b6859cefa1f0b423-0064c458ff-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.821 DEBUG:boto:path=/hiiqos-49/obj3 2023-07-29T00:10:39.821 DEBUG:boto:auth_path=/hiiqos-49/obj3 2023-07-29T00:10:39.822 DEBUG:boto:path=/hiiqos-49/obj3?versionId=null 2023-07-29T00:10:39.822 DEBUG:boto:auth_path=/hiiqos-49/obj3?versionId=null 2023-07-29T00:10:39.822 DEBUG:boto:Method: GET 2023-07-29T00:10:39.823 DEBUG:boto:Path: /hiiqos-49/obj3?versionId=null 2023-07-29T00:10:39.823 DEBUG:boto:Data: 2023-07-29T00:10:39.823 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.824 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:39.824 DEBUG:boto:Port: 8000 2023-07-29T00:10:39.825 DEBUG:boto:Params: {} 2023-07-29T00:10:39.825 DEBUG:boto:Token: None 2023-07-29T00:10:39.825 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/obj3?versionId=null 2023-07-29T00:10:39.826 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PdHTPAkr5jB/r8XB0ydefJmH8tg= 2023-07-29T00:10:39.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PdHTPAkr5jB/r8XB0ydefJmH8tg=', 'Content-Length': '0'} 2023-07-29T00:10:39.828 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008646b3fbfe8c501e-0064c458ff-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.829 DEBUG:boto:path=/hiiqos-49/obj3 2023-07-29T00:10:39.829 DEBUG:boto:auth_path=/hiiqos-49/obj3 2023-07-29T00:10:39.830 DEBUG:boto:path=/hiiqos-49/obj3?versionId=null 2023-07-29T00:10:39.830 DEBUG:boto:auth_path=/hiiqos-49/obj3?versionId=null 2023-07-29T00:10:39.830 DEBUG:boto:Method: HEAD 2023-07-29T00:10:39.831 DEBUG:boto:Path: /hiiqos-49/obj3?versionId=null 2023-07-29T00:10:39.831 DEBUG:boto:Data: 2023-07-29T00:10:39.831 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.832 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:39.832 DEBUG:boto:Port: 8001 2023-07-29T00:10:39.832 DEBUG:boto:Params: {} 2023-07-29T00:10:39.833 DEBUG:boto:Token: None 2023-07-29T00:10:39.833 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/obj3?versionId=null 2023-07-29T00:10:39.834 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zuMkudHeN6LRN9UQuxR9tEmVIr0= 2023-07-29T00:10:39.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zuMkudHeN6LRN9UQuxR9tEmVIr0=', 'Content-Length': '0'} 2023-07-29T00:10:39.836 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cca74814c240b5d5-0064c458ff-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.836 DEBUG:boto:path=/hiiqos-49/obj3 2023-07-29T00:10:39.837 DEBUG:boto:auth_path=/hiiqos-49/obj3 2023-07-29T00:10:39.837 DEBUG:boto:path=/hiiqos-49/obj3?versionId=null 2023-07-29T00:10:39.837 DEBUG:boto:auth_path=/hiiqos-49/obj3?versionId=null 2023-07-29T00:10:39.838 DEBUG:boto:Method: HEAD 2023-07-29T00:10:39.838 DEBUG:boto:Path: /hiiqos-49/obj3?versionId=null 2023-07-29T00:10:39.838 DEBUG:boto:Data: 2023-07-29T00:10:39.839 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.839 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:39.839 DEBUG:boto:Port: 8000 2023-07-29T00:10:39.840 DEBUG:boto:Params: {} 2023-07-29T00:10:39.840 DEBUG:boto:Token: None 2023-07-29T00:10:39.841 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/obj3?versionId=null 2023-07-29T00:10:39.841 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zuMkudHeN6LRN9UQuxR9tEmVIr0= 2023-07-29T00:10:39.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zuMkudHeN6LRN9UQuxR9tEmVIr0=', 'Content-Length': '0'} 2023-07-29T00:10:39.843 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9b476b106e9add6-0064c458ff-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-07-29T00:10:39.844 DEBUG:boto:path=/hiiqos-49/obj3 2023-07-29T00:10:39.844 DEBUG:boto:auth_path=/hiiqos-49/obj3 2023-07-29T00:10:39.845 DEBUG:boto:path=/hiiqos-49/obj3?versionId=null 2023-07-29T00:10:39.845 DEBUG:boto:auth_path=/hiiqos-49/obj3?versionId=null 2023-07-29T00:10:39.846 DEBUG:boto:Method: GET 2023-07-29T00:10:39.846 DEBUG:boto:Path: /hiiqos-49/obj3?versionId=null 2023-07-29T00:10:39.846 DEBUG:boto:Data: 2023-07-29T00:10:39.847 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.847 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:39.847 DEBUG:boto:Port: 8001 2023-07-29T00:10:39.848 DEBUG:boto:Params: {} 2023-07-29T00:10:39.848 DEBUG:boto:Token: None 2023-07-29T00:10:39.849 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/obj3?versionId=null 2023-07-29T00:10:39.849 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PdHTPAkr5jB/r8XB0ydefJmH8tg= 2023-07-29T00:10:39.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PdHTPAkr5jB/r8XB0ydefJmH8tg=', 'Content-Length': '0'} 2023-07-29T00:10:39.851 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1882d3b00860df0-0064c458ff-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.893 DEBUG:boto:path=/hiiqos-49/obj3 2023-07-29T00:10:39.893 DEBUG:boto:auth_path=/hiiqos-49/obj3 2023-07-29T00:10:39.894 DEBUG:boto:path=/hiiqos-49/obj3?versionId=null 2023-07-29T00:10:39.894 DEBUG:boto:auth_path=/hiiqos-49/obj3?versionId=null 2023-07-29T00:10:39.894 DEBUG:boto:Method: GET 2023-07-29T00:10:39.895 DEBUG:boto:Path: /hiiqos-49/obj3?versionId=null 2023-07-29T00:10:39.895 DEBUG:boto:Data: 2023-07-29T00:10:39.895 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.896 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:39.896 DEBUG:boto:Port: 8000 2023-07-29T00:10:39.896 DEBUG:boto:Params: {} 2023-07-29T00:10:39.897 DEBUG:boto:Token: None 2023-07-29T00:10:39.897 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/obj3?versionId=null 2023-07-29T00:10:39.898 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PdHTPAkr5jB/r8XB0ydefJmH8tg= 2023-07-29T00:10:39.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PdHTPAkr5jB/r8XB0ydefJmH8tg=', 'Content-Length': '0'} 2023-07-29T00:10:39.900 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000079f88515edf27fb6-0064c458ff-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.901 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:10:39.901 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:10:39.902 DEBUG:boto:path=/hiiqos-49/?prefix=obj3 2023-07-29T00:10:39.902 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj3 2023-07-29T00:10:39.902 DEBUG:boto:Method: GET 2023-07-29T00:10:39.903 DEBUG:boto:Path: /hiiqos-49/?prefix=obj3 2023-07-29T00:10:39.903 DEBUG:boto:Data: 2023-07-29T00:10:39.903 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.904 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:39.904 DEBUG:boto:Port: 8001 2023-07-29T00:10:39.904 DEBUG:boto:Params: {} 2023-07-29T00:10:39.905 DEBUG:boto:Token: None 2023-07-29T00:10:39.905 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/ 2023-07-29T00:10:39.906 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LsG1Uend1b5x+h5hPwHfXmQpEAM= 2023-07-29T00:10:39.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LsG1Uend1b5x+h5hPwHfXmQpEAM=', 'Content-Length': '0'} 2023-07-29T00:10:39.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3f592598ee3d655-0064c458ff-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.909 DEBUG:boto:b'hiiqos-49obj31000falseobj32023-07-29T00:08:34.246Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:10:39.909 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:10:39.910 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:10:39.910 DEBUG:boto:path=/hiiqos-49/?prefix=obj3 2023-07-29T00:10:39.911 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj3 2023-07-29T00:10:39.911 DEBUG:boto:Method: GET 2023-07-29T00:10:39.911 DEBUG:boto:Path: /hiiqos-49/?prefix=obj3 2023-07-29T00:10:39.912 DEBUG:boto:Data: 2023-07-29T00:10:39.912 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.912 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:39.913 DEBUG:boto:Port: 8000 2023-07-29T00:10:39.913 DEBUG:boto:Params: {} 2023-07-29T00:10:39.914 DEBUG:boto:Token: None 2023-07-29T00:10:39.914 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/ 2023-07-29T00:10:39.914 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LsG1Uend1b5x+h5hPwHfXmQpEAM= 2023-07-29T00:10:39.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LsG1Uend1b5x+h5hPwHfXmQpEAM=', 'Content-Length': '0'} 2023-07-29T00:10:39.917 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026f39a55ea654a1c-0064c458ff-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.918 DEBUG:boto:b'hiiqos-49obj31000falseobj32023-07-29T00:08:34.246Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:10:39.918 DEBUG:boto:path=/hiiqos-49/obj3 2023-07-29T00:10:39.919 DEBUG:boto:auth_path=/hiiqos-49/obj3 2023-07-29T00:10:39.919 DEBUG:boto:Method: HEAD 2023-07-29T00:10:39.919 DEBUG:boto:Path: /hiiqos-49/obj3 2023-07-29T00:10:39.920 DEBUG:boto:Data: 2023-07-29T00:10:39.920 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.920 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:39.921 DEBUG:boto:Port: 8001 2023-07-29T00:10:39.921 DEBUG:boto:Params: {} 2023-07-29T00:10:39.921 DEBUG:boto:Token: None 2023-07-29T00:10:39.922 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/obj3 2023-07-29T00:10:39.922 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P+xIqKKdY2iso/ix9Cj6qGFiMco= 2023-07-29T00:10:39.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P+xIqKKdY2iso/ix9Cj6qGFiMco=', 'Content-Length': '0'} 2023-07-29T00:10:39.924 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e9f5671e34bb403-0064c458ff-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.925 DEBUG:boto:path=/hiiqos-49/obj3 2023-07-29T00:10:39.925 DEBUG:boto:auth_path=/hiiqos-49/obj3 2023-07-29T00:10:39.925 DEBUG:boto:Method: HEAD 2023-07-29T00:10:39.926 DEBUG:boto:Path: /hiiqos-49/obj3 2023-07-29T00:10:39.926 DEBUG:boto:Data: 2023-07-29T00:10:39.926 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.927 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:39.927 DEBUG:boto:Port: 8000 2023-07-29T00:10:39.927 DEBUG:boto:Params: {} 2023-07-29T00:10:39.928 DEBUG:boto:Token: None 2023-07-29T00:10:39.928 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/obj3 2023-07-29T00:10:39.929 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P+xIqKKdY2iso/ix9Cj6qGFiMco= 2023-07-29T00:10:39.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P+xIqKKdY2iso/ix9Cj6qGFiMco=', 'Content-Length': '0'} 2023-07-29T00:10:39.931 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001cb46a041e6751e5-0064c458ff-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-07-29T00:10:39.932 DEBUG:boto:path=/hiiqos-49/obj3 2023-07-29T00:10:39.932 DEBUG:boto:auth_path=/hiiqos-49/obj3 2023-07-29T00:10:39.933 DEBUG:boto:Method: GET 2023-07-29T00:10:39.933 DEBUG:boto:Path: /hiiqos-49/obj3 2023-07-29T00:10:39.933 DEBUG:boto:Data: 2023-07-29T00:10:39.934 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.934 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:39.934 DEBUG:boto:Port: 8001 2023-07-29T00:10:39.935 DEBUG:boto:Params: {} 2023-07-29T00:10:39.935 DEBUG:boto:Token: None 2023-07-29T00:10:39.936 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/obj3 2023-07-29T00:10:39.936 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LuZW/brpwu+ppPReuOQ210FnQ1w= 2023-07-29T00:10:39.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LuZW/brpwu+ppPReuOQ210FnQ1w=', 'Content-Length': '0'} 2023-07-29T00:10:39.938 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007c724dc376d74ce7-0064c458ff-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.980 DEBUG:boto:path=/hiiqos-49/obj3 2023-07-29T00:10:39.981 DEBUG:boto:auth_path=/hiiqos-49/obj3 2023-07-29T00:10:39.981 DEBUG:boto:Method: GET 2023-07-29T00:10:39.982 DEBUG:boto:Path: /hiiqos-49/obj3 2023-07-29T00:10:39.982 DEBUG:boto:Data: 2023-07-29T00:10:39.982 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.983 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:39.983 DEBUG:boto:Port: 8000 2023-07-29T00:10:39.983 DEBUG:boto:Params: {} 2023-07-29T00:10:39.984 DEBUG:boto:Token: None 2023-07-29T00:10:39.984 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/obj3 2023-07-29T00:10:39.985 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LuZW/brpwu+ppPReuOQ210FnQ1w= 2023-07-29T00:10:39.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LuZW/brpwu+ppPReuOQ210FnQ1w=', 'Content-Length': '0'} 2023-07-29T00:10:39.987 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f0ab599f408bb31-0064c458ff-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.987 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:10:39.988 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:10:39.988 DEBUG:boto:path=/hiiqos-49/?prefix=obj3 2023-07-29T00:10:39.989 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj3 2023-07-29T00:10:39.989 DEBUG:boto:Method: GET 2023-07-29T00:10:39.989 DEBUG:boto:Path: /hiiqos-49/?prefix=obj3 2023-07-29T00:10:39.990 DEBUG:boto:Data: 2023-07-29T00:10:39.990 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.990 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:39.991 DEBUG:boto:Port: 8001 2023-07-29T00:10:39.991 DEBUG:boto:Params: {} 2023-07-29T00:10:39.991 DEBUG:boto:Token: None 2023-07-29T00:10:39.992 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:39 GMT /hiiqos-49/ 2023-07-29T00:10:39.992 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LsG1Uend1b5x+h5hPwHfXmQpEAM= 2023-07-29T00:10:39.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LsG1Uend1b5x+h5hPwHfXmQpEAM=', 'Content-Length': '0'} 2023-07-29T00:10:39.995 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005199682816a5eddb-0064c458ff-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:39.995 DEBUG:boto:b'hiiqos-49obj31000falseobj32023-07-29T00:08:34.246Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:10:39.996 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:10:39.996 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:10:39.997 DEBUG:boto:path=/hiiqos-49/?prefix=obj3 2023-07-29T00:10:39.997 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj3 2023-07-29T00:10:39.997 DEBUG:boto:Method: GET 2023-07-29T00:10:39.998 DEBUG:boto:Path: /hiiqos-49/?prefix=obj3 2023-07-29T00:10:39.998 DEBUG:boto:Data: 2023-07-29T00:10:39.998 DEBUG:boto:Headers: {} 2023-07-29T00:10:39.999 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:39.999 DEBUG:boto:Port: 8000 2023-07-29T00:10:39.999 DEBUG:boto:Params: {} 2023-07-29T00:10:40.000 DEBUG:boto:Token: None 2023-07-29T00:10:40.000 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/ 2023-07-29T00:10:40.001 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UQ1dXMReSrCpLgdbHZMnK072e8Y= 2023-07-29T00:10:40.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UQ1dXMReSrCpLgdbHZMnK072e8Y=', 'Content-Length': '0'} 2023-07-29T00:10:40.003 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000332cef1fc95b370a-0064c458ff-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.004 DEBUG:boto:b'hiiqos-49obj31000falseobj32023-07-29T00:08:34.246Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:10:40.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-07-29T00:10:40.005 DEBUG:boto:path=/hiiqos-49/obj4 2023-07-29T00:10:40.005 DEBUG:boto:auth_path=/hiiqos-49/obj4 2023-07-29T00:10:40.005 DEBUG:boto:path=/hiiqos-49/obj4?versionId=null 2023-07-29T00:10:40.006 DEBUG:boto:auth_path=/hiiqos-49/obj4?versionId=null 2023-07-29T00:10:40.006 DEBUG:boto:Method: GET 2023-07-29T00:10:40.006 DEBUG:boto:Path: /hiiqos-49/obj4?versionId=null 2023-07-29T00:10:40.007 DEBUG:boto:Data: 2023-07-29T00:10:40.007 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.007 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:40.008 DEBUG:boto:Port: 8001 2023-07-29T00:10:40.008 DEBUG:boto:Params: {} 2023-07-29T00:10:40.009 DEBUG:boto:Token: None 2023-07-29T00:10:40.009 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj4?versionId=null 2023-07-29T00:10:40.009 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fhg0jh6zdx2qDq+FswqVJOj8bTM= 2023-07-29T00:10:40.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fhg0jh6zdx2qDq+FswqVJOj8bTM=', 'Content-Length': '0'} 2023-07-29T00:10:40.012 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005e3cc8705824e947-0064c458ff-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.012 DEBUG:boto:path=/hiiqos-49/obj4 2023-07-29T00:10:40.013 DEBUG:boto:auth_path=/hiiqos-49/obj4 2023-07-29T00:10:40.013 DEBUG:boto:path=/hiiqos-49/obj4?versionId=null 2023-07-29T00:10:40.013 DEBUG:boto:auth_path=/hiiqos-49/obj4?versionId=null 2023-07-29T00:10:40.014 DEBUG:boto:Method: GET 2023-07-29T00:10:40.014 DEBUG:boto:Path: /hiiqos-49/obj4?versionId=null 2023-07-29T00:10:40.014 DEBUG:boto:Data: 2023-07-29T00:10:40.015 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.015 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:40.015 DEBUG:boto:Port: 8000 2023-07-29T00:10:40.016 DEBUG:boto:Params: {} 2023-07-29T00:10:40.016 DEBUG:boto:Token: None 2023-07-29T00:10:40.017 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj4?versionId=null 2023-07-29T00:10:40.017 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fhg0jh6zdx2qDq+FswqVJOj8bTM= 2023-07-29T00:10:40.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fhg0jh6zdx2qDq+FswqVJOj8bTM=', 'Content-Length': '0'} 2023-07-29T00:10:40.019 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000844975b7d47b40c6-0064c458ff-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.020 DEBUG:boto:path=/hiiqos-49/obj4 2023-07-29T00:10:40.020 DEBUG:boto:auth_path=/hiiqos-49/obj4 2023-07-29T00:10:40.020 DEBUG:boto:path=/hiiqos-49/obj4?versionId=null 2023-07-29T00:10:40.021 DEBUG:boto:auth_path=/hiiqos-49/obj4?versionId=null 2023-07-29T00:10:40.021 DEBUG:boto:Method: HEAD 2023-07-29T00:10:40.021 DEBUG:boto:Path: /hiiqos-49/obj4?versionId=null 2023-07-29T00:10:40.022 DEBUG:boto:Data: 2023-07-29T00:10:40.022 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.022 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:40.023 DEBUG:boto:Port: 8001 2023-07-29T00:10:40.023 DEBUG:boto:Params: {} 2023-07-29T00:10:40.024 DEBUG:boto:Token: None 2023-07-29T00:10:40.024 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj4?versionId=null 2023-07-29T00:10:40.024 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9g+15kIU6e7BjN0yhOMWGsHIowo= 2023-07-29T00:10:40.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9g+15kIU6e7BjN0yhOMWGsHIowo=', 'Content-Length': '0'} 2023-07-29T00:10:40.026 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000518bd512a0756916-0064c458ff-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.027 DEBUG:boto:path=/hiiqos-49/obj4 2023-07-29T00:10:40.027 DEBUG:boto:auth_path=/hiiqos-49/obj4 2023-07-29T00:10:40.028 DEBUG:boto:path=/hiiqos-49/obj4?versionId=null 2023-07-29T00:10:40.028 DEBUG:boto:auth_path=/hiiqos-49/obj4?versionId=null 2023-07-29T00:10:40.028 DEBUG:boto:Method: HEAD 2023-07-29T00:10:40.029 DEBUG:boto:Path: /hiiqos-49/obj4?versionId=null 2023-07-29T00:10:40.029 DEBUG:boto:Data: 2023-07-29T00:10:40.029 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.030 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:40.030 DEBUG:boto:Port: 8000 2023-07-29T00:10:40.030 DEBUG:boto:Params: {} 2023-07-29T00:10:40.031 DEBUG:boto:Token: None 2023-07-29T00:10:40.031 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj4?versionId=null 2023-07-29T00:10:40.032 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9g+15kIU6e7BjN0yhOMWGsHIowo= 2023-07-29T00:10:40.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9g+15kIU6e7BjN0yhOMWGsHIowo=', 'Content-Length': '0'} 2023-07-29T00:10:40.034 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000953e454d81791587-0064c45900-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-07-29T00:10:40.035 DEBUG:boto:path=/hiiqos-49/obj4 2023-07-29T00:10:40.035 DEBUG:boto:auth_path=/hiiqos-49/obj4 2023-07-29T00:10:40.035 DEBUG:boto:path=/hiiqos-49/obj4?versionId=null 2023-07-29T00:10:40.036 DEBUG:boto:auth_path=/hiiqos-49/obj4?versionId=null 2023-07-29T00:10:40.036 DEBUG:boto:Method: GET 2023-07-29T00:10:40.036 DEBUG:boto:Path: /hiiqos-49/obj4?versionId=null 2023-07-29T00:10:40.037 DEBUG:boto:Data: 2023-07-29T00:10:40.037 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.037 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:40.038 DEBUG:boto:Port: 8001 2023-07-29T00:10:40.038 DEBUG:boto:Params: {} 2023-07-29T00:10:40.039 DEBUG:boto:Token: None 2023-07-29T00:10:40.039 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj4?versionId=null 2023-07-29T00:10:40.039 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fhg0jh6zdx2qDq+FswqVJOj8bTM= 2023-07-29T00:10:40.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fhg0jh6zdx2qDq+FswqVJOj8bTM=', 'Content-Length': '0'} 2023-07-29T00:10:40.042 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000faeb4ca97042b902-0064c45900-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.042 DEBUG:boto:path=/hiiqos-49/obj4 2023-07-29T00:10:40.043 DEBUG:boto:auth_path=/hiiqos-49/obj4 2023-07-29T00:10:40.043 DEBUG:boto:path=/hiiqos-49/obj4?versionId=null 2023-07-29T00:10:40.043 DEBUG:boto:auth_path=/hiiqos-49/obj4?versionId=null 2023-07-29T00:10:40.044 DEBUG:boto:Method: GET 2023-07-29T00:10:40.045 DEBUG:boto:Path: /hiiqos-49/obj4?versionId=null 2023-07-29T00:10:40.045 DEBUG:boto:Data: 2023-07-29T00:10:40.045 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.046 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:40.046 DEBUG:boto:Port: 8000 2023-07-29T00:10:40.046 DEBUG:boto:Params: {} 2023-07-29T00:10:40.047 DEBUG:boto:Token: None 2023-07-29T00:10:40.047 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj4?versionId=null 2023-07-29T00:10:40.048 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fhg0jh6zdx2qDq+FswqVJOj8bTM= 2023-07-29T00:10:40.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fhg0jh6zdx2qDq+FswqVJOj8bTM=', 'Content-Length': '0'} 2023-07-29T00:10:40.050 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a78b10d317d18f09-0064c45900-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.051 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:10:40.051 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:10:40.051 DEBUG:boto:path=/hiiqos-49/?prefix=obj4 2023-07-29T00:10:40.052 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj4 2023-07-29T00:10:40.052 DEBUG:boto:Method: GET 2023-07-29T00:10:40.053 DEBUG:boto:Path: /hiiqos-49/?prefix=obj4 2023-07-29T00:10:40.053 DEBUG:boto:Data: 2023-07-29T00:10:40.053 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.054 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:40.054 DEBUG:boto:Port: 8001 2023-07-29T00:10:40.054 DEBUG:boto:Params: {} 2023-07-29T00:10:40.055 DEBUG:boto:Token: None 2023-07-29T00:10:40.055 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/ 2023-07-29T00:10:40.056 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UQ1dXMReSrCpLgdbHZMnK072e8Y= 2023-07-29T00:10:40.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UQ1dXMReSrCpLgdbHZMnK072e8Y=', 'Content-Length': '0'} 2023-07-29T00:10:40.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012099e4f22ca83fc-0064c45900-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.059 DEBUG:boto:b'hiiqos-49obj41000falseobj42023-07-29T00:08:34.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:10:40.059 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:10:40.060 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:10:40.060 DEBUG:boto:path=/hiiqos-49/?prefix=obj4 2023-07-29T00:10:40.061 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj4 2023-07-29T00:10:40.061 DEBUG:boto:Method: GET 2023-07-29T00:10:40.061 DEBUG:boto:Path: /hiiqos-49/?prefix=obj4 2023-07-29T00:10:40.062 DEBUG:boto:Data: 2023-07-29T00:10:40.062 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.062 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:40.063 DEBUG:boto:Port: 8000 2023-07-29T00:10:40.063 DEBUG:boto:Params: {} 2023-07-29T00:10:40.064 DEBUG:boto:Token: None 2023-07-29T00:10:40.064 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/ 2023-07-29T00:10:40.064 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UQ1dXMReSrCpLgdbHZMnK072e8Y= 2023-07-29T00:10:40.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UQ1dXMReSrCpLgdbHZMnK072e8Y=', 'Content-Length': '0'} 2023-07-29T00:10:40.067 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b97e0f7f206b01c-0064c45900-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.067 DEBUG:boto:b'hiiqos-49obj41000falseobj42023-07-29T00:08:34.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:10:40.068 DEBUG:boto:path=/hiiqos-49/obj4 2023-07-29T00:10:40.068 DEBUG:boto:auth_path=/hiiqos-49/obj4 2023-07-29T00:10:40.069 DEBUG:boto:Method: HEAD 2023-07-29T00:10:40.069 DEBUG:boto:Path: /hiiqos-49/obj4 2023-07-29T00:10:40.069 DEBUG:boto:Data: 2023-07-29T00:10:40.070 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.070 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:40.070 DEBUG:boto:Port: 8001 2023-07-29T00:10:40.071 DEBUG:boto:Params: {} 2023-07-29T00:10:40.071 DEBUG:boto:Token: None 2023-07-29T00:10:40.072 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj4 2023-07-29T00:10:40.072 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DSFEJd0NT7UfEHZ7rutGXF4HJoQ= 2023-07-29T00:10:40.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DSFEJd0NT7UfEHZ7rutGXF4HJoQ=', 'Content-Length': '0'} 2023-07-29T00:10:40.074 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c48a9addbd53769d-0064c45900-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.075 DEBUG:boto:path=/hiiqos-49/obj4 2023-07-29T00:10:40.075 DEBUG:boto:auth_path=/hiiqos-49/obj4 2023-07-29T00:10:40.075 DEBUG:boto:Method: HEAD 2023-07-29T00:10:40.076 DEBUG:boto:Path: /hiiqos-49/obj4 2023-07-29T00:10:40.076 DEBUG:boto:Data: 2023-07-29T00:10:40.076 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.077 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:40.077 DEBUG:boto:Port: 8000 2023-07-29T00:10:40.077 DEBUG:boto:Params: {} 2023-07-29T00:10:40.078 DEBUG:boto:Token: None 2023-07-29T00:10:40.078 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj4 2023-07-29T00:10:40.079 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DSFEJd0NT7UfEHZ7rutGXF4HJoQ= 2023-07-29T00:10:40.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DSFEJd0NT7UfEHZ7rutGXF4HJoQ=', 'Content-Length': '0'} 2023-07-29T00:10:40.081 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b31a51e9bd89308b-0064c45900-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-07-29T00:10:40.082 DEBUG:boto:path=/hiiqos-49/obj4 2023-07-29T00:10:40.082 DEBUG:boto:auth_path=/hiiqos-49/obj4 2023-07-29T00:10:40.082 DEBUG:boto:Method: GET 2023-07-29T00:10:40.083 DEBUG:boto:Path: /hiiqos-49/obj4 2023-07-29T00:10:40.083 DEBUG:boto:Data: 2023-07-29T00:10:40.084 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.084 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:40.084 DEBUG:boto:Port: 8001 2023-07-29T00:10:40.085 DEBUG:boto:Params: {} 2023-07-29T00:10:40.085 DEBUG:boto:Token: None 2023-07-29T00:10:40.085 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj4 2023-07-29T00:10:40.086 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OVWX5Z2QTRZRm8g2tYcz/hKXo2g= 2023-07-29T00:10:40.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OVWX5Z2QTRZRm8g2tYcz/hKXo2g=', 'Content-Length': '0'} 2023-07-29T00:10:40.088 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000059a3f0404d55d6b0-0064c45900-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.089 DEBUG:boto:path=/hiiqos-49/obj4 2023-07-29T00:10:40.089 DEBUG:boto:auth_path=/hiiqos-49/obj4 2023-07-29T00:10:40.090 DEBUG:boto:Method: GET 2023-07-29T00:10:40.090 DEBUG:boto:Path: /hiiqos-49/obj4 2023-07-29T00:10:40.090 DEBUG:boto:Data: 2023-07-29T00:10:40.091 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.091 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:40.092 DEBUG:boto:Port: 8000 2023-07-29T00:10:40.092 DEBUG:boto:Params: {} 2023-07-29T00:10:40.092 DEBUG:boto:Token: None 2023-07-29T00:10:40.093 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj4 2023-07-29T00:10:40.093 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OVWX5Z2QTRZRm8g2tYcz/hKXo2g= 2023-07-29T00:10:40.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OVWX5Z2QTRZRm8g2tYcz/hKXo2g=', 'Content-Length': '0'} 2023-07-29T00:10:40.096 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:08:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f6fae95dc9f65ea5-0064c45900-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.140 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:10:40.141 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:10:40.141 DEBUG:boto:path=/hiiqos-49/?prefix=obj4 2023-07-29T00:10:40.141 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj4 2023-07-29T00:10:40.142 DEBUG:boto:Method: GET 2023-07-29T00:10:40.143 DEBUG:boto:Path: /hiiqos-49/?prefix=obj4 2023-07-29T00:10:40.143 DEBUG:boto:Data: 2023-07-29T00:10:40.143 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.144 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:40.144 DEBUG:boto:Port: 8001 2023-07-29T00:10:40.144 DEBUG:boto:Params: {} 2023-07-29T00:10:40.145 DEBUG:boto:Token: None 2023-07-29T00:10:40.145 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/ 2023-07-29T00:10:40.146 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UQ1dXMReSrCpLgdbHZMnK072e8Y= 2023-07-29T00:10:40.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UQ1dXMReSrCpLgdbHZMnK072e8Y=', 'Content-Length': '0'} 2023-07-29T00:10:40.148 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db6dbb2a14824b5a-0064c45900-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.149 DEBUG:boto:b'hiiqos-49obj41000falseobj42023-07-29T00:08:34.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:10:40.149 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:10:40.150 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:10:40.150 DEBUG:boto:path=/hiiqos-49/?prefix=obj4 2023-07-29T00:10:40.150 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj4 2023-07-29T00:10:40.151 DEBUG:boto:Method: GET 2023-07-29T00:10:40.151 DEBUG:boto:Path: /hiiqos-49/?prefix=obj4 2023-07-29T00:10:40.151 DEBUG:boto:Data: 2023-07-29T00:10:40.152 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.152 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:40.153 DEBUG:boto:Port: 8000 2023-07-29T00:10:40.153 DEBUG:boto:Params: {} 2023-07-29T00:10:40.153 DEBUG:boto:Token: None 2023-07-29T00:10:40.154 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/ 2023-07-29T00:10:40.154 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UQ1dXMReSrCpLgdbHZMnK072e8Y= 2023-07-29T00:10:40.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UQ1dXMReSrCpLgdbHZMnK072e8Y=', 'Content-Length': '0'} 2023-07-29T00:10:40.157 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cc480cfdeb80b48-0064c45900-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.157 DEBUG:boto:b'hiiqos-49obj41000falseobj42023-07-29T00:08:34.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:10:40.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-07-29T00:10:40.158 DEBUG:boto:path=/hiiqos-49/obj5 2023-07-29T00:10:40.158 DEBUG:boto:auth_path=/hiiqos-49/obj5 2023-07-29T00:10:40.159 DEBUG:boto:path=/hiiqos-49/obj5?versionId=null 2023-07-29T00:10:40.159 DEBUG:boto:auth_path=/hiiqos-49/obj5?versionId=null 2023-07-29T00:10:40.160 DEBUG:boto:Method: GET 2023-07-29T00:10:40.160 DEBUG:boto:Path: /hiiqos-49/obj5?versionId=null 2023-07-29T00:10:40.160 DEBUG:boto:Data: 2023-07-29T00:10:40.161 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.161 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:40.161 DEBUG:boto:Port: 8001 2023-07-29T00:10:40.162 DEBUG:boto:Params: {} 2023-07-29T00:10:40.162 DEBUG:boto:Token: None 2023-07-29T00:10:40.162 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj5?versionId=null 2023-07-29T00:10:40.163 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4zXSlg9SpaNVJr+WBD4VNNfaIzk= 2023-07-29T00:10:40.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4zXSlg9SpaNVJr+WBD4VNNfaIzk=', 'Content-Length': '0'} 2023-07-29T00:10:40.165 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d2da4450a4105989-0064c45900-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.166 DEBUG:boto:path=/hiiqos-49/obj5 2023-07-29T00:10:40.166 DEBUG:boto:auth_path=/hiiqos-49/obj5 2023-07-29T00:10:40.166 DEBUG:boto:path=/hiiqos-49/obj5?versionId=null 2023-07-29T00:10:40.167 DEBUG:boto:auth_path=/hiiqos-49/obj5?versionId=null 2023-07-29T00:10:40.167 DEBUG:boto:Method: GET 2023-07-29T00:10:40.167 DEBUG:boto:Path: /hiiqos-49/obj5?versionId=null 2023-07-29T00:10:40.168 DEBUG:boto:Data: 2023-07-29T00:10:40.168 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.169 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:40.169 DEBUG:boto:Port: 8000 2023-07-29T00:10:40.169 DEBUG:boto:Params: {} 2023-07-29T00:10:40.170 DEBUG:boto:Token: None 2023-07-29T00:10:40.170 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj5?versionId=null 2023-07-29T00:10:40.170 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4zXSlg9SpaNVJr+WBD4VNNfaIzk= 2023-07-29T00:10:40.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4zXSlg9SpaNVJr+WBD4VNNfaIzk=', 'Content-Length': '0'} 2023-07-29T00:10:40.173 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000071a77140095c3952-0064c45900-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.173 DEBUG:boto:path=/hiiqos-49/obj5 2023-07-29T00:10:40.174 DEBUG:boto:auth_path=/hiiqos-49/obj5 2023-07-29T00:10:40.174 DEBUG:boto:path=/hiiqos-49/obj5?versionId=null 2023-07-29T00:10:40.174 DEBUG:boto:auth_path=/hiiqos-49/obj5?versionId=null 2023-07-29T00:10:40.175 DEBUG:boto:Method: HEAD 2023-07-29T00:10:40.175 DEBUG:boto:Path: /hiiqos-49/obj5?versionId=null 2023-07-29T00:10:40.175 DEBUG:boto:Data: 2023-07-29T00:10:40.176 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.176 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:40.176 DEBUG:boto:Port: 8001 2023-07-29T00:10:40.177 DEBUG:boto:Params: {} 2023-07-29T00:10:40.177 DEBUG:boto:Token: None 2023-07-29T00:10:40.177 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj5?versionId=null 2023-07-29T00:10:40.178 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kEBNWcjvEg3v2NDLyNKqWBPXXWc= 2023-07-29T00:10:40.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kEBNWcjvEg3v2NDLyNKqWBPXXWc=', 'Content-Length': '0'} 2023-07-29T00:10:40.179 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000061bfe709f4b451c0-0064c45900-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.180 DEBUG:boto:path=/hiiqos-49/obj5 2023-07-29T00:10:40.180 DEBUG:boto:auth_path=/hiiqos-49/obj5 2023-07-29T00:10:40.181 DEBUG:boto:path=/hiiqos-49/obj5?versionId=null 2023-07-29T00:10:40.181 DEBUG:boto:auth_path=/hiiqos-49/obj5?versionId=null 2023-07-29T00:10:40.181 DEBUG:boto:Method: HEAD 2023-07-29T00:10:40.182 DEBUG:boto:Path: /hiiqos-49/obj5?versionId=null 2023-07-29T00:10:40.182 DEBUG:boto:Data: 2023-07-29T00:10:40.182 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.183 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:40.183 DEBUG:boto:Port: 8000 2023-07-29T00:10:40.183 DEBUG:boto:Params: {} 2023-07-29T00:10:40.184 DEBUG:boto:Token: None 2023-07-29T00:10:40.184 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj5?versionId=null 2023-07-29T00:10:40.185 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kEBNWcjvEg3v2NDLyNKqWBPXXWc= 2023-07-29T00:10:40.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kEBNWcjvEg3v2NDLyNKqWBPXXWc=', 'Content-Length': '0'} 2023-07-29T00:10:40.187 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f3bf8bab79daee3b-0064c45900-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-07-29T00:10:40.188 DEBUG:boto:path=/hiiqos-49/obj5 2023-07-29T00:10:40.188 DEBUG:boto:auth_path=/hiiqos-49/obj5 2023-07-29T00:10:40.188 DEBUG:boto:path=/hiiqos-49/obj5?versionId=null 2023-07-29T00:10:40.189 DEBUG:boto:auth_path=/hiiqos-49/obj5?versionId=null 2023-07-29T00:10:40.189 DEBUG:boto:Method: GET 2023-07-29T00:10:40.189 DEBUG:boto:Path: /hiiqos-49/obj5?versionId=null 2023-07-29T00:10:40.190 DEBUG:boto:Data: 2023-07-29T00:10:40.190 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.190 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:40.191 DEBUG:boto:Port: 8001 2023-07-29T00:10:40.191 DEBUG:boto:Params: {} 2023-07-29T00:10:40.192 DEBUG:boto:Token: None 2023-07-29T00:10:40.192 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj5?versionId=null 2023-07-29T00:10:40.192 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4zXSlg9SpaNVJr+WBD4VNNfaIzk= 2023-07-29T00:10:40.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4zXSlg9SpaNVJr+WBD4VNNfaIzk=', 'Content-Length': '0'} 2023-07-29T00:10:40.194 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f331f47fdc91f897-0064c45900-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.237 DEBUG:boto:path=/hiiqos-49/obj5 2023-07-29T00:10:40.237 DEBUG:boto:auth_path=/hiiqos-49/obj5 2023-07-29T00:10:40.238 DEBUG:boto:path=/hiiqos-49/obj5?versionId=null 2023-07-29T00:10:40.238 DEBUG:boto:auth_path=/hiiqos-49/obj5?versionId=null 2023-07-29T00:10:40.238 DEBUG:boto:Method: GET 2023-07-29T00:10:40.239 DEBUG:boto:Path: /hiiqos-49/obj5?versionId=null 2023-07-29T00:10:40.239 DEBUG:boto:Data: 2023-07-29T00:10:40.240 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.240 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:40.241 DEBUG:boto:Port: 8000 2023-07-29T00:10:40.241 DEBUG:boto:Params: {} 2023-07-29T00:10:40.241 DEBUG:boto:Token: None 2023-07-29T00:10:40.242 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj5?versionId=null 2023-07-29T00:10:40.242 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4zXSlg9SpaNVJr+WBD4VNNfaIzk= 2023-07-29T00:10:40.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4zXSlg9SpaNVJr+WBD4VNNfaIzk=', 'Content-Length': '0'} 2023-07-29T00:10:40.245 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d71b252d11e23323-0064c45900-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.245 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:10:40.246 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:10:40.246 DEBUG:boto:path=/hiiqos-49/?prefix=obj5 2023-07-29T00:10:40.247 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj5 2023-07-29T00:10:40.247 DEBUG:boto:Method: GET 2023-07-29T00:10:40.247 DEBUG:boto:Path: /hiiqos-49/?prefix=obj5 2023-07-29T00:10:40.248 DEBUG:boto:Data: 2023-07-29T00:10:40.248 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.249 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:40.249 DEBUG:boto:Port: 8001 2023-07-29T00:10:40.249 DEBUG:boto:Params: {} 2023-07-29T00:10:40.250 DEBUG:boto:Token: None 2023-07-29T00:10:40.250 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/ 2023-07-29T00:10:40.251 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UQ1dXMReSrCpLgdbHZMnK072e8Y= 2023-07-29T00:10:40.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UQ1dXMReSrCpLgdbHZMnK072e8Y=', 'Content-Length': '0'} 2023-07-29T00:10:40.254 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6f71d75378521dd-0064c45900-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.254 DEBUG:boto:b'hiiqos-49obj51000falseobj52023-07-29T00:09:22.313Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:10:40.255 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:10:40.255 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:10:40.256 DEBUG:boto:path=/hiiqos-49/?prefix=obj5 2023-07-29T00:10:40.256 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj5 2023-07-29T00:10:40.257 DEBUG:boto:Method: GET 2023-07-29T00:10:40.257 DEBUG:boto:Path: /hiiqos-49/?prefix=obj5 2023-07-29T00:10:40.257 DEBUG:boto:Data: 2023-07-29T00:10:40.258 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.258 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:40.259 DEBUG:boto:Port: 8000 2023-07-29T00:10:40.259 DEBUG:boto:Params: {} 2023-07-29T00:10:40.259 DEBUG:boto:Token: None 2023-07-29T00:10:40.260 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/ 2023-07-29T00:10:40.260 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UQ1dXMReSrCpLgdbHZMnK072e8Y= 2023-07-29T00:10:40.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UQ1dXMReSrCpLgdbHZMnK072e8Y=', 'Content-Length': '0'} 2023-07-29T00:10:40.263 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fa98e628ae04428-0064c45900-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.264 DEBUG:boto:b'hiiqos-49obj51000falseobj52023-07-29T00:09:22.313Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:10:40.265 DEBUG:boto:path=/hiiqos-49/obj5 2023-07-29T00:10:40.265 DEBUG:boto:auth_path=/hiiqos-49/obj5 2023-07-29T00:10:40.266 DEBUG:boto:Method: HEAD 2023-07-29T00:10:40.266 DEBUG:boto:Path: /hiiqos-49/obj5 2023-07-29T00:10:40.267 DEBUG:boto:Data: 2023-07-29T00:10:40.267 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.267 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:40.268 DEBUG:boto:Port: 8001 2023-07-29T00:10:40.268 DEBUG:boto:Params: {} 2023-07-29T00:10:40.269 DEBUG:boto:Token: None 2023-07-29T00:10:40.269 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj5 2023-07-29T00:10:40.270 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cn40O5WOglecpbJys4NoNwRKMhk= 2023-07-29T00:10:40.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cn40O5WOglecpbJys4NoNwRKMhk=', 'Content-Length': '0'} 2023-07-29T00:10:40.272 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a88bc894400e70da-0064c45900-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.273 DEBUG:boto:path=/hiiqos-49/obj5 2023-07-29T00:10:40.274 DEBUG:boto:auth_path=/hiiqos-49/obj5 2023-07-29T00:10:40.274 DEBUG:boto:Method: HEAD 2023-07-29T00:10:40.274 DEBUG:boto:Path: /hiiqos-49/obj5 2023-07-29T00:10:40.275 DEBUG:boto:Data: 2023-07-29T00:10:40.275 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.276 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:40.276 DEBUG:boto:Port: 8000 2023-07-29T00:10:40.276 DEBUG:boto:Params: {} 2023-07-29T00:10:40.277 DEBUG:boto:Token: None 2023-07-29T00:10:40.278 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj5 2023-07-29T00:10:40.278 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cn40O5WOglecpbJys4NoNwRKMhk= 2023-07-29T00:10:40.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cn40O5WOglecpbJys4NoNwRKMhk=', 'Content-Length': '0'} 2023-07-29T00:10:40.280 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ca4be544d045b34a-0064c45900-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-07-29T00:10:40.282 DEBUG:boto:path=/hiiqos-49/obj5 2023-07-29T00:10:40.282 DEBUG:boto:auth_path=/hiiqos-49/obj5 2023-07-29T00:10:40.283 DEBUG:boto:Method: GET 2023-07-29T00:10:40.283 DEBUG:boto:Path: /hiiqos-49/obj5 2023-07-29T00:10:40.283 DEBUG:boto:Data: 2023-07-29T00:10:40.284 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.284 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:40.285 DEBUG:boto:Port: 8001 2023-07-29T00:10:40.285 DEBUG:boto:Params: {} 2023-07-29T00:10:40.285 DEBUG:boto:Token: None 2023-07-29T00:10:40.286 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj5 2023-07-29T00:10:40.286 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:d4LFk4L8d9UIqBEgIDQbJx+sV+o= 2023-07-29T00:10:40.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:d4LFk4L8d9UIqBEgIDQbJx+sV+o=', 'Content-Length': '0'} 2023-07-29T00:10:40.289 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d2fd020d7532e94f-0064c45900-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.290 DEBUG:boto:path=/hiiqos-49/obj5 2023-07-29T00:10:40.290 DEBUG:boto:auth_path=/hiiqos-49/obj5 2023-07-29T00:10:40.291 DEBUG:boto:Method: GET 2023-07-29T00:10:40.291 DEBUG:boto:Path: /hiiqos-49/obj5 2023-07-29T00:10:40.291 DEBUG:boto:Data: 2023-07-29T00:10:40.292 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.292 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:40.293 DEBUG:boto:Port: 8000 2023-07-29T00:10:40.293 DEBUG:boto:Params: {} 2023-07-29T00:10:40.293 DEBUG:boto:Token: None 2023-07-29T00:10:40.294 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj5 2023-07-29T00:10:40.294 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:d4LFk4L8d9UIqBEgIDQbJx+sV+o= 2023-07-29T00:10:40.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:d4LFk4L8d9UIqBEgIDQbJx+sV+o=', 'Content-Length': '0'} 2023-07-29T00:10:40.296 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000bb00d7777c858ae-0064c45900-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.297 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:10:40.298 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:10:40.299 DEBUG:boto:path=/hiiqos-49/?prefix=obj5 2023-07-29T00:10:40.299 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj5 2023-07-29T00:10:40.300 DEBUG:boto:Method: GET 2023-07-29T00:10:40.301 DEBUG:boto:Path: /hiiqos-49/?prefix=obj5 2023-07-29T00:10:40.302 DEBUG:boto:Data: 2023-07-29T00:10:40.303 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.304 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:40.305 DEBUG:boto:Port: 8001 2023-07-29T00:10:40.305 DEBUG:boto:Params: {} 2023-07-29T00:10:40.306 DEBUG:boto:Token: None 2023-07-29T00:10:40.307 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/ 2023-07-29T00:10:40.308 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UQ1dXMReSrCpLgdbHZMnK072e8Y= 2023-07-29T00:10:40.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UQ1dXMReSrCpLgdbHZMnK072e8Y=', 'Content-Length': '0'} 2023-07-29T00:10:40.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012b5d138729eb8ac-0064c45900-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.311 DEBUG:boto:b'hiiqos-49obj51000falseobj52023-07-29T00:09:22.313Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:10:40.312 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:10:40.312 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:10:40.313 DEBUG:boto:path=/hiiqos-49/?prefix=obj5 2023-07-29T00:10:40.313 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj5 2023-07-29T00:10:40.314 DEBUG:boto:Method: GET 2023-07-29T00:10:40.314 DEBUG:boto:Path: /hiiqos-49/?prefix=obj5 2023-07-29T00:10:40.314 DEBUG:boto:Data: 2023-07-29T00:10:40.315 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.315 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:40.315 DEBUG:boto:Port: 8000 2023-07-29T00:10:40.316 DEBUG:boto:Params: {} 2023-07-29T00:10:40.316 DEBUG:boto:Token: None 2023-07-29T00:10:40.317 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/ 2023-07-29T00:10:40.317 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UQ1dXMReSrCpLgdbHZMnK072e8Y= 2023-07-29T00:10:40.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UQ1dXMReSrCpLgdbHZMnK072e8Y=', 'Content-Length': '0'} 2023-07-29T00:10:40.320 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab5efa442387e0a0-0064c45900-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.360 DEBUG:boto:b'hiiqos-49obj51000falseobj52023-07-29T00:09:22.313Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:10:40.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-07-29T00:10:40.363 DEBUG:boto:path=/hiiqos-49/obj6 2023-07-29T00:10:40.363 DEBUG:boto:auth_path=/hiiqos-49/obj6 2023-07-29T00:10:40.364 DEBUG:boto:path=/hiiqos-49/obj6?versionId=null 2023-07-29T00:10:40.365 DEBUG:boto:auth_path=/hiiqos-49/obj6?versionId=null 2023-07-29T00:10:40.366 DEBUG:boto:Method: GET 2023-07-29T00:10:40.366 DEBUG:boto:Path: /hiiqos-49/obj6?versionId=null 2023-07-29T00:10:40.366 DEBUG:boto:Data: 2023-07-29T00:10:40.367 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.367 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:40.367 DEBUG:boto:Port: 8001 2023-07-29T00:10:40.368 DEBUG:boto:Params: {} 2023-07-29T00:10:40.368 DEBUG:boto:Token: None 2023-07-29T00:10:40.369 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj6?versionId=null 2023-07-29T00:10:40.369 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LNlsWFt9LJFsjGwtMIo6mNJKoGE= 2023-07-29T00:10:40.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LNlsWFt9LJFsjGwtMIo6mNJKoGE=', 'Content-Length': '0'} 2023-07-29T00:10:40.372 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001f5e7e82ec0dfcb1-0064c45900-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.372 DEBUG:boto:path=/hiiqos-49/obj6 2023-07-29T00:10:40.373 DEBUG:boto:auth_path=/hiiqos-49/obj6 2023-07-29T00:10:40.373 DEBUG:boto:path=/hiiqos-49/obj6?versionId=null 2023-07-29T00:10:40.373 DEBUG:boto:auth_path=/hiiqos-49/obj6?versionId=null 2023-07-29T00:10:40.374 DEBUG:boto:Method: GET 2023-07-29T00:10:40.374 DEBUG:boto:Path: /hiiqos-49/obj6?versionId=null 2023-07-29T00:10:40.375 DEBUG:boto:Data: 2023-07-29T00:10:40.375 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.375 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:40.376 DEBUG:boto:Port: 8000 2023-07-29T00:10:40.376 DEBUG:boto:Params: {} 2023-07-29T00:10:40.376 DEBUG:boto:Token: None 2023-07-29T00:10:40.377 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj6?versionId=null 2023-07-29T00:10:40.377 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LNlsWFt9LJFsjGwtMIo6mNJKoGE= 2023-07-29T00:10:40.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LNlsWFt9LJFsjGwtMIo6mNJKoGE=', 'Content-Length': '0'} 2023-07-29T00:10:40.380 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e3ecf66a24a6beb-0064c45900-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.380 DEBUG:boto:path=/hiiqos-49/obj6 2023-07-29T00:10:40.381 DEBUG:boto:auth_path=/hiiqos-49/obj6 2023-07-29T00:10:40.381 DEBUG:boto:path=/hiiqos-49/obj6?versionId=null 2023-07-29T00:10:40.382 DEBUG:boto:auth_path=/hiiqos-49/obj6?versionId=null 2023-07-29T00:10:40.382 DEBUG:boto:Method: HEAD 2023-07-29T00:10:40.382 DEBUG:boto:Path: /hiiqos-49/obj6?versionId=null 2023-07-29T00:10:40.383 DEBUG:boto:Data: 2023-07-29T00:10:40.383 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.383 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:40.384 DEBUG:boto:Port: 8001 2023-07-29T00:10:40.384 DEBUG:boto:Params: {} 2023-07-29T00:10:40.385 DEBUG:boto:Token: None 2023-07-29T00:10:40.385 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj6?versionId=null 2023-07-29T00:10:40.385 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Xantf/08nZV2FD9xiIH8Rzp0qm8= 2023-07-29T00:10:40.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Xantf/08nZV2FD9xiIH8Rzp0qm8=', 'Content-Length': '0'} 2023-07-29T00:10:40.387 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c88edfbc95cd8032-0064c45900-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.388 DEBUG:boto:path=/hiiqos-49/obj6 2023-07-29T00:10:40.389 DEBUG:boto:auth_path=/hiiqos-49/obj6 2023-07-29T00:10:40.389 DEBUG:boto:path=/hiiqos-49/obj6?versionId=null 2023-07-29T00:10:40.390 DEBUG:boto:auth_path=/hiiqos-49/obj6?versionId=null 2023-07-29T00:10:40.390 DEBUG:boto:Method: HEAD 2023-07-29T00:10:40.390 DEBUG:boto:Path: /hiiqos-49/obj6?versionId=null 2023-07-29T00:10:40.391 DEBUG:boto:Data: 2023-07-29T00:10:40.391 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.391 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:40.392 DEBUG:boto:Port: 8000 2023-07-29T00:10:40.392 DEBUG:boto:Params: {} 2023-07-29T00:10:40.393 DEBUG:boto:Token: None 2023-07-29T00:10:40.393 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj6?versionId=null 2023-07-29T00:10:40.393 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Xantf/08nZV2FD9xiIH8Rzp0qm8= 2023-07-29T00:10:40.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Xantf/08nZV2FD9xiIH8Rzp0qm8=', 'Content-Length': '0'} 2023-07-29T00:10:40.396 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c3a09f2d3de56328-0064c45900-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-07-29T00:10:40.397 DEBUG:boto:path=/hiiqos-49/obj6 2023-07-29T00:10:40.397 DEBUG:boto:auth_path=/hiiqos-49/obj6 2023-07-29T00:10:40.398 DEBUG:boto:path=/hiiqos-49/obj6?versionId=null 2023-07-29T00:10:40.398 DEBUG:boto:auth_path=/hiiqos-49/obj6?versionId=null 2023-07-29T00:10:40.398 DEBUG:boto:Method: GET 2023-07-29T00:10:40.399 DEBUG:boto:Path: /hiiqos-49/obj6?versionId=null 2023-07-29T00:10:40.399 DEBUG:boto:Data: 2023-07-29T00:10:40.399 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.400 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:40.400 DEBUG:boto:Port: 8001 2023-07-29T00:10:40.400 DEBUG:boto:Params: {} 2023-07-29T00:10:40.401 DEBUG:boto:Token: None 2023-07-29T00:10:40.401 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj6?versionId=null 2023-07-29T00:10:40.402 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LNlsWFt9LJFsjGwtMIo6mNJKoGE= 2023-07-29T00:10:40.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LNlsWFt9LJFsjGwtMIo6mNJKoGE=', 'Content-Length': '0'} 2023-07-29T00:10:40.404 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a32f5c1267d087c6-0064c45900-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.404 DEBUG:boto:path=/hiiqos-49/obj6 2023-07-29T00:10:40.405 DEBUG:boto:auth_path=/hiiqos-49/obj6 2023-07-29T00:10:40.405 DEBUG:boto:path=/hiiqos-49/obj6?versionId=null 2023-07-29T00:10:40.406 DEBUG:boto:auth_path=/hiiqos-49/obj6?versionId=null 2023-07-29T00:10:40.406 DEBUG:boto:Method: GET 2023-07-29T00:10:40.407 DEBUG:boto:Path: /hiiqos-49/obj6?versionId=null 2023-07-29T00:10:40.407 DEBUG:boto:Data: 2023-07-29T00:10:40.408 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.408 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:40.409 DEBUG:boto:Port: 8000 2023-07-29T00:10:40.409 DEBUG:boto:Params: {} 2023-07-29T00:10:40.409 DEBUG:boto:Token: None 2023-07-29T00:10:40.410 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj6?versionId=null 2023-07-29T00:10:40.411 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LNlsWFt9LJFsjGwtMIo6mNJKoGE= 2023-07-29T00:10:40.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LNlsWFt9LJFsjGwtMIo6mNJKoGE=', 'Content-Length': '0'} 2023-07-29T00:10:40.413 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ac935c764aebda50-0064c45900-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.413 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:10:40.414 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:10:40.414 DEBUG:boto:path=/hiiqos-49/?prefix=obj6 2023-07-29T00:10:40.415 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj6 2023-07-29T00:10:40.415 DEBUG:boto:Method: GET 2023-07-29T00:10:40.415 DEBUG:boto:Path: /hiiqos-49/?prefix=obj6 2023-07-29T00:10:40.416 DEBUG:boto:Data: 2023-07-29T00:10:40.416 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.417 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:40.417 DEBUG:boto:Port: 8001 2023-07-29T00:10:40.418 DEBUG:boto:Params: {} 2023-07-29T00:10:40.418 DEBUG:boto:Token: None 2023-07-29T00:10:40.418 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/ 2023-07-29T00:10:40.419 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UQ1dXMReSrCpLgdbHZMnK072e8Y= 2023-07-29T00:10:40.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UQ1dXMReSrCpLgdbHZMnK072e8Y=', 'Content-Length': '0'} 2023-07-29T00:10:40.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004c114e72d82864c-0064c45900-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.422 DEBUG:boto:b'hiiqos-49obj61000falseobj62023-07-29T00:09:22.330Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:10:40.423 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:10:40.423 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:10:40.423 DEBUG:boto:path=/hiiqos-49/?prefix=obj6 2023-07-29T00:10:40.424 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj6 2023-07-29T00:10:40.424 DEBUG:boto:Method: GET 2023-07-29T00:10:40.425 DEBUG:boto:Path: /hiiqos-49/?prefix=obj6 2023-07-29T00:10:40.425 DEBUG:boto:Data: 2023-07-29T00:10:40.425 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.426 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:40.426 DEBUG:boto:Port: 8000 2023-07-29T00:10:40.426 DEBUG:boto:Params: {} 2023-07-29T00:10:40.427 DEBUG:boto:Token: None 2023-07-29T00:10:40.427 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/ 2023-07-29T00:10:40.428 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UQ1dXMReSrCpLgdbHZMnK072e8Y= 2023-07-29T00:10:40.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UQ1dXMReSrCpLgdbHZMnK072e8Y=', 'Content-Length': '0'} 2023-07-29T00:10:40.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000532c599f3cafe284-0064c45900-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.431 DEBUG:boto:b'hiiqos-49obj61000falseobj62023-07-29T00:09:22.330Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:10:40.431 DEBUG:boto:path=/hiiqos-49/obj6 2023-07-29T00:10:40.432 DEBUG:boto:auth_path=/hiiqos-49/obj6 2023-07-29T00:10:40.432 DEBUG:boto:Method: HEAD 2023-07-29T00:10:40.432 DEBUG:boto:Path: /hiiqos-49/obj6 2023-07-29T00:10:40.433 DEBUG:boto:Data: 2023-07-29T00:10:40.433 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.434 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:40.434 DEBUG:boto:Port: 8001 2023-07-29T00:10:40.435 DEBUG:boto:Params: {} 2023-07-29T00:10:40.435 DEBUG:boto:Token: None 2023-07-29T00:10:40.436 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj6 2023-07-29T00:10:40.437 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OoJ5yOkDjC9slAUw6YR4BM3Ztk0= 2023-07-29T00:10:40.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OoJ5yOkDjC9slAUw6YR4BM3Ztk0=', 'Content-Length': '0'} 2023-07-29T00:10:40.439 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cce28a4ea64588d6-0064c45900-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.439 DEBUG:boto:path=/hiiqos-49/obj6 2023-07-29T00:10:40.440 DEBUG:boto:auth_path=/hiiqos-49/obj6 2023-07-29T00:10:40.440 DEBUG:boto:Method: HEAD 2023-07-29T00:10:40.440 DEBUG:boto:Path: /hiiqos-49/obj6 2023-07-29T00:10:40.441 DEBUG:boto:Data: 2023-07-29T00:10:40.441 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.442 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:40.442 DEBUG:boto:Port: 8000 2023-07-29T00:10:40.442 DEBUG:boto:Params: {} 2023-07-29T00:10:40.443 DEBUG:boto:Token: None 2023-07-29T00:10:40.443 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj6 2023-07-29T00:10:40.444 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OoJ5yOkDjC9slAUw6YR4BM3Ztk0= 2023-07-29T00:10:40.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OoJ5yOkDjC9slAUw6YR4BM3Ztk0=', 'Content-Length': '0'} 2023-07-29T00:10:40.446 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000062674bf4cdbbbf5a-0064c45900-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-07-29T00:10:40.447 DEBUG:boto:path=/hiiqos-49/obj6 2023-07-29T00:10:40.447 DEBUG:boto:auth_path=/hiiqos-49/obj6 2023-07-29T00:10:40.447 DEBUG:boto:Method: GET 2023-07-29T00:10:40.448 DEBUG:boto:Path: /hiiqos-49/obj6 2023-07-29T00:10:40.448 DEBUG:boto:Data: 2023-07-29T00:10:40.449 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.449 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:40.449 DEBUG:boto:Port: 8001 2023-07-29T00:10:40.450 DEBUG:boto:Params: {} 2023-07-29T00:10:40.450 DEBUG:boto:Token: None 2023-07-29T00:10:40.451 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj6 2023-07-29T00:10:40.451 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3ECL9+sMQpLfgfYcn9i5fXiYmB0= 2023-07-29T00:10:40.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3ECL9+sMQpLfgfYcn9i5fXiYmB0=', 'Content-Length': '0'} 2023-07-29T00:10:40.453 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000655f675e1f7aab65-0064c45900-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.500 DEBUG:boto:path=/hiiqos-49/obj6 2023-07-29T00:10:40.501 DEBUG:boto:auth_path=/hiiqos-49/obj6 2023-07-29T00:10:40.501 DEBUG:boto:Method: GET 2023-07-29T00:10:40.502 DEBUG:boto:Path: /hiiqos-49/obj6 2023-07-29T00:10:40.502 DEBUG:boto:Data: 2023-07-29T00:10:40.502 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.503 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:40.503 DEBUG:boto:Port: 8000 2023-07-29T00:10:40.503 DEBUG:boto:Params: {} 2023-07-29T00:10:40.504 DEBUG:boto:Token: None 2023-07-29T00:10:40.504 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj6 2023-07-29T00:10:40.505 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3ECL9+sMQpLfgfYcn9i5fXiYmB0= 2023-07-29T00:10:40.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3ECL9+sMQpLfgfYcn9i5fXiYmB0=', 'Content-Length': '0'} 2023-07-29T00:10:40.507 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000615f6400f5c0d5c3-0064c45900-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.508 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:10:40.508 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:10:40.508 DEBUG:boto:path=/hiiqos-49/?prefix=obj6 2023-07-29T00:10:40.509 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj6 2023-07-29T00:10:40.509 DEBUG:boto:Method: GET 2023-07-29T00:10:40.510 DEBUG:boto:Path: /hiiqos-49/?prefix=obj6 2023-07-29T00:10:40.510 DEBUG:boto:Data: 2023-07-29T00:10:40.510 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.511 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:40.511 DEBUG:boto:Port: 8001 2023-07-29T00:10:40.511 DEBUG:boto:Params: {} 2023-07-29T00:10:40.512 DEBUG:boto:Token: None 2023-07-29T00:10:40.512 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/ 2023-07-29T00:10:40.513 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UQ1dXMReSrCpLgdbHZMnK072e8Y= 2023-07-29T00:10:40.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UQ1dXMReSrCpLgdbHZMnK072e8Y=', 'Content-Length': '0'} 2023-07-29T00:10:40.516 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000661e04f9e13acd9b-0064c45900-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.516 DEBUG:boto:b'hiiqos-49obj61000falseobj62023-07-29T00:09:22.330Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:10:40.517 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:10:40.517 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:10:40.518 DEBUG:boto:path=/hiiqos-49/?prefix=obj6 2023-07-29T00:10:40.518 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj6 2023-07-29T00:10:40.519 DEBUG:boto:Method: GET 2023-07-29T00:10:40.519 DEBUG:boto:Path: /hiiqos-49/?prefix=obj6 2023-07-29T00:10:40.519 DEBUG:boto:Data: 2023-07-29T00:10:40.520 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.520 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:40.520 DEBUG:boto:Port: 8000 2023-07-29T00:10:40.521 DEBUG:boto:Params: {} 2023-07-29T00:10:40.522 DEBUG:boto:Token: None 2023-07-29T00:10:40.523 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/ 2023-07-29T00:10:40.524 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UQ1dXMReSrCpLgdbHZMnK072e8Y= 2023-07-29T00:10:40.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UQ1dXMReSrCpLgdbHZMnK072e8Y=', 'Content-Length': '0'} 2023-07-29T00:10:40.527 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072e68e0131514d63-0064c45900-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.528 DEBUG:boto:b'hiiqos-49obj61000falseobj62023-07-29T00:09:22.330Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:10:40.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-07-29T00:10:40.529 DEBUG:boto:path=/hiiqos-49/obj7 2023-07-29T00:10:40.529 DEBUG:boto:auth_path=/hiiqos-49/obj7 2023-07-29T00:10:40.530 DEBUG:boto:path=/hiiqos-49/obj7?versionId=null 2023-07-29T00:10:40.530 DEBUG:boto:auth_path=/hiiqos-49/obj7?versionId=null 2023-07-29T00:10:40.530 DEBUG:boto:Method: GET 2023-07-29T00:10:40.531 DEBUG:boto:Path: /hiiqos-49/obj7?versionId=null 2023-07-29T00:10:40.531 DEBUG:boto:Data: 2023-07-29T00:10:40.531 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.532 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:40.532 DEBUG:boto:Port: 8001 2023-07-29T00:10:40.533 DEBUG:boto:Params: {} 2023-07-29T00:10:40.533 DEBUG:boto:Token: None 2023-07-29T00:10:40.533 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj7?versionId=null 2023-07-29T00:10:40.534 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:veAeugJTqz3Xklw0q3F7kJhlTmM= 2023-07-29T00:10:40.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:veAeugJTqz3Xklw0q3F7kJhlTmM=', 'Content-Length': '0'} 2023-07-29T00:10:40.536 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f2ea8a6b3ca4d0e1-0064c45900-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.536 DEBUG:boto:path=/hiiqos-49/obj7 2023-07-29T00:10:40.537 DEBUG:boto:auth_path=/hiiqos-49/obj7 2023-07-29T00:10:40.537 DEBUG:boto:path=/hiiqos-49/obj7?versionId=null 2023-07-29T00:10:40.538 DEBUG:boto:auth_path=/hiiqos-49/obj7?versionId=null 2023-07-29T00:10:40.538 DEBUG:boto:Method: GET 2023-07-29T00:10:40.538 DEBUG:boto:Path: /hiiqos-49/obj7?versionId=null 2023-07-29T00:10:40.539 DEBUG:boto:Data: 2023-07-29T00:10:40.539 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.539 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:40.540 DEBUG:boto:Port: 8000 2023-07-29T00:10:40.541 DEBUG:boto:Params: {} 2023-07-29T00:10:40.541 DEBUG:boto:Token: None 2023-07-29T00:10:40.542 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj7?versionId=null 2023-07-29T00:10:40.542 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:veAeugJTqz3Xklw0q3F7kJhlTmM= 2023-07-29T00:10:40.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:veAeugJTqz3Xklw0q3F7kJhlTmM=', 'Content-Length': '0'} 2023-07-29T00:10:40.544 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba212c90c767e106-0064c45900-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.545 DEBUG:boto:path=/hiiqos-49/obj7 2023-07-29T00:10:40.545 DEBUG:boto:auth_path=/hiiqos-49/obj7 2023-07-29T00:10:40.546 DEBUG:boto:path=/hiiqos-49/obj7?versionId=null 2023-07-29T00:10:40.546 DEBUG:boto:auth_path=/hiiqos-49/obj7?versionId=null 2023-07-29T00:10:40.546 DEBUG:boto:Method: HEAD 2023-07-29T00:10:40.547 DEBUG:boto:Path: /hiiqos-49/obj7?versionId=null 2023-07-29T00:10:40.547 DEBUG:boto:Data: 2023-07-29T00:10:40.547 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.548 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:40.548 DEBUG:boto:Port: 8001 2023-07-29T00:10:40.549 DEBUG:boto:Params: {} 2023-07-29T00:10:40.550 DEBUG:boto:Token: None 2023-07-29T00:10:40.551 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj7?versionId=null 2023-07-29T00:10:40.552 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xl8nz3UV7qPQGIGWYzjjf+Zts6o= 2023-07-29T00:10:40.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xl8nz3UV7qPQGIGWYzjjf+Zts6o=', 'Content-Length': '0'} 2023-07-29T00:10:40.554 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa1bf4269a085af5-0064c45900-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.554 DEBUG:boto:path=/hiiqos-49/obj7 2023-07-29T00:10:40.555 DEBUG:boto:auth_path=/hiiqos-49/obj7 2023-07-29T00:10:40.555 DEBUG:boto:path=/hiiqos-49/obj7?versionId=null 2023-07-29T00:10:40.555 DEBUG:boto:auth_path=/hiiqos-49/obj7?versionId=null 2023-07-29T00:10:40.556 DEBUG:boto:Method: HEAD 2023-07-29T00:10:40.556 DEBUG:boto:Path: /hiiqos-49/obj7?versionId=null 2023-07-29T00:10:40.557 DEBUG:boto:Data: 2023-07-29T00:10:40.557 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.557 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:40.558 DEBUG:boto:Port: 8000 2023-07-29T00:10:40.558 DEBUG:boto:Params: {} 2023-07-29T00:10:40.558 DEBUG:boto:Token: None 2023-07-29T00:10:40.559 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj7?versionId=null 2023-07-29T00:10:40.559 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xl8nz3UV7qPQGIGWYzjjf+Zts6o= 2023-07-29T00:10:40.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xl8nz3UV7qPQGIGWYzjjf+Zts6o=', 'Content-Length': '0'} 2023-07-29T00:10:40.561 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d784eead529d347-0064c45900-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-07-29T00:10:40.564 DEBUG:boto:path=/hiiqos-49/obj7 2023-07-29T00:10:40.565 DEBUG:boto:auth_path=/hiiqos-49/obj7 2023-07-29T00:10:40.565 DEBUG:boto:path=/hiiqos-49/obj7?versionId=null 2023-07-29T00:10:40.566 DEBUG:boto:auth_path=/hiiqos-49/obj7?versionId=null 2023-07-29T00:10:40.567 DEBUG:boto:Method: GET 2023-07-29T00:10:40.568 DEBUG:boto:Path: /hiiqos-49/obj7?versionId=null 2023-07-29T00:10:40.569 DEBUG:boto:Data: 2023-07-29T00:10:40.569 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.570 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:40.571 DEBUG:boto:Port: 8001 2023-07-29T00:10:40.571 DEBUG:boto:Params: {} 2023-07-29T00:10:40.572 DEBUG:boto:Token: None 2023-07-29T00:10:40.572 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj7?versionId=null 2023-07-29T00:10:40.573 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:veAeugJTqz3Xklw0q3F7kJhlTmM= 2023-07-29T00:10:40.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:veAeugJTqz3Xklw0q3F7kJhlTmM=', 'Content-Length': '0'} 2023-07-29T00:10:40.575 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000015b2c1573e9c01b3-0064c45900-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.576 DEBUG:boto:path=/hiiqos-49/obj7 2023-07-29T00:10:40.576 DEBUG:boto:auth_path=/hiiqos-49/obj7 2023-07-29T00:10:40.577 DEBUG:boto:path=/hiiqos-49/obj7?versionId=null 2023-07-29T00:10:40.577 DEBUG:boto:auth_path=/hiiqos-49/obj7?versionId=null 2023-07-29T00:10:40.577 DEBUG:boto:Method: GET 2023-07-29T00:10:40.578 DEBUG:boto:Path: /hiiqos-49/obj7?versionId=null 2023-07-29T00:10:40.578 DEBUG:boto:Data: 2023-07-29T00:10:40.578 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.579 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:40.579 DEBUG:boto:Port: 8000 2023-07-29T00:10:40.580 DEBUG:boto:Params: {} 2023-07-29T00:10:40.580 DEBUG:boto:Token: None 2023-07-29T00:10:40.581 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj7?versionId=null 2023-07-29T00:10:40.581 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:veAeugJTqz3Xklw0q3F7kJhlTmM= 2023-07-29T00:10:40.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:veAeugJTqz3Xklw0q3F7kJhlTmM=', 'Content-Length': '0'} 2023-07-29T00:10:40.583 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e7206f11e83a106a-0064c45900-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.624 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:10:40.625 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:10:40.625 DEBUG:boto:path=/hiiqos-49/?prefix=obj7 2023-07-29T00:10:40.626 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj7 2023-07-29T00:10:40.626 DEBUG:boto:Method: GET 2023-07-29T00:10:40.626 DEBUG:boto:Path: /hiiqos-49/?prefix=obj7 2023-07-29T00:10:40.627 DEBUG:boto:Data: 2023-07-29T00:10:40.627 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.628 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:40.629 DEBUG:boto:Port: 8001 2023-07-29T00:10:40.629 DEBUG:boto:Params: {} 2023-07-29T00:10:40.630 DEBUG:boto:Token: None 2023-07-29T00:10:40.631 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/ 2023-07-29T00:10:40.631 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UQ1dXMReSrCpLgdbHZMnK072e8Y= 2023-07-29T00:10:40.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UQ1dXMReSrCpLgdbHZMnK072e8Y=', 'Content-Length': '0'} 2023-07-29T00:10:40.634 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004426898b6b88b99-0064c45900-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.635 DEBUG:boto:b'hiiqos-49obj71000falseobj72023-07-29T00:09:22.388Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:10:40.636 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:10:40.637 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:10:40.638 DEBUG:boto:path=/hiiqos-49/?prefix=obj7 2023-07-29T00:10:40.638 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj7 2023-07-29T00:10:40.639 DEBUG:boto:Method: GET 2023-07-29T00:10:40.640 DEBUG:boto:Path: /hiiqos-49/?prefix=obj7 2023-07-29T00:10:40.640 DEBUG:boto:Data: 2023-07-29T00:10:40.641 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.641 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:40.642 DEBUG:boto:Port: 8000 2023-07-29T00:10:40.643 DEBUG:boto:Params: {} 2023-07-29T00:10:40.643 DEBUG:boto:Token: None 2023-07-29T00:10:40.644 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/ 2023-07-29T00:10:40.644 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UQ1dXMReSrCpLgdbHZMnK072e8Y= 2023-07-29T00:10:40.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UQ1dXMReSrCpLgdbHZMnK072e8Y=', 'Content-Length': '0'} 2023-07-29T00:10:40.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4da12e1feb675bc-0064c45900-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.648 DEBUG:boto:b'hiiqos-49obj71000falseobj72023-07-29T00:09:22.388Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:10:40.649 DEBUG:boto:path=/hiiqos-49/obj7 2023-07-29T00:10:40.649 DEBUG:boto:auth_path=/hiiqos-49/obj7 2023-07-29T00:10:40.649 DEBUG:boto:Method: HEAD 2023-07-29T00:10:40.650 DEBUG:boto:Path: /hiiqos-49/obj7 2023-07-29T00:10:40.650 DEBUG:boto:Data: 2023-07-29T00:10:40.651 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.651 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:40.652 DEBUG:boto:Port: 8001 2023-07-29T00:10:40.652 DEBUG:boto:Params: {} 2023-07-29T00:10:40.652 DEBUG:boto:Token: None 2023-07-29T00:10:40.653 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj7 2023-07-29T00:10:40.653 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WfR18hfc52kOp38fMD/vWT3C2w8= 2023-07-29T00:10:40.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WfR18hfc52kOp38fMD/vWT3C2w8=', 'Content-Length': '0'} 2023-07-29T00:10:40.655 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000983ff9a33f3f3b79-0064c45900-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.656 DEBUG:boto:path=/hiiqos-49/obj7 2023-07-29T00:10:40.656 DEBUG:boto:auth_path=/hiiqos-49/obj7 2023-07-29T00:10:40.657 DEBUG:boto:Method: HEAD 2023-07-29T00:10:40.657 DEBUG:boto:Path: /hiiqos-49/obj7 2023-07-29T00:10:40.657 DEBUG:boto:Data: 2023-07-29T00:10:40.658 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.658 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:40.659 DEBUG:boto:Port: 8000 2023-07-29T00:10:40.659 DEBUG:boto:Params: {} 2023-07-29T00:10:40.659 DEBUG:boto:Token: None 2023-07-29T00:10:40.660 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj7 2023-07-29T00:10:40.660 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WfR18hfc52kOp38fMD/vWT3C2w8= 2023-07-29T00:10:40.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WfR18hfc52kOp38fMD/vWT3C2w8=', 'Content-Length': '0'} 2023-07-29T00:10:40.662 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ed2b93ec5b4e8d46-0064c45900-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-07-29T00:10:40.663 DEBUG:boto:path=/hiiqos-49/obj7 2023-07-29T00:10:40.663 DEBUG:boto:auth_path=/hiiqos-49/obj7 2023-07-29T00:10:40.664 DEBUG:boto:Method: GET 2023-07-29T00:10:40.664 DEBUG:boto:Path: /hiiqos-49/obj7 2023-07-29T00:10:40.664 DEBUG:boto:Data: 2023-07-29T00:10:40.665 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.665 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:40.666 DEBUG:boto:Port: 8001 2023-07-29T00:10:40.666 DEBUG:boto:Params: {} 2023-07-29T00:10:40.666 DEBUG:boto:Token: None 2023-07-29T00:10:40.667 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj7 2023-07-29T00:10:40.667 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tOvOX4pcCV2Exm2Od1SRXpL45eM= 2023-07-29T00:10:40.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tOvOX4pcCV2Exm2Od1SRXpL45eM=', 'Content-Length': '0'} 2023-07-29T00:10:40.669 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e9da410a48797f9-0064c45900-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.670 DEBUG:boto:path=/hiiqos-49/obj7 2023-07-29T00:10:40.670 DEBUG:boto:auth_path=/hiiqos-49/obj7 2023-07-29T00:10:40.671 DEBUG:boto:Method: GET 2023-07-29T00:10:40.671 DEBUG:boto:Path: /hiiqos-49/obj7 2023-07-29T00:10:40.671 DEBUG:boto:Data: 2023-07-29T00:10:40.672 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.672 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:40.673 DEBUG:boto:Port: 8000 2023-07-29T00:10:40.673 DEBUG:boto:Params: {} 2023-07-29T00:10:40.674 DEBUG:boto:Token: None 2023-07-29T00:10:40.674 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj7 2023-07-29T00:10:40.674 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tOvOX4pcCV2Exm2Od1SRXpL45eM= 2023-07-29T00:10:40.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tOvOX4pcCV2Exm2Od1SRXpL45eM=', 'Content-Length': '0'} 2023-07-29T00:10:40.676 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000033889c05cd54e3e4-0064c45900-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.677 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:10:40.678 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:10:40.678 DEBUG:boto:path=/hiiqos-49/?prefix=obj7 2023-07-29T00:10:40.678 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj7 2023-07-29T00:10:40.679 DEBUG:boto:Method: GET 2023-07-29T00:10:40.679 DEBUG:boto:Path: /hiiqos-49/?prefix=obj7 2023-07-29T00:10:40.679 DEBUG:boto:Data: 2023-07-29T00:10:40.680 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.680 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:40.681 DEBUG:boto:Port: 8001 2023-07-29T00:10:40.681 DEBUG:boto:Params: {} 2023-07-29T00:10:40.681 DEBUG:boto:Token: None 2023-07-29T00:10:40.682 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/ 2023-07-29T00:10:40.682 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UQ1dXMReSrCpLgdbHZMnK072e8Y= 2023-07-29T00:10:40.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UQ1dXMReSrCpLgdbHZMnK072e8Y=', 'Content-Length': '0'} 2023-07-29T00:10:40.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fa5c3d9592441aa-0064c45900-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.686 DEBUG:boto:b'hiiqos-49obj71000falseobj72023-07-29T00:09:22.388Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:10:40.686 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:10:40.687 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:10:40.687 DEBUG:boto:path=/hiiqos-49/?prefix=obj7 2023-07-29T00:10:40.688 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj7 2023-07-29T00:10:40.688 DEBUG:boto:Method: GET 2023-07-29T00:10:40.688 DEBUG:boto:Path: /hiiqos-49/?prefix=obj7 2023-07-29T00:10:40.689 DEBUG:boto:Data: 2023-07-29T00:10:40.689 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.690 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:40.690 DEBUG:boto:Port: 8000 2023-07-29T00:10:40.691 DEBUG:boto:Params: {} 2023-07-29T00:10:40.691 DEBUG:boto:Token: None 2023-07-29T00:10:40.691 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/ 2023-07-29T00:10:40.692 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UQ1dXMReSrCpLgdbHZMnK072e8Y= 2023-07-29T00:10:40.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UQ1dXMReSrCpLgdbHZMnK072e8Y=', 'Content-Length': '0'} 2023-07-29T00:10:40.694 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e4bb2639e2c34a3-0064c45900-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.695 DEBUG:boto:b'hiiqos-49obj71000falseobj72023-07-29T00:09:22.388Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:10:40.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-07-29T00:10:40.696 DEBUG:boto:path=/hiiqos-49/obj8 2023-07-29T00:10:40.696 DEBUG:boto:auth_path=/hiiqos-49/obj8 2023-07-29T00:10:40.697 DEBUG:boto:path=/hiiqos-49/obj8?versionId=null 2023-07-29T00:10:40.697 DEBUG:boto:auth_path=/hiiqos-49/obj8?versionId=null 2023-07-29T00:10:40.698 DEBUG:boto:Method: GET 2023-07-29T00:10:40.698 DEBUG:boto:Path: /hiiqos-49/obj8?versionId=null 2023-07-29T00:10:40.698 DEBUG:boto:Data: 2023-07-29T00:10:40.699 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.699 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:40.699 DEBUG:boto:Port: 8001 2023-07-29T00:10:40.700 DEBUG:boto:Params: {} 2023-07-29T00:10:40.700 DEBUG:boto:Token: None 2023-07-29T00:10:40.701 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj8?versionId=null 2023-07-29T00:10:40.701 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:D7ikapsovOTIBLPiLyGxhPgklLs= 2023-07-29T00:10:40.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:D7ikapsovOTIBLPiLyGxhPgklLs=', 'Content-Length': '0'} 2023-07-29T00:10:40.703 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c9e35685b109ac5-0064c45900-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.704 DEBUG:boto:path=/hiiqos-49/obj8 2023-07-29T00:10:40.705 DEBUG:boto:auth_path=/hiiqos-49/obj8 2023-07-29T00:10:40.705 DEBUG:boto:path=/hiiqos-49/obj8?versionId=null 2023-07-29T00:10:40.705 DEBUG:boto:auth_path=/hiiqos-49/obj8?versionId=null 2023-07-29T00:10:40.706 DEBUG:boto:Method: GET 2023-07-29T00:10:40.706 DEBUG:boto:Path: /hiiqos-49/obj8?versionId=null 2023-07-29T00:10:40.707 DEBUG:boto:Data: 2023-07-29T00:10:40.707 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.707 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:40.708 DEBUG:boto:Port: 8000 2023-07-29T00:10:40.708 DEBUG:boto:Params: {} 2023-07-29T00:10:40.708 DEBUG:boto:Token: None 2023-07-29T00:10:40.709 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj8?versionId=null 2023-07-29T00:10:40.709 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:D7ikapsovOTIBLPiLyGxhPgklLs= 2023-07-29T00:10:40.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:D7ikapsovOTIBLPiLyGxhPgklLs=', 'Content-Length': '0'} 2023-07-29T00:10:40.712 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000089616dc11155d53f-0064c45900-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.713 DEBUG:boto:path=/hiiqos-49/obj8 2023-07-29T00:10:40.713 DEBUG:boto:auth_path=/hiiqos-49/obj8 2023-07-29T00:10:40.714 DEBUG:boto:path=/hiiqos-49/obj8?versionId=null 2023-07-29T00:10:40.715 DEBUG:boto:auth_path=/hiiqos-49/obj8?versionId=null 2023-07-29T00:10:40.715 DEBUG:boto:Method: HEAD 2023-07-29T00:10:40.716 DEBUG:boto:Path: /hiiqos-49/obj8?versionId=null 2023-07-29T00:10:40.716 DEBUG:boto:Data: 2023-07-29T00:10:40.717 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.717 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:40.718 DEBUG:boto:Port: 8001 2023-07-29T00:10:40.718 DEBUG:boto:Params: {} 2023-07-29T00:10:40.718 DEBUG:boto:Token: None 2023-07-29T00:10:40.719 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj8?versionId=null 2023-07-29T00:10:40.719 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TcfXB+DbNb5L3jNHkm0kd6wSyio= 2023-07-29T00:10:40.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TcfXB+DbNb5L3jNHkm0kd6wSyio=', 'Content-Length': '0'} 2023-07-29T00:10:40.721 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c49514b31cc8795-0064c45900-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.722 DEBUG:boto:path=/hiiqos-49/obj8 2023-07-29T00:10:40.722 DEBUG:boto:auth_path=/hiiqos-49/obj8 2023-07-29T00:10:40.723 DEBUG:boto:path=/hiiqos-49/obj8?versionId=null 2023-07-29T00:10:40.723 DEBUG:boto:auth_path=/hiiqos-49/obj8?versionId=null 2023-07-29T00:10:40.724 DEBUG:boto:Method: HEAD 2023-07-29T00:10:40.724 DEBUG:boto:Path: /hiiqos-49/obj8?versionId=null 2023-07-29T00:10:40.724 DEBUG:boto:Data: 2023-07-29T00:10:40.725 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.725 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:40.726 DEBUG:boto:Port: 8000 2023-07-29T00:10:40.726 DEBUG:boto:Params: {} 2023-07-29T00:10:40.727 DEBUG:boto:Token: None 2023-07-29T00:10:40.727 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj8?versionId=null 2023-07-29T00:10:40.728 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TcfXB+DbNb5L3jNHkm0kd6wSyio= 2023-07-29T00:10:40.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TcfXB+DbNb5L3jNHkm0kd6wSyio=', 'Content-Length': '0'} 2023-07-29T00:10:40.730 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ed3448105207334-0064c45900-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-07-29T00:10:40.731 DEBUG:boto:path=/hiiqos-49/obj8 2023-07-29T00:10:40.731 DEBUG:boto:auth_path=/hiiqos-49/obj8 2023-07-29T00:10:40.731 DEBUG:boto:path=/hiiqos-49/obj8?versionId=null 2023-07-29T00:10:40.732 DEBUG:boto:auth_path=/hiiqos-49/obj8?versionId=null 2023-07-29T00:10:40.732 DEBUG:boto:Method: GET 2023-07-29T00:10:40.733 DEBUG:boto:Path: /hiiqos-49/obj8?versionId=null 2023-07-29T00:10:40.733 DEBUG:boto:Data: 2023-07-29T00:10:40.733 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.734 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:40.734 DEBUG:boto:Port: 8001 2023-07-29T00:10:40.734 DEBUG:boto:Params: {} 2023-07-29T00:10:40.735 DEBUG:boto:Token: None 2023-07-29T00:10:40.735 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj8?versionId=null 2023-07-29T00:10:40.736 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:D7ikapsovOTIBLPiLyGxhPgklLs= 2023-07-29T00:10:40.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:D7ikapsovOTIBLPiLyGxhPgklLs=', 'Content-Length': '0'} 2023-07-29T00:10:40.738 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000061f7c4be1b4dc5a7-0064c45900-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.739 DEBUG:boto:path=/hiiqos-49/obj8 2023-07-29T00:10:40.739 DEBUG:boto:auth_path=/hiiqos-49/obj8 2023-07-29T00:10:40.740 DEBUG:boto:path=/hiiqos-49/obj8?versionId=null 2023-07-29T00:10:40.740 DEBUG:boto:auth_path=/hiiqos-49/obj8?versionId=null 2023-07-29T00:10:40.740 DEBUG:boto:Method: GET 2023-07-29T00:10:40.741 DEBUG:boto:Path: /hiiqos-49/obj8?versionId=null 2023-07-29T00:10:40.741 DEBUG:boto:Data: 2023-07-29T00:10:40.741 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.742 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:40.742 DEBUG:boto:Port: 8000 2023-07-29T00:10:40.743 DEBUG:boto:Params: {} 2023-07-29T00:10:40.743 DEBUG:boto:Token: None 2023-07-29T00:10:40.743 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj8?versionId=null 2023-07-29T00:10:40.744 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:D7ikapsovOTIBLPiLyGxhPgklLs= 2023-07-29T00:10:40.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:D7ikapsovOTIBLPiLyGxhPgklLs=', 'Content-Length': '0'} 2023-07-29T00:10:40.746 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003a644b8db3a8403d-0064c45900-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.746 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:10:40.747 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:10:40.747 DEBUG:boto:path=/hiiqos-49/?prefix=obj8 2023-07-29T00:10:40.748 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj8 2023-07-29T00:10:40.748 DEBUG:boto:Method: GET 2023-07-29T00:10:40.748 DEBUG:boto:Path: /hiiqos-49/?prefix=obj8 2023-07-29T00:10:40.749 DEBUG:boto:Data: 2023-07-29T00:10:40.749 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.750 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:40.750 DEBUG:boto:Port: 8001 2023-07-29T00:10:40.750 DEBUG:boto:Params: {} 2023-07-29T00:10:40.751 DEBUG:boto:Token: None 2023-07-29T00:10:40.751 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/ 2023-07-29T00:10:40.752 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UQ1dXMReSrCpLgdbHZMnK072e8Y= 2023-07-29T00:10:40.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UQ1dXMReSrCpLgdbHZMnK072e8Y=', 'Content-Length': '0'} 2023-07-29T00:10:40.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a57177b6e32c1ee-0064c45900-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.755 DEBUG:boto:b'hiiqos-49obj81000falseobj82023-07-29T00:09:22.406Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:10:40.756 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:10:40.756 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:10:40.756 DEBUG:boto:path=/hiiqos-49/?prefix=obj8 2023-07-29T00:10:40.757 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj8 2023-07-29T00:10:40.757 DEBUG:boto:Method: GET 2023-07-29T00:10:40.758 DEBUG:boto:Path: /hiiqos-49/?prefix=obj8 2023-07-29T00:10:40.758 DEBUG:boto:Data: 2023-07-29T00:10:40.758 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.759 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:40.759 DEBUG:boto:Port: 8000 2023-07-29T00:10:40.760 DEBUG:boto:Params: {} 2023-07-29T00:10:40.760 DEBUG:boto:Token: None 2023-07-29T00:10:40.761 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/ 2023-07-29T00:10:40.761 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UQ1dXMReSrCpLgdbHZMnK072e8Y= 2023-07-29T00:10:40.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UQ1dXMReSrCpLgdbHZMnK072e8Y=', 'Content-Length': '0'} 2023-07-29T00:10:40.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047ff7af90bc8476c-0064c45900-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.764 DEBUG:boto:b'hiiqos-49obj81000falseobj82023-07-29T00:09:22.406Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:10:40.765 DEBUG:boto:path=/hiiqos-49/obj8 2023-07-29T00:10:40.765 DEBUG:boto:auth_path=/hiiqos-49/obj8 2023-07-29T00:10:40.766 DEBUG:boto:Method: HEAD 2023-07-29T00:10:40.766 DEBUG:boto:Path: /hiiqos-49/obj8 2023-07-29T00:10:40.766 DEBUG:boto:Data: 2023-07-29T00:10:40.767 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.767 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:40.767 DEBUG:boto:Port: 8001 2023-07-29T00:10:40.768 DEBUG:boto:Params: {} 2023-07-29T00:10:40.768 DEBUG:boto:Token: None 2023-07-29T00:10:40.768 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj8 2023-07-29T00:10:40.769 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P7so6j4gsm3a1gE8iXyv4SyctEA= 2023-07-29T00:10:40.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P7so6j4gsm3a1gE8iXyv4SyctEA=', 'Content-Length': '0'} 2023-07-29T00:10:40.771 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a0f81ca97e7d911-0064c45900-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.771 DEBUG:boto:path=/hiiqos-49/obj8 2023-07-29T00:10:40.772 DEBUG:boto:auth_path=/hiiqos-49/obj8 2023-07-29T00:10:40.773 DEBUG:boto:Method: HEAD 2023-07-29T00:10:40.773 DEBUG:boto:Path: /hiiqos-49/obj8 2023-07-29T00:10:40.774 DEBUG:boto:Data: 2023-07-29T00:10:40.774 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.775 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:40.775 DEBUG:boto:Port: 8000 2023-07-29T00:10:40.776 DEBUG:boto:Params: {} 2023-07-29T00:10:40.776 DEBUG:boto:Token: None 2023-07-29T00:10:40.777 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj8 2023-07-29T00:10:40.778 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P7so6j4gsm3a1gE8iXyv4SyctEA= 2023-07-29T00:10:40.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P7so6j4gsm3a1gE8iXyv4SyctEA=', 'Content-Length': '0'} 2023-07-29T00:10:40.780 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f4dc9b0bab1f811f-0064c45900-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-07-29T00:10:40.782 DEBUG:boto:path=/hiiqos-49/obj8 2023-07-29T00:10:40.782 DEBUG:boto:auth_path=/hiiqos-49/obj8 2023-07-29T00:10:40.783 DEBUG:boto:Method: GET 2023-07-29T00:10:40.783 DEBUG:boto:Path: /hiiqos-49/obj8 2023-07-29T00:10:40.784 DEBUG:boto:Data: 2023-07-29T00:10:40.784 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.785 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:40.785 DEBUG:boto:Port: 8001 2023-07-29T00:10:40.785 DEBUG:boto:Params: {} 2023-07-29T00:10:40.786 DEBUG:boto:Token: None 2023-07-29T00:10:40.786 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj8 2023-07-29T00:10:40.787 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YNBOORviZKkVgDryJUbB3CPDmdo= 2023-07-29T00:10:40.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YNBOORviZKkVgDryJUbB3CPDmdo=', 'Content-Length': '0'} 2023-07-29T00:10:40.789 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a62914980d6c7390-0064c45900-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.789 DEBUG:boto:path=/hiiqos-49/obj8 2023-07-29T00:10:40.790 DEBUG:boto:auth_path=/hiiqos-49/obj8 2023-07-29T00:10:40.790 DEBUG:boto:Method: GET 2023-07-29T00:10:40.791 DEBUG:boto:Path: /hiiqos-49/obj8 2023-07-29T00:10:40.791 DEBUG:boto:Data: 2023-07-29T00:10:40.791 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.792 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:40.792 DEBUG:boto:Port: 8000 2023-07-29T00:10:40.792 DEBUG:boto:Params: {} 2023-07-29T00:10:40.793 DEBUG:boto:Token: None 2023-07-29T00:10:40.793 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/obj8 2023-07-29T00:10:40.794 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YNBOORviZKkVgDryJUbB3CPDmdo= 2023-07-29T00:10:40.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YNBOORviZKkVgDryJUbB3CPDmdo=', 'Content-Length': '0'} 2023-07-29T00:10:40.796 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:09:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-49:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa6fed17aab1a3f5-0064c45900-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.836 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:10:40.837 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:10:40.837 DEBUG:boto:path=/hiiqos-49/?prefix=obj8 2023-07-29T00:10:40.838 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj8 2023-07-29T00:10:40.838 DEBUG:boto:Method: GET 2023-07-29T00:10:40.839 DEBUG:boto:Path: /hiiqos-49/?prefix=obj8 2023-07-29T00:10:40.839 DEBUG:boto:Data: 2023-07-29T00:10:40.839 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.840 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:40.840 DEBUG:boto:Port: 8001 2023-07-29T00:10:40.840 DEBUG:boto:Params: {} 2023-07-29T00:10:40.841 DEBUG:boto:Token: None 2023-07-29T00:10:40.841 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/ 2023-07-29T00:10:40.842 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UQ1dXMReSrCpLgdbHZMnK072e8Y= 2023-07-29T00:10:40.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UQ1dXMReSrCpLgdbHZMnK072e8Y=', 'Content-Length': '0'} 2023-07-29T00:10:40.844 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8509470634333db-0064c45900-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.845 DEBUG:boto:b'hiiqos-49obj81000falseobj82023-07-29T00:09:22.406Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:10:40.845 DEBUG:boto:path=/hiiqos-49/ 2023-07-29T00:10:40.846 DEBUG:boto:auth_path=/hiiqos-49/ 2023-07-29T00:10:40.846 DEBUG:boto:path=/hiiqos-49/?prefix=obj8 2023-07-29T00:10:40.846 DEBUG:boto:auth_path=/hiiqos-49/?prefix=obj8 2023-07-29T00:10:40.847 DEBUG:boto:Method: GET 2023-07-29T00:10:40.847 DEBUG:boto:Path: /hiiqos-49/?prefix=obj8 2023-07-29T00:10:40.848 DEBUG:boto:Data: 2023-07-29T00:10:40.848 DEBUG:boto:Headers: {} 2023-07-29T00:10:40.848 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:40.849 DEBUG:boto:Port: 8000 2023-07-29T00:10:40.849 DEBUG:boto:Params: {} 2023-07-29T00:10:40.849 DEBUG:boto:Token: None 2023-07-29T00:10:40.850 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:40 GMT /hiiqos-49/ 2023-07-29T00:10:40.850 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UQ1dXMReSrCpLgdbHZMnK072e8Y= 2023-07-29T00:10:40.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UQ1dXMReSrCpLgdbHZMnK072e8Y=', 'Content-Length': '0'} 2023-07-29T00:10:40.853 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b30555547a37acbe-0064c45900-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:10:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:40.853 DEBUG:boto:b'hiiqos-49obj81000falseobj82023-07-29T00:09:22.406Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:10:40.854 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-49 zones={a2, a1} 2023-07-29T00:10:40.854 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-07-29T00:10:40.855 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:10:40.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:10:40.855 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:10:40.904 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:10:40.904 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:10:40.949 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T00:10:40.950 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:10:40.950 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000010089", 2023-07-29T00:10:40.950 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:09:36.854629Z" 2023-07-29T00:10:40.950 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:10:40.951 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:10:40.951 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000007269", 2023-07-29T00:10:40.951 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:09:22.658599Z" 2023-07-29T00:10:40.951 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:10:40.951 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:10:40.951 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000006576", 2023-07-29T00:10:40.952 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:09:35.997703Z" 2023-07-29T00:10:40.952 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:10:40.952 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:10:40.952 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000004317", 2023-07-29T00:10:40.952 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:09:35.814480Z" 2023-07-29T00:10:40.953 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:10:40.953 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T00:10:40.954 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000010089', 1: '00000000000000000000:00000000000000007269', 2: '00000000000000000000:00000000000000006576', 3: '00000000000000000000:00000000000000004317'} 2023-07-29T00:10:40.954 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-07-29T00:10:40.954 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:10:40.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:10:40.955 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:10:41.002 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:10:41.002 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:10:41.044 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:10:41.045 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:10:41.045 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:10:41.045 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:10:41.045 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:10:41.045 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 16749483048653174838 2023-07-29T00:10:41.046 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:10:41.046 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:10:41.046 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:10:41.046 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:10:41.046 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:10:41.046 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:10:41.046 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000010089", 2023-07-29T00:10:41.047 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:10:41.047 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:10:41.047 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:10:41.047 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:09:36.854629Z" 2023-07-29T00:10:41.047 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:10:41.047 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:10:41.048 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:10:41.048 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:10:41.048 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:10:41.048 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:10:41.048 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000007269", 2023-07-29T00:10:41.049 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:10:41.049 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:10:41.049 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:10:41.049 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:09:22.658599Z" 2023-07-29T00:10:41.050 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:10:41.050 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:10:41.050 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:10:41.050 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:10:41.051 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:10:41.051 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:10:41.051 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000006576", 2023-07-29T00:10:41.051 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:10:41.051 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:10:41.051 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:10:41.052 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:09:35.997703Z" 2023-07-29T00:10:41.052 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:10:41.052 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:10:41.052 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:10:41.052 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:10:41.053 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:10:41.053 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:10:41.053 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000004317", 2023-07-29T00:10:41.053 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:10:41.053 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:10:41.053 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:10:41.053 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:09:35.814480Z" 2023-07-29T00:10:41.054 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:10:41.054 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:10:41.054 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:10:41.054 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:10:41.054 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:10:41.054 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 0, 2023-07-29T00:10:41.055 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 0 2023-07-29T00:10:41.055 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:10:41.055 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:10:41Z" 2023-07-29T00:10:41.055 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:10:41.056 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16749483048653174838 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000010089", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:09:36.854629Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007269", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:09:22.658599Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006576", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:09:35.997703Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004317", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:09:35.814480Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-07-29T00:10:41Z" } 2023-07-29T00:10:41.057 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000010089', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:09:36.854629Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007269', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:09:22.658599Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006576', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:09:35.997703Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004317', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:09:35.814480Z'}}] 2023-07-29T00:10:41.057 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000010089', 1: '00000000000000000000:00000000000000007269', 2: '00000000000000000000:00000000000000006576', 3: '00000000000000000000:00000000000000004317'} 2023-07-29T00:10:41.058 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000010089', 1: '00000000000000000000:00000000000000007269', 2: '00000000000000000000:00000000000000006576', 3: '00000000000000000000:00000000000000004317'} 2023-07-29T00:10:41.059 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-07-29T00:10:41.059 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-07-29T00:10:41.060 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:10:41.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:10:41.061 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:10:41.105 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:10:41.106 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:10:41.152 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T00:10:41.152 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:10:41.153 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000008397", 2023-07-29T00:10:41.153 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:09:39.221090Z" 2023-07-29T00:10:41.153 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:10:41.153 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:10:41.153 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000004593", 2023-07-29T00:10:41.153 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:09:39.222307Z" 2023-07-29T00:10:41.154 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:10:41.154 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:10:41.154 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000004597", 2023-07-29T00:10:41.154 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:09:39.223693Z" 2023-07-29T00:10:41.154 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:10:41.154 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:10:41.155 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000002907", 2023-07-29T00:10:41.155 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:09:39.224837Z" 2023-07-29T00:10:41.155 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:10:41.155 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T00:10:41.156 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000008397', 1: '00000000000000000000:00000000000000004593', 2: '00000000000000000000:00000000000000004597', 3: '00000000000000000000:00000000000000002907'} 2023-07-29T00:10:41.157 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-07-29T00:10:41.157 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:10:41.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:10:41.158 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:10:41.207 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:10:41.207 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:10:41.252 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:10:41.253 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:10:41.253 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:10:41.253 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:10:41.253 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:10:41.253 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 17882949198279251152 2023-07-29T00:10:41.254 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:10:41.254 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:10:41.254 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:10:41.254 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:10:41.254 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:10:41.254 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:10:41.254 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000008397", 2023-07-29T00:10:41.255 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:10:41.255 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:10:41.255 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:10:41.255 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:09:39.221090Z" 2023-07-29T00:10:41.255 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:10:41.255 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:10:41.256 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:10:41.256 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:10:41.256 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:10:41.256 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:10:41.256 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000004593", 2023-07-29T00:10:41.256 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:10:41.257 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:10:41.257 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:10:41.257 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:09:39.222307Z" 2023-07-29T00:10:41.257 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:10:41.257 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:10:41.257 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:10:41.257 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:10:41.258 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:10:41.258 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:10:41.258 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000004597", 2023-07-29T00:10:41.258 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:10:41.258 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:10:41.258 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:10:41.259 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:09:39.223693Z" 2023-07-29T00:10:41.259 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:10:41.259 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:10:41.259 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:10:41.259 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:10:41.259 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:10:41.260 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:10:41.260 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000002907", 2023-07-29T00:10:41.260 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:10:41.261 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:10:41.261 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:10:41.261 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:09:39.224837Z" 2023-07-29T00:10:41.261 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:10:41.261 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:10:41.261 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:10:41.262 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:10:41.262 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:10:41.262 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 0, 2023-07-29T00:10:41.262 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 0 2023-07-29T00:10:41.262 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:10:41.262 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:10:41Z" 2023-07-29T00:10:41.262 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:10:41.263 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17882949198279251152 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008397", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:09:39.221090Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004593", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:09:39.222307Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004597", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:09:39.223693Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002907", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:09:39.224837Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-07-29T00:10:41Z" } 2023-07-29T00:10:41.264 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008397', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:09:39.221090Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004593', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:09:39.222307Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004597', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:09:39.223693Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002907', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:09:39.224837Z'}}] 2023-07-29T00:10:41.264 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000008397', 1: '00000000000000000000:00000000000000004593', 2: '00000000000000000000:00000000000000004597', 3: '00000000000000000000:00000000000000002907'} 2023-07-29T00:10:41.265 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000008397', 1: '00000000000000000000:00000000000000004593', 2: '00000000000000000000:00000000000000004597', 3: '00000000000000000000:00000000000000002907'} 2023-07-29T00:10:41.265 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-07-29T00:10:41.266 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable_enable ... ok 2023-07-29T00:10:41.267 INFO:rgw_multi.tests:create bucket zone=a1 name=hiiqos-50 2023-07-29T00:10:41.267 DEBUG:boto:path=/hiiqos-50/ 2023-07-29T00:10:41.268 DEBUG:boto:auth_path=/hiiqos-50/ 2023-07-29T00:10:41.268 DEBUG:boto:Method: PUT 2023-07-29T00:10:41.269 DEBUG:boto:Path: /hiiqos-50/ 2023-07-29T00:10:41.269 DEBUG:boto:Data: 2023-07-29T00:10:41.270 DEBUG:boto:Headers: {} 2023-07-29T00:10:41.271 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:41.271 DEBUG:boto:Port: 8000 2023-07-29T00:10:41.272 DEBUG:boto:Params: {} 2023-07-29T00:10:41.273 DEBUG:boto:Token: None 2023-07-29T00:10:41.274 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 00:10:41 GMT /hiiqos-50/ 2023-07-29T00:10:41.274 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:diNnM3NiG1FgJjBRGbUS2o1ufb8= 2023-07-29T00:10:41.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:diNnM3NiG1FgJjBRGbUS2o1ufb8=', 'Content-Length': '0'} 2023-07-29T00:10:41.289 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e06fe9f7a5170064-0064c45901-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:10:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:41.289 INFO:rgw_multi.tests:create bucket zone=a2 name=hiiqos-51 2023-07-29T00:10:41.290 DEBUG:boto:path=/hiiqos-51/ 2023-07-29T00:10:41.290 DEBUG:boto:auth_path=/hiiqos-51/ 2023-07-29T00:10:41.290 DEBUG:boto:Method: PUT 2023-07-29T00:10:41.291 DEBUG:boto:Path: /hiiqos-51/ 2023-07-29T00:10:41.291 DEBUG:boto:Data: 2023-07-29T00:10:41.291 DEBUG:boto:Headers: {} 2023-07-29T00:10:41.292 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:10:41.292 DEBUG:boto:Port: 8001 2023-07-29T00:10:41.292 DEBUG:boto:Params: {} 2023-07-29T00:10:41.293 DEBUG:boto:Token: None 2023-07-29T00:10:41.293 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 00:10:41 GMT /hiiqos-51/ 2023-07-29T00:10:41.294 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uXT1zcTzgyhyETwDX1UotIYvG7A= 2023-07-29T00:10:41.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uXT1zcTzgyhyETwDX1UotIYvG7A=', 'Content-Length': '0'} 2023-07-29T00:10:41.320 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000df07150317d052c5-0064c45901-1663-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:10:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:41.320 DEBUG:boto:path=/hiiqos-50/MULTIPART 2023-07-29T00:10:41.321 DEBUG:boto:auth_path=/hiiqos-50/MULTIPART 2023-07-29T00:10:41.321 DEBUG:boto:path=/hiiqos-50/MULTIPART?uploads 2023-07-29T00:10:41.321 DEBUG:boto:auth_path=/hiiqos-50/MULTIPART?uploads 2023-07-29T00:10:41.322 DEBUG:boto:Method: POST 2023-07-29T00:10:41.322 DEBUG:boto:Path: /hiiqos-50/MULTIPART?uploads 2023-07-29T00:10:41.323 DEBUG:boto:Data: 2023-07-29T00:10:41.323 DEBUG:boto:Headers: {} 2023-07-29T00:10:41.323 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:41.324 DEBUG:boto:Port: 8000 2023-07-29T00:10:41.324 DEBUG:boto:Params: {} 2023-07-29T00:10:41.324 DEBUG:boto:Token: None 2023-07-29T00:10:41.325 DEBUG:boto:StringToSign: POST Sat, 29 Jul 2023 00:10:41 GMT /hiiqos-50/MULTIPART?uploads 2023-07-29T00:10:41.325 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gmYXM/uo+p6NNuZMgaWMNsMZ8QI= 2023-07-29T00:10:41.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gmYXM/uo+p6NNuZMgaWMNsMZ8QI=', 'Content-Length': '0'} 2023-07-29T00:10:42.493 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d6cf7b6444218561-0064c45901-165f-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '249'), ('Date', 'Sat, 29 Jul 2023 00:10:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:42.494 DEBUG:boto:b'hiiqos-50MULTIPART2~2V_VTfsjnq5bmSd4ZjNlcH3sTcBieGE' 2023-07-29T00:10:42.538 DEBUG:boto:path=/hiiqos-50/MULTIPART 2023-07-29T00:10:42.561 DEBUG:boto:auth_path=/hiiqos-50/MULTIPART 2023-07-29T00:10:42.562 DEBUG:boto:path=/hiiqos-50/MULTIPART?uploadId=2~2V_VTfsjnq5bmSd4ZjNlcH3sTcBieGE&partNumber=1 2023-07-29T00:10:42.562 DEBUG:boto:auth_path=/hiiqos-50/MULTIPART?uploadId=2~2V_VTfsjnq5bmSd4ZjNlcH3sTcBieGE&partNumber=1 2023-07-29T00:10:42.562 DEBUG:boto:Method: PUT 2023-07-29T00:10:42.563 DEBUG:boto:Path: /hiiqos-50/MULTIPART?uploadId=2~2V_VTfsjnq5bmSd4ZjNlcH3sTcBieGE&partNumber=1 2023-07-29T00:10:42.563 DEBUG:boto:Data: 2023-07-29T00:10:42.563 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-07-29T00:10:42.564 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:42.564 DEBUG:boto:Port: 8000 2023-07-29T00:10:42.564 DEBUG:boto:Params: {} 2023-07-29T00:10:42.565 DEBUG:boto:Token: None 2023-07-29T00:10:42.565 DEBUG:boto:StringToSign: PUT ebKBBg0ze5srhMzzkK3PdA== application/octet-stream Sat, 29 Jul 2023 00:10:42 GMT /hiiqos-50/MULTIPART?partNumber=1&uploadId=2~2V_VTfsjnq5bmSd4ZjNlcH3sTcBieGE 2023-07-29T00:10:42.566 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ykRi8dq8cce5YzE1U0uqNVGpMFA= 2023-07-29T00:10:42.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:10:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ykRi8dq8cce5YzE1U0uqNVGpMFA='} 2023-07-29T00:10:42.620 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"79b281060d337b9b2b84ccf390adcf74"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b4de19eaef7985a4-0064c45902-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:10:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:42.652 DEBUG:boto:path=/hiiqos-50/MULTIPART 2023-07-29T00:10:42.652 DEBUG:boto:auth_path=/hiiqos-50/MULTIPART 2023-07-29T00:10:42.653 DEBUG:boto:path=/hiiqos-50/MULTIPART?uploadId=2~2V_VTfsjnq5bmSd4ZjNlcH3sTcBieGE&partNumber=2 2023-07-29T00:10:42.653 DEBUG:boto:auth_path=/hiiqos-50/MULTIPART?uploadId=2~2V_VTfsjnq5bmSd4ZjNlcH3sTcBieGE&partNumber=2 2023-07-29T00:10:42.653 DEBUG:boto:Method: PUT 2023-07-29T00:10:42.654 DEBUG:boto:Path: /hiiqos-50/MULTIPART?uploadId=2~2V_VTfsjnq5bmSd4ZjNlcH3sTcBieGE&partNumber=2 2023-07-29T00:10:42.654 DEBUG:boto:Data: 2023-07-29T00:10:42.654 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-07-29T00:10:42.655 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:42.655 DEBUG:boto:Port: 8000 2023-07-29T00:10:42.655 DEBUG:boto:Params: {} 2023-07-29T00:10:42.656 DEBUG:boto:Token: None 2023-07-29T00:10:42.656 DEBUG:boto:StringToSign: PUT dIQ6OrGTo4m87YmUAtmdXw== application/octet-stream Sat, 29 Jul 2023 00:10:42 GMT /hiiqos-50/MULTIPART?partNumber=2&uploadId=2~2V_VTfsjnq5bmSd4ZjNlcH3sTcBieGE 2023-07-29T00:10:42.657 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0HevCukWW59pzMfCs5XInZZKWW0= 2023-07-29T00:10:42.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:10:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0HevCukWW59pzMfCs5XInZZKWW0='} 2023-07-29T00:10:42.703 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"74843a3ab193a389bced899402d99d5f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e8eb2c49aba82acb-0064c45902-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:10:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:42.721 DEBUG:boto:path=/hiiqos-50/MULTIPART 2023-07-29T00:10:42.722 DEBUG:boto:auth_path=/hiiqos-50/MULTIPART 2023-07-29T00:10:42.722 DEBUG:boto:path=/hiiqos-50/MULTIPART?uploadId=2~2V_VTfsjnq5bmSd4ZjNlcH3sTcBieGE&partNumber=3 2023-07-29T00:10:42.723 DEBUG:boto:auth_path=/hiiqos-50/MULTIPART?uploadId=2~2V_VTfsjnq5bmSd4ZjNlcH3sTcBieGE&partNumber=3 2023-07-29T00:10:42.723 DEBUG:boto:Method: PUT 2023-07-29T00:10:42.723 DEBUG:boto:Path: /hiiqos-50/MULTIPART?uploadId=2~2V_VTfsjnq5bmSd4ZjNlcH3sTcBieGE&partNumber=3 2023-07-29T00:10:42.724 DEBUG:boto:Data: 2023-07-29T00:10:42.724 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-07-29T00:10:42.724 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:42.725 DEBUG:boto:Port: 8000 2023-07-29T00:10:42.725 DEBUG:boto:Params: {} 2023-07-29T00:10:42.725 DEBUG:boto:Token: None 2023-07-29T00:10:42.726 DEBUG:boto:StringToSign: PUT e4RW4edMN49Fhh9TYZ51tg== application/octet-stream Sat, 29 Jul 2023 00:10:42 GMT /hiiqos-50/MULTIPART?partNumber=3&uploadId=2~2V_VTfsjnq5bmSd4ZjNlcH3sTcBieGE 2023-07-29T00:10:42.726 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:m0At3oRPYmaoqS8HJXVZqJbcLKw= 2023-07-29T00:10:42.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:10:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:m0At3oRPYmaoqS8HJXVZqJbcLKw='} 2023-07-29T00:10:42.740 INFO:tasks.rgw.c2.client.0.smithi139.stdout:2023-07-29T00:10:42.695+0000 7f5f2671b640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T00:10:42.772 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"7b8456e1e74c378f45861f53619e75b6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d527675ac74adc6f-0064c45902-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:10:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:42.790 DEBUG:boto:path=/hiiqos-50/MULTIPART 2023-07-29T00:10:42.791 DEBUG:boto:auth_path=/hiiqos-50/MULTIPART 2023-07-29T00:10:42.791 DEBUG:boto:path=/hiiqos-50/MULTIPART?uploadId=2~2V_VTfsjnq5bmSd4ZjNlcH3sTcBieGE&partNumber=4 2023-07-29T00:10:42.792 DEBUG:boto:auth_path=/hiiqos-50/MULTIPART?uploadId=2~2V_VTfsjnq5bmSd4ZjNlcH3sTcBieGE&partNumber=4 2023-07-29T00:10:42.792 DEBUG:boto:Method: PUT 2023-07-29T00:10:42.792 DEBUG:boto:Path: /hiiqos-50/MULTIPART?uploadId=2~2V_VTfsjnq5bmSd4ZjNlcH3sTcBieGE&partNumber=4 2023-07-29T00:10:42.793 DEBUG:boto:Data: 2023-07-29T00:10:42.793 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-07-29T00:10:42.793 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:42.794 DEBUG:boto:Port: 8000 2023-07-29T00:10:42.794 DEBUG:boto:Params: {} 2023-07-29T00:10:42.794 DEBUG:boto:Token: None 2023-07-29T00:10:42.795 DEBUG:boto:StringToSign: PUT l+jDi4yPcIlVvEiQwDQZTQ== application/octet-stream Sat, 29 Jul 2023 00:10:42 GMT /hiiqos-50/MULTIPART?partNumber=4&uploadId=2~2V_VTfsjnq5bmSd4ZjNlcH3sTcBieGE 2023-07-29T00:10:42.795 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O6JeHrqGSfr8cdGnkkP9B3eunME= 2023-07-29T00:10:42.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:10:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O6JeHrqGSfr8cdGnkkP9B3eunME='} 2023-07-29T00:10:42.841 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"97e8c38b8c8f708955bc4890c034194d"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009edc4651a3aab832-0064c45902-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:10:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:42.841 DEBUG:boto:path=/hiiqos-50/MULTIPART 2023-07-29T00:10:42.842 DEBUG:boto:auth_path=/hiiqos-50/MULTIPART 2023-07-29T00:10:42.842 DEBUG:boto:path=/hiiqos-50/MULTIPART?uploadId=2~2V_VTfsjnq5bmSd4ZjNlcH3sTcBieGE 2023-07-29T00:10:42.842 DEBUG:boto:auth_path=/hiiqos-50/MULTIPART?uploadId=2~2V_VTfsjnq5bmSd4ZjNlcH3sTcBieGE 2023-07-29T00:10:42.843 DEBUG:boto:Method: GET 2023-07-29T00:10:42.843 DEBUG:boto:Path: /hiiqos-50/MULTIPART?uploadId=2~2V_VTfsjnq5bmSd4ZjNlcH3sTcBieGE 2023-07-29T00:10:42.843 DEBUG:boto:Data: 2023-07-29T00:10:42.844 DEBUG:boto:Headers: {} 2023-07-29T00:10:42.844 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:42.844 DEBUG:boto:Port: 8000 2023-07-29T00:10:42.845 DEBUG:boto:Params: {} 2023-07-29T00:10:42.845 DEBUG:boto:Token: None 2023-07-29T00:10:42.845 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:10:42 GMT /hiiqos-50/MULTIPART?uploadId=2~2V_VTfsjnq5bmSd4ZjNlcH3sTcBieGE 2023-07-29T00:10:42.846 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4WRlpvb9+qT4NpCGDechQZ6bbmU= 2023-07-29T00:10:42.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4WRlpvb9+qT4NpCGDechQZ6bbmU=', 'Content-Length': '0'} 2023-07-29T00:10:42.848 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000316fba94f6c6e413-0064c45902-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:10:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:42.893 DEBUG:boto:path=/hiiqos-50/MULTIPART 2023-07-29T00:10:42.893 DEBUG:boto:auth_path=/hiiqos-50/MULTIPART 2023-07-29T00:10:42.894 DEBUG:boto:path=/hiiqos-50/MULTIPART?uploadId=2~2V_VTfsjnq5bmSd4ZjNlcH3sTcBieGE 2023-07-29T00:10:42.894 DEBUG:boto:auth_path=/hiiqos-50/MULTIPART?uploadId=2~2V_VTfsjnq5bmSd4ZjNlcH3sTcBieGE 2023-07-29T00:10:42.894 DEBUG:boto:Method: POST 2023-07-29T00:10:42.894 DEBUG:boto:Path: /hiiqos-50/MULTIPART?uploadId=2~2V_VTfsjnq5bmSd4ZjNlcH3sTcBieGE 2023-07-29T00:10:42.895 DEBUG:boto:Data: 1 "79b281060d337b9b2b84ccf390adcf74" 2 "74843a3ab193a389bced899402d99d5f" 3 "7b8456e1e74c378f45861f53619e75b6" 4 "97e8c38b8c8f708955bc4890c034194d" 2023-07-29T00:10:42.895 DEBUG:boto:Headers: {'Content-Type': 'text/xml'} 2023-07-29T00:10:42.895 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:10:42.896 DEBUG:boto:Port: 8000 2023-07-29T00:10:42.896 DEBUG:boto:Params: {} 2023-07-29T00:10:42.897 DEBUG:boto:Token: None 2023-07-29T00:10:42.897 DEBUG:boto:StringToSign: POST text/xml Sat, 29 Jul 2023 00:10:42 GMT /hiiqos-50/MULTIPART?uploadId=2~2V_VTfsjnq5bmSd4ZjNlcH3sTcBieGE 2023-07-29T00:10:42.897 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bgRBxAMYCuEYEs1Mu4ptIjW2ijA= 2023-07-29T00:10:42.898 DEBUG:boto:Final headers: {'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:10:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bgRBxAMYCuEYEs1Mu4ptIjW2ijA=', 'Content-Length': '460'} 2023-07-29T00:10:42.905 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005022d3187c17e760-0064c45902-165f-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '291'), ('Date', 'Sat, 29 Jul 2023 00:10:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:10:42.906 DEBUG:boto:http://smithi101.front.sepia.ceph.com:8000/hiiqos-50/MULTIPARThiiqos-50MULTIPART 2023-07-29T00:10:42.907 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:10:42.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:10:42.907 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:10:42.954 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:10:42.954 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:10:42.983 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T00:10:42.995 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T00:10:42.995 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:10:42.995 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:10:42.996 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:46:09.233742Z" 2023-07-29T00:10:42.996 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:10:42.996 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:10:42.996 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690589441.274978_771.1", 2023-07-29T00:10:42.996 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:10:41.274978Z" 2023-07-29T00:10:42.996 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:10:42.996 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:10:42.997 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:10:42.997 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T00:10:42.997 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:10:42.997 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:10:42.997 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:10:42.997 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T00:10:42.997 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:10:42.998 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T00:10:42.998 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690587969.233742_99.1', 1: '1_1690589441.274978_771.1', 2: '', 3: ''} 2023-07-29T00:10:42.999 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T00:10:42.999 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:10:43.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:10:43.000 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:10:43.047 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:10:43.047 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:10:43.076 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:10:43.042+0000 7f293382e880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:10:43.077 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:10:43.043+0000 7f293382e880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:10:43.089 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:10:43.090 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:10:43.090 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:10:43.090 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:10:43.090 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:10:43.090 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:10:43.090 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:10:43.091 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:10:43.091 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:10:43.091 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:10:43.091 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:10:43.091 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:10:43.091 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:10:43.091 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:10:43.092 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:10:43.092 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:10:43.092 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:10:43.092 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:10:43.092 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:10:43.092 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:10:43.093 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:10:43.093 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:10:43.093 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:10:43.093 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:10:43.093 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:10:43.093 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690589441.274978_771.1", 2023-07-29T00:10:43.093 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:10:43.094 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:10:43.094 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:10:43.094 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:10:41.274978Z", 2023-07-29T00:10:43.094 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:10:43.094 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:10:43.094 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:10:43.094 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:10:43.095 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:10:43.095 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:10:43.095 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:10:43.095 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:10:43.095 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:10:43.095 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:10:43.096 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:10:43.096 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:10:43.096 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:10:43.096 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:10:43.096 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:10:43.097 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:10:43.097 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:10:43.097 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:10:43.097 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:10:43.097 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:10:43.097 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:10:43.098 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:10:43.098 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:10:43.098 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:10:43.098 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:10:43.098 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:10:43.098 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:10:43.098 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:10:43.099 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:10:43.099 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:10:43.099 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:10:43.099 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:10:43.099 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:10:43.099 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:10:43Z" 2023-07-29T00:10:43.099 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:10:43.100 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690589441.274978_771.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:10:41.274978Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:10:43Z" } 2023-07-29T00:10:43.101 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690589441.274978_771.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:10:41.274978Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:10:43.101 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690589441.274978_771.1', 2: '', 3: ''} 2023-07-29T00:10:43.102 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690589441.274978_771.1', 2: '', 3: ''} 2023-07-29T00:10:43.102 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T00:10:43.102 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-50 2023-07-29T00:10:43.103 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-50', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:10:43.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-50', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:10:43.103 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-50 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:10:43.156 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:10:43.156 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:10:43.189 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:10:43.155+0000 7fd6c7135880 1 waiting to reach incremental sync.. 2023-07-29T00:10:48.190 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:10:48.156+0000 7fd6c7135880 1 waiting to reach incremental sync.. 2023-07-29T00:10:53.190 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:10:53.157+0000 7fd6c7135880 1 waiting to reach incremental sync.. 2023-07-29T00:10:58.192 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:10:58.158+0000 7fd6c7135880 1 bucket sync caught up with source: 2023-07-29T00:10:58.193 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000011.47.6, , , , , ] 2023-07-29T00:10:58.193 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000011.47.6, , , , , ] 2023-07-29T00:10:58.193 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:10:58.158+0000 7fd6c7135880 0 bucket checkpoint complete 2023-07-29T00:10:58.210 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-50 2023-07-29T00:10:58.210 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-50', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:10:58.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-50', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:10:58.211 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-50 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:10:58.270 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:10:58.271 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:10:58.306 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:10:58.271+0000 7efebcf26880 1 waiting to reach incremental sync.. 2023-07-29T00:11:03.306 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:11:03.272+0000 7efebcf26880 1 waiting to reach incremental sync.. 2023-07-29T00:11:08.307 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:11:08.272+0000 7efebcf26880 1 waiting to reach incremental sync.. 2023-07-29T00:11:13.307 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:11:13.273+0000 7efebcf26880 1 waiting to reach incremental sync.. 2023-07-29T00:11:14.688 INFO:tasks.rgw.c1.client.0.smithi101.stdout:2023-07-29T00:11:14.653+0000 7f0081a5e640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T00:11:18.308 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:11:18.274+0000 7efebcf26880 1 waiting to reach incremental sync.. 2023-07-29T00:11:23.309 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:11:23.274+0000 7efebcf26880 1 waiting to reach incremental sync.. 2023-07-29T00:11:28.310 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:11:28.275+0000 7efebcf26880 1 waiting to reach incremental sync.. 2023-07-29T00:11:33.311 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:11:33.276+0000 7efebcf26880 1 waiting to reach incremental sync.. 2023-07-29T00:11:38.312 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:11:38.276+0000 7efebcf26880 1 waiting to reach incremental sync.. 2023-07-29T00:11:43.312 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:11:43.277+0000 7efebcf26880 1 waiting to reach incremental sync.. 2023-07-29T00:11:48.313 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:11:48.277+0000 7efebcf26880 1 waiting to reach incremental sync.. 2023-07-29T00:11:53.314 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:11:53.278+0000 7efebcf26880 1 waiting to reach incremental sync.. 2023-07-29T00:11:58.314 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:11:58.278+0000 7efebcf26880 1 waiting to reach incremental sync.. 2023-07-29T00:12:03.315 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:12:03.279+0000 7efebcf26880 1 waiting to reach incremental sync.. 2023-07-29T00:12:08.316 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:12:08.280+0000 7efebcf26880 1 waiting to reach incremental sync.. 2023-07-29T00:12:13.316 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:12:13.280+0000 7efebcf26880 1 waiting to reach incremental sync.. 2023-07-29T00:12:18.318 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:12:18.281+0000 7efebcf26880 1 waiting to reach incremental sync.. 2023-07-29T00:12:23.318 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:12:23.282+0000 7efebcf26880 1 waiting to reach incremental sync.. 2023-07-29T00:12:28.319 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:12:28.282+0000 7efebcf26880 1 waiting to reach incremental sync.. 2023-07-29T00:12:33.320 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:12:33.283+0000 7efebcf26880 1 waiting to reach incremental sync.. 2023-07-29T00:12:38.320 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:12:38.283+0000 7efebcf26880 1 waiting to reach incremental sync.. 2023-07-29T00:12:43.321 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:12:43.284+0000 7efebcf26880 1 waiting to reach incremental sync.. 2023-07-29T00:12:48.322 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:12:48.285+0000 7efebcf26880 1 waiting to reach incremental sync.. 2023-07-29T00:12:53.323 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:12:53.285+0000 7efebcf26880 1 waiting to reach incremental sync.. 2023-07-29T00:12:58.325 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:12:58.287+0000 7efebcf26880 1 bucket sync caught up with source: 2023-07-29T00:12:58.325 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000001.702.6, , , , , ] 2023-07-29T00:12:58.325 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000001.702.6, , , , , ] 2023-07-29T00:12:58.325 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:12:58.287+0000 7efebcf26880 0 bucket checkpoint complete 2023-07-29T00:12:58.341 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-50 zones={a2, a1} 2023-07-29T00:12:58.342 DEBUG:boto:path=/hiiqos-50/ 2023-07-29T00:12:58.343 DEBUG:boto:auth_path=/hiiqos-50/ 2023-07-29T00:12:58.344 DEBUG:boto:Method: HEAD 2023-07-29T00:12:58.345 DEBUG:boto:Path: /hiiqos-50/ 2023-07-29T00:12:58.346 DEBUG:boto:Data: 2023-07-29T00:12:58.347 DEBUG:boto:Headers: {} 2023-07-29T00:12:58.347 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:12:58.348 DEBUG:boto:Port: 8001 2023-07-29T00:12:58.349 DEBUG:boto:Params: {} 2023-07-29T00:12:58.350 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-29T00:12:58.350 DEBUG:boto:Token: None 2023-07-29T00:12:58.351 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:12:58 GMT /hiiqos-50/ 2023-07-29T00:12:58.352 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:NX4W+wrfj19P8d2y+I7n8W0IU4o= 2023-07-29T00:12:58.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:12:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:NX4W+wrfj19P8d2y+I7n8W0IU4o=', 'Content-Length': '0'} 2023-07-29T00:12:58.356 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '20971520'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000633d410d88006470-0064c4598a-1663-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:12:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:12:58.357 DEBUG:boto:path=/hiiqos-50/ 2023-07-29T00:12:58.357 DEBUG:boto:auth_path=/hiiqos-50/ 2023-07-29T00:12:58.358 DEBUG:boto:Method: HEAD 2023-07-29T00:12:58.358 DEBUG:boto:Path: /hiiqos-50/ 2023-07-29T00:12:58.358 DEBUG:boto:Data: 2023-07-29T00:12:58.359 DEBUG:boto:Headers: {} 2023-07-29T00:12:58.359 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:12:58.359 DEBUG:boto:Port: 8000 2023-07-29T00:12:58.360 DEBUG:boto:Params: {} 2023-07-29T00:12:58.360 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-29T00:12:58.361 DEBUG:boto:Token: None 2023-07-29T00:12:58.361 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:12:58 GMT /hiiqos-50/ 2023-07-29T00:12:58.361 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:NX4W+wrfj19P8d2y+I7n8W0IU4o= 2023-07-29T00:12:58.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:12:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:NX4W+wrfj19P8d2y+I7n8W0IU4o=', 'Content-Length': '0'} 2023-07-29T00:12:58.365 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '20971520'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092de284a22d7e594-0064c4598a-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:12:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:12:58.365 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-29T00:12:58.366 DEBUG:boto:path=/hiiqos-50/ 2023-07-29T00:12:58.366 DEBUG:boto:auth_path=/hiiqos-50/ 2023-07-29T00:12:58.366 DEBUG:boto:path=/hiiqos-50/?versions&max-keys=999 2023-07-29T00:12:58.367 DEBUG:boto:auth_path=/hiiqos-50/?versions&max-keys=999 2023-07-29T00:12:58.367 DEBUG:boto:Method: GET 2023-07-29T00:12:58.367 DEBUG:boto:Path: /hiiqos-50/?versions&max-keys=999 2023-07-29T00:12:58.368 DEBUG:boto:Data: 2023-07-29T00:12:58.368 DEBUG:boto:Headers: {} 2023-07-29T00:12:58.369 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:12:58.369 DEBUG:boto:Port: 8001 2023-07-29T00:12:58.369 DEBUG:boto:Params: {} 2023-07-29T00:12:58.370 DEBUG:boto:Token: None 2023-07-29T00:12:58.370 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:12:58 GMT /hiiqos-50/?versions 2023-07-29T00:12:58.371 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2BgBqsYnWgh+g6IJK6jjlOBcR1Y= 2023-07-29T00:12:58.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:12:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2BgBqsYnWgh+g6IJK6jjlOBcR1Y=', 'Content-Length': '0'} 2023-07-29T00:12:58.373 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf94b58399e227ec-0064c4598a-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:12:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:12:58.374 DEBUG:boto:b'hiiqos-50999falseMULTIPARTnulltrue2023-07-29T00:10:42.868Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:12:58.375 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2023-07-29T00:12:58.376 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-29T00:12:58.376 DEBUG:boto:path=/hiiqos-50/ 2023-07-29T00:12:58.376 DEBUG:boto:auth_path=/hiiqos-50/ 2023-07-29T00:12:58.377 DEBUG:boto:path=/hiiqos-50/?versions&max-keys=999 2023-07-29T00:12:58.377 DEBUG:boto:auth_path=/hiiqos-50/?versions&max-keys=999 2023-07-29T00:12:58.377 DEBUG:boto:Method: GET 2023-07-29T00:12:58.378 DEBUG:boto:Path: /hiiqos-50/?versions&max-keys=999 2023-07-29T00:12:58.378 DEBUG:boto:Data: 2023-07-29T00:12:58.378 DEBUG:boto:Headers: {} 2023-07-29T00:12:58.379 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:12:58.379 DEBUG:boto:Port: 8000 2023-07-29T00:12:58.379 DEBUG:boto:Params: {} 2023-07-29T00:12:58.380 DEBUG:boto:Token: None 2023-07-29T00:12:58.380 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:12:58 GMT /hiiqos-50/?versions 2023-07-29T00:12:58.381 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2BgBqsYnWgh+g6IJK6jjlOBcR1Y= 2023-07-29T00:12:58.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:12:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2BgBqsYnWgh+g6IJK6jjlOBcR1Y=', 'Content-Length': '0'} 2023-07-29T00:12:58.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007133de7143d41dab-0064c4598a-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:12:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:12:58.384 DEBUG:boto:b'hiiqos-50999falseMULTIPARTnulltrue2023-07-29T00:10:42.868Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:12:58.385 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2023-07-29T00:12:58.385 DEBUG:boto:path=/hiiqos-50/ 2023-07-29T00:12:58.386 DEBUG:boto:auth_path=/hiiqos-50/ 2023-07-29T00:12:58.386 DEBUG:boto:path=/hiiqos-50/?versions&max-keys=999 2023-07-29T00:12:58.386 DEBUG:boto:auth_path=/hiiqos-50/?versions&max-keys=999 2023-07-29T00:12:58.387 DEBUG:boto:Method: GET 2023-07-29T00:12:58.387 DEBUG:boto:Path: /hiiqos-50/?versions&max-keys=999 2023-07-29T00:12:58.387 DEBUG:boto:Data: 2023-07-29T00:12:58.388 DEBUG:boto:Headers: {} 2023-07-29T00:12:58.388 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:12:58.389 DEBUG:boto:Port: 8001 2023-07-29T00:12:58.389 DEBUG:boto:Params: {} 2023-07-29T00:12:58.389 DEBUG:boto:Token: None 2023-07-29T00:12:58.390 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:12:58 GMT /hiiqos-50/?versions 2023-07-29T00:12:58.390 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2BgBqsYnWgh+g6IJK6jjlOBcR1Y= 2023-07-29T00:12:58.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:12:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2BgBqsYnWgh+g6IJK6jjlOBcR1Y=', 'Content-Length': '0'} 2023-07-29T00:12:58.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0fa201ec6c03abf-0064c4598a-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:12:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:12:58.394 DEBUG:boto:b'hiiqos-50999falseMULTIPARTnulltrue2023-07-29T00:10:42.868Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:12:58.394 DEBUG:boto:path=/hiiqos-50/ 2023-07-29T00:12:58.395 DEBUG:boto:auth_path=/hiiqos-50/ 2023-07-29T00:12:58.395 DEBUG:boto:path=/hiiqos-50/?versions&max-keys=999 2023-07-29T00:12:58.395 DEBUG:boto:auth_path=/hiiqos-50/?versions&max-keys=999 2023-07-29T00:12:58.396 DEBUG:boto:Method: GET 2023-07-29T00:12:58.396 DEBUG:boto:Path: /hiiqos-50/?versions&max-keys=999 2023-07-29T00:12:58.396 DEBUG:boto:Data: 2023-07-29T00:12:58.397 DEBUG:boto:Headers: {} 2023-07-29T00:12:58.397 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:12:58.397 DEBUG:boto:Port: 8000 2023-07-29T00:12:58.398 DEBUG:boto:Params: {} 2023-07-29T00:12:58.398 DEBUG:boto:Token: None 2023-07-29T00:12:58.399 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:12:58 GMT /hiiqos-50/?versions 2023-07-29T00:12:58.399 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2BgBqsYnWgh+g6IJK6jjlOBcR1Y= 2023-07-29T00:12:58.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:12:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2BgBqsYnWgh+g6IJK6jjlOBcR1Y=', 'Content-Length': '0'} 2023-07-29T00:12:58.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068e6f96484643865-0064c4598a-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:12:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:12:58.402 DEBUG:boto:b'hiiqos-50999falseMULTIPARTnulltrue2023-07-29T00:10:42.868Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:12:58.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2023-07-29T00:12:58.403 DEBUG:boto:path=/hiiqos-50/MULTIPART 2023-07-29T00:12:58.404 DEBUG:boto:auth_path=/hiiqos-50/MULTIPART 2023-07-29T00:12:58.404 DEBUG:boto:path=/hiiqos-50/MULTIPART?versionId=null 2023-07-29T00:12:58.405 DEBUG:boto:auth_path=/hiiqos-50/MULTIPART?versionId=null 2023-07-29T00:12:58.405 DEBUG:boto:Method: GET 2023-07-29T00:12:58.405 DEBUG:boto:Path: /hiiqos-50/MULTIPART?versionId=null 2023-07-29T00:12:58.406 DEBUG:boto:Data: 2023-07-29T00:12:58.406 DEBUG:boto:Headers: {} 2023-07-29T00:12:58.406 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:12:58.407 DEBUG:boto:Port: 8001 2023-07-29T00:12:58.407 DEBUG:boto:Params: {} 2023-07-29T00:12:58.407 DEBUG:boto:Token: None 2023-07-29T00:12:58.408 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:12:58 GMT /hiiqos-50/MULTIPART?versionId=null 2023-07-29T00:12:58.408 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/yuFTQUqaGDDf4B+FpeBHKnrdLY= 2023-07-29T00:12:58.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:12:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/yuFTQUqaGDDf4B+FpeBHKnrdLY=', 'Content-Length': '0'} 2023-07-29T00:12:58.413 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:10:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-50:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000e2d9053ca06d33cf-0064c4598a-1663-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:12:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:12:58.483 DEBUG:boto:path=/hiiqos-50/MULTIPART 2023-07-29T00:12:58.484 DEBUG:boto:auth_path=/hiiqos-50/MULTIPART 2023-07-29T00:12:58.484 DEBUG:boto:path=/hiiqos-50/MULTIPART?versionId=null 2023-07-29T00:12:58.485 DEBUG:boto:auth_path=/hiiqos-50/MULTIPART?versionId=null 2023-07-29T00:12:58.485 DEBUG:boto:Method: GET 2023-07-29T00:12:58.485 DEBUG:boto:Path: /hiiqos-50/MULTIPART?versionId=null 2023-07-29T00:12:58.486 DEBUG:boto:Data: 2023-07-29T00:12:58.486 DEBUG:boto:Headers: {} 2023-07-29T00:12:58.486 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:12:58.487 DEBUG:boto:Port: 8000 2023-07-29T00:12:58.487 DEBUG:boto:Params: {} 2023-07-29T00:12:58.488 DEBUG:boto:Token: None 2023-07-29T00:12:58.488 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:12:58 GMT /hiiqos-50/MULTIPART?versionId=null 2023-07-29T00:12:58.489 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/yuFTQUqaGDDf4B+FpeBHKnrdLY= 2023-07-29T00:12:58.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:12:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/yuFTQUqaGDDf4B+FpeBHKnrdLY=', 'Content-Length': '0'} 2023-07-29T00:12:58.498 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:10:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000159c05ef96239d8b-0064c4598a-165f-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:12:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:12:58.584 DEBUG:boto:path=/hiiqos-50/MULTIPART 2023-07-29T00:12:58.585 DEBUG:boto:auth_path=/hiiqos-50/MULTIPART 2023-07-29T00:12:58.585 DEBUG:boto:path=/hiiqos-50/MULTIPART?versionId=null 2023-07-29T00:12:58.586 DEBUG:boto:auth_path=/hiiqos-50/MULTIPART?versionId=null 2023-07-29T00:12:58.586 DEBUG:boto:Method: HEAD 2023-07-29T00:12:58.586 DEBUG:boto:Path: /hiiqos-50/MULTIPART?versionId=null 2023-07-29T00:12:58.587 DEBUG:boto:Data: 2023-07-29T00:12:58.587 DEBUG:boto:Headers: {} 2023-07-29T00:12:58.587 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:12:58.588 DEBUG:boto:Port: 8001 2023-07-29T00:12:58.588 DEBUG:boto:Params: {} 2023-07-29T00:12:58.588 DEBUG:boto:Token: None 2023-07-29T00:12:58.589 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:12:58 GMT /hiiqos-50/MULTIPART?versionId=null 2023-07-29T00:12:58.589 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:s+24Vq5izcuvOrADw2PxgZh3aeg= 2023-07-29T00:12:58.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:12:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:s+24Vq5izcuvOrADw2PxgZh3aeg=', 'Content-Length': '0'} 2023-07-29T00:12:58.592 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:10:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-50:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000004b86beb14889063a-0064c4598a-1663-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:12:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:12:58.593 DEBUG:boto:path=/hiiqos-50/MULTIPART 2023-07-29T00:12:58.593 DEBUG:boto:auth_path=/hiiqos-50/MULTIPART 2023-07-29T00:12:58.593 DEBUG:boto:path=/hiiqos-50/MULTIPART?versionId=null 2023-07-29T00:12:58.594 DEBUG:boto:auth_path=/hiiqos-50/MULTIPART?versionId=null 2023-07-29T00:12:58.594 DEBUG:boto:Method: HEAD 2023-07-29T00:12:58.595 DEBUG:boto:Path: /hiiqos-50/MULTIPART?versionId=null 2023-07-29T00:12:58.595 DEBUG:boto:Data: 2023-07-29T00:12:58.595 DEBUG:boto:Headers: {} 2023-07-29T00:12:58.596 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:12:58.596 DEBUG:boto:Port: 8000 2023-07-29T00:12:58.596 DEBUG:boto:Params: {} 2023-07-29T00:12:58.597 DEBUG:boto:Token: None 2023-07-29T00:12:58.597 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:12:58 GMT /hiiqos-50/MULTIPART?versionId=null 2023-07-29T00:12:58.597 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:s+24Vq5izcuvOrADw2PxgZh3aeg= 2023-07-29T00:12:58.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:12:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:s+24Vq5izcuvOrADw2PxgZh3aeg=', 'Content-Length': '0'} 2023-07-29T00:12:58.600 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:10:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000206268f9382a677d-0064c4598a-165f-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:12:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:12:58.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2023-07-29T00:12:58.601 DEBUG:boto:path=/hiiqos-50/MULTIPART 2023-07-29T00:12:58.601 DEBUG:boto:auth_path=/hiiqos-50/MULTIPART 2023-07-29T00:12:58.601 DEBUG:boto:path=/hiiqos-50/MULTIPART?versionId=null 2023-07-29T00:12:58.602 DEBUG:boto:auth_path=/hiiqos-50/MULTIPART?versionId=null 2023-07-29T00:12:58.602 DEBUG:boto:Method: GET 2023-07-29T00:12:58.603 DEBUG:boto:Path: /hiiqos-50/MULTIPART?versionId=null 2023-07-29T00:12:58.603 DEBUG:boto:Data: 2023-07-29T00:12:58.603 DEBUG:boto:Headers: {} 2023-07-29T00:12:58.604 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:12:58.604 DEBUG:boto:Port: 8001 2023-07-29T00:12:58.604 DEBUG:boto:Params: {} 2023-07-29T00:12:58.605 DEBUG:boto:Token: None 2023-07-29T00:12:58.605 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:12:58 GMT /hiiqos-50/MULTIPART?versionId=null 2023-07-29T00:12:58.606 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/yuFTQUqaGDDf4B+FpeBHKnrdLY= 2023-07-29T00:12:58.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:12:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/yuFTQUqaGDDf4B+FpeBHKnrdLY=', 'Content-Length': '0'} 2023-07-29T00:12:58.609 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:10:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-50:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000972619082fd4fc8a-0064c4598a-1663-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:12:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:12:58.701 DEBUG:boto:path=/hiiqos-50/MULTIPART 2023-07-29T00:12:58.702 DEBUG:boto:auth_path=/hiiqos-50/MULTIPART 2023-07-29T00:12:58.703 DEBUG:boto:path=/hiiqos-50/MULTIPART?versionId=null 2023-07-29T00:12:58.704 DEBUG:boto:auth_path=/hiiqos-50/MULTIPART?versionId=null 2023-07-29T00:12:58.705 DEBUG:boto:Method: GET 2023-07-29T00:12:58.706 DEBUG:boto:Path: /hiiqos-50/MULTIPART?versionId=null 2023-07-29T00:12:58.706 DEBUG:boto:Data: 2023-07-29T00:12:58.707 DEBUG:boto:Headers: {} 2023-07-29T00:12:58.708 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:12:58.708 DEBUG:boto:Port: 8000 2023-07-29T00:12:58.709 DEBUG:boto:Params: {} 2023-07-29T00:12:58.710 DEBUG:boto:Token: None 2023-07-29T00:12:58.710 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:12:58 GMT /hiiqos-50/MULTIPART?versionId=null 2023-07-29T00:12:58.711 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/yuFTQUqaGDDf4B+FpeBHKnrdLY= 2023-07-29T00:12:58.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:12:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/yuFTQUqaGDDf4B+FpeBHKnrdLY=', 'Content-Length': '0'} 2023-07-29T00:12:58.720 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:10:42 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000009e15244303ad320f-0064c4598a-165f-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:12:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:12:58.813 DEBUG:boto:path=/hiiqos-50/ 2023-07-29T00:12:58.813 DEBUG:boto:auth_path=/hiiqos-50/ 2023-07-29T00:12:58.814 DEBUG:boto:path=/hiiqos-50/?prefix=MULTIPART 2023-07-29T00:12:58.814 DEBUG:boto:auth_path=/hiiqos-50/?prefix=MULTIPART 2023-07-29T00:12:58.814 DEBUG:boto:Method: GET 2023-07-29T00:12:58.815 DEBUG:boto:Path: /hiiqos-50/?prefix=MULTIPART 2023-07-29T00:12:58.815 DEBUG:boto:Data: 2023-07-29T00:12:58.815 DEBUG:boto:Headers: {} 2023-07-29T00:12:58.816 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:12:58.816 DEBUG:boto:Port: 8001 2023-07-29T00:12:58.817 DEBUG:boto:Params: {} 2023-07-29T00:12:58.817 DEBUG:boto:Token: None 2023-07-29T00:12:58.817 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:12:58 GMT /hiiqos-50/ 2023-07-29T00:12:58.818 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Xcv9Lz5/qPzEOIAuzp8he1i1jL4= 2023-07-29T00:12:58.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:12:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Xcv9Lz5/qPzEOIAuzp8he1i1jL4=', 'Content-Length': '0'} 2023-07-29T00:12:58.821 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b76f7bc79293a783-0064c4598a-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:12:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:12:58.821 DEBUG:boto:b'hiiqos-50MULTIPART1000falseMULTIPART2023-07-29T00:10:42.868Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:12:58.822 DEBUG:boto:path=/hiiqos-50/ 2023-07-29T00:12:58.822 DEBUG:boto:auth_path=/hiiqos-50/ 2023-07-29T00:12:58.823 DEBUG:boto:path=/hiiqos-50/?prefix=MULTIPART 2023-07-29T00:12:58.823 DEBUG:boto:auth_path=/hiiqos-50/?prefix=MULTIPART 2023-07-29T00:12:58.824 DEBUG:boto:Method: GET 2023-07-29T00:12:58.824 DEBUG:boto:Path: /hiiqos-50/?prefix=MULTIPART 2023-07-29T00:12:58.824 DEBUG:boto:Data: 2023-07-29T00:12:58.825 DEBUG:boto:Headers: {} 2023-07-29T00:12:58.825 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:12:58.826 DEBUG:boto:Port: 8000 2023-07-29T00:12:58.826 DEBUG:boto:Params: {} 2023-07-29T00:12:58.826 DEBUG:boto:Token: None 2023-07-29T00:12:58.827 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:12:58 GMT /hiiqos-50/ 2023-07-29T00:12:58.827 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Xcv9Lz5/qPzEOIAuzp8he1i1jL4= 2023-07-29T00:12:58.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:12:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Xcv9Lz5/qPzEOIAuzp8he1i1jL4=', 'Content-Length': '0'} 2023-07-29T00:12:58.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9e265a0d5763385-0064c4598a-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:12:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:12:58.831 DEBUG:boto:b'hiiqos-50MULTIPART1000falseMULTIPART2023-07-29T00:10:42.868Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:12:58.831 DEBUG:boto:path=/hiiqos-50/MULTIPART 2023-07-29T00:12:58.832 DEBUG:boto:auth_path=/hiiqos-50/MULTIPART 2023-07-29T00:12:58.832 DEBUG:boto:Method: HEAD 2023-07-29T00:12:58.832 DEBUG:boto:Path: /hiiqos-50/MULTIPART 2023-07-29T00:12:58.833 DEBUG:boto:Data: 2023-07-29T00:12:58.833 DEBUG:boto:Headers: {} 2023-07-29T00:12:58.833 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:12:58.834 DEBUG:boto:Port: 8001 2023-07-29T00:12:58.834 DEBUG:boto:Params: {} 2023-07-29T00:12:58.834 DEBUG:boto:Token: None 2023-07-29T00:12:58.835 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:12:58 GMT /hiiqos-50/MULTIPART 2023-07-29T00:12:58.835 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vWV2CJPQbmb+9ZAwj3XA4HqcQKo= 2023-07-29T00:12:58.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:12:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vWV2CJPQbmb+9ZAwj3XA4HqcQKo=', 'Content-Length': '0'} 2023-07-29T00:12:58.838 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:10:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-50:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000002cc37ea57a51fd8d-0064c4598a-1663-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:12:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:12:58.838 DEBUG:boto:path=/hiiqos-50/MULTIPART 2023-07-29T00:12:58.839 DEBUG:boto:auth_path=/hiiqos-50/MULTIPART 2023-07-29T00:12:58.839 DEBUG:boto:Method: HEAD 2023-07-29T00:12:58.839 DEBUG:boto:Path: /hiiqos-50/MULTIPART 2023-07-29T00:12:58.840 DEBUG:boto:Data: 2023-07-29T00:12:58.840 DEBUG:boto:Headers: {} 2023-07-29T00:12:58.840 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:12:58.841 DEBUG:boto:Port: 8000 2023-07-29T00:12:58.841 DEBUG:boto:Params: {} 2023-07-29T00:12:58.842 DEBUG:boto:Token: None 2023-07-29T00:12:58.842 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:12:58 GMT /hiiqos-50/MULTIPART 2023-07-29T00:12:58.842 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vWV2CJPQbmb+9ZAwj3XA4HqcQKo= 2023-07-29T00:12:58.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:12:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vWV2CJPQbmb+9ZAwj3XA4HqcQKo=', 'Content-Length': '0'} 2023-07-29T00:12:58.846 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:10:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000aa4e650a5860506a-0064c4598a-165f-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:12:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:12:58.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2023-07-29T00:12:58.847 DEBUG:boto:path=/hiiqos-50/MULTIPART 2023-07-29T00:12:58.848 DEBUG:boto:auth_path=/hiiqos-50/MULTIPART 2023-07-29T00:12:58.849 DEBUG:boto:Method: GET 2023-07-29T00:12:58.850 DEBUG:boto:Path: /hiiqos-50/MULTIPART 2023-07-29T00:12:58.850 DEBUG:boto:Data: 2023-07-29T00:12:58.851 DEBUG:boto:Headers: {} 2023-07-29T00:12:58.852 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:12:58.852 DEBUG:boto:Port: 8001 2023-07-29T00:12:58.853 DEBUG:boto:Params: {} 2023-07-29T00:12:58.853 DEBUG:boto:Token: None 2023-07-29T00:12:58.854 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:12:58 GMT /hiiqos-50/MULTIPART 2023-07-29T00:12:58.854 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1zHUMHxtjehp4AG4VDy7cTUlUas= 2023-07-29T00:12:58.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:12:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1zHUMHxtjehp4AG4VDy7cTUlUas=', 'Content-Length': '0'} 2023-07-29T00:12:58.858 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:10:42 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-50:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000ee4d270a5d4b6f1b-0064c4598a-1663-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:12:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:12:58.936 DEBUG:boto:path=/hiiqos-50/MULTIPART 2023-07-29T00:12:58.937 DEBUG:boto:auth_path=/hiiqos-50/MULTIPART 2023-07-29T00:12:58.937 DEBUG:boto:Method: GET 2023-07-29T00:12:58.938 DEBUG:boto:Path: /hiiqos-50/MULTIPART 2023-07-29T00:12:58.938 DEBUG:boto:Data: 2023-07-29T00:12:58.938 DEBUG:boto:Headers: {} 2023-07-29T00:12:58.939 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:12:58.939 DEBUG:boto:Port: 8000 2023-07-29T00:12:58.939 DEBUG:boto:Params: {} 2023-07-29T00:12:58.940 DEBUG:boto:Token: None 2023-07-29T00:12:58.940 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:12:58 GMT /hiiqos-50/MULTIPART 2023-07-29T00:12:58.941 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1zHUMHxtjehp4AG4VDy7cTUlUas= 2023-07-29T00:12:58.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:12:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1zHUMHxtjehp4AG4VDy7cTUlUas=', 'Content-Length': '0'} 2023-07-29T00:12:58.950 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:10:42 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000f60a6d3274e8369b-0064c4598a-165f-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:12:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:12:59.040 DEBUG:boto:path=/hiiqos-50/ 2023-07-29T00:12:59.040 DEBUG:boto:auth_path=/hiiqos-50/ 2023-07-29T00:12:59.041 DEBUG:boto:path=/hiiqos-50/?prefix=MULTIPART 2023-07-29T00:12:59.041 DEBUG:boto:auth_path=/hiiqos-50/?prefix=MULTIPART 2023-07-29T00:12:59.042 DEBUG:boto:Method: GET 2023-07-29T00:12:59.042 DEBUG:boto:Path: /hiiqos-50/?prefix=MULTIPART 2023-07-29T00:12:59.042 DEBUG:boto:Data: 2023-07-29T00:12:59.043 DEBUG:boto:Headers: {} 2023-07-29T00:12:59.043 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:12:59.043 DEBUG:boto:Port: 8001 2023-07-29T00:12:59.044 DEBUG:boto:Params: {} 2023-07-29T00:12:59.044 DEBUG:boto:Token: None 2023-07-29T00:12:59.045 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:12:59 GMT /hiiqos-50/ 2023-07-29T00:12:59.045 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HFHF0Bof5GUrQijI74qFllSNrlg= 2023-07-29T00:12:59.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:12:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HFHF0Bof5GUrQijI74qFllSNrlg=', 'Content-Length': '0'} 2023-07-29T00:12:59.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca9113a6cb0141e5-0064c4598b-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:12:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:12:59.049 DEBUG:boto:b'hiiqos-50MULTIPART1000falseMULTIPART2023-07-29T00:10:42.868Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:12:59.050 DEBUG:boto:path=/hiiqos-50/ 2023-07-29T00:12:59.050 DEBUG:boto:auth_path=/hiiqos-50/ 2023-07-29T00:12:59.051 DEBUG:boto:path=/hiiqos-50/?prefix=MULTIPART 2023-07-29T00:12:59.051 DEBUG:boto:auth_path=/hiiqos-50/?prefix=MULTIPART 2023-07-29T00:12:59.052 DEBUG:boto:Method: GET 2023-07-29T00:12:59.052 DEBUG:boto:Path: /hiiqos-50/?prefix=MULTIPART 2023-07-29T00:12:59.053 DEBUG:boto:Data: 2023-07-29T00:12:59.053 DEBUG:boto:Headers: {} 2023-07-29T00:12:59.054 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:12:59.055 DEBUG:boto:Port: 8000 2023-07-29T00:12:59.056 DEBUG:boto:Params: {} 2023-07-29T00:12:59.057 DEBUG:boto:Token: None 2023-07-29T00:12:59.058 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:12:59 GMT /hiiqos-50/ 2023-07-29T00:12:59.059 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HFHF0Bof5GUrQijI74qFllSNrlg= 2023-07-29T00:12:59.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:12:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HFHF0Bof5GUrQijI74qFllSNrlg=', 'Content-Length': '0'} 2023-07-29T00:12:59.061 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf79a2b5976efc34-0064c4598b-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:12:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:12:59.062 DEBUG:boto:b'hiiqos-50MULTIPART1000falseMULTIPART2023-07-29T00:10:42.868Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:12:59.062 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-50 zones={a2, a1} 2023-07-29T00:12:59.063 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multipart_object_sync ... ok 2023-07-29T00:12:59.064 INFO:rgw_multi.tests:create bucket zone=a1 name=hiiqos-52 2023-07-29T00:12:59.064 DEBUG:boto:path=/hiiqos-52/ 2023-07-29T00:12:59.064 DEBUG:boto:auth_path=/hiiqos-52/ 2023-07-29T00:12:59.065 DEBUG:boto:Method: PUT 2023-07-29T00:12:59.065 DEBUG:boto:Path: /hiiqos-52/ 2023-07-29T00:12:59.065 DEBUG:boto:Data: 2023-07-29T00:12:59.066 DEBUG:boto:Headers: {} 2023-07-29T00:12:59.066 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:12:59.066 DEBUG:boto:Port: 8000 2023-07-29T00:12:59.067 DEBUG:boto:Params: {} 2023-07-29T00:12:59.067 DEBUG:boto:Token: None 2023-07-29T00:12:59.068 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 00:12:59 GMT /hiiqos-52/ 2023-07-29T00:12:59.069 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HIqXerDba/zg9+YZCbaHc4AvYFc= 2023-07-29T00:12:59.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:12:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HIqXerDba/zg9+YZCbaHc4AvYFc=', 'Content-Length': '0'} 2023-07-29T00:12:59.083 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004627bb7b1c6d711c-0064c4598b-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:12:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:12:59.084 DEBUG:boto:path=/hiiqos-52/testobj-sse-c 2023-07-29T00:12:59.085 DEBUG:boto:auth_path=/hiiqos-52/testobj-sse-c 2023-07-29T00:12:59.086 DEBUG:boto:Method: PUT 2023-07-29T00:12:59.086 DEBUG:boto:Path: /hiiqos-52/testobj-sse-c 2023-07-29T00:12:59.087 DEBUG:boto:Data: 2023-07-29T00:12:59.087 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-07-29T00:12:59.087 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:12:59.088 DEBUG:boto:Port: 8000 2023-07-29T00:12:59.088 DEBUG:boto:Params: {} 2023-07-29T00:12:59.088 DEBUG:boto:Token: None 2023-07-29T00:12:59.089 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Sat, 29 Jul 2023 00:12:59 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /hiiqos-52/testobj-sse-c 2023-07-29T00:12:59.089 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:N9tt6WLEYkfoUJnj2sdkdfGbmAE= 2023-07-29T00:12:59.090 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:12:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:N9tt6WLEYkfoUJnj2sdkdfGbmAE='} 2023-07-29T00:12:59.096 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('x-amz-request-id', 'tx00000a5b423ff9480dd83-0064c4598b-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:12:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:12:59.097 DEBUG:boto:path=/hiiqos-52/testobj-sse-kms 2023-07-29T00:12:59.097 DEBUG:boto:auth_path=/hiiqos-52/testobj-sse-kms 2023-07-29T00:12:59.098 DEBUG:boto:Method: PUT 2023-07-29T00:12:59.098 DEBUG:boto:Path: /hiiqos-52/testobj-sse-kms 2023-07-29T00:12:59.098 DEBUG:boto:Data: 2023-07-29T00:12:59.099 DEBUG:boto:Headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-07-29T00:12:59.099 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:12:59.099 DEBUG:boto:Port: 8000 2023-07-29T00:12:59.100 DEBUG:boto:Params: {} 2023-07-29T00:12:59.100 DEBUG:boto:Token: None 2023-07-29T00:12:59.100 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Sat, 29 Jul 2023 00:12:59 GMT x-amz-server-side-encryption:aws:kms x-amz-server-side-encryption-aws-kms-key-id:testkey-1 /hiiqos-52/testobj-sse-kms 2023-07-29T00:12:59.101 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VtJl7zH490iT+l0oSTQ43PBlghw= 2023-07-29T00:12:59.101 DEBUG:boto:Final headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:12:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VtJl7zH490iT+l0oSTQ43PBlghw='} 2023-07-29T00:12:59.204 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('x-amz-server-side-encryption-context', 'eyJhd3M6czM6YXJuIjoiYXJuOmF3czpzMzo6cmd3LW11bHRpc2l0ZS10ZXN0LXVzZXI6aGlpcW9zLTUyL3Rlc3RvYmotc3NlLWttcyJ9'), ('x-amz-request-id', 'tx00000ffb2eb7af83bdaf8-0064c4598b-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:12:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:12:59.204 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:12:59.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:12:59.205 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:12:59.253 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:12:59.253 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:12:59.286 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T00:12:59.298 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T00:12:59.299 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:12:59.299 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:12:59.299 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:46:09.233742Z" 2023-07-29T00:12:59.299 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:12:59.299 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:12:59.299 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690589579.045440_795.1", 2023-07-29T00:12:59.299 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:12:59.045440Z" 2023-07-29T00:12:59.300 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:12:59.300 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:12:59.300 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:12:59.300 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T00:12:59.300 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:12:59.300 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:12:59.301 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:12:59.301 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T00:12:59.301 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:12:59.301 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T00:12:59.302 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690587969.233742_99.1', 1: '1_1690589579.045440_795.1', 2: '', 3: ''} 2023-07-29T00:12:59.302 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T00:12:59.303 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:12:59.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:12:59.303 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:12:59.349 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:12:59.349 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:12:59.377 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:12:59.339+0000 7ff45412b880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:12:59.377 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:12:59.340+0000 7ff45412b880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:12:59.390 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:12:59.390 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:12:59.390 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:12:59.391 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:12:59.391 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:12:59.391 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:12:59.391 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:12:59.391 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:12:59.391 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:12:59.391 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:12:59.392 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:12:59.392 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:12:59.392 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:12:59.392 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:12:59.392 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:12:59.392 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:12:59.393 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:12:59.393 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:12:59.393 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:12:59.393 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:12:59.393 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:12:59.393 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:12:59.393 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:12:59.394 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:12:59.394 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:12:59.394 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690589441.274978_771.1", 2023-07-29T00:12:59.394 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:12:59.394 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:12:59.394 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:12:59.394 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:10:41.274978Z", 2023-07-29T00:12:59.395 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:12:59.395 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:12:59.395 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:12:59.395 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:12:59.395 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:12:59.395 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:12:59.395 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:12:59.396 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:12:59.396 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:12:59.396 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:12:59.396 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:12:59.396 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:12:59.396 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:12:59.397 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:12:59.397 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:12:59.397 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:12:59.397 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:12:59.397 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:12:59.397 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:12:59.397 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:12:59.398 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:12:59.398 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:12:59.398 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:12:59.398 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:12:59.398 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:12:59.398 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:12:59.398 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:12:59.399 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:12:59.399 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:12:59.399 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:12:59.399 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:12:59.399 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:12:59.399 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:12:59.399 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:12:59Z" 2023-07-29T00:12:59.400 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:12:59.400 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690589441.274978_771.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:10:41.274978Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:12:59Z" } 2023-07-29T00:12:59.401 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690589441.274978_771.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:10:41.274978Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:12:59.401 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690589579.045440_795.1', 2: '', 3: ''} 2023-07-29T00:12:59.402 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690589441.274978_771.1', 2: '', 3: ''} 2023-07-29T00:12:59.402 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690589579.045440_795.1 target=1_1690589441.274978_771.1 2023-07-29T00:13:04.403 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:13:04.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:13:04.404 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:13:04.458 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:13:04.458 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:13:04.490 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:13:04.453+0000 7f0a53413880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:13:04.491 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:13:04.453+0000 7f0a53413880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:13:04.504 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:13:04.505 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:13:04.505 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:13:04.505 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:13:04.505 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:13:04.505 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:13:04.505 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:13:04.506 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:13:04.506 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:13:04.506 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:13:04.506 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:13:04.506 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:13:04.506 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:13:04.506 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:13:04.507 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:13:04.507 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:13:04.507 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:13:04.507 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:13:04.507 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:13:04.507 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:13:04.507 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:13:04.508 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:13:04.508 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:13:04.508 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:13:04.508 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:13:04.508 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690589579.045440_795.1", 2023-07-29T00:13:04.508 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:13:04.509 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:13:04.509 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:13:04.509 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:12:59.045440Z", 2023-07-29T00:13:04.509 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:13:04.509 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:13:04.509 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:13:04.509 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:13:04.510 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:13:04.510 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:13:04.510 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:13:04.510 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:13:04.510 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:13:04.510 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:13:04.510 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:13:04.511 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:13:04.511 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:13:04.511 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:13:04.511 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:13:04.511 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:13:04.511 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:13:04.511 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:13:04.512 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:13:04.512 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:13:04.512 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:13:04.512 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:13:04.512 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:13:04.512 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:13:04.512 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:13:04.513 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:13:04.513 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:13:04.513 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:13:04.513 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:13:04.513 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:13:04.513 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:13:04.513 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:13:04.514 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:13:04.514 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:13:04Z" 2023-07-29T00:13:04.514 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:13:04.515 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690589579.045440_795.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:12:59.045440Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:13:04Z" } 2023-07-29T00:13:04.515 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690589579.045440_795.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:12:59.045440Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:13:04.516 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690589579.045440_795.1', 2: '', 3: ''} 2023-07-29T00:13:04.516 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690589579.045440_795.1', 2: '', 3: ''} 2023-07-29T00:13:04.516 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T00:13:04.517 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-52', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:13:04.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-52', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:13:04.517 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-52 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:13:04.571 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:13:04.572 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:13:04.604 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:13:04.567+0000 7f21ef010880 1 waiting to reach incremental sync.. 2023-07-29T00:13:09.605 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:13:09.567+0000 7f21ef010880 1 waiting to reach incremental sync.. 2023-07-29T00:13:14.606 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:13:14.568+0000 7f21ef010880 1 waiting to reach incremental sync.. 2023-07-29T00:13:19.606 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:13:19.568+0000 7f21ef010880 1 waiting to reach incremental sync.. 2023-07-29T00:13:24.607 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:13:24.569+0000 7f21ef010880 1 waiting to reach incremental sync.. 2023-07-29T00:13:29.608 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:13:29.570+0000 7f21ef010880 1 waiting to reach incremental sync.. 2023-07-29T00:13:34.609 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:13:34.570+0000 7f21ef010880 1 waiting to reach incremental sync.. 2023-07-29T00:13:39.609 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:13:39.571+0000 7f21ef010880 1 waiting to reach incremental sync.. 2023-07-29T00:13:44.610 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:13:44.571+0000 7f21ef010880 1 waiting to reach incremental sync.. 2023-07-29T00:13:49.611 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:13:49.572+0000 7f21ef010880 1 waiting to reach incremental sync.. 2023-07-29T00:13:54.612 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:13:54.573+0000 7f21ef010880 1 waiting to reach incremental sync.. 2023-07-29T00:13:59.614 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:13:59.574+0000 7f21ef010880 1 bucket sync caught up with source: 2023-07-29T00:13:59.614 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000001.33.6, , , , 00000000001.27.6, , , , , ] 2023-07-29T00:13:59.614 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000001.33.6, , , , 00000000001.27.6, , , , , ] 2023-07-29T00:13:59.615 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:13:59.574+0000 7f21ef010880 0 bucket checkpoint complete 2023-07-29T00:13:59.631 DEBUG:boto:path=/hiiqos-52/ 2023-07-29T00:13:59.631 DEBUG:boto:auth_path=/hiiqos-52/ 2023-07-29T00:13:59.632 DEBUG:boto:Method: HEAD 2023-07-29T00:13:59.632 DEBUG:boto:Path: /hiiqos-52/ 2023-07-29T00:13:59.632 DEBUG:boto:Data: 2023-07-29T00:13:59.633 DEBUG:boto:Headers: {} 2023-07-29T00:13:59.633 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:13:59.633 DEBUG:boto:Port: 8001 2023-07-29T00:13:59.634 DEBUG:boto:Params: {} 2023-07-29T00:13:59.634 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-29T00:13:59.635 DEBUG:boto:Token: None 2023-07-29T00:13:59.635 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:13:59 GMT /hiiqos-52/ 2023-07-29T00:13:59.636 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:f9MnE1QHk0VC+suQHepN7WHsV28= 2023-07-29T00:13:59.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:13:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:f9MnE1QHk0VC+suQHepN7WHsV28=', 'Content-Length': '0'} 2023-07-29T00:14:04.641 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '1024'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d3d2c57a161f435-0064c459cc-1663-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:14:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:04.642 DEBUG:boto:path=/hiiqos-52/testobj-sse-c 2023-07-29T00:14:04.643 DEBUG:boto:auth_path=/hiiqos-52/testobj-sse-c 2023-07-29T00:14:04.643 DEBUG:boto:Method: HEAD 2023-07-29T00:14:04.644 DEBUG:boto:Path: /hiiqos-52/testobj-sse-c 2023-07-29T00:14:04.644 DEBUG:boto:Data: 2023-07-29T00:14:04.644 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw=='} 2023-07-29T00:14:04.645 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:04.645 DEBUG:boto:Port: 8001 2023-07-29T00:14:04.645 DEBUG:boto:Params: {} 2023-07-29T00:14:04.646 DEBUG:boto:Token: None 2023-07-29T00:14:04.646 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:04 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /hiiqos-52/testobj-sse-c 2023-07-29T00:14:04.647 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PKPjwFofNRmvFlSqbsmSuiInhBE= 2023-07-29T00:14:04.647 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PKPjwFofNRmvFlSqbsmSuiInhBE=', 'Content-Length': '0'} 2023-07-29T00:14:04.649 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:12:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-52:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.17'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000b53779f0c57259bc-0064c459cc-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:04.651 DEBUG:boto:path=/hiiqos-52/testobj-sse-c 2023-07-29T00:14:04.651 DEBUG:boto:auth_path=/hiiqos-52/testobj-sse-c 2023-07-29T00:14:04.652 DEBUG:boto:Method: GET 2023-07-29T00:14:04.652 DEBUG:boto:Path: /hiiqos-52/testobj-sse-c 2023-07-29T00:14:04.652 DEBUG:boto:Data: 2023-07-29T00:14:04.653 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw=='} 2023-07-29T00:14:04.653 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:04.654 DEBUG:boto:Port: 8001 2023-07-29T00:14:04.654 DEBUG:boto:Params: {} 2023-07-29T00:14:04.654 DEBUG:boto:Token: None 2023-07-29T00:14:04.655 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:04 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /hiiqos-52/testobj-sse-c 2023-07-29T00:14:04.655 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3/9HtqbHRAqyOb4lPVDiOWC3YFU= 2023-07-29T00:14:04.656 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3/9HtqbHRAqyOb4lPVDiOWC3YFU=', 'Content-Length': '0'} 2023-07-29T00:14:04.658 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:12:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-52:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.17'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx000006f92a18f4910dcab-0064c459cc-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:04.659 DEBUG:boto:path=/hiiqos-52/testobj-sse-kms 2023-07-29T00:14:04.659 DEBUG:boto:auth_path=/hiiqos-52/testobj-sse-kms 2023-07-29T00:14:04.660 DEBUG:boto:Method: HEAD 2023-07-29T00:14:04.660 DEBUG:boto:Path: /hiiqos-52/testobj-sse-kms 2023-07-29T00:14:04.660 DEBUG:boto:Data: 2023-07-29T00:14:04.661 DEBUG:boto:Headers: {} 2023-07-29T00:14:04.661 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:04.661 DEBUG:boto:Port: 8001 2023-07-29T00:14:04.662 DEBUG:boto:Params: {} 2023-07-29T00:14:04.662 DEBUG:boto:Token: None 2023-07-29T00:14:04.662 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:04 GMT /hiiqos-52/testobj-sse-kms 2023-07-29T00:14:04.663 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CAlEj2f6rzqorSfehZpB5BbjOFU= 2023-07-29T00:14:04.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CAlEj2f6rzqorSfehZpB5BbjOFU=', 'Content-Length': '0'} 2023-07-29T00:14:04.665 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:12:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-52:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.17'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx000005734b9383006d3e1-0064c459cc-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:04.666 DEBUG:boto:path=/hiiqos-52/testobj-sse-kms 2023-07-29T00:14:04.666 DEBUG:boto:auth_path=/hiiqos-52/testobj-sse-kms 2023-07-29T00:14:04.667 DEBUG:boto:Method: GET 2023-07-29T00:14:04.667 DEBUG:boto:Path: /hiiqos-52/testobj-sse-kms 2023-07-29T00:14:04.668 DEBUG:boto:Data: 2023-07-29T00:14:04.668 DEBUG:boto:Headers: {} 2023-07-29T00:14:04.668 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:04.669 DEBUG:boto:Port: 8001 2023-07-29T00:14:04.669 DEBUG:boto:Params: {} 2023-07-29T00:14:04.669 DEBUG:boto:Token: None 2023-07-29T00:14:04.670 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:04 GMT /hiiqos-52/testobj-sse-kms 2023-07-29T00:14:04.670 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MmR++FMeNH6FaVLgHGzpF6fbqt4= 2023-07-29T00:14:04.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MmR++FMeNH6FaVLgHGzpF6fbqt4=', 'Content-Length': '0'} 2023-07-29T00:14:04.673 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:12:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-52:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.17'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000da9e8d3d541ebd03-0064c459cc-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:04.720 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_encrypted_object_sync ... ok 2023-07-29T00:14:04.722 INFO:rgw_multi.tests:create bucket zone=a1 name=hiiqos-53 2023-07-29T00:14:04.722 DEBUG:boto:path=/hiiqos-53/ 2023-07-29T00:14:04.724 DEBUG:boto:auth_path=/hiiqos-53/ 2023-07-29T00:14:04.725 DEBUG:boto:Method: PUT 2023-07-29T00:14:04.725 DEBUG:boto:Path: /hiiqos-53/ 2023-07-29T00:14:04.725 DEBUG:boto:Data: 2023-07-29T00:14:04.726 DEBUG:boto:Headers: {} 2023-07-29T00:14:04.726 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:04.726 DEBUG:boto:Port: 8000 2023-07-29T00:14:04.726 DEBUG:boto:Params: {} 2023-07-29T00:14:04.727 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-29T00:14:04.727 DEBUG:boto:Token: None 2023-07-29T00:14:04.728 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 00:14:04 GMT /hiiqos-53/ 2023-07-29T00:14:04.728 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fixyGM8tsdEsVjpgSNPEd1Jy7v8= 2023-07-29T00:14:04.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fixyGM8tsdEsVjpgSNPEd1Jy7v8=', 'Content-Length': '0'} 2023-07-29T00:14:09.746 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001ac098cbc9dcec0f-0064c459d1-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:09.747 DEBUG:boto:path=/hiiqos-53/ 2023-07-29T00:14:09.747 DEBUG:boto:auth_path=/hiiqos-53/ 2023-07-29T00:14:09.748 DEBUG:boto:Method: HEAD 2023-07-29T00:14:09.748 DEBUG:boto:Path: /hiiqos-53/ 2023-07-29T00:14:09.749 DEBUG:boto:Data: 2023-07-29T00:14:09.749 DEBUG:boto:Headers: {} 2023-07-29T00:14:09.749 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:09.750 DEBUG:boto:Port: 8000 2023-07-29T00:14:09.750 DEBUG:boto:Params: {} 2023-07-29T00:14:09.751 DEBUG:boto:Token: None 2023-07-29T00:14:09.751 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:09 GMT /hiiqos-53/ 2023-07-29T00:14:09.752 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aISbd6yfJzgKO+7Ll6cGofWLWvA= 2023-07-29T00:14:09.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aISbd6yfJzgKO+7Ll6cGofWLWvA=', 'Content-Length': '0'} 2023-07-29T00:14:09.755 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dab263ab809502e3-0064c459d1-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:09.755 DEBUG:boto:path=/hiiqos-53/a 2023-07-29T00:14:09.756 DEBUG:boto:auth_path=/hiiqos-53/a 2023-07-29T00:14:09.756 DEBUG:boto:Method: PUT 2023-07-29T00:14:09.757 DEBUG:boto:Path: /hiiqos-53/a 2023-07-29T00:14:09.757 DEBUG:boto:Data: 2023-07-29T00:14:09.757 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:14:09.758 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:09.758 DEBUG:boto:Port: 8000 2023-07-29T00:14:09.758 DEBUG:boto:Params: {} 2023-07-29T00:14:09.759 DEBUG:boto:Token: None 2023-07-29T00:14:09.759 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:14:09 GMT /hiiqos-53/a 2023-07-29T00:14:09.760 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ek+rxswe4Z541mfXL4xPh7tN1oY= 2023-07-29T00:14:09.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:14:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ek+rxswe4Z541mfXL4xPh7tN1oY='} 2023-07-29T00:14:09.767 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bdeedf3c234b4d00-0064c459d1-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:09.767 DEBUG:boto:path=/hiiqos-53/ 2023-07-29T00:14:09.768 DEBUG:boto:auth_path=/hiiqos-53/ 2023-07-29T00:14:09.768 DEBUG:boto:Method: HEAD 2023-07-29T00:14:09.769 DEBUG:boto:Path: /hiiqos-53/ 2023-07-29T00:14:09.769 DEBUG:boto:Data: 2023-07-29T00:14:09.770 DEBUG:boto:Headers: {} 2023-07-29T00:14:09.771 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:09.772 DEBUG:boto:Port: 8000 2023-07-29T00:14:09.772 DEBUG:boto:Params: {} 2023-07-29T00:14:09.772 DEBUG:boto:Token: None 2023-07-29T00:14:09.773 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:09 GMT /hiiqos-53/ 2023-07-29T00:14:09.773 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aISbd6yfJzgKO+7Ll6cGofWLWvA= 2023-07-29T00:14:09.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aISbd6yfJzgKO+7Ll6cGofWLWvA=', 'Content-Length': '0'} 2023-07-29T00:14:09.776 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001b4091b0eb380dc4-0064c459d1-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:09.776 DEBUG:boto:path=/hiiqos-53/b 2023-07-29T00:14:09.777 DEBUG:boto:auth_path=/hiiqos-53/b 2023-07-29T00:14:09.777 DEBUG:boto:Method: PUT 2023-07-29T00:14:09.778 DEBUG:boto:Path: /hiiqos-53/b 2023-07-29T00:14:09.778 DEBUG:boto:Data: 2023-07-29T00:14:09.778 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:14:09.779 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:09.779 DEBUG:boto:Port: 8000 2023-07-29T00:14:09.779 DEBUG:boto:Params: {} 2023-07-29T00:14:09.780 DEBUG:boto:Token: None 2023-07-29T00:14:09.780 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:14:09 GMT /hiiqos-53/b 2023-07-29T00:14:09.780 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wKqJ8jOfMO+46+TmmefU9MqFDd8= 2023-07-29T00:14:09.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:14:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wKqJ8jOfMO+46+TmmefU9MqFDd8='} 2023-07-29T00:14:09.786 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005de9503a09808756-0064c459d1-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:09.787 DEBUG:boto:path=/hiiqos-53/ 2023-07-29T00:14:09.787 DEBUG:boto:auth_path=/hiiqos-53/ 2023-07-29T00:14:09.788 DEBUG:boto:Method: HEAD 2023-07-29T00:14:09.788 DEBUG:boto:Path: /hiiqos-53/ 2023-07-29T00:14:09.788 DEBUG:boto:Data: 2023-07-29T00:14:09.789 DEBUG:boto:Headers: {} 2023-07-29T00:14:09.789 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:09.790 DEBUG:boto:Port: 8000 2023-07-29T00:14:09.790 DEBUG:boto:Params: {} 2023-07-29T00:14:09.790 DEBUG:boto:Token: None 2023-07-29T00:14:09.791 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:09 GMT /hiiqos-53/ 2023-07-29T00:14:09.791 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aISbd6yfJzgKO+7Ll6cGofWLWvA= 2023-07-29T00:14:09.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aISbd6yfJzgKO+7Ll6cGofWLWvA=', 'Content-Length': '0'} 2023-07-29T00:14:09.794 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ab92c48ab3026cb6-0064c459d1-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:09.795 DEBUG:boto:path=/hiiqos-53/c 2023-07-29T00:14:09.795 DEBUG:boto:auth_path=/hiiqos-53/c 2023-07-29T00:14:09.795 DEBUG:boto:Method: PUT 2023-07-29T00:14:09.796 DEBUG:boto:Path: /hiiqos-53/c 2023-07-29T00:14:09.796 DEBUG:boto:Data: 2023-07-29T00:14:09.796 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:14:09.797 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:09.797 DEBUG:boto:Port: 8000 2023-07-29T00:14:09.797 DEBUG:boto:Params: {} 2023-07-29T00:14:09.798 DEBUG:boto:Token: None 2023-07-29T00:14:09.798 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:14:09 GMT /hiiqos-53/c 2023-07-29T00:14:09.799 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:psjx3AgWnQUKCK0b7EbGKVWGYN8= 2023-07-29T00:14:09.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:14:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:psjx3AgWnQUKCK0b7EbGKVWGYN8='} 2023-07-29T00:14:09.844 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ea15e68a5f5145c8-0064c459d1-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:09.845 DEBUG:boto:path=/hiiqos-53/ 2023-07-29T00:14:09.846 DEBUG:boto:auth_path=/hiiqos-53/ 2023-07-29T00:14:09.846 DEBUG:boto:Method: HEAD 2023-07-29T00:14:09.846 DEBUG:boto:Path: /hiiqos-53/ 2023-07-29T00:14:09.847 DEBUG:boto:Data: 2023-07-29T00:14:09.847 DEBUG:boto:Headers: {} 2023-07-29T00:14:09.847 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:09.848 DEBUG:boto:Port: 8000 2023-07-29T00:14:09.848 DEBUG:boto:Params: {} 2023-07-29T00:14:09.848 DEBUG:boto:Token: None 2023-07-29T00:14:09.849 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:09 GMT /hiiqos-53/ 2023-07-29T00:14:09.849 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aISbd6yfJzgKO+7Ll6cGofWLWvA= 2023-07-29T00:14:09.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aISbd6yfJzgKO+7Ll6cGofWLWvA=', 'Content-Length': '0'} 2023-07-29T00:14:09.852 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000017b3dcdf64049111-0064c459d1-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:09.853 DEBUG:boto:path=/hiiqos-53/d 2023-07-29T00:14:09.853 DEBUG:boto:auth_path=/hiiqos-53/d 2023-07-29T00:14:09.853 DEBUG:boto:Method: PUT 2023-07-29T00:14:09.854 DEBUG:boto:Path: /hiiqos-53/d 2023-07-29T00:14:09.854 DEBUG:boto:Data: 2023-07-29T00:14:09.854 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:14:09.855 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:09.855 DEBUG:boto:Port: 8000 2023-07-29T00:14:09.856 DEBUG:boto:Params: {} 2023-07-29T00:14:09.856 DEBUG:boto:Token: None 2023-07-29T00:14:09.856 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:14:09 GMT /hiiqos-53/d 2023-07-29T00:14:09.857 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wpo3bqeSPdgS1ZDCvVXB0vybzLU= 2023-07-29T00:14:09.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:14:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wpo3bqeSPdgS1ZDCvVXB0vybzLU='} 2023-07-29T00:14:09.863 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000284de4a502853355-0064c459d1-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:09.864 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:14:09.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:14:09.865 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:14:09.914 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:14:09.914 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:14:09.944 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T00:14:09.957 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T00:14:09.957 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:14:09.957 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:14:09.957 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:46:09.233742Z" 2023-07-29T00:14:09.958 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:09.958 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:14:09.958 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690589649.706348_801.1", 2023-07-29T00:14:09.958 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:14:09.706348Z" 2023-07-29T00:14:09.958 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:09.958 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:14:09.959 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:14:09.959 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T00:14:09.959 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:09.959 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:14:09.959 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:14:09.959 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T00:14:09.959 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:14:09.960 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T00:14:09.961 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690587969.233742_99.1', 1: '1_1690589649.706348_801.1', 2: '', 3: ''} 2023-07-29T00:14:09.961 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T00:14:09.962 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:14:09.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:14:09.962 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:14:10.008 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:14:10.008 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:14:10.036 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:14:09.996+0000 7f831b016880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:14:10.036 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:14:09.997+0000 7f831b016880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:14:10.049 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:14:10.049 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:14:10.049 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:14:10.050 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:14:10.050 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:14:10.050 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:14:10.050 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:14:10.050 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:10.050 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:14:10.050 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:14:10.051 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:14:10.051 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:14:10.051 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:14:10.051 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:14:10.051 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:14:10.051 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:14:10.052 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:14:10.052 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:14:10.052 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:14:10.052 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:14:10.052 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:10.052 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:14:10.052 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:14:10.053 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:14:10.053 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:14:10.053 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690589649.699534_796.1", 2023-07-29T00:14:10.053 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:14:10.053 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:14:10.053 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:14:10.054 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:14:09.699534Z", 2023-07-29T00:14:10.054 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:14:10.054 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:14:10.054 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:10.054 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:14:10.054 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:14:10.054 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:14:10.055 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:14:10.055 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:14:10.055 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:14:10.055 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:14:10.055 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:14:10.055 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:14:10.055 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:14:10.056 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:14:10.056 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:10.056 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:14:10.056 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:14:10.056 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:14:10.056 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:14:10.056 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:14:10.057 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:14:10.057 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:14:10.057 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:14:10.057 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:14:10.057 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:14:10.057 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:14:10.057 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:14:10.058 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:14:10.058 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:10.058 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:14:10.058 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:14:10.058 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:14:10.058 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:10.058 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:14:09Z" 2023-07-29T00:14:10.059 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:14:10.059 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690589649.699534_796.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:14:09.699534Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:14:09Z" } 2023-07-29T00:14:10.060 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690589649.699534_796.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:14:09.699534Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:14:10.061 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690589649.706348_801.1', 2: '', 3: ''} 2023-07-29T00:14:10.061 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690589649.699534_796.1', 2: '', 3: ''} 2023-07-29T00:14:10.061 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690589649.706348_801.1 target=1_1690589649.699534_796.1 2023-07-29T00:14:15.062 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:14:15.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:14:15.063 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:14:15.115 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:14:15.115 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:14:15.146 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:14:15.107+0000 7f71ff012880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:14:15.147 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:14:15.108+0000 7f71ff012880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:14:15.161 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:14:15.161 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:14:15.161 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:14:15.161 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:14:15.161 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:14:15.162 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:14:15.162 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:14:15.162 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:15.162 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:14:15.163 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:14:15.163 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:14:15.163 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:14:15.164 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:14:15.164 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:14:15.164 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:14:15.164 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:14:15.165 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:14:15.165 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:14:15.165 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:14:15.166 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:14:15.166 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:15.166 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:14:15.166 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:14:15.167 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:14:15.167 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:14:15.167 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690589649.706348_801.1", 2023-07-29T00:14:15.168 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:14:15.168 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:14:15.168 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:14:15.169 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:14:09.706348Z", 2023-07-29T00:14:15.169 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:14:15.169 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:14:15.170 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:15.170 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:14:15.170 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:14:15.170 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:14:15.171 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:14:15.171 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:14:15.171 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:14:15.171 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:14:15.172 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:14:15.172 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:14:15.172 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:14:15.172 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:14:15.173 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:15.173 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:14:15.173 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:14:15.173 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:14:15.173 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:14:15.174 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:14:15.174 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:14:15.174 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:14:15.174 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:14:15.174 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:14:15.174 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:14:15.175 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:14:15.175 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:14:15.175 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:14:15.175 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:15.175 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:14:15.175 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:14:15.175 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:14:15.176 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:15.176 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:14:15Z" 2023-07-29T00:14:15.176 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:14:15.177 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690589649.706348_801.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:14:09.706348Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:14:15Z" } 2023-07-29T00:14:15.177 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690589649.706348_801.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:14:09.706348Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:14:15.178 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690589649.706348_801.1', 2: '', 3: ''} 2023-07-29T00:14:15.178 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690589649.706348_801.1', 2: '', 3: ''} 2023-07-29T00:14:15.179 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T00:14:15.179 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-53 2023-07-29T00:14:15.179 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-53', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:14:15.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-53', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:14:15.180 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-53 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:14:15.232 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:14:15.233 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:14:15.269 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:14:15.229+0000 7f191c218880 1 bucket sync caught up with source: 2023-07-29T00:14:15.269 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000001.36.6, , , , 00000000001.29.6, , , 00000000001.29.6, , 00000000001.89.6] 2023-07-29T00:14:15.269 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000001.36.6, , , , 00000000001.29.6, , , 00000000001.29.6, , 00000000001.89.6] 2023-07-29T00:14:15.269 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:14:15.229+0000 7f191c218880 0 bucket checkpoint complete 2023-07-29T00:14:15.283 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-53 2023-07-29T00:14:15.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-53', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:14:15.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-53', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:14:15.284 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-53 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:14:15.339 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:14:15.339 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:14:15.371 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:14:15.332+0000 7f8842522880 1 waiting to reach incremental sync.. 2023-07-29T00:14:20.373 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:14:20.334+0000 7f8842522880 1 bucket sync caught up with source: 2023-07-29T00:14:20.374 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000001.223.6, , , , 00000000001.1012.6, , , 00000000001.1027.6, , 00000000001.754.6] 2023-07-29T00:14:20.374 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000001.223.6, , , , 00000000001.1012.6, , , 00000000001.1027.6, , 00000000001.754.6] 2023-07-29T00:14:20.374 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:14:20.334+0000 7f8842522880 0 bucket checkpoint complete 2023-07-29T00:14:20.397 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-53 zones={a2, a1} 2023-07-29T00:14:20.397 DEBUG:boto:path=/hiiqos-53/ 2023-07-29T00:14:20.398 DEBUG:boto:auth_path=/hiiqos-53/ 2023-07-29T00:14:20.398 DEBUG:boto:Method: HEAD 2023-07-29T00:14:20.398 DEBUG:boto:Path: /hiiqos-53/ 2023-07-29T00:14:20.399 DEBUG:boto:Data: 2023-07-29T00:14:20.399 DEBUG:boto:Headers: {} 2023-07-29T00:14:20.400 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:20.400 DEBUG:boto:Port: 8001 2023-07-29T00:14:20.401 DEBUG:boto:Params: {} 2023-07-29T00:14:20.401 DEBUG:boto:Token: None 2023-07-29T00:14:20.401 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:20 GMT /hiiqos-53/ 2023-07-29T00:14:20.402 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cPEWr3I4KaYMwoopl+iYSgMm0Ac= 2023-07-29T00:14:20.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cPEWr3I4KaYMwoopl+iYSgMm0Ac=', 'Content-Length': '0'} 2023-07-29T00:14:20.405 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003941278b64128ed9-0064c459dc-1663-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:20.405 DEBUG:boto:path=/hiiqos-53/ 2023-07-29T00:14:20.406 DEBUG:boto:auth_path=/hiiqos-53/ 2023-07-29T00:14:20.406 DEBUG:boto:Method: HEAD 2023-07-29T00:14:20.407 DEBUG:boto:Path: /hiiqos-53/ 2023-07-29T00:14:20.407 DEBUG:boto:Data: 2023-07-29T00:14:20.407 DEBUG:boto:Headers: {} 2023-07-29T00:14:20.408 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:20.408 DEBUG:boto:Port: 8000 2023-07-29T00:14:20.408 DEBUG:boto:Params: {} 2023-07-29T00:14:20.409 DEBUG:boto:Token: None 2023-07-29T00:14:20.409 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:20 GMT /hiiqos-53/ 2023-07-29T00:14:20.410 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cPEWr3I4KaYMwoopl+iYSgMm0Ac= 2023-07-29T00:14:20.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cPEWr3I4KaYMwoopl+iYSgMm0Ac=', 'Content-Length': '0'} 2023-07-29T00:14:20.413 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000092a19a8b14cef41-0064c459dc-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:20.413 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-29T00:14:20.414 DEBUG:boto:path=/hiiqos-53/ 2023-07-29T00:14:20.414 DEBUG:boto:auth_path=/hiiqos-53/ 2023-07-29T00:14:20.414 DEBUG:boto:path=/hiiqos-53/?versions&max-keys=999 2023-07-29T00:14:20.415 DEBUG:boto:auth_path=/hiiqos-53/?versions&max-keys=999 2023-07-29T00:14:20.415 DEBUG:boto:Method: GET 2023-07-29T00:14:20.416 DEBUG:boto:Path: /hiiqos-53/?versions&max-keys=999 2023-07-29T00:14:20.416 DEBUG:boto:Data: 2023-07-29T00:14:20.416 DEBUG:boto:Headers: {} 2023-07-29T00:14:20.417 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:20.417 DEBUG:boto:Port: 8001 2023-07-29T00:14:20.418 DEBUG:boto:Params: {} 2023-07-29T00:14:20.418 DEBUG:boto:Token: None 2023-07-29T00:14:20.419 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:20 GMT /hiiqos-53/?versions 2023-07-29T00:14:20.419 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:y3HrjsNlkrl9Tqc/o5LqUQvUfYw= 2023-07-29T00:14:20.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:y3HrjsNlkrl9Tqc/o5LqUQvUfYw=', 'Content-Length': '0'} 2023-07-29T00:14:20.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c33eec89233dd35c-0064c459dc-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:20.423 DEBUG:boto:b'hiiqos-53999falseanulltrue2023-07-29T00:14:09.724Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:14:09.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:14:09.763Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:14:09.821Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:20.424 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-07-29T00:14:20.424 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-07-29T00:14:20.425 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-07-29T00:14:20.425 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-07-29T00:14:20.425 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-29T00:14:20.425 DEBUG:boto:path=/hiiqos-53/ 2023-07-29T00:14:20.426 DEBUG:boto:auth_path=/hiiqos-53/ 2023-07-29T00:14:20.426 DEBUG:boto:path=/hiiqos-53/?versions&max-keys=999 2023-07-29T00:14:20.427 DEBUG:boto:auth_path=/hiiqos-53/?versions&max-keys=999 2023-07-29T00:14:20.427 DEBUG:boto:Method: GET 2023-07-29T00:14:20.427 DEBUG:boto:Path: /hiiqos-53/?versions&max-keys=999 2023-07-29T00:14:20.428 DEBUG:boto:Data: 2023-07-29T00:14:20.428 DEBUG:boto:Headers: {} 2023-07-29T00:14:20.428 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:20.429 DEBUG:boto:Port: 8000 2023-07-29T00:14:20.429 DEBUG:boto:Params: {} 2023-07-29T00:14:20.429 DEBUG:boto:Token: None 2023-07-29T00:14:20.430 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:20 GMT /hiiqos-53/?versions 2023-07-29T00:14:20.430 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:y3HrjsNlkrl9Tqc/o5LqUQvUfYw= 2023-07-29T00:14:20.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:y3HrjsNlkrl9Tqc/o5LqUQvUfYw=', 'Content-Length': '0'} 2023-07-29T00:14:20.433 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000534c1fe30853bca7-0064c459dc-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:20.434 DEBUG:boto:b'hiiqos-53999falseanulltrue2023-07-29T00:14:09.724Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:14:09.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:14:09.763Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:14:09.821Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:20.435 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-07-29T00:14:20.435 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-07-29T00:14:20.435 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-07-29T00:14:20.436 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-07-29T00:14:20.436 DEBUG:boto:path=/hiiqos-53/ 2023-07-29T00:14:20.436 DEBUG:boto:auth_path=/hiiqos-53/ 2023-07-29T00:14:20.437 DEBUG:boto:path=/hiiqos-53/?versions&max-keys=999 2023-07-29T00:14:20.437 DEBUG:boto:auth_path=/hiiqos-53/?versions&max-keys=999 2023-07-29T00:14:20.438 DEBUG:boto:Method: GET 2023-07-29T00:14:20.438 DEBUG:boto:Path: /hiiqos-53/?versions&max-keys=999 2023-07-29T00:14:20.438 DEBUG:boto:Data: 2023-07-29T00:14:20.439 DEBUG:boto:Headers: {} 2023-07-29T00:14:20.439 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:20.440 DEBUG:boto:Port: 8001 2023-07-29T00:14:20.440 DEBUG:boto:Params: {} 2023-07-29T00:14:20.440 DEBUG:boto:Token: None 2023-07-29T00:14:20.441 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:20 GMT /hiiqos-53/?versions 2023-07-29T00:14:20.441 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:y3HrjsNlkrl9Tqc/o5LqUQvUfYw= 2023-07-29T00:14:20.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:y3HrjsNlkrl9Tqc/o5LqUQvUfYw=', 'Content-Length': '0'} 2023-07-29T00:14:20.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b192cd767395cb16-0064c459dc-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:20.445 DEBUG:boto:b'hiiqos-53999falseanulltrue2023-07-29T00:14:09.724Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:14:09.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:14:09.763Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:14:09.821Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:20.446 DEBUG:boto:path=/hiiqos-53/ 2023-07-29T00:14:20.446 DEBUG:boto:auth_path=/hiiqos-53/ 2023-07-29T00:14:20.447 DEBUG:boto:path=/hiiqos-53/?versions&max-keys=999 2023-07-29T00:14:20.447 DEBUG:boto:auth_path=/hiiqos-53/?versions&max-keys=999 2023-07-29T00:14:20.447 DEBUG:boto:Method: GET 2023-07-29T00:14:20.448 DEBUG:boto:Path: /hiiqos-53/?versions&max-keys=999 2023-07-29T00:14:20.448 DEBUG:boto:Data: 2023-07-29T00:14:20.449 DEBUG:boto:Headers: {} 2023-07-29T00:14:20.449 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:20.450 DEBUG:boto:Port: 8000 2023-07-29T00:14:20.450 DEBUG:boto:Params: {} 2023-07-29T00:14:20.451 DEBUG:boto:Token: None 2023-07-29T00:14:20.451 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:20 GMT /hiiqos-53/?versions 2023-07-29T00:14:20.452 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:y3HrjsNlkrl9Tqc/o5LqUQvUfYw= 2023-07-29T00:14:20.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:y3HrjsNlkrl9Tqc/o5LqUQvUfYw=', 'Content-Length': '0'} 2023-07-29T00:14:20.456 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000080ecffce8b6e00c-0064c459dc-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:20.456 DEBUG:boto:b'hiiqos-53999falseanulltrue2023-07-29T00:14:09.724Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:14:09.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:14:09.763Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:14:09.821Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:20.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-07-29T00:14:20.457 DEBUG:boto:path=/hiiqos-53/a 2023-07-29T00:14:20.458 DEBUG:boto:auth_path=/hiiqos-53/a 2023-07-29T00:14:20.458 DEBUG:boto:path=/hiiqos-53/a?versionId=null 2023-07-29T00:14:20.458 DEBUG:boto:auth_path=/hiiqos-53/a?versionId=null 2023-07-29T00:14:20.459 DEBUG:boto:Method: GET 2023-07-29T00:14:20.459 DEBUG:boto:Path: /hiiqos-53/a?versionId=null 2023-07-29T00:14:20.460 DEBUG:boto:Data: 2023-07-29T00:14:20.460 DEBUG:boto:Headers: {} 2023-07-29T00:14:20.460 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:20.461 DEBUG:boto:Port: 8001 2023-07-29T00:14:20.461 DEBUG:boto:Params: {} 2023-07-29T00:14:20.462 DEBUG:boto:Token: None 2023-07-29T00:14:20.462 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:20 GMT /hiiqos-53/a?versionId=null 2023-07-29T00:14:20.462 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9dsttstuCTS6+wyO42+Lp3H753g= 2023-07-29T00:14:20.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9dsttstuCTS6+wyO42+Lp3H753g=', 'Content-Length': '0'} 2023-07-29T00:14:20.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-53:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000223182c7b83c8a89-0064c459dc-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:20.465 DEBUG:boto:path=/hiiqos-53/a 2023-07-29T00:14:20.466 DEBUG:boto:auth_path=/hiiqos-53/a 2023-07-29T00:14:20.466 DEBUG:boto:path=/hiiqos-53/a?versionId=null 2023-07-29T00:14:20.467 DEBUG:boto:auth_path=/hiiqos-53/a?versionId=null 2023-07-29T00:14:20.467 DEBUG:boto:Method: GET 2023-07-29T00:14:20.467 DEBUG:boto:Path: /hiiqos-53/a?versionId=null 2023-07-29T00:14:20.468 DEBUG:boto:Data: 2023-07-29T00:14:20.468 DEBUG:boto:Headers: {} 2023-07-29T00:14:20.469 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:20.469 DEBUG:boto:Port: 8000 2023-07-29T00:14:20.469 DEBUG:boto:Params: {} 2023-07-29T00:14:20.470 DEBUG:boto:Token: None 2023-07-29T00:14:20.470 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:20 GMT /hiiqos-53/a?versionId=null 2023-07-29T00:14:20.471 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9dsttstuCTS6+wyO42+Lp3H753g= 2023-07-29T00:14:20.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9dsttstuCTS6+wyO42+Lp3H753g=', 'Content-Length': '0'} 2023-07-29T00:14:20.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052514169dff781ac-0064c459dc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:20.520 DEBUG:boto:path=/hiiqos-53/a 2023-07-29T00:14:20.521 DEBUG:boto:auth_path=/hiiqos-53/a 2023-07-29T00:14:20.522 DEBUG:boto:path=/hiiqos-53/a?versionId=null 2023-07-29T00:14:20.522 DEBUG:boto:auth_path=/hiiqos-53/a?versionId=null 2023-07-29T00:14:20.522 DEBUG:boto:Method: HEAD 2023-07-29T00:14:20.523 DEBUG:boto:Path: /hiiqos-53/a?versionId=null 2023-07-29T00:14:20.523 DEBUG:boto:Data: 2023-07-29T00:14:20.523 DEBUG:boto:Headers: {} 2023-07-29T00:14:20.524 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:20.524 DEBUG:boto:Port: 8001 2023-07-29T00:14:20.525 DEBUG:boto:Params: {} 2023-07-29T00:14:20.525 DEBUG:boto:Token: None 2023-07-29T00:14:20.525 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:20 GMT /hiiqos-53/a?versionId=null 2023-07-29T00:14:20.526 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3CnDqzfK1HMloIpFYfL0qUHZ5BM= 2023-07-29T00:14:20.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3CnDqzfK1HMloIpFYfL0qUHZ5BM=', 'Content-Length': '0'} 2023-07-29T00:14:20.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-53:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecbac4ce80b65426-0064c459dc-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:20.530 DEBUG:boto:path=/hiiqos-53/a 2023-07-29T00:14:20.530 DEBUG:boto:auth_path=/hiiqos-53/a 2023-07-29T00:14:20.531 DEBUG:boto:path=/hiiqos-53/a?versionId=null 2023-07-29T00:14:20.531 DEBUG:boto:auth_path=/hiiqos-53/a?versionId=null 2023-07-29T00:14:20.532 DEBUG:boto:Method: HEAD 2023-07-29T00:14:20.532 DEBUG:boto:Path: /hiiqos-53/a?versionId=null 2023-07-29T00:14:20.533 DEBUG:boto:Data: 2023-07-29T00:14:20.533 DEBUG:boto:Headers: {} 2023-07-29T00:14:20.534 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:20.534 DEBUG:boto:Port: 8000 2023-07-29T00:14:20.534 DEBUG:boto:Params: {} 2023-07-29T00:14:20.535 DEBUG:boto:Token: None 2023-07-29T00:14:20.535 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:20 GMT /hiiqos-53/a?versionId=null 2023-07-29T00:14:20.536 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3CnDqzfK1HMloIpFYfL0qUHZ5BM= 2023-07-29T00:14:20.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3CnDqzfK1HMloIpFYfL0qUHZ5BM=', 'Content-Length': '0'} 2023-07-29T00:14:20.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ae85f2c8fdf9533-0064c459dc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:20.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-07-29T00:14:20.539 DEBUG:boto:path=/hiiqos-53/a 2023-07-29T00:14:20.539 DEBUG:boto:auth_path=/hiiqos-53/a 2023-07-29T00:14:20.540 DEBUG:boto:path=/hiiqos-53/a?versionId=null 2023-07-29T00:14:20.540 DEBUG:boto:auth_path=/hiiqos-53/a?versionId=null 2023-07-29T00:14:20.540 DEBUG:boto:Method: GET 2023-07-29T00:14:20.541 DEBUG:boto:Path: /hiiqos-53/a?versionId=null 2023-07-29T00:14:20.541 DEBUG:boto:Data: 2023-07-29T00:14:20.541 DEBUG:boto:Headers: {} 2023-07-29T00:14:20.542 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:20.542 DEBUG:boto:Port: 8001 2023-07-29T00:14:20.542 DEBUG:boto:Params: {} 2023-07-29T00:14:20.543 DEBUG:boto:Token: None 2023-07-29T00:14:20.543 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:20 GMT /hiiqos-53/a?versionId=null 2023-07-29T00:14:20.544 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9dsttstuCTS6+wyO42+Lp3H753g= 2023-07-29T00:14:20.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9dsttstuCTS6+wyO42+Lp3H753g=', 'Content-Length': '0'} 2023-07-29T00:14:20.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-53:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000146de4a623c56096-0064c459dc-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:20.588 DEBUG:boto:path=/hiiqos-53/a 2023-07-29T00:14:20.589 DEBUG:boto:auth_path=/hiiqos-53/a 2023-07-29T00:14:20.590 DEBUG:boto:path=/hiiqos-53/a?versionId=null 2023-07-29T00:14:20.590 DEBUG:boto:auth_path=/hiiqos-53/a?versionId=null 2023-07-29T00:14:20.591 DEBUG:boto:Method: GET 2023-07-29T00:14:20.591 DEBUG:boto:Path: /hiiqos-53/a?versionId=null 2023-07-29T00:14:20.591 DEBUG:boto:Data: 2023-07-29T00:14:20.592 DEBUG:boto:Headers: {} 2023-07-29T00:14:20.592 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:20.593 DEBUG:boto:Port: 8000 2023-07-29T00:14:20.593 DEBUG:boto:Params: {} 2023-07-29T00:14:20.594 DEBUG:boto:Token: None 2023-07-29T00:14:20.594 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:20 GMT /hiiqos-53/a?versionId=null 2023-07-29T00:14:20.595 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9dsttstuCTS6+wyO42+Lp3H753g= 2023-07-29T00:14:20.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9dsttstuCTS6+wyO42+Lp3H753g=', 'Content-Length': '0'} 2023-07-29T00:14:20.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8c857fd4ef2a072-0064c459dc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:20.598 DEBUG:boto:path=/hiiqos-53/ 2023-07-29T00:14:20.599 DEBUG:boto:auth_path=/hiiqos-53/ 2023-07-29T00:14:20.599 DEBUG:boto:path=/hiiqos-53/?prefix=a 2023-07-29T00:14:20.599 DEBUG:boto:auth_path=/hiiqos-53/?prefix=a 2023-07-29T00:14:20.600 DEBUG:boto:Method: GET 2023-07-29T00:14:20.600 DEBUG:boto:Path: /hiiqos-53/?prefix=a 2023-07-29T00:14:20.601 DEBUG:boto:Data: 2023-07-29T00:14:20.601 DEBUG:boto:Headers: {} 2023-07-29T00:14:20.601 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:20.602 DEBUG:boto:Port: 8001 2023-07-29T00:14:20.602 DEBUG:boto:Params: {} 2023-07-29T00:14:20.603 DEBUG:boto:Token: None 2023-07-29T00:14:20.603 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:20 GMT /hiiqos-53/ 2023-07-29T00:14:20.603 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sNnB4vhsmHBCpY7VGemlM6zJVSU= 2023-07-29T00:14:20.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sNnB4vhsmHBCpY7VGemlM6zJVSU=', 'Content-Length': '0'} 2023-07-29T00:14:20.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000deb76c7e3d8d8d01-0064c459dc-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:20.607 DEBUG:boto:b'hiiqos-53a1000falsea2023-07-29T00:14:09.724Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:20.607 DEBUG:boto:path=/hiiqos-53/ 2023-07-29T00:14:20.608 DEBUG:boto:auth_path=/hiiqos-53/ 2023-07-29T00:14:20.608 DEBUG:boto:path=/hiiqos-53/?prefix=a 2023-07-29T00:14:20.609 DEBUG:boto:auth_path=/hiiqos-53/?prefix=a 2023-07-29T00:14:20.609 DEBUG:boto:Method: GET 2023-07-29T00:14:20.610 DEBUG:boto:Path: /hiiqos-53/?prefix=a 2023-07-29T00:14:20.610 DEBUG:boto:Data: 2023-07-29T00:14:20.611 DEBUG:boto:Headers: {} 2023-07-29T00:14:20.611 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:20.611 DEBUG:boto:Port: 8000 2023-07-29T00:14:20.612 DEBUG:boto:Params: {} 2023-07-29T00:14:20.612 DEBUG:boto:Token: None 2023-07-29T00:14:20.613 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:20 GMT /hiiqos-53/ 2023-07-29T00:14:20.613 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sNnB4vhsmHBCpY7VGemlM6zJVSU= 2023-07-29T00:14:20.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sNnB4vhsmHBCpY7VGemlM6zJVSU=', 'Content-Length': '0'} 2023-07-29T00:14:20.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1819c1fa25f0976-0064c459dc-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:20.617 DEBUG:boto:b'hiiqos-53a1000falsea2023-07-29T00:14:09.724Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:20.618 DEBUG:boto:path=/hiiqos-53/a 2023-07-29T00:14:20.619 DEBUG:boto:auth_path=/hiiqos-53/a 2023-07-29T00:14:20.619 DEBUG:boto:Method: HEAD 2023-07-29T00:14:20.620 DEBUG:boto:Path: /hiiqos-53/a 2023-07-29T00:14:20.620 DEBUG:boto:Data: 2023-07-29T00:14:20.621 DEBUG:boto:Headers: {} 2023-07-29T00:14:20.621 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:20.621 DEBUG:boto:Port: 8001 2023-07-29T00:14:20.622 DEBUG:boto:Params: {} 2023-07-29T00:14:20.622 DEBUG:boto:Token: None 2023-07-29T00:14:20.622 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:20 GMT /hiiqos-53/a 2023-07-29T00:14:20.623 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:42SIqMpXS9N1IYCwKlMOtf7MlEQ= 2023-07-29T00:14:20.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:42SIqMpXS9N1IYCwKlMOtf7MlEQ=', 'Content-Length': '0'} 2023-07-29T00:14:20.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-53:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013d7c778277266b0-0064c459dc-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:20.626 DEBUG:boto:path=/hiiqos-53/a 2023-07-29T00:14:20.626 DEBUG:boto:auth_path=/hiiqos-53/a 2023-07-29T00:14:20.627 DEBUG:boto:Method: HEAD 2023-07-29T00:14:20.627 DEBUG:boto:Path: /hiiqos-53/a 2023-07-29T00:14:20.627 DEBUG:boto:Data: 2023-07-29T00:14:20.628 DEBUG:boto:Headers: {} 2023-07-29T00:14:20.628 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:20.628 DEBUG:boto:Port: 8000 2023-07-29T00:14:20.629 DEBUG:boto:Params: {} 2023-07-29T00:14:20.629 DEBUG:boto:Token: None 2023-07-29T00:14:20.630 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:20 GMT /hiiqos-53/a 2023-07-29T00:14:20.630 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:42SIqMpXS9N1IYCwKlMOtf7MlEQ= 2023-07-29T00:14:20.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:42SIqMpXS9N1IYCwKlMOtf7MlEQ=', 'Content-Length': '0'} 2023-07-29T00:14:20.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019dc83935112df50-0064c459dc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:20.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-07-29T00:14:20.634 DEBUG:boto:path=/hiiqos-53/a 2023-07-29T00:14:20.634 DEBUG:boto:auth_path=/hiiqos-53/a 2023-07-29T00:14:20.634 DEBUG:boto:Method: GET 2023-07-29T00:14:20.635 DEBUG:boto:Path: /hiiqos-53/a 2023-07-29T00:14:20.635 DEBUG:boto:Data: 2023-07-29T00:14:20.635 DEBUG:boto:Headers: {} 2023-07-29T00:14:20.636 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:20.636 DEBUG:boto:Port: 8001 2023-07-29T00:14:20.637 DEBUG:boto:Params: {} 2023-07-29T00:14:20.637 DEBUG:boto:Token: None 2023-07-29T00:14:20.637 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:20 GMT /hiiqos-53/a 2023-07-29T00:14:20.638 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LjddlbNOLUj9o1NmvsrKuoKsl4Q= 2023-07-29T00:14:20.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LjddlbNOLUj9o1NmvsrKuoKsl4Q=', 'Content-Length': '0'} 2023-07-29T00:14:20.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-53:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e47a05407c0070c-0064c459dc-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:20.641 DEBUG:boto:path=/hiiqos-53/a 2023-07-29T00:14:20.641 DEBUG:boto:auth_path=/hiiqos-53/a 2023-07-29T00:14:20.641 DEBUG:boto:Method: GET 2023-07-29T00:14:20.642 DEBUG:boto:Path: /hiiqos-53/a 2023-07-29T00:14:20.642 DEBUG:boto:Data: 2023-07-29T00:14:20.643 DEBUG:boto:Headers: {} 2023-07-29T00:14:20.643 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:20.644 DEBUG:boto:Port: 8000 2023-07-29T00:14:20.644 DEBUG:boto:Params: {} 2023-07-29T00:14:20.644 DEBUG:boto:Token: None 2023-07-29T00:14:20.645 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:20 GMT /hiiqos-53/a 2023-07-29T00:14:20.645 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LjddlbNOLUj9o1NmvsrKuoKsl4Q= 2023-07-29T00:14:20.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LjddlbNOLUj9o1NmvsrKuoKsl4Q=', 'Content-Length': '0'} 2023-07-29T00:14:20.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a20781dcadb8620-0064c459dc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:20.688 DEBUG:boto:path=/hiiqos-53/ 2023-07-29T00:14:20.689 DEBUG:boto:auth_path=/hiiqos-53/ 2023-07-29T00:14:20.689 DEBUG:boto:path=/hiiqos-53/?prefix=a 2023-07-29T00:14:20.690 DEBUG:boto:auth_path=/hiiqos-53/?prefix=a 2023-07-29T00:14:20.690 DEBUG:boto:Method: GET 2023-07-29T00:14:20.691 DEBUG:boto:Path: /hiiqos-53/?prefix=a 2023-07-29T00:14:20.691 DEBUG:boto:Data: 2023-07-29T00:14:20.691 DEBUG:boto:Headers: {} 2023-07-29T00:14:20.692 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:20.692 DEBUG:boto:Port: 8001 2023-07-29T00:14:20.692 DEBUG:boto:Params: {} 2023-07-29T00:14:20.693 DEBUG:boto:Token: None 2023-07-29T00:14:20.693 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:20 GMT /hiiqos-53/ 2023-07-29T00:14:20.694 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sNnB4vhsmHBCpY7VGemlM6zJVSU= 2023-07-29T00:14:20.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sNnB4vhsmHBCpY7VGemlM6zJVSU=', 'Content-Length': '0'} 2023-07-29T00:14:20.697 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003adae1b284c454c3-0064c459dc-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:20.697 DEBUG:boto:b'hiiqos-53a1000falsea2023-07-29T00:14:09.724Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:20.698 DEBUG:boto:path=/hiiqos-53/ 2023-07-29T00:14:20.699 DEBUG:boto:auth_path=/hiiqos-53/ 2023-07-29T00:14:20.699 DEBUG:boto:path=/hiiqos-53/?prefix=a 2023-07-29T00:14:20.699 DEBUG:boto:auth_path=/hiiqos-53/?prefix=a 2023-07-29T00:14:20.700 DEBUG:boto:Method: GET 2023-07-29T00:14:20.700 DEBUG:boto:Path: /hiiqos-53/?prefix=a 2023-07-29T00:14:20.701 DEBUG:boto:Data: 2023-07-29T00:14:20.701 DEBUG:boto:Headers: {} 2023-07-29T00:14:20.702 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:20.702 DEBUG:boto:Port: 8000 2023-07-29T00:14:20.702 DEBUG:boto:Params: {} 2023-07-29T00:14:20.703 DEBUG:boto:Token: None 2023-07-29T00:14:20.703 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:20 GMT /hiiqos-53/ 2023-07-29T00:14:20.704 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sNnB4vhsmHBCpY7VGemlM6zJVSU= 2023-07-29T00:14:20.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sNnB4vhsmHBCpY7VGemlM6zJVSU=', 'Content-Length': '0'} 2023-07-29T00:14:20.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037460e45d38eae3a-0064c459dc-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:20.707 DEBUG:boto:b'hiiqos-53a1000falsea2023-07-29T00:14:09.724Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:20.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-07-29T00:14:20.709 DEBUG:boto:path=/hiiqos-53/b 2023-07-29T00:14:20.709 DEBUG:boto:auth_path=/hiiqos-53/b 2023-07-29T00:14:20.710 DEBUG:boto:path=/hiiqos-53/b?versionId=null 2023-07-29T00:14:20.710 DEBUG:boto:auth_path=/hiiqos-53/b?versionId=null 2023-07-29T00:14:20.711 DEBUG:boto:Method: GET 2023-07-29T00:14:20.711 DEBUG:boto:Path: /hiiqos-53/b?versionId=null 2023-07-29T00:14:20.712 DEBUG:boto:Data: 2023-07-29T00:14:20.712 DEBUG:boto:Headers: {} 2023-07-29T00:14:20.712 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:20.713 DEBUG:boto:Port: 8001 2023-07-29T00:14:20.713 DEBUG:boto:Params: {} 2023-07-29T00:14:20.714 DEBUG:boto:Token: None 2023-07-29T00:14:20.714 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:20 GMT /hiiqos-53/b?versionId=null 2023-07-29T00:14:20.714 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+tQmnunPy1SGgf3zv7G8M8TqV+0= 2023-07-29T00:14:20.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+tQmnunPy1SGgf3zv7G8M8TqV+0=', 'Content-Length': '0'} 2023-07-29T00:14:20.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-53:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c70a07d9aaedee05-0064c459dc-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:20.760 DEBUG:boto:path=/hiiqos-53/b 2023-07-29T00:14:20.761 DEBUG:boto:auth_path=/hiiqos-53/b 2023-07-29T00:14:20.762 DEBUG:boto:path=/hiiqos-53/b?versionId=null 2023-07-29T00:14:20.762 DEBUG:boto:auth_path=/hiiqos-53/b?versionId=null 2023-07-29T00:14:20.762 DEBUG:boto:Method: GET 2023-07-29T00:14:20.763 DEBUG:boto:Path: /hiiqos-53/b?versionId=null 2023-07-29T00:14:20.763 DEBUG:boto:Data: 2023-07-29T00:14:20.764 DEBUG:boto:Headers: {} 2023-07-29T00:14:20.764 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:20.765 DEBUG:boto:Port: 8000 2023-07-29T00:14:20.765 DEBUG:boto:Params: {} 2023-07-29T00:14:20.766 DEBUG:boto:Token: None 2023-07-29T00:14:20.766 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:20 GMT /hiiqos-53/b?versionId=null 2023-07-29T00:14:20.767 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+tQmnunPy1SGgf3zv7G8M8TqV+0= 2023-07-29T00:14:20.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+tQmnunPy1SGgf3zv7G8M8TqV+0=', 'Content-Length': '0'} 2023-07-29T00:14:20.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6fbcd1adec135c0-0064c459dc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:20.770 DEBUG:boto:path=/hiiqos-53/b 2023-07-29T00:14:20.770 DEBUG:boto:auth_path=/hiiqos-53/b 2023-07-29T00:14:20.771 DEBUG:boto:path=/hiiqos-53/b?versionId=null 2023-07-29T00:14:20.771 DEBUG:boto:auth_path=/hiiqos-53/b?versionId=null 2023-07-29T00:14:20.771 DEBUG:boto:Method: HEAD 2023-07-29T00:14:20.772 DEBUG:boto:Path: /hiiqos-53/b?versionId=null 2023-07-29T00:14:20.772 DEBUG:boto:Data: 2023-07-29T00:14:20.772 DEBUG:boto:Headers: {} 2023-07-29T00:14:20.773 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:20.773 DEBUG:boto:Port: 8001 2023-07-29T00:14:20.773 DEBUG:boto:Params: {} 2023-07-29T00:14:20.774 DEBUG:boto:Token: None 2023-07-29T00:14:20.774 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:20 GMT /hiiqos-53/b?versionId=null 2023-07-29T00:14:20.775 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WDm5/Jf/XS2pAPzhXdrCC4K/k3U= 2023-07-29T00:14:20.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WDm5/Jf/XS2pAPzhXdrCC4K/k3U=', 'Content-Length': '0'} 2023-07-29T00:14:20.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-53:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c4143ed10a089c9-0064c459dc-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:20.778 DEBUG:boto:path=/hiiqos-53/b 2023-07-29T00:14:20.778 DEBUG:boto:auth_path=/hiiqos-53/b 2023-07-29T00:14:20.778 DEBUG:boto:path=/hiiqos-53/b?versionId=null 2023-07-29T00:14:20.779 DEBUG:boto:auth_path=/hiiqos-53/b?versionId=null 2023-07-29T00:14:20.779 DEBUG:boto:Method: HEAD 2023-07-29T00:14:20.779 DEBUG:boto:Path: /hiiqos-53/b?versionId=null 2023-07-29T00:14:20.780 DEBUG:boto:Data: 2023-07-29T00:14:20.780 DEBUG:boto:Headers: {} 2023-07-29T00:14:20.780 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:20.781 DEBUG:boto:Port: 8000 2023-07-29T00:14:20.781 DEBUG:boto:Params: {} 2023-07-29T00:14:20.781 DEBUG:boto:Token: None 2023-07-29T00:14:20.782 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:20 GMT /hiiqos-53/b?versionId=null 2023-07-29T00:14:20.782 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WDm5/Jf/XS2pAPzhXdrCC4K/k3U= 2023-07-29T00:14:20.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WDm5/Jf/XS2pAPzhXdrCC4K/k3U=', 'Content-Length': '0'} 2023-07-29T00:14:20.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f0420cc73fb7842-0064c459dc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:20.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-07-29T00:14:20.786 DEBUG:boto:path=/hiiqos-53/b 2023-07-29T00:14:20.786 DEBUG:boto:auth_path=/hiiqos-53/b 2023-07-29T00:14:20.786 DEBUG:boto:path=/hiiqos-53/b?versionId=null 2023-07-29T00:14:20.787 DEBUG:boto:auth_path=/hiiqos-53/b?versionId=null 2023-07-29T00:14:20.787 DEBUG:boto:Method: GET 2023-07-29T00:14:20.787 DEBUG:boto:Path: /hiiqos-53/b?versionId=null 2023-07-29T00:14:20.788 DEBUG:boto:Data: 2023-07-29T00:14:20.788 DEBUG:boto:Headers: {} 2023-07-29T00:14:20.789 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:20.789 DEBUG:boto:Port: 8001 2023-07-29T00:14:20.789 DEBUG:boto:Params: {} 2023-07-29T00:14:20.790 DEBUG:boto:Token: None 2023-07-29T00:14:20.790 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:20 GMT /hiiqos-53/b?versionId=null 2023-07-29T00:14:20.791 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+tQmnunPy1SGgf3zv7G8M8TqV+0= 2023-07-29T00:14:20.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+tQmnunPy1SGgf3zv7G8M8TqV+0=', 'Content-Length': '0'} 2023-07-29T00:14:20.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-53:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0b9a312141e1baa-0064c459dc-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:20.794 DEBUG:boto:path=/hiiqos-53/b 2023-07-29T00:14:20.794 DEBUG:boto:auth_path=/hiiqos-53/b 2023-07-29T00:14:20.794 DEBUG:boto:path=/hiiqos-53/b?versionId=null 2023-07-29T00:14:20.795 DEBUG:boto:auth_path=/hiiqos-53/b?versionId=null 2023-07-29T00:14:20.795 DEBUG:boto:Method: GET 2023-07-29T00:14:20.796 DEBUG:boto:Path: /hiiqos-53/b?versionId=null 2023-07-29T00:14:20.796 DEBUG:boto:Data: 2023-07-29T00:14:20.796 DEBUG:boto:Headers: {} 2023-07-29T00:14:20.797 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:20.797 DEBUG:boto:Port: 8000 2023-07-29T00:14:20.797 DEBUG:boto:Params: {} 2023-07-29T00:14:20.798 DEBUG:boto:Token: None 2023-07-29T00:14:20.798 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:20 GMT /hiiqos-53/b?versionId=null 2023-07-29T00:14:20.799 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+tQmnunPy1SGgf3zv7G8M8TqV+0= 2023-07-29T00:14:20.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+tQmnunPy1SGgf3zv7G8M8TqV+0=', 'Content-Length': '0'} 2023-07-29T00:14:20.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000203a4911a43db7e9-0064c459dc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:20.844 DEBUG:boto:path=/hiiqos-53/ 2023-07-29T00:14:20.845 DEBUG:boto:auth_path=/hiiqos-53/ 2023-07-29T00:14:20.846 DEBUG:boto:path=/hiiqos-53/?prefix=b 2023-07-29T00:14:20.846 DEBUG:boto:auth_path=/hiiqos-53/?prefix=b 2023-07-29T00:14:20.846 DEBUG:boto:Method: GET 2023-07-29T00:14:20.847 DEBUG:boto:Path: /hiiqos-53/?prefix=b 2023-07-29T00:14:20.847 DEBUG:boto:Data: 2023-07-29T00:14:20.847 DEBUG:boto:Headers: {} 2023-07-29T00:14:20.848 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:20.848 DEBUG:boto:Port: 8001 2023-07-29T00:14:20.849 DEBUG:boto:Params: {} 2023-07-29T00:14:20.849 DEBUG:boto:Token: None 2023-07-29T00:14:20.849 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:20 GMT /hiiqos-53/ 2023-07-29T00:14:20.850 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sNnB4vhsmHBCpY7VGemlM6zJVSU= 2023-07-29T00:14:20.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sNnB4vhsmHBCpY7VGemlM6zJVSU=', 'Content-Length': '0'} 2023-07-29T00:14:20.853 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015980e620d50a620-0064c459dc-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:20.853 DEBUG:boto:b'hiiqos-53b1000falseb2023-07-29T00:14:09.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:20.854 DEBUG:boto:path=/hiiqos-53/ 2023-07-29T00:14:20.855 DEBUG:boto:auth_path=/hiiqos-53/ 2023-07-29T00:14:20.855 DEBUG:boto:path=/hiiqos-53/?prefix=b 2023-07-29T00:14:20.855 DEBUG:boto:auth_path=/hiiqos-53/?prefix=b 2023-07-29T00:14:20.856 DEBUG:boto:Method: GET 2023-07-29T00:14:20.856 DEBUG:boto:Path: /hiiqos-53/?prefix=b 2023-07-29T00:14:20.857 DEBUG:boto:Data: 2023-07-29T00:14:20.857 DEBUG:boto:Headers: {} 2023-07-29T00:14:20.857 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:20.858 DEBUG:boto:Port: 8000 2023-07-29T00:14:20.858 DEBUG:boto:Params: {} 2023-07-29T00:14:20.858 DEBUG:boto:Token: None 2023-07-29T00:14:20.859 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:20 GMT /hiiqos-53/ 2023-07-29T00:14:20.859 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sNnB4vhsmHBCpY7VGemlM6zJVSU= 2023-07-29T00:14:20.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sNnB4vhsmHBCpY7VGemlM6zJVSU=', 'Content-Length': '0'} 2023-07-29T00:14:20.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080c47988c2fe4a94-0064c459dc-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:20.863 DEBUG:boto:b'hiiqos-53b1000falseb2023-07-29T00:14:09.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:20.863 DEBUG:boto:path=/hiiqos-53/b 2023-07-29T00:14:20.864 DEBUG:boto:auth_path=/hiiqos-53/b 2023-07-29T00:14:20.864 DEBUG:boto:Method: HEAD 2023-07-29T00:14:20.865 DEBUG:boto:Path: /hiiqos-53/b 2023-07-29T00:14:20.865 DEBUG:boto:Data: 2023-07-29T00:14:20.865 DEBUG:boto:Headers: {} 2023-07-29T00:14:20.866 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:20.866 DEBUG:boto:Port: 8001 2023-07-29T00:14:20.866 DEBUG:boto:Params: {} 2023-07-29T00:14:20.867 DEBUG:boto:Token: None 2023-07-29T00:14:20.867 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:20 GMT /hiiqos-53/b 2023-07-29T00:14:20.867 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lVr2ADj7o+DtsrEdV/Ig2OBF5ys= 2023-07-29T00:14:20.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lVr2ADj7o+DtsrEdV/Ig2OBF5ys=', 'Content-Length': '0'} 2023-07-29T00:14:20.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-53:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdaf2a6c775a989e-0064c459dc-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:20.870 DEBUG:boto:path=/hiiqos-53/b 2023-07-29T00:14:20.871 DEBUG:boto:auth_path=/hiiqos-53/b 2023-07-29T00:14:20.871 DEBUG:boto:Method: HEAD 2023-07-29T00:14:20.872 DEBUG:boto:Path: /hiiqos-53/b 2023-07-29T00:14:20.872 DEBUG:boto:Data: 2023-07-29T00:14:20.872 DEBUG:boto:Headers: {} 2023-07-29T00:14:20.873 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:20.873 DEBUG:boto:Port: 8000 2023-07-29T00:14:20.873 DEBUG:boto:Params: {} 2023-07-29T00:14:20.874 DEBUG:boto:Token: None 2023-07-29T00:14:20.874 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:20 GMT /hiiqos-53/b 2023-07-29T00:14:20.875 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lVr2ADj7o+DtsrEdV/Ig2OBF5ys= 2023-07-29T00:14:20.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lVr2ADj7o+DtsrEdV/Ig2OBF5ys=', 'Content-Length': '0'} 2023-07-29T00:14:20.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094cd602d9b9779d3-0064c459dc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:20.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-07-29T00:14:20.878 DEBUG:boto:path=/hiiqos-53/b 2023-07-29T00:14:20.878 DEBUG:boto:auth_path=/hiiqos-53/b 2023-07-29T00:14:20.879 DEBUG:boto:Method: GET 2023-07-29T00:14:20.879 DEBUG:boto:Path: /hiiqos-53/b 2023-07-29T00:14:20.879 DEBUG:boto:Data: 2023-07-29T00:14:20.880 DEBUG:boto:Headers: {} 2023-07-29T00:14:20.880 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:20.880 DEBUG:boto:Port: 8001 2023-07-29T00:14:20.881 DEBUG:boto:Params: {} 2023-07-29T00:14:20.881 DEBUG:boto:Token: None 2023-07-29T00:14:20.882 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:20 GMT /hiiqos-53/b 2023-07-29T00:14:20.882 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O1i6huGcnbEY00PobevGBAQyJOI= 2023-07-29T00:14:20.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O1i6huGcnbEY00PobevGBAQyJOI=', 'Content-Length': '0'} 2023-07-29T00:14:20.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-53:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be4da38bbd24badc-0064c459dc-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:20.928 DEBUG:boto:path=/hiiqos-53/b 2023-07-29T00:14:20.929 DEBUG:boto:auth_path=/hiiqos-53/b 2023-07-29T00:14:20.930 DEBUG:boto:Method: GET 2023-07-29T00:14:20.931 DEBUG:boto:Path: /hiiqos-53/b 2023-07-29T00:14:20.932 DEBUG:boto:Data: 2023-07-29T00:14:20.932 DEBUG:boto:Headers: {} 2023-07-29T00:14:20.932 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:20.933 DEBUG:boto:Port: 8000 2023-07-29T00:14:20.933 DEBUG:boto:Params: {} 2023-07-29T00:14:20.934 DEBUG:boto:Token: None 2023-07-29T00:14:20.934 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:20 GMT /hiiqos-53/b 2023-07-29T00:14:20.935 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O1i6huGcnbEY00PobevGBAQyJOI= 2023-07-29T00:14:20.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O1i6huGcnbEY00PobevGBAQyJOI=', 'Content-Length': '0'} 2023-07-29T00:14:20.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057c384cbeb7a2796-0064c459dc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:20.939 DEBUG:boto:path=/hiiqos-53/ 2023-07-29T00:14:20.940 DEBUG:boto:auth_path=/hiiqos-53/ 2023-07-29T00:14:20.941 DEBUG:boto:path=/hiiqos-53/?prefix=b 2023-07-29T00:14:20.942 DEBUG:boto:auth_path=/hiiqos-53/?prefix=b 2023-07-29T00:14:20.943 DEBUG:boto:Method: GET 2023-07-29T00:14:20.944 DEBUG:boto:Path: /hiiqos-53/?prefix=b 2023-07-29T00:14:20.945 DEBUG:boto:Data: 2023-07-29T00:14:20.945 DEBUG:boto:Headers: {} 2023-07-29T00:14:20.946 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:20.947 DEBUG:boto:Port: 8001 2023-07-29T00:14:20.947 DEBUG:boto:Params: {} 2023-07-29T00:14:20.948 DEBUG:boto:Token: None 2023-07-29T00:14:20.949 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:20 GMT /hiiqos-53/ 2023-07-29T00:14:20.950 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sNnB4vhsmHBCpY7VGemlM6zJVSU= 2023-07-29T00:14:20.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sNnB4vhsmHBCpY7VGemlM6zJVSU=', 'Content-Length': '0'} 2023-07-29T00:14:20.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007540aa60fbe2a38d-0064c459dc-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:20.954 DEBUG:boto:b'hiiqos-53b1000falseb2023-07-29T00:14:09.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:20.955 DEBUG:boto:path=/hiiqos-53/ 2023-07-29T00:14:20.955 DEBUG:boto:auth_path=/hiiqos-53/ 2023-07-29T00:14:20.956 DEBUG:boto:path=/hiiqos-53/?prefix=b 2023-07-29T00:14:20.956 DEBUG:boto:auth_path=/hiiqos-53/?prefix=b 2023-07-29T00:14:20.957 DEBUG:boto:Method: GET 2023-07-29T00:14:20.957 DEBUG:boto:Path: /hiiqos-53/?prefix=b 2023-07-29T00:14:20.957 DEBUG:boto:Data: 2023-07-29T00:14:20.958 DEBUG:boto:Headers: {} 2023-07-29T00:14:20.958 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:20.959 DEBUG:boto:Port: 8000 2023-07-29T00:14:20.959 DEBUG:boto:Params: {} 2023-07-29T00:14:20.960 DEBUG:boto:Token: None 2023-07-29T00:14:20.960 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:20 GMT /hiiqos-53/ 2023-07-29T00:14:20.961 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sNnB4vhsmHBCpY7VGemlM6zJVSU= 2023-07-29T00:14:20.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sNnB4vhsmHBCpY7VGemlM6zJVSU=', 'Content-Length': '0'} 2023-07-29T00:14:20.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b629d79b916fad7d-0064c459dc-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:20.964 DEBUG:boto:b'hiiqos-53b1000falseb2023-07-29T00:14:09.744Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:20.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-07-29T00:14:20.965 DEBUG:boto:path=/hiiqos-53/c 2023-07-29T00:14:20.966 DEBUG:boto:auth_path=/hiiqos-53/c 2023-07-29T00:14:20.966 DEBUG:boto:path=/hiiqos-53/c?versionId=null 2023-07-29T00:14:20.967 DEBUG:boto:auth_path=/hiiqos-53/c?versionId=null 2023-07-29T00:14:20.967 DEBUG:boto:Method: GET 2023-07-29T00:14:20.967 DEBUG:boto:Path: /hiiqos-53/c?versionId=null 2023-07-29T00:14:20.968 DEBUG:boto:Data: 2023-07-29T00:14:20.968 DEBUG:boto:Headers: {} 2023-07-29T00:14:20.969 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:20.969 DEBUG:boto:Port: 8001 2023-07-29T00:14:20.970 DEBUG:boto:Params: {} 2023-07-29T00:14:20.970 DEBUG:boto:Token: None 2023-07-29T00:14:20.971 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:20 GMT /hiiqos-53/c?versionId=null 2023-07-29T00:14:20.972 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+Me2TLZUBawOfbqrnMnstFU87IM= 2023-07-29T00:14:20.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+Me2TLZUBawOfbqrnMnstFU87IM=', 'Content-Length': '0'} 2023-07-29T00:14:20.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-53:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f0c59cbab866f00-0064c459dc-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:20.975 DEBUG:boto:path=/hiiqos-53/c 2023-07-29T00:14:20.975 DEBUG:boto:auth_path=/hiiqos-53/c 2023-07-29T00:14:20.976 DEBUG:boto:path=/hiiqos-53/c?versionId=null 2023-07-29T00:14:20.976 DEBUG:boto:auth_path=/hiiqos-53/c?versionId=null 2023-07-29T00:14:20.977 DEBUG:boto:Method: GET 2023-07-29T00:14:20.977 DEBUG:boto:Path: /hiiqos-53/c?versionId=null 2023-07-29T00:14:20.977 DEBUG:boto:Data: 2023-07-29T00:14:20.978 DEBUG:boto:Headers: {} 2023-07-29T00:14:20.978 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:20.979 DEBUG:boto:Port: 8000 2023-07-29T00:14:20.979 DEBUG:boto:Params: {} 2023-07-29T00:14:20.980 DEBUG:boto:Token: None 2023-07-29T00:14:20.981 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:20 GMT /hiiqos-53/c?versionId=null 2023-07-29T00:14:20.982 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+Me2TLZUBawOfbqrnMnstFU87IM= 2023-07-29T00:14:20.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+Me2TLZUBawOfbqrnMnstFU87IM=', 'Content-Length': '0'} 2023-07-29T00:14:20.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e52e628efbf4cab-0064c459dc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:20.985 DEBUG:boto:path=/hiiqos-53/c 2023-07-29T00:14:20.985 DEBUG:boto:auth_path=/hiiqos-53/c 2023-07-29T00:14:20.985 DEBUG:boto:path=/hiiqos-53/c?versionId=null 2023-07-29T00:14:20.986 DEBUG:boto:auth_path=/hiiqos-53/c?versionId=null 2023-07-29T00:14:20.986 DEBUG:boto:Method: HEAD 2023-07-29T00:14:20.987 DEBUG:boto:Path: /hiiqos-53/c?versionId=null 2023-07-29T00:14:20.987 DEBUG:boto:Data: 2023-07-29T00:14:20.987 DEBUG:boto:Headers: {} 2023-07-29T00:14:20.988 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:20.988 DEBUG:boto:Port: 8001 2023-07-29T00:14:20.989 DEBUG:boto:Params: {} 2023-07-29T00:14:20.989 DEBUG:boto:Token: None 2023-07-29T00:14:20.989 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:20 GMT /hiiqos-53/c?versionId=null 2023-07-29T00:14:20.990 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BfqFl5nbD1xg+yberXGaXsyJdWc= 2023-07-29T00:14:20.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:20 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BfqFl5nbD1xg+yberXGaXsyJdWc=', 'Content-Length': '0'} 2023-07-29T00:14:20.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-53:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002436763ad63ec4d9-0064c459dc-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:20.993 DEBUG:boto:path=/hiiqos-53/c 2023-07-29T00:14:20.994 DEBUG:boto:auth_path=/hiiqos-53/c 2023-07-29T00:14:20.995 DEBUG:boto:path=/hiiqos-53/c?versionId=null 2023-07-29T00:14:20.995 DEBUG:boto:auth_path=/hiiqos-53/c?versionId=null 2023-07-29T00:14:20.996 DEBUG:boto:Method: HEAD 2023-07-29T00:14:20.997 DEBUG:boto:Path: /hiiqos-53/c?versionId=null 2023-07-29T00:14:20.998 DEBUG:boto:Data: 2023-07-29T00:14:20.998 DEBUG:boto:Headers: {} 2023-07-29T00:14:20.999 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:21.000 DEBUG:boto:Port: 8000 2023-07-29T00:14:21.000 DEBUG:boto:Params: {} 2023-07-29T00:14:21.001 DEBUG:boto:Token: None 2023-07-29T00:14:21.002 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:21 GMT /hiiqos-53/c?versionId=null 2023-07-29T00:14:21.002 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qSOqNCN4Zm+8JyC8ZuKJD2leg8E= 2023-07-29T00:14:21.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qSOqNCN4Zm+8JyC8ZuKJD2leg8E=', 'Content-Length': '0'} 2023-07-29T00:14:21.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004251bbc57eb7a236-0064c459dc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:21.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-07-29T00:14:21.007 DEBUG:boto:path=/hiiqos-53/c 2023-07-29T00:14:21.008 DEBUG:boto:auth_path=/hiiqos-53/c 2023-07-29T00:14:21.008 DEBUG:boto:path=/hiiqos-53/c?versionId=null 2023-07-29T00:14:21.009 DEBUG:boto:auth_path=/hiiqos-53/c?versionId=null 2023-07-29T00:14:21.010 DEBUG:boto:Method: GET 2023-07-29T00:14:21.010 DEBUG:boto:Path: /hiiqos-53/c?versionId=null 2023-07-29T00:14:21.011 DEBUG:boto:Data: 2023-07-29T00:14:21.012 DEBUG:boto:Headers: {} 2023-07-29T00:14:21.012 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:21.013 DEBUG:boto:Port: 8001 2023-07-29T00:14:21.014 DEBUG:boto:Params: {} 2023-07-29T00:14:21.015 DEBUG:boto:Token: None 2023-07-29T00:14:21.016 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:21 GMT /hiiqos-53/c?versionId=null 2023-07-29T00:14:21.017 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yRyNvEOv3NugEnafMS1Vx7hSRYI= 2023-07-29T00:14:21.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yRyNvEOv3NugEnafMS1Vx7hSRYI=', 'Content-Length': '0'} 2023-07-29T00:14:21.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-53:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d81896238bf2fee8-0064c459dc-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:20 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:21.064 DEBUG:boto:path=/hiiqos-53/c 2023-07-29T00:14:21.065 DEBUG:boto:auth_path=/hiiqos-53/c 2023-07-29T00:14:21.065 DEBUG:boto:path=/hiiqos-53/c?versionId=null 2023-07-29T00:14:21.066 DEBUG:boto:auth_path=/hiiqos-53/c?versionId=null 2023-07-29T00:14:21.066 DEBUG:boto:Method: GET 2023-07-29T00:14:21.067 DEBUG:boto:Path: /hiiqos-53/c?versionId=null 2023-07-29T00:14:21.067 DEBUG:boto:Data: 2023-07-29T00:14:21.068 DEBUG:boto:Headers: {} 2023-07-29T00:14:21.068 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:21.068 DEBUG:boto:Port: 8000 2023-07-29T00:14:21.069 DEBUG:boto:Params: {} 2023-07-29T00:14:21.070 DEBUG:boto:Token: None 2023-07-29T00:14:21.070 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:21 GMT /hiiqos-53/c?versionId=null 2023-07-29T00:14:21.071 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yRyNvEOv3NugEnafMS1Vx7hSRYI= 2023-07-29T00:14:21.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yRyNvEOv3NugEnafMS1Vx7hSRYI=', 'Content-Length': '0'} 2023-07-29T00:14:21.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004aef56e8a2a96c2a-0064c459dd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:21.075 DEBUG:boto:path=/hiiqos-53/ 2023-07-29T00:14:21.076 DEBUG:boto:auth_path=/hiiqos-53/ 2023-07-29T00:14:21.076 DEBUG:boto:path=/hiiqos-53/?prefix=c 2023-07-29T00:14:21.077 DEBUG:boto:auth_path=/hiiqos-53/?prefix=c 2023-07-29T00:14:21.078 DEBUG:boto:Method: GET 2023-07-29T00:14:21.079 DEBUG:boto:Path: /hiiqos-53/?prefix=c 2023-07-29T00:14:21.079 DEBUG:boto:Data: 2023-07-29T00:14:21.080 DEBUG:boto:Headers: {} 2023-07-29T00:14:21.080 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:21.081 DEBUG:boto:Port: 8001 2023-07-29T00:14:21.081 DEBUG:boto:Params: {} 2023-07-29T00:14:21.082 DEBUG:boto:Token: None 2023-07-29T00:14:21.082 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:21 GMT /hiiqos-53/ 2023-07-29T00:14:21.083 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HgZvQJ0dhuVtojzt7HOVfDO3lng= 2023-07-29T00:14:21.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HgZvQJ0dhuVtojzt7HOVfDO3lng=', 'Content-Length': '0'} 2023-07-29T00:14:21.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b8fa61f798a8eb6-0064c459dd-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:21.088 DEBUG:boto:b'hiiqos-53c1000falsec2023-07-29T00:14:09.763Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:21.089 DEBUG:boto:path=/hiiqos-53/ 2023-07-29T00:14:21.089 DEBUG:boto:auth_path=/hiiqos-53/ 2023-07-29T00:14:21.089 DEBUG:boto:path=/hiiqos-53/?prefix=c 2023-07-29T00:14:21.090 DEBUG:boto:auth_path=/hiiqos-53/?prefix=c 2023-07-29T00:14:21.090 DEBUG:boto:Method: GET 2023-07-29T00:14:21.091 DEBUG:boto:Path: /hiiqos-53/?prefix=c 2023-07-29T00:14:21.091 DEBUG:boto:Data: 2023-07-29T00:14:21.092 DEBUG:boto:Headers: {} 2023-07-29T00:14:21.092 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:21.092 DEBUG:boto:Port: 8000 2023-07-29T00:14:21.093 DEBUG:boto:Params: {} 2023-07-29T00:14:21.093 DEBUG:boto:Token: None 2023-07-29T00:14:21.094 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:21 GMT /hiiqos-53/ 2023-07-29T00:14:21.094 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HgZvQJ0dhuVtojzt7HOVfDO3lng= 2023-07-29T00:14:21.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HgZvQJ0dhuVtojzt7HOVfDO3lng=', 'Content-Length': '0'} 2023-07-29T00:14:21.097 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000636caa2ed880ab2a-0064c459dd-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:21.097 DEBUG:boto:b'hiiqos-53c1000falsec2023-07-29T00:14:09.763Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:21.098 DEBUG:boto:path=/hiiqos-53/c 2023-07-29T00:14:21.098 DEBUG:boto:auth_path=/hiiqos-53/c 2023-07-29T00:14:21.099 DEBUG:boto:Method: HEAD 2023-07-29T00:14:21.099 DEBUG:boto:Path: /hiiqos-53/c 2023-07-29T00:14:21.099 DEBUG:boto:Data: 2023-07-29T00:14:21.100 DEBUG:boto:Headers: {} 2023-07-29T00:14:21.100 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:21.100 DEBUG:boto:Port: 8001 2023-07-29T00:14:21.101 DEBUG:boto:Params: {} 2023-07-29T00:14:21.101 DEBUG:boto:Token: None 2023-07-29T00:14:21.102 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:21 GMT /hiiqos-53/c 2023-07-29T00:14:21.102 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w26HCTCj4SK6rK9gsItFDk0OOuE= 2023-07-29T00:14:21.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w26HCTCj4SK6rK9gsItFDk0OOuE=', 'Content-Length': '0'} 2023-07-29T00:14:21.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-53:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ba62409c278495f-0064c459dd-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:21.105 DEBUG:boto:path=/hiiqos-53/c 2023-07-29T00:14:21.105 DEBUG:boto:auth_path=/hiiqos-53/c 2023-07-29T00:14:21.106 DEBUG:boto:Method: HEAD 2023-07-29T00:14:21.106 DEBUG:boto:Path: /hiiqos-53/c 2023-07-29T00:14:21.106 DEBUG:boto:Data: 2023-07-29T00:14:21.107 DEBUG:boto:Headers: {} 2023-07-29T00:14:21.107 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:21.107 DEBUG:boto:Port: 8000 2023-07-29T00:14:21.108 DEBUG:boto:Params: {} 2023-07-29T00:14:21.108 DEBUG:boto:Token: None 2023-07-29T00:14:21.108 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:21 GMT /hiiqos-53/c 2023-07-29T00:14:21.109 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w26HCTCj4SK6rK9gsItFDk0OOuE= 2023-07-29T00:14:21.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w26HCTCj4SK6rK9gsItFDk0OOuE=', 'Content-Length': '0'} 2023-07-29T00:14:21.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fac1f4873c35d904-0064c459dd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:21.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-07-29T00:14:21.112 DEBUG:boto:path=/hiiqos-53/c 2023-07-29T00:14:21.113 DEBUG:boto:auth_path=/hiiqos-53/c 2023-07-29T00:14:21.113 DEBUG:boto:Method: GET 2023-07-29T00:14:21.114 DEBUG:boto:Path: /hiiqos-53/c 2023-07-29T00:14:21.114 DEBUG:boto:Data: 2023-07-29T00:14:21.114 DEBUG:boto:Headers: {} 2023-07-29T00:14:21.115 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:21.116 DEBUG:boto:Port: 8001 2023-07-29T00:14:21.117 DEBUG:boto:Params: {} 2023-07-29T00:14:21.118 DEBUG:boto:Token: None 2023-07-29T00:14:21.119 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:21 GMT /hiiqos-53/c 2023-07-29T00:14:21.119 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:irQStlZcDTkoMDMCnWw0NzmF4i0= 2023-07-29T00:14:21.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:irQStlZcDTkoMDMCnWw0NzmF4i0=', 'Content-Length': '0'} 2023-07-29T00:14:21.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-53:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1e7bf89e4036d81-0064c459dd-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:21.123 DEBUG:boto:path=/hiiqos-53/c 2023-07-29T00:14:21.124 DEBUG:boto:auth_path=/hiiqos-53/c 2023-07-29T00:14:21.124 DEBUG:boto:Method: GET 2023-07-29T00:14:21.124 DEBUG:boto:Path: /hiiqos-53/c 2023-07-29T00:14:21.125 DEBUG:boto:Data: 2023-07-29T00:14:21.125 DEBUG:boto:Headers: {} 2023-07-29T00:14:21.125 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:21.126 DEBUG:boto:Port: 8000 2023-07-29T00:14:21.126 DEBUG:boto:Params: {} 2023-07-29T00:14:21.127 DEBUG:boto:Token: None 2023-07-29T00:14:21.127 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:21 GMT /hiiqos-53/c 2023-07-29T00:14:21.127 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:irQStlZcDTkoMDMCnWw0NzmF4i0= 2023-07-29T00:14:21.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:irQStlZcDTkoMDMCnWw0NzmF4i0=', 'Content-Length': '0'} 2023-07-29T00:14:21.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa0f5a34a39d04f4-0064c459dd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:21.131 DEBUG:boto:path=/hiiqos-53/ 2023-07-29T00:14:21.131 DEBUG:boto:auth_path=/hiiqos-53/ 2023-07-29T00:14:21.132 DEBUG:boto:path=/hiiqos-53/?prefix=c 2023-07-29T00:14:21.132 DEBUG:boto:auth_path=/hiiqos-53/?prefix=c 2023-07-29T00:14:21.132 DEBUG:boto:Method: GET 2023-07-29T00:14:21.133 DEBUG:boto:Path: /hiiqos-53/?prefix=c 2023-07-29T00:14:21.133 DEBUG:boto:Data: 2023-07-29T00:14:21.133 DEBUG:boto:Headers: {} 2023-07-29T00:14:21.134 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:21.134 DEBUG:boto:Port: 8001 2023-07-29T00:14:21.135 DEBUG:boto:Params: {} 2023-07-29T00:14:21.135 DEBUG:boto:Token: None 2023-07-29T00:14:21.135 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:21 GMT /hiiqos-53/ 2023-07-29T00:14:21.136 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HgZvQJ0dhuVtojzt7HOVfDO3lng= 2023-07-29T00:14:21.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HgZvQJ0dhuVtojzt7HOVfDO3lng=', 'Content-Length': '0'} 2023-07-29T00:14:21.139 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c2358d278ac2554-0064c459dd-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:21.139 DEBUG:boto:b'hiiqos-53c1000falsec2023-07-29T00:14:09.763Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:21.140 DEBUG:boto:path=/hiiqos-53/ 2023-07-29T00:14:21.140 DEBUG:boto:auth_path=/hiiqos-53/ 2023-07-29T00:14:21.141 DEBUG:boto:path=/hiiqos-53/?prefix=c 2023-07-29T00:14:21.141 DEBUG:boto:auth_path=/hiiqos-53/?prefix=c 2023-07-29T00:14:21.142 DEBUG:boto:Method: GET 2023-07-29T00:14:21.142 DEBUG:boto:Path: /hiiqos-53/?prefix=c 2023-07-29T00:14:21.143 DEBUG:boto:Data: 2023-07-29T00:14:21.143 DEBUG:boto:Headers: {} 2023-07-29T00:14:21.143 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:21.144 DEBUG:boto:Port: 8000 2023-07-29T00:14:21.144 DEBUG:boto:Params: {} 2023-07-29T00:14:21.145 DEBUG:boto:Token: None 2023-07-29T00:14:21.145 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:21 GMT /hiiqos-53/ 2023-07-29T00:14:21.146 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HgZvQJ0dhuVtojzt7HOVfDO3lng= 2023-07-29T00:14:21.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HgZvQJ0dhuVtojzt7HOVfDO3lng=', 'Content-Length': '0'} 2023-07-29T00:14:21.149 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e3c40669261c0a0-0064c459dd-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:21.150 DEBUG:boto:b'hiiqos-53c1000falsec2023-07-29T00:14:09.763Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:21.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-07-29T00:14:21.152 DEBUG:boto:path=/hiiqos-53/d 2023-07-29T00:14:21.153 DEBUG:boto:auth_path=/hiiqos-53/d 2023-07-29T00:14:21.154 DEBUG:boto:path=/hiiqos-53/d?versionId=null 2023-07-29T00:14:21.155 DEBUG:boto:auth_path=/hiiqos-53/d?versionId=null 2023-07-29T00:14:21.155 DEBUG:boto:Method: GET 2023-07-29T00:14:21.156 DEBUG:boto:Path: /hiiqos-53/d?versionId=null 2023-07-29T00:14:21.157 DEBUG:boto:Data: 2023-07-29T00:14:21.157 DEBUG:boto:Headers: {} 2023-07-29T00:14:21.157 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:21.158 DEBUG:boto:Port: 8001 2023-07-29T00:14:21.158 DEBUG:boto:Params: {} 2023-07-29T00:14:21.159 DEBUG:boto:Token: None 2023-07-29T00:14:21.159 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:21 GMT /hiiqos-53/d?versionId=null 2023-07-29T00:14:21.159 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eU2v+NZI3leXBlOv7iEKLNVXFnM= 2023-07-29T00:14:21.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eU2v+NZI3leXBlOv7iEKLNVXFnM=', 'Content-Length': '0'} 2023-07-29T00:14:21.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-53:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c060e0932eef794f-0064c459dd-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:21.163 DEBUG:boto:path=/hiiqos-53/d 2023-07-29T00:14:21.163 DEBUG:boto:auth_path=/hiiqos-53/d 2023-07-29T00:14:21.163 DEBUG:boto:path=/hiiqos-53/d?versionId=null 2023-07-29T00:14:21.164 DEBUG:boto:auth_path=/hiiqos-53/d?versionId=null 2023-07-29T00:14:21.164 DEBUG:boto:Method: GET 2023-07-29T00:14:21.165 DEBUG:boto:Path: /hiiqos-53/d?versionId=null 2023-07-29T00:14:21.165 DEBUG:boto:Data: 2023-07-29T00:14:21.165 DEBUG:boto:Headers: {} 2023-07-29T00:14:21.166 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:21.166 DEBUG:boto:Port: 8000 2023-07-29T00:14:21.166 DEBUG:boto:Params: {} 2023-07-29T00:14:21.167 DEBUG:boto:Token: None 2023-07-29T00:14:21.167 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:21 GMT /hiiqos-53/d?versionId=null 2023-07-29T00:14:21.168 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eU2v+NZI3leXBlOv7iEKLNVXFnM= 2023-07-29T00:14:21.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eU2v+NZI3leXBlOv7iEKLNVXFnM=', 'Content-Length': '0'} 2023-07-29T00:14:21.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064fa8fffe81f0e2c-0064c459dd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:21.212 DEBUG:boto:path=/hiiqos-53/d 2023-07-29T00:14:21.213 DEBUG:boto:auth_path=/hiiqos-53/d 2023-07-29T00:14:21.213 DEBUG:boto:path=/hiiqos-53/d?versionId=null 2023-07-29T00:14:21.214 DEBUG:boto:auth_path=/hiiqos-53/d?versionId=null 2023-07-29T00:14:21.214 DEBUG:boto:Method: HEAD 2023-07-29T00:14:21.214 DEBUG:boto:Path: /hiiqos-53/d?versionId=null 2023-07-29T00:14:21.215 DEBUG:boto:Data: 2023-07-29T00:14:21.215 DEBUG:boto:Headers: {} 2023-07-29T00:14:21.215 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:21.216 DEBUG:boto:Port: 8001 2023-07-29T00:14:21.216 DEBUG:boto:Params: {} 2023-07-29T00:14:21.216 DEBUG:boto:Token: None 2023-07-29T00:14:21.217 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:21 GMT /hiiqos-53/d?versionId=null 2023-07-29T00:14:21.217 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:s+44en+FVTWffXr4vXjQLAUQQqk= 2023-07-29T00:14:21.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:s+44en+FVTWffXr4vXjQLAUQQqk=', 'Content-Length': '0'} 2023-07-29T00:14:21.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-53:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058f46a914930a466-0064c459dd-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:21.220 DEBUG:boto:path=/hiiqos-53/d 2023-07-29T00:14:21.220 DEBUG:boto:auth_path=/hiiqos-53/d 2023-07-29T00:14:21.221 DEBUG:boto:path=/hiiqos-53/d?versionId=null 2023-07-29T00:14:21.221 DEBUG:boto:auth_path=/hiiqos-53/d?versionId=null 2023-07-29T00:14:21.221 DEBUG:boto:Method: HEAD 2023-07-29T00:14:21.222 DEBUG:boto:Path: /hiiqos-53/d?versionId=null 2023-07-29T00:14:21.222 DEBUG:boto:Data: 2023-07-29T00:14:21.222 DEBUG:boto:Headers: {} 2023-07-29T00:14:21.223 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:21.223 DEBUG:boto:Port: 8000 2023-07-29T00:14:21.223 DEBUG:boto:Params: {} 2023-07-29T00:14:21.224 DEBUG:boto:Token: None 2023-07-29T00:14:21.224 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:21 GMT /hiiqos-53/d?versionId=null 2023-07-29T00:14:21.225 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:s+44en+FVTWffXr4vXjQLAUQQqk= 2023-07-29T00:14:21.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:s+44en+FVTWffXr4vXjQLAUQQqk=', 'Content-Length': '0'} 2023-07-29T00:14:21.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d7f3e5f63b4af13-0064c459dd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:21.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-07-29T00:14:21.228 DEBUG:boto:path=/hiiqos-53/d 2023-07-29T00:14:21.228 DEBUG:boto:auth_path=/hiiqos-53/d 2023-07-29T00:14:21.229 DEBUG:boto:path=/hiiqos-53/d?versionId=null 2023-07-29T00:14:21.229 DEBUG:boto:auth_path=/hiiqos-53/d?versionId=null 2023-07-29T00:14:21.229 DEBUG:boto:Method: GET 2023-07-29T00:14:21.230 DEBUG:boto:Path: /hiiqos-53/d?versionId=null 2023-07-29T00:14:21.230 DEBUG:boto:Data: 2023-07-29T00:14:21.230 DEBUG:boto:Headers: {} 2023-07-29T00:14:21.231 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:21.231 DEBUG:boto:Port: 8001 2023-07-29T00:14:21.231 DEBUG:boto:Params: {} 2023-07-29T00:14:21.232 DEBUG:boto:Token: None 2023-07-29T00:14:21.232 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:21 GMT /hiiqos-53/d?versionId=null 2023-07-29T00:14:21.233 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eU2v+NZI3leXBlOv7iEKLNVXFnM= 2023-07-29T00:14:21.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eU2v+NZI3leXBlOv7iEKLNVXFnM=', 'Content-Length': '0'} 2023-07-29T00:14:21.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-53:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe75900c36b5e042-0064c459dd-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:21.236 DEBUG:boto:path=/hiiqos-53/d 2023-07-29T00:14:21.236 DEBUG:boto:auth_path=/hiiqos-53/d 2023-07-29T00:14:21.237 DEBUG:boto:path=/hiiqos-53/d?versionId=null 2023-07-29T00:14:21.237 DEBUG:boto:auth_path=/hiiqos-53/d?versionId=null 2023-07-29T00:14:21.237 DEBUG:boto:Method: GET 2023-07-29T00:14:21.238 DEBUG:boto:Path: /hiiqos-53/d?versionId=null 2023-07-29T00:14:21.238 DEBUG:boto:Data: 2023-07-29T00:14:21.238 DEBUG:boto:Headers: {} 2023-07-29T00:14:21.239 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:21.239 DEBUG:boto:Port: 8000 2023-07-29T00:14:21.239 DEBUG:boto:Params: {} 2023-07-29T00:14:21.240 DEBUG:boto:Token: None 2023-07-29T00:14:21.240 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:21 GMT /hiiqos-53/d?versionId=null 2023-07-29T00:14:21.241 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eU2v+NZI3leXBlOv7iEKLNVXFnM= 2023-07-29T00:14:21.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eU2v+NZI3leXBlOv7iEKLNVXFnM=', 'Content-Length': '0'} 2023-07-29T00:14:21.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008244185280239082-0064c459dd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:21.244 DEBUG:boto:path=/hiiqos-53/ 2023-07-29T00:14:21.244 DEBUG:boto:auth_path=/hiiqos-53/ 2023-07-29T00:14:21.245 DEBUG:boto:path=/hiiqos-53/?prefix=d 2023-07-29T00:14:21.245 DEBUG:boto:auth_path=/hiiqos-53/?prefix=d 2023-07-29T00:14:21.245 DEBUG:boto:Method: GET 2023-07-29T00:14:21.246 DEBUG:boto:Path: /hiiqos-53/?prefix=d 2023-07-29T00:14:21.246 DEBUG:boto:Data: 2023-07-29T00:14:21.246 DEBUG:boto:Headers: {} 2023-07-29T00:14:21.247 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:21.247 DEBUG:boto:Port: 8001 2023-07-29T00:14:21.248 DEBUG:boto:Params: {} 2023-07-29T00:14:21.248 DEBUG:boto:Token: None 2023-07-29T00:14:21.248 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:21 GMT /hiiqos-53/ 2023-07-29T00:14:21.249 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HgZvQJ0dhuVtojzt7HOVfDO3lng= 2023-07-29T00:14:21.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HgZvQJ0dhuVtojzt7HOVfDO3lng=', 'Content-Length': '0'} 2023-07-29T00:14:21.251 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fe95d08a410e1fc-0064c459dd-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:21.252 DEBUG:boto:b'hiiqos-53d1000falsed2023-07-29T00:14:09.821Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:21.253 DEBUG:boto:path=/hiiqos-53/ 2023-07-29T00:14:21.253 DEBUG:boto:auth_path=/hiiqos-53/ 2023-07-29T00:14:21.253 DEBUG:boto:path=/hiiqos-53/?prefix=d 2023-07-29T00:14:21.254 DEBUG:boto:auth_path=/hiiqos-53/?prefix=d 2023-07-29T00:14:21.254 DEBUG:boto:Method: GET 2023-07-29T00:14:21.254 DEBUG:boto:Path: /hiiqos-53/?prefix=d 2023-07-29T00:14:21.255 DEBUG:boto:Data: 2023-07-29T00:14:21.255 DEBUG:boto:Headers: {} 2023-07-29T00:14:21.256 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:21.256 DEBUG:boto:Port: 8000 2023-07-29T00:14:21.256 DEBUG:boto:Params: {} 2023-07-29T00:14:21.257 DEBUG:boto:Token: None 2023-07-29T00:14:21.257 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:21 GMT /hiiqos-53/ 2023-07-29T00:14:21.258 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HgZvQJ0dhuVtojzt7HOVfDO3lng= 2023-07-29T00:14:21.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HgZvQJ0dhuVtojzt7HOVfDO3lng=', 'Content-Length': '0'} 2023-07-29T00:14:21.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd99549fc4a325eb-0064c459dd-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:21.265 DEBUG:boto:b'hiiqos-53d1000falsed2023-07-29T00:14:09.821Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:21.266 DEBUG:boto:path=/hiiqos-53/d 2023-07-29T00:14:21.266 DEBUG:boto:auth_path=/hiiqos-53/d 2023-07-29T00:14:21.266 DEBUG:boto:Method: HEAD 2023-07-29T00:14:21.267 DEBUG:boto:Path: /hiiqos-53/d 2023-07-29T00:14:21.267 DEBUG:boto:Data: 2023-07-29T00:14:21.267 DEBUG:boto:Headers: {} 2023-07-29T00:14:21.268 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:21.268 DEBUG:boto:Port: 8001 2023-07-29T00:14:21.268 DEBUG:boto:Params: {} 2023-07-29T00:14:21.269 DEBUG:boto:Token: None 2023-07-29T00:14:21.269 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:21 GMT /hiiqos-53/d 2023-07-29T00:14:21.270 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5UjKYwIXQxCqpBVq5kWHMnAMDL4= 2023-07-29T00:14:21.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5UjKYwIXQxCqpBVq5kWHMnAMDL4=', 'Content-Length': '0'} 2023-07-29T00:14:21.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-53:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c65caecac43fc3a-0064c459dd-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:21.273 DEBUG:boto:path=/hiiqos-53/d 2023-07-29T00:14:21.273 DEBUG:boto:auth_path=/hiiqos-53/d 2023-07-29T00:14:21.273 DEBUG:boto:Method: HEAD 2023-07-29T00:14:21.274 DEBUG:boto:Path: /hiiqos-53/d 2023-07-29T00:14:21.274 DEBUG:boto:Data: 2023-07-29T00:14:21.275 DEBUG:boto:Headers: {} 2023-07-29T00:14:21.275 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:21.275 DEBUG:boto:Port: 8000 2023-07-29T00:14:21.275 DEBUG:boto:Params: {} 2023-07-29T00:14:21.276 DEBUG:boto:Token: None 2023-07-29T00:14:21.276 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:21 GMT /hiiqos-53/d 2023-07-29T00:14:21.277 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5UjKYwIXQxCqpBVq5kWHMnAMDL4= 2023-07-29T00:14:21.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5UjKYwIXQxCqpBVq5kWHMnAMDL4=', 'Content-Length': '0'} 2023-07-29T00:14:21.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5df816bb1d91916-0064c459dd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:21.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-07-29T00:14:21.280 DEBUG:boto:path=/hiiqos-53/d 2023-07-29T00:14:21.280 DEBUG:boto:auth_path=/hiiqos-53/d 2023-07-29T00:14:21.281 DEBUG:boto:Method: GET 2023-07-29T00:14:21.281 DEBUG:boto:Path: /hiiqos-53/d 2023-07-29T00:14:21.281 DEBUG:boto:Data: 2023-07-29T00:14:21.282 DEBUG:boto:Headers: {} 2023-07-29T00:14:21.282 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:21.283 DEBUG:boto:Port: 8001 2023-07-29T00:14:21.283 DEBUG:boto:Params: {} 2023-07-29T00:14:21.283 DEBUG:boto:Token: None 2023-07-29T00:14:21.284 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:21 GMT /hiiqos-53/d 2023-07-29T00:14:21.284 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Jo1FZmEUq/keHVsB0I56lVSSj2I= 2023-07-29T00:14:21.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Jo1FZmEUq/keHVsB0I56lVSSj2I=', 'Content-Length': '0'} 2023-07-29T00:14:21.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-53:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023956a833eb27aa9-0064c459dd-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:21.287 DEBUG:boto:path=/hiiqos-53/d 2023-07-29T00:14:21.288 DEBUG:boto:auth_path=/hiiqos-53/d 2023-07-29T00:14:21.288 DEBUG:boto:Method: GET 2023-07-29T00:14:21.288 DEBUG:boto:Path: /hiiqos-53/d 2023-07-29T00:14:21.289 DEBUG:boto:Data: 2023-07-29T00:14:21.289 DEBUG:boto:Headers: {} 2023-07-29T00:14:21.289 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:21.290 DEBUG:boto:Port: 8000 2023-07-29T00:14:21.290 DEBUG:boto:Params: {} 2023-07-29T00:14:21.290 DEBUG:boto:Token: None 2023-07-29T00:14:21.291 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:21 GMT /hiiqos-53/d 2023-07-29T00:14:21.291 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Jo1FZmEUq/keHVsB0I56lVSSj2I= 2023-07-29T00:14:21.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Jo1FZmEUq/keHVsB0I56lVSSj2I=', 'Content-Length': '0'} 2023-07-29T00:14:21.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f3b1a31057daaf7-0064c459dd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:21.294 DEBUG:boto:path=/hiiqos-53/ 2023-07-29T00:14:21.295 DEBUG:boto:auth_path=/hiiqos-53/ 2023-07-29T00:14:21.295 DEBUG:boto:path=/hiiqos-53/?prefix=d 2023-07-29T00:14:21.296 DEBUG:boto:auth_path=/hiiqos-53/?prefix=d 2023-07-29T00:14:21.296 DEBUG:boto:Method: GET 2023-07-29T00:14:21.296 DEBUG:boto:Path: /hiiqos-53/?prefix=d 2023-07-29T00:14:21.297 DEBUG:boto:Data: 2023-07-29T00:14:21.297 DEBUG:boto:Headers: {} 2023-07-29T00:14:21.297 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:21.298 DEBUG:boto:Port: 8001 2023-07-29T00:14:21.298 DEBUG:boto:Params: {} 2023-07-29T00:14:21.299 DEBUG:boto:Token: None 2023-07-29T00:14:21.299 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:21 GMT /hiiqos-53/ 2023-07-29T00:14:21.299 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HgZvQJ0dhuVtojzt7HOVfDO3lng= 2023-07-29T00:14:21.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HgZvQJ0dhuVtojzt7HOVfDO3lng=', 'Content-Length': '0'} 2023-07-29T00:14:21.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c81e0cf3b953b6a4-0064c459dd-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:21.302 DEBUG:boto:b'hiiqos-53d1000falsed2023-07-29T00:14:09.821Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:21.303 DEBUG:boto:path=/hiiqos-53/ 2023-07-29T00:14:21.304 DEBUG:boto:auth_path=/hiiqos-53/ 2023-07-29T00:14:21.304 DEBUG:boto:path=/hiiqos-53/?prefix=d 2023-07-29T00:14:21.304 DEBUG:boto:auth_path=/hiiqos-53/?prefix=d 2023-07-29T00:14:21.305 DEBUG:boto:Method: GET 2023-07-29T00:14:21.305 DEBUG:boto:Path: /hiiqos-53/?prefix=d 2023-07-29T00:14:21.305 DEBUG:boto:Data: 2023-07-29T00:14:21.306 DEBUG:boto:Headers: {} 2023-07-29T00:14:21.306 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:21.306 DEBUG:boto:Port: 8000 2023-07-29T00:14:21.307 DEBUG:boto:Params: {} 2023-07-29T00:14:21.307 DEBUG:boto:Token: None 2023-07-29T00:14:21.308 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:21 GMT /hiiqos-53/ 2023-07-29T00:14:21.308 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HgZvQJ0dhuVtojzt7HOVfDO3lng= 2023-07-29T00:14:21.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HgZvQJ0dhuVtojzt7HOVfDO3lng=', 'Content-Length': '0'} 2023-07-29T00:14:21.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097fb1caf6e4680aa-0064c459dd-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:21.311 DEBUG:boto:b'hiiqos-53d1000falsed2023-07-29T00:14:09.821Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:21.312 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-53 zones={a2, a1} 2023-07-29T00:14:21.313 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:14:21.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:14:21.313 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:14:21.362 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:14:21.362 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:14:21.409 INFO:rgw_multi.tests:create bucket zone=a1 name=hiiqos-54 2023-07-29T00:14:21.409 DEBUG:boto:path=/hiiqos-54/ 2023-07-29T00:14:21.410 DEBUG:boto:auth_path=/hiiqos-54/ 2023-07-29T00:14:21.410 DEBUG:boto:Method: PUT 2023-07-29T00:14:21.410 DEBUG:boto:Path: /hiiqos-54/ 2023-07-29T00:14:21.411 DEBUG:boto:Data: 2023-07-29T00:14:21.411 DEBUG:boto:Headers: {} 2023-07-29T00:14:21.411 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:21.412 DEBUG:boto:Port: 8000 2023-07-29T00:14:21.412 DEBUG:boto:Params: {} 2023-07-29T00:14:21.413 DEBUG:boto:Token: None 2023-07-29T00:14:21.413 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 00:14:21 GMT /hiiqos-54/ 2023-07-29T00:14:21.413 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VstlytEiTdQYL5bAhvDpdv0FNNY= 2023-07-29T00:14:21.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VstlytEiTdQYL5bAhvDpdv0FNNY=', 'Content-Length': '0'} 2023-07-29T00:14:21.429 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000029ffdf06931779c1-0064c459dd-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:21.430 DEBUG:boto:path=/hiiqos-54/ 2023-07-29T00:14:21.430 DEBUG:boto:auth_path=/hiiqos-54/ 2023-07-29T00:14:21.431 DEBUG:boto:Method: HEAD 2023-07-29T00:14:21.431 DEBUG:boto:Path: /hiiqos-54/ 2023-07-29T00:14:21.431 DEBUG:boto:Data: 2023-07-29T00:14:21.432 DEBUG:boto:Headers: {} 2023-07-29T00:14:21.432 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:21.432 DEBUG:boto:Port: 8000 2023-07-29T00:14:21.433 DEBUG:boto:Params: {} 2023-07-29T00:14:21.433 DEBUG:boto:Token: None 2023-07-29T00:14:21.434 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:21 GMT /hiiqos-54/ 2023-07-29T00:14:21.434 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S7b/9oY1v+wTcrJZXlKmAoAerKA= 2023-07-29T00:14:21.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S7b/9oY1v+wTcrJZXlKmAoAerKA=', 'Content-Length': '0'} 2023-07-29T00:14:21.437 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ece2f6481da5cdee-0064c459dd-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:21.437 DEBUG:boto:path=/hiiqos-54/a 2023-07-29T00:14:21.438 DEBUG:boto:auth_path=/hiiqos-54/a 2023-07-29T00:14:21.438 DEBUG:boto:Method: PUT 2023-07-29T00:14:21.438 DEBUG:boto:Path: /hiiqos-54/a 2023-07-29T00:14:21.439 DEBUG:boto:Data: 2023-07-29T00:14:21.439 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:14:21.439 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:21.440 DEBUG:boto:Port: 8000 2023-07-29T00:14:21.440 DEBUG:boto:Params: {} 2023-07-29T00:14:21.440 DEBUG:boto:Token: None 2023-07-29T00:14:21.441 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:14:21 GMT /hiiqos-54/a 2023-07-29T00:14:21.441 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:04n9LwMtfTkUfpLMzWIFkh1zzUI= 2023-07-29T00:14:21.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:14:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:04n9LwMtfTkUfpLMzWIFkh1zzUI='} 2023-07-29T00:14:21.484 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006d40aa7fabf23fa9-0064c459dd-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:21.485 DEBUG:boto:path=/hiiqos-54/ 2023-07-29T00:14:21.485 DEBUG:boto:auth_path=/hiiqos-54/ 2023-07-29T00:14:21.486 DEBUG:boto:Method: HEAD 2023-07-29T00:14:21.486 DEBUG:boto:Path: /hiiqos-54/ 2023-07-29T00:14:21.486 DEBUG:boto:Data: 2023-07-29T00:14:21.487 DEBUG:boto:Headers: {} 2023-07-29T00:14:21.487 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:21.487 DEBUG:boto:Port: 8000 2023-07-29T00:14:21.488 DEBUG:boto:Params: {} 2023-07-29T00:14:21.488 DEBUG:boto:Token: None 2023-07-29T00:14:21.489 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:21 GMT /hiiqos-54/ 2023-07-29T00:14:21.489 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S7b/9oY1v+wTcrJZXlKmAoAerKA= 2023-07-29T00:14:21.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S7b/9oY1v+wTcrJZXlKmAoAerKA=', 'Content-Length': '0'} 2023-07-29T00:14:21.492 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000800905282041446f-0064c459dd-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:21.492 DEBUG:boto:path=/hiiqos-54/b 2023-07-29T00:14:21.493 DEBUG:boto:auth_path=/hiiqos-54/b 2023-07-29T00:14:21.493 DEBUG:boto:Method: PUT 2023-07-29T00:14:21.493 DEBUG:boto:Path: /hiiqos-54/b 2023-07-29T00:14:21.494 DEBUG:boto:Data: 2023-07-29T00:14:21.494 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:14:21.494 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:21.495 DEBUG:boto:Port: 8000 2023-07-29T00:14:21.495 DEBUG:boto:Params: {} 2023-07-29T00:14:21.495 DEBUG:boto:Token: None 2023-07-29T00:14:21.496 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:14:21 GMT /hiiqos-54/b 2023-07-29T00:14:21.496 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/TL7jej4uMtnYDRA2VmsbPV9GUs= 2023-07-29T00:14:21.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:14:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/TL7jej4uMtnYDRA2VmsbPV9GUs='} 2023-07-29T00:14:21.504 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cabba60f203b0dc6-0064c459dd-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:21.504 DEBUG:boto:path=/hiiqos-54/ 2023-07-29T00:14:21.504 DEBUG:boto:auth_path=/hiiqos-54/ 2023-07-29T00:14:21.505 DEBUG:boto:Method: HEAD 2023-07-29T00:14:21.505 DEBUG:boto:Path: /hiiqos-54/ 2023-07-29T00:14:21.505 DEBUG:boto:Data: 2023-07-29T00:14:21.506 DEBUG:boto:Headers: {} 2023-07-29T00:14:21.506 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:21.506 DEBUG:boto:Port: 8000 2023-07-29T00:14:21.507 DEBUG:boto:Params: {} 2023-07-29T00:14:21.507 DEBUG:boto:Token: None 2023-07-29T00:14:21.508 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:21 GMT /hiiqos-54/ 2023-07-29T00:14:21.508 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S7b/9oY1v+wTcrJZXlKmAoAerKA= 2023-07-29T00:14:21.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S7b/9oY1v+wTcrJZXlKmAoAerKA=', 'Content-Length': '0'} 2023-07-29T00:14:21.510 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cc44694b0dab32b1-0064c459dd-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:21.511 DEBUG:boto:path=/hiiqos-54/c 2023-07-29T00:14:21.511 DEBUG:boto:auth_path=/hiiqos-54/c 2023-07-29T00:14:21.512 DEBUG:boto:Method: PUT 2023-07-29T00:14:21.512 DEBUG:boto:Path: /hiiqos-54/c 2023-07-29T00:14:21.512 DEBUG:boto:Data: 2023-07-29T00:14:21.513 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:14:21.513 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:21.513 DEBUG:boto:Port: 8000 2023-07-29T00:14:21.514 DEBUG:boto:Params: {} 2023-07-29T00:14:21.514 DEBUG:boto:Token: None 2023-07-29T00:14:21.514 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:14:21 GMT /hiiqos-54/c 2023-07-29T00:14:21.515 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FWtPjBoUsochaqLjwYOVEpZK35g= 2023-07-29T00:14:21.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:14:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FWtPjBoUsochaqLjwYOVEpZK35g='} 2023-07-29T00:14:21.520 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000013f4f72b9f4b0b31-0064c459dd-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:21.520 DEBUG:boto:path=/hiiqos-54/ 2023-07-29T00:14:21.521 DEBUG:boto:auth_path=/hiiqos-54/ 2023-07-29T00:14:21.521 DEBUG:boto:Method: HEAD 2023-07-29T00:14:21.521 DEBUG:boto:Path: /hiiqos-54/ 2023-07-29T00:14:21.522 DEBUG:boto:Data: 2023-07-29T00:14:21.522 DEBUG:boto:Headers: {} 2023-07-29T00:14:21.522 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:21.523 DEBUG:boto:Port: 8000 2023-07-29T00:14:21.523 DEBUG:boto:Params: {} 2023-07-29T00:14:21.523 DEBUG:boto:Token: None 2023-07-29T00:14:21.524 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:21 GMT /hiiqos-54/ 2023-07-29T00:14:21.524 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S7b/9oY1v+wTcrJZXlKmAoAerKA= 2023-07-29T00:14:21.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S7b/9oY1v+wTcrJZXlKmAoAerKA=', 'Content-Length': '0'} 2023-07-29T00:14:21.527 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009061a14eb1c62bd4-0064c459dd-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:21.527 DEBUG:boto:path=/hiiqos-54/d 2023-07-29T00:14:21.527 DEBUG:boto:auth_path=/hiiqos-54/d 2023-07-29T00:14:21.528 DEBUG:boto:Method: PUT 2023-07-29T00:14:21.528 DEBUG:boto:Path: /hiiqos-54/d 2023-07-29T00:14:21.529 DEBUG:boto:Data: 2023-07-29T00:14:21.529 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:14:21.529 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:21.530 DEBUG:boto:Port: 8000 2023-07-29T00:14:21.530 DEBUG:boto:Params: {} 2023-07-29T00:14:21.530 DEBUG:boto:Token: None 2023-07-29T00:14:21.531 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:14:21 GMT /hiiqos-54/d 2023-07-29T00:14:21.531 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yBWz1GYK0Vd7PwlpfngZ47Fr3Mw= 2023-07-29T00:14:21.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:14:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yBWz1GYK0Vd7PwlpfngZ47Fr3Mw='} 2023-07-29T00:14:21.576 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003c7e49dc9c4e323b-0064c459dd-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:21.577 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:14:21.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:14:21.578 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:14:21.623 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:14:21.623 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:14:21.653 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T00:14:21.665 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T00:14:21.666 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:14:21.666 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:14:21.666 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:46:09.233742Z" 2023-07-29T00:14:21.666 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:21.666 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:14:21.666 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690589661.389622_811.1", 2023-07-29T00:14:21.666 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:14:21.389622Z" 2023-07-29T00:14:21.667 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:21.667 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:14:21.667 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:14:21.667 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T00:14:21.667 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:21.667 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:14:21.667 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:14:21.668 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T00:14:21.668 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:14:21.668 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T00:14:21.669 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690587969.233742_99.1', 1: '1_1690589661.389622_811.1', 2: '', 3: ''} 2023-07-29T00:14:21.669 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T00:14:21.670 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:14:21.670 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:14:21.671 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:14:21.717 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:14:21.717 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:14:21.745 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:14:21.705+0000 7f2d07616880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:14:21.746 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:14:21.706+0000 7f2d07616880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:14:21.758 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:14:21.759 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:14:21.759 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:14:21.759 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:14:21.759 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:14:21.759 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:14:21.759 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:14:21.760 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:21.760 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:14:21.760 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:14:21.760 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:14:21.760 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:14:21.760 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:14:21.760 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:14:21.761 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:14:21.761 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:14:21.761 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:14:21.761 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:14:21.761 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:14:21.761 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:14:21.761 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:21.762 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:14:21.762 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:14:21.762 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:14:21.762 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:14:21.762 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690589661.389622_811.1", 2023-07-29T00:14:21.762 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:14:21.762 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:14:21.763 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:14:21.763 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:14:21.389622Z", 2023-07-29T00:14:21.763 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:14:21.763 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:14:21.763 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:21.763 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:14:21.763 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:14:21.764 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:14:21.764 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:14:21.764 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:14:21.764 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:14:21.764 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:14:21.764 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:14:21.764 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:14:21.765 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:14:21.765 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:14:21.765 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:21.765 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:14:21.765 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:14:21.765 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:14:21.766 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:14:21.766 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:14:21.766 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:14:21.766 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:14:21.766 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:14:21.766 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:14:21.766 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:14:21.766 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:14:21.767 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:14:21.767 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:14:21.767 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:21.767 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:14:21.767 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:14:21.767 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:14:21.767 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:21.768 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:14:21Z" 2023-07-29T00:14:21.768 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:14:21.769 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690589661.389622_811.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:14:21.389622Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:14:21Z" } 2023-07-29T00:14:21.769 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690589661.389622_811.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:14:21.389622Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:14:21.770 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690589661.389622_811.1', 2: '', 3: ''} 2023-07-29T00:14:21.770 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690589661.389622_811.1', 2: '', 3: ''} 2023-07-29T00:14:21.770 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T00:14:21.771 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-54 2023-07-29T00:14:21.771 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-54', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:14:21.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-54', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:14:21.772 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-54 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:14:21.824 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:14:21.824 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:14:21.856 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:14:21.816+0000 7f5e34531880 1 waiting to reach incremental sync.. 2023-07-29T00:14:26.857 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:14:26.817+0000 7f5e34531880 1 waiting to reach incremental sync.. 2023-07-29T00:14:31.858 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:14:31.818+0000 7f5e34531880 1 waiting to reach incremental sync.. 2023-07-29T00:14:36.860 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:14:36.820+0000 7f5e34531880 1 bucket sync caught up with source: 2023-07-29T00:14:36.860 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000001.37.6, , , , 00000000001.17.6, , , 00000000001.37.6, , 00000000001.92.6] 2023-07-29T00:14:36.860 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000001.37.6, , , , 00000000001.17.6, , , 00000000001.37.6, , 00000000001.92.6] 2023-07-29T00:14:36.860 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:14:36.820+0000 7f5e34531880 0 bucket checkpoint complete 2023-07-29T00:14:36.877 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-54 2023-07-29T00:14:36.878 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-54', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:14:36.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-54', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:14:36.878 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-54 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:14:36.936 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:14:36.936 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:14:36.971 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:14:36.931+0000 7f61ac616880 1 waiting to reach incremental sync.. 2023-07-29T00:14:41.972 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:14:41.932+0000 7f61ac616880 1 bucket sync caught up with source: 2023-07-29T00:14:41.973 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000001.881.6, , , , 00000000001.358.6, , , 00000000001.104.6, , 00000000001.758.6] 2023-07-29T00:14:41.973 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000001.881.6, , , , 00000000001.358.6, , , 00000000001.104.6, , 00000000001.758.6] 2023-07-29T00:14:41.973 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:14:41.932+0000 7f61ac616880 0 bucket checkpoint complete 2023-07-29T00:14:41.989 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-54 zones={a2, a1} 2023-07-29T00:14:41.990 DEBUG:boto:path=/hiiqos-54/ 2023-07-29T00:14:41.990 DEBUG:boto:auth_path=/hiiqos-54/ 2023-07-29T00:14:41.991 DEBUG:boto:Method: HEAD 2023-07-29T00:14:41.991 DEBUG:boto:Path: /hiiqos-54/ 2023-07-29T00:14:41.991 DEBUG:boto:Data: 2023-07-29T00:14:41.992 DEBUG:boto:Headers: {} 2023-07-29T00:14:41.992 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:41.993 DEBUG:boto:Port: 8001 2023-07-29T00:14:41.993 DEBUG:boto:Params: {} 2023-07-29T00:14:41.993 DEBUG:boto:Token: None 2023-07-29T00:14:41.994 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:41 GMT /hiiqos-54/ 2023-07-29T00:14:41.994 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4RHIMA0qXSZQ91r4rY9t6tyn6OI= 2023-07-29T00:14:41.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4RHIMA0qXSZQ91r4rY9t6tyn6OI=', 'Content-Length': '0'} 2023-07-29T00:14:41.997 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c03ce4993a97b4e6-0064c459f1-1663-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:14:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:41.998 DEBUG:boto:path=/hiiqos-54/ 2023-07-29T00:14:41.998 DEBUG:boto:auth_path=/hiiqos-54/ 2023-07-29T00:14:41.998 DEBUG:boto:Method: HEAD 2023-07-29T00:14:41.999 DEBUG:boto:Path: /hiiqos-54/ 2023-07-29T00:14:41.999 DEBUG:boto:Data: 2023-07-29T00:14:41.999 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.000 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:42.000 DEBUG:boto:Port: 8000 2023-07-29T00:14:42.000 DEBUG:boto:Params: {} 2023-07-29T00:14:42.001 DEBUG:boto:Token: None 2023-07-29T00:14:42.001 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/ 2023-07-29T00:14:42.002 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Cx6gF2ZuEk5Ct217mBWKCUTXZBY= 2023-07-29T00:14:42.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Cx6gF2ZuEk5Ct217mBWKCUTXZBY=', 'Content-Length': '0'} 2023-07-29T00:14:42.004 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e526c205128333c-0064c459f1-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:14:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.005 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-29T00:14:42.005 DEBUG:boto:path=/hiiqos-54/ 2023-07-29T00:14:42.006 DEBUG:boto:auth_path=/hiiqos-54/ 2023-07-29T00:14:42.006 DEBUG:boto:path=/hiiqos-54/?versions&max-keys=999 2023-07-29T00:14:42.006 DEBUG:boto:auth_path=/hiiqos-54/?versions&max-keys=999 2023-07-29T00:14:42.007 DEBUG:boto:Method: GET 2023-07-29T00:14:42.007 DEBUG:boto:Path: /hiiqos-54/?versions&max-keys=999 2023-07-29T00:14:42.007 DEBUG:boto:Data: 2023-07-29T00:14:42.008 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.008 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:42.009 DEBUG:boto:Port: 8001 2023-07-29T00:14:42.009 DEBUG:boto:Params: {} 2023-07-29T00:14:42.009 DEBUG:boto:Token: None 2023-07-29T00:14:42.010 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/?versions 2023-07-29T00:14:42.010 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9FFjkm0G0/CPnK5gyWG345Hs+rs= 2023-07-29T00:14:42.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9FFjkm0G0/CPnK5gyWG345Hs+rs=', 'Content-Length': '0'} 2023-07-29T00:14:42.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afd40c11d57b74bf-0064c459f1-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.013 DEBUG:boto:b'hiiqos-54999falseanulltrue2023-07-29T00:14:21.405Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:14:21.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:14:21.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:14:21.494Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:42.014 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-07-29T00:14:42.014 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-07-29T00:14:42.015 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-07-29T00:14:42.015 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-07-29T00:14:42.015 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-29T00:14:42.016 DEBUG:boto:path=/hiiqos-54/ 2023-07-29T00:14:42.016 DEBUG:boto:auth_path=/hiiqos-54/ 2023-07-29T00:14:42.016 DEBUG:boto:path=/hiiqos-54/?versions&max-keys=999 2023-07-29T00:14:42.017 DEBUG:boto:auth_path=/hiiqos-54/?versions&max-keys=999 2023-07-29T00:14:42.017 DEBUG:boto:Method: GET 2023-07-29T00:14:42.017 DEBUG:boto:Path: /hiiqos-54/?versions&max-keys=999 2023-07-29T00:14:42.018 DEBUG:boto:Data: 2023-07-29T00:14:42.018 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.018 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:42.019 DEBUG:boto:Port: 8000 2023-07-29T00:14:42.019 DEBUG:boto:Params: {} 2023-07-29T00:14:42.019 DEBUG:boto:Token: None 2023-07-29T00:14:42.020 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/?versions 2023-07-29T00:14:42.020 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9FFjkm0G0/CPnK5gyWG345Hs+rs= 2023-07-29T00:14:42.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9FFjkm0G0/CPnK5gyWG345Hs+rs=', 'Content-Length': '0'} 2023-07-29T00:14:42.023 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9622e32d4083f16-0064c459f1-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.024 DEBUG:boto:b'hiiqos-54999falseanulltrue2023-07-29T00:14:21.405Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:14:21.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:14:21.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:14:21.494Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:42.025 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-07-29T00:14:42.025 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-07-29T00:14:42.025 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-07-29T00:14:42.026 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-07-29T00:14:42.026 DEBUG:boto:path=/hiiqos-54/ 2023-07-29T00:14:42.026 DEBUG:boto:auth_path=/hiiqos-54/ 2023-07-29T00:14:42.027 DEBUG:boto:path=/hiiqos-54/?versions&max-keys=999 2023-07-29T00:14:42.027 DEBUG:boto:auth_path=/hiiqos-54/?versions&max-keys=999 2023-07-29T00:14:42.027 DEBUG:boto:Method: GET 2023-07-29T00:14:42.028 DEBUG:boto:Path: /hiiqos-54/?versions&max-keys=999 2023-07-29T00:14:42.028 DEBUG:boto:Data: 2023-07-29T00:14:42.028 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.029 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:42.029 DEBUG:boto:Port: 8001 2023-07-29T00:14:42.029 DEBUG:boto:Params: {} 2023-07-29T00:14:42.030 DEBUG:boto:Token: None 2023-07-29T00:14:42.030 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/?versions 2023-07-29T00:14:42.031 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9FFjkm0G0/CPnK5gyWG345Hs+rs= 2023-07-29T00:14:42.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9FFjkm0G0/CPnK5gyWG345Hs+rs=', 'Content-Length': '0'} 2023-07-29T00:14:42.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcf3d73b65a37f4c-0064c459f1-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.034 DEBUG:boto:b'hiiqos-54999falseanulltrue2023-07-29T00:14:21.405Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:14:21.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:14:21.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:14:21.494Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:42.035 DEBUG:boto:path=/hiiqos-54/ 2023-07-29T00:14:42.035 DEBUG:boto:auth_path=/hiiqos-54/ 2023-07-29T00:14:42.035 DEBUG:boto:path=/hiiqos-54/?versions&max-keys=999 2023-07-29T00:14:42.036 DEBUG:boto:auth_path=/hiiqos-54/?versions&max-keys=999 2023-07-29T00:14:42.036 DEBUG:boto:Method: GET 2023-07-29T00:14:42.037 DEBUG:boto:Path: /hiiqos-54/?versions&max-keys=999 2023-07-29T00:14:42.037 DEBUG:boto:Data: 2023-07-29T00:14:42.037 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.038 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:42.038 DEBUG:boto:Port: 8000 2023-07-29T00:14:42.038 DEBUG:boto:Params: {} 2023-07-29T00:14:42.039 DEBUG:boto:Token: None 2023-07-29T00:14:42.039 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/?versions 2023-07-29T00:14:42.039 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9FFjkm0G0/CPnK5gyWG345Hs+rs= 2023-07-29T00:14:42.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9FFjkm0G0/CPnK5gyWG345Hs+rs=', 'Content-Length': '0'} 2023-07-29T00:14:42.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000966f0befcf37a939-0064c459f2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.043 DEBUG:boto:b'hiiqos-54999falseanulltrue2023-07-29T00:14:21.405Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:14:21.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:14:21.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:14:21.494Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:42.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-07-29T00:14:42.044 DEBUG:boto:path=/hiiqos-54/a 2023-07-29T00:14:42.044 DEBUG:boto:auth_path=/hiiqos-54/a 2023-07-29T00:14:42.045 DEBUG:boto:path=/hiiqos-54/a?versionId=null 2023-07-29T00:14:42.045 DEBUG:boto:auth_path=/hiiqos-54/a?versionId=null 2023-07-29T00:14:42.046 DEBUG:boto:Method: GET 2023-07-29T00:14:42.046 DEBUG:boto:Path: /hiiqos-54/a?versionId=null 2023-07-29T00:14:42.046 DEBUG:boto:Data: 2023-07-29T00:14:42.047 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.047 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:42.047 DEBUG:boto:Port: 8001 2023-07-29T00:14:42.048 DEBUG:boto:Params: {} 2023-07-29T00:14:42.048 DEBUG:boto:Token: None 2023-07-29T00:14:42.049 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/a?versionId=null 2023-07-29T00:14:42.049 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ovEOH5mN9ZT941vVNU3cV+hfRpc= 2023-07-29T00:14:42.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ovEOH5mN9ZT941vVNU3cV+hfRpc=', 'Content-Length': '0'} 2023-07-29T00:14:42.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-54:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000858708a43276bfe9-0064c459f2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.096 DEBUG:boto:path=/hiiqos-54/a 2023-07-29T00:14:42.097 DEBUG:boto:auth_path=/hiiqos-54/a 2023-07-29T00:14:42.097 DEBUG:boto:path=/hiiqos-54/a?versionId=null 2023-07-29T00:14:42.098 DEBUG:boto:auth_path=/hiiqos-54/a?versionId=null 2023-07-29T00:14:42.098 DEBUG:boto:Method: GET 2023-07-29T00:14:42.098 DEBUG:boto:Path: /hiiqos-54/a?versionId=null 2023-07-29T00:14:42.099 DEBUG:boto:Data: 2023-07-29T00:14:42.099 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.099 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:42.100 DEBUG:boto:Port: 8000 2023-07-29T00:14:42.100 DEBUG:boto:Params: {} 2023-07-29T00:14:42.101 DEBUG:boto:Token: None 2023-07-29T00:14:42.101 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/a?versionId=null 2023-07-29T00:14:42.101 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ovEOH5mN9ZT941vVNU3cV+hfRpc= 2023-07-29T00:14:42.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ovEOH5mN9ZT941vVNU3cV+hfRpc=', 'Content-Length': '0'} 2023-07-29T00:14:42.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007dbd44cfb2ee1d58-0064c459f2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.105 DEBUG:boto:path=/hiiqos-54/a 2023-07-29T00:14:42.105 DEBUG:boto:auth_path=/hiiqos-54/a 2023-07-29T00:14:42.106 DEBUG:boto:path=/hiiqos-54/a?versionId=null 2023-07-29T00:14:42.106 DEBUG:boto:auth_path=/hiiqos-54/a?versionId=null 2023-07-29T00:14:42.106 DEBUG:boto:Method: HEAD 2023-07-29T00:14:42.107 DEBUG:boto:Path: /hiiqos-54/a?versionId=null 2023-07-29T00:14:42.107 DEBUG:boto:Data: 2023-07-29T00:14:42.107 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.108 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:42.108 DEBUG:boto:Port: 8001 2023-07-29T00:14:42.108 DEBUG:boto:Params: {} 2023-07-29T00:14:42.109 DEBUG:boto:Token: None 2023-07-29T00:14:42.109 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/a?versionId=null 2023-07-29T00:14:42.110 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wpHXFegitLW1Fn3D9sR/CjItbpQ= 2023-07-29T00:14:42.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wpHXFegitLW1Fn3D9sR/CjItbpQ=', 'Content-Length': '0'} 2023-07-29T00:14:42.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-54:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011a2856b4340e750-0064c459f2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.112 DEBUG:boto:path=/hiiqos-54/a 2023-07-29T00:14:42.113 DEBUG:boto:auth_path=/hiiqos-54/a 2023-07-29T00:14:42.113 DEBUG:boto:path=/hiiqos-54/a?versionId=null 2023-07-29T00:14:42.114 DEBUG:boto:auth_path=/hiiqos-54/a?versionId=null 2023-07-29T00:14:42.114 DEBUG:boto:Method: HEAD 2023-07-29T00:14:42.115 DEBUG:boto:Path: /hiiqos-54/a?versionId=null 2023-07-29T00:14:42.115 DEBUG:boto:Data: 2023-07-29T00:14:42.115 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.116 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:42.116 DEBUG:boto:Port: 8000 2023-07-29T00:14:42.116 DEBUG:boto:Params: {} 2023-07-29T00:14:42.117 DEBUG:boto:Token: None 2023-07-29T00:14:42.117 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/a?versionId=null 2023-07-29T00:14:42.118 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wpHXFegitLW1Fn3D9sR/CjItbpQ= 2023-07-29T00:14:42.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wpHXFegitLW1Fn3D9sR/CjItbpQ=', 'Content-Length': '0'} 2023-07-29T00:14:42.120 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007871f6d0223fcd8f-0064c459f2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-07-29T00:14:42.121 DEBUG:boto:path=/hiiqos-54/a 2023-07-29T00:14:42.122 DEBUG:boto:auth_path=/hiiqos-54/a 2023-07-29T00:14:42.122 DEBUG:boto:path=/hiiqos-54/a?versionId=null 2023-07-29T00:14:42.122 DEBUG:boto:auth_path=/hiiqos-54/a?versionId=null 2023-07-29T00:14:42.123 DEBUG:boto:Method: GET 2023-07-29T00:14:42.123 DEBUG:boto:Path: /hiiqos-54/a?versionId=null 2023-07-29T00:14:42.123 DEBUG:boto:Data: 2023-07-29T00:14:42.124 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.124 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:42.125 DEBUG:boto:Port: 8001 2023-07-29T00:14:42.125 DEBUG:boto:Params: {} 2023-07-29T00:14:42.125 DEBUG:boto:Token: None 2023-07-29T00:14:42.126 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/a?versionId=null 2023-07-29T00:14:42.126 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ovEOH5mN9ZT941vVNU3cV+hfRpc= 2023-07-29T00:14:42.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ovEOH5mN9ZT941vVNU3cV+hfRpc=', 'Content-Length': '0'} 2023-07-29T00:14:42.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-54:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b651b62883fac7f-0064c459f2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.129 DEBUG:boto:path=/hiiqos-54/a 2023-07-29T00:14:42.130 DEBUG:boto:auth_path=/hiiqos-54/a 2023-07-29T00:14:42.130 DEBUG:boto:path=/hiiqos-54/a?versionId=null 2023-07-29T00:14:42.130 DEBUG:boto:auth_path=/hiiqos-54/a?versionId=null 2023-07-29T00:14:42.131 DEBUG:boto:Method: GET 2023-07-29T00:14:42.131 DEBUG:boto:Path: /hiiqos-54/a?versionId=null 2023-07-29T00:14:42.131 DEBUG:boto:Data: 2023-07-29T00:14:42.132 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.132 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:42.133 DEBUG:boto:Port: 8000 2023-07-29T00:14:42.133 DEBUG:boto:Params: {} 2023-07-29T00:14:42.133 DEBUG:boto:Token: None 2023-07-29T00:14:42.134 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/a?versionId=null 2023-07-29T00:14:42.134 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ovEOH5mN9ZT941vVNU3cV+hfRpc= 2023-07-29T00:14:42.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ovEOH5mN9ZT941vVNU3cV+hfRpc=', 'Content-Length': '0'} 2023-07-29T00:14:42.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3198e9d59619d85-0064c459f2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.180 DEBUG:boto:path=/hiiqos-54/ 2023-07-29T00:14:42.181 DEBUG:boto:auth_path=/hiiqos-54/ 2023-07-29T00:14:42.182 DEBUG:boto:path=/hiiqos-54/?prefix=a 2023-07-29T00:14:42.182 DEBUG:boto:auth_path=/hiiqos-54/?prefix=a 2023-07-29T00:14:42.182 DEBUG:boto:Method: GET 2023-07-29T00:14:42.183 DEBUG:boto:Path: /hiiqos-54/?prefix=a 2023-07-29T00:14:42.183 DEBUG:boto:Data: 2023-07-29T00:14:42.184 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.184 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:42.185 DEBUG:boto:Port: 8001 2023-07-29T00:14:42.185 DEBUG:boto:Params: {} 2023-07-29T00:14:42.186 DEBUG:boto:Token: None 2023-07-29T00:14:42.187 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/ 2023-07-29T00:14:42.187 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:p2xUYDIYoRCIFn5nBqae9D0avjg= 2023-07-29T00:14:42.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:p2xUYDIYoRCIFn5nBqae9D0avjg=', 'Content-Length': '0'} 2023-07-29T00:14:42.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4206a9867ee43a8-0064c459f2-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.191 DEBUG:boto:b'hiiqos-54a1000falsea2023-07-29T00:14:21.405Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:42.192 DEBUG:boto:path=/hiiqos-54/ 2023-07-29T00:14:42.193 DEBUG:boto:auth_path=/hiiqos-54/ 2023-07-29T00:14:42.193 DEBUG:boto:path=/hiiqos-54/?prefix=a 2023-07-29T00:14:42.194 DEBUG:boto:auth_path=/hiiqos-54/?prefix=a 2023-07-29T00:14:42.195 DEBUG:boto:Method: GET 2023-07-29T00:14:42.195 DEBUG:boto:Path: /hiiqos-54/?prefix=a 2023-07-29T00:14:42.196 DEBUG:boto:Data: 2023-07-29T00:14:42.196 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.197 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:42.197 DEBUG:boto:Port: 8000 2023-07-29T00:14:42.198 DEBUG:boto:Params: {} 2023-07-29T00:14:42.198 DEBUG:boto:Token: None 2023-07-29T00:14:42.199 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/ 2023-07-29T00:14:42.200 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:p2xUYDIYoRCIFn5nBqae9D0avjg= 2023-07-29T00:14:42.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:p2xUYDIYoRCIFn5nBqae9D0avjg=', 'Content-Length': '0'} 2023-07-29T00:14:42.203 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b24c138278127eb-0064c459f2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.204 DEBUG:boto:b'hiiqos-54a1000falsea2023-07-29T00:14:21.405Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:42.205 DEBUG:boto:path=/hiiqos-54/a 2023-07-29T00:14:42.205 DEBUG:boto:auth_path=/hiiqos-54/a 2023-07-29T00:14:42.206 DEBUG:boto:Method: HEAD 2023-07-29T00:14:42.206 DEBUG:boto:Path: /hiiqos-54/a 2023-07-29T00:14:42.207 DEBUG:boto:Data: 2023-07-29T00:14:42.207 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.208 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:42.208 DEBUG:boto:Port: 8001 2023-07-29T00:14:42.209 DEBUG:boto:Params: {} 2023-07-29T00:14:42.209 DEBUG:boto:Token: None 2023-07-29T00:14:42.210 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/a 2023-07-29T00:14:42.210 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wdzmu65qbGyz9oBSYRxFa0gh9JQ= 2023-07-29T00:14:42.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wdzmu65qbGyz9oBSYRxFa0gh9JQ=', 'Content-Length': '0'} 2023-07-29T00:14:42.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-54:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041864d9e2a8b937a-0064c459f2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.214 DEBUG:boto:path=/hiiqos-54/a 2023-07-29T00:14:42.214 DEBUG:boto:auth_path=/hiiqos-54/a 2023-07-29T00:14:42.215 DEBUG:boto:Method: HEAD 2023-07-29T00:14:42.215 DEBUG:boto:Path: /hiiqos-54/a 2023-07-29T00:14:42.216 DEBUG:boto:Data: 2023-07-29T00:14:42.217 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.217 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:42.218 DEBUG:boto:Port: 8000 2023-07-29T00:14:42.218 DEBUG:boto:Params: {} 2023-07-29T00:14:42.219 DEBUG:boto:Token: None 2023-07-29T00:14:42.219 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/a 2023-07-29T00:14:42.220 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wdzmu65qbGyz9oBSYRxFa0gh9JQ= 2023-07-29T00:14:42.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wdzmu65qbGyz9oBSYRxFa0gh9JQ=', 'Content-Length': '0'} 2023-07-29T00:14:42.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d84415515aab7ad-0064c459f2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-07-29T00:14:42.223 DEBUG:boto:path=/hiiqos-54/a 2023-07-29T00:14:42.224 DEBUG:boto:auth_path=/hiiqos-54/a 2023-07-29T00:14:42.225 DEBUG:boto:Method: GET 2023-07-29T00:14:42.225 DEBUG:boto:Path: /hiiqos-54/a 2023-07-29T00:14:42.226 DEBUG:boto:Data: 2023-07-29T00:14:42.226 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.227 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:42.227 DEBUG:boto:Port: 8001 2023-07-29T00:14:42.228 DEBUG:boto:Params: {} 2023-07-29T00:14:42.228 DEBUG:boto:Token: None 2023-07-29T00:14:42.229 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/a 2023-07-29T00:14:42.229 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:s5TBKadPqFgY56Q56gclisBRmJw= 2023-07-29T00:14:42.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:s5TBKadPqFgY56Q56gclisBRmJw=', 'Content-Length': '0'} 2023-07-29T00:14:42.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-54:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004671f9e42fd336c2-0064c459f2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.232 DEBUG:boto:path=/hiiqos-54/a 2023-07-29T00:14:42.233 DEBUG:boto:auth_path=/hiiqos-54/a 2023-07-29T00:14:42.233 DEBUG:boto:Method: GET 2023-07-29T00:14:42.234 DEBUG:boto:Path: /hiiqos-54/a 2023-07-29T00:14:42.235 DEBUG:boto:Data: 2023-07-29T00:14:42.235 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.236 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:42.236 DEBUG:boto:Port: 8000 2023-07-29T00:14:42.237 DEBUG:boto:Params: {} 2023-07-29T00:14:42.237 DEBUG:boto:Token: None 2023-07-29T00:14:42.238 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/a 2023-07-29T00:14:42.238 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:s5TBKadPqFgY56Q56gclisBRmJw= 2023-07-29T00:14:42.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:s5TBKadPqFgY56Q56gclisBRmJw=', 'Content-Length': '0'} 2023-07-29T00:14:42.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000276a91c3ed661a62-0064c459f2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.241 DEBUG:boto:path=/hiiqos-54/ 2023-07-29T00:14:42.242 DEBUG:boto:auth_path=/hiiqos-54/ 2023-07-29T00:14:42.242 DEBUG:boto:path=/hiiqos-54/?prefix=a 2023-07-29T00:14:42.242 DEBUG:boto:auth_path=/hiiqos-54/?prefix=a 2023-07-29T00:14:42.243 DEBUG:boto:Method: GET 2023-07-29T00:14:42.243 DEBUG:boto:Path: /hiiqos-54/?prefix=a 2023-07-29T00:14:42.244 DEBUG:boto:Data: 2023-07-29T00:14:42.245 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.246 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:42.247 DEBUG:boto:Port: 8001 2023-07-29T00:14:42.247 DEBUG:boto:Params: {} 2023-07-29T00:14:42.248 DEBUG:boto:Token: None 2023-07-29T00:14:42.249 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/ 2023-07-29T00:14:42.250 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:p2xUYDIYoRCIFn5nBqae9D0avjg= 2023-07-29T00:14:42.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:p2xUYDIYoRCIFn5nBqae9D0avjg=', 'Content-Length': '0'} 2023-07-29T00:14:42.254 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d61b17ab40f5062e-0064c459f2-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.255 DEBUG:boto:b'hiiqos-54a1000falsea2023-07-29T00:14:21.405Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:42.256 DEBUG:boto:path=/hiiqos-54/ 2023-07-29T00:14:42.256 DEBUG:boto:auth_path=/hiiqos-54/ 2023-07-29T00:14:42.257 DEBUG:boto:path=/hiiqos-54/?prefix=a 2023-07-29T00:14:42.257 DEBUG:boto:auth_path=/hiiqos-54/?prefix=a 2023-07-29T00:14:42.257 DEBUG:boto:Method: GET 2023-07-29T00:14:42.258 DEBUG:boto:Path: /hiiqos-54/?prefix=a 2023-07-29T00:14:42.258 DEBUG:boto:Data: 2023-07-29T00:14:42.258 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.259 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:42.259 DEBUG:boto:Port: 8000 2023-07-29T00:14:42.260 DEBUG:boto:Params: {} 2023-07-29T00:14:42.260 DEBUG:boto:Token: None 2023-07-29T00:14:42.260 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/ 2023-07-29T00:14:42.261 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:p2xUYDIYoRCIFn5nBqae9D0avjg= 2023-07-29T00:14:42.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:p2xUYDIYoRCIFn5nBqae9D0avjg=', 'Content-Length': '0'} 2023-07-29T00:14:42.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa0e651925e91880-0064c459f2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.264 DEBUG:boto:b'hiiqos-54a1000falsea2023-07-29T00:14:21.405Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:42.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-07-29T00:14:42.265 DEBUG:boto:path=/hiiqos-54/b 2023-07-29T00:14:42.266 DEBUG:boto:auth_path=/hiiqos-54/b 2023-07-29T00:14:42.266 DEBUG:boto:path=/hiiqos-54/b?versionId=null 2023-07-29T00:14:42.266 DEBUG:boto:auth_path=/hiiqos-54/b?versionId=null 2023-07-29T00:14:42.267 DEBUG:boto:Method: GET 2023-07-29T00:14:42.267 DEBUG:boto:Path: /hiiqos-54/b?versionId=null 2023-07-29T00:14:42.268 DEBUG:boto:Data: 2023-07-29T00:14:42.268 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.269 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:42.270 DEBUG:boto:Port: 8001 2023-07-29T00:14:42.271 DEBUG:boto:Params: {} 2023-07-29T00:14:42.272 DEBUG:boto:Token: None 2023-07-29T00:14:42.273 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/b?versionId=null 2023-07-29T00:14:42.273 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fkqvXxJWTrlQ6JcyHTOcxvHrMSg= 2023-07-29T00:14:42.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fkqvXxJWTrlQ6JcyHTOcxvHrMSg=', 'Content-Length': '0'} 2023-07-29T00:14:42.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-54:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061ade22db7025616-0064c459f2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.276 DEBUG:boto:path=/hiiqos-54/b 2023-07-29T00:14:42.276 DEBUG:boto:auth_path=/hiiqos-54/b 2023-07-29T00:14:42.277 DEBUG:boto:path=/hiiqos-54/b?versionId=null 2023-07-29T00:14:42.277 DEBUG:boto:auth_path=/hiiqos-54/b?versionId=null 2023-07-29T00:14:42.277 DEBUG:boto:Method: GET 2023-07-29T00:14:42.278 DEBUG:boto:Path: /hiiqos-54/b?versionId=null 2023-07-29T00:14:42.278 DEBUG:boto:Data: 2023-07-29T00:14:42.278 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.279 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:42.279 DEBUG:boto:Port: 8000 2023-07-29T00:14:42.280 DEBUG:boto:Params: {} 2023-07-29T00:14:42.280 DEBUG:boto:Token: None 2023-07-29T00:14:42.280 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/b?versionId=null 2023-07-29T00:14:42.281 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fkqvXxJWTrlQ6JcyHTOcxvHrMSg= 2023-07-29T00:14:42.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fkqvXxJWTrlQ6JcyHTOcxvHrMSg=', 'Content-Length': '0'} 2023-07-29T00:14:42.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9534fcc13ed2fd7-0064c459f2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.284 DEBUG:boto:path=/hiiqos-54/b 2023-07-29T00:14:42.284 DEBUG:boto:auth_path=/hiiqos-54/b 2023-07-29T00:14:42.284 DEBUG:boto:path=/hiiqos-54/b?versionId=null 2023-07-29T00:14:42.285 DEBUG:boto:auth_path=/hiiqos-54/b?versionId=null 2023-07-29T00:14:42.285 DEBUG:boto:Method: HEAD 2023-07-29T00:14:42.285 DEBUG:boto:Path: /hiiqos-54/b?versionId=null 2023-07-29T00:14:42.286 DEBUG:boto:Data: 2023-07-29T00:14:42.286 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.286 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:42.287 DEBUG:boto:Port: 8001 2023-07-29T00:14:42.287 DEBUG:boto:Params: {} 2023-07-29T00:14:42.287 DEBUG:boto:Token: None 2023-07-29T00:14:42.288 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/b?versionId=null 2023-07-29T00:14:42.288 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cI7gXKj64dHr3bEUZSBqpVsDv78= 2023-07-29T00:14:42.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cI7gXKj64dHr3bEUZSBqpVsDv78=', 'Content-Length': '0'} 2023-07-29T00:14:42.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-54:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f36ce3a91613651-0064c459f2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.291 DEBUG:boto:path=/hiiqos-54/b 2023-07-29T00:14:42.291 DEBUG:boto:auth_path=/hiiqos-54/b 2023-07-29T00:14:42.292 DEBUG:boto:path=/hiiqos-54/b?versionId=null 2023-07-29T00:14:42.292 DEBUG:boto:auth_path=/hiiqos-54/b?versionId=null 2023-07-29T00:14:42.293 DEBUG:boto:Method: HEAD 2023-07-29T00:14:42.293 DEBUG:boto:Path: /hiiqos-54/b?versionId=null 2023-07-29T00:14:42.293 DEBUG:boto:Data: 2023-07-29T00:14:42.294 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.294 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:42.294 DEBUG:boto:Port: 8000 2023-07-29T00:14:42.295 DEBUG:boto:Params: {} 2023-07-29T00:14:42.295 DEBUG:boto:Token: None 2023-07-29T00:14:42.295 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/b?versionId=null 2023-07-29T00:14:42.296 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cI7gXKj64dHr3bEUZSBqpVsDv78= 2023-07-29T00:14:42.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cI7gXKj64dHr3bEUZSBqpVsDv78=', 'Content-Length': '0'} 2023-07-29T00:14:42.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001082403d6c865cdc-0064c459f2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-07-29T00:14:42.299 DEBUG:boto:path=/hiiqos-54/b 2023-07-29T00:14:42.299 DEBUG:boto:auth_path=/hiiqos-54/b 2023-07-29T00:14:42.300 DEBUG:boto:path=/hiiqos-54/b?versionId=null 2023-07-29T00:14:42.300 DEBUG:boto:auth_path=/hiiqos-54/b?versionId=null 2023-07-29T00:14:42.301 DEBUG:boto:Method: GET 2023-07-29T00:14:42.301 DEBUG:boto:Path: /hiiqos-54/b?versionId=null 2023-07-29T00:14:42.302 DEBUG:boto:Data: 2023-07-29T00:14:42.302 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.302 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:42.303 DEBUG:boto:Port: 8001 2023-07-29T00:14:42.303 DEBUG:boto:Params: {} 2023-07-29T00:14:42.303 DEBUG:boto:Token: None 2023-07-29T00:14:42.304 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/b?versionId=null 2023-07-29T00:14:42.304 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fkqvXxJWTrlQ6JcyHTOcxvHrMSg= 2023-07-29T00:14:42.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fkqvXxJWTrlQ6JcyHTOcxvHrMSg=', 'Content-Length': '0'} 2023-07-29T00:14:42.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-54:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b452de8307d42105-0064c459f2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.307 DEBUG:boto:path=/hiiqos-54/b 2023-07-29T00:14:42.308 DEBUG:boto:auth_path=/hiiqos-54/b 2023-07-29T00:14:42.308 DEBUG:boto:path=/hiiqos-54/b?versionId=null 2023-07-29T00:14:42.308 DEBUG:boto:auth_path=/hiiqos-54/b?versionId=null 2023-07-29T00:14:42.309 DEBUG:boto:Method: GET 2023-07-29T00:14:42.309 DEBUG:boto:Path: /hiiqos-54/b?versionId=null 2023-07-29T00:14:42.309 DEBUG:boto:Data: 2023-07-29T00:14:42.310 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.310 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:42.310 DEBUG:boto:Port: 8000 2023-07-29T00:14:42.311 DEBUG:boto:Params: {} 2023-07-29T00:14:42.311 DEBUG:boto:Token: None 2023-07-29T00:14:42.312 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/b?versionId=null 2023-07-29T00:14:42.312 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fkqvXxJWTrlQ6JcyHTOcxvHrMSg= 2023-07-29T00:14:42.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fkqvXxJWTrlQ6JcyHTOcxvHrMSg=', 'Content-Length': '0'} 2023-07-29T00:14:42.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009dd9cf859629fc31-0064c459f2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.315 DEBUG:boto:path=/hiiqos-54/ 2023-07-29T00:14:42.315 DEBUG:boto:auth_path=/hiiqos-54/ 2023-07-29T00:14:42.316 DEBUG:boto:path=/hiiqos-54/?prefix=b 2023-07-29T00:14:42.316 DEBUG:boto:auth_path=/hiiqos-54/?prefix=b 2023-07-29T00:14:42.316 DEBUG:boto:Method: GET 2023-07-29T00:14:42.317 DEBUG:boto:Path: /hiiqos-54/?prefix=b 2023-07-29T00:14:42.317 DEBUG:boto:Data: 2023-07-29T00:14:42.317 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.318 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:42.318 DEBUG:boto:Port: 8001 2023-07-29T00:14:42.318 DEBUG:boto:Params: {} 2023-07-29T00:14:42.319 DEBUG:boto:Token: None 2023-07-29T00:14:42.319 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/ 2023-07-29T00:14:42.320 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:p2xUYDIYoRCIFn5nBqae9D0avjg= 2023-07-29T00:14:42.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:p2xUYDIYoRCIFn5nBqae9D0avjg=', 'Content-Length': '0'} 2023-07-29T00:14:42.323 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6097a715868f5c1-0064c459f2-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.323 DEBUG:boto:b'hiiqos-54b1000falseb2023-07-29T00:14:21.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:42.324 DEBUG:boto:path=/hiiqos-54/ 2023-07-29T00:14:42.324 DEBUG:boto:auth_path=/hiiqos-54/ 2023-07-29T00:14:42.325 DEBUG:boto:path=/hiiqos-54/?prefix=b 2023-07-29T00:14:42.325 DEBUG:boto:auth_path=/hiiqos-54/?prefix=b 2023-07-29T00:14:42.325 DEBUG:boto:Method: GET 2023-07-29T00:14:42.326 DEBUG:boto:Path: /hiiqos-54/?prefix=b 2023-07-29T00:14:42.326 DEBUG:boto:Data: 2023-07-29T00:14:42.326 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.327 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:42.327 DEBUG:boto:Port: 8000 2023-07-29T00:14:42.327 DEBUG:boto:Params: {} 2023-07-29T00:14:42.328 DEBUG:boto:Token: None 2023-07-29T00:14:42.328 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/ 2023-07-29T00:14:42.329 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:p2xUYDIYoRCIFn5nBqae9D0avjg= 2023-07-29T00:14:42.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:p2xUYDIYoRCIFn5nBqae9D0avjg=', 'Content-Length': '0'} 2023-07-29T00:14:42.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd174f8196705752-0064c459f2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.332 DEBUG:boto:b'hiiqos-54b1000falseb2023-07-29T00:14:21.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:42.332 DEBUG:boto:path=/hiiqos-54/b 2023-07-29T00:14:42.333 DEBUG:boto:auth_path=/hiiqos-54/b 2023-07-29T00:14:42.333 DEBUG:boto:Method: HEAD 2023-07-29T00:14:42.333 DEBUG:boto:Path: /hiiqos-54/b 2023-07-29T00:14:42.334 DEBUG:boto:Data: 2023-07-29T00:14:42.334 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.334 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:42.335 DEBUG:boto:Port: 8001 2023-07-29T00:14:42.335 DEBUG:boto:Params: {} 2023-07-29T00:14:42.336 DEBUG:boto:Token: None 2023-07-29T00:14:42.336 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/b 2023-07-29T00:14:42.336 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Ek2T+hUNWnZrlwAP05MqA44sRLM= 2023-07-29T00:14:42.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Ek2T+hUNWnZrlwAP05MqA44sRLM=', 'Content-Length': '0'} 2023-07-29T00:14:42.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-54:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c7cab7687c89937-0064c459f2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.339 DEBUG:boto:path=/hiiqos-54/b 2023-07-29T00:14:42.339 DEBUG:boto:auth_path=/hiiqos-54/b 2023-07-29T00:14:42.339 DEBUG:boto:Method: HEAD 2023-07-29T00:14:42.340 DEBUG:boto:Path: /hiiqos-54/b 2023-07-29T00:14:42.340 DEBUG:boto:Data: 2023-07-29T00:14:42.341 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.341 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:42.341 DEBUG:boto:Port: 8000 2023-07-29T00:14:42.342 DEBUG:boto:Params: {} 2023-07-29T00:14:42.342 DEBUG:boto:Token: None 2023-07-29T00:14:42.342 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/b 2023-07-29T00:14:42.343 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Ek2T+hUNWnZrlwAP05MqA44sRLM= 2023-07-29T00:14:42.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Ek2T+hUNWnZrlwAP05MqA44sRLM=', 'Content-Length': '0'} 2023-07-29T00:14:42.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004831cd08da51ee9a-0064c459f2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-07-29T00:14:42.346 DEBUG:boto:path=/hiiqos-54/b 2023-07-29T00:14:42.346 DEBUG:boto:auth_path=/hiiqos-54/b 2023-07-29T00:14:42.346 DEBUG:boto:Method: GET 2023-07-29T00:14:42.347 DEBUG:boto:Path: /hiiqos-54/b 2023-07-29T00:14:42.347 DEBUG:boto:Data: 2023-07-29T00:14:42.347 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.348 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:42.348 DEBUG:boto:Port: 8001 2023-07-29T00:14:42.349 DEBUG:boto:Params: {} 2023-07-29T00:14:42.349 DEBUG:boto:Token: None 2023-07-29T00:14:42.349 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/b 2023-07-29T00:14:42.350 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:L8PiLts56IsNt40LK8wXmlHUWe4= 2023-07-29T00:14:42.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:L8PiLts56IsNt40LK8wXmlHUWe4=', 'Content-Length': '0'} 2023-07-29T00:14:42.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-54:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da4f1fe89836e4c5-0064c459f2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.352 DEBUG:boto:path=/hiiqos-54/b 2023-07-29T00:14:42.353 DEBUG:boto:auth_path=/hiiqos-54/b 2023-07-29T00:14:42.353 DEBUG:boto:Method: GET 2023-07-29T00:14:42.354 DEBUG:boto:Path: /hiiqos-54/b 2023-07-29T00:14:42.354 DEBUG:boto:Data: 2023-07-29T00:14:42.354 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.355 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:42.355 DEBUG:boto:Port: 8000 2023-07-29T00:14:42.355 DEBUG:boto:Params: {} 2023-07-29T00:14:42.356 DEBUG:boto:Token: None 2023-07-29T00:14:42.356 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/b 2023-07-29T00:14:42.357 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:L8PiLts56IsNt40LK8wXmlHUWe4= 2023-07-29T00:14:42.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:L8PiLts56IsNt40LK8wXmlHUWe4=', 'Content-Length': '0'} 2023-07-29T00:14:42.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f68739167552e03-0064c459f2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.400 DEBUG:boto:path=/hiiqos-54/ 2023-07-29T00:14:42.401 DEBUG:boto:auth_path=/hiiqos-54/ 2023-07-29T00:14:42.401 DEBUG:boto:path=/hiiqos-54/?prefix=b 2023-07-29T00:14:42.402 DEBUG:boto:auth_path=/hiiqos-54/?prefix=b 2023-07-29T00:14:42.402 DEBUG:boto:Method: GET 2023-07-29T00:14:42.402 DEBUG:boto:Path: /hiiqos-54/?prefix=b 2023-07-29T00:14:42.403 DEBUG:boto:Data: 2023-07-29T00:14:42.403 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.403 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:42.404 DEBUG:boto:Port: 8001 2023-07-29T00:14:42.404 DEBUG:boto:Params: {} 2023-07-29T00:14:42.405 DEBUG:boto:Token: None 2023-07-29T00:14:42.405 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/ 2023-07-29T00:14:42.405 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:p2xUYDIYoRCIFn5nBqae9D0avjg= 2023-07-29T00:14:42.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:p2xUYDIYoRCIFn5nBqae9D0avjg=', 'Content-Length': '0'} 2023-07-29T00:14:42.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcb35c59d41e955b-0064c459f2-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.409 DEBUG:boto:b'hiiqos-54b1000falseb2023-07-29T00:14:21.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:42.409 DEBUG:boto:path=/hiiqos-54/ 2023-07-29T00:14:42.410 DEBUG:boto:auth_path=/hiiqos-54/ 2023-07-29T00:14:42.410 DEBUG:boto:path=/hiiqos-54/?prefix=b 2023-07-29T00:14:42.411 DEBUG:boto:auth_path=/hiiqos-54/?prefix=b 2023-07-29T00:14:42.411 DEBUG:boto:Method: GET 2023-07-29T00:14:42.411 DEBUG:boto:Path: /hiiqos-54/?prefix=b 2023-07-29T00:14:42.412 DEBUG:boto:Data: 2023-07-29T00:14:42.412 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.412 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:42.413 DEBUG:boto:Port: 8000 2023-07-29T00:14:42.413 DEBUG:boto:Params: {} 2023-07-29T00:14:42.414 DEBUG:boto:Token: None 2023-07-29T00:14:42.414 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/ 2023-07-29T00:14:42.414 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:p2xUYDIYoRCIFn5nBqae9D0avjg= 2023-07-29T00:14:42.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:p2xUYDIYoRCIFn5nBqae9D0avjg=', 'Content-Length': '0'} 2023-07-29T00:14:42.417 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fde20027096c1879-0064c459f2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.418 DEBUG:boto:b'hiiqos-54b1000falseb2023-07-29T00:14:21.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:42.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-07-29T00:14:42.419 DEBUG:boto:path=/hiiqos-54/c 2023-07-29T00:14:42.419 DEBUG:boto:auth_path=/hiiqos-54/c 2023-07-29T00:14:42.420 DEBUG:boto:path=/hiiqos-54/c?versionId=null 2023-07-29T00:14:42.420 DEBUG:boto:auth_path=/hiiqos-54/c?versionId=null 2023-07-29T00:14:42.420 DEBUG:boto:Method: GET 2023-07-29T00:14:42.421 DEBUG:boto:Path: /hiiqos-54/c?versionId=null 2023-07-29T00:14:42.421 DEBUG:boto:Data: 2023-07-29T00:14:42.422 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.422 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:42.422 DEBUG:boto:Port: 8001 2023-07-29T00:14:42.423 DEBUG:boto:Params: {} 2023-07-29T00:14:42.423 DEBUG:boto:Token: None 2023-07-29T00:14:42.423 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/c?versionId=null 2023-07-29T00:14:42.424 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IdWnVsj6T/1rlnltRvz24k6EkZc= 2023-07-29T00:14:42.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IdWnVsj6T/1rlnltRvz24k6EkZc=', 'Content-Length': '0'} 2023-07-29T00:14:42.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-54:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f7432b5ed8c294c-0064c459f2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.478 DEBUG:boto:path=/hiiqos-54/c 2023-07-29T00:14:42.478 DEBUG:boto:auth_path=/hiiqos-54/c 2023-07-29T00:14:42.479 DEBUG:boto:path=/hiiqos-54/c?versionId=null 2023-07-29T00:14:42.479 DEBUG:boto:auth_path=/hiiqos-54/c?versionId=null 2023-07-29T00:14:42.479 DEBUG:boto:Method: GET 2023-07-29T00:14:42.480 DEBUG:boto:Path: /hiiqos-54/c?versionId=null 2023-07-29T00:14:42.480 DEBUG:boto:Data: 2023-07-29T00:14:42.481 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.481 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:42.481 DEBUG:boto:Port: 8000 2023-07-29T00:14:42.482 DEBUG:boto:Params: {} 2023-07-29T00:14:42.482 DEBUG:boto:Token: None 2023-07-29T00:14:42.483 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/c?versionId=null 2023-07-29T00:14:42.483 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IdWnVsj6T/1rlnltRvz24k6EkZc= 2023-07-29T00:14:42.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IdWnVsj6T/1rlnltRvz24k6EkZc=', 'Content-Length': '0'} 2023-07-29T00:14:42.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048da24c7afe2097d-0064c459f2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.487 DEBUG:boto:path=/hiiqos-54/c 2023-07-29T00:14:42.488 DEBUG:boto:auth_path=/hiiqos-54/c 2023-07-29T00:14:42.489 DEBUG:boto:path=/hiiqos-54/c?versionId=null 2023-07-29T00:14:42.490 DEBUG:boto:auth_path=/hiiqos-54/c?versionId=null 2023-07-29T00:14:42.490 DEBUG:boto:Method: HEAD 2023-07-29T00:14:42.491 DEBUG:boto:Path: /hiiqos-54/c?versionId=null 2023-07-29T00:14:42.492 DEBUG:boto:Data: 2023-07-29T00:14:42.493 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.494 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:42.494 DEBUG:boto:Port: 8001 2023-07-29T00:14:42.495 DEBUG:boto:Params: {} 2023-07-29T00:14:42.495 DEBUG:boto:Token: None 2023-07-29T00:14:42.496 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/c?versionId=null 2023-07-29T00:14:42.497 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uSZ7HhpGNMcZ/z00FEgeWL6bbhY= 2023-07-29T00:14:42.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uSZ7HhpGNMcZ/z00FEgeWL6bbhY=', 'Content-Length': '0'} 2023-07-29T00:14:42.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-54:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6579148e4aad044-0064c459f2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.500 DEBUG:boto:path=/hiiqos-54/c 2023-07-29T00:14:42.501 DEBUG:boto:auth_path=/hiiqos-54/c 2023-07-29T00:14:42.501 DEBUG:boto:path=/hiiqos-54/c?versionId=null 2023-07-29T00:14:42.501 DEBUG:boto:auth_path=/hiiqos-54/c?versionId=null 2023-07-29T00:14:42.502 DEBUG:boto:Method: HEAD 2023-07-29T00:14:42.502 DEBUG:boto:Path: /hiiqos-54/c?versionId=null 2023-07-29T00:14:42.503 DEBUG:boto:Data: 2023-07-29T00:14:42.503 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.504 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:42.505 DEBUG:boto:Port: 8000 2023-07-29T00:14:42.505 DEBUG:boto:Params: {} 2023-07-29T00:14:42.506 DEBUG:boto:Token: None 2023-07-29T00:14:42.506 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/c?versionId=null 2023-07-29T00:14:42.506 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uSZ7HhpGNMcZ/z00FEgeWL6bbhY= 2023-07-29T00:14:42.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uSZ7HhpGNMcZ/z00FEgeWL6bbhY=', 'Content-Length': '0'} 2023-07-29T00:14:42.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000768bbd02461eac10-0064c459f2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-07-29T00:14:42.510 DEBUG:boto:path=/hiiqos-54/c 2023-07-29T00:14:42.510 DEBUG:boto:auth_path=/hiiqos-54/c 2023-07-29T00:14:42.510 DEBUG:boto:path=/hiiqos-54/c?versionId=null 2023-07-29T00:14:42.511 DEBUG:boto:auth_path=/hiiqos-54/c?versionId=null 2023-07-29T00:14:42.511 DEBUG:boto:Method: GET 2023-07-29T00:14:42.511 DEBUG:boto:Path: /hiiqos-54/c?versionId=null 2023-07-29T00:14:42.512 DEBUG:boto:Data: 2023-07-29T00:14:42.512 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.513 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:42.513 DEBUG:boto:Port: 8001 2023-07-29T00:14:42.513 DEBUG:boto:Params: {} 2023-07-29T00:14:42.514 DEBUG:boto:Token: None 2023-07-29T00:14:42.514 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/c?versionId=null 2023-07-29T00:14:42.515 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IdWnVsj6T/1rlnltRvz24k6EkZc= 2023-07-29T00:14:42.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IdWnVsj6T/1rlnltRvz24k6EkZc=', 'Content-Length': '0'} 2023-07-29T00:14:42.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-54:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e878b59977d5cb76-0064c459f2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.518 DEBUG:boto:path=/hiiqos-54/c 2023-07-29T00:14:42.518 DEBUG:boto:auth_path=/hiiqos-54/c 2023-07-29T00:14:42.518 DEBUG:boto:path=/hiiqos-54/c?versionId=null 2023-07-29T00:14:42.519 DEBUG:boto:auth_path=/hiiqos-54/c?versionId=null 2023-07-29T00:14:42.519 DEBUG:boto:Method: GET 2023-07-29T00:14:42.520 DEBUG:boto:Path: /hiiqos-54/c?versionId=null 2023-07-29T00:14:42.520 DEBUG:boto:Data: 2023-07-29T00:14:42.520 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.521 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:42.521 DEBUG:boto:Port: 8000 2023-07-29T00:14:42.521 DEBUG:boto:Params: {} 2023-07-29T00:14:42.522 DEBUG:boto:Token: None 2023-07-29T00:14:42.522 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/c?versionId=null 2023-07-29T00:14:42.523 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IdWnVsj6T/1rlnltRvz24k6EkZc= 2023-07-29T00:14:42.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IdWnVsj6T/1rlnltRvz24k6EkZc=', 'Content-Length': '0'} 2023-07-29T00:14:42.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094bbbec710e75ea0-0064c459f2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.526 DEBUG:boto:path=/hiiqos-54/ 2023-07-29T00:14:42.526 DEBUG:boto:auth_path=/hiiqos-54/ 2023-07-29T00:14:42.527 DEBUG:boto:path=/hiiqos-54/?prefix=c 2023-07-29T00:14:42.527 DEBUG:boto:auth_path=/hiiqos-54/?prefix=c 2023-07-29T00:14:42.527 DEBUG:boto:Method: GET 2023-07-29T00:14:42.528 DEBUG:boto:Path: /hiiqos-54/?prefix=c 2023-07-29T00:14:42.528 DEBUG:boto:Data: 2023-07-29T00:14:42.529 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.529 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:42.529 DEBUG:boto:Port: 8001 2023-07-29T00:14:42.530 DEBUG:boto:Params: {} 2023-07-29T00:14:42.530 DEBUG:boto:Token: None 2023-07-29T00:14:42.531 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/ 2023-07-29T00:14:42.531 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:p2xUYDIYoRCIFn5nBqae9D0avjg= 2023-07-29T00:14:42.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:p2xUYDIYoRCIFn5nBqae9D0avjg=', 'Content-Length': '0'} 2023-07-29T00:14:42.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007687e309cc7780b7-0064c459f2-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.534 DEBUG:boto:b'hiiqos-54c1000falsec2023-07-29T00:14:21.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:42.535 DEBUG:boto:path=/hiiqos-54/ 2023-07-29T00:14:42.535 DEBUG:boto:auth_path=/hiiqos-54/ 2023-07-29T00:14:42.536 DEBUG:boto:path=/hiiqos-54/?prefix=c 2023-07-29T00:14:42.536 DEBUG:boto:auth_path=/hiiqos-54/?prefix=c 2023-07-29T00:14:42.537 DEBUG:boto:Method: GET 2023-07-29T00:14:42.537 DEBUG:boto:Path: /hiiqos-54/?prefix=c 2023-07-29T00:14:42.537 DEBUG:boto:Data: 2023-07-29T00:14:42.538 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.538 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:42.538 DEBUG:boto:Port: 8000 2023-07-29T00:14:42.539 DEBUG:boto:Params: {} 2023-07-29T00:14:42.539 DEBUG:boto:Token: None 2023-07-29T00:14:42.540 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/ 2023-07-29T00:14:42.540 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:p2xUYDIYoRCIFn5nBqae9D0avjg= 2023-07-29T00:14:42.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:p2xUYDIYoRCIFn5nBqae9D0avjg=', 'Content-Length': '0'} 2023-07-29T00:14:42.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e571ffd3a8f0d15-0064c459f2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.584 DEBUG:boto:b'hiiqos-54c1000falsec2023-07-29T00:14:21.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:42.586 DEBUG:boto:path=/hiiqos-54/c 2023-07-29T00:14:42.587 DEBUG:boto:auth_path=/hiiqos-54/c 2023-07-29T00:14:42.587 DEBUG:boto:Method: HEAD 2023-07-29T00:14:42.588 DEBUG:boto:Path: /hiiqos-54/c 2023-07-29T00:14:42.589 DEBUG:boto:Data: 2023-07-29T00:14:42.589 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.590 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:42.591 DEBUG:boto:Port: 8001 2023-07-29T00:14:42.592 DEBUG:boto:Params: {} 2023-07-29T00:14:42.592 DEBUG:boto:Token: None 2023-07-29T00:14:42.593 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/c 2023-07-29T00:14:42.593 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lSHCUTZaOIA7nPGrpQxTHgMHPyU= 2023-07-29T00:14:42.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lSHCUTZaOIA7nPGrpQxTHgMHPyU=', 'Content-Length': '0'} 2023-07-29T00:14:42.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-54:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000632860d2d339624c-0064c459f2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.596 DEBUG:boto:path=/hiiqos-54/c 2023-07-29T00:14:42.596 DEBUG:boto:auth_path=/hiiqos-54/c 2023-07-29T00:14:42.597 DEBUG:boto:Method: HEAD 2023-07-29T00:14:42.597 DEBUG:boto:Path: /hiiqos-54/c 2023-07-29T00:14:42.597 DEBUG:boto:Data: 2023-07-29T00:14:42.598 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.598 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:42.598 DEBUG:boto:Port: 8000 2023-07-29T00:14:42.599 DEBUG:boto:Params: {} 2023-07-29T00:14:42.599 DEBUG:boto:Token: None 2023-07-29T00:14:42.599 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/c 2023-07-29T00:14:42.600 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lSHCUTZaOIA7nPGrpQxTHgMHPyU= 2023-07-29T00:14:42.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lSHCUTZaOIA7nPGrpQxTHgMHPyU=', 'Content-Length': '0'} 2023-07-29T00:14:42.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ea954ab4346ffa5-0064c459f2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-07-29T00:14:42.603 DEBUG:boto:path=/hiiqos-54/c 2023-07-29T00:14:42.603 DEBUG:boto:auth_path=/hiiqos-54/c 2023-07-29T00:14:42.604 DEBUG:boto:Method: GET 2023-07-29T00:14:42.604 DEBUG:boto:Path: /hiiqos-54/c 2023-07-29T00:14:42.604 DEBUG:boto:Data: 2023-07-29T00:14:42.605 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.605 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:42.605 DEBUG:boto:Port: 8001 2023-07-29T00:14:42.606 DEBUG:boto:Params: {} 2023-07-29T00:14:42.606 DEBUG:boto:Token: None 2023-07-29T00:14:42.607 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/c 2023-07-29T00:14:42.607 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2gI42KZYKdnqAdz1IsoY1tvxX10= 2023-07-29T00:14:42.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2gI42KZYKdnqAdz1IsoY1tvxX10=', 'Content-Length': '0'} 2023-07-29T00:14:42.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-54:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001850143e81823027-0064c459f2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.610 DEBUG:boto:path=/hiiqos-54/c 2023-07-29T00:14:42.610 DEBUG:boto:auth_path=/hiiqos-54/c 2023-07-29T00:14:42.610 DEBUG:boto:Method: GET 2023-07-29T00:14:42.611 DEBUG:boto:Path: /hiiqos-54/c 2023-07-29T00:14:42.611 DEBUG:boto:Data: 2023-07-29T00:14:42.612 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.612 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:42.612 DEBUG:boto:Port: 8000 2023-07-29T00:14:42.613 DEBUG:boto:Params: {} 2023-07-29T00:14:42.613 DEBUG:boto:Token: None 2023-07-29T00:14:42.613 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/c 2023-07-29T00:14:42.614 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2gI42KZYKdnqAdz1IsoY1tvxX10= 2023-07-29T00:14:42.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2gI42KZYKdnqAdz1IsoY1tvxX10=', 'Content-Length': '0'} 2023-07-29T00:14:42.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eab050c8105174ad-0064c459f2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.617 DEBUG:boto:path=/hiiqos-54/ 2023-07-29T00:14:42.617 DEBUG:boto:auth_path=/hiiqos-54/ 2023-07-29T00:14:42.618 DEBUG:boto:path=/hiiqos-54/?prefix=c 2023-07-29T00:14:42.618 DEBUG:boto:auth_path=/hiiqos-54/?prefix=c 2023-07-29T00:14:42.618 DEBUG:boto:Method: GET 2023-07-29T00:14:42.619 DEBUG:boto:Path: /hiiqos-54/?prefix=c 2023-07-29T00:14:42.619 DEBUG:boto:Data: 2023-07-29T00:14:42.619 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.620 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:42.620 DEBUG:boto:Port: 8001 2023-07-29T00:14:42.621 DEBUG:boto:Params: {} 2023-07-29T00:14:42.621 DEBUG:boto:Token: None 2023-07-29T00:14:42.621 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/ 2023-07-29T00:14:42.622 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:p2xUYDIYoRCIFn5nBqae9D0avjg= 2023-07-29T00:14:42.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:p2xUYDIYoRCIFn5nBqae9D0avjg=', 'Content-Length': '0'} 2023-07-29T00:14:42.625 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000131280987b278acb-0064c459f2-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.625 DEBUG:boto:b'hiiqos-54c1000falsec2023-07-29T00:14:21.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:42.626 DEBUG:boto:path=/hiiqos-54/ 2023-07-29T00:14:42.626 DEBUG:boto:auth_path=/hiiqos-54/ 2023-07-29T00:14:42.627 DEBUG:boto:path=/hiiqos-54/?prefix=c 2023-07-29T00:14:42.627 DEBUG:boto:auth_path=/hiiqos-54/?prefix=c 2023-07-29T00:14:42.627 DEBUG:boto:Method: GET 2023-07-29T00:14:42.628 DEBUG:boto:Path: /hiiqos-54/?prefix=c 2023-07-29T00:14:42.628 DEBUG:boto:Data: 2023-07-29T00:14:42.629 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.629 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:42.629 DEBUG:boto:Port: 8000 2023-07-29T00:14:42.630 DEBUG:boto:Params: {} 2023-07-29T00:14:42.630 DEBUG:boto:Token: None 2023-07-29T00:14:42.630 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/ 2023-07-29T00:14:42.631 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:p2xUYDIYoRCIFn5nBqae9D0avjg= 2023-07-29T00:14:42.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:p2xUYDIYoRCIFn5nBqae9D0avjg=', 'Content-Length': '0'} 2023-07-29T00:14:42.633 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000443d09d0f5ac1601-0064c459f2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.634 DEBUG:boto:b'hiiqos-54c1000falsec2023-07-29T00:14:21.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:42.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-07-29T00:14:42.635 DEBUG:boto:path=/hiiqos-54/d 2023-07-29T00:14:42.635 DEBUG:boto:auth_path=/hiiqos-54/d 2023-07-29T00:14:42.636 DEBUG:boto:path=/hiiqos-54/d?versionId=null 2023-07-29T00:14:42.636 DEBUG:boto:auth_path=/hiiqos-54/d?versionId=null 2023-07-29T00:14:42.636 DEBUG:boto:Method: GET 2023-07-29T00:14:42.637 DEBUG:boto:Path: /hiiqos-54/d?versionId=null 2023-07-29T00:14:42.637 DEBUG:boto:Data: 2023-07-29T00:14:42.638 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.638 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:42.638 DEBUG:boto:Port: 8001 2023-07-29T00:14:42.639 DEBUG:boto:Params: {} 2023-07-29T00:14:42.639 DEBUG:boto:Token: None 2023-07-29T00:14:42.639 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/d?versionId=null 2023-07-29T00:14:42.640 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Nei74ErziF87nn+uvMBTDF+kApU= 2023-07-29T00:14:42.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Nei74ErziF87nn+uvMBTDF+kApU=', 'Content-Length': '0'} 2023-07-29T00:14:42.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-54:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013fc8434fea5ac78-0064c459f2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.684 DEBUG:boto:path=/hiiqos-54/d 2023-07-29T00:14:42.685 DEBUG:boto:auth_path=/hiiqos-54/d 2023-07-29T00:14:42.685 DEBUG:boto:path=/hiiqos-54/d?versionId=null 2023-07-29T00:14:42.686 DEBUG:boto:auth_path=/hiiqos-54/d?versionId=null 2023-07-29T00:14:42.686 DEBUG:boto:Method: GET 2023-07-29T00:14:42.686 DEBUG:boto:Path: /hiiqos-54/d?versionId=null 2023-07-29T00:14:42.687 DEBUG:boto:Data: 2023-07-29T00:14:42.687 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.688 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:42.688 DEBUG:boto:Port: 8000 2023-07-29T00:14:42.689 DEBUG:boto:Params: {} 2023-07-29T00:14:42.689 DEBUG:boto:Token: None 2023-07-29T00:14:42.690 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/d?versionId=null 2023-07-29T00:14:42.690 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Nei74ErziF87nn+uvMBTDF+kApU= 2023-07-29T00:14:42.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Nei74ErziF87nn+uvMBTDF+kApU=', 'Content-Length': '0'} 2023-07-29T00:14:42.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062dc20d83871b415-0064c459f2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.693 DEBUG:boto:path=/hiiqos-54/d 2023-07-29T00:14:42.694 DEBUG:boto:auth_path=/hiiqos-54/d 2023-07-29T00:14:42.694 DEBUG:boto:path=/hiiqos-54/d?versionId=null 2023-07-29T00:14:42.694 DEBUG:boto:auth_path=/hiiqos-54/d?versionId=null 2023-07-29T00:14:42.695 DEBUG:boto:Method: HEAD 2023-07-29T00:14:42.695 DEBUG:boto:Path: /hiiqos-54/d?versionId=null 2023-07-29T00:14:42.696 DEBUG:boto:Data: 2023-07-29T00:14:42.696 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.696 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:42.697 DEBUG:boto:Port: 8001 2023-07-29T00:14:42.697 DEBUG:boto:Params: {} 2023-07-29T00:14:42.697 DEBUG:boto:Token: None 2023-07-29T00:14:42.698 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/d?versionId=null 2023-07-29T00:14:42.698 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OycSQigS8vH5OcDdwFOGsP3pylE= 2023-07-29T00:14:42.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OycSQigS8vH5OcDdwFOGsP3pylE=', 'Content-Length': '0'} 2023-07-29T00:14:42.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-54:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004eea102815ee7b1a-0064c459f2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.701 DEBUG:boto:path=/hiiqos-54/d 2023-07-29T00:14:42.701 DEBUG:boto:auth_path=/hiiqos-54/d 2023-07-29T00:14:42.702 DEBUG:boto:path=/hiiqos-54/d?versionId=null 2023-07-29T00:14:42.702 DEBUG:boto:auth_path=/hiiqos-54/d?versionId=null 2023-07-29T00:14:42.702 DEBUG:boto:Method: HEAD 2023-07-29T00:14:42.703 DEBUG:boto:Path: /hiiqos-54/d?versionId=null 2023-07-29T00:14:42.703 DEBUG:boto:Data: 2023-07-29T00:14:42.703 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.704 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:42.704 DEBUG:boto:Port: 8000 2023-07-29T00:14:42.705 DEBUG:boto:Params: {} 2023-07-29T00:14:42.705 DEBUG:boto:Token: None 2023-07-29T00:14:42.705 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/d?versionId=null 2023-07-29T00:14:42.706 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OycSQigS8vH5OcDdwFOGsP3pylE= 2023-07-29T00:14:42.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OycSQigS8vH5OcDdwFOGsP3pylE=', 'Content-Length': '0'} 2023-07-29T00:14:42.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc25351cecf1ff8f-0064c459f2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-07-29T00:14:42.709 DEBUG:boto:path=/hiiqos-54/d 2023-07-29T00:14:42.709 DEBUG:boto:auth_path=/hiiqos-54/d 2023-07-29T00:14:42.710 DEBUG:boto:path=/hiiqos-54/d?versionId=null 2023-07-29T00:14:42.710 DEBUG:boto:auth_path=/hiiqos-54/d?versionId=null 2023-07-29T00:14:42.711 DEBUG:boto:Method: GET 2023-07-29T00:14:42.711 DEBUG:boto:Path: /hiiqos-54/d?versionId=null 2023-07-29T00:14:42.711 DEBUG:boto:Data: 2023-07-29T00:14:42.712 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.712 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:42.713 DEBUG:boto:Port: 8001 2023-07-29T00:14:42.713 DEBUG:boto:Params: {} 2023-07-29T00:14:42.713 DEBUG:boto:Token: None 2023-07-29T00:14:42.714 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/d?versionId=null 2023-07-29T00:14:42.714 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Nei74ErziF87nn+uvMBTDF+kApU= 2023-07-29T00:14:42.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Nei74ErziF87nn+uvMBTDF+kApU=', 'Content-Length': '0'} 2023-07-29T00:14:42.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-54:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8f412937c1237c0-0064c459f2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.717 DEBUG:boto:path=/hiiqos-54/d 2023-07-29T00:14:42.718 DEBUG:boto:auth_path=/hiiqos-54/d 2023-07-29T00:14:42.718 DEBUG:boto:path=/hiiqos-54/d?versionId=null 2023-07-29T00:14:42.718 DEBUG:boto:auth_path=/hiiqos-54/d?versionId=null 2023-07-29T00:14:42.719 DEBUG:boto:Method: GET 2023-07-29T00:14:42.719 DEBUG:boto:Path: /hiiqos-54/d?versionId=null 2023-07-29T00:14:42.720 DEBUG:boto:Data: 2023-07-29T00:14:42.720 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.720 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:42.721 DEBUG:boto:Port: 8000 2023-07-29T00:14:42.721 DEBUG:boto:Params: {} 2023-07-29T00:14:42.722 DEBUG:boto:Token: None 2023-07-29T00:14:42.722 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/d?versionId=null 2023-07-29T00:14:42.723 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Nei74ErziF87nn+uvMBTDF+kApU= 2023-07-29T00:14:42.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Nei74ErziF87nn+uvMBTDF+kApU=', 'Content-Length': '0'} 2023-07-29T00:14:42.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c364fb4727ed373-0064c459f2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.768 DEBUG:boto:path=/hiiqos-54/ 2023-07-29T00:14:42.769 DEBUG:boto:auth_path=/hiiqos-54/ 2023-07-29T00:14:42.769 DEBUG:boto:path=/hiiqos-54/?prefix=d 2023-07-29T00:14:42.770 DEBUG:boto:auth_path=/hiiqos-54/?prefix=d 2023-07-29T00:14:42.770 DEBUG:boto:Method: GET 2023-07-29T00:14:42.770 DEBUG:boto:Path: /hiiqos-54/?prefix=d 2023-07-29T00:14:42.771 DEBUG:boto:Data: 2023-07-29T00:14:42.771 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.771 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:42.772 DEBUG:boto:Port: 8001 2023-07-29T00:14:42.772 DEBUG:boto:Params: {} 2023-07-29T00:14:42.773 DEBUG:boto:Token: None 2023-07-29T00:14:42.773 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/ 2023-07-29T00:14:42.774 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:p2xUYDIYoRCIFn5nBqae9D0avjg= 2023-07-29T00:14:42.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:p2xUYDIYoRCIFn5nBqae9D0avjg=', 'Content-Length': '0'} 2023-07-29T00:14:42.776 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fc0f3adea79e31c-0064c459f2-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.777 DEBUG:boto:b'hiiqos-54d1000falsed2023-07-29T00:14:21.494Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:42.777 DEBUG:boto:path=/hiiqos-54/ 2023-07-29T00:14:42.778 DEBUG:boto:auth_path=/hiiqos-54/ 2023-07-29T00:14:42.778 DEBUG:boto:path=/hiiqos-54/?prefix=d 2023-07-29T00:14:42.779 DEBUG:boto:auth_path=/hiiqos-54/?prefix=d 2023-07-29T00:14:42.779 DEBUG:boto:Method: GET 2023-07-29T00:14:42.779 DEBUG:boto:Path: /hiiqos-54/?prefix=d 2023-07-29T00:14:42.780 DEBUG:boto:Data: 2023-07-29T00:14:42.780 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.780 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:42.781 DEBUG:boto:Port: 8000 2023-07-29T00:14:42.781 DEBUG:boto:Params: {} 2023-07-29T00:14:42.782 DEBUG:boto:Token: None 2023-07-29T00:14:42.782 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/ 2023-07-29T00:14:42.782 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:p2xUYDIYoRCIFn5nBqae9D0avjg= 2023-07-29T00:14:42.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:p2xUYDIYoRCIFn5nBqae9D0avjg=', 'Content-Length': '0'} 2023-07-29T00:14:42.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e79dea64c2b4931-0064c459f2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.786 DEBUG:boto:b'hiiqos-54d1000falsed2023-07-29T00:14:21.494Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:42.786 DEBUG:boto:path=/hiiqos-54/d 2023-07-29T00:14:42.787 DEBUG:boto:auth_path=/hiiqos-54/d 2023-07-29T00:14:42.787 DEBUG:boto:Method: HEAD 2023-07-29T00:14:42.788 DEBUG:boto:Path: /hiiqos-54/d 2023-07-29T00:14:42.788 DEBUG:boto:Data: 2023-07-29T00:14:42.788 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.789 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:42.789 DEBUG:boto:Port: 8001 2023-07-29T00:14:42.789 DEBUG:boto:Params: {} 2023-07-29T00:14:42.790 DEBUG:boto:Token: None 2023-07-29T00:14:42.790 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/d 2023-07-29T00:14:42.790 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:s8bfDp17wgfpXrSKDT3V+Vv9RJo= 2023-07-29T00:14:42.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:s8bfDp17wgfpXrSKDT3V+Vv9RJo=', 'Content-Length': '0'} 2023-07-29T00:14:42.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-54:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003667e40b63ba88ed-0064c459f2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.793 DEBUG:boto:path=/hiiqos-54/d 2023-07-29T00:14:42.793 DEBUG:boto:auth_path=/hiiqos-54/d 2023-07-29T00:14:42.794 DEBUG:boto:Method: HEAD 2023-07-29T00:14:42.794 DEBUG:boto:Path: /hiiqos-54/d 2023-07-29T00:14:42.794 DEBUG:boto:Data: 2023-07-29T00:14:42.795 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.795 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:42.795 DEBUG:boto:Port: 8000 2023-07-29T00:14:42.796 DEBUG:boto:Params: {} 2023-07-29T00:14:42.796 DEBUG:boto:Token: None 2023-07-29T00:14:42.796 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/d 2023-07-29T00:14:42.797 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:s8bfDp17wgfpXrSKDT3V+Vv9RJo= 2023-07-29T00:14:42.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:s8bfDp17wgfpXrSKDT3V+Vv9RJo=', 'Content-Length': '0'} 2023-07-29T00:14:42.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e5b56b703c716af-0064c459f2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-07-29T00:14:42.800 DEBUG:boto:path=/hiiqos-54/d 2023-07-29T00:14:42.800 DEBUG:boto:auth_path=/hiiqos-54/d 2023-07-29T00:14:42.800 DEBUG:boto:Method: GET 2023-07-29T00:14:42.801 DEBUG:boto:Path: /hiiqos-54/d 2023-07-29T00:14:42.801 DEBUG:boto:Data: 2023-07-29T00:14:42.802 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.802 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:42.802 DEBUG:boto:Port: 8001 2023-07-29T00:14:42.803 DEBUG:boto:Params: {} 2023-07-29T00:14:42.803 DEBUG:boto:Token: None 2023-07-29T00:14:42.803 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/d 2023-07-29T00:14:42.804 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pKics6hkfFmgO7SXGoMRg38Wc/Y= 2023-07-29T00:14:42.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pKics6hkfFmgO7SXGoMRg38Wc/Y=', 'Content-Length': '0'} 2023-07-29T00:14:42.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-54:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc77196c0c72ff6f-0064c459f2-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.848 DEBUG:boto:path=/hiiqos-54/d 2023-07-29T00:14:42.849 DEBUG:boto:auth_path=/hiiqos-54/d 2023-07-29T00:14:42.849 DEBUG:boto:Method: GET 2023-07-29T00:14:42.850 DEBUG:boto:Path: /hiiqos-54/d 2023-07-29T00:14:42.850 DEBUG:boto:Data: 2023-07-29T00:14:42.850 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.851 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:42.851 DEBUG:boto:Port: 8000 2023-07-29T00:14:42.851 DEBUG:boto:Params: {} 2023-07-29T00:14:42.852 DEBUG:boto:Token: None 2023-07-29T00:14:42.852 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/d 2023-07-29T00:14:42.853 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pKics6hkfFmgO7SXGoMRg38Wc/Y= 2023-07-29T00:14:42.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pKics6hkfFmgO7SXGoMRg38Wc/Y=', 'Content-Length': '0'} 2023-07-29T00:14:42.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e03371f7a2345026-0064c459f2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.856 DEBUG:boto:path=/hiiqos-54/ 2023-07-29T00:14:42.856 DEBUG:boto:auth_path=/hiiqos-54/ 2023-07-29T00:14:42.856 DEBUG:boto:path=/hiiqos-54/?prefix=d 2023-07-29T00:14:42.857 DEBUG:boto:auth_path=/hiiqos-54/?prefix=d 2023-07-29T00:14:42.857 DEBUG:boto:Method: GET 2023-07-29T00:14:42.858 DEBUG:boto:Path: /hiiqos-54/?prefix=d 2023-07-29T00:14:42.858 DEBUG:boto:Data: 2023-07-29T00:14:42.858 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.859 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:42.859 DEBUG:boto:Port: 8001 2023-07-29T00:14:42.859 DEBUG:boto:Params: {} 2023-07-29T00:14:42.860 DEBUG:boto:Token: None 2023-07-29T00:14:42.860 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/ 2023-07-29T00:14:42.861 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:p2xUYDIYoRCIFn5nBqae9D0avjg= 2023-07-29T00:14:42.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:p2xUYDIYoRCIFn5nBqae9D0avjg=', 'Content-Length': '0'} 2023-07-29T00:14:42.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008714157a52cdf95c-0064c459f2-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.864 DEBUG:boto:b'hiiqos-54d1000falsed2023-07-29T00:14:21.494Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:42.864 DEBUG:boto:path=/hiiqos-54/ 2023-07-29T00:14:42.865 DEBUG:boto:auth_path=/hiiqos-54/ 2023-07-29T00:14:42.865 DEBUG:boto:path=/hiiqos-54/?prefix=d 2023-07-29T00:14:42.865 DEBUG:boto:auth_path=/hiiqos-54/?prefix=d 2023-07-29T00:14:42.866 DEBUG:boto:Method: GET 2023-07-29T00:14:42.866 DEBUG:boto:Path: /hiiqos-54/?prefix=d 2023-07-29T00:14:42.867 DEBUG:boto:Data: 2023-07-29T00:14:42.867 DEBUG:boto:Headers: {} 2023-07-29T00:14:42.867 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:42.868 DEBUG:boto:Port: 8000 2023-07-29T00:14:42.868 DEBUG:boto:Params: {} 2023-07-29T00:14:42.868 DEBUG:boto:Token: None 2023-07-29T00:14:42.869 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:42 GMT /hiiqos-54/ 2023-07-29T00:14:42.869 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:p2xUYDIYoRCIFn5nBqae9D0avjg= 2023-07-29T00:14:42.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:42 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:p2xUYDIYoRCIFn5nBqae9D0avjg=', 'Content-Length': '0'} 2023-07-29T00:14:42.872 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9c392cce6aab5b4-0064c459f2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:42 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:42.873 DEBUG:boto:b'hiiqos-54d1000falsed2023-07-29T00:14:21.494Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:42.873 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-54 zones={a2, a1} 2023-07-29T00:14:42.874 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:14:42.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:14:42.875 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 1 --rgw-sync-log-trim-min-cold-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:14:42.920 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:14:42.921 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:14:42.971 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'hiiqos-54', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:14:42.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'hiiqos-54', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:14:42.972 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket hiiqos-54 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:14:43.015 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:14:43.016 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:14:43.056 INFO:teuthology.orchestra.run.smithi101.stdout:[] 2023-07-29T00:14:43.058 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'hiiqos-53', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:14:43.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'hiiqos-53', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:14:43.059 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket hiiqos-53 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:14:43.102 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:14:43.103 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:14:43.143 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T00:14:43.143 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:14:43.144 INFO:teuthology.orchestra.run.smithi101.stdout: "op_id": "1#00000000001.36.6", 2023-07-29T00:14:43.144 INFO:teuthology.orchestra.run.smithi101.stdout: "op_tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.13686121767008685312", 2023-07-29T00:14:43.144 INFO:teuthology.orchestra.run.smithi101.stdout: "op": "write", 2023-07-29T00:14:43.144 INFO:teuthology.orchestra.run.smithi101.stdout: "object": "a", 2023-07-29T00:14:43.144 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:14:43.144 INFO:teuthology.orchestra.run.smithi101.stdout: "state": "complete", 2023-07-29T00:14:43.145 INFO:teuthology.orchestra.run.smithi101.stdout: "index_ver": 1, 2023-07-29T00:14:43.145 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:14:09.724437021Z", 2023-07-29T00:14:43.145 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:14:43.145 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:14:43.145 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 7 2023-07-29T00:14:43.145 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:43.146 INFO:teuthology.orchestra.run.smithi101.stdout: "bilog_flags": 0, 2023-07-29T00:14:43.146 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned": false, 2023-07-29T00:14:43.146 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "", 2023-07-29T00:14:43.146 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "", 2023-07-29T00:14:43.146 INFO:teuthology.orchestra.run.smithi101.stdout: "zones_trace": [ 2023-07-29T00:14:43.147 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:14:43.147 INFO:teuthology.orchestra.run.smithi101.stdout: "entry": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-53:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.18" 2023-07-29T00:14:43.147 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:14:43.147 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:14:43.147 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:43.147 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:14:43.148 INFO:teuthology.orchestra.run.smithi101.stdout: "op_id": "5#00000000001.29.6", 2023-07-29T00:14:43.148 INFO:teuthology.orchestra.run.smithi101.stdout: "op_tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.2904228732086858581", 2023-07-29T00:14:43.148 INFO:teuthology.orchestra.run.smithi101.stdout: "op": "write", 2023-07-29T00:14:43.148 INFO:teuthology.orchestra.run.smithi101.stdout: "object": "d", 2023-07-29T00:14:43.148 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:14:43.148 INFO:teuthology.orchestra.run.smithi101.stdout: "state": "complete", 2023-07-29T00:14:43.149 INFO:teuthology.orchestra.run.smithi101.stdout: "index_ver": 1, 2023-07-29T00:14:43.149 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:14:09.821385886Z", 2023-07-29T00:14:43.149 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:14:43.149 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:14:43.149 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 1 2023-07-29T00:14:43.149 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:43.150 INFO:teuthology.orchestra.run.smithi101.stdout: "bilog_flags": 0, 2023-07-29T00:14:43.150 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned": false, 2023-07-29T00:14:43.150 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "", 2023-07-29T00:14:43.150 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "", 2023-07-29T00:14:43.150 INFO:teuthology.orchestra.run.smithi101.stdout: "zones_trace": [ 2023-07-29T00:14:43.150 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:14:43.151 INFO:teuthology.orchestra.run.smithi101.stdout: "entry": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-53:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.18" 2023-07-29T00:14:43.151 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:14:43.151 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:14:43.151 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:43.151 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:14:43.151 INFO:teuthology.orchestra.run.smithi101.stdout: "op_id": "8#00000000001.29.6", 2023-07-29T00:14:43.152 INFO:teuthology.orchestra.run.smithi101.stdout: "op_tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.16867641461365097928", 2023-07-29T00:14:43.152 INFO:teuthology.orchestra.run.smithi101.stdout: "op": "write", 2023-07-29T00:14:43.152 INFO:teuthology.orchestra.run.smithi101.stdout: "object": "c", 2023-07-29T00:14:43.152 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:14:43.153 INFO:teuthology.orchestra.run.smithi101.stdout: "state": "complete", 2023-07-29T00:14:43.153 INFO:teuthology.orchestra.run.smithi101.stdout: "index_ver": 1, 2023-07-29T00:14:43.153 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:14:09.763038824Z", 2023-07-29T00:14:43.154 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:14:43.154 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:14:43.154 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 5 2023-07-29T00:14:43.155 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:43.155 INFO:teuthology.orchestra.run.smithi101.stdout: "bilog_flags": 0, 2023-07-29T00:14:43.155 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned": false, 2023-07-29T00:14:43.156 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "", 2023-07-29T00:14:43.156 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "", 2023-07-29T00:14:43.157 INFO:teuthology.orchestra.run.smithi101.stdout: "zones_trace": [ 2023-07-29T00:14:43.157 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:14:43.157 INFO:teuthology.orchestra.run.smithi101.stdout: "entry": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-53:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.18" 2023-07-29T00:14:43.158 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:14:43.158 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:14:43.158 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:43.158 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:14:43.159 INFO:teuthology.orchestra.run.smithi101.stdout: "op_id": "10#00000000001.89.6", 2023-07-29T00:14:43.159 INFO:teuthology.orchestra.run.smithi101.stdout: "op_tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.6767028125298624342", 2023-07-29T00:14:43.159 INFO:teuthology.orchestra.run.smithi101.stdout: "op": "write", 2023-07-29T00:14:43.159 INFO:teuthology.orchestra.run.smithi101.stdout: "object": "b", 2023-07-29T00:14:43.159 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:14:43.159 INFO:teuthology.orchestra.run.smithi101.stdout: "state": "complete", 2023-07-29T00:14:43.160 INFO:teuthology.orchestra.run.smithi101.stdout: "index_ver": 1, 2023-07-29T00:14:43.160 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:14:09.744670133Z", 2023-07-29T00:14:43.160 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:14:43.160 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:14:43.160 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 34 2023-07-29T00:14:43.160 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:43.161 INFO:teuthology.orchestra.run.smithi101.stdout: "bilog_flags": 0, 2023-07-29T00:14:43.161 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned": false, 2023-07-29T00:14:43.161 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "", 2023-07-29T00:14:43.161 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "", 2023-07-29T00:14:43.161 INFO:teuthology.orchestra.run.smithi101.stdout: "zones_trace": [ 2023-07-29T00:14:43.161 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:14:43.162 INFO:teuthology.orchestra.run.smithi101.stdout: "entry": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-53:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.18" 2023-07-29T00:14:43.162 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:14:43.162 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:14:43.162 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:14:43.162 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T00:14:43.163 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:14:43.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:14:43.164 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 999 --rgw-sync-log-trim-min-cold-buckets 999 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:14:43.210 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:14:43.210 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:14:43.251 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:14:43.211+0000 7f9135023640 -1 req 10506178200544863347 0.001999952s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-36:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.1,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-36:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.1,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T00:14:43.252 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:14:43.211+0000 7f913181c640 -1 req 15374149362609549887 0.001999952s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-19:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.19,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-19:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.19,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T00:14:43.252 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:14:43.211+0000 7f913081a640 -1 req 14276754990697261006 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-23:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.23,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-23:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.23,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T00:14:43.252 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:14:43.212+0000 7f911b7f0640 -1 req 15558792991327625126 0.001999951s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-6:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.6,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-6:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.6,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T00:14:43.256 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:14:43.216+0000 7f91137e0640 -1 req 14762474635273624071 0.002999927s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-5:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.5,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-5:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.5,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T00:14:43.299 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:14:43.259+0000 7f91007ba640 -1 req 13342205687668451185 0.003999903s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-51:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.16,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-51:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.16,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T00:14:43.342 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:14:43.302+0000 7f90edf95640 -1 req 3825749476286562763 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-42:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.7,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-42:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.7,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T00:14:43.348 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:14:43.308+0000 7f90def77640 -1 req 9928937820482236588 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-29:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.29,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-29:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.29,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T00:14:43.351 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:14:43.311+0000 7f90df778640 -1 req 115837166587138224 0.001999951s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-38:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.3,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-38:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.3,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T00:14:43.392 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:14:43.352+0000 7f90da76e640 -1 req 17501416642001785403 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-1:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.1,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-1:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.1,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T00:14:43.394 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:14:43.354+0000 7f91b7928640 -1 req 16177268343640997925 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-41:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.6,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-41:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.6,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T00:14:43.480 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:14:43.440+0000 7f91b7127640 -1 req 3391889588222824108 0.001999951s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-4:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.4,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-4:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.4,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T00:14:43.485 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:14:43.445+0000 7f918c0d1640 -1 req 7821156472719079854 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-25:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.25,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-25:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.25,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T00:14:43.521 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:14:43.481+0000 7f91c7948640 -1 req 6726738328127306704 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-3:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.3,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-3:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.3,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T00:14:43.524 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:14:43.484+0000 7f91980e9640 -1 req 11804301174991713438 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-43:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.8,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-43:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.8,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T00:14:43.564 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:14:43.524+0000 7f91910db640 -1 req 13134101385305084610 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-39:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.4,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-39:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.4,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T00:14:43.575 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:14:43.535+0000 7f915e075640 -1 req 17803901899199309 0.001999952s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-24:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.24,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-24:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.24,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T00:14:43.575 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:14:43.535+0000 7f9167087640 -1 req 13130905170185664012 0.001999952s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-27:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.27,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-27:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.27,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T00:14:43.618 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:14:43.578+0000 7f9144041640 -1 req 3156048081959293554 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-26:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.26,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-26:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.26,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T00:14:43.652 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:14:43.612+0000 7f913b02f640 -1 req 3256644701973088555 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-22:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.22,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-22:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.22,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T00:14:43.662 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:14:43.622+0000 7f9146045640 -1 req 12097681747531231470 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-40:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.5,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-40:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.5,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T00:14:43.706 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:14:43.665+0000 7f9134021640 -1 req 1376660257203556571 0.001999952s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-20:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.20,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-20:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.20,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T00:14:43.706 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:14:43.665+0000 7f9140039640 -1 req 8924567127280830651 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-30:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.30,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-30:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.30,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T00:14:43.740 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:14:43.700+0000 7f911d7f4640 -1 req 5737155578598720199 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-21:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.21,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-21:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.21,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T00:14:43.749 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:14:43.709+0000 7f910c7d2640 -1 req 16789572896352297746 0.001999952s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-2:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.2,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-2:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.2,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T00:14:43.751 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:14:43.711+0000 7f9115fe5640 -1 req 3422579826438594612 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-37:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.2,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-37:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.2,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T00:14:43.783 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:14:43.743+0000 7f90f57a4640 -1 req 1242324635530674354 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-28:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.28,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-28:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.28,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T00:14:43.853 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'hiiqos-53', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:14:43.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'hiiqos-53', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:14:43.854 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket hiiqos-53 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:14:43.898 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:14:43.898 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:14:43.938 INFO:teuthology.orchestra.run.smithi101.stdout:[] 2023-07-29T00:14:43.940 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_index_log_trim ... ok 2023-07-29T00:14:43.941 INFO:rgw_multi.tests:create bucket zone=a1 name=hiiqos-55 2023-07-29T00:14:43.942 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:14:43.942 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:14:43.942 DEBUG:boto:Method: PUT 2023-07-29T00:14:43.943 DEBUG:boto:Path: /hiiqos-55/ 2023-07-29T00:14:43.943 DEBUG:boto:Data: 2023-07-29T00:14:43.943 DEBUG:boto:Headers: {} 2023-07-29T00:14:43.944 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:43.944 DEBUG:boto:Port: 8000 2023-07-29T00:14:43.944 DEBUG:boto:Params: {} 2023-07-29T00:14:43.945 DEBUG:boto:Token: None 2023-07-29T00:14:43.945 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 00:14:43 GMT /hiiqos-55/ 2023-07-29T00:14:43.945 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3VTK0x/Na+wntAhkVteewKxndjk= 2023-07-29T00:14:43.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:43 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3VTK0x/Na+wntAhkVteewKxndjk=', 'Content-Length': '0'} 2023-07-29T00:14:43.959 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f09f13a3750c20b6-0064c459f3-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:43.960 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:14:43.960 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:14:43.961 DEBUG:boto:Method: HEAD 2023-07-29T00:14:43.961 DEBUG:boto:Path: /hiiqos-55/ 2023-07-29T00:14:43.961 DEBUG:boto:Data: 2023-07-29T00:14:43.962 DEBUG:boto:Headers: {} 2023-07-29T00:14:43.962 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:43.962 DEBUG:boto:Port: 8000 2023-07-29T00:14:43.963 DEBUG:boto:Params: {} 2023-07-29T00:14:43.963 DEBUG:boto:Token: None 2023-07-29T00:14:43.963 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:43 GMT /hiiqos-55/ 2023-07-29T00:14:43.964 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+kqoyPJALQOa5zDX6Bm/Q7wP5oo= 2023-07-29T00:14:43.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:43 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+kqoyPJALQOa5zDX6Bm/Q7wP5oo=', 'Content-Length': '0'} 2023-07-29T00:14:43.966 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000730d9cf93946d81-0064c459f3-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:43.967 DEBUG:boto:path=/hiiqos-55/a 2023-07-29T00:14:43.967 DEBUG:boto:auth_path=/hiiqos-55/a 2023-07-29T00:14:43.967 DEBUG:boto:Method: PUT 2023-07-29T00:14:43.968 DEBUG:boto:Path: /hiiqos-55/a 2023-07-29T00:14:43.968 DEBUG:boto:Data: 2023-07-29T00:14:43.969 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:14:43.969 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:43.969 DEBUG:boto:Port: 8000 2023-07-29T00:14:43.970 DEBUG:boto:Params: {} 2023-07-29T00:14:43.970 DEBUG:boto:Token: None 2023-07-29T00:14:43.970 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:14:43 GMT /hiiqos-55/a 2023-07-29T00:14:43.971 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pewjfxGD8GkfXmgP7eBxaAKEbxM= 2023-07-29T00:14:43.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:14:43 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pewjfxGD8GkfXmgP7eBxaAKEbxM='} 2023-07-29T00:14:44.016 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006f8abd58bd00bfbc-0064c459f3-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:44.017 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:14:44.017 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:14:44.018 DEBUG:boto:Method: HEAD 2023-07-29T00:14:44.018 DEBUG:boto:Path: /hiiqos-55/ 2023-07-29T00:14:44.018 DEBUG:boto:Data: 2023-07-29T00:14:44.019 DEBUG:boto:Headers: {} 2023-07-29T00:14:44.019 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:44.019 DEBUG:boto:Port: 8000 2023-07-29T00:14:44.020 DEBUG:boto:Params: {} 2023-07-29T00:14:44.020 DEBUG:boto:Token: None 2023-07-29T00:14:44.021 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:44 GMT /hiiqos-55/ 2023-07-29T00:14:44.021 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:N4CsLT7ZXMxE5bLtKrnMycmgYoE= 2023-07-29T00:14:44.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:44 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:N4CsLT7ZXMxE5bLtKrnMycmgYoE=', 'Content-Length': '0'} 2023-07-29T00:14:44.023 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a20eef557ccca2a2-0064c459f3-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:44.024 DEBUG:boto:path=/hiiqos-55/b 2023-07-29T00:14:44.024 DEBUG:boto:auth_path=/hiiqos-55/b 2023-07-29T00:14:44.025 DEBUG:boto:Method: PUT 2023-07-29T00:14:44.025 DEBUG:boto:Path: /hiiqos-55/b 2023-07-29T00:14:44.025 DEBUG:boto:Data: 2023-07-29T00:14:44.026 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:14:44.026 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:44.026 DEBUG:boto:Port: 8000 2023-07-29T00:14:44.027 DEBUG:boto:Params: {} 2023-07-29T00:14:44.027 DEBUG:boto:Token: None 2023-07-29T00:14:44.027 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:14:44 GMT /hiiqos-55/b 2023-07-29T00:14:44.028 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nYTRVtMAtJl6FmjI0Zq+riLs8Es= 2023-07-29T00:14:44.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:14:44 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nYTRVtMAtJl6FmjI0Zq+riLs8Es='} 2023-07-29T00:14:44.033 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ef207159b96557c9-0064c459f3-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:44.033 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:14:44.034 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:14:44.034 DEBUG:boto:Method: HEAD 2023-07-29T00:14:44.034 DEBUG:boto:Path: /hiiqos-55/ 2023-07-29T00:14:44.035 DEBUG:boto:Data: 2023-07-29T00:14:44.035 DEBUG:boto:Headers: {} 2023-07-29T00:14:44.035 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:44.036 DEBUG:boto:Port: 8000 2023-07-29T00:14:44.036 DEBUG:boto:Params: {} 2023-07-29T00:14:44.036 DEBUG:boto:Token: None 2023-07-29T00:14:44.037 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:44 GMT /hiiqos-55/ 2023-07-29T00:14:44.037 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:N4CsLT7ZXMxE5bLtKrnMycmgYoE= 2023-07-29T00:14:44.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:44 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:N4CsLT7ZXMxE5bLtKrnMycmgYoE=', 'Content-Length': '0'} 2023-07-29T00:14:44.040 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000872e0d52242325e7-0064c459f3-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:44.040 DEBUG:boto:path=/hiiqos-55/c 2023-07-29T00:14:44.041 DEBUG:boto:auth_path=/hiiqos-55/c 2023-07-29T00:14:44.041 DEBUG:boto:Method: PUT 2023-07-29T00:14:44.041 DEBUG:boto:Path: /hiiqos-55/c 2023-07-29T00:14:44.042 DEBUG:boto:Data: 2023-07-29T00:14:44.042 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:14:44.043 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:44.043 DEBUG:boto:Port: 8000 2023-07-29T00:14:44.043 DEBUG:boto:Params: {} 2023-07-29T00:14:44.044 DEBUG:boto:Token: None 2023-07-29T00:14:44.045 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:14:44 GMT /hiiqos-55/c 2023-07-29T00:14:44.045 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TFZ3JkxiZwzLgUJxlrKnDcJ3akA= 2023-07-29T00:14:44.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:14:44 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TFZ3JkxiZwzLgUJxlrKnDcJ3akA='} 2023-07-29T00:14:44.051 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d5ebee5d112fe9cc-0064c459f4-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:44.051 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:14:44.052 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:14:44.052 DEBUG:boto:Method: HEAD 2023-07-29T00:14:44.052 DEBUG:boto:Path: /hiiqos-55/ 2023-07-29T00:14:44.053 DEBUG:boto:Data: 2023-07-29T00:14:44.053 DEBUG:boto:Headers: {} 2023-07-29T00:14:44.053 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:44.054 DEBUG:boto:Port: 8000 2023-07-29T00:14:44.054 DEBUG:boto:Params: {} 2023-07-29T00:14:44.055 DEBUG:boto:Token: None 2023-07-29T00:14:44.055 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:44 GMT /hiiqos-55/ 2023-07-29T00:14:44.055 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:N4CsLT7ZXMxE5bLtKrnMycmgYoE= 2023-07-29T00:14:44.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:44 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:N4CsLT7ZXMxE5bLtKrnMycmgYoE=', 'Content-Length': '0'} 2023-07-29T00:14:44.058 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000966df5e2409e60d1-0064c459f4-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:44.059 DEBUG:boto:path=/hiiqos-55/d 2023-07-29T00:14:44.059 DEBUG:boto:auth_path=/hiiqos-55/d 2023-07-29T00:14:44.059 DEBUG:boto:Method: PUT 2023-07-29T00:14:44.060 DEBUG:boto:Path: /hiiqos-55/d 2023-07-29T00:14:44.060 DEBUG:boto:Data: 2023-07-29T00:14:44.061 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:14:44.061 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:44.062 DEBUG:boto:Port: 8000 2023-07-29T00:14:44.062 DEBUG:boto:Params: {} 2023-07-29T00:14:44.062 DEBUG:boto:Token: None 2023-07-29T00:14:44.063 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:14:44 GMT /hiiqos-55/d 2023-07-29T00:14:44.063 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4zJtmnnv8IpKzWYwPLr8krmyvGo= 2023-07-29T00:14:44.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:14:44 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4zJtmnnv8IpKzWYwPLr8krmyvGo='} 2023-07-29T00:14:44.108 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000652345e248f6894f-0064c459f4-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:44.109 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:14:44.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:14:44.110 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:14:44.157 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:14:44.157 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:14:44.186 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T00:14:44.198 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T00:14:44.199 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:14:44.199 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:14:44.199 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:46:09.233742Z" 2023-07-29T00:14:44.199 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:44.199 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:14:44.199 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690589683.919041_829.1", 2023-07-29T00:14:44.199 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:14:43.919041Z" 2023-07-29T00:14:44.200 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:44.200 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:14:44.200 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:14:44.200 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T00:14:44.200 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:44.200 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:14:44.201 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:14:44.201 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T00:14:44.201 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:14:44.201 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T00:14:44.202 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690587969.233742_99.1', 1: '1_1690589683.919041_829.1', 2: '', 3: ''} 2023-07-29T00:14:44.202 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T00:14:44.203 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:14:44.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:14:44.203 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:14:44.249 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:14:44.249 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:14:44.278 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:14:44.238+0000 7f70cbe0f880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:14:44.279 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:14:44.239+0000 7f70cbe0f880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:14:44.292 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:14:44.292 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:14:44.292 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:14:44.292 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:14:44.293 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:14:44.293 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:14:44.293 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:14:44.293 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:44.293 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:14:44.293 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:14:44.293 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:14:44.294 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:14:44.294 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:14:44.294 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:14:44.294 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:14:44.294 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:14:44.294 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:14:44.294 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:14:44.295 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:14:44.295 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:14:44.295 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:44.295 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:14:44.295 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:14:44.295 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:14:44.295 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:14:44.296 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690589683.913457_812.1", 2023-07-29T00:14:44.296 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:14:44.296 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:14:44.296 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:14:44.297 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:14:43.913457Z", 2023-07-29T00:14:44.297 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:14:44.297 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:14:44.297 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:44.297 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:14:44.297 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:14:44.298 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:14:44.298 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:14:44.298 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:14:44.298 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:14:44.298 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:14:44.298 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:14:44.299 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:14:44.299 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:14:44.299 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:14:44.299 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:44.299 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:14:44.300 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:14:44.300 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:14:44.300 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:14:44.300 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:14:44.300 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:14:44.300 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:14:44.301 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:14:44.301 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:14:44.301 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:14:44.301 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:14:44.301 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:14:44.301 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:14:44.301 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:44.302 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:14:44.302 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:14:44.302 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:14:44.302 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:44.302 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:14:44Z" 2023-07-29T00:14:44.302 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:14:44.303 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690589683.913457_812.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:14:43.913457Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:14:44Z" } 2023-07-29T00:14:44.304 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690589683.913457_812.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:14:43.913457Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:14:44.304 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690589683.919041_829.1', 2: '', 3: ''} 2023-07-29T00:14:44.305 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690589683.913457_812.1', 2: '', 3: ''} 2023-07-29T00:14:44.305 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690589683.919041_829.1 target=1_1690589683.913457_812.1 2023-07-29T00:14:49.306 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:14:49.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:14:49.307 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:14:49.360 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:14:49.360 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:14:49.392 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:14:49.352+0000 7f7a4f211880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:14:49.393 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:14:49.353+0000 7f7a4f211880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:14:49.406 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:14:49.407 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:14:49.407 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:14:49.407 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:14:49.407 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:14:49.407 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:14:49.407 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:14:49.408 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:49.408 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:14:49.408 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:14:49.408 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:14:49.408 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:14:49.408 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:14:49.409 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:14:49.409 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:14:49.410 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:14:49.410 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:14:49.410 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:14:49.410 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:14:49.410 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:14:49.411 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:49.411 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:14:49.411 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:14:49.411 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:14:49.411 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:14:49.411 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690589683.919041_829.1", 2023-07-29T00:14:49.411 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:14:49.412 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:14:49.412 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:14:49.412 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:14:43.919041Z", 2023-07-29T00:14:49.412 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:14:49.412 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:14:49.412 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:49.413 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:14:49.413 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:14:49.413 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:14:49.413 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:14:49.413 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:14:49.413 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:14:49.413 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:14:49.414 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:14:49.414 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:14:49.414 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:14:49.414 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:14:49.414 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:49.414 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:14:49.414 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:14:49.415 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:14:49.415 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:14:49.415 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:14:49.415 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:14:49.415 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:14:49.415 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:14:49.415 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:14:49.416 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:14:49.416 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:14:49.416 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:14:49.416 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:14:49.416 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:49.416 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:14:49.417 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:14:49.417 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:14:49.417 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:14:49.417 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:14:49Z" 2023-07-29T00:14:49.417 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:14:49.418 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690589683.919041_829.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:14:43.919041Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:14:49Z" } 2023-07-29T00:14:49.419 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690589683.919041_829.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:14:43.919041Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:14:49.419 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690589683.919041_829.1', 2: '', 3: ''} 2023-07-29T00:14:49.419 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690589683.919041_829.1', 2: '', 3: ''} 2023-07-29T00:14:49.420 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T00:14:49.420 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-55 2023-07-29T00:14:49.420 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:14:49.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:14:49.421 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:14:49.475 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:14:49.475 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:14:49.509 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:14:49.468+0000 7fa471416880 1 waiting to reach incremental sync.. 2023-07-29T00:14:54.510 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:14:54.469+0000 7fa471416880 1 waiting to reach incremental sync.. 2023-07-29T00:14:59.512 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:14:59.471+0000 7fa471416880 1 bucket sync caught up with source: 2023-07-29T00:14:59.512 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000001.56.6, , , , 00000000001.58.6, , , 00000000001.38.6, , 00000000001.23.6] 2023-07-29T00:14:59.512 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000001.56.6, , , , 00000000001.58.6, , , 00000000001.38.6, , 00000000001.23.6] 2023-07-29T00:14:59.512 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:14:59.471+0000 7fa471416880 0 bucket checkpoint complete 2023-07-29T00:14:59.529 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-55 2023-07-29T00:14:59.530 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:14:59.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:14:59.532 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:14:59.589 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:14:59.589 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:14:59.625 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:14:59.584+0000 7f0a33610880 1 bucket sync caught up with source: 2023-07-29T00:14:59.625 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000001.708.6, , , , 00000000001.707.6, , , 00000000001.382.6, , 00000000001.1165.6] 2023-07-29T00:14:59.625 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000001.708.6, , , , 00000000001.707.6, , , 00000000001.382.6, , 00000000001.1165.6] 2023-07-29T00:14:59.625 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:14:59.584+0000 7f0a33610880 0 bucket checkpoint complete 2023-07-29T00:14:59.639 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-55 zones={a2, a1} 2023-07-29T00:14:59.639 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:14:59.639 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:14:59.640 DEBUG:boto:Method: HEAD 2023-07-29T00:14:59.640 DEBUG:boto:Path: /hiiqos-55/ 2023-07-29T00:14:59.641 DEBUG:boto:Data: 2023-07-29T00:14:59.641 DEBUG:boto:Headers: {} 2023-07-29T00:14:59.641 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:59.642 DEBUG:boto:Port: 8001 2023-07-29T00:14:59.642 DEBUG:boto:Params: {} 2023-07-29T00:14:59.642 DEBUG:boto:Token: None 2023-07-29T00:14:59.643 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:59 GMT /hiiqos-55/ 2023-07-29T00:14:59.643 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1SfTQCuRQJkvt6bkf40LlGQrl50= 2023-07-29T00:14:59.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1SfTQCuRQJkvt6bkf40LlGQrl50=', 'Content-Length': '0'} 2023-07-29T00:14:59.646 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000084d72077fbcf1e7d-0064c45a03-1663-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:59.647 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:14:59.647 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:14:59.648 DEBUG:boto:Method: HEAD 2023-07-29T00:14:59.648 DEBUG:boto:Path: /hiiqos-55/ 2023-07-29T00:14:59.648 DEBUG:boto:Data: 2023-07-29T00:14:59.649 DEBUG:boto:Headers: {} 2023-07-29T00:14:59.649 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:59.649 DEBUG:boto:Port: 8000 2023-07-29T00:14:59.650 DEBUG:boto:Params: {} 2023-07-29T00:14:59.650 DEBUG:boto:Token: None 2023-07-29T00:14:59.651 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:59 GMT /hiiqos-55/ 2023-07-29T00:14:59.651 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1SfTQCuRQJkvt6bkf40LlGQrl50= 2023-07-29T00:14:59.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1SfTQCuRQJkvt6bkf40LlGQrl50=', 'Content-Length': '0'} 2023-07-29T00:14:59.654 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001d804cbaae5a3ef7-0064c45a03-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:59.654 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-29T00:14:59.655 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:14:59.655 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:14:59.656 DEBUG:boto:path=/hiiqos-55/?versions&max-keys=999 2023-07-29T00:14:59.656 DEBUG:boto:auth_path=/hiiqos-55/?versions&max-keys=999 2023-07-29T00:14:59.656 DEBUG:boto:Method: GET 2023-07-29T00:14:59.657 DEBUG:boto:Path: /hiiqos-55/?versions&max-keys=999 2023-07-29T00:14:59.657 DEBUG:boto:Data: 2023-07-29T00:14:59.657 DEBUG:boto:Headers: {} 2023-07-29T00:14:59.658 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:59.658 DEBUG:boto:Port: 8001 2023-07-29T00:14:59.658 DEBUG:boto:Params: {} 2023-07-29T00:14:59.659 DEBUG:boto:Token: None 2023-07-29T00:14:59.659 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:59 GMT /hiiqos-55/?versions 2023-07-29T00:14:59.660 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bU+pz49rai6Gr+8IjSkxokZURt0= 2023-07-29T00:14:59.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bU+pz49rai6Gr+8IjSkxokZURt0=', 'Content-Length': '0'} 2023-07-29T00:14:59.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a385bce7345584b1-0064c45a03-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:59.663 DEBUG:boto:b'hiiqos-55999falseanulltrue2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:59.663 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-07-29T00:14:59.664 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-07-29T00:14:59.664 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-07-29T00:14:59.665 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-07-29T00:14:59.665 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-29T00:14:59.665 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:14:59.666 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:14:59.666 DEBUG:boto:path=/hiiqos-55/?versions&max-keys=999 2023-07-29T00:14:59.666 DEBUG:boto:auth_path=/hiiqos-55/?versions&max-keys=999 2023-07-29T00:14:59.667 DEBUG:boto:Method: GET 2023-07-29T00:14:59.667 DEBUG:boto:Path: /hiiqos-55/?versions&max-keys=999 2023-07-29T00:14:59.668 DEBUG:boto:Data: 2023-07-29T00:14:59.668 DEBUG:boto:Headers: {} 2023-07-29T00:14:59.668 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:59.669 DEBUG:boto:Port: 8000 2023-07-29T00:14:59.669 DEBUG:boto:Params: {} 2023-07-29T00:14:59.669 DEBUG:boto:Token: None 2023-07-29T00:14:59.670 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:59 GMT /hiiqos-55/?versions 2023-07-29T00:14:59.670 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bU+pz49rai6Gr+8IjSkxokZURt0= 2023-07-29T00:14:59.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bU+pz49rai6Gr+8IjSkxokZURt0=', 'Content-Length': '0'} 2023-07-29T00:14:59.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086a14adb73f04b96-0064c45a03-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:59.673 DEBUG:boto:b'hiiqos-55999falseanulltrue2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:59.674 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-07-29T00:14:59.674 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-07-29T00:14:59.675 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-07-29T00:14:59.675 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-07-29T00:14:59.675 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:14:59.676 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:14:59.676 DEBUG:boto:path=/hiiqos-55/?versions&max-keys=999 2023-07-29T00:14:59.676 DEBUG:boto:auth_path=/hiiqos-55/?versions&max-keys=999 2023-07-29T00:14:59.677 DEBUG:boto:Method: GET 2023-07-29T00:14:59.677 DEBUG:boto:Path: /hiiqos-55/?versions&max-keys=999 2023-07-29T00:14:59.678 DEBUG:boto:Data: 2023-07-29T00:14:59.678 DEBUG:boto:Headers: {} 2023-07-29T00:14:59.678 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:59.679 DEBUG:boto:Port: 8001 2023-07-29T00:14:59.679 DEBUG:boto:Params: {} 2023-07-29T00:14:59.679 DEBUG:boto:Token: None 2023-07-29T00:14:59.680 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:59 GMT /hiiqos-55/?versions 2023-07-29T00:14:59.680 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bU+pz49rai6Gr+8IjSkxokZURt0= 2023-07-29T00:14:59.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bU+pz49rai6Gr+8IjSkxokZURt0=', 'Content-Length': '0'} 2023-07-29T00:14:59.683 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae92abfe846bb352-0064c45a03-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:59.683 DEBUG:boto:b'hiiqos-55999falseanulltrue2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:59.684 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:14:59.684 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:14:59.685 DEBUG:boto:path=/hiiqos-55/?versions&max-keys=999 2023-07-29T00:14:59.685 DEBUG:boto:auth_path=/hiiqos-55/?versions&max-keys=999 2023-07-29T00:14:59.685 DEBUG:boto:Method: GET 2023-07-29T00:14:59.686 DEBUG:boto:Path: /hiiqos-55/?versions&max-keys=999 2023-07-29T00:14:59.686 DEBUG:boto:Data: 2023-07-29T00:14:59.686 DEBUG:boto:Headers: {} 2023-07-29T00:14:59.687 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:59.687 DEBUG:boto:Port: 8000 2023-07-29T00:14:59.687 DEBUG:boto:Params: {} 2023-07-29T00:14:59.688 DEBUG:boto:Token: None 2023-07-29T00:14:59.688 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:59 GMT /hiiqos-55/?versions 2023-07-29T00:14:59.689 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bU+pz49rai6Gr+8IjSkxokZURt0= 2023-07-29T00:14:59.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bU+pz49rai6Gr+8IjSkxokZURt0=', 'Content-Length': '0'} 2023-07-29T00:14:59.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8b9f0918bddcfd2-0064c45a03-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:59.692 DEBUG:boto:b'hiiqos-55999falseanulltrue2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:59.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-07-29T00:14:59.693 DEBUG:boto:path=/hiiqos-55/a 2023-07-29T00:14:59.693 DEBUG:boto:auth_path=/hiiqos-55/a 2023-07-29T00:14:59.694 DEBUG:boto:path=/hiiqos-55/a?versionId=null 2023-07-29T00:14:59.694 DEBUG:boto:auth_path=/hiiqos-55/a?versionId=null 2023-07-29T00:14:59.694 DEBUG:boto:Method: GET 2023-07-29T00:14:59.695 DEBUG:boto:Path: /hiiqos-55/a?versionId=null 2023-07-29T00:14:59.695 DEBUG:boto:Data: 2023-07-29T00:14:59.696 DEBUG:boto:Headers: {} 2023-07-29T00:14:59.696 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:59.696 DEBUG:boto:Port: 8001 2023-07-29T00:14:59.697 DEBUG:boto:Params: {} 2023-07-29T00:14:59.697 DEBUG:boto:Token: None 2023-07-29T00:14:59.698 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:59 GMT /hiiqos-55/a?versionId=null 2023-07-29T00:14:59.698 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EYPfq7fKj53U2CLFCjFVFRG/mn4= 2023-07-29T00:14:59.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EYPfq7fKj53U2CLFCjFVFRG/mn4=', 'Content-Length': '0'} 2023-07-29T00:14:59.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7cc8bde0cb67603-0064c45a03-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:59.701 DEBUG:boto:path=/hiiqos-55/a 2023-07-29T00:14:59.701 DEBUG:boto:auth_path=/hiiqos-55/a 2023-07-29T00:14:59.701 DEBUG:boto:path=/hiiqos-55/a?versionId=null 2023-07-29T00:14:59.702 DEBUG:boto:auth_path=/hiiqos-55/a?versionId=null 2023-07-29T00:14:59.702 DEBUG:boto:Method: GET 2023-07-29T00:14:59.703 DEBUG:boto:Path: /hiiqos-55/a?versionId=null 2023-07-29T00:14:59.703 DEBUG:boto:Data: 2023-07-29T00:14:59.703 DEBUG:boto:Headers: {} 2023-07-29T00:14:59.704 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:59.704 DEBUG:boto:Port: 8000 2023-07-29T00:14:59.704 DEBUG:boto:Params: {} 2023-07-29T00:14:59.705 DEBUG:boto:Token: None 2023-07-29T00:14:59.705 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:59 GMT /hiiqos-55/a?versionId=null 2023-07-29T00:14:59.706 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EYPfq7fKj53U2CLFCjFVFRG/mn4= 2023-07-29T00:14:59.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EYPfq7fKj53U2CLFCjFVFRG/mn4=', 'Content-Length': '0'} 2023-07-29T00:14:59.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000127c742d076cf09e-0064c45a03-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:59.708 DEBUG:boto:path=/hiiqos-55/a 2023-07-29T00:14:59.709 DEBUG:boto:auth_path=/hiiqos-55/a 2023-07-29T00:14:59.709 DEBUG:boto:path=/hiiqos-55/a?versionId=null 2023-07-29T00:14:59.709 DEBUG:boto:auth_path=/hiiqos-55/a?versionId=null 2023-07-29T00:14:59.710 DEBUG:boto:Method: HEAD 2023-07-29T00:14:59.710 DEBUG:boto:Path: /hiiqos-55/a?versionId=null 2023-07-29T00:14:59.710 DEBUG:boto:Data: 2023-07-29T00:14:59.711 DEBUG:boto:Headers: {} 2023-07-29T00:14:59.711 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:59.711 DEBUG:boto:Port: 8001 2023-07-29T00:14:59.712 DEBUG:boto:Params: {} 2023-07-29T00:14:59.712 DEBUG:boto:Token: None 2023-07-29T00:14:59.713 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:59 GMT /hiiqos-55/a?versionId=null 2023-07-29T00:14:59.713 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rFL5sGpu4O7Gul3f3mTkl+UIxCY= 2023-07-29T00:14:59.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rFL5sGpu4O7Gul3f3mTkl+UIxCY=', 'Content-Length': '0'} 2023-07-29T00:14:59.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073ce0190e45b65f1-0064c45a03-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:59.716 DEBUG:boto:path=/hiiqos-55/a 2023-07-29T00:14:59.716 DEBUG:boto:auth_path=/hiiqos-55/a 2023-07-29T00:14:59.717 DEBUG:boto:path=/hiiqos-55/a?versionId=null 2023-07-29T00:14:59.717 DEBUG:boto:auth_path=/hiiqos-55/a?versionId=null 2023-07-29T00:14:59.717 DEBUG:boto:Method: HEAD 2023-07-29T00:14:59.718 DEBUG:boto:Path: /hiiqos-55/a?versionId=null 2023-07-29T00:14:59.718 DEBUG:boto:Data: 2023-07-29T00:14:59.718 DEBUG:boto:Headers: {} 2023-07-29T00:14:59.719 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:59.719 DEBUG:boto:Port: 8000 2023-07-29T00:14:59.719 DEBUG:boto:Params: {} 2023-07-29T00:14:59.720 DEBUG:boto:Token: None 2023-07-29T00:14:59.720 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:59 GMT /hiiqos-55/a?versionId=null 2023-07-29T00:14:59.721 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rFL5sGpu4O7Gul3f3mTkl+UIxCY= 2023-07-29T00:14:59.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rFL5sGpu4O7Gul3f3mTkl+UIxCY=', 'Content-Length': '0'} 2023-07-29T00:14:59.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099d5dca667bd64f7-0064c45a03-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:59.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-07-29T00:14:59.724 DEBUG:boto:path=/hiiqos-55/a 2023-07-29T00:14:59.724 DEBUG:boto:auth_path=/hiiqos-55/a 2023-07-29T00:14:59.725 DEBUG:boto:path=/hiiqos-55/a?versionId=null 2023-07-29T00:14:59.725 DEBUG:boto:auth_path=/hiiqos-55/a?versionId=null 2023-07-29T00:14:59.725 DEBUG:boto:Method: GET 2023-07-29T00:14:59.726 DEBUG:boto:Path: /hiiqos-55/a?versionId=null 2023-07-29T00:14:59.726 DEBUG:boto:Data: 2023-07-29T00:14:59.726 DEBUG:boto:Headers: {} 2023-07-29T00:14:59.727 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:59.727 DEBUG:boto:Port: 8001 2023-07-29T00:14:59.727 DEBUG:boto:Params: {} 2023-07-29T00:14:59.728 DEBUG:boto:Token: None 2023-07-29T00:14:59.728 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:59 GMT /hiiqos-55/a?versionId=null 2023-07-29T00:14:59.729 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EYPfq7fKj53U2CLFCjFVFRG/mn4= 2023-07-29T00:14:59.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EYPfq7fKj53U2CLFCjFVFRG/mn4=', 'Content-Length': '0'} 2023-07-29T00:14:59.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f8f6c6d63f3bef3-0064c45a03-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:59.732 DEBUG:boto:path=/hiiqos-55/a 2023-07-29T00:14:59.732 DEBUG:boto:auth_path=/hiiqos-55/a 2023-07-29T00:14:59.733 DEBUG:boto:path=/hiiqos-55/a?versionId=null 2023-07-29T00:14:59.733 DEBUG:boto:auth_path=/hiiqos-55/a?versionId=null 2023-07-29T00:14:59.733 DEBUG:boto:Method: GET 2023-07-29T00:14:59.734 DEBUG:boto:Path: /hiiqos-55/a?versionId=null 2023-07-29T00:14:59.734 DEBUG:boto:Data: 2023-07-29T00:14:59.734 DEBUG:boto:Headers: {} 2023-07-29T00:14:59.735 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:59.735 DEBUG:boto:Port: 8000 2023-07-29T00:14:59.736 DEBUG:boto:Params: {} 2023-07-29T00:14:59.736 DEBUG:boto:Token: None 2023-07-29T00:14:59.737 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:59 GMT /hiiqos-55/a?versionId=null 2023-07-29T00:14:59.737 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EYPfq7fKj53U2CLFCjFVFRG/mn4= 2023-07-29T00:14:59.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EYPfq7fKj53U2CLFCjFVFRG/mn4=', 'Content-Length': '0'} 2023-07-29T00:14:59.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e27a4a880ba3ff9-0064c45a03-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:59.740 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:14:59.740 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:14:59.741 DEBUG:boto:path=/hiiqos-55/?prefix=a 2023-07-29T00:14:59.741 DEBUG:boto:auth_path=/hiiqos-55/?prefix=a 2023-07-29T00:14:59.741 DEBUG:boto:Method: GET 2023-07-29T00:14:59.742 DEBUG:boto:Path: /hiiqos-55/?prefix=a 2023-07-29T00:14:59.742 DEBUG:boto:Data: 2023-07-29T00:14:59.742 DEBUG:boto:Headers: {} 2023-07-29T00:14:59.743 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:59.743 DEBUG:boto:Port: 8001 2023-07-29T00:14:59.743 DEBUG:boto:Params: {} 2023-07-29T00:14:59.744 DEBUG:boto:Token: None 2023-07-29T00:14:59.744 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:59 GMT /hiiqos-55/ 2023-07-29T00:14:59.745 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:t/PJ+bAhtkXlQy9TiCoPZfTxBw0= 2023-07-29T00:14:59.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t/PJ+bAhtkXlQy9TiCoPZfTxBw0=', 'Content-Length': '0'} 2023-07-29T00:14:59.747 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2f7cd5689b869dc-0064c45a03-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:59.748 DEBUG:boto:b'hiiqos-55a1000falsea2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:59.749 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:14:59.749 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:14:59.750 DEBUG:boto:path=/hiiqos-55/?prefix=a 2023-07-29T00:14:59.750 DEBUG:boto:auth_path=/hiiqos-55/?prefix=a 2023-07-29T00:14:59.751 DEBUG:boto:Method: GET 2023-07-29T00:14:59.751 DEBUG:boto:Path: /hiiqos-55/?prefix=a 2023-07-29T00:14:59.751 DEBUG:boto:Data: 2023-07-29T00:14:59.752 DEBUG:boto:Headers: {} 2023-07-29T00:14:59.752 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:59.752 DEBUG:boto:Port: 8000 2023-07-29T00:14:59.753 DEBUG:boto:Params: {} 2023-07-29T00:14:59.753 DEBUG:boto:Token: None 2023-07-29T00:14:59.754 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:59 GMT /hiiqos-55/ 2023-07-29T00:14:59.754 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:t/PJ+bAhtkXlQy9TiCoPZfTxBw0= 2023-07-29T00:14:59.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t/PJ+bAhtkXlQy9TiCoPZfTxBw0=', 'Content-Length': '0'} 2023-07-29T00:14:59.757 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b93c7218f9de916b-0064c45a03-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:59.758 DEBUG:boto:b'hiiqos-55a1000falsea2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:59.758 DEBUG:boto:path=/hiiqos-55/a 2023-07-29T00:14:59.759 DEBUG:boto:auth_path=/hiiqos-55/a 2023-07-29T00:14:59.759 DEBUG:boto:Method: HEAD 2023-07-29T00:14:59.760 DEBUG:boto:Path: /hiiqos-55/a 2023-07-29T00:14:59.760 DEBUG:boto:Data: 2023-07-29T00:14:59.760 DEBUG:boto:Headers: {} 2023-07-29T00:14:59.761 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:59.761 DEBUG:boto:Port: 8001 2023-07-29T00:14:59.761 DEBUG:boto:Params: {} 2023-07-29T00:14:59.762 DEBUG:boto:Token: None 2023-07-29T00:14:59.762 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:59 GMT /hiiqos-55/a 2023-07-29T00:14:59.763 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:NjbT39xb6ixLxCtixw8Gub03bq8= 2023-07-29T00:14:59.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:NjbT39xb6ixLxCtixw8Gub03bq8=', 'Content-Length': '0'} 2023-07-29T00:14:59.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb09aca9042a377c-0064c45a03-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:59.765 DEBUG:boto:path=/hiiqos-55/a 2023-07-29T00:14:59.766 DEBUG:boto:auth_path=/hiiqos-55/a 2023-07-29T00:14:59.766 DEBUG:boto:Method: HEAD 2023-07-29T00:14:59.766 DEBUG:boto:Path: /hiiqos-55/a 2023-07-29T00:14:59.767 DEBUG:boto:Data: 2023-07-29T00:14:59.767 DEBUG:boto:Headers: {} 2023-07-29T00:14:59.768 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:59.768 DEBUG:boto:Port: 8000 2023-07-29T00:14:59.768 DEBUG:boto:Params: {} 2023-07-29T00:14:59.769 DEBUG:boto:Token: None 2023-07-29T00:14:59.769 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:59 GMT /hiiqos-55/a 2023-07-29T00:14:59.770 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:NjbT39xb6ixLxCtixw8Gub03bq8= 2023-07-29T00:14:59.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:NjbT39xb6ixLxCtixw8Gub03bq8=', 'Content-Length': '0'} 2023-07-29T00:14:59.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b64393f344675ca-0064c45a03-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:59.772 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-07-29T00:14:59.773 DEBUG:boto:path=/hiiqos-55/a 2023-07-29T00:14:59.773 DEBUG:boto:auth_path=/hiiqos-55/a 2023-07-29T00:14:59.774 DEBUG:boto:Method: GET 2023-07-29T00:14:59.774 DEBUG:boto:Path: /hiiqos-55/a 2023-07-29T00:14:59.774 DEBUG:boto:Data: 2023-07-29T00:14:59.775 DEBUG:boto:Headers: {} 2023-07-29T00:14:59.775 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:59.775 DEBUG:boto:Port: 8001 2023-07-29T00:14:59.776 DEBUG:boto:Params: {} 2023-07-29T00:14:59.777 DEBUG:boto:Token: None 2023-07-29T00:14:59.777 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:59 GMT /hiiqos-55/a 2023-07-29T00:14:59.778 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yDoxt0Zmh2uge4Mq0f1WODESDrM= 2023-07-29T00:14:59.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yDoxt0Zmh2uge4Mq0f1WODESDrM=', 'Content-Length': '0'} 2023-07-29T00:14:59.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2d14ed4ebdadf3e-0064c45a03-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:59.781 DEBUG:boto:path=/hiiqos-55/a 2023-07-29T00:14:59.781 DEBUG:boto:auth_path=/hiiqos-55/a 2023-07-29T00:14:59.782 DEBUG:boto:Method: GET 2023-07-29T00:14:59.782 DEBUG:boto:Path: /hiiqos-55/a 2023-07-29T00:14:59.783 DEBUG:boto:Data: 2023-07-29T00:14:59.784 DEBUG:boto:Headers: {} 2023-07-29T00:14:59.784 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:59.785 DEBUG:boto:Port: 8000 2023-07-29T00:14:59.785 DEBUG:boto:Params: {} 2023-07-29T00:14:59.786 DEBUG:boto:Token: None 2023-07-29T00:14:59.786 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:59 GMT /hiiqos-55/a 2023-07-29T00:14:59.786 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yDoxt0Zmh2uge4Mq0f1WODESDrM= 2023-07-29T00:14:59.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yDoxt0Zmh2uge4Mq0f1WODESDrM=', 'Content-Length': '0'} 2023-07-29T00:14:59.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ad3c4665eb7f8c4-0064c45a03-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:59.790 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:14:59.790 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:14:59.791 DEBUG:boto:path=/hiiqos-55/?prefix=a 2023-07-29T00:14:59.791 DEBUG:boto:auth_path=/hiiqos-55/?prefix=a 2023-07-29T00:14:59.792 DEBUG:boto:Method: GET 2023-07-29T00:14:59.792 DEBUG:boto:Path: /hiiqos-55/?prefix=a 2023-07-29T00:14:59.793 DEBUG:boto:Data: 2023-07-29T00:14:59.793 DEBUG:boto:Headers: {} 2023-07-29T00:14:59.794 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:59.794 DEBUG:boto:Port: 8001 2023-07-29T00:14:59.794 DEBUG:boto:Params: {} 2023-07-29T00:14:59.795 DEBUG:boto:Token: None 2023-07-29T00:14:59.795 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:59 GMT /hiiqos-55/ 2023-07-29T00:14:59.796 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:t/PJ+bAhtkXlQy9TiCoPZfTxBw0= 2023-07-29T00:14:59.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t/PJ+bAhtkXlQy9TiCoPZfTxBw0=', 'Content-Length': '0'} 2023-07-29T00:14:59.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000599a42145668f9ee-0064c45a03-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:59.799 DEBUG:boto:b'hiiqos-55a1000falsea2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:59.799 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:14:59.800 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:14:59.800 DEBUG:boto:path=/hiiqos-55/?prefix=a 2023-07-29T00:14:59.801 DEBUG:boto:auth_path=/hiiqos-55/?prefix=a 2023-07-29T00:14:59.801 DEBUG:boto:Method: GET 2023-07-29T00:14:59.801 DEBUG:boto:Path: /hiiqos-55/?prefix=a 2023-07-29T00:14:59.802 DEBUG:boto:Data: 2023-07-29T00:14:59.802 DEBUG:boto:Headers: {} 2023-07-29T00:14:59.803 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:59.803 DEBUG:boto:Port: 8000 2023-07-29T00:14:59.804 DEBUG:boto:Params: {} 2023-07-29T00:14:59.804 DEBUG:boto:Token: None 2023-07-29T00:14:59.805 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:59 GMT /hiiqos-55/ 2023-07-29T00:14:59.805 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:t/PJ+bAhtkXlQy9TiCoPZfTxBw0= 2023-07-29T00:14:59.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t/PJ+bAhtkXlQy9TiCoPZfTxBw0=', 'Content-Length': '0'} 2023-07-29T00:14:59.807 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000539fe7f7ffee520f-0064c45a03-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:59.848 DEBUG:boto:b'hiiqos-55a1000falsea2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:59.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-07-29T00:14:59.849 DEBUG:boto:path=/hiiqos-55/b 2023-07-29T00:14:59.850 DEBUG:boto:auth_path=/hiiqos-55/b 2023-07-29T00:14:59.850 DEBUG:boto:path=/hiiqos-55/b?versionId=null 2023-07-29T00:14:59.851 DEBUG:boto:auth_path=/hiiqos-55/b?versionId=null 2023-07-29T00:14:59.851 DEBUG:boto:Method: GET 2023-07-29T00:14:59.851 DEBUG:boto:Path: /hiiqos-55/b?versionId=null 2023-07-29T00:14:59.852 DEBUG:boto:Data: 2023-07-29T00:14:59.852 DEBUG:boto:Headers: {} 2023-07-29T00:14:59.852 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:59.853 DEBUG:boto:Port: 8001 2023-07-29T00:14:59.853 DEBUG:boto:Params: {} 2023-07-29T00:14:59.854 DEBUG:boto:Token: None 2023-07-29T00:14:59.854 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:59 GMT /hiiqos-55/b?versionId=null 2023-07-29T00:14:59.854 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:K3jsoITgiOh1WpdTHXI210MlBVg= 2023-07-29T00:14:59.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K3jsoITgiOh1WpdTHXI210MlBVg=', 'Content-Length': '0'} 2023-07-29T00:14:59.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4e36fe16c02268a-0064c45a03-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:59.857 DEBUG:boto:path=/hiiqos-55/b 2023-07-29T00:14:59.858 DEBUG:boto:auth_path=/hiiqos-55/b 2023-07-29T00:14:59.858 DEBUG:boto:path=/hiiqos-55/b?versionId=null 2023-07-29T00:14:59.858 DEBUG:boto:auth_path=/hiiqos-55/b?versionId=null 2023-07-29T00:14:59.859 DEBUG:boto:Method: GET 2023-07-29T00:14:59.859 DEBUG:boto:Path: /hiiqos-55/b?versionId=null 2023-07-29T00:14:59.859 DEBUG:boto:Data: 2023-07-29T00:14:59.860 DEBUG:boto:Headers: {} 2023-07-29T00:14:59.860 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:59.860 DEBUG:boto:Port: 8000 2023-07-29T00:14:59.861 DEBUG:boto:Params: {} 2023-07-29T00:14:59.861 DEBUG:boto:Token: None 2023-07-29T00:14:59.862 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:59 GMT /hiiqos-55/b?versionId=null 2023-07-29T00:14:59.862 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:K3jsoITgiOh1WpdTHXI210MlBVg= 2023-07-29T00:14:59.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K3jsoITgiOh1WpdTHXI210MlBVg=', 'Content-Length': '0'} 2023-07-29T00:14:59.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009695d293b21300c9-0064c45a03-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:59.865 DEBUG:boto:path=/hiiqos-55/b 2023-07-29T00:14:59.865 DEBUG:boto:auth_path=/hiiqos-55/b 2023-07-29T00:14:59.866 DEBUG:boto:path=/hiiqos-55/b?versionId=null 2023-07-29T00:14:59.866 DEBUG:boto:auth_path=/hiiqos-55/b?versionId=null 2023-07-29T00:14:59.866 DEBUG:boto:Method: HEAD 2023-07-29T00:14:59.867 DEBUG:boto:Path: /hiiqos-55/b?versionId=null 2023-07-29T00:14:59.867 DEBUG:boto:Data: 2023-07-29T00:14:59.867 DEBUG:boto:Headers: {} 2023-07-29T00:14:59.868 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:59.868 DEBUG:boto:Port: 8001 2023-07-29T00:14:59.868 DEBUG:boto:Params: {} 2023-07-29T00:14:59.869 DEBUG:boto:Token: None 2023-07-29T00:14:59.869 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:59 GMT /hiiqos-55/b?versionId=null 2023-07-29T00:14:59.870 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:utXFtLpeF/logqu0Gi5UycTBPUs= 2023-07-29T00:14:59.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:utXFtLpeF/logqu0Gi5UycTBPUs=', 'Content-Length': '0'} 2023-07-29T00:14:59.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b85203c33fab547-0064c45a03-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:59.873 DEBUG:boto:path=/hiiqos-55/b 2023-07-29T00:14:59.873 DEBUG:boto:auth_path=/hiiqos-55/b 2023-07-29T00:14:59.873 DEBUG:boto:path=/hiiqos-55/b?versionId=null 2023-07-29T00:14:59.874 DEBUG:boto:auth_path=/hiiqos-55/b?versionId=null 2023-07-29T00:14:59.874 DEBUG:boto:Method: HEAD 2023-07-29T00:14:59.874 DEBUG:boto:Path: /hiiqos-55/b?versionId=null 2023-07-29T00:14:59.875 DEBUG:boto:Data: 2023-07-29T00:14:59.875 DEBUG:boto:Headers: {} 2023-07-29T00:14:59.875 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:59.876 DEBUG:boto:Port: 8000 2023-07-29T00:14:59.876 DEBUG:boto:Params: {} 2023-07-29T00:14:59.876 DEBUG:boto:Token: None 2023-07-29T00:14:59.877 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:59 GMT /hiiqos-55/b?versionId=null 2023-07-29T00:14:59.877 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:utXFtLpeF/logqu0Gi5UycTBPUs= 2023-07-29T00:14:59.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:utXFtLpeF/logqu0Gi5UycTBPUs=', 'Content-Length': '0'} 2023-07-29T00:14:59.879 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8bfc302683b6081-0064c45a03-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:59.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-07-29T00:14:59.880 DEBUG:boto:path=/hiiqos-55/b 2023-07-29T00:14:59.881 DEBUG:boto:auth_path=/hiiqos-55/b 2023-07-29T00:14:59.881 DEBUG:boto:path=/hiiqos-55/b?versionId=null 2023-07-29T00:14:59.881 DEBUG:boto:auth_path=/hiiqos-55/b?versionId=null 2023-07-29T00:14:59.882 DEBUG:boto:Method: GET 2023-07-29T00:14:59.882 DEBUG:boto:Path: /hiiqos-55/b?versionId=null 2023-07-29T00:14:59.882 DEBUG:boto:Data: 2023-07-29T00:14:59.883 DEBUG:boto:Headers: {} 2023-07-29T00:14:59.883 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:59.883 DEBUG:boto:Port: 8001 2023-07-29T00:14:59.884 DEBUG:boto:Params: {} 2023-07-29T00:14:59.884 DEBUG:boto:Token: None 2023-07-29T00:14:59.885 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:59 GMT /hiiqos-55/b?versionId=null 2023-07-29T00:14:59.885 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:K3jsoITgiOh1WpdTHXI210MlBVg= 2023-07-29T00:14:59.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K3jsoITgiOh1WpdTHXI210MlBVg=', 'Content-Length': '0'} 2023-07-29T00:14:59.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daaeeb853918d5e4-0064c45a03-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:59.928 DEBUG:boto:path=/hiiqos-55/b 2023-07-29T00:14:59.929 DEBUG:boto:auth_path=/hiiqos-55/b 2023-07-29T00:14:59.929 DEBUG:boto:path=/hiiqos-55/b?versionId=null 2023-07-29T00:14:59.930 DEBUG:boto:auth_path=/hiiqos-55/b?versionId=null 2023-07-29T00:14:59.930 DEBUG:boto:Method: GET 2023-07-29T00:14:59.930 DEBUG:boto:Path: /hiiqos-55/b?versionId=null 2023-07-29T00:14:59.931 DEBUG:boto:Data: 2023-07-29T00:14:59.931 DEBUG:boto:Headers: {} 2023-07-29T00:14:59.931 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:59.932 DEBUG:boto:Port: 8000 2023-07-29T00:14:59.932 DEBUG:boto:Params: {} 2023-07-29T00:14:59.933 DEBUG:boto:Token: None 2023-07-29T00:14:59.933 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:59 GMT /hiiqos-55/b?versionId=null 2023-07-29T00:14:59.934 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:K3jsoITgiOh1WpdTHXI210MlBVg= 2023-07-29T00:14:59.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K3jsoITgiOh1WpdTHXI210MlBVg=', 'Content-Length': '0'} 2023-07-29T00:14:59.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000380171446d61880b-0064c45a03-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:59.936 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:14:59.937 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:14:59.937 DEBUG:boto:path=/hiiqos-55/?prefix=b 2023-07-29T00:14:59.938 DEBUG:boto:auth_path=/hiiqos-55/?prefix=b 2023-07-29T00:14:59.938 DEBUG:boto:Method: GET 2023-07-29T00:14:59.938 DEBUG:boto:Path: /hiiqos-55/?prefix=b 2023-07-29T00:14:59.939 DEBUG:boto:Data: 2023-07-29T00:14:59.939 DEBUG:boto:Headers: {} 2023-07-29T00:14:59.939 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:59.940 DEBUG:boto:Port: 8001 2023-07-29T00:14:59.940 DEBUG:boto:Params: {} 2023-07-29T00:14:59.941 DEBUG:boto:Token: None 2023-07-29T00:14:59.941 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:59 GMT /hiiqos-55/ 2023-07-29T00:14:59.941 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:t/PJ+bAhtkXlQy9TiCoPZfTxBw0= 2023-07-29T00:14:59.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t/PJ+bAhtkXlQy9TiCoPZfTxBw0=', 'Content-Length': '0'} 2023-07-29T00:14:59.944 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd7e9de391681f78-0064c45a03-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:59.945 DEBUG:boto:b'hiiqos-55b1000falseb2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:59.945 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:14:59.946 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:14:59.946 DEBUG:boto:path=/hiiqos-55/?prefix=b 2023-07-29T00:14:59.946 DEBUG:boto:auth_path=/hiiqos-55/?prefix=b 2023-07-29T00:14:59.947 DEBUG:boto:Method: GET 2023-07-29T00:14:59.947 DEBUG:boto:Path: /hiiqos-55/?prefix=b 2023-07-29T00:14:59.947 DEBUG:boto:Data: 2023-07-29T00:14:59.948 DEBUG:boto:Headers: {} 2023-07-29T00:14:59.948 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:59.949 DEBUG:boto:Port: 8000 2023-07-29T00:14:59.949 DEBUG:boto:Params: {} 2023-07-29T00:14:59.949 DEBUG:boto:Token: None 2023-07-29T00:14:59.950 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:59 GMT /hiiqos-55/ 2023-07-29T00:14:59.950 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:t/PJ+bAhtkXlQy9TiCoPZfTxBw0= 2023-07-29T00:14:59.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t/PJ+bAhtkXlQy9TiCoPZfTxBw0=', 'Content-Length': '0'} 2023-07-29T00:14:59.953 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c51e3c05f377c66a-0064c45a03-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:59.953 DEBUG:boto:b'hiiqos-55b1000falseb2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:14:59.954 DEBUG:boto:path=/hiiqos-55/b 2023-07-29T00:14:59.954 DEBUG:boto:auth_path=/hiiqos-55/b 2023-07-29T00:14:59.955 DEBUG:boto:Method: HEAD 2023-07-29T00:14:59.955 DEBUG:boto:Path: /hiiqos-55/b 2023-07-29T00:14:59.955 DEBUG:boto:Data: 2023-07-29T00:14:59.956 DEBUG:boto:Headers: {} 2023-07-29T00:14:59.956 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:59.956 DEBUG:boto:Port: 8001 2023-07-29T00:14:59.957 DEBUG:boto:Params: {} 2023-07-29T00:14:59.957 DEBUG:boto:Token: None 2023-07-29T00:14:59.957 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:59 GMT /hiiqos-55/b 2023-07-29T00:14:59.958 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VWnzgs1UtCZ0poLsKOuJeiMZGmM= 2023-07-29T00:14:59.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VWnzgs1UtCZ0poLsKOuJeiMZGmM=', 'Content-Length': '0'} 2023-07-29T00:14:59.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001028bc788438768e-0064c45a03-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:59.961 DEBUG:boto:path=/hiiqos-55/b 2023-07-29T00:14:59.961 DEBUG:boto:auth_path=/hiiqos-55/b 2023-07-29T00:14:59.961 DEBUG:boto:Method: HEAD 2023-07-29T00:14:59.962 DEBUG:boto:Path: /hiiqos-55/b 2023-07-29T00:14:59.962 DEBUG:boto:Data: 2023-07-29T00:14:59.962 DEBUG:boto:Headers: {} 2023-07-29T00:14:59.963 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:59.963 DEBUG:boto:Port: 8000 2023-07-29T00:14:59.963 DEBUG:boto:Params: {} 2023-07-29T00:14:59.964 DEBUG:boto:Token: None 2023-07-29T00:14:59.964 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:14:59 GMT /hiiqos-55/b 2023-07-29T00:14:59.965 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VWnzgs1UtCZ0poLsKOuJeiMZGmM= 2023-07-29T00:14:59.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VWnzgs1UtCZ0poLsKOuJeiMZGmM=', 'Content-Length': '0'} 2023-07-29T00:14:59.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a50159a9e3485def-0064c45a03-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:59.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-07-29T00:14:59.968 DEBUG:boto:path=/hiiqos-55/b 2023-07-29T00:14:59.968 DEBUG:boto:auth_path=/hiiqos-55/b 2023-07-29T00:14:59.968 DEBUG:boto:Method: GET 2023-07-29T00:14:59.969 DEBUG:boto:Path: /hiiqos-55/b 2023-07-29T00:14:59.969 DEBUG:boto:Data: 2023-07-29T00:14:59.969 DEBUG:boto:Headers: {} 2023-07-29T00:14:59.970 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:14:59.970 DEBUG:boto:Port: 8001 2023-07-29T00:14:59.970 DEBUG:boto:Params: {} 2023-07-29T00:14:59.971 DEBUG:boto:Token: None 2023-07-29T00:14:59.971 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:59 GMT /hiiqos-55/b 2023-07-29T00:14:59.972 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4gJI8Oc2UBlMDSmw0aj3TixqZRw= 2023-07-29T00:14:59.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4gJI8Oc2UBlMDSmw0aj3TixqZRw=', 'Content-Length': '0'} 2023-07-29T00:14:59.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ce87ad2720ef04d-0064c45a03-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:14:59.974 DEBUG:boto:path=/hiiqos-55/b 2023-07-29T00:14:59.975 DEBUG:boto:auth_path=/hiiqos-55/b 2023-07-29T00:14:59.975 DEBUG:boto:Method: GET 2023-07-29T00:14:59.976 DEBUG:boto:Path: /hiiqos-55/b 2023-07-29T00:14:59.976 DEBUG:boto:Data: 2023-07-29T00:14:59.976 DEBUG:boto:Headers: {} 2023-07-29T00:14:59.977 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:14:59.977 DEBUG:boto:Port: 8000 2023-07-29T00:14:59.977 DEBUG:boto:Params: {} 2023-07-29T00:14:59.978 DEBUG:boto:Token: None 2023-07-29T00:14:59.978 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:14:59 GMT /hiiqos-55/b 2023-07-29T00:14:59.979 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4gJI8Oc2UBlMDSmw0aj3TixqZRw= 2023-07-29T00:14:59.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4gJI8Oc2UBlMDSmw0aj3TixqZRw=', 'Content-Length': '0'} 2023-07-29T00:14:59.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d00be8623b5eb837-0064c45a03-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:15:00.024 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:15:00.025 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:15:00.025 DEBUG:boto:path=/hiiqos-55/?prefix=b 2023-07-29T00:15:00.026 DEBUG:boto:auth_path=/hiiqos-55/?prefix=b 2023-07-29T00:15:00.026 DEBUG:boto:Method: GET 2023-07-29T00:15:00.026 DEBUG:boto:Path: /hiiqos-55/?prefix=b 2023-07-29T00:15:00.027 DEBUG:boto:Data: 2023-07-29T00:15:00.027 DEBUG:boto:Headers: {} 2023-07-29T00:15:00.027 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:15:00.028 DEBUG:boto:Port: 8001 2023-07-29T00:15:00.028 DEBUG:boto:Params: {} 2023-07-29T00:15:00.029 DEBUG:boto:Token: None 2023-07-29T00:15:00.029 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:15:00 GMT /hiiqos-55/ 2023-07-29T00:15:00.030 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8SN+2MFyD4AgqUBt82bgSDDOQ3k= 2023-07-29T00:15:00.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8SN+2MFyD4AgqUBt82bgSDDOQ3k=', 'Content-Length': '0'} 2023-07-29T00:15:00.032 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002362bed2c23187f0-0064c45a03-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:15:00.033 DEBUG:boto:b'hiiqos-55b1000falseb2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:15:00.033 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:15:00.034 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:15:00.034 DEBUG:boto:path=/hiiqos-55/?prefix=b 2023-07-29T00:15:00.034 DEBUG:boto:auth_path=/hiiqos-55/?prefix=b 2023-07-29T00:15:00.035 DEBUG:boto:Method: GET 2023-07-29T00:15:00.035 DEBUG:boto:Path: /hiiqos-55/?prefix=b 2023-07-29T00:15:00.036 DEBUG:boto:Data: 2023-07-29T00:15:00.036 DEBUG:boto:Headers: {} 2023-07-29T00:15:00.036 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:15:00.037 DEBUG:boto:Port: 8000 2023-07-29T00:15:00.037 DEBUG:boto:Params: {} 2023-07-29T00:15:00.037 DEBUG:boto:Token: None 2023-07-29T00:15:00.038 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:15:00 GMT /hiiqos-55/ 2023-07-29T00:15:00.038 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8SN+2MFyD4AgqUBt82bgSDDOQ3k= 2023-07-29T00:15:00.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8SN+2MFyD4AgqUBt82bgSDDOQ3k=', 'Content-Length': '0'} 2023-07-29T00:15:00.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3c8405590d1b286-0064c45a04-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:15:00.041 DEBUG:boto:b'hiiqos-55b1000falseb2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:15:00.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-07-29T00:15:00.042 DEBUG:boto:path=/hiiqos-55/c 2023-07-29T00:15:00.043 DEBUG:boto:auth_path=/hiiqos-55/c 2023-07-29T00:15:00.043 DEBUG:boto:path=/hiiqos-55/c?versionId=null 2023-07-29T00:15:00.044 DEBUG:boto:auth_path=/hiiqos-55/c?versionId=null 2023-07-29T00:15:00.044 DEBUG:boto:Method: GET 2023-07-29T00:15:00.044 DEBUG:boto:Path: /hiiqos-55/c?versionId=null 2023-07-29T00:15:00.045 DEBUG:boto:Data: 2023-07-29T00:15:00.045 DEBUG:boto:Headers: {} 2023-07-29T00:15:00.045 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:15:00.046 DEBUG:boto:Port: 8001 2023-07-29T00:15:00.046 DEBUG:boto:Params: {} 2023-07-29T00:15:00.047 DEBUG:boto:Token: None 2023-07-29T00:15:00.047 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:15:00 GMT /hiiqos-55/c?versionId=null 2023-07-29T00:15:00.047 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YjAmfQCPaTtbcqbhvrR1pWDHv10= 2023-07-29T00:15:00.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YjAmfQCPaTtbcqbhvrR1pWDHv10=', 'Content-Length': '0'} 2023-07-29T00:15:00.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd827131c7a4c227-0064c45a04-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:15:00.092 DEBUG:boto:path=/hiiqos-55/c 2023-07-29T00:15:00.093 DEBUG:boto:auth_path=/hiiqos-55/c 2023-07-29T00:15:00.093 DEBUG:boto:path=/hiiqos-55/c?versionId=null 2023-07-29T00:15:00.094 DEBUG:boto:auth_path=/hiiqos-55/c?versionId=null 2023-07-29T00:15:00.094 DEBUG:boto:Method: GET 2023-07-29T00:15:00.094 DEBUG:boto:Path: /hiiqos-55/c?versionId=null 2023-07-29T00:15:00.095 DEBUG:boto:Data: 2023-07-29T00:15:00.095 DEBUG:boto:Headers: {} 2023-07-29T00:15:00.096 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:15:00.096 DEBUG:boto:Port: 8000 2023-07-29T00:15:00.097 DEBUG:boto:Params: {} 2023-07-29T00:15:00.097 DEBUG:boto:Token: None 2023-07-29T00:15:00.097 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:15:00 GMT /hiiqos-55/c?versionId=null 2023-07-29T00:15:00.098 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YjAmfQCPaTtbcqbhvrR1pWDHv10= 2023-07-29T00:15:00.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YjAmfQCPaTtbcqbhvrR1pWDHv10=', 'Content-Length': '0'} 2023-07-29T00:15:00.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ae6dabfc8ab86af-0064c45a04-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:15:00.102 DEBUG:boto:path=/hiiqos-55/c 2023-07-29T00:15:00.102 DEBUG:boto:auth_path=/hiiqos-55/c 2023-07-29T00:15:00.103 DEBUG:boto:path=/hiiqos-55/c?versionId=null 2023-07-29T00:15:00.103 DEBUG:boto:auth_path=/hiiqos-55/c?versionId=null 2023-07-29T00:15:00.103 DEBUG:boto:Method: HEAD 2023-07-29T00:15:00.104 DEBUG:boto:Path: /hiiqos-55/c?versionId=null 2023-07-29T00:15:00.104 DEBUG:boto:Data: 2023-07-29T00:15:00.104 DEBUG:boto:Headers: {} 2023-07-29T00:15:00.105 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:15:00.105 DEBUG:boto:Port: 8001 2023-07-29T00:15:00.106 DEBUG:boto:Params: {} 2023-07-29T00:15:00.106 DEBUG:boto:Token: None 2023-07-29T00:15:00.106 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:15:00 GMT /hiiqos-55/c?versionId=null 2023-07-29T00:15:00.107 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aSaZD/M6zbTj6g2ntftT/xs5Ink= 2023-07-29T00:15:00.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aSaZD/M6zbTj6g2ntftT/xs5Ink=', 'Content-Length': '0'} 2023-07-29T00:15:00.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000304cff565cebc758-0064c45a04-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:15:00.109 DEBUG:boto:path=/hiiqos-55/c 2023-07-29T00:15:00.110 DEBUG:boto:auth_path=/hiiqos-55/c 2023-07-29T00:15:00.110 DEBUG:boto:path=/hiiqos-55/c?versionId=null 2023-07-29T00:15:00.110 DEBUG:boto:auth_path=/hiiqos-55/c?versionId=null 2023-07-29T00:15:00.111 DEBUG:boto:Method: HEAD 2023-07-29T00:15:00.111 DEBUG:boto:Path: /hiiqos-55/c?versionId=null 2023-07-29T00:15:00.111 DEBUG:boto:Data: 2023-07-29T00:15:00.112 DEBUG:boto:Headers: {} 2023-07-29T00:15:00.112 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:15:00.112 DEBUG:boto:Port: 8000 2023-07-29T00:15:00.113 DEBUG:boto:Params: {} 2023-07-29T00:15:00.113 DEBUG:boto:Token: None 2023-07-29T00:15:00.114 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:15:00 GMT /hiiqos-55/c?versionId=null 2023-07-29T00:15:00.114 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aSaZD/M6zbTj6g2ntftT/xs5Ink= 2023-07-29T00:15:00.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aSaZD/M6zbTj6g2ntftT/xs5Ink=', 'Content-Length': '0'} 2023-07-29T00:15:00.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daaf1a8ad1363dfb-0064c45a04-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:15:00.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-07-29T00:15:00.117 DEBUG:boto:path=/hiiqos-55/c 2023-07-29T00:15:00.118 DEBUG:boto:auth_path=/hiiqos-55/c 2023-07-29T00:15:00.118 DEBUG:boto:path=/hiiqos-55/c?versionId=null 2023-07-29T00:15:00.118 DEBUG:boto:auth_path=/hiiqos-55/c?versionId=null 2023-07-29T00:15:00.119 DEBUG:boto:Method: GET 2023-07-29T00:15:00.119 DEBUG:boto:Path: /hiiqos-55/c?versionId=null 2023-07-29T00:15:00.119 DEBUG:boto:Data: 2023-07-29T00:15:00.120 DEBUG:boto:Headers: {} 2023-07-29T00:15:00.120 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:15:00.120 DEBUG:boto:Port: 8001 2023-07-29T00:15:00.121 DEBUG:boto:Params: {} 2023-07-29T00:15:00.121 DEBUG:boto:Token: None 2023-07-29T00:15:00.122 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:15:00 GMT /hiiqos-55/c?versionId=null 2023-07-29T00:15:00.122 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YjAmfQCPaTtbcqbhvrR1pWDHv10= 2023-07-29T00:15:00.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YjAmfQCPaTtbcqbhvrR1pWDHv10=', 'Content-Length': '0'} 2023-07-29T00:15:00.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf003ad0742d1696-0064c45a04-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:15:00.125 DEBUG:boto:path=/hiiqos-55/c 2023-07-29T00:15:00.125 DEBUG:boto:auth_path=/hiiqos-55/c 2023-07-29T00:15:00.125 DEBUG:boto:path=/hiiqos-55/c?versionId=null 2023-07-29T00:15:00.126 DEBUG:boto:auth_path=/hiiqos-55/c?versionId=null 2023-07-29T00:15:00.126 DEBUG:boto:Method: GET 2023-07-29T00:15:00.127 DEBUG:boto:Path: /hiiqos-55/c?versionId=null 2023-07-29T00:15:00.127 DEBUG:boto:Data: 2023-07-29T00:15:00.127 DEBUG:boto:Headers: {} 2023-07-29T00:15:00.128 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:15:00.128 DEBUG:boto:Port: 8000 2023-07-29T00:15:00.128 DEBUG:boto:Params: {} 2023-07-29T00:15:00.129 DEBUG:boto:Token: None 2023-07-29T00:15:00.129 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:15:00 GMT /hiiqos-55/c?versionId=null 2023-07-29T00:15:00.130 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YjAmfQCPaTtbcqbhvrR1pWDHv10= 2023-07-29T00:15:00.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YjAmfQCPaTtbcqbhvrR1pWDHv10=', 'Content-Length': '0'} 2023-07-29T00:15:00.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c7fb8983862db78-0064c45a04-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:15:00.172 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:15:00.173 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:15:00.173 DEBUG:boto:path=/hiiqos-55/?prefix=c 2023-07-29T00:15:00.174 DEBUG:boto:auth_path=/hiiqos-55/?prefix=c 2023-07-29T00:15:00.174 DEBUG:boto:Method: GET 2023-07-29T00:15:00.174 DEBUG:boto:Path: /hiiqos-55/?prefix=c 2023-07-29T00:15:00.175 DEBUG:boto:Data: 2023-07-29T00:15:00.175 DEBUG:boto:Headers: {} 2023-07-29T00:15:00.175 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:15:00.176 DEBUG:boto:Port: 8001 2023-07-29T00:15:00.176 DEBUG:boto:Params: {} 2023-07-29T00:15:00.177 DEBUG:boto:Token: None 2023-07-29T00:15:00.177 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:15:00 GMT /hiiqos-55/ 2023-07-29T00:15:00.177 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8SN+2MFyD4AgqUBt82bgSDDOQ3k= 2023-07-29T00:15:00.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8SN+2MFyD4AgqUBt82bgSDDOQ3k=', 'Content-Length': '0'} 2023-07-29T00:15:00.180 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4f3aac13ed3cb5b-0064c45a04-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:15:00.181 DEBUG:boto:b'hiiqos-55c1000falsec2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:15:00.181 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:15:00.182 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:15:00.182 DEBUG:boto:path=/hiiqos-55/?prefix=c 2023-07-29T00:15:00.182 DEBUG:boto:auth_path=/hiiqos-55/?prefix=c 2023-07-29T00:15:00.183 DEBUG:boto:Method: GET 2023-07-29T00:15:00.183 DEBUG:boto:Path: /hiiqos-55/?prefix=c 2023-07-29T00:15:00.183 DEBUG:boto:Data: 2023-07-29T00:15:00.184 DEBUG:boto:Headers: {} 2023-07-29T00:15:00.184 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:15:00.185 DEBUG:boto:Port: 8000 2023-07-29T00:15:00.185 DEBUG:boto:Params: {} 2023-07-29T00:15:00.185 DEBUG:boto:Token: None 2023-07-29T00:15:00.186 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:15:00 GMT /hiiqos-55/ 2023-07-29T00:15:00.186 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8SN+2MFyD4AgqUBt82bgSDDOQ3k= 2023-07-29T00:15:00.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8SN+2MFyD4AgqUBt82bgSDDOQ3k=', 'Content-Length': '0'} 2023-07-29T00:15:00.188 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076227edfcce70e2b-0064c45a04-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:15:00.189 DEBUG:boto:b'hiiqos-55c1000falsec2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:15:00.190 DEBUG:boto:path=/hiiqos-55/c 2023-07-29T00:15:00.190 DEBUG:boto:auth_path=/hiiqos-55/c 2023-07-29T00:15:00.190 DEBUG:boto:Method: HEAD 2023-07-29T00:15:00.191 DEBUG:boto:Path: /hiiqos-55/c 2023-07-29T00:15:00.191 DEBUG:boto:Data: 2023-07-29T00:15:00.191 DEBUG:boto:Headers: {} 2023-07-29T00:15:00.192 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:15:00.192 DEBUG:boto:Port: 8001 2023-07-29T00:15:00.192 DEBUG:boto:Params: {} 2023-07-29T00:15:00.193 DEBUG:boto:Token: None 2023-07-29T00:15:00.193 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:15:00 GMT /hiiqos-55/c 2023-07-29T00:15:00.194 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ijOgwpG3/ky9gHXxa8KeS4yyK7s= 2023-07-29T00:15:00.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ijOgwpG3/ky9gHXxa8KeS4yyK7s=', 'Content-Length': '0'} 2023-07-29T00:15:00.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056dbe831a363d27f-0064c45a04-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:15:00.196 DEBUG:boto:path=/hiiqos-55/c 2023-07-29T00:15:00.196 DEBUG:boto:auth_path=/hiiqos-55/c 2023-07-29T00:15:00.197 DEBUG:boto:Method: HEAD 2023-07-29T00:15:00.197 DEBUG:boto:Path: /hiiqos-55/c 2023-07-29T00:15:00.197 DEBUG:boto:Data: 2023-07-29T00:15:00.198 DEBUG:boto:Headers: {} 2023-07-29T00:15:00.198 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:15:00.198 DEBUG:boto:Port: 8000 2023-07-29T00:15:00.199 DEBUG:boto:Params: {} 2023-07-29T00:15:00.199 DEBUG:boto:Token: None 2023-07-29T00:15:00.200 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:15:00 GMT /hiiqos-55/c 2023-07-29T00:15:00.200 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ijOgwpG3/ky9gHXxa8KeS4yyK7s= 2023-07-29T00:15:00.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ijOgwpG3/ky9gHXxa8KeS4yyK7s=', 'Content-Length': '0'} 2023-07-29T00:15:00.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000861104ec7672c6ec-0064c45a04-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:15:00.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-07-29T00:15:00.203 DEBUG:boto:path=/hiiqos-55/c 2023-07-29T00:15:00.203 DEBUG:boto:auth_path=/hiiqos-55/c 2023-07-29T00:15:00.203 DEBUG:boto:Method: GET 2023-07-29T00:15:00.204 DEBUG:boto:Path: /hiiqos-55/c 2023-07-29T00:15:00.204 DEBUG:boto:Data: 2023-07-29T00:15:00.205 DEBUG:boto:Headers: {} 2023-07-29T00:15:00.205 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:15:00.205 DEBUG:boto:Port: 8001 2023-07-29T00:15:00.206 DEBUG:boto:Params: {} 2023-07-29T00:15:00.206 DEBUG:boto:Token: None 2023-07-29T00:15:00.206 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:15:00 GMT /hiiqos-55/c 2023-07-29T00:15:00.207 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Bos7/aX/SnyTg1b0GI/CcwqdDJA= 2023-07-29T00:15:00.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Bos7/aX/SnyTg1b0GI/CcwqdDJA=', 'Content-Length': '0'} 2023-07-29T00:15:00.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073ee503875f1d8fd-0064c45a04-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:15:00.252 DEBUG:boto:path=/hiiqos-55/c 2023-07-29T00:15:00.253 DEBUG:boto:auth_path=/hiiqos-55/c 2023-07-29T00:15:00.253 DEBUG:boto:Method: GET 2023-07-29T00:15:00.254 DEBUG:boto:Path: /hiiqos-55/c 2023-07-29T00:15:00.254 DEBUG:boto:Data: 2023-07-29T00:15:00.254 DEBUG:boto:Headers: {} 2023-07-29T00:15:00.255 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:15:00.255 DEBUG:boto:Port: 8000 2023-07-29T00:15:00.255 DEBUG:boto:Params: {} 2023-07-29T00:15:00.256 DEBUG:boto:Token: None 2023-07-29T00:15:00.256 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:15:00 GMT /hiiqos-55/c 2023-07-29T00:15:00.257 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Bos7/aX/SnyTg1b0GI/CcwqdDJA= 2023-07-29T00:15:00.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Bos7/aX/SnyTg1b0GI/CcwqdDJA=', 'Content-Length': '0'} 2023-07-29T00:15:00.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cbc49c4282e4605-0064c45a04-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:15:00.260 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:15:00.260 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:15:00.261 DEBUG:boto:path=/hiiqos-55/?prefix=c 2023-07-29T00:15:00.261 DEBUG:boto:auth_path=/hiiqos-55/?prefix=c 2023-07-29T00:15:00.261 DEBUG:boto:Method: GET 2023-07-29T00:15:00.262 DEBUG:boto:Path: /hiiqos-55/?prefix=c 2023-07-29T00:15:00.262 DEBUG:boto:Data: 2023-07-29T00:15:00.262 DEBUG:boto:Headers: {} 2023-07-29T00:15:00.263 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:15:00.263 DEBUG:boto:Port: 8001 2023-07-29T00:15:00.263 DEBUG:boto:Params: {} 2023-07-29T00:15:00.264 DEBUG:boto:Token: None 2023-07-29T00:15:00.264 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:15:00 GMT /hiiqos-55/ 2023-07-29T00:15:00.265 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8SN+2MFyD4AgqUBt82bgSDDOQ3k= 2023-07-29T00:15:00.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8SN+2MFyD4AgqUBt82bgSDDOQ3k=', 'Content-Length': '0'} 2023-07-29T00:15:00.267 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c8921526bb23edc-0064c45a04-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:15:00.268 DEBUG:boto:b'hiiqos-55c1000falsec2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:15:00.269 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:15:00.269 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:15:00.269 DEBUG:boto:path=/hiiqos-55/?prefix=c 2023-07-29T00:15:00.270 DEBUG:boto:auth_path=/hiiqos-55/?prefix=c 2023-07-29T00:15:00.270 DEBUG:boto:Method: GET 2023-07-29T00:15:00.270 DEBUG:boto:Path: /hiiqos-55/?prefix=c 2023-07-29T00:15:00.271 DEBUG:boto:Data: 2023-07-29T00:15:00.271 DEBUG:boto:Headers: {} 2023-07-29T00:15:00.272 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:15:00.272 DEBUG:boto:Port: 8000 2023-07-29T00:15:00.272 DEBUG:boto:Params: {} 2023-07-29T00:15:00.273 DEBUG:boto:Token: None 2023-07-29T00:15:00.273 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:15:00 GMT /hiiqos-55/ 2023-07-29T00:15:00.274 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8SN+2MFyD4AgqUBt82bgSDDOQ3k= 2023-07-29T00:15:00.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8SN+2MFyD4AgqUBt82bgSDDOQ3k=', 'Content-Length': '0'} 2023-07-29T00:15:00.276 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca50dcad5c4ff2aa-0064c45a04-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:15:00.277 DEBUG:boto:b'hiiqos-55c1000falsec2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:15:00.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-07-29T00:15:00.278 DEBUG:boto:path=/hiiqos-55/d 2023-07-29T00:15:00.278 DEBUG:boto:auth_path=/hiiqos-55/d 2023-07-29T00:15:00.279 DEBUG:boto:path=/hiiqos-55/d?versionId=null 2023-07-29T00:15:00.279 DEBUG:boto:auth_path=/hiiqos-55/d?versionId=null 2023-07-29T00:15:00.279 DEBUG:boto:Method: GET 2023-07-29T00:15:00.280 DEBUG:boto:Path: /hiiqos-55/d?versionId=null 2023-07-29T00:15:00.280 DEBUG:boto:Data: 2023-07-29T00:15:00.281 DEBUG:boto:Headers: {} 2023-07-29T00:15:00.281 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:15:00.281 DEBUG:boto:Port: 8001 2023-07-29T00:15:00.282 DEBUG:boto:Params: {} 2023-07-29T00:15:00.282 DEBUG:boto:Token: None 2023-07-29T00:15:00.282 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:15:00 GMT /hiiqos-55/d?versionId=null 2023-07-29T00:15:00.283 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EsQHzPYxlLmznzSu7D4nua6U214= 2023-07-29T00:15:00.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EsQHzPYxlLmznzSu7D4nua6U214=', 'Content-Length': '0'} 2023-07-29T00:15:00.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b50165b009733bba-0064c45a04-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:15:00.286 DEBUG:boto:path=/hiiqos-55/d 2023-07-29T00:15:00.286 DEBUG:boto:auth_path=/hiiqos-55/d 2023-07-29T00:15:00.286 DEBUG:boto:path=/hiiqos-55/d?versionId=null 2023-07-29T00:15:00.287 DEBUG:boto:auth_path=/hiiqos-55/d?versionId=null 2023-07-29T00:15:00.287 DEBUG:boto:Method: GET 2023-07-29T00:15:00.287 DEBUG:boto:Path: /hiiqos-55/d?versionId=null 2023-07-29T00:15:00.288 DEBUG:boto:Data: 2023-07-29T00:15:00.288 DEBUG:boto:Headers: {} 2023-07-29T00:15:00.288 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:15:00.289 DEBUG:boto:Port: 8000 2023-07-29T00:15:00.289 DEBUG:boto:Params: {} 2023-07-29T00:15:00.290 DEBUG:boto:Token: None 2023-07-29T00:15:00.290 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:15:00 GMT /hiiqos-55/d?versionId=null 2023-07-29T00:15:00.290 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EsQHzPYxlLmznzSu7D4nua6U214= 2023-07-29T00:15:00.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EsQHzPYxlLmznzSu7D4nua6U214=', 'Content-Length': '0'} 2023-07-29T00:15:00.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bebe8766a26861a-0064c45a04-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:15:00.336 DEBUG:boto:path=/hiiqos-55/d 2023-07-29T00:15:00.337 DEBUG:boto:auth_path=/hiiqos-55/d 2023-07-29T00:15:00.337 DEBUG:boto:path=/hiiqos-55/d?versionId=null 2023-07-29T00:15:00.338 DEBUG:boto:auth_path=/hiiqos-55/d?versionId=null 2023-07-29T00:15:00.338 DEBUG:boto:Method: HEAD 2023-07-29T00:15:00.338 DEBUG:boto:Path: /hiiqos-55/d?versionId=null 2023-07-29T00:15:00.339 DEBUG:boto:Data: 2023-07-29T00:15:00.339 DEBUG:boto:Headers: {} 2023-07-29T00:15:00.339 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:15:00.340 DEBUG:boto:Port: 8001 2023-07-29T00:15:00.340 DEBUG:boto:Params: {} 2023-07-29T00:15:00.340 DEBUG:boto:Token: None 2023-07-29T00:15:00.341 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:15:00 GMT /hiiqos-55/d?versionId=null 2023-07-29T00:15:00.342 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9lIoYyf5lgi/bGQwGetmKHx14Rw= 2023-07-29T00:15:00.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9lIoYyf5lgi/bGQwGetmKHx14Rw=', 'Content-Length': '0'} 2023-07-29T00:15:00.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032ed55bdab7b78cf-0064c45a04-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:15:00.345 DEBUG:boto:path=/hiiqos-55/d 2023-07-29T00:15:00.345 DEBUG:boto:auth_path=/hiiqos-55/d 2023-07-29T00:15:00.345 DEBUG:boto:path=/hiiqos-55/d?versionId=null 2023-07-29T00:15:00.346 DEBUG:boto:auth_path=/hiiqos-55/d?versionId=null 2023-07-29T00:15:00.346 DEBUG:boto:Method: HEAD 2023-07-29T00:15:00.346 DEBUG:boto:Path: /hiiqos-55/d?versionId=null 2023-07-29T00:15:00.347 DEBUG:boto:Data: 2023-07-29T00:15:00.347 DEBUG:boto:Headers: {} 2023-07-29T00:15:00.347 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:15:00.348 DEBUG:boto:Port: 8000 2023-07-29T00:15:00.348 DEBUG:boto:Params: {} 2023-07-29T00:15:00.349 DEBUG:boto:Token: None 2023-07-29T00:15:00.349 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:15:00 GMT /hiiqos-55/d?versionId=null 2023-07-29T00:15:00.349 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9lIoYyf5lgi/bGQwGetmKHx14Rw= 2023-07-29T00:15:00.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9lIoYyf5lgi/bGQwGetmKHx14Rw=', 'Content-Length': '0'} 2023-07-29T00:15:00.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa07b2e116c593a2-0064c45a04-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:15:00.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-07-29T00:15:00.352 DEBUG:boto:path=/hiiqos-55/d 2023-07-29T00:15:00.353 DEBUG:boto:auth_path=/hiiqos-55/d 2023-07-29T00:15:00.353 DEBUG:boto:path=/hiiqos-55/d?versionId=null 2023-07-29T00:15:00.354 DEBUG:boto:auth_path=/hiiqos-55/d?versionId=null 2023-07-29T00:15:00.354 DEBUG:boto:Method: GET 2023-07-29T00:15:00.355 DEBUG:boto:Path: /hiiqos-55/d?versionId=null 2023-07-29T00:15:00.355 DEBUG:boto:Data: 2023-07-29T00:15:00.355 DEBUG:boto:Headers: {} 2023-07-29T00:15:00.356 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:15:00.356 DEBUG:boto:Port: 8001 2023-07-29T00:15:00.356 DEBUG:boto:Params: {} 2023-07-29T00:15:00.357 DEBUG:boto:Token: None 2023-07-29T00:15:00.357 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:15:00 GMT /hiiqos-55/d?versionId=null 2023-07-29T00:15:00.358 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EsQHzPYxlLmznzSu7D4nua6U214= 2023-07-29T00:15:00.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EsQHzPYxlLmznzSu7D4nua6U214=', 'Content-Length': '0'} 2023-07-29T00:15:00.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e5e0a328668838c-0064c45a04-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:15:00.360 DEBUG:boto:path=/hiiqos-55/d 2023-07-29T00:15:00.361 DEBUG:boto:auth_path=/hiiqos-55/d 2023-07-29T00:15:00.361 DEBUG:boto:path=/hiiqos-55/d?versionId=null 2023-07-29T00:15:00.361 DEBUG:boto:auth_path=/hiiqos-55/d?versionId=null 2023-07-29T00:15:00.362 DEBUG:boto:Method: GET 2023-07-29T00:15:00.362 DEBUG:boto:Path: /hiiqos-55/d?versionId=null 2023-07-29T00:15:00.362 DEBUG:boto:Data: 2023-07-29T00:15:00.363 DEBUG:boto:Headers: {} 2023-07-29T00:15:00.363 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:15:00.364 DEBUG:boto:Port: 8000 2023-07-29T00:15:00.364 DEBUG:boto:Params: {} 2023-07-29T00:15:00.364 DEBUG:boto:Token: None 2023-07-29T00:15:00.365 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:15:00 GMT /hiiqos-55/d?versionId=null 2023-07-29T00:15:00.365 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EsQHzPYxlLmznzSu7D4nua6U214= 2023-07-29T00:15:00.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EsQHzPYxlLmznzSu7D4nua6U214=', 'Content-Length': '0'} 2023-07-29T00:15:00.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c22e6d89637c1c44-0064c45a04-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:15:00.368 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:15:00.369 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:15:00.369 DEBUG:boto:path=/hiiqos-55/?prefix=d 2023-07-29T00:15:00.369 DEBUG:boto:auth_path=/hiiqos-55/?prefix=d 2023-07-29T00:15:00.370 DEBUG:boto:Method: GET 2023-07-29T00:15:00.370 DEBUG:boto:Path: /hiiqos-55/?prefix=d 2023-07-29T00:15:00.370 DEBUG:boto:Data: 2023-07-29T00:15:00.371 DEBUG:boto:Headers: {} 2023-07-29T00:15:00.371 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:15:00.371 DEBUG:boto:Port: 8001 2023-07-29T00:15:00.372 DEBUG:boto:Params: {} 2023-07-29T00:15:00.372 DEBUG:boto:Token: None 2023-07-29T00:15:00.373 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:15:00 GMT /hiiqos-55/ 2023-07-29T00:15:00.373 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8SN+2MFyD4AgqUBt82bgSDDOQ3k= 2023-07-29T00:15:00.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8SN+2MFyD4AgqUBt82bgSDDOQ3k=', 'Content-Length': '0'} 2023-07-29T00:15:00.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a77877f50e3c627-0064c45a04-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:15:00.376 DEBUG:boto:b'hiiqos-55d1000falsed2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:15:00.377 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:15:00.377 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:15:00.377 DEBUG:boto:path=/hiiqos-55/?prefix=d 2023-07-29T00:15:00.378 DEBUG:boto:auth_path=/hiiqos-55/?prefix=d 2023-07-29T00:15:00.378 DEBUG:boto:Method: GET 2023-07-29T00:15:00.379 DEBUG:boto:Path: /hiiqos-55/?prefix=d 2023-07-29T00:15:00.379 DEBUG:boto:Data: 2023-07-29T00:15:00.379 DEBUG:boto:Headers: {} 2023-07-29T00:15:00.380 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:15:00.380 DEBUG:boto:Port: 8000 2023-07-29T00:15:00.380 DEBUG:boto:Params: {} 2023-07-29T00:15:00.381 DEBUG:boto:Token: None 2023-07-29T00:15:00.381 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:15:00 GMT /hiiqos-55/ 2023-07-29T00:15:00.382 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8SN+2MFyD4AgqUBt82bgSDDOQ3k= 2023-07-29T00:15:00.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8SN+2MFyD4AgqUBt82bgSDDOQ3k=', 'Content-Length': '0'} 2023-07-29T00:15:00.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a096c63a9307ee52-0064c45a04-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:15:00.385 DEBUG:boto:b'hiiqos-55d1000falsed2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:15:00.385 DEBUG:boto:path=/hiiqos-55/d 2023-07-29T00:15:00.386 DEBUG:boto:auth_path=/hiiqos-55/d 2023-07-29T00:15:00.386 DEBUG:boto:Method: HEAD 2023-07-29T00:15:00.387 DEBUG:boto:Path: /hiiqos-55/d 2023-07-29T00:15:00.387 DEBUG:boto:Data: 2023-07-29T00:15:00.387 DEBUG:boto:Headers: {} 2023-07-29T00:15:00.388 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:15:00.388 DEBUG:boto:Port: 8001 2023-07-29T00:15:00.388 DEBUG:boto:Params: {} 2023-07-29T00:15:00.389 DEBUG:boto:Token: None 2023-07-29T00:15:00.389 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:15:00 GMT /hiiqos-55/d 2023-07-29T00:15:00.389 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vpIi4Sn3UkxFlRQyuw+9Syfas7U= 2023-07-29T00:15:00.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vpIi4Sn3UkxFlRQyuw+9Syfas7U=', 'Content-Length': '0'} 2023-07-29T00:15:00.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a871454aa75778f-0064c45a04-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:15:00.392 DEBUG:boto:path=/hiiqos-55/d 2023-07-29T00:15:00.393 DEBUG:boto:auth_path=/hiiqos-55/d 2023-07-29T00:15:00.393 DEBUG:boto:Method: HEAD 2023-07-29T00:15:00.393 DEBUG:boto:Path: /hiiqos-55/d 2023-07-29T00:15:00.394 DEBUG:boto:Data: 2023-07-29T00:15:00.394 DEBUG:boto:Headers: {} 2023-07-29T00:15:00.394 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:15:00.395 DEBUG:boto:Port: 8000 2023-07-29T00:15:00.395 DEBUG:boto:Params: {} 2023-07-29T00:15:00.395 DEBUG:boto:Token: None 2023-07-29T00:15:00.396 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:15:00 GMT /hiiqos-55/d 2023-07-29T00:15:00.396 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vpIi4Sn3UkxFlRQyuw+9Syfas7U= 2023-07-29T00:15:00.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vpIi4Sn3UkxFlRQyuw+9Syfas7U=', 'Content-Length': '0'} 2023-07-29T00:15:00.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cad3aec9f3a544ed-0064c45a04-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:15:00.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-07-29T00:15:00.399 DEBUG:boto:path=/hiiqos-55/d 2023-07-29T00:15:00.400 DEBUG:boto:auth_path=/hiiqos-55/d 2023-07-29T00:15:00.400 DEBUG:boto:Method: GET 2023-07-29T00:15:00.400 DEBUG:boto:Path: /hiiqos-55/d 2023-07-29T00:15:00.401 DEBUG:boto:Data: 2023-07-29T00:15:00.401 DEBUG:boto:Headers: {} 2023-07-29T00:15:00.401 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:15:00.402 DEBUG:boto:Port: 8001 2023-07-29T00:15:00.402 DEBUG:boto:Params: {} 2023-07-29T00:15:00.403 DEBUG:boto:Token: None 2023-07-29T00:15:00.403 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:15:00 GMT /hiiqos-55/d 2023-07-29T00:15:00.403 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MonvgxDid5FJgHcPN/I8YGfB6vY= 2023-07-29T00:15:00.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MonvgxDid5FJgHcPN/I8YGfB6vY=', 'Content-Length': '0'} 2023-07-29T00:15:00.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0179e24fcc46e1a-0064c45a04-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:15:00.406 DEBUG:boto:path=/hiiqos-55/d 2023-07-29T00:15:00.407 DEBUG:boto:auth_path=/hiiqos-55/d 2023-07-29T00:15:00.407 DEBUG:boto:Method: GET 2023-07-29T00:15:00.408 DEBUG:boto:Path: /hiiqos-55/d 2023-07-29T00:15:00.408 DEBUG:boto:Data: 2023-07-29T00:15:00.408 DEBUG:boto:Headers: {} 2023-07-29T00:15:00.409 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:15:00.409 DEBUG:boto:Port: 8000 2023-07-29T00:15:00.409 DEBUG:boto:Params: {} 2023-07-29T00:15:00.410 DEBUG:boto:Token: None 2023-07-29T00:15:00.410 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:15:00 GMT /hiiqos-55/d 2023-07-29T00:15:00.411 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MonvgxDid5FJgHcPN/I8YGfB6vY= 2023-07-29T00:15:00.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MonvgxDid5FJgHcPN/I8YGfB6vY=', 'Content-Length': '0'} 2023-07-29T00:15:00.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074fb0ab0c70b6a66-0064c45a04-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:15:00.414 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:15:00.414 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:15:00.414 DEBUG:boto:path=/hiiqos-55/?prefix=d 2023-07-29T00:15:00.415 DEBUG:boto:auth_path=/hiiqos-55/?prefix=d 2023-07-29T00:15:00.415 DEBUG:boto:Method: GET 2023-07-29T00:15:00.415 DEBUG:boto:Path: /hiiqos-55/?prefix=d 2023-07-29T00:15:00.416 DEBUG:boto:Data: 2023-07-29T00:15:00.416 DEBUG:boto:Headers: {} 2023-07-29T00:15:00.417 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:15:00.417 DEBUG:boto:Port: 8001 2023-07-29T00:15:00.417 DEBUG:boto:Params: {} 2023-07-29T00:15:00.418 DEBUG:boto:Token: None 2023-07-29T00:15:00.418 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:15:00 GMT /hiiqos-55/ 2023-07-29T00:15:00.419 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8SN+2MFyD4AgqUBt82bgSDDOQ3k= 2023-07-29T00:15:00.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8SN+2MFyD4AgqUBt82bgSDDOQ3k=', 'Content-Length': '0'} 2023-07-29T00:15:00.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be17c7f39e3478b5-0064c45a04-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:15:00.422 DEBUG:boto:b'hiiqos-55d1000falsed2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:15:00.423 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:15:00.423 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:15:00.424 DEBUG:boto:path=/hiiqos-55/?prefix=d 2023-07-29T00:15:00.424 DEBUG:boto:auth_path=/hiiqos-55/?prefix=d 2023-07-29T00:15:00.424 DEBUG:boto:Method: GET 2023-07-29T00:15:00.425 DEBUG:boto:Path: /hiiqos-55/?prefix=d 2023-07-29T00:15:00.425 DEBUG:boto:Data: 2023-07-29T00:15:00.426 DEBUG:boto:Headers: {} 2023-07-29T00:15:00.426 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:15:00.426 DEBUG:boto:Port: 8000 2023-07-29T00:15:00.427 DEBUG:boto:Params: {} 2023-07-29T00:15:00.427 DEBUG:boto:Token: None 2023-07-29T00:15:00.427 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:15:00 GMT /hiiqos-55/ 2023-07-29T00:15:00.428 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8SN+2MFyD4AgqUBt82bgSDDOQ3k= 2023-07-29T00:15:00.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8SN+2MFyD4AgqUBt82bgSDDOQ3k=', 'Content-Length': '0'} 2023-07-29T00:15:00.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000086cce5d9d0dc9d1-0064c45a04-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:15:00.472 DEBUG:boto:b'hiiqos-55d1000falsed2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:15:00.473 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-55 zones={a2, a1} 2023-07-29T00:15:00.473 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'hiiqos-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:15:00.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'hiiqos-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:15:00.474 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket hiiqos-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:15:00.520 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:15:00.521 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:15:00.562 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:15:00.562 INFO:teuthology.orchestra.run.smithi101.stdout: "layout": { 2023-07-29T00:15:00.563 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding": "None", 2023-07-29T00:15:00.563 INFO:teuthology.orchestra.run.smithi101.stdout: "current_index": { 2023-07-29T00:15:00.563 INFO:teuthology.orchestra.run.smithi101.stdout: "gen": 0, 2023-07-29T00:15:00.563 INFO:teuthology.orchestra.run.smithi101.stdout: "layout": { 2023-07-29T00:15:00.563 INFO:teuthology.orchestra.run.smithi101.stdout: "type": "Normal", 2023-07-29T00:15:00.563 INFO:teuthology.orchestra.run.smithi101.stdout: "normal": { 2023-07-29T00:15:00.564 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 11, 2023-07-29T00:15:00.564 INFO:teuthology.orchestra.run.smithi101.stdout: "hash_type": "Mod" 2023-07-29T00:15:00.565 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:15:00.565 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:15:00.565 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:15:00.565 INFO:teuthology.orchestra.run.smithi101.stdout: "logs": [ 2023-07-29T00:15:00.565 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:15:00.565 INFO:teuthology.orchestra.run.smithi101.stdout: "gen": 0, 2023-07-29T00:15:00.565 INFO:teuthology.orchestra.run.smithi101.stdout: "layout": { 2023-07-29T00:15:00.566 INFO:teuthology.orchestra.run.smithi101.stdout: "type": "InIndex", 2023-07-29T00:15:00.566 INFO:teuthology.orchestra.run.smithi101.stdout: "in_index": { 2023-07-29T00:15:00.566 INFO:teuthology.orchestra.run.smithi101.stdout: "gen": 0, 2023-07-29T00:15:00.566 INFO:teuthology.orchestra.run.smithi101.stdout: "layout": { 2023-07-29T00:15:00.566 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 11, 2023-07-29T00:15:00.567 INFO:teuthology.orchestra.run.smithi101.stdout: "hash_type": "Mod" 2023-07-29T00:15:00.567 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:15:00.567 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:15:00.567 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:15:00.567 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:15:00.567 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:15:00.568 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:15:00.568 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:15:00.569 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'hiiqos-55', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:15:00.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'hiiqos-55', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:15:00.570 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket hiiqos-55 --gen 0 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:15:00.616 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:15:00.616 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:15:00.656 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T00:15:00.656 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:15:00.657 INFO:teuthology.orchestra.run.smithi101.stdout: "op_id": "1#00000000001.56.6", 2023-07-29T00:15:00.657 INFO:teuthology.orchestra.run.smithi101.stdout: "op_tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.8037444673821786044", 2023-07-29T00:15:00.657 INFO:teuthology.orchestra.run.smithi101.stdout: "op": "write", 2023-07-29T00:15:00.658 INFO:teuthology.orchestra.run.smithi101.stdout: "object": "a", 2023-07-29T00:15:00.658 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:15:00.658 INFO:teuthology.orchestra.run.smithi101.stdout: "state": "complete", 2023-07-29T00:15:00.658 INFO:teuthology.orchestra.run.smithi101.stdout: "index_ver": 1, 2023-07-29T00:15:00.659 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:14:43.934557423Z", 2023-07-29T00:15:00.659 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:15:00.659 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:15:00.659 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 12 2023-07-29T00:15:00.659 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:15:00.659 INFO:teuthology.orchestra.run.smithi101.stdout: "bilog_flags": 0, 2023-07-29T00:15:00.659 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned": false, 2023-07-29T00:15:00.660 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "", 2023-07-29T00:15:00.660 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "", 2023-07-29T00:15:00.660 INFO:teuthology.orchestra.run.smithi101.stdout: "zones_trace": [ 2023-07-29T00:15:00.660 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:15:00.660 INFO:teuthology.orchestra.run.smithi101.stdout: "entry": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20" 2023-07-29T00:15:00.661 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:15:00.661 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:15:00.661 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:15:00.661 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:15:00.661 INFO:teuthology.orchestra.run.smithi101.stdout: "op_id": "5#00000000001.58.6", 2023-07-29T00:15:00.661 INFO:teuthology.orchestra.run.smithi101.stdout: "op_tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.7287745460204636495", 2023-07-29T00:15:00.661 INFO:teuthology.orchestra.run.smithi101.stdout: "op": "write", 2023-07-29T00:15:00.662 INFO:teuthology.orchestra.run.smithi101.stdout: "object": "d", 2023-07-29T00:15:00.662 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:15:00.662 INFO:teuthology.orchestra.run.smithi101.stdout: "state": "complete", 2023-07-29T00:15:00.662 INFO:teuthology.orchestra.run.smithi101.stdout: "index_ver": 1, 2023-07-29T00:15:00.662 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:14:44.026573109Z", 2023-07-29T00:15:00.662 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:15:00.662 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:15:00.663 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 2 2023-07-29T00:15:00.663 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:15:00.663 INFO:teuthology.orchestra.run.smithi101.stdout: "bilog_flags": 0, 2023-07-29T00:15:00.663 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned": false, 2023-07-29T00:15:00.663 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "", 2023-07-29T00:15:00.663 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "", 2023-07-29T00:15:00.664 INFO:teuthology.orchestra.run.smithi101.stdout: "zones_trace": [ 2023-07-29T00:15:00.664 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:15:00.664 INFO:teuthology.orchestra.run.smithi101.stdout: "entry": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20" 2023-07-29T00:15:00.664 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:15:00.664 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:15:00.664 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:15:00.664 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:15:00.665 INFO:teuthology.orchestra.run.smithi101.stdout: "op_id": "8#00000000001.38.6", 2023-07-29T00:15:00.665 INFO:teuthology.orchestra.run.smithi101.stdout: "op_tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.15414676233093376460", 2023-07-29T00:15:00.665 INFO:teuthology.orchestra.run.smithi101.stdout: "op": "write", 2023-07-29T00:15:00.665 INFO:teuthology.orchestra.run.smithi101.stdout: "object": "c", 2023-07-29T00:15:00.666 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:15:00.666 INFO:teuthology.orchestra.run.smithi101.stdout: "state": "complete", 2023-07-29T00:15:00.666 INFO:teuthology.orchestra.run.smithi101.stdout: "index_ver": 1, 2023-07-29T00:15:00.666 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:14:44.008991375Z", 2023-07-29T00:15:00.666 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:15:00.666 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:15:00.666 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 2 2023-07-29T00:15:00.667 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:15:00.667 INFO:teuthology.orchestra.run.smithi101.stdout: "bilog_flags": 0, 2023-07-29T00:15:00.667 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned": false, 2023-07-29T00:15:00.667 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "", 2023-07-29T00:15:00.667 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "", 2023-07-29T00:15:00.667 INFO:teuthology.orchestra.run.smithi101.stdout: "zones_trace": [ 2023-07-29T00:15:00.668 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:15:00.668 INFO:teuthology.orchestra.run.smithi101.stdout: "entry": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20" 2023-07-29T00:15:00.668 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:15:00.668 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:15:00.668 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:15:00.668 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:15:00.669 INFO:teuthology.orchestra.run.smithi101.stdout: "op_id": "10#00000000001.23.6", 2023-07-29T00:15:00.669 INFO:teuthology.orchestra.run.smithi101.stdout: "op_tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.17230896804495972297", 2023-07-29T00:15:00.669 INFO:teuthology.orchestra.run.smithi101.stdout: "op": "write", 2023-07-29T00:15:00.669 INFO:teuthology.orchestra.run.smithi101.stdout: "object": "b", 2023-07-29T00:15:00.669 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:15:00.670 INFO:teuthology.orchestra.run.smithi101.stdout: "state": "complete", 2023-07-29T00:15:00.670 INFO:teuthology.orchestra.run.smithi101.stdout: "index_ver": 1, 2023-07-29T00:15:00.670 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:14:43.991260949Z", 2023-07-29T00:15:00.670 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:15:00.671 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:15:00.671 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 2 2023-07-29T00:15:00.671 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:15:00.671 INFO:teuthology.orchestra.run.smithi101.stdout: "bilog_flags": 0, 2023-07-29T00:15:00.672 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned": false, 2023-07-29T00:15:00.672 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "", 2023-07-29T00:15:00.672 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "", 2023-07-29T00:15:00.672 INFO:teuthology.orchestra.run.smithi101.stdout: "zones_trace": [ 2023-07-29T00:15:00.672 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:15:00.672 INFO:teuthology.orchestra.run.smithi101.stdout: "entry": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20" 2023-07-29T00:15:00.673 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:15:00.673 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:15:00.673 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:15:00.673 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T00:15:00.674 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'hiiqos-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:15:00.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'hiiqos-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:15:00.675 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket hiiqos-55 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:15:00.721 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:15:00.721 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:15:00.759 INFO:teuthology.orchestra.run.smithi101.stdout:tenant: 2023-07-29T00:15:00.759 INFO:teuthology.orchestra.run.smithi101.stdout:bucket name: hiiqos-55 2023-07-29T00:15:00.763 INFO:teuthology.orchestra.run.smithi101.stdout:total entries: 4 2023-07-29T00:15:00.796 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'hiiqos-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:15:00.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'hiiqos-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:15:00.798 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket hiiqos-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:15:00.840 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:15:00.841 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:15:00.884 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:15:00.884 INFO:teuthology.orchestra.run.smithi101.stdout: "layout": { 2023-07-29T00:15:00.885 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding": "None", 2023-07-29T00:15:00.885 INFO:teuthology.orchestra.run.smithi101.stdout: "current_index": { 2023-07-29T00:15:00.885 INFO:teuthology.orchestra.run.smithi101.stdout: "gen": 1, 2023-07-29T00:15:00.885 INFO:teuthology.orchestra.run.smithi101.stdout: "layout": { 2023-07-29T00:15:00.885 INFO:teuthology.orchestra.run.smithi101.stdout: "type": "Normal", 2023-07-29T00:15:00.885 INFO:teuthology.orchestra.run.smithi101.stdout: "normal": { 2023-07-29T00:15:00.886 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 3, 2023-07-29T00:15:00.886 INFO:teuthology.orchestra.run.smithi101.stdout: "hash_type": "Mod" 2023-07-29T00:15:00.886 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:15:00.886 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:15:00.886 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:15:00.886 INFO:teuthology.orchestra.run.smithi101.stdout: "logs": [ 2023-07-29T00:15:00.886 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:15:00.887 INFO:teuthology.orchestra.run.smithi101.stdout: "gen": 0, 2023-07-29T00:15:00.887 INFO:teuthology.orchestra.run.smithi101.stdout: "layout": { 2023-07-29T00:15:00.887 INFO:teuthology.orchestra.run.smithi101.stdout: "type": "InIndex", 2023-07-29T00:15:00.887 INFO:teuthology.orchestra.run.smithi101.stdout: "in_index": { 2023-07-29T00:15:00.887 INFO:teuthology.orchestra.run.smithi101.stdout: "gen": 0, 2023-07-29T00:15:00.887 INFO:teuthology.orchestra.run.smithi101.stdout: "layout": { 2023-07-29T00:15:00.887 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 11, 2023-07-29T00:15:00.888 INFO:teuthology.orchestra.run.smithi101.stdout: "hash_type": "Mod" 2023-07-29T00:15:00.888 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:15:00.888 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:15:00.888 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:15:00.888 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:15:00.888 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:15:00.889 INFO:teuthology.orchestra.run.smithi101.stdout: "gen": 1, 2023-07-29T00:15:00.889 INFO:teuthology.orchestra.run.smithi101.stdout: "layout": { 2023-07-29T00:15:00.889 INFO:teuthology.orchestra.run.smithi101.stdout: "type": "InIndex", 2023-07-29T00:15:00.889 INFO:teuthology.orchestra.run.smithi101.stdout: "in_index": { 2023-07-29T00:15:00.889 INFO:teuthology.orchestra.run.smithi101.stdout: "gen": 1, 2023-07-29T00:15:00.889 INFO:teuthology.orchestra.run.smithi101.stdout: "layout": { 2023-07-29T00:15:00.889 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 3, 2023-07-29T00:15:00.890 INFO:teuthology.orchestra.run.smithi101.stdout: "hash_type": "Mod" 2023-07-29T00:15:00.890 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:15:00.890 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:15:00.890 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:15:00.890 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:15:00.890 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:15:00.890 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:15:00.891 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:15:00.891 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'hiiqos-55', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:15:00.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'hiiqos-55', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:15:00.892 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket hiiqos-55 --gen 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:15:00.970 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:15:00.971 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:15:01.011 INFO:teuthology.orchestra.run.smithi101.stdout:[] 2023-07-29T00:15:01.013 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:15:01.013 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:15:01.013 DEBUG:boto:Method: HEAD 2023-07-29T00:15:01.014 DEBUG:boto:Path: /hiiqos-55/ 2023-07-29T00:15:01.014 DEBUG:boto:Data: 2023-07-29T00:15:01.014 DEBUG:boto:Headers: {} 2023-07-29T00:15:01.015 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:15:01.015 DEBUG:boto:Port: 8000 2023-07-29T00:15:01.015 DEBUG:boto:Params: {} 2023-07-29T00:15:01.016 DEBUG:boto:Token: None 2023-07-29T00:15:01.016 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:15:01 GMT /hiiqos-55/ 2023-07-29T00:15:01.017 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DeuQEaQFwkMkxTy5Q9DpGMIBJeI= 2023-07-29T00:15:01.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DeuQEaQFwkMkxTy5Q9DpGMIBJeI=', 'Content-Length': '0'} 2023-07-29T00:15:01.019 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a527f7f9b1751137-0064c45a04-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:15:01.020 DEBUG:boto:path=/hiiqos-55/e 2023-07-29T00:15:01.020 DEBUG:boto:auth_path=/hiiqos-55/e 2023-07-29T00:15:01.020 DEBUG:boto:Method: PUT 2023-07-29T00:15:01.021 DEBUG:boto:Path: /hiiqos-55/e 2023-07-29T00:15:01.021 DEBUG:boto:Data: 2023-07-29T00:15:01.021 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:15:01.022 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:15:01.022 DEBUG:boto:Port: 8000 2023-07-29T00:15:01.022 DEBUG:boto:Params: {} 2023-07-29T00:15:01.023 DEBUG:boto:Token: None 2023-07-29T00:15:01.023 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:15:01 GMT /hiiqos-55/e 2023-07-29T00:15:01.024 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4egBKpz4HEG+RQEyrOl/R9L3Ph4= 2023-07-29T00:15:01.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:15:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4egBKpz4HEG+RQEyrOl/R9L3Ph4='} 2023-07-29T00:15:01.029 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005e2f734701b60eba-0064c45a04-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:15:01.029 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:15:01.029 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:15:01.030 DEBUG:boto:Method: HEAD 2023-07-29T00:15:01.030 DEBUG:boto:Path: /hiiqos-55/ 2023-07-29T00:15:01.030 DEBUG:boto:Data: 2023-07-29T00:15:01.031 DEBUG:boto:Headers: {} 2023-07-29T00:15:01.031 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:15:01.031 DEBUG:boto:Port: 8000 2023-07-29T00:15:01.032 DEBUG:boto:Params: {} 2023-07-29T00:15:01.032 DEBUG:boto:Token: None 2023-07-29T00:15:01.032 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:15:01 GMT /hiiqos-55/ 2023-07-29T00:15:01.033 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DeuQEaQFwkMkxTy5Q9DpGMIBJeI= 2023-07-29T00:15:01.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DeuQEaQFwkMkxTy5Q9DpGMIBJeI=', 'Content-Length': '0'} 2023-07-29T00:15:01.035 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000023f33a93db49114e-0064c45a04-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:15:01.035 DEBUG:boto:path=/hiiqos-55/f 2023-07-29T00:15:01.035 DEBUG:boto:auth_path=/hiiqos-55/f 2023-07-29T00:15:01.036 DEBUG:boto:Method: PUT 2023-07-29T00:15:01.036 DEBUG:boto:Path: /hiiqos-55/f 2023-07-29T00:15:01.037 DEBUG:boto:Data: 2023-07-29T00:15:01.037 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:15:01.037 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:15:01.037 DEBUG:boto:Port: 8000 2023-07-29T00:15:01.038 DEBUG:boto:Params: {} 2023-07-29T00:15:01.038 DEBUG:boto:Token: None 2023-07-29T00:15:01.039 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:15:01 GMT /hiiqos-55/f 2023-07-29T00:15:01.039 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Q9aDeK6k5VER/MLgN76iEUJj9lk= 2023-07-29T00:15:01.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:15:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q9aDeK6k5VER/MLgN76iEUJj9lk='} 2023-07-29T00:15:01.044 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006084addd4b540420-0064c45a05-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:15:01.045 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:15:01.045 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:15:01.046 DEBUG:boto:Method: HEAD 2023-07-29T00:15:01.046 DEBUG:boto:Path: /hiiqos-55/ 2023-07-29T00:15:01.046 DEBUG:boto:Data: 2023-07-29T00:15:01.047 DEBUG:boto:Headers: {} 2023-07-29T00:15:01.047 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:15:01.047 DEBUG:boto:Port: 8000 2023-07-29T00:15:01.048 DEBUG:boto:Params: {} 2023-07-29T00:15:01.048 DEBUG:boto:Token: None 2023-07-29T00:15:01.048 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:15:01 GMT /hiiqos-55/ 2023-07-29T00:15:01.049 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DeuQEaQFwkMkxTy5Q9DpGMIBJeI= 2023-07-29T00:15:01.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DeuQEaQFwkMkxTy5Q9DpGMIBJeI=', 'Content-Length': '0'} 2023-07-29T00:15:01.051 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009c73f91c92024568-0064c45a05-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:15:01.051 DEBUG:boto:path=/hiiqos-55/g 2023-07-29T00:15:01.052 DEBUG:boto:auth_path=/hiiqos-55/g 2023-07-29T00:15:01.052 DEBUG:boto:Method: PUT 2023-07-29T00:15:01.052 DEBUG:boto:Path: /hiiqos-55/g 2023-07-29T00:15:01.053 DEBUG:boto:Data: 2023-07-29T00:15:01.053 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:15:01.053 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:15:01.054 DEBUG:boto:Port: 8000 2023-07-29T00:15:01.054 DEBUG:boto:Params: {} 2023-07-29T00:15:01.054 DEBUG:boto:Token: None 2023-07-29T00:15:01.055 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:15:01 GMT /hiiqos-55/g 2023-07-29T00:15:01.055 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EN+oMQS9m6l/UexGwXkJzzAA86s= 2023-07-29T00:15:01.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:15:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EN+oMQS9m6l/UexGwXkJzzAA86s='} 2023-07-29T00:15:01.060 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000498c6a4f1cfe79df-0064c45a05-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:15:01.061 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:15:01.061 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:15:01.061 DEBUG:boto:Method: HEAD 2023-07-29T00:15:01.062 DEBUG:boto:Path: /hiiqos-55/ 2023-07-29T00:15:01.062 DEBUG:boto:Data: 2023-07-29T00:15:01.062 DEBUG:boto:Headers: {} 2023-07-29T00:15:01.063 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:15:01.063 DEBUG:boto:Port: 8000 2023-07-29T00:15:01.063 DEBUG:boto:Params: {} 2023-07-29T00:15:01.064 DEBUG:boto:Token: None 2023-07-29T00:15:01.064 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:15:01 GMT /hiiqos-55/ 2023-07-29T00:15:01.064 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DeuQEaQFwkMkxTy5Q9DpGMIBJeI= 2023-07-29T00:15:01.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DeuQEaQFwkMkxTy5Q9DpGMIBJeI=', 'Content-Length': '0'} 2023-07-29T00:15:01.066 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006730b230bf44059b-0064c45a05-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:15:01.067 DEBUG:boto:path=/hiiqos-55/h 2023-07-29T00:15:01.067 DEBUG:boto:auth_path=/hiiqos-55/h 2023-07-29T00:15:01.068 DEBUG:boto:Method: PUT 2023-07-29T00:15:01.068 DEBUG:boto:Path: /hiiqos-55/h 2023-07-29T00:15:01.068 DEBUG:boto:Data: 2023-07-29T00:15:01.069 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:15:01.069 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:15:01.069 DEBUG:boto:Port: 8000 2023-07-29T00:15:01.070 DEBUG:boto:Params: {} 2023-07-29T00:15:01.070 DEBUG:boto:Token: None 2023-07-29T00:15:01.071 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:15:01 GMT /hiiqos-55/h 2023-07-29T00:15:01.071 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YSotu97b1ZrT2gU7tvFJdNWA140= 2023-07-29T00:15:01.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:15:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YSotu97b1ZrT2gU7tvFJdNWA140='} 2023-07-29T00:15:01.120 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b0862846ad667a22-0064c45a05-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:15:01.121 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-55 2023-07-29T00:15:01.122 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:15:01.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:15:01.122 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:15:01.174 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:15:01.175 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:15:01.206 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:15:01.166+0000 7fb023329880 1 waiting to reach latest gen 1, on 0.. 2023-07-29T00:15:06.207 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:15:06.166+0000 7fb023329880 1 waiting to reach latest gen 1, on 0.. 2023-07-29T00:15:11.208 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:15:11.167+0000 7fb023329880 1 waiting to reach latest gen 1, on 0.. 2023-07-29T00:15:16.209 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:15:16.168+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T00:15:16.209 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , ] 2023-07-29T00:15:16.210 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T00:15:17.872 INFO:tasks.rgw.c2.client.1.smithi139.stdout:2023-07-29T00:15:17.831+0000 7f5731a6a640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T00:15:21.210 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:15:21.169+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T00:15:21.210 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , ] 2023-07-29T00:15:21.210 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T00:15:26.211 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:15:26.170+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T00:15:26.211 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , ] 2023-07-29T00:15:26.211 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T00:15:31.212 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:15:31.171+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T00:15:31.212 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , ] 2023-07-29T00:15:31.212 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T00:15:36.213 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:15:36.171+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T00:15:36.213 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , ] 2023-07-29T00:15:36.213 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T00:15:41.213 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:15:41.172+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T00:15:41.214 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , ] 2023-07-29T00:15:41.214 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T00:15:42.834 INFO:tasks.rgw.c2.client.0.smithi139.stdout:2023-07-29T00:15:42.793+0000 7f5f2671b640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T00:15:46.214 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:15:46.173+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T00:15:46.215 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , ] 2023-07-29T00:15:46.215 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T00:15:51.215 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:15:51.173+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T00:15:51.216 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , ] 2023-07-29T00:15:51.216 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T00:15:56.216 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:15:56.174+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T00:15:56.216 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.40.6, ] 2023-07-29T00:15:56.217 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T00:16:01.217 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:16:01.175+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T00:16:01.217 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.40.6, ] 2023-07-29T00:16:01.217 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T00:16:06.218 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:16:06.176+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T00:16:06.218 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.40.6, ] 2023-07-29T00:16:06.218 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T00:16:11.219 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:16:11.177+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T00:16:11.219 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.40.6, ] 2023-07-29T00:16:11.219 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T00:16:16.220 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:16:16.177+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T00:16:16.220 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.40.6, ] 2023-07-29T00:16:16.220 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T00:16:21.221 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:16:21.178+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T00:16:21.221 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.40.6, ] 2023-07-29T00:16:21.221 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T00:16:26.221 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:16:26.179+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T00:16:26.222 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.40.6, ] 2023-07-29T00:16:26.222 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T00:16:31.222 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:16:31.180+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T00:16:31.223 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.40.6, ] 2023-07-29T00:16:31.223 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T00:16:36.223 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:16:36.181+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T00:16:36.224 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.40.6, 00000000002.48.6] 2023-07-29T00:16:36.224 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T00:16:37.059 INFO:tasks.rgw.c1.client.0.smithi101.stdout:2023-07-29T00:16:37.016+0000 7f0081a5e640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T00:16:41.224 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:16:41.181+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T00:16:41.224 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.40.6, 00000000002.48.6] 2023-07-29T00:16:41.225 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T00:16:46.225 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:16:46.182+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T00:16:46.225 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.40.6, 00000000002.48.6] 2023-07-29T00:16:46.225 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T00:16:51.226 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:16:51.183+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T00:16:51.226 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.40.6, 00000000002.48.6] 2023-07-29T00:16:51.226 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T00:16:56.227 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:16:56.184+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T00:16:56.227 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.40.6, 00000000002.48.6] 2023-07-29T00:16:56.227 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T00:17:01.228 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:17:01.184+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T00:17:01.228 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.40.6, 00000000002.48.6] 2023-07-29T00:17:01.229 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T00:17:06.229 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:17:06.185+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T00:17:06.229 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.40.6, 00000000002.48.6] 2023-07-29T00:17:06.229 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T00:17:11.230 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:17:11.186+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T00:17:11.230 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.40.6, 00000000002.48.6] 2023-07-29T00:17:11.230 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T00:17:16.230 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:17:16.187+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T00:17:16.231 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.40.6, 00000000002.48.6] 2023-07-29T00:17:16.231 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T00:17:21.231 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:17:21.188+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T00:17:21.232 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.40.6, 00000000002.48.6] 2023-07-29T00:17:21.232 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T00:17:26.232 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:17:26.188+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T00:17:26.232 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.40.6, 00000000002.48.6] 2023-07-29T00:17:26.233 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T00:17:31.233 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:17:31.189+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T00:17:31.233 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.40.6, 00000000002.48.6] 2023-07-29T00:17:31.233 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T00:17:36.234 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:17:36.190+0000 7fb023329880 1 bucket sync caught up with source: 2023-07-29T00:17:36.234 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T00:17:36.234 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T00:17:36.235 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:17:36.190+0000 7fb023329880 0 bucket checkpoint complete 2023-07-29T00:17:36.251 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-55 2023-07-29T00:17:36.252 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:17:36.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:17:36.253 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:17:36.311 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:17:36.311 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:17:36.347 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:17:36.303+0000 7f4ba0126880 1 waiting for incremental sync to catch up: 2023-07-29T00:17:36.348 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.714.6, , , 00000000001.1347.6, 00000000001.707.6, , , 00000000002.386.6, , 00000000001.1165.6] 2023-07-29T00:17:36.348 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000002.714.6, , , 00000000001.1347.6, 00000000001.707.6, , , 00000000002.386.6, , 00000000002.1169.6] 2023-07-29T00:17:41.349 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:17:41.305+0000 7f4ba0126880 1 bucket sync caught up with source: 2023-07-29T00:17:41.349 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.714.6, , , 00000000001.1347.6, 00000000001.707.6, , , 00000000002.386.6, , 00000000002.1169.6] 2023-07-29T00:17:41.349 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000002.714.6, , , 00000000001.1347.6, 00000000001.707.6, , , 00000000002.386.6, , 00000000002.1169.6] 2023-07-29T00:17:41.349 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:17:41.305+0000 7f4ba0126880 0 bucket checkpoint complete 2023-07-29T00:17:41.366 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-55 zones={a2, a1} 2023-07-29T00:17:41.367 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:41.367 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:41.368 DEBUG:boto:Method: HEAD 2023-07-29T00:17:41.368 DEBUG:boto:Path: /hiiqos-55/ 2023-07-29T00:17:41.368 DEBUG:boto:Data: 2023-07-29T00:17:41.369 DEBUG:boto:Headers: {} 2023-07-29T00:17:41.369 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:41.370 DEBUG:boto:Port: 8001 2023-07-29T00:17:41.370 DEBUG:boto:Params: {} 2023-07-29T00:17:41.371 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-29T00:17:41.371 DEBUG:boto:Token: None 2023-07-29T00:17:41.371 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:41 GMT /hiiqos-55/ 2023-07-29T00:17:41.372 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aKtJjMgBOwNro+aqU8Irrs67zAY= 2023-07-29T00:17:41.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aKtJjMgBOwNro+aqU8Irrs67zAY=', 'Content-Length': '0'} 2023-07-29T00:17:46.379 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c71712ce5b4e363f-0064c45aaa-1663-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:17:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:46.379 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:46.380 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:46.380 DEBUG:boto:Method: HEAD 2023-07-29T00:17:46.381 DEBUG:boto:Path: /hiiqos-55/ 2023-07-29T00:17:46.381 DEBUG:boto:Data: 2023-07-29T00:17:46.381 DEBUG:boto:Headers: {} 2023-07-29T00:17:46.382 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:46.382 DEBUG:boto:Port: 8000 2023-07-29T00:17:46.382 DEBUG:boto:Params: {} 2023-07-29T00:17:46.383 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-29T00:17:46.383 DEBUG:boto:Token: None 2023-07-29T00:17:46.384 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:46 GMT /hiiqos-55/ 2023-07-29T00:17:46.384 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WOE4et53hzWWtjOpYL66yPz25AI= 2023-07-29T00:17:46.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WOE4et53hzWWtjOpYL66yPz25AI=', 'Content-Length': '0'} 2023-07-29T00:17:51.389 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000703a9bf9eec3bfa0-0064c45aaf-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.390 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-29T00:17:51.391 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:51.391 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:51.392 DEBUG:boto:path=/hiiqos-55/?versions&max-keys=999 2023-07-29T00:17:51.392 DEBUG:boto:auth_path=/hiiqos-55/?versions&max-keys=999 2023-07-29T00:17:51.393 DEBUG:boto:Method: GET 2023-07-29T00:17:51.393 DEBUG:boto:Path: /hiiqos-55/?versions&max-keys=999 2023-07-29T00:17:51.394 DEBUG:boto:Data: 2023-07-29T00:17:51.394 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.394 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:51.395 DEBUG:boto:Port: 8001 2023-07-29T00:17:51.395 DEBUG:boto:Params: {} 2023-07-29T00:17:51.396 DEBUG:boto:Token: None 2023-07-29T00:17:51.396 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/?versions 2023-07-29T00:17:51.397 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mJGH+AVFvJUrMB0TOMF+7f9NpsA= 2023-07-29T00:17:51.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mJGH+AVFvJUrMB0TOMF+7f9NpsA=', 'Content-Length': '0'} 2023-07-29T00:17:51.400 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c648f4c0a6589d05-0064c45aaf-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.401 DEBUG:boto:b'hiiqos-55999falseanulltrue2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-07-29T00:15:00.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-07-29T00:15:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-07-29T00:15:01.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-07-29T00:15:01.033Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:51.402 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-07-29T00:17:51.403 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-07-29T00:17:51.404 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-07-29T00:17:51.404 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-07-29T00:17:51.405 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-07-29T00:17:51.405 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-07-29T00:17:51.405 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-07-29T00:17:51.406 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-07-29T00:17:51.406 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-29T00:17:51.406 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:51.407 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:51.407 DEBUG:boto:path=/hiiqos-55/?versions&max-keys=999 2023-07-29T00:17:51.408 DEBUG:boto:auth_path=/hiiqos-55/?versions&max-keys=999 2023-07-29T00:17:51.409 DEBUG:boto:Method: GET 2023-07-29T00:17:51.409 DEBUG:boto:Path: /hiiqos-55/?versions&max-keys=999 2023-07-29T00:17:51.410 DEBUG:boto:Data: 2023-07-29T00:17:51.411 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.412 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:51.413 DEBUG:boto:Port: 8000 2023-07-29T00:17:51.414 DEBUG:boto:Params: {} 2023-07-29T00:17:51.415 DEBUG:boto:Token: None 2023-07-29T00:17:51.415 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/?versions 2023-07-29T00:17:51.416 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mJGH+AVFvJUrMB0TOMF+7f9NpsA= 2023-07-29T00:17:51.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mJGH+AVFvJUrMB0TOMF+7f9NpsA=', 'Content-Length': '0'} 2023-07-29T00:17:51.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000216b07236d99abe-0064c45aaf-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.420 DEBUG:boto:b'hiiqos-55999falseanulltrue2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-07-29T00:15:00.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-07-29T00:15:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-07-29T00:15:01.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-07-29T00:15:01.033Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:51.421 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-07-29T00:17:51.421 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-07-29T00:17:51.421 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-07-29T00:17:51.422 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-07-29T00:17:51.422 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-07-29T00:17:51.422 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-07-29T00:17:51.423 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-07-29T00:17:51.423 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-07-29T00:17:51.423 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:51.424 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:51.424 DEBUG:boto:path=/hiiqos-55/?versions&max-keys=999 2023-07-29T00:17:51.425 DEBUG:boto:auth_path=/hiiqos-55/?versions&max-keys=999 2023-07-29T00:17:51.425 DEBUG:boto:Method: GET 2023-07-29T00:17:51.425 DEBUG:boto:Path: /hiiqos-55/?versions&max-keys=999 2023-07-29T00:17:51.426 DEBUG:boto:Data: 2023-07-29T00:17:51.426 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.427 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:51.427 DEBUG:boto:Port: 8001 2023-07-29T00:17:51.427 DEBUG:boto:Params: {} 2023-07-29T00:17:51.428 DEBUG:boto:Token: None 2023-07-29T00:17:51.428 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/?versions 2023-07-29T00:17:51.429 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mJGH+AVFvJUrMB0TOMF+7f9NpsA= 2023-07-29T00:17:51.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mJGH+AVFvJUrMB0TOMF+7f9NpsA=', 'Content-Length': '0'} 2023-07-29T00:17:51.432 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003d929b53d1df2ea-0064c45aaf-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.432 DEBUG:boto:b'hiiqos-55999falseanulltrue2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-07-29T00:15:00.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-07-29T00:15:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-07-29T00:15:01.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-07-29T00:15:01.033Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:51.433 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:51.434 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:51.434 DEBUG:boto:path=/hiiqos-55/?versions&max-keys=999 2023-07-29T00:17:51.434 DEBUG:boto:auth_path=/hiiqos-55/?versions&max-keys=999 2023-07-29T00:17:51.435 DEBUG:boto:Method: GET 2023-07-29T00:17:51.435 DEBUG:boto:Path: /hiiqos-55/?versions&max-keys=999 2023-07-29T00:17:51.435 DEBUG:boto:Data: 2023-07-29T00:17:51.436 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.436 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:51.436 DEBUG:boto:Port: 8000 2023-07-29T00:17:51.437 DEBUG:boto:Params: {} 2023-07-29T00:17:51.437 DEBUG:boto:Token: None 2023-07-29T00:17:51.438 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/?versions 2023-07-29T00:17:51.438 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mJGH+AVFvJUrMB0TOMF+7f9NpsA= 2023-07-29T00:17:51.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mJGH+AVFvJUrMB0TOMF+7f9NpsA=', 'Content-Length': '0'} 2023-07-29T00:17:51.440 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fac22311d4146e3-0064c45aaf-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.441 DEBUG:boto:b'hiiqos-55999falseanulltrue2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-07-29T00:15:00.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-07-29T00:15:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-07-29T00:15:01.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-07-29T00:15:01.033Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:51.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-07-29T00:17:51.442 DEBUG:boto:path=/hiiqos-55/a 2023-07-29T00:17:51.443 DEBUG:boto:auth_path=/hiiqos-55/a 2023-07-29T00:17:51.443 DEBUG:boto:path=/hiiqos-55/a?versionId=null 2023-07-29T00:17:51.444 DEBUG:boto:auth_path=/hiiqos-55/a?versionId=null 2023-07-29T00:17:51.444 DEBUG:boto:Method: GET 2023-07-29T00:17:51.445 DEBUG:boto:Path: /hiiqos-55/a?versionId=null 2023-07-29T00:17:51.445 DEBUG:boto:Data: 2023-07-29T00:17:51.445 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.446 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:51.446 DEBUG:boto:Port: 8001 2023-07-29T00:17:51.446 DEBUG:boto:Params: {} 2023-07-29T00:17:51.447 DEBUG:boto:Token: None 2023-07-29T00:17:51.447 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/a?versionId=null 2023-07-29T00:17:51.448 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:e6wNrspmnivzuBOKojSzwt7KgT4= 2023-07-29T00:17:51.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:e6wNrspmnivzuBOKojSzwt7KgT4=', 'Content-Length': '0'} 2023-07-29T00:17:51.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ace96c9eee118183-0064c45aaf-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.451 DEBUG:boto:path=/hiiqos-55/a 2023-07-29T00:17:51.452 DEBUG:boto:auth_path=/hiiqos-55/a 2023-07-29T00:17:51.453 DEBUG:boto:path=/hiiqos-55/a?versionId=null 2023-07-29T00:17:51.454 DEBUG:boto:auth_path=/hiiqos-55/a?versionId=null 2023-07-29T00:17:51.455 DEBUG:boto:Method: GET 2023-07-29T00:17:51.456 DEBUG:boto:Path: /hiiqos-55/a?versionId=null 2023-07-29T00:17:51.457 DEBUG:boto:Data: 2023-07-29T00:17:51.458 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.458 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:51.459 DEBUG:boto:Port: 8000 2023-07-29T00:17:51.460 DEBUG:boto:Params: {} 2023-07-29T00:17:51.460 DEBUG:boto:Token: None 2023-07-29T00:17:51.461 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/a?versionId=null 2023-07-29T00:17:51.462 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:e6wNrspmnivzuBOKojSzwt7KgT4= 2023-07-29T00:17:51.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:e6wNrspmnivzuBOKojSzwt7KgT4=', 'Content-Length': '0'} 2023-07-29T00:17:51.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cb4ebeb01072aad-0064c45aaf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.465 DEBUG:boto:path=/hiiqos-55/a 2023-07-29T00:17:51.465 DEBUG:boto:auth_path=/hiiqos-55/a 2023-07-29T00:17:51.466 DEBUG:boto:path=/hiiqos-55/a?versionId=null 2023-07-29T00:17:51.466 DEBUG:boto:auth_path=/hiiqos-55/a?versionId=null 2023-07-29T00:17:51.466 DEBUG:boto:Method: HEAD 2023-07-29T00:17:51.467 DEBUG:boto:Path: /hiiqos-55/a?versionId=null 2023-07-29T00:17:51.467 DEBUG:boto:Data: 2023-07-29T00:17:51.467 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.468 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:51.468 DEBUG:boto:Port: 8001 2023-07-29T00:17:51.468 DEBUG:boto:Params: {} 2023-07-29T00:17:51.469 DEBUG:boto:Token: None 2023-07-29T00:17:51.469 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/a?versionId=null 2023-07-29T00:17:51.470 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2joamIz6RqsEMSyTGSOG0GVi6M8= 2023-07-29T00:17:51.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2joamIz6RqsEMSyTGSOG0GVi6M8=', 'Content-Length': '0'} 2023-07-29T00:17:51.472 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091f6c83479cb1df0-0064c45aaf-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.473 DEBUG:boto:path=/hiiqos-55/a 2023-07-29T00:17:51.473 DEBUG:boto:auth_path=/hiiqos-55/a 2023-07-29T00:17:51.473 DEBUG:boto:path=/hiiqos-55/a?versionId=null 2023-07-29T00:17:51.474 DEBUG:boto:auth_path=/hiiqos-55/a?versionId=null 2023-07-29T00:17:51.474 DEBUG:boto:Method: HEAD 2023-07-29T00:17:51.474 DEBUG:boto:Path: /hiiqos-55/a?versionId=null 2023-07-29T00:17:51.475 DEBUG:boto:Data: 2023-07-29T00:17:51.475 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.475 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:51.476 DEBUG:boto:Port: 8000 2023-07-29T00:17:51.476 DEBUG:boto:Params: {} 2023-07-29T00:17:51.477 DEBUG:boto:Token: None 2023-07-29T00:17:51.477 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/a?versionId=null 2023-07-29T00:17:51.477 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2joamIz6RqsEMSyTGSOG0GVi6M8= 2023-07-29T00:17:51.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2joamIz6RqsEMSyTGSOG0GVi6M8=', 'Content-Length': '0'} 2023-07-29T00:17:51.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d15bdb9685eee43d-0064c45aaf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.480 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-07-29T00:17:51.480 DEBUG:boto:path=/hiiqos-55/a 2023-07-29T00:17:51.481 DEBUG:boto:auth_path=/hiiqos-55/a 2023-07-29T00:17:51.481 DEBUG:boto:path=/hiiqos-55/a?versionId=null 2023-07-29T00:17:51.482 DEBUG:boto:auth_path=/hiiqos-55/a?versionId=null 2023-07-29T00:17:51.482 DEBUG:boto:Method: GET 2023-07-29T00:17:51.482 DEBUG:boto:Path: /hiiqos-55/a?versionId=null 2023-07-29T00:17:51.483 DEBUG:boto:Data: 2023-07-29T00:17:51.483 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.483 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:51.484 DEBUG:boto:Port: 8001 2023-07-29T00:17:51.485 DEBUG:boto:Params: {} 2023-07-29T00:17:51.485 DEBUG:boto:Token: None 2023-07-29T00:17:51.486 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/a?versionId=null 2023-07-29T00:17:51.487 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:e6wNrspmnivzuBOKojSzwt7KgT4= 2023-07-29T00:17:51.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:e6wNrspmnivzuBOKojSzwt7KgT4=', 'Content-Length': '0'} 2023-07-29T00:17:51.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000344f4965b17b2270-0064c45aaf-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.490 DEBUG:boto:path=/hiiqos-55/a 2023-07-29T00:17:51.491 DEBUG:boto:auth_path=/hiiqos-55/a 2023-07-29T00:17:51.491 DEBUG:boto:path=/hiiqos-55/a?versionId=null 2023-07-29T00:17:51.491 DEBUG:boto:auth_path=/hiiqos-55/a?versionId=null 2023-07-29T00:17:51.492 DEBUG:boto:Method: GET 2023-07-29T00:17:51.492 DEBUG:boto:Path: /hiiqos-55/a?versionId=null 2023-07-29T00:17:51.492 DEBUG:boto:Data: 2023-07-29T00:17:51.493 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.493 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:51.493 DEBUG:boto:Port: 8000 2023-07-29T00:17:51.494 DEBUG:boto:Params: {} 2023-07-29T00:17:51.494 DEBUG:boto:Token: None 2023-07-29T00:17:51.495 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/a?versionId=null 2023-07-29T00:17:51.495 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:e6wNrspmnivzuBOKojSzwt7KgT4= 2023-07-29T00:17:51.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:e6wNrspmnivzuBOKojSzwt7KgT4=', 'Content-Length': '0'} 2023-07-29T00:17:51.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7e67b63f9c69051-0064c45aaf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.540 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:51.541 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:51.541 DEBUG:boto:path=/hiiqos-55/?prefix=a 2023-07-29T00:17:51.541 DEBUG:boto:auth_path=/hiiqos-55/?prefix=a 2023-07-29T00:17:51.542 DEBUG:boto:Method: GET 2023-07-29T00:17:51.542 DEBUG:boto:Path: /hiiqos-55/?prefix=a 2023-07-29T00:17:51.543 DEBUG:boto:Data: 2023-07-29T00:17:51.543 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.543 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:51.544 DEBUG:boto:Port: 8001 2023-07-29T00:17:51.544 DEBUG:boto:Params: {} 2023-07-29T00:17:51.544 DEBUG:boto:Token: None 2023-07-29T00:17:51.545 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/ 2023-07-29T00:17:51.545 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ViuMHmbO3wFHX4uFdsobzPFwMFk= 2023-07-29T00:17:51.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ViuMHmbO3wFHX4uFdsobzPFwMFk=', 'Content-Length': '0'} 2023-07-29T00:17:51.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e20a9224bfddb9d-0064c45aaf-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.549 DEBUG:boto:b'hiiqos-55a1000falsea2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:51.549 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:51.550 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:51.550 DEBUG:boto:path=/hiiqos-55/?prefix=a 2023-07-29T00:17:51.550 DEBUG:boto:auth_path=/hiiqos-55/?prefix=a 2023-07-29T00:17:51.551 DEBUG:boto:Method: GET 2023-07-29T00:17:51.551 DEBUG:boto:Path: /hiiqos-55/?prefix=a 2023-07-29T00:17:51.551 DEBUG:boto:Data: 2023-07-29T00:17:51.552 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.552 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:51.553 DEBUG:boto:Port: 8000 2023-07-29T00:17:51.553 DEBUG:boto:Params: {} 2023-07-29T00:17:51.553 DEBUG:boto:Token: None 2023-07-29T00:17:51.554 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/ 2023-07-29T00:17:51.554 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ViuMHmbO3wFHX4uFdsobzPFwMFk= 2023-07-29T00:17:51.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ViuMHmbO3wFHX4uFdsobzPFwMFk=', 'Content-Length': '0'} 2023-07-29T00:17:51.556 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000647c6e0438458878-0064c45aaf-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.557 DEBUG:boto:b'hiiqos-55a1000falsea2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:51.557 DEBUG:boto:path=/hiiqos-55/a 2023-07-29T00:17:51.558 DEBUG:boto:auth_path=/hiiqos-55/a 2023-07-29T00:17:51.558 DEBUG:boto:Method: HEAD 2023-07-29T00:17:51.559 DEBUG:boto:Path: /hiiqos-55/a 2023-07-29T00:17:51.559 DEBUG:boto:Data: 2023-07-29T00:17:51.559 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.560 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:51.560 DEBUG:boto:Port: 8001 2023-07-29T00:17:51.560 DEBUG:boto:Params: {} 2023-07-29T00:17:51.561 DEBUG:boto:Token: None 2023-07-29T00:17:51.561 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/a 2023-07-29T00:17:51.561 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Gazq58+W+/UQCVQssQp5BiMFKHc= 2023-07-29T00:17:51.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Gazq58+W+/UQCVQssQp5BiMFKHc=', 'Content-Length': '0'} 2023-07-29T00:17:51.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000881ff93db5cadb72-0064c45aaf-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.564 DEBUG:boto:path=/hiiqos-55/a 2023-07-29T00:17:51.564 DEBUG:boto:auth_path=/hiiqos-55/a 2023-07-29T00:17:51.565 DEBUG:boto:Method: HEAD 2023-07-29T00:17:51.565 DEBUG:boto:Path: /hiiqos-55/a 2023-07-29T00:17:51.565 DEBUG:boto:Data: 2023-07-29T00:17:51.566 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.566 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:51.566 DEBUG:boto:Port: 8000 2023-07-29T00:17:51.567 DEBUG:boto:Params: {} 2023-07-29T00:17:51.567 DEBUG:boto:Token: None 2023-07-29T00:17:51.567 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/a 2023-07-29T00:17:51.568 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Gazq58+W+/UQCVQssQp5BiMFKHc= 2023-07-29T00:17:51.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Gazq58+W+/UQCVQssQp5BiMFKHc=', 'Content-Length': '0'} 2023-07-29T00:17:51.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e032f095dfd9c3d1-0064c45aaf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-07-29T00:17:51.571 DEBUG:boto:path=/hiiqos-55/a 2023-07-29T00:17:51.571 DEBUG:boto:auth_path=/hiiqos-55/a 2023-07-29T00:17:51.572 DEBUG:boto:Method: GET 2023-07-29T00:17:51.572 DEBUG:boto:Path: /hiiqos-55/a 2023-07-29T00:17:51.573 DEBUG:boto:Data: 2023-07-29T00:17:51.573 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.573 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:51.574 DEBUG:boto:Port: 8001 2023-07-29T00:17:51.574 DEBUG:boto:Params: {} 2023-07-29T00:17:51.574 DEBUG:boto:Token: None 2023-07-29T00:17:51.575 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/a 2023-07-29T00:17:51.575 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EGhkezi8Jc3wXAJ3VxuFv3J+aZM= 2023-07-29T00:17:51.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EGhkezi8Jc3wXAJ3VxuFv3J+aZM=', 'Content-Length': '0'} 2023-07-29T00:17:51.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071d9e4dce91afbe8-0064c45aaf-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.578 DEBUG:boto:path=/hiiqos-55/a 2023-07-29T00:17:51.579 DEBUG:boto:auth_path=/hiiqos-55/a 2023-07-29T00:17:51.579 DEBUG:boto:Method: GET 2023-07-29T00:17:51.579 DEBUG:boto:Path: /hiiqos-55/a 2023-07-29T00:17:51.580 DEBUG:boto:Data: 2023-07-29T00:17:51.580 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.580 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:51.581 DEBUG:boto:Port: 8000 2023-07-29T00:17:51.581 DEBUG:boto:Params: {} 2023-07-29T00:17:51.581 DEBUG:boto:Token: None 2023-07-29T00:17:51.582 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/a 2023-07-29T00:17:51.582 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EGhkezi8Jc3wXAJ3VxuFv3J+aZM= 2023-07-29T00:17:51.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EGhkezi8Jc3wXAJ3VxuFv3J+aZM=', 'Content-Length': '0'} 2023-07-29T00:17:51.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062f239f60daf50d8-0064c45aaf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.585 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:51.585 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:51.586 DEBUG:boto:path=/hiiqos-55/?prefix=a 2023-07-29T00:17:51.586 DEBUG:boto:auth_path=/hiiqos-55/?prefix=a 2023-07-29T00:17:51.586 DEBUG:boto:Method: GET 2023-07-29T00:17:51.587 DEBUG:boto:Path: /hiiqos-55/?prefix=a 2023-07-29T00:17:51.587 DEBUG:boto:Data: 2023-07-29T00:17:51.587 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.588 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:51.588 DEBUG:boto:Port: 8001 2023-07-29T00:17:51.588 DEBUG:boto:Params: {} 2023-07-29T00:17:51.589 DEBUG:boto:Token: None 2023-07-29T00:17:51.589 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/ 2023-07-29T00:17:51.590 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ViuMHmbO3wFHX4uFdsobzPFwMFk= 2023-07-29T00:17:51.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ViuMHmbO3wFHX4uFdsobzPFwMFk=', 'Content-Length': '0'} 2023-07-29T00:17:51.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdbdbe757151fe47-0064c45aaf-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.593 DEBUG:boto:b'hiiqos-55a1000falsea2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:51.594 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:51.594 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:51.595 DEBUG:boto:path=/hiiqos-55/?prefix=a 2023-07-29T00:17:51.595 DEBUG:boto:auth_path=/hiiqos-55/?prefix=a 2023-07-29T00:17:51.595 DEBUG:boto:Method: GET 2023-07-29T00:17:51.596 DEBUG:boto:Path: /hiiqos-55/?prefix=a 2023-07-29T00:17:51.596 DEBUG:boto:Data: 2023-07-29T00:17:51.596 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.597 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:51.597 DEBUG:boto:Port: 8000 2023-07-29T00:17:51.598 DEBUG:boto:Params: {} 2023-07-29T00:17:51.598 DEBUG:boto:Token: None 2023-07-29T00:17:51.598 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/ 2023-07-29T00:17:51.599 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ViuMHmbO3wFHX4uFdsobzPFwMFk= 2023-07-29T00:17:51.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ViuMHmbO3wFHX4uFdsobzPFwMFk=', 'Content-Length': '0'} 2023-07-29T00:17:51.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009aeaf1630171da10-0064c45aaf-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.644 DEBUG:boto:b'hiiqos-55a1000falsea2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:51.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-07-29T00:17:51.645 DEBUG:boto:path=/hiiqos-55/b 2023-07-29T00:17:51.646 DEBUG:boto:auth_path=/hiiqos-55/b 2023-07-29T00:17:51.646 DEBUG:boto:path=/hiiqos-55/b?versionId=null 2023-07-29T00:17:51.646 DEBUG:boto:auth_path=/hiiqos-55/b?versionId=null 2023-07-29T00:17:51.647 DEBUG:boto:Method: GET 2023-07-29T00:17:51.647 DEBUG:boto:Path: /hiiqos-55/b?versionId=null 2023-07-29T00:17:51.648 DEBUG:boto:Data: 2023-07-29T00:17:51.648 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.649 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:51.649 DEBUG:boto:Port: 8001 2023-07-29T00:17:51.649 DEBUG:boto:Params: {} 2023-07-29T00:17:51.650 DEBUG:boto:Token: None 2023-07-29T00:17:51.650 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/b?versionId=null 2023-07-29T00:17:51.651 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zJMcWDg9xOXxim63GHNQSwYIZHI= 2023-07-29T00:17:51.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zJMcWDg9xOXxim63GHNQSwYIZHI=', 'Content-Length': '0'} 2023-07-29T00:17:51.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004de841d55ed49eb9-0064c45aaf-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.653 DEBUG:boto:path=/hiiqos-55/b 2023-07-29T00:17:51.654 DEBUG:boto:auth_path=/hiiqos-55/b 2023-07-29T00:17:51.654 DEBUG:boto:path=/hiiqos-55/b?versionId=null 2023-07-29T00:17:51.655 DEBUG:boto:auth_path=/hiiqos-55/b?versionId=null 2023-07-29T00:17:51.655 DEBUG:boto:Method: GET 2023-07-29T00:17:51.655 DEBUG:boto:Path: /hiiqos-55/b?versionId=null 2023-07-29T00:17:51.656 DEBUG:boto:Data: 2023-07-29T00:17:51.656 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.656 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:51.657 DEBUG:boto:Port: 8000 2023-07-29T00:17:51.657 DEBUG:boto:Params: {} 2023-07-29T00:17:51.658 DEBUG:boto:Token: None 2023-07-29T00:17:51.658 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/b?versionId=null 2023-07-29T00:17:51.658 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zJMcWDg9xOXxim63GHNQSwYIZHI= 2023-07-29T00:17:51.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zJMcWDg9xOXxim63GHNQSwYIZHI=', 'Content-Length': '0'} 2023-07-29T00:17:51.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f767e89f3016ac84-0064c45aaf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.661 DEBUG:boto:path=/hiiqos-55/b 2023-07-29T00:17:51.662 DEBUG:boto:auth_path=/hiiqos-55/b 2023-07-29T00:17:51.662 DEBUG:boto:path=/hiiqos-55/b?versionId=null 2023-07-29T00:17:51.662 DEBUG:boto:auth_path=/hiiqos-55/b?versionId=null 2023-07-29T00:17:51.663 DEBUG:boto:Method: HEAD 2023-07-29T00:17:51.663 DEBUG:boto:Path: /hiiqos-55/b?versionId=null 2023-07-29T00:17:51.663 DEBUG:boto:Data: 2023-07-29T00:17:51.664 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.664 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:51.664 DEBUG:boto:Port: 8001 2023-07-29T00:17:51.665 DEBUG:boto:Params: {} 2023-07-29T00:17:51.665 DEBUG:boto:Token: None 2023-07-29T00:17:51.666 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/b?versionId=null 2023-07-29T00:17:51.666 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wsKKEj/6K/Xm0pDI7fjJ8Sxv2i8= 2023-07-29T00:17:51.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wsKKEj/6K/Xm0pDI7fjJ8Sxv2i8=', 'Content-Length': '0'} 2023-07-29T00:17:51.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fce70c835644a44-0064c45aaf-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.669 DEBUG:boto:path=/hiiqos-55/b 2023-07-29T00:17:51.669 DEBUG:boto:auth_path=/hiiqos-55/b 2023-07-29T00:17:51.669 DEBUG:boto:path=/hiiqos-55/b?versionId=null 2023-07-29T00:17:51.670 DEBUG:boto:auth_path=/hiiqos-55/b?versionId=null 2023-07-29T00:17:51.670 DEBUG:boto:Method: HEAD 2023-07-29T00:17:51.670 DEBUG:boto:Path: /hiiqos-55/b?versionId=null 2023-07-29T00:17:51.671 DEBUG:boto:Data: 2023-07-29T00:17:51.671 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.671 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:51.672 DEBUG:boto:Port: 8000 2023-07-29T00:17:51.672 DEBUG:boto:Params: {} 2023-07-29T00:17:51.672 DEBUG:boto:Token: None 2023-07-29T00:17:51.673 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/b?versionId=null 2023-07-29T00:17:51.673 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wsKKEj/6K/Xm0pDI7fjJ8Sxv2i8= 2023-07-29T00:17:51.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wsKKEj/6K/Xm0pDI7fjJ8Sxv2i8=', 'Content-Length': '0'} 2023-07-29T00:17:51.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fff2452cf575b76-0064c45aaf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-07-29T00:17:51.676 DEBUG:boto:path=/hiiqos-55/b 2023-07-29T00:17:51.676 DEBUG:boto:auth_path=/hiiqos-55/b 2023-07-29T00:17:51.677 DEBUG:boto:path=/hiiqos-55/b?versionId=null 2023-07-29T00:17:51.677 DEBUG:boto:auth_path=/hiiqos-55/b?versionId=null 2023-07-29T00:17:51.678 DEBUG:boto:Method: GET 2023-07-29T00:17:51.678 DEBUG:boto:Path: /hiiqos-55/b?versionId=null 2023-07-29T00:17:51.678 DEBUG:boto:Data: 2023-07-29T00:17:51.679 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.679 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:51.679 DEBUG:boto:Port: 8001 2023-07-29T00:17:51.680 DEBUG:boto:Params: {} 2023-07-29T00:17:51.680 DEBUG:boto:Token: None 2023-07-29T00:17:51.681 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/b?versionId=null 2023-07-29T00:17:51.681 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zJMcWDg9xOXxim63GHNQSwYIZHI= 2023-07-29T00:17:51.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zJMcWDg9xOXxim63GHNQSwYIZHI=', 'Content-Length': '0'} 2023-07-29T00:17:51.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072303957af4a1729-0064c45aaf-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.684 DEBUG:boto:path=/hiiqos-55/b 2023-07-29T00:17:51.684 DEBUG:boto:auth_path=/hiiqos-55/b 2023-07-29T00:17:51.685 DEBUG:boto:path=/hiiqos-55/b?versionId=null 2023-07-29T00:17:51.685 DEBUG:boto:auth_path=/hiiqos-55/b?versionId=null 2023-07-29T00:17:51.685 DEBUG:boto:Method: GET 2023-07-29T00:17:51.686 DEBUG:boto:Path: /hiiqos-55/b?versionId=null 2023-07-29T00:17:51.686 DEBUG:boto:Data: 2023-07-29T00:17:51.687 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.687 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:51.687 DEBUG:boto:Port: 8000 2023-07-29T00:17:51.688 DEBUG:boto:Params: {} 2023-07-29T00:17:51.688 DEBUG:boto:Token: None 2023-07-29T00:17:51.689 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/b?versionId=null 2023-07-29T00:17:51.689 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zJMcWDg9xOXxim63GHNQSwYIZHI= 2023-07-29T00:17:51.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zJMcWDg9xOXxim63GHNQSwYIZHI=', 'Content-Length': '0'} 2023-07-29T00:17:51.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000011d9770f70c9acf-0064c45aaf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.692 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:51.692 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:51.693 DEBUG:boto:path=/hiiqos-55/?prefix=b 2023-07-29T00:17:51.693 DEBUG:boto:auth_path=/hiiqos-55/?prefix=b 2023-07-29T00:17:51.694 DEBUG:boto:Method: GET 2023-07-29T00:17:51.694 DEBUG:boto:Path: /hiiqos-55/?prefix=b 2023-07-29T00:17:51.694 DEBUG:boto:Data: 2023-07-29T00:17:51.695 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.695 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:51.695 DEBUG:boto:Port: 8001 2023-07-29T00:17:51.696 DEBUG:boto:Params: {} 2023-07-29T00:17:51.696 DEBUG:boto:Token: None 2023-07-29T00:17:51.697 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/ 2023-07-29T00:17:51.697 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ViuMHmbO3wFHX4uFdsobzPFwMFk= 2023-07-29T00:17:51.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ViuMHmbO3wFHX4uFdsobzPFwMFk=', 'Content-Length': '0'} 2023-07-29T00:17:51.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000270f530221e8aff0-0064c45aaf-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.701 DEBUG:boto:b'hiiqos-55b1000falseb2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:51.702 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:51.703 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:51.704 DEBUG:boto:path=/hiiqos-55/?prefix=b 2023-07-29T00:17:51.705 DEBUG:boto:auth_path=/hiiqos-55/?prefix=b 2023-07-29T00:17:51.706 DEBUG:boto:Method: GET 2023-07-29T00:17:51.706 DEBUG:boto:Path: /hiiqos-55/?prefix=b 2023-07-29T00:17:51.707 DEBUG:boto:Data: 2023-07-29T00:17:51.708 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.709 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:51.709 DEBUG:boto:Port: 8000 2023-07-29T00:17:51.710 DEBUG:boto:Params: {} 2023-07-29T00:17:51.711 DEBUG:boto:Token: None 2023-07-29T00:17:51.712 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/ 2023-07-29T00:17:51.712 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ViuMHmbO3wFHX4uFdsobzPFwMFk= 2023-07-29T00:17:51.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ViuMHmbO3wFHX4uFdsobzPFwMFk=', 'Content-Length': '0'} 2023-07-29T00:17:51.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077f03034941a8ff1-0064c45aaf-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.715 DEBUG:boto:b'hiiqos-55b1000falseb2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:51.716 DEBUG:boto:path=/hiiqos-55/b 2023-07-29T00:17:51.716 DEBUG:boto:auth_path=/hiiqos-55/b 2023-07-29T00:17:51.716 DEBUG:boto:Method: HEAD 2023-07-29T00:17:51.717 DEBUG:boto:Path: /hiiqos-55/b 2023-07-29T00:17:51.717 DEBUG:boto:Data: 2023-07-29T00:17:51.717 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.718 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:51.718 DEBUG:boto:Port: 8001 2023-07-29T00:17:51.718 DEBUG:boto:Params: {} 2023-07-29T00:17:51.719 DEBUG:boto:Token: None 2023-07-29T00:17:51.719 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/b 2023-07-29T00:17:51.720 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RllwATRheJgeJUxadVoKtPSQuYg= 2023-07-29T00:17:51.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RllwATRheJgeJUxadVoKtPSQuYg=', 'Content-Length': '0'} 2023-07-29T00:17:51.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df660554483b3d90-0064c45aaf-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.723 DEBUG:boto:path=/hiiqos-55/b 2023-07-29T00:17:51.723 DEBUG:boto:auth_path=/hiiqos-55/b 2023-07-29T00:17:51.723 DEBUG:boto:Method: HEAD 2023-07-29T00:17:51.724 DEBUG:boto:Path: /hiiqos-55/b 2023-07-29T00:17:51.724 DEBUG:boto:Data: 2023-07-29T00:17:51.725 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.725 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:51.725 DEBUG:boto:Port: 8000 2023-07-29T00:17:51.726 DEBUG:boto:Params: {} 2023-07-29T00:17:51.726 DEBUG:boto:Token: None 2023-07-29T00:17:51.726 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/b 2023-07-29T00:17:51.727 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RllwATRheJgeJUxadVoKtPSQuYg= 2023-07-29T00:17:51.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RllwATRheJgeJUxadVoKtPSQuYg=', 'Content-Length': '0'} 2023-07-29T00:17:51.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bfec04c540d16c1-0064c45aaf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-07-29T00:17:51.730 DEBUG:boto:path=/hiiqos-55/b 2023-07-29T00:17:51.730 DEBUG:boto:auth_path=/hiiqos-55/b 2023-07-29T00:17:51.730 DEBUG:boto:Method: GET 2023-07-29T00:17:51.731 DEBUG:boto:Path: /hiiqos-55/b 2023-07-29T00:17:51.731 DEBUG:boto:Data: 2023-07-29T00:17:51.731 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.732 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:51.732 DEBUG:boto:Port: 8001 2023-07-29T00:17:51.733 DEBUG:boto:Params: {} 2023-07-29T00:17:51.733 DEBUG:boto:Token: None 2023-07-29T00:17:51.733 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/b 2023-07-29T00:17:51.734 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VMDvXjs1rzQLghcfDyLtGxUs7no= 2023-07-29T00:17:51.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VMDvXjs1rzQLghcfDyLtGxUs7no=', 'Content-Length': '0'} 2023-07-29T00:17:51.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000192749cbe01f146b-0064c45aaf-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.780 DEBUG:boto:path=/hiiqos-55/b 2023-07-29T00:17:51.781 DEBUG:boto:auth_path=/hiiqos-55/b 2023-07-29T00:17:51.781 DEBUG:boto:Method: GET 2023-07-29T00:17:51.782 DEBUG:boto:Path: /hiiqos-55/b 2023-07-29T00:17:51.782 DEBUG:boto:Data: 2023-07-29T00:17:51.782 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.783 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:51.783 DEBUG:boto:Port: 8000 2023-07-29T00:17:51.783 DEBUG:boto:Params: {} 2023-07-29T00:17:51.784 DEBUG:boto:Token: None 2023-07-29T00:17:51.784 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/b 2023-07-29T00:17:51.785 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VMDvXjs1rzQLghcfDyLtGxUs7no= 2023-07-29T00:17:51.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VMDvXjs1rzQLghcfDyLtGxUs7no=', 'Content-Length': '0'} 2023-07-29T00:17:51.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efb1e43051348f85-0064c45aaf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.788 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:51.788 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:51.789 DEBUG:boto:path=/hiiqos-55/?prefix=b 2023-07-29T00:17:51.789 DEBUG:boto:auth_path=/hiiqos-55/?prefix=b 2023-07-29T00:17:51.789 DEBUG:boto:Method: GET 2023-07-29T00:17:51.790 DEBUG:boto:Path: /hiiqos-55/?prefix=b 2023-07-29T00:17:51.790 DEBUG:boto:Data: 2023-07-29T00:17:51.790 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.791 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:51.791 DEBUG:boto:Port: 8001 2023-07-29T00:17:51.791 DEBUG:boto:Params: {} 2023-07-29T00:17:51.792 DEBUG:boto:Token: None 2023-07-29T00:17:51.792 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/ 2023-07-29T00:17:51.793 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ViuMHmbO3wFHX4uFdsobzPFwMFk= 2023-07-29T00:17:51.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ViuMHmbO3wFHX4uFdsobzPFwMFk=', 'Content-Length': '0'} 2023-07-29T00:17:51.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005736aa35aee835aa-0064c45aaf-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.796 DEBUG:boto:b'hiiqos-55b1000falseb2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:51.797 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:51.797 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:51.797 DEBUG:boto:path=/hiiqos-55/?prefix=b 2023-07-29T00:17:51.798 DEBUG:boto:auth_path=/hiiqos-55/?prefix=b 2023-07-29T00:17:51.798 DEBUG:boto:Method: GET 2023-07-29T00:17:51.798 DEBUG:boto:Path: /hiiqos-55/?prefix=b 2023-07-29T00:17:51.799 DEBUG:boto:Data: 2023-07-29T00:17:51.800 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.801 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:51.801 DEBUG:boto:Port: 8000 2023-07-29T00:17:51.802 DEBUG:boto:Params: {} 2023-07-29T00:17:51.803 DEBUG:boto:Token: None 2023-07-29T00:17:51.804 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/ 2023-07-29T00:17:51.805 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ViuMHmbO3wFHX4uFdsobzPFwMFk= 2023-07-29T00:17:51.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ViuMHmbO3wFHX4uFdsobzPFwMFk=', 'Content-Length': '0'} 2023-07-29T00:17:51.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031dfcaba6f6ebfdd-0064c45aaf-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.809 DEBUG:boto:b'hiiqos-55b1000falseb2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:51.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-07-29T00:17:51.811 DEBUG:boto:path=/hiiqos-55/c 2023-07-29T00:17:51.812 DEBUG:boto:auth_path=/hiiqos-55/c 2023-07-29T00:17:51.813 DEBUG:boto:path=/hiiqos-55/c?versionId=null 2023-07-29T00:17:51.814 DEBUG:boto:auth_path=/hiiqos-55/c?versionId=null 2023-07-29T00:17:51.814 DEBUG:boto:Method: GET 2023-07-29T00:17:51.815 DEBUG:boto:Path: /hiiqos-55/c?versionId=null 2023-07-29T00:17:51.816 DEBUG:boto:Data: 2023-07-29T00:17:51.817 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.817 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:51.818 DEBUG:boto:Port: 8001 2023-07-29T00:17:51.819 DEBUG:boto:Params: {} 2023-07-29T00:17:51.820 DEBUG:boto:Token: None 2023-07-29T00:17:51.820 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/c?versionId=null 2023-07-29T00:17:51.821 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YAnGgL8OAhPLvsjllCt2QFxMvls= 2023-07-29T00:17:51.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YAnGgL8OAhPLvsjllCt2QFxMvls=', 'Content-Length': '0'} 2023-07-29T00:17:51.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0bdf7a2cca4c099-0064c45aaf-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.825 DEBUG:boto:path=/hiiqos-55/c 2023-07-29T00:17:51.826 DEBUG:boto:auth_path=/hiiqos-55/c 2023-07-29T00:17:51.827 DEBUG:boto:path=/hiiqos-55/c?versionId=null 2023-07-29T00:17:51.828 DEBUG:boto:auth_path=/hiiqos-55/c?versionId=null 2023-07-29T00:17:51.828 DEBUG:boto:Method: GET 2023-07-29T00:17:51.829 DEBUG:boto:Path: /hiiqos-55/c?versionId=null 2023-07-29T00:17:51.830 DEBUG:boto:Data: 2023-07-29T00:17:51.831 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.832 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:51.832 DEBUG:boto:Port: 8000 2023-07-29T00:17:51.833 DEBUG:boto:Params: {} 2023-07-29T00:17:51.834 DEBUG:boto:Token: None 2023-07-29T00:17:51.835 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/c?versionId=null 2023-07-29T00:17:51.836 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YAnGgL8OAhPLvsjllCt2QFxMvls= 2023-07-29T00:17:51.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YAnGgL8OAhPLvsjllCt2QFxMvls=', 'Content-Length': '0'} 2023-07-29T00:17:51.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ade5ad9acc43ea3-0064c45aaf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.880 DEBUG:boto:path=/hiiqos-55/c 2023-07-29T00:17:51.881 DEBUG:boto:auth_path=/hiiqos-55/c 2023-07-29T00:17:51.882 DEBUG:boto:path=/hiiqos-55/c?versionId=null 2023-07-29T00:17:51.883 DEBUG:boto:auth_path=/hiiqos-55/c?versionId=null 2023-07-29T00:17:51.884 DEBUG:boto:Method: HEAD 2023-07-29T00:17:51.884 DEBUG:boto:Path: /hiiqos-55/c?versionId=null 2023-07-29T00:17:51.885 DEBUG:boto:Data: 2023-07-29T00:17:51.886 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.887 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:51.888 DEBUG:boto:Port: 8001 2023-07-29T00:17:51.888 DEBUG:boto:Params: {} 2023-07-29T00:17:51.889 DEBUG:boto:Token: None 2023-07-29T00:17:51.890 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/c?versionId=null 2023-07-29T00:17:51.891 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZthH3qNkOgcqhxzwANftn3UWglE= 2023-07-29T00:17:51.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZthH3qNkOgcqhxzwANftn3UWglE=', 'Content-Length': '0'} 2023-07-29T00:17:51.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fcaeb35c7056457-0064c45aaf-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.894 DEBUG:boto:path=/hiiqos-55/c 2023-07-29T00:17:51.895 DEBUG:boto:auth_path=/hiiqos-55/c 2023-07-29T00:17:51.895 DEBUG:boto:path=/hiiqos-55/c?versionId=null 2023-07-29T00:17:51.896 DEBUG:boto:auth_path=/hiiqos-55/c?versionId=null 2023-07-29T00:17:51.896 DEBUG:boto:Method: HEAD 2023-07-29T00:17:51.896 DEBUG:boto:Path: /hiiqos-55/c?versionId=null 2023-07-29T00:17:51.897 DEBUG:boto:Data: 2023-07-29T00:17:51.897 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.897 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:51.898 DEBUG:boto:Port: 8000 2023-07-29T00:17:51.898 DEBUG:boto:Params: {} 2023-07-29T00:17:51.898 DEBUG:boto:Token: None 2023-07-29T00:17:51.899 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/c?versionId=null 2023-07-29T00:17:51.899 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZthH3qNkOgcqhxzwANftn3UWglE= 2023-07-29T00:17:51.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZthH3qNkOgcqhxzwANftn3UWglE=', 'Content-Length': '0'} 2023-07-29T00:17:51.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008219af0940008497-0064c45aaf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-07-29T00:17:51.902 DEBUG:boto:path=/hiiqos-55/c 2023-07-29T00:17:51.903 DEBUG:boto:auth_path=/hiiqos-55/c 2023-07-29T00:17:51.903 DEBUG:boto:path=/hiiqos-55/c?versionId=null 2023-07-29T00:17:51.903 DEBUG:boto:auth_path=/hiiqos-55/c?versionId=null 2023-07-29T00:17:51.904 DEBUG:boto:Method: GET 2023-07-29T00:17:51.904 DEBUG:boto:Path: /hiiqos-55/c?versionId=null 2023-07-29T00:17:51.905 DEBUG:boto:Data: 2023-07-29T00:17:51.905 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.905 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:51.906 DEBUG:boto:Port: 8001 2023-07-29T00:17:51.906 DEBUG:boto:Params: {} 2023-07-29T00:17:51.906 DEBUG:boto:Token: None 2023-07-29T00:17:51.907 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/c?versionId=null 2023-07-29T00:17:51.907 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YAnGgL8OAhPLvsjllCt2QFxMvls= 2023-07-29T00:17:51.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YAnGgL8OAhPLvsjllCt2QFxMvls=', 'Content-Length': '0'} 2023-07-29T00:17:51.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be75d7525b4245e6-0064c45aaf-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.910 DEBUG:boto:path=/hiiqos-55/c 2023-07-29T00:17:51.911 DEBUG:boto:auth_path=/hiiqos-55/c 2023-07-29T00:17:51.911 DEBUG:boto:path=/hiiqos-55/c?versionId=null 2023-07-29T00:17:51.911 DEBUG:boto:auth_path=/hiiqos-55/c?versionId=null 2023-07-29T00:17:51.912 DEBUG:boto:Method: GET 2023-07-29T00:17:51.912 DEBUG:boto:Path: /hiiqos-55/c?versionId=null 2023-07-29T00:17:51.912 DEBUG:boto:Data: 2023-07-29T00:17:51.913 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.913 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:51.913 DEBUG:boto:Port: 8000 2023-07-29T00:17:51.914 DEBUG:boto:Params: {} 2023-07-29T00:17:51.914 DEBUG:boto:Token: None 2023-07-29T00:17:51.915 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/c?versionId=null 2023-07-29T00:17:51.915 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YAnGgL8OAhPLvsjllCt2QFxMvls= 2023-07-29T00:17:51.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YAnGgL8OAhPLvsjllCt2QFxMvls=', 'Content-Length': '0'} 2023-07-29T00:17:51.917 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060dcdc8c04b51011-0064c45aaf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.918 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:51.918 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:51.919 DEBUG:boto:path=/hiiqos-55/?prefix=c 2023-07-29T00:17:51.919 DEBUG:boto:auth_path=/hiiqos-55/?prefix=c 2023-07-29T00:17:51.919 DEBUG:boto:Method: GET 2023-07-29T00:17:51.920 DEBUG:boto:Path: /hiiqos-55/?prefix=c 2023-07-29T00:17:51.920 DEBUG:boto:Data: 2023-07-29T00:17:51.920 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.921 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:51.921 DEBUG:boto:Port: 8001 2023-07-29T00:17:51.921 DEBUG:boto:Params: {} 2023-07-29T00:17:51.922 DEBUG:boto:Token: None 2023-07-29T00:17:51.922 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/ 2023-07-29T00:17:51.923 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ViuMHmbO3wFHX4uFdsobzPFwMFk= 2023-07-29T00:17:51.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ViuMHmbO3wFHX4uFdsobzPFwMFk=', 'Content-Length': '0'} 2023-07-29T00:17:51.926 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afa3d1db99c0bd4c-0064c45aaf-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.926 DEBUG:boto:b'hiiqos-55c1000falsec2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:51.927 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:51.927 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:51.928 DEBUG:boto:path=/hiiqos-55/?prefix=c 2023-07-29T00:17:51.928 DEBUG:boto:auth_path=/hiiqos-55/?prefix=c 2023-07-29T00:17:51.928 DEBUG:boto:Method: GET 2023-07-29T00:17:51.929 DEBUG:boto:Path: /hiiqos-55/?prefix=c 2023-07-29T00:17:51.929 DEBUG:boto:Data: 2023-07-29T00:17:51.930 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.930 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:51.930 DEBUG:boto:Port: 8000 2023-07-29T00:17:51.931 DEBUG:boto:Params: {} 2023-07-29T00:17:51.931 DEBUG:boto:Token: None 2023-07-29T00:17:51.932 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/ 2023-07-29T00:17:51.932 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ViuMHmbO3wFHX4uFdsobzPFwMFk= 2023-07-29T00:17:51.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ViuMHmbO3wFHX4uFdsobzPFwMFk=', 'Content-Length': '0'} 2023-07-29T00:17:51.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000177dd346ad7cc145-0064c45aaf-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.935 DEBUG:boto:b'hiiqos-55c1000falsec2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:51.935 DEBUG:boto:path=/hiiqos-55/c 2023-07-29T00:17:51.936 DEBUG:boto:auth_path=/hiiqos-55/c 2023-07-29T00:17:51.936 DEBUG:boto:Method: HEAD 2023-07-29T00:17:51.937 DEBUG:boto:Path: /hiiqos-55/c 2023-07-29T00:17:51.937 DEBUG:boto:Data: 2023-07-29T00:17:51.937 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.938 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:51.938 DEBUG:boto:Port: 8001 2023-07-29T00:17:51.938 DEBUG:boto:Params: {} 2023-07-29T00:17:51.939 DEBUG:boto:Token: None 2023-07-29T00:17:51.939 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/c 2023-07-29T00:17:51.940 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SBwl7ICkhbIsIF4nZ0KaBwypOvY= 2023-07-29T00:17:51.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SBwl7ICkhbIsIF4nZ0KaBwypOvY=', 'Content-Length': '0'} 2023-07-29T00:17:51.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e101e0c38092bbf2-0064c45aaf-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.942 DEBUG:boto:path=/hiiqos-55/c 2023-07-29T00:17:51.943 DEBUG:boto:auth_path=/hiiqos-55/c 2023-07-29T00:17:51.943 DEBUG:boto:Method: HEAD 2023-07-29T00:17:51.944 DEBUG:boto:Path: /hiiqos-55/c 2023-07-29T00:17:51.944 DEBUG:boto:Data: 2023-07-29T00:17:51.944 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.945 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:51.945 DEBUG:boto:Port: 8000 2023-07-29T00:17:51.945 DEBUG:boto:Params: {} 2023-07-29T00:17:51.946 DEBUG:boto:Token: None 2023-07-29T00:17:51.946 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/c 2023-07-29T00:17:51.947 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SBwl7ICkhbIsIF4nZ0KaBwypOvY= 2023-07-29T00:17:51.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SBwl7ICkhbIsIF4nZ0KaBwypOvY=', 'Content-Length': '0'} 2023-07-29T00:17:51.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b882c32e3fb00164-0064c45aaf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-07-29T00:17:51.950 DEBUG:boto:path=/hiiqos-55/c 2023-07-29T00:17:51.950 DEBUG:boto:auth_path=/hiiqos-55/c 2023-07-29T00:17:51.950 DEBUG:boto:Method: GET 2023-07-29T00:17:51.951 DEBUG:boto:Path: /hiiqos-55/c 2023-07-29T00:17:51.951 DEBUG:boto:Data: 2023-07-29T00:17:51.951 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.952 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:51.952 DEBUG:boto:Port: 8001 2023-07-29T00:17:51.953 DEBUG:boto:Params: {} 2023-07-29T00:17:51.953 DEBUG:boto:Token: None 2023-07-29T00:17:51.954 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/c 2023-07-29T00:17:51.954 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I/IkE5WOh5yt9p/ptd3zOOL5dA8= 2023-07-29T00:17:51.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I/IkE5WOh5yt9p/ptd3zOOL5dA8=', 'Content-Length': '0'} 2023-07-29T00:17:51.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007564ab6c72d456d-0064c45aaf-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:51.957 DEBUG:boto:path=/hiiqos-55/c 2023-07-29T00:17:51.957 DEBUG:boto:auth_path=/hiiqos-55/c 2023-07-29T00:17:51.958 DEBUG:boto:Method: GET 2023-07-29T00:17:51.958 DEBUG:boto:Path: /hiiqos-55/c 2023-07-29T00:17:51.958 DEBUG:boto:Data: 2023-07-29T00:17:51.959 DEBUG:boto:Headers: {} 2023-07-29T00:17:51.959 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:51.960 DEBUG:boto:Port: 8000 2023-07-29T00:17:51.960 DEBUG:boto:Params: {} 2023-07-29T00:17:51.960 DEBUG:boto:Token: None 2023-07-29T00:17:51.961 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:51 GMT /hiiqos-55/c 2023-07-29T00:17:51.961 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I/IkE5WOh5yt9p/ptd3zOOL5dA8= 2023-07-29T00:17:51.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I/IkE5WOh5yt9p/ptd3zOOL5dA8=', 'Content-Length': '0'} 2023-07-29T00:17:51.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042120bc7553f86e3-0064c45aaf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.004 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:52.005 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:52.005 DEBUG:boto:path=/hiiqos-55/?prefix=c 2023-07-29T00:17:52.005 DEBUG:boto:auth_path=/hiiqos-55/?prefix=c 2023-07-29T00:17:52.006 DEBUG:boto:Method: GET 2023-07-29T00:17:52.006 DEBUG:boto:Path: /hiiqos-55/?prefix=c 2023-07-29T00:17:52.007 DEBUG:boto:Data: 2023-07-29T00:17:52.007 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.007 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:52.008 DEBUG:boto:Port: 8001 2023-07-29T00:17:52.008 DEBUG:boto:Params: {} 2023-07-29T00:17:52.009 DEBUG:boto:Token: None 2023-07-29T00:17:52.009 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/ 2023-07-29T00:17:52.009 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T00:17:52.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T00:17:52.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d4f19e5531f42bc-0064c45aaf-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.014 DEBUG:boto:b'hiiqos-55c1000falsec2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:52.014 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:52.015 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:52.015 DEBUG:boto:path=/hiiqos-55/?prefix=c 2023-07-29T00:17:52.016 DEBUG:boto:auth_path=/hiiqos-55/?prefix=c 2023-07-29T00:17:52.016 DEBUG:boto:Method: GET 2023-07-29T00:17:52.017 DEBUG:boto:Path: /hiiqos-55/?prefix=c 2023-07-29T00:17:52.017 DEBUG:boto:Data: 2023-07-29T00:17:52.017 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.018 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:52.018 DEBUG:boto:Port: 8000 2023-07-29T00:17:52.018 DEBUG:boto:Params: {} 2023-07-29T00:17:52.019 DEBUG:boto:Token: None 2023-07-29T00:17:52.019 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/ 2023-07-29T00:17:52.020 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T00:17:52.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T00:17:52.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa290e70b24479bf-0064c45aaf-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.023 DEBUG:boto:b'hiiqos-55c1000falsec2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:52.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-07-29T00:17:52.024 DEBUG:boto:path=/hiiqos-55/d 2023-07-29T00:17:52.025 DEBUG:boto:auth_path=/hiiqos-55/d 2023-07-29T00:17:52.025 DEBUG:boto:path=/hiiqos-55/d?versionId=null 2023-07-29T00:17:52.025 DEBUG:boto:auth_path=/hiiqos-55/d?versionId=null 2023-07-29T00:17:52.026 DEBUG:boto:Method: GET 2023-07-29T00:17:52.027 DEBUG:boto:Path: /hiiqos-55/d?versionId=null 2023-07-29T00:17:52.027 DEBUG:boto:Data: 2023-07-29T00:17:52.028 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.029 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:52.030 DEBUG:boto:Port: 8001 2023-07-29T00:17:52.031 DEBUG:boto:Params: {} 2023-07-29T00:17:52.032 DEBUG:boto:Token: None 2023-07-29T00:17:52.032 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/d?versionId=null 2023-07-29T00:17:52.033 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TlZH8Pn8FdUzbwrkh/xPyg3TCgs= 2023-07-29T00:17:52.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TlZH8Pn8FdUzbwrkh/xPyg3TCgs=', 'Content-Length': '0'} 2023-07-29T00:17:52.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9268d239f60b174-0064c45aaf-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.076 DEBUG:boto:path=/hiiqos-55/d 2023-07-29T00:17:52.077 DEBUG:boto:auth_path=/hiiqos-55/d 2023-07-29T00:17:52.077 DEBUG:boto:path=/hiiqos-55/d?versionId=null 2023-07-29T00:17:52.078 DEBUG:boto:auth_path=/hiiqos-55/d?versionId=null 2023-07-29T00:17:52.078 DEBUG:boto:Method: GET 2023-07-29T00:17:52.078 DEBUG:boto:Path: /hiiqos-55/d?versionId=null 2023-07-29T00:17:52.079 DEBUG:boto:Data: 2023-07-29T00:17:52.079 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.080 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:52.080 DEBUG:boto:Port: 8000 2023-07-29T00:17:52.080 DEBUG:boto:Params: {} 2023-07-29T00:17:52.081 DEBUG:boto:Token: None 2023-07-29T00:17:52.081 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/d?versionId=null 2023-07-29T00:17:52.082 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TlZH8Pn8FdUzbwrkh/xPyg3TCgs= 2023-07-29T00:17:52.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TlZH8Pn8FdUzbwrkh/xPyg3TCgs=', 'Content-Length': '0'} 2023-07-29T00:17:52.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d972603c6251719-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.085 DEBUG:boto:path=/hiiqos-55/d 2023-07-29T00:17:52.086 DEBUG:boto:auth_path=/hiiqos-55/d 2023-07-29T00:17:52.086 DEBUG:boto:path=/hiiqos-55/d?versionId=null 2023-07-29T00:17:52.086 DEBUG:boto:auth_path=/hiiqos-55/d?versionId=null 2023-07-29T00:17:52.087 DEBUG:boto:Method: HEAD 2023-07-29T00:17:52.087 DEBUG:boto:Path: /hiiqos-55/d?versionId=null 2023-07-29T00:17:52.087 DEBUG:boto:Data: 2023-07-29T00:17:52.088 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.088 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:52.088 DEBUG:boto:Port: 8001 2023-07-29T00:17:52.089 DEBUG:boto:Params: {} 2023-07-29T00:17:52.089 DEBUG:boto:Token: None 2023-07-29T00:17:52.090 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/d?versionId=null 2023-07-29T00:17:52.090 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lN85Q9cM22ERpzAKcYPyYmsxToc= 2023-07-29T00:17:52.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lN85Q9cM22ERpzAKcYPyYmsxToc=', 'Content-Length': '0'} 2023-07-29T00:17:52.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da2e5c45caa1ee85-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.093 DEBUG:boto:path=/hiiqos-55/d 2023-07-29T00:17:52.093 DEBUG:boto:auth_path=/hiiqos-55/d 2023-07-29T00:17:52.093 DEBUG:boto:path=/hiiqos-55/d?versionId=null 2023-07-29T00:17:52.094 DEBUG:boto:auth_path=/hiiqos-55/d?versionId=null 2023-07-29T00:17:52.094 DEBUG:boto:Method: HEAD 2023-07-29T00:17:52.094 DEBUG:boto:Path: /hiiqos-55/d?versionId=null 2023-07-29T00:17:52.095 DEBUG:boto:Data: 2023-07-29T00:17:52.095 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.095 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:52.096 DEBUG:boto:Port: 8000 2023-07-29T00:17:52.096 DEBUG:boto:Params: {} 2023-07-29T00:17:52.096 DEBUG:boto:Token: None 2023-07-29T00:17:52.097 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/d?versionId=null 2023-07-29T00:17:52.097 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lN85Q9cM22ERpzAKcYPyYmsxToc= 2023-07-29T00:17:52.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lN85Q9cM22ERpzAKcYPyYmsxToc=', 'Content-Length': '0'} 2023-07-29T00:17:52.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070203ba4e7d867f3-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-07-29T00:17:52.101 DEBUG:boto:path=/hiiqos-55/d 2023-07-29T00:17:52.101 DEBUG:boto:auth_path=/hiiqos-55/d 2023-07-29T00:17:52.101 DEBUG:boto:path=/hiiqos-55/d?versionId=null 2023-07-29T00:17:52.102 DEBUG:boto:auth_path=/hiiqos-55/d?versionId=null 2023-07-29T00:17:52.102 DEBUG:boto:Method: GET 2023-07-29T00:17:52.102 DEBUG:boto:Path: /hiiqos-55/d?versionId=null 2023-07-29T00:17:52.103 DEBUG:boto:Data: 2023-07-29T00:17:52.103 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.103 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:52.104 DEBUG:boto:Port: 8001 2023-07-29T00:17:52.104 DEBUG:boto:Params: {} 2023-07-29T00:17:52.105 DEBUG:boto:Token: None 2023-07-29T00:17:52.105 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/d?versionId=null 2023-07-29T00:17:52.105 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TlZH8Pn8FdUzbwrkh/xPyg3TCgs= 2023-07-29T00:17:52.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TlZH8Pn8FdUzbwrkh/xPyg3TCgs=', 'Content-Length': '0'} 2023-07-29T00:17:52.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c846ee8e30ba607-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.109 DEBUG:boto:path=/hiiqos-55/d 2023-07-29T00:17:52.109 DEBUG:boto:auth_path=/hiiqos-55/d 2023-07-29T00:17:52.109 DEBUG:boto:path=/hiiqos-55/d?versionId=null 2023-07-29T00:17:52.110 DEBUG:boto:auth_path=/hiiqos-55/d?versionId=null 2023-07-29T00:17:52.110 DEBUG:boto:Method: GET 2023-07-29T00:17:52.110 DEBUG:boto:Path: /hiiqos-55/d?versionId=null 2023-07-29T00:17:52.111 DEBUG:boto:Data: 2023-07-29T00:17:52.111 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.111 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:52.112 DEBUG:boto:Port: 8000 2023-07-29T00:17:52.112 DEBUG:boto:Params: {} 2023-07-29T00:17:52.113 DEBUG:boto:Token: None 2023-07-29T00:17:52.113 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/d?versionId=null 2023-07-29T00:17:52.113 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TlZH8Pn8FdUzbwrkh/xPyg3TCgs= 2023-07-29T00:17:52.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TlZH8Pn8FdUzbwrkh/xPyg3TCgs=', 'Content-Length': '0'} 2023-07-29T00:17:52.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000432769ff14a36c30-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.116 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:52.117 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:52.117 DEBUG:boto:path=/hiiqos-55/?prefix=d 2023-07-29T00:17:52.118 DEBUG:boto:auth_path=/hiiqos-55/?prefix=d 2023-07-29T00:17:52.118 DEBUG:boto:Method: GET 2023-07-29T00:17:52.118 DEBUG:boto:Path: /hiiqos-55/?prefix=d 2023-07-29T00:17:52.119 DEBUG:boto:Data: 2023-07-29T00:17:52.119 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.119 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:52.120 DEBUG:boto:Port: 8001 2023-07-29T00:17:52.120 DEBUG:boto:Params: {} 2023-07-29T00:17:52.120 DEBUG:boto:Token: None 2023-07-29T00:17:52.121 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/ 2023-07-29T00:17:52.121 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T00:17:52.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T00:17:52.124 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000249a28b1f763824b-0064c45ab0-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.125 DEBUG:boto:b'hiiqos-55d1000falsed2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:52.125 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:52.126 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:52.126 DEBUG:boto:path=/hiiqos-55/?prefix=d 2023-07-29T00:17:52.127 DEBUG:boto:auth_path=/hiiqos-55/?prefix=d 2023-07-29T00:17:52.127 DEBUG:boto:Method: GET 2023-07-29T00:17:52.127 DEBUG:boto:Path: /hiiqos-55/?prefix=d 2023-07-29T00:17:52.128 DEBUG:boto:Data: 2023-07-29T00:17:52.128 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.128 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:52.129 DEBUG:boto:Port: 8000 2023-07-29T00:17:52.129 DEBUG:boto:Params: {} 2023-07-29T00:17:52.130 DEBUG:boto:Token: None 2023-07-29T00:17:52.130 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/ 2023-07-29T00:17:52.130 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T00:17:52.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T00:17:52.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c4c89f5d8b921c9-0064c45ab0-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.133 DEBUG:boto:b'hiiqos-55d1000falsed2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:52.134 DEBUG:boto:path=/hiiqos-55/d 2023-07-29T00:17:52.134 DEBUG:boto:auth_path=/hiiqos-55/d 2023-07-29T00:17:52.135 DEBUG:boto:Method: HEAD 2023-07-29T00:17:52.135 DEBUG:boto:Path: /hiiqos-55/d 2023-07-29T00:17:52.135 DEBUG:boto:Data: 2023-07-29T00:17:52.136 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.136 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:52.136 DEBUG:boto:Port: 8001 2023-07-29T00:17:52.137 DEBUG:boto:Params: {} 2023-07-29T00:17:52.137 DEBUG:boto:Token: None 2023-07-29T00:17:52.138 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/d 2023-07-29T00:17:52.138 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ImAhZGRvKKXv+DmADCWJaUW6tgo= 2023-07-29T00:17:52.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ImAhZGRvKKXv+DmADCWJaUW6tgo=', 'Content-Length': '0'} 2023-07-29T00:17:52.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011f0caa0e74b4997-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.141 DEBUG:boto:path=/hiiqos-55/d 2023-07-29T00:17:52.141 DEBUG:boto:auth_path=/hiiqos-55/d 2023-07-29T00:17:52.142 DEBUG:boto:Method: HEAD 2023-07-29T00:17:52.142 DEBUG:boto:Path: /hiiqos-55/d 2023-07-29T00:17:52.142 DEBUG:boto:Data: 2023-07-29T00:17:52.143 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.143 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:52.143 DEBUG:boto:Port: 8000 2023-07-29T00:17:52.144 DEBUG:boto:Params: {} 2023-07-29T00:17:52.144 DEBUG:boto:Token: None 2023-07-29T00:17:52.145 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/d 2023-07-29T00:17:52.145 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ImAhZGRvKKXv+DmADCWJaUW6tgo= 2023-07-29T00:17:52.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ImAhZGRvKKXv+DmADCWJaUW6tgo=', 'Content-Length': '0'} 2023-07-29T00:17:52.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c05c10a737259b12-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-07-29T00:17:52.148 DEBUG:boto:path=/hiiqos-55/d 2023-07-29T00:17:52.149 DEBUG:boto:auth_path=/hiiqos-55/d 2023-07-29T00:17:52.149 DEBUG:boto:Method: GET 2023-07-29T00:17:52.149 DEBUG:boto:Path: /hiiqos-55/d 2023-07-29T00:17:52.150 DEBUG:boto:Data: 2023-07-29T00:17:52.150 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.150 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:52.151 DEBUG:boto:Port: 8001 2023-07-29T00:17:52.151 DEBUG:boto:Params: {} 2023-07-29T00:17:52.152 DEBUG:boto:Token: None 2023-07-29T00:17:52.152 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/d 2023-07-29T00:17:52.152 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HT+lbcN3VtKadl92wINcp5Z9skk= 2023-07-29T00:17:52.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HT+lbcN3VtKadl92wINcp5Z9skk=', 'Content-Length': '0'} 2023-07-29T00:17:52.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d59ee5858f6514b2-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.200 DEBUG:boto:path=/hiiqos-55/d 2023-07-29T00:17:52.201 DEBUG:boto:auth_path=/hiiqos-55/d 2023-07-29T00:17:52.201 DEBUG:boto:Method: GET 2023-07-29T00:17:52.202 DEBUG:boto:Path: /hiiqos-55/d 2023-07-29T00:17:52.202 DEBUG:boto:Data: 2023-07-29T00:17:52.202 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.203 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:52.203 DEBUG:boto:Port: 8000 2023-07-29T00:17:52.203 DEBUG:boto:Params: {} 2023-07-29T00:17:52.204 DEBUG:boto:Token: None 2023-07-29T00:17:52.204 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/d 2023-07-29T00:17:52.205 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HT+lbcN3VtKadl92wINcp5Z9skk= 2023-07-29T00:17:52.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HT+lbcN3VtKadl92wINcp5Z9skk=', 'Content-Length': '0'} 2023-07-29T00:17:52.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af0db3789ccf4b14-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.208 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:52.208 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:52.209 DEBUG:boto:path=/hiiqos-55/?prefix=d 2023-07-29T00:17:52.209 DEBUG:boto:auth_path=/hiiqos-55/?prefix=d 2023-07-29T00:17:52.209 DEBUG:boto:Method: GET 2023-07-29T00:17:52.210 DEBUG:boto:Path: /hiiqos-55/?prefix=d 2023-07-29T00:17:52.210 DEBUG:boto:Data: 2023-07-29T00:17:52.210 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.211 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:52.211 DEBUG:boto:Port: 8001 2023-07-29T00:17:52.212 DEBUG:boto:Params: {} 2023-07-29T00:17:52.212 DEBUG:boto:Token: None 2023-07-29T00:17:52.212 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/ 2023-07-29T00:17:52.213 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T00:17:52.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T00:17:52.216 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a2080980f354d29-0064c45ab0-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.216 DEBUG:boto:b'hiiqos-55d1000falsed2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:52.217 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:52.217 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:52.218 DEBUG:boto:path=/hiiqos-55/?prefix=d 2023-07-29T00:17:52.218 DEBUG:boto:auth_path=/hiiqos-55/?prefix=d 2023-07-29T00:17:52.218 DEBUG:boto:Method: GET 2023-07-29T00:17:52.219 DEBUG:boto:Path: /hiiqos-55/?prefix=d 2023-07-29T00:17:52.219 DEBUG:boto:Data: 2023-07-29T00:17:52.219 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.220 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:52.220 DEBUG:boto:Port: 8000 2023-07-29T00:17:52.221 DEBUG:boto:Params: {} 2023-07-29T00:17:52.221 DEBUG:boto:Token: None 2023-07-29T00:17:52.221 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/ 2023-07-29T00:17:52.222 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T00:17:52.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T00:17:52.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012ee2600564992b6-0064c45ab0-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.225 DEBUG:boto:b'hiiqos-55d1000falsed2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:52.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-07-29T00:17:52.225 DEBUG:boto:path=/hiiqos-55/e 2023-07-29T00:17:52.226 DEBUG:boto:auth_path=/hiiqos-55/e 2023-07-29T00:17:52.226 DEBUG:boto:path=/hiiqos-55/e?versionId=null 2023-07-29T00:17:52.227 DEBUG:boto:auth_path=/hiiqos-55/e?versionId=null 2023-07-29T00:17:52.227 DEBUG:boto:Method: GET 2023-07-29T00:17:52.227 DEBUG:boto:Path: /hiiqos-55/e?versionId=null 2023-07-29T00:17:52.228 DEBUG:boto:Data: 2023-07-29T00:17:52.228 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.229 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:52.229 DEBUG:boto:Port: 8001 2023-07-29T00:17:52.229 DEBUG:boto:Params: {} 2023-07-29T00:17:52.230 DEBUG:boto:Token: None 2023-07-29T00:17:52.230 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/e?versionId=null 2023-07-29T00:17:52.231 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:N3uBRkJaALjdWYqGPneTe3xR1gw= 2023-07-29T00:17:52.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:N3uBRkJaALjdWYqGPneTe3xR1gw=', 'Content-Length': '0'} 2023-07-29T00:17:52.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d26d4a9040c2e79a-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.234 DEBUG:boto:path=/hiiqos-55/e 2023-07-29T00:17:52.234 DEBUG:boto:auth_path=/hiiqos-55/e 2023-07-29T00:17:52.234 DEBUG:boto:path=/hiiqos-55/e?versionId=null 2023-07-29T00:17:52.235 DEBUG:boto:auth_path=/hiiqos-55/e?versionId=null 2023-07-29T00:17:52.235 DEBUG:boto:Method: GET 2023-07-29T00:17:52.235 DEBUG:boto:Path: /hiiqos-55/e?versionId=null 2023-07-29T00:17:52.236 DEBUG:boto:Data: 2023-07-29T00:17:52.236 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.237 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:52.237 DEBUG:boto:Port: 8000 2023-07-29T00:17:52.237 DEBUG:boto:Params: {} 2023-07-29T00:17:52.238 DEBUG:boto:Token: None 2023-07-29T00:17:52.238 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/e?versionId=null 2023-07-29T00:17:52.239 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:N3uBRkJaALjdWYqGPneTe3xR1gw= 2023-07-29T00:17:52.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:N3uBRkJaALjdWYqGPneTe3xR1gw=', 'Content-Length': '0'} 2023-07-29T00:17:52.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004373202868bd0f57-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.242 DEBUG:boto:path=/hiiqos-55/e 2023-07-29T00:17:52.242 DEBUG:boto:auth_path=/hiiqos-55/e 2023-07-29T00:17:52.242 DEBUG:boto:path=/hiiqos-55/e?versionId=null 2023-07-29T00:17:52.243 DEBUG:boto:auth_path=/hiiqos-55/e?versionId=null 2023-07-29T00:17:52.243 DEBUG:boto:Method: HEAD 2023-07-29T00:17:52.243 DEBUG:boto:Path: /hiiqos-55/e?versionId=null 2023-07-29T00:17:52.244 DEBUG:boto:Data: 2023-07-29T00:17:52.244 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.244 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:52.247 DEBUG:boto:Port: 8001 2023-07-29T00:17:52.248 DEBUG:boto:Params: {} 2023-07-29T00:17:52.248 DEBUG:boto:Token: None 2023-07-29T00:17:52.249 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/e?versionId=null 2023-07-29T00:17:52.249 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gEvnc1UMMCN8fiNCcgF3RYoBzrM= 2023-07-29T00:17:52.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gEvnc1UMMCN8fiNCcgF3RYoBzrM=', 'Content-Length': '0'} 2023-07-29T00:17:52.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b5435277d8bf831-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.252 DEBUG:boto:path=/hiiqos-55/e 2023-07-29T00:17:52.252 DEBUG:boto:auth_path=/hiiqos-55/e 2023-07-29T00:17:52.252 DEBUG:boto:path=/hiiqos-55/e?versionId=null 2023-07-29T00:17:52.253 DEBUG:boto:auth_path=/hiiqos-55/e?versionId=null 2023-07-29T00:17:52.254 DEBUG:boto:Method: HEAD 2023-07-29T00:17:52.254 DEBUG:boto:Path: /hiiqos-55/e?versionId=null 2023-07-29T00:17:52.255 DEBUG:boto:Data: 2023-07-29T00:17:52.255 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.256 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:52.256 DEBUG:boto:Port: 8000 2023-07-29T00:17:52.256 DEBUG:boto:Params: {} 2023-07-29T00:17:52.257 DEBUG:boto:Token: None 2023-07-29T00:17:52.258 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/e?versionId=null 2023-07-29T00:17:52.258 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gEvnc1UMMCN8fiNCcgF3RYoBzrM= 2023-07-29T00:17:52.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gEvnc1UMMCN8fiNCcgF3RYoBzrM=', 'Content-Length': '0'} 2023-07-29T00:17:52.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083475c4476da0b80-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-07-29T00:17:52.262 DEBUG:boto:path=/hiiqos-55/e 2023-07-29T00:17:52.263 DEBUG:boto:auth_path=/hiiqos-55/e 2023-07-29T00:17:52.264 DEBUG:boto:path=/hiiqos-55/e?versionId=null 2023-07-29T00:17:52.264 DEBUG:boto:auth_path=/hiiqos-55/e?versionId=null 2023-07-29T00:17:52.265 DEBUG:boto:Method: GET 2023-07-29T00:17:52.266 DEBUG:boto:Path: /hiiqos-55/e?versionId=null 2023-07-29T00:17:52.266 DEBUG:boto:Data: 2023-07-29T00:17:52.267 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.268 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:52.268 DEBUG:boto:Port: 8001 2023-07-29T00:17:52.269 DEBUG:boto:Params: {} 2023-07-29T00:17:52.269 DEBUG:boto:Token: None 2023-07-29T00:17:52.269 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/e?versionId=null 2023-07-29T00:17:52.270 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:N3uBRkJaALjdWYqGPneTe3xR1gw= 2023-07-29T00:17:52.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:N3uBRkJaALjdWYqGPneTe3xR1gw=', 'Content-Length': '0'} 2023-07-29T00:17:52.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008972ee1a0b127a0b-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.273 DEBUG:boto:path=/hiiqos-55/e 2023-07-29T00:17:52.274 DEBUG:boto:auth_path=/hiiqos-55/e 2023-07-29T00:17:52.274 DEBUG:boto:path=/hiiqos-55/e?versionId=null 2023-07-29T00:17:52.275 DEBUG:boto:auth_path=/hiiqos-55/e?versionId=null 2023-07-29T00:17:52.275 DEBUG:boto:Method: GET 2023-07-29T00:17:52.275 DEBUG:boto:Path: /hiiqos-55/e?versionId=null 2023-07-29T00:17:52.276 DEBUG:boto:Data: 2023-07-29T00:17:52.276 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.277 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:52.277 DEBUG:boto:Port: 8000 2023-07-29T00:17:52.277 DEBUG:boto:Params: {} 2023-07-29T00:17:52.278 DEBUG:boto:Token: None 2023-07-29T00:17:52.278 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/e?versionId=null 2023-07-29T00:17:52.279 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:N3uBRkJaALjdWYqGPneTe3xR1gw= 2023-07-29T00:17:52.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:N3uBRkJaALjdWYqGPneTe3xR1gw=', 'Content-Length': '0'} 2023-07-29T00:17:52.281 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000630dedb542ac032a-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.324 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:52.325 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:52.325 DEBUG:boto:path=/hiiqos-55/?prefix=e 2023-07-29T00:17:52.326 DEBUG:boto:auth_path=/hiiqos-55/?prefix=e 2023-07-29T00:17:52.326 DEBUG:boto:Method: GET 2023-07-29T00:17:52.326 DEBUG:boto:Path: /hiiqos-55/?prefix=e 2023-07-29T00:17:52.327 DEBUG:boto:Data: 2023-07-29T00:17:52.327 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.328 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:52.328 DEBUG:boto:Port: 8001 2023-07-29T00:17:52.329 DEBUG:boto:Params: {} 2023-07-29T00:17:52.329 DEBUG:boto:Token: None 2023-07-29T00:17:52.329 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/ 2023-07-29T00:17:52.330 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T00:17:52.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T00:17:52.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1e85a493360275d-0064c45ab0-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.334 DEBUG:boto:b'hiiqos-55e1000falsee2023-07-29T00:15:00.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:52.335 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:52.335 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:52.336 DEBUG:boto:path=/hiiqos-55/?prefix=e 2023-07-29T00:17:52.337 DEBUG:boto:auth_path=/hiiqos-55/?prefix=e 2023-07-29T00:17:52.337 DEBUG:boto:Method: GET 2023-07-29T00:17:52.337 DEBUG:boto:Path: /hiiqos-55/?prefix=e 2023-07-29T00:17:52.338 DEBUG:boto:Data: 2023-07-29T00:17:52.338 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.339 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:52.339 DEBUG:boto:Port: 8000 2023-07-29T00:17:52.339 DEBUG:boto:Params: {} 2023-07-29T00:17:52.340 DEBUG:boto:Token: None 2023-07-29T00:17:52.340 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/ 2023-07-29T00:17:52.341 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T00:17:52.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T00:17:52.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0b81074a5a417d0-0064c45ab0-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.344 DEBUG:boto:b'hiiqos-55e1000falsee2023-07-29T00:15:00.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:52.345 DEBUG:boto:path=/hiiqos-55/e 2023-07-29T00:17:52.345 DEBUG:boto:auth_path=/hiiqos-55/e 2023-07-29T00:17:52.346 DEBUG:boto:Method: HEAD 2023-07-29T00:17:52.346 DEBUG:boto:Path: /hiiqos-55/e 2023-07-29T00:17:52.347 DEBUG:boto:Data: 2023-07-29T00:17:52.347 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.348 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:52.348 DEBUG:boto:Port: 8001 2023-07-29T00:17:52.349 DEBUG:boto:Params: {} 2023-07-29T00:17:52.350 DEBUG:boto:Token: None 2023-07-29T00:17:52.351 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/e 2023-07-29T00:17:52.352 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SgWvTdm5halu9UEj51FV4sX1P4E= 2023-07-29T00:17:52.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SgWvTdm5halu9UEj51FV4sX1P4E=', 'Content-Length': '0'} 2023-07-29T00:17:52.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000790df59022b3b1af-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.355 DEBUG:boto:path=/hiiqos-55/e 2023-07-29T00:17:52.356 DEBUG:boto:auth_path=/hiiqos-55/e 2023-07-29T00:17:52.356 DEBUG:boto:Method: HEAD 2023-07-29T00:17:52.357 DEBUG:boto:Path: /hiiqos-55/e 2023-07-29T00:17:52.357 DEBUG:boto:Data: 2023-07-29T00:17:52.357 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.358 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:52.358 DEBUG:boto:Port: 8000 2023-07-29T00:17:52.358 DEBUG:boto:Params: {} 2023-07-29T00:17:52.359 DEBUG:boto:Token: None 2023-07-29T00:17:52.359 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/e 2023-07-29T00:17:52.359 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SgWvTdm5halu9UEj51FV4sX1P4E= 2023-07-29T00:17:52.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SgWvTdm5halu9UEj51FV4sX1P4E=', 'Content-Length': '0'} 2023-07-29T00:17:52.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d58b51a2051ab62-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-07-29T00:17:52.363 DEBUG:boto:path=/hiiqos-55/e 2023-07-29T00:17:52.363 DEBUG:boto:auth_path=/hiiqos-55/e 2023-07-29T00:17:52.364 DEBUG:boto:Method: GET 2023-07-29T00:17:52.364 DEBUG:boto:Path: /hiiqos-55/e 2023-07-29T00:17:52.364 DEBUG:boto:Data: 2023-07-29T00:17:52.365 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.365 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:52.365 DEBUG:boto:Port: 8001 2023-07-29T00:17:52.366 DEBUG:boto:Params: {} 2023-07-29T00:17:52.366 DEBUG:boto:Token: None 2023-07-29T00:17:52.366 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/e 2023-07-29T00:17:52.367 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YxNeDhSQ/lfu2QO3WerXuLe7XSo= 2023-07-29T00:17:52.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YxNeDhSQ/lfu2QO3WerXuLe7XSo=', 'Content-Length': '0'} 2023-07-29T00:17:52.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf6dc73622af9cdd-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.412 DEBUG:boto:path=/hiiqos-55/e 2023-07-29T00:17:52.413 DEBUG:boto:auth_path=/hiiqos-55/e 2023-07-29T00:17:52.413 DEBUG:boto:Method: GET 2023-07-29T00:17:52.414 DEBUG:boto:Path: /hiiqos-55/e 2023-07-29T00:17:52.414 DEBUG:boto:Data: 2023-07-29T00:17:52.415 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.415 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:52.415 DEBUG:boto:Port: 8000 2023-07-29T00:17:52.416 DEBUG:boto:Params: {} 2023-07-29T00:17:52.416 DEBUG:boto:Token: None 2023-07-29T00:17:52.417 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/e 2023-07-29T00:17:52.417 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YxNeDhSQ/lfu2QO3WerXuLe7XSo= 2023-07-29T00:17:52.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YxNeDhSQ/lfu2QO3WerXuLe7XSo=', 'Content-Length': '0'} 2023-07-29T00:17:52.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005001bcc8041f564-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.420 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:52.421 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:52.421 DEBUG:boto:path=/hiiqos-55/?prefix=e 2023-07-29T00:17:52.421 DEBUG:boto:auth_path=/hiiqos-55/?prefix=e 2023-07-29T00:17:52.422 DEBUG:boto:Method: GET 2023-07-29T00:17:52.422 DEBUG:boto:Path: /hiiqos-55/?prefix=e 2023-07-29T00:17:52.423 DEBUG:boto:Data: 2023-07-29T00:17:52.423 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.423 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:52.424 DEBUG:boto:Port: 8001 2023-07-29T00:17:52.424 DEBUG:boto:Params: {} 2023-07-29T00:17:52.424 DEBUG:boto:Token: None 2023-07-29T00:17:52.425 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/ 2023-07-29T00:17:52.425 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T00:17:52.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T00:17:52.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b35f907dc0a680e-0064c45ab0-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.429 DEBUG:boto:b'hiiqos-55e1000falsee2023-07-29T00:15:00.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:52.429 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:52.430 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:52.431 DEBUG:boto:path=/hiiqos-55/?prefix=e 2023-07-29T00:17:52.432 DEBUG:boto:auth_path=/hiiqos-55/?prefix=e 2023-07-29T00:17:52.433 DEBUG:boto:Method: GET 2023-07-29T00:17:52.434 DEBUG:boto:Path: /hiiqos-55/?prefix=e 2023-07-29T00:17:52.435 DEBUG:boto:Data: 2023-07-29T00:17:52.436 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.436 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:52.437 DEBUG:boto:Port: 8000 2023-07-29T00:17:52.438 DEBUG:boto:Params: {} 2023-07-29T00:17:52.438 DEBUG:boto:Token: None 2023-07-29T00:17:52.439 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/ 2023-07-29T00:17:52.440 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T00:17:52.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T00:17:52.443 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031238a5a6413035c-0064c45ab0-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.443 DEBUG:boto:b'hiiqos-55e1000falsee2023-07-29T00:15:00.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:52.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-07-29T00:17:52.444 DEBUG:boto:path=/hiiqos-55/f 2023-07-29T00:17:52.445 DEBUG:boto:auth_path=/hiiqos-55/f 2023-07-29T00:17:52.445 DEBUG:boto:path=/hiiqos-55/f?versionId=null 2023-07-29T00:17:52.446 DEBUG:boto:auth_path=/hiiqos-55/f?versionId=null 2023-07-29T00:17:52.446 DEBUG:boto:Method: GET 2023-07-29T00:17:52.446 DEBUG:boto:Path: /hiiqos-55/f?versionId=null 2023-07-29T00:17:52.447 DEBUG:boto:Data: 2023-07-29T00:17:52.447 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.447 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:52.448 DEBUG:boto:Port: 8001 2023-07-29T00:17:52.448 DEBUG:boto:Params: {} 2023-07-29T00:17:52.449 DEBUG:boto:Token: None 2023-07-29T00:17:52.449 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/f?versionId=null 2023-07-29T00:17:52.449 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oPKjocaacwAeufKTae4WMimfxpk= 2023-07-29T00:17:52.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oPKjocaacwAeufKTae4WMimfxpk=', 'Content-Length': '0'} 2023-07-29T00:17:52.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003483810c790d6c46-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.452 DEBUG:boto:path=/hiiqos-55/f 2023-07-29T00:17:52.453 DEBUG:boto:auth_path=/hiiqos-55/f 2023-07-29T00:17:52.453 DEBUG:boto:path=/hiiqos-55/f?versionId=null 2023-07-29T00:17:52.453 DEBUG:boto:auth_path=/hiiqos-55/f?versionId=null 2023-07-29T00:17:52.454 DEBUG:boto:Method: GET 2023-07-29T00:17:52.454 DEBUG:boto:Path: /hiiqos-55/f?versionId=null 2023-07-29T00:17:52.455 DEBUG:boto:Data: 2023-07-29T00:17:52.455 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.455 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:52.456 DEBUG:boto:Port: 8000 2023-07-29T00:17:52.456 DEBUG:boto:Params: {} 2023-07-29T00:17:52.457 DEBUG:boto:Token: None 2023-07-29T00:17:52.457 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/f?versionId=null 2023-07-29T00:17:52.457 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oPKjocaacwAeufKTae4WMimfxpk= 2023-07-29T00:17:52.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oPKjocaacwAeufKTae4WMimfxpk=', 'Content-Length': '0'} 2023-07-29T00:17:52.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099503e7261290d66-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.460 DEBUG:boto:path=/hiiqos-55/f 2023-07-29T00:17:52.461 DEBUG:boto:auth_path=/hiiqos-55/f 2023-07-29T00:17:52.461 DEBUG:boto:path=/hiiqos-55/f?versionId=null 2023-07-29T00:17:52.461 DEBUG:boto:auth_path=/hiiqos-55/f?versionId=null 2023-07-29T00:17:52.462 DEBUG:boto:Method: HEAD 2023-07-29T00:17:52.462 DEBUG:boto:Path: /hiiqos-55/f?versionId=null 2023-07-29T00:17:52.462 DEBUG:boto:Data: 2023-07-29T00:17:52.463 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.463 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:52.463 DEBUG:boto:Port: 8001 2023-07-29T00:17:52.464 DEBUG:boto:Params: {} 2023-07-29T00:17:52.464 DEBUG:boto:Token: None 2023-07-29T00:17:52.465 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/f?versionId=null 2023-07-29T00:17:52.465 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VX//KDpAivFYiTnzXu27tVbaLhQ= 2023-07-29T00:17:52.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VX//KDpAivFYiTnzXu27tVbaLhQ=', 'Content-Length': '0'} 2023-07-29T00:17:52.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce015b253e0bab7d-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.468 DEBUG:boto:path=/hiiqos-55/f 2023-07-29T00:17:52.468 DEBUG:boto:auth_path=/hiiqos-55/f 2023-07-29T00:17:52.468 DEBUG:boto:path=/hiiqos-55/f?versionId=null 2023-07-29T00:17:52.469 DEBUG:boto:auth_path=/hiiqos-55/f?versionId=null 2023-07-29T00:17:52.469 DEBUG:boto:Method: HEAD 2023-07-29T00:17:52.469 DEBUG:boto:Path: /hiiqos-55/f?versionId=null 2023-07-29T00:17:52.470 DEBUG:boto:Data: 2023-07-29T00:17:52.470 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.470 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:52.471 DEBUG:boto:Port: 8000 2023-07-29T00:17:52.471 DEBUG:boto:Params: {} 2023-07-29T00:17:52.472 DEBUG:boto:Token: None 2023-07-29T00:17:52.472 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/f?versionId=null 2023-07-29T00:17:52.472 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VX//KDpAivFYiTnzXu27tVbaLhQ= 2023-07-29T00:17:52.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VX//KDpAivFYiTnzXu27tVbaLhQ=', 'Content-Length': '0'} 2023-07-29T00:17:52.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022a5abaccabd8c32-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-07-29T00:17:52.476 DEBUG:boto:path=/hiiqos-55/f 2023-07-29T00:17:52.476 DEBUG:boto:auth_path=/hiiqos-55/f 2023-07-29T00:17:52.476 DEBUG:boto:path=/hiiqos-55/f?versionId=null 2023-07-29T00:17:52.477 DEBUG:boto:auth_path=/hiiqos-55/f?versionId=null 2023-07-29T00:17:52.477 DEBUG:boto:Method: GET 2023-07-29T00:17:52.478 DEBUG:boto:Path: /hiiqos-55/f?versionId=null 2023-07-29T00:17:52.478 DEBUG:boto:Data: 2023-07-29T00:17:52.478 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.479 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:52.479 DEBUG:boto:Port: 8001 2023-07-29T00:17:52.479 DEBUG:boto:Params: {} 2023-07-29T00:17:52.480 DEBUG:boto:Token: None 2023-07-29T00:17:52.480 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/f?versionId=null 2023-07-29T00:17:52.481 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oPKjocaacwAeufKTae4WMimfxpk= 2023-07-29T00:17:52.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oPKjocaacwAeufKTae4WMimfxpk=', 'Content-Length': '0'} 2023-07-29T00:17:52.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb3693e28b767431-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.483 DEBUG:boto:path=/hiiqos-55/f 2023-07-29T00:17:52.484 DEBUG:boto:auth_path=/hiiqos-55/f 2023-07-29T00:17:52.484 DEBUG:boto:path=/hiiqos-55/f?versionId=null 2023-07-29T00:17:52.484 DEBUG:boto:auth_path=/hiiqos-55/f?versionId=null 2023-07-29T00:17:52.485 DEBUG:boto:Method: GET 2023-07-29T00:17:52.485 DEBUG:boto:Path: /hiiqos-55/f?versionId=null 2023-07-29T00:17:52.486 DEBUG:boto:Data: 2023-07-29T00:17:52.486 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.486 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:52.487 DEBUG:boto:Port: 8000 2023-07-29T00:17:52.487 DEBUG:boto:Params: {} 2023-07-29T00:17:52.487 DEBUG:boto:Token: None 2023-07-29T00:17:52.488 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/f?versionId=null 2023-07-29T00:17:52.488 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oPKjocaacwAeufKTae4WMimfxpk= 2023-07-29T00:17:52.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oPKjocaacwAeufKTae4WMimfxpk=', 'Content-Length': '0'} 2023-07-29T00:17:52.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000587ba367585c93c6-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.491 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:52.492 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:52.492 DEBUG:boto:path=/hiiqos-55/?prefix=f 2023-07-29T00:17:52.492 DEBUG:boto:auth_path=/hiiqos-55/?prefix=f 2023-07-29T00:17:52.493 DEBUG:boto:Method: GET 2023-07-29T00:17:52.493 DEBUG:boto:Path: /hiiqos-55/?prefix=f 2023-07-29T00:17:52.493 DEBUG:boto:Data: 2023-07-29T00:17:52.494 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.494 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:52.495 DEBUG:boto:Port: 8001 2023-07-29T00:17:52.495 DEBUG:boto:Params: {} 2023-07-29T00:17:52.495 DEBUG:boto:Token: None 2023-07-29T00:17:52.496 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/ 2023-07-29T00:17:52.496 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T00:17:52.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T00:17:52.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8b1a70d2bd9b193-0064c45ab0-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.500 DEBUG:boto:b'hiiqos-55f1000falsef2023-07-29T00:15:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:52.500 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:52.501 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:52.501 DEBUG:boto:path=/hiiqos-55/?prefix=f 2023-07-29T00:17:52.501 DEBUG:boto:auth_path=/hiiqos-55/?prefix=f 2023-07-29T00:17:52.502 DEBUG:boto:Method: GET 2023-07-29T00:17:52.502 DEBUG:boto:Path: /hiiqos-55/?prefix=f 2023-07-29T00:17:52.502 DEBUG:boto:Data: 2023-07-29T00:17:52.503 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.503 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:52.503 DEBUG:boto:Port: 8000 2023-07-29T00:17:52.504 DEBUG:boto:Params: {} 2023-07-29T00:17:52.504 DEBUG:boto:Token: None 2023-07-29T00:17:52.505 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/ 2023-07-29T00:17:52.505 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T00:17:52.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T00:17:52.507 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006274d6c33fad8a99-0064c45ab0-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.508 DEBUG:boto:b'hiiqos-55f1000falsef2023-07-29T00:15:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:52.508 DEBUG:boto:path=/hiiqos-55/f 2023-07-29T00:17:52.509 DEBUG:boto:auth_path=/hiiqos-55/f 2023-07-29T00:17:52.509 DEBUG:boto:Method: HEAD 2023-07-29T00:17:52.510 DEBUG:boto:Path: /hiiqos-55/f 2023-07-29T00:17:52.510 DEBUG:boto:Data: 2023-07-29T00:17:52.510 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.511 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:52.511 DEBUG:boto:Port: 8001 2023-07-29T00:17:52.511 DEBUG:boto:Params: {} 2023-07-29T00:17:52.512 DEBUG:boto:Token: None 2023-07-29T00:17:52.512 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/f 2023-07-29T00:17:52.512 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dPtyzLDV1b6Yqrf28S//81SkE/0= 2023-07-29T00:17:52.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dPtyzLDV1b6Yqrf28S//81SkE/0=', 'Content-Length': '0'} 2023-07-29T00:17:52.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ca3f1998c05b17c-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.515 DEBUG:boto:path=/hiiqos-55/f 2023-07-29T00:17:52.515 DEBUG:boto:auth_path=/hiiqos-55/f 2023-07-29T00:17:52.516 DEBUG:boto:Method: HEAD 2023-07-29T00:17:52.516 DEBUG:boto:Path: /hiiqos-55/f 2023-07-29T00:17:52.516 DEBUG:boto:Data: 2023-07-29T00:17:52.517 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.517 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:52.517 DEBUG:boto:Port: 8000 2023-07-29T00:17:52.518 DEBUG:boto:Params: {} 2023-07-29T00:17:52.518 DEBUG:boto:Token: None 2023-07-29T00:17:52.518 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/f 2023-07-29T00:17:52.519 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dPtyzLDV1b6Yqrf28S//81SkE/0= 2023-07-29T00:17:52.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dPtyzLDV1b6Yqrf28S//81SkE/0=', 'Content-Length': '0'} 2023-07-29T00:17:52.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e03fa0ab61b4f36-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-07-29T00:17:52.522 DEBUG:boto:path=/hiiqos-55/f 2023-07-29T00:17:52.522 DEBUG:boto:auth_path=/hiiqos-55/f 2023-07-29T00:17:52.523 DEBUG:boto:Method: GET 2023-07-29T00:17:52.523 DEBUG:boto:Path: /hiiqos-55/f 2023-07-29T00:17:52.523 DEBUG:boto:Data: 2023-07-29T00:17:52.524 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.524 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:52.524 DEBUG:boto:Port: 8001 2023-07-29T00:17:52.525 DEBUG:boto:Params: {} 2023-07-29T00:17:52.525 DEBUG:boto:Token: None 2023-07-29T00:17:52.526 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/f 2023-07-29T00:17:52.526 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mTgT+KfdB6wYOWIZ6BMsqVEEoNs= 2023-07-29T00:17:52.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mTgT+KfdB6wYOWIZ6BMsqVEEoNs=', 'Content-Length': '0'} 2023-07-29T00:17:52.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cbe500111f3f202-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.529 DEBUG:boto:path=/hiiqos-55/f 2023-07-29T00:17:52.529 DEBUG:boto:auth_path=/hiiqos-55/f 2023-07-29T00:17:52.530 DEBUG:boto:Method: GET 2023-07-29T00:17:52.530 DEBUG:boto:Path: /hiiqos-55/f 2023-07-29T00:17:52.530 DEBUG:boto:Data: 2023-07-29T00:17:52.531 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.531 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:52.531 DEBUG:boto:Port: 8000 2023-07-29T00:17:52.532 DEBUG:boto:Params: {} 2023-07-29T00:17:52.532 DEBUG:boto:Token: None 2023-07-29T00:17:52.533 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/f 2023-07-29T00:17:52.533 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mTgT+KfdB6wYOWIZ6BMsqVEEoNs= 2023-07-29T00:17:52.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mTgT+KfdB6wYOWIZ6BMsqVEEoNs=', 'Content-Length': '0'} 2023-07-29T00:17:52.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ead464c090fd8bb-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.536 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:52.536 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:52.536 DEBUG:boto:path=/hiiqos-55/?prefix=f 2023-07-29T00:17:52.537 DEBUG:boto:auth_path=/hiiqos-55/?prefix=f 2023-07-29T00:17:52.537 DEBUG:boto:Method: GET 2023-07-29T00:17:52.537 DEBUG:boto:Path: /hiiqos-55/?prefix=f 2023-07-29T00:17:52.538 DEBUG:boto:Data: 2023-07-29T00:17:52.538 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.538 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:52.539 DEBUG:boto:Port: 8001 2023-07-29T00:17:52.539 DEBUG:boto:Params: {} 2023-07-29T00:17:52.540 DEBUG:boto:Token: None 2023-07-29T00:17:52.540 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/ 2023-07-29T00:17:52.540 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T00:17:52.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T00:17:52.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057da9643c0159c5e-0064c45ab0-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.584 DEBUG:boto:b'hiiqos-55f1000falsef2023-07-29T00:15:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:52.585 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:52.586 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:52.586 DEBUG:boto:path=/hiiqos-55/?prefix=f 2023-07-29T00:17:52.586 DEBUG:boto:auth_path=/hiiqos-55/?prefix=f 2023-07-29T00:17:52.587 DEBUG:boto:Method: GET 2023-07-29T00:17:52.587 DEBUG:boto:Path: /hiiqos-55/?prefix=f 2023-07-29T00:17:52.587 DEBUG:boto:Data: 2023-07-29T00:17:52.588 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.588 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:52.588 DEBUG:boto:Port: 8000 2023-07-29T00:17:52.589 DEBUG:boto:Params: {} 2023-07-29T00:17:52.589 DEBUG:boto:Token: None 2023-07-29T00:17:52.590 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/ 2023-07-29T00:17:52.590 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T00:17:52.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T00:17:52.592 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d66be81f32fe6f9-0064c45ab0-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.593 DEBUG:boto:b'hiiqos-55f1000falsef2023-07-29T00:15:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:52.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-07-29T00:17:52.594 DEBUG:boto:path=/hiiqos-55/g 2023-07-29T00:17:52.594 DEBUG:boto:auth_path=/hiiqos-55/g 2023-07-29T00:17:52.595 DEBUG:boto:path=/hiiqos-55/g?versionId=null 2023-07-29T00:17:52.595 DEBUG:boto:auth_path=/hiiqos-55/g?versionId=null 2023-07-29T00:17:52.595 DEBUG:boto:Method: GET 2023-07-29T00:17:52.596 DEBUG:boto:Path: /hiiqos-55/g?versionId=null 2023-07-29T00:17:52.596 DEBUG:boto:Data: 2023-07-29T00:17:52.597 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.597 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:52.597 DEBUG:boto:Port: 8001 2023-07-29T00:17:52.598 DEBUG:boto:Params: {} 2023-07-29T00:17:52.598 DEBUG:boto:Token: None 2023-07-29T00:17:52.598 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/g?versionId=null 2023-07-29T00:17:52.599 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IzU7oefaY1bviV3nIiOVEj01G8Y= 2023-07-29T00:17:52.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IzU7oefaY1bviV3nIiOVEj01G8Y=', 'Content-Length': '0'} 2023-07-29T00:17:52.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000193e7837c393c3bb-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.602 DEBUG:boto:path=/hiiqos-55/g 2023-07-29T00:17:52.602 DEBUG:boto:auth_path=/hiiqos-55/g 2023-07-29T00:17:52.602 DEBUG:boto:path=/hiiqos-55/g?versionId=null 2023-07-29T00:17:52.603 DEBUG:boto:auth_path=/hiiqos-55/g?versionId=null 2023-07-29T00:17:52.603 DEBUG:boto:Method: GET 2023-07-29T00:17:52.604 DEBUG:boto:Path: /hiiqos-55/g?versionId=null 2023-07-29T00:17:52.604 DEBUG:boto:Data: 2023-07-29T00:17:52.604 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.605 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:52.605 DEBUG:boto:Port: 8000 2023-07-29T00:17:52.605 DEBUG:boto:Params: {} 2023-07-29T00:17:52.606 DEBUG:boto:Token: None 2023-07-29T00:17:52.606 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/g?versionId=null 2023-07-29T00:17:52.607 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IzU7oefaY1bviV3nIiOVEj01G8Y= 2023-07-29T00:17:52.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IzU7oefaY1bviV3nIiOVEj01G8Y=', 'Content-Length': '0'} 2023-07-29T00:17:52.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f74116d3c78d689-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.652 DEBUG:boto:path=/hiiqos-55/g 2023-07-29T00:17:52.653 DEBUG:boto:auth_path=/hiiqos-55/g 2023-07-29T00:17:52.653 DEBUG:boto:path=/hiiqos-55/g?versionId=null 2023-07-29T00:17:52.653 DEBUG:boto:auth_path=/hiiqos-55/g?versionId=null 2023-07-29T00:17:52.654 DEBUG:boto:Method: HEAD 2023-07-29T00:17:52.654 DEBUG:boto:Path: /hiiqos-55/g?versionId=null 2023-07-29T00:17:52.655 DEBUG:boto:Data: 2023-07-29T00:17:52.655 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.655 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:52.656 DEBUG:boto:Port: 8001 2023-07-29T00:17:52.656 DEBUG:boto:Params: {} 2023-07-29T00:17:52.656 DEBUG:boto:Token: None 2023-07-29T00:17:52.657 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/g?versionId=null 2023-07-29T00:17:52.657 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Q/yqVXQpcZZ/bXbp1gBVOaL8fr4= 2023-07-29T00:17:52.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q/yqVXQpcZZ/bXbp1gBVOaL8fr4=', 'Content-Length': '0'} 2023-07-29T00:17:52.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbda2a0bf0971958-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.660 DEBUG:boto:path=/hiiqos-55/g 2023-07-29T00:17:52.660 DEBUG:boto:auth_path=/hiiqos-55/g 2023-07-29T00:17:52.661 DEBUG:boto:path=/hiiqos-55/g?versionId=null 2023-07-29T00:17:52.661 DEBUG:boto:auth_path=/hiiqos-55/g?versionId=null 2023-07-29T00:17:52.661 DEBUG:boto:Method: HEAD 2023-07-29T00:17:52.662 DEBUG:boto:Path: /hiiqos-55/g?versionId=null 2023-07-29T00:17:52.662 DEBUG:boto:Data: 2023-07-29T00:17:52.662 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.663 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:52.663 DEBUG:boto:Port: 8000 2023-07-29T00:17:52.663 DEBUG:boto:Params: {} 2023-07-29T00:17:52.664 DEBUG:boto:Token: None 2023-07-29T00:17:52.664 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/g?versionId=null 2023-07-29T00:17:52.665 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Q/yqVXQpcZZ/bXbp1gBVOaL8fr4= 2023-07-29T00:17:52.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q/yqVXQpcZZ/bXbp1gBVOaL8fr4=', 'Content-Length': '0'} 2023-07-29T00:17:52.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d99a7e52f565f72-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-07-29T00:17:52.667 DEBUG:boto:path=/hiiqos-55/g 2023-07-29T00:17:52.668 DEBUG:boto:auth_path=/hiiqos-55/g 2023-07-29T00:17:52.668 DEBUG:boto:path=/hiiqos-55/g?versionId=null 2023-07-29T00:17:52.669 DEBUG:boto:auth_path=/hiiqos-55/g?versionId=null 2023-07-29T00:17:52.669 DEBUG:boto:Method: GET 2023-07-29T00:17:52.669 DEBUG:boto:Path: /hiiqos-55/g?versionId=null 2023-07-29T00:17:52.670 DEBUG:boto:Data: 2023-07-29T00:17:52.670 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.671 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:52.671 DEBUG:boto:Port: 8001 2023-07-29T00:17:52.671 DEBUG:boto:Params: {} 2023-07-29T00:17:52.672 DEBUG:boto:Token: None 2023-07-29T00:17:52.672 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/g?versionId=null 2023-07-29T00:17:52.673 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IzU7oefaY1bviV3nIiOVEj01G8Y= 2023-07-29T00:17:52.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IzU7oefaY1bviV3nIiOVEj01G8Y=', 'Content-Length': '0'} 2023-07-29T00:17:52.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf425de4bf3e2491-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.675 DEBUG:boto:path=/hiiqos-55/g 2023-07-29T00:17:52.676 DEBUG:boto:auth_path=/hiiqos-55/g 2023-07-29T00:17:52.676 DEBUG:boto:path=/hiiqos-55/g?versionId=null 2023-07-29T00:17:52.677 DEBUG:boto:auth_path=/hiiqos-55/g?versionId=null 2023-07-29T00:17:52.677 DEBUG:boto:Method: GET 2023-07-29T00:17:52.677 DEBUG:boto:Path: /hiiqos-55/g?versionId=null 2023-07-29T00:17:52.678 DEBUG:boto:Data: 2023-07-29T00:17:52.678 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.678 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:52.679 DEBUG:boto:Port: 8000 2023-07-29T00:17:52.679 DEBUG:boto:Params: {} 2023-07-29T00:17:52.679 DEBUG:boto:Token: None 2023-07-29T00:17:52.680 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/g?versionId=null 2023-07-29T00:17:52.680 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IzU7oefaY1bviV3nIiOVEj01G8Y= 2023-07-29T00:17:52.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IzU7oefaY1bviV3nIiOVEj01G8Y=', 'Content-Length': '0'} 2023-07-29T00:17:52.682 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000048d9a5cc6ec3a8e-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.683 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:52.684 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:52.684 DEBUG:boto:path=/hiiqos-55/?prefix=g 2023-07-29T00:17:52.684 DEBUG:boto:auth_path=/hiiqos-55/?prefix=g 2023-07-29T00:17:52.685 DEBUG:boto:Method: GET 2023-07-29T00:17:52.685 DEBUG:boto:Path: /hiiqos-55/?prefix=g 2023-07-29T00:17:52.686 DEBUG:boto:Data: 2023-07-29T00:17:52.686 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.686 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:52.687 DEBUG:boto:Port: 8001 2023-07-29T00:17:52.687 DEBUG:boto:Params: {} 2023-07-29T00:17:52.687 DEBUG:boto:Token: None 2023-07-29T00:17:52.688 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/ 2023-07-29T00:17:52.688 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T00:17:52.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T00:17:52.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f90c91286bed80cd-0064c45ab0-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.691 DEBUG:boto:b'hiiqos-55g1000falseg2023-07-29T00:15:01.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:52.692 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:52.692 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:52.693 DEBUG:boto:path=/hiiqos-55/?prefix=g 2023-07-29T00:17:52.693 DEBUG:boto:auth_path=/hiiqos-55/?prefix=g 2023-07-29T00:17:52.694 DEBUG:boto:Method: GET 2023-07-29T00:17:52.694 DEBUG:boto:Path: /hiiqos-55/?prefix=g 2023-07-29T00:17:52.694 DEBUG:boto:Data: 2023-07-29T00:17:52.695 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.695 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:52.695 DEBUG:boto:Port: 8000 2023-07-29T00:17:52.696 DEBUG:boto:Params: {} 2023-07-29T00:17:52.696 DEBUG:boto:Token: None 2023-07-29T00:17:52.697 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/ 2023-07-29T00:17:52.697 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T00:17:52.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T00:17:52.699 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc60fa27340da666-0064c45ab0-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.700 DEBUG:boto:b'hiiqos-55g1000falseg2023-07-29T00:15:01.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:52.700 DEBUG:boto:path=/hiiqos-55/g 2023-07-29T00:17:52.701 DEBUG:boto:auth_path=/hiiqos-55/g 2023-07-29T00:17:52.701 DEBUG:boto:Method: HEAD 2023-07-29T00:17:52.701 DEBUG:boto:Path: /hiiqos-55/g 2023-07-29T00:17:52.702 DEBUG:boto:Data: 2023-07-29T00:17:52.702 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.702 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:52.703 DEBUG:boto:Port: 8001 2023-07-29T00:17:52.703 DEBUG:boto:Params: {} 2023-07-29T00:17:52.704 DEBUG:boto:Token: None 2023-07-29T00:17:52.704 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/g 2023-07-29T00:17:52.705 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QxryRujaF8rFUevgWB9LcvBjN+E= 2023-07-29T00:17:52.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QxryRujaF8rFUevgWB9LcvBjN+E=', 'Content-Length': '0'} 2023-07-29T00:17:52.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bde0d67e70c38c05-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.709 DEBUG:boto:path=/hiiqos-55/g 2023-07-29T00:17:52.710 DEBUG:boto:auth_path=/hiiqos-55/g 2023-07-29T00:17:52.711 DEBUG:boto:Method: HEAD 2023-07-29T00:17:52.711 DEBUG:boto:Path: /hiiqos-55/g 2023-07-29T00:17:52.712 DEBUG:boto:Data: 2023-07-29T00:17:52.712 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.713 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:52.713 DEBUG:boto:Port: 8000 2023-07-29T00:17:52.713 DEBUG:boto:Params: {} 2023-07-29T00:17:52.714 DEBUG:boto:Token: None 2023-07-29T00:17:52.714 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/g 2023-07-29T00:17:52.714 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QxryRujaF8rFUevgWB9LcvBjN+E= 2023-07-29T00:17:52.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QxryRujaF8rFUevgWB9LcvBjN+E=', 'Content-Length': '0'} 2023-07-29T00:17:52.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011e15deafe34baeb-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-07-29T00:17:52.717 DEBUG:boto:path=/hiiqos-55/g 2023-07-29T00:17:52.718 DEBUG:boto:auth_path=/hiiqos-55/g 2023-07-29T00:17:52.718 DEBUG:boto:Method: GET 2023-07-29T00:17:52.719 DEBUG:boto:Path: /hiiqos-55/g 2023-07-29T00:17:52.719 DEBUG:boto:Data: 2023-07-29T00:17:52.719 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.720 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:52.720 DEBUG:boto:Port: 8001 2023-07-29T00:17:52.720 DEBUG:boto:Params: {} 2023-07-29T00:17:52.721 DEBUG:boto:Token: None 2023-07-29T00:17:52.721 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/g 2023-07-29T00:17:52.722 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Lflq/q/L06omp8cwv/ba+0g9MTs= 2023-07-29T00:17:52.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Lflq/q/L06omp8cwv/ba+0g9MTs=', 'Content-Length': '0'} 2023-07-29T00:17:52.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d5adbef7ad18ead-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.725 DEBUG:boto:path=/hiiqos-55/g 2023-07-29T00:17:52.726 DEBUG:boto:auth_path=/hiiqos-55/g 2023-07-29T00:17:52.726 DEBUG:boto:Method: GET 2023-07-29T00:17:52.726 DEBUG:boto:Path: /hiiqos-55/g 2023-07-29T00:17:52.727 DEBUG:boto:Data: 2023-07-29T00:17:52.727 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.727 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:52.728 DEBUG:boto:Port: 8000 2023-07-29T00:17:52.728 DEBUG:boto:Params: {} 2023-07-29T00:17:52.729 DEBUG:boto:Token: None 2023-07-29T00:17:52.729 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/g 2023-07-29T00:17:52.729 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Lflq/q/L06omp8cwv/ba+0g9MTs= 2023-07-29T00:17:52.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Lflq/q/L06omp8cwv/ba+0g9MTs=', 'Content-Length': '0'} 2023-07-29T00:17:52.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e5a5990e1f57479-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.732 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:52.733 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:52.733 DEBUG:boto:path=/hiiqos-55/?prefix=g 2023-07-29T00:17:52.733 DEBUG:boto:auth_path=/hiiqos-55/?prefix=g 2023-07-29T00:17:52.734 DEBUG:boto:Method: GET 2023-07-29T00:17:52.735 DEBUG:boto:Path: /hiiqos-55/?prefix=g 2023-07-29T00:17:52.735 DEBUG:boto:Data: 2023-07-29T00:17:52.736 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.736 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:52.736 DEBUG:boto:Port: 8001 2023-07-29T00:17:52.737 DEBUG:boto:Params: {} 2023-07-29T00:17:52.737 DEBUG:boto:Token: None 2023-07-29T00:17:52.738 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/ 2023-07-29T00:17:52.738 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T00:17:52.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T00:17:52.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003003618b1be507b9-0064c45ab0-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.741 DEBUG:boto:b'hiiqos-55g1000falseg2023-07-29T00:15:01.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:52.742 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:52.742 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:52.742 DEBUG:boto:path=/hiiqos-55/?prefix=g 2023-07-29T00:17:52.743 DEBUG:boto:auth_path=/hiiqos-55/?prefix=g 2023-07-29T00:17:52.743 DEBUG:boto:Method: GET 2023-07-29T00:17:52.744 DEBUG:boto:Path: /hiiqos-55/?prefix=g 2023-07-29T00:17:52.744 DEBUG:boto:Data: 2023-07-29T00:17:52.744 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.745 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:52.745 DEBUG:boto:Port: 8000 2023-07-29T00:17:52.745 DEBUG:boto:Params: {} 2023-07-29T00:17:52.746 DEBUG:boto:Token: None 2023-07-29T00:17:52.747 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/ 2023-07-29T00:17:52.747 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T00:17:52.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T00:17:52.749 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d992e58a0fef678d-0064c45ab0-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.750 DEBUG:boto:b'hiiqos-55g1000falseg2023-07-29T00:15:01.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:52.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-07-29T00:17:52.751 DEBUG:boto:path=/hiiqos-55/h 2023-07-29T00:17:52.751 DEBUG:boto:auth_path=/hiiqos-55/h 2023-07-29T00:17:52.752 DEBUG:boto:path=/hiiqos-55/h?versionId=null 2023-07-29T00:17:52.752 DEBUG:boto:auth_path=/hiiqos-55/h?versionId=null 2023-07-29T00:17:52.752 DEBUG:boto:Method: GET 2023-07-29T00:17:52.753 DEBUG:boto:Path: /hiiqos-55/h?versionId=null 2023-07-29T00:17:52.753 DEBUG:boto:Data: 2023-07-29T00:17:52.753 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.754 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:52.754 DEBUG:boto:Port: 8001 2023-07-29T00:17:52.754 DEBUG:boto:Params: {} 2023-07-29T00:17:52.755 DEBUG:boto:Token: None 2023-07-29T00:17:52.755 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/h?versionId=null 2023-07-29T00:17:52.756 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZX0bat4jFiw5mdqaom7C1S6xQCY= 2023-07-29T00:17:52.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZX0bat4jFiw5mdqaom7C1S6xQCY=', 'Content-Length': '0'} 2023-07-29T00:17:52.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d93beb4bf4bd5748-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.800 DEBUG:boto:path=/hiiqos-55/h 2023-07-29T00:17:52.801 DEBUG:boto:auth_path=/hiiqos-55/h 2023-07-29T00:17:52.802 DEBUG:boto:path=/hiiqos-55/h?versionId=null 2023-07-29T00:17:52.802 DEBUG:boto:auth_path=/hiiqos-55/h?versionId=null 2023-07-29T00:17:52.803 DEBUG:boto:Method: GET 2023-07-29T00:17:52.803 DEBUG:boto:Path: /hiiqos-55/h?versionId=null 2023-07-29T00:17:52.803 DEBUG:boto:Data: 2023-07-29T00:17:52.804 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.804 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:52.805 DEBUG:boto:Port: 8000 2023-07-29T00:17:52.805 DEBUG:boto:Params: {} 2023-07-29T00:17:52.805 DEBUG:boto:Token: None 2023-07-29T00:17:52.806 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/h?versionId=null 2023-07-29T00:17:52.806 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZX0bat4jFiw5mdqaom7C1S6xQCY= 2023-07-29T00:17:52.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZX0bat4jFiw5mdqaom7C1S6xQCY=', 'Content-Length': '0'} 2023-07-29T00:17:52.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ed3f4083f27005b-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.809 DEBUG:boto:path=/hiiqos-55/h 2023-07-29T00:17:52.810 DEBUG:boto:auth_path=/hiiqos-55/h 2023-07-29T00:17:52.810 DEBUG:boto:path=/hiiqos-55/h?versionId=null 2023-07-29T00:17:52.810 DEBUG:boto:auth_path=/hiiqos-55/h?versionId=null 2023-07-29T00:17:52.811 DEBUG:boto:Method: HEAD 2023-07-29T00:17:52.811 DEBUG:boto:Path: /hiiqos-55/h?versionId=null 2023-07-29T00:17:52.811 DEBUG:boto:Data: 2023-07-29T00:17:52.812 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.812 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:52.813 DEBUG:boto:Port: 8001 2023-07-29T00:17:52.813 DEBUG:boto:Params: {} 2023-07-29T00:17:52.814 DEBUG:boto:Token: None 2023-07-29T00:17:52.814 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/h?versionId=null 2023-07-29T00:17:52.815 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7uSLB+kRtp7NRIh0+tHASwEKFJM= 2023-07-29T00:17:52.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7uSLB+kRtp7NRIh0+tHASwEKFJM=', 'Content-Length': '0'} 2023-07-29T00:17:52.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac6d46c542d001ff-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.817 DEBUG:boto:path=/hiiqos-55/h 2023-07-29T00:17:52.818 DEBUG:boto:auth_path=/hiiqos-55/h 2023-07-29T00:17:52.818 DEBUG:boto:path=/hiiqos-55/h?versionId=null 2023-07-29T00:17:52.818 DEBUG:boto:auth_path=/hiiqos-55/h?versionId=null 2023-07-29T00:17:52.819 DEBUG:boto:Method: HEAD 2023-07-29T00:17:52.819 DEBUG:boto:Path: /hiiqos-55/h?versionId=null 2023-07-29T00:17:52.819 DEBUG:boto:Data: 2023-07-29T00:17:52.820 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.820 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:52.820 DEBUG:boto:Port: 8000 2023-07-29T00:17:52.821 DEBUG:boto:Params: {} 2023-07-29T00:17:52.821 DEBUG:boto:Token: None 2023-07-29T00:17:52.822 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/h?versionId=null 2023-07-29T00:17:52.822 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7uSLB+kRtp7NRIh0+tHASwEKFJM= 2023-07-29T00:17:52.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7uSLB+kRtp7NRIh0+tHASwEKFJM=', 'Content-Length': '0'} 2023-07-29T00:17:52.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e898eb45fc9c94f4-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-07-29T00:17:52.825 DEBUG:boto:path=/hiiqos-55/h 2023-07-29T00:17:52.826 DEBUG:boto:auth_path=/hiiqos-55/h 2023-07-29T00:17:52.826 DEBUG:boto:path=/hiiqos-55/h?versionId=null 2023-07-29T00:17:52.826 DEBUG:boto:auth_path=/hiiqos-55/h?versionId=null 2023-07-29T00:17:52.827 DEBUG:boto:Method: GET 2023-07-29T00:17:52.827 DEBUG:boto:Path: /hiiqos-55/h?versionId=null 2023-07-29T00:17:52.828 DEBUG:boto:Data: 2023-07-29T00:17:52.828 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.828 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:52.829 DEBUG:boto:Port: 8001 2023-07-29T00:17:52.829 DEBUG:boto:Params: {} 2023-07-29T00:17:52.829 DEBUG:boto:Token: None 2023-07-29T00:17:52.830 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/h?versionId=null 2023-07-29T00:17:52.830 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZX0bat4jFiw5mdqaom7C1S6xQCY= 2023-07-29T00:17:52.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZX0bat4jFiw5mdqaom7C1S6xQCY=', 'Content-Length': '0'} 2023-07-29T00:17:52.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a20eda04f1de0637-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.833 DEBUG:boto:path=/hiiqos-55/h 2023-07-29T00:17:52.833 DEBUG:boto:auth_path=/hiiqos-55/h 2023-07-29T00:17:52.834 DEBUG:boto:path=/hiiqos-55/h?versionId=null 2023-07-29T00:17:52.834 DEBUG:boto:auth_path=/hiiqos-55/h?versionId=null 2023-07-29T00:17:52.834 DEBUG:boto:Method: GET 2023-07-29T00:17:52.835 DEBUG:boto:Path: /hiiqos-55/h?versionId=null 2023-07-29T00:17:52.835 DEBUG:boto:Data: 2023-07-29T00:17:52.836 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.837 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:52.837 DEBUG:boto:Port: 8000 2023-07-29T00:17:52.837 DEBUG:boto:Params: {} 2023-07-29T00:17:52.838 DEBUG:boto:Token: None 2023-07-29T00:17:52.838 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/h?versionId=null 2023-07-29T00:17:52.839 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZX0bat4jFiw5mdqaom7C1S6xQCY= 2023-07-29T00:17:52.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZX0bat4jFiw5mdqaom7C1S6xQCY=', 'Content-Length': '0'} 2023-07-29T00:17:52.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000314d547e5297ddef-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.884 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:52.885 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:52.885 DEBUG:boto:path=/hiiqos-55/?prefix=h 2023-07-29T00:17:52.886 DEBUG:boto:auth_path=/hiiqos-55/?prefix=h 2023-07-29T00:17:52.886 DEBUG:boto:Method: GET 2023-07-29T00:17:52.886 DEBUG:boto:Path: /hiiqos-55/?prefix=h 2023-07-29T00:17:52.887 DEBUG:boto:Data: 2023-07-29T00:17:52.887 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.887 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:52.888 DEBUG:boto:Port: 8001 2023-07-29T00:17:52.888 DEBUG:boto:Params: {} 2023-07-29T00:17:52.889 DEBUG:boto:Token: None 2023-07-29T00:17:52.889 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/ 2023-07-29T00:17:52.889 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T00:17:52.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T00:17:52.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008028961dd08ecc06-0064c45ab0-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.893 DEBUG:boto:b'hiiqos-55h1000falseh2023-07-29T00:15:01.033Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:52.893 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:52.894 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:52.894 DEBUG:boto:path=/hiiqos-55/?prefix=h 2023-07-29T00:17:52.894 DEBUG:boto:auth_path=/hiiqos-55/?prefix=h 2023-07-29T00:17:52.895 DEBUG:boto:Method: GET 2023-07-29T00:17:52.895 DEBUG:boto:Path: /hiiqos-55/?prefix=h 2023-07-29T00:17:52.896 DEBUG:boto:Data: 2023-07-29T00:17:52.896 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.896 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:52.897 DEBUG:boto:Port: 8000 2023-07-29T00:17:52.897 DEBUG:boto:Params: {} 2023-07-29T00:17:52.897 DEBUG:boto:Token: None 2023-07-29T00:17:52.898 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/ 2023-07-29T00:17:52.898 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T00:17:52.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T00:17:52.900 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a39f56dfd58fad1-0064c45ab0-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.901 DEBUG:boto:b'hiiqos-55h1000falseh2023-07-29T00:15:01.033Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:52.902 DEBUG:boto:path=/hiiqos-55/h 2023-07-29T00:17:52.902 DEBUG:boto:auth_path=/hiiqos-55/h 2023-07-29T00:17:52.903 DEBUG:boto:Method: HEAD 2023-07-29T00:17:52.903 DEBUG:boto:Path: /hiiqos-55/h 2023-07-29T00:17:52.903 DEBUG:boto:Data: 2023-07-29T00:17:52.904 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.904 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:52.904 DEBUG:boto:Port: 8001 2023-07-29T00:17:52.905 DEBUG:boto:Params: {} 2023-07-29T00:17:52.905 DEBUG:boto:Token: None 2023-07-29T00:17:52.906 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/h 2023-07-29T00:17:52.906 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:skFzbXXcGibuPHPSRPhi/hHw9r4= 2023-07-29T00:17:52.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:skFzbXXcGibuPHPSRPhi/hHw9r4=', 'Content-Length': '0'} 2023-07-29T00:17:52.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae27e2664064634f-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.909 DEBUG:boto:path=/hiiqos-55/h 2023-07-29T00:17:52.909 DEBUG:boto:auth_path=/hiiqos-55/h 2023-07-29T00:17:52.909 DEBUG:boto:Method: HEAD 2023-07-29T00:17:52.910 DEBUG:boto:Path: /hiiqos-55/h 2023-07-29T00:17:52.910 DEBUG:boto:Data: 2023-07-29T00:17:52.910 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.911 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:52.911 DEBUG:boto:Port: 8000 2023-07-29T00:17:52.911 DEBUG:boto:Params: {} 2023-07-29T00:17:52.912 DEBUG:boto:Token: None 2023-07-29T00:17:52.912 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/h 2023-07-29T00:17:52.913 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:skFzbXXcGibuPHPSRPhi/hHw9r4= 2023-07-29T00:17:52.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:skFzbXXcGibuPHPSRPhi/hHw9r4=', 'Content-Length': '0'} 2023-07-29T00:17:52.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009723aa127059076b-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-07-29T00:17:52.916 DEBUG:boto:path=/hiiqos-55/h 2023-07-29T00:17:52.916 DEBUG:boto:auth_path=/hiiqos-55/h 2023-07-29T00:17:52.917 DEBUG:boto:Method: GET 2023-07-29T00:17:52.917 DEBUG:boto:Path: /hiiqos-55/h 2023-07-29T00:17:52.918 DEBUG:boto:Data: 2023-07-29T00:17:52.918 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.918 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:52.919 DEBUG:boto:Port: 8001 2023-07-29T00:17:52.919 DEBUG:boto:Params: {} 2023-07-29T00:17:52.919 DEBUG:boto:Token: None 2023-07-29T00:17:52.920 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/h 2023-07-29T00:17:52.920 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1AhkFk7LYGQONbZN3SfIj+1M53c= 2023-07-29T00:17:52.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1AhkFk7LYGQONbZN3SfIj+1M53c=', 'Content-Length': '0'} 2023-07-29T00:17:52.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d65cf23586dcd09-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.964 DEBUG:boto:path=/hiiqos-55/h 2023-07-29T00:17:52.965 DEBUG:boto:auth_path=/hiiqos-55/h 2023-07-29T00:17:52.966 DEBUG:boto:Method: GET 2023-07-29T00:17:52.967 DEBUG:boto:Path: /hiiqos-55/h 2023-07-29T00:17:52.968 DEBUG:boto:Data: 2023-07-29T00:17:52.969 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.970 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:52.970 DEBUG:boto:Port: 8000 2023-07-29T00:17:52.971 DEBUG:boto:Params: {} 2023-07-29T00:17:52.972 DEBUG:boto:Token: None 2023-07-29T00:17:52.973 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/h 2023-07-29T00:17:52.974 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1AhkFk7LYGQONbZN3SfIj+1M53c= 2023-07-29T00:17:52.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1AhkFk7LYGQONbZN3SfIj+1M53c=', 'Content-Length': '0'} 2023-07-29T00:17:52.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085e1812d984bf485-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.979 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:52.979 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:52.980 DEBUG:boto:path=/hiiqos-55/?prefix=h 2023-07-29T00:17:52.981 DEBUG:boto:auth_path=/hiiqos-55/?prefix=h 2023-07-29T00:17:52.982 DEBUG:boto:Method: GET 2023-07-29T00:17:52.982 DEBUG:boto:Path: /hiiqos-55/?prefix=h 2023-07-29T00:17:52.983 DEBUG:boto:Data: 2023-07-29T00:17:52.984 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.985 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:52.985 DEBUG:boto:Port: 8001 2023-07-29T00:17:52.986 DEBUG:boto:Params: {} 2023-07-29T00:17:52.987 DEBUG:boto:Token: None 2023-07-29T00:17:52.988 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/ 2023-07-29T00:17:52.989 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T00:17:52.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T00:17:52.993 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c1403d11bbf1693-0064c45ab0-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:52.993 DEBUG:boto:b'hiiqos-55h1000falseh2023-07-29T00:15:01.033Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:52.994 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:52.994 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:52.995 DEBUG:boto:path=/hiiqos-55/?prefix=h 2023-07-29T00:17:52.995 DEBUG:boto:auth_path=/hiiqos-55/?prefix=h 2023-07-29T00:17:52.995 DEBUG:boto:Method: GET 2023-07-29T00:17:52.996 DEBUG:boto:Path: /hiiqos-55/?prefix=h 2023-07-29T00:17:52.996 DEBUG:boto:Data: 2023-07-29T00:17:52.996 DEBUG:boto:Headers: {} 2023-07-29T00:17:52.997 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:52.997 DEBUG:boto:Port: 8000 2023-07-29T00:17:52.997 DEBUG:boto:Params: {} 2023-07-29T00:17:52.998 DEBUG:boto:Token: None 2023-07-29T00:17:52.998 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:52 GMT /hiiqos-55/ 2023-07-29T00:17:52.999 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T00:17:52.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T00:17:53.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035f5b21c532acc62-0064c45ab0-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:53.002 DEBUG:boto:b'hiiqos-55h1000falseh2023-07-29T00:15:01.033Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:53.002 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-55 zones={a2, a1} 2023-07-29T00:17:53.003 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'hiiqos-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:17:53.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'hiiqos-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:17:53.004 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket hiiqos-55 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:17:53.053 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:17:53.053 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:17:53.091 INFO:teuthology.orchestra.run.smithi101.stdout:tenant: 2023-07-29T00:17:53.091 INFO:teuthology.orchestra.run.smithi101.stdout:bucket name: hiiqos-55 2023-07-29T00:17:53.092 INFO:teuthology.orchestra.run.smithi101.stdout:total entries: 8 2023-07-29T00:17:53.117 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'hiiqos-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:17:53.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'hiiqos-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:17:53.118 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket hiiqos-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:17:53.161 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:17:53.162 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:17:53.202 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:17:53.202 INFO:teuthology.orchestra.run.smithi101.stdout: "layout": { 2023-07-29T00:17:53.202 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding": "None", 2023-07-29T00:17:53.202 INFO:teuthology.orchestra.run.smithi101.stdout: "current_index": { 2023-07-29T00:17:53.202 INFO:teuthology.orchestra.run.smithi101.stdout: "gen": 2, 2023-07-29T00:17:53.203 INFO:teuthology.orchestra.run.smithi101.stdout: "layout": { 2023-07-29T00:17:53.203 INFO:teuthology.orchestra.run.smithi101.stdout: "type": "Normal", 2023-07-29T00:17:53.203 INFO:teuthology.orchestra.run.smithi101.stdout: "normal": { 2023-07-29T00:17:53.203 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 3, 2023-07-29T00:17:53.203 INFO:teuthology.orchestra.run.smithi101.stdout: "hash_type": "Mod" 2023-07-29T00:17:53.203 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:17:53.203 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:17:53.204 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:17:53.204 INFO:teuthology.orchestra.run.smithi101.stdout: "logs": [ 2023-07-29T00:17:53.204 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:17:53.204 INFO:teuthology.orchestra.run.smithi101.stdout: "gen": 0, 2023-07-29T00:17:53.204 INFO:teuthology.orchestra.run.smithi101.stdout: "layout": { 2023-07-29T00:17:53.204 INFO:teuthology.orchestra.run.smithi101.stdout: "type": "InIndex", 2023-07-29T00:17:53.205 INFO:teuthology.orchestra.run.smithi101.stdout: "in_index": { 2023-07-29T00:17:53.205 INFO:teuthology.orchestra.run.smithi101.stdout: "gen": 0, 2023-07-29T00:17:53.205 INFO:teuthology.orchestra.run.smithi101.stdout: "layout": { 2023-07-29T00:17:53.205 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 11, 2023-07-29T00:17:53.205 INFO:teuthology.orchestra.run.smithi101.stdout: "hash_type": "Mod" 2023-07-29T00:17:53.205 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:17:53.205 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:17:53.206 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:17:53.206 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:17:53.206 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:17:53.206 INFO:teuthology.orchestra.run.smithi101.stdout: "gen": 1, 2023-07-29T00:17:53.206 INFO:teuthology.orchestra.run.smithi101.stdout: "layout": { 2023-07-29T00:17:53.206 INFO:teuthology.orchestra.run.smithi101.stdout: "type": "InIndex", 2023-07-29T00:17:53.206 INFO:teuthology.orchestra.run.smithi101.stdout: "in_index": { 2023-07-29T00:17:53.207 INFO:teuthology.orchestra.run.smithi101.stdout: "gen": 1, 2023-07-29T00:17:53.207 INFO:teuthology.orchestra.run.smithi101.stdout: "layout": { 2023-07-29T00:17:53.207 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 3, 2023-07-29T00:17:53.207 INFO:teuthology.orchestra.run.smithi101.stdout: "hash_type": "Mod" 2023-07-29T00:17:53.207 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:17:53.207 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:17:53.208 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:17:53.208 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:17:53.208 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:17:53.208 INFO:teuthology.orchestra.run.smithi101.stdout: "gen": 2, 2023-07-29T00:17:53.208 INFO:teuthology.orchestra.run.smithi101.stdout: "layout": { 2023-07-29T00:17:53.208 INFO:teuthology.orchestra.run.smithi101.stdout: "type": "InIndex", 2023-07-29T00:17:53.208 INFO:teuthology.orchestra.run.smithi101.stdout: "in_index": { 2023-07-29T00:17:53.209 INFO:teuthology.orchestra.run.smithi101.stdout: "gen": 2, 2023-07-29T00:17:53.209 INFO:teuthology.orchestra.run.smithi101.stdout: "layout": { 2023-07-29T00:17:53.209 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 3, 2023-07-29T00:17:53.209 INFO:teuthology.orchestra.run.smithi101.stdout: "hash_type": "Mod" 2023-07-29T00:17:53.209 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:17:53.209 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:17:53.209 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:17:53.210 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:17:53.210 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:17:53.210 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:17:53.210 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:17:53.211 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-55 2023-07-29T00:17:53.211 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:17:53.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:17:53.212 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:17:53.304 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:17:53.305 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:17:53.336 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:17:53.292+0000 7ff75a135880 1 waiting to reach latest gen 2, on 1.. 2023-07-29T00:17:58.337 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:17:58.293+0000 7ff75a135880 1 bucket sync caught up with empty source 2023-07-29T00:17:58.338 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:17:58.293+0000 7ff75a135880 0 bucket checkpoint complete 2023-07-29T00:17:58.355 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-55 2023-07-29T00:17:58.355 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:17:58.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:17:58.356 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:17:58.413 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:17:58.413 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:17:58.449 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:17:58.405+0000 7fd187f26880 1 bucket sync caught up with source: 2023-07-29T00:17:58.450 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.714.6, , , 00000000001.1347.6, 00000000001.707.6, , , 00000000002.386.6, , 00000000002.1169.6] 2023-07-29T00:17:58.450 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000002.714.6, , , 00000000001.1347.6, 00000000001.707.6, , , 00000000002.386.6, , 00000000002.1169.6] 2023-07-29T00:17:58.450 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:17:58.405+0000 7fd187f26880 0 bucket checkpoint complete 2023-07-29T00:17:58.462 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-55 zones={a2, a1} 2023-07-29T00:17:58.463 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:58.463 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:58.464 DEBUG:boto:Method: HEAD 2023-07-29T00:17:58.464 DEBUG:boto:Path: /hiiqos-55/ 2023-07-29T00:17:58.464 DEBUG:boto:Data: 2023-07-29T00:17:58.465 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.466 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:58.467 DEBUG:boto:Port: 8001 2023-07-29T00:17:58.468 DEBUG:boto:Params: {} 2023-07-29T00:17:58.469 DEBUG:boto:Token: None 2023-07-29T00:17:58.470 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/ 2023-07-29T00:17:58.471 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:t3BmvdFePG5jN7YR86Da8NTlLX8= 2023-07-29T00:17:58.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t3BmvdFePG5jN7YR86Da8NTlLX8=', 'Content-Length': '0'} 2023-07-29T00:17:58.473 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000595b2192f6874fbe-0064c45ab6-1663-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.474 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:58.474 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:58.475 DEBUG:boto:Method: HEAD 2023-07-29T00:17:58.475 DEBUG:boto:Path: /hiiqos-55/ 2023-07-29T00:17:58.475 DEBUG:boto:Data: 2023-07-29T00:17:58.476 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.476 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:58.476 DEBUG:boto:Port: 8000 2023-07-29T00:17:58.477 DEBUG:boto:Params: {} 2023-07-29T00:17:58.477 DEBUG:boto:Token: None 2023-07-29T00:17:58.477 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/ 2023-07-29T00:17:58.478 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:t3BmvdFePG5jN7YR86Da8NTlLX8= 2023-07-29T00:17:58.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t3BmvdFePG5jN7YR86Da8NTlLX8=', 'Content-Length': '0'} 2023-07-29T00:17:58.480 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000062d5f1618c5e584e-0064c45ab6-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.481 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-29T00:17:58.481 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:58.481 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:58.482 DEBUG:boto:path=/hiiqos-55/?versions&max-keys=999 2023-07-29T00:17:58.482 DEBUG:boto:auth_path=/hiiqos-55/?versions&max-keys=999 2023-07-29T00:17:58.482 DEBUG:boto:Method: GET 2023-07-29T00:17:58.483 DEBUG:boto:Path: /hiiqos-55/?versions&max-keys=999 2023-07-29T00:17:58.483 DEBUG:boto:Data: 2023-07-29T00:17:58.483 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.484 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:58.484 DEBUG:boto:Port: 8001 2023-07-29T00:17:58.485 DEBUG:boto:Params: {} 2023-07-29T00:17:58.485 DEBUG:boto:Token: None 2023-07-29T00:17:58.485 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/?versions 2023-07-29T00:17:58.486 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nkO23lZHuGy1GRu9+rLtGNhwFFA= 2023-07-29T00:17:58.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nkO23lZHuGy1GRu9+rLtGNhwFFA=', 'Content-Length': '0'} 2023-07-29T00:17:58.488 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080b4886b16007f7d-0064c45ab6-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.489 DEBUG:boto:b'hiiqos-55999falseanulltrue2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-07-29T00:15:00.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-07-29T00:15:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-07-29T00:15:01.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-07-29T00:15:01.033Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:58.490 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-07-29T00:17:58.490 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-07-29T00:17:58.490 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-07-29T00:17:58.491 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-07-29T00:17:58.491 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-07-29T00:17:58.491 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-07-29T00:17:58.492 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-07-29T00:17:58.492 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-07-29T00:17:58.492 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-29T00:17:58.493 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:58.493 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:58.493 DEBUG:boto:path=/hiiqos-55/?versions&max-keys=999 2023-07-29T00:17:58.494 DEBUG:boto:auth_path=/hiiqos-55/?versions&max-keys=999 2023-07-29T00:17:58.494 DEBUG:boto:Method: GET 2023-07-29T00:17:58.494 DEBUG:boto:Path: /hiiqos-55/?versions&max-keys=999 2023-07-29T00:17:58.495 DEBUG:boto:Data: 2023-07-29T00:17:58.495 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.495 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:58.496 DEBUG:boto:Port: 8000 2023-07-29T00:17:58.496 DEBUG:boto:Params: {} 2023-07-29T00:17:58.497 DEBUG:boto:Token: None 2023-07-29T00:17:58.497 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/?versions 2023-07-29T00:17:58.497 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nkO23lZHuGy1GRu9+rLtGNhwFFA= 2023-07-29T00:17:58.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nkO23lZHuGy1GRu9+rLtGNhwFFA=', 'Content-Length': '0'} 2023-07-29T00:17:58.500 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd795f2900cae01a-0064c45ab6-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.500 DEBUG:boto:b'hiiqos-55999falseanulltrue2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-07-29T00:15:00.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-07-29T00:15:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-07-29T00:15:01.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-07-29T00:15:01.033Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:58.501 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-07-29T00:17:58.501 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-07-29T00:17:58.502 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-07-29T00:17:58.502 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-07-29T00:17:58.502 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-07-29T00:17:58.503 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-07-29T00:17:58.503 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-07-29T00:17:58.503 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-07-29T00:17:58.504 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:58.504 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:58.505 DEBUG:boto:path=/hiiqos-55/?versions&max-keys=999 2023-07-29T00:17:58.505 DEBUG:boto:auth_path=/hiiqos-55/?versions&max-keys=999 2023-07-29T00:17:58.505 DEBUG:boto:Method: GET 2023-07-29T00:17:58.506 DEBUG:boto:Path: /hiiqos-55/?versions&max-keys=999 2023-07-29T00:17:58.506 DEBUG:boto:Data: 2023-07-29T00:17:58.506 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.507 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:58.507 DEBUG:boto:Port: 8001 2023-07-29T00:17:58.507 DEBUG:boto:Params: {} 2023-07-29T00:17:58.508 DEBUG:boto:Token: None 2023-07-29T00:17:58.508 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/?versions 2023-07-29T00:17:58.509 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nkO23lZHuGy1GRu9+rLtGNhwFFA= 2023-07-29T00:17:58.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nkO23lZHuGy1GRu9+rLtGNhwFFA=', 'Content-Length': '0'} 2023-07-29T00:17:58.511 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000397f81de5e512417-0064c45ab6-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.512 DEBUG:boto:b'hiiqos-55999falseanulltrue2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-07-29T00:15:00.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-07-29T00:15:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-07-29T00:15:01.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-07-29T00:15:01.033Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:58.515 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:58.515 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:58.516 DEBUG:boto:path=/hiiqos-55/?versions&max-keys=999 2023-07-29T00:17:58.517 DEBUG:boto:auth_path=/hiiqos-55/?versions&max-keys=999 2023-07-29T00:17:58.518 DEBUG:boto:Method: GET 2023-07-29T00:17:58.518 DEBUG:boto:Path: /hiiqos-55/?versions&max-keys=999 2023-07-29T00:17:58.519 DEBUG:boto:Data: 2023-07-29T00:17:58.519 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.520 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:58.521 DEBUG:boto:Port: 8000 2023-07-29T00:17:58.521 DEBUG:boto:Params: {} 2023-07-29T00:17:58.522 DEBUG:boto:Token: None 2023-07-29T00:17:58.522 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/?versions 2023-07-29T00:17:58.522 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nkO23lZHuGy1GRu9+rLtGNhwFFA= 2023-07-29T00:17:58.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nkO23lZHuGy1GRu9+rLtGNhwFFA=', 'Content-Length': '0'} 2023-07-29T00:17:58.525 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000875153d43b6a0d9f-0064c45ab6-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.525 DEBUG:boto:b'hiiqos-55999falseanulltrue2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-07-29T00:15:00.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-07-29T00:15:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-07-29T00:15:01.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-07-29T00:15:01.033Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:58.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-07-29T00:17:58.526 DEBUG:boto:path=/hiiqos-55/a 2023-07-29T00:17:58.527 DEBUG:boto:auth_path=/hiiqos-55/a 2023-07-29T00:17:58.527 DEBUG:boto:path=/hiiqos-55/a?versionId=null 2023-07-29T00:17:58.528 DEBUG:boto:auth_path=/hiiqos-55/a?versionId=null 2023-07-29T00:17:58.528 DEBUG:boto:Method: GET 2023-07-29T00:17:58.528 DEBUG:boto:Path: /hiiqos-55/a?versionId=null 2023-07-29T00:17:58.529 DEBUG:boto:Data: 2023-07-29T00:17:58.529 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.529 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:58.530 DEBUG:boto:Port: 8001 2023-07-29T00:17:58.530 DEBUG:boto:Params: {} 2023-07-29T00:17:58.530 DEBUG:boto:Token: None 2023-07-29T00:17:58.531 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/a?versionId=null 2023-07-29T00:17:58.531 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FLpEUX7sV/aMDO7rT2S5vRKCWbQ= 2023-07-29T00:17:58.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FLpEUX7sV/aMDO7rT2S5vRKCWbQ=', 'Content-Length': '0'} 2023-07-29T00:17:58.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc5b4220253a8d4a-0064c45ab6-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.576 DEBUG:boto:path=/hiiqos-55/a 2023-07-29T00:17:58.577 DEBUG:boto:auth_path=/hiiqos-55/a 2023-07-29T00:17:58.577 DEBUG:boto:path=/hiiqos-55/a?versionId=null 2023-07-29T00:17:58.578 DEBUG:boto:auth_path=/hiiqos-55/a?versionId=null 2023-07-29T00:17:58.578 DEBUG:boto:Method: GET 2023-07-29T00:17:58.578 DEBUG:boto:Path: /hiiqos-55/a?versionId=null 2023-07-29T00:17:58.579 DEBUG:boto:Data: 2023-07-29T00:17:58.579 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.579 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:58.580 DEBUG:boto:Port: 8000 2023-07-29T00:17:58.580 DEBUG:boto:Params: {} 2023-07-29T00:17:58.581 DEBUG:boto:Token: None 2023-07-29T00:17:58.581 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/a?versionId=null 2023-07-29T00:17:58.582 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FLpEUX7sV/aMDO7rT2S5vRKCWbQ= 2023-07-29T00:17:58.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FLpEUX7sV/aMDO7rT2S5vRKCWbQ=', 'Content-Length': '0'} 2023-07-29T00:17:58.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000208daf13698bf92d-0064c45ab6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.585 DEBUG:boto:path=/hiiqos-55/a 2023-07-29T00:17:58.585 DEBUG:boto:auth_path=/hiiqos-55/a 2023-07-29T00:17:58.585 DEBUG:boto:path=/hiiqos-55/a?versionId=null 2023-07-29T00:17:58.586 DEBUG:boto:auth_path=/hiiqos-55/a?versionId=null 2023-07-29T00:17:58.586 DEBUG:boto:Method: HEAD 2023-07-29T00:17:58.586 DEBUG:boto:Path: /hiiqos-55/a?versionId=null 2023-07-29T00:17:58.587 DEBUG:boto:Data: 2023-07-29T00:17:58.587 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.587 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:58.588 DEBUG:boto:Port: 8001 2023-07-29T00:17:58.588 DEBUG:boto:Params: {} 2023-07-29T00:17:58.588 DEBUG:boto:Token: None 2023-07-29T00:17:58.589 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/a?versionId=null 2023-07-29T00:17:58.589 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:x1PArLs/dXmbVDoWX7D8h1wmzks= 2023-07-29T00:17:58.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:x1PArLs/dXmbVDoWX7D8h1wmzks=', 'Content-Length': '0'} 2023-07-29T00:17:58.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080fb246b4c3d97c5-0064c45ab6-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.592 DEBUG:boto:path=/hiiqos-55/a 2023-07-29T00:17:58.592 DEBUG:boto:auth_path=/hiiqos-55/a 2023-07-29T00:17:58.593 DEBUG:boto:path=/hiiqos-55/a?versionId=null 2023-07-29T00:17:58.593 DEBUG:boto:auth_path=/hiiqos-55/a?versionId=null 2023-07-29T00:17:58.593 DEBUG:boto:Method: HEAD 2023-07-29T00:17:58.594 DEBUG:boto:Path: /hiiqos-55/a?versionId=null 2023-07-29T00:17:58.594 DEBUG:boto:Data: 2023-07-29T00:17:58.594 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.595 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:58.595 DEBUG:boto:Port: 8000 2023-07-29T00:17:58.595 DEBUG:boto:Params: {} 2023-07-29T00:17:58.596 DEBUG:boto:Token: None 2023-07-29T00:17:58.596 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/a?versionId=null 2023-07-29T00:17:58.596 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:x1PArLs/dXmbVDoWX7D8h1wmzks= 2023-07-29T00:17:58.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:x1PArLs/dXmbVDoWX7D8h1wmzks=', 'Content-Length': '0'} 2023-07-29T00:17:58.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d516d7e44917c09d-0064c45ab6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-07-29T00:17:58.599 DEBUG:boto:path=/hiiqos-55/a 2023-07-29T00:17:58.600 DEBUG:boto:auth_path=/hiiqos-55/a 2023-07-29T00:17:58.600 DEBUG:boto:path=/hiiqos-55/a?versionId=null 2023-07-29T00:17:58.601 DEBUG:boto:auth_path=/hiiqos-55/a?versionId=null 2023-07-29T00:17:58.601 DEBUG:boto:Method: GET 2023-07-29T00:17:58.601 DEBUG:boto:Path: /hiiqos-55/a?versionId=null 2023-07-29T00:17:58.602 DEBUG:boto:Data: 2023-07-29T00:17:58.602 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.602 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:58.603 DEBUG:boto:Port: 8001 2023-07-29T00:17:58.603 DEBUG:boto:Params: {} 2023-07-29T00:17:58.603 DEBUG:boto:Token: None 2023-07-29T00:17:58.604 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/a?versionId=null 2023-07-29T00:17:58.604 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FLpEUX7sV/aMDO7rT2S5vRKCWbQ= 2023-07-29T00:17:58.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FLpEUX7sV/aMDO7rT2S5vRKCWbQ=', 'Content-Length': '0'} 2023-07-29T00:17:58.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d3b5c37d359afd2-0064c45ab6-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.607 DEBUG:boto:path=/hiiqos-55/a 2023-07-29T00:17:58.608 DEBUG:boto:auth_path=/hiiqos-55/a 2023-07-29T00:17:58.608 DEBUG:boto:path=/hiiqos-55/a?versionId=null 2023-07-29T00:17:58.608 DEBUG:boto:auth_path=/hiiqos-55/a?versionId=null 2023-07-29T00:17:58.609 DEBUG:boto:Method: GET 2023-07-29T00:17:58.609 DEBUG:boto:Path: /hiiqos-55/a?versionId=null 2023-07-29T00:17:58.609 DEBUG:boto:Data: 2023-07-29T00:17:58.610 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.610 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:58.610 DEBUG:boto:Port: 8000 2023-07-29T00:17:58.611 DEBUG:boto:Params: {} 2023-07-29T00:17:58.611 DEBUG:boto:Token: None 2023-07-29T00:17:58.612 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/a?versionId=null 2023-07-29T00:17:58.612 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FLpEUX7sV/aMDO7rT2S5vRKCWbQ= 2023-07-29T00:17:58.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FLpEUX7sV/aMDO7rT2S5vRKCWbQ=', 'Content-Length': '0'} 2023-07-29T00:17:58.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000154b1861952f482c-0064c45ab6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.615 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:58.615 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:58.616 DEBUG:boto:path=/hiiqos-55/?prefix=a 2023-07-29T00:17:58.616 DEBUG:boto:auth_path=/hiiqos-55/?prefix=a 2023-07-29T00:17:58.616 DEBUG:boto:Method: GET 2023-07-29T00:17:58.617 DEBUG:boto:Path: /hiiqos-55/?prefix=a 2023-07-29T00:17:58.617 DEBUG:boto:Data: 2023-07-29T00:17:58.617 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.618 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:58.618 DEBUG:boto:Port: 8001 2023-07-29T00:17:58.618 DEBUG:boto:Params: {} 2023-07-29T00:17:58.619 DEBUG:boto:Token: None 2023-07-29T00:17:58.619 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/ 2023-07-29T00:17:58.620 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0= 2023-07-29T00:17:58.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0=', 'Content-Length': '0'} 2023-07-29T00:17:58.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000195a36f67e346268-0064c45ab6-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.623 DEBUG:boto:b'hiiqos-55a1000falsea2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:58.623 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:58.624 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:58.624 DEBUG:boto:path=/hiiqos-55/?prefix=a 2023-07-29T00:17:58.624 DEBUG:boto:auth_path=/hiiqos-55/?prefix=a 2023-07-29T00:17:58.625 DEBUG:boto:Method: GET 2023-07-29T00:17:58.625 DEBUG:boto:Path: /hiiqos-55/?prefix=a 2023-07-29T00:17:58.625 DEBUG:boto:Data: 2023-07-29T00:17:58.626 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.626 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:58.627 DEBUG:boto:Port: 8000 2023-07-29T00:17:58.627 DEBUG:boto:Params: {} 2023-07-29T00:17:58.627 DEBUG:boto:Token: None 2023-07-29T00:17:58.628 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/ 2023-07-29T00:17:58.628 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0= 2023-07-29T00:17:58.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0=', 'Content-Length': '0'} 2023-07-29T00:17:58.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000451ff7ca570b4be3-0064c45ab6-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.631 DEBUG:boto:b'hiiqos-55a1000falsea2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:58.631 DEBUG:boto:path=/hiiqos-55/a 2023-07-29T00:17:58.632 DEBUG:boto:auth_path=/hiiqos-55/a 2023-07-29T00:17:58.632 DEBUG:boto:Method: HEAD 2023-07-29T00:17:58.632 DEBUG:boto:Path: /hiiqos-55/a 2023-07-29T00:17:58.633 DEBUG:boto:Data: 2023-07-29T00:17:58.633 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.633 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:58.634 DEBUG:boto:Port: 8001 2023-07-29T00:17:58.634 DEBUG:boto:Params: {} 2023-07-29T00:17:58.634 DEBUG:boto:Token: None 2023-07-29T00:17:58.635 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/a 2023-07-29T00:17:58.635 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vaTRr3LNOYjSJ/0XsCl26xX2gD8= 2023-07-29T00:17:58.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vaTRr3LNOYjSJ/0XsCl26xX2gD8=', 'Content-Length': '0'} 2023-07-29T00:17:58.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008e03213ca6ebee3-0064c45ab6-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.638 DEBUG:boto:path=/hiiqos-55/a 2023-07-29T00:17:58.638 DEBUG:boto:auth_path=/hiiqos-55/a 2023-07-29T00:17:58.639 DEBUG:boto:Method: HEAD 2023-07-29T00:17:58.639 DEBUG:boto:Path: /hiiqos-55/a 2023-07-29T00:17:58.639 DEBUG:boto:Data: 2023-07-29T00:17:58.640 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.640 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:58.640 DEBUG:boto:Port: 8000 2023-07-29T00:17:58.641 DEBUG:boto:Params: {} 2023-07-29T00:17:58.641 DEBUG:boto:Token: None 2023-07-29T00:17:58.641 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/a 2023-07-29T00:17:58.642 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vaTRr3LNOYjSJ/0XsCl26xX2gD8= 2023-07-29T00:17:58.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vaTRr3LNOYjSJ/0XsCl26xX2gD8=', 'Content-Length': '0'} 2023-07-29T00:17:58.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000650f9da0342e2027-0064c45ab6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-07-29T00:17:58.645 DEBUG:boto:path=/hiiqos-55/a 2023-07-29T00:17:58.645 DEBUG:boto:auth_path=/hiiqos-55/a 2023-07-29T00:17:58.645 DEBUG:boto:Method: GET 2023-07-29T00:17:58.646 DEBUG:boto:Path: /hiiqos-55/a 2023-07-29T00:17:58.646 DEBUG:boto:Data: 2023-07-29T00:17:58.646 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.647 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:58.647 DEBUG:boto:Port: 8001 2023-07-29T00:17:58.647 DEBUG:boto:Params: {} 2023-07-29T00:17:58.648 DEBUG:boto:Token: None 2023-07-29T00:17:58.648 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/a 2023-07-29T00:17:58.649 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jbWIkcwnNqnA+a2EoaUQKHtDObo= 2023-07-29T00:17:58.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jbWIkcwnNqnA+a2EoaUQKHtDObo=', 'Content-Length': '0'} 2023-07-29T00:17:58.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000129a48f453ee88e8-0064c45ab6-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.651 DEBUG:boto:path=/hiiqos-55/a 2023-07-29T00:17:58.652 DEBUG:boto:auth_path=/hiiqos-55/a 2023-07-29T00:17:58.652 DEBUG:boto:Method: GET 2023-07-29T00:17:58.652 DEBUG:boto:Path: /hiiqos-55/a 2023-07-29T00:17:58.653 DEBUG:boto:Data: 2023-07-29T00:17:58.653 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.653 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:58.654 DEBUG:boto:Port: 8000 2023-07-29T00:17:58.654 DEBUG:boto:Params: {} 2023-07-29T00:17:58.654 DEBUG:boto:Token: None 2023-07-29T00:17:58.655 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/a 2023-07-29T00:17:58.655 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jbWIkcwnNqnA+a2EoaUQKHtDObo= 2023-07-29T00:17:58.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jbWIkcwnNqnA+a2EoaUQKHtDObo=', 'Content-Length': '0'} 2023-07-29T00:17:58.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043f6b9a6e65329ef-0064c45ab6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.658 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:58.659 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:58.659 DEBUG:boto:path=/hiiqos-55/?prefix=a 2023-07-29T00:17:58.659 DEBUG:boto:auth_path=/hiiqos-55/?prefix=a 2023-07-29T00:17:58.660 DEBUG:boto:Method: GET 2023-07-29T00:17:58.660 DEBUG:boto:Path: /hiiqos-55/?prefix=a 2023-07-29T00:17:58.660 DEBUG:boto:Data: 2023-07-29T00:17:58.661 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.661 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:58.661 DEBUG:boto:Port: 8001 2023-07-29T00:17:58.662 DEBUG:boto:Params: {} 2023-07-29T00:17:58.662 DEBUG:boto:Token: None 2023-07-29T00:17:58.663 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/ 2023-07-29T00:17:58.663 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0= 2023-07-29T00:17:58.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0=', 'Content-Length': '0'} 2023-07-29T00:17:58.665 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bcaccaaaf5ab687-0064c45ab6-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.666 DEBUG:boto:b'hiiqos-55a1000falsea2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:58.666 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:58.667 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:58.667 DEBUG:boto:path=/hiiqos-55/?prefix=a 2023-07-29T00:17:58.668 DEBUG:boto:auth_path=/hiiqos-55/?prefix=a 2023-07-29T00:17:58.668 DEBUG:boto:Method: GET 2023-07-29T00:17:58.668 DEBUG:boto:Path: /hiiqos-55/?prefix=a 2023-07-29T00:17:58.669 DEBUG:boto:Data: 2023-07-29T00:17:58.669 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.669 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:58.670 DEBUG:boto:Port: 8000 2023-07-29T00:17:58.670 DEBUG:boto:Params: {} 2023-07-29T00:17:58.671 DEBUG:boto:Token: None 2023-07-29T00:17:58.671 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/ 2023-07-29T00:17:58.671 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0= 2023-07-29T00:17:58.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0=', 'Content-Length': '0'} 2023-07-29T00:17:58.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6741626a4e7e98f-0064c45ab6-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.674 DEBUG:boto:b'hiiqos-55a1000falsea2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:58.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-07-29T00:17:58.675 DEBUG:boto:path=/hiiqos-55/b 2023-07-29T00:17:58.675 DEBUG:boto:auth_path=/hiiqos-55/b 2023-07-29T00:17:58.676 DEBUG:boto:path=/hiiqos-55/b?versionId=null 2023-07-29T00:17:58.677 DEBUG:boto:auth_path=/hiiqos-55/b?versionId=null 2023-07-29T00:17:58.677 DEBUG:boto:Method: GET 2023-07-29T00:17:58.677 DEBUG:boto:Path: /hiiqos-55/b?versionId=null 2023-07-29T00:17:58.678 DEBUG:boto:Data: 2023-07-29T00:17:58.678 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.678 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:58.679 DEBUG:boto:Port: 8001 2023-07-29T00:17:58.679 DEBUG:boto:Params: {} 2023-07-29T00:17:58.680 DEBUG:boto:Token: None 2023-07-29T00:17:58.680 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/b?versionId=null 2023-07-29T00:17:58.681 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:F40X/fWMus0VglY4g75YfbW+5Gc= 2023-07-29T00:17:58.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:F40X/fWMus0VglY4g75YfbW+5Gc=', 'Content-Length': '0'} 2023-07-29T00:17:58.682 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089473e88239a03aa-0064c45ab6-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.683 DEBUG:boto:path=/hiiqos-55/b 2023-07-29T00:17:58.684 DEBUG:boto:auth_path=/hiiqos-55/b 2023-07-29T00:17:58.684 DEBUG:boto:path=/hiiqos-55/b?versionId=null 2023-07-29T00:17:58.684 DEBUG:boto:auth_path=/hiiqos-55/b?versionId=null 2023-07-29T00:17:58.685 DEBUG:boto:Method: GET 2023-07-29T00:17:58.685 DEBUG:boto:Path: /hiiqos-55/b?versionId=null 2023-07-29T00:17:58.685 DEBUG:boto:Data: 2023-07-29T00:17:58.686 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.686 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:58.686 DEBUG:boto:Port: 8000 2023-07-29T00:17:58.687 DEBUG:boto:Params: {} 2023-07-29T00:17:58.687 DEBUG:boto:Token: None 2023-07-29T00:17:58.688 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/b?versionId=null 2023-07-29T00:17:58.688 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:F40X/fWMus0VglY4g75YfbW+5Gc= 2023-07-29T00:17:58.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:F40X/fWMus0VglY4g75YfbW+5Gc=', 'Content-Length': '0'} 2023-07-29T00:17:58.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000536bd1a74974ad15-0064c45ab6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.691 DEBUG:boto:path=/hiiqos-55/b 2023-07-29T00:17:58.691 DEBUG:boto:auth_path=/hiiqos-55/b 2023-07-29T00:17:58.692 DEBUG:boto:path=/hiiqos-55/b?versionId=null 2023-07-29T00:17:58.692 DEBUG:boto:auth_path=/hiiqos-55/b?versionId=null 2023-07-29T00:17:58.692 DEBUG:boto:Method: HEAD 2023-07-29T00:17:58.693 DEBUG:boto:Path: /hiiqos-55/b?versionId=null 2023-07-29T00:17:58.693 DEBUG:boto:Data: 2023-07-29T00:17:58.693 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.694 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:58.694 DEBUG:boto:Port: 8001 2023-07-29T00:17:58.694 DEBUG:boto:Params: {} 2023-07-29T00:17:58.695 DEBUG:boto:Token: None 2023-07-29T00:17:58.695 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/b?versionId=null 2023-07-29T00:17:58.696 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KM9msgrVxaszSpAPyPsyIBQoi4c= 2023-07-29T00:17:58.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KM9msgrVxaszSpAPyPsyIBQoi4c=', 'Content-Length': '0'} 2023-07-29T00:17:58.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c30d8b135018450d-0064c45ab6-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.698 DEBUG:boto:path=/hiiqos-55/b 2023-07-29T00:17:58.698 DEBUG:boto:auth_path=/hiiqos-55/b 2023-07-29T00:17:58.699 DEBUG:boto:path=/hiiqos-55/b?versionId=null 2023-07-29T00:17:58.699 DEBUG:boto:auth_path=/hiiqos-55/b?versionId=null 2023-07-29T00:17:58.699 DEBUG:boto:Method: HEAD 2023-07-29T00:17:58.700 DEBUG:boto:Path: /hiiqos-55/b?versionId=null 2023-07-29T00:17:58.700 DEBUG:boto:Data: 2023-07-29T00:17:58.700 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.701 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:58.701 DEBUG:boto:Port: 8000 2023-07-29T00:17:58.701 DEBUG:boto:Params: {} 2023-07-29T00:17:58.702 DEBUG:boto:Token: None 2023-07-29T00:17:58.702 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/b?versionId=null 2023-07-29T00:17:58.703 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KM9msgrVxaszSpAPyPsyIBQoi4c= 2023-07-29T00:17:58.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KM9msgrVxaszSpAPyPsyIBQoi4c=', 'Content-Length': '0'} 2023-07-29T00:17:58.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3a89c5a96c33d50-0064c45ab6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-07-29T00:17:58.705 DEBUG:boto:path=/hiiqos-55/b 2023-07-29T00:17:58.706 DEBUG:boto:auth_path=/hiiqos-55/b 2023-07-29T00:17:58.706 DEBUG:boto:path=/hiiqos-55/b?versionId=null 2023-07-29T00:17:58.707 DEBUG:boto:auth_path=/hiiqos-55/b?versionId=null 2023-07-29T00:17:58.707 DEBUG:boto:Method: GET 2023-07-29T00:17:58.707 DEBUG:boto:Path: /hiiqos-55/b?versionId=null 2023-07-29T00:17:58.708 DEBUG:boto:Data: 2023-07-29T00:17:58.708 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.709 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:58.709 DEBUG:boto:Port: 8001 2023-07-29T00:17:58.709 DEBUG:boto:Params: {} 2023-07-29T00:17:58.710 DEBUG:boto:Token: None 2023-07-29T00:17:58.710 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/b?versionId=null 2023-07-29T00:17:58.711 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:F40X/fWMus0VglY4g75YfbW+5Gc= 2023-07-29T00:17:58.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:F40X/fWMus0VglY4g75YfbW+5Gc=', 'Content-Length': '0'} 2023-07-29T00:17:58.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093c5a7c05bfc6243-0064c45ab6-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.713 DEBUG:boto:path=/hiiqos-55/b 2023-07-29T00:17:58.713 DEBUG:boto:auth_path=/hiiqos-55/b 2023-07-29T00:17:58.714 DEBUG:boto:path=/hiiqos-55/b?versionId=null 2023-07-29T00:17:58.714 DEBUG:boto:auth_path=/hiiqos-55/b?versionId=null 2023-07-29T00:17:58.715 DEBUG:boto:Method: GET 2023-07-29T00:17:58.715 DEBUG:boto:Path: /hiiqos-55/b?versionId=null 2023-07-29T00:17:58.715 DEBUG:boto:Data: 2023-07-29T00:17:58.716 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.716 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:58.716 DEBUG:boto:Port: 8000 2023-07-29T00:17:58.717 DEBUG:boto:Params: {} 2023-07-29T00:17:58.717 DEBUG:boto:Token: None 2023-07-29T00:17:58.718 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/b?versionId=null 2023-07-29T00:17:58.718 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:F40X/fWMus0VglY4g75YfbW+5Gc= 2023-07-29T00:17:58.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:F40X/fWMus0VglY4g75YfbW+5Gc=', 'Content-Length': '0'} 2023-07-29T00:17:58.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001654eaf368c6dd71-0064c45ab6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.764 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:58.765 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:58.765 DEBUG:boto:path=/hiiqos-55/?prefix=b 2023-07-29T00:17:58.766 DEBUG:boto:auth_path=/hiiqos-55/?prefix=b 2023-07-29T00:17:58.766 DEBUG:boto:Method: GET 2023-07-29T00:17:58.766 DEBUG:boto:Path: /hiiqos-55/?prefix=b 2023-07-29T00:17:58.767 DEBUG:boto:Data: 2023-07-29T00:17:58.767 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.768 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:58.769 DEBUG:boto:Port: 8001 2023-07-29T00:17:58.770 DEBUG:boto:Params: {} 2023-07-29T00:17:58.771 DEBUG:boto:Token: None 2023-07-29T00:17:58.772 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/ 2023-07-29T00:17:58.773 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0= 2023-07-29T00:17:58.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0=', 'Content-Length': '0'} 2023-07-29T00:17:58.777 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a959977f4c6e829b-0064c45ab6-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.777 DEBUG:boto:b'hiiqos-55b1000falseb2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:58.778 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:58.779 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:58.779 DEBUG:boto:path=/hiiqos-55/?prefix=b 2023-07-29T00:17:58.779 DEBUG:boto:auth_path=/hiiqos-55/?prefix=b 2023-07-29T00:17:58.780 DEBUG:boto:Method: GET 2023-07-29T00:17:58.780 DEBUG:boto:Path: /hiiqos-55/?prefix=b 2023-07-29T00:17:58.781 DEBUG:boto:Data: 2023-07-29T00:17:58.781 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.781 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:58.782 DEBUG:boto:Port: 8000 2023-07-29T00:17:58.782 DEBUG:boto:Params: {} 2023-07-29T00:17:58.782 DEBUG:boto:Token: None 2023-07-29T00:17:58.783 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/ 2023-07-29T00:17:58.783 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0= 2023-07-29T00:17:58.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0=', 'Content-Length': '0'} 2023-07-29T00:17:58.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac301243df40d177-0064c45ab6-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.786 DEBUG:boto:b'hiiqos-55b1000falseb2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:58.787 DEBUG:boto:path=/hiiqos-55/b 2023-07-29T00:17:58.787 DEBUG:boto:auth_path=/hiiqos-55/b 2023-07-29T00:17:58.787 DEBUG:boto:Method: HEAD 2023-07-29T00:17:58.788 DEBUG:boto:Path: /hiiqos-55/b 2023-07-29T00:17:58.788 DEBUG:boto:Data: 2023-07-29T00:17:58.788 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.789 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:58.789 DEBUG:boto:Port: 8001 2023-07-29T00:17:58.789 DEBUG:boto:Params: {} 2023-07-29T00:17:58.790 DEBUG:boto:Token: None 2023-07-29T00:17:58.790 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/b 2023-07-29T00:17:58.790 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UZL2zVcfSioLjNM2EB5i90prEFg= 2023-07-29T00:17:58.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UZL2zVcfSioLjNM2EB5i90prEFg=', 'Content-Length': '0'} 2023-07-29T00:17:58.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000162a096e6afa9dd3-0064c45ab6-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.793 DEBUG:boto:path=/hiiqos-55/b 2023-07-29T00:17:58.793 DEBUG:boto:auth_path=/hiiqos-55/b 2023-07-29T00:17:58.793 DEBUG:boto:Method: HEAD 2023-07-29T00:17:58.794 DEBUG:boto:Path: /hiiqos-55/b 2023-07-29T00:17:58.794 DEBUG:boto:Data: 2023-07-29T00:17:58.794 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.795 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:58.795 DEBUG:boto:Port: 8000 2023-07-29T00:17:58.795 DEBUG:boto:Params: {} 2023-07-29T00:17:58.796 DEBUG:boto:Token: None 2023-07-29T00:17:58.796 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/b 2023-07-29T00:17:58.797 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UZL2zVcfSioLjNM2EB5i90prEFg= 2023-07-29T00:17:58.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UZL2zVcfSioLjNM2EB5i90prEFg=', 'Content-Length': '0'} 2023-07-29T00:17:58.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1b2e660499c6acc-0064c45ab6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-07-29T00:17:58.800 DEBUG:boto:path=/hiiqos-55/b 2023-07-29T00:17:58.800 DEBUG:boto:auth_path=/hiiqos-55/b 2023-07-29T00:17:58.801 DEBUG:boto:Method: GET 2023-07-29T00:17:58.801 DEBUG:boto:Path: /hiiqos-55/b 2023-07-29T00:17:58.802 DEBUG:boto:Data: 2023-07-29T00:17:58.802 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.803 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:58.803 DEBUG:boto:Port: 8001 2023-07-29T00:17:58.803 DEBUG:boto:Params: {} 2023-07-29T00:17:58.804 DEBUG:boto:Token: None 2023-07-29T00:17:58.804 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/b 2023-07-29T00:17:58.805 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YUgMFWbH8rRwqAFjfcnARdRhZfQ= 2023-07-29T00:17:58.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YUgMFWbH8rRwqAFjfcnARdRhZfQ=', 'Content-Length': '0'} 2023-07-29T00:17:58.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f0610a37245d16a-0064c45ab6-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.848 DEBUG:boto:path=/hiiqos-55/b 2023-07-29T00:17:58.849 DEBUG:boto:auth_path=/hiiqos-55/b 2023-07-29T00:17:58.849 DEBUG:boto:Method: GET 2023-07-29T00:17:58.850 DEBUG:boto:Path: /hiiqos-55/b 2023-07-29T00:17:58.850 DEBUG:boto:Data: 2023-07-29T00:17:58.850 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.851 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:58.851 DEBUG:boto:Port: 8000 2023-07-29T00:17:58.851 DEBUG:boto:Params: {} 2023-07-29T00:17:58.852 DEBUG:boto:Token: None 2023-07-29T00:17:58.852 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/b 2023-07-29T00:17:58.853 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YUgMFWbH8rRwqAFjfcnARdRhZfQ= 2023-07-29T00:17:58.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YUgMFWbH8rRwqAFjfcnARdRhZfQ=', 'Content-Length': '0'} 2023-07-29T00:17:58.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e222814f0641eb1-0064c45ab6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.856 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:58.856 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:58.857 DEBUG:boto:path=/hiiqos-55/?prefix=b 2023-07-29T00:17:58.857 DEBUG:boto:auth_path=/hiiqos-55/?prefix=b 2023-07-29T00:17:58.857 DEBUG:boto:Method: GET 2023-07-29T00:17:58.858 DEBUG:boto:Path: /hiiqos-55/?prefix=b 2023-07-29T00:17:58.858 DEBUG:boto:Data: 2023-07-29T00:17:58.859 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.859 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:58.859 DEBUG:boto:Port: 8001 2023-07-29T00:17:58.860 DEBUG:boto:Params: {} 2023-07-29T00:17:58.860 DEBUG:boto:Token: None 2023-07-29T00:17:58.861 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/ 2023-07-29T00:17:58.861 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0= 2023-07-29T00:17:58.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0=', 'Content-Length': '0'} 2023-07-29T00:17:58.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000950b59d064400c31-0064c45ab6-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.864 DEBUG:boto:b'hiiqos-55b1000falseb2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:58.865 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:58.865 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:58.865 DEBUG:boto:path=/hiiqos-55/?prefix=b 2023-07-29T00:17:58.866 DEBUG:boto:auth_path=/hiiqos-55/?prefix=b 2023-07-29T00:17:58.866 DEBUG:boto:Method: GET 2023-07-29T00:17:58.866 DEBUG:boto:Path: /hiiqos-55/?prefix=b 2023-07-29T00:17:58.867 DEBUG:boto:Data: 2023-07-29T00:17:58.867 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.868 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:58.868 DEBUG:boto:Port: 8000 2023-07-29T00:17:58.868 DEBUG:boto:Params: {} 2023-07-29T00:17:58.869 DEBUG:boto:Token: None 2023-07-29T00:17:58.869 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/ 2023-07-29T00:17:58.869 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0= 2023-07-29T00:17:58.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0=', 'Content-Length': '0'} 2023-07-29T00:17:58.872 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be1b991e1357ae82-0064c45ab6-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.872 DEBUG:boto:b'hiiqos-55b1000falseb2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:58.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-07-29T00:17:58.873 DEBUG:boto:path=/hiiqos-55/c 2023-07-29T00:17:58.874 DEBUG:boto:auth_path=/hiiqos-55/c 2023-07-29T00:17:58.874 DEBUG:boto:path=/hiiqos-55/c?versionId=null 2023-07-29T00:17:58.874 DEBUG:boto:auth_path=/hiiqos-55/c?versionId=null 2023-07-29T00:17:58.875 DEBUG:boto:Method: GET 2023-07-29T00:17:58.875 DEBUG:boto:Path: /hiiqos-55/c?versionId=null 2023-07-29T00:17:58.876 DEBUG:boto:Data: 2023-07-29T00:17:58.876 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.876 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:58.877 DEBUG:boto:Port: 8001 2023-07-29T00:17:58.877 DEBUG:boto:Params: {} 2023-07-29T00:17:58.878 DEBUG:boto:Token: None 2023-07-29T00:17:58.878 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/c?versionId=null 2023-07-29T00:17:58.878 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:in0lt/VJy+SUY7l7ukxeojvT/Zw= 2023-07-29T00:17:58.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:in0lt/VJy+SUY7l7ukxeojvT/Zw=', 'Content-Length': '0'} 2023-07-29T00:17:58.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088b0439a358db043-0064c45ab6-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.881 DEBUG:boto:path=/hiiqos-55/c 2023-07-29T00:17:58.881 DEBUG:boto:auth_path=/hiiqos-55/c 2023-07-29T00:17:58.882 DEBUG:boto:path=/hiiqos-55/c?versionId=null 2023-07-29T00:17:58.882 DEBUG:boto:auth_path=/hiiqos-55/c?versionId=null 2023-07-29T00:17:58.883 DEBUG:boto:Method: GET 2023-07-29T00:17:58.883 DEBUG:boto:Path: /hiiqos-55/c?versionId=null 2023-07-29T00:17:58.883 DEBUG:boto:Data: 2023-07-29T00:17:58.884 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.884 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:58.884 DEBUG:boto:Port: 8000 2023-07-29T00:17:58.885 DEBUG:boto:Params: {} 2023-07-29T00:17:58.885 DEBUG:boto:Token: None 2023-07-29T00:17:58.886 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/c?versionId=null 2023-07-29T00:17:58.886 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:in0lt/VJy+SUY7l7ukxeojvT/Zw= 2023-07-29T00:17:58.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:in0lt/VJy+SUY7l7ukxeojvT/Zw=', 'Content-Length': '0'} 2023-07-29T00:17:58.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000422d1f69b81f5b78-0064c45ab6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.889 DEBUG:boto:path=/hiiqos-55/c 2023-07-29T00:17:58.889 DEBUG:boto:auth_path=/hiiqos-55/c 2023-07-29T00:17:58.890 DEBUG:boto:path=/hiiqos-55/c?versionId=null 2023-07-29T00:17:58.890 DEBUG:boto:auth_path=/hiiqos-55/c?versionId=null 2023-07-29T00:17:58.890 DEBUG:boto:Method: HEAD 2023-07-29T00:17:58.891 DEBUG:boto:Path: /hiiqos-55/c?versionId=null 2023-07-29T00:17:58.891 DEBUG:boto:Data: 2023-07-29T00:17:58.891 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.892 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:58.892 DEBUG:boto:Port: 8001 2023-07-29T00:17:58.892 DEBUG:boto:Params: {} 2023-07-29T00:17:58.893 DEBUG:boto:Token: None 2023-07-29T00:17:58.893 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/c?versionId=null 2023-07-29T00:17:58.894 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:M2kiVXMdN1Bb5T+of81Txn3s958= 2023-07-29T00:17:58.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:M2kiVXMdN1Bb5T+of81Txn3s958=', 'Content-Length': '0'} 2023-07-29T00:17:58.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000817d65fb19713beb-0064c45ab6-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.896 DEBUG:boto:path=/hiiqos-55/c 2023-07-29T00:17:58.897 DEBUG:boto:auth_path=/hiiqos-55/c 2023-07-29T00:17:58.897 DEBUG:boto:path=/hiiqos-55/c?versionId=null 2023-07-29T00:17:58.897 DEBUG:boto:auth_path=/hiiqos-55/c?versionId=null 2023-07-29T00:17:58.898 DEBUG:boto:Method: HEAD 2023-07-29T00:17:58.898 DEBUG:boto:Path: /hiiqos-55/c?versionId=null 2023-07-29T00:17:58.898 DEBUG:boto:Data: 2023-07-29T00:17:58.899 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.899 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:58.899 DEBUG:boto:Port: 8000 2023-07-29T00:17:58.900 DEBUG:boto:Params: {} 2023-07-29T00:17:58.900 DEBUG:boto:Token: None 2023-07-29T00:17:58.901 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/c?versionId=null 2023-07-29T00:17:58.901 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:M2kiVXMdN1Bb5T+of81Txn3s958= 2023-07-29T00:17:58.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:M2kiVXMdN1Bb5T+of81Txn3s958=', 'Content-Length': '0'} 2023-07-29T00:17:58.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058cfc84e7f9173e2-0064c45ab6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-07-29T00:17:58.904 DEBUG:boto:path=/hiiqos-55/c 2023-07-29T00:17:58.904 DEBUG:boto:auth_path=/hiiqos-55/c 2023-07-29T00:17:58.905 DEBUG:boto:path=/hiiqos-55/c?versionId=null 2023-07-29T00:17:58.905 DEBUG:boto:auth_path=/hiiqos-55/c?versionId=null 2023-07-29T00:17:58.906 DEBUG:boto:Method: GET 2023-07-29T00:17:58.906 DEBUG:boto:Path: /hiiqos-55/c?versionId=null 2023-07-29T00:17:58.906 DEBUG:boto:Data: 2023-07-29T00:17:58.907 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.907 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:58.907 DEBUG:boto:Port: 8001 2023-07-29T00:17:58.908 DEBUG:boto:Params: {} 2023-07-29T00:17:58.908 DEBUG:boto:Token: None 2023-07-29T00:17:58.908 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/c?versionId=null 2023-07-29T00:17:58.909 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:in0lt/VJy+SUY7l7ukxeojvT/Zw= 2023-07-29T00:17:58.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:in0lt/VJy+SUY7l7ukxeojvT/Zw=', 'Content-Length': '0'} 2023-07-29T00:17:58.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d7e98540e8c084d-0064c45ab6-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.911 DEBUG:boto:path=/hiiqos-55/c 2023-07-29T00:17:58.912 DEBUG:boto:auth_path=/hiiqos-55/c 2023-07-29T00:17:58.912 DEBUG:boto:path=/hiiqos-55/c?versionId=null 2023-07-29T00:17:58.913 DEBUG:boto:auth_path=/hiiqos-55/c?versionId=null 2023-07-29T00:17:58.913 DEBUG:boto:Method: GET 2023-07-29T00:17:58.914 DEBUG:boto:Path: /hiiqos-55/c?versionId=null 2023-07-29T00:17:58.914 DEBUG:boto:Data: 2023-07-29T00:17:58.915 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.915 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:58.915 DEBUG:boto:Port: 8000 2023-07-29T00:17:58.916 DEBUG:boto:Params: {} 2023-07-29T00:17:58.916 DEBUG:boto:Token: None 2023-07-29T00:17:58.916 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/c?versionId=null 2023-07-29T00:17:58.917 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:in0lt/VJy+SUY7l7ukxeojvT/Zw= 2023-07-29T00:17:58.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:in0lt/VJy+SUY7l7ukxeojvT/Zw=', 'Content-Length': '0'} 2023-07-29T00:17:58.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e80422942ee77ae7-0064c45ab6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.920 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:58.920 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:58.920 DEBUG:boto:path=/hiiqos-55/?prefix=c 2023-07-29T00:17:58.921 DEBUG:boto:auth_path=/hiiqos-55/?prefix=c 2023-07-29T00:17:58.921 DEBUG:boto:Method: GET 2023-07-29T00:17:58.921 DEBUG:boto:Path: /hiiqos-55/?prefix=c 2023-07-29T00:17:58.922 DEBUG:boto:Data: 2023-07-29T00:17:58.922 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.922 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:58.923 DEBUG:boto:Port: 8001 2023-07-29T00:17:58.923 DEBUG:boto:Params: {} 2023-07-29T00:17:58.924 DEBUG:boto:Token: None 2023-07-29T00:17:58.924 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/ 2023-07-29T00:17:58.924 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0= 2023-07-29T00:17:58.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0=', 'Content-Length': '0'} 2023-07-29T00:17:58.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000396588c58c5757c2-0064c45ab6-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.927 DEBUG:boto:b'hiiqos-55c1000falsec2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:58.928 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:58.929 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:58.929 DEBUG:boto:path=/hiiqos-55/?prefix=c 2023-07-29T00:17:58.929 DEBUG:boto:auth_path=/hiiqos-55/?prefix=c 2023-07-29T00:17:58.930 DEBUG:boto:Method: GET 2023-07-29T00:17:58.930 DEBUG:boto:Path: /hiiqos-55/?prefix=c 2023-07-29T00:17:58.930 DEBUG:boto:Data: 2023-07-29T00:17:58.931 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.931 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:58.931 DEBUG:boto:Port: 8000 2023-07-29T00:17:58.932 DEBUG:boto:Params: {} 2023-07-29T00:17:58.932 DEBUG:boto:Token: None 2023-07-29T00:17:58.933 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/ 2023-07-29T00:17:58.933 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0= 2023-07-29T00:17:58.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0=', 'Content-Length': '0'} 2023-07-29T00:17:58.935 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b30d55281470da69-0064c45ab6-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.936 DEBUG:boto:b'hiiqos-55c1000falsec2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:58.937 DEBUG:boto:path=/hiiqos-55/c 2023-07-29T00:17:58.937 DEBUG:boto:auth_path=/hiiqos-55/c 2023-07-29T00:17:58.938 DEBUG:boto:Method: HEAD 2023-07-29T00:17:58.938 DEBUG:boto:Path: /hiiqos-55/c 2023-07-29T00:17:58.938 DEBUG:boto:Data: 2023-07-29T00:17:58.939 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.939 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:58.939 DEBUG:boto:Port: 8001 2023-07-29T00:17:58.940 DEBUG:boto:Params: {} 2023-07-29T00:17:58.940 DEBUG:boto:Token: None 2023-07-29T00:17:58.940 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/c 2023-07-29T00:17:58.941 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Sn0Wzw5kIMGHTkEvKPnrZBf0rDA= 2023-07-29T00:17:58.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Sn0Wzw5kIMGHTkEvKPnrZBf0rDA=', 'Content-Length': '0'} 2023-07-29T00:17:58.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c55ced919a98346-0064c45ab6-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.943 DEBUG:boto:path=/hiiqos-55/c 2023-07-29T00:17:58.944 DEBUG:boto:auth_path=/hiiqos-55/c 2023-07-29T00:17:58.944 DEBUG:boto:Method: HEAD 2023-07-29T00:17:58.944 DEBUG:boto:Path: /hiiqos-55/c 2023-07-29T00:17:58.945 DEBUG:boto:Data: 2023-07-29T00:17:58.945 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.945 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:58.946 DEBUG:boto:Port: 8000 2023-07-29T00:17:58.946 DEBUG:boto:Params: {} 2023-07-29T00:17:58.946 DEBUG:boto:Token: None 2023-07-29T00:17:58.947 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/c 2023-07-29T00:17:58.947 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Sn0Wzw5kIMGHTkEvKPnrZBf0rDA= 2023-07-29T00:17:58.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Sn0Wzw5kIMGHTkEvKPnrZBf0rDA=', 'Content-Length': '0'} 2023-07-29T00:17:58.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d44c50646b3b776-0064c45ab6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-07-29T00:17:58.950 DEBUG:boto:path=/hiiqos-55/c 2023-07-29T00:17:58.951 DEBUG:boto:auth_path=/hiiqos-55/c 2023-07-29T00:17:58.951 DEBUG:boto:Method: GET 2023-07-29T00:17:58.951 DEBUG:boto:Path: /hiiqos-55/c 2023-07-29T00:17:58.952 DEBUG:boto:Data: 2023-07-29T00:17:58.952 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.952 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:58.953 DEBUG:boto:Port: 8001 2023-07-29T00:17:58.953 DEBUG:boto:Params: {} 2023-07-29T00:17:58.953 DEBUG:boto:Token: None 2023-07-29T00:17:58.954 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/c 2023-07-29T00:17:58.954 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:43Jkn9X3FAOqavz5PVYX/nOwzbA= 2023-07-29T00:17:58.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:43Jkn9X3FAOqavz5PVYX/nOwzbA=', 'Content-Length': '0'} 2023-07-29T00:17:58.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c91126f1e5e676af-0064c45ab6-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.957 DEBUG:boto:path=/hiiqos-55/c 2023-07-29T00:17:58.957 DEBUG:boto:auth_path=/hiiqos-55/c 2023-07-29T00:17:58.958 DEBUG:boto:Method: GET 2023-07-29T00:17:58.958 DEBUG:boto:Path: /hiiqos-55/c 2023-07-29T00:17:58.958 DEBUG:boto:Data: 2023-07-29T00:17:58.959 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.959 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:58.959 DEBUG:boto:Port: 8000 2023-07-29T00:17:58.960 DEBUG:boto:Params: {} 2023-07-29T00:17:58.960 DEBUG:boto:Token: None 2023-07-29T00:17:58.961 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/c 2023-07-29T00:17:58.961 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:43Jkn9X3FAOqavz5PVYX/nOwzbA= 2023-07-29T00:17:58.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:43Jkn9X3FAOqavz5PVYX/nOwzbA=', 'Content-Length': '0'} 2023-07-29T00:17:58.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f3e402302d22d77-0064c45ab6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:58.964 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:58.964 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:58.965 DEBUG:boto:path=/hiiqos-55/?prefix=c 2023-07-29T00:17:58.965 DEBUG:boto:auth_path=/hiiqos-55/?prefix=c 2023-07-29T00:17:58.966 DEBUG:boto:Method: GET 2023-07-29T00:17:58.966 DEBUG:boto:Path: /hiiqos-55/?prefix=c 2023-07-29T00:17:58.966 DEBUG:boto:Data: 2023-07-29T00:17:58.967 DEBUG:boto:Headers: {} 2023-07-29T00:17:58.967 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:58.967 DEBUG:boto:Port: 8001 2023-07-29T00:17:58.968 DEBUG:boto:Params: {} 2023-07-29T00:17:58.968 DEBUG:boto:Token: None 2023-07-29T00:17:58.969 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:58 GMT /hiiqos-55/ 2023-07-29T00:17:58.969 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0= 2023-07-29T00:17:58.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0=', 'Content-Length': '0'} 2023-07-29T00:17:58.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004dfb3465966e6c6c-0064c45ab6-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.012 DEBUG:boto:b'hiiqos-55c1000falsec2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:59.013 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:59.014 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:59.014 DEBUG:boto:path=/hiiqos-55/?prefix=c 2023-07-29T00:17:59.014 DEBUG:boto:auth_path=/hiiqos-55/?prefix=c 2023-07-29T00:17:59.015 DEBUG:boto:Method: GET 2023-07-29T00:17:59.015 DEBUG:boto:Path: /hiiqos-55/?prefix=c 2023-07-29T00:17:59.015 DEBUG:boto:Data: 2023-07-29T00:17:59.016 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.016 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:59.016 DEBUG:boto:Port: 8000 2023-07-29T00:17:59.017 DEBUG:boto:Params: {} 2023-07-29T00:17:59.017 DEBUG:boto:Token: None 2023-07-29T00:17:59.018 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/ 2023-07-29T00:17:59.018 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4= 2023-07-29T00:17:59.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4=', 'Content-Length': '0'} 2023-07-29T00:17:59.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005579d3a28c84b2be-0064c45ab6-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.021 DEBUG:boto:b'hiiqos-55c1000falsec2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:59.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-07-29T00:17:59.022 DEBUG:boto:path=/hiiqos-55/d 2023-07-29T00:17:59.022 DEBUG:boto:auth_path=/hiiqos-55/d 2023-07-29T00:17:59.023 DEBUG:boto:path=/hiiqos-55/d?versionId=null 2023-07-29T00:17:59.023 DEBUG:boto:auth_path=/hiiqos-55/d?versionId=null 2023-07-29T00:17:59.023 DEBUG:boto:Method: GET 2023-07-29T00:17:59.024 DEBUG:boto:Path: /hiiqos-55/d?versionId=null 2023-07-29T00:17:59.024 DEBUG:boto:Data: 2023-07-29T00:17:59.025 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.025 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:59.025 DEBUG:boto:Port: 8001 2023-07-29T00:17:59.026 DEBUG:boto:Params: {} 2023-07-29T00:17:59.026 DEBUG:boto:Token: None 2023-07-29T00:17:59.026 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/d?versionId=null 2023-07-29T00:17:59.027 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5KQOUDw8e4yJatFUb6ylI+bnIL8= 2023-07-29T00:17:59.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5KQOUDw8e4yJatFUb6ylI+bnIL8=', 'Content-Length': '0'} 2023-07-29T00:17:59.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000108945ec86768e79-0064c45ab6-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.030 DEBUG:boto:path=/hiiqos-55/d 2023-07-29T00:17:59.030 DEBUG:boto:auth_path=/hiiqos-55/d 2023-07-29T00:17:59.030 DEBUG:boto:path=/hiiqos-55/d?versionId=null 2023-07-29T00:17:59.031 DEBUG:boto:auth_path=/hiiqos-55/d?versionId=null 2023-07-29T00:17:59.031 DEBUG:boto:Method: GET 2023-07-29T00:17:59.031 DEBUG:boto:Path: /hiiqos-55/d?versionId=null 2023-07-29T00:17:59.032 DEBUG:boto:Data: 2023-07-29T00:17:59.032 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.033 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:59.033 DEBUG:boto:Port: 8000 2023-07-29T00:17:59.033 DEBUG:boto:Params: {} 2023-07-29T00:17:59.034 DEBUG:boto:Token: None 2023-07-29T00:17:59.034 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/d?versionId=null 2023-07-29T00:17:59.034 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5KQOUDw8e4yJatFUb6ylI+bnIL8= 2023-07-29T00:17:59.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5KQOUDw8e4yJatFUb6ylI+bnIL8=', 'Content-Length': '0'} 2023-07-29T00:17:59.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000502a67ebdf71270b-0064c45ab6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.080 DEBUG:boto:path=/hiiqos-55/d 2023-07-29T00:17:59.081 DEBUG:boto:auth_path=/hiiqos-55/d 2023-07-29T00:17:59.081 DEBUG:boto:path=/hiiqos-55/d?versionId=null 2023-07-29T00:17:59.081 DEBUG:boto:auth_path=/hiiqos-55/d?versionId=null 2023-07-29T00:17:59.082 DEBUG:boto:Method: HEAD 2023-07-29T00:17:59.082 DEBUG:boto:Path: /hiiqos-55/d?versionId=null 2023-07-29T00:17:59.082 DEBUG:boto:Data: 2023-07-29T00:17:59.083 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.083 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:59.083 DEBUG:boto:Port: 8001 2023-07-29T00:17:59.084 DEBUG:boto:Params: {} 2023-07-29T00:17:59.084 DEBUG:boto:Token: None 2023-07-29T00:17:59.085 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/d?versionId=null 2023-07-29T00:17:59.085 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mAAcVZoZTIkH9qt6YdN+KDWtwDE= 2023-07-29T00:17:59.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mAAcVZoZTIkH9qt6YdN+KDWtwDE=', 'Content-Length': '0'} 2023-07-29T00:17:59.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbdcbaade15861e5-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.088 DEBUG:boto:path=/hiiqos-55/d 2023-07-29T00:17:59.088 DEBUG:boto:auth_path=/hiiqos-55/d 2023-07-29T00:17:59.088 DEBUG:boto:path=/hiiqos-55/d?versionId=null 2023-07-29T00:17:59.089 DEBUG:boto:auth_path=/hiiqos-55/d?versionId=null 2023-07-29T00:17:59.089 DEBUG:boto:Method: HEAD 2023-07-29T00:17:59.089 DEBUG:boto:Path: /hiiqos-55/d?versionId=null 2023-07-29T00:17:59.090 DEBUG:boto:Data: 2023-07-29T00:17:59.090 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.090 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:59.091 DEBUG:boto:Port: 8000 2023-07-29T00:17:59.091 DEBUG:boto:Params: {} 2023-07-29T00:17:59.091 DEBUG:boto:Token: None 2023-07-29T00:17:59.092 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/d?versionId=null 2023-07-29T00:17:59.092 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mAAcVZoZTIkH9qt6YdN+KDWtwDE= 2023-07-29T00:17:59.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mAAcVZoZTIkH9qt6YdN+KDWtwDE=', 'Content-Length': '0'} 2023-07-29T00:17:59.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049b3d11e51fe1edd-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-07-29T00:17:59.095 DEBUG:boto:path=/hiiqos-55/d 2023-07-29T00:17:59.095 DEBUG:boto:auth_path=/hiiqos-55/d 2023-07-29T00:17:59.096 DEBUG:boto:path=/hiiqos-55/d?versionId=null 2023-07-29T00:17:59.096 DEBUG:boto:auth_path=/hiiqos-55/d?versionId=null 2023-07-29T00:17:59.097 DEBUG:boto:Method: GET 2023-07-29T00:17:59.097 DEBUG:boto:Path: /hiiqos-55/d?versionId=null 2023-07-29T00:17:59.097 DEBUG:boto:Data: 2023-07-29T00:17:59.098 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.098 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:59.098 DEBUG:boto:Port: 8001 2023-07-29T00:17:59.099 DEBUG:boto:Params: {} 2023-07-29T00:17:59.099 DEBUG:boto:Token: None 2023-07-29T00:17:59.100 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/d?versionId=null 2023-07-29T00:17:59.100 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5KQOUDw8e4yJatFUb6ylI+bnIL8= 2023-07-29T00:17:59.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5KQOUDw8e4yJatFUb6ylI+bnIL8=', 'Content-Length': '0'} 2023-07-29T00:17:59.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bbc2ff9d0f430f5-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.103 DEBUG:boto:path=/hiiqos-55/d 2023-07-29T00:17:59.103 DEBUG:boto:auth_path=/hiiqos-55/d 2023-07-29T00:17:59.103 DEBUG:boto:path=/hiiqos-55/d?versionId=null 2023-07-29T00:17:59.104 DEBUG:boto:auth_path=/hiiqos-55/d?versionId=null 2023-07-29T00:17:59.104 DEBUG:boto:Method: GET 2023-07-29T00:17:59.104 DEBUG:boto:Path: /hiiqos-55/d?versionId=null 2023-07-29T00:17:59.105 DEBUG:boto:Data: 2023-07-29T00:17:59.105 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.105 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:59.106 DEBUG:boto:Port: 8000 2023-07-29T00:17:59.106 DEBUG:boto:Params: {} 2023-07-29T00:17:59.107 DEBUG:boto:Token: None 2023-07-29T00:17:59.107 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/d?versionId=null 2023-07-29T00:17:59.107 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5KQOUDw8e4yJatFUb6ylI+bnIL8= 2023-07-29T00:17:59.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5KQOUDw8e4yJatFUb6ylI+bnIL8=', 'Content-Length': '0'} 2023-07-29T00:17:59.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f2e6376004fecce-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.110 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:59.110 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:59.111 DEBUG:boto:path=/hiiqos-55/?prefix=d 2023-07-29T00:17:59.111 DEBUG:boto:auth_path=/hiiqos-55/?prefix=d 2023-07-29T00:17:59.112 DEBUG:boto:Method: GET 2023-07-29T00:17:59.112 DEBUG:boto:Path: /hiiqos-55/?prefix=d 2023-07-29T00:17:59.112 DEBUG:boto:Data: 2023-07-29T00:17:59.113 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.113 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:59.113 DEBUG:boto:Port: 8001 2023-07-29T00:17:59.114 DEBUG:boto:Params: {} 2023-07-29T00:17:59.114 DEBUG:boto:Token: None 2023-07-29T00:17:59.114 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/ 2023-07-29T00:17:59.115 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4= 2023-07-29T00:17:59.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4=', 'Content-Length': '0'} 2023-07-29T00:17:59.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b94ba521c560ed15-0064c45ab7-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.118 DEBUG:boto:b'hiiqos-55d1000falsed2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:59.119 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:59.119 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:59.119 DEBUG:boto:path=/hiiqos-55/?prefix=d 2023-07-29T00:17:59.120 DEBUG:boto:auth_path=/hiiqos-55/?prefix=d 2023-07-29T00:17:59.120 DEBUG:boto:Method: GET 2023-07-29T00:17:59.121 DEBUG:boto:Path: /hiiqos-55/?prefix=d 2023-07-29T00:17:59.121 DEBUG:boto:Data: 2023-07-29T00:17:59.121 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.122 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:59.122 DEBUG:boto:Port: 8000 2023-07-29T00:17:59.122 DEBUG:boto:Params: {} 2023-07-29T00:17:59.123 DEBUG:boto:Token: None 2023-07-29T00:17:59.123 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/ 2023-07-29T00:17:59.124 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4= 2023-07-29T00:17:59.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4=', 'Content-Length': '0'} 2023-07-29T00:17:59.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ec61d12a6273cd6-0064c45ab7-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.126 DEBUG:boto:b'hiiqos-55d1000falsed2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:59.127 DEBUG:boto:path=/hiiqos-55/d 2023-07-29T00:17:59.127 DEBUG:boto:auth_path=/hiiqos-55/d 2023-07-29T00:17:59.128 DEBUG:boto:Method: HEAD 2023-07-29T00:17:59.128 DEBUG:boto:Path: /hiiqos-55/d 2023-07-29T00:17:59.128 DEBUG:boto:Data: 2023-07-29T00:17:59.129 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.129 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:59.129 DEBUG:boto:Port: 8001 2023-07-29T00:17:59.130 DEBUG:boto:Params: {} 2023-07-29T00:17:59.130 DEBUG:boto:Token: None 2023-07-29T00:17:59.130 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/d 2023-07-29T00:17:59.131 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cnK+njR1eCbNgdNpxRIaNnWloQE= 2023-07-29T00:17:59.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cnK+njR1eCbNgdNpxRIaNnWloQE=', 'Content-Length': '0'} 2023-07-29T00:17:59.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2f1ca5974730f1c-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.133 DEBUG:boto:path=/hiiqos-55/d 2023-07-29T00:17:59.134 DEBUG:boto:auth_path=/hiiqos-55/d 2023-07-29T00:17:59.134 DEBUG:boto:Method: HEAD 2023-07-29T00:17:59.134 DEBUG:boto:Path: /hiiqos-55/d 2023-07-29T00:17:59.135 DEBUG:boto:Data: 2023-07-29T00:17:59.135 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.135 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:59.136 DEBUG:boto:Port: 8000 2023-07-29T00:17:59.136 DEBUG:boto:Params: {} 2023-07-29T00:17:59.136 DEBUG:boto:Token: None 2023-07-29T00:17:59.137 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/d 2023-07-29T00:17:59.137 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cnK+njR1eCbNgdNpxRIaNnWloQE= 2023-07-29T00:17:59.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cnK+njR1eCbNgdNpxRIaNnWloQE=', 'Content-Length': '0'} 2023-07-29T00:17:59.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007dc3f5c221695af8-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-07-29T00:17:59.140 DEBUG:boto:path=/hiiqos-55/d 2023-07-29T00:17:59.140 DEBUG:boto:auth_path=/hiiqos-55/d 2023-07-29T00:17:59.141 DEBUG:boto:Method: GET 2023-07-29T00:17:59.141 DEBUG:boto:Path: /hiiqos-55/d 2023-07-29T00:17:59.141 DEBUG:boto:Data: 2023-07-29T00:17:59.142 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.142 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:59.142 DEBUG:boto:Port: 8001 2023-07-29T00:17:59.143 DEBUG:boto:Params: {} 2023-07-29T00:17:59.143 DEBUG:boto:Token: None 2023-07-29T00:17:59.144 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/d 2023-07-29T00:17:59.144 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0JFteaOb3BIH/EffAi7cmQ8mFVM= 2023-07-29T00:17:59.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0JFteaOb3BIH/EffAi7cmQ8mFVM=', 'Content-Length': '0'} 2023-07-29T00:17:59.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000700450029633ad4-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.188 DEBUG:boto:path=/hiiqos-55/d 2023-07-29T00:17:59.189 DEBUG:boto:auth_path=/hiiqos-55/d 2023-07-29T00:17:59.189 DEBUG:boto:Method: GET 2023-07-29T00:17:59.190 DEBUG:boto:Path: /hiiqos-55/d 2023-07-29T00:17:59.190 DEBUG:boto:Data: 2023-07-29T00:17:59.190 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.191 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:59.191 DEBUG:boto:Port: 8000 2023-07-29T00:17:59.191 DEBUG:boto:Params: {} 2023-07-29T00:17:59.192 DEBUG:boto:Token: None 2023-07-29T00:17:59.192 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/d 2023-07-29T00:17:59.193 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0JFteaOb3BIH/EffAi7cmQ8mFVM= 2023-07-29T00:17:59.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0JFteaOb3BIH/EffAi7cmQ8mFVM=', 'Content-Length': '0'} 2023-07-29T00:17:59.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049cac657181d2c6f-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.195 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:59.196 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:59.196 DEBUG:boto:path=/hiiqos-55/?prefix=d 2023-07-29T00:17:59.197 DEBUG:boto:auth_path=/hiiqos-55/?prefix=d 2023-07-29T00:17:59.197 DEBUG:boto:Method: GET 2023-07-29T00:17:59.197 DEBUG:boto:Path: /hiiqos-55/?prefix=d 2023-07-29T00:17:59.198 DEBUG:boto:Data: 2023-07-29T00:17:59.198 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.198 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:59.199 DEBUG:boto:Port: 8001 2023-07-29T00:17:59.199 DEBUG:boto:Params: {} 2023-07-29T00:17:59.200 DEBUG:boto:Token: None 2023-07-29T00:17:59.200 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/ 2023-07-29T00:17:59.200 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4= 2023-07-29T00:17:59.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4=', 'Content-Length': '0'} 2023-07-29T00:17:59.203 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a781b5b1cb817c0-0064c45ab7-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.204 DEBUG:boto:b'hiiqos-55d1000falsed2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:59.205 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:59.205 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:59.205 DEBUG:boto:path=/hiiqos-55/?prefix=d 2023-07-29T00:17:59.206 DEBUG:boto:auth_path=/hiiqos-55/?prefix=d 2023-07-29T00:17:59.206 DEBUG:boto:Method: GET 2023-07-29T00:17:59.206 DEBUG:boto:Path: /hiiqos-55/?prefix=d 2023-07-29T00:17:59.207 DEBUG:boto:Data: 2023-07-29T00:17:59.207 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.208 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:59.208 DEBUG:boto:Port: 8000 2023-07-29T00:17:59.209 DEBUG:boto:Params: {} 2023-07-29T00:17:59.209 DEBUG:boto:Token: None 2023-07-29T00:17:59.210 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/ 2023-07-29T00:17:59.210 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4= 2023-07-29T00:17:59.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4=', 'Content-Length': '0'} 2023-07-29T00:17:59.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ade1402206f0af5f-0064c45ab7-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.213 DEBUG:boto:b'hiiqos-55d1000falsed2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:59.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-07-29T00:17:59.214 DEBUG:boto:path=/hiiqos-55/e 2023-07-29T00:17:59.214 DEBUG:boto:auth_path=/hiiqos-55/e 2023-07-29T00:17:59.214 DEBUG:boto:path=/hiiqos-55/e?versionId=null 2023-07-29T00:17:59.215 DEBUG:boto:auth_path=/hiiqos-55/e?versionId=null 2023-07-29T00:17:59.215 DEBUG:boto:Method: GET 2023-07-29T00:17:59.215 DEBUG:boto:Path: /hiiqos-55/e?versionId=null 2023-07-29T00:17:59.216 DEBUG:boto:Data: 2023-07-29T00:17:59.216 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.217 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:59.217 DEBUG:boto:Port: 8001 2023-07-29T00:17:59.217 DEBUG:boto:Params: {} 2023-07-29T00:17:59.218 DEBUG:boto:Token: None 2023-07-29T00:17:59.218 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/e?versionId=null 2023-07-29T00:17:59.219 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1E/UqzTuNI784IcAGiKLyGG931s= 2023-07-29T00:17:59.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1E/UqzTuNI784IcAGiKLyGG931s=', 'Content-Length': '0'} 2023-07-29T00:17:59.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b45d68b1efbd1a3-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.224 DEBUG:boto:path=/hiiqos-55/e 2023-07-29T00:17:59.225 DEBUG:boto:auth_path=/hiiqos-55/e 2023-07-29T00:17:59.226 DEBUG:boto:path=/hiiqos-55/e?versionId=null 2023-07-29T00:17:59.226 DEBUG:boto:auth_path=/hiiqos-55/e?versionId=null 2023-07-29T00:17:59.227 DEBUG:boto:Method: GET 2023-07-29T00:17:59.227 DEBUG:boto:Path: /hiiqos-55/e?versionId=null 2023-07-29T00:17:59.228 DEBUG:boto:Data: 2023-07-29T00:17:59.229 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.229 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:59.229 DEBUG:boto:Port: 8000 2023-07-29T00:17:59.230 DEBUG:boto:Params: {} 2023-07-29T00:17:59.230 DEBUG:boto:Token: None 2023-07-29T00:17:59.231 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/e?versionId=null 2023-07-29T00:17:59.231 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1E/UqzTuNI784IcAGiKLyGG931s= 2023-07-29T00:17:59.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1E/UqzTuNI784IcAGiKLyGG931s=', 'Content-Length': '0'} 2023-07-29T00:17:59.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051668bd88135902c-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.234 DEBUG:boto:path=/hiiqos-55/e 2023-07-29T00:17:59.234 DEBUG:boto:auth_path=/hiiqos-55/e 2023-07-29T00:17:59.234 DEBUG:boto:path=/hiiqos-55/e?versionId=null 2023-07-29T00:17:59.235 DEBUG:boto:auth_path=/hiiqos-55/e?versionId=null 2023-07-29T00:17:59.235 DEBUG:boto:Method: HEAD 2023-07-29T00:17:59.235 DEBUG:boto:Path: /hiiqos-55/e?versionId=null 2023-07-29T00:17:59.236 DEBUG:boto:Data: 2023-07-29T00:17:59.236 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.237 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:59.237 DEBUG:boto:Port: 8001 2023-07-29T00:17:59.237 DEBUG:boto:Params: {} 2023-07-29T00:17:59.238 DEBUG:boto:Token: None 2023-07-29T00:17:59.238 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/e?versionId=null 2023-07-29T00:17:59.238 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aaGWO4PYLoNIX0J0lONsLW3S1dA= 2023-07-29T00:17:59.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aaGWO4PYLoNIX0J0lONsLW3S1dA=', 'Content-Length': '0'} 2023-07-29T00:17:59.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003beb56383f269697-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.241 DEBUG:boto:path=/hiiqos-55/e 2023-07-29T00:17:59.241 DEBUG:boto:auth_path=/hiiqos-55/e 2023-07-29T00:17:59.242 DEBUG:boto:path=/hiiqos-55/e?versionId=null 2023-07-29T00:17:59.242 DEBUG:boto:auth_path=/hiiqos-55/e?versionId=null 2023-07-29T00:17:59.242 DEBUG:boto:Method: HEAD 2023-07-29T00:17:59.242 DEBUG:boto:Path: /hiiqos-55/e?versionId=null 2023-07-29T00:17:59.243 DEBUG:boto:Data: 2023-07-29T00:17:59.243 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.243 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:59.244 DEBUG:boto:Port: 8000 2023-07-29T00:17:59.244 DEBUG:boto:Params: {} 2023-07-29T00:17:59.245 DEBUG:boto:Token: None 2023-07-29T00:17:59.245 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/e?versionId=null 2023-07-29T00:17:59.245 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aaGWO4PYLoNIX0J0lONsLW3S1dA= 2023-07-29T00:17:59.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aaGWO4PYLoNIX0J0lONsLW3S1dA=', 'Content-Length': '0'} 2023-07-29T00:17:59.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004995a77e59f97824-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-07-29T00:17:59.249 DEBUG:boto:path=/hiiqos-55/e 2023-07-29T00:17:59.249 DEBUG:boto:auth_path=/hiiqos-55/e 2023-07-29T00:17:59.249 DEBUG:boto:path=/hiiqos-55/e?versionId=null 2023-07-29T00:17:59.250 DEBUG:boto:auth_path=/hiiqos-55/e?versionId=null 2023-07-29T00:17:59.250 DEBUG:boto:Method: GET 2023-07-29T00:17:59.250 DEBUG:boto:Path: /hiiqos-55/e?versionId=null 2023-07-29T00:17:59.251 DEBUG:boto:Data: 2023-07-29T00:17:59.251 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.251 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:59.252 DEBUG:boto:Port: 8001 2023-07-29T00:17:59.252 DEBUG:boto:Params: {} 2023-07-29T00:17:59.253 DEBUG:boto:Token: None 2023-07-29T00:17:59.253 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/e?versionId=null 2023-07-29T00:17:59.254 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1E/UqzTuNI784IcAGiKLyGG931s= 2023-07-29T00:17:59.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1E/UqzTuNI784IcAGiKLyGG931s=', 'Content-Length': '0'} 2023-07-29T00:17:59.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b678acb67f46189c-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.256 DEBUG:boto:path=/hiiqos-55/e 2023-07-29T00:17:59.257 DEBUG:boto:auth_path=/hiiqos-55/e 2023-07-29T00:17:59.257 DEBUG:boto:path=/hiiqos-55/e?versionId=null 2023-07-29T00:17:59.258 DEBUG:boto:auth_path=/hiiqos-55/e?versionId=null 2023-07-29T00:17:59.258 DEBUG:boto:Method: GET 2023-07-29T00:17:59.258 DEBUG:boto:Path: /hiiqos-55/e?versionId=null 2023-07-29T00:17:59.259 DEBUG:boto:Data: 2023-07-29T00:17:59.259 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.260 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:59.260 DEBUG:boto:Port: 8000 2023-07-29T00:17:59.260 DEBUG:boto:Params: {} 2023-07-29T00:17:59.261 DEBUG:boto:Token: None 2023-07-29T00:17:59.261 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/e?versionId=null 2023-07-29T00:17:59.262 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1E/UqzTuNI784IcAGiKLyGG931s= 2023-07-29T00:17:59.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1E/UqzTuNI784IcAGiKLyGG931s=', 'Content-Length': '0'} 2023-07-29T00:17:59.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6bf6bd06784381f-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.264 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:59.265 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:59.265 DEBUG:boto:path=/hiiqos-55/?prefix=e 2023-07-29T00:17:59.265 DEBUG:boto:auth_path=/hiiqos-55/?prefix=e 2023-07-29T00:17:59.266 DEBUG:boto:Method: GET 2023-07-29T00:17:59.266 DEBUG:boto:Path: /hiiqos-55/?prefix=e 2023-07-29T00:17:59.266 DEBUG:boto:Data: 2023-07-29T00:17:59.267 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.267 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:59.268 DEBUG:boto:Port: 8001 2023-07-29T00:17:59.268 DEBUG:boto:Params: {} 2023-07-29T00:17:59.268 DEBUG:boto:Token: None 2023-07-29T00:17:59.269 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/ 2023-07-29T00:17:59.269 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4= 2023-07-29T00:17:59.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4=', 'Content-Length': '0'} 2023-07-29T00:17:59.272 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003207164e967754cd-0064c45ab7-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.272 DEBUG:boto:b'hiiqos-55e1000falsee2023-07-29T00:15:00.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:59.273 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:59.274 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:59.275 DEBUG:boto:path=/hiiqos-55/?prefix=e 2023-07-29T00:17:59.275 DEBUG:boto:auth_path=/hiiqos-55/?prefix=e 2023-07-29T00:17:59.276 DEBUG:boto:Method: GET 2023-07-29T00:17:59.276 DEBUG:boto:Path: /hiiqos-55/?prefix=e 2023-07-29T00:17:59.277 DEBUG:boto:Data: 2023-07-29T00:17:59.278 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.278 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:59.279 DEBUG:boto:Port: 8000 2023-07-29T00:17:59.279 DEBUG:boto:Params: {} 2023-07-29T00:17:59.280 DEBUG:boto:Token: None 2023-07-29T00:17:59.281 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/ 2023-07-29T00:17:59.281 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4= 2023-07-29T00:17:59.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4=', 'Content-Length': '0'} 2023-07-29T00:17:59.284 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009022c770f61ed90d-0064c45ab7-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.285 DEBUG:boto:b'hiiqos-55e1000falsee2023-07-29T00:15:00.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:59.285 DEBUG:boto:path=/hiiqos-55/e 2023-07-29T00:17:59.286 DEBUG:boto:auth_path=/hiiqos-55/e 2023-07-29T00:17:59.286 DEBUG:boto:Method: HEAD 2023-07-29T00:17:59.286 DEBUG:boto:Path: /hiiqos-55/e 2023-07-29T00:17:59.287 DEBUG:boto:Data: 2023-07-29T00:17:59.287 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.287 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:59.288 DEBUG:boto:Port: 8001 2023-07-29T00:17:59.288 DEBUG:boto:Params: {} 2023-07-29T00:17:59.288 DEBUG:boto:Token: None 2023-07-29T00:17:59.289 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/e 2023-07-29T00:17:59.289 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4uyE2n56zIVyLz8hhjN209tqjPs= 2023-07-29T00:17:59.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4uyE2n56zIVyLz8hhjN209tqjPs=', 'Content-Length': '0'} 2023-07-29T00:17:59.291 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f9bb6cb518e61fd-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.292 DEBUG:boto:path=/hiiqos-55/e 2023-07-29T00:17:59.292 DEBUG:boto:auth_path=/hiiqos-55/e 2023-07-29T00:17:59.292 DEBUG:boto:Method: HEAD 2023-07-29T00:17:59.293 DEBUG:boto:Path: /hiiqos-55/e 2023-07-29T00:17:59.293 DEBUG:boto:Data: 2023-07-29T00:17:59.293 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.294 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:59.294 DEBUG:boto:Port: 8000 2023-07-29T00:17:59.294 DEBUG:boto:Params: {} 2023-07-29T00:17:59.295 DEBUG:boto:Token: None 2023-07-29T00:17:59.295 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/e 2023-07-29T00:17:59.296 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4uyE2n56zIVyLz8hhjN209tqjPs= 2023-07-29T00:17:59.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4uyE2n56zIVyLz8hhjN209tqjPs=', 'Content-Length': '0'} 2023-07-29T00:17:59.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000356242b0b68ba80-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-07-29T00:17:59.299 DEBUG:boto:path=/hiiqos-55/e 2023-07-29T00:17:59.299 DEBUG:boto:auth_path=/hiiqos-55/e 2023-07-29T00:17:59.300 DEBUG:boto:Method: GET 2023-07-29T00:17:59.300 DEBUG:boto:Path: /hiiqos-55/e 2023-07-29T00:17:59.301 DEBUG:boto:Data: 2023-07-29T00:17:59.301 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.301 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:59.302 DEBUG:boto:Port: 8001 2023-07-29T00:17:59.302 DEBUG:boto:Params: {} 2023-07-29T00:17:59.302 DEBUG:boto:Token: None 2023-07-29T00:17:59.303 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/e 2023-07-29T00:17:59.303 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IoV2Ky/ucAZhdNFujM4/+mF7J/U= 2023-07-29T00:17:59.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IoV2Ky/ucAZhdNFujM4/+mF7J/U=', 'Content-Length': '0'} 2023-07-29T00:17:59.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c862ac8e6ad9802-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.306 DEBUG:boto:path=/hiiqos-55/e 2023-07-29T00:17:59.306 DEBUG:boto:auth_path=/hiiqos-55/e 2023-07-29T00:17:59.307 DEBUG:boto:Method: GET 2023-07-29T00:17:59.307 DEBUG:boto:Path: /hiiqos-55/e 2023-07-29T00:17:59.307 DEBUG:boto:Data: 2023-07-29T00:17:59.308 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.308 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:59.309 DEBUG:boto:Port: 8000 2023-07-29T00:17:59.309 DEBUG:boto:Params: {} 2023-07-29T00:17:59.310 DEBUG:boto:Token: None 2023-07-29T00:17:59.310 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/e 2023-07-29T00:17:59.310 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IoV2Ky/ucAZhdNFujM4/+mF7J/U= 2023-07-29T00:17:59.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IoV2Ky/ucAZhdNFujM4/+mF7J/U=', 'Content-Length': '0'} 2023-07-29T00:17:59.313 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d65710e082e9080-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.356 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:59.357 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:59.357 DEBUG:boto:path=/hiiqos-55/?prefix=e 2023-07-29T00:17:59.358 DEBUG:boto:auth_path=/hiiqos-55/?prefix=e 2023-07-29T00:17:59.358 DEBUG:boto:Method: GET 2023-07-29T00:17:59.358 DEBUG:boto:Path: /hiiqos-55/?prefix=e 2023-07-29T00:17:59.359 DEBUG:boto:Data: 2023-07-29T00:17:59.359 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.359 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:59.360 DEBUG:boto:Port: 8001 2023-07-29T00:17:59.360 DEBUG:boto:Params: {} 2023-07-29T00:17:59.361 DEBUG:boto:Token: None 2023-07-29T00:17:59.361 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/ 2023-07-29T00:17:59.361 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4= 2023-07-29T00:17:59.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4=', 'Content-Length': '0'} 2023-07-29T00:17:59.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052193816a2ff14cf-0064c45ab7-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.365 DEBUG:boto:b'hiiqos-55e1000falsee2023-07-29T00:15:00.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:59.365 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:59.366 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:59.366 DEBUG:boto:path=/hiiqos-55/?prefix=e 2023-07-29T00:17:59.366 DEBUG:boto:auth_path=/hiiqos-55/?prefix=e 2023-07-29T00:17:59.367 DEBUG:boto:Method: GET 2023-07-29T00:17:59.367 DEBUG:boto:Path: /hiiqos-55/?prefix=e 2023-07-29T00:17:59.367 DEBUG:boto:Data: 2023-07-29T00:17:59.368 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.368 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:59.368 DEBUG:boto:Port: 8000 2023-07-29T00:17:59.369 DEBUG:boto:Params: {} 2023-07-29T00:17:59.369 DEBUG:boto:Token: None 2023-07-29T00:17:59.370 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/ 2023-07-29T00:17:59.370 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4= 2023-07-29T00:17:59.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4=', 'Content-Length': '0'} 2023-07-29T00:17:59.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb72bf4b47127052-0064c45ab7-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.373 DEBUG:boto:b'hiiqos-55e1000falsee2023-07-29T00:15:00.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:59.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-07-29T00:17:59.374 DEBUG:boto:path=/hiiqos-55/f 2023-07-29T00:17:59.374 DEBUG:boto:auth_path=/hiiqos-55/f 2023-07-29T00:17:59.375 DEBUG:boto:path=/hiiqos-55/f?versionId=null 2023-07-29T00:17:59.375 DEBUG:boto:auth_path=/hiiqos-55/f?versionId=null 2023-07-29T00:17:59.375 DEBUG:boto:Method: GET 2023-07-29T00:17:59.376 DEBUG:boto:Path: /hiiqos-55/f?versionId=null 2023-07-29T00:17:59.376 DEBUG:boto:Data: 2023-07-29T00:17:59.376 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.377 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:59.377 DEBUG:boto:Port: 8001 2023-07-29T00:17:59.378 DEBUG:boto:Params: {} 2023-07-29T00:17:59.378 DEBUG:boto:Token: None 2023-07-29T00:17:59.378 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/f?versionId=null 2023-07-29T00:17:59.379 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gLvuIDy4hi0Zr6zvW0YeFpRN28Y= 2023-07-29T00:17:59.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gLvuIDy4hi0Zr6zvW0YeFpRN28Y=', 'Content-Length': '0'} 2023-07-29T00:17:59.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4d651fed1ecabed-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.424 DEBUG:boto:path=/hiiqos-55/f 2023-07-29T00:17:59.425 DEBUG:boto:auth_path=/hiiqos-55/f 2023-07-29T00:17:59.425 DEBUG:boto:path=/hiiqos-55/f?versionId=null 2023-07-29T00:17:59.425 DEBUG:boto:auth_path=/hiiqos-55/f?versionId=null 2023-07-29T00:17:59.426 DEBUG:boto:Method: GET 2023-07-29T00:17:59.426 DEBUG:boto:Path: /hiiqos-55/f?versionId=null 2023-07-29T00:17:59.426 DEBUG:boto:Data: 2023-07-29T00:17:59.427 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.427 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:59.428 DEBUG:boto:Port: 8000 2023-07-29T00:17:59.428 DEBUG:boto:Params: {} 2023-07-29T00:17:59.428 DEBUG:boto:Token: None 2023-07-29T00:17:59.429 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/f?versionId=null 2023-07-29T00:17:59.429 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gLvuIDy4hi0Zr6zvW0YeFpRN28Y= 2023-07-29T00:17:59.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gLvuIDy4hi0Zr6zvW0YeFpRN28Y=', 'Content-Length': '0'} 2023-07-29T00:17:59.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005598885b4bdf9b58-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.432 DEBUG:boto:path=/hiiqos-55/f 2023-07-29T00:17:59.432 DEBUG:boto:auth_path=/hiiqos-55/f 2023-07-29T00:17:59.433 DEBUG:boto:path=/hiiqos-55/f?versionId=null 2023-07-29T00:17:59.433 DEBUG:boto:auth_path=/hiiqos-55/f?versionId=null 2023-07-29T00:17:59.433 DEBUG:boto:Method: HEAD 2023-07-29T00:17:59.434 DEBUG:boto:Path: /hiiqos-55/f?versionId=null 2023-07-29T00:17:59.434 DEBUG:boto:Data: 2023-07-29T00:17:59.434 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.435 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:59.435 DEBUG:boto:Port: 8001 2023-07-29T00:17:59.435 DEBUG:boto:Params: {} 2023-07-29T00:17:59.436 DEBUG:boto:Token: None 2023-07-29T00:17:59.436 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/f?versionId=null 2023-07-29T00:17:59.436 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pwJ581L8kNUghlSeFpOLE2E029o= 2023-07-29T00:17:59.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pwJ581L8kNUghlSeFpOLE2E029o=', 'Content-Length': '0'} 2023-07-29T00:17:59.438 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfdb624d78bb8902-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.439 DEBUG:boto:path=/hiiqos-55/f 2023-07-29T00:17:59.439 DEBUG:boto:auth_path=/hiiqos-55/f 2023-07-29T00:17:59.439 DEBUG:boto:path=/hiiqos-55/f?versionId=null 2023-07-29T00:17:59.440 DEBUG:boto:auth_path=/hiiqos-55/f?versionId=null 2023-07-29T00:17:59.440 DEBUG:boto:Method: HEAD 2023-07-29T00:17:59.441 DEBUG:boto:Path: /hiiqos-55/f?versionId=null 2023-07-29T00:17:59.441 DEBUG:boto:Data: 2023-07-29T00:17:59.441 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.442 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:59.442 DEBUG:boto:Port: 8000 2023-07-29T00:17:59.442 DEBUG:boto:Params: {} 2023-07-29T00:17:59.443 DEBUG:boto:Token: None 2023-07-29T00:17:59.443 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/f?versionId=null 2023-07-29T00:17:59.443 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pwJ581L8kNUghlSeFpOLE2E029o= 2023-07-29T00:17:59.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pwJ581L8kNUghlSeFpOLE2E029o=', 'Content-Length': '0'} 2023-07-29T00:17:59.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7cceadb5d6cad9c-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-07-29T00:17:59.446 DEBUG:boto:path=/hiiqos-55/f 2023-07-29T00:17:59.447 DEBUG:boto:auth_path=/hiiqos-55/f 2023-07-29T00:17:59.447 DEBUG:boto:path=/hiiqos-55/f?versionId=null 2023-07-29T00:17:59.447 DEBUG:boto:auth_path=/hiiqos-55/f?versionId=null 2023-07-29T00:17:59.448 DEBUG:boto:Method: GET 2023-07-29T00:17:59.448 DEBUG:boto:Path: /hiiqos-55/f?versionId=null 2023-07-29T00:17:59.448 DEBUG:boto:Data: 2023-07-29T00:17:59.449 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.449 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:59.449 DEBUG:boto:Port: 8001 2023-07-29T00:17:59.450 DEBUG:boto:Params: {} 2023-07-29T00:17:59.450 DEBUG:boto:Token: None 2023-07-29T00:17:59.451 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/f?versionId=null 2023-07-29T00:17:59.451 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gLvuIDy4hi0Zr6zvW0YeFpRN28Y= 2023-07-29T00:17:59.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gLvuIDy4hi0Zr6zvW0YeFpRN28Y=', 'Content-Length': '0'} 2023-07-29T00:17:59.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eda0b573ed529569-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.453 DEBUG:boto:path=/hiiqos-55/f 2023-07-29T00:17:59.454 DEBUG:boto:auth_path=/hiiqos-55/f 2023-07-29T00:17:59.454 DEBUG:boto:path=/hiiqos-55/f?versionId=null 2023-07-29T00:17:59.455 DEBUG:boto:auth_path=/hiiqos-55/f?versionId=null 2023-07-29T00:17:59.455 DEBUG:boto:Method: GET 2023-07-29T00:17:59.455 DEBUG:boto:Path: /hiiqos-55/f?versionId=null 2023-07-29T00:17:59.456 DEBUG:boto:Data: 2023-07-29T00:17:59.456 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.456 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:59.457 DEBUG:boto:Port: 8000 2023-07-29T00:17:59.457 DEBUG:boto:Params: {} 2023-07-29T00:17:59.458 DEBUG:boto:Token: None 2023-07-29T00:17:59.458 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/f?versionId=null 2023-07-29T00:17:59.458 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gLvuIDy4hi0Zr6zvW0YeFpRN28Y= 2023-07-29T00:17:59.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gLvuIDy4hi0Zr6zvW0YeFpRN28Y=', 'Content-Length': '0'} 2023-07-29T00:17:59.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005beadd3ac11a88ef-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.461 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:59.462 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:59.462 DEBUG:boto:path=/hiiqos-55/?prefix=f 2023-07-29T00:17:59.462 DEBUG:boto:auth_path=/hiiqos-55/?prefix=f 2023-07-29T00:17:59.463 DEBUG:boto:Method: GET 2023-07-29T00:17:59.463 DEBUG:boto:Path: /hiiqos-55/?prefix=f 2023-07-29T00:17:59.464 DEBUG:boto:Data: 2023-07-29T00:17:59.464 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.464 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:59.465 DEBUG:boto:Port: 8001 2023-07-29T00:17:59.465 DEBUG:boto:Params: {} 2023-07-29T00:17:59.465 DEBUG:boto:Token: None 2023-07-29T00:17:59.466 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/ 2023-07-29T00:17:59.466 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4= 2023-07-29T00:17:59.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4=', 'Content-Length': '0'} 2023-07-29T00:17:59.469 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005a9c67883a34fca-0064c45ab7-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.469 DEBUG:boto:b'hiiqos-55f1000falsef2023-07-29T00:15:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:59.470 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:59.470 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:59.471 DEBUG:boto:path=/hiiqos-55/?prefix=f 2023-07-29T00:17:59.471 DEBUG:boto:auth_path=/hiiqos-55/?prefix=f 2023-07-29T00:17:59.471 DEBUG:boto:Method: GET 2023-07-29T00:17:59.472 DEBUG:boto:Path: /hiiqos-55/?prefix=f 2023-07-29T00:17:59.472 DEBUG:boto:Data: 2023-07-29T00:17:59.473 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.473 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:59.473 DEBUG:boto:Port: 8000 2023-07-29T00:17:59.474 DEBUG:boto:Params: {} 2023-07-29T00:17:59.474 DEBUG:boto:Token: None 2023-07-29T00:17:59.475 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/ 2023-07-29T00:17:59.475 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4= 2023-07-29T00:17:59.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4=', 'Content-Length': '0'} 2023-07-29T00:17:59.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffa2f312c780d4d8-0064c45ab7-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.477 DEBUG:boto:b'hiiqos-55f1000falsef2023-07-29T00:15:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:59.478 DEBUG:boto:path=/hiiqos-55/f 2023-07-29T00:17:59.478 DEBUG:boto:auth_path=/hiiqos-55/f 2023-07-29T00:17:59.479 DEBUG:boto:Method: HEAD 2023-07-29T00:17:59.479 DEBUG:boto:Path: /hiiqos-55/f 2023-07-29T00:17:59.479 DEBUG:boto:Data: 2023-07-29T00:17:59.480 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.480 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:59.480 DEBUG:boto:Port: 8001 2023-07-29T00:17:59.481 DEBUG:boto:Params: {} 2023-07-29T00:17:59.481 DEBUG:boto:Token: None 2023-07-29T00:17:59.482 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/f 2023-07-29T00:17:59.482 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Uy41MOvlnmjNiKyRMFohexsmmoo= 2023-07-29T00:17:59.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Uy41MOvlnmjNiKyRMFohexsmmoo=', 'Content-Length': '0'} 2023-07-29T00:17:59.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004798b34cd5e549a5-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.484 DEBUG:boto:path=/hiiqos-55/f 2023-07-29T00:17:59.485 DEBUG:boto:auth_path=/hiiqos-55/f 2023-07-29T00:17:59.485 DEBUG:boto:Method: HEAD 2023-07-29T00:17:59.485 DEBUG:boto:Path: /hiiqos-55/f 2023-07-29T00:17:59.486 DEBUG:boto:Data: 2023-07-29T00:17:59.486 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.486 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:59.487 DEBUG:boto:Port: 8000 2023-07-29T00:17:59.487 DEBUG:boto:Params: {} 2023-07-29T00:17:59.487 DEBUG:boto:Token: None 2023-07-29T00:17:59.488 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/f 2023-07-29T00:17:59.488 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Uy41MOvlnmjNiKyRMFohexsmmoo= 2023-07-29T00:17:59.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Uy41MOvlnmjNiKyRMFohexsmmoo=', 'Content-Length': '0'} 2023-07-29T00:17:59.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a3ff7b1d2401937-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-07-29T00:17:59.491 DEBUG:boto:path=/hiiqos-55/f 2023-07-29T00:17:59.492 DEBUG:boto:auth_path=/hiiqos-55/f 2023-07-29T00:17:59.492 DEBUG:boto:Method: GET 2023-07-29T00:17:59.492 DEBUG:boto:Path: /hiiqos-55/f 2023-07-29T00:17:59.493 DEBUG:boto:Data: 2023-07-29T00:17:59.493 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.493 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:59.494 DEBUG:boto:Port: 8001 2023-07-29T00:17:59.494 DEBUG:boto:Params: {} 2023-07-29T00:17:59.495 DEBUG:boto:Token: None 2023-07-29T00:17:59.495 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/f 2023-07-29T00:17:59.496 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9aaUUp3s170DLQ1Y8LBuXurTmws= 2023-07-29T00:17:59.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9aaUUp3s170DLQ1Y8LBuXurTmws=', 'Content-Length': '0'} 2023-07-29T00:17:59.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e0021329d06c01b-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.498 DEBUG:boto:path=/hiiqos-55/f 2023-07-29T00:17:59.499 DEBUG:boto:auth_path=/hiiqos-55/f 2023-07-29T00:17:59.499 DEBUG:boto:Method: GET 2023-07-29T00:17:59.499 DEBUG:boto:Path: /hiiqos-55/f 2023-07-29T00:17:59.500 DEBUG:boto:Data: 2023-07-29T00:17:59.500 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.501 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:59.501 DEBUG:boto:Port: 8000 2023-07-29T00:17:59.501 DEBUG:boto:Params: {} 2023-07-29T00:17:59.502 DEBUG:boto:Token: None 2023-07-29T00:17:59.502 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/f 2023-07-29T00:17:59.502 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9aaUUp3s170DLQ1Y8LBuXurTmws= 2023-07-29T00:17:59.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9aaUUp3s170DLQ1Y8LBuXurTmws=', 'Content-Length': '0'} 2023-07-29T00:17:59.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000549c6a3c7f95cd5-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.548 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:59.549 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:59.549 DEBUG:boto:path=/hiiqos-55/?prefix=f 2023-07-29T00:17:59.550 DEBUG:boto:auth_path=/hiiqos-55/?prefix=f 2023-07-29T00:17:59.550 DEBUG:boto:Method: GET 2023-07-29T00:17:59.550 DEBUG:boto:Path: /hiiqos-55/?prefix=f 2023-07-29T00:17:59.551 DEBUG:boto:Data: 2023-07-29T00:17:59.551 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.551 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:59.552 DEBUG:boto:Port: 8001 2023-07-29T00:17:59.552 DEBUG:boto:Params: {} 2023-07-29T00:17:59.553 DEBUG:boto:Token: None 2023-07-29T00:17:59.553 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/ 2023-07-29T00:17:59.553 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4= 2023-07-29T00:17:59.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4=', 'Content-Length': '0'} 2023-07-29T00:17:59.556 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d480abe8c52d56d0-0064c45ab7-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.556 DEBUG:boto:b'hiiqos-55f1000falsef2023-07-29T00:15:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:59.557 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:59.557 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:59.558 DEBUG:boto:path=/hiiqos-55/?prefix=f 2023-07-29T00:17:59.558 DEBUG:boto:auth_path=/hiiqos-55/?prefix=f 2023-07-29T00:17:59.558 DEBUG:boto:Method: GET 2023-07-29T00:17:59.559 DEBUG:boto:Path: /hiiqos-55/?prefix=f 2023-07-29T00:17:59.559 DEBUG:boto:Data: 2023-07-29T00:17:59.559 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.560 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:59.560 DEBUG:boto:Port: 8000 2023-07-29T00:17:59.561 DEBUG:boto:Params: {} 2023-07-29T00:17:59.561 DEBUG:boto:Token: None 2023-07-29T00:17:59.561 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/ 2023-07-29T00:17:59.562 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4= 2023-07-29T00:17:59.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4=', 'Content-Length': '0'} 2023-07-29T00:17:59.564 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1e1ccd3b718d768-0064c45ab7-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.565 DEBUG:boto:b'hiiqos-55f1000falsef2023-07-29T00:15:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:59.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-07-29T00:17:59.566 DEBUG:boto:path=/hiiqos-55/g 2023-07-29T00:17:59.566 DEBUG:boto:auth_path=/hiiqos-55/g 2023-07-29T00:17:59.566 DEBUG:boto:path=/hiiqos-55/g?versionId=null 2023-07-29T00:17:59.567 DEBUG:boto:auth_path=/hiiqos-55/g?versionId=null 2023-07-29T00:17:59.567 DEBUG:boto:Method: GET 2023-07-29T00:17:59.567 DEBUG:boto:Path: /hiiqos-55/g?versionId=null 2023-07-29T00:17:59.568 DEBUG:boto:Data: 2023-07-29T00:17:59.568 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.568 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:59.569 DEBUG:boto:Port: 8001 2023-07-29T00:17:59.569 DEBUG:boto:Params: {} 2023-07-29T00:17:59.570 DEBUG:boto:Token: None 2023-07-29T00:17:59.570 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/g?versionId=null 2023-07-29T00:17:59.570 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZNRr3CNlRcBpQFGsX4+KIN9uMY4= 2023-07-29T00:17:59.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZNRr3CNlRcBpQFGsX4+KIN9uMY4=', 'Content-Length': '0'} 2023-07-29T00:17:59.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9dd95e7bdc62af7-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.573 DEBUG:boto:path=/hiiqos-55/g 2023-07-29T00:17:59.574 DEBUG:boto:auth_path=/hiiqos-55/g 2023-07-29T00:17:59.574 DEBUG:boto:path=/hiiqos-55/g?versionId=null 2023-07-29T00:17:59.574 DEBUG:boto:auth_path=/hiiqos-55/g?versionId=null 2023-07-29T00:17:59.575 DEBUG:boto:Method: GET 2023-07-29T00:17:59.575 DEBUG:boto:Path: /hiiqos-55/g?versionId=null 2023-07-29T00:17:59.576 DEBUG:boto:Data: 2023-07-29T00:17:59.576 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.577 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:59.577 DEBUG:boto:Port: 8000 2023-07-29T00:17:59.578 DEBUG:boto:Params: {} 2023-07-29T00:17:59.578 DEBUG:boto:Token: None 2023-07-29T00:17:59.578 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/g?versionId=null 2023-07-29T00:17:59.579 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZNRr3CNlRcBpQFGsX4+KIN9uMY4= 2023-07-29T00:17:59.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZNRr3CNlRcBpQFGsX4+KIN9uMY4=', 'Content-Length': '0'} 2023-07-29T00:17:59.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bad10613f68aca98-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.581 DEBUG:boto:path=/hiiqos-55/g 2023-07-29T00:17:59.582 DEBUG:boto:auth_path=/hiiqos-55/g 2023-07-29T00:17:59.582 DEBUG:boto:path=/hiiqos-55/g?versionId=null 2023-07-29T00:17:59.583 DEBUG:boto:auth_path=/hiiqos-55/g?versionId=null 2023-07-29T00:17:59.583 DEBUG:boto:Method: HEAD 2023-07-29T00:17:59.583 DEBUG:boto:Path: /hiiqos-55/g?versionId=null 2023-07-29T00:17:59.584 DEBUG:boto:Data: 2023-07-29T00:17:59.584 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.584 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:59.585 DEBUG:boto:Port: 8001 2023-07-29T00:17:59.585 DEBUG:boto:Params: {} 2023-07-29T00:17:59.585 DEBUG:boto:Token: None 2023-07-29T00:17:59.586 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/g?versionId=null 2023-07-29T00:17:59.586 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BnK/6Xcas2ogfhpOar48vO0D6mg= 2023-07-29T00:17:59.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BnK/6Xcas2ogfhpOar48vO0D6mg=', 'Content-Length': '0'} 2023-07-29T00:17:59.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddd7256a3859c453-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.589 DEBUG:boto:path=/hiiqos-55/g 2023-07-29T00:17:59.589 DEBUG:boto:auth_path=/hiiqos-55/g 2023-07-29T00:17:59.589 DEBUG:boto:path=/hiiqos-55/g?versionId=null 2023-07-29T00:17:59.590 DEBUG:boto:auth_path=/hiiqos-55/g?versionId=null 2023-07-29T00:17:59.590 DEBUG:boto:Method: HEAD 2023-07-29T00:17:59.590 DEBUG:boto:Path: /hiiqos-55/g?versionId=null 2023-07-29T00:17:59.591 DEBUG:boto:Data: 2023-07-29T00:17:59.591 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.591 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:59.592 DEBUG:boto:Port: 8000 2023-07-29T00:17:59.592 DEBUG:boto:Params: {} 2023-07-29T00:17:59.593 DEBUG:boto:Token: None 2023-07-29T00:17:59.593 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/g?versionId=null 2023-07-29T00:17:59.593 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BnK/6Xcas2ogfhpOar48vO0D6mg= 2023-07-29T00:17:59.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BnK/6Xcas2ogfhpOar48vO0D6mg=', 'Content-Length': '0'} 2023-07-29T00:17:59.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0814290aa6ff181-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-07-29T00:17:59.596 DEBUG:boto:path=/hiiqos-55/g 2023-07-29T00:17:59.596 DEBUG:boto:auth_path=/hiiqos-55/g 2023-07-29T00:17:59.597 DEBUG:boto:path=/hiiqos-55/g?versionId=null 2023-07-29T00:17:59.597 DEBUG:boto:auth_path=/hiiqos-55/g?versionId=null 2023-07-29T00:17:59.597 DEBUG:boto:Method: GET 2023-07-29T00:17:59.598 DEBUG:boto:Path: /hiiqos-55/g?versionId=null 2023-07-29T00:17:59.598 DEBUG:boto:Data: 2023-07-29T00:17:59.599 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.599 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:59.600 DEBUG:boto:Port: 8001 2023-07-29T00:17:59.601 DEBUG:boto:Params: {} 2023-07-29T00:17:59.601 DEBUG:boto:Token: None 2023-07-29T00:17:59.602 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/g?versionId=null 2023-07-29T00:17:59.603 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZNRr3CNlRcBpQFGsX4+KIN9uMY4= 2023-07-29T00:17:59.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZNRr3CNlRcBpQFGsX4+KIN9uMY4=', 'Content-Length': '0'} 2023-07-29T00:17:59.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f9053e0baa52689-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.648 DEBUG:boto:path=/hiiqos-55/g 2023-07-29T00:17:59.649 DEBUG:boto:auth_path=/hiiqos-55/g 2023-07-29T00:17:59.649 DEBUG:boto:path=/hiiqos-55/g?versionId=null 2023-07-29T00:17:59.650 DEBUG:boto:auth_path=/hiiqos-55/g?versionId=null 2023-07-29T00:17:59.651 DEBUG:boto:Method: GET 2023-07-29T00:17:59.652 DEBUG:boto:Path: /hiiqos-55/g?versionId=null 2023-07-29T00:17:59.653 DEBUG:boto:Data: 2023-07-29T00:17:59.653 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.654 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:59.655 DEBUG:boto:Port: 8000 2023-07-29T00:17:59.656 DEBUG:boto:Params: {} 2023-07-29T00:17:59.657 DEBUG:boto:Token: None 2023-07-29T00:17:59.657 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/g?versionId=null 2023-07-29T00:17:59.657 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZNRr3CNlRcBpQFGsX4+KIN9uMY4= 2023-07-29T00:17:59.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZNRr3CNlRcBpQFGsX4+KIN9uMY4=', 'Content-Length': '0'} 2023-07-29T00:17:59.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdcc814503478369-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.660 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:59.661 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:59.661 DEBUG:boto:path=/hiiqos-55/?prefix=g 2023-07-29T00:17:59.661 DEBUG:boto:auth_path=/hiiqos-55/?prefix=g 2023-07-29T00:17:59.662 DEBUG:boto:Method: GET 2023-07-29T00:17:59.662 DEBUG:boto:Path: /hiiqos-55/?prefix=g 2023-07-29T00:17:59.663 DEBUG:boto:Data: 2023-07-29T00:17:59.663 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.663 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:59.664 DEBUG:boto:Port: 8001 2023-07-29T00:17:59.664 DEBUG:boto:Params: {} 2023-07-29T00:17:59.664 DEBUG:boto:Token: None 2023-07-29T00:17:59.665 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/ 2023-07-29T00:17:59.665 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4= 2023-07-29T00:17:59.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4=', 'Content-Length': '0'} 2023-07-29T00:17:59.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002dcf10373327872e-0064c45ab7-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.668 DEBUG:boto:b'hiiqos-55g1000falseg2023-07-29T00:15:01.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:59.669 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:59.669 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:59.670 DEBUG:boto:path=/hiiqos-55/?prefix=g 2023-07-29T00:17:59.670 DEBUG:boto:auth_path=/hiiqos-55/?prefix=g 2023-07-29T00:17:59.670 DEBUG:boto:Method: GET 2023-07-29T00:17:59.671 DEBUG:boto:Path: /hiiqos-55/?prefix=g 2023-07-29T00:17:59.671 DEBUG:boto:Data: 2023-07-29T00:17:59.671 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.672 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:59.672 DEBUG:boto:Port: 8000 2023-07-29T00:17:59.672 DEBUG:boto:Params: {} 2023-07-29T00:17:59.673 DEBUG:boto:Token: None 2023-07-29T00:17:59.673 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/ 2023-07-29T00:17:59.674 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4= 2023-07-29T00:17:59.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4=', 'Content-Length': '0'} 2023-07-29T00:17:59.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083f2675daef90c69-0064c45ab7-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.676 DEBUG:boto:b'hiiqos-55g1000falseg2023-07-29T00:15:01.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:59.677 DEBUG:boto:path=/hiiqos-55/g 2023-07-29T00:17:59.677 DEBUG:boto:auth_path=/hiiqos-55/g 2023-07-29T00:17:59.678 DEBUG:boto:Method: HEAD 2023-07-29T00:17:59.678 DEBUG:boto:Path: /hiiqos-55/g 2023-07-29T00:17:59.678 DEBUG:boto:Data: 2023-07-29T00:17:59.679 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.679 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:59.679 DEBUG:boto:Port: 8001 2023-07-29T00:17:59.680 DEBUG:boto:Params: {} 2023-07-29T00:17:59.680 DEBUG:boto:Token: None 2023-07-29T00:17:59.681 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/g 2023-07-29T00:17:59.681 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rtczgeAVntG5FH/w9a/I55RBkEk= 2023-07-29T00:17:59.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rtczgeAVntG5FH/w9a/I55RBkEk=', 'Content-Length': '0'} 2023-07-29T00:17:59.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013c57afdc505dd5a-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.683 DEBUG:boto:path=/hiiqos-55/g 2023-07-29T00:17:59.684 DEBUG:boto:auth_path=/hiiqos-55/g 2023-07-29T00:17:59.684 DEBUG:boto:Method: HEAD 2023-07-29T00:17:59.685 DEBUG:boto:Path: /hiiqos-55/g 2023-07-29T00:17:59.685 DEBUG:boto:Data: 2023-07-29T00:17:59.685 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.686 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:59.686 DEBUG:boto:Port: 8000 2023-07-29T00:17:59.686 DEBUG:boto:Params: {} 2023-07-29T00:17:59.687 DEBUG:boto:Token: None 2023-07-29T00:17:59.687 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/g 2023-07-29T00:17:59.687 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rtczgeAVntG5FH/w9a/I55RBkEk= 2023-07-29T00:17:59.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rtczgeAVntG5FH/w9a/I55RBkEk=', 'Content-Length': '0'} 2023-07-29T00:17:59.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000071f706f259fd7b-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-07-29T00:17:59.691 DEBUG:boto:path=/hiiqos-55/g 2023-07-29T00:17:59.691 DEBUG:boto:auth_path=/hiiqos-55/g 2023-07-29T00:17:59.691 DEBUG:boto:Method: GET 2023-07-29T00:17:59.692 DEBUG:boto:Path: /hiiqos-55/g 2023-07-29T00:17:59.692 DEBUG:boto:Data: 2023-07-29T00:17:59.693 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.693 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:59.693 DEBUG:boto:Port: 8001 2023-07-29T00:17:59.694 DEBUG:boto:Params: {} 2023-07-29T00:17:59.694 DEBUG:boto:Token: None 2023-07-29T00:17:59.694 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/g 2023-07-29T00:17:59.695 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:q24ibV8HAnGltAk+0FvPg/aq+fk= 2023-07-29T00:17:59.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:q24ibV8HAnGltAk+0FvPg/aq+fk=', 'Content-Length': '0'} 2023-07-29T00:17:59.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099617417b4963fcf-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.697 DEBUG:boto:path=/hiiqos-55/g 2023-07-29T00:17:59.698 DEBUG:boto:auth_path=/hiiqos-55/g 2023-07-29T00:17:59.698 DEBUG:boto:Method: GET 2023-07-29T00:17:59.699 DEBUG:boto:Path: /hiiqos-55/g 2023-07-29T00:17:59.699 DEBUG:boto:Data: 2023-07-29T00:17:59.699 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.700 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:59.700 DEBUG:boto:Port: 8000 2023-07-29T00:17:59.700 DEBUG:boto:Params: {} 2023-07-29T00:17:59.701 DEBUG:boto:Token: None 2023-07-29T00:17:59.701 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/g 2023-07-29T00:17:59.702 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:q24ibV8HAnGltAk+0FvPg/aq+fk= 2023-07-29T00:17:59.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:q24ibV8HAnGltAk+0FvPg/aq+fk=', 'Content-Length': '0'} 2023-07-29T00:17:59.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b456d3548f3fd87-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.704 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:59.705 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:59.705 DEBUG:boto:path=/hiiqos-55/?prefix=g 2023-07-29T00:17:59.705 DEBUG:boto:auth_path=/hiiqos-55/?prefix=g 2023-07-29T00:17:59.706 DEBUG:boto:Method: GET 2023-07-29T00:17:59.706 DEBUG:boto:Path: /hiiqos-55/?prefix=g 2023-07-29T00:17:59.706 DEBUG:boto:Data: 2023-07-29T00:17:59.707 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.707 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:59.708 DEBUG:boto:Port: 8001 2023-07-29T00:17:59.708 DEBUG:boto:Params: {} 2023-07-29T00:17:59.708 DEBUG:boto:Token: None 2023-07-29T00:17:59.709 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/ 2023-07-29T00:17:59.709 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4= 2023-07-29T00:17:59.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4=', 'Content-Length': '0'} 2023-07-29T00:17:59.712 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004080cc390dbd7c3c-0064c45ab7-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.712 DEBUG:boto:b'hiiqos-55g1000falseg2023-07-29T00:15:01.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:59.713 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:59.713 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:59.714 DEBUG:boto:path=/hiiqos-55/?prefix=g 2023-07-29T00:17:59.714 DEBUG:boto:auth_path=/hiiqos-55/?prefix=g 2023-07-29T00:17:59.714 DEBUG:boto:Method: GET 2023-07-29T00:17:59.715 DEBUG:boto:Path: /hiiqos-55/?prefix=g 2023-07-29T00:17:59.715 DEBUG:boto:Data: 2023-07-29T00:17:59.715 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.716 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:59.716 DEBUG:boto:Port: 8000 2023-07-29T00:17:59.716 DEBUG:boto:Params: {} 2023-07-29T00:17:59.717 DEBUG:boto:Token: None 2023-07-29T00:17:59.717 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/ 2023-07-29T00:17:59.718 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4= 2023-07-29T00:17:59.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4=', 'Content-Length': '0'} 2023-07-29T00:17:59.720 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000237e45f4e04cfde0-0064c45ab7-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.720 DEBUG:boto:b'hiiqos-55g1000falseg2023-07-29T00:15:01.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:59.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-07-29T00:17:59.721 DEBUG:boto:path=/hiiqos-55/h 2023-07-29T00:17:59.722 DEBUG:boto:auth_path=/hiiqos-55/h 2023-07-29T00:17:59.722 DEBUG:boto:path=/hiiqos-55/h?versionId=null 2023-07-29T00:17:59.722 DEBUG:boto:auth_path=/hiiqos-55/h?versionId=null 2023-07-29T00:17:59.723 DEBUG:boto:Method: GET 2023-07-29T00:17:59.723 DEBUG:boto:Path: /hiiqos-55/h?versionId=null 2023-07-29T00:17:59.723 DEBUG:boto:Data: 2023-07-29T00:17:59.724 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.724 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:59.724 DEBUG:boto:Port: 8001 2023-07-29T00:17:59.725 DEBUG:boto:Params: {} 2023-07-29T00:17:59.725 DEBUG:boto:Token: None 2023-07-29T00:17:59.726 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/h?versionId=null 2023-07-29T00:17:59.726 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZEYei45sbVojlw8zUXRDZ8zvJjs= 2023-07-29T00:17:59.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZEYei45sbVojlw8zUXRDZ8zvJjs=', 'Content-Length': '0'} 2023-07-29T00:17:59.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db3b743a4bacf9c6-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.729 DEBUG:boto:path=/hiiqos-55/h 2023-07-29T00:17:59.729 DEBUG:boto:auth_path=/hiiqos-55/h 2023-07-29T00:17:59.730 DEBUG:boto:path=/hiiqos-55/h?versionId=null 2023-07-29T00:17:59.730 DEBUG:boto:auth_path=/hiiqos-55/h?versionId=null 2023-07-29T00:17:59.730 DEBUG:boto:Method: GET 2023-07-29T00:17:59.731 DEBUG:boto:Path: /hiiqos-55/h?versionId=null 2023-07-29T00:17:59.731 DEBUG:boto:Data: 2023-07-29T00:17:59.731 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.732 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:59.732 DEBUG:boto:Port: 8000 2023-07-29T00:17:59.732 DEBUG:boto:Params: {} 2023-07-29T00:17:59.733 DEBUG:boto:Token: None 2023-07-29T00:17:59.733 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/h?versionId=null 2023-07-29T00:17:59.734 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZEYei45sbVojlw8zUXRDZ8zvJjs= 2023-07-29T00:17:59.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZEYei45sbVojlw8zUXRDZ8zvJjs=', 'Content-Length': '0'} 2023-07-29T00:17:59.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcc9bcb14de3353d-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.736 DEBUG:boto:path=/hiiqos-55/h 2023-07-29T00:17:59.737 DEBUG:boto:auth_path=/hiiqos-55/h 2023-07-29T00:17:59.737 DEBUG:boto:path=/hiiqos-55/h?versionId=null 2023-07-29T00:17:59.737 DEBUG:boto:auth_path=/hiiqos-55/h?versionId=null 2023-07-29T00:17:59.738 DEBUG:boto:Method: HEAD 2023-07-29T00:17:59.738 DEBUG:boto:Path: /hiiqos-55/h?versionId=null 2023-07-29T00:17:59.738 DEBUG:boto:Data: 2023-07-29T00:17:59.739 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.739 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:59.739 DEBUG:boto:Port: 8001 2023-07-29T00:17:59.740 DEBUG:boto:Params: {} 2023-07-29T00:17:59.740 DEBUG:boto:Token: None 2023-07-29T00:17:59.740 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/h?versionId=null 2023-07-29T00:17:59.741 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6o39inkh5b4GqFq6WT4ng7z3+Fc= 2023-07-29T00:17:59.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6o39inkh5b4GqFq6WT4ng7z3+Fc=', 'Content-Length': '0'} 2023-07-29T00:17:59.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d96ab5e573f07457-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.743 DEBUG:boto:path=/hiiqos-55/h 2023-07-29T00:17:59.744 DEBUG:boto:auth_path=/hiiqos-55/h 2023-07-29T00:17:59.744 DEBUG:boto:path=/hiiqos-55/h?versionId=null 2023-07-29T00:17:59.744 DEBUG:boto:auth_path=/hiiqos-55/h?versionId=null 2023-07-29T00:17:59.745 DEBUG:boto:Method: HEAD 2023-07-29T00:17:59.745 DEBUG:boto:Path: /hiiqos-55/h?versionId=null 2023-07-29T00:17:59.745 DEBUG:boto:Data: 2023-07-29T00:17:59.746 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.746 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:59.746 DEBUG:boto:Port: 8000 2023-07-29T00:17:59.747 DEBUG:boto:Params: {} 2023-07-29T00:17:59.747 DEBUG:boto:Token: None 2023-07-29T00:17:59.747 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/h?versionId=null 2023-07-29T00:17:59.748 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6o39inkh5b4GqFq6WT4ng7z3+Fc= 2023-07-29T00:17:59.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6o39inkh5b4GqFq6WT4ng7z3+Fc=', 'Content-Length': '0'} 2023-07-29T00:17:59.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000167e0044fa8fbbaa-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-07-29T00:17:59.751 DEBUG:boto:path=/hiiqos-55/h 2023-07-29T00:17:59.751 DEBUG:boto:auth_path=/hiiqos-55/h 2023-07-29T00:17:59.751 DEBUG:boto:path=/hiiqos-55/h?versionId=null 2023-07-29T00:17:59.752 DEBUG:boto:auth_path=/hiiqos-55/h?versionId=null 2023-07-29T00:17:59.752 DEBUG:boto:Method: GET 2023-07-29T00:17:59.752 DEBUG:boto:Path: /hiiqos-55/h?versionId=null 2023-07-29T00:17:59.753 DEBUG:boto:Data: 2023-07-29T00:17:59.753 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.754 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:59.754 DEBUG:boto:Port: 8001 2023-07-29T00:17:59.754 DEBUG:boto:Params: {} 2023-07-29T00:17:59.755 DEBUG:boto:Token: None 2023-07-29T00:17:59.755 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/h?versionId=null 2023-07-29T00:17:59.755 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZEYei45sbVojlw8zUXRDZ8zvJjs= 2023-07-29T00:17:59.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZEYei45sbVojlw8zUXRDZ8zvJjs=', 'Content-Length': '0'} 2023-07-29T00:17:59.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df69615601489a82-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.804 DEBUG:boto:path=/hiiqos-55/h 2023-07-29T00:17:59.805 DEBUG:boto:auth_path=/hiiqos-55/h 2023-07-29T00:17:59.805 DEBUG:boto:path=/hiiqos-55/h?versionId=null 2023-07-29T00:17:59.806 DEBUG:boto:auth_path=/hiiqos-55/h?versionId=null 2023-07-29T00:17:59.806 DEBUG:boto:Method: GET 2023-07-29T00:17:59.806 DEBUG:boto:Path: /hiiqos-55/h?versionId=null 2023-07-29T00:17:59.807 DEBUG:boto:Data: 2023-07-29T00:17:59.807 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.807 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:59.808 DEBUG:boto:Port: 8000 2023-07-29T00:17:59.808 DEBUG:boto:Params: {} 2023-07-29T00:17:59.809 DEBUG:boto:Token: None 2023-07-29T00:17:59.809 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/h?versionId=null 2023-07-29T00:17:59.809 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZEYei45sbVojlw8zUXRDZ8zvJjs= 2023-07-29T00:17:59.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZEYei45sbVojlw8zUXRDZ8zvJjs=', 'Content-Length': '0'} 2023-07-29T00:17:59.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1f70643599f1dbf-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.812 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:59.813 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:59.813 DEBUG:boto:path=/hiiqos-55/?prefix=h 2023-07-29T00:17:59.814 DEBUG:boto:auth_path=/hiiqos-55/?prefix=h 2023-07-29T00:17:59.814 DEBUG:boto:Method: GET 2023-07-29T00:17:59.814 DEBUG:boto:Path: /hiiqos-55/?prefix=h 2023-07-29T00:17:59.815 DEBUG:boto:Data: 2023-07-29T00:17:59.815 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.815 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:59.816 DEBUG:boto:Port: 8001 2023-07-29T00:17:59.816 DEBUG:boto:Params: {} 2023-07-29T00:17:59.816 DEBUG:boto:Token: None 2023-07-29T00:17:59.817 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/ 2023-07-29T00:17:59.817 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4= 2023-07-29T00:17:59.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4=', 'Content-Length': '0'} 2023-07-29T00:17:59.820 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cce1a630b150b48-0064c45ab7-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.820 DEBUG:boto:b'hiiqos-55h1000falseh2023-07-29T00:15:01.033Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:59.821 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:59.821 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:59.822 DEBUG:boto:path=/hiiqos-55/?prefix=h 2023-07-29T00:17:59.822 DEBUG:boto:auth_path=/hiiqos-55/?prefix=h 2023-07-29T00:17:59.822 DEBUG:boto:Method: GET 2023-07-29T00:17:59.823 DEBUG:boto:Path: /hiiqos-55/?prefix=h 2023-07-29T00:17:59.823 DEBUG:boto:Data: 2023-07-29T00:17:59.823 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.824 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:59.824 DEBUG:boto:Port: 8000 2023-07-29T00:17:59.824 DEBUG:boto:Params: {} 2023-07-29T00:17:59.825 DEBUG:boto:Token: None 2023-07-29T00:17:59.825 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/ 2023-07-29T00:17:59.826 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4= 2023-07-29T00:17:59.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4=', 'Content-Length': '0'} 2023-07-29T00:17:59.828 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff79c5232917fef3-0064c45ab7-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.828 DEBUG:boto:b'hiiqos-55h1000falseh2023-07-29T00:15:01.033Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:59.829 DEBUG:boto:path=/hiiqos-55/h 2023-07-29T00:17:59.829 DEBUG:boto:auth_path=/hiiqos-55/h 2023-07-29T00:17:59.830 DEBUG:boto:Method: HEAD 2023-07-29T00:17:59.830 DEBUG:boto:Path: /hiiqos-55/h 2023-07-29T00:17:59.830 DEBUG:boto:Data: 2023-07-29T00:17:59.831 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.831 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:59.831 DEBUG:boto:Port: 8001 2023-07-29T00:17:59.832 DEBUG:boto:Params: {} 2023-07-29T00:17:59.832 DEBUG:boto:Token: None 2023-07-29T00:17:59.832 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/h 2023-07-29T00:17:59.833 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4wOvzVQ/RNGzUG918pfrTmMumes= 2023-07-29T00:17:59.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4wOvzVQ/RNGzUG918pfrTmMumes=', 'Content-Length': '0'} 2023-07-29T00:17:59.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007179146942739fa2-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.835 DEBUG:boto:path=/hiiqos-55/h 2023-07-29T00:17:59.836 DEBUG:boto:auth_path=/hiiqos-55/h 2023-07-29T00:17:59.836 DEBUG:boto:Method: HEAD 2023-07-29T00:17:59.836 DEBUG:boto:Path: /hiiqos-55/h 2023-07-29T00:17:59.837 DEBUG:boto:Data: 2023-07-29T00:17:59.837 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.837 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:59.838 DEBUG:boto:Port: 8000 2023-07-29T00:17:59.838 DEBUG:boto:Params: {} 2023-07-29T00:17:59.838 DEBUG:boto:Token: None 2023-07-29T00:17:59.839 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/h 2023-07-29T00:17:59.839 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4wOvzVQ/RNGzUG918pfrTmMumes= 2023-07-29T00:17:59.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4wOvzVQ/RNGzUG918pfrTmMumes=', 'Content-Length': '0'} 2023-07-29T00:17:59.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb22204b2e671787-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-07-29T00:17:59.842 DEBUG:boto:path=/hiiqos-55/h 2023-07-29T00:17:59.843 DEBUG:boto:auth_path=/hiiqos-55/h 2023-07-29T00:17:59.843 DEBUG:boto:Method: GET 2023-07-29T00:17:59.843 DEBUG:boto:Path: /hiiqos-55/h 2023-07-29T00:17:59.844 DEBUG:boto:Data: 2023-07-29T00:17:59.844 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.845 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:59.845 DEBUG:boto:Port: 8001 2023-07-29T00:17:59.845 DEBUG:boto:Params: {} 2023-07-29T00:17:59.846 DEBUG:boto:Token: None 2023-07-29T00:17:59.846 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/h 2023-07-29T00:17:59.847 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3xolI2FkEQ6M3/+29sdFYA3Jits= 2023-07-29T00:17:59.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3xolI2FkEQ6M3/+29sdFYA3Jits=', 'Content-Length': '0'} 2023-07-29T00:17:59.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0d196ff1cdd8964-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.849 DEBUG:boto:path=/hiiqos-55/h 2023-07-29T00:17:59.849 DEBUG:boto:auth_path=/hiiqos-55/h 2023-07-29T00:17:59.850 DEBUG:boto:Method: GET 2023-07-29T00:17:59.850 DEBUG:boto:Path: /hiiqos-55/h 2023-07-29T00:17:59.850 DEBUG:boto:Data: 2023-07-29T00:17:59.851 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.851 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:59.852 DEBUG:boto:Port: 8000 2023-07-29T00:17:59.852 DEBUG:boto:Params: {} 2023-07-29T00:17:59.852 DEBUG:boto:Token: None 2023-07-29T00:17:59.853 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/h 2023-07-29T00:17:59.853 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3xolI2FkEQ6M3/+29sdFYA3Jits= 2023-07-29T00:17:59.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3xolI2FkEQ6M3/+29sdFYA3Jits=', 'Content-Length': '0'} 2023-07-29T00:17:59.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048a712a8fed930f8-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.856 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:59.856 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:59.856 DEBUG:boto:path=/hiiqos-55/?prefix=h 2023-07-29T00:17:59.857 DEBUG:boto:auth_path=/hiiqos-55/?prefix=h 2023-07-29T00:17:59.857 DEBUG:boto:Method: GET 2023-07-29T00:17:59.858 DEBUG:boto:Path: /hiiqos-55/?prefix=h 2023-07-29T00:17:59.858 DEBUG:boto:Data: 2023-07-29T00:17:59.858 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.859 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:17:59.859 DEBUG:boto:Port: 8001 2023-07-29T00:17:59.859 DEBUG:boto:Params: {} 2023-07-29T00:17:59.860 DEBUG:boto:Token: None 2023-07-29T00:17:59.860 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/ 2023-07-29T00:17:59.861 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4= 2023-07-29T00:17:59.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4=', 'Content-Length': '0'} 2023-07-29T00:17:59.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5611b30beff0877-0064c45ab7-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.864 DEBUG:boto:b'hiiqos-55h1000falseh2023-07-29T00:15:01.033Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:59.865 DEBUG:boto:path=/hiiqos-55/ 2023-07-29T00:17:59.865 DEBUG:boto:auth_path=/hiiqos-55/ 2023-07-29T00:17:59.866 DEBUG:boto:path=/hiiqos-55/?prefix=h 2023-07-29T00:17:59.866 DEBUG:boto:auth_path=/hiiqos-55/?prefix=h 2023-07-29T00:17:59.866 DEBUG:boto:Method: GET 2023-07-29T00:17:59.867 DEBUG:boto:Path: /hiiqos-55/?prefix=h 2023-07-29T00:17:59.867 DEBUG:boto:Data: 2023-07-29T00:17:59.867 DEBUG:boto:Headers: {} 2023-07-29T00:17:59.868 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:17:59.868 DEBUG:boto:Port: 8000 2023-07-29T00:17:59.868 DEBUG:boto:Params: {} 2023-07-29T00:17:59.869 DEBUG:boto:Token: None 2023-07-29T00:17:59.869 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:17:59 GMT /hiiqos-55/ 2023-07-29T00:17:59.870 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4= 2023-07-29T00:17:59.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4=', 'Content-Length': '0'} 2023-07-29T00:17:59.872 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3ac07bdb7b3e088-0064c45ab7-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:17:59.872 DEBUG:boto:b'hiiqos-55h1000falseh2023-07-29T00:15:01.033Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:17:59.873 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-55 zones={a2, a1} 2023-07-29T00:17:59.873 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:17:59.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:17:59.874 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:17:59.916 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:17:59.916 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:17:59.957 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:17:59.912+0000 7f915205d640 -1 req 5459550028987789389 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-6:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.6,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-6:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.6,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T00:17:59.957 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:17:59.912+0000 7f9167888640 -1 req 13781160531294753919 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-36:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.1,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-36:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.1,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T00:17:59.957 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:17:59.912+0000 7f913d033640 -1 req 2754071433038047717 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-19:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.19,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-19:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.19,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T00:17:59.960 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:17:59.916+0000 7f9124001640 -1 req 3483033020764464808 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-23:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.23,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-23:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.23,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T00:18:00.002 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:17:59.957+0000 7f91197ec640 -1 req 264704333263720778 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-5:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.5,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-5:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.5,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T00:18:00.002 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:17:59.958+0000 7f911a7ee640 -1 req 4787960672522150306 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-51:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.16,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-51:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.16,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T00:18:00.045 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:18:00.000+0000 7f90fefb7640 -1 req 15118785672761743039 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-42:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.7,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-42:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.7,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T00:18:00.048 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:18:00.004+0000 7f91137e0640 -1 req 14210760628869684679 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-29:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.29,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-29:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.29,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T00:18:00.116 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'hiiqos-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:18:00.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'hiiqos-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:18:00.117 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket hiiqos-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:18:00.200 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:18:00.200 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:18:00.239 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:18:00.240 INFO:teuthology.orchestra.run.smithi101.stdout: "layout": { 2023-07-29T00:18:00.240 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding": "None", 2023-07-29T00:18:00.240 INFO:teuthology.orchestra.run.smithi101.stdout: "current_index": { 2023-07-29T00:18:00.240 INFO:teuthology.orchestra.run.smithi101.stdout: "gen": 2, 2023-07-29T00:18:00.240 INFO:teuthology.orchestra.run.smithi101.stdout: "layout": { 2023-07-29T00:18:00.240 INFO:teuthology.orchestra.run.smithi101.stdout: "type": "Normal", 2023-07-29T00:18:00.241 INFO:teuthology.orchestra.run.smithi101.stdout: "normal": { 2023-07-29T00:18:00.241 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 3, 2023-07-29T00:18:00.241 INFO:teuthology.orchestra.run.smithi101.stdout: "hash_type": "Mod" 2023-07-29T00:18:00.241 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:18:00.241 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:18:00.241 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:18:00.241 INFO:teuthology.orchestra.run.smithi101.stdout: "logs": [ 2023-07-29T00:18:00.242 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:18:00.242 INFO:teuthology.orchestra.run.smithi101.stdout: "gen": 1, 2023-07-29T00:18:00.242 INFO:teuthology.orchestra.run.smithi101.stdout: "layout": { 2023-07-29T00:18:00.242 INFO:teuthology.orchestra.run.smithi101.stdout: "type": "InIndex", 2023-07-29T00:18:00.242 INFO:teuthology.orchestra.run.smithi101.stdout: "in_index": { 2023-07-29T00:18:00.242 INFO:teuthology.orchestra.run.smithi101.stdout: "gen": 1, 2023-07-29T00:18:00.243 INFO:teuthology.orchestra.run.smithi101.stdout: "layout": { 2023-07-29T00:18:00.243 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 3, 2023-07-29T00:18:00.243 INFO:teuthology.orchestra.run.smithi101.stdout: "hash_type": "Mod" 2023-07-29T00:18:00.243 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:18:00.243 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:18:00.243 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:18:00.243 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:18:00.244 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:18:00.244 INFO:teuthology.orchestra.run.smithi101.stdout: "gen": 2, 2023-07-29T00:18:00.244 INFO:teuthology.orchestra.run.smithi101.stdout: "layout": { 2023-07-29T00:18:00.244 INFO:teuthology.orchestra.run.smithi101.stdout: "type": "InIndex", 2023-07-29T00:18:00.244 INFO:teuthology.orchestra.run.smithi101.stdout: "in_index": { 2023-07-29T00:18:00.244 INFO:teuthology.orchestra.run.smithi101.stdout: "gen": 2, 2023-07-29T00:18:00.244 INFO:teuthology.orchestra.run.smithi101.stdout: "layout": { 2023-07-29T00:18:00.245 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 3, 2023-07-29T00:18:00.245 INFO:teuthology.orchestra.run.smithi101.stdout: "hash_type": "Mod" 2023-07-29T00:18:00.245 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:18:00.245 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:18:00.245 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:18:00.245 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:18:00.245 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:18:00.246 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:18:00.246 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:18:00.247 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:18:00.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:18:00.247 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:18:00.290 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:18:00.290 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:18:00.330 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:18:00.285+0000 7f90ff7b8640 -1 req 5304715750979617260 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-41:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.6,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-41:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.6,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T00:18:00.331 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:18:00.287+0000 7f9121ffd640 -1 req 357210395645382615 0.000999975s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-38:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.3,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-38:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.3,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T00:18:00.332 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:18:00.287+0000 7f90e5784640 -1 req 4449530075996876019 0.000999975s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-1:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.1,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-1:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.1,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T00:18:00.380 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:18:00.336+0000 7f90d2f5f640 -1 req 14306619141323723770 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-4:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.4,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-4:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.4,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T00:18:00.418 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:18:00.373+0000 7f91b3920640 -1 req 121437746836377053 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-25:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.25,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-25:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.25,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T00:18:00.422 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:18:00.378+0000 7f91c213d640 -1 req 11353311073100918810 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-3:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.3,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-3:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.3,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T00:18:00.423 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:18:00.378+0000 7f91ad113640 -1 req 3289727646538734759 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-43:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.8,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-43:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.8,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T00:18:00.487 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'hiiqos-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:18:00.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'hiiqos-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:18:00.488 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket hiiqos-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:18:00.571 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:18:00.571 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:18:00.611 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:18:00.612 INFO:teuthology.orchestra.run.smithi101.stdout: "layout": { 2023-07-29T00:18:00.612 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding": "None", 2023-07-29T00:18:00.612 INFO:teuthology.orchestra.run.smithi101.stdout: "current_index": { 2023-07-29T00:18:00.612 INFO:teuthology.orchestra.run.smithi101.stdout: "gen": 2, 2023-07-29T00:18:00.612 INFO:teuthology.orchestra.run.smithi101.stdout: "layout": { 2023-07-29T00:18:00.612 INFO:teuthology.orchestra.run.smithi101.stdout: "type": "Normal", 2023-07-29T00:18:00.613 INFO:teuthology.orchestra.run.smithi101.stdout: "normal": { 2023-07-29T00:18:00.613 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 3, 2023-07-29T00:18:00.613 INFO:teuthology.orchestra.run.smithi101.stdout: "hash_type": "Mod" 2023-07-29T00:18:00.613 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:18:00.613 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:18:00.613 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:18:00.614 INFO:teuthology.orchestra.run.smithi101.stdout: "logs": [ 2023-07-29T00:18:00.614 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:18:00.614 INFO:teuthology.orchestra.run.smithi101.stdout: "gen": 1, 2023-07-29T00:18:00.614 INFO:teuthology.orchestra.run.smithi101.stdout: "layout": { 2023-07-29T00:18:00.614 INFO:teuthology.orchestra.run.smithi101.stdout: "type": "InIndex", 2023-07-29T00:18:00.614 INFO:teuthology.orchestra.run.smithi101.stdout: "in_index": { 2023-07-29T00:18:00.615 INFO:teuthology.orchestra.run.smithi101.stdout: "gen": 1, 2023-07-29T00:18:00.615 INFO:teuthology.orchestra.run.smithi101.stdout: "layout": { 2023-07-29T00:18:00.615 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 3, 2023-07-29T00:18:00.615 INFO:teuthology.orchestra.run.smithi101.stdout: "hash_type": "Mod" 2023-07-29T00:18:00.615 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:18:00.615 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:18:00.616 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:18:00.616 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:18:00.616 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:18:00.616 INFO:teuthology.orchestra.run.smithi101.stdout: "gen": 2, 2023-07-29T00:18:00.616 INFO:teuthology.orchestra.run.smithi101.stdout: "layout": { 2023-07-29T00:18:00.616 INFO:teuthology.orchestra.run.smithi101.stdout: "type": "InIndex", 2023-07-29T00:18:00.617 INFO:teuthology.orchestra.run.smithi101.stdout: "in_index": { 2023-07-29T00:18:00.617 INFO:teuthology.orchestra.run.smithi101.stdout: "gen": 2, 2023-07-29T00:18:00.617 INFO:teuthology.orchestra.run.smithi101.stdout: "layout": { 2023-07-29T00:18:00.617 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 3, 2023-07-29T00:18:00.617 INFO:teuthology.orchestra.run.smithi101.stdout: "hash_type": "Mod" 2023-07-29T00:18:00.617 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:18:00.617 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:18:00.618 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:18:00.618 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:18:00.618 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:18:00.618 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:18:00.618 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:18:00.623 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_index_log_trim ... FAIL 2023-07-29T00:18:00.624 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:18:00.624 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:18:00.625 DEBUG:boto:Method: PUT 2023-07-29T00:18:00.625 DEBUG:boto:Path: /hiiqos-56/ 2023-07-29T00:18:00.625 DEBUG:boto:Data: 2023-07-29T00:18:00.626 DEBUG:boto:Headers: {} 2023-07-29T00:18:00.626 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:00.626 DEBUG:boto:Port: 8000 2023-07-29T00:18:00.627 DEBUG:boto:Params: {} 2023-07-29T00:18:00.627 DEBUG:boto:Token: None 2023-07-29T00:18:00.628 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 00:18:00 GMT /hiiqos-56/ 2023-07-29T00:18:00.628 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:j96ADP3d7NwVBaq4d14BzMJbpV8= 2023-07-29T00:18:00.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:j96ADP3d7NwVBaq4d14BzMJbpV8=', 'Content-Length': '0'} 2023-07-29T00:18:00.642 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dc5d9132945c8445-0064c45ab8-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:00.642 DEBUG:rgw_multi.tests:created bucket=hiiqos-56 2023-07-29T00:18:00.643 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:18:00.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:18:00.643 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:18:00.704 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:18:00.704 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:18:00.733 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T00:18:00.746 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T00:18:00.746 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:18:00.747 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:18:00.747 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:46:09.233742Z" 2023-07-29T00:18:00.747 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:18:00.747 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:18:00.747 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690589880.597311_873.1", 2023-07-29T00:18:00.747 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:18:00.597311Z" 2023-07-29T00:18:00.747 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:18:00.748 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:18:00.748 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:18:00.748 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T00:18:00.748 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:18:00.748 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:18:00.748 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:18:00.749 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T00:18:00.749 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:18:00.749 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T00:18:00.750 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690587969.233742_99.1', 1: '1_1690589880.597311_873.1', 2: '', 3: ''} 2023-07-29T00:18:00.750 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T00:18:00.750 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:18:00.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:18:00.751 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:18:00.799 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:18:00.800 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:18:00.827 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:18:00.783+0000 7fb33e922880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:18:00.828 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:18:00.784+0000 7fb33e922880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:18:00.841 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:18:00.841 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:18:00.841 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:18:00.842 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:18:00.842 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:18:00.842 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:18:00.842 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:18:00.842 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:18:00.842 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:18:00.843 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:18:00.843 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:18:00.843 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:18:00.843 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:18:00.843 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:18:00.843 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:18:00.843 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:18:00.844 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:18:00.844 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:18:00.844 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:18:00.844 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:18:00.844 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:18:00.844 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:18:00.845 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:18:00.845 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:18:00.845 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:18:00.845 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690589880.597311_873.1", 2023-07-29T00:18:00.845 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:18:00.845 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:18:00.845 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:18:00.846 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:18:00.597311Z", 2023-07-29T00:18:00.846 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:18:00.846 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:18:00.846 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:18:00.846 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:18:00.846 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:18:00.846 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:18:00.847 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:18:00.847 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:18:00.847 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:18:00.847 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:18:00.847 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:18:00.847 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:18:00.848 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:18:00.848 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:18:00.848 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:18:00.848 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:18:00.849 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:18:00.849 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:18:00.849 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:18:00.849 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:18:00.849 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:18:00.849 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:18:00.850 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:18:00.850 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:18:00.850 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:18:00.850 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:18:00.850 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:18:00.850 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:18:00.850 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:18:00.851 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:18:00.851 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:18:00.851 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:18:00.851 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:18:00.851 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:18:00Z" 2023-07-29T00:18:00.851 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:18:00.852 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690589880.597311_873.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:18:00.597311Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:18:00Z" } 2023-07-29T00:18:00.853 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690589880.597311_873.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:18:00.597311Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:18:00.853 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690589880.597311_873.1', 2: '', 3: ''} 2023-07-29T00:18:00.853 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690589880.597311_873.1', 2: '', 3: ''} 2023-07-29T00:18:00.854 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T00:18:00.854 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:18:00.854 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:18:00.855 DEBUG:boto:Method: HEAD 2023-07-29T00:18:00.855 DEBUG:boto:Path: /hiiqos-56/ 2023-07-29T00:18:00.855 DEBUG:boto:Data: 2023-07-29T00:18:00.856 DEBUG:boto:Headers: {} 2023-07-29T00:18:00.856 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:00.856 DEBUG:boto:Port: 8000 2023-07-29T00:18:00.857 DEBUG:boto:Params: {} 2023-07-29T00:18:00.857 DEBUG:boto:Token: None 2023-07-29T00:18:00.857 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:18:00 GMT /hiiqos-56/ 2023-07-29T00:18:00.858 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:raxH0YpgFogCN5B22qwFdVLTqwA= 2023-07-29T00:18:00.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:raxH0YpgFogCN5B22qwFdVLTqwA=', 'Content-Length': '0'} 2023-07-29T00:18:00.861 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001484ab95d227b20e-0064c45ab8-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:00.861 DEBUG:boto:path=/hiiqos-56/a 2023-07-29T00:18:00.862 DEBUG:boto:auth_path=/hiiqos-56/a 2023-07-29T00:18:00.862 DEBUG:boto:Method: PUT 2023-07-29T00:18:00.862 DEBUG:boto:Path: /hiiqos-56/a 2023-07-29T00:18:00.863 DEBUG:boto:Data: 2023-07-29T00:18:00.863 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:18:00.863 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:00.864 DEBUG:boto:Port: 8000 2023-07-29T00:18:00.864 DEBUG:boto:Params: {} 2023-07-29T00:18:00.864 DEBUG:boto:Token: None 2023-07-29T00:18:00.865 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:18:00 GMT /hiiqos-56/a 2023-07-29T00:18:00.865 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ezzVC1ShdYiTg1q1lA6U6iXdAAQ= 2023-07-29T00:18:00.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:18:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ezzVC1ShdYiTg1q1lA6U6iXdAAQ='} 2023-07-29T00:18:00.871 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000003ebad1c3bb5a3c3-0064c45ab8-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:00.872 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:18:00.872 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:18:00.872 DEBUG:boto:Method: HEAD 2023-07-29T00:18:00.873 DEBUG:boto:Path: /hiiqos-56/ 2023-07-29T00:18:00.873 DEBUG:boto:Data: 2023-07-29T00:18:00.873 DEBUG:boto:Headers: {} 2023-07-29T00:18:00.874 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:00.874 DEBUG:boto:Port: 8000 2023-07-29T00:18:00.874 DEBUG:boto:Params: {} 2023-07-29T00:18:00.875 DEBUG:boto:Token: None 2023-07-29T00:18:00.875 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:18:00 GMT /hiiqos-56/ 2023-07-29T00:18:00.875 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:raxH0YpgFogCN5B22qwFdVLTqwA= 2023-07-29T00:18:00.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:raxH0YpgFogCN5B22qwFdVLTqwA=', 'Content-Length': '0'} 2023-07-29T00:18:00.878 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b7a2312014085ccc-0064c45ab8-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:00.879 DEBUG:boto:path=/hiiqos-56/b 2023-07-29T00:18:00.879 DEBUG:boto:auth_path=/hiiqos-56/b 2023-07-29T00:18:00.879 DEBUG:boto:Method: PUT 2023-07-29T00:18:00.880 DEBUG:boto:Path: /hiiqos-56/b 2023-07-29T00:18:00.880 DEBUG:boto:Data: 2023-07-29T00:18:00.880 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:18:00.881 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:00.881 DEBUG:boto:Port: 8000 2023-07-29T00:18:00.882 DEBUG:boto:Params: {} 2023-07-29T00:18:00.882 DEBUG:boto:Token: None 2023-07-29T00:18:00.882 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:18:00 GMT /hiiqos-56/b 2023-07-29T00:18:00.883 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4MdBn8ZCGpJPjDVkrn0em6iLGU8= 2023-07-29T00:18:00.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:18:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4MdBn8ZCGpJPjDVkrn0em6iLGU8='} 2023-07-29T00:18:00.888 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c00ce8c01ed8dcfb-0064c45ab8-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:00.888 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:18:00.889 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:18:00.889 DEBUG:boto:Method: HEAD 2023-07-29T00:18:00.890 DEBUG:boto:Path: /hiiqos-56/ 2023-07-29T00:18:00.890 DEBUG:boto:Data: 2023-07-29T00:18:00.890 DEBUG:boto:Headers: {} 2023-07-29T00:18:00.891 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:00.891 DEBUG:boto:Port: 8000 2023-07-29T00:18:00.891 DEBUG:boto:Params: {} 2023-07-29T00:18:00.892 DEBUG:boto:Token: None 2023-07-29T00:18:00.892 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:18:00 GMT /hiiqos-56/ 2023-07-29T00:18:00.892 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:raxH0YpgFogCN5B22qwFdVLTqwA= 2023-07-29T00:18:00.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:raxH0YpgFogCN5B22qwFdVLTqwA=', 'Content-Length': '0'} 2023-07-29T00:18:00.895 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003f61b365d10bd1cb-0064c45ab8-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:00.895 DEBUG:boto:path=/hiiqos-56/c 2023-07-29T00:18:00.896 DEBUG:boto:auth_path=/hiiqos-56/c 2023-07-29T00:18:00.896 DEBUG:boto:Method: PUT 2023-07-29T00:18:00.897 DEBUG:boto:Path: /hiiqos-56/c 2023-07-29T00:18:00.897 DEBUG:boto:Data: 2023-07-29T00:18:00.897 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:18:00.898 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:00.898 DEBUG:boto:Port: 8000 2023-07-29T00:18:00.898 DEBUG:boto:Params: {} 2023-07-29T00:18:00.899 DEBUG:boto:Token: None 2023-07-29T00:18:00.899 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:18:00 GMT /hiiqos-56/c 2023-07-29T00:18:00.899 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:t/6k4nU0SJ2FGQL8pNVDBki4KFM= 2023-07-29T00:18:00.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:18:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t/6k4nU0SJ2FGQL8pNVDBki4KFM='} 2023-07-29T00:18:00.944 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f056980748d79154-0064c45ab8-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:00.945 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:18:00.945 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:18:00.945 DEBUG:boto:Method: HEAD 2023-07-29T00:18:00.946 DEBUG:boto:Path: /hiiqos-56/ 2023-07-29T00:18:00.946 DEBUG:boto:Data: 2023-07-29T00:18:00.946 DEBUG:boto:Headers: {} 2023-07-29T00:18:00.947 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:00.947 DEBUG:boto:Port: 8000 2023-07-29T00:18:00.947 DEBUG:boto:Params: {} 2023-07-29T00:18:00.948 DEBUG:boto:Token: None 2023-07-29T00:18:00.948 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:18:00 GMT /hiiqos-56/ 2023-07-29T00:18:00.949 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:raxH0YpgFogCN5B22qwFdVLTqwA= 2023-07-29T00:18:00.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:raxH0YpgFogCN5B22qwFdVLTqwA=', 'Content-Length': '0'} 2023-07-29T00:18:00.951 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a9cca5384db6258b-0064c45ab8-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:00.951 DEBUG:boto:path=/hiiqos-56/d 2023-07-29T00:18:00.952 DEBUG:boto:auth_path=/hiiqos-56/d 2023-07-29T00:18:00.952 DEBUG:boto:Method: PUT 2023-07-29T00:18:00.952 DEBUG:boto:Path: /hiiqos-56/d 2023-07-29T00:18:00.953 DEBUG:boto:Data: 2023-07-29T00:18:00.953 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:18:00.954 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:00.954 DEBUG:boto:Port: 8000 2023-07-29T00:18:00.954 DEBUG:boto:Params: {} 2023-07-29T00:18:00.955 DEBUG:boto:Token: None 2023-07-29T00:18:00.955 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:18:00 GMT /hiiqos-56/d 2023-07-29T00:18:00.955 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+9/CXfg+dn7dU0tkKr7SEuL3FP4= 2023-07-29T00:18:00.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:18:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+9/CXfg+dn7dU0tkKr7SEuL3FP4='} 2023-07-29T00:18:00.960 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000974bae697622cdba-0064c45ab8-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:00.961 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-56 2023-07-29T00:18:00.961 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:18:00.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:18:00.962 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-56 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:18:01.015 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:18:01.015 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:18:01.047 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:18:01.002+0000 7f314b72c880 1 waiting to reach incremental sync.. 2023-07-29T00:18:06.048 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:18:06.003+0000 7f314b72c880 1 waiting to reach incremental sync.. 2023-07-29T00:18:11.048 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:18:11.004+0000 7f314b72c880 1 waiting to reach incremental sync.. 2023-07-29T00:18:16.050 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:18:16.005+0000 7f314b72c880 1 bucket sync caught up with source: 2023-07-29T00:18:16.050 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000001.22.6, , , , 00000000001.102.6, , , 00000000001.19.6, , 00000000001.68.6] 2023-07-29T00:18:16.051 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000001.22.6, , , , 00000000001.102.6, , , 00000000001.19.6, , 00000000001.68.6] 2023-07-29T00:18:16.051 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:18:16.005+0000 7f314b72c880 0 bucket checkpoint complete 2023-07-29T00:18:16.067 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-56 2023-07-29T00:18:16.068 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:18:16.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:18:16.068 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-56 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:18:16.126 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:18:16.127 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:18:16.161 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:18:16.116+0000 7f73d3d2a880 1 waiting to reach incremental sync.. 2023-07-29T00:18:21.162 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:18:21.118+0000 7f73d3d2a880 1 bucket sync caught up with source: 2023-07-29T00:18:21.163 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000001.987.6, , , , 00000000001.762.6, , , 00000000001.991.6, , 00000000001.876.6] 2023-07-29T00:18:21.163 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000001.987.6, , , , 00000000001.762.6, , , 00000000001.991.6, , 00000000001.876.6] 2023-07-29T00:18:21.163 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:18:21.118+0000 7f73d3d2a880 0 bucket checkpoint complete 2023-07-29T00:18:21.180 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-56 zones={a2, a1} 2023-07-29T00:18:21.180 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:18:21.180 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:18:21.181 DEBUG:boto:Method: HEAD 2023-07-29T00:18:21.181 DEBUG:boto:Path: /hiiqos-56/ 2023-07-29T00:18:21.182 DEBUG:boto:Data: 2023-07-29T00:18:21.182 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.182 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:18:21.183 DEBUG:boto:Port: 8001 2023-07-29T00:18:21.183 DEBUG:boto:Params: {} 2023-07-29T00:18:21.183 DEBUG:boto:Token: None 2023-07-29T00:18:21.184 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/ 2023-07-29T00:18:21.184 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7qRzxITA9vSElYoRF0McIh5wNFE= 2023-07-29T00:18:21.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7qRzxITA9vSElYoRF0McIh5wNFE=', 'Content-Length': '0'} 2023-07-29T00:18:21.187 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007e6802adc3719a33-0064c45acd-1663-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.187 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:18:21.188 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:18:21.188 DEBUG:boto:Method: HEAD 2023-07-29T00:18:21.188 DEBUG:boto:Path: /hiiqos-56/ 2023-07-29T00:18:21.189 DEBUG:boto:Data: 2023-07-29T00:18:21.189 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.189 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:21.190 DEBUG:boto:Port: 8000 2023-07-29T00:18:21.190 DEBUG:boto:Params: {} 2023-07-29T00:18:21.190 DEBUG:boto:Token: None 2023-07-29T00:18:21.191 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/ 2023-07-29T00:18:21.191 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7qRzxITA9vSElYoRF0McIh5wNFE= 2023-07-29T00:18:21.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7qRzxITA9vSElYoRF0McIh5wNFE=', 'Content-Length': '0'} 2023-07-29T00:18:21.194 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006ac46f4de1becc4e-0064c45acd-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.195 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-29T00:18:21.195 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:18:21.195 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:18:21.196 DEBUG:boto:path=/hiiqos-56/?versions&max-keys=999 2023-07-29T00:18:21.196 DEBUG:boto:auth_path=/hiiqos-56/?versions&max-keys=999 2023-07-29T00:18:21.196 DEBUG:boto:Method: GET 2023-07-29T00:18:21.197 DEBUG:boto:Path: /hiiqos-56/?versions&max-keys=999 2023-07-29T00:18:21.197 DEBUG:boto:Data: 2023-07-29T00:18:21.197 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.198 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:18:21.198 DEBUG:boto:Port: 8001 2023-07-29T00:18:21.198 DEBUG:boto:Params: {} 2023-07-29T00:18:21.199 DEBUG:boto:Token: None 2023-07-29T00:18:21.199 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/?versions 2023-07-29T00:18:21.200 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UYMx+nMVQjmkfB9OYTsQKd4SaQY= 2023-07-29T00:18:21.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UYMx+nMVQjmkfB9OYTsQKd4SaQY=', 'Content-Length': '0'} 2023-07-29T00:18:21.203 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048269ecd1e6e1edc-0064c45acd-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.204 DEBUG:boto:b'hiiqos-56999falseanulltrue2023-07-29T00:18:00.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:18:00.841Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:18:00.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:18:00.914Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:18:21.204 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-07-29T00:18:21.205 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-07-29T00:18:21.205 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-07-29T00:18:21.205 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-07-29T00:18:21.206 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-29T00:18:21.206 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:18:21.206 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:18:21.207 DEBUG:boto:path=/hiiqos-56/?versions&max-keys=999 2023-07-29T00:18:21.207 DEBUG:boto:auth_path=/hiiqos-56/?versions&max-keys=999 2023-07-29T00:18:21.207 DEBUG:boto:Method: GET 2023-07-29T00:18:21.208 DEBUG:boto:Path: /hiiqos-56/?versions&max-keys=999 2023-07-29T00:18:21.208 DEBUG:boto:Data: 2023-07-29T00:18:21.208 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.209 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:21.209 DEBUG:boto:Port: 8000 2023-07-29T00:18:21.209 DEBUG:boto:Params: {} 2023-07-29T00:18:21.210 DEBUG:boto:Token: None 2023-07-29T00:18:21.210 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/?versions 2023-07-29T00:18:21.211 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UYMx+nMVQjmkfB9OYTsQKd4SaQY= 2023-07-29T00:18:21.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UYMx+nMVQjmkfB9OYTsQKd4SaQY=', 'Content-Length': '0'} 2023-07-29T00:18:21.213 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077c7b2d9d041041f-0064c45acd-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.214 DEBUG:boto:b'hiiqos-56999falseanulltrue2023-07-29T00:18:00.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:18:00.841Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:18:00.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:18:00.914Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:18:21.215 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-07-29T00:18:21.215 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-07-29T00:18:21.215 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-07-29T00:18:21.216 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-07-29T00:18:21.216 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:18:21.216 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:18:21.217 DEBUG:boto:path=/hiiqos-56/?versions&max-keys=999 2023-07-29T00:18:21.217 DEBUG:boto:auth_path=/hiiqos-56/?versions&max-keys=999 2023-07-29T00:18:21.217 DEBUG:boto:Method: GET 2023-07-29T00:18:21.218 DEBUG:boto:Path: /hiiqos-56/?versions&max-keys=999 2023-07-29T00:18:21.218 DEBUG:boto:Data: 2023-07-29T00:18:21.218 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.219 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:18:21.219 DEBUG:boto:Port: 8001 2023-07-29T00:18:21.219 DEBUG:boto:Params: {} 2023-07-29T00:18:21.220 DEBUG:boto:Token: None 2023-07-29T00:18:21.220 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/?versions 2023-07-29T00:18:21.221 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UYMx+nMVQjmkfB9OYTsQKd4SaQY= 2023-07-29T00:18:21.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UYMx+nMVQjmkfB9OYTsQKd4SaQY=', 'Content-Length': '0'} 2023-07-29T00:18:21.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059c106f3385bff8c-0064c45acd-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.224 DEBUG:boto:b'hiiqos-56999falseanulltrue2023-07-29T00:18:00.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:18:00.841Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:18:00.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:18:00.914Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:18:21.225 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:18:21.225 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:18:21.225 DEBUG:boto:path=/hiiqos-56/?versions&max-keys=999 2023-07-29T00:18:21.226 DEBUG:boto:auth_path=/hiiqos-56/?versions&max-keys=999 2023-07-29T00:18:21.226 DEBUG:boto:Method: GET 2023-07-29T00:18:21.226 DEBUG:boto:Path: /hiiqos-56/?versions&max-keys=999 2023-07-29T00:18:21.227 DEBUG:boto:Data: 2023-07-29T00:18:21.227 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.227 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:21.228 DEBUG:boto:Port: 8000 2023-07-29T00:18:21.228 DEBUG:boto:Params: {} 2023-07-29T00:18:21.228 DEBUG:boto:Token: None 2023-07-29T00:18:21.229 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/?versions 2023-07-29T00:18:21.229 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UYMx+nMVQjmkfB9OYTsQKd4SaQY= 2023-07-29T00:18:21.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UYMx+nMVQjmkfB9OYTsQKd4SaQY=', 'Content-Length': '0'} 2023-07-29T00:18:21.232 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a2069be237c6f9c-0064c45acd-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.233 DEBUG:boto:b'hiiqos-56999falseanulltrue2023-07-29T00:18:00.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:18:00.841Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:18:00.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:18:00.914Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:18:21.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-07-29T00:18:21.234 DEBUG:boto:path=/hiiqos-56/a 2023-07-29T00:18:21.234 DEBUG:boto:auth_path=/hiiqos-56/a 2023-07-29T00:18:21.235 DEBUG:boto:path=/hiiqos-56/a?versionId=null 2023-07-29T00:18:21.235 DEBUG:boto:auth_path=/hiiqos-56/a?versionId=null 2023-07-29T00:18:21.235 DEBUG:boto:Method: GET 2023-07-29T00:18:21.236 DEBUG:boto:Path: /hiiqos-56/a?versionId=null 2023-07-29T00:18:21.236 DEBUG:boto:Data: 2023-07-29T00:18:21.236 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.237 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:18:21.237 DEBUG:boto:Port: 8001 2023-07-29T00:18:21.237 DEBUG:boto:Params: {} 2023-07-29T00:18:21.238 DEBUG:boto:Token: None 2023-07-29T00:18:21.238 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/a?versionId=null 2023-07-29T00:18:21.239 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZyXs6bVRD2cKb4+gIm8GSuOemDI= 2023-07-29T00:18:21.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZyXs6bVRD2cKb4+gIm8GSuOemDI=', 'Content-Length': '0'} 2023-07-29T00:18:21.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079e38cd0acd76caf-0064c45acd-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.242 DEBUG:boto:path=/hiiqos-56/a 2023-07-29T00:18:21.242 DEBUG:boto:auth_path=/hiiqos-56/a 2023-07-29T00:18:21.242 DEBUG:boto:path=/hiiqos-56/a?versionId=null 2023-07-29T00:18:21.243 DEBUG:boto:auth_path=/hiiqos-56/a?versionId=null 2023-07-29T00:18:21.243 DEBUG:boto:Method: GET 2023-07-29T00:18:21.243 DEBUG:boto:Path: /hiiqos-56/a?versionId=null 2023-07-29T00:18:21.244 DEBUG:boto:Data: 2023-07-29T00:18:21.244 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.245 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:21.245 DEBUG:boto:Port: 8000 2023-07-29T00:18:21.245 DEBUG:boto:Params: {} 2023-07-29T00:18:21.246 DEBUG:boto:Token: None 2023-07-29T00:18:21.246 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/a?versionId=null 2023-07-29T00:18:21.246 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZyXs6bVRD2cKb4+gIm8GSuOemDI= 2023-07-29T00:18:21.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZyXs6bVRD2cKb4+gIm8GSuOemDI=', 'Content-Length': '0'} 2023-07-29T00:18:21.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8ea35b62d3369a7-0064c45acd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.292 DEBUG:boto:path=/hiiqos-56/a 2023-07-29T00:18:21.293 DEBUG:boto:auth_path=/hiiqos-56/a 2023-07-29T00:18:21.293 DEBUG:boto:path=/hiiqos-56/a?versionId=null 2023-07-29T00:18:21.293 DEBUG:boto:auth_path=/hiiqos-56/a?versionId=null 2023-07-29T00:18:21.294 DEBUG:boto:Method: HEAD 2023-07-29T00:18:21.294 DEBUG:boto:Path: /hiiqos-56/a?versionId=null 2023-07-29T00:18:21.295 DEBUG:boto:Data: 2023-07-29T00:18:21.295 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.295 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:18:21.296 DEBUG:boto:Port: 8001 2023-07-29T00:18:21.296 DEBUG:boto:Params: {} 2023-07-29T00:18:21.296 DEBUG:boto:Token: None 2023-07-29T00:18:21.297 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/a?versionId=null 2023-07-29T00:18:21.297 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1wqushcqPgHD3uM21vh3Cw42nFo= 2023-07-29T00:18:21.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1wqushcqPgHD3uM21vh3Cw42nFo=', 'Content-Length': '0'} 2023-07-29T00:18:21.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca460dda5159eefb-0064c45acd-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.300 DEBUG:boto:path=/hiiqos-56/a 2023-07-29T00:18:21.300 DEBUG:boto:auth_path=/hiiqos-56/a 2023-07-29T00:18:21.301 DEBUG:boto:path=/hiiqos-56/a?versionId=null 2023-07-29T00:18:21.301 DEBUG:boto:auth_path=/hiiqos-56/a?versionId=null 2023-07-29T00:18:21.301 DEBUG:boto:Method: HEAD 2023-07-29T00:18:21.302 DEBUG:boto:Path: /hiiqos-56/a?versionId=null 2023-07-29T00:18:21.302 DEBUG:boto:Data: 2023-07-29T00:18:21.302 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.303 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:21.303 DEBUG:boto:Port: 8000 2023-07-29T00:18:21.303 DEBUG:boto:Params: {} 2023-07-29T00:18:21.304 DEBUG:boto:Token: None 2023-07-29T00:18:21.304 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/a?versionId=null 2023-07-29T00:18:21.305 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1wqushcqPgHD3uM21vh3Cw42nFo= 2023-07-29T00:18:21.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1wqushcqPgHD3uM21vh3Cw42nFo=', 'Content-Length': '0'} 2023-07-29T00:18:21.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000164039c0cc1ecd43-0064c45acd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-07-29T00:18:21.308 DEBUG:boto:path=/hiiqos-56/a 2023-07-29T00:18:21.308 DEBUG:boto:auth_path=/hiiqos-56/a 2023-07-29T00:18:21.309 DEBUG:boto:path=/hiiqos-56/a?versionId=null 2023-07-29T00:18:21.309 DEBUG:boto:auth_path=/hiiqos-56/a?versionId=null 2023-07-29T00:18:21.309 DEBUG:boto:Method: GET 2023-07-29T00:18:21.310 DEBUG:boto:Path: /hiiqos-56/a?versionId=null 2023-07-29T00:18:21.310 DEBUG:boto:Data: 2023-07-29T00:18:21.310 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.311 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:18:21.311 DEBUG:boto:Port: 8001 2023-07-29T00:18:21.312 DEBUG:boto:Params: {} 2023-07-29T00:18:21.312 DEBUG:boto:Token: None 2023-07-29T00:18:21.312 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/a?versionId=null 2023-07-29T00:18:21.313 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZyXs6bVRD2cKb4+gIm8GSuOemDI= 2023-07-29T00:18:21.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZyXs6bVRD2cKb4+gIm8GSuOemDI=', 'Content-Length': '0'} 2023-07-29T00:18:21.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000024cac66e6628d22-0064c45acd-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.316 DEBUG:boto:path=/hiiqos-56/a 2023-07-29T00:18:21.316 DEBUG:boto:auth_path=/hiiqos-56/a 2023-07-29T00:18:21.316 DEBUG:boto:path=/hiiqos-56/a?versionId=null 2023-07-29T00:18:21.317 DEBUG:boto:auth_path=/hiiqos-56/a?versionId=null 2023-07-29T00:18:21.317 DEBUG:boto:Method: GET 2023-07-29T00:18:21.317 DEBUG:boto:Path: /hiiqos-56/a?versionId=null 2023-07-29T00:18:21.318 DEBUG:boto:Data: 2023-07-29T00:18:21.318 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.318 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:21.319 DEBUG:boto:Port: 8000 2023-07-29T00:18:21.319 DEBUG:boto:Params: {} 2023-07-29T00:18:21.320 DEBUG:boto:Token: None 2023-07-29T00:18:21.320 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/a?versionId=null 2023-07-29T00:18:21.320 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZyXs6bVRD2cKb4+gIm8GSuOemDI= 2023-07-29T00:18:21.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZyXs6bVRD2cKb4+gIm8GSuOemDI=', 'Content-Length': '0'} 2023-07-29T00:18:21.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b82b352458b1d426-0064c45acd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.324 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:18:21.324 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:18:21.324 DEBUG:boto:path=/hiiqos-56/?prefix=a 2023-07-29T00:18:21.325 DEBUG:boto:auth_path=/hiiqos-56/?prefix=a 2023-07-29T00:18:21.325 DEBUG:boto:Method: GET 2023-07-29T00:18:21.325 DEBUG:boto:Path: /hiiqos-56/?prefix=a 2023-07-29T00:18:21.326 DEBUG:boto:Data: 2023-07-29T00:18:21.326 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.327 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:18:21.327 DEBUG:boto:Port: 8001 2023-07-29T00:18:21.327 DEBUG:boto:Params: {} 2023-07-29T00:18:21.328 DEBUG:boto:Token: None 2023-07-29T00:18:21.328 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/ 2023-07-29T00:18:21.328 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FUNiQpHia71tPYJ7FWyG9lcF0mM= 2023-07-29T00:18:21.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FUNiQpHia71tPYJ7FWyG9lcF0mM=', 'Content-Length': '0'} 2023-07-29T00:18:21.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e10b2ba1c4a2e6f-0064c45acd-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.332 DEBUG:boto:b'hiiqos-56a1000falsea2023-07-29T00:18:00.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:18:21.333 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:18:21.333 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:18:21.333 DEBUG:boto:path=/hiiqos-56/?prefix=a 2023-07-29T00:18:21.334 DEBUG:boto:auth_path=/hiiqos-56/?prefix=a 2023-07-29T00:18:21.334 DEBUG:boto:Method: GET 2023-07-29T00:18:21.334 DEBUG:boto:Path: /hiiqos-56/?prefix=a 2023-07-29T00:18:21.335 DEBUG:boto:Data: 2023-07-29T00:18:21.335 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.335 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:21.336 DEBUG:boto:Port: 8000 2023-07-29T00:18:21.336 DEBUG:boto:Params: {} 2023-07-29T00:18:21.337 DEBUG:boto:Token: None 2023-07-29T00:18:21.337 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/ 2023-07-29T00:18:21.337 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FUNiQpHia71tPYJ7FWyG9lcF0mM= 2023-07-29T00:18:21.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FUNiQpHia71tPYJ7FWyG9lcF0mM=', 'Content-Length': '0'} 2023-07-29T00:18:21.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003eee9257c5da1949-0064c45acd-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.341 DEBUG:boto:b'hiiqos-56a1000falsea2023-07-29T00:18:00.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:18:21.341 DEBUG:boto:path=/hiiqos-56/a 2023-07-29T00:18:21.342 DEBUG:boto:auth_path=/hiiqos-56/a 2023-07-29T00:18:21.342 DEBUG:boto:Method: HEAD 2023-07-29T00:18:21.342 DEBUG:boto:Path: /hiiqos-56/a 2023-07-29T00:18:21.343 DEBUG:boto:Data: 2023-07-29T00:18:21.343 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.343 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:18:21.344 DEBUG:boto:Port: 8001 2023-07-29T00:18:21.344 DEBUG:boto:Params: {} 2023-07-29T00:18:21.344 DEBUG:boto:Token: None 2023-07-29T00:18:21.345 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/a 2023-07-29T00:18:21.345 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/QUtjYn76FBKihBQutnQfZu8bUk= 2023-07-29T00:18:21.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/QUtjYn76FBKihBQutnQfZu8bUk=', 'Content-Length': '0'} 2023-07-29T00:18:21.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d126adca4020430f-0064c45acd-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.348 DEBUG:boto:path=/hiiqos-56/a 2023-07-29T00:18:21.348 DEBUG:boto:auth_path=/hiiqos-56/a 2023-07-29T00:18:21.349 DEBUG:boto:Method: HEAD 2023-07-29T00:18:21.349 DEBUG:boto:Path: /hiiqos-56/a 2023-07-29T00:18:21.349 DEBUG:boto:Data: 2023-07-29T00:18:21.350 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.350 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:21.350 DEBUG:boto:Port: 8000 2023-07-29T00:18:21.351 DEBUG:boto:Params: {} 2023-07-29T00:18:21.351 DEBUG:boto:Token: None 2023-07-29T00:18:21.351 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/a 2023-07-29T00:18:21.352 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/QUtjYn76FBKihBQutnQfZu8bUk= 2023-07-29T00:18:21.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/QUtjYn76FBKihBQutnQfZu8bUk=', 'Content-Length': '0'} 2023-07-29T00:18:21.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001dc414c6714ff50f-0064c45acd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-07-29T00:18:21.355 DEBUG:boto:path=/hiiqos-56/a 2023-07-29T00:18:21.355 DEBUG:boto:auth_path=/hiiqos-56/a 2023-07-29T00:18:21.355 DEBUG:boto:Method: GET 2023-07-29T00:18:21.356 DEBUG:boto:Path: /hiiqos-56/a 2023-07-29T00:18:21.356 DEBUG:boto:Data: 2023-07-29T00:18:21.356 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.357 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:18:21.357 DEBUG:boto:Port: 8001 2023-07-29T00:18:21.358 DEBUG:boto:Params: {} 2023-07-29T00:18:21.358 DEBUG:boto:Token: None 2023-07-29T00:18:21.358 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/a 2023-07-29T00:18:21.359 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eQCCsIh+HnNreyd91BbusHiFyZM= 2023-07-29T00:18:21.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eQCCsIh+HnNreyd91BbusHiFyZM=', 'Content-Length': '0'} 2023-07-29T00:18:21.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000070031cf1f8b9baa-0064c45acd-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.361 DEBUG:boto:path=/hiiqos-56/a 2023-07-29T00:18:21.362 DEBUG:boto:auth_path=/hiiqos-56/a 2023-07-29T00:18:21.362 DEBUG:boto:Method: GET 2023-07-29T00:18:21.363 DEBUG:boto:Path: /hiiqos-56/a 2023-07-29T00:18:21.363 DEBUG:boto:Data: 2023-07-29T00:18:21.363 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.364 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:21.364 DEBUG:boto:Port: 8000 2023-07-29T00:18:21.364 DEBUG:boto:Params: {} 2023-07-29T00:18:21.365 DEBUG:boto:Token: None 2023-07-29T00:18:21.365 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/a 2023-07-29T00:18:21.366 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eQCCsIh+HnNreyd91BbusHiFyZM= 2023-07-29T00:18:21.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eQCCsIh+HnNreyd91BbusHiFyZM=', 'Content-Length': '0'} 2023-07-29T00:18:21.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1ea76bee88dda7d-0064c45acd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.369 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:18:21.369 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:18:21.369 DEBUG:boto:path=/hiiqos-56/?prefix=a 2023-07-29T00:18:21.370 DEBUG:boto:auth_path=/hiiqos-56/?prefix=a 2023-07-29T00:18:21.370 DEBUG:boto:Method: GET 2023-07-29T00:18:21.370 DEBUG:boto:Path: /hiiqos-56/?prefix=a 2023-07-29T00:18:21.371 DEBUG:boto:Data: 2023-07-29T00:18:21.371 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.371 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:18:21.372 DEBUG:boto:Port: 8001 2023-07-29T00:18:21.372 DEBUG:boto:Params: {} 2023-07-29T00:18:21.373 DEBUG:boto:Token: None 2023-07-29T00:18:21.373 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/ 2023-07-29T00:18:21.373 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FUNiQpHia71tPYJ7FWyG9lcF0mM= 2023-07-29T00:18:21.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FUNiQpHia71tPYJ7FWyG9lcF0mM=', 'Content-Length': '0'} 2023-07-29T00:18:21.376 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000880cdabce7a848ba-0064c45acd-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.420 DEBUG:boto:b'hiiqos-56a1000falsea2023-07-29T00:18:00.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:18:21.421 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:18:21.421 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:18:21.422 DEBUG:boto:path=/hiiqos-56/?prefix=a 2023-07-29T00:18:21.422 DEBUG:boto:auth_path=/hiiqos-56/?prefix=a 2023-07-29T00:18:21.423 DEBUG:boto:Method: GET 2023-07-29T00:18:21.423 DEBUG:boto:Path: /hiiqos-56/?prefix=a 2023-07-29T00:18:21.423 DEBUG:boto:Data: 2023-07-29T00:18:21.424 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.424 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:21.424 DEBUG:boto:Port: 8000 2023-07-29T00:18:21.425 DEBUG:boto:Params: {} 2023-07-29T00:18:21.425 DEBUG:boto:Token: None 2023-07-29T00:18:21.426 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/ 2023-07-29T00:18:21.426 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FUNiQpHia71tPYJ7FWyG9lcF0mM= 2023-07-29T00:18:21.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FUNiQpHia71tPYJ7FWyG9lcF0mM=', 'Content-Length': '0'} 2023-07-29T00:18:21.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095f8663cf6212a01-0064c45acd-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.429 DEBUG:boto:b'hiiqos-56a1000falsea2023-07-29T00:18:00.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:18:21.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-07-29T00:18:21.430 DEBUG:boto:path=/hiiqos-56/b 2023-07-29T00:18:21.431 DEBUG:boto:auth_path=/hiiqos-56/b 2023-07-29T00:18:21.431 DEBUG:boto:path=/hiiqos-56/b?versionId=null 2023-07-29T00:18:21.431 DEBUG:boto:auth_path=/hiiqos-56/b?versionId=null 2023-07-29T00:18:21.432 DEBUG:boto:Method: GET 2023-07-29T00:18:21.432 DEBUG:boto:Path: /hiiqos-56/b?versionId=null 2023-07-29T00:18:21.433 DEBUG:boto:Data: 2023-07-29T00:18:21.433 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.433 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:18:21.434 DEBUG:boto:Port: 8001 2023-07-29T00:18:21.434 DEBUG:boto:Params: {} 2023-07-29T00:18:21.434 DEBUG:boto:Token: None 2023-07-29T00:18:21.435 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/b?versionId=null 2023-07-29T00:18:21.435 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oBWTci84LruNM/kAmAdwjwgd4dQ= 2023-07-29T00:18:21.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oBWTci84LruNM/kAmAdwjwgd4dQ=', 'Content-Length': '0'} 2023-07-29T00:18:21.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000902b4bcfd454086-0064c45acd-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.438 DEBUG:boto:path=/hiiqos-56/b 2023-07-29T00:18:21.438 DEBUG:boto:auth_path=/hiiqos-56/b 2023-07-29T00:18:21.439 DEBUG:boto:path=/hiiqos-56/b?versionId=null 2023-07-29T00:18:21.439 DEBUG:boto:auth_path=/hiiqos-56/b?versionId=null 2023-07-29T00:18:21.439 DEBUG:boto:Method: GET 2023-07-29T00:18:21.440 DEBUG:boto:Path: /hiiqos-56/b?versionId=null 2023-07-29T00:18:21.440 DEBUG:boto:Data: 2023-07-29T00:18:21.441 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.441 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:21.441 DEBUG:boto:Port: 8000 2023-07-29T00:18:21.442 DEBUG:boto:Params: {} 2023-07-29T00:18:21.442 DEBUG:boto:Token: None 2023-07-29T00:18:21.442 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/b?versionId=null 2023-07-29T00:18:21.443 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oBWTci84LruNM/kAmAdwjwgd4dQ= 2023-07-29T00:18:21.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oBWTci84LruNM/kAmAdwjwgd4dQ=', 'Content-Length': '0'} 2023-07-29T00:18:21.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b826b1dd88368c9b-0064c45acd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.488 DEBUG:boto:path=/hiiqos-56/b 2023-07-29T00:18:21.489 DEBUG:boto:auth_path=/hiiqos-56/b 2023-07-29T00:18:21.489 DEBUG:boto:path=/hiiqos-56/b?versionId=null 2023-07-29T00:18:21.489 DEBUG:boto:auth_path=/hiiqos-56/b?versionId=null 2023-07-29T00:18:21.490 DEBUG:boto:Method: HEAD 2023-07-29T00:18:21.490 DEBUG:boto:Path: /hiiqos-56/b?versionId=null 2023-07-29T00:18:21.490 DEBUG:boto:Data: 2023-07-29T00:18:21.491 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.491 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:18:21.491 DEBUG:boto:Port: 8001 2023-07-29T00:18:21.492 DEBUG:boto:Params: {} 2023-07-29T00:18:21.492 DEBUG:boto:Token: None 2023-07-29T00:18:21.492 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/b?versionId=null 2023-07-29T00:18:21.493 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SfDsMMRy1ZDt4GrnYl+M0nOcKkc= 2023-07-29T00:18:21.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SfDsMMRy1ZDt4GrnYl+M0nOcKkc=', 'Content-Length': '0'} 2023-07-29T00:18:21.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000116a3c366586221-0064c45acd-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.496 DEBUG:boto:path=/hiiqos-56/b 2023-07-29T00:18:21.496 DEBUG:boto:auth_path=/hiiqos-56/b 2023-07-29T00:18:21.497 DEBUG:boto:path=/hiiqos-56/b?versionId=null 2023-07-29T00:18:21.497 DEBUG:boto:auth_path=/hiiqos-56/b?versionId=null 2023-07-29T00:18:21.497 DEBUG:boto:Method: HEAD 2023-07-29T00:18:21.498 DEBUG:boto:Path: /hiiqos-56/b?versionId=null 2023-07-29T00:18:21.498 DEBUG:boto:Data: 2023-07-29T00:18:21.498 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.499 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:21.499 DEBUG:boto:Port: 8000 2023-07-29T00:18:21.499 DEBUG:boto:Params: {} 2023-07-29T00:18:21.500 DEBUG:boto:Token: None 2023-07-29T00:18:21.500 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/b?versionId=null 2023-07-29T00:18:21.500 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SfDsMMRy1ZDt4GrnYl+M0nOcKkc= 2023-07-29T00:18:21.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SfDsMMRy1ZDt4GrnYl+M0nOcKkc=', 'Content-Length': '0'} 2023-07-29T00:18:21.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7e8d067abf4f46e-0064c45acd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-07-29T00:18:21.504 DEBUG:boto:path=/hiiqos-56/b 2023-07-29T00:18:21.504 DEBUG:boto:auth_path=/hiiqos-56/b 2023-07-29T00:18:21.505 DEBUG:boto:path=/hiiqos-56/b?versionId=null 2023-07-29T00:18:21.505 DEBUG:boto:auth_path=/hiiqos-56/b?versionId=null 2023-07-29T00:18:21.505 DEBUG:boto:Method: GET 2023-07-29T00:18:21.506 DEBUG:boto:Path: /hiiqos-56/b?versionId=null 2023-07-29T00:18:21.506 DEBUG:boto:Data: 2023-07-29T00:18:21.506 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.507 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:18:21.507 DEBUG:boto:Port: 8001 2023-07-29T00:18:21.507 DEBUG:boto:Params: {} 2023-07-29T00:18:21.508 DEBUG:boto:Token: None 2023-07-29T00:18:21.508 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/b?versionId=null 2023-07-29T00:18:21.509 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oBWTci84LruNM/kAmAdwjwgd4dQ= 2023-07-29T00:18:21.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oBWTci84LruNM/kAmAdwjwgd4dQ=', 'Content-Length': '0'} 2023-07-29T00:18:21.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e04a7494ef3baa8-0064c45acd-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.511 DEBUG:boto:path=/hiiqos-56/b 2023-07-29T00:18:21.512 DEBUG:boto:auth_path=/hiiqos-56/b 2023-07-29T00:18:21.512 DEBUG:boto:path=/hiiqos-56/b?versionId=null 2023-07-29T00:18:21.513 DEBUG:boto:auth_path=/hiiqos-56/b?versionId=null 2023-07-29T00:18:21.513 DEBUG:boto:Method: GET 2023-07-29T00:18:21.513 DEBUG:boto:Path: /hiiqos-56/b?versionId=null 2023-07-29T00:18:21.514 DEBUG:boto:Data: 2023-07-29T00:18:21.514 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.514 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:21.515 DEBUG:boto:Port: 8000 2023-07-29T00:18:21.515 DEBUG:boto:Params: {} 2023-07-29T00:18:21.515 DEBUG:boto:Token: None 2023-07-29T00:18:21.516 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/b?versionId=null 2023-07-29T00:18:21.516 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oBWTci84LruNM/kAmAdwjwgd4dQ= 2023-07-29T00:18:21.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oBWTci84LruNM/kAmAdwjwgd4dQ=', 'Content-Length': '0'} 2023-07-29T00:18:21.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6ec1425ae900dce-0064c45acd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.519 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:18:21.519 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:18:21.520 DEBUG:boto:path=/hiiqos-56/?prefix=b 2023-07-29T00:18:21.520 DEBUG:boto:auth_path=/hiiqos-56/?prefix=b 2023-07-29T00:18:21.520 DEBUG:boto:Method: GET 2023-07-29T00:18:21.521 DEBUG:boto:Path: /hiiqos-56/?prefix=b 2023-07-29T00:18:21.521 DEBUG:boto:Data: 2023-07-29T00:18:21.522 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.522 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:18:21.522 DEBUG:boto:Port: 8001 2023-07-29T00:18:21.523 DEBUG:boto:Params: {} 2023-07-29T00:18:21.523 DEBUG:boto:Token: None 2023-07-29T00:18:21.523 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/ 2023-07-29T00:18:21.524 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FUNiQpHia71tPYJ7FWyG9lcF0mM= 2023-07-29T00:18:21.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FUNiQpHia71tPYJ7FWyG9lcF0mM=', 'Content-Length': '0'} 2023-07-29T00:18:21.527 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000007dbe587cc4285-0064c45acd-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.527 DEBUG:boto:b'hiiqos-56b1000falseb2023-07-29T00:18:00.841Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:18:21.528 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:18:21.528 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:18:21.529 DEBUG:boto:path=/hiiqos-56/?prefix=b 2023-07-29T00:18:21.529 DEBUG:boto:auth_path=/hiiqos-56/?prefix=b 2023-07-29T00:18:21.529 DEBUG:boto:Method: GET 2023-07-29T00:18:21.530 DEBUG:boto:Path: /hiiqos-56/?prefix=b 2023-07-29T00:18:21.530 DEBUG:boto:Data: 2023-07-29T00:18:21.530 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.531 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:21.531 DEBUG:boto:Port: 8000 2023-07-29T00:18:21.531 DEBUG:boto:Params: {} 2023-07-29T00:18:21.532 DEBUG:boto:Token: None 2023-07-29T00:18:21.532 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/ 2023-07-29T00:18:21.533 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FUNiQpHia71tPYJ7FWyG9lcF0mM= 2023-07-29T00:18:21.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FUNiQpHia71tPYJ7FWyG9lcF0mM=', 'Content-Length': '0'} 2023-07-29T00:18:21.536 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acf37769147dfdf1-0064c45acd-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.536 DEBUG:boto:b'hiiqos-56b1000falseb2023-07-29T00:18:00.841Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:18:21.537 DEBUG:boto:path=/hiiqos-56/b 2023-07-29T00:18:21.537 DEBUG:boto:auth_path=/hiiqos-56/b 2023-07-29T00:18:21.538 DEBUG:boto:Method: HEAD 2023-07-29T00:18:21.538 DEBUG:boto:Path: /hiiqos-56/b 2023-07-29T00:18:21.538 DEBUG:boto:Data: 2023-07-29T00:18:21.539 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.539 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:18:21.539 DEBUG:boto:Port: 8001 2023-07-29T00:18:21.540 DEBUG:boto:Params: {} 2023-07-29T00:18:21.540 DEBUG:boto:Token: None 2023-07-29T00:18:21.540 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/b 2023-07-29T00:18:21.541 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZIQaKuQDcYOERPnMZ36uDhBSpp8= 2023-07-29T00:18:21.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZIQaKuQDcYOERPnMZ36uDhBSpp8=', 'Content-Length': '0'} 2023-07-29T00:18:21.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcbae2ad30071d6e-0064c45acd-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.544 DEBUG:boto:path=/hiiqos-56/b 2023-07-29T00:18:21.544 DEBUG:boto:auth_path=/hiiqos-56/b 2023-07-29T00:18:21.544 DEBUG:boto:Method: HEAD 2023-07-29T00:18:21.545 DEBUG:boto:Path: /hiiqos-56/b 2023-07-29T00:18:21.545 DEBUG:boto:Data: 2023-07-29T00:18:21.546 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.546 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:21.546 DEBUG:boto:Port: 8000 2023-07-29T00:18:21.546 DEBUG:boto:Params: {} 2023-07-29T00:18:21.547 DEBUG:boto:Token: None 2023-07-29T00:18:21.547 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/b 2023-07-29T00:18:21.548 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZIQaKuQDcYOERPnMZ36uDhBSpp8= 2023-07-29T00:18:21.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZIQaKuQDcYOERPnMZ36uDhBSpp8=', 'Content-Length': '0'} 2023-07-29T00:18:21.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af37fef3cda94f73-0064c45acd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-07-29T00:18:21.551 DEBUG:boto:path=/hiiqos-56/b 2023-07-29T00:18:21.551 DEBUG:boto:auth_path=/hiiqos-56/b 2023-07-29T00:18:21.551 DEBUG:boto:Method: GET 2023-07-29T00:18:21.552 DEBUG:boto:Path: /hiiqos-56/b 2023-07-29T00:18:21.552 DEBUG:boto:Data: 2023-07-29T00:18:21.552 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.553 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:18:21.553 DEBUG:boto:Port: 8001 2023-07-29T00:18:21.553 DEBUG:boto:Params: {} 2023-07-29T00:18:21.554 DEBUG:boto:Token: None 2023-07-29T00:18:21.554 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/b 2023-07-29T00:18:21.555 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qMjbhO1LriqbWqXDQf4+/yOeZis= 2023-07-29T00:18:21.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qMjbhO1LriqbWqXDQf4+/yOeZis=', 'Content-Length': '0'} 2023-07-29T00:18:21.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062f7fa5900cb967c-0064c45acd-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.557 DEBUG:boto:path=/hiiqos-56/b 2023-07-29T00:18:21.558 DEBUG:boto:auth_path=/hiiqos-56/b 2023-07-29T00:18:21.558 DEBUG:boto:Method: GET 2023-07-29T00:18:21.559 DEBUG:boto:Path: /hiiqos-56/b 2023-07-29T00:18:21.559 DEBUG:boto:Data: 2023-07-29T00:18:21.559 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.560 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:21.560 DEBUG:boto:Port: 8000 2023-07-29T00:18:21.560 DEBUG:boto:Params: {} 2023-07-29T00:18:21.561 DEBUG:boto:Token: None 2023-07-29T00:18:21.561 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/b 2023-07-29T00:18:21.562 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qMjbhO1LriqbWqXDQf4+/yOeZis= 2023-07-29T00:18:21.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qMjbhO1LriqbWqXDQf4+/yOeZis=', 'Content-Length': '0'} 2023-07-29T00:18:21.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c9b8d677e548e10-0064c45acd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.564 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:18:21.565 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:18:21.565 DEBUG:boto:path=/hiiqos-56/?prefix=b 2023-07-29T00:18:21.566 DEBUG:boto:auth_path=/hiiqos-56/?prefix=b 2023-07-29T00:18:21.566 DEBUG:boto:Method: GET 2023-07-29T00:18:21.566 DEBUG:boto:Path: /hiiqos-56/?prefix=b 2023-07-29T00:18:21.567 DEBUG:boto:Data: 2023-07-29T00:18:21.567 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.567 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:18:21.568 DEBUG:boto:Port: 8001 2023-07-29T00:18:21.568 DEBUG:boto:Params: {} 2023-07-29T00:18:21.569 DEBUG:boto:Token: None 2023-07-29T00:18:21.569 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/ 2023-07-29T00:18:21.569 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FUNiQpHia71tPYJ7FWyG9lcF0mM= 2023-07-29T00:18:21.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FUNiQpHia71tPYJ7FWyG9lcF0mM=', 'Content-Length': '0'} 2023-07-29T00:18:21.572 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067e19b82f9ccafaa-0064c45acd-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.573 DEBUG:boto:b'hiiqos-56b1000falseb2023-07-29T00:18:00.841Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:18:21.574 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:18:21.574 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:18:21.574 DEBUG:boto:path=/hiiqos-56/?prefix=b 2023-07-29T00:18:21.575 DEBUG:boto:auth_path=/hiiqos-56/?prefix=b 2023-07-29T00:18:21.575 DEBUG:boto:Method: GET 2023-07-29T00:18:21.575 DEBUG:boto:Path: /hiiqos-56/?prefix=b 2023-07-29T00:18:21.576 DEBUG:boto:Data: 2023-07-29T00:18:21.576 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.576 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:21.577 DEBUG:boto:Port: 8000 2023-07-29T00:18:21.577 DEBUG:boto:Params: {} 2023-07-29T00:18:21.578 DEBUG:boto:Token: None 2023-07-29T00:18:21.578 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/ 2023-07-29T00:18:21.578 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FUNiQpHia71tPYJ7FWyG9lcF0mM= 2023-07-29T00:18:21.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FUNiQpHia71tPYJ7FWyG9lcF0mM=', 'Content-Length': '0'} 2023-07-29T00:18:21.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020fb40161a6c6d7c-0064c45acd-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.582 DEBUG:boto:b'hiiqos-56b1000falseb2023-07-29T00:18:00.841Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:18:21.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-07-29T00:18:21.583 DEBUG:boto:path=/hiiqos-56/c 2023-07-29T00:18:21.583 DEBUG:boto:auth_path=/hiiqos-56/c 2023-07-29T00:18:21.584 DEBUG:boto:path=/hiiqos-56/c?versionId=null 2023-07-29T00:18:21.584 DEBUG:boto:auth_path=/hiiqos-56/c?versionId=null 2023-07-29T00:18:21.584 DEBUG:boto:Method: GET 2023-07-29T00:18:21.585 DEBUG:boto:Path: /hiiqos-56/c?versionId=null 2023-07-29T00:18:21.585 DEBUG:boto:Data: 2023-07-29T00:18:21.585 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.586 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:18:21.586 DEBUG:boto:Port: 8001 2023-07-29T00:18:21.586 DEBUG:boto:Params: {} 2023-07-29T00:18:21.587 DEBUG:boto:Token: None 2023-07-29T00:18:21.587 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/c?versionId=null 2023-07-29T00:18:21.588 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+W0+zSgVaqjsWuQbJJxWhoM06HI= 2023-07-29T00:18:21.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+W0+zSgVaqjsWuQbJJxWhoM06HI=', 'Content-Length': '0'} 2023-07-29T00:18:21.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdfb663e90009b46-0064c45acd-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.632 DEBUG:boto:path=/hiiqos-56/c 2023-07-29T00:18:21.633 DEBUG:boto:auth_path=/hiiqos-56/c 2023-07-29T00:18:21.633 DEBUG:boto:path=/hiiqos-56/c?versionId=null 2023-07-29T00:18:21.633 DEBUG:boto:auth_path=/hiiqos-56/c?versionId=null 2023-07-29T00:18:21.634 DEBUG:boto:Method: GET 2023-07-29T00:18:21.634 DEBUG:boto:Path: /hiiqos-56/c?versionId=null 2023-07-29T00:18:21.635 DEBUG:boto:Data: 2023-07-29T00:18:21.635 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.635 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:21.636 DEBUG:boto:Port: 8000 2023-07-29T00:18:21.636 DEBUG:boto:Params: {} 2023-07-29T00:18:21.636 DEBUG:boto:Token: None 2023-07-29T00:18:21.637 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/c?versionId=null 2023-07-29T00:18:21.637 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+W0+zSgVaqjsWuQbJJxWhoM06HI= 2023-07-29T00:18:21.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+W0+zSgVaqjsWuQbJJxWhoM06HI=', 'Content-Length': '0'} 2023-07-29T00:18:21.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030612bbab577c854-0064c45acd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.640 DEBUG:boto:path=/hiiqos-56/c 2023-07-29T00:18:21.641 DEBUG:boto:auth_path=/hiiqos-56/c 2023-07-29T00:18:21.641 DEBUG:boto:path=/hiiqos-56/c?versionId=null 2023-07-29T00:18:21.641 DEBUG:boto:auth_path=/hiiqos-56/c?versionId=null 2023-07-29T00:18:21.642 DEBUG:boto:Method: HEAD 2023-07-29T00:18:21.642 DEBUG:boto:Path: /hiiqos-56/c?versionId=null 2023-07-29T00:18:21.642 DEBUG:boto:Data: 2023-07-29T00:18:21.643 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.643 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:18:21.643 DEBUG:boto:Port: 8001 2023-07-29T00:18:21.644 DEBUG:boto:Params: {} 2023-07-29T00:18:21.644 DEBUG:boto:Token: None 2023-07-29T00:18:21.644 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/c?versionId=null 2023-07-29T00:18:21.645 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I0fmexU1hSzpZjQny9DHCwdqwkQ= 2023-07-29T00:18:21.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I0fmexU1hSzpZjQny9DHCwdqwkQ=', 'Content-Length': '0'} 2023-07-29T00:18:21.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c6bec713132352d-0064c45acd-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.648 DEBUG:boto:path=/hiiqos-56/c 2023-07-29T00:18:21.648 DEBUG:boto:auth_path=/hiiqos-56/c 2023-07-29T00:18:21.648 DEBUG:boto:path=/hiiqos-56/c?versionId=null 2023-07-29T00:18:21.649 DEBUG:boto:auth_path=/hiiqos-56/c?versionId=null 2023-07-29T00:18:21.649 DEBUG:boto:Method: HEAD 2023-07-29T00:18:21.649 DEBUG:boto:Path: /hiiqos-56/c?versionId=null 2023-07-29T00:18:21.650 DEBUG:boto:Data: 2023-07-29T00:18:21.650 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.651 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:21.651 DEBUG:boto:Port: 8000 2023-07-29T00:18:21.651 DEBUG:boto:Params: {} 2023-07-29T00:18:21.652 DEBUG:boto:Token: None 2023-07-29T00:18:21.652 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/c?versionId=null 2023-07-29T00:18:21.652 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I0fmexU1hSzpZjQny9DHCwdqwkQ= 2023-07-29T00:18:21.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I0fmexU1hSzpZjQny9DHCwdqwkQ=', 'Content-Length': '0'} 2023-07-29T00:18:21.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc5eca49b4edc9ec-0064c45acd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-07-29T00:18:21.656 DEBUG:boto:path=/hiiqos-56/c 2023-07-29T00:18:21.656 DEBUG:boto:auth_path=/hiiqos-56/c 2023-07-29T00:18:21.656 DEBUG:boto:path=/hiiqos-56/c?versionId=null 2023-07-29T00:18:21.657 DEBUG:boto:auth_path=/hiiqos-56/c?versionId=null 2023-07-29T00:18:21.657 DEBUG:boto:Method: GET 2023-07-29T00:18:21.657 DEBUG:boto:Path: /hiiqos-56/c?versionId=null 2023-07-29T00:18:21.658 DEBUG:boto:Data: 2023-07-29T00:18:21.658 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.658 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:18:21.659 DEBUG:boto:Port: 8001 2023-07-29T00:18:21.659 DEBUG:boto:Params: {} 2023-07-29T00:18:21.659 DEBUG:boto:Token: None 2023-07-29T00:18:21.660 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/c?versionId=null 2023-07-29T00:18:21.660 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+W0+zSgVaqjsWuQbJJxWhoM06HI= 2023-07-29T00:18:21.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+W0+zSgVaqjsWuQbJJxWhoM06HI=', 'Content-Length': '0'} 2023-07-29T00:18:21.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000125c10ab6b24e88e-0064c45acd-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.663 DEBUG:boto:path=/hiiqos-56/c 2023-07-29T00:18:21.664 DEBUG:boto:auth_path=/hiiqos-56/c 2023-07-29T00:18:21.664 DEBUG:boto:path=/hiiqos-56/c?versionId=null 2023-07-29T00:18:21.665 DEBUG:boto:auth_path=/hiiqos-56/c?versionId=null 2023-07-29T00:18:21.665 DEBUG:boto:Method: GET 2023-07-29T00:18:21.665 DEBUG:boto:Path: /hiiqos-56/c?versionId=null 2023-07-29T00:18:21.666 DEBUG:boto:Data: 2023-07-29T00:18:21.666 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.666 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:21.667 DEBUG:boto:Port: 8000 2023-07-29T00:18:21.667 DEBUG:boto:Params: {} 2023-07-29T00:18:21.668 DEBUG:boto:Token: None 2023-07-29T00:18:21.668 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/c?versionId=null 2023-07-29T00:18:21.668 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+W0+zSgVaqjsWuQbJJxWhoM06HI= 2023-07-29T00:18:21.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+W0+zSgVaqjsWuQbJJxWhoM06HI=', 'Content-Length': '0'} 2023-07-29T00:18:21.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7f0713d55765034-0064c45acd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.671 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:18:21.672 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:18:21.672 DEBUG:boto:path=/hiiqos-56/?prefix=c 2023-07-29T00:18:21.672 DEBUG:boto:auth_path=/hiiqos-56/?prefix=c 2023-07-29T00:18:21.673 DEBUG:boto:Method: GET 2023-07-29T00:18:21.673 DEBUG:boto:Path: /hiiqos-56/?prefix=c 2023-07-29T00:18:21.673 DEBUG:boto:Data: 2023-07-29T00:18:21.674 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.674 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:18:21.674 DEBUG:boto:Port: 8001 2023-07-29T00:18:21.675 DEBUG:boto:Params: {} 2023-07-29T00:18:21.675 DEBUG:boto:Token: None 2023-07-29T00:18:21.676 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/ 2023-07-29T00:18:21.676 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FUNiQpHia71tPYJ7FWyG9lcF0mM= 2023-07-29T00:18:21.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FUNiQpHia71tPYJ7FWyG9lcF0mM=', 'Content-Length': '0'} 2023-07-29T00:18:21.679 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d2d4cbe9d6058fa-0064c45acd-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.679 DEBUG:boto:b'hiiqos-56c1000falsec2023-07-29T00:18:00.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:18:21.680 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:18:21.680 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:18:21.681 DEBUG:boto:path=/hiiqos-56/?prefix=c 2023-07-29T00:18:21.681 DEBUG:boto:auth_path=/hiiqos-56/?prefix=c 2023-07-29T00:18:21.681 DEBUG:boto:Method: GET 2023-07-29T00:18:21.682 DEBUG:boto:Path: /hiiqos-56/?prefix=c 2023-07-29T00:18:21.682 DEBUG:boto:Data: 2023-07-29T00:18:21.682 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.683 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:21.683 DEBUG:boto:Port: 8000 2023-07-29T00:18:21.683 DEBUG:boto:Params: {} 2023-07-29T00:18:21.684 DEBUG:boto:Token: None 2023-07-29T00:18:21.684 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/ 2023-07-29T00:18:21.685 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FUNiQpHia71tPYJ7FWyG9lcF0mM= 2023-07-29T00:18:21.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FUNiQpHia71tPYJ7FWyG9lcF0mM=', 'Content-Length': '0'} 2023-07-29T00:18:21.687 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000625160e2901b9610-0064c45acd-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.688 DEBUG:boto:b'hiiqos-56c1000falsec2023-07-29T00:18:00.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:18:21.688 DEBUG:boto:path=/hiiqos-56/c 2023-07-29T00:18:21.689 DEBUG:boto:auth_path=/hiiqos-56/c 2023-07-29T00:18:21.689 DEBUG:boto:Method: HEAD 2023-07-29T00:18:21.689 DEBUG:boto:Path: /hiiqos-56/c 2023-07-29T00:18:21.690 DEBUG:boto:Data: 2023-07-29T00:18:21.690 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.690 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:18:21.691 DEBUG:boto:Port: 8001 2023-07-29T00:18:21.691 DEBUG:boto:Params: {} 2023-07-29T00:18:21.691 DEBUG:boto:Token: None 2023-07-29T00:18:21.692 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/c 2023-07-29T00:18:21.692 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/zaP5NEl97K/LjVTEVYEaNEVLKs= 2023-07-29T00:18:21.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/zaP5NEl97K/LjVTEVYEaNEVLKs=', 'Content-Length': '0'} 2023-07-29T00:18:21.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1d4b837c1c4abc9-0064c45acd-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.695 DEBUG:boto:path=/hiiqos-56/c 2023-07-29T00:18:21.695 DEBUG:boto:auth_path=/hiiqos-56/c 2023-07-29T00:18:21.695 DEBUG:boto:Method: HEAD 2023-07-29T00:18:21.696 DEBUG:boto:Path: /hiiqos-56/c 2023-07-29T00:18:21.696 DEBUG:boto:Data: 2023-07-29T00:18:21.696 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.697 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:21.697 DEBUG:boto:Port: 8000 2023-07-29T00:18:21.697 DEBUG:boto:Params: {} 2023-07-29T00:18:21.698 DEBUG:boto:Token: None 2023-07-29T00:18:21.698 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/c 2023-07-29T00:18:21.699 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/zaP5NEl97K/LjVTEVYEaNEVLKs= 2023-07-29T00:18:21.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/zaP5NEl97K/LjVTEVYEaNEVLKs=', 'Content-Length': '0'} 2023-07-29T00:18:21.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052824bf6868a73fb-0064c45acd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-07-29T00:18:21.701 DEBUG:boto:path=/hiiqos-56/c 2023-07-29T00:18:21.702 DEBUG:boto:auth_path=/hiiqos-56/c 2023-07-29T00:18:21.702 DEBUG:boto:Method: GET 2023-07-29T00:18:21.703 DEBUG:boto:Path: /hiiqos-56/c 2023-07-29T00:18:21.703 DEBUG:boto:Data: 2023-07-29T00:18:21.703 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.704 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:18:21.704 DEBUG:boto:Port: 8001 2023-07-29T00:18:21.704 DEBUG:boto:Params: {} 2023-07-29T00:18:21.705 DEBUG:boto:Token: None 2023-07-29T00:18:21.705 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/c 2023-07-29T00:18:21.706 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6jxmEpYNhszBnAJjYtuyio3cgio= 2023-07-29T00:18:21.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6jxmEpYNhszBnAJjYtuyio3cgio=', 'Content-Length': '0'} 2023-07-29T00:18:21.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000feb9b847aafff43b-0064c45acd-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.708 DEBUG:boto:path=/hiiqos-56/c 2023-07-29T00:18:21.709 DEBUG:boto:auth_path=/hiiqos-56/c 2023-07-29T00:18:21.709 DEBUG:boto:Method: GET 2023-07-29T00:18:21.710 DEBUG:boto:Path: /hiiqos-56/c 2023-07-29T00:18:21.710 DEBUG:boto:Data: 2023-07-29T00:18:21.710 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.711 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:21.711 DEBUG:boto:Port: 8000 2023-07-29T00:18:21.711 DEBUG:boto:Params: {} 2023-07-29T00:18:21.712 DEBUG:boto:Token: None 2023-07-29T00:18:21.712 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/c 2023-07-29T00:18:21.713 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6jxmEpYNhszBnAJjYtuyio3cgio= 2023-07-29T00:18:21.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6jxmEpYNhszBnAJjYtuyio3cgio=', 'Content-Length': '0'} 2023-07-29T00:18:21.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e2064de68fbb66d-0064c45acd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.715 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:18:21.716 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:18:21.716 DEBUG:boto:path=/hiiqos-56/?prefix=c 2023-07-29T00:18:21.716 DEBUG:boto:auth_path=/hiiqos-56/?prefix=c 2023-07-29T00:18:21.717 DEBUG:boto:Method: GET 2023-07-29T00:18:21.717 DEBUG:boto:Path: /hiiqos-56/?prefix=c 2023-07-29T00:18:21.717 DEBUG:boto:Data: 2023-07-29T00:18:21.718 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.718 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:18:21.718 DEBUG:boto:Port: 8001 2023-07-29T00:18:21.719 DEBUG:boto:Params: {} 2023-07-29T00:18:21.719 DEBUG:boto:Token: None 2023-07-29T00:18:21.720 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/ 2023-07-29T00:18:21.720 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FUNiQpHia71tPYJ7FWyG9lcF0mM= 2023-07-29T00:18:21.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FUNiQpHia71tPYJ7FWyG9lcF0mM=', 'Content-Length': '0'} 2023-07-29T00:18:21.723 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000977a9e3b086c54f9-0064c45acd-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.723 DEBUG:boto:b'hiiqos-56c1000falsec2023-07-29T00:18:00.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:18:21.724 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:18:21.724 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:18:21.725 DEBUG:boto:path=/hiiqos-56/?prefix=c 2023-07-29T00:18:21.725 DEBUG:boto:auth_path=/hiiqos-56/?prefix=c 2023-07-29T00:18:21.725 DEBUG:boto:Method: GET 2023-07-29T00:18:21.726 DEBUG:boto:Path: /hiiqos-56/?prefix=c 2023-07-29T00:18:21.726 DEBUG:boto:Data: 2023-07-29T00:18:21.726 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.727 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:21.727 DEBUG:boto:Port: 8000 2023-07-29T00:18:21.727 DEBUG:boto:Params: {} 2023-07-29T00:18:21.728 DEBUG:boto:Token: None 2023-07-29T00:18:21.728 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/ 2023-07-29T00:18:21.729 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FUNiQpHia71tPYJ7FWyG9lcF0mM= 2023-07-29T00:18:21.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FUNiQpHia71tPYJ7FWyG9lcF0mM=', 'Content-Length': '0'} 2023-07-29T00:18:21.732 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c5639b6b0776dbe-0064c45acd-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.732 DEBUG:boto:b'hiiqos-56c1000falsec2023-07-29T00:18:00.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:18:21.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-07-29T00:18:21.733 DEBUG:boto:path=/hiiqos-56/d 2023-07-29T00:18:21.734 DEBUG:boto:auth_path=/hiiqos-56/d 2023-07-29T00:18:21.734 DEBUG:boto:path=/hiiqos-56/d?versionId=null 2023-07-29T00:18:21.734 DEBUG:boto:auth_path=/hiiqos-56/d?versionId=null 2023-07-29T00:18:21.735 DEBUG:boto:Method: GET 2023-07-29T00:18:21.735 DEBUG:boto:Path: /hiiqos-56/d?versionId=null 2023-07-29T00:18:21.735 DEBUG:boto:Data: 2023-07-29T00:18:21.736 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.736 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:18:21.736 DEBUG:boto:Port: 8001 2023-07-29T00:18:21.737 DEBUG:boto:Params: {} 2023-07-29T00:18:21.737 DEBUG:boto:Token: None 2023-07-29T00:18:21.738 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/d?versionId=null 2023-07-29T00:18:21.738 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J4seB4ppF3Dh+bV7ryXPSKRNQHg= 2023-07-29T00:18:21.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J4seB4ppF3Dh+bV7ryXPSKRNQHg=', 'Content-Length': '0'} 2023-07-29T00:18:21.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e107ccb70a068adb-0064c45acd-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.741 DEBUG:boto:path=/hiiqos-56/d 2023-07-29T00:18:21.741 DEBUG:boto:auth_path=/hiiqos-56/d 2023-07-29T00:18:21.741 DEBUG:boto:path=/hiiqos-56/d?versionId=null 2023-07-29T00:18:21.742 DEBUG:boto:auth_path=/hiiqos-56/d?versionId=null 2023-07-29T00:18:21.742 DEBUG:boto:Method: GET 2023-07-29T00:18:21.742 DEBUG:boto:Path: /hiiqos-56/d?versionId=null 2023-07-29T00:18:21.743 DEBUG:boto:Data: 2023-07-29T00:18:21.743 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.743 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:21.744 DEBUG:boto:Port: 8000 2023-07-29T00:18:21.744 DEBUG:boto:Params: {} 2023-07-29T00:18:21.745 DEBUG:boto:Token: None 2023-07-29T00:18:21.745 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/d?versionId=null 2023-07-29T00:18:21.745 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J4seB4ppF3Dh+bV7ryXPSKRNQHg= 2023-07-29T00:18:21.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J4seB4ppF3Dh+bV7ryXPSKRNQHg=', 'Content-Length': '0'} 2023-07-29T00:18:21.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7d31d050ae3b25f-0064c45acd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.748 DEBUG:boto:path=/hiiqos-56/d 2023-07-29T00:18:21.748 DEBUG:boto:auth_path=/hiiqos-56/d 2023-07-29T00:18:21.749 DEBUG:boto:path=/hiiqos-56/d?versionId=null 2023-07-29T00:18:21.749 DEBUG:boto:auth_path=/hiiqos-56/d?versionId=null 2023-07-29T00:18:21.749 DEBUG:boto:Method: HEAD 2023-07-29T00:18:21.750 DEBUG:boto:Path: /hiiqos-56/d?versionId=null 2023-07-29T00:18:21.750 DEBUG:boto:Data: 2023-07-29T00:18:21.750 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.751 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:18:21.751 DEBUG:boto:Port: 8001 2023-07-29T00:18:21.751 DEBUG:boto:Params: {} 2023-07-29T00:18:21.752 DEBUG:boto:Token: None 2023-07-29T00:18:21.752 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/d?versionId=null 2023-07-29T00:18:21.753 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kapxnCv2vFl4ZV3vecEE7suO10c= 2023-07-29T00:18:21.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kapxnCv2vFl4ZV3vecEE7suO10c=', 'Content-Length': '0'} 2023-07-29T00:18:21.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c74db0505db3d48-0064c45acd-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.755 DEBUG:boto:path=/hiiqos-56/d 2023-07-29T00:18:21.756 DEBUG:boto:auth_path=/hiiqos-56/d 2023-07-29T00:18:21.756 DEBUG:boto:path=/hiiqos-56/d?versionId=null 2023-07-29T00:18:21.756 DEBUG:boto:auth_path=/hiiqos-56/d?versionId=null 2023-07-29T00:18:21.757 DEBUG:boto:Method: HEAD 2023-07-29T00:18:21.757 DEBUG:boto:Path: /hiiqos-56/d?versionId=null 2023-07-29T00:18:21.757 DEBUG:boto:Data: 2023-07-29T00:18:21.758 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.758 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:21.758 DEBUG:boto:Port: 8000 2023-07-29T00:18:21.759 DEBUG:boto:Params: {} 2023-07-29T00:18:21.759 DEBUG:boto:Token: None 2023-07-29T00:18:21.759 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/d?versionId=null 2023-07-29T00:18:21.760 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kapxnCv2vFl4ZV3vecEE7suO10c= 2023-07-29T00:18:21.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kapxnCv2vFl4ZV3vecEE7suO10c=', 'Content-Length': '0'} 2023-07-29T00:18:21.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc907ae0f0a87d3d-0064c45acd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-07-29T00:18:21.763 DEBUG:boto:path=/hiiqos-56/d 2023-07-29T00:18:21.763 DEBUG:boto:auth_path=/hiiqos-56/d 2023-07-29T00:18:21.763 DEBUG:boto:path=/hiiqos-56/d?versionId=null 2023-07-29T00:18:21.764 DEBUG:boto:auth_path=/hiiqos-56/d?versionId=null 2023-07-29T00:18:21.764 DEBUG:boto:Method: GET 2023-07-29T00:18:21.765 DEBUG:boto:Path: /hiiqos-56/d?versionId=null 2023-07-29T00:18:21.765 DEBUG:boto:Data: 2023-07-29T00:18:21.765 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.766 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:18:21.766 DEBUG:boto:Port: 8001 2023-07-29T00:18:21.766 DEBUG:boto:Params: {} 2023-07-29T00:18:21.767 DEBUG:boto:Token: None 2023-07-29T00:18:21.767 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/d?versionId=null 2023-07-29T00:18:21.767 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J4seB4ppF3Dh+bV7ryXPSKRNQHg= 2023-07-29T00:18:21.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J4seB4ppF3Dh+bV7ryXPSKRNQHg=', 'Content-Length': '0'} 2023-07-29T00:18:21.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029e7fb9e52d143f3-0064c45acd-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.770 DEBUG:boto:path=/hiiqos-56/d 2023-07-29T00:18:21.771 DEBUG:boto:auth_path=/hiiqos-56/d 2023-07-29T00:18:21.771 DEBUG:boto:path=/hiiqos-56/d?versionId=null 2023-07-29T00:18:21.771 DEBUG:boto:auth_path=/hiiqos-56/d?versionId=null 2023-07-29T00:18:21.772 DEBUG:boto:Method: GET 2023-07-29T00:18:21.772 DEBUG:boto:Path: /hiiqos-56/d?versionId=null 2023-07-29T00:18:21.773 DEBUG:boto:Data: 2023-07-29T00:18:21.773 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.773 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:21.774 DEBUG:boto:Port: 8000 2023-07-29T00:18:21.774 DEBUG:boto:Params: {} 2023-07-29T00:18:21.774 DEBUG:boto:Token: None 2023-07-29T00:18:21.775 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/d?versionId=null 2023-07-29T00:18:21.775 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J4seB4ppF3Dh+bV7ryXPSKRNQHg= 2023-07-29T00:18:21.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J4seB4ppF3Dh+bV7ryXPSKRNQHg=', 'Content-Length': '0'} 2023-07-29T00:18:21.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049a472c2f0444780-0064c45acd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.778 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:18:21.779 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:18:21.779 DEBUG:boto:path=/hiiqos-56/?prefix=d 2023-07-29T00:18:21.779 DEBUG:boto:auth_path=/hiiqos-56/?prefix=d 2023-07-29T00:18:21.780 DEBUG:boto:Method: GET 2023-07-29T00:18:21.780 DEBUG:boto:Path: /hiiqos-56/?prefix=d 2023-07-29T00:18:21.780 DEBUG:boto:Data: 2023-07-29T00:18:21.781 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.781 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:18:21.781 DEBUG:boto:Port: 8001 2023-07-29T00:18:21.782 DEBUG:boto:Params: {} 2023-07-29T00:18:21.782 DEBUG:boto:Token: None 2023-07-29T00:18:21.783 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/ 2023-07-29T00:18:21.783 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FUNiQpHia71tPYJ7FWyG9lcF0mM= 2023-07-29T00:18:21.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FUNiQpHia71tPYJ7FWyG9lcF0mM=', 'Content-Length': '0'} 2023-07-29T00:18:21.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b767704ef274a72-0064c45acd-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.786 DEBUG:boto:b'hiiqos-56d1000falsed2023-07-29T00:18:00.914Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:18:21.787 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:18:21.787 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:18:21.788 DEBUG:boto:path=/hiiqos-56/?prefix=d 2023-07-29T00:18:21.788 DEBUG:boto:auth_path=/hiiqos-56/?prefix=d 2023-07-29T00:18:21.788 DEBUG:boto:Method: GET 2023-07-29T00:18:21.789 DEBUG:boto:Path: /hiiqos-56/?prefix=d 2023-07-29T00:18:21.789 DEBUG:boto:Data: 2023-07-29T00:18:21.789 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.790 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:21.790 DEBUG:boto:Port: 8000 2023-07-29T00:18:21.790 DEBUG:boto:Params: {} 2023-07-29T00:18:21.791 DEBUG:boto:Token: None 2023-07-29T00:18:21.791 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/ 2023-07-29T00:18:21.792 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FUNiQpHia71tPYJ7FWyG9lcF0mM= 2023-07-29T00:18:21.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FUNiQpHia71tPYJ7FWyG9lcF0mM=', 'Content-Length': '0'} 2023-07-29T00:18:21.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008474009d25dad7c9-0064c45acd-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.795 DEBUG:boto:b'hiiqos-56d1000falsed2023-07-29T00:18:00.914Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:18:21.796 DEBUG:boto:path=/hiiqos-56/d 2023-07-29T00:18:21.796 DEBUG:boto:auth_path=/hiiqos-56/d 2023-07-29T00:18:21.796 DEBUG:boto:Method: HEAD 2023-07-29T00:18:21.797 DEBUG:boto:Path: /hiiqos-56/d 2023-07-29T00:18:21.797 DEBUG:boto:Data: 2023-07-29T00:18:21.797 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.798 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:18:21.798 DEBUG:boto:Port: 8001 2023-07-29T00:18:21.798 DEBUG:boto:Params: {} 2023-07-29T00:18:21.799 DEBUG:boto:Token: None 2023-07-29T00:18:21.799 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/d 2023-07-29T00:18:21.799 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sttgIck58wxvL/iJVpkdLC/4bRk= 2023-07-29T00:18:21.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sttgIck58wxvL/iJVpkdLC/4bRk=', 'Content-Length': '0'} 2023-07-29T00:18:21.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c81ee492a7e8679b-0064c45acd-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.802 DEBUG:boto:path=/hiiqos-56/d 2023-07-29T00:18:21.803 DEBUG:boto:auth_path=/hiiqos-56/d 2023-07-29T00:18:21.803 DEBUG:boto:Method: HEAD 2023-07-29T00:18:21.803 DEBUG:boto:Path: /hiiqos-56/d 2023-07-29T00:18:21.804 DEBUG:boto:Data: 2023-07-29T00:18:21.804 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.804 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:21.805 DEBUG:boto:Port: 8000 2023-07-29T00:18:21.805 DEBUG:boto:Params: {} 2023-07-29T00:18:21.805 DEBUG:boto:Token: None 2023-07-29T00:18:21.806 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/d 2023-07-29T00:18:21.806 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sttgIck58wxvL/iJVpkdLC/4bRk= 2023-07-29T00:18:21.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sttgIck58wxvL/iJVpkdLC/4bRk=', 'Content-Length': '0'} 2023-07-29T00:18:21.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a917afbd7fc97fe9-0064c45acd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-07-29T00:18:21.809 DEBUG:boto:path=/hiiqos-56/d 2023-07-29T00:18:21.809 DEBUG:boto:auth_path=/hiiqos-56/d 2023-07-29T00:18:21.810 DEBUG:boto:Method: GET 2023-07-29T00:18:21.810 DEBUG:boto:Path: /hiiqos-56/d 2023-07-29T00:18:21.811 DEBUG:boto:Data: 2023-07-29T00:18:21.811 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.811 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:18:21.812 DEBUG:boto:Port: 8001 2023-07-29T00:18:21.812 DEBUG:boto:Params: {} 2023-07-29T00:18:21.812 DEBUG:boto:Token: None 2023-07-29T00:18:21.813 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/d 2023-07-29T00:18:21.813 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Ta4HMbxB04S9Oz6oKGeThlMvSWI= 2023-07-29T00:18:21.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Ta4HMbxB04S9Oz6oKGeThlMvSWI=', 'Content-Length': '0'} 2023-07-29T00:18:21.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000977afd88b94ccb86-0064c45acd-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.816 DEBUG:boto:path=/hiiqos-56/d 2023-07-29T00:18:21.817 DEBUG:boto:auth_path=/hiiqos-56/d 2023-07-29T00:18:21.817 DEBUG:boto:Method: GET 2023-07-29T00:18:21.817 DEBUG:boto:Path: /hiiqos-56/d 2023-07-29T00:18:21.818 DEBUG:boto:Data: 2023-07-29T00:18:21.818 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.818 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:21.819 DEBUG:boto:Port: 8000 2023-07-29T00:18:21.819 DEBUG:boto:Params: {} 2023-07-29T00:18:21.820 DEBUG:boto:Token: None 2023-07-29T00:18:21.820 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/d 2023-07-29T00:18:21.820 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Ta4HMbxB04S9Oz6oKGeThlMvSWI= 2023-07-29T00:18:21.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Ta4HMbxB04S9Oz6oKGeThlMvSWI=', 'Content-Length': '0'} 2023-07-29T00:18:21.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea30881e07b27506-0064c45acd-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.864 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:18:21.865 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:18:21.866 DEBUG:boto:path=/hiiqos-56/?prefix=d 2023-07-29T00:18:21.866 DEBUG:boto:auth_path=/hiiqos-56/?prefix=d 2023-07-29T00:18:21.866 DEBUG:boto:Method: GET 2023-07-29T00:18:21.867 DEBUG:boto:Path: /hiiqos-56/?prefix=d 2023-07-29T00:18:21.867 DEBUG:boto:Data: 2023-07-29T00:18:21.867 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.868 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:18:21.868 DEBUG:boto:Port: 8001 2023-07-29T00:18:21.868 DEBUG:boto:Params: {} 2023-07-29T00:18:21.869 DEBUG:boto:Token: None 2023-07-29T00:18:21.869 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/ 2023-07-29T00:18:21.870 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FUNiQpHia71tPYJ7FWyG9lcF0mM= 2023-07-29T00:18:21.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FUNiQpHia71tPYJ7FWyG9lcF0mM=', 'Content-Length': '0'} 2023-07-29T00:18:21.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027a17a8a1662b5d6-0064c45acd-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.873 DEBUG:boto:b'hiiqos-56d1000falsed2023-07-29T00:18:00.914Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:18:21.874 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:18:21.874 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:18:21.875 DEBUG:boto:path=/hiiqos-56/?prefix=d 2023-07-29T00:18:21.875 DEBUG:boto:auth_path=/hiiqos-56/?prefix=d 2023-07-29T00:18:21.875 DEBUG:boto:Method: GET 2023-07-29T00:18:21.876 DEBUG:boto:Path: /hiiqos-56/?prefix=d 2023-07-29T00:18:21.876 DEBUG:boto:Data: 2023-07-29T00:18:21.877 DEBUG:boto:Headers: {} 2023-07-29T00:18:21.877 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:21.877 DEBUG:boto:Port: 8000 2023-07-29T00:18:21.878 DEBUG:boto:Params: {} 2023-07-29T00:18:21.878 DEBUG:boto:Token: None 2023-07-29T00:18:21.879 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:18:21 GMT /hiiqos-56/ 2023-07-29T00:18:21.879 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FUNiQpHia71tPYJ7FWyG9lcF0mM= 2023-07-29T00:18:21.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FUNiQpHia71tPYJ7FWyG9lcF0mM=', 'Content-Length': '0'} 2023-07-29T00:18:21.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b548d291aad019b2-0064c45acd-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:18:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:21.882 DEBUG:boto:b'hiiqos-56d1000falsed2023-07-29T00:18:00.914Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:18:21.883 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-56 zones={a2, a1} 2023-07-29T00:18:21.883 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'hiiqos-56', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:18:21.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'hiiqos-56', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:18:21.884 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket hiiqos-56 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:18:21.934 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:18:21.934 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:18:21.974 INFO:teuthology.orchestra.run.smithi101.stdout:tenant: 2023-07-29T00:18:21.974 INFO:teuthology.orchestra.run.smithi101.stdout:bucket name: hiiqos-56 2023-07-29T00:18:21.978 INFO:teuthology.orchestra.run.smithi101.stdout:total entries: 4 2023-07-29T00:18:22.012 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'hiiqos-56', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:18:22.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'hiiqos-56', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:18:22.013 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket hiiqos-56 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:18:22.059 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:18:22.060 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:18:22.095 INFO:teuthology.orchestra.run.smithi101.stdout:tenant: 2023-07-29T00:18:22.096 INFO:teuthology.orchestra.run.smithi101.stdout:bucket name: hiiqos-56 2023-07-29T00:18:22.096 INFO:teuthology.orchestra.run.smithi101.stdout:total entries: 4 2023-07-29T00:18:22.122 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:18:22.122 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:18:22.123 DEBUG:boto:Method: HEAD 2023-07-29T00:18:22.123 DEBUG:boto:Path: /hiiqos-56/ 2023-07-29T00:18:22.123 DEBUG:boto:Data: 2023-07-29T00:18:22.124 DEBUG:boto:Headers: {} 2023-07-29T00:18:22.124 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:22.124 DEBUG:boto:Port: 8000 2023-07-29T00:18:22.125 DEBUG:boto:Params: {} 2023-07-29T00:18:22.125 DEBUG:boto:Token: None 2023-07-29T00:18:22.126 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:18:22 GMT /hiiqos-56/ 2023-07-29T00:18:22.126 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1213pH3DBqtCOzwPowpEG2EoxQE= 2023-07-29T00:18:22.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1213pH3DBqtCOzwPowpEG2EoxQE=', 'Content-Length': '0'} 2023-07-29T00:18:22.128 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005305b99beda3d6d9-0064c45ace-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:22.129 DEBUG:boto:path=/hiiqos-56/e 2023-07-29T00:18:22.129 DEBUG:boto:auth_path=/hiiqos-56/e 2023-07-29T00:18:22.130 DEBUG:boto:Method: PUT 2023-07-29T00:18:22.130 DEBUG:boto:Path: /hiiqos-56/e 2023-07-29T00:18:22.130 DEBUG:boto:Data: 2023-07-29T00:18:22.131 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:18:22.131 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:22.131 DEBUG:boto:Port: 8000 2023-07-29T00:18:22.132 DEBUG:boto:Params: {} 2023-07-29T00:18:22.132 DEBUG:boto:Token: None 2023-07-29T00:18:22.132 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:18:22 GMT /hiiqos-56/e 2023-07-29T00:18:22.133 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JeVOFpZuWx7cVo3aP1lCxcYZLFk= 2023-07-29T00:18:22.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:18:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JeVOFpZuWx7cVo3aP1lCxcYZLFk='} 2023-07-29T00:18:22.138 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e04d122c46977ced-0064c45ace-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:22.139 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:18:22.139 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:18:22.139 DEBUG:boto:Method: HEAD 2023-07-29T00:18:22.140 DEBUG:boto:Path: /hiiqos-56/ 2023-07-29T00:18:22.140 DEBUG:boto:Data: 2023-07-29T00:18:22.140 DEBUG:boto:Headers: {} 2023-07-29T00:18:22.141 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:22.141 DEBUG:boto:Port: 8000 2023-07-29T00:18:22.141 DEBUG:boto:Params: {} 2023-07-29T00:18:22.142 DEBUG:boto:Token: None 2023-07-29T00:18:22.142 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:18:22 GMT /hiiqos-56/ 2023-07-29T00:18:22.142 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1213pH3DBqtCOzwPowpEG2EoxQE= 2023-07-29T00:18:22.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1213pH3DBqtCOzwPowpEG2EoxQE=', 'Content-Length': '0'} 2023-07-29T00:18:22.145 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000d24d913a7bc0944-0064c45ace-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:22.145 DEBUG:boto:path=/hiiqos-56/f 2023-07-29T00:18:22.146 DEBUG:boto:auth_path=/hiiqos-56/f 2023-07-29T00:18:22.146 DEBUG:boto:Method: PUT 2023-07-29T00:18:22.146 DEBUG:boto:Path: /hiiqos-56/f 2023-07-29T00:18:22.147 DEBUG:boto:Data: 2023-07-29T00:18:22.147 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:18:22.147 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:22.148 DEBUG:boto:Port: 8000 2023-07-29T00:18:22.148 DEBUG:boto:Params: {} 2023-07-29T00:18:22.149 DEBUG:boto:Token: None 2023-07-29T00:18:22.149 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:18:22 GMT /hiiqos-56/f 2023-07-29T00:18:22.149 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:f+Lyy2JqmCgXrz61RjYxJvuMRXM= 2023-07-29T00:18:22.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:18:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:f+Lyy2JqmCgXrz61RjYxJvuMRXM='} 2023-07-29T00:18:22.192 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ff715a58d80380e1-0064c45ace-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:22.193 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:18:22.193 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:18:22.194 DEBUG:boto:Method: HEAD 2023-07-29T00:18:22.194 DEBUG:boto:Path: /hiiqos-56/ 2023-07-29T00:18:22.194 DEBUG:boto:Data: 2023-07-29T00:18:22.195 DEBUG:boto:Headers: {} 2023-07-29T00:18:22.195 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:22.195 DEBUG:boto:Port: 8000 2023-07-29T00:18:22.196 DEBUG:boto:Params: {} 2023-07-29T00:18:22.196 DEBUG:boto:Token: None 2023-07-29T00:18:22.197 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:18:22 GMT /hiiqos-56/ 2023-07-29T00:18:22.197 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1213pH3DBqtCOzwPowpEG2EoxQE= 2023-07-29T00:18:22.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1213pH3DBqtCOzwPowpEG2EoxQE=', 'Content-Length': '0'} 2023-07-29T00:18:22.199 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c9b4f847695fba8e-0064c45ace-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:22.200 DEBUG:boto:path=/hiiqos-56/g 2023-07-29T00:18:22.200 DEBUG:boto:auth_path=/hiiqos-56/g 2023-07-29T00:18:22.201 DEBUG:boto:Method: PUT 2023-07-29T00:18:22.201 DEBUG:boto:Path: /hiiqos-56/g 2023-07-29T00:18:22.201 DEBUG:boto:Data: 2023-07-29T00:18:22.202 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:18:22.202 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:22.202 DEBUG:boto:Port: 8000 2023-07-29T00:18:22.203 DEBUG:boto:Params: {} 2023-07-29T00:18:22.203 DEBUG:boto:Token: None 2023-07-29T00:18:22.203 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:18:22 GMT /hiiqos-56/g 2023-07-29T00:18:22.204 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nqceCHLoOefYKEDLUe17vnu5lnA= 2023-07-29T00:18:22.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:18:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nqceCHLoOefYKEDLUe17vnu5lnA='} 2023-07-29T00:18:22.209 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dcbff5f20a5b60df-0064c45ace-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:22.209 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:18:22.210 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:18:22.210 DEBUG:boto:Method: HEAD 2023-07-29T00:18:22.210 DEBUG:boto:Path: /hiiqos-56/ 2023-07-29T00:18:22.211 DEBUG:boto:Data: 2023-07-29T00:18:22.211 DEBUG:boto:Headers: {} 2023-07-29T00:18:22.211 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:22.211 DEBUG:boto:Port: 8000 2023-07-29T00:18:22.212 DEBUG:boto:Params: {} 2023-07-29T00:18:22.212 DEBUG:boto:Token: None 2023-07-29T00:18:22.213 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:18:22 GMT /hiiqos-56/ 2023-07-29T00:18:22.213 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1213pH3DBqtCOzwPowpEG2EoxQE= 2023-07-29T00:18:22.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:18:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1213pH3DBqtCOzwPowpEG2EoxQE=', 'Content-Length': '0'} 2023-07-29T00:18:22.215 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000067211e63c2226be-0064c45ace-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:22.215 DEBUG:boto:path=/hiiqos-56/h 2023-07-29T00:18:22.216 DEBUG:boto:auth_path=/hiiqos-56/h 2023-07-29T00:18:22.216 DEBUG:boto:Method: PUT 2023-07-29T00:18:22.216 DEBUG:boto:Path: /hiiqos-56/h 2023-07-29T00:18:22.217 DEBUG:boto:Data: 2023-07-29T00:18:22.217 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:18:22.217 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:18:22.218 DEBUG:boto:Port: 8000 2023-07-29T00:18:22.218 DEBUG:boto:Params: {} 2023-07-29T00:18:22.218 DEBUG:boto:Token: None 2023-07-29T00:18:22.219 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:18:22 GMT /hiiqos-56/h 2023-07-29T00:18:22.219 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/d/WT7rCkcVVBbIZ4U/L0obqN7E= 2023-07-29T00:18:22.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:18:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/d/WT7rCkcVVBbIZ4U/L0obqN7E='} 2023-07-29T00:18:22.223 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000914216e84ed5f93f-0064c45ace-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:18:22.223 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-56 2023-07-29T00:18:22.224 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:18:22.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:18:22.224 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-56 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:18:22.276 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:18:22.277 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:18:22.308 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:18:22.264+0000 7fa00b836880 1 waiting to reach latest gen 2, on 0.. 2023-07-29T00:18:27.309 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:18:27.264+0000 7fa00b836880 1 waiting to reach latest gen 2, on 0.. 2023-07-29T00:18:32.310 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:18:32.265+0000 7fa00b836880 1 waiting to reach latest gen 2, on 0.. 2023-07-29T00:18:37.311 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:18:37.266+0000 7fa00b836880 1 waiting to reach latest gen 2, on 1.. 2023-07-29T00:18:42.311 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:18:42.266+0000 7fa00b836880 1 waiting to reach latest gen 2, on 1.. 2023-07-29T00:18:47.312 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:18:47.267+0000 7fa00b836880 1 waiting to reach latest gen 2, on 1.. 2023-07-29T00:18:52.313 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:18:52.267+0000 7fa00b836880 1 waiting to reach latest gen 2, on 1.. 2023-07-29T00:18:57.314 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:18:57.268+0000 7fa00b836880 1 waiting to reach latest gen 2, on 1.. 2023-07-29T00:19:02.314 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:19:02.269+0000 7fa00b836880 1 waiting to reach latest gen 2, on 1.. 2023-07-29T00:19:07.315 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:19:07.270+0000 7fa00b836880 1 waiting to reach latest gen 2, on 1.. 2023-07-29T00:19:12.316 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:19:12.270+0000 7fa00b836880 1 waiting to reach latest gen 2, on 1.. 2023-07-29T00:19:17.317 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:19:17.271+0000 7fa00b836880 1 waiting to reach latest gen 2, on 1.. 2023-07-29T00:19:22.318 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:19:22.271+0000 7fa00b836880 1 waiting to reach latest gen 2, on 1.. 2023-07-29T00:19:27.318 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:19:27.272+0000 7fa00b836880 1 waiting to reach latest gen 2, on 1.. 2023-07-29T00:19:32.319 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:19:32.273+0000 7fa00b836880 1 waiting to reach latest gen 2, on 1.. 2023-07-29T00:19:37.320 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:19:37.273+0000 7fa00b836880 1 waiting to reach latest gen 2, on 1.. 2023-07-29T00:19:42.320 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:19:42.274+0000 7fa00b836880 1 waiting to reach latest gen 2, on 1.. 2023-07-29T00:19:47.321 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:19:47.274+0000 7fa00b836880 1 waiting to reach latest gen 2, on 1.. 2023-07-29T00:19:52.321 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:19:52.275+0000 7fa00b836880 1 waiting to reach latest gen 2, on 1.. 2023-07-29T00:19:57.323 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:19:57.276+0000 7fa00b836880 1 waiting for incremental sync to catch up: 2023-07-29T00:19:57.323 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , 00000000002.55.6] 2023-07-29T00:19:57.323 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.27.6, 00000000002.30.6, 00000000002.55.6] 2023-07-29T00:20:02.323 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:20:02.277+0000 7fa00b836880 1 waiting for incremental sync to catch up: 2023-07-29T00:20:02.324 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , 00000000002.55.6] 2023-07-29T00:20:02.324 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.27.6, 00000000002.30.6, 00000000002.55.6] 2023-07-29T00:20:07.324 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:20:07.277+0000 7fa00b836880 1 waiting for incremental sync to catch up: 2023-07-29T00:20:07.325 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , 00000000002.55.6] 2023-07-29T00:20:07.325 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.27.6, 00000000002.30.6, 00000000002.55.6] 2023-07-29T00:20:12.325 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:20:12.278+0000 7fa00b836880 1 waiting for incremental sync to catch up: 2023-07-29T00:20:12.325 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , 00000000002.55.6] 2023-07-29T00:20:12.326 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.27.6, 00000000002.30.6, 00000000002.55.6] 2023-07-29T00:20:17.326 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:20:17.279+0000 7fa00b836880 1 waiting for incremental sync to catch up: 2023-07-29T00:20:17.326 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , 00000000002.55.6] 2023-07-29T00:20:17.327 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.27.6, 00000000002.30.6, 00000000002.55.6] 2023-07-29T00:20:18.085 INFO:tasks.rgw.c2.client.1.smithi139.stdout:2023-07-29T00:20:18.038+0000 7f5731a6a640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T00:20:22.327 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:20:22.280+0000 7fa00b836880 1 waiting for incremental sync to catch up: 2023-07-29T00:20:22.327 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , 00000000002.55.6] 2023-07-29T00:20:22.327 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.27.6, 00000000002.30.6, 00000000002.55.6] 2023-07-29T00:20:27.328 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:20:27.280+0000 7fa00b836880 1 waiting for incremental sync to catch up: 2023-07-29T00:20:27.328 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , 00000000002.55.6] 2023-07-29T00:20:27.328 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.27.6, 00000000002.30.6, 00000000002.55.6] 2023-07-29T00:20:32.328 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:20:32.281+0000 7fa00b836880 1 waiting for incremental sync to catch up: 2023-07-29T00:20:32.329 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , 00000000002.55.6] 2023-07-29T00:20:32.329 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.27.6, 00000000002.30.6, 00000000002.55.6] 2023-07-29T00:20:37.329 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:20:37.282+0000 7fa00b836880 1 waiting for incremental sync to catch up: 2023-07-29T00:20:37.330 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.30.6, 00000000002.55.6] 2023-07-29T00:20:37.330 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.27.6, 00000000002.30.6, 00000000002.55.6] 2023-07-29T00:20:42.330 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:20:42.283+0000 7fa00b836880 1 waiting for incremental sync to catch up: 2023-07-29T00:20:42.331 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.30.6, 00000000002.55.6] 2023-07-29T00:20:42.331 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.27.6, 00000000002.30.6, 00000000002.55.6] 2023-07-29T00:20:47.332 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:20:47.284+0000 7fa00b836880 1 waiting for incremental sync to catch up: 2023-07-29T00:20:47.332 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.30.6, 00000000002.55.6] 2023-07-29T00:20:47.332 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.27.6, 00000000002.30.6, 00000000002.55.6] 2023-07-29T00:20:52.332 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:20:52.285+0000 7fa00b836880 1 waiting for incremental sync to catch up: 2023-07-29T00:20:52.333 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.30.6, 00000000002.55.6] 2023-07-29T00:20:52.333 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.27.6, 00000000002.30.6, 00000000002.55.6] 2023-07-29T00:20:57.333 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:20:57.285+0000 7fa00b836880 1 waiting for incremental sync to catch up: 2023-07-29T00:20:57.334 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.30.6, 00000000002.55.6] 2023-07-29T00:20:57.334 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.27.6, 00000000002.30.6, 00000000002.55.6] 2023-07-29T00:21:01.973 INFO:tasks.rgw.c2.client.0.smithi139.stdout:2023-07-29T00:21:01.925+0000 7f5f2671b640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T00:21:02.335 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:21:02.286+0000 7fa00b836880 1 waiting for incremental sync to catch up: 2023-07-29T00:21:02.335 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.30.6, 00000000002.55.6] 2023-07-29T00:21:02.335 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.27.6, 00000000002.30.6, 00000000002.55.6] 2023-07-29T00:21:07.335 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:21:07.287+0000 7fa00b836880 1 waiting for incremental sync to catch up: 2023-07-29T00:21:07.335 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.30.6, 00000000002.55.6] 2023-07-29T00:21:07.336 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.27.6, 00000000002.30.6, 00000000002.55.6] 2023-07-29T00:21:12.336 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:21:12.288+0000 7fa00b836880 1 waiting for incremental sync to catch up: 2023-07-29T00:21:12.337 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.30.6, 00000000002.55.6] 2023-07-29T00:21:12.337 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.27.6, 00000000002.30.6, 00000000002.55.6] 2023-07-29T00:21:17.337 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:21:17.289+0000 7fa00b836880 1 bucket sync caught up with source: 2023-07-29T00:21:17.338 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000003.27.6, 00000000002.30.6, 00000000002.55.6] 2023-07-29T00:21:17.338 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.27.6, 00000000002.30.6, 00000000002.55.6] 2023-07-29T00:21:17.339 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:21:17.289+0000 7fa00b836880 0 bucket checkpoint complete 2023-07-29T00:21:17.354 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-56 2023-07-29T00:21:17.355 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:21:17.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:21:17.356 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-56 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:21:17.414 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:21:17.414 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:21:17.451 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:21:17.402+0000 7faf03d32880 1 waiting for incremental sync to catch up: 2023-07-29T00:21:17.451 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.997.6, , , , 00000000001.762.6, , , 00000000002.997.6, , 00000000001.876.6] 2023-07-29T00:21:17.451 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000002.997.6, , , 00000000001.707.6, 00000000001.762.6, , , 00000000002.997.6, , 00000000002.882.6] 2023-07-29T00:21:22.452 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:21:22.404+0000 7faf03d32880 1 bucket sync caught up with source: 2023-07-29T00:21:22.453 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.997.6, , , 00000000001.707.6, 00000000001.762.6, , , 00000000002.997.6, , 00000000002.882.6] 2023-07-29T00:21:22.453 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000002.997.6, , , 00000000001.707.6, 00000000001.762.6, , , 00000000002.997.6, , 00000000002.882.6] 2023-07-29T00:21:22.453 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:21:22.404+0000 7faf03d32880 0 bucket checkpoint complete 2023-07-29T00:21:22.469 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-56 zones={a2, a1} 2023-07-29T00:21:22.470 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:21:22.470 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:21:22.471 DEBUG:boto:Method: HEAD 2023-07-29T00:21:22.471 DEBUG:boto:Path: /hiiqos-56/ 2023-07-29T00:21:22.472 DEBUG:boto:Data: 2023-07-29T00:21:22.472 DEBUG:boto:Headers: {} 2023-07-29T00:21:22.472 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:22.473 DEBUG:boto:Port: 8001 2023-07-29T00:21:22.473 DEBUG:boto:Params: {} 2023-07-29T00:21:22.473 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-29T00:21:22.474 DEBUG:boto:Token: None 2023-07-29T00:21:22.474 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:21:22 GMT /hiiqos-56/ 2023-07-29T00:21:22.475 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zEdGx0I0vmOmIPPHsQWRNvedzUk= 2023-07-29T00:21:22.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zEdGx0I0vmOmIPPHsQWRNvedzUk=', 'Content-Length': '0'} 2023-07-29T00:21:27.479 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d55df908c39d02f3-0064c45b87-1663-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:21:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:27.480 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:21:27.481 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:21:27.481 DEBUG:boto:Method: HEAD 2023-07-29T00:21:27.481 DEBUG:boto:Path: /hiiqos-56/ 2023-07-29T00:21:27.482 DEBUG:boto:Data: 2023-07-29T00:21:27.482 DEBUG:boto:Headers: {} 2023-07-29T00:21:27.482 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:27.483 DEBUG:boto:Port: 8000 2023-07-29T00:21:27.483 DEBUG:boto:Params: {} 2023-07-29T00:21:27.484 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-29T00:21:27.484 DEBUG:boto:Token: None 2023-07-29T00:21:27.484 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:21:27 GMT /hiiqos-56/ 2023-07-29T00:21:27.485 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O5xZRbCQ6D4sOqzt6cslO/NpmRI= 2023-07-29T00:21:27.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O5xZRbCQ6D4sOqzt6cslO/NpmRI=', 'Content-Length': '0'} 2023-07-29T00:21:32.489 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d3f08d40dbd481ac-0064c45b8c-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:21:32 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:32.489 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-29T00:21:32.490 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:21:32.490 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:21:32.491 DEBUG:boto:path=/hiiqos-56/?versions&max-keys=999 2023-07-29T00:21:32.491 DEBUG:boto:auth_path=/hiiqos-56/?versions&max-keys=999 2023-07-29T00:21:32.491 DEBUG:boto:Method: GET 2023-07-29T00:21:32.492 DEBUG:boto:Path: /hiiqos-56/?versions&max-keys=999 2023-07-29T00:21:32.492 DEBUG:boto:Data: 2023-07-29T00:21:32.493 DEBUG:boto:Headers: {} 2023-07-29T00:21:32.493 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:32.493 DEBUG:boto:Port: 8001 2023-07-29T00:21:32.494 DEBUG:boto:Params: {} 2023-07-29T00:21:32.494 DEBUG:boto:Token: None 2023-07-29T00:21:32.495 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:32 GMT /hiiqos-56/?versions 2023-07-29T00:21:32.495 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+PMRTp/3yqGg/Pi65FFV9AXibNk= 2023-07-29T00:21:32.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:32 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+PMRTp/3yqGg/Pi65FFV9AXibNk=', 'Content-Length': '0'} 2023-07-29T00:21:32.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb0e83ab7f96a5ba-0064c45b8c-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:21:32 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:32.499 DEBUG:boto:b'hiiqos-56999falseanulltrue2023-07-29T00:18:00.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:18:00.841Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:18:00.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:18:00.914Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-07-29T00:18:22.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-07-29T00:18:22.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-07-29T00:18:22.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-07-29T00:18:22.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:21:32.500 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-07-29T00:21:32.501 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-07-29T00:21:32.501 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-07-29T00:21:32.501 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-07-29T00:21:32.502 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-07-29T00:21:32.502 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-07-29T00:21:32.502 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-07-29T00:21:32.503 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-07-29T00:21:32.503 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-29T00:21:32.503 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:21:32.504 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:21:32.504 DEBUG:boto:path=/hiiqos-56/?versions&max-keys=999 2023-07-29T00:21:32.504 DEBUG:boto:auth_path=/hiiqos-56/?versions&max-keys=999 2023-07-29T00:21:32.505 DEBUG:boto:Method: GET 2023-07-29T00:21:32.505 DEBUG:boto:Path: /hiiqos-56/?versions&max-keys=999 2023-07-29T00:21:32.505 DEBUG:boto:Data: 2023-07-29T00:21:32.506 DEBUG:boto:Headers: {} 2023-07-29T00:21:32.506 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:32.507 DEBUG:boto:Port: 8000 2023-07-29T00:21:32.507 DEBUG:boto:Params: {} 2023-07-29T00:21:32.507 DEBUG:boto:Token: None 2023-07-29T00:21:32.508 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:32 GMT /hiiqos-56/?versions 2023-07-29T00:21:32.508 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+PMRTp/3yqGg/Pi65FFV9AXibNk= 2023-07-29T00:21:32.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:32 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+PMRTp/3yqGg/Pi65FFV9AXibNk=', 'Content-Length': '0'} 2023-07-29T00:21:32.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4c7ec31afaaf524-0064c45b8c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:21:32 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:32.511 DEBUG:boto:b'hiiqos-56999falseanulltrue2023-07-29T00:18:00.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:18:00.841Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:18:00.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:18:00.914Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-07-29T00:18:22.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-07-29T00:18:22.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-07-29T00:18:22.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-07-29T00:18:22.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:21:32.512 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-07-29T00:21:32.512 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-07-29T00:21:32.513 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-07-29T00:21:32.513 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-07-29T00:21:32.513 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-07-29T00:21:32.514 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-07-29T00:21:32.514 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-07-29T00:21:32.514 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-07-29T00:21:32.515 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:21:32.515 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:21:32.515 DEBUG:boto:path=/hiiqos-56/?versions&max-keys=999 2023-07-29T00:21:32.516 DEBUG:boto:auth_path=/hiiqos-56/?versions&max-keys=999 2023-07-29T00:21:32.516 DEBUG:boto:Method: GET 2023-07-29T00:21:32.516 DEBUG:boto:Path: /hiiqos-56/?versions&max-keys=999 2023-07-29T00:21:32.517 DEBUG:boto:Data: 2023-07-29T00:21:32.517 DEBUG:boto:Headers: {} 2023-07-29T00:21:32.517 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:32.518 DEBUG:boto:Port: 8001 2023-07-29T00:21:32.518 DEBUG:boto:Params: {} 2023-07-29T00:21:32.519 DEBUG:boto:Token: None 2023-07-29T00:21:32.519 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:32 GMT /hiiqos-56/?versions 2023-07-29T00:21:32.520 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+PMRTp/3yqGg/Pi65FFV9AXibNk= 2023-07-29T00:21:32.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:32 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+PMRTp/3yqGg/Pi65FFV9AXibNk=', 'Content-Length': '0'} 2023-07-29T00:21:32.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b806ea64bbf9de6-0064c45b8c-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:21:32 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:32.523 DEBUG:boto:b'hiiqos-56999falseanulltrue2023-07-29T00:18:00.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:18:00.841Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:18:00.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:18:00.914Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-07-29T00:18:22.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-07-29T00:18:22.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-07-29T00:18:22.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-07-29T00:18:22.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:21:32.524 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:21:32.525 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:21:32.526 DEBUG:boto:path=/hiiqos-56/?versions&max-keys=999 2023-07-29T00:21:32.527 DEBUG:boto:auth_path=/hiiqos-56/?versions&max-keys=999 2023-07-29T00:21:32.527 DEBUG:boto:Method: GET 2023-07-29T00:21:32.528 DEBUG:boto:Path: /hiiqos-56/?versions&max-keys=999 2023-07-29T00:21:32.529 DEBUG:boto:Data: 2023-07-29T00:21:32.530 DEBUG:boto:Headers: {} 2023-07-29T00:21:32.530 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:32.531 DEBUG:boto:Port: 8000 2023-07-29T00:21:32.532 DEBUG:boto:Params: {} 2023-07-29T00:21:32.533 DEBUG:boto:Token: None 2023-07-29T00:21:32.534 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:32 GMT /hiiqos-56/?versions 2023-07-29T00:21:32.534 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+PMRTp/3yqGg/Pi65FFV9AXibNk= 2023-07-29T00:21:32.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:32 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+PMRTp/3yqGg/Pi65FFV9AXibNk=', 'Content-Length': '0'} 2023-07-29T00:21:32.536 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023beac1c6730bc0d-0064c45b8c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:21:32 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:32.537 DEBUG:boto:b'hiiqos-56999falseanulltrue2023-07-29T00:18:00.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:18:00.841Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:18:00.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:18:00.914Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-07-29T00:18:22.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-07-29T00:18:22.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-07-29T00:18:22.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-07-29T00:18:22.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:21:32.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-07-29T00:21:32.538 DEBUG:boto:path=/hiiqos-56/a 2023-07-29T00:21:32.539 DEBUG:boto:auth_path=/hiiqos-56/a 2023-07-29T00:21:32.539 DEBUG:boto:path=/hiiqos-56/a?versionId=null 2023-07-29T00:21:32.539 DEBUG:boto:auth_path=/hiiqos-56/a?versionId=null 2023-07-29T00:21:32.540 DEBUG:boto:Method: GET 2023-07-29T00:21:32.540 DEBUG:boto:Path: /hiiqos-56/a?versionId=null 2023-07-29T00:21:32.541 DEBUG:boto:Data: 2023-07-29T00:21:32.541 DEBUG:boto:Headers: {} 2023-07-29T00:21:32.541 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:32.542 DEBUG:boto:Port: 8001 2023-07-29T00:21:32.542 DEBUG:boto:Params: {} 2023-07-29T00:21:32.543 DEBUG:boto:Token: None 2023-07-29T00:21:32.543 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:32 GMT /hiiqos-56/a?versionId=null 2023-07-29T00:21:32.543 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aOvA+aXcnctGcxlfVYzyYvdIS2Y= 2023-07-29T00:21:32.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:32 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aOvA+aXcnctGcxlfVYzyYvdIS2Y=', 'Content-Length': '0'} 2023-07-29T00:21:32.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027e68cf782c81244-0064c45b8c-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:32 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:32.547 DEBUG:boto:path=/hiiqos-56/a 2023-07-29T00:21:32.547 DEBUG:boto:auth_path=/hiiqos-56/a 2023-07-29T00:21:32.547 DEBUG:boto:path=/hiiqos-56/a?versionId=null 2023-07-29T00:21:32.548 DEBUG:boto:auth_path=/hiiqos-56/a?versionId=null 2023-07-29T00:21:32.548 DEBUG:boto:Method: GET 2023-07-29T00:21:32.549 DEBUG:boto:Path: /hiiqos-56/a?versionId=null 2023-07-29T00:21:32.549 DEBUG:boto:Data: 2023-07-29T00:21:32.549 DEBUG:boto:Headers: {} 2023-07-29T00:21:32.550 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:32.551 DEBUG:boto:Port: 8000 2023-07-29T00:21:32.552 DEBUG:boto:Params: {} 2023-07-29T00:21:32.553 DEBUG:boto:Token: None 2023-07-29T00:21:32.554 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:32 GMT /hiiqos-56/a?versionId=null 2023-07-29T00:21:32.555 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aOvA+aXcnctGcxlfVYzyYvdIS2Y= 2023-07-29T00:21:32.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:32 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aOvA+aXcnctGcxlfVYzyYvdIS2Y=', 'Content-Length': '0'} 2023-07-29T00:21:32.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe8ce9092fff107c-0064c45b8c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:32 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:32.558 DEBUG:boto:path=/hiiqos-56/a 2023-07-29T00:21:32.559 DEBUG:boto:auth_path=/hiiqos-56/a 2023-07-29T00:21:32.559 DEBUG:boto:path=/hiiqos-56/a?versionId=null 2023-07-29T00:21:32.559 DEBUG:boto:auth_path=/hiiqos-56/a?versionId=null 2023-07-29T00:21:32.560 DEBUG:boto:Method: HEAD 2023-07-29T00:21:32.560 DEBUG:boto:Path: /hiiqos-56/a?versionId=null 2023-07-29T00:21:32.560 DEBUG:boto:Data: 2023-07-29T00:21:32.561 DEBUG:boto:Headers: {} 2023-07-29T00:21:32.561 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:32.561 DEBUG:boto:Port: 8001 2023-07-29T00:21:32.562 DEBUG:boto:Params: {} 2023-07-29T00:21:32.562 DEBUG:boto:Token: None 2023-07-29T00:21:32.563 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:21:32 GMT /hiiqos-56/a?versionId=null 2023-07-29T00:21:32.563 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mhhFlUDF+ypKp5IoVGqSjpA2EFs= 2023-07-29T00:21:32.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:32 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mhhFlUDF+ypKp5IoVGqSjpA2EFs=', 'Content-Length': '0'} 2023-07-29T00:21:32.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d853e1b574cb870-0064c45b8c-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:32 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:32.566 DEBUG:boto:path=/hiiqos-56/a 2023-07-29T00:21:32.566 DEBUG:boto:auth_path=/hiiqos-56/a 2023-07-29T00:21:32.567 DEBUG:boto:path=/hiiqos-56/a?versionId=null 2023-07-29T00:21:32.567 DEBUG:boto:auth_path=/hiiqos-56/a?versionId=null 2023-07-29T00:21:32.567 DEBUG:boto:Method: HEAD 2023-07-29T00:21:32.568 DEBUG:boto:Path: /hiiqos-56/a?versionId=null 2023-07-29T00:21:32.568 DEBUG:boto:Data: 2023-07-29T00:21:32.568 DEBUG:boto:Headers: {} 2023-07-29T00:21:32.569 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:32.569 DEBUG:boto:Port: 8000 2023-07-29T00:21:32.569 DEBUG:boto:Params: {} 2023-07-29T00:21:32.570 DEBUG:boto:Token: None 2023-07-29T00:21:32.570 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:21:32 GMT /hiiqos-56/a?versionId=null 2023-07-29T00:21:32.570 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mhhFlUDF+ypKp5IoVGqSjpA2EFs= 2023-07-29T00:21:32.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:32 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mhhFlUDF+ypKp5IoVGqSjpA2EFs=', 'Content-Length': '0'} 2023-07-29T00:21:32.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025d5184a7b07cf23-0064c45b8c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:32 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:32.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-07-29T00:21:32.574 DEBUG:boto:path=/hiiqos-56/a 2023-07-29T00:21:32.574 DEBUG:boto:auth_path=/hiiqos-56/a 2023-07-29T00:21:32.575 DEBUG:boto:path=/hiiqos-56/a?versionId=null 2023-07-29T00:21:32.575 DEBUG:boto:auth_path=/hiiqos-56/a?versionId=null 2023-07-29T00:21:32.576 DEBUG:boto:Method: GET 2023-07-29T00:21:32.576 DEBUG:boto:Path: /hiiqos-56/a?versionId=null 2023-07-29T00:21:32.577 DEBUG:boto:Data: 2023-07-29T00:21:32.577 DEBUG:boto:Headers: {} 2023-07-29T00:21:32.577 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:32.578 DEBUG:boto:Port: 8001 2023-07-29T00:21:32.578 DEBUG:boto:Params: {} 2023-07-29T00:21:32.579 DEBUG:boto:Token: None 2023-07-29T00:21:32.579 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:32 GMT /hiiqos-56/a?versionId=null 2023-07-29T00:21:32.579 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aOvA+aXcnctGcxlfVYzyYvdIS2Y= 2023-07-29T00:21:32.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:32 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aOvA+aXcnctGcxlfVYzyYvdIS2Y=', 'Content-Length': '0'} 2023-07-29T00:21:32.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2b2a2f7910a8687-0064c45b8c-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:32 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:32.583 DEBUG:boto:path=/hiiqos-56/a 2023-07-29T00:21:32.583 DEBUG:boto:auth_path=/hiiqos-56/a 2023-07-29T00:21:32.583 DEBUG:boto:path=/hiiqos-56/a?versionId=null 2023-07-29T00:21:32.584 DEBUG:boto:auth_path=/hiiqos-56/a?versionId=null 2023-07-29T00:21:32.584 DEBUG:boto:Method: GET 2023-07-29T00:21:32.585 DEBUG:boto:Path: /hiiqos-56/a?versionId=null 2023-07-29T00:21:32.585 DEBUG:boto:Data: 2023-07-29T00:21:32.586 DEBUG:boto:Headers: {} 2023-07-29T00:21:32.586 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:32.586 DEBUG:boto:Port: 8000 2023-07-29T00:21:32.587 DEBUG:boto:Params: {} 2023-07-29T00:21:32.587 DEBUG:boto:Token: None 2023-07-29T00:21:32.588 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:32 GMT /hiiqos-56/a?versionId=null 2023-07-29T00:21:32.588 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aOvA+aXcnctGcxlfVYzyYvdIS2Y= 2023-07-29T00:21:32.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:32 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aOvA+aXcnctGcxlfVYzyYvdIS2Y=', 'Content-Length': '0'} 2023-07-29T00:21:32.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015e71618151b030d-0064c45b8c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:32 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:32.633 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:21:32.633 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:21:32.634 DEBUG:boto:path=/hiiqos-56/?prefix=a 2023-07-29T00:21:32.634 DEBUG:boto:auth_path=/hiiqos-56/?prefix=a 2023-07-29T00:21:32.634 DEBUG:boto:Method: GET 2023-07-29T00:21:32.635 DEBUG:boto:Path: /hiiqos-56/?prefix=a 2023-07-29T00:21:32.635 DEBUG:boto:Data: 2023-07-29T00:21:32.635 DEBUG:boto:Headers: {} 2023-07-29T00:21:32.636 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:32.636 DEBUG:boto:Port: 8001 2023-07-29T00:21:32.636 DEBUG:boto:Params: {} 2023-07-29T00:21:32.637 DEBUG:boto:Token: None 2023-07-29T00:21:32.637 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:32 GMT /hiiqos-56/ 2023-07-29T00:21:32.638 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MbA4bkqS3ZaSkI+WR3OfIFnpI0U= 2023-07-29T00:21:32.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:32 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MbA4bkqS3ZaSkI+WR3OfIFnpI0U=', 'Content-Length': '0'} 2023-07-29T00:21:32.641 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5cd14d35ef2b53f-0064c45b8c-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:21:32 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:32.641 DEBUG:boto:b'hiiqos-56a1000falsea2023-07-29T00:18:00.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:21:32.642 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:21:32.642 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:21:32.643 DEBUG:boto:path=/hiiqos-56/?prefix=a 2023-07-29T00:21:32.643 DEBUG:boto:auth_path=/hiiqos-56/?prefix=a 2023-07-29T00:21:32.643 DEBUG:boto:Method: GET 2023-07-29T00:21:32.644 DEBUG:boto:Path: /hiiqos-56/?prefix=a 2023-07-29T00:21:32.644 DEBUG:boto:Data: 2023-07-29T00:21:32.645 DEBUG:boto:Headers: {} 2023-07-29T00:21:32.645 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:32.645 DEBUG:boto:Port: 8000 2023-07-29T00:21:32.646 DEBUG:boto:Params: {} 2023-07-29T00:21:32.646 DEBUG:boto:Token: None 2023-07-29T00:21:32.646 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:32 GMT /hiiqos-56/ 2023-07-29T00:21:32.647 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MbA4bkqS3ZaSkI+WR3OfIFnpI0U= 2023-07-29T00:21:32.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:32 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MbA4bkqS3ZaSkI+WR3OfIFnpI0U=', 'Content-Length': '0'} 2023-07-29T00:21:32.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e5a3aec923cc0b4-0064c45b8c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:21:32 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:32.650 DEBUG:boto:b'hiiqos-56a1000falsea2023-07-29T00:18:00.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:21:32.651 DEBUG:boto:path=/hiiqos-56/a 2023-07-29T00:21:32.651 DEBUG:boto:auth_path=/hiiqos-56/a 2023-07-29T00:21:32.652 DEBUG:boto:Method: HEAD 2023-07-29T00:21:32.652 DEBUG:boto:Path: /hiiqos-56/a 2023-07-29T00:21:32.652 DEBUG:boto:Data: 2023-07-29T00:21:32.653 DEBUG:boto:Headers: {} 2023-07-29T00:21:32.653 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:32.653 DEBUG:boto:Port: 8001 2023-07-29T00:21:32.654 DEBUG:boto:Params: {} 2023-07-29T00:21:32.654 DEBUG:boto:Token: None 2023-07-29T00:21:32.654 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:21:32 GMT /hiiqos-56/a 2023-07-29T00:21:32.655 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:h6RzU9i2zWBuyEtELIrq61IOBVo= 2023-07-29T00:21:32.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:32 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:h6RzU9i2zWBuyEtELIrq61IOBVo=', 'Content-Length': '0'} 2023-07-29T00:21:32.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b83cbbe3d5b6c613-0064c45b8c-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:32 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:32.658 DEBUG:boto:path=/hiiqos-56/a 2023-07-29T00:21:32.658 DEBUG:boto:auth_path=/hiiqos-56/a 2023-07-29T00:21:32.658 DEBUG:boto:Method: HEAD 2023-07-29T00:21:32.659 DEBUG:boto:Path: /hiiqos-56/a 2023-07-29T00:21:32.659 DEBUG:boto:Data: 2023-07-29T00:21:32.659 DEBUG:boto:Headers: {} 2023-07-29T00:21:32.660 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:32.660 DEBUG:boto:Port: 8000 2023-07-29T00:21:32.660 DEBUG:boto:Params: {} 2023-07-29T00:21:32.661 DEBUG:boto:Token: None 2023-07-29T00:21:32.661 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:21:32 GMT /hiiqos-56/a 2023-07-29T00:21:32.662 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:h6RzU9i2zWBuyEtELIrq61IOBVo= 2023-07-29T00:21:32.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:32 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:h6RzU9i2zWBuyEtELIrq61IOBVo=', 'Content-Length': '0'} 2023-07-29T00:21:32.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d30c63505882290-0064c45b8c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:32 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:32.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-07-29T00:21:32.665 DEBUG:boto:path=/hiiqos-56/a 2023-07-29T00:21:32.665 DEBUG:boto:auth_path=/hiiqos-56/a 2023-07-29T00:21:32.665 DEBUG:boto:Method: GET 2023-07-29T00:21:32.666 DEBUG:boto:Path: /hiiqos-56/a 2023-07-29T00:21:32.666 DEBUG:boto:Data: 2023-07-29T00:21:32.666 DEBUG:boto:Headers: {} 2023-07-29T00:21:32.667 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:32.667 DEBUG:boto:Port: 8001 2023-07-29T00:21:32.667 DEBUG:boto:Params: {} 2023-07-29T00:21:32.668 DEBUG:boto:Token: None 2023-07-29T00:21:32.668 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:32 GMT /hiiqos-56/a 2023-07-29T00:21:32.669 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YM1M/bKwO0WEw775SNnKahK1A1c= 2023-07-29T00:21:32.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:32 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YM1M/bKwO0WEw775SNnKahK1A1c=', 'Content-Length': '0'} 2023-07-29T00:21:32.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d37c933be15dacc3-0064c45b8c-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:32 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:32.712 DEBUG:boto:path=/hiiqos-56/a 2023-07-29T00:21:32.713 DEBUG:boto:auth_path=/hiiqos-56/a 2023-07-29T00:21:32.713 DEBUG:boto:Method: GET 2023-07-29T00:21:32.713 DEBUG:boto:Path: /hiiqos-56/a 2023-07-29T00:21:32.714 DEBUG:boto:Data: 2023-07-29T00:21:32.714 DEBUG:boto:Headers: {} 2023-07-29T00:21:32.715 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:32.715 DEBUG:boto:Port: 8000 2023-07-29T00:21:32.715 DEBUG:boto:Params: {} 2023-07-29T00:21:32.716 DEBUG:boto:Token: None 2023-07-29T00:21:32.716 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:32 GMT /hiiqos-56/a 2023-07-29T00:21:32.717 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YM1M/bKwO0WEw775SNnKahK1A1c= 2023-07-29T00:21:32.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:32 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YM1M/bKwO0WEw775SNnKahK1A1c=', 'Content-Length': '0'} 2023-07-29T00:21:32.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c488f3ece46be1f1-0064c45b8c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:32 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:32.719 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:21:32.720 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:21:32.720 DEBUG:boto:path=/hiiqos-56/?prefix=a 2023-07-29T00:21:32.721 DEBUG:boto:auth_path=/hiiqos-56/?prefix=a 2023-07-29T00:21:32.721 DEBUG:boto:Method: GET 2023-07-29T00:21:32.721 DEBUG:boto:Path: /hiiqos-56/?prefix=a 2023-07-29T00:21:32.722 DEBUG:boto:Data: 2023-07-29T00:21:32.722 DEBUG:boto:Headers: {} 2023-07-29T00:21:32.722 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:32.723 DEBUG:boto:Port: 8001 2023-07-29T00:21:32.723 DEBUG:boto:Params: {} 2023-07-29T00:21:32.723 DEBUG:boto:Token: None 2023-07-29T00:21:32.724 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:32 GMT /hiiqos-56/ 2023-07-29T00:21:32.724 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MbA4bkqS3ZaSkI+WR3OfIFnpI0U= 2023-07-29T00:21:32.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:32 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MbA4bkqS3ZaSkI+WR3OfIFnpI0U=', 'Content-Length': '0'} 2023-07-29T00:21:32.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e06aaee376586ef1-0064c45b8c-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:21:32 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:32.728 DEBUG:boto:b'hiiqos-56a1000falsea2023-07-29T00:18:00.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:21:32.728 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:21:32.729 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:21:32.729 DEBUG:boto:path=/hiiqos-56/?prefix=a 2023-07-29T00:21:32.729 DEBUG:boto:auth_path=/hiiqos-56/?prefix=a 2023-07-29T00:21:32.730 DEBUG:boto:Method: GET 2023-07-29T00:21:32.730 DEBUG:boto:Path: /hiiqos-56/?prefix=a 2023-07-29T00:21:32.730 DEBUG:boto:Data: 2023-07-29T00:21:32.731 DEBUG:boto:Headers: {} 2023-07-29T00:21:32.731 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:32.731 DEBUG:boto:Port: 8000 2023-07-29T00:21:32.732 DEBUG:boto:Params: {} 2023-07-29T00:21:32.732 DEBUG:boto:Token: None 2023-07-29T00:21:32.733 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:32 GMT /hiiqos-56/ 2023-07-29T00:21:32.733 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MbA4bkqS3ZaSkI+WR3OfIFnpI0U= 2023-07-29T00:21:32.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:32 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MbA4bkqS3ZaSkI+WR3OfIFnpI0U=', 'Content-Length': '0'} 2023-07-29T00:21:32.736 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000198c9c8e56c2888c-0064c45b8c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:21:32 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:32.736 DEBUG:boto:b'hiiqos-56a1000falsea2023-07-29T00:18:00.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:21:32.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-07-29T00:21:32.737 DEBUG:boto:path=/hiiqos-56/b 2023-07-29T00:21:32.737 DEBUG:boto:auth_path=/hiiqos-56/b 2023-07-29T00:21:32.738 DEBUG:boto:path=/hiiqos-56/b?versionId=null 2023-07-29T00:21:32.738 DEBUG:boto:auth_path=/hiiqos-56/b?versionId=null 2023-07-29T00:21:32.739 DEBUG:boto:Method: GET 2023-07-29T00:21:32.739 DEBUG:boto:Path: /hiiqos-56/b?versionId=null 2023-07-29T00:21:32.739 DEBUG:boto:Data: 2023-07-29T00:21:32.740 DEBUG:boto:Headers: {} 2023-07-29T00:21:32.740 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:32.741 DEBUG:boto:Port: 8001 2023-07-29T00:21:32.741 DEBUG:boto:Params: {} 2023-07-29T00:21:32.741 DEBUG:boto:Token: None 2023-07-29T00:21:32.742 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:32 GMT /hiiqos-56/b?versionId=null 2023-07-29T00:21:32.742 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9ORLhP8tYxnDKcQpzrVDB5Wd7z4= 2023-07-29T00:21:32.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:32 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9ORLhP8tYxnDKcQpzrVDB5Wd7z4=', 'Content-Length': '0'} 2023-07-29T00:21:32.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe79c0dfdce837e9-0064c45b8c-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:32 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:32.745 DEBUG:boto:path=/hiiqos-56/b 2023-07-29T00:21:32.746 DEBUG:boto:auth_path=/hiiqos-56/b 2023-07-29T00:21:32.746 DEBUG:boto:path=/hiiqos-56/b?versionId=null 2023-07-29T00:21:32.747 DEBUG:boto:auth_path=/hiiqos-56/b?versionId=null 2023-07-29T00:21:32.747 DEBUG:boto:Method: GET 2023-07-29T00:21:32.747 DEBUG:boto:Path: /hiiqos-56/b?versionId=null 2023-07-29T00:21:32.748 DEBUG:boto:Data: 2023-07-29T00:21:32.748 DEBUG:boto:Headers: {} 2023-07-29T00:21:32.749 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:32.749 DEBUG:boto:Port: 8000 2023-07-29T00:21:32.749 DEBUG:boto:Params: {} 2023-07-29T00:21:32.750 DEBUG:boto:Token: None 2023-07-29T00:21:32.750 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:32 GMT /hiiqos-56/b?versionId=null 2023-07-29T00:21:32.751 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9ORLhP8tYxnDKcQpzrVDB5Wd7z4= 2023-07-29T00:21:32.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:32 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9ORLhP8tYxnDKcQpzrVDB5Wd7z4=', 'Content-Length': '0'} 2023-07-29T00:21:32.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af43daa8e28470d1-0064c45b8c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:32 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:32.754 DEBUG:boto:path=/hiiqos-56/b 2023-07-29T00:21:32.754 DEBUG:boto:auth_path=/hiiqos-56/b 2023-07-29T00:21:32.755 DEBUG:boto:path=/hiiqos-56/b?versionId=null 2023-07-29T00:21:32.755 DEBUG:boto:auth_path=/hiiqos-56/b?versionId=null 2023-07-29T00:21:32.755 DEBUG:boto:Method: HEAD 2023-07-29T00:21:32.756 DEBUG:boto:Path: /hiiqos-56/b?versionId=null 2023-07-29T00:21:32.756 DEBUG:boto:Data: 2023-07-29T00:21:32.757 DEBUG:boto:Headers: {} 2023-07-29T00:21:32.757 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:32.757 DEBUG:boto:Port: 8001 2023-07-29T00:21:32.758 DEBUG:boto:Params: {} 2023-07-29T00:21:32.758 DEBUG:boto:Token: None 2023-07-29T00:21:32.759 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:21:32 GMT /hiiqos-56/b?versionId=null 2023-07-29T00:21:32.759 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8ColYv88Kls0yDqg6NygCwNaAQ8= 2023-07-29T00:21:32.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:32 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8ColYv88Kls0yDqg6NygCwNaAQ8=', 'Content-Length': '0'} 2023-07-29T00:21:32.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015d520535f266aae-0064c45b8c-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:32 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:32.762 DEBUG:boto:path=/hiiqos-56/b 2023-07-29T00:21:32.762 DEBUG:boto:auth_path=/hiiqos-56/b 2023-07-29T00:21:32.763 DEBUG:boto:path=/hiiqos-56/b?versionId=null 2023-07-29T00:21:32.763 DEBUG:boto:auth_path=/hiiqos-56/b?versionId=null 2023-07-29T00:21:32.764 DEBUG:boto:Method: HEAD 2023-07-29T00:21:32.764 DEBUG:boto:Path: /hiiqos-56/b?versionId=null 2023-07-29T00:21:32.764 DEBUG:boto:Data: 2023-07-29T00:21:32.765 DEBUG:boto:Headers: {} 2023-07-29T00:21:32.765 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:32.766 DEBUG:boto:Port: 8000 2023-07-29T00:21:32.766 DEBUG:boto:Params: {} 2023-07-29T00:21:32.766 DEBUG:boto:Token: None 2023-07-29T00:21:32.767 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:21:32 GMT /hiiqos-56/b?versionId=null 2023-07-29T00:21:32.767 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8ColYv88Kls0yDqg6NygCwNaAQ8= 2023-07-29T00:21:32.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:32 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8ColYv88Kls0yDqg6NygCwNaAQ8=', 'Content-Length': '0'} 2023-07-29T00:21:32.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d508f8c62be8e7e3-0064c45b8c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:32 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:32.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-07-29T00:21:32.771 DEBUG:boto:path=/hiiqos-56/b 2023-07-29T00:21:32.771 DEBUG:boto:auth_path=/hiiqos-56/b 2023-07-29T00:21:32.772 DEBUG:boto:path=/hiiqos-56/b?versionId=null 2023-07-29T00:21:32.772 DEBUG:boto:auth_path=/hiiqos-56/b?versionId=null 2023-07-29T00:21:32.772 DEBUG:boto:Method: GET 2023-07-29T00:21:32.773 DEBUG:boto:Path: /hiiqos-56/b?versionId=null 2023-07-29T00:21:32.773 DEBUG:boto:Data: 2023-07-29T00:21:32.774 DEBUG:boto:Headers: {} 2023-07-29T00:21:32.774 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:32.774 DEBUG:boto:Port: 8001 2023-07-29T00:21:32.775 DEBUG:boto:Params: {} 2023-07-29T00:21:32.775 DEBUG:boto:Token: None 2023-07-29T00:21:32.776 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:32 GMT /hiiqos-56/b?versionId=null 2023-07-29T00:21:32.776 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9ORLhP8tYxnDKcQpzrVDB5Wd7z4= 2023-07-29T00:21:32.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:32 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9ORLhP8tYxnDKcQpzrVDB5Wd7z4=', 'Content-Length': '0'} 2023-07-29T00:21:32.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fdd63545fd345d9-0064c45b8c-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:32 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:32.779 DEBUG:boto:path=/hiiqos-56/b 2023-07-29T00:21:32.780 DEBUG:boto:auth_path=/hiiqos-56/b 2023-07-29T00:21:32.780 DEBUG:boto:path=/hiiqos-56/b?versionId=null 2023-07-29T00:21:32.781 DEBUG:boto:auth_path=/hiiqos-56/b?versionId=null 2023-07-29T00:21:32.781 DEBUG:boto:Method: GET 2023-07-29T00:21:32.782 DEBUG:boto:Path: /hiiqos-56/b?versionId=null 2023-07-29T00:21:32.782 DEBUG:boto:Data: 2023-07-29T00:21:32.782 DEBUG:boto:Headers: {} 2023-07-29T00:21:32.783 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:32.783 DEBUG:boto:Port: 8000 2023-07-29T00:21:32.784 DEBUG:boto:Params: {} 2023-07-29T00:21:32.784 DEBUG:boto:Token: None 2023-07-29T00:21:32.785 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:32 GMT /hiiqos-56/b?versionId=null 2023-07-29T00:21:32.785 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9ORLhP8tYxnDKcQpzrVDB5Wd7z4= 2023-07-29T00:21:32.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:32 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9ORLhP8tYxnDKcQpzrVDB5Wd7z4=', 'Content-Length': '0'} 2023-07-29T00:21:32.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c68347790f88f016-0064c45b8c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:32 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:32.828 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:21:32.829 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:21:32.829 DEBUG:boto:path=/hiiqos-56/?prefix=b 2023-07-29T00:21:32.830 DEBUG:boto:auth_path=/hiiqos-56/?prefix=b 2023-07-29T00:21:32.830 DEBUG:boto:Method: GET 2023-07-29T00:21:32.830 DEBUG:boto:Path: /hiiqos-56/?prefix=b 2023-07-29T00:21:32.831 DEBUG:boto:Data: 2023-07-29T00:21:32.831 DEBUG:boto:Headers: {} 2023-07-29T00:21:32.831 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:32.832 DEBUG:boto:Port: 8001 2023-07-29T00:21:32.832 DEBUG:boto:Params: {} 2023-07-29T00:21:32.833 DEBUG:boto:Token: None 2023-07-29T00:21:32.833 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:32 GMT /hiiqos-56/ 2023-07-29T00:21:32.833 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MbA4bkqS3ZaSkI+WR3OfIFnpI0U= 2023-07-29T00:21:32.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:32 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MbA4bkqS3ZaSkI+WR3OfIFnpI0U=', 'Content-Length': '0'} 2023-07-29T00:21:32.836 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f43bac992296c0d-0064c45b8c-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:21:32 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:32.837 DEBUG:boto:b'hiiqos-56b1000falseb2023-07-29T00:18:00.841Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:21:32.838 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:21:32.838 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:21:32.838 DEBUG:boto:path=/hiiqos-56/?prefix=b 2023-07-29T00:21:32.839 DEBUG:boto:auth_path=/hiiqos-56/?prefix=b 2023-07-29T00:21:32.839 DEBUG:boto:Method: GET 2023-07-29T00:21:32.839 DEBUG:boto:Path: /hiiqos-56/?prefix=b 2023-07-29T00:21:32.840 DEBUG:boto:Data: 2023-07-29T00:21:32.840 DEBUG:boto:Headers: {} 2023-07-29T00:21:32.841 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:32.841 DEBUG:boto:Port: 8000 2023-07-29T00:21:32.841 DEBUG:boto:Params: {} 2023-07-29T00:21:32.842 DEBUG:boto:Token: None 2023-07-29T00:21:32.842 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:32 GMT /hiiqos-56/ 2023-07-29T00:21:32.842 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MbA4bkqS3ZaSkI+WR3OfIFnpI0U= 2023-07-29T00:21:32.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:32 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MbA4bkqS3ZaSkI+WR3OfIFnpI0U=', 'Content-Length': '0'} 2023-07-29T00:21:32.845 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000717a3d055a9130bb-0064c45b8c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:21:32 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:32.846 DEBUG:boto:b'hiiqos-56b1000falseb2023-07-29T00:18:00.841Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:21:32.846 DEBUG:boto:path=/hiiqos-56/b 2023-07-29T00:21:32.847 DEBUG:boto:auth_path=/hiiqos-56/b 2023-07-29T00:21:32.847 DEBUG:boto:Method: HEAD 2023-07-29T00:21:32.847 DEBUG:boto:Path: /hiiqos-56/b 2023-07-29T00:21:32.848 DEBUG:boto:Data: 2023-07-29T00:21:32.848 DEBUG:boto:Headers: {} 2023-07-29T00:21:32.848 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:32.849 DEBUG:boto:Port: 8001 2023-07-29T00:21:32.849 DEBUG:boto:Params: {} 2023-07-29T00:21:32.849 DEBUG:boto:Token: None 2023-07-29T00:21:32.850 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:21:32 GMT /hiiqos-56/b 2023-07-29T00:21:32.850 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7rtg3b2pFIJRKAXep/SAxUWUTX4= 2023-07-29T00:21:32.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:32 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7rtg3b2pFIJRKAXep/SAxUWUTX4=', 'Content-Length': '0'} 2023-07-29T00:21:32.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000830d38f7a5fb7002-0064c45b8c-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:32 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:32.853 DEBUG:boto:path=/hiiqos-56/b 2023-07-29T00:21:32.854 DEBUG:boto:auth_path=/hiiqos-56/b 2023-07-29T00:21:32.854 DEBUG:boto:Method: HEAD 2023-07-29T00:21:32.855 DEBUG:boto:Path: /hiiqos-56/b 2023-07-29T00:21:32.855 DEBUG:boto:Data: 2023-07-29T00:21:32.856 DEBUG:boto:Headers: {} 2023-07-29T00:21:32.856 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:32.856 DEBUG:boto:Port: 8000 2023-07-29T00:21:32.857 DEBUG:boto:Params: {} 2023-07-29T00:21:32.858 DEBUG:boto:Token: None 2023-07-29T00:21:32.858 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:21:32 GMT /hiiqos-56/b 2023-07-29T00:21:32.859 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7rtg3b2pFIJRKAXep/SAxUWUTX4= 2023-07-29T00:21:32.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:32 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7rtg3b2pFIJRKAXep/SAxUWUTX4=', 'Content-Length': '0'} 2023-07-29T00:21:32.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000096447fe07d414b4-0064c45b8c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:32 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:32.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-07-29T00:21:32.863 DEBUG:boto:path=/hiiqos-56/b 2023-07-29T00:21:32.863 DEBUG:boto:auth_path=/hiiqos-56/b 2023-07-29T00:21:32.864 DEBUG:boto:Method: GET 2023-07-29T00:21:32.865 DEBUG:boto:Path: /hiiqos-56/b 2023-07-29T00:21:32.866 DEBUG:boto:Data: 2023-07-29T00:21:32.866 DEBUG:boto:Headers: {} 2023-07-29T00:21:32.867 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:32.867 DEBUG:boto:Port: 8001 2023-07-29T00:21:32.868 DEBUG:boto:Params: {} 2023-07-29T00:21:32.868 DEBUG:boto:Token: None 2023-07-29T00:21:32.869 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:32 GMT /hiiqos-56/b 2023-07-29T00:21:32.870 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RJ0JREHbNvqsUHSDMmfUaX6dL5U= 2023-07-29T00:21:32.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:32 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RJ0JREHbNvqsUHSDMmfUaX6dL5U=', 'Content-Length': '0'} 2023-07-29T00:21:32.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080056f7b49a07969-0064c45b8c-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:32 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:32.916 DEBUG:boto:path=/hiiqos-56/b 2023-07-29T00:21:32.917 DEBUG:boto:auth_path=/hiiqos-56/b 2023-07-29T00:21:32.918 DEBUG:boto:Method: GET 2023-07-29T00:21:32.918 DEBUG:boto:Path: /hiiqos-56/b 2023-07-29T00:21:32.919 DEBUG:boto:Data: 2023-07-29T00:21:32.919 DEBUG:boto:Headers: {} 2023-07-29T00:21:32.919 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:32.920 DEBUG:boto:Port: 8000 2023-07-29T00:21:32.920 DEBUG:boto:Params: {} 2023-07-29T00:21:32.921 DEBUG:boto:Token: None 2023-07-29T00:21:32.921 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:32 GMT /hiiqos-56/b 2023-07-29T00:21:32.922 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RJ0JREHbNvqsUHSDMmfUaX6dL5U= 2023-07-29T00:21:32.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:32 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RJ0JREHbNvqsUHSDMmfUaX6dL5U=', 'Content-Length': '0'} 2023-07-29T00:21:32.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfe665cf1eb612a2-0064c45b8c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:32 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:32.925 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:21:32.926 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:21:32.926 DEBUG:boto:path=/hiiqos-56/?prefix=b 2023-07-29T00:21:32.926 DEBUG:boto:auth_path=/hiiqos-56/?prefix=b 2023-07-29T00:21:32.927 DEBUG:boto:Method: GET 2023-07-29T00:21:32.927 DEBUG:boto:Path: /hiiqos-56/?prefix=b 2023-07-29T00:21:32.927 DEBUG:boto:Data: 2023-07-29T00:21:32.928 DEBUG:boto:Headers: {} 2023-07-29T00:21:32.928 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:32.928 DEBUG:boto:Port: 8001 2023-07-29T00:21:32.929 DEBUG:boto:Params: {} 2023-07-29T00:21:32.929 DEBUG:boto:Token: None 2023-07-29T00:21:32.930 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:32 GMT /hiiqos-56/ 2023-07-29T00:21:32.930 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MbA4bkqS3ZaSkI+WR3OfIFnpI0U= 2023-07-29T00:21:32.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:32 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MbA4bkqS3ZaSkI+WR3OfIFnpI0U=', 'Content-Length': '0'} 2023-07-29T00:21:32.933 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac2098356f521db6-0064c45b8c-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:21:32 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:32.934 DEBUG:boto:b'hiiqos-56b1000falseb2023-07-29T00:18:00.841Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:21:32.934 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:21:32.935 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:21:32.935 DEBUG:boto:path=/hiiqos-56/?prefix=b 2023-07-29T00:21:32.935 DEBUG:boto:auth_path=/hiiqos-56/?prefix=b 2023-07-29T00:21:32.936 DEBUG:boto:Method: GET 2023-07-29T00:21:32.936 DEBUG:boto:Path: /hiiqos-56/?prefix=b 2023-07-29T00:21:32.936 DEBUG:boto:Data: 2023-07-29T00:21:32.937 DEBUG:boto:Headers: {} 2023-07-29T00:21:32.937 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:32.938 DEBUG:boto:Port: 8000 2023-07-29T00:21:32.938 DEBUG:boto:Params: {} 2023-07-29T00:21:32.938 DEBUG:boto:Token: None 2023-07-29T00:21:32.939 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:32 GMT /hiiqos-56/ 2023-07-29T00:21:32.939 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MbA4bkqS3ZaSkI+WR3OfIFnpI0U= 2023-07-29T00:21:32.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:32 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MbA4bkqS3ZaSkI+WR3OfIFnpI0U=', 'Content-Length': '0'} 2023-07-29T00:21:32.941 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d3d7ce70ab20811-0064c45b8c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:21:32 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:32.984 DEBUG:boto:b'hiiqos-56b1000falseb2023-07-29T00:18:00.841Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:21:32.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-07-29T00:21:32.986 DEBUG:boto:path=/hiiqos-56/c 2023-07-29T00:21:32.986 DEBUG:boto:auth_path=/hiiqos-56/c 2023-07-29T00:21:32.986 DEBUG:boto:path=/hiiqos-56/c?versionId=null 2023-07-29T00:21:32.987 DEBUG:boto:auth_path=/hiiqos-56/c?versionId=null 2023-07-29T00:21:32.987 DEBUG:boto:Method: GET 2023-07-29T00:21:32.987 DEBUG:boto:Path: /hiiqos-56/c?versionId=null 2023-07-29T00:21:32.988 DEBUG:boto:Data: 2023-07-29T00:21:32.988 DEBUG:boto:Headers: {} 2023-07-29T00:21:32.988 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:32.989 DEBUG:boto:Port: 8001 2023-07-29T00:21:32.989 DEBUG:boto:Params: {} 2023-07-29T00:21:32.990 DEBUG:boto:Token: None 2023-07-29T00:21:32.990 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:32 GMT /hiiqos-56/c?versionId=null 2023-07-29T00:21:32.990 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Idg/6ux0u4gWhvfN7LKvp6F/yyQ= 2023-07-29T00:21:32.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:32 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Idg/6ux0u4gWhvfN7LKvp6F/yyQ=', 'Content-Length': '0'} 2023-07-29T00:21:32.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b27366b122eb0665-0064c45b8c-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:32 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:32.993 DEBUG:boto:path=/hiiqos-56/c 2023-07-29T00:21:32.994 DEBUG:boto:auth_path=/hiiqos-56/c 2023-07-29T00:21:32.994 DEBUG:boto:path=/hiiqos-56/c?versionId=null 2023-07-29T00:21:32.995 DEBUG:boto:auth_path=/hiiqos-56/c?versionId=null 2023-07-29T00:21:32.995 DEBUG:boto:Method: GET 2023-07-29T00:21:32.996 DEBUG:boto:Path: /hiiqos-56/c?versionId=null 2023-07-29T00:21:32.996 DEBUG:boto:Data: 2023-07-29T00:21:32.996 DEBUG:boto:Headers: {} 2023-07-29T00:21:32.997 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:32.999 DEBUG:boto:Port: 8000 2023-07-29T00:21:33.000 DEBUG:boto:Params: {} 2023-07-29T00:21:33.001 DEBUG:boto:Token: None 2023-07-29T00:21:33.001 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/c?versionId=null 2023-07-29T00:21:33.002 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/BHR1ZOTJ5hct5S0sCKPdtCmS7s= 2023-07-29T00:21:33.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/BHR1ZOTJ5hct5S0sCKPdtCmS7s=', 'Content-Length': '0'} 2023-07-29T00:21:33.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b96dae24b9ea1ed-0064c45b8c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:32 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.007 DEBUG:boto:path=/hiiqos-56/c 2023-07-29T00:21:33.007 DEBUG:boto:auth_path=/hiiqos-56/c 2023-07-29T00:21:33.008 DEBUG:boto:path=/hiiqos-56/c?versionId=null 2023-07-29T00:21:33.008 DEBUG:boto:auth_path=/hiiqos-56/c?versionId=null 2023-07-29T00:21:33.009 DEBUG:boto:Method: HEAD 2023-07-29T00:21:33.010 DEBUG:boto:Path: /hiiqos-56/c?versionId=null 2023-07-29T00:21:33.011 DEBUG:boto:Data: 2023-07-29T00:21:33.011 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.012 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:33.013 DEBUG:boto:Port: 8001 2023-07-29T00:21:33.013 DEBUG:boto:Params: {} 2023-07-29T00:21:33.014 DEBUG:boto:Token: None 2023-07-29T00:21:33.015 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/c?versionId=null 2023-07-29T00:21:33.015 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:suzgDiyKMeOCywkCme8mR0AaVig= 2023-07-29T00:21:33.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:suzgDiyKMeOCywkCme8mR0AaVig=', 'Content-Length': '0'} 2023-07-29T00:21:33.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f38152ea10d3818c-0064c45b8c-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:32 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.018 DEBUG:boto:path=/hiiqos-56/c 2023-07-29T00:21:33.019 DEBUG:boto:auth_path=/hiiqos-56/c 2023-07-29T00:21:33.019 DEBUG:boto:path=/hiiqos-56/c?versionId=null 2023-07-29T00:21:33.020 DEBUG:boto:auth_path=/hiiqos-56/c?versionId=null 2023-07-29T00:21:33.020 DEBUG:boto:Method: HEAD 2023-07-29T00:21:33.021 DEBUG:boto:Path: /hiiqos-56/c?versionId=null 2023-07-29T00:21:33.022 DEBUG:boto:Data: 2023-07-29T00:21:33.022 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.023 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:33.024 DEBUG:boto:Port: 8000 2023-07-29T00:21:33.025 DEBUG:boto:Params: {} 2023-07-29T00:21:33.026 DEBUG:boto:Token: None 2023-07-29T00:21:33.027 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/c?versionId=null 2023-07-29T00:21:33.027 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:suzgDiyKMeOCywkCme8mR0AaVig= 2023-07-29T00:21:33.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:suzgDiyKMeOCywkCme8mR0AaVig=', 'Content-Length': '0'} 2023-07-29T00:21:33.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef28462dbd915919-0064c45b8c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:32 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-07-29T00:21:33.033 DEBUG:boto:path=/hiiqos-56/c 2023-07-29T00:21:33.034 DEBUG:boto:auth_path=/hiiqos-56/c 2023-07-29T00:21:33.035 DEBUG:boto:path=/hiiqos-56/c?versionId=null 2023-07-29T00:21:33.035 DEBUG:boto:auth_path=/hiiqos-56/c?versionId=null 2023-07-29T00:21:33.036 DEBUG:boto:Method: GET 2023-07-29T00:21:33.036 DEBUG:boto:Path: /hiiqos-56/c?versionId=null 2023-07-29T00:21:33.036 DEBUG:boto:Data: 2023-07-29T00:21:33.037 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.037 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:33.037 DEBUG:boto:Port: 8001 2023-07-29T00:21:33.038 DEBUG:boto:Params: {} 2023-07-29T00:21:33.039 DEBUG:boto:Token: None 2023-07-29T00:21:33.040 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/c?versionId=null 2023-07-29T00:21:33.041 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/BHR1ZOTJ5hct5S0sCKPdtCmS7s= 2023-07-29T00:21:33.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/BHR1ZOTJ5hct5S0sCKPdtCmS7s=', 'Content-Length': '0'} 2023-07-29T00:21:33.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bf5289e88a944d4-0064c45b8c-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:32 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.045 DEBUG:boto:path=/hiiqos-56/c 2023-07-29T00:21:33.046 DEBUG:boto:auth_path=/hiiqos-56/c 2023-07-29T00:21:33.046 DEBUG:boto:path=/hiiqos-56/c?versionId=null 2023-07-29T00:21:33.046 DEBUG:boto:auth_path=/hiiqos-56/c?versionId=null 2023-07-29T00:21:33.047 DEBUG:boto:Method: GET 2023-07-29T00:21:33.047 DEBUG:boto:Path: /hiiqos-56/c?versionId=null 2023-07-29T00:21:33.047 DEBUG:boto:Data: 2023-07-29T00:21:33.048 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.048 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:33.049 DEBUG:boto:Port: 8000 2023-07-29T00:21:33.049 DEBUG:boto:Params: {} 2023-07-29T00:21:33.049 DEBUG:boto:Token: None 2023-07-29T00:21:33.050 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/c?versionId=null 2023-07-29T00:21:33.050 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/BHR1ZOTJ5hct5S0sCKPdtCmS7s= 2023-07-29T00:21:33.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/BHR1ZOTJ5hct5S0sCKPdtCmS7s=', 'Content-Length': '0'} 2023-07-29T00:21:33.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000705af67206694e71-0064c45b8d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.053 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:21:33.053 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:21:33.054 DEBUG:boto:path=/hiiqos-56/?prefix=c 2023-07-29T00:21:33.054 DEBUG:boto:auth_path=/hiiqos-56/?prefix=c 2023-07-29T00:21:33.055 DEBUG:boto:Method: GET 2023-07-29T00:21:33.055 DEBUG:boto:Path: /hiiqos-56/?prefix=c 2023-07-29T00:21:33.056 DEBUG:boto:Data: 2023-07-29T00:21:33.057 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.058 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:33.059 DEBUG:boto:Port: 8001 2023-07-29T00:21:33.060 DEBUG:boto:Params: {} 2023-07-29T00:21:33.061 DEBUG:boto:Token: None 2023-07-29T00:21:33.061 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/ 2023-07-29T00:21:33.062 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YZogNM5YkcSQXyEAlbgfE0GcTX4= 2023-07-29T00:21:33.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YZogNM5YkcSQXyEAlbgfE0GcTX4=', 'Content-Length': '0'} 2023-07-29T00:21:33.065 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1e8c28248d46906-0064c45b8d-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.112 DEBUG:boto:b'hiiqos-56c1000falsec2023-07-29T00:18:00.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:21:33.113 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:21:33.114 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:21:33.114 DEBUG:boto:path=/hiiqos-56/?prefix=c 2023-07-29T00:21:33.114 DEBUG:boto:auth_path=/hiiqos-56/?prefix=c 2023-07-29T00:21:33.115 DEBUG:boto:Method: GET 2023-07-29T00:21:33.115 DEBUG:boto:Path: /hiiqos-56/?prefix=c 2023-07-29T00:21:33.116 DEBUG:boto:Data: 2023-07-29T00:21:33.116 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.116 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:33.117 DEBUG:boto:Port: 8000 2023-07-29T00:21:33.117 DEBUG:boto:Params: {} 2023-07-29T00:21:33.117 DEBUG:boto:Token: None 2023-07-29T00:21:33.118 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/ 2023-07-29T00:21:33.118 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YZogNM5YkcSQXyEAlbgfE0GcTX4= 2023-07-29T00:21:33.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YZogNM5YkcSQXyEAlbgfE0GcTX4=', 'Content-Length': '0'} 2023-07-29T00:21:33.121 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064d9775d6ef30937-0064c45b8d-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.121 DEBUG:boto:b'hiiqos-56c1000falsec2023-07-29T00:18:00.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:21:33.122 DEBUG:boto:path=/hiiqos-56/c 2023-07-29T00:21:33.122 DEBUG:boto:auth_path=/hiiqos-56/c 2023-07-29T00:21:33.123 DEBUG:boto:Method: HEAD 2023-07-29T00:21:33.123 DEBUG:boto:Path: /hiiqos-56/c 2023-07-29T00:21:33.123 DEBUG:boto:Data: 2023-07-29T00:21:33.124 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.124 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:33.124 DEBUG:boto:Port: 8001 2023-07-29T00:21:33.125 DEBUG:boto:Params: {} 2023-07-29T00:21:33.125 DEBUG:boto:Token: None 2023-07-29T00:21:33.126 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/c 2023-07-29T00:21:33.126 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SU73jNm2ARvtFIWT/d5awpVw9zg= 2023-07-29T00:21:33.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SU73jNm2ARvtFIWT/d5awpVw9zg=', 'Content-Length': '0'} 2023-07-29T00:21:33.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056312553104fa1ff-0064c45b8d-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.129 DEBUG:boto:path=/hiiqos-56/c 2023-07-29T00:21:33.129 DEBUG:boto:auth_path=/hiiqos-56/c 2023-07-29T00:21:33.129 DEBUG:boto:Method: HEAD 2023-07-29T00:21:33.130 DEBUG:boto:Path: /hiiqos-56/c 2023-07-29T00:21:33.130 DEBUG:boto:Data: 2023-07-29T00:21:33.130 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.131 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:33.131 DEBUG:boto:Port: 8000 2023-07-29T00:21:33.131 DEBUG:boto:Params: {} 2023-07-29T00:21:33.132 DEBUG:boto:Token: None 2023-07-29T00:21:33.132 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/c 2023-07-29T00:21:33.133 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SU73jNm2ARvtFIWT/d5awpVw9zg= 2023-07-29T00:21:33.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SU73jNm2ARvtFIWT/d5awpVw9zg=', 'Content-Length': '0'} 2023-07-29T00:21:33.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b437b7d89b31b420-0064c45b8d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-07-29T00:21:33.136 DEBUG:boto:path=/hiiqos-56/c 2023-07-29T00:21:33.136 DEBUG:boto:auth_path=/hiiqos-56/c 2023-07-29T00:21:33.137 DEBUG:boto:Method: GET 2023-07-29T00:21:33.137 DEBUG:boto:Path: /hiiqos-56/c 2023-07-29T00:21:33.138 DEBUG:boto:Data: 2023-07-29T00:21:33.138 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.138 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:33.139 DEBUG:boto:Port: 8001 2023-07-29T00:21:33.139 DEBUG:boto:Params: {} 2023-07-29T00:21:33.140 DEBUG:boto:Token: None 2023-07-29T00:21:33.140 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/c 2023-07-29T00:21:33.140 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:h0dW/q26vVLzEhfH1YgN1O11ayk= 2023-07-29T00:21:33.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:h0dW/q26vVLzEhfH1YgN1O11ayk=', 'Content-Length': '0'} 2023-07-29T00:21:33.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011a3185f8e9e3dd8-0064c45b8d-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.143 DEBUG:boto:path=/hiiqos-56/c 2023-07-29T00:21:33.144 DEBUG:boto:auth_path=/hiiqos-56/c 2023-07-29T00:21:33.144 DEBUG:boto:Method: GET 2023-07-29T00:21:33.144 DEBUG:boto:Path: /hiiqos-56/c 2023-07-29T00:21:33.145 DEBUG:boto:Data: 2023-07-29T00:21:33.145 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.146 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:33.146 DEBUG:boto:Port: 8000 2023-07-29T00:21:33.146 DEBUG:boto:Params: {} 2023-07-29T00:21:33.147 DEBUG:boto:Token: None 2023-07-29T00:21:33.147 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/c 2023-07-29T00:21:33.147 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:h0dW/q26vVLzEhfH1YgN1O11ayk= 2023-07-29T00:21:33.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:h0dW/q26vVLzEhfH1YgN1O11ayk=', 'Content-Length': '0'} 2023-07-29T00:21:33.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a496f7575437ba29-0064c45b8d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.192 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:21:33.193 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:21:33.193 DEBUG:boto:path=/hiiqos-56/?prefix=c 2023-07-29T00:21:33.194 DEBUG:boto:auth_path=/hiiqos-56/?prefix=c 2023-07-29T00:21:33.194 DEBUG:boto:Method: GET 2023-07-29T00:21:33.195 DEBUG:boto:Path: /hiiqos-56/?prefix=c 2023-07-29T00:21:33.195 DEBUG:boto:Data: 2023-07-29T00:21:33.196 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.196 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:33.196 DEBUG:boto:Port: 8001 2023-07-29T00:21:33.197 DEBUG:boto:Params: {} 2023-07-29T00:21:33.197 DEBUG:boto:Token: None 2023-07-29T00:21:33.198 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/ 2023-07-29T00:21:33.198 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YZogNM5YkcSQXyEAlbgfE0GcTX4= 2023-07-29T00:21:33.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YZogNM5YkcSQXyEAlbgfE0GcTX4=', 'Content-Length': '0'} 2023-07-29T00:21:33.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090f8430df215ec4a-0064c45b8d-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.201 DEBUG:boto:b'hiiqos-56c1000falsec2023-07-29T00:18:00.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:21:33.202 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:21:33.202 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:21:33.203 DEBUG:boto:path=/hiiqos-56/?prefix=c 2023-07-29T00:21:33.203 DEBUG:boto:auth_path=/hiiqos-56/?prefix=c 2023-07-29T00:21:33.204 DEBUG:boto:Method: GET 2023-07-29T00:21:33.204 DEBUG:boto:Path: /hiiqos-56/?prefix=c 2023-07-29T00:21:33.204 DEBUG:boto:Data: 2023-07-29T00:21:33.205 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.205 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:33.205 DEBUG:boto:Port: 8000 2023-07-29T00:21:33.206 DEBUG:boto:Params: {} 2023-07-29T00:21:33.206 DEBUG:boto:Token: None 2023-07-29T00:21:33.207 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/ 2023-07-29T00:21:33.207 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YZogNM5YkcSQXyEAlbgfE0GcTX4= 2023-07-29T00:21:33.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YZogNM5YkcSQXyEAlbgfE0GcTX4=', 'Content-Length': '0'} 2023-07-29T00:21:33.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000beb7335ae03f3a1e-0064c45b8d-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.210 DEBUG:boto:b'hiiqos-56c1000falsec2023-07-29T00:18:00.858Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:21:33.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-07-29T00:21:33.211 DEBUG:boto:path=/hiiqos-56/d 2023-07-29T00:21:33.212 DEBUG:boto:auth_path=/hiiqos-56/d 2023-07-29T00:21:33.212 DEBUG:boto:path=/hiiqos-56/d?versionId=null 2023-07-29T00:21:33.213 DEBUG:boto:auth_path=/hiiqos-56/d?versionId=null 2023-07-29T00:21:33.213 DEBUG:boto:Method: GET 2023-07-29T00:21:33.214 DEBUG:boto:Path: /hiiqos-56/d?versionId=null 2023-07-29T00:21:33.215 DEBUG:boto:Data: 2023-07-29T00:21:33.215 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.216 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:33.217 DEBUG:boto:Port: 8001 2023-07-29T00:21:33.217 DEBUG:boto:Params: {} 2023-07-29T00:21:33.218 DEBUG:boto:Token: None 2023-07-29T00:21:33.219 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/d?versionId=null 2023-07-29T00:21:33.219 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MSZhjLyuHT6loziLvLmhkxzT2r8= 2023-07-29T00:21:33.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MSZhjLyuHT6loziLvLmhkxzT2r8=', 'Content-Length': '0'} 2023-07-29T00:21:33.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a60dad047f800851-0064c45b8d-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.268 DEBUG:boto:path=/hiiqos-56/d 2023-07-29T00:21:33.269 DEBUG:boto:auth_path=/hiiqos-56/d 2023-07-29T00:21:33.269 DEBUG:boto:path=/hiiqos-56/d?versionId=null 2023-07-29T00:21:33.270 DEBUG:boto:auth_path=/hiiqos-56/d?versionId=null 2023-07-29T00:21:33.270 DEBUG:boto:Method: GET 2023-07-29T00:21:33.270 DEBUG:boto:Path: /hiiqos-56/d?versionId=null 2023-07-29T00:21:33.271 DEBUG:boto:Data: 2023-07-29T00:21:33.271 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.272 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:33.273 DEBUG:boto:Port: 8000 2023-07-29T00:21:33.274 DEBUG:boto:Params: {} 2023-07-29T00:21:33.275 DEBUG:boto:Token: None 2023-07-29T00:21:33.276 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/d?versionId=null 2023-07-29T00:21:33.277 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MSZhjLyuHT6loziLvLmhkxzT2r8= 2023-07-29T00:21:33.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MSZhjLyuHT6loziLvLmhkxzT2r8=', 'Content-Length': '0'} 2023-07-29T00:21:33.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f576de86c847290-0064c45b8d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.280 DEBUG:boto:path=/hiiqos-56/d 2023-07-29T00:21:33.281 DEBUG:boto:auth_path=/hiiqos-56/d 2023-07-29T00:21:33.281 DEBUG:boto:path=/hiiqos-56/d?versionId=null 2023-07-29T00:21:33.281 DEBUG:boto:auth_path=/hiiqos-56/d?versionId=null 2023-07-29T00:21:33.282 DEBUG:boto:Method: HEAD 2023-07-29T00:21:33.282 DEBUG:boto:Path: /hiiqos-56/d?versionId=null 2023-07-29T00:21:33.282 DEBUG:boto:Data: 2023-07-29T00:21:33.283 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.283 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:33.284 DEBUG:boto:Port: 8001 2023-07-29T00:21:33.284 DEBUG:boto:Params: {} 2023-07-29T00:21:33.284 DEBUG:boto:Token: None 2023-07-29T00:21:33.285 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/d?versionId=null 2023-07-29T00:21:33.285 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pA2N8jPeaI6JcWU7H8hU/6P64fc= 2023-07-29T00:21:33.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pA2N8jPeaI6JcWU7H8hU/6P64fc=', 'Content-Length': '0'} 2023-07-29T00:21:33.288 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d7e0a8465d3347a-0064c45b8d-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.289 DEBUG:boto:path=/hiiqos-56/d 2023-07-29T00:21:33.289 DEBUG:boto:auth_path=/hiiqos-56/d 2023-07-29T00:21:33.290 DEBUG:boto:path=/hiiqos-56/d?versionId=null 2023-07-29T00:21:33.290 DEBUG:boto:auth_path=/hiiqos-56/d?versionId=null 2023-07-29T00:21:33.290 DEBUG:boto:Method: HEAD 2023-07-29T00:21:33.291 DEBUG:boto:Path: /hiiqos-56/d?versionId=null 2023-07-29T00:21:33.291 DEBUG:boto:Data: 2023-07-29T00:21:33.291 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.292 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:33.292 DEBUG:boto:Port: 8000 2023-07-29T00:21:33.292 DEBUG:boto:Params: {} 2023-07-29T00:21:33.293 DEBUG:boto:Token: None 2023-07-29T00:21:33.293 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/d?versionId=null 2023-07-29T00:21:33.294 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pA2N8jPeaI6JcWU7H8hU/6P64fc= 2023-07-29T00:21:33.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pA2N8jPeaI6JcWU7H8hU/6P64fc=', 'Content-Length': '0'} 2023-07-29T00:21:33.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005369021ed8756d3-0064c45b8d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-07-29T00:21:33.297 DEBUG:boto:path=/hiiqos-56/d 2023-07-29T00:21:33.297 DEBUG:boto:auth_path=/hiiqos-56/d 2023-07-29T00:21:33.298 DEBUG:boto:path=/hiiqos-56/d?versionId=null 2023-07-29T00:21:33.298 DEBUG:boto:auth_path=/hiiqos-56/d?versionId=null 2023-07-29T00:21:33.299 DEBUG:boto:Method: GET 2023-07-29T00:21:33.299 DEBUG:boto:Path: /hiiqos-56/d?versionId=null 2023-07-29T00:21:33.300 DEBUG:boto:Data: 2023-07-29T00:21:33.300 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.301 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:33.301 DEBUG:boto:Port: 8001 2023-07-29T00:21:33.301 DEBUG:boto:Params: {} 2023-07-29T00:21:33.302 DEBUG:boto:Token: None 2023-07-29T00:21:33.303 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/d?versionId=null 2023-07-29T00:21:33.303 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MSZhjLyuHT6loziLvLmhkxzT2r8= 2023-07-29T00:21:33.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MSZhjLyuHT6loziLvLmhkxzT2r8=', 'Content-Length': '0'} 2023-07-29T00:21:33.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9dce233219d3dc2-0064c45b8d-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.306 DEBUG:boto:path=/hiiqos-56/d 2023-07-29T00:21:33.307 DEBUG:boto:auth_path=/hiiqos-56/d 2023-07-29T00:21:33.307 DEBUG:boto:path=/hiiqos-56/d?versionId=null 2023-07-29T00:21:33.308 DEBUG:boto:auth_path=/hiiqos-56/d?versionId=null 2023-07-29T00:21:33.308 DEBUG:boto:Method: GET 2023-07-29T00:21:33.308 DEBUG:boto:Path: /hiiqos-56/d?versionId=null 2023-07-29T00:21:33.309 DEBUG:boto:Data: 2023-07-29T00:21:33.309 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.310 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:33.310 DEBUG:boto:Port: 8000 2023-07-29T00:21:33.310 DEBUG:boto:Params: {} 2023-07-29T00:21:33.311 DEBUG:boto:Token: None 2023-07-29T00:21:33.311 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/d?versionId=null 2023-07-29T00:21:33.312 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MSZhjLyuHT6loziLvLmhkxzT2r8= 2023-07-29T00:21:33.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MSZhjLyuHT6loziLvLmhkxzT2r8=', 'Content-Length': '0'} 2023-07-29T00:21:33.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fed4e35899e4df3-0064c45b8d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.317 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:21:33.318 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:21:33.318 DEBUG:boto:path=/hiiqos-56/?prefix=d 2023-07-29T00:21:33.318 DEBUG:boto:auth_path=/hiiqos-56/?prefix=d 2023-07-29T00:21:33.319 DEBUG:boto:Method: GET 2023-07-29T00:21:33.319 DEBUG:boto:Path: /hiiqos-56/?prefix=d 2023-07-29T00:21:33.320 DEBUG:boto:Data: 2023-07-29T00:21:33.320 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.320 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:33.321 DEBUG:boto:Port: 8001 2023-07-29T00:21:33.321 DEBUG:boto:Params: {} 2023-07-29T00:21:33.321 DEBUG:boto:Token: None 2023-07-29T00:21:33.322 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/ 2023-07-29T00:21:33.322 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YZogNM5YkcSQXyEAlbgfE0GcTX4= 2023-07-29T00:21:33.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YZogNM5YkcSQXyEAlbgfE0GcTX4=', 'Content-Length': '0'} 2023-07-29T00:21:33.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092e4448dc512dd43-0064c45b8d-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.325 DEBUG:boto:b'hiiqos-56d1000falsed2023-07-29T00:18:00.914Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:21:33.326 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:21:33.327 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:21:33.327 DEBUG:boto:path=/hiiqos-56/?prefix=d 2023-07-29T00:21:33.328 DEBUG:boto:auth_path=/hiiqos-56/?prefix=d 2023-07-29T00:21:33.328 DEBUG:boto:Method: GET 2023-07-29T00:21:33.329 DEBUG:boto:Path: /hiiqos-56/?prefix=d 2023-07-29T00:21:33.329 DEBUG:boto:Data: 2023-07-29T00:21:33.329 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.330 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:33.330 DEBUG:boto:Port: 8000 2023-07-29T00:21:33.330 DEBUG:boto:Params: {} 2023-07-29T00:21:33.331 DEBUG:boto:Token: None 2023-07-29T00:21:33.331 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/ 2023-07-29T00:21:33.332 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YZogNM5YkcSQXyEAlbgfE0GcTX4= 2023-07-29T00:21:33.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YZogNM5YkcSQXyEAlbgfE0GcTX4=', 'Content-Length': '0'} 2023-07-29T00:21:33.336 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e3d61ae26bcf5cc-0064c45b8d-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.337 DEBUG:boto:b'hiiqos-56d1000falsed2023-07-29T00:18:00.914Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:21:33.338 DEBUG:boto:path=/hiiqos-56/d 2023-07-29T00:21:33.339 DEBUG:boto:auth_path=/hiiqos-56/d 2023-07-29T00:21:33.339 DEBUG:boto:Method: HEAD 2023-07-29T00:21:33.340 DEBUG:boto:Path: /hiiqos-56/d 2023-07-29T00:21:33.341 DEBUG:boto:Data: 2023-07-29T00:21:33.341 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.342 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:33.342 DEBUG:boto:Port: 8001 2023-07-29T00:21:33.343 DEBUG:boto:Params: {} 2023-07-29T00:21:33.343 DEBUG:boto:Token: None 2023-07-29T00:21:33.343 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/d 2023-07-29T00:21:33.344 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YzKQ92gM/1/czVE1G0IUViJZSaA= 2023-07-29T00:21:33.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YzKQ92gM/1/czVE1G0IUViJZSaA=', 'Content-Length': '0'} 2023-07-29T00:21:33.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cfb0e03b8bf2d11-0064c45b8d-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.347 DEBUG:boto:path=/hiiqos-56/d 2023-07-29T00:21:33.347 DEBUG:boto:auth_path=/hiiqos-56/d 2023-07-29T00:21:33.348 DEBUG:boto:Method: HEAD 2023-07-29T00:21:33.348 DEBUG:boto:Path: /hiiqos-56/d 2023-07-29T00:21:33.348 DEBUG:boto:Data: 2023-07-29T00:21:33.349 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.349 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:33.349 DEBUG:boto:Port: 8000 2023-07-29T00:21:33.350 DEBUG:boto:Params: {} 2023-07-29T00:21:33.350 DEBUG:boto:Token: None 2023-07-29T00:21:33.350 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/d 2023-07-29T00:21:33.351 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YzKQ92gM/1/czVE1G0IUViJZSaA= 2023-07-29T00:21:33.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YzKQ92gM/1/czVE1G0IUViJZSaA=', 'Content-Length': '0'} 2023-07-29T00:21:33.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000aecdff35f12a3be-0064c45b8d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-07-29T00:21:33.354 DEBUG:boto:path=/hiiqos-56/d 2023-07-29T00:21:33.355 DEBUG:boto:auth_path=/hiiqos-56/d 2023-07-29T00:21:33.355 DEBUG:boto:Method: GET 2023-07-29T00:21:33.356 DEBUG:boto:Path: /hiiqos-56/d 2023-07-29T00:21:33.356 DEBUG:boto:Data: 2023-07-29T00:21:33.356 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.357 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:33.357 DEBUG:boto:Port: 8001 2023-07-29T00:21:33.358 DEBUG:boto:Params: {} 2023-07-29T00:21:33.358 DEBUG:boto:Token: None 2023-07-29T00:21:33.358 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/d 2023-07-29T00:21:33.359 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bXLJwAZSIhjYq6lHsceJq/q8cRM= 2023-07-29T00:21:33.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bXLJwAZSIhjYq6lHsceJq/q8cRM=', 'Content-Length': '0'} 2023-07-29T00:21:33.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3f810e9565ed3ba-0064c45b8d-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.404 DEBUG:boto:path=/hiiqos-56/d 2023-07-29T00:21:33.405 DEBUG:boto:auth_path=/hiiqos-56/d 2023-07-29T00:21:33.405 DEBUG:boto:Method: GET 2023-07-29T00:21:33.406 DEBUG:boto:Path: /hiiqos-56/d 2023-07-29T00:21:33.406 DEBUG:boto:Data: 2023-07-29T00:21:33.406 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.407 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:33.407 DEBUG:boto:Port: 8000 2023-07-29T00:21:33.408 DEBUG:boto:Params: {} 2023-07-29T00:21:33.408 DEBUG:boto:Token: None 2023-07-29T00:21:33.408 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/d 2023-07-29T00:21:33.409 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bXLJwAZSIhjYq6lHsceJq/q8cRM= 2023-07-29T00:21:33.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bXLJwAZSIhjYq6lHsceJq/q8cRM=', 'Content-Length': '0'} 2023-07-29T00:21:33.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009dac7465536c031d-0064c45b8d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.412 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:21:33.413 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:21:33.413 DEBUG:boto:path=/hiiqos-56/?prefix=d 2023-07-29T00:21:33.414 DEBUG:boto:auth_path=/hiiqos-56/?prefix=d 2023-07-29T00:21:33.414 DEBUG:boto:Method: GET 2023-07-29T00:21:33.414 DEBUG:boto:Path: /hiiqos-56/?prefix=d 2023-07-29T00:21:33.415 DEBUG:boto:Data: 2023-07-29T00:21:33.415 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.415 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:33.416 DEBUG:boto:Port: 8001 2023-07-29T00:21:33.416 DEBUG:boto:Params: {} 2023-07-29T00:21:33.417 DEBUG:boto:Token: None 2023-07-29T00:21:33.417 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/ 2023-07-29T00:21:33.417 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YZogNM5YkcSQXyEAlbgfE0GcTX4= 2023-07-29T00:21:33.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YZogNM5YkcSQXyEAlbgfE0GcTX4=', 'Content-Length': '0'} 2023-07-29T00:21:33.420 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cddadf028187368-0064c45b8d-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.421 DEBUG:boto:b'hiiqos-56d1000falsed2023-07-29T00:18:00.914Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:21:33.422 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:21:33.422 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:21:33.422 DEBUG:boto:path=/hiiqos-56/?prefix=d 2023-07-29T00:21:33.423 DEBUG:boto:auth_path=/hiiqos-56/?prefix=d 2023-07-29T00:21:33.423 DEBUG:boto:Method: GET 2023-07-29T00:21:33.423 DEBUG:boto:Path: /hiiqos-56/?prefix=d 2023-07-29T00:21:33.424 DEBUG:boto:Data: 2023-07-29T00:21:33.424 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.425 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:33.425 DEBUG:boto:Port: 8000 2023-07-29T00:21:33.425 DEBUG:boto:Params: {} 2023-07-29T00:21:33.426 DEBUG:boto:Token: None 2023-07-29T00:21:33.426 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/ 2023-07-29T00:21:33.427 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YZogNM5YkcSQXyEAlbgfE0GcTX4= 2023-07-29T00:21:33.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YZogNM5YkcSQXyEAlbgfE0GcTX4=', 'Content-Length': '0'} 2023-07-29T00:21:33.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d9f3496a7efefb0-0064c45b8d-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.429 DEBUG:boto:b'hiiqos-56d1000falsed2023-07-29T00:18:00.914Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:21:33.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-07-29T00:21:33.431 DEBUG:boto:path=/hiiqos-56/e 2023-07-29T00:21:33.431 DEBUG:boto:auth_path=/hiiqos-56/e 2023-07-29T00:21:33.431 DEBUG:boto:path=/hiiqos-56/e?versionId=null 2023-07-29T00:21:33.432 DEBUG:boto:auth_path=/hiiqos-56/e?versionId=null 2023-07-29T00:21:33.432 DEBUG:boto:Method: GET 2023-07-29T00:21:33.432 DEBUG:boto:Path: /hiiqos-56/e?versionId=null 2023-07-29T00:21:33.433 DEBUG:boto:Data: 2023-07-29T00:21:33.433 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.433 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:33.434 DEBUG:boto:Port: 8001 2023-07-29T00:21:33.434 DEBUG:boto:Params: {} 2023-07-29T00:21:33.435 DEBUG:boto:Token: None 2023-07-29T00:21:33.435 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/e?versionId=null 2023-07-29T00:21:33.435 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fevbA8LV5gJ4WAbuA2RyEJMwG+U= 2023-07-29T00:21:33.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fevbA8LV5gJ4WAbuA2RyEJMwG+U=', 'Content-Length': '0'} 2023-07-29T00:21:33.438 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d299317ac49da710-0064c45b8d-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.439 DEBUG:boto:path=/hiiqos-56/e 2023-07-29T00:21:33.439 DEBUG:boto:auth_path=/hiiqos-56/e 2023-07-29T00:21:33.440 DEBUG:boto:path=/hiiqos-56/e?versionId=null 2023-07-29T00:21:33.440 DEBUG:boto:auth_path=/hiiqos-56/e?versionId=null 2023-07-29T00:21:33.440 DEBUG:boto:Method: GET 2023-07-29T00:21:33.441 DEBUG:boto:Path: /hiiqos-56/e?versionId=null 2023-07-29T00:21:33.441 DEBUG:boto:Data: 2023-07-29T00:21:33.441 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.442 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:33.442 DEBUG:boto:Port: 8000 2023-07-29T00:21:33.442 DEBUG:boto:Params: {} 2023-07-29T00:21:33.443 DEBUG:boto:Token: None 2023-07-29T00:21:33.443 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/e?versionId=null 2023-07-29T00:21:33.444 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fevbA8LV5gJ4WAbuA2RyEJMwG+U= 2023-07-29T00:21:33.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fevbA8LV5gJ4WAbuA2RyEJMwG+U=', 'Content-Length': '0'} 2023-07-29T00:21:33.446 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5dcf20681bb074b-0064c45b8d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.447 DEBUG:boto:path=/hiiqos-56/e 2023-07-29T00:21:33.447 DEBUG:boto:auth_path=/hiiqos-56/e 2023-07-29T00:21:33.448 DEBUG:boto:path=/hiiqos-56/e?versionId=null 2023-07-29T00:21:33.448 DEBUG:boto:auth_path=/hiiqos-56/e?versionId=null 2023-07-29T00:21:33.448 DEBUG:boto:Method: HEAD 2023-07-29T00:21:33.449 DEBUG:boto:Path: /hiiqos-56/e?versionId=null 2023-07-29T00:21:33.449 DEBUG:boto:Data: 2023-07-29T00:21:33.449 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.450 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:33.450 DEBUG:boto:Port: 8001 2023-07-29T00:21:33.450 DEBUG:boto:Params: {} 2023-07-29T00:21:33.451 DEBUG:boto:Token: None 2023-07-29T00:21:33.451 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/e?versionId=null 2023-07-29T00:21:33.452 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HTrLV3ix//+QLq7dUs5NeIkvDPU= 2023-07-29T00:21:33.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HTrLV3ix//+QLq7dUs5NeIkvDPU=', 'Content-Length': '0'} 2023-07-29T00:21:33.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000699ac57f39cec846-0064c45b8d-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.455 DEBUG:boto:path=/hiiqos-56/e 2023-07-29T00:21:33.455 DEBUG:boto:auth_path=/hiiqos-56/e 2023-07-29T00:21:33.455 DEBUG:boto:path=/hiiqos-56/e?versionId=null 2023-07-29T00:21:33.456 DEBUG:boto:auth_path=/hiiqos-56/e?versionId=null 2023-07-29T00:21:33.456 DEBUG:boto:Method: HEAD 2023-07-29T00:21:33.456 DEBUG:boto:Path: /hiiqos-56/e?versionId=null 2023-07-29T00:21:33.457 DEBUG:boto:Data: 2023-07-29T00:21:33.457 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.457 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:33.458 DEBUG:boto:Port: 8000 2023-07-29T00:21:33.458 DEBUG:boto:Params: {} 2023-07-29T00:21:33.458 DEBUG:boto:Token: None 2023-07-29T00:21:33.459 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/e?versionId=null 2023-07-29T00:21:33.459 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HTrLV3ix//+QLq7dUs5NeIkvDPU= 2023-07-29T00:21:33.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HTrLV3ix//+QLq7dUs5NeIkvDPU=', 'Content-Length': '0'} 2023-07-29T00:21:33.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000358e5f80560716e0-0064c45b8d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-07-29T00:21:33.462 DEBUG:boto:path=/hiiqos-56/e 2023-07-29T00:21:33.463 DEBUG:boto:auth_path=/hiiqos-56/e 2023-07-29T00:21:33.463 DEBUG:boto:path=/hiiqos-56/e?versionId=null 2023-07-29T00:21:33.464 DEBUG:boto:auth_path=/hiiqos-56/e?versionId=null 2023-07-29T00:21:33.464 DEBUG:boto:Method: GET 2023-07-29T00:21:33.464 DEBUG:boto:Path: /hiiqos-56/e?versionId=null 2023-07-29T00:21:33.465 DEBUG:boto:Data: 2023-07-29T00:21:33.465 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.465 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:33.466 DEBUG:boto:Port: 8001 2023-07-29T00:21:33.466 DEBUG:boto:Params: {} 2023-07-29T00:21:33.466 DEBUG:boto:Token: None 2023-07-29T00:21:33.467 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/e?versionId=null 2023-07-29T00:21:33.467 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fevbA8LV5gJ4WAbuA2RyEJMwG+U= 2023-07-29T00:21:33.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fevbA8LV5gJ4WAbuA2RyEJMwG+U=', 'Content-Length': '0'} 2023-07-29T00:21:33.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f1fc84462aba4ef-0064c45b8d-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.470 DEBUG:boto:path=/hiiqos-56/e 2023-07-29T00:21:33.471 DEBUG:boto:auth_path=/hiiqos-56/e 2023-07-29T00:21:33.471 DEBUG:boto:path=/hiiqos-56/e?versionId=null 2023-07-29T00:21:33.471 DEBUG:boto:auth_path=/hiiqos-56/e?versionId=null 2023-07-29T00:21:33.472 DEBUG:boto:Method: GET 2023-07-29T00:21:33.472 DEBUG:boto:Path: /hiiqos-56/e?versionId=null 2023-07-29T00:21:33.472 DEBUG:boto:Data: 2023-07-29T00:21:33.473 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.473 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:33.474 DEBUG:boto:Port: 8000 2023-07-29T00:21:33.474 DEBUG:boto:Params: {} 2023-07-29T00:21:33.474 DEBUG:boto:Token: None 2023-07-29T00:21:33.475 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/e?versionId=null 2023-07-29T00:21:33.475 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fevbA8LV5gJ4WAbuA2RyEJMwG+U= 2023-07-29T00:21:33.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fevbA8LV5gJ4WAbuA2RyEJMwG+U=', 'Content-Length': '0'} 2023-07-29T00:21:33.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb8ff575423b9694-0064c45b8d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.478 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:21:33.479 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:21:33.479 DEBUG:boto:path=/hiiqos-56/?prefix=e 2023-07-29T00:21:33.479 DEBUG:boto:auth_path=/hiiqos-56/?prefix=e 2023-07-29T00:21:33.480 DEBUG:boto:Method: GET 2023-07-29T00:21:33.480 DEBUG:boto:Path: /hiiqos-56/?prefix=e 2023-07-29T00:21:33.481 DEBUG:boto:Data: 2023-07-29T00:21:33.481 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.481 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:33.482 DEBUG:boto:Port: 8001 2023-07-29T00:21:33.482 DEBUG:boto:Params: {} 2023-07-29T00:21:33.482 DEBUG:boto:Token: None 2023-07-29T00:21:33.483 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/ 2023-07-29T00:21:33.483 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YZogNM5YkcSQXyEAlbgfE0GcTX4= 2023-07-29T00:21:33.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YZogNM5YkcSQXyEAlbgfE0GcTX4=', 'Content-Length': '0'} 2023-07-29T00:21:33.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c40dd729c13a3f0-0064c45b8d-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.487 DEBUG:boto:b'hiiqos-56e1000falsee2023-07-29T00:18:22.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:21:33.487 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:21:33.488 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:21:33.488 DEBUG:boto:path=/hiiqos-56/?prefix=e 2023-07-29T00:21:33.488 DEBUG:boto:auth_path=/hiiqos-56/?prefix=e 2023-07-29T00:21:33.489 DEBUG:boto:Method: GET 2023-07-29T00:21:33.489 DEBUG:boto:Path: /hiiqos-56/?prefix=e 2023-07-29T00:21:33.490 DEBUG:boto:Data: 2023-07-29T00:21:33.490 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.490 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:33.491 DEBUG:boto:Port: 8000 2023-07-29T00:21:33.491 DEBUG:boto:Params: {} 2023-07-29T00:21:33.491 DEBUG:boto:Token: None 2023-07-29T00:21:33.492 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/ 2023-07-29T00:21:33.492 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YZogNM5YkcSQXyEAlbgfE0GcTX4= 2023-07-29T00:21:33.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YZogNM5YkcSQXyEAlbgfE0GcTX4=', 'Content-Length': '0'} 2023-07-29T00:21:33.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a2a84c7d23e1ce5-0064c45b8d-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.495 DEBUG:boto:b'hiiqos-56e1000falsee2023-07-29T00:18:22.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:21:33.496 DEBUG:boto:path=/hiiqos-56/e 2023-07-29T00:21:33.496 DEBUG:boto:auth_path=/hiiqos-56/e 2023-07-29T00:21:33.496 DEBUG:boto:Method: HEAD 2023-07-29T00:21:33.497 DEBUG:boto:Path: /hiiqos-56/e 2023-07-29T00:21:33.497 DEBUG:boto:Data: 2023-07-29T00:21:33.497 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.498 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:33.498 DEBUG:boto:Port: 8001 2023-07-29T00:21:33.498 DEBUG:boto:Params: {} 2023-07-29T00:21:33.499 DEBUG:boto:Token: None 2023-07-29T00:21:33.499 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/e 2023-07-29T00:21:33.500 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:V27o0O02MhyIykTCukpnwRyQp/Q= 2023-07-29T00:21:33.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:V27o0O02MhyIykTCukpnwRyQp/Q=', 'Content-Length': '0'} 2023-07-29T00:21:33.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037fd741e61afe90b-0064c45b8d-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.502 DEBUG:boto:path=/hiiqos-56/e 2023-07-29T00:21:33.503 DEBUG:boto:auth_path=/hiiqos-56/e 2023-07-29T00:21:33.503 DEBUG:boto:Method: HEAD 2023-07-29T00:21:33.503 DEBUG:boto:Path: /hiiqos-56/e 2023-07-29T00:21:33.504 DEBUG:boto:Data: 2023-07-29T00:21:33.504 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.504 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:33.505 DEBUG:boto:Port: 8000 2023-07-29T00:21:33.505 DEBUG:boto:Params: {} 2023-07-29T00:21:33.505 DEBUG:boto:Token: None 2023-07-29T00:21:33.506 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/e 2023-07-29T00:21:33.506 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:V27o0O02MhyIykTCukpnwRyQp/Q= 2023-07-29T00:21:33.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:V27o0O02MhyIykTCukpnwRyQp/Q=', 'Content-Length': '0'} 2023-07-29T00:21:33.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001de4d0f93b1474ef-0064c45b8d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-07-29T00:21:33.509 DEBUG:boto:path=/hiiqos-56/e 2023-07-29T00:21:33.509 DEBUG:boto:auth_path=/hiiqos-56/e 2023-07-29T00:21:33.510 DEBUG:boto:Method: GET 2023-07-29T00:21:33.510 DEBUG:boto:Path: /hiiqos-56/e 2023-07-29T00:21:33.511 DEBUG:boto:Data: 2023-07-29T00:21:33.511 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.511 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:33.512 DEBUG:boto:Port: 8001 2023-07-29T00:21:33.512 DEBUG:boto:Params: {} 2023-07-29T00:21:33.512 DEBUG:boto:Token: None 2023-07-29T00:21:33.513 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/e 2023-07-29T00:21:33.513 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:d9tesdxB7fq5/Q/GpeMP4LQXvnM= 2023-07-29T00:21:33.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:d9tesdxB7fq5/Q/GpeMP4LQXvnM=', 'Content-Length': '0'} 2023-07-29T00:21:33.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dece00d1d1fb27bb-0064c45b8d-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.516 DEBUG:boto:path=/hiiqos-56/e 2023-07-29T00:21:33.516 DEBUG:boto:auth_path=/hiiqos-56/e 2023-07-29T00:21:33.517 DEBUG:boto:Method: GET 2023-07-29T00:21:33.517 DEBUG:boto:Path: /hiiqos-56/e 2023-07-29T00:21:33.518 DEBUG:boto:Data: 2023-07-29T00:21:33.518 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.518 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:33.519 DEBUG:boto:Port: 8000 2023-07-29T00:21:33.519 DEBUG:boto:Params: {} 2023-07-29T00:21:33.519 DEBUG:boto:Token: None 2023-07-29T00:21:33.520 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/e 2023-07-29T00:21:33.520 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:d9tesdxB7fq5/Q/GpeMP4LQXvnM= 2023-07-29T00:21:33.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:d9tesdxB7fq5/Q/GpeMP4LQXvnM=', 'Content-Length': '0'} 2023-07-29T00:21:33.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ca87f1f01c40831-0064c45b8d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.564 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:21:33.565 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:21:33.565 DEBUG:boto:path=/hiiqos-56/?prefix=e 2023-07-29T00:21:33.566 DEBUG:boto:auth_path=/hiiqos-56/?prefix=e 2023-07-29T00:21:33.566 DEBUG:boto:Method: GET 2023-07-29T00:21:33.566 DEBUG:boto:Path: /hiiqos-56/?prefix=e 2023-07-29T00:21:33.567 DEBUG:boto:Data: 2023-07-29T00:21:33.567 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.567 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:33.568 DEBUG:boto:Port: 8001 2023-07-29T00:21:33.568 DEBUG:boto:Params: {} 2023-07-29T00:21:33.569 DEBUG:boto:Token: None 2023-07-29T00:21:33.569 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/ 2023-07-29T00:21:33.569 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YZogNM5YkcSQXyEAlbgfE0GcTX4= 2023-07-29T00:21:33.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YZogNM5YkcSQXyEAlbgfE0GcTX4=', 'Content-Length': '0'} 2023-07-29T00:21:33.572 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000247f46d42e099e03-0064c45b8d-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.573 DEBUG:boto:b'hiiqos-56e1000falsee2023-07-29T00:18:22.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:21:33.573 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:21:33.574 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:21:33.574 DEBUG:boto:path=/hiiqos-56/?prefix=e 2023-07-29T00:21:33.575 DEBUG:boto:auth_path=/hiiqos-56/?prefix=e 2023-07-29T00:21:33.575 DEBUG:boto:Method: GET 2023-07-29T00:21:33.575 DEBUG:boto:Path: /hiiqos-56/?prefix=e 2023-07-29T00:21:33.576 DEBUG:boto:Data: 2023-07-29T00:21:33.576 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.576 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:33.577 DEBUG:boto:Port: 8000 2023-07-29T00:21:33.577 DEBUG:boto:Params: {} 2023-07-29T00:21:33.577 DEBUG:boto:Token: None 2023-07-29T00:21:33.578 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/ 2023-07-29T00:21:33.578 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YZogNM5YkcSQXyEAlbgfE0GcTX4= 2023-07-29T00:21:33.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YZogNM5YkcSQXyEAlbgfE0GcTX4=', 'Content-Length': '0'} 2023-07-29T00:21:33.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3b4cdd7f79bcdf5-0064c45b8d-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.581 DEBUG:boto:b'hiiqos-56e1000falsee2023-07-29T00:18:22.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:21:33.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-07-29T00:21:33.582 DEBUG:boto:path=/hiiqos-56/f 2023-07-29T00:21:33.583 DEBUG:boto:auth_path=/hiiqos-56/f 2023-07-29T00:21:33.583 DEBUG:boto:path=/hiiqos-56/f?versionId=null 2023-07-29T00:21:33.583 DEBUG:boto:auth_path=/hiiqos-56/f?versionId=null 2023-07-29T00:21:33.584 DEBUG:boto:Method: GET 2023-07-29T00:21:33.584 DEBUG:boto:Path: /hiiqos-56/f?versionId=null 2023-07-29T00:21:33.585 DEBUG:boto:Data: 2023-07-29T00:21:33.585 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.585 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:33.586 DEBUG:boto:Port: 8001 2023-07-29T00:21:33.586 DEBUG:boto:Params: {} 2023-07-29T00:21:33.586 DEBUG:boto:Token: None 2023-07-29T00:21:33.587 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/f?versionId=null 2023-07-29T00:21:33.587 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9LqUmJ3vKJIHH62hm2vNnq1USj0= 2023-07-29T00:21:33.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9LqUmJ3vKJIHH62hm2vNnq1USj0=', 'Content-Length': '0'} 2023-07-29T00:21:33.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000248f7abefd49e102-0064c45b8d-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.632 DEBUG:boto:path=/hiiqos-56/f 2023-07-29T00:21:33.633 DEBUG:boto:auth_path=/hiiqos-56/f 2023-07-29T00:21:33.633 DEBUG:boto:path=/hiiqos-56/f?versionId=null 2023-07-29T00:21:33.633 DEBUG:boto:auth_path=/hiiqos-56/f?versionId=null 2023-07-29T00:21:33.634 DEBUG:boto:Method: GET 2023-07-29T00:21:33.634 DEBUG:boto:Path: /hiiqos-56/f?versionId=null 2023-07-29T00:21:33.635 DEBUG:boto:Data: 2023-07-29T00:21:33.635 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.635 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:33.636 DEBUG:boto:Port: 8000 2023-07-29T00:21:33.636 DEBUG:boto:Params: {} 2023-07-29T00:21:33.636 DEBUG:boto:Token: None 2023-07-29T00:21:33.637 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/f?versionId=null 2023-07-29T00:21:33.637 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9LqUmJ3vKJIHH62hm2vNnq1USj0= 2023-07-29T00:21:33.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9LqUmJ3vKJIHH62hm2vNnq1USj0=', 'Content-Length': '0'} 2023-07-29T00:21:33.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008abe9720226f691b-0064c45b8d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.640 DEBUG:boto:path=/hiiqos-56/f 2023-07-29T00:21:33.641 DEBUG:boto:auth_path=/hiiqos-56/f 2023-07-29T00:21:33.641 DEBUG:boto:path=/hiiqos-56/f?versionId=null 2023-07-29T00:21:33.642 DEBUG:boto:auth_path=/hiiqos-56/f?versionId=null 2023-07-29T00:21:33.642 DEBUG:boto:Method: HEAD 2023-07-29T00:21:33.642 DEBUG:boto:Path: /hiiqos-56/f?versionId=null 2023-07-29T00:21:33.643 DEBUG:boto:Data: 2023-07-29T00:21:33.643 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.643 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:33.644 DEBUG:boto:Port: 8001 2023-07-29T00:21:33.644 DEBUG:boto:Params: {} 2023-07-29T00:21:33.645 DEBUG:boto:Token: None 2023-07-29T00:21:33.645 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/f?versionId=null 2023-07-29T00:21:33.646 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:chUYoEvMvU1ky5Q4poI4EblbVwM= 2023-07-29T00:21:33.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:chUYoEvMvU1ky5Q4poI4EblbVwM=', 'Content-Length': '0'} 2023-07-29T00:21:33.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ded6fae26c89e209-0064c45b8d-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.649 DEBUG:boto:path=/hiiqos-56/f 2023-07-29T00:21:33.649 DEBUG:boto:auth_path=/hiiqos-56/f 2023-07-29T00:21:33.649 DEBUG:boto:path=/hiiqos-56/f?versionId=null 2023-07-29T00:21:33.650 DEBUG:boto:auth_path=/hiiqos-56/f?versionId=null 2023-07-29T00:21:33.650 DEBUG:boto:Method: HEAD 2023-07-29T00:21:33.651 DEBUG:boto:Path: /hiiqos-56/f?versionId=null 2023-07-29T00:21:33.651 DEBUG:boto:Data: 2023-07-29T00:21:33.651 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.652 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:33.652 DEBUG:boto:Port: 8000 2023-07-29T00:21:33.652 DEBUG:boto:Params: {} 2023-07-29T00:21:33.653 DEBUG:boto:Token: None 2023-07-29T00:21:33.653 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/f?versionId=null 2023-07-29T00:21:33.654 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:chUYoEvMvU1ky5Q4poI4EblbVwM= 2023-07-29T00:21:33.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:chUYoEvMvU1ky5Q4poI4EblbVwM=', 'Content-Length': '0'} 2023-07-29T00:21:33.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c1d3ecf9798f69d-0064c45b8d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-07-29T00:21:33.657 DEBUG:boto:path=/hiiqos-56/f 2023-07-29T00:21:33.657 DEBUG:boto:auth_path=/hiiqos-56/f 2023-07-29T00:21:33.658 DEBUG:boto:path=/hiiqos-56/f?versionId=null 2023-07-29T00:21:33.658 DEBUG:boto:auth_path=/hiiqos-56/f?versionId=null 2023-07-29T00:21:33.659 DEBUG:boto:Method: GET 2023-07-29T00:21:33.659 DEBUG:boto:Path: /hiiqos-56/f?versionId=null 2023-07-29T00:21:33.659 DEBUG:boto:Data: 2023-07-29T00:21:33.660 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.660 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:33.660 DEBUG:boto:Port: 8001 2023-07-29T00:21:33.661 DEBUG:boto:Params: {} 2023-07-29T00:21:33.661 DEBUG:boto:Token: None 2023-07-29T00:21:33.662 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/f?versionId=null 2023-07-29T00:21:33.662 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9LqUmJ3vKJIHH62hm2vNnq1USj0= 2023-07-29T00:21:33.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9LqUmJ3vKJIHH62hm2vNnq1USj0=', 'Content-Length': '0'} 2023-07-29T00:21:33.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ab1030e2bbd65d9-0064c45b8d-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.665 DEBUG:boto:path=/hiiqos-56/f 2023-07-29T00:21:33.665 DEBUG:boto:auth_path=/hiiqos-56/f 2023-07-29T00:21:33.666 DEBUG:boto:path=/hiiqos-56/f?versionId=null 2023-07-29T00:21:33.666 DEBUG:boto:auth_path=/hiiqos-56/f?versionId=null 2023-07-29T00:21:33.666 DEBUG:boto:Method: GET 2023-07-29T00:21:33.667 DEBUG:boto:Path: /hiiqos-56/f?versionId=null 2023-07-29T00:21:33.667 DEBUG:boto:Data: 2023-07-29T00:21:33.668 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.668 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:33.668 DEBUG:boto:Port: 8000 2023-07-29T00:21:33.669 DEBUG:boto:Params: {} 2023-07-29T00:21:33.669 DEBUG:boto:Token: None 2023-07-29T00:21:33.670 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/f?versionId=null 2023-07-29T00:21:33.670 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9LqUmJ3vKJIHH62hm2vNnq1USj0= 2023-07-29T00:21:33.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9LqUmJ3vKJIHH62hm2vNnq1USj0=', 'Content-Length': '0'} 2023-07-29T00:21:33.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059fb31c50d1594d6-0064c45b8d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.675 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:21:33.675 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:21:33.676 DEBUG:boto:path=/hiiqos-56/?prefix=f 2023-07-29T00:21:33.677 DEBUG:boto:auth_path=/hiiqos-56/?prefix=f 2023-07-29T00:21:33.677 DEBUG:boto:Method: GET 2023-07-29T00:21:33.678 DEBUG:boto:Path: /hiiqos-56/?prefix=f 2023-07-29T00:21:33.679 DEBUG:boto:Data: 2023-07-29T00:21:33.679 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.680 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:33.681 DEBUG:boto:Port: 8001 2023-07-29T00:21:33.681 DEBUG:boto:Params: {} 2023-07-29T00:21:33.682 DEBUG:boto:Token: None 2023-07-29T00:21:33.683 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/ 2023-07-29T00:21:33.684 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YZogNM5YkcSQXyEAlbgfE0GcTX4= 2023-07-29T00:21:33.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YZogNM5YkcSQXyEAlbgfE0GcTX4=', 'Content-Length': '0'} 2023-07-29T00:21:33.687 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c1f7f01bf1a21ab-0064c45b8d-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.688 DEBUG:boto:b'hiiqos-56f1000falsef2023-07-29T00:18:22.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:21:33.689 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:21:33.690 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:21:33.691 DEBUG:boto:path=/hiiqos-56/?prefix=f 2023-07-29T00:21:33.692 DEBUG:boto:auth_path=/hiiqos-56/?prefix=f 2023-07-29T00:21:33.693 DEBUG:boto:Method: GET 2023-07-29T00:21:33.693 DEBUG:boto:Path: /hiiqos-56/?prefix=f 2023-07-29T00:21:33.693 DEBUG:boto:Data: 2023-07-29T00:21:33.694 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.694 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:33.694 DEBUG:boto:Port: 8000 2023-07-29T00:21:33.695 DEBUG:boto:Params: {} 2023-07-29T00:21:33.695 DEBUG:boto:Token: None 2023-07-29T00:21:33.696 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/ 2023-07-29T00:21:33.696 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YZogNM5YkcSQXyEAlbgfE0GcTX4= 2023-07-29T00:21:33.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YZogNM5YkcSQXyEAlbgfE0GcTX4=', 'Content-Length': '0'} 2023-07-29T00:21:33.698 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000340b1381b62fe242-0064c45b8d-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.699 DEBUG:boto:b'hiiqos-56f1000falsef2023-07-29T00:18:22.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:21:33.700 DEBUG:boto:path=/hiiqos-56/f 2023-07-29T00:21:33.700 DEBUG:boto:auth_path=/hiiqos-56/f 2023-07-29T00:21:33.701 DEBUG:boto:Method: HEAD 2023-07-29T00:21:33.702 DEBUG:boto:Path: /hiiqos-56/f 2023-07-29T00:21:33.703 DEBUG:boto:Data: 2023-07-29T00:21:33.703 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.704 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:33.705 DEBUG:boto:Port: 8001 2023-07-29T00:21:33.705 DEBUG:boto:Params: {} 2023-07-29T00:21:33.706 DEBUG:boto:Token: None 2023-07-29T00:21:33.706 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/f 2023-07-29T00:21:33.706 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:blebXDa2D6wQCcBCwBeRSeTGBww= 2023-07-29T00:21:33.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:blebXDa2D6wQCcBCwBeRSeTGBww=', 'Content-Length': '0'} 2023-07-29T00:21:33.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb6f62eccdd4ff05-0064c45b8d-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.709 DEBUG:boto:path=/hiiqos-56/f 2023-07-29T00:21:33.710 DEBUG:boto:auth_path=/hiiqos-56/f 2023-07-29T00:21:33.710 DEBUG:boto:Method: HEAD 2023-07-29T00:21:33.711 DEBUG:boto:Path: /hiiqos-56/f 2023-07-29T00:21:33.711 DEBUG:boto:Data: 2023-07-29T00:21:33.712 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.712 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:33.713 DEBUG:boto:Port: 8000 2023-07-29T00:21:33.713 DEBUG:boto:Params: {} 2023-07-29T00:21:33.714 DEBUG:boto:Token: None 2023-07-29T00:21:33.714 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/f 2023-07-29T00:21:33.715 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:blebXDa2D6wQCcBCwBeRSeTGBww= 2023-07-29T00:21:33.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:blebXDa2D6wQCcBCwBeRSeTGBww=', 'Content-Length': '0'} 2023-07-29T00:21:33.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000226d0dd341999e99-0064c45b8d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-07-29T00:21:33.718 DEBUG:boto:path=/hiiqos-56/f 2023-07-29T00:21:33.718 DEBUG:boto:auth_path=/hiiqos-56/f 2023-07-29T00:21:33.719 DEBUG:boto:Method: GET 2023-07-29T00:21:33.719 DEBUG:boto:Path: /hiiqos-56/f 2023-07-29T00:21:33.720 DEBUG:boto:Data: 2023-07-29T00:21:33.720 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.720 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:33.721 DEBUG:boto:Port: 8001 2023-07-29T00:21:33.721 DEBUG:boto:Params: {} 2023-07-29T00:21:33.721 DEBUG:boto:Token: None 2023-07-29T00:21:33.722 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/f 2023-07-29T00:21:33.722 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0qvg/1/gSJammvqDt2c/dpjwtVQ= 2023-07-29T00:21:33.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0qvg/1/gSJammvqDt2c/dpjwtVQ=', 'Content-Length': '0'} 2023-07-29T00:21:33.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000305bb8f76a2ad2f1-0064c45b8d-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.773 DEBUG:boto:path=/hiiqos-56/f 2023-07-29T00:21:33.774 DEBUG:boto:auth_path=/hiiqos-56/f 2023-07-29T00:21:33.774 DEBUG:boto:Method: GET 2023-07-29T00:21:33.775 DEBUG:boto:Path: /hiiqos-56/f 2023-07-29T00:21:33.775 DEBUG:boto:Data: 2023-07-29T00:21:33.776 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.776 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:33.777 DEBUG:boto:Port: 8000 2023-07-29T00:21:33.777 DEBUG:boto:Params: {} 2023-07-29T00:21:33.778 DEBUG:boto:Token: None 2023-07-29T00:21:33.778 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/f 2023-07-29T00:21:33.778 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0qvg/1/gSJammvqDt2c/dpjwtVQ= 2023-07-29T00:21:33.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0qvg/1/gSJammvqDt2c/dpjwtVQ=', 'Content-Length': '0'} 2023-07-29T00:21:33.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034e0064b6bfb2ddb-0064c45b8d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.781 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:21:33.782 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:21:33.782 DEBUG:boto:path=/hiiqos-56/?prefix=f 2023-07-29T00:21:33.782 DEBUG:boto:auth_path=/hiiqos-56/?prefix=f 2023-07-29T00:21:33.783 DEBUG:boto:Method: GET 2023-07-29T00:21:33.783 DEBUG:boto:Path: /hiiqos-56/?prefix=f 2023-07-29T00:21:33.783 DEBUG:boto:Data: 2023-07-29T00:21:33.784 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.784 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:33.785 DEBUG:boto:Port: 8001 2023-07-29T00:21:33.785 DEBUG:boto:Params: {} 2023-07-29T00:21:33.786 DEBUG:boto:Token: None 2023-07-29T00:21:33.787 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/ 2023-07-29T00:21:33.788 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YZogNM5YkcSQXyEAlbgfE0GcTX4= 2023-07-29T00:21:33.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YZogNM5YkcSQXyEAlbgfE0GcTX4=', 'Content-Length': '0'} 2023-07-29T00:21:33.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d882d1740702561a-0064c45b8d-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.792 DEBUG:boto:b'hiiqos-56f1000falsef2023-07-29T00:18:22.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:21:33.793 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:21:33.794 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:21:33.794 DEBUG:boto:path=/hiiqos-56/?prefix=f 2023-07-29T00:21:33.795 DEBUG:boto:auth_path=/hiiqos-56/?prefix=f 2023-07-29T00:21:33.796 DEBUG:boto:Method: GET 2023-07-29T00:21:33.796 DEBUG:boto:Path: /hiiqos-56/?prefix=f 2023-07-29T00:21:33.797 DEBUG:boto:Data: 2023-07-29T00:21:33.798 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.798 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:33.798 DEBUG:boto:Port: 8000 2023-07-29T00:21:33.799 DEBUG:boto:Params: {} 2023-07-29T00:21:33.799 DEBUG:boto:Token: None 2023-07-29T00:21:33.800 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/ 2023-07-29T00:21:33.800 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YZogNM5YkcSQXyEAlbgfE0GcTX4= 2023-07-29T00:21:33.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YZogNM5YkcSQXyEAlbgfE0GcTX4=', 'Content-Length': '0'} 2023-07-29T00:21:33.803 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011feaffb87ace54e-0064c45b8d-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.804 DEBUG:boto:b'hiiqos-56f1000falsef2023-07-29T00:18:22.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:21:33.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-07-29T00:21:33.805 DEBUG:boto:path=/hiiqos-56/g 2023-07-29T00:21:33.806 DEBUG:boto:auth_path=/hiiqos-56/g 2023-07-29T00:21:33.807 DEBUG:boto:path=/hiiqos-56/g?versionId=null 2023-07-29T00:21:33.808 DEBUG:boto:auth_path=/hiiqos-56/g?versionId=null 2023-07-29T00:21:33.809 DEBUG:boto:Method: GET 2023-07-29T00:21:33.809 DEBUG:boto:Path: /hiiqos-56/g?versionId=null 2023-07-29T00:21:33.810 DEBUG:boto:Data: 2023-07-29T00:21:33.810 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.810 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:33.811 DEBUG:boto:Port: 8001 2023-07-29T00:21:33.811 DEBUG:boto:Params: {} 2023-07-29T00:21:33.812 DEBUG:boto:Token: None 2023-07-29T00:21:33.813 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/g?versionId=null 2023-07-29T00:21:33.814 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4/9n3rqg6GiyjaeDy90aLXDCpIE= 2023-07-29T00:21:33.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4/9n3rqg6GiyjaeDy90aLXDCpIE=', 'Content-Length': '0'} 2023-07-29T00:21:33.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b3bbc92f0c62643-0064c45b8d-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.819 DEBUG:boto:path=/hiiqos-56/g 2023-07-29T00:21:33.820 DEBUG:boto:auth_path=/hiiqos-56/g 2023-07-29T00:21:33.820 DEBUG:boto:path=/hiiqos-56/g?versionId=null 2023-07-29T00:21:33.820 DEBUG:boto:auth_path=/hiiqos-56/g?versionId=null 2023-07-29T00:21:33.821 DEBUG:boto:Method: GET 2023-07-29T00:21:33.821 DEBUG:boto:Path: /hiiqos-56/g?versionId=null 2023-07-29T00:21:33.822 DEBUG:boto:Data: 2023-07-29T00:21:33.822 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.822 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:33.823 DEBUG:boto:Port: 8000 2023-07-29T00:21:33.823 DEBUG:boto:Params: {} 2023-07-29T00:21:33.824 DEBUG:boto:Token: None 2023-07-29T00:21:33.824 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/g?versionId=null 2023-07-29T00:21:33.830 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4/9n3rqg6GiyjaeDy90aLXDCpIE= 2023-07-29T00:21:33.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4/9n3rqg6GiyjaeDy90aLXDCpIE=', 'Content-Length': '0'} 2023-07-29T00:21:33.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007880ad972e087824-0064c45b8d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.834 DEBUG:boto:path=/hiiqos-56/g 2023-07-29T00:21:33.834 DEBUG:boto:auth_path=/hiiqos-56/g 2023-07-29T00:21:33.834 DEBUG:boto:path=/hiiqos-56/g?versionId=null 2023-07-29T00:21:33.835 DEBUG:boto:auth_path=/hiiqos-56/g?versionId=null 2023-07-29T00:21:33.835 DEBUG:boto:Method: HEAD 2023-07-29T00:21:33.835 DEBUG:boto:Path: /hiiqos-56/g?versionId=null 2023-07-29T00:21:33.836 DEBUG:boto:Data: 2023-07-29T00:21:33.836 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.836 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:33.837 DEBUG:boto:Port: 8001 2023-07-29T00:21:33.837 DEBUG:boto:Params: {} 2023-07-29T00:21:33.837 DEBUG:boto:Token: None 2023-07-29T00:21:33.838 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/g?versionId=null 2023-07-29T00:21:33.838 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zRaWprB0H0JIGfasPg/DZVyVsA0= 2023-07-29T00:21:33.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zRaWprB0H0JIGfasPg/DZVyVsA0=', 'Content-Length': '0'} 2023-07-29T00:21:33.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae0f6782326e314e-0064c45b8d-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.841 DEBUG:boto:path=/hiiqos-56/g 2023-07-29T00:21:33.841 DEBUG:boto:auth_path=/hiiqos-56/g 2023-07-29T00:21:33.842 DEBUG:boto:path=/hiiqos-56/g?versionId=null 2023-07-29T00:21:33.842 DEBUG:boto:auth_path=/hiiqos-56/g?versionId=null 2023-07-29T00:21:33.842 DEBUG:boto:Method: HEAD 2023-07-29T00:21:33.843 DEBUG:boto:Path: /hiiqos-56/g?versionId=null 2023-07-29T00:21:33.843 DEBUG:boto:Data: 2023-07-29T00:21:33.843 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.844 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:33.844 DEBUG:boto:Port: 8000 2023-07-29T00:21:33.844 DEBUG:boto:Params: {} 2023-07-29T00:21:33.845 DEBUG:boto:Token: None 2023-07-29T00:21:33.845 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/g?versionId=null 2023-07-29T00:21:33.846 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zRaWprB0H0JIGfasPg/DZVyVsA0= 2023-07-29T00:21:33.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zRaWprB0H0JIGfasPg/DZVyVsA0=', 'Content-Length': '0'} 2023-07-29T00:21:33.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088cc0cccf58af21e-0064c45b8d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-07-29T00:21:33.849 DEBUG:boto:path=/hiiqos-56/g 2023-07-29T00:21:33.849 DEBUG:boto:auth_path=/hiiqos-56/g 2023-07-29T00:21:33.849 DEBUG:boto:path=/hiiqos-56/g?versionId=null 2023-07-29T00:21:33.850 DEBUG:boto:auth_path=/hiiqos-56/g?versionId=null 2023-07-29T00:21:33.850 DEBUG:boto:Method: GET 2023-07-29T00:21:33.850 DEBUG:boto:Path: /hiiqos-56/g?versionId=null 2023-07-29T00:21:33.851 DEBUG:boto:Data: 2023-07-29T00:21:33.851 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.852 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:33.852 DEBUG:boto:Port: 8001 2023-07-29T00:21:33.852 DEBUG:boto:Params: {} 2023-07-29T00:21:33.853 DEBUG:boto:Token: None 2023-07-29T00:21:33.853 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/g?versionId=null 2023-07-29T00:21:33.854 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4/9n3rqg6GiyjaeDy90aLXDCpIE= 2023-07-29T00:21:33.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4/9n3rqg6GiyjaeDy90aLXDCpIE=', 'Content-Length': '0'} 2023-07-29T00:21:33.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2dca0b0ba041ce2-0064c45b8d-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.856 DEBUG:boto:path=/hiiqos-56/g 2023-07-29T00:21:33.857 DEBUG:boto:auth_path=/hiiqos-56/g 2023-07-29T00:21:33.857 DEBUG:boto:path=/hiiqos-56/g?versionId=null 2023-07-29T00:21:33.858 DEBUG:boto:auth_path=/hiiqos-56/g?versionId=null 2023-07-29T00:21:33.858 DEBUG:boto:Method: GET 2023-07-29T00:21:33.858 DEBUG:boto:Path: /hiiqos-56/g?versionId=null 2023-07-29T00:21:33.859 DEBUG:boto:Data: 2023-07-29T00:21:33.859 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.859 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:33.860 DEBUG:boto:Port: 8000 2023-07-29T00:21:33.860 DEBUG:boto:Params: {} 2023-07-29T00:21:33.860 DEBUG:boto:Token: None 2023-07-29T00:21:33.861 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/g?versionId=null 2023-07-29T00:21:33.861 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4/9n3rqg6GiyjaeDy90aLXDCpIE= 2023-07-29T00:21:33.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4/9n3rqg6GiyjaeDy90aLXDCpIE=', 'Content-Length': '0'} 2023-07-29T00:21:33.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000220355c989e0611-0064c45b8d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.904 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:21:33.905 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:21:33.905 DEBUG:boto:path=/hiiqos-56/?prefix=g 2023-07-29T00:21:33.905 DEBUG:boto:auth_path=/hiiqos-56/?prefix=g 2023-07-29T00:21:33.906 DEBUG:boto:Method: GET 2023-07-29T00:21:33.906 DEBUG:boto:Path: /hiiqos-56/?prefix=g 2023-07-29T00:21:33.906 DEBUG:boto:Data: 2023-07-29T00:21:33.907 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.907 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:33.907 DEBUG:boto:Port: 8001 2023-07-29T00:21:33.908 DEBUG:boto:Params: {} 2023-07-29T00:21:33.908 DEBUG:boto:Token: None 2023-07-29T00:21:33.909 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/ 2023-07-29T00:21:33.909 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YZogNM5YkcSQXyEAlbgfE0GcTX4= 2023-07-29T00:21:33.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YZogNM5YkcSQXyEAlbgfE0GcTX4=', 'Content-Length': '0'} 2023-07-29T00:21:33.912 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005be63aff60bf3137-0064c45b8d-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.912 DEBUG:boto:b'hiiqos-56g1000falseg2023-07-29T00:18:22.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:21:33.913 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:21:33.914 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:21:33.914 DEBUG:boto:path=/hiiqos-56/?prefix=g 2023-07-29T00:21:33.914 DEBUG:boto:auth_path=/hiiqos-56/?prefix=g 2023-07-29T00:21:33.915 DEBUG:boto:Method: GET 2023-07-29T00:21:33.915 DEBUG:boto:Path: /hiiqos-56/?prefix=g 2023-07-29T00:21:33.915 DEBUG:boto:Data: 2023-07-29T00:21:33.916 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.916 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:33.916 DEBUG:boto:Port: 8000 2023-07-29T00:21:33.917 DEBUG:boto:Params: {} 2023-07-29T00:21:33.917 DEBUG:boto:Token: None 2023-07-29T00:21:33.918 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/ 2023-07-29T00:21:33.918 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YZogNM5YkcSQXyEAlbgfE0GcTX4= 2023-07-29T00:21:33.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YZogNM5YkcSQXyEAlbgfE0GcTX4=', 'Content-Length': '0'} 2023-07-29T00:21:33.920 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0fe61abf5865485-0064c45b8d-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.921 DEBUG:boto:b'hiiqos-56g1000falseg2023-07-29T00:18:22.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:21:33.921 DEBUG:boto:path=/hiiqos-56/g 2023-07-29T00:21:33.922 DEBUG:boto:auth_path=/hiiqos-56/g 2023-07-29T00:21:33.922 DEBUG:boto:Method: HEAD 2023-07-29T00:21:33.922 DEBUG:boto:Path: /hiiqos-56/g 2023-07-29T00:21:33.923 DEBUG:boto:Data: 2023-07-29T00:21:33.923 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.923 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:33.924 DEBUG:boto:Port: 8001 2023-07-29T00:21:33.924 DEBUG:boto:Params: {} 2023-07-29T00:21:33.924 DEBUG:boto:Token: None 2023-07-29T00:21:33.925 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/g 2023-07-29T00:21:33.925 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vcpaZEsA+ahZNt1pnIp3V8c1JxA= 2023-07-29T00:21:33.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vcpaZEsA+ahZNt1pnIp3V8c1JxA=', 'Content-Length': '0'} 2023-07-29T00:21:33.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8ab39ff802f4cf7-0064c45b8d-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.928 DEBUG:boto:path=/hiiqos-56/g 2023-07-29T00:21:33.928 DEBUG:boto:auth_path=/hiiqos-56/g 2023-07-29T00:21:33.929 DEBUG:boto:Method: HEAD 2023-07-29T00:21:33.929 DEBUG:boto:Path: /hiiqos-56/g 2023-07-29T00:21:33.929 DEBUG:boto:Data: 2023-07-29T00:21:33.930 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.930 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:33.930 DEBUG:boto:Port: 8000 2023-07-29T00:21:33.931 DEBUG:boto:Params: {} 2023-07-29T00:21:33.931 DEBUG:boto:Token: None 2023-07-29T00:21:33.931 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/g 2023-07-29T00:21:33.932 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vcpaZEsA+ahZNt1pnIp3V8c1JxA= 2023-07-29T00:21:33.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vcpaZEsA+ahZNt1pnIp3V8c1JxA=', 'Content-Length': '0'} 2023-07-29T00:21:33.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090da5aea1ba665c3-0064c45b8d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-07-29T00:21:33.935 DEBUG:boto:path=/hiiqos-56/g 2023-07-29T00:21:33.935 DEBUG:boto:auth_path=/hiiqos-56/g 2023-07-29T00:21:33.935 DEBUG:boto:Method: GET 2023-07-29T00:21:33.936 DEBUG:boto:Path: /hiiqos-56/g 2023-07-29T00:21:33.936 DEBUG:boto:Data: 2023-07-29T00:21:33.936 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.937 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:33.937 DEBUG:boto:Port: 8001 2023-07-29T00:21:33.938 DEBUG:boto:Params: {} 2023-07-29T00:21:33.938 DEBUG:boto:Token: None 2023-07-29T00:21:33.938 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/g 2023-07-29T00:21:33.939 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0Uude8UZF16Mb0s4e0Q+k6mB5PU= 2023-07-29T00:21:33.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0Uude8UZF16Mb0s4e0Q+k6mB5PU=', 'Content-Length': '0'} 2023-07-29T00:21:33.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d420117b3a3aaa46-0064c45b8d-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.942 DEBUG:boto:path=/hiiqos-56/g 2023-07-29T00:21:33.942 DEBUG:boto:auth_path=/hiiqos-56/g 2023-07-29T00:21:33.943 DEBUG:boto:Method: GET 2023-07-29T00:21:33.943 DEBUG:boto:Path: /hiiqos-56/g 2023-07-29T00:21:33.943 DEBUG:boto:Data: 2023-07-29T00:21:33.944 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.944 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:33.944 DEBUG:boto:Port: 8000 2023-07-29T00:21:33.945 DEBUG:boto:Params: {} 2023-07-29T00:21:33.945 DEBUG:boto:Token: None 2023-07-29T00:21:33.946 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/g 2023-07-29T00:21:33.946 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0Uude8UZF16Mb0s4e0Q+k6mB5PU= 2023-07-29T00:21:33.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0Uude8UZF16Mb0s4e0Q+k6mB5PU=', 'Content-Length': '0'} 2023-07-29T00:21:33.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f703bbd75037624d-0064c45b8d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:33.949 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:21:33.949 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:21:33.950 DEBUG:boto:path=/hiiqos-56/?prefix=g 2023-07-29T00:21:33.950 DEBUG:boto:auth_path=/hiiqos-56/?prefix=g 2023-07-29T00:21:33.950 DEBUG:boto:Method: GET 2023-07-29T00:21:33.951 DEBUG:boto:Path: /hiiqos-56/?prefix=g 2023-07-29T00:21:33.951 DEBUG:boto:Data: 2023-07-29T00:21:33.951 DEBUG:boto:Headers: {} 2023-07-29T00:21:33.952 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:33.952 DEBUG:boto:Port: 8001 2023-07-29T00:21:33.952 DEBUG:boto:Params: {} 2023-07-29T00:21:33.953 DEBUG:boto:Token: None 2023-07-29T00:21:33.953 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:33 GMT /hiiqos-56/ 2023-07-29T00:21:33.954 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YZogNM5YkcSQXyEAlbgfE0GcTX4= 2023-07-29T00:21:33.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YZogNM5YkcSQXyEAlbgfE0GcTX4=', 'Content-Length': '0'} 2023-07-29T00:21:33.956 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e7f3aa6cbcab9cb-0064c45b8d-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:34.004 DEBUG:boto:b'hiiqos-56g1000falseg2023-07-29T00:18:22.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:21:34.005 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:21:34.005 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:21:34.006 DEBUG:boto:path=/hiiqos-56/?prefix=g 2023-07-29T00:21:34.006 DEBUG:boto:auth_path=/hiiqos-56/?prefix=g 2023-07-29T00:21:34.006 DEBUG:boto:Method: GET 2023-07-29T00:21:34.007 DEBUG:boto:Path: /hiiqos-56/?prefix=g 2023-07-29T00:21:34.007 DEBUG:boto:Data: 2023-07-29T00:21:34.008 DEBUG:boto:Headers: {} 2023-07-29T00:21:34.008 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:34.008 DEBUG:boto:Port: 8000 2023-07-29T00:21:34.009 DEBUG:boto:Params: {} 2023-07-29T00:21:34.009 DEBUG:boto:Token: None 2023-07-29T00:21:34.009 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:34 GMT /hiiqos-56/ 2023-07-29T00:21:34.010 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5sY+bXpxT3XVO7BRyU3SAli5WHo= 2023-07-29T00:21:34.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5sY+bXpxT3XVO7BRyU3SAli5WHo=', 'Content-Length': '0'} 2023-07-29T00:21:34.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f3c2aa5313713f8-0064c45b8d-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:34.013 DEBUG:boto:b'hiiqos-56g1000falseg2023-07-29T00:18:22.162Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:21:34.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-07-29T00:21:34.014 DEBUG:boto:path=/hiiqos-56/h 2023-07-29T00:21:34.014 DEBUG:boto:auth_path=/hiiqos-56/h 2023-07-29T00:21:34.015 DEBUG:boto:path=/hiiqos-56/h?versionId=null 2023-07-29T00:21:34.015 DEBUG:boto:auth_path=/hiiqos-56/h?versionId=null 2023-07-29T00:21:34.015 DEBUG:boto:Method: GET 2023-07-29T00:21:34.016 DEBUG:boto:Path: /hiiqos-56/h?versionId=null 2023-07-29T00:21:34.016 DEBUG:boto:Data: 2023-07-29T00:21:34.016 DEBUG:boto:Headers: {} 2023-07-29T00:21:34.017 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:34.017 DEBUG:boto:Port: 8001 2023-07-29T00:21:34.017 DEBUG:boto:Params: {} 2023-07-29T00:21:34.018 DEBUG:boto:Token: None 2023-07-29T00:21:34.018 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:34 GMT /hiiqos-56/h?versionId=null 2023-07-29T00:21:34.019 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3CV2mjxZXoqSpyaJPCe3R2QtsNM= 2023-07-29T00:21:34.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3CV2mjxZXoqSpyaJPCe3R2QtsNM=', 'Content-Length': '0'} 2023-07-29T00:21:34.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a50c932724f5cbd-0064c45b8d-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:34.022 DEBUG:boto:path=/hiiqos-56/h 2023-07-29T00:21:34.023 DEBUG:boto:auth_path=/hiiqos-56/h 2023-07-29T00:21:34.023 DEBUG:boto:path=/hiiqos-56/h?versionId=null 2023-07-29T00:21:34.023 DEBUG:boto:auth_path=/hiiqos-56/h?versionId=null 2023-07-29T00:21:34.024 DEBUG:boto:Method: GET 2023-07-29T00:21:34.024 DEBUG:boto:Path: /hiiqos-56/h?versionId=null 2023-07-29T00:21:34.024 DEBUG:boto:Data: 2023-07-29T00:21:34.025 DEBUG:boto:Headers: {} 2023-07-29T00:21:34.025 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:34.026 DEBUG:boto:Port: 8000 2023-07-29T00:21:34.026 DEBUG:boto:Params: {} 2023-07-29T00:21:34.027 DEBUG:boto:Token: None 2023-07-29T00:21:34.027 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:34 GMT /hiiqos-56/h?versionId=null 2023-07-29T00:21:34.028 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3CV2mjxZXoqSpyaJPCe3R2QtsNM= 2023-07-29T00:21:34.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3CV2mjxZXoqSpyaJPCe3R2QtsNM=', 'Content-Length': '0'} 2023-07-29T00:21:34.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044e275234b68125e-0064c45b8d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:34.072 DEBUG:boto:path=/hiiqos-56/h 2023-07-29T00:21:34.073 DEBUG:boto:auth_path=/hiiqos-56/h 2023-07-29T00:21:34.073 DEBUG:boto:path=/hiiqos-56/h?versionId=null 2023-07-29T00:21:34.073 DEBUG:boto:auth_path=/hiiqos-56/h?versionId=null 2023-07-29T00:21:34.074 DEBUG:boto:Method: HEAD 2023-07-29T00:21:34.074 DEBUG:boto:Path: /hiiqos-56/h?versionId=null 2023-07-29T00:21:34.074 DEBUG:boto:Data: 2023-07-29T00:21:34.075 DEBUG:boto:Headers: {} 2023-07-29T00:21:34.075 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:34.075 DEBUG:boto:Port: 8001 2023-07-29T00:21:34.076 DEBUG:boto:Params: {} 2023-07-29T00:21:34.076 DEBUG:boto:Token: None 2023-07-29T00:21:34.077 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:21:34 GMT /hiiqos-56/h?versionId=null 2023-07-29T00:21:34.077 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:btf4cIpG7M+UAMFa1mSBRvTcdHc= 2023-07-29T00:21:34.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:btf4cIpG7M+UAMFa1mSBRvTcdHc=', 'Content-Length': '0'} 2023-07-29T00:21:34.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb4f09c216db2da3-0064c45b8e-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:34.080 DEBUG:boto:path=/hiiqos-56/h 2023-07-29T00:21:34.080 DEBUG:boto:auth_path=/hiiqos-56/h 2023-07-29T00:21:34.080 DEBUG:boto:path=/hiiqos-56/h?versionId=null 2023-07-29T00:21:34.081 DEBUG:boto:auth_path=/hiiqos-56/h?versionId=null 2023-07-29T00:21:34.081 DEBUG:boto:Method: HEAD 2023-07-29T00:21:34.081 DEBUG:boto:Path: /hiiqos-56/h?versionId=null 2023-07-29T00:21:34.082 DEBUG:boto:Data: 2023-07-29T00:21:34.082 DEBUG:boto:Headers: {} 2023-07-29T00:21:34.082 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:34.083 DEBUG:boto:Port: 8000 2023-07-29T00:21:34.083 DEBUG:boto:Params: {} 2023-07-29T00:21:34.083 DEBUG:boto:Token: None 2023-07-29T00:21:34.084 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:21:34 GMT /hiiqos-56/h?versionId=null 2023-07-29T00:21:34.084 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:btf4cIpG7M+UAMFa1mSBRvTcdHc= 2023-07-29T00:21:34.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:btf4cIpG7M+UAMFa1mSBRvTcdHc=', 'Content-Length': '0'} 2023-07-29T00:21:34.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000432705f7615669df-0064c45b8e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:34.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-07-29T00:21:34.087 DEBUG:boto:path=/hiiqos-56/h 2023-07-29T00:21:34.088 DEBUG:boto:auth_path=/hiiqos-56/h 2023-07-29T00:21:34.088 DEBUG:boto:path=/hiiqos-56/h?versionId=null 2023-07-29T00:21:34.089 DEBUG:boto:auth_path=/hiiqos-56/h?versionId=null 2023-07-29T00:21:34.089 DEBUG:boto:Method: GET 2023-07-29T00:21:34.089 DEBUG:boto:Path: /hiiqos-56/h?versionId=null 2023-07-29T00:21:34.090 DEBUG:boto:Data: 2023-07-29T00:21:34.090 DEBUG:boto:Headers: {} 2023-07-29T00:21:34.091 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:34.091 DEBUG:boto:Port: 8001 2023-07-29T00:21:34.091 DEBUG:boto:Params: {} 2023-07-29T00:21:34.092 DEBUG:boto:Token: None 2023-07-29T00:21:34.092 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:34 GMT /hiiqos-56/h?versionId=null 2023-07-29T00:21:34.093 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3CV2mjxZXoqSpyaJPCe3R2QtsNM= 2023-07-29T00:21:34.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3CV2mjxZXoqSpyaJPCe3R2QtsNM=', 'Content-Length': '0'} 2023-07-29T00:21:34.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ea8aa65ebf4370d-0064c45b8e-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:34.095 DEBUG:boto:path=/hiiqos-56/h 2023-07-29T00:21:34.096 DEBUG:boto:auth_path=/hiiqos-56/h 2023-07-29T00:21:34.096 DEBUG:boto:path=/hiiqos-56/h?versionId=null 2023-07-29T00:21:34.097 DEBUG:boto:auth_path=/hiiqos-56/h?versionId=null 2023-07-29T00:21:34.097 DEBUG:boto:Method: GET 2023-07-29T00:21:34.097 DEBUG:boto:Path: /hiiqos-56/h?versionId=null 2023-07-29T00:21:34.098 DEBUG:boto:Data: 2023-07-29T00:21:34.098 DEBUG:boto:Headers: {} 2023-07-29T00:21:34.098 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:34.099 DEBUG:boto:Port: 8000 2023-07-29T00:21:34.099 DEBUG:boto:Params: {} 2023-07-29T00:21:34.099 DEBUG:boto:Token: None 2023-07-29T00:21:34.100 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:34 GMT /hiiqos-56/h?versionId=null 2023-07-29T00:21:34.100 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3CV2mjxZXoqSpyaJPCe3R2QtsNM= 2023-07-29T00:21:34.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3CV2mjxZXoqSpyaJPCe3R2QtsNM=', 'Content-Length': '0'} 2023-07-29T00:21:34.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d14968090f7f42e-0064c45b8e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:34.103 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:21:34.104 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:21:34.104 DEBUG:boto:path=/hiiqos-56/?prefix=h 2023-07-29T00:21:34.105 DEBUG:boto:auth_path=/hiiqos-56/?prefix=h 2023-07-29T00:21:34.105 DEBUG:boto:Method: GET 2023-07-29T00:21:34.105 DEBUG:boto:Path: /hiiqos-56/?prefix=h 2023-07-29T00:21:34.106 DEBUG:boto:Data: 2023-07-29T00:21:34.106 DEBUG:boto:Headers: {} 2023-07-29T00:21:34.107 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:34.107 DEBUG:boto:Port: 8001 2023-07-29T00:21:34.107 DEBUG:boto:Params: {} 2023-07-29T00:21:34.108 DEBUG:boto:Token: None 2023-07-29T00:21:34.108 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:34 GMT /hiiqos-56/ 2023-07-29T00:21:34.108 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5sY+bXpxT3XVO7BRyU3SAli5WHo= 2023-07-29T00:21:34.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5sY+bXpxT3XVO7BRyU3SAli5WHo=', 'Content-Length': '0'} 2023-07-29T00:21:34.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8f94a92c93c368a-0064c45b8e-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:34.112 DEBUG:boto:b'hiiqos-56h1000falseh2023-07-29T00:18:22.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:21:34.113 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:21:34.113 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:21:34.114 DEBUG:boto:path=/hiiqos-56/?prefix=h 2023-07-29T00:21:34.114 DEBUG:boto:auth_path=/hiiqos-56/?prefix=h 2023-07-29T00:21:34.114 DEBUG:boto:Method: GET 2023-07-29T00:21:34.115 DEBUG:boto:Path: /hiiqos-56/?prefix=h 2023-07-29T00:21:34.115 DEBUG:boto:Data: 2023-07-29T00:21:34.116 DEBUG:boto:Headers: {} 2023-07-29T00:21:34.116 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:34.116 DEBUG:boto:Port: 8000 2023-07-29T00:21:34.117 DEBUG:boto:Params: {} 2023-07-29T00:21:34.117 DEBUG:boto:Token: None 2023-07-29T00:21:34.118 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:34 GMT /hiiqos-56/ 2023-07-29T00:21:34.118 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5sY+bXpxT3XVO7BRyU3SAli5WHo= 2023-07-29T00:21:34.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5sY+bXpxT3XVO7BRyU3SAli5WHo=', 'Content-Length': '0'} 2023-07-29T00:21:34.120 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a18b204f63c276b9-0064c45b8e-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:34.121 DEBUG:boto:b'hiiqos-56h1000falseh2023-07-29T00:18:22.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:21:34.122 DEBUG:boto:path=/hiiqos-56/h 2023-07-29T00:21:34.122 DEBUG:boto:auth_path=/hiiqos-56/h 2023-07-29T00:21:34.122 DEBUG:boto:Method: HEAD 2023-07-29T00:21:34.123 DEBUG:boto:Path: /hiiqos-56/h 2023-07-29T00:21:34.123 DEBUG:boto:Data: 2023-07-29T00:21:34.123 DEBUG:boto:Headers: {} 2023-07-29T00:21:34.124 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:34.124 DEBUG:boto:Port: 8001 2023-07-29T00:21:34.124 DEBUG:boto:Params: {} 2023-07-29T00:21:34.125 DEBUG:boto:Token: None 2023-07-29T00:21:34.125 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:21:34 GMT /hiiqos-56/h 2023-07-29T00:21:34.126 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PGDzqcuye8x0uIMXHNHJc+QhKHc= 2023-07-29T00:21:34.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PGDzqcuye8x0uIMXHNHJc+QhKHc=', 'Content-Length': '0'} 2023-07-29T00:21:34.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0523639a9866ee1-0064c45b8e-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:34.128 DEBUG:boto:path=/hiiqos-56/h 2023-07-29T00:21:34.129 DEBUG:boto:auth_path=/hiiqos-56/h 2023-07-29T00:21:34.129 DEBUG:boto:Method: HEAD 2023-07-29T00:21:34.129 DEBUG:boto:Path: /hiiqos-56/h 2023-07-29T00:21:34.130 DEBUG:boto:Data: 2023-07-29T00:21:34.130 DEBUG:boto:Headers: {} 2023-07-29T00:21:34.130 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:34.131 DEBUG:boto:Port: 8000 2023-07-29T00:21:34.131 DEBUG:boto:Params: {} 2023-07-29T00:21:34.131 DEBUG:boto:Token: None 2023-07-29T00:21:34.132 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:21:34 GMT /hiiqos-56/h 2023-07-29T00:21:34.132 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PGDzqcuye8x0uIMXHNHJc+QhKHc= 2023-07-29T00:21:34.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PGDzqcuye8x0uIMXHNHJc+QhKHc=', 'Content-Length': '0'} 2023-07-29T00:21:34.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a97d1131fe02989-0064c45b8e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:34.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-07-29T00:21:34.135 DEBUG:boto:path=/hiiqos-56/h 2023-07-29T00:21:34.136 DEBUG:boto:auth_path=/hiiqos-56/h 2023-07-29T00:21:34.136 DEBUG:boto:Method: GET 2023-07-29T00:21:34.136 DEBUG:boto:Path: /hiiqos-56/h 2023-07-29T00:21:34.137 DEBUG:boto:Data: 2023-07-29T00:21:34.137 DEBUG:boto:Headers: {} 2023-07-29T00:21:34.137 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:34.138 DEBUG:boto:Port: 8001 2023-07-29T00:21:34.138 DEBUG:boto:Params: {} 2023-07-29T00:21:34.139 DEBUG:boto:Token: None 2023-07-29T00:21:34.139 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:34 GMT /hiiqos-56/h 2023-07-29T00:21:34.139 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xeD8Q/9O5a/z8K5F41ANRKYgjyo= 2023-07-29T00:21:34.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xeD8Q/9O5a/z8K5F41ANRKYgjyo=', 'Content-Length': '0'} 2023-07-29T00:21:34.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-56:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f83b11f7fb4b394-0064c45b8e-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:34.142 DEBUG:boto:path=/hiiqos-56/h 2023-07-29T00:21:34.143 DEBUG:boto:auth_path=/hiiqos-56/h 2023-07-29T00:21:34.143 DEBUG:boto:Method: GET 2023-07-29T00:21:34.143 DEBUG:boto:Path: /hiiqos-56/h 2023-07-29T00:21:34.144 DEBUG:boto:Data: 2023-07-29T00:21:34.144 DEBUG:boto:Headers: {} 2023-07-29T00:21:34.144 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:34.145 DEBUG:boto:Port: 8000 2023-07-29T00:21:34.145 DEBUG:boto:Params: {} 2023-07-29T00:21:34.146 DEBUG:boto:Token: None 2023-07-29T00:21:34.146 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:34 GMT /hiiqos-56/h 2023-07-29T00:21:34.146 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xeD8Q/9O5a/z8K5F41ANRKYgjyo= 2023-07-29T00:21:34.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xeD8Q/9O5a/z8K5F41ANRKYgjyo=', 'Content-Length': '0'} 2023-07-29T00:21:34.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:18:22 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8dc2705b79e769b-0064c45b8e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:34.149 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:21:34.150 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:21:34.150 DEBUG:boto:path=/hiiqos-56/?prefix=h 2023-07-29T00:21:34.150 DEBUG:boto:auth_path=/hiiqos-56/?prefix=h 2023-07-29T00:21:34.151 DEBUG:boto:Method: GET 2023-07-29T00:21:34.151 DEBUG:boto:Path: /hiiqos-56/?prefix=h 2023-07-29T00:21:34.151 DEBUG:boto:Data: 2023-07-29T00:21:34.152 DEBUG:boto:Headers: {} 2023-07-29T00:21:34.152 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:34.152 DEBUG:boto:Port: 8001 2023-07-29T00:21:34.153 DEBUG:boto:Params: {} 2023-07-29T00:21:34.153 DEBUG:boto:Token: None 2023-07-29T00:21:34.154 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:34 GMT /hiiqos-56/ 2023-07-29T00:21:34.154 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5sY+bXpxT3XVO7BRyU3SAli5WHo= 2023-07-29T00:21:34.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5sY+bXpxT3XVO7BRyU3SAli5WHo=', 'Content-Length': '0'} 2023-07-29T00:21:34.157 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001dcdd3dbf0e931d0-0064c45b8e-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:34.157 DEBUG:boto:b'hiiqos-56h1000falseh2023-07-29T00:18:22.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:21:34.158 DEBUG:boto:path=/hiiqos-56/ 2023-07-29T00:21:34.158 DEBUG:boto:auth_path=/hiiqos-56/ 2023-07-29T00:21:34.159 DEBUG:boto:path=/hiiqos-56/?prefix=h 2023-07-29T00:21:34.159 DEBUG:boto:auth_path=/hiiqos-56/?prefix=h 2023-07-29T00:21:34.159 DEBUG:boto:Method: GET 2023-07-29T00:21:34.160 DEBUG:boto:Path: /hiiqos-56/?prefix=h 2023-07-29T00:21:34.160 DEBUG:boto:Data: 2023-07-29T00:21:34.161 DEBUG:boto:Headers: {} 2023-07-29T00:21:34.161 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:34.161 DEBUG:boto:Port: 8000 2023-07-29T00:21:34.162 DEBUG:boto:Params: {} 2023-07-29T00:21:34.162 DEBUG:boto:Token: None 2023-07-29T00:21:34.162 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:21:34 GMT /hiiqos-56/ 2023-07-29T00:21:34.163 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5sY+bXpxT3XVO7BRyU3SAli5WHo= 2023-07-29T00:21:34.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5sY+bXpxT3XVO7BRyU3SAli5WHo=', 'Content-Length': '0'} 2023-07-29T00:21:34.165 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f07e7cca3e860be-0064c45b8e-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:34.208 DEBUG:boto:b'hiiqos-56h1000falseh2023-07-29T00:18:22.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:21:34.209 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-56 zones={a2, a1} 2023-07-29T00:21:34.210 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_incremental ... ok 2023-07-29T00:21:34.211 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:21:34.211 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:21:34.211 DEBUG:boto:Method: PUT 2023-07-29T00:21:34.212 DEBUG:boto:Path: /hiiqos-57/ 2023-07-29T00:21:34.212 DEBUG:boto:Data: 2023-07-29T00:21:34.212 DEBUG:boto:Headers: {} 2023-07-29T00:21:34.213 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:34.213 DEBUG:boto:Port: 8000 2023-07-29T00:21:34.213 DEBUG:boto:Params: {} 2023-07-29T00:21:34.214 DEBUG:boto:Token: None 2023-07-29T00:21:34.214 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 00:21:34 GMT /hiiqos-57/ 2023-07-29T00:21:34.215 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:D8WRvsukrQVKQUZeplt1d/T5zuM= 2023-07-29T00:21:34.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:D8WRvsukrQVKQUZeplt1d/T5zuM=', 'Content-Length': '0'} 2023-07-29T00:21:34.231 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000fbc8fe00a21888a-0064c45b8e-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:21:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:34.232 DEBUG:rgw_multi.tests:created bucket=hiiqos-57 2023-07-29T00:21:34.232 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:21:34.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:21:34.233 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:21:34.283 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:21:34.284 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:21:34.318 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T00:21:34.331 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T00:21:34.332 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:21:34.332 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:21:34.332 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:46:09.233742Z" 2023-07-29T00:21:34.332 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:21:34.332 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:21:34.332 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690590094.182190_907.1", 2023-07-29T00:21:34.332 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:21:34.182190Z" 2023-07-29T00:21:34.333 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:21:34.333 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:21:34.333 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:21:34.333 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T00:21:34.333 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:21:34.333 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:21:34.334 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:21:34.334 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T00:21:34.334 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:21:34.334 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T00:21:34.335 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690587969.233742_99.1', 1: '1_1690590094.182190_907.1', 2: '', 3: ''} 2023-07-29T00:21:34.335 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T00:21:34.336 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:21:34.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:21:34.336 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:21:34.382 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:21:34.383 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:21:34.411 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:21:34.363+0000 7f403e60f880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:21:34.412 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:21:34.363+0000 7f403e60f880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:21:34.424 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:21:34.425 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:21:34.425 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:21:34.425 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:21:34.425 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:21:34.425 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:21:34.425 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:21:34.426 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:21:34.426 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:21:34.426 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:21:34.426 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:21:34.426 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:21:34.426 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:21:34.427 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:21:34.427 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:21:34.427 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:21:34.427 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:21:34.427 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:21:34.427 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:21:34.427 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:21:34.428 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:21:34.428 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:21:34.428 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:21:34.428 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:21:34.428 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:21:34.428 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690590094.182190_907.1", 2023-07-29T00:21:34.428 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:21:34.429 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:21:34.429 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:21:34.429 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:21:34.182190Z", 2023-07-29T00:21:34.429 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:21:34.429 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:21:34.429 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:21:34.430 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:21:34.430 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:21:34.430 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:21:34.430 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:21:34.430 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:21:34.430 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:21:34.430 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:21:34.431 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:21:34.431 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:21:34.431 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:21:34.431 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:21:34.431 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:21:34.431 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:21:34.431 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:21:34.432 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:21:34.432 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:21:34.432 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:21:34.432 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:21:34.432 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:21:34.432 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:21:34.432 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:21:34.433 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:21:34.433 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:21:34.433 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:21:34.433 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:21:34.433 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:21:34.433 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:21:34.433 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:21:34.434 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:21:34.434 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:21:34.434 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:21:34Z" 2023-07-29T00:21:34.434 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:21:34.435 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690590094.182190_907.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:21:34.182190Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:21:34Z" } 2023-07-29T00:21:34.436 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690590094.182190_907.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:21:34.182190Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:21:34.436 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690590094.182190_907.1', 2: '', 3: ''} 2023-07-29T00:21:34.436 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690590094.182190_907.1', 2: '', 3: ''} 2023-07-29T00:21:34.437 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T00:21:34.437 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-07-29T00:21:34.438 INFO:teuthology.orchestra.run:waiting for 300 2023-07-29T00:21:34.438 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:21:34.390+0000 7f91f321c640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 90101) UID: 0 2023-07-29T00:21:34.439 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:21:34.390+0000 7f91f8e0f880 -1 shutting down 2023-07-29T00:21:36.328 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-07-29T00:21:40.540 INFO:tasks.rgw.c1.client.1:Stopped 2023-07-29T00:21:40.541 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:21:40.542 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:21:40.542 DEBUG:boto:Method: HEAD 2023-07-29T00:21:40.543 DEBUG:boto:Path: /hiiqos-57/ 2023-07-29T00:21:40.544 DEBUG:boto:Data: 2023-07-29T00:21:40.544 DEBUG:boto:Headers: {} 2023-07-29T00:21:40.545 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:40.545 DEBUG:boto:Port: 8000 2023-07-29T00:21:40.546 DEBUG:boto:Params: {} 2023-07-29T00:21:40.546 DEBUG:boto:Token: None 2023-07-29T00:21:40.547 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:21:40 GMT /hiiqos-57/ 2023-07-29T00:21:40.547 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oDMswg2Y0EB1C6/+mZF+Hu6RJTQ= 2023-07-29T00:21:40.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oDMswg2Y0EB1C6/+mZF+Hu6RJTQ=', 'Content-Length': '0'} 2023-07-29T00:21:40.551 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000009036982d9fa921d-0064c45b94-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:40.551 DEBUG:boto:path=/hiiqos-57/a 2023-07-29T00:21:40.552 DEBUG:boto:auth_path=/hiiqos-57/a 2023-07-29T00:21:40.552 DEBUG:boto:Method: PUT 2023-07-29T00:21:40.552 DEBUG:boto:Path: /hiiqos-57/a 2023-07-29T00:21:40.553 DEBUG:boto:Data: 2023-07-29T00:21:40.553 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:21:40.553 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:40.554 DEBUG:boto:Port: 8000 2023-07-29T00:21:40.554 DEBUG:boto:Params: {} 2023-07-29T00:21:40.555 DEBUG:boto:Token: None 2023-07-29T00:21:40.555 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:21:40 GMT /hiiqos-57/a 2023-07-29T00:21:40.556 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Y3pM8o/8AGk59KcDVx8qRdQ0jrg= 2023-07-29T00:21:40.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:21:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Y3pM8o/8AGk59KcDVx8qRdQ0jrg='} 2023-07-29T00:21:40.564 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e2552321fd9a481f-0064c45b94-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:40.564 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:21:40.565 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:21:40.565 DEBUG:boto:Method: HEAD 2023-07-29T00:21:40.566 DEBUG:boto:Path: /hiiqos-57/ 2023-07-29T00:21:40.566 DEBUG:boto:Data: 2023-07-29T00:21:40.566 DEBUG:boto:Headers: {} 2023-07-29T00:21:40.567 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:40.567 DEBUG:boto:Port: 8000 2023-07-29T00:21:40.567 DEBUG:boto:Params: {} 2023-07-29T00:21:40.568 DEBUG:boto:Token: None 2023-07-29T00:21:40.568 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:21:40 GMT /hiiqos-57/ 2023-07-29T00:21:40.569 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oDMswg2Y0EB1C6/+mZF+Hu6RJTQ= 2023-07-29T00:21:40.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oDMswg2Y0EB1C6/+mZF+Hu6RJTQ=', 'Content-Length': '0'} 2023-07-29T00:21:40.572 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c79865caae276941-0064c45b94-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:40.572 DEBUG:boto:path=/hiiqos-57/b 2023-07-29T00:21:40.573 DEBUG:boto:auth_path=/hiiqos-57/b 2023-07-29T00:21:40.573 DEBUG:boto:Method: PUT 2023-07-29T00:21:40.573 DEBUG:boto:Path: /hiiqos-57/b 2023-07-29T00:21:40.574 DEBUG:boto:Data: 2023-07-29T00:21:40.574 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:21:40.574 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:40.575 DEBUG:boto:Port: 8000 2023-07-29T00:21:40.575 DEBUG:boto:Params: {} 2023-07-29T00:21:40.576 DEBUG:boto:Token: None 2023-07-29T00:21:40.576 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:21:40 GMT /hiiqos-57/b 2023-07-29T00:21:40.577 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8wl3mxlgAbMk1dZprHHY02J1pe0= 2023-07-29T00:21:40.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:21:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8wl3mxlgAbMk1dZprHHY02J1pe0='} 2023-07-29T00:21:40.584 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000001cc6e5e219244aa-0064c45b94-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:40.585 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:21:40.586 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:21:40.587 DEBUG:boto:Method: HEAD 2023-07-29T00:21:40.588 DEBUG:boto:Path: /hiiqos-57/ 2023-07-29T00:21:40.588 DEBUG:boto:Data: 2023-07-29T00:21:40.589 DEBUG:boto:Headers: {} 2023-07-29T00:21:40.590 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:40.590 DEBUG:boto:Port: 8000 2023-07-29T00:21:40.591 DEBUG:boto:Params: {} 2023-07-29T00:21:40.591 DEBUG:boto:Token: None 2023-07-29T00:21:40.592 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:21:40 GMT /hiiqos-57/ 2023-07-29T00:21:40.592 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oDMswg2Y0EB1C6/+mZF+Hu6RJTQ= 2023-07-29T00:21:40.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oDMswg2Y0EB1C6/+mZF+Hu6RJTQ=', 'Content-Length': '0'} 2023-07-29T00:21:40.595 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af3dffa061bdc6d6-0064c45b94-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:40.595 DEBUG:boto:path=/hiiqos-57/c 2023-07-29T00:21:40.596 DEBUG:boto:auth_path=/hiiqos-57/c 2023-07-29T00:21:40.596 DEBUG:boto:Method: PUT 2023-07-29T00:21:40.597 DEBUG:boto:Path: /hiiqos-57/c 2023-07-29T00:21:40.597 DEBUG:boto:Data: 2023-07-29T00:21:40.597 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:21:40.598 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:40.598 DEBUG:boto:Port: 8000 2023-07-29T00:21:40.598 DEBUG:boto:Params: {} 2023-07-29T00:21:40.599 DEBUG:boto:Token: None 2023-07-29T00:21:40.599 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:21:40 GMT /hiiqos-57/c 2023-07-29T00:21:40.600 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RtLu/DY1bSz3V442oF41oBAOZQY= 2023-07-29T00:21:40.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:21:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RtLu/DY1bSz3V442oF41oBAOZQY='} 2023-07-29T00:21:40.644 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004d973f9fd3489a6f-0064c45b94-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:40.645 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:21:40.646 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:21:40.646 DEBUG:boto:Method: HEAD 2023-07-29T00:21:40.646 DEBUG:boto:Path: /hiiqos-57/ 2023-07-29T00:21:40.647 DEBUG:boto:Data: 2023-07-29T00:21:40.647 DEBUG:boto:Headers: {} 2023-07-29T00:21:40.647 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:40.648 DEBUG:boto:Port: 8000 2023-07-29T00:21:40.648 DEBUG:boto:Params: {} 2023-07-29T00:21:40.648 DEBUG:boto:Token: None 2023-07-29T00:21:40.649 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:21:40 GMT /hiiqos-57/ 2023-07-29T00:21:40.649 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oDMswg2Y0EB1C6/+mZF+Hu6RJTQ= 2023-07-29T00:21:40.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oDMswg2Y0EB1C6/+mZF+Hu6RJTQ=', 'Content-Length': '0'} 2023-07-29T00:21:40.652 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c4fc946530f5eebd-0064c45b94-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:40.653 DEBUG:boto:path=/hiiqos-57/d 2023-07-29T00:21:40.653 DEBUG:boto:auth_path=/hiiqos-57/d 2023-07-29T00:21:40.654 DEBUG:boto:Method: PUT 2023-07-29T00:21:40.654 DEBUG:boto:Path: /hiiqos-57/d 2023-07-29T00:21:40.654 DEBUG:boto:Data: 2023-07-29T00:21:40.655 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:21:40.655 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:40.655 DEBUG:boto:Port: 8000 2023-07-29T00:21:40.656 DEBUG:boto:Params: {} 2023-07-29T00:21:40.656 DEBUG:boto:Token: None 2023-07-29T00:21:40.657 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:21:40 GMT /hiiqos-57/d 2023-07-29T00:21:40.657 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gsS+TPfxJCVmFh4uwPoJmhfujyM= 2023-07-29T00:21:40.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:21:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gsS+TPfxJCVmFh4uwPoJmhfujyM='} 2023-07-29T00:21:40.664 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000055fa3e106819c10e-0064c45b94-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:40.664 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'hiiqos-57', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:21:40.665 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'hiiqos-57', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:21:40.665 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket hiiqos-57 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:21:40.718 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:21:40.719 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:21:40.761 INFO:teuthology.orchestra.run.smithi101.stdout:tenant: 2023-07-29T00:21:40.761 INFO:teuthology.orchestra.run.smithi101.stdout:bucket name: hiiqos-57 2023-07-29T00:21:40.764 INFO:teuthology.orchestra.run.smithi101.stdout:total entries: 4 2023-07-29T00:21:40.801 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:21:40.802 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:21:40.802 DEBUG:boto:Method: HEAD 2023-07-29T00:21:40.802 DEBUG:boto:Path: /hiiqos-57/ 2023-07-29T00:21:40.803 DEBUG:boto:Data: 2023-07-29T00:21:40.803 DEBUG:boto:Headers: {} 2023-07-29T00:21:40.803 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:40.804 DEBUG:boto:Port: 8000 2023-07-29T00:21:40.804 DEBUG:boto:Params: {} 2023-07-29T00:21:40.804 DEBUG:boto:Token: None 2023-07-29T00:21:40.805 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:21:40 GMT /hiiqos-57/ 2023-07-29T00:21:40.805 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oDMswg2Y0EB1C6/+mZF+Hu6RJTQ= 2023-07-29T00:21:40.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oDMswg2Y0EB1C6/+mZF+Hu6RJTQ=', 'Content-Length': '0'} 2023-07-29T00:21:40.808 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a23c780bd6fbf49-0064c45b94-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:40.808 DEBUG:boto:path=/hiiqos-57/e 2023-07-29T00:21:40.809 DEBUG:boto:auth_path=/hiiqos-57/e 2023-07-29T00:21:40.809 DEBUG:boto:Method: PUT 2023-07-29T00:21:40.810 DEBUG:boto:Path: /hiiqos-57/e 2023-07-29T00:21:40.810 DEBUG:boto:Data: 2023-07-29T00:21:40.810 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:21:40.811 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:40.811 DEBUG:boto:Port: 8000 2023-07-29T00:21:40.811 DEBUG:boto:Params: {} 2023-07-29T00:21:40.812 DEBUG:boto:Token: None 2023-07-29T00:21:40.812 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:21:40 GMT /hiiqos-57/e 2023-07-29T00:21:40.813 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:874BsuRmjwL9zpho55db90yFp/c= 2023-07-29T00:21:40.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:21:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:874BsuRmjwL9zpho55db90yFp/c='} 2023-07-29T00:21:40.818 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008e6ed833e4af78ac-0064c45b94-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:40.818 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:21:40.819 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:21:40.819 DEBUG:boto:Method: HEAD 2023-07-29T00:21:40.819 DEBUG:boto:Path: /hiiqos-57/ 2023-07-29T00:21:40.820 DEBUG:boto:Data: 2023-07-29T00:21:40.820 DEBUG:boto:Headers: {} 2023-07-29T00:21:40.820 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:40.821 DEBUG:boto:Port: 8000 2023-07-29T00:21:40.821 DEBUG:boto:Params: {} 2023-07-29T00:21:40.821 DEBUG:boto:Token: None 2023-07-29T00:21:40.822 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:21:40 GMT /hiiqos-57/ 2023-07-29T00:21:40.822 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oDMswg2Y0EB1C6/+mZF+Hu6RJTQ= 2023-07-29T00:21:40.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oDMswg2Y0EB1C6/+mZF+Hu6RJTQ=', 'Content-Length': '0'} 2023-07-29T00:21:40.824 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bb9837101e3029e2-0064c45b94-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:40.825 DEBUG:boto:path=/hiiqos-57/f 2023-07-29T00:21:40.825 DEBUG:boto:auth_path=/hiiqos-57/f 2023-07-29T00:21:40.826 DEBUG:boto:Method: PUT 2023-07-29T00:21:40.826 DEBUG:boto:Path: /hiiqos-57/f 2023-07-29T00:21:40.826 DEBUG:boto:Data: 2023-07-29T00:21:40.827 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:21:40.827 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:40.827 DEBUG:boto:Port: 8000 2023-07-29T00:21:40.828 DEBUG:boto:Params: {} 2023-07-29T00:21:40.828 DEBUG:boto:Token: None 2023-07-29T00:21:40.829 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:21:40 GMT /hiiqos-57/f 2023-07-29T00:21:40.829 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3H0PhmTwhcInszGVYE0/OLrHfQI= 2023-07-29T00:21:40.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:21:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3H0PhmTwhcInszGVYE0/OLrHfQI='} 2023-07-29T00:21:40.876 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000828a4726fe63d7b7-0064c45b94-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:40.877 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:21:40.878 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:21:40.878 DEBUG:boto:Method: HEAD 2023-07-29T00:21:40.879 DEBUG:boto:Path: /hiiqos-57/ 2023-07-29T00:21:40.879 DEBUG:boto:Data: 2023-07-29T00:21:40.880 DEBUG:boto:Headers: {} 2023-07-29T00:21:40.880 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:40.881 DEBUG:boto:Port: 8000 2023-07-29T00:21:40.881 DEBUG:boto:Params: {} 2023-07-29T00:21:40.882 DEBUG:boto:Token: None 2023-07-29T00:21:40.882 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:21:40 GMT /hiiqos-57/ 2023-07-29T00:21:40.883 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oDMswg2Y0EB1C6/+mZF+Hu6RJTQ= 2023-07-29T00:21:40.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oDMswg2Y0EB1C6/+mZF+Hu6RJTQ=', 'Content-Length': '0'} 2023-07-29T00:21:40.885 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003646c4a2e714fb16-0064c45b94-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:40.886 DEBUG:boto:path=/hiiqos-57/g 2023-07-29T00:21:40.886 DEBUG:boto:auth_path=/hiiqos-57/g 2023-07-29T00:21:40.886 DEBUG:boto:Method: PUT 2023-07-29T00:21:40.887 DEBUG:boto:Path: /hiiqos-57/g 2023-07-29T00:21:40.887 DEBUG:boto:Data: 2023-07-29T00:21:40.887 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:21:40.888 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:40.888 DEBUG:boto:Port: 8000 2023-07-29T00:21:40.888 DEBUG:boto:Params: {} 2023-07-29T00:21:40.889 DEBUG:boto:Token: None 2023-07-29T00:21:40.889 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:21:40 GMT /hiiqos-57/g 2023-07-29T00:21:40.890 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ocMw+aRn8BGb2Bo6GRH3OKN0fHQ= 2023-07-29T00:21:40.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:21:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ocMw+aRn8BGb2Bo6GRH3OKN0fHQ='} 2023-07-29T00:21:40.894 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000036c727f63f056c3d-0064c45b94-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:40.895 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:21:40.895 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:21:40.896 DEBUG:boto:Method: HEAD 2023-07-29T00:21:40.896 DEBUG:boto:Path: /hiiqos-57/ 2023-07-29T00:21:40.896 DEBUG:boto:Data: 2023-07-29T00:21:40.897 DEBUG:boto:Headers: {} 2023-07-29T00:21:40.897 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:40.897 DEBUG:boto:Port: 8000 2023-07-29T00:21:40.898 DEBUG:boto:Params: {} 2023-07-29T00:21:40.898 DEBUG:boto:Token: None 2023-07-29T00:21:40.899 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:21:40 GMT /hiiqos-57/ 2023-07-29T00:21:40.900 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oDMswg2Y0EB1C6/+mZF+Hu6RJTQ= 2023-07-29T00:21:40.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:21:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oDMswg2Y0EB1C6/+mZF+Hu6RJTQ=', 'Content-Length': '0'} 2023-07-29T00:21:40.903 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aec4213c61b65171-0064c45b94-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:40.904 DEBUG:boto:path=/hiiqos-57/h 2023-07-29T00:21:40.904 DEBUG:boto:auth_path=/hiiqos-57/h 2023-07-29T00:21:40.905 DEBUG:boto:Method: PUT 2023-07-29T00:21:40.905 DEBUG:boto:Path: /hiiqos-57/h 2023-07-29T00:21:40.906 DEBUG:boto:Data: 2023-07-29T00:21:40.906 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:21:40.906 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:21:40.907 DEBUG:boto:Port: 8000 2023-07-29T00:21:40.907 DEBUG:boto:Params: {} 2023-07-29T00:21:40.907 DEBUG:boto:Token: None 2023-07-29T00:21:40.908 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:21:40 GMT /hiiqos-57/h 2023-07-29T00:21:40.908 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XVs8NIpbtv2n8u5xAEokP1tspSI= 2023-07-29T00:21:40.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:21:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XVs8NIpbtv2n8u5xAEokP1tspSI='} 2023-07-29T00:21:40.956 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000744d5c23b484cdf2-0064c45b94-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:21:40.957 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-07-29T00:21:40.958 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-07-29T00:21:40.961 INFO:tasks.rgw.c1.client.1:Started 2023-07-29T00:21:40.961 DEBUG:teuthology.orchestra.run.smithi101:> curl http://smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:40.979 INFO:teuthology.orchestra.run.smithi101.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-07-29T00:21:40.979 INFO:teuthology.orchestra.run.smithi101.stderr: Dload Upload Total Spent Left Speed 2023-07-29T00:21:40.979 INFO:teuthology.orchestra.run.smithi101.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-07-29T00:21:40.980 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-07-29T00:21:40.981 INFO:teuthology.orchestra.run.smithi101.stderr:curl: (7) Failed to connect to smithi101.front.sepia.ceph.com port 8001: Connection refused 2023-07-29T00:21:41.981 DEBUG:teuthology.orchestra.run.smithi101:> curl http://smithi101.front.sepia.ceph.com:8001 2023-07-29T00:21:42.001 INFO:teuthology.orchestra.run.smithi101.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-07-29T00:21:42.001 INFO:teuthology.orchestra.run.smithi101.stderr: Dload Upload Total Spent Left Speed 2023-07-29T00:21:42.001 INFO:teuthology.orchestra.run.smithi101.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-07-29T00:21:42.002 INFO:teuthology.orchestra.run.smithi101.stdout:anonymous 2023-07-29T00:21:42.002 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-57 2023-07-29T00:21:42.003 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:21:42.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:21:42.004 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-57 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:21:42.034 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-07-29T00:21:42.095 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:21:42.096 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:21:42.130 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:21:42.081+0000 7fd328d26880 1 waiting to reach incremental sync.. 2023-07-29T00:21:47.131 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:21:47.082+0000 7fd328d26880 1 waiting to reach incremental sync.. 2023-07-29T00:21:52.131 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:21:52.083+0000 7fd328d26880 1 waiting to reach incremental sync.. 2023-07-29T00:21:57.132 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:21:57.083+0000 7fd328d26880 1 waiting to reach incremental sync.. 2023-07-29T00:22:02.133 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:22:02.084+0000 7fd328d26880 1 waiting to reach incremental sync.. 2023-07-29T00:22:07.133 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:22:07.084+0000 7fd328d26880 1 waiting to reach incremental sync.. 2023-07-29T00:22:12.134 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:22:12.085+0000 7fd328d26880 1 waiting to reach incremental sync.. 2023-07-29T00:22:17.135 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:22:17.085+0000 7fd328d26880 1 waiting to reach incremental sync.. 2023-07-29T00:22:22.135 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:22:22.086+0000 7fd328d26880 1 waiting to reach incremental sync.. 2023-07-29T00:22:27.136 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:22:27.087+0000 7fd328d26880 1 waiting to reach incremental sync.. 2023-07-29T00:22:32.137 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:22:32.087+0000 7fd328d26880 1 waiting to reach incremental sync.. 2023-07-29T00:22:37.138 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:22:37.088+0000 7fd328d26880 1 waiting to reach incremental sync.. 2023-07-29T00:22:42.139 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:22:42.090+0000 7fd328d26880 1 waiting to reach incremental sync.. 2023-07-29T00:22:47.141 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:22:47.091+0000 7fd328d26880 1 waiting to reach incremental sync.. 2023-07-29T00:22:52.141 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:22:52.091+0000 7fd328d26880 1 waiting to reach incremental sync.. 2023-07-29T00:22:57.142 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:22:57.092+0000 7fd328d26880 1 waiting to reach incremental sync.. 2023-07-29T00:23:01.153 INFO:tasks.rgw.c1.client.0.smithi101.stdout:2023-07-29T00:23:01.103+0000 7f0081a5e640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T00:23:02.143 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:23:02.092+0000 7fd328d26880 1 waiting to reach incremental sync.. 2023-07-29T00:23:07.143 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:23:07.093+0000 7fd328d26880 1 waiting to reach incremental sync.. 2023-07-29T00:23:12.144 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:23:12.094+0000 7fd328d26880 1 waiting to reach incremental sync.. 2023-07-29T00:23:17.145 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:23:17.094+0000 7fd328d26880 1 waiting to reach incremental sync.. 2023-07-29T00:23:22.145 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:23:22.095+0000 7fd328d26880 1 waiting to reach incremental sync.. 2023-07-29T00:23:27.146 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:23:27.095+0000 7fd328d26880 1 waiting to reach incremental sync.. 2023-07-29T00:23:32.146 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:23:32.096+0000 7fd328d26880 1 waiting to reach incremental sync.. 2023-07-29T00:23:37.147 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:23:37.096+0000 7fd328d26880 1 waiting to reach incremental sync.. 2023-07-29T00:23:42.150 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:23:42.099+0000 7fd328d26880 1 waiting to reach incremental sync.. 2023-07-29T00:23:47.150 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:23:47.100+0000 7fd328d26880 1 waiting to reach incremental sync.. 2023-07-29T00:23:52.151 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:23:52.100+0000 7fd328d26880 1 waiting to reach incremental sync.. 2023-07-29T00:23:57.152 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:23:57.101+0000 7fd328d26880 1 waiting to reach incremental sync.. 2023-07-29T00:24:02.152 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:24:02.101+0000 7fd328d26880 1 waiting to reach incremental sync.. 2023-07-29T00:24:07.154 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:24:07.103+0000 7fd328d26880 1 bucket sync caught up with source: 2023-07-29T00:24:07.154 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000003.33.6, 00000000002.27.6, 00000000002.29.6] 2023-07-29T00:24:07.154 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000003.33.6, 00000000002.27.6, 00000000002.29.6] 2023-07-29T00:24:07.155 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:24:07.103+0000 7fd328d26880 0 bucket checkpoint complete 2023-07-29T00:24:07.172 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-57 2023-07-29T00:24:07.173 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:24:07.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:24:07.174 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-57 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:24:07.233 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:24:07.233 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:24:07.269 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:24:07.218+0000 7efed980f880 1 waiting for incremental sync to catch up: 2023-07-29T00:24:07.270 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.892.6, , , 00000000001.881.6, 00000000001.551.6, , , 00000000002.410.6, , 00000000001.2430.6] 2023-07-29T00:24:07.270 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000002.892.6, , , 00000000001.881.6, 00000000001.551.6, , , 00000000002.410.6, , 00000000002.2435.6] 2023-07-29T00:24:12.271 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:24:12.220+0000 7efed980f880 1 waiting for incremental sync to catch up: 2023-07-29T00:24:12.271 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.892.6, , , 00000000001.881.6, 00000000001.551.6, , , 00000000002.410.6, , 00000000001.2430.6] 2023-07-29T00:24:12.271 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000002.892.6, , , 00000000001.881.6, 00000000001.551.6, , , 00000000002.410.6, , 00000000002.2435.6] 2023-07-29T00:24:17.272 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:24:17.221+0000 7efed980f880 1 waiting for incremental sync to catch up: 2023-07-29T00:24:17.273 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.892.6, , , 00000000001.881.6, 00000000001.551.6, , , 00000000002.410.6, , 00000000001.2430.6] 2023-07-29T00:24:17.273 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000002.892.6, , , 00000000001.881.6, 00000000001.551.6, , , 00000000002.410.6, , 00000000002.2435.6] 2023-07-29T00:24:22.273 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:24:22.222+0000 7efed980f880 1 bucket sync caught up with source: 2023-07-29T00:24:22.274 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.892.6, , , 00000000001.881.6, 00000000001.551.6, , , 00000000002.410.6, , 00000000002.2435.6] 2023-07-29T00:24:22.274 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000002.892.6, , , 00000000001.881.6, 00000000001.551.6, , , 00000000002.410.6, , 00000000002.2435.6] 2023-07-29T00:24:22.274 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:24:22.222+0000 7efed980f880 0 bucket checkpoint complete 2023-07-29T00:24:22.291 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-57 zones={a2, a1} 2023-07-29T00:24:22.291 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:24:22.292 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:24:22.292 DEBUG:boto:Method: HEAD 2023-07-29T00:24:22.293 DEBUG:boto:Path: /hiiqos-57/ 2023-07-29T00:24:22.293 DEBUG:boto:Data: 2023-07-29T00:24:22.293 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.294 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:22.294 DEBUG:boto:Port: 8001 2023-07-29T00:24:22.294 DEBUG:boto:Params: {} 2023-07-29T00:24:22.295 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-29T00:24:22.296 DEBUG:boto:Token: None 2023-07-29T00:24:22.296 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/ 2023-07-29T00:24:22.296 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:B7AjlL9VFaB+U7Uw/Vu7peLNKag= 2023-07-29T00:24:22.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:B7AjlL9VFaB+U7Uw/Vu7peLNKag=', 'Content-Length': '0'} 2023-07-29T00:24:22.302 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000890028f8af055966-0064c45c36-1b40-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.303 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:24:22.303 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:24:22.304 DEBUG:boto:Method: HEAD 2023-07-29T00:24:22.304 DEBUG:boto:Path: /hiiqos-57/ 2023-07-29T00:24:22.304 DEBUG:boto:Data: 2023-07-29T00:24:22.305 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.305 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:22.305 DEBUG:boto:Port: 8000 2023-07-29T00:24:22.306 DEBUG:boto:Params: {} 2023-07-29T00:24:22.306 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-29T00:24:22.307 DEBUG:boto:Token: None 2023-07-29T00:24:22.307 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/ 2023-07-29T00:24:22.307 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:B7AjlL9VFaB+U7Uw/Vu7peLNKag= 2023-07-29T00:24:22.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:B7AjlL9VFaB+U7Uw/Vu7peLNKag=', 'Content-Length': '0'} 2023-07-29T00:24:22.311 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b0ea413083156c2d-0064c45c36-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.312 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-29T00:24:22.312 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:24:22.312 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:24:22.313 DEBUG:boto:path=/hiiqos-57/?versions&max-keys=999 2023-07-29T00:24:22.313 DEBUG:boto:auth_path=/hiiqos-57/?versions&max-keys=999 2023-07-29T00:24:22.313 DEBUG:boto:Method: GET 2023-07-29T00:24:22.314 DEBUG:boto:Path: /hiiqos-57/?versions&max-keys=999 2023-07-29T00:24:22.314 DEBUG:boto:Data: 2023-07-29T00:24:22.314 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.315 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:22.315 DEBUG:boto:Port: 8001 2023-07-29T00:24:22.315 DEBUG:boto:Params: {} 2023-07-29T00:24:22.316 DEBUG:boto:Token: None 2023-07-29T00:24:22.316 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/?versions 2023-07-29T00:24:22.317 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KXyGhJFBqVDtCQNWYxxyeJAA4Ps= 2023-07-29T00:24:22.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KXyGhJFBqVDtCQNWYxxyeJAA4Ps=', 'Content-Length': '0'} 2023-07-29T00:24:22.320 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002221663574f5509f-0064c45c36-1b40-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.320 DEBUG:boto:b'hiiqos-57999falseanulltrue2023-07-29T00:21:40.511Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:21:40.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:21:40.555Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:21:40.612Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-07-29T00:21:40.767Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-07-29T00:21:40.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-07-29T00:21:40.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-07-29T00:21:40.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:22.321 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-07-29T00:24:22.322 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-07-29T00:24:22.322 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-07-29T00:24:22.322 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-07-29T00:24:22.323 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-07-29T00:24:22.323 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-07-29T00:24:22.323 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-07-29T00:24:22.324 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-07-29T00:24:22.324 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-29T00:24:22.324 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:24:22.325 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:24:22.325 DEBUG:boto:path=/hiiqos-57/?versions&max-keys=999 2023-07-29T00:24:22.325 DEBUG:boto:auth_path=/hiiqos-57/?versions&max-keys=999 2023-07-29T00:24:22.326 DEBUG:boto:Method: GET 2023-07-29T00:24:22.326 DEBUG:boto:Path: /hiiqos-57/?versions&max-keys=999 2023-07-29T00:24:22.326 DEBUG:boto:Data: 2023-07-29T00:24:22.327 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.327 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:22.327 DEBUG:boto:Port: 8000 2023-07-29T00:24:22.328 DEBUG:boto:Params: {} 2023-07-29T00:24:22.328 DEBUG:boto:Token: None 2023-07-29T00:24:22.328 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/?versions 2023-07-29T00:24:22.329 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KXyGhJFBqVDtCQNWYxxyeJAA4Ps= 2023-07-29T00:24:22.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KXyGhJFBqVDtCQNWYxxyeJAA4Ps=', 'Content-Length': '0'} 2023-07-29T00:24:22.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e90fe4c480ac6f7f-0064c45c36-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.332 DEBUG:boto:b'hiiqos-57999falseanulltrue2023-07-29T00:21:40.511Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:21:40.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:21:40.555Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:21:40.612Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-07-29T00:21:40.767Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-07-29T00:21:40.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-07-29T00:21:40.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-07-29T00:21:40.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:22.333 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-07-29T00:24:22.334 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-07-29T00:24:22.334 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-07-29T00:24:22.334 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-07-29T00:24:22.335 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-07-29T00:24:22.335 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-07-29T00:24:22.336 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-07-29T00:24:22.336 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-07-29T00:24:22.336 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:24:22.337 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:24:22.337 DEBUG:boto:path=/hiiqos-57/?versions&max-keys=999 2023-07-29T00:24:22.338 DEBUG:boto:auth_path=/hiiqos-57/?versions&max-keys=999 2023-07-29T00:24:22.338 DEBUG:boto:Method: GET 2023-07-29T00:24:22.338 DEBUG:boto:Path: /hiiqos-57/?versions&max-keys=999 2023-07-29T00:24:22.339 DEBUG:boto:Data: 2023-07-29T00:24:22.339 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.339 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:22.340 DEBUG:boto:Port: 8001 2023-07-29T00:24:22.340 DEBUG:boto:Params: {} 2023-07-29T00:24:22.340 DEBUG:boto:Token: None 2023-07-29T00:24:22.341 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/?versions 2023-07-29T00:24:22.341 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KXyGhJFBqVDtCQNWYxxyeJAA4Ps= 2023-07-29T00:24:22.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KXyGhJFBqVDtCQNWYxxyeJAA4Ps=', 'Content-Length': '0'} 2023-07-29T00:24:22.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000295da56660b5f0db-0064c45c36-1b40-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.344 DEBUG:boto:b'hiiqos-57999falseanulltrue2023-07-29T00:21:40.511Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:21:40.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:21:40.555Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:21:40.612Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-07-29T00:21:40.767Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-07-29T00:21:40.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-07-29T00:21:40.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-07-29T00:21:40.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:22.345 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:24:22.346 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:24:22.346 DEBUG:boto:path=/hiiqos-57/?versions&max-keys=999 2023-07-29T00:24:22.346 DEBUG:boto:auth_path=/hiiqos-57/?versions&max-keys=999 2023-07-29T00:24:22.347 DEBUG:boto:Method: GET 2023-07-29T00:24:22.347 DEBUG:boto:Path: /hiiqos-57/?versions&max-keys=999 2023-07-29T00:24:22.347 DEBUG:boto:Data: 2023-07-29T00:24:22.348 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.348 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:22.348 DEBUG:boto:Port: 8000 2023-07-29T00:24:22.349 DEBUG:boto:Params: {} 2023-07-29T00:24:22.349 DEBUG:boto:Token: None 2023-07-29T00:24:22.350 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/?versions 2023-07-29T00:24:22.350 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KXyGhJFBqVDtCQNWYxxyeJAA4Ps= 2023-07-29T00:24:22.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KXyGhJFBqVDtCQNWYxxyeJAA4Ps=', 'Content-Length': '0'} 2023-07-29T00:24:22.352 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f86b4bc2b6cc8051-0064c45c36-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.353 DEBUG:boto:b'hiiqos-57999falseanulltrue2023-07-29T00:21:40.511Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:21:40.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:21:40.555Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:21:40.612Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-07-29T00:21:40.767Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-07-29T00:21:40.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-07-29T00:21:40.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-07-29T00:21:40.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:22.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-07-29T00:24:22.354 DEBUG:boto:path=/hiiqos-57/a 2023-07-29T00:24:22.355 DEBUG:boto:auth_path=/hiiqos-57/a 2023-07-29T00:24:22.355 DEBUG:boto:path=/hiiqos-57/a?versionId=null 2023-07-29T00:24:22.355 DEBUG:boto:auth_path=/hiiqos-57/a?versionId=null 2023-07-29T00:24:22.356 DEBUG:boto:Method: GET 2023-07-29T00:24:22.356 DEBUG:boto:Path: /hiiqos-57/a?versionId=null 2023-07-29T00:24:22.356 DEBUG:boto:Data: 2023-07-29T00:24:22.357 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.357 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:22.357 DEBUG:boto:Port: 8001 2023-07-29T00:24:22.358 DEBUG:boto:Params: {} 2023-07-29T00:24:22.358 DEBUG:boto:Token: None 2023-07-29T00:24:22.359 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/a?versionId=null 2023-07-29T00:24:22.359 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uz4yM5zJ6RX3qFcUShJf5z0uIv4= 2023-07-29T00:24:22.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uz4yM5zJ6RX3qFcUShJf5z0uIv4=', 'Content-Length': '0'} 2023-07-29T00:24:22.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-57:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b6d2b144a9c5c65-0064c45c36-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.362 DEBUG:boto:path=/hiiqos-57/a 2023-07-29T00:24:22.363 DEBUG:boto:auth_path=/hiiqos-57/a 2023-07-29T00:24:22.363 DEBUG:boto:path=/hiiqos-57/a?versionId=null 2023-07-29T00:24:22.364 DEBUG:boto:auth_path=/hiiqos-57/a?versionId=null 2023-07-29T00:24:22.364 DEBUG:boto:Method: GET 2023-07-29T00:24:22.364 DEBUG:boto:Path: /hiiqos-57/a?versionId=null 2023-07-29T00:24:22.365 DEBUG:boto:Data: 2023-07-29T00:24:22.365 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.365 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:22.366 DEBUG:boto:Port: 8000 2023-07-29T00:24:22.366 DEBUG:boto:Params: {} 2023-07-29T00:24:22.366 DEBUG:boto:Token: None 2023-07-29T00:24:22.367 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/a?versionId=null 2023-07-29T00:24:22.367 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uz4yM5zJ6RX3qFcUShJf5z0uIv4= 2023-07-29T00:24:22.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uz4yM5zJ6RX3qFcUShJf5z0uIv4=', 'Content-Length': '0'} 2023-07-29T00:24:22.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f17acf66a5a530f5-0064c45c36-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.370 DEBUG:boto:path=/hiiqos-57/a 2023-07-29T00:24:22.371 DEBUG:boto:auth_path=/hiiqos-57/a 2023-07-29T00:24:22.371 DEBUG:boto:path=/hiiqos-57/a?versionId=null 2023-07-29T00:24:22.371 DEBUG:boto:auth_path=/hiiqos-57/a?versionId=null 2023-07-29T00:24:22.372 DEBUG:boto:Method: HEAD 2023-07-29T00:24:22.372 DEBUG:boto:Path: /hiiqos-57/a?versionId=null 2023-07-29T00:24:22.372 DEBUG:boto:Data: 2023-07-29T00:24:22.373 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.373 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:22.373 DEBUG:boto:Port: 8001 2023-07-29T00:24:22.374 DEBUG:boto:Params: {} 2023-07-29T00:24:22.374 DEBUG:boto:Token: None 2023-07-29T00:24:22.374 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/a?versionId=null 2023-07-29T00:24:22.375 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:79X8X3Qs/8+AHs4q2reMJ9MhiSU= 2023-07-29T00:24:22.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:79X8X3Qs/8+AHs4q2reMJ9MhiSU=', 'Content-Length': '0'} 2023-07-29T00:24:22.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-57:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdb3c0f351b63741-0064c45c36-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.377 DEBUG:boto:path=/hiiqos-57/a 2023-07-29T00:24:22.378 DEBUG:boto:auth_path=/hiiqos-57/a 2023-07-29T00:24:22.378 DEBUG:boto:path=/hiiqos-57/a?versionId=null 2023-07-29T00:24:22.378 DEBUG:boto:auth_path=/hiiqos-57/a?versionId=null 2023-07-29T00:24:22.379 DEBUG:boto:Method: HEAD 2023-07-29T00:24:22.379 DEBUG:boto:Path: /hiiqos-57/a?versionId=null 2023-07-29T00:24:22.379 DEBUG:boto:Data: 2023-07-29T00:24:22.380 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.380 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:22.381 DEBUG:boto:Port: 8000 2023-07-29T00:24:22.381 DEBUG:boto:Params: {} 2023-07-29T00:24:22.381 DEBUG:boto:Token: None 2023-07-29T00:24:22.382 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/a?versionId=null 2023-07-29T00:24:22.382 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:79X8X3Qs/8+AHs4q2reMJ9MhiSU= 2023-07-29T00:24:22.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:79X8X3Qs/8+AHs4q2reMJ9MhiSU=', 'Content-Length': '0'} 2023-07-29T00:24:22.384 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000333dd6eaf4db18c9-0064c45c36-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-07-29T00:24:22.385 DEBUG:boto:path=/hiiqos-57/a 2023-07-29T00:24:22.386 DEBUG:boto:auth_path=/hiiqos-57/a 2023-07-29T00:24:22.386 DEBUG:boto:path=/hiiqos-57/a?versionId=null 2023-07-29T00:24:22.387 DEBUG:boto:auth_path=/hiiqos-57/a?versionId=null 2023-07-29T00:24:22.387 DEBUG:boto:Method: GET 2023-07-29T00:24:22.387 DEBUG:boto:Path: /hiiqos-57/a?versionId=null 2023-07-29T00:24:22.388 DEBUG:boto:Data: 2023-07-29T00:24:22.388 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.388 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:22.389 DEBUG:boto:Port: 8001 2023-07-29T00:24:22.389 DEBUG:boto:Params: {} 2023-07-29T00:24:22.389 DEBUG:boto:Token: None 2023-07-29T00:24:22.390 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/a?versionId=null 2023-07-29T00:24:22.390 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uz4yM5zJ6RX3qFcUShJf5z0uIv4= 2023-07-29T00:24:22.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uz4yM5zJ6RX3qFcUShJf5z0uIv4=', 'Content-Length': '0'} 2023-07-29T00:24:22.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-57:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000475d775d751aa953-0064c45c36-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.393 DEBUG:boto:path=/hiiqos-57/a 2023-07-29T00:24:22.393 DEBUG:boto:auth_path=/hiiqos-57/a 2023-07-29T00:24:22.394 DEBUG:boto:path=/hiiqos-57/a?versionId=null 2023-07-29T00:24:22.394 DEBUG:boto:auth_path=/hiiqos-57/a?versionId=null 2023-07-29T00:24:22.394 DEBUG:boto:Method: GET 2023-07-29T00:24:22.395 DEBUG:boto:Path: /hiiqos-57/a?versionId=null 2023-07-29T00:24:22.395 DEBUG:boto:Data: 2023-07-29T00:24:22.396 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.396 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:22.396 DEBUG:boto:Port: 8000 2023-07-29T00:24:22.397 DEBUG:boto:Params: {} 2023-07-29T00:24:22.397 DEBUG:boto:Token: None 2023-07-29T00:24:22.397 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/a?versionId=null 2023-07-29T00:24:22.398 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uz4yM5zJ6RX3qFcUShJf5z0uIv4= 2023-07-29T00:24:22.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uz4yM5zJ6RX3qFcUShJf5z0uIv4=', 'Content-Length': '0'} 2023-07-29T00:24:22.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afd810df651a00aa-0064c45c36-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.400 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:24:22.401 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:24:22.401 DEBUG:boto:path=/hiiqos-57/?prefix=a 2023-07-29T00:24:22.402 DEBUG:boto:auth_path=/hiiqos-57/?prefix=a 2023-07-29T00:24:22.402 DEBUG:boto:Method: GET 2023-07-29T00:24:22.402 DEBUG:boto:Path: /hiiqos-57/?prefix=a 2023-07-29T00:24:22.403 DEBUG:boto:Data: 2023-07-29T00:24:22.403 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.403 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:22.404 DEBUG:boto:Port: 8001 2023-07-29T00:24:22.404 DEBUG:boto:Params: {} 2023-07-29T00:24:22.404 DEBUG:boto:Token: None 2023-07-29T00:24:22.405 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/ 2023-07-29T00:24:22.405 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OKCZY2qj6scFWJ1bIWz07KyqoOY= 2023-07-29T00:24:22.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OKCZY2qj6scFWJ1bIWz07KyqoOY=', 'Content-Length': '0'} 2023-07-29T00:24:22.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000927ea82da8ae4775-0064c45c36-1b40-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.448 DEBUG:boto:b'hiiqos-57a1000falsea2023-07-29T00:21:40.511Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:22.449 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:24:22.449 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:24:22.450 DEBUG:boto:path=/hiiqos-57/?prefix=a 2023-07-29T00:24:22.450 DEBUG:boto:auth_path=/hiiqos-57/?prefix=a 2023-07-29T00:24:22.451 DEBUG:boto:Method: GET 2023-07-29T00:24:22.451 DEBUG:boto:Path: /hiiqos-57/?prefix=a 2023-07-29T00:24:22.451 DEBUG:boto:Data: 2023-07-29T00:24:22.452 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.452 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:22.452 DEBUG:boto:Port: 8000 2023-07-29T00:24:22.453 DEBUG:boto:Params: {} 2023-07-29T00:24:22.453 DEBUG:boto:Token: None 2023-07-29T00:24:22.454 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/ 2023-07-29T00:24:22.454 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OKCZY2qj6scFWJ1bIWz07KyqoOY= 2023-07-29T00:24:22.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OKCZY2qj6scFWJ1bIWz07KyqoOY=', 'Content-Length': '0'} 2023-07-29T00:24:22.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e99d7acc1d5fa4bb-0064c45c36-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.458 DEBUG:boto:b'hiiqos-57a1000falsea2023-07-29T00:21:40.511Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:22.458 DEBUG:boto:path=/hiiqos-57/a 2023-07-29T00:24:22.459 DEBUG:boto:auth_path=/hiiqos-57/a 2023-07-29T00:24:22.459 DEBUG:boto:Method: HEAD 2023-07-29T00:24:22.460 DEBUG:boto:Path: /hiiqos-57/a 2023-07-29T00:24:22.460 DEBUG:boto:Data: 2023-07-29T00:24:22.460 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.461 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:22.461 DEBUG:boto:Port: 8001 2023-07-29T00:24:22.461 DEBUG:boto:Params: {} 2023-07-29T00:24:22.462 DEBUG:boto:Token: None 2023-07-29T00:24:22.462 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/a 2023-07-29T00:24:22.463 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8NDHx/jwWF5Uqd2qnvxxr0E1JyI= 2023-07-29T00:24:22.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8NDHx/jwWF5Uqd2qnvxxr0E1JyI=', 'Content-Length': '0'} 2023-07-29T00:24:22.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-57:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000269cd68aeb1fd015-0064c45c36-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.465 DEBUG:boto:path=/hiiqos-57/a 2023-07-29T00:24:22.466 DEBUG:boto:auth_path=/hiiqos-57/a 2023-07-29T00:24:22.466 DEBUG:boto:Method: HEAD 2023-07-29T00:24:22.466 DEBUG:boto:Path: /hiiqos-57/a 2023-07-29T00:24:22.467 DEBUG:boto:Data: 2023-07-29T00:24:22.467 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.467 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:22.468 DEBUG:boto:Port: 8000 2023-07-29T00:24:22.468 DEBUG:boto:Params: {} 2023-07-29T00:24:22.469 DEBUG:boto:Token: None 2023-07-29T00:24:22.469 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/a 2023-07-29T00:24:22.469 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8NDHx/jwWF5Uqd2qnvxxr0E1JyI= 2023-07-29T00:24:22.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8NDHx/jwWF5Uqd2qnvxxr0E1JyI=', 'Content-Length': '0'} 2023-07-29T00:24:22.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000708dcc0c98691cce-0064c45c36-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-07-29T00:24:22.472 DEBUG:boto:path=/hiiqos-57/a 2023-07-29T00:24:22.473 DEBUG:boto:auth_path=/hiiqos-57/a 2023-07-29T00:24:22.473 DEBUG:boto:Method: GET 2023-07-29T00:24:22.473 DEBUG:boto:Path: /hiiqos-57/a 2023-07-29T00:24:22.474 DEBUG:boto:Data: 2023-07-29T00:24:22.474 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.474 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:22.475 DEBUG:boto:Port: 8001 2023-07-29T00:24:22.475 DEBUG:boto:Params: {} 2023-07-29T00:24:22.475 DEBUG:boto:Token: None 2023-07-29T00:24:22.476 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/a 2023-07-29T00:24:22.476 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cnSrMZu2Q8KZiQJTeKnWxZyi4v8= 2023-07-29T00:24:22.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cnSrMZu2Q8KZiQJTeKnWxZyi4v8=', 'Content-Length': '0'} 2023-07-29T00:24:22.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-57:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d408abbe3a678a8c-0064c45c36-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.479 DEBUG:boto:path=/hiiqos-57/a 2023-07-29T00:24:22.480 DEBUG:boto:auth_path=/hiiqos-57/a 2023-07-29T00:24:22.481 DEBUG:boto:Method: GET 2023-07-29T00:24:22.481 DEBUG:boto:Path: /hiiqos-57/a 2023-07-29T00:24:22.481 DEBUG:boto:Data: 2023-07-29T00:24:22.482 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.482 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:22.482 DEBUG:boto:Port: 8000 2023-07-29T00:24:22.483 DEBUG:boto:Params: {} 2023-07-29T00:24:22.483 DEBUG:boto:Token: None 2023-07-29T00:24:22.484 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/a 2023-07-29T00:24:22.484 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cnSrMZu2Q8KZiQJTeKnWxZyi4v8= 2023-07-29T00:24:22.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cnSrMZu2Q8KZiQJTeKnWxZyi4v8=', 'Content-Length': '0'} 2023-07-29T00:24:22.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b61d582328986cc1-0064c45c36-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.528 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:24:22.529 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:24:22.529 DEBUG:boto:path=/hiiqos-57/?prefix=a 2023-07-29T00:24:22.530 DEBUG:boto:auth_path=/hiiqos-57/?prefix=a 2023-07-29T00:24:22.530 DEBUG:boto:Method: GET 2023-07-29T00:24:22.530 DEBUG:boto:Path: /hiiqos-57/?prefix=a 2023-07-29T00:24:22.531 DEBUG:boto:Data: 2023-07-29T00:24:22.531 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.531 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:22.532 DEBUG:boto:Port: 8001 2023-07-29T00:24:22.532 DEBUG:boto:Params: {} 2023-07-29T00:24:22.532 DEBUG:boto:Token: None 2023-07-29T00:24:22.533 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/ 2023-07-29T00:24:22.533 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OKCZY2qj6scFWJ1bIWz07KyqoOY= 2023-07-29T00:24:22.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OKCZY2qj6scFWJ1bIWz07KyqoOY=', 'Content-Length': '0'} 2023-07-29T00:24:22.536 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007767dc34a9b6890b-0064c45c36-1b40-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.537 DEBUG:boto:b'hiiqos-57a1000falsea2023-07-29T00:21:40.511Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:22.537 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:24:22.538 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:24:22.538 DEBUG:boto:path=/hiiqos-57/?prefix=a 2023-07-29T00:24:22.539 DEBUG:boto:auth_path=/hiiqos-57/?prefix=a 2023-07-29T00:24:22.539 DEBUG:boto:Method: GET 2023-07-29T00:24:22.539 DEBUG:boto:Path: /hiiqos-57/?prefix=a 2023-07-29T00:24:22.540 DEBUG:boto:Data: 2023-07-29T00:24:22.540 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.540 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:22.541 DEBUG:boto:Port: 8000 2023-07-29T00:24:22.541 DEBUG:boto:Params: {} 2023-07-29T00:24:22.541 DEBUG:boto:Token: None 2023-07-29T00:24:22.542 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/ 2023-07-29T00:24:22.542 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OKCZY2qj6scFWJ1bIWz07KyqoOY= 2023-07-29T00:24:22.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OKCZY2qj6scFWJ1bIWz07KyqoOY=', 'Content-Length': '0'} 2023-07-29T00:24:22.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acb533b0a5d700dc-0064c45c36-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.545 DEBUG:boto:b'hiiqos-57a1000falsea2023-07-29T00:21:40.511Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:22.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-07-29T00:24:22.547 DEBUG:boto:path=/hiiqos-57/b 2023-07-29T00:24:22.547 DEBUG:boto:auth_path=/hiiqos-57/b 2023-07-29T00:24:22.547 DEBUG:boto:path=/hiiqos-57/b?versionId=null 2023-07-29T00:24:22.548 DEBUG:boto:auth_path=/hiiqos-57/b?versionId=null 2023-07-29T00:24:22.548 DEBUG:boto:Method: GET 2023-07-29T00:24:22.548 DEBUG:boto:Path: /hiiqos-57/b?versionId=null 2023-07-29T00:24:22.549 DEBUG:boto:Data: 2023-07-29T00:24:22.549 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.549 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:22.550 DEBUG:boto:Port: 8001 2023-07-29T00:24:22.550 DEBUG:boto:Params: {} 2023-07-29T00:24:22.551 DEBUG:boto:Token: None 2023-07-29T00:24:22.551 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/b?versionId=null 2023-07-29T00:24:22.551 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kzNqvXF9h1r5IXL/Wj+9VR16Tfg= 2023-07-29T00:24:22.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kzNqvXF9h1r5IXL/Wj+9VR16Tfg=', 'Content-Length': '0'} 2023-07-29T00:24:22.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-57:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8290c273d188402-0064c45c36-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.596 DEBUG:boto:path=/hiiqos-57/b 2023-07-29T00:24:22.597 DEBUG:boto:auth_path=/hiiqos-57/b 2023-07-29T00:24:22.597 DEBUG:boto:path=/hiiqos-57/b?versionId=null 2023-07-29T00:24:22.598 DEBUG:boto:auth_path=/hiiqos-57/b?versionId=null 2023-07-29T00:24:22.598 DEBUG:boto:Method: GET 2023-07-29T00:24:22.598 DEBUG:boto:Path: /hiiqos-57/b?versionId=null 2023-07-29T00:24:22.599 DEBUG:boto:Data: 2023-07-29T00:24:22.599 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.599 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:22.600 DEBUG:boto:Port: 8000 2023-07-29T00:24:22.600 DEBUG:boto:Params: {} 2023-07-29T00:24:22.601 DEBUG:boto:Token: None 2023-07-29T00:24:22.601 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/b?versionId=null 2023-07-29T00:24:22.601 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kzNqvXF9h1r5IXL/Wj+9VR16Tfg= 2023-07-29T00:24:22.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kzNqvXF9h1r5IXL/Wj+9VR16Tfg=', 'Content-Length': '0'} 2023-07-29T00:24:22.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdbff3a4b32cfe95-0064c45c36-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.606 DEBUG:boto:path=/hiiqos-57/b 2023-07-29T00:24:22.607 DEBUG:boto:auth_path=/hiiqos-57/b 2023-07-29T00:24:22.607 DEBUG:boto:path=/hiiqos-57/b?versionId=null 2023-07-29T00:24:22.608 DEBUG:boto:auth_path=/hiiqos-57/b?versionId=null 2023-07-29T00:24:22.609 DEBUG:boto:Method: HEAD 2023-07-29T00:24:22.609 DEBUG:boto:Path: /hiiqos-57/b?versionId=null 2023-07-29T00:24:22.613 DEBUG:boto:Data: 2023-07-29T00:24:22.614 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.614 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:22.615 DEBUG:boto:Port: 8001 2023-07-29T00:24:22.615 DEBUG:boto:Params: {} 2023-07-29T00:24:22.616 DEBUG:boto:Token: None 2023-07-29T00:24:22.616 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/b?versionId=null 2023-07-29T00:24:22.617 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BEwv0pXoeNs/Fex/ZaeT3Da8V2A= 2023-07-29T00:24:22.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BEwv0pXoeNs/Fex/ZaeT3Da8V2A=', 'Content-Length': '0'} 2023-07-29T00:24:22.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-57:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f73fd8a3246f483-0064c45c36-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.620 DEBUG:boto:path=/hiiqos-57/b 2023-07-29T00:24:22.620 DEBUG:boto:auth_path=/hiiqos-57/b 2023-07-29T00:24:22.621 DEBUG:boto:path=/hiiqos-57/b?versionId=null 2023-07-29T00:24:22.621 DEBUG:boto:auth_path=/hiiqos-57/b?versionId=null 2023-07-29T00:24:22.622 DEBUG:boto:Method: HEAD 2023-07-29T00:24:22.622 DEBUG:boto:Path: /hiiqos-57/b?versionId=null 2023-07-29T00:24:22.622 DEBUG:boto:Data: 2023-07-29T00:24:22.623 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.623 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:22.623 DEBUG:boto:Port: 8000 2023-07-29T00:24:22.624 DEBUG:boto:Params: {} 2023-07-29T00:24:22.624 DEBUG:boto:Token: None 2023-07-29T00:24:22.625 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/b?versionId=null 2023-07-29T00:24:22.625 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BEwv0pXoeNs/Fex/ZaeT3Da8V2A= 2023-07-29T00:24:22.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BEwv0pXoeNs/Fex/ZaeT3Da8V2A=', 'Content-Length': '0'} 2023-07-29T00:24:22.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000755e544d3269c789-0064c45c36-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-07-29T00:24:22.629 DEBUG:boto:path=/hiiqos-57/b 2023-07-29T00:24:22.629 DEBUG:boto:auth_path=/hiiqos-57/b 2023-07-29T00:24:22.629 DEBUG:boto:path=/hiiqos-57/b?versionId=null 2023-07-29T00:24:22.630 DEBUG:boto:auth_path=/hiiqos-57/b?versionId=null 2023-07-29T00:24:22.630 DEBUG:boto:Method: GET 2023-07-29T00:24:22.631 DEBUG:boto:Path: /hiiqos-57/b?versionId=null 2023-07-29T00:24:22.631 DEBUG:boto:Data: 2023-07-29T00:24:22.631 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.632 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:22.632 DEBUG:boto:Port: 8001 2023-07-29T00:24:22.632 DEBUG:boto:Params: {} 2023-07-29T00:24:22.633 DEBUG:boto:Token: None 2023-07-29T00:24:22.633 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/b?versionId=null 2023-07-29T00:24:22.634 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kzNqvXF9h1r5IXL/Wj+9VR16Tfg= 2023-07-29T00:24:22.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kzNqvXF9h1r5IXL/Wj+9VR16Tfg=', 'Content-Length': '0'} 2023-07-29T00:24:22.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-57:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083ca48559b7efbf0-0064c45c36-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.637 DEBUG:boto:path=/hiiqos-57/b 2023-07-29T00:24:22.638 DEBUG:boto:auth_path=/hiiqos-57/b 2023-07-29T00:24:22.638 DEBUG:boto:path=/hiiqos-57/b?versionId=null 2023-07-29T00:24:22.639 DEBUG:boto:auth_path=/hiiqos-57/b?versionId=null 2023-07-29T00:24:22.639 DEBUG:boto:Method: GET 2023-07-29T00:24:22.640 DEBUG:boto:Path: /hiiqos-57/b?versionId=null 2023-07-29T00:24:22.641 DEBUG:boto:Data: 2023-07-29T00:24:22.641 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.641 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:22.642 DEBUG:boto:Port: 8000 2023-07-29T00:24:22.642 DEBUG:boto:Params: {} 2023-07-29T00:24:22.643 DEBUG:boto:Token: None 2023-07-29T00:24:22.644 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/b?versionId=null 2023-07-29T00:24:22.644 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kzNqvXF9h1r5IXL/Wj+9VR16Tfg= 2023-07-29T00:24:22.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kzNqvXF9h1r5IXL/Wj+9VR16Tfg=', 'Content-Length': '0'} 2023-07-29T00:24:22.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c498d5e589a59efc-0064c45c36-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.658 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:24:22.659 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:24:22.660 DEBUG:boto:path=/hiiqos-57/?prefix=b 2023-07-29T00:24:22.661 DEBUG:boto:auth_path=/hiiqos-57/?prefix=b 2023-07-29T00:24:22.661 DEBUG:boto:Method: GET 2023-07-29T00:24:22.662 DEBUG:boto:Path: /hiiqos-57/?prefix=b 2023-07-29T00:24:22.662 DEBUG:boto:Data: 2023-07-29T00:24:22.662 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.663 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:22.663 DEBUG:boto:Port: 8001 2023-07-29T00:24:22.664 DEBUG:boto:Params: {} 2023-07-29T00:24:22.665 DEBUG:boto:Token: None 2023-07-29T00:24:22.665 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/ 2023-07-29T00:24:22.666 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OKCZY2qj6scFWJ1bIWz07KyqoOY= 2023-07-29T00:24:22.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OKCZY2qj6scFWJ1bIWz07KyqoOY=', 'Content-Length': '0'} 2023-07-29T00:24:22.669 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fade055c20efb5b3-0064c45c36-1b40-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.670 DEBUG:boto:b'hiiqos-57b1000falseb2023-07-29T00:21:40.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:22.670 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:24:22.671 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:24:22.671 DEBUG:boto:path=/hiiqos-57/?prefix=b 2023-07-29T00:24:22.671 DEBUG:boto:auth_path=/hiiqos-57/?prefix=b 2023-07-29T00:24:22.672 DEBUG:boto:Method: GET 2023-07-29T00:24:22.672 DEBUG:boto:Path: /hiiqos-57/?prefix=b 2023-07-29T00:24:22.673 DEBUG:boto:Data: 2023-07-29T00:24:22.673 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.673 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:22.674 DEBUG:boto:Port: 8000 2023-07-29T00:24:22.674 DEBUG:boto:Params: {} 2023-07-29T00:24:22.675 DEBUG:boto:Token: None 2023-07-29T00:24:22.675 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/ 2023-07-29T00:24:22.676 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OKCZY2qj6scFWJ1bIWz07KyqoOY= 2023-07-29T00:24:22.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OKCZY2qj6scFWJ1bIWz07KyqoOY=', 'Content-Length': '0'} 2023-07-29T00:24:22.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006513b5d8cdc20a00-0064c45c36-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.720 DEBUG:boto:b'hiiqos-57b1000falseb2023-07-29T00:21:40.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:22.721 DEBUG:boto:path=/hiiqos-57/b 2023-07-29T00:24:22.722 DEBUG:boto:auth_path=/hiiqos-57/b 2023-07-29T00:24:22.722 DEBUG:boto:Method: HEAD 2023-07-29T00:24:22.722 DEBUG:boto:Path: /hiiqos-57/b 2023-07-29T00:24:22.723 DEBUG:boto:Data: 2023-07-29T00:24:22.723 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.724 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:22.724 DEBUG:boto:Port: 8001 2023-07-29T00:24:22.724 DEBUG:boto:Params: {} 2023-07-29T00:24:22.725 DEBUG:boto:Token: None 2023-07-29T00:24:22.726 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/b 2023-07-29T00:24:22.726 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KfQtHVgWN0oZ2Z/4CewOGnxNjSg= 2023-07-29T00:24:22.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KfQtHVgWN0oZ2Z/4CewOGnxNjSg=', 'Content-Length': '0'} 2023-07-29T00:24:22.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-57:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e95a82b9c512779d-0064c45c36-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.729 DEBUG:boto:path=/hiiqos-57/b 2023-07-29T00:24:22.730 DEBUG:boto:auth_path=/hiiqos-57/b 2023-07-29T00:24:22.730 DEBUG:boto:Method: HEAD 2023-07-29T00:24:22.731 DEBUG:boto:Path: /hiiqos-57/b 2023-07-29T00:24:22.731 DEBUG:boto:Data: 2023-07-29T00:24:22.732 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.732 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:22.733 DEBUG:boto:Port: 8000 2023-07-29T00:24:22.734 DEBUG:boto:Params: {} 2023-07-29T00:24:22.734 DEBUG:boto:Token: None 2023-07-29T00:24:22.735 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/b 2023-07-29T00:24:22.735 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KfQtHVgWN0oZ2Z/4CewOGnxNjSg= 2023-07-29T00:24:22.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KfQtHVgWN0oZ2Z/4CewOGnxNjSg=', 'Content-Length': '0'} 2023-07-29T00:24:22.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0dd73c6f2450f21-0064c45c36-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-07-29T00:24:22.739 DEBUG:boto:path=/hiiqos-57/b 2023-07-29T00:24:22.740 DEBUG:boto:auth_path=/hiiqos-57/b 2023-07-29T00:24:22.741 DEBUG:boto:Method: GET 2023-07-29T00:24:22.741 DEBUG:boto:Path: /hiiqos-57/b 2023-07-29T00:24:22.741 DEBUG:boto:Data: 2023-07-29T00:24:22.742 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.742 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:22.743 DEBUG:boto:Port: 8001 2023-07-29T00:24:22.743 DEBUG:boto:Params: {} 2023-07-29T00:24:22.744 DEBUG:boto:Token: None 2023-07-29T00:24:22.745 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/b 2023-07-29T00:24:22.746 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FYXsrIW4DwwW/DEMuwgue399gK0= 2023-07-29T00:24:22.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FYXsrIW4DwwW/DEMuwgue399gK0=', 'Content-Length': '0'} 2023-07-29T00:24:22.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-57:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000742a637ac52a71f4-0064c45c36-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.792 DEBUG:boto:path=/hiiqos-57/b 2023-07-29T00:24:22.793 DEBUG:boto:auth_path=/hiiqos-57/b 2023-07-29T00:24:22.793 DEBUG:boto:Method: GET 2023-07-29T00:24:22.794 DEBUG:boto:Path: /hiiqos-57/b 2023-07-29T00:24:22.794 DEBUG:boto:Data: 2023-07-29T00:24:22.794 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.795 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:22.795 DEBUG:boto:Port: 8000 2023-07-29T00:24:22.795 DEBUG:boto:Params: {} 2023-07-29T00:24:22.796 DEBUG:boto:Token: None 2023-07-29T00:24:22.796 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/b 2023-07-29T00:24:22.797 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FYXsrIW4DwwW/DEMuwgue399gK0= 2023-07-29T00:24:22.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FYXsrIW4DwwW/DEMuwgue399gK0=', 'Content-Length': '0'} 2023-07-29T00:24:22.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000969105cfbbe9d2f-0064c45c36-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.800 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:24:22.800 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:24:22.801 DEBUG:boto:path=/hiiqos-57/?prefix=b 2023-07-29T00:24:22.801 DEBUG:boto:auth_path=/hiiqos-57/?prefix=b 2023-07-29T00:24:22.801 DEBUG:boto:Method: GET 2023-07-29T00:24:22.802 DEBUG:boto:Path: /hiiqos-57/?prefix=b 2023-07-29T00:24:22.803 DEBUG:boto:Data: 2023-07-29T00:24:22.804 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.804 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:22.805 DEBUG:boto:Port: 8001 2023-07-29T00:24:22.805 DEBUG:boto:Params: {} 2023-07-29T00:24:22.806 DEBUG:boto:Token: None 2023-07-29T00:24:22.806 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/ 2023-07-29T00:24:22.807 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OKCZY2qj6scFWJ1bIWz07KyqoOY= 2023-07-29T00:24:22.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OKCZY2qj6scFWJ1bIWz07KyqoOY=', 'Content-Length': '0'} 2023-07-29T00:24:22.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051a14e62a3be6a4c-0064c45c36-1b40-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.810 DEBUG:boto:b'hiiqos-57b1000falseb2023-07-29T00:21:40.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:22.811 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:24:22.812 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:24:22.812 DEBUG:boto:path=/hiiqos-57/?prefix=b 2023-07-29T00:24:22.812 DEBUG:boto:auth_path=/hiiqos-57/?prefix=b 2023-07-29T00:24:22.813 DEBUG:boto:Method: GET 2023-07-29T00:24:22.813 DEBUG:boto:Path: /hiiqos-57/?prefix=b 2023-07-29T00:24:22.814 DEBUG:boto:Data: 2023-07-29T00:24:22.814 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.814 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:22.815 DEBUG:boto:Port: 8000 2023-07-29T00:24:22.815 DEBUG:boto:Params: {} 2023-07-29T00:24:22.816 DEBUG:boto:Token: None 2023-07-29T00:24:22.816 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/ 2023-07-29T00:24:22.817 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OKCZY2qj6scFWJ1bIWz07KyqoOY= 2023-07-29T00:24:22.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OKCZY2qj6scFWJ1bIWz07KyqoOY=', 'Content-Length': '0'} 2023-07-29T00:24:22.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000559bd09c7acbc125-0064c45c36-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.820 DEBUG:boto:b'hiiqos-57b1000falseb2023-07-29T00:21:40.531Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:22.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-07-29T00:24:22.821 DEBUG:boto:path=/hiiqos-57/c 2023-07-29T00:24:22.822 DEBUG:boto:auth_path=/hiiqos-57/c 2023-07-29T00:24:22.822 DEBUG:boto:path=/hiiqos-57/c?versionId=null 2023-07-29T00:24:22.823 DEBUG:boto:auth_path=/hiiqos-57/c?versionId=null 2023-07-29T00:24:22.823 DEBUG:boto:Method: GET 2023-07-29T00:24:22.824 DEBUG:boto:Path: /hiiqos-57/c?versionId=null 2023-07-29T00:24:22.825 DEBUG:boto:Data: 2023-07-29T00:24:22.825 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.826 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:22.826 DEBUG:boto:Port: 8001 2023-07-29T00:24:22.826 DEBUG:boto:Params: {} 2023-07-29T00:24:22.827 DEBUG:boto:Token: None 2023-07-29T00:24:22.827 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/c?versionId=null 2023-07-29T00:24:22.828 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tRPLZbfX3UbYh3xi0WMtc5mGvfc= 2023-07-29T00:24:22.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tRPLZbfX3UbYh3xi0WMtc5mGvfc=', 'Content-Length': '0'} 2023-07-29T00:24:22.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-57:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030233e5a73190a55-0064c45c36-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.831 DEBUG:boto:path=/hiiqos-57/c 2023-07-29T00:24:22.832 DEBUG:boto:auth_path=/hiiqos-57/c 2023-07-29T00:24:22.833 DEBUG:boto:path=/hiiqos-57/c?versionId=null 2023-07-29T00:24:22.833 DEBUG:boto:auth_path=/hiiqos-57/c?versionId=null 2023-07-29T00:24:22.834 DEBUG:boto:Method: GET 2023-07-29T00:24:22.834 DEBUG:boto:Path: /hiiqos-57/c?versionId=null 2023-07-29T00:24:22.834 DEBUG:boto:Data: 2023-07-29T00:24:22.835 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.835 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:22.836 DEBUG:boto:Port: 8000 2023-07-29T00:24:22.836 DEBUG:boto:Params: {} 2023-07-29T00:24:22.837 DEBUG:boto:Token: None 2023-07-29T00:24:22.837 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/c?versionId=null 2023-07-29T00:24:22.838 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tRPLZbfX3UbYh3xi0WMtc5mGvfc= 2023-07-29T00:24:22.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tRPLZbfX3UbYh3xi0WMtc5mGvfc=', 'Content-Length': '0'} 2023-07-29T00:24:22.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000208775c8386e0879-0064c45c36-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.841 DEBUG:boto:path=/hiiqos-57/c 2023-07-29T00:24:22.841 DEBUG:boto:auth_path=/hiiqos-57/c 2023-07-29T00:24:22.842 DEBUG:boto:path=/hiiqos-57/c?versionId=null 2023-07-29T00:24:22.842 DEBUG:boto:auth_path=/hiiqos-57/c?versionId=null 2023-07-29T00:24:22.843 DEBUG:boto:Method: HEAD 2023-07-29T00:24:22.843 DEBUG:boto:Path: /hiiqos-57/c?versionId=null 2023-07-29T00:24:22.844 DEBUG:boto:Data: 2023-07-29T00:24:22.844 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.845 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:22.845 DEBUG:boto:Port: 8001 2023-07-29T00:24:22.845 DEBUG:boto:Params: {} 2023-07-29T00:24:22.846 DEBUG:boto:Token: None 2023-07-29T00:24:22.846 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/c?versionId=null 2023-07-29T00:24:22.847 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:W+J5JzOgIxoWqmM2agmZSdRwBFM= 2023-07-29T00:24:22.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:W+J5JzOgIxoWqmM2agmZSdRwBFM=', 'Content-Length': '0'} 2023-07-29T00:24:22.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-57:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5e5dae0b3763667-0064c45c36-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.850 DEBUG:boto:path=/hiiqos-57/c 2023-07-29T00:24:22.851 DEBUG:boto:auth_path=/hiiqos-57/c 2023-07-29T00:24:22.851 DEBUG:boto:path=/hiiqos-57/c?versionId=null 2023-07-29T00:24:22.852 DEBUG:boto:auth_path=/hiiqos-57/c?versionId=null 2023-07-29T00:24:22.853 DEBUG:boto:Method: HEAD 2023-07-29T00:24:22.853 DEBUG:boto:Path: /hiiqos-57/c?versionId=null 2023-07-29T00:24:22.854 DEBUG:boto:Data: 2023-07-29T00:24:22.855 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.855 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:22.856 DEBUG:boto:Port: 8000 2023-07-29T00:24:22.856 DEBUG:boto:Params: {} 2023-07-29T00:24:22.857 DEBUG:boto:Token: None 2023-07-29T00:24:22.857 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/c?versionId=null 2023-07-29T00:24:22.858 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:W+J5JzOgIxoWqmM2agmZSdRwBFM= 2023-07-29T00:24:22.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:W+J5JzOgIxoWqmM2agmZSdRwBFM=', 'Content-Length': '0'} 2023-07-29T00:24:22.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024cb550baf4c20dc-0064c45c36-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-07-29T00:24:22.861 DEBUG:boto:path=/hiiqos-57/c 2023-07-29T00:24:22.861 DEBUG:boto:auth_path=/hiiqos-57/c 2023-07-29T00:24:22.862 DEBUG:boto:path=/hiiqos-57/c?versionId=null 2023-07-29T00:24:22.862 DEBUG:boto:auth_path=/hiiqos-57/c?versionId=null 2023-07-29T00:24:22.862 DEBUG:boto:Method: GET 2023-07-29T00:24:22.863 DEBUG:boto:Path: /hiiqos-57/c?versionId=null 2023-07-29T00:24:22.863 DEBUG:boto:Data: 2023-07-29T00:24:22.863 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.864 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:22.864 DEBUG:boto:Port: 8001 2023-07-29T00:24:22.865 DEBUG:boto:Params: {} 2023-07-29T00:24:22.865 DEBUG:boto:Token: None 2023-07-29T00:24:22.865 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/c?versionId=null 2023-07-29T00:24:22.866 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tRPLZbfX3UbYh3xi0WMtc5mGvfc= 2023-07-29T00:24:22.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tRPLZbfX3UbYh3xi0WMtc5mGvfc=', 'Content-Length': '0'} 2023-07-29T00:24:22.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-57:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a694c004454f8db-0064c45c36-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.912 DEBUG:boto:path=/hiiqos-57/c 2023-07-29T00:24:22.913 DEBUG:boto:auth_path=/hiiqos-57/c 2023-07-29T00:24:22.913 DEBUG:boto:path=/hiiqos-57/c?versionId=null 2023-07-29T00:24:22.914 DEBUG:boto:auth_path=/hiiqos-57/c?versionId=null 2023-07-29T00:24:22.914 DEBUG:boto:Method: GET 2023-07-29T00:24:22.914 DEBUG:boto:Path: /hiiqos-57/c?versionId=null 2023-07-29T00:24:22.915 DEBUG:boto:Data: 2023-07-29T00:24:22.915 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.915 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:22.916 DEBUG:boto:Port: 8000 2023-07-29T00:24:22.916 DEBUG:boto:Params: {} 2023-07-29T00:24:22.917 DEBUG:boto:Token: None 2023-07-29T00:24:22.917 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/c?versionId=null 2023-07-29T00:24:22.917 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tRPLZbfX3UbYh3xi0WMtc5mGvfc= 2023-07-29T00:24:22.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tRPLZbfX3UbYh3xi0WMtc5mGvfc=', 'Content-Length': '0'} 2023-07-29T00:24:22.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a1e58a9f425f1c3-0064c45c36-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.920 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:24:22.921 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:24:22.921 DEBUG:boto:path=/hiiqos-57/?prefix=c 2023-07-29T00:24:22.922 DEBUG:boto:auth_path=/hiiqos-57/?prefix=c 2023-07-29T00:24:22.922 DEBUG:boto:Method: GET 2023-07-29T00:24:22.923 DEBUG:boto:Path: /hiiqos-57/?prefix=c 2023-07-29T00:24:22.923 DEBUG:boto:Data: 2023-07-29T00:24:22.923 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.924 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:22.924 DEBUG:boto:Port: 8001 2023-07-29T00:24:22.925 DEBUG:boto:Params: {} 2023-07-29T00:24:22.925 DEBUG:boto:Token: None 2023-07-29T00:24:22.925 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/ 2023-07-29T00:24:22.926 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OKCZY2qj6scFWJ1bIWz07KyqoOY= 2023-07-29T00:24:22.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OKCZY2qj6scFWJ1bIWz07KyqoOY=', 'Content-Length': '0'} 2023-07-29T00:24:22.928 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a4e9bbe546ce3cb-0064c45c36-1b40-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.929 DEBUG:boto:b'hiiqos-57c1000falsec2023-07-29T00:21:40.555Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:22.930 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:24:22.930 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:24:22.930 DEBUG:boto:path=/hiiqos-57/?prefix=c 2023-07-29T00:24:22.931 DEBUG:boto:auth_path=/hiiqos-57/?prefix=c 2023-07-29T00:24:22.931 DEBUG:boto:Method: GET 2023-07-29T00:24:22.932 DEBUG:boto:Path: /hiiqos-57/?prefix=c 2023-07-29T00:24:22.932 DEBUG:boto:Data: 2023-07-29T00:24:22.933 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.933 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:22.933 DEBUG:boto:Port: 8000 2023-07-29T00:24:22.934 DEBUG:boto:Params: {} 2023-07-29T00:24:22.934 DEBUG:boto:Token: None 2023-07-29T00:24:22.935 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/ 2023-07-29T00:24:22.935 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OKCZY2qj6scFWJ1bIWz07KyqoOY= 2023-07-29T00:24:22.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OKCZY2qj6scFWJ1bIWz07KyqoOY=', 'Content-Length': '0'} 2023-07-29T00:24:22.937 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf0b154085178b01-0064c45c36-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.938 DEBUG:boto:b'hiiqos-57c1000falsec2023-07-29T00:21:40.555Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:22.939 DEBUG:boto:path=/hiiqos-57/c 2023-07-29T00:24:22.939 DEBUG:boto:auth_path=/hiiqos-57/c 2023-07-29T00:24:22.939 DEBUG:boto:Method: HEAD 2023-07-29T00:24:22.940 DEBUG:boto:Path: /hiiqos-57/c 2023-07-29T00:24:22.940 DEBUG:boto:Data: 2023-07-29T00:24:22.940 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.941 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:22.941 DEBUG:boto:Port: 8001 2023-07-29T00:24:22.941 DEBUG:boto:Params: {} 2023-07-29T00:24:22.942 DEBUG:boto:Token: None 2023-07-29T00:24:22.942 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/c 2023-07-29T00:24:22.943 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hKncaBjqAuDSY35yfmbkhA7NqPQ= 2023-07-29T00:24:22.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hKncaBjqAuDSY35yfmbkhA7NqPQ=', 'Content-Length': '0'} 2023-07-29T00:24:22.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-57:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb9125f87e66f782-0064c45c36-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.946 DEBUG:boto:path=/hiiqos-57/c 2023-07-29T00:24:22.946 DEBUG:boto:auth_path=/hiiqos-57/c 2023-07-29T00:24:22.947 DEBUG:boto:Method: HEAD 2023-07-29T00:24:22.947 DEBUG:boto:Path: /hiiqos-57/c 2023-07-29T00:24:22.947 DEBUG:boto:Data: 2023-07-29T00:24:22.948 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.948 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:22.948 DEBUG:boto:Port: 8000 2023-07-29T00:24:22.949 DEBUG:boto:Params: {} 2023-07-29T00:24:22.949 DEBUG:boto:Token: None 2023-07-29T00:24:22.949 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/c 2023-07-29T00:24:22.950 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hKncaBjqAuDSY35yfmbkhA7NqPQ= 2023-07-29T00:24:22.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hKncaBjqAuDSY35yfmbkhA7NqPQ=', 'Content-Length': '0'} 2023-07-29T00:24:22.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e75bb8c250126d86-0064c45c36-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:22.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-07-29T00:24:22.953 DEBUG:boto:path=/hiiqos-57/c 2023-07-29T00:24:22.954 DEBUG:boto:auth_path=/hiiqos-57/c 2023-07-29T00:24:22.954 DEBUG:boto:Method: GET 2023-07-29T00:24:22.954 DEBUG:boto:Path: /hiiqos-57/c 2023-07-29T00:24:22.955 DEBUG:boto:Data: 2023-07-29T00:24:22.955 DEBUG:boto:Headers: {} 2023-07-29T00:24:22.956 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:22.956 DEBUG:boto:Port: 8001 2023-07-29T00:24:22.956 DEBUG:boto:Params: {} 2023-07-29T00:24:22.957 DEBUG:boto:Token: None 2023-07-29T00:24:22.957 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:22 GMT /hiiqos-57/c 2023-07-29T00:24:22.958 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:p4Ry77dG1exs+r+QI+eyNaRhSVA= 2023-07-29T00:24:22.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:p4Ry77dG1exs+r+QI+eyNaRhSVA=', 'Content-Length': '0'} 2023-07-29T00:24:22.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-57:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000697ebf9c5ab22d31-0064c45c36-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.000 DEBUG:boto:path=/hiiqos-57/c 2023-07-29T00:24:23.001 DEBUG:boto:auth_path=/hiiqos-57/c 2023-07-29T00:24:23.001 DEBUG:boto:Method: GET 2023-07-29T00:24:23.002 DEBUG:boto:Path: /hiiqos-57/c 2023-07-29T00:24:23.002 DEBUG:boto:Data: 2023-07-29T00:24:23.002 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.003 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:23.003 DEBUG:boto:Port: 8000 2023-07-29T00:24:23.003 DEBUG:boto:Params: {} 2023-07-29T00:24:23.004 DEBUG:boto:Token: None 2023-07-29T00:24:23.005 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/c 2023-07-29T00:24:23.005 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kafV7JfEZPMUm6SCEbWMc9h5F50= 2023-07-29T00:24:23.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kafV7JfEZPMUm6SCEbWMc9h5F50=', 'Content-Length': '0'} 2023-07-29T00:24:23.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b638bfe0b69a6224-0064c45c36-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.008 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:24:23.009 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:24:23.009 DEBUG:boto:path=/hiiqos-57/?prefix=c 2023-07-29T00:24:23.009 DEBUG:boto:auth_path=/hiiqos-57/?prefix=c 2023-07-29T00:24:23.010 DEBUG:boto:Method: GET 2023-07-29T00:24:23.010 DEBUG:boto:Path: /hiiqos-57/?prefix=c 2023-07-29T00:24:23.010 DEBUG:boto:Data: 2023-07-29T00:24:23.011 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.011 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:23.011 DEBUG:boto:Port: 8001 2023-07-29T00:24:23.012 DEBUG:boto:Params: {} 2023-07-29T00:24:23.012 DEBUG:boto:Token: None 2023-07-29T00:24:23.013 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/ 2023-07-29T00:24:23.013 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5xXylTj5tC8sh36uFSqkRh09kgM= 2023-07-29T00:24:23.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5xXylTj5tC8sh36uFSqkRh09kgM=', 'Content-Length': '0'} 2023-07-29T00:24:23.016 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec2935845a500a2d-0064c45c36-1b40-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.016 DEBUG:boto:b'hiiqos-57c1000falsec2023-07-29T00:21:40.555Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:23.017 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:24:23.017 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:24:23.018 DEBUG:boto:path=/hiiqos-57/?prefix=c 2023-07-29T00:24:23.018 DEBUG:boto:auth_path=/hiiqos-57/?prefix=c 2023-07-29T00:24:23.019 DEBUG:boto:Method: GET 2023-07-29T00:24:23.019 DEBUG:boto:Path: /hiiqos-57/?prefix=c 2023-07-29T00:24:23.019 DEBUG:boto:Data: 2023-07-29T00:24:23.020 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.020 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:23.020 DEBUG:boto:Port: 8000 2023-07-29T00:24:23.021 DEBUG:boto:Params: {} 2023-07-29T00:24:23.021 DEBUG:boto:Token: None 2023-07-29T00:24:23.021 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/ 2023-07-29T00:24:23.022 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5xXylTj5tC8sh36uFSqkRh09kgM= 2023-07-29T00:24:23.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5xXylTj5tC8sh36uFSqkRh09kgM=', 'Content-Length': '0'} 2023-07-29T00:24:23.024 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069af066c1ce2d3dd-0064c45c36-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.025 DEBUG:boto:b'hiiqos-57c1000falsec2023-07-29T00:21:40.555Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:23.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-07-29T00:24:23.026 DEBUG:boto:path=/hiiqos-57/d 2023-07-29T00:24:23.026 DEBUG:boto:auth_path=/hiiqos-57/d 2023-07-29T00:24:23.027 DEBUG:boto:path=/hiiqos-57/d?versionId=null 2023-07-29T00:24:23.027 DEBUG:boto:auth_path=/hiiqos-57/d?versionId=null 2023-07-29T00:24:23.027 DEBUG:boto:Method: GET 2023-07-29T00:24:23.028 DEBUG:boto:Path: /hiiqos-57/d?versionId=null 2023-07-29T00:24:23.028 DEBUG:boto:Data: 2023-07-29T00:24:23.029 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.029 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:23.029 DEBUG:boto:Port: 8001 2023-07-29T00:24:23.030 DEBUG:boto:Params: {} 2023-07-29T00:24:23.030 DEBUG:boto:Token: None 2023-07-29T00:24:23.030 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/d?versionId=null 2023-07-29T00:24:23.031 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3uF0VtQixRFK0e7e9urzsvIY+Tk= 2023-07-29T00:24:23.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3uF0VtQixRFK0e7e9urzsvIY+Tk=', 'Content-Length': '0'} 2023-07-29T00:24:23.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-57:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c4ce8a88949d8e2-0064c45c36-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.034 DEBUG:boto:path=/hiiqos-57/d 2023-07-29T00:24:23.034 DEBUG:boto:auth_path=/hiiqos-57/d 2023-07-29T00:24:23.035 DEBUG:boto:path=/hiiqos-57/d?versionId=null 2023-07-29T00:24:23.035 DEBUG:boto:auth_path=/hiiqos-57/d?versionId=null 2023-07-29T00:24:23.035 DEBUG:boto:Method: GET 2023-07-29T00:24:23.036 DEBUG:boto:Path: /hiiqos-57/d?versionId=null 2023-07-29T00:24:23.036 DEBUG:boto:Data: 2023-07-29T00:24:23.036 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.037 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:23.037 DEBUG:boto:Port: 8000 2023-07-29T00:24:23.037 DEBUG:boto:Params: {} 2023-07-29T00:24:23.038 DEBUG:boto:Token: None 2023-07-29T00:24:23.038 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/d?versionId=null 2023-07-29T00:24:23.039 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3uF0VtQixRFK0e7e9urzsvIY+Tk= 2023-07-29T00:24:23.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3uF0VtQixRFK0e7e9urzsvIY+Tk=', 'Content-Length': '0'} 2023-07-29T00:24:23.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c5f4f3c79193bbd-0064c45c36-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.084 DEBUG:boto:path=/hiiqos-57/d 2023-07-29T00:24:23.085 DEBUG:boto:auth_path=/hiiqos-57/d 2023-07-29T00:24:23.086 DEBUG:boto:path=/hiiqos-57/d?versionId=null 2023-07-29T00:24:23.086 DEBUG:boto:auth_path=/hiiqos-57/d?versionId=null 2023-07-29T00:24:23.086 DEBUG:boto:Method: HEAD 2023-07-29T00:24:23.087 DEBUG:boto:Path: /hiiqos-57/d?versionId=null 2023-07-29T00:24:23.087 DEBUG:boto:Data: 2023-07-29T00:24:23.087 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.088 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:23.088 DEBUG:boto:Port: 8001 2023-07-29T00:24:23.088 DEBUG:boto:Params: {} 2023-07-29T00:24:23.089 DEBUG:boto:Token: None 2023-07-29T00:24:23.089 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/d?versionId=null 2023-07-29T00:24:23.090 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6V9rqxTiU0ogi5EjGBIndUQQCaM= 2023-07-29T00:24:23.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6V9rqxTiU0ogi5EjGBIndUQQCaM=', 'Content-Length': '0'} 2023-07-29T00:24:23.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-57:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acc19c88031395b4-0064c45c37-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.093 DEBUG:boto:path=/hiiqos-57/d 2023-07-29T00:24:23.093 DEBUG:boto:auth_path=/hiiqos-57/d 2023-07-29T00:24:23.094 DEBUG:boto:path=/hiiqos-57/d?versionId=null 2023-07-29T00:24:23.094 DEBUG:boto:auth_path=/hiiqos-57/d?versionId=null 2023-07-29T00:24:23.094 DEBUG:boto:Method: HEAD 2023-07-29T00:24:23.095 DEBUG:boto:Path: /hiiqos-57/d?versionId=null 2023-07-29T00:24:23.095 DEBUG:boto:Data: 2023-07-29T00:24:23.095 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.096 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:23.096 DEBUG:boto:Port: 8000 2023-07-29T00:24:23.096 DEBUG:boto:Params: {} 2023-07-29T00:24:23.097 DEBUG:boto:Token: None 2023-07-29T00:24:23.097 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/d?versionId=null 2023-07-29T00:24:23.098 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6V9rqxTiU0ogi5EjGBIndUQQCaM= 2023-07-29T00:24:23.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6V9rqxTiU0ogi5EjGBIndUQQCaM=', 'Content-Length': '0'} 2023-07-29T00:24:23.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c91f461b23ed7397-0064c45c37-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-07-29T00:24:23.101 DEBUG:boto:path=/hiiqos-57/d 2023-07-29T00:24:23.102 DEBUG:boto:auth_path=/hiiqos-57/d 2023-07-29T00:24:23.102 DEBUG:boto:path=/hiiqos-57/d?versionId=null 2023-07-29T00:24:23.102 DEBUG:boto:auth_path=/hiiqos-57/d?versionId=null 2023-07-29T00:24:23.103 DEBUG:boto:Method: GET 2023-07-29T00:24:23.103 DEBUG:boto:Path: /hiiqos-57/d?versionId=null 2023-07-29T00:24:23.104 DEBUG:boto:Data: 2023-07-29T00:24:23.104 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.104 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:23.105 DEBUG:boto:Port: 8001 2023-07-29T00:24:23.105 DEBUG:boto:Params: {} 2023-07-29T00:24:23.106 DEBUG:boto:Token: None 2023-07-29T00:24:23.106 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/d?versionId=null 2023-07-29T00:24:23.106 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3uF0VtQixRFK0e7e9urzsvIY+Tk= 2023-07-29T00:24:23.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3uF0VtQixRFK0e7e9urzsvIY+Tk=', 'Content-Length': '0'} 2023-07-29T00:24:23.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-57:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cd41190da4bacba-0064c45c37-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.110 DEBUG:boto:path=/hiiqos-57/d 2023-07-29T00:24:23.110 DEBUG:boto:auth_path=/hiiqos-57/d 2023-07-29T00:24:23.110 DEBUG:boto:path=/hiiqos-57/d?versionId=null 2023-07-29T00:24:23.111 DEBUG:boto:auth_path=/hiiqos-57/d?versionId=null 2023-07-29T00:24:23.111 DEBUG:boto:Method: GET 2023-07-29T00:24:23.111 DEBUG:boto:Path: /hiiqos-57/d?versionId=null 2023-07-29T00:24:23.112 DEBUG:boto:Data: 2023-07-29T00:24:23.112 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.113 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:23.113 DEBUG:boto:Port: 8000 2023-07-29T00:24:23.113 DEBUG:boto:Params: {} 2023-07-29T00:24:23.114 DEBUG:boto:Token: None 2023-07-29T00:24:23.114 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/d?versionId=null 2023-07-29T00:24:23.115 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3uF0VtQixRFK0e7e9urzsvIY+Tk= 2023-07-29T00:24:23.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3uF0VtQixRFK0e7e9urzsvIY+Tk=', 'Content-Length': '0'} 2023-07-29T00:24:23.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000db27fa74cebec9e-0064c45c37-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.118 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:24:23.118 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:24:23.119 DEBUG:boto:path=/hiiqos-57/?prefix=d 2023-07-29T00:24:23.119 DEBUG:boto:auth_path=/hiiqos-57/?prefix=d 2023-07-29T00:24:23.119 DEBUG:boto:Method: GET 2023-07-29T00:24:23.120 DEBUG:boto:Path: /hiiqos-57/?prefix=d 2023-07-29T00:24:23.120 DEBUG:boto:Data: 2023-07-29T00:24:23.121 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.121 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:23.121 DEBUG:boto:Port: 8001 2023-07-29T00:24:23.122 DEBUG:boto:Params: {} 2023-07-29T00:24:23.122 DEBUG:boto:Token: None 2023-07-29T00:24:23.122 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/ 2023-07-29T00:24:23.123 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5xXylTj5tC8sh36uFSqkRh09kgM= 2023-07-29T00:24:23.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5xXylTj5tC8sh36uFSqkRh09kgM=', 'Content-Length': '0'} 2023-07-29T00:24:23.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094d1eedc71ec456e-0064c45c37-1b40-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.126 DEBUG:boto:b'hiiqos-57d1000falsed2023-07-29T00:21:40.612Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:23.127 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:24:23.128 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:24:23.128 DEBUG:boto:path=/hiiqos-57/?prefix=d 2023-07-29T00:24:23.128 DEBUG:boto:auth_path=/hiiqos-57/?prefix=d 2023-07-29T00:24:23.129 DEBUG:boto:Method: GET 2023-07-29T00:24:23.129 DEBUG:boto:Path: /hiiqos-57/?prefix=d 2023-07-29T00:24:23.129 DEBUG:boto:Data: 2023-07-29T00:24:23.130 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.130 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:23.130 DEBUG:boto:Port: 8000 2023-07-29T00:24:23.131 DEBUG:boto:Params: {} 2023-07-29T00:24:23.131 DEBUG:boto:Token: None 2023-07-29T00:24:23.132 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/ 2023-07-29T00:24:23.132 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5xXylTj5tC8sh36uFSqkRh09kgM= 2023-07-29T00:24:23.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5xXylTj5tC8sh36uFSqkRh09kgM=', 'Content-Length': '0'} 2023-07-29T00:24:23.135 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5f2254be11721ba-0064c45c37-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.176 DEBUG:boto:b'hiiqos-57d1000falsed2023-07-29T00:21:40.612Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:23.177 DEBUG:boto:path=/hiiqos-57/d 2023-07-29T00:24:23.177 DEBUG:boto:auth_path=/hiiqos-57/d 2023-07-29T00:24:23.178 DEBUG:boto:Method: HEAD 2023-07-29T00:24:23.178 DEBUG:boto:Path: /hiiqos-57/d 2023-07-29T00:24:23.179 DEBUG:boto:Data: 2023-07-29T00:24:23.179 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.179 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:23.180 DEBUG:boto:Port: 8001 2023-07-29T00:24:23.180 DEBUG:boto:Params: {} 2023-07-29T00:24:23.180 DEBUG:boto:Token: None 2023-07-29T00:24:23.181 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/d 2023-07-29T00:24:23.181 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:X3kIrAKChU5bimnaDDj/GN+kfEU= 2023-07-29T00:24:23.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:X3kIrAKChU5bimnaDDj/GN+kfEU=', 'Content-Length': '0'} 2023-07-29T00:24:23.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-57:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ded8246f05230bcd-0064c45c37-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.184 DEBUG:boto:path=/hiiqos-57/d 2023-07-29T00:24:23.185 DEBUG:boto:auth_path=/hiiqos-57/d 2023-07-29T00:24:23.185 DEBUG:boto:Method: HEAD 2023-07-29T00:24:23.185 DEBUG:boto:Path: /hiiqos-57/d 2023-07-29T00:24:23.186 DEBUG:boto:Data: 2023-07-29T00:24:23.186 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.186 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:23.187 DEBUG:boto:Port: 8000 2023-07-29T00:24:23.187 DEBUG:boto:Params: {} 2023-07-29T00:24:23.187 DEBUG:boto:Token: None 2023-07-29T00:24:23.188 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/d 2023-07-29T00:24:23.189 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:X3kIrAKChU5bimnaDDj/GN+kfEU= 2023-07-29T00:24:23.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:X3kIrAKChU5bimnaDDj/GN+kfEU=', 'Content-Length': '0'} 2023-07-29T00:24:23.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a55d1d82731e5c0-0064c45c37-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-07-29T00:24:23.192 DEBUG:boto:path=/hiiqos-57/d 2023-07-29T00:24:23.193 DEBUG:boto:auth_path=/hiiqos-57/d 2023-07-29T00:24:23.193 DEBUG:boto:Method: GET 2023-07-29T00:24:23.194 DEBUG:boto:Path: /hiiqos-57/d 2023-07-29T00:24:23.194 DEBUG:boto:Data: 2023-07-29T00:24:23.194 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.195 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:23.195 DEBUG:boto:Port: 8001 2023-07-29T00:24:23.195 DEBUG:boto:Params: {} 2023-07-29T00:24:23.196 DEBUG:boto:Token: None 2023-07-29T00:24:23.196 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/d 2023-07-29T00:24:23.197 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4awQC6EIp/ZvND0VQbfIqVm8p3k= 2023-07-29T00:24:23.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4awQC6EIp/ZvND0VQbfIqVm8p3k=', 'Content-Length': '0'} 2023-07-29T00:24:23.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-57:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fa1652c9b6e5ca3-0064c45c37-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.200 DEBUG:boto:path=/hiiqos-57/d 2023-07-29T00:24:23.201 DEBUG:boto:auth_path=/hiiqos-57/d 2023-07-29T00:24:23.201 DEBUG:boto:Method: GET 2023-07-29T00:24:23.201 DEBUG:boto:Path: /hiiqos-57/d 2023-07-29T00:24:23.202 DEBUG:boto:Data: 2023-07-29T00:24:23.202 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.202 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:23.203 DEBUG:boto:Port: 8000 2023-07-29T00:24:23.203 DEBUG:boto:Params: {} 2023-07-29T00:24:23.204 DEBUG:boto:Token: None 2023-07-29T00:24:23.204 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/d 2023-07-29T00:24:23.204 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4awQC6EIp/ZvND0VQbfIqVm8p3k= 2023-07-29T00:24:23.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4awQC6EIp/ZvND0VQbfIqVm8p3k=', 'Content-Length': '0'} 2023-07-29T00:24:23.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a564056bfdd8800-0064c45c37-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.207 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:24:23.208 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:24:23.208 DEBUG:boto:path=/hiiqos-57/?prefix=d 2023-07-29T00:24:23.208 DEBUG:boto:auth_path=/hiiqos-57/?prefix=d 2023-07-29T00:24:23.209 DEBUG:boto:Method: GET 2023-07-29T00:24:23.209 DEBUG:boto:Path: /hiiqos-57/?prefix=d 2023-07-29T00:24:23.210 DEBUG:boto:Data: 2023-07-29T00:24:23.210 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.210 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:23.211 DEBUG:boto:Port: 8001 2023-07-29T00:24:23.211 DEBUG:boto:Params: {} 2023-07-29T00:24:23.212 DEBUG:boto:Token: None 2023-07-29T00:24:23.212 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/ 2023-07-29T00:24:23.213 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5xXylTj5tC8sh36uFSqkRh09kgM= 2023-07-29T00:24:23.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5xXylTj5tC8sh36uFSqkRh09kgM=', 'Content-Length': '0'} 2023-07-29T00:24:23.216 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bc8421921b79d06-0064c45c37-1b40-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.216 DEBUG:boto:b'hiiqos-57d1000falsed2023-07-29T00:21:40.612Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:23.217 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:24:23.217 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:24:23.218 DEBUG:boto:path=/hiiqos-57/?prefix=d 2023-07-29T00:24:23.218 DEBUG:boto:auth_path=/hiiqos-57/?prefix=d 2023-07-29T00:24:23.218 DEBUG:boto:Method: GET 2023-07-29T00:24:23.219 DEBUG:boto:Path: /hiiqos-57/?prefix=d 2023-07-29T00:24:23.219 DEBUG:boto:Data: 2023-07-29T00:24:23.219 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.220 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:23.220 DEBUG:boto:Port: 8000 2023-07-29T00:24:23.220 DEBUG:boto:Params: {} 2023-07-29T00:24:23.221 DEBUG:boto:Token: None 2023-07-29T00:24:23.222 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/ 2023-07-29T00:24:23.222 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5xXylTj5tC8sh36uFSqkRh09kgM= 2023-07-29T00:24:23.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5xXylTj5tC8sh36uFSqkRh09kgM=', 'Content-Length': '0'} 2023-07-29T00:24:23.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004751e33b84f5b9b-0064c45c37-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.225 DEBUG:boto:b'hiiqos-57d1000falsed2023-07-29T00:21:40.612Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:23.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-07-29T00:24:23.226 DEBUG:boto:path=/hiiqos-57/e 2023-07-29T00:24:23.226 DEBUG:boto:auth_path=/hiiqos-57/e 2023-07-29T00:24:23.227 DEBUG:boto:path=/hiiqos-57/e?versionId=null 2023-07-29T00:24:23.227 DEBUG:boto:auth_path=/hiiqos-57/e?versionId=null 2023-07-29T00:24:23.227 DEBUG:boto:Method: GET 2023-07-29T00:24:23.228 DEBUG:boto:Path: /hiiqos-57/e?versionId=null 2023-07-29T00:24:23.228 DEBUG:boto:Data: 2023-07-29T00:24:23.229 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.229 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:23.229 DEBUG:boto:Port: 8001 2023-07-29T00:24:23.230 DEBUG:boto:Params: {} 2023-07-29T00:24:23.230 DEBUG:boto:Token: None 2023-07-29T00:24:23.231 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/e?versionId=null 2023-07-29T00:24:23.231 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:77zWolXrJow05AchPdQ61rw9A48= 2023-07-29T00:24:23.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:77zWolXrJow05AchPdQ61rw9A48=', 'Content-Length': '0'} 2023-07-29T00:24:23.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-57:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089b541328ec408e4-0064c45c37-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.234 DEBUG:boto:path=/hiiqos-57/e 2023-07-29T00:24:23.235 DEBUG:boto:auth_path=/hiiqos-57/e 2023-07-29T00:24:23.235 DEBUG:boto:path=/hiiqos-57/e?versionId=null 2023-07-29T00:24:23.236 DEBUG:boto:auth_path=/hiiqos-57/e?versionId=null 2023-07-29T00:24:23.236 DEBUG:boto:Method: GET 2023-07-29T00:24:23.236 DEBUG:boto:Path: /hiiqos-57/e?versionId=null 2023-07-29T00:24:23.237 DEBUG:boto:Data: 2023-07-29T00:24:23.237 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.237 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:23.238 DEBUG:boto:Port: 8000 2023-07-29T00:24:23.238 DEBUG:boto:Params: {} 2023-07-29T00:24:23.238 DEBUG:boto:Token: None 2023-07-29T00:24:23.239 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/e?versionId=null 2023-07-29T00:24:23.239 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:77zWolXrJow05AchPdQ61rw9A48= 2023-07-29T00:24:23.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:77zWolXrJow05AchPdQ61rw9A48=', 'Content-Length': '0'} 2023-07-29T00:24:23.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8a4f7e70a8e4c7b-0064c45c37-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.243 DEBUG:boto:path=/hiiqos-57/e 2023-07-29T00:24:23.243 DEBUG:boto:auth_path=/hiiqos-57/e 2023-07-29T00:24:23.244 DEBUG:boto:path=/hiiqos-57/e?versionId=null 2023-07-29T00:24:23.244 DEBUG:boto:auth_path=/hiiqos-57/e?versionId=null 2023-07-29T00:24:23.244 DEBUG:boto:Method: HEAD 2023-07-29T00:24:23.245 DEBUG:boto:Path: /hiiqos-57/e?versionId=null 2023-07-29T00:24:23.245 DEBUG:boto:Data: 2023-07-29T00:24:23.245 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.246 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:23.246 DEBUG:boto:Port: 8001 2023-07-29T00:24:23.246 DEBUG:boto:Params: {} 2023-07-29T00:24:23.247 DEBUG:boto:Token: None 2023-07-29T00:24:23.247 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/e?versionId=null 2023-07-29T00:24:23.248 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TzGFGZgu+Hdu9ie8FXERtYrcbuo= 2023-07-29T00:24:23.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TzGFGZgu+Hdu9ie8FXERtYrcbuo=', 'Content-Length': '0'} 2023-07-29T00:24:23.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-57:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe551da892cbe4de-0064c45c37-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.250 DEBUG:boto:path=/hiiqos-57/e 2023-07-29T00:24:23.251 DEBUG:boto:auth_path=/hiiqos-57/e 2023-07-29T00:24:23.251 DEBUG:boto:path=/hiiqos-57/e?versionId=null 2023-07-29T00:24:23.251 DEBUG:boto:auth_path=/hiiqos-57/e?versionId=null 2023-07-29T00:24:23.252 DEBUG:boto:Method: HEAD 2023-07-29T00:24:23.252 DEBUG:boto:Path: /hiiqos-57/e?versionId=null 2023-07-29T00:24:23.252 DEBUG:boto:Data: 2023-07-29T00:24:23.253 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.253 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:23.253 DEBUG:boto:Port: 8000 2023-07-29T00:24:23.254 DEBUG:boto:Params: {} 2023-07-29T00:24:23.254 DEBUG:boto:Token: None 2023-07-29T00:24:23.255 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/e?versionId=null 2023-07-29T00:24:23.255 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TzGFGZgu+Hdu9ie8FXERtYrcbuo= 2023-07-29T00:24:23.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TzGFGZgu+Hdu9ie8FXERtYrcbuo=', 'Content-Length': '0'} 2023-07-29T00:24:23.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9a6d27267ed2e66-0064c45c37-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-07-29T00:24:23.258 DEBUG:boto:path=/hiiqos-57/e 2023-07-29T00:24:23.258 DEBUG:boto:auth_path=/hiiqos-57/e 2023-07-29T00:24:23.259 DEBUG:boto:path=/hiiqos-57/e?versionId=null 2023-07-29T00:24:23.259 DEBUG:boto:auth_path=/hiiqos-57/e?versionId=null 2023-07-29T00:24:23.259 DEBUG:boto:Method: GET 2023-07-29T00:24:23.260 DEBUG:boto:Path: /hiiqos-57/e?versionId=null 2023-07-29T00:24:23.260 DEBUG:boto:Data: 2023-07-29T00:24:23.261 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.261 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:23.261 DEBUG:boto:Port: 8001 2023-07-29T00:24:23.262 DEBUG:boto:Params: {} 2023-07-29T00:24:23.262 DEBUG:boto:Token: None 2023-07-29T00:24:23.262 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/e?versionId=null 2023-07-29T00:24:23.263 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:77zWolXrJow05AchPdQ61rw9A48= 2023-07-29T00:24:23.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:77zWolXrJow05AchPdQ61rw9A48=', 'Content-Length': '0'} 2023-07-29T00:24:23.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-57:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d5ff212169cd739-0064c45c37-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.266 DEBUG:boto:path=/hiiqos-57/e 2023-07-29T00:24:23.267 DEBUG:boto:auth_path=/hiiqos-57/e 2023-07-29T00:24:23.267 DEBUG:boto:path=/hiiqos-57/e?versionId=null 2023-07-29T00:24:23.267 DEBUG:boto:auth_path=/hiiqos-57/e?versionId=null 2023-07-29T00:24:23.268 DEBUG:boto:Method: GET 2023-07-29T00:24:23.268 DEBUG:boto:Path: /hiiqos-57/e?versionId=null 2023-07-29T00:24:23.269 DEBUG:boto:Data: 2023-07-29T00:24:23.269 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.269 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:23.270 DEBUG:boto:Port: 8000 2023-07-29T00:24:23.270 DEBUG:boto:Params: {} 2023-07-29T00:24:23.270 DEBUG:boto:Token: None 2023-07-29T00:24:23.271 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/e?versionId=null 2023-07-29T00:24:23.271 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:77zWolXrJow05AchPdQ61rw9A48= 2023-07-29T00:24:23.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:77zWolXrJow05AchPdQ61rw9A48=', 'Content-Length': '0'} 2023-07-29T00:24:23.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b1c3f512a931aa5-0064c45c37-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.316 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:24:23.317 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:24:23.317 DEBUG:boto:path=/hiiqos-57/?prefix=e 2023-07-29T00:24:23.318 DEBUG:boto:auth_path=/hiiqos-57/?prefix=e 2023-07-29T00:24:23.318 DEBUG:boto:Method: GET 2023-07-29T00:24:23.318 DEBUG:boto:Path: /hiiqos-57/?prefix=e 2023-07-29T00:24:23.319 DEBUG:boto:Data: 2023-07-29T00:24:23.319 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.319 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:23.320 DEBUG:boto:Port: 8001 2023-07-29T00:24:23.320 DEBUG:boto:Params: {} 2023-07-29T00:24:23.321 DEBUG:boto:Token: None 2023-07-29T00:24:23.321 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/ 2023-07-29T00:24:23.321 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5xXylTj5tC8sh36uFSqkRh09kgM= 2023-07-29T00:24:23.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5xXylTj5tC8sh36uFSqkRh09kgM=', 'Content-Length': '0'} 2023-07-29T00:24:23.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005698ffb1d3135ee0-0064c45c37-1b40-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.325 DEBUG:boto:b'hiiqos-57e1000falsee2023-07-29T00:21:40.767Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:23.326 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:24:23.327 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:24:23.328 DEBUG:boto:path=/hiiqos-57/?prefix=e 2023-07-29T00:24:23.329 DEBUG:boto:auth_path=/hiiqos-57/?prefix=e 2023-07-29T00:24:23.329 DEBUG:boto:Method: GET 2023-07-29T00:24:23.330 DEBUG:boto:Path: /hiiqos-57/?prefix=e 2023-07-29T00:24:23.331 DEBUG:boto:Data: 2023-07-29T00:24:23.331 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.331 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:23.332 DEBUG:boto:Port: 8000 2023-07-29T00:24:23.332 DEBUG:boto:Params: {} 2023-07-29T00:24:23.333 DEBUG:boto:Token: None 2023-07-29T00:24:23.333 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/ 2023-07-29T00:24:23.334 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5xXylTj5tC8sh36uFSqkRh09kgM= 2023-07-29T00:24:23.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5xXylTj5tC8sh36uFSqkRh09kgM=', 'Content-Length': '0'} 2023-07-29T00:24:23.336 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bf2f640683f950e-0064c45c37-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.337 DEBUG:boto:b'hiiqos-57e1000falsee2023-07-29T00:21:40.767Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:23.338 DEBUG:boto:path=/hiiqos-57/e 2023-07-29T00:24:23.338 DEBUG:boto:auth_path=/hiiqos-57/e 2023-07-29T00:24:23.339 DEBUG:boto:Method: HEAD 2023-07-29T00:24:23.339 DEBUG:boto:Path: /hiiqos-57/e 2023-07-29T00:24:23.339 DEBUG:boto:Data: 2023-07-29T00:24:23.340 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.340 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:23.341 DEBUG:boto:Port: 8001 2023-07-29T00:24:23.341 DEBUG:boto:Params: {} 2023-07-29T00:24:23.341 DEBUG:boto:Token: None 2023-07-29T00:24:23.342 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/e 2023-07-29T00:24:23.342 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HoXmpHpdcNegKh2JLmGJnSAl2nw= 2023-07-29T00:24:23.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HoXmpHpdcNegKh2JLmGJnSAl2nw=', 'Content-Length': '0'} 2023-07-29T00:24:23.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-57:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047c2338e1e85e13f-0064c45c37-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.345 DEBUG:boto:path=/hiiqos-57/e 2023-07-29T00:24:23.346 DEBUG:boto:auth_path=/hiiqos-57/e 2023-07-29T00:24:23.347 DEBUG:boto:Method: HEAD 2023-07-29T00:24:23.347 DEBUG:boto:Path: /hiiqos-57/e 2023-07-29T00:24:23.348 DEBUG:boto:Data: 2023-07-29T00:24:23.349 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.350 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:23.350 DEBUG:boto:Port: 8000 2023-07-29T00:24:23.351 DEBUG:boto:Params: {} 2023-07-29T00:24:23.351 DEBUG:boto:Token: None 2023-07-29T00:24:23.352 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/e 2023-07-29T00:24:23.353 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HoXmpHpdcNegKh2JLmGJnSAl2nw= 2023-07-29T00:24:23.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HoXmpHpdcNegKh2JLmGJnSAl2nw=', 'Content-Length': '0'} 2023-07-29T00:24:23.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000472efcc99ef3cd3-0064c45c37-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-07-29T00:24:23.356 DEBUG:boto:path=/hiiqos-57/e 2023-07-29T00:24:23.356 DEBUG:boto:auth_path=/hiiqos-57/e 2023-07-29T00:24:23.357 DEBUG:boto:Method: GET 2023-07-29T00:24:23.357 DEBUG:boto:Path: /hiiqos-57/e 2023-07-29T00:24:23.357 DEBUG:boto:Data: 2023-07-29T00:24:23.358 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.358 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:23.358 DEBUG:boto:Port: 8001 2023-07-29T00:24:23.359 DEBUG:boto:Params: {} 2023-07-29T00:24:23.359 DEBUG:boto:Token: None 2023-07-29T00:24:23.359 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/e 2023-07-29T00:24:23.360 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:iyy46eUlgqwABJsARStGOulOUZo= 2023-07-29T00:24:23.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:iyy46eUlgqwABJsARStGOulOUZo=', 'Content-Length': '0'} 2023-07-29T00:24:23.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-57:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b6aa5173f4f7f89-0064c45c37-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.404 DEBUG:boto:path=/hiiqos-57/e 2023-07-29T00:24:23.405 DEBUG:boto:auth_path=/hiiqos-57/e 2023-07-29T00:24:23.405 DEBUG:boto:Method: GET 2023-07-29T00:24:23.406 DEBUG:boto:Path: /hiiqos-57/e 2023-07-29T00:24:23.406 DEBUG:boto:Data: 2023-07-29T00:24:23.406 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.407 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:23.407 DEBUG:boto:Port: 8000 2023-07-29T00:24:23.407 DEBUG:boto:Params: {} 2023-07-29T00:24:23.408 DEBUG:boto:Token: None 2023-07-29T00:24:23.408 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/e 2023-07-29T00:24:23.409 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:iyy46eUlgqwABJsARStGOulOUZo= 2023-07-29T00:24:23.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:iyy46eUlgqwABJsARStGOulOUZo=', 'Content-Length': '0'} 2023-07-29T00:24:23.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076cb3e3957992b37-0064c45c37-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.414 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:24:23.414 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:24:23.415 DEBUG:boto:path=/hiiqos-57/?prefix=e 2023-07-29T00:24:23.415 DEBUG:boto:auth_path=/hiiqos-57/?prefix=e 2023-07-29T00:24:23.416 DEBUG:boto:Method: GET 2023-07-29T00:24:23.416 DEBUG:boto:Path: /hiiqos-57/?prefix=e 2023-07-29T00:24:23.416 DEBUG:boto:Data: 2023-07-29T00:24:23.417 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.418 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:23.418 DEBUG:boto:Port: 8001 2023-07-29T00:24:23.418 DEBUG:boto:Params: {} 2023-07-29T00:24:23.419 DEBUG:boto:Token: None 2023-07-29T00:24:23.419 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/ 2023-07-29T00:24:23.420 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5xXylTj5tC8sh36uFSqkRh09kgM= 2023-07-29T00:24:23.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5xXylTj5tC8sh36uFSqkRh09kgM=', 'Content-Length': '0'} 2023-07-29T00:24:23.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de2cb89f1b6a70fd-0064c45c37-1b40-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.423 DEBUG:boto:b'hiiqos-57e1000falsee2023-07-29T00:21:40.767Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:23.424 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:24:23.424 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:24:23.425 DEBUG:boto:path=/hiiqos-57/?prefix=e 2023-07-29T00:24:23.425 DEBUG:boto:auth_path=/hiiqos-57/?prefix=e 2023-07-29T00:24:23.425 DEBUG:boto:Method: GET 2023-07-29T00:24:23.426 DEBUG:boto:Path: /hiiqos-57/?prefix=e 2023-07-29T00:24:23.426 DEBUG:boto:Data: 2023-07-29T00:24:23.426 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.427 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:23.427 DEBUG:boto:Port: 8000 2023-07-29T00:24:23.428 DEBUG:boto:Params: {} 2023-07-29T00:24:23.428 DEBUG:boto:Token: None 2023-07-29T00:24:23.428 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/ 2023-07-29T00:24:23.429 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5xXylTj5tC8sh36uFSqkRh09kgM= 2023-07-29T00:24:23.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5xXylTj5tC8sh36uFSqkRh09kgM=', 'Content-Length': '0'} 2023-07-29T00:24:23.432 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000610502779bb788cd-0064c45c37-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.476 DEBUG:boto:b'hiiqos-57e1000falsee2023-07-29T00:21:40.767Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:23.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-07-29T00:24:23.477 DEBUG:boto:path=/hiiqos-57/f 2023-07-29T00:24:23.478 DEBUG:boto:auth_path=/hiiqos-57/f 2023-07-29T00:24:23.478 DEBUG:boto:path=/hiiqos-57/f?versionId=null 2023-07-29T00:24:23.479 DEBUG:boto:auth_path=/hiiqos-57/f?versionId=null 2023-07-29T00:24:23.479 DEBUG:boto:Method: GET 2023-07-29T00:24:23.479 DEBUG:boto:Path: /hiiqos-57/f?versionId=null 2023-07-29T00:24:23.480 DEBUG:boto:Data: 2023-07-29T00:24:23.480 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.480 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:23.481 DEBUG:boto:Port: 8001 2023-07-29T00:24:23.481 DEBUG:boto:Params: {} 2023-07-29T00:24:23.482 DEBUG:boto:Token: None 2023-07-29T00:24:23.482 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/f?versionId=null 2023-07-29T00:24:23.482 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HbUU2WqVlCNOKGL1QXc+FbyD9qc= 2023-07-29T00:24:23.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HbUU2WqVlCNOKGL1QXc+FbyD9qc=', 'Content-Length': '0'} 2023-07-29T00:24:23.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-57:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000271241ec23037d97-0064c45c37-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.486 DEBUG:boto:path=/hiiqos-57/f 2023-07-29T00:24:23.486 DEBUG:boto:auth_path=/hiiqos-57/f 2023-07-29T00:24:23.486 DEBUG:boto:path=/hiiqos-57/f?versionId=null 2023-07-29T00:24:23.487 DEBUG:boto:auth_path=/hiiqos-57/f?versionId=null 2023-07-29T00:24:23.487 DEBUG:boto:Method: GET 2023-07-29T00:24:23.487 DEBUG:boto:Path: /hiiqos-57/f?versionId=null 2023-07-29T00:24:23.488 DEBUG:boto:Data: 2023-07-29T00:24:23.488 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.489 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:23.489 DEBUG:boto:Port: 8000 2023-07-29T00:24:23.489 DEBUG:boto:Params: {} 2023-07-29T00:24:23.490 DEBUG:boto:Token: None 2023-07-29T00:24:23.490 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/f?versionId=null 2023-07-29T00:24:23.491 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HbUU2WqVlCNOKGL1QXc+FbyD9qc= 2023-07-29T00:24:23.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HbUU2WqVlCNOKGL1QXc+FbyD9qc=', 'Content-Length': '0'} 2023-07-29T00:24:23.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b30af2ba5b738cb-0064c45c37-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.494 DEBUG:boto:path=/hiiqos-57/f 2023-07-29T00:24:23.494 DEBUG:boto:auth_path=/hiiqos-57/f 2023-07-29T00:24:23.494 DEBUG:boto:path=/hiiqos-57/f?versionId=null 2023-07-29T00:24:23.495 DEBUG:boto:auth_path=/hiiqos-57/f?versionId=null 2023-07-29T00:24:23.495 DEBUG:boto:Method: HEAD 2023-07-29T00:24:23.495 DEBUG:boto:Path: /hiiqos-57/f?versionId=null 2023-07-29T00:24:23.496 DEBUG:boto:Data: 2023-07-29T00:24:23.496 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.496 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:23.497 DEBUG:boto:Port: 8001 2023-07-29T00:24:23.497 DEBUG:boto:Params: {} 2023-07-29T00:24:23.497 DEBUG:boto:Token: None 2023-07-29T00:24:23.498 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/f?versionId=null 2023-07-29T00:24:23.498 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yVOG8lIt46mdrC46YYgZRmygxs0= 2023-07-29T00:24:23.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yVOG8lIt46mdrC46YYgZRmygxs0=', 'Content-Length': '0'} 2023-07-29T00:24:23.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-57:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000761ff36b7ece20fb-0064c45c37-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.501 DEBUG:boto:path=/hiiqos-57/f 2023-07-29T00:24:23.502 DEBUG:boto:auth_path=/hiiqos-57/f 2023-07-29T00:24:23.502 DEBUG:boto:path=/hiiqos-57/f?versionId=null 2023-07-29T00:24:23.502 DEBUG:boto:auth_path=/hiiqos-57/f?versionId=null 2023-07-29T00:24:23.503 DEBUG:boto:Method: HEAD 2023-07-29T00:24:23.503 DEBUG:boto:Path: /hiiqos-57/f?versionId=null 2023-07-29T00:24:23.503 DEBUG:boto:Data: 2023-07-29T00:24:23.504 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.504 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:23.504 DEBUG:boto:Port: 8000 2023-07-29T00:24:23.505 DEBUG:boto:Params: {} 2023-07-29T00:24:23.505 DEBUG:boto:Token: None 2023-07-29T00:24:23.505 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/f?versionId=null 2023-07-29T00:24:23.506 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yVOG8lIt46mdrC46YYgZRmygxs0= 2023-07-29T00:24:23.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yVOG8lIt46mdrC46YYgZRmygxs0=', 'Content-Length': '0'} 2023-07-29T00:24:23.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000457cccdaf2e8962d-0064c45c37-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-07-29T00:24:23.509 DEBUG:boto:path=/hiiqos-57/f 2023-07-29T00:24:23.509 DEBUG:boto:auth_path=/hiiqos-57/f 2023-07-29T00:24:23.510 DEBUG:boto:path=/hiiqos-57/f?versionId=null 2023-07-29T00:24:23.510 DEBUG:boto:auth_path=/hiiqos-57/f?versionId=null 2023-07-29T00:24:23.511 DEBUG:boto:Method: GET 2023-07-29T00:24:23.511 DEBUG:boto:Path: /hiiqos-57/f?versionId=null 2023-07-29T00:24:23.511 DEBUG:boto:Data: 2023-07-29T00:24:23.512 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.512 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:23.512 DEBUG:boto:Port: 8001 2023-07-29T00:24:23.513 DEBUG:boto:Params: {} 2023-07-29T00:24:23.513 DEBUG:boto:Token: None 2023-07-29T00:24:23.514 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/f?versionId=null 2023-07-29T00:24:23.514 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HbUU2WqVlCNOKGL1QXc+FbyD9qc= 2023-07-29T00:24:23.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HbUU2WqVlCNOKGL1QXc+FbyD9qc=', 'Content-Length': '0'} 2023-07-29T00:24:23.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-57:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fea7b66feed9e3b8-0064c45c37-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.560 DEBUG:boto:path=/hiiqos-57/f 2023-07-29T00:24:23.561 DEBUG:boto:auth_path=/hiiqos-57/f 2023-07-29T00:24:23.561 DEBUG:boto:path=/hiiqos-57/f?versionId=null 2023-07-29T00:24:23.562 DEBUG:boto:auth_path=/hiiqos-57/f?versionId=null 2023-07-29T00:24:23.562 DEBUG:boto:Method: GET 2023-07-29T00:24:23.562 DEBUG:boto:Path: /hiiqos-57/f?versionId=null 2023-07-29T00:24:23.563 DEBUG:boto:Data: 2023-07-29T00:24:23.563 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.563 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:23.564 DEBUG:boto:Port: 8000 2023-07-29T00:24:23.564 DEBUG:boto:Params: {} 2023-07-29T00:24:23.565 DEBUG:boto:Token: None 2023-07-29T00:24:23.565 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/f?versionId=null 2023-07-29T00:24:23.565 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HbUU2WqVlCNOKGL1QXc+FbyD9qc= 2023-07-29T00:24:23.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HbUU2WqVlCNOKGL1QXc+FbyD9qc=', 'Content-Length': '0'} 2023-07-29T00:24:23.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c579961bccf8d17e-0064c45c37-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.568 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:24:23.569 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:24:23.569 DEBUG:boto:path=/hiiqos-57/?prefix=f 2023-07-29T00:24:23.570 DEBUG:boto:auth_path=/hiiqos-57/?prefix=f 2023-07-29T00:24:23.570 DEBUG:boto:Method: GET 2023-07-29T00:24:23.570 DEBUG:boto:Path: /hiiqos-57/?prefix=f 2023-07-29T00:24:23.571 DEBUG:boto:Data: 2023-07-29T00:24:23.571 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.571 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:23.572 DEBUG:boto:Port: 8001 2023-07-29T00:24:23.572 DEBUG:boto:Params: {} 2023-07-29T00:24:23.573 DEBUG:boto:Token: None 2023-07-29T00:24:23.573 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/ 2023-07-29T00:24:23.573 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5xXylTj5tC8sh36uFSqkRh09kgM= 2023-07-29T00:24:23.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5xXylTj5tC8sh36uFSqkRh09kgM=', 'Content-Length': '0'} 2023-07-29T00:24:23.576 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0523dd9ad6f8321-0064c45c37-1b40-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.577 DEBUG:boto:b'hiiqos-57f1000falsef2023-07-29T00:21:40.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:23.577 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:24:23.578 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:24:23.578 DEBUG:boto:path=/hiiqos-57/?prefix=f 2023-07-29T00:24:23.578 DEBUG:boto:auth_path=/hiiqos-57/?prefix=f 2023-07-29T00:24:23.579 DEBUG:boto:Method: GET 2023-07-29T00:24:23.579 DEBUG:boto:Path: /hiiqos-57/?prefix=f 2023-07-29T00:24:23.579 DEBUG:boto:Data: 2023-07-29T00:24:23.580 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.580 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:23.581 DEBUG:boto:Port: 8000 2023-07-29T00:24:23.581 DEBUG:boto:Params: {} 2023-07-29T00:24:23.581 DEBUG:boto:Token: None 2023-07-29T00:24:23.582 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/ 2023-07-29T00:24:23.582 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5xXylTj5tC8sh36uFSqkRh09kgM= 2023-07-29T00:24:23.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5xXylTj5tC8sh36uFSqkRh09kgM=', 'Content-Length': '0'} 2023-07-29T00:24:23.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a2df846b7578d08-0064c45c37-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.585 DEBUG:boto:b'hiiqos-57f1000falsef2023-07-29T00:21:40.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:23.586 DEBUG:boto:path=/hiiqos-57/f 2023-07-29T00:24:23.587 DEBUG:boto:auth_path=/hiiqos-57/f 2023-07-29T00:24:23.587 DEBUG:boto:Method: HEAD 2023-07-29T00:24:23.587 DEBUG:boto:Path: /hiiqos-57/f 2023-07-29T00:24:23.588 DEBUG:boto:Data: 2023-07-29T00:24:23.588 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.588 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:23.589 DEBUG:boto:Port: 8001 2023-07-29T00:24:23.589 DEBUG:boto:Params: {} 2023-07-29T00:24:23.589 DEBUG:boto:Token: None 2023-07-29T00:24:23.590 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/f 2023-07-29T00:24:23.590 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aoNw94zI0sXxrDYFVE9raPu+m24= 2023-07-29T00:24:23.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aoNw94zI0sXxrDYFVE9raPu+m24=', 'Content-Length': '0'} 2023-07-29T00:24:23.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-57:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b98c41f3684d7666-0064c45c37-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.593 DEBUG:boto:path=/hiiqos-57/f 2023-07-29T00:24:23.593 DEBUG:boto:auth_path=/hiiqos-57/f 2023-07-29T00:24:23.593 DEBUG:boto:Method: HEAD 2023-07-29T00:24:23.594 DEBUG:boto:Path: /hiiqos-57/f 2023-07-29T00:24:23.594 DEBUG:boto:Data: 2023-07-29T00:24:23.594 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.595 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:23.595 DEBUG:boto:Port: 8000 2023-07-29T00:24:23.595 DEBUG:boto:Params: {} 2023-07-29T00:24:23.596 DEBUG:boto:Token: None 2023-07-29T00:24:23.596 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/f 2023-07-29T00:24:23.597 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aoNw94zI0sXxrDYFVE9raPu+m24= 2023-07-29T00:24:23.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aoNw94zI0sXxrDYFVE9raPu+m24=', 'Content-Length': '0'} 2023-07-29T00:24:23.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007327128447c6e629-0064c45c37-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-07-29T00:24:23.600 DEBUG:boto:path=/hiiqos-57/f 2023-07-29T00:24:23.600 DEBUG:boto:auth_path=/hiiqos-57/f 2023-07-29T00:24:23.600 DEBUG:boto:Method: GET 2023-07-29T00:24:23.601 DEBUG:boto:Path: /hiiqos-57/f 2023-07-29T00:24:23.601 DEBUG:boto:Data: 2023-07-29T00:24:23.602 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.602 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:23.602 DEBUG:boto:Port: 8001 2023-07-29T00:24:23.603 DEBUG:boto:Params: {} 2023-07-29T00:24:23.603 DEBUG:boto:Token: None 2023-07-29T00:24:23.603 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/f 2023-07-29T00:24:23.604 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mrVb9pENSFc0R1Bwfbyk2g+dliI= 2023-07-29T00:24:23.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mrVb9pENSFc0R1Bwfbyk2g+dliI=', 'Content-Length': '0'} 2023-07-29T00:24:23.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-57:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037a9a0d26454f0bd-0064c45c37-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.648 DEBUG:boto:path=/hiiqos-57/f 2023-07-29T00:24:23.649 DEBUG:boto:auth_path=/hiiqos-57/f 2023-07-29T00:24:23.649 DEBUG:boto:Method: GET 2023-07-29T00:24:23.650 DEBUG:boto:Path: /hiiqos-57/f 2023-07-29T00:24:23.650 DEBUG:boto:Data: 2023-07-29T00:24:23.650 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.651 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:23.651 DEBUG:boto:Port: 8000 2023-07-29T00:24:23.651 DEBUG:boto:Params: {} 2023-07-29T00:24:23.652 DEBUG:boto:Token: None 2023-07-29T00:24:23.652 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/f 2023-07-29T00:24:23.653 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mrVb9pENSFc0R1Bwfbyk2g+dliI= 2023-07-29T00:24:23.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mrVb9pENSFc0R1Bwfbyk2g+dliI=', 'Content-Length': '0'} 2023-07-29T00:24:23.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d34667fc199809b-0064c45c37-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.656 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:24:23.657 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:24:23.657 DEBUG:boto:path=/hiiqos-57/?prefix=f 2023-07-29T00:24:23.657 DEBUG:boto:auth_path=/hiiqos-57/?prefix=f 2023-07-29T00:24:23.658 DEBUG:boto:Method: GET 2023-07-29T00:24:23.658 DEBUG:boto:Path: /hiiqos-57/?prefix=f 2023-07-29T00:24:23.659 DEBUG:boto:Data: 2023-07-29T00:24:23.659 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.659 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:23.660 DEBUG:boto:Port: 8001 2023-07-29T00:24:23.660 DEBUG:boto:Params: {} 2023-07-29T00:24:23.661 DEBUG:boto:Token: None 2023-07-29T00:24:23.661 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/ 2023-07-29T00:24:23.661 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5xXylTj5tC8sh36uFSqkRh09kgM= 2023-07-29T00:24:23.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5xXylTj5tC8sh36uFSqkRh09kgM=', 'Content-Length': '0'} 2023-07-29T00:24:23.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5176739b298f9a6-0064c45c37-1b40-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.665 DEBUG:boto:b'hiiqos-57f1000falsef2023-07-29T00:21:40.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:23.666 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:24:23.666 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:24:23.666 DEBUG:boto:path=/hiiqos-57/?prefix=f 2023-07-29T00:24:23.667 DEBUG:boto:auth_path=/hiiqos-57/?prefix=f 2023-07-29T00:24:23.667 DEBUG:boto:Method: GET 2023-07-29T00:24:23.668 DEBUG:boto:Path: /hiiqos-57/?prefix=f 2023-07-29T00:24:23.668 DEBUG:boto:Data: 2023-07-29T00:24:23.668 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.669 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:23.669 DEBUG:boto:Port: 8000 2023-07-29T00:24:23.670 DEBUG:boto:Params: {} 2023-07-29T00:24:23.670 DEBUG:boto:Token: None 2023-07-29T00:24:23.670 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/ 2023-07-29T00:24:23.671 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5xXylTj5tC8sh36uFSqkRh09kgM= 2023-07-29T00:24:23.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5xXylTj5tC8sh36uFSqkRh09kgM=', 'Content-Length': '0'} 2023-07-29T00:24:23.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d61f5e7bf273d43-0064c45c37-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.674 DEBUG:boto:b'hiiqos-57f1000falsef2023-07-29T00:21:40.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:23.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-07-29T00:24:23.675 DEBUG:boto:path=/hiiqos-57/g 2023-07-29T00:24:23.676 DEBUG:boto:auth_path=/hiiqos-57/g 2023-07-29T00:24:23.676 DEBUG:boto:path=/hiiqos-57/g?versionId=null 2023-07-29T00:24:23.676 DEBUG:boto:auth_path=/hiiqos-57/g?versionId=null 2023-07-29T00:24:23.677 DEBUG:boto:Method: GET 2023-07-29T00:24:23.677 DEBUG:boto:Path: /hiiqos-57/g?versionId=null 2023-07-29T00:24:23.678 DEBUG:boto:Data: 2023-07-29T00:24:23.678 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.678 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:23.679 DEBUG:boto:Port: 8001 2023-07-29T00:24:23.679 DEBUG:boto:Params: {} 2023-07-29T00:24:23.680 DEBUG:boto:Token: None 2023-07-29T00:24:23.680 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/g?versionId=null 2023-07-29T00:24:23.681 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:V11NXMbyc2Anco3xbSwQRM8Bz7A= 2023-07-29T00:24:23.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:V11NXMbyc2Anco3xbSwQRM8Bz7A=', 'Content-Length': '0'} 2023-07-29T00:24:23.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-57:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1a8078c16723a02-0064c45c37-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.684 DEBUG:boto:path=/hiiqos-57/g 2023-07-29T00:24:23.684 DEBUG:boto:auth_path=/hiiqos-57/g 2023-07-29T00:24:23.685 DEBUG:boto:path=/hiiqos-57/g?versionId=null 2023-07-29T00:24:23.685 DEBUG:boto:auth_path=/hiiqos-57/g?versionId=null 2023-07-29T00:24:23.685 DEBUG:boto:Method: GET 2023-07-29T00:24:23.686 DEBUG:boto:Path: /hiiqos-57/g?versionId=null 2023-07-29T00:24:23.686 DEBUG:boto:Data: 2023-07-29T00:24:23.686 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.687 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:23.687 DEBUG:boto:Port: 8000 2023-07-29T00:24:23.687 DEBUG:boto:Params: {} 2023-07-29T00:24:23.688 DEBUG:boto:Token: None 2023-07-29T00:24:23.688 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/g?versionId=null 2023-07-29T00:24:23.689 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:V11NXMbyc2Anco3xbSwQRM8Bz7A= 2023-07-29T00:24:23.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:V11NXMbyc2Anco3xbSwQRM8Bz7A=', 'Content-Length': '0'} 2023-07-29T00:24:23.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d2335507078077e-0064c45c37-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.692 DEBUG:boto:path=/hiiqos-57/g 2023-07-29T00:24:23.692 DEBUG:boto:auth_path=/hiiqos-57/g 2023-07-29T00:24:23.693 DEBUG:boto:path=/hiiqos-57/g?versionId=null 2023-07-29T00:24:23.693 DEBUG:boto:auth_path=/hiiqos-57/g?versionId=null 2023-07-29T00:24:23.693 DEBUG:boto:Method: HEAD 2023-07-29T00:24:23.693 DEBUG:boto:Path: /hiiqos-57/g?versionId=null 2023-07-29T00:24:23.694 DEBUG:boto:Data: 2023-07-29T00:24:23.694 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.694 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:23.695 DEBUG:boto:Port: 8001 2023-07-29T00:24:23.695 DEBUG:boto:Params: {} 2023-07-29T00:24:23.695 DEBUG:boto:Token: None 2023-07-29T00:24:23.696 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/g?versionId=null 2023-07-29T00:24:23.696 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KMb6QU8R2NTYJ4+WCA8TGs4XvR4= 2023-07-29T00:24:23.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KMb6QU8R2NTYJ4+WCA8TGs4XvR4=', 'Content-Length': '0'} 2023-07-29T00:24:23.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-57:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e2fcbb62f0fa6de-0064c45c37-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.699 DEBUG:boto:path=/hiiqos-57/g 2023-07-29T00:24:23.699 DEBUG:boto:auth_path=/hiiqos-57/g 2023-07-29T00:24:23.700 DEBUG:boto:path=/hiiqos-57/g?versionId=null 2023-07-29T00:24:23.700 DEBUG:boto:auth_path=/hiiqos-57/g?versionId=null 2023-07-29T00:24:23.700 DEBUG:boto:Method: HEAD 2023-07-29T00:24:23.701 DEBUG:boto:Path: /hiiqos-57/g?versionId=null 2023-07-29T00:24:23.701 DEBUG:boto:Data: 2023-07-29T00:24:23.701 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.702 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:23.702 DEBUG:boto:Port: 8000 2023-07-29T00:24:23.702 DEBUG:boto:Params: {} 2023-07-29T00:24:23.703 DEBUG:boto:Token: None 2023-07-29T00:24:23.703 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/g?versionId=null 2023-07-29T00:24:23.703 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KMb6QU8R2NTYJ4+WCA8TGs4XvR4= 2023-07-29T00:24:23.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KMb6QU8R2NTYJ4+WCA8TGs4XvR4=', 'Content-Length': '0'} 2023-07-29T00:24:23.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f927348f953e71ae-0064c45c37-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-07-29T00:24:23.707 DEBUG:boto:path=/hiiqos-57/g 2023-07-29T00:24:23.707 DEBUG:boto:auth_path=/hiiqos-57/g 2023-07-29T00:24:23.707 DEBUG:boto:path=/hiiqos-57/g?versionId=null 2023-07-29T00:24:23.708 DEBUG:boto:auth_path=/hiiqos-57/g?versionId=null 2023-07-29T00:24:23.708 DEBUG:boto:Method: GET 2023-07-29T00:24:23.708 DEBUG:boto:Path: /hiiqos-57/g?versionId=null 2023-07-29T00:24:23.709 DEBUG:boto:Data: 2023-07-29T00:24:23.709 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.709 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:23.710 DEBUG:boto:Port: 8001 2023-07-29T00:24:23.710 DEBUG:boto:Params: {} 2023-07-29T00:24:23.711 DEBUG:boto:Token: None 2023-07-29T00:24:23.711 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/g?versionId=null 2023-07-29T00:24:23.711 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:V11NXMbyc2Anco3xbSwQRM8Bz7A= 2023-07-29T00:24:23.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:V11NXMbyc2Anco3xbSwQRM8Bz7A=', 'Content-Length': '0'} 2023-07-29T00:24:23.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-57:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a27a55b111390c03-0064c45c37-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.714 DEBUG:boto:path=/hiiqos-57/g 2023-07-29T00:24:23.715 DEBUG:boto:auth_path=/hiiqos-57/g 2023-07-29T00:24:23.715 DEBUG:boto:path=/hiiqos-57/g?versionId=null 2023-07-29T00:24:23.715 DEBUG:boto:auth_path=/hiiqos-57/g?versionId=null 2023-07-29T00:24:23.716 DEBUG:boto:Method: GET 2023-07-29T00:24:23.716 DEBUG:boto:Path: /hiiqos-57/g?versionId=null 2023-07-29T00:24:23.716 DEBUG:boto:Data: 2023-07-29T00:24:23.717 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.717 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:23.717 DEBUG:boto:Port: 8000 2023-07-29T00:24:23.718 DEBUG:boto:Params: {} 2023-07-29T00:24:23.718 DEBUG:boto:Token: None 2023-07-29T00:24:23.719 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/g?versionId=null 2023-07-29T00:24:23.719 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:V11NXMbyc2Anco3xbSwQRM8Bz7A= 2023-07-29T00:24:23.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:V11NXMbyc2Anco3xbSwQRM8Bz7A=', 'Content-Length': '0'} 2023-07-29T00:24:23.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7149c4517efd7a3-0064c45c37-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.722 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:24:23.722 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:24:23.723 DEBUG:boto:path=/hiiqos-57/?prefix=g 2023-07-29T00:24:23.723 DEBUG:boto:auth_path=/hiiqos-57/?prefix=g 2023-07-29T00:24:23.723 DEBUG:boto:Method: GET 2023-07-29T00:24:23.724 DEBUG:boto:Path: /hiiqos-57/?prefix=g 2023-07-29T00:24:23.724 DEBUG:boto:Data: 2023-07-29T00:24:23.725 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.725 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:23.725 DEBUG:boto:Port: 8001 2023-07-29T00:24:23.726 DEBUG:boto:Params: {} 2023-07-29T00:24:23.726 DEBUG:boto:Token: None 2023-07-29T00:24:23.726 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/ 2023-07-29T00:24:23.727 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5xXylTj5tC8sh36uFSqkRh09kgM= 2023-07-29T00:24:23.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5xXylTj5tC8sh36uFSqkRh09kgM=', 'Content-Length': '0'} 2023-07-29T00:24:23.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a9cd0004fc57320-0064c45c37-1b40-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.730 DEBUG:boto:b'hiiqos-57g1000falseg2023-07-29T00:21:40.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:23.731 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:24:23.731 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:24:23.732 DEBUG:boto:path=/hiiqos-57/?prefix=g 2023-07-29T00:24:23.732 DEBUG:boto:auth_path=/hiiqos-57/?prefix=g 2023-07-29T00:24:23.732 DEBUG:boto:Method: GET 2023-07-29T00:24:23.733 DEBUG:boto:Path: /hiiqos-57/?prefix=g 2023-07-29T00:24:23.733 DEBUG:boto:Data: 2023-07-29T00:24:23.733 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.734 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:23.734 DEBUG:boto:Port: 8000 2023-07-29T00:24:23.734 DEBUG:boto:Params: {} 2023-07-29T00:24:23.735 DEBUG:boto:Token: None 2023-07-29T00:24:23.735 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/ 2023-07-29T00:24:23.736 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5xXylTj5tC8sh36uFSqkRh09kgM= 2023-07-29T00:24:23.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5xXylTj5tC8sh36uFSqkRh09kgM=', 'Content-Length': '0'} 2023-07-29T00:24:23.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b608b5e7139c91ae-0064c45c37-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.739 DEBUG:boto:b'hiiqos-57g1000falseg2023-07-29T00:21:40.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:23.739 DEBUG:boto:path=/hiiqos-57/g 2023-07-29T00:24:23.740 DEBUG:boto:auth_path=/hiiqos-57/g 2023-07-29T00:24:23.740 DEBUG:boto:Method: HEAD 2023-07-29T00:24:23.740 DEBUG:boto:Path: /hiiqos-57/g 2023-07-29T00:24:23.741 DEBUG:boto:Data: 2023-07-29T00:24:23.741 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.741 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:23.742 DEBUG:boto:Port: 8001 2023-07-29T00:24:23.742 DEBUG:boto:Params: {} 2023-07-29T00:24:23.742 DEBUG:boto:Token: None 2023-07-29T00:24:23.743 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/g 2023-07-29T00:24:23.743 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4AmiYdAhvleAAwaxdBZCmAEr3OY= 2023-07-29T00:24:23.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4AmiYdAhvleAAwaxdBZCmAEr3OY=', 'Content-Length': '0'} 2023-07-29T00:24:23.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-57:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c76229d75dcc7a26-0064c45c37-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.746 DEBUG:boto:path=/hiiqos-57/g 2023-07-29T00:24:23.746 DEBUG:boto:auth_path=/hiiqos-57/g 2023-07-29T00:24:23.747 DEBUG:boto:Method: HEAD 2023-07-29T00:24:23.747 DEBUG:boto:Path: /hiiqos-57/g 2023-07-29T00:24:23.747 DEBUG:boto:Data: 2023-07-29T00:24:23.748 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.748 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:23.748 DEBUG:boto:Port: 8000 2023-07-29T00:24:23.749 DEBUG:boto:Params: {} 2023-07-29T00:24:23.749 DEBUG:boto:Token: None 2023-07-29T00:24:23.749 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/g 2023-07-29T00:24:23.750 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4AmiYdAhvleAAwaxdBZCmAEr3OY= 2023-07-29T00:24:23.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4AmiYdAhvleAAwaxdBZCmAEr3OY=', 'Content-Length': '0'} 2023-07-29T00:24:23.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0dbf3f4ad6d64c6-0064c45c37-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-07-29T00:24:23.753 DEBUG:boto:path=/hiiqos-57/g 2023-07-29T00:24:23.753 DEBUG:boto:auth_path=/hiiqos-57/g 2023-07-29T00:24:23.754 DEBUG:boto:Method: GET 2023-07-29T00:24:23.754 DEBUG:boto:Path: /hiiqos-57/g 2023-07-29T00:24:23.754 DEBUG:boto:Data: 2023-07-29T00:24:23.755 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.755 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:23.755 DEBUG:boto:Port: 8001 2023-07-29T00:24:23.756 DEBUG:boto:Params: {} 2023-07-29T00:24:23.756 DEBUG:boto:Token: None 2023-07-29T00:24:23.757 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/g 2023-07-29T00:24:23.757 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I4Dxevif6JcX4T6k234njsdxS5k= 2023-07-29T00:24:23.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I4Dxevif6JcX4T6k234njsdxS5k=', 'Content-Length': '0'} 2023-07-29T00:24:23.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-57:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061ff7de61148c33f-0064c45c37-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.800 DEBUG:boto:path=/hiiqos-57/g 2023-07-29T00:24:23.801 DEBUG:boto:auth_path=/hiiqos-57/g 2023-07-29T00:24:23.801 DEBUG:boto:Method: GET 2023-07-29T00:24:23.802 DEBUG:boto:Path: /hiiqos-57/g 2023-07-29T00:24:23.802 DEBUG:boto:Data: 2023-07-29T00:24:23.802 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.803 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:23.803 DEBUG:boto:Port: 8000 2023-07-29T00:24:23.803 DEBUG:boto:Params: {} 2023-07-29T00:24:23.804 DEBUG:boto:Token: None 2023-07-29T00:24:23.804 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/g 2023-07-29T00:24:23.805 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I4Dxevif6JcX4T6k234njsdxS5k= 2023-07-29T00:24:23.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I4Dxevif6JcX4T6k234njsdxS5k=', 'Content-Length': '0'} 2023-07-29T00:24:23.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a1c8a492704f8c6-0064c45c37-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.808 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:24:23.809 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:24:23.809 DEBUG:boto:path=/hiiqos-57/?prefix=g 2023-07-29T00:24:23.810 DEBUG:boto:auth_path=/hiiqos-57/?prefix=g 2023-07-29T00:24:23.810 DEBUG:boto:Method: GET 2023-07-29T00:24:23.810 DEBUG:boto:Path: /hiiqos-57/?prefix=g 2023-07-29T00:24:23.811 DEBUG:boto:Data: 2023-07-29T00:24:23.811 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.811 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:23.812 DEBUG:boto:Port: 8001 2023-07-29T00:24:23.812 DEBUG:boto:Params: {} 2023-07-29T00:24:23.812 DEBUG:boto:Token: None 2023-07-29T00:24:23.813 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/ 2023-07-29T00:24:23.813 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5xXylTj5tC8sh36uFSqkRh09kgM= 2023-07-29T00:24:23.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5xXylTj5tC8sh36uFSqkRh09kgM=', 'Content-Length': '0'} 2023-07-29T00:24:23.816 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eff336d6d161a769-0064c45c37-1b40-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.817 DEBUG:boto:b'hiiqos-57g1000falseg2023-07-29T00:21:40.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:23.817 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:24:23.818 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:24:23.818 DEBUG:boto:path=/hiiqos-57/?prefix=g 2023-07-29T00:24:23.818 DEBUG:boto:auth_path=/hiiqos-57/?prefix=g 2023-07-29T00:24:23.819 DEBUG:boto:Method: GET 2023-07-29T00:24:23.819 DEBUG:boto:Path: /hiiqos-57/?prefix=g 2023-07-29T00:24:23.820 DEBUG:boto:Data: 2023-07-29T00:24:23.820 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.820 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:23.821 DEBUG:boto:Port: 8000 2023-07-29T00:24:23.821 DEBUG:boto:Params: {} 2023-07-29T00:24:23.821 DEBUG:boto:Token: None 2023-07-29T00:24:23.822 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/ 2023-07-29T00:24:23.822 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5xXylTj5tC8sh36uFSqkRh09kgM= 2023-07-29T00:24:23.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5xXylTj5tC8sh36uFSqkRh09kgM=', 'Content-Length': '0'} 2023-07-29T00:24:23.825 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b810c23f48416ed9-0064c45c37-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.825 DEBUG:boto:b'hiiqos-57g1000falseg2023-07-29T00:21:40.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:23.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-07-29T00:24:23.826 DEBUG:boto:path=/hiiqos-57/h 2023-07-29T00:24:23.826 DEBUG:boto:auth_path=/hiiqos-57/h 2023-07-29T00:24:23.827 DEBUG:boto:path=/hiiqos-57/h?versionId=null 2023-07-29T00:24:23.827 DEBUG:boto:auth_path=/hiiqos-57/h?versionId=null 2023-07-29T00:24:23.828 DEBUG:boto:Method: GET 2023-07-29T00:24:23.828 DEBUG:boto:Path: /hiiqos-57/h?versionId=null 2023-07-29T00:24:23.828 DEBUG:boto:Data: 2023-07-29T00:24:23.829 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.829 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:23.829 DEBUG:boto:Port: 8001 2023-07-29T00:24:23.830 DEBUG:boto:Params: {} 2023-07-29T00:24:23.830 DEBUG:boto:Token: None 2023-07-29T00:24:23.830 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/h?versionId=null 2023-07-29T00:24:23.831 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gHbJ00hShuPLC7oPGEuhXzEUPbw= 2023-07-29T00:24:23.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gHbJ00hShuPLC7oPGEuhXzEUPbw=', 'Content-Length': '0'} 2023-07-29T00:24:23.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-57:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4208512558206dd-0064c45c37-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.834 DEBUG:boto:path=/hiiqos-57/h 2023-07-29T00:24:23.834 DEBUG:boto:auth_path=/hiiqos-57/h 2023-07-29T00:24:23.835 DEBUG:boto:path=/hiiqos-57/h?versionId=null 2023-07-29T00:24:23.835 DEBUG:boto:auth_path=/hiiqos-57/h?versionId=null 2023-07-29T00:24:23.835 DEBUG:boto:Method: GET 2023-07-29T00:24:23.836 DEBUG:boto:Path: /hiiqos-57/h?versionId=null 2023-07-29T00:24:23.836 DEBUG:boto:Data: 2023-07-29T00:24:23.837 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.837 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:23.837 DEBUG:boto:Port: 8000 2023-07-29T00:24:23.838 DEBUG:boto:Params: {} 2023-07-29T00:24:23.838 DEBUG:boto:Token: None 2023-07-29T00:24:23.838 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/h?versionId=null 2023-07-29T00:24:23.839 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gHbJ00hShuPLC7oPGEuhXzEUPbw= 2023-07-29T00:24:23.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gHbJ00hShuPLC7oPGEuhXzEUPbw=', 'Content-Length': '0'} 2023-07-29T00:24:23.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c335b27ce7feb44-0064c45c37-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.842 DEBUG:boto:path=/hiiqos-57/h 2023-07-29T00:24:23.842 DEBUG:boto:auth_path=/hiiqos-57/h 2023-07-29T00:24:23.843 DEBUG:boto:path=/hiiqos-57/h?versionId=null 2023-07-29T00:24:23.843 DEBUG:boto:auth_path=/hiiqos-57/h?versionId=null 2023-07-29T00:24:23.844 DEBUG:boto:Method: HEAD 2023-07-29T00:24:23.844 DEBUG:boto:Path: /hiiqos-57/h?versionId=null 2023-07-29T00:24:23.844 DEBUG:boto:Data: 2023-07-29T00:24:23.845 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.845 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:23.845 DEBUG:boto:Port: 8001 2023-07-29T00:24:23.846 DEBUG:boto:Params: {} 2023-07-29T00:24:23.846 DEBUG:boto:Token: None 2023-07-29T00:24:23.846 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/h?versionId=null 2023-07-29T00:24:23.847 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZViwfgfwQUmaLFBXpd/hjKhQifE= 2023-07-29T00:24:23.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZViwfgfwQUmaLFBXpd/hjKhQifE=', 'Content-Length': '0'} 2023-07-29T00:24:23.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-57:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013487b1b29f2bc3b-0064c45c37-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.850 DEBUG:boto:path=/hiiqos-57/h 2023-07-29T00:24:23.850 DEBUG:boto:auth_path=/hiiqos-57/h 2023-07-29T00:24:23.850 DEBUG:boto:path=/hiiqos-57/h?versionId=null 2023-07-29T00:24:23.851 DEBUG:boto:auth_path=/hiiqos-57/h?versionId=null 2023-07-29T00:24:23.851 DEBUG:boto:Method: HEAD 2023-07-29T00:24:23.851 DEBUG:boto:Path: /hiiqos-57/h?versionId=null 2023-07-29T00:24:23.852 DEBUG:boto:Data: 2023-07-29T00:24:23.852 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.852 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:23.853 DEBUG:boto:Port: 8000 2023-07-29T00:24:23.853 DEBUG:boto:Params: {} 2023-07-29T00:24:23.853 DEBUG:boto:Token: None 2023-07-29T00:24:23.854 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/h?versionId=null 2023-07-29T00:24:23.854 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZViwfgfwQUmaLFBXpd/hjKhQifE= 2023-07-29T00:24:23.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZViwfgfwQUmaLFBXpd/hjKhQifE=', 'Content-Length': '0'} 2023-07-29T00:24:23.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b01b06e7bc5b373e-0064c45c37-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-07-29T00:24:23.857 DEBUG:boto:path=/hiiqos-57/h 2023-07-29T00:24:23.858 DEBUG:boto:auth_path=/hiiqos-57/h 2023-07-29T00:24:23.858 DEBUG:boto:path=/hiiqos-57/h?versionId=null 2023-07-29T00:24:23.858 DEBUG:boto:auth_path=/hiiqos-57/h?versionId=null 2023-07-29T00:24:23.859 DEBUG:boto:Method: GET 2023-07-29T00:24:23.859 DEBUG:boto:Path: /hiiqos-57/h?versionId=null 2023-07-29T00:24:23.859 DEBUG:boto:Data: 2023-07-29T00:24:23.860 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.860 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:23.860 DEBUG:boto:Port: 8001 2023-07-29T00:24:23.861 DEBUG:boto:Params: {} 2023-07-29T00:24:23.861 DEBUG:boto:Token: None 2023-07-29T00:24:23.862 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/h?versionId=null 2023-07-29T00:24:23.862 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gHbJ00hShuPLC7oPGEuhXzEUPbw= 2023-07-29T00:24:23.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gHbJ00hShuPLC7oPGEuhXzEUPbw=', 'Content-Length': '0'} 2023-07-29T00:24:23.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-57:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010889a3f2e6b25d1-0064c45c37-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.908 DEBUG:boto:path=/hiiqos-57/h 2023-07-29T00:24:23.909 DEBUG:boto:auth_path=/hiiqos-57/h 2023-07-29T00:24:23.909 DEBUG:boto:path=/hiiqos-57/h?versionId=null 2023-07-29T00:24:23.910 DEBUG:boto:auth_path=/hiiqos-57/h?versionId=null 2023-07-29T00:24:23.910 DEBUG:boto:Method: GET 2023-07-29T00:24:23.910 DEBUG:boto:Path: /hiiqos-57/h?versionId=null 2023-07-29T00:24:23.911 DEBUG:boto:Data: 2023-07-29T00:24:23.911 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.911 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:23.912 DEBUG:boto:Port: 8000 2023-07-29T00:24:23.912 DEBUG:boto:Params: {} 2023-07-29T00:24:23.912 DEBUG:boto:Token: None 2023-07-29T00:24:23.913 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/h?versionId=null 2023-07-29T00:24:23.913 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gHbJ00hShuPLC7oPGEuhXzEUPbw= 2023-07-29T00:24:23.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gHbJ00hShuPLC7oPGEuhXzEUPbw=', 'Content-Length': '0'} 2023-07-29T00:24:23.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd9121214feb2bf1-0064c45c37-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.917 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:24:23.917 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:24:23.917 DEBUG:boto:path=/hiiqos-57/?prefix=h 2023-07-29T00:24:23.918 DEBUG:boto:auth_path=/hiiqos-57/?prefix=h 2023-07-29T00:24:23.918 DEBUG:boto:Method: GET 2023-07-29T00:24:23.918 DEBUG:boto:Path: /hiiqos-57/?prefix=h 2023-07-29T00:24:23.919 DEBUG:boto:Data: 2023-07-29T00:24:23.919 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.919 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:23.920 DEBUG:boto:Port: 8001 2023-07-29T00:24:23.921 DEBUG:boto:Params: {} 2023-07-29T00:24:23.921 DEBUG:boto:Token: None 2023-07-29T00:24:23.921 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/ 2023-07-29T00:24:23.922 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5xXylTj5tC8sh36uFSqkRh09kgM= 2023-07-29T00:24:23.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5xXylTj5tC8sh36uFSqkRh09kgM=', 'Content-Length': '0'} 2023-07-29T00:24:23.925 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d159ce457899d37-0064c45c37-1b40-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.925 DEBUG:boto:b'hiiqos-57h1000falseh2023-07-29T00:21:40.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:23.926 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:24:23.926 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:24:23.927 DEBUG:boto:path=/hiiqos-57/?prefix=h 2023-07-29T00:24:23.927 DEBUG:boto:auth_path=/hiiqos-57/?prefix=h 2023-07-29T00:24:23.927 DEBUG:boto:Method: GET 2023-07-29T00:24:23.928 DEBUG:boto:Path: /hiiqos-57/?prefix=h 2023-07-29T00:24:23.928 DEBUG:boto:Data: 2023-07-29T00:24:23.928 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.929 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:23.929 DEBUG:boto:Port: 8000 2023-07-29T00:24:23.929 DEBUG:boto:Params: {} 2023-07-29T00:24:23.930 DEBUG:boto:Token: None 2023-07-29T00:24:23.930 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/ 2023-07-29T00:24:23.931 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5xXylTj5tC8sh36uFSqkRh09kgM= 2023-07-29T00:24:23.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5xXylTj5tC8sh36uFSqkRh09kgM=', 'Content-Length': '0'} 2023-07-29T00:24:23.933 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e00d6a77a15fa4ff-0064c45c37-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.934 DEBUG:boto:b'hiiqos-57h1000falseh2023-07-29T00:21:40.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:23.934 DEBUG:boto:path=/hiiqos-57/h 2023-07-29T00:24:23.935 DEBUG:boto:auth_path=/hiiqos-57/h 2023-07-29T00:24:23.935 DEBUG:boto:Method: HEAD 2023-07-29T00:24:23.935 DEBUG:boto:Path: /hiiqos-57/h 2023-07-29T00:24:23.936 DEBUG:boto:Data: 2023-07-29T00:24:23.936 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.936 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:23.937 DEBUG:boto:Port: 8001 2023-07-29T00:24:23.937 DEBUG:boto:Params: {} 2023-07-29T00:24:23.937 DEBUG:boto:Token: None 2023-07-29T00:24:23.938 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/h 2023-07-29T00:24:23.938 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/zFFIl1HApDXB8X2j6VxPhT3YQI= 2023-07-29T00:24:23.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/zFFIl1HApDXB8X2j6VxPhT3YQI=', 'Content-Length': '0'} 2023-07-29T00:24:23.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-57:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a595d7798ab326e-0064c45c37-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.941 DEBUG:boto:path=/hiiqos-57/h 2023-07-29T00:24:23.941 DEBUG:boto:auth_path=/hiiqos-57/h 2023-07-29T00:24:23.942 DEBUG:boto:Method: HEAD 2023-07-29T00:24:23.942 DEBUG:boto:Path: /hiiqos-57/h 2023-07-29T00:24:23.942 DEBUG:boto:Data: 2023-07-29T00:24:23.943 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.943 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:23.943 DEBUG:boto:Port: 8000 2023-07-29T00:24:23.944 DEBUG:boto:Params: {} 2023-07-29T00:24:23.944 DEBUG:boto:Token: None 2023-07-29T00:24:23.945 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/h 2023-07-29T00:24:23.945 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/zFFIl1HApDXB8X2j6VxPhT3YQI= 2023-07-29T00:24:23.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/zFFIl1HApDXB8X2j6VxPhT3YQI=', 'Content-Length': '0'} 2023-07-29T00:24:23.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c53da6d210330982-0064c45c37-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-07-29T00:24:23.949 DEBUG:boto:path=/hiiqos-57/h 2023-07-29T00:24:23.949 DEBUG:boto:auth_path=/hiiqos-57/h 2023-07-29T00:24:23.950 DEBUG:boto:Method: GET 2023-07-29T00:24:23.950 DEBUG:boto:Path: /hiiqos-57/h 2023-07-29T00:24:23.950 DEBUG:boto:Data: 2023-07-29T00:24:23.951 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.951 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:23.951 DEBUG:boto:Port: 8001 2023-07-29T00:24:23.952 DEBUG:boto:Params: {} 2023-07-29T00:24:23.952 DEBUG:boto:Token: None 2023-07-29T00:24:23.953 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:23 GMT /hiiqos-57/h 2023-07-29T00:24:23.953 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:y3I4sTcYO6rG9Nced/4QGJqgqH0= 2023-07-29T00:24:23.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:y3I4sTcYO6rG9Nced/4QGJqgqH0=', 'Content-Length': '0'} 2023-07-29T00:24:23.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-57:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b74c7ccd200167b-0064c45c37-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:23.996 DEBUG:boto:path=/hiiqos-57/h 2023-07-29T00:24:23.997 DEBUG:boto:auth_path=/hiiqos-57/h 2023-07-29T00:24:23.998 DEBUG:boto:Method: GET 2023-07-29T00:24:23.998 DEBUG:boto:Path: /hiiqos-57/h 2023-07-29T00:24:23.999 DEBUG:boto:Data: 2023-07-29T00:24:23.999 DEBUG:boto:Headers: {} 2023-07-29T00:24:23.999 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:24.000 DEBUG:boto:Port: 8000 2023-07-29T00:24:24.000 DEBUG:boto:Params: {} 2023-07-29T00:24:24.001 DEBUG:boto:Token: None 2023-07-29T00:24:24.001 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:24 GMT /hiiqos-57/h 2023-07-29T00:24:24.002 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:A8iSVPZadyzpHg4PZlPwvZjYYCg= 2023-07-29T00:24:24.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:A8iSVPZadyzpHg4PZlPwvZjYYCg=', 'Content-Length': '0'} 2023-07-29T00:24:24.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:21:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dfc472fe35ad657-0064c45c37-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:24.006 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:24:24.006 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:24:24.007 DEBUG:boto:path=/hiiqos-57/?prefix=h 2023-07-29T00:24:24.007 DEBUG:boto:auth_path=/hiiqos-57/?prefix=h 2023-07-29T00:24:24.008 DEBUG:boto:Method: GET 2023-07-29T00:24:24.008 DEBUG:boto:Path: /hiiqos-57/?prefix=h 2023-07-29T00:24:24.008 DEBUG:boto:Data: 2023-07-29T00:24:24.009 DEBUG:boto:Headers: {} 2023-07-29T00:24:24.009 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:24.009 DEBUG:boto:Port: 8001 2023-07-29T00:24:24.010 DEBUG:boto:Params: {} 2023-07-29T00:24:24.010 DEBUG:boto:Token: None 2023-07-29T00:24:24.011 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:24 GMT /hiiqos-57/ 2023-07-29T00:24:24.011 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/rxZ4zdsbCru9SBG1cMZIYexIKo= 2023-07-29T00:24:24.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/rxZ4zdsbCru9SBG1cMZIYexIKo=', 'Content-Length': '0'} 2023-07-29T00:24:24.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b6f5dc94b2f447a-0064c45c37-1b40-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:24.015 DEBUG:boto:b'hiiqos-57h1000falseh2023-07-29T00:21:40.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:24.016 DEBUG:boto:path=/hiiqos-57/ 2023-07-29T00:24:24.016 DEBUG:boto:auth_path=/hiiqos-57/ 2023-07-29T00:24:24.016 DEBUG:boto:path=/hiiqos-57/?prefix=h 2023-07-29T00:24:24.017 DEBUG:boto:auth_path=/hiiqos-57/?prefix=h 2023-07-29T00:24:24.017 DEBUG:boto:Method: GET 2023-07-29T00:24:24.018 DEBUG:boto:Path: /hiiqos-57/?prefix=h 2023-07-29T00:24:24.018 DEBUG:boto:Data: 2023-07-29T00:24:24.018 DEBUG:boto:Headers: {} 2023-07-29T00:24:24.019 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:24.019 DEBUG:boto:Port: 8000 2023-07-29T00:24:24.020 DEBUG:boto:Params: {} 2023-07-29T00:24:24.021 DEBUG:boto:Token: None 2023-07-29T00:24:24.021 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:24 GMT /hiiqos-57/ 2023-07-29T00:24:24.022 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/rxZ4zdsbCru9SBG1cMZIYexIKo= 2023-07-29T00:24:24.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/rxZ4zdsbCru9SBG1cMZIYexIKo=', 'Content-Length': '0'} 2023-07-29T00:24:24.024 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c13c7ad1ce580bc1-0064c45c37-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:24.025 DEBUG:boto:b'hiiqos-57h1000falseh2023-07-29T00:21:40.863Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:24.025 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-57 zones={a2, a1} 2023-07-29T00:24:24.026 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_full ... ok 2023-07-29T00:24:24.027 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:24:24.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:24:24.028 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:24:24.080 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:24:24.081 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:24:24.113 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T00:24:24.127 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T00:24:24.127 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:24:24.128 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:24:24.128 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:46:09.233742Z" 2023-07-29T00:24:24.128 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:24:24.128 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:24:24.128 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690590100.721445_914.1", 2023-07-29T00:24:24.128 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:21:40.721445Z" 2023-07-29T00:24:24.129 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:24:24.129 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:24:24.129 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:24:24.129 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T00:24:24.129 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:24:24.129 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:24:24.129 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:24:24.130 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T00:24:24.130 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:24:24.130 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T00:24:24.131 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690587969.233742_99.1', 1: '1_1690590100.721445_914.1', 2: '', 3: ''} 2023-07-29T00:24:24.131 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T00:24:24.132 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:24:24.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:24:24.132 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:24:24.179 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:24:24.179 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:24:24.207 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:24:24.156+0000 7fa557d31880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:24:24.208 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:24:24.157+0000 7fa557d31880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:24:24.221 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:24:24.221 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:24:24.221 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:24:24.221 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:24:24.222 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:24:24.222 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:24:24.222 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:24:24.222 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:24:24.222 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:24:24.222 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:24:24.222 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:24:24.223 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:24:24.223 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:24:24.223 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:24:24.223 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:24:24.223 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:24:24.223 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:24:24.223 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:24:24.224 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:24:24.224 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:24:24.224 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:24:24.224 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:24:24.224 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:24:24.224 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:24:24.225 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:24:24.225 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690590100.721445_914.1", 2023-07-29T00:24:24.225 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:24:24.225 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:24:24.225 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:24:24.225 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:21:40.721445Z", 2023-07-29T00:24:24.225 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:24:24.226 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:24:24.226 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:24:24.226 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:24:24.226 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:24:24.226 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:24:24.226 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:24:24.226 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:24:24.227 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:24:24.227 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:24:24.227 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:24:24.227 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:24:24.227 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:24:24.227 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:24:24.227 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:24:24.228 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:24:24.228 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:24:24.228 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:24:24.228 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:24:24.228 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:24:24.228 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:24:24.229 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:24:24.229 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:24:24.229 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:24:24.229 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:24:24.229 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:24:24.229 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:24:24.229 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:24:24.230 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:24:24.230 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:24:24.230 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:24:24.230 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:24:24.230 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:24:24.230 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:24:24Z" 2023-07-29T00:24:24.230 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:24:24.231 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690590100.721445_914.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:21:40.721445Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:24:24Z" } 2023-07-29T00:24:24.232 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690590100.721445_914.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:21:40.721445Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:24:24.232 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690590100.721445_914.1', 2: '', 3: ''} 2023-07-29T00:24:24.233 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690590100.721445_914.1', 2: '', 3: ''} 2023-07-29T00:24:24.233 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T00:24:24.233 DEBUG:boto:path=/ 2023-07-29T00:24:24.234 DEBUG:boto:auth_path=/ 2023-07-29T00:24:24.234 DEBUG:boto:Method: GET 2023-07-29T00:24:24.234 DEBUG:boto:Path: / 2023-07-29T00:24:24.235 DEBUG:boto:Data: 2023-07-29T00:24:24.235 DEBUG:boto:Headers: {} 2023-07-29T00:24:24.235 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:24.236 DEBUG:boto:Port: 8000 2023-07-29T00:24:24.236 DEBUG:boto:Params: {} 2023-07-29T00:24:24.236 DEBUG:boto:Token: None 2023-07-29T00:24:24.237 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:24 GMT / 2023-07-29T00:24:24.237 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3LdwBjvKLy5vgBlj2+wKKFaSIlo= 2023-07-29T00:24:24.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3LdwBjvKLy5vgBlj2+wKKFaSIlo=', 'Content-Length': '0'} 2023-07-29T00:24:24.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050e2a8bcddded732-0064c45c38-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:24.242 DEBUG:boto:path=/ 2023-07-29T00:24:24.243 DEBUG:boto:auth_path=/ 2023-07-29T00:24:24.243 DEBUG:boto:Method: GET 2023-07-29T00:24:24.243 DEBUG:boto:Path: / 2023-07-29T00:24:24.244 DEBUG:boto:Data: 2023-07-29T00:24:24.244 DEBUG:boto:Headers: {} 2023-07-29T00:24:24.244 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:24.245 DEBUG:boto:Port: 8001 2023-07-29T00:24:24.245 DEBUG:boto:Params: {} 2023-07-29T00:24:24.245 DEBUG:boto:Token: None 2023-07-29T00:24:24.246 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:24 GMT / 2023-07-29T00:24:24.246 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3LdwBjvKLy5vgBlj2+wKKFaSIlo= 2023-07-29T00:24:24.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3LdwBjvKLy5vgBlj2+wKKFaSIlo=', 'Content-Length': '0'} 2023-07-29T00:24:24.248 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000291cdba526bd9a29-0064c45c38-1b40-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:24.250 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_creation_time ... ok 2023-07-29T00:24:24.250 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:24.251 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:24.251 DEBUG:boto:Method: PUT 2023-07-29T00:24:24.251 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:24.252 DEBUG:boto:Data: 2023-07-29T00:24:24.252 DEBUG:boto:Headers: {} 2023-07-29T00:24:24.252 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:24.253 DEBUG:boto:Port: 8000 2023-07-29T00:24:24.253 DEBUG:boto:Params: {} 2023-07-29T00:24:24.253 DEBUG:boto:Token: None 2023-07-29T00:24:24.254 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 00:24:24 GMT /hiiqos-58/ 2023-07-29T00:24:24.254 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HH+5b8/+AO9LXFlGJIEq7wpgtDc= 2023-07-29T00:24:24.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HH+5b8/+AO9LXFlGJIEq7wpgtDc=', 'Content-Length': '0'} 2023-07-29T00:24:24.268 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008c3c1ee0cd3029c7-0064c45c38-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:24.268 DEBUG:rgw_multi.tests:created bucket=hiiqos-58 2023-07-29T00:24:24.269 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:24:24.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:24:24.270 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:24:24.314 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:24:24.315 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:24:24.345 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T00:24:24.358 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T00:24:24.359 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:24:24.359 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:24:24.359 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:46:09.233742Z" 2023-07-29T00:24:24.359 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:24:24.359 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:24:24.359 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690590264.216448_923.1", 2023-07-29T00:24:24.360 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:24:24.216448Z" 2023-07-29T00:24:24.360 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:24:24.360 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:24:24.360 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:24:24.360 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T00:24:24.360 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:24:24.360 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:24:24.361 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:24:24.361 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T00:24:24.361 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:24:24.361 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T00:24:24.362 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690587969.233742_99.1', 1: '1_1690590264.216448_923.1', 2: '', 3: ''} 2023-07-29T00:24:24.362 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T00:24:24.363 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:24:24.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:24:24.364 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:24:24.412 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:24:24.412 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:24:24.440 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:24:24.388+0000 7f4f0580f880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:24:24.440 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:24:24.389+0000 7f4f0580f880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:24:24.453 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:24:24.453 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:24:24.454 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:24:24.454 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:24:24.454 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:24:24.454 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:24:24.454 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:24:24.454 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:24:24.454 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:24:24.455 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:24:24.455 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:24:24.455 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:24:24.455 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:24:24.455 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:24:24.455 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:24:24.455 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:24:24.456 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:24:24.456 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:24:24.456 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:24:24.456 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:24:24.456 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:24:24.456 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:24:24.457 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:24:24.457 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:24:24.457 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:24:24.457 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690590264.216448_923.1", 2023-07-29T00:24:24.457 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:24:24.457 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:24:24.457 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:24:24.458 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:24:24.216448Z", 2023-07-29T00:24:24.458 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:24:24.458 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:24:24.458 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:24:24.458 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:24:24.458 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:24:24.458 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:24:24.459 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:24:24.459 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:24:24.459 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:24:24.459 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:24:24.459 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:24:24.459 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:24:24.459 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:24:24.460 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:24:24.460 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:24:24.460 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:24:24.460 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:24:24.460 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:24:24.460 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:24:24.460 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:24:24.461 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:24:24.461 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:24:24.461 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:24:24.461 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:24:24.461 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:24:24.461 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:24:24.461 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:24:24.462 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:24:24.462 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:24:24.462 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:24:24.462 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:24:24.462 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:24:24.462 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:24:24.462 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:24:24Z" 2023-07-29T00:24:24.463 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:24:24.463 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690590264.216448_923.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:24:24.216448Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:24:24Z" } 2023-07-29T00:24:24.464 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690590264.216448_923.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:24:24.216448Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:24:24.464 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690590264.216448_923.1', 2: '', 3: ''} 2023-07-29T00:24:24.465 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690590264.216448_923.1', 2: '', 3: ''} 2023-07-29T00:24:24.465 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T00:24:24.465 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:24:24.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:24:24.466 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:24:24.514 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:24:24.514 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:24:24.553 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:24:24.553 INFO:teuthology.orchestra.run.smithi101.stdout: "objs": [ 2023-07-29T00:24:24.553 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000001", 2023-07-29T00:24:24.553 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000005", 2023-07-29T00:24:24.554 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000012", 2023-07-29T00:24:24.554 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000004", 2023-07-29T00:24:24.554 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000011", 2023-07-29T00:24:24.554 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000000", 2023-07-29T00:24:24.554 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000010", 2023-07-29T00:24:24.554 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000003", 2023-07-29T00:24:24.555 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000013", 2023-07-29T00:24:24.555 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000002", 2023-07-29T00:24:24.555 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000006" 2023-07-29T00:24:24.555 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:24:24.555 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:24:24.556 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:24.556 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:24.557 DEBUG:boto:Method: HEAD 2023-07-29T00:24:24.557 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:24.558 DEBUG:boto:Data: 2023-07-29T00:24:24.558 DEBUG:boto:Headers: {} 2023-07-29T00:24:24.558 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:24.558 DEBUG:boto:Port: 8000 2023-07-29T00:24:24.559 DEBUG:boto:Params: {} 2023-07-29T00:24:24.559 DEBUG:boto:Token: None 2023-07-29T00:24:24.560 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:24 GMT /hiiqos-58/ 2023-07-29T00:24:24.560 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HrDBCNCRUmRiF4JseE4QBDxWvxg= 2023-07-29T00:24:24.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HrDBCNCRUmRiF4JseE4QBDxWvxg=', 'Content-Length': '0'} 2023-07-29T00:24:24.563 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f5d3647cbb8ad16c-0064c45c38-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:24.564 DEBUG:boto:path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:24:24.564 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:24:24.564 DEBUG:boto:Method: PUT 2023-07-29T00:24:24.565 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000003 2023-07-29T00:24:24.565 DEBUG:boto:Data: 2023-07-29T00:24:24.565 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:24.566 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:24.566 DEBUG:boto:Port: 8000 2023-07-29T00:24:24.566 DEBUG:boto:Params: {} 2023-07-29T00:24:24.567 DEBUG:boto:Token: None 2023-07-29T00:24:24.567 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:24 GMT /hiiqos-58/obj00000000000000000003 2023-07-29T00:24:24.568 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PYAulW/QhHTycKiXC6uTmTMmJIM= 2023-07-29T00:24:24.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PYAulW/QhHTycKiXC6uTmTMmJIM='} 2023-07-29T00:24:24.574 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ea0e007052493296-0064c45c38-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:24.575 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:24.575 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:24.576 DEBUG:boto:Method: HEAD 2023-07-29T00:24:24.576 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:24.576 DEBUG:boto:Data: 2023-07-29T00:24:24.577 DEBUG:boto:Headers: {} 2023-07-29T00:24:24.577 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:24.577 DEBUG:boto:Port: 8000 2023-07-29T00:24:24.578 DEBUG:boto:Params: {} 2023-07-29T00:24:24.578 DEBUG:boto:Token: None 2023-07-29T00:24:24.578 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:24 GMT /hiiqos-58/ 2023-07-29T00:24:24.579 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HrDBCNCRUmRiF4JseE4QBDxWvxg= 2023-07-29T00:24:24.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HrDBCNCRUmRiF4JseE4QBDxWvxg=', 'Content-Length': '0'} 2023-07-29T00:24:24.581 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000def407ac527af28b-0064c45c38-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:24.582 DEBUG:boto:path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:24:24.582 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:24:24.583 DEBUG:boto:Method: PUT 2023-07-29T00:24:24.583 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000006 2023-07-29T00:24:24.583 DEBUG:boto:Data: 2023-07-29T00:24:24.584 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:24.584 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:24.584 DEBUG:boto:Port: 8000 2023-07-29T00:24:24.585 DEBUG:boto:Params: {} 2023-07-29T00:24:24.585 DEBUG:boto:Token: None 2023-07-29T00:24:24.585 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:24 GMT /hiiqos-58/obj00000000000000000006 2023-07-29T00:24:24.586 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hEjxPV2e7UeXor2+gCfj0V4MPK0= 2023-07-29T00:24:24.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hEjxPV2e7UeXor2+gCfj0V4MPK0='} 2023-07-29T00:24:24.591 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d3cfa3467a95afa1-0064c45c38-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:24.591 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:24.591 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:24.592 DEBUG:boto:Method: HEAD 2023-07-29T00:24:24.592 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:24.593 DEBUG:boto:Data: 2023-07-29T00:24:24.593 DEBUG:boto:Headers: {} 2023-07-29T00:24:24.593 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:24.594 DEBUG:boto:Port: 8000 2023-07-29T00:24:24.594 DEBUG:boto:Params: {} 2023-07-29T00:24:24.594 DEBUG:boto:Token: None 2023-07-29T00:24:24.595 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:24 GMT /hiiqos-58/ 2023-07-29T00:24:24.595 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HrDBCNCRUmRiF4JseE4QBDxWvxg= 2023-07-29T00:24:24.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HrDBCNCRUmRiF4JseE4QBDxWvxg=', 'Content-Length': '0'} 2023-07-29T00:24:24.597 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004ae00cab4b407dd3-0064c45c38-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:24.598 DEBUG:boto:path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:24:24.598 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:24:24.599 DEBUG:boto:Method: PUT 2023-07-29T00:24:24.599 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000004 2023-07-29T00:24:24.599 DEBUG:boto:Data: 2023-07-29T00:24:24.600 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:24.600 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:24.600 DEBUG:boto:Port: 8000 2023-07-29T00:24:24.601 DEBUG:boto:Params: {} 2023-07-29T00:24:24.601 DEBUG:boto:Token: None 2023-07-29T00:24:24.602 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:24 GMT /hiiqos-58/obj00000000000000000004 2023-07-29T00:24:24.602 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cj8cUy8Y1pad4T/ivtNk2xCr8tE= 2023-07-29T00:24:24.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cj8cUy8Y1pad4T/ivtNk2xCr8tE='} 2023-07-29T00:24:24.644 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a44288ec074f0de5-0064c45c38-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:24.645 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:24.646 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:24.646 DEBUG:boto:Method: HEAD 2023-07-29T00:24:24.646 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:24.647 DEBUG:boto:Data: 2023-07-29T00:24:24.647 DEBUG:boto:Headers: {} 2023-07-29T00:24:24.647 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:24.648 DEBUG:boto:Port: 8000 2023-07-29T00:24:24.648 DEBUG:boto:Params: {} 2023-07-29T00:24:24.648 DEBUG:boto:Token: None 2023-07-29T00:24:24.649 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:24 GMT /hiiqos-58/ 2023-07-29T00:24:24.649 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HrDBCNCRUmRiF4JseE4QBDxWvxg= 2023-07-29T00:24:24.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HrDBCNCRUmRiF4JseE4QBDxWvxg=', 'Content-Length': '0'} 2023-07-29T00:24:24.652 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000012a29d0d195f6185-0064c45c38-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:24.653 DEBUG:boto:path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:24:24.653 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:24:24.653 DEBUG:boto:Method: PUT 2023-07-29T00:24:24.654 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000001 2023-07-29T00:24:24.654 DEBUG:boto:Data: 2023-07-29T00:24:24.654 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:24.655 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:24.655 DEBUG:boto:Port: 8000 2023-07-29T00:24:24.655 DEBUG:boto:Params: {} 2023-07-29T00:24:24.656 DEBUG:boto:Token: None 2023-07-29T00:24:24.656 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:24 GMT /hiiqos-58/obj00000000000000000001 2023-07-29T00:24:24.657 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ky1EdMclhbYsu3FmEI9QnUY/r+I= 2023-07-29T00:24:24.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ky1EdMclhbYsu3FmEI9QnUY/r+I='} 2023-07-29T00:24:24.662 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ba1d239cb5a86e7a-0064c45c38-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:24.663 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:24.663 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:24.664 DEBUG:boto:Method: HEAD 2023-07-29T00:24:24.665 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:24.666 DEBUG:boto:Data: 2023-07-29T00:24:24.667 DEBUG:boto:Headers: {} 2023-07-29T00:24:24.668 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:24.668 DEBUG:boto:Port: 8000 2023-07-29T00:24:24.669 DEBUG:boto:Params: {} 2023-07-29T00:24:24.670 DEBUG:boto:Token: None 2023-07-29T00:24:24.671 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:24 GMT /hiiqos-58/ 2023-07-29T00:24:24.672 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HrDBCNCRUmRiF4JseE4QBDxWvxg= 2023-07-29T00:24:24.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HrDBCNCRUmRiF4JseE4QBDxWvxg=', 'Content-Length': '0'} 2023-07-29T00:24:24.676 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f10715434bf17961-0064c45c38-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:24.677 DEBUG:boto:path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:24:24.677 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:24:24.678 DEBUG:boto:Method: PUT 2023-07-29T00:24:24.678 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000002 2023-07-29T00:24:24.678 DEBUG:boto:Data: 2023-07-29T00:24:24.679 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:24.679 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:24.679 DEBUG:boto:Port: 8000 2023-07-29T00:24:24.680 DEBUG:boto:Params: {} 2023-07-29T00:24:24.680 DEBUG:boto:Token: None 2023-07-29T00:24:24.681 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:24 GMT /hiiqos-58/obj00000000000000000002 2023-07-29T00:24:24.681 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zVxF5iJmPR8BkgKB9QisZjiKKfU= 2023-07-29T00:24:24.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zVxF5iJmPR8BkgKB9QisZjiKKfU='} 2023-07-29T00:24:24.687 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000055849cd51497b2e2-0064c45c38-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:24.687 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:24.688 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:24.688 DEBUG:boto:Method: HEAD 2023-07-29T00:24:24.689 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:24.689 DEBUG:boto:Data: 2023-07-29T00:24:24.689 DEBUG:boto:Headers: {} 2023-07-29T00:24:24.690 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:24.690 DEBUG:boto:Port: 8000 2023-07-29T00:24:24.690 DEBUG:boto:Params: {} 2023-07-29T00:24:24.691 DEBUG:boto:Token: None 2023-07-29T00:24:24.691 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:24 GMT /hiiqos-58/ 2023-07-29T00:24:24.691 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HrDBCNCRUmRiF4JseE4QBDxWvxg= 2023-07-29T00:24:24.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HrDBCNCRUmRiF4JseE4QBDxWvxg=', 'Content-Length': '0'} 2023-07-29T00:24:24.694 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001df9f2f6919500cc-0064c45c38-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:24.695 DEBUG:boto:path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:24:24.695 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:24:24.696 DEBUG:boto:Method: PUT 2023-07-29T00:24:24.696 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000000 2023-07-29T00:24:24.696 DEBUG:boto:Data: 2023-07-29T00:24:24.697 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:24.697 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:24.698 DEBUG:boto:Port: 8000 2023-07-29T00:24:24.698 DEBUG:boto:Params: {} 2023-07-29T00:24:24.698 DEBUG:boto:Token: None 2023-07-29T00:24:24.699 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:24 GMT /hiiqos-58/obj00000000000000000000 2023-07-29T00:24:24.699 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:m8WtnP2PzKiVYp/jwehx0yI2hrA= 2023-07-29T00:24:24.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:m8WtnP2PzKiVYp/jwehx0yI2hrA='} 2023-07-29T00:24:24.745 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f088bec9812c0b9b-0064c45c38-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:24.746 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:24.746 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:24.746 DEBUG:boto:Method: HEAD 2023-07-29T00:24:24.747 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:24.747 DEBUG:boto:Data: 2023-07-29T00:24:24.747 DEBUG:boto:Headers: {} 2023-07-29T00:24:24.748 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:24.748 DEBUG:boto:Port: 8000 2023-07-29T00:24:24.748 DEBUG:boto:Params: {} 2023-07-29T00:24:24.749 DEBUG:boto:Token: None 2023-07-29T00:24:24.749 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:24 GMT /hiiqos-58/ 2023-07-29T00:24:24.750 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HrDBCNCRUmRiF4JseE4QBDxWvxg= 2023-07-29T00:24:24.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HrDBCNCRUmRiF4JseE4QBDxWvxg=', 'Content-Length': '0'} 2023-07-29T00:24:24.752 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000319c899271a67208-0064c45c38-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:24.753 DEBUG:boto:path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:24:24.753 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:24:24.754 DEBUG:boto:Method: PUT 2023-07-29T00:24:24.754 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000013 2023-07-29T00:24:24.754 DEBUG:boto:Data: 2023-07-29T00:24:24.755 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:24.755 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:24.755 DEBUG:boto:Port: 8000 2023-07-29T00:24:24.756 DEBUG:boto:Params: {} 2023-07-29T00:24:24.756 DEBUG:boto:Token: None 2023-07-29T00:24:24.756 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:24 GMT /hiiqos-58/obj00000000000000000013 2023-07-29T00:24:24.757 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:monfZyeNZCvaokREmrDmU//x8oM= 2023-07-29T00:24:24.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:monfZyeNZCvaokREmrDmU//x8oM='} 2023-07-29T00:24:24.762 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f1f94ec505635e48-0064c45c38-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:24.763 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:24.763 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:24.763 DEBUG:boto:Method: HEAD 2023-07-29T00:24:24.764 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:24.764 DEBUG:boto:Data: 2023-07-29T00:24:24.764 DEBUG:boto:Headers: {} 2023-07-29T00:24:24.765 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:24.765 DEBUG:boto:Port: 8000 2023-07-29T00:24:24.766 DEBUG:boto:Params: {} 2023-07-29T00:24:24.766 DEBUG:boto:Token: None 2023-07-29T00:24:24.766 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:24 GMT /hiiqos-58/ 2023-07-29T00:24:24.767 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HrDBCNCRUmRiF4JseE4QBDxWvxg= 2023-07-29T00:24:24.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HrDBCNCRUmRiF4JseE4QBDxWvxg=', 'Content-Length': '0'} 2023-07-29T00:24:24.769 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ff4e0eeabe3c4ed-0064c45c38-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:24.770 DEBUG:boto:path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:24:24.771 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:24:24.771 DEBUG:boto:Method: PUT 2023-07-29T00:24:24.771 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000005 2023-07-29T00:24:24.772 DEBUG:boto:Data: 2023-07-29T00:24:24.772 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:24.772 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:24.773 DEBUG:boto:Port: 8000 2023-07-29T00:24:24.773 DEBUG:boto:Params: {} 2023-07-29T00:24:24.773 DEBUG:boto:Token: None 2023-07-29T00:24:24.774 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:24 GMT /hiiqos-58/obj00000000000000000005 2023-07-29T00:24:24.774 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Ay3eiAXhdd3QHhCv2h7WBdog4hM= 2023-07-29T00:24:24.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Ay3eiAXhdd3QHhCv2h7WBdog4hM='} 2023-07-29T00:24:24.820 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ce5aa34407c261cf-0064c45c38-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:24.821 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:24.822 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:24.822 DEBUG:boto:Method: HEAD 2023-07-29T00:24:24.822 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:24.823 DEBUG:boto:Data: 2023-07-29T00:24:24.823 DEBUG:boto:Headers: {} 2023-07-29T00:24:24.823 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:24.824 DEBUG:boto:Port: 8000 2023-07-29T00:24:24.824 DEBUG:boto:Params: {} 2023-07-29T00:24:24.824 DEBUG:boto:Token: None 2023-07-29T00:24:24.825 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:24 GMT /hiiqos-58/ 2023-07-29T00:24:24.825 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HrDBCNCRUmRiF4JseE4QBDxWvxg= 2023-07-29T00:24:24.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HrDBCNCRUmRiF4JseE4QBDxWvxg=', 'Content-Length': '0'} 2023-07-29T00:24:24.828 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009069b9d3d8a992f4-0064c45c38-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:24.829 DEBUG:boto:path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:24:24.829 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:24:24.829 DEBUG:boto:Method: PUT 2023-07-29T00:24:24.830 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000011 2023-07-29T00:24:24.830 DEBUG:boto:Data: 2023-07-29T00:24:24.830 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:24.831 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:24.831 DEBUG:boto:Port: 8000 2023-07-29T00:24:24.831 DEBUG:boto:Params: {} 2023-07-29T00:24:24.832 DEBUG:boto:Token: None 2023-07-29T00:24:24.832 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:24 GMT /hiiqos-58/obj00000000000000000011 2023-07-29T00:24:24.833 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J0eFyRRryvkNay5vRu5zb5OrD7A= 2023-07-29T00:24:24.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J0eFyRRryvkNay5vRu5zb5OrD7A='} 2023-07-29T00:24:24.837 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000da7cb45df85badc5-0064c45c38-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:24.838 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:24.838 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:24.839 DEBUG:boto:Method: HEAD 2023-07-29T00:24:24.839 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:24.840 DEBUG:boto:Data: 2023-07-29T00:24:24.840 DEBUG:boto:Headers: {} 2023-07-29T00:24:24.840 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:24.841 DEBUG:boto:Port: 8000 2023-07-29T00:24:24.841 DEBUG:boto:Params: {} 2023-07-29T00:24:24.841 DEBUG:boto:Token: None 2023-07-29T00:24:24.842 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:24 GMT /hiiqos-58/ 2023-07-29T00:24:24.842 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HrDBCNCRUmRiF4JseE4QBDxWvxg= 2023-07-29T00:24:24.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HrDBCNCRUmRiF4JseE4QBDxWvxg=', 'Content-Length': '0'} 2023-07-29T00:24:24.845 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c90d6503f8b3ed9-0064c45c38-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:24.845 DEBUG:boto:path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:24:24.846 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:24:24.846 DEBUG:boto:Method: PUT 2023-07-29T00:24:24.847 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000012 2023-07-29T00:24:24.847 DEBUG:boto:Data: 2023-07-29T00:24:24.847 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:24.848 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:24.848 DEBUG:boto:Port: 8000 2023-07-29T00:24:24.848 DEBUG:boto:Params: {} 2023-07-29T00:24:24.849 DEBUG:boto:Token: None 2023-07-29T00:24:24.849 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:24 GMT /hiiqos-58/obj00000000000000000012 2023-07-29T00:24:24.849 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:F9FrcSFKDbYaLz2jKPBjXP5cGXM= 2023-07-29T00:24:24.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:F9FrcSFKDbYaLz2jKPBjXP5cGXM='} 2023-07-29T00:24:24.892 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cb3a379e094d4e9e-0064c45c38-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:24.893 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-58 2023-07-29T00:24:24.894 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:24:24.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:24:24.895 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-58 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:24:24.947 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:24:24.947 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:24:24.980 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:24:24.928+0000 7f5368322880 1 waiting to reach incremental sync.. 2023-07-29T00:24:29.980 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:24:29.929+0000 7f5368322880 1 waiting to reach incremental sync.. 2023-07-29T00:24:34.981 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:24:34.930+0000 7f5368322880 1 waiting to reach incremental sync.. 2023-07-29T00:24:39.982 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:24:39.930+0000 7f5368322880 1 waiting to reach incremental sync.. 2023-07-29T00:24:44.984 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:24:44.932+0000 7f5368322880 1 bucket sync caught up with source: 2023-07-29T00:24:44.984 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000001.72.6, 00000000001.37.6, 00000000001.30.6, 00000000001.37.6, 00000000001.58.6, 00000000001.46.6, , 00000000001.58.6, 00000000001.38.6, 00000000001.30.6, 00000000001.35.6] 2023-07-29T00:24:44.984 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000001.72.6, 00000000001.37.6, 00000000001.30.6, 00000000001.37.6, 00000000001.58.6, 00000000001.46.6, , 00000000001.58.6, 00000000001.38.6, 00000000001.30.6, 00000000001.35.6] 2023-07-29T00:24:44.984 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:24:44.932+0000 7f5368322880 0 bucket checkpoint complete 2023-07-29T00:24:45.001 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-58 2023-07-29T00:24:45.002 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:24:45.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:24:45.003 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-58 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:24:45.061 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:24:45.062 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:24:45.098 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:24:45.046+0000 7fa692f2a880 1 bucket sync caught up with source: 2023-07-29T00:24:45.098 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000001.885.6, 00000000001.876.6, 00000000001.1336.6, 00000000001.1183.6, 00000000001.113.6, 00000000001.1035.6, , 00000000001.886.6, 00000000001.2438.6, 00000000001.247.6, 00000000001.877.6] 2023-07-29T00:24:45.099 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000001.885.6, 00000000001.876.6, 00000000001.1336.6, 00000000001.1183.6, 00000000001.113.6, 00000000001.1035.6, , 00000000001.886.6, 00000000001.2438.6, 00000000001.247.6, 00000000001.877.6] 2023-07-29T00:24:45.099 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:24:45.046+0000 7fa692f2a880 0 bucket checkpoint complete 2023-07-29T00:24:45.112 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-58 zones={a2, a1} 2023-07-29T00:24:45.112 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:45.113 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:45.113 DEBUG:boto:Method: HEAD 2023-07-29T00:24:45.113 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:45.114 DEBUG:boto:Data: 2023-07-29T00:24:45.114 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.114 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:45.115 DEBUG:boto:Port: 8001 2023-07-29T00:24:45.115 DEBUG:boto:Params: {} 2023-07-29T00:24:45.115 DEBUG:boto:Token: None 2023-07-29T00:24:45.116 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/ 2023-07-29T00:24:45.116 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yb6mZeRJVVo/KGNLy9x3dGjPEEw= 2023-07-29T00:24:45.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yb6mZeRJVVo/KGNLy9x3dGjPEEw=', 'Content-Length': '0'} 2023-07-29T00:24:45.119 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000013f6b989064312ba-0064c45c4d-1b40-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.120 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:45.120 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:45.120 DEBUG:boto:Method: HEAD 2023-07-29T00:24:45.121 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:45.121 DEBUG:boto:Data: 2023-07-29T00:24:45.121 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.122 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:45.122 DEBUG:boto:Port: 8000 2023-07-29T00:24:45.122 DEBUG:boto:Params: {} 2023-07-29T00:24:45.123 DEBUG:boto:Token: None 2023-07-29T00:24:45.123 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/ 2023-07-29T00:24:45.124 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yb6mZeRJVVo/KGNLy9x3dGjPEEw= 2023-07-29T00:24:45.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yb6mZeRJVVo/KGNLy9x3dGjPEEw=', 'Content-Length': '0'} 2023-07-29T00:24:45.127 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000acf5a1c6b0cc157a-0064c45c4d-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.127 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-29T00:24:45.127 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:45.128 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:45.128 DEBUG:boto:path=/hiiqos-58/?versions&max-keys=999 2023-07-29T00:24:45.129 DEBUG:boto:auth_path=/hiiqos-58/?versions&max-keys=999 2023-07-29T00:24:45.129 DEBUG:boto:Method: GET 2023-07-29T00:24:45.129 DEBUG:boto:Path: /hiiqos-58/?versions&max-keys=999 2023-07-29T00:24:45.130 DEBUG:boto:Data: 2023-07-29T00:24:45.130 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.130 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:45.131 DEBUG:boto:Port: 8001 2023-07-29T00:24:45.131 DEBUG:boto:Params: {} 2023-07-29T00:24:45.131 DEBUG:boto:Token: None 2023-07-29T00:24:45.132 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/?versions 2023-07-29T00:24:45.132 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0zK9ehWt415hMfhlkkffOSF1Dmg= 2023-07-29T00:24:45.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0zK9ehWt415hMfhlkkffOSF1Dmg=', 'Content-Length': '0'} 2023-07-29T00:24:45.135 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ebe0b47a2b7892f-0064c45c4d-1b40-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.135 DEBUG:boto:b'hiiqos-58999falseobj00000000000000000000nulltrue2023-07-29T00:24:24.651Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-07-29T00:24:24.608Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-07-29T00:24:24.633Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-07-29T00:24:24.520Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-07-29T00:24:24.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-07-29T00:24:24.726Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-07-29T00:24:24.537Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-07-29T00:24:24.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-07-29T00:24:24.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-07-29T00:24:24.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:45.136 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2023-07-29T00:24:45.136 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2023-07-29T00:24:45.137 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2023-07-29T00:24:45.137 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2023-07-29T00:24:45.137 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2023-07-29T00:24:45.138 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2023-07-29T00:24:45.138 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2023-07-29T00:24:45.138 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2023-07-29T00:24:45.139 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2023-07-29T00:24:45.139 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2023-07-29T00:24:45.139 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-29T00:24:45.140 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:45.140 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:45.140 DEBUG:boto:path=/hiiqos-58/?versions&max-keys=999 2023-07-29T00:24:45.141 DEBUG:boto:auth_path=/hiiqos-58/?versions&max-keys=999 2023-07-29T00:24:45.141 DEBUG:boto:Method: GET 2023-07-29T00:24:45.141 DEBUG:boto:Path: /hiiqos-58/?versions&max-keys=999 2023-07-29T00:24:45.142 DEBUG:boto:Data: 2023-07-29T00:24:45.142 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.142 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:45.143 DEBUG:boto:Port: 8000 2023-07-29T00:24:45.143 DEBUG:boto:Params: {} 2023-07-29T00:24:45.143 DEBUG:boto:Token: None 2023-07-29T00:24:45.144 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/?versions 2023-07-29T00:24:45.144 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0zK9ehWt415hMfhlkkffOSF1Dmg= 2023-07-29T00:24:45.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0zK9ehWt415hMfhlkkffOSF1Dmg=', 'Content-Length': '0'} 2023-07-29T00:24:45.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000892e08a835bed0d0-0064c45c4d-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.147 DEBUG:boto:b'hiiqos-58999falseobj00000000000000000000nulltrue2023-07-29T00:24:24.651Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-07-29T00:24:24.608Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-07-29T00:24:24.633Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-07-29T00:24:24.520Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-07-29T00:24:24.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-07-29T00:24:24.726Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-07-29T00:24:24.537Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-07-29T00:24:24.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-07-29T00:24:24.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-07-29T00:24:24.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:45.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2023-07-29T00:24:45.149 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2023-07-29T00:24:45.149 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2023-07-29T00:24:45.149 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2023-07-29T00:24:45.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2023-07-29T00:24:45.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2023-07-29T00:24:45.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2023-07-29T00:24:45.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2023-07-29T00:24:45.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2023-07-29T00:24:45.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2023-07-29T00:24:45.151 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:45.152 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:45.152 DEBUG:boto:path=/hiiqos-58/?versions&max-keys=999 2023-07-29T00:24:45.153 DEBUG:boto:auth_path=/hiiqos-58/?versions&max-keys=999 2023-07-29T00:24:45.153 DEBUG:boto:Method: GET 2023-07-29T00:24:45.153 DEBUG:boto:Path: /hiiqos-58/?versions&max-keys=999 2023-07-29T00:24:45.154 DEBUG:boto:Data: 2023-07-29T00:24:45.154 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.154 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:45.155 DEBUG:boto:Port: 8001 2023-07-29T00:24:45.155 DEBUG:boto:Params: {} 2023-07-29T00:24:45.155 DEBUG:boto:Token: None 2023-07-29T00:24:45.156 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/?versions 2023-07-29T00:24:45.156 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0zK9ehWt415hMfhlkkffOSF1Dmg= 2023-07-29T00:24:45.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0zK9ehWt415hMfhlkkffOSF1Dmg=', 'Content-Length': '0'} 2023-07-29T00:24:45.159 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000095ff2454eb5a235-0064c45c4d-1b40-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.159 DEBUG:boto:b'hiiqos-58999falseobj00000000000000000000nulltrue2023-07-29T00:24:24.651Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-07-29T00:24:24.608Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-07-29T00:24:24.633Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-07-29T00:24:24.520Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-07-29T00:24:24.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-07-29T00:24:24.726Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-07-29T00:24:24.537Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-07-29T00:24:24.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-07-29T00:24:24.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-07-29T00:24:24.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:45.160 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:45.161 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:45.161 DEBUG:boto:path=/hiiqos-58/?versions&max-keys=999 2023-07-29T00:24:45.162 DEBUG:boto:auth_path=/hiiqos-58/?versions&max-keys=999 2023-07-29T00:24:45.162 DEBUG:boto:Method: GET 2023-07-29T00:24:45.162 DEBUG:boto:Path: /hiiqos-58/?versions&max-keys=999 2023-07-29T00:24:45.163 DEBUG:boto:Data: 2023-07-29T00:24:45.163 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.163 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:45.164 DEBUG:boto:Port: 8000 2023-07-29T00:24:45.164 DEBUG:boto:Params: {} 2023-07-29T00:24:45.164 DEBUG:boto:Token: None 2023-07-29T00:24:45.165 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/?versions 2023-07-29T00:24:45.165 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0zK9ehWt415hMfhlkkffOSF1Dmg= 2023-07-29T00:24:45.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0zK9ehWt415hMfhlkkffOSF1Dmg=', 'Content-Length': '0'} 2023-07-29T00:24:45.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000739c866a840aa30-0064c45c4d-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.168 DEBUG:boto:b'hiiqos-58999falseobj00000000000000000000nulltrue2023-07-29T00:24:24.651Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-07-29T00:24:24.608Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-07-29T00:24:24.633Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-07-29T00:24:24.520Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-07-29T00:24:24.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-07-29T00:24:24.726Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-07-29T00:24:24.537Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-07-29T00:24:24.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-07-29T00:24:24.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-07-29T00:24:24.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:45.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-07-29T00:24:45.170 DEBUG:boto:path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:24:45.170 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:24:45.170 DEBUG:boto:path=/hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:24:45.171 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:24:45.171 DEBUG:boto:Method: GET 2023-07-29T00:24:45.171 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:24:45.172 DEBUG:boto:Data: 2023-07-29T00:24:45.172 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.173 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:45.173 DEBUG:boto:Port: 8001 2023-07-29T00:24:45.173 DEBUG:boto:Params: {} 2023-07-29T00:24:45.174 DEBUG:boto:Token: None 2023-07-29T00:24:45.174 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:24:45.175 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vxVmE84YuiNS0YzbAGXgRiQdo9o= 2023-07-29T00:24:45.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vxVmE84YuiNS0YzbAGXgRiQdo9o=', 'Content-Length': '0'} 2023-07-29T00:24:45.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1921bfe1891d4c0-0064c45c4d-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.177 DEBUG:boto:path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:24:45.178 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:24:45.178 DEBUG:boto:path=/hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:24:45.178 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:24:45.179 DEBUG:boto:Method: GET 2023-07-29T00:24:45.179 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:24:45.179 DEBUG:boto:Data: 2023-07-29T00:24:45.180 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.180 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:45.181 DEBUG:boto:Port: 8000 2023-07-29T00:24:45.181 DEBUG:boto:Params: {} 2023-07-29T00:24:45.181 DEBUG:boto:Token: None 2023-07-29T00:24:45.182 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:24:45.182 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vxVmE84YuiNS0YzbAGXgRiQdo9o= 2023-07-29T00:24:45.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vxVmE84YuiNS0YzbAGXgRiQdo9o=', 'Content-Length': '0'} 2023-07-29T00:24:45.184 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000466da5ab7779f4e9-0064c45c4d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.186 DEBUG:boto:path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:24:45.186 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:24:45.186 DEBUG:boto:path=/hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:24:45.187 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:24:45.187 DEBUG:boto:Method: HEAD 2023-07-29T00:24:45.188 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:24:45.188 DEBUG:boto:Data: 2023-07-29T00:24:45.189 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.189 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:45.189 DEBUG:boto:Port: 8001 2023-07-29T00:24:45.190 DEBUG:boto:Params: {} 2023-07-29T00:24:45.190 DEBUG:boto:Token: None 2023-07-29T00:24:45.191 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:24:45.191 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:63U1jzBCXBecfo29a7H+dAsd+xc= 2023-07-29T00:24:45.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:63U1jzBCXBecfo29a7H+dAsd+xc=', 'Content-Length': '0'} 2023-07-29T00:24:45.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a42c602faa0ed1d1-0064c45c4d-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.194 DEBUG:boto:path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:24:45.194 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:24:45.194 DEBUG:boto:path=/hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:24:45.195 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:24:45.195 DEBUG:boto:Method: HEAD 2023-07-29T00:24:45.196 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:24:45.196 DEBUG:boto:Data: 2023-07-29T00:24:45.196 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.197 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:45.197 DEBUG:boto:Port: 8000 2023-07-29T00:24:45.197 DEBUG:boto:Params: {} 2023-07-29T00:24:45.198 DEBUG:boto:Token: None 2023-07-29T00:24:45.198 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:24:45.199 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:63U1jzBCXBecfo29a7H+dAsd+xc= 2023-07-29T00:24:45.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:63U1jzBCXBecfo29a7H+dAsd+xc=', 'Content-Length': '0'} 2023-07-29T00:24:45.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad53e7ab09d03768-0064c45c4d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-07-29T00:24:45.202 DEBUG:boto:path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:24:45.203 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:24:45.204 DEBUG:boto:path=/hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:24:45.205 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:24:45.206 DEBUG:boto:Method: GET 2023-07-29T00:24:45.207 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:24:45.208 DEBUG:boto:Data: 2023-07-29T00:24:45.209 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.210 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:45.210 DEBUG:boto:Port: 8001 2023-07-29T00:24:45.211 DEBUG:boto:Params: {} 2023-07-29T00:24:45.212 DEBUG:boto:Token: None 2023-07-29T00:24:45.213 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:24:45.213 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vxVmE84YuiNS0YzbAGXgRiQdo9o= 2023-07-29T00:24:45.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vxVmE84YuiNS0YzbAGXgRiQdo9o=', 'Content-Length': '0'} 2023-07-29T00:24:45.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b779d8fa636a6a3-0064c45c4d-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.216 DEBUG:boto:path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:24:45.217 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:24:45.217 DEBUG:boto:path=/hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:24:45.218 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:24:45.218 DEBUG:boto:Method: GET 2023-07-29T00:24:45.218 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:24:45.219 DEBUG:boto:Data: 2023-07-29T00:24:45.219 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.220 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:45.220 DEBUG:boto:Port: 8000 2023-07-29T00:24:45.221 DEBUG:boto:Params: {} 2023-07-29T00:24:45.221 DEBUG:boto:Token: None 2023-07-29T00:24:45.221 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:24:45.222 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vxVmE84YuiNS0YzbAGXgRiQdo9o= 2023-07-29T00:24:45.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vxVmE84YuiNS0YzbAGXgRiQdo9o=', 'Content-Length': '0'} 2023-07-29T00:24:45.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7b8f71a41ecc6cd-0064c45c4d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.225 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:45.225 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:45.226 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000000 2023-07-29T00:24:45.226 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000000 2023-07-29T00:24:45.226 DEBUG:boto:Method: GET 2023-07-29T00:24:45.227 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000000 2023-07-29T00:24:45.227 DEBUG:boto:Data: 2023-07-29T00:24:45.227 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.228 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:45.228 DEBUG:boto:Port: 8001 2023-07-29T00:24:45.229 DEBUG:boto:Params: {} 2023-07-29T00:24:45.229 DEBUG:boto:Token: None 2023-07-29T00:24:45.229 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/ 2023-07-29T00:24:45.230 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fRgrRTvpOgxlOYybJvqGctoAw+Q= 2023-07-29T00:24:45.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fRgrRTvpOgxlOYybJvqGctoAw+Q=', 'Content-Length': '0'} 2023-07-29T00:24:45.233 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ff6fbda40465128-0064c45c4d-1b40-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.233 DEBUG:boto:b'hiiqos-58obj000000000000000000001000falseobj000000000000000000002023-07-29T00:24:24.651Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:45.234 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:45.235 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:45.235 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000000 2023-07-29T00:24:45.235 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000000 2023-07-29T00:24:45.236 DEBUG:boto:Method: GET 2023-07-29T00:24:45.236 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000000 2023-07-29T00:24:45.237 DEBUG:boto:Data: 2023-07-29T00:24:45.237 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.238 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:45.238 DEBUG:boto:Port: 8000 2023-07-29T00:24:45.239 DEBUG:boto:Params: {} 2023-07-29T00:24:45.239 DEBUG:boto:Token: None 2023-07-29T00:24:45.240 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/ 2023-07-29T00:24:45.241 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fRgrRTvpOgxlOYybJvqGctoAw+Q= 2023-07-29T00:24:45.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fRgrRTvpOgxlOYybJvqGctoAw+Q=', 'Content-Length': '0'} 2023-07-29T00:24:45.244 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb953c4f37033d25-0064c45c4d-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.245 DEBUG:boto:b'hiiqos-58obj000000000000000000001000falseobj000000000000000000002023-07-29T00:24:24.651Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:45.245 DEBUG:boto:path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:24:45.246 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:24:45.246 DEBUG:boto:Method: HEAD 2023-07-29T00:24:45.247 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000000 2023-07-29T00:24:45.247 DEBUG:boto:Data: 2023-07-29T00:24:45.248 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.249 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:45.249 DEBUG:boto:Port: 8001 2023-07-29T00:24:45.250 DEBUG:boto:Params: {} 2023-07-29T00:24:45.251 DEBUG:boto:Token: None 2023-07-29T00:24:45.251 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/obj00000000000000000000 2023-07-29T00:24:45.252 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cr1MCJG9Zt5edXHdOeEmdf/jP90= 2023-07-29T00:24:45.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cr1MCJG9Zt5edXHdOeEmdf/jP90=', 'Content-Length': '0'} 2023-07-29T00:24:45.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075549ccc36b3b935-0064c45c4d-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.254 DEBUG:boto:path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:24:45.255 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:24:45.255 DEBUG:boto:Method: HEAD 2023-07-29T00:24:45.255 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000000 2023-07-29T00:24:45.256 DEBUG:boto:Data: 2023-07-29T00:24:45.256 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.257 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:45.257 DEBUG:boto:Port: 8000 2023-07-29T00:24:45.258 DEBUG:boto:Params: {} 2023-07-29T00:24:45.258 DEBUG:boto:Token: None 2023-07-29T00:24:45.259 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/obj00000000000000000000 2023-07-29T00:24:45.259 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cr1MCJG9Zt5edXHdOeEmdf/jP90= 2023-07-29T00:24:45.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cr1MCJG9Zt5edXHdOeEmdf/jP90=', 'Content-Length': '0'} 2023-07-29T00:24:45.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3c12e5ed095d0a0-0064c45c4d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-07-29T00:24:45.262 DEBUG:boto:path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:24:45.263 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:24:45.263 DEBUG:boto:Method: GET 2023-07-29T00:24:45.263 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000000 2023-07-29T00:24:45.264 DEBUG:boto:Data: 2023-07-29T00:24:45.264 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.265 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:45.265 DEBUG:boto:Port: 8001 2023-07-29T00:24:45.265 DEBUG:boto:Params: {} 2023-07-29T00:24:45.266 DEBUG:boto:Token: None 2023-07-29T00:24:45.266 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/obj00000000000000000000 2023-07-29T00:24:45.267 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Iw5NXxR7ksV2yZko2tkVjxcw0DE= 2023-07-29T00:24:45.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Iw5NXxR7ksV2yZko2tkVjxcw0DE=', 'Content-Length': '0'} 2023-07-29T00:24:45.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a62496a5bfacf3f2-0064c45c4d-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.313 DEBUG:boto:path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:24:45.313 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:24:45.314 DEBUG:boto:Method: GET 2023-07-29T00:24:45.314 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000000 2023-07-29T00:24:45.315 DEBUG:boto:Data: 2023-07-29T00:24:45.315 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.316 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:45.316 DEBUG:boto:Port: 8000 2023-07-29T00:24:45.317 DEBUG:boto:Params: {} 2023-07-29T00:24:45.317 DEBUG:boto:Token: None 2023-07-29T00:24:45.318 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/obj00000000000000000000 2023-07-29T00:24:45.319 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Iw5NXxR7ksV2yZko2tkVjxcw0DE= 2023-07-29T00:24:45.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Iw5NXxR7ksV2yZko2tkVjxcw0DE=', 'Content-Length': '0'} 2023-07-29T00:24:45.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097f336b3614e11ca-0064c45c4d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.324 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:45.324 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:45.324 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000000 2023-07-29T00:24:45.325 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000000 2023-07-29T00:24:45.325 DEBUG:boto:Method: GET 2023-07-29T00:24:45.326 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000000 2023-07-29T00:24:45.326 DEBUG:boto:Data: 2023-07-29T00:24:45.326 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.327 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:45.327 DEBUG:boto:Port: 8001 2023-07-29T00:24:45.327 DEBUG:boto:Params: {} 2023-07-29T00:24:45.328 DEBUG:boto:Token: None 2023-07-29T00:24:45.328 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/ 2023-07-29T00:24:45.329 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fRgrRTvpOgxlOYybJvqGctoAw+Q= 2023-07-29T00:24:45.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fRgrRTvpOgxlOYybJvqGctoAw+Q=', 'Content-Length': '0'} 2023-07-29T00:24:45.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e92705245d016f3-0064c45c4d-1b40-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.333 DEBUG:boto:b'hiiqos-58obj000000000000000000001000falseobj000000000000000000002023-07-29T00:24:24.651Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:45.333 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:45.334 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:45.335 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000000 2023-07-29T00:24:45.335 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000000 2023-07-29T00:24:45.336 DEBUG:boto:Method: GET 2023-07-29T00:24:45.336 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000000 2023-07-29T00:24:45.337 DEBUG:boto:Data: 2023-07-29T00:24:45.338 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.339 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:45.340 DEBUG:boto:Port: 8000 2023-07-29T00:24:45.341 DEBUG:boto:Params: {} 2023-07-29T00:24:45.342 DEBUG:boto:Token: None 2023-07-29T00:24:45.342 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/ 2023-07-29T00:24:45.343 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fRgrRTvpOgxlOYybJvqGctoAw+Q= 2023-07-29T00:24:45.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fRgrRTvpOgxlOYybJvqGctoAw+Q=', 'Content-Length': '0'} 2023-07-29T00:24:45.345 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de2bf549373cf820-0064c45c4d-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.346 DEBUG:boto:b'hiiqos-58obj000000000000000000001000falseobj000000000000000000002023-07-29T00:24:24.651Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:45.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-07-29T00:24:45.347 DEBUG:boto:path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:24:45.347 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:24:45.347 DEBUG:boto:path=/hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:24:45.348 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:24:45.348 DEBUG:boto:Method: GET 2023-07-29T00:24:45.349 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:24:45.349 DEBUG:boto:Data: 2023-07-29T00:24:45.349 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.350 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:45.350 DEBUG:boto:Port: 8001 2023-07-29T00:24:45.351 DEBUG:boto:Params: {} 2023-07-29T00:24:45.351 DEBUG:boto:Token: None 2023-07-29T00:24:45.352 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:24:45.353 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:G3lX4zevN+9GhtioLU5Ud/3YJgw= 2023-07-29T00:24:45.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:G3lX4zevN+9GhtioLU5Ud/3YJgw=', 'Content-Length': '0'} 2023-07-29T00:24:45.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb50171c00fdc599-0064c45c4d-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.358 DEBUG:boto:path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:24:45.359 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:24:45.360 DEBUG:boto:path=/hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:24:45.360 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:24:45.361 DEBUG:boto:Method: GET 2023-07-29T00:24:45.362 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:24:45.363 DEBUG:boto:Data: 2023-07-29T00:24:45.363 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.363 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:45.364 DEBUG:boto:Port: 8000 2023-07-29T00:24:45.364 DEBUG:boto:Params: {} 2023-07-29T00:24:45.365 DEBUG:boto:Token: None 2023-07-29T00:24:45.365 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:24:45.366 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:G3lX4zevN+9GhtioLU5Ud/3YJgw= 2023-07-29T00:24:45.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:G3lX4zevN+9GhtioLU5Ud/3YJgw=', 'Content-Length': '0'} 2023-07-29T00:24:45.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c3d1e8ff04fead5-0064c45c4d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.412 DEBUG:boto:path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:24:45.413 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:24:45.413 DEBUG:boto:path=/hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:24:45.413 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:24:45.414 DEBUG:boto:Method: HEAD 2023-07-29T00:24:45.414 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:24:45.414 DEBUG:boto:Data: 2023-07-29T00:24:45.415 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.415 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:45.415 DEBUG:boto:Port: 8001 2023-07-29T00:24:45.416 DEBUG:boto:Params: {} 2023-07-29T00:24:45.416 DEBUG:boto:Token: None 2023-07-29T00:24:45.417 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:24:45.417 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1XnE7MBYG67JCEGFF1G2laKqboA= 2023-07-29T00:24:45.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1XnE7MBYG67JCEGFF1G2laKqboA=', 'Content-Length': '0'} 2023-07-29T00:24:45.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d361ed7636d5b111-0064c45c4d-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.420 DEBUG:boto:path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:24:45.421 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:24:45.421 DEBUG:boto:path=/hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:24:45.421 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:24:45.422 DEBUG:boto:Method: HEAD 2023-07-29T00:24:45.422 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:24:45.422 DEBUG:boto:Data: 2023-07-29T00:24:45.423 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.423 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:45.423 DEBUG:boto:Port: 8000 2023-07-29T00:24:45.424 DEBUG:boto:Params: {} 2023-07-29T00:24:45.424 DEBUG:boto:Token: None 2023-07-29T00:24:45.424 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:24:45.425 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1XnE7MBYG67JCEGFF1G2laKqboA= 2023-07-29T00:24:45.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1XnE7MBYG67JCEGFF1G2laKqboA=', 'Content-Length': '0'} 2023-07-29T00:24:45.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011f30d4f9011db8a-0064c45c4d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-07-29T00:24:45.428 DEBUG:boto:path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:24:45.428 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:24:45.429 DEBUG:boto:path=/hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:24:45.429 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:24:45.429 DEBUG:boto:Method: GET 2023-07-29T00:24:45.430 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:24:45.430 DEBUG:boto:Data: 2023-07-29T00:24:45.430 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.431 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:45.431 DEBUG:boto:Port: 8001 2023-07-29T00:24:45.431 DEBUG:boto:Params: {} 2023-07-29T00:24:45.432 DEBUG:boto:Token: None 2023-07-29T00:24:45.432 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:24:45.433 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:G3lX4zevN+9GhtioLU5Ud/3YJgw= 2023-07-29T00:24:45.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:G3lX4zevN+9GhtioLU5Ud/3YJgw=', 'Content-Length': '0'} 2023-07-29T00:24:45.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a46f2de66ca6c6b-0064c45c4d-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.436 DEBUG:boto:path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:24:45.436 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:24:45.437 DEBUG:boto:path=/hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:24:45.437 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:24:45.437 DEBUG:boto:Method: GET 2023-07-29T00:24:45.438 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:24:45.438 DEBUG:boto:Data: 2023-07-29T00:24:45.438 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.439 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:45.439 DEBUG:boto:Port: 8000 2023-07-29T00:24:45.440 DEBUG:boto:Params: {} 2023-07-29T00:24:45.440 DEBUG:boto:Token: None 2023-07-29T00:24:45.440 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:24:45.441 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:G3lX4zevN+9GhtioLU5Ud/3YJgw= 2023-07-29T00:24:45.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:G3lX4zevN+9GhtioLU5Ud/3YJgw=', 'Content-Length': '0'} 2023-07-29T00:24:45.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b60fa5e8226e74ff-0064c45c4d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.444 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:45.444 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:45.445 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000001 2023-07-29T00:24:45.445 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000001 2023-07-29T00:24:45.445 DEBUG:boto:Method: GET 2023-07-29T00:24:45.446 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000001 2023-07-29T00:24:45.446 DEBUG:boto:Data: 2023-07-29T00:24:45.447 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.447 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:45.447 DEBUG:boto:Port: 8001 2023-07-29T00:24:45.448 DEBUG:boto:Params: {} 2023-07-29T00:24:45.448 DEBUG:boto:Token: None 2023-07-29T00:24:45.449 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/ 2023-07-29T00:24:45.449 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fRgrRTvpOgxlOYybJvqGctoAw+Q= 2023-07-29T00:24:45.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fRgrRTvpOgxlOYybJvqGctoAw+Q=', 'Content-Length': '0'} 2023-07-29T00:24:45.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004db3c516f3d313f2-0064c45c4d-1b40-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.452 DEBUG:boto:b'hiiqos-58obj000000000000000000011000falseobj000000000000000000012023-07-29T00:24:24.608Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:45.453 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:45.453 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:45.453 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000001 2023-07-29T00:24:45.454 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000001 2023-07-29T00:24:45.454 DEBUG:boto:Method: GET 2023-07-29T00:24:45.455 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000001 2023-07-29T00:24:45.455 DEBUG:boto:Data: 2023-07-29T00:24:45.455 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.456 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:45.456 DEBUG:boto:Port: 8000 2023-07-29T00:24:45.456 DEBUG:boto:Params: {} 2023-07-29T00:24:45.457 DEBUG:boto:Token: None 2023-07-29T00:24:45.457 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/ 2023-07-29T00:24:45.458 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fRgrRTvpOgxlOYybJvqGctoAw+Q= 2023-07-29T00:24:45.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fRgrRTvpOgxlOYybJvqGctoAw+Q=', 'Content-Length': '0'} 2023-07-29T00:24:45.460 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006dc729e026017398-0064c45c4d-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.461 DEBUG:boto:b'hiiqos-58obj000000000000000000011000falseobj000000000000000000012023-07-29T00:24:24.608Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:45.461 DEBUG:boto:path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:24:45.462 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:24:45.462 DEBUG:boto:Method: HEAD 2023-07-29T00:24:45.463 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000001 2023-07-29T00:24:45.463 DEBUG:boto:Data: 2023-07-29T00:24:45.463 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.464 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:45.464 DEBUG:boto:Port: 8001 2023-07-29T00:24:45.464 DEBUG:boto:Params: {} 2023-07-29T00:24:45.465 DEBUG:boto:Token: None 2023-07-29T00:24:45.465 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/obj00000000000000000001 2023-07-29T00:24:45.465 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jCYkv4aBL3FDxLUSphoGU2bYl0U= 2023-07-29T00:24:45.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jCYkv4aBL3FDxLUSphoGU2bYl0U=', 'Content-Length': '0'} 2023-07-29T00:24:45.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a41c5a95f2a94bb-0064c45c4d-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.468 DEBUG:boto:path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:24:45.468 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:24:45.469 DEBUG:boto:Method: HEAD 2023-07-29T00:24:45.469 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000001 2023-07-29T00:24:45.470 DEBUG:boto:Data: 2023-07-29T00:24:45.470 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.470 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:45.471 DEBUG:boto:Port: 8000 2023-07-29T00:24:45.471 DEBUG:boto:Params: {} 2023-07-29T00:24:45.471 DEBUG:boto:Token: None 2023-07-29T00:24:45.472 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/obj00000000000000000001 2023-07-29T00:24:45.472 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jCYkv4aBL3FDxLUSphoGU2bYl0U= 2023-07-29T00:24:45.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jCYkv4aBL3FDxLUSphoGU2bYl0U=', 'Content-Length': '0'} 2023-07-29T00:24:45.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e082bdb64ce2b108-0064c45c4d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-07-29T00:24:45.475 DEBUG:boto:path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:24:45.475 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:24:45.476 DEBUG:boto:Method: GET 2023-07-29T00:24:45.476 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000001 2023-07-29T00:24:45.476 DEBUG:boto:Data: 2023-07-29T00:24:45.477 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.477 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:45.478 DEBUG:boto:Port: 8001 2023-07-29T00:24:45.478 DEBUG:boto:Params: {} 2023-07-29T00:24:45.478 DEBUG:boto:Token: None 2023-07-29T00:24:45.479 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/obj00000000000000000001 2023-07-29T00:24:45.479 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:v3UZU9hZ3ten4VBvPuAGaEV+oDs= 2023-07-29T00:24:45.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:v3UZU9hZ3ten4VBvPuAGaEV+oDs=', 'Content-Length': '0'} 2023-07-29T00:24:45.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000034c94612410cc12-0064c45c4d-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.482 DEBUG:boto:path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:24:45.482 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:24:45.483 DEBUG:boto:Method: GET 2023-07-29T00:24:45.483 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000001 2023-07-29T00:24:45.483 DEBUG:boto:Data: 2023-07-29T00:24:45.484 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.484 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:45.485 DEBUG:boto:Port: 8000 2023-07-29T00:24:45.485 DEBUG:boto:Params: {} 2023-07-29T00:24:45.485 DEBUG:boto:Token: None 2023-07-29T00:24:45.486 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/obj00000000000000000001 2023-07-29T00:24:45.486 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:v3UZU9hZ3ten4VBvPuAGaEV+oDs= 2023-07-29T00:24:45.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:v3UZU9hZ3ten4VBvPuAGaEV+oDs=', 'Content-Length': '0'} 2023-07-29T00:24:45.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab7b7a2b41380e83-0064c45c4d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.489 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:45.490 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:45.490 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000001 2023-07-29T00:24:45.490 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000001 2023-07-29T00:24:45.491 DEBUG:boto:Method: GET 2023-07-29T00:24:45.491 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000001 2023-07-29T00:24:45.491 DEBUG:boto:Data: 2023-07-29T00:24:45.492 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.492 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:45.492 DEBUG:boto:Port: 8001 2023-07-29T00:24:45.493 DEBUG:boto:Params: {} 2023-07-29T00:24:45.493 DEBUG:boto:Token: None 2023-07-29T00:24:45.494 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/ 2023-07-29T00:24:45.494 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fRgrRTvpOgxlOYybJvqGctoAw+Q= 2023-07-29T00:24:45.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fRgrRTvpOgxlOYybJvqGctoAw+Q=', 'Content-Length': '0'} 2023-07-29T00:24:45.497 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000703e6cf2e3cd929e-0064c45c4d-1b40-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.497 DEBUG:boto:b'hiiqos-58obj000000000000000000011000falseobj000000000000000000012023-07-29T00:24:24.608Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:45.498 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:45.499 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:45.499 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000001 2023-07-29T00:24:45.499 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000001 2023-07-29T00:24:45.500 DEBUG:boto:Method: GET 2023-07-29T00:24:45.500 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000001 2023-07-29T00:24:45.500 DEBUG:boto:Data: 2023-07-29T00:24:45.501 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.501 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:45.501 DEBUG:boto:Port: 8000 2023-07-29T00:24:45.502 DEBUG:boto:Params: {} 2023-07-29T00:24:45.502 DEBUG:boto:Token: None 2023-07-29T00:24:45.503 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/ 2023-07-29T00:24:45.503 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fRgrRTvpOgxlOYybJvqGctoAw+Q= 2023-07-29T00:24:45.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fRgrRTvpOgxlOYybJvqGctoAw+Q=', 'Content-Length': '0'} 2023-07-29T00:24:45.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008624cd189b9927b8-0064c45c4d-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.548 DEBUG:boto:b'hiiqos-58obj000000000000000000011000falseobj000000000000000000012023-07-29T00:24:24.608Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:45.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-07-29T00:24:45.550 DEBUG:boto:path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:24:45.550 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:24:45.550 DEBUG:boto:path=/hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:24:45.551 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:24:45.551 DEBUG:boto:Method: GET 2023-07-29T00:24:45.551 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:24:45.552 DEBUG:boto:Data: 2023-07-29T00:24:45.552 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.552 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:45.553 DEBUG:boto:Port: 8001 2023-07-29T00:24:45.553 DEBUG:boto:Params: {} 2023-07-29T00:24:45.554 DEBUG:boto:Token: None 2023-07-29T00:24:45.554 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:24:45.555 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/OkZXNEsSywmxiFZO/wAZWzfF7U= 2023-07-29T00:24:45.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/OkZXNEsSywmxiFZO/wAZWzfF7U=', 'Content-Length': '0'} 2023-07-29T00:24:45.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7815c39bd9a26c7-0064c45c4d-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.557 DEBUG:boto:path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:24:45.558 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:24:45.558 DEBUG:boto:path=/hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:24:45.558 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:24:45.559 DEBUG:boto:Method: GET 2023-07-29T00:24:45.559 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:24:45.559 DEBUG:boto:Data: 2023-07-29T00:24:45.560 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.560 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:45.561 DEBUG:boto:Port: 8000 2023-07-29T00:24:45.561 DEBUG:boto:Params: {} 2023-07-29T00:24:45.561 DEBUG:boto:Token: None 2023-07-29T00:24:45.562 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:24:45.562 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/OkZXNEsSywmxiFZO/wAZWzfF7U= 2023-07-29T00:24:45.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/OkZXNEsSywmxiFZO/wAZWzfF7U=', 'Content-Length': '0'} 2023-07-29T00:24:45.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8e9a29101bd025e-0064c45c4d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.566 DEBUG:boto:path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:24:45.566 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:24:45.566 DEBUG:boto:path=/hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:24:45.567 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:24:45.567 DEBUG:boto:Method: HEAD 2023-07-29T00:24:45.567 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:24:45.568 DEBUG:boto:Data: 2023-07-29T00:24:45.568 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.568 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:45.569 DEBUG:boto:Port: 8001 2023-07-29T00:24:45.569 DEBUG:boto:Params: {} 2023-07-29T00:24:45.569 DEBUG:boto:Token: None 2023-07-29T00:24:45.570 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:24:45.570 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JDp5/ZJgUB3GYfaC/zklBWyxd04= 2023-07-29T00:24:45.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JDp5/ZJgUB3GYfaC/zklBWyxd04=', 'Content-Length': '0'} 2023-07-29T00:24:45.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0e06cc9f88d9302-0064c45c4d-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.573 DEBUG:boto:path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:24:45.573 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:24:45.574 DEBUG:boto:path=/hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:24:45.574 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:24:45.574 DEBUG:boto:Method: HEAD 2023-07-29T00:24:45.575 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:24:45.575 DEBUG:boto:Data: 2023-07-29T00:24:45.575 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.576 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:45.576 DEBUG:boto:Port: 8000 2023-07-29T00:24:45.576 DEBUG:boto:Params: {} 2023-07-29T00:24:45.577 DEBUG:boto:Token: None 2023-07-29T00:24:45.577 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:24:45.577 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JDp5/ZJgUB3GYfaC/zklBWyxd04= 2023-07-29T00:24:45.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JDp5/ZJgUB3GYfaC/zklBWyxd04=', 'Content-Length': '0'} 2023-07-29T00:24:45.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000125f776a29b41722-0064c45c4d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-07-29T00:24:45.580 DEBUG:boto:path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:24:45.581 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:24:45.581 DEBUG:boto:path=/hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:24:45.582 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:24:45.582 DEBUG:boto:Method: GET 2023-07-29T00:24:45.582 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:24:45.583 DEBUG:boto:Data: 2023-07-29T00:24:45.583 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.583 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:45.584 DEBUG:boto:Port: 8001 2023-07-29T00:24:45.584 DEBUG:boto:Params: {} 2023-07-29T00:24:45.585 DEBUG:boto:Token: None 2023-07-29T00:24:45.585 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:24:45.585 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/OkZXNEsSywmxiFZO/wAZWzfF7U= 2023-07-29T00:24:45.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/OkZXNEsSywmxiFZO/wAZWzfF7U=', 'Content-Length': '0'} 2023-07-29T00:24:45.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000571f93d257c6d24f-0064c45c4d-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.628 DEBUG:boto:path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:24:45.629 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:24:45.629 DEBUG:boto:path=/hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:24:45.630 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:24:45.630 DEBUG:boto:Method: GET 2023-07-29T00:24:45.631 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:24:45.631 DEBUG:boto:Data: 2023-07-29T00:24:45.631 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.632 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:45.632 DEBUG:boto:Port: 8000 2023-07-29T00:24:45.632 DEBUG:boto:Params: {} 2023-07-29T00:24:45.633 DEBUG:boto:Token: None 2023-07-29T00:24:45.633 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:24:45.634 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/OkZXNEsSywmxiFZO/wAZWzfF7U= 2023-07-29T00:24:45.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/OkZXNEsSywmxiFZO/wAZWzfF7U=', 'Content-Length': '0'} 2023-07-29T00:24:45.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059f4ee5956c6153d-0064c45c4d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.637 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:45.637 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:45.637 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000002 2023-07-29T00:24:45.638 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000002 2023-07-29T00:24:45.638 DEBUG:boto:Method: GET 2023-07-29T00:24:45.638 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000002 2023-07-29T00:24:45.639 DEBUG:boto:Data: 2023-07-29T00:24:45.639 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.640 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:45.640 DEBUG:boto:Port: 8001 2023-07-29T00:24:45.640 DEBUG:boto:Params: {} 2023-07-29T00:24:45.641 DEBUG:boto:Token: None 2023-07-29T00:24:45.641 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/ 2023-07-29T00:24:45.642 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fRgrRTvpOgxlOYybJvqGctoAw+Q= 2023-07-29T00:24:45.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fRgrRTvpOgxlOYybJvqGctoAw+Q=', 'Content-Length': '0'} 2023-07-29T00:24:45.644 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006539c005cff60229-0064c45c4d-1b40-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.645 DEBUG:boto:b'hiiqos-58obj000000000000000000021000falseobj000000000000000000022023-07-29T00:24:24.633Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:45.645 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:45.646 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:45.646 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000002 2023-07-29T00:24:45.647 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000002 2023-07-29T00:24:45.647 DEBUG:boto:Method: GET 2023-07-29T00:24:45.648 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000002 2023-07-29T00:24:45.648 DEBUG:boto:Data: 2023-07-29T00:24:45.649 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.649 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:45.649 DEBUG:boto:Port: 8000 2023-07-29T00:24:45.650 DEBUG:boto:Params: {} 2023-07-29T00:24:45.650 DEBUG:boto:Token: None 2023-07-29T00:24:45.651 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/ 2023-07-29T00:24:45.651 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fRgrRTvpOgxlOYybJvqGctoAw+Q= 2023-07-29T00:24:45.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fRgrRTvpOgxlOYybJvqGctoAw+Q=', 'Content-Length': '0'} 2023-07-29T00:24:45.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000753626699750d777-0064c45c4d-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.654 DEBUG:boto:b'hiiqos-58obj000000000000000000021000falseobj000000000000000000022023-07-29T00:24:24.633Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:45.655 DEBUG:boto:path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:24:45.656 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:24:45.656 DEBUG:boto:Method: HEAD 2023-07-29T00:24:45.657 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000002 2023-07-29T00:24:45.657 DEBUG:boto:Data: 2023-07-29T00:24:45.657 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.658 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:45.658 DEBUG:boto:Port: 8001 2023-07-29T00:24:45.658 DEBUG:boto:Params: {} 2023-07-29T00:24:45.659 DEBUG:boto:Token: None 2023-07-29T00:24:45.659 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/obj00000000000000000002 2023-07-29T00:24:45.660 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GwXTVPc7JHdaPxnaMeaEX1PWLNE= 2023-07-29T00:24:45.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GwXTVPc7JHdaPxnaMeaEX1PWLNE=', 'Content-Length': '0'} 2023-07-29T00:24:45.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050351c4e5cd7b30f-0064c45c4d-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.662 DEBUG:boto:path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:24:45.663 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:24:45.663 DEBUG:boto:Method: HEAD 2023-07-29T00:24:45.663 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000002 2023-07-29T00:24:45.664 DEBUG:boto:Data: 2023-07-29T00:24:45.664 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.664 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:45.665 DEBUG:boto:Port: 8000 2023-07-29T00:24:45.665 DEBUG:boto:Params: {} 2023-07-29T00:24:45.665 DEBUG:boto:Token: None 2023-07-29T00:24:45.666 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/obj00000000000000000002 2023-07-29T00:24:45.666 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GwXTVPc7JHdaPxnaMeaEX1PWLNE= 2023-07-29T00:24:45.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GwXTVPc7JHdaPxnaMeaEX1PWLNE=', 'Content-Length': '0'} 2023-07-29T00:24:45.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024581976967fa854-0064c45c4d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-07-29T00:24:45.670 DEBUG:boto:path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:24:45.670 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:24:45.670 DEBUG:boto:Method: GET 2023-07-29T00:24:45.671 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000002 2023-07-29T00:24:45.671 DEBUG:boto:Data: 2023-07-29T00:24:45.672 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.673 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:45.673 DEBUG:boto:Port: 8001 2023-07-29T00:24:45.673 DEBUG:boto:Params: {} 2023-07-29T00:24:45.674 DEBUG:boto:Token: None 2023-07-29T00:24:45.674 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/obj00000000000000000002 2023-07-29T00:24:45.675 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QhzaYlEoEricKKt/XzYKDBRcxdU= 2023-07-29T00:24:45.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QhzaYlEoEricKKt/XzYKDBRcxdU=', 'Content-Length': '0'} 2023-07-29T00:24:45.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000525870438f045912-0064c45c4d-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.678 DEBUG:boto:path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:24:45.678 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:24:45.678 DEBUG:boto:Method: GET 2023-07-29T00:24:45.679 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000002 2023-07-29T00:24:45.679 DEBUG:boto:Data: 2023-07-29T00:24:45.679 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.680 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:45.680 DEBUG:boto:Port: 8000 2023-07-29T00:24:45.681 DEBUG:boto:Params: {} 2023-07-29T00:24:45.681 DEBUG:boto:Token: None 2023-07-29T00:24:45.681 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/obj00000000000000000002 2023-07-29T00:24:45.682 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QhzaYlEoEricKKt/XzYKDBRcxdU= 2023-07-29T00:24:45.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QhzaYlEoEricKKt/XzYKDBRcxdU=', 'Content-Length': '0'} 2023-07-29T00:24:45.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dfbd03d865b1146-0064c45c4d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.728 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:45.729 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:45.730 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000002 2023-07-29T00:24:45.730 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000002 2023-07-29T00:24:45.731 DEBUG:boto:Method: GET 2023-07-29T00:24:45.731 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000002 2023-07-29T00:24:45.732 DEBUG:boto:Data: 2023-07-29T00:24:45.732 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.733 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:45.734 DEBUG:boto:Port: 8001 2023-07-29T00:24:45.734 DEBUG:boto:Params: {} 2023-07-29T00:24:45.735 DEBUG:boto:Token: None 2023-07-29T00:24:45.736 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/ 2023-07-29T00:24:45.737 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fRgrRTvpOgxlOYybJvqGctoAw+Q= 2023-07-29T00:24:45.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fRgrRTvpOgxlOYybJvqGctoAw+Q=', 'Content-Length': '0'} 2023-07-29T00:24:45.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099fcbbf0d73d1028-0064c45c4d-1b40-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.743 DEBUG:boto:b'hiiqos-58obj000000000000000000021000falseobj000000000000000000022023-07-29T00:24:24.633Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:45.744 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:45.745 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:45.746 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000002 2023-07-29T00:24:45.747 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000002 2023-07-29T00:24:45.747 DEBUG:boto:Method: GET 2023-07-29T00:24:45.748 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000002 2023-07-29T00:24:45.749 DEBUG:boto:Data: 2023-07-29T00:24:45.750 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.750 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:45.750 DEBUG:boto:Port: 8000 2023-07-29T00:24:45.751 DEBUG:boto:Params: {} 2023-07-29T00:24:45.751 DEBUG:boto:Token: None 2023-07-29T00:24:45.752 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/ 2023-07-29T00:24:45.752 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fRgrRTvpOgxlOYybJvqGctoAw+Q= 2023-07-29T00:24:45.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fRgrRTvpOgxlOYybJvqGctoAw+Q=', 'Content-Length': '0'} 2023-07-29T00:24:45.755 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008958a3478f2f2197-0064c45c4d-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.755 DEBUG:boto:b'hiiqos-58obj000000000000000000021000falseobj000000000000000000022023-07-29T00:24:24.633Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:45.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-07-29T00:24:45.757 DEBUG:boto:path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:24:45.757 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:24:45.758 DEBUG:boto:path=/hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:24:45.758 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:24:45.758 DEBUG:boto:Method: GET 2023-07-29T00:24:45.759 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:24:45.759 DEBUG:boto:Data: 2023-07-29T00:24:45.759 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.760 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:45.760 DEBUG:boto:Port: 8001 2023-07-29T00:24:45.760 DEBUG:boto:Params: {} 2023-07-29T00:24:45.761 DEBUG:boto:Token: None 2023-07-29T00:24:45.761 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:24:45.762 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wR2Zmh17rjmy66rkhLmjBP38D9s= 2023-07-29T00:24:45.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wR2Zmh17rjmy66rkhLmjBP38D9s=', 'Content-Length': '0'} 2023-07-29T00:24:45.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e3ae300bf30367b-0064c45c4d-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.804 DEBUG:boto:path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:24:45.805 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:24:45.805 DEBUG:boto:path=/hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:24:45.806 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:24:45.806 DEBUG:boto:Method: GET 2023-07-29T00:24:45.806 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:24:45.807 DEBUG:boto:Data: 2023-07-29T00:24:45.808 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.808 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:45.809 DEBUG:boto:Port: 8000 2023-07-29T00:24:45.809 DEBUG:boto:Params: {} 2023-07-29T00:24:45.809 DEBUG:boto:Token: None 2023-07-29T00:24:45.810 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:24:45.810 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wR2Zmh17rjmy66rkhLmjBP38D9s= 2023-07-29T00:24:45.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wR2Zmh17rjmy66rkhLmjBP38D9s=', 'Content-Length': '0'} 2023-07-29T00:24:45.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f1b900dd8af9213-0064c45c4d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.814 DEBUG:boto:path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:24:45.814 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:24:45.815 DEBUG:boto:path=/hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:24:45.815 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:24:45.815 DEBUG:boto:Method: HEAD 2023-07-29T00:24:45.816 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:24:45.816 DEBUG:boto:Data: 2023-07-29T00:24:45.816 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.817 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:45.817 DEBUG:boto:Port: 8001 2023-07-29T00:24:45.818 DEBUG:boto:Params: {} 2023-07-29T00:24:45.818 DEBUG:boto:Token: None 2023-07-29T00:24:45.818 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:24:45.819 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OTQNzphTKNpfbLN3OHAAwrTFiqQ= 2023-07-29T00:24:45.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OTQNzphTKNpfbLN3OHAAwrTFiqQ=', 'Content-Length': '0'} 2023-07-29T00:24:45.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e5aac11ec43b675-0064c45c4d-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.822 DEBUG:boto:path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:24:45.823 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:24:45.823 DEBUG:boto:path=/hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:24:45.823 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:24:45.824 DEBUG:boto:Method: HEAD 2023-07-29T00:24:45.824 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:24:45.825 DEBUG:boto:Data: 2023-07-29T00:24:45.825 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.825 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:45.826 DEBUG:boto:Port: 8000 2023-07-29T00:24:45.826 DEBUG:boto:Params: {} 2023-07-29T00:24:45.827 DEBUG:boto:Token: None 2023-07-29T00:24:45.828 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:24:45.828 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OTQNzphTKNpfbLN3OHAAwrTFiqQ= 2023-07-29T00:24:45.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OTQNzphTKNpfbLN3OHAAwrTFiqQ=', 'Content-Length': '0'} 2023-07-29T00:24:45.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe2fb2bd62447a24-0064c45c4d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-07-29T00:24:45.832 DEBUG:boto:path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:24:45.832 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:24:45.832 DEBUG:boto:path=/hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:24:45.833 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:24:45.833 DEBUG:boto:Method: GET 2023-07-29T00:24:45.834 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:24:45.834 DEBUG:boto:Data: 2023-07-29T00:24:45.835 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.835 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:45.835 DEBUG:boto:Port: 8001 2023-07-29T00:24:45.836 DEBUG:boto:Params: {} 2023-07-29T00:24:45.836 DEBUG:boto:Token: None 2023-07-29T00:24:45.837 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:24:45.837 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wR2Zmh17rjmy66rkhLmjBP38D9s= 2023-07-29T00:24:45.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wR2Zmh17rjmy66rkhLmjBP38D9s=', 'Content-Length': '0'} 2023-07-29T00:24:45.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c6bb3ca54250406-0064c45c4d-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.840 DEBUG:boto:path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:24:45.840 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:24:45.841 DEBUG:boto:path=/hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:24:45.841 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:24:45.841 DEBUG:boto:Method: GET 2023-07-29T00:24:45.842 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:24:45.842 DEBUG:boto:Data: 2023-07-29T00:24:45.842 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.843 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:45.843 DEBUG:boto:Port: 8000 2023-07-29T00:24:45.844 DEBUG:boto:Params: {} 2023-07-29T00:24:45.845 DEBUG:boto:Token: None 2023-07-29T00:24:45.845 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:24:45.845 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wR2Zmh17rjmy66rkhLmjBP38D9s= 2023-07-29T00:24:45.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wR2Zmh17rjmy66rkhLmjBP38D9s=', 'Content-Length': '0'} 2023-07-29T00:24:45.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a161c7d5afade5f-0064c45c4d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.848 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:45.849 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:45.849 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000003 2023-07-29T00:24:45.850 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000003 2023-07-29T00:24:45.850 DEBUG:boto:Method: GET 2023-07-29T00:24:45.850 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000003 2023-07-29T00:24:45.851 DEBUG:boto:Data: 2023-07-29T00:24:45.851 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.852 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:45.852 DEBUG:boto:Port: 8001 2023-07-29T00:24:45.852 DEBUG:boto:Params: {} 2023-07-29T00:24:45.853 DEBUG:boto:Token: None 2023-07-29T00:24:45.853 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/ 2023-07-29T00:24:45.854 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fRgrRTvpOgxlOYybJvqGctoAw+Q= 2023-07-29T00:24:45.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fRgrRTvpOgxlOYybJvqGctoAw+Q=', 'Content-Length': '0'} 2023-07-29T00:24:45.856 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e504fc1bc076af7f-0064c45c4d-1b40-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.857 DEBUG:boto:b'hiiqos-58obj000000000000000000031000falseobj000000000000000000032023-07-29T00:24:24.520Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:45.858 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:45.858 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:45.858 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000003 2023-07-29T00:24:45.859 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000003 2023-07-29T00:24:45.859 DEBUG:boto:Method: GET 2023-07-29T00:24:45.859 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000003 2023-07-29T00:24:45.860 DEBUG:boto:Data: 2023-07-29T00:24:45.860 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.861 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:45.861 DEBUG:boto:Port: 8000 2023-07-29T00:24:45.862 DEBUG:boto:Params: {} 2023-07-29T00:24:45.862 DEBUG:boto:Token: None 2023-07-29T00:24:45.863 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/ 2023-07-29T00:24:45.863 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fRgrRTvpOgxlOYybJvqGctoAw+Q= 2023-07-29T00:24:45.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fRgrRTvpOgxlOYybJvqGctoAw+Q=', 'Content-Length': '0'} 2023-07-29T00:24:45.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f51f0581bae3bdd-0064c45c4d-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.866 DEBUG:boto:b'hiiqos-58obj000000000000000000031000falseobj000000000000000000032023-07-29T00:24:24.520Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:45.867 DEBUG:boto:path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:24:45.867 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:24:45.868 DEBUG:boto:Method: HEAD 2023-07-29T00:24:45.868 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000003 2023-07-29T00:24:45.869 DEBUG:boto:Data: 2023-07-29T00:24:45.869 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.869 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:45.870 DEBUG:boto:Port: 8001 2023-07-29T00:24:45.870 DEBUG:boto:Params: {} 2023-07-29T00:24:45.870 DEBUG:boto:Token: None 2023-07-29T00:24:45.871 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/obj00000000000000000003 2023-07-29T00:24:45.871 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4gv6cLM+41IaJHT+BbesMxDPVuE= 2023-07-29T00:24:45.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4gv6cLM+41IaJHT+BbesMxDPVuE=', 'Content-Length': '0'} 2023-07-29T00:24:45.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086caf773534db60b-0064c45c4d-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.874 DEBUG:boto:path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:24:45.875 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:24:45.875 DEBUG:boto:Method: HEAD 2023-07-29T00:24:45.875 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000003 2023-07-29T00:24:45.876 DEBUG:boto:Data: 2023-07-29T00:24:45.876 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.876 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:45.877 DEBUG:boto:Port: 8000 2023-07-29T00:24:45.877 DEBUG:boto:Params: {} 2023-07-29T00:24:45.878 DEBUG:boto:Token: None 2023-07-29T00:24:45.878 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/obj00000000000000000003 2023-07-29T00:24:45.879 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4gv6cLM+41IaJHT+BbesMxDPVuE= 2023-07-29T00:24:45.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4gv6cLM+41IaJHT+BbesMxDPVuE=', 'Content-Length': '0'} 2023-07-29T00:24:45.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000127fce30bf7501ca-0064c45c4d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-07-29T00:24:45.882 DEBUG:boto:path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:24:45.882 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:24:45.883 DEBUG:boto:Method: GET 2023-07-29T00:24:45.883 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000003 2023-07-29T00:24:45.884 DEBUG:boto:Data: 2023-07-29T00:24:45.884 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.884 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:45.885 DEBUG:boto:Port: 8001 2023-07-29T00:24:45.885 DEBUG:boto:Params: {} 2023-07-29T00:24:45.886 DEBUG:boto:Token: None 2023-07-29T00:24:45.886 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/obj00000000000000000003 2023-07-29T00:24:45.887 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I0j4fqLewFk/0rKgQ7ARMj3Q6KI= 2023-07-29T00:24:45.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I0j4fqLewFk/0rKgQ7ARMj3Q6KI=', 'Content-Length': '0'} 2023-07-29T00:24:45.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b099b70cb51d1172-0064c45c4d-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.892 DEBUG:boto:path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:24:45.893 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:24:45.894 DEBUG:boto:Method: GET 2023-07-29T00:24:45.894 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000003 2023-07-29T00:24:45.895 DEBUG:boto:Data: 2023-07-29T00:24:45.895 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.896 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:45.897 DEBUG:boto:Port: 8000 2023-07-29T00:24:45.897 DEBUG:boto:Params: {} 2023-07-29T00:24:45.898 DEBUG:boto:Token: None 2023-07-29T00:24:45.898 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/obj00000000000000000003 2023-07-29T00:24:45.899 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I0j4fqLewFk/0rKgQ7ARMj3Q6KI= 2023-07-29T00:24:45.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I0j4fqLewFk/0rKgQ7ARMj3Q6KI=', 'Content-Length': '0'} 2023-07-29T00:24:45.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be36b8f9fc7c3690-0064c45c4d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.948 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:45.949 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:45.949 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000003 2023-07-29T00:24:45.950 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000003 2023-07-29T00:24:45.950 DEBUG:boto:Method: GET 2023-07-29T00:24:45.950 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000003 2023-07-29T00:24:45.951 DEBUG:boto:Data: 2023-07-29T00:24:45.951 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.951 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:45.952 DEBUG:boto:Port: 8001 2023-07-29T00:24:45.952 DEBUG:boto:Params: {} 2023-07-29T00:24:45.953 DEBUG:boto:Token: None 2023-07-29T00:24:45.953 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/ 2023-07-29T00:24:45.953 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fRgrRTvpOgxlOYybJvqGctoAw+Q= 2023-07-29T00:24:45.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fRgrRTvpOgxlOYybJvqGctoAw+Q=', 'Content-Length': '0'} 2023-07-29T00:24:45.956 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077e03fade76a3580-0064c45c4d-1b40-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.957 DEBUG:boto:b'hiiqos-58obj000000000000000000031000falseobj000000000000000000032023-07-29T00:24:24.520Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:45.959 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:45.959 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:45.960 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000003 2023-07-29T00:24:45.960 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000003 2023-07-29T00:24:45.960 DEBUG:boto:Method: GET 2023-07-29T00:24:45.961 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000003 2023-07-29T00:24:45.961 DEBUG:boto:Data: 2023-07-29T00:24:45.961 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.962 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:45.962 DEBUG:boto:Port: 8000 2023-07-29T00:24:45.962 DEBUG:boto:Params: {} 2023-07-29T00:24:45.963 DEBUG:boto:Token: None 2023-07-29T00:24:45.963 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/ 2023-07-29T00:24:45.964 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fRgrRTvpOgxlOYybJvqGctoAw+Q= 2023-07-29T00:24:45.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fRgrRTvpOgxlOYybJvqGctoAw+Q=', 'Content-Length': '0'} 2023-07-29T00:24:45.966 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1e664a1f21b128f-0064c45c4d-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:45.967 DEBUG:boto:b'hiiqos-58obj000000000000000000031000falseobj000000000000000000032023-07-29T00:24:24.520Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:45.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-07-29T00:24:45.968 DEBUG:boto:path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:24:45.968 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:24:45.969 DEBUG:boto:path=/hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:24:45.969 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:24:45.969 DEBUG:boto:Method: GET 2023-07-29T00:24:45.970 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:24:45.970 DEBUG:boto:Data: 2023-07-29T00:24:45.970 DEBUG:boto:Headers: {} 2023-07-29T00:24:45.971 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:45.971 DEBUG:boto:Port: 8001 2023-07-29T00:24:45.972 DEBUG:boto:Params: {} 2023-07-29T00:24:45.972 DEBUG:boto:Token: None 2023-07-29T00:24:45.972 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:45 GMT /hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:24:45.973 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:e5DSRwVNsc7neoIFFhIiaRPwk9U= 2023-07-29T00:24:45.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:e5DSRwVNsc7neoIFFhIiaRPwk9U=', 'Content-Length': '0'} 2023-07-29T00:24:45.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c81c1ac139d3dc89-0064c45c4d-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.016 DEBUG:boto:path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:24:46.017 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:24:46.017 DEBUG:boto:path=/hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:24:46.018 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:24:46.018 DEBUG:boto:Method: GET 2023-07-29T00:24:46.018 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:24:46.019 DEBUG:boto:Data: 2023-07-29T00:24:46.019 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.020 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:46.020 DEBUG:boto:Port: 8000 2023-07-29T00:24:46.020 DEBUG:boto:Params: {} 2023-07-29T00:24:46.021 DEBUG:boto:Token: None 2023-07-29T00:24:46.021 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:24:46.022 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:632VkSjgV692kFaPyV01JqNBomg= 2023-07-29T00:24:46.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:632VkSjgV692kFaPyV01JqNBomg=', 'Content-Length': '0'} 2023-07-29T00:24:46.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d049b1d1eb4fc62-0064c45c4d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.024 DEBUG:boto:path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:24:46.025 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:24:46.025 DEBUG:boto:path=/hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:24:46.026 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:24:46.026 DEBUG:boto:Method: HEAD 2023-07-29T00:24:46.026 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:24:46.027 DEBUG:boto:Data: 2023-07-29T00:24:46.027 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.027 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:46.028 DEBUG:boto:Port: 8001 2023-07-29T00:24:46.028 DEBUG:boto:Params: {} 2023-07-29T00:24:46.028 DEBUG:boto:Token: None 2023-07-29T00:24:46.029 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:24:46.029 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EWvoG//23aD3J9cRTLPoJNS2zj4= 2023-07-29T00:24:46.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EWvoG//23aD3J9cRTLPoJNS2zj4=', 'Content-Length': '0'} 2023-07-29T00:24:46.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072bf009d688883f8-0064c45c4d-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.032 DEBUG:boto:path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:24:46.032 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:24:46.033 DEBUG:boto:path=/hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:24:46.033 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:24:46.033 DEBUG:boto:Method: HEAD 2023-07-29T00:24:46.034 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:24:46.034 DEBUG:boto:Data: 2023-07-29T00:24:46.035 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.035 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:46.035 DEBUG:boto:Port: 8000 2023-07-29T00:24:46.036 DEBUG:boto:Params: {} 2023-07-29T00:24:46.036 DEBUG:boto:Token: None 2023-07-29T00:24:46.036 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:24:46.037 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EWvoG//23aD3J9cRTLPoJNS2zj4= 2023-07-29T00:24:46.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EWvoG//23aD3J9cRTLPoJNS2zj4=', 'Content-Length': '0'} 2023-07-29T00:24:46.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0777a7526370d9c-0064c45c4d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-07-29T00:24:46.040 DEBUG:boto:path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:24:46.041 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:24:46.041 DEBUG:boto:path=/hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:24:46.041 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:24:46.042 DEBUG:boto:Method: GET 2023-07-29T00:24:46.042 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:24:46.042 DEBUG:boto:Data: 2023-07-29T00:24:46.043 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.043 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:46.044 DEBUG:boto:Port: 8001 2023-07-29T00:24:46.044 DEBUG:boto:Params: {} 2023-07-29T00:24:46.044 DEBUG:boto:Token: None 2023-07-29T00:24:46.045 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:24:46.045 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:632VkSjgV692kFaPyV01JqNBomg= 2023-07-29T00:24:46.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:632VkSjgV692kFaPyV01JqNBomg=', 'Content-Length': '0'} 2023-07-29T00:24:46.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060e791b98283b7c3-0064c45c4d-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.048 DEBUG:boto:path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:24:46.048 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:24:46.049 DEBUG:boto:path=/hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:24:46.049 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:24:46.050 DEBUG:boto:Method: GET 2023-07-29T00:24:46.050 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:24:46.050 DEBUG:boto:Data: 2023-07-29T00:24:46.051 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.051 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:46.051 DEBUG:boto:Port: 8000 2023-07-29T00:24:46.052 DEBUG:boto:Params: {} 2023-07-29T00:24:46.052 DEBUG:boto:Token: None 2023-07-29T00:24:46.053 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:24:46.053 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:632VkSjgV692kFaPyV01JqNBomg= 2023-07-29T00:24:46.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:632VkSjgV692kFaPyV01JqNBomg=', 'Content-Length': '0'} 2023-07-29T00:24:46.055 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002905ce39d435b67e-0064c45c4e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.056 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:46.056 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:46.057 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000004 2023-07-29T00:24:46.057 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000004 2023-07-29T00:24:46.057 DEBUG:boto:Method: GET 2023-07-29T00:24:46.058 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000004 2023-07-29T00:24:46.058 DEBUG:boto:Data: 2023-07-29T00:24:46.059 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.059 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:46.059 DEBUG:boto:Port: 8001 2023-07-29T00:24:46.060 DEBUG:boto:Params: {} 2023-07-29T00:24:46.060 DEBUG:boto:Token: None 2023-07-29T00:24:46.060 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/ 2023-07-29T00:24:46.061 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TRm3KbIAOgxXRanghL6T0MIAtl0= 2023-07-29T00:24:46.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TRm3KbIAOgxXRanghL6T0MIAtl0=', 'Content-Length': '0'} 2023-07-29T00:24:46.063 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088126ff444a08fa5-0064c45c4e-1b40-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.064 DEBUG:boto:b'hiiqos-58obj000000000000000000041000falseobj000000000000000000042023-07-29T00:24:24.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:46.065 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:46.065 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:46.065 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000004 2023-07-29T00:24:46.066 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000004 2023-07-29T00:24:46.066 DEBUG:boto:Method: GET 2023-07-29T00:24:46.067 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000004 2023-07-29T00:24:46.067 DEBUG:boto:Data: 2023-07-29T00:24:46.067 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.068 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:46.068 DEBUG:boto:Port: 8000 2023-07-29T00:24:46.068 DEBUG:boto:Params: {} 2023-07-29T00:24:46.069 DEBUG:boto:Token: None 2023-07-29T00:24:46.069 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/ 2023-07-29T00:24:46.070 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TRm3KbIAOgxXRanghL6T0MIAtl0= 2023-07-29T00:24:46.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TRm3KbIAOgxXRanghL6T0MIAtl0=', 'Content-Length': '0'} 2023-07-29T00:24:46.073 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbf8823f007fb7cd-0064c45c4e-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.073 DEBUG:boto:b'hiiqos-58obj000000000000000000041000falseobj000000000000000000042023-07-29T00:24:24.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:46.074 DEBUG:boto:path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:24:46.074 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:24:46.075 DEBUG:boto:Method: HEAD 2023-07-29T00:24:46.075 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000004 2023-07-29T00:24:46.075 DEBUG:boto:Data: 2023-07-29T00:24:46.076 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.076 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:46.076 DEBUG:boto:Port: 8001 2023-07-29T00:24:46.077 DEBUG:boto:Params: {} 2023-07-29T00:24:46.077 DEBUG:boto:Token: None 2023-07-29T00:24:46.078 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000004 2023-07-29T00:24:46.078 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JW4l6Rfnen/QeXJnTCMdpaVniak= 2023-07-29T00:24:46.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JW4l6Rfnen/QeXJnTCMdpaVniak=', 'Content-Length': '0'} 2023-07-29T00:24:46.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa3d772649c0f772-0064c45c4e-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.081 DEBUG:boto:path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:24:46.081 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:24:46.081 DEBUG:boto:Method: HEAD 2023-07-29T00:24:46.082 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000004 2023-07-29T00:24:46.082 DEBUG:boto:Data: 2023-07-29T00:24:46.082 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.083 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:46.083 DEBUG:boto:Port: 8000 2023-07-29T00:24:46.084 DEBUG:boto:Params: {} 2023-07-29T00:24:46.084 DEBUG:boto:Token: None 2023-07-29T00:24:46.084 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000004 2023-07-29T00:24:46.085 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JW4l6Rfnen/QeXJnTCMdpaVniak= 2023-07-29T00:24:46.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JW4l6Rfnen/QeXJnTCMdpaVniak=', 'Content-Length': '0'} 2023-07-29T00:24:46.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5c3f08b0ebe14ef-0064c45c4e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-07-29T00:24:46.088 DEBUG:boto:path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:24:46.088 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:24:46.088 DEBUG:boto:Method: GET 2023-07-29T00:24:46.089 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000004 2023-07-29T00:24:46.089 DEBUG:boto:Data: 2023-07-29T00:24:46.090 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.090 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:46.090 DEBUG:boto:Port: 8001 2023-07-29T00:24:46.091 DEBUG:boto:Params: {} 2023-07-29T00:24:46.091 DEBUG:boto:Token: None 2023-07-29T00:24:46.092 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000004 2023-07-29T00:24:46.092 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gsfGnaCbzer47u9afgYOuLb6hxw= 2023-07-29T00:24:46.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gsfGnaCbzer47u9afgYOuLb6hxw=', 'Content-Length': '0'} 2023-07-29T00:24:46.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057e0d18c7238d8d9-0064c45c4e-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.095 DEBUG:boto:path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:24:46.095 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:24:46.095 DEBUG:boto:Method: GET 2023-07-29T00:24:46.096 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000004 2023-07-29T00:24:46.096 DEBUG:boto:Data: 2023-07-29T00:24:46.097 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.097 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:46.097 DEBUG:boto:Port: 8000 2023-07-29T00:24:46.098 DEBUG:boto:Params: {} 2023-07-29T00:24:46.098 DEBUG:boto:Token: None 2023-07-29T00:24:46.098 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000004 2023-07-29T00:24:46.099 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gsfGnaCbzer47u9afgYOuLb6hxw= 2023-07-29T00:24:46.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gsfGnaCbzer47u9afgYOuLb6hxw=', 'Content-Length': '0'} 2023-07-29T00:24:46.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000064c2a844bc61e43-0064c45c4e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.102 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:46.102 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:46.103 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000004 2023-07-29T00:24:46.103 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000004 2023-07-29T00:24:46.103 DEBUG:boto:Method: GET 2023-07-29T00:24:46.104 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000004 2023-07-29T00:24:46.104 DEBUG:boto:Data: 2023-07-29T00:24:46.104 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.105 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:46.105 DEBUG:boto:Port: 8001 2023-07-29T00:24:46.105 DEBUG:boto:Params: {} 2023-07-29T00:24:46.106 DEBUG:boto:Token: None 2023-07-29T00:24:46.106 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/ 2023-07-29T00:24:46.107 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TRm3KbIAOgxXRanghL6T0MIAtl0= 2023-07-29T00:24:46.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TRm3KbIAOgxXRanghL6T0MIAtl0=', 'Content-Length': '0'} 2023-07-29T00:24:46.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa05a67329d30a7b-0064c45c4e-1b40-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.110 DEBUG:boto:b'hiiqos-58obj000000000000000000041000falseobj000000000000000000042023-07-29T00:24:24.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:46.110 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:46.111 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:46.111 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000004 2023-07-29T00:24:46.111 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000004 2023-07-29T00:24:46.112 DEBUG:boto:Method: GET 2023-07-29T00:24:46.112 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000004 2023-07-29T00:24:46.113 DEBUG:boto:Data: 2023-07-29T00:24:46.113 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.113 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:46.114 DEBUG:boto:Port: 8000 2023-07-29T00:24:46.114 DEBUG:boto:Params: {} 2023-07-29T00:24:46.115 DEBUG:boto:Token: None 2023-07-29T00:24:46.115 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/ 2023-07-29T00:24:46.115 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TRm3KbIAOgxXRanghL6T0MIAtl0= 2023-07-29T00:24:46.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TRm3KbIAOgxXRanghL6T0MIAtl0=', 'Content-Length': '0'} 2023-07-29T00:24:46.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a261c2d476f7c2ad-0064c45c4e-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.160 DEBUG:boto:b'hiiqos-58obj000000000000000000041000falseobj000000000000000000042023-07-29T00:24:24.554Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:46.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-07-29T00:24:46.162 DEBUG:boto:path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:24:46.163 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:24:46.163 DEBUG:boto:path=/hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:24:46.163 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:24:46.164 DEBUG:boto:Method: GET 2023-07-29T00:24:46.164 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:24:46.165 DEBUG:boto:Data: 2023-07-29T00:24:46.165 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.166 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:46.166 DEBUG:boto:Port: 8001 2023-07-29T00:24:46.166 DEBUG:boto:Params: {} 2023-07-29T00:24:46.167 DEBUG:boto:Token: None 2023-07-29T00:24:46.167 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:24:46.168 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Hr/rX7IhOKP34HxOZCJdmjEofE8= 2023-07-29T00:24:46.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Hr/rX7IhOKP34HxOZCJdmjEofE8=', 'Content-Length': '0'} 2023-07-29T00:24:46.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006d18fb9b51fc0c3-0064c45c4e-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.171 DEBUG:boto:path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:24:46.171 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:24:46.172 DEBUG:boto:path=/hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:24:46.172 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:24:46.172 DEBUG:boto:Method: GET 2023-07-29T00:24:46.173 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:24:46.173 DEBUG:boto:Data: 2023-07-29T00:24:46.174 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.174 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:46.174 DEBUG:boto:Port: 8000 2023-07-29T00:24:46.175 DEBUG:boto:Params: {} 2023-07-29T00:24:46.175 DEBUG:boto:Token: None 2023-07-29T00:24:46.176 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:24:46.176 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Hr/rX7IhOKP34HxOZCJdmjEofE8= 2023-07-29T00:24:46.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Hr/rX7IhOKP34HxOZCJdmjEofE8=', 'Content-Length': '0'} 2023-07-29T00:24:46.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002af673018817b24c-0064c45c4e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.179 DEBUG:boto:path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:24:46.180 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:24:46.180 DEBUG:boto:path=/hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:24:46.180 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:24:46.181 DEBUG:boto:Method: HEAD 2023-07-29T00:24:46.181 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:24:46.182 DEBUG:boto:Data: 2023-07-29T00:24:46.182 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.182 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:46.183 DEBUG:boto:Port: 8001 2023-07-29T00:24:46.183 DEBUG:boto:Params: {} 2023-07-29T00:24:46.184 DEBUG:boto:Token: None 2023-07-29T00:24:46.185 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:24:46.186 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BJcmffctBbsvprH+hUdF4uMvTOg= 2023-07-29T00:24:46.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BJcmffctBbsvprH+hUdF4uMvTOg=', 'Content-Length': '0'} 2023-07-29T00:24:46.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014616b7d3e2aa155-0064c45c4e-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.190 DEBUG:boto:path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:24:46.190 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:24:46.191 DEBUG:boto:path=/hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:24:46.191 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:24:46.191 DEBUG:boto:Method: HEAD 2023-07-29T00:24:46.192 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:24:46.192 DEBUG:boto:Data: 2023-07-29T00:24:46.193 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.194 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:46.195 DEBUG:boto:Port: 8000 2023-07-29T00:24:46.195 DEBUG:boto:Params: {} 2023-07-29T00:24:46.196 DEBUG:boto:Token: None 2023-07-29T00:24:46.197 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:24:46.198 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BJcmffctBbsvprH+hUdF4uMvTOg= 2023-07-29T00:24:46.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BJcmffctBbsvprH+hUdF4uMvTOg=', 'Content-Length': '0'} 2023-07-29T00:24:46.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7550394bf6bb56f-0064c45c4e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-07-29T00:24:46.202 DEBUG:boto:path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:24:46.202 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:24:46.203 DEBUG:boto:path=/hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:24:46.203 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:24:46.203 DEBUG:boto:Method: GET 2023-07-29T00:24:46.204 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:24:46.204 DEBUG:boto:Data: 2023-07-29T00:24:46.205 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.205 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:46.205 DEBUG:boto:Port: 8001 2023-07-29T00:24:46.206 DEBUG:boto:Params: {} 2023-07-29T00:24:46.206 DEBUG:boto:Token: None 2023-07-29T00:24:46.207 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:24:46.207 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Hr/rX7IhOKP34HxOZCJdmjEofE8= 2023-07-29T00:24:46.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Hr/rX7IhOKP34HxOZCJdmjEofE8=', 'Content-Length': '0'} 2023-07-29T00:24:46.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea2ca42588ea65ad-0064c45c4e-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.210 DEBUG:boto:path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:24:46.211 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:24:46.211 DEBUG:boto:path=/hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:24:46.211 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:24:46.212 DEBUG:boto:Method: GET 2023-07-29T00:24:46.212 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:24:46.213 DEBUG:boto:Data: 2023-07-29T00:24:46.213 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.214 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:46.214 DEBUG:boto:Port: 8000 2023-07-29T00:24:46.214 DEBUG:boto:Params: {} 2023-07-29T00:24:46.215 DEBUG:boto:Token: None 2023-07-29T00:24:46.215 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:24:46.216 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Hr/rX7IhOKP34HxOZCJdmjEofE8= 2023-07-29T00:24:46.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Hr/rX7IhOKP34HxOZCJdmjEofE8=', 'Content-Length': '0'} 2023-07-29T00:24:46.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012d1e5f2d9582632-0064c45c4e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.220 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:46.220 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:46.221 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000005 2023-07-29T00:24:46.222 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000005 2023-07-29T00:24:46.222 DEBUG:boto:Method: GET 2023-07-29T00:24:46.223 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000005 2023-07-29T00:24:46.223 DEBUG:boto:Data: 2023-07-29T00:24:46.224 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.225 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:46.225 DEBUG:boto:Port: 8001 2023-07-29T00:24:46.226 DEBUG:boto:Params: {} 2023-07-29T00:24:46.227 DEBUG:boto:Token: None 2023-07-29T00:24:46.227 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/ 2023-07-29T00:24:46.228 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TRm3KbIAOgxXRanghL6T0MIAtl0= 2023-07-29T00:24:46.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TRm3KbIAOgxXRanghL6T0MIAtl0=', 'Content-Length': '0'} 2023-07-29T00:24:46.231 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffffc739e2e61d7d-0064c45c4e-1b40-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.232 DEBUG:boto:b'hiiqos-58obj000000000000000000051000falseobj000000000000000000052023-07-29T00:24:24.726Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:46.232 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:46.233 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:46.233 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000005 2023-07-29T00:24:46.234 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000005 2023-07-29T00:24:46.234 DEBUG:boto:Method: GET 2023-07-29T00:24:46.234 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000005 2023-07-29T00:24:46.235 DEBUG:boto:Data: 2023-07-29T00:24:46.235 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.236 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:46.236 DEBUG:boto:Port: 8000 2023-07-29T00:24:46.236 DEBUG:boto:Params: {} 2023-07-29T00:24:46.237 DEBUG:boto:Token: None 2023-07-29T00:24:46.237 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/ 2023-07-29T00:24:46.238 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TRm3KbIAOgxXRanghL6T0MIAtl0= 2023-07-29T00:24:46.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TRm3KbIAOgxXRanghL6T0MIAtl0=', 'Content-Length': '0'} 2023-07-29T00:24:46.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000332679a0795d01a8-0064c45c4e-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.241 DEBUG:boto:b'hiiqos-58obj000000000000000000051000falseobj000000000000000000052023-07-29T00:24:24.726Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:46.242 DEBUG:boto:path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:24:46.243 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:24:46.243 DEBUG:boto:Method: HEAD 2023-07-29T00:24:46.243 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000005 2023-07-29T00:24:46.244 DEBUG:boto:Data: 2023-07-29T00:24:46.244 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.244 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:46.245 DEBUG:boto:Port: 8001 2023-07-29T00:24:46.245 DEBUG:boto:Params: {} 2023-07-29T00:24:46.246 DEBUG:boto:Token: None 2023-07-29T00:24:46.246 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000005 2023-07-29T00:24:46.246 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:C/r1jCFFBm5DBRTJ6fUHE8f7QWo= 2023-07-29T00:24:46.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:C/r1jCFFBm5DBRTJ6fUHE8f7QWo=', 'Content-Length': '0'} 2023-07-29T00:24:46.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb75da997ef190f0-0064c45c4e-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.249 DEBUG:boto:path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:24:46.249 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:24:46.250 DEBUG:boto:Method: HEAD 2023-07-29T00:24:46.250 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000005 2023-07-29T00:24:46.250 DEBUG:boto:Data: 2023-07-29T00:24:46.251 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.251 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:46.251 DEBUG:boto:Port: 8000 2023-07-29T00:24:46.252 DEBUG:boto:Params: {} 2023-07-29T00:24:46.252 DEBUG:boto:Token: None 2023-07-29T00:24:46.252 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000005 2023-07-29T00:24:46.253 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:C/r1jCFFBm5DBRTJ6fUHE8f7QWo= 2023-07-29T00:24:46.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:C/r1jCFFBm5DBRTJ6fUHE8f7QWo=', 'Content-Length': '0'} 2023-07-29T00:24:46.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065acb5a644e22484-0064c45c4e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-07-29T00:24:46.256 DEBUG:boto:path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:24:46.256 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:24:46.257 DEBUG:boto:Method: GET 2023-07-29T00:24:46.257 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000005 2023-07-29T00:24:46.257 DEBUG:boto:Data: 2023-07-29T00:24:46.258 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.258 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:46.258 DEBUG:boto:Port: 8001 2023-07-29T00:24:46.259 DEBUG:boto:Params: {} 2023-07-29T00:24:46.259 DEBUG:boto:Token: None 2023-07-29T00:24:46.260 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000005 2023-07-29T00:24:46.260 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PyRcDVGIFDDfK8pwIBfcpvw0K9U= 2023-07-29T00:24:46.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PyRcDVGIFDDfK8pwIBfcpvw0K9U=', 'Content-Length': '0'} 2023-07-29T00:24:46.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ad06005415e9d2f-0064c45c4e-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.263 DEBUG:boto:path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:24:46.263 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:24:46.263 DEBUG:boto:Method: GET 2023-07-29T00:24:46.264 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000005 2023-07-29T00:24:46.264 DEBUG:boto:Data: 2023-07-29T00:24:46.265 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.265 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:46.265 DEBUG:boto:Port: 8000 2023-07-29T00:24:46.266 DEBUG:boto:Params: {} 2023-07-29T00:24:46.266 DEBUG:boto:Token: None 2023-07-29T00:24:46.266 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000005 2023-07-29T00:24:46.267 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PyRcDVGIFDDfK8pwIBfcpvw0K9U= 2023-07-29T00:24:46.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PyRcDVGIFDDfK8pwIBfcpvw0K9U=', 'Content-Length': '0'} 2023-07-29T00:24:46.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a42bc787dbc4b940-0064c45c4e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.312 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:46.313 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:46.313 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000005 2023-07-29T00:24:46.314 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000005 2023-07-29T00:24:46.314 DEBUG:boto:Method: GET 2023-07-29T00:24:46.314 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000005 2023-07-29T00:24:46.315 DEBUG:boto:Data: 2023-07-29T00:24:46.315 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.315 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:46.316 DEBUG:boto:Port: 8001 2023-07-29T00:24:46.316 DEBUG:boto:Params: {} 2023-07-29T00:24:46.317 DEBUG:boto:Token: None 2023-07-29T00:24:46.317 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/ 2023-07-29T00:24:46.317 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TRm3KbIAOgxXRanghL6T0MIAtl0= 2023-07-29T00:24:46.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TRm3KbIAOgxXRanghL6T0MIAtl0=', 'Content-Length': '0'} 2023-07-29T00:24:46.320 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035f1f9046a60036e-0064c45c4e-1b40-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.321 DEBUG:boto:b'hiiqos-58obj000000000000000000051000falseobj000000000000000000052023-07-29T00:24:24.726Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:46.322 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:46.322 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:46.322 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000005 2023-07-29T00:24:46.323 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000005 2023-07-29T00:24:46.323 DEBUG:boto:Method: GET 2023-07-29T00:24:46.324 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000005 2023-07-29T00:24:46.324 DEBUG:boto:Data: 2023-07-29T00:24:46.324 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.325 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:46.325 DEBUG:boto:Port: 8000 2023-07-29T00:24:46.325 DEBUG:boto:Params: {} 2023-07-29T00:24:46.326 DEBUG:boto:Token: None 2023-07-29T00:24:46.326 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/ 2023-07-29T00:24:46.327 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TRm3KbIAOgxXRanghL6T0MIAtl0= 2023-07-29T00:24:46.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TRm3KbIAOgxXRanghL6T0MIAtl0=', 'Content-Length': '0'} 2023-07-29T00:24:46.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008967caa5d4b59cd2-0064c45c4e-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.330 DEBUG:boto:b'hiiqos-58obj000000000000000000051000falseobj000000000000000000052023-07-29T00:24:24.726Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:46.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-07-29T00:24:46.332 DEBUG:boto:path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:24:46.332 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:24:46.332 DEBUG:boto:path=/hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:24:46.333 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:24:46.334 DEBUG:boto:Method: GET 2023-07-29T00:24:46.334 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:24:46.334 DEBUG:boto:Data: 2023-07-29T00:24:46.335 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.335 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:46.335 DEBUG:boto:Port: 8001 2023-07-29T00:24:46.336 DEBUG:boto:Params: {} 2023-07-29T00:24:46.336 DEBUG:boto:Token: None 2023-07-29T00:24:46.337 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:24:46.337 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dlRtDy0a/Tp+GVfarVqHxoOt6BU= 2023-07-29T00:24:46.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dlRtDy0a/Tp+GVfarVqHxoOt6BU=', 'Content-Length': '0'} 2023-07-29T00:24:46.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000982b9fb74f59792c-0064c45c4e-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.380 DEBUG:boto:path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:24:46.381 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:24:46.381 DEBUG:boto:path=/hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:24:46.382 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:24:46.382 DEBUG:boto:Method: GET 2023-07-29T00:24:46.383 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:24:46.383 DEBUG:boto:Data: 2023-07-29T00:24:46.383 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.384 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:46.384 DEBUG:boto:Port: 8000 2023-07-29T00:24:46.385 DEBUG:boto:Params: {} 2023-07-29T00:24:46.385 DEBUG:boto:Token: None 2023-07-29T00:24:46.386 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:24:46.386 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dlRtDy0a/Tp+GVfarVqHxoOt6BU= 2023-07-29T00:24:46.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dlRtDy0a/Tp+GVfarVqHxoOt6BU=', 'Content-Length': '0'} 2023-07-29T00:24:46.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9fc64452a47fcad-0064c45c4e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.389 DEBUG:boto:path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:24:46.389 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:24:46.390 DEBUG:boto:path=/hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:24:46.390 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:24:46.390 DEBUG:boto:Method: HEAD 2023-07-29T00:24:46.391 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:24:46.391 DEBUG:boto:Data: 2023-07-29T00:24:46.391 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.392 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:46.392 DEBUG:boto:Port: 8001 2023-07-29T00:24:46.392 DEBUG:boto:Params: {} 2023-07-29T00:24:46.393 DEBUG:boto:Token: None 2023-07-29T00:24:46.393 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:24:46.394 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DYYJ2bmzSW0D5C25uJP7JlK/HCs= 2023-07-29T00:24:46.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DYYJ2bmzSW0D5C25uJP7JlK/HCs=', 'Content-Length': '0'} 2023-07-29T00:24:46.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e91bdcaca3fdd09b-0064c45c4e-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.397 DEBUG:boto:path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:24:46.397 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:24:46.397 DEBUG:boto:path=/hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:24:46.398 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:24:46.398 DEBUG:boto:Method: HEAD 2023-07-29T00:24:46.398 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:24:46.399 DEBUG:boto:Data: 2023-07-29T00:24:46.399 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.399 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:46.400 DEBUG:boto:Port: 8000 2023-07-29T00:24:46.400 DEBUG:boto:Params: {} 2023-07-29T00:24:46.400 DEBUG:boto:Token: None 2023-07-29T00:24:46.401 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:24:46.401 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DYYJ2bmzSW0D5C25uJP7JlK/HCs= 2023-07-29T00:24:46.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DYYJ2bmzSW0D5C25uJP7JlK/HCs=', 'Content-Length': '0'} 2023-07-29T00:24:46.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d2fb093d725265e-0064c45c4e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-07-29T00:24:46.404 DEBUG:boto:path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:24:46.405 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:24:46.405 DEBUG:boto:path=/hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:24:46.405 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:24:46.406 DEBUG:boto:Method: GET 2023-07-29T00:24:46.406 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:24:46.407 DEBUG:boto:Data: 2023-07-29T00:24:46.407 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.407 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:46.408 DEBUG:boto:Port: 8001 2023-07-29T00:24:46.408 DEBUG:boto:Params: {} 2023-07-29T00:24:46.408 DEBUG:boto:Token: None 2023-07-29T00:24:46.409 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:24:46.409 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dlRtDy0a/Tp+GVfarVqHxoOt6BU= 2023-07-29T00:24:46.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dlRtDy0a/Tp+GVfarVqHxoOt6BU=', 'Content-Length': '0'} 2023-07-29T00:24:46.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000447c56a46c48443f-0064c45c4e-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.413 DEBUG:boto:path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:24:46.414 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:24:46.414 DEBUG:boto:path=/hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:24:46.414 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:24:46.415 DEBUG:boto:Method: GET 2023-07-29T00:24:46.415 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:24:46.415 DEBUG:boto:Data: 2023-07-29T00:24:46.416 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.416 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:46.416 DEBUG:boto:Port: 8000 2023-07-29T00:24:46.417 DEBUG:boto:Params: {} 2023-07-29T00:24:46.417 DEBUG:boto:Token: None 2023-07-29T00:24:46.418 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:24:46.418 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dlRtDy0a/Tp+GVfarVqHxoOt6BU= 2023-07-29T00:24:46.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dlRtDy0a/Tp+GVfarVqHxoOt6BU=', 'Content-Length': '0'} 2023-07-29T00:24:46.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003dc0b94d7888fb8a-0064c45c4e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.421 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:46.421 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:46.422 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000006 2023-07-29T00:24:46.422 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000006 2023-07-29T00:24:46.422 DEBUG:boto:Method: GET 2023-07-29T00:24:46.423 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000006 2023-07-29T00:24:46.423 DEBUG:boto:Data: 2023-07-29T00:24:46.423 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.424 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:46.424 DEBUG:boto:Port: 8001 2023-07-29T00:24:46.425 DEBUG:boto:Params: {} 2023-07-29T00:24:46.425 DEBUG:boto:Token: None 2023-07-29T00:24:46.425 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/ 2023-07-29T00:24:46.426 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TRm3KbIAOgxXRanghL6T0MIAtl0= 2023-07-29T00:24:46.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TRm3KbIAOgxXRanghL6T0MIAtl0=', 'Content-Length': '0'} 2023-07-29T00:24:46.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049aaf9bd493939ca-0064c45c4e-1b40-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.429 DEBUG:boto:b'hiiqos-58obj000000000000000000061000falseobj000000000000000000062023-07-29T00:24:24.537Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:46.430 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:46.430 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:46.430 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000006 2023-07-29T00:24:46.431 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000006 2023-07-29T00:24:46.431 DEBUG:boto:Method: GET 2023-07-29T00:24:46.432 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000006 2023-07-29T00:24:46.432 DEBUG:boto:Data: 2023-07-29T00:24:46.432 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.433 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:46.433 DEBUG:boto:Port: 8000 2023-07-29T00:24:46.433 DEBUG:boto:Params: {} 2023-07-29T00:24:46.434 DEBUG:boto:Token: None 2023-07-29T00:24:46.434 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/ 2023-07-29T00:24:46.435 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TRm3KbIAOgxXRanghL6T0MIAtl0= 2023-07-29T00:24:46.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TRm3KbIAOgxXRanghL6T0MIAtl0=', 'Content-Length': '0'} 2023-07-29T00:24:46.438 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f691205f0987c782-0064c45c4e-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.480 DEBUG:boto:b'hiiqos-58obj000000000000000000061000falseobj000000000000000000062023-07-29T00:24:24.537Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:46.482 DEBUG:boto:path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:24:46.482 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:24:46.483 DEBUG:boto:Method: HEAD 2023-07-29T00:24:46.483 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000006 2023-07-29T00:24:46.484 DEBUG:boto:Data: 2023-07-29T00:24:46.484 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.484 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:46.485 DEBUG:boto:Port: 8001 2023-07-29T00:24:46.485 DEBUG:boto:Params: {} 2023-07-29T00:24:46.485 DEBUG:boto:Token: None 2023-07-29T00:24:46.486 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000006 2023-07-29T00:24:46.487 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XbKQ+uAL8VjmUAywNK1AjzOOoOY= 2023-07-29T00:24:46.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XbKQ+uAL8VjmUAywNK1AjzOOoOY=', 'Content-Length': '0'} 2023-07-29T00:24:46.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d425b6f09f875d1d-0064c45c4e-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.489 DEBUG:boto:path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:24:46.490 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:24:46.491 DEBUG:boto:Method: HEAD 2023-07-29T00:24:46.491 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000006 2023-07-29T00:24:46.492 DEBUG:boto:Data: 2023-07-29T00:24:46.492 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.493 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:46.494 DEBUG:boto:Port: 8000 2023-07-29T00:24:46.494 DEBUG:boto:Params: {} 2023-07-29T00:24:46.495 DEBUG:boto:Token: None 2023-07-29T00:24:46.496 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000006 2023-07-29T00:24:46.497 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XbKQ+uAL8VjmUAywNK1AjzOOoOY= 2023-07-29T00:24:46.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XbKQ+uAL8VjmUAywNK1AjzOOoOY=', 'Content-Length': '0'} 2023-07-29T00:24:46.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001255754a07b919d8-0064c45c4e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-07-29T00:24:46.501 DEBUG:boto:path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:24:46.501 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:24:46.502 DEBUG:boto:Method: GET 2023-07-29T00:24:46.503 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000006 2023-07-29T00:24:46.503 DEBUG:boto:Data: 2023-07-29T00:24:46.503 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.504 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:46.505 DEBUG:boto:Port: 8001 2023-07-29T00:24:46.506 DEBUG:boto:Params: {} 2023-07-29T00:24:46.507 DEBUG:boto:Token: None 2023-07-29T00:24:46.507 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000006 2023-07-29T00:24:46.508 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:o5Q5Ck522nPY14eOsjrvnjlT1Gc= 2023-07-29T00:24:46.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:o5Q5Ck522nPY14eOsjrvnjlT1Gc=', 'Content-Length': '0'} 2023-07-29T00:24:46.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000309f17a67717becc-0064c45c4e-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.556 DEBUG:boto:path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:24:46.557 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:24:46.557 DEBUG:boto:Method: GET 2023-07-29T00:24:46.558 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000006 2023-07-29T00:24:46.558 DEBUG:boto:Data: 2023-07-29T00:24:46.558 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.559 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:46.559 DEBUG:boto:Port: 8000 2023-07-29T00:24:46.560 DEBUG:boto:Params: {} 2023-07-29T00:24:46.560 DEBUG:boto:Token: None 2023-07-29T00:24:46.561 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000006 2023-07-29T00:24:46.561 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:o5Q5Ck522nPY14eOsjrvnjlT1Gc= 2023-07-29T00:24:46.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:o5Q5Ck522nPY14eOsjrvnjlT1Gc=', 'Content-Length': '0'} 2023-07-29T00:24:46.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d096434b2ff7c59-0064c45c4e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.565 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:46.565 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:46.566 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000006 2023-07-29T00:24:46.566 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000006 2023-07-29T00:24:46.566 DEBUG:boto:Method: GET 2023-07-29T00:24:46.567 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000006 2023-07-29T00:24:46.567 DEBUG:boto:Data: 2023-07-29T00:24:46.567 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.568 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:46.568 DEBUG:boto:Port: 8001 2023-07-29T00:24:46.569 DEBUG:boto:Params: {} 2023-07-29T00:24:46.569 DEBUG:boto:Token: None 2023-07-29T00:24:46.570 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/ 2023-07-29T00:24:46.571 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TRm3KbIAOgxXRanghL6T0MIAtl0= 2023-07-29T00:24:46.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TRm3KbIAOgxXRanghL6T0MIAtl0=', 'Content-Length': '0'} 2023-07-29T00:24:46.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008358ab3dd7444a8b-0064c45c4e-1b40-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.575 DEBUG:boto:b'hiiqos-58obj000000000000000000061000falseobj000000000000000000062023-07-29T00:24:24.537Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:46.576 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:46.577 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:46.578 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000006 2023-07-29T00:24:46.578 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000006 2023-07-29T00:24:46.579 DEBUG:boto:Method: GET 2023-07-29T00:24:46.579 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000006 2023-07-29T00:24:46.579 DEBUG:boto:Data: 2023-07-29T00:24:46.580 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.580 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:46.581 DEBUG:boto:Port: 8000 2023-07-29T00:24:46.581 DEBUG:boto:Params: {} 2023-07-29T00:24:46.582 DEBUG:boto:Token: None 2023-07-29T00:24:46.582 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/ 2023-07-29T00:24:46.582 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TRm3KbIAOgxXRanghL6T0MIAtl0= 2023-07-29T00:24:46.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TRm3KbIAOgxXRanghL6T0MIAtl0=', 'Content-Length': '0'} 2023-07-29T00:24:46.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008a657381eb728b5-0064c45c4e-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.586 DEBUG:boto:b'hiiqos-58obj000000000000000000061000falseobj000000000000000000062023-07-29T00:24:24.537Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:46.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-07-29T00:24:46.587 DEBUG:boto:path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:24:46.587 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:24:46.587 DEBUG:boto:path=/hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:24:46.588 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:24:46.588 DEBUG:boto:Method: GET 2023-07-29T00:24:46.588 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:24:46.589 DEBUG:boto:Data: 2023-07-29T00:24:46.589 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.590 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:46.590 DEBUG:boto:Port: 8001 2023-07-29T00:24:46.590 DEBUG:boto:Params: {} 2023-07-29T00:24:46.591 DEBUG:boto:Token: None 2023-07-29T00:24:46.591 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:24:46.592 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wZ8vwyaIEwyc3vX19XuNeTzTBkA= 2023-07-29T00:24:46.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wZ8vwyaIEwyc3vX19XuNeTzTBkA=', 'Content-Length': '0'} 2023-07-29T00:24:46.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038b1d7ffe4f12d73-0064c45c4e-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.595 DEBUG:boto:path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:24:46.595 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:24:46.596 DEBUG:boto:path=/hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:24:46.596 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:24:46.596 DEBUG:boto:Method: GET 2023-07-29T00:24:46.597 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:24:46.597 DEBUG:boto:Data: 2023-07-29T00:24:46.598 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.598 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:46.599 DEBUG:boto:Port: 8000 2023-07-29T00:24:46.599 DEBUG:boto:Params: {} 2023-07-29T00:24:46.600 DEBUG:boto:Token: None 2023-07-29T00:24:46.600 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:24:46.600 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wZ8vwyaIEwyc3vX19XuNeTzTBkA= 2023-07-29T00:24:46.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wZ8vwyaIEwyc3vX19XuNeTzTBkA=', 'Content-Length': '0'} 2023-07-29T00:24:46.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000507b90b07ec37dab-0064c45c4e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.604 DEBUG:boto:path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:24:46.605 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:24:46.606 DEBUG:boto:path=/hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:24:46.606 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:24:46.607 DEBUG:boto:Method: HEAD 2023-07-29T00:24:46.608 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:24:46.608 DEBUG:boto:Data: 2023-07-29T00:24:46.608 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.609 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:46.609 DEBUG:boto:Port: 8001 2023-07-29T00:24:46.609 DEBUG:boto:Params: {} 2023-07-29T00:24:46.610 DEBUG:boto:Token: None 2023-07-29T00:24:46.610 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:24:46.611 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9XIF+wYYww+C0u2qPl/MGwSpapg= 2023-07-29T00:24:46.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9XIF+wYYww+C0u2qPl/MGwSpapg=', 'Content-Length': '0'} 2023-07-29T00:24:46.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae0ad6755ae2de2d-0064c45c4e-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.613 DEBUG:boto:path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:24:46.614 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:24:46.614 DEBUG:boto:path=/hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:24:46.614 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:24:46.615 DEBUG:boto:Method: HEAD 2023-07-29T00:24:46.615 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:24:46.615 DEBUG:boto:Data: 2023-07-29T00:24:46.616 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.616 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:46.616 DEBUG:boto:Port: 8000 2023-07-29T00:24:46.617 DEBUG:boto:Params: {} 2023-07-29T00:24:46.617 DEBUG:boto:Token: None 2023-07-29T00:24:46.618 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:24:46.618 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9XIF+wYYww+C0u2qPl/MGwSpapg= 2023-07-29T00:24:46.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9XIF+wYYww+C0u2qPl/MGwSpapg=', 'Content-Length': '0'} 2023-07-29T00:24:46.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0f8d0bc9ac0226a-0064c45c4e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-07-29T00:24:46.621 DEBUG:boto:path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:24:46.622 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:24:46.622 DEBUG:boto:path=/hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:24:46.623 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:24:46.623 DEBUG:boto:Method: GET 2023-07-29T00:24:46.623 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:24:46.624 DEBUG:boto:Data: 2023-07-29T00:24:46.624 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.625 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:46.625 DEBUG:boto:Port: 8001 2023-07-29T00:24:46.626 DEBUG:boto:Params: {} 2023-07-29T00:24:46.626 DEBUG:boto:Token: None 2023-07-29T00:24:46.626 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:24:46.627 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wZ8vwyaIEwyc3vX19XuNeTzTBkA= 2023-07-29T00:24:46.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wZ8vwyaIEwyc3vX19XuNeTzTBkA=', 'Content-Length': '0'} 2023-07-29T00:24:46.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6c408f1a5423b35-0064c45c4e-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.672 DEBUG:boto:path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:24:46.673 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:24:46.673 DEBUG:boto:path=/hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:24:46.674 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:24:46.674 DEBUG:boto:Method: GET 2023-07-29T00:24:46.675 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:24:46.675 DEBUG:boto:Data: 2023-07-29T00:24:46.675 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.676 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:46.676 DEBUG:boto:Port: 8000 2023-07-29T00:24:46.676 DEBUG:boto:Params: {} 2023-07-29T00:24:46.677 DEBUG:boto:Token: None 2023-07-29T00:24:46.677 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:24:46.678 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wZ8vwyaIEwyc3vX19XuNeTzTBkA= 2023-07-29T00:24:46.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wZ8vwyaIEwyc3vX19XuNeTzTBkA=', 'Content-Length': '0'} 2023-07-29T00:24:46.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000282fc808912cd879-0064c45c4e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.681 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:46.682 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:46.683 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000011 2023-07-29T00:24:46.683 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000011 2023-07-29T00:24:46.684 DEBUG:boto:Method: GET 2023-07-29T00:24:46.684 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000011 2023-07-29T00:24:46.684 DEBUG:boto:Data: 2023-07-29T00:24:46.685 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.685 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:46.686 DEBUG:boto:Port: 8001 2023-07-29T00:24:46.686 DEBUG:boto:Params: {} 2023-07-29T00:24:46.686 DEBUG:boto:Token: None 2023-07-29T00:24:46.687 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/ 2023-07-29T00:24:46.687 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TRm3KbIAOgxXRanghL6T0MIAtl0= 2023-07-29T00:24:46.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TRm3KbIAOgxXRanghL6T0MIAtl0=', 'Content-Length': '0'} 2023-07-29T00:24:46.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dea951af01c19e77-0064c45c4e-1b40-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.691 DEBUG:boto:b'hiiqos-58obj000000000000000000111000falseobj000000000000000000112023-07-29T00:24:24.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:46.693 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:46.694 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:46.694 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000011 2023-07-29T00:24:46.695 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000011 2023-07-29T00:24:46.696 DEBUG:boto:Method: GET 2023-07-29T00:24:46.697 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000011 2023-07-29T00:24:46.698 DEBUG:boto:Data: 2023-07-29T00:24:46.699 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.700 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:46.700 DEBUG:boto:Port: 8000 2023-07-29T00:24:46.701 DEBUG:boto:Params: {} 2023-07-29T00:24:46.702 DEBUG:boto:Token: None 2023-07-29T00:24:46.703 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/ 2023-07-29T00:24:46.703 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TRm3KbIAOgxXRanghL6T0MIAtl0= 2023-07-29T00:24:46.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TRm3KbIAOgxXRanghL6T0MIAtl0=', 'Content-Length': '0'} 2023-07-29T00:24:46.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d1f75ac844fa8e2-0064c45c4e-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.707 DEBUG:boto:b'hiiqos-58obj000000000000000000111000falseobj000000000000000000112023-07-29T00:24:24.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:46.707 DEBUG:boto:path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:24:46.708 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:24:46.708 DEBUG:boto:Method: HEAD 2023-07-29T00:24:46.709 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000011 2023-07-29T00:24:46.709 DEBUG:boto:Data: 2023-07-29T00:24:46.710 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.710 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:46.710 DEBUG:boto:Port: 8001 2023-07-29T00:24:46.711 DEBUG:boto:Params: {} 2023-07-29T00:24:46.711 DEBUG:boto:Token: None 2023-07-29T00:24:46.712 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000011 2023-07-29T00:24:46.712 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6CdwTeWV+ZS6hpx3of6B/VkfhiE= 2023-07-29T00:24:46.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6CdwTeWV+ZS6hpx3of6B/VkfhiE=', 'Content-Length': '0'} 2023-07-29T00:24:46.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000246ce47ac0656391-0064c45c4e-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.715 DEBUG:boto:path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:24:46.715 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:24:46.715 DEBUG:boto:Method: HEAD 2023-07-29T00:24:46.716 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000011 2023-07-29T00:24:46.716 DEBUG:boto:Data: 2023-07-29T00:24:46.716 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.717 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:46.717 DEBUG:boto:Port: 8000 2023-07-29T00:24:46.718 DEBUG:boto:Params: {} 2023-07-29T00:24:46.718 DEBUG:boto:Token: None 2023-07-29T00:24:46.718 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000011 2023-07-29T00:24:46.719 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6CdwTeWV+ZS6hpx3of6B/VkfhiE= 2023-07-29T00:24:46.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6CdwTeWV+ZS6hpx3of6B/VkfhiE=', 'Content-Length': '0'} 2023-07-29T00:24:46.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069862479a5726c1a-0064c45c4e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-07-29T00:24:46.723 DEBUG:boto:path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:24:46.724 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:24:46.725 DEBUG:boto:Method: GET 2023-07-29T00:24:46.726 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000011 2023-07-29T00:24:46.727 DEBUG:boto:Data: 2023-07-29T00:24:46.728 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.729 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:46.730 DEBUG:boto:Port: 8001 2023-07-29T00:24:46.731 DEBUG:boto:Params: {} 2023-07-29T00:24:46.732 DEBUG:boto:Token: None 2023-07-29T00:24:46.733 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000011 2023-07-29T00:24:46.734 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SkbcMqLPHh6HmJi/ZF465oKoDXE= 2023-07-29T00:24:46.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SkbcMqLPHh6HmJi/ZF465oKoDXE=', 'Content-Length': '0'} 2023-07-29T00:24:46.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea6ddb4dfafe1125-0064c45c4e-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.739 DEBUG:boto:path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:24:46.740 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:24:46.741 DEBUG:boto:Method: GET 2023-07-29T00:24:46.742 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000011 2023-07-29T00:24:46.742 DEBUG:boto:Data: 2023-07-29T00:24:46.743 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.744 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:46.745 DEBUG:boto:Port: 8000 2023-07-29T00:24:46.746 DEBUG:boto:Params: {} 2023-07-29T00:24:46.747 DEBUG:boto:Token: None 2023-07-29T00:24:46.748 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000011 2023-07-29T00:24:46.749 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SkbcMqLPHh6HmJi/ZF465oKoDXE= 2023-07-29T00:24:46.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SkbcMqLPHh6HmJi/ZF465oKoDXE=', 'Content-Length': '0'} 2023-07-29T00:24:46.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045ccd5cffea08c51-0064c45c4e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.796 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:46.797 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:46.798 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000011 2023-07-29T00:24:46.799 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000011 2023-07-29T00:24:46.799 DEBUG:boto:Method: GET 2023-07-29T00:24:46.800 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000011 2023-07-29T00:24:46.801 DEBUG:boto:Data: 2023-07-29T00:24:46.801 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.802 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:46.802 DEBUG:boto:Port: 8001 2023-07-29T00:24:46.803 DEBUG:boto:Params: {} 2023-07-29T00:24:46.804 DEBUG:boto:Token: None 2023-07-29T00:24:46.805 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/ 2023-07-29T00:24:46.805 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TRm3KbIAOgxXRanghL6T0MIAtl0= 2023-07-29T00:24:46.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TRm3KbIAOgxXRanghL6T0MIAtl0=', 'Content-Length': '0'} 2023-07-29T00:24:46.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e84491213a1996eb-0064c45c4e-1b40-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.810 DEBUG:boto:b'hiiqos-58obj000000000000000000111000falseobj000000000000000000112023-07-29T00:24:24.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:46.811 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:46.811 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:46.812 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000011 2023-07-29T00:24:46.813 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000011 2023-07-29T00:24:46.813 DEBUG:boto:Method: GET 2023-07-29T00:24:46.814 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000011 2023-07-29T00:24:46.814 DEBUG:boto:Data: 2023-07-29T00:24:46.815 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.816 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:46.816 DEBUG:boto:Port: 8000 2023-07-29T00:24:46.817 DEBUG:boto:Params: {} 2023-07-29T00:24:46.818 DEBUG:boto:Token: None 2023-07-29T00:24:46.819 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/ 2023-07-29T00:24:46.819 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TRm3KbIAOgxXRanghL6T0MIAtl0= 2023-07-29T00:24:46.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TRm3KbIAOgxXRanghL6T0MIAtl0=', 'Content-Length': '0'} 2023-07-29T00:24:46.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018bd47aa0263102a-0064c45c4e-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.824 DEBUG:boto:b'hiiqos-58obj000000000000000000111000falseobj000000000000000000112023-07-29T00:24:24.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:46.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-07-29T00:24:46.826 DEBUG:boto:path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:24:46.826 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:24:46.827 DEBUG:boto:path=/hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:24:46.828 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:24:46.828 DEBUG:boto:Method: GET 2023-07-29T00:24:46.829 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:24:46.830 DEBUG:boto:Data: 2023-07-29T00:24:46.830 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.831 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:46.831 DEBUG:boto:Port: 8001 2023-07-29T00:24:46.832 DEBUG:boto:Params: {} 2023-07-29T00:24:46.833 DEBUG:boto:Token: None 2023-07-29T00:24:46.834 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:24:46.834 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FC4NBHbrhYZAEincSg87UiS5LK8= 2023-07-29T00:24:46.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FC4NBHbrhYZAEincSg87UiS5LK8=', 'Content-Length': '0'} 2023-07-29T00:24:46.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1d843e6d3463d82-0064c45c4e-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.838 DEBUG:boto:path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:24:46.839 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:24:46.839 DEBUG:boto:path=/hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:24:46.839 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:24:46.840 DEBUG:boto:Method: GET 2023-07-29T00:24:46.840 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:24:46.841 DEBUG:boto:Data: 2023-07-29T00:24:46.841 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.841 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:46.842 DEBUG:boto:Port: 8000 2023-07-29T00:24:46.842 DEBUG:boto:Params: {} 2023-07-29T00:24:46.843 DEBUG:boto:Token: None 2023-07-29T00:24:46.843 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:24:46.843 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FC4NBHbrhYZAEincSg87UiS5LK8= 2023-07-29T00:24:46.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FC4NBHbrhYZAEincSg87UiS5LK8=', 'Content-Length': '0'} 2023-07-29T00:24:46.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e962a0a66ebb6528-0064c45c4e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.847 DEBUG:boto:path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:24:46.848 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:24:46.849 DEBUG:boto:path=/hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:24:46.849 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:24:46.850 DEBUG:boto:Method: HEAD 2023-07-29T00:24:46.850 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:24:46.851 DEBUG:boto:Data: 2023-07-29T00:24:46.852 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.853 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:46.853 DEBUG:boto:Port: 8001 2023-07-29T00:24:46.854 DEBUG:boto:Params: {} 2023-07-29T00:24:46.855 DEBUG:boto:Token: None 2023-07-29T00:24:46.856 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:24:46.857 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OQbLiLgaEY4vbfERPx7tHdGogxo= 2023-07-29T00:24:46.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OQbLiLgaEY4vbfERPx7tHdGogxo=', 'Content-Length': '0'} 2023-07-29T00:24:46.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000330a8a2ea5c27497-0064c45c4e-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.860 DEBUG:boto:path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:24:46.861 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:24:46.861 DEBUG:boto:path=/hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:24:46.862 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:24:46.862 DEBUG:boto:Method: HEAD 2023-07-29T00:24:46.862 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:24:46.863 DEBUG:boto:Data: 2023-07-29T00:24:46.863 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.863 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:46.864 DEBUG:boto:Port: 8000 2023-07-29T00:24:46.864 DEBUG:boto:Params: {} 2023-07-29T00:24:46.864 DEBUG:boto:Token: None 2023-07-29T00:24:46.865 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:24:46.865 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OQbLiLgaEY4vbfERPx7tHdGogxo= 2023-07-29T00:24:46.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OQbLiLgaEY4vbfERPx7tHdGogxo=', 'Content-Length': '0'} 2023-07-29T00:24:46.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000991ab371eb8d9d8a-0064c45c4e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-07-29T00:24:46.868 DEBUG:boto:path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:24:46.869 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:24:46.869 DEBUG:boto:path=/hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:24:46.870 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:24:46.870 DEBUG:boto:Method: GET 2023-07-29T00:24:46.870 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:24:46.871 DEBUG:boto:Data: 2023-07-29T00:24:46.871 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.871 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:46.872 DEBUG:boto:Port: 8001 2023-07-29T00:24:46.872 DEBUG:boto:Params: {} 2023-07-29T00:24:46.872 DEBUG:boto:Token: None 2023-07-29T00:24:46.873 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:24:46.873 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FC4NBHbrhYZAEincSg87UiS5LK8= 2023-07-29T00:24:46.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FC4NBHbrhYZAEincSg87UiS5LK8=', 'Content-Length': '0'} 2023-07-29T00:24:46.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ff41759a4833709-0064c45c4e-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.876 DEBUG:boto:path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:24:46.877 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:24:46.877 DEBUG:boto:path=/hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:24:46.877 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:24:46.878 DEBUG:boto:Method: GET 2023-07-29T00:24:46.878 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:24:46.878 DEBUG:boto:Data: 2023-07-29T00:24:46.879 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.879 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:46.879 DEBUG:boto:Port: 8000 2023-07-29T00:24:46.880 DEBUG:boto:Params: {} 2023-07-29T00:24:46.880 DEBUG:boto:Token: None 2023-07-29T00:24:46.881 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:24:46.881 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FC4NBHbrhYZAEincSg87UiS5LK8= 2023-07-29T00:24:46.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FC4NBHbrhYZAEincSg87UiS5LK8=', 'Content-Length': '0'} 2023-07-29T00:24:46.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065cd7704914d6577-0064c45c4e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.884 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:46.884 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:46.885 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000012 2023-07-29T00:24:46.885 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000012 2023-07-29T00:24:46.885 DEBUG:boto:Method: GET 2023-07-29T00:24:46.886 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000012 2023-07-29T00:24:46.886 DEBUG:boto:Data: 2023-07-29T00:24:46.887 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.887 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:46.887 DEBUG:boto:Port: 8001 2023-07-29T00:24:46.888 DEBUG:boto:Params: {} 2023-07-29T00:24:46.888 DEBUG:boto:Token: None 2023-07-29T00:24:46.888 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/ 2023-07-29T00:24:46.889 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TRm3KbIAOgxXRanghL6T0MIAtl0= 2023-07-29T00:24:46.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TRm3KbIAOgxXRanghL6T0MIAtl0=', 'Content-Length': '0'} 2023-07-29T00:24:46.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fdfc84e3a3281b9-0064c45c4e-1b40-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.892 DEBUG:boto:b'hiiqos-58obj000000000000000000121000falseobj000000000000000000122023-07-29T00:24:24.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:46.893 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:46.893 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:46.894 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000012 2023-07-29T00:24:46.894 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000012 2023-07-29T00:24:46.895 DEBUG:boto:Method: GET 2023-07-29T00:24:46.895 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000012 2023-07-29T00:24:46.895 DEBUG:boto:Data: 2023-07-29T00:24:46.896 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.896 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:46.897 DEBUG:boto:Port: 8000 2023-07-29T00:24:46.897 DEBUG:boto:Params: {} 2023-07-29T00:24:46.897 DEBUG:boto:Token: None 2023-07-29T00:24:46.898 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/ 2023-07-29T00:24:46.898 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TRm3KbIAOgxXRanghL6T0MIAtl0= 2023-07-29T00:24:46.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TRm3KbIAOgxXRanghL6T0MIAtl0=', 'Content-Length': '0'} 2023-07-29T00:24:46.901 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000414e56c3770c26f2-0064c45c4e-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.901 DEBUG:boto:b'hiiqos-58obj000000000000000000121000falseobj000000000000000000122023-07-29T00:24:24.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:46.902 DEBUG:boto:path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:24:46.902 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:24:46.903 DEBUG:boto:Method: HEAD 2023-07-29T00:24:46.903 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000012 2023-07-29T00:24:46.904 DEBUG:boto:Data: 2023-07-29T00:24:46.904 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.904 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:46.905 DEBUG:boto:Port: 8001 2023-07-29T00:24:46.905 DEBUG:boto:Params: {} 2023-07-29T00:24:46.905 DEBUG:boto:Token: None 2023-07-29T00:24:46.906 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000012 2023-07-29T00:24:46.906 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ObesN1UK0tQdK68gP60jeHO/dTI= 2023-07-29T00:24:46.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ObesN1UK0tQdK68gP60jeHO/dTI=', 'Content-Length': '0'} 2023-07-29T00:24:46.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c582af5ef787a98b-0064c45c4e-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.909 DEBUG:boto:path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:24:46.909 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:24:46.910 DEBUG:boto:Method: HEAD 2023-07-29T00:24:46.910 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000012 2023-07-29T00:24:46.910 DEBUG:boto:Data: 2023-07-29T00:24:46.911 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.911 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:46.911 DEBUG:boto:Port: 8000 2023-07-29T00:24:46.912 DEBUG:boto:Params: {} 2023-07-29T00:24:46.912 DEBUG:boto:Token: None 2023-07-29T00:24:46.912 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000012 2023-07-29T00:24:46.913 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ObesN1UK0tQdK68gP60jeHO/dTI= 2023-07-29T00:24:46.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ObesN1UK0tQdK68gP60jeHO/dTI=', 'Content-Length': '0'} 2023-07-29T00:24:46.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040692bae6fc1f2f5-0064c45c4e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-07-29T00:24:46.916 DEBUG:boto:path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:24:46.916 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:24:46.917 DEBUG:boto:Method: GET 2023-07-29T00:24:46.917 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000012 2023-07-29T00:24:46.918 DEBUG:boto:Data: 2023-07-29T00:24:46.918 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.919 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:46.919 DEBUG:boto:Port: 8001 2023-07-29T00:24:46.919 DEBUG:boto:Params: {} 2023-07-29T00:24:46.920 DEBUG:boto:Token: None 2023-07-29T00:24:46.920 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000012 2023-07-29T00:24:46.921 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LKpZzufUsWkSTrVRqHMxyCFEC1s= 2023-07-29T00:24:46.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LKpZzufUsWkSTrVRqHMxyCFEC1s=', 'Content-Length': '0'} 2023-07-29T00:24:46.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084f00e06449373a2-0064c45c4e-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.964 DEBUG:boto:path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:24:46.965 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:24:46.965 DEBUG:boto:Method: GET 2023-07-29T00:24:46.966 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000012 2023-07-29T00:24:46.966 DEBUG:boto:Data: 2023-07-29T00:24:46.966 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.967 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:46.967 DEBUG:boto:Port: 8000 2023-07-29T00:24:46.967 DEBUG:boto:Params: {} 2023-07-29T00:24:46.968 DEBUG:boto:Token: None 2023-07-29T00:24:46.968 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000012 2023-07-29T00:24:46.969 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LKpZzufUsWkSTrVRqHMxyCFEC1s= 2023-07-29T00:24:46.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LKpZzufUsWkSTrVRqHMxyCFEC1s=', 'Content-Length': '0'} 2023-07-29T00:24:46.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b608ff60b94b3cf3-0064c45c4e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.972 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:46.972 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:46.972 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000012 2023-07-29T00:24:46.973 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000012 2023-07-29T00:24:46.973 DEBUG:boto:Method: GET 2023-07-29T00:24:46.974 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000012 2023-07-29T00:24:46.974 DEBUG:boto:Data: 2023-07-29T00:24:46.974 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.975 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:46.975 DEBUG:boto:Port: 8001 2023-07-29T00:24:46.975 DEBUG:boto:Params: {} 2023-07-29T00:24:46.976 DEBUG:boto:Token: None 2023-07-29T00:24:46.976 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/ 2023-07-29T00:24:46.977 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TRm3KbIAOgxXRanghL6T0MIAtl0= 2023-07-29T00:24:46.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TRm3KbIAOgxXRanghL6T0MIAtl0=', 'Content-Length': '0'} 2023-07-29T00:24:46.979 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fee2e9c24b39f14e-0064c45c4e-1b40-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.980 DEBUG:boto:b'hiiqos-58obj000000000000000000121000falseobj000000000000000000122023-07-29T00:24:24.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:46.980 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:46.981 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:46.981 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000012 2023-07-29T00:24:46.981 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000012 2023-07-29T00:24:46.982 DEBUG:boto:Method: GET 2023-07-29T00:24:46.982 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000012 2023-07-29T00:24:46.982 DEBUG:boto:Data: 2023-07-29T00:24:46.983 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.983 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:46.983 DEBUG:boto:Port: 8000 2023-07-29T00:24:46.984 DEBUG:boto:Params: {} 2023-07-29T00:24:46.984 DEBUG:boto:Token: None 2023-07-29T00:24:46.985 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/ 2023-07-29T00:24:46.985 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TRm3KbIAOgxXRanghL6T0MIAtl0= 2023-07-29T00:24:46.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TRm3KbIAOgxXRanghL6T0MIAtl0=', 'Content-Length': '0'} 2023-07-29T00:24:46.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acdaf6bb038fc18c-0064c45c4e-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.989 DEBUG:boto:b'hiiqos-58obj000000000000000000121000falseobj000000000000000000122023-07-29T00:24:24.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:46.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-07-29T00:24:46.990 DEBUG:boto:path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:24:46.990 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:24:46.990 DEBUG:boto:path=/hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:24:46.991 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:24:46.991 DEBUG:boto:Method: GET 2023-07-29T00:24:46.991 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:24:46.992 DEBUG:boto:Data: 2023-07-29T00:24:46.992 DEBUG:boto:Headers: {} 2023-07-29T00:24:46.992 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:46.993 DEBUG:boto:Port: 8001 2023-07-29T00:24:46.993 DEBUG:boto:Params: {} 2023-07-29T00:24:46.994 DEBUG:boto:Token: None 2023-07-29T00:24:46.994 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:46 GMT /hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:24:46.994 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qgPfzxr2TV3rYf8Qo17tfhI7g4w= 2023-07-29T00:24:46.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qgPfzxr2TV3rYf8Qo17tfhI7g4w=', 'Content-Length': '0'} 2023-07-29T00:24:46.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000084cdac0ba28074e-0064c45c4e-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:46.997 DEBUG:boto:path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:24:46.998 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:24:46.998 DEBUG:boto:path=/hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:24:46.999 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:24:46.999 DEBUG:boto:Method: GET 2023-07-29T00:24:46.999 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:24:47.000 DEBUG:boto:Data: 2023-07-29T00:24:47.000 DEBUG:boto:Headers: {} 2023-07-29T00:24:47.000 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:47.001 DEBUG:boto:Port: 8000 2023-07-29T00:24:47.001 DEBUG:boto:Params: {} 2023-07-29T00:24:47.001 DEBUG:boto:Token: None 2023-07-29T00:24:47.002 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:47 GMT /hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:24:47.002 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J9NASkQBHNLxicy4x5/UBlg3Or4= 2023-07-29T00:24:47.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J9NASkQBHNLxicy4x5/UBlg3Or4=', 'Content-Length': '0'} 2023-07-29T00:24:47.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008179ec6ed140479-0064c45c4e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:47.052 DEBUG:boto:path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:24:47.053 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:24:47.053 DEBUG:boto:path=/hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:24:47.053 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:24:47.054 DEBUG:boto:Method: HEAD 2023-07-29T00:24:47.054 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:24:47.054 DEBUG:boto:Data: 2023-07-29T00:24:47.055 DEBUG:boto:Headers: {} 2023-07-29T00:24:47.055 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:47.056 DEBUG:boto:Port: 8001 2023-07-29T00:24:47.057 DEBUG:boto:Params: {} 2023-07-29T00:24:47.057 DEBUG:boto:Token: None 2023-07-29T00:24:47.058 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:47 GMT /hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:24:47.058 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JuovaIG/gQJZcsemOnhZ0mSLv4M= 2023-07-29T00:24:47.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JuovaIG/gQJZcsemOnhZ0mSLv4M=', 'Content-Length': '0'} 2023-07-29T00:24:47.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006656d2bc02f1948-0064c45c4f-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:47.061 DEBUG:boto:path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:24:47.062 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:24:47.062 DEBUG:boto:path=/hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:24:47.063 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:24:47.063 DEBUG:boto:Method: HEAD 2023-07-29T00:24:47.063 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:24:47.064 DEBUG:boto:Data: 2023-07-29T00:24:47.064 DEBUG:boto:Headers: {} 2023-07-29T00:24:47.064 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:47.065 DEBUG:boto:Port: 8000 2023-07-29T00:24:47.065 DEBUG:boto:Params: {} 2023-07-29T00:24:47.066 DEBUG:boto:Token: None 2023-07-29T00:24:47.066 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:47 GMT /hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:24:47.066 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JuovaIG/gQJZcsemOnhZ0mSLv4M= 2023-07-29T00:24:47.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JuovaIG/gQJZcsemOnhZ0mSLv4M=', 'Content-Length': '0'} 2023-07-29T00:24:47.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057295caf5c63f93b-0064c45c4f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:47.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-07-29T00:24:47.069 DEBUG:boto:path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:24:47.070 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:24:47.070 DEBUG:boto:path=/hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:24:47.071 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:24:47.071 DEBUG:boto:Method: GET 2023-07-29T00:24:47.071 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:24:47.072 DEBUG:boto:Data: 2023-07-29T00:24:47.072 DEBUG:boto:Headers: {} 2023-07-29T00:24:47.073 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:47.073 DEBUG:boto:Port: 8001 2023-07-29T00:24:47.073 DEBUG:boto:Params: {} 2023-07-29T00:24:47.074 DEBUG:boto:Token: None 2023-07-29T00:24:47.074 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:47 GMT /hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:24:47.075 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J9NASkQBHNLxicy4x5/UBlg3Or4= 2023-07-29T00:24:47.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J9NASkQBHNLxicy4x5/UBlg3Or4=', 'Content-Length': '0'} 2023-07-29T00:24:47.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000824ac6046cf39a57-0064c45c4f-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:47.077 DEBUG:boto:path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:24:47.078 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:24:47.078 DEBUG:boto:path=/hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:24:47.078 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:24:47.079 DEBUG:boto:Method: GET 2023-07-29T00:24:47.079 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:24:47.080 DEBUG:boto:Data: 2023-07-29T00:24:47.080 DEBUG:boto:Headers: {} 2023-07-29T00:24:47.080 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:47.081 DEBUG:boto:Port: 8000 2023-07-29T00:24:47.081 DEBUG:boto:Params: {} 2023-07-29T00:24:47.081 DEBUG:boto:Token: None 2023-07-29T00:24:47.082 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:47 GMT /hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:24:47.082 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J9NASkQBHNLxicy4x5/UBlg3Or4= 2023-07-29T00:24:47.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J9NASkQBHNLxicy4x5/UBlg3Or4=', 'Content-Length': '0'} 2023-07-29T00:24:47.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da8add53bb6966f0-0064c45c4f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:47.085 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:47.085 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:47.086 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000013 2023-07-29T00:24:47.086 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000013 2023-07-29T00:24:47.086 DEBUG:boto:Method: GET 2023-07-29T00:24:47.087 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000013 2023-07-29T00:24:47.087 DEBUG:boto:Data: 2023-07-29T00:24:47.087 DEBUG:boto:Headers: {} 2023-07-29T00:24:47.088 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:47.088 DEBUG:boto:Port: 8001 2023-07-29T00:24:47.088 DEBUG:boto:Params: {} 2023-07-29T00:24:47.089 DEBUG:boto:Token: None 2023-07-29T00:24:47.090 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:47 GMT /hiiqos-58/ 2023-07-29T00:24:47.091 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MGrwEsgtfJ+S8SWzygHsT8pP87M= 2023-07-29T00:24:47.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MGrwEsgtfJ+S8SWzygHsT8pP87M=', 'Content-Length': '0'} 2023-07-29T00:24:47.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000552c2a443866a343-0064c45c4f-1b40-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:47.136 DEBUG:boto:b'hiiqos-58obj000000000000000000131000falseobj000000000000000000132023-07-29T00:24:24.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:47.137 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:47.138 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:47.138 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000013 2023-07-29T00:24:47.138 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000013 2023-07-29T00:24:47.139 DEBUG:boto:Method: GET 2023-07-29T00:24:47.139 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000013 2023-07-29T00:24:47.139 DEBUG:boto:Data: 2023-07-29T00:24:47.140 DEBUG:boto:Headers: {} 2023-07-29T00:24:47.140 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:47.141 DEBUG:boto:Port: 8000 2023-07-29T00:24:47.141 DEBUG:boto:Params: {} 2023-07-29T00:24:47.141 DEBUG:boto:Token: None 2023-07-29T00:24:47.142 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:47 GMT /hiiqos-58/ 2023-07-29T00:24:47.142 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MGrwEsgtfJ+S8SWzygHsT8pP87M= 2023-07-29T00:24:47.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MGrwEsgtfJ+S8SWzygHsT8pP87M=', 'Content-Length': '0'} 2023-07-29T00:24:47.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000686a5de09ccab7bd-0064c45c4f-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:47.145 DEBUG:boto:b'hiiqos-58obj000000000000000000131000falseobj000000000000000000132023-07-29T00:24:24.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:47.146 DEBUG:boto:path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:24:47.146 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:24:47.147 DEBUG:boto:Method: HEAD 2023-07-29T00:24:47.147 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000013 2023-07-29T00:24:47.147 DEBUG:boto:Data: 2023-07-29T00:24:47.148 DEBUG:boto:Headers: {} 2023-07-29T00:24:47.148 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:47.148 DEBUG:boto:Port: 8001 2023-07-29T00:24:47.149 DEBUG:boto:Params: {} 2023-07-29T00:24:47.150 DEBUG:boto:Token: None 2023-07-29T00:24:47.150 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:47 GMT /hiiqos-58/obj00000000000000000013 2023-07-29T00:24:47.151 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:a2K/RW4gL0oZBzFdC5FiAg9emi0= 2023-07-29T00:24:47.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:a2K/RW4gL0oZBzFdC5FiAg9emi0=', 'Content-Length': '0'} 2023-07-29T00:24:47.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afd5eec44d0853da-0064c45c4f-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:47.153 DEBUG:boto:path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:24:47.154 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:24:47.154 DEBUG:boto:Method: HEAD 2023-07-29T00:24:47.155 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000013 2023-07-29T00:24:47.155 DEBUG:boto:Data: 2023-07-29T00:24:47.155 DEBUG:boto:Headers: {} 2023-07-29T00:24:47.156 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:47.156 DEBUG:boto:Port: 8000 2023-07-29T00:24:47.156 DEBUG:boto:Params: {} 2023-07-29T00:24:47.157 DEBUG:boto:Token: None 2023-07-29T00:24:47.157 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:47 GMT /hiiqos-58/obj00000000000000000013 2023-07-29T00:24:47.158 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:a2K/RW4gL0oZBzFdC5FiAg9emi0= 2023-07-29T00:24:47.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:a2K/RW4gL0oZBzFdC5FiAg9emi0=', 'Content-Length': '0'} 2023-07-29T00:24:47.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a3d99bb7b79aeda-0064c45c4f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:47.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-07-29T00:24:47.161 DEBUG:boto:path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:24:47.162 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:24:47.162 DEBUG:boto:Method: GET 2023-07-29T00:24:47.163 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000013 2023-07-29T00:24:47.163 DEBUG:boto:Data: 2023-07-29T00:24:47.163 DEBUG:boto:Headers: {} 2023-07-29T00:24:47.164 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:47.164 DEBUG:boto:Port: 8001 2023-07-29T00:24:47.165 DEBUG:boto:Params: {} 2023-07-29T00:24:47.165 DEBUG:boto:Token: None 2023-07-29T00:24:47.166 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:47 GMT /hiiqos-58/obj00000000000000000013 2023-07-29T00:24:47.166 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Op+Uybl3bYxZa1X05Jw8OeHbrqU= 2023-07-29T00:24:47.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Op+Uybl3bYxZa1X05Jw8OeHbrqU=', 'Content-Length': '0'} 2023-07-29T00:24:47.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079e9373794ad21d5-0064c45c4f-1b40-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:47.170 DEBUG:boto:path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:24:47.171 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:24:47.171 DEBUG:boto:Method: GET 2023-07-29T00:24:47.171 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000013 2023-07-29T00:24:47.172 DEBUG:boto:Data: 2023-07-29T00:24:47.173 DEBUG:boto:Headers: {} 2023-07-29T00:24:47.173 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:47.174 DEBUG:boto:Port: 8000 2023-07-29T00:24:47.174 DEBUG:boto:Params: {} 2023-07-29T00:24:47.174 DEBUG:boto:Token: None 2023-07-29T00:24:47.175 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:47 GMT /hiiqos-58/obj00000000000000000013 2023-07-29T00:24:47.175 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Op+Uybl3bYxZa1X05Jw8OeHbrqU= 2023-07-29T00:24:47.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Op+Uybl3bYxZa1X05Jw8OeHbrqU=', 'Content-Length': '0'} 2023-07-29T00:24:47.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fe8a2b9a4844816-0064c45c4f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:24:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:47.220 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:47.221 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:47.221 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000013 2023-07-29T00:24:47.222 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000013 2023-07-29T00:24:47.222 DEBUG:boto:Method: GET 2023-07-29T00:24:47.222 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000013 2023-07-29T00:24:47.223 DEBUG:boto:Data: 2023-07-29T00:24:47.223 DEBUG:boto:Headers: {} 2023-07-29T00:24:47.223 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:24:47.224 DEBUG:boto:Port: 8001 2023-07-29T00:24:47.224 DEBUG:boto:Params: {} 2023-07-29T00:24:47.225 DEBUG:boto:Token: None 2023-07-29T00:24:47.225 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:47 GMT /hiiqos-58/ 2023-07-29T00:24:47.226 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MGrwEsgtfJ+S8SWzygHsT8pP87M= 2023-07-29T00:24:47.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MGrwEsgtfJ+S8SWzygHsT8pP87M=', 'Content-Length': '0'} 2023-07-29T00:24:47.228 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddd50d15a54b7e64-0064c45c4f-1b40-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:47.229 DEBUG:boto:b'hiiqos-58obj000000000000000000131000falseobj000000000000000000132023-07-29T00:24:24.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:47.230 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:47.230 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:47.230 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000013 2023-07-29T00:24:47.231 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000013 2023-07-29T00:24:47.231 DEBUG:boto:Method: GET 2023-07-29T00:24:47.232 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000013 2023-07-29T00:24:47.232 DEBUG:boto:Data: 2023-07-29T00:24:47.232 DEBUG:boto:Headers: {} 2023-07-29T00:24:47.233 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:47.233 DEBUG:boto:Port: 8000 2023-07-29T00:24:47.234 DEBUG:boto:Params: {} 2023-07-29T00:24:47.234 DEBUG:boto:Token: None 2023-07-29T00:24:47.234 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:24:47 GMT /hiiqos-58/ 2023-07-29T00:24:47.235 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MGrwEsgtfJ+S8SWzygHsT8pP87M= 2023-07-29T00:24:47.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MGrwEsgtfJ+S8SWzygHsT8pP87M=', 'Content-Length': '0'} 2023-07-29T00:24:47.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c46fa9e6ec3b6a1b-0064c45c4f-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:24:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:47.239 DEBUG:boto:b'hiiqos-58obj000000000000000000131000falseobj000000000000000000132023-07-29T00:24:24.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:24:47.239 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-58 zones={a2, a1} 2023-07-29T00:24:47.240 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-07-29T00:24:47.240 INFO:teuthology.orchestra.run:waiting for 300 2023-07-29T00:24:47.241 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:24:47.189+0000 7f05629f5640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 125391) UID: 0 2023-07-29T00:24:47.242 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:24:47.189+0000 7f0568a34880 -1 shutting down 2023-07-29T00:24:50.222 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-07-29T00:24:53.339 INFO:tasks.rgw.c1.client.1:Stopped 2023-07-29T00:24:53.339 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-58', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:24:53.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-58', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:24:53.340 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-58 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:24:53.396 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:24:53.396 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:24:53.439 INFO:teuthology.orchestra.run.smithi101.stdout:tenant: 2023-07-29T00:24:53.440 INFO:teuthology.orchestra.run.smithi101.stdout:bucket name: hiiqos-58 2023-07-29T00:24:53.443 INFO:teuthology.orchestra.run.smithi101.stdout:total entries: 10 2023-07-29T00:24:53.481 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:24:53.482 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:24:53.482 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:24:53.530 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:24:53.530 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:24:53.571 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:24:53.571 INFO:teuthology.orchestra.run.smithi101.stdout: "objs": [ 2023-07-29T00:24:53.571 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000007", 2023-07-29T00:24:53.571 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000016", 2023-07-29T00:24:53.572 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000024", 2023-07-29T00:24:53.572 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000020", 2023-07-29T00:24:53.572 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000028", 2023-07-29T00:24:53.572 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000000", 2023-07-29T00:24:53.572 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000032", 2023-07-29T00:24:53.572 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000001", 2023-07-29T00:24:53.573 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000010", 2023-07-29T00:24:53.573 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000019", 2023-07-29T00:24:53.573 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000013", 2023-07-29T00:24:53.573 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000021", 2023-07-29T00:24:53.573 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000030", 2023-07-29T00:24:53.573 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000026", 2023-07-29T00:24:53.573 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000033", 2023-07-29T00:24:53.574 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000004", 2023-07-29T00:24:53.574 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000076" 2023-07-29T00:24:53.574 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:24:53.574 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:24:53.575 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:53.576 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:53.576 DEBUG:boto:Method: HEAD 2023-07-29T00:24:53.576 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:53.577 DEBUG:boto:Data: 2023-07-29T00:24:53.577 DEBUG:boto:Headers: {} 2023-07-29T00:24:53.577 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:53.578 DEBUG:boto:Port: 8000 2023-07-29T00:24:53.578 DEBUG:boto:Params: {} 2023-07-29T00:24:53.578 DEBUG:boto:Token: None 2023-07-29T00:24:53.579 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:53 GMT /hiiqos-58/ 2023-07-29T00:24:53.579 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/2x4mBUU98gvOQOLwUFLrPADylM= 2023-07-29T00:24:53.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/2x4mBUU98gvOQOLwUFLrPADylM=', 'Content-Length': '0'} 2023-07-29T00:24:53.582 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006119a31c2fe225ef-0064c45c55-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:53.583 DEBUG:boto:path=/hiiqos-58/obj00000000000000000028 2023-07-29T00:24:53.583 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000028 2023-07-29T00:24:53.584 DEBUG:boto:Method: PUT 2023-07-29T00:24:53.584 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000028 2023-07-29T00:24:53.588 DEBUG:boto:Data: 2023-07-29T00:24:53.588 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:53.588 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:53.589 DEBUG:boto:Port: 8000 2023-07-29T00:24:53.589 DEBUG:boto:Params: {} 2023-07-29T00:24:53.589 DEBUG:boto:Token: None 2023-07-29T00:24:53.590 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:53 GMT /hiiqos-58/obj00000000000000000028 2023-07-29T00:24:53.590 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WIXsSPwdCHhDD4VniRxMDxbvHMU= 2023-07-29T00:24:53.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WIXsSPwdCHhDD4VniRxMDxbvHMU='} 2023-07-29T00:24:53.595 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000722b13a28b8191d2-0064c45c55-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:53.596 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:53.596 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:53.597 DEBUG:boto:Method: HEAD 2023-07-29T00:24:53.597 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:53.598 DEBUG:boto:Data: 2023-07-29T00:24:53.598 DEBUG:boto:Headers: {} 2023-07-29T00:24:53.598 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:53.599 DEBUG:boto:Port: 8000 2023-07-29T00:24:53.599 DEBUG:boto:Params: {} 2023-07-29T00:24:53.599 DEBUG:boto:Token: None 2023-07-29T00:24:53.600 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:53 GMT /hiiqos-58/ 2023-07-29T00:24:53.600 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/2x4mBUU98gvOQOLwUFLrPADylM= 2023-07-29T00:24:53.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/2x4mBUU98gvOQOLwUFLrPADylM=', 'Content-Length': '0'} 2023-07-29T00:24:53.603 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000073d74b28a7b5c2c-0064c45c55-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:53.604 DEBUG:boto:path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:24:53.604 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:24:53.605 DEBUG:boto:Method: PUT 2023-07-29T00:24:53.605 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000000 2023-07-29T00:24:53.606 DEBUG:boto:Data: 2023-07-29T00:24:53.606 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:53.606 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:53.607 DEBUG:boto:Port: 8000 2023-07-29T00:24:53.607 DEBUG:boto:Params: {} 2023-07-29T00:24:53.607 DEBUG:boto:Token: None 2023-07-29T00:24:53.608 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:53 GMT /hiiqos-58/obj00000000000000000000 2023-07-29T00:24:53.608 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1VvPW9sALLT9xtfC6Y4tRWzMhSQ= 2023-07-29T00:24:53.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1VvPW9sALLT9xtfC6Y4tRWzMhSQ='} 2023-07-29T00:24:53.652 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000076382fedecc6664d-0064c45c55-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:53.653 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:53.654 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:53.654 DEBUG:boto:Method: HEAD 2023-07-29T00:24:53.654 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:53.655 DEBUG:boto:Data: 2023-07-29T00:24:53.655 DEBUG:boto:Headers: {} 2023-07-29T00:24:53.656 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:53.656 DEBUG:boto:Port: 8000 2023-07-29T00:24:53.656 DEBUG:boto:Params: {} 2023-07-29T00:24:53.657 DEBUG:boto:Token: None 2023-07-29T00:24:53.657 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:53 GMT /hiiqos-58/ 2023-07-29T00:24:53.658 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/2x4mBUU98gvOQOLwUFLrPADylM= 2023-07-29T00:24:53.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/2x4mBUU98gvOQOLwUFLrPADylM=', 'Content-Length': '0'} 2023-07-29T00:24:53.661 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007d1d6fc9bab876f8-0064c45c55-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:53.661 DEBUG:boto:path=/hiiqos-58/obj00000000000000000026 2023-07-29T00:24:53.662 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000026 2023-07-29T00:24:53.662 DEBUG:boto:Method: PUT 2023-07-29T00:24:53.663 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000026 2023-07-29T00:24:53.663 DEBUG:boto:Data: 2023-07-29T00:24:53.664 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:53.664 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:53.665 DEBUG:boto:Port: 8000 2023-07-29T00:24:53.665 DEBUG:boto:Params: {} 2023-07-29T00:24:53.666 DEBUG:boto:Token: None 2023-07-29T00:24:53.666 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:53 GMT /hiiqos-58/obj00000000000000000026 2023-07-29T00:24:53.666 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gPCclOpnDAGhu4zH75nQyOH/+iE= 2023-07-29T00:24:53.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gPCclOpnDAGhu4zH75nQyOH/+iE='} 2023-07-29T00:24:53.671 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000096d0269d1befe69e-0064c45c55-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:53.672 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:53.673 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:53.673 DEBUG:boto:Method: HEAD 2023-07-29T00:24:53.674 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:53.674 DEBUG:boto:Data: 2023-07-29T00:24:53.674 DEBUG:boto:Headers: {} 2023-07-29T00:24:53.675 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:53.675 DEBUG:boto:Port: 8000 2023-07-29T00:24:53.675 DEBUG:boto:Params: {} 2023-07-29T00:24:53.676 DEBUG:boto:Token: None 2023-07-29T00:24:53.677 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:53 GMT /hiiqos-58/ 2023-07-29T00:24:53.678 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/2x4mBUU98gvOQOLwUFLrPADylM= 2023-07-29T00:24:53.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/2x4mBUU98gvOQOLwUFLrPADylM=', 'Content-Length': '0'} 2023-07-29T00:24:53.680 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d0689811c4109177-0064c45c55-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:53.681 DEBUG:boto:path=/hiiqos-58/obj00000000000000000016 2023-07-29T00:24:53.682 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000016 2023-07-29T00:24:53.682 DEBUG:boto:Method: PUT 2023-07-29T00:24:53.683 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000016 2023-07-29T00:24:53.683 DEBUG:boto:Data: 2023-07-29T00:24:53.684 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:53.684 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:53.685 DEBUG:boto:Port: 8000 2023-07-29T00:24:53.685 DEBUG:boto:Params: {} 2023-07-29T00:24:53.686 DEBUG:boto:Token: None 2023-07-29T00:24:53.686 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:53 GMT /hiiqos-58/obj00000000000000000016 2023-07-29T00:24:53.687 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rLWQuXEe5jCoAe9EWxEOeaZXt0w= 2023-07-29T00:24:53.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rLWQuXEe5jCoAe9EWxEOeaZXt0w='} 2023-07-29T00:24:53.737 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d1be468bff1d7019-0064c45c55-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:53.738 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:53.739 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:53.739 DEBUG:boto:Method: HEAD 2023-07-29T00:24:53.741 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:53.741 DEBUG:boto:Data: 2023-07-29T00:24:53.742 DEBUG:boto:Headers: {} 2023-07-29T00:24:53.742 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:53.742 DEBUG:boto:Port: 8000 2023-07-29T00:24:53.743 DEBUG:boto:Params: {} 2023-07-29T00:24:53.743 DEBUG:boto:Token: None 2023-07-29T00:24:53.746 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:53 GMT /hiiqos-58/ 2023-07-29T00:24:53.747 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/2x4mBUU98gvOQOLwUFLrPADylM= 2023-07-29T00:24:53.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/2x4mBUU98gvOQOLwUFLrPADylM=', 'Content-Length': '0'} 2023-07-29T00:24:53.750 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a7828c42b9854bbb-0064c45c55-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:53.751 DEBUG:boto:path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:24:53.752 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:24:53.753 DEBUG:boto:Method: PUT 2023-07-29T00:24:53.754 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000004 2023-07-29T00:24:53.754 DEBUG:boto:Data: 2023-07-29T00:24:53.755 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:53.756 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:53.757 DEBUG:boto:Port: 8000 2023-07-29T00:24:53.757 DEBUG:boto:Params: {} 2023-07-29T00:24:53.758 DEBUG:boto:Token: None 2023-07-29T00:24:53.758 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:53 GMT /hiiqos-58/obj00000000000000000004 2023-07-29T00:24:53.759 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:g/Hz7/z9P0cGFqJrq/CewQ5SUlk= 2023-07-29T00:24:53.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:g/Hz7/z9P0cGFqJrq/CewQ5SUlk='} 2023-07-29T00:24:53.765 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002bff25e74826da32-0064c45c55-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:53.766 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:53.766 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:53.767 DEBUG:boto:Method: HEAD 2023-07-29T00:24:53.767 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:53.767 DEBUG:boto:Data: 2023-07-29T00:24:53.768 DEBUG:boto:Headers: {} 2023-07-29T00:24:53.768 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:53.768 DEBUG:boto:Port: 8000 2023-07-29T00:24:53.769 DEBUG:boto:Params: {} 2023-07-29T00:24:53.769 DEBUG:boto:Token: None 2023-07-29T00:24:53.770 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:53 GMT /hiiqos-58/ 2023-07-29T00:24:53.770 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/2x4mBUU98gvOQOLwUFLrPADylM= 2023-07-29T00:24:53.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/2x4mBUU98gvOQOLwUFLrPADylM=', 'Content-Length': '0'} 2023-07-29T00:24:53.773 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d8e1ed484dbe1f69-0064c45c55-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:53.774 DEBUG:boto:path=/hiiqos-58/obj00000000000000000024 2023-07-29T00:24:53.774 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000024 2023-07-29T00:24:53.774 DEBUG:boto:Method: PUT 2023-07-29T00:24:53.775 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000024 2023-07-29T00:24:53.775 DEBUG:boto:Data: 2023-07-29T00:24:53.776 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:53.776 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:53.776 DEBUG:boto:Port: 8000 2023-07-29T00:24:53.777 DEBUG:boto:Params: {} 2023-07-29T00:24:53.777 DEBUG:boto:Token: None 2023-07-29T00:24:53.777 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:53 GMT /hiiqos-58/obj00000000000000000024 2023-07-29T00:24:53.778 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hXG65279AocgI0Q+2WBIQ0g+Bek= 2023-07-29T00:24:53.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hXG65279AocgI0Q+2WBIQ0g+Bek='} 2023-07-29T00:24:53.820 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000085136e35e9059001-0064c45c55-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:53.821 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:53.821 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:53.822 DEBUG:boto:Method: HEAD 2023-07-29T00:24:53.822 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:53.823 DEBUG:boto:Data: 2023-07-29T00:24:53.823 DEBUG:boto:Headers: {} 2023-07-29T00:24:53.823 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:53.824 DEBUG:boto:Port: 8000 2023-07-29T00:24:53.824 DEBUG:boto:Params: {} 2023-07-29T00:24:53.825 DEBUG:boto:Token: None 2023-07-29T00:24:53.825 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:53 GMT /hiiqos-58/ 2023-07-29T00:24:53.826 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/2x4mBUU98gvOQOLwUFLrPADylM= 2023-07-29T00:24:53.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/2x4mBUU98gvOQOLwUFLrPADylM=', 'Content-Length': '0'} 2023-07-29T00:24:53.829 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001b0acf0ad912d791-0064c45c55-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:53.829 DEBUG:boto:path=/hiiqos-58/obj00000000000000000019 2023-07-29T00:24:53.830 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000019 2023-07-29T00:24:53.830 DEBUG:boto:Method: PUT 2023-07-29T00:24:53.830 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000019 2023-07-29T00:24:53.831 DEBUG:boto:Data: 2023-07-29T00:24:53.831 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:53.831 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:53.832 DEBUG:boto:Port: 8000 2023-07-29T00:24:53.832 DEBUG:boto:Params: {} 2023-07-29T00:24:53.833 DEBUG:boto:Token: None 2023-07-29T00:24:53.833 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:53 GMT /hiiqos-58/obj00000000000000000019 2023-07-29T00:24:53.833 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:r623SEq9cq6Yg8gAGtG0t6INz80= 2023-07-29T00:24:53.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:r623SEq9cq6Yg8gAGtG0t6INz80='} 2023-07-29T00:24:53.839 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004f94014a3d72959c-0064c45c55-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:53.839 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:53.840 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:53.840 DEBUG:boto:Method: HEAD 2023-07-29T00:24:53.840 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:53.841 DEBUG:boto:Data: 2023-07-29T00:24:53.841 DEBUG:boto:Headers: {} 2023-07-29T00:24:53.841 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:53.842 DEBUG:boto:Port: 8000 2023-07-29T00:24:53.842 DEBUG:boto:Params: {} 2023-07-29T00:24:53.843 DEBUG:boto:Token: None 2023-07-29T00:24:53.843 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:53 GMT /hiiqos-58/ 2023-07-29T00:24:53.843 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/2x4mBUU98gvOQOLwUFLrPADylM= 2023-07-29T00:24:53.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/2x4mBUU98gvOQOLwUFLrPADylM=', 'Content-Length': '0'} 2023-07-29T00:24:53.847 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006bac196bb0c31242-0064c45c55-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:53.848 DEBUG:boto:path=/hiiqos-58/obj00000000000000000007 2023-07-29T00:24:53.848 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000007 2023-07-29T00:24:53.849 DEBUG:boto:Method: PUT 2023-07-29T00:24:53.849 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000007 2023-07-29T00:24:53.849 DEBUG:boto:Data: 2023-07-29T00:24:53.850 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:53.850 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:53.851 DEBUG:boto:Port: 8000 2023-07-29T00:24:53.851 DEBUG:boto:Params: {} 2023-07-29T00:24:53.851 DEBUG:boto:Token: None 2023-07-29T00:24:53.852 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:53 GMT /hiiqos-58/obj00000000000000000007 2023-07-29T00:24:53.852 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I9JUP6pcgLgkZ/udSTW0uMUi04E= 2023-07-29T00:24:53.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I9JUP6pcgLgkZ/udSTW0uMUi04E='} 2023-07-29T00:24:53.896 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aaa4796cfdb09cbb-0064c45c55-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:53.897 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:53.897 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:53.898 DEBUG:boto:Method: HEAD 2023-07-29T00:24:53.898 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:53.898 DEBUG:boto:Data: 2023-07-29T00:24:53.899 DEBUG:boto:Headers: {} 2023-07-29T00:24:53.899 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:53.899 DEBUG:boto:Port: 8000 2023-07-29T00:24:53.900 DEBUG:boto:Params: {} 2023-07-29T00:24:53.900 DEBUG:boto:Token: None 2023-07-29T00:24:53.901 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:53 GMT /hiiqos-58/ 2023-07-29T00:24:53.901 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/2x4mBUU98gvOQOLwUFLrPADylM= 2023-07-29T00:24:53.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/2x4mBUU98gvOQOLwUFLrPADylM=', 'Content-Length': '0'} 2023-07-29T00:24:53.904 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000310307ac37cdf149-0064c45c55-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:53.904 DEBUG:boto:path=/hiiqos-58/obj00000000000000000030 2023-07-29T00:24:53.905 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000030 2023-07-29T00:24:53.905 DEBUG:boto:Method: PUT 2023-07-29T00:24:53.906 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000030 2023-07-29T00:24:53.906 DEBUG:boto:Data: 2023-07-29T00:24:53.906 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:53.907 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:53.907 DEBUG:boto:Port: 8000 2023-07-29T00:24:53.907 DEBUG:boto:Params: {} 2023-07-29T00:24:53.908 DEBUG:boto:Token: None 2023-07-29T00:24:53.908 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:53 GMT /hiiqos-58/obj00000000000000000030 2023-07-29T00:24:53.909 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ezjiQYegsot1yid453cdiKFox9U= 2023-07-29T00:24:53.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ezjiQYegsot1yid453cdiKFox9U='} 2023-07-29T00:24:53.914 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002c8986693f894654-0064c45c55-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:53.914 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:53.915 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:53.915 DEBUG:boto:Method: HEAD 2023-07-29T00:24:53.915 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:53.916 DEBUG:boto:Data: 2023-07-29T00:24:53.916 DEBUG:boto:Headers: {} 2023-07-29T00:24:53.916 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:53.917 DEBUG:boto:Port: 8000 2023-07-29T00:24:53.917 DEBUG:boto:Params: {} 2023-07-29T00:24:53.917 DEBUG:boto:Token: None 2023-07-29T00:24:53.918 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:53 GMT /hiiqos-58/ 2023-07-29T00:24:53.918 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/2x4mBUU98gvOQOLwUFLrPADylM= 2023-07-29T00:24:53.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/2x4mBUU98gvOQOLwUFLrPADylM=', 'Content-Length': '0'} 2023-07-29T00:24:53.921 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003a570e9869a300b8-0064c45c55-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:53.922 DEBUG:boto:path=/hiiqos-58/obj00000000000000000032 2023-07-29T00:24:53.922 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000032 2023-07-29T00:24:53.923 DEBUG:boto:Method: PUT 2023-07-29T00:24:53.923 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000032 2023-07-29T00:24:53.923 DEBUG:boto:Data: 2023-07-29T00:24:53.924 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:53.924 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:53.924 DEBUG:boto:Port: 8000 2023-07-29T00:24:53.925 DEBUG:boto:Params: {} 2023-07-29T00:24:53.925 DEBUG:boto:Token: None 2023-07-29T00:24:53.925 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:53 GMT /hiiqos-58/obj00000000000000000032 2023-07-29T00:24:53.926 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ofDipKKpdYNVnRd1dXyOSddRufc= 2023-07-29T00:24:53.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ofDipKKpdYNVnRd1dXyOSddRufc='} 2023-07-29T00:24:53.972 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000626ae0dc23f740c9-0064c45c55-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:53.973 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:53.973 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:53.974 DEBUG:boto:Method: HEAD 2023-07-29T00:24:53.974 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:53.975 DEBUG:boto:Data: 2023-07-29T00:24:53.975 DEBUG:boto:Headers: {} 2023-07-29T00:24:53.975 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:53.975 DEBUG:boto:Port: 8000 2023-07-29T00:24:53.976 DEBUG:boto:Params: {} 2023-07-29T00:24:53.976 DEBUG:boto:Token: None 2023-07-29T00:24:53.977 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:53 GMT /hiiqos-58/ 2023-07-29T00:24:53.977 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/2x4mBUU98gvOQOLwUFLrPADylM= 2023-07-29T00:24:53.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/2x4mBUU98gvOQOLwUFLrPADylM=', 'Content-Length': '0'} 2023-07-29T00:24:53.980 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000043ddd28aa0dd07be-0064c45c55-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:53.981 DEBUG:boto:path=/hiiqos-58/obj00000000000000000020 2023-07-29T00:24:53.981 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000020 2023-07-29T00:24:53.981 DEBUG:boto:Method: PUT 2023-07-29T00:24:53.982 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000020 2023-07-29T00:24:53.982 DEBUG:boto:Data: 2023-07-29T00:24:53.984 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:53.984 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:53.985 DEBUG:boto:Port: 8000 2023-07-29T00:24:53.985 DEBUG:boto:Params: {} 2023-07-29T00:24:53.985 DEBUG:boto:Token: None 2023-07-29T00:24:53.986 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:53 GMT /hiiqos-58/obj00000000000000000020 2023-07-29T00:24:53.986 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:s1lv3yNTRc2RBJDEe1CaUan6KY0= 2023-07-29T00:24:53.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:s1lv3yNTRc2RBJDEe1CaUan6KY0='} 2023-07-29T00:24:53.992 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003befd2d60c3734a2-0064c45c55-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:53.992 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:53.993 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:53.993 DEBUG:boto:Method: HEAD 2023-07-29T00:24:53.993 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:53.994 DEBUG:boto:Data: 2023-07-29T00:24:53.994 DEBUG:boto:Headers: {} 2023-07-29T00:24:53.994 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:53.995 DEBUG:boto:Port: 8000 2023-07-29T00:24:53.995 DEBUG:boto:Params: {} 2023-07-29T00:24:53.995 DEBUG:boto:Token: None 2023-07-29T00:24:53.996 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:53 GMT /hiiqos-58/ 2023-07-29T00:24:53.996 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/2x4mBUU98gvOQOLwUFLrPADylM= 2023-07-29T00:24:53.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/2x4mBUU98gvOQOLwUFLrPADylM=', 'Content-Length': '0'} 2023-07-29T00:24:53.999 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f9d7be7f9edc6369-0064c45c55-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.000 DEBUG:boto:path=/hiiqos-58/obj00000000000000000076 2023-07-29T00:24:54.000 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000076 2023-07-29T00:24:54.000 DEBUG:boto:Method: PUT 2023-07-29T00:24:54.001 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000076 2023-07-29T00:24:54.001 DEBUG:boto:Data: 2023-07-29T00:24:54.002 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:54.002 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.002 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.003 DEBUG:boto:Params: {} 2023-07-29T00:24:54.003 DEBUG:boto:Token: None 2023-07-29T00:24:54.003 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/obj00000000000000000076 2023-07-29T00:24:54.004 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:v4sfrY/ejH51YKppVZ3h29ZghAI= 2023-07-29T00:24:54.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:v4sfrY/ejH51YKppVZ3h29ZghAI='} 2023-07-29T00:24:54.009 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000de8d8b48c33412df-0064c45c55-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.010 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:54.010 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:54.011 DEBUG:boto:Method: HEAD 2023-07-29T00:24:54.011 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:54.011 DEBUG:boto:Data: 2023-07-29T00:24:54.012 DEBUG:boto:Headers: {} 2023-07-29T00:24:54.012 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.012 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.013 DEBUG:boto:Params: {} 2023-07-29T00:24:54.013 DEBUG:boto:Token: None 2023-07-29T00:24:54.014 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/ 2023-07-29T00:24:54.014 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0= 2023-07-29T00:24:54.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0=', 'Content-Length': '0'} 2023-07-29T00:24:54.017 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cfe2d6b02b718499-0064c45c55-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.017 DEBUG:boto:path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:24:54.018 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:24:54.018 DEBUG:boto:Method: PUT 2023-07-29T00:24:54.019 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000001 2023-07-29T00:24:54.019 DEBUG:boto:Data: 2023-07-29T00:24:54.019 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:54.020 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.020 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.020 DEBUG:boto:Params: {} 2023-07-29T00:24:54.021 DEBUG:boto:Token: None 2023-07-29T00:24:54.021 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/obj00000000000000000001 2023-07-29T00:24:54.022 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:X29rKXFlkbABR2yBCd+Q6fHFMt8= 2023-07-29T00:24:54.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:X29rKXFlkbABR2yBCd+Q6fHFMt8='} 2023-07-29T00:24:54.068 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000036ed87656cf2cf7a-0064c45c55-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.070 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:54.071 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:54.072 DEBUG:boto:Method: HEAD 2023-07-29T00:24:54.073 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:54.073 DEBUG:boto:Data: 2023-07-29T00:24:54.074 DEBUG:boto:Headers: {} 2023-07-29T00:24:54.074 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.075 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.075 DEBUG:boto:Params: {} 2023-07-29T00:24:54.075 DEBUG:boto:Token: None 2023-07-29T00:24:54.076 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/ 2023-07-29T00:24:54.077 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0= 2023-07-29T00:24:54.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0=', 'Content-Length': '0'} 2023-07-29T00:24:54.080 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c934ce15aebbf196-0064c45c56-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.081 DEBUG:boto:path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:24:54.081 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:24:54.081 DEBUG:boto:Method: PUT 2023-07-29T00:24:54.082 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000013 2023-07-29T00:24:54.082 DEBUG:boto:Data: 2023-07-29T00:24:54.083 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:54.083 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.083 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.084 DEBUG:boto:Params: {} 2023-07-29T00:24:54.084 DEBUG:boto:Token: None 2023-07-29T00:24:54.085 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/obj00000000000000000013 2023-07-29T00:24:54.085 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SMQ+z0sBG7ZRf+zOrE1XhIS0PYw= 2023-07-29T00:24:54.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SMQ+z0sBG7ZRf+zOrE1XhIS0PYw='} 2023-07-29T00:24:54.092 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a7a9ad014a963e74-0064c45c56-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.092 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:54.092 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:54.093 DEBUG:boto:Method: HEAD 2023-07-29T00:24:54.093 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:54.094 DEBUG:boto:Data: 2023-07-29T00:24:54.094 DEBUG:boto:Headers: {} 2023-07-29T00:24:54.094 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.095 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.095 DEBUG:boto:Params: {} 2023-07-29T00:24:54.095 DEBUG:boto:Token: None 2023-07-29T00:24:54.096 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/ 2023-07-29T00:24:54.096 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0= 2023-07-29T00:24:54.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0=', 'Content-Length': '0'} 2023-07-29T00:24:54.099 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e2b885cd75d29098-0064c45c56-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.099 DEBUG:boto:path=/hiiqos-58/obj00000000000000000010 2023-07-29T00:24:54.100 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000010 2023-07-29T00:24:54.100 DEBUG:boto:Method: PUT 2023-07-29T00:24:54.100 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000010 2023-07-29T00:24:54.101 DEBUG:boto:Data: 2023-07-29T00:24:54.101 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:54.101 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.102 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.102 DEBUG:boto:Params: {} 2023-07-29T00:24:54.102 DEBUG:boto:Token: None 2023-07-29T00:24:54.103 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/obj00000000000000000010 2023-07-29T00:24:54.103 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Q0/ZuzqVjAwRYPLhHPvJhKTw8Oc= 2023-07-29T00:24:54.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q0/ZuzqVjAwRYPLhHPvJhKTw8Oc='} 2023-07-29T00:24:54.108 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a725a9cc16c9319b-0064c45c56-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.109 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:54.109 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:54.109 DEBUG:boto:Method: HEAD 2023-07-29T00:24:54.110 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:54.110 DEBUG:boto:Data: 2023-07-29T00:24:54.110 DEBUG:boto:Headers: {} 2023-07-29T00:24:54.111 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.111 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.111 DEBUG:boto:Params: {} 2023-07-29T00:24:54.112 DEBUG:boto:Token: None 2023-07-29T00:24:54.112 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/ 2023-07-29T00:24:54.113 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0= 2023-07-29T00:24:54.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0=', 'Content-Length': '0'} 2023-07-29T00:24:54.115 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000904e61fa52cde179-0064c45c56-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.117 DEBUG:boto:path=/hiiqos-58/obj00000000000000000033 2023-07-29T00:24:54.117 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000033 2023-07-29T00:24:54.118 DEBUG:boto:Method: PUT 2023-07-29T00:24:54.118 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000033 2023-07-29T00:24:54.119 DEBUG:boto:Data: 2023-07-29T00:24:54.119 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:54.119 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.120 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.120 DEBUG:boto:Params: {} 2023-07-29T00:24:54.120 DEBUG:boto:Token: None 2023-07-29T00:24:54.121 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/obj00000000000000000033 2023-07-29T00:24:54.121 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gThZzkFjKC32tek26B8aeMIeEg8= 2023-07-29T00:24:54.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gThZzkFjKC32tek26B8aeMIeEg8='} 2023-07-29T00:24:54.126 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aa436583b4705e58-0064c45c56-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.127 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-58', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:24:54.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-58', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:24:54.128 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-58 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:24:54.176 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:24:54.176 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:24:54.217 INFO:teuthology.orchestra.run.smithi101.stdout:tenant: 2023-07-29T00:24:54.217 INFO:teuthology.orchestra.run.smithi101.stdout:bucket name: hiiqos-58 2023-07-29T00:24:54.223 INFO:teuthology.orchestra.run.smithi101.stdout:total entries: 22 2023-07-29T00:24:54.264 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:24:54.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:24:54.266 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:24:54.314 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:24:54.315 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:24:54.354 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:24:54.354 INFO:teuthology.orchestra.run.smithi101.stdout: "objs": [ 2023-07-29T00:24:54.354 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000005", 2023-07-29T00:24:54.355 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000010", 2023-07-29T00:24:54.355 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000016", 2023-07-29T00:24:54.355 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000021", 2023-07-29T00:24:54.355 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000004", 2023-07-29T00:24:54.355 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000025", 2023-07-29T00:24:54.355 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000030", 2023-07-29T00:24:54.355 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000015", 2023-07-29T00:24:54.356 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000019", 2023-07-29T00:24:54.356 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000003", 2023-07-29T00:24:54.356 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000009", 2023-07-29T00:24:54.356 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000014", 2023-07-29T00:24:54.356 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000012", 2023-07-29T00:24:54.356 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000002", 2023-07-29T00:24:54.357 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000000", 2023-07-29T00:24:54.357 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000006", 2023-07-29T00:24:54.357 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000011", 2023-07-29T00:24:54.357 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000001", 2023-07-29T00:24:54.357 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000007" 2023-07-29T00:24:54.358 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:24:54.358 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:24:54.359 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:54.359 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:54.360 DEBUG:boto:Method: HEAD 2023-07-29T00:24:54.360 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:54.361 DEBUG:boto:Data: 2023-07-29T00:24:54.361 DEBUG:boto:Headers: {} 2023-07-29T00:24:54.361 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.362 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.362 DEBUG:boto:Params: {} 2023-07-29T00:24:54.362 DEBUG:boto:Token: None 2023-07-29T00:24:54.363 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/ 2023-07-29T00:24:54.363 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0= 2023-07-29T00:24:54.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0=', 'Content-Length': '0'} 2023-07-29T00:24:54.367 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cca5305caf82b2a3-0064c45c56-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.367 DEBUG:boto:path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:24:54.368 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:24:54.368 DEBUG:boto:Method: PUT 2023-07-29T00:24:54.369 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000003 2023-07-29T00:24:54.369 DEBUG:boto:Data: 2023-07-29T00:24:54.370 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:54.370 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.371 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.371 DEBUG:boto:Params: {} 2023-07-29T00:24:54.371 DEBUG:boto:Token: None 2023-07-29T00:24:54.372 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/obj00000000000000000003 2023-07-29T00:24:54.373 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fx+dVjtddr8hM5OI9F/gcKuIu/g= 2023-07-29T00:24:54.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fx+dVjtddr8hM5OI9F/gcKuIu/g='} 2023-07-29T00:24:54.379 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006a1dcb9e6bd0ac68-0064c45c56-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.380 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:54.380 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:54.381 DEBUG:boto:Method: HEAD 2023-07-29T00:24:54.381 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:54.381 DEBUG:boto:Data: 2023-07-29T00:24:54.382 DEBUG:boto:Headers: {} 2023-07-29T00:24:54.382 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.382 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.383 DEBUG:boto:Params: {} 2023-07-29T00:24:54.383 DEBUG:boto:Token: None 2023-07-29T00:24:54.384 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/ 2023-07-29T00:24:54.384 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0= 2023-07-29T00:24:54.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0=', 'Content-Length': '0'} 2023-07-29T00:24:54.387 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d4e13f714c96d628-0064c45c56-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.388 DEBUG:boto:path=/hiiqos-58/obj00000000000000000021 2023-07-29T00:24:54.388 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000021 2023-07-29T00:24:54.389 DEBUG:boto:Method: PUT 2023-07-29T00:24:54.389 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000021 2023-07-29T00:24:54.389 DEBUG:boto:Data: 2023-07-29T00:24:54.390 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:54.390 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.390 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.391 DEBUG:boto:Params: {} 2023-07-29T00:24:54.391 DEBUG:boto:Token: None 2023-07-29T00:24:54.392 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/obj00000000000000000021 2023-07-29T00:24:54.392 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:iHA3dc235dFQ2qY7q58lI38Rbeg= 2023-07-29T00:24:54.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:iHA3dc235dFQ2qY7q58lI38Rbeg='} 2023-07-29T00:24:54.436 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000029530a447042b2fc-0064c45c56-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.437 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:54.438 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:54.438 DEBUG:boto:Method: HEAD 2023-07-29T00:24:54.438 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:54.439 DEBUG:boto:Data: 2023-07-29T00:24:54.439 DEBUG:boto:Headers: {} 2023-07-29T00:24:54.439 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.440 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.440 DEBUG:boto:Params: {} 2023-07-29T00:24:54.441 DEBUG:boto:Token: None 2023-07-29T00:24:54.441 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/ 2023-07-29T00:24:54.441 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0= 2023-07-29T00:24:54.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0=', 'Content-Length': '0'} 2023-07-29T00:24:54.444 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000075cb61301f8768f1-0064c45c56-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.445 DEBUG:boto:path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:24:54.445 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:24:54.446 DEBUG:boto:Method: PUT 2023-07-29T00:24:54.446 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000002 2023-07-29T00:24:54.446 DEBUG:boto:Data: 2023-07-29T00:24:54.447 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:54.447 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.447 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.448 DEBUG:boto:Params: {} 2023-07-29T00:24:54.448 DEBUG:boto:Token: None 2023-07-29T00:24:54.449 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/obj00000000000000000002 2023-07-29T00:24:54.449 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2JFq0uJjVZWyquDmF/0fnT+TKbA= 2023-07-29T00:24:54.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2JFq0uJjVZWyquDmF/0fnT+TKbA='} 2023-07-29T00:24:54.456 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000060577c2c9dd6556d-0064c45c56-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.456 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:54.457 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:54.457 DEBUG:boto:Method: HEAD 2023-07-29T00:24:54.458 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:54.458 DEBUG:boto:Data: 2023-07-29T00:24:54.458 DEBUG:boto:Headers: {} 2023-07-29T00:24:54.458 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.459 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.459 DEBUG:boto:Params: {} 2023-07-29T00:24:54.460 DEBUG:boto:Token: None 2023-07-29T00:24:54.460 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/ 2023-07-29T00:24:54.460 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0= 2023-07-29T00:24:54.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0=', 'Content-Length': '0'} 2023-07-29T00:24:54.463 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fb2cc0c00195cb46-0064c45c56-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.464 DEBUG:boto:path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:24:54.464 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:24:54.465 DEBUG:boto:Method: PUT 2023-07-29T00:24:54.465 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000005 2023-07-29T00:24:54.465 DEBUG:boto:Data: 2023-07-29T00:24:54.466 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:54.466 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.467 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.467 DEBUG:boto:Params: {} 2023-07-29T00:24:54.467 DEBUG:boto:Token: None 2023-07-29T00:24:54.468 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/obj00000000000000000005 2023-07-29T00:24:54.468 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:g0JpkCduvVd8KuE3UpebJ5gYPpA= 2023-07-29T00:24:54.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:g0JpkCduvVd8KuE3UpebJ5gYPpA='} 2023-07-29T00:24:54.512 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004aedd391dc0431a8-0064c45c56-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.513 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:54.513 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:54.514 DEBUG:boto:Method: HEAD 2023-07-29T00:24:54.514 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:54.514 DEBUG:boto:Data: 2023-07-29T00:24:54.515 DEBUG:boto:Headers: {} 2023-07-29T00:24:54.515 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.515 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.516 DEBUG:boto:Params: {} 2023-07-29T00:24:54.516 DEBUG:boto:Token: None 2023-07-29T00:24:54.517 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/ 2023-07-29T00:24:54.517 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0= 2023-07-29T00:24:54.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0=', 'Content-Length': '0'} 2023-07-29T00:24:54.520 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c515f4c4b49ca8f2-0064c45c56-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.521 DEBUG:boto:path=/hiiqos-58/obj00000000000000000009 2023-07-29T00:24:54.521 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000009 2023-07-29T00:24:54.522 DEBUG:boto:Method: PUT 2023-07-29T00:24:54.522 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000009 2023-07-29T00:24:54.522 DEBUG:boto:Data: 2023-07-29T00:24:54.523 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:54.523 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.523 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.524 DEBUG:boto:Params: {} 2023-07-29T00:24:54.524 DEBUG:boto:Token: None 2023-07-29T00:24:54.524 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/obj00000000000000000009 2023-07-29T00:24:54.525 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:u+WMJRZqfZBHlw9ohh9LIXT6WVA= 2023-07-29T00:24:54.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:u+WMJRZqfZBHlw9ohh9LIXT6WVA='} 2023-07-29T00:24:54.530 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000736eaceb007bfb93-0064c45c56-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.531 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:54.531 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:54.531 DEBUG:boto:Method: HEAD 2023-07-29T00:24:54.532 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:54.532 DEBUG:boto:Data: 2023-07-29T00:24:54.532 DEBUG:boto:Headers: {} 2023-07-29T00:24:54.533 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.533 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.533 DEBUG:boto:Params: {} 2023-07-29T00:24:54.534 DEBUG:boto:Token: None 2023-07-29T00:24:54.534 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/ 2023-07-29T00:24:54.535 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0= 2023-07-29T00:24:54.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0=', 'Content-Length': '0'} 2023-07-29T00:24:54.537 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000324b2000fb187a2c-0064c45c56-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.538 DEBUG:boto:path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:24:54.538 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:24:54.539 DEBUG:boto:Method: PUT 2023-07-29T00:24:54.539 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000006 2023-07-29T00:24:54.540 DEBUG:boto:Data: 2023-07-29T00:24:54.540 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:54.541 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.541 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.542 DEBUG:boto:Params: {} 2023-07-29T00:24:54.542 DEBUG:boto:Token: None 2023-07-29T00:24:54.542 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/obj00000000000000000006 2023-07-29T00:24:54.543 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gs6zTSmIbhQII0efRMyD8vSMFjA= 2023-07-29T00:24:54.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gs6zTSmIbhQII0efRMyD8vSMFjA='} 2023-07-29T00:24:54.592 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000d8d1c662052b708-0064c45c56-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.593 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:54.594 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:54.594 DEBUG:boto:Method: HEAD 2023-07-29T00:24:54.594 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:54.595 DEBUG:boto:Data: 2023-07-29T00:24:54.595 DEBUG:boto:Headers: {} 2023-07-29T00:24:54.595 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.596 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.596 DEBUG:boto:Params: {} 2023-07-29T00:24:54.596 DEBUG:boto:Token: None 2023-07-29T00:24:54.597 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/ 2023-07-29T00:24:54.597 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0= 2023-07-29T00:24:54.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0=', 'Content-Length': '0'} 2023-07-29T00:24:54.600 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ca0bdc48794d480-0064c45c56-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.601 DEBUG:boto:path=/hiiqos-58/obj00000000000000000030 2023-07-29T00:24:54.601 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000030 2023-07-29T00:24:54.601 DEBUG:boto:Method: PUT 2023-07-29T00:24:54.602 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000030 2023-07-29T00:24:54.602 DEBUG:boto:Data: 2023-07-29T00:24:54.602 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:54.603 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.603 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.603 DEBUG:boto:Params: {} 2023-07-29T00:24:54.604 DEBUG:boto:Token: None 2023-07-29T00:24:54.604 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/obj00000000000000000030 2023-07-29T00:24:54.605 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LH3gNRB5PCs6Hjt2YdE6JOW+J7M= 2023-07-29T00:24:54.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LH3gNRB5PCs6Hjt2YdE6JOW+J7M='} 2023-07-29T00:24:54.611 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c21a04fe6c16c301-0064c45c56-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.612 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:54.612 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:54.612 DEBUG:boto:Method: HEAD 2023-07-29T00:24:54.613 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:54.613 DEBUG:boto:Data: 2023-07-29T00:24:54.613 DEBUG:boto:Headers: {} 2023-07-29T00:24:54.614 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.614 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.614 DEBUG:boto:Params: {} 2023-07-29T00:24:54.615 DEBUG:boto:Token: None 2023-07-29T00:24:54.615 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/ 2023-07-29T00:24:54.616 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0= 2023-07-29T00:24:54.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0=', 'Content-Length': '0'} 2023-07-29T00:24:54.619 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000070891569ad96e8ab-0064c45c56-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.619 DEBUG:boto:path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:24:54.620 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:24:54.620 DEBUG:boto:Method: PUT 2023-07-29T00:24:54.620 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000011 2023-07-29T00:24:54.621 DEBUG:boto:Data: 2023-07-29T00:24:54.621 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:54.621 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.622 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.622 DEBUG:boto:Params: {} 2023-07-29T00:24:54.622 DEBUG:boto:Token: None 2023-07-29T00:24:54.623 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/obj00000000000000000011 2023-07-29T00:24:54.623 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9HkqV4sye1UuNQT3pedzSWhQZ4Y= 2023-07-29T00:24:54.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9HkqV4sye1UuNQT3pedzSWhQZ4Y='} 2023-07-29T00:24:54.629 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007843a7c6626ad67b-0064c45c56-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.630 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:54.631 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:54.631 DEBUG:boto:Method: HEAD 2023-07-29T00:24:54.631 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:54.632 DEBUG:boto:Data: 2023-07-29T00:24:54.632 DEBUG:boto:Headers: {} 2023-07-29T00:24:54.632 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.633 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.633 DEBUG:boto:Params: {} 2023-07-29T00:24:54.634 DEBUG:boto:Token: None 2023-07-29T00:24:54.634 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/ 2023-07-29T00:24:54.635 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0= 2023-07-29T00:24:54.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0=', 'Content-Length': '0'} 2023-07-29T00:24:54.637 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b0c2f808dd6e739-0064c45c56-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.638 DEBUG:boto:path=/hiiqos-58/obj00000000000000000019 2023-07-29T00:24:54.638 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000019 2023-07-29T00:24:54.639 DEBUG:boto:Method: PUT 2023-07-29T00:24:54.639 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000019 2023-07-29T00:24:54.639 DEBUG:boto:Data: 2023-07-29T00:24:54.640 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:54.640 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.641 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.641 DEBUG:boto:Params: {} 2023-07-29T00:24:54.642 DEBUG:boto:Token: None 2023-07-29T00:24:54.642 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/obj00000000000000000019 2023-07-29T00:24:54.642 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:H+Z0/jj9uiiTUza12avG63wyKfk= 2023-07-29T00:24:54.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:H+Z0/jj9uiiTUza12avG63wyKfk='} 2023-07-29T00:24:54.688 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000783ba0ab4475f7a5-0064c45c56-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.689 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:54.690 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:54.690 DEBUG:boto:Method: HEAD 2023-07-29T00:24:54.690 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:54.691 DEBUG:boto:Data: 2023-07-29T00:24:54.691 DEBUG:boto:Headers: {} 2023-07-29T00:24:54.691 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.692 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.692 DEBUG:boto:Params: {} 2023-07-29T00:24:54.693 DEBUG:boto:Token: None 2023-07-29T00:24:54.693 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/ 2023-07-29T00:24:54.693 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0= 2023-07-29T00:24:54.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0=', 'Content-Length': '0'} 2023-07-29T00:24:54.696 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bcf216bc7f9fd6fc-0064c45c56-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.697 DEBUG:boto:path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:24:54.697 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:24:54.698 DEBUG:boto:Method: PUT 2023-07-29T00:24:54.698 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000012 2023-07-29T00:24:54.698 DEBUG:boto:Data: 2023-07-29T00:24:54.699 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:54.699 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.699 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.700 DEBUG:boto:Params: {} 2023-07-29T00:24:54.700 DEBUG:boto:Token: None 2023-07-29T00:24:54.701 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/obj00000000000000000012 2023-07-29T00:24:54.701 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DJjm0FtBIUDzTiKC/MHeP3Bwzn4= 2023-07-29T00:24:54.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DJjm0FtBIUDzTiKC/MHeP3Bwzn4='} 2023-07-29T00:24:54.707 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bc3d5f37d179fc76-0064c45c56-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.709 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:54.710 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:54.711 DEBUG:boto:Method: HEAD 2023-07-29T00:24:54.712 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:54.712 DEBUG:boto:Data: 2023-07-29T00:24:54.713 DEBUG:boto:Headers: {} 2023-07-29T00:24:54.713 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.714 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.715 DEBUG:boto:Params: {} 2023-07-29T00:24:54.716 DEBUG:boto:Token: None 2023-07-29T00:24:54.716 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/ 2023-07-29T00:24:54.717 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0= 2023-07-29T00:24:54.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0=', 'Content-Length': '0'} 2023-07-29T00:24:54.720 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d910e9f2e69e6862-0064c45c56-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.721 DEBUG:boto:path=/hiiqos-58/obj00000000000000000010 2023-07-29T00:24:54.721 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000010 2023-07-29T00:24:54.722 DEBUG:boto:Method: PUT 2023-07-29T00:24:54.722 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000010 2023-07-29T00:24:54.722 DEBUG:boto:Data: 2023-07-29T00:24:54.723 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:54.723 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.724 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.724 DEBUG:boto:Params: {} 2023-07-29T00:24:54.725 DEBUG:boto:Token: None 2023-07-29T00:24:54.725 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/obj00000000000000000010 2023-07-29T00:24:54.726 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Q0/ZuzqVjAwRYPLhHPvJhKTw8Oc= 2023-07-29T00:24:54.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q0/ZuzqVjAwRYPLhHPvJhKTw8Oc='} 2023-07-29T00:24:54.772 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000073e4682261518097-0064c45c56-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.773 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:54.774 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:54.774 DEBUG:boto:Method: HEAD 2023-07-29T00:24:54.775 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:54.775 DEBUG:boto:Data: 2023-07-29T00:24:54.775 DEBUG:boto:Headers: {} 2023-07-29T00:24:54.776 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.776 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.777 DEBUG:boto:Params: {} 2023-07-29T00:24:54.778 DEBUG:boto:Token: None 2023-07-29T00:24:54.778 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/ 2023-07-29T00:24:54.779 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0= 2023-07-29T00:24:54.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0=', 'Content-Length': '0'} 2023-07-29T00:24:54.782 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004942b9ba52552f81-0064c45c56-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.782 DEBUG:boto:path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:24:54.783 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:24:54.783 DEBUG:boto:Method: PUT 2023-07-29T00:24:54.783 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000000 2023-07-29T00:24:54.784 DEBUG:boto:Data: 2023-07-29T00:24:54.784 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:54.784 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.785 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.785 DEBUG:boto:Params: {} 2023-07-29T00:24:54.786 DEBUG:boto:Token: None 2023-07-29T00:24:54.786 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/obj00000000000000000000 2023-07-29T00:24:54.787 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6oAEfHnciCQlbIUcFTEA/EVdVVw= 2023-07-29T00:24:54.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6oAEfHnciCQlbIUcFTEA/EVdVVw='} 2023-07-29T00:24:54.793 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a1b038e06a0dc3c9-0064c45c56-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.794 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:54.794 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:54.795 DEBUG:boto:Method: HEAD 2023-07-29T00:24:54.795 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:54.795 DEBUG:boto:Data: 2023-07-29T00:24:54.796 DEBUG:boto:Headers: {} 2023-07-29T00:24:54.796 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.796 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.797 DEBUG:boto:Params: {} 2023-07-29T00:24:54.797 DEBUG:boto:Token: None 2023-07-29T00:24:54.798 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/ 2023-07-29T00:24:54.798 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0= 2023-07-29T00:24:54.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0=', 'Content-Length': '0'} 2023-07-29T00:24:54.801 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000023cbadc712cdb889-0064c45c56-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.802 DEBUG:boto:path=/hiiqos-58/obj00000000000000000007 2023-07-29T00:24:54.802 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000007 2023-07-29T00:24:54.803 DEBUG:boto:Method: PUT 2023-07-29T00:24:54.803 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000007 2023-07-29T00:24:54.803 DEBUG:boto:Data: 2023-07-29T00:24:54.804 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:54.804 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.804 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.805 DEBUG:boto:Params: {} 2023-07-29T00:24:54.805 DEBUG:boto:Token: None 2023-07-29T00:24:54.806 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/obj00000000000000000007 2023-07-29T00:24:54.806 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sGo5iKXuoXIk1xxRLijA/6l6veU= 2023-07-29T00:24:54.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sGo5iKXuoXIk1xxRLijA/6l6veU='} 2023-07-29T00:24:54.848 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ab22a2feb16e15cb-0064c45c56-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.850 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:54.850 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:54.851 DEBUG:boto:Method: HEAD 2023-07-29T00:24:54.851 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:54.852 DEBUG:boto:Data: 2023-07-29T00:24:54.852 DEBUG:boto:Headers: {} 2023-07-29T00:24:54.853 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.853 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.853 DEBUG:boto:Params: {} 2023-07-29T00:24:54.854 DEBUG:boto:Token: None 2023-07-29T00:24:54.855 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/ 2023-07-29T00:24:54.855 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0= 2023-07-29T00:24:54.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0=', 'Content-Length': '0'} 2023-07-29T00:24:54.858 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000042bc744db9fe91f3-0064c45c56-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.859 DEBUG:boto:path=/hiiqos-58/obj00000000000000000025 2023-07-29T00:24:54.860 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000025 2023-07-29T00:24:54.860 DEBUG:boto:Method: PUT 2023-07-29T00:24:54.861 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000025 2023-07-29T00:24:54.861 DEBUG:boto:Data: 2023-07-29T00:24:54.861 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:54.862 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.862 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.863 DEBUG:boto:Params: {} 2023-07-29T00:24:54.863 DEBUG:boto:Token: None 2023-07-29T00:24:54.864 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/obj00000000000000000025 2023-07-29T00:24:54.864 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4bKkIFbfM2uEYY2d4ZITssHnMhE= 2023-07-29T00:24:54.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4bKkIFbfM2uEYY2d4ZITssHnMhE='} 2023-07-29T00:24:54.870 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000163aa8ed7166b59a-0064c45c56-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.870 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:54.871 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:54.872 DEBUG:boto:Method: HEAD 2023-07-29T00:24:54.872 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:54.872 DEBUG:boto:Data: 2023-07-29T00:24:54.873 DEBUG:boto:Headers: {} 2023-07-29T00:24:54.873 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.874 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.874 DEBUG:boto:Params: {} 2023-07-29T00:24:54.874 DEBUG:boto:Token: None 2023-07-29T00:24:54.875 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/ 2023-07-29T00:24:54.875 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0= 2023-07-29T00:24:54.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0=', 'Content-Length': '0'} 2023-07-29T00:24:54.878 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000080d51dfc4bf4ee88-0064c45c56-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.879 DEBUG:boto:path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:24:54.879 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:24:54.880 DEBUG:boto:Method: PUT 2023-07-29T00:24:54.880 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000001 2023-07-29T00:24:54.881 DEBUG:boto:Data: 2023-07-29T00:24:54.881 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:54.882 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.882 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.882 DEBUG:boto:Params: {} 2023-07-29T00:24:54.883 DEBUG:boto:Token: None 2023-07-29T00:24:54.883 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/obj00000000000000000001 2023-07-29T00:24:54.884 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:X29rKXFlkbABR2yBCd+Q6fHFMt8= 2023-07-29T00:24:54.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:X29rKXFlkbABR2yBCd+Q6fHFMt8='} 2023-07-29T00:24:54.891 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000011dbfc42bbad8856-0064c45c56-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.891 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:54.892 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:54.892 DEBUG:boto:Method: HEAD 2023-07-29T00:24:54.893 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:54.893 DEBUG:boto:Data: 2023-07-29T00:24:54.893 DEBUG:boto:Headers: {} 2023-07-29T00:24:54.894 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.894 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.894 DEBUG:boto:Params: {} 2023-07-29T00:24:54.895 DEBUG:boto:Token: None 2023-07-29T00:24:54.895 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/ 2023-07-29T00:24:54.895 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0= 2023-07-29T00:24:54.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0=', 'Content-Length': '0'} 2023-07-29T00:24:54.898 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000035113243a3b1cbbd-0064c45c56-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.899 DEBUG:boto:path=/hiiqos-58/obj00000000000000000014 2023-07-29T00:24:54.899 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000014 2023-07-29T00:24:54.899 DEBUG:boto:Method: PUT 2023-07-29T00:24:54.900 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000014 2023-07-29T00:24:54.900 DEBUG:boto:Data: 2023-07-29T00:24:54.901 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:54.901 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.902 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.902 DEBUG:boto:Params: {} 2023-07-29T00:24:54.902 DEBUG:boto:Token: None 2023-07-29T00:24:54.903 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/obj00000000000000000014 2023-07-29T00:24:54.904 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1sDQMkRGq3oXf3OFgngSbGK2mvQ= 2023-07-29T00:24:54.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1sDQMkRGq3oXf3OFgngSbGK2mvQ='} 2023-07-29T00:24:54.949 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000006d0b26dc3a1e4ca-0064c45c56-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.950 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:54.951 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:54.951 DEBUG:boto:Method: HEAD 2023-07-29T00:24:54.952 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:54.953 DEBUG:boto:Data: 2023-07-29T00:24:54.953 DEBUG:boto:Headers: {} 2023-07-29T00:24:54.954 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.955 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.956 DEBUG:boto:Params: {} 2023-07-29T00:24:54.956 DEBUG:boto:Token: None 2023-07-29T00:24:54.957 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/ 2023-07-29T00:24:54.958 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0= 2023-07-29T00:24:54.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0=', 'Content-Length': '0'} 2023-07-29T00:24:54.962 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cc2e2c1425ea5bb4-0064c45c56-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.963 DEBUG:boto:path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:24:54.964 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:24:54.964 DEBUG:boto:Method: PUT 2023-07-29T00:24:54.965 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000004 2023-07-29T00:24:54.966 DEBUG:boto:Data: 2023-07-29T00:24:54.966 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:54.966 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.967 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.967 DEBUG:boto:Params: {} 2023-07-29T00:24:54.968 DEBUG:boto:Token: None 2023-07-29T00:24:54.968 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/obj00000000000000000004 2023-07-29T00:24:54.969 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:t2DcmaYUv/2Yae2kj4RK708WueM= 2023-07-29T00:24:54.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t2DcmaYUv/2Yae2kj4RK708WueM='} 2023-07-29T00:24:54.975 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a245264c4a98a99a-0064c45c56-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.976 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:54.976 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:54.977 DEBUG:boto:Method: HEAD 2023-07-29T00:24:54.977 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:54.977 DEBUG:boto:Data: 2023-07-29T00:24:54.978 DEBUG:boto:Headers: {} 2023-07-29T00:24:54.978 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.979 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.979 DEBUG:boto:Params: {} 2023-07-29T00:24:54.980 DEBUG:boto:Token: None 2023-07-29T00:24:54.980 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/ 2023-07-29T00:24:54.981 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0= 2023-07-29T00:24:54.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TMIZPjtJTEUQ/QaMcjEy8o1meF0=', 'Content-Length': '0'} 2023-07-29T00:24:54.984 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001147b27aaf0e9a48-0064c45c56-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.985 DEBUG:boto:path=/hiiqos-58/obj00000000000000000015 2023-07-29T00:24:54.985 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000015 2023-07-29T00:24:54.986 DEBUG:boto:Method: PUT 2023-07-29T00:24:54.986 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000015 2023-07-29T00:24:54.986 DEBUG:boto:Data: 2023-07-29T00:24:54.987 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:54.987 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:54.988 DEBUG:boto:Port: 8000 2023-07-29T00:24:54.988 DEBUG:boto:Params: {} 2023-07-29T00:24:54.988 DEBUG:boto:Token: None 2023-07-29T00:24:54.989 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:54 GMT /hiiqos-58/obj00000000000000000015 2023-07-29T00:24:54.990 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JKXQOUuat/rqKAtR7ScUBzpXCg0= 2023-07-29T00:24:54.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JKXQOUuat/rqKAtR7ScUBzpXCg0='} 2023-07-29T00:24:54.995 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000082159c64a948f386-0064c45c56-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:54.996 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-58', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:24:54.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-58', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:24:54.998 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-58 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:24:55.046 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:24:55.047 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:24:55.088 INFO:teuthology.orchestra.run.smithi101.stdout:tenant: 2023-07-29T00:24:55.089 INFO:teuthology.orchestra.run.smithi101.stdout:bucket name: hiiqos-58 2023-07-29T00:24:55.095 INFO:teuthology.orchestra.run.smithi101.stdout:total entries: 27 2023-07-29T00:24:55.138 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:24:55.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:24:55.140 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:24:55.187 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:24:55.188 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:24:55.227 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:24:55.227 INFO:teuthology.orchestra.run.smithi101.stdout: "objs": [ 2023-07-29T00:24:55.227 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000058", 2023-07-29T00:24:55.227 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000026", 2023-07-29T00:24:55.227 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000011", 2023-07-29T00:24:55.228 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000046", 2023-07-29T00:24:55.228 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000014", 2023-07-29T00:24:55.228 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000050", 2023-07-29T00:24:55.228 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000032", 2023-07-29T00:24:55.228 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000002", 2023-07-29T00:24:55.228 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000039", 2023-07-29T00:24:55.229 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000007", 2023-07-29T00:24:55.229 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000006", 2023-07-29T00:24:55.229 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000040", 2023-07-29T00:24:55.229 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000009", 2023-07-29T00:24:55.229 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000045", 2023-07-29T00:24:55.229 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000013", 2023-07-29T00:24:55.229 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000012", 2023-07-29T00:24:55.230 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000033", 2023-07-29T00:24:55.230 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000001", 2023-07-29T00:24:55.230 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000000", 2023-07-29T00:24:55.230 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000021", 2023-07-29T00:24:55.230 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000003", 2023-07-29T00:24:55.230 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000055", 2023-07-29T00:24:55.230 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000008" 2023-07-29T00:24:55.231 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:24:55.231 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:24:55.232 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:55.232 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:55.233 DEBUG:boto:Method: HEAD 2023-07-29T00:24:55.233 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:55.234 DEBUG:boto:Data: 2023-07-29T00:24:55.234 DEBUG:boto:Headers: {} 2023-07-29T00:24:55.234 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:55.235 DEBUG:boto:Port: 8000 2023-07-29T00:24:55.235 DEBUG:boto:Params: {} 2023-07-29T00:24:55.235 DEBUG:boto:Token: None 2023-07-29T00:24:55.236 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:55 GMT /hiiqos-58/ 2023-07-29T00:24:55.236 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HyOqaYwDa5ZSg9TdbLPmcy5bYL0= 2023-07-29T00:24:55.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HyOqaYwDa5ZSg9TdbLPmcy5bYL0=', 'Content-Length': '0'} 2023-07-29T00:24:55.239 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005f685038f535c3ef-0064c45c57-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:55.240 DEBUG:boto:path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:24:55.241 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:24:55.241 DEBUG:boto:Method: PUT 2023-07-29T00:24:55.241 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000006 2023-07-29T00:24:55.242 DEBUG:boto:Data: 2023-07-29T00:24:55.242 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:55.242 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:55.243 DEBUG:boto:Port: 8000 2023-07-29T00:24:55.243 DEBUG:boto:Params: {} 2023-07-29T00:24:55.243 DEBUG:boto:Token: None 2023-07-29T00:24:55.244 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:55 GMT /hiiqos-58/obj00000000000000000006 2023-07-29T00:24:55.244 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kZt7WK7QczMEKAroicm9Nhm1tZM= 2023-07-29T00:24:55.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kZt7WK7QczMEKAroicm9Nhm1tZM='} 2023-07-29T00:24:55.251 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007b3902d34aeed246-0064c45c57-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:55.251 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:55.252 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:55.252 DEBUG:boto:Method: HEAD 2023-07-29T00:24:55.253 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:55.253 DEBUG:boto:Data: 2023-07-29T00:24:55.254 DEBUG:boto:Headers: {} 2023-07-29T00:24:55.254 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:55.254 DEBUG:boto:Port: 8000 2023-07-29T00:24:55.255 DEBUG:boto:Params: {} 2023-07-29T00:24:55.255 DEBUG:boto:Token: None 2023-07-29T00:24:55.255 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:55 GMT /hiiqos-58/ 2023-07-29T00:24:55.256 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HyOqaYwDa5ZSg9TdbLPmcy5bYL0= 2023-07-29T00:24:55.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HyOqaYwDa5ZSg9TdbLPmcy5bYL0=', 'Content-Length': '0'} 2023-07-29T00:24:55.259 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000011d1b4d0f6258b57-0064c45c57-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:55.259 DEBUG:boto:path=/hiiqos-58/obj00000000000000000046 2023-07-29T00:24:55.260 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000046 2023-07-29T00:24:55.260 DEBUG:boto:Method: PUT 2023-07-29T00:24:55.261 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000046 2023-07-29T00:24:55.262 DEBUG:boto:Data: 2023-07-29T00:24:55.263 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:55.264 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:55.265 DEBUG:boto:Port: 8000 2023-07-29T00:24:55.265 DEBUG:boto:Params: {} 2023-07-29T00:24:55.266 DEBUG:boto:Token: None 2023-07-29T00:24:55.267 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:55 GMT /hiiqos-58/obj00000000000000000046 2023-07-29T00:24:55.268 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oLC+1uQ+iHBVE59UH50PcnzcxT8= 2023-07-29T00:24:55.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oLC+1uQ+iHBVE59UH50PcnzcxT8='} 2023-07-29T00:24:55.312 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a03d0f35107135df-0064c45c57-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:55.313 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:55.314 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:55.314 DEBUG:boto:Method: HEAD 2023-07-29T00:24:55.314 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:55.315 DEBUG:boto:Data: 2023-07-29T00:24:55.315 DEBUG:boto:Headers: {} 2023-07-29T00:24:55.315 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:55.316 DEBUG:boto:Port: 8000 2023-07-29T00:24:55.316 DEBUG:boto:Params: {} 2023-07-29T00:24:55.316 DEBUG:boto:Token: None 2023-07-29T00:24:55.317 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:55 GMT /hiiqos-58/ 2023-07-29T00:24:55.318 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HyOqaYwDa5ZSg9TdbLPmcy5bYL0= 2023-07-29T00:24:55.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HyOqaYwDa5ZSg9TdbLPmcy5bYL0=', 'Content-Length': '0'} 2023-07-29T00:24:55.322 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004375ee92d92d39a3-0064c45c57-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:55.323 DEBUG:boto:path=/hiiqos-58/obj00000000000000000009 2023-07-29T00:24:55.324 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000009 2023-07-29T00:24:55.325 DEBUG:boto:Method: PUT 2023-07-29T00:24:55.326 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000009 2023-07-29T00:24:55.326 DEBUG:boto:Data: 2023-07-29T00:24:55.327 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:55.328 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:55.328 DEBUG:boto:Port: 8000 2023-07-29T00:24:55.328 DEBUG:boto:Params: {} 2023-07-29T00:24:55.329 DEBUG:boto:Token: None 2023-07-29T00:24:55.329 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:55 GMT /hiiqos-58/obj00000000000000000009 2023-07-29T00:24:55.330 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QkOkWi/um6tsVWZBl+Udstoqwjo= 2023-07-29T00:24:55.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QkOkWi/um6tsVWZBl+Udstoqwjo='} 2023-07-29T00:24:55.336 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003a5bea87e7962278-0064c45c57-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:55.337 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:55.338 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:55.338 DEBUG:boto:Method: HEAD 2023-07-29T00:24:55.339 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:55.339 DEBUG:boto:Data: 2023-07-29T00:24:55.340 DEBUG:boto:Headers: {} 2023-07-29T00:24:55.341 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:55.341 DEBUG:boto:Port: 8000 2023-07-29T00:24:55.342 DEBUG:boto:Params: {} 2023-07-29T00:24:55.342 DEBUG:boto:Token: None 2023-07-29T00:24:55.343 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:55 GMT /hiiqos-58/ 2023-07-29T00:24:55.343 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HyOqaYwDa5ZSg9TdbLPmcy5bYL0= 2023-07-29T00:24:55.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HyOqaYwDa5ZSg9TdbLPmcy5bYL0=', 'Content-Length': '0'} 2023-07-29T00:24:55.346 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d3b25749745cd95-0064c45c57-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:55.347 DEBUG:boto:path=/hiiqos-58/obj00000000000000000045 2023-07-29T00:24:55.347 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000045 2023-07-29T00:24:55.348 DEBUG:boto:Method: PUT 2023-07-29T00:24:55.348 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000045 2023-07-29T00:24:55.349 DEBUG:boto:Data: 2023-07-29T00:24:55.349 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:55.350 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:55.350 DEBUG:boto:Port: 8000 2023-07-29T00:24:55.350 DEBUG:boto:Params: {} 2023-07-29T00:24:55.351 DEBUG:boto:Token: None 2023-07-29T00:24:55.351 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:55 GMT /hiiqos-58/obj00000000000000000045 2023-07-29T00:24:55.352 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Nqx/WnrT4Ds/u9DWv85xpEmhsFk= 2023-07-29T00:24:55.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Nqx/WnrT4Ds/u9DWv85xpEmhsFk='} 2023-07-29T00:24:55.400 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007f493abe443c415f-0064c45c57-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:55.401 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:55.401 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:55.402 DEBUG:boto:Method: HEAD 2023-07-29T00:24:55.402 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:55.402 DEBUG:boto:Data: 2023-07-29T00:24:55.403 DEBUG:boto:Headers: {} 2023-07-29T00:24:55.403 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:55.403 DEBUG:boto:Port: 8000 2023-07-29T00:24:55.404 DEBUG:boto:Params: {} 2023-07-29T00:24:55.405 DEBUG:boto:Token: None 2023-07-29T00:24:55.405 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:55 GMT /hiiqos-58/ 2023-07-29T00:24:55.406 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HyOqaYwDa5ZSg9TdbLPmcy5bYL0= 2023-07-29T00:24:55.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HyOqaYwDa5ZSg9TdbLPmcy5bYL0=', 'Content-Length': '0'} 2023-07-29T00:24:55.409 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ba49cb899c8046c-0064c45c57-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:55.409 DEBUG:boto:path=/hiiqos-58/obj00000000000000000033 2023-07-29T00:24:55.410 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000033 2023-07-29T00:24:55.410 DEBUG:boto:Method: PUT 2023-07-29T00:24:55.411 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000033 2023-07-29T00:24:55.412 DEBUG:boto:Data: 2023-07-29T00:24:55.413 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:55.414 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:55.415 DEBUG:boto:Port: 8000 2023-07-29T00:24:55.415 DEBUG:boto:Params: {} 2023-07-29T00:24:55.416 DEBUG:boto:Token: None 2023-07-29T00:24:55.417 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:55 GMT /hiiqos-58/obj00000000000000000033 2023-07-29T00:24:55.417 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mSOYMmjbeRevxuZ8utwGjeUAb28= 2023-07-29T00:24:55.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mSOYMmjbeRevxuZ8utwGjeUAb28='} 2023-07-29T00:24:55.424 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000001e1f1ffaa25f5ef-0064c45c57-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:55.425 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:55.426 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:55.427 DEBUG:boto:Method: HEAD 2023-07-29T00:24:55.427 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:55.428 DEBUG:boto:Data: 2023-07-29T00:24:55.429 DEBUG:boto:Headers: {} 2023-07-29T00:24:55.430 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:55.430 DEBUG:boto:Port: 8000 2023-07-29T00:24:55.431 DEBUG:boto:Params: {} 2023-07-29T00:24:55.431 DEBUG:boto:Token: None 2023-07-29T00:24:55.432 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:55 GMT /hiiqos-58/ 2023-07-29T00:24:55.433 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HyOqaYwDa5ZSg9TdbLPmcy5bYL0= 2023-07-29T00:24:55.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HyOqaYwDa5ZSg9TdbLPmcy5bYL0=', 'Content-Length': '0'} 2023-07-29T00:24:55.436 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f95bd95a14eac89-0064c45c57-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:55.437 DEBUG:boto:path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:24:55.437 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:24:55.437 DEBUG:boto:Method: PUT 2023-07-29T00:24:55.438 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000002 2023-07-29T00:24:55.438 DEBUG:boto:Data: 2023-07-29T00:24:55.438 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:55.439 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:55.439 DEBUG:boto:Port: 8000 2023-07-29T00:24:55.439 DEBUG:boto:Params: {} 2023-07-29T00:24:55.440 DEBUG:boto:Token: None 2023-07-29T00:24:55.440 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:55 GMT /hiiqos-58/obj00000000000000000002 2023-07-29T00:24:55.441 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:F2y1PDfBSfz7cQC20mNVDIG/ohE= 2023-07-29T00:24:55.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:F2y1PDfBSfz7cQC20mNVDIG/ohE='} 2023-07-29T00:24:55.484 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000271c3c2240ef733d-0064c45c57-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:55.485 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:55.485 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:55.486 DEBUG:boto:Method: HEAD 2023-07-29T00:24:55.486 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:55.487 DEBUG:boto:Data: 2023-07-29T00:24:55.487 DEBUG:boto:Headers: {} 2023-07-29T00:24:55.487 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:55.488 DEBUG:boto:Port: 8000 2023-07-29T00:24:55.488 DEBUG:boto:Params: {} 2023-07-29T00:24:55.489 DEBUG:boto:Token: None 2023-07-29T00:24:55.489 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:55 GMT /hiiqos-58/ 2023-07-29T00:24:55.490 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HyOqaYwDa5ZSg9TdbLPmcy5bYL0= 2023-07-29T00:24:55.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HyOqaYwDa5ZSg9TdbLPmcy5bYL0=', 'Content-Length': '0'} 2023-07-29T00:24:55.493 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c54cd0d4d71475ca-0064c45c57-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:55.494 DEBUG:boto:path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:24:55.495 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:24:55.495 DEBUG:boto:Method: PUT 2023-07-29T00:24:55.496 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000013 2023-07-29T00:24:55.497 DEBUG:boto:Data: 2023-07-29T00:24:55.497 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:55.498 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:55.498 DEBUG:boto:Port: 8000 2023-07-29T00:24:55.499 DEBUG:boto:Params: {} 2023-07-29T00:24:55.500 DEBUG:boto:Token: None 2023-07-29T00:24:55.500 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:55 GMT /hiiqos-58/obj00000000000000000013 2023-07-29T00:24:55.501 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:prHYDg87Itvy3wRVhhdG4ZP83hg= 2023-07-29T00:24:55.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:prHYDg87Itvy3wRVhhdG4ZP83hg='} 2023-07-29T00:24:55.508 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009233e85f52ec6b9d-0064c45c57-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:55.508 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:55.509 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:55.509 DEBUG:boto:Method: HEAD 2023-07-29T00:24:55.509 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:55.510 DEBUG:boto:Data: 2023-07-29T00:24:55.510 DEBUG:boto:Headers: {} 2023-07-29T00:24:55.511 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:55.511 DEBUG:boto:Port: 8000 2023-07-29T00:24:55.511 DEBUG:boto:Params: {} 2023-07-29T00:24:55.512 DEBUG:boto:Token: None 2023-07-29T00:24:55.512 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:55 GMT /hiiqos-58/ 2023-07-29T00:24:55.513 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HyOqaYwDa5ZSg9TdbLPmcy5bYL0= 2023-07-29T00:24:55.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HyOqaYwDa5ZSg9TdbLPmcy5bYL0=', 'Content-Length': '0'} 2023-07-29T00:24:55.516 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000124c4c835535bcee-0064c45c57-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:55.517 DEBUG:boto:path=/hiiqos-58/obj00000000000000000026 2023-07-29T00:24:55.517 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000026 2023-07-29T00:24:55.518 DEBUG:boto:Method: PUT 2023-07-29T00:24:55.518 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000026 2023-07-29T00:24:55.518 DEBUG:boto:Data: 2023-07-29T00:24:55.519 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:55.519 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:55.519 DEBUG:boto:Port: 8000 2023-07-29T00:24:55.520 DEBUG:boto:Params: {} 2023-07-29T00:24:55.520 DEBUG:boto:Token: None 2023-07-29T00:24:55.521 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:55 GMT /hiiqos-58/obj00000000000000000026 2023-07-29T00:24:55.521 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0iJrB9FLlPluYAxwn8yhl203LhY= 2023-07-29T00:24:55.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0iJrB9FLlPluYAxwn8yhl203LhY='} 2023-07-29T00:24:55.569 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fd4cb68ef99bc84a-0064c45c57-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:55.570 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:55.570 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:55.570 DEBUG:boto:Method: HEAD 2023-07-29T00:24:55.571 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:55.571 DEBUG:boto:Data: 2023-07-29T00:24:55.571 DEBUG:boto:Headers: {} 2023-07-29T00:24:55.572 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:55.572 DEBUG:boto:Port: 8000 2023-07-29T00:24:55.572 DEBUG:boto:Params: {} 2023-07-29T00:24:55.573 DEBUG:boto:Token: None 2023-07-29T00:24:55.573 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:55 GMT /hiiqos-58/ 2023-07-29T00:24:55.574 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HyOqaYwDa5ZSg9TdbLPmcy5bYL0= 2023-07-29T00:24:55.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HyOqaYwDa5ZSg9TdbLPmcy5bYL0=', 'Content-Length': '0'} 2023-07-29T00:24:55.577 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005888197d3c63f82f-0064c45c57-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:55.578 DEBUG:boto:path=/hiiqos-58/obj00000000000000000021 2023-07-29T00:24:55.579 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000021 2023-07-29T00:24:55.582 DEBUG:boto:Method: PUT 2023-07-29T00:24:55.583 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000021 2023-07-29T00:24:55.583 DEBUG:boto:Data: 2023-07-29T00:24:55.584 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:55.585 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:55.585 DEBUG:boto:Port: 8000 2023-07-29T00:24:55.586 DEBUG:boto:Params: {} 2023-07-29T00:24:55.587 DEBUG:boto:Token: None 2023-07-29T00:24:55.588 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:55 GMT /hiiqos-58/obj00000000000000000021 2023-07-29T00:24:55.588 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AlXP7WRaVW0A6eZ2U0uWmiGwSnU= 2023-07-29T00:24:55.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AlXP7WRaVW0A6eZ2U0uWmiGwSnU='} 2023-07-29T00:24:55.595 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006ad4491582c4d689-0064c45c57-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:55.596 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:55.597 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:55.598 DEBUG:boto:Method: HEAD 2023-07-29T00:24:55.599 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:55.599 DEBUG:boto:Data: 2023-07-29T00:24:55.600 DEBUG:boto:Headers: {} 2023-07-29T00:24:55.604 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:55.605 DEBUG:boto:Port: 8000 2023-07-29T00:24:55.605 DEBUG:boto:Params: {} 2023-07-29T00:24:55.605 DEBUG:boto:Token: None 2023-07-29T00:24:55.606 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:55 GMT /hiiqos-58/ 2023-07-29T00:24:55.606 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HyOqaYwDa5ZSg9TdbLPmcy5bYL0= 2023-07-29T00:24:55.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HyOqaYwDa5ZSg9TdbLPmcy5bYL0=', 'Content-Length': '0'} 2023-07-29T00:24:55.610 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cf88f8a214d62a31-0064c45c57-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:55.610 DEBUG:boto:path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:24:55.611 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:24:55.611 DEBUG:boto:Method: PUT 2023-07-29T00:24:55.612 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000000 2023-07-29T00:24:55.612 DEBUG:boto:Data: 2023-07-29T00:24:55.613 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:55.614 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:55.614 DEBUG:boto:Port: 8000 2023-07-29T00:24:55.615 DEBUG:boto:Params: {} 2023-07-29T00:24:55.615 DEBUG:boto:Token: None 2023-07-29T00:24:55.615 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:55 GMT /hiiqos-58/obj00000000000000000000 2023-07-29T00:24:55.616 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PoagxmLy41zJiatcX1lWEl5HZY4= 2023-07-29T00:24:55.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PoagxmLy41zJiatcX1lWEl5HZY4='} 2023-07-29T00:24:55.661 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000075a32dfc01b4ffc0-0064c45c57-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:55.661 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:55.662 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:55.662 DEBUG:boto:Method: HEAD 2023-07-29T00:24:55.662 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:55.663 DEBUG:boto:Data: 2023-07-29T00:24:55.663 DEBUG:boto:Headers: {} 2023-07-29T00:24:55.663 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:55.664 DEBUG:boto:Port: 8000 2023-07-29T00:24:55.664 DEBUG:boto:Params: {} 2023-07-29T00:24:55.664 DEBUG:boto:Token: None 2023-07-29T00:24:55.665 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:55 GMT /hiiqos-58/ 2023-07-29T00:24:55.665 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HyOqaYwDa5ZSg9TdbLPmcy5bYL0= 2023-07-29T00:24:55.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HyOqaYwDa5ZSg9TdbLPmcy5bYL0=', 'Content-Length': '0'} 2023-07-29T00:24:55.668 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004cdcd9093ae7ad09-0064c45c57-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:55.669 DEBUG:boto:path=/hiiqos-58/obj00000000000000000007 2023-07-29T00:24:55.670 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000007 2023-07-29T00:24:55.670 DEBUG:boto:Method: PUT 2023-07-29T00:24:55.670 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000007 2023-07-29T00:24:55.671 DEBUG:boto:Data: 2023-07-29T00:24:55.671 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:55.671 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:55.672 DEBUG:boto:Port: 8000 2023-07-29T00:24:55.672 DEBUG:boto:Params: {} 2023-07-29T00:24:55.673 DEBUG:boto:Token: None 2023-07-29T00:24:55.673 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:55 GMT /hiiqos-58/obj00000000000000000007 2023-07-29T00:24:55.674 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J5284WyjZXOsZHRUQsPdIuHcEKs= 2023-07-29T00:24:55.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J5284WyjZXOsZHRUQsPdIuHcEKs='} 2023-07-29T00:24:55.680 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000be9ce6896355d0f4-0064c45c57-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:55.681 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:55.681 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:55.681 DEBUG:boto:Method: HEAD 2023-07-29T00:24:55.682 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:55.682 DEBUG:boto:Data: 2023-07-29T00:24:55.682 DEBUG:boto:Headers: {} 2023-07-29T00:24:55.683 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:55.683 DEBUG:boto:Port: 8000 2023-07-29T00:24:55.683 DEBUG:boto:Params: {} 2023-07-29T00:24:55.684 DEBUG:boto:Token: None 2023-07-29T00:24:55.684 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:55 GMT /hiiqos-58/ 2023-07-29T00:24:55.685 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HyOqaYwDa5ZSg9TdbLPmcy5bYL0= 2023-07-29T00:24:55.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HyOqaYwDa5ZSg9TdbLPmcy5bYL0=', 'Content-Length': '0'} 2023-07-29T00:24:55.688 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a6177aad8692f635-0064c45c57-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:55.689 DEBUG:boto:path=/hiiqos-58/obj00000000000000000039 2023-07-29T00:24:55.689 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000039 2023-07-29T00:24:55.690 DEBUG:boto:Method: PUT 2023-07-29T00:24:55.690 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000039 2023-07-29T00:24:55.691 DEBUG:boto:Data: 2023-07-29T00:24:55.692 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:55.692 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:55.693 DEBUG:boto:Port: 8000 2023-07-29T00:24:55.693 DEBUG:boto:Params: {} 2023-07-29T00:24:55.694 DEBUG:boto:Token: None 2023-07-29T00:24:55.694 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:55 GMT /hiiqos-58/obj00000000000000000039 2023-07-29T00:24:55.694 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JxumWD/Rd6D94sUbxS596ZMrLYU= 2023-07-29T00:24:55.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JxumWD/Rd6D94sUbxS596ZMrLYU='} 2023-07-29T00:24:55.744 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004d983418d0a9306a-0064c45c57-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:55.746 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:55.747 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:55.747 DEBUG:boto:Method: HEAD 2023-07-29T00:24:55.748 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:55.749 DEBUG:boto:Data: 2023-07-29T00:24:55.749 DEBUG:boto:Headers: {} 2023-07-29T00:24:55.750 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:55.751 DEBUG:boto:Port: 8000 2023-07-29T00:24:55.752 DEBUG:boto:Params: {} 2023-07-29T00:24:55.752 DEBUG:boto:Token: None 2023-07-29T00:24:55.753 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:55 GMT /hiiqos-58/ 2023-07-29T00:24:55.754 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HyOqaYwDa5ZSg9TdbLPmcy5bYL0= 2023-07-29T00:24:55.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HyOqaYwDa5ZSg9TdbLPmcy5bYL0=', 'Content-Length': '0'} 2023-07-29T00:24:55.758 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f66854089d536d6-0064c45c57-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:55.758 DEBUG:boto:path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:24:55.759 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:24:55.759 DEBUG:boto:Method: PUT 2023-07-29T00:24:55.760 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000003 2023-07-29T00:24:55.760 DEBUG:boto:Data: 2023-07-29T00:24:55.760 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:55.761 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:55.761 DEBUG:boto:Port: 8000 2023-07-29T00:24:55.762 DEBUG:boto:Params: {} 2023-07-29T00:24:55.762 DEBUG:boto:Token: None 2023-07-29T00:24:55.763 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:55 GMT /hiiqos-58/obj00000000000000000003 2023-07-29T00:24:55.763 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jTr+BZoCbP9L+u1qgAAszLyVNqE= 2023-07-29T00:24:55.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jTr+BZoCbP9L+u1qgAAszLyVNqE='} 2023-07-29T00:24:55.770 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eed518af953eb858-0064c45c57-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:55.770 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:55.771 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:55.771 DEBUG:boto:Method: HEAD 2023-07-29T00:24:55.771 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:55.772 DEBUG:boto:Data: 2023-07-29T00:24:55.772 DEBUG:boto:Headers: {} 2023-07-29T00:24:55.773 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:55.773 DEBUG:boto:Port: 8000 2023-07-29T00:24:55.773 DEBUG:boto:Params: {} 2023-07-29T00:24:55.774 DEBUG:boto:Token: None 2023-07-29T00:24:55.774 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:55 GMT /hiiqos-58/ 2023-07-29T00:24:55.775 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HyOqaYwDa5ZSg9TdbLPmcy5bYL0= 2023-07-29T00:24:55.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HyOqaYwDa5ZSg9TdbLPmcy5bYL0=', 'Content-Length': '0'} 2023-07-29T00:24:55.778 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a2927fb3f5b47854-0064c45c57-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:55.779 DEBUG:boto:path=/hiiqos-58/obj00000000000000000055 2023-07-29T00:24:55.779 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000055 2023-07-29T00:24:55.779 DEBUG:boto:Method: PUT 2023-07-29T00:24:55.780 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000055 2023-07-29T00:24:55.780 DEBUG:boto:Data: 2023-07-29T00:24:55.781 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:55.781 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:55.781 DEBUG:boto:Port: 8000 2023-07-29T00:24:55.782 DEBUG:boto:Params: {} 2023-07-29T00:24:55.782 DEBUG:boto:Token: None 2023-07-29T00:24:55.782 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:55 GMT /hiiqos-58/obj00000000000000000055 2023-07-29T00:24:55.783 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lWBvdNtKixoVpijH3qIU3JQ8Dtg= 2023-07-29T00:24:55.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lWBvdNtKixoVpijH3qIU3JQ8Dtg='} 2023-07-29T00:24:55.828 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000017103b33f8fedde-0064c45c57-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:55.829 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:55.830 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:55.830 DEBUG:boto:Method: HEAD 2023-07-29T00:24:55.830 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:55.831 DEBUG:boto:Data: 2023-07-29T00:24:55.831 DEBUG:boto:Headers: {} 2023-07-29T00:24:55.831 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:55.832 DEBUG:boto:Port: 8000 2023-07-29T00:24:55.833 DEBUG:boto:Params: {} 2023-07-29T00:24:55.834 DEBUG:boto:Token: None 2023-07-29T00:24:55.834 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:55 GMT /hiiqos-58/ 2023-07-29T00:24:55.835 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HyOqaYwDa5ZSg9TdbLPmcy5bYL0= 2023-07-29T00:24:55.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HyOqaYwDa5ZSg9TdbLPmcy5bYL0=', 'Content-Length': '0'} 2023-07-29T00:24:55.837 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-07-29T00:24:55.839 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000052651a998dd797fb-0064c45c57-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:55.840 DEBUG:boto:path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:24:55.841 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:24:55.841 DEBUG:boto:Method: PUT 2023-07-29T00:24:55.841 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000011 2023-07-29T00:24:55.842 DEBUG:boto:Data: 2023-07-29T00:24:55.842 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:55.842 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:55.843 DEBUG:boto:Port: 8000 2023-07-29T00:24:55.843 DEBUG:boto:Params: {} 2023-07-29T00:24:55.843 DEBUG:boto:Token: None 2023-07-29T00:24:55.844 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:55 GMT /hiiqos-58/obj00000000000000000011 2023-07-29T00:24:55.844 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Ogw35iX3sss7oaOOY8J3GYSJ5lg= 2023-07-29T00:24:55.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Ogw35iX3sss7oaOOY8J3GYSJ5lg='} 2023-07-29T00:24:55.851 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006e526d9f11edc65d-0064c45c57-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:55.852 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:55.852 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:55.852 DEBUG:boto:Method: HEAD 2023-07-29T00:24:55.853 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:55.853 DEBUG:boto:Data: 2023-07-29T00:24:55.853 DEBUG:boto:Headers: {} 2023-07-29T00:24:55.854 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:55.854 DEBUG:boto:Port: 8000 2023-07-29T00:24:55.854 DEBUG:boto:Params: {} 2023-07-29T00:24:55.855 DEBUG:boto:Token: None 2023-07-29T00:24:55.855 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:55 GMT /hiiqos-58/ 2023-07-29T00:24:55.856 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HyOqaYwDa5ZSg9TdbLPmcy5bYL0= 2023-07-29T00:24:55.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HyOqaYwDa5ZSg9TdbLPmcy5bYL0=', 'Content-Length': '0'} 2023-07-29T00:24:55.859 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000370a1b6e94f63531-0064c45c57-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:55.859 DEBUG:boto:path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:24:55.860 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:24:55.860 DEBUG:boto:Method: PUT 2023-07-29T00:24:55.860 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000012 2023-07-29T00:24:55.861 DEBUG:boto:Data: 2023-07-29T00:24:55.861 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:55.861 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:55.862 DEBUG:boto:Port: 8000 2023-07-29T00:24:55.862 DEBUG:boto:Params: {} 2023-07-29T00:24:55.862 DEBUG:boto:Token: None 2023-07-29T00:24:55.863 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:55 GMT /hiiqos-58/obj00000000000000000012 2023-07-29T00:24:55.863 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:etokUqJw/aXvAJVnVSWH6gJq6yE= 2023-07-29T00:24:55.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:etokUqJw/aXvAJVnVSWH6gJq6yE='} 2023-07-29T00:24:55.870 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d44aa8d711fe3d64-0064c45c57-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:55.870 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:55.871 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:55.871 DEBUG:boto:Method: HEAD 2023-07-29T00:24:55.871 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:55.872 DEBUG:boto:Data: 2023-07-29T00:24:55.872 DEBUG:boto:Headers: {} 2023-07-29T00:24:55.872 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:55.873 DEBUG:boto:Port: 8000 2023-07-29T00:24:55.873 DEBUG:boto:Params: {} 2023-07-29T00:24:55.873 DEBUG:boto:Token: None 2023-07-29T00:24:55.874 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:55 GMT /hiiqos-58/ 2023-07-29T00:24:55.874 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HyOqaYwDa5ZSg9TdbLPmcy5bYL0= 2023-07-29T00:24:55.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HyOqaYwDa5ZSg9TdbLPmcy5bYL0=', 'Content-Length': '0'} 2023-07-29T00:24:55.877 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e4c7b3d5d2c3f904-0064c45c57-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:55.878 DEBUG:boto:path=/hiiqos-58/obj00000000000000000040 2023-07-29T00:24:55.878 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000040 2023-07-29T00:24:55.879 DEBUG:boto:Method: PUT 2023-07-29T00:24:55.879 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000040 2023-07-29T00:24:55.879 DEBUG:boto:Data: 2023-07-29T00:24:55.880 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:55.880 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:55.880 DEBUG:boto:Port: 8000 2023-07-29T00:24:55.881 DEBUG:boto:Params: {} 2023-07-29T00:24:55.881 DEBUG:boto:Token: None 2023-07-29T00:24:55.882 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:55 GMT /hiiqos-58/obj00000000000000000040 2023-07-29T00:24:55.882 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mE0rWoGb1P3GuFhm6VKQ3Kr3FCc= 2023-07-29T00:24:55.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mE0rWoGb1P3GuFhm6VKQ3Kr3FCc='} 2023-07-29T00:24:55.925 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001b294173766b6365-0064c45c57-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:55.925 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:55.926 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:55.926 DEBUG:boto:Method: HEAD 2023-07-29T00:24:55.926 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:55.927 DEBUG:boto:Data: 2023-07-29T00:24:55.927 DEBUG:boto:Headers: {} 2023-07-29T00:24:55.928 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:55.928 DEBUG:boto:Port: 8000 2023-07-29T00:24:55.929 DEBUG:boto:Params: {} 2023-07-29T00:24:55.929 DEBUG:boto:Token: None 2023-07-29T00:24:55.930 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:55 GMT /hiiqos-58/ 2023-07-29T00:24:55.930 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HyOqaYwDa5ZSg9TdbLPmcy5bYL0= 2023-07-29T00:24:55.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HyOqaYwDa5ZSg9TdbLPmcy5bYL0=', 'Content-Length': '0'} 2023-07-29T00:24:55.933 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001ccada33204bcc92-0064c45c57-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:55.934 DEBUG:boto:path=/hiiqos-58/obj00000000000000000008 2023-07-29T00:24:55.934 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000008 2023-07-29T00:24:55.934 DEBUG:boto:Method: PUT 2023-07-29T00:24:55.935 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000008 2023-07-29T00:24:55.935 DEBUG:boto:Data: 2023-07-29T00:24:55.936 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:55.936 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:55.936 DEBUG:boto:Port: 8000 2023-07-29T00:24:55.937 DEBUG:boto:Params: {} 2023-07-29T00:24:55.937 DEBUG:boto:Token: None 2023-07-29T00:24:55.938 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:55 GMT /hiiqos-58/obj00000000000000000008 2023-07-29T00:24:55.938 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EXrdi3wkj/9V7n8gUQOR2diGA/0= 2023-07-29T00:24:55.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EXrdi3wkj/9V7n8gUQOR2diGA/0='} 2023-07-29T00:24:55.943 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000040783457a5893084-0064c45c57-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:55.944 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:55.944 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:55.944 DEBUG:boto:Method: HEAD 2023-07-29T00:24:55.945 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:55.945 DEBUG:boto:Data: 2023-07-29T00:24:55.946 DEBUG:boto:Headers: {} 2023-07-29T00:24:55.946 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:55.946 DEBUG:boto:Port: 8000 2023-07-29T00:24:55.947 DEBUG:boto:Params: {} 2023-07-29T00:24:55.947 DEBUG:boto:Token: None 2023-07-29T00:24:55.947 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:55 GMT /hiiqos-58/ 2023-07-29T00:24:55.948 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HyOqaYwDa5ZSg9TdbLPmcy5bYL0= 2023-07-29T00:24:55.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HyOqaYwDa5ZSg9TdbLPmcy5bYL0=', 'Content-Length': '0'} 2023-07-29T00:24:55.951 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d417ce37fd6bae3a-0064c45c57-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:55.952 DEBUG:boto:path=/hiiqos-58/obj00000000000000000050 2023-07-29T00:24:55.952 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000050 2023-07-29T00:24:55.953 DEBUG:boto:Method: PUT 2023-07-29T00:24:55.953 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000050 2023-07-29T00:24:55.954 DEBUG:boto:Data: 2023-07-29T00:24:55.954 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:55.954 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:55.955 DEBUG:boto:Port: 8000 2023-07-29T00:24:55.955 DEBUG:boto:Params: {} 2023-07-29T00:24:55.955 DEBUG:boto:Token: None 2023-07-29T00:24:55.956 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:55 GMT /hiiqos-58/obj00000000000000000050 2023-07-29T00:24:55.956 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:If2F+CiIjouQoZPD85veN7onvZ0= 2023-07-29T00:24:55.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:55 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:If2F+CiIjouQoZPD85veN7onvZ0='} 2023-07-29T00:24:56.004 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d860ba5d0b56e451-0064c45c57-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:56.005 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:56.005 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:56.006 DEBUG:boto:Method: HEAD 2023-07-29T00:24:56.006 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:56.007 DEBUG:boto:Data: 2023-07-29T00:24:56.007 DEBUG:boto:Headers: {} 2023-07-29T00:24:56.007 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:56.008 DEBUG:boto:Port: 8000 2023-07-29T00:24:56.008 DEBUG:boto:Params: {} 2023-07-29T00:24:56.008 DEBUG:boto:Token: None 2023-07-29T00:24:56.009 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:56 GMT /hiiqos-58/ 2023-07-29T00:24:56.009 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Daq7rVxx4zjpp/4mkT3GlUKRs8Y= 2023-07-29T00:24:56.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Daq7rVxx4zjpp/4mkT3GlUKRs8Y=', 'Content-Length': '0'} 2023-07-29T00:24:56.013 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001c7fd2f725e800ed-0064c45c57-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:56.013 DEBUG:boto:path=/hiiqos-58/obj00000000000000000014 2023-07-29T00:24:56.014 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000014 2023-07-29T00:24:56.014 DEBUG:boto:Method: PUT 2023-07-29T00:24:56.014 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000014 2023-07-29T00:24:56.015 DEBUG:boto:Data: 2023-07-29T00:24:56.015 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:56.015 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:56.016 DEBUG:boto:Port: 8000 2023-07-29T00:24:56.016 DEBUG:boto:Params: {} 2023-07-29T00:24:56.017 DEBUG:boto:Token: None 2023-07-29T00:24:56.017 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:56 GMT /hiiqos-58/obj00000000000000000014 2023-07-29T00:24:56.017 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:odkw7GQtXyUFqex+Ti1ZmLrY1gE= 2023-07-29T00:24:56.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:odkw7GQtXyUFqex+Ti1ZmLrY1gE='} 2023-07-29T00:24:56.024 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e89909b08c35df3d-0064c45c57-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:56.024 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:56.025 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:56.025 DEBUG:boto:Method: HEAD 2023-07-29T00:24:56.026 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:56.026 DEBUG:boto:Data: 2023-07-29T00:24:56.026 DEBUG:boto:Headers: {} 2023-07-29T00:24:56.027 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:56.027 DEBUG:boto:Port: 8000 2023-07-29T00:24:56.027 DEBUG:boto:Params: {} 2023-07-29T00:24:56.028 DEBUG:boto:Token: None 2023-07-29T00:24:56.028 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:56 GMT /hiiqos-58/ 2023-07-29T00:24:56.029 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Daq7rVxx4zjpp/4mkT3GlUKRs8Y= 2023-07-29T00:24:56.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Daq7rVxx4zjpp/4mkT3GlUKRs8Y=', 'Content-Length': '0'} 2023-07-29T00:24:56.032 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e9e5f7b84a47f50-0064c45c57-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:56.032 DEBUG:boto:path=/hiiqos-58/obj00000000000000000058 2023-07-29T00:24:56.033 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000058 2023-07-29T00:24:56.033 DEBUG:boto:Method: PUT 2023-07-29T00:24:56.033 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000058 2023-07-29T00:24:56.034 DEBUG:boto:Data: 2023-07-29T00:24:56.034 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:56.035 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:56.035 DEBUG:boto:Port: 8000 2023-07-29T00:24:56.036 DEBUG:boto:Params: {} 2023-07-29T00:24:56.036 DEBUG:boto:Token: None 2023-07-29T00:24:56.037 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:56 GMT /hiiqos-58/obj00000000000000000058 2023-07-29T00:24:56.037 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:D0eBPJi73p0oJrPnM7S4OVt/4to= 2023-07-29T00:24:56.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:D0eBPJi73p0oJrPnM7S4OVt/4to='} 2023-07-29T00:24:56.080 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008a128bfa66ffff14-0064c45c57-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:56.081 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-58', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:24:56.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-58', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:24:56.082 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-58 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:24:56.131 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:24:56.131 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:24:56.177 INFO:teuthology.orchestra.run.smithi101.stdout:tenant: 2023-07-29T00:24:56.178 INFO:teuthology.orchestra.run.smithi101.stdout:bucket name: hiiqos-58 2023-07-29T00:24:56.187 INFO:teuthology.orchestra.run.smithi101.stdout:total entries: 35 2023-07-29T00:24:56.236 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:24:56.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:24:56.237 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 29 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:24:56.285 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:24:56.285 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:24:56.325 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:24:56.325 INFO:teuthology.orchestra.run.smithi101.stdout: "objs": [ 2023-07-29T00:24:56.326 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000006", 2023-07-29T00:24:56.326 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000059", 2023-07-29T00:24:56.326 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000027", 2023-07-29T00:24:56.326 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000004", 2023-07-29T00:24:56.326 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000011", 2023-07-29T00:24:56.327 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000002", 2023-07-29T00:24:56.327 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000019", 2023-07-29T00:24:56.327 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000049", 2023-07-29T00:24:56.327 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000017", 2023-07-29T00:24:56.327 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000008", 2023-07-29T00:24:56.327 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000015", 2023-07-29T00:24:56.327 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000031", 2023-07-29T00:24:56.328 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000022", 2023-07-29T00:24:56.328 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000013", 2023-07-29T00:24:56.328 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000020", 2023-07-29T00:24:56.328 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000037", 2023-07-29T00:24:56.328 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000005", 2023-07-29T00:24:56.328 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000012", 2023-07-29T00:24:56.329 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000003", 2023-07-29T00:24:56.329 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000042", 2023-07-29T00:24:56.329 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000010", 2023-07-29T00:24:56.329 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000001", 2023-07-29T00:24:56.329 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000009", 2023-07-29T00:24:56.329 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000025", 2023-07-29T00:24:56.329 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000016", 2023-07-29T00:24:56.330 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000000", 2023-07-29T00:24:56.330 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000014", 2023-07-29T00:24:56.330 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000030", 2023-07-29T00:24:56.330 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000021" 2023-07-29T00:24:56.330 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:24:56.330 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:24:56.331 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:56.332 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:56.332 DEBUG:boto:Method: HEAD 2023-07-29T00:24:56.332 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:56.333 DEBUG:boto:Data: 2023-07-29T00:24:56.333 DEBUG:boto:Headers: {} 2023-07-29T00:24:56.334 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:56.334 DEBUG:boto:Port: 8000 2023-07-29T00:24:56.334 DEBUG:boto:Params: {} 2023-07-29T00:24:56.335 DEBUG:boto:Token: None 2023-07-29T00:24:56.335 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:56 GMT /hiiqos-58/ 2023-07-29T00:24:56.336 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Daq7rVxx4zjpp/4mkT3GlUKRs8Y= 2023-07-29T00:24:56.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Daq7rVxx4zjpp/4mkT3GlUKRs8Y=', 'Content-Length': '0'} 2023-07-29T00:24:56.339 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000026966b63206f5e7e-0064c45c58-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:56.340 DEBUG:boto:path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:24:56.341 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:24:56.341 DEBUG:boto:Method: PUT 2023-07-29T00:24:56.341 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000001 2023-07-29T00:24:56.342 DEBUG:boto:Data: 2023-07-29T00:24:56.343 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:56.344 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:56.345 DEBUG:boto:Port: 8000 2023-07-29T00:24:56.345 DEBUG:boto:Params: {} 2023-07-29T00:24:56.346 DEBUG:boto:Token: None 2023-07-29T00:24:56.347 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:56 GMT /hiiqos-58/obj00000000000000000001 2023-07-29T00:24:56.348 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sLjJ11X/xgUaMObm6zBAvN5SS1Y= 2023-07-29T00:24:56.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sLjJ11X/xgUaMObm6zBAvN5SS1Y='} 2023-07-29T00:24:56.355 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000091d9af060d7f0ae1-0064c45c58-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:56.356 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:56.356 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:56.357 DEBUG:boto:Method: HEAD 2023-07-29T00:24:56.357 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:56.358 DEBUG:boto:Data: 2023-07-29T00:24:56.358 DEBUG:boto:Headers: {} 2023-07-29T00:24:56.359 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:56.359 DEBUG:boto:Port: 8000 2023-07-29T00:24:56.359 DEBUG:boto:Params: {} 2023-07-29T00:24:56.360 DEBUG:boto:Token: None 2023-07-29T00:24:56.360 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:56 GMT /hiiqos-58/ 2023-07-29T00:24:56.361 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Daq7rVxx4zjpp/4mkT3GlUKRs8Y= 2023-07-29T00:24:56.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Daq7rVxx4zjpp/4mkT3GlUKRs8Y=', 'Content-Length': '0'} 2023-07-29T00:24:56.364 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f6a7c70aa75df952-0064c45c58-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:56.365 DEBUG:boto:path=/hiiqos-58/obj00000000000000000008 2023-07-29T00:24:56.365 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000008 2023-07-29T00:24:56.365 DEBUG:boto:Method: PUT 2023-07-29T00:24:56.366 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000008 2023-07-29T00:24:56.366 DEBUG:boto:Data: 2023-07-29T00:24:56.367 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:56.367 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:56.367 DEBUG:boto:Port: 8000 2023-07-29T00:24:56.368 DEBUG:boto:Params: {} 2023-07-29T00:24:56.368 DEBUG:boto:Token: None 2023-07-29T00:24:56.369 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:56 GMT /hiiqos-58/obj00000000000000000008 2023-07-29T00:24:56.369 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:L0P/UfyQjHio7IFXs1IpaOzXECA= 2023-07-29T00:24:56.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:L0P/UfyQjHio7IFXs1IpaOzXECA='} 2023-07-29T00:24:56.375 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003f381ba77c5f790f-0064c45c58-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:56.376 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:56.377 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:56.377 DEBUG:boto:Method: HEAD 2023-07-29T00:24:56.377 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:56.378 DEBUG:boto:Data: 2023-07-29T00:24:56.378 DEBUG:boto:Headers: {} 2023-07-29T00:24:56.378 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:56.379 DEBUG:boto:Port: 8000 2023-07-29T00:24:56.379 DEBUG:boto:Params: {} 2023-07-29T00:24:56.380 DEBUG:boto:Token: None 2023-07-29T00:24:56.380 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:56 GMT /hiiqos-58/ 2023-07-29T00:24:56.380 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Daq7rVxx4zjpp/4mkT3GlUKRs8Y= 2023-07-29T00:24:56.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Daq7rVxx4zjpp/4mkT3GlUKRs8Y=', 'Content-Length': '0'} 2023-07-29T00:24:56.384 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001037124b5e483e9b-0064c45c58-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:56.384 DEBUG:boto:path=/hiiqos-58/obj00000000000000000030 2023-07-29T00:24:56.385 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000030 2023-07-29T00:24:56.385 DEBUG:boto:Method: PUT 2023-07-29T00:24:56.386 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000030 2023-07-29T00:24:56.386 DEBUG:boto:Data: 2023-07-29T00:24:56.386 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:56.387 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:56.387 DEBUG:boto:Port: 8000 2023-07-29T00:24:56.387 DEBUG:boto:Params: {} 2023-07-29T00:24:56.388 DEBUG:boto:Token: None 2023-07-29T00:24:56.388 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:56 GMT /hiiqos-58/obj00000000000000000030 2023-07-29T00:24:56.389 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Px4ASU8ktQzKaTFDrFQsmYHK9n8= 2023-07-29T00:24:56.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Px4ASU8ktQzKaTFDrFQsmYHK9n8='} 2023-07-29T00:24:56.395 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fefb40b746c27339-0064c45c58-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:56.395 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:56.396 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:56.396 DEBUG:boto:Method: HEAD 2023-07-29T00:24:56.396 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:56.397 DEBUG:boto:Data: 2023-07-29T00:24:56.397 DEBUG:boto:Headers: {} 2023-07-29T00:24:56.397 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:56.398 DEBUG:boto:Port: 8000 2023-07-29T00:24:56.398 DEBUG:boto:Params: {} 2023-07-29T00:24:56.399 DEBUG:boto:Token: None 2023-07-29T00:24:56.399 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:56 GMT /hiiqos-58/ 2023-07-29T00:24:56.399 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Daq7rVxx4zjpp/4mkT3GlUKRs8Y= 2023-07-29T00:24:56.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Daq7rVxx4zjpp/4mkT3GlUKRs8Y=', 'Content-Length': '0'} 2023-07-29T00:24:56.403 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000827e1ad5403111f6-0064c45c58-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:56.403 DEBUG:boto:path=/hiiqos-58/obj00000000000000000016 2023-07-29T00:24:56.404 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000016 2023-07-29T00:24:56.404 DEBUG:boto:Method: PUT 2023-07-29T00:24:56.404 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000016 2023-07-29T00:24:56.405 DEBUG:boto:Data: 2023-07-29T00:24:56.405 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:56.405 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:56.406 DEBUG:boto:Port: 8000 2023-07-29T00:24:56.406 DEBUG:boto:Params: {} 2023-07-29T00:24:56.407 DEBUG:boto:Token: None 2023-07-29T00:24:56.407 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:56 GMT /hiiqos-58/obj00000000000000000016 2023-07-29T00:24:56.407 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ffBXdZYEAB0U3xbHY5jnWPIkKlE= 2023-07-29T00:24:56.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ffBXdZYEAB0U3xbHY5jnWPIkKlE='} 2023-07-29T00:24:56.452 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000343dbf9ab2e66cb0-0064c45c58-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:56.453 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:56.453 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:56.454 DEBUG:boto:Method: HEAD 2023-07-29T00:24:56.454 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:56.455 DEBUG:boto:Data: 2023-07-29T00:24:56.455 DEBUG:boto:Headers: {} 2023-07-29T00:24:56.455 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:56.456 DEBUG:boto:Port: 8000 2023-07-29T00:24:56.456 DEBUG:boto:Params: {} 2023-07-29T00:24:56.457 DEBUG:boto:Token: None 2023-07-29T00:24:56.457 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:56 GMT /hiiqos-58/ 2023-07-29T00:24:56.457 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Daq7rVxx4zjpp/4mkT3GlUKRs8Y= 2023-07-29T00:24:56.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Daq7rVxx4zjpp/4mkT3GlUKRs8Y=', 'Content-Length': '0'} 2023-07-29T00:24:56.462 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092ded17b799d2264-0064c45c58-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:56.462 DEBUG:boto:path=/hiiqos-58/obj00000000000000000059 2023-07-29T00:24:56.463 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000059 2023-07-29T00:24:56.463 DEBUG:boto:Method: PUT 2023-07-29T00:24:56.463 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000059 2023-07-29T00:24:56.464 DEBUG:boto:Data: 2023-07-29T00:24:56.464 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:56.465 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:56.465 DEBUG:boto:Port: 8000 2023-07-29T00:24:56.466 DEBUG:boto:Params: {} 2023-07-29T00:24:56.466 DEBUG:boto:Token: None 2023-07-29T00:24:56.467 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:56 GMT /hiiqos-58/obj00000000000000000059 2023-07-29T00:24:56.467 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:o6KAsmvXLwow2OZ3CucS6WoeYuw= 2023-07-29T00:24:56.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:o6KAsmvXLwow2OZ3CucS6WoeYuw='} 2023-07-29T00:24:56.473 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b5582c350a1ead85-0064c45c58-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:56.474 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:56.475 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:56.475 DEBUG:boto:Method: HEAD 2023-07-29T00:24:56.476 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:56.477 DEBUG:boto:Data: 2023-07-29T00:24:56.477 DEBUG:boto:Headers: {} 2023-07-29T00:24:56.478 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:56.478 DEBUG:boto:Port: 8000 2023-07-29T00:24:56.479 DEBUG:boto:Params: {} 2023-07-29T00:24:56.480 DEBUG:boto:Token: None 2023-07-29T00:24:56.481 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:56 GMT /hiiqos-58/ 2023-07-29T00:24:56.481 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Daq7rVxx4zjpp/4mkT3GlUKRs8Y= 2023-07-29T00:24:56.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Daq7rVxx4zjpp/4mkT3GlUKRs8Y=', 'Content-Length': '0'} 2023-07-29T00:24:56.485 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a4026b8c25be6fe3-0064c45c58-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:56.485 DEBUG:boto:path=/hiiqos-58/obj00000000000000000010 2023-07-29T00:24:56.486 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000010 2023-07-29T00:24:56.486 DEBUG:boto:Method: PUT 2023-07-29T00:24:56.487 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000010 2023-07-29T00:24:56.487 DEBUG:boto:Data: 2023-07-29T00:24:56.487 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:56.488 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:56.489 DEBUG:boto:Port: 8000 2023-07-29T00:24:56.489 DEBUG:boto:Params: {} 2023-07-29T00:24:56.489 DEBUG:boto:Token: None 2023-07-29T00:24:56.490 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:56 GMT /hiiqos-58/obj00000000000000000010 2023-07-29T00:24:56.491 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xvNW8ESfbRxNMDC0nubfhyVHPlQ= 2023-07-29T00:24:56.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xvNW8ESfbRxNMDC0nubfhyVHPlQ='} 2023-07-29T00:24:56.541 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002512f499eaa5ea31-0064c45c58-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:56.542 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:56.542 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:56.543 DEBUG:boto:Method: HEAD 2023-07-29T00:24:56.543 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:56.543 DEBUG:boto:Data: 2023-07-29T00:24:56.544 DEBUG:boto:Headers: {} 2023-07-29T00:24:56.544 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:56.545 DEBUG:boto:Port: 8000 2023-07-29T00:24:56.545 DEBUG:boto:Params: {} 2023-07-29T00:24:56.545 DEBUG:boto:Token: None 2023-07-29T00:24:56.546 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:56 GMT /hiiqos-58/ 2023-07-29T00:24:56.546 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Daq7rVxx4zjpp/4mkT3GlUKRs8Y= 2023-07-29T00:24:56.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Daq7rVxx4zjpp/4mkT3GlUKRs8Y=', 'Content-Length': '0'} 2023-07-29T00:24:56.550 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000efbca17ad32c24f-0064c45c58-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:56.551 DEBUG:boto:path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:24:56.551 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:24:56.552 DEBUG:boto:Method: PUT 2023-07-29T00:24:56.552 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000003 2023-07-29T00:24:56.552 DEBUG:boto:Data: 2023-07-29T00:24:56.553 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:56.553 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:56.554 DEBUG:boto:Port: 8000 2023-07-29T00:24:56.554 DEBUG:boto:Params: {} 2023-07-29T00:24:56.554 DEBUG:boto:Token: None 2023-07-29T00:24:56.555 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:56 GMT /hiiqos-58/obj00000000000000000003 2023-07-29T00:24:56.555 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pP2RGp+jiyxaj3g8O1QIC6UxZ/E= 2023-07-29T00:24:56.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pP2RGp+jiyxaj3g8O1QIC6UxZ/E='} 2023-07-29T00:24:56.562 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009cf99e571c30f85a-0064c45c58-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:56.563 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:56.563 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:56.563 DEBUG:boto:Method: HEAD 2023-07-29T00:24:56.564 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:56.564 DEBUG:boto:Data: 2023-07-29T00:24:56.564 DEBUG:boto:Headers: {} 2023-07-29T00:24:56.565 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:56.565 DEBUG:boto:Port: 8000 2023-07-29T00:24:56.566 DEBUG:boto:Params: {} 2023-07-29T00:24:56.566 DEBUG:boto:Token: None 2023-07-29T00:24:56.566 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:56 GMT /hiiqos-58/ 2023-07-29T00:24:56.567 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Daq7rVxx4zjpp/4mkT3GlUKRs8Y= 2023-07-29T00:24:56.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Daq7rVxx4zjpp/4mkT3GlUKRs8Y=', 'Content-Length': '0'} 2023-07-29T00:24:56.570 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d06a2dfc943ce6c1-0064c45c58-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:56.571 DEBUG:boto:path=/hiiqos-58/obj00000000000000000015 2023-07-29T00:24:56.572 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000015 2023-07-29T00:24:56.573 DEBUG:boto:Method: PUT 2023-07-29T00:24:56.573 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000015 2023-07-29T00:24:56.574 DEBUG:boto:Data: 2023-07-29T00:24:56.574 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:56.574 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:56.575 DEBUG:boto:Port: 8000 2023-07-29T00:24:56.575 DEBUG:boto:Params: {} 2023-07-29T00:24:56.576 DEBUG:boto:Token: None 2023-07-29T00:24:56.577 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:56 GMT /hiiqos-58/obj00000000000000000015 2023-07-29T00:24:56.578 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1CcCBVgHoGEdXHV407SyZqvWbGQ= 2023-07-29T00:24:56.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1CcCBVgHoGEdXHV407SyZqvWbGQ='} 2023-07-29T00:24:56.620 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002b8a7e08f487b98f-0064c45c58-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:56.621 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:56.622 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:56.623 DEBUG:boto:Method: HEAD 2023-07-29T00:24:56.623 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:56.623 DEBUG:boto:Data: 2023-07-29T00:24:56.624 DEBUG:boto:Headers: {} 2023-07-29T00:24:56.624 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:56.624 DEBUG:boto:Port: 8000 2023-07-29T00:24:56.625 DEBUG:boto:Params: {} 2023-07-29T00:24:56.625 DEBUG:boto:Token: None 2023-07-29T00:24:56.626 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:56 GMT /hiiqos-58/ 2023-07-29T00:24:56.626 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Daq7rVxx4zjpp/4mkT3GlUKRs8Y= 2023-07-29T00:24:56.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Daq7rVxx4zjpp/4mkT3GlUKRs8Y=', 'Content-Length': '0'} 2023-07-29T00:24:56.630 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f05d51e13be396b9-0064c45c58-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:56.630 DEBUG:boto:path=/hiiqos-58/obj00000000000000000021 2023-07-29T00:24:56.631 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000021 2023-07-29T00:24:56.631 DEBUG:boto:Method: PUT 2023-07-29T00:24:56.632 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000021 2023-07-29T00:24:56.632 DEBUG:boto:Data: 2023-07-29T00:24:56.633 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:56.633 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:56.633 DEBUG:boto:Port: 8000 2023-07-29T00:24:56.634 DEBUG:boto:Params: {} 2023-07-29T00:24:56.634 DEBUG:boto:Token: None 2023-07-29T00:24:56.635 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:56 GMT /hiiqos-58/obj00000000000000000021 2023-07-29T00:24:56.635 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XNXpAeT8zZ7Z2yj9w8qvUATZDTw= 2023-07-29T00:24:56.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XNXpAeT8zZ7Z2yj9w8qvUATZDTw='} 2023-07-29T00:24:56.642 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000904d9c45ed695dcb-0064c45c58-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:56.643 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:56.643 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:56.643 DEBUG:boto:Method: HEAD 2023-07-29T00:24:56.644 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:56.644 DEBUG:boto:Data: 2023-07-29T00:24:56.644 DEBUG:boto:Headers: {} 2023-07-29T00:24:56.645 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:56.645 DEBUG:boto:Port: 8000 2023-07-29T00:24:56.645 DEBUG:boto:Params: {} 2023-07-29T00:24:56.646 DEBUG:boto:Token: None 2023-07-29T00:24:56.646 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:56 GMT /hiiqos-58/ 2023-07-29T00:24:56.647 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Daq7rVxx4zjpp/4mkT3GlUKRs8Y= 2023-07-29T00:24:56.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Daq7rVxx4zjpp/4mkT3GlUKRs8Y=', 'Content-Length': '0'} 2023-07-29T00:24:56.650 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000719d3d0bcdb28640-0064c45c58-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:56.651 DEBUG:boto:path=/hiiqos-58/obj00000000000000000017 2023-07-29T00:24:56.651 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000017 2023-07-29T00:24:56.651 DEBUG:boto:Method: PUT 2023-07-29T00:24:56.652 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000017 2023-07-29T00:24:56.652 DEBUG:boto:Data: 2023-07-29T00:24:56.652 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:56.653 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:56.653 DEBUG:boto:Port: 8000 2023-07-29T00:24:56.653 DEBUG:boto:Params: {} 2023-07-29T00:24:56.654 DEBUG:boto:Token: None 2023-07-29T00:24:56.654 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:56 GMT /hiiqos-58/obj00000000000000000017 2023-07-29T00:24:56.655 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:F2dCbX9iuT0pJJPYGy1+1lFv4Pc= 2023-07-29T00:24:56.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:F2dCbX9iuT0pJJPYGy1+1lFv4Pc='} 2023-07-29T00:24:56.701 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000008eb64340851ddd4-0064c45c58-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:56.701 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:56.702 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:56.702 DEBUG:boto:Method: HEAD 2023-07-29T00:24:56.703 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:56.703 DEBUG:boto:Data: 2023-07-29T00:24:56.704 DEBUG:boto:Headers: {} 2023-07-29T00:24:56.704 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:56.705 DEBUG:boto:Port: 8000 2023-07-29T00:24:56.705 DEBUG:boto:Params: {} 2023-07-29T00:24:56.705 DEBUG:boto:Token: None 2023-07-29T00:24:56.706 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:56 GMT /hiiqos-58/ 2023-07-29T00:24:56.706 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Daq7rVxx4zjpp/4mkT3GlUKRs8Y= 2023-07-29T00:24:56.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Daq7rVxx4zjpp/4mkT3GlUKRs8Y=', 'Content-Length': '0'} 2023-07-29T00:24:56.710 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '111'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000079c5563e2e2eaeb7-0064c45c58-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:56.710 DEBUG:boto:path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:24:56.711 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:24:56.711 DEBUG:boto:Method: PUT 2023-07-29T00:24:56.712 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000013 2023-07-29T00:24:56.712 DEBUG:boto:Data: 2023-07-29T00:24:56.713 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:56.713 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:56.713 DEBUG:boto:Port: 8000 2023-07-29T00:24:56.714 DEBUG:boto:Params: {} 2023-07-29T00:24:56.714 DEBUG:boto:Token: None 2023-07-29T00:24:56.714 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:56 GMT /hiiqos-58/obj00000000000000000013 2023-07-29T00:24:56.715 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:07390w/FEEFHwsMOCdSnnb0eCV8= 2023-07-29T00:24:56.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:07390w/FEEFHwsMOCdSnnb0eCV8='} 2023-07-29T00:24:56.721 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000017e031dca0529960-0064c45c58-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:56.722 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:56.722 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:56.722 DEBUG:boto:Method: HEAD 2023-07-29T00:24:56.723 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:56.723 DEBUG:boto:Data: 2023-07-29T00:24:56.723 DEBUG:boto:Headers: {} 2023-07-29T00:24:56.724 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:56.724 DEBUG:boto:Port: 8000 2023-07-29T00:24:56.725 DEBUG:boto:Params: {} 2023-07-29T00:24:56.725 DEBUG:boto:Token: None 2023-07-29T00:24:56.725 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:56 GMT /hiiqos-58/ 2023-07-29T00:24:56.726 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Daq7rVxx4zjpp/4mkT3GlUKRs8Y= 2023-07-29T00:24:56.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Daq7rVxx4zjpp/4mkT3GlUKRs8Y=', 'Content-Length': '0'} 2023-07-29T00:24:56.729 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '111'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005efc1f35f8db0fa2-0064c45c58-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:56.730 DEBUG:boto:path=/hiiqos-58/obj00000000000000000049 2023-07-29T00:24:56.730 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000049 2023-07-29T00:24:56.731 DEBUG:boto:Method: PUT 2023-07-29T00:24:56.731 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000049 2023-07-29T00:24:56.731 DEBUG:boto:Data: 2023-07-29T00:24:56.732 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:56.732 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:56.732 DEBUG:boto:Port: 8000 2023-07-29T00:24:56.733 DEBUG:boto:Params: {} 2023-07-29T00:24:56.733 DEBUG:boto:Token: None 2023-07-29T00:24:56.733 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:56 GMT /hiiqos-58/obj00000000000000000049 2023-07-29T00:24:56.734 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vFZsVjdV0TTM3JAYuOarOn7aWWo= 2023-07-29T00:24:56.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vFZsVjdV0TTM3JAYuOarOn7aWWo='} 2023-07-29T00:24:56.776 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000077fbeb50183db870-0064c45c58-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:56.777 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:56.778 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:56.778 DEBUG:boto:Method: HEAD 2023-07-29T00:24:56.778 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:56.779 DEBUG:boto:Data: 2023-07-29T00:24:56.779 DEBUG:boto:Headers: {} 2023-07-29T00:24:56.779 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:56.779 DEBUG:boto:Port: 8000 2023-07-29T00:24:56.780 DEBUG:boto:Params: {} 2023-07-29T00:24:56.780 DEBUG:boto:Token: None 2023-07-29T00:24:56.781 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:56 GMT /hiiqos-58/ 2023-07-29T00:24:56.781 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Daq7rVxx4zjpp/4mkT3GlUKRs8Y= 2023-07-29T00:24:56.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Daq7rVxx4zjpp/4mkT3GlUKRs8Y=', 'Content-Length': '0'} 2023-07-29T00:24:56.787 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000487c502f90f8cd2d-0064c45c58-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:56.788 DEBUG:boto:path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:24:56.788 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:24:56.788 DEBUG:boto:Method: PUT 2023-07-29T00:24:56.789 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000004 2023-07-29T00:24:56.789 DEBUG:boto:Data: 2023-07-29T00:24:56.789 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:56.790 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:56.790 DEBUG:boto:Port: 8000 2023-07-29T00:24:56.790 DEBUG:boto:Params: {} 2023-07-29T00:24:56.791 DEBUG:boto:Token: None 2023-07-29T00:24:56.792 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:56 GMT /hiiqos-58/obj00000000000000000004 2023-07-29T00:24:56.793 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ui8/9V1a39GVpLujx8Qtweo3ivI= 2023-07-29T00:24:56.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ui8/9V1a39GVpLujx8Qtweo3ivI='} 2023-07-29T00:24:56.799 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ee0e3c59990304b0-0064c45c58-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:56.800 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:56.800 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:56.800 DEBUG:boto:Method: HEAD 2023-07-29T00:24:56.801 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:56.801 DEBUG:boto:Data: 2023-07-29T00:24:56.802 DEBUG:boto:Headers: {} 2023-07-29T00:24:56.802 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:56.802 DEBUG:boto:Port: 8000 2023-07-29T00:24:56.803 DEBUG:boto:Params: {} 2023-07-29T00:24:56.803 DEBUG:boto:Token: None 2023-07-29T00:24:56.804 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:56 GMT /hiiqos-58/ 2023-07-29T00:24:56.804 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Daq7rVxx4zjpp/4mkT3GlUKRs8Y= 2023-07-29T00:24:56.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Daq7rVxx4zjpp/4mkT3GlUKRs8Y=', 'Content-Length': '0'} 2023-07-29T00:24:56.807 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000095e0e91d6a4f5d0-0064c45c58-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:56.808 DEBUG:boto:path=/hiiqos-58/obj00000000000000000014 2023-07-29T00:24:56.808 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000014 2023-07-29T00:24:56.809 DEBUG:boto:Method: PUT 2023-07-29T00:24:56.809 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000014 2023-07-29T00:24:56.809 DEBUG:boto:Data: 2023-07-29T00:24:56.810 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:56.810 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:56.810 DEBUG:boto:Port: 8000 2023-07-29T00:24:56.811 DEBUG:boto:Params: {} 2023-07-29T00:24:56.811 DEBUG:boto:Token: None 2023-07-29T00:24:56.811 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:56 GMT /hiiqos-58/obj00000000000000000014 2023-07-29T00:24:56.812 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:odkw7GQtXyUFqex+Ti1ZmLrY1gE= 2023-07-29T00:24:56.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:odkw7GQtXyUFqex+Ti1ZmLrY1gE='} 2023-07-29T00:24:56.857 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000083aafe470f8e281f-0064c45c58-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:56.857 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:56.858 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:56.858 DEBUG:boto:Method: HEAD 2023-07-29T00:24:56.858 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:56.859 DEBUG:boto:Data: 2023-07-29T00:24:56.859 DEBUG:boto:Headers: {} 2023-07-29T00:24:56.859 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:56.860 DEBUG:boto:Port: 8000 2023-07-29T00:24:56.860 DEBUG:boto:Params: {} 2023-07-29T00:24:56.861 DEBUG:boto:Token: None 2023-07-29T00:24:56.861 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:56 GMT /hiiqos-58/ 2023-07-29T00:24:56.861 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Daq7rVxx4zjpp/4mkT3GlUKRs8Y= 2023-07-29T00:24:56.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Daq7rVxx4zjpp/4mkT3GlUKRs8Y=', 'Content-Length': '0'} 2023-07-29T00:24:56.865 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000db99a70f1aeaaab9-0064c45c58-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:56.865 DEBUG:boto:path=/hiiqos-58/obj00000000000000000019 2023-07-29T00:24:56.866 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000019 2023-07-29T00:24:56.866 DEBUG:boto:Method: PUT 2023-07-29T00:24:56.866 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000019 2023-07-29T00:24:56.867 DEBUG:boto:Data: 2023-07-29T00:24:56.867 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:56.867 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:56.868 DEBUG:boto:Port: 8000 2023-07-29T00:24:56.868 DEBUG:boto:Params: {} 2023-07-29T00:24:56.868 DEBUG:boto:Token: None 2023-07-29T00:24:56.869 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:56 GMT /hiiqos-58/obj00000000000000000019 2023-07-29T00:24:56.869 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:V1Ru8XC22YOXHqb0Aa/OmbQxGDA= 2023-07-29T00:24:56.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:V1Ru8XC22YOXHqb0Aa/OmbQxGDA='} 2023-07-29T00:24:56.875 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009d6cf78d442f646c-0064c45c58-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:56.876 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:56.876 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:56.877 DEBUG:boto:Method: HEAD 2023-07-29T00:24:56.877 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:56.878 DEBUG:boto:Data: 2023-07-29T00:24:56.878 DEBUG:boto:Headers: {} 2023-07-29T00:24:56.879 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:56.879 DEBUG:boto:Port: 8000 2023-07-29T00:24:56.879 DEBUG:boto:Params: {} 2023-07-29T00:24:56.880 DEBUG:boto:Token: None 2023-07-29T00:24:56.880 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:56 GMT /hiiqos-58/ 2023-07-29T00:24:56.881 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Daq7rVxx4zjpp/4mkT3GlUKRs8Y= 2023-07-29T00:24:56.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Daq7rVxx4zjpp/4mkT3GlUKRs8Y=', 'Content-Length': '0'} 2023-07-29T00:24:56.884 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b19b77d7e546efce-0064c45c58-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:56.885 DEBUG:boto:path=/hiiqos-58/obj00000000000000000042 2023-07-29T00:24:56.885 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000042 2023-07-29T00:24:56.886 DEBUG:boto:Method: PUT 2023-07-29T00:24:56.886 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000042 2023-07-29T00:24:56.886 DEBUG:boto:Data: 2023-07-29T00:24:56.887 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:56.888 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:56.889 DEBUG:boto:Port: 8000 2023-07-29T00:24:56.890 DEBUG:boto:Params: {} 2023-07-29T00:24:56.891 DEBUG:boto:Token: None 2023-07-29T00:24:56.892 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:56 GMT /hiiqos-58/obj00000000000000000042 2023-07-29T00:24:56.893 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0UfsfzWNIHh0dNSM/xnZ5s9yQGg= 2023-07-29T00:24:56.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0UfsfzWNIHh0dNSM/xnZ5s9yQGg='} 2023-07-29T00:24:56.940 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000077394983c72f8eb1-0064c45c58-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:56.941 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:56.942 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:56.943 DEBUG:boto:Method: HEAD 2023-07-29T00:24:56.943 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:56.944 DEBUG:boto:Data: 2023-07-29T00:24:56.945 DEBUG:boto:Headers: {} 2023-07-29T00:24:56.946 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:56.947 DEBUG:boto:Port: 8000 2023-07-29T00:24:56.948 DEBUG:boto:Params: {} 2023-07-29T00:24:56.948 DEBUG:boto:Token: None 2023-07-29T00:24:56.949 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:56 GMT /hiiqos-58/ 2023-07-29T00:24:56.949 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Daq7rVxx4zjpp/4mkT3GlUKRs8Y= 2023-07-29T00:24:56.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Daq7rVxx4zjpp/4mkT3GlUKRs8Y=', 'Content-Length': '0'} 2023-07-29T00:24:56.953 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000043fbfff21ad21e48-0064c45c58-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:56.954 DEBUG:boto:path=/hiiqos-58/obj00000000000000000025 2023-07-29T00:24:56.955 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000025 2023-07-29T00:24:56.956 DEBUG:boto:Method: PUT 2023-07-29T00:24:56.956 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000025 2023-07-29T00:24:56.956 DEBUG:boto:Data: 2023-07-29T00:24:56.957 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:56.957 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:56.958 DEBUG:boto:Port: 8000 2023-07-29T00:24:56.958 DEBUG:boto:Params: {} 2023-07-29T00:24:56.958 DEBUG:boto:Token: None 2023-07-29T00:24:56.959 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:56 GMT /hiiqos-58/obj00000000000000000025 2023-07-29T00:24:56.959 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jJvI5/MAWRMeYd82pxc+OOKetrQ= 2023-07-29T00:24:56.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jJvI5/MAWRMeYd82pxc+OOKetrQ='} 2023-07-29T00:24:56.966 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a66b9d7a2cb3ab28-0064c45c58-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:56.967 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:56.967 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:56.968 DEBUG:boto:Method: HEAD 2023-07-29T00:24:56.968 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:56.969 DEBUG:boto:Data: 2023-07-29T00:24:56.969 DEBUG:boto:Headers: {} 2023-07-29T00:24:56.969 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:56.970 DEBUG:boto:Port: 8000 2023-07-29T00:24:56.970 DEBUG:boto:Params: {} 2023-07-29T00:24:56.970 DEBUG:boto:Token: None 2023-07-29T00:24:56.971 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:56 GMT /hiiqos-58/ 2023-07-29T00:24:56.972 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Daq7rVxx4zjpp/4mkT3GlUKRs8Y= 2023-07-29T00:24:56.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Daq7rVxx4zjpp/4mkT3GlUKRs8Y=', 'Content-Length': '0'} 2023-07-29T00:24:56.975 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001c8bb954c03a7080-0064c45c58-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:56.976 DEBUG:boto:path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:24:56.977 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:24:56.978 DEBUG:boto:Method: PUT 2023-07-29T00:24:56.979 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000011 2023-07-29T00:24:56.980 DEBUG:boto:Data: 2023-07-29T00:24:56.980 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:56.981 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:56.981 DEBUG:boto:Port: 8000 2023-07-29T00:24:56.981 DEBUG:boto:Params: {} 2023-07-29T00:24:56.982 DEBUG:boto:Token: None 2023-07-29T00:24:56.983 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:56 GMT /hiiqos-58/obj00000000000000000011 2023-07-29T00:24:56.984 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:liWwo5AavTNjo9PzmiVmOo2BwNI= 2023-07-29T00:24:56.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:liWwo5AavTNjo9PzmiVmOo2BwNI='} 2023-07-29T00:24:57.028 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b23397086935c5ef-0064c45c58-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:57.029 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:57.030 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:57.030 DEBUG:boto:Method: HEAD 2023-07-29T00:24:57.031 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:57.031 DEBUG:boto:Data: 2023-07-29T00:24:57.032 DEBUG:boto:Headers: {} 2023-07-29T00:24:57.032 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:57.032 DEBUG:boto:Port: 8000 2023-07-29T00:24:57.033 DEBUG:boto:Params: {} 2023-07-29T00:24:57.033 DEBUG:boto:Token: None 2023-07-29T00:24:57.034 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:57 GMT /hiiqos-58/ 2023-07-29T00:24:57.034 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GtI6bJ0lM5N4fQ6+f2oz8G4xFdg= 2023-07-29T00:24:57.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GtI6bJ0lM5N4fQ6+f2oz8G4xFdg=', 'Content-Length': '0'} 2023-07-29T00:24:57.038 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003a5f337b16ddd35f-0064c45c58-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:57.038 DEBUG:boto:path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:24:57.039 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:24:57.039 DEBUG:boto:Method: PUT 2023-07-29T00:24:57.040 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000012 2023-07-29T00:24:57.040 DEBUG:boto:Data: 2023-07-29T00:24:57.041 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:57.041 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:57.041 DEBUG:boto:Port: 8000 2023-07-29T00:24:57.042 DEBUG:boto:Params: {} 2023-07-29T00:24:57.042 DEBUG:boto:Token: None 2023-07-29T00:24:57.042 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:57 GMT /hiiqos-58/obj00000000000000000012 2023-07-29T00:24:57.043 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:V4Am0x27liAsfE+BOb03bAMP5cw= 2023-07-29T00:24:57.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:V4Am0x27liAsfE+BOb03bAMP5cw='} 2023-07-29T00:24:57.049 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b4fbee923beab4ae-0064c45c58-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:57.050 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:57.050 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:57.051 DEBUG:boto:Method: HEAD 2023-07-29T00:24:57.051 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:57.051 DEBUG:boto:Data: 2023-07-29T00:24:57.052 DEBUG:boto:Headers: {} 2023-07-29T00:24:57.052 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:57.052 DEBUG:boto:Port: 8000 2023-07-29T00:24:57.053 DEBUG:boto:Params: {} 2023-07-29T00:24:57.053 DEBUG:boto:Token: None 2023-07-29T00:24:57.054 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:57 GMT /hiiqos-58/ 2023-07-29T00:24:57.054 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GtI6bJ0lM5N4fQ6+f2oz8G4xFdg= 2023-07-29T00:24:57.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GtI6bJ0lM5N4fQ6+f2oz8G4xFdg=', 'Content-Length': '0'} 2023-07-29T00:24:57.057 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e95420d1ff20b42c-0064c45c59-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:57.058 DEBUG:boto:path=/hiiqos-58/obj00000000000000000031 2023-07-29T00:24:57.058 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000031 2023-07-29T00:24:57.058 DEBUG:boto:Method: PUT 2023-07-29T00:24:57.059 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000031 2023-07-29T00:24:57.059 DEBUG:boto:Data: 2023-07-29T00:24:57.059 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:57.060 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:57.060 DEBUG:boto:Port: 8000 2023-07-29T00:24:57.061 DEBUG:boto:Params: {} 2023-07-29T00:24:57.061 DEBUG:boto:Token: None 2023-07-29T00:24:57.061 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:57 GMT /hiiqos-58/obj00000000000000000031 2023-07-29T00:24:57.062 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BxzP+hH6aPK4ivm8928hhZUiiF0= 2023-07-29T00:24:57.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BxzP+hH6aPK4ivm8928hhZUiiF0='} 2023-07-29T00:24:57.104 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004e1fc89a58d862e4-0064c45c59-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:57.105 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:57.106 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:57.106 DEBUG:boto:Method: HEAD 2023-07-29T00:24:57.106 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:57.107 DEBUG:boto:Data: 2023-07-29T00:24:57.107 DEBUG:boto:Headers: {} 2023-07-29T00:24:57.107 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:57.108 DEBUG:boto:Port: 8000 2023-07-29T00:24:57.108 DEBUG:boto:Params: {} 2023-07-29T00:24:57.108 DEBUG:boto:Token: None 2023-07-29T00:24:57.109 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:57 GMT /hiiqos-58/ 2023-07-29T00:24:57.109 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GtI6bJ0lM5N4fQ6+f2oz8G4xFdg= 2023-07-29T00:24:57.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GtI6bJ0lM5N4fQ6+f2oz8G4xFdg=', 'Content-Length': '0'} 2023-07-29T00:24:57.112 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000093d8a3319823e400-0064c45c59-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:57.113 DEBUG:boto:path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:24:57.114 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:24:57.114 DEBUG:boto:Method: PUT 2023-07-29T00:24:57.114 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000005 2023-07-29T00:24:57.115 DEBUG:boto:Data: 2023-07-29T00:24:57.115 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:57.115 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:57.116 DEBUG:boto:Port: 8000 2023-07-29T00:24:57.116 DEBUG:boto:Params: {} 2023-07-29T00:24:57.116 DEBUG:boto:Token: None 2023-07-29T00:24:57.117 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:57 GMT /hiiqos-58/obj00000000000000000005 2023-07-29T00:24:57.117 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wyZwkkTP670TYR5xEeXBkWTmNJQ= 2023-07-29T00:24:57.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wyZwkkTP670TYR5xEeXBkWTmNJQ='} 2023-07-29T00:24:57.123 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009edb8b52c3c42d7b-0064c45c59-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:57.124 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:57.124 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:57.125 DEBUG:boto:Method: HEAD 2023-07-29T00:24:57.125 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:57.125 DEBUG:boto:Data: 2023-07-29T00:24:57.126 DEBUG:boto:Headers: {} 2023-07-29T00:24:57.126 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:57.126 DEBUG:boto:Port: 8000 2023-07-29T00:24:57.127 DEBUG:boto:Params: {} 2023-07-29T00:24:57.127 DEBUG:boto:Token: None 2023-07-29T00:24:57.127 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:57 GMT /hiiqos-58/ 2023-07-29T00:24:57.128 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GtI6bJ0lM5N4fQ6+f2oz8G4xFdg= 2023-07-29T00:24:57.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GtI6bJ0lM5N4fQ6+f2oz8G4xFdg=', 'Content-Length': '0'} 2023-07-29T00:24:57.131 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a76ec3eb14489bdc-0064c45c59-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:57.132 DEBUG:boto:path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:24:57.132 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:24:57.132 DEBUG:boto:Method: PUT 2023-07-29T00:24:57.133 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000002 2023-07-29T00:24:57.133 DEBUG:boto:Data: 2023-07-29T00:24:57.134 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:57.134 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:57.134 DEBUG:boto:Port: 8000 2023-07-29T00:24:57.134 DEBUG:boto:Params: {} 2023-07-29T00:24:57.135 DEBUG:boto:Token: None 2023-07-29T00:24:57.135 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:57 GMT /hiiqos-58/obj00000000000000000002 2023-07-29T00:24:57.136 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:D/SDf1lvP4CgIl/hgQ+004wGy/A= 2023-07-29T00:24:57.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:D/SDf1lvP4CgIl/hgQ+004wGy/A='} 2023-07-29T00:24:57.180 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000044d687ae2d728f80-0064c45c59-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:57.181 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:57.181 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:57.182 DEBUG:boto:Method: HEAD 2023-07-29T00:24:57.182 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:57.183 DEBUG:boto:Data: 2023-07-29T00:24:57.183 DEBUG:boto:Headers: {} 2023-07-29T00:24:57.183 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:57.183 DEBUG:boto:Port: 8000 2023-07-29T00:24:57.184 DEBUG:boto:Params: {} 2023-07-29T00:24:57.184 DEBUG:boto:Token: None 2023-07-29T00:24:57.185 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:57 GMT /hiiqos-58/ 2023-07-29T00:24:57.185 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GtI6bJ0lM5N4fQ6+f2oz8G4xFdg= 2023-07-29T00:24:57.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GtI6bJ0lM5N4fQ6+f2oz8G4xFdg=', 'Content-Length': '0'} 2023-07-29T00:24:57.189 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c9ac3462ea992cf8-0064c45c59-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:57.189 DEBUG:boto:path=/hiiqos-58/obj00000000000000000020 2023-07-29T00:24:57.190 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000020 2023-07-29T00:24:57.190 DEBUG:boto:Method: PUT 2023-07-29T00:24:57.190 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000020 2023-07-29T00:24:57.191 DEBUG:boto:Data: 2023-07-29T00:24:57.191 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:57.191 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:57.192 DEBUG:boto:Port: 8000 2023-07-29T00:24:57.192 DEBUG:boto:Params: {} 2023-07-29T00:24:57.193 DEBUG:boto:Token: None 2023-07-29T00:24:57.193 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:57 GMT /hiiqos-58/obj00000000000000000020 2023-07-29T00:24:57.193 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YCN9GjlzeXdQvBwNTVT2rlxiAy4= 2023-07-29T00:24:57.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YCN9GjlzeXdQvBwNTVT2rlxiAy4='} 2023-07-29T00:24:57.200 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000938292e3f4eb84c1-0064c45c59-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:57.200 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:57.201 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:57.201 DEBUG:boto:Method: HEAD 2023-07-29T00:24:57.201 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:57.202 DEBUG:boto:Data: 2023-07-29T00:24:57.202 DEBUG:boto:Headers: {} 2023-07-29T00:24:57.202 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:57.203 DEBUG:boto:Port: 8000 2023-07-29T00:24:57.203 DEBUG:boto:Params: {} 2023-07-29T00:24:57.203 DEBUG:boto:Token: None 2023-07-29T00:24:57.204 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:57 GMT /hiiqos-58/ 2023-07-29T00:24:57.204 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GtI6bJ0lM5N4fQ6+f2oz8G4xFdg= 2023-07-29T00:24:57.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GtI6bJ0lM5N4fQ6+f2oz8G4xFdg=', 'Content-Length': '0'} 2023-07-29T00:24:57.207 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d0d58405a04a60c9-0064c45c59-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:57.208 DEBUG:boto:path=/hiiqos-58/obj00000000000000000027 2023-07-29T00:24:57.209 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000027 2023-07-29T00:24:57.209 DEBUG:boto:Method: PUT 2023-07-29T00:24:57.210 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000027 2023-07-29T00:24:57.211 DEBUG:boto:Data: 2023-07-29T00:24:57.212 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:57.213 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:57.213 DEBUG:boto:Port: 8000 2023-07-29T00:24:57.214 DEBUG:boto:Params: {} 2023-07-29T00:24:57.214 DEBUG:boto:Token: None 2023-07-29T00:24:57.215 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:57 GMT /hiiqos-58/obj00000000000000000027 2023-07-29T00:24:57.216 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QxUF83c6pxU/y+Ot7Ty/Xpe6+pY= 2023-07-29T00:24:57.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QxUF83c6pxU/y+Ot7Ty/Xpe6+pY='} 2023-07-29T00:24:57.268 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000329daa444fb4fc84-0064c45c59-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:57.270 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:57.271 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:57.272 DEBUG:boto:Method: HEAD 2023-07-29T00:24:57.273 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:57.273 DEBUG:boto:Data: 2023-07-29T00:24:57.274 DEBUG:boto:Headers: {} 2023-07-29T00:24:57.276 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:57.276 DEBUG:boto:Port: 8000 2023-07-29T00:24:57.277 DEBUG:boto:Params: {} 2023-07-29T00:24:57.278 DEBUG:boto:Token: None 2023-07-29T00:24:57.279 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:57 GMT /hiiqos-58/ 2023-07-29T00:24:57.279 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GtI6bJ0lM5N4fQ6+f2oz8G4xFdg= 2023-07-29T00:24:57.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GtI6bJ0lM5N4fQ6+f2oz8G4xFdg=', 'Content-Length': '0'} 2023-07-29T00:24:57.287 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '123'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000018b351e466bdf489-0064c45c59-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:57.288 DEBUG:boto:path=/hiiqos-58/obj00000000000000000009 2023-07-29T00:24:57.288 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000009 2023-07-29T00:24:57.289 DEBUG:boto:Method: PUT 2023-07-29T00:24:57.289 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000009 2023-07-29T00:24:57.289 DEBUG:boto:Data: 2023-07-29T00:24:57.290 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:57.290 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:57.290 DEBUG:boto:Port: 8000 2023-07-29T00:24:57.291 DEBUG:boto:Params: {} 2023-07-29T00:24:57.291 DEBUG:boto:Token: None 2023-07-29T00:24:57.292 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:57 GMT /hiiqos-58/obj00000000000000000009 2023-07-29T00:24:57.293 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:B/mWHteFDHwhMm2zA1XHQXqgQUc= 2023-07-29T00:24:57.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:B/mWHteFDHwhMm2zA1XHQXqgQUc='} 2023-07-29T00:24:57.299 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000067ad7bdf3e8a9e0c-0064c45c59-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:57.300 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:57.300 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:57.301 DEBUG:boto:Method: HEAD 2023-07-29T00:24:57.301 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:57.301 DEBUG:boto:Data: 2023-07-29T00:24:57.302 DEBUG:boto:Headers: {} 2023-07-29T00:24:57.302 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:57.302 DEBUG:boto:Port: 8000 2023-07-29T00:24:57.303 DEBUG:boto:Params: {} 2023-07-29T00:24:57.303 DEBUG:boto:Token: None 2023-07-29T00:24:57.304 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:57 GMT /hiiqos-58/ 2023-07-29T00:24:57.304 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GtI6bJ0lM5N4fQ6+f2oz8G4xFdg= 2023-07-29T00:24:57.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GtI6bJ0lM5N4fQ6+f2oz8G4xFdg=', 'Content-Length': '0'} 2023-07-29T00:24:57.308 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '123'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001048baf96cb40af9-0064c45c59-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:57.308 DEBUG:boto:path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:24:57.309 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:24:57.309 DEBUG:boto:Method: PUT 2023-07-29T00:24:57.310 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000000 2023-07-29T00:24:57.310 DEBUG:boto:Data: 2023-07-29T00:24:57.310 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:57.311 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:57.311 DEBUG:boto:Port: 8000 2023-07-29T00:24:57.312 DEBUG:boto:Params: {} 2023-07-29T00:24:57.312 DEBUG:boto:Token: None 2023-07-29T00:24:57.312 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:57 GMT /hiiqos-58/obj00000000000000000000 2023-07-29T00:24:57.313 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:epdFqyQ8wJze8SDqiuOG8YmhGxM= 2023-07-29T00:24:57.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:epdFqyQ8wJze8SDqiuOG8YmhGxM='} 2023-07-29T00:24:57.356 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a8d3941691f30c5d-0064c45c59-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:57.357 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:57.358 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:57.358 DEBUG:boto:Method: HEAD 2023-07-29T00:24:57.359 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:57.359 DEBUG:boto:Data: 2023-07-29T00:24:57.360 DEBUG:boto:Headers: {} 2023-07-29T00:24:57.360 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:57.361 DEBUG:boto:Port: 8000 2023-07-29T00:24:57.361 DEBUG:boto:Params: {} 2023-07-29T00:24:57.362 DEBUG:boto:Token: None 2023-07-29T00:24:57.362 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:57 GMT /hiiqos-58/ 2023-07-29T00:24:57.363 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GtI6bJ0lM5N4fQ6+f2oz8G4xFdg= 2023-07-29T00:24:57.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GtI6bJ0lM5N4fQ6+f2oz8G4xFdg=', 'Content-Length': '0'} 2023-07-29T00:24:57.367 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '123'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000086f139597215b654-0064c45c59-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:57.367 DEBUG:boto:path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:24:57.368 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:24:57.368 DEBUG:boto:Method: PUT 2023-07-29T00:24:57.368 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000006 2023-07-29T00:24:57.369 DEBUG:boto:Data: 2023-07-29T00:24:57.369 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:57.370 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:57.370 DEBUG:boto:Port: 8000 2023-07-29T00:24:57.371 DEBUG:boto:Params: {} 2023-07-29T00:24:57.371 DEBUG:boto:Token: None 2023-07-29T00:24:57.372 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:57 GMT /hiiqos-58/obj00000000000000000006 2023-07-29T00:24:57.372 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8A7B6wfPm81wX7ufMW/NeE6NGkM= 2023-07-29T00:24:57.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8A7B6wfPm81wX7ufMW/NeE6NGkM='} 2023-07-29T00:24:57.379 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e08455941e0f3136-0064c45c59-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:57.380 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-58', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:24:57.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-58', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:24:57.381 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-58 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:24:57.433 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:24:57.434 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:24:57.479 INFO:teuthology.orchestra.run.smithi101.stdout:tenant: 2023-07-29T00:24:57.480 INFO:teuthology.orchestra.run.smithi101.stdout:bucket name: hiiqos-58 2023-07-29T00:24:57.490 INFO:teuthology.orchestra.run.smithi101.stdout:total entries: 41 2023-07-29T00:24:57.544 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:24:57.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:24:57.546 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 31 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:24:57.593 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:24:57.594 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:24:57.633 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:24:57.633 INFO:teuthology.orchestra.run.smithi101.stdout: "objs": [ 2023-07-29T00:24:57.634 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000019", 2023-07-29T00:24:57.634 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000006", 2023-07-29T00:24:57.634 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000467", 2023-07-29T00:24:57.634 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000005", 2023-07-29T00:24:57.634 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000056", 2023-07-29T00:24:57.635 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000004", 2023-07-29T00:24:57.635 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000023", 2023-07-29T00:24:57.635 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000317", 2023-07-29T00:24:57.635 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000022", 2023-07-29T00:24:57.635 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000000", 2023-07-29T00:24:57.635 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000021", 2023-07-29T00:24:57.636 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000009", 2023-07-29T00:24:57.636 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000271", 2023-07-29T00:24:57.636 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000008", 2023-07-29T00:24:57.636 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000018", 2023-07-29T00:24:57.636 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000007", 2023-07-29T00:24:57.636 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000017", 2023-07-29T00:24:57.637 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000162", 2023-07-29T00:24:57.637 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000016", 2023-07-29T00:24:57.637 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000003", 2023-07-29T00:24:57.637 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000423", 2023-07-29T00:24:57.637 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000002", 2023-07-29T00:24:57.637 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000012", 2023-07-29T00:24:57.637 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000001", 2023-07-29T00:24:57.638 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000011", 2023-07-29T00:24:57.638 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000283", 2023-07-29T00:24:57.638 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000010", 2023-07-29T00:24:57.638 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000061", 2023-07-29T00:24:57.638 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000050", 2023-07-29T00:24:57.638 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000029", 2023-07-29T00:24:57.638 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000133" 2023-07-29T00:24:57.639 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:24:57.639 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:24:57.640 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:57.640 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:57.641 DEBUG:boto:Method: HEAD 2023-07-29T00:24:57.641 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:57.641 DEBUG:boto:Data: 2023-07-29T00:24:57.642 DEBUG:boto:Headers: {} 2023-07-29T00:24:57.642 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:57.642 DEBUG:boto:Port: 8000 2023-07-29T00:24:57.643 DEBUG:boto:Params: {} 2023-07-29T00:24:57.643 DEBUG:boto:Token: None 2023-07-29T00:24:57.644 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:57 GMT /hiiqos-58/ 2023-07-29T00:24:57.644 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GtI6bJ0lM5N4fQ6+f2oz8G4xFdg= 2023-07-29T00:24:57.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GtI6bJ0lM5N4fQ6+f2oz8G4xFdg=', 'Content-Length': '0'} 2023-07-29T00:24:57.648 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '123'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000385db0ee3c9fed8e-0064c45c59-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:57.649 DEBUG:boto:path=/hiiqos-58/obj00000000000000000056 2023-07-29T00:24:57.649 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000056 2023-07-29T00:24:57.649 DEBUG:boto:Method: PUT 2023-07-29T00:24:57.650 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000056 2023-07-29T00:24:57.650 DEBUG:boto:Data: 2023-07-29T00:24:57.650 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:57.651 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:57.651 DEBUG:boto:Port: 8000 2023-07-29T00:24:57.651 DEBUG:boto:Params: {} 2023-07-29T00:24:57.652 DEBUG:boto:Token: None 2023-07-29T00:24:57.652 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:57 GMT /hiiqos-58/obj00000000000000000056 2023-07-29T00:24:57.653 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HuEc+jprQir7oN76NTf7jUzxkLI= 2023-07-29T00:24:57.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HuEc+jprQir7oN76NTf7jUzxkLI='} 2023-07-29T00:24:57.658 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d4ba2588bf075f4b-0064c45c59-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:57.659 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:57.659 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:57.659 DEBUG:boto:Method: HEAD 2023-07-29T00:24:57.660 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:57.660 DEBUG:boto:Data: 2023-07-29T00:24:57.660 DEBUG:boto:Headers: {} 2023-07-29T00:24:57.661 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:57.661 DEBUG:boto:Port: 8000 2023-07-29T00:24:57.661 DEBUG:boto:Params: {} 2023-07-29T00:24:57.662 DEBUG:boto:Token: None 2023-07-29T00:24:57.662 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:57 GMT /hiiqos-58/ 2023-07-29T00:24:57.663 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GtI6bJ0lM5N4fQ6+f2oz8G4xFdg= 2023-07-29T00:24:57.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GtI6bJ0lM5N4fQ6+f2oz8G4xFdg=', 'Content-Length': '0'} 2023-07-29T00:24:57.666 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002bcec8377a3af6c6-0064c45c59-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:57.667 DEBUG:boto:path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:24:57.667 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:24:57.667 DEBUG:boto:Method: PUT 2023-07-29T00:24:57.668 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000003 2023-07-29T00:24:57.668 DEBUG:boto:Data: 2023-07-29T00:24:57.668 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:57.669 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:57.669 DEBUG:boto:Port: 8000 2023-07-29T00:24:57.669 DEBUG:boto:Params: {} 2023-07-29T00:24:57.670 DEBUG:boto:Token: None 2023-07-29T00:24:57.670 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:57 GMT /hiiqos-58/obj00000000000000000003 2023-07-29T00:24:57.671 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Q+ALgXGxvzYr5WMcvRToFckqfD8= 2023-07-29T00:24:57.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q+ALgXGxvzYr5WMcvRToFckqfD8='} 2023-07-29T00:24:57.716 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007e916fd602469fed-0064c45c59-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:57.717 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:57.717 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:57.718 DEBUG:boto:Method: HEAD 2023-07-29T00:24:57.718 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:57.719 DEBUG:boto:Data: 2023-07-29T00:24:57.719 DEBUG:boto:Headers: {} 2023-07-29T00:24:57.719 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:57.720 DEBUG:boto:Port: 8000 2023-07-29T00:24:57.720 DEBUG:boto:Params: {} 2023-07-29T00:24:57.720 DEBUG:boto:Token: None 2023-07-29T00:24:57.721 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:57 GMT /hiiqos-58/ 2023-07-29T00:24:57.721 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GtI6bJ0lM5N4fQ6+f2oz8G4xFdg= 2023-07-29T00:24:57.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GtI6bJ0lM5N4fQ6+f2oz8G4xFdg=', 'Content-Length': '0'} 2023-07-29T00:24:57.725 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004713452ee7fa5c16-0064c45c59-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:57.726 DEBUG:boto:path=/hiiqos-58/obj00000000000000000008 2023-07-29T00:24:57.728 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000008 2023-07-29T00:24:57.728 DEBUG:boto:Method: PUT 2023-07-29T00:24:57.729 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000008 2023-07-29T00:24:57.729 DEBUG:boto:Data: 2023-07-29T00:24:57.729 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:57.730 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:57.730 DEBUG:boto:Port: 8000 2023-07-29T00:24:57.730 DEBUG:boto:Params: {} 2023-07-29T00:24:57.731 DEBUG:boto:Token: None 2023-07-29T00:24:57.731 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:57 GMT /hiiqos-58/obj00000000000000000008 2023-07-29T00:24:57.732 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oOkkUr9jIcd/YuqiBzvHCcn1B2A= 2023-07-29T00:24:57.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oOkkUr9jIcd/YuqiBzvHCcn1B2A='} 2023-07-29T00:24:57.739 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e6034de4add42f9e-0064c45c59-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:57.740 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:57.740 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:57.741 DEBUG:boto:Method: HEAD 2023-07-29T00:24:57.741 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:57.741 DEBUG:boto:Data: 2023-07-29T00:24:57.742 DEBUG:boto:Headers: {} 2023-07-29T00:24:57.742 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:57.742 DEBUG:boto:Port: 8000 2023-07-29T00:24:57.743 DEBUG:boto:Params: {} 2023-07-29T00:24:57.743 DEBUG:boto:Token: None 2023-07-29T00:24:57.744 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:57 GMT /hiiqos-58/ 2023-07-29T00:24:57.744 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GtI6bJ0lM5N4fQ6+f2oz8G4xFdg= 2023-07-29T00:24:57.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GtI6bJ0lM5N4fQ6+f2oz8G4xFdg=', 'Content-Length': '0'} 2023-07-29T00:24:57.747 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000674f5094732afaa3-0064c45c59-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:57.748 DEBUG:boto:path=/hiiqos-58/obj00000000000000000133 2023-07-29T00:24:57.749 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000133 2023-07-29T00:24:57.750 DEBUG:boto:Method: PUT 2023-07-29T00:24:57.751 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000133 2023-07-29T00:24:57.752 DEBUG:boto:Data: 2023-07-29T00:24:57.753 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:57.753 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:57.754 DEBUG:boto:Port: 8000 2023-07-29T00:24:57.755 DEBUG:boto:Params: {} 2023-07-29T00:24:57.756 DEBUG:boto:Token: None 2023-07-29T00:24:57.757 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:57 GMT /hiiqos-58/obj00000000000000000133 2023-07-29T00:24:57.758 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cxM4c43+eq81+tl4x06ODJCjjo4= 2023-07-29T00:24:57.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cxM4c43+eq81+tl4x06ODJCjjo4='} 2023-07-29T00:24:57.804 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003889a9fa49c70f33-0064c45c59-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:57.806 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:57.807 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:57.807 DEBUG:boto:Method: HEAD 2023-07-29T00:24:57.808 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:57.809 DEBUG:boto:Data: 2023-07-29T00:24:57.810 DEBUG:boto:Headers: {} 2023-07-29T00:24:57.810 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:57.811 DEBUG:boto:Port: 8000 2023-07-29T00:24:57.811 DEBUG:boto:Params: {} 2023-07-29T00:24:57.812 DEBUG:boto:Token: None 2023-07-29T00:24:57.812 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:57 GMT /hiiqos-58/ 2023-07-29T00:24:57.812 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GtI6bJ0lM5N4fQ6+f2oz8G4xFdg= 2023-07-29T00:24:57.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GtI6bJ0lM5N4fQ6+f2oz8G4xFdg=', 'Content-Length': '0'} 2023-07-29T00:24:57.816 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003f1b14ac99694509-0064c45c59-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:57.816 DEBUG:boto:path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:24:57.817 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:24:57.818 DEBUG:boto:Method: PUT 2023-07-29T00:24:57.819 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000000 2023-07-29T00:24:57.819 DEBUG:boto:Data: 2023-07-29T00:24:57.820 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:57.820 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:57.820 DEBUG:boto:Port: 8000 2023-07-29T00:24:57.821 DEBUG:boto:Params: {} 2023-07-29T00:24:57.822 DEBUG:boto:Token: None 2023-07-29T00:24:57.822 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:57 GMT /hiiqos-58/obj00000000000000000000 2023-07-29T00:24:57.822 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:epdFqyQ8wJze8SDqiuOG8YmhGxM= 2023-07-29T00:24:57.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:epdFqyQ8wJze8SDqiuOG8YmhGxM='} 2023-07-29T00:24:57.829 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007296d4face97bf56-0064c45c59-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:57.829 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:57.830 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:57.830 DEBUG:boto:Method: HEAD 2023-07-29T00:24:57.830 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:57.831 DEBUG:boto:Data: 2023-07-29T00:24:57.831 DEBUG:boto:Headers: {} 2023-07-29T00:24:57.831 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:57.832 DEBUG:boto:Port: 8000 2023-07-29T00:24:57.832 DEBUG:boto:Params: {} 2023-07-29T00:24:57.832 DEBUG:boto:Token: None 2023-07-29T00:24:57.833 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:57 GMT /hiiqos-58/ 2023-07-29T00:24:57.833 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GtI6bJ0lM5N4fQ6+f2oz8G4xFdg= 2023-07-29T00:24:57.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GtI6bJ0lM5N4fQ6+f2oz8G4xFdg=', 'Content-Length': '0'} 2023-07-29T00:24:57.840 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dbc0baaf25413721-0064c45c59-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:57.841 DEBUG:boto:path=/hiiqos-58/obj00000000000000000021 2023-07-29T00:24:57.841 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000021 2023-07-29T00:24:57.842 DEBUG:boto:Method: PUT 2023-07-29T00:24:57.842 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000021 2023-07-29T00:24:57.843 DEBUG:boto:Data: 2023-07-29T00:24:57.843 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:57.843 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:57.844 DEBUG:boto:Port: 8000 2023-07-29T00:24:57.844 DEBUG:boto:Params: {} 2023-07-29T00:24:57.845 DEBUG:boto:Token: None 2023-07-29T00:24:57.845 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:57 GMT /hiiqos-58/obj00000000000000000021 2023-07-29T00:24:57.846 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1uXAaH3GCVC6ll/dZnqCH+sX4yk= 2023-07-29T00:24:57.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1uXAaH3GCVC6ll/dZnqCH+sX4yk='} 2023-07-29T00:24:57.888 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003bdaf554d8d62a43-0064c45c59-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:57.889 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:57.890 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:57.890 DEBUG:boto:Method: HEAD 2023-07-29T00:24:57.890 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:57.891 DEBUG:boto:Data: 2023-07-29T00:24:57.891 DEBUG:boto:Headers: {} 2023-07-29T00:24:57.891 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:57.892 DEBUG:boto:Port: 8000 2023-07-29T00:24:57.892 DEBUG:boto:Params: {} 2023-07-29T00:24:57.893 DEBUG:boto:Token: None 2023-07-29T00:24:57.893 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:57 GMT /hiiqos-58/ 2023-07-29T00:24:57.893 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GtI6bJ0lM5N4fQ6+f2oz8G4xFdg= 2023-07-29T00:24:57.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GtI6bJ0lM5N4fQ6+f2oz8G4xFdg=', 'Content-Length': '0'} 2023-07-29T00:24:57.897 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c421e29508c117bd-0064c45c59-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:57.898 DEBUG:boto:path=/hiiqos-58/obj00000000000000000017 2023-07-29T00:24:57.898 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000017 2023-07-29T00:24:57.899 DEBUG:boto:Method: PUT 2023-07-29T00:24:57.899 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000017 2023-07-29T00:24:57.900 DEBUG:boto:Data: 2023-07-29T00:24:57.900 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:57.900 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:57.901 DEBUG:boto:Port: 8000 2023-07-29T00:24:57.901 DEBUG:boto:Params: {} 2023-07-29T00:24:57.902 DEBUG:boto:Token: None 2023-07-29T00:24:57.903 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:57 GMT /hiiqos-58/obj00000000000000000017 2023-07-29T00:24:57.903 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S4nO2oMTaSG6HJGBcn8Syuz4vxE= 2023-07-29T00:24:57.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S4nO2oMTaSG6HJGBcn8Syuz4vxE='} 2023-07-29T00:24:57.910 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001d99e04dcc43208f-0064c45c59-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:57.910 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:57.911 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:57.911 DEBUG:boto:Method: HEAD 2023-07-29T00:24:57.912 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:57.912 DEBUG:boto:Data: 2023-07-29T00:24:57.912 DEBUG:boto:Headers: {} 2023-07-29T00:24:57.913 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:57.914 DEBUG:boto:Port: 8000 2023-07-29T00:24:57.914 DEBUG:boto:Params: {} 2023-07-29T00:24:57.914 DEBUG:boto:Token: None 2023-07-29T00:24:57.915 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:57 GMT /hiiqos-58/ 2023-07-29T00:24:57.915 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GtI6bJ0lM5N4fQ6+f2oz8G4xFdg= 2023-07-29T00:24:57.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GtI6bJ0lM5N4fQ6+f2oz8G4xFdg=', 'Content-Length': '0'} 2023-07-29T00:24:57.919 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7b331594c10e095-0064c45c59-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:57.920 DEBUG:boto:path=/hiiqos-58/obj00000000000000000162 2023-07-29T00:24:57.920 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000162 2023-07-29T00:24:57.921 DEBUG:boto:Method: PUT 2023-07-29T00:24:57.921 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000162 2023-07-29T00:24:57.922 DEBUG:boto:Data: 2023-07-29T00:24:57.922 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:57.923 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:57.923 DEBUG:boto:Port: 8000 2023-07-29T00:24:57.923 DEBUG:boto:Params: {} 2023-07-29T00:24:57.924 DEBUG:boto:Token: None 2023-07-29T00:24:57.925 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:57 GMT /hiiqos-58/obj00000000000000000162 2023-07-29T00:24:57.925 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/mc6YPlqvU1DNbfmG1cctuNzQLI= 2023-07-29T00:24:57.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/mc6YPlqvU1DNbfmG1cctuNzQLI='} 2023-07-29T00:24:57.972 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ed96ef0f69f9ced2-0064c45c59-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:57.973 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:57.974 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:57.974 DEBUG:boto:Method: HEAD 2023-07-29T00:24:57.975 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:57.975 DEBUG:boto:Data: 2023-07-29T00:24:57.976 DEBUG:boto:Headers: {} 2023-07-29T00:24:57.976 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:57.976 DEBUG:boto:Port: 8000 2023-07-29T00:24:57.977 DEBUG:boto:Params: {} 2023-07-29T00:24:57.977 DEBUG:boto:Token: None 2023-07-29T00:24:57.978 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:57 GMT /hiiqos-58/ 2023-07-29T00:24:57.978 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GtI6bJ0lM5N4fQ6+f2oz8G4xFdg= 2023-07-29T00:24:57.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GtI6bJ0lM5N4fQ6+f2oz8G4xFdg=', 'Content-Length': '0'} 2023-07-29T00:24:57.982 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001da050ea19bfc353-0064c45c59-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:57.983 DEBUG:boto:path=/hiiqos-58/obj00000000000000000050 2023-07-29T00:24:57.983 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000050 2023-07-29T00:24:57.984 DEBUG:boto:Method: PUT 2023-07-29T00:24:57.984 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000050 2023-07-29T00:24:57.984 DEBUG:boto:Data: 2023-07-29T00:24:57.985 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:57.985 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:57.986 DEBUG:boto:Port: 8000 2023-07-29T00:24:57.986 DEBUG:boto:Params: {} 2023-07-29T00:24:57.986 DEBUG:boto:Token: None 2023-07-29T00:24:57.987 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:57 GMT /hiiqos-58/obj00000000000000000050 2023-07-29T00:24:57.987 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BrEKW17/ofB7aRHvQHmRjrzBIro= 2023-07-29T00:24:57.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BrEKW17/ofB7aRHvQHmRjrzBIro='} 2023-07-29T00:24:57.993 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000066ba3409fa75da81-0064c45c59-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:57.995 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:57.995 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:57.995 DEBUG:boto:Method: HEAD 2023-07-29T00:24:57.996 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:57.996 DEBUG:boto:Data: 2023-07-29T00:24:57.997 DEBUG:boto:Headers: {} 2023-07-29T00:24:57.998 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:57.999 DEBUG:boto:Port: 8000 2023-07-29T00:24:58.000 DEBUG:boto:Params: {} 2023-07-29T00:24:58.001 DEBUG:boto:Token: None 2023-07-29T00:24:58.001 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:58 GMT /hiiqos-58/ 2023-07-29T00:24:58.002 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oBFDLeKeY7ytS2Tybci/Xn5QPu8= 2023-07-29T00:24:58.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oBFDLeKeY7ytS2Tybci/Xn5QPu8=', 'Content-Length': '0'} 2023-07-29T00:24:58.005 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000061b0c61db27c0979-0064c45c59-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:58.007 DEBUG:boto:path=/hiiqos-58/obj00000000000000000423 2023-07-29T00:24:58.007 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000423 2023-07-29T00:24:58.008 DEBUG:boto:Method: PUT 2023-07-29T00:24:58.009 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000423 2023-07-29T00:24:58.010 DEBUG:boto:Data: 2023-07-29T00:24:58.011 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:58.011 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:58.012 DEBUG:boto:Port: 8000 2023-07-29T00:24:58.013 DEBUG:boto:Params: {} 2023-07-29T00:24:58.014 DEBUG:boto:Token: None 2023-07-29T00:24:58.014 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:58 GMT /hiiqos-58/obj00000000000000000423 2023-07-29T00:24:58.015 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EfMxNA8JRV7/kd35EnhrhbTUFeE= 2023-07-29T00:24:58.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EfMxNA8JRV7/kd35EnhrhbTUFeE='} 2023-07-29T00:24:58.060 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000043f2c92f612280e0-0064c45c59-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:58.061 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:58.062 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:58.062 DEBUG:boto:Method: HEAD 2023-07-29T00:24:58.063 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:58.063 DEBUG:boto:Data: 2023-07-29T00:24:58.064 DEBUG:boto:Headers: {} 2023-07-29T00:24:58.064 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:58.065 DEBUG:boto:Port: 8000 2023-07-29T00:24:58.065 DEBUG:boto:Params: {} 2023-07-29T00:24:58.066 DEBUG:boto:Token: None 2023-07-29T00:24:58.067 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:58 GMT /hiiqos-58/ 2023-07-29T00:24:58.067 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oBFDLeKeY7ytS2Tybci/Xn5QPu8= 2023-07-29T00:24:58.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oBFDLeKeY7ytS2Tybci/Xn5QPu8=', 'Content-Length': '0'} 2023-07-29T00:24:58.071 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '135'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001387a4ee8e9c0e1f-0064c45c5a-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:58.072 DEBUG:boto:path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:24:58.072 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:24:58.073 DEBUG:boto:Method: PUT 2023-07-29T00:24:58.073 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000011 2023-07-29T00:24:58.074 DEBUG:boto:Data: 2023-07-29T00:24:58.074 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:58.075 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:58.075 DEBUG:boto:Port: 8000 2023-07-29T00:24:58.075 DEBUG:boto:Params: {} 2023-07-29T00:24:58.076 DEBUG:boto:Token: None 2023-07-29T00:24:58.076 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:58 GMT /hiiqos-58/obj00000000000000000011 2023-07-29T00:24:58.077 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+9RHpDXN7nFbHEy9uxf8zg52uzE= 2023-07-29T00:24:58.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+9RHpDXN7nFbHEy9uxf8zg52uzE='} 2023-07-29T00:24:58.083 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000d8f608ac9d958bd-0064c45c5a-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:58.084 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:58.085 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:58.086 DEBUG:boto:Method: HEAD 2023-07-29T00:24:58.086 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:58.087 DEBUG:boto:Data: 2023-07-29T00:24:58.087 DEBUG:boto:Headers: {} 2023-07-29T00:24:58.088 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:58.088 DEBUG:boto:Port: 8000 2023-07-29T00:24:58.088 DEBUG:boto:Params: {} 2023-07-29T00:24:58.089 DEBUG:boto:Token: None 2023-07-29T00:24:58.089 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:58 GMT /hiiqos-58/ 2023-07-29T00:24:58.089 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oBFDLeKeY7ytS2Tybci/Xn5QPu8= 2023-07-29T00:24:58.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oBFDLeKeY7ytS2Tybci/Xn5QPu8=', 'Content-Length': '0'} 2023-07-29T00:24:58.093 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '135'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b51b7b3a6a03555e-0064c45c5a-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:58.093 DEBUG:boto:path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:24:58.094 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:24:58.094 DEBUG:boto:Method: PUT 2023-07-29T00:24:58.094 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000004 2023-07-29T00:24:58.095 DEBUG:boto:Data: 2023-07-29T00:24:58.095 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:58.095 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:58.096 DEBUG:boto:Port: 8000 2023-07-29T00:24:58.096 DEBUG:boto:Params: {} 2023-07-29T00:24:58.097 DEBUG:boto:Token: None 2023-07-29T00:24:58.097 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:58 GMT /hiiqos-58/obj00000000000000000004 2023-07-29T00:24:58.097 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:R423gCzrbO+GH0Dw69QlHNyiqCM= 2023-07-29T00:24:58.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:R423gCzrbO+GH0Dw69QlHNyiqCM='} 2023-07-29T00:24:58.148 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000204aa9928a5ba044-0064c45c5a-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:58.150 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:58.150 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:58.151 DEBUG:boto:Method: HEAD 2023-07-29T00:24:58.152 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:58.153 DEBUG:boto:Data: 2023-07-29T00:24:58.153 DEBUG:boto:Headers: {} 2023-07-29T00:24:58.154 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:58.154 DEBUG:boto:Port: 8000 2023-07-29T00:24:58.154 DEBUG:boto:Params: {} 2023-07-29T00:24:58.155 DEBUG:boto:Token: None 2023-07-29T00:24:58.155 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:58 GMT /hiiqos-58/ 2023-07-29T00:24:58.156 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oBFDLeKeY7ytS2Tybci/Xn5QPu8= 2023-07-29T00:24:58.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oBFDLeKeY7ytS2Tybci/Xn5QPu8=', 'Content-Length': '0'} 2023-07-29T00:24:58.161 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '135'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005e84f54c5bf67529-0064c45c5a-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:58.162 DEBUG:boto:path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:24:58.163 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:24:58.164 DEBUG:boto:Method: PUT 2023-07-29T00:24:58.165 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000005 2023-07-29T00:24:58.167 DEBUG:boto:Data: 2023-07-29T00:24:58.167 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:58.168 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:58.169 DEBUG:boto:Port: 8000 2023-07-29T00:24:58.170 DEBUG:boto:Params: {} 2023-07-29T00:24:58.170 DEBUG:boto:Token: None 2023-07-29T00:24:58.171 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:58 GMT /hiiqos-58/obj00000000000000000005 2023-07-29T00:24:58.172 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zvmftrdQ6hFUOU3WU6H0XU1sr0Q= 2023-07-29T00:24:58.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zvmftrdQ6hFUOU3WU6H0XU1sr0Q='} 2023-07-29T00:24:58.178 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b0173ad149d2f6df-0064c45c5a-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:58.179 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:58.180 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:58.181 DEBUG:boto:Method: HEAD 2023-07-29T00:24:58.182 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:58.183 DEBUG:boto:Data: 2023-07-29T00:24:58.184 DEBUG:boto:Headers: {} 2023-07-29T00:24:58.184 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:58.185 DEBUG:boto:Port: 8000 2023-07-29T00:24:58.186 DEBUG:boto:Params: {} 2023-07-29T00:24:58.187 DEBUG:boto:Token: None 2023-07-29T00:24:58.187 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:58 GMT /hiiqos-58/ 2023-07-29T00:24:58.188 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oBFDLeKeY7ytS2Tybci/Xn5QPu8= 2023-07-29T00:24:58.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oBFDLeKeY7ytS2Tybci/Xn5QPu8=', 'Content-Length': '0'} 2023-07-29T00:24:58.192 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '135'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b2354870c77b87a6-0064c45c5a-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:58.193 DEBUG:boto:path=/hiiqos-58/obj00000000000000000271 2023-07-29T00:24:58.193 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000271 2023-07-29T00:24:58.194 DEBUG:boto:Method: PUT 2023-07-29T00:24:58.194 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000271 2023-07-29T00:24:58.194 DEBUG:boto:Data: 2023-07-29T00:24:58.195 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:58.195 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:58.195 DEBUG:boto:Port: 8000 2023-07-29T00:24:58.196 DEBUG:boto:Params: {} 2023-07-29T00:24:58.196 DEBUG:boto:Token: None 2023-07-29T00:24:58.197 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:58 GMT /hiiqos-58/obj00000000000000000271 2023-07-29T00:24:58.197 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yvSyRpqRJJvQBamTglcDrSX7vRE= 2023-07-29T00:24:58.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yvSyRpqRJJvQBamTglcDrSX7vRE='} 2023-07-29T00:24:58.244 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005c2d11ad351449cc-0064c45c5a-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:58.245 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:58.245 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:58.246 DEBUG:boto:Method: HEAD 2023-07-29T00:24:58.246 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:58.246 DEBUG:boto:Data: 2023-07-29T00:24:58.247 DEBUG:boto:Headers: {} 2023-07-29T00:24:58.247 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:58.247 DEBUG:boto:Port: 8000 2023-07-29T00:24:58.248 DEBUG:boto:Params: {} 2023-07-29T00:24:58.248 DEBUG:boto:Token: None 2023-07-29T00:24:58.249 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:58 GMT /hiiqos-58/ 2023-07-29T00:24:58.249 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oBFDLeKeY7ytS2Tybci/Xn5QPu8= 2023-07-29T00:24:58.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oBFDLeKeY7ytS2Tybci/Xn5QPu8=', 'Content-Length': '0'} 2023-07-29T00:24:58.253 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000099d6cc256290196-0064c45c5a-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:58.253 DEBUG:boto:path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:24:58.254 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:24:58.254 DEBUG:boto:Method: PUT 2023-07-29T00:24:58.255 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000012 2023-07-29T00:24:58.255 DEBUG:boto:Data: 2023-07-29T00:24:58.255 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:58.256 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:58.256 DEBUG:boto:Port: 8000 2023-07-29T00:24:58.256 DEBUG:boto:Params: {} 2023-07-29T00:24:58.257 DEBUG:boto:Token: None 2023-07-29T00:24:58.257 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:58 GMT /hiiqos-58/obj00000000000000000012 2023-07-29T00:24:58.257 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LFqRs7z7nTWwEToNfnejWEf4/7M= 2023-07-29T00:24:58.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LFqRs7z7nTWwEToNfnejWEf4/7M='} 2023-07-29T00:24:58.264 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dfa468ff47448e3e-0064c45c5a-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:58.265 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:58.265 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:58.265 DEBUG:boto:Method: HEAD 2023-07-29T00:24:58.266 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:58.266 DEBUG:boto:Data: 2023-07-29T00:24:58.266 DEBUG:boto:Headers: {} 2023-07-29T00:24:58.267 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:58.267 DEBUG:boto:Port: 8000 2023-07-29T00:24:58.267 DEBUG:boto:Params: {} 2023-07-29T00:24:58.268 DEBUG:boto:Token: None 2023-07-29T00:24:58.268 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:58 GMT /hiiqos-58/ 2023-07-29T00:24:58.268 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oBFDLeKeY7ytS2Tybci/Xn5QPu8= 2023-07-29T00:24:58.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oBFDLeKeY7ytS2Tybci/Xn5QPu8=', 'Content-Length': '0'} 2023-07-29T00:24:58.272 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004bb6707dbe4e1108-0064c45c5a-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:58.273 DEBUG:boto:path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:24:58.273 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:24:58.274 DEBUG:boto:Method: PUT 2023-07-29T00:24:58.274 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000001 2023-07-29T00:24:58.274 DEBUG:boto:Data: 2023-07-29T00:24:58.275 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:58.275 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:58.275 DEBUG:boto:Port: 8000 2023-07-29T00:24:58.276 DEBUG:boto:Params: {} 2023-07-29T00:24:58.276 DEBUG:boto:Token: None 2023-07-29T00:24:58.277 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:58 GMT /hiiqos-58/obj00000000000000000001 2023-07-29T00:24:58.277 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:z71DzUB73dPtfrEoxRPusnBDqwM= 2023-07-29T00:24:58.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:z71DzUB73dPtfrEoxRPusnBDqwM='} 2023-07-29T00:24:58.320 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008f5bb4825de353d0-0064c45c5a-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:58.321 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:58.321 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:58.322 DEBUG:boto:Method: HEAD 2023-07-29T00:24:58.322 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:58.322 DEBUG:boto:Data: 2023-07-29T00:24:58.323 DEBUG:boto:Headers: {} 2023-07-29T00:24:58.323 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:58.323 DEBUG:boto:Port: 8000 2023-07-29T00:24:58.324 DEBUG:boto:Params: {} 2023-07-29T00:24:58.324 DEBUG:boto:Token: None 2023-07-29T00:24:58.325 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:58 GMT /hiiqos-58/ 2023-07-29T00:24:58.325 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oBFDLeKeY7ytS2Tybci/Xn5QPu8= 2023-07-29T00:24:58.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oBFDLeKeY7ytS2Tybci/Xn5QPu8=', 'Content-Length': '0'} 2023-07-29T00:24:58.329 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c3c2669090b0f08e-0064c45c5a-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:58.329 DEBUG:boto:path=/hiiqos-58/obj00000000000000000029 2023-07-29T00:24:58.330 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000029 2023-07-29T00:24:58.330 DEBUG:boto:Method: PUT 2023-07-29T00:24:58.330 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000029 2023-07-29T00:24:58.331 DEBUG:boto:Data: 2023-07-29T00:24:58.331 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:58.331 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:58.332 DEBUG:boto:Port: 8000 2023-07-29T00:24:58.332 DEBUG:boto:Params: {} 2023-07-29T00:24:58.333 DEBUG:boto:Token: None 2023-07-29T00:24:58.333 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:58 GMT /hiiqos-58/obj00000000000000000029 2023-07-29T00:24:58.333 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:t2VmdwoN4gA22PHE5T6kU97t6hc= 2023-07-29T00:24:58.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t2VmdwoN4gA22PHE5T6kU97t6hc='} 2023-07-29T00:24:58.339 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f31d1b7b2b7dae0f-0064c45c5a-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:58.339 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:58.340 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:58.340 DEBUG:boto:Method: HEAD 2023-07-29T00:24:58.340 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:58.341 DEBUG:boto:Data: 2023-07-29T00:24:58.341 DEBUG:boto:Headers: {} 2023-07-29T00:24:58.341 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:58.342 DEBUG:boto:Port: 8000 2023-07-29T00:24:58.342 DEBUG:boto:Params: {} 2023-07-29T00:24:58.342 DEBUG:boto:Token: None 2023-07-29T00:24:58.343 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:58 GMT /hiiqos-58/ 2023-07-29T00:24:58.343 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oBFDLeKeY7ytS2Tybci/Xn5QPu8= 2023-07-29T00:24:58.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oBFDLeKeY7ytS2Tybci/Xn5QPu8=', 'Content-Length': '0'} 2023-07-29T00:24:58.347 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '141'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001a81513bd5f38c2e-0064c45c5a-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:58.348 DEBUG:boto:path=/hiiqos-58/obj00000000000000000009 2023-07-29T00:24:58.348 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000009 2023-07-29T00:24:58.348 DEBUG:boto:Method: PUT 2023-07-29T00:24:58.349 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000009 2023-07-29T00:24:58.349 DEBUG:boto:Data: 2023-07-29T00:24:58.349 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:58.350 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:58.350 DEBUG:boto:Port: 8000 2023-07-29T00:24:58.350 DEBUG:boto:Params: {} 2023-07-29T00:24:58.351 DEBUG:boto:Token: None 2023-07-29T00:24:58.351 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:58 GMT /hiiqos-58/obj00000000000000000009 2023-07-29T00:24:58.352 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GwoansxfjeZKwazATrfQCR6Smi8= 2023-07-29T00:24:58.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GwoansxfjeZKwazATrfQCR6Smi8='} 2023-07-29T00:24:58.396 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004eb1c9bcb9a11d4d-0064c45c5a-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:58.397 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:58.397 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:58.398 DEBUG:boto:Method: HEAD 2023-07-29T00:24:58.398 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:58.398 DEBUG:boto:Data: 2023-07-29T00:24:58.399 DEBUG:boto:Headers: {} 2023-07-29T00:24:58.399 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:58.399 DEBUG:boto:Port: 8000 2023-07-29T00:24:58.400 DEBUG:boto:Params: {} 2023-07-29T00:24:58.400 DEBUG:boto:Token: None 2023-07-29T00:24:58.401 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:58 GMT /hiiqos-58/ 2023-07-29T00:24:58.401 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oBFDLeKeY7ytS2Tybci/Xn5QPu8= 2023-07-29T00:24:58.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oBFDLeKeY7ytS2Tybci/Xn5QPu8=', 'Content-Length': '0'} 2023-07-29T00:24:58.405 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '141'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000574b19ee29fbea94-0064c45c5a-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:58.405 DEBUG:boto:path=/hiiqos-58/obj00000000000000000016 2023-07-29T00:24:58.406 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000016 2023-07-29T00:24:58.406 DEBUG:boto:Method: PUT 2023-07-29T00:24:58.406 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000016 2023-07-29T00:24:58.407 DEBUG:boto:Data: 2023-07-29T00:24:58.407 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:58.407 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:58.408 DEBUG:boto:Port: 8000 2023-07-29T00:24:58.408 DEBUG:boto:Params: {} 2023-07-29T00:24:58.409 DEBUG:boto:Token: None 2023-07-29T00:24:58.409 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:58 GMT /hiiqos-58/obj00000000000000000016 2023-07-29T00:24:58.409 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0/MBxJLtly8GtkfofJ5SVHQwZwc= 2023-07-29T00:24:58.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0/MBxJLtly8GtkfofJ5SVHQwZwc='} 2023-07-29T00:24:58.416 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005b5e73de3e99be22-0064c45c5a-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:58.417 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:58.417 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:58.417 DEBUG:boto:Method: HEAD 2023-07-29T00:24:58.418 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:58.418 DEBUG:boto:Data: 2023-07-29T00:24:58.418 DEBUG:boto:Headers: {} 2023-07-29T00:24:58.419 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:58.419 DEBUG:boto:Port: 8000 2023-07-29T00:24:58.419 DEBUG:boto:Params: {} 2023-07-29T00:24:58.420 DEBUG:boto:Token: None 2023-07-29T00:24:58.420 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:58 GMT /hiiqos-58/ 2023-07-29T00:24:58.421 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oBFDLeKeY7ytS2Tybci/Xn5QPu8= 2023-07-29T00:24:58.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oBFDLeKeY7ytS2Tybci/Xn5QPu8=', 'Content-Length': '0'} 2023-07-29T00:24:58.424 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '141'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd208b6cd2c3e3b1-0064c45c5a-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:58.425 DEBUG:boto:path=/hiiqos-58/obj00000000000000000023 2023-07-29T00:24:58.425 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000023 2023-07-29T00:24:58.425 DEBUG:boto:Method: PUT 2023-07-29T00:24:58.426 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000023 2023-07-29T00:24:58.426 DEBUG:boto:Data: 2023-07-29T00:24:58.427 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:58.427 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:58.427 DEBUG:boto:Port: 8000 2023-07-29T00:24:58.427 DEBUG:boto:Params: {} 2023-07-29T00:24:58.428 DEBUG:boto:Token: None 2023-07-29T00:24:58.428 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:58 GMT /hiiqos-58/obj00000000000000000023 2023-07-29T00:24:58.429 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3l0oeLhkfjYMgzRSKPYoF1xMjRM= 2023-07-29T00:24:58.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3l0oeLhkfjYMgzRSKPYoF1xMjRM='} 2023-07-29T00:24:58.472 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a86b616aa1e6623a-0064c45c5a-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:58.473 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:58.474 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:58.474 DEBUG:boto:Method: HEAD 2023-07-29T00:24:58.474 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:58.475 DEBUG:boto:Data: 2023-07-29T00:24:58.475 DEBUG:boto:Headers: {} 2023-07-29T00:24:58.475 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:58.476 DEBUG:boto:Port: 8000 2023-07-29T00:24:58.476 DEBUG:boto:Params: {} 2023-07-29T00:24:58.477 DEBUG:boto:Token: None 2023-07-29T00:24:58.477 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:58 GMT /hiiqos-58/ 2023-07-29T00:24:58.477 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oBFDLeKeY7ytS2Tybci/Xn5QPu8= 2023-07-29T00:24:58.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oBFDLeKeY7ytS2Tybci/Xn5QPu8=', 'Content-Length': '0'} 2023-07-29T00:24:58.481 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000047739a6d4f71ba64-0064c45c5a-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:58.482 DEBUG:boto:path=/hiiqos-58/obj00000000000000000010 2023-07-29T00:24:58.482 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000010 2023-07-29T00:24:58.483 DEBUG:boto:Method: PUT 2023-07-29T00:24:58.483 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000010 2023-07-29T00:24:58.483 DEBUG:boto:Data: 2023-07-29T00:24:58.484 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:58.484 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:58.484 DEBUG:boto:Port: 8000 2023-07-29T00:24:58.485 DEBUG:boto:Params: {} 2023-07-29T00:24:58.485 DEBUG:boto:Token: None 2023-07-29T00:24:58.486 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:58 GMT /hiiqos-58/obj00000000000000000010 2023-07-29T00:24:58.486 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FmDzOGIP2J444IMUTnTfgsTfXaU= 2023-07-29T00:24:58.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FmDzOGIP2J444IMUTnTfgsTfXaU='} 2023-07-29T00:24:58.493 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c4dd3333992781ce-0064c45c5a-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:58.494 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:58.494 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:58.494 DEBUG:boto:Method: HEAD 2023-07-29T00:24:58.495 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:58.495 DEBUG:boto:Data: 2023-07-29T00:24:58.495 DEBUG:boto:Headers: {} 2023-07-29T00:24:58.496 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:58.496 DEBUG:boto:Port: 8000 2023-07-29T00:24:58.497 DEBUG:boto:Params: {} 2023-07-29T00:24:58.497 DEBUG:boto:Token: None 2023-07-29T00:24:58.497 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:58 GMT /hiiqos-58/ 2023-07-29T00:24:58.498 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oBFDLeKeY7ytS2Tybci/Xn5QPu8= 2023-07-29T00:24:58.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oBFDLeKeY7ytS2Tybci/Xn5QPu8=', 'Content-Length': '0'} 2023-07-29T00:24:58.502 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d65a05e9070f5cb5-0064c45c5a-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:58.502 DEBUG:boto:path=/hiiqos-58/obj00000000000000000467 2023-07-29T00:24:58.503 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000467 2023-07-29T00:24:58.503 DEBUG:boto:Method: PUT 2023-07-29T00:24:58.503 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000467 2023-07-29T00:24:58.504 DEBUG:boto:Data: 2023-07-29T00:24:58.504 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:58.505 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:58.505 DEBUG:boto:Port: 8000 2023-07-29T00:24:58.506 DEBUG:boto:Params: {} 2023-07-29T00:24:58.506 DEBUG:boto:Token: None 2023-07-29T00:24:58.507 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:58 GMT /hiiqos-58/obj00000000000000000467 2023-07-29T00:24:58.507 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O0Nhlom1T8a8gnbnvCkB00DEYzQ= 2023-07-29T00:24:58.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O0Nhlom1T8a8gnbnvCkB00DEYzQ='} 2023-07-29T00:24:58.552 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f5097383aadb7bfd-0064c45c5a-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:58.553 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:58.554 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:58.554 DEBUG:boto:Method: HEAD 2023-07-29T00:24:58.554 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:58.555 DEBUG:boto:Data: 2023-07-29T00:24:58.555 DEBUG:boto:Headers: {} 2023-07-29T00:24:58.555 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:58.556 DEBUG:boto:Port: 8000 2023-07-29T00:24:58.556 DEBUG:boto:Params: {} 2023-07-29T00:24:58.556 DEBUG:boto:Token: None 2023-07-29T00:24:58.557 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:58 GMT /hiiqos-58/ 2023-07-29T00:24:58.557 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oBFDLeKeY7ytS2Tybci/Xn5QPu8= 2023-07-29T00:24:58.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oBFDLeKeY7ytS2Tybci/Xn5QPu8=', 'Content-Length': '0'} 2023-07-29T00:24:58.560 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '49'), ('X-RGW-Bytes-Used', '147'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007a361212fee2b817-0064c45c5a-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:58.561 DEBUG:boto:path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:24:58.561 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:24:58.562 DEBUG:boto:Method: PUT 2023-07-29T00:24:58.562 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000002 2023-07-29T00:24:58.563 DEBUG:boto:Data: 2023-07-29T00:24:58.563 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:58.563 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:58.564 DEBUG:boto:Port: 8000 2023-07-29T00:24:58.564 DEBUG:boto:Params: {} 2023-07-29T00:24:58.564 DEBUG:boto:Token: None 2023-07-29T00:24:58.565 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:58 GMT /hiiqos-58/obj00000000000000000002 2023-07-29T00:24:58.566 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7vgxsgm6NAfbUvENR564P+X6uvo= 2023-07-29T00:24:58.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7vgxsgm6NAfbUvENR564P+X6uvo='} 2023-07-29T00:24:58.572 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e455591d2d4955ba-0064c45c5a-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:58.573 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:58.573 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:58.573 DEBUG:boto:Method: HEAD 2023-07-29T00:24:58.574 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:58.574 DEBUG:boto:Data: 2023-07-29T00:24:58.574 DEBUG:boto:Headers: {} 2023-07-29T00:24:58.575 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:58.575 DEBUG:boto:Port: 8000 2023-07-29T00:24:58.575 DEBUG:boto:Params: {} 2023-07-29T00:24:58.576 DEBUG:boto:Token: None 2023-07-29T00:24:58.577 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:58 GMT /hiiqos-58/ 2023-07-29T00:24:58.577 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oBFDLeKeY7ytS2Tybci/Xn5QPu8= 2023-07-29T00:24:58.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oBFDLeKeY7ytS2Tybci/Xn5QPu8=', 'Content-Length': '0'} 2023-07-29T00:24:58.580 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '49'), ('X-RGW-Bytes-Used', '147'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c15cc0ec5b8bdf0b-0064c45c5a-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:58.581 DEBUG:boto:path=/hiiqos-58/obj00000000000000000018 2023-07-29T00:24:58.581 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000018 2023-07-29T00:24:58.582 DEBUG:boto:Method: PUT 2023-07-29T00:24:58.582 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000018 2023-07-29T00:24:58.583 DEBUG:boto:Data: 2023-07-29T00:24:58.583 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:58.584 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:58.585 DEBUG:boto:Port: 8000 2023-07-29T00:24:58.586 DEBUG:boto:Params: {} 2023-07-29T00:24:58.587 DEBUG:boto:Token: None 2023-07-29T00:24:58.588 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:58 GMT /hiiqos-58/obj00000000000000000018 2023-07-29T00:24:58.588 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mTthGFfUEpxLu44PTd5cp1aeYMk= 2023-07-29T00:24:58.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mTthGFfUEpxLu44PTd5cp1aeYMk='} 2023-07-29T00:24:58.632 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005e815da38451234b-0064c45c5a-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:58.633 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:58.634 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:58.634 DEBUG:boto:Method: HEAD 2023-07-29T00:24:58.634 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:58.635 DEBUG:boto:Data: 2023-07-29T00:24:58.635 DEBUG:boto:Headers: {} 2023-07-29T00:24:58.635 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:58.636 DEBUG:boto:Port: 8000 2023-07-29T00:24:58.636 DEBUG:boto:Params: {} 2023-07-29T00:24:58.636 DEBUG:boto:Token: None 2023-07-29T00:24:58.637 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:58 GMT /hiiqos-58/ 2023-07-29T00:24:58.637 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oBFDLeKeY7ytS2Tybci/Xn5QPu8= 2023-07-29T00:24:58.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oBFDLeKeY7ytS2Tybci/Xn5QPu8=', 'Content-Length': '0'} 2023-07-29T00:24:58.641 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '50'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bceb366cf8da517a-0064c45c5a-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:58.641 DEBUG:boto:path=/hiiqos-58/obj00000000000000000061 2023-07-29T00:24:58.642 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000061 2023-07-29T00:24:58.642 DEBUG:boto:Method: PUT 2023-07-29T00:24:58.642 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000061 2023-07-29T00:24:58.643 DEBUG:boto:Data: 2023-07-29T00:24:58.643 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:58.643 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:58.644 DEBUG:boto:Port: 8000 2023-07-29T00:24:58.644 DEBUG:boto:Params: {} 2023-07-29T00:24:58.645 DEBUG:boto:Token: None 2023-07-29T00:24:58.645 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:58 GMT /hiiqos-58/obj00000000000000000061 2023-07-29T00:24:58.645 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:M9JucMy9auDj/vjx++qqRMtA+Dk= 2023-07-29T00:24:58.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:M9JucMy9auDj/vjx++qqRMtA+Dk='} 2023-07-29T00:24:58.651 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d1d3155e2af6aee2-0064c45c5a-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:58.651 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:58.652 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:58.652 DEBUG:boto:Method: HEAD 2023-07-29T00:24:58.652 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:58.653 DEBUG:boto:Data: 2023-07-29T00:24:58.653 DEBUG:boto:Headers: {} 2023-07-29T00:24:58.653 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:58.654 DEBUG:boto:Port: 8000 2023-07-29T00:24:58.654 DEBUG:boto:Params: {} 2023-07-29T00:24:58.654 DEBUG:boto:Token: None 2023-07-29T00:24:58.655 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:58 GMT /hiiqos-58/ 2023-07-29T00:24:58.655 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oBFDLeKeY7ytS2Tybci/Xn5QPu8= 2023-07-29T00:24:58.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oBFDLeKeY7ytS2Tybci/Xn5QPu8=', 'Content-Length': '0'} 2023-07-29T00:24:58.658 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '51'), ('X-RGW-Bytes-Used', '153'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fd92f9ae02440c88-0064c45c5a-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:58.659 DEBUG:boto:path=/hiiqos-58/obj00000000000000000317 2023-07-29T00:24:58.659 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000317 2023-07-29T00:24:58.660 DEBUG:boto:Method: PUT 2023-07-29T00:24:58.660 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000317 2023-07-29T00:24:58.661 DEBUG:boto:Data: 2023-07-29T00:24:58.661 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:58.661 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:58.662 DEBUG:boto:Port: 8000 2023-07-29T00:24:58.662 DEBUG:boto:Params: {} 2023-07-29T00:24:58.662 DEBUG:boto:Token: None 2023-07-29T00:24:58.663 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:58 GMT /hiiqos-58/obj00000000000000000317 2023-07-29T00:24:58.663 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1Pgw7JFKAfRKDY1Z0te+K1Gg1As= 2023-07-29T00:24:58.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1Pgw7JFKAfRKDY1Z0te+K1Gg1As='} 2023-07-29T00:24:58.712 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002b33ba4f754b3917-0064c45c5a-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:58.713 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:58.713 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:58.714 DEBUG:boto:Method: HEAD 2023-07-29T00:24:58.714 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:58.715 DEBUG:boto:Data: 2023-07-29T00:24:58.715 DEBUG:boto:Headers: {} 2023-07-29T00:24:58.716 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:58.716 DEBUG:boto:Port: 8000 2023-07-29T00:24:58.717 DEBUG:boto:Params: {} 2023-07-29T00:24:58.717 DEBUG:boto:Token: None 2023-07-29T00:24:58.717 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:58 GMT /hiiqos-58/ 2023-07-29T00:24:58.718 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oBFDLeKeY7ytS2Tybci/Xn5QPu8= 2023-07-29T00:24:58.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oBFDLeKeY7ytS2Tybci/Xn5QPu8=', 'Content-Length': '0'} 2023-07-29T00:24:58.721 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '52'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000084b3adce4de650a-0064c45c5a-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:58.722 DEBUG:boto:path=/hiiqos-58/obj00000000000000000019 2023-07-29T00:24:58.722 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000019 2023-07-29T00:24:58.723 DEBUG:boto:Method: PUT 2023-07-29T00:24:58.723 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000019 2023-07-29T00:24:58.723 DEBUG:boto:Data: 2023-07-29T00:24:58.724 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:58.724 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:58.724 DEBUG:boto:Port: 8000 2023-07-29T00:24:58.725 DEBUG:boto:Params: {} 2023-07-29T00:24:58.725 DEBUG:boto:Token: None 2023-07-29T00:24:58.725 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:58 GMT /hiiqos-58/obj00000000000000000019 2023-07-29T00:24:58.726 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:biZx1XV6X1Cp6RRg/RF2xdENxtc= 2023-07-29T00:24:58.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:biZx1XV6X1Cp6RRg/RF2xdENxtc='} 2023-07-29T00:24:58.733 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000db660a6222d2d982-0064c45c5a-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:58.734 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:58.734 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:58.734 DEBUG:boto:Method: HEAD 2023-07-29T00:24:58.735 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:58.735 DEBUG:boto:Data: 2023-07-29T00:24:58.735 DEBUG:boto:Headers: {} 2023-07-29T00:24:58.736 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:58.736 DEBUG:boto:Port: 8000 2023-07-29T00:24:58.736 DEBUG:boto:Params: {} 2023-07-29T00:24:58.737 DEBUG:boto:Token: None 2023-07-29T00:24:58.737 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:58 GMT /hiiqos-58/ 2023-07-29T00:24:58.738 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oBFDLeKeY7ytS2Tybci/Xn5QPu8= 2023-07-29T00:24:58.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oBFDLeKeY7ytS2Tybci/Xn5QPu8=', 'Content-Length': '0'} 2023-07-29T00:24:58.741 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '52'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000569239ac59c950c2-0064c45c5a-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:58.742 DEBUG:boto:path=/hiiqos-58/obj00000000000000000022 2023-07-29T00:24:58.742 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000022 2023-07-29T00:24:58.743 DEBUG:boto:Method: PUT 2023-07-29T00:24:58.743 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000022 2023-07-29T00:24:58.744 DEBUG:boto:Data: 2023-07-29T00:24:58.744 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:58.744 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:58.745 DEBUG:boto:Port: 8000 2023-07-29T00:24:58.745 DEBUG:boto:Params: {} 2023-07-29T00:24:58.746 DEBUG:boto:Token: None 2023-07-29T00:24:58.747 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:58 GMT /hiiqos-58/obj00000000000000000022 2023-07-29T00:24:58.748 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AQM4H91YqjJkK5Gd7xcxaiAAPwQ= 2023-07-29T00:24:58.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AQM4H91YqjJkK5Gd7xcxaiAAPwQ='} 2023-07-29T00:24:58.792 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e9dedd46a5aacbe7-0064c45c5a-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:58.793 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-58', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:24:58.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-58', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:24:58.794 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-58 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:24:58.844 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:24:58.845 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:24:58.891 INFO:teuthology.orchestra.run.smithi101.stdout:tenant: 2023-07-29T00:24:58.891 INFO:teuthology.orchestra.run.smithi101.stdout:bucket name: hiiqos-58 2023-07-29T00:24:58.902 INFO:teuthology.orchestra.run.smithi101.stdout:total entries: 53 2023-07-29T00:24:58.959 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:24:58.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:24:58.961 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 37 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:24:59.009 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:24:59.009 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:24:59.049 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:24:59.049 INFO:teuthology.orchestra.run.smithi101.stdout: "objs": [ 2023-07-29T00:24:59.049 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000060", 2023-07-29T00:24:59.050 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000000", 2023-07-29T00:24:59.050 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000011", 2023-07-29T00:24:59.050 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000001", 2023-07-29T00:24:59.050 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000033", 2023-07-29T00:24:59.050 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000065", 2023-07-29T00:24:59.050 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000005", 2023-07-29T00:24:59.051 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000016", 2023-07-29T00:24:59.051 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000048", 2023-07-29T00:24:59.051 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000015", 2023-07-29T00:24:59.051 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000026", 2023-07-29T00:24:59.051 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000030", 2023-07-29T00:24:59.051 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000062", 2023-07-29T00:24:59.051 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000009", 2023-07-29T00:24:59.052 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000040", 2023-07-29T00:24:59.052 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000024", 2023-07-29T00:24:59.052 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000012", 2023-07-29T00:24:59.052 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000002", 2023-07-29T00:24:59.052 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000013", 2023-07-29T00:24:59.053 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000045", 2023-07-29T00:24:59.053 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000006", 2023-07-29T00:24:59.053 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000017", 2023-07-29T00:24:59.054 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000007", 2023-07-29T00:24:59.054 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000039", 2023-07-29T00:24:59.054 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000027", 2023-07-29T00:24:59.055 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000010", 2023-07-29T00:24:59.055 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000021", 2023-07-29T00:24:59.055 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000053", 2023-07-29T00:24:59.056 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000020", 2023-07-29T00:24:59.057 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000004", 2023-07-29T00:24:59.057 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000036", 2023-07-29T00:24:59.057 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000003", 2023-07-29T00:24:59.058 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000014", 2023-07-29T00:24:59.058 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000019", 2023-07-29T00:24:59.058 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000057", 2023-07-29T00:24:59.058 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000018", 2023-07-29T00:24:59.059 INFO:teuthology.orchestra.run.smithi101.stdout: "obj00000000000000000008" 2023-07-29T00:24:59.059 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:24:59.059 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:24:59.061 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:59.061 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:59.062 DEBUG:boto:Method: HEAD 2023-07-29T00:24:59.063 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:59.063 DEBUG:boto:Data: 2023-07-29T00:24:59.064 DEBUG:boto:Headers: {} 2023-07-29T00:24:59.065 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.065 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.066 DEBUG:boto:Params: {} 2023-07-29T00:24:59.066 DEBUG:boto:Token: None 2023-07-29T00:24:59.067 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/ 2023-07-29T00:24:59.068 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw= 2023-07-29T00:24:59.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw=', 'Content-Length': '0'} 2023-07-29T00:24:59.072 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bc469ab58d966746-0064c45c5b-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.073 DEBUG:boto:path=/hiiqos-58/obj00000000000000000048 2023-07-29T00:24:59.073 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000048 2023-07-29T00:24:59.074 DEBUG:boto:Method: PUT 2023-07-29T00:24:59.074 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000048 2023-07-29T00:24:59.074 DEBUG:boto:Data: 2023-07-29T00:24:59.075 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:59.075 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.075 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.076 DEBUG:boto:Params: {} 2023-07-29T00:24:59.076 DEBUG:boto:Token: None 2023-07-29T00:24:59.077 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/obj00000000000000000048 2023-07-29T00:24:59.077 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:u1RCqzPWfUDGLE/9ZWnLRiFur9U= 2023-07-29T00:24:59.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:u1RCqzPWfUDGLE/9ZWnLRiFur9U='} 2023-07-29T00:24:59.083 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000030fd6dfa5889af75-0064c45c5b-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.084 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:59.084 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:59.085 DEBUG:boto:Method: HEAD 2023-07-29T00:24:59.086 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:59.086 DEBUG:boto:Data: 2023-07-29T00:24:59.087 DEBUG:boto:Headers: {} 2023-07-29T00:24:59.087 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.088 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.089 DEBUG:boto:Params: {} 2023-07-29T00:24:59.089 DEBUG:boto:Token: None 2023-07-29T00:24:59.090 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/ 2023-07-29T00:24:59.091 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw= 2023-07-29T00:24:59.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw=', 'Content-Length': '0'} 2023-07-29T00:24:59.095 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '54'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000457b9200d16fc382-0064c45c5b-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.096 DEBUG:boto:path=/hiiqos-58/obj00000000000000000016 2023-07-29T00:24:59.097 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000016 2023-07-29T00:24:59.098 DEBUG:boto:Method: PUT 2023-07-29T00:24:59.099 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000016 2023-07-29T00:24:59.100 DEBUG:boto:Data: 2023-07-29T00:24:59.101 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:59.102 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.102 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.103 DEBUG:boto:Params: {} 2023-07-29T00:24:59.103 DEBUG:boto:Token: None 2023-07-29T00:24:59.104 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/obj00000000000000000016 2023-07-29T00:24:59.104 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wyMPBt5GoxiXjzgvy//STd96s7Y= 2023-07-29T00:24:59.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wyMPBt5GoxiXjzgvy//STd96s7Y='} 2023-07-29T00:24:59.111 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000062b765f0622ef4df-0064c45c5b-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.112 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:59.112 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:59.112 DEBUG:boto:Method: HEAD 2023-07-29T00:24:59.113 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:59.113 DEBUG:boto:Data: 2023-07-29T00:24:59.113 DEBUG:boto:Headers: {} 2023-07-29T00:24:59.114 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.114 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.115 DEBUG:boto:Params: {} 2023-07-29T00:24:59.115 DEBUG:boto:Token: None 2023-07-29T00:24:59.116 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/ 2023-07-29T00:24:59.116 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw= 2023-07-29T00:24:59.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw=', 'Content-Length': '0'} 2023-07-29T00:24:59.121 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '54'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000068e146289806b507-0064c45c5b-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.122 DEBUG:boto:path=/hiiqos-58/obj00000000000000000030 2023-07-29T00:24:59.123 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000030 2023-07-29T00:24:59.123 DEBUG:boto:Method: PUT 2023-07-29T00:24:59.124 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000030 2023-07-29T00:24:59.124 DEBUG:boto:Data: 2023-07-29T00:24:59.124 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:59.125 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.125 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.125 DEBUG:boto:Params: {} 2023-07-29T00:24:59.126 DEBUG:boto:Token: None 2023-07-29T00:24:59.126 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/obj00000000000000000030 2023-07-29T00:24:59.127 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:n9z8uRuItRsKqcQseQbDMTm8GaI= 2023-07-29T00:24:59.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:n9z8uRuItRsKqcQseQbDMTm8GaI='} 2023-07-29T00:24:59.172 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003307459b78b96916-0064c45c5b-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.174 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:59.175 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:59.176 DEBUG:boto:Method: HEAD 2023-07-29T00:24:59.176 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:59.177 DEBUG:boto:Data: 2023-07-29T00:24:59.178 DEBUG:boto:Headers: {} 2023-07-29T00:24:59.179 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.180 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.181 DEBUG:boto:Params: {} 2023-07-29T00:24:59.182 DEBUG:boto:Token: None 2023-07-29T00:24:59.183 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/ 2023-07-29T00:24:59.184 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw= 2023-07-29T00:24:59.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw=', 'Content-Length': '0'} 2023-07-29T00:24:59.188 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '54'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000140e7a8eb4e6f08f-0064c45c5b-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.189 DEBUG:boto:path=/hiiqos-58/obj00000000000000000036 2023-07-29T00:24:59.190 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000036 2023-07-29T00:24:59.190 DEBUG:boto:Method: PUT 2023-07-29T00:24:59.191 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000036 2023-07-29T00:24:59.191 DEBUG:boto:Data: 2023-07-29T00:24:59.191 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:59.192 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.192 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.193 DEBUG:boto:Params: {} 2023-07-29T00:24:59.193 DEBUG:boto:Token: None 2023-07-29T00:24:59.194 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/obj00000000000000000036 2023-07-29T00:24:59.195 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UESKxGUEZW+pWclvr0Wvk+xqWkE= 2023-07-29T00:24:59.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UESKxGUEZW+pWclvr0Wvk+xqWkE='} 2023-07-29T00:24:59.200 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000c97ebf061a69396-0064c45c5b-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.201 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:59.201 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:59.202 DEBUG:boto:Method: HEAD 2023-07-29T00:24:59.202 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:59.202 DEBUG:boto:Data: 2023-07-29T00:24:59.203 DEBUG:boto:Headers: {} 2023-07-29T00:24:59.203 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.204 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.204 DEBUG:boto:Params: {} 2023-07-29T00:24:59.205 DEBUG:boto:Token: None 2023-07-29T00:24:59.205 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/ 2023-07-29T00:24:59.206 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw= 2023-07-29T00:24:59.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw=', 'Content-Length': '0'} 2023-07-29T00:24:59.210 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000def9286e73b9b0a4-0064c45c5b-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.211 DEBUG:boto:path=/hiiqos-58/obj00000000000000000026 2023-07-29T00:24:59.211 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000026 2023-07-29T00:24:59.212 DEBUG:boto:Method: PUT 2023-07-29T00:24:59.213 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000026 2023-07-29T00:24:59.214 DEBUG:boto:Data: 2023-07-29T00:24:59.215 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:59.217 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.217 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.218 DEBUG:boto:Params: {} 2023-07-29T00:24:59.218 DEBUG:boto:Token: None 2023-07-29T00:24:59.219 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/obj00000000000000000026 2023-07-29T00:24:59.219 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:m3RbQz68ZH7R0jIZakEIzlWfpDw= 2023-07-29T00:24:59.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:m3RbQz68ZH7R0jIZakEIzlWfpDw='} 2023-07-29T00:24:59.264 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bf1f3b2bba516ff3-0064c45c5b-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.265 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:59.265 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:59.266 DEBUG:boto:Method: HEAD 2023-07-29T00:24:59.267 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:59.267 DEBUG:boto:Data: 2023-07-29T00:24:59.267 DEBUG:boto:Headers: {} 2023-07-29T00:24:59.268 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.268 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.268 DEBUG:boto:Params: {} 2023-07-29T00:24:59.269 DEBUG:boto:Token: None 2023-07-29T00:24:59.269 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/ 2023-07-29T00:24:59.270 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw= 2023-07-29T00:24:59.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw=', 'Content-Length': '0'} 2023-07-29T00:24:59.273 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003675c82e037c2274-0064c45c5b-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.274 DEBUG:boto:path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:24:59.275 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:24:59.275 DEBUG:boto:Method: PUT 2023-07-29T00:24:59.276 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000004 2023-07-29T00:24:59.277 DEBUG:boto:Data: 2023-07-29T00:24:59.277 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:59.278 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.279 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.279 DEBUG:boto:Params: {} 2023-07-29T00:24:59.280 DEBUG:boto:Token: None 2023-07-29T00:24:59.281 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/obj00000000000000000004 2023-07-29T00:24:59.281 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1xJr4XzQdB1h0T/JHp/iG3a4Sw8= 2023-07-29T00:24:59.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1xJr4XzQdB1h0T/JHp/iG3a4Sw8='} 2023-07-29T00:24:59.289 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000037cc874cca5c3fca-0064c45c5b-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.290 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:59.291 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:59.292 DEBUG:boto:Method: HEAD 2023-07-29T00:24:59.293 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:59.293 DEBUG:boto:Data: 2023-07-29T00:24:59.294 DEBUG:boto:Headers: {} 2023-07-29T00:24:59.295 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.295 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.296 DEBUG:boto:Params: {} 2023-07-29T00:24:59.297 DEBUG:boto:Token: None 2023-07-29T00:24:59.298 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/ 2023-07-29T00:24:59.299 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw= 2023-07-29T00:24:59.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw=', 'Content-Length': '0'} 2023-07-29T00:24:59.302 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eb9536b07421e44d-0064c45c5b-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.303 DEBUG:boto:path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:24:59.304 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:24:59.305 DEBUG:boto:Method: PUT 2023-07-29T00:24:59.305 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000013 2023-07-29T00:24:59.305 DEBUG:boto:Data: 2023-07-29T00:24:59.306 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:59.306 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.306 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.307 DEBUG:boto:Params: {} 2023-07-29T00:24:59.307 DEBUG:boto:Token: None 2023-07-29T00:24:59.307 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/obj00000000000000000013 2023-07-29T00:24:59.308 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+RhEAldQnRrIWl0SK/6oLwUm5O4= 2023-07-29T00:24:59.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+RhEAldQnRrIWl0SK/6oLwUm5O4='} 2023-07-29T00:24:59.356 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eee937a666e810d3-0064c45c5b-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.357 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:59.358 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:59.358 DEBUG:boto:Method: HEAD 2023-07-29T00:24:59.358 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:59.359 DEBUG:boto:Data: 2023-07-29T00:24:59.359 DEBUG:boto:Headers: {} 2023-07-29T00:24:59.359 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.360 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.361 DEBUG:boto:Params: {} 2023-07-29T00:24:59.362 DEBUG:boto:Token: None 2023-07-29T00:24:59.363 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/ 2023-07-29T00:24:59.364 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw= 2023-07-29T00:24:59.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw=', 'Content-Length': '0'} 2023-07-29T00:24:59.369 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000009bb6581351ca5eb-0064c45c5b-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.370 DEBUG:boto:path=/hiiqos-58/obj00000000000000000040 2023-07-29T00:24:59.370 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000040 2023-07-29T00:24:59.371 DEBUG:boto:Method: PUT 2023-07-29T00:24:59.371 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000040 2023-07-29T00:24:59.372 DEBUG:boto:Data: 2023-07-29T00:24:59.372 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:59.372 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.373 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.373 DEBUG:boto:Params: {} 2023-07-29T00:24:59.373 DEBUG:boto:Token: None 2023-07-29T00:24:59.374 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/obj00000000000000000040 2023-07-29T00:24:59.375 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sltITQMVYFIhxbPHMeSUZh9471c= 2023-07-29T00:24:59.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sltITQMVYFIhxbPHMeSUZh9471c='} 2023-07-29T00:24:59.382 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000876d27b880e33b49-0064c45c5b-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.382 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:59.383 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:59.383 DEBUG:boto:Method: HEAD 2023-07-29T00:24:59.384 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:59.384 DEBUG:boto:Data: 2023-07-29T00:24:59.384 DEBUG:boto:Headers: {} 2023-07-29T00:24:59.385 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.385 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.386 DEBUG:boto:Params: {} 2023-07-29T00:24:59.386 DEBUG:boto:Token: None 2023-07-29T00:24:59.386 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/ 2023-07-29T00:24:59.387 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw= 2023-07-29T00:24:59.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw=', 'Content-Length': '0'} 2023-07-29T00:24:59.392 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006fce3929d8debf11-0064c45c5b-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.394 DEBUG:boto:path=/hiiqos-58/obj00000000000000000015 2023-07-29T00:24:59.395 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000015 2023-07-29T00:24:59.396 DEBUG:boto:Method: PUT 2023-07-29T00:24:59.396 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000015 2023-07-29T00:24:59.397 DEBUG:boto:Data: 2023-07-29T00:24:59.398 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:59.398 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.400 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.400 DEBUG:boto:Params: {} 2023-07-29T00:24:59.401 DEBUG:boto:Token: None 2023-07-29T00:24:59.401 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/obj00000000000000000015 2023-07-29T00:24:59.402 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YvkX4+AB+yVISFNtvYJlTCRsYtk= 2023-07-29T00:24:59.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YvkX4+AB+yVISFNtvYJlTCRsYtk='} 2023-07-29T00:24:59.448 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fd2a1ca53ad8ce28-0064c45c5b-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.449 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:59.449 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:59.450 DEBUG:boto:Method: HEAD 2023-07-29T00:24:59.450 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:59.451 DEBUG:boto:Data: 2023-07-29T00:24:59.451 DEBUG:boto:Headers: {} 2023-07-29T00:24:59.451 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.451 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.452 DEBUG:boto:Params: {} 2023-07-29T00:24:59.453 DEBUG:boto:Token: None 2023-07-29T00:24:59.453 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/ 2023-07-29T00:24:59.454 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw= 2023-07-29T00:24:59.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw=', 'Content-Length': '0'} 2023-07-29T00:24:59.457 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000301765e0dfe64aaa-0064c45c5b-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.458 DEBUG:boto:path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:24:59.458 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:24:59.459 DEBUG:boto:Method: PUT 2023-07-29T00:24:59.459 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000005 2023-07-29T00:24:59.459 DEBUG:boto:Data: 2023-07-29T00:24:59.460 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:59.460 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.461 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.461 DEBUG:boto:Params: {} 2023-07-29T00:24:59.461 DEBUG:boto:Token: None 2023-07-29T00:24:59.462 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/obj00000000000000000005 2023-07-29T00:24:59.462 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zhjU0unXrFThRtHSl1tufAsfIgA= 2023-07-29T00:24:59.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zhjU0unXrFThRtHSl1tufAsfIgA='} 2023-07-29T00:24:59.469 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000434daaf232ee8b95-0064c45c5b-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.470 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:59.470 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:59.470 DEBUG:boto:Method: HEAD 2023-07-29T00:24:59.471 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:59.471 DEBUG:boto:Data: 2023-07-29T00:24:59.472 DEBUG:boto:Headers: {} 2023-07-29T00:24:59.472 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.473 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.473 DEBUG:boto:Params: {} 2023-07-29T00:24:59.474 DEBUG:boto:Token: None 2023-07-29T00:24:59.474 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/ 2023-07-29T00:24:59.474 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw= 2023-07-29T00:24:59.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw=', 'Content-Length': '0'} 2023-07-29T00:24:59.479 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c47ee3ed3b53838c-0064c45c5b-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.479 DEBUG:boto:path=/hiiqos-58/obj00000000000000000024 2023-07-29T00:24:59.480 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000024 2023-07-29T00:24:59.480 DEBUG:boto:Method: PUT 2023-07-29T00:24:59.481 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000024 2023-07-29T00:24:59.481 DEBUG:boto:Data: 2023-07-29T00:24:59.482 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:59.482 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.482 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.483 DEBUG:boto:Params: {} 2023-07-29T00:24:59.483 DEBUG:boto:Token: None 2023-07-29T00:24:59.483 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/obj00000000000000000024 2023-07-29T00:24:59.484 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JuUMkOxvhM1e33GoxC1RXKHSdro= 2023-07-29T00:24:59.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JuUMkOxvhM1e33GoxC1RXKHSdro='} 2023-07-29T00:24:59.532 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000d3c56d4b3e02626-0064c45c5b-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.534 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:59.535 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:59.536 DEBUG:boto:Method: HEAD 2023-07-29T00:24:59.536 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:59.537 DEBUG:boto:Data: 2023-07-29T00:24:59.538 DEBUG:boto:Headers: {} 2023-07-29T00:24:59.538 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.538 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.539 DEBUG:boto:Params: {} 2023-07-29T00:24:59.539 DEBUG:boto:Token: None 2023-07-29T00:24:59.539 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/ 2023-07-29T00:24:59.540 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw= 2023-07-29T00:24:59.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw=', 'Content-Length': '0'} 2023-07-29T00:24:59.544 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008cc43c471b941d74-0064c45c5b-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.544 DEBUG:boto:path=/hiiqos-58/obj00000000000000000009 2023-07-29T00:24:59.545 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000009 2023-07-29T00:24:59.545 DEBUG:boto:Method: PUT 2023-07-29T00:24:59.546 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000009 2023-07-29T00:24:59.546 DEBUG:boto:Data: 2023-07-29T00:24:59.547 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:59.547 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.547 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.548 DEBUG:boto:Params: {} 2023-07-29T00:24:59.548 DEBUG:boto:Token: None 2023-07-29T00:24:59.549 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/obj00000000000000000009 2023-07-29T00:24:59.549 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ymROuow76w+F0nN4HL3SnjUtCwk= 2023-07-29T00:24:59.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ymROuow76w+F0nN4HL3SnjUtCwk='} 2023-07-29T00:24:59.555 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b6544e21d85cc40a-0064c45c5b-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.556 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:59.556 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:59.557 DEBUG:boto:Method: HEAD 2023-07-29T00:24:59.557 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:59.557 DEBUG:boto:Data: 2023-07-29T00:24:59.558 DEBUG:boto:Headers: {} 2023-07-29T00:24:59.558 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.558 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.559 DEBUG:boto:Params: {} 2023-07-29T00:24:59.559 DEBUG:boto:Token: None 2023-07-29T00:24:59.559 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/ 2023-07-29T00:24:59.560 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw= 2023-07-29T00:24:59.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw=', 'Content-Length': '0'} 2023-07-29T00:24:59.563 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dcd8d75c4ce60942-0064c45c5b-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.564 DEBUG:boto:path=/hiiqos-58/obj00000000000000000060 2023-07-29T00:24:59.564 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000060 2023-07-29T00:24:59.565 DEBUG:boto:Method: PUT 2023-07-29T00:24:59.566 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000060 2023-07-29T00:24:59.566 DEBUG:boto:Data: 2023-07-29T00:24:59.566 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:59.567 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.567 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.567 DEBUG:boto:Params: {} 2023-07-29T00:24:59.568 DEBUG:boto:Token: None 2023-07-29T00:24:59.568 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/obj00000000000000000060 2023-07-29T00:24:59.569 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lv+sjL0Qa3NM5ZDV/JaOBZqGZRY= 2023-07-29T00:24:59.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lv+sjL0Qa3NM5ZDV/JaOBZqGZRY='} 2023-07-29T00:24:59.574 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008fd3a4c1607f0381-0064c45c5b-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.574 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:59.575 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:59.575 DEBUG:boto:Method: HEAD 2023-07-29T00:24:59.575 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:59.576 DEBUG:boto:Data: 2023-07-29T00:24:59.576 DEBUG:boto:Headers: {} 2023-07-29T00:24:59.576 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.577 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.577 DEBUG:boto:Params: {} 2023-07-29T00:24:59.577 DEBUG:boto:Token: None 2023-07-29T00:24:59.578 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/ 2023-07-29T00:24:59.578 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw= 2023-07-29T00:24:59.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw=', 'Content-Length': '0'} 2023-07-29T00:24:59.582 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fde8b27df89de900-0064c45c5b-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.583 DEBUG:boto:path=/hiiqos-58/obj00000000000000000020 2023-07-29T00:24:59.583 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000020 2023-07-29T00:24:59.584 DEBUG:boto:Method: PUT 2023-07-29T00:24:59.584 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000020 2023-07-29T00:24:59.584 DEBUG:boto:Data: 2023-07-29T00:24:59.585 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:59.585 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.585 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.586 DEBUG:boto:Params: {} 2023-07-29T00:24:59.586 DEBUG:boto:Token: None 2023-07-29T00:24:59.586 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/obj00000000000000000020 2023-07-29T00:24:59.587 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pH2ugujf4/J3d0vbTO0FLlpCs10= 2023-07-29T00:24:59.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pH2ugujf4/J3d0vbTO0FLlpCs10='} 2023-07-29T00:24:59.636 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004de36cbd6463bdc4-0064c45c5b-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.637 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:59.637 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:59.638 DEBUG:boto:Method: HEAD 2023-07-29T00:24:59.641 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:59.641 DEBUG:boto:Data: 2023-07-29T00:24:59.641 DEBUG:boto:Headers: {} 2023-07-29T00:24:59.642 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.642 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.642 DEBUG:boto:Params: {} 2023-07-29T00:24:59.643 DEBUG:boto:Token: None 2023-07-29T00:24:59.643 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/ 2023-07-29T00:24:59.644 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw= 2023-07-29T00:24:59.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw=', 'Content-Length': '0'} 2023-07-29T00:24:59.648 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aee11649bb7ace50-0064c45c5b-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.648 DEBUG:boto:path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:24:59.649 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:24:59.649 DEBUG:boto:Method: PUT 2023-07-29T00:24:59.650 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000012 2023-07-29T00:24:59.650 DEBUG:boto:Data: 2023-07-29T00:24:59.650 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:59.651 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.651 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.651 DEBUG:boto:Params: {} 2023-07-29T00:24:59.652 DEBUG:boto:Token: None 2023-07-29T00:24:59.652 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/obj00000000000000000012 2023-07-29T00:24:59.653 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:NXs+YGyjDud4bKnqdEMpgUY4tCs= 2023-07-29T00:24:59.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:NXs+YGyjDud4bKnqdEMpgUY4tCs='} 2023-07-29T00:24:59.659 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004fd100cd05d47ab9-0064c45c5b-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.660 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:59.660 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:59.661 DEBUG:boto:Method: HEAD 2023-07-29T00:24:59.661 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:59.661 DEBUG:boto:Data: 2023-07-29T00:24:59.662 DEBUG:boto:Headers: {} 2023-07-29T00:24:59.662 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.662 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.663 DEBUG:boto:Params: {} 2023-07-29T00:24:59.663 DEBUG:boto:Token: None 2023-07-29T00:24:59.663 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/ 2023-07-29T00:24:59.664 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw= 2023-07-29T00:24:59.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw=', 'Content-Length': '0'} 2023-07-29T00:24:59.668 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b8f2b91f2ae76c81-0064c45c5b-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.668 DEBUG:boto:path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:24:59.669 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:24:59.669 DEBUG:boto:Method: PUT 2023-07-29T00:24:59.670 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000006 2023-07-29T00:24:59.670 DEBUG:boto:Data: 2023-07-29T00:24:59.670 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:59.671 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.671 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.671 DEBUG:boto:Params: {} 2023-07-29T00:24:59.672 DEBUG:boto:Token: None 2023-07-29T00:24:59.672 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/obj00000000000000000006 2023-07-29T00:24:59.673 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LDG5cyBArCdtGh27R0GY4EnF2Ek= 2023-07-29T00:24:59.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LDG5cyBArCdtGh27R0GY4EnF2Ek='} 2023-07-29T00:24:59.679 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000def2fa884873ba22-0064c45c5b-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.680 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:59.680 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:59.680 DEBUG:boto:Method: HEAD 2023-07-29T00:24:59.681 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:59.681 DEBUG:boto:Data: 2023-07-29T00:24:59.681 DEBUG:boto:Headers: {} 2023-07-29T00:24:59.682 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.682 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.682 DEBUG:boto:Params: {} 2023-07-29T00:24:59.683 DEBUG:boto:Token: None 2023-07-29T00:24:59.683 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/ 2023-07-29T00:24:59.684 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw= 2023-07-29T00:24:59.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw=', 'Content-Length': '0'} 2023-07-29T00:24:59.688 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f986214f43d52190-0064c45c5b-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.689 DEBUG:boto:path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:24:59.689 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:24:59.689 DEBUG:boto:Method: PUT 2023-07-29T00:24:59.690 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000002 2023-07-29T00:24:59.690 DEBUG:boto:Data: 2023-07-29T00:24:59.690 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:59.691 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.691 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.691 DEBUG:boto:Params: {} 2023-07-29T00:24:59.692 DEBUG:boto:Token: None 2023-07-29T00:24:59.692 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/obj00000000000000000002 2023-07-29T00:24:59.693 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Q0eZMJl3Tgwy8erEMvIUu7FvjU8= 2023-07-29T00:24:59.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q0eZMJl3Tgwy8erEMvIUu7FvjU8='} 2023-07-29T00:24:59.699 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009a335fba5cd8851f-0064c45c5b-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.700 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:59.701 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:59.701 DEBUG:boto:Method: HEAD 2023-07-29T00:24:59.701 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:59.702 DEBUG:boto:Data: 2023-07-29T00:24:59.702 DEBUG:boto:Headers: {} 2023-07-29T00:24:59.702 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.703 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.703 DEBUG:boto:Params: {} 2023-07-29T00:24:59.703 DEBUG:boto:Token: None 2023-07-29T00:24:59.704 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/ 2023-07-29T00:24:59.705 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw= 2023-07-29T00:24:59.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw=', 'Content-Length': '0'} 2023-07-29T00:24:59.709 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000180b3b09b2758934-0064c45c5b-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.710 DEBUG:boto:path=/hiiqos-58/obj00000000000000000010 2023-07-29T00:24:59.710 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000010 2023-07-29T00:24:59.711 DEBUG:boto:Method: PUT 2023-07-29T00:24:59.711 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000010 2023-07-29T00:24:59.711 DEBUG:boto:Data: 2023-07-29T00:24:59.712 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:59.712 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.713 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.713 DEBUG:boto:Params: {} 2023-07-29T00:24:59.713 DEBUG:boto:Token: None 2023-07-29T00:24:59.714 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/obj00000000000000000010 2023-07-29T00:24:59.714 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2og5Ivk8LE/nKM+GKd8mppCdL80= 2023-07-29T00:24:59.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2og5Ivk8LE/nKM+GKd8mppCdL80='} 2023-07-29T00:24:59.760 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001ad8d04b35433583-0064c45c5b-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.761 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:59.762 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:59.762 DEBUG:boto:Method: HEAD 2023-07-29T00:24:59.763 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:59.763 DEBUG:boto:Data: 2023-07-29T00:24:59.763 DEBUG:boto:Headers: {} 2023-07-29T00:24:59.764 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.764 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.764 DEBUG:boto:Params: {} 2023-07-29T00:24:59.765 DEBUG:boto:Token: None 2023-07-29T00:24:59.765 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/ 2023-07-29T00:24:59.766 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw= 2023-07-29T00:24:59.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw=', 'Content-Length': '0'} 2023-07-29T00:24:59.769 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d6fcfc131a220380-0064c45c5b-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.770 DEBUG:boto:path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:24:59.770 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:24:59.771 DEBUG:boto:Method: PUT 2023-07-29T00:24:59.771 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000000 2023-07-29T00:24:59.771 DEBUG:boto:Data: 2023-07-29T00:24:59.772 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:59.772 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.773 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.773 DEBUG:boto:Params: {} 2023-07-29T00:24:59.773 DEBUG:boto:Token: None 2023-07-29T00:24:59.774 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/obj00000000000000000000 2023-07-29T00:24:59.774 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6FIT9cgMGyFwVQNtz/pl6Zq3U7k= 2023-07-29T00:24:59.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6FIT9cgMGyFwVQNtz/pl6Zq3U7k='} 2023-07-29T00:24:59.781 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004876ef3ececf747d-0064c45c5b-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.781 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:59.781 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:59.782 DEBUG:boto:Method: HEAD 2023-07-29T00:24:59.782 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:59.783 DEBUG:boto:Data: 2023-07-29T00:24:59.783 DEBUG:boto:Headers: {} 2023-07-29T00:24:59.783 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.784 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.785 DEBUG:boto:Params: {} 2023-07-29T00:24:59.785 DEBUG:boto:Token: None 2023-07-29T00:24:59.786 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/ 2023-07-29T00:24:59.787 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw= 2023-07-29T00:24:59.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw=', 'Content-Length': '0'} 2023-07-29T00:24:59.790 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f1dead42dd0b2899-0064c45c5b-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.791 DEBUG:boto:path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:24:59.792 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:24:59.793 DEBUG:boto:Method: PUT 2023-07-29T00:24:59.793 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000001 2023-07-29T00:24:59.794 DEBUG:boto:Data: 2023-07-29T00:24:59.794 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:59.795 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.795 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.796 DEBUG:boto:Params: {} 2023-07-29T00:24:59.797 DEBUG:boto:Token: None 2023-07-29T00:24:59.797 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/obj00000000000000000001 2023-07-29T00:24:59.798 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:iUOVofYthggm9sjz5eYWEsXAj3Q= 2023-07-29T00:24:59.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:iUOVofYthggm9sjz5eYWEsXAj3Q='} 2023-07-29T00:24:59.840 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005d3fdd8ea2d79e95-0064c45c5b-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.841 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:59.841 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:59.842 DEBUG:boto:Method: HEAD 2023-07-29T00:24:59.842 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:59.843 DEBUG:boto:Data: 2023-07-29T00:24:59.843 DEBUG:boto:Headers: {} 2023-07-29T00:24:59.843 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.844 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.844 DEBUG:boto:Params: {} 2023-07-29T00:24:59.845 DEBUG:boto:Token: None 2023-07-29T00:24:59.845 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/ 2023-07-29T00:24:59.845 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw= 2023-07-29T00:24:59.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw=', 'Content-Length': '0'} 2023-07-29T00:24:59.849 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003fc95441f650e5cd-0064c45c5b-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.850 DEBUG:boto:path=/hiiqos-58/obj00000000000000000065 2023-07-29T00:24:59.850 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000065 2023-07-29T00:24:59.851 DEBUG:boto:Method: PUT 2023-07-29T00:24:59.851 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000065 2023-07-29T00:24:59.851 DEBUG:boto:Data: 2023-07-29T00:24:59.852 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:59.852 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.852 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.853 DEBUG:boto:Params: {} 2023-07-29T00:24:59.853 DEBUG:boto:Token: None 2023-07-29T00:24:59.853 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/obj00000000000000000065 2023-07-29T00:24:59.854 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IyNMZ+5f+cGO80rfd0QXO4kq5D0= 2023-07-29T00:24:59.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IyNMZ+5f+cGO80rfd0QXO4kq5D0='} 2023-07-29T00:24:59.859 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e9268cf89f4baa9e-0064c45c5b-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.860 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:59.860 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:59.861 DEBUG:boto:Method: HEAD 2023-07-29T00:24:59.861 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:59.861 DEBUG:boto:Data: 2023-07-29T00:24:59.862 DEBUG:boto:Headers: {} 2023-07-29T00:24:59.862 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.863 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.863 DEBUG:boto:Params: {} 2023-07-29T00:24:59.863 DEBUG:boto:Token: None 2023-07-29T00:24:59.864 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/ 2023-07-29T00:24:59.864 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw= 2023-07-29T00:24:59.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw=', 'Content-Length': '0'} 2023-07-29T00:24:59.868 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004272233c1d072e18-0064c45c5b-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.869 DEBUG:boto:path=/hiiqos-58/obj00000000000000000021 2023-07-29T00:24:59.869 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000021 2023-07-29T00:24:59.869 DEBUG:boto:Method: PUT 2023-07-29T00:24:59.870 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000021 2023-07-29T00:24:59.870 DEBUG:boto:Data: 2023-07-29T00:24:59.871 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:59.871 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.871 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.872 DEBUG:boto:Params: {} 2023-07-29T00:24:59.872 DEBUG:boto:Token: None 2023-07-29T00:24:59.872 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/obj00000000000000000021 2023-07-29T00:24:59.873 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Cu97/LCOz1GlideBdrhx+jCX+As= 2023-07-29T00:24:59.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Cu97/LCOz1GlideBdrhx+jCX+As='} 2023-07-29T00:24:59.920 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000edc3a986da52dc0b-0064c45c5b-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.921 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:59.922 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:59.922 DEBUG:boto:Method: HEAD 2023-07-29T00:24:59.923 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:59.923 DEBUG:boto:Data: 2023-07-29T00:24:59.924 DEBUG:boto:Headers: {} 2023-07-29T00:24:59.924 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.924 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.925 DEBUG:boto:Params: {} 2023-07-29T00:24:59.925 DEBUG:boto:Token: None 2023-07-29T00:24:59.926 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/ 2023-07-29T00:24:59.926 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw= 2023-07-29T00:24:59.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw=', 'Content-Length': '0'} 2023-07-29T00:24:59.930 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009d41bc3b98558a31-0064c45c5b-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.930 DEBUG:boto:path=/hiiqos-58/obj00000000000000000007 2023-07-29T00:24:59.931 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000007 2023-07-29T00:24:59.931 DEBUG:boto:Method: PUT 2023-07-29T00:24:59.932 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000007 2023-07-29T00:24:59.932 DEBUG:boto:Data: 2023-07-29T00:24:59.932 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:59.933 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.933 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.933 DEBUG:boto:Params: {} 2023-07-29T00:24:59.934 DEBUG:boto:Token: None 2023-07-29T00:24:59.934 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/obj00000000000000000007 2023-07-29T00:24:59.935 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FaQJn5zgOAEOWzmLZKRIZrGSSnk= 2023-07-29T00:24:59.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FaQJn5zgOAEOWzmLZKRIZrGSSnk='} 2023-07-29T00:24:59.943 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e4dfcc962e65a242-0064c45c5b-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.944 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:24:59.944 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:24:59.944 DEBUG:boto:Method: HEAD 2023-07-29T00:24:59.945 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:24:59.945 DEBUG:boto:Data: 2023-07-29T00:24:59.945 DEBUG:boto:Headers: {} 2023-07-29T00:24:59.946 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.946 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.946 DEBUG:boto:Params: {} 2023-07-29T00:24:59.947 DEBUG:boto:Token: None 2023-07-29T00:24:59.947 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/ 2023-07-29T00:24:59.949 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw= 2023-07-29T00:24:59.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rIN3UeW796zJN7RmqIQHpwRW/pw=', 'Content-Length': '0'} 2023-07-29T00:24:59.953 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f3c7d70d3ad3de98-0064c45c5b-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:24:59.953 DEBUG:boto:path=/hiiqos-58/obj00000000000000000027 2023-07-29T00:24:59.954 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000027 2023-07-29T00:24:59.954 DEBUG:boto:Method: PUT 2023-07-29T00:24:59.955 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000027 2023-07-29T00:24:59.955 DEBUG:boto:Data: 2023-07-29T00:24:59.955 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:24:59.956 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:24:59.956 DEBUG:boto:Port: 8000 2023-07-29T00:24:59.956 DEBUG:boto:Params: {} 2023-07-29T00:24:59.957 DEBUG:boto:Token: None 2023-07-29T00:24:59.957 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:24:59 GMT /hiiqos-58/obj00000000000000000027 2023-07-29T00:24:59.958 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w16O39lBjLQSx7YVsxr/PkNquVQ= 2023-07-29T00:24:59.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:24:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w16O39lBjLQSx7YVsxr/PkNquVQ='} 2023-07-29T00:25:00.004 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e1be7b0eba0bb761-0064c45c5b-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:00.005 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:00.006 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:00.006 DEBUG:boto:Method: HEAD 2023-07-29T00:25:00.007 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:25:00.007 DEBUG:boto:Data: 2023-07-29T00:25:00.007 DEBUG:boto:Headers: {} 2023-07-29T00:25:00.008 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:00.008 DEBUG:boto:Port: 8000 2023-07-29T00:25:00.008 DEBUG:boto:Params: {} 2023-07-29T00:25:00.009 DEBUG:boto:Token: None 2023-07-29T00:25:00.009 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:00 GMT /hiiqos-58/ 2023-07-29T00:25:00.010 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w/8cldGnsO241D20CiUq6gwsmRk= 2023-07-29T00:25:00.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w/8cldGnsO241D20CiUq6gwsmRk=', 'Content-Length': '0'} 2023-07-29T00:25:00.013 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001e198df8470993db-0064c45c5b-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:00.014 DEBUG:boto:path=/hiiqos-58/obj00000000000000000019 2023-07-29T00:25:00.014 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000019 2023-07-29T00:25:00.015 DEBUG:boto:Method: PUT 2023-07-29T00:25:00.015 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000019 2023-07-29T00:25:00.015 DEBUG:boto:Data: 2023-07-29T00:25:00.016 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:25:00.016 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:00.017 DEBUG:boto:Port: 8000 2023-07-29T00:25:00.017 DEBUG:boto:Params: {} 2023-07-29T00:25:00.017 DEBUG:boto:Token: None 2023-07-29T00:25:00.018 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:25:00 GMT /hiiqos-58/obj00000000000000000019 2023-07-29T00:25:00.018 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:motjOgmg8t9MHPlvbZ37J6/shWI= 2023-07-29T00:25:00.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:25:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:motjOgmg8t9MHPlvbZ37J6/shWI='} 2023-07-29T00:25:00.025 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000248dc3f59c41ee41-0064c45c5b-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:00.026 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:00.026 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:00.026 DEBUG:boto:Method: HEAD 2023-07-29T00:25:00.027 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:25:00.027 DEBUG:boto:Data: 2023-07-29T00:25:00.027 DEBUG:boto:Headers: {} 2023-07-29T00:25:00.028 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:00.028 DEBUG:boto:Port: 8000 2023-07-29T00:25:00.029 DEBUG:boto:Params: {} 2023-07-29T00:25:00.029 DEBUG:boto:Token: None 2023-07-29T00:25:00.029 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:00 GMT /hiiqos-58/ 2023-07-29T00:25:00.030 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w/8cldGnsO241D20CiUq6gwsmRk= 2023-07-29T00:25:00.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w/8cldGnsO241D20CiUq6gwsmRk=', 'Content-Length': '0'} 2023-07-29T00:25:00.033 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004b2e475ee9e7922c-0064c45c5b-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:00.034 DEBUG:boto:path=/hiiqos-58/obj00000000000000000014 2023-07-29T00:25:00.034 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000014 2023-07-29T00:25:00.035 DEBUG:boto:Method: PUT 2023-07-29T00:25:00.035 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000014 2023-07-29T00:25:00.035 DEBUG:boto:Data: 2023-07-29T00:25:00.036 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:25:00.036 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:00.037 DEBUG:boto:Port: 8000 2023-07-29T00:25:00.038 DEBUG:boto:Params: {} 2023-07-29T00:25:00.038 DEBUG:boto:Token: None 2023-07-29T00:25:00.038 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:25:00 GMT /hiiqos-58/obj00000000000000000014 2023-07-29T00:25:00.039 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lDoQ25CM8lS9CJidZKoZsgSUY2I= 2023-07-29T00:25:00.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:25:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lDoQ25CM8lS9CJidZKoZsgSUY2I='} 2023-07-29T00:25:00.084 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eccafd0e522f56b7-0064c45c5b-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:00.086 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:00.087 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:00.087 DEBUG:boto:Method: HEAD 2023-07-29T00:25:00.088 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:25:00.088 DEBUG:boto:Data: 2023-07-29T00:25:00.088 DEBUG:boto:Headers: {} 2023-07-29T00:25:00.089 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:00.090 DEBUG:boto:Port: 8000 2023-07-29T00:25:00.090 DEBUG:boto:Params: {} 2023-07-29T00:25:00.090 DEBUG:boto:Token: None 2023-07-29T00:25:00.091 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:00 GMT /hiiqos-58/ 2023-07-29T00:25:00.091 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w/8cldGnsO241D20CiUq6gwsmRk= 2023-07-29T00:25:00.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w/8cldGnsO241D20CiUq6gwsmRk=', 'Content-Length': '0'} 2023-07-29T00:25:00.095 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fd95c5dcf18f1f21-0064c45c5c-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:00.096 DEBUG:boto:path=/hiiqos-58/obj00000000000000000033 2023-07-29T00:25:00.096 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000033 2023-07-29T00:25:00.097 DEBUG:boto:Method: PUT 2023-07-29T00:25:00.097 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000033 2023-07-29T00:25:00.097 DEBUG:boto:Data: 2023-07-29T00:25:00.098 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:25:00.098 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:00.098 DEBUG:boto:Port: 8000 2023-07-29T00:25:00.099 DEBUG:boto:Params: {} 2023-07-29T00:25:00.099 DEBUG:boto:Token: None 2023-07-29T00:25:00.100 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:25:00 GMT /hiiqos-58/obj00000000000000000033 2023-07-29T00:25:00.100 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:775Se0k10f/VL2LO8Vz+73TYJco= 2023-07-29T00:25:00.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:25:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:775Se0k10f/VL2LO8Vz+73TYJco='} 2023-07-29T00:25:00.106 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000afda83f026b0b4f1-0064c45c5c-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:00.107 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:00.107 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:00.107 DEBUG:boto:Method: HEAD 2023-07-29T00:25:00.108 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:25:00.108 DEBUG:boto:Data: 2023-07-29T00:25:00.108 DEBUG:boto:Headers: {} 2023-07-29T00:25:00.109 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:00.109 DEBUG:boto:Port: 8000 2023-07-29T00:25:00.109 DEBUG:boto:Params: {} 2023-07-29T00:25:00.110 DEBUG:boto:Token: None 2023-07-29T00:25:00.110 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:00 GMT /hiiqos-58/ 2023-07-29T00:25:00.110 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w/8cldGnsO241D20CiUq6gwsmRk= 2023-07-29T00:25:00.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w/8cldGnsO241D20CiUq6gwsmRk=', 'Content-Length': '0'} 2023-07-29T00:25:00.114 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a7b33b8631477ff0-0064c45c5c-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:00.115 DEBUG:boto:path=/hiiqos-58/obj00000000000000000045 2023-07-29T00:25:00.115 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000045 2023-07-29T00:25:00.116 DEBUG:boto:Method: PUT 2023-07-29T00:25:00.116 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000045 2023-07-29T00:25:00.116 DEBUG:boto:Data: 2023-07-29T00:25:00.117 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:25:00.117 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:00.117 DEBUG:boto:Port: 8000 2023-07-29T00:25:00.118 DEBUG:boto:Params: {} 2023-07-29T00:25:00.118 DEBUG:boto:Token: None 2023-07-29T00:25:00.118 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:25:00 GMT /hiiqos-58/obj00000000000000000045 2023-07-29T00:25:00.119 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ejOSNECs7Oac3hccMYe1y118p8o= 2023-07-29T00:25:00.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:25:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ejOSNECs7Oac3hccMYe1y118p8o='} 2023-07-29T00:25:00.164 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000def4d14a39e11438-0064c45c5c-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:00.165 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:00.166 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:00.166 DEBUG:boto:Method: HEAD 2023-07-29T00:25:00.167 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:25:00.167 DEBUG:boto:Data: 2023-07-29T00:25:00.167 DEBUG:boto:Headers: {} 2023-07-29T00:25:00.168 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:00.168 DEBUG:boto:Port: 8000 2023-07-29T00:25:00.169 DEBUG:boto:Params: {} 2023-07-29T00:25:00.169 DEBUG:boto:Token: None 2023-07-29T00:25:00.169 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:00 GMT /hiiqos-58/ 2023-07-29T00:25:00.170 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w/8cldGnsO241D20CiUq6gwsmRk= 2023-07-29T00:25:00.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w/8cldGnsO241D20CiUq6gwsmRk=', 'Content-Length': '0'} 2023-07-29T00:25:00.174 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b360c44e5e6faf60-0064c45c5c-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:00.175 DEBUG:boto:path=/hiiqos-58/obj00000000000000000017 2023-07-29T00:25:00.175 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000017 2023-07-29T00:25:00.175 DEBUG:boto:Method: PUT 2023-07-29T00:25:00.176 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000017 2023-07-29T00:25:00.176 DEBUG:boto:Data: 2023-07-29T00:25:00.176 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:25:00.177 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:00.177 DEBUG:boto:Port: 8000 2023-07-29T00:25:00.177 DEBUG:boto:Params: {} 2023-07-29T00:25:00.178 DEBUG:boto:Token: None 2023-07-29T00:25:00.178 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:25:00 GMT /hiiqos-58/obj00000000000000000017 2023-07-29T00:25:00.179 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hjkVYM4e+dSoshtwns8nnsTXcWE= 2023-07-29T00:25:00.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:25:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hjkVYM4e+dSoshtwns8nnsTXcWE='} 2023-07-29T00:25:00.185 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000804a0870f7ba5126-0064c45c5c-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:00.186 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:00.186 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:00.187 DEBUG:boto:Method: HEAD 2023-07-29T00:25:00.187 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:25:00.187 DEBUG:boto:Data: 2023-07-29T00:25:00.188 DEBUG:boto:Headers: {} 2023-07-29T00:25:00.188 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:00.188 DEBUG:boto:Port: 8000 2023-07-29T00:25:00.189 DEBUG:boto:Params: {} 2023-07-29T00:25:00.189 DEBUG:boto:Token: None 2023-07-29T00:25:00.189 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:00 GMT /hiiqos-58/ 2023-07-29T00:25:00.190 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w/8cldGnsO241D20CiUq6gwsmRk= 2023-07-29T00:25:00.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w/8cldGnsO241D20CiUq6gwsmRk=', 'Content-Length': '0'} 2023-07-29T00:25:00.194 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000062135fddc3a9bb5e-0064c45c5c-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:00.194 DEBUG:boto:path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:25:00.195 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:25:00.195 DEBUG:boto:Method: PUT 2023-07-29T00:25:00.196 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000003 2023-07-29T00:25:00.196 DEBUG:boto:Data: 2023-07-29T00:25:00.196 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:25:00.197 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:00.197 DEBUG:boto:Port: 8000 2023-07-29T00:25:00.197 DEBUG:boto:Params: {} 2023-07-29T00:25:00.198 DEBUG:boto:Token: None 2023-07-29T00:25:00.198 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:25:00 GMT /hiiqos-58/obj00000000000000000003 2023-07-29T00:25:00.198 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:d1PZKo8bJ+r1EJ+rQJoZDqdnIxw= 2023-07-29T00:25:00.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:25:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:d1PZKo8bJ+r1EJ+rQJoZDqdnIxw='} 2023-07-29T00:25:00.241 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a4adce7e0aeae087-0064c45c5c-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:00.241 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:00.242 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:00.242 DEBUG:boto:Method: HEAD 2023-07-29T00:25:00.242 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:25:00.243 DEBUG:boto:Data: 2023-07-29T00:25:00.243 DEBUG:boto:Headers: {} 2023-07-29T00:25:00.244 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:00.244 DEBUG:boto:Port: 8000 2023-07-29T00:25:00.245 DEBUG:boto:Params: {} 2023-07-29T00:25:00.245 DEBUG:boto:Token: None 2023-07-29T00:25:00.246 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:00 GMT /hiiqos-58/ 2023-07-29T00:25:00.246 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w/8cldGnsO241D20CiUq6gwsmRk= 2023-07-29T00:25:00.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w/8cldGnsO241D20CiUq6gwsmRk=', 'Content-Length': '0'} 2023-07-29T00:25:00.250 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000da15d2bfb52a9bb0-0064c45c5c-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:00.251 DEBUG:boto:path=/hiiqos-58/obj00000000000000000057 2023-07-29T00:25:00.251 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000057 2023-07-29T00:25:00.252 DEBUG:boto:Method: PUT 2023-07-29T00:25:00.252 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000057 2023-07-29T00:25:00.253 DEBUG:boto:Data: 2023-07-29T00:25:00.253 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:25:00.253 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:00.254 DEBUG:boto:Port: 8000 2023-07-29T00:25:00.254 DEBUG:boto:Params: {} 2023-07-29T00:25:00.255 DEBUG:boto:Token: None 2023-07-29T00:25:00.255 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:25:00 GMT /hiiqos-58/obj00000000000000000057 2023-07-29T00:25:00.256 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PEaZjupX3t6L0h0YlIAGvDasin4= 2023-07-29T00:25:00.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:25:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PEaZjupX3t6L0h0YlIAGvDasin4='} 2023-07-29T00:25:00.261 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000862b87d88f1b633f-0064c45c5c-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:00.261 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:00.262 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:00.262 DEBUG:boto:Method: HEAD 2023-07-29T00:25:00.263 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:25:00.263 DEBUG:boto:Data: 2023-07-29T00:25:00.263 DEBUG:boto:Headers: {} 2023-07-29T00:25:00.264 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:00.264 DEBUG:boto:Port: 8000 2023-07-29T00:25:00.264 DEBUG:boto:Params: {} 2023-07-29T00:25:00.265 DEBUG:boto:Token: None 2023-07-29T00:25:00.265 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:00 GMT /hiiqos-58/ 2023-07-29T00:25:00.266 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w/8cldGnsO241D20CiUq6gwsmRk= 2023-07-29T00:25:00.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w/8cldGnsO241D20CiUq6gwsmRk=', 'Content-Length': '0'} 2023-07-29T00:25:00.269 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000923f16f50efe635f-0064c45c5c-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:00.270 DEBUG:boto:path=/hiiqos-58/obj00000000000000000008 2023-07-29T00:25:00.270 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000008 2023-07-29T00:25:00.271 DEBUG:boto:Method: PUT 2023-07-29T00:25:00.271 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000008 2023-07-29T00:25:00.271 DEBUG:boto:Data: 2023-07-29T00:25:00.272 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:25:00.272 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:00.273 DEBUG:boto:Port: 8000 2023-07-29T00:25:00.273 DEBUG:boto:Params: {} 2023-07-29T00:25:00.273 DEBUG:boto:Token: None 2023-07-29T00:25:00.274 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:25:00 GMT /hiiqos-58/obj00000000000000000008 2023-07-29T00:25:00.274 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fQH7vo+z762AHCFUa7oQqBoF7Jg= 2023-07-29T00:25:00.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:25:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fQH7vo+z762AHCFUa7oQqBoF7Jg='} 2023-07-29T00:25:00.316 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000b6c378fadb6cb58-0064c45c5c-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:00.317 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:00.318 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:00.318 DEBUG:boto:Method: HEAD 2023-07-29T00:25:00.318 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:25:00.319 DEBUG:boto:Data: 2023-07-29T00:25:00.319 DEBUG:boto:Headers: {} 2023-07-29T00:25:00.319 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:00.320 DEBUG:boto:Port: 8000 2023-07-29T00:25:00.320 DEBUG:boto:Params: {} 2023-07-29T00:25:00.320 DEBUG:boto:Token: None 2023-07-29T00:25:00.321 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:00 GMT /hiiqos-58/ 2023-07-29T00:25:00.321 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w/8cldGnsO241D20CiUq6gwsmRk= 2023-07-29T00:25:00.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w/8cldGnsO241D20CiUq6gwsmRk=', 'Content-Length': '0'} 2023-07-29T00:25:00.325 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f974edacf18a76c-0064c45c5c-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:00.325 DEBUG:boto:path=/hiiqos-58/obj00000000000000000062 2023-07-29T00:25:00.326 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000062 2023-07-29T00:25:00.326 DEBUG:boto:Method: PUT 2023-07-29T00:25:00.327 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000062 2023-07-29T00:25:00.327 DEBUG:boto:Data: 2023-07-29T00:25:00.327 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:25:00.328 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:00.328 DEBUG:boto:Port: 8000 2023-07-29T00:25:00.328 DEBUG:boto:Params: {} 2023-07-29T00:25:00.329 DEBUG:boto:Token: None 2023-07-29T00:25:00.329 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:25:00 GMT /hiiqos-58/obj00000000000000000062 2023-07-29T00:25:00.330 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3m8kFjh2u28GiQ+JcG6b9kwW0RY= 2023-07-29T00:25:00.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:25:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3m8kFjh2u28GiQ+JcG6b9kwW0RY='} 2023-07-29T00:25:00.335 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d0396eca944f0c1b-0064c45c5c-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:00.336 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:00.336 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:00.337 DEBUG:boto:Method: HEAD 2023-07-29T00:25:00.337 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:25:00.337 DEBUG:boto:Data: 2023-07-29T00:25:00.338 DEBUG:boto:Headers: {} 2023-07-29T00:25:00.338 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:00.338 DEBUG:boto:Port: 8000 2023-07-29T00:25:00.339 DEBUG:boto:Params: {} 2023-07-29T00:25:00.339 DEBUG:boto:Token: None 2023-07-29T00:25:00.339 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:00 GMT /hiiqos-58/ 2023-07-29T00:25:00.340 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w/8cldGnsO241D20CiUq6gwsmRk= 2023-07-29T00:25:00.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w/8cldGnsO241D20CiUq6gwsmRk=', 'Content-Length': '0'} 2023-07-29T00:25:00.343 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002f00235ba822be74-0064c45c5c-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:00.344 DEBUG:boto:path=/hiiqos-58/obj00000000000000000053 2023-07-29T00:25:00.345 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000053 2023-07-29T00:25:00.345 DEBUG:boto:Method: PUT 2023-07-29T00:25:00.345 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000053 2023-07-29T00:25:00.346 DEBUG:boto:Data: 2023-07-29T00:25:00.346 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:25:00.346 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:00.347 DEBUG:boto:Port: 8000 2023-07-29T00:25:00.347 DEBUG:boto:Params: {} 2023-07-29T00:25:00.348 DEBUG:boto:Token: None 2023-07-29T00:25:00.348 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:25:00 GMT /hiiqos-58/obj00000000000000000053 2023-07-29T00:25:00.349 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0MYK2p2nTkQNc93XAatDr1NiD20= 2023-07-29T00:25:00.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:25:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0MYK2p2nTkQNc93XAatDr1NiD20='} 2023-07-29T00:25:00.392 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002fd7bd2d3a2b6a60-0064c45c5c-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:00.393 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-58', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:25:00.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-58', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:25:00.394 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync run --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-58 --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:25:00.443 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:25:00.443 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:25:00.510 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.457+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.511 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.458+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.511 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.458+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: sync status for source bucket shard: 2 2023-07-29T00:25:00.511 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.458+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.511 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.458+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: listing bilog for incremental sync; position=00000000001.72.6 2023-07-29T00:25:00.511 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.458+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.511 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.458+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: sync status for source bucket shard: 2 2023-07-29T00:25:00.512 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.458+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: listing bilog for incremental sync; position=00000000001.37.6 2023-07-29T00:25:00.513 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.458+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.513 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.458+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.513 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.458+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: sync status for source bucket shard: 2 2023-07-29T00:25:00.514 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.458+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: listing bilog for incremental sync; position=00000000001.37.6 2023-07-29T00:25:00.514 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.459+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: sync status for source bucket shard: 2 2023-07-29T00:25:00.514 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.459+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: listing bilog for incremental sync; position=00000000001.46.6 2023-07-29T00:25:00.514 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.459+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.514 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.459+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: sync status for source bucket shard: 2 2023-07-29T00:25:00.514 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.459+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.515 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.459+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: listing bilog for incremental sync; position=00000000001.30.6 2023-07-29T00:25:00.515 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.460+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.515 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.460+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: sync status for source bucket shard: 2 2023-07-29T00:25:00.515 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.460+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: listing bilog for incremental sync; position=00000000001.58.6 2023-07-29T00:25:00.515 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.460+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.515 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.460+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.515 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.460+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: sync status for source bucket shard: 2 2023-07-29T00:25:00.516 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.460+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.516 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.461+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: sync status for source bucket shard: 2 2023-07-29T00:25:00.516 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.461+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: listing bilog for incremental sync; position=00000000001.30.6 2023-07-29T00:25:00.516 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.461+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: sync status for source bucket shard: 2 2023-07-29T00:25:00.516 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.461+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: sync status for source bucket shard: 2 2023-07-29T00:25:00.516 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.461+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: sync status for source bucket shard: 2 2023-07-29T00:25:00.517 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.461+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: listing bilog for incremental sync; position=00000000001.58.6 2023-07-29T00:25:00.517 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.461+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: listing bilog for incremental sync; position=00000000001.38.6 2023-07-29T00:25:00.517 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.461+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: listing bilog for incremental sync; position=00000000001.35.6 2023-07-29T00:25:00.565 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.512+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.565 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.512+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.565 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.512+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.566 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.512+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: sync status for source bucket shard: 0 2023-07-29T00:25:00.566 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.512+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.566 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.513+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.566 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.513+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: sync status for source bucket shard: 0 2023-07-29T00:25:00.566 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.513+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.567 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.513+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.569 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.513+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.569 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.513+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: sync status for source bucket shard: 0 2023-07-29T00:25:00.569 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.513+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.569 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.513+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: sync status for source bucket shard: 0 2023-07-29T00:25:00.569 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.513+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: sync status for source bucket shard: 0 2023-07-29T00:25:00.569 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.513+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.570 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.513+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.570 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.513+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.570 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.514+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.570 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.514+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.570 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.514+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: sync status for source bucket shard: 0 2023-07-29T00:25:00.570 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.514+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.571 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.514+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: sync status for source bucket shard: 0 2023-07-29T00:25:00.571 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.514+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.571 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.515+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.571 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.515+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.571 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.515+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.572 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.515+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: sync status for source bucket shard: 0 2023-07-29T00:25:00.572 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.515+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.572 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.515+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: sync status for source bucket shard: 0 2023-07-29T00:25:00.572 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.515+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.572 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.515+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.572 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.516+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: sync status for source bucket shard: 0 2023-07-29T00:25:00.573 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.516+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.573 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.516+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: parsed entry: id=00000000001.37.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2023-07-29T00:25:00.573 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.516+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3/obj00000000000000000020 2023-07-29T00:25:00.573 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.516+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: entry->timestamp=2023-07-29T00:24:53.937979+0000 2023-07-29T00:25:00.573 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.516+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.37.6 op=0 op_state=1 2023-07-29T00:25:00.573 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.516+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.573 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.516+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:entry[obj00000000000000000020]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000020[0] 2023-07-29T00:25:00.574 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.516+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: listing bilog for incremental sync; position=00000000001.37.6 2023-07-29T00:25:00.574 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.516+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.574 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.516+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.574 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.516+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: sync status for source bucket shard: 0 2023-07-29T00:25:00.574 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.516+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.574 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.516+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: parsed entry: id=00000000002.46.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-07-29T00:25:00.575 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.516+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8/obj00000000000000000010 2023-07-29T00:25:00.575 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.516+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: entry->timestamp=2023-07-29T00:24:54.054726+0000 2023-07-29T00:25:00.575 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.516+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.46.6 op=0 op_state=1 2023-07-29T00:25:00.575 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.516+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:entry[obj00000000000000000010]: : injecting data sync error on key=obj00000000000000000010 2023-07-29T00:25:00.575 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.516+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: listing bilog for incremental sync; position=00000000002.46.6 2023-07-29T00:25:00.575 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.516+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:entry[obj00000000000000000010]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:25:00.576 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.516+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:entry[obj00000000000000000010]: ERROR: failed to sync object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8/obj00000000000000000010 2023-07-29T00:25:00.578 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.517+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: sync status for source bucket shard: 0 2023-07-29T00:25:00.578 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.517+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: sync status for source bucket shard: 0 2023-07-29T00:25:00.578 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.517+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.578 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.517+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.578 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.517+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.578 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.517+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: parsed entry: id=00000000001.52.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-07-29T00:25:00.579 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.517+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9/obj00000000000000000019 2023-07-29T00:25:00.579 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.517+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: entry->timestamp=2023-07-29T00:24:53.785118+0000 2023-07-29T00:25:00.579 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.517+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.52.6 op=0 op_state=1 2023-07-29T00:25:00.579 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.517+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:entry[obj00000000000000000019]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000019[0] 2023-07-29T00:25:00.579 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.517+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: listing bilog for incremental sync; position=00000000001.52.6 2023-07-29T00:25:00.579 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.518+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: sync status for source bucket shard: 0 2023-07-29T00:25:00.580 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.518+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: sync status for source bucket shard: 0 2023-07-29T00:25:00.580 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.518+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: sync status for source bucket shard: 0 2023-07-29T00:25:00.580 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.518+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.580 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.518+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.580 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.518+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.580 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.518+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: sync status for source bucket shard: 0 2023-07-29T00:25:00.580 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.518+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.581 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.519+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: parsed entry: id=00000000001.41.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2023-07-29T00:25:00.581 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.519+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12/obj00000000000000000030 2023-07-29T00:25:00.581 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.519+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: entry->timestamp=2023-07-29T00:24:53.860195+0000 2023-07-29T00:25:00.581 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.519+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.41.6 op=0 op_state=1 2023-07-29T00:25:00.581 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.519+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:entry[obj00000000000000000030]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000030[0] 2023-07-29T00:25:00.581 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.519+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: listing bilog for incremental sync; position=00000000001.41.6 2023-07-29T00:25:00.582 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.520+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: parsed entry: id=00000000002.50.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-07-29T00:25:00.582 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.520+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15/obj00000000000000000004 2023-07-29T00:25:00.582 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.520+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: entry->timestamp=2023-07-29T00:24:53.710755+0000 2023-07-29T00:25:00.582 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.520+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.50.6 op=0 op_state=1 2023-07-29T00:25:00.582 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.520+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: a sync operation returned error 2023-07-29T00:25:00.582 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.520+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:entry[obj00000000000000000004]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000004[0] 2023-07-29T00:25:00.583 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.520+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: listing bilog for incremental sync; position=00000000002.50.6 2023-07-29T00:25:00.583 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.520+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: backing out with sync_status=-5 2023-07-29T00:25:00.583 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.520+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: incremental sync on bucket failed, retcode=-5 2023-07-29T00:25:00.583 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.520+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: ERROR: incremental sync failed. error: -5 2023-07-29T00:25:00.583 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.520+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: parsed entry: id=00000000001.49.6 iter->object=obj00000000000000000076 iter->instance= name=obj00000000000000000076 instance= ns= 2023-07-29T00:25:00.583 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.520+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16/obj00000000000000000076 2023-07-29T00:25:00.583 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.520+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: entry->timestamp=2023-07-29T00:24:53.955523+0000 2023-07-29T00:25:00.584 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.520+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:entry[obj00000000000000000076]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.49.6 op=0 op_state=1 2023-07-29T00:25:00.584 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.520+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:entry[obj00000000000000000076]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000076[0] 2023-07-29T00:25:00.584 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.520+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: listing bilog for incremental sync; position=00000000001.49.6 2023-07-29T00:25:00.584 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.521+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: parsed entry: id=00000000002.57.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-07-29T00:25:00.584 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.521+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5/obj00000000000000000000 2023-07-29T00:25:00.584 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.521+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: entry->timestamp=2023-07-29T00:24:53.559981+0000 2023-07-29T00:25:00.585 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.521+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.57.6 op=0 op_state=1 2023-07-29T00:25:00.585 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.521+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:entry[obj00000000000000000000]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000000[0] 2023-07-29T00:25:00.585 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.521+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: listing bilog for incremental sync; position=00000000002.57.6 2023-07-29T00:25:00.585 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.522+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.585 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.523+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: sync status for source bucket shard: 0 2023-07-29T00:25:00.585 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.523+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.586 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.529+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:entry[obj00000000000000000020]: success 2023-07-29T00:25:00.586 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.529+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3:1 marker=00000000001.37.6 timestamp=2023-07-29T00:24:53.937979+0000 2023-07-29T00:25:00.587 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.530+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:entry[obj00000000000000000030]: success 2023-07-29T00:25:00.587 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.530+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12:1 marker=00000000001.41.6 timestamp=2023-07-29T00:24:53.860195+0000 2023-07-29T00:25:00.587 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.530+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:entry[obj00000000000000000019]: success 2023-07-29T00:25:00.587 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.530+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9:1 marker=00000000001.52.6 timestamp=2023-07-29T00:24:53.785118+0000 2023-07-29T00:25:00.587 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.530+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:entry[obj00000000000000000076]: success 2023-07-29T00:25:00.588 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.530+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16:1 marker=00000000001.49.6 timestamp=2023-07-29T00:24:53.955523+0000 2023-07-29T00:25:00.608 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.555+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: parsed entry: id=00000000001.59.6 iter->object=obj00000000000000000024 iter->instance= name=obj00000000000000000024 instance= ns= 2023-07-29T00:25:00.608 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.555+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2/obj00000000000000000024 2023-07-29T00:25:00.609 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.555+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: entry->timestamp=2023-07-29T00:24:53.729628+0000 2023-07-29T00:25:00.609 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.555+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:entry[obj00000000000000000024]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.59.6 op=0 op_state=1 2023-07-29T00:25:00.609 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.555+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:entry[obj00000000000000000024]: : injecting data sync error on key=obj00000000000000000024 2023-07-29T00:25:00.609 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.555+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: listing bilog for incremental sync; position=00000000001.59.6 2023-07-29T00:25:00.609 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.555+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: parsed entry: id=00000000001.108.6 iter->object=obj00000000000000000028 iter->instance= name=obj00000000000000000028 instance= ns= 2023-07-29T00:25:00.609 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.555+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4/obj00000000000000000028 2023-07-29T00:25:00.610 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.555+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: entry->timestamp=2023-07-29T00:24:53.541854+0000 2023-07-29T00:25:00.610 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.555+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:entry[obj00000000000000000028]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.108.6 op=0 op_state=1 2023-07-29T00:25:00.610 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.555+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: parsed entry: id=00000000001.44.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-07-29T00:25:00.610 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.555+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1/obj00000000000000000016 2023-07-29T00:25:00.610 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.555+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: entry->timestamp=2023-07-29T00:24:53.638713+0000 2023-07-29T00:25:00.610 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.555+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.44.6 op=0 op_state=1 2023-07-29T00:25:00.611 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.555+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: parsed entry: id=00000000002.36.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-07-29T00:25:00.611 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.555+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7/obj00000000000000000001 2023-07-29T00:25:00.611 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.555+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: entry->timestamp=2023-07-29T00:24:53.973227+0000 2023-07-29T00:25:00.611 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.555+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.36.6 op=0 op_state=1 2023-07-29T00:25:00.611 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.555+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:entry[obj00000000000000000024]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:25:00.611 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.555+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:entry[obj00000000000000000024]: ERROR: failed to sync object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2/obj00000000000000000024 2023-07-29T00:25:00.612 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.555+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:entry[obj00000000000000000028]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000028[0] 2023-07-29T00:25:00.612 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.555+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: listing bilog for incremental sync; position=00000000001.108.6 2023-07-29T00:25:00.612 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.555+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:entry[obj00000000000000000016]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000016[0] 2023-07-29T00:25:00.612 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.555+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: listing bilog for incremental sync; position=00000000001.44.6 2023-07-29T00:25:00.612 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.555+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:entry[obj00000000000000000001]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000001[0] 2023-07-29T00:25:00.612 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.555+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: listing bilog for incremental sync; position=00000000002.36.6 2023-07-29T00:25:00.613 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.556+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: parsed entry: id=00000000001.110.6 iter->object=obj00000000000000000032 iter->instance= name=obj00000000000000000032 instance= ns= 2023-07-29T00:25:00.613 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.556+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6/obj00000000000000000032 2023-07-29T00:25:00.614 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.557+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: entry->timestamp=2023-07-29T00:24:53.877366+0000 2023-07-29T00:25:00.614 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.557+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:entry[obj00000000000000000032]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.110.6 op=0 op_state=1 2023-07-29T00:25:00.615 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.557+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:entry[obj00000000000000000032]: : injecting data sync error on key=obj00000000000000000032 2023-07-29T00:25:00.615 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.557+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: listing bilog for incremental sync; position=00000000001.110.6 2023-07-29T00:25:00.615 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.557+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:entry[obj00000000000000000032]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:25:00.615 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.557+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:entry[obj00000000000000000032]: ERROR: failed to sync object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6/obj00000000000000000032 2023-07-29T00:25:00.615 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.558+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: parsed entry: id=00000000002.38.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2023-07-29T00:25:00.615 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.558+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10/obj00000000000000000013 2023-07-29T00:25:00.615 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.558+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: entry->timestamp=2023-07-29T00:24:54.036836+0000 2023-07-29T00:25:00.616 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.558+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.38.6 op=0 op_state=1 2023-07-29T00:25:00.616 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.558+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:entry[obj00000000000000000013]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000013[0] 2023-07-29T00:25:00.616 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.558+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: listing bilog for incremental sync; position=00000000002.38.6 2023-07-29T00:25:00.616 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.560+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: parsed entry: id=00000000001.35.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2023-07-29T00:25:00.616 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.560+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13/obj00000000000000000026 2023-07-29T00:25:00.616 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.560+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: entry->timestamp=2023-07-29T00:24:53.617963+0000 2023-07-29T00:25:00.617 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.560+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.35.6 op=0 op_state=1 2023-07-29T00:25:00.617 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.560+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:entry[obj00000000000000000026]: : injecting data sync error on key=obj00000000000000000026 2023-07-29T00:25:00.617 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.560+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: listing bilog for incremental sync; position=00000000001.35.6 2023-07-29T00:25:00.617 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.560+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:entry[obj00000000000000000026]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:25:00.617 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.560+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:entry[obj00000000000000000026]: ERROR: failed to sync object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13/obj00000000000000000026 2023-07-29T00:25:00.617 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.561+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: parsed entry: id=00000000001.76.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2023-07-29T00:25:00.618 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.561+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0/obj00000000000000000007 2023-07-29T00:25:00.618 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.561+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: entry->timestamp=2023-07-29T00:24:53.804231+0000 2023-07-29T00:25:00.618 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.561+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.76.6 op=0 op_state=1 2023-07-29T00:25:00.618 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.561+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:entry[obj00000000000000000007]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000007[0] 2023-07-29T00:25:00.618 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.561+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: listing bilog for incremental sync; position=00000000001.76.6 2023-07-29T00:25:00.618 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.561+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: parsed entry: id=00000000001.71.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2023-07-29T00:25:00.618 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.561+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14/obj00000000000000000033 2023-07-29T00:25:00.619 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.561+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: entry->timestamp=2023-07-29T00:24:54.072785+0000 2023-07-29T00:25:00.619 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.561+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.71.6 op=0 op_state=1 2023-07-29T00:25:00.619 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.561+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:entry[obj00000000000000000033]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000033[0] 2023-07-29T00:25:00.619 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.561+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: listing bilog for incremental sync; position=00000000001.71.6 2023-07-29T00:25:00.619 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.562+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:entry[obj00000000000000000016]: success 2023-07-29T00:25:00.619 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.562+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1:1 marker=00000000001.44.6 timestamp=2023-07-29T00:24:53.638713+0000 2023-07-29T00:25:00.620 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.565+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: parsed entry: id=00000000002.46.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-07-29T00:25:00.620 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.565+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8/obj00000000000000000010 2023-07-29T00:25:00.621 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.565+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: entry->timestamp=2023-07-29T00:24:54.054726+0000 2023-07-29T00:25:00.621 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.565+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.46.6 op=0 op_state=1 2023-07-29T00:25:00.621 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.565+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:entry[obj00000000000000000010]: : injecting data sync error on key=obj00000000000000000010 2023-07-29T00:25:00.621 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.565+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: listing bilog for incremental sync; position=00000000002.46.6 2023-07-29T00:25:00.621 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.565+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:entry[obj00000000000000000010]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:25:00.621 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.565+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:entry[obj00000000000000000010]: ERROR: failed to sync object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8/obj00000000000000000010 2023-07-29T00:25:00.621 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.567+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:entry[obj00000000000000000007]: success 2023-07-29T00:25:00.622 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.567+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0:1 marker=00000000001.76.6 timestamp=2023-07-29T00:24:53.804231+0000 2023-07-29T00:25:00.622 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.569+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:entry[obj00000000000000000004]: success 2023-07-29T00:25:00.622 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.569+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15:1 marker=00000000002.50.6 timestamp=2023-07-29T00:24:53.710755+0000 2023-07-29T00:25:00.623 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.571+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:entry[obj00000000000000000000]: success 2023-07-29T00:25:00.623 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.571+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5:1 marker=00000000002.57.6 timestamp=2023-07-29T00:24:53.559981+0000 2023-07-29T00:25:00.623 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.571+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:entry[obj00000000000000000033]: success 2023-07-29T00:25:00.623 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.571+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14:1 marker=00000000001.71.6 timestamp=2023-07-29T00:24:54.072785+0000 2023-07-29T00:25:00.650 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.598+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: a sync operation returned error 2023-07-29T00:25:00.650 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.598+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: backing out with sync_status=-5 2023-07-29T00:25:00.651 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.598+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: incremental sync on bucket failed, retcode=-5 2023-07-29T00:25:00.651 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.598+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: ERROR: incremental sync failed. error: -5 2023-07-29T00:25:00.651 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.599+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.652 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.599+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: a sync operation returned error 2023-07-29T00:25:00.652 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.599+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: backing out with sync_status=-5 2023-07-29T00:25:00.652 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.599+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: incremental sync on bucket failed, retcode=-5 2023-07-29T00:25:00.652 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.599+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: ERROR: incremental sync failed. error: -5 2023-07-29T00:25:00.652 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.599+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: sync status for source bucket shard: 0 2023-07-29T00:25:00.652 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.599+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.653 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.600+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.653 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.600+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: sync status for source bucket shard: 0 2023-07-29T00:25:00.653 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.600+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.654 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.603+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: a sync operation returned error 2023-07-29T00:25:00.655 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.603+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: backing out with sync_status=-5 2023-07-29T00:25:00.655 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.603+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: incremental sync on bucket failed, retcode=-5 2023-07-29T00:25:00.655 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.603+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: ERROR: incremental sync failed. error: -5 2023-07-29T00:25:00.656 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.603+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:entry[obj00000000000000000001]: success 2023-07-29T00:25:00.656 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.603+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7:1 marker=00000000002.36.6 timestamp=2023-07-29T00:24:53.973227+0000 2023-07-29T00:25:00.656 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.604+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.656 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.604+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: sync status for source bucket shard: 0 2023-07-29T00:25:00.656 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.604+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.657 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.605+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:entry[obj00000000000000000028]: success 2023-07-29T00:25:00.657 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.605+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4:1 marker=00000000001.108.6 timestamp=2023-07-29T00:24:53.541854+0000 2023-07-29T00:25:00.657 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.605+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:entry[obj00000000000000000013]: success 2023-07-29T00:25:00.657 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.605+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10:1 marker=00000000002.38.6 timestamp=2023-07-29T00:24:54.036836+0000 2023-07-29T00:25:00.659 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.608+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: a sync operation returned error 2023-07-29T00:25:00.660 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.608+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: backing out with sync_status=-5 2023-07-29T00:25:00.660 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.608+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: incremental sync on bucket failed, retcode=-5 2023-07-29T00:25:00.660 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.608+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: ERROR: incremental sync failed. error: -5 2023-07-29T00:25:00.661 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.608+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.661 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.609+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: sync status for source bucket shard: 0 2023-07-29T00:25:00.661 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.609+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.694 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.642+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: parsed entry: id=00000000001.59.6 iter->object=obj00000000000000000024 iter->instance= name=obj00000000000000000024 instance= ns= 2023-07-29T00:25:00.695 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.642+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2/obj00000000000000000024 2023-07-29T00:25:00.696 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.642+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: entry->timestamp=2023-07-29T00:24:53.729628+0000 2023-07-29T00:25:00.696 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.642+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:entry[obj00000000000000000024]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.59.6 op=0 op_state=1 2023-07-29T00:25:00.696 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.642+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:entry[obj00000000000000000024]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000024[0] 2023-07-29T00:25:00.696 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.642+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: listing bilog for incremental sync; position=00000000001.59.6 2023-07-29T00:25:00.697 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.643+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: parsed entry: id=00000000001.110.6 iter->object=obj00000000000000000032 iter->instance= name=obj00000000000000000032 instance= ns= 2023-07-29T00:25:00.697 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.643+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6/obj00000000000000000032 2023-07-29T00:25:00.697 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.643+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: entry->timestamp=2023-07-29T00:24:53.877366+0000 2023-07-29T00:25:00.697 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.643+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:entry[obj00000000000000000032]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.110.6 op=0 op_state=1 2023-07-29T00:25:00.698 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.643+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:entry[obj00000000000000000032]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000032[0] 2023-07-29T00:25:00.698 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.643+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: listing bilog for incremental sync; position=00000000001.110.6 2023-07-29T00:25:00.699 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.647+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: parsed entry: id=00000000001.35.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2023-07-29T00:25:00.699 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.647+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13/obj00000000000000000026 2023-07-29T00:25:00.699 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.647+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: entry->timestamp=2023-07-29T00:24:53.617963+0000 2023-07-29T00:25:00.699 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.647+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.35.6 op=0 op_state=1 2023-07-29T00:25:00.700 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.647+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:entry[obj00000000000000000026]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000026[0] 2023-07-29T00:25:00.700 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.647+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: listing bilog for incremental sync; position=00000000001.35.6 2023-07-29T00:25:00.700 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.647+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:entry[obj00000000000000000024]: success 2023-07-29T00:25:00.700 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.647+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2:1 marker=00000000001.59.6 timestamp=2023-07-29T00:24:53.729628+0000 2023-07-29T00:25:00.701 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.649+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:entry[obj00000000000000000032]: success 2023-07-29T00:25:00.701 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.649+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6:1 marker=00000000001.110.6 timestamp=2023-07-29T00:24:53.877366+0000 2023-07-29T00:25:00.703 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.651+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: parsed entry: id=00000000002.46.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-07-29T00:25:00.703 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.651+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8/obj00000000000000000010 2023-07-29T00:25:00.703 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.651+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: entry->timestamp=2023-07-29T00:24:54.054726+0000 2023-07-29T00:25:00.703 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.651+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.46.6 op=0 op_state=1 2023-07-29T00:25:00.703 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.651+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:entry[obj00000000000000000010]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000010[0] 2023-07-29T00:25:00.703 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.651+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: listing bilog for incremental sync; position=00000000002.46.6 2023-07-29T00:25:00.745 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.693+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:entry[obj00000000000000000026]: success 2023-07-29T00:25:00.746 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.693+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13:1 marker=00000000001.35.6 timestamp=2023-07-29T00:24:53.617963+0000 2023-07-29T00:25:00.748 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.696+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:entry[obj00000000000000000010]: success 2023-07-29T00:25:00.748 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.696+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8:1 marker=00000000002.46.6 timestamp=2023-07-29T00:24:54.054726+0000 2023-07-29T00:25:00.754 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.701+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.754 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.701+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.754 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.701+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.754 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.701+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.755 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.701+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: sync status for source bucket shard: 0 2023-07-29T00:25:00.755 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.701+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: sync status for source bucket shard: 0 2023-07-29T00:25:00.755 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.701+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.755 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.701+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.755 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.701+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: sync status for source bucket shard: 0 2023-07-29T00:25:00.756 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.701+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.756 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.701+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.756 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.702+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: sync status for source bucket shard: 0 2023-07-29T00:25:00.756 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.702+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.756 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.702+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.758 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.702+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: sync status for source bucket shard: 0 2023-07-29T00:25:00.758 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.702+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.758 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.702+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.758 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.702+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.758 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.702+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.759 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.702+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.759 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.703+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.759 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.703+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: sync status for source bucket shard: 0 2023-07-29T00:25:00.759 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.703+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.759 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.703+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: sync status for source bucket shard: 0 2023-07-29T00:25:00.759 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.703+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.760 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.703+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: sync status for source bucket shard: 0 2023-07-29T00:25:00.760 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.703+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.760 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.704+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: sync status for source bucket shard: 0 2023-07-29T00:25:00.760 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.704+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: sync status for source bucket shard: 0 2023-07-29T00:25:00.760 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.704+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.760 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.704+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.761 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.704+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: parsed entry: id=00000000002.30.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-07-29T00:25:00.761 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.704+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4/obj00000000000000000004 2023-07-29T00:25:00.761 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.704+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: entry->timestamp=2023-07-29T00:24:54.920460+0000 2023-07-29T00:25:00.761 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.704+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.30.6 op=0 op_state=1 2023-07-29T00:25:00.761 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.704+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.761 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.704+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.761 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.704+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.762 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.704+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.762 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.704+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:entry[obj00000000000000000004]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000004[0] 2023-07-29T00:25:00.762 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.704+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: listing bilog for incremental sync; position=00000000002.30.6 2023-07-29T00:25:00.762 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.705+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: sync status for source bucket shard: 0 2023-07-29T00:25:00.762 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.705+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.762 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.705+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.763 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.705+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.763 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.705+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.763 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.705+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: parsed entry: id=00000000002.30.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2023-07-29T00:25:00.763 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.705+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7/obj00000000000000000015 2023-07-29T00:25:00.763 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.705+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: entry->timestamp=2023-07-29T00:24:54.941573+0000 2023-07-29T00:25:00.763 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.705+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.30.6 op=0 op_state=1 2023-07-29T00:25:00.763 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.705+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.764 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.705+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:entry[obj00000000000000000015]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000015[0] 2023-07-29T00:25:00.764 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.705+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: listing bilog for incremental sync; position=00000000002.30.6 2023-07-29T00:25:00.764 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.706+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: sync status for source bucket shard: 0 2023-07-29T00:25:00.764 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.706+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: sync status for source bucket shard: 0 2023-07-29T00:25:00.764 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.706+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.764 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.706+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: sync status for source bucket shard: 0 2023-07-29T00:25:00.765 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.706+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: sync status for source bucket shard: 0 2023-07-29T00:25:00.765 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.706+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]: sync status for source bucket shard: 0 2023-07-29T00:25:00.765 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.706+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.765 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.706+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.765 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.706+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.765 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.706+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.766 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.706+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: parsed entry: id=00000000002.45.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2023-07-29T00:25:00.766 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.706+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6/obj00000000000000000030 2023-07-29T00:25:00.766 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.706+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: entry->timestamp=2023-07-29T00:24:54.556315+0000 2023-07-29T00:25:00.766 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.706+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.45.6 op=0 op_state=1 2023-07-29T00:25:00.766 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.706+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:entry[obj00000000000000000030]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000030[0] 2023-07-29T00:25:00.766 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.706+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: listing bilog for incremental sync; position=00000000002.45.6 2023-07-29T00:25:00.768 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.707+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: sync status for source bucket shard: 0 2023-07-29T00:25:00.768 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.707+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: sync status for source bucket shard: 0 2023-07-29T00:25:00.768 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.707+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.768 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.707+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]: sync status for source bucket shard: 0 2023-07-29T00:25:00.769 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.707+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.769 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.707+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.769 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.707+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: parsed entry: id=00000000002.37.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-07-29T00:25:00.769 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.707+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8/obj00000000000000000019 2023-07-29T00:25:00.769 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.707+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: entry->timestamp=2023-07-29T00:24:54.594467+0000 2023-07-29T00:25:00.769 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.707+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.37.6 op=0 op_state=1 2023-07-29T00:25:00.770 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.707+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:entry[obj00000000000000000019]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000019[0] 2023-07-29T00:25:00.770 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.707+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: listing bilog for incremental sync; position=00000000002.37.6 2023-07-29T00:25:00.770 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.707+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: parsed entry: id=00000000001.28.6 iter->object=obj00000000000000000025 iter->instance= name=obj00000000000000000025 instance= ns= 2023-07-29T00:25:00.771 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.707+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5/obj00000000000000000025 2023-07-29T00:25:00.771 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.707+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: entry->timestamp=2023-07-29T00:24:54.816289+0000 2023-07-29T00:25:00.771 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.707+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:entry[obj00000000000000000025]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.28.6 op=0 op_state=1 2023-07-29T00:25:00.771 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.707+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:entry[obj00000000000000000025]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000025[0] 2023-07-29T00:25:00.771 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.707+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: listing bilog for incremental sync; position=00000000001.28.6 2023-07-29T00:25:00.771 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.708+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: parsed entry: id=00000000002.42.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-07-29T00:25:00.772 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.708+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15/obj00000000000000000006 2023-07-29T00:25:00.772 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.708+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: entry->timestamp=2023-07-29T00:24:54.494465+0000 2023-07-29T00:25:00.772 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.708+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.42.6 op=0 op_state=1 2023-07-29T00:25:00.772 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.708+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:entry[obj00000000000000000006]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000006[0] 2023-07-29T00:25:00.772 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.708+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: listing bilog for incremental sync; position=00000000002.42.6 2023-07-29T00:25:00.772 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.709+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:entry[obj00000000000000000004]: success 2023-07-29T00:25:00.772 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.709+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4:2 marker=00000000002.30.6 timestamp=2023-07-29T00:24:54.920460+0000 2023-07-29T00:25:00.773 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.709+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: parsed entry: id=00000000002.64.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-07-29T00:25:00.773 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.709+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16/obj00000000000000000011 2023-07-29T00:25:00.773 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.709+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: entry->timestamp=2023-07-29T00:24:54.574842+0000 2023-07-29T00:25:00.773 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.709+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.64.6 op=0 op_state=1 2023-07-29T00:25:00.773 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.709+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:entry[obj00000000000000000011]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000011[0] 2023-07-29T00:25:00.773 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.709+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: listing bilog for incremental sync; position=00000000002.64.6 2023-07-29T00:25:00.774 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.709+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]: parsed entry: id=00000000002.33.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2023-07-29T00:25:00.774 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.709+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18/obj00000000000000000007 2023-07-29T00:25:00.774 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.709+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]: entry->timestamp=2023-07-29T00:24:54.757744+0000 2023-07-29T00:25:00.774 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.709+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.33.6 op=0 op_state=1 2023-07-29T00:25:00.774 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.709+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:entry[obj00000000000000000007]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000007[0] 2023-07-29T00:25:00.774 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.709+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]: listing bilog for incremental sync; position=00000000002.33.6 2023-07-29T00:25:00.775 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.710+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]: parsed entry: id=00000000002.63.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-07-29T00:25:00.775 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.710+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17/obj00000000000000000001 2023-07-29T00:25:00.775 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.710+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]: entry->timestamp=2023-07-29T00:24:54.835676+0000 2023-07-29T00:25:00.775 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.710+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.63.6 op=0 op_state=1 2023-07-29T00:25:00.775 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.710+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:entry[obj00000000000000000001]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000001[0] 2023-07-29T00:25:00.775 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.710+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]: listing bilog for incremental sync; position=00000000002.63.6 2023-07-29T00:25:00.776 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.711+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:entry[obj00000000000000000030]: success 2023-07-29T00:25:00.776 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.711+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6:2 marker=00000000002.45.6 timestamp=2023-07-29T00:24:54.556315+0000 2023-07-29T00:25:00.776 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.713+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: parsed entry: id=00000000002.52.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-07-29T00:25:00.776 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.713+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13/obj00000000000000000002 2023-07-29T00:25:00.776 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.713+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: entry->timestamp=2023-07-29T00:24:54.400639+0000 2023-07-29T00:25:00.776 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.713+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.52.6 op=0 op_state=1 2023-07-29T00:25:00.777 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.713+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:entry[obj00000000000000000002]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000002[0] 2023-07-29T00:25:00.777 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.713+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: listing bilog for incremental sync; position=00000000002.52.6 2023-07-29T00:25:00.777 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.714+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:entry[obj00000000000000000019]: success 2023-07-29T00:25:00.777 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.714+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8:2 marker=00000000002.37.6 timestamp=2023-07-29T00:24:54.594467+0000 2023-07-29T00:25:00.777 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.714+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:entry[obj00000000000000000007]: success 2023-07-29T00:25:00.777 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.714+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18:2 marker=00000000002.33.6 timestamp=2023-07-29T00:24:54.757744+0000 2023-07-29T00:25:00.777 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.714+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:entry[obj00000000000000000001]: success 2023-07-29T00:25:00.778 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.714+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17:2 marker=00000000002.63.6 timestamp=2023-07-29T00:24:54.835676+0000 2023-07-29T00:25:00.778 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.718+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:entry[obj00000000000000000011]: success 2023-07-29T00:25:00.778 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.718+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16:2 marker=00000000002.64.6 timestamp=2023-07-29T00:24:54.574842+0000 2023-07-29T00:25:00.778 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.720+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:entry[obj00000000000000000006]: success 2023-07-29T00:25:00.779 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.720+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15:2 marker=00000000002.42.6 timestamp=2023-07-29T00:24:54.494465+0000 2023-07-29T00:25:00.797 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.744+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: parsed entry: id=00000000002.34.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2023-07-29T00:25:00.797 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.744+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0/obj00000000000000000005 2023-07-29T00:25:00.798 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.744+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: entry->timestamp=2023-07-29T00:24:54.420187+0000 2023-07-29T00:25:00.798 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.744+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.34.6 op=0 op_state=1 2023-07-29T00:25:00.798 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.744+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:entry[obj00000000000000000005]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000005[0] 2023-07-29T00:25:00.798 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.744+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: listing bilog for incremental sync; position=00000000002.34.6 2023-07-29T00:25:00.798 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.744+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: parsed entry: id=00000000002.47.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-07-29T00:25:00.798 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.744+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1/obj00000000000000000010 2023-07-29T00:25:00.799 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.744+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: entry->timestamp=2023-07-29T00:24:54.678611+0000 2023-07-29T00:25:00.799 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.744+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.47.6 op=0 op_state=1 2023-07-29T00:25:00.799 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.744+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: parsed entry: id=00000000001.66.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-07-29T00:25:00.799 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.744+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3/obj00000000000000000021 2023-07-29T00:25:00.799 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.744+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: entry->timestamp=2023-07-29T00:24:54.343723+0000 2023-07-29T00:25:00.799 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.744+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.66.6 op=0 op_state=1 2023-07-29T00:25:00.800 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.744+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:entry[obj00000000000000000010]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000010[0] 2023-07-29T00:25:00.800 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.744+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: listing bilog for incremental sync; position=00000000002.47.6 2023-07-29T00:25:00.800 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.744+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:entry[obj00000000000000000021]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000021[0] 2023-07-29T00:25:00.800 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.744+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: listing bilog for incremental sync; position=00000000001.66.6 2023-07-29T00:25:00.800 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.745+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: parsed entry: id=00000000002.17.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-07-29T00:25:00.802 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.746+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9/obj00000000000000000003 2023-07-29T00:25:00.802 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.746+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: entry->timestamp=2023-07-29T00:24:54.324153+0000 2023-07-29T00:25:00.802 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.746+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.17.6 op=0 op_state=1 2023-07-29T00:25:00.802 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.746+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:entry[obj00000000000000000003]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000003[0] 2023-07-29T00:25:00.802 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.746+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: listing bilog for incremental sync; position=00000000002.17.6 2023-07-29T00:25:00.803 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.746+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: parsed entry: id=00000000001.69.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-07-29T00:25:00.803 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.746+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10/obj00000000000000000009 2023-07-29T00:25:00.804 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.746+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: entry->timestamp=2023-07-29T00:24:54.476578+0000 2023-07-29T00:25:00.804 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.746+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.69.6 op=0 op_state=1 2023-07-29T00:25:00.804 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.746+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:entry[obj00000000000000000009]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000009[0] 2023-07-29T00:25:00.804 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.746+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: listing bilog for incremental sync; position=00000000001.69.6 2023-07-29T00:25:00.804 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.747+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:entry[obj00000000000000000010]: success 2023-07-29T00:25:00.804 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.747+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1:2 marker=00000000002.47.6 timestamp=2023-07-29T00:24:54.678611+0000 2023-07-29T00:25:00.805 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.748+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: parsed entry: id=00000000002.68.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-07-29T00:25:00.805 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.748+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14/obj00000000000000000000 2023-07-29T00:25:00.805 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.748+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: entry->timestamp=2023-07-29T00:24:54.738555+0000 2023-07-29T00:25:00.805 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.748+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.68.6 op=0 op_state=1 2023-07-29T00:25:00.805 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.748+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:entry[obj00000000000000000000]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000000[0] 2023-07-29T00:25:00.805 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.748+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: listing bilog for incremental sync; position=00000000002.68.6 2023-07-29T00:25:00.806 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.749+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: parsed entry: id=00000000001.66.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2023-07-29T00:25:00.806 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.749+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11/obj00000000000000000014 2023-07-29T00:25:00.806 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.749+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: entry->timestamp=2023-07-29T00:24:54.855220+0000 2023-07-29T00:25:00.806 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.749+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.66.6 op=0 op_state=1 2023-07-29T00:25:00.806 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.749+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:entry[obj00000000000000000014]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000014[0] 2023-07-29T00:25:00.806 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.749+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: listing bilog for incremental sync; position=00000000001.66.6 2023-07-29T00:25:00.807 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.750+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: parsed entry: id=00000000002.116.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-07-29T00:25:00.807 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.750+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12/obj00000000000000000012 2023-07-29T00:25:00.807 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.750+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: entry->timestamp=2023-07-29T00:24:54.652513+0000 2023-07-29T00:25:00.807 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.750+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.116.6 op=0 op_state=1 2023-07-29T00:25:00.807 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.750+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:entry[obj00000000000000000012]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000012[0] 2023-07-29T00:25:00.807 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.750+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: listing bilog for incremental sync; position=00000000002.116.6 2023-07-29T00:25:00.808 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.751+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:entry[obj00000000000000000000]: success 2023-07-29T00:25:00.808 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.751+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14:2 marker=00000000002.68.6 timestamp=2023-07-29T00:24:54.738555+0000 2023-07-29T00:25:00.808 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.754+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:entry[obj00000000000000000015]: success 2023-07-29T00:25:00.808 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.754+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7:2 marker=00000000002.30.6 timestamp=2023-07-29T00:24:54.941573+0000 2023-07-29T00:25:00.809 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.755+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:entry[obj00000000000000000009]: success 2023-07-29T00:25:00.809 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.755+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10:2 marker=00000000001.69.6 timestamp=2023-07-29T00:24:54.476578+0000 2023-07-29T00:25:00.809 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.757+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:entry[obj00000000000000000025]: success 2023-07-29T00:25:00.809 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.757+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5:2 marker=00000000001.28.6 timestamp=2023-07-29T00:24:54.816289+0000 2023-07-29T00:25:00.812 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.761+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:entry[obj00000000000000000002]: success 2023-07-29T00:25:00.813 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.761+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13:2 marker=00000000002.52.6 timestamp=2023-07-29T00:24:54.400639+0000 2023-07-29T00:25:00.843 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.790+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:entry[obj00000000000000000005]: success 2023-07-29T00:25:00.843 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.790+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0:2 marker=00000000002.34.6 timestamp=2023-07-29T00:24:54.420187+0000 2023-07-29T00:25:00.844 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.791+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:entry[obj00000000000000000021]: success 2023-07-29T00:25:00.844 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.791+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3:2 marker=00000000001.66.6 timestamp=2023-07-29T00:24:54.343723+0000 2023-07-29T00:25:00.847 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.795+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:entry[obj00000000000000000003]: success 2023-07-29T00:25:00.847 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.795+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9:2 marker=00000000002.17.6 timestamp=2023-07-29T00:24:54.324153+0000 2023-07-29T00:25:00.849 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.797+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:entry[obj00000000000000000014]: success 2023-07-29T00:25:00.849 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.797+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11:2 marker=00000000001.66.6 timestamp=2023-07-29T00:24:54.855220+0000 2023-07-29T00:25:00.849 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.797+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:entry[obj00000000000000000012]: success 2023-07-29T00:25:00.849 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.797+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12:2 marker=00000000002.116.6 timestamp=2023-07-29T00:24:54.652513+0000 2023-07-29T00:25:00.863 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.810+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.863 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.810+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.864 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.810+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.864 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.810+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: sync status for source bucket shard: 0 2023-07-29T00:25:00.864 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.810+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.864 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.810+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: sync status for source bucket shard: 0 2023-07-29T00:25:00.865 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.810+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.865 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.811+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.865 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.811+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: sync status for source bucket shard: 0 2023-07-29T00:25:00.865 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.811+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.866 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.811+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.866 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.811+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: sync status for source bucket shard: 0 2023-07-29T00:25:00.866 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.811+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.866 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.811+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.867 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.811+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.868 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.812+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.868 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.812+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: sync status for source bucket shard: 0 2023-07-29T00:25:00.868 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.812+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.869 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.812+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: sync status for source bucket shard: 0 2023-07-29T00:25:00.869 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.812+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.869 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.812+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.870 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.813+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.870 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.813+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: sync status for source bucket shard: 0 2023-07-29T00:25:00.870 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.813+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.870 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.813+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: sync status for source bucket shard: 0 2023-07-29T00:25:00.870 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.813+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.870 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.813+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.871 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.813+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.871 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.813+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.871 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.813+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: sync status for source bucket shard: 0 2023-07-29T00:25:00.871 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.813+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.871 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.813+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.871 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.813+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.872 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.814+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: sync status for source bucket shard: 0 2023-07-29T00:25:00.872 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.814+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.872 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.814+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: sync status for source bucket shard: 0 2023-07-29T00:25:00.872 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.814+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: sync status for source bucket shard: 0 2023-07-29T00:25:00.872 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.814+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.872 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.814+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.872 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.814+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: sync status for source bucket shard: 0 2023-07-29T00:25:00.873 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.814+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.873 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.814+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.873 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.814+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.873 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.815+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: sync status for source bucket shard: 0 2023-07-29T00:25:00.873 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.815+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.873 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.815+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.874 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.815+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.874 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.815+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: parsed entry: id=00000000002.80.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-07-29T00:25:00.874 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.815+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7/obj00000000000000000002 2023-07-29T00:25:00.874 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.815+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: entry->timestamp=2023-07-29T00:24:55.392159+0000 2023-07-29T00:25:00.874 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.815+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.80.6 op=0 op_state=1 2023-07-29T00:25:00.874 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.815+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.875 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.815+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.875 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.815+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:entry[obj00000000000000000002]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000002[0] 2023-07-29T00:25:00.875 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.815+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: listing bilog for incremental sync; position=00000000002.80.6 2023-07-29T00:25:00.876 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.816+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.877 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.816+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: sync status for source bucket shard: 0 2023-07-29T00:25:00.877 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.816+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.877 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.816+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: sync status for source bucket shard: 0 2023-07-29T00:25:00.877 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.816+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: parsed entry: id=00000000002.74.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2023-07-29T00:25:00.877 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.816+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9/obj00000000000000000007 2023-07-29T00:25:00.877 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.816+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: entry->timestamp=2023-07-29T00:24:55.625568+0000 2023-07-29T00:25:00.878 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.816+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.74.6 op=0 op_state=1 2023-07-29T00:25:00.878 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.816+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.878 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.816+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: sync status for source bucket shard: 0 2023-07-29T00:25:00.878 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.816+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.878 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.816+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:entry[obj00000000000000000007]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000007[0] 2023-07-29T00:25:00.879 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.816+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: listing bilog for incremental sync; position=00000000002.74.6 2023-07-29T00:25:00.879 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.817+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.879 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.817+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]: sync status for source bucket shard: 0 2023-07-29T00:25:00.879 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.817+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.879 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.817+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]: sync status for source bucket shard: 0 2023-07-29T00:25:00.880 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.817+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.880 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.817+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: parsed entry: id=00000000002.78.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-07-29T00:25:00.880 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.817+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12/obj00000000000000000009 2023-07-29T00:25:00.880 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.817+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: entry->timestamp=2023-07-29T00:24:55.281553+0000 2023-07-29T00:25:00.880 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.817+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.78.6 op=0 op_state=1 2023-07-29T00:25:00.880 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.818+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]: sync status for source bucket shard: 0 2023-07-29T00:25:00.881 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.818+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:entry[obj00000000000000000009]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000009[0] 2023-07-29T00:25:00.881 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.818+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: listing bilog for incremental sync; position=00000000002.78.6 2023-07-29T00:25:00.881 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.818+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.881 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.818+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]: sync status for source bucket shard: 0 2023-07-29T00:25:00.881 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.818+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]: sync status for source bucket shard: 0 2023-07-29T00:25:00.881 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.818+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.882 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.818+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.882 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.819+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: parsed entry: id=00000000002.42.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2023-07-29T00:25:00.882 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.819+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14/obj00000000000000000013 2023-07-29T00:25:00.882 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.819+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: entry->timestamp=2023-07-29T00:24:55.453055+0000 2023-07-29T00:25:00.882 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.819+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.42.6 op=0 op_state=1 2023-07-29T00:25:00.882 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.819+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: parsed entry: id=00000000001.55.6 iter->object=obj00000000000000000045 iter->instance= name=obj00000000000000000045 instance= ns= 2023-07-29T00:25:00.882 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.819+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13/obj00000000000000000045 2023-07-29T00:25:00.883 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.819+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: entry->timestamp=2023-07-29T00:24:55.303471+0000 2023-07-29T00:25:00.883 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.819+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:entry[obj00000000000000000045]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.55.6 op=0 op_state=1 2023-07-29T00:25:00.883 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.819+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:entry[obj00000000000000000013]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000013[0] 2023-07-29T00:25:00.883 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.819+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: listing bilog for incremental sync; position=00000000002.42.6 2023-07-29T00:25:00.883 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.819+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:entry[obj00000000000000000045]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000045[0] 2023-07-29T00:25:00.884 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.819+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: listing bilog for incremental sync; position=00000000001.55.6 2023-07-29T00:25:00.884 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.820+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: sync status for source bucket shard: 0 2023-07-29T00:25:00.884 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.820+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.884 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.822+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:entry[obj00000000000000000002]: success 2023-07-29T00:25:00.884 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.822+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7:3 marker=00000000002.80.6 timestamp=2023-07-29T00:24:55.392159+0000 2023-07-29T00:25:00.885 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.822+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: parsed entry: id=00000000002.57.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2023-07-29T00:25:00.885 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.822+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16/obj00000000000000000033 2023-07-29T00:25:00.885 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.822+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: entry->timestamp=2023-07-29T00:24:55.369624+0000 2023-07-29T00:25:00.885 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.822+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.57.6 op=0 op_state=1 2023-07-29T00:25:00.885 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.822+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:entry[obj00000000000000000033]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000033[0] 2023-07-29T00:25:00.885 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.822+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: listing bilog for incremental sync; position=00000000002.57.6 2023-07-29T00:25:00.885 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.822+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]: parsed entry: id=00000000002.26.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-07-29T00:25:00.886 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.822+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20/obj00000000000000000003 2023-07-29T00:25:00.886 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.822+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]: entry->timestamp=2023-07-29T00:24:55.714869+0000 2023-07-29T00:25:00.886 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.822+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.26.6 op=0 op_state=1 2023-07-29T00:25:00.886 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.822+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]: parsed entry: id=00000000002.44.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-07-29T00:25:00.886 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.822+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18/obj00000000000000000000 2023-07-29T00:25:00.886 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.822+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]: entry->timestamp=2023-07-29T00:24:55.567686+0000 2023-07-29T00:25:00.887 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.822+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.44.6 op=0 op_state=1 2023-07-29T00:25:00.887 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.822+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:entry[obj00000000000000000009]: success 2023-07-29T00:25:00.887 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.822+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12:3 marker=00000000002.78.6 timestamp=2023-07-29T00:24:55.281553+0000 2023-07-29T00:25:00.887 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.823+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:entry[obj00000000000000000003]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000003[0] 2023-07-29T00:25:00.887 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.823+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]: listing bilog for incremental sync; position=00000000002.26.6 2023-07-29T00:25:00.887 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.823+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:entry[obj00000000000000000000]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000000[0] 2023-07-29T00:25:00.887 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.823+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]: listing bilog for incremental sync; position=00000000002.44.6 2023-07-29T00:25:00.888 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.823+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:entry[obj00000000000000000007]: success 2023-07-29T00:25:00.888 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.823+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9:3 marker=00000000002.74.6 timestamp=2023-07-29T00:24:55.625568+0000 2023-07-29T00:25:00.888 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.823+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:entry[obj00000000000000000013]: success 2023-07-29T00:25:00.888 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.823+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14:3 marker=00000000002.42.6 timestamp=2023-07-29T00:24:55.453055+0000 2023-07-29T00:25:00.889 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.825+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:entry[obj00000000000000000033]: success 2023-07-29T00:25:00.889 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.825+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16:3 marker=00000000002.57.6 timestamp=2023-07-29T00:24:55.369624+0000 2023-07-29T00:25:00.889 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.827+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:entry[obj00000000000000000000]: success 2023-07-29T00:25:00.889 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.827+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18:3 marker=00000000002.44.6 timestamp=2023-07-29T00:24:55.567686+0000 2023-07-29T00:25:00.889 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.827+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:entry[obj00000000000000000003]: success 2023-07-29T00:25:00.890 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.827+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20:3 marker=00000000002.26.6 timestamp=2023-07-29T00:24:55.714869+0000 2023-07-29T00:25:00.905 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.853+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: parsed entry: id=00000000001.28.6 iter->object=obj00000000000000000058 iter->instance= name=obj00000000000000000058 instance= ns= 2023-07-29T00:25:00.905 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.853+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0/obj00000000000000000058 2023-07-29T00:25:00.905 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.853+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: entry->timestamp=2023-07-29T00:24:55.988907+0000 2023-07-29T00:25:00.905 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.853+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:entry[obj00000000000000000058]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.28.6 op=0 op_state=1 2023-07-29T00:25:00.906 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.853+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:entry[obj00000000000000000058]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000058[0] 2023-07-29T00:25:00.906 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.853+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: listing bilog for incremental sync; position=00000000001.28.6 2023-07-29T00:25:00.906 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.853+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: parsed entry: id=00000000002.38.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2023-07-29T00:25:00.906 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.853+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1/obj00000000000000000026 2023-07-29T00:25:00.907 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.853+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: entry->timestamp=2023-07-29T00:24:55.473015+0000 2023-07-29T00:25:00.907 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.853+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.38.6 op=0 op_state=1 2023-07-29T00:25:00.907 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.853+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: parsed entry: id=00000000001.58.6 iter->object=obj00000000000000000046 iter->instance= name=obj00000000000000000046 instance= ns= 2023-07-29T00:25:00.907 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.853+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3/obj00000000000000000046 2023-07-29T00:25:00.907 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.853+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: entry->timestamp=2023-07-29T00:24:55.220568+0000 2023-07-29T00:25:00.908 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.853+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:entry[obj00000000000000000046]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.58.6 op=0 op_state=1 2023-07-29T00:25:00.908 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.853+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:entry[obj00000000000000000026]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000026[0] 2023-07-29T00:25:00.908 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.853+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: listing bilog for incremental sync; position=00000000002.38.6 2023-07-29T00:25:00.908 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.853+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:entry[obj00000000000000000046]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000046[0] 2023-07-29T00:25:00.908 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.853+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: listing bilog for incremental sync; position=00000000001.58.6 2023-07-29T00:25:00.909 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.854+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: parsed entry: id=00000000001.46.6 iter->object=obj00000000000000000050 iter->instance= name=obj00000000000000000050 instance= ns= 2023-07-29T00:25:00.909 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.854+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5/obj00000000000000000050 2023-07-29T00:25:00.909 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.854+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: entry->timestamp=2023-07-29T00:24:55.908171+0000 2023-07-29T00:25:00.909 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.854+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:entry[obj00000000000000000050]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.46.6 op=0 op_state=1 2023-07-29T00:25:00.910 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.854+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:entry[obj00000000000000000050]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000050[0] 2023-07-29T00:25:00.910 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.854+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: listing bilog for incremental sync; position=00000000001.46.6 2023-07-29T00:25:00.910 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.854+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: parsed entry: id=00000000002.51.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-07-29T00:25:00.910 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.854+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2/obj00000000000000000011 2023-07-29T00:25:00.910 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.854+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: entry->timestamp=2023-07-29T00:24:55.796080+0000 2023-07-29T00:25:00.910 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.854+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.51.6 op=0 op_state=1 2023-07-29T00:25:00.910 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.854+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:entry[obj00000000000000000011]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000011[0] 2023-07-29T00:25:00.911 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.854+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: listing bilog for incremental sync; position=00000000002.51.6 2023-07-29T00:25:00.911 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.856+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: parsed entry: id=00000000002.76.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2023-07-29T00:25:00.911 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.856+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4/obj00000000000000000014 2023-07-29T00:25:00.911 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.856+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: entry->timestamp=2023-07-29T00:24:55.969029+0000 2023-07-29T00:25:00.911 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.856+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.76.6 op=0 op_state=1 2023-07-29T00:25:00.911 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.856+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: parsed entry: id=00000000001.60.6 iter->object=obj00000000000000000039 iter->instance= name=obj00000000000000000039 instance= ns= 2023-07-29T00:25:00.912 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.856+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8/obj00000000000000000039 2023-07-29T00:25:00.912 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.856+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: entry->timestamp=2023-07-29T00:24:55.646289+0000 2023-07-29T00:25:00.912 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.856+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:entry[obj00000000000000000039]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.60.6 op=0 op_state=1 2023-07-29T00:25:00.912 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.856+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:entry[obj00000000000000000014]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000014[0] 2023-07-29T00:25:00.912 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.856+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: listing bilog for incremental sync; position=00000000002.76.6 2023-07-29T00:25:00.912 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.856+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:entry[obj00000000000000000026]: success 2023-07-29T00:25:00.913 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.856+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1:3 marker=00000000002.38.6 timestamp=2023-07-29T00:24:55.473015+0000 2023-07-29T00:25:00.913 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.856+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:entry[obj00000000000000000039]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000039[0] 2023-07-29T00:25:00.913 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.856+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: listing bilog for incremental sync; position=00000000001.60.6 2023-07-29T00:25:00.914 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.858+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: parsed entry: id=00000000001.59.6 iter->object=obj00000000000000000040 iter->instance= name=obj00000000000000000040 instance= ns= 2023-07-29T00:25:00.914 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.858+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11/obj00000000000000000040 2023-07-29T00:25:00.914 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.858+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: entry->timestamp=2023-07-29T00:24:55.833791+0000 2023-07-29T00:25:00.914 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.858+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:entry[obj00000000000000000040]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.59.6 op=0 op_state=1 2023-07-29T00:25:00.914 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.859+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:entry[obj00000000000000000040]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000040[0] 2023-07-29T00:25:00.914 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.859+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: listing bilog for incremental sync; position=00000000001.59.6 2023-07-29T00:25:00.915 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.859+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:entry[obj00000000000000000011]: success 2023-07-29T00:25:00.915 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.859+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2:3 marker=00000000002.51.6 timestamp=2023-07-29T00:24:55.796080+0000 2023-07-29T00:25:00.915 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.859+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:entry[obj00000000000000000014]: success 2023-07-29T00:25:00.915 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.859+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4:3 marker=00000000002.76.6 timestamp=2023-07-29T00:24:55.969029+0000 2023-07-29T00:25:00.915 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.860+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: parsed entry: id=00000000002.35.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-07-29T00:25:00.915 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.860+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15/obj00000000000000000012 2023-07-29T00:25:00.916 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.860+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: entry->timestamp=2023-07-29T00:24:55.814873+0000 2023-07-29T00:25:00.916 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.860+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.35.6 op=0 op_state=1 2023-07-29T00:25:00.916 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.860+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:entry[obj00000000000000000012]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000012[0] 2023-07-29T00:25:00.916 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.860+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: listing bilog for incremental sync; position=00000000002.35.6 2023-07-29T00:25:00.916 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.861+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: parsed entry: id=00000000002.40.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-07-29T00:25:00.916 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.861+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10/obj00000000000000000006 2023-07-29T00:25:00.917 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.861+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: entry->timestamp=2023-07-29T00:24:55.196240+0000 2023-07-29T00:25:00.917 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.861+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.40.6 op=0 op_state=1 2023-07-29T00:25:00.917 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.861+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:entry[obj00000000000000000006]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000006[0] 2023-07-29T00:25:00.917 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.861+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: listing bilog for incremental sync; position=00000000002.40.6 2023-07-29T00:25:00.918 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.862+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]: parsed entry: id=00000000002.24.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-07-29T00:25:00.918 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.862+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19/obj00000000000000000021 2023-07-29T00:25:00.918 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.862+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]: entry->timestamp=2023-07-29T00:24:55.540146+0000 2023-07-29T00:25:00.919 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.862+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.24.6 op=0 op_state=1 2023-07-29T00:25:00.919 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.862+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:entry[obj00000000000000000021]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000021[0] 2023-07-29T00:25:00.919 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.862+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]: listing bilog for incremental sync; position=00000000002.24.6 2023-07-29T00:25:00.919 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.863+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:entry[obj00000000000000000050]: success 2023-07-29T00:25:00.919 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.863+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5:3 marker=00000000001.46.6 timestamp=2023-07-29T00:24:55.908171+0000 2023-07-29T00:25:00.919 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.863+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]: parsed entry: id=00000000002.46.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2023-07-29T00:25:00.920 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.863+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22/obj00000000000000000008 2023-07-29T00:25:00.920 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.863+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]: entry->timestamp=2023-07-29T00:24:55.889720+0000 2023-07-29T00:25:00.920 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.863+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.46.6 op=0 op_state=1 2023-07-29T00:25:00.920 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.863+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:entry[obj00000000000000000008]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000008[0] 2023-07-29T00:25:00.920 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.863+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]: listing bilog for incremental sync; position=00000000002.46.6 2023-07-29T00:25:00.920 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.864+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:entry[obj00000000000000000012]: success 2023-07-29T00:25:00.921 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.864+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15:3 marker=00000000002.35.6 timestamp=2023-07-29T00:24:55.814873+0000 2023-07-29T00:25:00.921 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.865+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:entry[obj00000000000000000021]: success 2023-07-29T00:25:00.921 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.865+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19:3 marker=00000000002.24.6 timestamp=2023-07-29T00:24:55.540146+0000 2023-07-29T00:25:00.921 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.866+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: parsed entry: id=00000000001.43.6 iter->object=obj00000000000000000055 iter->instance= name=obj00000000000000000055 instance= ns= 2023-07-29T00:25:00.921 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.866+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21/obj00000000000000000055 2023-07-29T00:25:00.921 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.866+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: entry->timestamp=2023-07-29T00:24:55.734741+0000 2023-07-29T00:25:00.921 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.866+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:entry[obj00000000000000000055]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.43.6 op=0 op_state=1 2023-07-29T00:25:00.922 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.866+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:entry[obj00000000000000000055]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000055[0] 2023-07-29T00:25:00.922 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.866+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: listing bilog for incremental sync; position=00000000001.43.6 2023-07-29T00:25:00.922 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.867+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:entry[obj00000000000000000006]: success 2023-07-29T00:25:00.923 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.867+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10:3 marker=00000000002.40.6 timestamp=2023-07-29T00:24:55.196240+0000 2023-07-29T00:25:00.923 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.869+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:entry[obj00000000000000000045]: success 2023-07-29T00:25:00.923 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.870+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13:3 marker=00000000001.55.6 timestamp=2023-07-29T00:24:55.303471+0000 2023-07-29T00:25:00.953 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.900+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:entry[obj00000000000000000046]: success 2023-07-29T00:25:00.953 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.900+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3:3 marker=00000000001.58.6 timestamp=2023-07-29T00:24:55.220568+0000 2023-07-29T00:25:00.953 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.902+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:entry[obj00000000000000000058]: success 2023-07-29T00:25:00.954 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.902+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0:3 marker=00000000001.28.6 timestamp=2023-07-29T00:24:55.988907+0000 2023-07-29T00:25:00.956 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.904+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:entry[obj00000000000000000039]: success 2023-07-29T00:25:00.956 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.904+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8:3 marker=00000000001.60.6 timestamp=2023-07-29T00:24:55.646289+0000 2023-07-29T00:25:00.958 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.907+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:entry[obj00000000000000000040]: success 2023-07-29T00:25:00.959 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.907+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11:3 marker=00000000001.59.6 timestamp=2023-07-29T00:24:55.833791+0000 2023-07-29T00:25:00.963 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.911+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:entry[obj00000000000000000008]: success 2023-07-29T00:25:00.963 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.911+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22:3 marker=00000000002.46.6 timestamp=2023-07-29T00:24:55.889720+0000 2023-07-29T00:25:00.964 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.913+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:entry[obj00000000000000000055]: success 2023-07-29T00:25:00.965 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.913+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21:3 marker=00000000001.43.6 timestamp=2023-07-29T00:24:55.734741+0000 2023-07-29T00:25:00.995 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.943+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.996 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.943+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.996 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.943+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.996 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.943+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:00.996 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.943+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: sync status for source bucket shard: 0 2023-07-29T00:25:00.997 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.943+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: sync status for source bucket shard: 0 2023-07-29T00:25:00.997 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.943+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.997 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.943+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.997 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.943+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: sync status for source bucket shard: 0 2023-07-29T00:25:00.999 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.944+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: listing bilog for incremental sync; position= 2023-07-29T00:25:00.999 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.944+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.000 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.944+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: sync status for source bucket shard: 0 2023-07-29T00:25:01.000 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.944+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.000 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.944+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.000 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.944+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.000 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.945+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.000 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.946+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: sync status for source bucket shard: 0 2023-07-29T00:25:01.001 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.946+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.001 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.946+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.001 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.946+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.001 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.946+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.001 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.946+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.001 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.946+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.001 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.946+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: sync status for source bucket shard: 0 2023-07-29T00:25:01.002 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.946+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: sync status for source bucket shard: 0 2023-07-29T00:25:01.002 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.946+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.002 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.946+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.002 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.946+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.002 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.946+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.002 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.946+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: parsed entry: id=00000000002.91.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-07-29T00:25:01.002 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.946+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5/obj00000000000000000002 2023-07-29T00:25:01.003 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.947+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: entry->timestamp=2023-07-29T00:24:57.087288+0000 2023-07-29T00:25:01.003 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.947+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.91.6 op=0 op_state=1 2023-07-29T00:25:01.003 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.947+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.003 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.947+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: parsed entry: id=00000000001.66.6 iter->object=obj00000000000000000059 iter->instance= name=obj00000000000000000059 instance= ns= 2023-07-29T00:25:01.003 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.947+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1/obj00000000000000000059 2023-07-29T00:25:01.003 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.947+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: entry->timestamp=2023-07-29T00:24:56.419498+0000 2023-07-29T00:25:01.004 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.947+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:entry[obj00000000000000000059]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.66.6 op=0 op_state=1 2023-07-29T00:25:01.004 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.947+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: parsed entry: id=00000000002.93.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-07-29T00:25:01.004 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.947+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0/obj00000000000000000006 2023-07-29T00:25:01.004 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.947+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: entry->timestamp=2023-07-29T00:24:57.324357+0000 2023-07-29T00:25:01.004 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.947+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.93.6 op=0 op_state=1 2023-07-29T00:25:01.004 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.947+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.004 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.947+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: parsed entry: id=00000000001.81.6 iter->object=obj00000000000000000027 iter->instance= name=obj00000000000000000027 instance= ns= 2023-07-29T00:25:01.005 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.947+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2/obj00000000000000000027 2023-07-29T00:25:01.005 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.947+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: entry->timestamp=2023-07-29T00:24:57.170121+0000 2023-07-29T00:25:01.005 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.947+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:entry[obj00000000000000000027]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.81.6 op=0 op_state=1 2023-07-29T00:25:01.005 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.947+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:entry[obj00000000000000000002]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000002[0] 2023-07-29T00:25:01.005 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.947+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: listing bilog for incremental sync; position=00000000002.91.6 2023-07-29T00:25:01.005 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.947+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:entry[obj00000000000000000059]: : injecting data sync error on key=obj00000000000000000059 2023-07-29T00:25:01.006 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.947+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: listing bilog for incremental sync; position=00000000001.66.6 2023-07-29T00:25:01.006 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.947+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:entry[obj00000000000000000006]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000006[0] 2023-07-29T00:25:01.006 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.947+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: listing bilog for incremental sync; position=00000000002.93.6 2023-07-29T00:25:01.006 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.947+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:entry[obj00000000000000000027]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000027[0] 2023-07-29T00:25:01.006 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.947+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: listing bilog for incremental sync; position=00000000001.81.6 2023-07-29T00:25:01.009 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.947+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:entry[obj00000000000000000059]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:25:01.009 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.947+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:entry[obj00000000000000000059]: ERROR: failed to sync object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1/obj00000000000000000059 2023-07-29T00:25:01.009 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.947+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.009 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.948+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: sync status for source bucket shard: 0 2023-07-29T00:25:01.009 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.949+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: sync status for source bucket shard: 0 2023-07-29T00:25:01.010 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.949+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: sync status for source bucket shard: 0 2023-07-29T00:25:01.010 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.949+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: sync status for source bucket shard: 0 2023-07-29T00:25:01.010 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.949+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.010 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.949+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.010 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.949+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.010 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.949+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.011 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.949+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.011 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.949+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.011 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.949+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: sync status for source bucket shard: 0 2023-07-29T00:25:01.011 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.949+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: sync status for source bucket shard: 0 2023-07-29T00:25:01.011 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.949+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.011 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.949+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.011 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.949+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.012 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.949+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.012 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.949+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.012 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.949+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: sync status for source bucket shard: 0 2023-07-29T00:25:01.012 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.949+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.012 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.949+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.012 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.950+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.013 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.951+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]: sync status for source bucket shard: 0 2023-07-29T00:25:01.013 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.951+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.013 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.951+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]: sync status for source bucket shard: 0 2023-07-29T00:25:01.013 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.951+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.013 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.951+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]: sync status for source bucket shard: 0 2023-07-29T00:25:01.013 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.953+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.013 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:entry[obj00000000000000000002]: success 2023-07-29T00:25:01.014 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5:4 marker=00000000002.91.6 timestamp=2023-07-29T00:24:57.087288+0000 2023-07-29T00:25:01.014 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]: sync status for source bucket shard: 0 2023-07-29T00:25:01.014 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: sync status for source bucket shard: 0 2023-07-29T00:25:01.014 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.014 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.014 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: sync status for source bucket shard: 0 2023-07-29T00:25:01.015 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]: sync status for source bucket shard: 0 2023-07-29T00:25:01.015 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]: sync status for source bucket shard: 0 2023-07-29T00:25:01.015 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: sync status for source bucket shard: 0 2023-07-29T00:25:01.015 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: sync status for source bucket shard: 0 2023-07-29T00:25:01.015 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]: sync status for source bucket shard: 0 2023-07-29T00:25:01.015 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.015 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.016 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.016 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.016 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.016 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.016 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.016 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.017 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.017 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: a sync operation returned error 2023-07-29T00:25:01.017 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:entry[obj00000000000000000006]: success 2023-07-29T00:25:01.017 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0:4 marker=00000000002.93.6 timestamp=2023-07-29T00:24:57.324357+0000 2023-07-29T00:25:01.017 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: parsed entry: id=00000000002.29.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2023-07-29T00:25:01.017 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9/obj00000000000000000008 2023-07-29T00:25:01.018 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: entry->timestamp=2023-07-29T00:24:56.320579+0000 2023-07-29T00:25:01.018 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.29.6 op=0 op_state=1 2023-07-29T00:25:01.018 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: parsed entry: id=00000000001.31.6 iter->object=obj00000000000000000031 iter->instance= name=obj00000000000000000031 instance= ns= 2023-07-29T00:25:01.018 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11/obj00000000000000000031 2023-07-29T00:25:01.018 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: entry->timestamp=2023-07-29T00:24:57.013288+0000 2023-07-29T00:25:01.018 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:entry[obj00000000000000000031]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.31.6 op=0 op_state=1 2023-07-29T00:25:01.018 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.019 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: parsed entry: id=00000000002.43.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2023-07-29T00:25:01.019 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13/obj00000000000000000013 2023-07-29T00:25:01.019 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: entry->timestamp=2023-07-29T00:24:56.666413+0000 2023-07-29T00:25:01.019 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.43.6 op=0 op_state=1 2023-07-29T00:25:01.019 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]: parsed entry: id=00000000002.51.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-07-29T00:25:01.019 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17/obj00000000000000000012 2023-07-29T00:25:01.019 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]: entry->timestamp=2023-07-29T00:24:56.994653+0000 2023-07-29T00:25:01.020 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.51.6 op=0 op_state=1 2023-07-29T00:25:01.020 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:entry[obj00000000000000000008]: : injecting data sync error on key=obj00000000000000000008 2023-07-29T00:25:01.020 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: listing bilog for incremental sync; position=00000000002.29.6 2023-07-29T00:25:01.020 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:entry[obj00000000000000000031]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000031[0] 2023-07-29T00:25:01.020 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: listing bilog for incremental sync; position=00000000001.31.6 2023-07-29T00:25:01.020 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:entry[obj00000000000000000013]: : injecting data sync error on key=obj00000000000000000013 2023-07-29T00:25:01.021 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: listing bilog for incremental sync; position=00000000002.43.6 2023-07-29T00:25:01.021 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:entry[obj00000000000000000012]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000012[0] 2023-07-29T00:25:01.021 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]: listing bilog for incremental sync; position=00000000002.51.6 2023-07-29T00:25:01.021 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: backing out with sync_status=-5 2023-07-29T00:25:01.021 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:entry[obj00000000000000000008]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:25:01.021 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.954+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:entry[obj00000000000000000008]: ERROR: failed to sync object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9/obj00000000000000000008 2023-07-29T00:25:01.022 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.955+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:entry[obj00000000000000000013]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:25:01.022 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.955+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:entry[obj00000000000000000013]: ERROR: failed to sync object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13/obj00000000000000000013 2023-07-29T00:25:01.022 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.956+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: incremental sync on bucket failed, retcode=-5 2023-07-29T00:25:01.022 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.957+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: ERROR: incremental sync failed. error: -5 2023-07-29T00:25:01.023 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.959+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]: sync status for source bucket shard: 0 2023-07-29T00:25:01.024 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.959+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]: sync status for source bucket shard: 0 2023-07-29T00:25:01.024 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.959+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]: sync status for source bucket shard: 0 2023-07-29T00:25:01.024 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.959+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.024 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.959+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.024 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.959+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.024 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.959+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]: sync status for source bucket shard: 0 2023-07-29T00:25:01.025 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.959+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.025 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.959+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: parsed entry: id=00000000002.31.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-07-29T00:25:01.025 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.959+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21/obj00000000000000000001 2023-07-29T00:25:01.025 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.959+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: entry->timestamp=2023-07-29T00:24:56.300706+0000 2023-07-29T00:25:01.025 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.959+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.31.6 op=0 op_state=1 2023-07-29T00:25:01.025 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.959+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:entry[obj00000000000000000001]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000001[0] 2023-07-29T00:25:01.026 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.959+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: listing bilog for incremental sync; position=00000000002.31.6 2023-07-29T00:25:01.026 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.959+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: parsed entry: id=00000000001.89.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2023-07-29T00:25:01.026 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.959+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8/obj00000000000000000017 2023-07-29T00:25:01.026 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.959+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: entry->timestamp=2023-07-29T00:24:56.606139+0000 2023-07-29T00:25:01.026 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.959+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.89.6 op=0 op_state=1 2023-07-29T00:25:01.026 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.959+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]: parsed entry: id=00000000002.52.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-07-29T00:25:01.026 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.959+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22/obj00000000000000000009 2023-07-29T00:25:01.027 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.959+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]: entry->timestamp=2023-07-29T00:24:57.244560+0000 2023-07-29T00:25:01.027 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.959+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.52.6 op=0 op_state=1 2023-07-29T00:25:01.027 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.959+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:entry[obj00000000000000000017]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000017[0] 2023-07-29T00:25:01.027 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.959+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: listing bilog for incremental sync; position=00000000001.89.6 2023-07-29T00:25:01.027 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.959+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:entry[obj00000000000000000009]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000009[0] 2023-07-29T00:25:01.027 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.959+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]: listing bilog for incremental sync; position=00000000002.52.6 2023-07-29T00:25:01.028 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.962+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]: parsed entry: id=00000000002.45.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-07-29T00:25:01.028 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.962+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28/obj00000000000000000021 2023-07-29T00:25:01.028 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.962+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]: entry->timestamp=2023-07-29T00:24:56.586907+0000 2023-07-29T00:25:01.028 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.962+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.45.6 op=0 op_state=1 2023-07-29T00:25:01.028 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.962+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:entry[obj00000000000000000021]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000021[0] 2023-07-29T00:25:01.028 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.962+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]: listing bilog for incremental sync; position=00000000002.45.6 2023-07-29T00:25:01.029 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.962+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]: parsed entry: id=00000000002.69.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2023-07-29T00:25:01.029 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.962+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26/obj00000000000000000014 2023-07-29T00:25:01.029 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.962+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]: entry->timestamp=2023-07-29T00:24:56.763798+0000 2023-07-29T00:25:01.029 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.962+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.69.6 op=0 op_state=1 2023-07-29T00:25:01.029 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.963+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]:entry[obj00000000000000000014]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000014[0] 2023-07-29T00:25:01.029 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.963+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]: listing bilog for incremental sync; position=00000000002.69.6 2023-07-29T00:25:01.029 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.965+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:entry[obj00000000000000000012]: success 2023-07-29T00:25:01.030 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.965+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17:4 marker=00000000002.51.6 timestamp=2023-07-29T00:24:56.994653+0000 2023-07-29T00:25:01.030 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.965+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]: parsed entry: id=00000000002.33.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2023-07-29T00:25:01.030 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.965+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27/obj00000000000000000030 2023-07-29T00:25:01.030 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.965+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]: entry->timestamp=2023-07-29T00:24:56.340337+0000 2023-07-29T00:25:01.031 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.965+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.33.6 op=0 op_state=1 2023-07-29T00:25:01.031 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.965+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]: parsed entry: id=00000000002.70.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-07-29T00:25:01.031 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.965+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25/obj00000000000000000000 2023-07-29T00:25:01.031 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.965+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]: entry->timestamp=2023-07-29T00:24:57.264830+0000 2023-07-29T00:25:01.031 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.965+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.70.6 op=0 op_state=1 2023-07-29T00:25:01.031 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.965+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:entry[obj00000000000000000030]: : injecting data sync error on key=obj00000000000000000030 2023-07-29T00:25:01.031 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.965+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]: listing bilog for incremental sync; position=00000000002.33.6 2023-07-29T00:25:01.032 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.965+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:entry[obj00000000000000000009]: success 2023-07-29T00:25:01.032 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.965+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22:4 marker=00000000002.52.6 timestamp=2023-07-29T00:24:57.244560+0000 2023-07-29T00:25:01.032 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.965+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]:entry[obj00000000000000000000]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000000[0] 2023-07-29T00:25:01.032 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.965+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]: listing bilog for incremental sync; position=00000000002.70.6 2023-07-29T00:25:01.032 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.965+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:entry[obj00000000000000000001]: success 2023-07-29T00:25:01.032 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.965+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21:4 marker=00000000002.31.6 timestamp=2023-07-29T00:24:56.300706+0000 2023-07-29T00:25:01.033 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.965+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:entry[obj00000000000000000030]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:25:01.033 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.965+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:entry[obj00000000000000000030]: ERROR: failed to sync object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27/obj00000000000000000030 2023-07-29T00:25:01.033 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.967+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:entry[obj00000000000000000021]: success 2023-07-29T00:25:01.033 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.967+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28:4 marker=00000000002.45.6 timestamp=2023-07-29T00:24:56.586907+0000 2023-07-29T00:25:01.033 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.968+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.033 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.968+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]:entry[obj00000000000000000014]: success 2023-07-29T00:25:01.034 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.968+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26:4 marker=00000000002.69.6 timestamp=2023-07-29T00:24:56.763798+0000 2023-07-29T00:25:01.034 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.969+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: sync status for source bucket shard: 0 2023-07-29T00:25:01.034 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.969+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.034 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.970+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]:entry[obj00000000000000000000]: success 2023-07-29T00:25:01.034 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.970+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25:4 marker=00000000002.70.6 timestamp=2023-07-29T00:24:57.264830+0000 2023-07-29T00:25:01.035 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.972+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:entry[obj00000000000000000017]: success 2023-07-29T00:25:01.035 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.972+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8:4 marker=00000000001.89.6 timestamp=2023-07-29T00:24:56.606139+0000 2023-07-29T00:25:01.041 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.989+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: parsed entry: id=00000000001.55.6 iter->object=obj00000000000000000049 iter->instance= name=obj00000000000000000049 instance= ns= 2023-07-29T00:25:01.041 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.989+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7/obj00000000000000000049 2023-07-29T00:25:01.041 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.989+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: entry->timestamp=2023-07-29T00:24:56.685390+0000 2023-07-29T00:25:01.041 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.989+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:entry[obj00000000000000000049]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.55.6 op=0 op_state=1 2023-07-29T00:25:01.042 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.989+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:entry[obj00000000000000000049]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000049[0] 2023-07-29T00:25:01.042 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.989+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: listing bilog for incremental sync; position=00000000001.55.6 2023-07-29T00:25:01.042 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.989+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: parsed entry: id=00000000002.56.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-07-29T00:25:01.042 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.989+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3/obj00000000000000000004 2023-07-29T00:25:01.042 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.989+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: entry->timestamp=2023-07-29T00:24:56.744196+0000 2023-07-29T00:25:01.042 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.989+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.56.6 op=0 op_state=1 2023-07-29T00:25:01.043 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.989+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:entry[obj00000000000000000004]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000004[0] 2023-07-29T00:25:01.043 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.989+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: listing bilog for incremental sync; position=00000000002.56.6 2023-07-29T00:25:01.043 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.990+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: parsed entry: id=00000000002.82.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-07-29T00:25:01.043 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.990+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6/obj00000000000000000019 2023-07-29T00:25:01.044 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.990+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: entry->timestamp=2023-07-29T00:24:56.820858+0000 2023-07-29T00:25:01.044 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.990+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.82.6 op=0 op_state=1 2023-07-29T00:25:01.044 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.990+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:entry[obj00000000000000000019]: : injecting data sync error on key=obj00000000000000000019 2023-07-29T00:25:01.044 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.990+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: listing bilog for incremental sync; position=00000000002.82.6 2023-07-29T00:25:01.044 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.990+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:entry[obj00000000000000000019]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:25:01.044 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.990+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:entry[obj00000000000000000019]: ERROR: failed to sync object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6/obj00000000000000000019 2023-07-29T00:25:01.045 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.991+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:entry[obj00000000000000000004]: success 2023-07-29T00:25:01.045 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.991+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3:4 marker=00000000002.56.6 timestamp=2023-07-29T00:24:56.744196+0000 2023-07-29T00:25:01.045 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.993+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: parsed entry: id=00000000002.41.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2023-07-29T00:25:01.045 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.993+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10/obj00000000000000000015 2023-07-29T00:25:01.046 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.993+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: entry->timestamp=2023-07-29T00:24:56.529463+0000 2023-07-29T00:25:01.046 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.993+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.41.6 op=0 op_state=1 2023-07-29T00:25:01.046 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.993+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:entry[obj00000000000000000015]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000015[0] 2023-07-29T00:25:01.046 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.993+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: listing bilog for incremental sync; position=00000000002.41.6 2023-07-29T00:25:01.047 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.994+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:entry[obj00000000000000000015]: success 2023-07-29T00:25:01.047 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.994+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10:4 marker=00000000002.41.6 timestamp=2023-07-29T00:24:56.529463+0000 2023-07-29T00:25:01.047 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.995+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: parsed entry: id=00000000002.74.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2023-07-29T00:25:01.047 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.995+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16/obj00000000000000000005 2023-07-29T00:25:01.048 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.995+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: entry->timestamp=2023-07-29T00:24:57.068776+0000 2023-07-29T00:25:01.048 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.995+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.74.6 op=0 op_state=1 2023-07-29T00:25:01.048 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.995+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:entry[obj00000000000000000005]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000005[0] 2023-07-29T00:25:01.048 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.995+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: listing bilog for incremental sync; position=00000000002.74.6 2023-07-29T00:25:01.048 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.995+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]: parsed entry: id=00000000001.84.6 iter->object=obj00000000000000000042 iter->instance= name=obj00000000000000000042 instance= ns= 2023-07-29T00:25:01.048 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.995+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19/obj00000000000000000042 2023-07-29T00:25:01.049 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.995+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]: entry->timestamp=2023-07-29T00:24:56.845159+0000 2023-07-29T00:25:01.049 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.995+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:entry[obj00000000000000000042]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.84.6 op=0 op_state=1 2023-07-29T00:25:01.049 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.995+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:entry[obj00000000000000000042]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000042[0] 2023-07-29T00:25:01.049 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.995+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]: listing bilog for incremental sync; position=00000000001.84.6 2023-07-29T00:25:01.050 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.996+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]: parsed entry: id=00000000002.54.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-07-29T00:25:01.051 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.996+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24/obj00000000000000000016 2023-07-29T00:25:01.051 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.996+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]: entry->timestamp=2023-07-29T00:24:56.359684+0000 2023-07-29T00:25:01.051 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.996+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.54.6 op=0 op_state=1 2023-07-29T00:25:01.051 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.996+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]:entry[obj00000000000000000016]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000016[0] 2023-07-29T00:25:01.051 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.996+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]: listing bilog for incremental sync; position=00000000002.54.6 2023-07-29T00:25:01.052 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.998+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]: parsed entry: id=00000000002.51.6 iter->object=obj00000000000000000025 iter->instance= name=obj00000000000000000025 instance= ns= 2023-07-29T00:25:01.052 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.998+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23/obj00000000000000000025 2023-07-29T00:25:01.052 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.998+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]: entry->timestamp=2023-07-29T00:24:56.911728+0000 2023-07-29T00:25:01.052 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.998+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]:entry[obj00000000000000000025]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.51.6 op=0 op_state=1 2023-07-29T00:25:01.052 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.998+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]:entry[obj00000000000000000025]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000025[0] 2023-07-29T00:25:01.052 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.998+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]: listing bilog for incremental sync; position=00000000002.51.6 2023-07-29T00:25:01.052 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.998+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: parsed entry: id=00000000002.21.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2023-07-29T00:25:01.053 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.998+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14/obj00000000000000000020 2023-07-29T00:25:01.053 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.998+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: entry->timestamp=2023-07-29T00:24:57.144955+0000 2023-07-29T00:25:01.053 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.998+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.21.6 op=0 op_state=1 2023-07-29T00:25:01.053 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.998+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: parsed entry: id=00000000002.47.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-07-29T00:25:01.053 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.998+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4/obj00000000000000000011 2023-07-29T00:25:01.053 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.998+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: entry->timestamp=2023-07-29T00:24:56.936676+0000 2023-07-29T00:25:01.054 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.998+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.47.6 op=0 op_state=1 2023-07-29T00:25:01.054 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.998+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]: parsed entry: id=00000000002.68.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-07-29T00:25:01.054 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.998+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18/obj00000000000000000003 2023-07-29T00:25:01.054 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.998+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]: entry->timestamp=2023-07-29T00:24:56.507140+0000 2023-07-29T00:25:01.054 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.998+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.68.6 op=0 op_state=1 2023-07-29T00:25:01.054 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.998+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]: parsed entry: id=00000000002.19.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-07-29T00:25:01.054 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.998+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20/obj00000000000000000010 2023-07-29T00:25:01.055 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.998+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]: entry->timestamp=2023-07-29T00:24:56.442832+0000 2023-07-29T00:25:01.055 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.998+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.19.6 op=0 op_state=1 2023-07-29T00:25:01.055 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.998+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:entry[obj00000000000000000005]: success 2023-07-29T00:25:01.055 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.998+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16:4 marker=00000000002.74.6 timestamp=2023-07-29T00:24:57.068776+0000 2023-07-29T00:25:01.055 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.998+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:entry[obj00000000000000000020]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000020[0] 2023-07-29T00:25:01.055 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.998+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: listing bilog for incremental sync; position=00000000002.21.6 2023-07-29T00:25:01.056 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.998+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:entry[obj00000000000000000011]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000011[0] 2023-07-29T00:25:01.056 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.998+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: listing bilog for incremental sync; position=00000000002.47.6 2023-07-29T00:25:01.056 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.998+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:entry[obj00000000000000000003]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000003[0] 2023-07-29T00:25:01.056 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.998+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]: listing bilog for incremental sync; position=00000000002.68.6 2023-07-29T00:25:01.056 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.998+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:entry[obj00000000000000000010]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000010[0] 2023-07-29T00:25:01.056 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:00.998+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]: listing bilog for incremental sync; position=00000000002.19.6 2023-07-29T00:25:01.057 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.001+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]:entry[obj00000000000000000016]: success 2023-07-29T00:25:01.057 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.001+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24:4 marker=00000000002.54.6 timestamp=2023-07-29T00:24:56.359684+0000 2023-07-29T00:25:01.058 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.002+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: a sync operation returned error 2023-07-29T00:25:01.058 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.002+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: backing out with sync_status=-5 2023-07-29T00:25:01.058 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.002+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:entry[obj00000000000000000020]: success 2023-07-29T00:25:01.058 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.002+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14:4 marker=00000000002.21.6 timestamp=2023-07-29T00:24:57.144955+0000 2023-07-29T00:25:01.058 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.002+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:entry[obj00000000000000000011]: success 2023-07-29T00:25:01.058 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.002+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4:4 marker=00000000002.47.6 timestamp=2023-07-29T00:24:56.936676+0000 2023-07-29T00:25:01.059 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.002+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: incremental sync on bucket failed, retcode=-5 2023-07-29T00:25:01.059 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.002+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: ERROR: incremental sync failed. error: -5 2023-07-29T00:25:01.059 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.002+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]:entry[obj00000000000000000025]: success 2023-07-29T00:25:01.059 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.002+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23:4 marker=00000000002.51.6 timestamp=2023-07-29T00:24:56.911728+0000 2023-07-29T00:25:01.059 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.002+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:entry[obj00000000000000000027]: success 2023-07-29T00:25:01.059 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.002+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2:4 marker=00000000001.81.6 timestamp=2023-07-29T00:24:57.170121+0000 2023-07-29T00:25:01.060 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.002+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:entry[obj00000000000000000010]: success 2023-07-29T00:25:01.060 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.002+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20:4 marker=00000000002.19.6 timestamp=2023-07-29T00:24:56.442832+0000 2023-07-29T00:25:01.060 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.003+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:entry[obj00000000000000000003]: success 2023-07-29T00:25:01.060 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.003+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18:4 marker=00000000002.68.6 timestamp=2023-07-29T00:24:56.507140+0000 2023-07-29T00:25:01.060 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.004+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: a sync operation returned error 2023-07-29T00:25:01.060 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.004+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: backing out with sync_status=-5 2023-07-29T00:25:01.060 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.004+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: incremental sync on bucket failed, retcode=-5 2023-07-29T00:25:01.061 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.004+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: ERROR: incremental sync failed. error: -5 2023-07-29T00:25:01.061 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.006+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.062 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.006+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.062 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.006+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:entry[obj00000000000000000031]: success 2023-07-29T00:25:01.062 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.006+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11:4 marker=00000000001.31.6 timestamp=2023-07-29T00:24:57.013288+0000 2023-07-29T00:25:01.062 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.007+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: sync status for source bucket shard: 0 2023-07-29T00:25:01.062 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.007+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.062 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.007+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: sync status for source bucket shard: 0 2023-07-29T00:25:01.063 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.007+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.063 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.010+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]: a sync operation returned error 2023-07-29T00:25:01.063 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.010+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]: backing out with sync_status=-5 2023-07-29T00:25:01.063 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.010+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]: incremental sync on bucket failed, retcode=-5 2023-07-29T00:25:01.063 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.010+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]: ERROR: incremental sync failed. error: -5 2023-07-29T00:25:01.064 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.011+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: parsed entry: id=00000000001.66.6 iter->object=obj00000000000000000059 iter->instance= name=obj00000000000000000059 instance= ns= 2023-07-29T00:25:01.064 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.011+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1/obj00000000000000000059 2023-07-29T00:25:01.064 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.011+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: entry->timestamp=2023-07-29T00:24:56.419498+0000 2023-07-29T00:25:01.065 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.011+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:entry[obj00000000000000000059]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.66.6 op=0 op_state=1 2023-07-29T00:25:01.065 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.011+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:entry[obj00000000000000000059]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000059[0] 2023-07-29T00:25:01.065 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.011+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: listing bilog for incremental sync; position=00000000001.66.6 2023-07-29T00:25:01.065 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.011+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.065 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.012+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]: sync status for source bucket shard: 0 2023-07-29T00:25:01.065 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.012+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.084 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.032+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: a sync operation returned error 2023-07-29T00:25:01.084 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.032+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: backing out with sync_status=-5 2023-07-29T00:25:01.084 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.032+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: incremental sync on bucket failed, retcode=-5 2023-07-29T00:25:01.084 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.032+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: ERROR: incremental sync failed. error: -5 2023-07-29T00:25:01.085 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.032+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.085 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.033+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: sync status for source bucket shard: 0 2023-07-29T00:25:01.085 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.033+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.087 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.035+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:entry[obj00000000000000000049]: success 2023-07-29T00:25:01.087 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.035+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7:4 marker=00000000001.55.6 timestamp=2023-07-29T00:24:56.685390+0000 2023-07-29T00:25:01.095 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.043+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:entry[obj00000000000000000042]: success 2023-07-29T00:25:01.095 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.043+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19:4 marker=00000000001.84.6 timestamp=2023-07-29T00:24:56.845159+0000 2023-07-29T00:25:01.102 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.050+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: parsed entry: id=00000000002.29.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2023-07-29T00:25:01.102 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.050+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9/obj00000000000000000008 2023-07-29T00:25:01.102 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.050+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: entry->timestamp=2023-07-29T00:24:56.320579+0000 2023-07-29T00:25:01.102 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.050+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.29.6 op=0 op_state=1 2023-07-29T00:25:01.103 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.050+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:entry[obj00000000000000000008]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000008[0] 2023-07-29T00:25:01.103 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.050+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: listing bilog for incremental sync; position=00000000002.29.6 2023-07-29T00:25:01.103 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.050+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: parsed entry: id=00000000002.43.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2023-07-29T00:25:01.103 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.050+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13/obj00000000000000000013 2023-07-29T00:25:01.103 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.050+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: entry->timestamp=2023-07-29T00:24:56.666413+0000 2023-07-29T00:25:01.103 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.050+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.43.6 op=0 op_state=1 2023-07-29T00:25:01.104 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.050+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:entry[obj00000000000000000013]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000013[0] 2023-07-29T00:25:01.104 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.050+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: listing bilog for incremental sync; position=00000000002.43.6 2023-07-29T00:25:01.104 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.052+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:entry[obj00000000000000000013]: success 2023-07-29T00:25:01.104 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.052+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13:4 marker=00000000002.43.6 timestamp=2023-07-29T00:24:56.666413+0000 2023-07-29T00:25:01.105 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.053+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:entry[obj00000000000000000008]: success 2023-07-29T00:25:01.105 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.053+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9:4 marker=00000000002.29.6 timestamp=2023-07-29T00:24:56.320579+0000 2023-07-29T00:25:01.108 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.056+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]: parsed entry: id=00000000002.33.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2023-07-29T00:25:01.108 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.056+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27/obj00000000000000000030 2023-07-29T00:25:01.108 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.056+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]: entry->timestamp=2023-07-29T00:24:56.340337+0000 2023-07-29T00:25:01.108 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.056+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.33.6 op=0 op_state=1 2023-07-29T00:25:01.109 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.056+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:entry[obj00000000000000000030]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000030[0] 2023-07-29T00:25:01.109 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.056+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]: listing bilog for incremental sync; position=00000000002.33.6 2023-07-29T00:25:01.109 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.057+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:entry[obj00000000000000000059]: success 2023-07-29T00:25:01.109 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.057+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1:4 marker=00000000001.66.6 timestamp=2023-07-29T00:24:56.419498+0000 2023-07-29T00:25:01.110 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.058+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:entry[obj00000000000000000030]: success 2023-07-29T00:25:01.110 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.058+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27:4 marker=00000000002.33.6 timestamp=2023-07-29T00:24:56.340337+0000 2023-07-29T00:25:01.129 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.077+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: parsed entry: id=00000000002.82.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-07-29T00:25:01.129 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.077+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6/obj00000000000000000019 2023-07-29T00:25:01.129 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.077+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: entry->timestamp=2023-07-29T00:24:56.820858+0000 2023-07-29T00:25:01.130 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.077+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.82.6 op=0 op_state=1 2023-07-29T00:25:01.130 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.077+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:entry[obj00000000000000000019]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000019[0] 2023-07-29T00:25:01.130 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.077+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: listing bilog for incremental sync; position=00000000002.82.6 2023-07-29T00:25:01.131 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.079+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:entry[obj00000000000000000019]: success 2023-07-29T00:25:01.131 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.079+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6:4 marker=00000000002.82.6 timestamp=2023-07-29T00:24:56.820858+0000 2023-07-29T00:25:01.176 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.123+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.176 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.123+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.176 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.123+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.176 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.123+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: sync status for source bucket shard: 0 2023-07-29T00:25:01.176 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.123+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.177 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.123+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.177 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.123+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: sync status for source bucket shard: 0 2023-07-29T00:25:01.177 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.124+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: sync status for source bucket shard: 0 2023-07-29T00:25:01.177 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.124+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.177 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.124+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.177 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.124+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.179 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.124+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.179 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.124+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.179 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.124+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.180 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.124+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: sync status for source bucket shard: 0 2023-07-29T00:25:01.180 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.124+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.180 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.124+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: sync status for source bucket shard: 0 2023-07-29T00:25:01.180 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.124+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.180 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.125+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: sync status for source bucket shard: 0 2023-07-29T00:25:01.180 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.125+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.181 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.125+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: sync status for source bucket shard: 0 2023-07-29T00:25:01.181 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.125+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: sync status for source bucket shard: 0 2023-07-29T00:25:01.181 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.125+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.181 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.125+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.181 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.125+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.181 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.126+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: sync status for source bucket shard: 0 2023-07-29T00:25:01.181 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.126+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.182 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.126+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.182 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.126+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.182 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.126+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.182 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.126+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.184 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.127+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: parsed entry: id=00000000001.88.6 iter->object=obj00000000000000000317 iter->instance= name=obj00000000000000000317 instance= ns= 2023-07-29T00:25:01.184 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.127+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7/obj00000000000000000317 2023-07-29T00:25:01.184 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.127+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: entry->timestamp=2023-07-29T00:24:58.614561+0000 2023-07-29T00:25:01.184 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.127+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:entry[obj00000000000000000317]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.88.6 op=0 op_state=1 2023-07-29T00:25:01.184 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.127+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.185 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.127+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.185 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.127+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: parsed entry: id=00000000001.45.6 iter->object=obj00000000000000000022 iter->instance= name=obj00000000000000000022 instance= ns= 2023-07-29T00:25:01.185 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.127+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8/obj00000000000000000022 2023-07-29T00:25:01.185 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.127+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: entry->timestamp=2023-07-29T00:24:58.700216+0000 2023-07-29T00:25:01.185 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.127+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:entry[obj00000000000000000022]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.45.6 op=0 op_state=1 2023-07-29T00:25:01.185 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.127+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: parsed entry: id=00000000002.29.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-07-29T00:25:01.185 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.127+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5/obj00000000000000000004 2023-07-29T00:25:01.186 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.127+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: entry->timestamp=2023-07-29T00:24:58.051099+0000 2023-07-29T00:25:01.186 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.127+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.29.6 op=0 op_state=1 2023-07-29T00:25:01.186 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.127+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: parsed entry: id=00000000001.60.6 iter->object=obj00000000000000000467 iter->instance= name=obj00000000000000000467 instance= ns= 2023-07-29T00:25:01.186 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.127+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2/obj00000000000000000467 2023-07-29T00:25:01.186 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.127+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: entry->timestamp=2023-07-29T00:24:58.460850+0000 2023-07-29T00:25:01.186 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.127+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:entry[obj00000000000000000467]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.60.6 op=0 op_state=1 2023-07-29T00:25:01.187 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.127+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.187 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.127+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:entry[obj00000000000000000317]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000317[0] 2023-07-29T00:25:01.187 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.127+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: listing bilog for incremental sync; position=00000000001.88.6 2023-07-29T00:25:01.187 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.127+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:entry[obj00000000000000000022]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000022[0] 2023-07-29T00:25:01.187 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.127+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: listing bilog for incremental sync; position=00000000001.45.6 2023-07-29T00:25:01.187 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.127+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:entry[obj00000000000000000004]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000004[0] 2023-07-29T00:25:01.188 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.128+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: listing bilog for incremental sync; position=00000000002.29.6 2023-07-29T00:25:01.188 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.128+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:entry[obj00000000000000000467]: : injecting data sync error on key=obj00000000000000000467 2023-07-29T00:25:01.188 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.128+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: listing bilog for incremental sync; position=00000000001.60.6 2023-07-29T00:25:01.188 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.128+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: parsed entry: id=00000000002.56.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-07-29T00:25:01.188 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.128+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10/obj00000000000000000021 2023-07-29T00:25:01.188 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.128+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: entry->timestamp=2023-07-29T00:24:57.797257+0000 2023-07-29T00:25:01.189 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.128+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.56.6 op=0 op_state=1 2023-07-29T00:25:01.189 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.128+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:entry[obj00000000000000000467]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:25:01.189 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.128+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:entry[obj00000000000000000467]: ERROR: failed to sync object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2/obj00000000000000000467 2023-07-29T00:25:01.189 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.128+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.189 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.128+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:entry[obj00000000000000000021]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000021[0] 2023-07-29T00:25:01.189 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.128+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: listing bilog for incremental sync; position=00000000002.56.6 2023-07-29T00:25:01.190 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.128+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: sync status for source bucket shard: 0 2023-07-29T00:25:01.190 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.128+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: sync status for source bucket shard: 0 2023-07-29T00:25:01.190 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.128+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.190 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.129+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: sync status for source bucket shard: 0 2023-07-29T00:25:01.190 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.129+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.190 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.129+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: sync status for source bucket shard: 0 2023-07-29T00:25:01.190 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.129+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.191 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.129+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.191 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.129+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.191 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.129+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: sync status for source bucket shard: 0 2023-07-29T00:25:01.191 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.130+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.191 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.130+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.191 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.130+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.192 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.130+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.192 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.131+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: sync status for source bucket shard: 0 2023-07-29T00:25:01.192 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.131+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.192 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.131+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.192 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.131+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.194 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: sync status for source bucket shard: 0 2023-07-29T00:25:01.195 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: sync status for source bucket shard: 0 2023-07-29T00:25:01.195 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]: sync status for source bucket shard: 0 2023-07-29T00:25:01.195 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]: sync status for source bucket shard: 0 2023-07-29T00:25:01.195 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]: sync status for source bucket shard: 0 2023-07-29T00:25:01.195 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]: sync status for source bucket shard: 0 2023-07-29T00:25:01.195 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: sync status for source bucket shard: 0 2023-07-29T00:25:01.196 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.196 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.196 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.196 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.196 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.196 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.197 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.197 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.197 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: parsed entry: id=00000000002.43.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-07-29T00:25:01.197 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9/obj00000000000000000000 2023-07-29T00:25:01.197 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: entry->timestamp=2023-07-29T00:24:57.774204+0000 2023-07-29T00:25:01.197 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.43.6 op=0 op_state=1 2023-07-29T00:25:01.198 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.198 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.198 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.198 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.198 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:entry[obj00000000000000000004]: success 2023-07-29T00:25:01.198 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5:5 marker=00000000002.29.6 timestamp=2023-07-29T00:24:58.051099+0000 2023-07-29T00:25:01.198 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: parsed entry: id=00000000001.42.6 iter->object=obj00000000000000000271 iter->instance= name=obj00000000000000000271 instance= ns= 2023-07-29T00:25:01.199 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12/obj00000000000000000271 2023-07-29T00:25:01.199 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: entry->timestamp=2023-07-29T00:24:58.148659+0000 2023-07-29T00:25:01.199 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:entry[obj00000000000000000271]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.42.6 op=0 op_state=1 2023-07-29T00:25:01.199 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.199 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: parsed entry: id=00000000002.49.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-07-29T00:25:01.199 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11/obj00000000000000000009 2023-07-29T00:25:01.200 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: entry->timestamp=2023-07-29T00:24:58.303302+0000 2023-07-29T00:25:01.200 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.49.6 op=0 op_state=1 2023-07-29T00:25:01.200 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: parsed entry: id=00000000002.42.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2023-07-29T00:25:01.200 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13/obj00000000000000000008 2023-07-29T00:25:01.200 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: entry->timestamp=2023-07-29T00:24:57.684655+0000 2023-07-29T00:25:01.200 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.42.6 op=0 op_state=1 2023-07-29T00:25:01.200 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:entry[obj00000000000000000021]: success 2023-07-29T00:25:01.201 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10:5 marker=00000000002.56.6 timestamp=2023-07-29T00:24:57.797257+0000 2023-07-29T00:25:01.201 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:entry[obj00000000000000000000]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000000[0] 2023-07-29T00:25:01.201 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: listing bilog for incremental sync; position=00000000002.43.6 2023-07-29T00:25:01.201 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: parsed entry: id=00000000002.22.6 iter->object=obj00000000000000000018 iter->instance= name=obj00000000000000000018 instance= ns= 2023-07-29T00:25:01.201 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14/obj00000000000000000018 2023-07-29T00:25:01.201 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: entry->timestamp=2023-07-29T00:24:58.540506+0000 2023-07-29T00:25:01.202 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:entry[obj00000000000000000018]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.22.6 op=0 op_state=1 2023-07-29T00:25:01.202 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.202 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:entry[obj00000000000000000271]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000271[0] 2023-07-29T00:25:01.202 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: listing bilog for incremental sync; position=00000000001.42.6 2023-07-29T00:25:01.202 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:entry[obj00000000000000000009]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000009[0] 2023-07-29T00:25:01.202 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: listing bilog for incremental sync; position=00000000002.49.6 2023-07-29T00:25:01.202 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:entry[obj00000000000000000008]: : injecting data sync error on key=obj00000000000000000008 2023-07-29T00:25:01.203 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: listing bilog for incremental sync; position=00000000002.42.6 2023-07-29T00:25:01.203 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: parsed entry: id=00000000002.31.6 iter->object=obj00000000000000000023 iter->instance= name=obj00000000000000000023 instance= ns= 2023-07-29T00:25:01.203 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6/obj00000000000000000023 2023-07-29T00:25:01.203 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: entry->timestamp=2023-07-29T00:24:58.380161+0000 2023-07-29T00:25:01.204 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:entry[obj00000000000000000023]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.31.6 op=0 op_state=1 2023-07-29T00:25:01.204 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.134+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.204 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.135+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:entry[obj00000000000000000018]: : injecting data sync error on key=obj00000000000000000018 2023-07-29T00:25:01.204 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.135+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: listing bilog for incremental sync; position=00000000002.22.6 2023-07-29T00:25:01.204 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.136+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:entry[obj00000000000000000008]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:25:01.204 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.136+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:entry[obj00000000000000000008]: ERROR: failed to sync object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13/obj00000000000000000008 2023-07-29T00:25:01.205 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.136+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:entry[obj00000000000000000023]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000023[0] 2023-07-29T00:25:01.205 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.136+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: listing bilog for incremental sync; position=00000000002.31.6 2023-07-29T00:25:01.205 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.136+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]: sync status for source bucket shard: 0 2023-07-29T00:25:01.205 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.136+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:entry[obj00000000000000000018]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:25:01.205 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.136+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:entry[obj00000000000000000018]: ERROR: failed to sync object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14/obj00000000000000000018 2023-07-29T00:25:01.205 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.136+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.205 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.139+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]: sync status for source bucket shard: 0 2023-07-29T00:25:01.206 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.139+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]: sync status for source bucket shard: 0 2023-07-29T00:25:01.206 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.139+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]: sync status for source bucket shard: 0 2023-07-29T00:25:01.206 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.139+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]: sync status for source bucket shard: 0 2023-07-29T00:25:01.206 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.139+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]: sync status for source bucket shard: 0 2023-07-29T00:25:01.206 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.139+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]: sync status for source bucket shard: 0 2023-07-29T00:25:01.206 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.139+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]: sync status for source bucket shard: 0 2023-07-29T00:25:01.206 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.139+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.207 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.139+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.207 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.139+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.207 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.139+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.207 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.139+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.207 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.139+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.207 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.139+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.208 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.139+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: parsed entry: id=00000000002.37.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-07-29T00:25:01.208 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.139+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21/obj00000000000000000002 2023-07-29T00:25:01.208 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.139+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: entry->timestamp=2023-07-29T00:24:58.517449+0000 2023-07-29T00:25:01.208 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.139+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.37.6 op=0 op_state=1 2023-07-29T00:25:01.208 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.139+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:entry[obj00000000000000000002]: : injecting data sync error on key=obj00000000000000000002 2023-07-29T00:25:01.208 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.139+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: listing bilog for incremental sync; position=00000000002.37.6 2023-07-29T00:25:01.209 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.140+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:entry[obj00000000000000000002]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:25:01.209 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.140+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:entry[obj00000000000000000002]: ERROR: failed to sync object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21/obj00000000000000000002 2023-07-29T00:25:01.209 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.140+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:entry[obj00000000000000000317]: success 2023-07-29T00:25:01.209 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.140+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7:5 marker=00000000001.88.6 timestamp=2023-07-29T00:24:58.614561+0000 2023-07-29T00:25:01.209 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.140+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30]: sync status for source bucket shard: 0 2023-07-29T00:25:01.209 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.140+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.209 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.140+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:entry[obj00000000000000000000]: success 2023-07-29T00:25:01.210 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.140+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9:5 marker=00000000002.43.6 timestamp=2023-07-29T00:24:57.774204+0000 2023-07-29T00:25:01.210 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.140+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:entry[obj00000000000000000009]: success 2023-07-29T00:25:01.210 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.140+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11:5 marker=00000000002.49.6 timestamp=2023-07-29T00:24:58.303302+0000 2023-07-29T00:25:01.212 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.144+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: a sync operation returned error 2023-07-29T00:25:01.212 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.144+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: backing out with sync_status=-5 2023-07-29T00:25:01.212 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.144+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: incremental sync on bucket failed, retcode=-5 2023-07-29T00:25:01.212 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.144+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: ERROR: incremental sync failed. error: -5 2023-07-29T00:25:01.213 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.145+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]: parsed entry: id=00000000002.58.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-07-29T00:25:01.213 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.145+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24/obj00000000000000000011 2023-07-29T00:25:01.213 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.145+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]: entry->timestamp=2023-07-29T00:24:58.028530+0000 2023-07-29T00:25:01.213 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.145+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.58.6 op=0 op_state=1 2023-07-29T00:25:01.213 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.145+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]: parsed entry: id=00000000002.104.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-07-29T00:25:01.213 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.145+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26/obj00000000000000000010 2023-07-29T00:25:01.214 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.145+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]: entry->timestamp=2023-07-29T00:24:58.437978+0000 2023-07-29T00:25:01.214 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.145+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.104.6 op=0 op_state=1 2023-07-29T00:25:01.214 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.145+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.214 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.145+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]:entry[obj00000000000000000011]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000011[0] 2023-07-29T00:25:01.214 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.145+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]: listing bilog for incremental sync; position=00000000002.58.6 2023-07-29T00:25:01.214 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.145+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]:entry[obj00000000000000000010]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000010[0] 2023-07-29T00:25:01.214 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.145+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]: listing bilog for incremental sync; position=00000000002.104.6 2023-07-29T00:25:01.215 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.146+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: sync status for source bucket shard: 0 2023-07-29T00:25:01.215 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.146+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]: parsed entry: id=00000000002.39.6 iter->object=obj00000000000000000050 iter->instance= name=obj00000000000000000050 instance= ns= 2023-07-29T00:25:01.215 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.146+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28/obj00000000000000000050 2023-07-29T00:25:01.215 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.146+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]: entry->timestamp=2023-07-29T00:24:57.938921+0000 2023-07-29T00:25:01.215 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.146+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:entry[obj00000000000000000050]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.39.6 op=0 op_state=1 2023-07-29T00:25:01.215 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.146+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]: parsed entry: id=00000000001.48.6 iter->object=obj00000000000000000061 iter->instance= name=obj00000000000000000061 instance= ns= 2023-07-29T00:25:01.215 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.146+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27/obj00000000000000000061 2023-07-29T00:25:01.216 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.146+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]: entry->timestamp=2023-07-29T00:24:58.597135+0000 2023-07-29T00:25:01.216 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.146+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:entry[obj00000000000000000061]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.48.6 op=0 op_state=1 2023-07-29T00:25:01.216 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.146+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.216 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.146+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30]: parsed entry: id=00000000001.58.6 iter->object=obj00000000000000000133 iter->instance= name=obj00000000000000000133 instance= ns= 2023-07-29T00:25:01.216 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.146+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30/obj00000000000000000133 2023-07-29T00:25:01.216 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.146+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30]: entry->timestamp=2023-07-29T00:24:57.710367+0000 2023-07-29T00:25:01.217 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.146+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30]:entry[obj00000000000000000133]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.58.6 op=0 op_state=1 2023-07-29T00:25:01.217 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.146+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:entry[obj00000000000000000050]: : injecting data sync error on key=obj00000000000000000050 2023-07-29T00:25:01.217 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.146+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]: listing bilog for incremental sync; position=00000000002.39.6 2023-07-29T00:25:01.217 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.146+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:entry[obj00000000000000000061]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000061[0] 2023-07-29T00:25:01.217 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.146+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]: listing bilog for incremental sync; position=00000000001.48.6 2023-07-29T00:25:01.217 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.146+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]: parsed entry: id=00000000001.60.6 iter->object=obj00000000000000000029 iter->instance= name=obj00000000000000000029 instance= ns= 2023-07-29T00:25:01.217 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.146+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29/obj00000000000000000029 2023-07-29T00:25:01.218 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.146+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]: entry->timestamp=2023-07-29T00:24:58.285139+0000 2023-07-29T00:25:01.218 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.147+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:entry[obj00000000000000000029]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.60.6 op=0 op_state=1 2023-07-29T00:25:01.218 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.147+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30]:entry[obj00000000000000000133]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000133[0] 2023-07-29T00:25:01.218 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.147+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30]: listing bilog for incremental sync; position=00000000001.58.6 2023-07-29T00:25:01.218 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.147+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:entry[obj00000000000000000050]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:25:01.218 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.147+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:entry[obj00000000000000000050]: ERROR: failed to sync object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28/obj00000000000000000050 2023-07-29T00:25:01.219 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.147+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:entry[obj00000000000000000029]: : injecting data sync error on key=obj00000000000000000029 2023-07-29T00:25:01.219 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.147+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]: listing bilog for incremental sync; position=00000000001.60.6 2023-07-29T00:25:01.219 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.147+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:entry[obj00000000000000000029]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:25:01.219 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.147+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:entry[obj00000000000000000029]: ERROR: failed to sync object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29/obj00000000000000000029 2023-07-29T00:25:01.219 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.148+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: a sync operation returned error 2023-07-29T00:25:01.219 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.148+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: backing out with sync_status=-5 2023-07-29T00:25:01.219 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.148+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: incremental sync on bucket failed, retcode=-5 2023-07-29T00:25:01.220 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.148+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]:entry[obj00000000000000000011]: success 2023-07-29T00:25:01.220 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.148+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24:5 marker=00000000002.58.6 timestamp=2023-07-29T00:24:58.028530+0000 2023-07-29T00:25:01.220 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.148+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: ERROR: incremental sync failed. error: -5 2023-07-29T00:25:01.220 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.149+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]:entry[obj00000000000000000010]: success 2023-07-29T00:25:01.220 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.149+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26:5 marker=00000000002.104.6 timestamp=2023-07-29T00:24:58.437978+0000 2023-07-29T00:25:01.220 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.151+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.221 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.151+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: sync status for source bucket shard: 0 2023-07-29T00:25:01.221 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.151+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.221 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.153+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: parsed entry: id=00000000002.37.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-07-29T00:25:01.221 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.153+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21/obj00000000000000000002 2023-07-29T00:25:01.221 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.153+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: entry->timestamp=2023-07-29T00:24:58.517449+0000 2023-07-29T00:25:01.221 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.153+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.37.6 op=0 op_state=1 2023-07-29T00:25:01.221 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.153+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:entry[obj00000000000000000002]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000002[0] 2023-07-29T00:25:01.222 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.153+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: listing bilog for incremental sync; position=00000000002.37.6 2023-07-29T00:25:01.222 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.155+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:entry[obj00000000000000000002]: success 2023-07-29T00:25:01.222 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.155+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21:5 marker=00000000002.37.6 timestamp=2023-07-29T00:24:58.517449+0000 2023-07-29T00:25:01.223 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.167+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: parsed entry: id=00000000002.69.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2023-07-29T00:25:01.223 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.167+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3/obj00000000000000000005 2023-07-29T00:25:01.223 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.167+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: entry->timestamp=2023-07-29T00:24:58.123402+0000 2023-07-29T00:25:01.223 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.167+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.69.6 op=0 op_state=1 2023-07-29T00:25:01.224 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.167+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: parsed entry: id=00000000002.44.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-07-29T00:25:01.224 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.167+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0/obj00000000000000000019 2023-07-29T00:25:01.224 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.167+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: entry->timestamp=2023-07-29T00:24:58.678084+0000 2023-07-29T00:25:01.224 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.167+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.44.6 op=0 op_state=1 2023-07-29T00:25:01.224 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.167+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: parsed entry: id=00000000001.86.6 iter->object=obj00000000000000000056 iter->instance= name=obj00000000000000000056 instance= ns= 2023-07-29T00:25:01.224 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.167+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4/obj00000000000000000056 2023-07-29T00:25:01.225 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.167+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: entry->timestamp=2023-07-29T00:24:57.604582+0000 2023-07-29T00:25:01.225 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.167+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:entry[obj00000000000000000056]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.86.6 op=0 op_state=1 2023-07-29T00:25:01.225 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.167+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:entry[obj00000000000000000005]: : injecting data sync error on key=obj00000000000000000005 2023-07-29T00:25:01.225 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.167+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: listing bilog for incremental sync; position=00000000002.69.6 2023-07-29T00:25:01.225 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.167+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:entry[obj00000000000000000019]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000019[0] 2023-07-29T00:25:01.225 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.167+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: listing bilog for incremental sync; position=00000000002.44.6 2023-07-29T00:25:01.225 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.167+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:entry[obj00000000000000000056]: : injecting data sync error on key=obj00000000000000000056 2023-07-29T00:25:01.226 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.167+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: listing bilog for incremental sync; position=00000000001.86.6 2023-07-29T00:25:01.226 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.167+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:entry[obj00000000000000000005]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:25:01.226 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.167+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:entry[obj00000000000000000005]: ERROR: failed to sync object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3/obj00000000000000000005 2023-07-29T00:25:01.226 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.167+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:entry[obj00000000000000000056]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:25:01.226 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.167+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:entry[obj00000000000000000056]: ERROR: failed to sync object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4/obj00000000000000000056 2023-07-29T00:25:01.226 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.170+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:entry[obj00000000000000000019]: success 2023-07-29T00:25:01.227 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.170+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0:5 marker=00000000002.44.6 timestamp=2023-07-29T00:24:58.678084+0000 2023-07-29T00:25:01.227 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.173+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: a sync operation returned error 2023-07-29T00:25:01.228 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.173+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: backing out with sync_status=-5 2023-07-29T00:25:01.228 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.173+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: incremental sync on bucket failed, retcode=-5 2023-07-29T00:25:01.228 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.173+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: ERROR: incremental sync failed. error: -5 2023-07-29T00:25:01.228 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.174+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.228 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.174+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: sync status for source bucket shard: 0 2023-07-29T00:25:01.228 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.174+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.229 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.177+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: parsed entry: id=00000000002.51.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2023-07-29T00:25:01.229 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.177+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16/obj00000000000000000017 2023-07-29T00:25:01.229 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.177+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: entry->timestamp=2023-07-29T00:24:57.855168+0000 2023-07-29T00:25:01.230 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.177+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.51.6 op=0 op_state=1 2023-07-29T00:25:01.230 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.177+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:entry[obj00000000000000000017]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000017[0] 2023-07-29T00:25:01.230 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.177+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: listing bilog for incremental sync; position=00000000002.51.6 2023-07-29T00:25:01.230 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.177+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]: parsed entry: id=00000000002.37.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-07-29T00:25:01.230 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.177+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18/obj00000000000000000016 2023-07-29T00:25:01.230 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.177+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]: entry->timestamp=2023-07-29T00:24:58.361072+0000 2023-07-29T00:25:01.231 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.177+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.37.6 op=0 op_state=1 2023-07-29T00:25:01.231 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.177+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:entry[obj00000000000000000016]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000016[0] 2023-07-29T00:25:01.231 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.177+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]: listing bilog for incremental sync; position=00000000002.37.6 2023-07-29T00:25:01.231 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.178+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]: parsed entry: id=00000000002.102.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-07-29T00:25:01.232 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.178+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19/obj00000000000000000003 2023-07-29T00:25:01.232 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.178+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]: entry->timestamp=2023-07-29T00:24:57.622282+0000 2023-07-29T00:25:01.232 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.178+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.102.6 op=0 op_state=1 2023-07-29T00:25:01.232 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.178+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]: parsed entry: id=00000000001.58.6 iter->object=obj00000000000000000423 iter->instance= name=obj00000000000000000423 instance= ns= 2023-07-29T00:25:01.232 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.178+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20/obj00000000000000000423 2023-07-29T00:25:01.233 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.178+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]: entry->timestamp=2023-07-29T00:24:57.967362+0000 2023-07-29T00:25:01.233 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.178+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:entry[obj00000000000000000423]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.58.6 op=0 op_state=1 2023-07-29T00:25:01.233 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.178+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:entry[obj00000000000000000003]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000003[0] 2023-07-29T00:25:01.233 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.178+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]: listing bilog for incremental sync; position=00000000002.102.6 2023-07-29T00:25:01.233 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.178+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:entry[obj00000000000000000423]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000423[0] 2023-07-29T00:25:01.233 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.178+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]: listing bilog for incremental sync; position=00000000001.58.6 2023-07-29T00:25:01.234 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.179+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:entry[obj00000000000000000017]: success 2023-07-29T00:25:01.234 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.179+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16:5 marker=00000000002.51.6 timestamp=2023-07-29T00:24:57.855168+0000 2023-07-29T00:25:01.234 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.180+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]: parsed entry: id=00000000001.61.6 iter->object=obj00000000000000000162 iter->instance= name=obj00000000000000000162 instance= ns= 2023-07-29T00:25:01.235 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.180+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17/obj00000000000000000162 2023-07-29T00:25:01.235 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.180+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]: entry->timestamp=2023-07-29T00:24:57.876925+0000 2023-07-29T00:25:01.235 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.180+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:entry[obj00000000000000000162]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.61.6 op=0 op_state=1 2023-07-29T00:25:01.235 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.180+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:entry[obj00000000000000000162]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000162[0] 2023-07-29T00:25:01.235 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.180+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]: listing bilog for incremental sync; position=00000000001.61.6 2023-07-29T00:25:01.235 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.180+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:entry[obj00000000000000000022]: success 2023-07-29T00:25:01.236 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.180+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8:5 marker=00000000001.45.6 timestamp=2023-07-29T00:24:58.700216+0000 2023-07-29T00:25:01.236 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.180+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:entry[obj00000000000000000016]: success 2023-07-29T00:25:01.236 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.180+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18:5 marker=00000000002.37.6 timestamp=2023-07-29T00:24:58.361072+0000 2023-07-29T00:25:01.236 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.181+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: a sync operation returned error 2023-07-29T00:25:01.236 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.181+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: backing out with sync_status=-5 2023-07-29T00:25:01.236 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.181+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: incremental sync on bucket failed, retcode=-5 2023-07-29T00:25:01.237 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.181+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: ERROR: incremental sync failed. error: -5 2023-07-29T00:25:01.237 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.181+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:entry[obj00000000000000000003]: success 2023-07-29T00:25:01.237 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.181+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19:5 marker=00000000002.102.6 timestamp=2023-07-29T00:24:57.622282+0000 2023-07-29T00:25:01.238 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.183+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.238 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.184+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: sync status for source bucket shard: 0 2023-07-29T00:25:01.238 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.184+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.238 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.185+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]: parsed entry: id=00000000002.56.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-07-29T00:25:01.238 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.185+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22/obj00000000000000000012 2023-07-29T00:25:01.239 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.185+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]: entry->timestamp=2023-07-29T00:24:58.208961+0000 2023-07-29T00:25:01.239 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.185+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.56.6 op=0 op_state=1 2023-07-29T00:25:01.239 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.185+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:entry[obj00000000000000000012]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000012[0] 2023-07-29T00:25:01.239 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.185+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]: listing bilog for incremental sync; position=00000000002.56.6 2023-07-29T00:25:01.240 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.186+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:entry[obj00000000000000000271]: success 2023-07-29T00:25:01.240 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.186+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12:5 marker=00000000001.42.6 timestamp=2023-07-29T00:24:58.148659+0000 2023-07-29T00:25:01.240 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.187+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]: parsed entry: id=00000000002.60.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-07-29T00:25:01.240 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.187+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23/obj00000000000000000001 2023-07-29T00:25:01.240 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.187+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]: entry->timestamp=2023-07-29T00:24:58.228492+0000 2023-07-29T00:25:01.241 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.187+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.60.6 op=0 op_state=1 2023-07-29T00:25:01.241 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.187+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]:entry[obj00000000000000000001]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000001[0] 2023-07-29T00:25:01.241 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.187+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]: listing bilog for incremental sync; position=00000000002.60.6 2023-07-29T00:25:01.241 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.188+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:entry[obj00000000000000000012]: success 2023-07-29T00:25:01.241 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.188+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22:5 marker=00000000002.56.6 timestamp=2023-07-29T00:24:58.208961+0000 2023-07-29T00:25:01.242 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.189+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]:entry[obj00000000000000000001]: success 2023-07-29T00:25:01.242 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.189+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23:5 marker=00000000002.60.6 timestamp=2023-07-29T00:24:58.228492+0000 2023-07-29T00:25:01.242 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.190+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]: a sync operation returned error 2023-07-29T00:25:01.242 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.190+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]: backing out with sync_status=-5 2023-07-29T00:25:01.243 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.190+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]: incremental sync on bucket failed, retcode=-5 2023-07-29T00:25:01.243 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.190+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]: ERROR: incremental sync failed. error: -5 2023-07-29T00:25:01.244 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.191+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:entry[obj00000000000000000023]: success 2023-07-29T00:25:01.244 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.191+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6:5 marker=00000000002.31.6 timestamp=2023-07-29T00:24:58.380161+0000 2023-07-29T00:25:01.244 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.191+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: parsed entry: id=00000000002.42.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2023-07-29T00:25:01.244 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.191+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13/obj00000000000000000008 2023-07-29T00:25:01.244 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.191+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: entry->timestamp=2023-07-29T00:24:57.684655+0000 2023-07-29T00:25:01.245 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.191+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.42.6 op=0 op_state=1 2023-07-29T00:25:01.245 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.191+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:entry[obj00000000000000000008]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000008[0] 2023-07-29T00:25:01.245 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.191+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: listing bilog for incremental sync; position=00000000002.42.6 2023-07-29T00:25:01.245 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.191+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]: a sync operation returned error 2023-07-29T00:25:01.245 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.191+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]: backing out with sync_status=-5 2023-07-29T00:25:01.245 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.191+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]: incremental sync on bucket failed, retcode=-5 2023-07-29T00:25:01.245 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.191+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]: ERROR: incremental sync failed. error: -5 2023-07-29T00:25:01.246 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.193+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:entry[obj00000000000000000008]: success 2023-07-29T00:25:01.246 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.193+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13:5 marker=00000000002.42.6 timestamp=2023-07-29T00:24:57.684655+0000 2023-07-29T00:25:01.246 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.194+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.247 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.194+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.247 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.194+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]: sync status for source bucket shard: 0 2023-07-29T00:25:01.247 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.194+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.247 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.194+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]: sync status for source bucket shard: 0 2023-07-29T00:25:01.247 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.194+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.248 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.195+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30]:entry[obj00000000000000000133]: success 2023-07-29T00:25:01.248 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.195+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30:5 marker=00000000001.58.6 timestamp=2023-07-29T00:24:57.710367+0000 2023-07-29T00:25:01.248 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.197+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:entry[obj00000000000000000061]: success 2023-07-29T00:25:01.249 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.197+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27:5 marker=00000000001.48.6 timestamp=2023-07-29T00:24:58.597135+0000 2023-07-29T00:25:01.262 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.210+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: a sync operation returned error 2023-07-29T00:25:01.262 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.210+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: backing out with sync_status=-5 2023-07-29T00:25:01.262 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.210+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: incremental sync on bucket failed, retcode=-5 2023-07-29T00:25:01.262 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.210+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: ERROR: incremental sync failed. error: -5 2023-07-29T00:25:01.263 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.210+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.264 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.210+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: sync status for source bucket shard: 0 2023-07-29T00:25:01.264 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.210+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.264 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.211+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: a sync operation returned error 2023-07-29T00:25:01.264 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.211+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: backing out with sync_status=-5 2023-07-29T00:25:01.264 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.211+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: incremental sync on bucket failed, retcode=-5 2023-07-29T00:25:01.264 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.211+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: ERROR: incremental sync failed. error: -5 2023-07-29T00:25:01.265 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.211+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.265 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.211+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: sync status for source bucket shard: 0 2023-07-29T00:25:01.265 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.211+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.269 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.217+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: parsed entry: id=00000000001.60.6 iter->object=obj00000000000000000467 iter->instance= name=obj00000000000000000467 instance= ns= 2023-07-29T00:25:01.269 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.217+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2/obj00000000000000000467 2023-07-29T00:25:01.269 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.217+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: entry->timestamp=2023-07-29T00:24:58.460850+0000 2023-07-29T00:25:01.269 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.217+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:entry[obj00000000000000000467]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.60.6 op=0 op_state=1 2023-07-29T00:25:01.269 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.217+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:entry[obj00000000000000000467]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000467[0] 2023-07-29T00:25:01.270 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.217+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: listing bilog for incremental sync; position=00000000001.60.6 2023-07-29T00:25:01.278 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.226+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: parsed entry: id=00000000002.22.6 iter->object=obj00000000000000000018 iter->instance= name=obj00000000000000000018 instance= ns= 2023-07-29T00:25:01.278 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.226+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14/obj00000000000000000018 2023-07-29T00:25:01.278 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.226+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: entry->timestamp=2023-07-29T00:24:58.540506+0000 2023-07-29T00:25:01.278 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.226+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:entry[obj00000000000000000018]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.22.6 op=0 op_state=1 2023-07-29T00:25:01.279 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.226+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:entry[obj00000000000000000018]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000018[0] 2023-07-29T00:25:01.279 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.226+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: listing bilog for incremental sync; position=00000000002.22.6 2023-07-29T00:25:01.279 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.227+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:entry[obj00000000000000000423]: success 2023-07-29T00:25:01.279 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.227+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20:5 marker=00000000001.58.6 timestamp=2023-07-29T00:24:57.967362+0000 2023-07-29T00:25:01.281 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.229+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:entry[obj00000000000000000162]: success 2023-07-29T00:25:01.281 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.229+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17:5 marker=00000000001.61.6 timestamp=2023-07-29T00:24:57.876925+0000 2023-07-29T00:25:01.289 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.237+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]: parsed entry: id=00000000002.39.6 iter->object=obj00000000000000000050 iter->instance= name=obj00000000000000000050 instance= ns= 2023-07-29T00:25:01.289 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.237+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28/obj00000000000000000050 2023-07-29T00:25:01.289 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.237+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]: entry->timestamp=2023-07-29T00:24:57.938921+0000 2023-07-29T00:25:01.289 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.237+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:entry[obj00000000000000000050]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.39.6 op=0 op_state=1 2023-07-29T00:25:01.290 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.237+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:entry[obj00000000000000000050]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000050[0] 2023-07-29T00:25:01.290 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.237+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]: listing bilog for incremental sync; position=00000000002.39.6 2023-07-29T00:25:01.290 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.238+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]: parsed entry: id=00000000001.60.6 iter->object=obj00000000000000000029 iter->instance= name=obj00000000000000000029 instance= ns= 2023-07-29T00:25:01.291 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.238+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29/obj00000000000000000029 2023-07-29T00:25:01.291 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.238+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]: entry->timestamp=2023-07-29T00:24:58.285139+0000 2023-07-29T00:25:01.291 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.238+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:entry[obj00000000000000000029]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.60.6 op=0 op_state=1 2023-07-29T00:25:01.291 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.238+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:entry[obj00000000000000000029]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000029[0] 2023-07-29T00:25:01.291 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.238+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]: listing bilog for incremental sync; position=00000000001.60.6 2023-07-29T00:25:01.292 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.239+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:entry[obj00000000000000000050]: success 2023-07-29T00:25:01.292 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.239+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28:5 marker=00000000002.39.6 timestamp=2023-07-29T00:24:57.938921+0000 2023-07-29T00:25:01.294 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.242+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:entry[obj00000000000000000029]: success 2023-07-29T00:25:01.294 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.242+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29:5 marker=00000000001.60.6 timestamp=2023-07-29T00:24:58.285139+0000 2023-07-29T00:25:01.305 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.253+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: parsed entry: id=00000000002.69.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2023-07-29T00:25:01.305 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.253+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3/obj00000000000000000005 2023-07-29T00:25:01.306 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.253+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: entry->timestamp=2023-07-29T00:24:58.123402+0000 2023-07-29T00:25:01.306 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.253+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.69.6 op=0 op_state=1 2023-07-29T00:25:01.306 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.253+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:entry[obj00000000000000000005]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000005[0] 2023-07-29T00:25:01.306 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.253+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: listing bilog for incremental sync; position=00000000002.69.6 2023-07-29T00:25:01.306 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.253+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: parsed entry: id=00000000001.86.6 iter->object=obj00000000000000000056 iter->instance= name=obj00000000000000000056 instance= ns= 2023-07-29T00:25:01.306 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.253+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4/obj00000000000000000056 2023-07-29T00:25:01.307 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.253+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: entry->timestamp=2023-07-29T00:24:57.604582+0000 2023-07-29T00:25:01.307 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.253+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:entry[obj00000000000000000056]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.86.6 op=0 op_state=1 2023-07-29T00:25:01.307 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.253+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:entry[obj00000000000000000056]: : injecting data sync error on key=obj00000000000000000056 2023-07-29T00:25:01.307 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.253+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: listing bilog for incremental sync; position=00000000001.86.6 2023-07-29T00:25:01.307 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.253+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:entry[obj00000000000000000056]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:25:01.307 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.253+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:entry[obj00000000000000000056]: ERROR: failed to sync object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4/obj00000000000000000056 2023-07-29T00:25:01.308 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.255+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:entry[obj00000000000000000005]: success 2023-07-29T00:25:01.308 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.255+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3:5 marker=00000000002.69.6 timestamp=2023-07-29T00:24:58.123402+0000 2023-07-29T00:25:01.314 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.262+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:entry[obj00000000000000000467]: success 2023-07-29T00:25:01.314 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.262+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2:5 marker=00000000001.60.6 timestamp=2023-07-29T00:24:58.460850+0000 2023-07-29T00:25:01.324 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.272+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:entry[obj00000000000000000018]: success 2023-07-29T00:25:01.324 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.272+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14:5 marker=00000000002.22.6 timestamp=2023-07-29T00:24:58.540506+0000 2023-07-29T00:25:01.348 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.296+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: a sync operation returned error 2023-07-29T00:25:01.348 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.296+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: backing out with sync_status=-5 2023-07-29T00:25:01.348 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.296+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: incremental sync on bucket failed, retcode=-5 2023-07-29T00:25:01.349 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.296+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: ERROR: incremental sync failed. error: -5 2023-07-29T00:25:01.349 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.296+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.349 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.297+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: sync status for source bucket shard: 0 2023-07-29T00:25:01.350 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.297+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.391 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.339+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: parsed entry: id=00000000001.86.6 iter->object=obj00000000000000000056 iter->instance= name=obj00000000000000000056 instance= ns= 2023-07-29T00:25:01.392 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.339+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4/obj00000000000000000056 2023-07-29T00:25:01.392 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.339+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: entry->timestamp=2023-07-29T00:24:57.604582+0000 2023-07-29T00:25:01.392 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.339+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:entry[obj00000000000000000056]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.86.6 op=0 op_state=1 2023-07-29T00:25:01.392 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.339+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:entry[obj00000000000000000056]: : injecting data sync error on key=obj00000000000000000056 2023-07-29T00:25:01.393 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.339+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: listing bilog for incremental sync; position=00000000001.86.6 2023-07-29T00:25:01.393 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.339+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:entry[obj00000000000000000056]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:25:01.393 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.339+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:entry[obj00000000000000000056]: ERROR: failed to sync object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4/obj00000000000000000056 2023-07-29T00:25:01.433 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.381+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: a sync operation returned error 2023-07-29T00:25:01.434 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.381+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: backing out with sync_status=-5 2023-07-29T00:25:01.434 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.381+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: incremental sync on bucket failed, retcode=-5 2023-07-29T00:25:01.434 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.381+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: ERROR: incremental sync failed. error: -5 2023-07-29T00:25:01.435 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.381+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.435 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.381+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: sync status for source bucket shard: 0 2023-07-29T00:25:01.435 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.381+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.475 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.423+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: parsed entry: id=00000000001.86.6 iter->object=obj00000000000000000056 iter->instance= name=obj00000000000000000056 instance= ns= 2023-07-29T00:25:01.475 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.423+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4/obj00000000000000000056 2023-07-29T00:25:01.476 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.423+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: entry->timestamp=2023-07-29T00:24:57.604582+0000 2023-07-29T00:25:01.476 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.423+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:entry[obj00000000000000000056]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.86.6 op=0 op_state=1 2023-07-29T00:25:01.476 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.423+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:entry[obj00000000000000000056]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000056[0] 2023-07-29T00:25:01.476 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.423+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: listing bilog for incremental sync; position=00000000001.86.6 2023-07-29T00:25:01.479 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.427+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:entry[obj00000000000000000056]: success 2023-07-29T00:25:01.479 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.427+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4:5 marker=00000000001.86.6 timestamp=2023-07-29T00:24:57.604582+0000 2023-07-29T00:25:01.522 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.469+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.523 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.469+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.523 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.469+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.523 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.469+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.523 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.470+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.523 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.470+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: sync status for source bucket shard: 0 2023-07-29T00:25:01.524 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.470+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.524 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.470+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: sync status for source bucket shard: 0 2023-07-29T00:25:01.524 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.470+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: sync status for source bucket shard: 0 2023-07-29T00:25:01.524 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.470+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: sync status for source bucket shard: 0 2023-07-29T00:25:01.524 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.470+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.525 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.470+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:2]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.525 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.470+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.525 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.470+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: sync status for source bucket shard: 0 2023-07-29T00:25:01.525 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.470+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.525 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.470+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.528 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.471+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.529 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.471+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.529 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.471+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: sync status for source bucket shard: 0 2023-07-29T00:25:01.530 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.471+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: sync status for source bucket shard: 0 2023-07-29T00:25:01.530 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.471+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.530 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.471+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.530 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.471+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.530 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.471+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.530 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.471+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.531 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.471+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.531 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.472+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: sync status for source bucket shard: 0 2023-07-29T00:25:01.531 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.472+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.531 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.472+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.531 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.472+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: parsed entry: id=00000000002.74.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-07-29T00:25:01.531 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.472+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1/obj00000000000000000000 2023-07-29T00:25:01.532 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.472+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: entry->timestamp=2023-07-29T00:24:59.725669+0000 2023-07-29T00:25:01.532 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.472+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.74.6 op=0 op_state=1 2023-07-29T00:25:01.532 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.472+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.532 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.472+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:entry[obj00000000000000000000]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000000[0] 2023-07-29T00:25:01.532 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.472+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: listing bilog for incremental sync; position=00000000002.74.6 2023-07-29T00:25:01.532 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.472+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: parsed entry: id=00000000001.43.6 iter->object=obj00000000000000000065 iter->instance= name=obj00000000000000000065 instance= ns= 2023-07-29T00:25:01.533 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.472+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5/obj00000000000000000065 2023-07-29T00:25:01.533 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.472+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: entry->timestamp=2023-07-29T00:24:59.805482+0000 2023-07-29T00:25:01.533 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.472+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:entry[obj00000000000000000065]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.43.6 op=0 op_state=1 2023-07-29T00:25:01.533 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.472+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:entry[obj00000000000000000065]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000065[0] 2023-07-29T00:25:01.533 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.472+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: listing bilog for incremental sync; position=00000000001.43.6 2023-07-29T00:25:01.533 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.472+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: parsed entry: id=00000000002.101.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2023-07-29T00:25:01.534 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.472+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4/obj00000000000000000033 2023-07-29T00:25:01.534 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.472+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: entry->timestamp=2023-07-29T00:25:00.051388+0000 2023-07-29T00:25:01.534 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.472+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.101.6 op=0 op_state=1 2023-07-29T00:25:01.534 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.473+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: sync status for source bucket shard: 0 2023-07-29T00:25:01.534 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.473+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: sync status for source bucket shard: 0 2023-07-29T00:25:01.535 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.473+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:entry[obj00000000000000000033]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000033[0] 2023-07-29T00:25:01.535 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.473+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: listing bilog for incremental sync; position=00000000002.101.6 2023-07-29T00:25:01.535 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.473+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: sync status for source bucket shard: 0 2023-07-29T00:25:01.535 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.473+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: sync status for source bucket shard: 0 2023-07-29T00:25:01.536 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.473+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.536 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.473+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.536 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.473+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.536 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.473+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.537 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.473+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.537 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.473+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.537 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.474+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: sync status for source bucket shard: 0 2023-07-29T00:25:01.538 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.474+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.538 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.474+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.538 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.475+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: sync status for source bucket shard: 0 2023-07-29T00:25:01.539 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.475+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.539 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.475+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.539 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.475+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.539 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.475+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.540 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.475+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.540 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.475+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.540 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.475+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.540 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.475+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: sync status for source bucket shard: 0 2023-07-29T00:25:01.540 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.475+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.540 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.475+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.541 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.475+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.541 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.475+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.544 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.476+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: sync status for source bucket shard: 0 2023-07-29T00:25:01.545 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.476+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]: sync status for source bucket shard: 0 2023-07-29T00:25:01.545 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.476+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:entry[obj00000000000000000000]: success 2023-07-29T00:25:01.545 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.476+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:1:6 marker=00000000002.74.6 timestamp=2023-07-29T00:24:59.725669+0000 2023-07-29T00:25:01.545 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.476+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.545 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.476+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.545 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.477+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: parsed entry: id=00000000002.55.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2023-07-29T00:25:01.546 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.477+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10/obj00000000000000000026 2023-07-29T00:25:01.546 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.477+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: entry->timestamp=2023-07-29T00:24:59.170768+0000 2023-07-29T00:25:01.546 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.477+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.55.6 op=0 op_state=1 2023-07-29T00:25:01.546 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.477+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.546 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.477+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.546 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.477+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.547 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.477+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:entry[obj00000000000000000026]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000026[0] 2023-07-29T00:25:01.547 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.477+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: listing bilog for incremental sync; position=00000000002.55.6 2023-07-29T00:25:01.547 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.477+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]: sync status for source bucket shard: 0 2023-07-29T00:25:01.547 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.477+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]: sync status for source bucket shard: 0 2023-07-29T00:25:01.547 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.477+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]: sync status for source bucket shard: 0 2023-07-29T00:25:01.547 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.477+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]: sync status for source bucket shard: 0 2023-07-29T00:25:01.547 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.477+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]: sync status for source bucket shard: 0 2023-07-29T00:25:01.548 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.477+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: sync status for source bucket shard: 0 2023-07-29T00:25:01.548 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.477+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: sync status for source bucket shard: 0 2023-07-29T00:25:01.548 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.477+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]: sync status for source bucket shard: 0 2023-07-29T00:25:01.548 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.478+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.548 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.478+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.548 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.478+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.549 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.478+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.549 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.478+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:23]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.549 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.478+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.549 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.478+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.549 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.478+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.549 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.478+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:entry[obj00000000000000000033]: success 2023-07-29T00:25:01.550 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.478+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:4:6 marker=00000000002.101.6 timestamp=2023-07-29T00:25:00.051388+0000 2023-07-29T00:25:01.550 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.478+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]: sync status for source bucket shard: 0 2023-07-29T00:25:01.550 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.478+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: parsed entry: id=00000000002.52.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2023-07-29T00:25:01.550 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.478+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11/obj00000000000000000030 2023-07-29T00:25:01.550 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.478+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: entry->timestamp=2023-07-29T00:24:59.078383+0000 2023-07-29T00:25:01.550 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.478+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.52.6 op=0 op_state=1 2023-07-29T00:25:01.550 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.478+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.551 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.478+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.551 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.478+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.551 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.478+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:entry[obj00000000000000000030]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000030[0] 2023-07-29T00:25:01.551 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.478+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: listing bilog for incremental sync; position=00000000002.52.6 2023-07-29T00:25:01.551 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.478+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:31]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.551 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.479+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: parsed entry: id=00000000002.43.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-07-29T00:25:01.552 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.479+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16/obj00000000000000000012 2023-07-29T00:25:01.552 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.479+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: entry->timestamp=2023-07-29T00:24:59.604469+0000 2023-07-29T00:25:01.552 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.479+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.43.6 op=0 op_state=1 2023-07-29T00:25:01.552 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.479+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:32]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.552 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.480+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:entry[obj00000000000000000012]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000012[0] 2023-07-29T00:25:01.552 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.480+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: listing bilog for incremental sync; position=00000000002.43.6 2023-07-29T00:25:01.553 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.480+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: parsed entry: id=00000000001.68.6 iter->object=obj00000000000000000048 iter->instance= name=obj00000000000000000048 instance= ns= 2023-07-29T00:25:01.553 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.480+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8/obj00000000000000000048 2023-07-29T00:25:01.553 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.480+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: entry->timestamp=2023-07-29T00:24:59.028892+0000 2023-07-29T00:25:01.553 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.480+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:entry[obj00000000000000000048]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.68.6 op=0 op_state=1 2023-07-29T00:25:01.553 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.481+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: parsed entry: id=00000000001.69.6 iter->object=obj00000000000000000062 iter->instance= name=obj00000000000000000062 instance= ns= 2023-07-29T00:25:01.553 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.481+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12/obj00000000000000000062 2023-07-29T00:25:01.554 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.481+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: entry->timestamp=2023-07-29T00:25:00.281200+0000 2023-07-29T00:25:01.554 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.481+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:entry[obj00000000000000000062]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.69.6 op=0 op_state=1 2023-07-29T00:25:01.554 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.481+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:33]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.554 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.482+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:entry[obj00000000000000000048]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000048[0] 2023-07-29T00:25:01.554 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.482+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: listing bilog for incremental sync; position=00000000001.68.6 2023-07-29T00:25:01.554 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.482+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:entry[obj00000000000000000062]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000062[0] 2023-07-29T00:25:01.554 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.482+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: listing bilog for incremental sync; position=00000000001.69.6 2023-07-29T00:25:01.555 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.482+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: parsed entry: id=00000000002.54.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-07-29T00:25:01.555 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.482+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13/obj00000000000000000009 2023-07-29T00:25:01.555 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.482+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: entry->timestamp=2023-07-29T00:24:59.500645+0000 2023-07-29T00:25:01.555 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.482+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.54.6 op=0 op_state=1 2023-07-29T00:25:01.555 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.483+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:34]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.555 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.483+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]: sync status for source bucket shard: 0 2023-07-29T00:25:01.556 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.483+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]: sync status for source bucket shard: 0 2023-07-29T00:25:01.556 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.483+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]: sync status for source bucket shard: 0 2023-07-29T00:25:01.556 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.483+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:entry[obj00000000000000000009]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000009[0] 2023-07-29T00:25:01.556 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.483+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: listing bilog for incremental sync; position=00000000002.54.6 2023-07-29T00:25:01.556 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.483+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.556 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.483+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.557 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.483+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.557 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.483+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:35]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.557 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.483+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:36]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.557 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.483+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]: sync status for source bucket shard: 0 2023-07-29T00:25:01.557 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.484+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.557 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.484+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30]: sync status for source bucket shard: 0 2023-07-29T00:25:01.557 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.485+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: parsed entry: id=00000000002.52.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2023-07-29T00:25:01.558 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.485+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21/obj00000000000000000017 2023-07-29T00:25:01.558 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.485+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: entry->timestamp=2023-07-29T00:25:00.130458+0000 2023-07-29T00:25:01.558 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.485+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.52.6 op=0 op_state=1 2023-07-29T00:25:01.558 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.485+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:entry[obj00000000000000000026]: success 2023-07-29T00:25:01.558 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.485+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:10:6 marker=00000000002.55.6 timestamp=2023-07-29T00:24:59.170768+0000 2023-07-29T00:25:01.558 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.485+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: parsed entry: id=00000000002.65.6 iter->object=obj00000000000000000040 iter->instance= name=obj00000000000000000040 instance= ns= 2023-07-29T00:25:01.559 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.485+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14/obj00000000000000000040 2023-07-29T00:25:01.559 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.485+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: entry->timestamp=2023-07-29T00:24:59.326766+0000 2023-07-29T00:25:01.559 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.485+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:entry[obj00000000000000000040]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.65.6 op=0 op_state=1 2023-07-29T00:25:01.559 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.485+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]: parsed entry: id=00000000002.53.6 iter->object=obj00000000000000000027 iter->instance= name=obj00000000000000000027 instance= ns= 2023-07-29T00:25:01.559 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.485+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24/obj00000000000000000027 2023-07-29T00:25:01.559 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.485+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]: entry->timestamp=2023-07-29T00:24:59.909186+0000 2023-07-29T00:25:01.559 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.485+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]:entry[obj00000000000000000027]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.53.6 op=0 op_state=1 2023-07-29T00:25:01.560 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.485+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.560 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.485+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:entry[obj00000000000000000017]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000017[0] 2023-07-29T00:25:01.560 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.485+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: listing bilog for incremental sync; position=00000000002.52.6 2023-07-29T00:25:01.560 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.485+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]: parsed entry: id=00000000002.90.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-07-29T00:25:01.560 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.485+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25/obj00000000000000000010 2023-07-29T00:25:01.560 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.485+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]: entry->timestamp=2023-07-29T00:24:59.665592+0000 2023-07-29T00:25:01.561 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.485+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.90.6 op=0 op_state=1 2023-07-29T00:25:01.561 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.485+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:entry[obj00000000000000000040]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000040[0] 2023-07-29T00:25:01.561 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.485+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: listing bilog for incremental sync; position=00000000002.65.6 2023-07-29T00:25:01.561 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.485+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]:entry[obj00000000000000000027]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000027[0] 2023-07-29T00:25:01.561 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.485+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]: listing bilog for incremental sync; position=00000000002.53.6 2023-07-29T00:25:01.561 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.485+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]:entry[obj00000000000000000010]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000010[0] 2023-07-29T00:25:01.561 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.485+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]: listing bilog for incremental sync; position=00000000002.90.6 2023-07-29T00:25:01.562 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.486+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:entry[obj00000000000000000065]: success 2023-07-29T00:25:01.562 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.486+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:5:6 marker=00000000001.43.6 timestamp=2023-07-29T00:24:59.805482+0000 2023-07-29T00:25:01.562 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.488+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:entry[obj00000000000000000030]: success 2023-07-29T00:25:01.562 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.488+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:11:6 marker=00000000002.52.6 timestamp=2023-07-29T00:24:59.078383+0000 2023-07-29T00:25:01.562 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.488+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:34]: sync status for source bucket shard: 0 2023-07-29T00:25:01.562 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.488+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:34]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:34]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.563 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.488+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:31]: sync status for source bucket shard: 0 2023-07-29T00:25:01.563 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.488+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:32]: sync status for source bucket shard: 0 2023-07-29T00:25:01.563 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.488+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:33]: sync status for source bucket shard: 0 2023-07-29T00:25:01.563 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.488+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:35]: sync status for source bucket shard: 0 2023-07-29T00:25:01.563 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.488+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:31]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:31]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.563 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.488+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:32]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:32]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.563 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.488+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:36]: sync status for source bucket shard: 0 2023-07-29T00:25:01.564 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.488+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:33]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:33]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.564 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.488+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:35]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:35]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.564 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.488+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:36]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:36]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.564 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.489+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:entry[obj00000000000000000012]: success 2023-07-29T00:25:01.564 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.489+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:16:6 marker=00000000002.43.6 timestamp=2023-07-29T00:24:59.604469+0000 2023-07-29T00:25:01.564 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.489+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]: parsed entry: id=00000000002.58.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-07-29T00:25:01.565 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.489+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26/obj00000000000000000021 2023-07-29T00:25:01.565 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.489+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]: entry->timestamp=2023-07-29T00:24:59.824586+0000 2023-07-29T00:25:01.565 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.489+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.58.6 op=0 op_state=1 2023-07-29T00:25:01.565 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.489+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:entry[obj00000000000000000009]: success 2023-07-29T00:25:01.565 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.489+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:13:6 marker=00000000002.54.6 timestamp=2023-07-29T00:24:59.500645+0000 2023-07-29T00:25:01.565 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.490+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]:entry[obj00000000000000000021]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000021[0] 2023-07-29T00:25:01.566 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.490+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]: listing bilog for incremental sync; position=00000000002.58.6 2023-07-29T00:25:01.566 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.491+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30]: parsed entry: id=00000000001.53.6 iter->object=obj00000000000000000036 iter->instance= name=obj00000000000000000036 instance= ns= 2023-07-29T00:25:01.566 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.491+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30/obj00000000000000000036 2023-07-29T00:25:01.566 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.491+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30]: entry->timestamp=2023-07-29T00:24:59.146626+0000 2023-07-29T00:25:01.566 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.491+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30]:entry[obj00000000000000000036]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.53.6 op=0 op_state=1 2023-07-29T00:25:01.566 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.491+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30]:entry[obj00000000000000000036]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000036[0] 2023-07-29T00:25:01.566 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.491+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30]: listing bilog for incremental sync; position=00000000001.53.6 2023-07-29T00:25:01.567 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.491+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]: parsed entry: id=00000000002.63.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-07-29T00:25:01.567 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.491+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29/obj00000000000000000004 2023-07-29T00:25:01.567 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.491+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]: entry->timestamp=2023-07-29T00:24:59.233760+0000 2023-07-29T00:25:01.567 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.491+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.63.6 op=0 op_state=1 2023-07-29T00:25:01.567 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.491+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]: parsed entry: id=00000000002.54.6 iter->object=obj00000000000000000053 iter->instance= name=obj00000000000000000053 instance= ns= 2023-07-29T00:25:01.567 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.491+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27/obj00000000000000000053 2023-07-29T00:25:01.567 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.491+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]: entry->timestamp=2023-07-29T00:25:00.300321+0000 2023-07-29T00:25:01.568 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.491+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:entry[obj00000000000000000053]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.54.6 op=0 op_state=1 2023-07-29T00:25:01.568 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.491+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]: parsed entry: id=00000000002.50.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2023-07-29T00:25:01.568 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.491+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28/obj00000000000000000020 2023-07-29T00:25:01.568 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.491+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]: entry->timestamp=2023-07-29T00:24:59.538340+0000 2023-07-29T00:25:01.568 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.491+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.50.6 op=0 op_state=1 2023-07-29T00:25:01.568 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.491+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:entry[obj00000000000000000004]: : injecting data sync error on key=obj00000000000000000004 2023-07-29T00:25:01.569 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.491+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]: listing bilog for incremental sync; position=00000000002.63.6 2023-07-29T00:25:01.569 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.491+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:entry[obj00000000000000000053]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000053[0] 2023-07-29T00:25:01.569 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.491+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]: listing bilog for incremental sync; position=00000000002.54.6 2023-07-29T00:25:01.569 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.491+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:entry[obj00000000000000000020]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000020[0] 2023-07-29T00:25:01.569 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.491+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]: listing bilog for incremental sync; position=00000000002.50.6 2023-07-29T00:25:01.569 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.491+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:entry[obj00000000000000000004]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:25:01.570 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.491+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:entry[obj00000000000000000004]: ERROR: failed to sync object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29/obj00000000000000000004 2023-07-29T00:25:01.570 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.492+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:entry[obj00000000000000000040]: success 2023-07-29T00:25:01.570 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.492+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:14:6 marker=00000000002.65.6 timestamp=2023-07-29T00:24:59.326766+0000 2023-07-29T00:25:01.570 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.492+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:entry[obj00000000000000000017]: success 2023-07-29T00:25:01.570 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.492+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:21:6 marker=00000000002.52.6 timestamp=2023-07-29T00:25:00.130458+0000 2023-07-29T00:25:01.573 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.493+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]:entry[obj00000000000000000027]: success 2023-07-29T00:25:01.573 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.493+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:24:6 marker=00000000002.53.6 timestamp=2023-07-29T00:24:59.909186+0000 2023-07-29T00:25:01.573 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.494+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]:entry[obj00000000000000000010]: success 2023-07-29T00:25:01.573 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.494+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:25:6 marker=00000000002.90.6 timestamp=2023-07-29T00:24:59.665592+0000 2023-07-29T00:25:01.574 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.495+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]:entry[obj00000000000000000021]: success 2023-07-29T00:25:01.574 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.495+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:26:6 marker=00000000002.58.6 timestamp=2023-07-29T00:24:59.824586+0000 2023-07-29T00:25:01.574 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.495+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:entry[obj00000000000000000062]: success 2023-07-29T00:25:01.574 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.495+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:12:6 marker=00000000001.69.6 timestamp=2023-07-29T00:25:00.281200+0000 2023-07-29T00:25:01.574 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.495+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:entry[obj00000000000000000020]: success 2023-07-29T00:25:01.574 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.495+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:28:6 marker=00000000002.50.6 timestamp=2023-07-29T00:24:59.538340+0000 2023-07-29T00:25:01.574 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.496+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]: a sync operation returned error 2023-07-29T00:25:01.575 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.496+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]: backing out with sync_status=-5 2023-07-29T00:25:01.575 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.496+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]: incremental sync on bucket failed, retcode=-5 2023-07-29T00:25:01.575 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.496+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]: ERROR: incremental sync failed. error: -5 2023-07-29T00:25:01.575 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.498+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.575 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.498+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]: sync status for source bucket shard: 0 2023-07-29T00:25:01.575 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.498+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.575 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.513+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: parsed entry: id=00000000001.97.6 iter->object=obj00000000000000000060 iter->instance= name=obj00000000000000000060 instance= ns= 2023-07-29T00:25:01.576 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.513+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0/obj00000000000000000060 2023-07-29T00:25:01.576 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.513+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: entry->timestamp=2023-07-29T00:24:59.520291+0000 2023-07-29T00:25:01.576 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.513+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:entry[obj00000000000000000060]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.97.6 op=0 op_state=1 2023-07-29T00:25:01.576 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.513+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:entry[obj00000000000000000060]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000060[0] 2023-07-29T00:25:01.576 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.513+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: listing bilog for incremental sync; position=00000000001.97.6 2023-07-29T00:25:01.576 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.514+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: parsed entry: id=00000000002.51.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-07-29T00:25:01.577 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.514+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3/obj00000000000000000001 2023-07-29T00:25:01.577 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.514+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: entry->timestamp=2023-07-29T00:24:59.749457+0000 2023-07-29T00:25:01.577 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.514+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.51.6 op=0 op_state=1 2023-07-29T00:25:01.577 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.514+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:entry[obj00000000000000000001]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000001[0] 2023-07-29T00:25:01.577 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.514+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: listing bilog for incremental sync; position=00000000002.51.6 2023-07-29T00:25:01.577 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.514+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: parsed entry: id=00000000002.57.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-07-29T00:25:01.578 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.514+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7/obj00000000000000000016 2023-07-29T00:25:01.578 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.514+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: entry->timestamp=2023-07-29T00:24:59.056052+0000 2023-07-29T00:25:01.578 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.514+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.57.6 op=0 op_state=1 2023-07-29T00:25:01.578 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.514+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: parsed entry: id=00000000002.67.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2023-07-29T00:25:01.578 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.514+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6/obj00000000000000000005 2023-07-29T00:25:01.579 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.514+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: entry->timestamp=2023-07-29T00:24:59.414494+0000 2023-07-29T00:25:01.579 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.514+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.67.6 op=0 op_state=1 2023-07-29T00:25:01.579 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.514+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:entry[obj00000000000000000016]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000016[0] 2023-07-29T00:25:01.579 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.514+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: listing bilog for incremental sync; position=00000000002.57.6 2023-07-29T00:25:01.579 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.514+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:entry[obj00000000000000000005]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000005[0] 2023-07-29T00:25:01.579 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.514+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: listing bilog for incremental sync; position=00000000002.67.6 2023-07-29T00:25:01.580 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.516+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:entry[obj00000000000000000001]: success 2023-07-29T00:25:01.580 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.516+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:3:6 marker=00000000002.51.6 timestamp=2023-07-29T00:24:59.749457+0000 2023-07-29T00:25:01.580 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.516+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:entry[obj00000000000000000016]: success 2023-07-29T00:25:01.580 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.516+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:7:6 marker=00000000002.57.6 timestamp=2023-07-29T00:24:59.056052+0000 2023-07-29T00:25:01.580 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.516+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:entry[obj00000000000000000005]: success 2023-07-29T00:25:01.580 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.516+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:6:6 marker=00000000002.67.6 timestamp=2023-07-29T00:24:59.414494+0000 2023-07-29T00:25:01.581 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.518+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: parsed entry: id=00000000002.70.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2023-07-29T00:25:01.581 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.518+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9/obj00000000000000000015 2023-07-29T00:25:01.581 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.518+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: entry->timestamp=2023-07-29T00:24:59.353507+0000 2023-07-29T00:25:01.581 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.518+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.70.6 op=0 op_state=1 2023-07-29T00:25:01.581 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.518+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:entry[obj00000000000000000015]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000015[0] 2023-07-29T00:25:01.581 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.518+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: listing bilog for incremental sync; position=00000000002.70.6 2023-07-29T00:25:01.581 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.519+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: parsed entry: id=00000000002.37.6 iter->object=obj00000000000000000024 iter->instance= name=obj00000000000000000024 instance= ns= 2023-07-29T00:25:01.582 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.519+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15/obj00000000000000000024 2023-07-29T00:25:01.582 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.519+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: entry->timestamp=2023-07-29T00:24:59.435609+0000 2023-07-29T00:25:01.582 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.519+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:entry[obj00000000000000000024]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.37.6 op=0 op_state=1 2023-07-29T00:25:01.582 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.519+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:entry[obj00000000000000000024]: : injecting data sync error on key=obj00000000000000000024 2023-07-29T00:25:01.582 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.519+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: listing bilog for incremental sync; position=00000000002.37.6 2023-07-29T00:25:01.582 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.519+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:entry[obj00000000000000000024]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:25:01.583 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.519+0000 7fb978526880 0 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:entry[obj00000000000000000024]: ERROR: failed to sync object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15/obj00000000000000000024 2023-07-29T00:25:01.583 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.519+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:entry[obj00000000000000000015]: success 2023-07-29T00:25:01.583 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.519+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:9:6 marker=00000000002.70.6 timestamp=2023-07-29T00:24:59.353507+0000 2023-07-29T00:25:01.583 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.520+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]: parsed entry: id=00000000002.60.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2023-07-29T00:25:01.583 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.520+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22/obj00000000000000000007 2023-07-29T00:25:01.583 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.520+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]: entry->timestamp=2023-07-29T00:24:59.887579+0000 2023-07-29T00:25:01.583 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.520+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.60.6 op=0 op_state=1 2023-07-29T00:25:01.584 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.520+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:entry[obj00000000000000000007]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000007[0] 2023-07-29T00:25:01.584 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.520+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]: listing bilog for incremental sync; position=00000000002.60.6 2023-07-29T00:25:01.584 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.521+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]: parsed entry: id=00000000002.77.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-07-29T00:25:01.584 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.521+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20/obj00000000000000000006 2023-07-29T00:25:01.584 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.521+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]: entry->timestamp=2023-07-29T00:24:59.624215+0000 2023-07-29T00:25:01.584 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.521+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.77.6 op=0 op_state=1 2023-07-29T00:25:01.585 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.521+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:entry[obj00000000000000000006]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000006[0] 2023-07-29T00:25:01.585 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.521+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]: listing bilog for incremental sync; position=00000000002.77.6 2023-07-29T00:25:01.587 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.521+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:entry[obj00000000000000000007]: success 2023-07-29T00:25:01.587 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.522+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:22:6 marker=00000000002.60.6 timestamp=2023-07-29T00:24:59.887579+0000 2023-07-29T00:25:01.587 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.522+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]: parsed entry: id=00000000002.66.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-07-29T00:25:01.588 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.522+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17/obj00000000000000000002 2023-07-29T00:25:01.588 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.522+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]: entry->timestamp=2023-07-29T00:24:59.644274+0000 2023-07-29T00:25:01.588 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.522+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.66.6 op=0 op_state=1 2023-07-29T00:25:01.588 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.522+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:entry[obj00000000000000000002]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000002[0] 2023-07-29T00:25:01.588 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.522+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]: listing bilog for incremental sync; position=00000000002.66.6 2023-07-29T00:25:01.588 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.522+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]: parsed entry: id=00000000002.41.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2023-07-29T00:25:01.588 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.522+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18/obj00000000000000000013 2023-07-29T00:25:01.589 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.522+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]: entry->timestamp=2023-07-29T00:24:59.259576+0000 2023-07-29T00:25:01.589 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.522+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.41.6 op=0 op_state=1 2023-07-29T00:25:01.589 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.522+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]: parsed entry: id=00000000002.42.6 iter->object=obj00000000000000000045 iter->instance= name=obj00000000000000000045 instance= ns= 2023-07-29T00:25:01.589 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.522+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19/obj00000000000000000045 2023-07-29T00:25:01.589 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.522+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]: entry->timestamp=2023-07-29T00:25:00.070217+0000 2023-07-29T00:25:01.589 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.522+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:entry[obj00000000000000000045]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.42.6 op=0 op_state=1 2023-07-29T00:25:01.590 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.522+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:entry[obj00000000000000000013]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000013[0] 2023-07-29T00:25:01.590 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.522+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]: listing bilog for incremental sync; position=00000000002.41.6 2023-07-29T00:25:01.590 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.522+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:entry[obj00000000000000000045]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000045[0] 2023-07-29T00:25:01.590 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.522+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]: listing bilog for incremental sync; position=00000000002.42.6 2023-07-29T00:25:01.590 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.523+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:entry[obj00000000000000000006]: success 2023-07-29T00:25:01.590 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.523+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:20:6 marker=00000000002.77.6 timestamp=2023-07-29T00:24:59.624215+0000 2023-07-29T00:25:01.591 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.524+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:entry[obj00000000000000000002]: success 2023-07-29T00:25:01.591 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.524+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:17:6 marker=00000000002.66.6 timestamp=2023-07-29T00:24:59.644274+0000 2023-07-29T00:25:01.591 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.524+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:entry[obj00000000000000000045]: success 2023-07-29T00:25:01.591 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.524+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:19:6 marker=00000000002.42.6 timestamp=2023-07-29T00:25:00.070217+0000 2023-07-29T00:25:01.591 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.524+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:entry[obj00000000000000000013]: success 2023-07-29T00:25:01.591 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.524+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:18:6 marker=00000000002.41.6 timestamp=2023-07-29T00:24:59.259576+0000 2023-07-29T00:25:01.591 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.531+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:31]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:31]: parsed entry: id=00000000002.55.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-07-29T00:25:01.592 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.531+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:31]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:31]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:31/obj00000000000000000003 2023-07-29T00:25:01.592 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.531+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:31]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:31]: entry->timestamp=2023-07-29T00:25:00.149911+0000 2023-07-29T00:25:01.592 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.531+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:31]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:31]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.55.6 op=0 op_state=1 2023-07-29T00:25:01.592 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.531+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:31]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:31]:entry[obj00000000000000000003]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000003[0] 2023-07-29T00:25:01.592 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.531+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:31]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:31]: listing bilog for incremental sync; position=00000000002.55.6 2023-07-29T00:25:01.592 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.531+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:entry[obj00000000000000000048]: success 2023-07-29T00:25:01.593 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.531+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:8:6 marker=00000000001.68.6 timestamp=2023-07-29T00:24:59.028892+0000 2023-07-29T00:25:01.593 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.533+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:31]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:31]:entry[obj00000000000000000003]: success 2023-07-29T00:25:01.593 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.533+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:31]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:31]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:31:6 marker=00000000002.55.6 timestamp=2023-07-29T00:25:00.149911+0000 2023-07-29T00:25:01.593 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.533+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:32]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:32]: parsed entry: id=00000000002.53.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2023-07-29T00:25:01.593 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.533+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:32]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:32]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:32/obj00000000000000000014 2023-07-29T00:25:01.593 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.533+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:32]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:32]: entry->timestamp=2023-07-29T00:24:59.990758+0000 2023-07-29T00:25:01.593 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.533+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:32]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:32]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.53.6 op=0 op_state=1 2023-07-29T00:25:01.594 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.533+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:32]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:32]:entry[obj00000000000000000014]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000014[0] 2023-07-29T00:25:01.594 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.533+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:32]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:32]: listing bilog for incremental sync; position=00000000002.53.6 2023-07-29T00:25:01.594 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.534+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:34]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:34]: parsed entry: id=00000000001.65.6 iter->object=obj00000000000000000057 iter->instance= name=obj00000000000000000057 instance= ns= 2023-07-29T00:25:01.594 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.534+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:34]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:34]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:34/obj00000000000000000057 2023-07-29T00:25:01.594 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.534+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:34]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:34]: entry->timestamp=2023-07-29T00:25:00.207399+0000 2023-07-29T00:25:01.595 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.534+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:34]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:34]:entry[obj00000000000000000057]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000001.65.6 op=0 op_state=1 2023-07-29T00:25:01.595 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.534+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:34]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:34]:entry[obj00000000000000000057]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000057[0] 2023-07-29T00:25:01.595 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.534+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:34]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:34]: listing bilog for incremental sync; position=00000000001.65.6 2023-07-29T00:25:01.595 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.534+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:33]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:33]: parsed entry: id=00000000002.99.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-07-29T00:25:01.595 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.534+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:33]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:33]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:33/obj00000000000000000019 2023-07-29T00:25:01.595 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.534+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:33]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:33]: entry->timestamp=2023-07-29T00:24:59.969687+0000 2023-07-29T00:25:01.596 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.534+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:33]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:33]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.99.6 op=0 op_state=1 2023-07-29T00:25:01.596 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.534+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:33]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:33]:entry[obj00000000000000000019]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000019[0] 2023-07-29T00:25:01.596 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.534+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:33]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:33]: listing bilog for incremental sync; position=00000000002.99.6 2023-07-29T00:25:01.596 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.535+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:32]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:32]:entry[obj00000000000000000014]: success 2023-07-29T00:25:01.596 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.535+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:32]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:32]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:32:6 marker=00000000002.53.6 timestamp=2023-07-29T00:24:59.990758+0000 2023-07-29T00:25:01.596 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.535+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:36]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:36]: parsed entry: id=00000000002.114.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2023-07-29T00:25:01.596 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.535+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:36]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:36]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:36/obj00000000000000000008 2023-07-29T00:25:01.597 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.535+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:36]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:36]: entry->timestamp=2023-07-29T00:25:00.226007+0000 2023-07-29T00:25:01.597 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.535+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:36]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:36]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.114.6 op=0 op_state=1 2023-07-29T00:25:01.597 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.535+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:36]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:36]:entry[obj00000000000000000008]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000008[0] 2023-07-29T00:25:01.597 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.535+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:36]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:36]: listing bilog for incremental sync; position=00000000002.114.6 2023-07-29T00:25:01.598 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.536+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:33]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:33]:entry[obj00000000000000000019]: success 2023-07-29T00:25:01.598 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.536+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:33]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:33]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:33:6 marker=00000000002.99.6 timestamp=2023-07-29T00:24:59.969687+0000 2023-07-29T00:25:01.599 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.537+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:36]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:36]:entry[obj00000000000000000008]: success 2023-07-29T00:25:01.599 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.537+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:36]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:36]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:36:6 marker=00000000002.114.6 timestamp=2023-07-29T00:25:00.226007+0000 2023-07-29T00:25:01.599 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.539+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30]:entry[obj00000000000000000036]: success 2023-07-29T00:25:01.599 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.539+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:30:6 marker=00000000001.53.6 timestamp=2023-07-29T00:24:59.146626+0000 2023-07-29T00:25:01.599 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.540+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]: parsed entry: id=00000000002.63.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-07-29T00:25:01.599 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.540+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29/obj00000000000000000004 2023-07-29T00:25:01.600 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.540+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]: entry->timestamp=2023-07-29T00:24:59.233760+0000 2023-07-29T00:25:01.600 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.540+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.63.6 op=0 op_state=1 2023-07-29T00:25:01.600 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.540+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:entry[obj00000000000000000004]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000004[0] 2023-07-29T00:25:01.600 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.540+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]: listing bilog for incremental sync; position=00000000002.63.6 2023-07-29T00:25:01.600 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.541+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:entry[obj00000000000000000053]: success 2023-07-29T00:25:01.600 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.541+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:27:6 marker=00000000002.54.6 timestamp=2023-07-29T00:25:00.300321+0000 2023-07-29T00:25:01.600 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.542+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:entry[obj00000000000000000004]: success 2023-07-29T00:25:01.601 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.542+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:29:6 marker=00000000002.63.6 timestamp=2023-07-29T00:24:59.233760+0000 2023-07-29T00:25:01.609 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.557+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:entry[obj00000000000000000060]: success 2023-07-29T00:25:01.609 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.557+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:0:6 marker=00000000001.97.6 timestamp=2023-07-29T00:24:59.520291+0000 2023-07-29T00:25:01.613 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.561+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: a sync operation returned error 2023-07-29T00:25:01.613 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.561+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: backing out with sync_status=-5 2023-07-29T00:25:01.613 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.561+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: incremental sync on bucket failed, retcode=-5 2023-07-29T00:25:01.613 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.561+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: ERROR: incremental sync failed. error: -5 2023-07-29T00:25:01.613 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.561+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:25:01.614 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.561+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: sync status for source bucket shard: 0 2023-07-29T00:25:01.614 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.561+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: listing bilog for incremental sync; position= 2023-07-29T00:25:01.631 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.579+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:34]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:34]:entry[obj00000000000000000057]: success 2023-07-29T00:25:01.631 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.579+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:34]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:34]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:34:6 marker=00000000001.65.6 timestamp=2023-07-29T00:25:00.207399+0000 2023-07-29T00:25:01.655 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.603+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: parsed entry: id=00000000002.37.6 iter->object=obj00000000000000000024 iter->instance= name=obj00000000000000000024 instance= ns= 2023-07-29T00:25:01.655 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.603+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: syncing object: hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15/obj00000000000000000024 2023-07-29T00:25:01.655 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.603+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: entry->timestamp=2023-07-29T00:24:59.435609+0000 2023-07-29T00:25:01.656 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.603+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:entry[obj00000000000000000024]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=00000000002.37.6 op=0 op_state=1 2023-07-29T00:25:01.656 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.603+0000 7fb978526880 5 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:entry[obj00000000000000000024]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-58[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23])/obj00000000000000000024[0] 2023-07-29T00:25:01.656 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.603+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: listing bilog for incremental sync; position=00000000002.37.6 2023-07-29T00:25:01.656 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.604+0000 7fb978526880 10 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:entry[obj00000000000000000024]: success 2023-07-29T00:25:01.657 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:01.604+0000 7fb978526880 20 RGW-SYNC:bucket[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23<-hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]:inc_sync[hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15]: updating marker marker_oid=bucket.sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23:15:6 marker=00000000002.37.6 timestamp=2023-07-29T00:24:59.435609+0000 2023-07-29T00:25:01.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-58', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:25:01.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-58', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:25:01.719 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-58 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:25:01.806 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:25:01.807 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:25:01.839 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T00:25:01.840 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:01.840 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000000", 2023-07-29T00:25:01.840 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:01.840 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:01.840 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:01.840 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 25 2023-07-29T00:25:01.841 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.841 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:01.841 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:01.841 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:01.841 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:01.841 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:01.841 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.725669Z", 2023-07-29T00:25:01.842 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:01.842 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:01.842 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:01.842 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:01.842 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:01.842 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:01.843 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:01.843 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:01.843 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.843 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.5221623871019381885", 2023-07-29T00:25:01.843 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:01.843 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:01.843 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:01.844 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.844 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:01.844 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000001", 2023-07-29T00:25:01.844 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:01.844 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:01.845 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:01.845 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 25 2023-07-29T00:25:01.845 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.845 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:01.845 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:01.845 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:01.845 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:01.846 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:25:01.846 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.749457Z", 2023-07-29T00:25:01.846 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:01.846 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:01.846 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:01.846 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:01.846 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:01.847 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:01.847 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:01.847 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:01.847 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.847 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.6719332773747203733", 2023-07-29T00:25:01.847 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:01.848 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:01.848 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:01.848 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.848 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:01.848 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000002", 2023-07-29T00:25:01.848 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:01.848 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:01.849 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:01.849 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 12 2023-07-29T00:25:01.849 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.849 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:01.849 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:01.849 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:01.849 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:01.850 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:25:01.850 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.644274Z", 2023-07-29T00:25:01.850 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:01.850 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:01.850 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:01.850 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:01.850 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:01.851 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:01.851 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:01.851 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:01.851 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.851 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.11111329959679395103", 2023-07-29T00:25:01.851 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:01.851 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:01.852 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:01.852 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.852 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:01.852 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000003", 2023-07-29T00:25:01.852 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:01.852 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:01.852 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:01.853 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 18 2023-07-29T00:25:01.853 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.853 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:01.853 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:01.853 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:01.853 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:01.853 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:01.854 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:25:00.149911Z", 2023-07-29T00:25:01.854 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:01.854 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:01.854 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:01.854 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:01.854 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:01.854 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:01.855 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:01.855 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:01.855 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.855 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.11866367633935491207", 2023-07-29T00:25:01.855 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:01.855 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:01.855 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:01.856 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.856 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:01.856 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000004", 2023-07-29T00:25:01.856 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:01.856 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:01.856 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:01.857 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 28 2023-07-29T00:25:01.857 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.857 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:01.857 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:01.857 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:01.857 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:01.857 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 6, 2023-07-29T00:25:01.858 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.233760Z", 2023-07-29T00:25:01.858 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:01.858 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:01.858 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:01.858 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:01.858 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:01.858 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:01.859 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:01.859 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:01.859 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.859 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.4020737331217317834", 2023-07-29T00:25:01.859 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:01.859 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:01.859 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:01.860 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.860 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:01.860 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000005", 2023-07-29T00:25:01.860 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:01.860 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:01.860 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:01.860 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 11 2023-07-29T00:25:01.861 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.861 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:01.861 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:01.861 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:01.861 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:01.861 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 6, 2023-07-29T00:25:01.861 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.414494Z", 2023-07-29T00:25:01.862 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:01.862 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:01.862 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:01.862 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:01.862 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:01.862 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:01.862 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:01.863 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:01.863 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.863 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.4849720330961193877", 2023-07-29T00:25:01.863 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:01.863 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:01.863 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:01.863 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.864 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:01.864 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000006", 2023-07-29T00:25:01.864 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:01.864 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:01.864 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:01.864 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 23 2023-07-29T00:25:01.864 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.865 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:01.865 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:01.865 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:01.865 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:01.865 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:25:01.865 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.624215Z", 2023-07-29T00:25:01.865 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:01.866 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:01.866 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:01.866 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:01.866 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:01.866 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:01.866 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:01.866 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:01.867 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.867 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.16065178284022020642", 2023-07-29T00:25:01.867 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:01.867 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:01.867 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:01.867 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.867 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:01.868 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000007", 2023-07-29T00:25:01.868 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:01.868 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:01.868 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:01.868 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 11 2023-07-29T00:25:01.868 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.868 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:01.869 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:01.869 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:01.869 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:01.869 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:25:01.869 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.887579Z", 2023-07-29T00:25:01.869 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:01.869 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:01.870 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:01.870 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:01.870 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:01.870 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:01.870 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:01.870 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:01.870 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.871 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.16492125305849619010", 2023-07-29T00:25:01.871 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:01.871 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:01.871 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:01.871 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.871 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:01.871 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000008", 2023-07-29T00:25:01.872 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:01.872 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:01.872 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:01.872 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 9 2023-07-29T00:25:01.872 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.872 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:01.872 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:01.873 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:01.873 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:01.873 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:01.873 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:25:00.226007Z", 2023-07-29T00:25:01.873 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:01.873 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:01.873 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:01.874 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:01.874 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:01.874 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:01.874 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:01.874 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:01.874 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.874 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.823093922136247128", 2023-07-29T00:25:01.875 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:01.875 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:01.875 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:01.875 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.875 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:01.875 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000009", 2023-07-29T00:25:01.875 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:01.876 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:01.876 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:01.876 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 52 2023-07-29T00:25:01.876 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.876 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:01.876 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:01.876 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:01.877 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:01.877 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:25:01.877 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.500645Z", 2023-07-29T00:25:01.877 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:01.877 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:01.877 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:01.877 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:01.878 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:01.878 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:01.878 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:01.878 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:01.878 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.878 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.13138211920217424906", 2023-07-29T00:25:01.879 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:01.879 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:01.879 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:01.879 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.879 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:01.879 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000010", 2023-07-29T00:25:01.879 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:01.880 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:01.880 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:01.880 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 26 2023-07-29T00:25:01.880 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.880 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:01.880 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:01.880 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:01.881 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:01.881 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:25:01.881 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.665592Z", 2023-07-29T00:25:01.881 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:01.881 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:01.881 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:01.881 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:01.882 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:01.882 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:01.882 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:01.882 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:01.882 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.882 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.1934525061390349699", 2023-07-29T00:25:01.882 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:01.883 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:01.883 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:01.883 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.883 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:01.883 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000011", 2023-07-29T00:25:01.883 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:01.883 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:01.884 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:01.884 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 14 2023-07-29T00:25:01.884 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.884 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:01.884 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:01.884 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:01.884 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:01.885 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:01.885 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:58.028530Z", 2023-07-29T00:25:01.885 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:01.885 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:01.885 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:01.885 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:01.885 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:01.886 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:01.886 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:01.886 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:01.886 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.886 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.977105793370904765", 2023-07-29T00:25:01.886 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:01.886 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:01.887 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:01.887 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.887 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:01.887 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000012", 2023-07-29T00:25:01.887 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:01.887 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:01.887 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:01.888 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 30 2023-07-29T00:25:01.888 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.888 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:01.888 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:01.888 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:01.888 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:01.888 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 6, 2023-07-29T00:25:01.889 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.604469Z", 2023-07-29T00:25:01.889 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:01.889 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:01.889 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:01.889 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:01.889 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:01.889 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:01.890 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:01.890 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:01.890 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.890 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.5751379079694940857", 2023-07-29T00:25:01.890 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:01.890 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:01.890 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:01.891 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.891 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:01.891 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000013", 2023-07-29T00:25:01.891 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:01.891 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:01.891 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:01.891 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 10 2023-07-29T00:25:01.892 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.892 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:01.892 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:01.892 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:01.892 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:01.892 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:01.892 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.259576Z", 2023-07-29T00:25:01.893 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:01.893 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:01.893 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:01.893 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:01.893 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:01.893 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:01.893 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:01.894 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:01.894 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.894 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.17215352238431015123", 2023-07-29T00:25:01.894 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:01.894 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:01.894 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:01.894 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.895 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:01.895 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000014", 2023-07-29T00:25:01.895 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:01.895 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:01.895 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:01.895 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 14 2023-07-29T00:25:01.895 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.896 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:01.896 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:01.896 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:01.896 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:01.896 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:25:01.896 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.990758Z", 2023-07-29T00:25:01.896 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:01.897 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:01.897 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:01.897 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:01.897 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:01.897 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:01.897 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:01.897 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:01.898 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.898 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.17062728376196748983", 2023-07-29T00:25:01.898 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:01.898 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:01.898 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:01.898 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.898 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:01.899 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000015", 2023-07-29T00:25:01.899 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:01.899 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:01.899 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:01.899 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 10 2023-07-29T00:25:01.899 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.899 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:01.900 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:01.900 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:01.900 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:01.900 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:01.900 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.353507Z", 2023-07-29T00:25:01.900 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:01.900 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:01.901 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:01.901 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:01.901 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:01.901 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:01.901 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:01.901 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:01.901 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.902 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.18242424736600084008", 2023-07-29T00:25:01.902 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:01.902 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:01.902 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:01.902 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.902 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:01.902 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000016", 2023-07-29T00:25:01.903 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:01.903 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:01.903 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:01.903 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 22 2023-07-29T00:25:01.903 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.903 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:01.903 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:01.904 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:01.904 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:01.904 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:01.904 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.056052Z", 2023-07-29T00:25:01.904 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:01.904 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:01.904 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:01.905 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:01.905 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:01.905 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:01.905 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:01.905 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:01.905 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.905 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.7113266219568788703", 2023-07-29T00:25:01.906 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:01.906 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:01.906 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:01.906 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.906 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:01.906 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000017", 2023-07-29T00:25:01.906 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:01.907 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:01.907 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:01.907 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 6 2023-07-29T00:25:01.907 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.907 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:01.907 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:01.907 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:01.908 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:01.908 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:01.908 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:25:00.130458Z", 2023-07-29T00:25:01.908 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:01.908 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:01.908 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:01.908 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:01.909 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:01.909 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:01.909 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:01.909 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:01.909 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.909 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.9244210466416906534", 2023-07-29T00:25:01.909 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:01.910 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:01.910 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:01.910 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.910 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:01.910 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000018", 2023-07-29T00:25:01.910 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:01.910 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:01.911 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:01.911 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 1 2023-07-29T00:25:01.911 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.911 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:01.911 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:01.911 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:01.911 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:01.912 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:01.912 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:58.540506Z", 2023-07-29T00:25:01.912 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:01.912 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:01.912 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:01.912 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:01.913 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:01.913 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:01.913 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:01.913 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:01.913 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.913 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.6809827068441862987", 2023-07-29T00:25:01.913 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:01.914 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:01.914 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:01.914 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.914 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:01.914 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000019", 2023-07-29T00:25:01.914 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:01.914 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:01.915 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:01.915 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 54 2023-07-29T00:25:01.915 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.915 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:01.915 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:01.915 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:01.915 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:01.915 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:25:01.916 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.969687Z", 2023-07-29T00:25:01.916 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:01.916 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:01.916 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:01.916 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:01.916 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:01.917 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:01.917 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:01.917 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:01.917 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.917 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.2633976816737578561", 2023-07-29T00:25:01.917 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:01.917 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:01.918 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:01.918 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.918 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:01.918 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000020", 2023-07-29T00:25:01.918 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:01.918 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:01.918 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:01.919 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 6 2023-07-29T00:25:01.919 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.919 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:01.919 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:01.919 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:01.919 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:01.919 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:25:01.920 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.538340Z", 2023-07-29T00:25:01.920 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:01.920 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:01.920 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:01.920 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:01.920 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:01.920 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:01.921 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:01.921 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:01.921 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.921 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.5612449121322646980", 2023-07-29T00:25:01.921 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:01.921 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:01.921 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:01.922 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.922 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:01.922 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000021", 2023-07-29T00:25:01.922 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:01.922 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:01.922 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:01.922 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 27 2023-07-29T00:25:01.923 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.923 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:01.923 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:01.923 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:01.923 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:01.923 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:25:01.923 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.824586Z", 2023-07-29T00:25:01.924 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:01.924 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:01.924 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:01.924 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:01.924 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:01.924 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:01.924 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:01.925 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:01.925 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.925 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.17132723804101073931", 2023-07-29T00:25:01.925 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:01.925 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:01.925 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:01.925 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.926 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:01.926 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000022", 2023-07-29T00:25:01.926 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:01.926 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:01.926 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:01.926 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 4 2023-07-29T00:25:01.926 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.927 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:01.927 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:01.927 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:01.927 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:01.927 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:25:01.927 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:58.700216Z", 2023-07-29T00:25:01.927 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:01.928 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:01.928 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:01.928 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:01.928 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:01.928 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:01.928 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:01.928 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:01.929 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.929 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.16852150151163857895", 2023-07-29T00:25:01.929 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:01.929 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:01.929 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:01.929 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.929 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:01.930 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000023", 2023-07-29T00:25:01.930 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:01.930 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:01.930 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:01.930 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 1 2023-07-29T00:25:01.930 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.930 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:01.931 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:01.931 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:01.931 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:01.931 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:25:01.931 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:58.380161Z", 2023-07-29T00:25:01.931 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:01.931 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:01.932 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:01.932 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:01.932 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:01.932 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:01.932 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:01.932 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:01.932 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.933 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.12135900731490591290", 2023-07-29T00:25:01.933 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:01.933 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:01.933 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:01.933 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.933 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:01.933 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000024", 2023-07-29T00:25:01.934 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:01.934 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:01.934 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:01.934 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 24 2023-07-29T00:25:01.934 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.935 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:01.935 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:01.935 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:01.935 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:01.935 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:25:01.935 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.435609Z", 2023-07-29T00:25:01.935 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:01.936 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:01.936 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:01.936 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:01.936 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:01.936 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:01.936 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:01.936 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:01.937 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.937 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.953732692646569510", 2023-07-29T00:25:01.937 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:01.937 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:01.937 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:01.937 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.937 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:01.938 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000025", 2023-07-29T00:25:01.938 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:01.938 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:01.938 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:01.938 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 5 2023-07-29T00:25:01.938 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.938 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:01.939 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:01.939 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:01.939 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:01.939 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:01.939 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:56.911728Z", 2023-07-29T00:25:01.939 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:01.940 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:01.940 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:01.940 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:01.941 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:01.941 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:01.941 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:01.941 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:01.941 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.941 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.11991851580865620776", 2023-07-29T00:25:01.941 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:01.942 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:01.942 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:01.942 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.942 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:01.942 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000026", 2023-07-29T00:25:01.942 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:01.942 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:01.943 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:01.943 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 11 2023-07-29T00:25:01.943 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.943 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:01.943 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:01.943 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:01.943 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:01.944 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:25:01.944 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.170768Z", 2023-07-29T00:25:01.944 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:01.944 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:01.944 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:01.945 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:01.945 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:01.945 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:01.945 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:01.945 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:01.945 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.946 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.13771791244517797875", 2023-07-29T00:25:01.946 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:01.946 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:01.946 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:01.946 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.946 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:01.946 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000027", 2023-07-29T00:25:01.947 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:01.947 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:01.947 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:01.947 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 26 2023-07-29T00:25:01.947 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.947 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:01.947 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:01.948 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:01.948 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:01.948 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:01.948 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.909186Z", 2023-07-29T00:25:01.948 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:01.948 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:01.948 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:01.949 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:01.949 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:01.949 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:01.949 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:01.949 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:01.949 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.949 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.16266574207289898849", 2023-07-29T00:25:01.950 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:01.950 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:01.950 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:01.950 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.950 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:01.950 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000028", 2023-07-29T00:25:01.950 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:01.951 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:01.951 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:01.951 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 2 2023-07-29T00:25:01.951 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.951 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:01.951 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:01.951 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:01.952 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:01.952 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:25:01.952 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:53.541854Z", 2023-07-29T00:25:01.952 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:01.952 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:01.952 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:01.952 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:01.953 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:01.953 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:01.953 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:01.953 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:01.953 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.953 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.8226690733168497106", 2023-07-29T00:25:01.953 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:01.954 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:01.954 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:01.954 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.954 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:01.954 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000029", 2023-07-29T00:25:01.954 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:01.954 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:01.955 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:01.955 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 8 2023-07-29T00:25:01.955 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.955 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:01.955 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:01.955 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:01.955 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:01.956 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 6, 2023-07-29T00:25:01.956 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:58.285139Z", 2023-07-29T00:25:01.956 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:01.956 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:01.956 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:01.956 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:01.956 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:01.957 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:01.957 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:01.957 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:01.957 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.957 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.17518188341365681679", 2023-07-29T00:25:01.957 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:01.957 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:01.958 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:01.958 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.958 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:01.958 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000030", 2023-07-29T00:25:01.958 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:01.958 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:01.958 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:01.959 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 8 2023-07-29T00:25:01.959 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.959 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:01.959 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:01.959 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:01.959 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:01.959 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:01.960 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.078383Z", 2023-07-29T00:25:01.960 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:01.960 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:01.960 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:01.960 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:01.960 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:01.960 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:01.961 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:01.961 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:01.961 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.961 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.3676984154818963734", 2023-07-29T00:25:01.961 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:01.961 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:01.961 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:01.962 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.962 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:01.962 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000031", 2023-07-29T00:25:01.962 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:01.962 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:01.962 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:01.962 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 3 2023-07-29T00:25:01.963 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.963 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:01.963 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:01.963 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:01.963 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:01.963 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:25:01.963 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:57.013288Z", 2023-07-29T00:25:01.964 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:01.964 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:01.964 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:01.964 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:01.964 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:01.964 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:01.965 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:01.965 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:01.965 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.965 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.5629438624477504228", 2023-07-29T00:25:01.965 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:01.965 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:01.965 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:01.966 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.966 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:01.966 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000032", 2023-07-29T00:25:01.966 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:01.966 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:01.966 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:01.966 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 2 2023-07-29T00:25:01.967 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.967 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:01.967 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:01.967 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:01.967 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:01.967 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:25:01.967 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:53.877366Z", 2023-07-29T00:25:01.968 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:01.968 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:01.968 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:01.968 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:01.968 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:01.968 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:01.968 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:01.969 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:01.969 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.969 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.7091727799349100745", 2023-07-29T00:25:01.969 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:01.969 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:01.969 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:01.969 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.970 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:01.970 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000033", 2023-07-29T00:25:01.970 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:01.970 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:01.970 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:01.970 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 9 2023-07-29T00:25:01.970 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.971 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:01.971 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:01.971 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:01.971 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:01.971 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:01.971 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:25:00.051388Z", 2023-07-29T00:25:01.971 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:01.972 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:01.972 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:01.972 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:01.972 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:01.972 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:01.972 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:01.972 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:01.973 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.973 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.12671585569024816369", 2023-07-29T00:25:01.973 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:01.973 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:01.973 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:01.973 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.973 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:01.974 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000036", 2023-07-29T00:25:01.974 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:01.974 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:01.974 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:01.974 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 3 2023-07-29T00:25:01.974 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.974 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:01.975 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:01.975 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:01.975 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:01.975 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:25:01.975 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.146626Z", 2023-07-29T00:25:01.975 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:01.975 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:01.976 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:01.976 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:01.976 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:01.976 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:01.976 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:01.976 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:01.976 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.977 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.907453267601429398", 2023-07-29T00:25:01.977 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:01.977 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:01.977 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:01.977 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.977 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:01.977 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000039", 2023-07-29T00:25:01.978 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:01.978 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:01.978 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:01.978 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 5 2023-07-29T00:25:01.978 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.978 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:01.978 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:01.979 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:01.979 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:01.979 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:25:01.979 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:55.646289Z", 2023-07-29T00:25:01.979 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:01.979 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:01.979 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:01.980 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:01.980 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:01.980 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:01.980 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:01.980 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:01.980 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.980 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.5591276218565079146", 2023-07-29T00:25:01.981 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:01.981 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:01.981 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:01.981 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.981 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:01.981 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000040", 2023-07-29T00:25:01.981 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:01.982 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:01.982 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:01.982 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 9 2023-07-29T00:25:01.982 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.982 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:01.982 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:01.982 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:01.983 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:01.983 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:25:01.983 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.326766Z", 2023-07-29T00:25:01.983 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:01.983 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:01.983 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:01.983 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:01.984 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:01.984 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:01.984 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:01.984 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:01.984 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.984 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.9758499640971574089", 2023-07-29T00:25:01.984 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:01.985 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:01.985 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:01.985 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.985 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:01.985 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000042", 2023-07-29T00:25:01.985 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:01.985 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:01.986 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:01.986 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 2 2023-07-29T00:25:01.986 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.986 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:01.986 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:01.986 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:01.986 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:01.987 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 6, 2023-07-29T00:25:01.987 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:56.845159Z", 2023-07-29T00:25:01.987 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:01.987 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:01.987 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:01.987 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:01.987 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:01.988 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:01.988 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:01.988 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:01.988 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.988 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.8590978594517257905", 2023-07-29T00:25:01.988 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:01.988 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:01.989 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:01.989 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.989 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:01.989 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000045", 2023-07-29T00:25:01.989 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:01.989 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:01.989 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:01.990 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 5 2023-07-29T00:25:01.990 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.990 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:01.990 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:01.990 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:01.990 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:01.990 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:01.991 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:25:00.070217Z", 2023-07-29T00:25:01.991 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:01.991 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:01.991 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:01.991 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:01.991 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:01.991 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:01.992 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:01.992 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:01.992 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.992 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.16065695887466239032", 2023-07-29T00:25:01.992 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:01.992 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:01.992 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:01.993 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.993 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:01.993 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000046", 2023-07-29T00:25:01.993 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:01.993 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:01.993 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:01.993 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 23 2023-07-29T00:25:01.994 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.994 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:01.994 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:01.994 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:01.994 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:01.994 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:25:01.994 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:55.220568Z", 2023-07-29T00:25:01.995 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:01.995 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:01.995 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:01.995 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:01.995 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:01.995 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:01.996 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:01.996 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:01.996 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.996 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.11546401740231357919", 2023-07-29T00:25:01.996 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:01.996 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:01.996 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:01.997 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.997 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:01.997 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000048", 2023-07-29T00:25:01.997 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:01.997 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:01.997 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:01.997 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 4 2023-07-29T00:25:01.998 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:01.998 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:01.998 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:01.998 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:01.998 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:01.998 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:01.998 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.028892Z", 2023-07-29T00:25:01.999 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:01.999 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:01.999 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:01.999 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:01.999 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:01.999 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:01.999 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.000 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.000 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.000 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.3530098604923006837", 2023-07-29T00:25:02.000 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.000 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.000 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.000 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.001 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.001 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000049", 2023-07-29T00:25:02.001 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.001 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.001 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:02.001 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 10 2023-07-29T00:25:02.001 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.002 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.002 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.002 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.002 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.002 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:02.002 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:56.685390Z", 2023-07-29T00:25:02.002 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.003 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.003 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.003 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.003 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.003 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.003 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.003 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.004 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.004 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.8645762638904408176", 2023-07-29T00:25:02.004 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.004 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.004 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.004 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.004 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.005 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000050", 2023-07-29T00:25:02.005 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.005 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.005 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:02.005 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 7 2023-07-29T00:25:02.005 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.005 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.005 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.006 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.006 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.006 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:02.006 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:57.938921Z", 2023-07-29T00:25:02.006 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.006 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.007 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.007 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.007 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.007 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.007 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.007 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.007 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.007 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.7402286154998209153", 2023-07-29T00:25:02.008 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.008 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.008 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.008 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.008 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.008 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000053", 2023-07-29T00:25:02.009 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.009 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.009 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:02.009 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 19 2023-07-29T00:25:02.009 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.009 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.009 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.010 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.010 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.010 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:02.010 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:25:00.300321Z", 2023-07-29T00:25:02.010 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.010 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.010 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.010 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.011 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.011 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.011 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.011 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.011 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.011 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.3447432041722505824", 2023-07-29T00:25:02.011 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.012 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.012 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.012 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.012 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.012 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000055", 2023-07-29T00:25:02.012 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.012 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.013 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:02.013 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 1 2023-07-29T00:25:02.013 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.013 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.013 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.013 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.013 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.014 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:25:02.014 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:55.734741Z", 2023-07-29T00:25:02.014 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.014 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.014 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.014 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.014 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.015 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.015 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.015 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.015 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.015 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.103868334806658526", 2023-07-29T00:25:02.015 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.015 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.016 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.016 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.016 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.016 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000056", 2023-07-29T00:25:02.016 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.016 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.016 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:02.017 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 4 2023-07-29T00:25:02.017 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.017 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.017 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.017 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.017 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.017 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:25:02.018 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:57.604582Z", 2023-07-29T00:25:02.018 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.018 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.018 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.018 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.018 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.018 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.019 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.019 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.019 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.019 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.15328605550959615819", 2023-07-29T00:25:02.019 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.019 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.019 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.020 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.020 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.020 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000057", 2023-07-29T00:25:02.020 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.020 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.020 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:02.020 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 5 2023-07-29T00:25:02.021 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.021 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.021 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.021 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.021 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.021 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:25:02.021 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:25:00.207399Z", 2023-07-29T00:25:02.022 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.022 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.022 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.022 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.022 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.022 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.022 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.023 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.023 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.023 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.9667970389264524095", 2023-07-29T00:25:02.023 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.023 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.023 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.023 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.024 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.024 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000058", 2023-07-29T00:25:02.024 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.024 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.024 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:02.024 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 2 2023-07-29T00:25:02.025 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.025 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.025 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.025 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.025 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.025 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:02.025 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:55.988907Z", 2023-07-29T00:25:02.025 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.026 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.026 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.026 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.026 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.026 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.026 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.026 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.027 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.027 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.9949168434400984852", 2023-07-29T00:25:02.027 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.027 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.027 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.027 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.027 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.028 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000059", 2023-07-29T00:25:02.028 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.028 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.028 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:02.028 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 5 2023-07-29T00:25:02.028 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.028 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.029 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.029 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.029 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.029 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:25:02.029 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:56.419498Z", 2023-07-29T00:25:02.029 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.029 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.030 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.030 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.030 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.030 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.030 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.030 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.030 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.031 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.13067242925130165637", 2023-07-29T00:25:02.031 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.031 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.031 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.031 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.031 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.031 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000060", 2023-07-29T00:25:02.032 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.032 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.032 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:02.032 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 23 2023-07-29T00:25:02.032 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.032 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.032 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.033 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.033 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.033 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:02.033 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.520291Z", 2023-07-29T00:25:02.033 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.033 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.033 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.034 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.034 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.034 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.034 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.034 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.034 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.034 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.10363808317964223361", 2023-07-29T00:25:02.035 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.035 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.035 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.035 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.035 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.035 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000061", 2023-07-29T00:25:02.035 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.036 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.036 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:02.036 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 2 2023-07-29T00:25:02.036 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.036 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.036 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.037 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.037 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.037 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:25:02.037 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:58.597135Z", 2023-07-29T00:25:02.037 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.037 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.037 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.038 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.038 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.038 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.038 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.038 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.038 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.038 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.15119451868204805858", 2023-07-29T00:25:02.039 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.039 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.039 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.039 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.039 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.039 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000062", 2023-07-29T00:25:02.039 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.040 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.040 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:02.040 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 11 2023-07-29T00:25:02.040 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.040 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.040 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.040 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.041 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.041 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:02.041 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:25:00.281200Z", 2023-07-29T00:25:02.041 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.041 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.041 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.041 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.041 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.042 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.042 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.042 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.042 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.042 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.15004145449912175643", 2023-07-29T00:25:02.042 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.042 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.043 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.043 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.043 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.043 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000065", 2023-07-29T00:25:02.043 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.043 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.043 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:02.044 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 12 2023-07-29T00:25:02.044 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.044 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.044 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.044 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.044 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.044 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:25:02.045 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.805482Z", 2023-07-29T00:25:02.045 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.045 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.045 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.045 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.045 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.045 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.046 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.046 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.046 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.046 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.16800270459404528286", 2023-07-29T00:25:02.046 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.046 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.046 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.047 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.047 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.047 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000076", 2023-07-29T00:25:02.047 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.047 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.047 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:02.047 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 2 2023-07-29T00:25:02.048 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.048 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.048 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.048 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.048 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.048 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:25:02.048 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:53.955523Z", 2023-07-29T00:25:02.049 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.049 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.049 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.049 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.049 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.049 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.049 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.050 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.050 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.050 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.16036626992765080287", 2023-07-29T00:25:02.050 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.050 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.050 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.050 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.051 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.051 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000133", 2023-07-29T00:25:02.051 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.051 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.051 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:02.051 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 3 2023-07-29T00:25:02.051 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.052 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.052 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.052 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.052 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.052 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:25:02.052 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:57.710367Z", 2023-07-29T00:25:02.052 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.053 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.053 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.053 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.053 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.053 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.053 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.053 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.054 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.054 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.4073974230378024755", 2023-07-29T00:25:02.054 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.054 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.054 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.054 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.054 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.055 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000162", 2023-07-29T00:25:02.055 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.055 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.055 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:02.055 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 3 2023-07-29T00:25:02.055 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.055 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.056 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.056 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.056 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.056 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:02.056 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:57.876925Z", 2023-07-29T00:25:02.056 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.056 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.057 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.057 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.057 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.057 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.057 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.057 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.057 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.058 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.17120133882977046226", 2023-07-29T00:25:02.058 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.058 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.058 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.058 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.058 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.058 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000271", 2023-07-29T00:25:02.059 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.059 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.059 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:02.059 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 2 2023-07-29T00:25:02.059 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.059 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.059 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.060 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.060 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.060 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:25:02.060 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:58.148659Z", 2023-07-29T00:25:02.060 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.060 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.060 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.061 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.061 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.061 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.061 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.061 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.061 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.061 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.6641984461058886092", 2023-07-29T00:25:02.062 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.062 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.062 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.062 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.062 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.062 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000317", 2023-07-29T00:25:02.062 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.063 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.063 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:02.063 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 7 2023-07-29T00:25:02.063 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.063 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.063 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.063 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.063 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.064 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:02.064 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:58.614561Z", 2023-07-29T00:25:02.064 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.064 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.064 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.064 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.065 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.065 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.065 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.065 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.065 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.065 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.3113036617876191511", 2023-07-29T00:25:02.065 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.065 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.066 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.066 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.066 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.066 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000423", 2023-07-29T00:25:02.066 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.066 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.066 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:02.067 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 6 2023-07-29T00:25:02.067 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.067 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.067 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.067 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.067 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.067 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:25:02.068 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:57.967362Z", 2023-07-29T00:25:02.068 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.068 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.068 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.068 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.068 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.068 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.069 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.069 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.069 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.069 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.4896196950235447520", 2023-07-29T00:25:02.069 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.069 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.070 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.071 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.071 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.071 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000467", 2023-07-29T00:25:02.071 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.071 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.071 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:25:02.071 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 2 2023-07-29T00:25:02.072 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.072 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.072 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.072 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.072 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.072 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:25:02.073 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:58.460850Z", 2023-07-29T00:25:02.073 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.073 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.073 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.073 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.073 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.073 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.074 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.074 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.074 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.074 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.17656770823427161085", 2023-07-29T00:25:02.074 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.074 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.074 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.075 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:02.075 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T00:25:02.075 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-58', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:25:02.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-58', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:25:02.076 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-58 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:25:02.126 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:25:02.126 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:25:02.157 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T00:25:02.157 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.157 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000000", 2023-07-29T00:25:02.157 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.157 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.158 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.158 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 4 2023-07-29T00:25:02.158 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.158 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.158 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.158 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.159 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.159 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:25:02.159 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.725669Z", 2023-07-29T00:25:02.159 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.159 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.159 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.159 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.160 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.160 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.160 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.160 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.160 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.160 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_slTUEpfBO5jVBj6GsaB6G9UgLni3JsP", 2023-07-29T00:25:02.161 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.161 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.161 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.161 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.161 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.161 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000001", 2023-07-29T00:25:02.161 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.162 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.162 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.162 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 8 2023-07-29T00:25:02.162 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.162 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.162 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.162 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.163 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.163 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:02.163 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.749457Z", 2023-07-29T00:25:02.163 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.163 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.163 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.163 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.164 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.164 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.164 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.164 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.165 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.165 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_vp09hCJbfML2C4DclfS3f2SN5yfcnqt", 2023-07-29T00:25:02.165 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.165 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.165 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.165 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.165 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.166 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000002", 2023-07-29T00:25:02.166 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.166 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.166 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.166 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 2 2023-07-29T00:25:02.166 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.166 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.167 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.167 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.167 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.167 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:02.167 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.644274Z", 2023-07-29T00:25:02.167 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.168 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.168 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.168 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.168 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.168 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.168 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.169 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.169 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.169 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_bQQcvDxYdcGYWMq9mLyojConVZ3F4Yh", 2023-07-29T00:25:02.169 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.169 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.169 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.169 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.170 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.170 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000003", 2023-07-29T00:25:02.170 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.170 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.170 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.170 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 6 2023-07-29T00:25:02.170 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.171 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.171 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.171 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.171 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.171 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 6, 2023-07-29T00:25:02.171 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:25:00.149911Z", 2023-07-29T00:25:02.171 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.172 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.172 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.172 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.172 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.172 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.172 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.172 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.173 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.173 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_tUiwTH9dWDlU3zRd-h2k9EzV97D4x14", 2023-07-29T00:25:02.173 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.173 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.173 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.173 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.174 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.174 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000004", 2023-07-29T00:25:02.174 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.174 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.174 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.174 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 10 2023-07-29T00:25:02.174 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.175 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.175 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.175 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.175 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.175 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:25:02.175 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.233760Z", 2023-07-29T00:25:02.175 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.176 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.176 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.176 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.176 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.176 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.176 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.176 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.177 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.177 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "__PVrn_aHIs3HkaDQi83TkWY0pxEkumy", 2023-07-29T00:25:02.177 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.177 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.177 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.177 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.177 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.178 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000005", 2023-07-29T00:25:02.178 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.178 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.178 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.178 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 3 2023-07-29T00:25:02.178 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.178 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.179 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.179 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.179 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.179 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 6, 2023-07-29T00:25:02.179 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.414494Z", 2023-07-29T00:25:02.179 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.179 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.180 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.180 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.180 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.180 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.180 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.180 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.180 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.181 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_Y4FICw1yAGHK5GgzZ1WJ3C3Ugc83OHY", 2023-07-29T00:25:02.181 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.181 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.181 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.181 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.181 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.181 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000006", 2023-07-29T00:25:02.182 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.182 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.182 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.182 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 9 2023-07-29T00:25:02.182 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.182 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.182 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.183 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.183 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.183 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 6, 2023-07-29T00:25:02.183 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.624215Z", 2023-07-29T00:25:02.183 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.183 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.183 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.184 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.184 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.184 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.184 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.184 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.184 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.184 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_4fl5D3sxCSBVYxYAB356pKHpAwVp9B4", 2023-07-29T00:25:02.185 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.185 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.185 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.185 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.185 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.185 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000007", 2023-07-29T00:25:02.185 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.186 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.186 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.186 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 3 2023-07-29T00:25:02.186 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.186 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.186 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.186 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.187 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.187 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:25:02.187 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.887579Z", 2023-07-29T00:25:02.187 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.187 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.187 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.187 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.188 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.188 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.188 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.188 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.188 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.188 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_ed48DoaB5fQpNfIAaW5kiCitkjVjTdm", 2023-07-29T00:25:02.188 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.189 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.189 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.189 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.189 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.189 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000008", 2023-07-29T00:25:02.189 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.189 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.190 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.190 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 3 2023-07-29T00:25:02.190 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.190 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.190 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.190 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.190 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.191 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:02.191 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:25:00.226007Z", 2023-07-29T00:25:02.191 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.191 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.191 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.191 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.191 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.192 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.192 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.192 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.192 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.192 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_WmYXvPefcC1MDL_5Iefpu3a0p_KhPw5", 2023-07-29T00:25:02.192 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.193 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.193 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.193 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.193 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.193 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000009", 2023-07-29T00:25:02.193 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.193 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.194 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.194 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 38 2023-07-29T00:25:02.194 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.194 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.194 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.194 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.194 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.194 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 6, 2023-07-29T00:25:02.195 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.500645Z", 2023-07-29T00:25:02.195 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.195 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.195 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.195 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.195 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.195 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.196 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.196 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.196 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.196 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_4LhzGInFycSvS9-6_uqI5dADglU5kaE", 2023-07-29T00:25:02.196 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.196 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.197 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.197 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.197 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.197 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000010", 2023-07-29T00:25:02.197 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.197 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.197 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.197 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 13 2023-07-29T00:25:02.198 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.198 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.198 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.198 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.198 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.198 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:02.199 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.665592Z", 2023-07-29T00:25:02.199 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.199 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.199 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.199 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.199 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.199 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.200 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.200 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.200 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.200 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_xQ9B7iYIb6ihyYrZE-viEkG4xfqz059", 2023-07-29T00:25:02.200 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.200 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.200 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.201 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.201 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.201 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000011", 2023-07-29T00:25:02.201 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.201 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.201 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.201 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 6 2023-07-29T00:25:02.202 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.202 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.202 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.202 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.202 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.202 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:02.202 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:58.028530Z", 2023-07-29T00:25:02.203 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.203 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.203 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.203 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.203 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.203 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.203 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.204 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.204 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.204 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_SN_ttOk3J6jG6MvpHS-UluuYHhGPK91", 2023-07-29T00:25:02.204 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.204 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.204 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.204 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.205 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.205 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000012", 2023-07-29T00:25:02.205 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.205 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.205 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.205 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 11 2023-07-29T00:25:02.205 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.206 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.206 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.206 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.206 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.206 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:02.206 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.604469Z", 2023-07-29T00:25:02.206 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.207 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.207 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.207 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.207 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.207 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.207 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.207 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.208 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.208 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_C9BI7wVEU34qwi3qNOwEXfX1NovJj2h", 2023-07-29T00:25:02.208 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.208 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.208 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.208 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.208 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.209 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000013", 2023-07-29T00:25:02.209 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.209 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.209 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.209 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 3 2023-07-29T00:25:02.209 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.209 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.210 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.210 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.210 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.210 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:25:02.210 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.259576Z", 2023-07-29T00:25:02.210 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.210 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.211 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.211 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.211 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.211 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.211 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.211 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.211 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.212 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_QtOY06IMVWT1KZBrJ9NGEild5vqLZhS", 2023-07-29T00:25:02.212 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.212 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.212 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.212 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.212 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.213 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000014", 2023-07-29T00:25:02.213 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.213 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.213 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.213 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 7 2023-07-29T00:25:02.213 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.213 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.214 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.214 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.214 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.214 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:02.214 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.990758Z", 2023-07-29T00:25:02.214 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.214 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.214 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.215 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.215 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.215 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.215 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.216 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.216 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.216 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_ki-HdX8JyzPTrFQBRTv1iZ13BkNyRD5", 2023-07-29T00:25:02.216 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.216 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.216 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.216 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.217 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.217 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000015", 2023-07-29T00:25:02.217 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.217 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.217 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.217 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 4 2023-07-29T00:25:02.217 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.218 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.218 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.218 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.218 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.218 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:25:02.218 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.353507Z", 2023-07-29T00:25:02.219 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.219 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.219 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.219 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.219 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.219 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.219 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.220 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.220 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.220 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_R94pZiBy1k7xMoC4bNhWNdlIk7Gw5Cn", 2023-07-29T00:25:02.220 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.220 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.220 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.221 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.221 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.221 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000016", 2023-07-29T00:25:02.221 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.221 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.221 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.221 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 11 2023-07-29T00:25:02.222 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.222 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.222 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.222 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.222 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.222 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:25:02.222 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.056052Z", 2023-07-29T00:25:02.223 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.223 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.223 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.223 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.223 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.223 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.223 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.224 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.224 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.224 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_2BFx_RpxvKcypOtK06Hz0QiERri3ELL", 2023-07-29T00:25:02.224 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.224 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.224 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.224 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.225 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.225 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000017", 2023-07-29T00:25:02.225 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.225 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.225 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.225 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 2 2023-07-29T00:25:02.225 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.226 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.226 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.226 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.226 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.226 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:25:02.226 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:25:00.130458Z", 2023-07-29T00:25:02.226 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.227 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.227 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.227 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.227 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.227 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.228 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.228 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.228 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.228 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_fNhhWpJrKQCQnggc-GSM1NrFASCdlG0", 2023-07-29T00:25:02.228 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.228 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.228 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.229 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.229 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.229 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000018", 2023-07-29T00:25:02.229 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.229 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.229 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.229 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 2 2023-07-29T00:25:02.230 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.230 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.230 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.230 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.230 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.230 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:25:02.230 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:58.540506Z", 2023-07-29T00:25:02.231 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.231 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.231 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.231 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.231 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.231 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.231 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.232 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.232 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.232 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_5JwpVkaWMv2KIcBtKYG0TkLeOIEeDRi", 2023-07-29T00:25:02.232 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.232 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.232 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.233 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.233 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.233 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000019", 2023-07-29T00:25:02.233 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.233 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.233 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.234 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 37 2023-07-29T00:25:02.234 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.234 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.234 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.234 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.234 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.234 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:25:02.235 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.969687Z", 2023-07-29T00:25:02.235 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.235 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.235 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.235 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.235 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.235 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.236 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.236 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.236 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.236 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_z9QLke8psECMRiv5V5b1aYTSuVAd6vG", 2023-07-29T00:25:02.236 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.236 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.236 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.237 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.237 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.237 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000020", 2023-07-29T00:25:02.237 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.237 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.237 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.237 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 2 2023-07-29T00:25:02.238 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.238 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.238 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.238 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.238 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.238 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:25:02.238 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.538340Z", 2023-07-29T00:25:02.239 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.239 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.239 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.239 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.239 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.239 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.239 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.240 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.240 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.240 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_hdqnFKk7rm22X01P8Jinjx2Te0V83s-", 2023-07-29T00:25:02.240 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.240 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.240 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.240 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.241 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.241 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000021", 2023-07-29T00:25:02.241 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.241 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.241 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.241 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 5 2023-07-29T00:25:02.241 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.242 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.242 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.242 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.242 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.242 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:02.242 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.824586Z", 2023-07-29T00:25:02.242 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.243 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.243 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.243 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.243 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.243 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.243 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.244 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.244 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.244 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_9Lt7k3mueTuDzyKvom8BQ1a4G_deRcX", 2023-07-29T00:25:02.244 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.244 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.244 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.244 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.245 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.245 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000022", 2023-07-29T00:25:02.245 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.245 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.245 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.245 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 4 2023-07-29T00:25:02.245 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.246 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.246 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.246 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.246 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.246 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:02.246 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:58.700216Z", 2023-07-29T00:25:02.246 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.247 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.247 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.247 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.247 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.247 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.247 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.247 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.248 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.248 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_5liilFLCFZaxLMDX1M6p6ay6X-93CjM", 2023-07-29T00:25:02.248 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.248 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.248 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.248 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.248 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.249 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000023", 2023-07-29T00:25:02.249 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.249 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.249 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.249 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 1 2023-07-29T00:25:02.249 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.249 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.250 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.250 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.250 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.250 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:02.250 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:58.380161Z", 2023-07-29T00:25:02.250 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.250 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.251 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.251 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.251 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.251 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.251 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.251 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.251 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.252 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_WzDjHFC74Y1sSCbCWDn4bC09QBbpy17", 2023-07-29T00:25:02.252 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.252 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.252 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.252 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.252 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.252 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000024", 2023-07-29T00:25:02.253 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.253 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.253 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.253 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 12 2023-07-29T00:25:02.253 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.253 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.253 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.254 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.254 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.254 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:25:02.254 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.435609Z", 2023-07-29T00:25:02.254 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.254 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.254 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.255 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.255 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.255 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.255 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.255 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.255 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.255 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_bqLKJRYIhxLBy2NbhroyB8jND0UeOwE", 2023-07-29T00:25:02.256 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.256 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.256 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.256 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.256 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.256 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000025", 2023-07-29T00:25:02.256 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.257 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.257 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.257 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 5 2023-07-29T00:25:02.257 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.257 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.257 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.257 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.258 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.258 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:02.258 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:56.911728Z", 2023-07-29T00:25:02.258 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.258 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.258 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.258 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.259 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.259 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.259 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.259 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.259 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.259 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_8xSuKvRmEryjCIQFzF30lUpVmCgGQ8y", 2023-07-29T00:25:02.259 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.260 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.260 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.260 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.260 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.260 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000026", 2023-07-29T00:25:02.260 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.261 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.261 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.261 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 5 2023-07-29T00:25:02.261 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.261 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.261 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.261 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.262 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.262 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:25:02.262 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.170768Z", 2023-07-29T00:25:02.262 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.262 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.262 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.262 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.263 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.263 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.263 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.263 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.263 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.263 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_1gj4JRNtZYZzmge0SP-ZH9DPIa-hlHL", 2023-07-29T00:25:02.263 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.264 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.264 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.264 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.264 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.264 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000027", 2023-07-29T00:25:02.264 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.264 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.265 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.265 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 21 2023-07-29T00:25:02.265 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.265 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.265 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.265 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.265 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.266 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:02.266 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.909186Z", 2023-07-29T00:25:02.266 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.266 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.266 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.266 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.266 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.267 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.267 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.267 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.267 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.267 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_hJrPOIvk_cCS9IBKWGbdGN6MwtAWy16", 2023-07-29T00:25:02.267 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.267 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.268 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.268 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.268 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.268 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000028", 2023-07-29T00:25:02.268 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.268 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.268 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.269 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 2 2023-07-29T00:25:02.269 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.269 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.269 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.269 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.269 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.269 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:25:02.270 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:53.541854Z", 2023-07-29T00:25:02.270 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.270 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.270 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.270 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.270 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.270 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.271 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.271 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.271 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.271 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_t0nl74dFRLfrNaQHcnlYTF_LnF3ZDlB", 2023-07-29T00:25:02.271 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.271 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.271 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.272 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.272 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.272 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000029", 2023-07-29T00:25:02.272 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.272 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.272 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.273 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 3 2023-07-29T00:25:02.273 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.273 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.273 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.273 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.273 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.273 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:02.274 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:58.285139Z", 2023-07-29T00:25:02.274 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.274 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.274 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.274 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.274 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.274 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.274 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.275 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.275 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.275 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_bX0aH-SXlKMSWwLjXxhu8nyDFXrNQPc", 2023-07-29T00:25:02.275 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.275 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.275 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.276 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.276 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.276 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000030", 2023-07-29T00:25:02.276 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.276 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.276 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.276 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 2 2023-07-29T00:25:02.277 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.277 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.277 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.277 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.277 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.277 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 6, 2023-07-29T00:25:02.277 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.078383Z", 2023-07-29T00:25:02.278 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.278 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.278 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.278 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.278 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.278 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.278 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.279 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.279 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.279 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_yj0xXpPvj-DeVH3wfoQSxGe48zPVkTa", 2023-07-29T00:25:02.279 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.279 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.279 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.279 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.280 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.280 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000031", 2023-07-29T00:25:02.280 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.280 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.280 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.280 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 5 2023-07-29T00:25:02.281 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.281 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.281 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.281 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.281 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.281 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:25:02.281 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:57.013288Z", 2023-07-29T00:25:02.282 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.282 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.282 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.282 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.282 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.282 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.282 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.283 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.283 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.283 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_nGajaE5HbeTdWRQk_o2sztPm66mfwR0", 2023-07-29T00:25:02.283 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.283 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.283 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.284 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.284 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.284 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000032", 2023-07-29T00:25:02.284 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.284 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.284 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.284 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 2 2023-07-29T00:25:02.285 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.285 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.285 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.285 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.285 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.285 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:25:02.285 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:53.877366Z", 2023-07-29T00:25:02.286 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.286 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.286 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.286 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.286 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.286 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.286 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.287 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.287 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.287 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_iX6tiUpA_5K5QO_PVnBkQOAGGUXwtx7", 2023-07-29T00:25:02.287 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.287 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.287 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.287 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.288 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.288 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000033", 2023-07-29T00:25:02.288 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.288 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.288 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.288 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 3 2023-07-29T00:25:02.289 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.289 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.289 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.289 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.289 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.289 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:02.289 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:25:00.051388Z", 2023-07-29T00:25:02.290 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.290 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.290 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.290 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.290 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.290 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.290 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.291 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.291 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.291 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_gLC5B0ObXBfuVrYmg1kgOzH3nzTYFWP", 2023-07-29T00:25:02.291 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.291 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.291 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.291 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.292 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.292 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000036", 2023-07-29T00:25:02.292 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.292 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.292 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.292 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 3 2023-07-29T00:25:02.292 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.293 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.293 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.293 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.293 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.293 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:02.293 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.146626Z", 2023-07-29T00:25:02.293 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.294 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.294 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.294 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.294 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.294 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.294 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.294 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.295 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.295 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_grINbN2SERC7_CEmjQ4FS4YHu0cE4-H", 2023-07-29T00:25:02.295 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.295 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.295 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.295 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.295 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.296 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000039", 2023-07-29T00:25:02.296 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.296 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.296 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.296 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 4 2023-07-29T00:25:02.296 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.297 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.297 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.297 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.297 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.297 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:02.297 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:55.646289Z", 2023-07-29T00:25:02.297 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.298 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.298 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.298 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.298 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.298 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.298 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.298 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.299 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.299 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_Z0Jwm0DG5P9IbonhiGOE2z5Q52-D3BF", 2023-07-29T00:25:02.299 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.299 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.299 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.299 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.299 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.300 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000040", 2023-07-29T00:25:02.300 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.300 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.300 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.300 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 5 2023-07-29T00:25:02.300 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.300 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.301 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.301 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.301 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.301 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:25:02.301 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.326766Z", 2023-07-29T00:25:02.301 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.301 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.302 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.302 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.302 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.302 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.302 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.302 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.302 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.303 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_aW8ZzoHFm9Hjw5WYi6acJKPmYTaFcw4", 2023-07-29T00:25:02.303 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.303 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.303 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.303 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.303 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.303 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000042", 2023-07-29T00:25:02.304 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.304 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.304 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.304 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 2 2023-07-29T00:25:02.304 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.304 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.305 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.305 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.305 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.305 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:25:02.305 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:56.845159Z", 2023-07-29T00:25:02.305 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.305 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.306 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.306 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.306 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.306 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.306 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.306 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.306 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.307 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_uujPFbfTNsKtcHlOFjkeVOA7c6fErLw", 2023-07-29T00:25:02.307 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.307 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.307 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.307 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.307 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.307 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000045", 2023-07-29T00:25:02.308 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.308 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.308 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.308 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 3 2023-07-29T00:25:02.308 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.308 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.308 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.309 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.309 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.309 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:02.309 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:25:00.070217Z", 2023-07-29T00:25:02.309 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.309 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.309 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.310 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.310 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.310 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.313 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.313 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.313 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.313 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_Vf_6F3KlKn2ydeFNm5cpq0azS7Fle5o", 2023-07-29T00:25:02.313 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.313 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.313 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.314 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.314 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.314 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000046", 2023-07-29T00:25:02.314 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.314 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.314 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.314 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 20 2023-07-29T00:25:02.315 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.315 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.315 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.315 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.315 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.315 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:02.315 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:55.220568Z", 2023-07-29T00:25:02.316 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.316 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.316 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.316 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.316 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.316 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.317 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.317 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.317 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.317 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_LjbBbkffQbDhJ835-pK8tKlXLy1YzOv", 2023-07-29T00:25:02.317 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.317 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.317 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.318 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.318 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.318 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000048", 2023-07-29T00:25:02.318 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.318 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.318 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.318 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 4 2023-07-29T00:25:02.319 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.319 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.319 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.319 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.319 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.319 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:25:02.319 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.028892Z", 2023-07-29T00:25:02.320 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.320 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.320 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.320 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.320 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.320 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.321 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.321 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.321 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.321 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_Ce-jGvZqPrdWXPBSjbDpixVSyebuIC1", 2023-07-29T00:25:02.321 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.321 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.322 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.322 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.322 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.322 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000049", 2023-07-29T00:25:02.322 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.322 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.322 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.323 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 7 2023-07-29T00:25:02.323 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.323 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.323 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.323 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.323 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.323 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 6, 2023-07-29T00:25:02.324 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:56.685390Z", 2023-07-29T00:25:02.324 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.324 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.324 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.324 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.324 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.324 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.325 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.325 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.325 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.325 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_0oilpWmJp6Ind2n1IpFdWFZzFvdpKsg", 2023-07-29T00:25:02.325 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.325 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.325 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.326 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.326 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.326 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000050", 2023-07-29T00:25:02.326 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.326 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.326 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.327 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 4 2023-07-29T00:25:02.327 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.327 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.327 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.327 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.327 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.327 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:25:02.328 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:57.938921Z", 2023-07-29T00:25:02.328 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.328 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.328 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.328 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.328 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.329 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.329 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.329 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.329 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.329 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_NKzLk4AmwKH87QM69RYlJYp_oGhmnIY", 2023-07-29T00:25:02.329 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.329 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.330 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.330 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.330 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.330 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000053", 2023-07-29T00:25:02.330 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.330 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.330 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.331 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 7 2023-07-29T00:25:02.331 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.331 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.331 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.331 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.331 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.332 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 6, 2023-07-29T00:25:02.332 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:25:00.300321Z", 2023-07-29T00:25:02.332 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.332 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.332 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.332 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.332 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.333 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.333 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.333 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.333 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.333 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_vWpy7mgdFgKDGTeOdfoIFrW4xssBcfx", 2023-07-29T00:25:02.333 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.334 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.334 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.334 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.334 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.334 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000055", 2023-07-29T00:25:02.334 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.334 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.335 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.335 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 1 2023-07-29T00:25:02.335 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.335 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.335 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.335 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.336 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.336 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:02.336 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:55.734741Z", 2023-07-29T00:25:02.336 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.336 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.336 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.337 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.337 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.337 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.337 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.337 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.337 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.338 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_3tmVfIJcDi8UNojyRPDOCP9loGsMwQm", 2023-07-29T00:25:02.338 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.338 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.338 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.338 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.338 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.338 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000056", 2023-07-29T00:25:02.339 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.339 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.339 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.339 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 4 2023-07-29T00:25:02.339 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.339 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.340 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.340 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.340 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.340 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:02.340 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:57.604582Z", 2023-07-29T00:25:02.340 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.341 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.341 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.341 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.341 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.341 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.341 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.341 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.342 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.342 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_at0wQGu8xw9ErJELbkfn2S-sejON1JB", 2023-07-29T00:25:02.342 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.342 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.342 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.342 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.343 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.343 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000057", 2023-07-29T00:25:02.343 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.343 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.343 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.343 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 4 2023-07-29T00:25:02.344 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.344 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.344 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.344 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.344 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.345 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:25:02.345 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:25:00.207399Z", 2023-07-29T00:25:02.345 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.345 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.345 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.345 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.345 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.346 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.346 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.346 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.346 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.346 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_ewyukjDoIZ4CvdguR-Hj2-MPXo3dk3y", 2023-07-29T00:25:02.346 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.347 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.347 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.347 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.347 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.347 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000058", 2023-07-29T00:25:02.347 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.348 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.348 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.348 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 2 2023-07-29T00:25:02.348 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.348 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.349 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.349 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.349 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.349 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 6, 2023-07-29T00:25:02.349 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:55.988907Z", 2023-07-29T00:25:02.349 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.350 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.350 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.350 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.350 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.350 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.350 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.351 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.351 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.351 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_qCJPc5wxrpVsBz5jRhTkiwjQtiAvauu", 2023-07-29T00:25:02.351 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.351 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.351 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.352 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.352 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.352 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000059", 2023-07-29T00:25:02.352 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.352 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.352 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.353 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 5 2023-07-29T00:25:02.353 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.353 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.353 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.353 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.353 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.353 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:25:02.354 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:56.419498Z", 2023-07-29T00:25:02.354 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.354 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.354 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.354 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.355 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.355 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.355 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.355 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.355 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.355 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_tG-eli1MZ5zMdl1pqVVGj5emDQ17FPW", 2023-07-29T00:25:02.356 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.356 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.356 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.356 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.356 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.356 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000060", 2023-07-29T00:25:02.357 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.357 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.357 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.357 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 6 2023-07-29T00:25:02.357 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.357 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.357 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.358 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.358 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.358 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:25:02.358 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.520291Z", 2023-07-29T00:25:02.358 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.358 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.359 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.359 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.359 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.359 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.359 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.359 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.360 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.360 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_UsHlj-waBJYvPWEi48A8OKGgQ5x0a28", 2023-07-29T00:25:02.360 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.360 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.360 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.360 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.361 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.361 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000061", 2023-07-29T00:25:02.361 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.361 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.361 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.361 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 1 2023-07-29T00:25:02.362 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.362 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.362 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.362 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.362 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.362 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:25:02.362 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:58.597135Z", 2023-07-29T00:25:02.363 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.363 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.363 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.363 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.363 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.363 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.364 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.364 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.364 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.364 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_7hcp9owFYxAKUdxMgrxoBH-iBhKjzRN", 2023-07-29T00:25:02.364 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.364 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.365 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.365 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.365 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.365 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000062", 2023-07-29T00:25:02.365 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.365 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.366 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.366 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 3 2023-07-29T00:25:02.366 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.366 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.366 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.366 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.366 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.367 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:25:02.367 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:25:00.281200Z", 2023-07-29T00:25:02.367 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.367 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.367 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.367 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.367 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.368 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.368 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.368 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.368 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.368 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_ST2bbBRxDVkrlRkydzJW3pcqZGZzOuF", 2023-07-29T00:25:02.368 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.369 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.369 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.369 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.369 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.369 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000065", 2023-07-29T00:25:02.369 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.369 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.370 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.370 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 8 2023-07-29T00:25:02.370 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.370 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.370 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.370 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.370 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.371 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:02.371 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:59.805482Z", 2023-07-29T00:25:02.371 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.371 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.371 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.371 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.372 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.372 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.372 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.372 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.372 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.372 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_ki5P3I0vEcxFjs6399SHgCHqj4nvDb7", 2023-07-29T00:25:02.373 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.373 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.373 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.373 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.373 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.373 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000076", 2023-07-29T00:25:02.373 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.374 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.374 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.374 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 2 2023-07-29T00:25:02.374 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.374 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.374 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.375 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.375 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.375 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:25:02.375 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:53.955523Z", 2023-07-29T00:25:02.375 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.375 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.376 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.376 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.376 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.376 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.376 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.376 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.376 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.377 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_SwpiBDYtS1q-iDJ_C4XduGK8ZfpQpe2", 2023-07-29T00:25:02.377 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.377 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.377 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.377 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.377 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.378 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000133", 2023-07-29T00:25:02.378 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.378 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.378 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.378 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 3 2023-07-29T00:25:02.378 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.378 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.379 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.379 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.379 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.379 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:02.379 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:57.710367Z", 2023-07-29T00:25:02.379 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.379 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.380 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.380 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.380 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.380 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.380 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.380 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.381 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.381 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_U5I9xbxNzqIHuHnN4K5Z2hVxHNVBkpe", 2023-07-29T00:25:02.381 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.381 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.381 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.381 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.381 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.382 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000162", 2023-07-29T00:25:02.382 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.382 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.382 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.382 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 3 2023-07-29T00:25:02.382 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.383 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.383 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.383 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.383 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.383 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:02.383 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:57.876925Z", 2023-07-29T00:25:02.383 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.384 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.384 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.384 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.384 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.384 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.384 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.385 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.385 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.385 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_LnDhCdIFkJrly3PWlQ1F9xgQJBOc3ro", 2023-07-29T00:25:02.385 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.385 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.385 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.386 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.387 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.387 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000271", 2023-07-29T00:25:02.387 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.387 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.387 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.387 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 2 2023-07-29T00:25:02.388 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.388 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.388 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.388 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.388 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.388 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:25:02.389 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:58.148659Z", 2023-07-29T00:25:02.389 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.389 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.389 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.389 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.390 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.390 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.390 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.390 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.390 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.390 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_xMUiqRUmBL6RtmGqM4XF6WHucYtXan6", 2023-07-29T00:25:02.391 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.391 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.391 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.391 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.391 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.391 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000317", 2023-07-29T00:25:02.391 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.392 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.392 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.392 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 6 2023-07-29T00:25:02.392 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.392 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.392 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.393 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.393 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.393 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:02.393 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:58.614561Z", 2023-07-29T00:25:02.393 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.393 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.394 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.394 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.394 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.394 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.394 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.394 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.395 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.395 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_lCQudZu1AFZ_J97G5F4w3quwBMgttVZ", 2023-07-29T00:25:02.395 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.395 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.395 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.395 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.396 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.396 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000423", 2023-07-29T00:25:02.396 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.396 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.396 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.396 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 7 2023-07-29T00:25:02.397 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.397 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.397 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.397 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.397 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.397 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:25:02.397 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:57.967362Z", 2023-07-29T00:25:02.398 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.398 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.398 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.398 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.398 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.398 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.399 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.399 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.399 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.399 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_ySFl_qmP8AmRboObvgeQpaZs1DDQGug", 2023-07-29T00:25:02.399 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.399 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.400 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.400 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.400 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:02.400 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj00000000000000000467", 2023-07-29T00:25:02.400 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:25:02.400 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:25:02.400 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:25:02.401 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 2 2023-07-29T00:25:02.401 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.401 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:25:02.401 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:25:02.401 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:25:02.401 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:25:02.402 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:25:02.402 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:24:58.460850Z", 2023-07-29T00:25:02.402 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:25:02.402 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:25:02.402 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:25:02.402 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:25:02.402 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:25:02.403 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:25:02.403 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:25:02.403 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:25:02.403 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:02.403 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_fvvEzUuVJeapmkCWEKyGeM4iWQqyv2h", 2023-07-29T00:25:02.403 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:25:02.403 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:25:02.404 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:25:02.404 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:02.404 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T00:25:02.405 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-07-29T00:25:02.405 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-07-29T00:25:02.408 INFO:tasks.rgw.c1.client.1:Started 2023-07-29T00:25:02.409 DEBUG:teuthology.orchestra.run.smithi101:> curl http://smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:02.427 INFO:teuthology.orchestra.run.smithi101.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-07-29T00:25:02.427 INFO:teuthology.orchestra.run.smithi101.stderr: Dload Upload Total Spent Left Speed 2023-07-29T00:25:02.428 INFO:teuthology.orchestra.run.smithi101.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-07-29T00:25:02.429 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-07-29T00:25:02.430 INFO:teuthology.orchestra.run.smithi101.stderr:curl: (7) Failed to connect to smithi101.front.sepia.ceph.com port 8001: Connection refused 2023-07-29T00:25:03.431 DEBUG:teuthology.orchestra.run.smithi101:> curl http://smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:03.450 INFO:teuthology.orchestra.run.smithi101.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-07-29T00:25:03.450 INFO:teuthology.orchestra.run.smithi101.stderr: Dload Upload Total Spent Left Speed 2023-07-29T00:25:03.451 INFO:teuthology.orchestra.run.smithi101.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-07-29T00:25:03.451 INFO:teuthology.orchestra.run.smithi101.stdout:anonymous 2023-07-29T00:25:03.452 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-58 2023-07-29T00:25:03.452 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:25:03.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:25:03.453 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-58 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:25:03.545 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:25:03.546 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:25:03.583 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:03.531+0000 7f6304baf880 1 bucket sync caught up with source: 2023-07-29T00:25:03.583 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000001.97.6, 00000000002.74.6, , 00000000002.51.6, 00000000002.101.6, 00000000001.43.6, 00000000002.67.6, 00000000002.57.6, 00000000001.68.6, 00000000002.70.6, 00000000002.55.6, 00000000002.52.6, 00000000001.69.6, 00000000002.54.6, 00000000002.65.6, 00000000002.37.6, 00000000002.43.6, 00000000002.66.6, 00000000002.41.6, 00000000002.42.6, 00000000002.77.6, 00000000002.52.6, 00000000002.60.6, , 00000000002.53.6, 00000000002.90.6, 00000000002.58.6, 00000000002.54.6, 00000000002.50.6, 00000000002.63.6, 00000000001.53.6, 00000000002.55.6, 00000000002.53.6, 00000000002.99.6, 00000000001.65.6, , 00000000002.114.6] 2023-07-29T00:25:03.584 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000001.97.6, 00000000002.74.6, , 00000000002.51.6, 00000000002.101.6, 00000000001.43.6, 00000000002.67.6, 00000000002.57.6, 00000000001.68.6, 00000000002.70.6, 00000000002.55.6, 00000000002.52.6, 00000000001.69.6, 00000000002.54.6, 00000000002.65.6, 00000000002.37.6, 00000000002.43.6, 00000000002.66.6, 00000000002.41.6, 00000000002.42.6, 00000000002.77.6, 00000000002.52.6, 00000000002.60.6, , 00000000002.53.6, 00000000002.90.6, 00000000002.58.6, 00000000002.54.6, 00000000002.50.6, 00000000002.63.6, 00000000001.53.6, 00000000002.55.6, 00000000002.53.6, 00000000002.99.6, 00000000001.65.6, , 00000000002.114.6] 2023-07-29T00:25:03.584 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:03.531+0000 7f6304baf880 0 bucket checkpoint complete 2023-07-29T00:25:03.597 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-58 2023-07-29T00:25:03.597 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:25:03.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:25:03.598 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-58 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:25:03.653 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:25:03.653 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:25:03.687 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:03.635+0000 7fe47d729880 1 bucket sync caught up with source: 2023-07-29T00:25:03.688 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000005.893.6, 00000000007.895.6, 00000000006.1346.6, 00000000009.1199.6, 00000000005.121.6, 00000000008.1049.6, 00000000006.908.6, 00000000006.896.6, 00000000007.2450.6, 00000000007.259.6, 00000000004.891.6] 2023-07-29T00:25:03.688 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000005.893.6, 00000000007.895.6, 00000000006.1346.6, 00000000009.1199.6, 00000000005.121.6, 00000000008.1049.6, 00000000006.908.6, 00000000006.896.6, 00000000007.2450.6, 00000000007.259.6, 00000000004.891.6] 2023-07-29T00:25:03.688 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:03.635+0000 7fe47d729880 0 bucket checkpoint complete 2023-07-29T00:25:03.701 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-58 zones={a2, a1} 2023-07-29T00:25:03.701 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:03.702 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:03.702 DEBUG:boto:Method: HEAD 2023-07-29T00:25:03.702 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:25:03.703 DEBUG:boto:Data: 2023-07-29T00:25:03.703 DEBUG:boto:Headers: {} 2023-07-29T00:25:03.703 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:03.704 DEBUG:boto:Port: 8001 2023-07-29T00:25:03.704 DEBUG:boto:Params: {} 2023-07-29T00:25:03.705 DEBUG:boto:Token: None 2023-07-29T00:25:03.705 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:03 GMT /hiiqos-58/ 2023-07-29T00:25:03.706 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5Xt1mu+1zO3sbOr4b2/jq9rC9B0= 2023-07-29T00:25:03.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5Xt1mu+1zO3sbOr4b2/jq9rC9B0=', 'Content-Length': '0'} 2023-07-29T00:25:03.707 DEBUG:boto:encountered RemoteDisconnected exception, reconnecting 2023-07-29T00:25:03.756 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-29T00:25:04.174 DEBUG:boto:Token: None 2023-07-29T00:25:04.175 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:03 GMT /hiiqos-58/ 2023-07-29T00:25:04.175 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5Xt1mu+1zO3sbOr4b2/jq9rC9B0= 2023-07-29T00:25:04.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5Xt1mu+1zO3sbOr4b2/jq9rC9B0=', 'Content-Length': '0'} 2023-07-29T00:25:04.181 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e4574f052eb022cc-0064c45c60-1c08-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.182 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:04.182 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:04.183 DEBUG:boto:Method: HEAD 2023-07-29T00:25:04.183 DEBUG:boto:Path: /hiiqos-58/ 2023-07-29T00:25:04.183 DEBUG:boto:Data: 2023-07-29T00:25:04.184 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.184 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:04.185 DEBUG:boto:Port: 8000 2023-07-29T00:25:04.185 DEBUG:boto:Params: {} 2023-07-29T00:25:04.185 DEBUG:boto:Token: None 2023-07-29T00:25:04.186 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/ 2023-07-29T00:25:04.186 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QzNDrTHx4SY6xFF5zjhkRc3YPn0= 2023-07-29T00:25:04.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QzNDrTHx4SY6xFF5zjhkRc3YPn0=', 'Content-Length': '0'} 2023-07-29T00:25:04.191 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008a25fd6ddface50b-0064c45c60-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.191 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-29T00:25:04.192 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:04.192 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:04.193 DEBUG:boto:path=/hiiqos-58/?versions&max-keys=999 2023-07-29T00:25:04.193 DEBUG:boto:auth_path=/hiiqos-58/?versions&max-keys=999 2023-07-29T00:25:04.193 DEBUG:boto:Method: GET 2023-07-29T00:25:04.194 DEBUG:boto:Path: /hiiqos-58/?versions&max-keys=999 2023-07-29T00:25:04.194 DEBUG:boto:Data: 2023-07-29T00:25:04.194 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.195 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:04.195 DEBUG:boto:Port: 8001 2023-07-29T00:25:04.195 DEBUG:boto:Params: {} 2023-07-29T00:25:04.196 DEBUG:boto:Token: None 2023-07-29T00:25:04.196 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/?versions 2023-07-29T00:25:04.197 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rioXZnS/gEBaSA3ita4esg+3cmQ= 2023-07-29T00:25:04.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rioXZnS/gEBaSA3ita4esg+3cmQ=', 'Content-Length': '0'} 2023-07-29T00:25:04.200 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065cca15ea3bbb17a-0064c45c60-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.201 DEBUG:boto:b'hiiqos-58999falseobj00000000000000000000nulltrue2023-07-29T00:24:59.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-07-29T00:24:59.749Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-07-29T00:24:59.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-07-29T00:25:00.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-07-29T00:24:59.233Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-07-29T00:24:59.414Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-07-29T00:24:59.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2023-07-29T00:24:59.887Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2023-07-29T00:25:00.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2023-07-29T00:24:59.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-07-29T00:24:59.665Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-07-29T00:24:58.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-07-29T00:24:59.604Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-07-29T00:24:59.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2023-07-29T00:24:59.990Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2023-07-29T00:24:59.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2023-07-29T00:24:59.056Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2023-07-29T00:25:00.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2023-07-29T00:24:58.540Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2023-07-29T00:24:59.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2023-07-29T00:24:59.538Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2023-07-29T00:24:59.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2023-07-29T00:24:58.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2023-07-29T00:24:58.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2023-07-29T00:24:59.435Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2023-07-29T00:24:56.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2023-07-29T00:24:59.170Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2023-07-29T00:24:59.909Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2023-07-29T00:24:53.541Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2023-07-29T00:24:58.285Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2023-07-29T00:24:59.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2023-07-29T00:24:57.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2023-07-29T00:24:53.877Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2023-07-29T00:25:00.051Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2023-07-29T00:24:59.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2023-07-29T00:24:55.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2023-07-29T00:24:59.326Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2023-07-29T00:24:56.845Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2023-07-29T00:25:00.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000046nulltrue2023-07-29T00:24:55.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000048nulltrue2023-07-29T00:24:59.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000049nulltrue2023-07-29T00:24:56.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2023-07-29T00:24:57.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2023-07-29T00:25:00.300Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2023-07-29T00:24:55.734Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000056nulltrue2023-07-29T00:24:57.604Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2023-07-29T00:25:00.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2023-07-29T00:24:55.988Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2023-07-29T00:24:56.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2023-07-29T00:24:59.520Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2023-07-29T00:24:58.597Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2023-07-29T00:25:00.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2023-07-29T00:24:59.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000076nulltrue2023-07-29T00:24:53.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2023-07-29T00:24:57.710Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000162nulltrue2023-07-29T00:24:57.876Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2023-07-29T00:24:58.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2023-07-29T00:24:58.614Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2023-07-29T00:24:57.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2023-07-29T00:24:58.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:04.204 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2023-07-29T00:25:04.204 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2023-07-29T00:25:04.204 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2023-07-29T00:25:04.205 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2023-07-29T00:25:04.205 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2023-07-29T00:25:04.205 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2023-07-29T00:25:04.206 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2023-07-29T00:25:04.206 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000007 2023-07-29T00:25:04.206 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000008 2023-07-29T00:25:04.207 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000009 2023-07-29T00:25:04.207 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2023-07-29T00:25:04.207 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2023-07-29T00:25:04.207 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2023-07-29T00:25:04.208 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2023-07-29T00:25:04.208 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000014 2023-07-29T00:25:04.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000015 2023-07-29T00:25:04.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000016 2023-07-29T00:25:04.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000017 2023-07-29T00:25:04.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000018 2023-07-29T00:25:04.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000019 2023-07-29T00:25:04.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000020 2023-07-29T00:25:04.211 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000021 2023-07-29T00:25:04.211 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000022 2023-07-29T00:25:04.211 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000023 2023-07-29T00:25:04.212 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000024 2023-07-29T00:25:04.212 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000025 2023-07-29T00:25:04.212 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000026 2023-07-29T00:25:04.213 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000027 2023-07-29T00:25:04.213 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000028 2023-07-29T00:25:04.213 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000029 2023-07-29T00:25:04.214 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000030 2023-07-29T00:25:04.214 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000031 2023-07-29T00:25:04.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000032 2023-07-29T00:25:04.216 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000033 2023-07-29T00:25:04.216 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000036 2023-07-29T00:25:04.217 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000039 2023-07-29T00:25:04.218 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000040 2023-07-29T00:25:04.219 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000042 2023-07-29T00:25:04.219 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000045 2023-07-29T00:25:04.220 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000046 2023-07-29T00:25:04.221 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000048 2023-07-29T00:25:04.221 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000049 2023-07-29T00:25:04.222 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000050 2023-07-29T00:25:04.223 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000053 2023-07-29T00:25:04.223 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000055 2023-07-29T00:25:04.224 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000056 2023-07-29T00:25:04.224 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000057 2023-07-29T00:25:04.225 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000058 2023-07-29T00:25:04.226 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000059 2023-07-29T00:25:04.226 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000060 2023-07-29T00:25:04.227 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000061 2023-07-29T00:25:04.227 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000062 2023-07-29T00:25:04.227 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000065 2023-07-29T00:25:04.228 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000076 2023-07-29T00:25:04.228 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000133 2023-07-29T00:25:04.228 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000162 2023-07-29T00:25:04.229 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000271 2023-07-29T00:25:04.229 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000317 2023-07-29T00:25:04.229 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000423 2023-07-29T00:25:04.230 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000467 2023-07-29T00:25:04.230 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-29T00:25:04.230 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:04.231 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:04.232 DEBUG:boto:path=/hiiqos-58/?versions&max-keys=999 2023-07-29T00:25:04.232 DEBUG:boto:auth_path=/hiiqos-58/?versions&max-keys=999 2023-07-29T00:25:04.232 DEBUG:boto:Method: GET 2023-07-29T00:25:04.233 DEBUG:boto:Path: /hiiqos-58/?versions&max-keys=999 2023-07-29T00:25:04.233 DEBUG:boto:Data: 2023-07-29T00:25:04.233 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.234 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:04.235 DEBUG:boto:Port: 8000 2023-07-29T00:25:04.235 DEBUG:boto:Params: {} 2023-07-29T00:25:04.235 DEBUG:boto:Token: None 2023-07-29T00:25:04.236 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/?versions 2023-07-29T00:25:04.237 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rioXZnS/gEBaSA3ita4esg+3cmQ= 2023-07-29T00:25:04.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rioXZnS/gEBaSA3ita4esg+3cmQ=', 'Content-Length': '0'} 2023-07-29T00:25:04.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090164d93102704d4-0064c45c60-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.242 DEBUG:boto:b'hiiqos-58999falseobj00000000000000000000nulltrue2023-07-29T00:24:59.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-07-29T00:24:59.749Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-07-29T00:24:59.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-07-29T00:25:00.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-07-29T00:24:59.233Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-07-29T00:24:59.414Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-07-29T00:24:59.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2023-07-29T00:24:59.887Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2023-07-29T00:25:00.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2023-07-29T00:24:59.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-07-29T00:24:59.665Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-07-29T00:24:58.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-07-29T00:24:59.604Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-07-29T00:24:59.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2023-07-29T00:24:59.990Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2023-07-29T00:24:59.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2023-07-29T00:24:59.056Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2023-07-29T00:25:00.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2023-07-29T00:24:58.540Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2023-07-29T00:24:59.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2023-07-29T00:24:59.538Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2023-07-29T00:24:59.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2023-07-29T00:24:58.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2023-07-29T00:24:58.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2023-07-29T00:24:59.435Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2023-07-29T00:24:56.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2023-07-29T00:24:59.170Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2023-07-29T00:24:59.909Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2023-07-29T00:24:53.541Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2023-07-29T00:24:58.285Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2023-07-29T00:24:59.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2023-07-29T00:24:57.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2023-07-29T00:24:53.877Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2023-07-29T00:25:00.051Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2023-07-29T00:24:59.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2023-07-29T00:24:55.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2023-07-29T00:24:59.326Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2023-07-29T00:24:56.845Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2023-07-29T00:25:00.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000046nulltrue2023-07-29T00:24:55.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000048nulltrue2023-07-29T00:24:59.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000049nulltrue2023-07-29T00:24:56.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2023-07-29T00:24:57.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2023-07-29T00:25:00.300Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2023-07-29T00:24:55.734Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000056nulltrue2023-07-29T00:24:57.604Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2023-07-29T00:25:00.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2023-07-29T00:24:55.988Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2023-07-29T00:24:56.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2023-07-29T00:24:59.520Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2023-07-29T00:24:58.597Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2023-07-29T00:25:00.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2023-07-29T00:24:59.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000076nulltrue2023-07-29T00:24:53.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2023-07-29T00:24:57.710Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000162nulltrue2023-07-29T00:24:57.876Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2023-07-29T00:24:58.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2023-07-29T00:24:58.614Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2023-07-29T00:24:57.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2023-07-29T00:24:58.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:04.246 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2023-07-29T00:25:04.247 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2023-07-29T00:25:04.247 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2023-07-29T00:25:04.248 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2023-07-29T00:25:04.248 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2023-07-29T00:25:04.249 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2023-07-29T00:25:04.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2023-07-29T00:25:04.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000007 2023-07-29T00:25:04.251 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000008 2023-07-29T00:25:04.251 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000009 2023-07-29T00:25:04.252 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2023-07-29T00:25:04.252 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2023-07-29T00:25:04.252 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2023-07-29T00:25:04.253 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2023-07-29T00:25:04.253 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000014 2023-07-29T00:25:04.253 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000015 2023-07-29T00:25:04.253 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000016 2023-07-29T00:25:04.254 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000017 2023-07-29T00:25:04.254 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000018 2023-07-29T00:25:04.254 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000019 2023-07-29T00:25:04.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000020 2023-07-29T00:25:04.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000021 2023-07-29T00:25:04.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000022 2023-07-29T00:25:04.256 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000023 2023-07-29T00:25:04.256 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000024 2023-07-29T00:25:04.256 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000025 2023-07-29T00:25:04.256 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000026 2023-07-29T00:25:04.257 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000027 2023-07-29T00:25:04.257 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000028 2023-07-29T00:25:04.257 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000029 2023-07-29T00:25:04.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000030 2023-07-29T00:25:04.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000031 2023-07-29T00:25:04.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000032 2023-07-29T00:25:04.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000033 2023-07-29T00:25:04.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000036 2023-07-29T00:25:04.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000039 2023-07-29T00:25:04.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000040 2023-07-29T00:25:04.260 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000042 2023-07-29T00:25:04.260 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000045 2023-07-29T00:25:04.260 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000046 2023-07-29T00:25:04.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000048 2023-07-29T00:25:04.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000049 2023-07-29T00:25:04.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000050 2023-07-29T00:25:04.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000053 2023-07-29T00:25:04.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000055 2023-07-29T00:25:04.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000056 2023-07-29T00:25:04.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000057 2023-07-29T00:25:04.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000058 2023-07-29T00:25:04.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000059 2023-07-29T00:25:04.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000060 2023-07-29T00:25:04.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000061 2023-07-29T00:25:04.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000062 2023-07-29T00:25:04.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000065 2023-07-29T00:25:04.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000076 2023-07-29T00:25:04.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000133 2023-07-29T00:25:04.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000162 2023-07-29T00:25:04.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000271 2023-07-29T00:25:04.266 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000317 2023-07-29T00:25:04.266 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000423 2023-07-29T00:25:04.266 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000467 2023-07-29T00:25:04.267 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:04.267 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:04.267 DEBUG:boto:path=/hiiqos-58/?versions&max-keys=999 2023-07-29T00:25:04.268 DEBUG:boto:auth_path=/hiiqos-58/?versions&max-keys=999 2023-07-29T00:25:04.268 DEBUG:boto:Method: GET 2023-07-29T00:25:04.268 DEBUG:boto:Path: /hiiqos-58/?versions&max-keys=999 2023-07-29T00:25:04.269 DEBUG:boto:Data: 2023-07-29T00:25:04.269 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.269 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:04.270 DEBUG:boto:Port: 8001 2023-07-29T00:25:04.270 DEBUG:boto:Params: {} 2023-07-29T00:25:04.270 DEBUG:boto:Token: None 2023-07-29T00:25:04.271 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/?versions 2023-07-29T00:25:04.271 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rioXZnS/gEBaSA3ita4esg+3cmQ= 2023-07-29T00:25:04.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rioXZnS/gEBaSA3ita4esg+3cmQ=', 'Content-Length': '0'} 2023-07-29T00:25:04.274 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002be943990f058590-0064c45c60-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.275 DEBUG:boto:b'hiiqos-58999falseobj00000000000000000000nulltrue2023-07-29T00:24:59.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-07-29T00:24:59.749Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-07-29T00:24:59.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-07-29T00:25:00.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-07-29T00:24:59.233Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-07-29T00:24:59.414Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-07-29T00:24:59.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2023-07-29T00:24:59.887Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2023-07-29T00:25:00.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2023-07-29T00:24:59.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-07-29T00:24:59.665Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-07-29T00:24:58.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-07-29T00:24:59.604Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-07-29T00:24:59.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2023-07-29T00:24:59.990Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2023-07-29T00:24:59.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2023-07-29T00:24:59.056Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2023-07-29T00:25:00.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2023-07-29T00:24:58.540Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2023-07-29T00:24:59.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2023-07-29T00:24:59.538Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2023-07-29T00:24:59.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2023-07-29T00:24:58.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2023-07-29T00:24:58.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2023-07-29T00:24:59.435Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2023-07-29T00:24:56.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2023-07-29T00:24:59.170Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2023-07-29T00:24:59.909Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2023-07-29T00:24:53.541Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2023-07-29T00:24:58.285Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2023-07-29T00:24:59.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2023-07-29T00:24:57.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2023-07-29T00:24:53.877Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2023-07-29T00:25:00.051Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2023-07-29T00:24:59.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2023-07-29T00:24:55.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2023-07-29T00:24:59.326Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2023-07-29T00:24:56.845Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2023-07-29T00:25:00.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000046nulltrue2023-07-29T00:24:55.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000048nulltrue2023-07-29T00:24:59.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000049nulltrue2023-07-29T00:24:56.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2023-07-29T00:24:57.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2023-07-29T00:25:00.300Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2023-07-29T00:24:55.734Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000056nulltrue2023-07-29T00:24:57.604Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2023-07-29T00:25:00.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2023-07-29T00:24:55.988Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2023-07-29T00:24:56.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2023-07-29T00:24:59.520Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2023-07-29T00:24:58.597Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2023-07-29T00:25:00.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2023-07-29T00:24:59.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000076nulltrue2023-07-29T00:24:53.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2023-07-29T00:24:57.710Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000162nulltrue2023-07-29T00:24:57.876Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2023-07-29T00:24:58.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2023-07-29T00:24:58.614Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2023-07-29T00:24:57.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2023-07-29T00:24:58.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:04.278 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:04.278 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:04.279 DEBUG:boto:path=/hiiqos-58/?versions&max-keys=999 2023-07-29T00:25:04.279 DEBUG:boto:auth_path=/hiiqos-58/?versions&max-keys=999 2023-07-29T00:25:04.279 DEBUG:boto:Method: GET 2023-07-29T00:25:04.280 DEBUG:boto:Path: /hiiqos-58/?versions&max-keys=999 2023-07-29T00:25:04.280 DEBUG:boto:Data: 2023-07-29T00:25:04.281 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.281 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:04.281 DEBUG:boto:Port: 8000 2023-07-29T00:25:04.282 DEBUG:boto:Params: {} 2023-07-29T00:25:04.282 DEBUG:boto:Token: None 2023-07-29T00:25:04.282 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/?versions 2023-07-29T00:25:04.283 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rioXZnS/gEBaSA3ita4esg+3cmQ= 2023-07-29T00:25:04.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rioXZnS/gEBaSA3ita4esg+3cmQ=', 'Content-Length': '0'} 2023-07-29T00:25:04.287 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034ed8087556d3741-0064c45c60-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.288 DEBUG:boto:b'hiiqos-58999falseobj00000000000000000000nulltrue2023-07-29T00:24:59.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-07-29T00:24:59.749Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-07-29T00:24:59.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-07-29T00:25:00.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-07-29T00:24:59.233Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-07-29T00:24:59.414Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-07-29T00:24:59.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2023-07-29T00:24:59.887Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2023-07-29T00:25:00.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2023-07-29T00:24:59.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-07-29T00:24:59.665Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-07-29T00:24:58.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-07-29T00:24:59.604Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-07-29T00:24:59.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2023-07-29T00:24:59.990Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2023-07-29T00:24:59.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2023-07-29T00:24:59.056Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2023-07-29T00:25:00.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2023-07-29T00:24:58.540Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2023-07-29T00:24:59.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2023-07-29T00:24:59.538Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2023-07-29T00:24:59.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2023-07-29T00:24:58.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2023-07-29T00:24:58.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2023-07-29T00:24:59.435Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2023-07-29T00:24:56.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2023-07-29T00:24:59.170Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2023-07-29T00:24:59.909Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2023-07-29T00:24:53.541Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2023-07-29T00:24:58.285Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2023-07-29T00:24:59.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2023-07-29T00:24:57.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2023-07-29T00:24:53.877Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2023-07-29T00:25:00.051Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2023-07-29T00:24:59.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2023-07-29T00:24:55.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2023-07-29T00:24:59.326Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2023-07-29T00:24:56.845Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2023-07-29T00:25:00.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000046nulltrue2023-07-29T00:24:55.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000048nulltrue2023-07-29T00:24:59.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000049nulltrue2023-07-29T00:24:56.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2023-07-29T00:24:57.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2023-07-29T00:25:00.300Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2023-07-29T00:24:55.734Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000056nulltrue2023-07-29T00:24:57.604Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2023-07-29T00:25:00.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2023-07-29T00:24:55.988Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2023-07-29T00:24:56.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2023-07-29T00:24:59.520Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2023-07-29T00:24:58.597Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2023-07-29T00:25:00.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2023-07-29T00:24:59.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000076nulltrue2023-07-29T00:24:53.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2023-07-29T00:24:57.710Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000162nulltrue2023-07-29T00:24:57.876Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2023-07-29T00:24:58.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2023-07-29T00:24:58.614Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2023-07-29T00:24:57.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2023-07-29T00:24:58.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:04.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-07-29T00:25:04.293 DEBUG:boto:path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:25:04.293 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:25:04.293 DEBUG:boto:path=/hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:25:04.294 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:25:04.294 DEBUG:boto:Method: GET 2023-07-29T00:25:04.294 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:25:04.295 DEBUG:boto:Data: 2023-07-29T00:25:04.295 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.295 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:04.296 DEBUG:boto:Port: 8001 2023-07-29T00:25:04.296 DEBUG:boto:Params: {} 2023-07-29T00:25:04.297 DEBUG:boto:Token: None 2023-07-29T00:25:04.297 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:25:04.297 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xa7eH0hTXg3D4CNdi6nhFhuo8OY= 2023-07-29T00:25:04.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xa7eH0hTXg3D4CNdi6nhFhuo8OY=', 'Content-Length': '0'} 2023-07-29T00:25:04.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5f365d7057b2c8f-0064c45c60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.301 DEBUG:boto:path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:25:04.301 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:25:04.302 DEBUG:boto:path=/hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:25:04.302 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:25:04.303 DEBUG:boto:Method: GET 2023-07-29T00:25:04.303 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:25:04.303 DEBUG:boto:Data: 2023-07-29T00:25:04.304 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.304 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:04.304 DEBUG:boto:Port: 8000 2023-07-29T00:25:04.305 DEBUG:boto:Params: {} 2023-07-29T00:25:04.305 DEBUG:boto:Token: None 2023-07-29T00:25:04.306 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:25:04.306 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xa7eH0hTXg3D4CNdi6nhFhuo8OY= 2023-07-29T00:25:04.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xa7eH0hTXg3D4CNdi6nhFhuo8OY=', 'Content-Length': '0'} 2023-07-29T00:25:04.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003155447064ac0feb-0064c45c60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.309 DEBUG:boto:path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:25:04.310 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:25:04.310 DEBUG:boto:path=/hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:25:04.311 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:25:04.311 DEBUG:boto:Method: HEAD 2023-07-29T00:25:04.312 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:25:04.312 DEBUG:boto:Data: 2023-07-29T00:25:04.313 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.313 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:04.313 DEBUG:boto:Port: 8001 2023-07-29T00:25:04.313 DEBUG:boto:Params: {} 2023-07-29T00:25:04.314 DEBUG:boto:Token: None 2023-07-29T00:25:04.314 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:25:04.315 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xeib5jAanbZnXUwat3NbtEYYx2E= 2023-07-29T00:25:04.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xeib5jAanbZnXUwat3NbtEYYx2E=', 'Content-Length': '0'} 2023-07-29T00:25:04.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030b385a17146fa6f-0064c45c60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.320 DEBUG:boto:path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:25:04.320 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:25:04.321 DEBUG:boto:path=/hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:25:04.322 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:25:04.323 DEBUG:boto:Method: HEAD 2023-07-29T00:25:04.323 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:25:04.324 DEBUG:boto:Data: 2023-07-29T00:25:04.324 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.325 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:04.326 DEBUG:boto:Port: 8000 2023-07-29T00:25:04.326 DEBUG:boto:Params: {} 2023-07-29T00:25:04.327 DEBUG:boto:Token: None 2023-07-29T00:25:04.328 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:25:04.328 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xeib5jAanbZnXUwat3NbtEYYx2E= 2023-07-29T00:25:04.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xeib5jAanbZnXUwat3NbtEYYx2E=', 'Content-Length': '0'} 2023-07-29T00:25:04.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096a11ddefcde2582-0064c45c60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-07-29T00:25:04.332 DEBUG:boto:path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:25:04.333 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:25:04.334 DEBUG:boto:path=/hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:25:04.334 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:25:04.335 DEBUG:boto:Method: GET 2023-07-29T00:25:04.336 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:25:04.336 DEBUG:boto:Data: 2023-07-29T00:25:04.337 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.338 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:04.338 DEBUG:boto:Port: 8001 2023-07-29T00:25:04.339 DEBUG:boto:Params: {} 2023-07-29T00:25:04.340 DEBUG:boto:Token: None 2023-07-29T00:25:04.340 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:25:04.341 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xa7eH0hTXg3D4CNdi6nhFhuo8OY= 2023-07-29T00:25:04.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xa7eH0hTXg3D4CNdi6nhFhuo8OY=', 'Content-Length': '0'} 2023-07-29T00:25:04.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2ca0adff1fb3ddd-0064c45c60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.343 DEBUG:boto:path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:25:04.344 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:25:04.345 DEBUG:boto:path=/hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:25:04.345 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:25:04.345 DEBUG:boto:Method: GET 2023-07-29T00:25:04.346 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:25:04.346 DEBUG:boto:Data: 2023-07-29T00:25:04.347 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.347 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:04.348 DEBUG:boto:Port: 8000 2023-07-29T00:25:04.348 DEBUG:boto:Params: {} 2023-07-29T00:25:04.349 DEBUG:boto:Token: None 2023-07-29T00:25:04.349 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/obj00000000000000000000?versionId=null 2023-07-29T00:25:04.349 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xa7eH0hTXg3D4CNdi6nhFhuo8OY= 2023-07-29T00:25:04.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xa7eH0hTXg3D4CNdi6nhFhuo8OY=', 'Content-Length': '0'} 2023-07-29T00:25:04.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b153d6f2293cbb08-0064c45c60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.353 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:04.353 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:04.353 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000000 2023-07-29T00:25:04.354 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000000 2023-07-29T00:25:04.354 DEBUG:boto:Method: GET 2023-07-29T00:25:04.355 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000000 2023-07-29T00:25:04.355 DEBUG:boto:Data: 2023-07-29T00:25:04.355 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.356 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:04.356 DEBUG:boto:Port: 8001 2023-07-29T00:25:04.357 DEBUG:boto:Params: {} 2023-07-29T00:25:04.357 DEBUG:boto:Token: None 2023-07-29T00:25:04.358 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/ 2023-07-29T00:25:04.358 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TBsjLEJxMDSPwueQSOR6oKQJ65c= 2023-07-29T00:25:04.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TBsjLEJxMDSPwueQSOR6oKQJ65c=', 'Content-Length': '0'} 2023-07-29T00:25:04.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbb6a35d3866df43-0064c45c60-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.362 DEBUG:boto:b'hiiqos-58obj000000000000000000001000falseobj000000000000000000002023-07-29T00:24:59.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:04.363 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:04.364 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:04.365 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000000 2023-07-29T00:25:04.365 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000000 2023-07-29T00:25:04.366 DEBUG:boto:Method: GET 2023-07-29T00:25:04.366 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000000 2023-07-29T00:25:04.367 DEBUG:boto:Data: 2023-07-29T00:25:04.367 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.367 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:04.368 DEBUG:boto:Port: 8000 2023-07-29T00:25:04.368 DEBUG:boto:Params: {} 2023-07-29T00:25:04.368 DEBUG:boto:Token: None 2023-07-29T00:25:04.369 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/ 2023-07-29T00:25:04.369 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TBsjLEJxMDSPwueQSOR6oKQJ65c= 2023-07-29T00:25:04.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TBsjLEJxMDSPwueQSOR6oKQJ65c=', 'Content-Length': '0'} 2023-07-29T00:25:04.373 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025db649cb5e7d4dd-0064c45c60-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.374 DEBUG:boto:b'hiiqos-58obj000000000000000000001000falseobj000000000000000000002023-07-29T00:24:59.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:04.374 DEBUG:boto:path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:25:04.375 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:25:04.375 DEBUG:boto:Method: HEAD 2023-07-29T00:25:04.375 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000000 2023-07-29T00:25:04.376 DEBUG:boto:Data: 2023-07-29T00:25:04.376 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.376 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:04.377 DEBUG:boto:Port: 8001 2023-07-29T00:25:04.377 DEBUG:boto:Params: {} 2023-07-29T00:25:04.377 DEBUG:boto:Token: None 2023-07-29T00:25:04.378 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/obj00000000000000000000 2023-07-29T00:25:04.378 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rfUfvIDr/7MDcaSkt+LLvFiC9K4= 2023-07-29T00:25:04.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rfUfvIDr/7MDcaSkt+LLvFiC9K4=', 'Content-Length': '0'} 2023-07-29T00:25:04.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bb9e5d64459a201-0064c45c60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.382 DEBUG:boto:path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:25:04.382 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:25:04.383 DEBUG:boto:Method: HEAD 2023-07-29T00:25:04.383 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000000 2023-07-29T00:25:04.384 DEBUG:boto:Data: 2023-07-29T00:25:04.385 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.386 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:04.386 DEBUG:boto:Port: 8000 2023-07-29T00:25:04.386 DEBUG:boto:Params: {} 2023-07-29T00:25:04.387 DEBUG:boto:Token: None 2023-07-29T00:25:04.387 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/obj00000000000000000000 2023-07-29T00:25:04.388 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rfUfvIDr/7MDcaSkt+LLvFiC9K4= 2023-07-29T00:25:04.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rfUfvIDr/7MDcaSkt+LLvFiC9K4=', 'Content-Length': '0'} 2023-07-29T00:25:04.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f34e745cfa3d0596-0064c45c60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-07-29T00:25:04.391 DEBUG:boto:path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:25:04.392 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:25:04.393 DEBUG:boto:Method: GET 2023-07-29T00:25:04.393 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000000 2023-07-29T00:25:04.393 DEBUG:boto:Data: 2023-07-29T00:25:04.394 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.394 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:04.394 DEBUG:boto:Port: 8001 2023-07-29T00:25:04.395 DEBUG:boto:Params: {} 2023-07-29T00:25:04.395 DEBUG:boto:Token: None 2023-07-29T00:25:04.396 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/obj00000000000000000000 2023-07-29T00:25:04.396 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:za2KCFtRj080XHSQDyNqc+97+ng= 2023-07-29T00:25:04.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:za2KCFtRj080XHSQDyNqc+97+ng=', 'Content-Length': '0'} 2023-07-29T00:25:04.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b338ccd7df49bec-0064c45c60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.440 DEBUG:boto:path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:25:04.441 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000000 2023-07-29T00:25:04.441 DEBUG:boto:Method: GET 2023-07-29T00:25:04.442 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000000 2023-07-29T00:25:04.442 DEBUG:boto:Data: 2023-07-29T00:25:04.442 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.443 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:04.443 DEBUG:boto:Port: 8000 2023-07-29T00:25:04.443 DEBUG:boto:Params: {} 2023-07-29T00:25:04.444 DEBUG:boto:Token: None 2023-07-29T00:25:04.444 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/obj00000000000000000000 2023-07-29T00:25:04.445 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:za2KCFtRj080XHSQDyNqc+97+ng= 2023-07-29T00:25:04.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:za2KCFtRj080XHSQDyNqc+97+ng=', 'Content-Length': '0'} 2023-07-29T00:25:04.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ccfccdd89e6df51-0064c45c60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.448 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:04.448 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:04.449 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000000 2023-07-29T00:25:04.449 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000000 2023-07-29T00:25:04.449 DEBUG:boto:Method: GET 2023-07-29T00:25:04.450 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000000 2023-07-29T00:25:04.450 DEBUG:boto:Data: 2023-07-29T00:25:04.450 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.451 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:04.451 DEBUG:boto:Port: 8001 2023-07-29T00:25:04.451 DEBUG:boto:Params: {} 2023-07-29T00:25:04.452 DEBUG:boto:Token: None 2023-07-29T00:25:04.452 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/ 2023-07-29T00:25:04.453 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TBsjLEJxMDSPwueQSOR6oKQJ65c= 2023-07-29T00:25:04.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TBsjLEJxMDSPwueQSOR6oKQJ65c=', 'Content-Length': '0'} 2023-07-29T00:25:04.455 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043182d2a441ad78e-0064c45c60-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.456 DEBUG:boto:b'hiiqos-58obj000000000000000000001000falseobj000000000000000000002023-07-29T00:24:59.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:04.457 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:04.457 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:04.457 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000000 2023-07-29T00:25:04.458 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000000 2023-07-29T00:25:04.458 DEBUG:boto:Method: GET 2023-07-29T00:25:04.458 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000000 2023-07-29T00:25:04.459 DEBUG:boto:Data: 2023-07-29T00:25:04.459 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.459 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:04.460 DEBUG:boto:Port: 8000 2023-07-29T00:25:04.460 DEBUG:boto:Params: {} 2023-07-29T00:25:04.461 DEBUG:boto:Token: None 2023-07-29T00:25:04.461 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/ 2023-07-29T00:25:04.461 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TBsjLEJxMDSPwueQSOR6oKQJ65c= 2023-07-29T00:25:04.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TBsjLEJxMDSPwueQSOR6oKQJ65c=', 'Content-Length': '0'} 2023-07-29T00:25:04.465 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001eb7e433faeef430-0064c45c60-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.466 DEBUG:boto:b'hiiqos-58obj000000000000000000001000falseobj000000000000000000002023-07-29T00:24:59.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:04.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-07-29T00:25:04.467 DEBUG:boto:path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:25:04.467 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:25:04.467 DEBUG:boto:path=/hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:25:04.468 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:25:04.468 DEBUG:boto:Method: GET 2023-07-29T00:25:04.469 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:25:04.469 DEBUG:boto:Data: 2023-07-29T00:25:04.469 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.470 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:04.470 DEBUG:boto:Port: 8001 2023-07-29T00:25:04.470 DEBUG:boto:Params: {} 2023-07-29T00:25:04.471 DEBUG:boto:Token: None 2023-07-29T00:25:04.471 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:25:04.472 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:En93UPeXExw/NB2ZGmJtw+4mRSc= 2023-07-29T00:25:04.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:En93UPeXExw/NB2ZGmJtw+4mRSc=', 'Content-Length': '0'} 2023-07-29T00:25:04.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab1620bcc728a8aa-0064c45c60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.475 DEBUG:boto:path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:25:04.475 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:25:04.476 DEBUG:boto:path=/hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:25:04.476 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:25:04.477 DEBUG:boto:Method: GET 2023-07-29T00:25:04.477 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:25:04.477 DEBUG:boto:Data: 2023-07-29T00:25:04.478 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.478 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:04.478 DEBUG:boto:Port: 8000 2023-07-29T00:25:04.479 DEBUG:boto:Params: {} 2023-07-29T00:25:04.479 DEBUG:boto:Token: None 2023-07-29T00:25:04.480 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:25:04.480 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:En93UPeXExw/NB2ZGmJtw+4mRSc= 2023-07-29T00:25:04.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:En93UPeXExw/NB2ZGmJtw+4mRSc=', 'Content-Length': '0'} 2023-07-29T00:25:04.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ce2357b9ac4564f-0064c45c60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.483 DEBUG:boto:path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:25:04.483 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:25:04.484 DEBUG:boto:path=/hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:25:04.484 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:25:04.485 DEBUG:boto:Method: HEAD 2023-07-29T00:25:04.485 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:25:04.485 DEBUG:boto:Data: 2023-07-29T00:25:04.486 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.486 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:04.486 DEBUG:boto:Port: 8001 2023-07-29T00:25:04.487 DEBUG:boto:Params: {} 2023-07-29T00:25:04.487 DEBUG:boto:Token: None 2023-07-29T00:25:04.487 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:25:04.488 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bEv037Gw22Fcw07P6E/tjGnfnfc= 2023-07-29T00:25:04.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bEv037Gw22Fcw07P6E/tjGnfnfc=', 'Content-Length': '0'} 2023-07-29T00:25:04.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c5c8e556f9940bb-0064c45c60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.490 DEBUG:boto:path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:25:04.491 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:25:04.491 DEBUG:boto:path=/hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:25:04.491 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:25:04.492 DEBUG:boto:Method: HEAD 2023-07-29T00:25:04.492 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:25:04.493 DEBUG:boto:Data: 2023-07-29T00:25:04.493 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.493 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:04.494 DEBUG:boto:Port: 8000 2023-07-29T00:25:04.494 DEBUG:boto:Params: {} 2023-07-29T00:25:04.494 DEBUG:boto:Token: None 2023-07-29T00:25:04.495 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:25:04.495 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bEv037Gw22Fcw07P6E/tjGnfnfc= 2023-07-29T00:25:04.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bEv037Gw22Fcw07P6E/tjGnfnfc=', 'Content-Length': '0'} 2023-07-29T00:25:04.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083a93c56a8ce0718-0064c45c60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-07-29T00:25:04.498 DEBUG:boto:path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:25:04.498 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:25:04.499 DEBUG:boto:path=/hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:25:04.499 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:25:04.499 DEBUG:boto:Method: GET 2023-07-29T00:25:04.500 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:25:04.500 DEBUG:boto:Data: 2023-07-29T00:25:04.500 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.501 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:04.501 DEBUG:boto:Port: 8001 2023-07-29T00:25:04.502 DEBUG:boto:Params: {} 2023-07-29T00:25:04.502 DEBUG:boto:Token: None 2023-07-29T00:25:04.503 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:25:04.503 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:En93UPeXExw/NB2ZGmJtw+4mRSc= 2023-07-29T00:25:04.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:En93UPeXExw/NB2ZGmJtw+4mRSc=', 'Content-Length': '0'} 2023-07-29T00:25:04.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0ce6f989ecb044f-0064c45c60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.506 DEBUG:boto:path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:25:04.506 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:25:04.507 DEBUG:boto:path=/hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:25:04.507 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:25:04.507 DEBUG:boto:Method: GET 2023-07-29T00:25:04.508 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:25:04.508 DEBUG:boto:Data: 2023-07-29T00:25:04.508 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.509 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:04.509 DEBUG:boto:Port: 8000 2023-07-29T00:25:04.510 DEBUG:boto:Params: {} 2023-07-29T00:25:04.510 DEBUG:boto:Token: None 2023-07-29T00:25:04.510 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/obj00000000000000000001?versionId=null 2023-07-29T00:25:04.511 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:En93UPeXExw/NB2ZGmJtw+4mRSc= 2023-07-29T00:25:04.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:En93UPeXExw/NB2ZGmJtw+4mRSc=', 'Content-Length': '0'} 2023-07-29T00:25:04.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000530a802be0a0ec7e-0064c45c60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.514 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:04.514 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:04.514 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000001 2023-07-29T00:25:04.515 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000001 2023-07-29T00:25:04.515 DEBUG:boto:Method: GET 2023-07-29T00:25:04.515 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000001 2023-07-29T00:25:04.516 DEBUG:boto:Data: 2023-07-29T00:25:04.516 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.517 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:04.517 DEBUG:boto:Port: 8001 2023-07-29T00:25:04.517 DEBUG:boto:Params: {} 2023-07-29T00:25:04.518 DEBUG:boto:Token: None 2023-07-29T00:25:04.518 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/ 2023-07-29T00:25:04.519 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TBsjLEJxMDSPwueQSOR6oKQJ65c= 2023-07-29T00:25:04.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TBsjLEJxMDSPwueQSOR6oKQJ65c=', 'Content-Length': '0'} 2023-07-29T00:25:04.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092fce3f4e5c2a622-0064c45c60-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.521 DEBUG:boto:b'hiiqos-58obj000000000000000000011000falseobj000000000000000000012023-07-29T00:24:59.749Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:04.522 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:04.523 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:04.523 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000001 2023-07-29T00:25:04.523 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000001 2023-07-29T00:25:04.524 DEBUG:boto:Method: GET 2023-07-29T00:25:04.524 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000001 2023-07-29T00:25:04.524 DEBUG:boto:Data: 2023-07-29T00:25:04.525 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.525 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:04.525 DEBUG:boto:Port: 8000 2023-07-29T00:25:04.526 DEBUG:boto:Params: {} 2023-07-29T00:25:04.526 DEBUG:boto:Token: None 2023-07-29T00:25:04.527 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/ 2023-07-29T00:25:04.527 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TBsjLEJxMDSPwueQSOR6oKQJ65c= 2023-07-29T00:25:04.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TBsjLEJxMDSPwueQSOR6oKQJ65c=', 'Content-Length': '0'} 2023-07-29T00:25:04.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe24d5774fe5b1dd-0064c45c60-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.532 DEBUG:boto:b'hiiqos-58obj000000000000000000011000falseobj000000000000000000012023-07-29T00:24:59.749Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:04.532 DEBUG:boto:path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:25:04.533 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:25:04.533 DEBUG:boto:Method: HEAD 2023-07-29T00:25:04.533 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000001 2023-07-29T00:25:04.534 DEBUG:boto:Data: 2023-07-29T00:25:04.534 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.534 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:04.535 DEBUG:boto:Port: 8001 2023-07-29T00:25:04.535 DEBUG:boto:Params: {} 2023-07-29T00:25:04.536 DEBUG:boto:Token: None 2023-07-29T00:25:04.536 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/obj00000000000000000001 2023-07-29T00:25:04.537 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:537X8IIPGZs5kAEImimdfLEe29g= 2023-07-29T00:25:04.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:537X8IIPGZs5kAEImimdfLEe29g=', 'Content-Length': '0'} 2023-07-29T00:25:04.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007639e8a94ecdbf02-0064c45c60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.539 DEBUG:boto:path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:25:04.539 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:25:04.540 DEBUG:boto:Method: HEAD 2023-07-29T00:25:04.540 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000001 2023-07-29T00:25:04.540 DEBUG:boto:Data: 2023-07-29T00:25:04.541 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.541 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:04.541 DEBUG:boto:Port: 8000 2023-07-29T00:25:04.542 DEBUG:boto:Params: {} 2023-07-29T00:25:04.542 DEBUG:boto:Token: None 2023-07-29T00:25:04.543 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/obj00000000000000000001 2023-07-29T00:25:04.543 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:537X8IIPGZs5kAEImimdfLEe29g= 2023-07-29T00:25:04.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:537X8IIPGZs5kAEImimdfLEe29g=', 'Content-Length': '0'} 2023-07-29T00:25:04.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc95f2d7bea08fa9-0064c45c60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-07-29T00:25:04.546 DEBUG:boto:path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:25:04.546 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:25:04.547 DEBUG:boto:Method: GET 2023-07-29T00:25:04.547 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000001 2023-07-29T00:25:04.547 DEBUG:boto:Data: 2023-07-29T00:25:04.548 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.548 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:04.549 DEBUG:boto:Port: 8001 2023-07-29T00:25:04.549 DEBUG:boto:Params: {} 2023-07-29T00:25:04.549 DEBUG:boto:Token: None 2023-07-29T00:25:04.550 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/obj00000000000000000001 2023-07-29T00:25:04.550 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+ev1mFupz/mIW2EJYf2CkUOQJ1Y= 2023-07-29T00:25:04.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+ev1mFupz/mIW2EJYf2CkUOQJ1Y=', 'Content-Length': '0'} 2023-07-29T00:25:04.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000798b37605f96284f-0064c45c60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.553 DEBUG:boto:path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:25:04.553 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000001 2023-07-29T00:25:04.554 DEBUG:boto:Method: GET 2023-07-29T00:25:04.554 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000001 2023-07-29T00:25:04.554 DEBUG:boto:Data: 2023-07-29T00:25:04.555 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.555 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:04.555 DEBUG:boto:Port: 8000 2023-07-29T00:25:04.556 DEBUG:boto:Params: {} 2023-07-29T00:25:04.556 DEBUG:boto:Token: None 2023-07-29T00:25:04.557 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/obj00000000000000000001 2023-07-29T00:25:04.557 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+ev1mFupz/mIW2EJYf2CkUOQJ1Y= 2023-07-29T00:25:04.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+ev1mFupz/mIW2EJYf2CkUOQJ1Y=', 'Content-Length': '0'} 2023-07-29T00:25:04.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b730b9cee33623f-0064c45c60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.604 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:04.605 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:04.605 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000001 2023-07-29T00:25:04.606 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000001 2023-07-29T00:25:04.606 DEBUG:boto:Method: GET 2023-07-29T00:25:04.607 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000001 2023-07-29T00:25:04.607 DEBUG:boto:Data: 2023-07-29T00:25:04.607 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.608 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:04.608 DEBUG:boto:Port: 8001 2023-07-29T00:25:04.609 DEBUG:boto:Params: {} 2023-07-29T00:25:04.609 DEBUG:boto:Token: None 2023-07-29T00:25:04.609 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/ 2023-07-29T00:25:04.610 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TBsjLEJxMDSPwueQSOR6oKQJ65c= 2023-07-29T00:25:04.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TBsjLEJxMDSPwueQSOR6oKQJ65c=', 'Content-Length': '0'} 2023-07-29T00:25:04.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee1286c44c5548a3-0064c45c60-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.613 DEBUG:boto:b'hiiqos-58obj000000000000000000011000falseobj000000000000000000012023-07-29T00:24:59.749Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:04.614 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:04.614 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:04.615 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000001 2023-07-29T00:25:04.615 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000001 2023-07-29T00:25:04.615 DEBUG:boto:Method: GET 2023-07-29T00:25:04.616 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000001 2023-07-29T00:25:04.616 DEBUG:boto:Data: 2023-07-29T00:25:04.617 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.617 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:04.618 DEBUG:boto:Port: 8000 2023-07-29T00:25:04.618 DEBUG:boto:Params: {} 2023-07-29T00:25:04.619 DEBUG:boto:Token: None 2023-07-29T00:25:04.619 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/ 2023-07-29T00:25:04.619 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TBsjLEJxMDSPwueQSOR6oKQJ65c= 2023-07-29T00:25:04.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TBsjLEJxMDSPwueQSOR6oKQJ65c=', 'Content-Length': '0'} 2023-07-29T00:25:04.624 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c24ab022e1bbe940-0064c45c60-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.624 DEBUG:boto:b'hiiqos-58obj000000000000000000011000falseobj000000000000000000012023-07-29T00:24:59.749Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:04.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-07-29T00:25:04.625 DEBUG:boto:path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:25:04.626 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:25:04.626 DEBUG:boto:path=/hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:25:04.626 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:25:04.627 DEBUG:boto:Method: GET 2023-07-29T00:25:04.628 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:25:04.628 DEBUG:boto:Data: 2023-07-29T00:25:04.628 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.629 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:04.629 DEBUG:boto:Port: 8001 2023-07-29T00:25:04.629 DEBUG:boto:Params: {} 2023-07-29T00:25:04.630 DEBUG:boto:Token: None 2023-07-29T00:25:04.630 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:25:04.631 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BdaJcxS9ro4Y0GBCQW6r9hGjns4= 2023-07-29T00:25:04.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BdaJcxS9ro4Y0GBCQW6r9hGjns4=', 'Content-Length': '0'} 2023-07-29T00:25:04.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044a99dd124cade70-0064c45c60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.676 DEBUG:boto:path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:25:04.677 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:25:04.678 DEBUG:boto:path=/hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:25:04.678 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:25:04.678 DEBUG:boto:Method: GET 2023-07-29T00:25:04.679 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:25:04.679 DEBUG:boto:Data: 2023-07-29T00:25:04.679 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.680 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:04.680 DEBUG:boto:Port: 8000 2023-07-29T00:25:04.681 DEBUG:boto:Params: {} 2023-07-29T00:25:04.681 DEBUG:boto:Token: None 2023-07-29T00:25:04.682 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:25:04.682 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BdaJcxS9ro4Y0GBCQW6r9hGjns4= 2023-07-29T00:25:04.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BdaJcxS9ro4Y0GBCQW6r9hGjns4=', 'Content-Length': '0'} 2023-07-29T00:25:04.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d352695641bd1fc1-0064c45c60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.685 DEBUG:boto:path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:25:04.686 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:25:04.686 DEBUG:boto:path=/hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:25:04.686 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:25:04.687 DEBUG:boto:Method: HEAD 2023-07-29T00:25:04.687 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:25:04.688 DEBUG:boto:Data: 2023-07-29T00:25:04.688 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.688 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:04.689 DEBUG:boto:Port: 8001 2023-07-29T00:25:04.689 DEBUG:boto:Params: {} 2023-07-29T00:25:04.689 DEBUG:boto:Token: None 2023-07-29T00:25:04.690 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:25:04.691 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TBZVK5J6V2p1gSIXMUYSSPcysbY= 2023-07-29T00:25:04.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TBZVK5J6V2p1gSIXMUYSSPcysbY=', 'Content-Length': '0'} 2023-07-29T00:25:04.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d92ad0d5af414f1b-0064c45c60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.694 DEBUG:boto:path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:25:04.694 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:25:04.695 DEBUG:boto:path=/hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:25:04.695 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:25:04.695 DEBUG:boto:Method: HEAD 2023-07-29T00:25:04.696 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:25:04.696 DEBUG:boto:Data: 2023-07-29T00:25:04.697 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.697 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:04.697 DEBUG:boto:Port: 8000 2023-07-29T00:25:04.698 DEBUG:boto:Params: {} 2023-07-29T00:25:04.698 DEBUG:boto:Token: None 2023-07-29T00:25:04.699 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:25:04.699 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TBZVK5J6V2p1gSIXMUYSSPcysbY= 2023-07-29T00:25:04.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TBZVK5J6V2p1gSIXMUYSSPcysbY=', 'Content-Length': '0'} 2023-07-29T00:25:04.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6b9a39d906650c4-0064c45c60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-07-29T00:25:04.702 DEBUG:boto:path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:25:04.702 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:25:04.703 DEBUG:boto:path=/hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:25:04.703 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:25:04.704 DEBUG:boto:Method: GET 2023-07-29T00:25:04.704 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:25:04.704 DEBUG:boto:Data: 2023-07-29T00:25:04.705 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.705 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:04.705 DEBUG:boto:Port: 8001 2023-07-29T00:25:04.706 DEBUG:boto:Params: {} 2023-07-29T00:25:04.706 DEBUG:boto:Token: None 2023-07-29T00:25:04.707 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:25:04.707 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BdaJcxS9ro4Y0GBCQW6r9hGjns4= 2023-07-29T00:25:04.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BdaJcxS9ro4Y0GBCQW6r9hGjns4=', 'Content-Length': '0'} 2023-07-29T00:25:04.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000087194298fa65ace-0064c45c60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.710 DEBUG:boto:path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:25:04.711 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:25:04.711 DEBUG:boto:path=/hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:25:04.711 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:25:04.712 DEBUG:boto:Method: GET 2023-07-29T00:25:04.712 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:25:04.713 DEBUG:boto:Data: 2023-07-29T00:25:04.713 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.713 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:04.714 DEBUG:boto:Port: 8000 2023-07-29T00:25:04.714 DEBUG:boto:Params: {} 2023-07-29T00:25:04.714 DEBUG:boto:Token: None 2023-07-29T00:25:04.715 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/obj00000000000000000002?versionId=null 2023-07-29T00:25:04.715 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BdaJcxS9ro4Y0GBCQW6r9hGjns4= 2023-07-29T00:25:04.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BdaJcxS9ro4Y0GBCQW6r9hGjns4=', 'Content-Length': '0'} 2023-07-29T00:25:04.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e1d72f58658a511-0064c45c60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.718 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:04.719 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:04.719 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000002 2023-07-29T00:25:04.720 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000002 2023-07-29T00:25:04.720 DEBUG:boto:Method: GET 2023-07-29T00:25:04.721 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000002 2023-07-29T00:25:04.721 DEBUG:boto:Data: 2023-07-29T00:25:04.722 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.723 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:04.723 DEBUG:boto:Port: 8001 2023-07-29T00:25:04.724 DEBUG:boto:Params: {} 2023-07-29T00:25:04.725 DEBUG:boto:Token: None 2023-07-29T00:25:04.726 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/ 2023-07-29T00:25:04.727 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TBsjLEJxMDSPwueQSOR6oKQJ65c= 2023-07-29T00:25:04.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TBsjLEJxMDSPwueQSOR6oKQJ65c=', 'Content-Length': '0'} 2023-07-29T00:25:04.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6053171c5d068d9-0064c45c60-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.732 DEBUG:boto:b'hiiqos-58obj000000000000000000021000falseobj000000000000000000022023-07-29T00:24:59.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:04.733 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:04.733 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:04.733 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000002 2023-07-29T00:25:04.734 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000002 2023-07-29T00:25:04.734 DEBUG:boto:Method: GET 2023-07-29T00:25:04.734 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000002 2023-07-29T00:25:04.735 DEBUG:boto:Data: 2023-07-29T00:25:04.735 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.735 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:04.736 DEBUG:boto:Port: 8000 2023-07-29T00:25:04.736 DEBUG:boto:Params: {} 2023-07-29T00:25:04.737 DEBUG:boto:Token: None 2023-07-29T00:25:04.737 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/ 2023-07-29T00:25:04.737 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TBsjLEJxMDSPwueQSOR6oKQJ65c= 2023-07-29T00:25:04.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TBsjLEJxMDSPwueQSOR6oKQJ65c=', 'Content-Length': '0'} 2023-07-29T00:25:04.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f092b2c89bf9807-0064c45c60-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.742 DEBUG:boto:b'hiiqos-58obj000000000000000000021000falseobj000000000000000000022023-07-29T00:24:59.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:04.743 DEBUG:boto:path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:25:04.743 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:25:04.744 DEBUG:boto:Method: HEAD 2023-07-29T00:25:04.744 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000002 2023-07-29T00:25:04.745 DEBUG:boto:Data: 2023-07-29T00:25:04.745 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.745 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:04.746 DEBUG:boto:Port: 8001 2023-07-29T00:25:04.746 DEBUG:boto:Params: {} 2023-07-29T00:25:04.746 DEBUG:boto:Token: None 2023-07-29T00:25:04.747 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/obj00000000000000000002 2023-07-29T00:25:04.747 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LfdDJHg1eDkpPi2L+6PlgUc/aZI= 2023-07-29T00:25:04.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LfdDJHg1eDkpPi2L+6PlgUc/aZI=', 'Content-Length': '0'} 2023-07-29T00:25:04.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea3518435ec75062-0064c45c60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.750 DEBUG:boto:path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:25:04.750 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:25:04.750 DEBUG:boto:Method: HEAD 2023-07-29T00:25:04.751 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000002 2023-07-29T00:25:04.751 DEBUG:boto:Data: 2023-07-29T00:25:04.751 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.752 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:04.752 DEBUG:boto:Port: 8000 2023-07-29T00:25:04.752 DEBUG:boto:Params: {} 2023-07-29T00:25:04.753 DEBUG:boto:Token: None 2023-07-29T00:25:04.753 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/obj00000000000000000002 2023-07-29T00:25:04.754 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LfdDJHg1eDkpPi2L+6PlgUc/aZI= 2023-07-29T00:25:04.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LfdDJHg1eDkpPi2L+6PlgUc/aZI=', 'Content-Length': '0'} 2023-07-29T00:25:04.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007df65dca7b37857b-0064c45c60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-07-29T00:25:04.757 DEBUG:boto:path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:25:04.757 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:25:04.757 DEBUG:boto:Method: GET 2023-07-29T00:25:04.758 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000002 2023-07-29T00:25:04.758 DEBUG:boto:Data: 2023-07-29T00:25:04.758 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.759 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:04.759 DEBUG:boto:Port: 8001 2023-07-29T00:25:04.759 DEBUG:boto:Params: {} 2023-07-29T00:25:04.760 DEBUG:boto:Token: None 2023-07-29T00:25:04.760 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/obj00000000000000000002 2023-07-29T00:25:04.761 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZL8keOKIE6t5he1FGjnI0dbj1Kk= 2023-07-29T00:25:04.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZL8keOKIE6t5he1FGjnI0dbj1Kk=', 'Content-Length': '0'} 2023-07-29T00:25:04.763 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3b2e7bcfe4f7925-0064c45c60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.763 DEBUG:boto:path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:25:04.764 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000002 2023-07-29T00:25:04.764 DEBUG:boto:Method: GET 2023-07-29T00:25:04.765 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000002 2023-07-29T00:25:04.765 DEBUG:boto:Data: 2023-07-29T00:25:04.765 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.766 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:04.766 DEBUG:boto:Port: 8000 2023-07-29T00:25:04.766 DEBUG:boto:Params: {} 2023-07-29T00:25:04.767 DEBUG:boto:Token: None 2023-07-29T00:25:04.767 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/obj00000000000000000002 2023-07-29T00:25:04.768 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZL8keOKIE6t5he1FGjnI0dbj1Kk= 2023-07-29T00:25:04.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZL8keOKIE6t5he1FGjnI0dbj1Kk=', 'Content-Length': '0'} 2023-07-29T00:25:04.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f43031233f27119-0064c45c60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.812 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:04.813 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:04.813 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000002 2023-07-29T00:25:04.814 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000002 2023-07-29T00:25:04.814 DEBUG:boto:Method: GET 2023-07-29T00:25:04.814 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000002 2023-07-29T00:25:04.815 DEBUG:boto:Data: 2023-07-29T00:25:04.815 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.815 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:04.816 DEBUG:boto:Port: 8001 2023-07-29T00:25:04.816 DEBUG:boto:Params: {} 2023-07-29T00:25:04.817 DEBUG:boto:Token: None 2023-07-29T00:25:04.817 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/ 2023-07-29T00:25:04.818 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TBsjLEJxMDSPwueQSOR6oKQJ65c= 2023-07-29T00:25:04.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TBsjLEJxMDSPwueQSOR6oKQJ65c=', 'Content-Length': '0'} 2023-07-29T00:25:04.820 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d83ab77a8ce02731-0064c45c60-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.821 DEBUG:boto:b'hiiqos-58obj000000000000000000021000falseobj000000000000000000022023-07-29T00:24:59.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:04.821 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:04.822 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:04.822 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000002 2023-07-29T00:25:04.822 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000002 2023-07-29T00:25:04.823 DEBUG:boto:Method: GET 2023-07-29T00:25:04.823 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000002 2023-07-29T00:25:04.824 DEBUG:boto:Data: 2023-07-29T00:25:04.824 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.824 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:04.825 DEBUG:boto:Port: 8000 2023-07-29T00:25:04.825 DEBUG:boto:Params: {} 2023-07-29T00:25:04.825 DEBUG:boto:Token: None 2023-07-29T00:25:04.826 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/ 2023-07-29T00:25:04.826 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TBsjLEJxMDSPwueQSOR6oKQJ65c= 2023-07-29T00:25:04.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TBsjLEJxMDSPwueQSOR6oKQJ65c=', 'Content-Length': '0'} 2023-07-29T00:25:04.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029675a615b20b273-0064c45c60-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.833 DEBUG:boto:b'hiiqos-58obj000000000000000000021000falseobj000000000000000000022023-07-29T00:24:59.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:04.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-07-29T00:25:04.834 DEBUG:boto:path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:25:04.834 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:25:04.835 DEBUG:boto:path=/hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:25:04.835 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:25:04.835 DEBUG:boto:Method: GET 2023-07-29T00:25:04.836 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:25:04.836 DEBUG:boto:Data: 2023-07-29T00:25:04.836 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.837 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:04.837 DEBUG:boto:Port: 8001 2023-07-29T00:25:04.837 DEBUG:boto:Params: {} 2023-07-29T00:25:04.838 DEBUG:boto:Token: None 2023-07-29T00:25:04.838 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:25:04.839 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yTTMMzS7ELFWnSh+OGwaVcx0uNM= 2023-07-29T00:25:04.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yTTMMzS7ELFWnSh+OGwaVcx0uNM=', 'Content-Length': '0'} 2023-07-29T00:25:04.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000954c105114588f18-0064c45c60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.884 DEBUG:boto:path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:25:04.885 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:25:04.886 DEBUG:boto:path=/hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:25:04.886 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:25:04.886 DEBUG:boto:Method: GET 2023-07-29T00:25:04.887 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:25:04.887 DEBUG:boto:Data: 2023-07-29T00:25:04.887 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.888 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:04.888 DEBUG:boto:Port: 8000 2023-07-29T00:25:04.888 DEBUG:boto:Params: {} 2023-07-29T00:25:04.889 DEBUG:boto:Token: None 2023-07-29T00:25:04.889 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:25:04.890 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yTTMMzS7ELFWnSh+OGwaVcx0uNM= 2023-07-29T00:25:04.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yTTMMzS7ELFWnSh+OGwaVcx0uNM=', 'Content-Length': '0'} 2023-07-29T00:25:04.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eba59857dad6d549-0064c45c60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.893 DEBUG:boto:path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:25:04.894 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:25:04.894 DEBUG:boto:path=/hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:25:04.894 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:25:04.895 DEBUG:boto:Method: HEAD 2023-07-29T00:25:04.895 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:25:04.896 DEBUG:boto:Data: 2023-07-29T00:25:04.896 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.896 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:04.897 DEBUG:boto:Port: 8001 2023-07-29T00:25:04.897 DEBUG:boto:Params: {} 2023-07-29T00:25:04.897 DEBUG:boto:Token: None 2023-07-29T00:25:04.898 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:25:04.898 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5/hSu3IxwBTh7jLRj1Bre9GcPoA= 2023-07-29T00:25:04.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5/hSu3IxwBTh7jLRj1Bre9GcPoA=', 'Content-Length': '0'} 2023-07-29T00:25:04.900 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3194bbab71359f4-0064c45c60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.901 DEBUG:boto:path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:25:04.901 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:25:04.902 DEBUG:boto:path=/hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:25:04.902 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:25:04.902 DEBUG:boto:Method: HEAD 2023-07-29T00:25:04.903 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:25:04.903 DEBUG:boto:Data: 2023-07-29T00:25:04.903 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.904 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:04.904 DEBUG:boto:Port: 8000 2023-07-29T00:25:04.905 DEBUG:boto:Params: {} 2023-07-29T00:25:04.905 DEBUG:boto:Token: None 2023-07-29T00:25:04.905 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:25:04.906 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5/hSu3IxwBTh7jLRj1Bre9GcPoA= 2023-07-29T00:25:04.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5/hSu3IxwBTh7jLRj1Bre9GcPoA=', 'Content-Length': '0'} 2023-07-29T00:25:04.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e12f92e6eff0233e-0064c45c60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-07-29T00:25:04.909 DEBUG:boto:path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:25:04.909 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:25:04.910 DEBUG:boto:path=/hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:25:04.910 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:25:04.910 DEBUG:boto:Method: GET 2023-07-29T00:25:04.911 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:25:04.911 DEBUG:boto:Data: 2023-07-29T00:25:04.911 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.912 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:04.912 DEBUG:boto:Port: 8001 2023-07-29T00:25:04.913 DEBUG:boto:Params: {} 2023-07-29T00:25:04.913 DEBUG:boto:Token: None 2023-07-29T00:25:04.913 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:25:04.914 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yTTMMzS7ELFWnSh+OGwaVcx0uNM= 2023-07-29T00:25:04.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yTTMMzS7ELFWnSh+OGwaVcx0uNM=', 'Content-Length': '0'} 2023-07-29T00:25:04.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f379874191727da-0064c45c60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.917 DEBUG:boto:path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:25:04.918 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:25:04.918 DEBUG:boto:path=/hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:25:04.918 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:25:04.919 DEBUG:boto:Method: GET 2023-07-29T00:25:04.919 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:25:04.919 DEBUG:boto:Data: 2023-07-29T00:25:04.920 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.920 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:04.921 DEBUG:boto:Port: 8000 2023-07-29T00:25:04.921 DEBUG:boto:Params: {} 2023-07-29T00:25:04.922 DEBUG:boto:Token: None 2023-07-29T00:25:04.922 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/obj00000000000000000003?versionId=null 2023-07-29T00:25:04.923 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yTTMMzS7ELFWnSh+OGwaVcx0uNM= 2023-07-29T00:25:04.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yTTMMzS7ELFWnSh+OGwaVcx0uNM=', 'Content-Length': '0'} 2023-07-29T00:25:04.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c104e24f4b4eab9-0064c45c60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.926 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:04.926 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:04.927 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000003 2023-07-29T00:25:04.927 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000003 2023-07-29T00:25:04.928 DEBUG:boto:Method: GET 2023-07-29T00:25:04.928 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000003 2023-07-29T00:25:04.928 DEBUG:boto:Data: 2023-07-29T00:25:04.929 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.929 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:04.930 DEBUG:boto:Port: 8001 2023-07-29T00:25:04.930 DEBUG:boto:Params: {} 2023-07-29T00:25:04.930 DEBUG:boto:Token: None 2023-07-29T00:25:04.931 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/ 2023-07-29T00:25:04.931 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TBsjLEJxMDSPwueQSOR6oKQJ65c= 2023-07-29T00:25:04.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TBsjLEJxMDSPwueQSOR6oKQJ65c=', 'Content-Length': '0'} 2023-07-29T00:25:04.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b554ae303f9ff2b5-0064c45c60-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.977 DEBUG:boto:b'hiiqos-58obj000000000000000000031000falseobj000000000000000000032023-07-29T00:25:00.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:04.978 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:04.978 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:04.978 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000003 2023-07-29T00:25:04.979 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000003 2023-07-29T00:25:04.979 DEBUG:boto:Method: GET 2023-07-29T00:25:04.979 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000003 2023-07-29T00:25:04.980 DEBUG:boto:Data: 2023-07-29T00:25:04.980 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.981 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:04.981 DEBUG:boto:Port: 8000 2023-07-29T00:25:04.981 DEBUG:boto:Params: {} 2023-07-29T00:25:04.982 DEBUG:boto:Token: None 2023-07-29T00:25:04.982 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/ 2023-07-29T00:25:04.983 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TBsjLEJxMDSPwueQSOR6oKQJ65c= 2023-07-29T00:25:04.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TBsjLEJxMDSPwueQSOR6oKQJ65c=', 'Content-Length': '0'} 2023-07-29T00:25:04.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000064d944b854fc870-0064c45c60-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.987 DEBUG:boto:b'hiiqos-58obj000000000000000000031000falseobj000000000000000000032023-07-29T00:25:00.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:04.988 DEBUG:boto:path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:25:04.988 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:25:04.989 DEBUG:boto:Method: HEAD 2023-07-29T00:25:04.989 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000003 2023-07-29T00:25:04.990 DEBUG:boto:Data: 2023-07-29T00:25:04.990 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.991 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:04.991 DEBUG:boto:Port: 8001 2023-07-29T00:25:04.991 DEBUG:boto:Params: {} 2023-07-29T00:25:04.992 DEBUG:boto:Token: None 2023-07-29T00:25:04.992 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/obj00000000000000000003 2023-07-29T00:25:04.993 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:U6iZcLz3PqewPiIRJnDOwUmpKd0= 2023-07-29T00:25:04.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:U6iZcLz3PqewPiIRJnDOwUmpKd0=', 'Content-Length': '0'} 2023-07-29T00:25:04.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000048fbdcedfae52d-0064c45c60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:04.996 DEBUG:boto:path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:25:04.996 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:25:04.996 DEBUG:boto:Method: HEAD 2023-07-29T00:25:04.997 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000003 2023-07-29T00:25:04.997 DEBUG:boto:Data: 2023-07-29T00:25:04.997 DEBUG:boto:Headers: {} 2023-07-29T00:25:04.998 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:04.998 DEBUG:boto:Port: 8000 2023-07-29T00:25:04.998 DEBUG:boto:Params: {} 2023-07-29T00:25:04.999 DEBUG:boto:Token: None 2023-07-29T00:25:04.999 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:04 GMT /hiiqos-58/obj00000000000000000003 2023-07-29T00:25:05.000 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:U6iZcLz3PqewPiIRJnDOwUmpKd0= 2023-07-29T00:25:05.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:U6iZcLz3PqewPiIRJnDOwUmpKd0=', 'Content-Length': '0'} 2023-07-29T00:25:05.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000876af404dfcb2d3e-0064c45c60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-07-29T00:25:05.003 DEBUG:boto:path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:25:05.004 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:25:05.004 DEBUG:boto:Method: GET 2023-07-29T00:25:05.004 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000003 2023-07-29T00:25:05.005 DEBUG:boto:Data: 2023-07-29T00:25:05.005 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.006 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:05.006 DEBUG:boto:Port: 8001 2023-07-29T00:25:05.006 DEBUG:boto:Params: {} 2023-07-29T00:25:05.007 DEBUG:boto:Token: None 2023-07-29T00:25:05.007 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000003 2023-07-29T00:25:05.008 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WoHuVjnBjERCjqrCNHIGGB3Tbvo= 2023-07-29T00:25:05.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WoHuVjnBjERCjqrCNHIGGB3Tbvo=', 'Content-Length': '0'} 2023-07-29T00:25:05.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6b4301864b5edb6-0064c45c60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.011 DEBUG:boto:path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:25:05.012 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000003 2023-07-29T00:25:05.012 DEBUG:boto:Method: GET 2023-07-29T00:25:05.012 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000003 2023-07-29T00:25:05.013 DEBUG:boto:Data: 2023-07-29T00:25:05.013 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.014 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:05.014 DEBUG:boto:Port: 8000 2023-07-29T00:25:05.014 DEBUG:boto:Params: {} 2023-07-29T00:25:05.015 DEBUG:boto:Token: None 2023-07-29T00:25:05.016 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000003 2023-07-29T00:25:05.016 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WoHuVjnBjERCjqrCNHIGGB3Tbvo= 2023-07-29T00:25:05.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WoHuVjnBjERCjqrCNHIGGB3Tbvo=', 'Content-Length': '0'} 2023-07-29T00:25:05.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000286ed37c6643cc8-0064c45c60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.060 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:05.061 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:05.062 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000003 2023-07-29T00:25:05.062 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000003 2023-07-29T00:25:05.062 DEBUG:boto:Method: GET 2023-07-29T00:25:05.063 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000003 2023-07-29T00:25:05.063 DEBUG:boto:Data: 2023-07-29T00:25:05.064 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.064 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:05.065 DEBUG:boto:Port: 8001 2023-07-29T00:25:05.065 DEBUG:boto:Params: {} 2023-07-29T00:25:05.065 DEBUG:boto:Token: None 2023-07-29T00:25:05.066 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/ 2023-07-29T00:25:05.066 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:p0mCby/Aoe7TFeGHqvTp57nCytE= 2023-07-29T00:25:05.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:p0mCby/Aoe7TFeGHqvTp57nCytE=', 'Content-Length': '0'} 2023-07-29T00:25:05.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c677f62e41aa0f5c-0064c45c61-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.071 DEBUG:boto:b'hiiqos-58obj000000000000000000031000falseobj000000000000000000032023-07-29T00:25:00.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:05.072 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:05.073 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:05.073 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000003 2023-07-29T00:25:05.074 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000003 2023-07-29T00:25:05.075 DEBUG:boto:Method: GET 2023-07-29T00:25:05.075 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000003 2023-07-29T00:25:05.075 DEBUG:boto:Data: 2023-07-29T00:25:05.076 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.076 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:05.077 DEBUG:boto:Port: 8000 2023-07-29T00:25:05.077 DEBUG:boto:Params: {} 2023-07-29T00:25:05.078 DEBUG:boto:Token: None 2023-07-29T00:25:05.078 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/ 2023-07-29T00:25:05.079 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:p0mCby/Aoe7TFeGHqvTp57nCytE= 2023-07-29T00:25:05.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:p0mCby/Aoe7TFeGHqvTp57nCytE=', 'Content-Length': '0'} 2023-07-29T00:25:05.083 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c10b70fda823b847-0064c45c61-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.084 DEBUG:boto:b'hiiqos-58obj000000000000000000031000falseobj000000000000000000032023-07-29T00:25:00.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:05.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-07-29T00:25:05.085 DEBUG:boto:path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:25:05.085 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:25:05.086 DEBUG:boto:path=/hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:25:05.086 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:25:05.086 DEBUG:boto:Method: GET 2023-07-29T00:25:05.087 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:25:05.087 DEBUG:boto:Data: 2023-07-29T00:25:05.087 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.088 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:05.088 DEBUG:boto:Port: 8001 2023-07-29T00:25:05.089 DEBUG:boto:Params: {} 2023-07-29T00:25:05.089 DEBUG:boto:Token: None 2023-07-29T00:25:05.089 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:25:05.090 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EFjsz7ZDhCqwY5/h3TZ/1uyUcww= 2023-07-29T00:25:05.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EFjsz7ZDhCqwY5/h3TZ/1uyUcww=', 'Content-Length': '0'} 2023-07-29T00:25:05.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c41862fb29bb882-0064c45c61-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.093 DEBUG:boto:path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:25:05.094 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:25:05.095 DEBUG:boto:path=/hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:25:05.095 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:25:05.096 DEBUG:boto:Method: GET 2023-07-29T00:25:05.096 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:25:05.097 DEBUG:boto:Data: 2023-07-29T00:25:05.098 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.099 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:05.099 DEBUG:boto:Port: 8000 2023-07-29T00:25:05.100 DEBUG:boto:Params: {} 2023-07-29T00:25:05.101 DEBUG:boto:Token: None 2023-07-29T00:25:05.101 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:25:05.102 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EFjsz7ZDhCqwY5/h3TZ/1uyUcww= 2023-07-29T00:25:05.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EFjsz7ZDhCqwY5/h3TZ/1uyUcww=', 'Content-Length': '0'} 2023-07-29T00:25:05.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e18213173f620318-0064c45c61-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.106 DEBUG:boto:path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:25:05.107 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:25:05.107 DEBUG:boto:path=/hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:25:05.107 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:25:05.108 DEBUG:boto:Method: HEAD 2023-07-29T00:25:05.108 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:25:05.109 DEBUG:boto:Data: 2023-07-29T00:25:05.110 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.110 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:05.111 DEBUG:boto:Port: 8001 2023-07-29T00:25:05.112 DEBUG:boto:Params: {} 2023-07-29T00:25:05.113 DEBUG:boto:Token: None 2023-07-29T00:25:05.114 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:25:05.115 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:34Qb8i66vz9PtV2i6K6prSoFm3o= 2023-07-29T00:25:05.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:34Qb8i66vz9PtV2i6K6prSoFm3o=', 'Content-Length': '0'} 2023-07-29T00:25:05.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000169645bf6d2180ac-0064c45c61-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.118 DEBUG:boto:path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:25:05.119 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:25:05.119 DEBUG:boto:path=/hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:25:05.120 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:25:05.120 DEBUG:boto:Method: HEAD 2023-07-29T00:25:05.121 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:25:05.121 DEBUG:boto:Data: 2023-07-29T00:25:05.121 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.122 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:05.123 DEBUG:boto:Port: 8000 2023-07-29T00:25:05.123 DEBUG:boto:Params: {} 2023-07-29T00:25:05.123 DEBUG:boto:Token: None 2023-07-29T00:25:05.124 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:25:05.124 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:34Qb8i66vz9PtV2i6K6prSoFm3o= 2023-07-29T00:25:05.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:34Qb8i66vz9PtV2i6K6prSoFm3o=', 'Content-Length': '0'} 2023-07-29T00:25:05.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa659d8e4101f104-0064c45c61-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-07-29T00:25:05.127 DEBUG:boto:path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:25:05.128 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:25:05.128 DEBUG:boto:path=/hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:25:05.129 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:25:05.129 DEBUG:boto:Method: GET 2023-07-29T00:25:05.129 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:25:05.130 DEBUG:boto:Data: 2023-07-29T00:25:05.130 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.130 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:05.131 DEBUG:boto:Port: 8001 2023-07-29T00:25:05.131 DEBUG:boto:Params: {} 2023-07-29T00:25:05.132 DEBUG:boto:Token: None 2023-07-29T00:25:05.132 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:25:05.132 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EFjsz7ZDhCqwY5/h3TZ/1uyUcww= 2023-07-29T00:25:05.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EFjsz7ZDhCqwY5/h3TZ/1uyUcww=', 'Content-Length': '0'} 2023-07-29T00:25:05.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000053d3d985a38dcf5-0064c45c61-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.177 DEBUG:boto:path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:25:05.178 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:25:05.178 DEBUG:boto:path=/hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:25:05.178 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:25:05.179 DEBUG:boto:Method: GET 2023-07-29T00:25:05.179 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:25:05.180 DEBUG:boto:Data: 2023-07-29T00:25:05.180 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.180 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:05.181 DEBUG:boto:Port: 8000 2023-07-29T00:25:05.181 DEBUG:boto:Params: {} 2023-07-29T00:25:05.182 DEBUG:boto:Token: None 2023-07-29T00:25:05.182 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000004?versionId=null 2023-07-29T00:25:05.182 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EFjsz7ZDhCqwY5/h3TZ/1uyUcww= 2023-07-29T00:25:05.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EFjsz7ZDhCqwY5/h3TZ/1uyUcww=', 'Content-Length': '0'} 2023-07-29T00:25:05.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afa71e69cae1ca01-0064c45c61-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.185 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:05.186 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:05.186 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000004 2023-07-29T00:25:05.187 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000004 2023-07-29T00:25:05.187 DEBUG:boto:Method: GET 2023-07-29T00:25:05.187 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000004 2023-07-29T00:25:05.188 DEBUG:boto:Data: 2023-07-29T00:25:05.188 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.189 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:05.189 DEBUG:boto:Port: 8001 2023-07-29T00:25:05.189 DEBUG:boto:Params: {} 2023-07-29T00:25:05.190 DEBUG:boto:Token: None 2023-07-29T00:25:05.190 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/ 2023-07-29T00:25:05.191 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:p0mCby/Aoe7TFeGHqvTp57nCytE= 2023-07-29T00:25:05.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:p0mCby/Aoe7TFeGHqvTp57nCytE=', 'Content-Length': '0'} 2023-07-29T00:25:05.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d10413598139633-0064c45c61-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.194 DEBUG:boto:b'hiiqos-58obj000000000000000000041000falseobj000000000000000000042023-07-29T00:24:59.233Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:05.195 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:05.195 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:05.196 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000004 2023-07-29T00:25:05.196 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000004 2023-07-29T00:25:05.196 DEBUG:boto:Method: GET 2023-07-29T00:25:05.197 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000004 2023-07-29T00:25:05.197 DEBUG:boto:Data: 2023-07-29T00:25:05.197 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.198 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:05.198 DEBUG:boto:Port: 8000 2023-07-29T00:25:05.199 DEBUG:boto:Params: {} 2023-07-29T00:25:05.199 DEBUG:boto:Token: None 2023-07-29T00:25:05.199 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/ 2023-07-29T00:25:05.200 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:p0mCby/Aoe7TFeGHqvTp57nCytE= 2023-07-29T00:25:05.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:p0mCby/Aoe7TFeGHqvTp57nCytE=', 'Content-Length': '0'} 2023-07-29T00:25:05.204 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4f92cb2d1ea5228-0064c45c61-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.205 DEBUG:boto:b'hiiqos-58obj000000000000000000041000falseobj000000000000000000042023-07-29T00:24:59.233Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:05.205 DEBUG:boto:path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:25:05.206 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:25:05.206 DEBUG:boto:Method: HEAD 2023-07-29T00:25:05.206 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000004 2023-07-29T00:25:05.207 DEBUG:boto:Data: 2023-07-29T00:25:05.207 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.207 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:05.208 DEBUG:boto:Port: 8001 2023-07-29T00:25:05.208 DEBUG:boto:Params: {} 2023-07-29T00:25:05.209 DEBUG:boto:Token: None 2023-07-29T00:25:05.209 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000004 2023-07-29T00:25:05.209 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:64+T/sMgVxJ+6EaIgHlfnkQmUZs= 2023-07-29T00:25:05.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:64+T/sMgVxJ+6EaIgHlfnkQmUZs=', 'Content-Length': '0'} 2023-07-29T00:25:05.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069ab507e35b7730d-0064c45c61-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.212 DEBUG:boto:path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:25:05.212 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:25:05.213 DEBUG:boto:Method: HEAD 2023-07-29T00:25:05.213 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000004 2023-07-29T00:25:05.213 DEBUG:boto:Data: 2023-07-29T00:25:05.214 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.214 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:05.214 DEBUG:boto:Port: 8000 2023-07-29T00:25:05.215 DEBUG:boto:Params: {} 2023-07-29T00:25:05.215 DEBUG:boto:Token: None 2023-07-29T00:25:05.215 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000004 2023-07-29T00:25:05.216 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:64+T/sMgVxJ+6EaIgHlfnkQmUZs= 2023-07-29T00:25:05.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:64+T/sMgVxJ+6EaIgHlfnkQmUZs=', 'Content-Length': '0'} 2023-07-29T00:25:05.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba7d844322f9927e-0064c45c61-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-07-29T00:25:05.219 DEBUG:boto:path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:25:05.219 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:25:05.220 DEBUG:boto:Method: GET 2023-07-29T00:25:05.220 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000004 2023-07-29T00:25:05.220 DEBUG:boto:Data: 2023-07-29T00:25:05.221 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.221 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:05.222 DEBUG:boto:Port: 8001 2023-07-29T00:25:05.222 DEBUG:boto:Params: {} 2023-07-29T00:25:05.222 DEBUG:boto:Token: None 2023-07-29T00:25:05.223 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000004 2023-07-29T00:25:05.223 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dRtedm5ve5S5EDDA2iozxTSiv7s= 2023-07-29T00:25:05.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dRtedm5ve5S5EDDA2iozxTSiv7s=', 'Content-Length': '0'} 2023-07-29T00:25:05.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f73a7301dfc3144e-0064c45c61-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.226 DEBUG:boto:path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:25:05.227 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000004 2023-07-29T00:25:05.227 DEBUG:boto:Method: GET 2023-07-29T00:25:05.227 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000004 2023-07-29T00:25:05.228 DEBUG:boto:Data: 2023-07-29T00:25:05.228 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.229 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:05.229 DEBUG:boto:Port: 8000 2023-07-29T00:25:05.229 DEBUG:boto:Params: {} 2023-07-29T00:25:05.230 DEBUG:boto:Token: None 2023-07-29T00:25:05.230 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000004 2023-07-29T00:25:05.230 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dRtedm5ve5S5EDDA2iozxTSiv7s= 2023-07-29T00:25:05.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dRtedm5ve5S5EDDA2iozxTSiv7s=', 'Content-Length': '0'} 2023-07-29T00:25:05.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000284dc05befe95b3b-0064c45c61-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.233 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:05.234 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:05.234 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000004 2023-07-29T00:25:05.234 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000004 2023-07-29T00:25:05.235 DEBUG:boto:Method: GET 2023-07-29T00:25:05.235 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000004 2023-07-29T00:25:05.235 DEBUG:boto:Data: 2023-07-29T00:25:05.236 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.236 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:05.237 DEBUG:boto:Port: 8001 2023-07-29T00:25:05.237 DEBUG:boto:Params: {} 2023-07-29T00:25:05.237 DEBUG:boto:Token: None 2023-07-29T00:25:05.238 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/ 2023-07-29T00:25:05.238 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:p0mCby/Aoe7TFeGHqvTp57nCytE= 2023-07-29T00:25:05.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:p0mCby/Aoe7TFeGHqvTp57nCytE=', 'Content-Length': '0'} 2023-07-29T00:25:05.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001758eca140ce789-0064c45c61-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.241 DEBUG:boto:b'hiiqos-58obj000000000000000000041000falseobj000000000000000000042023-07-29T00:24:59.233Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:05.242 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:05.242 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:05.243 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000004 2023-07-29T00:25:05.243 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000004 2023-07-29T00:25:05.244 DEBUG:boto:Method: GET 2023-07-29T00:25:05.244 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000004 2023-07-29T00:25:05.244 DEBUG:boto:Data: 2023-07-29T00:25:05.245 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.245 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:05.245 DEBUG:boto:Port: 8000 2023-07-29T00:25:05.246 DEBUG:boto:Params: {} 2023-07-29T00:25:05.246 DEBUG:boto:Token: None 2023-07-29T00:25:05.246 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/ 2023-07-29T00:25:05.247 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:p0mCby/Aoe7TFeGHqvTp57nCytE= 2023-07-29T00:25:05.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:p0mCby/Aoe7TFeGHqvTp57nCytE=', 'Content-Length': '0'} 2023-07-29T00:25:05.251 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9c172568f700bd1-0064c45c61-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.292 DEBUG:boto:b'hiiqos-58obj000000000000000000041000falseobj000000000000000000042023-07-29T00:24:59.233Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:05.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-07-29T00:25:05.293 DEBUG:boto:path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:25:05.294 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:25:05.294 DEBUG:boto:path=/hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:25:05.295 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:25:05.295 DEBUG:boto:Method: GET 2023-07-29T00:25:05.295 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:25:05.296 DEBUG:boto:Data: 2023-07-29T00:25:05.296 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.296 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:05.297 DEBUG:boto:Port: 8001 2023-07-29T00:25:05.297 DEBUG:boto:Params: {} 2023-07-29T00:25:05.298 DEBUG:boto:Token: None 2023-07-29T00:25:05.298 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:25:05.298 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SWnJAKx2qoVdDBuLTgyKxK1eEpY= 2023-07-29T00:25:05.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SWnJAKx2qoVdDBuLTgyKxK1eEpY=', 'Content-Length': '0'} 2023-07-29T00:25:05.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002c4ac9171285966-0064c45c61-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.302 DEBUG:boto:path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:25:05.302 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:25:05.303 DEBUG:boto:path=/hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:25:05.303 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:25:05.303 DEBUG:boto:Method: GET 2023-07-29T00:25:05.304 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:25:05.304 DEBUG:boto:Data: 2023-07-29T00:25:05.304 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.305 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:05.305 DEBUG:boto:Port: 8000 2023-07-29T00:25:05.306 DEBUG:boto:Params: {} 2023-07-29T00:25:05.306 DEBUG:boto:Token: None 2023-07-29T00:25:05.306 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:25:05.307 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SWnJAKx2qoVdDBuLTgyKxK1eEpY= 2023-07-29T00:25:05.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SWnJAKx2qoVdDBuLTgyKxK1eEpY=', 'Content-Length': '0'} 2023-07-29T00:25:05.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce7d57aed41d49e1-0064c45c61-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.310 DEBUG:boto:path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:25:05.310 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:25:05.310 DEBUG:boto:path=/hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:25:05.311 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:25:05.311 DEBUG:boto:Method: HEAD 2023-07-29T00:25:05.311 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:25:05.312 DEBUG:boto:Data: 2023-07-29T00:25:05.312 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.312 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:05.313 DEBUG:boto:Port: 8001 2023-07-29T00:25:05.313 DEBUG:boto:Params: {} 2023-07-29T00:25:05.313 DEBUG:boto:Token: None 2023-07-29T00:25:05.314 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:25:05.314 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DEgQhtKkE2BttGaT9BV4dMkSnFM= 2023-07-29T00:25:05.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DEgQhtKkE2BttGaT9BV4dMkSnFM=', 'Content-Length': '0'} 2023-07-29T00:25:05.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cedf49460f278d0-0064c45c61-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.317 DEBUG:boto:path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:25:05.317 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:25:05.317 DEBUG:boto:path=/hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:25:05.318 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:25:05.318 DEBUG:boto:Method: HEAD 2023-07-29T00:25:05.318 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:25:05.319 DEBUG:boto:Data: 2023-07-29T00:25:05.319 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.320 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:05.320 DEBUG:boto:Port: 8000 2023-07-29T00:25:05.320 DEBUG:boto:Params: {} 2023-07-29T00:25:05.321 DEBUG:boto:Token: None 2023-07-29T00:25:05.321 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:25:05.321 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DEgQhtKkE2BttGaT9BV4dMkSnFM= 2023-07-29T00:25:05.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DEgQhtKkE2BttGaT9BV4dMkSnFM=', 'Content-Length': '0'} 2023-07-29T00:25:05.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008425f36e1e8bfae3-0064c45c61-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-07-29T00:25:05.324 DEBUG:boto:path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:25:05.325 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:25:05.325 DEBUG:boto:path=/hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:25:05.326 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:25:05.326 DEBUG:boto:Method: GET 2023-07-29T00:25:05.327 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:25:05.327 DEBUG:boto:Data: 2023-07-29T00:25:05.327 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.328 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:05.328 DEBUG:boto:Port: 8001 2023-07-29T00:25:05.328 DEBUG:boto:Params: {} 2023-07-29T00:25:05.329 DEBUG:boto:Token: None 2023-07-29T00:25:05.329 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:25:05.330 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SWnJAKx2qoVdDBuLTgyKxK1eEpY= 2023-07-29T00:25:05.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SWnJAKx2qoVdDBuLTgyKxK1eEpY=', 'Content-Length': '0'} 2023-07-29T00:25:05.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2533eb8852b3e86-0064c45c61-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.378 DEBUG:boto:path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:25:05.378 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:25:05.379 DEBUG:boto:path=/hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:25:05.380 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:25:05.380 DEBUG:boto:Method: GET 2023-07-29T00:25:05.381 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:25:05.381 DEBUG:boto:Data: 2023-07-29T00:25:05.381 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.382 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:05.382 DEBUG:boto:Port: 8000 2023-07-29T00:25:05.382 DEBUG:boto:Params: {} 2023-07-29T00:25:05.383 DEBUG:boto:Token: None 2023-07-29T00:25:05.383 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000005?versionId=null 2023-07-29T00:25:05.384 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SWnJAKx2qoVdDBuLTgyKxK1eEpY= 2023-07-29T00:25:05.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SWnJAKx2qoVdDBuLTgyKxK1eEpY=', 'Content-Length': '0'} 2023-07-29T00:25:05.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d67a9d00bdd2795-0064c45c61-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.387 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:05.387 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:05.388 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000005 2023-07-29T00:25:05.388 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000005 2023-07-29T00:25:05.389 DEBUG:boto:Method: GET 2023-07-29T00:25:05.389 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000005 2023-07-29T00:25:05.389 DEBUG:boto:Data: 2023-07-29T00:25:05.390 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.390 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:05.391 DEBUG:boto:Port: 8001 2023-07-29T00:25:05.391 DEBUG:boto:Params: {} 2023-07-29T00:25:05.392 DEBUG:boto:Token: None 2023-07-29T00:25:05.392 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/ 2023-07-29T00:25:05.393 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:p0mCby/Aoe7TFeGHqvTp57nCytE= 2023-07-29T00:25:05.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:p0mCby/Aoe7TFeGHqvTp57nCytE=', 'Content-Length': '0'} 2023-07-29T00:25:05.395 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000330f56541b9ca7f3-0064c45c61-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.396 DEBUG:boto:b'hiiqos-58obj000000000000000000051000falseobj000000000000000000052023-07-29T00:24:59.414Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:05.396 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:05.397 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:05.397 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000005 2023-07-29T00:25:05.398 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000005 2023-07-29T00:25:05.398 DEBUG:boto:Method: GET 2023-07-29T00:25:05.399 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000005 2023-07-29T00:25:05.399 DEBUG:boto:Data: 2023-07-29T00:25:05.399 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.400 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:05.401 DEBUG:boto:Port: 8000 2023-07-29T00:25:05.401 DEBUG:boto:Params: {} 2023-07-29T00:25:05.401 DEBUG:boto:Token: None 2023-07-29T00:25:05.402 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/ 2023-07-29T00:25:05.402 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:p0mCby/Aoe7TFeGHqvTp57nCytE= 2023-07-29T00:25:05.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:p0mCby/Aoe7TFeGHqvTp57nCytE=', 'Content-Length': '0'} 2023-07-29T00:25:05.406 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bf78aea21a43390-0064c45c61-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.407 DEBUG:boto:b'hiiqos-58obj000000000000000000051000falseobj000000000000000000052023-07-29T00:24:59.414Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:05.408 DEBUG:boto:path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:25:05.408 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:25:05.409 DEBUG:boto:Method: HEAD 2023-07-29T00:25:05.410 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000005 2023-07-29T00:25:05.410 DEBUG:boto:Data: 2023-07-29T00:25:05.410 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.411 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:05.411 DEBUG:boto:Port: 8001 2023-07-29T00:25:05.412 DEBUG:boto:Params: {} 2023-07-29T00:25:05.413 DEBUG:boto:Token: None 2023-07-29T00:25:05.413 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000005 2023-07-29T00:25:05.413 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PO1ba6B3v6Czm85XKuJkIrH45Ew= 2023-07-29T00:25:05.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PO1ba6B3v6Czm85XKuJkIrH45Ew=', 'Content-Length': '0'} 2023-07-29T00:25:05.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000115fb51cff5884da-0064c45c61-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.417 DEBUG:boto:path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:25:05.417 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:25:05.418 DEBUG:boto:Method: HEAD 2023-07-29T00:25:05.418 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000005 2023-07-29T00:25:05.419 DEBUG:boto:Data: 2023-07-29T00:25:05.419 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.419 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:05.420 DEBUG:boto:Port: 8000 2023-07-29T00:25:05.420 DEBUG:boto:Params: {} 2023-07-29T00:25:05.421 DEBUG:boto:Token: None 2023-07-29T00:25:05.422 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000005 2023-07-29T00:25:05.422 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PO1ba6B3v6Czm85XKuJkIrH45Ew= 2023-07-29T00:25:05.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PO1ba6B3v6Czm85XKuJkIrH45Ew=', 'Content-Length': '0'} 2023-07-29T00:25:05.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d37e0e50a3c90d0-0064c45c61-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-07-29T00:25:05.428 DEBUG:boto:path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:25:05.428 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:25:05.429 DEBUG:boto:Method: GET 2023-07-29T00:25:05.429 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000005 2023-07-29T00:25:05.429 DEBUG:boto:Data: 2023-07-29T00:25:05.430 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.430 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:05.431 DEBUG:boto:Port: 8001 2023-07-29T00:25:05.431 DEBUG:boto:Params: {} 2023-07-29T00:25:05.432 DEBUG:boto:Token: None 2023-07-29T00:25:05.432 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000005 2023-07-29T00:25:05.433 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+Ok3Il4C5Ub6lxuAB0EhxzgjQKg= 2023-07-29T00:25:05.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+Ok3Il4C5Ub6lxuAB0EhxzgjQKg=', 'Content-Length': '0'} 2023-07-29T00:25:05.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020f044c5f8a596ab-0064c45c61-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.435 DEBUG:boto:path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:25:05.436 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000005 2023-07-29T00:25:05.436 DEBUG:boto:Method: GET 2023-07-29T00:25:05.437 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000005 2023-07-29T00:25:05.437 DEBUG:boto:Data: 2023-07-29T00:25:05.438 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.438 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:05.438 DEBUG:boto:Port: 8000 2023-07-29T00:25:05.439 DEBUG:boto:Params: {} 2023-07-29T00:25:05.439 DEBUG:boto:Token: None 2023-07-29T00:25:05.440 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000005 2023-07-29T00:25:05.440 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+Ok3Il4C5Ub6lxuAB0EhxzgjQKg= 2023-07-29T00:25:05.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+Ok3Il4C5Ub6lxuAB0EhxzgjQKg=', 'Content-Length': '0'} 2023-07-29T00:25:05.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa1a5e51a9e5b3ba-0064c45c61-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.484 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:05.486 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:05.487 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000005 2023-07-29T00:25:05.487 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000005 2023-07-29T00:25:05.487 DEBUG:boto:Method: GET 2023-07-29T00:25:05.488 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000005 2023-07-29T00:25:05.488 DEBUG:boto:Data: 2023-07-29T00:25:05.489 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.489 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:05.489 DEBUG:boto:Port: 8001 2023-07-29T00:25:05.490 DEBUG:boto:Params: {} 2023-07-29T00:25:05.490 DEBUG:boto:Token: None 2023-07-29T00:25:05.491 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/ 2023-07-29T00:25:05.491 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:p0mCby/Aoe7TFeGHqvTp57nCytE= 2023-07-29T00:25:05.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:p0mCby/Aoe7TFeGHqvTp57nCytE=', 'Content-Length': '0'} 2023-07-29T00:25:05.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008df6065fee6354e8-0064c45c61-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.494 DEBUG:boto:b'hiiqos-58obj000000000000000000051000falseobj000000000000000000052023-07-29T00:24:59.414Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:05.495 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:05.496 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:05.496 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000005 2023-07-29T00:25:05.496 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000005 2023-07-29T00:25:05.497 DEBUG:boto:Method: GET 2023-07-29T00:25:05.497 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000005 2023-07-29T00:25:05.497 DEBUG:boto:Data: 2023-07-29T00:25:05.498 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.498 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:05.499 DEBUG:boto:Port: 8000 2023-07-29T00:25:05.499 DEBUG:boto:Params: {} 2023-07-29T00:25:05.499 DEBUG:boto:Token: None 2023-07-29T00:25:05.500 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/ 2023-07-29T00:25:05.500 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:p0mCby/Aoe7TFeGHqvTp57nCytE= 2023-07-29T00:25:05.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:p0mCby/Aoe7TFeGHqvTp57nCytE=', 'Content-Length': '0'} 2023-07-29T00:25:05.504 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4e482c7f8fd90ac-0064c45c61-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.506 DEBUG:boto:b'hiiqos-58obj000000000000000000051000falseobj000000000000000000052023-07-29T00:24:59.414Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:05.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-07-29T00:25:05.508 DEBUG:boto:path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:25:05.509 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:25:05.509 DEBUG:boto:path=/hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:25:05.509 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:25:05.510 DEBUG:boto:Method: GET 2023-07-29T00:25:05.511 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:25:05.511 DEBUG:boto:Data: 2023-07-29T00:25:05.512 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.512 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:05.513 DEBUG:boto:Port: 8001 2023-07-29T00:25:05.514 DEBUG:boto:Params: {} 2023-07-29T00:25:05.514 DEBUG:boto:Token: None 2023-07-29T00:25:05.515 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:25:05.515 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PcmrIZYceaMtowhgpaunOH2zhh0= 2023-07-29T00:25:05.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PcmrIZYceaMtowhgpaunOH2zhh0=', 'Content-Length': '0'} 2023-07-29T00:25:05.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000e60c957af024a3-0064c45c61-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.518 DEBUG:boto:path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:25:05.519 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:25:05.519 DEBUG:boto:path=/hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:25:05.520 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:25:05.521 DEBUG:boto:Method: GET 2023-07-29T00:25:05.521 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:25:05.522 DEBUG:boto:Data: 2023-07-29T00:25:05.522 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.522 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:05.523 DEBUG:boto:Port: 8000 2023-07-29T00:25:05.523 DEBUG:boto:Params: {} 2023-07-29T00:25:05.524 DEBUG:boto:Token: None 2023-07-29T00:25:05.524 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:25:05.525 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PcmrIZYceaMtowhgpaunOH2zhh0= 2023-07-29T00:25:05.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PcmrIZYceaMtowhgpaunOH2zhh0=', 'Content-Length': '0'} 2023-07-29T00:25:05.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002c023801db50778-0064c45c61-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.529 DEBUG:boto:path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:25:05.529 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:25:05.530 DEBUG:boto:path=/hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:25:05.530 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:25:05.531 DEBUG:boto:Method: HEAD 2023-07-29T00:25:05.531 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:25:05.531 DEBUG:boto:Data: 2023-07-29T00:25:05.532 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.532 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:05.532 DEBUG:boto:Port: 8001 2023-07-29T00:25:05.533 DEBUG:boto:Params: {} 2023-07-29T00:25:05.533 DEBUG:boto:Token: None 2023-07-29T00:25:05.534 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:25:05.534 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J7UmWMBDVv+Y8haLnp+Erq1d3Sw= 2023-07-29T00:25:05.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J7UmWMBDVv+Y8haLnp+Erq1d3Sw=', 'Content-Length': '0'} 2023-07-29T00:25:05.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031b78cdc4c244388-0064c45c61-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.537 DEBUG:boto:path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:25:05.537 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:25:05.538 DEBUG:boto:path=/hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:25:05.539 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:25:05.539 DEBUG:boto:Method: HEAD 2023-07-29T00:25:05.540 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:25:05.541 DEBUG:boto:Data: 2023-07-29T00:25:05.542 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.542 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:05.543 DEBUG:boto:Port: 8000 2023-07-29T00:25:05.543 DEBUG:boto:Params: {} 2023-07-29T00:25:05.544 DEBUG:boto:Token: None 2023-07-29T00:25:05.545 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:25:05.545 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J7UmWMBDVv+Y8haLnp+Erq1d3Sw= 2023-07-29T00:25:05.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J7UmWMBDVv+Y8haLnp+Erq1d3Sw=', 'Content-Length': '0'} 2023-07-29T00:25:05.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000556d12c78224e263-0064c45c61-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-07-29T00:25:05.549 DEBUG:boto:path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:25:05.550 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:25:05.550 DEBUG:boto:path=/hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:25:05.550 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:25:05.551 DEBUG:boto:Method: GET 2023-07-29T00:25:05.551 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:25:05.552 DEBUG:boto:Data: 2023-07-29T00:25:05.552 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.553 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:05.553 DEBUG:boto:Port: 8001 2023-07-29T00:25:05.554 DEBUG:boto:Params: {} 2023-07-29T00:25:05.555 DEBUG:boto:Token: None 2023-07-29T00:25:05.555 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:25:05.556 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PcmrIZYceaMtowhgpaunOH2zhh0= 2023-07-29T00:25:05.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PcmrIZYceaMtowhgpaunOH2zhh0=', 'Content-Length': '0'} 2023-07-29T00:25:05.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c85c9ad44e029b49-0064c45c61-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.561 DEBUG:boto:path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:25:05.562 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:25:05.563 DEBUG:boto:path=/hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:25:05.564 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:25:05.564 DEBUG:boto:Method: GET 2023-07-29T00:25:05.565 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:25:05.566 DEBUG:boto:Data: 2023-07-29T00:25:05.567 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.567 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:05.568 DEBUG:boto:Port: 8000 2023-07-29T00:25:05.569 DEBUG:boto:Params: {} 2023-07-29T00:25:05.570 DEBUG:boto:Token: None 2023-07-29T00:25:05.570 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000006?versionId=null 2023-07-29T00:25:05.571 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PcmrIZYceaMtowhgpaunOH2zhh0= 2023-07-29T00:25:05.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PcmrIZYceaMtowhgpaunOH2zhh0=', 'Content-Length': '0'} 2023-07-29T00:25:05.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df04a03dd559812b-0064c45c61-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.574 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:05.575 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:05.575 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000006 2023-07-29T00:25:05.576 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000006 2023-07-29T00:25:05.576 DEBUG:boto:Method: GET 2023-07-29T00:25:05.577 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000006 2023-07-29T00:25:05.577 DEBUG:boto:Data: 2023-07-29T00:25:05.578 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.578 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:05.578 DEBUG:boto:Port: 8001 2023-07-29T00:25:05.579 DEBUG:boto:Params: {} 2023-07-29T00:25:05.579 DEBUG:boto:Token: None 2023-07-29T00:25:05.580 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/ 2023-07-29T00:25:05.581 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:p0mCby/Aoe7TFeGHqvTp57nCytE= 2023-07-29T00:25:05.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:p0mCby/Aoe7TFeGHqvTp57nCytE=', 'Content-Length': '0'} 2023-07-29T00:25:05.583 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a4e8e658d93d8f1-0064c45c61-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.584 DEBUG:boto:b'hiiqos-58obj000000000000000000061000falseobj000000000000000000062023-07-29T00:24:59.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:05.584 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:05.585 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:05.585 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000006 2023-07-29T00:25:05.586 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000006 2023-07-29T00:25:05.586 DEBUG:boto:Method: GET 2023-07-29T00:25:05.587 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000006 2023-07-29T00:25:05.587 DEBUG:boto:Data: 2023-07-29T00:25:05.588 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.589 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:05.590 DEBUG:boto:Port: 8000 2023-07-29T00:25:05.591 DEBUG:boto:Params: {} 2023-07-29T00:25:05.592 DEBUG:boto:Token: None 2023-07-29T00:25:05.593 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/ 2023-07-29T00:25:05.593 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:p0mCby/Aoe7TFeGHqvTp57nCytE= 2023-07-29T00:25:05.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:p0mCby/Aoe7TFeGHqvTp57nCytE=', 'Content-Length': '0'} 2023-07-29T00:25:05.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053e4198cf903e667-0064c45c61-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.598 DEBUG:boto:b'hiiqos-58obj000000000000000000061000falseobj000000000000000000062023-07-29T00:24:59.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:05.599 DEBUG:boto:path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:25:05.599 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:25:05.600 DEBUG:boto:Method: HEAD 2023-07-29T00:25:05.600 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000006 2023-07-29T00:25:05.600 DEBUG:boto:Data: 2023-07-29T00:25:05.601 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.601 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:05.601 DEBUG:boto:Port: 8001 2023-07-29T00:25:05.602 DEBUG:boto:Params: {} 2023-07-29T00:25:05.602 DEBUG:boto:Token: None 2023-07-29T00:25:05.603 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000006 2023-07-29T00:25:05.603 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bXbyTrAc4VAWGi1oKx6DIUZGQ1g= 2023-07-29T00:25:05.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bXbyTrAc4VAWGi1oKx6DIUZGQ1g=', 'Content-Length': '0'} 2023-07-29T00:25:05.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075bd6618f4000f3d-0064c45c61-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.607 DEBUG:boto:path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:25:05.607 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:25:05.607 DEBUG:boto:Method: HEAD 2023-07-29T00:25:05.608 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000006 2023-07-29T00:25:05.608 DEBUG:boto:Data: 2023-07-29T00:25:05.609 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.609 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:05.610 DEBUG:boto:Port: 8000 2023-07-29T00:25:05.610 DEBUG:boto:Params: {} 2023-07-29T00:25:05.610 DEBUG:boto:Token: None 2023-07-29T00:25:05.611 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000006 2023-07-29T00:25:05.611 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bXbyTrAc4VAWGi1oKx6DIUZGQ1g= 2023-07-29T00:25:05.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bXbyTrAc4VAWGi1oKx6DIUZGQ1g=', 'Content-Length': '0'} 2023-07-29T00:25:05.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c6cd254e716aa60-0064c45c61-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-07-29T00:25:05.615 DEBUG:boto:path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:25:05.615 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:25:05.615 DEBUG:boto:Method: GET 2023-07-29T00:25:05.616 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000006 2023-07-29T00:25:05.616 DEBUG:boto:Data: 2023-07-29T00:25:05.617 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.617 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:05.617 DEBUG:boto:Port: 8001 2023-07-29T00:25:05.618 DEBUG:boto:Params: {} 2023-07-29T00:25:05.618 DEBUG:boto:Token: None 2023-07-29T00:25:05.618 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000006 2023-07-29T00:25:05.619 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5up3Dx7nXiuf7SVPPWUP2OIrA0A= 2023-07-29T00:25:05.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5up3Dx7nXiuf7SVPPWUP2OIrA0A=', 'Content-Length': '0'} 2023-07-29T00:25:05.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cd6649c0b764eea-0064c45c61-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.622 DEBUG:boto:path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:25:05.623 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000006 2023-07-29T00:25:05.623 DEBUG:boto:Method: GET 2023-07-29T00:25:05.624 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000006 2023-07-29T00:25:05.625 DEBUG:boto:Data: 2023-07-29T00:25:05.626 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.626 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:05.627 DEBUG:boto:Port: 8000 2023-07-29T00:25:05.628 DEBUG:boto:Params: {} 2023-07-29T00:25:05.629 DEBUG:boto:Token: None 2023-07-29T00:25:05.630 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000006 2023-07-29T00:25:05.631 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5up3Dx7nXiuf7SVPPWUP2OIrA0A= 2023-07-29T00:25:05.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5up3Dx7nXiuf7SVPPWUP2OIrA0A=', 'Content-Length': '0'} 2023-07-29T00:25:05.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063f0af270d354507-0064c45c61-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.676 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:05.677 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:05.678 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000006 2023-07-29T00:25:05.679 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000006 2023-07-29T00:25:05.680 DEBUG:boto:Method: GET 2023-07-29T00:25:05.680 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000006 2023-07-29T00:25:05.681 DEBUG:boto:Data: 2023-07-29T00:25:05.682 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.682 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:05.683 DEBUG:boto:Port: 8001 2023-07-29T00:25:05.683 DEBUG:boto:Params: {} 2023-07-29T00:25:05.684 DEBUG:boto:Token: None 2023-07-29T00:25:05.684 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/ 2023-07-29T00:25:05.685 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:p0mCby/Aoe7TFeGHqvTp57nCytE= 2023-07-29T00:25:05.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:p0mCby/Aoe7TFeGHqvTp57nCytE=', 'Content-Length': '0'} 2023-07-29T00:25:05.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006cf32d55ee7fd4db-0064c45c61-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.690 DEBUG:boto:b'hiiqos-58obj000000000000000000061000falseobj000000000000000000062023-07-29T00:24:59.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:05.691 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:05.691 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:05.691 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000006 2023-07-29T00:25:05.692 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000006 2023-07-29T00:25:05.692 DEBUG:boto:Method: GET 2023-07-29T00:25:05.693 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000006 2023-07-29T00:25:05.693 DEBUG:boto:Data: 2023-07-29T00:25:05.693 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.694 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:05.694 DEBUG:boto:Port: 8000 2023-07-29T00:25:05.694 DEBUG:boto:Params: {} 2023-07-29T00:25:05.695 DEBUG:boto:Token: None 2023-07-29T00:25:05.695 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/ 2023-07-29T00:25:05.696 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:p0mCby/Aoe7TFeGHqvTp57nCytE= 2023-07-29T00:25:05.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:p0mCby/Aoe7TFeGHqvTp57nCytE=', 'Content-Length': '0'} 2023-07-29T00:25:05.701 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023a6eb24b8944dea-0064c45c61-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.701 DEBUG:boto:b'hiiqos-58obj000000000000000000061000falseobj000000000000000000062023-07-29T00:24:59.624Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:05.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2023-07-29T00:25:05.702 DEBUG:boto:path=/hiiqos-58/obj00000000000000000007 2023-07-29T00:25:05.702 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000007 2023-07-29T00:25:05.703 DEBUG:boto:path=/hiiqos-58/obj00000000000000000007?versionId=null 2023-07-29T00:25:05.703 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000007?versionId=null 2023-07-29T00:25:05.704 DEBUG:boto:Method: GET 2023-07-29T00:25:05.704 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000007?versionId=null 2023-07-29T00:25:05.704 DEBUG:boto:Data: 2023-07-29T00:25:05.705 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.705 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:05.705 DEBUG:boto:Port: 8001 2023-07-29T00:25:05.706 DEBUG:boto:Params: {} 2023-07-29T00:25:05.706 DEBUG:boto:Token: None 2023-07-29T00:25:05.707 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000007?versionId=null 2023-07-29T00:25:05.707 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:iFJwiwbYhWDvcAfieLm/J9FeESk= 2023-07-29T00:25:05.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:iFJwiwbYhWDvcAfieLm/J9FeESk=', 'Content-Length': '0'} 2023-07-29T00:25:05.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000357f0f11fb714b5f-0064c45c61-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.752 DEBUG:boto:path=/hiiqos-58/obj00000000000000000007 2023-07-29T00:25:05.753 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000007 2023-07-29T00:25:05.753 DEBUG:boto:path=/hiiqos-58/obj00000000000000000007?versionId=null 2023-07-29T00:25:05.754 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000007?versionId=null 2023-07-29T00:25:05.754 DEBUG:boto:Method: GET 2023-07-29T00:25:05.754 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000007?versionId=null 2023-07-29T00:25:05.755 DEBUG:boto:Data: 2023-07-29T00:25:05.755 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.755 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:05.756 DEBUG:boto:Port: 8000 2023-07-29T00:25:05.756 DEBUG:boto:Params: {} 2023-07-29T00:25:05.757 DEBUG:boto:Token: None 2023-07-29T00:25:05.757 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000007?versionId=null 2023-07-29T00:25:05.758 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:iFJwiwbYhWDvcAfieLm/J9FeESk= 2023-07-29T00:25:05.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:iFJwiwbYhWDvcAfieLm/J9FeESk=', 'Content-Length': '0'} 2023-07-29T00:25:05.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000274b15d6c7e200c-0064c45c61-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.761 DEBUG:boto:path=/hiiqos-58/obj00000000000000000007 2023-07-29T00:25:05.761 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000007 2023-07-29T00:25:05.762 DEBUG:boto:path=/hiiqos-58/obj00000000000000000007?versionId=null 2023-07-29T00:25:05.762 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000007?versionId=null 2023-07-29T00:25:05.762 DEBUG:boto:Method: HEAD 2023-07-29T00:25:05.763 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000007?versionId=null 2023-07-29T00:25:05.763 DEBUG:boto:Data: 2023-07-29T00:25:05.763 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.764 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:05.764 DEBUG:boto:Port: 8001 2023-07-29T00:25:05.764 DEBUG:boto:Params: {} 2023-07-29T00:25:05.765 DEBUG:boto:Token: None 2023-07-29T00:25:05.765 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000007?versionId=null 2023-07-29T00:25:05.766 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:biLCZw9IkDf8Un651HHdrKOSd3I= 2023-07-29T00:25:05.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:biLCZw9IkDf8Un651HHdrKOSd3I=', 'Content-Length': '0'} 2023-07-29T00:25:05.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006877e1f88b4b2ebb-0064c45c61-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.770 DEBUG:boto:path=/hiiqos-58/obj00000000000000000007 2023-07-29T00:25:05.771 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000007 2023-07-29T00:25:05.771 DEBUG:boto:path=/hiiqos-58/obj00000000000000000007?versionId=null 2023-07-29T00:25:05.771 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000007?versionId=null 2023-07-29T00:25:05.772 DEBUG:boto:Method: HEAD 2023-07-29T00:25:05.772 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000007?versionId=null 2023-07-29T00:25:05.773 DEBUG:boto:Data: 2023-07-29T00:25:05.773 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.773 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:05.774 DEBUG:boto:Port: 8000 2023-07-29T00:25:05.774 DEBUG:boto:Params: {} 2023-07-29T00:25:05.775 DEBUG:boto:Token: None 2023-07-29T00:25:05.775 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000007?versionId=null 2023-07-29T00:25:05.776 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:biLCZw9IkDf8Un651HHdrKOSd3I= 2023-07-29T00:25:05.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:biLCZw9IkDf8Un651HHdrKOSd3I=', 'Content-Length': '0'} 2023-07-29T00:25:05.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000847d1bbcd15dcd6d-0064c45c61-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2023-07-29T00:25:05.779 DEBUG:boto:path=/hiiqos-58/obj00000000000000000007 2023-07-29T00:25:05.780 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000007 2023-07-29T00:25:05.780 DEBUG:boto:path=/hiiqos-58/obj00000000000000000007?versionId=null 2023-07-29T00:25:05.781 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000007?versionId=null 2023-07-29T00:25:05.781 DEBUG:boto:Method: GET 2023-07-29T00:25:05.782 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000007?versionId=null 2023-07-29T00:25:05.782 DEBUG:boto:Data: 2023-07-29T00:25:05.782 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.783 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:05.783 DEBUG:boto:Port: 8001 2023-07-29T00:25:05.784 DEBUG:boto:Params: {} 2023-07-29T00:25:05.784 DEBUG:boto:Token: None 2023-07-29T00:25:05.785 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000007?versionId=null 2023-07-29T00:25:05.786 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:iFJwiwbYhWDvcAfieLm/J9FeESk= 2023-07-29T00:25:05.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:iFJwiwbYhWDvcAfieLm/J9FeESk=', 'Content-Length': '0'} 2023-07-29T00:25:05.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb7fd89d9ed04530-0064c45c61-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.789 DEBUG:boto:path=/hiiqos-58/obj00000000000000000007 2023-07-29T00:25:05.789 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000007 2023-07-29T00:25:05.790 DEBUG:boto:path=/hiiqos-58/obj00000000000000000007?versionId=null 2023-07-29T00:25:05.790 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000007?versionId=null 2023-07-29T00:25:05.791 DEBUG:boto:Method: GET 2023-07-29T00:25:05.791 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000007?versionId=null 2023-07-29T00:25:05.791 DEBUG:boto:Data: 2023-07-29T00:25:05.792 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.792 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:05.792 DEBUG:boto:Port: 8000 2023-07-29T00:25:05.793 DEBUG:boto:Params: {} 2023-07-29T00:25:05.793 DEBUG:boto:Token: None 2023-07-29T00:25:05.794 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000007?versionId=null 2023-07-29T00:25:05.794 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:iFJwiwbYhWDvcAfieLm/J9FeESk= 2023-07-29T00:25:05.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:iFJwiwbYhWDvcAfieLm/J9FeESk=', 'Content-Length': '0'} 2023-07-29T00:25:05.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050555757e45ea06f-0064c45c61-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.797 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:05.798 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:05.798 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000007 2023-07-29T00:25:05.798 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000007 2023-07-29T00:25:05.799 DEBUG:boto:Method: GET 2023-07-29T00:25:05.799 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000007 2023-07-29T00:25:05.800 DEBUG:boto:Data: 2023-07-29T00:25:05.800 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.800 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:05.801 DEBUG:boto:Port: 8001 2023-07-29T00:25:05.801 DEBUG:boto:Params: {} 2023-07-29T00:25:05.802 DEBUG:boto:Token: None 2023-07-29T00:25:05.802 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/ 2023-07-29T00:25:05.802 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:p0mCby/Aoe7TFeGHqvTp57nCytE= 2023-07-29T00:25:05.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:p0mCby/Aoe7TFeGHqvTp57nCytE=', 'Content-Length': '0'} 2023-07-29T00:25:05.805 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dafe435169f59b2d-0064c45c61-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.806 DEBUG:boto:b'hiiqos-58obj000000000000000000071000falseobj000000000000000000072023-07-29T00:24:59.887Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:05.807 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:05.807 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:05.808 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000007 2023-07-29T00:25:05.808 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000007 2023-07-29T00:25:05.808 DEBUG:boto:Method: GET 2023-07-29T00:25:05.809 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000007 2023-07-29T00:25:05.809 DEBUG:boto:Data: 2023-07-29T00:25:05.810 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.810 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:05.810 DEBUG:boto:Port: 8000 2023-07-29T00:25:05.811 DEBUG:boto:Params: {} 2023-07-29T00:25:05.811 DEBUG:boto:Token: None 2023-07-29T00:25:05.812 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/ 2023-07-29T00:25:05.812 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:p0mCby/Aoe7TFeGHqvTp57nCytE= 2023-07-29T00:25:05.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:p0mCby/Aoe7TFeGHqvTp57nCytE=', 'Content-Length': '0'} 2023-07-29T00:25:05.817 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e917f0dea13971c-0064c45c61-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.817 DEBUG:boto:b'hiiqos-58obj000000000000000000071000falseobj000000000000000000072023-07-29T00:24:59.887Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:05.818 DEBUG:boto:path=/hiiqos-58/obj00000000000000000007 2023-07-29T00:25:05.819 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000007 2023-07-29T00:25:05.819 DEBUG:boto:Method: HEAD 2023-07-29T00:25:05.819 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000007 2023-07-29T00:25:05.820 DEBUG:boto:Data: 2023-07-29T00:25:05.820 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.820 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:05.821 DEBUG:boto:Port: 8001 2023-07-29T00:25:05.821 DEBUG:boto:Params: {} 2023-07-29T00:25:05.822 DEBUG:boto:Token: None 2023-07-29T00:25:05.822 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000007 2023-07-29T00:25:05.823 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LHfmTxpXBYMxlxjZ90W35cMsdNI= 2023-07-29T00:25:05.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LHfmTxpXBYMxlxjZ90W35cMsdNI=', 'Content-Length': '0'} 2023-07-29T00:25:05.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ea4b264e2e70a57-0064c45c61-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.826 DEBUG:boto:path=/hiiqos-58/obj00000000000000000007 2023-07-29T00:25:05.826 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000007 2023-07-29T00:25:05.826 DEBUG:boto:Method: HEAD 2023-07-29T00:25:05.827 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000007 2023-07-29T00:25:05.827 DEBUG:boto:Data: 2023-07-29T00:25:05.827 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.828 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:05.828 DEBUG:boto:Port: 8000 2023-07-29T00:25:05.829 DEBUG:boto:Params: {} 2023-07-29T00:25:05.829 DEBUG:boto:Token: None 2023-07-29T00:25:05.829 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000007 2023-07-29T00:25:05.830 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LHfmTxpXBYMxlxjZ90W35cMsdNI= 2023-07-29T00:25:05.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LHfmTxpXBYMxlxjZ90W35cMsdNI=', 'Content-Length': '0'} 2023-07-29T00:25:05.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021d13c7cd86fd778-0064c45c61-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2023-07-29T00:25:05.833 DEBUG:boto:path=/hiiqos-58/obj00000000000000000007 2023-07-29T00:25:05.834 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000007 2023-07-29T00:25:05.834 DEBUG:boto:Method: GET 2023-07-29T00:25:05.834 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000007 2023-07-29T00:25:05.835 DEBUG:boto:Data: 2023-07-29T00:25:05.835 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.836 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:05.836 DEBUG:boto:Port: 8001 2023-07-29T00:25:05.836 DEBUG:boto:Params: {} 2023-07-29T00:25:05.837 DEBUG:boto:Token: None 2023-07-29T00:25:05.837 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000007 2023-07-29T00:25:05.838 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qlEiK3LxMyPWdjGAvRYOvFN0kPw= 2023-07-29T00:25:05.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qlEiK3LxMyPWdjGAvRYOvFN0kPw=', 'Content-Length': '0'} 2023-07-29T00:25:05.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003233139ca5cd7786-0064c45c61-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.884 DEBUG:boto:path=/hiiqos-58/obj00000000000000000007 2023-07-29T00:25:05.885 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000007 2023-07-29T00:25:05.886 DEBUG:boto:Method: GET 2023-07-29T00:25:05.887 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000007 2023-07-29T00:25:05.888 DEBUG:boto:Data: 2023-07-29T00:25:05.889 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.889 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:05.889 DEBUG:boto:Port: 8000 2023-07-29T00:25:05.890 DEBUG:boto:Params: {} 2023-07-29T00:25:05.890 DEBUG:boto:Token: None 2023-07-29T00:25:05.891 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000007 2023-07-29T00:25:05.891 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qlEiK3LxMyPWdjGAvRYOvFN0kPw= 2023-07-29T00:25:05.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qlEiK3LxMyPWdjGAvRYOvFN0kPw=', 'Content-Length': '0'} 2023-07-29T00:25:05.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4952307040b7b71-0064c45c61-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.895 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:05.895 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:05.895 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000007 2023-07-29T00:25:05.896 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000007 2023-07-29T00:25:05.896 DEBUG:boto:Method: GET 2023-07-29T00:25:05.897 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000007 2023-07-29T00:25:05.897 DEBUG:boto:Data: 2023-07-29T00:25:05.898 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.898 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:05.898 DEBUG:boto:Port: 8001 2023-07-29T00:25:05.899 DEBUG:boto:Params: {} 2023-07-29T00:25:05.899 DEBUG:boto:Token: None 2023-07-29T00:25:05.900 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/ 2023-07-29T00:25:05.901 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:p0mCby/Aoe7TFeGHqvTp57nCytE= 2023-07-29T00:25:05.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:p0mCby/Aoe7TFeGHqvTp57nCytE=', 'Content-Length': '0'} 2023-07-29T00:25:05.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068d75b85821778df-0064c45c61-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.907 DEBUG:boto:b'hiiqos-58obj000000000000000000071000falseobj000000000000000000072023-07-29T00:24:59.887Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:05.908 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:05.909 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:05.909 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000007 2023-07-29T00:25:05.910 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000007 2023-07-29T00:25:05.911 DEBUG:boto:Method: GET 2023-07-29T00:25:05.911 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000007 2023-07-29T00:25:05.912 DEBUG:boto:Data: 2023-07-29T00:25:05.913 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.913 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:05.913 DEBUG:boto:Port: 8000 2023-07-29T00:25:05.914 DEBUG:boto:Params: {} 2023-07-29T00:25:05.914 DEBUG:boto:Token: None 2023-07-29T00:25:05.915 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/ 2023-07-29T00:25:05.915 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:p0mCby/Aoe7TFeGHqvTp57nCytE= 2023-07-29T00:25:05.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:p0mCby/Aoe7TFeGHqvTp57nCytE=', 'Content-Length': '0'} 2023-07-29T00:25:05.920 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000506d2b0f1c98786d-0064c45c61-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.920 DEBUG:boto:b'hiiqos-58obj000000000000000000071000falseobj000000000000000000072023-07-29T00:24:59.887Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:05.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2023-07-29T00:25:05.922 DEBUG:boto:path=/hiiqos-58/obj00000000000000000008 2023-07-29T00:25:05.922 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000008 2023-07-29T00:25:05.923 DEBUG:boto:path=/hiiqos-58/obj00000000000000000008?versionId=null 2023-07-29T00:25:05.923 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000008?versionId=null 2023-07-29T00:25:05.923 DEBUG:boto:Method: GET 2023-07-29T00:25:05.924 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000008?versionId=null 2023-07-29T00:25:05.924 DEBUG:boto:Data: 2023-07-29T00:25:05.925 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.925 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:05.925 DEBUG:boto:Port: 8001 2023-07-29T00:25:05.926 DEBUG:boto:Params: {} 2023-07-29T00:25:05.926 DEBUG:boto:Token: None 2023-07-29T00:25:05.928 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000008?versionId=null 2023-07-29T00:25:05.928 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WjzNgHL3E2DL86x8Ltuijw2bJmc= 2023-07-29T00:25:05.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WjzNgHL3E2DL86x8Ltuijw2bJmc=', 'Content-Length': '0'} 2023-07-29T00:25:05.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000501e44ec260608fe-0064c45c61-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.932 DEBUG:boto:path=/hiiqos-58/obj00000000000000000008 2023-07-29T00:25:05.932 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000008 2023-07-29T00:25:05.932 DEBUG:boto:path=/hiiqos-58/obj00000000000000000008?versionId=null 2023-07-29T00:25:05.933 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000008?versionId=null 2023-07-29T00:25:05.933 DEBUG:boto:Method: GET 2023-07-29T00:25:05.934 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000008?versionId=null 2023-07-29T00:25:05.934 DEBUG:boto:Data: 2023-07-29T00:25:05.934 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.935 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:05.935 DEBUG:boto:Port: 8000 2023-07-29T00:25:05.935 DEBUG:boto:Params: {} 2023-07-29T00:25:05.936 DEBUG:boto:Token: None 2023-07-29T00:25:05.936 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000008?versionId=null 2023-07-29T00:25:05.937 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WjzNgHL3E2DL86x8Ltuijw2bJmc= 2023-07-29T00:25:05.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WjzNgHL3E2DL86x8Ltuijw2bJmc=', 'Content-Length': '0'} 2023-07-29T00:25:05.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025d199b2303d2d8a-0064c45c61-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.940 DEBUG:boto:path=/hiiqos-58/obj00000000000000000008 2023-07-29T00:25:05.940 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000008 2023-07-29T00:25:05.940 DEBUG:boto:path=/hiiqos-58/obj00000000000000000008?versionId=null 2023-07-29T00:25:05.941 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000008?versionId=null 2023-07-29T00:25:05.941 DEBUG:boto:Method: HEAD 2023-07-29T00:25:05.941 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000008?versionId=null 2023-07-29T00:25:05.942 DEBUG:boto:Data: 2023-07-29T00:25:05.942 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.943 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:05.943 DEBUG:boto:Port: 8001 2023-07-29T00:25:05.943 DEBUG:boto:Params: {} 2023-07-29T00:25:05.944 DEBUG:boto:Token: None 2023-07-29T00:25:05.944 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000008?versionId=null 2023-07-29T00:25:05.945 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Ujus3E1bWYAjXQOrBstFhLGN4iM= 2023-07-29T00:25:05.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Ujus3E1bWYAjXQOrBstFhLGN4iM=', 'Content-Length': '0'} 2023-07-29T00:25:05.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cc260402771e7d4-0064c45c61-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.948 DEBUG:boto:path=/hiiqos-58/obj00000000000000000008 2023-07-29T00:25:05.948 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000008 2023-07-29T00:25:05.948 DEBUG:boto:path=/hiiqos-58/obj00000000000000000008?versionId=null 2023-07-29T00:25:05.949 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000008?versionId=null 2023-07-29T00:25:05.949 DEBUG:boto:Method: HEAD 2023-07-29T00:25:05.950 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000008?versionId=null 2023-07-29T00:25:05.950 DEBUG:boto:Data: 2023-07-29T00:25:05.950 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.951 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:05.951 DEBUG:boto:Port: 8000 2023-07-29T00:25:05.951 DEBUG:boto:Params: {} 2023-07-29T00:25:05.952 DEBUG:boto:Token: None 2023-07-29T00:25:05.952 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000008?versionId=null 2023-07-29T00:25:05.953 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Ujus3E1bWYAjXQOrBstFhLGN4iM= 2023-07-29T00:25:05.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Ujus3E1bWYAjXQOrBstFhLGN4iM=', 'Content-Length': '0'} 2023-07-29T00:25:05.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071db189b86ee6c7a-0064c45c61-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2023-07-29T00:25:05.956 DEBUG:boto:path=/hiiqos-58/obj00000000000000000008 2023-07-29T00:25:05.956 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000008 2023-07-29T00:25:05.957 DEBUG:boto:path=/hiiqos-58/obj00000000000000000008?versionId=null 2023-07-29T00:25:05.957 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000008?versionId=null 2023-07-29T00:25:05.958 DEBUG:boto:Method: GET 2023-07-29T00:25:05.958 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000008?versionId=null 2023-07-29T00:25:05.959 DEBUG:boto:Data: 2023-07-29T00:25:05.959 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.960 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:05.960 DEBUG:boto:Port: 8001 2023-07-29T00:25:05.961 DEBUG:boto:Params: {} 2023-07-29T00:25:05.962 DEBUG:boto:Token: None 2023-07-29T00:25:05.963 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000008?versionId=null 2023-07-29T00:25:05.964 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WjzNgHL3E2DL86x8Ltuijw2bJmc= 2023-07-29T00:25:05.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WjzNgHL3E2DL86x8Ltuijw2bJmc=', 'Content-Length': '0'} 2023-07-29T00:25:05.968 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8e2e38a0c082eb9-0064c45c61-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.969 DEBUG:boto:path=/hiiqos-58/obj00000000000000000008 2023-07-29T00:25:05.969 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000008 2023-07-29T00:25:05.970 DEBUG:boto:path=/hiiqos-58/obj00000000000000000008?versionId=null 2023-07-29T00:25:05.971 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000008?versionId=null 2023-07-29T00:25:05.971 DEBUG:boto:Method: GET 2023-07-29T00:25:05.972 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000008?versionId=null 2023-07-29T00:25:05.972 DEBUG:boto:Data: 2023-07-29T00:25:05.972 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.973 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:05.973 DEBUG:boto:Port: 8000 2023-07-29T00:25:05.973 DEBUG:boto:Params: {} 2023-07-29T00:25:05.974 DEBUG:boto:Token: None 2023-07-29T00:25:05.974 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/obj00000000000000000008?versionId=null 2023-07-29T00:25:05.975 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WjzNgHL3E2DL86x8Ltuijw2bJmc= 2023-07-29T00:25:05.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WjzNgHL3E2DL86x8Ltuijw2bJmc=', 'Content-Length': '0'} 2023-07-29T00:25:05.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005514c6ebb9cda216-0064c45c61-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.978 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:05.978 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:05.979 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000008 2023-07-29T00:25:05.980 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000008 2023-07-29T00:25:05.981 DEBUG:boto:Method: GET 2023-07-29T00:25:05.982 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000008 2023-07-29T00:25:05.983 DEBUG:boto:Data: 2023-07-29T00:25:05.984 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.985 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:05.986 DEBUG:boto:Port: 8001 2023-07-29T00:25:05.986 DEBUG:boto:Params: {} 2023-07-29T00:25:05.987 DEBUG:boto:Token: None 2023-07-29T00:25:05.988 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:05 GMT /hiiqos-58/ 2023-07-29T00:25:05.989 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:p0mCby/Aoe7TFeGHqvTp57nCytE= 2023-07-29T00:25:05.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:p0mCby/Aoe7TFeGHqvTp57nCytE=', 'Content-Length': '0'} 2023-07-29T00:25:05.993 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021a58d0971070568-0064c45c61-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:05.993 DEBUG:boto:b'hiiqos-58obj000000000000000000081000falseobj000000000000000000082023-07-29T00:25:00.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:05.994 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:05.994 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:05.995 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000008 2023-07-29T00:25:05.995 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000008 2023-07-29T00:25:05.995 DEBUG:boto:Method: GET 2023-07-29T00:25:05.996 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000008 2023-07-29T00:25:05.997 DEBUG:boto:Data: 2023-07-29T00:25:05.998 DEBUG:boto:Headers: {} 2023-07-29T00:25:05.999 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:06.000 DEBUG:boto:Port: 8000 2023-07-29T00:25:06.001 DEBUG:boto:Params: {} 2023-07-29T00:25:06.002 DEBUG:boto:Token: None 2023-07-29T00:25:06.002 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/ 2023-07-29T00:25:06.003 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vTz8RfPkYzQQtKEDzipz/ZiOrvc= 2023-07-29T00:25:06.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vTz8RfPkYzQQtKEDzipz/ZiOrvc=', 'Content-Length': '0'} 2023-07-29T00:25:06.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008248c390e51fd7a-0064c45c61-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.008 DEBUG:boto:b'hiiqos-58obj000000000000000000081000falseobj000000000000000000082023-07-29T00:25:00.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:06.008 DEBUG:boto:path=/hiiqos-58/obj00000000000000000008 2023-07-29T00:25:06.009 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000008 2023-07-29T00:25:06.009 DEBUG:boto:Method: HEAD 2023-07-29T00:25:06.009 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000008 2023-07-29T00:25:06.010 DEBUG:boto:Data: 2023-07-29T00:25:06.010 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.010 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:06.011 DEBUG:boto:Port: 8001 2023-07-29T00:25:06.011 DEBUG:boto:Params: {} 2023-07-29T00:25:06.011 DEBUG:boto:Token: None 2023-07-29T00:25:06.012 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000008 2023-07-29T00:25:06.012 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PlQqaCOdxobFbmLKC8SyRP8bpKE= 2023-07-29T00:25:06.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PlQqaCOdxobFbmLKC8SyRP8bpKE=', 'Content-Length': '0'} 2023-07-29T00:25:06.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000757114dad05f214-0064c45c61-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.015 DEBUG:boto:path=/hiiqos-58/obj00000000000000000008 2023-07-29T00:25:06.015 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000008 2023-07-29T00:25:06.016 DEBUG:boto:Method: HEAD 2023-07-29T00:25:06.016 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000008 2023-07-29T00:25:06.016 DEBUG:boto:Data: 2023-07-29T00:25:06.017 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.017 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:06.017 DEBUG:boto:Port: 8000 2023-07-29T00:25:06.018 DEBUG:boto:Params: {} 2023-07-29T00:25:06.018 DEBUG:boto:Token: None 2023-07-29T00:25:06.018 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000008 2023-07-29T00:25:06.019 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PlQqaCOdxobFbmLKC8SyRP8bpKE= 2023-07-29T00:25:06.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PlQqaCOdxobFbmLKC8SyRP8bpKE=', 'Content-Length': '0'} 2023-07-29T00:25:06.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b87a1f0c51332987-0064c45c61-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2023-07-29T00:25:06.022 DEBUG:boto:path=/hiiqos-58/obj00000000000000000008 2023-07-29T00:25:06.022 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000008 2023-07-29T00:25:06.023 DEBUG:boto:Method: GET 2023-07-29T00:25:06.023 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000008 2023-07-29T00:25:06.023 DEBUG:boto:Data: 2023-07-29T00:25:06.024 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.024 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:06.025 DEBUG:boto:Port: 8001 2023-07-29T00:25:06.025 DEBUG:boto:Params: {} 2023-07-29T00:25:06.025 DEBUG:boto:Token: None 2023-07-29T00:25:06.026 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000008 2023-07-29T00:25:06.026 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8fQRFwLGmM/f5J5V4G3cM5I/fgI= 2023-07-29T00:25:06.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8fQRFwLGmM/f5J5V4G3cM5I/fgI=', 'Content-Length': '0'} 2023-07-29T00:25:06.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031d314cf6aaae8e2-0064c45c61-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.076 DEBUG:boto:path=/hiiqos-58/obj00000000000000000008 2023-07-29T00:25:06.077 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000008 2023-07-29T00:25:06.077 DEBUG:boto:Method: GET 2023-07-29T00:25:06.078 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000008 2023-07-29T00:25:06.078 DEBUG:boto:Data: 2023-07-29T00:25:06.078 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.079 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:06.079 DEBUG:boto:Port: 8000 2023-07-29T00:25:06.079 DEBUG:boto:Params: {} 2023-07-29T00:25:06.080 DEBUG:boto:Token: None 2023-07-29T00:25:06.080 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000008 2023-07-29T00:25:06.081 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8fQRFwLGmM/f5J5V4G3cM5I/fgI= 2023-07-29T00:25:06.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8fQRFwLGmM/f5J5V4G3cM5I/fgI=', 'Content-Length': '0'} 2023-07-29T00:25:06.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f19a2c8fbb740d5-0064c45c62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.084 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:06.084 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:06.085 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000008 2023-07-29T00:25:06.085 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000008 2023-07-29T00:25:06.085 DEBUG:boto:Method: GET 2023-07-29T00:25:06.086 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000008 2023-07-29T00:25:06.086 DEBUG:boto:Data: 2023-07-29T00:25:06.086 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.087 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:06.087 DEBUG:boto:Port: 8001 2023-07-29T00:25:06.087 DEBUG:boto:Params: {} 2023-07-29T00:25:06.088 DEBUG:boto:Token: None 2023-07-29T00:25:06.088 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/ 2023-07-29T00:25:06.089 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vTz8RfPkYzQQtKEDzipz/ZiOrvc= 2023-07-29T00:25:06.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vTz8RfPkYzQQtKEDzipz/ZiOrvc=', 'Content-Length': '0'} 2023-07-29T00:25:06.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078bccd4c03a3000e-0064c45c62-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.092 DEBUG:boto:b'hiiqos-58obj000000000000000000081000falseobj000000000000000000082023-07-29T00:25:00.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:06.093 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:06.093 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:06.094 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000008 2023-07-29T00:25:06.094 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000008 2023-07-29T00:25:06.094 DEBUG:boto:Method: GET 2023-07-29T00:25:06.095 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000008 2023-07-29T00:25:06.095 DEBUG:boto:Data: 2023-07-29T00:25:06.095 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.096 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:06.096 DEBUG:boto:Port: 8000 2023-07-29T00:25:06.096 DEBUG:boto:Params: {} 2023-07-29T00:25:06.097 DEBUG:boto:Token: None 2023-07-29T00:25:06.097 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/ 2023-07-29T00:25:06.098 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vTz8RfPkYzQQtKEDzipz/ZiOrvc= 2023-07-29T00:25:06.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vTz8RfPkYzQQtKEDzipz/ZiOrvc=', 'Content-Length': '0'} 2023-07-29T00:25:06.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0a654298d284278-0064c45c62-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.103 DEBUG:boto:b'hiiqos-58obj000000000000000000081000falseobj000000000000000000082023-07-29T00:25:00.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:06.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2023-07-29T00:25:06.104 DEBUG:boto:path=/hiiqos-58/obj00000000000000000009 2023-07-29T00:25:06.104 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000009 2023-07-29T00:25:06.105 DEBUG:boto:path=/hiiqos-58/obj00000000000000000009?versionId=null 2023-07-29T00:25:06.105 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000009?versionId=null 2023-07-29T00:25:06.105 DEBUG:boto:Method: GET 2023-07-29T00:25:06.106 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000009?versionId=null 2023-07-29T00:25:06.106 DEBUG:boto:Data: 2023-07-29T00:25:06.106 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.107 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:06.107 DEBUG:boto:Port: 8001 2023-07-29T00:25:06.107 DEBUG:boto:Params: {} 2023-07-29T00:25:06.108 DEBUG:boto:Token: None 2023-07-29T00:25:06.108 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000009?versionId=null 2023-07-29T00:25:06.109 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aCz6f0h3D8WQ/N6puqxcjpcsT1M= 2023-07-29T00:25:06.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aCz6f0h3D8WQ/N6puqxcjpcsT1M=', 'Content-Length': '0'} 2023-07-29T00:25:06.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ad6c375081ab22e-0064c45c62-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.112 DEBUG:boto:path=/hiiqos-58/obj00000000000000000009 2023-07-29T00:25:06.112 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000009 2023-07-29T00:25:06.113 DEBUG:boto:path=/hiiqos-58/obj00000000000000000009?versionId=null 2023-07-29T00:25:06.113 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000009?versionId=null 2023-07-29T00:25:06.114 DEBUG:boto:Method: GET 2023-07-29T00:25:06.114 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000009?versionId=null 2023-07-29T00:25:06.114 DEBUG:boto:Data: 2023-07-29T00:25:06.115 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.115 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:06.115 DEBUG:boto:Port: 8000 2023-07-29T00:25:06.116 DEBUG:boto:Params: {} 2023-07-29T00:25:06.116 DEBUG:boto:Token: None 2023-07-29T00:25:06.116 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000009?versionId=null 2023-07-29T00:25:06.117 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aCz6f0h3D8WQ/N6puqxcjpcsT1M= 2023-07-29T00:25:06.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aCz6f0h3D8WQ/N6puqxcjpcsT1M=', 'Content-Length': '0'} 2023-07-29T00:25:06.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1a50ba90eb04161-0064c45c62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.164 DEBUG:boto:path=/hiiqos-58/obj00000000000000000009 2023-07-29T00:25:06.165 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000009 2023-07-29T00:25:06.165 DEBUG:boto:path=/hiiqos-58/obj00000000000000000009?versionId=null 2023-07-29T00:25:06.165 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000009?versionId=null 2023-07-29T00:25:06.166 DEBUG:boto:Method: HEAD 2023-07-29T00:25:06.166 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000009?versionId=null 2023-07-29T00:25:06.166 DEBUG:boto:Data: 2023-07-29T00:25:06.167 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.167 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:06.167 DEBUG:boto:Port: 8001 2023-07-29T00:25:06.168 DEBUG:boto:Params: {} 2023-07-29T00:25:06.168 DEBUG:boto:Token: None 2023-07-29T00:25:06.169 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000009?versionId=null 2023-07-29T00:25:06.169 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IV515DuEtC6pe/rHOK9Gi4Gbmlc= 2023-07-29T00:25:06.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IV515DuEtC6pe/rHOK9Gi4Gbmlc=', 'Content-Length': '0'} 2023-07-29T00:25:06.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000444a7167d58e0b0f-0064c45c62-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.172 DEBUG:boto:path=/hiiqos-58/obj00000000000000000009 2023-07-29T00:25:06.172 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000009 2023-07-29T00:25:06.172 DEBUG:boto:path=/hiiqos-58/obj00000000000000000009?versionId=null 2023-07-29T00:25:06.173 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000009?versionId=null 2023-07-29T00:25:06.173 DEBUG:boto:Method: HEAD 2023-07-29T00:25:06.173 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000009?versionId=null 2023-07-29T00:25:06.174 DEBUG:boto:Data: 2023-07-29T00:25:06.174 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.174 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:06.175 DEBUG:boto:Port: 8000 2023-07-29T00:25:06.175 DEBUG:boto:Params: {} 2023-07-29T00:25:06.175 DEBUG:boto:Token: None 2023-07-29T00:25:06.176 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000009?versionId=null 2023-07-29T00:25:06.176 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IV515DuEtC6pe/rHOK9Gi4Gbmlc= 2023-07-29T00:25:06.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IV515DuEtC6pe/rHOK9Gi4Gbmlc=', 'Content-Length': '0'} 2023-07-29T00:25:06.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ec822d68fafe170-0064c45c62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2023-07-29T00:25:06.180 DEBUG:boto:path=/hiiqos-58/obj00000000000000000009 2023-07-29T00:25:06.180 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000009 2023-07-29T00:25:06.180 DEBUG:boto:path=/hiiqos-58/obj00000000000000000009?versionId=null 2023-07-29T00:25:06.181 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000009?versionId=null 2023-07-29T00:25:06.181 DEBUG:boto:Method: GET 2023-07-29T00:25:06.182 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000009?versionId=null 2023-07-29T00:25:06.182 DEBUG:boto:Data: 2023-07-29T00:25:06.182 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.183 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:06.183 DEBUG:boto:Port: 8001 2023-07-29T00:25:06.183 DEBUG:boto:Params: {} 2023-07-29T00:25:06.184 DEBUG:boto:Token: None 2023-07-29T00:25:06.184 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000009?versionId=null 2023-07-29T00:25:06.185 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aCz6f0h3D8WQ/N6puqxcjpcsT1M= 2023-07-29T00:25:06.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aCz6f0h3D8WQ/N6puqxcjpcsT1M=', 'Content-Length': '0'} 2023-07-29T00:25:06.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000851f8ad91f2d81d1-0064c45c62-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.188 DEBUG:boto:path=/hiiqos-58/obj00000000000000000009 2023-07-29T00:25:06.188 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000009 2023-07-29T00:25:06.188 DEBUG:boto:path=/hiiqos-58/obj00000000000000000009?versionId=null 2023-07-29T00:25:06.189 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000009?versionId=null 2023-07-29T00:25:06.189 DEBUG:boto:Method: GET 2023-07-29T00:25:06.190 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000009?versionId=null 2023-07-29T00:25:06.190 DEBUG:boto:Data: 2023-07-29T00:25:06.190 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.191 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:06.191 DEBUG:boto:Port: 8000 2023-07-29T00:25:06.192 DEBUG:boto:Params: {} 2023-07-29T00:25:06.192 DEBUG:boto:Token: None 2023-07-29T00:25:06.192 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000009?versionId=null 2023-07-29T00:25:06.193 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aCz6f0h3D8WQ/N6puqxcjpcsT1M= 2023-07-29T00:25:06.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aCz6f0h3D8WQ/N6puqxcjpcsT1M=', 'Content-Length': '0'} 2023-07-29T00:25:06.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ace49049efea51ec-0064c45c62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.196 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:06.196 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:06.197 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000009 2023-07-29T00:25:06.197 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000009 2023-07-29T00:25:06.197 DEBUG:boto:Method: GET 2023-07-29T00:25:06.198 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000009 2023-07-29T00:25:06.198 DEBUG:boto:Data: 2023-07-29T00:25:06.198 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.199 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:06.199 DEBUG:boto:Port: 8001 2023-07-29T00:25:06.199 DEBUG:boto:Params: {} 2023-07-29T00:25:06.200 DEBUG:boto:Token: None 2023-07-29T00:25:06.200 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/ 2023-07-29T00:25:06.201 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vTz8RfPkYzQQtKEDzipz/ZiOrvc= 2023-07-29T00:25:06.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vTz8RfPkYzQQtKEDzipz/ZiOrvc=', 'Content-Length': '0'} 2023-07-29T00:25:06.204 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095e1d028e0cd8c22-0064c45c62-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.204 DEBUG:boto:b'hiiqos-58obj000000000000000000091000falseobj000000000000000000092023-07-29T00:24:59.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:06.205 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:06.205 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:06.206 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000009 2023-07-29T00:25:06.206 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000009 2023-07-29T00:25:06.206 DEBUG:boto:Method: GET 2023-07-29T00:25:06.207 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000009 2023-07-29T00:25:06.207 DEBUG:boto:Data: 2023-07-29T00:25:06.207 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.208 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:06.208 DEBUG:boto:Port: 8000 2023-07-29T00:25:06.208 DEBUG:boto:Params: {} 2023-07-29T00:25:06.209 DEBUG:boto:Token: None 2023-07-29T00:25:06.209 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/ 2023-07-29T00:25:06.210 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vTz8RfPkYzQQtKEDzipz/ZiOrvc= 2023-07-29T00:25:06.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vTz8RfPkYzQQtKEDzipz/ZiOrvc=', 'Content-Length': '0'} 2023-07-29T00:25:06.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000697d70ac2eec4a2a-0064c45c62-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.215 DEBUG:boto:b'hiiqos-58obj000000000000000000091000falseobj000000000000000000092023-07-29T00:24:59.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:06.216 DEBUG:boto:path=/hiiqos-58/obj00000000000000000009 2023-07-29T00:25:06.216 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000009 2023-07-29T00:25:06.217 DEBUG:boto:Method: HEAD 2023-07-29T00:25:06.217 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000009 2023-07-29T00:25:06.217 DEBUG:boto:Data: 2023-07-29T00:25:06.218 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.218 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:06.219 DEBUG:boto:Port: 8001 2023-07-29T00:25:06.219 DEBUG:boto:Params: {} 2023-07-29T00:25:06.219 DEBUG:boto:Token: None 2023-07-29T00:25:06.220 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000009 2023-07-29T00:25:06.220 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UC6J6nFaO023YhsW4mQgkp0lzyA= 2023-07-29T00:25:06.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UC6J6nFaO023YhsW4mQgkp0lzyA=', 'Content-Length': '0'} 2023-07-29T00:25:06.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8cdf1c8ee451d90-0064c45c62-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.223 DEBUG:boto:path=/hiiqos-58/obj00000000000000000009 2023-07-29T00:25:06.224 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000009 2023-07-29T00:25:06.224 DEBUG:boto:Method: HEAD 2023-07-29T00:25:06.224 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000009 2023-07-29T00:25:06.225 DEBUG:boto:Data: 2023-07-29T00:25:06.225 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.225 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:06.226 DEBUG:boto:Port: 8000 2023-07-29T00:25:06.226 DEBUG:boto:Params: {} 2023-07-29T00:25:06.226 DEBUG:boto:Token: None 2023-07-29T00:25:06.227 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000009 2023-07-29T00:25:06.227 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UC6J6nFaO023YhsW4mQgkp0lzyA= 2023-07-29T00:25:06.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UC6J6nFaO023YhsW4mQgkp0lzyA=', 'Content-Length': '0'} 2023-07-29T00:25:06.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fe329d336131d0f-0064c45c62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2023-07-29T00:25:06.231 DEBUG:boto:path=/hiiqos-58/obj00000000000000000009 2023-07-29T00:25:06.231 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000009 2023-07-29T00:25:06.231 DEBUG:boto:Method: GET 2023-07-29T00:25:06.232 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000009 2023-07-29T00:25:06.232 DEBUG:boto:Data: 2023-07-29T00:25:06.232 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.233 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:06.233 DEBUG:boto:Port: 8001 2023-07-29T00:25:06.233 DEBUG:boto:Params: {} 2023-07-29T00:25:06.234 DEBUG:boto:Token: None 2023-07-29T00:25:06.234 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000009 2023-07-29T00:25:06.235 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bT2UEMIDV3jYW4aOhu924c1NidQ= 2023-07-29T00:25:06.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bT2UEMIDV3jYW4aOhu924c1NidQ=', 'Content-Length': '0'} 2023-07-29T00:25:06.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cd58e9bfdb66280-0064c45c62-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.238 DEBUG:boto:path=/hiiqos-58/obj00000000000000000009 2023-07-29T00:25:06.238 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000009 2023-07-29T00:25:06.239 DEBUG:boto:Method: GET 2023-07-29T00:25:06.239 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000009 2023-07-29T00:25:06.239 DEBUG:boto:Data: 2023-07-29T00:25:06.240 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.240 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:06.240 DEBUG:boto:Port: 8000 2023-07-29T00:25:06.241 DEBUG:boto:Params: {} 2023-07-29T00:25:06.241 DEBUG:boto:Token: None 2023-07-29T00:25:06.241 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000009 2023-07-29T00:25:06.242 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bT2UEMIDV3jYW4aOhu924c1NidQ= 2023-07-29T00:25:06.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bT2UEMIDV3jYW4aOhu924c1NidQ=', 'Content-Length': '0'} 2023-07-29T00:25:06.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007b901ca67656d9b-0064c45c62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.245 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:06.245 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:06.246 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000009 2023-07-29T00:25:06.246 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000009 2023-07-29T00:25:06.246 DEBUG:boto:Method: GET 2023-07-29T00:25:06.247 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000009 2023-07-29T00:25:06.247 DEBUG:boto:Data: 2023-07-29T00:25:06.247 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.248 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:06.248 DEBUG:boto:Port: 8001 2023-07-29T00:25:06.248 DEBUG:boto:Params: {} 2023-07-29T00:25:06.249 DEBUG:boto:Token: None 2023-07-29T00:25:06.249 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/ 2023-07-29T00:25:06.250 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vTz8RfPkYzQQtKEDzipz/ZiOrvc= 2023-07-29T00:25:06.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vTz8RfPkYzQQtKEDzipz/ZiOrvc=', 'Content-Length': '0'} 2023-07-29T00:25:06.253 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a5e27124d382cbb-0064c45c62-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.253 DEBUG:boto:b'hiiqos-58obj000000000000000000091000falseobj000000000000000000092023-07-29T00:24:59.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:06.254 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:06.254 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:06.255 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000009 2023-07-29T00:25:06.255 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000009 2023-07-29T00:25:06.255 DEBUG:boto:Method: GET 2023-07-29T00:25:06.256 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000009 2023-07-29T00:25:06.256 DEBUG:boto:Data: 2023-07-29T00:25:06.257 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.257 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:06.257 DEBUG:boto:Port: 8000 2023-07-29T00:25:06.258 DEBUG:boto:Params: {} 2023-07-29T00:25:06.258 DEBUG:boto:Token: None 2023-07-29T00:25:06.259 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/ 2023-07-29T00:25:06.259 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vTz8RfPkYzQQtKEDzipz/ZiOrvc= 2023-07-29T00:25:06.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vTz8RfPkYzQQtKEDzipz/ZiOrvc=', 'Content-Length': '0'} 2023-07-29T00:25:06.263 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072c7bfe2e4f05e98-0064c45c62-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.304 DEBUG:boto:b'hiiqos-58obj000000000000000000091000falseobj000000000000000000092023-07-29T00:24:59.500Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:06.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-07-29T00:25:06.306 DEBUG:boto:path=/hiiqos-58/obj00000000000000000010 2023-07-29T00:25:06.306 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000010 2023-07-29T00:25:06.306 DEBUG:boto:path=/hiiqos-58/obj00000000000000000010?versionId=null 2023-07-29T00:25:06.307 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000010?versionId=null 2023-07-29T00:25:06.307 DEBUG:boto:Method: GET 2023-07-29T00:25:06.307 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000010?versionId=null 2023-07-29T00:25:06.308 DEBUG:boto:Data: 2023-07-29T00:25:06.308 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.308 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:06.309 DEBUG:boto:Port: 8001 2023-07-29T00:25:06.309 DEBUG:boto:Params: {} 2023-07-29T00:25:06.310 DEBUG:boto:Token: None 2023-07-29T00:25:06.310 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000010?versionId=null 2023-07-29T00:25:06.310 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SBVIEGdZpUiRGSOJNR07dFu60YU= 2023-07-29T00:25:06.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SBVIEGdZpUiRGSOJNR07dFu60YU=', 'Content-Length': '0'} 2023-07-29T00:25:06.313 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d82ff4b7082c46d-0064c45c62-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.314 DEBUG:boto:path=/hiiqos-58/obj00000000000000000010 2023-07-29T00:25:06.315 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000010 2023-07-29T00:25:06.315 DEBUG:boto:path=/hiiqos-58/obj00000000000000000010?versionId=null 2023-07-29T00:25:06.315 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000010?versionId=null 2023-07-29T00:25:06.316 DEBUG:boto:Method: GET 2023-07-29T00:25:06.316 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000010?versionId=null 2023-07-29T00:25:06.317 DEBUG:boto:Data: 2023-07-29T00:25:06.317 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.317 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:06.318 DEBUG:boto:Port: 8000 2023-07-29T00:25:06.318 DEBUG:boto:Params: {} 2023-07-29T00:25:06.318 DEBUG:boto:Token: None 2023-07-29T00:25:06.319 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000010?versionId=null 2023-07-29T00:25:06.319 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SBVIEGdZpUiRGSOJNR07dFu60YU= 2023-07-29T00:25:06.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SBVIEGdZpUiRGSOJNR07dFu60YU=', 'Content-Length': '0'} 2023-07-29T00:25:06.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7731eb8d00203c7-0064c45c62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.322 DEBUG:boto:path=/hiiqos-58/obj00000000000000000010 2023-07-29T00:25:06.323 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000010 2023-07-29T00:25:06.323 DEBUG:boto:path=/hiiqos-58/obj00000000000000000010?versionId=null 2023-07-29T00:25:06.323 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000010?versionId=null 2023-07-29T00:25:06.324 DEBUG:boto:Method: HEAD 2023-07-29T00:25:06.325 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000010?versionId=null 2023-07-29T00:25:06.325 DEBUG:boto:Data: 2023-07-29T00:25:06.325 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.326 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:06.326 DEBUG:boto:Port: 8001 2023-07-29T00:25:06.326 DEBUG:boto:Params: {} 2023-07-29T00:25:06.327 DEBUG:boto:Token: None 2023-07-29T00:25:06.327 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000010?versionId=null 2023-07-29T00:25:06.328 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bVMy9ORLOHejX5lPbQ2oibpSD9U= 2023-07-29T00:25:06.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bVMy9ORLOHejX5lPbQ2oibpSD9U=', 'Content-Length': '0'} 2023-07-29T00:25:06.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d3195ef70109dfa-0064c45c62-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.330 DEBUG:boto:path=/hiiqos-58/obj00000000000000000010 2023-07-29T00:25:06.331 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000010 2023-07-29T00:25:06.331 DEBUG:boto:path=/hiiqos-58/obj00000000000000000010?versionId=null 2023-07-29T00:25:06.331 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000010?versionId=null 2023-07-29T00:25:06.332 DEBUG:boto:Method: HEAD 2023-07-29T00:25:06.332 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000010?versionId=null 2023-07-29T00:25:06.332 DEBUG:boto:Data: 2023-07-29T00:25:06.333 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.333 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:06.333 DEBUG:boto:Port: 8000 2023-07-29T00:25:06.334 DEBUG:boto:Params: {} 2023-07-29T00:25:06.334 DEBUG:boto:Token: None 2023-07-29T00:25:06.335 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000010?versionId=null 2023-07-29T00:25:06.335 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bVMy9ORLOHejX5lPbQ2oibpSD9U= 2023-07-29T00:25:06.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bVMy9ORLOHejX5lPbQ2oibpSD9U=', 'Content-Length': '0'} 2023-07-29T00:25:06.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ffbf708ee3fc887-0064c45c62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-07-29T00:25:06.338 DEBUG:boto:path=/hiiqos-58/obj00000000000000000010 2023-07-29T00:25:06.339 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000010 2023-07-29T00:25:06.339 DEBUG:boto:path=/hiiqos-58/obj00000000000000000010?versionId=null 2023-07-29T00:25:06.339 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000010?versionId=null 2023-07-29T00:25:06.340 DEBUG:boto:Method: GET 2023-07-29T00:25:06.340 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000010?versionId=null 2023-07-29T00:25:06.340 DEBUG:boto:Data: 2023-07-29T00:25:06.341 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.341 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:06.341 DEBUG:boto:Port: 8001 2023-07-29T00:25:06.342 DEBUG:boto:Params: {} 2023-07-29T00:25:06.342 DEBUG:boto:Token: None 2023-07-29T00:25:06.343 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000010?versionId=null 2023-07-29T00:25:06.343 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SBVIEGdZpUiRGSOJNR07dFu60YU= 2023-07-29T00:25:06.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SBVIEGdZpUiRGSOJNR07dFu60YU=', 'Content-Length': '0'} 2023-07-29T00:25:06.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b41ca4bcb8971b92-0064c45c62-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.388 DEBUG:boto:path=/hiiqos-58/obj00000000000000000010 2023-07-29T00:25:06.389 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000010 2023-07-29T00:25:06.389 DEBUG:boto:path=/hiiqos-58/obj00000000000000000010?versionId=null 2023-07-29T00:25:06.390 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000010?versionId=null 2023-07-29T00:25:06.390 DEBUG:boto:Method: GET 2023-07-29T00:25:06.391 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000010?versionId=null 2023-07-29T00:25:06.391 DEBUG:boto:Data: 2023-07-29T00:25:06.391 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.392 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:06.392 DEBUG:boto:Port: 8000 2023-07-29T00:25:06.393 DEBUG:boto:Params: {} 2023-07-29T00:25:06.393 DEBUG:boto:Token: None 2023-07-29T00:25:06.394 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000010?versionId=null 2023-07-29T00:25:06.394 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SBVIEGdZpUiRGSOJNR07dFu60YU= 2023-07-29T00:25:06.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SBVIEGdZpUiRGSOJNR07dFu60YU=', 'Content-Length': '0'} 2023-07-29T00:25:06.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e4f4c1a96de9404-0064c45c62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.398 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:06.398 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:06.398 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000010 2023-07-29T00:25:06.399 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000010 2023-07-29T00:25:06.399 DEBUG:boto:Method: GET 2023-07-29T00:25:06.400 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000010 2023-07-29T00:25:06.400 DEBUG:boto:Data: 2023-07-29T00:25:06.400 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.401 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:06.401 DEBUG:boto:Port: 8001 2023-07-29T00:25:06.401 DEBUG:boto:Params: {} 2023-07-29T00:25:06.402 DEBUG:boto:Token: None 2023-07-29T00:25:06.402 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/ 2023-07-29T00:25:06.403 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vTz8RfPkYzQQtKEDzipz/ZiOrvc= 2023-07-29T00:25:06.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vTz8RfPkYzQQtKEDzipz/ZiOrvc=', 'Content-Length': '0'} 2023-07-29T00:25:06.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebb8858f8fb2cde7-0064c45c62-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.406 DEBUG:boto:b'hiiqos-58obj000000000000000000101000falseobj000000000000000000102023-07-29T00:24:59.665Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:06.406 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:06.407 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:06.407 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000010 2023-07-29T00:25:06.408 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000010 2023-07-29T00:25:06.408 DEBUG:boto:Method: GET 2023-07-29T00:25:06.408 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000010 2023-07-29T00:25:06.409 DEBUG:boto:Data: 2023-07-29T00:25:06.409 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.409 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:06.410 DEBUG:boto:Port: 8000 2023-07-29T00:25:06.410 DEBUG:boto:Params: {} 2023-07-29T00:25:06.411 DEBUG:boto:Token: None 2023-07-29T00:25:06.411 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/ 2023-07-29T00:25:06.411 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vTz8RfPkYzQQtKEDzipz/ZiOrvc= 2023-07-29T00:25:06.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vTz8RfPkYzQQtKEDzipz/ZiOrvc=', 'Content-Length': '0'} 2023-07-29T00:25:06.416 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f74e14abf129cd86-0064c45c62-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.417 DEBUG:boto:b'hiiqos-58obj000000000000000000101000falseobj000000000000000000102023-07-29T00:24:59.665Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:06.417 DEBUG:boto:path=/hiiqos-58/obj00000000000000000010 2023-07-29T00:25:06.418 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000010 2023-07-29T00:25:06.418 DEBUG:boto:Method: HEAD 2023-07-29T00:25:06.419 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000010 2023-07-29T00:25:06.419 DEBUG:boto:Data: 2023-07-29T00:25:06.419 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.420 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:06.420 DEBUG:boto:Port: 8001 2023-07-29T00:25:06.422 DEBUG:boto:Params: {} 2023-07-29T00:25:06.422 DEBUG:boto:Token: None 2023-07-29T00:25:06.423 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000010 2023-07-29T00:25:06.423 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:z0TmazVMR0bRJc/vYVDucuw8HBE= 2023-07-29T00:25:06.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:z0TmazVMR0bRJc/vYVDucuw8HBE=', 'Content-Length': '0'} 2023-07-29T00:25:06.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000898f3d638fd401e7-0064c45c62-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.426 DEBUG:boto:path=/hiiqos-58/obj00000000000000000010 2023-07-29T00:25:06.427 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000010 2023-07-29T00:25:06.427 DEBUG:boto:Method: HEAD 2023-07-29T00:25:06.427 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000010 2023-07-29T00:25:06.428 DEBUG:boto:Data: 2023-07-29T00:25:06.428 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.429 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:06.429 DEBUG:boto:Port: 8000 2023-07-29T00:25:06.429 DEBUG:boto:Params: {} 2023-07-29T00:25:06.430 DEBUG:boto:Token: None 2023-07-29T00:25:06.430 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000010 2023-07-29T00:25:06.431 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:z0TmazVMR0bRJc/vYVDucuw8HBE= 2023-07-29T00:25:06.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:z0TmazVMR0bRJc/vYVDucuw8HBE=', 'Content-Length': '0'} 2023-07-29T00:25:06.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fa7faf2dbfaae6d-0064c45c62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-07-29T00:25:06.434 DEBUG:boto:path=/hiiqos-58/obj00000000000000000010 2023-07-29T00:25:06.434 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000010 2023-07-29T00:25:06.435 DEBUG:boto:Method: GET 2023-07-29T00:25:06.435 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000010 2023-07-29T00:25:06.436 DEBUG:boto:Data: 2023-07-29T00:25:06.436 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.436 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:06.437 DEBUG:boto:Port: 8001 2023-07-29T00:25:06.437 DEBUG:boto:Params: {} 2023-07-29T00:25:06.438 DEBUG:boto:Token: None 2023-07-29T00:25:06.438 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000010 2023-07-29T00:25:06.438 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wt6oUKMXjITqeBU6WsOmcm1uKI0= 2023-07-29T00:25:06.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wt6oUKMXjITqeBU6WsOmcm1uKI0=', 'Content-Length': '0'} 2023-07-29T00:25:06.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098dde4adbc2b8ca4-0064c45c62-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.441 DEBUG:boto:path=/hiiqos-58/obj00000000000000000010 2023-07-29T00:25:06.442 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000010 2023-07-29T00:25:06.442 DEBUG:boto:Method: GET 2023-07-29T00:25:06.442 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000010 2023-07-29T00:25:06.443 DEBUG:boto:Data: 2023-07-29T00:25:06.443 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.444 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:06.445 DEBUG:boto:Port: 8000 2023-07-29T00:25:06.445 DEBUG:boto:Params: {} 2023-07-29T00:25:06.446 DEBUG:boto:Token: None 2023-07-29T00:25:06.446 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000010 2023-07-29T00:25:06.446 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wt6oUKMXjITqeBU6WsOmcm1uKI0= 2023-07-29T00:25:06.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wt6oUKMXjITqeBU6WsOmcm1uKI0=', 'Content-Length': '0'} 2023-07-29T00:25:06.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005870e0ecd3bbacd2-0064c45c62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.492 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:06.493 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:06.493 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000010 2023-07-29T00:25:06.494 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000010 2023-07-29T00:25:06.494 DEBUG:boto:Method: GET 2023-07-29T00:25:06.494 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000010 2023-07-29T00:25:06.495 DEBUG:boto:Data: 2023-07-29T00:25:06.495 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.495 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:06.496 DEBUG:boto:Port: 8001 2023-07-29T00:25:06.496 DEBUG:boto:Params: {} 2023-07-29T00:25:06.497 DEBUG:boto:Token: None 2023-07-29T00:25:06.497 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/ 2023-07-29T00:25:06.497 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vTz8RfPkYzQQtKEDzipz/ZiOrvc= 2023-07-29T00:25:06.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vTz8RfPkYzQQtKEDzipz/ZiOrvc=', 'Content-Length': '0'} 2023-07-29T00:25:06.500 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8ef31f1bbb5bec7-0064c45c62-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.501 DEBUG:boto:b'hiiqos-58obj000000000000000000101000falseobj000000000000000000102023-07-29T00:24:59.665Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:06.502 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:06.503 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:06.503 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000010 2023-07-29T00:25:06.503 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000010 2023-07-29T00:25:06.504 DEBUG:boto:Method: GET 2023-07-29T00:25:06.505 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000010 2023-07-29T00:25:06.505 DEBUG:boto:Data: 2023-07-29T00:25:06.505 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.506 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:06.506 DEBUG:boto:Port: 8000 2023-07-29T00:25:06.507 DEBUG:boto:Params: {} 2023-07-29T00:25:06.507 DEBUG:boto:Token: None 2023-07-29T00:25:06.507 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/ 2023-07-29T00:25:06.508 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vTz8RfPkYzQQtKEDzipz/ZiOrvc= 2023-07-29T00:25:06.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vTz8RfPkYzQQtKEDzipz/ZiOrvc=', 'Content-Length': '0'} 2023-07-29T00:25:06.512 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fbade222bb1facc-0064c45c62-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.513 DEBUG:boto:b'hiiqos-58obj000000000000000000101000falseobj000000000000000000102023-07-29T00:24:59.665Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:06.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-07-29T00:25:06.514 DEBUG:boto:path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:25:06.515 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:25:06.515 DEBUG:boto:path=/hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:25:06.515 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:25:06.516 DEBUG:boto:Method: GET 2023-07-29T00:25:06.516 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:25:06.517 DEBUG:boto:Data: 2023-07-29T00:25:06.517 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.517 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:06.518 DEBUG:boto:Port: 8001 2023-07-29T00:25:06.518 DEBUG:boto:Params: {} 2023-07-29T00:25:06.519 DEBUG:boto:Token: None 2023-07-29T00:25:06.519 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:25:06.520 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ULeEd0gAlQzMwRrDKs4OMSqoxTk= 2023-07-29T00:25:06.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ULeEd0gAlQzMwRrDKs4OMSqoxTk=', 'Content-Length': '0'} 2023-07-29T00:25:06.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f80130e44c60e013-0064c45c62-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.568 DEBUG:boto:path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:25:06.569 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:25:06.569 DEBUG:boto:path=/hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:25:06.570 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:25:06.570 DEBUG:boto:Method: GET 2023-07-29T00:25:06.570 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:25:06.571 DEBUG:boto:Data: 2023-07-29T00:25:06.571 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.572 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:06.573 DEBUG:boto:Port: 8000 2023-07-29T00:25:06.573 DEBUG:boto:Params: {} 2023-07-29T00:25:06.574 DEBUG:boto:Token: None 2023-07-29T00:25:06.575 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:25:06.576 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ULeEd0gAlQzMwRrDKs4OMSqoxTk= 2023-07-29T00:25:06.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ULeEd0gAlQzMwRrDKs4OMSqoxTk=', 'Content-Length': '0'} 2023-07-29T00:25:06.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012fe199f84905bbc-0064c45c62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.580 DEBUG:boto:path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:25:06.581 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:25:06.581 DEBUG:boto:path=/hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:25:06.582 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:25:06.582 DEBUG:boto:Method: HEAD 2023-07-29T00:25:06.582 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:25:06.583 DEBUG:boto:Data: 2023-07-29T00:25:06.583 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.583 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:06.584 DEBUG:boto:Port: 8001 2023-07-29T00:25:06.584 DEBUG:boto:Params: {} 2023-07-29T00:25:06.584 DEBUG:boto:Token: None 2023-07-29T00:25:06.585 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:25:06.585 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AddM6Edmwop/jYm+nUgdH+H43B8= 2023-07-29T00:25:06.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AddM6Edmwop/jYm+nUgdH+H43B8=', 'Content-Length': '0'} 2023-07-29T00:25:06.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7dccbb54213ded0-0064c45c62-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.588 DEBUG:boto:path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:25:06.588 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:25:06.589 DEBUG:boto:path=/hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:25:06.589 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:25:06.589 DEBUG:boto:Method: HEAD 2023-07-29T00:25:06.590 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:25:06.590 DEBUG:boto:Data: 2023-07-29T00:25:06.590 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.591 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:06.591 DEBUG:boto:Port: 8000 2023-07-29T00:25:06.591 DEBUG:boto:Params: {} 2023-07-29T00:25:06.592 DEBUG:boto:Token: None 2023-07-29T00:25:06.592 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:25:06.593 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AddM6Edmwop/jYm+nUgdH+H43B8= 2023-07-29T00:25:06.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AddM6Edmwop/jYm+nUgdH+H43B8=', 'Content-Length': '0'} 2023-07-29T00:25:06.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc202a5e56fd68f9-0064c45c62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-07-29T00:25:06.596 DEBUG:boto:path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:25:06.596 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:25:06.597 DEBUG:boto:path=/hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:25:06.597 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:25:06.597 DEBUG:boto:Method: GET 2023-07-29T00:25:06.598 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:25:06.598 DEBUG:boto:Data: 2023-07-29T00:25:06.598 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.599 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:06.599 DEBUG:boto:Port: 8001 2023-07-29T00:25:06.600 DEBUG:boto:Params: {} 2023-07-29T00:25:06.600 DEBUG:boto:Token: None 2023-07-29T00:25:06.600 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:25:06.601 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ULeEd0gAlQzMwRrDKs4OMSqoxTk= 2023-07-29T00:25:06.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ULeEd0gAlQzMwRrDKs4OMSqoxTk=', 'Content-Length': '0'} 2023-07-29T00:25:06.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6a148cd52e40304-0064c45c62-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.604 DEBUG:boto:path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:25:06.604 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:25:06.604 DEBUG:boto:path=/hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:25:06.605 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:25:06.605 DEBUG:boto:Method: GET 2023-07-29T00:25:06.605 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:25:06.606 DEBUG:boto:Data: 2023-07-29T00:25:06.606 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.607 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:06.607 DEBUG:boto:Port: 8000 2023-07-29T00:25:06.607 DEBUG:boto:Params: {} 2023-07-29T00:25:06.608 DEBUG:boto:Token: None 2023-07-29T00:25:06.608 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000011?versionId=null 2023-07-29T00:25:06.608 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ULeEd0gAlQzMwRrDKs4OMSqoxTk= 2023-07-29T00:25:06.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ULeEd0gAlQzMwRrDKs4OMSqoxTk=', 'Content-Length': '0'} 2023-07-29T00:25:06.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d887b2e607ed412-0064c45c62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.611 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:06.612 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:06.612 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000011 2023-07-29T00:25:06.614 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000011 2023-07-29T00:25:06.615 DEBUG:boto:Method: GET 2023-07-29T00:25:06.615 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000011 2023-07-29T00:25:06.615 DEBUG:boto:Data: 2023-07-29T00:25:06.616 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.616 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:06.616 DEBUG:boto:Port: 8001 2023-07-29T00:25:06.617 DEBUG:boto:Params: {} 2023-07-29T00:25:06.617 DEBUG:boto:Token: None 2023-07-29T00:25:06.618 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/ 2023-07-29T00:25:06.618 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vTz8RfPkYzQQtKEDzipz/ZiOrvc= 2023-07-29T00:25:06.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vTz8RfPkYzQQtKEDzipz/ZiOrvc=', 'Content-Length': '0'} 2023-07-29T00:25:06.621 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c81306d61202d992-0064c45c62-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.622 DEBUG:boto:b'hiiqos-58obj000000000000000000111000falseobj000000000000000000112023-07-29T00:24:58.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:06.622 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:06.623 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:06.623 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000011 2023-07-29T00:25:06.623 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000011 2023-07-29T00:25:06.624 DEBUG:boto:Method: GET 2023-07-29T00:25:06.624 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000011 2023-07-29T00:25:06.624 DEBUG:boto:Data: 2023-07-29T00:25:06.625 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.625 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:06.626 DEBUG:boto:Port: 8000 2023-07-29T00:25:06.626 DEBUG:boto:Params: {} 2023-07-29T00:25:06.626 DEBUG:boto:Token: None 2023-07-29T00:25:06.627 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/ 2023-07-29T00:25:06.627 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vTz8RfPkYzQQtKEDzipz/ZiOrvc= 2023-07-29T00:25:06.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vTz8RfPkYzQQtKEDzipz/ZiOrvc=', 'Content-Length': '0'} 2023-07-29T00:25:06.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033e07302d9bc8060-0064c45c62-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.636 DEBUG:boto:b'hiiqos-58obj000000000000000000111000falseobj000000000000000000112023-07-29T00:24:58.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:06.637 DEBUG:boto:path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:25:06.637 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:25:06.638 DEBUG:boto:Method: HEAD 2023-07-29T00:25:06.638 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000011 2023-07-29T00:25:06.639 DEBUG:boto:Data: 2023-07-29T00:25:06.639 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.639 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:06.640 DEBUG:boto:Port: 8001 2023-07-29T00:25:06.640 DEBUG:boto:Params: {} 2023-07-29T00:25:06.640 DEBUG:boto:Token: None 2023-07-29T00:25:06.641 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000011 2023-07-29T00:25:06.641 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9otRqRo7nmJmXhEPCcCIMncPork= 2023-07-29T00:25:06.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9otRqRo7nmJmXhEPCcCIMncPork=', 'Content-Length': '0'} 2023-07-29T00:25:06.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008770a1cc495c7f6f-0064c45c62-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.644 DEBUG:boto:path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:25:06.644 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:25:06.645 DEBUG:boto:Method: HEAD 2023-07-29T00:25:06.645 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000011 2023-07-29T00:25:06.645 DEBUG:boto:Data: 2023-07-29T00:25:06.646 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.646 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:06.646 DEBUG:boto:Port: 8000 2023-07-29T00:25:06.647 DEBUG:boto:Params: {} 2023-07-29T00:25:06.647 DEBUG:boto:Token: None 2023-07-29T00:25:06.648 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000011 2023-07-29T00:25:06.648 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9otRqRo7nmJmXhEPCcCIMncPork= 2023-07-29T00:25:06.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9otRqRo7nmJmXhEPCcCIMncPork=', 'Content-Length': '0'} 2023-07-29T00:25:06.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a10334d72f131f0-0064c45c62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-07-29T00:25:06.651 DEBUG:boto:path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:25:06.652 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:25:06.652 DEBUG:boto:Method: GET 2023-07-29T00:25:06.653 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000011 2023-07-29T00:25:06.653 DEBUG:boto:Data: 2023-07-29T00:25:06.654 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.654 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:06.654 DEBUG:boto:Port: 8001 2023-07-29T00:25:06.655 DEBUG:boto:Params: {} 2023-07-29T00:25:06.655 DEBUG:boto:Token: None 2023-07-29T00:25:06.656 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000011 2023-07-29T00:25:06.656 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vTGfjKi+4cppDqZ4AFlyeOlQJzM= 2023-07-29T00:25:06.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vTGfjKi+4cppDqZ4AFlyeOlQJzM=', 'Content-Length': '0'} 2023-07-29T00:25:06.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8a645def5d41a0d-0064c45c62-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.659 DEBUG:boto:path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:25:06.659 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000011 2023-07-29T00:25:06.660 DEBUG:boto:Method: GET 2023-07-29T00:25:06.660 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000011 2023-07-29T00:25:06.661 DEBUG:boto:Data: 2023-07-29T00:25:06.661 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.661 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:06.662 DEBUG:boto:Port: 8000 2023-07-29T00:25:06.662 DEBUG:boto:Params: {} 2023-07-29T00:25:06.662 DEBUG:boto:Token: None 2023-07-29T00:25:06.663 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000011 2023-07-29T00:25:06.663 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vTGfjKi+4cppDqZ4AFlyeOlQJzM= 2023-07-29T00:25:06.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vTGfjKi+4cppDqZ4AFlyeOlQJzM=', 'Content-Length': '0'} 2023-07-29T00:25:06.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000beef45d1a1e775be-0064c45c62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.667 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:06.667 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:06.667 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000011 2023-07-29T00:25:06.668 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000011 2023-07-29T00:25:06.668 DEBUG:boto:Method: GET 2023-07-29T00:25:06.668 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000011 2023-07-29T00:25:06.669 DEBUG:boto:Data: 2023-07-29T00:25:06.669 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.670 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:06.670 DEBUG:boto:Port: 8001 2023-07-29T00:25:06.670 DEBUG:boto:Params: {} 2023-07-29T00:25:06.671 DEBUG:boto:Token: None 2023-07-29T00:25:06.671 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/ 2023-07-29T00:25:06.672 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vTz8RfPkYzQQtKEDzipz/ZiOrvc= 2023-07-29T00:25:06.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vTz8RfPkYzQQtKEDzipz/ZiOrvc=', 'Content-Length': '0'} 2023-07-29T00:25:06.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003dd99e983d13f9c-0064c45c62-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.675 DEBUG:boto:b'hiiqos-58obj000000000000000000111000falseobj000000000000000000112023-07-29T00:24:58.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:06.676 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:06.676 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:06.677 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000011 2023-07-29T00:25:06.677 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000011 2023-07-29T00:25:06.677 DEBUG:boto:Method: GET 2023-07-29T00:25:06.678 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000011 2023-07-29T00:25:06.678 DEBUG:boto:Data: 2023-07-29T00:25:06.679 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.679 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:06.679 DEBUG:boto:Port: 8000 2023-07-29T00:25:06.680 DEBUG:boto:Params: {} 2023-07-29T00:25:06.680 DEBUG:boto:Token: None 2023-07-29T00:25:06.681 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/ 2023-07-29T00:25:06.681 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vTz8RfPkYzQQtKEDzipz/ZiOrvc= 2023-07-29T00:25:06.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vTz8RfPkYzQQtKEDzipz/ZiOrvc=', 'Content-Length': '0'} 2023-07-29T00:25:06.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7d46cbed70ff5f6-0064c45c62-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.687 DEBUG:boto:b'hiiqos-58obj000000000000000000111000falseobj000000000000000000112023-07-29T00:24:58.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:06.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-07-29T00:25:06.688 DEBUG:boto:path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:25:06.688 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:25:06.689 DEBUG:boto:path=/hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:25:06.689 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:25:06.689 DEBUG:boto:Method: GET 2023-07-29T00:25:06.690 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:25:06.690 DEBUG:boto:Data: 2023-07-29T00:25:06.691 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.691 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:06.691 DEBUG:boto:Port: 8001 2023-07-29T00:25:06.692 DEBUG:boto:Params: {} 2023-07-29T00:25:06.692 DEBUG:boto:Token: None 2023-07-29T00:25:06.692 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:25:06.693 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JTmZ/hibJlGmqFgcXDvLUO3Zg+I= 2023-07-29T00:25:06.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JTmZ/hibJlGmqFgcXDvLUO3Zg+I=', 'Content-Length': '0'} 2023-07-29T00:25:06.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051c846eaa02fd9c4-0064c45c62-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.696 DEBUG:boto:path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:25:06.697 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:25:06.697 DEBUG:boto:path=/hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:25:06.697 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:25:06.698 DEBUG:boto:Method: GET 2023-07-29T00:25:06.698 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:25:06.698 DEBUG:boto:Data: 2023-07-29T00:25:06.699 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.699 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:06.700 DEBUG:boto:Port: 8000 2023-07-29T00:25:06.700 DEBUG:boto:Params: {} 2023-07-29T00:25:06.700 DEBUG:boto:Token: None 2023-07-29T00:25:06.701 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:25:06.701 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JTmZ/hibJlGmqFgcXDvLUO3Zg+I= 2023-07-29T00:25:06.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JTmZ/hibJlGmqFgcXDvLUO3Zg+I=', 'Content-Length': '0'} 2023-07-29T00:25:06.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d074bd69feb5093-0064c45c62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.744 DEBUG:boto:path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:25:06.745 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:25:06.745 DEBUG:boto:path=/hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:25:06.746 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:25:06.747 DEBUG:boto:Method: HEAD 2023-07-29T00:25:06.747 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:25:06.747 DEBUG:boto:Data: 2023-07-29T00:25:06.748 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.748 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:06.748 DEBUG:boto:Port: 8001 2023-07-29T00:25:06.749 DEBUG:boto:Params: {} 2023-07-29T00:25:06.749 DEBUG:boto:Token: None 2023-07-29T00:25:06.750 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:25:06.750 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1ZRqZPjQJUT/uk1lrbgfzPK0HIQ= 2023-07-29T00:25:06.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1ZRqZPjQJUT/uk1lrbgfzPK0HIQ=', 'Content-Length': '0'} 2023-07-29T00:25:06.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e4be673f8e05708-0064c45c62-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.753 DEBUG:boto:path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:25:06.754 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:25:06.754 DEBUG:boto:path=/hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:25:06.754 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:25:06.755 DEBUG:boto:Method: HEAD 2023-07-29T00:25:06.755 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:25:06.755 DEBUG:boto:Data: 2023-07-29T00:25:06.756 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.756 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:06.757 DEBUG:boto:Port: 8000 2023-07-29T00:25:06.757 DEBUG:boto:Params: {} 2023-07-29T00:25:06.758 DEBUG:boto:Token: None 2023-07-29T00:25:06.758 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:25:06.758 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1ZRqZPjQJUT/uk1lrbgfzPK0HIQ= 2023-07-29T00:25:06.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1ZRqZPjQJUT/uk1lrbgfzPK0HIQ=', 'Content-Length': '0'} 2023-07-29T00:25:06.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d81af49c42fa38e5-0064c45c62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-07-29T00:25:06.762 DEBUG:boto:path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:25:06.762 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:25:06.762 DEBUG:boto:path=/hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:25:06.763 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:25:06.763 DEBUG:boto:Method: GET 2023-07-29T00:25:06.763 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:25:06.764 DEBUG:boto:Data: 2023-07-29T00:25:06.765 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.766 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:06.766 DEBUG:boto:Port: 8001 2023-07-29T00:25:06.767 DEBUG:boto:Params: {} 2023-07-29T00:25:06.767 DEBUG:boto:Token: None 2023-07-29T00:25:06.767 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:25:06.768 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JTmZ/hibJlGmqFgcXDvLUO3Zg+I= 2023-07-29T00:25:06.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JTmZ/hibJlGmqFgcXDvLUO3Zg+I=', 'Content-Length': '0'} 2023-07-29T00:25:06.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004285e3483e02d5a4-0064c45c62-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.816 DEBUG:boto:path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:25:06.817 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:25:06.817 DEBUG:boto:path=/hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:25:06.818 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:25:06.818 DEBUG:boto:Method: GET 2023-07-29T00:25:06.819 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:25:06.819 DEBUG:boto:Data: 2023-07-29T00:25:06.819 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.820 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:06.820 DEBUG:boto:Port: 8000 2023-07-29T00:25:06.820 DEBUG:boto:Params: {} 2023-07-29T00:25:06.821 DEBUG:boto:Token: None 2023-07-29T00:25:06.821 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000012?versionId=null 2023-07-29T00:25:06.822 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JTmZ/hibJlGmqFgcXDvLUO3Zg+I= 2023-07-29T00:25:06.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JTmZ/hibJlGmqFgcXDvLUO3Zg+I=', 'Content-Length': '0'} 2023-07-29T00:25:06.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de0f3faa7e001b0c-0064c45c62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.825 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:06.826 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:06.826 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000012 2023-07-29T00:25:06.827 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000012 2023-07-29T00:25:06.827 DEBUG:boto:Method: GET 2023-07-29T00:25:06.827 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000012 2023-07-29T00:25:06.828 DEBUG:boto:Data: 2023-07-29T00:25:06.828 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.829 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:06.829 DEBUG:boto:Port: 8001 2023-07-29T00:25:06.829 DEBUG:boto:Params: {} 2023-07-29T00:25:06.830 DEBUG:boto:Token: None 2023-07-29T00:25:06.830 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/ 2023-07-29T00:25:06.830 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vTz8RfPkYzQQtKEDzipz/ZiOrvc= 2023-07-29T00:25:06.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vTz8RfPkYzQQtKEDzipz/ZiOrvc=', 'Content-Length': '0'} 2023-07-29T00:25:06.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c1b059453acf5ae-0064c45c62-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.834 DEBUG:boto:b'hiiqos-58obj000000000000000000121000falseobj000000000000000000122023-07-29T00:24:59.604Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:06.835 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:06.836 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:06.836 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000012 2023-07-29T00:25:06.836 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000012 2023-07-29T00:25:06.837 DEBUG:boto:Method: GET 2023-07-29T00:25:06.837 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000012 2023-07-29T00:25:06.837 DEBUG:boto:Data: 2023-07-29T00:25:06.838 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.838 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:06.839 DEBUG:boto:Port: 8000 2023-07-29T00:25:06.839 DEBUG:boto:Params: {} 2023-07-29T00:25:06.839 DEBUG:boto:Token: None 2023-07-29T00:25:06.840 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/ 2023-07-29T00:25:06.840 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vTz8RfPkYzQQtKEDzipz/ZiOrvc= 2023-07-29T00:25:06.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vTz8RfPkYzQQtKEDzipz/ZiOrvc=', 'Content-Length': '0'} 2023-07-29T00:25:06.845 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008421a87f6af604c8-0064c45c62-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.845 DEBUG:boto:b'hiiqos-58obj000000000000000000121000falseobj000000000000000000122023-07-29T00:24:59.604Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:06.846 DEBUG:boto:path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:25:06.846 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:25:06.847 DEBUG:boto:Method: HEAD 2023-07-29T00:25:06.847 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000012 2023-07-29T00:25:06.847 DEBUG:boto:Data: 2023-07-29T00:25:06.848 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.848 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:06.849 DEBUG:boto:Port: 8001 2023-07-29T00:25:06.849 DEBUG:boto:Params: {} 2023-07-29T00:25:06.849 DEBUG:boto:Token: None 2023-07-29T00:25:06.850 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000012 2023-07-29T00:25:06.851 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aHeAWu6kryTOmbabrhVX2q0ps1A= 2023-07-29T00:25:06.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aHeAWu6kryTOmbabrhVX2q0ps1A=', 'Content-Length': '0'} 2023-07-29T00:25:06.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c2f0693f5f663cf-0064c45c62-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.854 DEBUG:boto:path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:25:06.854 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:25:06.855 DEBUG:boto:Method: HEAD 2023-07-29T00:25:06.855 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000012 2023-07-29T00:25:06.855 DEBUG:boto:Data: 2023-07-29T00:25:06.856 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.856 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:06.856 DEBUG:boto:Port: 8000 2023-07-29T00:25:06.857 DEBUG:boto:Params: {} 2023-07-29T00:25:06.857 DEBUG:boto:Token: None 2023-07-29T00:25:06.858 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000012 2023-07-29T00:25:06.858 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aHeAWu6kryTOmbabrhVX2q0ps1A= 2023-07-29T00:25:06.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aHeAWu6kryTOmbabrhVX2q0ps1A=', 'Content-Length': '0'} 2023-07-29T00:25:06.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9b5d0b1f85cd941-0064c45c62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-07-29T00:25:06.861 DEBUG:boto:path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:25:06.862 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:25:06.862 DEBUG:boto:Method: GET 2023-07-29T00:25:06.863 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000012 2023-07-29T00:25:06.864 DEBUG:boto:Data: 2023-07-29T00:25:06.864 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.864 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:06.865 DEBUG:boto:Port: 8001 2023-07-29T00:25:06.865 DEBUG:boto:Params: {} 2023-07-29T00:25:06.866 DEBUG:boto:Token: None 2023-07-29T00:25:06.866 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000012 2023-07-29T00:25:06.867 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:L000Meghh1iPyO0xXvTOeJeE1j8= 2023-07-29T00:25:06.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:L000Meghh1iPyO0xXvTOeJeE1j8=', 'Content-Length': '0'} 2023-07-29T00:25:06.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020b2291b1e8957a3-0064c45c62-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.912 DEBUG:boto:path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:25:06.913 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000012 2023-07-29T00:25:06.913 DEBUG:boto:Method: GET 2023-07-29T00:25:06.914 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000012 2023-07-29T00:25:06.914 DEBUG:boto:Data: 2023-07-29T00:25:06.915 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.915 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:06.915 DEBUG:boto:Port: 8000 2023-07-29T00:25:06.916 DEBUG:boto:Params: {} 2023-07-29T00:25:06.916 DEBUG:boto:Token: None 2023-07-29T00:25:06.917 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000012 2023-07-29T00:25:06.917 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:L000Meghh1iPyO0xXvTOeJeE1j8= 2023-07-29T00:25:06.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:L000Meghh1iPyO0xXvTOeJeE1j8=', 'Content-Length': '0'} 2023-07-29T00:25:06.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c39a570a8530444-0064c45c62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.921 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:06.921 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:06.921 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000012 2023-07-29T00:25:06.922 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000012 2023-07-29T00:25:06.922 DEBUG:boto:Method: GET 2023-07-29T00:25:06.922 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000012 2023-07-29T00:25:06.923 DEBUG:boto:Data: 2023-07-29T00:25:06.923 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.924 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:06.924 DEBUG:boto:Port: 8001 2023-07-29T00:25:06.924 DEBUG:boto:Params: {} 2023-07-29T00:25:06.925 DEBUG:boto:Token: None 2023-07-29T00:25:06.925 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/ 2023-07-29T00:25:06.926 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vTz8RfPkYzQQtKEDzipz/ZiOrvc= 2023-07-29T00:25:06.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vTz8RfPkYzQQtKEDzipz/ZiOrvc=', 'Content-Length': '0'} 2023-07-29T00:25:06.929 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf472d2436af8b71-0064c45c62-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.929 DEBUG:boto:b'hiiqos-58obj000000000000000000121000falseobj000000000000000000122023-07-29T00:24:59.604Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:06.930 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:06.930 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:06.931 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000012 2023-07-29T00:25:06.931 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000012 2023-07-29T00:25:06.932 DEBUG:boto:Method: GET 2023-07-29T00:25:06.932 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000012 2023-07-29T00:25:06.932 DEBUG:boto:Data: 2023-07-29T00:25:06.933 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.933 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:06.933 DEBUG:boto:Port: 8000 2023-07-29T00:25:06.934 DEBUG:boto:Params: {} 2023-07-29T00:25:06.934 DEBUG:boto:Token: None 2023-07-29T00:25:06.935 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/ 2023-07-29T00:25:06.935 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vTz8RfPkYzQQtKEDzipz/ZiOrvc= 2023-07-29T00:25:06.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vTz8RfPkYzQQtKEDzipz/ZiOrvc=', 'Content-Length': '0'} 2023-07-29T00:25:06.940 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a1794a91be51881-0064c45c62-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.941 DEBUG:boto:b'hiiqos-58obj000000000000000000121000falseobj000000000000000000122023-07-29T00:24:59.604Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:06.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-07-29T00:25:06.942 DEBUG:boto:path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:25:06.942 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:25:06.943 DEBUG:boto:path=/hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:25:06.943 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:25:06.943 DEBUG:boto:Method: GET 2023-07-29T00:25:06.944 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:25:06.944 DEBUG:boto:Data: 2023-07-29T00:25:06.944 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.945 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:06.945 DEBUG:boto:Port: 8001 2023-07-29T00:25:06.945 DEBUG:boto:Params: {} 2023-07-29T00:25:06.946 DEBUG:boto:Token: None 2023-07-29T00:25:06.946 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:25:06.947 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Q5uRHC9xwdkbPuq9i6USB8vbAAo= 2023-07-29T00:25:06.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q5uRHC9xwdkbPuq9i6USB8vbAAo=', 'Content-Length': '0'} 2023-07-29T00:25:06.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046af60e62d490395-0064c45c62-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.950 DEBUG:boto:path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:25:06.951 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:25:06.951 DEBUG:boto:path=/hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:25:06.952 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:25:06.952 DEBUG:boto:Method: GET 2023-07-29T00:25:06.952 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:25:06.953 DEBUG:boto:Data: 2023-07-29T00:25:06.953 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.954 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:06.954 DEBUG:boto:Port: 8000 2023-07-29T00:25:06.954 DEBUG:boto:Params: {} 2023-07-29T00:25:06.955 DEBUG:boto:Token: None 2023-07-29T00:25:06.955 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:25:06.956 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Q5uRHC9xwdkbPuq9i6USB8vbAAo= 2023-07-29T00:25:06.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q5uRHC9xwdkbPuq9i6USB8vbAAo=', 'Content-Length': '0'} 2023-07-29T00:25:06.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007499e9bd771b491f-0064c45c62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.959 DEBUG:boto:path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:25:06.959 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:25:06.960 DEBUG:boto:path=/hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:25:06.960 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:25:06.960 DEBUG:boto:Method: HEAD 2023-07-29T00:25:06.961 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:25:06.961 DEBUG:boto:Data: 2023-07-29T00:25:06.961 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.962 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:06.962 DEBUG:boto:Port: 8001 2023-07-29T00:25:06.962 DEBUG:boto:Params: {} 2023-07-29T00:25:06.963 DEBUG:boto:Token: None 2023-07-29T00:25:06.963 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:25:06.964 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WUkPB0GksoYAAS0B5Sc+NlWnxDA= 2023-07-29T00:25:06.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WUkPB0GksoYAAS0B5Sc+NlWnxDA=', 'Content-Length': '0'} 2023-07-29T00:25:06.966 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cdf9dffc67e0b44-0064c45c62-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.967 DEBUG:boto:path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:25:06.967 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:25:06.967 DEBUG:boto:path=/hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:25:06.968 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:25:06.968 DEBUG:boto:Method: HEAD 2023-07-29T00:25:06.968 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:25:06.969 DEBUG:boto:Data: 2023-07-29T00:25:06.969 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.969 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:06.970 DEBUG:boto:Port: 8000 2023-07-29T00:25:06.970 DEBUG:boto:Params: {} 2023-07-29T00:25:06.971 DEBUG:boto:Token: None 2023-07-29T00:25:06.971 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:25:06.971 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WUkPB0GksoYAAS0B5Sc+NlWnxDA= 2023-07-29T00:25:06.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WUkPB0GksoYAAS0B5Sc+NlWnxDA=', 'Content-Length': '0'} 2023-07-29T00:25:06.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000707cefbf72ce077b-0064c45c62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:06.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-07-29T00:25:06.975 DEBUG:boto:path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:25:06.976 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:25:06.977 DEBUG:boto:path=/hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:25:06.977 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:25:06.978 DEBUG:boto:Method: GET 2023-07-29T00:25:06.978 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:25:06.979 DEBUG:boto:Data: 2023-07-29T00:25:06.979 DEBUG:boto:Headers: {} 2023-07-29T00:25:06.979 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:06.980 DEBUG:boto:Port: 8001 2023-07-29T00:25:06.980 DEBUG:boto:Params: {} 2023-07-29T00:25:06.980 DEBUG:boto:Token: None 2023-07-29T00:25:06.981 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:06 GMT /hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:25:06.981 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Q5uRHC9xwdkbPuq9i6USB8vbAAo= 2023-07-29T00:25:06.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q5uRHC9xwdkbPuq9i6USB8vbAAo=', 'Content-Length': '0'} 2023-07-29T00:25:06.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c0d53d4adc0fb77-0064c45c62-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.024 DEBUG:boto:path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:25:07.025 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:25:07.025 DEBUG:boto:path=/hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:25:07.026 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:25:07.026 DEBUG:boto:Method: GET 2023-07-29T00:25:07.026 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:25:07.027 DEBUG:boto:Data: 2023-07-29T00:25:07.027 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.028 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:07.028 DEBUG:boto:Port: 8000 2023-07-29T00:25:07.028 DEBUG:boto:Params: {} 2023-07-29T00:25:07.029 DEBUG:boto:Token: None 2023-07-29T00:25:07.029 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000013?versionId=null 2023-07-29T00:25:07.030 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JSCmmLZuPLB/HrLxu7fQiTWSJ3c= 2023-07-29T00:25:07.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JSCmmLZuPLB/HrLxu7fQiTWSJ3c=', 'Content-Length': '0'} 2023-07-29T00:25:07.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000646fedc84f09c313-0064c45c62-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.033 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:07.034 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:07.034 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000013 2023-07-29T00:25:07.035 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000013 2023-07-29T00:25:07.035 DEBUG:boto:Method: GET 2023-07-29T00:25:07.035 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000013 2023-07-29T00:25:07.036 DEBUG:boto:Data: 2023-07-29T00:25:07.036 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.037 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:07.037 DEBUG:boto:Port: 8001 2023-07-29T00:25:07.037 DEBUG:boto:Params: {} 2023-07-29T00:25:07.038 DEBUG:boto:Token: None 2023-07-29T00:25:07.038 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/ 2023-07-29T00:25:07.039 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lWrDRkW/GNAXJIbFCESeE57zgGQ= 2023-07-29T00:25:07.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lWrDRkW/GNAXJIbFCESeE57zgGQ=', 'Content-Length': '0'} 2023-07-29T00:25:07.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f0d4bd847a81795-0064c45c62-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.043 DEBUG:boto:b'hiiqos-58obj000000000000000000131000falseobj000000000000000000132023-07-29T00:24:59.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:07.044 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:07.044 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:07.044 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000013 2023-07-29T00:25:07.045 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000013 2023-07-29T00:25:07.045 DEBUG:boto:Method: GET 2023-07-29T00:25:07.046 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000013 2023-07-29T00:25:07.046 DEBUG:boto:Data: 2023-07-29T00:25:07.047 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.048 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:07.048 DEBUG:boto:Port: 8000 2023-07-29T00:25:07.049 DEBUG:boto:Params: {} 2023-07-29T00:25:07.049 DEBUG:boto:Token: None 2023-07-29T00:25:07.050 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/ 2023-07-29T00:25:07.050 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lWrDRkW/GNAXJIbFCESeE57zgGQ= 2023-07-29T00:25:07.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lWrDRkW/GNAXJIbFCESeE57zgGQ=', 'Content-Length': '0'} 2023-07-29T00:25:07.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c886dda2fa6fffd-0064c45c63-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.057 DEBUG:boto:b'hiiqos-58obj000000000000000000131000falseobj000000000000000000132023-07-29T00:24:59.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:07.057 DEBUG:boto:path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:25:07.058 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:25:07.058 DEBUG:boto:Method: HEAD 2023-07-29T00:25:07.059 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000013 2023-07-29T00:25:07.059 DEBUG:boto:Data: 2023-07-29T00:25:07.059 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.060 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:07.060 DEBUG:boto:Port: 8001 2023-07-29T00:25:07.060 DEBUG:boto:Params: {} 2023-07-29T00:25:07.061 DEBUG:boto:Token: None 2023-07-29T00:25:07.061 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000013 2023-07-29T00:25:07.062 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7pPfQ1WqnbK383MKf4vRVzRTD6Y= 2023-07-29T00:25:07.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7pPfQ1WqnbK383MKf4vRVzRTD6Y=', 'Content-Length': '0'} 2023-07-29T00:25:07.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e56a93f5daa34b0-0064c45c63-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.065 DEBUG:boto:path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:25:07.066 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:25:07.066 DEBUG:boto:Method: HEAD 2023-07-29T00:25:07.066 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000013 2023-07-29T00:25:07.067 DEBUG:boto:Data: 2023-07-29T00:25:07.067 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.068 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:07.068 DEBUG:boto:Port: 8000 2023-07-29T00:25:07.069 DEBUG:boto:Params: {} 2023-07-29T00:25:07.070 DEBUG:boto:Token: None 2023-07-29T00:25:07.070 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000013 2023-07-29T00:25:07.071 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7pPfQ1WqnbK383MKf4vRVzRTD6Y= 2023-07-29T00:25:07.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7pPfQ1WqnbK383MKf4vRVzRTD6Y=', 'Content-Length': '0'} 2023-07-29T00:25:07.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9ecc66d47fdddfe-0064c45c63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-07-29T00:25:07.074 DEBUG:boto:path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:25:07.074 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:25:07.075 DEBUG:boto:Method: GET 2023-07-29T00:25:07.075 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000013 2023-07-29T00:25:07.076 DEBUG:boto:Data: 2023-07-29T00:25:07.076 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.077 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:07.077 DEBUG:boto:Port: 8001 2023-07-29T00:25:07.077 DEBUG:boto:Params: {} 2023-07-29T00:25:07.078 DEBUG:boto:Token: None 2023-07-29T00:25:07.078 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000013 2023-07-29T00:25:07.079 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kjzITG069pu3k8DJ+5Jzvgbdxpg= 2023-07-29T00:25:07.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kjzITG069pu3k8DJ+5Jzvgbdxpg=', 'Content-Length': '0'} 2023-07-29T00:25:07.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028bfd2e3f05e4f00-0064c45c63-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.124 DEBUG:boto:path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:25:07.125 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000013 2023-07-29T00:25:07.125 DEBUG:boto:Method: GET 2023-07-29T00:25:07.126 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000013 2023-07-29T00:25:07.126 DEBUG:boto:Data: 2023-07-29T00:25:07.126 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.127 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:07.127 DEBUG:boto:Port: 8000 2023-07-29T00:25:07.127 DEBUG:boto:Params: {} 2023-07-29T00:25:07.128 DEBUG:boto:Token: None 2023-07-29T00:25:07.129 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000013 2023-07-29T00:25:07.129 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kjzITG069pu3k8DJ+5Jzvgbdxpg= 2023-07-29T00:25:07.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kjzITG069pu3k8DJ+5Jzvgbdxpg=', 'Content-Length': '0'} 2023-07-29T00:25:07.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d797960ce39d7463-0064c45c63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.132 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:07.133 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:07.133 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000013 2023-07-29T00:25:07.133 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000013 2023-07-29T00:25:07.134 DEBUG:boto:Method: GET 2023-07-29T00:25:07.134 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000013 2023-07-29T00:25:07.134 DEBUG:boto:Data: 2023-07-29T00:25:07.135 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.135 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:07.136 DEBUG:boto:Port: 8001 2023-07-29T00:25:07.136 DEBUG:boto:Params: {} 2023-07-29T00:25:07.136 DEBUG:boto:Token: None 2023-07-29T00:25:07.137 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/ 2023-07-29T00:25:07.137 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lWrDRkW/GNAXJIbFCESeE57zgGQ= 2023-07-29T00:25:07.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lWrDRkW/GNAXJIbFCESeE57zgGQ=', 'Content-Length': '0'} 2023-07-29T00:25:07.140 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007eb2995304179531-0064c45c63-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.141 DEBUG:boto:b'hiiqos-58obj000000000000000000131000falseobj000000000000000000132023-07-29T00:24:59.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:07.141 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:07.142 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:07.142 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000013 2023-07-29T00:25:07.142 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000013 2023-07-29T00:25:07.143 DEBUG:boto:Method: GET 2023-07-29T00:25:07.143 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000013 2023-07-29T00:25:07.143 DEBUG:boto:Data: 2023-07-29T00:25:07.144 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.144 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:07.144 DEBUG:boto:Port: 8000 2023-07-29T00:25:07.145 DEBUG:boto:Params: {} 2023-07-29T00:25:07.145 DEBUG:boto:Token: None 2023-07-29T00:25:07.146 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/ 2023-07-29T00:25:07.146 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lWrDRkW/GNAXJIbFCESeE57zgGQ= 2023-07-29T00:25:07.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lWrDRkW/GNAXJIbFCESeE57zgGQ=', 'Content-Length': '0'} 2023-07-29T00:25:07.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a17f38fec90ccb1-0064c45c63-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.152 DEBUG:boto:b'hiiqos-58obj000000000000000000131000falseobj000000000000000000132023-07-29T00:24:59.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:07.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2023-07-29T00:25:07.153 DEBUG:boto:path=/hiiqos-58/obj00000000000000000014 2023-07-29T00:25:07.153 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000014 2023-07-29T00:25:07.153 DEBUG:boto:path=/hiiqos-58/obj00000000000000000014?versionId=null 2023-07-29T00:25:07.154 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000014?versionId=null 2023-07-29T00:25:07.154 DEBUG:boto:Method: GET 2023-07-29T00:25:07.154 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000014?versionId=null 2023-07-29T00:25:07.155 DEBUG:boto:Data: 2023-07-29T00:25:07.155 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.156 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:07.156 DEBUG:boto:Port: 8001 2023-07-29T00:25:07.156 DEBUG:boto:Params: {} 2023-07-29T00:25:07.157 DEBUG:boto:Token: None 2023-07-29T00:25:07.157 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000014?versionId=null 2023-07-29T00:25:07.158 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+JSZguyrIGIE1c6oSVCF1ooLLd0= 2023-07-29T00:25:07.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+JSZguyrIGIE1c6oSVCF1ooLLd0=', 'Content-Length': '0'} 2023-07-29T00:25:07.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f2f4677d4ba7a62-0064c45c63-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.161 DEBUG:boto:path=/hiiqos-58/obj00000000000000000014 2023-07-29T00:25:07.161 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000014 2023-07-29T00:25:07.162 DEBUG:boto:path=/hiiqos-58/obj00000000000000000014?versionId=null 2023-07-29T00:25:07.162 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000014?versionId=null 2023-07-29T00:25:07.162 DEBUG:boto:Method: GET 2023-07-29T00:25:07.163 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000014?versionId=null 2023-07-29T00:25:07.163 DEBUG:boto:Data: 2023-07-29T00:25:07.163 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.164 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:07.164 DEBUG:boto:Port: 8000 2023-07-29T00:25:07.165 DEBUG:boto:Params: {} 2023-07-29T00:25:07.165 DEBUG:boto:Token: None 2023-07-29T00:25:07.166 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000014?versionId=null 2023-07-29T00:25:07.166 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+JSZguyrIGIE1c6oSVCF1ooLLd0= 2023-07-29T00:25:07.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+JSZguyrIGIE1c6oSVCF1ooLLd0=', 'Content-Length': '0'} 2023-07-29T00:25:07.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029665bae5881380e-0064c45c63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.212 DEBUG:boto:path=/hiiqos-58/obj00000000000000000014 2023-07-29T00:25:07.213 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000014 2023-07-29T00:25:07.213 DEBUG:boto:path=/hiiqos-58/obj00000000000000000014?versionId=null 2023-07-29T00:25:07.214 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000014?versionId=null 2023-07-29T00:25:07.214 DEBUG:boto:Method: HEAD 2023-07-29T00:25:07.215 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000014?versionId=null 2023-07-29T00:25:07.215 DEBUG:boto:Data: 2023-07-29T00:25:07.215 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.216 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:07.216 DEBUG:boto:Port: 8001 2023-07-29T00:25:07.216 DEBUG:boto:Params: {} 2023-07-29T00:25:07.217 DEBUG:boto:Token: None 2023-07-29T00:25:07.217 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000014?versionId=null 2023-07-29T00:25:07.218 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hjiJemzSAP4aLPZsaMbRjhOQTA0= 2023-07-29T00:25:07.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hjiJemzSAP4aLPZsaMbRjhOQTA0=', 'Content-Length': '0'} 2023-07-29T00:25:07.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078ff445c19c69af2-0064c45c63-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.221 DEBUG:boto:path=/hiiqos-58/obj00000000000000000014 2023-07-29T00:25:07.222 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000014 2023-07-29T00:25:07.222 DEBUG:boto:path=/hiiqos-58/obj00000000000000000014?versionId=null 2023-07-29T00:25:07.223 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000014?versionId=null 2023-07-29T00:25:07.223 DEBUG:boto:Method: HEAD 2023-07-29T00:25:07.223 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000014?versionId=null 2023-07-29T00:25:07.224 DEBUG:boto:Data: 2023-07-29T00:25:07.224 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.225 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:07.225 DEBUG:boto:Port: 8000 2023-07-29T00:25:07.225 DEBUG:boto:Params: {} 2023-07-29T00:25:07.226 DEBUG:boto:Token: None 2023-07-29T00:25:07.226 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000014?versionId=null 2023-07-29T00:25:07.227 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hjiJemzSAP4aLPZsaMbRjhOQTA0= 2023-07-29T00:25:07.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hjiJemzSAP4aLPZsaMbRjhOQTA0=', 'Content-Length': '0'} 2023-07-29T00:25:07.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033693c76365cd5c9-0064c45c63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2023-07-29T00:25:07.230 DEBUG:boto:path=/hiiqos-58/obj00000000000000000014 2023-07-29T00:25:07.231 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000014 2023-07-29T00:25:07.231 DEBUG:boto:path=/hiiqos-58/obj00000000000000000014?versionId=null 2023-07-29T00:25:07.232 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000014?versionId=null 2023-07-29T00:25:07.232 DEBUG:boto:Method: GET 2023-07-29T00:25:07.233 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000014?versionId=null 2023-07-29T00:25:07.233 DEBUG:boto:Data: 2023-07-29T00:25:07.233 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.234 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:07.234 DEBUG:boto:Port: 8001 2023-07-29T00:25:07.235 DEBUG:boto:Params: {} 2023-07-29T00:25:07.235 DEBUG:boto:Token: None 2023-07-29T00:25:07.236 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000014?versionId=null 2023-07-29T00:25:07.236 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+JSZguyrIGIE1c6oSVCF1ooLLd0= 2023-07-29T00:25:07.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+JSZguyrIGIE1c6oSVCF1ooLLd0=', 'Content-Length': '0'} 2023-07-29T00:25:07.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b4316f5962490aa-0064c45c63-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.280 DEBUG:boto:path=/hiiqos-58/obj00000000000000000014 2023-07-29T00:25:07.281 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000014 2023-07-29T00:25:07.282 DEBUG:boto:path=/hiiqos-58/obj00000000000000000014?versionId=null 2023-07-29T00:25:07.282 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000014?versionId=null 2023-07-29T00:25:07.282 DEBUG:boto:Method: GET 2023-07-29T00:25:07.284 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000014?versionId=null 2023-07-29T00:25:07.284 DEBUG:boto:Data: 2023-07-29T00:25:07.285 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.285 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:07.286 DEBUG:boto:Port: 8000 2023-07-29T00:25:07.288 DEBUG:boto:Params: {} 2023-07-29T00:25:07.288 DEBUG:boto:Token: None 2023-07-29T00:25:07.289 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000014?versionId=null 2023-07-29T00:25:07.290 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+JSZguyrIGIE1c6oSVCF1ooLLd0= 2023-07-29T00:25:07.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+JSZguyrIGIE1c6oSVCF1ooLLd0=', 'Content-Length': '0'} 2023-07-29T00:25:07.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073283564dcf3dc8f-0064c45c63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.293 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:07.294 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:07.295 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000014 2023-07-29T00:25:07.296 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000014 2023-07-29T00:25:07.296 DEBUG:boto:Method: GET 2023-07-29T00:25:07.297 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000014 2023-07-29T00:25:07.298 DEBUG:boto:Data: 2023-07-29T00:25:07.299 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.299 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:07.299 DEBUG:boto:Port: 8001 2023-07-29T00:25:07.300 DEBUG:boto:Params: {} 2023-07-29T00:25:07.300 DEBUG:boto:Token: None 2023-07-29T00:25:07.301 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/ 2023-07-29T00:25:07.301 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lWrDRkW/GNAXJIbFCESeE57zgGQ= 2023-07-29T00:25:07.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lWrDRkW/GNAXJIbFCESeE57zgGQ=', 'Content-Length': '0'} 2023-07-29T00:25:07.305 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b00c044d89afa8ab-0064c45c63-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.305 DEBUG:boto:b'hiiqos-58obj000000000000000000141000falseobj000000000000000000142023-07-29T00:24:59.990Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:07.306 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:07.307 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:07.307 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000014 2023-07-29T00:25:07.308 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000014 2023-07-29T00:25:07.308 DEBUG:boto:Method: GET 2023-07-29T00:25:07.308 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000014 2023-07-29T00:25:07.309 DEBUG:boto:Data: 2023-07-29T00:25:07.309 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.310 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:07.310 DEBUG:boto:Port: 8000 2023-07-29T00:25:07.311 DEBUG:boto:Params: {} 2023-07-29T00:25:07.311 DEBUG:boto:Token: None 2023-07-29T00:25:07.312 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/ 2023-07-29T00:25:07.312 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lWrDRkW/GNAXJIbFCESeE57zgGQ= 2023-07-29T00:25:07.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lWrDRkW/GNAXJIbFCESeE57zgGQ=', 'Content-Length': '0'} 2023-07-29T00:25:07.318 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000648fe6113454e34e-0064c45c63-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.319 DEBUG:boto:b'hiiqos-58obj000000000000000000141000falseobj000000000000000000142023-07-29T00:24:59.990Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:07.320 DEBUG:boto:path=/hiiqos-58/obj00000000000000000014 2023-07-29T00:25:07.320 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000014 2023-07-29T00:25:07.320 DEBUG:boto:Method: HEAD 2023-07-29T00:25:07.321 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000014 2023-07-29T00:25:07.321 DEBUG:boto:Data: 2023-07-29T00:25:07.322 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.322 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:07.322 DEBUG:boto:Port: 8001 2023-07-29T00:25:07.323 DEBUG:boto:Params: {} 2023-07-29T00:25:07.323 DEBUG:boto:Token: None 2023-07-29T00:25:07.324 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000014 2023-07-29T00:25:07.324 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UfDatPUDH7Dc0jQR1a+BAKn+0hc= 2023-07-29T00:25:07.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UfDatPUDH7Dc0jQR1a+BAKn+0hc=', 'Content-Length': '0'} 2023-07-29T00:25:07.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba249e2b0b88858d-0064c45c63-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.327 DEBUG:boto:path=/hiiqos-58/obj00000000000000000014 2023-07-29T00:25:07.328 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000014 2023-07-29T00:25:07.328 DEBUG:boto:Method: HEAD 2023-07-29T00:25:07.329 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000014 2023-07-29T00:25:07.329 DEBUG:boto:Data: 2023-07-29T00:25:07.329 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.330 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:07.330 DEBUG:boto:Port: 8000 2023-07-29T00:25:07.330 DEBUG:boto:Params: {} 2023-07-29T00:25:07.331 DEBUG:boto:Token: None 2023-07-29T00:25:07.331 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000014 2023-07-29T00:25:07.331 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UfDatPUDH7Dc0jQR1a+BAKn+0hc= 2023-07-29T00:25:07.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UfDatPUDH7Dc0jQR1a+BAKn+0hc=', 'Content-Length': '0'} 2023-07-29T00:25:07.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000278ab534e8d32c2d-0064c45c63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2023-07-29T00:25:07.335 DEBUG:boto:path=/hiiqos-58/obj00000000000000000014 2023-07-29T00:25:07.335 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000014 2023-07-29T00:25:07.335 DEBUG:boto:Method: GET 2023-07-29T00:25:07.336 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000014 2023-07-29T00:25:07.336 DEBUG:boto:Data: 2023-07-29T00:25:07.337 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.337 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:07.337 DEBUG:boto:Port: 8001 2023-07-29T00:25:07.338 DEBUG:boto:Params: {} 2023-07-29T00:25:07.338 DEBUG:boto:Token: None 2023-07-29T00:25:07.339 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000014 2023-07-29T00:25:07.339 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0WTMfqII0af6duvicE8/GR3Ak7o= 2023-07-29T00:25:07.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0WTMfqII0af6duvicE8/GR3Ak7o=', 'Content-Length': '0'} 2023-07-29T00:25:07.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc2ee2a2838a0c66-0064c45c63-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.342 DEBUG:boto:path=/hiiqos-58/obj00000000000000000014 2023-07-29T00:25:07.343 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000014 2023-07-29T00:25:07.343 DEBUG:boto:Method: GET 2023-07-29T00:25:07.344 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000014 2023-07-29T00:25:07.344 DEBUG:boto:Data: 2023-07-29T00:25:07.344 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.345 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:07.345 DEBUG:boto:Port: 8000 2023-07-29T00:25:07.345 DEBUG:boto:Params: {} 2023-07-29T00:25:07.346 DEBUG:boto:Token: None 2023-07-29T00:25:07.346 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000014 2023-07-29T00:25:07.347 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0WTMfqII0af6duvicE8/GR3Ak7o= 2023-07-29T00:25:07.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0WTMfqII0af6duvicE8/GR3Ak7o=', 'Content-Length': '0'} 2023-07-29T00:25:07.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009dc7b0344ae63a0f-0064c45c63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.396 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:07.397 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:07.397 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000014 2023-07-29T00:25:07.398 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000014 2023-07-29T00:25:07.398 DEBUG:boto:Method: GET 2023-07-29T00:25:07.398 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000014 2023-07-29T00:25:07.399 DEBUG:boto:Data: 2023-07-29T00:25:07.399 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.399 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:07.400 DEBUG:boto:Port: 8001 2023-07-29T00:25:07.400 DEBUG:boto:Params: {} 2023-07-29T00:25:07.401 DEBUG:boto:Token: None 2023-07-29T00:25:07.402 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/ 2023-07-29T00:25:07.403 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lWrDRkW/GNAXJIbFCESeE57zgGQ= 2023-07-29T00:25:07.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lWrDRkW/GNAXJIbFCESeE57zgGQ=', 'Content-Length': '0'} 2023-07-29T00:25:07.406 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9a25f26dfa3d596-0064c45c63-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.407 DEBUG:boto:b'hiiqos-58obj000000000000000000141000falseobj000000000000000000142023-07-29T00:24:59.990Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:07.408 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:07.408 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:07.409 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000014 2023-07-29T00:25:07.409 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000014 2023-07-29T00:25:07.410 DEBUG:boto:Method: GET 2023-07-29T00:25:07.410 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000014 2023-07-29T00:25:07.410 DEBUG:boto:Data: 2023-07-29T00:25:07.411 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.412 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:07.412 DEBUG:boto:Port: 8000 2023-07-29T00:25:07.412 DEBUG:boto:Params: {} 2023-07-29T00:25:07.413 DEBUG:boto:Token: None 2023-07-29T00:25:07.413 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/ 2023-07-29T00:25:07.414 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lWrDRkW/GNAXJIbFCESeE57zgGQ= 2023-07-29T00:25:07.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lWrDRkW/GNAXJIbFCESeE57zgGQ=', 'Content-Length': '0'} 2023-07-29T00:25:07.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed5ecf21a169c9cc-0064c45c63-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.419 DEBUG:boto:b'hiiqos-58obj000000000000000000141000falseobj000000000000000000142023-07-29T00:24:59.990Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:07.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2023-07-29T00:25:07.420 DEBUG:boto:path=/hiiqos-58/obj00000000000000000015 2023-07-29T00:25:07.421 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000015 2023-07-29T00:25:07.421 DEBUG:boto:path=/hiiqos-58/obj00000000000000000015?versionId=null 2023-07-29T00:25:07.421 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000015?versionId=null 2023-07-29T00:25:07.422 DEBUG:boto:Method: GET 2023-07-29T00:25:07.422 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000015?versionId=null 2023-07-29T00:25:07.423 DEBUG:boto:Data: 2023-07-29T00:25:07.423 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.423 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:07.424 DEBUG:boto:Port: 8001 2023-07-29T00:25:07.424 DEBUG:boto:Params: {} 2023-07-29T00:25:07.425 DEBUG:boto:Token: None 2023-07-29T00:25:07.425 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000015?versionId=null 2023-07-29T00:25:07.425 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ykPlpJzC6VBk/ARhx5X/tuxlG3c= 2023-07-29T00:25:07.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ykPlpJzC6VBk/ARhx5X/tuxlG3c=', 'Content-Length': '0'} 2023-07-29T00:25:07.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000623f9ce9f3b08503-0064c45c63-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.429 DEBUG:boto:path=/hiiqos-58/obj00000000000000000015 2023-07-29T00:25:07.429 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000015 2023-07-29T00:25:07.430 DEBUG:boto:path=/hiiqos-58/obj00000000000000000015?versionId=null 2023-07-29T00:25:07.430 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000015?versionId=null 2023-07-29T00:25:07.430 DEBUG:boto:Method: GET 2023-07-29T00:25:07.431 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000015?versionId=null 2023-07-29T00:25:07.431 DEBUG:boto:Data: 2023-07-29T00:25:07.431 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.432 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:07.432 DEBUG:boto:Port: 8000 2023-07-29T00:25:07.433 DEBUG:boto:Params: {} 2023-07-29T00:25:07.433 DEBUG:boto:Token: None 2023-07-29T00:25:07.433 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000015?versionId=null 2023-07-29T00:25:07.434 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ykPlpJzC6VBk/ARhx5X/tuxlG3c= 2023-07-29T00:25:07.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ykPlpJzC6VBk/ARhx5X/tuxlG3c=', 'Content-Length': '0'} 2023-07-29T00:25:07.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a058489f86b9c4a-0064c45c63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.437 DEBUG:boto:path=/hiiqos-58/obj00000000000000000015 2023-07-29T00:25:07.438 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000015 2023-07-29T00:25:07.438 DEBUG:boto:path=/hiiqos-58/obj00000000000000000015?versionId=null 2023-07-29T00:25:07.438 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000015?versionId=null 2023-07-29T00:25:07.439 DEBUG:boto:Method: HEAD 2023-07-29T00:25:07.439 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000015?versionId=null 2023-07-29T00:25:07.439 DEBUG:boto:Data: 2023-07-29T00:25:07.440 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.440 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:07.440 DEBUG:boto:Port: 8001 2023-07-29T00:25:07.441 DEBUG:boto:Params: {} 2023-07-29T00:25:07.441 DEBUG:boto:Token: None 2023-07-29T00:25:07.441 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000015?versionId=null 2023-07-29T00:25:07.442 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rBHEaOSdeZeQoS7kF3RZqV21tmM= 2023-07-29T00:25:07.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rBHEaOSdeZeQoS7kF3RZqV21tmM=', 'Content-Length': '0'} 2023-07-29T00:25:07.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb2d716d5c31a5ee-0064c45c63-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.445 DEBUG:boto:path=/hiiqos-58/obj00000000000000000015 2023-07-29T00:25:07.446 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000015 2023-07-29T00:25:07.446 DEBUG:boto:path=/hiiqos-58/obj00000000000000000015?versionId=null 2023-07-29T00:25:07.446 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000015?versionId=null 2023-07-29T00:25:07.447 DEBUG:boto:Method: HEAD 2023-07-29T00:25:07.447 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000015?versionId=null 2023-07-29T00:25:07.447 DEBUG:boto:Data: 2023-07-29T00:25:07.448 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.448 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:07.448 DEBUG:boto:Port: 8000 2023-07-29T00:25:07.449 DEBUG:boto:Params: {} 2023-07-29T00:25:07.449 DEBUG:boto:Token: None 2023-07-29T00:25:07.450 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000015?versionId=null 2023-07-29T00:25:07.450 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rBHEaOSdeZeQoS7kF3RZqV21tmM= 2023-07-29T00:25:07.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rBHEaOSdeZeQoS7kF3RZqV21tmM=', 'Content-Length': '0'} 2023-07-29T00:25:07.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a46fde98c7490a8-0064c45c63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2023-07-29T00:25:07.455 DEBUG:boto:path=/hiiqos-58/obj00000000000000000015 2023-07-29T00:25:07.456 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000015 2023-07-29T00:25:07.456 DEBUG:boto:path=/hiiqos-58/obj00000000000000000015?versionId=null 2023-07-29T00:25:07.458 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000015?versionId=null 2023-07-29T00:25:07.460 DEBUG:boto:Method: GET 2023-07-29T00:25:07.460 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000015?versionId=null 2023-07-29T00:25:07.460 DEBUG:boto:Data: 2023-07-29T00:25:07.461 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.461 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:07.462 DEBUG:boto:Port: 8001 2023-07-29T00:25:07.462 DEBUG:boto:Params: {} 2023-07-29T00:25:07.462 DEBUG:boto:Token: None 2023-07-29T00:25:07.463 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000015?versionId=null 2023-07-29T00:25:07.463 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ykPlpJzC6VBk/ARhx5X/tuxlG3c= 2023-07-29T00:25:07.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ykPlpJzC6VBk/ARhx5X/tuxlG3c=', 'Content-Length': '0'} 2023-07-29T00:25:07.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002398a3e165ff3da-0064c45c63-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.508 DEBUG:boto:path=/hiiqos-58/obj00000000000000000015 2023-07-29T00:25:07.509 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000015 2023-07-29T00:25:07.510 DEBUG:boto:path=/hiiqos-58/obj00000000000000000015?versionId=null 2023-07-29T00:25:07.510 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000015?versionId=null 2023-07-29T00:25:07.510 DEBUG:boto:Method: GET 2023-07-29T00:25:07.511 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000015?versionId=null 2023-07-29T00:25:07.511 DEBUG:boto:Data: 2023-07-29T00:25:07.511 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.512 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:07.512 DEBUG:boto:Port: 8000 2023-07-29T00:25:07.513 DEBUG:boto:Params: {} 2023-07-29T00:25:07.513 DEBUG:boto:Token: None 2023-07-29T00:25:07.513 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000015?versionId=null 2023-07-29T00:25:07.514 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ykPlpJzC6VBk/ARhx5X/tuxlG3c= 2023-07-29T00:25:07.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ykPlpJzC6VBk/ARhx5X/tuxlG3c=', 'Content-Length': '0'} 2023-07-29T00:25:07.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008254baee882865ab-0064c45c63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.517 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:07.518 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:07.518 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000015 2023-07-29T00:25:07.518 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000015 2023-07-29T00:25:07.519 DEBUG:boto:Method: GET 2023-07-29T00:25:07.519 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000015 2023-07-29T00:25:07.519 DEBUG:boto:Data: 2023-07-29T00:25:07.520 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.520 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:07.520 DEBUG:boto:Port: 8001 2023-07-29T00:25:07.521 DEBUG:boto:Params: {} 2023-07-29T00:25:07.521 DEBUG:boto:Token: None 2023-07-29T00:25:07.522 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/ 2023-07-29T00:25:07.522 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lWrDRkW/GNAXJIbFCESeE57zgGQ= 2023-07-29T00:25:07.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lWrDRkW/GNAXJIbFCESeE57zgGQ=', 'Content-Length': '0'} 2023-07-29T00:25:07.525 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cd81e52cdff9901-0064c45c63-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.526 DEBUG:boto:b'hiiqos-58obj000000000000000000151000falseobj000000000000000000152023-07-29T00:24:59.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:07.526 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:07.527 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:07.527 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000015 2023-07-29T00:25:07.528 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000015 2023-07-29T00:25:07.528 DEBUG:boto:Method: GET 2023-07-29T00:25:07.528 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000015 2023-07-29T00:25:07.529 DEBUG:boto:Data: 2023-07-29T00:25:07.529 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.529 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:07.530 DEBUG:boto:Port: 8000 2023-07-29T00:25:07.530 DEBUG:boto:Params: {} 2023-07-29T00:25:07.530 DEBUG:boto:Token: None 2023-07-29T00:25:07.531 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/ 2023-07-29T00:25:07.531 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lWrDRkW/GNAXJIbFCESeE57zgGQ= 2023-07-29T00:25:07.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lWrDRkW/GNAXJIbFCESeE57zgGQ=', 'Content-Length': '0'} 2023-07-29T00:25:07.536 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000663bb95e9d20a1ec-0064c45c63-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.537 DEBUG:boto:b'hiiqos-58obj000000000000000000151000falseobj000000000000000000152023-07-29T00:24:59.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:07.538 DEBUG:boto:path=/hiiqos-58/obj00000000000000000015 2023-07-29T00:25:07.539 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000015 2023-07-29T00:25:07.539 DEBUG:boto:Method: HEAD 2023-07-29T00:25:07.539 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000015 2023-07-29T00:25:07.540 DEBUG:boto:Data: 2023-07-29T00:25:07.540 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.540 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:07.541 DEBUG:boto:Port: 8001 2023-07-29T00:25:07.541 DEBUG:boto:Params: {} 2023-07-29T00:25:07.542 DEBUG:boto:Token: None 2023-07-29T00:25:07.542 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000015 2023-07-29T00:25:07.543 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:F4xGhhtyomcDlGB5XYKFTmYWe2I= 2023-07-29T00:25:07.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:F4xGhhtyomcDlGB5XYKFTmYWe2I=', 'Content-Length': '0'} 2023-07-29T00:25:07.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033f59d5eb487d30e-0064c45c63-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.547 DEBUG:boto:path=/hiiqos-58/obj00000000000000000015 2023-07-29T00:25:07.547 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000015 2023-07-29T00:25:07.547 DEBUG:boto:Method: HEAD 2023-07-29T00:25:07.548 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000015 2023-07-29T00:25:07.548 DEBUG:boto:Data: 2023-07-29T00:25:07.548 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.549 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:07.549 DEBUG:boto:Port: 8000 2023-07-29T00:25:07.549 DEBUG:boto:Params: {} 2023-07-29T00:25:07.550 DEBUG:boto:Token: None 2023-07-29T00:25:07.550 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000015 2023-07-29T00:25:07.551 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:F4xGhhtyomcDlGB5XYKFTmYWe2I= 2023-07-29T00:25:07.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:F4xGhhtyomcDlGB5XYKFTmYWe2I=', 'Content-Length': '0'} 2023-07-29T00:25:07.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000290c42eca3cb0b43-0064c45c63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2023-07-29T00:25:07.554 DEBUG:boto:path=/hiiqos-58/obj00000000000000000015 2023-07-29T00:25:07.554 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000015 2023-07-29T00:25:07.555 DEBUG:boto:Method: GET 2023-07-29T00:25:07.556 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000015 2023-07-29T00:25:07.556 DEBUG:boto:Data: 2023-07-29T00:25:07.557 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.557 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:07.557 DEBUG:boto:Port: 8001 2023-07-29T00:25:07.558 DEBUG:boto:Params: {} 2023-07-29T00:25:07.558 DEBUG:boto:Token: None 2023-07-29T00:25:07.559 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000015 2023-07-29T00:25:07.559 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JdPjnDyFc+laVq7aU+8qNrw2ZX8= 2023-07-29T00:25:07.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JdPjnDyFc+laVq7aU+8qNrw2ZX8=', 'Content-Length': '0'} 2023-07-29T00:25:07.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c2aceba2c29a05d-0064c45c63-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.562 DEBUG:boto:path=/hiiqos-58/obj00000000000000000015 2023-07-29T00:25:07.563 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000015 2023-07-29T00:25:07.563 DEBUG:boto:Method: GET 2023-07-29T00:25:07.563 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000015 2023-07-29T00:25:07.564 DEBUG:boto:Data: 2023-07-29T00:25:07.564 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.564 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:07.565 DEBUG:boto:Port: 8000 2023-07-29T00:25:07.565 DEBUG:boto:Params: {} 2023-07-29T00:25:07.566 DEBUG:boto:Token: None 2023-07-29T00:25:07.566 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000015 2023-07-29T00:25:07.566 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JdPjnDyFc+laVq7aU+8qNrw2ZX8= 2023-07-29T00:25:07.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JdPjnDyFc+laVq7aU+8qNrw2ZX8=', 'Content-Length': '0'} 2023-07-29T00:25:07.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057c9fdcf3d45ed20-0064c45c63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.612 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:07.613 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:07.613 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000015 2023-07-29T00:25:07.614 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000015 2023-07-29T00:25:07.614 DEBUG:boto:Method: GET 2023-07-29T00:25:07.614 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000015 2023-07-29T00:25:07.615 DEBUG:boto:Data: 2023-07-29T00:25:07.615 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.615 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:07.616 DEBUG:boto:Port: 8001 2023-07-29T00:25:07.616 DEBUG:boto:Params: {} 2023-07-29T00:25:07.617 DEBUG:boto:Token: None 2023-07-29T00:25:07.617 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/ 2023-07-29T00:25:07.617 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lWrDRkW/GNAXJIbFCESeE57zgGQ= 2023-07-29T00:25:07.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lWrDRkW/GNAXJIbFCESeE57zgGQ=', 'Content-Length': '0'} 2023-07-29T00:25:07.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc3281b47efc4e07-0064c45c63-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.621 DEBUG:boto:b'hiiqos-58obj000000000000000000151000falseobj000000000000000000152023-07-29T00:24:59.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:07.622 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:07.622 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:07.623 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000015 2023-07-29T00:25:07.623 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000015 2023-07-29T00:25:07.623 DEBUG:boto:Method: GET 2023-07-29T00:25:07.624 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000015 2023-07-29T00:25:07.624 DEBUG:boto:Data: 2023-07-29T00:25:07.624 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.625 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:07.625 DEBUG:boto:Port: 8000 2023-07-29T00:25:07.625 DEBUG:boto:Params: {} 2023-07-29T00:25:07.626 DEBUG:boto:Token: None 2023-07-29T00:25:07.626 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/ 2023-07-29T00:25:07.627 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lWrDRkW/GNAXJIbFCESeE57zgGQ= 2023-07-29T00:25:07.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lWrDRkW/GNAXJIbFCESeE57zgGQ=', 'Content-Length': '0'} 2023-07-29T00:25:07.632 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092ed8c690786b39a-0064c45c63-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.632 DEBUG:boto:b'hiiqos-58obj000000000000000000151000falseobj000000000000000000152023-07-29T00:24:59.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:07.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2023-07-29T00:25:07.633 DEBUG:boto:path=/hiiqos-58/obj00000000000000000016 2023-07-29T00:25:07.634 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000016 2023-07-29T00:25:07.634 DEBUG:boto:path=/hiiqos-58/obj00000000000000000016?versionId=null 2023-07-29T00:25:07.635 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000016?versionId=null 2023-07-29T00:25:07.635 DEBUG:boto:Method: GET 2023-07-29T00:25:07.635 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000016?versionId=null 2023-07-29T00:25:07.636 DEBUG:boto:Data: 2023-07-29T00:25:07.636 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.636 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:07.637 DEBUG:boto:Port: 8001 2023-07-29T00:25:07.637 DEBUG:boto:Params: {} 2023-07-29T00:25:07.637 DEBUG:boto:Token: None 2023-07-29T00:25:07.638 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000016?versionId=null 2023-07-29T00:25:07.638 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:j/uWRxpC/wzRfqGCG6+mxODXtvA= 2023-07-29T00:25:07.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:j/uWRxpC/wzRfqGCG6+mxODXtvA=', 'Content-Length': '0'} 2023-07-29T00:25:07.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c35885a4c3fd27e9-0064c45c63-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.685 DEBUG:boto:path=/hiiqos-58/obj00000000000000000016 2023-07-29T00:25:07.686 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000016 2023-07-29T00:25:07.686 DEBUG:boto:path=/hiiqos-58/obj00000000000000000016?versionId=null 2023-07-29T00:25:07.688 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000016?versionId=null 2023-07-29T00:25:07.688 DEBUG:boto:Method: GET 2023-07-29T00:25:07.689 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000016?versionId=null 2023-07-29T00:25:07.689 DEBUG:boto:Data: 2023-07-29T00:25:07.689 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.691 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:07.691 DEBUG:boto:Port: 8000 2023-07-29T00:25:07.692 DEBUG:boto:Params: {} 2023-07-29T00:25:07.692 DEBUG:boto:Token: None 2023-07-29T00:25:07.693 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000016?versionId=null 2023-07-29T00:25:07.693 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:j/uWRxpC/wzRfqGCG6+mxODXtvA= 2023-07-29T00:25:07.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:j/uWRxpC/wzRfqGCG6+mxODXtvA=', 'Content-Length': '0'} 2023-07-29T00:25:07.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069fa613f42250e28-0064c45c63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.697 DEBUG:boto:path=/hiiqos-58/obj00000000000000000016 2023-07-29T00:25:07.702 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000016 2023-07-29T00:25:07.703 DEBUG:boto:path=/hiiqos-58/obj00000000000000000016?versionId=null 2023-07-29T00:25:07.703 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000016?versionId=null 2023-07-29T00:25:07.704 DEBUG:boto:Method: HEAD 2023-07-29T00:25:07.704 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000016?versionId=null 2023-07-29T00:25:07.704 DEBUG:boto:Data: 2023-07-29T00:25:07.705 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.705 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:07.705 DEBUG:boto:Port: 8001 2023-07-29T00:25:07.707 DEBUG:boto:Params: {} 2023-07-29T00:25:07.708 DEBUG:boto:Token: None 2023-07-29T00:25:07.708 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000016?versionId=null 2023-07-29T00:25:07.709 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+7n2PXehB8NYxmbUu4xTv47FGck= 2023-07-29T00:25:07.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+7n2PXehB8NYxmbUu4xTv47FGck=', 'Content-Length': '0'} 2023-07-29T00:25:07.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdfa9e6a4985ae0e-0064c45c63-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.713 DEBUG:boto:path=/hiiqos-58/obj00000000000000000016 2023-07-29T00:25:07.713 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000016 2023-07-29T00:25:07.714 DEBUG:boto:path=/hiiqos-58/obj00000000000000000016?versionId=null 2023-07-29T00:25:07.714 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000016?versionId=null 2023-07-29T00:25:07.714 DEBUG:boto:Method: HEAD 2023-07-29T00:25:07.715 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000016?versionId=null 2023-07-29T00:25:07.715 DEBUG:boto:Data: 2023-07-29T00:25:07.716 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.716 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:07.717 DEBUG:boto:Port: 8000 2023-07-29T00:25:07.717 DEBUG:boto:Params: {} 2023-07-29T00:25:07.718 DEBUG:boto:Token: None 2023-07-29T00:25:07.718 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000016?versionId=null 2023-07-29T00:25:07.719 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+7n2PXehB8NYxmbUu4xTv47FGck= 2023-07-29T00:25:07.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+7n2PXehB8NYxmbUu4xTv47FGck=', 'Content-Length': '0'} 2023-07-29T00:25:07.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004af9564d433dacc1-0064c45c63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2023-07-29T00:25:07.722 DEBUG:boto:path=/hiiqos-58/obj00000000000000000016 2023-07-29T00:25:07.723 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000016 2023-07-29T00:25:07.723 DEBUG:boto:path=/hiiqos-58/obj00000000000000000016?versionId=null 2023-07-29T00:25:07.724 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000016?versionId=null 2023-07-29T00:25:07.724 DEBUG:boto:Method: GET 2023-07-29T00:25:07.725 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000016?versionId=null 2023-07-29T00:25:07.725 DEBUG:boto:Data: 2023-07-29T00:25:07.726 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.726 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:07.726 DEBUG:boto:Port: 8001 2023-07-29T00:25:07.727 DEBUG:boto:Params: {} 2023-07-29T00:25:07.727 DEBUG:boto:Token: None 2023-07-29T00:25:07.728 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000016?versionId=null 2023-07-29T00:25:07.728 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:j/uWRxpC/wzRfqGCG6+mxODXtvA= 2023-07-29T00:25:07.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:j/uWRxpC/wzRfqGCG6+mxODXtvA=', 'Content-Length': '0'} 2023-07-29T00:25:07.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7f2ae093b38a9a7-0064c45c63-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.732 DEBUG:boto:path=/hiiqos-58/obj00000000000000000016 2023-07-29T00:25:07.732 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000016 2023-07-29T00:25:07.732 DEBUG:boto:path=/hiiqos-58/obj00000000000000000016?versionId=null 2023-07-29T00:25:07.733 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000016?versionId=null 2023-07-29T00:25:07.735 DEBUG:boto:Method: GET 2023-07-29T00:25:07.735 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000016?versionId=null 2023-07-29T00:25:07.735 DEBUG:boto:Data: 2023-07-29T00:25:07.736 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.736 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:07.737 DEBUG:boto:Port: 8000 2023-07-29T00:25:07.737 DEBUG:boto:Params: {} 2023-07-29T00:25:07.737 DEBUG:boto:Token: None 2023-07-29T00:25:07.738 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000016?versionId=null 2023-07-29T00:25:07.738 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:j/uWRxpC/wzRfqGCG6+mxODXtvA= 2023-07-29T00:25:07.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:j/uWRxpC/wzRfqGCG6+mxODXtvA=', 'Content-Length': '0'} 2023-07-29T00:25:07.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fd5e945aa35a922-0064c45c63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.742 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:07.742 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:07.742 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000016 2023-07-29T00:25:07.743 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000016 2023-07-29T00:25:07.743 DEBUG:boto:Method: GET 2023-07-29T00:25:07.743 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000016 2023-07-29T00:25:07.744 DEBUG:boto:Data: 2023-07-29T00:25:07.744 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.745 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:07.745 DEBUG:boto:Port: 8001 2023-07-29T00:25:07.745 DEBUG:boto:Params: {} 2023-07-29T00:25:07.746 DEBUG:boto:Token: None 2023-07-29T00:25:07.746 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/ 2023-07-29T00:25:07.747 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lWrDRkW/GNAXJIbFCESeE57zgGQ= 2023-07-29T00:25:07.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lWrDRkW/GNAXJIbFCESeE57zgGQ=', 'Content-Length': '0'} 2023-07-29T00:25:07.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb6be62603f95a21-0064c45c63-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.750 DEBUG:boto:b'hiiqos-58obj000000000000000000161000falseobj000000000000000000162023-07-29T00:24:59.056Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:07.751 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:07.751 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:07.752 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000016 2023-07-29T00:25:07.752 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000016 2023-07-29T00:25:07.752 DEBUG:boto:Method: GET 2023-07-29T00:25:07.753 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000016 2023-07-29T00:25:07.753 DEBUG:boto:Data: 2023-07-29T00:25:07.754 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.754 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:07.754 DEBUG:boto:Port: 8000 2023-07-29T00:25:07.755 DEBUG:boto:Params: {} 2023-07-29T00:25:07.755 DEBUG:boto:Token: None 2023-07-29T00:25:07.756 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/ 2023-07-29T00:25:07.756 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lWrDRkW/GNAXJIbFCESeE57zgGQ= 2023-07-29T00:25:07.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lWrDRkW/GNAXJIbFCESeE57zgGQ=', 'Content-Length': '0'} 2023-07-29T00:25:07.763 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052b2311516a14a7d-0064c45c63-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.808 DEBUG:boto:b'hiiqos-58obj000000000000000000161000falseobj000000000000000000162023-07-29T00:24:59.056Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:07.809 DEBUG:boto:path=/hiiqos-58/obj00000000000000000016 2023-07-29T00:25:07.810 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000016 2023-07-29T00:25:07.810 DEBUG:boto:Method: HEAD 2023-07-29T00:25:07.810 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000016 2023-07-29T00:25:07.811 DEBUG:boto:Data: 2023-07-29T00:25:07.811 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.812 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:07.813 DEBUG:boto:Port: 8001 2023-07-29T00:25:07.813 DEBUG:boto:Params: {} 2023-07-29T00:25:07.814 DEBUG:boto:Token: None 2023-07-29T00:25:07.814 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000016 2023-07-29T00:25:07.815 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aV3n1lHPSxA8QNkAH1E0Wq2oI1g= 2023-07-29T00:25:07.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aV3n1lHPSxA8QNkAH1E0Wq2oI1g=', 'Content-Length': '0'} 2023-07-29T00:25:07.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000433216590deb4c3f-0064c45c63-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.819 DEBUG:boto:path=/hiiqos-58/obj00000000000000000016 2023-07-29T00:25:07.820 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000016 2023-07-29T00:25:07.820 DEBUG:boto:Method: HEAD 2023-07-29T00:25:07.820 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000016 2023-07-29T00:25:07.821 DEBUG:boto:Data: 2023-07-29T00:25:07.821 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.821 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:07.822 DEBUG:boto:Port: 8000 2023-07-29T00:25:07.822 DEBUG:boto:Params: {} 2023-07-29T00:25:07.822 DEBUG:boto:Token: None 2023-07-29T00:25:07.823 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000016 2023-07-29T00:25:07.823 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aV3n1lHPSxA8QNkAH1E0Wq2oI1g= 2023-07-29T00:25:07.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aV3n1lHPSxA8QNkAH1E0Wq2oI1g=', 'Content-Length': '0'} 2023-07-29T00:25:07.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a73936ec9dbce76-0064c45c63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2023-07-29T00:25:07.827 DEBUG:boto:path=/hiiqos-58/obj00000000000000000016 2023-07-29T00:25:07.827 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000016 2023-07-29T00:25:07.827 DEBUG:boto:Method: GET 2023-07-29T00:25:07.828 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000016 2023-07-29T00:25:07.828 DEBUG:boto:Data: 2023-07-29T00:25:07.829 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.829 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:07.829 DEBUG:boto:Port: 8001 2023-07-29T00:25:07.830 DEBUG:boto:Params: {} 2023-07-29T00:25:07.830 DEBUG:boto:Token: None 2023-07-29T00:25:07.830 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000016 2023-07-29T00:25:07.831 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:b1bwR9X4eagC5imRrgiGjsnnrfY= 2023-07-29T00:25:07.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:b1bwR9X4eagC5imRrgiGjsnnrfY=', 'Content-Length': '0'} 2023-07-29T00:25:07.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fd9b75f7dc53203-0064c45c63-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.835 DEBUG:boto:path=/hiiqos-58/obj00000000000000000016 2023-07-29T00:25:07.836 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000016 2023-07-29T00:25:07.836 DEBUG:boto:Method: GET 2023-07-29T00:25:07.837 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000016 2023-07-29T00:25:07.837 DEBUG:boto:Data: 2023-07-29T00:25:07.837 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.838 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:07.838 DEBUG:boto:Port: 8000 2023-07-29T00:25:07.839 DEBUG:boto:Params: {} 2023-07-29T00:25:07.839 DEBUG:boto:Token: None 2023-07-29T00:25:07.840 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000016 2023-07-29T00:25:07.840 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:b1bwR9X4eagC5imRrgiGjsnnrfY= 2023-07-29T00:25:07.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:b1bwR9X4eagC5imRrgiGjsnnrfY=', 'Content-Length': '0'} 2023-07-29T00:25:07.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030d6848b63d66837-0064c45c63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.843 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:07.844 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:07.844 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000016 2023-07-29T00:25:07.845 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000016 2023-07-29T00:25:07.845 DEBUG:boto:Method: GET 2023-07-29T00:25:07.845 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000016 2023-07-29T00:25:07.846 DEBUG:boto:Data: 2023-07-29T00:25:07.846 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.846 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:07.847 DEBUG:boto:Port: 8001 2023-07-29T00:25:07.847 DEBUG:boto:Params: {} 2023-07-29T00:25:07.847 DEBUG:boto:Token: None 2023-07-29T00:25:07.848 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/ 2023-07-29T00:25:07.848 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lWrDRkW/GNAXJIbFCESeE57zgGQ= 2023-07-29T00:25:07.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lWrDRkW/GNAXJIbFCESeE57zgGQ=', 'Content-Length': '0'} 2023-07-29T00:25:07.851 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b7aebc95832afc5-0064c45c63-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.852 DEBUG:boto:b'hiiqos-58obj000000000000000000161000falseobj000000000000000000162023-07-29T00:24:59.056Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:07.853 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:07.853 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:07.854 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000016 2023-07-29T00:25:07.854 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000016 2023-07-29T00:25:07.854 DEBUG:boto:Method: GET 2023-07-29T00:25:07.855 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000016 2023-07-29T00:25:07.855 DEBUG:boto:Data: 2023-07-29T00:25:07.855 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.856 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:07.856 DEBUG:boto:Port: 8000 2023-07-29T00:25:07.857 DEBUG:boto:Params: {} 2023-07-29T00:25:07.857 DEBUG:boto:Token: None 2023-07-29T00:25:07.857 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/ 2023-07-29T00:25:07.858 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lWrDRkW/GNAXJIbFCESeE57zgGQ= 2023-07-29T00:25:07.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lWrDRkW/GNAXJIbFCESeE57zgGQ=', 'Content-Length': '0'} 2023-07-29T00:25:07.864 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091c25a58166b867d-0064c45c63-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.865 DEBUG:boto:b'hiiqos-58obj000000000000000000161000falseobj000000000000000000162023-07-29T00:24:59.056Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:07.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2023-07-29T00:25:07.866 DEBUG:boto:path=/hiiqos-58/obj00000000000000000017 2023-07-29T00:25:07.867 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000017 2023-07-29T00:25:07.867 DEBUG:boto:path=/hiiqos-58/obj00000000000000000017?versionId=null 2023-07-29T00:25:07.867 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000017?versionId=null 2023-07-29T00:25:07.868 DEBUG:boto:Method: GET 2023-07-29T00:25:07.868 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000017?versionId=null 2023-07-29T00:25:07.869 DEBUG:boto:Data: 2023-07-29T00:25:07.869 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.870 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:07.870 DEBUG:boto:Port: 8001 2023-07-29T00:25:07.870 DEBUG:boto:Params: {} 2023-07-29T00:25:07.871 DEBUG:boto:Token: None 2023-07-29T00:25:07.873 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000017?versionId=null 2023-07-29T00:25:07.873 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GDbTf2EeGqrsPYPU/T22EmtFPFs= 2023-07-29T00:25:07.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GDbTf2EeGqrsPYPU/T22EmtFPFs=', 'Content-Length': '0'} 2023-07-29T00:25:07.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4ac50a9c719ca30-0064c45c63-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.877 DEBUG:boto:path=/hiiqos-58/obj00000000000000000017 2023-07-29T00:25:07.877 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000017 2023-07-29T00:25:07.878 DEBUG:boto:path=/hiiqos-58/obj00000000000000000017?versionId=null 2023-07-29T00:25:07.878 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000017?versionId=null 2023-07-29T00:25:07.879 DEBUG:boto:Method: GET 2023-07-29T00:25:07.879 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000017?versionId=null 2023-07-29T00:25:07.879 DEBUG:boto:Data: 2023-07-29T00:25:07.880 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.880 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:07.880 DEBUG:boto:Port: 8000 2023-07-29T00:25:07.881 DEBUG:boto:Params: {} 2023-07-29T00:25:07.881 DEBUG:boto:Token: None 2023-07-29T00:25:07.882 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000017?versionId=null 2023-07-29T00:25:07.882 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GDbTf2EeGqrsPYPU/T22EmtFPFs= 2023-07-29T00:25:07.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GDbTf2EeGqrsPYPU/T22EmtFPFs=', 'Content-Length': '0'} 2023-07-29T00:25:07.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cce068480e5780f-0064c45c63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.885 DEBUG:boto:path=/hiiqos-58/obj00000000000000000017 2023-07-29T00:25:07.885 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000017 2023-07-29T00:25:07.886 DEBUG:boto:path=/hiiqos-58/obj00000000000000000017?versionId=null 2023-07-29T00:25:07.886 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000017?versionId=null 2023-07-29T00:25:07.887 DEBUG:boto:Method: HEAD 2023-07-29T00:25:07.887 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000017?versionId=null 2023-07-29T00:25:07.887 DEBUG:boto:Data: 2023-07-29T00:25:07.888 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.888 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:07.888 DEBUG:boto:Port: 8001 2023-07-29T00:25:07.889 DEBUG:boto:Params: {} 2023-07-29T00:25:07.889 DEBUG:boto:Token: None 2023-07-29T00:25:07.890 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000017?versionId=null 2023-07-29T00:25:07.890 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TbtlPuwZejYTuxSfmsAxxOA4MFg= 2023-07-29T00:25:07.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TbtlPuwZejYTuxSfmsAxxOA4MFg=', 'Content-Length': '0'} 2023-07-29T00:25:07.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aee716db4f81ae8b-0064c45c63-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.893 DEBUG:boto:path=/hiiqos-58/obj00000000000000000017 2023-07-29T00:25:07.894 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000017 2023-07-29T00:25:07.894 DEBUG:boto:path=/hiiqos-58/obj00000000000000000017?versionId=null 2023-07-29T00:25:07.894 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000017?versionId=null 2023-07-29T00:25:07.895 DEBUG:boto:Method: HEAD 2023-07-29T00:25:07.895 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000017?versionId=null 2023-07-29T00:25:07.895 DEBUG:boto:Data: 2023-07-29T00:25:07.896 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.896 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:07.897 DEBUG:boto:Port: 8000 2023-07-29T00:25:07.897 DEBUG:boto:Params: {} 2023-07-29T00:25:07.897 DEBUG:boto:Token: None 2023-07-29T00:25:07.898 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000017?versionId=null 2023-07-29T00:25:07.898 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TbtlPuwZejYTuxSfmsAxxOA4MFg= 2023-07-29T00:25:07.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TbtlPuwZejYTuxSfmsAxxOA4MFg=', 'Content-Length': '0'} 2023-07-29T00:25:07.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000954b69153e94c565-0064c45c63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2023-07-29T00:25:07.902 DEBUG:boto:path=/hiiqos-58/obj00000000000000000017 2023-07-29T00:25:07.902 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000017 2023-07-29T00:25:07.903 DEBUG:boto:path=/hiiqos-58/obj00000000000000000017?versionId=null 2023-07-29T00:25:07.903 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000017?versionId=null 2023-07-29T00:25:07.903 DEBUG:boto:Method: GET 2023-07-29T00:25:07.904 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000017?versionId=null 2023-07-29T00:25:07.904 DEBUG:boto:Data: 2023-07-29T00:25:07.904 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.905 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:07.905 DEBUG:boto:Port: 8001 2023-07-29T00:25:07.906 DEBUG:boto:Params: {} 2023-07-29T00:25:07.906 DEBUG:boto:Token: None 2023-07-29T00:25:07.906 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000017?versionId=null 2023-07-29T00:25:07.907 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GDbTf2EeGqrsPYPU/T22EmtFPFs= 2023-07-29T00:25:07.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GDbTf2EeGqrsPYPU/T22EmtFPFs=', 'Content-Length': '0'} 2023-07-29T00:25:07.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b3520a78d1391a9-0064c45c63-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.910 DEBUG:boto:path=/hiiqos-58/obj00000000000000000017 2023-07-29T00:25:07.911 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000017 2023-07-29T00:25:07.911 DEBUG:boto:path=/hiiqos-58/obj00000000000000000017?versionId=null 2023-07-29T00:25:07.911 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000017?versionId=null 2023-07-29T00:25:07.912 DEBUG:boto:Method: GET 2023-07-29T00:25:07.912 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000017?versionId=null 2023-07-29T00:25:07.912 DEBUG:boto:Data: 2023-07-29T00:25:07.913 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.913 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:07.913 DEBUG:boto:Port: 8000 2023-07-29T00:25:07.914 DEBUG:boto:Params: {} 2023-07-29T00:25:07.914 DEBUG:boto:Token: None 2023-07-29T00:25:07.915 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000017?versionId=null 2023-07-29T00:25:07.915 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GDbTf2EeGqrsPYPU/T22EmtFPFs= 2023-07-29T00:25:07.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GDbTf2EeGqrsPYPU/T22EmtFPFs=', 'Content-Length': '0'} 2023-07-29T00:25:07.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051506b60283283c0-0064c45c63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.960 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:07.961 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:07.962 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000017 2023-07-29T00:25:07.962 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000017 2023-07-29T00:25:07.962 DEBUG:boto:Method: GET 2023-07-29T00:25:07.963 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000017 2023-07-29T00:25:07.963 DEBUG:boto:Data: 2023-07-29T00:25:07.963 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.964 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:07.964 DEBUG:boto:Port: 8001 2023-07-29T00:25:07.964 DEBUG:boto:Params: {} 2023-07-29T00:25:07.965 DEBUG:boto:Token: None 2023-07-29T00:25:07.965 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/ 2023-07-29T00:25:07.966 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lWrDRkW/GNAXJIbFCESeE57zgGQ= 2023-07-29T00:25:07.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lWrDRkW/GNAXJIbFCESeE57zgGQ=', 'Content-Length': '0'} 2023-07-29T00:25:07.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006e8f3e634257b38-0064c45c63-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.970 DEBUG:boto:b'hiiqos-58obj000000000000000000171000falseobj000000000000000000172023-07-29T00:25:00.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:07.970 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:07.971 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:07.971 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000017 2023-07-29T00:25:07.972 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000017 2023-07-29T00:25:07.972 DEBUG:boto:Method: GET 2023-07-29T00:25:07.973 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000017 2023-07-29T00:25:07.973 DEBUG:boto:Data: 2023-07-29T00:25:07.973 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.974 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:07.974 DEBUG:boto:Port: 8000 2023-07-29T00:25:07.974 DEBUG:boto:Params: {} 2023-07-29T00:25:07.975 DEBUG:boto:Token: None 2023-07-29T00:25:07.975 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/ 2023-07-29T00:25:07.976 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lWrDRkW/GNAXJIbFCESeE57zgGQ= 2023-07-29T00:25:07.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lWrDRkW/GNAXJIbFCESeE57zgGQ=', 'Content-Length': '0'} 2023-07-29T00:25:07.980 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002ef8850ebfedd6d-0064c45c63-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.980 DEBUG:boto:b'hiiqos-58obj000000000000000000171000falseobj000000000000000000172023-07-29T00:25:00.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:07.981 DEBUG:boto:path=/hiiqos-58/obj00000000000000000017 2023-07-29T00:25:07.981 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000017 2023-07-29T00:25:07.982 DEBUG:boto:Method: HEAD 2023-07-29T00:25:07.982 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000017 2023-07-29T00:25:07.983 DEBUG:boto:Data: 2023-07-29T00:25:07.983 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.983 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:07.984 DEBUG:boto:Port: 8001 2023-07-29T00:25:07.984 DEBUG:boto:Params: {} 2023-07-29T00:25:07.984 DEBUG:boto:Token: None 2023-07-29T00:25:07.985 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000017 2023-07-29T00:25:07.985 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cc87u4aDWhp0iLY8ihq0ogBtN7o= 2023-07-29T00:25:07.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cc87u4aDWhp0iLY8ihq0ogBtN7o=', 'Content-Length': '0'} 2023-07-29T00:25:07.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bba4ba3988ec0ec8-0064c45c63-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.988 DEBUG:boto:path=/hiiqos-58/obj00000000000000000017 2023-07-29T00:25:07.989 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000017 2023-07-29T00:25:07.989 DEBUG:boto:Method: HEAD 2023-07-29T00:25:07.989 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000017 2023-07-29T00:25:07.990 DEBUG:boto:Data: 2023-07-29T00:25:07.990 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.990 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:07.991 DEBUG:boto:Port: 8000 2023-07-29T00:25:07.991 DEBUG:boto:Params: {} 2023-07-29T00:25:07.992 DEBUG:boto:Token: None 2023-07-29T00:25:07.992 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000017 2023-07-29T00:25:07.992 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cc87u4aDWhp0iLY8ihq0ogBtN7o= 2023-07-29T00:25:07.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cc87u4aDWhp0iLY8ihq0ogBtN7o=', 'Content-Length': '0'} 2023-07-29T00:25:07.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb10fc72e41b3306-0064c45c63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:07.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2023-07-29T00:25:07.995 DEBUG:boto:path=/hiiqos-58/obj00000000000000000017 2023-07-29T00:25:07.996 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000017 2023-07-29T00:25:07.996 DEBUG:boto:Method: GET 2023-07-29T00:25:07.997 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000017 2023-07-29T00:25:07.997 DEBUG:boto:Data: 2023-07-29T00:25:07.997 DEBUG:boto:Headers: {} 2023-07-29T00:25:07.998 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:07.998 DEBUG:boto:Port: 8001 2023-07-29T00:25:07.998 DEBUG:boto:Params: {} 2023-07-29T00:25:07.999 DEBUG:boto:Token: None 2023-07-29T00:25:07.999 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:07 GMT /hiiqos-58/obj00000000000000000017 2023-07-29T00:25:08.000 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jZXh99iMQahYjeDiSIa5R4uVsco= 2023-07-29T00:25:08.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jZXh99iMQahYjeDiSIa5R4uVsco=', 'Content-Length': '0'} 2023-07-29T00:25:08.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000741cefe25f1ffe51-0064c45c63-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.044 DEBUG:boto:path=/hiiqos-58/obj00000000000000000017 2023-07-29T00:25:08.045 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000017 2023-07-29T00:25:08.046 DEBUG:boto:Method: GET 2023-07-29T00:25:08.046 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000017 2023-07-29T00:25:08.046 DEBUG:boto:Data: 2023-07-29T00:25:08.047 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.047 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:08.047 DEBUG:boto:Port: 8000 2023-07-29T00:25:08.048 DEBUG:boto:Params: {} 2023-07-29T00:25:08.048 DEBUG:boto:Token: None 2023-07-29T00:25:08.049 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000017 2023-07-29T00:25:08.049 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:E4SEe62TbwSFNfd+zVXNd3so79w= 2023-07-29T00:25:08.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:E4SEe62TbwSFNfd+zVXNd3so79w=', 'Content-Length': '0'} 2023-07-29T00:25:08.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010e73b2b361e6e6a-0064c45c63-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.052 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:08.053 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:08.053 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000017 2023-07-29T00:25:08.054 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000017 2023-07-29T00:25:08.054 DEBUG:boto:Method: GET 2023-07-29T00:25:08.054 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000017 2023-07-29T00:25:08.055 DEBUG:boto:Data: 2023-07-29T00:25:08.055 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.055 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:08.056 DEBUG:boto:Port: 8001 2023-07-29T00:25:08.056 DEBUG:boto:Params: {} 2023-07-29T00:25:08.057 DEBUG:boto:Token: None 2023-07-29T00:25:08.057 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/ 2023-07-29T00:25:08.057 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qGAMiLgaCAmNrEigoEpXqCHju0Q= 2023-07-29T00:25:08.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qGAMiLgaCAmNrEigoEpXqCHju0Q=', 'Content-Length': '0'} 2023-07-29T00:25:08.061 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fd6a46d4a315b86-0064c45c64-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.061 DEBUG:boto:b'hiiqos-58obj000000000000000000171000falseobj000000000000000000172023-07-29T00:25:00.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:08.064 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:08.064 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:08.064 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000017 2023-07-29T00:25:08.065 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000017 2023-07-29T00:25:08.065 DEBUG:boto:Method: GET 2023-07-29T00:25:08.066 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000017 2023-07-29T00:25:08.066 DEBUG:boto:Data: 2023-07-29T00:25:08.066 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.067 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:08.067 DEBUG:boto:Port: 8000 2023-07-29T00:25:08.068 DEBUG:boto:Params: {} 2023-07-29T00:25:08.068 DEBUG:boto:Token: None 2023-07-29T00:25:08.069 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/ 2023-07-29T00:25:08.069 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qGAMiLgaCAmNrEigoEpXqCHju0Q= 2023-07-29T00:25:08.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qGAMiLgaCAmNrEigoEpXqCHju0Q=', 'Content-Length': '0'} 2023-07-29T00:25:08.074 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001aa9c89c96bcb162-0064c45c64-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.074 DEBUG:boto:b'hiiqos-58obj000000000000000000171000falseobj000000000000000000172023-07-29T00:25:00.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:08.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2023-07-29T00:25:08.076 DEBUG:boto:path=/hiiqos-58/obj00000000000000000018 2023-07-29T00:25:08.076 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000018 2023-07-29T00:25:08.076 DEBUG:boto:path=/hiiqos-58/obj00000000000000000018?versionId=null 2023-07-29T00:25:08.077 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000018?versionId=null 2023-07-29T00:25:08.077 DEBUG:boto:Method: GET 2023-07-29T00:25:08.077 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000018?versionId=null 2023-07-29T00:25:08.078 DEBUG:boto:Data: 2023-07-29T00:25:08.078 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.079 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:08.079 DEBUG:boto:Port: 8001 2023-07-29T00:25:08.079 DEBUG:boto:Params: {} 2023-07-29T00:25:08.080 DEBUG:boto:Token: None 2023-07-29T00:25:08.080 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000018?versionId=null 2023-07-29T00:25:08.081 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:71ZXZagWk39AQQsbsN1GX+2AN+w= 2023-07-29T00:25:08.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:71ZXZagWk39AQQsbsN1GX+2AN+w=', 'Content-Length': '0'} 2023-07-29T00:25:08.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9852f7387cb417e-0064c45c64-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.084 DEBUG:boto:path=/hiiqos-58/obj00000000000000000018 2023-07-29T00:25:08.084 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000018 2023-07-29T00:25:08.085 DEBUG:boto:path=/hiiqos-58/obj00000000000000000018?versionId=null 2023-07-29T00:25:08.085 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000018?versionId=null 2023-07-29T00:25:08.086 DEBUG:boto:Method: GET 2023-07-29T00:25:08.086 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000018?versionId=null 2023-07-29T00:25:08.086 DEBUG:boto:Data: 2023-07-29T00:25:08.087 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.087 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:08.087 DEBUG:boto:Port: 8000 2023-07-29T00:25:08.088 DEBUG:boto:Params: {} 2023-07-29T00:25:08.088 DEBUG:boto:Token: None 2023-07-29T00:25:08.089 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000018?versionId=null 2023-07-29T00:25:08.089 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:71ZXZagWk39AQQsbsN1GX+2AN+w= 2023-07-29T00:25:08.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:71ZXZagWk39AQQsbsN1GX+2AN+w=', 'Content-Length': '0'} 2023-07-29T00:25:08.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7f46f87a1bd4d22-0064c45c64-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.092 DEBUG:boto:path=/hiiqos-58/obj00000000000000000018 2023-07-29T00:25:08.093 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000018 2023-07-29T00:25:08.093 DEBUG:boto:path=/hiiqos-58/obj00000000000000000018?versionId=null 2023-07-29T00:25:08.093 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000018?versionId=null 2023-07-29T00:25:08.094 DEBUG:boto:Method: HEAD 2023-07-29T00:25:08.094 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000018?versionId=null 2023-07-29T00:25:08.094 DEBUG:boto:Data: 2023-07-29T00:25:08.095 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.095 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:08.096 DEBUG:boto:Port: 8001 2023-07-29T00:25:08.096 DEBUG:boto:Params: {} 2023-07-29T00:25:08.096 DEBUG:boto:Token: None 2023-07-29T00:25:08.097 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000018?versionId=null 2023-07-29T00:25:08.097 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fGY2FC1sAHwbGF9o6Qk+pI1s8XM= 2023-07-29T00:25:08.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fGY2FC1sAHwbGF9o6Qk+pI1s8XM=', 'Content-Length': '0'} 2023-07-29T00:25:08.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ed0984ca63cd51c-0064c45c64-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.100 DEBUG:boto:path=/hiiqos-58/obj00000000000000000018 2023-07-29T00:25:08.100 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000018 2023-07-29T00:25:08.101 DEBUG:boto:path=/hiiqos-58/obj00000000000000000018?versionId=null 2023-07-29T00:25:08.101 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000018?versionId=null 2023-07-29T00:25:08.101 DEBUG:boto:Method: HEAD 2023-07-29T00:25:08.102 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000018?versionId=null 2023-07-29T00:25:08.102 DEBUG:boto:Data: 2023-07-29T00:25:08.102 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.103 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:08.103 DEBUG:boto:Port: 8000 2023-07-29T00:25:08.103 DEBUG:boto:Params: {} 2023-07-29T00:25:08.104 DEBUG:boto:Token: None 2023-07-29T00:25:08.104 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000018?versionId=null 2023-07-29T00:25:08.104 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fGY2FC1sAHwbGF9o6Qk+pI1s8XM= 2023-07-29T00:25:08.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fGY2FC1sAHwbGF9o6Qk+pI1s8XM=', 'Content-Length': '0'} 2023-07-29T00:25:08.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdf441d0e64bc972-0064c45c64-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2023-07-29T00:25:08.108 DEBUG:boto:path=/hiiqos-58/obj00000000000000000018 2023-07-29T00:25:08.108 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000018 2023-07-29T00:25:08.108 DEBUG:boto:path=/hiiqos-58/obj00000000000000000018?versionId=null 2023-07-29T00:25:08.109 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000018?versionId=null 2023-07-29T00:25:08.109 DEBUG:boto:Method: GET 2023-07-29T00:25:08.110 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000018?versionId=null 2023-07-29T00:25:08.110 DEBUG:boto:Data: 2023-07-29T00:25:08.110 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.111 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:08.111 DEBUG:boto:Port: 8001 2023-07-29T00:25:08.111 DEBUG:boto:Params: {} 2023-07-29T00:25:08.112 DEBUG:boto:Token: None 2023-07-29T00:25:08.112 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000018?versionId=null 2023-07-29T00:25:08.113 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:71ZXZagWk39AQQsbsN1GX+2AN+w= 2023-07-29T00:25:08.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:71ZXZagWk39AQQsbsN1GX+2AN+w=', 'Content-Length': '0'} 2023-07-29T00:25:08.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c5483740fb30c4c-0064c45c64-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.116 DEBUG:boto:path=/hiiqos-58/obj00000000000000000018 2023-07-29T00:25:08.116 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000018 2023-07-29T00:25:08.116 DEBUG:boto:path=/hiiqos-58/obj00000000000000000018?versionId=null 2023-07-29T00:25:08.117 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000018?versionId=null 2023-07-29T00:25:08.117 DEBUG:boto:Method: GET 2023-07-29T00:25:08.117 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000018?versionId=null 2023-07-29T00:25:08.118 DEBUG:boto:Data: 2023-07-29T00:25:08.118 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.118 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:08.119 DEBUG:boto:Port: 8000 2023-07-29T00:25:08.119 DEBUG:boto:Params: {} 2023-07-29T00:25:08.120 DEBUG:boto:Token: None 2023-07-29T00:25:08.120 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000018?versionId=null 2023-07-29T00:25:08.120 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:71ZXZagWk39AQQsbsN1GX+2AN+w= 2023-07-29T00:25:08.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:71ZXZagWk39AQQsbsN1GX+2AN+w=', 'Content-Length': '0'} 2023-07-29T00:25:08.123 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004a57cae86a39c5b-0064c45c64-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.123 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:08.124 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:08.124 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000018 2023-07-29T00:25:08.125 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000018 2023-07-29T00:25:08.125 DEBUG:boto:Method: GET 2023-07-29T00:25:08.125 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000018 2023-07-29T00:25:08.126 DEBUG:boto:Data: 2023-07-29T00:25:08.126 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.126 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:08.127 DEBUG:boto:Port: 8001 2023-07-29T00:25:08.127 DEBUG:boto:Params: {} 2023-07-29T00:25:08.127 DEBUG:boto:Token: None 2023-07-29T00:25:08.128 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/ 2023-07-29T00:25:08.128 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qGAMiLgaCAmNrEigoEpXqCHju0Q= 2023-07-29T00:25:08.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qGAMiLgaCAmNrEigoEpXqCHju0Q=', 'Content-Length': '0'} 2023-07-29T00:25:08.132 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000491a94913bb6767e-0064c45c64-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.132 DEBUG:boto:b'hiiqos-58obj000000000000000000181000falseobj000000000000000000182023-07-29T00:24:58.540Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:08.133 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:08.133 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:08.134 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000018 2023-07-29T00:25:08.134 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000018 2023-07-29T00:25:08.134 DEBUG:boto:Method: GET 2023-07-29T00:25:08.135 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000018 2023-07-29T00:25:08.135 DEBUG:boto:Data: 2023-07-29T00:25:08.136 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.136 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:08.136 DEBUG:boto:Port: 8000 2023-07-29T00:25:08.137 DEBUG:boto:Params: {} 2023-07-29T00:25:08.137 DEBUG:boto:Token: None 2023-07-29T00:25:08.138 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/ 2023-07-29T00:25:08.138 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qGAMiLgaCAmNrEigoEpXqCHju0Q= 2023-07-29T00:25:08.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qGAMiLgaCAmNrEigoEpXqCHju0Q=', 'Content-Length': '0'} 2023-07-29T00:25:08.143 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e4b9bfa12adbda4-0064c45c64-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.144 DEBUG:boto:b'hiiqos-58obj000000000000000000181000falseobj000000000000000000182023-07-29T00:24:58.540Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:08.145 DEBUG:boto:path=/hiiqos-58/obj00000000000000000018 2023-07-29T00:25:08.145 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000018 2023-07-29T00:25:08.145 DEBUG:boto:Method: HEAD 2023-07-29T00:25:08.146 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000018 2023-07-29T00:25:08.146 DEBUG:boto:Data: 2023-07-29T00:25:08.147 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.147 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:08.148 DEBUG:boto:Port: 8001 2023-07-29T00:25:08.148 DEBUG:boto:Params: {} 2023-07-29T00:25:08.148 DEBUG:boto:Token: None 2023-07-29T00:25:08.149 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000018 2023-07-29T00:25:08.149 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ELbg5Lpvjg/wVwjcrSYfLzHMXTs= 2023-07-29T00:25:08.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ELbg5Lpvjg/wVwjcrSYfLzHMXTs=', 'Content-Length': '0'} 2023-07-29T00:25:08.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000630170435ade6a29-0064c45c64-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.152 DEBUG:boto:path=/hiiqos-58/obj00000000000000000018 2023-07-29T00:25:08.152 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000018 2023-07-29T00:25:08.153 DEBUG:boto:Method: HEAD 2023-07-29T00:25:08.153 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000018 2023-07-29T00:25:08.154 DEBUG:boto:Data: 2023-07-29T00:25:08.154 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.154 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:08.155 DEBUG:boto:Port: 8000 2023-07-29T00:25:08.155 DEBUG:boto:Params: {} 2023-07-29T00:25:08.155 DEBUG:boto:Token: None 2023-07-29T00:25:08.156 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000018 2023-07-29T00:25:08.156 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ELbg5Lpvjg/wVwjcrSYfLzHMXTs= 2023-07-29T00:25:08.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ELbg5Lpvjg/wVwjcrSYfLzHMXTs=', 'Content-Length': '0'} 2023-07-29T00:25:08.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2550cc1121f36e6-0064c45c64-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2023-07-29T00:25:08.160 DEBUG:boto:path=/hiiqos-58/obj00000000000000000018 2023-07-29T00:25:08.160 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000018 2023-07-29T00:25:08.160 DEBUG:boto:Method: GET 2023-07-29T00:25:08.161 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000018 2023-07-29T00:25:08.161 DEBUG:boto:Data: 2023-07-29T00:25:08.162 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.162 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:08.162 DEBUG:boto:Port: 8001 2023-07-29T00:25:08.163 DEBUG:boto:Params: {} 2023-07-29T00:25:08.163 DEBUG:boto:Token: None 2023-07-29T00:25:08.164 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000018 2023-07-29T00:25:08.164 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KFW1aVrnRwMvOXRhKaEc533KX3Q= 2023-07-29T00:25:08.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KFW1aVrnRwMvOXRhKaEc533KX3Q=', 'Content-Length': '0'} 2023-07-29T00:25:08.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083e00a6444167fb7-0064c45c64-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.167 DEBUG:boto:path=/hiiqos-58/obj00000000000000000018 2023-07-29T00:25:08.167 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000018 2023-07-29T00:25:08.168 DEBUG:boto:Method: GET 2023-07-29T00:25:08.168 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000018 2023-07-29T00:25:08.169 DEBUG:boto:Data: 2023-07-29T00:25:08.169 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.169 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:08.170 DEBUG:boto:Port: 8000 2023-07-29T00:25:08.170 DEBUG:boto:Params: {} 2023-07-29T00:25:08.170 DEBUG:boto:Token: None 2023-07-29T00:25:08.171 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000018 2023-07-29T00:25:08.171 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KFW1aVrnRwMvOXRhKaEc533KX3Q= 2023-07-29T00:25:08.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KFW1aVrnRwMvOXRhKaEc533KX3Q=', 'Content-Length': '0'} 2023-07-29T00:25:08.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c33c6f9cdc98b17c-0064c45c64-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.216 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:08.217 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:08.218 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000018 2023-07-29T00:25:08.218 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000018 2023-07-29T00:25:08.218 DEBUG:boto:Method: GET 2023-07-29T00:25:08.219 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000018 2023-07-29T00:25:08.219 DEBUG:boto:Data: 2023-07-29T00:25:08.219 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.220 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:08.220 DEBUG:boto:Port: 8001 2023-07-29T00:25:08.220 DEBUG:boto:Params: {} 2023-07-29T00:25:08.221 DEBUG:boto:Token: None 2023-07-29T00:25:08.221 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/ 2023-07-29T00:25:08.222 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qGAMiLgaCAmNrEigoEpXqCHju0Q= 2023-07-29T00:25:08.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qGAMiLgaCAmNrEigoEpXqCHju0Q=', 'Content-Length': '0'} 2023-07-29T00:25:08.225 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008aaa22e1a509d375-0064c45c64-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.225 DEBUG:boto:b'hiiqos-58obj000000000000000000181000falseobj000000000000000000182023-07-29T00:24:58.540Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:08.226 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:08.227 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:08.227 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000018 2023-07-29T00:25:08.227 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000018 2023-07-29T00:25:08.228 DEBUG:boto:Method: GET 2023-07-29T00:25:08.228 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000018 2023-07-29T00:25:08.229 DEBUG:boto:Data: 2023-07-29T00:25:08.229 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.229 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:08.230 DEBUG:boto:Port: 8000 2023-07-29T00:25:08.230 DEBUG:boto:Params: {} 2023-07-29T00:25:08.231 DEBUG:boto:Token: None 2023-07-29T00:25:08.231 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/ 2023-07-29T00:25:08.232 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qGAMiLgaCAmNrEigoEpXqCHju0Q= 2023-07-29T00:25:08.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qGAMiLgaCAmNrEigoEpXqCHju0Q=', 'Content-Length': '0'} 2023-07-29T00:25:08.236 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064148e2c72b38d43-0064c45c64-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.237 DEBUG:boto:b'hiiqos-58obj000000000000000000181000falseobj000000000000000000182023-07-29T00:24:58.540Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:08.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2023-07-29T00:25:08.238 DEBUG:boto:path=/hiiqos-58/obj00000000000000000019 2023-07-29T00:25:08.239 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000019 2023-07-29T00:25:08.239 DEBUG:boto:path=/hiiqos-58/obj00000000000000000019?versionId=null 2023-07-29T00:25:08.239 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000019?versionId=null 2023-07-29T00:25:08.240 DEBUG:boto:Method: GET 2023-07-29T00:25:08.240 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000019?versionId=null 2023-07-29T00:25:08.240 DEBUG:boto:Data: 2023-07-29T00:25:08.241 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.241 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:08.241 DEBUG:boto:Port: 8001 2023-07-29T00:25:08.242 DEBUG:boto:Params: {} 2023-07-29T00:25:08.242 DEBUG:boto:Token: None 2023-07-29T00:25:08.243 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000019?versionId=null 2023-07-29T00:25:08.243 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uNEua7gYp3r6cI1dg0+JJWBBBrc= 2023-07-29T00:25:08.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uNEua7gYp3r6cI1dg0+JJWBBBrc=', 'Content-Length': '0'} 2023-07-29T00:25:08.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000713ea28249457da0-0064c45c64-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.288 DEBUG:boto:path=/hiiqos-58/obj00000000000000000019 2023-07-29T00:25:08.289 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000019 2023-07-29T00:25:08.289 DEBUG:boto:path=/hiiqos-58/obj00000000000000000019?versionId=null 2023-07-29T00:25:08.290 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000019?versionId=null 2023-07-29T00:25:08.290 DEBUG:boto:Method: GET 2023-07-29T00:25:08.290 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000019?versionId=null 2023-07-29T00:25:08.291 DEBUG:boto:Data: 2023-07-29T00:25:08.293 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.293 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:08.293 DEBUG:boto:Port: 8000 2023-07-29T00:25:08.294 DEBUG:boto:Params: {} 2023-07-29T00:25:08.294 DEBUG:boto:Token: None 2023-07-29T00:25:08.295 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000019?versionId=null 2023-07-29T00:25:08.295 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uNEua7gYp3r6cI1dg0+JJWBBBrc= 2023-07-29T00:25:08.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uNEua7gYp3r6cI1dg0+JJWBBBrc=', 'Content-Length': '0'} 2023-07-29T00:25:08.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000873161fc596c9d0e-0064c45c64-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.298 DEBUG:boto:path=/hiiqos-58/obj00000000000000000019 2023-07-29T00:25:08.299 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000019 2023-07-29T00:25:08.299 DEBUG:boto:path=/hiiqos-58/obj00000000000000000019?versionId=null 2023-07-29T00:25:08.300 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000019?versionId=null 2023-07-29T00:25:08.300 DEBUG:boto:Method: HEAD 2023-07-29T00:25:08.300 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000019?versionId=null 2023-07-29T00:25:08.301 DEBUG:boto:Data: 2023-07-29T00:25:08.301 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.302 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:08.303 DEBUG:boto:Port: 8001 2023-07-29T00:25:08.303 DEBUG:boto:Params: {} 2023-07-29T00:25:08.304 DEBUG:boto:Token: None 2023-07-29T00:25:08.304 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000019?versionId=null 2023-07-29T00:25:08.305 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TBobf8CUtPI3/+i+jYkRk7C0w2Y= 2023-07-29T00:25:08.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TBobf8CUtPI3/+i+jYkRk7C0w2Y=', 'Content-Length': '0'} 2023-07-29T00:25:08.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3c91c1461836107-0064c45c64-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.308 DEBUG:boto:path=/hiiqos-58/obj00000000000000000019 2023-07-29T00:25:08.309 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000019 2023-07-29T00:25:08.309 DEBUG:boto:path=/hiiqos-58/obj00000000000000000019?versionId=null 2023-07-29T00:25:08.310 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000019?versionId=null 2023-07-29T00:25:08.310 DEBUG:boto:Method: HEAD 2023-07-29T00:25:08.310 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000019?versionId=null 2023-07-29T00:25:08.311 DEBUG:boto:Data: 2023-07-29T00:25:08.311 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.311 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:08.312 DEBUG:boto:Port: 8000 2023-07-29T00:25:08.312 DEBUG:boto:Params: {} 2023-07-29T00:25:08.312 DEBUG:boto:Token: None 2023-07-29T00:25:08.313 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000019?versionId=null 2023-07-29T00:25:08.313 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TBobf8CUtPI3/+i+jYkRk7C0w2Y= 2023-07-29T00:25:08.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TBobf8CUtPI3/+i+jYkRk7C0w2Y=', 'Content-Length': '0'} 2023-07-29T00:25:08.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f119cee0665677d6-0064c45c64-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2023-07-29T00:25:08.317 DEBUG:boto:path=/hiiqos-58/obj00000000000000000019 2023-07-29T00:25:08.317 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000019 2023-07-29T00:25:08.317 DEBUG:boto:path=/hiiqos-58/obj00000000000000000019?versionId=null 2023-07-29T00:25:08.318 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000019?versionId=null 2023-07-29T00:25:08.318 DEBUG:boto:Method: GET 2023-07-29T00:25:08.319 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000019?versionId=null 2023-07-29T00:25:08.319 DEBUG:boto:Data: 2023-07-29T00:25:08.319 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.320 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:08.320 DEBUG:boto:Port: 8001 2023-07-29T00:25:08.320 DEBUG:boto:Params: {} 2023-07-29T00:25:08.321 DEBUG:boto:Token: None 2023-07-29T00:25:08.321 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000019?versionId=null 2023-07-29T00:25:08.322 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uNEua7gYp3r6cI1dg0+JJWBBBrc= 2023-07-29T00:25:08.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uNEua7gYp3r6cI1dg0+JJWBBBrc=', 'Content-Length': '0'} 2023-07-29T00:25:08.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c77792eed8c3e187-0064c45c64-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.325 DEBUG:boto:path=/hiiqos-58/obj00000000000000000019 2023-07-29T00:25:08.325 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000019 2023-07-29T00:25:08.326 DEBUG:boto:path=/hiiqos-58/obj00000000000000000019?versionId=null 2023-07-29T00:25:08.326 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000019?versionId=null 2023-07-29T00:25:08.326 DEBUG:boto:Method: GET 2023-07-29T00:25:08.327 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000019?versionId=null 2023-07-29T00:25:08.327 DEBUG:boto:Data: 2023-07-29T00:25:08.328 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.328 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:08.328 DEBUG:boto:Port: 8000 2023-07-29T00:25:08.329 DEBUG:boto:Params: {} 2023-07-29T00:25:08.329 DEBUG:boto:Token: None 2023-07-29T00:25:08.330 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000019?versionId=null 2023-07-29T00:25:08.330 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uNEua7gYp3r6cI1dg0+JJWBBBrc= 2023-07-29T00:25:08.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uNEua7gYp3r6cI1dg0+JJWBBBrc=', 'Content-Length': '0'} 2023-07-29T00:25:08.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f908012194e849c2-0064c45c64-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.333 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:08.334 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:08.334 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000019 2023-07-29T00:25:08.334 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000019 2023-07-29T00:25:08.335 DEBUG:boto:Method: GET 2023-07-29T00:25:08.335 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000019 2023-07-29T00:25:08.335 DEBUG:boto:Data: 2023-07-29T00:25:08.336 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.336 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:08.337 DEBUG:boto:Port: 8001 2023-07-29T00:25:08.337 DEBUG:boto:Params: {} 2023-07-29T00:25:08.337 DEBUG:boto:Token: None 2023-07-29T00:25:08.338 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/ 2023-07-29T00:25:08.338 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qGAMiLgaCAmNrEigoEpXqCHju0Q= 2023-07-29T00:25:08.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qGAMiLgaCAmNrEigoEpXqCHju0Q=', 'Content-Length': '0'} 2023-07-29T00:25:08.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc1ed6c328d2ed2a-0064c45c64-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.342 DEBUG:boto:b'hiiqos-58obj000000000000000000191000falseobj000000000000000000192023-07-29T00:24:59.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:08.342 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:08.343 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:08.343 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000019 2023-07-29T00:25:08.344 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000019 2023-07-29T00:25:08.344 DEBUG:boto:Method: GET 2023-07-29T00:25:08.344 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000019 2023-07-29T00:25:08.345 DEBUG:boto:Data: 2023-07-29T00:25:08.345 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.346 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:08.346 DEBUG:boto:Port: 8000 2023-07-29T00:25:08.346 DEBUG:boto:Params: {} 2023-07-29T00:25:08.347 DEBUG:boto:Token: None 2023-07-29T00:25:08.347 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/ 2023-07-29T00:25:08.348 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qGAMiLgaCAmNrEigoEpXqCHju0Q= 2023-07-29T00:25:08.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qGAMiLgaCAmNrEigoEpXqCHju0Q=', 'Content-Length': '0'} 2023-07-29T00:25:08.352 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c03eac32243eb222-0064c45c64-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.353 DEBUG:boto:b'hiiqos-58obj000000000000000000191000falseobj000000000000000000192023-07-29T00:24:59.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:08.354 DEBUG:boto:path=/hiiqos-58/obj00000000000000000019 2023-07-29T00:25:08.354 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000019 2023-07-29T00:25:08.355 DEBUG:boto:Method: HEAD 2023-07-29T00:25:08.355 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000019 2023-07-29T00:25:08.355 DEBUG:boto:Data: 2023-07-29T00:25:08.356 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.356 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:08.356 DEBUG:boto:Port: 8001 2023-07-29T00:25:08.357 DEBUG:boto:Params: {} 2023-07-29T00:25:08.357 DEBUG:boto:Token: None 2023-07-29T00:25:08.358 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000019 2023-07-29T00:25:08.358 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TEHFgHi9aA7L6f2Bc1034vdBuZQ= 2023-07-29T00:25:08.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TEHFgHi9aA7L6f2Bc1034vdBuZQ=', 'Content-Length': '0'} 2023-07-29T00:25:08.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ae6040dba921c5a-0064c45c64-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.361 DEBUG:boto:path=/hiiqos-58/obj00000000000000000019 2023-07-29T00:25:08.362 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000019 2023-07-29T00:25:08.362 DEBUG:boto:Method: HEAD 2023-07-29T00:25:08.362 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000019 2023-07-29T00:25:08.363 DEBUG:boto:Data: 2023-07-29T00:25:08.363 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.363 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:08.364 DEBUG:boto:Port: 8000 2023-07-29T00:25:08.364 DEBUG:boto:Params: {} 2023-07-29T00:25:08.364 DEBUG:boto:Token: None 2023-07-29T00:25:08.365 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000019 2023-07-29T00:25:08.365 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TEHFgHi9aA7L6f2Bc1034vdBuZQ= 2023-07-29T00:25:08.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TEHFgHi9aA7L6f2Bc1034vdBuZQ=', 'Content-Length': '0'} 2023-07-29T00:25:08.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d50c709cf34069c-0064c45c64-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2023-07-29T00:25:08.369 DEBUG:boto:path=/hiiqos-58/obj00000000000000000019 2023-07-29T00:25:08.369 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000019 2023-07-29T00:25:08.370 DEBUG:boto:Method: GET 2023-07-29T00:25:08.370 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000019 2023-07-29T00:25:08.370 DEBUG:boto:Data: 2023-07-29T00:25:08.371 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.371 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:08.371 DEBUG:boto:Port: 8001 2023-07-29T00:25:08.372 DEBUG:boto:Params: {} 2023-07-29T00:25:08.372 DEBUG:boto:Token: None 2023-07-29T00:25:08.373 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000019 2023-07-29T00:25:08.373 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GXl6JtGVac4dxJa9nGtMls2CECE= 2023-07-29T00:25:08.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GXl6JtGVac4dxJa9nGtMls2CECE=', 'Content-Length': '0'} 2023-07-29T00:25:08.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002df413d52eb46d00-0064c45c64-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.416 DEBUG:boto:path=/hiiqos-58/obj00000000000000000019 2023-07-29T00:25:08.417 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000019 2023-07-29T00:25:08.417 DEBUG:boto:Method: GET 2023-07-29T00:25:08.418 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000019 2023-07-29T00:25:08.418 DEBUG:boto:Data: 2023-07-29T00:25:08.418 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.419 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:08.419 DEBUG:boto:Port: 8000 2023-07-29T00:25:08.420 DEBUG:boto:Params: {} 2023-07-29T00:25:08.420 DEBUG:boto:Token: None 2023-07-29T00:25:08.421 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000019 2023-07-29T00:25:08.421 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GXl6JtGVac4dxJa9nGtMls2CECE= 2023-07-29T00:25:08.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GXl6JtGVac4dxJa9nGtMls2CECE=', 'Content-Length': '0'} 2023-07-29T00:25:08.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009892cb8bb899e283-0064c45c64-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.424 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:08.425 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:08.425 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000019 2023-07-29T00:25:08.426 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000019 2023-07-29T00:25:08.426 DEBUG:boto:Method: GET 2023-07-29T00:25:08.426 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000019 2023-07-29T00:25:08.427 DEBUG:boto:Data: 2023-07-29T00:25:08.427 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.427 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:08.428 DEBUG:boto:Port: 8001 2023-07-29T00:25:08.428 DEBUG:boto:Params: {} 2023-07-29T00:25:08.429 DEBUG:boto:Token: None 2023-07-29T00:25:08.429 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/ 2023-07-29T00:25:08.429 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qGAMiLgaCAmNrEigoEpXqCHju0Q= 2023-07-29T00:25:08.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qGAMiLgaCAmNrEigoEpXqCHju0Q=', 'Content-Length': '0'} 2023-07-29T00:25:08.432 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2d003b06207cb3f-0064c45c64-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.433 DEBUG:boto:b'hiiqos-58obj000000000000000000191000falseobj000000000000000000192023-07-29T00:24:59.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:08.434 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:08.434 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:08.435 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000019 2023-07-29T00:25:08.435 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000019 2023-07-29T00:25:08.435 DEBUG:boto:Method: GET 2023-07-29T00:25:08.436 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000019 2023-07-29T00:25:08.436 DEBUG:boto:Data: 2023-07-29T00:25:08.436 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.437 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:08.437 DEBUG:boto:Port: 8000 2023-07-29T00:25:08.438 DEBUG:boto:Params: {} 2023-07-29T00:25:08.438 DEBUG:boto:Token: None 2023-07-29T00:25:08.438 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/ 2023-07-29T00:25:08.439 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qGAMiLgaCAmNrEigoEpXqCHju0Q= 2023-07-29T00:25:08.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qGAMiLgaCAmNrEigoEpXqCHju0Q=', 'Content-Length': '0'} 2023-07-29T00:25:08.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b7de6715f510f28-0064c45c64-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.445 DEBUG:boto:b'hiiqos-58obj000000000000000000191000falseobj000000000000000000192023-07-29T00:24:59.969Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:08.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2023-07-29T00:25:08.446 DEBUG:boto:path=/hiiqos-58/obj00000000000000000020 2023-07-29T00:25:08.446 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000020 2023-07-29T00:25:08.446 DEBUG:boto:path=/hiiqos-58/obj00000000000000000020?versionId=null 2023-07-29T00:25:08.447 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000020?versionId=null 2023-07-29T00:25:08.447 DEBUG:boto:Method: GET 2023-07-29T00:25:08.448 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000020?versionId=null 2023-07-29T00:25:08.448 DEBUG:boto:Data: 2023-07-29T00:25:08.448 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.449 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:08.449 DEBUG:boto:Port: 8001 2023-07-29T00:25:08.450 DEBUG:boto:Params: {} 2023-07-29T00:25:08.450 DEBUG:boto:Token: None 2023-07-29T00:25:08.450 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000020?versionId=null 2023-07-29T00:25:08.451 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5l6ByY04tK1N7+KmZgzEuILWL8g= 2023-07-29T00:25:08.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5l6ByY04tK1N7+KmZgzEuILWL8g=', 'Content-Length': '0'} 2023-07-29T00:25:08.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a27f43b51871e530-0064c45c64-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.455 DEBUG:boto:path=/hiiqos-58/obj00000000000000000020 2023-07-29T00:25:08.455 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000020 2023-07-29T00:25:08.455 DEBUG:boto:path=/hiiqos-58/obj00000000000000000020?versionId=null 2023-07-29T00:25:08.456 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000020?versionId=null 2023-07-29T00:25:08.456 DEBUG:boto:Method: GET 2023-07-29T00:25:08.456 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000020?versionId=null 2023-07-29T00:25:08.457 DEBUG:boto:Data: 2023-07-29T00:25:08.457 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.457 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:08.458 DEBUG:boto:Port: 8000 2023-07-29T00:25:08.458 DEBUG:boto:Params: {} 2023-07-29T00:25:08.459 DEBUG:boto:Token: None 2023-07-29T00:25:08.459 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000020?versionId=null 2023-07-29T00:25:08.459 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5l6ByY04tK1N7+KmZgzEuILWL8g= 2023-07-29T00:25:08.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5l6ByY04tK1N7+KmZgzEuILWL8g=', 'Content-Length': '0'} 2023-07-29T00:25:08.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdec8aa171548dc7-0064c45c64-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.463 DEBUG:boto:path=/hiiqos-58/obj00000000000000000020 2023-07-29T00:25:08.463 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000020 2023-07-29T00:25:08.463 DEBUG:boto:path=/hiiqos-58/obj00000000000000000020?versionId=null 2023-07-29T00:25:08.464 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000020?versionId=null 2023-07-29T00:25:08.464 DEBUG:boto:Method: HEAD 2023-07-29T00:25:08.464 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000020?versionId=null 2023-07-29T00:25:08.465 DEBUG:boto:Data: 2023-07-29T00:25:08.465 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.466 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:08.466 DEBUG:boto:Port: 8001 2023-07-29T00:25:08.466 DEBUG:boto:Params: {} 2023-07-29T00:25:08.467 DEBUG:boto:Token: None 2023-07-29T00:25:08.467 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000020?versionId=null 2023-07-29T00:25:08.467 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:osvgN0Nd8EVGtzcpSh+CWPu+6Iw= 2023-07-29T00:25:08.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:osvgN0Nd8EVGtzcpSh+CWPu+6Iw=', 'Content-Length': '0'} 2023-07-29T00:25:08.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de9298745a6723f7-0064c45c64-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.470 DEBUG:boto:path=/hiiqos-58/obj00000000000000000020 2023-07-29T00:25:08.471 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000020 2023-07-29T00:25:08.471 DEBUG:boto:path=/hiiqos-58/obj00000000000000000020?versionId=null 2023-07-29T00:25:08.471 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000020?versionId=null 2023-07-29T00:25:08.472 DEBUG:boto:Method: HEAD 2023-07-29T00:25:08.472 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000020?versionId=null 2023-07-29T00:25:08.472 DEBUG:boto:Data: 2023-07-29T00:25:08.473 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.473 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:08.473 DEBUG:boto:Port: 8000 2023-07-29T00:25:08.474 DEBUG:boto:Params: {} 2023-07-29T00:25:08.474 DEBUG:boto:Token: None 2023-07-29T00:25:08.474 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000020?versionId=null 2023-07-29T00:25:08.475 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:osvgN0Nd8EVGtzcpSh+CWPu+6Iw= 2023-07-29T00:25:08.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:osvgN0Nd8EVGtzcpSh+CWPu+6Iw=', 'Content-Length': '0'} 2023-07-29T00:25:08.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd219d959f5ec2ac-0064c45c64-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2023-07-29T00:25:08.478 DEBUG:boto:path=/hiiqos-58/obj00000000000000000020 2023-07-29T00:25:08.478 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000020 2023-07-29T00:25:08.479 DEBUG:boto:path=/hiiqos-58/obj00000000000000000020?versionId=null 2023-07-29T00:25:08.479 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000020?versionId=null 2023-07-29T00:25:08.479 DEBUG:boto:Method: GET 2023-07-29T00:25:08.480 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000020?versionId=null 2023-07-29T00:25:08.480 DEBUG:boto:Data: 2023-07-29T00:25:08.480 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.481 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:08.482 DEBUG:boto:Port: 8001 2023-07-29T00:25:08.482 DEBUG:boto:Params: {} 2023-07-29T00:25:08.482 DEBUG:boto:Token: None 2023-07-29T00:25:08.483 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000020?versionId=null 2023-07-29T00:25:08.483 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5l6ByY04tK1N7+KmZgzEuILWL8g= 2023-07-29T00:25:08.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5l6ByY04tK1N7+KmZgzEuILWL8g=', 'Content-Length': '0'} 2023-07-29T00:25:08.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021c651aa3f1e3f14-0064c45c64-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.528 DEBUG:boto:path=/hiiqos-58/obj00000000000000000020 2023-07-29T00:25:08.529 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000020 2023-07-29T00:25:08.530 DEBUG:boto:path=/hiiqos-58/obj00000000000000000020?versionId=null 2023-07-29T00:25:08.530 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000020?versionId=null 2023-07-29T00:25:08.530 DEBUG:boto:Method: GET 2023-07-29T00:25:08.531 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000020?versionId=null 2023-07-29T00:25:08.531 DEBUG:boto:Data: 2023-07-29T00:25:08.532 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.532 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:08.532 DEBUG:boto:Port: 8000 2023-07-29T00:25:08.533 DEBUG:boto:Params: {} 2023-07-29T00:25:08.533 DEBUG:boto:Token: None 2023-07-29T00:25:08.534 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000020?versionId=null 2023-07-29T00:25:08.534 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5l6ByY04tK1N7+KmZgzEuILWL8g= 2023-07-29T00:25:08.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5l6ByY04tK1N7+KmZgzEuILWL8g=', 'Content-Length': '0'} 2023-07-29T00:25:08.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bfefca1b2fd6233-0064c45c64-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.540 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:08.541 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:08.541 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000020 2023-07-29T00:25:08.542 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000020 2023-07-29T00:25:08.542 DEBUG:boto:Method: GET 2023-07-29T00:25:08.542 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000020 2023-07-29T00:25:08.543 DEBUG:boto:Data: 2023-07-29T00:25:08.543 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.544 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:08.544 DEBUG:boto:Port: 8001 2023-07-29T00:25:08.544 DEBUG:boto:Params: {} 2023-07-29T00:25:08.545 DEBUG:boto:Token: None 2023-07-29T00:25:08.545 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/ 2023-07-29T00:25:08.546 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qGAMiLgaCAmNrEigoEpXqCHju0Q= 2023-07-29T00:25:08.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qGAMiLgaCAmNrEigoEpXqCHju0Q=', 'Content-Length': '0'} 2023-07-29T00:25:08.549 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026a3ca9c7c0751ab-0064c45c64-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.549 DEBUG:boto:b'hiiqos-58obj000000000000000000201000falseobj000000000000000000202023-07-29T00:24:59.538Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:08.550 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:08.550 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:08.551 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000020 2023-07-29T00:25:08.551 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000020 2023-07-29T00:25:08.551 DEBUG:boto:Method: GET 2023-07-29T00:25:08.552 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000020 2023-07-29T00:25:08.552 DEBUG:boto:Data: 2023-07-29T00:25:08.552 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.553 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:08.553 DEBUG:boto:Port: 8000 2023-07-29T00:25:08.554 DEBUG:boto:Params: {} 2023-07-29T00:25:08.554 DEBUG:boto:Token: None 2023-07-29T00:25:08.554 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/ 2023-07-29T00:25:08.555 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qGAMiLgaCAmNrEigoEpXqCHju0Q= 2023-07-29T00:25:08.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qGAMiLgaCAmNrEigoEpXqCHju0Q=', 'Content-Length': '0'} 2023-07-29T00:25:08.560 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9d31ecd1fcd903c-0064c45c64-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.562 DEBUG:boto:b'hiiqos-58obj000000000000000000201000falseobj000000000000000000202023-07-29T00:24:59.538Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:08.563 DEBUG:boto:path=/hiiqos-58/obj00000000000000000020 2023-07-29T00:25:08.563 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000020 2023-07-29T00:25:08.563 DEBUG:boto:Method: HEAD 2023-07-29T00:25:08.564 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000020 2023-07-29T00:25:08.564 DEBUG:boto:Data: 2023-07-29T00:25:08.565 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.565 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:08.565 DEBUG:boto:Port: 8001 2023-07-29T00:25:08.566 DEBUG:boto:Params: {} 2023-07-29T00:25:08.566 DEBUG:boto:Token: None 2023-07-29T00:25:08.566 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000020 2023-07-29T00:25:08.567 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LYKHRXY0WPvHGqU7iXaXez4shu8= 2023-07-29T00:25:08.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LYKHRXY0WPvHGqU7iXaXez4shu8=', 'Content-Length': '0'} 2023-07-29T00:25:08.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dffcef842db1800b-0064c45c64-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.570 DEBUG:boto:path=/hiiqos-58/obj00000000000000000020 2023-07-29T00:25:08.570 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000020 2023-07-29T00:25:08.571 DEBUG:boto:Method: HEAD 2023-07-29T00:25:08.571 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000020 2023-07-29T00:25:08.572 DEBUG:boto:Data: 2023-07-29T00:25:08.572 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.572 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:08.573 DEBUG:boto:Port: 8000 2023-07-29T00:25:08.573 DEBUG:boto:Params: {} 2023-07-29T00:25:08.573 DEBUG:boto:Token: None 2023-07-29T00:25:08.574 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000020 2023-07-29T00:25:08.575 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LYKHRXY0WPvHGqU7iXaXez4shu8= 2023-07-29T00:25:08.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LYKHRXY0WPvHGqU7iXaXez4shu8=', 'Content-Length': '0'} 2023-07-29T00:25:08.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a79ad718aed66fb0-0064c45c64-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2023-07-29T00:25:08.579 DEBUG:boto:path=/hiiqos-58/obj00000000000000000020 2023-07-29T00:25:08.580 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000020 2023-07-29T00:25:08.580 DEBUG:boto:Method: GET 2023-07-29T00:25:08.580 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000020 2023-07-29T00:25:08.581 DEBUG:boto:Data: 2023-07-29T00:25:08.581 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.581 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:08.582 DEBUG:boto:Port: 8001 2023-07-29T00:25:08.582 DEBUG:boto:Params: {} 2023-07-29T00:25:08.583 DEBUG:boto:Token: None 2023-07-29T00:25:08.583 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000020 2023-07-29T00:25:08.583 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vRJ5p5hBzcm2emtRpNeORtp1tcM= 2023-07-29T00:25:08.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vRJ5p5hBzcm2emtRpNeORtp1tcM=', 'Content-Length': '0'} 2023-07-29T00:25:08.586 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091a838da2fba5df6-0064c45c64-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.628 DEBUG:boto:path=/hiiqos-58/obj00000000000000000020 2023-07-29T00:25:08.629 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000020 2023-07-29T00:25:08.630 DEBUG:boto:Method: GET 2023-07-29T00:25:08.630 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000020 2023-07-29T00:25:08.630 DEBUG:boto:Data: 2023-07-29T00:25:08.631 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.631 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:08.631 DEBUG:boto:Port: 8000 2023-07-29T00:25:08.632 DEBUG:boto:Params: {} 2023-07-29T00:25:08.632 DEBUG:boto:Token: None 2023-07-29T00:25:08.633 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000020 2023-07-29T00:25:08.633 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vRJ5p5hBzcm2emtRpNeORtp1tcM= 2023-07-29T00:25:08.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vRJ5p5hBzcm2emtRpNeORtp1tcM=', 'Content-Length': '0'} 2023-07-29T00:25:08.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fd062fb94e45599-0064c45c64-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.636 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:08.637 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:08.637 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000020 2023-07-29T00:25:08.638 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000020 2023-07-29T00:25:08.638 DEBUG:boto:Method: GET 2023-07-29T00:25:08.638 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000020 2023-07-29T00:25:08.639 DEBUG:boto:Data: 2023-07-29T00:25:08.639 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.639 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:08.640 DEBUG:boto:Port: 8001 2023-07-29T00:25:08.640 DEBUG:boto:Params: {} 2023-07-29T00:25:08.641 DEBUG:boto:Token: None 2023-07-29T00:25:08.641 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/ 2023-07-29T00:25:08.642 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qGAMiLgaCAmNrEigoEpXqCHju0Q= 2023-07-29T00:25:08.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qGAMiLgaCAmNrEigoEpXqCHju0Q=', 'Content-Length': '0'} 2023-07-29T00:25:08.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001be4fa68bc547b9d-0064c45c64-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.645 DEBUG:boto:b'hiiqos-58obj000000000000000000201000falseobj000000000000000000202023-07-29T00:24:59.538Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:08.646 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:08.646 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:08.647 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000020 2023-07-29T00:25:08.647 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000020 2023-07-29T00:25:08.647 DEBUG:boto:Method: GET 2023-07-29T00:25:08.648 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000020 2023-07-29T00:25:08.648 DEBUG:boto:Data: 2023-07-29T00:25:08.649 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.649 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:08.649 DEBUG:boto:Port: 8000 2023-07-29T00:25:08.650 DEBUG:boto:Params: {} 2023-07-29T00:25:08.650 DEBUG:boto:Token: None 2023-07-29T00:25:08.650 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/ 2023-07-29T00:25:08.651 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qGAMiLgaCAmNrEigoEpXqCHju0Q= 2023-07-29T00:25:08.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qGAMiLgaCAmNrEigoEpXqCHju0Q=', 'Content-Length': '0'} 2023-07-29T00:25:08.656 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef67f175e49ef5d0-0064c45c64-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.657 DEBUG:boto:b'hiiqos-58obj000000000000000000201000falseobj000000000000000000202023-07-29T00:24:59.538Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:08.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2023-07-29T00:25:08.658 DEBUG:boto:path=/hiiqos-58/obj00000000000000000021 2023-07-29T00:25:08.658 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000021 2023-07-29T00:25:08.659 DEBUG:boto:path=/hiiqos-58/obj00000000000000000021?versionId=null 2023-07-29T00:25:08.659 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000021?versionId=null 2023-07-29T00:25:08.659 DEBUG:boto:Method: GET 2023-07-29T00:25:08.660 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000021?versionId=null 2023-07-29T00:25:08.660 DEBUG:boto:Data: 2023-07-29T00:25:08.661 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.661 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:08.661 DEBUG:boto:Port: 8001 2023-07-29T00:25:08.662 DEBUG:boto:Params: {} 2023-07-29T00:25:08.662 DEBUG:boto:Token: None 2023-07-29T00:25:08.663 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000021?versionId=null 2023-07-29T00:25:08.663 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rHxJcLe96MP7p8uQerQufDGGcqg= 2023-07-29T00:25:08.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rHxJcLe96MP7p8uQerQufDGGcqg=', 'Content-Length': '0'} 2023-07-29T00:25:08.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ada73ffac81f59b-0064c45c64-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.667 DEBUG:boto:path=/hiiqos-58/obj00000000000000000021 2023-07-29T00:25:08.667 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000021 2023-07-29T00:25:08.668 DEBUG:boto:path=/hiiqos-58/obj00000000000000000021?versionId=null 2023-07-29T00:25:08.668 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000021?versionId=null 2023-07-29T00:25:08.669 DEBUG:boto:Method: GET 2023-07-29T00:25:08.669 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000021?versionId=null 2023-07-29T00:25:08.669 DEBUG:boto:Data: 2023-07-29T00:25:08.670 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.670 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:08.670 DEBUG:boto:Port: 8000 2023-07-29T00:25:08.671 DEBUG:boto:Params: {} 2023-07-29T00:25:08.671 DEBUG:boto:Token: None 2023-07-29T00:25:08.672 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000021?versionId=null 2023-07-29T00:25:08.672 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rHxJcLe96MP7p8uQerQufDGGcqg= 2023-07-29T00:25:08.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rHxJcLe96MP7p8uQerQufDGGcqg=', 'Content-Length': '0'} 2023-07-29T00:25:08.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1e32ffb8d3d4b09-0064c45c64-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.676 DEBUG:boto:path=/hiiqos-58/obj00000000000000000021 2023-07-29T00:25:08.676 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000021 2023-07-29T00:25:08.677 DEBUG:boto:path=/hiiqos-58/obj00000000000000000021?versionId=null 2023-07-29T00:25:08.677 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000021?versionId=null 2023-07-29T00:25:08.677 DEBUG:boto:Method: HEAD 2023-07-29T00:25:08.678 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000021?versionId=null 2023-07-29T00:25:08.678 DEBUG:boto:Data: 2023-07-29T00:25:08.678 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.679 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:08.679 DEBUG:boto:Port: 8001 2023-07-29T00:25:08.679 DEBUG:boto:Params: {} 2023-07-29T00:25:08.680 DEBUG:boto:Token: None 2023-07-29T00:25:08.680 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000021?versionId=null 2023-07-29T00:25:08.681 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:C9R/a6k9dNYJfzmp64/i2FkL3ac= 2023-07-29T00:25:08.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:C9R/a6k9dNYJfzmp64/i2FkL3ac=', 'Content-Length': '0'} 2023-07-29T00:25:08.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000803ce2d439ae9e0-0064c45c64-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.684 DEBUG:boto:path=/hiiqos-58/obj00000000000000000021 2023-07-29T00:25:08.684 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000021 2023-07-29T00:25:08.685 DEBUG:boto:path=/hiiqos-58/obj00000000000000000021?versionId=null 2023-07-29T00:25:08.685 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000021?versionId=null 2023-07-29T00:25:08.685 DEBUG:boto:Method: HEAD 2023-07-29T00:25:08.686 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000021?versionId=null 2023-07-29T00:25:08.686 DEBUG:boto:Data: 2023-07-29T00:25:08.686 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.687 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:08.687 DEBUG:boto:Port: 8000 2023-07-29T00:25:08.687 DEBUG:boto:Params: {} 2023-07-29T00:25:08.688 DEBUG:boto:Token: None 2023-07-29T00:25:08.688 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000021?versionId=null 2023-07-29T00:25:08.689 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:C9R/a6k9dNYJfzmp64/i2FkL3ac= 2023-07-29T00:25:08.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:C9R/a6k9dNYJfzmp64/i2FkL3ac=', 'Content-Length': '0'} 2023-07-29T00:25:08.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000022232c79ddf0801-0064c45c64-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2023-07-29T00:25:08.692 DEBUG:boto:path=/hiiqos-58/obj00000000000000000021 2023-07-29T00:25:08.692 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000021 2023-07-29T00:25:08.693 DEBUG:boto:path=/hiiqos-58/obj00000000000000000021?versionId=null 2023-07-29T00:25:08.693 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000021?versionId=null 2023-07-29T00:25:08.694 DEBUG:boto:Method: GET 2023-07-29T00:25:08.694 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000021?versionId=null 2023-07-29T00:25:08.694 DEBUG:boto:Data: 2023-07-29T00:25:08.695 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.695 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:08.695 DEBUG:boto:Port: 8001 2023-07-29T00:25:08.696 DEBUG:boto:Params: {} 2023-07-29T00:25:08.696 DEBUG:boto:Token: None 2023-07-29T00:25:08.697 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000021?versionId=null 2023-07-29T00:25:08.697 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rHxJcLe96MP7p8uQerQufDGGcqg= 2023-07-29T00:25:08.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rHxJcLe96MP7p8uQerQufDGGcqg=', 'Content-Length': '0'} 2023-07-29T00:25:08.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067ee4433f7c805e1-0064c45c64-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.700 DEBUG:boto:path=/hiiqos-58/obj00000000000000000021 2023-07-29T00:25:08.700 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000021 2023-07-29T00:25:08.701 DEBUG:boto:path=/hiiqos-58/obj00000000000000000021?versionId=null 2023-07-29T00:25:08.701 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000021?versionId=null 2023-07-29T00:25:08.701 DEBUG:boto:Method: GET 2023-07-29T00:25:08.702 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000021?versionId=null 2023-07-29T00:25:08.702 DEBUG:boto:Data: 2023-07-29T00:25:08.703 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.703 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:08.703 DEBUG:boto:Port: 8000 2023-07-29T00:25:08.704 DEBUG:boto:Params: {} 2023-07-29T00:25:08.704 DEBUG:boto:Token: None 2023-07-29T00:25:08.705 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000021?versionId=null 2023-07-29T00:25:08.705 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rHxJcLe96MP7p8uQerQufDGGcqg= 2023-07-29T00:25:08.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rHxJcLe96MP7p8uQerQufDGGcqg=', 'Content-Length': '0'} 2023-07-29T00:25:08.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000de884858106ef01-0064c45c64-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.708 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:08.709 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:08.709 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000021 2023-07-29T00:25:08.709 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000021 2023-07-29T00:25:08.710 DEBUG:boto:Method: GET 2023-07-29T00:25:08.710 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000021 2023-07-29T00:25:08.710 DEBUG:boto:Data: 2023-07-29T00:25:08.711 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.711 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:08.711 DEBUG:boto:Port: 8001 2023-07-29T00:25:08.712 DEBUG:boto:Params: {} 2023-07-29T00:25:08.712 DEBUG:boto:Token: None 2023-07-29T00:25:08.713 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/ 2023-07-29T00:25:08.713 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qGAMiLgaCAmNrEigoEpXqCHju0Q= 2023-07-29T00:25:08.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qGAMiLgaCAmNrEigoEpXqCHju0Q=', 'Content-Length': '0'} 2023-07-29T00:25:08.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5dbc052e4f38d41-0064c45c64-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.717 DEBUG:boto:b'hiiqos-58obj000000000000000000211000falseobj000000000000000000212023-07-29T00:24:59.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:08.717 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:08.718 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:08.718 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000021 2023-07-29T00:25:08.718 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000021 2023-07-29T00:25:08.719 DEBUG:boto:Method: GET 2023-07-29T00:25:08.719 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000021 2023-07-29T00:25:08.719 DEBUG:boto:Data: 2023-07-29T00:25:08.720 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.720 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:08.721 DEBUG:boto:Port: 8000 2023-07-29T00:25:08.721 DEBUG:boto:Params: {} 2023-07-29T00:25:08.721 DEBUG:boto:Token: None 2023-07-29T00:25:08.722 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/ 2023-07-29T00:25:08.722 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qGAMiLgaCAmNrEigoEpXqCHju0Q= 2023-07-29T00:25:08.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qGAMiLgaCAmNrEigoEpXqCHju0Q=', 'Content-Length': '0'} 2023-07-29T00:25:08.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bba919e3c73fa79-0064c45c64-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.728 DEBUG:boto:b'hiiqos-58obj000000000000000000211000falseobj000000000000000000212023-07-29T00:24:59.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:08.728 DEBUG:boto:path=/hiiqos-58/obj00000000000000000021 2023-07-29T00:25:08.729 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000021 2023-07-29T00:25:08.730 DEBUG:boto:Method: HEAD 2023-07-29T00:25:08.730 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000021 2023-07-29T00:25:08.730 DEBUG:boto:Data: 2023-07-29T00:25:08.731 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.731 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:08.731 DEBUG:boto:Port: 8001 2023-07-29T00:25:08.732 DEBUG:boto:Params: {} 2023-07-29T00:25:08.732 DEBUG:boto:Token: None 2023-07-29T00:25:08.733 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000021 2023-07-29T00:25:08.733 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KLi7DLIPNsr3WO4dDhoSVwhdSdk= 2023-07-29T00:25:08.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KLi7DLIPNsr3WO4dDhoSVwhdSdk=', 'Content-Length': '0'} 2023-07-29T00:25:08.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc4b455672401cd5-0064c45c64-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.736 DEBUG:boto:path=/hiiqos-58/obj00000000000000000021 2023-07-29T00:25:08.736 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000021 2023-07-29T00:25:08.737 DEBUG:boto:Method: HEAD 2023-07-29T00:25:08.737 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000021 2023-07-29T00:25:08.737 DEBUG:boto:Data: 2023-07-29T00:25:08.738 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.738 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:08.738 DEBUG:boto:Port: 8000 2023-07-29T00:25:08.739 DEBUG:boto:Params: {} 2023-07-29T00:25:08.739 DEBUG:boto:Token: None 2023-07-29T00:25:08.740 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000021 2023-07-29T00:25:08.740 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KLi7DLIPNsr3WO4dDhoSVwhdSdk= 2023-07-29T00:25:08.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KLi7DLIPNsr3WO4dDhoSVwhdSdk=', 'Content-Length': '0'} 2023-07-29T00:25:08.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095ca715664866bb1-0064c45c64-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2023-07-29T00:25:08.744 DEBUG:boto:path=/hiiqos-58/obj00000000000000000021 2023-07-29T00:25:08.744 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000021 2023-07-29T00:25:08.745 DEBUG:boto:Method: GET 2023-07-29T00:25:08.745 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000021 2023-07-29T00:25:08.745 DEBUG:boto:Data: 2023-07-29T00:25:08.746 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.746 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:08.746 DEBUG:boto:Port: 8001 2023-07-29T00:25:08.747 DEBUG:boto:Params: {} 2023-07-29T00:25:08.747 DEBUG:boto:Token: None 2023-07-29T00:25:08.747 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000021 2023-07-29T00:25:08.748 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:iowItBC1fIm7d5y6MwHFz65D0IQ= 2023-07-29T00:25:08.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:iowItBC1fIm7d5y6MwHFz65D0IQ=', 'Content-Length': '0'} 2023-07-29T00:25:08.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000030f72fcca4eb746-0064c45c64-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.792 DEBUG:boto:path=/hiiqos-58/obj00000000000000000021 2023-07-29T00:25:08.793 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000021 2023-07-29T00:25:08.793 DEBUG:boto:Method: GET 2023-07-29T00:25:08.794 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000021 2023-07-29T00:25:08.794 DEBUG:boto:Data: 2023-07-29T00:25:08.794 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.795 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:08.795 DEBUG:boto:Port: 8000 2023-07-29T00:25:08.795 DEBUG:boto:Params: {} 2023-07-29T00:25:08.796 DEBUG:boto:Token: None 2023-07-29T00:25:08.796 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000021 2023-07-29T00:25:08.797 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:iowItBC1fIm7d5y6MwHFz65D0IQ= 2023-07-29T00:25:08.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:iowItBC1fIm7d5y6MwHFz65D0IQ=', 'Content-Length': '0'} 2023-07-29T00:25:08.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b5a3df2d5ca016b-0064c45c64-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.800 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:08.801 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:08.801 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000021 2023-07-29T00:25:08.802 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000021 2023-07-29T00:25:08.802 DEBUG:boto:Method: GET 2023-07-29T00:25:08.803 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000021 2023-07-29T00:25:08.803 DEBUG:boto:Data: 2023-07-29T00:25:08.803 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.804 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:08.804 DEBUG:boto:Port: 8001 2023-07-29T00:25:08.804 DEBUG:boto:Params: {} 2023-07-29T00:25:08.805 DEBUG:boto:Token: None 2023-07-29T00:25:08.805 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/ 2023-07-29T00:25:08.806 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qGAMiLgaCAmNrEigoEpXqCHju0Q= 2023-07-29T00:25:08.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qGAMiLgaCAmNrEigoEpXqCHju0Q=', 'Content-Length': '0'} 2023-07-29T00:25:08.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff61737b30476c4a-0064c45c64-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.809 DEBUG:boto:b'hiiqos-58obj000000000000000000211000falseobj000000000000000000212023-07-29T00:24:59.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:08.810 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:08.810 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:08.811 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000021 2023-07-29T00:25:08.811 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000021 2023-07-29T00:25:08.811 DEBUG:boto:Method: GET 2023-07-29T00:25:08.812 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000021 2023-07-29T00:25:08.812 DEBUG:boto:Data: 2023-07-29T00:25:08.813 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.813 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:08.814 DEBUG:boto:Port: 8000 2023-07-29T00:25:08.814 DEBUG:boto:Params: {} 2023-07-29T00:25:08.814 DEBUG:boto:Token: None 2023-07-29T00:25:08.815 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/ 2023-07-29T00:25:08.815 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qGAMiLgaCAmNrEigoEpXqCHju0Q= 2023-07-29T00:25:08.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qGAMiLgaCAmNrEigoEpXqCHju0Q=', 'Content-Length': '0'} 2023-07-29T00:25:08.820 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6be192659d56563-0064c45c64-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.820 DEBUG:boto:b'hiiqos-58obj000000000000000000211000falseobj000000000000000000212023-07-29T00:24:59.824Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:08.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2023-07-29T00:25:08.821 DEBUG:boto:path=/hiiqos-58/obj00000000000000000022 2023-07-29T00:25:08.822 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000022 2023-07-29T00:25:08.822 DEBUG:boto:path=/hiiqos-58/obj00000000000000000022?versionId=null 2023-07-29T00:25:08.823 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000022?versionId=null 2023-07-29T00:25:08.823 DEBUG:boto:Method: GET 2023-07-29T00:25:08.823 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000022?versionId=null 2023-07-29T00:25:08.824 DEBUG:boto:Data: 2023-07-29T00:25:08.824 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.824 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:08.825 DEBUG:boto:Port: 8001 2023-07-29T00:25:08.825 DEBUG:boto:Params: {} 2023-07-29T00:25:08.826 DEBUG:boto:Token: None 2023-07-29T00:25:08.826 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000022?versionId=null 2023-07-29T00:25:08.826 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vMqF5yXHTwVh9OsUB3BYfaYXi1w= 2023-07-29T00:25:08.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vMqF5yXHTwVh9OsUB3BYfaYXi1w=', 'Content-Length': '0'} 2023-07-29T00:25:08.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f53651a22d7d8e22-0064c45c64-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.830 DEBUG:boto:path=/hiiqos-58/obj00000000000000000022 2023-07-29T00:25:08.830 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000022 2023-07-29T00:25:08.830 DEBUG:boto:path=/hiiqos-58/obj00000000000000000022?versionId=null 2023-07-29T00:25:08.831 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000022?versionId=null 2023-07-29T00:25:08.831 DEBUG:boto:Method: GET 2023-07-29T00:25:08.832 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000022?versionId=null 2023-07-29T00:25:08.832 DEBUG:boto:Data: 2023-07-29T00:25:08.832 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.833 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:08.833 DEBUG:boto:Port: 8000 2023-07-29T00:25:08.833 DEBUG:boto:Params: {} 2023-07-29T00:25:08.834 DEBUG:boto:Token: None 2023-07-29T00:25:08.834 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000022?versionId=null 2023-07-29T00:25:08.835 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vMqF5yXHTwVh9OsUB3BYfaYXi1w= 2023-07-29T00:25:08.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vMqF5yXHTwVh9OsUB3BYfaYXi1w=', 'Content-Length': '0'} 2023-07-29T00:25:08.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b563f5ec95bfe31c-0064c45c64-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.837 DEBUG:boto:path=/hiiqos-58/obj00000000000000000022 2023-07-29T00:25:08.838 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000022 2023-07-29T00:25:08.838 DEBUG:boto:path=/hiiqos-58/obj00000000000000000022?versionId=null 2023-07-29T00:25:08.839 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000022?versionId=null 2023-07-29T00:25:08.839 DEBUG:boto:Method: HEAD 2023-07-29T00:25:08.839 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000022?versionId=null 2023-07-29T00:25:08.840 DEBUG:boto:Data: 2023-07-29T00:25:08.840 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.840 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:08.841 DEBUG:boto:Port: 8001 2023-07-29T00:25:08.841 DEBUG:boto:Params: {} 2023-07-29T00:25:08.841 DEBUG:boto:Token: None 2023-07-29T00:25:08.842 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000022?versionId=null 2023-07-29T00:25:08.842 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EOie+4cQXho0XmuBYLrBsv5jCWI= 2023-07-29T00:25:08.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EOie+4cQXho0XmuBYLrBsv5jCWI=', 'Content-Length': '0'} 2023-07-29T00:25:08.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000820fe36639f73e6f-0064c45c64-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.845 DEBUG:boto:path=/hiiqos-58/obj00000000000000000022 2023-07-29T00:25:08.846 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000022 2023-07-29T00:25:08.846 DEBUG:boto:path=/hiiqos-58/obj00000000000000000022?versionId=null 2023-07-29T00:25:08.846 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000022?versionId=null 2023-07-29T00:25:08.847 DEBUG:boto:Method: HEAD 2023-07-29T00:25:08.847 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000022?versionId=null 2023-07-29T00:25:08.847 DEBUG:boto:Data: 2023-07-29T00:25:08.848 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.848 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:08.848 DEBUG:boto:Port: 8000 2023-07-29T00:25:08.849 DEBUG:boto:Params: {} 2023-07-29T00:25:08.849 DEBUG:boto:Token: None 2023-07-29T00:25:08.849 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000022?versionId=null 2023-07-29T00:25:08.850 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EOie+4cQXho0XmuBYLrBsv5jCWI= 2023-07-29T00:25:08.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EOie+4cQXho0XmuBYLrBsv5jCWI=', 'Content-Length': '0'} 2023-07-29T00:25:08.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb4613a97e5438c7-0064c45c64-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.853 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2023-07-29T00:25:08.853 DEBUG:boto:path=/hiiqos-58/obj00000000000000000022 2023-07-29T00:25:08.854 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000022 2023-07-29T00:25:08.854 DEBUG:boto:path=/hiiqos-58/obj00000000000000000022?versionId=null 2023-07-29T00:25:08.854 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000022?versionId=null 2023-07-29T00:25:08.855 DEBUG:boto:Method: GET 2023-07-29T00:25:08.855 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000022?versionId=null 2023-07-29T00:25:08.855 DEBUG:boto:Data: 2023-07-29T00:25:08.856 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.856 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:08.856 DEBUG:boto:Port: 8001 2023-07-29T00:25:08.857 DEBUG:boto:Params: {} 2023-07-29T00:25:08.857 DEBUG:boto:Token: None 2023-07-29T00:25:08.858 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000022?versionId=null 2023-07-29T00:25:08.858 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vMqF5yXHTwVh9OsUB3BYfaYXi1w= 2023-07-29T00:25:08.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vMqF5yXHTwVh9OsUB3BYfaYXi1w=', 'Content-Length': '0'} 2023-07-29T00:25:08.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071a46f567e832f50-0064c45c64-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.908 DEBUG:boto:path=/hiiqos-58/obj00000000000000000022 2023-07-29T00:25:08.909 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000022 2023-07-29T00:25:08.909 DEBUG:boto:path=/hiiqos-58/obj00000000000000000022?versionId=null 2023-07-29T00:25:08.910 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000022?versionId=null 2023-07-29T00:25:08.910 DEBUG:boto:Method: GET 2023-07-29T00:25:08.911 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000022?versionId=null 2023-07-29T00:25:08.911 DEBUG:boto:Data: 2023-07-29T00:25:08.911 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.912 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:08.912 DEBUG:boto:Port: 8000 2023-07-29T00:25:08.912 DEBUG:boto:Params: {} 2023-07-29T00:25:08.913 DEBUG:boto:Token: None 2023-07-29T00:25:08.913 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000022?versionId=null 2023-07-29T00:25:08.914 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vMqF5yXHTwVh9OsUB3BYfaYXi1w= 2023-07-29T00:25:08.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vMqF5yXHTwVh9OsUB3BYfaYXi1w=', 'Content-Length': '0'} 2023-07-29T00:25:08.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a521c8ce85886dc3-0064c45c64-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.917 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:08.917 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:08.918 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000022 2023-07-29T00:25:08.918 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000022 2023-07-29T00:25:08.918 DEBUG:boto:Method: GET 2023-07-29T00:25:08.919 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000022 2023-07-29T00:25:08.919 DEBUG:boto:Data: 2023-07-29T00:25:08.919 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.920 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:08.920 DEBUG:boto:Port: 8001 2023-07-29T00:25:08.921 DEBUG:boto:Params: {} 2023-07-29T00:25:08.921 DEBUG:boto:Token: None 2023-07-29T00:25:08.921 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/ 2023-07-29T00:25:08.922 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qGAMiLgaCAmNrEigoEpXqCHju0Q= 2023-07-29T00:25:08.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qGAMiLgaCAmNrEigoEpXqCHju0Q=', 'Content-Length': '0'} 2023-07-29T00:25:08.925 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004704fa9013a96d11-0064c45c64-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.925 DEBUG:boto:b'hiiqos-58obj000000000000000000221000falseobj000000000000000000222023-07-29T00:24:58.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:08.926 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:08.926 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:08.927 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000022 2023-07-29T00:25:08.927 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000022 2023-07-29T00:25:08.928 DEBUG:boto:Method: GET 2023-07-29T00:25:08.928 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000022 2023-07-29T00:25:08.928 DEBUG:boto:Data: 2023-07-29T00:25:08.929 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.929 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:08.929 DEBUG:boto:Port: 8000 2023-07-29T00:25:08.930 DEBUG:boto:Params: {} 2023-07-29T00:25:08.930 DEBUG:boto:Token: None 2023-07-29T00:25:08.931 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/ 2023-07-29T00:25:08.931 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qGAMiLgaCAmNrEigoEpXqCHju0Q= 2023-07-29T00:25:08.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qGAMiLgaCAmNrEigoEpXqCHju0Q=', 'Content-Length': '0'} 2023-07-29T00:25:08.936 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b9794670ac1f8b9-0064c45c64-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.937 DEBUG:boto:b'hiiqos-58obj000000000000000000221000falseobj000000000000000000222023-07-29T00:24:58.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:08.938 DEBUG:boto:path=/hiiqos-58/obj00000000000000000022 2023-07-29T00:25:08.938 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000022 2023-07-29T00:25:08.938 DEBUG:boto:Method: HEAD 2023-07-29T00:25:08.939 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000022 2023-07-29T00:25:08.939 DEBUG:boto:Data: 2023-07-29T00:25:08.939 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.940 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:08.940 DEBUG:boto:Port: 8001 2023-07-29T00:25:08.940 DEBUG:boto:Params: {} 2023-07-29T00:25:08.941 DEBUG:boto:Token: None 2023-07-29T00:25:08.941 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000022 2023-07-29T00:25:08.942 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:R4WtIacG/gYgtlpsYLcE2xsk8QU= 2023-07-29T00:25:08.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:R4WtIacG/gYgtlpsYLcE2xsk8QU=', 'Content-Length': '0'} 2023-07-29T00:25:08.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000951c48189315d24-0064c45c64-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.945 DEBUG:boto:path=/hiiqos-58/obj00000000000000000022 2023-07-29T00:25:08.945 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000022 2023-07-29T00:25:08.945 DEBUG:boto:Method: HEAD 2023-07-29T00:25:08.946 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000022 2023-07-29T00:25:08.946 DEBUG:boto:Data: 2023-07-29T00:25:08.946 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.947 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:08.947 DEBUG:boto:Port: 8000 2023-07-29T00:25:08.948 DEBUG:boto:Params: {} 2023-07-29T00:25:08.948 DEBUG:boto:Token: None 2023-07-29T00:25:08.948 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000022 2023-07-29T00:25:08.949 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:R4WtIacG/gYgtlpsYLcE2xsk8QU= 2023-07-29T00:25:08.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:R4WtIacG/gYgtlpsYLcE2xsk8QU=', 'Content-Length': '0'} 2023-07-29T00:25:08.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e4f7135c9e5abf0-0064c45c64-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2023-07-29T00:25:08.953 DEBUG:boto:path=/hiiqos-58/obj00000000000000000022 2023-07-29T00:25:08.954 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000022 2023-07-29T00:25:08.954 DEBUG:boto:Method: GET 2023-07-29T00:25:08.955 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000022 2023-07-29T00:25:08.955 DEBUG:boto:Data: 2023-07-29T00:25:08.955 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.956 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:08.956 DEBUG:boto:Port: 8001 2023-07-29T00:25:08.957 DEBUG:boto:Params: {} 2023-07-29T00:25:08.957 DEBUG:boto:Token: None 2023-07-29T00:25:08.958 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000022 2023-07-29T00:25:08.958 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9atI3JR+D4jE0+ldHsgNCWjueOU= 2023-07-29T00:25:08.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9atI3JR+D4jE0+ldHsgNCWjueOU=', 'Content-Length': '0'} 2023-07-29T00:25:08.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c28a741a1572358d-0064c45c64-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.961 DEBUG:boto:path=/hiiqos-58/obj00000000000000000022 2023-07-29T00:25:08.962 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000022 2023-07-29T00:25:08.962 DEBUG:boto:Method: GET 2023-07-29T00:25:08.963 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000022 2023-07-29T00:25:08.963 DEBUG:boto:Data: 2023-07-29T00:25:08.963 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.964 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:08.964 DEBUG:boto:Port: 8000 2023-07-29T00:25:08.965 DEBUG:boto:Params: {} 2023-07-29T00:25:08.965 DEBUG:boto:Token: None 2023-07-29T00:25:08.966 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/obj00000000000000000022 2023-07-29T00:25:08.966 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9atI3JR+D4jE0+ldHsgNCWjueOU= 2023-07-29T00:25:08.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9atI3JR+D4jE0+ldHsgNCWjueOU=', 'Content-Length': '0'} 2023-07-29T00:25:08.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bf84f45e65983a3-0064c45c64-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:08.969 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:08.970 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:08.970 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000022 2023-07-29T00:25:08.970 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000022 2023-07-29T00:25:08.971 DEBUG:boto:Method: GET 2023-07-29T00:25:08.971 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000022 2023-07-29T00:25:08.972 DEBUG:boto:Data: 2023-07-29T00:25:08.972 DEBUG:boto:Headers: {} 2023-07-29T00:25:08.972 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:08.973 DEBUG:boto:Port: 8001 2023-07-29T00:25:08.973 DEBUG:boto:Params: {} 2023-07-29T00:25:08.973 DEBUG:boto:Token: None 2023-07-29T00:25:08.974 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:08 GMT /hiiqos-58/ 2023-07-29T00:25:08.974 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qGAMiLgaCAmNrEigoEpXqCHju0Q= 2023-07-29T00:25:08.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qGAMiLgaCAmNrEigoEpXqCHju0Q=', 'Content-Length': '0'} 2023-07-29T00:25:08.977 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df9ace825a6fef12-0064c45c64-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.024 DEBUG:boto:b'hiiqos-58obj000000000000000000221000falseobj000000000000000000222023-07-29T00:24:58.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:09.025 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:09.026 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:09.026 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000022 2023-07-29T00:25:09.027 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000022 2023-07-29T00:25:09.027 DEBUG:boto:Method: GET 2023-07-29T00:25:09.027 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000022 2023-07-29T00:25:09.028 DEBUG:boto:Data: 2023-07-29T00:25:09.028 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.029 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:09.029 DEBUG:boto:Port: 8000 2023-07-29T00:25:09.029 DEBUG:boto:Params: {} 2023-07-29T00:25:09.030 DEBUG:boto:Token: None 2023-07-29T00:25:09.030 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/ 2023-07-29T00:25:09.031 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:d4avFOwp7+ujArmIRItvC+vgGPg= 2023-07-29T00:25:09.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:d4avFOwp7+ujArmIRItvC+vgGPg=', 'Content-Length': '0'} 2023-07-29T00:25:09.036 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009efea83645b2f1e2-0064c45c64-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.036 DEBUG:boto:b'hiiqos-58obj000000000000000000221000falseobj000000000000000000222023-07-29T00:24:58.700Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:09.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2023-07-29T00:25:09.038 DEBUG:boto:path=/hiiqos-58/obj00000000000000000023 2023-07-29T00:25:09.038 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000023 2023-07-29T00:25:09.039 DEBUG:boto:path=/hiiqos-58/obj00000000000000000023?versionId=null 2023-07-29T00:25:09.039 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000023?versionId=null 2023-07-29T00:25:09.039 DEBUG:boto:Method: GET 2023-07-29T00:25:09.040 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000023?versionId=null 2023-07-29T00:25:09.040 DEBUG:boto:Data: 2023-07-29T00:25:09.041 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.041 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:09.041 DEBUG:boto:Port: 8001 2023-07-29T00:25:09.042 DEBUG:boto:Params: {} 2023-07-29T00:25:09.043 DEBUG:boto:Token: None 2023-07-29T00:25:09.043 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000023?versionId=null 2023-07-29T00:25:09.043 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8IeEHgNieY1KfYkYHrPmHNAIpJA= 2023-07-29T00:25:09.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8IeEHgNieY1KfYkYHrPmHNAIpJA=', 'Content-Length': '0'} 2023-07-29T00:25:09.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007eecd2e4e061efa1-0064c45c64-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.047 DEBUG:boto:path=/hiiqos-58/obj00000000000000000023 2023-07-29T00:25:09.047 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000023 2023-07-29T00:25:09.048 DEBUG:boto:path=/hiiqos-58/obj00000000000000000023?versionId=null 2023-07-29T00:25:09.048 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000023?versionId=null 2023-07-29T00:25:09.049 DEBUG:boto:Method: GET 2023-07-29T00:25:09.049 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000023?versionId=null 2023-07-29T00:25:09.049 DEBUG:boto:Data: 2023-07-29T00:25:09.050 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.050 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:09.051 DEBUG:boto:Port: 8000 2023-07-29T00:25:09.051 DEBUG:boto:Params: {} 2023-07-29T00:25:09.051 DEBUG:boto:Token: None 2023-07-29T00:25:09.052 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000023?versionId=null 2023-07-29T00:25:09.052 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8IeEHgNieY1KfYkYHrPmHNAIpJA= 2023-07-29T00:25:09.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8IeEHgNieY1KfYkYHrPmHNAIpJA=', 'Content-Length': '0'} 2023-07-29T00:25:09.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000299c6c8592c05d11-0064c45c65-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.055 DEBUG:boto:path=/hiiqos-58/obj00000000000000000023 2023-07-29T00:25:09.056 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000023 2023-07-29T00:25:09.056 DEBUG:boto:path=/hiiqos-58/obj00000000000000000023?versionId=null 2023-07-29T00:25:09.056 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000023?versionId=null 2023-07-29T00:25:09.057 DEBUG:boto:Method: HEAD 2023-07-29T00:25:09.057 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000023?versionId=null 2023-07-29T00:25:09.057 DEBUG:boto:Data: 2023-07-29T00:25:09.058 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.058 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:09.058 DEBUG:boto:Port: 8001 2023-07-29T00:25:09.059 DEBUG:boto:Params: {} 2023-07-29T00:25:09.059 DEBUG:boto:Token: None 2023-07-29T00:25:09.059 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000023?versionId=null 2023-07-29T00:25:09.060 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SWtqvFMEvXeGlXgtKU1bFSaiwUo= 2023-07-29T00:25:09.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SWtqvFMEvXeGlXgtKU1bFSaiwUo=', 'Content-Length': '0'} 2023-07-29T00:25:09.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097011cceb812a745-0064c45c65-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.063 DEBUG:boto:path=/hiiqos-58/obj00000000000000000023 2023-07-29T00:25:09.063 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000023 2023-07-29T00:25:09.064 DEBUG:boto:path=/hiiqos-58/obj00000000000000000023?versionId=null 2023-07-29T00:25:09.064 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000023?versionId=null 2023-07-29T00:25:09.064 DEBUG:boto:Method: HEAD 2023-07-29T00:25:09.065 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000023?versionId=null 2023-07-29T00:25:09.065 DEBUG:boto:Data: 2023-07-29T00:25:09.066 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.066 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:09.066 DEBUG:boto:Port: 8000 2023-07-29T00:25:09.067 DEBUG:boto:Params: {} 2023-07-29T00:25:09.067 DEBUG:boto:Token: None 2023-07-29T00:25:09.068 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000023?versionId=null 2023-07-29T00:25:09.068 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SWtqvFMEvXeGlXgtKU1bFSaiwUo= 2023-07-29T00:25:09.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SWtqvFMEvXeGlXgtKU1bFSaiwUo=', 'Content-Length': '0'} 2023-07-29T00:25:09.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6cb73d0d315e40b-0064c45c65-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2023-07-29T00:25:09.071 DEBUG:boto:path=/hiiqos-58/obj00000000000000000023 2023-07-29T00:25:09.072 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000023 2023-07-29T00:25:09.072 DEBUG:boto:path=/hiiqos-58/obj00000000000000000023?versionId=null 2023-07-29T00:25:09.072 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000023?versionId=null 2023-07-29T00:25:09.073 DEBUG:boto:Method: GET 2023-07-29T00:25:09.073 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000023?versionId=null 2023-07-29T00:25:09.074 DEBUG:boto:Data: 2023-07-29T00:25:09.074 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.075 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:09.075 DEBUG:boto:Port: 8001 2023-07-29T00:25:09.075 DEBUG:boto:Params: {} 2023-07-29T00:25:09.076 DEBUG:boto:Token: None 2023-07-29T00:25:09.077 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000023?versionId=null 2023-07-29T00:25:09.077 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8IeEHgNieY1KfYkYHrPmHNAIpJA= 2023-07-29T00:25:09.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8IeEHgNieY1KfYkYHrPmHNAIpJA=', 'Content-Length': '0'} 2023-07-29T00:25:09.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091b686c005b99481-0064c45c65-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.080 DEBUG:boto:path=/hiiqos-58/obj00000000000000000023 2023-07-29T00:25:09.081 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000023 2023-07-29T00:25:09.081 DEBUG:boto:path=/hiiqos-58/obj00000000000000000023?versionId=null 2023-07-29T00:25:09.081 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000023?versionId=null 2023-07-29T00:25:09.082 DEBUG:boto:Method: GET 2023-07-29T00:25:09.083 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000023?versionId=null 2023-07-29T00:25:09.083 DEBUG:boto:Data: 2023-07-29T00:25:09.084 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.084 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:09.085 DEBUG:boto:Port: 8000 2023-07-29T00:25:09.085 DEBUG:boto:Params: {} 2023-07-29T00:25:09.086 DEBUG:boto:Token: None 2023-07-29T00:25:09.086 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000023?versionId=null 2023-07-29T00:25:09.087 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8IeEHgNieY1KfYkYHrPmHNAIpJA= 2023-07-29T00:25:09.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8IeEHgNieY1KfYkYHrPmHNAIpJA=', 'Content-Length': '0'} 2023-07-29T00:25:09.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c55e1d2c36254da7-0064c45c65-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.090 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:09.090 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:09.091 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000023 2023-07-29T00:25:09.091 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000023 2023-07-29T00:25:09.092 DEBUG:boto:Method: GET 2023-07-29T00:25:09.092 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000023 2023-07-29T00:25:09.093 DEBUG:boto:Data: 2023-07-29T00:25:09.093 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.094 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:09.094 DEBUG:boto:Port: 8001 2023-07-29T00:25:09.095 DEBUG:boto:Params: {} 2023-07-29T00:25:09.095 DEBUG:boto:Token: None 2023-07-29T00:25:09.096 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/ 2023-07-29T00:25:09.096 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:d4avFOwp7+ujArmIRItvC+vgGPg= 2023-07-29T00:25:09.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:d4avFOwp7+ujArmIRItvC+vgGPg=', 'Content-Length': '0'} 2023-07-29T00:25:09.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000472a903f5a22d5e-0064c45c65-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.100 DEBUG:boto:b'hiiqos-58obj000000000000000000231000falseobj000000000000000000232023-07-29T00:24:58.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:09.101 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:09.101 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:09.102 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000023 2023-07-29T00:25:09.102 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000023 2023-07-29T00:25:09.103 DEBUG:boto:Method: GET 2023-07-29T00:25:09.103 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000023 2023-07-29T00:25:09.104 DEBUG:boto:Data: 2023-07-29T00:25:09.104 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.104 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:09.105 DEBUG:boto:Port: 8000 2023-07-29T00:25:09.105 DEBUG:boto:Params: {} 2023-07-29T00:25:09.106 DEBUG:boto:Token: None 2023-07-29T00:25:09.106 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/ 2023-07-29T00:25:09.107 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:d4avFOwp7+ujArmIRItvC+vgGPg= 2023-07-29T00:25:09.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:d4avFOwp7+ujArmIRItvC+vgGPg=', 'Content-Length': '0'} 2023-07-29T00:25:09.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009171ab4735833840-0064c45c65-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.112 DEBUG:boto:b'hiiqos-58obj000000000000000000231000falseobj000000000000000000232023-07-29T00:24:58.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:09.113 DEBUG:boto:path=/hiiqos-58/obj00000000000000000023 2023-07-29T00:25:09.113 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000023 2023-07-29T00:25:09.114 DEBUG:boto:Method: HEAD 2023-07-29T00:25:09.114 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000023 2023-07-29T00:25:09.115 DEBUG:boto:Data: 2023-07-29T00:25:09.115 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.115 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:09.116 DEBUG:boto:Port: 8001 2023-07-29T00:25:09.116 DEBUG:boto:Params: {} 2023-07-29T00:25:09.117 DEBUG:boto:Token: None 2023-07-29T00:25:09.117 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000023 2023-07-29T00:25:09.118 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P/xY6/tie4hJEh207A0WPS5/alQ= 2023-07-29T00:25:09.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P/xY6/tie4hJEh207A0WPS5/alQ=', 'Content-Length': '0'} 2023-07-29T00:25:09.120 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd382fe612c3ed79-0064c45c65-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.121 DEBUG:boto:path=/hiiqos-58/obj00000000000000000023 2023-07-29T00:25:09.121 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000023 2023-07-29T00:25:09.122 DEBUG:boto:Method: HEAD 2023-07-29T00:25:09.122 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000023 2023-07-29T00:25:09.122 DEBUG:boto:Data: 2023-07-29T00:25:09.123 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.123 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:09.124 DEBUG:boto:Port: 8000 2023-07-29T00:25:09.124 DEBUG:boto:Params: {} 2023-07-29T00:25:09.125 DEBUG:boto:Token: None 2023-07-29T00:25:09.125 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000023 2023-07-29T00:25:09.126 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P/xY6/tie4hJEh207A0WPS5/alQ= 2023-07-29T00:25:09.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P/xY6/tie4hJEh207A0WPS5/alQ=', 'Content-Length': '0'} 2023-07-29T00:25:09.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df8ab44a185db2d2-0064c45c65-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2023-07-29T00:25:09.129 DEBUG:boto:path=/hiiqos-58/obj00000000000000000023 2023-07-29T00:25:09.129 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000023 2023-07-29T00:25:09.130 DEBUG:boto:Method: GET 2023-07-29T00:25:09.130 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000023 2023-07-29T00:25:09.130 DEBUG:boto:Data: 2023-07-29T00:25:09.131 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.131 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:09.132 DEBUG:boto:Port: 8001 2023-07-29T00:25:09.132 DEBUG:boto:Params: {} 2023-07-29T00:25:09.133 DEBUG:boto:Token: None 2023-07-29T00:25:09.133 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000023 2023-07-29T00:25:09.133 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8sWWqyq5OUHWIQGIXH5JjxdP3YU= 2023-07-29T00:25:09.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8sWWqyq5OUHWIQGIXH5JjxdP3YU=', 'Content-Length': '0'} 2023-07-29T00:25:09.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026fab52c0f2e8dff-0064c45c65-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.136 DEBUG:boto:path=/hiiqos-58/obj00000000000000000023 2023-07-29T00:25:09.137 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000023 2023-07-29T00:25:09.137 DEBUG:boto:Method: GET 2023-07-29T00:25:09.138 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000023 2023-07-29T00:25:09.138 DEBUG:boto:Data: 2023-07-29T00:25:09.138 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.139 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:09.139 DEBUG:boto:Port: 8000 2023-07-29T00:25:09.139 DEBUG:boto:Params: {} 2023-07-29T00:25:09.140 DEBUG:boto:Token: None 2023-07-29T00:25:09.140 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000023 2023-07-29T00:25:09.141 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8sWWqyq5OUHWIQGIXH5JjxdP3YU= 2023-07-29T00:25:09.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8sWWqyq5OUHWIQGIXH5JjxdP3YU=', 'Content-Length': '0'} 2023-07-29T00:25:09.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b240c6f72c8cac6d-0064c45c65-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.144 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:09.144 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:09.145 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000023 2023-07-29T00:25:09.145 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000023 2023-07-29T00:25:09.146 DEBUG:boto:Method: GET 2023-07-29T00:25:09.146 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000023 2023-07-29T00:25:09.146 DEBUG:boto:Data: 2023-07-29T00:25:09.147 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.147 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:09.147 DEBUG:boto:Port: 8001 2023-07-29T00:25:09.148 DEBUG:boto:Params: {} 2023-07-29T00:25:09.148 DEBUG:boto:Token: None 2023-07-29T00:25:09.149 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/ 2023-07-29T00:25:09.149 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:d4avFOwp7+ujArmIRItvC+vgGPg= 2023-07-29T00:25:09.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:d4avFOwp7+ujArmIRItvC+vgGPg=', 'Content-Length': '0'} 2023-07-29T00:25:09.152 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebb7d8c737164271-0064c45c65-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.153 DEBUG:boto:b'hiiqos-58obj000000000000000000231000falseobj000000000000000000232023-07-29T00:24:58.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:09.153 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:09.154 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:09.154 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000023 2023-07-29T00:25:09.155 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000023 2023-07-29T00:25:09.155 DEBUG:boto:Method: GET 2023-07-29T00:25:09.156 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000023 2023-07-29T00:25:09.156 DEBUG:boto:Data: 2023-07-29T00:25:09.156 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.157 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:09.157 DEBUG:boto:Port: 8000 2023-07-29T00:25:09.158 DEBUG:boto:Params: {} 2023-07-29T00:25:09.158 DEBUG:boto:Token: None 2023-07-29T00:25:09.158 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/ 2023-07-29T00:25:09.159 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:d4avFOwp7+ujArmIRItvC+vgGPg= 2023-07-29T00:25:09.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:d4avFOwp7+ujArmIRItvC+vgGPg=', 'Content-Length': '0'} 2023-07-29T00:25:09.163 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b06393eaf17cc50c-0064c45c65-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.164 DEBUG:boto:b'hiiqos-58obj000000000000000000231000falseobj000000000000000000232023-07-29T00:24:58.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:09.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2023-07-29T00:25:09.165 DEBUG:boto:path=/hiiqos-58/obj00000000000000000024 2023-07-29T00:25:09.165 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000024 2023-07-29T00:25:09.166 DEBUG:boto:path=/hiiqos-58/obj00000000000000000024?versionId=null 2023-07-29T00:25:09.166 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000024?versionId=null 2023-07-29T00:25:09.166 DEBUG:boto:Method: GET 2023-07-29T00:25:09.167 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000024?versionId=null 2023-07-29T00:25:09.167 DEBUG:boto:Data: 2023-07-29T00:25:09.168 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.168 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:09.168 DEBUG:boto:Port: 8001 2023-07-29T00:25:09.169 DEBUG:boto:Params: {} 2023-07-29T00:25:09.169 DEBUG:boto:Token: None 2023-07-29T00:25:09.170 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000024?versionId=null 2023-07-29T00:25:09.170 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jy/kpCOZsSsGEkQy3szNtzQB8bs= 2023-07-29T00:25:09.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jy/kpCOZsSsGEkQy3szNtzQB8bs=', 'Content-Length': '0'} 2023-07-29T00:25:09.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae4f87da9584d4a5-0064c45c65-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.174 DEBUG:boto:path=/hiiqos-58/obj00000000000000000024 2023-07-29T00:25:09.174 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000024 2023-07-29T00:25:09.174 DEBUG:boto:path=/hiiqos-58/obj00000000000000000024?versionId=null 2023-07-29T00:25:09.175 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000024?versionId=null 2023-07-29T00:25:09.175 DEBUG:boto:Method: GET 2023-07-29T00:25:09.176 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000024?versionId=null 2023-07-29T00:25:09.176 DEBUG:boto:Data: 2023-07-29T00:25:09.176 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.177 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:09.177 DEBUG:boto:Port: 8000 2023-07-29T00:25:09.177 DEBUG:boto:Params: {} 2023-07-29T00:25:09.178 DEBUG:boto:Token: None 2023-07-29T00:25:09.178 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000024?versionId=null 2023-07-29T00:25:09.179 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jy/kpCOZsSsGEkQy3szNtzQB8bs= 2023-07-29T00:25:09.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jy/kpCOZsSsGEkQy3szNtzQB8bs=', 'Content-Length': '0'} 2023-07-29T00:25:09.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001765b83350a9f309-0064c45c65-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.182 DEBUG:boto:path=/hiiqos-58/obj00000000000000000024 2023-07-29T00:25:09.182 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000024 2023-07-29T00:25:09.182 DEBUG:boto:path=/hiiqos-58/obj00000000000000000024?versionId=null 2023-07-29T00:25:09.183 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000024?versionId=null 2023-07-29T00:25:09.183 DEBUG:boto:Method: HEAD 2023-07-29T00:25:09.184 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000024?versionId=null 2023-07-29T00:25:09.184 DEBUG:boto:Data: 2023-07-29T00:25:09.184 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.185 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:09.185 DEBUG:boto:Port: 8001 2023-07-29T00:25:09.185 DEBUG:boto:Params: {} 2023-07-29T00:25:09.186 DEBUG:boto:Token: None 2023-07-29T00:25:09.186 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000024?versionId=null 2023-07-29T00:25:09.187 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:944OFkKUuyUnJbhaWIhWPEGXYTc= 2023-07-29T00:25:09.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:944OFkKUuyUnJbhaWIhWPEGXYTc=', 'Content-Length': '0'} 2023-07-29T00:25:09.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cdbe6295e592a57-0064c45c65-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.189 DEBUG:boto:path=/hiiqos-58/obj00000000000000000024 2023-07-29T00:25:09.190 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000024 2023-07-29T00:25:09.190 DEBUG:boto:path=/hiiqos-58/obj00000000000000000024?versionId=null 2023-07-29T00:25:09.191 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000024?versionId=null 2023-07-29T00:25:09.191 DEBUG:boto:Method: HEAD 2023-07-29T00:25:09.191 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000024?versionId=null 2023-07-29T00:25:09.192 DEBUG:boto:Data: 2023-07-29T00:25:09.192 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.193 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:09.193 DEBUG:boto:Port: 8000 2023-07-29T00:25:09.193 DEBUG:boto:Params: {} 2023-07-29T00:25:09.194 DEBUG:boto:Token: None 2023-07-29T00:25:09.194 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000024?versionId=null 2023-07-29T00:25:09.194 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:944OFkKUuyUnJbhaWIhWPEGXYTc= 2023-07-29T00:25:09.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:944OFkKUuyUnJbhaWIhWPEGXYTc=', 'Content-Length': '0'} 2023-07-29T00:25:09.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088ade4287e0704f2-0064c45c65-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2023-07-29T00:25:09.198 DEBUG:boto:path=/hiiqos-58/obj00000000000000000024 2023-07-29T00:25:09.198 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000024 2023-07-29T00:25:09.199 DEBUG:boto:path=/hiiqos-58/obj00000000000000000024?versionId=null 2023-07-29T00:25:09.199 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000024?versionId=null 2023-07-29T00:25:09.199 DEBUG:boto:Method: GET 2023-07-29T00:25:09.200 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000024?versionId=null 2023-07-29T00:25:09.200 DEBUG:boto:Data: 2023-07-29T00:25:09.201 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.201 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:09.201 DEBUG:boto:Port: 8001 2023-07-29T00:25:09.202 DEBUG:boto:Params: {} 2023-07-29T00:25:09.202 DEBUG:boto:Token: None 2023-07-29T00:25:09.203 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000024?versionId=null 2023-07-29T00:25:09.203 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jy/kpCOZsSsGEkQy3szNtzQB8bs= 2023-07-29T00:25:09.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jy/kpCOZsSsGEkQy3szNtzQB8bs=', 'Content-Length': '0'} 2023-07-29T00:25:09.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000899374ae395a726-0064c45c65-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.248 DEBUG:boto:path=/hiiqos-58/obj00000000000000000024 2023-07-29T00:25:09.249 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000024 2023-07-29T00:25:09.249 DEBUG:boto:path=/hiiqos-58/obj00000000000000000024?versionId=null 2023-07-29T00:25:09.250 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000024?versionId=null 2023-07-29T00:25:09.250 DEBUG:boto:Method: GET 2023-07-29T00:25:09.250 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000024?versionId=null 2023-07-29T00:25:09.251 DEBUG:boto:Data: 2023-07-29T00:25:09.251 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.251 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:09.252 DEBUG:boto:Port: 8000 2023-07-29T00:25:09.252 DEBUG:boto:Params: {} 2023-07-29T00:25:09.253 DEBUG:boto:Token: None 2023-07-29T00:25:09.253 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000024?versionId=null 2023-07-29T00:25:09.253 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jy/kpCOZsSsGEkQy3szNtzQB8bs= 2023-07-29T00:25:09.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jy/kpCOZsSsGEkQy3szNtzQB8bs=', 'Content-Length': '0'} 2023-07-29T00:25:09.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000527c7024055c319c-0064c45c65-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.257 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:09.257 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:09.258 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000024 2023-07-29T00:25:09.258 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000024 2023-07-29T00:25:09.259 DEBUG:boto:Method: GET 2023-07-29T00:25:09.259 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000024 2023-07-29T00:25:09.259 DEBUG:boto:Data: 2023-07-29T00:25:09.260 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.260 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:09.261 DEBUG:boto:Port: 8001 2023-07-29T00:25:09.261 DEBUG:boto:Params: {} 2023-07-29T00:25:09.261 DEBUG:boto:Token: None 2023-07-29T00:25:09.262 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/ 2023-07-29T00:25:09.262 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:d4avFOwp7+ujArmIRItvC+vgGPg= 2023-07-29T00:25:09.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:d4avFOwp7+ujArmIRItvC+vgGPg=', 'Content-Length': '0'} 2023-07-29T00:25:09.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000556891cdb2d7edc0-0064c45c65-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.266 DEBUG:boto:b'hiiqos-58obj000000000000000000241000falseobj000000000000000000242023-07-29T00:24:59.435Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:09.267 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:09.267 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:09.268 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000024 2023-07-29T00:25:09.268 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000024 2023-07-29T00:25:09.269 DEBUG:boto:Method: GET 2023-07-29T00:25:09.269 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000024 2023-07-29T00:25:09.269 DEBUG:boto:Data: 2023-07-29T00:25:09.270 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.270 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:09.271 DEBUG:boto:Port: 8000 2023-07-29T00:25:09.271 DEBUG:boto:Params: {} 2023-07-29T00:25:09.271 DEBUG:boto:Token: None 2023-07-29T00:25:09.272 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/ 2023-07-29T00:25:09.272 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:d4avFOwp7+ujArmIRItvC+vgGPg= 2023-07-29T00:25:09.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:d4avFOwp7+ujArmIRItvC+vgGPg=', 'Content-Length': '0'} 2023-07-29T00:25:09.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e9bb14ae4bbf7e8-0064c45c65-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.277 DEBUG:boto:b'hiiqos-58obj000000000000000000241000falseobj000000000000000000242023-07-29T00:24:59.435Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:09.278 DEBUG:boto:path=/hiiqos-58/obj00000000000000000024 2023-07-29T00:25:09.279 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000024 2023-07-29T00:25:09.279 DEBUG:boto:Method: HEAD 2023-07-29T00:25:09.279 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000024 2023-07-29T00:25:09.280 DEBUG:boto:Data: 2023-07-29T00:25:09.280 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.280 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:09.281 DEBUG:boto:Port: 8001 2023-07-29T00:25:09.281 DEBUG:boto:Params: {} 2023-07-29T00:25:09.281 DEBUG:boto:Token: None 2023-07-29T00:25:09.282 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000024 2023-07-29T00:25:09.282 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:x0O4o4aHIkm8gq1PoGNnLlLbym8= 2023-07-29T00:25:09.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:x0O4o4aHIkm8gq1PoGNnLlLbym8=', 'Content-Length': '0'} 2023-07-29T00:25:09.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a71142fea048519f-0064c45c65-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.285 DEBUG:boto:path=/hiiqos-58/obj00000000000000000024 2023-07-29T00:25:09.285 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000024 2023-07-29T00:25:09.286 DEBUG:boto:Method: HEAD 2023-07-29T00:25:09.286 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000024 2023-07-29T00:25:09.286 DEBUG:boto:Data: 2023-07-29T00:25:09.287 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.287 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:09.287 DEBUG:boto:Port: 8000 2023-07-29T00:25:09.288 DEBUG:boto:Params: {} 2023-07-29T00:25:09.288 DEBUG:boto:Token: None 2023-07-29T00:25:09.289 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000024 2023-07-29T00:25:09.289 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:x0O4o4aHIkm8gq1PoGNnLlLbym8= 2023-07-29T00:25:09.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:x0O4o4aHIkm8gq1PoGNnLlLbym8=', 'Content-Length': '0'} 2023-07-29T00:25:09.291 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7b13c588b2028d0-0064c45c65-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2023-07-29T00:25:09.292 DEBUG:boto:path=/hiiqos-58/obj00000000000000000024 2023-07-29T00:25:09.292 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000024 2023-07-29T00:25:09.293 DEBUG:boto:Method: GET 2023-07-29T00:25:09.293 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000024 2023-07-29T00:25:09.294 DEBUG:boto:Data: 2023-07-29T00:25:09.294 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.294 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:09.295 DEBUG:boto:Port: 8001 2023-07-29T00:25:09.295 DEBUG:boto:Params: {} 2023-07-29T00:25:09.295 DEBUG:boto:Token: None 2023-07-29T00:25:09.296 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000024 2023-07-29T00:25:09.296 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:28PPrMbIYCOU8xCpKTi6cp+BiBE= 2023-07-29T00:25:09.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:28PPrMbIYCOU8xCpKTi6cp+BiBE=', 'Content-Length': '0'} 2023-07-29T00:25:09.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d43af72f859e617-0064c45c65-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.344 DEBUG:boto:path=/hiiqos-58/obj00000000000000000024 2023-07-29T00:25:09.345 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000024 2023-07-29T00:25:09.345 DEBUG:boto:Method: GET 2023-07-29T00:25:09.346 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000024 2023-07-29T00:25:09.346 DEBUG:boto:Data: 2023-07-29T00:25:09.347 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.347 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:09.347 DEBUG:boto:Port: 8000 2023-07-29T00:25:09.348 DEBUG:boto:Params: {} 2023-07-29T00:25:09.348 DEBUG:boto:Token: None 2023-07-29T00:25:09.349 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000024 2023-07-29T00:25:09.349 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:28PPrMbIYCOU8xCpKTi6cp+BiBE= 2023-07-29T00:25:09.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:28PPrMbIYCOU8xCpKTi6cp+BiBE=', 'Content-Length': '0'} 2023-07-29T00:25:09.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8c12cc801c4b8a2-0064c45c65-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.354 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:09.354 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:09.355 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000024 2023-07-29T00:25:09.355 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000024 2023-07-29T00:25:09.355 DEBUG:boto:Method: GET 2023-07-29T00:25:09.356 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000024 2023-07-29T00:25:09.356 DEBUG:boto:Data: 2023-07-29T00:25:09.356 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.357 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:09.357 DEBUG:boto:Port: 8001 2023-07-29T00:25:09.357 DEBUG:boto:Params: {} 2023-07-29T00:25:09.358 DEBUG:boto:Token: None 2023-07-29T00:25:09.358 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/ 2023-07-29T00:25:09.360 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:d4avFOwp7+ujArmIRItvC+vgGPg= 2023-07-29T00:25:09.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:d4avFOwp7+ujArmIRItvC+vgGPg=', 'Content-Length': '0'} 2023-07-29T00:25:09.363 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096180970afc9f32a-0064c45c65-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.363 DEBUG:boto:b'hiiqos-58obj000000000000000000241000falseobj000000000000000000242023-07-29T00:24:59.435Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:09.364 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:09.365 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:09.365 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000024 2023-07-29T00:25:09.365 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000024 2023-07-29T00:25:09.366 DEBUG:boto:Method: GET 2023-07-29T00:25:09.366 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000024 2023-07-29T00:25:09.366 DEBUG:boto:Data: 2023-07-29T00:25:09.367 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.367 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:09.367 DEBUG:boto:Port: 8000 2023-07-29T00:25:09.368 DEBUG:boto:Params: {} 2023-07-29T00:25:09.368 DEBUG:boto:Token: None 2023-07-29T00:25:09.369 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/ 2023-07-29T00:25:09.369 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:d4avFOwp7+ujArmIRItvC+vgGPg= 2023-07-29T00:25:09.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:d4avFOwp7+ujArmIRItvC+vgGPg=', 'Content-Length': '0'} 2023-07-29T00:25:09.374 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4c8007e9a13a6b0-0064c45c65-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.376 DEBUG:boto:b'hiiqos-58obj000000000000000000241000falseobj000000000000000000242023-07-29T00:24:59.435Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:09.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2023-07-29T00:25:09.378 DEBUG:boto:path=/hiiqos-58/obj00000000000000000025 2023-07-29T00:25:09.379 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000025 2023-07-29T00:25:09.379 DEBUG:boto:path=/hiiqos-58/obj00000000000000000025?versionId=null 2023-07-29T00:25:09.380 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000025?versionId=null 2023-07-29T00:25:09.381 DEBUG:boto:Method: GET 2023-07-29T00:25:09.382 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000025?versionId=null 2023-07-29T00:25:09.383 DEBUG:boto:Data: 2023-07-29T00:25:09.384 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.385 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:09.386 DEBUG:boto:Port: 8001 2023-07-29T00:25:09.387 DEBUG:boto:Params: {} 2023-07-29T00:25:09.387 DEBUG:boto:Token: None 2023-07-29T00:25:09.388 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000025?versionId=null 2023-07-29T00:25:09.389 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:maI7wg6oPmzgT56K9b1AFptJ2sQ= 2023-07-29T00:25:09.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:maI7wg6oPmzgT56K9b1AFptJ2sQ=', 'Content-Length': '0'} 2023-07-29T00:25:09.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e39617bd178c4f36-0064c45c65-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.393 DEBUG:boto:path=/hiiqos-58/obj00000000000000000025 2023-07-29T00:25:09.394 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000025 2023-07-29T00:25:09.395 DEBUG:boto:path=/hiiqos-58/obj00000000000000000025?versionId=null 2023-07-29T00:25:09.396 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000025?versionId=null 2023-07-29T00:25:09.397 DEBUG:boto:Method: GET 2023-07-29T00:25:09.398 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000025?versionId=null 2023-07-29T00:25:09.399 DEBUG:boto:Data: 2023-07-29T00:25:09.400 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.400 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:09.401 DEBUG:boto:Port: 8000 2023-07-29T00:25:09.402 DEBUG:boto:Params: {} 2023-07-29T00:25:09.403 DEBUG:boto:Token: None 2023-07-29T00:25:09.403 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000025?versionId=null 2023-07-29T00:25:09.404 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:maI7wg6oPmzgT56K9b1AFptJ2sQ= 2023-07-29T00:25:09.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:maI7wg6oPmzgT56K9b1AFptJ2sQ=', 'Content-Length': '0'} 2023-07-29T00:25:09.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b590b53f8899ffb-0064c45c65-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.448 DEBUG:boto:path=/hiiqos-58/obj00000000000000000025 2023-07-29T00:25:09.449 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000025 2023-07-29T00:25:09.449 DEBUG:boto:path=/hiiqos-58/obj00000000000000000025?versionId=null 2023-07-29T00:25:09.450 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000025?versionId=null 2023-07-29T00:25:09.450 DEBUG:boto:Method: HEAD 2023-07-29T00:25:09.451 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000025?versionId=null 2023-07-29T00:25:09.451 DEBUG:boto:Data: 2023-07-29T00:25:09.451 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.452 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:09.452 DEBUG:boto:Port: 8001 2023-07-29T00:25:09.452 DEBUG:boto:Params: {} 2023-07-29T00:25:09.453 DEBUG:boto:Token: None 2023-07-29T00:25:09.453 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000025?versionId=null 2023-07-29T00:25:09.454 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:C3iYkaGgC120he+sz4a7Ht8Fy7I= 2023-07-29T00:25:09.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:C3iYkaGgC120he+sz4a7Ht8Fy7I=', 'Content-Length': '0'} 2023-07-29T00:25:09.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000845d416cfb7203d2-0064c45c65-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.457 DEBUG:boto:path=/hiiqos-58/obj00000000000000000025 2023-07-29T00:25:09.457 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000025 2023-07-29T00:25:09.458 DEBUG:boto:path=/hiiqos-58/obj00000000000000000025?versionId=null 2023-07-29T00:25:09.458 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000025?versionId=null 2023-07-29T00:25:09.458 DEBUG:boto:Method: HEAD 2023-07-29T00:25:09.459 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000025?versionId=null 2023-07-29T00:25:09.459 DEBUG:boto:Data: 2023-07-29T00:25:09.459 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.460 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:09.460 DEBUG:boto:Port: 8000 2023-07-29T00:25:09.460 DEBUG:boto:Params: {} 2023-07-29T00:25:09.461 DEBUG:boto:Token: None 2023-07-29T00:25:09.461 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000025?versionId=null 2023-07-29T00:25:09.461 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:C3iYkaGgC120he+sz4a7Ht8Fy7I= 2023-07-29T00:25:09.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:C3iYkaGgC120he+sz4a7Ht8Fy7I=', 'Content-Length': '0'} 2023-07-29T00:25:09.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b970f8b069ecffec-0064c45c65-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2023-07-29T00:25:09.465 DEBUG:boto:path=/hiiqos-58/obj00000000000000000025 2023-07-29T00:25:09.466 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000025 2023-07-29T00:25:09.466 DEBUG:boto:path=/hiiqos-58/obj00000000000000000025?versionId=null 2023-07-29T00:25:09.466 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000025?versionId=null 2023-07-29T00:25:09.467 DEBUG:boto:Method: GET 2023-07-29T00:25:09.467 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000025?versionId=null 2023-07-29T00:25:09.467 DEBUG:boto:Data: 2023-07-29T00:25:09.468 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.468 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:09.468 DEBUG:boto:Port: 8001 2023-07-29T00:25:09.469 DEBUG:boto:Params: {} 2023-07-29T00:25:09.469 DEBUG:boto:Token: None 2023-07-29T00:25:09.470 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000025?versionId=null 2023-07-29T00:25:09.470 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:maI7wg6oPmzgT56K9b1AFptJ2sQ= 2023-07-29T00:25:09.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:maI7wg6oPmzgT56K9b1AFptJ2sQ=', 'Content-Length': '0'} 2023-07-29T00:25:09.472 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042dff3396e33c4aa-0064c45c65-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.473 DEBUG:boto:path=/hiiqos-58/obj00000000000000000025 2023-07-29T00:25:09.474 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000025 2023-07-29T00:25:09.474 DEBUG:boto:path=/hiiqos-58/obj00000000000000000025?versionId=null 2023-07-29T00:25:09.474 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000025?versionId=null 2023-07-29T00:25:09.475 DEBUG:boto:Method: GET 2023-07-29T00:25:09.475 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000025?versionId=null 2023-07-29T00:25:09.475 DEBUG:boto:Data: 2023-07-29T00:25:09.476 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.476 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:09.477 DEBUG:boto:Port: 8000 2023-07-29T00:25:09.477 DEBUG:boto:Params: {} 2023-07-29T00:25:09.477 DEBUG:boto:Token: None 2023-07-29T00:25:09.478 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000025?versionId=null 2023-07-29T00:25:09.478 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:maI7wg6oPmzgT56K9b1AFptJ2sQ= 2023-07-29T00:25:09.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:maI7wg6oPmzgT56K9b1AFptJ2sQ=', 'Content-Length': '0'} 2023-07-29T00:25:09.480 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076ba0b7105a3e3b0-0064c45c65-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.481 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:09.481 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:09.482 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000025 2023-07-29T00:25:09.482 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000025 2023-07-29T00:25:09.483 DEBUG:boto:Method: GET 2023-07-29T00:25:09.483 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000025 2023-07-29T00:25:09.484 DEBUG:boto:Data: 2023-07-29T00:25:09.484 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.484 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:09.485 DEBUG:boto:Port: 8001 2023-07-29T00:25:09.485 DEBUG:boto:Params: {} 2023-07-29T00:25:09.486 DEBUG:boto:Token: None 2023-07-29T00:25:09.486 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/ 2023-07-29T00:25:09.487 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:d4avFOwp7+ujArmIRItvC+vgGPg= 2023-07-29T00:25:09.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:d4avFOwp7+ujArmIRItvC+vgGPg=', 'Content-Length': '0'} 2023-07-29T00:25:09.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f78cfe0ab4e0509-0064c45c65-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.490 DEBUG:boto:b'hiiqos-58obj000000000000000000251000falseobj000000000000000000252023-07-29T00:24:56.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:09.491 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:09.491 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:09.492 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000025 2023-07-29T00:25:09.492 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000025 2023-07-29T00:25:09.492 DEBUG:boto:Method: GET 2023-07-29T00:25:09.493 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000025 2023-07-29T00:25:09.493 DEBUG:boto:Data: 2023-07-29T00:25:09.493 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.494 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:09.494 DEBUG:boto:Port: 8000 2023-07-29T00:25:09.495 DEBUG:boto:Params: {} 2023-07-29T00:25:09.495 DEBUG:boto:Token: None 2023-07-29T00:25:09.495 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/ 2023-07-29T00:25:09.496 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:d4avFOwp7+ujArmIRItvC+vgGPg= 2023-07-29T00:25:09.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:d4avFOwp7+ujArmIRItvC+vgGPg=', 'Content-Length': '0'} 2023-07-29T00:25:09.500 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fa13201e13145bc-0064c45c65-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.501 DEBUG:boto:b'hiiqos-58obj000000000000000000251000falseobj000000000000000000252023-07-29T00:24:56.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:09.502 DEBUG:boto:path=/hiiqos-58/obj00000000000000000025 2023-07-29T00:25:09.502 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000025 2023-07-29T00:25:09.502 DEBUG:boto:Method: HEAD 2023-07-29T00:25:09.503 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000025 2023-07-29T00:25:09.503 DEBUG:boto:Data: 2023-07-29T00:25:09.503 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.504 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:09.504 DEBUG:boto:Port: 8001 2023-07-29T00:25:09.504 DEBUG:boto:Params: {} 2023-07-29T00:25:09.505 DEBUG:boto:Token: None 2023-07-29T00:25:09.505 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000025 2023-07-29T00:25:09.506 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tbU7pKXh/jr7ubsfZMWvdru8RII= 2023-07-29T00:25:09.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tbU7pKXh/jr7ubsfZMWvdru8RII=', 'Content-Length': '0'} 2023-07-29T00:25:09.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b46ed64a5fb4d74d-0064c45c65-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.508 DEBUG:boto:path=/hiiqos-58/obj00000000000000000025 2023-07-29T00:25:09.509 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000025 2023-07-29T00:25:09.509 DEBUG:boto:Method: HEAD 2023-07-29T00:25:09.509 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000025 2023-07-29T00:25:09.510 DEBUG:boto:Data: 2023-07-29T00:25:09.510 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.510 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:09.511 DEBUG:boto:Port: 8000 2023-07-29T00:25:09.511 DEBUG:boto:Params: {} 2023-07-29T00:25:09.511 DEBUG:boto:Token: None 2023-07-29T00:25:09.512 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000025 2023-07-29T00:25:09.512 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tbU7pKXh/jr7ubsfZMWvdru8RII= 2023-07-29T00:25:09.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tbU7pKXh/jr7ubsfZMWvdru8RII=', 'Content-Length': '0'} 2023-07-29T00:25:09.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e46eec0acaf3e80-0064c45c65-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2023-07-29T00:25:09.516 DEBUG:boto:path=/hiiqos-58/obj00000000000000000025 2023-07-29T00:25:09.517 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000025 2023-07-29T00:25:09.517 DEBUG:boto:Method: GET 2023-07-29T00:25:09.517 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000025 2023-07-29T00:25:09.518 DEBUG:boto:Data: 2023-07-29T00:25:09.518 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.518 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:09.519 DEBUG:boto:Port: 8001 2023-07-29T00:25:09.519 DEBUG:boto:Params: {} 2023-07-29T00:25:09.519 DEBUG:boto:Token: None 2023-07-29T00:25:09.520 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000025 2023-07-29T00:25:09.520 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HY2xX0Vog4cmhJ60zGAW7uKel4o= 2023-07-29T00:25:09.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HY2xX0Vog4cmhJ60zGAW7uKel4o=', 'Content-Length': '0'} 2023-07-29T00:25:09.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048183fa9bf97740a-0064c45c65-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.568 DEBUG:boto:path=/hiiqos-58/obj00000000000000000025 2023-07-29T00:25:09.569 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000025 2023-07-29T00:25:09.569 DEBUG:boto:Method: GET 2023-07-29T00:25:09.570 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000025 2023-07-29T00:25:09.570 DEBUG:boto:Data: 2023-07-29T00:25:09.570 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.571 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:09.571 DEBUG:boto:Port: 8000 2023-07-29T00:25:09.572 DEBUG:boto:Params: {} 2023-07-29T00:25:09.572 DEBUG:boto:Token: None 2023-07-29T00:25:09.572 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000025 2023-07-29T00:25:09.573 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HY2xX0Vog4cmhJ60zGAW7uKel4o= 2023-07-29T00:25:09.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HY2xX0Vog4cmhJ60zGAW7uKel4o=', 'Content-Length': '0'} 2023-07-29T00:25:09.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a74df56cc1222fa9-0064c45c65-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.577 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:09.577 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:09.577 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000025 2023-07-29T00:25:09.578 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000025 2023-07-29T00:25:09.578 DEBUG:boto:Method: GET 2023-07-29T00:25:09.578 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000025 2023-07-29T00:25:09.579 DEBUG:boto:Data: 2023-07-29T00:25:09.579 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.580 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:09.580 DEBUG:boto:Port: 8001 2023-07-29T00:25:09.580 DEBUG:boto:Params: {} 2023-07-29T00:25:09.581 DEBUG:boto:Token: None 2023-07-29T00:25:09.581 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/ 2023-07-29T00:25:09.582 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:d4avFOwp7+ujArmIRItvC+vgGPg= 2023-07-29T00:25:09.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:d4avFOwp7+ujArmIRItvC+vgGPg=', 'Content-Length': '0'} 2023-07-29T00:25:09.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008274bca3f248831d-0064c45c65-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.585 DEBUG:boto:b'hiiqos-58obj000000000000000000251000falseobj000000000000000000252023-07-29T00:24:56.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:09.586 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:09.586 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:09.586 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000025 2023-07-29T00:25:09.587 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000025 2023-07-29T00:25:09.588 DEBUG:boto:Method: GET 2023-07-29T00:25:09.588 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000025 2023-07-29T00:25:09.589 DEBUG:boto:Data: 2023-07-29T00:25:09.589 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.590 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:09.591 DEBUG:boto:Port: 8000 2023-07-29T00:25:09.591 DEBUG:boto:Params: {} 2023-07-29T00:25:09.592 DEBUG:boto:Token: None 2023-07-29T00:25:09.593 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/ 2023-07-29T00:25:09.594 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:d4avFOwp7+ujArmIRItvC+vgGPg= 2023-07-29T00:25:09.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:d4avFOwp7+ujArmIRItvC+vgGPg=', 'Content-Length': '0'} 2023-07-29T00:25:09.599 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a26467e452388f3-0064c45c65-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.600 DEBUG:boto:b'hiiqos-58obj000000000000000000251000falseobj000000000000000000252023-07-29T00:24:56.911Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:09.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2023-07-29T00:25:09.601 DEBUG:boto:path=/hiiqos-58/obj00000000000000000026 2023-07-29T00:25:09.602 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000026 2023-07-29T00:25:09.602 DEBUG:boto:path=/hiiqos-58/obj00000000000000000026?versionId=null 2023-07-29T00:25:09.603 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000026?versionId=null 2023-07-29T00:25:09.603 DEBUG:boto:Method: GET 2023-07-29T00:25:09.604 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000026?versionId=null 2023-07-29T00:25:09.604 DEBUG:boto:Data: 2023-07-29T00:25:09.605 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.605 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:09.605 DEBUG:boto:Port: 8001 2023-07-29T00:25:09.606 DEBUG:boto:Params: {} 2023-07-29T00:25:09.606 DEBUG:boto:Token: None 2023-07-29T00:25:09.607 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000026?versionId=null 2023-07-29T00:25:09.607 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OLANXKNEq5iLwCc0nv0QTEO+q1E= 2023-07-29T00:25:09.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OLANXKNEq5iLwCc0nv0QTEO+q1E=', 'Content-Length': '0'} 2023-07-29T00:25:09.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044b63e329e831f2c-0064c45c65-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.611 DEBUG:boto:path=/hiiqos-58/obj00000000000000000026 2023-07-29T00:25:09.612 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000026 2023-07-29T00:25:09.612 DEBUG:boto:path=/hiiqos-58/obj00000000000000000026?versionId=null 2023-07-29T00:25:09.613 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000026?versionId=null 2023-07-29T00:25:09.613 DEBUG:boto:Method: GET 2023-07-29T00:25:09.613 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000026?versionId=null 2023-07-29T00:25:09.614 DEBUG:boto:Data: 2023-07-29T00:25:09.614 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.615 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:09.615 DEBUG:boto:Port: 8000 2023-07-29T00:25:09.615 DEBUG:boto:Params: {} 2023-07-29T00:25:09.616 DEBUG:boto:Token: None 2023-07-29T00:25:09.616 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000026?versionId=null 2023-07-29T00:25:09.617 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OLANXKNEq5iLwCc0nv0QTEO+q1E= 2023-07-29T00:25:09.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OLANXKNEq5iLwCc0nv0QTEO+q1E=', 'Content-Length': '0'} 2023-07-29T00:25:09.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af3a4777f96dd19d-0064c45c65-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.621 DEBUG:boto:path=/hiiqos-58/obj00000000000000000026 2023-07-29T00:25:09.622 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000026 2023-07-29T00:25:09.623 DEBUG:boto:path=/hiiqos-58/obj00000000000000000026?versionId=null 2023-07-29T00:25:09.623 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000026?versionId=null 2023-07-29T00:25:09.624 DEBUG:boto:Method: HEAD 2023-07-29T00:25:09.625 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000026?versionId=null 2023-07-29T00:25:09.625 DEBUG:boto:Data: 2023-07-29T00:25:09.626 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.626 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:09.627 DEBUG:boto:Port: 8001 2023-07-29T00:25:09.628 DEBUG:boto:Params: {} 2023-07-29T00:25:09.629 DEBUG:boto:Token: None 2023-07-29T00:25:09.630 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000026?versionId=null 2023-07-29T00:25:09.630 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:loitUUr4tEK+L5NADu5NOy04Ook= 2023-07-29T00:25:09.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:loitUUr4tEK+L5NADu5NOy04Ook=', 'Content-Length': '0'} 2023-07-29T00:25:09.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053735b58a7180d46-0064c45c65-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.633 DEBUG:boto:path=/hiiqos-58/obj00000000000000000026 2023-07-29T00:25:09.633 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000026 2023-07-29T00:25:09.634 DEBUG:boto:path=/hiiqos-58/obj00000000000000000026?versionId=null 2023-07-29T00:25:09.634 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000026?versionId=null 2023-07-29T00:25:09.635 DEBUG:boto:Method: HEAD 2023-07-29T00:25:09.635 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000026?versionId=null 2023-07-29T00:25:09.636 DEBUG:boto:Data: 2023-07-29T00:25:09.636 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.637 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:09.638 DEBUG:boto:Port: 8000 2023-07-29T00:25:09.638 DEBUG:boto:Params: {} 2023-07-29T00:25:09.639 DEBUG:boto:Token: None 2023-07-29T00:25:09.639 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000026?versionId=null 2023-07-29T00:25:09.640 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:loitUUr4tEK+L5NADu5NOy04Ook= 2023-07-29T00:25:09.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:loitUUr4tEK+L5NADu5NOy04Ook=', 'Content-Length': '0'} 2023-07-29T00:25:09.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a1f624c1c61f93f-0064c45c65-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2023-07-29T00:25:09.644 DEBUG:boto:path=/hiiqos-58/obj00000000000000000026 2023-07-29T00:25:09.644 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000026 2023-07-29T00:25:09.645 DEBUG:boto:path=/hiiqos-58/obj00000000000000000026?versionId=null 2023-07-29T00:25:09.645 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000026?versionId=null 2023-07-29T00:25:09.646 DEBUG:boto:Method: GET 2023-07-29T00:25:09.646 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000026?versionId=null 2023-07-29T00:25:09.646 DEBUG:boto:Data: 2023-07-29T00:25:09.647 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.648 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:09.648 DEBUG:boto:Port: 8001 2023-07-29T00:25:09.649 DEBUG:boto:Params: {} 2023-07-29T00:25:09.649 DEBUG:boto:Token: None 2023-07-29T00:25:09.649 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000026?versionId=null 2023-07-29T00:25:09.650 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OLANXKNEq5iLwCc0nv0QTEO+q1E= 2023-07-29T00:25:09.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OLANXKNEq5iLwCc0nv0QTEO+q1E=', 'Content-Length': '0'} 2023-07-29T00:25:09.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000709a87221fd33192-0064c45c65-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.700 DEBUG:boto:path=/hiiqos-58/obj00000000000000000026 2023-07-29T00:25:09.701 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000026 2023-07-29T00:25:09.702 DEBUG:boto:path=/hiiqos-58/obj00000000000000000026?versionId=null 2023-07-29T00:25:09.702 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000026?versionId=null 2023-07-29T00:25:09.702 DEBUG:boto:Method: GET 2023-07-29T00:25:09.703 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000026?versionId=null 2023-07-29T00:25:09.703 DEBUG:boto:Data: 2023-07-29T00:25:09.703 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.704 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:09.704 DEBUG:boto:Port: 8000 2023-07-29T00:25:09.705 DEBUG:boto:Params: {} 2023-07-29T00:25:09.705 DEBUG:boto:Token: None 2023-07-29T00:25:09.706 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000026?versionId=null 2023-07-29T00:25:09.706 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OLANXKNEq5iLwCc0nv0QTEO+q1E= 2023-07-29T00:25:09.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OLANXKNEq5iLwCc0nv0QTEO+q1E=', 'Content-Length': '0'} 2023-07-29T00:25:09.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b66f50c932873f11-0064c45c65-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.709 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:09.710 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:09.710 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000026 2023-07-29T00:25:09.710 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000026 2023-07-29T00:25:09.711 DEBUG:boto:Method: GET 2023-07-29T00:25:09.711 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000026 2023-07-29T00:25:09.711 DEBUG:boto:Data: 2023-07-29T00:25:09.712 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.712 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:09.712 DEBUG:boto:Port: 8001 2023-07-29T00:25:09.713 DEBUG:boto:Params: {} 2023-07-29T00:25:09.713 DEBUG:boto:Token: None 2023-07-29T00:25:09.714 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/ 2023-07-29T00:25:09.714 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:d4avFOwp7+ujArmIRItvC+vgGPg= 2023-07-29T00:25:09.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:d4avFOwp7+ujArmIRItvC+vgGPg=', 'Content-Length': '0'} 2023-07-29T00:25:09.720 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000728f90dc14a23158-0064c45c65-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.721 DEBUG:boto:b'hiiqos-58obj000000000000000000261000falseobj000000000000000000262023-07-29T00:24:59.170Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:09.721 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:09.722 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:09.722 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000026 2023-07-29T00:25:09.722 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000026 2023-07-29T00:25:09.723 DEBUG:boto:Method: GET 2023-07-29T00:25:09.723 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000026 2023-07-29T00:25:09.724 DEBUG:boto:Data: 2023-07-29T00:25:09.724 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.724 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:09.725 DEBUG:boto:Port: 8000 2023-07-29T00:25:09.725 DEBUG:boto:Params: {} 2023-07-29T00:25:09.725 DEBUG:boto:Token: None 2023-07-29T00:25:09.726 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/ 2023-07-29T00:25:09.726 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:d4avFOwp7+ujArmIRItvC+vgGPg= 2023-07-29T00:25:09.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:d4avFOwp7+ujArmIRItvC+vgGPg=', 'Content-Length': '0'} 2023-07-29T00:25:09.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002b33bf67f7ebd2c-0064c45c65-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.732 DEBUG:boto:b'hiiqos-58obj000000000000000000261000falseobj000000000000000000262023-07-29T00:24:59.170Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:09.732 DEBUG:boto:path=/hiiqos-58/obj00000000000000000026 2023-07-29T00:25:09.733 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000026 2023-07-29T00:25:09.733 DEBUG:boto:Method: HEAD 2023-07-29T00:25:09.733 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000026 2023-07-29T00:25:09.734 DEBUG:boto:Data: 2023-07-29T00:25:09.734 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.734 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:09.735 DEBUG:boto:Port: 8001 2023-07-29T00:25:09.735 DEBUG:boto:Params: {} 2023-07-29T00:25:09.735 DEBUG:boto:Token: None 2023-07-29T00:25:09.736 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000026 2023-07-29T00:25:09.736 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LG9WROBiukeWqxZB2NbMxwxlKd8= 2023-07-29T00:25:09.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LG9WROBiukeWqxZB2NbMxwxlKd8=', 'Content-Length': '0'} 2023-07-29T00:25:09.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027305cfc9390c1fb-0064c45c65-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.739 DEBUG:boto:path=/hiiqos-58/obj00000000000000000026 2023-07-29T00:25:09.740 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000026 2023-07-29T00:25:09.740 DEBUG:boto:Method: HEAD 2023-07-29T00:25:09.741 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000026 2023-07-29T00:25:09.741 DEBUG:boto:Data: 2023-07-29T00:25:09.741 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.742 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:09.743 DEBUG:boto:Port: 8000 2023-07-29T00:25:09.743 DEBUG:boto:Params: {} 2023-07-29T00:25:09.743 DEBUG:boto:Token: None 2023-07-29T00:25:09.744 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000026 2023-07-29T00:25:09.744 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LG9WROBiukeWqxZB2NbMxwxlKd8= 2023-07-29T00:25:09.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LG9WROBiukeWqxZB2NbMxwxlKd8=', 'Content-Length': '0'} 2023-07-29T00:25:09.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057f250e46b71b05b-0064c45c65-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2023-07-29T00:25:09.748 DEBUG:boto:path=/hiiqos-58/obj00000000000000000026 2023-07-29T00:25:09.749 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000026 2023-07-29T00:25:09.749 DEBUG:boto:Method: GET 2023-07-29T00:25:09.750 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000026 2023-07-29T00:25:09.751 DEBUG:boto:Data: 2023-07-29T00:25:09.751 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.752 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:09.753 DEBUG:boto:Port: 8001 2023-07-29T00:25:09.753 DEBUG:boto:Params: {} 2023-07-29T00:25:09.754 DEBUG:boto:Token: None 2023-07-29T00:25:09.754 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000026 2023-07-29T00:25:09.755 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:j6Ncj1+XUX+GHtpxl2sFjUGT4N8= 2023-07-29T00:25:09.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:j6Ncj1+XUX+GHtpxl2sFjUGT4N8=', 'Content-Length': '0'} 2023-07-29T00:25:09.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb1d6814c2c5f441-0064c45c65-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.758 DEBUG:boto:path=/hiiqos-58/obj00000000000000000026 2023-07-29T00:25:09.758 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000026 2023-07-29T00:25:09.759 DEBUG:boto:Method: GET 2023-07-29T00:25:09.759 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000026 2023-07-29T00:25:09.759 DEBUG:boto:Data: 2023-07-29T00:25:09.760 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.760 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:09.760 DEBUG:boto:Port: 8000 2023-07-29T00:25:09.761 DEBUG:boto:Params: {} 2023-07-29T00:25:09.761 DEBUG:boto:Token: None 2023-07-29T00:25:09.762 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000026 2023-07-29T00:25:09.763 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:j6Ncj1+XUX+GHtpxl2sFjUGT4N8= 2023-07-29T00:25:09.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:j6Ncj1+XUX+GHtpxl2sFjUGT4N8=', 'Content-Length': '0'} 2023-07-29T00:25:09.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b7bd37aef03c274-0064c45c65-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.812 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:09.813 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:09.813 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000026 2023-07-29T00:25:09.814 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000026 2023-07-29T00:25:09.814 DEBUG:boto:Method: GET 2023-07-29T00:25:09.814 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000026 2023-07-29T00:25:09.815 DEBUG:boto:Data: 2023-07-29T00:25:09.815 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.816 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:09.816 DEBUG:boto:Port: 8001 2023-07-29T00:25:09.816 DEBUG:boto:Params: {} 2023-07-29T00:25:09.817 DEBUG:boto:Token: None 2023-07-29T00:25:09.817 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/ 2023-07-29T00:25:09.818 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:d4avFOwp7+ujArmIRItvC+vgGPg= 2023-07-29T00:25:09.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:d4avFOwp7+ujArmIRItvC+vgGPg=', 'Content-Length': '0'} 2023-07-29T00:25:09.820 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcccbdffd86cc171-0064c45c65-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.821 DEBUG:boto:b'hiiqos-58obj000000000000000000261000falseobj000000000000000000262023-07-29T00:24:59.170Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:09.822 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:09.822 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:09.823 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000026 2023-07-29T00:25:09.823 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000026 2023-07-29T00:25:09.823 DEBUG:boto:Method: GET 2023-07-29T00:25:09.824 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000026 2023-07-29T00:25:09.824 DEBUG:boto:Data: 2023-07-29T00:25:09.825 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.825 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:09.825 DEBUG:boto:Port: 8000 2023-07-29T00:25:09.826 DEBUG:boto:Params: {} 2023-07-29T00:25:09.826 DEBUG:boto:Token: None 2023-07-29T00:25:09.827 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/ 2023-07-29T00:25:09.827 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:d4avFOwp7+ujArmIRItvC+vgGPg= 2023-07-29T00:25:09.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:d4avFOwp7+ujArmIRItvC+vgGPg=', 'Content-Length': '0'} 2023-07-29T00:25:09.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006582026d5ed42ade-0064c45c65-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.833 DEBUG:boto:b'hiiqos-58obj000000000000000000261000falseobj000000000000000000262023-07-29T00:24:59.170Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:09.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2023-07-29T00:25:09.834 DEBUG:boto:path=/hiiqos-58/obj00000000000000000027 2023-07-29T00:25:09.834 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000027 2023-07-29T00:25:09.835 DEBUG:boto:path=/hiiqos-58/obj00000000000000000027?versionId=null 2023-07-29T00:25:09.835 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000027?versionId=null 2023-07-29T00:25:09.835 DEBUG:boto:Method: GET 2023-07-29T00:25:09.836 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000027?versionId=null 2023-07-29T00:25:09.836 DEBUG:boto:Data: 2023-07-29T00:25:09.837 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.837 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:09.837 DEBUG:boto:Port: 8001 2023-07-29T00:25:09.838 DEBUG:boto:Params: {} 2023-07-29T00:25:09.838 DEBUG:boto:Token: None 2023-07-29T00:25:09.839 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000027?versionId=null 2023-07-29T00:25:09.839 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fHgB8IHpZZlURefxl1wFQzDTRGQ= 2023-07-29T00:25:09.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fHgB8IHpZZlURefxl1wFQzDTRGQ=', 'Content-Length': '0'} 2023-07-29T00:25:09.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001494b72c0cb8cd7a-0064c45c65-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.884 DEBUG:boto:path=/hiiqos-58/obj00000000000000000027 2023-07-29T00:25:09.885 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000027 2023-07-29T00:25:09.885 DEBUG:boto:path=/hiiqos-58/obj00000000000000000027?versionId=null 2023-07-29T00:25:09.886 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000027?versionId=null 2023-07-29T00:25:09.886 DEBUG:boto:Method: GET 2023-07-29T00:25:09.886 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000027?versionId=null 2023-07-29T00:25:09.887 DEBUG:boto:Data: 2023-07-29T00:25:09.887 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.888 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:09.888 DEBUG:boto:Port: 8000 2023-07-29T00:25:09.888 DEBUG:boto:Params: {} 2023-07-29T00:25:09.889 DEBUG:boto:Token: None 2023-07-29T00:25:09.889 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000027?versionId=null 2023-07-29T00:25:09.890 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fHgB8IHpZZlURefxl1wFQzDTRGQ= 2023-07-29T00:25:09.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fHgB8IHpZZlURefxl1wFQzDTRGQ=', 'Content-Length': '0'} 2023-07-29T00:25:09.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0248ed07fa9b1d1-0064c45c65-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.894 DEBUG:boto:path=/hiiqos-58/obj00000000000000000027 2023-07-29T00:25:09.894 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000027 2023-07-29T00:25:09.894 DEBUG:boto:path=/hiiqos-58/obj00000000000000000027?versionId=null 2023-07-29T00:25:09.895 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000027?versionId=null 2023-07-29T00:25:09.895 DEBUG:boto:Method: HEAD 2023-07-29T00:25:09.895 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000027?versionId=null 2023-07-29T00:25:09.896 DEBUG:boto:Data: 2023-07-29T00:25:09.896 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.896 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:09.897 DEBUG:boto:Port: 8001 2023-07-29T00:25:09.897 DEBUG:boto:Params: {} 2023-07-29T00:25:09.898 DEBUG:boto:Token: None 2023-07-29T00:25:09.898 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000027?versionId=null 2023-07-29T00:25:09.899 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TTw0ctgBeByZIDT4rdRTr1Rx8h4= 2023-07-29T00:25:09.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TTw0ctgBeByZIDT4rdRTr1Rx8h4=', 'Content-Length': '0'} 2023-07-29T00:25:09.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063d56842900bd95e-0064c45c65-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.902 DEBUG:boto:path=/hiiqos-58/obj00000000000000000027 2023-07-29T00:25:09.902 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000027 2023-07-29T00:25:09.902 DEBUG:boto:path=/hiiqos-58/obj00000000000000000027?versionId=null 2023-07-29T00:25:09.903 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000027?versionId=null 2023-07-29T00:25:09.903 DEBUG:boto:Method: HEAD 2023-07-29T00:25:09.903 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000027?versionId=null 2023-07-29T00:25:09.904 DEBUG:boto:Data: 2023-07-29T00:25:09.904 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.904 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:09.905 DEBUG:boto:Port: 8000 2023-07-29T00:25:09.905 DEBUG:boto:Params: {} 2023-07-29T00:25:09.905 DEBUG:boto:Token: None 2023-07-29T00:25:09.906 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000027?versionId=null 2023-07-29T00:25:09.906 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TTw0ctgBeByZIDT4rdRTr1Rx8h4= 2023-07-29T00:25:09.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TTw0ctgBeByZIDT4rdRTr1Rx8h4=', 'Content-Length': '0'} 2023-07-29T00:25:09.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060eb1ef154cdf051-0064c45c65-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2023-07-29T00:25:09.910 DEBUG:boto:path=/hiiqos-58/obj00000000000000000027 2023-07-29T00:25:09.910 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000027 2023-07-29T00:25:09.911 DEBUG:boto:path=/hiiqos-58/obj00000000000000000027?versionId=null 2023-07-29T00:25:09.911 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000027?versionId=null 2023-07-29T00:25:09.912 DEBUG:boto:Method: GET 2023-07-29T00:25:09.912 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000027?versionId=null 2023-07-29T00:25:09.912 DEBUG:boto:Data: 2023-07-29T00:25:09.913 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.913 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:09.913 DEBUG:boto:Port: 8001 2023-07-29T00:25:09.914 DEBUG:boto:Params: {} 2023-07-29T00:25:09.914 DEBUG:boto:Token: None 2023-07-29T00:25:09.915 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000027?versionId=null 2023-07-29T00:25:09.915 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fHgB8IHpZZlURefxl1wFQzDTRGQ= 2023-07-29T00:25:09.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fHgB8IHpZZlURefxl1wFQzDTRGQ=', 'Content-Length': '0'} 2023-07-29T00:25:09.917 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac631a070af9d909-0064c45c65-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.918 DEBUG:boto:path=/hiiqos-58/obj00000000000000000027 2023-07-29T00:25:09.918 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000027 2023-07-29T00:25:09.919 DEBUG:boto:path=/hiiqos-58/obj00000000000000000027?versionId=null 2023-07-29T00:25:09.919 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000027?versionId=null 2023-07-29T00:25:09.919 DEBUG:boto:Method: GET 2023-07-29T00:25:09.920 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000027?versionId=null 2023-07-29T00:25:09.920 DEBUG:boto:Data: 2023-07-29T00:25:09.921 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.921 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:09.921 DEBUG:boto:Port: 8000 2023-07-29T00:25:09.922 DEBUG:boto:Params: {} 2023-07-29T00:25:09.922 DEBUG:boto:Token: None 2023-07-29T00:25:09.922 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/obj00000000000000000027?versionId=null 2023-07-29T00:25:09.923 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fHgB8IHpZZlURefxl1wFQzDTRGQ= 2023-07-29T00:25:09.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fHgB8IHpZZlURefxl1wFQzDTRGQ=', 'Content-Length': '0'} 2023-07-29T00:25:09.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8983df543ba5a98-0064c45c65-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.972 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:09.973 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:09.974 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000027 2023-07-29T00:25:09.974 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000027 2023-07-29T00:25:09.974 DEBUG:boto:Method: GET 2023-07-29T00:25:09.975 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000027 2023-07-29T00:25:09.975 DEBUG:boto:Data: 2023-07-29T00:25:09.975 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.976 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:09.976 DEBUG:boto:Port: 8001 2023-07-29T00:25:09.977 DEBUG:boto:Params: {} 2023-07-29T00:25:09.977 DEBUG:boto:Token: None 2023-07-29T00:25:09.977 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/ 2023-07-29T00:25:09.981 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:d4avFOwp7+ujArmIRItvC+vgGPg= 2023-07-29T00:25:09.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:d4avFOwp7+ujArmIRItvC+vgGPg=', 'Content-Length': '0'} 2023-07-29T00:25:09.984 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b7eca67d43a17ea-0064c45c65-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.984 DEBUG:boto:b'hiiqos-58obj000000000000000000271000falseobj000000000000000000272023-07-29T00:24:59.909Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:09.985 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:09.985 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:09.986 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000027 2023-07-29T00:25:09.986 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000027 2023-07-29T00:25:09.987 DEBUG:boto:Method: GET 2023-07-29T00:25:09.987 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000027 2023-07-29T00:25:09.987 DEBUG:boto:Data: 2023-07-29T00:25:09.988 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.988 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:09.988 DEBUG:boto:Port: 8000 2023-07-29T00:25:09.989 DEBUG:boto:Params: {} 2023-07-29T00:25:09.989 DEBUG:boto:Token: None 2023-07-29T00:25:09.990 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:09 GMT /hiiqos-58/ 2023-07-29T00:25:09.990 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:d4avFOwp7+ujArmIRItvC+vgGPg= 2023-07-29T00:25:09.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:d4avFOwp7+ujArmIRItvC+vgGPg=', 'Content-Length': '0'} 2023-07-29T00:25:09.995 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071d04dd6fe0d4222-0064c45c65-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:09.995 DEBUG:boto:b'hiiqos-58obj000000000000000000271000falseobj000000000000000000272023-07-29T00:24:59.909Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:09.996 DEBUG:boto:path=/hiiqos-58/obj00000000000000000027 2023-07-29T00:25:09.997 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000027 2023-07-29T00:25:09.997 DEBUG:boto:Method: HEAD 2023-07-29T00:25:09.997 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000027 2023-07-29T00:25:09.998 DEBUG:boto:Data: 2023-07-29T00:25:09.998 DEBUG:boto:Headers: {} 2023-07-29T00:25:09.998 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:09.999 DEBUG:boto:Port: 8001 2023-07-29T00:25:09.999 DEBUG:boto:Params: {} 2023-07-29T00:25:09.999 DEBUG:boto:Token: None 2023-07-29T00:25:10.000 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000027 2023-07-29T00:25:10.000 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Or3PYfCP+W7psfBdtOwRcguA8oo= 2023-07-29T00:25:10.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Or3PYfCP+W7psfBdtOwRcguA8oo=', 'Content-Length': '0'} 2023-07-29T00:25:10.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003544b5a18ecedf1e-0064c45c65-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.003 DEBUG:boto:path=/hiiqos-58/obj00000000000000000027 2023-07-29T00:25:10.004 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000027 2023-07-29T00:25:10.004 DEBUG:boto:Method: HEAD 2023-07-29T00:25:10.004 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000027 2023-07-29T00:25:10.005 DEBUG:boto:Data: 2023-07-29T00:25:10.005 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.005 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:10.006 DEBUG:boto:Port: 8000 2023-07-29T00:25:10.006 DEBUG:boto:Params: {} 2023-07-29T00:25:10.007 DEBUG:boto:Token: None 2023-07-29T00:25:10.007 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000027 2023-07-29T00:25:10.007 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Or3PYfCP+W7psfBdtOwRcguA8oo= 2023-07-29T00:25:10.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Or3PYfCP+W7psfBdtOwRcguA8oo=', 'Content-Length': '0'} 2023-07-29T00:25:10.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4aaca031ec0b45f-0064c45c65-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2023-07-29T00:25:10.011 DEBUG:boto:path=/hiiqos-58/obj00000000000000000027 2023-07-29T00:25:10.011 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000027 2023-07-29T00:25:10.012 DEBUG:boto:Method: GET 2023-07-29T00:25:10.012 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000027 2023-07-29T00:25:10.012 DEBUG:boto:Data: 2023-07-29T00:25:10.013 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.013 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:10.014 DEBUG:boto:Port: 8001 2023-07-29T00:25:10.014 DEBUG:boto:Params: {} 2023-07-29T00:25:10.014 DEBUG:boto:Token: None 2023-07-29T00:25:10.015 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000027 2023-07-29T00:25:10.015 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CDnpYCnZaVkILb0WX4/KBjRpJWA= 2023-07-29T00:25:10.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CDnpYCnZaVkILb0WX4/KBjRpJWA=', 'Content-Length': '0'} 2023-07-29T00:25:10.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f470cf04628ef1ab-0064c45c65-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.019 DEBUG:boto:path=/hiiqos-58/obj00000000000000000027 2023-07-29T00:25:10.019 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000027 2023-07-29T00:25:10.019 DEBUG:boto:Method: GET 2023-07-29T00:25:10.020 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000027 2023-07-29T00:25:10.020 DEBUG:boto:Data: 2023-07-29T00:25:10.021 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.021 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:10.021 DEBUG:boto:Port: 8000 2023-07-29T00:25:10.022 DEBUG:boto:Params: {} 2023-07-29T00:25:10.022 DEBUG:boto:Token: None 2023-07-29T00:25:10.023 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000027 2023-07-29T00:25:10.023 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CDnpYCnZaVkILb0WX4/KBjRpJWA= 2023-07-29T00:25:10.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CDnpYCnZaVkILb0WX4/KBjRpJWA=', 'Content-Length': '0'} 2023-07-29T00:25:10.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003797ca26be532098-0064c45c65-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.026 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:10.026 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:10.027 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000027 2023-07-29T00:25:10.027 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000027 2023-07-29T00:25:10.027 DEBUG:boto:Method: GET 2023-07-29T00:25:10.028 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000027 2023-07-29T00:25:10.028 DEBUG:boto:Data: 2023-07-29T00:25:10.029 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.029 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:10.029 DEBUG:boto:Port: 8001 2023-07-29T00:25:10.030 DEBUG:boto:Params: {} 2023-07-29T00:25:10.030 DEBUG:boto:Token: None 2023-07-29T00:25:10.030 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/ 2023-07-29T00:25:10.031 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VLNhdLobIM31WkKxiP5QjruUTBs= 2023-07-29T00:25:10.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VLNhdLobIM31WkKxiP5QjruUTBs=', 'Content-Length': '0'} 2023-07-29T00:25:10.034 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004553210613eeeb70-0064c45c65-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.035 DEBUG:boto:b'hiiqos-58obj000000000000000000271000falseobj000000000000000000272023-07-29T00:24:59.909Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:10.035 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:10.036 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:10.036 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000027 2023-07-29T00:25:10.036 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000027 2023-07-29T00:25:10.037 DEBUG:boto:Method: GET 2023-07-29T00:25:10.037 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000027 2023-07-29T00:25:10.038 DEBUG:boto:Data: 2023-07-29T00:25:10.038 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.038 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:10.039 DEBUG:boto:Port: 8000 2023-07-29T00:25:10.039 DEBUG:boto:Params: {} 2023-07-29T00:25:10.039 DEBUG:boto:Token: None 2023-07-29T00:25:10.040 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/ 2023-07-29T00:25:10.041 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VLNhdLobIM31WkKxiP5QjruUTBs= 2023-07-29T00:25:10.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VLNhdLobIM31WkKxiP5QjruUTBs=', 'Content-Length': '0'} 2023-07-29T00:25:10.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006090300fb2477e45-0064c45c65-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.046 DEBUG:boto:b'hiiqos-58obj000000000000000000271000falseobj000000000000000000272023-07-29T00:24:59.909Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:10.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2023-07-29T00:25:10.047 DEBUG:boto:path=/hiiqos-58/obj00000000000000000028 2023-07-29T00:25:10.047 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000028 2023-07-29T00:25:10.048 DEBUG:boto:path=/hiiqos-58/obj00000000000000000028?versionId=null 2023-07-29T00:25:10.048 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000028?versionId=null 2023-07-29T00:25:10.048 DEBUG:boto:Method: GET 2023-07-29T00:25:10.049 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000028?versionId=null 2023-07-29T00:25:10.049 DEBUG:boto:Data: 2023-07-29T00:25:10.049 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.050 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:10.050 DEBUG:boto:Port: 8001 2023-07-29T00:25:10.050 DEBUG:boto:Params: {} 2023-07-29T00:25:10.051 DEBUG:boto:Token: None 2023-07-29T00:25:10.051 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000028?versionId=null 2023-07-29T00:25:10.052 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:shMLrATofXZ6Cq5bHyck09EH7a0= 2023-07-29T00:25:10.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:shMLrATofXZ6Cq5bHyck09EH7a0=', 'Content-Length': '0'} 2023-07-29T00:25:10.055 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d0e47f6b4f491da-0064c45c66-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.055 DEBUG:boto:path=/hiiqos-58/obj00000000000000000028 2023-07-29T00:25:10.056 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000028 2023-07-29T00:25:10.056 DEBUG:boto:path=/hiiqos-58/obj00000000000000000028?versionId=null 2023-07-29T00:25:10.057 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000028?versionId=null 2023-07-29T00:25:10.057 DEBUG:boto:Method: GET 2023-07-29T00:25:10.057 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000028?versionId=null 2023-07-29T00:25:10.058 DEBUG:boto:Data: 2023-07-29T00:25:10.058 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.058 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:10.059 DEBUG:boto:Port: 8000 2023-07-29T00:25:10.059 DEBUG:boto:Params: {} 2023-07-29T00:25:10.059 DEBUG:boto:Token: None 2023-07-29T00:25:10.060 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000028?versionId=null 2023-07-29T00:25:10.060 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:shMLrATofXZ6Cq5bHyck09EH7a0= 2023-07-29T00:25:10.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:shMLrATofXZ6Cq5bHyck09EH7a0=', 'Content-Length': '0'} 2023-07-29T00:25:10.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a43d8758760e1ebd-0064c45c66-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.104 DEBUG:boto:path=/hiiqos-58/obj00000000000000000028 2023-07-29T00:25:10.105 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000028 2023-07-29T00:25:10.105 DEBUG:boto:path=/hiiqos-58/obj00000000000000000028?versionId=null 2023-07-29T00:25:10.105 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000028?versionId=null 2023-07-29T00:25:10.106 DEBUG:boto:Method: HEAD 2023-07-29T00:25:10.106 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000028?versionId=null 2023-07-29T00:25:10.107 DEBUG:boto:Data: 2023-07-29T00:25:10.107 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.107 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:10.108 DEBUG:boto:Port: 8001 2023-07-29T00:25:10.108 DEBUG:boto:Params: {} 2023-07-29T00:25:10.108 DEBUG:boto:Token: None 2023-07-29T00:25:10.109 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000028?versionId=null 2023-07-29T00:25:10.109 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kGaz4KQYvRQfTPY5LWmK41jl85o= 2023-07-29T00:25:10.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kGaz4KQYvRQfTPY5LWmK41jl85o=', 'Content-Length': '0'} 2023-07-29T00:25:10.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b46f7cc1b5141c59-0064c45c66-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.112 DEBUG:boto:path=/hiiqos-58/obj00000000000000000028 2023-07-29T00:25:10.113 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000028 2023-07-29T00:25:10.113 DEBUG:boto:path=/hiiqos-58/obj00000000000000000028?versionId=null 2023-07-29T00:25:10.113 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000028?versionId=null 2023-07-29T00:25:10.114 DEBUG:boto:Method: HEAD 2023-07-29T00:25:10.114 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000028?versionId=null 2023-07-29T00:25:10.114 DEBUG:boto:Data: 2023-07-29T00:25:10.115 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.115 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:10.115 DEBUG:boto:Port: 8000 2023-07-29T00:25:10.116 DEBUG:boto:Params: {} 2023-07-29T00:25:10.116 DEBUG:boto:Token: None 2023-07-29T00:25:10.116 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000028?versionId=null 2023-07-29T00:25:10.117 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kGaz4KQYvRQfTPY5LWmK41jl85o= 2023-07-29T00:25:10.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kGaz4KQYvRQfTPY5LWmK41jl85o=', 'Content-Length': '0'} 2023-07-29T00:25:10.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009003b21105328b2d-0064c45c66-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2023-07-29T00:25:10.120 DEBUG:boto:path=/hiiqos-58/obj00000000000000000028 2023-07-29T00:25:10.121 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000028 2023-07-29T00:25:10.121 DEBUG:boto:path=/hiiqos-58/obj00000000000000000028?versionId=null 2023-07-29T00:25:10.122 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000028?versionId=null 2023-07-29T00:25:10.122 DEBUG:boto:Method: GET 2023-07-29T00:25:10.122 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000028?versionId=null 2023-07-29T00:25:10.123 DEBUG:boto:Data: 2023-07-29T00:25:10.123 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.123 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:10.124 DEBUG:boto:Port: 8001 2023-07-29T00:25:10.124 DEBUG:boto:Params: {} 2023-07-29T00:25:10.125 DEBUG:boto:Token: None 2023-07-29T00:25:10.125 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000028?versionId=null 2023-07-29T00:25:10.126 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:shMLrATofXZ6Cq5bHyck09EH7a0= 2023-07-29T00:25:10.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:shMLrATofXZ6Cq5bHyck09EH7a0=', 'Content-Length': '0'} 2023-07-29T00:25:10.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046006252ee6e3364-0064c45c66-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.129 DEBUG:boto:path=/hiiqos-58/obj00000000000000000028 2023-07-29T00:25:10.129 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000028 2023-07-29T00:25:10.130 DEBUG:boto:path=/hiiqos-58/obj00000000000000000028?versionId=null 2023-07-29T00:25:10.130 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000028?versionId=null 2023-07-29T00:25:10.130 DEBUG:boto:Method: GET 2023-07-29T00:25:10.131 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000028?versionId=null 2023-07-29T00:25:10.131 DEBUG:boto:Data: 2023-07-29T00:25:10.131 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.132 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:10.132 DEBUG:boto:Port: 8000 2023-07-29T00:25:10.132 DEBUG:boto:Params: {} 2023-07-29T00:25:10.133 DEBUG:boto:Token: None 2023-07-29T00:25:10.133 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000028?versionId=null 2023-07-29T00:25:10.134 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:shMLrATofXZ6Cq5bHyck09EH7a0= 2023-07-29T00:25:10.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:shMLrATofXZ6Cq5bHyck09EH7a0=', 'Content-Length': '0'} 2023-07-29T00:25:10.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c96eb629613b439a-0064c45c66-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.137 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:10.137 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:10.138 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000028 2023-07-29T00:25:10.138 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000028 2023-07-29T00:25:10.139 DEBUG:boto:Method: GET 2023-07-29T00:25:10.139 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000028 2023-07-29T00:25:10.139 DEBUG:boto:Data: 2023-07-29T00:25:10.140 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.140 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:10.140 DEBUG:boto:Port: 8001 2023-07-29T00:25:10.141 DEBUG:boto:Params: {} 2023-07-29T00:25:10.141 DEBUG:boto:Token: None 2023-07-29T00:25:10.141 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/ 2023-07-29T00:25:10.142 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VLNhdLobIM31WkKxiP5QjruUTBs= 2023-07-29T00:25:10.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VLNhdLobIM31WkKxiP5QjruUTBs=', 'Content-Length': '0'} 2023-07-29T00:25:10.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002478480717a10fba-0064c45c66-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.146 DEBUG:boto:b'hiiqos-58obj000000000000000000281000falseobj000000000000000000282023-07-29T00:24:53.541Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:10.148 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:10.149 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:10.149 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000028 2023-07-29T00:25:10.150 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000028 2023-07-29T00:25:10.151 DEBUG:boto:Method: GET 2023-07-29T00:25:10.152 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000028 2023-07-29T00:25:10.152 DEBUG:boto:Data: 2023-07-29T00:25:10.153 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.153 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:10.153 DEBUG:boto:Port: 8000 2023-07-29T00:25:10.154 DEBUG:boto:Params: {} 2023-07-29T00:25:10.154 DEBUG:boto:Token: None 2023-07-29T00:25:10.155 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/ 2023-07-29T00:25:10.155 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VLNhdLobIM31WkKxiP5QjruUTBs= 2023-07-29T00:25:10.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VLNhdLobIM31WkKxiP5QjruUTBs=', 'Content-Length': '0'} 2023-07-29T00:25:10.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdd1e90648ef85de-0064c45c66-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.161 DEBUG:boto:b'hiiqos-58obj000000000000000000281000falseobj000000000000000000282023-07-29T00:24:53.541Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:10.161 DEBUG:boto:path=/hiiqos-58/obj00000000000000000028 2023-07-29T00:25:10.162 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000028 2023-07-29T00:25:10.163 DEBUG:boto:Method: HEAD 2023-07-29T00:25:10.164 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000028 2023-07-29T00:25:10.165 DEBUG:boto:Data: 2023-07-29T00:25:10.166 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.167 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:10.167 DEBUG:boto:Port: 8001 2023-07-29T00:25:10.168 DEBUG:boto:Params: {} 2023-07-29T00:25:10.169 DEBUG:boto:Token: None 2023-07-29T00:25:10.169 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000028 2023-07-29T00:25:10.169 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:megqtzvuPKuu0n5/YZzMlhuHLlg= 2023-07-29T00:25:10.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:megqtzvuPKuu0n5/YZzMlhuHLlg=', 'Content-Length': '0'} 2023-07-29T00:25:10.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2481fedf133cf73-0064c45c66-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.172 DEBUG:boto:path=/hiiqos-58/obj00000000000000000028 2023-07-29T00:25:10.173 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000028 2023-07-29T00:25:10.173 DEBUG:boto:Method: HEAD 2023-07-29T00:25:10.173 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000028 2023-07-29T00:25:10.174 DEBUG:boto:Data: 2023-07-29T00:25:10.174 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.174 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:10.175 DEBUG:boto:Port: 8000 2023-07-29T00:25:10.175 DEBUG:boto:Params: {} 2023-07-29T00:25:10.175 DEBUG:boto:Token: None 2023-07-29T00:25:10.176 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000028 2023-07-29T00:25:10.176 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:megqtzvuPKuu0n5/YZzMlhuHLlg= 2023-07-29T00:25:10.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:megqtzvuPKuu0n5/YZzMlhuHLlg=', 'Content-Length': '0'} 2023-07-29T00:25:10.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7440b552b7a7e54-0064c45c66-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2023-07-29T00:25:10.179 DEBUG:boto:path=/hiiqos-58/obj00000000000000000028 2023-07-29T00:25:10.180 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000028 2023-07-29T00:25:10.180 DEBUG:boto:Method: GET 2023-07-29T00:25:10.180 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000028 2023-07-29T00:25:10.181 DEBUG:boto:Data: 2023-07-29T00:25:10.181 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.182 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:10.182 DEBUG:boto:Port: 8001 2023-07-29T00:25:10.182 DEBUG:boto:Params: {} 2023-07-29T00:25:10.183 DEBUG:boto:Token: None 2023-07-29T00:25:10.183 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000028 2023-07-29T00:25:10.183 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EXNZVnvEBD1JrCR/bHM658ImjOc= 2023-07-29T00:25:10.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EXNZVnvEBD1JrCR/bHM658ImjOc=', 'Content-Length': '0'} 2023-07-29T00:25:10.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ef3b352c737090b-0064c45c66-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.232 DEBUG:boto:path=/hiiqos-58/obj00000000000000000028 2023-07-29T00:25:10.233 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000028 2023-07-29T00:25:10.234 DEBUG:boto:Method: GET 2023-07-29T00:25:10.234 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000028 2023-07-29T00:25:10.234 DEBUG:boto:Data: 2023-07-29T00:25:10.235 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.235 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:10.236 DEBUG:boto:Port: 8000 2023-07-29T00:25:10.236 DEBUG:boto:Params: {} 2023-07-29T00:25:10.237 DEBUG:boto:Token: None 2023-07-29T00:25:10.237 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000028 2023-07-29T00:25:10.238 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EXNZVnvEBD1JrCR/bHM658ImjOc= 2023-07-29T00:25:10.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EXNZVnvEBD1JrCR/bHM658ImjOc=', 'Content-Length': '0'} 2023-07-29T00:25:10.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d10cb46fee2842c-0064c45c66-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.241 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:10.241 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:10.241 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000028 2023-07-29T00:25:10.242 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000028 2023-07-29T00:25:10.242 DEBUG:boto:Method: GET 2023-07-29T00:25:10.243 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000028 2023-07-29T00:25:10.243 DEBUG:boto:Data: 2023-07-29T00:25:10.243 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.244 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:10.244 DEBUG:boto:Port: 8001 2023-07-29T00:25:10.244 DEBUG:boto:Params: {} 2023-07-29T00:25:10.245 DEBUG:boto:Token: None 2023-07-29T00:25:10.245 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/ 2023-07-29T00:25:10.246 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VLNhdLobIM31WkKxiP5QjruUTBs= 2023-07-29T00:25:10.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VLNhdLobIM31WkKxiP5QjruUTBs=', 'Content-Length': '0'} 2023-07-29T00:25:10.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2929245bf3b30a8-0064c45c66-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.249 DEBUG:boto:b'hiiqos-58obj000000000000000000281000falseobj000000000000000000282023-07-29T00:24:53.541Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:10.250 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:10.250 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:10.251 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000028 2023-07-29T00:25:10.251 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000028 2023-07-29T00:25:10.251 DEBUG:boto:Method: GET 2023-07-29T00:25:10.252 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000028 2023-07-29T00:25:10.252 DEBUG:boto:Data: 2023-07-29T00:25:10.253 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.253 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:10.253 DEBUG:boto:Port: 8000 2023-07-29T00:25:10.254 DEBUG:boto:Params: {} 2023-07-29T00:25:10.254 DEBUG:boto:Token: None 2023-07-29T00:25:10.255 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/ 2023-07-29T00:25:10.255 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VLNhdLobIM31WkKxiP5QjruUTBs= 2023-07-29T00:25:10.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VLNhdLobIM31WkKxiP5QjruUTBs=', 'Content-Length': '0'} 2023-07-29T00:25:10.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000797eaf16f23c7bf-0064c45c66-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.260 DEBUG:boto:b'hiiqos-58obj000000000000000000281000falseobj000000000000000000282023-07-29T00:24:53.541Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:10.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2023-07-29T00:25:10.262 DEBUG:boto:path=/hiiqos-58/obj00000000000000000029 2023-07-29T00:25:10.262 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000029 2023-07-29T00:25:10.262 DEBUG:boto:path=/hiiqos-58/obj00000000000000000029?versionId=null 2023-07-29T00:25:10.263 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000029?versionId=null 2023-07-29T00:25:10.263 DEBUG:boto:Method: GET 2023-07-29T00:25:10.263 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000029?versionId=null 2023-07-29T00:25:10.264 DEBUG:boto:Data: 2023-07-29T00:25:10.264 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.265 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:10.265 DEBUG:boto:Port: 8001 2023-07-29T00:25:10.265 DEBUG:boto:Params: {} 2023-07-29T00:25:10.266 DEBUG:boto:Token: None 2023-07-29T00:25:10.266 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000029?versionId=null 2023-07-29T00:25:10.267 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yv6woNo6shDjjli7uBy6I29m/SI= 2023-07-29T00:25:10.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yv6woNo6shDjjli7uBy6I29m/SI=', 'Content-Length': '0'} 2023-07-29T00:25:10.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009505c617880badb1-0064c45c66-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.270 DEBUG:boto:path=/hiiqos-58/obj00000000000000000029 2023-07-29T00:25:10.271 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000029 2023-07-29T00:25:10.271 DEBUG:boto:path=/hiiqos-58/obj00000000000000000029?versionId=null 2023-07-29T00:25:10.272 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000029?versionId=null 2023-07-29T00:25:10.272 DEBUG:boto:Method: GET 2023-07-29T00:25:10.273 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000029?versionId=null 2023-07-29T00:25:10.273 DEBUG:boto:Data: 2023-07-29T00:25:10.273 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.274 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:10.274 DEBUG:boto:Port: 8000 2023-07-29T00:25:10.274 DEBUG:boto:Params: {} 2023-07-29T00:25:10.275 DEBUG:boto:Token: None 2023-07-29T00:25:10.275 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000029?versionId=null 2023-07-29T00:25:10.276 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yv6woNo6shDjjli7uBy6I29m/SI= 2023-07-29T00:25:10.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yv6woNo6shDjjli7uBy6I29m/SI=', 'Content-Length': '0'} 2023-07-29T00:25:10.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6b9dc20ed6253c0-0064c45c66-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.281 DEBUG:boto:path=/hiiqos-58/obj00000000000000000029 2023-07-29T00:25:10.282 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000029 2023-07-29T00:25:10.283 DEBUG:boto:path=/hiiqos-58/obj00000000000000000029?versionId=null 2023-07-29T00:25:10.283 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000029?versionId=null 2023-07-29T00:25:10.284 DEBUG:boto:Method: HEAD 2023-07-29T00:25:10.285 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000029?versionId=null 2023-07-29T00:25:10.285 DEBUG:boto:Data: 2023-07-29T00:25:10.286 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.286 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:10.286 DEBUG:boto:Port: 8001 2023-07-29T00:25:10.287 DEBUG:boto:Params: {} 2023-07-29T00:25:10.287 DEBUG:boto:Token: None 2023-07-29T00:25:10.287 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000029?versionId=null 2023-07-29T00:25:10.288 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WH6zsGPgj0jbhFvJ5sYl/M2yBQ8= 2023-07-29T00:25:10.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WH6zsGPgj0jbhFvJ5sYl/M2yBQ8=', 'Content-Length': '0'} 2023-07-29T00:25:10.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc34f9928309f0ae-0064c45c66-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.291 DEBUG:boto:path=/hiiqos-58/obj00000000000000000029 2023-07-29T00:25:10.291 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000029 2023-07-29T00:25:10.292 DEBUG:boto:path=/hiiqos-58/obj00000000000000000029?versionId=null 2023-07-29T00:25:10.292 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000029?versionId=null 2023-07-29T00:25:10.292 DEBUG:boto:Method: HEAD 2023-07-29T00:25:10.293 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000029?versionId=null 2023-07-29T00:25:10.293 DEBUG:boto:Data: 2023-07-29T00:25:10.293 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.294 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:10.294 DEBUG:boto:Port: 8000 2023-07-29T00:25:10.294 DEBUG:boto:Params: {} 2023-07-29T00:25:10.295 DEBUG:boto:Token: None 2023-07-29T00:25:10.295 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000029?versionId=null 2023-07-29T00:25:10.296 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WH6zsGPgj0jbhFvJ5sYl/M2yBQ8= 2023-07-29T00:25:10.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WH6zsGPgj0jbhFvJ5sYl/M2yBQ8=', 'Content-Length': '0'} 2023-07-29T00:25:10.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbaebbbf2a98118f-0064c45c66-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2023-07-29T00:25:10.299 DEBUG:boto:path=/hiiqos-58/obj00000000000000000029 2023-07-29T00:25:10.299 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000029 2023-07-29T00:25:10.299 DEBUG:boto:path=/hiiqos-58/obj00000000000000000029?versionId=null 2023-07-29T00:25:10.300 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000029?versionId=null 2023-07-29T00:25:10.300 DEBUG:boto:Method: GET 2023-07-29T00:25:10.301 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000029?versionId=null 2023-07-29T00:25:10.301 DEBUG:boto:Data: 2023-07-29T00:25:10.301 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.302 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:10.302 DEBUG:boto:Port: 8001 2023-07-29T00:25:10.302 DEBUG:boto:Params: {} 2023-07-29T00:25:10.303 DEBUG:boto:Token: None 2023-07-29T00:25:10.303 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000029?versionId=null 2023-07-29T00:25:10.304 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yv6woNo6shDjjli7uBy6I29m/SI= 2023-07-29T00:25:10.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yv6woNo6shDjjli7uBy6I29m/SI=', 'Content-Length': '0'} 2023-07-29T00:25:10.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000537b1d5a219ce63d-0064c45c66-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.348 DEBUG:boto:path=/hiiqos-58/obj00000000000000000029 2023-07-29T00:25:10.349 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000029 2023-07-29T00:25:10.349 DEBUG:boto:path=/hiiqos-58/obj00000000000000000029?versionId=null 2023-07-29T00:25:10.350 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000029?versionId=null 2023-07-29T00:25:10.350 DEBUG:boto:Method: GET 2023-07-29T00:25:10.350 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000029?versionId=null 2023-07-29T00:25:10.351 DEBUG:boto:Data: 2023-07-29T00:25:10.351 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.352 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:10.352 DEBUG:boto:Port: 8000 2023-07-29T00:25:10.353 DEBUG:boto:Params: {} 2023-07-29T00:25:10.354 DEBUG:boto:Token: None 2023-07-29T00:25:10.355 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000029?versionId=null 2023-07-29T00:25:10.356 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yv6woNo6shDjjli7uBy6I29m/SI= 2023-07-29T00:25:10.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yv6woNo6shDjjli7uBy6I29m/SI=', 'Content-Length': '0'} 2023-07-29T00:25:10.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d383ff29afdeffe-0064c45c66-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.360 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:10.361 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:10.361 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000029 2023-07-29T00:25:10.362 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000029 2023-07-29T00:25:10.363 DEBUG:boto:Method: GET 2023-07-29T00:25:10.363 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000029 2023-07-29T00:25:10.364 DEBUG:boto:Data: 2023-07-29T00:25:10.365 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.365 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:10.366 DEBUG:boto:Port: 8001 2023-07-29T00:25:10.367 DEBUG:boto:Params: {} 2023-07-29T00:25:10.368 DEBUG:boto:Token: None 2023-07-29T00:25:10.368 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/ 2023-07-29T00:25:10.369 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VLNhdLobIM31WkKxiP5QjruUTBs= 2023-07-29T00:25:10.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VLNhdLobIM31WkKxiP5QjruUTBs=', 'Content-Length': '0'} 2023-07-29T00:25:10.373 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072dc29aedcfa57a5-0064c45c66-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.373 DEBUG:boto:b'hiiqos-58obj000000000000000000291000falseobj000000000000000000292023-07-29T00:24:58.285Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:10.375 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:10.375 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:10.376 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000029 2023-07-29T00:25:10.377 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000029 2023-07-29T00:25:10.378 DEBUG:boto:Method: GET 2023-07-29T00:25:10.379 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000029 2023-07-29T00:25:10.379 DEBUG:boto:Data: 2023-07-29T00:25:10.380 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.381 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:10.382 DEBUG:boto:Port: 8000 2023-07-29T00:25:10.382 DEBUG:boto:Params: {} 2023-07-29T00:25:10.383 DEBUG:boto:Token: None 2023-07-29T00:25:10.383 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/ 2023-07-29T00:25:10.384 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VLNhdLobIM31WkKxiP5QjruUTBs= 2023-07-29T00:25:10.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VLNhdLobIM31WkKxiP5QjruUTBs=', 'Content-Length': '0'} 2023-07-29T00:25:10.388 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006fcff907be91a16-0064c45c66-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.389 DEBUG:boto:b'hiiqos-58obj000000000000000000291000falseobj000000000000000000292023-07-29T00:24:58.285Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:10.390 DEBUG:boto:path=/hiiqos-58/obj00000000000000000029 2023-07-29T00:25:10.390 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000029 2023-07-29T00:25:10.391 DEBUG:boto:Method: HEAD 2023-07-29T00:25:10.392 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000029 2023-07-29T00:25:10.392 DEBUG:boto:Data: 2023-07-29T00:25:10.392 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.393 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:10.393 DEBUG:boto:Port: 8001 2023-07-29T00:25:10.393 DEBUG:boto:Params: {} 2023-07-29T00:25:10.394 DEBUG:boto:Token: None 2023-07-29T00:25:10.394 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000029 2023-07-29T00:25:10.394 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w55ULF0egFKZSrt/rtfBDqOeWqU= 2023-07-29T00:25:10.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w55ULF0egFKZSrt/rtfBDqOeWqU=', 'Content-Length': '0'} 2023-07-29T00:25:10.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bce12f7dd48bea2-0064c45c66-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.397 DEBUG:boto:path=/hiiqos-58/obj00000000000000000029 2023-07-29T00:25:10.397 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000029 2023-07-29T00:25:10.398 DEBUG:boto:Method: HEAD 2023-07-29T00:25:10.398 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000029 2023-07-29T00:25:10.399 DEBUG:boto:Data: 2023-07-29T00:25:10.399 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.399 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:10.400 DEBUG:boto:Port: 8000 2023-07-29T00:25:10.400 DEBUG:boto:Params: {} 2023-07-29T00:25:10.400 DEBUG:boto:Token: None 2023-07-29T00:25:10.401 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000029 2023-07-29T00:25:10.401 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w55ULF0egFKZSrt/rtfBDqOeWqU= 2023-07-29T00:25:10.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w55ULF0egFKZSrt/rtfBDqOeWqU=', 'Content-Length': '0'} 2023-07-29T00:25:10.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6934a4f859de820-0064c45c66-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2023-07-29T00:25:10.404 DEBUG:boto:path=/hiiqos-58/obj00000000000000000029 2023-07-29T00:25:10.405 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000029 2023-07-29T00:25:10.405 DEBUG:boto:Method: GET 2023-07-29T00:25:10.405 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000029 2023-07-29T00:25:10.406 DEBUG:boto:Data: 2023-07-29T00:25:10.406 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.406 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:10.407 DEBUG:boto:Port: 8001 2023-07-29T00:25:10.407 DEBUG:boto:Params: {} 2023-07-29T00:25:10.408 DEBUG:boto:Token: None 2023-07-29T00:25:10.408 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000029 2023-07-29T00:25:10.408 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P6OlrD6Heieuhv1YLR2QYDmpb98= 2023-07-29T00:25:10.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P6OlrD6Heieuhv1YLR2QYDmpb98=', 'Content-Length': '0'} 2023-07-29T00:25:10.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6107d48f0e45039-0064c45c66-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.452 DEBUG:boto:path=/hiiqos-58/obj00000000000000000029 2023-07-29T00:25:10.453 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000029 2023-07-29T00:25:10.453 DEBUG:boto:Method: GET 2023-07-29T00:25:10.454 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000029 2023-07-29T00:25:10.454 DEBUG:boto:Data: 2023-07-29T00:25:10.454 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.455 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:10.455 DEBUG:boto:Port: 8000 2023-07-29T00:25:10.456 DEBUG:boto:Params: {} 2023-07-29T00:25:10.456 DEBUG:boto:Token: None 2023-07-29T00:25:10.457 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000029 2023-07-29T00:25:10.457 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P6OlrD6Heieuhv1YLR2QYDmpb98= 2023-07-29T00:25:10.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P6OlrD6Heieuhv1YLR2QYDmpb98=', 'Content-Length': '0'} 2023-07-29T00:25:10.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fc3c6643d1cd6b9-0064c45c66-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.462 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:10.463 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:10.463 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000029 2023-07-29T00:25:10.463 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000029 2023-07-29T00:25:10.464 DEBUG:boto:Method: GET 2023-07-29T00:25:10.464 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000029 2023-07-29T00:25:10.465 DEBUG:boto:Data: 2023-07-29T00:25:10.465 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.465 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:10.466 DEBUG:boto:Port: 8001 2023-07-29T00:25:10.466 DEBUG:boto:Params: {} 2023-07-29T00:25:10.467 DEBUG:boto:Token: None 2023-07-29T00:25:10.467 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/ 2023-07-29T00:25:10.467 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VLNhdLobIM31WkKxiP5QjruUTBs= 2023-07-29T00:25:10.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VLNhdLobIM31WkKxiP5QjruUTBs=', 'Content-Length': '0'} 2023-07-29T00:25:10.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006785812d3db3106f-0064c45c66-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.472 DEBUG:boto:b'hiiqos-58obj000000000000000000291000falseobj000000000000000000292023-07-29T00:24:58.285Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:10.473 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:10.474 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:10.474 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000029 2023-07-29T00:25:10.474 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000029 2023-07-29T00:25:10.475 DEBUG:boto:Method: GET 2023-07-29T00:25:10.475 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000029 2023-07-29T00:25:10.476 DEBUG:boto:Data: 2023-07-29T00:25:10.476 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.477 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:10.477 DEBUG:boto:Port: 8000 2023-07-29T00:25:10.478 DEBUG:boto:Params: {} 2023-07-29T00:25:10.479 DEBUG:boto:Token: None 2023-07-29T00:25:10.479 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/ 2023-07-29T00:25:10.480 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VLNhdLobIM31WkKxiP5QjruUTBs= 2023-07-29T00:25:10.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VLNhdLobIM31WkKxiP5QjruUTBs=', 'Content-Length': '0'} 2023-07-29T00:25:10.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062ee41be84852a49-0064c45c66-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.487 DEBUG:boto:b'hiiqos-58obj000000000000000000291000falseobj000000000000000000292023-07-29T00:24:58.285Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:10.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2023-07-29T00:25:10.489 DEBUG:boto:path=/hiiqos-58/obj00000000000000000030 2023-07-29T00:25:10.490 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000030 2023-07-29T00:25:10.491 DEBUG:boto:path=/hiiqos-58/obj00000000000000000030?versionId=null 2023-07-29T00:25:10.493 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000030?versionId=null 2023-07-29T00:25:10.494 DEBUG:boto:Method: GET 2023-07-29T00:25:10.494 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000030?versionId=null 2023-07-29T00:25:10.494 DEBUG:boto:Data: 2023-07-29T00:25:10.495 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.495 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:10.495 DEBUG:boto:Port: 8001 2023-07-29T00:25:10.496 DEBUG:boto:Params: {} 2023-07-29T00:25:10.496 DEBUG:boto:Token: None 2023-07-29T00:25:10.497 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000030?versionId=null 2023-07-29T00:25:10.498 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VkLwJR9AHDAvZTpqwLEuzodIwN0= 2023-07-29T00:25:10.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VkLwJR9AHDAvZTpqwLEuzodIwN0=', 'Content-Length': '0'} 2023-07-29T00:25:10.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000988fda75b661f024-0064c45c66-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.501 DEBUG:boto:path=/hiiqos-58/obj00000000000000000030 2023-07-29T00:25:10.502 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000030 2023-07-29T00:25:10.502 DEBUG:boto:path=/hiiqos-58/obj00000000000000000030?versionId=null 2023-07-29T00:25:10.503 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000030?versionId=null 2023-07-29T00:25:10.503 DEBUG:boto:Method: GET 2023-07-29T00:25:10.504 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000030?versionId=null 2023-07-29T00:25:10.504 DEBUG:boto:Data: 2023-07-29T00:25:10.505 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.505 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:10.505 DEBUG:boto:Port: 8000 2023-07-29T00:25:10.506 DEBUG:boto:Params: {} 2023-07-29T00:25:10.506 DEBUG:boto:Token: None 2023-07-29T00:25:10.507 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000030?versionId=null 2023-07-29T00:25:10.507 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VkLwJR9AHDAvZTpqwLEuzodIwN0= 2023-07-29T00:25:10.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VkLwJR9AHDAvZTpqwLEuzodIwN0=', 'Content-Length': '0'} 2023-07-29T00:25:10.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acdf35bb3106f790-0064c45c66-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.510 DEBUG:boto:path=/hiiqos-58/obj00000000000000000030 2023-07-29T00:25:10.511 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000030 2023-07-29T00:25:10.511 DEBUG:boto:path=/hiiqos-58/obj00000000000000000030?versionId=null 2023-07-29T00:25:10.511 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000030?versionId=null 2023-07-29T00:25:10.512 DEBUG:boto:Method: HEAD 2023-07-29T00:25:10.512 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000030?versionId=null 2023-07-29T00:25:10.512 DEBUG:boto:Data: 2023-07-29T00:25:10.513 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.513 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:10.514 DEBUG:boto:Port: 8001 2023-07-29T00:25:10.514 DEBUG:boto:Params: {} 2023-07-29T00:25:10.515 DEBUG:boto:Token: None 2023-07-29T00:25:10.515 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000030?versionId=null 2023-07-29T00:25:10.515 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bG+xZWRYUoSzid51kffMF7dincI= 2023-07-29T00:25:10.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bG+xZWRYUoSzid51kffMF7dincI=', 'Content-Length': '0'} 2023-07-29T00:25:10.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097e5200727a18004-0064c45c66-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.518 DEBUG:boto:path=/hiiqos-58/obj00000000000000000030 2023-07-29T00:25:10.519 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000030 2023-07-29T00:25:10.519 DEBUG:boto:path=/hiiqos-58/obj00000000000000000030?versionId=null 2023-07-29T00:25:10.519 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000030?versionId=null 2023-07-29T00:25:10.520 DEBUG:boto:Method: HEAD 2023-07-29T00:25:10.520 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000030?versionId=null 2023-07-29T00:25:10.520 DEBUG:boto:Data: 2023-07-29T00:25:10.521 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.521 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:10.521 DEBUG:boto:Port: 8000 2023-07-29T00:25:10.522 DEBUG:boto:Params: {} 2023-07-29T00:25:10.522 DEBUG:boto:Token: None 2023-07-29T00:25:10.523 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000030?versionId=null 2023-07-29T00:25:10.523 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bG+xZWRYUoSzid51kffMF7dincI= 2023-07-29T00:25:10.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bG+xZWRYUoSzid51kffMF7dincI=', 'Content-Length': '0'} 2023-07-29T00:25:10.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec8ab5205acbc1a9-0064c45c66-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2023-07-29T00:25:10.530 DEBUG:boto:path=/hiiqos-58/obj00000000000000000030 2023-07-29T00:25:10.531 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000030 2023-07-29T00:25:10.531 DEBUG:boto:path=/hiiqos-58/obj00000000000000000030?versionId=null 2023-07-29T00:25:10.531 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000030?versionId=null 2023-07-29T00:25:10.532 DEBUG:boto:Method: GET 2023-07-29T00:25:10.532 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000030?versionId=null 2023-07-29T00:25:10.533 DEBUG:boto:Data: 2023-07-29T00:25:10.533 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.534 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:10.534 DEBUG:boto:Port: 8001 2023-07-29T00:25:10.534 DEBUG:boto:Params: {} 2023-07-29T00:25:10.535 DEBUG:boto:Token: None 2023-07-29T00:25:10.535 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000030?versionId=null 2023-07-29T00:25:10.536 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VkLwJR9AHDAvZTpqwLEuzodIwN0= 2023-07-29T00:25:10.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VkLwJR9AHDAvZTpqwLEuzodIwN0=', 'Content-Length': '0'} 2023-07-29T00:25:10.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ac203ad23f74629-0064c45c66-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.540 DEBUG:boto:path=/hiiqos-58/obj00000000000000000030 2023-07-29T00:25:10.540 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000030 2023-07-29T00:25:10.541 DEBUG:boto:path=/hiiqos-58/obj00000000000000000030?versionId=null 2023-07-29T00:25:10.541 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000030?versionId=null 2023-07-29T00:25:10.541 DEBUG:boto:Method: GET 2023-07-29T00:25:10.542 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000030?versionId=null 2023-07-29T00:25:10.542 DEBUG:boto:Data: 2023-07-29T00:25:10.542 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.543 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:10.543 DEBUG:boto:Port: 8000 2023-07-29T00:25:10.543 DEBUG:boto:Params: {} 2023-07-29T00:25:10.544 DEBUG:boto:Token: None 2023-07-29T00:25:10.544 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000030?versionId=null 2023-07-29T00:25:10.545 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VkLwJR9AHDAvZTpqwLEuzodIwN0= 2023-07-29T00:25:10.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VkLwJR9AHDAvZTpqwLEuzodIwN0=', 'Content-Length': '0'} 2023-07-29T00:25:10.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a655c20c771934eb-0064c45c66-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.548 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:10.548 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:10.549 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000030 2023-07-29T00:25:10.549 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000030 2023-07-29T00:25:10.550 DEBUG:boto:Method: GET 2023-07-29T00:25:10.550 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000030 2023-07-29T00:25:10.550 DEBUG:boto:Data: 2023-07-29T00:25:10.551 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.551 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:10.551 DEBUG:boto:Port: 8001 2023-07-29T00:25:10.552 DEBUG:boto:Params: {} 2023-07-29T00:25:10.552 DEBUG:boto:Token: None 2023-07-29T00:25:10.553 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/ 2023-07-29T00:25:10.553 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VLNhdLobIM31WkKxiP5QjruUTBs= 2023-07-29T00:25:10.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VLNhdLobIM31WkKxiP5QjruUTBs=', 'Content-Length': '0'} 2023-07-29T00:25:10.556 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000122487336fbec3a5-0064c45c66-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.557 DEBUG:boto:b'hiiqos-58obj000000000000000000301000falseobj000000000000000000302023-07-29T00:24:59.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:10.557 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:10.558 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:10.558 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000030 2023-07-29T00:25:10.558 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000030 2023-07-29T00:25:10.559 DEBUG:boto:Method: GET 2023-07-29T00:25:10.559 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000030 2023-07-29T00:25:10.559 DEBUG:boto:Data: 2023-07-29T00:25:10.560 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.560 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:10.561 DEBUG:boto:Port: 8000 2023-07-29T00:25:10.561 DEBUG:boto:Params: {} 2023-07-29T00:25:10.561 DEBUG:boto:Token: None 2023-07-29T00:25:10.562 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/ 2023-07-29T00:25:10.562 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VLNhdLobIM31WkKxiP5QjruUTBs= 2023-07-29T00:25:10.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VLNhdLobIM31WkKxiP5QjruUTBs=', 'Content-Length': '0'} 2023-07-29T00:25:10.567 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044c7c2d91a74aadf-0064c45c66-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.567 DEBUG:boto:b'hiiqos-58obj000000000000000000301000falseobj000000000000000000302023-07-29T00:24:59.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:10.568 DEBUG:boto:path=/hiiqos-58/obj00000000000000000030 2023-07-29T00:25:10.568 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000030 2023-07-29T00:25:10.569 DEBUG:boto:Method: HEAD 2023-07-29T00:25:10.569 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000030 2023-07-29T00:25:10.570 DEBUG:boto:Data: 2023-07-29T00:25:10.570 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.570 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:10.570 DEBUG:boto:Port: 8001 2023-07-29T00:25:10.571 DEBUG:boto:Params: {} 2023-07-29T00:25:10.571 DEBUG:boto:Token: None 2023-07-29T00:25:10.572 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000030 2023-07-29T00:25:10.572 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UNxW8lPLNyphmSFtiGec53ZfYJE= 2023-07-29T00:25:10.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UNxW8lPLNyphmSFtiGec53ZfYJE=', 'Content-Length': '0'} 2023-07-29T00:25:10.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cb21a59eefa1849-0064c45c66-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.575 DEBUG:boto:path=/hiiqos-58/obj00000000000000000030 2023-07-29T00:25:10.576 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000030 2023-07-29T00:25:10.576 DEBUG:boto:Method: HEAD 2023-07-29T00:25:10.576 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000030 2023-07-29T00:25:10.577 DEBUG:boto:Data: 2023-07-29T00:25:10.577 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.577 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:10.578 DEBUG:boto:Port: 8000 2023-07-29T00:25:10.578 DEBUG:boto:Params: {} 2023-07-29T00:25:10.578 DEBUG:boto:Token: None 2023-07-29T00:25:10.579 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000030 2023-07-29T00:25:10.579 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UNxW8lPLNyphmSFtiGec53ZfYJE= 2023-07-29T00:25:10.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UNxW8lPLNyphmSFtiGec53ZfYJE=', 'Content-Length': '0'} 2023-07-29T00:25:10.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f70e929bdf4fa93-0064c45c66-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2023-07-29T00:25:10.583 DEBUG:boto:path=/hiiqos-58/obj00000000000000000030 2023-07-29T00:25:10.583 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000030 2023-07-29T00:25:10.583 DEBUG:boto:Method: GET 2023-07-29T00:25:10.584 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000030 2023-07-29T00:25:10.584 DEBUG:boto:Data: 2023-07-29T00:25:10.585 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.585 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:10.585 DEBUG:boto:Port: 8001 2023-07-29T00:25:10.586 DEBUG:boto:Params: {} 2023-07-29T00:25:10.586 DEBUG:boto:Token: None 2023-07-29T00:25:10.587 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000030 2023-07-29T00:25:10.587 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Q1O/eeEy2w03QqhmcCdMmq7b6c0= 2023-07-29T00:25:10.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q1O/eeEy2w03QqhmcCdMmq7b6c0=', 'Content-Length': '0'} 2023-07-29T00:25:10.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002149b8e4d8ed04fd-0064c45c66-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.632 DEBUG:boto:path=/hiiqos-58/obj00000000000000000030 2023-07-29T00:25:10.633 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000030 2023-07-29T00:25:10.634 DEBUG:boto:Method: GET 2023-07-29T00:25:10.635 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000030 2023-07-29T00:25:10.635 DEBUG:boto:Data: 2023-07-29T00:25:10.636 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.637 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:10.638 DEBUG:boto:Port: 8000 2023-07-29T00:25:10.638 DEBUG:boto:Params: {} 2023-07-29T00:25:10.639 DEBUG:boto:Token: None 2023-07-29T00:25:10.640 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000030 2023-07-29T00:25:10.641 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Q1O/eeEy2w03QqhmcCdMmq7b6c0= 2023-07-29T00:25:10.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q1O/eeEy2w03QqhmcCdMmq7b6c0=', 'Content-Length': '0'} 2023-07-29T00:25:10.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e060b76d59d584cb-0064c45c66-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.645 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:10.645 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:10.646 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000030 2023-07-29T00:25:10.646 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000030 2023-07-29T00:25:10.646 DEBUG:boto:Method: GET 2023-07-29T00:25:10.647 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000030 2023-07-29T00:25:10.648 DEBUG:boto:Data: 2023-07-29T00:25:10.648 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.649 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:10.650 DEBUG:boto:Port: 8001 2023-07-29T00:25:10.651 DEBUG:boto:Params: {} 2023-07-29T00:25:10.651 DEBUG:boto:Token: None 2023-07-29T00:25:10.652 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/ 2023-07-29T00:25:10.654 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VLNhdLobIM31WkKxiP5QjruUTBs= 2023-07-29T00:25:10.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VLNhdLobIM31WkKxiP5QjruUTBs=', 'Content-Length': '0'} 2023-07-29T00:25:10.657 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e3afe920d52feeb-0064c45c66-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.658 DEBUG:boto:b'hiiqos-58obj000000000000000000301000falseobj000000000000000000302023-07-29T00:24:59.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:10.659 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:10.660 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:10.660 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000030 2023-07-29T00:25:10.661 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000030 2023-07-29T00:25:10.661 DEBUG:boto:Method: GET 2023-07-29T00:25:10.661 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000030 2023-07-29T00:25:10.662 DEBUG:boto:Data: 2023-07-29T00:25:10.662 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.662 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:10.663 DEBUG:boto:Port: 8000 2023-07-29T00:25:10.663 DEBUG:boto:Params: {} 2023-07-29T00:25:10.664 DEBUG:boto:Token: None 2023-07-29T00:25:10.664 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/ 2023-07-29T00:25:10.665 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VLNhdLobIM31WkKxiP5QjruUTBs= 2023-07-29T00:25:10.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VLNhdLobIM31WkKxiP5QjruUTBs=', 'Content-Length': '0'} 2023-07-29T00:25:10.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000122bcd6d2ac87ccb-0064c45c66-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.670 DEBUG:boto:b'hiiqos-58obj000000000000000000301000falseobj000000000000000000302023-07-29T00:24:59.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:10.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2023-07-29T00:25:10.672 DEBUG:boto:path=/hiiqos-58/obj00000000000000000031 2023-07-29T00:25:10.672 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000031 2023-07-29T00:25:10.673 DEBUG:boto:path=/hiiqos-58/obj00000000000000000031?versionId=null 2023-07-29T00:25:10.673 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000031?versionId=null 2023-07-29T00:25:10.673 DEBUG:boto:Method: GET 2023-07-29T00:25:10.674 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000031?versionId=null 2023-07-29T00:25:10.675 DEBUG:boto:Data: 2023-07-29T00:25:10.675 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.676 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:10.677 DEBUG:boto:Port: 8001 2023-07-29T00:25:10.679 DEBUG:boto:Params: {} 2023-07-29T00:25:10.680 DEBUG:boto:Token: None 2023-07-29T00:25:10.681 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000031?versionId=null 2023-07-29T00:25:10.682 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:E+OXf5ufsaFhMDcfTiqOCkhmqyk= 2023-07-29T00:25:10.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:E+OXf5ufsaFhMDcfTiqOCkhmqyk=', 'Content-Length': '0'} 2023-07-29T00:25:10.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007dffca67d085838c-0064c45c66-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.685 DEBUG:boto:path=/hiiqos-58/obj00000000000000000031 2023-07-29T00:25:10.685 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000031 2023-07-29T00:25:10.686 DEBUG:boto:path=/hiiqos-58/obj00000000000000000031?versionId=null 2023-07-29T00:25:10.686 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000031?versionId=null 2023-07-29T00:25:10.687 DEBUG:boto:Method: GET 2023-07-29T00:25:10.687 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000031?versionId=null 2023-07-29T00:25:10.687 DEBUG:boto:Data: 2023-07-29T00:25:10.688 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.688 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:10.689 DEBUG:boto:Port: 8000 2023-07-29T00:25:10.689 DEBUG:boto:Params: {} 2023-07-29T00:25:10.689 DEBUG:boto:Token: None 2023-07-29T00:25:10.690 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000031?versionId=null 2023-07-29T00:25:10.691 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:E+OXf5ufsaFhMDcfTiqOCkhmqyk= 2023-07-29T00:25:10.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:E+OXf5ufsaFhMDcfTiqOCkhmqyk=', 'Content-Length': '0'} 2023-07-29T00:25:10.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a17daeaf7ebb0011-0064c45c66-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.696 DEBUG:boto:path=/hiiqos-58/obj00000000000000000031 2023-07-29T00:25:10.696 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000031 2023-07-29T00:25:10.696 DEBUG:boto:path=/hiiqos-58/obj00000000000000000031?versionId=null 2023-07-29T00:25:10.697 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000031?versionId=null 2023-07-29T00:25:10.697 DEBUG:boto:Method: HEAD 2023-07-29T00:25:10.697 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000031?versionId=null 2023-07-29T00:25:10.698 DEBUG:boto:Data: 2023-07-29T00:25:10.698 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.699 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:10.699 DEBUG:boto:Port: 8001 2023-07-29T00:25:10.699 DEBUG:boto:Params: {} 2023-07-29T00:25:10.700 DEBUG:boto:Token: None 2023-07-29T00:25:10.700 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000031?versionId=null 2023-07-29T00:25:10.701 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Rz+382D5qdBUAWSiB2cLTwle+JM= 2023-07-29T00:25:10.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Rz+382D5qdBUAWSiB2cLTwle+JM=', 'Content-Length': '0'} 2023-07-29T00:25:10.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a94c108edc81693-0064c45c66-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.703 DEBUG:boto:path=/hiiqos-58/obj00000000000000000031 2023-07-29T00:25:10.704 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000031 2023-07-29T00:25:10.704 DEBUG:boto:path=/hiiqos-58/obj00000000000000000031?versionId=null 2023-07-29T00:25:10.705 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000031?versionId=null 2023-07-29T00:25:10.706 DEBUG:boto:Method: HEAD 2023-07-29T00:25:10.706 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000031?versionId=null 2023-07-29T00:25:10.707 DEBUG:boto:Data: 2023-07-29T00:25:10.707 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.708 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:10.709 DEBUG:boto:Port: 8000 2023-07-29T00:25:10.709 DEBUG:boto:Params: {} 2023-07-29T00:25:10.710 DEBUG:boto:Token: None 2023-07-29T00:25:10.712 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000031?versionId=null 2023-07-29T00:25:10.713 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Rz+382D5qdBUAWSiB2cLTwle+JM= 2023-07-29T00:25:10.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Rz+382D5qdBUAWSiB2cLTwle+JM=', 'Content-Length': '0'} 2023-07-29T00:25:10.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bf43d358fed76e1-0064c45c66-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2023-07-29T00:25:10.718 DEBUG:boto:path=/hiiqos-58/obj00000000000000000031 2023-07-29T00:25:10.718 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000031 2023-07-29T00:25:10.718 DEBUG:boto:path=/hiiqos-58/obj00000000000000000031?versionId=null 2023-07-29T00:25:10.719 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000031?versionId=null 2023-07-29T00:25:10.719 DEBUG:boto:Method: GET 2023-07-29T00:25:10.719 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000031?versionId=null 2023-07-29T00:25:10.720 DEBUG:boto:Data: 2023-07-29T00:25:10.721 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.721 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:10.722 DEBUG:boto:Port: 8001 2023-07-29T00:25:10.722 DEBUG:boto:Params: {} 2023-07-29T00:25:10.722 DEBUG:boto:Token: None 2023-07-29T00:25:10.723 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000031?versionId=null 2023-07-29T00:25:10.723 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:E+OXf5ufsaFhMDcfTiqOCkhmqyk= 2023-07-29T00:25:10.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:E+OXf5ufsaFhMDcfTiqOCkhmqyk=', 'Content-Length': '0'} 2023-07-29T00:25:10.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6a1cd8e96f81f36-0064c45c66-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.772 DEBUG:boto:path=/hiiqos-58/obj00000000000000000031 2023-07-29T00:25:10.773 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000031 2023-07-29T00:25:10.774 DEBUG:boto:path=/hiiqos-58/obj00000000000000000031?versionId=null 2023-07-29T00:25:10.774 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000031?versionId=null 2023-07-29T00:25:10.774 DEBUG:boto:Method: GET 2023-07-29T00:25:10.775 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000031?versionId=null 2023-07-29T00:25:10.775 DEBUG:boto:Data: 2023-07-29T00:25:10.775 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.776 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:10.776 DEBUG:boto:Port: 8000 2023-07-29T00:25:10.777 DEBUG:boto:Params: {} 2023-07-29T00:25:10.777 DEBUG:boto:Token: None 2023-07-29T00:25:10.777 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000031?versionId=null 2023-07-29T00:25:10.778 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:E+OXf5ufsaFhMDcfTiqOCkhmqyk= 2023-07-29T00:25:10.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:E+OXf5ufsaFhMDcfTiqOCkhmqyk=', 'Content-Length': '0'} 2023-07-29T00:25:10.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000452df7dd05210a70-0064c45c66-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.781 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:10.782 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:10.782 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000031 2023-07-29T00:25:10.783 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000031 2023-07-29T00:25:10.783 DEBUG:boto:Method: GET 2023-07-29T00:25:10.783 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000031 2023-07-29T00:25:10.784 DEBUG:boto:Data: 2023-07-29T00:25:10.784 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.784 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:10.785 DEBUG:boto:Port: 8001 2023-07-29T00:25:10.785 DEBUG:boto:Params: {} 2023-07-29T00:25:10.785 DEBUG:boto:Token: None 2023-07-29T00:25:10.786 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/ 2023-07-29T00:25:10.786 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VLNhdLobIM31WkKxiP5QjruUTBs= 2023-07-29T00:25:10.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VLNhdLobIM31WkKxiP5QjruUTBs=', 'Content-Length': '0'} 2023-07-29T00:25:10.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a3177966280feb5-0064c45c66-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.790 DEBUG:boto:b'hiiqos-58obj000000000000000000311000falseobj000000000000000000312023-07-29T00:24:57.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:10.791 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:10.791 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:10.791 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000031 2023-07-29T00:25:10.792 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000031 2023-07-29T00:25:10.792 DEBUG:boto:Method: GET 2023-07-29T00:25:10.793 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000031 2023-07-29T00:25:10.793 DEBUG:boto:Data: 2023-07-29T00:25:10.793 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.794 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:10.794 DEBUG:boto:Port: 8000 2023-07-29T00:25:10.794 DEBUG:boto:Params: {} 2023-07-29T00:25:10.795 DEBUG:boto:Token: None 2023-07-29T00:25:10.795 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/ 2023-07-29T00:25:10.796 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VLNhdLobIM31WkKxiP5QjruUTBs= 2023-07-29T00:25:10.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VLNhdLobIM31WkKxiP5QjruUTBs=', 'Content-Length': '0'} 2023-07-29T00:25:10.800 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9cd4717c0baa0b6-0064c45c66-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.801 DEBUG:boto:b'hiiqos-58obj000000000000000000311000falseobj000000000000000000312023-07-29T00:24:57.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:10.802 DEBUG:boto:path=/hiiqos-58/obj00000000000000000031 2023-07-29T00:25:10.802 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000031 2023-07-29T00:25:10.802 DEBUG:boto:Method: HEAD 2023-07-29T00:25:10.803 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000031 2023-07-29T00:25:10.803 DEBUG:boto:Data: 2023-07-29T00:25:10.804 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.804 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:10.804 DEBUG:boto:Port: 8001 2023-07-29T00:25:10.805 DEBUG:boto:Params: {} 2023-07-29T00:25:10.806 DEBUG:boto:Token: None 2023-07-29T00:25:10.806 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000031 2023-07-29T00:25:10.807 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:y4g3BNHkL4CavENzuGLWs5YCBRw= 2023-07-29T00:25:10.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:y4g3BNHkL4CavENzuGLWs5YCBRw=', 'Content-Length': '0'} 2023-07-29T00:25:10.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7ae882b682468ac-0064c45c66-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.811 DEBUG:boto:path=/hiiqos-58/obj00000000000000000031 2023-07-29T00:25:10.811 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000031 2023-07-29T00:25:10.812 DEBUG:boto:Method: HEAD 2023-07-29T00:25:10.812 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000031 2023-07-29T00:25:10.813 DEBUG:boto:Data: 2023-07-29T00:25:10.813 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.813 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:10.814 DEBUG:boto:Port: 8000 2023-07-29T00:25:10.814 DEBUG:boto:Params: {} 2023-07-29T00:25:10.814 DEBUG:boto:Token: None 2023-07-29T00:25:10.815 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000031 2023-07-29T00:25:10.815 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:y4g3BNHkL4CavENzuGLWs5YCBRw= 2023-07-29T00:25:10.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:y4g3BNHkL4CavENzuGLWs5YCBRw=', 'Content-Length': '0'} 2023-07-29T00:25:10.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d77391026aa608e-0064c45c66-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2023-07-29T00:25:10.819 DEBUG:boto:path=/hiiqos-58/obj00000000000000000031 2023-07-29T00:25:10.819 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000031 2023-07-29T00:25:10.820 DEBUG:boto:Method: GET 2023-07-29T00:25:10.820 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000031 2023-07-29T00:25:10.820 DEBUG:boto:Data: 2023-07-29T00:25:10.821 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.821 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:10.821 DEBUG:boto:Port: 8001 2023-07-29T00:25:10.822 DEBUG:boto:Params: {} 2023-07-29T00:25:10.822 DEBUG:boto:Token: None 2023-07-29T00:25:10.823 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000031 2023-07-29T00:25:10.823 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ob9VPwO/W3Ec0vl4ZGV2Z1huu+M= 2023-07-29T00:25:10.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ob9VPwO/W3Ec0vl4ZGV2Z1huu+M=', 'Content-Length': '0'} 2023-07-29T00:25:10.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013ac7bb9f32c4798-0064c45c66-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.826 DEBUG:boto:path=/hiiqos-58/obj00000000000000000031 2023-07-29T00:25:10.827 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000031 2023-07-29T00:25:10.827 DEBUG:boto:Method: GET 2023-07-29T00:25:10.827 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000031 2023-07-29T00:25:10.828 DEBUG:boto:Data: 2023-07-29T00:25:10.828 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.829 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:10.829 DEBUG:boto:Port: 8000 2023-07-29T00:25:10.829 DEBUG:boto:Params: {} 2023-07-29T00:25:10.830 DEBUG:boto:Token: None 2023-07-29T00:25:10.830 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000031 2023-07-29T00:25:10.831 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ob9VPwO/W3Ec0vl4ZGV2Z1huu+M= 2023-07-29T00:25:10.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ob9VPwO/W3Ec0vl4ZGV2Z1huu+M=', 'Content-Length': '0'} 2023-07-29T00:25:10.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059830f4ef0b93c3f-0064c45c66-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.876 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:10.877 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:10.878 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000031 2023-07-29T00:25:10.878 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000031 2023-07-29T00:25:10.878 DEBUG:boto:Method: GET 2023-07-29T00:25:10.879 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000031 2023-07-29T00:25:10.879 DEBUG:boto:Data: 2023-07-29T00:25:10.880 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.880 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:10.880 DEBUG:boto:Port: 8001 2023-07-29T00:25:10.881 DEBUG:boto:Params: {} 2023-07-29T00:25:10.881 DEBUG:boto:Token: None 2023-07-29T00:25:10.882 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/ 2023-07-29T00:25:10.882 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VLNhdLobIM31WkKxiP5QjruUTBs= 2023-07-29T00:25:10.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VLNhdLobIM31WkKxiP5QjruUTBs=', 'Content-Length': '0'} 2023-07-29T00:25:10.885 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000306d754502a6e9ea-0064c45c66-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.886 DEBUG:boto:b'hiiqos-58obj000000000000000000311000falseobj000000000000000000312023-07-29T00:24:57.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:10.886 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:10.887 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:10.887 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000031 2023-07-29T00:25:10.887 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000031 2023-07-29T00:25:10.888 DEBUG:boto:Method: GET 2023-07-29T00:25:10.888 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000031 2023-07-29T00:25:10.889 DEBUG:boto:Data: 2023-07-29T00:25:10.889 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.889 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:10.890 DEBUG:boto:Port: 8000 2023-07-29T00:25:10.890 DEBUG:boto:Params: {} 2023-07-29T00:25:10.890 DEBUG:boto:Token: None 2023-07-29T00:25:10.891 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/ 2023-07-29T00:25:10.891 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VLNhdLobIM31WkKxiP5QjruUTBs= 2023-07-29T00:25:10.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VLNhdLobIM31WkKxiP5QjruUTBs=', 'Content-Length': '0'} 2023-07-29T00:25:10.896 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025f47ae4138edc22-0064c45c66-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.897 DEBUG:boto:b'hiiqos-58obj000000000000000000311000falseobj000000000000000000312023-07-29T00:24:57.013Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:10.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2023-07-29T00:25:10.898 DEBUG:boto:path=/hiiqos-58/obj00000000000000000032 2023-07-29T00:25:10.898 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000032 2023-07-29T00:25:10.899 DEBUG:boto:path=/hiiqos-58/obj00000000000000000032?versionId=null 2023-07-29T00:25:10.901 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000032?versionId=null 2023-07-29T00:25:10.902 DEBUG:boto:Method: GET 2023-07-29T00:25:10.902 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000032?versionId=null 2023-07-29T00:25:10.903 DEBUG:boto:Data: 2023-07-29T00:25:10.903 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.903 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:10.904 DEBUG:boto:Port: 8001 2023-07-29T00:25:10.904 DEBUG:boto:Params: {} 2023-07-29T00:25:10.905 DEBUG:boto:Token: None 2023-07-29T00:25:10.905 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000032?versionId=null 2023-07-29T00:25:10.906 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EDPoS6eBNWpPsObAHw39BKwKOBQ= 2023-07-29T00:25:10.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EDPoS6eBNWpPsObAHw39BKwKOBQ=', 'Content-Length': '0'} 2023-07-29T00:25:10.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050045b4bdbfba0a4-0064c45c66-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.948 DEBUG:boto:path=/hiiqos-58/obj00000000000000000032 2023-07-29T00:25:10.949 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000032 2023-07-29T00:25:10.949 DEBUG:boto:path=/hiiqos-58/obj00000000000000000032?versionId=null 2023-07-29T00:25:10.950 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000032?versionId=null 2023-07-29T00:25:10.950 DEBUG:boto:Method: GET 2023-07-29T00:25:10.950 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000032?versionId=null 2023-07-29T00:25:10.951 DEBUG:boto:Data: 2023-07-29T00:25:10.951 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.952 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:10.952 DEBUG:boto:Port: 8000 2023-07-29T00:25:10.953 DEBUG:boto:Params: {} 2023-07-29T00:25:10.953 DEBUG:boto:Token: None 2023-07-29T00:25:10.954 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000032?versionId=null 2023-07-29T00:25:10.954 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EDPoS6eBNWpPsObAHw39BKwKOBQ= 2023-07-29T00:25:10.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EDPoS6eBNWpPsObAHw39BKwKOBQ=', 'Content-Length': '0'} 2023-07-29T00:25:10.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031bea2a26e54e0cb-0064c45c66-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.958 DEBUG:boto:path=/hiiqos-58/obj00000000000000000032 2023-07-29T00:25:10.958 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000032 2023-07-29T00:25:10.958 DEBUG:boto:path=/hiiqos-58/obj00000000000000000032?versionId=null 2023-07-29T00:25:10.959 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000032?versionId=null 2023-07-29T00:25:10.959 DEBUG:boto:Method: HEAD 2023-07-29T00:25:10.960 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000032?versionId=null 2023-07-29T00:25:10.960 DEBUG:boto:Data: 2023-07-29T00:25:10.960 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.961 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:10.961 DEBUG:boto:Port: 8001 2023-07-29T00:25:10.962 DEBUG:boto:Params: {} 2023-07-29T00:25:10.962 DEBUG:boto:Token: None 2023-07-29T00:25:10.962 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000032?versionId=null 2023-07-29T00:25:10.963 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QZM+zX2k7ummwPS5rRN6R7ZntRs= 2023-07-29T00:25:10.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QZM+zX2k7ummwPS5rRN6R7ZntRs=', 'Content-Length': '0'} 2023-07-29T00:25:10.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057fb4fdb08838725-0064c45c66-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.966 DEBUG:boto:path=/hiiqos-58/obj00000000000000000032 2023-07-29T00:25:10.966 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000032 2023-07-29T00:25:10.967 DEBUG:boto:path=/hiiqos-58/obj00000000000000000032?versionId=null 2023-07-29T00:25:10.967 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000032?versionId=null 2023-07-29T00:25:10.967 DEBUG:boto:Method: HEAD 2023-07-29T00:25:10.968 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000032?versionId=null 2023-07-29T00:25:10.968 DEBUG:boto:Data: 2023-07-29T00:25:10.969 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.969 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:10.969 DEBUG:boto:Port: 8000 2023-07-29T00:25:10.970 DEBUG:boto:Params: {} 2023-07-29T00:25:10.970 DEBUG:boto:Token: None 2023-07-29T00:25:10.971 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000032?versionId=null 2023-07-29T00:25:10.971 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QZM+zX2k7ummwPS5rRN6R7ZntRs= 2023-07-29T00:25:10.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QZM+zX2k7ummwPS5rRN6R7ZntRs=', 'Content-Length': '0'} 2023-07-29T00:25:10.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000045c35df7239373a-0064c45c66-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2023-07-29T00:25:10.974 DEBUG:boto:path=/hiiqos-58/obj00000000000000000032 2023-07-29T00:25:10.975 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000032 2023-07-29T00:25:10.975 DEBUG:boto:path=/hiiqos-58/obj00000000000000000032?versionId=null 2023-07-29T00:25:10.975 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000032?versionId=null 2023-07-29T00:25:10.976 DEBUG:boto:Method: GET 2023-07-29T00:25:10.976 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000032?versionId=null 2023-07-29T00:25:10.977 DEBUG:boto:Data: 2023-07-29T00:25:10.977 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.977 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:10.978 DEBUG:boto:Port: 8001 2023-07-29T00:25:10.978 DEBUG:boto:Params: {} 2023-07-29T00:25:10.979 DEBUG:boto:Token: None 2023-07-29T00:25:10.979 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000032?versionId=null 2023-07-29T00:25:10.980 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EDPoS6eBNWpPsObAHw39BKwKOBQ= 2023-07-29T00:25:10.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EDPoS6eBNWpPsObAHw39BKwKOBQ=', 'Content-Length': '0'} 2023-07-29T00:25:10.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000502a45934dd3eff6-0064c45c66-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.983 DEBUG:boto:path=/hiiqos-58/obj00000000000000000032 2023-07-29T00:25:10.983 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000032 2023-07-29T00:25:10.984 DEBUG:boto:path=/hiiqos-58/obj00000000000000000032?versionId=null 2023-07-29T00:25:10.984 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000032?versionId=null 2023-07-29T00:25:10.985 DEBUG:boto:Method: GET 2023-07-29T00:25:10.985 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000032?versionId=null 2023-07-29T00:25:10.986 DEBUG:boto:Data: 2023-07-29T00:25:10.986 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.986 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:10.987 DEBUG:boto:Port: 8000 2023-07-29T00:25:10.987 DEBUG:boto:Params: {} 2023-07-29T00:25:10.988 DEBUG:boto:Token: None 2023-07-29T00:25:10.989 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/obj00000000000000000032?versionId=null 2023-07-29T00:25:10.989 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EDPoS6eBNWpPsObAHw39BKwKOBQ= 2023-07-29T00:25:10.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EDPoS6eBNWpPsObAHw39BKwKOBQ=', 'Content-Length': '0'} 2023-07-29T00:25:10.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb6cbda6727e66c7-0064c45c66-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:10.993 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:10.993 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:10.994 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000032 2023-07-29T00:25:10.994 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000032 2023-07-29T00:25:10.995 DEBUG:boto:Method: GET 2023-07-29T00:25:10.995 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000032 2023-07-29T00:25:10.996 DEBUG:boto:Data: 2023-07-29T00:25:10.996 DEBUG:boto:Headers: {} 2023-07-29T00:25:10.997 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:10.997 DEBUG:boto:Port: 8001 2023-07-29T00:25:10.998 DEBUG:boto:Params: {} 2023-07-29T00:25:10.999 DEBUG:boto:Token: None 2023-07-29T00:25:10.999 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:10 GMT /hiiqos-58/ 2023-07-29T00:25:11.000 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VLNhdLobIM31WkKxiP5QjruUTBs= 2023-07-29T00:25:11.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VLNhdLobIM31WkKxiP5QjruUTBs=', 'Content-Length': '0'} 2023-07-29T00:25:11.003 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043f822c01f8397d0-0064c45c66-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.003 DEBUG:boto:b'hiiqos-58obj000000000000000000321000falseobj000000000000000000322023-07-29T00:24:53.877Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:11.004 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:11.005 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:11.005 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000032 2023-07-29T00:25:11.006 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000032 2023-07-29T00:25:11.006 DEBUG:boto:Method: GET 2023-07-29T00:25:11.007 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000032 2023-07-29T00:25:11.007 DEBUG:boto:Data: 2023-07-29T00:25:11.008 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.008 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:11.009 DEBUG:boto:Port: 8000 2023-07-29T00:25:11.009 DEBUG:boto:Params: {} 2023-07-29T00:25:11.010 DEBUG:boto:Token: None 2023-07-29T00:25:11.011 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/ 2023-07-29T00:25:11.011 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:roI/S2Qgkbc4BRAtVkkc6h5zINs= 2023-07-29T00:25:11.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:roI/S2Qgkbc4BRAtVkkc6h5zINs=', 'Content-Length': '0'} 2023-07-29T00:25:11.016 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d24b82596dac7a8-0064c45c66-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.017 DEBUG:boto:b'hiiqos-58obj000000000000000000321000falseobj000000000000000000322023-07-29T00:24:53.877Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:11.017 DEBUG:boto:path=/hiiqos-58/obj00000000000000000032 2023-07-29T00:25:11.018 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000032 2023-07-29T00:25:11.018 DEBUG:boto:Method: HEAD 2023-07-29T00:25:11.018 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000032 2023-07-29T00:25:11.019 DEBUG:boto:Data: 2023-07-29T00:25:11.019 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.019 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:11.020 DEBUG:boto:Port: 8001 2023-07-29T00:25:11.020 DEBUG:boto:Params: {} 2023-07-29T00:25:11.020 DEBUG:boto:Token: None 2023-07-29T00:25:11.021 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000032 2023-07-29T00:25:11.021 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TtlIq51CgXIH40vJF8qkW4ENRTs= 2023-07-29T00:25:11.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TtlIq51CgXIH40vJF8qkW4ENRTs=', 'Content-Length': '0'} 2023-07-29T00:25:11.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfbb201f6d8add95-0064c45c66-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.024 DEBUG:boto:path=/hiiqos-58/obj00000000000000000032 2023-07-29T00:25:11.025 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000032 2023-07-29T00:25:11.025 DEBUG:boto:Method: HEAD 2023-07-29T00:25:11.025 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000032 2023-07-29T00:25:11.026 DEBUG:boto:Data: 2023-07-29T00:25:11.026 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.026 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:11.027 DEBUG:boto:Port: 8000 2023-07-29T00:25:11.027 DEBUG:boto:Params: {} 2023-07-29T00:25:11.027 DEBUG:boto:Token: None 2023-07-29T00:25:11.028 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000032 2023-07-29T00:25:11.028 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TtlIq51CgXIH40vJF8qkW4ENRTs= 2023-07-29T00:25:11.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TtlIq51CgXIH40vJF8qkW4ENRTs=', 'Content-Length': '0'} 2023-07-29T00:25:11.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1816511e621de68-0064c45c66-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2023-07-29T00:25:11.031 DEBUG:boto:path=/hiiqos-58/obj00000000000000000032 2023-07-29T00:25:11.032 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000032 2023-07-29T00:25:11.032 DEBUG:boto:Method: GET 2023-07-29T00:25:11.033 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000032 2023-07-29T00:25:11.033 DEBUG:boto:Data: 2023-07-29T00:25:11.033 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.034 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:11.034 DEBUG:boto:Port: 8001 2023-07-29T00:25:11.034 DEBUG:boto:Params: {} 2023-07-29T00:25:11.035 DEBUG:boto:Token: None 2023-07-29T00:25:11.035 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000032 2023-07-29T00:25:11.036 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AvjDiDfx16uuKNXbrL6wLUtldMw= 2023-07-29T00:25:11.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AvjDiDfx16uuKNXbrL6wLUtldMw=', 'Content-Length': '0'} 2023-07-29T00:25:11.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e6669c471bae39a-0064c45c66-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.039 DEBUG:boto:path=/hiiqos-58/obj00000000000000000032 2023-07-29T00:25:11.039 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000032 2023-07-29T00:25:11.039 DEBUG:boto:Method: GET 2023-07-29T00:25:11.040 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000032 2023-07-29T00:25:11.040 DEBUG:boto:Data: 2023-07-29T00:25:11.041 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.041 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:11.041 DEBUG:boto:Port: 8000 2023-07-29T00:25:11.042 DEBUG:boto:Params: {} 2023-07-29T00:25:11.042 DEBUG:boto:Token: None 2023-07-29T00:25:11.042 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000032 2023-07-29T00:25:11.043 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AvjDiDfx16uuKNXbrL6wLUtldMw= 2023-07-29T00:25:11.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AvjDiDfx16uuKNXbrL6wLUtldMw=', 'Content-Length': '0'} 2023-07-29T00:25:11.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a5598d1c9f1ad4e-0064c45c66-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.092 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:11.093 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:11.093 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000032 2023-07-29T00:25:11.094 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000032 2023-07-29T00:25:11.094 DEBUG:boto:Method: GET 2023-07-29T00:25:11.094 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000032 2023-07-29T00:25:11.095 DEBUG:boto:Data: 2023-07-29T00:25:11.095 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.095 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:11.096 DEBUG:boto:Port: 8001 2023-07-29T00:25:11.096 DEBUG:boto:Params: {} 2023-07-29T00:25:11.097 DEBUG:boto:Token: None 2023-07-29T00:25:11.098 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/ 2023-07-29T00:25:11.099 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:roI/S2Qgkbc4BRAtVkkc6h5zINs= 2023-07-29T00:25:11.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:roI/S2Qgkbc4BRAtVkkc6h5zINs=', 'Content-Length': '0'} 2023-07-29T00:25:11.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd4c26a96fcf5cfa-0064c45c67-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.103 DEBUG:boto:b'hiiqos-58obj000000000000000000321000falseobj000000000000000000322023-07-29T00:24:53.877Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:11.104 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:11.105 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:11.106 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000032 2023-07-29T00:25:11.107 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000032 2023-07-29T00:25:11.108 DEBUG:boto:Method: GET 2023-07-29T00:25:11.109 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000032 2023-07-29T00:25:11.109 DEBUG:boto:Data: 2023-07-29T00:25:11.110 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.110 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:11.111 DEBUG:boto:Port: 8000 2023-07-29T00:25:11.111 DEBUG:boto:Params: {} 2023-07-29T00:25:11.112 DEBUG:boto:Token: None 2023-07-29T00:25:11.113 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/ 2023-07-29T00:25:11.114 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:roI/S2Qgkbc4BRAtVkkc6h5zINs= 2023-07-29T00:25:11.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:roI/S2Qgkbc4BRAtVkkc6h5zINs=', 'Content-Length': '0'} 2023-07-29T00:25:11.120 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062bf7f9509a5cd9a-0064c45c67-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.120 DEBUG:boto:b'hiiqos-58obj000000000000000000321000falseobj000000000000000000322023-07-29T00:24:53.877Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:11.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2023-07-29T00:25:11.122 DEBUG:boto:path=/hiiqos-58/obj00000000000000000033 2023-07-29T00:25:11.122 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000033 2023-07-29T00:25:11.122 DEBUG:boto:path=/hiiqos-58/obj00000000000000000033?versionId=null 2023-07-29T00:25:11.123 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000033?versionId=null 2023-07-29T00:25:11.123 DEBUG:boto:Method: GET 2023-07-29T00:25:11.124 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000033?versionId=null 2023-07-29T00:25:11.124 DEBUG:boto:Data: 2023-07-29T00:25:11.124 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.125 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:11.126 DEBUG:boto:Port: 8001 2023-07-29T00:25:11.126 DEBUG:boto:Params: {} 2023-07-29T00:25:11.128 DEBUG:boto:Token: None 2023-07-29T00:25:11.129 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000033?versionId=null 2023-07-29T00:25:11.129 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3Rk4pt+tqPGTisquebks+mcydMY= 2023-07-29T00:25:11.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3Rk4pt+tqPGTisquebks+mcydMY=', 'Content-Length': '0'} 2023-07-29T00:25:11.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023a598f88b260931-0064c45c67-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.133 DEBUG:boto:path=/hiiqos-58/obj00000000000000000033 2023-07-29T00:25:11.133 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000033 2023-07-29T00:25:11.133 DEBUG:boto:path=/hiiqos-58/obj00000000000000000033?versionId=null 2023-07-29T00:25:11.134 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000033?versionId=null 2023-07-29T00:25:11.134 DEBUG:boto:Method: GET 2023-07-29T00:25:11.134 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000033?versionId=null 2023-07-29T00:25:11.135 DEBUG:boto:Data: 2023-07-29T00:25:11.135 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.135 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:11.136 DEBUG:boto:Port: 8000 2023-07-29T00:25:11.136 DEBUG:boto:Params: {} 2023-07-29T00:25:11.137 DEBUG:boto:Token: None 2023-07-29T00:25:11.137 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000033?versionId=null 2023-07-29T00:25:11.137 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3Rk4pt+tqPGTisquebks+mcydMY= 2023-07-29T00:25:11.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3Rk4pt+tqPGTisquebks+mcydMY=', 'Content-Length': '0'} 2023-07-29T00:25:11.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000307ea8f555132c81-0064c45c67-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.141 DEBUG:boto:path=/hiiqos-58/obj00000000000000000033 2023-07-29T00:25:11.141 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000033 2023-07-29T00:25:11.142 DEBUG:boto:path=/hiiqos-58/obj00000000000000000033?versionId=null 2023-07-29T00:25:11.142 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000033?versionId=null 2023-07-29T00:25:11.142 DEBUG:boto:Method: HEAD 2023-07-29T00:25:11.143 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000033?versionId=null 2023-07-29T00:25:11.143 DEBUG:boto:Data: 2023-07-29T00:25:11.143 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.144 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:11.144 DEBUG:boto:Port: 8001 2023-07-29T00:25:11.144 DEBUG:boto:Params: {} 2023-07-29T00:25:11.145 DEBUG:boto:Token: None 2023-07-29T00:25:11.145 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000033?versionId=null 2023-07-29T00:25:11.146 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tYrixxV/32xFSuKTgDMvYTGR46A= 2023-07-29T00:25:11.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tYrixxV/32xFSuKTgDMvYTGR46A=', 'Content-Length': '0'} 2023-07-29T00:25:11.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf902867c45c4b75-0064c45c67-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.149 DEBUG:boto:path=/hiiqos-58/obj00000000000000000033 2023-07-29T00:25:11.149 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000033 2023-07-29T00:25:11.149 DEBUG:boto:path=/hiiqos-58/obj00000000000000000033?versionId=null 2023-07-29T00:25:11.150 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000033?versionId=null 2023-07-29T00:25:11.150 DEBUG:boto:Method: HEAD 2023-07-29T00:25:11.150 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000033?versionId=null 2023-07-29T00:25:11.151 DEBUG:boto:Data: 2023-07-29T00:25:11.151 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.151 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:11.152 DEBUG:boto:Port: 8000 2023-07-29T00:25:11.152 DEBUG:boto:Params: {} 2023-07-29T00:25:11.152 DEBUG:boto:Token: None 2023-07-29T00:25:11.153 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000033?versionId=null 2023-07-29T00:25:11.153 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tYrixxV/32xFSuKTgDMvYTGR46A= 2023-07-29T00:25:11.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tYrixxV/32xFSuKTgDMvYTGR46A=', 'Content-Length': '0'} 2023-07-29T00:25:11.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a596f5941d506fd2-0064c45c67-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2023-07-29T00:25:11.157 DEBUG:boto:path=/hiiqos-58/obj00000000000000000033 2023-07-29T00:25:11.157 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000033 2023-07-29T00:25:11.157 DEBUG:boto:path=/hiiqos-58/obj00000000000000000033?versionId=null 2023-07-29T00:25:11.158 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000033?versionId=null 2023-07-29T00:25:11.158 DEBUG:boto:Method: GET 2023-07-29T00:25:11.158 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000033?versionId=null 2023-07-29T00:25:11.159 DEBUG:boto:Data: 2023-07-29T00:25:11.159 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.160 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:11.160 DEBUG:boto:Port: 8001 2023-07-29T00:25:11.160 DEBUG:boto:Params: {} 2023-07-29T00:25:11.161 DEBUG:boto:Token: None 2023-07-29T00:25:11.161 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000033?versionId=null 2023-07-29T00:25:11.162 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3Rk4pt+tqPGTisquebks+mcydMY= 2023-07-29T00:25:11.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3Rk4pt+tqPGTisquebks+mcydMY=', 'Content-Length': '0'} 2023-07-29T00:25:11.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa028d6173187e42-0064c45c67-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.165 DEBUG:boto:path=/hiiqos-58/obj00000000000000000033 2023-07-29T00:25:11.165 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000033 2023-07-29T00:25:11.166 DEBUG:boto:path=/hiiqos-58/obj00000000000000000033?versionId=null 2023-07-29T00:25:11.166 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000033?versionId=null 2023-07-29T00:25:11.166 DEBUG:boto:Method: GET 2023-07-29T00:25:11.167 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000033?versionId=null 2023-07-29T00:25:11.167 DEBUG:boto:Data: 2023-07-29T00:25:11.168 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.168 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:11.168 DEBUG:boto:Port: 8000 2023-07-29T00:25:11.169 DEBUG:boto:Params: {} 2023-07-29T00:25:11.169 DEBUG:boto:Token: None 2023-07-29T00:25:11.169 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000033?versionId=null 2023-07-29T00:25:11.170 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3Rk4pt+tqPGTisquebks+mcydMY= 2023-07-29T00:25:11.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3Rk4pt+tqPGTisquebks+mcydMY=', 'Content-Length': '0'} 2023-07-29T00:25:11.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bba0c23f7feaef7-0064c45c67-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.173 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:11.173 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:11.174 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000033 2023-07-29T00:25:11.174 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000033 2023-07-29T00:25:11.175 DEBUG:boto:Method: GET 2023-07-29T00:25:11.175 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000033 2023-07-29T00:25:11.175 DEBUG:boto:Data: 2023-07-29T00:25:11.176 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.176 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:11.176 DEBUG:boto:Port: 8001 2023-07-29T00:25:11.177 DEBUG:boto:Params: {} 2023-07-29T00:25:11.177 DEBUG:boto:Token: None 2023-07-29T00:25:11.178 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/ 2023-07-29T00:25:11.178 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:roI/S2Qgkbc4BRAtVkkc6h5zINs= 2023-07-29T00:25:11.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:roI/S2Qgkbc4BRAtVkkc6h5zINs=', 'Content-Length': '0'} 2023-07-29T00:25:11.181 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062ac4c1930c077ae-0064c45c67-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.182 DEBUG:boto:b'hiiqos-58obj000000000000000000331000falseobj000000000000000000332023-07-29T00:25:00.051Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:11.183 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:11.184 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:11.185 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000033 2023-07-29T00:25:11.185 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000033 2023-07-29T00:25:11.186 DEBUG:boto:Method: GET 2023-07-29T00:25:11.186 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000033 2023-07-29T00:25:11.186 DEBUG:boto:Data: 2023-07-29T00:25:11.187 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.187 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:11.188 DEBUG:boto:Port: 8000 2023-07-29T00:25:11.188 DEBUG:boto:Params: {} 2023-07-29T00:25:11.189 DEBUG:boto:Token: None 2023-07-29T00:25:11.189 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/ 2023-07-29T00:25:11.190 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:roI/S2Qgkbc4BRAtVkkc6h5zINs= 2023-07-29T00:25:11.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:roI/S2Qgkbc4BRAtVkkc6h5zINs=', 'Content-Length': '0'} 2023-07-29T00:25:11.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b3fd209e8f02f44-0064c45c67-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.236 DEBUG:boto:b'hiiqos-58obj000000000000000000331000falseobj000000000000000000332023-07-29T00:25:00.051Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:11.237 DEBUG:boto:path=/hiiqos-58/obj00000000000000000033 2023-07-29T00:25:11.238 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000033 2023-07-29T00:25:11.238 DEBUG:boto:Method: HEAD 2023-07-29T00:25:11.238 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000033 2023-07-29T00:25:11.239 DEBUG:boto:Data: 2023-07-29T00:25:11.239 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.239 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:11.240 DEBUG:boto:Port: 8001 2023-07-29T00:25:11.240 DEBUG:boto:Params: {} 2023-07-29T00:25:11.240 DEBUG:boto:Token: None 2023-07-29T00:25:11.241 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000033 2023-07-29T00:25:11.241 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:y9UrPOcwsoOfDSTaFcDD3fUp7oc= 2023-07-29T00:25:11.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:y9UrPOcwsoOfDSTaFcDD3fUp7oc=', 'Content-Length': '0'} 2023-07-29T00:25:11.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000840b52b6f69be0e2-0064c45c67-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.244 DEBUG:boto:path=/hiiqos-58/obj00000000000000000033 2023-07-29T00:25:11.245 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000033 2023-07-29T00:25:11.245 DEBUG:boto:Method: HEAD 2023-07-29T00:25:11.246 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000033 2023-07-29T00:25:11.246 DEBUG:boto:Data: 2023-07-29T00:25:11.246 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.247 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:11.247 DEBUG:boto:Port: 8000 2023-07-29T00:25:11.247 DEBUG:boto:Params: {} 2023-07-29T00:25:11.248 DEBUG:boto:Token: None 2023-07-29T00:25:11.248 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000033 2023-07-29T00:25:11.249 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:y9UrPOcwsoOfDSTaFcDD3fUp7oc= 2023-07-29T00:25:11.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:y9UrPOcwsoOfDSTaFcDD3fUp7oc=', 'Content-Length': '0'} 2023-07-29T00:25:11.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1395c0374acd46d-0064c45c67-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2023-07-29T00:25:11.252 DEBUG:boto:path=/hiiqos-58/obj00000000000000000033 2023-07-29T00:25:11.252 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000033 2023-07-29T00:25:11.253 DEBUG:boto:Method: GET 2023-07-29T00:25:11.253 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000033 2023-07-29T00:25:11.254 DEBUG:boto:Data: 2023-07-29T00:25:11.254 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.254 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:11.255 DEBUG:boto:Port: 8001 2023-07-29T00:25:11.255 DEBUG:boto:Params: {} 2023-07-29T00:25:11.256 DEBUG:boto:Token: None 2023-07-29T00:25:11.256 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000033 2023-07-29T00:25:11.257 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P0yZxvrsOJWfFZfnP60p/DEROKw= 2023-07-29T00:25:11.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P0yZxvrsOJWfFZfnP60p/DEROKw=', 'Content-Length': '0'} 2023-07-29T00:25:11.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008937d328bb331b7f-0064c45c67-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.260 DEBUG:boto:path=/hiiqos-58/obj00000000000000000033 2023-07-29T00:25:11.261 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000033 2023-07-29T00:25:11.261 DEBUG:boto:Method: GET 2023-07-29T00:25:11.261 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000033 2023-07-29T00:25:11.262 DEBUG:boto:Data: 2023-07-29T00:25:11.262 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.263 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:11.263 DEBUG:boto:Port: 8000 2023-07-29T00:25:11.263 DEBUG:boto:Params: {} 2023-07-29T00:25:11.264 DEBUG:boto:Token: None 2023-07-29T00:25:11.264 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000033 2023-07-29T00:25:11.265 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P0yZxvrsOJWfFZfnP60p/DEROKw= 2023-07-29T00:25:11.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P0yZxvrsOJWfFZfnP60p/DEROKw=', 'Content-Length': '0'} 2023-07-29T00:25:11.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a87845383e3577f-0064c45c67-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.268 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:11.268 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:11.269 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000033 2023-07-29T00:25:11.269 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000033 2023-07-29T00:25:11.269 DEBUG:boto:Method: GET 2023-07-29T00:25:11.270 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000033 2023-07-29T00:25:11.270 DEBUG:boto:Data: 2023-07-29T00:25:11.271 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.271 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:11.271 DEBUG:boto:Port: 8001 2023-07-29T00:25:11.272 DEBUG:boto:Params: {} 2023-07-29T00:25:11.272 DEBUG:boto:Token: None 2023-07-29T00:25:11.272 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/ 2023-07-29T00:25:11.273 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:roI/S2Qgkbc4BRAtVkkc6h5zINs= 2023-07-29T00:25:11.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:roI/S2Qgkbc4BRAtVkkc6h5zINs=', 'Content-Length': '0'} 2023-07-29T00:25:11.276 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3adbc8237b1b8a8-0064c45c67-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.276 DEBUG:boto:b'hiiqos-58obj000000000000000000331000falseobj000000000000000000332023-07-29T00:25:00.051Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:11.277 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:11.277 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:11.278 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000033 2023-07-29T00:25:11.278 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000033 2023-07-29T00:25:11.279 DEBUG:boto:Method: GET 2023-07-29T00:25:11.279 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000033 2023-07-29T00:25:11.279 DEBUG:boto:Data: 2023-07-29T00:25:11.280 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.280 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:11.280 DEBUG:boto:Port: 8000 2023-07-29T00:25:11.281 DEBUG:boto:Params: {} 2023-07-29T00:25:11.281 DEBUG:boto:Token: None 2023-07-29T00:25:11.282 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/ 2023-07-29T00:25:11.282 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:roI/S2Qgkbc4BRAtVkkc6h5zINs= 2023-07-29T00:25:11.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:roI/S2Qgkbc4BRAtVkkc6h5zINs=', 'Content-Length': '0'} 2023-07-29T00:25:11.287 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ccc834ada4eaa0d-0064c45c67-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.287 DEBUG:boto:b'hiiqos-58obj000000000000000000331000falseobj000000000000000000332023-07-29T00:25:00.051Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:11.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2023-07-29T00:25:11.288 DEBUG:boto:path=/hiiqos-58/obj00000000000000000036 2023-07-29T00:25:11.289 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000036 2023-07-29T00:25:11.289 DEBUG:boto:path=/hiiqos-58/obj00000000000000000036?versionId=null 2023-07-29T00:25:11.289 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000036?versionId=null 2023-07-29T00:25:11.290 DEBUG:boto:Method: GET 2023-07-29T00:25:11.290 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000036?versionId=null 2023-07-29T00:25:11.290 DEBUG:boto:Data: 2023-07-29T00:25:11.291 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.291 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:11.291 DEBUG:boto:Port: 8001 2023-07-29T00:25:11.292 DEBUG:boto:Params: {} 2023-07-29T00:25:11.292 DEBUG:boto:Token: None 2023-07-29T00:25:11.293 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000036?versionId=null 2023-07-29T00:25:11.293 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1ma2tqY1aEMMwUt0d/vc16gUKfw= 2023-07-29T00:25:11.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1ma2tqY1aEMMwUt0d/vc16gUKfw=', 'Content-Length': '0'} 2023-07-29T00:25:11.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4b8234f574f5e93-0064c45c67-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.296 DEBUG:boto:path=/hiiqos-58/obj00000000000000000036 2023-07-29T00:25:11.297 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000036 2023-07-29T00:25:11.297 DEBUG:boto:path=/hiiqos-58/obj00000000000000000036?versionId=null 2023-07-29T00:25:11.297 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000036?versionId=null 2023-07-29T00:25:11.298 DEBUG:boto:Method: GET 2023-07-29T00:25:11.298 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000036?versionId=null 2023-07-29T00:25:11.298 DEBUG:boto:Data: 2023-07-29T00:25:11.299 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.299 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:11.299 DEBUG:boto:Port: 8000 2023-07-29T00:25:11.300 DEBUG:boto:Params: {} 2023-07-29T00:25:11.300 DEBUG:boto:Token: None 2023-07-29T00:25:11.301 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000036?versionId=null 2023-07-29T00:25:11.301 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1ma2tqY1aEMMwUt0d/vc16gUKfw= 2023-07-29T00:25:11.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1ma2tqY1aEMMwUt0d/vc16gUKfw=', 'Content-Length': '0'} 2023-07-29T00:25:11.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089ec4293c45f1226-0064c45c67-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.304 DEBUG:boto:path=/hiiqos-58/obj00000000000000000036 2023-07-29T00:25:11.304 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000036 2023-07-29T00:25:11.305 DEBUG:boto:path=/hiiqos-58/obj00000000000000000036?versionId=null 2023-07-29T00:25:11.305 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000036?versionId=null 2023-07-29T00:25:11.305 DEBUG:boto:Method: HEAD 2023-07-29T00:25:11.306 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000036?versionId=null 2023-07-29T00:25:11.306 DEBUG:boto:Data: 2023-07-29T00:25:11.306 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.307 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:11.307 DEBUG:boto:Port: 8001 2023-07-29T00:25:11.307 DEBUG:boto:Params: {} 2023-07-29T00:25:11.308 DEBUG:boto:Token: None 2023-07-29T00:25:11.308 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000036?versionId=null 2023-07-29T00:25:11.309 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fqHiJE32qjfKoxdWWCo1t5ISz6o= 2023-07-29T00:25:11.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fqHiJE32qjfKoxdWWCo1t5ISz6o=', 'Content-Length': '0'} 2023-07-29T00:25:11.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e856ee87a545fd8-0064c45c67-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.311 DEBUG:boto:path=/hiiqos-58/obj00000000000000000036 2023-07-29T00:25:11.312 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000036 2023-07-29T00:25:11.312 DEBUG:boto:path=/hiiqos-58/obj00000000000000000036?versionId=null 2023-07-29T00:25:11.312 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000036?versionId=null 2023-07-29T00:25:11.313 DEBUG:boto:Method: HEAD 2023-07-29T00:25:11.313 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000036?versionId=null 2023-07-29T00:25:11.313 DEBUG:boto:Data: 2023-07-29T00:25:11.314 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.314 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:11.314 DEBUG:boto:Port: 8000 2023-07-29T00:25:11.315 DEBUG:boto:Params: {} 2023-07-29T00:25:11.315 DEBUG:boto:Token: None 2023-07-29T00:25:11.316 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000036?versionId=null 2023-07-29T00:25:11.316 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fqHiJE32qjfKoxdWWCo1t5ISz6o= 2023-07-29T00:25:11.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fqHiJE32qjfKoxdWWCo1t5ISz6o=', 'Content-Length': '0'} 2023-07-29T00:25:11.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a33654987b01322f-0064c45c67-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2023-07-29T00:25:11.319 DEBUG:boto:path=/hiiqos-58/obj00000000000000000036 2023-07-29T00:25:11.320 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000036 2023-07-29T00:25:11.320 DEBUG:boto:path=/hiiqos-58/obj00000000000000000036?versionId=null 2023-07-29T00:25:11.320 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000036?versionId=null 2023-07-29T00:25:11.321 DEBUG:boto:Method: GET 2023-07-29T00:25:11.321 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000036?versionId=null 2023-07-29T00:25:11.321 DEBUG:boto:Data: 2023-07-29T00:25:11.322 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.322 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:11.323 DEBUG:boto:Port: 8001 2023-07-29T00:25:11.323 DEBUG:boto:Params: {} 2023-07-29T00:25:11.323 DEBUG:boto:Token: None 2023-07-29T00:25:11.324 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000036?versionId=null 2023-07-29T00:25:11.324 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1ma2tqY1aEMMwUt0d/vc16gUKfw= 2023-07-29T00:25:11.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1ma2tqY1aEMMwUt0d/vc16gUKfw=', 'Content-Length': '0'} 2023-07-29T00:25:11.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035a96639b247f51b-0064c45c67-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.327 DEBUG:boto:path=/hiiqos-58/obj00000000000000000036 2023-07-29T00:25:11.327 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000036 2023-07-29T00:25:11.328 DEBUG:boto:path=/hiiqos-58/obj00000000000000000036?versionId=null 2023-07-29T00:25:11.328 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000036?versionId=null 2023-07-29T00:25:11.328 DEBUG:boto:Method: GET 2023-07-29T00:25:11.329 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000036?versionId=null 2023-07-29T00:25:11.329 DEBUG:boto:Data: 2023-07-29T00:25:11.329 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.330 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:11.330 DEBUG:boto:Port: 8000 2023-07-29T00:25:11.330 DEBUG:boto:Params: {} 2023-07-29T00:25:11.331 DEBUG:boto:Token: None 2023-07-29T00:25:11.331 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000036?versionId=null 2023-07-29T00:25:11.332 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1ma2tqY1aEMMwUt0d/vc16gUKfw= 2023-07-29T00:25:11.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1ma2tqY1aEMMwUt0d/vc16gUKfw=', 'Content-Length': '0'} 2023-07-29T00:25:11.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdb662f82b619bb6-0064c45c67-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.335 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:11.335 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:11.336 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000036 2023-07-29T00:25:11.336 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000036 2023-07-29T00:25:11.336 DEBUG:boto:Method: GET 2023-07-29T00:25:11.337 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000036 2023-07-29T00:25:11.337 DEBUG:boto:Data: 2023-07-29T00:25:11.337 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.338 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:11.338 DEBUG:boto:Port: 8001 2023-07-29T00:25:11.338 DEBUG:boto:Params: {} 2023-07-29T00:25:11.339 DEBUG:boto:Token: None 2023-07-29T00:25:11.339 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/ 2023-07-29T00:25:11.340 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:roI/S2Qgkbc4BRAtVkkc6h5zINs= 2023-07-29T00:25:11.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:roI/S2Qgkbc4BRAtVkkc6h5zINs=', 'Content-Length': '0'} 2023-07-29T00:25:11.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4d4f61cdf83141f-0064c45c67-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.343 DEBUG:boto:b'hiiqos-58obj000000000000000000361000falseobj000000000000000000362023-07-29T00:24:59.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:11.344 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:11.344 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:11.344 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000036 2023-07-29T00:25:11.345 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000036 2023-07-29T00:25:11.345 DEBUG:boto:Method: GET 2023-07-29T00:25:11.345 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000036 2023-07-29T00:25:11.346 DEBUG:boto:Data: 2023-07-29T00:25:11.346 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.347 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:11.347 DEBUG:boto:Port: 8000 2023-07-29T00:25:11.347 DEBUG:boto:Params: {} 2023-07-29T00:25:11.348 DEBUG:boto:Token: None 2023-07-29T00:25:11.348 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/ 2023-07-29T00:25:11.348 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:roI/S2Qgkbc4BRAtVkkc6h5zINs= 2023-07-29T00:25:11.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:roI/S2Qgkbc4BRAtVkkc6h5zINs=', 'Content-Length': '0'} 2023-07-29T00:25:11.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a27071c2feba5f2d-0064c45c67-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.355 DEBUG:boto:b'hiiqos-58obj000000000000000000361000falseobj000000000000000000362023-07-29T00:24:59.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:11.356 DEBUG:boto:path=/hiiqos-58/obj00000000000000000036 2023-07-29T00:25:11.356 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000036 2023-07-29T00:25:11.356 DEBUG:boto:Method: HEAD 2023-07-29T00:25:11.357 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000036 2023-07-29T00:25:11.357 DEBUG:boto:Data: 2023-07-29T00:25:11.357 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.358 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:11.358 DEBUG:boto:Port: 8001 2023-07-29T00:25:11.358 DEBUG:boto:Params: {} 2023-07-29T00:25:11.359 DEBUG:boto:Token: None 2023-07-29T00:25:11.359 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000036 2023-07-29T00:25:11.360 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:E6sVKbjAVaLBMTzzJpSwypr5+O4= 2023-07-29T00:25:11.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:E6sVKbjAVaLBMTzzJpSwypr5+O4=', 'Content-Length': '0'} 2023-07-29T00:25:11.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddaf947a6dbf0eca-0064c45c67-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.362 DEBUG:boto:path=/hiiqos-58/obj00000000000000000036 2023-07-29T00:25:11.363 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000036 2023-07-29T00:25:11.363 DEBUG:boto:Method: HEAD 2023-07-29T00:25:11.363 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000036 2023-07-29T00:25:11.364 DEBUG:boto:Data: 2023-07-29T00:25:11.364 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.364 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:11.365 DEBUG:boto:Port: 8000 2023-07-29T00:25:11.365 DEBUG:boto:Params: {} 2023-07-29T00:25:11.365 DEBUG:boto:Token: None 2023-07-29T00:25:11.366 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000036 2023-07-29T00:25:11.366 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:E6sVKbjAVaLBMTzzJpSwypr5+O4= 2023-07-29T00:25:11.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:E6sVKbjAVaLBMTzzJpSwypr5+O4=', 'Content-Length': '0'} 2023-07-29T00:25:11.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d414c3b8ee2ba881-0064c45c67-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2023-07-29T00:25:11.369 DEBUG:boto:path=/hiiqos-58/obj00000000000000000036 2023-07-29T00:25:11.370 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000036 2023-07-29T00:25:11.370 DEBUG:boto:Method: GET 2023-07-29T00:25:11.370 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000036 2023-07-29T00:25:11.371 DEBUG:boto:Data: 2023-07-29T00:25:11.371 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.371 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:11.372 DEBUG:boto:Port: 8001 2023-07-29T00:25:11.372 DEBUG:boto:Params: {} 2023-07-29T00:25:11.373 DEBUG:boto:Token: None 2023-07-29T00:25:11.373 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000036 2023-07-29T00:25:11.373 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J2QxB3l5MZzmvFmmfjatjk4kMNI= 2023-07-29T00:25:11.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J2QxB3l5MZzmvFmmfjatjk4kMNI=', 'Content-Length': '0'} 2023-07-29T00:25:11.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a358dcc5b0bf554-0064c45c67-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.376 DEBUG:boto:path=/hiiqos-58/obj00000000000000000036 2023-07-29T00:25:11.376 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000036 2023-07-29T00:25:11.377 DEBUG:boto:Method: GET 2023-07-29T00:25:11.377 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000036 2023-07-29T00:25:11.378 DEBUG:boto:Data: 2023-07-29T00:25:11.378 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.378 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:11.379 DEBUG:boto:Port: 8000 2023-07-29T00:25:11.379 DEBUG:boto:Params: {} 2023-07-29T00:25:11.379 DEBUG:boto:Token: None 2023-07-29T00:25:11.380 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000036 2023-07-29T00:25:11.380 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J2QxB3l5MZzmvFmmfjatjk4kMNI= 2023-07-29T00:25:11.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J2QxB3l5MZzmvFmmfjatjk4kMNI=', 'Content-Length': '0'} 2023-07-29T00:25:11.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ce5a7b2bf331ff9-0064c45c67-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.424 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:11.425 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:11.425 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000036 2023-07-29T00:25:11.426 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000036 2023-07-29T00:25:11.426 DEBUG:boto:Method: GET 2023-07-29T00:25:11.426 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000036 2023-07-29T00:25:11.427 DEBUG:boto:Data: 2023-07-29T00:25:11.427 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.427 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:11.428 DEBUG:boto:Port: 8001 2023-07-29T00:25:11.428 DEBUG:boto:Params: {} 2023-07-29T00:25:11.429 DEBUG:boto:Token: None 2023-07-29T00:25:11.429 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/ 2023-07-29T00:25:11.430 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:roI/S2Qgkbc4BRAtVkkc6h5zINs= 2023-07-29T00:25:11.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:roI/S2Qgkbc4BRAtVkkc6h5zINs=', 'Content-Length': '0'} 2023-07-29T00:25:11.432 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071f2bd2ba6f9ae32-0064c45c67-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.433 DEBUG:boto:b'hiiqos-58obj000000000000000000361000falseobj000000000000000000362023-07-29T00:24:59.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:11.434 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:11.434 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:11.435 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000036 2023-07-29T00:25:11.435 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000036 2023-07-29T00:25:11.435 DEBUG:boto:Method: GET 2023-07-29T00:25:11.436 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000036 2023-07-29T00:25:11.436 DEBUG:boto:Data: 2023-07-29T00:25:11.437 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.437 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:11.437 DEBUG:boto:Port: 8000 2023-07-29T00:25:11.438 DEBUG:boto:Params: {} 2023-07-29T00:25:11.438 DEBUG:boto:Token: None 2023-07-29T00:25:11.438 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/ 2023-07-29T00:25:11.439 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:roI/S2Qgkbc4BRAtVkkc6h5zINs= 2023-07-29T00:25:11.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:roI/S2Qgkbc4BRAtVkkc6h5zINs=', 'Content-Length': '0'} 2023-07-29T00:25:11.443 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099dc2c5aa459ccab-0064c45c67-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.444 DEBUG:boto:b'hiiqos-58obj000000000000000000361000falseobj000000000000000000362023-07-29T00:24:59.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:11.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2023-07-29T00:25:11.445 DEBUG:boto:path=/hiiqos-58/obj00000000000000000039 2023-07-29T00:25:11.445 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000039 2023-07-29T00:25:11.446 DEBUG:boto:path=/hiiqos-58/obj00000000000000000039?versionId=null 2023-07-29T00:25:11.446 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000039?versionId=null 2023-07-29T00:25:11.446 DEBUG:boto:Method: GET 2023-07-29T00:25:11.447 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000039?versionId=null 2023-07-29T00:25:11.447 DEBUG:boto:Data: 2023-07-29T00:25:11.447 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.448 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:11.448 DEBUG:boto:Port: 8001 2023-07-29T00:25:11.448 DEBUG:boto:Params: {} 2023-07-29T00:25:11.449 DEBUG:boto:Token: None 2023-07-29T00:25:11.449 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000039?versionId=null 2023-07-29T00:25:11.450 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:l/JV2qQB1On68TYSzuX2v7lVBoA= 2023-07-29T00:25:11.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:l/JV2qQB1On68TYSzuX2v7lVBoA=', 'Content-Length': '0'} 2023-07-29T00:25:11.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f2aac08386030b4-0064c45c67-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.496 DEBUG:boto:path=/hiiqos-58/obj00000000000000000039 2023-07-29T00:25:11.497 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000039 2023-07-29T00:25:11.497 DEBUG:boto:path=/hiiqos-58/obj00000000000000000039?versionId=null 2023-07-29T00:25:11.497 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000039?versionId=null 2023-07-29T00:25:11.498 DEBUG:boto:Method: GET 2023-07-29T00:25:11.498 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000039?versionId=null 2023-07-29T00:25:11.498 DEBUG:boto:Data: 2023-07-29T00:25:11.499 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.499 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:11.500 DEBUG:boto:Port: 8000 2023-07-29T00:25:11.500 DEBUG:boto:Params: {} 2023-07-29T00:25:11.500 DEBUG:boto:Token: None 2023-07-29T00:25:11.501 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000039?versionId=null 2023-07-29T00:25:11.501 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:l/JV2qQB1On68TYSzuX2v7lVBoA= 2023-07-29T00:25:11.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:l/JV2qQB1On68TYSzuX2v7lVBoA=', 'Content-Length': '0'} 2023-07-29T00:25:11.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6514f9a0f403cd6-0064c45c67-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.505 DEBUG:boto:path=/hiiqos-58/obj00000000000000000039 2023-07-29T00:25:11.505 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000039 2023-07-29T00:25:11.505 DEBUG:boto:path=/hiiqos-58/obj00000000000000000039?versionId=null 2023-07-29T00:25:11.506 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000039?versionId=null 2023-07-29T00:25:11.506 DEBUG:boto:Method: HEAD 2023-07-29T00:25:11.506 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000039?versionId=null 2023-07-29T00:25:11.507 DEBUG:boto:Data: 2023-07-29T00:25:11.507 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.507 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:11.508 DEBUG:boto:Port: 8001 2023-07-29T00:25:11.508 DEBUG:boto:Params: {} 2023-07-29T00:25:11.508 DEBUG:boto:Token: None 2023-07-29T00:25:11.509 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000039?versionId=null 2023-07-29T00:25:11.509 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VQpPgYGkT/nRbmJrCB6DGqSWW0A= 2023-07-29T00:25:11.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VQpPgYGkT/nRbmJrCB6DGqSWW0A=', 'Content-Length': '0'} 2023-07-29T00:25:11.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022460d7cc12741f0-0064c45c67-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.512 DEBUG:boto:path=/hiiqos-58/obj00000000000000000039 2023-07-29T00:25:11.512 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000039 2023-07-29T00:25:11.513 DEBUG:boto:path=/hiiqos-58/obj00000000000000000039?versionId=null 2023-07-29T00:25:11.513 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000039?versionId=null 2023-07-29T00:25:11.513 DEBUG:boto:Method: HEAD 2023-07-29T00:25:11.514 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000039?versionId=null 2023-07-29T00:25:11.514 DEBUG:boto:Data: 2023-07-29T00:25:11.514 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.515 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:11.515 DEBUG:boto:Port: 8000 2023-07-29T00:25:11.515 DEBUG:boto:Params: {} 2023-07-29T00:25:11.516 DEBUG:boto:Token: None 2023-07-29T00:25:11.516 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000039?versionId=null 2023-07-29T00:25:11.517 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VQpPgYGkT/nRbmJrCB6DGqSWW0A= 2023-07-29T00:25:11.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VQpPgYGkT/nRbmJrCB6DGqSWW0A=', 'Content-Length': '0'} 2023-07-29T00:25:11.519 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074f3de1f1929d22a-0064c45c67-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2023-07-29T00:25:11.520 DEBUG:boto:path=/hiiqos-58/obj00000000000000000039 2023-07-29T00:25:11.520 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000039 2023-07-29T00:25:11.520 DEBUG:boto:path=/hiiqos-58/obj00000000000000000039?versionId=null 2023-07-29T00:25:11.521 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000039?versionId=null 2023-07-29T00:25:11.521 DEBUG:boto:Method: GET 2023-07-29T00:25:11.521 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000039?versionId=null 2023-07-29T00:25:11.522 DEBUG:boto:Data: 2023-07-29T00:25:11.522 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.523 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:11.523 DEBUG:boto:Port: 8001 2023-07-29T00:25:11.523 DEBUG:boto:Params: {} 2023-07-29T00:25:11.524 DEBUG:boto:Token: None 2023-07-29T00:25:11.524 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000039?versionId=null 2023-07-29T00:25:11.524 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:l/JV2qQB1On68TYSzuX2v7lVBoA= 2023-07-29T00:25:11.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:l/JV2qQB1On68TYSzuX2v7lVBoA=', 'Content-Length': '0'} 2023-07-29T00:25:11.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009efa9e3bb36cdff5-0064c45c67-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.527 DEBUG:boto:path=/hiiqos-58/obj00000000000000000039 2023-07-29T00:25:11.528 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000039 2023-07-29T00:25:11.528 DEBUG:boto:path=/hiiqos-58/obj00000000000000000039?versionId=null 2023-07-29T00:25:11.529 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000039?versionId=null 2023-07-29T00:25:11.529 DEBUG:boto:Method: GET 2023-07-29T00:25:11.529 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000039?versionId=null 2023-07-29T00:25:11.530 DEBUG:boto:Data: 2023-07-29T00:25:11.530 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.530 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:11.531 DEBUG:boto:Port: 8000 2023-07-29T00:25:11.531 DEBUG:boto:Params: {} 2023-07-29T00:25:11.531 DEBUG:boto:Token: None 2023-07-29T00:25:11.532 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000039?versionId=null 2023-07-29T00:25:11.532 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:l/JV2qQB1On68TYSzuX2v7lVBoA= 2023-07-29T00:25:11.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:l/JV2qQB1On68TYSzuX2v7lVBoA=', 'Content-Length': '0'} 2023-07-29T00:25:11.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b8f57fb31752d0f-0064c45c67-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.535 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:11.536 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:11.536 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000039 2023-07-29T00:25:11.537 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000039 2023-07-29T00:25:11.537 DEBUG:boto:Method: GET 2023-07-29T00:25:11.537 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000039 2023-07-29T00:25:11.538 DEBUG:boto:Data: 2023-07-29T00:25:11.538 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.538 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:11.539 DEBUG:boto:Port: 8001 2023-07-29T00:25:11.539 DEBUG:boto:Params: {} 2023-07-29T00:25:11.539 DEBUG:boto:Token: None 2023-07-29T00:25:11.540 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/ 2023-07-29T00:25:11.540 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:roI/S2Qgkbc4BRAtVkkc6h5zINs= 2023-07-29T00:25:11.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:roI/S2Qgkbc4BRAtVkkc6h5zINs=', 'Content-Length': '0'} 2023-07-29T00:25:11.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020fe542a2e967747-0064c45c67-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.544 DEBUG:boto:b'hiiqos-58obj000000000000000000391000falseobj000000000000000000392023-07-29T00:24:55.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:11.544 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:11.545 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:11.545 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000039 2023-07-29T00:25:11.545 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000039 2023-07-29T00:25:11.546 DEBUG:boto:Method: GET 2023-07-29T00:25:11.546 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000039 2023-07-29T00:25:11.547 DEBUG:boto:Data: 2023-07-29T00:25:11.547 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.547 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:11.548 DEBUG:boto:Port: 8000 2023-07-29T00:25:11.548 DEBUG:boto:Params: {} 2023-07-29T00:25:11.548 DEBUG:boto:Token: None 2023-07-29T00:25:11.549 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/ 2023-07-29T00:25:11.549 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:roI/S2Qgkbc4BRAtVkkc6h5zINs= 2023-07-29T00:25:11.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:roI/S2Qgkbc4BRAtVkkc6h5zINs=', 'Content-Length': '0'} 2023-07-29T00:25:11.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d132c687744820bd-0064c45c67-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.554 DEBUG:boto:b'hiiqos-58obj000000000000000000391000falseobj000000000000000000392023-07-29T00:24:55.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:11.555 DEBUG:boto:path=/hiiqos-58/obj00000000000000000039 2023-07-29T00:25:11.555 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000039 2023-07-29T00:25:11.556 DEBUG:boto:Method: HEAD 2023-07-29T00:25:11.556 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000039 2023-07-29T00:25:11.557 DEBUG:boto:Data: 2023-07-29T00:25:11.557 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.557 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:11.558 DEBUG:boto:Port: 8001 2023-07-29T00:25:11.558 DEBUG:boto:Params: {} 2023-07-29T00:25:11.558 DEBUG:boto:Token: None 2023-07-29T00:25:11.559 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000039 2023-07-29T00:25:11.559 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vFyhJd3faRaBVI26ECZ+8DaQulA= 2023-07-29T00:25:11.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vFyhJd3faRaBVI26ECZ+8DaQulA=', 'Content-Length': '0'} 2023-07-29T00:25:11.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003faa753ea145e5d0-0064c45c67-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.562 DEBUG:boto:path=/hiiqos-58/obj00000000000000000039 2023-07-29T00:25:11.562 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000039 2023-07-29T00:25:11.563 DEBUG:boto:Method: HEAD 2023-07-29T00:25:11.563 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000039 2023-07-29T00:25:11.563 DEBUG:boto:Data: 2023-07-29T00:25:11.564 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.564 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:11.564 DEBUG:boto:Port: 8000 2023-07-29T00:25:11.565 DEBUG:boto:Params: {} 2023-07-29T00:25:11.565 DEBUG:boto:Token: None 2023-07-29T00:25:11.566 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000039 2023-07-29T00:25:11.566 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vFyhJd3faRaBVI26ECZ+8DaQulA= 2023-07-29T00:25:11.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vFyhJd3faRaBVI26ECZ+8DaQulA=', 'Content-Length': '0'} 2023-07-29T00:25:11.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099dda1d063e0e123-0064c45c67-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2023-07-29T00:25:11.569 DEBUG:boto:path=/hiiqos-58/obj00000000000000000039 2023-07-29T00:25:11.570 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000039 2023-07-29T00:25:11.570 DEBUG:boto:Method: GET 2023-07-29T00:25:11.571 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000039 2023-07-29T00:25:11.571 DEBUG:boto:Data: 2023-07-29T00:25:11.571 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.572 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:11.572 DEBUG:boto:Port: 8001 2023-07-29T00:25:11.572 DEBUG:boto:Params: {} 2023-07-29T00:25:11.573 DEBUG:boto:Token: None 2023-07-29T00:25:11.573 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000039 2023-07-29T00:25:11.574 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MaEoVvA+8yN/l9w+3waJJyaiJR0= 2023-07-29T00:25:11.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MaEoVvA+8yN/l9w+3waJJyaiJR0=', 'Content-Length': '0'} 2023-07-29T00:25:11.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8f858114cddd89a-0064c45c67-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.577 DEBUG:boto:path=/hiiqos-58/obj00000000000000000039 2023-07-29T00:25:11.577 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000039 2023-07-29T00:25:11.577 DEBUG:boto:Method: GET 2023-07-29T00:25:11.578 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000039 2023-07-29T00:25:11.578 DEBUG:boto:Data: 2023-07-29T00:25:11.578 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.579 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:11.579 DEBUG:boto:Port: 8000 2023-07-29T00:25:11.579 DEBUG:boto:Params: {} 2023-07-29T00:25:11.580 DEBUG:boto:Token: None 2023-07-29T00:25:11.580 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000039 2023-07-29T00:25:11.581 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MaEoVvA+8yN/l9w+3waJJyaiJR0= 2023-07-29T00:25:11.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MaEoVvA+8yN/l9w+3waJJyaiJR0=', 'Content-Length': '0'} 2023-07-29T00:25:11.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4266d3e3d05558b-0064c45c67-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.624 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:11.625 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:11.625 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000039 2023-07-29T00:25:11.626 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000039 2023-07-29T00:25:11.626 DEBUG:boto:Method: GET 2023-07-29T00:25:11.627 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000039 2023-07-29T00:25:11.627 DEBUG:boto:Data: 2023-07-29T00:25:11.627 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.628 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:11.628 DEBUG:boto:Port: 8001 2023-07-29T00:25:11.628 DEBUG:boto:Params: {} 2023-07-29T00:25:11.629 DEBUG:boto:Token: None 2023-07-29T00:25:11.629 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/ 2023-07-29T00:25:11.630 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:roI/S2Qgkbc4BRAtVkkc6h5zINs= 2023-07-29T00:25:11.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:roI/S2Qgkbc4BRAtVkkc6h5zINs=', 'Content-Length': '0'} 2023-07-29T00:25:11.632 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000998e2a6122ca8dbc-0064c45c67-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.633 DEBUG:boto:b'hiiqos-58obj000000000000000000391000falseobj000000000000000000392023-07-29T00:24:55.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:11.634 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:11.634 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:11.635 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000039 2023-07-29T00:25:11.635 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000039 2023-07-29T00:25:11.636 DEBUG:boto:Method: GET 2023-07-29T00:25:11.636 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000039 2023-07-29T00:25:11.636 DEBUG:boto:Data: 2023-07-29T00:25:11.637 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.637 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:11.637 DEBUG:boto:Port: 8000 2023-07-29T00:25:11.638 DEBUG:boto:Params: {} 2023-07-29T00:25:11.638 DEBUG:boto:Token: None 2023-07-29T00:25:11.639 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/ 2023-07-29T00:25:11.639 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:roI/S2Qgkbc4BRAtVkkc6h5zINs= 2023-07-29T00:25:11.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:roI/S2Qgkbc4BRAtVkkc6h5zINs=', 'Content-Length': '0'} 2023-07-29T00:25:11.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a1f1061725f0a50-0064c45c67-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.644 DEBUG:boto:b'hiiqos-58obj000000000000000000391000falseobj000000000000000000392023-07-29T00:24:55.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:11.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2023-07-29T00:25:11.645 DEBUG:boto:path=/hiiqos-58/obj00000000000000000040 2023-07-29T00:25:11.646 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000040 2023-07-29T00:25:11.646 DEBUG:boto:path=/hiiqos-58/obj00000000000000000040?versionId=null 2023-07-29T00:25:11.646 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000040?versionId=null 2023-07-29T00:25:11.647 DEBUG:boto:Method: GET 2023-07-29T00:25:11.647 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000040?versionId=null 2023-07-29T00:25:11.647 DEBUG:boto:Data: 2023-07-29T00:25:11.648 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.648 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:11.648 DEBUG:boto:Port: 8001 2023-07-29T00:25:11.649 DEBUG:boto:Params: {} 2023-07-29T00:25:11.649 DEBUG:boto:Token: None 2023-07-29T00:25:11.650 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000040?versionId=null 2023-07-29T00:25:11.650 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Jqoico3jjv2z6EeuKDNtFye6aBc= 2023-07-29T00:25:11.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Jqoico3jjv2z6EeuKDNtFye6aBc=', 'Content-Length': '0'} 2023-07-29T00:25:11.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a96de07633b682f-0064c45c67-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.653 DEBUG:boto:path=/hiiqos-58/obj00000000000000000040 2023-07-29T00:25:11.654 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000040 2023-07-29T00:25:11.654 DEBUG:boto:path=/hiiqos-58/obj00000000000000000040?versionId=null 2023-07-29T00:25:11.654 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000040?versionId=null 2023-07-29T00:25:11.655 DEBUG:boto:Method: GET 2023-07-29T00:25:11.655 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000040?versionId=null 2023-07-29T00:25:11.655 DEBUG:boto:Data: 2023-07-29T00:25:11.656 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.656 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:11.657 DEBUG:boto:Port: 8000 2023-07-29T00:25:11.657 DEBUG:boto:Params: {} 2023-07-29T00:25:11.659 DEBUG:boto:Token: None 2023-07-29T00:25:11.659 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000040?versionId=null 2023-07-29T00:25:11.659 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Jqoico3jjv2z6EeuKDNtFye6aBc= 2023-07-29T00:25:11.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Jqoico3jjv2z6EeuKDNtFye6aBc=', 'Content-Length': '0'} 2023-07-29T00:25:11.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099842c0982c00041-0064c45c67-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.662 DEBUG:boto:path=/hiiqos-58/obj00000000000000000040 2023-07-29T00:25:11.663 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000040 2023-07-29T00:25:11.663 DEBUG:boto:path=/hiiqos-58/obj00000000000000000040?versionId=null 2023-07-29T00:25:11.663 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000040?versionId=null 2023-07-29T00:25:11.664 DEBUG:boto:Method: HEAD 2023-07-29T00:25:11.664 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000040?versionId=null 2023-07-29T00:25:11.664 DEBUG:boto:Data: 2023-07-29T00:25:11.665 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.665 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:11.665 DEBUG:boto:Port: 8001 2023-07-29T00:25:11.666 DEBUG:boto:Params: {} 2023-07-29T00:25:11.666 DEBUG:boto:Token: None 2023-07-29T00:25:11.666 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000040?versionId=null 2023-07-29T00:25:11.667 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3HqpdCIVfo5F7l94rKKuY15e5Yk= 2023-07-29T00:25:11.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3HqpdCIVfo5F7l94rKKuY15e5Yk=', 'Content-Length': '0'} 2023-07-29T00:25:11.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e1e030410388d58-0064c45c67-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.669 DEBUG:boto:path=/hiiqos-58/obj00000000000000000040 2023-07-29T00:25:11.670 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000040 2023-07-29T00:25:11.670 DEBUG:boto:path=/hiiqos-58/obj00000000000000000040?versionId=null 2023-07-29T00:25:11.670 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000040?versionId=null 2023-07-29T00:25:11.671 DEBUG:boto:Method: HEAD 2023-07-29T00:25:11.671 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000040?versionId=null 2023-07-29T00:25:11.671 DEBUG:boto:Data: 2023-07-29T00:25:11.672 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.672 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:11.673 DEBUG:boto:Port: 8000 2023-07-29T00:25:11.673 DEBUG:boto:Params: {} 2023-07-29T00:25:11.673 DEBUG:boto:Token: None 2023-07-29T00:25:11.674 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000040?versionId=null 2023-07-29T00:25:11.674 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3HqpdCIVfo5F7l94rKKuY15e5Yk= 2023-07-29T00:25:11.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3HqpdCIVfo5F7l94rKKuY15e5Yk=', 'Content-Length': '0'} 2023-07-29T00:25:11.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a988aaacf0da9818-0064c45c67-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2023-07-29T00:25:11.677 DEBUG:boto:path=/hiiqos-58/obj00000000000000000040 2023-07-29T00:25:11.678 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000040 2023-07-29T00:25:11.678 DEBUG:boto:path=/hiiqos-58/obj00000000000000000040?versionId=null 2023-07-29T00:25:11.679 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000040?versionId=null 2023-07-29T00:25:11.679 DEBUG:boto:Method: GET 2023-07-29T00:25:11.679 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000040?versionId=null 2023-07-29T00:25:11.680 DEBUG:boto:Data: 2023-07-29T00:25:11.680 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.680 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:11.681 DEBUG:boto:Port: 8001 2023-07-29T00:25:11.681 DEBUG:boto:Params: {} 2023-07-29T00:25:11.681 DEBUG:boto:Token: None 2023-07-29T00:25:11.682 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000040?versionId=null 2023-07-29T00:25:11.682 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Jqoico3jjv2z6EeuKDNtFye6aBc= 2023-07-29T00:25:11.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Jqoico3jjv2z6EeuKDNtFye6aBc=', 'Content-Length': '0'} 2023-07-29T00:25:11.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006193cab5d575254b-0064c45c67-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.728 DEBUG:boto:path=/hiiqos-58/obj00000000000000000040 2023-07-29T00:25:11.729 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000040 2023-07-29T00:25:11.729 DEBUG:boto:path=/hiiqos-58/obj00000000000000000040?versionId=null 2023-07-29T00:25:11.730 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000040?versionId=null 2023-07-29T00:25:11.730 DEBUG:boto:Method: GET 2023-07-29T00:25:11.731 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000040?versionId=null 2023-07-29T00:25:11.731 DEBUG:boto:Data: 2023-07-29T00:25:11.731 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.732 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:11.732 DEBUG:boto:Port: 8000 2023-07-29T00:25:11.732 DEBUG:boto:Params: {} 2023-07-29T00:25:11.736 DEBUG:boto:Token: None 2023-07-29T00:25:11.737 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000040?versionId=null 2023-07-29T00:25:11.738 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Jqoico3jjv2z6EeuKDNtFye6aBc= 2023-07-29T00:25:11.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Jqoico3jjv2z6EeuKDNtFye6aBc=', 'Content-Length': '0'} 2023-07-29T00:25:11.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d73a2072f58131f-0064c45c67-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.744 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:11.744 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:11.745 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000040 2023-07-29T00:25:11.746 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000040 2023-07-29T00:25:11.747 DEBUG:boto:Method: GET 2023-07-29T00:25:11.747 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000040 2023-07-29T00:25:11.748 DEBUG:boto:Data: 2023-07-29T00:25:11.749 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.749 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:11.750 DEBUG:boto:Port: 8001 2023-07-29T00:25:11.751 DEBUG:boto:Params: {} 2023-07-29T00:25:11.752 DEBUG:boto:Token: None 2023-07-29T00:25:11.752 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/ 2023-07-29T00:25:11.753 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:roI/S2Qgkbc4BRAtVkkc6h5zINs= 2023-07-29T00:25:11.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:roI/S2Qgkbc4BRAtVkkc6h5zINs=', 'Content-Length': '0'} 2023-07-29T00:25:11.757 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d50e2d4bd74a66ff-0064c45c67-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.757 DEBUG:boto:b'hiiqos-58obj000000000000000000401000falseobj000000000000000000402023-07-29T00:24:59.326Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:11.758 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:11.759 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:11.759 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000040 2023-07-29T00:25:11.759 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000040 2023-07-29T00:25:11.760 DEBUG:boto:Method: GET 2023-07-29T00:25:11.760 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000040 2023-07-29T00:25:11.760 DEBUG:boto:Data: 2023-07-29T00:25:11.761 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.761 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:11.762 DEBUG:boto:Port: 8000 2023-07-29T00:25:11.763 DEBUG:boto:Params: {} 2023-07-29T00:25:11.764 DEBUG:boto:Token: None 2023-07-29T00:25:11.765 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/ 2023-07-29T00:25:11.766 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:roI/S2Qgkbc4BRAtVkkc6h5zINs= 2023-07-29T00:25:11.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:roI/S2Qgkbc4BRAtVkkc6h5zINs=', 'Content-Length': '0'} 2023-07-29T00:25:11.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8064b926a81a064-0064c45c67-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.773 DEBUG:boto:b'hiiqos-58obj000000000000000000401000falseobj000000000000000000402023-07-29T00:24:59.326Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:11.774 DEBUG:boto:path=/hiiqos-58/obj00000000000000000040 2023-07-29T00:25:11.774 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000040 2023-07-29T00:25:11.775 DEBUG:boto:Method: HEAD 2023-07-29T00:25:11.775 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000040 2023-07-29T00:25:11.776 DEBUG:boto:Data: 2023-07-29T00:25:11.776 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.777 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:11.777 DEBUG:boto:Port: 8001 2023-07-29T00:25:11.777 DEBUG:boto:Params: {} 2023-07-29T00:25:11.778 DEBUG:boto:Token: None 2023-07-29T00:25:11.778 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000040 2023-07-29T00:25:11.779 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PKYcldAjXyxDY2auBfY8jNunDqU= 2023-07-29T00:25:11.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PKYcldAjXyxDY2auBfY8jNunDqU=', 'Content-Length': '0'} 2023-07-29T00:25:11.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008988541d8475c1a4-0064c45c67-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.782 DEBUG:boto:path=/hiiqos-58/obj00000000000000000040 2023-07-29T00:25:11.782 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000040 2023-07-29T00:25:11.782 DEBUG:boto:Method: HEAD 2023-07-29T00:25:11.783 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000040 2023-07-29T00:25:11.784 DEBUG:boto:Data: 2023-07-29T00:25:11.784 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.784 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:11.785 DEBUG:boto:Port: 8000 2023-07-29T00:25:11.785 DEBUG:boto:Params: {} 2023-07-29T00:25:11.786 DEBUG:boto:Token: None 2023-07-29T00:25:11.787 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000040 2023-07-29T00:25:11.787 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PKYcldAjXyxDY2auBfY8jNunDqU= 2023-07-29T00:25:11.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PKYcldAjXyxDY2auBfY8jNunDqU=', 'Content-Length': '0'} 2023-07-29T00:25:11.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0598bd32c731c40-0064c45c67-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2023-07-29T00:25:11.791 DEBUG:boto:path=/hiiqos-58/obj00000000000000000040 2023-07-29T00:25:11.791 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000040 2023-07-29T00:25:11.791 DEBUG:boto:Method: GET 2023-07-29T00:25:11.792 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000040 2023-07-29T00:25:11.792 DEBUG:boto:Data: 2023-07-29T00:25:11.793 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.794 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:11.795 DEBUG:boto:Port: 8001 2023-07-29T00:25:11.795 DEBUG:boto:Params: {} 2023-07-29T00:25:11.797 DEBUG:boto:Token: None 2023-07-29T00:25:11.798 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000040 2023-07-29T00:25:11.798 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SaD0m2zaXEySbe/Fi+xjOJgEi5Q= 2023-07-29T00:25:11.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SaD0m2zaXEySbe/Fi+xjOJgEi5Q=', 'Content-Length': '0'} 2023-07-29T00:25:11.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b827eb2fa8fffac-0064c45c67-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.844 DEBUG:boto:path=/hiiqos-58/obj00000000000000000040 2023-07-29T00:25:11.845 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000040 2023-07-29T00:25:11.845 DEBUG:boto:Method: GET 2023-07-29T00:25:11.846 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000040 2023-07-29T00:25:11.846 DEBUG:boto:Data: 2023-07-29T00:25:11.847 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.847 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:11.847 DEBUG:boto:Port: 8000 2023-07-29T00:25:11.848 DEBUG:boto:Params: {} 2023-07-29T00:25:11.848 DEBUG:boto:Token: None 2023-07-29T00:25:11.849 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000040 2023-07-29T00:25:11.849 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SaD0m2zaXEySbe/Fi+xjOJgEi5Q= 2023-07-29T00:25:11.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SaD0m2zaXEySbe/Fi+xjOJgEi5Q=', 'Content-Length': '0'} 2023-07-29T00:25:11.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b840a6acd7fd7f31-0064c45c67-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.852 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:11.853 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:11.853 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000040 2023-07-29T00:25:11.853 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000040 2023-07-29T00:25:11.854 DEBUG:boto:Method: GET 2023-07-29T00:25:11.854 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000040 2023-07-29T00:25:11.854 DEBUG:boto:Data: 2023-07-29T00:25:11.855 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.855 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:11.855 DEBUG:boto:Port: 8001 2023-07-29T00:25:11.856 DEBUG:boto:Params: {} 2023-07-29T00:25:11.856 DEBUG:boto:Token: None 2023-07-29T00:25:11.857 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/ 2023-07-29T00:25:11.857 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:roI/S2Qgkbc4BRAtVkkc6h5zINs= 2023-07-29T00:25:11.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:roI/S2Qgkbc4BRAtVkkc6h5zINs=', 'Content-Length': '0'} 2023-07-29T00:25:11.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec67b702e8a5347e-0064c45c67-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.861 DEBUG:boto:b'hiiqos-58obj000000000000000000401000falseobj000000000000000000402023-07-29T00:24:59.326Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:11.861 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:11.862 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:11.862 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000040 2023-07-29T00:25:11.863 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000040 2023-07-29T00:25:11.863 DEBUG:boto:Method: GET 2023-07-29T00:25:11.864 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000040 2023-07-29T00:25:11.864 DEBUG:boto:Data: 2023-07-29T00:25:11.865 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.865 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:11.865 DEBUG:boto:Port: 8000 2023-07-29T00:25:11.866 DEBUG:boto:Params: {} 2023-07-29T00:25:11.866 DEBUG:boto:Token: None 2023-07-29T00:25:11.867 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/ 2023-07-29T00:25:11.867 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:roI/S2Qgkbc4BRAtVkkc6h5zINs= 2023-07-29T00:25:11.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:roI/S2Qgkbc4BRAtVkkc6h5zINs=', 'Content-Length': '0'} 2023-07-29T00:25:11.872 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031f889e5a0690364-0064c45c67-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.873 DEBUG:boto:b'hiiqos-58obj000000000000000000401000falseobj000000000000000000402023-07-29T00:24:59.326Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:11.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2023-07-29T00:25:11.874 DEBUG:boto:path=/hiiqos-58/obj00000000000000000042 2023-07-29T00:25:11.874 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000042 2023-07-29T00:25:11.875 DEBUG:boto:path=/hiiqos-58/obj00000000000000000042?versionId=null 2023-07-29T00:25:11.875 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000042?versionId=null 2023-07-29T00:25:11.876 DEBUG:boto:Method: GET 2023-07-29T00:25:11.876 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000042?versionId=null 2023-07-29T00:25:11.876 DEBUG:boto:Data: 2023-07-29T00:25:11.877 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.877 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:11.877 DEBUG:boto:Port: 8001 2023-07-29T00:25:11.878 DEBUG:boto:Params: {} 2023-07-29T00:25:11.878 DEBUG:boto:Token: None 2023-07-29T00:25:11.879 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000042?versionId=null 2023-07-29T00:25:11.879 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jQG6jGBAA6bAQqw2utpJPnpLVdo= 2023-07-29T00:25:11.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jQG6jGBAA6bAQqw2utpJPnpLVdo=', 'Content-Length': '0'} 2023-07-29T00:25:11.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3726ece6dca7872-0064c45c67-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.882 DEBUG:boto:path=/hiiqos-58/obj00000000000000000042 2023-07-29T00:25:11.883 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000042 2023-07-29T00:25:11.883 DEBUG:boto:path=/hiiqos-58/obj00000000000000000042?versionId=null 2023-07-29T00:25:11.884 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000042?versionId=null 2023-07-29T00:25:11.884 DEBUG:boto:Method: GET 2023-07-29T00:25:11.884 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000042?versionId=null 2023-07-29T00:25:11.885 DEBUG:boto:Data: 2023-07-29T00:25:11.885 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.886 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:11.886 DEBUG:boto:Port: 8000 2023-07-29T00:25:11.886 DEBUG:boto:Params: {} 2023-07-29T00:25:11.887 DEBUG:boto:Token: None 2023-07-29T00:25:11.887 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000042?versionId=null 2023-07-29T00:25:11.888 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jQG6jGBAA6bAQqw2utpJPnpLVdo= 2023-07-29T00:25:11.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jQG6jGBAA6bAQqw2utpJPnpLVdo=', 'Content-Length': '0'} 2023-07-29T00:25:11.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019c773be83e24ca8-0064c45c67-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.932 DEBUG:boto:path=/hiiqos-58/obj00000000000000000042 2023-07-29T00:25:11.933 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000042 2023-07-29T00:25:11.933 DEBUG:boto:path=/hiiqos-58/obj00000000000000000042?versionId=null 2023-07-29T00:25:11.934 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000042?versionId=null 2023-07-29T00:25:11.934 DEBUG:boto:Method: HEAD 2023-07-29T00:25:11.934 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000042?versionId=null 2023-07-29T00:25:11.935 DEBUG:boto:Data: 2023-07-29T00:25:11.935 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.935 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:11.936 DEBUG:boto:Port: 8001 2023-07-29T00:25:11.936 DEBUG:boto:Params: {} 2023-07-29T00:25:11.937 DEBUG:boto:Token: None 2023-07-29T00:25:11.937 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000042?versionId=null 2023-07-29T00:25:11.937 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gqYUOD8p/v3B7stAfnXkkAgPn3A= 2023-07-29T00:25:11.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gqYUOD8p/v3B7stAfnXkkAgPn3A=', 'Content-Length': '0'} 2023-07-29T00:25:11.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f3281cd3d07caae-0064c45c67-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.940 DEBUG:boto:path=/hiiqos-58/obj00000000000000000042 2023-07-29T00:25:11.941 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000042 2023-07-29T00:25:11.941 DEBUG:boto:path=/hiiqos-58/obj00000000000000000042?versionId=null 2023-07-29T00:25:11.941 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000042?versionId=null 2023-07-29T00:25:11.942 DEBUG:boto:Method: HEAD 2023-07-29T00:25:11.942 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000042?versionId=null 2023-07-29T00:25:11.942 DEBUG:boto:Data: 2023-07-29T00:25:11.943 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.943 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:11.943 DEBUG:boto:Port: 8000 2023-07-29T00:25:11.944 DEBUG:boto:Params: {} 2023-07-29T00:25:11.944 DEBUG:boto:Token: None 2023-07-29T00:25:11.945 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000042?versionId=null 2023-07-29T00:25:11.945 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gqYUOD8p/v3B7stAfnXkkAgPn3A= 2023-07-29T00:25:11.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gqYUOD8p/v3B7stAfnXkkAgPn3A=', 'Content-Length': '0'} 2023-07-29T00:25:11.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d958e91e53016dbf-0064c45c67-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2023-07-29T00:25:11.948 DEBUG:boto:path=/hiiqos-58/obj00000000000000000042 2023-07-29T00:25:11.949 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000042 2023-07-29T00:25:11.949 DEBUG:boto:path=/hiiqos-58/obj00000000000000000042?versionId=null 2023-07-29T00:25:11.950 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000042?versionId=null 2023-07-29T00:25:11.950 DEBUG:boto:Method: GET 2023-07-29T00:25:11.950 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000042?versionId=null 2023-07-29T00:25:11.951 DEBUG:boto:Data: 2023-07-29T00:25:11.951 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.951 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:11.952 DEBUG:boto:Port: 8001 2023-07-29T00:25:11.952 DEBUG:boto:Params: {} 2023-07-29T00:25:11.952 DEBUG:boto:Token: None 2023-07-29T00:25:11.953 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000042?versionId=null 2023-07-29T00:25:11.953 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jQG6jGBAA6bAQqw2utpJPnpLVdo= 2023-07-29T00:25:11.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jQG6jGBAA6bAQqw2utpJPnpLVdo=', 'Content-Length': '0'} 2023-07-29T00:25:11.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cfeceb0056d9754-0064c45c67-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.956 DEBUG:boto:path=/hiiqos-58/obj00000000000000000042 2023-07-29T00:25:11.957 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000042 2023-07-29T00:25:11.957 DEBUG:boto:path=/hiiqos-58/obj00000000000000000042?versionId=null 2023-07-29T00:25:11.957 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000042?versionId=null 2023-07-29T00:25:11.958 DEBUG:boto:Method: GET 2023-07-29T00:25:11.958 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000042?versionId=null 2023-07-29T00:25:11.958 DEBUG:boto:Data: 2023-07-29T00:25:11.959 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.959 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:11.960 DEBUG:boto:Port: 8000 2023-07-29T00:25:11.960 DEBUG:boto:Params: {} 2023-07-29T00:25:11.960 DEBUG:boto:Token: None 2023-07-29T00:25:11.961 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000042?versionId=null 2023-07-29T00:25:11.961 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jQG6jGBAA6bAQqw2utpJPnpLVdo= 2023-07-29T00:25:11.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jQG6jGBAA6bAQqw2utpJPnpLVdo=', 'Content-Length': '0'} 2023-07-29T00:25:11.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064c1dab7d2f6cf22-0064c45c67-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.964 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:11.965 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:11.965 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000042 2023-07-29T00:25:11.965 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000042 2023-07-29T00:25:11.966 DEBUG:boto:Method: GET 2023-07-29T00:25:11.966 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000042 2023-07-29T00:25:11.967 DEBUG:boto:Data: 2023-07-29T00:25:11.967 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.967 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:11.968 DEBUG:boto:Port: 8001 2023-07-29T00:25:11.968 DEBUG:boto:Params: {} 2023-07-29T00:25:11.968 DEBUG:boto:Token: None 2023-07-29T00:25:11.969 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/ 2023-07-29T00:25:11.969 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:roI/S2Qgkbc4BRAtVkkc6h5zINs= 2023-07-29T00:25:11.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:roI/S2Qgkbc4BRAtVkkc6h5zINs=', 'Content-Length': '0'} 2023-07-29T00:25:11.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000046b31e4a1849fbb-0064c45c67-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.973 DEBUG:boto:b'hiiqos-58obj000000000000000000421000falseobj000000000000000000422023-07-29T00:24:56.845Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:11.973 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:11.974 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:11.974 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000042 2023-07-29T00:25:11.975 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000042 2023-07-29T00:25:11.975 DEBUG:boto:Method: GET 2023-07-29T00:25:11.975 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000042 2023-07-29T00:25:11.976 DEBUG:boto:Data: 2023-07-29T00:25:11.976 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.976 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:11.977 DEBUG:boto:Port: 8000 2023-07-29T00:25:11.977 DEBUG:boto:Params: {} 2023-07-29T00:25:11.978 DEBUG:boto:Token: None 2023-07-29T00:25:11.978 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/ 2023-07-29T00:25:11.978 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:roI/S2Qgkbc4BRAtVkkc6h5zINs= 2023-07-29T00:25:11.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:roI/S2Qgkbc4BRAtVkkc6h5zINs=', 'Content-Length': '0'} 2023-07-29T00:25:11.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065aa8d41a99709f5-0064c45c67-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.984 DEBUG:boto:b'hiiqos-58obj000000000000000000421000falseobj000000000000000000422023-07-29T00:24:56.845Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:11.985 DEBUG:boto:path=/hiiqos-58/obj00000000000000000042 2023-07-29T00:25:11.985 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000042 2023-07-29T00:25:11.985 DEBUG:boto:Method: HEAD 2023-07-29T00:25:11.986 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000042 2023-07-29T00:25:11.986 DEBUG:boto:Data: 2023-07-29T00:25:11.986 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.987 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:11.987 DEBUG:boto:Port: 8001 2023-07-29T00:25:11.987 DEBUG:boto:Params: {} 2023-07-29T00:25:11.988 DEBUG:boto:Token: None 2023-07-29T00:25:11.988 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000042 2023-07-29T00:25:11.989 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:swjhFhPhLmmUtSNDW9fdrqEvmlg= 2023-07-29T00:25:11.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:swjhFhPhLmmUtSNDW9fdrqEvmlg=', 'Content-Length': '0'} 2023-07-29T00:25:11.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000106bf601fe187512-0064c45c67-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.992 DEBUG:boto:path=/hiiqos-58/obj00000000000000000042 2023-07-29T00:25:11.992 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000042 2023-07-29T00:25:11.992 DEBUG:boto:Method: HEAD 2023-07-29T00:25:11.993 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000042 2023-07-29T00:25:11.993 DEBUG:boto:Data: 2023-07-29T00:25:11.993 DEBUG:boto:Headers: {} 2023-07-29T00:25:11.994 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:11.994 DEBUG:boto:Port: 8000 2023-07-29T00:25:11.994 DEBUG:boto:Params: {} 2023-07-29T00:25:11.995 DEBUG:boto:Token: None 2023-07-29T00:25:11.995 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:11 GMT /hiiqos-58/obj00000000000000000042 2023-07-29T00:25:11.996 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:swjhFhPhLmmUtSNDW9fdrqEvmlg= 2023-07-29T00:25:11.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:swjhFhPhLmmUtSNDW9fdrqEvmlg=', 'Content-Length': '0'} 2023-07-29T00:25:11.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008effab6eaaa377a9-0064c45c67-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:11.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2023-07-29T00:25:11.999 DEBUG:boto:path=/hiiqos-58/obj00000000000000000042 2023-07-29T00:25:11.999 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000042 2023-07-29T00:25:12.000 DEBUG:boto:Method: GET 2023-07-29T00:25:12.000 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000042 2023-07-29T00:25:12.000 DEBUG:boto:Data: 2023-07-29T00:25:12.001 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.001 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:12.002 DEBUG:boto:Port: 8001 2023-07-29T00:25:12.002 DEBUG:boto:Params: {} 2023-07-29T00:25:12.002 DEBUG:boto:Token: None 2023-07-29T00:25:12.003 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/obj00000000000000000042 2023-07-29T00:25:12.003 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:E5MJNA1ZecvK23spUTlj4aW1OYE= 2023-07-29T00:25:12.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:E5MJNA1ZecvK23spUTlj4aW1OYE=', 'Content-Length': '0'} 2023-07-29T00:25:12.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd35877e95d2fd93-0064c45c67-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.052 DEBUG:boto:path=/hiiqos-58/obj00000000000000000042 2023-07-29T00:25:12.053 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000042 2023-07-29T00:25:12.053 DEBUG:boto:Method: GET 2023-07-29T00:25:12.054 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000042 2023-07-29T00:25:12.054 DEBUG:boto:Data: 2023-07-29T00:25:12.054 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.055 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:12.055 DEBUG:boto:Port: 8000 2023-07-29T00:25:12.055 DEBUG:boto:Params: {} 2023-07-29T00:25:12.056 DEBUG:boto:Token: None 2023-07-29T00:25:12.057 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/obj00000000000000000042 2023-07-29T00:25:12.058 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:E5MJNA1ZecvK23spUTlj4aW1OYE= 2023-07-29T00:25:12.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:E5MJNA1ZecvK23spUTlj4aW1OYE=', 'Content-Length': '0'} 2023-07-29T00:25:12.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf3b3fb2b8bc2626-0064c45c68-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.062 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:12.062 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:12.063 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000042 2023-07-29T00:25:12.063 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000042 2023-07-29T00:25:12.063 DEBUG:boto:Method: GET 2023-07-29T00:25:12.064 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000042 2023-07-29T00:25:12.064 DEBUG:boto:Data: 2023-07-29T00:25:12.064 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.065 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:12.066 DEBUG:boto:Port: 8001 2023-07-29T00:25:12.066 DEBUG:boto:Params: {} 2023-07-29T00:25:12.067 DEBUG:boto:Token: None 2023-07-29T00:25:12.067 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/ 2023-07-29T00:25:12.068 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7kh5dIuFkes7J+Ul3u5NWFfVa5E= 2023-07-29T00:25:12.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7kh5dIuFkes7J+Ul3u5NWFfVa5E=', 'Content-Length': '0'} 2023-07-29T00:25:12.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab3053f3b6f09bab-0064c45c68-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.073 DEBUG:boto:b'hiiqos-58obj000000000000000000421000falseobj000000000000000000422023-07-29T00:24:56.845Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:12.074 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:12.074 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:12.075 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000042 2023-07-29T00:25:12.076 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000042 2023-07-29T00:25:12.077 DEBUG:boto:Method: GET 2023-07-29T00:25:12.077 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000042 2023-07-29T00:25:12.078 DEBUG:boto:Data: 2023-07-29T00:25:12.080 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.080 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:12.082 DEBUG:boto:Port: 8000 2023-07-29T00:25:12.082 DEBUG:boto:Params: {} 2023-07-29T00:25:12.083 DEBUG:boto:Token: None 2023-07-29T00:25:12.084 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/ 2023-07-29T00:25:12.085 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7kh5dIuFkes7J+Ul3u5NWFfVa5E= 2023-07-29T00:25:12.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7kh5dIuFkes7J+Ul3u5NWFfVa5E=', 'Content-Length': '0'} 2023-07-29T00:25:12.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd686a9ecec19225-0064c45c68-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.132 DEBUG:boto:b'hiiqos-58obj000000000000000000421000falseobj000000000000000000422023-07-29T00:24:56.845Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:12.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2023-07-29T00:25:12.134 DEBUG:boto:path=/hiiqos-58/obj00000000000000000045 2023-07-29T00:25:12.134 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000045 2023-07-29T00:25:12.134 DEBUG:boto:path=/hiiqos-58/obj00000000000000000045?versionId=null 2023-07-29T00:25:12.135 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000045?versionId=null 2023-07-29T00:25:12.135 DEBUG:boto:Method: GET 2023-07-29T00:25:12.135 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000045?versionId=null 2023-07-29T00:25:12.136 DEBUG:boto:Data: 2023-07-29T00:25:12.136 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.137 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:12.137 DEBUG:boto:Port: 8001 2023-07-29T00:25:12.137 DEBUG:boto:Params: {} 2023-07-29T00:25:12.138 DEBUG:boto:Token: None 2023-07-29T00:25:12.138 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/obj00000000000000000045?versionId=null 2023-07-29T00:25:12.139 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TusilXPrxmkqvXGr3Zdlv0QWRxE= 2023-07-29T00:25:12.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TusilXPrxmkqvXGr3Zdlv0QWRxE=', 'Content-Length': '0'} 2023-07-29T00:25:12.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae6a9922772fab91-0064c45c68-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.142 DEBUG:boto:path=/hiiqos-58/obj00000000000000000045 2023-07-29T00:25:12.142 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000045 2023-07-29T00:25:12.143 DEBUG:boto:path=/hiiqos-58/obj00000000000000000045?versionId=null 2023-07-29T00:25:12.143 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000045?versionId=null 2023-07-29T00:25:12.144 DEBUG:boto:Method: GET 2023-07-29T00:25:12.144 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000045?versionId=null 2023-07-29T00:25:12.144 DEBUG:boto:Data: 2023-07-29T00:25:12.145 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.145 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:12.146 DEBUG:boto:Port: 8000 2023-07-29T00:25:12.146 DEBUG:boto:Params: {} 2023-07-29T00:25:12.147 DEBUG:boto:Token: None 2023-07-29T00:25:12.147 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/obj00000000000000000045?versionId=null 2023-07-29T00:25:12.147 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TusilXPrxmkqvXGr3Zdlv0QWRxE= 2023-07-29T00:25:12.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TusilXPrxmkqvXGr3Zdlv0QWRxE=', 'Content-Length': '0'} 2023-07-29T00:25:12.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056e22b1cd971ad19-0064c45c68-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.151 DEBUG:boto:path=/hiiqos-58/obj00000000000000000045 2023-07-29T00:25:12.151 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000045 2023-07-29T00:25:12.151 DEBUG:boto:path=/hiiqos-58/obj00000000000000000045?versionId=null 2023-07-29T00:25:12.152 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000045?versionId=null 2023-07-29T00:25:12.152 DEBUG:boto:Method: HEAD 2023-07-29T00:25:12.152 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000045?versionId=null 2023-07-29T00:25:12.153 DEBUG:boto:Data: 2023-07-29T00:25:12.153 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.154 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:12.154 DEBUG:boto:Port: 8001 2023-07-29T00:25:12.154 DEBUG:boto:Params: {} 2023-07-29T00:25:12.155 DEBUG:boto:Token: None 2023-07-29T00:25:12.155 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/obj00000000000000000045?versionId=null 2023-07-29T00:25:12.155 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:N77bwL1lkHKsdTTCiwYCLTlpvY0= 2023-07-29T00:25:12.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:N77bwL1lkHKsdTTCiwYCLTlpvY0=', 'Content-Length': '0'} 2023-07-29T00:25:12.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094974454dbb3a387-0064c45c68-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.158 DEBUG:boto:path=/hiiqos-58/obj00000000000000000045 2023-07-29T00:25:12.159 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000045 2023-07-29T00:25:12.160 DEBUG:boto:path=/hiiqos-58/obj00000000000000000045?versionId=null 2023-07-29T00:25:12.160 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000045?versionId=null 2023-07-29T00:25:12.161 DEBUG:boto:Method: HEAD 2023-07-29T00:25:12.161 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000045?versionId=null 2023-07-29T00:25:12.162 DEBUG:boto:Data: 2023-07-29T00:25:12.162 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.162 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:12.163 DEBUG:boto:Port: 8000 2023-07-29T00:25:12.164 DEBUG:boto:Params: {} 2023-07-29T00:25:12.165 DEBUG:boto:Token: None 2023-07-29T00:25:12.166 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/obj00000000000000000045?versionId=null 2023-07-29T00:25:12.167 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:N77bwL1lkHKsdTTCiwYCLTlpvY0= 2023-07-29T00:25:12.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:N77bwL1lkHKsdTTCiwYCLTlpvY0=', 'Content-Length': '0'} 2023-07-29T00:25:12.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ca4224604fc7bbb-0064c45c68-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2023-07-29T00:25:12.172 DEBUG:boto:path=/hiiqos-58/obj00000000000000000045 2023-07-29T00:25:12.173 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000045 2023-07-29T00:25:12.173 DEBUG:boto:path=/hiiqos-58/obj00000000000000000045?versionId=null 2023-07-29T00:25:12.174 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000045?versionId=null 2023-07-29T00:25:12.175 DEBUG:boto:Method: GET 2023-07-29T00:25:12.175 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000045?versionId=null 2023-07-29T00:25:12.175 DEBUG:boto:Data: 2023-07-29T00:25:12.176 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.176 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:12.177 DEBUG:boto:Port: 8001 2023-07-29T00:25:12.177 DEBUG:boto:Params: {} 2023-07-29T00:25:12.177 DEBUG:boto:Token: None 2023-07-29T00:25:12.178 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/obj00000000000000000045?versionId=null 2023-07-29T00:25:12.178 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TusilXPrxmkqvXGr3Zdlv0QWRxE= 2023-07-29T00:25:12.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TusilXPrxmkqvXGr3Zdlv0QWRxE=', 'Content-Length': '0'} 2023-07-29T00:25:12.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b2a88b51e40c27b-0064c45c68-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.182 DEBUG:boto:path=/hiiqos-58/obj00000000000000000045 2023-07-29T00:25:12.182 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000045 2023-07-29T00:25:12.183 DEBUG:boto:path=/hiiqos-58/obj00000000000000000045?versionId=null 2023-07-29T00:25:12.183 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000045?versionId=null 2023-07-29T00:25:12.183 DEBUG:boto:Method: GET 2023-07-29T00:25:12.184 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000045?versionId=null 2023-07-29T00:25:12.185 DEBUG:boto:Data: 2023-07-29T00:25:12.185 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.186 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:12.186 DEBUG:boto:Port: 8000 2023-07-29T00:25:12.187 DEBUG:boto:Params: {} 2023-07-29T00:25:12.187 DEBUG:boto:Token: None 2023-07-29T00:25:12.188 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/obj00000000000000000045?versionId=null 2023-07-29T00:25:12.188 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TusilXPrxmkqvXGr3Zdlv0QWRxE= 2023-07-29T00:25:12.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TusilXPrxmkqvXGr3Zdlv0QWRxE=', 'Content-Length': '0'} 2023-07-29T00:25:12.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f7125843ce86a2c-0064c45c68-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.192 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:12.192 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:12.193 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000045 2023-07-29T00:25:12.193 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000045 2023-07-29T00:25:12.194 DEBUG:boto:Method: GET 2023-07-29T00:25:12.194 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000045 2023-07-29T00:25:12.194 DEBUG:boto:Data: 2023-07-29T00:25:12.195 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.196 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:12.197 DEBUG:boto:Port: 8001 2023-07-29T00:25:12.197 DEBUG:boto:Params: {} 2023-07-29T00:25:12.198 DEBUG:boto:Token: None 2023-07-29T00:25:12.200 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/ 2023-07-29T00:25:12.200 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7kh5dIuFkes7J+Ul3u5NWFfVa5E= 2023-07-29T00:25:12.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7kh5dIuFkes7J+Ul3u5NWFfVa5E=', 'Content-Length': '0'} 2023-07-29T00:25:12.204 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c5ce5d55c78679c-0064c45c68-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.244 DEBUG:boto:b'hiiqos-58obj000000000000000000451000falseobj000000000000000000452023-07-29T00:25:00.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:12.246 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:12.247 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:12.248 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000045 2023-07-29T00:25:12.249 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000045 2023-07-29T00:25:12.250 DEBUG:boto:Method: GET 2023-07-29T00:25:12.250 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000045 2023-07-29T00:25:12.251 DEBUG:boto:Data: 2023-07-29T00:25:12.252 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.253 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:12.253 DEBUG:boto:Port: 8000 2023-07-29T00:25:12.254 DEBUG:boto:Params: {} 2023-07-29T00:25:12.255 DEBUG:boto:Token: None 2023-07-29T00:25:12.255 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/ 2023-07-29T00:25:12.256 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7kh5dIuFkes7J+Ul3u5NWFfVa5E= 2023-07-29T00:25:12.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7kh5dIuFkes7J+Ul3u5NWFfVa5E=', 'Content-Length': '0'} 2023-07-29T00:25:12.262 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050edc51710971741-0064c45c68-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.262 DEBUG:boto:b'hiiqos-58obj000000000000000000451000falseobj000000000000000000452023-07-29T00:25:00.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:12.264 DEBUG:boto:path=/hiiqos-58/obj00000000000000000045 2023-07-29T00:25:12.264 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000045 2023-07-29T00:25:12.265 DEBUG:boto:Method: HEAD 2023-07-29T00:25:12.266 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000045 2023-07-29T00:25:12.267 DEBUG:boto:Data: 2023-07-29T00:25:12.268 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.269 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:12.270 DEBUG:boto:Port: 8001 2023-07-29T00:25:12.271 DEBUG:boto:Params: {} 2023-07-29T00:25:12.272 DEBUG:boto:Token: None 2023-07-29T00:25:12.273 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/obj00000000000000000045 2023-07-29T00:25:12.274 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:g80X72mC5dtzHJkV/7rRzJR1VJY= 2023-07-29T00:25:12.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:g80X72mC5dtzHJkV/7rRzJR1VJY=', 'Content-Length': '0'} 2023-07-29T00:25:12.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d400964d9531e80-0064c45c68-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.279 DEBUG:boto:path=/hiiqos-58/obj00000000000000000045 2023-07-29T00:25:12.279 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000045 2023-07-29T00:25:12.280 DEBUG:boto:Method: HEAD 2023-07-29T00:25:12.280 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000045 2023-07-29T00:25:12.280 DEBUG:boto:Data: 2023-07-29T00:25:12.281 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.281 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:12.281 DEBUG:boto:Port: 8000 2023-07-29T00:25:12.282 DEBUG:boto:Params: {} 2023-07-29T00:25:12.282 DEBUG:boto:Token: None 2023-07-29T00:25:12.282 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/obj00000000000000000045 2023-07-29T00:25:12.283 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:g80X72mC5dtzHJkV/7rRzJR1VJY= 2023-07-29T00:25:12.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:g80X72mC5dtzHJkV/7rRzJR1VJY=', 'Content-Length': '0'} 2023-07-29T00:25:12.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070d88a856d7c1715-0064c45c68-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2023-07-29T00:25:12.286 DEBUG:boto:path=/hiiqos-58/obj00000000000000000045 2023-07-29T00:25:12.287 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000045 2023-07-29T00:25:12.287 DEBUG:boto:Method: GET 2023-07-29T00:25:12.288 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000045 2023-07-29T00:25:12.288 DEBUG:boto:Data: 2023-07-29T00:25:12.289 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.289 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:12.289 DEBUG:boto:Port: 8001 2023-07-29T00:25:12.290 DEBUG:boto:Params: {} 2023-07-29T00:25:12.290 DEBUG:boto:Token: None 2023-07-29T00:25:12.290 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/obj00000000000000000045 2023-07-29T00:25:12.291 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qO0NE2zfZOGCrFPZuuuDxpXwlfc= 2023-07-29T00:25:12.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qO0NE2zfZOGCrFPZuuuDxpXwlfc=', 'Content-Length': '0'} 2023-07-29T00:25:12.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f13a40ea8a291de-0064c45c68-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.294 DEBUG:boto:path=/hiiqos-58/obj00000000000000000045 2023-07-29T00:25:12.294 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000045 2023-07-29T00:25:12.295 DEBUG:boto:Method: GET 2023-07-29T00:25:12.295 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000045 2023-07-29T00:25:12.295 DEBUG:boto:Data: 2023-07-29T00:25:12.296 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.296 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:12.296 DEBUG:boto:Port: 8000 2023-07-29T00:25:12.297 DEBUG:boto:Params: {} 2023-07-29T00:25:12.297 DEBUG:boto:Token: None 2023-07-29T00:25:12.298 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/obj00000000000000000045 2023-07-29T00:25:12.298 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qO0NE2zfZOGCrFPZuuuDxpXwlfc= 2023-07-29T00:25:12.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qO0NE2zfZOGCrFPZuuuDxpXwlfc=', 'Content-Length': '0'} 2023-07-29T00:25:12.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7d511472b715eec-0064c45c68-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.303 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:12.304 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:12.305 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000045 2023-07-29T00:25:12.305 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000045 2023-07-29T00:25:12.306 DEBUG:boto:Method: GET 2023-07-29T00:25:12.306 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000045 2023-07-29T00:25:12.307 DEBUG:boto:Data: 2023-07-29T00:25:12.307 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.307 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:12.308 DEBUG:boto:Port: 8001 2023-07-29T00:25:12.308 DEBUG:boto:Params: {} 2023-07-29T00:25:12.309 DEBUG:boto:Token: None 2023-07-29T00:25:12.309 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/ 2023-07-29T00:25:12.309 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7kh5dIuFkes7J+Ul3u5NWFfVa5E= 2023-07-29T00:25:12.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7kh5dIuFkes7J+Ul3u5NWFfVa5E=', 'Content-Length': '0'} 2023-07-29T00:25:12.312 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7a19cd467fb702f-0064c45c68-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.313 DEBUG:boto:b'hiiqos-58obj000000000000000000451000falseobj000000000000000000452023-07-29T00:25:00.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:12.314 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:12.315 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:12.315 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000045 2023-07-29T00:25:12.316 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000045 2023-07-29T00:25:12.316 DEBUG:boto:Method: GET 2023-07-29T00:25:12.317 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000045 2023-07-29T00:25:12.317 DEBUG:boto:Data: 2023-07-29T00:25:12.317 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.318 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:12.318 DEBUG:boto:Port: 8000 2023-07-29T00:25:12.319 DEBUG:boto:Params: {} 2023-07-29T00:25:12.319 DEBUG:boto:Token: None 2023-07-29T00:25:12.320 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/ 2023-07-29T00:25:12.320 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7kh5dIuFkes7J+Ul3u5NWFfVa5E= 2023-07-29T00:25:12.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7kh5dIuFkes7J+Ul3u5NWFfVa5E=', 'Content-Length': '0'} 2023-07-29T00:25:12.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f53d09851f8a6f4-0064c45c68-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.368 DEBUG:boto:b'hiiqos-58obj000000000000000000451000falseobj000000000000000000452023-07-29T00:25:00.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:12.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000046 2023-07-29T00:25:12.370 DEBUG:boto:path=/hiiqos-58/obj00000000000000000046 2023-07-29T00:25:12.370 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000046 2023-07-29T00:25:12.371 DEBUG:boto:path=/hiiqos-58/obj00000000000000000046?versionId=null 2023-07-29T00:25:12.371 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000046?versionId=null 2023-07-29T00:25:12.371 DEBUG:boto:Method: GET 2023-07-29T00:25:12.372 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000046?versionId=null 2023-07-29T00:25:12.372 DEBUG:boto:Data: 2023-07-29T00:25:12.373 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.373 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:12.373 DEBUG:boto:Port: 8001 2023-07-29T00:25:12.374 DEBUG:boto:Params: {} 2023-07-29T00:25:12.374 DEBUG:boto:Token: None 2023-07-29T00:25:12.375 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/obj00000000000000000046?versionId=null 2023-07-29T00:25:12.375 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FoValeOblv83knkKXHd0hgib1N4= 2023-07-29T00:25:12.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FoValeOblv83knkKXHd0hgib1N4=', 'Content-Length': '0'} 2023-07-29T00:25:12.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000573c4aa811a2eca3-0064c45c68-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.379 DEBUG:boto:path=/hiiqos-58/obj00000000000000000046 2023-07-29T00:25:12.379 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000046 2023-07-29T00:25:12.380 DEBUG:boto:path=/hiiqos-58/obj00000000000000000046?versionId=null 2023-07-29T00:25:12.380 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000046?versionId=null 2023-07-29T00:25:12.380 DEBUG:boto:Method: GET 2023-07-29T00:25:12.381 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000046?versionId=null 2023-07-29T00:25:12.381 DEBUG:boto:Data: 2023-07-29T00:25:12.381 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.382 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:12.382 DEBUG:boto:Port: 8000 2023-07-29T00:25:12.383 DEBUG:boto:Params: {} 2023-07-29T00:25:12.384 DEBUG:boto:Token: None 2023-07-29T00:25:12.385 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/obj00000000000000000046?versionId=null 2023-07-29T00:25:12.386 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FoValeOblv83knkKXHd0hgib1N4= 2023-07-29T00:25:12.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FoValeOblv83knkKXHd0hgib1N4=', 'Content-Length': '0'} 2023-07-29T00:25:12.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050f4810204b2aa36-0064c45c68-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.389 DEBUG:boto:path=/hiiqos-58/obj00000000000000000046 2023-07-29T00:25:12.389 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000046 2023-07-29T00:25:12.390 DEBUG:boto:path=/hiiqos-58/obj00000000000000000046?versionId=null 2023-07-29T00:25:12.390 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000046?versionId=null 2023-07-29T00:25:12.391 DEBUG:boto:Method: HEAD 2023-07-29T00:25:12.391 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000046?versionId=null 2023-07-29T00:25:12.392 DEBUG:boto:Data: 2023-07-29T00:25:12.392 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.393 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:12.393 DEBUG:boto:Port: 8001 2023-07-29T00:25:12.394 DEBUG:boto:Params: {} 2023-07-29T00:25:12.394 DEBUG:boto:Token: None 2023-07-29T00:25:12.394 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/obj00000000000000000046?versionId=null 2023-07-29T00:25:12.395 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QlgpBU6lHDvaTV5yuuq7BiIJLyw= 2023-07-29T00:25:12.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QlgpBU6lHDvaTV5yuuq7BiIJLyw=', 'Content-Length': '0'} 2023-07-29T00:25:12.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a570e58e981cfd94-0064c45c68-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.399 DEBUG:boto:path=/hiiqos-58/obj00000000000000000046 2023-07-29T00:25:12.399 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000046 2023-07-29T00:25:12.400 DEBUG:boto:path=/hiiqos-58/obj00000000000000000046?versionId=null 2023-07-29T00:25:12.400 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000046?versionId=null 2023-07-29T00:25:12.400 DEBUG:boto:Method: HEAD 2023-07-29T00:25:12.401 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000046?versionId=null 2023-07-29T00:25:12.402 DEBUG:boto:Data: 2023-07-29T00:25:12.403 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.404 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:12.405 DEBUG:boto:Port: 8000 2023-07-29T00:25:12.406 DEBUG:boto:Params: {} 2023-07-29T00:25:12.406 DEBUG:boto:Token: None 2023-07-29T00:25:12.407 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/obj00000000000000000046?versionId=null 2023-07-29T00:25:12.408 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QlgpBU6lHDvaTV5yuuq7BiIJLyw= 2023-07-29T00:25:12.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QlgpBU6lHDvaTV5yuuq7BiIJLyw=', 'Content-Length': '0'} 2023-07-29T00:25:12.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069da5e33d7554a34-0064c45c68-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000046 2023-07-29T00:25:12.413 DEBUG:boto:path=/hiiqos-58/obj00000000000000000046 2023-07-29T00:25:12.414 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000046 2023-07-29T00:25:12.414 DEBUG:boto:path=/hiiqos-58/obj00000000000000000046?versionId=null 2023-07-29T00:25:12.415 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000046?versionId=null 2023-07-29T00:25:12.416 DEBUG:boto:Method: GET 2023-07-29T00:25:12.416 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000046?versionId=null 2023-07-29T00:25:12.417 DEBUG:boto:Data: 2023-07-29T00:25:12.417 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.418 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:12.418 DEBUG:boto:Port: 8001 2023-07-29T00:25:12.419 DEBUG:boto:Params: {} 2023-07-29T00:25:12.419 DEBUG:boto:Token: None 2023-07-29T00:25:12.420 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/obj00000000000000000046?versionId=null 2023-07-29T00:25:12.420 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FoValeOblv83knkKXHd0hgib1N4= 2023-07-29T00:25:12.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FoValeOblv83knkKXHd0hgib1N4=', 'Content-Length': '0'} 2023-07-29T00:25:12.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb0c1effc561c9a5-0064c45c68-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.423 DEBUG:boto:path=/hiiqos-58/obj00000000000000000046 2023-07-29T00:25:12.424 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000046 2023-07-29T00:25:12.424 DEBUG:boto:path=/hiiqos-58/obj00000000000000000046?versionId=null 2023-07-29T00:25:12.425 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000046?versionId=null 2023-07-29T00:25:12.426 DEBUG:boto:Method: GET 2023-07-29T00:25:12.427 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000046?versionId=null 2023-07-29T00:25:12.428 DEBUG:boto:Data: 2023-07-29T00:25:12.429 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.429 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:12.430 DEBUG:boto:Port: 8000 2023-07-29T00:25:12.431 DEBUG:boto:Params: {} 2023-07-29T00:25:12.432 DEBUG:boto:Token: None 2023-07-29T00:25:12.433 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/obj00000000000000000046?versionId=null 2023-07-29T00:25:12.434 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FoValeOblv83knkKXHd0hgib1N4= 2023-07-29T00:25:12.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FoValeOblv83knkKXHd0hgib1N4=', 'Content-Length': '0'} 2023-07-29T00:25:12.438 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083bcd6aab6719f74-0064c45c68-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.440 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:12.441 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:12.442 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000046 2023-07-29T00:25:12.442 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000046 2023-07-29T00:25:12.443 DEBUG:boto:Method: GET 2023-07-29T00:25:12.444 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000046 2023-07-29T00:25:12.444 DEBUG:boto:Data: 2023-07-29T00:25:12.445 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.446 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:12.446 DEBUG:boto:Port: 8001 2023-07-29T00:25:12.447 DEBUG:boto:Params: {} 2023-07-29T00:25:12.448 DEBUG:boto:Token: None 2023-07-29T00:25:12.449 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/ 2023-07-29T00:25:12.450 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7kh5dIuFkes7J+Ul3u5NWFfVa5E= 2023-07-29T00:25:12.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7kh5dIuFkes7J+Ul3u5NWFfVa5E=', 'Content-Length': '0'} 2023-07-29T00:25:12.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f702718ba6ee330-0064c45c68-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.455 DEBUG:boto:b'hiiqos-58obj000000000000000000461000falseobj000000000000000000462023-07-29T00:24:55.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:12.455 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:12.456 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:12.456 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000046 2023-07-29T00:25:12.457 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000046 2023-07-29T00:25:12.457 DEBUG:boto:Method: GET 2023-07-29T00:25:12.458 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000046 2023-07-29T00:25:12.458 DEBUG:boto:Data: 2023-07-29T00:25:12.458 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.459 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:12.460 DEBUG:boto:Port: 8000 2023-07-29T00:25:12.461 DEBUG:boto:Params: {} 2023-07-29T00:25:12.461 DEBUG:boto:Token: None 2023-07-29T00:25:12.462 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/ 2023-07-29T00:25:12.463 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7kh5dIuFkes7J+Ul3u5NWFfVa5E= 2023-07-29T00:25:12.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7kh5dIuFkes7J+Ul3u5NWFfVa5E=', 'Content-Length': '0'} 2023-07-29T00:25:12.469 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f788b66882e003bf-0064c45c68-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.471 DEBUG:boto:b'hiiqos-58obj000000000000000000461000falseobj000000000000000000462023-07-29T00:24:55.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:12.472 DEBUG:boto:path=/hiiqos-58/obj00000000000000000046 2023-07-29T00:25:12.472 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000046 2023-07-29T00:25:12.473 DEBUG:boto:Method: HEAD 2023-07-29T00:25:12.473 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000046 2023-07-29T00:25:12.473 DEBUG:boto:Data: 2023-07-29T00:25:12.474 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.474 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:12.475 DEBUG:boto:Port: 8001 2023-07-29T00:25:12.475 DEBUG:boto:Params: {} 2023-07-29T00:25:12.475 DEBUG:boto:Token: None 2023-07-29T00:25:12.476 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/obj00000000000000000046 2023-07-29T00:25:12.476 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Q/HneqZP7VmqYsPNIdqivpXz5HQ= 2023-07-29T00:25:12.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q/HneqZP7VmqYsPNIdqivpXz5HQ=', 'Content-Length': '0'} 2023-07-29T00:25:12.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7432891972f9bd9-0064c45c68-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.480 DEBUG:boto:path=/hiiqos-58/obj00000000000000000046 2023-07-29T00:25:12.480 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000046 2023-07-29T00:25:12.480 DEBUG:boto:Method: HEAD 2023-07-29T00:25:12.481 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000046 2023-07-29T00:25:12.481 DEBUG:boto:Data: 2023-07-29T00:25:12.482 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.483 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:12.484 DEBUG:boto:Port: 8000 2023-07-29T00:25:12.484 DEBUG:boto:Params: {} 2023-07-29T00:25:12.485 DEBUG:boto:Token: None 2023-07-29T00:25:12.485 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/obj00000000000000000046 2023-07-29T00:25:12.486 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Q/HneqZP7VmqYsPNIdqivpXz5HQ= 2023-07-29T00:25:12.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q/HneqZP7VmqYsPNIdqivpXz5HQ=', 'Content-Length': '0'} 2023-07-29T00:25:12.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a393270a95865d82-0064c45c68-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000046 2023-07-29T00:25:12.489 DEBUG:boto:path=/hiiqos-58/obj00000000000000000046 2023-07-29T00:25:12.489 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000046 2023-07-29T00:25:12.490 DEBUG:boto:Method: GET 2023-07-29T00:25:12.490 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000046 2023-07-29T00:25:12.491 DEBUG:boto:Data: 2023-07-29T00:25:12.491 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.491 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:12.492 DEBUG:boto:Port: 8001 2023-07-29T00:25:12.492 DEBUG:boto:Params: {} 2023-07-29T00:25:12.492 DEBUG:boto:Token: None 2023-07-29T00:25:12.493 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/obj00000000000000000046 2023-07-29T00:25:12.493 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nOM4OU5KNjw5HNPJVSQ8kBt+gIw= 2023-07-29T00:25:12.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nOM4OU5KNjw5HNPJVSQ8kBt+gIw=', 'Content-Length': '0'} 2023-07-29T00:25:12.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056149f74ef9055b6-0064c45c68-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.497 DEBUG:boto:path=/hiiqos-58/obj00000000000000000046 2023-07-29T00:25:12.498 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000046 2023-07-29T00:25:12.498 DEBUG:boto:Method: GET 2023-07-29T00:25:12.499 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000046 2023-07-29T00:25:12.499 DEBUG:boto:Data: 2023-07-29T00:25:12.499 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.500 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:12.500 DEBUG:boto:Port: 8000 2023-07-29T00:25:12.501 DEBUG:boto:Params: {} 2023-07-29T00:25:12.501 DEBUG:boto:Token: None 2023-07-29T00:25:12.502 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/obj00000000000000000046 2023-07-29T00:25:12.502 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nOM4OU5KNjw5HNPJVSQ8kBt+gIw= 2023-07-29T00:25:12.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nOM4OU5KNjw5HNPJVSQ8kBt+gIw=', 'Content-Length': '0'} 2023-07-29T00:25:12.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6e7cf27de05d04e-0064c45c68-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.552 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:12.553 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:12.554 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000046 2023-07-29T00:25:12.554 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000046 2023-07-29T00:25:12.554 DEBUG:boto:Method: GET 2023-07-29T00:25:12.555 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000046 2023-07-29T00:25:12.555 DEBUG:boto:Data: 2023-07-29T00:25:12.556 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.556 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:12.557 DEBUG:boto:Port: 8001 2023-07-29T00:25:12.557 DEBUG:boto:Params: {} 2023-07-29T00:25:12.558 DEBUG:boto:Token: None 2023-07-29T00:25:12.558 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/ 2023-07-29T00:25:12.558 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7kh5dIuFkes7J+Ul3u5NWFfVa5E= 2023-07-29T00:25:12.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7kh5dIuFkes7J+Ul3u5NWFfVa5E=', 'Content-Length': '0'} 2023-07-29T00:25:12.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce522d6430feaf06-0064c45c68-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.621 DEBUG:boto:b'hiiqos-58obj000000000000000000461000falseobj000000000000000000462023-07-29T00:24:55.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:12.622 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:12.622 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:12.623 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000046 2023-07-29T00:25:12.623 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000046 2023-07-29T00:25:12.624 DEBUG:boto:Method: GET 2023-07-29T00:25:12.624 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000046 2023-07-29T00:25:12.624 DEBUG:boto:Data: 2023-07-29T00:25:12.625 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.625 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:12.625 DEBUG:boto:Port: 8000 2023-07-29T00:25:12.626 DEBUG:boto:Params: {} 2023-07-29T00:25:12.626 DEBUG:boto:Token: None 2023-07-29T00:25:12.627 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/ 2023-07-29T00:25:12.627 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7kh5dIuFkes7J+Ul3u5NWFfVa5E= 2023-07-29T00:25:12.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7kh5dIuFkes7J+Ul3u5NWFfVa5E=', 'Content-Length': '0'} 2023-07-29T00:25:12.632 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046b3155c13b38fb5-0064c45c68-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.632 DEBUG:boto:b'hiiqos-58obj000000000000000000461000falseobj000000000000000000462023-07-29T00:24:55.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:12.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000048 2023-07-29T00:25:12.634 DEBUG:boto:path=/hiiqos-58/obj00000000000000000048 2023-07-29T00:25:12.634 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000048 2023-07-29T00:25:12.634 DEBUG:boto:path=/hiiqos-58/obj00000000000000000048?versionId=null 2023-07-29T00:25:12.635 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000048?versionId=null 2023-07-29T00:25:12.635 DEBUG:boto:Method: GET 2023-07-29T00:25:12.636 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000048?versionId=null 2023-07-29T00:25:12.636 DEBUG:boto:Data: 2023-07-29T00:25:12.636 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.637 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:12.637 DEBUG:boto:Port: 8001 2023-07-29T00:25:12.637 DEBUG:boto:Params: {} 2023-07-29T00:25:12.638 DEBUG:boto:Token: None 2023-07-29T00:25:12.638 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/obj00000000000000000048?versionId=null 2023-07-29T00:25:12.639 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Fg78P6Fp+Vov0kDzPvukmd9Bkxs= 2023-07-29T00:25:12.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Fg78P6Fp+Vov0kDzPvukmd9Bkxs=', 'Content-Length': '0'} 2023-07-29T00:25:12.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a12b111ba03d202-0064c45c68-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.644 DEBUG:boto:path=/hiiqos-58/obj00000000000000000048 2023-07-29T00:25:12.644 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000048 2023-07-29T00:25:12.645 DEBUG:boto:path=/hiiqos-58/obj00000000000000000048?versionId=null 2023-07-29T00:25:12.645 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000048?versionId=null 2023-07-29T00:25:12.645 DEBUG:boto:Method: GET 2023-07-29T00:25:12.646 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000048?versionId=null 2023-07-29T00:25:12.646 DEBUG:boto:Data: 2023-07-29T00:25:12.647 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.647 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:12.647 DEBUG:boto:Port: 8000 2023-07-29T00:25:12.648 DEBUG:boto:Params: {} 2023-07-29T00:25:12.648 DEBUG:boto:Token: None 2023-07-29T00:25:12.648 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/obj00000000000000000048?versionId=null 2023-07-29T00:25:12.649 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Fg78P6Fp+Vov0kDzPvukmd9Bkxs= 2023-07-29T00:25:12.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Fg78P6Fp+Vov0kDzPvukmd9Bkxs=', 'Content-Length': '0'} 2023-07-29T00:25:12.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfb5e08dfc7e64e1-0064c45c68-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.652 DEBUG:boto:path=/hiiqos-58/obj00000000000000000048 2023-07-29T00:25:12.653 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000048 2023-07-29T00:25:12.653 DEBUG:boto:path=/hiiqos-58/obj00000000000000000048?versionId=null 2023-07-29T00:25:12.653 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000048?versionId=null 2023-07-29T00:25:12.654 DEBUG:boto:Method: HEAD 2023-07-29T00:25:12.654 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000048?versionId=null 2023-07-29T00:25:12.654 DEBUG:boto:Data: 2023-07-29T00:25:12.655 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.655 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:12.655 DEBUG:boto:Port: 8001 2023-07-29T00:25:12.656 DEBUG:boto:Params: {} 2023-07-29T00:25:12.656 DEBUG:boto:Token: None 2023-07-29T00:25:12.657 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/obj00000000000000000048?versionId=null 2023-07-29T00:25:12.657 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TgEZMi41AJkTNmGpShMQolNfTLQ= 2023-07-29T00:25:12.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TgEZMi41AJkTNmGpShMQolNfTLQ=', 'Content-Length': '0'} 2023-07-29T00:25:12.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015fcab99a6d97f12-0064c45c68-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.660 DEBUG:boto:path=/hiiqos-58/obj00000000000000000048 2023-07-29T00:25:12.660 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000048 2023-07-29T00:25:12.661 DEBUG:boto:path=/hiiqos-58/obj00000000000000000048?versionId=null 2023-07-29T00:25:12.661 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000048?versionId=null 2023-07-29T00:25:12.662 DEBUG:boto:Method: HEAD 2023-07-29T00:25:12.662 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000048?versionId=null 2023-07-29T00:25:12.662 DEBUG:boto:Data: 2023-07-29T00:25:12.663 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.663 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:12.663 DEBUG:boto:Port: 8000 2023-07-29T00:25:12.664 DEBUG:boto:Params: {} 2023-07-29T00:25:12.664 DEBUG:boto:Token: None 2023-07-29T00:25:12.665 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/obj00000000000000000048?versionId=null 2023-07-29T00:25:12.665 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TgEZMi41AJkTNmGpShMQolNfTLQ= 2023-07-29T00:25:12.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TgEZMi41AJkTNmGpShMQolNfTLQ=', 'Content-Length': '0'} 2023-07-29T00:25:12.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058ec2e4d0ff82b10-0064c45c68-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000048 2023-07-29T00:25:12.669 DEBUG:boto:path=/hiiqos-58/obj00000000000000000048 2023-07-29T00:25:12.669 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000048 2023-07-29T00:25:12.669 DEBUG:boto:path=/hiiqos-58/obj00000000000000000048?versionId=null 2023-07-29T00:25:12.670 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000048?versionId=null 2023-07-29T00:25:12.670 DEBUG:boto:Method: GET 2023-07-29T00:25:12.671 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000048?versionId=null 2023-07-29T00:25:12.672 DEBUG:boto:Data: 2023-07-29T00:25:12.672 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.673 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:12.673 DEBUG:boto:Port: 8001 2023-07-29T00:25:12.674 DEBUG:boto:Params: {} 2023-07-29T00:25:12.674 DEBUG:boto:Token: None 2023-07-29T00:25:12.675 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/obj00000000000000000048?versionId=null 2023-07-29T00:25:12.676 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Fg78P6Fp+Vov0kDzPvukmd9Bkxs= 2023-07-29T00:25:12.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Fg78P6Fp+Vov0kDzPvukmd9Bkxs=', 'Content-Length': '0'} 2023-07-29T00:25:12.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000decad8d1b39997b8-0064c45c68-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.681 DEBUG:boto:path=/hiiqos-58/obj00000000000000000048 2023-07-29T00:25:12.682 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000048 2023-07-29T00:25:12.683 DEBUG:boto:path=/hiiqos-58/obj00000000000000000048?versionId=null 2023-07-29T00:25:12.683 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000048?versionId=null 2023-07-29T00:25:12.683 DEBUG:boto:Method: GET 2023-07-29T00:25:12.684 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000048?versionId=null 2023-07-29T00:25:12.684 DEBUG:boto:Data: 2023-07-29T00:25:12.685 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.685 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:12.685 DEBUG:boto:Port: 8000 2023-07-29T00:25:12.686 DEBUG:boto:Params: {} 2023-07-29T00:25:12.686 DEBUG:boto:Token: None 2023-07-29T00:25:12.687 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/obj00000000000000000048?versionId=null 2023-07-29T00:25:12.687 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Fg78P6Fp+Vov0kDzPvukmd9Bkxs= 2023-07-29T00:25:12.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Fg78P6Fp+Vov0kDzPvukmd9Bkxs=', 'Content-Length': '0'} 2023-07-29T00:25:12.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e0ff3573a222c38-0064c45c68-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.691 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:12.692 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:12.693 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000048 2023-07-29T00:25:12.693 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000048 2023-07-29T00:25:12.694 DEBUG:boto:Method: GET 2023-07-29T00:25:12.694 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000048 2023-07-29T00:25:12.694 DEBUG:boto:Data: 2023-07-29T00:25:12.695 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.695 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:12.695 DEBUG:boto:Port: 8001 2023-07-29T00:25:12.696 DEBUG:boto:Params: {} 2023-07-29T00:25:12.696 DEBUG:boto:Token: None 2023-07-29T00:25:12.697 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/ 2023-07-29T00:25:12.697 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7kh5dIuFkes7J+Ul3u5NWFfVa5E= 2023-07-29T00:25:12.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7kh5dIuFkes7J+Ul3u5NWFfVa5E=', 'Content-Length': '0'} 2023-07-29T00:25:12.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000900405e9b0329142-0064c45c68-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.701 DEBUG:boto:b'hiiqos-58obj000000000000000000481000falseobj000000000000000000482023-07-29T00:24:59.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:12.701 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:12.702 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:12.702 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000048 2023-07-29T00:25:12.703 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000048 2023-07-29T00:25:12.703 DEBUG:boto:Method: GET 2023-07-29T00:25:12.703 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000048 2023-07-29T00:25:12.704 DEBUG:boto:Data: 2023-07-29T00:25:12.704 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.705 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:12.705 DEBUG:boto:Port: 8000 2023-07-29T00:25:12.705 DEBUG:boto:Params: {} 2023-07-29T00:25:12.706 DEBUG:boto:Token: None 2023-07-29T00:25:12.706 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/ 2023-07-29T00:25:12.707 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7kh5dIuFkes7J+Ul3u5NWFfVa5E= 2023-07-29T00:25:12.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7kh5dIuFkes7J+Ul3u5NWFfVa5E=', 'Content-Length': '0'} 2023-07-29T00:25:12.712 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b884d2554e233e2-0064c45c68-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.712 DEBUG:boto:b'hiiqos-58obj000000000000000000481000falseobj000000000000000000482023-07-29T00:24:59.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:12.713 DEBUG:boto:path=/hiiqos-58/obj00000000000000000048 2023-07-29T00:25:12.713 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000048 2023-07-29T00:25:12.714 DEBUG:boto:Method: HEAD 2023-07-29T00:25:12.714 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000048 2023-07-29T00:25:12.715 DEBUG:boto:Data: 2023-07-29T00:25:12.715 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.715 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:12.716 DEBUG:boto:Port: 8001 2023-07-29T00:25:12.716 DEBUG:boto:Params: {} 2023-07-29T00:25:12.717 DEBUG:boto:Token: None 2023-07-29T00:25:12.717 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/obj00000000000000000048 2023-07-29T00:25:12.718 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:whHNX7I0bTEkxwrdckhla/iyTuM= 2023-07-29T00:25:12.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:whHNX7I0bTEkxwrdckhla/iyTuM=', 'Content-Length': '0'} 2023-07-29T00:25:12.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a13f09e65e4c4111-0064c45c68-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.721 DEBUG:boto:path=/hiiqos-58/obj00000000000000000048 2023-07-29T00:25:12.721 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000048 2023-07-29T00:25:12.721 DEBUG:boto:Method: HEAD 2023-07-29T00:25:12.722 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000048 2023-07-29T00:25:12.722 DEBUG:boto:Data: 2023-07-29T00:25:12.722 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.723 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:12.723 DEBUG:boto:Port: 8000 2023-07-29T00:25:12.723 DEBUG:boto:Params: {} 2023-07-29T00:25:12.724 DEBUG:boto:Token: None 2023-07-29T00:25:12.725 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/obj00000000000000000048 2023-07-29T00:25:12.725 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:whHNX7I0bTEkxwrdckhla/iyTuM= 2023-07-29T00:25:12.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:whHNX7I0bTEkxwrdckhla/iyTuM=', 'Content-Length': '0'} 2023-07-29T00:25:12.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015bcef4a7710a2fb-0064c45c68-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000048 2023-07-29T00:25:12.728 DEBUG:boto:path=/hiiqos-58/obj00000000000000000048 2023-07-29T00:25:12.729 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000048 2023-07-29T00:25:12.729 DEBUG:boto:Method: GET 2023-07-29T00:25:12.729 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000048 2023-07-29T00:25:12.730 DEBUG:boto:Data: 2023-07-29T00:25:12.730 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.730 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:12.731 DEBUG:boto:Port: 8001 2023-07-29T00:25:12.731 DEBUG:boto:Params: {} 2023-07-29T00:25:12.732 DEBUG:boto:Token: None 2023-07-29T00:25:12.733 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/obj00000000000000000048 2023-07-29T00:25:12.734 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ylyJyYlIiE6naLBKWSsUzNdXeBQ= 2023-07-29T00:25:12.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ylyJyYlIiE6naLBKWSsUzNdXeBQ=', 'Content-Length': '0'} 2023-07-29T00:25:12.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010a162ea6444e9fe-0064c45c68-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.785 DEBUG:boto:path=/hiiqos-58/obj00000000000000000048 2023-07-29T00:25:12.785 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000048 2023-07-29T00:25:12.786 DEBUG:boto:Method: GET 2023-07-29T00:25:12.786 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000048 2023-07-29T00:25:12.787 DEBUG:boto:Data: 2023-07-29T00:25:12.787 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.787 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:12.788 DEBUG:boto:Port: 8000 2023-07-29T00:25:12.788 DEBUG:boto:Params: {} 2023-07-29T00:25:12.788 DEBUG:boto:Token: None 2023-07-29T00:25:12.789 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/obj00000000000000000048 2023-07-29T00:25:12.789 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ylyJyYlIiE6naLBKWSsUzNdXeBQ= 2023-07-29T00:25:12.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ylyJyYlIiE6naLBKWSsUzNdXeBQ=', 'Content-Length': '0'} 2023-07-29T00:25:12.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac8732bc8e0dde25-0064c45c68-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.793 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:12.793 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:12.794 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000048 2023-07-29T00:25:12.794 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000048 2023-07-29T00:25:12.795 DEBUG:boto:Method: GET 2023-07-29T00:25:12.795 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000048 2023-07-29T00:25:12.796 DEBUG:boto:Data: 2023-07-29T00:25:12.796 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.796 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:12.797 DEBUG:boto:Port: 8001 2023-07-29T00:25:12.797 DEBUG:boto:Params: {} 2023-07-29T00:25:12.797 DEBUG:boto:Token: None 2023-07-29T00:25:12.798 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/ 2023-07-29T00:25:12.798 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7kh5dIuFkes7J+Ul3u5NWFfVa5E= 2023-07-29T00:25:12.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7kh5dIuFkes7J+Ul3u5NWFfVa5E=', 'Content-Length': '0'} 2023-07-29T00:25:12.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053a0fa020c78a6f4-0064c45c68-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.802 DEBUG:boto:b'hiiqos-58obj000000000000000000481000falseobj000000000000000000482023-07-29T00:24:59.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:12.803 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:12.803 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:12.803 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000048 2023-07-29T00:25:12.804 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000048 2023-07-29T00:25:12.804 DEBUG:boto:Method: GET 2023-07-29T00:25:12.804 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000048 2023-07-29T00:25:12.805 DEBUG:boto:Data: 2023-07-29T00:25:12.805 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.806 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:12.806 DEBUG:boto:Port: 8000 2023-07-29T00:25:12.806 DEBUG:boto:Params: {} 2023-07-29T00:25:12.807 DEBUG:boto:Token: None 2023-07-29T00:25:12.807 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/ 2023-07-29T00:25:12.808 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7kh5dIuFkes7J+Ul3u5NWFfVa5E= 2023-07-29T00:25:12.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7kh5dIuFkes7J+Ul3u5NWFfVa5E=', 'Content-Length': '0'} 2023-07-29T00:25:12.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ef1ec091db933c4-0064c45c68-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.813 DEBUG:boto:b'hiiqos-58obj000000000000000000481000falseobj000000000000000000482023-07-29T00:24:59.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:12.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000049 2023-07-29T00:25:12.814 DEBUG:boto:path=/hiiqos-58/obj00000000000000000049 2023-07-29T00:25:12.815 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000049 2023-07-29T00:25:12.815 DEBUG:boto:path=/hiiqos-58/obj00000000000000000049?versionId=null 2023-07-29T00:25:12.816 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000049?versionId=null 2023-07-29T00:25:12.816 DEBUG:boto:Method: GET 2023-07-29T00:25:12.817 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000049?versionId=null 2023-07-29T00:25:12.817 DEBUG:boto:Data: 2023-07-29T00:25:12.817 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.818 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:12.818 DEBUG:boto:Port: 8001 2023-07-29T00:25:12.818 DEBUG:boto:Params: {} 2023-07-29T00:25:12.819 DEBUG:boto:Token: None 2023-07-29T00:25:12.819 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/obj00000000000000000049?versionId=null 2023-07-29T00:25:12.820 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PbVd35cCWV2cV7GXDyYGWQpbE1M= 2023-07-29T00:25:12.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PbVd35cCWV2cV7GXDyYGWQpbE1M=', 'Content-Length': '0'} 2023-07-29T00:25:12.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a7ff06681003d8c-0064c45c68-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.823 DEBUG:boto:path=/hiiqos-58/obj00000000000000000049 2023-07-29T00:25:12.823 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000049 2023-07-29T00:25:12.824 DEBUG:boto:path=/hiiqos-58/obj00000000000000000049?versionId=null 2023-07-29T00:25:12.824 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000049?versionId=null 2023-07-29T00:25:12.825 DEBUG:boto:Method: GET 2023-07-29T00:25:12.825 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000049?versionId=null 2023-07-29T00:25:12.825 DEBUG:boto:Data: 2023-07-29T00:25:12.826 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.826 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:12.826 DEBUG:boto:Port: 8000 2023-07-29T00:25:12.827 DEBUG:boto:Params: {} 2023-07-29T00:25:12.827 DEBUG:boto:Token: None 2023-07-29T00:25:12.828 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/obj00000000000000000049?versionId=null 2023-07-29T00:25:12.828 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PbVd35cCWV2cV7GXDyYGWQpbE1M= 2023-07-29T00:25:12.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PbVd35cCWV2cV7GXDyYGWQpbE1M=', 'Content-Length': '0'} 2023-07-29T00:25:12.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016ad14229f55d261-0064c45c68-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.876 DEBUG:boto:path=/hiiqos-58/obj00000000000000000049 2023-07-29T00:25:12.877 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000049 2023-07-29T00:25:12.877 DEBUG:boto:path=/hiiqos-58/obj00000000000000000049?versionId=null 2023-07-29T00:25:12.878 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000049?versionId=null 2023-07-29T00:25:12.878 DEBUG:boto:Method: HEAD 2023-07-29T00:25:12.878 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000049?versionId=null 2023-07-29T00:25:12.879 DEBUG:boto:Data: 2023-07-29T00:25:12.879 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.879 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:12.880 DEBUG:boto:Port: 8001 2023-07-29T00:25:12.880 DEBUG:boto:Params: {} 2023-07-29T00:25:12.880 DEBUG:boto:Token: None 2023-07-29T00:25:12.881 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/obj00000000000000000049?versionId=null 2023-07-29T00:25:12.881 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XrB5d+/oXSCLf23tKYxSoENif5Y= 2023-07-29T00:25:12.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XrB5d+/oXSCLf23tKYxSoENif5Y=', 'Content-Length': '0'} 2023-07-29T00:25:12.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031559cb991f2f005-0064c45c68-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.884 DEBUG:boto:path=/hiiqos-58/obj00000000000000000049 2023-07-29T00:25:12.885 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000049 2023-07-29T00:25:12.885 DEBUG:boto:path=/hiiqos-58/obj00000000000000000049?versionId=null 2023-07-29T00:25:12.885 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000049?versionId=null 2023-07-29T00:25:12.886 DEBUG:boto:Method: HEAD 2023-07-29T00:25:12.886 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000049?versionId=null 2023-07-29T00:25:12.886 DEBUG:boto:Data: 2023-07-29T00:25:12.887 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.887 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:12.888 DEBUG:boto:Port: 8000 2023-07-29T00:25:12.888 DEBUG:boto:Params: {} 2023-07-29T00:25:12.889 DEBUG:boto:Token: None 2023-07-29T00:25:12.889 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/obj00000000000000000049?versionId=null 2023-07-29T00:25:12.890 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XrB5d+/oXSCLf23tKYxSoENif5Y= 2023-07-29T00:25:12.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XrB5d+/oXSCLf23tKYxSoENif5Y=', 'Content-Length': '0'} 2023-07-29T00:25:12.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000404dce3de7085ed4-0064c45c68-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000049 2023-07-29T00:25:12.893 DEBUG:boto:path=/hiiqos-58/obj00000000000000000049 2023-07-29T00:25:12.893 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000049 2023-07-29T00:25:12.894 DEBUG:boto:path=/hiiqos-58/obj00000000000000000049?versionId=null 2023-07-29T00:25:12.894 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000049?versionId=null 2023-07-29T00:25:12.894 DEBUG:boto:Method: GET 2023-07-29T00:25:12.895 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000049?versionId=null 2023-07-29T00:25:12.895 DEBUG:boto:Data: 2023-07-29T00:25:12.896 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.896 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:12.896 DEBUG:boto:Port: 8001 2023-07-29T00:25:12.897 DEBUG:boto:Params: {} 2023-07-29T00:25:12.897 DEBUG:boto:Token: None 2023-07-29T00:25:12.898 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/obj00000000000000000049?versionId=null 2023-07-29T00:25:12.898 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PbVd35cCWV2cV7GXDyYGWQpbE1M= 2023-07-29T00:25:12.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PbVd35cCWV2cV7GXDyYGWQpbE1M=', 'Content-Length': '0'} 2023-07-29T00:25:12.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000374ad97348e9b1e6-0064c45c68-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.901 DEBUG:boto:path=/hiiqos-58/obj00000000000000000049 2023-07-29T00:25:12.902 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000049 2023-07-29T00:25:12.902 DEBUG:boto:path=/hiiqos-58/obj00000000000000000049?versionId=null 2023-07-29T00:25:12.903 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000049?versionId=null 2023-07-29T00:25:12.903 DEBUG:boto:Method: GET 2023-07-29T00:25:12.903 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000049?versionId=null 2023-07-29T00:25:12.904 DEBUG:boto:Data: 2023-07-29T00:25:12.904 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.905 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:12.905 DEBUG:boto:Port: 8000 2023-07-29T00:25:12.905 DEBUG:boto:Params: {} 2023-07-29T00:25:12.906 DEBUG:boto:Token: None 2023-07-29T00:25:12.906 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/obj00000000000000000049?versionId=null 2023-07-29T00:25:12.907 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PbVd35cCWV2cV7GXDyYGWQpbE1M= 2023-07-29T00:25:12.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PbVd35cCWV2cV7GXDyYGWQpbE1M=', 'Content-Length': '0'} 2023-07-29T00:25:12.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000411b85ea677cbd04-0064c45c68-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.910 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:12.910 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:12.911 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000049 2023-07-29T00:25:12.911 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000049 2023-07-29T00:25:12.911 DEBUG:boto:Method: GET 2023-07-29T00:25:12.912 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000049 2023-07-29T00:25:12.912 DEBUG:boto:Data: 2023-07-29T00:25:12.913 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.913 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:12.913 DEBUG:boto:Port: 8001 2023-07-29T00:25:12.914 DEBUG:boto:Params: {} 2023-07-29T00:25:12.914 DEBUG:boto:Token: None 2023-07-29T00:25:12.915 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/ 2023-07-29T00:25:12.915 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7kh5dIuFkes7J+Ul3u5NWFfVa5E= 2023-07-29T00:25:12.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7kh5dIuFkes7J+Ul3u5NWFfVa5E=', 'Content-Length': '0'} 2023-07-29T00:25:12.918 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097004c5bcd2b8d06-0064c45c68-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.964 DEBUG:boto:b'hiiqos-58obj000000000000000000491000falseobj000000000000000000492023-07-29T00:24:56.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:12.965 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:12.966 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:12.966 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000049 2023-07-29T00:25:12.966 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000049 2023-07-29T00:25:12.967 DEBUG:boto:Method: GET 2023-07-29T00:25:12.967 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000049 2023-07-29T00:25:12.967 DEBUG:boto:Data: 2023-07-29T00:25:12.968 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.968 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:12.968 DEBUG:boto:Port: 8000 2023-07-29T00:25:12.969 DEBUG:boto:Params: {} 2023-07-29T00:25:12.969 DEBUG:boto:Token: None 2023-07-29T00:25:12.970 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/ 2023-07-29T00:25:12.970 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7kh5dIuFkes7J+Ul3u5NWFfVa5E= 2023-07-29T00:25:12.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7kh5dIuFkes7J+Ul3u5NWFfVa5E=', 'Content-Length': '0'} 2023-07-29T00:25:12.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eaaaa3902709ba77-0064c45c68-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.975 DEBUG:boto:b'hiiqos-58obj000000000000000000491000falseobj000000000000000000492023-07-29T00:24:56.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:12.976 DEBUG:boto:path=/hiiqos-58/obj00000000000000000049 2023-07-29T00:25:12.976 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000049 2023-07-29T00:25:12.977 DEBUG:boto:Method: HEAD 2023-07-29T00:25:12.977 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000049 2023-07-29T00:25:12.977 DEBUG:boto:Data: 2023-07-29T00:25:12.978 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.978 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:12.978 DEBUG:boto:Port: 8001 2023-07-29T00:25:12.979 DEBUG:boto:Params: {} 2023-07-29T00:25:12.979 DEBUG:boto:Token: None 2023-07-29T00:25:12.980 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/obj00000000000000000049 2023-07-29T00:25:12.980 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XMOLMccnTvyBsXDOxUBnZ14hVyc= 2023-07-29T00:25:12.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XMOLMccnTvyBsXDOxUBnZ14hVyc=', 'Content-Length': '0'} 2023-07-29T00:25:12.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af8cd203ce2f826a-0064c45c68-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.983 DEBUG:boto:path=/hiiqos-58/obj00000000000000000049 2023-07-29T00:25:12.983 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000049 2023-07-29T00:25:12.984 DEBUG:boto:Method: HEAD 2023-07-29T00:25:12.984 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000049 2023-07-29T00:25:12.984 DEBUG:boto:Data: 2023-07-29T00:25:12.985 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.985 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:12.985 DEBUG:boto:Port: 8000 2023-07-29T00:25:12.986 DEBUG:boto:Params: {} 2023-07-29T00:25:12.986 DEBUG:boto:Token: None 2023-07-29T00:25:12.987 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/obj00000000000000000049 2023-07-29T00:25:12.987 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XMOLMccnTvyBsXDOxUBnZ14hVyc= 2023-07-29T00:25:12.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XMOLMccnTvyBsXDOxUBnZ14hVyc=', 'Content-Length': '0'} 2023-07-29T00:25:12.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e591fadc0f94d535-0064c45c68-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000049 2023-07-29T00:25:12.990 DEBUG:boto:path=/hiiqos-58/obj00000000000000000049 2023-07-29T00:25:12.991 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000049 2023-07-29T00:25:12.991 DEBUG:boto:Method: GET 2023-07-29T00:25:12.992 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000049 2023-07-29T00:25:12.992 DEBUG:boto:Data: 2023-07-29T00:25:12.993 DEBUG:boto:Headers: {} 2023-07-29T00:25:12.993 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:12.993 DEBUG:boto:Port: 8001 2023-07-29T00:25:12.994 DEBUG:boto:Params: {} 2023-07-29T00:25:12.994 DEBUG:boto:Token: None 2023-07-29T00:25:12.995 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:12 GMT /hiiqos-58/obj00000000000000000049 2023-07-29T00:25:12.995 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3nrwtDW7/H1QM8xKSUJNilYn/bM= 2023-07-29T00:25:12.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:12 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3nrwtDW7/H1QM8xKSUJNilYn/bM=', 'Content-Length': '0'} 2023-07-29T00:25:12.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f838b5eec581b4d5-0064c45c68-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:12.998 DEBUG:boto:path=/hiiqos-58/obj00000000000000000049 2023-07-29T00:25:12.999 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000049 2023-07-29T00:25:12.999 DEBUG:boto:Method: GET 2023-07-29T00:25:12.999 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000049 2023-07-29T00:25:13.000 DEBUG:boto:Data: 2023-07-29T00:25:13.000 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.000 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:13.001 DEBUG:boto:Port: 8000 2023-07-29T00:25:13.001 DEBUG:boto:Params: {} 2023-07-29T00:25:13.002 DEBUG:boto:Token: None 2023-07-29T00:25:13.002 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000049 2023-07-29T00:25:13.002 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Gk4gmiM2xosI9he16eu7AszaHik= 2023-07-29T00:25:13.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Gk4gmiM2xosI9he16eu7AszaHik=', 'Content-Length': '0'} 2023-07-29T00:25:13.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095280a7b1af25d17-0064c45c68-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.005 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:13.006 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:13.006 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000049 2023-07-29T00:25:13.007 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000049 2023-07-29T00:25:13.007 DEBUG:boto:Method: GET 2023-07-29T00:25:13.007 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000049 2023-07-29T00:25:13.008 DEBUG:boto:Data: 2023-07-29T00:25:13.008 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.008 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:13.009 DEBUG:boto:Port: 8001 2023-07-29T00:25:13.009 DEBUG:boto:Params: {} 2023-07-29T00:25:13.010 DEBUG:boto:Token: None 2023-07-29T00:25:13.010 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/ 2023-07-29T00:25:13.010 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CHnlLplrkwjE2rhRRoGuf8tEYTM= 2023-07-29T00:25:13.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CHnlLplrkwjE2rhRRoGuf8tEYTM=', 'Content-Length': '0'} 2023-07-29T00:25:13.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000134025df63fd10-0064c45c68-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.014 DEBUG:boto:b'hiiqos-58obj000000000000000000491000falseobj000000000000000000492023-07-29T00:24:56.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:13.014 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:13.015 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:13.015 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000049 2023-07-29T00:25:13.015 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000049 2023-07-29T00:25:13.016 DEBUG:boto:Method: GET 2023-07-29T00:25:13.016 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000049 2023-07-29T00:25:13.017 DEBUG:boto:Data: 2023-07-29T00:25:13.017 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.018 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:13.018 DEBUG:boto:Port: 8000 2023-07-29T00:25:13.018 DEBUG:boto:Params: {} 2023-07-29T00:25:13.019 DEBUG:boto:Token: None 2023-07-29T00:25:13.019 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/ 2023-07-29T00:25:13.020 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CHnlLplrkwjE2rhRRoGuf8tEYTM= 2023-07-29T00:25:13.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CHnlLplrkwjE2rhRRoGuf8tEYTM=', 'Content-Length': '0'} 2023-07-29T00:25:13.024 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006863a070a7e7a556-0064c45c68-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.068 DEBUG:boto:b'hiiqos-58obj000000000000000000491000falseobj000000000000000000492023-07-29T00:24:56.685Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:13.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2023-07-29T00:25:13.070 DEBUG:boto:path=/hiiqos-58/obj00000000000000000050 2023-07-29T00:25:13.071 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000050 2023-07-29T00:25:13.072 DEBUG:boto:path=/hiiqos-58/obj00000000000000000050?versionId=null 2023-07-29T00:25:13.073 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000050?versionId=null 2023-07-29T00:25:13.073 DEBUG:boto:Method: GET 2023-07-29T00:25:13.073 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000050?versionId=null 2023-07-29T00:25:13.074 DEBUG:boto:Data: 2023-07-29T00:25:13.074 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.075 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:13.075 DEBUG:boto:Port: 8001 2023-07-29T00:25:13.075 DEBUG:boto:Params: {} 2023-07-29T00:25:13.076 DEBUG:boto:Token: None 2023-07-29T00:25:13.076 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000050?versionId=null 2023-07-29T00:25:13.077 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4hrV3mYobTRF5p5Id0MPX5E0NUQ= 2023-07-29T00:25:13.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4hrV3mYobTRF5p5Id0MPX5E0NUQ=', 'Content-Length': '0'} 2023-07-29T00:25:13.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058c8fb9a02d9ea2e-0064c45c69-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.080 DEBUG:boto:path=/hiiqos-58/obj00000000000000000050 2023-07-29T00:25:13.081 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000050 2023-07-29T00:25:13.081 DEBUG:boto:path=/hiiqos-58/obj00000000000000000050?versionId=null 2023-07-29T00:25:13.082 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000050?versionId=null 2023-07-29T00:25:13.083 DEBUG:boto:Method: GET 2023-07-29T00:25:13.084 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000050?versionId=null 2023-07-29T00:25:13.085 DEBUG:boto:Data: 2023-07-29T00:25:13.086 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.086 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:13.087 DEBUG:boto:Port: 8000 2023-07-29T00:25:13.087 DEBUG:boto:Params: {} 2023-07-29T00:25:13.088 DEBUG:boto:Token: None 2023-07-29T00:25:13.088 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000050?versionId=null 2023-07-29T00:25:13.089 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4hrV3mYobTRF5p5Id0MPX5E0NUQ= 2023-07-29T00:25:13.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4hrV3mYobTRF5p5Id0MPX5E0NUQ=', 'Content-Length': '0'} 2023-07-29T00:25:13.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa943c1c499fc8e7-0064c45c69-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.092 DEBUG:boto:path=/hiiqos-58/obj00000000000000000050 2023-07-29T00:25:13.092 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000050 2023-07-29T00:25:13.093 DEBUG:boto:path=/hiiqos-58/obj00000000000000000050?versionId=null 2023-07-29T00:25:13.094 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000050?versionId=null 2023-07-29T00:25:13.094 DEBUG:boto:Method: HEAD 2023-07-29T00:25:13.095 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000050?versionId=null 2023-07-29T00:25:13.096 DEBUG:boto:Data: 2023-07-29T00:25:13.097 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.097 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:13.098 DEBUG:boto:Port: 8001 2023-07-29T00:25:13.098 DEBUG:boto:Params: {} 2023-07-29T00:25:13.100 DEBUG:boto:Token: None 2023-07-29T00:25:13.101 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000050?versionId=null 2023-07-29T00:25:13.102 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dKFdAkh7XLQxZOQJ1MGTVCiJNiQ= 2023-07-29T00:25:13.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dKFdAkh7XLQxZOQJ1MGTVCiJNiQ=', 'Content-Length': '0'} 2023-07-29T00:25:13.105 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4bcd3753eccb4fc-0064c45c69-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.106 DEBUG:boto:path=/hiiqos-58/obj00000000000000000050 2023-07-29T00:25:13.107 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000050 2023-07-29T00:25:13.108 DEBUG:boto:path=/hiiqos-58/obj00000000000000000050?versionId=null 2023-07-29T00:25:13.109 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000050?versionId=null 2023-07-29T00:25:13.110 DEBUG:boto:Method: HEAD 2023-07-29T00:25:13.110 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000050?versionId=null 2023-07-29T00:25:13.111 DEBUG:boto:Data: 2023-07-29T00:25:13.112 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.112 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:13.113 DEBUG:boto:Port: 8000 2023-07-29T00:25:13.114 DEBUG:boto:Params: {} 2023-07-29T00:25:13.114 DEBUG:boto:Token: None 2023-07-29T00:25:13.115 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000050?versionId=null 2023-07-29T00:25:13.115 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dKFdAkh7XLQxZOQJ1MGTVCiJNiQ= 2023-07-29T00:25:13.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dKFdAkh7XLQxZOQJ1MGTVCiJNiQ=', 'Content-Length': '0'} 2023-07-29T00:25:13.120 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004adcf470b4f58d67-0064c45c69-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2023-07-29T00:25:13.121 DEBUG:boto:path=/hiiqos-58/obj00000000000000000050 2023-07-29T00:25:13.122 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000050 2023-07-29T00:25:13.122 DEBUG:boto:path=/hiiqos-58/obj00000000000000000050?versionId=null 2023-07-29T00:25:13.123 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000050?versionId=null 2023-07-29T00:25:13.123 DEBUG:boto:Method: GET 2023-07-29T00:25:13.123 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000050?versionId=null 2023-07-29T00:25:13.124 DEBUG:boto:Data: 2023-07-29T00:25:13.124 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.125 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:13.125 DEBUG:boto:Port: 8001 2023-07-29T00:25:13.125 DEBUG:boto:Params: {} 2023-07-29T00:25:13.126 DEBUG:boto:Token: None 2023-07-29T00:25:13.127 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000050?versionId=null 2023-07-29T00:25:13.127 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4hrV3mYobTRF5p5Id0MPX5E0NUQ= 2023-07-29T00:25:13.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4hrV3mYobTRF5p5Id0MPX5E0NUQ=', 'Content-Length': '0'} 2023-07-29T00:25:13.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000953113603d1c5ff8-0064c45c69-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.131 DEBUG:boto:path=/hiiqos-58/obj00000000000000000050 2023-07-29T00:25:13.132 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000050 2023-07-29T00:25:13.132 DEBUG:boto:path=/hiiqos-58/obj00000000000000000050?versionId=null 2023-07-29T00:25:13.133 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000050?versionId=null 2023-07-29T00:25:13.133 DEBUG:boto:Method: GET 2023-07-29T00:25:13.133 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000050?versionId=null 2023-07-29T00:25:13.134 DEBUG:boto:Data: 2023-07-29T00:25:13.134 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.134 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:13.135 DEBUG:boto:Port: 8000 2023-07-29T00:25:13.135 DEBUG:boto:Params: {} 2023-07-29T00:25:13.136 DEBUG:boto:Token: None 2023-07-29T00:25:13.136 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000050?versionId=null 2023-07-29T00:25:13.136 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4hrV3mYobTRF5p5Id0MPX5E0NUQ= 2023-07-29T00:25:13.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4hrV3mYobTRF5p5Id0MPX5E0NUQ=', 'Content-Length': '0'} 2023-07-29T00:25:13.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097fcc89949baf050-0064c45c69-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.141 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:13.142 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:13.143 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000050 2023-07-29T00:25:13.143 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000050 2023-07-29T00:25:13.144 DEBUG:boto:Method: GET 2023-07-29T00:25:13.145 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000050 2023-07-29T00:25:13.146 DEBUG:boto:Data: 2023-07-29T00:25:13.146 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.147 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:13.148 DEBUG:boto:Port: 8001 2023-07-29T00:25:13.149 DEBUG:boto:Params: {} 2023-07-29T00:25:13.150 DEBUG:boto:Token: None 2023-07-29T00:25:13.151 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/ 2023-07-29T00:25:13.151 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CHnlLplrkwjE2rhRRoGuf8tEYTM= 2023-07-29T00:25:13.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CHnlLplrkwjE2rhRRoGuf8tEYTM=', 'Content-Length': '0'} 2023-07-29T00:25:13.156 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cdcc61711300b05-0064c45c69-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.156 DEBUG:boto:b'hiiqos-58obj000000000000000000501000falseobj000000000000000000502023-07-29T00:24:57.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:13.157 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:13.157 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:13.158 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000050 2023-07-29T00:25:13.158 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000050 2023-07-29T00:25:13.158 DEBUG:boto:Method: GET 2023-07-29T00:25:13.159 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000050 2023-07-29T00:25:13.159 DEBUG:boto:Data: 2023-07-29T00:25:13.159 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.160 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:13.160 DEBUG:boto:Port: 8000 2023-07-29T00:25:13.160 DEBUG:boto:Params: {} 2023-07-29T00:25:13.161 DEBUG:boto:Token: None 2023-07-29T00:25:13.161 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/ 2023-07-29T00:25:13.162 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CHnlLplrkwjE2rhRRoGuf8tEYTM= 2023-07-29T00:25:13.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CHnlLplrkwjE2rhRRoGuf8tEYTM=', 'Content-Length': '0'} 2023-07-29T00:25:13.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f77b9553c7369ad-0064c45c69-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.167 DEBUG:boto:b'hiiqos-58obj000000000000000000501000falseobj000000000000000000502023-07-29T00:24:57.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:13.168 DEBUG:boto:path=/hiiqos-58/obj00000000000000000050 2023-07-29T00:25:13.168 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000050 2023-07-29T00:25:13.169 DEBUG:boto:Method: HEAD 2023-07-29T00:25:13.169 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000050 2023-07-29T00:25:13.170 DEBUG:boto:Data: 2023-07-29T00:25:13.170 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.170 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:13.171 DEBUG:boto:Port: 8001 2023-07-29T00:25:13.171 DEBUG:boto:Params: {} 2023-07-29T00:25:13.171 DEBUG:boto:Token: None 2023-07-29T00:25:13.172 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000050 2023-07-29T00:25:13.172 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:x+LtsidEdUyzYOEE6JaL/vxbg2M= 2023-07-29T00:25:13.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:x+LtsidEdUyzYOEE6JaL/vxbg2M=', 'Content-Length': '0'} 2023-07-29T00:25:13.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000eec03c5162736e6-0064c45c69-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.175 DEBUG:boto:path=/hiiqos-58/obj00000000000000000050 2023-07-29T00:25:13.175 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000050 2023-07-29T00:25:13.176 DEBUG:boto:Method: HEAD 2023-07-29T00:25:13.176 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000050 2023-07-29T00:25:13.177 DEBUG:boto:Data: 2023-07-29T00:25:13.177 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.177 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:13.178 DEBUG:boto:Port: 8000 2023-07-29T00:25:13.178 DEBUG:boto:Params: {} 2023-07-29T00:25:13.178 DEBUG:boto:Token: None 2023-07-29T00:25:13.179 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000050 2023-07-29T00:25:13.179 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:x+LtsidEdUyzYOEE6JaL/vxbg2M= 2023-07-29T00:25:13.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:x+LtsidEdUyzYOEE6JaL/vxbg2M=', 'Content-Length': '0'} 2023-07-29T00:25:13.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023721acc22b2ca9f-0064c45c69-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2023-07-29T00:25:13.182 DEBUG:boto:path=/hiiqos-58/obj00000000000000000050 2023-07-29T00:25:13.183 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000050 2023-07-29T00:25:13.183 DEBUG:boto:Method: GET 2023-07-29T00:25:13.184 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000050 2023-07-29T00:25:13.184 DEBUG:boto:Data: 2023-07-29T00:25:13.184 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.185 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:13.185 DEBUG:boto:Port: 8001 2023-07-29T00:25:13.185 DEBUG:boto:Params: {} 2023-07-29T00:25:13.186 DEBUG:boto:Token: None 2023-07-29T00:25:13.186 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000050 2023-07-29T00:25:13.187 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eVXyp9UjzC+WCeIEC1NxwCMG5Wk= 2023-07-29T00:25:13.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eVXyp9UjzC+WCeIEC1NxwCMG5Wk=', 'Content-Length': '0'} 2023-07-29T00:25:13.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b49db140e75e39a-0064c45c69-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.189 DEBUG:boto:path=/hiiqos-58/obj00000000000000000050 2023-07-29T00:25:13.190 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000050 2023-07-29T00:25:13.190 DEBUG:boto:Method: GET 2023-07-29T00:25:13.191 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000050 2023-07-29T00:25:13.191 DEBUG:boto:Data: 2023-07-29T00:25:13.191 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.192 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:13.192 DEBUG:boto:Port: 8000 2023-07-29T00:25:13.192 DEBUG:boto:Params: {} 2023-07-29T00:25:13.193 DEBUG:boto:Token: None 2023-07-29T00:25:13.193 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000050 2023-07-29T00:25:13.194 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eVXyp9UjzC+WCeIEC1NxwCMG5Wk= 2023-07-29T00:25:13.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eVXyp9UjzC+WCeIEC1NxwCMG5Wk=', 'Content-Length': '0'} 2023-07-29T00:25:13.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006df279165c0f3d06-0064c45c69-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.197 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:13.197 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:13.198 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000050 2023-07-29T00:25:13.198 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000050 2023-07-29T00:25:13.198 DEBUG:boto:Method: GET 2023-07-29T00:25:13.199 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000050 2023-07-29T00:25:13.199 DEBUG:boto:Data: 2023-07-29T00:25:13.199 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.200 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:13.200 DEBUG:boto:Port: 8001 2023-07-29T00:25:13.200 DEBUG:boto:Params: {} 2023-07-29T00:25:13.201 DEBUG:boto:Token: None 2023-07-29T00:25:13.201 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/ 2023-07-29T00:25:13.202 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CHnlLplrkwjE2rhRRoGuf8tEYTM= 2023-07-29T00:25:13.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CHnlLplrkwjE2rhRRoGuf8tEYTM=', 'Content-Length': '0'} 2023-07-29T00:25:13.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000455a319db3773dc2-0064c45c69-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.248 DEBUG:boto:b'hiiqos-58obj000000000000000000501000falseobj000000000000000000502023-07-29T00:24:57.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:13.249 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:13.250 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:13.250 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000050 2023-07-29T00:25:13.250 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000050 2023-07-29T00:25:13.251 DEBUG:boto:Method: GET 2023-07-29T00:25:13.251 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000050 2023-07-29T00:25:13.251 DEBUG:boto:Data: 2023-07-29T00:25:13.252 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.252 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:13.252 DEBUG:boto:Port: 8000 2023-07-29T00:25:13.253 DEBUG:boto:Params: {} 2023-07-29T00:25:13.253 DEBUG:boto:Token: None 2023-07-29T00:25:13.254 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/ 2023-07-29T00:25:13.254 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CHnlLplrkwjE2rhRRoGuf8tEYTM= 2023-07-29T00:25:13.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CHnlLplrkwjE2rhRRoGuf8tEYTM=', 'Content-Length': '0'} 2023-07-29T00:25:13.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2ff57392774ca64-0064c45c69-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.260 DEBUG:boto:b'hiiqos-58obj000000000000000000501000falseobj000000000000000000502023-07-29T00:24:57.938Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:13.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2023-07-29T00:25:13.261 DEBUG:boto:path=/hiiqos-58/obj00000000000000000053 2023-07-29T00:25:13.262 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000053 2023-07-29T00:25:13.262 DEBUG:boto:path=/hiiqos-58/obj00000000000000000053?versionId=null 2023-07-29T00:25:13.262 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000053?versionId=null 2023-07-29T00:25:13.263 DEBUG:boto:Method: GET 2023-07-29T00:25:13.263 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000053?versionId=null 2023-07-29T00:25:13.263 DEBUG:boto:Data: 2023-07-29T00:25:13.264 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.264 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:13.264 DEBUG:boto:Port: 8001 2023-07-29T00:25:13.265 DEBUG:boto:Params: {} 2023-07-29T00:25:13.265 DEBUG:boto:Token: None 2023-07-29T00:25:13.266 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000053?versionId=null 2023-07-29T00:25:13.266 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yrBU8mra7xtmb4PjGmc3yDBuEmo= 2023-07-29T00:25:13.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yrBU8mra7xtmb4PjGmc3yDBuEmo=', 'Content-Length': '0'} 2023-07-29T00:25:13.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7e26b4a845df36f-0064c45c69-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.270 DEBUG:boto:path=/hiiqos-58/obj00000000000000000053 2023-07-29T00:25:13.270 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000053 2023-07-29T00:25:13.270 DEBUG:boto:path=/hiiqos-58/obj00000000000000000053?versionId=null 2023-07-29T00:25:13.271 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000053?versionId=null 2023-07-29T00:25:13.271 DEBUG:boto:Method: GET 2023-07-29T00:25:13.271 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000053?versionId=null 2023-07-29T00:25:13.272 DEBUG:boto:Data: 2023-07-29T00:25:13.272 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.273 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:13.273 DEBUG:boto:Port: 8000 2023-07-29T00:25:13.273 DEBUG:boto:Params: {} 2023-07-29T00:25:13.274 DEBUG:boto:Token: None 2023-07-29T00:25:13.274 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000053?versionId=null 2023-07-29T00:25:13.275 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yrBU8mra7xtmb4PjGmc3yDBuEmo= 2023-07-29T00:25:13.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yrBU8mra7xtmb4PjGmc3yDBuEmo=', 'Content-Length': '0'} 2023-07-29T00:25:13.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009db4a7a6880516ef-0064c45c69-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.324 DEBUG:boto:path=/hiiqos-58/obj00000000000000000053 2023-07-29T00:25:13.325 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000053 2023-07-29T00:25:13.325 DEBUG:boto:path=/hiiqos-58/obj00000000000000000053?versionId=null 2023-07-29T00:25:13.326 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000053?versionId=null 2023-07-29T00:25:13.326 DEBUG:boto:Method: HEAD 2023-07-29T00:25:13.326 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000053?versionId=null 2023-07-29T00:25:13.327 DEBUG:boto:Data: 2023-07-29T00:25:13.327 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.327 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:13.328 DEBUG:boto:Port: 8001 2023-07-29T00:25:13.328 DEBUG:boto:Params: {} 2023-07-29T00:25:13.329 DEBUG:boto:Token: None 2023-07-29T00:25:13.329 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000053?versionId=null 2023-07-29T00:25:13.330 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lwEiBe1YAEzyo7FCZk+cL+oEXx4= 2023-07-29T00:25:13.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lwEiBe1YAEzyo7FCZk+cL+oEXx4=', 'Content-Length': '0'} 2023-07-29T00:25:13.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000283d4edd8114d119-0064c45c69-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.334 DEBUG:boto:path=/hiiqos-58/obj00000000000000000053 2023-07-29T00:25:13.335 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000053 2023-07-29T00:25:13.336 DEBUG:boto:path=/hiiqos-58/obj00000000000000000053?versionId=null 2023-07-29T00:25:13.336 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000053?versionId=null 2023-07-29T00:25:13.337 DEBUG:boto:Method: HEAD 2023-07-29T00:25:13.338 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000053?versionId=null 2023-07-29T00:25:13.339 DEBUG:boto:Data: 2023-07-29T00:25:13.339 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.340 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:13.340 DEBUG:boto:Port: 8000 2023-07-29T00:25:13.341 DEBUG:boto:Params: {} 2023-07-29T00:25:13.342 DEBUG:boto:Token: None 2023-07-29T00:25:13.343 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000053?versionId=null 2023-07-29T00:25:13.344 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lwEiBe1YAEzyo7FCZk+cL+oEXx4= 2023-07-29T00:25:13.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lwEiBe1YAEzyo7FCZk+cL+oEXx4=', 'Content-Length': '0'} 2023-07-29T00:25:13.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006aa24fd9e1c958a3-0064c45c69-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2023-07-29T00:25:13.349 DEBUG:boto:path=/hiiqos-58/obj00000000000000000053 2023-07-29T00:25:13.350 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000053 2023-07-29T00:25:13.351 DEBUG:boto:path=/hiiqos-58/obj00000000000000000053?versionId=null 2023-07-29T00:25:13.352 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000053?versionId=null 2023-07-29T00:25:13.353 DEBUG:boto:Method: GET 2023-07-29T00:25:13.354 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000053?versionId=null 2023-07-29T00:25:13.355 DEBUG:boto:Data: 2023-07-29T00:25:13.356 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.357 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:13.357 DEBUG:boto:Port: 8001 2023-07-29T00:25:13.358 DEBUG:boto:Params: {} 2023-07-29T00:25:13.359 DEBUG:boto:Token: None 2023-07-29T00:25:13.360 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000053?versionId=null 2023-07-29T00:25:13.360 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yrBU8mra7xtmb4PjGmc3yDBuEmo= 2023-07-29T00:25:13.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yrBU8mra7xtmb4PjGmc3yDBuEmo=', 'Content-Length': '0'} 2023-07-29T00:25:13.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1aa6cb8f0e99070-0064c45c69-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.364 DEBUG:boto:path=/hiiqos-58/obj00000000000000000053 2023-07-29T00:25:13.365 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000053 2023-07-29T00:25:13.366 DEBUG:boto:path=/hiiqos-58/obj00000000000000000053?versionId=null 2023-07-29T00:25:13.366 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000053?versionId=null 2023-07-29T00:25:13.367 DEBUG:boto:Method: GET 2023-07-29T00:25:13.368 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000053?versionId=null 2023-07-29T00:25:13.368 DEBUG:boto:Data: 2023-07-29T00:25:13.369 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.369 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:13.370 DEBUG:boto:Port: 8000 2023-07-29T00:25:13.371 DEBUG:boto:Params: {} 2023-07-29T00:25:13.371 DEBUG:boto:Token: None 2023-07-29T00:25:13.372 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000053?versionId=null 2023-07-29T00:25:13.373 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yrBU8mra7xtmb4PjGmc3yDBuEmo= 2023-07-29T00:25:13.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yrBU8mra7xtmb4PjGmc3yDBuEmo=', 'Content-Length': '0'} 2023-07-29T00:25:13.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e23b6565ca24f40-0064c45c69-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.377 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:13.378 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:13.378 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000053 2023-07-29T00:25:13.379 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000053 2023-07-29T00:25:13.379 DEBUG:boto:Method: GET 2023-07-29T00:25:13.379 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000053 2023-07-29T00:25:13.380 DEBUG:boto:Data: 2023-07-29T00:25:13.380 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.381 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:13.381 DEBUG:boto:Port: 8001 2023-07-29T00:25:13.382 DEBUG:boto:Params: {} 2023-07-29T00:25:13.382 DEBUG:boto:Token: None 2023-07-29T00:25:13.383 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/ 2023-07-29T00:25:13.383 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CHnlLplrkwjE2rhRRoGuf8tEYTM= 2023-07-29T00:25:13.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CHnlLplrkwjE2rhRRoGuf8tEYTM=', 'Content-Length': '0'} 2023-07-29T00:25:13.386 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057138a42ab67a048-0064c45c69-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.387 DEBUG:boto:b'hiiqos-58obj000000000000000000531000falseobj000000000000000000532023-07-29T00:25:00.300Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:13.387 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:13.388 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:13.388 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000053 2023-07-29T00:25:13.389 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000053 2023-07-29T00:25:13.389 DEBUG:boto:Method: GET 2023-07-29T00:25:13.390 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000053 2023-07-29T00:25:13.390 DEBUG:boto:Data: 2023-07-29T00:25:13.390 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.391 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:13.391 DEBUG:boto:Port: 8000 2023-07-29T00:25:13.392 DEBUG:boto:Params: {} 2023-07-29T00:25:13.392 DEBUG:boto:Token: None 2023-07-29T00:25:13.393 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/ 2023-07-29T00:25:13.393 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CHnlLplrkwjE2rhRRoGuf8tEYTM= 2023-07-29T00:25:13.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CHnlLplrkwjE2rhRRoGuf8tEYTM=', 'Content-Length': '0'} 2023-07-29T00:25:13.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6448c87a3bdb3ff-0064c45c69-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.399 DEBUG:boto:b'hiiqos-58obj000000000000000000531000falseobj000000000000000000532023-07-29T00:25:00.300Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:13.400 DEBUG:boto:path=/hiiqos-58/obj00000000000000000053 2023-07-29T00:25:13.400 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000053 2023-07-29T00:25:13.401 DEBUG:boto:Method: HEAD 2023-07-29T00:25:13.401 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000053 2023-07-29T00:25:13.401 DEBUG:boto:Data: 2023-07-29T00:25:13.402 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.402 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:13.402 DEBUG:boto:Port: 8001 2023-07-29T00:25:13.403 DEBUG:boto:Params: {} 2023-07-29T00:25:13.403 DEBUG:boto:Token: None 2023-07-29T00:25:13.404 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000053 2023-07-29T00:25:13.404 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xz9XPtf+LO0FiqwY1QQGN7jIBLE= 2023-07-29T00:25:13.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xz9XPtf+LO0FiqwY1QQGN7jIBLE=', 'Content-Length': '0'} 2023-07-29T00:25:13.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a06a3be2bb68f62-0064c45c69-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.407 DEBUG:boto:path=/hiiqos-58/obj00000000000000000053 2023-07-29T00:25:13.407 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000053 2023-07-29T00:25:13.408 DEBUG:boto:Method: HEAD 2023-07-29T00:25:13.408 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000053 2023-07-29T00:25:13.408 DEBUG:boto:Data: 2023-07-29T00:25:13.409 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.409 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:13.409 DEBUG:boto:Port: 8000 2023-07-29T00:25:13.410 DEBUG:boto:Params: {} 2023-07-29T00:25:13.410 DEBUG:boto:Token: None 2023-07-29T00:25:13.411 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000053 2023-07-29T00:25:13.411 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xz9XPtf+LO0FiqwY1QQGN7jIBLE= 2023-07-29T00:25:13.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xz9XPtf+LO0FiqwY1QQGN7jIBLE=', 'Content-Length': '0'} 2023-07-29T00:25:13.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c06d0d316d5fca1-0064c45c69-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2023-07-29T00:25:13.414 DEBUG:boto:path=/hiiqos-58/obj00000000000000000053 2023-07-29T00:25:13.415 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000053 2023-07-29T00:25:13.415 DEBUG:boto:Method: GET 2023-07-29T00:25:13.415 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000053 2023-07-29T00:25:13.416 DEBUG:boto:Data: 2023-07-29T00:25:13.416 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.417 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:13.417 DEBUG:boto:Port: 8001 2023-07-29T00:25:13.417 DEBUG:boto:Params: {} 2023-07-29T00:25:13.418 DEBUG:boto:Token: None 2023-07-29T00:25:13.418 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000053 2023-07-29T00:25:13.419 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:r/1xamYkXw1Wp6+nSNHEqhu5KvM= 2023-07-29T00:25:13.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:r/1xamYkXw1Wp6+nSNHEqhu5KvM=', 'Content-Length': '0'} 2023-07-29T00:25:13.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000942304c23d914b37-0064c45c69-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.422 DEBUG:boto:path=/hiiqos-58/obj00000000000000000053 2023-07-29T00:25:13.422 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000053 2023-07-29T00:25:13.423 DEBUG:boto:Method: GET 2023-07-29T00:25:13.424 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000053 2023-07-29T00:25:13.424 DEBUG:boto:Data: 2023-07-29T00:25:13.425 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.425 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:13.425 DEBUG:boto:Port: 8000 2023-07-29T00:25:13.426 DEBUG:boto:Params: {} 2023-07-29T00:25:13.426 DEBUG:boto:Token: None 2023-07-29T00:25:13.427 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000053 2023-07-29T00:25:13.427 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:r/1xamYkXw1Wp6+nSNHEqhu5KvM= 2023-07-29T00:25:13.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:r/1xamYkXw1Wp6+nSNHEqhu5KvM=', 'Content-Length': '0'} 2023-07-29T00:25:13.429 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6d27b464df022b8-0064c45c69-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.472 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:13.473 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:13.473 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000053 2023-07-29T00:25:13.474 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000053 2023-07-29T00:25:13.474 DEBUG:boto:Method: GET 2023-07-29T00:25:13.475 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000053 2023-07-29T00:25:13.475 DEBUG:boto:Data: 2023-07-29T00:25:13.475 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.476 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:13.476 DEBUG:boto:Port: 8001 2023-07-29T00:25:13.476 DEBUG:boto:Params: {} 2023-07-29T00:25:13.477 DEBUG:boto:Token: None 2023-07-29T00:25:13.477 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/ 2023-07-29T00:25:13.478 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CHnlLplrkwjE2rhRRoGuf8tEYTM= 2023-07-29T00:25:13.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CHnlLplrkwjE2rhRRoGuf8tEYTM=', 'Content-Length': '0'} 2023-07-29T00:25:13.480 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000265c326dc47546cb-0064c45c69-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.481 DEBUG:boto:b'hiiqos-58obj000000000000000000531000falseobj000000000000000000532023-07-29T00:25:00.300Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:13.482 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:13.482 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:13.483 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000053 2023-07-29T00:25:13.483 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000053 2023-07-29T00:25:13.483 DEBUG:boto:Method: GET 2023-07-29T00:25:13.484 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000053 2023-07-29T00:25:13.484 DEBUG:boto:Data: 2023-07-29T00:25:13.485 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.485 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:13.485 DEBUG:boto:Port: 8000 2023-07-29T00:25:13.486 DEBUG:boto:Params: {} 2023-07-29T00:25:13.486 DEBUG:boto:Token: None 2023-07-29T00:25:13.487 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/ 2023-07-29T00:25:13.487 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CHnlLplrkwjE2rhRRoGuf8tEYTM= 2023-07-29T00:25:13.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CHnlLplrkwjE2rhRRoGuf8tEYTM=', 'Content-Length': '0'} 2023-07-29T00:25:13.492 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000653f8a51c8276027-0064c45c69-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.492 DEBUG:boto:b'hiiqos-58obj000000000000000000531000falseobj000000000000000000532023-07-29T00:25:00.300Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:13.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2023-07-29T00:25:13.494 DEBUG:boto:path=/hiiqos-58/obj00000000000000000055 2023-07-29T00:25:13.494 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000055 2023-07-29T00:25:13.494 DEBUG:boto:path=/hiiqos-58/obj00000000000000000055?versionId=null 2023-07-29T00:25:13.495 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000055?versionId=null 2023-07-29T00:25:13.495 DEBUG:boto:Method: GET 2023-07-29T00:25:13.495 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000055?versionId=null 2023-07-29T00:25:13.496 DEBUG:boto:Data: 2023-07-29T00:25:13.496 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.497 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:13.497 DEBUG:boto:Port: 8001 2023-07-29T00:25:13.497 DEBUG:boto:Params: {} 2023-07-29T00:25:13.498 DEBUG:boto:Token: None 2023-07-29T00:25:13.498 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000055?versionId=null 2023-07-29T00:25:13.499 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:41gFV7MPw4uLbjOzo5kpzroAJp0= 2023-07-29T00:25:13.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:41gFV7MPw4uLbjOzo5kpzroAJp0=', 'Content-Length': '0'} 2023-07-29T00:25:13.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051a50f63109775f5-0064c45c69-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.548 DEBUG:boto:path=/hiiqos-58/obj00000000000000000055 2023-07-29T00:25:13.549 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000055 2023-07-29T00:25:13.549 DEBUG:boto:path=/hiiqos-58/obj00000000000000000055?versionId=null 2023-07-29T00:25:13.550 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000055?versionId=null 2023-07-29T00:25:13.550 DEBUG:boto:Method: GET 2023-07-29T00:25:13.550 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000055?versionId=null 2023-07-29T00:25:13.551 DEBUG:boto:Data: 2023-07-29T00:25:13.551 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.552 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:13.552 DEBUG:boto:Port: 8000 2023-07-29T00:25:13.552 DEBUG:boto:Params: {} 2023-07-29T00:25:13.553 DEBUG:boto:Token: None 2023-07-29T00:25:13.553 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000055?versionId=null 2023-07-29T00:25:13.554 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:41gFV7MPw4uLbjOzo5kpzroAJp0= 2023-07-29T00:25:13.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:41gFV7MPw4uLbjOzo5kpzroAJp0=', 'Content-Length': '0'} 2023-07-29T00:25:13.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091a698425a146baa-0064c45c69-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.557 DEBUG:boto:path=/hiiqos-58/obj00000000000000000055 2023-07-29T00:25:13.557 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000055 2023-07-29T00:25:13.558 DEBUG:boto:path=/hiiqos-58/obj00000000000000000055?versionId=null 2023-07-29T00:25:13.558 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000055?versionId=null 2023-07-29T00:25:13.558 DEBUG:boto:Method: HEAD 2023-07-29T00:25:13.559 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000055?versionId=null 2023-07-29T00:25:13.559 DEBUG:boto:Data: 2023-07-29T00:25:13.559 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.560 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:13.560 DEBUG:boto:Port: 8001 2023-07-29T00:25:13.560 DEBUG:boto:Params: {} 2023-07-29T00:25:13.561 DEBUG:boto:Token: None 2023-07-29T00:25:13.561 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000055?versionId=null 2023-07-29T00:25:13.561 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JsrS3F+OXtEH4Pd4xsaYHsCNKmg= 2023-07-29T00:25:13.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JsrS3F+OXtEH4Pd4xsaYHsCNKmg=', 'Content-Length': '0'} 2023-07-29T00:25:13.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5fc4b2295641d83-0064c45c69-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.564 DEBUG:boto:path=/hiiqos-58/obj00000000000000000055 2023-07-29T00:25:13.565 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000055 2023-07-29T00:25:13.565 DEBUG:boto:path=/hiiqos-58/obj00000000000000000055?versionId=null 2023-07-29T00:25:13.565 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000055?versionId=null 2023-07-29T00:25:13.566 DEBUG:boto:Method: HEAD 2023-07-29T00:25:13.566 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000055?versionId=null 2023-07-29T00:25:13.566 DEBUG:boto:Data: 2023-07-29T00:25:13.567 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.567 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:13.567 DEBUG:boto:Port: 8000 2023-07-29T00:25:13.568 DEBUG:boto:Params: {} 2023-07-29T00:25:13.568 DEBUG:boto:Token: None 2023-07-29T00:25:13.569 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000055?versionId=null 2023-07-29T00:25:13.569 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JsrS3F+OXtEH4Pd4xsaYHsCNKmg= 2023-07-29T00:25:13.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JsrS3F+OXtEH4Pd4xsaYHsCNKmg=', 'Content-Length': '0'} 2023-07-29T00:25:13.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e170f7adf96baa8-0064c45c69-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2023-07-29T00:25:13.573 DEBUG:boto:path=/hiiqos-58/obj00000000000000000055 2023-07-29T00:25:13.573 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000055 2023-07-29T00:25:13.574 DEBUG:boto:path=/hiiqos-58/obj00000000000000000055?versionId=null 2023-07-29T00:25:13.574 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000055?versionId=null 2023-07-29T00:25:13.574 DEBUG:boto:Method: GET 2023-07-29T00:25:13.575 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000055?versionId=null 2023-07-29T00:25:13.575 DEBUG:boto:Data: 2023-07-29T00:25:13.575 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.576 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:13.576 DEBUG:boto:Port: 8001 2023-07-29T00:25:13.576 DEBUG:boto:Params: {} 2023-07-29T00:25:13.577 DEBUG:boto:Token: None 2023-07-29T00:25:13.577 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000055?versionId=null 2023-07-29T00:25:13.578 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:41gFV7MPw4uLbjOzo5kpzroAJp0= 2023-07-29T00:25:13.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:41gFV7MPw4uLbjOzo5kpzroAJp0=', 'Content-Length': '0'} 2023-07-29T00:25:13.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000096f83fdcac72b1a-0064c45c69-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.581 DEBUG:boto:path=/hiiqos-58/obj00000000000000000055 2023-07-29T00:25:13.581 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000055 2023-07-29T00:25:13.582 DEBUG:boto:path=/hiiqos-58/obj00000000000000000055?versionId=null 2023-07-29T00:25:13.582 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000055?versionId=null 2023-07-29T00:25:13.583 DEBUG:boto:Method: GET 2023-07-29T00:25:13.583 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000055?versionId=null 2023-07-29T00:25:13.583 DEBUG:boto:Data: 2023-07-29T00:25:13.584 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.584 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:13.584 DEBUG:boto:Port: 8000 2023-07-29T00:25:13.585 DEBUG:boto:Params: {} 2023-07-29T00:25:13.585 DEBUG:boto:Token: None 2023-07-29T00:25:13.585 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000055?versionId=null 2023-07-29T00:25:13.586 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:41gFV7MPw4uLbjOzo5kpzroAJp0= 2023-07-29T00:25:13.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:41gFV7MPw4uLbjOzo5kpzroAJp0=', 'Content-Length': '0'} 2023-07-29T00:25:13.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053d9f61ac5f25169-0064c45c69-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.589 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:13.589 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:13.590 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000055 2023-07-29T00:25:13.590 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000055 2023-07-29T00:25:13.590 DEBUG:boto:Method: GET 2023-07-29T00:25:13.591 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000055 2023-07-29T00:25:13.591 DEBUG:boto:Data: 2023-07-29T00:25:13.592 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.592 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:13.592 DEBUG:boto:Port: 8001 2023-07-29T00:25:13.593 DEBUG:boto:Params: {} 2023-07-29T00:25:13.593 DEBUG:boto:Token: None 2023-07-29T00:25:13.593 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/ 2023-07-29T00:25:13.594 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CHnlLplrkwjE2rhRRoGuf8tEYTM= 2023-07-29T00:25:13.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CHnlLplrkwjE2rhRRoGuf8tEYTM=', 'Content-Length': '0'} 2023-07-29T00:25:13.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009463f15f0969c636-0064c45c69-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.597 DEBUG:boto:b'hiiqos-58obj000000000000000000551000falseobj000000000000000000552023-07-29T00:24:55.734Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:13.598 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:13.598 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:13.599 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000055 2023-07-29T00:25:13.599 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000055 2023-07-29T00:25:13.600 DEBUG:boto:Method: GET 2023-07-29T00:25:13.600 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000055 2023-07-29T00:25:13.600 DEBUG:boto:Data: 2023-07-29T00:25:13.601 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.601 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:13.601 DEBUG:boto:Port: 8000 2023-07-29T00:25:13.602 DEBUG:boto:Params: {} 2023-07-29T00:25:13.602 DEBUG:boto:Token: None 2023-07-29T00:25:13.602 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/ 2023-07-29T00:25:13.603 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CHnlLplrkwjE2rhRRoGuf8tEYTM= 2023-07-29T00:25:13.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CHnlLplrkwjE2rhRRoGuf8tEYTM=', 'Content-Length': '0'} 2023-07-29T00:25:13.608 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc1b8bfb989b8b0c-0064c45c69-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.652 DEBUG:boto:b'hiiqos-58obj000000000000000000551000falseobj000000000000000000552023-07-29T00:24:55.734Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:13.653 DEBUG:boto:path=/hiiqos-58/obj00000000000000000055 2023-07-29T00:25:13.654 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000055 2023-07-29T00:25:13.654 DEBUG:boto:Method: HEAD 2023-07-29T00:25:13.654 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000055 2023-07-29T00:25:13.655 DEBUG:boto:Data: 2023-07-29T00:25:13.655 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.655 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:13.656 DEBUG:boto:Port: 8001 2023-07-29T00:25:13.656 DEBUG:boto:Params: {} 2023-07-29T00:25:13.657 DEBUG:boto:Token: None 2023-07-29T00:25:13.657 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000055 2023-07-29T00:25:13.658 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:T5OQmWyoaZ16Q/BCPqFA8Cpnakk= 2023-07-29T00:25:13.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:T5OQmWyoaZ16Q/BCPqFA8Cpnakk=', 'Content-Length': '0'} 2023-07-29T00:25:13.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091ec6e642ad77620-0064c45c69-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.662 DEBUG:boto:path=/hiiqos-58/obj00000000000000000055 2023-07-29T00:25:13.663 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000055 2023-07-29T00:25:13.663 DEBUG:boto:Method: HEAD 2023-07-29T00:25:13.664 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000055 2023-07-29T00:25:13.665 DEBUG:boto:Data: 2023-07-29T00:25:13.666 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.666 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:13.667 DEBUG:boto:Port: 8000 2023-07-29T00:25:13.667 DEBUG:boto:Params: {} 2023-07-29T00:25:13.668 DEBUG:boto:Token: None 2023-07-29T00:25:13.669 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000055 2023-07-29T00:25:13.669 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:T5OQmWyoaZ16Q/BCPqFA8Cpnakk= 2023-07-29T00:25:13.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:T5OQmWyoaZ16Q/BCPqFA8Cpnakk=', 'Content-Length': '0'} 2023-07-29T00:25:13.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c11d2320bb691f77-0064c45c69-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2023-07-29T00:25:13.673 DEBUG:boto:path=/hiiqos-58/obj00000000000000000055 2023-07-29T00:25:13.674 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000055 2023-07-29T00:25:13.674 DEBUG:boto:Method: GET 2023-07-29T00:25:13.674 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000055 2023-07-29T00:25:13.675 DEBUG:boto:Data: 2023-07-29T00:25:13.675 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.676 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:13.676 DEBUG:boto:Port: 8001 2023-07-29T00:25:13.676 DEBUG:boto:Params: {} 2023-07-29T00:25:13.677 DEBUG:boto:Token: None 2023-07-29T00:25:13.677 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000055 2023-07-29T00:25:13.678 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tz76NWQS3Zco2wAwCsvnQodSbOk= 2023-07-29T00:25:13.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tz76NWQS3Zco2wAwCsvnQodSbOk=', 'Content-Length': '0'} 2023-07-29T00:25:13.682 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079f85fc1dacdb96e-0064c45c69-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.683 DEBUG:boto:path=/hiiqos-58/obj00000000000000000055 2023-07-29T00:25:13.683 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000055 2023-07-29T00:25:13.684 DEBUG:boto:Method: GET 2023-07-29T00:25:13.684 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000055 2023-07-29T00:25:13.685 DEBUG:boto:Data: 2023-07-29T00:25:13.685 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.685 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:13.686 DEBUG:boto:Port: 8000 2023-07-29T00:25:13.686 DEBUG:boto:Params: {} 2023-07-29T00:25:13.686 DEBUG:boto:Token: None 2023-07-29T00:25:13.687 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000055 2023-07-29T00:25:13.687 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tz76NWQS3Zco2wAwCsvnQodSbOk= 2023-07-29T00:25:13.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tz76NWQS3Zco2wAwCsvnQodSbOk=', 'Content-Length': '0'} 2023-07-29T00:25:13.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1d2fff57ffbf4c4-0064c45c69-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.690 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:13.691 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:13.691 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000055 2023-07-29T00:25:13.692 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000055 2023-07-29T00:25:13.692 DEBUG:boto:Method: GET 2023-07-29T00:25:13.693 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000055 2023-07-29T00:25:13.693 DEBUG:boto:Data: 2023-07-29T00:25:13.693 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.694 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:13.694 DEBUG:boto:Port: 8001 2023-07-29T00:25:13.694 DEBUG:boto:Params: {} 2023-07-29T00:25:13.695 DEBUG:boto:Token: None 2023-07-29T00:25:13.695 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/ 2023-07-29T00:25:13.696 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CHnlLplrkwjE2rhRRoGuf8tEYTM= 2023-07-29T00:25:13.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CHnlLplrkwjE2rhRRoGuf8tEYTM=', 'Content-Length': '0'} 2023-07-29T00:25:13.699 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b69ff846a6670a2-0064c45c69-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.699 DEBUG:boto:b'hiiqos-58obj000000000000000000551000falseobj000000000000000000552023-07-29T00:24:55.734Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:13.700 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:13.701 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:13.701 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000055 2023-07-29T00:25:13.701 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000055 2023-07-29T00:25:13.702 DEBUG:boto:Method: GET 2023-07-29T00:25:13.702 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000055 2023-07-29T00:25:13.702 DEBUG:boto:Data: 2023-07-29T00:25:13.703 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.704 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:13.704 DEBUG:boto:Port: 8000 2023-07-29T00:25:13.704 DEBUG:boto:Params: {} 2023-07-29T00:25:13.705 DEBUG:boto:Token: None 2023-07-29T00:25:13.705 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/ 2023-07-29T00:25:13.706 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CHnlLplrkwjE2rhRRoGuf8tEYTM= 2023-07-29T00:25:13.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CHnlLplrkwjE2rhRRoGuf8tEYTM=', 'Content-Length': '0'} 2023-07-29T00:25:13.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098a41f821b9309b0-0064c45c69-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.711 DEBUG:boto:b'hiiqos-58obj000000000000000000551000falseobj000000000000000000552023-07-29T00:24:55.734Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:13.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000056 2023-07-29T00:25:13.713 DEBUG:boto:path=/hiiqos-58/obj00000000000000000056 2023-07-29T00:25:13.713 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000056 2023-07-29T00:25:13.713 DEBUG:boto:path=/hiiqos-58/obj00000000000000000056?versionId=null 2023-07-29T00:25:13.714 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000056?versionId=null 2023-07-29T00:25:13.714 DEBUG:boto:Method: GET 2023-07-29T00:25:13.714 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000056?versionId=null 2023-07-29T00:25:13.715 DEBUG:boto:Data: 2023-07-29T00:25:13.715 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.715 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:13.716 DEBUG:boto:Port: 8001 2023-07-29T00:25:13.716 DEBUG:boto:Params: {} 2023-07-29T00:25:13.717 DEBUG:boto:Token: None 2023-07-29T00:25:13.717 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000056?versionId=null 2023-07-29T00:25:13.718 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:akIrW0v/6kYkZhTLmutmid8mkz8= 2023-07-29T00:25:13.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:akIrW0v/6kYkZhTLmutmid8mkz8=', 'Content-Length': '0'} 2023-07-29T00:25:13.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072320ff820a64310-0064c45c69-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.724 DEBUG:boto:path=/hiiqos-58/obj00000000000000000056 2023-07-29T00:25:13.725 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000056 2023-07-29T00:25:13.726 DEBUG:boto:path=/hiiqos-58/obj00000000000000000056?versionId=null 2023-07-29T00:25:13.726 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000056?versionId=null 2023-07-29T00:25:13.727 DEBUG:boto:Method: GET 2023-07-29T00:25:13.728 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000056?versionId=null 2023-07-29T00:25:13.729 DEBUG:boto:Data: 2023-07-29T00:25:13.729 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.730 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:13.730 DEBUG:boto:Port: 8000 2023-07-29T00:25:13.731 DEBUG:boto:Params: {} 2023-07-29T00:25:13.731 DEBUG:boto:Token: None 2023-07-29T00:25:13.732 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000056?versionId=null 2023-07-29T00:25:13.732 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:akIrW0v/6kYkZhTLmutmid8mkz8= 2023-07-29T00:25:13.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:akIrW0v/6kYkZhTLmutmid8mkz8=', 'Content-Length': '0'} 2023-07-29T00:25:13.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f74eb8b345490e32-0064c45c69-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.736 DEBUG:boto:path=/hiiqos-58/obj00000000000000000056 2023-07-29T00:25:13.736 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000056 2023-07-29T00:25:13.737 DEBUG:boto:path=/hiiqos-58/obj00000000000000000056?versionId=null 2023-07-29T00:25:13.737 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000056?versionId=null 2023-07-29T00:25:13.738 DEBUG:boto:Method: HEAD 2023-07-29T00:25:13.738 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000056?versionId=null 2023-07-29T00:25:13.738 DEBUG:boto:Data: 2023-07-29T00:25:13.739 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.739 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:13.739 DEBUG:boto:Port: 8001 2023-07-29T00:25:13.740 DEBUG:boto:Params: {} 2023-07-29T00:25:13.740 DEBUG:boto:Token: None 2023-07-29T00:25:13.741 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000056?versionId=null 2023-07-29T00:25:13.741 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sCpTBreVfTAMJhATqVv7si55IX4= 2023-07-29T00:25:13.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sCpTBreVfTAMJhATqVv7si55IX4=', 'Content-Length': '0'} 2023-07-29T00:25:13.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cfdc173bb5ab41b-0064c45c69-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.744 DEBUG:boto:path=/hiiqos-58/obj00000000000000000056 2023-07-29T00:25:13.744 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000056 2023-07-29T00:25:13.745 DEBUG:boto:path=/hiiqos-58/obj00000000000000000056?versionId=null 2023-07-29T00:25:13.745 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000056?versionId=null 2023-07-29T00:25:13.746 DEBUG:boto:Method: HEAD 2023-07-29T00:25:13.747 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000056?versionId=null 2023-07-29T00:25:13.748 DEBUG:boto:Data: 2023-07-29T00:25:13.748 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.748 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:13.749 DEBUG:boto:Port: 8000 2023-07-29T00:25:13.749 DEBUG:boto:Params: {} 2023-07-29T00:25:13.749 DEBUG:boto:Token: None 2023-07-29T00:25:13.750 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000056?versionId=null 2023-07-29T00:25:13.750 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sCpTBreVfTAMJhATqVv7si55IX4= 2023-07-29T00:25:13.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sCpTBreVfTAMJhATqVv7si55IX4=', 'Content-Length': '0'} 2023-07-29T00:25:13.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b1b2dc8fe4add9e-0064c45c69-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000056 2023-07-29T00:25:13.754 DEBUG:boto:path=/hiiqos-58/obj00000000000000000056 2023-07-29T00:25:13.754 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000056 2023-07-29T00:25:13.755 DEBUG:boto:path=/hiiqos-58/obj00000000000000000056?versionId=null 2023-07-29T00:25:13.755 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000056?versionId=null 2023-07-29T00:25:13.755 DEBUG:boto:Method: GET 2023-07-29T00:25:13.756 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000056?versionId=null 2023-07-29T00:25:13.756 DEBUG:boto:Data: 2023-07-29T00:25:13.757 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.757 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:13.757 DEBUG:boto:Port: 8001 2023-07-29T00:25:13.758 DEBUG:boto:Params: {} 2023-07-29T00:25:13.758 DEBUG:boto:Token: None 2023-07-29T00:25:13.758 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000056?versionId=null 2023-07-29T00:25:13.759 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:akIrW0v/6kYkZhTLmutmid8mkz8= 2023-07-29T00:25:13.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:akIrW0v/6kYkZhTLmutmid8mkz8=', 'Content-Length': '0'} 2023-07-29T00:25:13.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000419264025d7f386e-0064c45c69-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.762 DEBUG:boto:path=/hiiqos-58/obj00000000000000000056 2023-07-29T00:25:13.762 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000056 2023-07-29T00:25:13.763 DEBUG:boto:path=/hiiqos-58/obj00000000000000000056?versionId=null 2023-07-29T00:25:13.763 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000056?versionId=null 2023-07-29T00:25:13.763 DEBUG:boto:Method: GET 2023-07-29T00:25:13.764 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000056?versionId=null 2023-07-29T00:25:13.764 DEBUG:boto:Data: 2023-07-29T00:25:13.764 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.765 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:13.765 DEBUG:boto:Port: 8000 2023-07-29T00:25:13.766 DEBUG:boto:Params: {} 2023-07-29T00:25:13.766 DEBUG:boto:Token: None 2023-07-29T00:25:13.767 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000056?versionId=null 2023-07-29T00:25:13.767 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:akIrW0v/6kYkZhTLmutmid8mkz8= 2023-07-29T00:25:13.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:akIrW0v/6kYkZhTLmutmid8mkz8=', 'Content-Length': '0'} 2023-07-29T00:25:13.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1c5c948392f5e0f-0064c45c69-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.812 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:13.813 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:13.813 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000056 2023-07-29T00:25:13.814 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000056 2023-07-29T00:25:13.814 DEBUG:boto:Method: GET 2023-07-29T00:25:13.814 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000056 2023-07-29T00:25:13.815 DEBUG:boto:Data: 2023-07-29T00:25:13.815 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.815 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:13.816 DEBUG:boto:Port: 8001 2023-07-29T00:25:13.816 DEBUG:boto:Params: {} 2023-07-29T00:25:13.817 DEBUG:boto:Token: None 2023-07-29T00:25:13.817 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/ 2023-07-29T00:25:13.818 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CHnlLplrkwjE2rhRRoGuf8tEYTM= 2023-07-29T00:25:13.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CHnlLplrkwjE2rhRRoGuf8tEYTM=', 'Content-Length': '0'} 2023-07-29T00:25:13.821 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8945aecc56c5a07-0064c45c69-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.821 DEBUG:boto:b'hiiqos-58obj000000000000000000561000falseobj000000000000000000562023-07-29T00:24:57.604Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:13.822 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:13.823 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:13.823 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000056 2023-07-29T00:25:13.823 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000056 2023-07-29T00:25:13.824 DEBUG:boto:Method: GET 2023-07-29T00:25:13.824 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000056 2023-07-29T00:25:13.824 DEBUG:boto:Data: 2023-07-29T00:25:13.825 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.825 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:13.825 DEBUG:boto:Port: 8000 2023-07-29T00:25:13.826 DEBUG:boto:Params: {} 2023-07-29T00:25:13.826 DEBUG:boto:Token: None 2023-07-29T00:25:13.827 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/ 2023-07-29T00:25:13.827 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CHnlLplrkwjE2rhRRoGuf8tEYTM= 2023-07-29T00:25:13.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CHnlLplrkwjE2rhRRoGuf8tEYTM=', 'Content-Length': '0'} 2023-07-29T00:25:13.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d99ca9f69245b474-0064c45c69-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.833 DEBUG:boto:b'hiiqos-58obj000000000000000000561000falseobj000000000000000000562023-07-29T00:24:57.604Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:13.833 DEBUG:boto:path=/hiiqos-58/obj00000000000000000056 2023-07-29T00:25:13.834 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000056 2023-07-29T00:25:13.834 DEBUG:boto:Method: HEAD 2023-07-29T00:25:13.834 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000056 2023-07-29T00:25:13.835 DEBUG:boto:Data: 2023-07-29T00:25:13.835 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.835 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:13.836 DEBUG:boto:Port: 8001 2023-07-29T00:25:13.836 DEBUG:boto:Params: {} 2023-07-29T00:25:13.837 DEBUG:boto:Token: None 2023-07-29T00:25:13.837 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000056 2023-07-29T00:25:13.837 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:t1S5epBkmt23n1tQfrJ4tMuWfcE= 2023-07-29T00:25:13.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t1S5epBkmt23n1tQfrJ4tMuWfcE=', 'Content-Length': '0'} 2023-07-29T00:25:13.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023913c3119c920f1-0064c45c69-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.841 DEBUG:boto:path=/hiiqos-58/obj00000000000000000056 2023-07-29T00:25:13.841 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000056 2023-07-29T00:25:13.841 DEBUG:boto:Method: HEAD 2023-07-29T00:25:13.842 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000056 2023-07-29T00:25:13.842 DEBUG:boto:Data: 2023-07-29T00:25:13.842 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.843 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:13.843 DEBUG:boto:Port: 8000 2023-07-29T00:25:13.843 DEBUG:boto:Params: {} 2023-07-29T00:25:13.844 DEBUG:boto:Token: None 2023-07-29T00:25:13.844 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000056 2023-07-29T00:25:13.845 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:t1S5epBkmt23n1tQfrJ4tMuWfcE= 2023-07-29T00:25:13.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t1S5epBkmt23n1tQfrJ4tMuWfcE=', 'Content-Length': '0'} 2023-07-29T00:25:13.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae8c4e7c40bae412-0064c45c69-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000056 2023-07-29T00:25:13.848 DEBUG:boto:path=/hiiqos-58/obj00000000000000000056 2023-07-29T00:25:13.848 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000056 2023-07-29T00:25:13.849 DEBUG:boto:Method: GET 2023-07-29T00:25:13.849 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000056 2023-07-29T00:25:13.850 DEBUG:boto:Data: 2023-07-29T00:25:13.850 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.850 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:13.851 DEBUG:boto:Port: 8001 2023-07-29T00:25:13.851 DEBUG:boto:Params: {} 2023-07-29T00:25:13.852 DEBUG:boto:Token: None 2023-07-29T00:25:13.852 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000056 2023-07-29T00:25:13.853 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LblBBTBBOPKGIFaBcPdu1rsIe7A= 2023-07-29T00:25:13.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LblBBTBBOPKGIFaBcPdu1rsIe7A=', 'Content-Length': '0'} 2023-07-29T00:25:13.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f29ddda96a139a7e-0064c45c69-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.900 DEBUG:boto:path=/hiiqos-58/obj00000000000000000056 2023-07-29T00:25:13.901 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000056 2023-07-29T00:25:13.901 DEBUG:boto:Method: GET 2023-07-29T00:25:13.902 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000056 2023-07-29T00:25:13.902 DEBUG:boto:Data: 2023-07-29T00:25:13.902 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.903 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:13.903 DEBUG:boto:Port: 8000 2023-07-29T00:25:13.904 DEBUG:boto:Params: {} 2023-07-29T00:25:13.904 DEBUG:boto:Token: None 2023-07-29T00:25:13.905 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000056 2023-07-29T00:25:13.905 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LblBBTBBOPKGIFaBcPdu1rsIe7A= 2023-07-29T00:25:13.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LblBBTBBOPKGIFaBcPdu1rsIe7A=', 'Content-Length': '0'} 2023-07-29T00:25:13.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2754020f5b8e550-0064c45c69-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.909 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:13.910 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:13.910 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000056 2023-07-29T00:25:13.910 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000056 2023-07-29T00:25:13.911 DEBUG:boto:Method: GET 2023-07-29T00:25:13.911 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000056 2023-07-29T00:25:13.911 DEBUG:boto:Data: 2023-07-29T00:25:13.912 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.912 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:13.913 DEBUG:boto:Port: 8001 2023-07-29T00:25:13.913 DEBUG:boto:Params: {} 2023-07-29T00:25:13.913 DEBUG:boto:Token: None 2023-07-29T00:25:13.914 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/ 2023-07-29T00:25:13.914 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CHnlLplrkwjE2rhRRoGuf8tEYTM= 2023-07-29T00:25:13.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CHnlLplrkwjE2rhRRoGuf8tEYTM=', 'Content-Length': '0'} 2023-07-29T00:25:13.917 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085612bb847781f8a-0064c45c69-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.918 DEBUG:boto:b'hiiqos-58obj000000000000000000561000falseobj000000000000000000562023-07-29T00:24:57.604Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:13.918 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:13.919 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:13.919 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000056 2023-07-29T00:25:13.920 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000056 2023-07-29T00:25:13.920 DEBUG:boto:Method: GET 2023-07-29T00:25:13.920 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000056 2023-07-29T00:25:13.921 DEBUG:boto:Data: 2023-07-29T00:25:13.921 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.921 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:13.922 DEBUG:boto:Port: 8000 2023-07-29T00:25:13.922 DEBUG:boto:Params: {} 2023-07-29T00:25:13.923 DEBUG:boto:Token: None 2023-07-29T00:25:13.923 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/ 2023-07-29T00:25:13.923 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CHnlLplrkwjE2rhRRoGuf8tEYTM= 2023-07-29T00:25:13.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CHnlLplrkwjE2rhRRoGuf8tEYTM=', 'Content-Length': '0'} 2023-07-29T00:25:13.928 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000829ca1c96812f09d-0064c45c69-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.968 DEBUG:boto:b'hiiqos-58obj000000000000000000561000falseobj000000000000000000562023-07-29T00:24:57.604Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:13.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2023-07-29T00:25:13.970 DEBUG:boto:path=/hiiqos-58/obj00000000000000000057 2023-07-29T00:25:13.970 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000057 2023-07-29T00:25:13.971 DEBUG:boto:path=/hiiqos-58/obj00000000000000000057?versionId=null 2023-07-29T00:25:13.971 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000057?versionId=null 2023-07-29T00:25:13.971 DEBUG:boto:Method: GET 2023-07-29T00:25:13.972 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000057?versionId=null 2023-07-29T00:25:13.972 DEBUG:boto:Data: 2023-07-29T00:25:13.973 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.973 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:13.974 DEBUG:boto:Port: 8001 2023-07-29T00:25:13.974 DEBUG:boto:Params: {} 2023-07-29T00:25:13.975 DEBUG:boto:Token: None 2023-07-29T00:25:13.975 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000057?versionId=null 2023-07-29T00:25:13.976 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:X+A5T7siZAhs4mupx8pnr1UmIJU= 2023-07-29T00:25:13.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:X+A5T7siZAhs4mupx8pnr1UmIJU=', 'Content-Length': '0'} 2023-07-29T00:25:13.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e83a1c7b5d62ece4-0064c45c69-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.979 DEBUG:boto:path=/hiiqos-58/obj00000000000000000057 2023-07-29T00:25:13.979 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000057 2023-07-29T00:25:13.980 DEBUG:boto:path=/hiiqos-58/obj00000000000000000057?versionId=null 2023-07-29T00:25:13.980 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000057?versionId=null 2023-07-29T00:25:13.981 DEBUG:boto:Method: GET 2023-07-29T00:25:13.981 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000057?versionId=null 2023-07-29T00:25:13.981 DEBUG:boto:Data: 2023-07-29T00:25:13.982 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.982 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:13.982 DEBUG:boto:Port: 8000 2023-07-29T00:25:13.983 DEBUG:boto:Params: {} 2023-07-29T00:25:13.984 DEBUG:boto:Token: None 2023-07-29T00:25:13.984 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000057?versionId=null 2023-07-29T00:25:13.984 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:X+A5T7siZAhs4mupx8pnr1UmIJU= 2023-07-29T00:25:13.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:X+A5T7siZAhs4mupx8pnr1UmIJU=', 'Content-Length': '0'} 2023-07-29T00:25:13.987 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000317277c65e61e8e5-0064c45c69-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.988 DEBUG:boto:path=/hiiqos-58/obj00000000000000000057 2023-07-29T00:25:13.988 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000057 2023-07-29T00:25:13.988 DEBUG:boto:path=/hiiqos-58/obj00000000000000000057?versionId=null 2023-07-29T00:25:13.989 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000057?versionId=null 2023-07-29T00:25:13.989 DEBUG:boto:Method: HEAD 2023-07-29T00:25:13.990 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000057?versionId=null 2023-07-29T00:25:13.990 DEBUG:boto:Data: 2023-07-29T00:25:13.991 DEBUG:boto:Headers: {} 2023-07-29T00:25:13.991 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:13.991 DEBUG:boto:Port: 8001 2023-07-29T00:25:13.992 DEBUG:boto:Params: {} 2023-07-29T00:25:13.992 DEBUG:boto:Token: None 2023-07-29T00:25:13.992 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:13 GMT /hiiqos-58/obj00000000000000000057?versionId=null 2023-07-29T00:25:13.993 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Jsn9TKCSbD651wiATZCy06zubq8= 2023-07-29T00:25:13.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:13 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Jsn9TKCSbD651wiATZCy06zubq8=', 'Content-Length': '0'} 2023-07-29T00:25:13.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5c32e064afc1d8b-0064c45c69-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:13.996 DEBUG:boto:path=/hiiqos-58/obj00000000000000000057 2023-07-29T00:25:13.998 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000057 2023-07-29T00:25:13.998 DEBUG:boto:path=/hiiqos-58/obj00000000000000000057?versionId=null 2023-07-29T00:25:13.998 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000057?versionId=null 2023-07-29T00:25:13.999 DEBUG:boto:Method: HEAD 2023-07-29T00:25:13.999 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000057?versionId=null 2023-07-29T00:25:13.999 DEBUG:boto:Data: 2023-07-29T00:25:14.000 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.000 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:14.000 DEBUG:boto:Port: 8000 2023-07-29T00:25:14.001 DEBUG:boto:Params: {} 2023-07-29T00:25:14.001 DEBUG:boto:Token: None 2023-07-29T00:25:14.002 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000057?versionId=null 2023-07-29T00:25:14.002 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Mxs3oazVtrkGgqGEn+ZXxlHp+PE= 2023-07-29T00:25:14.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Mxs3oazVtrkGgqGEn+ZXxlHp+PE=', 'Content-Length': '0'} 2023-07-29T00:25:14.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098893e3ed288a337-0064c45c69-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2023-07-29T00:25:14.005 DEBUG:boto:path=/hiiqos-58/obj00000000000000000057 2023-07-29T00:25:14.006 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000057 2023-07-29T00:25:14.006 DEBUG:boto:path=/hiiqos-58/obj00000000000000000057?versionId=null 2023-07-29T00:25:14.007 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000057?versionId=null 2023-07-29T00:25:14.007 DEBUG:boto:Method: GET 2023-07-29T00:25:14.007 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000057?versionId=null 2023-07-29T00:25:14.008 DEBUG:boto:Data: 2023-07-29T00:25:14.008 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.008 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:14.009 DEBUG:boto:Port: 8001 2023-07-29T00:25:14.009 DEBUG:boto:Params: {} 2023-07-29T00:25:14.010 DEBUG:boto:Token: None 2023-07-29T00:25:14.010 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000057?versionId=null 2023-07-29T00:25:14.010 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:C8jgzW6OHSzTeVuTF6VRXZ1BWWw= 2023-07-29T00:25:14.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:C8jgzW6OHSzTeVuTF6VRXZ1BWWw=', 'Content-Length': '0'} 2023-07-29T00:25:14.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee5655c212dad6af-0064c45c69-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.016 DEBUG:boto:path=/hiiqos-58/obj00000000000000000057 2023-07-29T00:25:14.016 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000057 2023-07-29T00:25:14.016 DEBUG:boto:path=/hiiqos-58/obj00000000000000000057?versionId=null 2023-07-29T00:25:14.017 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000057?versionId=null 2023-07-29T00:25:14.017 DEBUG:boto:Method: GET 2023-07-29T00:25:14.017 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000057?versionId=null 2023-07-29T00:25:14.018 DEBUG:boto:Data: 2023-07-29T00:25:14.018 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.019 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:14.019 DEBUG:boto:Port: 8000 2023-07-29T00:25:14.019 DEBUG:boto:Params: {} 2023-07-29T00:25:14.020 DEBUG:boto:Token: None 2023-07-29T00:25:14.020 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000057?versionId=null 2023-07-29T00:25:14.021 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:C8jgzW6OHSzTeVuTF6VRXZ1BWWw= 2023-07-29T00:25:14.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:C8jgzW6OHSzTeVuTF6VRXZ1BWWw=', 'Content-Length': '0'} 2023-07-29T00:25:14.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8711f3413ae6197-0064c45c69-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.023 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:14.024 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:14.024 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000057 2023-07-29T00:25:14.025 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000057 2023-07-29T00:25:14.025 DEBUG:boto:Method: GET 2023-07-29T00:25:14.025 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000057 2023-07-29T00:25:14.026 DEBUG:boto:Data: 2023-07-29T00:25:14.026 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.026 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:14.027 DEBUG:boto:Port: 8001 2023-07-29T00:25:14.027 DEBUG:boto:Params: {} 2023-07-29T00:25:14.028 DEBUG:boto:Token: None 2023-07-29T00:25:14.028 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/ 2023-07-29T00:25:14.028 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S5R3ASd1HfDELycrH0uGJ4cyYaU= 2023-07-29T00:25:14.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S5R3ASd1HfDELycrH0uGJ4cyYaU=', 'Content-Length': '0'} 2023-07-29T00:25:14.031 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000222bfb2ae23e0021-0064c45c69-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.032 DEBUG:boto:b'hiiqos-58obj000000000000000000571000falseobj000000000000000000572023-07-29T00:25:00.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:14.032 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:14.033 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:14.033 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000057 2023-07-29T00:25:14.034 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000057 2023-07-29T00:25:14.034 DEBUG:boto:Method: GET 2023-07-29T00:25:14.034 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000057 2023-07-29T00:25:14.035 DEBUG:boto:Data: 2023-07-29T00:25:14.035 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.035 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:14.036 DEBUG:boto:Port: 8000 2023-07-29T00:25:14.036 DEBUG:boto:Params: {} 2023-07-29T00:25:14.037 DEBUG:boto:Token: None 2023-07-29T00:25:14.037 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/ 2023-07-29T00:25:14.037 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S5R3ASd1HfDELycrH0uGJ4cyYaU= 2023-07-29T00:25:14.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S5R3ASd1HfDELycrH0uGJ4cyYaU=', 'Content-Length': '0'} 2023-07-29T00:25:14.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddb42129430d4665-0064c45c69-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.043 DEBUG:boto:b'hiiqos-58obj000000000000000000571000falseobj000000000000000000572023-07-29T00:25:00.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:14.043 DEBUG:boto:path=/hiiqos-58/obj00000000000000000057 2023-07-29T00:25:14.044 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000057 2023-07-29T00:25:14.044 DEBUG:boto:Method: HEAD 2023-07-29T00:25:14.044 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000057 2023-07-29T00:25:14.045 DEBUG:boto:Data: 2023-07-29T00:25:14.045 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.045 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:14.046 DEBUG:boto:Port: 8001 2023-07-29T00:25:14.046 DEBUG:boto:Params: {} 2023-07-29T00:25:14.046 DEBUG:boto:Token: None 2023-07-29T00:25:14.047 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000057 2023-07-29T00:25:14.047 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vw5kMSd5Q492qN52hGqJsXUMC1c= 2023-07-29T00:25:14.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vw5kMSd5Q492qN52hGqJsXUMC1c=', 'Content-Length': '0'} 2023-07-29T00:25:14.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5507d519967b53a-0064c45c69-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.050 DEBUG:boto:path=/hiiqos-58/obj00000000000000000057 2023-07-29T00:25:14.051 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000057 2023-07-29T00:25:14.051 DEBUG:boto:Method: HEAD 2023-07-29T00:25:14.051 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000057 2023-07-29T00:25:14.052 DEBUG:boto:Data: 2023-07-29T00:25:14.052 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.052 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:14.053 DEBUG:boto:Port: 8000 2023-07-29T00:25:14.053 DEBUG:boto:Params: {} 2023-07-29T00:25:14.053 DEBUG:boto:Token: None 2023-07-29T00:25:14.054 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000057 2023-07-29T00:25:14.054 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vw5kMSd5Q492qN52hGqJsXUMC1c= 2023-07-29T00:25:14.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vw5kMSd5Q492qN52hGqJsXUMC1c=', 'Content-Length': '0'} 2023-07-29T00:25:14.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007846434038bb35c2-0064c45c6a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2023-07-29T00:25:14.058 DEBUG:boto:path=/hiiqos-58/obj00000000000000000057 2023-07-29T00:25:14.058 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000057 2023-07-29T00:25:14.058 DEBUG:boto:Method: GET 2023-07-29T00:25:14.059 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000057 2023-07-29T00:25:14.059 DEBUG:boto:Data: 2023-07-29T00:25:14.059 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.060 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:14.060 DEBUG:boto:Port: 8001 2023-07-29T00:25:14.061 DEBUG:boto:Params: {} 2023-07-29T00:25:14.061 DEBUG:boto:Token: None 2023-07-29T00:25:14.061 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000057 2023-07-29T00:25:14.062 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:N2Ujrjds7V52ImRyPbmlzrV+KOI= 2023-07-29T00:25:14.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:N2Ujrjds7V52ImRyPbmlzrV+KOI=', 'Content-Length': '0'} 2023-07-29T00:25:14.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c2da93ce2ce775e-0064c45c6a-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.065 DEBUG:boto:path=/hiiqos-58/obj00000000000000000057 2023-07-29T00:25:14.065 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000057 2023-07-29T00:25:14.065 DEBUG:boto:Method: GET 2023-07-29T00:25:14.066 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000057 2023-07-29T00:25:14.066 DEBUG:boto:Data: 2023-07-29T00:25:14.066 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.067 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:14.067 DEBUG:boto:Port: 8000 2023-07-29T00:25:14.067 DEBUG:boto:Params: {} 2023-07-29T00:25:14.068 DEBUG:boto:Token: None 2023-07-29T00:25:14.068 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000057 2023-07-29T00:25:14.069 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:N2Ujrjds7V52ImRyPbmlzrV+KOI= 2023-07-29T00:25:14.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:N2Ujrjds7V52ImRyPbmlzrV+KOI=', 'Content-Length': '0'} 2023-07-29T00:25:14.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6dfdd884b1559ab-0064c45c6a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.112 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:14.113 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:14.113 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000057 2023-07-29T00:25:14.114 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000057 2023-07-29T00:25:14.114 DEBUG:boto:Method: GET 2023-07-29T00:25:14.114 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000057 2023-07-29T00:25:14.115 DEBUG:boto:Data: 2023-07-29T00:25:14.115 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.115 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:14.116 DEBUG:boto:Port: 8001 2023-07-29T00:25:14.116 DEBUG:boto:Params: {} 2023-07-29T00:25:14.117 DEBUG:boto:Token: None 2023-07-29T00:25:14.117 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/ 2023-07-29T00:25:14.118 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S5R3ASd1HfDELycrH0uGJ4cyYaU= 2023-07-29T00:25:14.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S5R3ASd1HfDELycrH0uGJ4cyYaU=', 'Content-Length': '0'} 2023-07-29T00:25:14.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000380c32df718fcfe4-0064c45c6a-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.123 DEBUG:boto:b'hiiqos-58obj000000000000000000571000falseobj000000000000000000572023-07-29T00:25:00.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:14.124 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:14.125 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:14.125 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000057 2023-07-29T00:25:14.126 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000057 2023-07-29T00:25:14.126 DEBUG:boto:Method: GET 2023-07-29T00:25:14.127 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000057 2023-07-29T00:25:14.127 DEBUG:boto:Data: 2023-07-29T00:25:14.128 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.128 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:14.129 DEBUG:boto:Port: 8000 2023-07-29T00:25:14.129 DEBUG:boto:Params: {} 2023-07-29T00:25:14.130 DEBUG:boto:Token: None 2023-07-29T00:25:14.130 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/ 2023-07-29T00:25:14.131 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S5R3ASd1HfDELycrH0uGJ4cyYaU= 2023-07-29T00:25:14.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S5R3ASd1HfDELycrH0uGJ4cyYaU=', 'Content-Length': '0'} 2023-07-29T00:25:14.135 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c681ca4f4c0e43f0-0064c45c6a-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.136 DEBUG:boto:b'hiiqos-58obj000000000000000000571000falseobj000000000000000000572023-07-29T00:25:00.207Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:14.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2023-07-29T00:25:14.138 DEBUG:boto:path=/hiiqos-58/obj00000000000000000058 2023-07-29T00:25:14.139 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000058 2023-07-29T00:25:14.140 DEBUG:boto:path=/hiiqos-58/obj00000000000000000058?versionId=null 2023-07-29T00:25:14.141 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000058?versionId=null 2023-07-29T00:25:14.141 DEBUG:boto:Method: GET 2023-07-29T00:25:14.141 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000058?versionId=null 2023-07-29T00:25:14.142 DEBUG:boto:Data: 2023-07-29T00:25:14.142 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.143 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:14.143 DEBUG:boto:Port: 8001 2023-07-29T00:25:14.144 DEBUG:boto:Params: {} 2023-07-29T00:25:14.145 DEBUG:boto:Token: None 2023-07-29T00:25:14.145 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000058?versionId=null 2023-07-29T00:25:14.146 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HpgsJzNSLah8r8Y8s5ilHYZ2ysw= 2023-07-29T00:25:14.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HpgsJzNSLah8r8Y8s5ilHYZ2ysw=', 'Content-Length': '0'} 2023-07-29T00:25:14.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033431d01293e9d2a-0064c45c6a-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.150 DEBUG:boto:path=/hiiqos-58/obj00000000000000000058 2023-07-29T00:25:14.151 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000058 2023-07-29T00:25:14.151 DEBUG:boto:path=/hiiqos-58/obj00000000000000000058?versionId=null 2023-07-29T00:25:14.152 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000058?versionId=null 2023-07-29T00:25:14.152 DEBUG:boto:Method: GET 2023-07-29T00:25:14.153 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000058?versionId=null 2023-07-29T00:25:14.154 DEBUG:boto:Data: 2023-07-29T00:25:14.154 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.155 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:14.156 DEBUG:boto:Port: 8000 2023-07-29T00:25:14.156 DEBUG:boto:Params: {} 2023-07-29T00:25:14.157 DEBUG:boto:Token: None 2023-07-29T00:25:14.158 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000058?versionId=null 2023-07-29T00:25:14.159 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HpgsJzNSLah8r8Y8s5ilHYZ2ysw= 2023-07-29T00:25:14.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HpgsJzNSLah8r8Y8s5ilHYZ2ysw=', 'Content-Length': '0'} 2023-07-29T00:25:14.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b99a5a81a2f7a2b-0064c45c6a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.162 DEBUG:boto:path=/hiiqos-58/obj00000000000000000058 2023-07-29T00:25:14.163 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000058 2023-07-29T00:25:14.163 DEBUG:boto:path=/hiiqos-58/obj00000000000000000058?versionId=null 2023-07-29T00:25:14.164 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000058?versionId=null 2023-07-29T00:25:14.164 DEBUG:boto:Method: HEAD 2023-07-29T00:25:14.164 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000058?versionId=null 2023-07-29T00:25:14.165 DEBUG:boto:Data: 2023-07-29T00:25:14.165 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.166 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:14.166 DEBUG:boto:Port: 8001 2023-07-29T00:25:14.166 DEBUG:boto:Params: {} 2023-07-29T00:25:14.167 DEBUG:boto:Token: None 2023-07-29T00:25:14.167 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000058?versionId=null 2023-07-29T00:25:14.168 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jMMCisC0RP4pSJpbYG6FXjyKxSo= 2023-07-29T00:25:14.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jMMCisC0RP4pSJpbYG6FXjyKxSo=', 'Content-Length': '0'} 2023-07-29T00:25:14.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f9a22b00250c7c6-0064c45c6a-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.170 DEBUG:boto:path=/hiiqos-58/obj00000000000000000058 2023-07-29T00:25:14.171 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000058 2023-07-29T00:25:14.172 DEBUG:boto:path=/hiiqos-58/obj00000000000000000058?versionId=null 2023-07-29T00:25:14.172 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000058?versionId=null 2023-07-29T00:25:14.172 DEBUG:boto:Method: HEAD 2023-07-29T00:25:14.173 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000058?versionId=null 2023-07-29T00:25:14.173 DEBUG:boto:Data: 2023-07-29T00:25:14.173 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.174 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:14.174 DEBUG:boto:Port: 8000 2023-07-29T00:25:14.174 DEBUG:boto:Params: {} 2023-07-29T00:25:14.175 DEBUG:boto:Token: None 2023-07-29T00:25:14.175 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000058?versionId=null 2023-07-29T00:25:14.176 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jMMCisC0RP4pSJpbYG6FXjyKxSo= 2023-07-29T00:25:14.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jMMCisC0RP4pSJpbYG6FXjyKxSo=', 'Content-Length': '0'} 2023-07-29T00:25:14.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b36e8595ec512499-0064c45c6a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2023-07-29T00:25:14.179 DEBUG:boto:path=/hiiqos-58/obj00000000000000000058 2023-07-29T00:25:14.180 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000058 2023-07-29T00:25:14.180 DEBUG:boto:path=/hiiqos-58/obj00000000000000000058?versionId=null 2023-07-29T00:25:14.180 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000058?versionId=null 2023-07-29T00:25:14.181 DEBUG:boto:Method: GET 2023-07-29T00:25:14.181 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000058?versionId=null 2023-07-29T00:25:14.182 DEBUG:boto:Data: 2023-07-29T00:25:14.182 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.182 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:14.183 DEBUG:boto:Port: 8001 2023-07-29T00:25:14.183 DEBUG:boto:Params: {} 2023-07-29T00:25:14.184 DEBUG:boto:Token: None 2023-07-29T00:25:14.184 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000058?versionId=null 2023-07-29T00:25:14.185 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HpgsJzNSLah8r8Y8s5ilHYZ2ysw= 2023-07-29T00:25:14.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HpgsJzNSLah8r8Y8s5ilHYZ2ysw=', 'Content-Length': '0'} 2023-07-29T00:25:14.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d456d7d5d1b3b3a-0064c45c6a-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.237 DEBUG:boto:path=/hiiqos-58/obj00000000000000000058 2023-07-29T00:25:14.237 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000058 2023-07-29T00:25:14.238 DEBUG:boto:path=/hiiqos-58/obj00000000000000000058?versionId=null 2023-07-29T00:25:14.238 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000058?versionId=null 2023-07-29T00:25:14.239 DEBUG:boto:Method: GET 2023-07-29T00:25:14.239 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000058?versionId=null 2023-07-29T00:25:14.239 DEBUG:boto:Data: 2023-07-29T00:25:14.240 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.241 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:14.241 DEBUG:boto:Port: 8000 2023-07-29T00:25:14.241 DEBUG:boto:Params: {} 2023-07-29T00:25:14.242 DEBUG:boto:Token: None 2023-07-29T00:25:14.242 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000058?versionId=null 2023-07-29T00:25:14.243 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HpgsJzNSLah8r8Y8s5ilHYZ2ysw= 2023-07-29T00:25:14.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HpgsJzNSLah8r8Y8s5ilHYZ2ysw=', 'Content-Length': '0'} 2023-07-29T00:25:14.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000935325b12254dad2-0064c45c6a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.246 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:14.246 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:14.247 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000058 2023-07-29T00:25:14.247 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000058 2023-07-29T00:25:14.248 DEBUG:boto:Method: GET 2023-07-29T00:25:14.248 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000058 2023-07-29T00:25:14.248 DEBUG:boto:Data: 2023-07-29T00:25:14.249 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.249 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:14.250 DEBUG:boto:Port: 8001 2023-07-29T00:25:14.253 DEBUG:boto:Params: {} 2023-07-29T00:25:14.255 DEBUG:boto:Token: None 2023-07-29T00:25:14.255 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/ 2023-07-29T00:25:14.256 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S5R3ASd1HfDELycrH0uGJ4cyYaU= 2023-07-29T00:25:14.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S5R3ASd1HfDELycrH0uGJ4cyYaU=', 'Content-Length': '0'} 2023-07-29T00:25:14.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a28942003875c7e8-0064c45c6a-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.259 DEBUG:boto:b'hiiqos-58obj000000000000000000581000falseobj000000000000000000582023-07-29T00:24:55.988Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:14.260 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:14.260 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:14.261 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000058 2023-07-29T00:25:14.261 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000058 2023-07-29T00:25:14.261 DEBUG:boto:Method: GET 2023-07-29T00:25:14.262 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000058 2023-07-29T00:25:14.262 DEBUG:boto:Data: 2023-07-29T00:25:14.262 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.263 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:14.263 DEBUG:boto:Port: 8000 2023-07-29T00:25:14.264 DEBUG:boto:Params: {} 2023-07-29T00:25:14.264 DEBUG:boto:Token: None 2023-07-29T00:25:14.265 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/ 2023-07-29T00:25:14.265 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S5R3ASd1HfDELycrH0uGJ4cyYaU= 2023-07-29T00:25:14.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S5R3ASd1HfDELycrH0uGJ4cyYaU=', 'Content-Length': '0'} 2023-07-29T00:25:14.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c94c867c58f3b2cf-0064c45c6a-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.312 DEBUG:boto:b'hiiqos-58obj000000000000000000581000falseobj000000000000000000582023-07-29T00:24:55.988Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:14.313 DEBUG:boto:path=/hiiqos-58/obj00000000000000000058 2023-07-29T00:25:14.314 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000058 2023-07-29T00:25:14.314 DEBUG:boto:Method: HEAD 2023-07-29T00:25:14.315 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000058 2023-07-29T00:25:14.315 DEBUG:boto:Data: 2023-07-29T00:25:14.315 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.316 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:14.316 DEBUG:boto:Port: 8001 2023-07-29T00:25:14.316 DEBUG:boto:Params: {} 2023-07-29T00:25:14.317 DEBUG:boto:Token: None 2023-07-29T00:25:14.317 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000058 2023-07-29T00:25:14.318 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6NuZ74zXCHPv7bxgKrn+N/sMNKM= 2023-07-29T00:25:14.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6NuZ74zXCHPv7bxgKrn+N/sMNKM=', 'Content-Length': '0'} 2023-07-29T00:25:14.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3552ca6cc7ad05a-0064c45c6a-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.321 DEBUG:boto:path=/hiiqos-58/obj00000000000000000058 2023-07-29T00:25:14.321 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000058 2023-07-29T00:25:14.321 DEBUG:boto:Method: HEAD 2023-07-29T00:25:14.322 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000058 2023-07-29T00:25:14.322 DEBUG:boto:Data: 2023-07-29T00:25:14.322 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.323 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:14.323 DEBUG:boto:Port: 8000 2023-07-29T00:25:14.323 DEBUG:boto:Params: {} 2023-07-29T00:25:14.324 DEBUG:boto:Token: None 2023-07-29T00:25:14.324 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000058 2023-07-29T00:25:14.325 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6NuZ74zXCHPv7bxgKrn+N/sMNKM= 2023-07-29T00:25:14.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6NuZ74zXCHPv7bxgKrn+N/sMNKM=', 'Content-Length': '0'} 2023-07-29T00:25:14.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f51b9ac97a33a84c-0064c45c6a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2023-07-29T00:25:14.328 DEBUG:boto:path=/hiiqos-58/obj00000000000000000058 2023-07-29T00:25:14.329 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000058 2023-07-29T00:25:14.329 DEBUG:boto:Method: GET 2023-07-29T00:25:14.330 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000058 2023-07-29T00:25:14.330 DEBUG:boto:Data: 2023-07-29T00:25:14.330 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.331 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:14.331 DEBUG:boto:Port: 8001 2023-07-29T00:25:14.331 DEBUG:boto:Params: {} 2023-07-29T00:25:14.332 DEBUG:boto:Token: None 2023-07-29T00:25:14.332 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000058 2023-07-29T00:25:14.333 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aFSSJQZwO/U0KA0Xt0N+77TrTV8= 2023-07-29T00:25:14.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aFSSJQZwO/U0KA0Xt0N+77TrTV8=', 'Content-Length': '0'} 2023-07-29T00:25:14.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053d8eeaf3eda29d5-0064c45c6a-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.335 DEBUG:boto:path=/hiiqos-58/obj00000000000000000058 2023-07-29T00:25:14.336 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000058 2023-07-29T00:25:14.336 DEBUG:boto:Method: GET 2023-07-29T00:25:14.337 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000058 2023-07-29T00:25:14.337 DEBUG:boto:Data: 2023-07-29T00:25:14.337 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.338 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:14.338 DEBUG:boto:Port: 8000 2023-07-29T00:25:14.338 DEBUG:boto:Params: {} 2023-07-29T00:25:14.339 DEBUG:boto:Token: None 2023-07-29T00:25:14.339 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000058 2023-07-29T00:25:14.340 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aFSSJQZwO/U0KA0Xt0N+77TrTV8= 2023-07-29T00:25:14.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aFSSJQZwO/U0KA0Xt0N+77TrTV8=', 'Content-Length': '0'} 2023-07-29T00:25:14.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:55 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dc0932144f51cc3-0064c45c6a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.343 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:14.343 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:14.344 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000058 2023-07-29T00:25:14.344 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000058 2023-07-29T00:25:14.344 DEBUG:boto:Method: GET 2023-07-29T00:25:14.345 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000058 2023-07-29T00:25:14.345 DEBUG:boto:Data: 2023-07-29T00:25:14.345 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.346 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:14.346 DEBUG:boto:Port: 8001 2023-07-29T00:25:14.347 DEBUG:boto:Params: {} 2023-07-29T00:25:14.347 DEBUG:boto:Token: None 2023-07-29T00:25:14.347 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/ 2023-07-29T00:25:14.348 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S5R3ASd1HfDELycrH0uGJ4cyYaU= 2023-07-29T00:25:14.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S5R3ASd1HfDELycrH0uGJ4cyYaU=', 'Content-Length': '0'} 2023-07-29T00:25:14.351 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000741e8b3532125d51-0064c45c6a-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.351 DEBUG:boto:b'hiiqos-58obj000000000000000000581000falseobj000000000000000000582023-07-29T00:24:55.988Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:14.352 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:14.352 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:14.353 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000058 2023-07-29T00:25:14.353 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000058 2023-07-29T00:25:14.353 DEBUG:boto:Method: GET 2023-07-29T00:25:14.354 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000058 2023-07-29T00:25:14.354 DEBUG:boto:Data: 2023-07-29T00:25:14.354 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.355 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:14.355 DEBUG:boto:Port: 8000 2023-07-29T00:25:14.355 DEBUG:boto:Params: {} 2023-07-29T00:25:14.356 DEBUG:boto:Token: None 2023-07-29T00:25:14.356 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/ 2023-07-29T00:25:14.357 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S5R3ASd1HfDELycrH0uGJ4cyYaU= 2023-07-29T00:25:14.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S5R3ASd1HfDELycrH0uGJ4cyYaU=', 'Content-Length': '0'} 2023-07-29T00:25:14.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091e59d8aabffcaba-0064c45c6a-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.362 DEBUG:boto:b'hiiqos-58obj000000000000000000581000falseobj000000000000000000582023-07-29T00:24:55.988Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:14.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2023-07-29T00:25:14.363 DEBUG:boto:path=/hiiqos-58/obj00000000000000000059 2023-07-29T00:25:14.363 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000059 2023-07-29T00:25:14.364 DEBUG:boto:path=/hiiqos-58/obj00000000000000000059?versionId=null 2023-07-29T00:25:14.364 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000059?versionId=null 2023-07-29T00:25:14.364 DEBUG:boto:Method: GET 2023-07-29T00:25:14.365 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000059?versionId=null 2023-07-29T00:25:14.365 DEBUG:boto:Data: 2023-07-29T00:25:14.366 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.366 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:14.366 DEBUG:boto:Port: 8001 2023-07-29T00:25:14.367 DEBUG:boto:Params: {} 2023-07-29T00:25:14.367 DEBUG:boto:Token: None 2023-07-29T00:25:14.367 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000059?versionId=null 2023-07-29T00:25:14.368 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OKO9Jz00/f5mdHjAjgIZtY1WIv0= 2023-07-29T00:25:14.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OKO9Jz00/f5mdHjAjgIZtY1WIv0=', 'Content-Length': '0'} 2023-07-29T00:25:14.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001dab54d8bff1f1cf-0064c45c6a-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.371 DEBUG:boto:path=/hiiqos-58/obj00000000000000000059 2023-07-29T00:25:14.371 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000059 2023-07-29T00:25:14.372 DEBUG:boto:path=/hiiqos-58/obj00000000000000000059?versionId=null 2023-07-29T00:25:14.372 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000059?versionId=null 2023-07-29T00:25:14.373 DEBUG:boto:Method: GET 2023-07-29T00:25:14.373 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000059?versionId=null 2023-07-29T00:25:14.373 DEBUG:boto:Data: 2023-07-29T00:25:14.374 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.374 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:14.374 DEBUG:boto:Port: 8000 2023-07-29T00:25:14.375 DEBUG:boto:Params: {} 2023-07-29T00:25:14.375 DEBUG:boto:Token: None 2023-07-29T00:25:14.375 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000059?versionId=null 2023-07-29T00:25:14.376 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OKO9Jz00/f5mdHjAjgIZtY1WIv0= 2023-07-29T00:25:14.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OKO9Jz00/f5mdHjAjgIZtY1WIv0=', 'Content-Length': '0'} 2023-07-29T00:25:14.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a489a712004b372a-0064c45c6a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.381 DEBUG:boto:path=/hiiqos-58/obj00000000000000000059 2023-07-29T00:25:14.381 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000059 2023-07-29T00:25:14.381 DEBUG:boto:path=/hiiqos-58/obj00000000000000000059?versionId=null 2023-07-29T00:25:14.382 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000059?versionId=null 2023-07-29T00:25:14.382 DEBUG:boto:Method: HEAD 2023-07-29T00:25:14.383 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000059?versionId=null 2023-07-29T00:25:14.383 DEBUG:boto:Data: 2023-07-29T00:25:14.383 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.384 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:14.384 DEBUG:boto:Port: 8001 2023-07-29T00:25:14.384 DEBUG:boto:Params: {} 2023-07-29T00:25:14.385 DEBUG:boto:Token: None 2023-07-29T00:25:14.385 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000059?versionId=null 2023-07-29T00:25:14.386 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sxEJ1Zz9HMWGispZzLMwbd0zwkg= 2023-07-29T00:25:14.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sxEJ1Zz9HMWGispZzLMwbd0zwkg=', 'Content-Length': '0'} 2023-07-29T00:25:14.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000251bbd07ebc3fcc-0064c45c6a-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.390 DEBUG:boto:path=/hiiqos-58/obj00000000000000000059 2023-07-29T00:25:14.390 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000059 2023-07-29T00:25:14.391 DEBUG:boto:path=/hiiqos-58/obj00000000000000000059?versionId=null 2023-07-29T00:25:14.391 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000059?versionId=null 2023-07-29T00:25:14.391 DEBUG:boto:Method: HEAD 2023-07-29T00:25:14.392 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000059?versionId=null 2023-07-29T00:25:14.392 DEBUG:boto:Data: 2023-07-29T00:25:14.392 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.393 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:14.393 DEBUG:boto:Port: 8000 2023-07-29T00:25:14.393 DEBUG:boto:Params: {} 2023-07-29T00:25:14.394 DEBUG:boto:Token: None 2023-07-29T00:25:14.394 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000059?versionId=null 2023-07-29T00:25:14.395 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sxEJ1Zz9HMWGispZzLMwbd0zwkg= 2023-07-29T00:25:14.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sxEJ1Zz9HMWGispZzLMwbd0zwkg=', 'Content-Length': '0'} 2023-07-29T00:25:14.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e07d7e8b1c104ce-0064c45c6a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2023-07-29T00:25:14.398 DEBUG:boto:path=/hiiqos-58/obj00000000000000000059 2023-07-29T00:25:14.398 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000059 2023-07-29T00:25:14.399 DEBUG:boto:path=/hiiqos-58/obj00000000000000000059?versionId=null 2023-07-29T00:25:14.399 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000059?versionId=null 2023-07-29T00:25:14.399 DEBUG:boto:Method: GET 2023-07-29T00:25:14.400 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000059?versionId=null 2023-07-29T00:25:14.400 DEBUG:boto:Data: 2023-07-29T00:25:14.400 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.401 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:14.401 DEBUG:boto:Port: 8001 2023-07-29T00:25:14.401 DEBUG:boto:Params: {} 2023-07-29T00:25:14.402 DEBUG:boto:Token: None 2023-07-29T00:25:14.402 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000059?versionId=null 2023-07-29T00:25:14.403 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OKO9Jz00/f5mdHjAjgIZtY1WIv0= 2023-07-29T00:25:14.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OKO9Jz00/f5mdHjAjgIZtY1WIv0=', 'Content-Length': '0'} 2023-07-29T00:25:14.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005037c403396d8abe-0064c45c6a-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.406 DEBUG:boto:path=/hiiqos-58/obj00000000000000000059 2023-07-29T00:25:14.406 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000059 2023-07-29T00:25:14.407 DEBUG:boto:path=/hiiqos-58/obj00000000000000000059?versionId=null 2023-07-29T00:25:14.407 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000059?versionId=null 2023-07-29T00:25:14.407 DEBUG:boto:Method: GET 2023-07-29T00:25:14.408 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000059?versionId=null 2023-07-29T00:25:14.408 DEBUG:boto:Data: 2023-07-29T00:25:14.408 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.409 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:14.409 DEBUG:boto:Port: 8000 2023-07-29T00:25:14.409 DEBUG:boto:Params: {} 2023-07-29T00:25:14.410 DEBUG:boto:Token: None 2023-07-29T00:25:14.410 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000059?versionId=null 2023-07-29T00:25:14.411 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OKO9Jz00/f5mdHjAjgIZtY1WIv0= 2023-07-29T00:25:14.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OKO9Jz00/f5mdHjAjgIZtY1WIv0=', 'Content-Length': '0'} 2023-07-29T00:25:14.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3fe320b3b1971b4-0064c45c6a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.456 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:14.457 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:14.457 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000059 2023-07-29T00:25:14.458 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000059 2023-07-29T00:25:14.458 DEBUG:boto:Method: GET 2023-07-29T00:25:14.459 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000059 2023-07-29T00:25:14.459 DEBUG:boto:Data: 2023-07-29T00:25:14.459 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.460 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:14.460 DEBUG:boto:Port: 8001 2023-07-29T00:25:14.460 DEBUG:boto:Params: {} 2023-07-29T00:25:14.461 DEBUG:boto:Token: None 2023-07-29T00:25:14.461 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/ 2023-07-29T00:25:14.462 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S5R3ASd1HfDELycrH0uGJ4cyYaU= 2023-07-29T00:25:14.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S5R3ASd1HfDELycrH0uGJ4cyYaU=', 'Content-Length': '0'} 2023-07-29T00:25:14.465 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000236c49b827071ed1-0064c45c6a-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.465 DEBUG:boto:b'hiiqos-58obj000000000000000000591000falseobj000000000000000000592023-07-29T00:24:56.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:14.466 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:14.466 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:14.467 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000059 2023-07-29T00:25:14.467 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000059 2023-07-29T00:25:14.468 DEBUG:boto:Method: GET 2023-07-29T00:25:14.468 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000059 2023-07-29T00:25:14.468 DEBUG:boto:Data: 2023-07-29T00:25:14.469 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.469 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:14.469 DEBUG:boto:Port: 8000 2023-07-29T00:25:14.470 DEBUG:boto:Params: {} 2023-07-29T00:25:14.470 DEBUG:boto:Token: None 2023-07-29T00:25:14.471 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/ 2023-07-29T00:25:14.471 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S5R3ASd1HfDELycrH0uGJ4cyYaU= 2023-07-29T00:25:14.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S5R3ASd1HfDELycrH0uGJ4cyYaU=', 'Content-Length': '0'} 2023-07-29T00:25:14.476 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061395debf5a6548a-0064c45c6a-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.476 DEBUG:boto:b'hiiqos-58obj000000000000000000591000falseobj000000000000000000592023-07-29T00:24:56.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:14.477 DEBUG:boto:path=/hiiqos-58/obj00000000000000000059 2023-07-29T00:25:14.477 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000059 2023-07-29T00:25:14.478 DEBUG:boto:Method: HEAD 2023-07-29T00:25:14.478 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000059 2023-07-29T00:25:14.478 DEBUG:boto:Data: 2023-07-29T00:25:14.479 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.479 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:14.479 DEBUG:boto:Port: 8001 2023-07-29T00:25:14.480 DEBUG:boto:Params: {} 2023-07-29T00:25:14.480 DEBUG:boto:Token: None 2023-07-29T00:25:14.480 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000059 2023-07-29T00:25:14.481 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CaaxOuVoORg8670SUuB1U3alsJs= 2023-07-29T00:25:14.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CaaxOuVoORg8670SUuB1U3alsJs=', 'Content-Length': '0'} 2023-07-29T00:25:14.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028db228672037b95-0064c45c6a-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.484 DEBUG:boto:path=/hiiqos-58/obj00000000000000000059 2023-07-29T00:25:14.484 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000059 2023-07-29T00:25:14.484 DEBUG:boto:Method: HEAD 2023-07-29T00:25:14.485 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000059 2023-07-29T00:25:14.485 DEBUG:boto:Data: 2023-07-29T00:25:14.485 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.486 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:14.486 DEBUG:boto:Port: 8000 2023-07-29T00:25:14.486 DEBUG:boto:Params: {} 2023-07-29T00:25:14.487 DEBUG:boto:Token: None 2023-07-29T00:25:14.487 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000059 2023-07-29T00:25:14.487 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CaaxOuVoORg8670SUuB1U3alsJs= 2023-07-29T00:25:14.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CaaxOuVoORg8670SUuB1U3alsJs=', 'Content-Length': '0'} 2023-07-29T00:25:14.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065f3b2762cc5a8ed-0064c45c6a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2023-07-29T00:25:14.491 DEBUG:boto:path=/hiiqos-58/obj00000000000000000059 2023-07-29T00:25:14.491 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000059 2023-07-29T00:25:14.491 DEBUG:boto:Method: GET 2023-07-29T00:25:14.492 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000059 2023-07-29T00:25:14.492 DEBUG:boto:Data: 2023-07-29T00:25:14.492 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.493 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:14.493 DEBUG:boto:Port: 8001 2023-07-29T00:25:14.494 DEBUG:boto:Params: {} 2023-07-29T00:25:14.494 DEBUG:boto:Token: None 2023-07-29T00:25:14.494 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000059 2023-07-29T00:25:14.495 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kYYd5byk23IXrI+2dA+YWKtUask= 2023-07-29T00:25:14.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kYYd5byk23IXrI+2dA+YWKtUask=', 'Content-Length': '0'} 2023-07-29T00:25:14.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002938301aaadc24bc-0064c45c6a-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.541 DEBUG:boto:path=/hiiqos-58/obj00000000000000000059 2023-07-29T00:25:14.542 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000059 2023-07-29T00:25:14.543 DEBUG:boto:Method: GET 2023-07-29T00:25:14.545 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000059 2023-07-29T00:25:14.546 DEBUG:boto:Data: 2023-07-29T00:25:14.547 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.548 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:14.548 DEBUG:boto:Port: 8000 2023-07-29T00:25:14.549 DEBUG:boto:Params: {} 2023-07-29T00:25:14.550 DEBUG:boto:Token: None 2023-07-29T00:25:14.550 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000059 2023-07-29T00:25:14.551 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kYYd5byk23IXrI+2dA+YWKtUask= 2023-07-29T00:25:14.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kYYd5byk23IXrI+2dA+YWKtUask=', 'Content-Length': '0'} 2023-07-29T00:25:14.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b93989d0fc2a9498-0064c45c6a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.555 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:14.556 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:14.557 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000059 2023-07-29T00:25:14.557 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000059 2023-07-29T00:25:14.558 DEBUG:boto:Method: GET 2023-07-29T00:25:14.558 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000059 2023-07-29T00:25:14.559 DEBUG:boto:Data: 2023-07-29T00:25:14.559 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.560 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:14.560 DEBUG:boto:Port: 8001 2023-07-29T00:25:14.560 DEBUG:boto:Params: {} 2023-07-29T00:25:14.561 DEBUG:boto:Token: None 2023-07-29T00:25:14.561 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/ 2023-07-29T00:25:14.562 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S5R3ASd1HfDELycrH0uGJ4cyYaU= 2023-07-29T00:25:14.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S5R3ASd1HfDELycrH0uGJ4cyYaU=', 'Content-Length': '0'} 2023-07-29T00:25:14.565 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077275a3194892945-0064c45c6a-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.566 DEBUG:boto:b'hiiqos-58obj000000000000000000591000falseobj000000000000000000592023-07-29T00:24:56.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:14.566 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:14.567 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:14.567 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000059 2023-07-29T00:25:14.567 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000059 2023-07-29T00:25:14.568 DEBUG:boto:Method: GET 2023-07-29T00:25:14.568 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000059 2023-07-29T00:25:14.569 DEBUG:boto:Data: 2023-07-29T00:25:14.569 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.569 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:14.570 DEBUG:boto:Port: 8000 2023-07-29T00:25:14.570 DEBUG:boto:Params: {} 2023-07-29T00:25:14.570 DEBUG:boto:Token: None 2023-07-29T00:25:14.571 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/ 2023-07-29T00:25:14.571 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S5R3ASd1HfDELycrH0uGJ4cyYaU= 2023-07-29T00:25:14.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S5R3ASd1HfDELycrH0uGJ4cyYaU=', 'Content-Length': '0'} 2023-07-29T00:25:14.576 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f20ead7c7b08a5e9-0064c45c6a-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.577 DEBUG:boto:b'hiiqos-58obj000000000000000000591000falseobj000000000000000000592023-07-29T00:24:56.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:14.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2023-07-29T00:25:14.578 DEBUG:boto:path=/hiiqos-58/obj00000000000000000060 2023-07-29T00:25:14.578 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000060 2023-07-29T00:25:14.579 DEBUG:boto:path=/hiiqos-58/obj00000000000000000060?versionId=null 2023-07-29T00:25:14.579 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000060?versionId=null 2023-07-29T00:25:14.580 DEBUG:boto:Method: GET 2023-07-29T00:25:14.580 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000060?versionId=null 2023-07-29T00:25:14.580 DEBUG:boto:Data: 2023-07-29T00:25:14.581 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.581 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:14.581 DEBUG:boto:Port: 8001 2023-07-29T00:25:14.582 DEBUG:boto:Params: {} 2023-07-29T00:25:14.582 DEBUG:boto:Token: None 2023-07-29T00:25:14.583 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000060?versionId=null 2023-07-29T00:25:14.583 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:l6JuSzWJYr8mWFrAq4oO1HSjtPU= 2023-07-29T00:25:14.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:l6JuSzWJYr8mWFrAq4oO1HSjtPU=', 'Content-Length': '0'} 2023-07-29T00:25:14.586 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009faef92f3219c591-0064c45c6a-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.586 DEBUG:boto:path=/hiiqos-58/obj00000000000000000060 2023-07-29T00:25:14.587 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000060 2023-07-29T00:25:14.587 DEBUG:boto:path=/hiiqos-58/obj00000000000000000060?versionId=null 2023-07-29T00:25:14.588 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000060?versionId=null 2023-07-29T00:25:14.588 DEBUG:boto:Method: GET 2023-07-29T00:25:14.588 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000060?versionId=null 2023-07-29T00:25:14.589 DEBUG:boto:Data: 2023-07-29T00:25:14.589 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.589 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:14.590 DEBUG:boto:Port: 8000 2023-07-29T00:25:14.590 DEBUG:boto:Params: {} 2023-07-29T00:25:14.591 DEBUG:boto:Token: None 2023-07-29T00:25:14.591 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000060?versionId=null 2023-07-29T00:25:14.591 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:l6JuSzWJYr8mWFrAq4oO1HSjtPU= 2023-07-29T00:25:14.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:l6JuSzWJYr8mWFrAq4oO1HSjtPU=', 'Content-Length': '0'} 2023-07-29T00:25:14.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000952e49304ed75e15-0064c45c6a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.594 DEBUG:boto:path=/hiiqos-58/obj00000000000000000060 2023-07-29T00:25:14.595 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000060 2023-07-29T00:25:14.595 DEBUG:boto:path=/hiiqos-58/obj00000000000000000060?versionId=null 2023-07-29T00:25:14.595 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000060?versionId=null 2023-07-29T00:25:14.596 DEBUG:boto:Method: HEAD 2023-07-29T00:25:14.596 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000060?versionId=null 2023-07-29T00:25:14.597 DEBUG:boto:Data: 2023-07-29T00:25:14.597 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.597 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:14.597 DEBUG:boto:Port: 8001 2023-07-29T00:25:14.598 DEBUG:boto:Params: {} 2023-07-29T00:25:14.598 DEBUG:boto:Token: None 2023-07-29T00:25:14.599 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000060?versionId=null 2023-07-29T00:25:14.599 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eLLlioeHk77U1uHFgcohflw4mog= 2023-07-29T00:25:14.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eLLlioeHk77U1uHFgcohflw4mog=', 'Content-Length': '0'} 2023-07-29T00:25:14.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ca8a1cff38e91f2-0064c45c6a-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.602 DEBUG:boto:path=/hiiqos-58/obj00000000000000000060 2023-07-29T00:25:14.602 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000060 2023-07-29T00:25:14.602 DEBUG:boto:path=/hiiqos-58/obj00000000000000000060?versionId=null 2023-07-29T00:25:14.603 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000060?versionId=null 2023-07-29T00:25:14.603 DEBUG:boto:Method: HEAD 2023-07-29T00:25:14.603 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000060?versionId=null 2023-07-29T00:25:14.604 DEBUG:boto:Data: 2023-07-29T00:25:14.604 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.604 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:14.605 DEBUG:boto:Port: 8000 2023-07-29T00:25:14.605 DEBUG:boto:Params: {} 2023-07-29T00:25:14.605 DEBUG:boto:Token: None 2023-07-29T00:25:14.606 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000060?versionId=null 2023-07-29T00:25:14.606 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eLLlioeHk77U1uHFgcohflw4mog= 2023-07-29T00:25:14.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eLLlioeHk77U1uHFgcohflw4mog=', 'Content-Length': '0'} 2023-07-29T00:25:14.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dae17f4029503b26-0064c45c6a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2023-07-29T00:25:14.609 DEBUG:boto:path=/hiiqos-58/obj00000000000000000060 2023-07-29T00:25:14.610 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000060 2023-07-29T00:25:14.610 DEBUG:boto:path=/hiiqos-58/obj00000000000000000060?versionId=null 2023-07-29T00:25:14.611 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000060?versionId=null 2023-07-29T00:25:14.611 DEBUG:boto:Method: GET 2023-07-29T00:25:14.611 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000060?versionId=null 2023-07-29T00:25:14.612 DEBUG:boto:Data: 2023-07-29T00:25:14.612 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.612 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:14.613 DEBUG:boto:Port: 8001 2023-07-29T00:25:14.613 DEBUG:boto:Params: {} 2023-07-29T00:25:14.613 DEBUG:boto:Token: None 2023-07-29T00:25:14.614 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000060?versionId=null 2023-07-29T00:25:14.614 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:l6JuSzWJYr8mWFrAq4oO1HSjtPU= 2023-07-29T00:25:14.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:l6JuSzWJYr8mWFrAq4oO1HSjtPU=', 'Content-Length': '0'} 2023-07-29T00:25:14.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f46f8dc1bde9009-0064c45c6a-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.617 DEBUG:boto:path=/hiiqos-58/obj00000000000000000060 2023-07-29T00:25:14.618 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000060 2023-07-29T00:25:14.618 DEBUG:boto:path=/hiiqos-58/obj00000000000000000060?versionId=null 2023-07-29T00:25:14.618 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000060?versionId=null 2023-07-29T00:25:14.619 DEBUG:boto:Method: GET 2023-07-29T00:25:14.619 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000060?versionId=null 2023-07-29T00:25:14.619 DEBUG:boto:Data: 2023-07-29T00:25:14.620 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.620 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:14.620 DEBUG:boto:Port: 8000 2023-07-29T00:25:14.621 DEBUG:boto:Params: {} 2023-07-29T00:25:14.621 DEBUG:boto:Token: None 2023-07-29T00:25:14.622 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000060?versionId=null 2023-07-29T00:25:14.622 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:l6JuSzWJYr8mWFrAq4oO1HSjtPU= 2023-07-29T00:25:14.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:l6JuSzWJYr8mWFrAq4oO1HSjtPU=', 'Content-Length': '0'} 2023-07-29T00:25:14.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e5f58eb721086f7-0064c45c6a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.629 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:14.629 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:14.630 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000060 2023-07-29T00:25:14.630 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000060 2023-07-29T00:25:14.630 DEBUG:boto:Method: GET 2023-07-29T00:25:14.631 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000060 2023-07-29T00:25:14.631 DEBUG:boto:Data: 2023-07-29T00:25:14.631 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.632 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:14.632 DEBUG:boto:Port: 8001 2023-07-29T00:25:14.632 DEBUG:boto:Params: {} 2023-07-29T00:25:14.633 DEBUG:boto:Token: None 2023-07-29T00:25:14.633 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/ 2023-07-29T00:25:14.634 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S5R3ASd1HfDELycrH0uGJ4cyYaU= 2023-07-29T00:25:14.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S5R3ASd1HfDELycrH0uGJ4cyYaU=', 'Content-Length': '0'} 2023-07-29T00:25:14.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000982042cbfb841a8d-0064c45c6a-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.680 DEBUG:boto:b'hiiqos-58obj000000000000000000601000falseobj000000000000000000602023-07-29T00:24:59.520Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:14.681 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:14.682 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:14.682 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000060 2023-07-29T00:25:14.682 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000060 2023-07-29T00:25:14.683 DEBUG:boto:Method: GET 2023-07-29T00:25:14.683 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000060 2023-07-29T00:25:14.683 DEBUG:boto:Data: 2023-07-29T00:25:14.684 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.684 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:14.685 DEBUG:boto:Port: 8000 2023-07-29T00:25:14.685 DEBUG:boto:Params: {} 2023-07-29T00:25:14.685 DEBUG:boto:Token: None 2023-07-29T00:25:14.686 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/ 2023-07-29T00:25:14.686 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S5R3ASd1HfDELycrH0uGJ4cyYaU= 2023-07-29T00:25:14.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S5R3ASd1HfDELycrH0uGJ4cyYaU=', 'Content-Length': '0'} 2023-07-29T00:25:14.692 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000848f16be0a02206f-0064c45c6a-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.693 DEBUG:boto:b'hiiqos-58obj000000000000000000601000falseobj000000000000000000602023-07-29T00:24:59.520Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:14.694 DEBUG:boto:path=/hiiqos-58/obj00000000000000000060 2023-07-29T00:25:14.694 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000060 2023-07-29T00:25:14.695 DEBUG:boto:Method: HEAD 2023-07-29T00:25:14.696 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000060 2023-07-29T00:25:14.697 DEBUG:boto:Data: 2023-07-29T00:25:14.698 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.699 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:14.700 DEBUG:boto:Port: 8001 2023-07-29T00:25:14.701 DEBUG:boto:Params: {} 2023-07-29T00:25:14.702 DEBUG:boto:Token: None 2023-07-29T00:25:14.702 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000060 2023-07-29T00:25:14.703 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Xl+0J4lS8mhgNriRuHF6jRUw1Gg= 2023-07-29T00:25:14.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Xl+0J4lS8mhgNriRuHF6jRUw1Gg=', 'Content-Length': '0'} 2023-07-29T00:25:14.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000545b5a6521ab2d51-0064c45c6a-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.707 DEBUG:boto:path=/hiiqos-58/obj00000000000000000060 2023-07-29T00:25:14.707 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000060 2023-07-29T00:25:14.708 DEBUG:boto:Method: HEAD 2023-07-29T00:25:14.708 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000060 2023-07-29T00:25:14.709 DEBUG:boto:Data: 2023-07-29T00:25:14.709 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.709 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:14.710 DEBUG:boto:Port: 8000 2023-07-29T00:25:14.711 DEBUG:boto:Params: {} 2023-07-29T00:25:14.712 DEBUG:boto:Token: None 2023-07-29T00:25:14.712 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000060 2023-07-29T00:25:14.713 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Xl+0J4lS8mhgNriRuHF6jRUw1Gg= 2023-07-29T00:25:14.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Xl+0J4lS8mhgNriRuHF6jRUw1Gg=', 'Content-Length': '0'} 2023-07-29T00:25:14.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e763032d8d42f321-0064c45c6a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2023-07-29T00:25:14.717 DEBUG:boto:path=/hiiqos-58/obj00000000000000000060 2023-07-29T00:25:14.717 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000060 2023-07-29T00:25:14.718 DEBUG:boto:Method: GET 2023-07-29T00:25:14.718 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000060 2023-07-29T00:25:14.719 DEBUG:boto:Data: 2023-07-29T00:25:14.719 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.720 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:14.720 DEBUG:boto:Port: 8001 2023-07-29T00:25:14.721 DEBUG:boto:Params: {} 2023-07-29T00:25:14.721 DEBUG:boto:Token: None 2023-07-29T00:25:14.722 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000060 2023-07-29T00:25:14.722 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CNhBYUOKtPTU5USmgXoQhfuvvkw= 2023-07-29T00:25:14.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CNhBYUOKtPTU5USmgXoQhfuvvkw=', 'Content-Length': '0'} 2023-07-29T00:25:14.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1b10f8ab825c51c-0064c45c6a-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.725 DEBUG:boto:path=/hiiqos-58/obj00000000000000000060 2023-07-29T00:25:14.726 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000060 2023-07-29T00:25:14.727 DEBUG:boto:Method: GET 2023-07-29T00:25:14.727 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000060 2023-07-29T00:25:14.727 DEBUG:boto:Data: 2023-07-29T00:25:14.728 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.729 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:14.729 DEBUG:boto:Port: 8000 2023-07-29T00:25:14.729 DEBUG:boto:Params: {} 2023-07-29T00:25:14.730 DEBUG:boto:Token: None 2023-07-29T00:25:14.730 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000060 2023-07-29T00:25:14.731 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CNhBYUOKtPTU5USmgXoQhfuvvkw= 2023-07-29T00:25:14.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CNhBYUOKtPTU5USmgXoQhfuvvkw=', 'Content-Length': '0'} 2023-07-29T00:25:14.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb758ec4f80d7521-0064c45c6a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.776 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:14.777 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:14.778 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000060 2023-07-29T00:25:14.778 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000060 2023-07-29T00:25:14.779 DEBUG:boto:Method: GET 2023-07-29T00:25:14.779 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000060 2023-07-29T00:25:14.779 DEBUG:boto:Data: 2023-07-29T00:25:14.780 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.780 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:14.781 DEBUG:boto:Port: 8001 2023-07-29T00:25:14.781 DEBUG:boto:Params: {} 2023-07-29T00:25:14.781 DEBUG:boto:Token: None 2023-07-29T00:25:14.782 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/ 2023-07-29T00:25:14.782 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S5R3ASd1HfDELycrH0uGJ4cyYaU= 2023-07-29T00:25:14.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S5R3ASd1HfDELycrH0uGJ4cyYaU=', 'Content-Length': '0'} 2023-07-29T00:25:14.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035b32f93b7253729-0064c45c6a-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.786 DEBUG:boto:b'hiiqos-58obj000000000000000000601000falseobj000000000000000000602023-07-29T00:24:59.520Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:14.787 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:14.787 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:14.788 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000060 2023-07-29T00:25:14.788 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000060 2023-07-29T00:25:14.789 DEBUG:boto:Method: GET 2023-07-29T00:25:14.789 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000060 2023-07-29T00:25:14.789 DEBUG:boto:Data: 2023-07-29T00:25:14.790 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.790 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:14.790 DEBUG:boto:Port: 8000 2023-07-29T00:25:14.791 DEBUG:boto:Params: {} 2023-07-29T00:25:14.791 DEBUG:boto:Token: None 2023-07-29T00:25:14.792 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/ 2023-07-29T00:25:14.792 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S5R3ASd1HfDELycrH0uGJ4cyYaU= 2023-07-29T00:25:14.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S5R3ASd1HfDELycrH0uGJ4cyYaU=', 'Content-Length': '0'} 2023-07-29T00:25:14.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6a57eae5bad6703-0064c45c6a-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.799 DEBUG:boto:b'hiiqos-58obj000000000000000000601000falseobj000000000000000000602023-07-29T00:24:59.520Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:14.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2023-07-29T00:25:14.800 DEBUG:boto:path=/hiiqos-58/obj00000000000000000061 2023-07-29T00:25:14.800 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000061 2023-07-29T00:25:14.801 DEBUG:boto:path=/hiiqos-58/obj00000000000000000061?versionId=null 2023-07-29T00:25:14.801 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000061?versionId=null 2023-07-29T00:25:14.801 DEBUG:boto:Method: GET 2023-07-29T00:25:14.802 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000061?versionId=null 2023-07-29T00:25:14.802 DEBUG:boto:Data: 2023-07-29T00:25:14.802 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.803 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:14.803 DEBUG:boto:Port: 8001 2023-07-29T00:25:14.803 DEBUG:boto:Params: {} 2023-07-29T00:25:14.804 DEBUG:boto:Token: None 2023-07-29T00:25:14.804 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000061?versionId=null 2023-07-29T00:25:14.805 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LJdYX7cuIJoILmhXdcL2n3qOd1I= 2023-07-29T00:25:14.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LJdYX7cuIJoILmhXdcL2n3qOd1I=', 'Content-Length': '0'} 2023-07-29T00:25:14.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fec8157f08da4cd5-0064c45c6a-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.808 DEBUG:boto:path=/hiiqos-58/obj00000000000000000061 2023-07-29T00:25:14.808 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000061 2023-07-29T00:25:14.809 DEBUG:boto:path=/hiiqos-58/obj00000000000000000061?versionId=null 2023-07-29T00:25:14.809 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000061?versionId=null 2023-07-29T00:25:14.810 DEBUG:boto:Method: GET 2023-07-29T00:25:14.810 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000061?versionId=null 2023-07-29T00:25:14.810 DEBUG:boto:Data: 2023-07-29T00:25:14.811 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.811 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:14.811 DEBUG:boto:Port: 8000 2023-07-29T00:25:14.812 DEBUG:boto:Params: {} 2023-07-29T00:25:14.812 DEBUG:boto:Token: None 2023-07-29T00:25:14.813 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000061?versionId=null 2023-07-29T00:25:14.813 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LJdYX7cuIJoILmhXdcL2n3qOd1I= 2023-07-29T00:25:14.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LJdYX7cuIJoILmhXdcL2n3qOd1I=', 'Content-Length': '0'} 2023-07-29T00:25:14.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc3b391ff75d55a3-0064c45c6a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.816 DEBUG:boto:path=/hiiqos-58/obj00000000000000000061 2023-07-29T00:25:14.817 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000061 2023-07-29T00:25:14.817 DEBUG:boto:path=/hiiqos-58/obj00000000000000000061?versionId=null 2023-07-29T00:25:14.817 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000061?versionId=null 2023-07-29T00:25:14.818 DEBUG:boto:Method: HEAD 2023-07-29T00:25:14.818 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000061?versionId=null 2023-07-29T00:25:14.818 DEBUG:boto:Data: 2023-07-29T00:25:14.819 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.819 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:14.819 DEBUG:boto:Port: 8001 2023-07-29T00:25:14.820 DEBUG:boto:Params: {} 2023-07-29T00:25:14.820 DEBUG:boto:Token: None 2023-07-29T00:25:14.821 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000061?versionId=null 2023-07-29T00:25:14.821 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:awohq98yynzVWNrfhrEWvycHOWE= 2023-07-29T00:25:14.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:awohq98yynzVWNrfhrEWvycHOWE=', 'Content-Length': '0'} 2023-07-29T00:25:14.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc503342996d3ac8-0064c45c6a-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.824 DEBUG:boto:path=/hiiqos-58/obj00000000000000000061 2023-07-29T00:25:14.824 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000061 2023-07-29T00:25:14.824 DEBUG:boto:path=/hiiqos-58/obj00000000000000000061?versionId=null 2023-07-29T00:25:14.825 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000061?versionId=null 2023-07-29T00:25:14.825 DEBUG:boto:Method: HEAD 2023-07-29T00:25:14.825 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000061?versionId=null 2023-07-29T00:25:14.826 DEBUG:boto:Data: 2023-07-29T00:25:14.826 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.826 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:14.827 DEBUG:boto:Port: 8000 2023-07-29T00:25:14.827 DEBUG:boto:Params: {} 2023-07-29T00:25:14.827 DEBUG:boto:Token: None 2023-07-29T00:25:14.828 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000061?versionId=null 2023-07-29T00:25:14.828 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:awohq98yynzVWNrfhrEWvycHOWE= 2023-07-29T00:25:14.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:awohq98yynzVWNrfhrEWvycHOWE=', 'Content-Length': '0'} 2023-07-29T00:25:14.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031e0dd0cfc1e4337-0064c45c6a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2023-07-29T00:25:14.832 DEBUG:boto:path=/hiiqos-58/obj00000000000000000061 2023-07-29T00:25:14.832 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000061 2023-07-29T00:25:14.832 DEBUG:boto:path=/hiiqos-58/obj00000000000000000061?versionId=null 2023-07-29T00:25:14.833 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000061?versionId=null 2023-07-29T00:25:14.833 DEBUG:boto:Method: GET 2023-07-29T00:25:14.834 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000061?versionId=null 2023-07-29T00:25:14.834 DEBUG:boto:Data: 2023-07-29T00:25:14.834 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.835 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:14.835 DEBUG:boto:Port: 8001 2023-07-29T00:25:14.836 DEBUG:boto:Params: {} 2023-07-29T00:25:14.836 DEBUG:boto:Token: None 2023-07-29T00:25:14.836 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000061?versionId=null 2023-07-29T00:25:14.837 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LJdYX7cuIJoILmhXdcL2n3qOd1I= 2023-07-29T00:25:14.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LJdYX7cuIJoILmhXdcL2n3qOd1I=', 'Content-Length': '0'} 2023-07-29T00:25:14.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c103d31ec04b213-0064c45c6a-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.840 DEBUG:boto:path=/hiiqos-58/obj00000000000000000061 2023-07-29T00:25:14.840 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000061 2023-07-29T00:25:14.841 DEBUG:boto:path=/hiiqos-58/obj00000000000000000061?versionId=null 2023-07-29T00:25:14.841 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000061?versionId=null 2023-07-29T00:25:14.842 DEBUG:boto:Method: GET 2023-07-29T00:25:14.842 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000061?versionId=null 2023-07-29T00:25:14.842 DEBUG:boto:Data: 2023-07-29T00:25:14.843 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.843 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:14.843 DEBUG:boto:Port: 8000 2023-07-29T00:25:14.844 DEBUG:boto:Params: {} 2023-07-29T00:25:14.844 DEBUG:boto:Token: None 2023-07-29T00:25:14.845 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000061?versionId=null 2023-07-29T00:25:14.845 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LJdYX7cuIJoILmhXdcL2n3qOd1I= 2023-07-29T00:25:14.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LJdYX7cuIJoILmhXdcL2n3qOd1I=', 'Content-Length': '0'} 2023-07-29T00:25:14.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c45c64490a09b201-0064c45c6a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.849 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:14.849 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:14.850 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000061 2023-07-29T00:25:14.850 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000061 2023-07-29T00:25:14.851 DEBUG:boto:Method: GET 2023-07-29T00:25:14.851 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000061 2023-07-29T00:25:14.851 DEBUG:boto:Data: 2023-07-29T00:25:14.852 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.852 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:14.853 DEBUG:boto:Port: 8001 2023-07-29T00:25:14.853 DEBUG:boto:Params: {} 2023-07-29T00:25:14.854 DEBUG:boto:Token: None 2023-07-29T00:25:14.854 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/ 2023-07-29T00:25:14.854 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S5R3ASd1HfDELycrH0uGJ4cyYaU= 2023-07-29T00:25:14.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S5R3ASd1HfDELycrH0uGJ4cyYaU=', 'Content-Length': '0'} 2023-07-29T00:25:14.857 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007167562328f48bae-0064c45c6a-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.858 DEBUG:boto:b'hiiqos-58obj000000000000000000611000falseobj000000000000000000612023-07-29T00:24:58.597Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:14.859 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:14.859 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:14.859 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000061 2023-07-29T00:25:14.860 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000061 2023-07-29T00:25:14.860 DEBUG:boto:Method: GET 2023-07-29T00:25:14.861 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000061 2023-07-29T00:25:14.861 DEBUG:boto:Data: 2023-07-29T00:25:14.862 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.862 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:14.862 DEBUG:boto:Port: 8000 2023-07-29T00:25:14.863 DEBUG:boto:Params: {} 2023-07-29T00:25:14.863 DEBUG:boto:Token: None 2023-07-29T00:25:14.864 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/ 2023-07-29T00:25:14.865 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S5R3ASd1HfDELycrH0uGJ4cyYaU= 2023-07-29T00:25:14.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S5R3ASd1HfDELycrH0uGJ4cyYaU=', 'Content-Length': '0'} 2023-07-29T00:25:14.869 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078819e92299e91b3-0064c45c6a-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.870 DEBUG:boto:b'hiiqos-58obj000000000000000000611000falseobj000000000000000000612023-07-29T00:24:58.597Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:14.871 DEBUG:boto:path=/hiiqos-58/obj00000000000000000061 2023-07-29T00:25:14.871 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000061 2023-07-29T00:25:14.872 DEBUG:boto:Method: HEAD 2023-07-29T00:25:14.872 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000061 2023-07-29T00:25:14.873 DEBUG:boto:Data: 2023-07-29T00:25:14.873 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.873 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:14.874 DEBUG:boto:Port: 8001 2023-07-29T00:25:14.874 DEBUG:boto:Params: {} 2023-07-29T00:25:14.875 DEBUG:boto:Token: None 2023-07-29T00:25:14.875 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000061 2023-07-29T00:25:14.875 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OQeQdampEMLIGG+MHX7wGVw4Us4= 2023-07-29T00:25:14.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OQeQdampEMLIGG+MHX7wGVw4Us4=', 'Content-Length': '0'} 2023-07-29T00:25:14.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5da45aa42c2311a-0064c45c6a-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.879 DEBUG:boto:path=/hiiqos-58/obj00000000000000000061 2023-07-29T00:25:14.879 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000061 2023-07-29T00:25:14.879 DEBUG:boto:Method: HEAD 2023-07-29T00:25:14.880 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000061 2023-07-29T00:25:14.880 DEBUG:boto:Data: 2023-07-29T00:25:14.881 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.881 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:14.881 DEBUG:boto:Port: 8000 2023-07-29T00:25:14.882 DEBUG:boto:Params: {} 2023-07-29T00:25:14.882 DEBUG:boto:Token: None 2023-07-29T00:25:14.883 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000061 2023-07-29T00:25:14.883 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OQeQdampEMLIGG+MHX7wGVw4Us4= 2023-07-29T00:25:14.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OQeQdampEMLIGG+MHX7wGVw4Us4=', 'Content-Length': '0'} 2023-07-29T00:25:14.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007dbbdedbfb5ac8ce-0064c45c6a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2023-07-29T00:25:14.886 DEBUG:boto:path=/hiiqos-58/obj00000000000000000061 2023-07-29T00:25:14.887 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000061 2023-07-29T00:25:14.887 DEBUG:boto:Method: GET 2023-07-29T00:25:14.887 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000061 2023-07-29T00:25:14.888 DEBUG:boto:Data: 2023-07-29T00:25:14.888 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.888 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:14.889 DEBUG:boto:Port: 8001 2023-07-29T00:25:14.889 DEBUG:boto:Params: {} 2023-07-29T00:25:14.890 DEBUG:boto:Token: None 2023-07-29T00:25:14.890 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000061 2023-07-29T00:25:14.890 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kK+iJRtJibZMOGyzV2BJhCS0w7o= 2023-07-29T00:25:14.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kK+iJRtJibZMOGyzV2BJhCS0w7o=', 'Content-Length': '0'} 2023-07-29T00:25:14.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008de8b7749b280247-0064c45c6a-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.936 DEBUG:boto:path=/hiiqos-58/obj00000000000000000061 2023-07-29T00:25:14.937 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000061 2023-07-29T00:25:14.938 DEBUG:boto:Method: GET 2023-07-29T00:25:14.939 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000061 2023-07-29T00:25:14.939 DEBUG:boto:Data: 2023-07-29T00:25:14.940 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.941 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:14.942 DEBUG:boto:Port: 8000 2023-07-29T00:25:14.943 DEBUG:boto:Params: {} 2023-07-29T00:25:14.944 DEBUG:boto:Token: None 2023-07-29T00:25:14.945 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000061 2023-07-29T00:25:14.945 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kK+iJRtJibZMOGyzV2BJhCS0w7o= 2023-07-29T00:25:14.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kK+iJRtJibZMOGyzV2BJhCS0w7o=', 'Content-Length': '0'} 2023-07-29T00:25:14.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000972678a5e0de3891-0064c45c6a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.949 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:14.950 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:14.950 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000061 2023-07-29T00:25:14.950 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000061 2023-07-29T00:25:14.951 DEBUG:boto:Method: GET 2023-07-29T00:25:14.951 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000061 2023-07-29T00:25:14.951 DEBUG:boto:Data: 2023-07-29T00:25:14.952 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.952 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:14.953 DEBUG:boto:Port: 8001 2023-07-29T00:25:14.953 DEBUG:boto:Params: {} 2023-07-29T00:25:14.954 DEBUG:boto:Token: None 2023-07-29T00:25:14.955 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/ 2023-07-29T00:25:14.955 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S5R3ASd1HfDELycrH0uGJ4cyYaU= 2023-07-29T00:25:14.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S5R3ASd1HfDELycrH0uGJ4cyYaU=', 'Content-Length': '0'} 2023-07-29T00:25:14.958 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000276cd1e594ddc893-0064c45c6a-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.959 DEBUG:boto:b'hiiqos-58obj000000000000000000611000falseobj000000000000000000612023-07-29T00:24:58.597Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:14.959 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:14.960 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:14.961 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000061 2023-07-29T00:25:14.962 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000061 2023-07-29T00:25:14.963 DEBUG:boto:Method: GET 2023-07-29T00:25:14.964 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000061 2023-07-29T00:25:14.965 DEBUG:boto:Data: 2023-07-29T00:25:14.965 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.966 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:14.967 DEBUG:boto:Port: 8000 2023-07-29T00:25:14.968 DEBUG:boto:Params: {} 2023-07-29T00:25:14.969 DEBUG:boto:Token: None 2023-07-29T00:25:14.969 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/ 2023-07-29T00:25:14.970 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S5R3ASd1HfDELycrH0uGJ4cyYaU= 2023-07-29T00:25:14.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S5R3ASd1HfDELycrH0uGJ4cyYaU=', 'Content-Length': '0'} 2023-07-29T00:25:14.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1f4fd598e797038-0064c45c6a-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.976 DEBUG:boto:b'hiiqos-58obj000000000000000000611000falseobj000000000000000000612023-07-29T00:24:58.597Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:14.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2023-07-29T00:25:14.977 DEBUG:boto:path=/hiiqos-58/obj00000000000000000062 2023-07-29T00:25:14.977 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000062 2023-07-29T00:25:14.977 DEBUG:boto:path=/hiiqos-58/obj00000000000000000062?versionId=null 2023-07-29T00:25:14.978 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000062?versionId=null 2023-07-29T00:25:14.978 DEBUG:boto:Method: GET 2023-07-29T00:25:14.979 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000062?versionId=null 2023-07-29T00:25:14.979 DEBUG:boto:Data: 2023-07-29T00:25:14.979 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.980 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:14.980 DEBUG:boto:Port: 8001 2023-07-29T00:25:14.980 DEBUG:boto:Params: {} 2023-07-29T00:25:14.981 DEBUG:boto:Token: None 2023-07-29T00:25:14.981 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000062?versionId=null 2023-07-29T00:25:14.982 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dru9H66OG55OFJAprnNkXK0sF7U= 2023-07-29T00:25:14.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dru9H66OG55OFJAprnNkXK0sF7U=', 'Content-Length': '0'} 2023-07-29T00:25:14.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006843f1d6e866a057-0064c45c6a-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.985 DEBUG:boto:path=/hiiqos-58/obj00000000000000000062 2023-07-29T00:25:14.985 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000062 2023-07-29T00:25:14.986 DEBUG:boto:path=/hiiqos-58/obj00000000000000000062?versionId=null 2023-07-29T00:25:14.986 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000062?versionId=null 2023-07-29T00:25:14.986 DEBUG:boto:Method: GET 2023-07-29T00:25:14.987 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000062?versionId=null 2023-07-29T00:25:14.987 DEBUG:boto:Data: 2023-07-29T00:25:14.988 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.988 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:14.988 DEBUG:boto:Port: 8000 2023-07-29T00:25:14.989 DEBUG:boto:Params: {} 2023-07-29T00:25:14.989 DEBUG:boto:Token: None 2023-07-29T00:25:14.989 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000062?versionId=null 2023-07-29T00:25:14.990 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dru9H66OG55OFJAprnNkXK0sF7U= 2023-07-29T00:25:14.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dru9H66OG55OFJAprnNkXK0sF7U=', 'Content-Length': '0'} 2023-07-29T00:25:14.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076f1aa018bdbab2e-0064c45c6a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:14.993 DEBUG:boto:path=/hiiqos-58/obj00000000000000000062 2023-07-29T00:25:14.993 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000062 2023-07-29T00:25:14.994 DEBUG:boto:path=/hiiqos-58/obj00000000000000000062?versionId=null 2023-07-29T00:25:14.994 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000062?versionId=null 2023-07-29T00:25:14.994 DEBUG:boto:Method: HEAD 2023-07-29T00:25:14.995 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000062?versionId=null 2023-07-29T00:25:14.995 DEBUG:boto:Data: 2023-07-29T00:25:14.995 DEBUG:boto:Headers: {} 2023-07-29T00:25:14.996 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:14.996 DEBUG:boto:Port: 8001 2023-07-29T00:25:14.996 DEBUG:boto:Params: {} 2023-07-29T00:25:14.997 DEBUG:boto:Token: None 2023-07-29T00:25:14.997 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:14 GMT /hiiqos-58/obj00000000000000000062?versionId=null 2023-07-29T00:25:14.998 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sti2cbeEQChLkNM4OWNj1GTJAYw= 2023-07-29T00:25:14.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:14 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sti2cbeEQChLkNM4OWNj1GTJAYw=', 'Content-Length': '0'} 2023-07-29T00:25:15.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007043ace5150ae497-0064c45c6a-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.001 DEBUG:boto:path=/hiiqos-58/obj00000000000000000062 2023-07-29T00:25:15.001 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000062 2023-07-29T00:25:15.001 DEBUG:boto:path=/hiiqos-58/obj00000000000000000062?versionId=null 2023-07-29T00:25:15.002 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000062?versionId=null 2023-07-29T00:25:15.002 DEBUG:boto:Method: HEAD 2023-07-29T00:25:15.002 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000062?versionId=null 2023-07-29T00:25:15.003 DEBUG:boto:Data: 2023-07-29T00:25:15.003 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.004 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:15.004 DEBUG:boto:Port: 8000 2023-07-29T00:25:15.004 DEBUG:boto:Params: {} 2023-07-29T00:25:15.005 DEBUG:boto:Token: None 2023-07-29T00:25:15.006 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/obj00000000000000000062?versionId=null 2023-07-29T00:25:15.006 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3pAVJuI7G5iZxo4Rq/e1YQuVkYU= 2023-07-29T00:25:15.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3pAVJuI7G5iZxo4Rq/e1YQuVkYU=', 'Content-Length': '0'} 2023-07-29T00:25:15.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009531ae940cb4975b-0064c45c6a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2023-07-29T00:25:15.010 DEBUG:boto:path=/hiiqos-58/obj00000000000000000062 2023-07-29T00:25:15.010 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000062 2023-07-29T00:25:15.011 DEBUG:boto:path=/hiiqos-58/obj00000000000000000062?versionId=null 2023-07-29T00:25:15.011 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000062?versionId=null 2023-07-29T00:25:15.012 DEBUG:boto:Method: GET 2023-07-29T00:25:15.012 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000062?versionId=null 2023-07-29T00:25:15.013 DEBUG:boto:Data: 2023-07-29T00:25:15.013 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.014 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:15.014 DEBUG:boto:Port: 8001 2023-07-29T00:25:15.015 DEBUG:boto:Params: {} 2023-07-29T00:25:15.015 DEBUG:boto:Token: None 2023-07-29T00:25:15.016 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/obj00000000000000000062?versionId=null 2023-07-29T00:25:15.017 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Yyn0HorHKIg5TmtFTc57amOg/4g= 2023-07-29T00:25:15.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Yyn0HorHKIg5TmtFTc57amOg/4g=', 'Content-Length': '0'} 2023-07-29T00:25:15.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6a4db2f07d4274d-0064c45c6a-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.020 DEBUG:boto:path=/hiiqos-58/obj00000000000000000062 2023-07-29T00:25:15.020 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000062 2023-07-29T00:25:15.021 DEBUG:boto:path=/hiiqos-58/obj00000000000000000062?versionId=null 2023-07-29T00:25:15.021 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000062?versionId=null 2023-07-29T00:25:15.022 DEBUG:boto:Method: GET 2023-07-29T00:25:15.022 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000062?versionId=null 2023-07-29T00:25:15.023 DEBUG:boto:Data: 2023-07-29T00:25:15.023 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.024 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:15.024 DEBUG:boto:Port: 8000 2023-07-29T00:25:15.025 DEBUG:boto:Params: {} 2023-07-29T00:25:15.026 DEBUG:boto:Token: None 2023-07-29T00:25:15.027 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/obj00000000000000000062?versionId=null 2023-07-29T00:25:15.028 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Yyn0HorHKIg5TmtFTc57amOg/4g= 2023-07-29T00:25:15.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Yyn0HorHKIg5TmtFTc57amOg/4g=', 'Content-Length': '0'} 2023-07-29T00:25:15.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039ce5e0f4fc9dd92-0064c45c6a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:14 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.072 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:15.073 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:15.073 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000062 2023-07-29T00:25:15.074 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000062 2023-07-29T00:25:15.074 DEBUG:boto:Method: GET 2023-07-29T00:25:15.075 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000062 2023-07-29T00:25:15.075 DEBUG:boto:Data: 2023-07-29T00:25:15.076 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.076 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:15.076 DEBUG:boto:Port: 8001 2023-07-29T00:25:15.077 DEBUG:boto:Params: {} 2023-07-29T00:25:15.077 DEBUG:boto:Token: None 2023-07-29T00:25:15.078 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/ 2023-07-29T00:25:15.079 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P5MaecRSkISG+pYr7xzhbV+KLto= 2023-07-29T00:25:15.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P5MaecRSkISG+pYr7xzhbV+KLto=', 'Content-Length': '0'} 2023-07-29T00:25:15.083 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca4a4e11ff712730-0064c45c6b-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.084 DEBUG:boto:b'hiiqos-58obj000000000000000000621000falseobj000000000000000000622023-07-29T00:25:00.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:15.085 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:15.085 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:15.086 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000062 2023-07-29T00:25:15.086 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000062 2023-07-29T00:25:15.086 DEBUG:boto:Method: GET 2023-07-29T00:25:15.087 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000062 2023-07-29T00:25:15.087 DEBUG:boto:Data: 2023-07-29T00:25:15.088 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.088 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:15.088 DEBUG:boto:Port: 8000 2023-07-29T00:25:15.089 DEBUG:boto:Params: {} 2023-07-29T00:25:15.089 DEBUG:boto:Token: None 2023-07-29T00:25:15.090 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/ 2023-07-29T00:25:15.090 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P5MaecRSkISG+pYr7xzhbV+KLto= 2023-07-29T00:25:15.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P5MaecRSkISG+pYr7xzhbV+KLto=', 'Content-Length': '0'} 2023-07-29T00:25:15.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d158a724e12ebab-0064c45c6b-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.096 DEBUG:boto:b'hiiqos-58obj000000000000000000621000falseobj000000000000000000622023-07-29T00:25:00.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:15.097 DEBUG:boto:path=/hiiqos-58/obj00000000000000000062 2023-07-29T00:25:15.097 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000062 2023-07-29T00:25:15.098 DEBUG:boto:Method: HEAD 2023-07-29T00:25:15.098 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000062 2023-07-29T00:25:15.098 DEBUG:boto:Data: 2023-07-29T00:25:15.099 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.099 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:15.099 DEBUG:boto:Port: 8001 2023-07-29T00:25:15.100 DEBUG:boto:Params: {} 2023-07-29T00:25:15.100 DEBUG:boto:Token: None 2023-07-29T00:25:15.101 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/obj00000000000000000062 2023-07-29T00:25:15.101 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6tjzddAYHQmjLcClD+4oGfIcJi4= 2023-07-29T00:25:15.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6tjzddAYHQmjLcClD+4oGfIcJi4=', 'Content-Length': '0'} 2023-07-29T00:25:15.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e93df0611dece06b-0064c45c6b-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.105 DEBUG:boto:path=/hiiqos-58/obj00000000000000000062 2023-07-29T00:25:15.106 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000062 2023-07-29T00:25:15.107 DEBUG:boto:Method: HEAD 2023-07-29T00:25:15.108 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000062 2023-07-29T00:25:15.109 DEBUG:boto:Data: 2023-07-29T00:25:15.110 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.111 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:15.111 DEBUG:boto:Port: 8000 2023-07-29T00:25:15.112 DEBUG:boto:Params: {} 2023-07-29T00:25:15.113 DEBUG:boto:Token: None 2023-07-29T00:25:15.113 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/obj00000000000000000062 2023-07-29T00:25:15.114 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6tjzddAYHQmjLcClD+4oGfIcJi4= 2023-07-29T00:25:15.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6tjzddAYHQmjLcClD+4oGfIcJi4=', 'Content-Length': '0'} 2023-07-29T00:25:15.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f37538275ba9ef3-0064c45c6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2023-07-29T00:25:15.119 DEBUG:boto:path=/hiiqos-58/obj00000000000000000062 2023-07-29T00:25:15.119 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000062 2023-07-29T00:25:15.119 DEBUG:boto:Method: GET 2023-07-29T00:25:15.120 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000062 2023-07-29T00:25:15.120 DEBUG:boto:Data: 2023-07-29T00:25:15.121 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.121 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:15.121 DEBUG:boto:Port: 8001 2023-07-29T00:25:15.122 DEBUG:boto:Params: {} 2023-07-29T00:25:15.122 DEBUG:boto:Token: None 2023-07-29T00:25:15.123 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/obj00000000000000000062 2023-07-29T00:25:15.124 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hgLZOF6j1YAlSJteYMxjPKtBsOU= 2023-07-29T00:25:15.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hgLZOF6j1YAlSJteYMxjPKtBsOU=', 'Content-Length': '0'} 2023-07-29T00:25:15.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e462aba3825e0429-0064c45c6b-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.127 DEBUG:boto:path=/hiiqos-58/obj00000000000000000062 2023-07-29T00:25:15.127 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000062 2023-07-29T00:25:15.127 DEBUG:boto:Method: GET 2023-07-29T00:25:15.128 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000062 2023-07-29T00:25:15.128 DEBUG:boto:Data: 2023-07-29T00:25:15.129 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.129 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:15.129 DEBUG:boto:Port: 8000 2023-07-29T00:25:15.130 DEBUG:boto:Params: {} 2023-07-29T00:25:15.130 DEBUG:boto:Token: None 2023-07-29T00:25:15.131 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/obj00000000000000000062 2023-07-29T00:25:15.131 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hgLZOF6j1YAlSJteYMxjPKtBsOU= 2023-07-29T00:25:15.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hgLZOF6j1YAlSJteYMxjPKtBsOU=', 'Content-Length': '0'} 2023-07-29T00:25:15.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:25:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009348bdfacc40be2c-0064c45c6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.176 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:15.177 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:15.178 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000062 2023-07-29T00:25:15.179 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000062 2023-07-29T00:25:15.179 DEBUG:boto:Method: GET 2023-07-29T00:25:15.180 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000062 2023-07-29T00:25:15.181 DEBUG:boto:Data: 2023-07-29T00:25:15.182 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.182 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:15.183 DEBUG:boto:Port: 8001 2023-07-29T00:25:15.184 DEBUG:boto:Params: {} 2023-07-29T00:25:15.184 DEBUG:boto:Token: None 2023-07-29T00:25:15.185 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/ 2023-07-29T00:25:15.186 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P5MaecRSkISG+pYr7xzhbV+KLto= 2023-07-29T00:25:15.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P5MaecRSkISG+pYr7xzhbV+KLto=', 'Content-Length': '0'} 2023-07-29T00:25:15.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d2b952e3bf257b3-0064c45c6b-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.191 DEBUG:boto:b'hiiqos-58obj000000000000000000621000falseobj000000000000000000622023-07-29T00:25:00.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:15.192 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:15.193 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:15.194 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000062 2023-07-29T00:25:15.195 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000062 2023-07-29T00:25:15.195 DEBUG:boto:Method: GET 2023-07-29T00:25:15.196 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000062 2023-07-29T00:25:15.197 DEBUG:boto:Data: 2023-07-29T00:25:15.198 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.198 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:15.199 DEBUG:boto:Port: 8000 2023-07-29T00:25:15.199 DEBUG:boto:Params: {} 2023-07-29T00:25:15.200 DEBUG:boto:Token: None 2023-07-29T00:25:15.201 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/ 2023-07-29T00:25:15.202 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P5MaecRSkISG+pYr7xzhbV+KLto= 2023-07-29T00:25:15.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P5MaecRSkISG+pYr7xzhbV+KLto=', 'Content-Length': '0'} 2023-07-29T00:25:15.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001743cffc6d08ae00-0064c45c6b-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.208 DEBUG:boto:b'hiiqos-58obj000000000000000000621000falseobj000000000000000000622023-07-29T00:25:00.281Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:15.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2023-07-29T00:25:15.209 DEBUG:boto:path=/hiiqos-58/obj00000000000000000065 2023-07-29T00:25:15.209 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000065 2023-07-29T00:25:15.210 DEBUG:boto:path=/hiiqos-58/obj00000000000000000065?versionId=null 2023-07-29T00:25:15.210 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000065?versionId=null 2023-07-29T00:25:15.211 DEBUG:boto:Method: GET 2023-07-29T00:25:15.211 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000065?versionId=null 2023-07-29T00:25:15.211 DEBUG:boto:Data: 2023-07-29T00:25:15.212 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.212 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:15.212 DEBUG:boto:Port: 8001 2023-07-29T00:25:15.213 DEBUG:boto:Params: {} 2023-07-29T00:25:15.213 DEBUG:boto:Token: None 2023-07-29T00:25:15.214 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/obj00000000000000000065?versionId=null 2023-07-29T00:25:15.214 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tMUQ5aRrF4TwflsKCL7whd81/ms= 2023-07-29T00:25:15.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tMUQ5aRrF4TwflsKCL7whd81/ms=', 'Content-Length': '0'} 2023-07-29T00:25:15.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000394c48ba9947825-0064c45c6b-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.260 DEBUG:boto:path=/hiiqos-58/obj00000000000000000065 2023-07-29T00:25:15.261 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000065 2023-07-29T00:25:15.262 DEBUG:boto:path=/hiiqos-58/obj00000000000000000065?versionId=null 2023-07-29T00:25:15.262 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000065?versionId=null 2023-07-29T00:25:15.263 DEBUG:boto:Method: GET 2023-07-29T00:25:15.263 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000065?versionId=null 2023-07-29T00:25:15.263 DEBUG:boto:Data: 2023-07-29T00:25:15.264 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.264 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:15.265 DEBUG:boto:Port: 8000 2023-07-29T00:25:15.265 DEBUG:boto:Params: {} 2023-07-29T00:25:15.266 DEBUG:boto:Token: None 2023-07-29T00:25:15.267 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/obj00000000000000000065?versionId=null 2023-07-29T00:25:15.268 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tMUQ5aRrF4TwflsKCL7whd81/ms= 2023-07-29T00:25:15.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tMUQ5aRrF4TwflsKCL7whd81/ms=', 'Content-Length': '0'} 2023-07-29T00:25:15.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055341c184b720693-0064c45c6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.273 DEBUG:boto:path=/hiiqos-58/obj00000000000000000065 2023-07-29T00:25:15.274 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000065 2023-07-29T00:25:15.274 DEBUG:boto:path=/hiiqos-58/obj00000000000000000065?versionId=null 2023-07-29T00:25:15.275 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000065?versionId=null 2023-07-29T00:25:15.275 DEBUG:boto:Method: HEAD 2023-07-29T00:25:15.277 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000065?versionId=null 2023-07-29T00:25:15.278 DEBUG:boto:Data: 2023-07-29T00:25:15.278 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.279 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:15.281 DEBUG:boto:Port: 8001 2023-07-29T00:25:15.282 DEBUG:boto:Params: {} 2023-07-29T00:25:15.282 DEBUG:boto:Token: None 2023-07-29T00:25:15.283 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/obj00000000000000000065?versionId=null 2023-07-29T00:25:15.284 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bE/ZJjy5I1cbEPNJd7Gr9noOfZw= 2023-07-29T00:25:15.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bE/ZJjy5I1cbEPNJd7Gr9noOfZw=', 'Content-Length': '0'} 2023-07-29T00:25:15.288 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cd46b05882b1f4f-0064c45c6b-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.289 DEBUG:boto:path=/hiiqos-58/obj00000000000000000065 2023-07-29T00:25:15.290 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000065 2023-07-29T00:25:15.290 DEBUG:boto:path=/hiiqos-58/obj00000000000000000065?versionId=null 2023-07-29T00:25:15.291 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000065?versionId=null 2023-07-29T00:25:15.292 DEBUG:boto:Method: HEAD 2023-07-29T00:25:15.292 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000065?versionId=null 2023-07-29T00:25:15.293 DEBUG:boto:Data: 2023-07-29T00:25:15.294 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.294 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:15.295 DEBUG:boto:Port: 8000 2023-07-29T00:25:15.296 DEBUG:boto:Params: {} 2023-07-29T00:25:15.296 DEBUG:boto:Token: None 2023-07-29T00:25:15.297 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/obj00000000000000000065?versionId=null 2023-07-29T00:25:15.298 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bE/ZJjy5I1cbEPNJd7Gr9noOfZw= 2023-07-29T00:25:15.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bE/ZJjy5I1cbEPNJd7Gr9noOfZw=', 'Content-Length': '0'} 2023-07-29T00:25:15.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013ee61853ecfd72f-0064c45c6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2023-07-29T00:25:15.302 DEBUG:boto:path=/hiiqos-58/obj00000000000000000065 2023-07-29T00:25:15.303 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000065 2023-07-29T00:25:15.304 DEBUG:boto:path=/hiiqos-58/obj00000000000000000065?versionId=null 2023-07-29T00:25:15.305 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000065?versionId=null 2023-07-29T00:25:15.305 DEBUG:boto:Method: GET 2023-07-29T00:25:15.306 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000065?versionId=null 2023-07-29T00:25:15.306 DEBUG:boto:Data: 2023-07-29T00:25:15.307 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.307 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:15.308 DEBUG:boto:Port: 8001 2023-07-29T00:25:15.308 DEBUG:boto:Params: {} 2023-07-29T00:25:15.309 DEBUG:boto:Token: None 2023-07-29T00:25:15.309 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/obj00000000000000000065?versionId=null 2023-07-29T00:25:15.310 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tMUQ5aRrF4TwflsKCL7whd81/ms= 2023-07-29T00:25:15.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tMUQ5aRrF4TwflsKCL7whd81/ms=', 'Content-Length': '0'} 2023-07-29T00:25:15.313 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ec9c536063efb05-0064c45c6b-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.314 DEBUG:boto:path=/hiiqos-58/obj00000000000000000065 2023-07-29T00:25:15.314 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000065 2023-07-29T00:25:15.315 DEBUG:boto:path=/hiiqos-58/obj00000000000000000065?versionId=null 2023-07-29T00:25:15.315 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000065?versionId=null 2023-07-29T00:25:15.315 DEBUG:boto:Method: GET 2023-07-29T00:25:15.316 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000065?versionId=null 2023-07-29T00:25:15.316 DEBUG:boto:Data: 2023-07-29T00:25:15.316 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.317 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:15.317 DEBUG:boto:Port: 8000 2023-07-29T00:25:15.317 DEBUG:boto:Params: {} 2023-07-29T00:25:15.318 DEBUG:boto:Token: None 2023-07-29T00:25:15.318 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/obj00000000000000000065?versionId=null 2023-07-29T00:25:15.319 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tMUQ5aRrF4TwflsKCL7whd81/ms= 2023-07-29T00:25:15.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tMUQ5aRrF4TwflsKCL7whd81/ms=', 'Content-Length': '0'} 2023-07-29T00:25:15.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8e74fe0f0c0989a-0064c45c6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.322 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:15.323 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:15.323 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000065 2023-07-29T00:25:15.324 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000065 2023-07-29T00:25:15.324 DEBUG:boto:Method: GET 2023-07-29T00:25:15.324 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000065 2023-07-29T00:25:15.325 DEBUG:boto:Data: 2023-07-29T00:25:15.325 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.325 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:15.326 DEBUG:boto:Port: 8001 2023-07-29T00:25:15.326 DEBUG:boto:Params: {} 2023-07-29T00:25:15.327 DEBUG:boto:Token: None 2023-07-29T00:25:15.327 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/ 2023-07-29T00:25:15.328 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P5MaecRSkISG+pYr7xzhbV+KLto= 2023-07-29T00:25:15.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P5MaecRSkISG+pYr7xzhbV+KLto=', 'Content-Length': '0'} 2023-07-29T00:25:15.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003866026480b6dbb2-0064c45c6b-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.331 DEBUG:boto:b'hiiqos-58obj000000000000000000651000falseobj000000000000000000652023-07-29T00:24:59.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:15.332 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:15.332 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:15.333 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000065 2023-07-29T00:25:15.333 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000065 2023-07-29T00:25:15.333 DEBUG:boto:Method: GET 2023-07-29T00:25:15.334 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000065 2023-07-29T00:25:15.334 DEBUG:boto:Data: 2023-07-29T00:25:15.334 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.335 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:15.335 DEBUG:boto:Port: 8000 2023-07-29T00:25:15.335 DEBUG:boto:Params: {} 2023-07-29T00:25:15.336 DEBUG:boto:Token: None 2023-07-29T00:25:15.336 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/ 2023-07-29T00:25:15.337 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P5MaecRSkISG+pYr7xzhbV+KLto= 2023-07-29T00:25:15.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P5MaecRSkISG+pYr7xzhbV+KLto=', 'Content-Length': '0'} 2023-07-29T00:25:15.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fd051f2cc28721e-0064c45c6b-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.343 DEBUG:boto:b'hiiqos-58obj000000000000000000651000falseobj000000000000000000652023-07-29T00:24:59.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:15.344 DEBUG:boto:path=/hiiqos-58/obj00000000000000000065 2023-07-29T00:25:15.344 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000065 2023-07-29T00:25:15.345 DEBUG:boto:Method: HEAD 2023-07-29T00:25:15.346 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000065 2023-07-29T00:25:15.346 DEBUG:boto:Data: 2023-07-29T00:25:15.347 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.347 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:15.348 DEBUG:boto:Port: 8001 2023-07-29T00:25:15.348 DEBUG:boto:Params: {} 2023-07-29T00:25:15.348 DEBUG:boto:Token: None 2023-07-29T00:25:15.349 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/obj00000000000000000065 2023-07-29T00:25:15.349 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ab0QjcRBjfA9/2YsZ1334enEZCk= 2023-07-29T00:25:15.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ab0QjcRBjfA9/2YsZ1334enEZCk=', 'Content-Length': '0'} 2023-07-29T00:25:15.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5ca818c2cd8265e-0064c45c6b-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.352 DEBUG:boto:path=/hiiqos-58/obj00000000000000000065 2023-07-29T00:25:15.353 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000065 2023-07-29T00:25:15.353 DEBUG:boto:Method: HEAD 2023-07-29T00:25:15.353 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000065 2023-07-29T00:25:15.354 DEBUG:boto:Data: 2023-07-29T00:25:15.354 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.354 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:15.355 DEBUG:boto:Port: 8000 2023-07-29T00:25:15.355 DEBUG:boto:Params: {} 2023-07-29T00:25:15.355 DEBUG:boto:Token: None 2023-07-29T00:25:15.356 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/obj00000000000000000065 2023-07-29T00:25:15.356 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ab0QjcRBjfA9/2YsZ1334enEZCk= 2023-07-29T00:25:15.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ab0QjcRBjfA9/2YsZ1334enEZCk=', 'Content-Length': '0'} 2023-07-29T00:25:15.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a54d889275667e9-0064c45c6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2023-07-29T00:25:15.361 DEBUG:boto:path=/hiiqos-58/obj00000000000000000065 2023-07-29T00:25:15.362 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000065 2023-07-29T00:25:15.363 DEBUG:boto:Method: GET 2023-07-29T00:25:15.364 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000065 2023-07-29T00:25:15.365 DEBUG:boto:Data: 2023-07-29T00:25:15.365 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.368 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:15.368 DEBUG:boto:Port: 8001 2023-07-29T00:25:15.369 DEBUG:boto:Params: {} 2023-07-29T00:25:15.370 DEBUG:boto:Token: None 2023-07-29T00:25:15.370 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/obj00000000000000000065 2023-07-29T00:25:15.371 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HwUXAXKPFtrEQLdLf6GNB0o8kIk= 2023-07-29T00:25:15.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HwUXAXKPFtrEQLdLf6GNB0o8kIk=', 'Content-Length': '0'} 2023-07-29T00:25:15.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b0a005b4f150576-0064c45c6b-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.376 DEBUG:boto:path=/hiiqos-58/obj00000000000000000065 2023-07-29T00:25:15.377 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000065 2023-07-29T00:25:15.378 DEBUG:boto:Method: GET 2023-07-29T00:25:15.379 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000065 2023-07-29T00:25:15.380 DEBUG:boto:Data: 2023-07-29T00:25:15.380 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.381 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:15.382 DEBUG:boto:Port: 8000 2023-07-29T00:25:15.382 DEBUG:boto:Params: {} 2023-07-29T00:25:15.383 DEBUG:boto:Token: None 2023-07-29T00:25:15.384 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/obj00000000000000000065 2023-07-29T00:25:15.384 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HwUXAXKPFtrEQLdLf6GNB0o8kIk= 2023-07-29T00:25:15.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HwUXAXKPFtrEQLdLf6GNB0o8kIk=', 'Content-Length': '0'} 2023-07-29T00:25:15.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004119522f466117f4-0064c45c6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.428 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:15.430 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:15.430 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000065 2023-07-29T00:25:15.431 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000065 2023-07-29T00:25:15.432 DEBUG:boto:Method: GET 2023-07-29T00:25:15.433 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000065 2023-07-29T00:25:15.433 DEBUG:boto:Data: 2023-07-29T00:25:15.434 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.436 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:15.436 DEBUG:boto:Port: 8001 2023-07-29T00:25:15.437 DEBUG:boto:Params: {} 2023-07-29T00:25:15.438 DEBUG:boto:Token: None 2023-07-29T00:25:15.439 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/ 2023-07-29T00:25:15.440 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P5MaecRSkISG+pYr7xzhbV+KLto= 2023-07-29T00:25:15.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P5MaecRSkISG+pYr7xzhbV+KLto=', 'Content-Length': '0'} 2023-07-29T00:25:15.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c829e8e5ce4e20a4-0064c45c6b-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.445 DEBUG:boto:b'hiiqos-58obj000000000000000000651000falseobj000000000000000000652023-07-29T00:24:59.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:15.445 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:15.446 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:15.446 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000065 2023-07-29T00:25:15.447 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000065 2023-07-29T00:25:15.447 DEBUG:boto:Method: GET 2023-07-29T00:25:15.448 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000065 2023-07-29T00:25:15.448 DEBUG:boto:Data: 2023-07-29T00:25:15.449 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.449 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:15.450 DEBUG:boto:Port: 8000 2023-07-29T00:25:15.450 DEBUG:boto:Params: {} 2023-07-29T00:25:15.451 DEBUG:boto:Token: None 2023-07-29T00:25:15.451 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/ 2023-07-29T00:25:15.451 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P5MaecRSkISG+pYr7xzhbV+KLto= 2023-07-29T00:25:15.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P5MaecRSkISG+pYr7xzhbV+KLto=', 'Content-Length': '0'} 2023-07-29T00:25:15.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d7e0d7327526e33-0064c45c6b-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.458 DEBUG:boto:b'hiiqos-58obj000000000000000000651000falseobj000000000000000000652023-07-29T00:24:59.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:15.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000076 2023-07-29T00:25:15.460 DEBUG:boto:path=/hiiqos-58/obj00000000000000000076 2023-07-29T00:25:15.461 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000076 2023-07-29T00:25:15.462 DEBUG:boto:path=/hiiqos-58/obj00000000000000000076?versionId=null 2023-07-29T00:25:15.462 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000076?versionId=null 2023-07-29T00:25:15.463 DEBUG:boto:Method: GET 2023-07-29T00:25:15.464 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000076?versionId=null 2023-07-29T00:25:15.465 DEBUG:boto:Data: 2023-07-29T00:25:15.466 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.467 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:15.468 DEBUG:boto:Port: 8001 2023-07-29T00:25:15.469 DEBUG:boto:Params: {} 2023-07-29T00:25:15.470 DEBUG:boto:Token: None 2023-07-29T00:25:15.471 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/obj00000000000000000076?versionId=null 2023-07-29T00:25:15.472 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CDT1Yv98CmSfNNPo4u0kHdUSD30= 2023-07-29T00:25:15.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CDT1Yv98CmSfNNPo4u0kHdUSD30=', 'Content-Length': '0'} 2023-07-29T00:25:15.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5574b1296a8f20b-0064c45c6b-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.477 DEBUG:boto:path=/hiiqos-58/obj00000000000000000076 2023-07-29T00:25:15.477 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000076 2023-07-29T00:25:15.478 DEBUG:boto:path=/hiiqos-58/obj00000000000000000076?versionId=null 2023-07-29T00:25:15.478 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000076?versionId=null 2023-07-29T00:25:15.478 DEBUG:boto:Method: GET 2023-07-29T00:25:15.479 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000076?versionId=null 2023-07-29T00:25:15.480 DEBUG:boto:Data: 2023-07-29T00:25:15.480 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.480 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:15.481 DEBUG:boto:Port: 8000 2023-07-29T00:25:15.481 DEBUG:boto:Params: {} 2023-07-29T00:25:15.481 DEBUG:boto:Token: None 2023-07-29T00:25:15.482 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/obj00000000000000000076?versionId=null 2023-07-29T00:25:15.482 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CDT1Yv98CmSfNNPo4u0kHdUSD30= 2023-07-29T00:25:15.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CDT1Yv98CmSfNNPo4u0kHdUSD30=', 'Content-Length': '0'} 2023-07-29T00:25:15.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059fbcf8fde021285-0064c45c6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.485 DEBUG:boto:path=/hiiqos-58/obj00000000000000000076 2023-07-29T00:25:15.486 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000076 2023-07-29T00:25:15.487 DEBUG:boto:path=/hiiqos-58/obj00000000000000000076?versionId=null 2023-07-29T00:25:15.487 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000076?versionId=null 2023-07-29T00:25:15.488 DEBUG:boto:Method: HEAD 2023-07-29T00:25:15.488 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000076?versionId=null 2023-07-29T00:25:15.489 DEBUG:boto:Data: 2023-07-29T00:25:15.489 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.490 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:15.490 DEBUG:boto:Port: 8001 2023-07-29T00:25:15.491 DEBUG:boto:Params: {} 2023-07-29T00:25:15.491 DEBUG:boto:Token: None 2023-07-29T00:25:15.491 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/obj00000000000000000076?versionId=null 2023-07-29T00:25:15.492 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RheDCsHJ+mmWWmxVp87npYRPL3U= 2023-07-29T00:25:15.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RheDCsHJ+mmWWmxVp87npYRPL3U=', 'Content-Length': '0'} 2023-07-29T00:25:15.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae8ad39afe6c3c07-0064c45c6b-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.495 DEBUG:boto:path=/hiiqos-58/obj00000000000000000076 2023-07-29T00:25:15.495 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000076 2023-07-29T00:25:15.496 DEBUG:boto:path=/hiiqos-58/obj00000000000000000076?versionId=null 2023-07-29T00:25:15.496 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000076?versionId=null 2023-07-29T00:25:15.496 DEBUG:boto:Method: HEAD 2023-07-29T00:25:15.497 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000076?versionId=null 2023-07-29T00:25:15.497 DEBUG:boto:Data: 2023-07-29T00:25:15.498 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.498 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:15.499 DEBUG:boto:Port: 8000 2023-07-29T00:25:15.500 DEBUG:boto:Params: {} 2023-07-29T00:25:15.500 DEBUG:boto:Token: None 2023-07-29T00:25:15.501 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/obj00000000000000000076?versionId=null 2023-07-29T00:25:15.502 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RheDCsHJ+mmWWmxVp87npYRPL3U= 2023-07-29T00:25:15.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RheDCsHJ+mmWWmxVp87npYRPL3U=', 'Content-Length': '0'} 2023-07-29T00:25:15.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7786ae987871cf7-0064c45c6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000076 2023-07-29T00:25:15.508 DEBUG:boto:path=/hiiqos-58/obj00000000000000000076 2023-07-29T00:25:15.509 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000076 2023-07-29T00:25:15.510 DEBUG:boto:path=/hiiqos-58/obj00000000000000000076?versionId=null 2023-07-29T00:25:15.511 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000076?versionId=null 2023-07-29T00:25:15.512 DEBUG:boto:Method: GET 2023-07-29T00:25:15.512 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000076?versionId=null 2023-07-29T00:25:15.513 DEBUG:boto:Data: 2023-07-29T00:25:15.514 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.515 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:15.516 DEBUG:boto:Port: 8001 2023-07-29T00:25:15.516 DEBUG:boto:Params: {} 2023-07-29T00:25:15.517 DEBUG:boto:Token: None 2023-07-29T00:25:15.517 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/obj00000000000000000076?versionId=null 2023-07-29T00:25:15.518 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CDT1Yv98CmSfNNPo4u0kHdUSD30= 2023-07-29T00:25:15.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CDT1Yv98CmSfNNPo4u0kHdUSD30=', 'Content-Length': '0'} 2023-07-29T00:25:15.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7011fc679999260-0064c45c6b-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.521 DEBUG:boto:path=/hiiqos-58/obj00000000000000000076 2023-07-29T00:25:15.521 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000076 2023-07-29T00:25:15.522 DEBUG:boto:path=/hiiqos-58/obj00000000000000000076?versionId=null 2023-07-29T00:25:15.522 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000076?versionId=null 2023-07-29T00:25:15.523 DEBUG:boto:Method: GET 2023-07-29T00:25:15.523 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000076?versionId=null 2023-07-29T00:25:15.523 DEBUG:boto:Data: 2023-07-29T00:25:15.524 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.524 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:15.524 DEBUG:boto:Port: 8000 2023-07-29T00:25:15.525 DEBUG:boto:Params: {} 2023-07-29T00:25:15.525 DEBUG:boto:Token: None 2023-07-29T00:25:15.526 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/obj00000000000000000076?versionId=null 2023-07-29T00:25:15.526 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CDT1Yv98CmSfNNPo4u0kHdUSD30= 2023-07-29T00:25:15.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CDT1Yv98CmSfNNPo4u0kHdUSD30=', 'Content-Length': '0'} 2023-07-29T00:25:15.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a447588ffef60c8-0064c45c6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.572 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:15.573 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:15.573 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000076 2023-07-29T00:25:15.574 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000076 2023-07-29T00:25:15.574 DEBUG:boto:Method: GET 2023-07-29T00:25:15.575 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000076 2023-07-29T00:25:15.575 DEBUG:boto:Data: 2023-07-29T00:25:15.575 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.576 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:15.576 DEBUG:boto:Port: 8001 2023-07-29T00:25:15.577 DEBUG:boto:Params: {} 2023-07-29T00:25:15.577 DEBUG:boto:Token: None 2023-07-29T00:25:15.578 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/ 2023-07-29T00:25:15.578 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P5MaecRSkISG+pYr7xzhbV+KLto= 2023-07-29T00:25:15.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P5MaecRSkISG+pYr7xzhbV+KLto=', 'Content-Length': '0'} 2023-07-29T00:25:15.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013dddd63c66c80ef-0064c45c6b-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.581 DEBUG:boto:b'hiiqos-58obj000000000000000000761000falseobj000000000000000000762023-07-29T00:24:53.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:15.582 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:15.583 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:15.583 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000076 2023-07-29T00:25:15.584 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000076 2023-07-29T00:25:15.584 DEBUG:boto:Method: GET 2023-07-29T00:25:15.585 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000076 2023-07-29T00:25:15.585 DEBUG:boto:Data: 2023-07-29T00:25:15.585 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.586 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:15.586 DEBUG:boto:Port: 8000 2023-07-29T00:25:15.586 DEBUG:boto:Params: {} 2023-07-29T00:25:15.587 DEBUG:boto:Token: None 2023-07-29T00:25:15.588 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/ 2023-07-29T00:25:15.588 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P5MaecRSkISG+pYr7xzhbV+KLto= 2023-07-29T00:25:15.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P5MaecRSkISG+pYr7xzhbV+KLto=', 'Content-Length': '0'} 2023-07-29T00:25:15.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee86653638a7aa68-0064c45c6b-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.594 DEBUG:boto:b'hiiqos-58obj000000000000000000761000falseobj000000000000000000762023-07-29T00:24:53.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:15.595 DEBUG:boto:path=/hiiqos-58/obj00000000000000000076 2023-07-29T00:25:15.595 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000076 2023-07-29T00:25:15.595 DEBUG:boto:Method: HEAD 2023-07-29T00:25:15.596 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000076 2023-07-29T00:25:15.596 DEBUG:boto:Data: 2023-07-29T00:25:15.597 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.597 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:15.597 DEBUG:boto:Port: 8001 2023-07-29T00:25:15.598 DEBUG:boto:Params: {} 2023-07-29T00:25:15.598 DEBUG:boto:Token: None 2023-07-29T00:25:15.598 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/obj00000000000000000076 2023-07-29T00:25:15.599 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9oN57CDwwKxVklEyJY2yMN0ZcII= 2023-07-29T00:25:15.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9oN57CDwwKxVklEyJY2yMN0ZcII=', 'Content-Length': '0'} 2023-07-29T00:25:15.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cee4e8d2f1737db-0064c45c6b-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.602 DEBUG:boto:path=/hiiqos-58/obj00000000000000000076 2023-07-29T00:25:15.602 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000076 2023-07-29T00:25:15.602 DEBUG:boto:Method: HEAD 2023-07-29T00:25:15.603 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000076 2023-07-29T00:25:15.603 DEBUG:boto:Data: 2023-07-29T00:25:15.603 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.604 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:15.604 DEBUG:boto:Port: 8000 2023-07-29T00:25:15.605 DEBUG:boto:Params: {} 2023-07-29T00:25:15.605 DEBUG:boto:Token: None 2023-07-29T00:25:15.605 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/obj00000000000000000076 2023-07-29T00:25:15.606 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9oN57CDwwKxVklEyJY2yMN0ZcII= 2023-07-29T00:25:15.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9oN57CDwwKxVklEyJY2yMN0ZcII=', 'Content-Length': '0'} 2023-07-29T00:25:15.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fb1b5f0f97b4999-0064c45c6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000076 2023-07-29T00:25:15.609 DEBUG:boto:path=/hiiqos-58/obj00000000000000000076 2023-07-29T00:25:15.609 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000076 2023-07-29T00:25:15.610 DEBUG:boto:Method: GET 2023-07-29T00:25:15.610 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000076 2023-07-29T00:25:15.611 DEBUG:boto:Data: 2023-07-29T00:25:15.611 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.611 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:15.612 DEBUG:boto:Port: 8001 2023-07-29T00:25:15.612 DEBUG:boto:Params: {} 2023-07-29T00:25:15.613 DEBUG:boto:Token: None 2023-07-29T00:25:15.613 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/obj00000000000000000076 2023-07-29T00:25:15.613 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0B3/DGDaIA2BhqzvaVgbowv0MqI= 2023-07-29T00:25:15.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0B3/DGDaIA2BhqzvaVgbowv0MqI=', 'Content-Length': '0'} 2023-07-29T00:25:15.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001700a073f84be25-0064c45c6b-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.656 DEBUG:boto:path=/hiiqos-58/obj00000000000000000076 2023-07-29T00:25:15.657 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000076 2023-07-29T00:25:15.657 DEBUG:boto:Method: GET 2023-07-29T00:25:15.658 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000076 2023-07-29T00:25:15.658 DEBUG:boto:Data: 2023-07-29T00:25:15.658 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.659 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:15.659 DEBUG:boto:Port: 8000 2023-07-29T00:25:15.659 DEBUG:boto:Params: {} 2023-07-29T00:25:15.660 DEBUG:boto:Token: None 2023-07-29T00:25:15.660 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/obj00000000000000000076 2023-07-29T00:25:15.661 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0B3/DGDaIA2BhqzvaVgbowv0MqI= 2023-07-29T00:25:15.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0B3/DGDaIA2BhqzvaVgbowv0MqI=', 'Content-Length': '0'} 2023-07-29T00:25:15.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dd83809e52f09ce-0064c45c6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.665 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:15.665 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:15.665 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000076 2023-07-29T00:25:15.666 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000076 2023-07-29T00:25:15.666 DEBUG:boto:Method: GET 2023-07-29T00:25:15.667 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000076 2023-07-29T00:25:15.667 DEBUG:boto:Data: 2023-07-29T00:25:15.667 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.668 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:15.668 DEBUG:boto:Port: 8001 2023-07-29T00:25:15.668 DEBUG:boto:Params: {} 2023-07-29T00:25:15.669 DEBUG:boto:Token: None 2023-07-29T00:25:15.669 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/ 2023-07-29T00:25:15.670 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P5MaecRSkISG+pYr7xzhbV+KLto= 2023-07-29T00:25:15.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P5MaecRSkISG+pYr7xzhbV+KLto=', 'Content-Length': '0'} 2023-07-29T00:25:15.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023b8165a06a72bf3-0064c45c6b-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.673 DEBUG:boto:b'hiiqos-58obj000000000000000000761000falseobj000000000000000000762023-07-29T00:24:53.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:15.674 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:15.675 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:15.675 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000076 2023-07-29T00:25:15.675 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000076 2023-07-29T00:25:15.676 DEBUG:boto:Method: GET 2023-07-29T00:25:15.676 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000076 2023-07-29T00:25:15.676 DEBUG:boto:Data: 2023-07-29T00:25:15.677 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.677 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:15.678 DEBUG:boto:Port: 8000 2023-07-29T00:25:15.678 DEBUG:boto:Params: {} 2023-07-29T00:25:15.678 DEBUG:boto:Token: None 2023-07-29T00:25:15.679 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/ 2023-07-29T00:25:15.679 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P5MaecRSkISG+pYr7xzhbV+KLto= 2023-07-29T00:25:15.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P5MaecRSkISG+pYr7xzhbV+KLto=', 'Content-Length': '0'} 2023-07-29T00:25:15.684 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e9b8c86d1e10b40-0064c45c6b-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.685 DEBUG:boto:b'hiiqos-58obj000000000000000000761000falseobj000000000000000000762023-07-29T00:24:53.955Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:15.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2023-07-29T00:25:15.686 DEBUG:boto:path=/hiiqos-58/obj00000000000000000133 2023-07-29T00:25:15.687 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000133 2023-07-29T00:25:15.687 DEBUG:boto:path=/hiiqos-58/obj00000000000000000133?versionId=null 2023-07-29T00:25:15.687 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000133?versionId=null 2023-07-29T00:25:15.688 DEBUG:boto:Method: GET 2023-07-29T00:25:15.688 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000133?versionId=null 2023-07-29T00:25:15.689 DEBUG:boto:Data: 2023-07-29T00:25:15.689 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.689 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:15.690 DEBUG:boto:Port: 8001 2023-07-29T00:25:15.690 DEBUG:boto:Params: {} 2023-07-29T00:25:15.691 DEBUG:boto:Token: None 2023-07-29T00:25:15.691 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/obj00000000000000000133?versionId=null 2023-07-29T00:25:15.692 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zOjNuQlbgP2RDgQ31rILTpsC4Io= 2023-07-29T00:25:15.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zOjNuQlbgP2RDgQ31rILTpsC4Io=', 'Content-Length': '0'} 2023-07-29T00:25:15.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000037f1ad98d7b549b-0064c45c6b-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.695 DEBUG:boto:path=/hiiqos-58/obj00000000000000000133 2023-07-29T00:25:15.696 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000133 2023-07-29T00:25:15.696 DEBUG:boto:path=/hiiqos-58/obj00000000000000000133?versionId=null 2023-07-29T00:25:15.696 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000133?versionId=null 2023-07-29T00:25:15.697 DEBUG:boto:Method: GET 2023-07-29T00:25:15.697 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000133?versionId=null 2023-07-29T00:25:15.698 DEBUG:boto:Data: 2023-07-29T00:25:15.698 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.698 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:15.699 DEBUG:boto:Port: 8000 2023-07-29T00:25:15.699 DEBUG:boto:Params: {} 2023-07-29T00:25:15.700 DEBUG:boto:Token: None 2023-07-29T00:25:15.700 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/obj00000000000000000133?versionId=null 2023-07-29T00:25:15.701 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zOjNuQlbgP2RDgQ31rILTpsC4Io= 2023-07-29T00:25:15.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zOjNuQlbgP2RDgQ31rILTpsC4Io=', 'Content-Length': '0'} 2023-07-29T00:25:15.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffec7e4cff01c1e7-0064c45c6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.756 DEBUG:boto:path=/hiiqos-58/obj00000000000000000133 2023-07-29T00:25:15.757 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000133 2023-07-29T00:25:15.757 DEBUG:boto:path=/hiiqos-58/obj00000000000000000133?versionId=null 2023-07-29T00:25:15.758 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000133?versionId=null 2023-07-29T00:25:15.758 DEBUG:boto:Method: HEAD 2023-07-29T00:25:15.758 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000133?versionId=null 2023-07-29T00:25:15.759 DEBUG:boto:Data: 2023-07-29T00:25:15.759 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.759 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:15.760 DEBUG:boto:Port: 8001 2023-07-29T00:25:15.760 DEBUG:boto:Params: {} 2023-07-29T00:25:15.760 DEBUG:boto:Token: None 2023-07-29T00:25:15.761 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/obj00000000000000000133?versionId=null 2023-07-29T00:25:15.761 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YJGrP5Uijxkji4dDSjRCPC/5sxI= 2023-07-29T00:25:15.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YJGrP5Uijxkji4dDSjRCPC/5sxI=', 'Content-Length': '0'} 2023-07-29T00:25:15.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b00013e6ea37b77e-0064c45c6b-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.764 DEBUG:boto:path=/hiiqos-58/obj00000000000000000133 2023-07-29T00:25:15.765 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000133 2023-07-29T00:25:15.765 DEBUG:boto:path=/hiiqos-58/obj00000000000000000133?versionId=null 2023-07-29T00:25:15.765 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000133?versionId=null 2023-07-29T00:25:15.768 DEBUG:boto:Method: HEAD 2023-07-29T00:25:15.768 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000133?versionId=null 2023-07-29T00:25:15.769 DEBUG:boto:Data: 2023-07-29T00:25:15.769 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.769 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:15.770 DEBUG:boto:Port: 8000 2023-07-29T00:25:15.770 DEBUG:boto:Params: {} 2023-07-29T00:25:15.771 DEBUG:boto:Token: None 2023-07-29T00:25:15.771 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/obj00000000000000000133?versionId=null 2023-07-29T00:25:15.771 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YJGrP5Uijxkji4dDSjRCPC/5sxI= 2023-07-29T00:25:15.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YJGrP5Uijxkji4dDSjRCPC/5sxI=', 'Content-Length': '0'} 2023-07-29T00:25:15.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000003d86d40dc7b72d-0064c45c6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2023-07-29T00:25:15.776 DEBUG:boto:path=/hiiqos-58/obj00000000000000000133 2023-07-29T00:25:15.776 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000133 2023-07-29T00:25:15.777 DEBUG:boto:path=/hiiqos-58/obj00000000000000000133?versionId=null 2023-07-29T00:25:15.778 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000133?versionId=null 2023-07-29T00:25:15.778 DEBUG:boto:Method: GET 2023-07-29T00:25:15.779 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000133?versionId=null 2023-07-29T00:25:15.779 DEBUG:boto:Data: 2023-07-29T00:25:15.780 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.781 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:15.781 DEBUG:boto:Port: 8001 2023-07-29T00:25:15.782 DEBUG:boto:Params: {} 2023-07-29T00:25:15.783 DEBUG:boto:Token: None 2023-07-29T00:25:15.784 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/obj00000000000000000133?versionId=null 2023-07-29T00:25:15.785 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zOjNuQlbgP2RDgQ31rILTpsC4Io= 2023-07-29T00:25:15.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zOjNuQlbgP2RDgQ31rILTpsC4Io=', 'Content-Length': '0'} 2023-07-29T00:25:15.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001122a1042c67ea61-0064c45c6b-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.789 DEBUG:boto:path=/hiiqos-58/obj00000000000000000133 2023-07-29T00:25:15.790 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000133 2023-07-29T00:25:15.790 DEBUG:boto:path=/hiiqos-58/obj00000000000000000133?versionId=null 2023-07-29T00:25:15.790 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000133?versionId=null 2023-07-29T00:25:15.791 DEBUG:boto:Method: GET 2023-07-29T00:25:15.791 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000133?versionId=null 2023-07-29T00:25:15.791 DEBUG:boto:Data: 2023-07-29T00:25:15.792 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.792 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:15.793 DEBUG:boto:Port: 8000 2023-07-29T00:25:15.794 DEBUG:boto:Params: {} 2023-07-29T00:25:15.795 DEBUG:boto:Token: None 2023-07-29T00:25:15.796 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/obj00000000000000000133?versionId=null 2023-07-29T00:25:15.797 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zOjNuQlbgP2RDgQ31rILTpsC4Io= 2023-07-29T00:25:15.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zOjNuQlbgP2RDgQ31rILTpsC4Io=', 'Content-Length': '0'} 2023-07-29T00:25:15.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f939e4aa2ac32a90-0064c45c6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.802 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:15.803 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:15.803 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000133 2023-07-29T00:25:15.804 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000133 2023-07-29T00:25:15.805 DEBUG:boto:Method: GET 2023-07-29T00:25:15.805 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000133 2023-07-29T00:25:15.806 DEBUG:boto:Data: 2023-07-29T00:25:15.806 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.807 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:15.808 DEBUG:boto:Port: 8001 2023-07-29T00:25:15.808 DEBUG:boto:Params: {} 2023-07-29T00:25:15.809 DEBUG:boto:Token: None 2023-07-29T00:25:15.810 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/ 2023-07-29T00:25:15.811 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P5MaecRSkISG+pYr7xzhbV+KLto= 2023-07-29T00:25:15.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P5MaecRSkISG+pYr7xzhbV+KLto=', 'Content-Length': '0'} 2023-07-29T00:25:15.814 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1e1fcf54fc8273d-0064c45c6b-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.857 DEBUG:boto:b'hiiqos-58obj000000000000000001331000falseobj000000000000000001332023-07-29T00:24:57.710Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:15.860 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:15.861 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:15.862 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000133 2023-07-29T00:25:15.863 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000133 2023-07-29T00:25:15.863 DEBUG:boto:Method: GET 2023-07-29T00:25:15.864 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000133 2023-07-29T00:25:15.864 DEBUG:boto:Data: 2023-07-29T00:25:15.864 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.865 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:15.865 DEBUG:boto:Port: 8000 2023-07-29T00:25:15.866 DEBUG:boto:Params: {} 2023-07-29T00:25:15.867 DEBUG:boto:Token: None 2023-07-29T00:25:15.867 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/ 2023-07-29T00:25:15.867 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P5MaecRSkISG+pYr7xzhbV+KLto= 2023-07-29T00:25:15.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P5MaecRSkISG+pYr7xzhbV+KLto=', 'Content-Length': '0'} 2023-07-29T00:25:15.872 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005864eed433652831-0064c45c6b-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.873 DEBUG:boto:b'hiiqos-58obj000000000000000001331000falseobj000000000000000001332023-07-29T00:24:57.710Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:15.873 DEBUG:boto:path=/hiiqos-58/obj00000000000000000133 2023-07-29T00:25:15.874 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000133 2023-07-29T00:25:15.875 DEBUG:boto:Method: HEAD 2023-07-29T00:25:15.875 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000133 2023-07-29T00:25:15.876 DEBUG:boto:Data: 2023-07-29T00:25:15.877 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.878 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:15.879 DEBUG:boto:Port: 8001 2023-07-29T00:25:15.880 DEBUG:boto:Params: {} 2023-07-29T00:25:15.882 DEBUG:boto:Token: None 2023-07-29T00:25:15.882 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/obj00000000000000000133 2023-07-29T00:25:15.883 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:T8PmYS1zzwsiwFJsiAoDHawgtBo= 2023-07-29T00:25:15.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:T8PmYS1zzwsiwFJsiAoDHawgtBo=', 'Content-Length': '0'} 2023-07-29T00:25:15.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3287348fcb2730f-0064c45c6b-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.886 DEBUG:boto:path=/hiiqos-58/obj00000000000000000133 2023-07-29T00:25:15.887 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000133 2023-07-29T00:25:15.888 DEBUG:boto:Method: HEAD 2023-07-29T00:25:15.888 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000133 2023-07-29T00:25:15.888 DEBUG:boto:Data: 2023-07-29T00:25:15.889 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.889 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:15.889 DEBUG:boto:Port: 8000 2023-07-29T00:25:15.890 DEBUG:boto:Params: {} 2023-07-29T00:25:15.891 DEBUG:boto:Token: None 2023-07-29T00:25:15.891 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/obj00000000000000000133 2023-07-29T00:25:15.891 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:T8PmYS1zzwsiwFJsiAoDHawgtBo= 2023-07-29T00:25:15.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:T8PmYS1zzwsiwFJsiAoDHawgtBo=', 'Content-Length': '0'} 2023-07-29T00:25:15.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f7fc92e5ee551ad-0064c45c6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2023-07-29T00:25:15.895 DEBUG:boto:path=/hiiqos-58/obj00000000000000000133 2023-07-29T00:25:15.896 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000133 2023-07-29T00:25:15.896 DEBUG:boto:Method: GET 2023-07-29T00:25:15.896 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000133 2023-07-29T00:25:15.897 DEBUG:boto:Data: 2023-07-29T00:25:15.897 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.898 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:15.898 DEBUG:boto:Port: 8001 2023-07-29T00:25:15.898 DEBUG:boto:Params: {} 2023-07-29T00:25:15.899 DEBUG:boto:Token: None 2023-07-29T00:25:15.899 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/obj00000000000000000133 2023-07-29T00:25:15.900 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:o3QvQDN5jaGJRuDnGiTZOku02Qo= 2023-07-29T00:25:15.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:o3QvQDN5jaGJRuDnGiTZOku02Qo=', 'Content-Length': '0'} 2023-07-29T00:25:15.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000023b75c31273174e-0064c45c6b-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.903 DEBUG:boto:path=/hiiqos-58/obj00000000000000000133 2023-07-29T00:25:15.904 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000133 2023-07-29T00:25:15.905 DEBUG:boto:Method: GET 2023-07-29T00:25:15.906 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000133 2023-07-29T00:25:15.906 DEBUG:boto:Data: 2023-07-29T00:25:15.906 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.907 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:15.907 DEBUG:boto:Port: 8000 2023-07-29T00:25:15.908 DEBUG:boto:Params: {} 2023-07-29T00:25:15.908 DEBUG:boto:Token: None 2023-07-29T00:25:15.908 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/obj00000000000000000133 2023-07-29T00:25:15.909 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:o3QvQDN5jaGJRuDnGiTZOku02Qo= 2023-07-29T00:25:15.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:o3QvQDN5jaGJRuDnGiTZOku02Qo=', 'Content-Length': '0'} 2023-07-29T00:25:15.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e5336ab99099b8a-0064c45c6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.913 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:15.914 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:15.914 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000133 2023-07-29T00:25:15.914 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000133 2023-07-29T00:25:15.915 DEBUG:boto:Method: GET 2023-07-29T00:25:15.915 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000133 2023-07-29T00:25:15.915 DEBUG:boto:Data: 2023-07-29T00:25:15.916 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.916 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:15.917 DEBUG:boto:Port: 8001 2023-07-29T00:25:15.917 DEBUG:boto:Params: {} 2023-07-29T00:25:15.917 DEBUG:boto:Token: None 2023-07-29T00:25:15.918 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/ 2023-07-29T00:25:15.918 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P5MaecRSkISG+pYr7xzhbV+KLto= 2023-07-29T00:25:15.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P5MaecRSkISG+pYr7xzhbV+KLto=', 'Content-Length': '0'} 2023-07-29T00:25:15.921 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007aaff1b8ff8f5090-0064c45c6b-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.922 DEBUG:boto:b'hiiqos-58obj000000000000000001331000falseobj000000000000000001332023-07-29T00:24:57.710Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:15.922 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:15.923 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:15.923 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000133 2023-07-29T00:25:15.924 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000133 2023-07-29T00:25:15.924 DEBUG:boto:Method: GET 2023-07-29T00:25:15.924 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000133 2023-07-29T00:25:15.925 DEBUG:boto:Data: 2023-07-29T00:25:15.925 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.925 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:15.926 DEBUG:boto:Port: 8000 2023-07-29T00:25:15.926 DEBUG:boto:Params: {} 2023-07-29T00:25:15.927 DEBUG:boto:Token: None 2023-07-29T00:25:15.927 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/ 2023-07-29T00:25:15.928 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P5MaecRSkISG+pYr7xzhbV+KLto= 2023-07-29T00:25:15.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P5MaecRSkISG+pYr7xzhbV+KLto=', 'Content-Length': '0'} 2023-07-29T00:25:15.933 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000663f6237fd96bd4f-0064c45c6b-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.933 DEBUG:boto:b'hiiqos-58obj000000000000000001331000falseobj000000000000000001332023-07-29T00:24:57.710Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:15.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000162 2023-07-29T00:25:15.934 DEBUG:boto:path=/hiiqos-58/obj00000000000000000162 2023-07-29T00:25:15.935 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000162 2023-07-29T00:25:15.935 DEBUG:boto:path=/hiiqos-58/obj00000000000000000162?versionId=null 2023-07-29T00:25:15.936 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000162?versionId=null 2023-07-29T00:25:15.936 DEBUG:boto:Method: GET 2023-07-29T00:25:15.936 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000162?versionId=null 2023-07-29T00:25:15.937 DEBUG:boto:Data: 2023-07-29T00:25:15.937 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.938 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:15.938 DEBUG:boto:Port: 8001 2023-07-29T00:25:15.938 DEBUG:boto:Params: {} 2023-07-29T00:25:15.939 DEBUG:boto:Token: None 2023-07-29T00:25:15.939 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/obj00000000000000000162?versionId=null 2023-07-29T00:25:15.940 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xxmLZq/RBShVPl1GTb/R5GMAGfM= 2023-07-29T00:25:15.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xxmLZq/RBShVPl1GTb/R5GMAGfM=', 'Content-Length': '0'} 2023-07-29T00:25:15.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004caac0aa89f44605-0064c45c6b-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.984 DEBUG:boto:path=/hiiqos-58/obj00000000000000000162 2023-07-29T00:25:15.985 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000162 2023-07-29T00:25:15.985 DEBUG:boto:path=/hiiqos-58/obj00000000000000000162?versionId=null 2023-07-29T00:25:15.986 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000162?versionId=null 2023-07-29T00:25:15.986 DEBUG:boto:Method: GET 2023-07-29T00:25:15.986 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000162?versionId=null 2023-07-29T00:25:15.987 DEBUG:boto:Data: 2023-07-29T00:25:15.987 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.987 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:15.988 DEBUG:boto:Port: 8000 2023-07-29T00:25:15.988 DEBUG:boto:Params: {} 2023-07-29T00:25:15.989 DEBUG:boto:Token: None 2023-07-29T00:25:15.989 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/obj00000000000000000162?versionId=null 2023-07-29T00:25:15.989 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xxmLZq/RBShVPl1GTb/R5GMAGfM= 2023-07-29T00:25:15.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xxmLZq/RBShVPl1GTb/R5GMAGfM=', 'Content-Length': '0'} 2023-07-29T00:25:15.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086c9a562a5ed8bc5-0064c45c6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:15.993 DEBUG:boto:path=/hiiqos-58/obj00000000000000000162 2023-07-29T00:25:15.993 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000162 2023-07-29T00:25:15.994 DEBUG:boto:path=/hiiqos-58/obj00000000000000000162?versionId=null 2023-07-29T00:25:15.994 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000162?versionId=null 2023-07-29T00:25:15.994 DEBUG:boto:Method: HEAD 2023-07-29T00:25:15.995 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000162?versionId=null 2023-07-29T00:25:15.995 DEBUG:boto:Data: 2023-07-29T00:25:15.995 DEBUG:boto:Headers: {} 2023-07-29T00:25:15.996 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:15.996 DEBUG:boto:Port: 8001 2023-07-29T00:25:15.996 DEBUG:boto:Params: {} 2023-07-29T00:25:15.997 DEBUG:boto:Token: None 2023-07-29T00:25:15.997 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:15 GMT /hiiqos-58/obj00000000000000000162?versionId=null 2023-07-29T00:25:15.997 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wQs+QZ0EFRkPzPO1N5COJzRVgnE= 2023-07-29T00:25:15.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wQs+QZ0EFRkPzPO1N5COJzRVgnE=', 'Content-Length': '0'} 2023-07-29T00:25:16.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfd0f4351f616e08-0064c45c6b-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.000 DEBUG:boto:path=/hiiqos-58/obj00000000000000000162 2023-07-29T00:25:16.001 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000162 2023-07-29T00:25:16.001 DEBUG:boto:path=/hiiqos-58/obj00000000000000000162?versionId=null 2023-07-29T00:25:16.001 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000162?versionId=null 2023-07-29T00:25:16.002 DEBUG:boto:Method: HEAD 2023-07-29T00:25:16.002 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000162?versionId=null 2023-07-29T00:25:16.002 DEBUG:boto:Data: 2023-07-29T00:25:16.003 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.003 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:16.003 DEBUG:boto:Port: 8000 2023-07-29T00:25:16.004 DEBUG:boto:Params: {} 2023-07-29T00:25:16.004 DEBUG:boto:Token: None 2023-07-29T00:25:16.004 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000162?versionId=null 2023-07-29T00:25:16.005 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/JmXhK8e2YF0mpz3RGzdWCgQR4s= 2023-07-29T00:25:16.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/JmXhK8e2YF0mpz3RGzdWCgQR4s=', 'Content-Length': '0'} 2023-07-29T00:25:16.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccd905a744a379e1-0064c45c6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000162 2023-07-29T00:25:16.008 DEBUG:boto:path=/hiiqos-58/obj00000000000000000162 2023-07-29T00:25:16.008 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000162 2023-07-29T00:25:16.009 DEBUG:boto:path=/hiiqos-58/obj00000000000000000162?versionId=null 2023-07-29T00:25:16.009 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000162?versionId=null 2023-07-29T00:25:16.010 DEBUG:boto:Method: GET 2023-07-29T00:25:16.010 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000162?versionId=null 2023-07-29T00:25:16.010 DEBUG:boto:Data: 2023-07-29T00:25:16.011 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.011 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:16.011 DEBUG:boto:Port: 8001 2023-07-29T00:25:16.012 DEBUG:boto:Params: {} 2023-07-29T00:25:16.012 DEBUG:boto:Token: None 2023-07-29T00:25:16.013 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000162?versionId=null 2023-07-29T00:25:16.013 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wFGfZOtuKpqQXaeDt25gKp+f5vo= 2023-07-29T00:25:16.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wFGfZOtuKpqQXaeDt25gKp+f5vo=', 'Content-Length': '0'} 2023-07-29T00:25:16.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e78b255fa25c6bbe-0064c45c6b-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.016 DEBUG:boto:path=/hiiqos-58/obj00000000000000000162 2023-07-29T00:25:16.016 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000162 2023-07-29T00:25:16.017 DEBUG:boto:path=/hiiqos-58/obj00000000000000000162?versionId=null 2023-07-29T00:25:16.017 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000162?versionId=null 2023-07-29T00:25:16.017 DEBUG:boto:Method: GET 2023-07-29T00:25:16.018 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000162?versionId=null 2023-07-29T00:25:16.018 DEBUG:boto:Data: 2023-07-29T00:25:16.018 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.019 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:16.019 DEBUG:boto:Port: 8000 2023-07-29T00:25:16.019 DEBUG:boto:Params: {} 2023-07-29T00:25:16.020 DEBUG:boto:Token: None 2023-07-29T00:25:16.020 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000162?versionId=null 2023-07-29T00:25:16.021 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wFGfZOtuKpqQXaeDt25gKp+f5vo= 2023-07-29T00:25:16.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wFGfZOtuKpqQXaeDt25gKp+f5vo=', 'Content-Length': '0'} 2023-07-29T00:25:16.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ba5473f313c8546-0064c45c6b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.024 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:16.024 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:16.025 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000162 2023-07-29T00:25:16.025 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000162 2023-07-29T00:25:16.025 DEBUG:boto:Method: GET 2023-07-29T00:25:16.026 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000162 2023-07-29T00:25:16.026 DEBUG:boto:Data: 2023-07-29T00:25:16.026 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.027 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:16.027 DEBUG:boto:Port: 8001 2023-07-29T00:25:16.027 DEBUG:boto:Params: {} 2023-07-29T00:25:16.028 DEBUG:boto:Token: None 2023-07-29T00:25:16.028 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/ 2023-07-29T00:25:16.029 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ctvzq1HqhVpcvSTsF416pCLFISQ= 2023-07-29T00:25:16.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ctvzq1HqhVpcvSTsF416pCLFISQ=', 'Content-Length': '0'} 2023-07-29T00:25:16.032 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005427b258a3be33ff-0064c45c6b-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.032 DEBUG:boto:b'hiiqos-58obj000000000000000001621000falseobj000000000000000001622023-07-29T00:24:57.876Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:16.033 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:16.033 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:16.034 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000162 2023-07-29T00:25:16.034 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000162 2023-07-29T00:25:16.034 DEBUG:boto:Method: GET 2023-07-29T00:25:16.035 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000162 2023-07-29T00:25:16.035 DEBUG:boto:Data: 2023-07-29T00:25:16.035 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.036 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:16.036 DEBUG:boto:Port: 8000 2023-07-29T00:25:16.036 DEBUG:boto:Params: {} 2023-07-29T00:25:16.037 DEBUG:boto:Token: None 2023-07-29T00:25:16.037 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/ 2023-07-29T00:25:16.038 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ctvzq1HqhVpcvSTsF416pCLFISQ= 2023-07-29T00:25:16.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ctvzq1HqhVpcvSTsF416pCLFISQ=', 'Content-Length': '0'} 2023-07-29T00:25:16.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034abf5bbb874f5ab-0064c45c6b-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.043 DEBUG:boto:b'hiiqos-58obj000000000000000001621000falseobj000000000000000001622023-07-29T00:24:57.876Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:16.044 DEBUG:boto:path=/hiiqos-58/obj00000000000000000162 2023-07-29T00:25:16.044 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000162 2023-07-29T00:25:16.045 DEBUG:boto:Method: HEAD 2023-07-29T00:25:16.046 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000162 2023-07-29T00:25:16.046 DEBUG:boto:Data: 2023-07-29T00:25:16.046 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.047 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:16.047 DEBUG:boto:Port: 8001 2023-07-29T00:25:16.047 DEBUG:boto:Params: {} 2023-07-29T00:25:16.048 DEBUG:boto:Token: None 2023-07-29T00:25:16.048 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000162 2023-07-29T00:25:16.049 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uSg8qE1JfNBJ63gFo7ERY6p0E3E= 2023-07-29T00:25:16.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uSg8qE1JfNBJ63gFo7ERY6p0E3E=', 'Content-Length': '0'} 2023-07-29T00:25:16.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b28842be558d8ae2-0064c45c6b-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.052 DEBUG:boto:path=/hiiqos-58/obj00000000000000000162 2023-07-29T00:25:16.052 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000162 2023-07-29T00:25:16.053 DEBUG:boto:Method: HEAD 2023-07-29T00:25:16.053 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000162 2023-07-29T00:25:16.053 DEBUG:boto:Data: 2023-07-29T00:25:16.054 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.054 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:16.054 DEBUG:boto:Port: 8000 2023-07-29T00:25:16.055 DEBUG:boto:Params: {} 2023-07-29T00:25:16.055 DEBUG:boto:Token: None 2023-07-29T00:25:16.056 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000162 2023-07-29T00:25:16.057 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uSg8qE1JfNBJ63gFo7ERY6p0E3E= 2023-07-29T00:25:16.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uSg8qE1JfNBJ63gFo7ERY6p0E3E=', 'Content-Length': '0'} 2023-07-29T00:25:16.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7e56d26ef304246-0064c45c6c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000162 2023-07-29T00:25:16.060 DEBUG:boto:path=/hiiqos-58/obj00000000000000000162 2023-07-29T00:25:16.061 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000162 2023-07-29T00:25:16.061 DEBUG:boto:Method: GET 2023-07-29T00:25:16.062 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000162 2023-07-29T00:25:16.062 DEBUG:boto:Data: 2023-07-29T00:25:16.063 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.063 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:16.064 DEBUG:boto:Port: 8001 2023-07-29T00:25:16.064 DEBUG:boto:Params: {} 2023-07-29T00:25:16.065 DEBUG:boto:Token: None 2023-07-29T00:25:16.065 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000162 2023-07-29T00:25:16.066 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+B4R2jPF0gncoaHZe3nJWW/iJi8= 2023-07-29T00:25:16.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+B4R2jPF0gncoaHZe3nJWW/iJi8=', 'Content-Length': '0'} 2023-07-29T00:25:16.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000591b678b47e9b2d2-0064c45c6c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.069 DEBUG:boto:path=/hiiqos-58/obj00000000000000000162 2023-07-29T00:25:16.069 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000162 2023-07-29T00:25:16.070 DEBUG:boto:Method: GET 2023-07-29T00:25:16.070 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000162 2023-07-29T00:25:16.070 DEBUG:boto:Data: 2023-07-29T00:25:16.071 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.071 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:16.072 DEBUG:boto:Port: 8000 2023-07-29T00:25:16.072 DEBUG:boto:Params: {} 2023-07-29T00:25:16.072 DEBUG:boto:Token: None 2023-07-29T00:25:16.073 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000162 2023-07-29T00:25:16.073 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+B4R2jPF0gncoaHZe3nJWW/iJi8= 2023-07-29T00:25:16.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+B4R2jPF0gncoaHZe3nJWW/iJi8=', 'Content-Length': '0'} 2023-07-29T00:25:16.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bff13605ea6181ce-0064c45c6c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.116 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:16.117 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:16.118 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000162 2023-07-29T00:25:16.118 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000162 2023-07-29T00:25:16.118 DEBUG:boto:Method: GET 2023-07-29T00:25:16.119 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000162 2023-07-29T00:25:16.119 DEBUG:boto:Data: 2023-07-29T00:25:16.119 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.120 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:16.120 DEBUG:boto:Port: 8001 2023-07-29T00:25:16.121 DEBUG:boto:Params: {} 2023-07-29T00:25:16.122 DEBUG:boto:Token: None 2023-07-29T00:25:16.122 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/ 2023-07-29T00:25:16.123 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ctvzq1HqhVpcvSTsF416pCLFISQ= 2023-07-29T00:25:16.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ctvzq1HqhVpcvSTsF416pCLFISQ=', 'Content-Length': '0'} 2023-07-29T00:25:16.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000681d046a71a77952-0064c45c6c-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.127 DEBUG:boto:b'hiiqos-58obj000000000000000001621000falseobj000000000000000001622023-07-29T00:24:57.876Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:16.128 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:16.128 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:16.129 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000162 2023-07-29T00:25:16.129 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000162 2023-07-29T00:25:16.130 DEBUG:boto:Method: GET 2023-07-29T00:25:16.130 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000162 2023-07-29T00:25:16.130 DEBUG:boto:Data: 2023-07-29T00:25:16.131 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.131 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:16.131 DEBUG:boto:Port: 8000 2023-07-29T00:25:16.132 DEBUG:boto:Params: {} 2023-07-29T00:25:16.132 DEBUG:boto:Token: None 2023-07-29T00:25:16.133 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/ 2023-07-29T00:25:16.133 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ctvzq1HqhVpcvSTsF416pCLFISQ= 2023-07-29T00:25:16.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ctvzq1HqhVpcvSTsF416pCLFISQ=', 'Content-Length': '0'} 2023-07-29T00:25:16.140 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008733cb0515de94ab-0064c45c6c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.141 DEBUG:boto:b'hiiqos-58obj000000000000000001621000falseobj000000000000000001622023-07-29T00:24:57.876Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:16.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2023-07-29T00:25:16.142 DEBUG:boto:path=/hiiqos-58/obj00000000000000000271 2023-07-29T00:25:16.143 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000271 2023-07-29T00:25:16.143 DEBUG:boto:path=/hiiqos-58/obj00000000000000000271?versionId=null 2023-07-29T00:25:16.143 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000271?versionId=null 2023-07-29T00:25:16.144 DEBUG:boto:Method: GET 2023-07-29T00:25:16.144 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000271?versionId=null 2023-07-29T00:25:16.145 DEBUG:boto:Data: 2023-07-29T00:25:16.145 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.145 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:16.146 DEBUG:boto:Port: 8001 2023-07-29T00:25:16.146 DEBUG:boto:Params: {} 2023-07-29T00:25:16.146 DEBUG:boto:Token: None 2023-07-29T00:25:16.147 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000271?versionId=null 2023-07-29T00:25:16.147 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cN6y8N97dN4z3BoINrA1wPE5Bzw= 2023-07-29T00:25:16.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cN6y8N97dN4z3BoINrA1wPE5Bzw=', 'Content-Length': '0'} 2023-07-29T00:25:16.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008e923ce60b74ffb-0064c45c6c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.151 DEBUG:boto:path=/hiiqos-58/obj00000000000000000271 2023-07-29T00:25:16.151 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000271 2023-07-29T00:25:16.151 DEBUG:boto:path=/hiiqos-58/obj00000000000000000271?versionId=null 2023-07-29T00:25:16.152 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000271?versionId=null 2023-07-29T00:25:16.152 DEBUG:boto:Method: GET 2023-07-29T00:25:16.152 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000271?versionId=null 2023-07-29T00:25:16.153 DEBUG:boto:Data: 2023-07-29T00:25:16.153 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.154 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:16.154 DEBUG:boto:Port: 8000 2023-07-29T00:25:16.154 DEBUG:boto:Params: {} 2023-07-29T00:25:16.155 DEBUG:boto:Token: None 2023-07-29T00:25:16.155 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000271?versionId=null 2023-07-29T00:25:16.156 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cN6y8N97dN4z3BoINrA1wPE5Bzw= 2023-07-29T00:25:16.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cN6y8N97dN4z3BoINrA1wPE5Bzw=', 'Content-Length': '0'} 2023-07-29T00:25:16.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e32e9d90061df6ca-0064c45c6c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.159 DEBUG:boto:path=/hiiqos-58/obj00000000000000000271 2023-07-29T00:25:16.159 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000271 2023-07-29T00:25:16.159 DEBUG:boto:path=/hiiqos-58/obj00000000000000000271?versionId=null 2023-07-29T00:25:16.160 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000271?versionId=null 2023-07-29T00:25:16.160 DEBUG:boto:Method: HEAD 2023-07-29T00:25:16.160 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000271?versionId=null 2023-07-29T00:25:16.161 DEBUG:boto:Data: 2023-07-29T00:25:16.161 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.161 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:16.162 DEBUG:boto:Port: 8001 2023-07-29T00:25:16.162 DEBUG:boto:Params: {} 2023-07-29T00:25:16.162 DEBUG:boto:Token: None 2023-07-29T00:25:16.163 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000271?versionId=null 2023-07-29T00:25:16.163 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BN8a7SerAaQC/WsKIhrjSHhtJPI= 2023-07-29T00:25:16.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BN8a7SerAaQC/WsKIhrjSHhtJPI=', 'Content-Length': '0'} 2023-07-29T00:25:16.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023e2362e211ddb66-0064c45c6c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.166 DEBUG:boto:path=/hiiqos-58/obj00000000000000000271 2023-07-29T00:25:16.166 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000271 2023-07-29T00:25:16.167 DEBUG:boto:path=/hiiqos-58/obj00000000000000000271?versionId=null 2023-07-29T00:25:16.167 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000271?versionId=null 2023-07-29T00:25:16.167 DEBUG:boto:Method: HEAD 2023-07-29T00:25:16.168 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000271?versionId=null 2023-07-29T00:25:16.168 DEBUG:boto:Data: 2023-07-29T00:25:16.168 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.169 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:16.169 DEBUG:boto:Port: 8000 2023-07-29T00:25:16.169 DEBUG:boto:Params: {} 2023-07-29T00:25:16.170 DEBUG:boto:Token: None 2023-07-29T00:25:16.170 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000271?versionId=null 2023-07-29T00:25:16.171 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BN8a7SerAaQC/WsKIhrjSHhtJPI= 2023-07-29T00:25:16.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BN8a7SerAaQC/WsKIhrjSHhtJPI=', 'Content-Length': '0'} 2023-07-29T00:25:16.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c250993969b7cd47-0064c45c6c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2023-07-29T00:25:16.174 DEBUG:boto:path=/hiiqos-58/obj00000000000000000271 2023-07-29T00:25:16.174 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000271 2023-07-29T00:25:16.175 DEBUG:boto:path=/hiiqos-58/obj00000000000000000271?versionId=null 2023-07-29T00:25:16.175 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000271?versionId=null 2023-07-29T00:25:16.175 DEBUG:boto:Method: GET 2023-07-29T00:25:16.176 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000271?versionId=null 2023-07-29T00:25:16.176 DEBUG:boto:Data: 2023-07-29T00:25:16.176 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.177 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:16.177 DEBUG:boto:Port: 8001 2023-07-29T00:25:16.178 DEBUG:boto:Params: {} 2023-07-29T00:25:16.178 DEBUG:boto:Token: None 2023-07-29T00:25:16.178 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000271?versionId=null 2023-07-29T00:25:16.179 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cN6y8N97dN4z3BoINrA1wPE5Bzw= 2023-07-29T00:25:16.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cN6y8N97dN4z3BoINrA1wPE5Bzw=', 'Content-Length': '0'} 2023-07-29T00:25:16.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005002af1800613d9a-0064c45c6c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.182 DEBUG:boto:path=/hiiqos-58/obj00000000000000000271 2023-07-29T00:25:16.182 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000271 2023-07-29T00:25:16.183 DEBUG:boto:path=/hiiqos-58/obj00000000000000000271?versionId=null 2023-07-29T00:25:16.183 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000271?versionId=null 2023-07-29T00:25:16.183 DEBUG:boto:Method: GET 2023-07-29T00:25:16.184 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000271?versionId=null 2023-07-29T00:25:16.184 DEBUG:boto:Data: 2023-07-29T00:25:16.185 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.185 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:16.185 DEBUG:boto:Port: 8000 2023-07-29T00:25:16.186 DEBUG:boto:Params: {} 2023-07-29T00:25:16.186 DEBUG:boto:Token: None 2023-07-29T00:25:16.186 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000271?versionId=null 2023-07-29T00:25:16.187 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cN6y8N97dN4z3BoINrA1wPE5Bzw= 2023-07-29T00:25:16.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cN6y8N97dN4z3BoINrA1wPE5Bzw=', 'Content-Length': '0'} 2023-07-29T00:25:16.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046d3017cf2c2b460-0064c45c6c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.232 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:16.233 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:16.233 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000271 2023-07-29T00:25:16.234 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000271 2023-07-29T00:25:16.234 DEBUG:boto:Method: GET 2023-07-29T00:25:16.234 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000271 2023-07-29T00:25:16.235 DEBUG:boto:Data: 2023-07-29T00:25:16.235 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.236 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:16.236 DEBUG:boto:Port: 8001 2023-07-29T00:25:16.236 DEBUG:boto:Params: {} 2023-07-29T00:25:16.237 DEBUG:boto:Token: None 2023-07-29T00:25:16.237 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/ 2023-07-29T00:25:16.238 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ctvzq1HqhVpcvSTsF416pCLFISQ= 2023-07-29T00:25:16.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ctvzq1HqhVpcvSTsF416pCLFISQ=', 'Content-Length': '0'} 2023-07-29T00:25:16.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000305a8e3850f4baba-0064c45c6c-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.241 DEBUG:boto:b'hiiqos-58obj000000000000000002711000falseobj000000000000000002712023-07-29T00:24:58.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:16.242 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:16.242 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:16.243 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000271 2023-07-29T00:25:16.243 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000271 2023-07-29T00:25:16.243 DEBUG:boto:Method: GET 2023-07-29T00:25:16.244 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000271 2023-07-29T00:25:16.244 DEBUG:boto:Data: 2023-07-29T00:25:16.245 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.245 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:16.245 DEBUG:boto:Port: 8000 2023-07-29T00:25:16.246 DEBUG:boto:Params: {} 2023-07-29T00:25:16.246 DEBUG:boto:Token: None 2023-07-29T00:25:16.247 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/ 2023-07-29T00:25:16.247 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ctvzq1HqhVpcvSTsF416pCLFISQ= 2023-07-29T00:25:16.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ctvzq1HqhVpcvSTsF416pCLFISQ=', 'Content-Length': '0'} 2023-07-29T00:25:16.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e63c2cdd5861a5e-0064c45c6c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.253 DEBUG:boto:b'hiiqos-58obj000000000000000002711000falseobj000000000000000002712023-07-29T00:24:58.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:16.253 DEBUG:boto:path=/hiiqos-58/obj00000000000000000271 2023-07-29T00:25:16.254 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000271 2023-07-29T00:25:16.254 DEBUG:boto:Method: HEAD 2023-07-29T00:25:16.254 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000271 2023-07-29T00:25:16.255 DEBUG:boto:Data: 2023-07-29T00:25:16.255 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.256 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:16.256 DEBUG:boto:Port: 8001 2023-07-29T00:25:16.256 DEBUG:boto:Params: {} 2023-07-29T00:25:16.257 DEBUG:boto:Token: None 2023-07-29T00:25:16.257 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000271 2023-07-29T00:25:16.258 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2XYUFRIlLl90dc0Rek2PLUL2ZX4= 2023-07-29T00:25:16.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2XYUFRIlLl90dc0Rek2PLUL2ZX4=', 'Content-Length': '0'} 2023-07-29T00:25:16.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a2b1d51a079d962-0064c45c6c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.261 DEBUG:boto:path=/hiiqos-58/obj00000000000000000271 2023-07-29T00:25:16.261 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000271 2023-07-29T00:25:16.262 DEBUG:boto:Method: HEAD 2023-07-29T00:25:16.262 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000271 2023-07-29T00:25:16.262 DEBUG:boto:Data: 2023-07-29T00:25:16.263 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.263 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:16.263 DEBUG:boto:Port: 8000 2023-07-29T00:25:16.264 DEBUG:boto:Params: {} 2023-07-29T00:25:16.264 DEBUG:boto:Token: None 2023-07-29T00:25:16.265 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000271 2023-07-29T00:25:16.265 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2XYUFRIlLl90dc0Rek2PLUL2ZX4= 2023-07-29T00:25:16.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2XYUFRIlLl90dc0Rek2PLUL2ZX4=', 'Content-Length': '0'} 2023-07-29T00:25:16.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007eda58cbe68fd954-0064c45c6c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2023-07-29T00:25:16.268 DEBUG:boto:path=/hiiqos-58/obj00000000000000000271 2023-07-29T00:25:16.269 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000271 2023-07-29T00:25:16.270 DEBUG:boto:Method: GET 2023-07-29T00:25:16.270 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000271 2023-07-29T00:25:16.271 DEBUG:boto:Data: 2023-07-29T00:25:16.271 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.272 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:16.272 DEBUG:boto:Port: 8001 2023-07-29T00:25:16.272 DEBUG:boto:Params: {} 2023-07-29T00:25:16.273 DEBUG:boto:Token: None 2023-07-29T00:25:16.273 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000271 2023-07-29T00:25:16.274 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:r+J5PW3AILSTPPd2yoWVD9U8ELQ= 2023-07-29T00:25:16.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:r+J5PW3AILSTPPd2yoWVD9U8ELQ=', 'Content-Length': '0'} 2023-07-29T00:25:16.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096f7814b4ab768b8-0064c45c6c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.316 DEBUG:boto:path=/hiiqos-58/obj00000000000000000271 2023-07-29T00:25:16.317 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000271 2023-07-29T00:25:16.317 DEBUG:boto:Method: GET 2023-07-29T00:25:16.318 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000271 2023-07-29T00:25:16.318 DEBUG:boto:Data: 2023-07-29T00:25:16.319 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.319 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:16.319 DEBUG:boto:Port: 8000 2023-07-29T00:25:16.320 DEBUG:boto:Params: {} 2023-07-29T00:25:16.320 DEBUG:boto:Token: None 2023-07-29T00:25:16.321 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000271 2023-07-29T00:25:16.322 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:r+J5PW3AILSTPPd2yoWVD9U8ELQ= 2023-07-29T00:25:16.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:r+J5PW3AILSTPPd2yoWVD9U8ELQ=', 'Content-Length': '0'} 2023-07-29T00:25:16.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001148b63b3445316e-0064c45c6c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.325 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:16.325 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:16.326 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000271 2023-07-29T00:25:16.326 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000271 2023-07-29T00:25:16.326 DEBUG:boto:Method: GET 2023-07-29T00:25:16.327 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000271 2023-07-29T00:25:16.327 DEBUG:boto:Data: 2023-07-29T00:25:16.327 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.328 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:16.328 DEBUG:boto:Port: 8001 2023-07-29T00:25:16.328 DEBUG:boto:Params: {} 2023-07-29T00:25:16.329 DEBUG:boto:Token: None 2023-07-29T00:25:16.329 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/ 2023-07-29T00:25:16.330 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ctvzq1HqhVpcvSTsF416pCLFISQ= 2023-07-29T00:25:16.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ctvzq1HqhVpcvSTsF416pCLFISQ=', 'Content-Length': '0'} 2023-07-29T00:25:16.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a724ea513495e70c-0064c45c6c-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.333 DEBUG:boto:b'hiiqos-58obj000000000000000002711000falseobj000000000000000002712023-07-29T00:24:58.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:16.334 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:16.334 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:16.335 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000271 2023-07-29T00:25:16.335 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000271 2023-07-29T00:25:16.335 DEBUG:boto:Method: GET 2023-07-29T00:25:16.336 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000271 2023-07-29T00:25:16.336 DEBUG:boto:Data: 2023-07-29T00:25:16.336 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.337 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:16.337 DEBUG:boto:Port: 8000 2023-07-29T00:25:16.337 DEBUG:boto:Params: {} 2023-07-29T00:25:16.338 DEBUG:boto:Token: None 2023-07-29T00:25:16.338 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/ 2023-07-29T00:25:16.339 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ctvzq1HqhVpcvSTsF416pCLFISQ= 2023-07-29T00:25:16.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ctvzq1HqhVpcvSTsF416pCLFISQ=', 'Content-Length': '0'} 2023-07-29T00:25:16.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1e8740960952aec-0064c45c6c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.344 DEBUG:boto:b'hiiqos-58obj000000000000000002711000falseobj000000000000000002712023-07-29T00:24:58.148Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:16.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2023-07-29T00:25:16.345 DEBUG:boto:path=/hiiqos-58/obj00000000000000000317 2023-07-29T00:25:16.346 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000317 2023-07-29T00:25:16.346 DEBUG:boto:path=/hiiqos-58/obj00000000000000000317?versionId=null 2023-07-29T00:25:16.346 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000317?versionId=null 2023-07-29T00:25:16.347 DEBUG:boto:Method: GET 2023-07-29T00:25:16.347 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000317?versionId=null 2023-07-29T00:25:16.348 DEBUG:boto:Data: 2023-07-29T00:25:16.348 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.348 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:16.349 DEBUG:boto:Port: 8001 2023-07-29T00:25:16.349 DEBUG:boto:Params: {} 2023-07-29T00:25:16.349 DEBUG:boto:Token: None 2023-07-29T00:25:16.350 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000317?versionId=null 2023-07-29T00:25:16.350 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZGue8Jp7gxL0k13i7o7ogIA3gos= 2023-07-29T00:25:16.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZGue8Jp7gxL0k13i7o7ogIA3gos=', 'Content-Length': '0'} 2023-07-29T00:25:16.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000307d79b7b98eca93-0064c45c6c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.353 DEBUG:boto:path=/hiiqos-58/obj00000000000000000317 2023-07-29T00:25:16.354 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000317 2023-07-29T00:25:16.354 DEBUG:boto:path=/hiiqos-58/obj00000000000000000317?versionId=null 2023-07-29T00:25:16.355 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000317?versionId=null 2023-07-29T00:25:16.355 DEBUG:boto:Method: GET 2023-07-29T00:25:16.356 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000317?versionId=null 2023-07-29T00:25:16.356 DEBUG:boto:Data: 2023-07-29T00:25:16.356 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.357 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:16.357 DEBUG:boto:Port: 8000 2023-07-29T00:25:16.357 DEBUG:boto:Params: {} 2023-07-29T00:25:16.358 DEBUG:boto:Token: None 2023-07-29T00:25:16.358 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000317?versionId=null 2023-07-29T00:25:16.359 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZGue8Jp7gxL0k13i7o7ogIA3gos= 2023-07-29T00:25:16.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZGue8Jp7gxL0k13i7o7ogIA3gos=', 'Content-Length': '0'} 2023-07-29T00:25:16.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e98ea53d81cdfdfb-0064c45c6c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.362 DEBUG:boto:path=/hiiqos-58/obj00000000000000000317 2023-07-29T00:25:16.362 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000317 2023-07-29T00:25:16.362 DEBUG:boto:path=/hiiqos-58/obj00000000000000000317?versionId=null 2023-07-29T00:25:16.363 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000317?versionId=null 2023-07-29T00:25:16.363 DEBUG:boto:Method: HEAD 2023-07-29T00:25:16.363 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000317?versionId=null 2023-07-29T00:25:16.364 DEBUG:boto:Data: 2023-07-29T00:25:16.364 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.364 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:16.365 DEBUG:boto:Port: 8001 2023-07-29T00:25:16.365 DEBUG:boto:Params: {} 2023-07-29T00:25:16.365 DEBUG:boto:Token: None 2023-07-29T00:25:16.366 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000317?versionId=null 2023-07-29T00:25:16.366 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:L8XtsN3vYp9s6xakp3YTBCfFtYE= 2023-07-29T00:25:16.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:L8XtsN3vYp9s6xakp3YTBCfFtYE=', 'Content-Length': '0'} 2023-07-29T00:25:16.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001468972feb999a75-0064c45c6c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.369 DEBUG:boto:path=/hiiqos-58/obj00000000000000000317 2023-07-29T00:25:16.370 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000317 2023-07-29T00:25:16.370 DEBUG:boto:path=/hiiqos-58/obj00000000000000000317?versionId=null 2023-07-29T00:25:16.370 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000317?versionId=null 2023-07-29T00:25:16.371 DEBUG:boto:Method: HEAD 2023-07-29T00:25:16.371 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000317?versionId=null 2023-07-29T00:25:16.371 DEBUG:boto:Data: 2023-07-29T00:25:16.372 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.372 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:16.373 DEBUG:boto:Port: 8000 2023-07-29T00:25:16.373 DEBUG:boto:Params: {} 2023-07-29T00:25:16.373 DEBUG:boto:Token: None 2023-07-29T00:25:16.374 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000317?versionId=null 2023-07-29T00:25:16.374 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:L8XtsN3vYp9s6xakp3YTBCfFtYE= 2023-07-29T00:25:16.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:L8XtsN3vYp9s6xakp3YTBCfFtYE=', 'Content-Length': '0'} 2023-07-29T00:25:16.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039e479d326633051-0064c45c6c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2023-07-29T00:25:16.377 DEBUG:boto:path=/hiiqos-58/obj00000000000000000317 2023-07-29T00:25:16.378 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000317 2023-07-29T00:25:16.378 DEBUG:boto:path=/hiiqos-58/obj00000000000000000317?versionId=null 2023-07-29T00:25:16.378 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000317?versionId=null 2023-07-29T00:25:16.379 DEBUG:boto:Method: GET 2023-07-29T00:25:16.379 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000317?versionId=null 2023-07-29T00:25:16.380 DEBUG:boto:Data: 2023-07-29T00:25:16.380 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.380 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:16.381 DEBUG:boto:Port: 8001 2023-07-29T00:25:16.381 DEBUG:boto:Params: {} 2023-07-29T00:25:16.381 DEBUG:boto:Token: None 2023-07-29T00:25:16.382 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000317?versionId=null 2023-07-29T00:25:16.382 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZGue8Jp7gxL0k13i7o7ogIA3gos= 2023-07-29T00:25:16.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZGue8Jp7gxL0k13i7o7ogIA3gos=', 'Content-Length': '0'} 2023-07-29T00:25:16.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfbcb5773d6c1107-0064c45c6c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.432 DEBUG:boto:path=/hiiqos-58/obj00000000000000000317 2023-07-29T00:25:16.433 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000317 2023-07-29T00:25:16.434 DEBUG:boto:path=/hiiqos-58/obj00000000000000000317?versionId=null 2023-07-29T00:25:16.434 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000317?versionId=null 2023-07-29T00:25:16.434 DEBUG:boto:Method: GET 2023-07-29T00:25:16.435 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000317?versionId=null 2023-07-29T00:25:16.435 DEBUG:boto:Data: 2023-07-29T00:25:16.436 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.436 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:16.436 DEBUG:boto:Port: 8000 2023-07-29T00:25:16.437 DEBUG:boto:Params: {} 2023-07-29T00:25:16.437 DEBUG:boto:Token: None 2023-07-29T00:25:16.438 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000317?versionId=null 2023-07-29T00:25:16.439 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZGue8Jp7gxL0k13i7o7ogIA3gos= 2023-07-29T00:25:16.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZGue8Jp7gxL0k13i7o7ogIA3gos=', 'Content-Length': '0'} 2023-07-29T00:25:16.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024a80cb797163c70-0064c45c6c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.445 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:16.446 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:16.446 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000317 2023-07-29T00:25:16.447 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000317 2023-07-29T00:25:16.447 DEBUG:boto:Method: GET 2023-07-29T00:25:16.448 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000317 2023-07-29T00:25:16.448 DEBUG:boto:Data: 2023-07-29T00:25:16.449 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.449 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:16.450 DEBUG:boto:Port: 8001 2023-07-29T00:25:16.450 DEBUG:boto:Params: {} 2023-07-29T00:25:16.450 DEBUG:boto:Token: None 2023-07-29T00:25:16.451 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/ 2023-07-29T00:25:16.452 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ctvzq1HqhVpcvSTsF416pCLFISQ= 2023-07-29T00:25:16.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ctvzq1HqhVpcvSTsF416pCLFISQ=', 'Content-Length': '0'} 2023-07-29T00:25:16.455 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000895936ac5ac66bca-0064c45c6c-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.456 DEBUG:boto:b'hiiqos-58obj000000000000000003171000falseobj000000000000000003172023-07-29T00:24:58.614Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:16.457 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:16.457 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:16.458 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000317 2023-07-29T00:25:16.458 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000317 2023-07-29T00:25:16.458 DEBUG:boto:Method: GET 2023-07-29T00:25:16.459 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000317 2023-07-29T00:25:16.459 DEBUG:boto:Data: 2023-07-29T00:25:16.459 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.460 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:16.460 DEBUG:boto:Port: 8000 2023-07-29T00:25:16.461 DEBUG:boto:Params: {} 2023-07-29T00:25:16.461 DEBUG:boto:Token: None 2023-07-29T00:25:16.462 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/ 2023-07-29T00:25:16.462 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ctvzq1HqhVpcvSTsF416pCLFISQ= 2023-07-29T00:25:16.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ctvzq1HqhVpcvSTsF416pCLFISQ=', 'Content-Length': '0'} 2023-07-29T00:25:16.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010b6b97ad5bbe159-0064c45c6c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.467 DEBUG:boto:b'hiiqos-58obj000000000000000003171000falseobj000000000000000003172023-07-29T00:24:58.614Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:16.468 DEBUG:boto:path=/hiiqos-58/obj00000000000000000317 2023-07-29T00:25:16.469 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000317 2023-07-29T00:25:16.469 DEBUG:boto:Method: HEAD 2023-07-29T00:25:16.469 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000317 2023-07-29T00:25:16.470 DEBUG:boto:Data: 2023-07-29T00:25:16.470 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.470 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:16.471 DEBUG:boto:Port: 8001 2023-07-29T00:25:16.471 DEBUG:boto:Params: {} 2023-07-29T00:25:16.471 DEBUG:boto:Token: None 2023-07-29T00:25:16.472 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000317 2023-07-29T00:25:16.473 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KdGPlR0cSRObuBhitrY80BsWA7g= 2023-07-29T00:25:16.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KdGPlR0cSRObuBhitrY80BsWA7g=', 'Content-Length': '0'} 2023-07-29T00:25:16.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e186c313a6cb179a-0064c45c6c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.476 DEBUG:boto:path=/hiiqos-58/obj00000000000000000317 2023-07-29T00:25:16.476 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000317 2023-07-29T00:25:16.477 DEBUG:boto:Method: HEAD 2023-07-29T00:25:16.477 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000317 2023-07-29T00:25:16.477 DEBUG:boto:Data: 2023-07-29T00:25:16.478 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.478 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:16.478 DEBUG:boto:Port: 8000 2023-07-29T00:25:16.479 DEBUG:boto:Params: {} 2023-07-29T00:25:16.479 DEBUG:boto:Token: None 2023-07-29T00:25:16.479 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000317 2023-07-29T00:25:16.480 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KdGPlR0cSRObuBhitrY80BsWA7g= 2023-07-29T00:25:16.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KdGPlR0cSRObuBhitrY80BsWA7g=', 'Content-Length': '0'} 2023-07-29T00:25:16.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009454493301d4a56e-0064c45c6c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2023-07-29T00:25:16.484 DEBUG:boto:path=/hiiqos-58/obj00000000000000000317 2023-07-29T00:25:16.484 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000317 2023-07-29T00:25:16.485 DEBUG:boto:Method: GET 2023-07-29T00:25:16.486 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000317 2023-07-29T00:25:16.486 DEBUG:boto:Data: 2023-07-29T00:25:16.486 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.487 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:16.487 DEBUG:boto:Port: 8001 2023-07-29T00:25:16.487 DEBUG:boto:Params: {} 2023-07-29T00:25:16.488 DEBUG:boto:Token: None 2023-07-29T00:25:16.488 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000317 2023-07-29T00:25:16.489 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:v/YjQtrQwo8pzIXEhs0TUxfJpB8= 2023-07-29T00:25:16.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:v/YjQtrQwo8pzIXEhs0TUxfJpB8=', 'Content-Length': '0'} 2023-07-29T00:25:16.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075a16ab51a453838-0064c45c6c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.492 DEBUG:boto:path=/hiiqos-58/obj00000000000000000317 2023-07-29T00:25:16.492 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000317 2023-07-29T00:25:16.493 DEBUG:boto:Method: GET 2023-07-29T00:25:16.494 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000317 2023-07-29T00:25:16.494 DEBUG:boto:Data: 2023-07-29T00:25:16.494 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.495 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:16.495 DEBUG:boto:Port: 8000 2023-07-29T00:25:16.496 DEBUG:boto:Params: {} 2023-07-29T00:25:16.496 DEBUG:boto:Token: None 2023-07-29T00:25:16.497 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000317 2023-07-29T00:25:16.497 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:v/YjQtrQwo8pzIXEhs0TUxfJpB8= 2023-07-29T00:25:16.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:v/YjQtrQwo8pzIXEhs0TUxfJpB8=', 'Content-Length': '0'} 2023-07-29T00:25:16.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cabb5cce7ab35c9-0064c45c6c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.500 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:16.501 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:16.501 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000317 2023-07-29T00:25:16.501 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000317 2023-07-29T00:25:16.502 DEBUG:boto:Method: GET 2023-07-29T00:25:16.502 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000317 2023-07-29T00:25:16.503 DEBUG:boto:Data: 2023-07-29T00:25:16.503 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.504 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:16.504 DEBUG:boto:Port: 8001 2023-07-29T00:25:16.505 DEBUG:boto:Params: {} 2023-07-29T00:25:16.505 DEBUG:boto:Token: None 2023-07-29T00:25:16.506 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/ 2023-07-29T00:25:16.506 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ctvzq1HqhVpcvSTsF416pCLFISQ= 2023-07-29T00:25:16.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ctvzq1HqhVpcvSTsF416pCLFISQ=', 'Content-Length': '0'} 2023-07-29T00:25:16.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092222250b56f7e34-0064c45c6c-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.510 DEBUG:boto:b'hiiqos-58obj000000000000000003171000falseobj000000000000000003172023-07-29T00:24:58.614Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:16.511 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:16.511 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:16.511 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000317 2023-07-29T00:25:16.512 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000317 2023-07-29T00:25:16.512 DEBUG:boto:Method: GET 2023-07-29T00:25:16.512 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000317 2023-07-29T00:25:16.513 DEBUG:boto:Data: 2023-07-29T00:25:16.513 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.514 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:16.514 DEBUG:boto:Port: 8000 2023-07-29T00:25:16.519 DEBUG:boto:Params: {} 2023-07-29T00:25:16.520 DEBUG:boto:Token: None 2023-07-29T00:25:16.520 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/ 2023-07-29T00:25:16.521 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ctvzq1HqhVpcvSTsF416pCLFISQ= 2023-07-29T00:25:16.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ctvzq1HqhVpcvSTsF416pCLFISQ=', 'Content-Length': '0'} 2023-07-29T00:25:16.525 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011bcec3ccd7fd47a-0064c45c6c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.526 DEBUG:boto:b'hiiqos-58obj000000000000000003171000falseobj000000000000000003172023-07-29T00:24:58.614Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:16.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2023-07-29T00:25:16.527 DEBUG:boto:path=/hiiqos-58/obj00000000000000000423 2023-07-29T00:25:16.528 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000423 2023-07-29T00:25:16.528 DEBUG:boto:path=/hiiqos-58/obj00000000000000000423?versionId=null 2023-07-29T00:25:16.528 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000423?versionId=null 2023-07-29T00:25:16.529 DEBUG:boto:Method: GET 2023-07-29T00:25:16.529 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000423?versionId=null 2023-07-29T00:25:16.530 DEBUG:boto:Data: 2023-07-29T00:25:16.530 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.530 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:16.531 DEBUG:boto:Port: 8001 2023-07-29T00:25:16.531 DEBUG:boto:Params: {} 2023-07-29T00:25:16.531 DEBUG:boto:Token: None 2023-07-29T00:25:16.532 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000423?versionId=null 2023-07-29T00:25:16.532 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bPq1+dJKvJrPtrixT3zDhZOYub0= 2023-07-29T00:25:16.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bPq1+dJKvJrPtrixT3zDhZOYub0=', 'Content-Length': '0'} 2023-07-29T00:25:16.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000780b285068dc9c25-0064c45c6c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.536 DEBUG:boto:path=/hiiqos-58/obj00000000000000000423 2023-07-29T00:25:16.536 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000423 2023-07-29T00:25:16.537 DEBUG:boto:path=/hiiqos-58/obj00000000000000000423?versionId=null 2023-07-29T00:25:16.537 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000423?versionId=null 2023-07-29T00:25:16.537 DEBUG:boto:Method: GET 2023-07-29T00:25:16.538 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000423?versionId=null 2023-07-29T00:25:16.538 DEBUG:boto:Data: 2023-07-29T00:25:16.539 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.539 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:16.539 DEBUG:boto:Port: 8000 2023-07-29T00:25:16.540 DEBUG:boto:Params: {} 2023-07-29T00:25:16.540 DEBUG:boto:Token: None 2023-07-29T00:25:16.541 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000423?versionId=null 2023-07-29T00:25:16.541 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bPq1+dJKvJrPtrixT3zDhZOYub0= 2023-07-29T00:25:16.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bPq1+dJKvJrPtrixT3zDhZOYub0=', 'Content-Length': '0'} 2023-07-29T00:25:16.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f6c4f0c41c40a0f-0064c45c6c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.584 DEBUG:boto:path=/hiiqos-58/obj00000000000000000423 2023-07-29T00:25:16.585 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000423 2023-07-29T00:25:16.585 DEBUG:boto:path=/hiiqos-58/obj00000000000000000423?versionId=null 2023-07-29T00:25:16.586 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000423?versionId=null 2023-07-29T00:25:16.586 DEBUG:boto:Method: HEAD 2023-07-29T00:25:16.586 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000423?versionId=null 2023-07-29T00:25:16.587 DEBUG:boto:Data: 2023-07-29T00:25:16.587 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.587 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:16.588 DEBUG:boto:Port: 8001 2023-07-29T00:25:16.588 DEBUG:boto:Params: {} 2023-07-29T00:25:16.588 DEBUG:boto:Token: None 2023-07-29T00:25:16.589 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000423?versionId=null 2023-07-29T00:25:16.589 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZgwEdTtwADP6cGJQsPMZfRfdmsA= 2023-07-29T00:25:16.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZgwEdTtwADP6cGJQsPMZfRfdmsA=', 'Content-Length': '0'} 2023-07-29T00:25:16.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001d1c01ddbf0e48a-0064c45c6c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.595 DEBUG:boto:path=/hiiqos-58/obj00000000000000000423 2023-07-29T00:25:16.595 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000423 2023-07-29T00:25:16.596 DEBUG:boto:path=/hiiqos-58/obj00000000000000000423?versionId=null 2023-07-29T00:25:16.596 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000423?versionId=null 2023-07-29T00:25:16.596 DEBUG:boto:Method: HEAD 2023-07-29T00:25:16.597 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000423?versionId=null 2023-07-29T00:25:16.597 DEBUG:boto:Data: 2023-07-29T00:25:16.597 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.598 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:16.598 DEBUG:boto:Port: 8000 2023-07-29T00:25:16.598 DEBUG:boto:Params: {} 2023-07-29T00:25:16.599 DEBUG:boto:Token: None 2023-07-29T00:25:16.599 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000423?versionId=null 2023-07-29T00:25:16.600 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZgwEdTtwADP6cGJQsPMZfRfdmsA= 2023-07-29T00:25:16.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZgwEdTtwADP6cGJQsPMZfRfdmsA=', 'Content-Length': '0'} 2023-07-29T00:25:16.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffe08e305fafc41e-0064c45c6c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2023-07-29T00:25:16.604 DEBUG:boto:path=/hiiqos-58/obj00000000000000000423 2023-07-29T00:25:16.605 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000423 2023-07-29T00:25:16.605 DEBUG:boto:path=/hiiqos-58/obj00000000000000000423?versionId=null 2023-07-29T00:25:16.605 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000423?versionId=null 2023-07-29T00:25:16.606 DEBUG:boto:Method: GET 2023-07-29T00:25:16.606 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000423?versionId=null 2023-07-29T00:25:16.607 DEBUG:boto:Data: 2023-07-29T00:25:16.607 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.607 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:16.608 DEBUG:boto:Port: 8001 2023-07-29T00:25:16.608 DEBUG:boto:Params: {} 2023-07-29T00:25:16.609 DEBUG:boto:Token: None 2023-07-29T00:25:16.609 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000423?versionId=null 2023-07-29T00:25:16.610 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bPq1+dJKvJrPtrixT3zDhZOYub0= 2023-07-29T00:25:16.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bPq1+dJKvJrPtrixT3zDhZOYub0=', 'Content-Length': '0'} 2023-07-29T00:25:16.612 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c49a0962a757781f-0064c45c6c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.613 DEBUG:boto:path=/hiiqos-58/obj00000000000000000423 2023-07-29T00:25:16.614 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000423 2023-07-29T00:25:16.614 DEBUG:boto:path=/hiiqos-58/obj00000000000000000423?versionId=null 2023-07-29T00:25:16.614 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000423?versionId=null 2023-07-29T00:25:16.615 DEBUG:boto:Method: GET 2023-07-29T00:25:16.615 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000423?versionId=null 2023-07-29T00:25:16.615 DEBUG:boto:Data: 2023-07-29T00:25:16.616 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.616 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:16.616 DEBUG:boto:Port: 8000 2023-07-29T00:25:16.617 DEBUG:boto:Params: {} 2023-07-29T00:25:16.617 DEBUG:boto:Token: None 2023-07-29T00:25:16.618 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000423?versionId=null 2023-07-29T00:25:16.618 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bPq1+dJKvJrPtrixT3zDhZOYub0= 2023-07-29T00:25:16.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bPq1+dJKvJrPtrixT3zDhZOYub0=', 'Content-Length': '0'} 2023-07-29T00:25:16.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000127a3c9c6459ef57-0064c45c6c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.621 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:16.622 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:16.622 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000423 2023-07-29T00:25:16.622 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000423 2023-07-29T00:25:16.623 DEBUG:boto:Method: GET 2023-07-29T00:25:16.623 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000423 2023-07-29T00:25:16.623 DEBUG:boto:Data: 2023-07-29T00:25:16.624 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.624 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:16.625 DEBUG:boto:Port: 8001 2023-07-29T00:25:16.625 DEBUG:boto:Params: {} 2023-07-29T00:25:16.625 DEBUG:boto:Token: None 2023-07-29T00:25:16.626 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/ 2023-07-29T00:25:16.626 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ctvzq1HqhVpcvSTsF416pCLFISQ= 2023-07-29T00:25:16.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ctvzq1HqhVpcvSTsF416pCLFISQ=', 'Content-Length': '0'} 2023-07-29T00:25:16.629 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001faf47d3da6ca16e-0064c45c6c-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.672 DEBUG:boto:b'hiiqos-58obj000000000000000004231000falseobj000000000000000004232023-07-29T00:24:57.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:16.674 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:16.675 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:16.675 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000423 2023-07-29T00:25:16.675 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000423 2023-07-29T00:25:16.676 DEBUG:boto:Method: GET 2023-07-29T00:25:16.676 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000423 2023-07-29T00:25:16.677 DEBUG:boto:Data: 2023-07-29T00:25:16.678 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.678 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:16.678 DEBUG:boto:Port: 8000 2023-07-29T00:25:16.679 DEBUG:boto:Params: {} 2023-07-29T00:25:16.680 DEBUG:boto:Token: None 2023-07-29T00:25:16.681 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/ 2023-07-29T00:25:16.682 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ctvzq1HqhVpcvSTsF416pCLFISQ= 2023-07-29T00:25:16.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ctvzq1HqhVpcvSTsF416pCLFISQ=', 'Content-Length': '0'} 2023-07-29T00:25:16.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f5af200ba20e983-0064c45c6c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.689 DEBUG:boto:b'hiiqos-58obj000000000000000004231000falseobj000000000000000004232023-07-29T00:24:57.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:16.690 DEBUG:boto:path=/hiiqos-58/obj00000000000000000423 2023-07-29T00:25:16.690 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000423 2023-07-29T00:25:16.691 DEBUG:boto:Method: HEAD 2023-07-29T00:25:16.691 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000423 2023-07-29T00:25:16.691 DEBUG:boto:Data: 2023-07-29T00:25:16.692 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.692 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:16.692 DEBUG:boto:Port: 8001 2023-07-29T00:25:16.693 DEBUG:boto:Params: {} 2023-07-29T00:25:16.693 DEBUG:boto:Token: None 2023-07-29T00:25:16.693 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000423 2023-07-29T00:25:16.694 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9Q7mvOTaSCSFNqkHXX6Gdtw6crI= 2023-07-29T00:25:16.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9Q7mvOTaSCSFNqkHXX6Gdtw6crI=', 'Content-Length': '0'} 2023-07-29T00:25:16.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7531c18e56aa9a4-0064c45c6c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.698 DEBUG:boto:path=/hiiqos-58/obj00000000000000000423 2023-07-29T00:25:16.698 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000423 2023-07-29T00:25:16.698 DEBUG:boto:Method: HEAD 2023-07-29T00:25:16.699 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000423 2023-07-29T00:25:16.699 DEBUG:boto:Data: 2023-07-29T00:25:16.699 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.700 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:16.700 DEBUG:boto:Port: 8000 2023-07-29T00:25:16.702 DEBUG:boto:Params: {} 2023-07-29T00:25:16.703 DEBUG:boto:Token: None 2023-07-29T00:25:16.704 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000423 2023-07-29T00:25:16.704 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9Q7mvOTaSCSFNqkHXX6Gdtw6crI= 2023-07-29T00:25:16.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9Q7mvOTaSCSFNqkHXX6Gdtw6crI=', 'Content-Length': '0'} 2023-07-29T00:25:16.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001355b99574bf7b32-0064c45c6c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2023-07-29T00:25:16.709 DEBUG:boto:path=/hiiqos-58/obj00000000000000000423 2023-07-29T00:25:16.709 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000423 2023-07-29T00:25:16.709 DEBUG:boto:Method: GET 2023-07-29T00:25:16.710 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000423 2023-07-29T00:25:16.710 DEBUG:boto:Data: 2023-07-29T00:25:16.711 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.711 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:16.711 DEBUG:boto:Port: 8001 2023-07-29T00:25:16.712 DEBUG:boto:Params: {} 2023-07-29T00:25:16.712 DEBUG:boto:Token: None 2023-07-29T00:25:16.713 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000423 2023-07-29T00:25:16.713 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:D9YrqeuYiIa5VoPXakFuJOs6SE4= 2023-07-29T00:25:16.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:D9YrqeuYiIa5VoPXakFuJOs6SE4=', 'Content-Length': '0'} 2023-07-29T00:25:16.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d851ef96f54cf506-0064c45c6c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.717 DEBUG:boto:path=/hiiqos-58/obj00000000000000000423 2023-07-29T00:25:16.717 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000423 2023-07-29T00:25:16.718 DEBUG:boto:Method: GET 2023-07-29T00:25:16.719 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000423 2023-07-29T00:25:16.719 DEBUG:boto:Data: 2023-07-29T00:25:16.720 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.720 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:16.720 DEBUG:boto:Port: 8000 2023-07-29T00:25:16.721 DEBUG:boto:Params: {} 2023-07-29T00:25:16.721 DEBUG:boto:Token: None 2023-07-29T00:25:16.722 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000423 2023-07-29T00:25:16.722 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:D9YrqeuYiIa5VoPXakFuJOs6SE4= 2023-07-29T00:25:16.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:D9YrqeuYiIa5VoPXakFuJOs6SE4=', 'Content-Length': '0'} 2023-07-29T00:25:16.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:57 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063dbad2d471fc4ad-0064c45c6c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.725 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:16.726 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:16.726 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000423 2023-07-29T00:25:16.727 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000423 2023-07-29T00:25:16.727 DEBUG:boto:Method: GET 2023-07-29T00:25:16.727 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000423 2023-07-29T00:25:16.728 DEBUG:boto:Data: 2023-07-29T00:25:16.728 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.728 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:16.729 DEBUG:boto:Port: 8001 2023-07-29T00:25:16.729 DEBUG:boto:Params: {} 2023-07-29T00:25:16.730 DEBUG:boto:Token: None 2023-07-29T00:25:16.730 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/ 2023-07-29T00:25:16.730 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ctvzq1HqhVpcvSTsF416pCLFISQ= 2023-07-29T00:25:16.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ctvzq1HqhVpcvSTsF416pCLFISQ=', 'Content-Length': '0'} 2023-07-29T00:25:16.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c257feeb34e548c3-0064c45c6c-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.734 DEBUG:boto:b'hiiqos-58obj000000000000000004231000falseobj000000000000000004232023-07-29T00:24:57.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:16.734 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:16.735 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:16.735 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000423 2023-07-29T00:25:16.736 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000423 2023-07-29T00:25:16.736 DEBUG:boto:Method: GET 2023-07-29T00:25:16.736 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000423 2023-07-29T00:25:16.737 DEBUG:boto:Data: 2023-07-29T00:25:16.737 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.737 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:16.738 DEBUG:boto:Port: 8000 2023-07-29T00:25:16.738 DEBUG:boto:Params: {} 2023-07-29T00:25:16.739 DEBUG:boto:Token: None 2023-07-29T00:25:16.739 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/ 2023-07-29T00:25:16.739 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ctvzq1HqhVpcvSTsF416pCLFISQ= 2023-07-29T00:25:16.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ctvzq1HqhVpcvSTsF416pCLFISQ=', 'Content-Length': '0'} 2023-07-29T00:25:16.744 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4937a346a2d9b36-0064c45c6c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.784 DEBUG:boto:b'hiiqos-58obj000000000000000004231000falseobj000000000000000004232023-07-29T00:24:57.967Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:16.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2023-07-29T00:25:16.786 DEBUG:boto:path=/hiiqos-58/obj00000000000000000467 2023-07-29T00:25:16.786 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000467 2023-07-29T00:25:16.787 DEBUG:boto:path=/hiiqos-58/obj00000000000000000467?versionId=null 2023-07-29T00:25:16.787 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000467?versionId=null 2023-07-29T00:25:16.788 DEBUG:boto:Method: GET 2023-07-29T00:25:16.788 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000467?versionId=null 2023-07-29T00:25:16.789 DEBUG:boto:Data: 2023-07-29T00:25:16.789 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.789 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:16.790 DEBUG:boto:Port: 8001 2023-07-29T00:25:16.790 DEBUG:boto:Params: {} 2023-07-29T00:25:16.790 DEBUG:boto:Token: None 2023-07-29T00:25:16.791 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000467?versionId=null 2023-07-29T00:25:16.791 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FV8ntxI+9fmo93h1VICc4Qu82MM= 2023-07-29T00:25:16.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FV8ntxI+9fmo93h1VICc4Qu82MM=', 'Content-Length': '0'} 2023-07-29T00:25:16.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a45001a66fff3c1-0064c45c6c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.795 DEBUG:boto:path=/hiiqos-58/obj00000000000000000467 2023-07-29T00:25:16.795 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000467 2023-07-29T00:25:16.796 DEBUG:boto:path=/hiiqos-58/obj00000000000000000467?versionId=null 2023-07-29T00:25:16.796 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000467?versionId=null 2023-07-29T00:25:16.797 DEBUG:boto:Method: GET 2023-07-29T00:25:16.797 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000467?versionId=null 2023-07-29T00:25:16.798 DEBUG:boto:Data: 2023-07-29T00:25:16.798 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.798 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:16.799 DEBUG:boto:Port: 8000 2023-07-29T00:25:16.799 DEBUG:boto:Params: {} 2023-07-29T00:25:16.799 DEBUG:boto:Token: None 2023-07-29T00:25:16.800 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000467?versionId=null 2023-07-29T00:25:16.801 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FV8ntxI+9fmo93h1VICc4Qu82MM= 2023-07-29T00:25:16.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FV8ntxI+9fmo93h1VICc4Qu82MM=', 'Content-Length': '0'} 2023-07-29T00:25:16.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bdc308eda7fe69f-0064c45c6c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.804 DEBUG:boto:path=/hiiqos-58/obj00000000000000000467 2023-07-29T00:25:16.804 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000467 2023-07-29T00:25:16.805 DEBUG:boto:path=/hiiqos-58/obj00000000000000000467?versionId=null 2023-07-29T00:25:16.805 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000467?versionId=null 2023-07-29T00:25:16.806 DEBUG:boto:Method: HEAD 2023-07-29T00:25:16.806 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000467?versionId=null 2023-07-29T00:25:16.807 DEBUG:boto:Data: 2023-07-29T00:25:16.807 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.807 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:16.808 DEBUG:boto:Port: 8001 2023-07-29T00:25:16.808 DEBUG:boto:Params: {} 2023-07-29T00:25:16.808 DEBUG:boto:Token: None 2023-07-29T00:25:16.809 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000467?versionId=null 2023-07-29T00:25:16.809 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:um8G98ij5cfVxBWPB0TsqOgM6/s= 2023-07-29T00:25:16.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:um8G98ij5cfVxBWPB0TsqOgM6/s=', 'Content-Length': '0'} 2023-07-29T00:25:16.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f56f27f653987ea4-0064c45c6c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.812 DEBUG:boto:path=/hiiqos-58/obj00000000000000000467 2023-07-29T00:25:16.812 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000467 2023-07-29T00:25:16.813 DEBUG:boto:path=/hiiqos-58/obj00000000000000000467?versionId=null 2023-07-29T00:25:16.813 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000467?versionId=null 2023-07-29T00:25:16.813 DEBUG:boto:Method: HEAD 2023-07-29T00:25:16.814 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000467?versionId=null 2023-07-29T00:25:16.814 DEBUG:boto:Data: 2023-07-29T00:25:16.814 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.815 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:16.815 DEBUG:boto:Port: 8000 2023-07-29T00:25:16.815 DEBUG:boto:Params: {} 2023-07-29T00:25:16.816 DEBUG:boto:Token: None 2023-07-29T00:25:16.816 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000467?versionId=null 2023-07-29T00:25:16.816 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:um8G98ij5cfVxBWPB0TsqOgM6/s= 2023-07-29T00:25:16.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:um8G98ij5cfVxBWPB0TsqOgM6/s=', 'Content-Length': '0'} 2023-07-29T00:25:16.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008730b98b830a5b09-0064c45c6c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2023-07-29T00:25:16.820 DEBUG:boto:path=/hiiqos-58/obj00000000000000000467 2023-07-29T00:25:16.821 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000467 2023-07-29T00:25:16.821 DEBUG:boto:path=/hiiqos-58/obj00000000000000000467?versionId=null 2023-07-29T00:25:16.821 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000467?versionId=null 2023-07-29T00:25:16.822 DEBUG:boto:Method: GET 2023-07-29T00:25:16.822 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000467?versionId=null 2023-07-29T00:25:16.823 DEBUG:boto:Data: 2023-07-29T00:25:16.823 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.823 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:16.824 DEBUG:boto:Port: 8001 2023-07-29T00:25:16.824 DEBUG:boto:Params: {} 2023-07-29T00:25:16.825 DEBUG:boto:Token: None 2023-07-29T00:25:16.825 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000467?versionId=null 2023-07-29T00:25:16.826 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FV8ntxI+9fmo93h1VICc4Qu82MM= 2023-07-29T00:25:16.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FV8ntxI+9fmo93h1VICc4Qu82MM=', 'Content-Length': '0'} 2023-07-29T00:25:16.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9d0a48e5a873811-0064c45c6c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.829 DEBUG:boto:path=/hiiqos-58/obj00000000000000000467 2023-07-29T00:25:16.830 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000467 2023-07-29T00:25:16.830 DEBUG:boto:path=/hiiqos-58/obj00000000000000000467?versionId=null 2023-07-29T00:25:16.831 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000467?versionId=null 2023-07-29T00:25:16.831 DEBUG:boto:Method: GET 2023-07-29T00:25:16.831 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000467?versionId=null 2023-07-29T00:25:16.832 DEBUG:boto:Data: 2023-07-29T00:25:16.833 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.833 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:16.833 DEBUG:boto:Port: 8000 2023-07-29T00:25:16.834 DEBUG:boto:Params: {} 2023-07-29T00:25:16.834 DEBUG:boto:Token: None 2023-07-29T00:25:16.835 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000467?versionId=null 2023-07-29T00:25:16.835 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FV8ntxI+9fmo93h1VICc4Qu82MM= 2023-07-29T00:25:16.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FV8ntxI+9fmo93h1VICc4Qu82MM=', 'Content-Length': '0'} 2023-07-29T00:25:16.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bee2bead3534769-0064c45c6c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.839 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:16.839 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:16.840 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000467 2023-07-29T00:25:16.840 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000467 2023-07-29T00:25:16.841 DEBUG:boto:Method: GET 2023-07-29T00:25:16.841 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000467 2023-07-29T00:25:16.841 DEBUG:boto:Data: 2023-07-29T00:25:16.842 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.842 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:16.843 DEBUG:boto:Port: 8001 2023-07-29T00:25:16.843 DEBUG:boto:Params: {} 2023-07-29T00:25:16.844 DEBUG:boto:Token: None 2023-07-29T00:25:16.844 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/ 2023-07-29T00:25:16.845 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ctvzq1HqhVpcvSTsF416pCLFISQ= 2023-07-29T00:25:16.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ctvzq1HqhVpcvSTsF416pCLFISQ=', 'Content-Length': '0'} 2023-07-29T00:25:16.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061c9d25c0afd8f24-0064c45c6c-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.888 DEBUG:boto:b'hiiqos-58obj000000000000000004671000falseobj000000000000000004672023-07-29T00:24:58.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:16.890 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:16.890 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:16.890 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000467 2023-07-29T00:25:16.891 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000467 2023-07-29T00:25:16.891 DEBUG:boto:Method: GET 2023-07-29T00:25:16.892 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000467 2023-07-29T00:25:16.892 DEBUG:boto:Data: 2023-07-29T00:25:16.893 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.893 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:16.894 DEBUG:boto:Port: 8000 2023-07-29T00:25:16.894 DEBUG:boto:Params: {} 2023-07-29T00:25:16.895 DEBUG:boto:Token: None 2023-07-29T00:25:16.895 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/ 2023-07-29T00:25:16.896 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ctvzq1HqhVpcvSTsF416pCLFISQ= 2023-07-29T00:25:16.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ctvzq1HqhVpcvSTsF416pCLFISQ=', 'Content-Length': '0'} 2023-07-29T00:25:16.904 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c880e3ebe9f6ea9-0064c45c6c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.905 DEBUG:boto:b'hiiqos-58obj000000000000000004671000falseobj000000000000000004672023-07-29T00:24:58.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:16.906 DEBUG:boto:path=/hiiqos-58/obj00000000000000000467 2023-07-29T00:25:16.906 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000467 2023-07-29T00:25:16.907 DEBUG:boto:Method: HEAD 2023-07-29T00:25:16.907 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000467 2023-07-29T00:25:16.908 DEBUG:boto:Data: 2023-07-29T00:25:16.908 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.909 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:16.909 DEBUG:boto:Port: 8001 2023-07-29T00:25:16.909 DEBUG:boto:Params: {} 2023-07-29T00:25:16.910 DEBUG:boto:Token: None 2023-07-29T00:25:16.910 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000467 2023-07-29T00:25:16.911 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DNIGkdsOokUmOPl+NPAE0uUsxPU= 2023-07-29T00:25:16.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DNIGkdsOokUmOPl+NPAE0uUsxPU=', 'Content-Length': '0'} 2023-07-29T00:25:16.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e30205172733f8a2-0064c45c6c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.914 DEBUG:boto:path=/hiiqos-58/obj00000000000000000467 2023-07-29T00:25:16.915 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000467 2023-07-29T00:25:16.915 DEBUG:boto:Method: HEAD 2023-07-29T00:25:16.916 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000467 2023-07-29T00:25:16.916 DEBUG:boto:Data: 2023-07-29T00:25:16.917 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.918 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:16.918 DEBUG:boto:Port: 8000 2023-07-29T00:25:16.919 DEBUG:boto:Params: {} 2023-07-29T00:25:16.919 DEBUG:boto:Token: None 2023-07-29T00:25:16.920 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000467 2023-07-29T00:25:16.920 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DNIGkdsOokUmOPl+NPAE0uUsxPU= 2023-07-29T00:25:16.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DNIGkdsOokUmOPl+NPAE0uUsxPU=', 'Content-Length': '0'} 2023-07-29T00:25:16.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077f57b2ecb006be0-0064c45c6c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2023-07-29T00:25:16.926 DEBUG:boto:path=/hiiqos-58/obj00000000000000000467 2023-07-29T00:25:16.926 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000467 2023-07-29T00:25:16.927 DEBUG:boto:Method: GET 2023-07-29T00:25:16.928 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000467 2023-07-29T00:25:16.929 DEBUG:boto:Data: 2023-07-29T00:25:16.930 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.931 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:16.932 DEBUG:boto:Port: 8001 2023-07-29T00:25:16.932 DEBUG:boto:Params: {} 2023-07-29T00:25:16.933 DEBUG:boto:Token: None 2023-07-29T00:25:16.934 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000467 2023-07-29T00:25:16.935 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qeBoE6edLIg2wg5Xl4Dm14kS/g8= 2023-07-29T00:25:16.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qeBoE6edLIg2wg5Xl4Dm14kS/g8=', 'Content-Length': '0'} 2023-07-29T00:25:16.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-58:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082e34a6524c781fe-0064c45c6c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.938 DEBUG:boto:path=/hiiqos-58/obj00000000000000000467 2023-07-29T00:25:16.939 DEBUG:boto:auth_path=/hiiqos-58/obj00000000000000000467 2023-07-29T00:25:16.939 DEBUG:boto:Method: GET 2023-07-29T00:25:16.940 DEBUG:boto:Path: /hiiqos-58/obj00000000000000000467 2023-07-29T00:25:16.940 DEBUG:boto:Data: 2023-07-29T00:25:16.940 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.941 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:16.941 DEBUG:boto:Port: 8000 2023-07-29T00:25:16.942 DEBUG:boto:Params: {} 2023-07-29T00:25:16.942 DEBUG:boto:Token: None 2023-07-29T00:25:16.942 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/obj00000000000000000467 2023-07-29T00:25:16.943 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qeBoE6edLIg2wg5Xl4Dm14kS/g8= 2023-07-29T00:25:16.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qeBoE6edLIg2wg5Xl4Dm14kS/g8=', 'Content-Length': '0'} 2023-07-29T00:25:16.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:24:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccfa116ab735eb06-0064c45c6c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.946 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:16.946 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:16.947 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000467 2023-07-29T00:25:16.947 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000467 2023-07-29T00:25:16.948 DEBUG:boto:Method: GET 2023-07-29T00:25:16.949 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000467 2023-07-29T00:25:16.949 DEBUG:boto:Data: 2023-07-29T00:25:16.950 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.950 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:25:16.950 DEBUG:boto:Port: 8001 2023-07-29T00:25:16.951 DEBUG:boto:Params: {} 2023-07-29T00:25:16.952 DEBUG:boto:Token: None 2023-07-29T00:25:16.953 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/ 2023-07-29T00:25:16.954 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ctvzq1HqhVpcvSTsF416pCLFISQ= 2023-07-29T00:25:16.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ctvzq1HqhVpcvSTsF416pCLFISQ=', 'Content-Length': '0'} 2023-07-29T00:25:16.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069aa983f5739f3d6-0064c45c6c-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:16.959 DEBUG:boto:b'hiiqos-58obj000000000000000004671000falseobj000000000000000004672023-07-29T00:24:58.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:16.960 DEBUG:boto:path=/hiiqos-58/ 2023-07-29T00:25:16.961 DEBUG:boto:auth_path=/hiiqos-58/ 2023-07-29T00:25:16.961 DEBUG:boto:path=/hiiqos-58/?prefix=obj00000000000000000467 2023-07-29T00:25:16.962 DEBUG:boto:auth_path=/hiiqos-58/?prefix=obj00000000000000000467 2023-07-29T00:25:16.962 DEBUG:boto:Method: GET 2023-07-29T00:25:16.962 DEBUG:boto:Path: /hiiqos-58/?prefix=obj00000000000000000467 2023-07-29T00:25:16.963 DEBUG:boto:Data: 2023-07-29T00:25:16.963 DEBUG:boto:Headers: {} 2023-07-29T00:25:16.964 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:16.964 DEBUG:boto:Port: 8000 2023-07-29T00:25:16.964 DEBUG:boto:Params: {} 2023-07-29T00:25:16.965 DEBUG:boto:Token: None 2023-07-29T00:25:16.966 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:25:16 GMT /hiiqos-58/ 2023-07-29T00:25:16.966 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ctvzq1HqhVpcvSTsF416pCLFISQ= 2023-07-29T00:25:16.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ctvzq1HqhVpcvSTsF416pCLFISQ=', 'Content-Length': '0'} 2023-07-29T00:25:16.971 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce39cbc0affbcb2a-0064c45c6c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:17.016 DEBUG:boto:b'hiiqos-58obj000000000000000004671000falseobj000000000000000004672023-07-29T00:24:58.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:25:17.017 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-58 zones={a2, a1} 2023-07-29T00:25:17.017 INFO:tasks.rgw_multisite_tests:Create several generations of objects, then run bucket sync ... ok 2023-07-29T00:25:17.019 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:25:17.020 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:25:17.020 DEBUG:boto:Method: PUT 2023-07-29T00:25:17.020 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:25:17.021 DEBUG:boto:Data: 2023-07-29T00:25:17.021 DEBUG:boto:Headers: {} 2023-07-29T00:25:17.021 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:25:17.022 DEBUG:boto:Port: 8000 2023-07-29T00:25:17.022 DEBUG:boto:Params: {} 2023-07-29T00:25:17.022 DEBUG:boto:Token: None 2023-07-29T00:25:17.023 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 00:25:17 GMT /hiiqos-59/ 2023-07-29T00:25:17.023 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kWqYpwWJvZR2dM15eoo9Y8N0buA= 2023-07-29T00:25:17.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:25:17 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kWqYpwWJvZR2dM15eoo9Y8N0buA=', 'Content-Length': '0'} 2023-07-29T00:25:17.041 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005e235800cc415a72-0064c45c6c-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:25:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:25:17.042 DEBUG:rgw_multi.tests:created bucket=hiiqos-59 2023-07-29T00:25:17.043 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:25:17.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:25:17.043 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:25:17.093 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:25:17.093 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:25:17.124 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T00:25:17.138 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T00:25:17.138 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:17.138 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:25:17.138 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-28T23:46:09.233742Z" 2023-07-29T00:25:17.139 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:17.139 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:17.139 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690590316.988705_997.1", 2023-07-29T00:25:17.139 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:25:16.988705Z" 2023-07-29T00:25:17.139 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:17.139 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:17.140 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:25:17.140 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T00:25:17.140 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:17.140 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:17.140 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:25:17.140 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T00:25:17.140 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:17.141 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T00:25:17.141 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690587969.233742_99.1', 1: '1_1690590316.988705_997.1', 2: '', 3: ''} 2023-07-29T00:25:17.142 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T00:25:17.142 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:25:17.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:25:17.143 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:25:17.190 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:25:17.190 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:25:17.219 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:17.166+0000 7f81cc016880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:25:17.219 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:17.167+0000 7f81cc016880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:25:17.232 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:25:17.233 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:25:17.233 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:25:17.233 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:25:17.233 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:25:17.234 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:25:17.234 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:17.234 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:17.234 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:25:17.234 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:17.236 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:25:17.236 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:25:17.236 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:25:17.236 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:25:17.237 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:25:17.237 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:25:17.237 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:25:17.237 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:25:17.237 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:17.237 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:17.238 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:17.238 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:17.238 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:25:17.238 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:25:17.238 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:25:17.238 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690590264.216448_923.1", 2023-07-29T00:25:17.239 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:25:17.239 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:25:17.239 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:25:17.239 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:24:24.216448Z", 2023-07-29T00:25:17.239 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:17.239 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:17.239 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:17.240 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:17.240 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:25:17.240 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:25:17.240 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:25:17.240 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:25:17.240 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:25:17.241 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:25:17.241 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:25:17.241 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:25:17.241 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:17.241 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:17.241 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:17.242 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:17.242 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:25:17.242 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:25:17.242 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:25:17.242 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:25:17.242 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:25:17.243 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:25:17.243 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:25:17.243 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:25:17.243 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:17.243 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:17.244 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:17.244 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:25:17.244 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:17.244 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:25:17.245 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:25:17.245 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:25:17.245 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:17.245 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:25:17Z" 2023-07-29T00:25:17.245 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:25:17.246 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690590264.216448_923.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:24:24.216448Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:25:17Z" } 2023-07-29T00:25:17.247 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690590264.216448_923.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:24:24.216448Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:25:17.247 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690590316.988705_997.1', 2: '', 3: ''} 2023-07-29T00:25:17.248 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690590264.216448_923.1', 2: '', 3: ''} 2023-07-29T00:25:17.248 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690590316.988705_997.1 target=1_1690590264.216448_923.1 2023-07-29T00:25:22.250 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:25:22.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:25:22.251 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:25:22.304 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:25:22.305 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:25:22.337 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:22.284+0000 7fbe60016880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:25:22.338 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:22.285+0000 7fbe60016880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:25:22.352 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:25:22.352 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:25:22.353 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:25:22.353 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:25:22.353 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:25:22.353 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:25:22.353 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:22.353 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:22.353 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:25:22.354 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:22.354 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:25:22.354 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:25:22.354 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:25:22.354 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:25:22.354 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:25:22.355 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:25:22.355 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:25:22.355 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:25:22.355 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:22.355 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:22.355 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:22.355 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:22.356 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:25:22.356 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:25:22.356 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:25:22.356 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690590264.216448_923.1", 2023-07-29T00:25:22.356 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:25:22.356 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:25:22.357 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:25:22.357 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:24:24.216448Z", 2023-07-29T00:25:22.357 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:22.357 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:22.357 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:22.357 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:22.357 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:25:22.358 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:25:22.358 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:25:22.358 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:25:22.358 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:25:22.358 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:25:22.358 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:25:22.358 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:25:22.359 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:22.359 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:22.359 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:22.359 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:22.359 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:25:22.359 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:25:22.360 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:25:22.360 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:25:22.360 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:25:22.360 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:25:22.360 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:25:22.360 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:25:22.360 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:22.361 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:22.361 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:22.361 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:25:22.361 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:22.361 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:25:22.361 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:25:22.361 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:25:22.362 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:22.362 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:25:22Z" 2023-07-29T00:25:22.362 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:25:22.363 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690590264.216448_923.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:24:24.216448Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:25:22Z" } 2023-07-29T00:25:22.363 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690590264.216448_923.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:24:24.216448Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:25:22.364 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690590316.988705_997.1', 2: '', 3: ''} 2023-07-29T00:25:22.364 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690590264.216448_923.1', 2: '', 3: ''} 2023-07-29T00:25:22.364 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690590316.988705_997.1 target=1_1690590264.216448_923.1 2023-07-29T00:25:27.366 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:25:27.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:25:27.368 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:25:27.421 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:25:27.422 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:25:27.454 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:27.401+0000 7f061912d880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:25:27.455 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:27.402+0000 7f061912d880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:25:27.468 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:25:27.469 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:25:27.469 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:25:27.469 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:25:27.469 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:25:27.469 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:25:27.470 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:27.470 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:27.470 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:25:27.470 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:27.470 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:25:27.470 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:25:27.470 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:25:27.471 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:25:27.471 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:25:27.471 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:25:27.471 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:25:27.471 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:25:27.471 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:27.471 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:27.472 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:27.472 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:27.472 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:25:27.472 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:25:27.472 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:25:27.472 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690590264.216448_923.1", 2023-07-29T00:25:27.473 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:25:27.473 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:25:27.473 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:25:27.473 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:24:24.216448Z", 2023-07-29T00:25:27.473 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:27.473 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:27.474 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:27.474 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:27.474 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:25:27.474 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:25:27.474 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:25:27.474 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:25:27.474 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:25:27.475 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:25:27.475 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:25:27.475 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:25:27.475 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:27.475 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:27.476 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:27.476 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:27.476 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:25:27.476 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:25:27.476 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:25:27.477 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:25:27.477 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:25:27.477 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:25:27.477 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:25:27.477 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:25:27.477 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:27.477 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:27.478 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:27.478 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:25:27.478 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:27.478 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:25:27.479 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:25:27.479 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:25:27.479 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:27.479 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:25:27Z" 2023-07-29T00:25:27.479 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:25:27.480 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690590264.216448_923.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:24:24.216448Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:25:27Z" } 2023-07-29T00:25:27.481 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690590264.216448_923.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:24:24.216448Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:25:27.481 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690590316.988705_997.1', 2: '', 3: ''} 2023-07-29T00:25:27.481 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690590264.216448_923.1', 2: '', 3: ''} 2023-07-29T00:25:27.482 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690590316.988705_997.1 target=1_1690590264.216448_923.1 2023-07-29T00:25:32.482 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:25:32.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:25:32.484 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:25:32.539 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:25:32.539 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:25:32.571 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:32.518+0000 7f227192e880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:25:32.572 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:32.519+0000 7f227192e880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:25:32.586 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:25:32.587 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:25:32.587 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:25:32.587 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:25:32.587 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:25:32.587 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:25:32.587 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:32.588 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:32.588 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:25:32.588 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:32.588 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:25:32.588 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:25:32.588 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:25:32.588 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:25:32.589 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:25:32.589 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:25:32.589 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:25:32.589 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:25:32.589 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:32.589 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:32.589 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:32.590 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:32.590 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:25:32.590 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:25:32.590 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:25:32.590 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690590264.216448_923.1", 2023-07-29T00:25:32.590 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:25:32.591 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:25:32.591 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:25:32.591 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:24:24.216448Z", 2023-07-29T00:25:32.591 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:32.591 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:32.591 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:32.591 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:32.592 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:25:32.592 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:25:32.592 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:25:32.592 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:25:32.592 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:25:32.592 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:25:32.592 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:25:32.593 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:25:32.593 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:32.593 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:32.593 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:32.593 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:32.593 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:25:32.594 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:25:32.594 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:25:32.594 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:25:32.594 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:25:32.594 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:25:32.594 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:25:32.595 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:25:32.595 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:32.595 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:32.595 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:32.595 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:25:32.595 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:32.595 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:25:32.596 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:25:32.596 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:25:32.596 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:32.596 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:25:32Z" 2023-07-29T00:25:32.596 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:25:32.597 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690590264.216448_923.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:24:24.216448Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:25:32Z" } 2023-07-29T00:25:32.598 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690590264.216448_923.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:24:24.216448Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:25:32.599 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690590316.988705_997.1', 2: '', 3: ''} 2023-07-29T00:25:32.599 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690590264.216448_923.1', 2: '', 3: ''} 2023-07-29T00:25:32.600 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690590316.988705_997.1 target=1_1690590264.216448_923.1 2023-07-29T00:25:37.601 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:25:37.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:25:37.602 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:25:37.655 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:25:37.655 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:25:37.687 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:37.634+0000 7fab62522880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:25:37.688 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:37.635+0000 7fab62522880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:25:37.701 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:25:37.702 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:25:37.702 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:25:37.702 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:25:37.702 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:25:37.702 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:25:37.703 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:37.703 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:37.703 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:25:37.703 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:37.703 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:25:37.703 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:25:37.703 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:25:37.704 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:25:37.704 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:25:37.704 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:25:37.704 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:25:37.704 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:25:37.704 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:37.705 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:37.705 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:37.705 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:37.705 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:25:37.705 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:25:37.705 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:25:37.705 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690590264.216448_923.1", 2023-07-29T00:25:37.706 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:25:37.706 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:25:37.706 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:25:37.706 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:24:24.216448Z", 2023-07-29T00:25:37.706 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:37.706 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:37.706 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:37.707 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:37.707 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:25:37.707 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:25:37.707 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:25:37.707 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:25:37.707 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:25:37.707 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:25:37.708 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:25:37.708 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:25:37.708 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:37.708 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:37.708 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:37.708 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:37.709 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:25:37.709 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:25:37.709 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:25:37.709 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:25:37.709 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:25:37.709 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:25:37.709 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:25:37.710 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:25:37.710 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:37.710 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:37.710 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:37.710 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:25:37.710 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:37.710 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:25:37.711 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:25:37.711 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:25:37.711 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:37.711 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:25:37Z" 2023-07-29T00:25:37.711 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:25:37.712 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690590264.216448_923.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:24:24.216448Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:25:37Z" } 2023-07-29T00:25:37.713 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690590264.216448_923.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:24:24.216448Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:25:37.713 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690590316.988705_997.1', 2: '', 3: ''} 2023-07-29T00:25:37.713 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690590264.216448_923.1', 2: '', 3: ''} 2023-07-29T00:25:37.714 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690590316.988705_997.1 target=1_1690590264.216448_923.1 2023-07-29T00:25:38.096 INFO:tasks.rgw.c2.client.1.smithi139.stdout:2023-07-29T00:25:38.043+0000 7f5731a6a640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T00:25:42.715 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:25:42.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:25:42.716 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:25:42.770 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:25:42.771 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:25:42.803 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:42.750+0000 7f869df11880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:25:42.804 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:42.751+0000 7f869df11880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:25:42.817 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:25:42.818 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:25:42.818 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:25:42.818 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:25:42.818 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:25:42.819 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:25:42.819 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:42.819 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:42.819 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:25:42.819 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:42.819 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:25:42.819 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:25:42.820 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:25:42.820 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:25:42.820 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:25:42.820 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:25:42.820 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:25:42.820 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:25:42.821 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:42.821 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:42.821 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:42.821 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:42.821 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:25:42.821 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:25:42.822 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:25:42.822 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690590264.216448_923.1", 2023-07-29T00:25:42.822 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:25:42.822 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:25:42.822 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:25:42.822 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:24:24.216448Z", 2023-07-29T00:25:42.822 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:42.823 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:42.823 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:42.823 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:42.823 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:25:42.823 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:25:42.823 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:25:42.823 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:25:42.824 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:25:42.824 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:25:42.824 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:25:42.824 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:25:42.824 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:42.824 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:42.825 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:42.825 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:42.825 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:25:42.825 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:25:42.825 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:25:42.825 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:25:42.825 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:25:42.826 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:25:42.826 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:25:42.826 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:25:42.826 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:42.826 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:42.826 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:42.826 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:25:42.827 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:42.827 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:25:42.827 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:25:42.827 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:25:42.827 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:42.827 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:25:42Z" 2023-07-29T00:25:42.828 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:25:42.829 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690590264.216448_923.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:24:24.216448Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:25:42Z" } 2023-07-29T00:25:42.829 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690590264.216448_923.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:24:24.216448Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:25:42.829 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690590316.988705_997.1', 2: '', 3: ''} 2023-07-29T00:25:42.830 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690590264.216448_923.1', 2: '', 3: ''} 2023-07-29T00:25:42.830 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690590316.988705_997.1 target=1_1690590264.216448_923.1 2023-07-29T00:25:47.831 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:25:47.832 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:25:47.832 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:25:47.887 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:25:47.887 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:25:47.920 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:47.867+0000 7f923ac22880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:25:47.921 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:47.868+0000 7f923ac22880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:25:47.935 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:25:47.935 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:25:47.936 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:25:47.936 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:25:47.936 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:25:47.936 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:25:47.936 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:47.936 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:47.937 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:25:47.937 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:47.937 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:25:47.937 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:25:47.937 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:25:47.937 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:25:47.937 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:25:47.938 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:25:47.938 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:25:47.938 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:25:47.938 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:47.938 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:47.938 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:47.939 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:47.939 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:25:47.939 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:25:47.939 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:25:47.939 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690590264.216448_923.1", 2023-07-29T00:25:47.939 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:25:47.939 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:25:47.940 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:25:47.940 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:24:24.216448Z", 2023-07-29T00:25:47.940 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:47.940 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:47.940 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:47.940 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:47.940 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:25:47.941 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:25:47.941 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:25:47.941 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:25:47.941 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:25:47.941 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:25:47.941 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:25:47.941 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:25:47.942 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:47.942 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:47.942 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:47.942 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:47.942 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:25:47.942 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:25:47.942 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:25:47.943 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:25:47.943 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:25:47.943 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:25:47.943 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:25:47.943 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:25:47.943 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:47.944 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:47.944 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:47.944 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:25:47.944 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:47.944 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:25:47.944 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:25:47.944 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:25:47.945 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:47.945 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:25:47Z" 2023-07-29T00:25:47.945 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:25:47.946 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690590264.216448_923.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:24:24.216448Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:25:47Z" } 2023-07-29T00:25:47.946 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690590264.216448_923.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:24:24.216448Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:25:47.947 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690590316.988705_997.1', 2: '', 3: ''} 2023-07-29T00:25:47.947 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690590264.216448_923.1', 2: '', 3: ''} 2023-07-29T00:25:47.947 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690590316.988705_997.1 target=1_1690590264.216448_923.1 2023-07-29T00:25:52.948 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:25:52.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:25:52.949 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:25:53.008 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:25:53.009 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:25:53.037 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:52.985+0000 7fa0ff722880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:25:53.039 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:52.986+0000 7fa0ff722880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:25:53.055 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:25:53.056 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:25:53.056 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:25:53.056 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:25:53.056 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:25:53.057 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:25:53.057 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:53.057 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:53.058 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:25:53.058 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:53.058 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:25:53.058 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:25:53.058 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:25:53.058 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:25:53.058 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:25:53.059 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:25:53.059 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:25:53.059 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:25:53.059 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:53.059 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:53.059 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:53.060 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:53.060 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:25:53.060 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:25:53.060 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:25:53.060 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690590264.216448_923.1", 2023-07-29T00:25:53.060 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:25:53.061 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:25:53.061 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:25:53.061 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:24:24.216448Z", 2023-07-29T00:25:53.061 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:53.061 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:53.061 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:53.061 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:53.062 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:25:53.062 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:25:53.062 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:25:53.062 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:25:53.062 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:25:53.062 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:25:53.062 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:25:53.063 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:25:53.063 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:53.063 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:53.063 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:53.064 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:53.064 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:25:53.064 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:25:53.064 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:25:53.064 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:25:53.064 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:25:53.065 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:25:53.065 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:25:53.065 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:25:53.065 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:53.065 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:53.065 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:53.065 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:25:53.066 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:53.066 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:25:53.066 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:25:53.066 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:25:53.066 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:53.066 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:25:52Z" 2023-07-29T00:25:53.066 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:25:53.067 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690590264.216448_923.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:24:24.216448Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:25:52Z" } 2023-07-29T00:25:53.068 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690590264.216448_923.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:24:24.216448Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:25:53.068 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690590316.988705_997.1', 2: '', 3: ''} 2023-07-29T00:25:53.069 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690590264.216448_923.1', 2: '', 3: ''} 2023-07-29T00:25:53.069 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690590316.988705_997.1 target=1_1690590264.216448_923.1 2023-07-29T00:25:58.070 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:25:58.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:25:58.071 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:25:58.126 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:25:58.126 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:25:58.159 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:58.106+0000 7fef10012880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:25:58.159 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:25:58.107+0000 7fef10012880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:25:58.173 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:25:58.174 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:25:58.174 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:25:58.174 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:25:58.174 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:25:58.174 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:25:58.175 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:58.175 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:58.175 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:25:58.175 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:58.175 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:25:58.175 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:25:58.175 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:25:58.176 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:25:58.176 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:25:58.176 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:25:58.176 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:25:58.176 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:25:58.176 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:58.177 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:58.177 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:58.177 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:58.177 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:25:58.177 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:25:58.177 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:25:58.177 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690590264.216448_923.1", 2023-07-29T00:25:58.178 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:25:58.178 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:25:58.178 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:25:58.178 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:24:24.216448Z", 2023-07-29T00:25:58.178 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:58.178 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:58.178 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:58.179 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:58.179 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:25:58.179 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:25:58.179 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:25:58.179 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:25:58.179 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:25:58.179 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:25:58.180 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:25:58.180 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:25:58.180 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:58.180 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:58.180 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:58.180 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:25:58.181 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:25:58.181 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:25:58.181 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:25:58.181 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:25:58.181 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:25:58.181 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:25:58.181 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:25:58.182 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:25:58.182 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:25:58.182 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:58.182 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:25:58.182 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:25:58.182 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:58.182 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:25:58.183 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:25:58.183 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:25:58.183 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:25:58.183 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:25:58Z" 2023-07-29T00:25:58.183 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:25:58.184 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690590264.216448_923.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:24:24.216448Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:25:58Z" } 2023-07-29T00:25:58.185 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690590264.216448_923.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:24:24.216448Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:25:58.185 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690590316.988705_997.1', 2: '', 3: ''} 2023-07-29T00:25:58.185 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690590264.216448_923.1', 2: '', 3: ''} 2023-07-29T00:25:58.186 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690590316.988705_997.1 target=1_1690590264.216448_923.1 2023-07-29T00:26:02.905 INFO:tasks.rgw.c2.client.0.smithi139.stdout:2023-07-29T00:26:02.853+0000 7f5f2671b640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T00:26:03.187 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:26:03.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:26:03.188 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:26:03.242 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:26:03.242 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:26:03.275 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:26:03.221+0000 7f719a922880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:26:03.275 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:26:03.222+0000 7f719a922880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:26:03.290 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:26:03.290 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:26:03.291 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:26:03.291 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:26:03.291 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:26:03.291 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:26:03.291 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:03.291 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:03.291 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:26:03.292 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:03.292 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:26:03.292 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:03.292 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:03.292 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:26:03.292 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:03.293 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:26:03.293 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:03.293 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:26:03.293 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:03.293 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:03.293 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:03.293 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:03.294 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:26:03.294 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:03.294 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:03.294 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690590264.216448_923.1", 2023-07-29T00:26:03.294 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:03.294 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:26:03.294 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:03.295 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:24:24.216448Z", 2023-07-29T00:26:03.295 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:03.295 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:03.295 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:03.295 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:03.295 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:26:03.296 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:03.296 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:03.296 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:26:03.296 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:03.296 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:26:03.296 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:03.296 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:26:03.297 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:03.297 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:03.297 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:03.297 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:03.297 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:26:03.297 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:03.297 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:03.298 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:26:03.298 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:03.298 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:26:03.298 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:03.298 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:26:03.298 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:03.298 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:03.299 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:03.299 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:26:03.299 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:03.299 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:26:03.299 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:26:03.299 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:26:03.299 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:03.300 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:26:03Z" 2023-07-29T00:26:03.300 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:26:03.301 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690590264.216448_923.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:24:24.216448Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:26:03Z" } 2023-07-29T00:26:03.301 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690590264.216448_923.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:24:24.216448Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:26:03.302 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690590316.988705_997.1', 2: '', 3: ''} 2023-07-29T00:26:03.302 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690590264.216448_923.1', 2: '', 3: ''} 2023-07-29T00:26:03.302 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690590316.988705_997.1 target=1_1690590264.216448_923.1 2023-07-29T00:26:08.303 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:26:08.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:26:08.304 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:26:08.357 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:26:08.358 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:26:08.390 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:26:08.336+0000 7f1850749880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:26:08.390 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:26:08.337+0000 7f1850749880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:26:08.404 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:26:08.405 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:26:08.405 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:26:08.405 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:26:08.405 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:26:08.406 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:26:08.406 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:08.406 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:08.406 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:26:08.406 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:08.406 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:26:08.407 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:08.407 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:08.407 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:26:08.407 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:08.407 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:26:08.407 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:08.407 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:26:08.408 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:08.408 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:08.408 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:08.408 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:08.408 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:26:08.408 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:08.409 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:08.409 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690590264.216448_923.1", 2023-07-29T00:26:08.409 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:08.409 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:26:08.409 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:08.409 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:24:24.216448Z", 2023-07-29T00:26:08.410 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:08.410 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:08.410 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:08.410 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:08.410 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:26:08.410 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:08.411 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:08.411 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:26:08.411 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:08.411 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:26:08.411 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:08.411 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:26:08.411 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:08.412 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:08.412 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:08.412 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:08.412 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:26:08.412 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:08.412 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:08.413 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:26:08.413 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:08.413 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:26:08.413 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:08.413 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:26:08.413 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:08.414 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:08.414 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:08.414 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:26:08.414 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:08.414 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:26:08.414 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:26:08.414 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:26:08.415 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:08.415 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:26:08Z" 2023-07-29T00:26:08.415 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:26:08.416 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690590264.216448_923.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:24:24.216448Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:26:08Z" } 2023-07-29T00:26:08.416 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690590264.216448_923.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:24:24.216448Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:26:08.417 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690590316.988705_997.1', 2: '', 3: ''} 2023-07-29T00:26:08.417 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690590264.216448_923.1', 2: '', 3: ''} 2023-07-29T00:26:08.418 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690590316.988705_997.1 target=1_1690590264.216448_923.1 2023-07-29T00:26:13.418 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:26:13.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:26:13.419 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:26:13.475 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:26:13.476 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:26:13.507 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:26:13.454+0000 7f84efb28880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:26:13.508 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:26:13.455+0000 7f84efb28880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:26:13.521 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:26:13.522 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:26:13.522 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:26:13.522 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:26:13.522 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:26:13.522 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:26:13.523 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:13.523 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:13.523 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:26:13.523 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:13.523 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:26:13.523 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:13.523 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:13.524 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:26:13.524 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:13.524 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:26:13.524 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:13.524 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:26:13.524 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:13.525 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:13.525 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:13.525 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:13.525 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:26:13.525 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:13.525 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:13.526 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690590264.216448_923.1", 2023-07-29T00:26:13.526 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:13.526 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:26:13.526 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:13.526 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:24:24.216448Z", 2023-07-29T00:26:13.526 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:13.527 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:13.527 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:13.527 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:13.527 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:26:13.527 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:13.528 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:13.528 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:26:13.528 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:13.528 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:26:13.528 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:13.529 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:26:13.529 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:13.529 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:13.529 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:13.529 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:13.529 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:26:13.530 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:13.530 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:13.530 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:26:13.530 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:13.530 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:26:13.530 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:13.530 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:26:13.531 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:13.531 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:13.531 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:13.531 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:26:13.531 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:13.531 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:26:13.531 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:26:13.532 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:26:13.532 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:13.532 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:26:13Z" 2023-07-29T00:26:13.532 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:26:13.533 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690590264.216448_923.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:24:24.216448Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:26:13Z" } 2023-07-29T00:26:13.534 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690590264.216448_923.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:24:24.216448Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:26:13.534 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690590316.988705_997.1', 2: '', 3: ''} 2023-07-29T00:26:13.535 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690590264.216448_923.1', 2: '', 3: ''} 2023-07-29T00:26:13.535 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690590316.988705_997.1 target=1_1690590264.216448_923.1 2023-07-29T00:26:18.536 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:26:18.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:26:18.538 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:26:18.590 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:26:18.591 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:26:18.623 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:26:18.569+0000 7fdde0f35880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:26:18.623 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:26:18.570+0000 7fdde0f35880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:26:18.638 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:26:18.638 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:26:18.638 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:26:18.638 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:26:18.639 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:26:18.639 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:26:18.639 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:18.639 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:18.639 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:26:18.640 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:18.640 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:26:18.640 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:18.640 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:18.640 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:26:18.641 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:18.641 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:26:18.641 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:18.641 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:26:18.642 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:18.642 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:18.642 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:18.642 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:18.642 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:26:18.643 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:18.643 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:18.643 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690590264.216448_923.1", 2023-07-29T00:26:18.643 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:18.643 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:26:18.644 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:18.644 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:24:24.216448Z", 2023-07-29T00:26:18.644 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:18.644 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:18.644 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:18.644 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:18.645 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:26:18.645 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:18.645 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:18.645 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:26:18.646 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:18.646 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:26:18.646 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:18.647 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:26:18.647 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:18.647 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:18.648 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:18.648 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:18.648 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:26:18.649 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:18.649 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:18.649 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:26:18.650 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:18.650 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:26:18.650 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:18.651 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:26:18.651 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:18.652 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:18.652 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:18.652 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:26:18.652 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:18.653 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:26:18.653 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:26:18.653 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:26:18.653 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:18.654 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:26:18Z" 2023-07-29T00:26:18.654 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:26:18.656 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690590264.216448_923.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:24:24.216448Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:26:18Z" } 2023-07-29T00:26:18.656 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690590264.216448_923.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:24:24.216448Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:26:18.657 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690590316.988705_997.1', 2: '', 3: ''} 2023-07-29T00:26:18.657 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690590264.216448_923.1', 2: '', 3: ''} 2023-07-29T00:26:18.657 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690590316.988705_997.1 target=1_1690590264.216448_923.1 2023-07-29T00:26:23.660 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:26:23.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:26:23.661 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:26:23.715 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:26:23.716 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:26:23.747 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:26:23.694+0000 7fc576e24880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:26:23.748 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:26:23.695+0000 7fc576e24880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:26:23.762 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:26:23.762 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:26:23.763 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:26:23.763 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:26:23.763 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:26:23.763 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:26:23.763 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:23.763 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:23.764 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:26:23.764 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:23.764 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:26:23.764 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:23.764 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:23.764 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:26:23.765 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:23.766 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:26:23.766 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:23.766 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:26:23.767 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:23.767 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:23.767 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:23.767 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:23.768 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:26:23.768 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:23.768 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:23.768 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690590264.216448_923.1", 2023-07-29T00:26:23.768 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:23.768 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:26:23.769 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:23.769 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:24:24.216448Z", 2023-07-29T00:26:23.769 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:23.769 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:23.769 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:23.769 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:23.769 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:26:23.770 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:23.770 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:23.770 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:26:23.770 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:23.770 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:26:23.770 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:23.770 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:26:23.771 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:23.771 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:23.771 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:23.771 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:23.771 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:26:23.772 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:23.772 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:23.772 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:26:23.772 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:23.772 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:26:23.773 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:23.773 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:26:23.773 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:23.773 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:23.773 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:23.773 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:26:23.774 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:23.774 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:26:23.774 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:26:23.774 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:26:23.774 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:23.775 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:26:23Z" 2023-07-29T00:26:23.775 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:26:23.777 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690590264.216448_923.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:24:24.216448Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:26:23Z" } 2023-07-29T00:26:23.778 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690590264.216448_923.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:24:24.216448Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:26:23.779 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690590316.988705_997.1', 2: '', 3: ''} 2023-07-29T00:26:23.780 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690590264.216448_923.1', 2: '', 3: ''} 2023-07-29T00:26:23.781 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690590316.988705_997.1 target=1_1690590264.216448_923.1 2023-07-29T00:26:28.782 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:26:28.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:26:28.783 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:26:28.839 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:26:28.839 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:26:28.871 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:26:28.817+0000 7fb97ff29880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:26:28.872 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:26:28.818+0000 7fb97ff29880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:26:28.885 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:26:28.886 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:26:28.886 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:26:28.886 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:26:28.886 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:26:28.886 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:26:28.887 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:28.887 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:28.887 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:26:28.887 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:28.887 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:26:28.887 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:28.888 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:28.888 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:26:28.888 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:28.888 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:26:28.888 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:28.888 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:26:28.889 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:28.889 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:28.889 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:28.889 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:28.889 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:26:28.890 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:28.890 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:28.890 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690590264.216448_923.1", 2023-07-29T00:26:28.890 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:28.890 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:26:28.890 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:28.891 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:24:24.216448Z", 2023-07-29T00:26:28.891 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:28.891 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:28.891 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:28.891 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:28.891 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:26:28.891 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:28.892 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:28.892 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:26:28.892 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:28.892 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:26:28.892 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:28.892 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:26:28.893 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:28.893 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:28.893 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:28.893 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:28.893 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:26:28.893 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:28.894 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:28.894 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:26:28.894 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:28.894 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:26:28.894 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:28.894 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:26:28.894 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:28.895 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:28.895 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:28.895 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:26:28.895 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:28.895 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:26:28.895 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:26:28.895 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:26:28.896 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:28.896 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:26:28Z" 2023-07-29T00:26:28.896 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:26:28.897 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690590264.216448_923.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:24:24.216448Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:26:28Z" } 2023-07-29T00:26:28.898 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690590264.216448_923.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:24:24.216448Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:26:28.899 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690590316.988705_997.1', 2: '', 3: ''} 2023-07-29T00:26:28.899 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690590264.216448_923.1', 2: '', 3: ''} 2023-07-29T00:26:28.899 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690590316.988705_997.1 target=1_1690590264.216448_923.1 2023-07-29T00:26:33.900 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:26:33.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:26:33.901 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:26:33.957 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:26:33.957 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:26:33.988 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:26:33.935+0000 7f6605413880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:26:33.989 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:26:33.936+0000 7f6605413880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:26:34.003 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:26:34.003 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:26:34.003 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:26:34.003 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:26:34.004 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:26:34.004 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:26:34.004 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:34.004 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:34.004 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:26:34.004 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:34.005 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:26:34.005 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:34.005 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:34.005 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:26:34.005 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:34.005 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:26:34.006 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:34.006 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:26:34.006 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:34.006 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:34.006 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:34.006 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:34.006 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:26:34.007 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:34.007 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:34.007 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690590264.216448_923.1", 2023-07-29T00:26:34.007 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:34.007 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:26:34.007 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:34.007 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:24:24.216448Z", 2023-07-29T00:26:34.008 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:34.008 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:34.008 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:34.008 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:34.008 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:26:34.008 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:34.009 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:34.009 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:26:34.009 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:34.009 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:26:34.009 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:34.009 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:26:34.009 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:34.010 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:34.010 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:34.010 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:34.010 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:26:34.010 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:34.010 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:34.010 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:26:34.011 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:34.011 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:26:34.011 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:34.012 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:26:34.012 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:34.012 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:34.012 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:34.012 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:26:34.012 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:34.013 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:26:34.013 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:26:34.013 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:26:34.013 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:34.013 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:26:33Z" 2023-07-29T00:26:34.013 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:26:34.014 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690590264.216448_923.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:24:24.216448Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:26:33Z" } 2023-07-29T00:26:34.016 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690590264.216448_923.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:24:24.216448Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:26:34.017 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690590316.988705_997.1', 2: '', 3: ''} 2023-07-29T00:26:34.017 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690590264.216448_923.1', 2: '', 3: ''} 2023-07-29T00:26:34.018 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690590316.988705_997.1 target=1_1690590264.216448_923.1 2023-07-29T00:26:39.019 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:26:39.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:26:39.021 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:26:39.074 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:26:39.074 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:26:39.106 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:26:39.052+0000 7f79adf22880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:26:39.106 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:26:39.053+0000 7f79adf22880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:26:39.120 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:26:39.121 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:26:39.121 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:26:39.122 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:26:39.123 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:26:39.123 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:26:39.123 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:39.123 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:39.124 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:26:39.124 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:39.124 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:26:39.125 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:39.125 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:39.125 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:26:39.125 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:39.126 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:26:39.126 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:39.126 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:26:39.127 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:39.127 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:39.127 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:39.128 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:39.128 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:26:39.128 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:39.129 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:39.129 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690590264.216448_923.1", 2023-07-29T00:26:39.130 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:39.131 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:26:39.131 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:39.131 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:24:24.216448Z", 2023-07-29T00:26:39.132 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:39.132 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:39.133 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:39.134 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:39.134 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:26:39.134 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:39.135 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:39.135 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:26:39.136 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:39.136 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:26:39.136 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:39.137 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:26:39.137 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:39.137 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:39.138 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:39.139 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:39.139 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:26:39.139 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:39.139 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:39.140 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:26:39.140 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:39.140 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:26:39.140 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:39.140 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:26:39.141 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:39.141 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:39.141 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:39.141 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:26:39.141 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:39.141 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:26:39.142 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:26:39.142 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:26:39.142 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:39.142 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:26:39Z" 2023-07-29T00:26:39.142 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:26:39.143 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690590264.216448_923.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:24:24.216448Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:26:39Z" } 2023-07-29T00:26:39.144 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690590264.216448_923.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:24:24.216448Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:26:39.144 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690590316.988705_997.1', 2: '', 3: ''} 2023-07-29T00:26:39.145 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690590264.216448_923.1', 2: '', 3: ''} 2023-07-29T00:26:39.145 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690590316.988705_997.1 target=1_1690590264.216448_923.1 2023-07-29T00:26:44.148 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:26:44.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:26:44.150 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:26:44.205 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:26:44.205 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:26:44.237 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:26:44.183+0000 7fc3cf68b880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:26:44.238 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:26:44.184+0000 7fc3cf68b880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:26:44.252 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:26:44.253 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:26:44.253 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:26:44.253 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:26:44.253 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:26:44.253 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:26:44.253 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:44.253 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:44.254 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:26:44.254 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:44.254 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:26:44.254 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:44.254 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:44.255 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:26:44.255 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:44.255 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:26:44.255 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:44.255 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:26:44.256 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:44.256 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:44.256 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:44.256 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:44.256 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:26:44.257 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:44.257 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:44.257 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690590264.216448_923.1", 2023-07-29T00:26:44.257 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:44.257 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:26:44.257 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:44.257 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:24:24.216448Z", 2023-07-29T00:26:44.258 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:44.258 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:44.258 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:44.258 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:44.258 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:26:44.259 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:44.259 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:44.259 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:26:44.259 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:44.259 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:26:44.260 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:44.260 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:26:44.260 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:44.260 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:44.261 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:44.261 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:44.261 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:26:44.261 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:44.261 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:44.261 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:26:44.262 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:44.262 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:26:44.262 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:44.262 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:26:44.262 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:44.262 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:44.262 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:44.263 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:26:44.263 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:44.263 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:26:44.263 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:26:44.263 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:26:44.263 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:44.263 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:26:44Z" 2023-07-29T00:26:44.264 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:26:44.265 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690590264.216448_923.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:24:24.216448Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:26:44Z" } 2023-07-29T00:26:44.265 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690590264.216448_923.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:24:24.216448Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:26:44.266 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690590316.988705_997.1', 2: '', 3: ''} 2023-07-29T00:26:44.266 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690590264.216448_923.1', 2: '', 3: ''} 2023-07-29T00:26:44.267 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690590316.988705_997.1 target=1_1690590264.216448_923.1 2023-07-29T00:26:49.268 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:26:49.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:26:49.269 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:26:49.325 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:26:49.325 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:26:49.357 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:26:49.304+0000 7fce38210880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:26:49.358 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:26:49.305+0000 7fce38210880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:26:49.372 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:26:49.373 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:26:49.373 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:26:49.373 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:26:49.374 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:26:49.374 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:26:49.374 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:49.374 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:49.375 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:26:49.375 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:49.376 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:26:49.376 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:49.376 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:49.376 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:26:49.377 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:49.377 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:26:49.377 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:49.377 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:26:49.377 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:49.377 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:49.378 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:49.378 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:49.378 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:26:49.378 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:49.378 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:49.378 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690590293.382770_932.1", 2023-07-29T00:26:49.378 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:49.379 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:26:49.379 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:49.379 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:24:53.382770Z", 2023-07-29T00:26:49.379 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:49.379 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:49.379 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:49.379 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:49.380 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:26:49.380 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:49.380 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:49.380 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:26:49.380 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:49.380 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:26:49.381 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:49.381 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:26:49.381 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:49.381 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:49.381 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:49.381 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:49.381 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:26:49.382 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:49.382 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:49.382 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:26:49.382 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:49.382 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:26:49.382 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:49.382 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:26:49.383 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:49.383 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:49.383 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:49.383 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:26:49.383 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:49.383 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:26:49.383 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:26:49.384 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:26:49.384 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:49.384 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:26:49Z" 2023-07-29T00:26:49.384 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:26:49.385 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690590293.382770_932.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:24:53.382770Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:26:49Z" } 2023-07-29T00:26:49.386 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690590293.382770_932.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:24:53.382770Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:26:49.387 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690590316.988705_997.1', 2: '', 3: ''} 2023-07-29T00:26:49.387 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690590293.382770_932.1', 2: '', 3: ''} 2023-07-29T00:26:49.388 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690590316.988705_997.1 target=1_1690590293.382770_932.1 2023-07-29T00:26:54.389 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:26:54.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:26:54.391 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:26:54.444 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:26:54.444 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:26:54.481 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:26:54.427+0000 7fb09fb4b880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:26:54.482 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:26:54.428+0000 7fb09fb4b880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:26:54.496 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:26:54.496 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:26:54.497 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:26:54.497 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:26:54.497 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:26:54.497 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:26:54.497 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:54.497 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:54.498 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:26:54.498 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:54.498 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:26:54.498 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:54.498 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:54.498 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:26:54.498 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:54.499 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:26:54.499 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:54.499 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:26:54.499 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:54.499 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:54.499 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:54.499 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:54.500 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:26:54.500 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:54.500 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:54.500 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690590316.988705_997.1", 2023-07-29T00:26:54.500 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:54.500 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:26:54.501 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:54.501 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:25:16.988705Z", 2023-07-29T00:26:54.501 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:54.501 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:54.501 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:54.501 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:54.501 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:26:54.502 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:54.502 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:54.502 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:26:54.502 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:54.502 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:26:54.502 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:54.502 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:26:54.503 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:54.503 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:54.503 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:54.503 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:26:54.503 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:26:54.503 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:26:54.503 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:26:54.504 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:26:54.504 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:26:54.504 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:26:54.504 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:26:54.504 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:26:54.504 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:26:54.505 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:54.505 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:26:54.505 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:26:54.505 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:54.505 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:26:54.505 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:26:54.505 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:26:54.506 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:26:54.506 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:26:54Z" 2023-07-29T00:26:54.506 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:26:54.507 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690590316.988705_997.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:25:16.988705Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:26:54Z" } 2023-07-29T00:26:54.507 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690590316.988705_997.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:25:16.988705Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:26:54.508 DEBUG:rgw_multi.tests:log_status={0: '1_1690587969.233742_99.1', 1: '1_1690590316.988705_997.1', 2: '', 3: ''} 2023-07-29T00:26:54.508 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690590316.988705_997.1', 2: '', 3: ''} 2023-07-29T00:26:54.508 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T00:26:54.509 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:26:54.509 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:26:54.510 DEBUG:boto:Method: HEAD 2023-07-29T00:26:54.510 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:26:54.510 DEBUG:boto:Data: 2023-07-29T00:26:54.511 DEBUG:boto:Headers: {} 2023-07-29T00:26:54.511 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:26:54.511 DEBUG:boto:Port: 8000 2023-07-29T00:26:54.512 DEBUG:boto:Params: {} 2023-07-29T00:26:54.512 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-29T00:26:54.513 DEBUG:boto:Token: None 2023-07-29T00:26:54.513 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:26:54 GMT /hiiqos-59/ 2023-07-29T00:26:54.513 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:397tZ73H3iJ2Ieu1TW2MlPQlGlw= 2023-07-29T00:26:54.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:26:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:397tZ73H3iJ2Ieu1TW2MlPQlGlw=', 'Content-Length': '0'} 2023-07-29T00:26:54.524 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000017212c34a93787f-0064c45cce-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:26:54.525 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:26:54.526 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:26:54.526 DEBUG:boto:Method: PUT 2023-07-29T00:26:54.526 DEBUG:boto:Path: /hiiqos-59/obj11 2023-07-29T00:26:54.527 DEBUG:boto:Data: 2023-07-29T00:26:54.527 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:26:54.527 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:26:54.528 DEBUG:boto:Port: 8000 2023-07-29T00:26:54.528 DEBUG:boto:Params: {} 2023-07-29T00:26:54.529 DEBUG:boto:Token: None 2023-07-29T00:26:54.529 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:26:54 GMT /hiiqos-59/obj11 2023-07-29T00:26:54.530 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:z85i7ou7pMYFDKqv04x28A4rh4c= 2023-07-29T00:26:54.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:26:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:z85i7ou7pMYFDKqv04x28A4rh4c='} 2023-07-29T00:26:54.538 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cdd6c329e5f64bc4-0064c45cce-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:26:54.539 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:26:54.539 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:26:54.539 DEBUG:boto:Method: HEAD 2023-07-29T00:26:54.540 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:26:54.540 DEBUG:boto:Data: 2023-07-29T00:26:54.541 DEBUG:boto:Headers: {} 2023-07-29T00:26:54.541 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:26:54.541 DEBUG:boto:Port: 8000 2023-07-29T00:26:54.541 DEBUG:boto:Params: {} 2023-07-29T00:26:54.542 DEBUG:boto:Token: None 2023-07-29T00:26:54.542 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:26:54 GMT /hiiqos-59/ 2023-07-29T00:26:54.543 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:397tZ73H3iJ2Ieu1TW2MlPQlGlw= 2023-07-29T00:26:54.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:26:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:397tZ73H3iJ2Ieu1TW2MlPQlGlw=', 'Content-Length': '0'} 2023-07-29T00:26:54.549 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb1ff0bd8f6ff01e-0064c45cce-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:26:54.550 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:26:54.550 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:26:54.550 DEBUG:boto:Method: PUT 2023-07-29T00:26:54.551 DEBUG:boto:Path: /hiiqos-59/obj22 2023-07-29T00:26:54.551 DEBUG:boto:Data: 2023-07-29T00:26:54.551 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:26:54.552 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:26:54.552 DEBUG:boto:Port: 8000 2023-07-29T00:26:54.553 DEBUG:boto:Params: {} 2023-07-29T00:26:54.553 DEBUG:boto:Token: None 2023-07-29T00:26:54.553 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:26:54 GMT /hiiqos-59/obj22 2023-07-29T00:26:54.554 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KQC/doofNlIxw4KGb+IRTu7w5Vo= 2023-07-29T00:26:54.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:26:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KQC/doofNlIxw4KGb+IRTu7w5Vo='} 2023-07-29T00:26:54.601 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000beeedf60ed5e0613-0064c45cce-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:26:54.602 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:26:54.602 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:26:54.603 DEBUG:boto:Method: HEAD 2023-07-29T00:26:54.603 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:26:54.604 DEBUG:boto:Data: 2023-07-29T00:26:54.604 DEBUG:boto:Headers: {} 2023-07-29T00:26:54.604 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:26:54.605 DEBUG:boto:Port: 8000 2023-07-29T00:26:54.605 DEBUG:boto:Params: {} 2023-07-29T00:26:54.606 DEBUG:boto:Token: None 2023-07-29T00:26:54.606 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:26:54 GMT /hiiqos-59/ 2023-07-29T00:26:54.607 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:397tZ73H3iJ2Ieu1TW2MlPQlGlw= 2023-07-29T00:26:54.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:26:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:397tZ73H3iJ2Ieu1TW2MlPQlGlw=', 'Content-Length': '0'} 2023-07-29T00:26:54.612 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4bc1f0a824725b4-0064c45cce-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:26:54.613 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:26:54.613 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:26:54.614 DEBUG:boto:Method: PUT 2023-07-29T00:26:54.614 DEBUG:boto:Path: /hiiqos-59/obj33 2023-07-29T00:26:54.614 DEBUG:boto:Data: 2023-07-29T00:26:54.615 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:26:54.615 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:26:54.616 DEBUG:boto:Port: 8000 2023-07-29T00:26:54.616 DEBUG:boto:Params: {} 2023-07-29T00:26:54.616 DEBUG:boto:Token: None 2023-07-29T00:26:54.617 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:26:54 GMT /hiiqos-59/obj33 2023-07-29T00:26:54.617 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pBm0fIs2zspHJhXPsId/fRTIA+w= 2023-07-29T00:26:54.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:26:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pBm0fIs2zspHJhXPsId/fRTIA+w='} 2023-07-29T00:26:54.624 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005ee24d12badc686a-0064c45cce-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:26:54.625 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:26:54.625 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:26:54.626 DEBUG:boto:Method: HEAD 2023-07-29T00:26:54.626 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:26:54.626 DEBUG:boto:Data: 2023-07-29T00:26:54.627 DEBUG:boto:Headers: {} 2023-07-29T00:26:54.627 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:26:54.627 DEBUG:boto:Port: 8000 2023-07-29T00:26:54.628 DEBUG:boto:Params: {} 2023-07-29T00:26:54.628 DEBUG:boto:Token: None 2023-07-29T00:26:54.629 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:26:54 GMT /hiiqos-59/ 2023-07-29T00:26:54.629 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:397tZ73H3iJ2Ieu1TW2MlPQlGlw= 2023-07-29T00:26:54.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:26:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:397tZ73H3iJ2Ieu1TW2MlPQlGlw=', 'Content-Length': '0'} 2023-07-29T00:26:54.637 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d23303715d857b68-0064c45cce-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:26:54.637 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:26:54.638 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:26:54.638 DEBUG:boto:Method: PUT 2023-07-29T00:26:54.639 DEBUG:boto:Path: /hiiqos-59/obj44 2023-07-29T00:26:54.639 DEBUG:boto:Data: 2023-07-29T00:26:54.640 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:26:54.640 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:26:54.640 DEBUG:boto:Port: 8000 2023-07-29T00:26:54.641 DEBUG:boto:Params: {} 2023-07-29T00:26:54.641 DEBUG:boto:Token: None 2023-07-29T00:26:54.642 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:26:54 GMT /hiiqos-59/obj44 2023-07-29T00:26:54.643 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UmWb7wC4smeEEcQQEC93NQa/adY= 2023-07-29T00:26:54.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:26:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UmWb7wC4smeEEcQQEC93NQa/adY='} 2023-07-29T00:26:54.691 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006b03015bf8922e9a-0064c45cce-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:26:54.692 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:26:54.692 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:26:54.693 DEBUG:boto:Method: HEAD 2023-07-29T00:26:54.694 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:26:54.694 DEBUG:boto:Data: 2023-07-29T00:26:54.694 DEBUG:boto:Headers: {} 2023-07-29T00:26:54.695 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:26:54.695 DEBUG:boto:Port: 8000 2023-07-29T00:26:54.695 DEBUG:boto:Params: {} 2023-07-29T00:26:54.696 DEBUG:boto:Token: None 2023-07-29T00:26:54.696 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:26:54 GMT /hiiqos-59/ 2023-07-29T00:26:54.697 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:397tZ73H3iJ2Ieu1TW2MlPQlGlw= 2023-07-29T00:26:54.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:26:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:397tZ73H3iJ2Ieu1TW2MlPQlGlw=', 'Content-Length': '0'} 2023-07-29T00:26:54.700 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bf766703eef3a4bc-0064c45cce-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:26:54.700 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:26:54.701 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:26:54.701 DEBUG:boto:Method: PUT 2023-07-29T00:26:54.701 DEBUG:boto:Path: /hiiqos-59/obj55 2023-07-29T00:26:54.702 DEBUG:boto:Data: 2023-07-29T00:26:54.702 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:26:54.703 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:26:54.703 DEBUG:boto:Port: 8000 2023-07-29T00:26:54.704 DEBUG:boto:Params: {} 2023-07-29T00:26:54.705 DEBUG:boto:Token: None 2023-07-29T00:26:54.705 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:26:54 GMT /hiiqos-59/obj55 2023-07-29T00:26:54.705 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/nqzlRmc1voVL5aY2ZCNehHPtM0= 2023-07-29T00:26:54.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:26:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/nqzlRmc1voVL5aY2ZCNehHPtM0='} 2023-07-29T00:26:54.712 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007165d6bee4558cc5-0064c45cce-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:26:54.713 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-59 2023-07-29T00:26:54.713 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:26:54.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:26:54.714 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:26:54.768 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:26:54.769 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:26:54.802 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:26:54.748+0000 7f8ffb024880 1 waiting to reach incremental sync.. 2023-07-29T00:26:59.803 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:26:59.748+0000 7f8ffb024880 1 waiting to reach incremental sync.. 2023-07-29T00:27:04.803 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:27:04.749+0000 7f8ffb024880 1 waiting to reach incremental sync.. 2023-07-29T00:27:09.804 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:27:09.750+0000 7f8ffb024880 1 waiting to reach incremental sync.. 2023-07-29T00:27:14.805 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:27:14.751+0000 7f8ffb024880 1 waiting to reach incremental sync.. 2023-07-29T00:27:19.805 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:27:19.751+0000 7f8ffb024880 1 waiting to reach incremental sync.. 2023-07-29T00:27:24.806 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:27:24.752+0000 7f8ffb024880 1 waiting to reach incremental sync.. 2023-07-29T00:27:29.808 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:27:29.754+0000 7f8ffb024880 1 bucket sync caught up with source: 2023-07-29T00:27:29.809 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , 00000000001.58.6, 00000000002.122.6, 00000000002.70.6] 2023-07-29T00:27:29.809 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000001.58.6, 00000000002.122.6, 00000000002.70.6] 2023-07-29T00:27:29.809 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:27:29.754+0000 7f8ffb024880 0 bucket checkpoint complete 2023-07-29T00:27:29.826 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-59 2023-07-29T00:27:29.827 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:27:29.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:27:29.828 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:27:29.886 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:27:29.887 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:27:29.924 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:27:29.869+0000 7f189d322880 1 waiting for incremental sync to catch up: 2023-07-29T00:27:29.924 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , , 00000000002.775.6, 00000000002.1355.6] 2023-07-29T00:27:29.924 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000001.285.6, 00000000002.775.6, 00000000002.1355.6] 2023-07-29T00:27:34.925 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:27:34.871+0000 7f189d322880 1 waiting for incremental sync to catch up: 2023-07-29T00:27:34.926 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , , 00000000002.775.6, 00000000002.1355.6] 2023-07-29T00:27:34.926 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000001.285.6, 00000000002.775.6, 00000000002.1355.6] 2023-07-29T00:27:39.927 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:27:39.872+0000 7f189d322880 1 waiting for incremental sync to catch up: 2023-07-29T00:27:39.927 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , , 00000000002.775.6, 00000000002.1355.6] 2023-07-29T00:27:39.927 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000001.285.6, 00000000002.775.6, 00000000002.1355.6] 2023-07-29T00:27:44.928 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:27:44.873+0000 7f189d322880 1 bucket sync caught up with source: 2023-07-29T00:27:44.929 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , 00000000001.285.6, 00000000002.775.6, 00000000002.1355.6] 2023-07-29T00:27:44.929 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000001.285.6, 00000000002.775.6, 00000000002.1355.6] 2023-07-29T00:27:44.930 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:27:44.873+0000 7f189d322880 0 bucket checkpoint complete 2023-07-29T00:27:44.946 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-59 zones={a2, a1} 2023-07-29T00:27:44.946 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:27:44.947 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:27:44.947 DEBUG:boto:Method: HEAD 2023-07-29T00:27:44.948 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:27:44.948 DEBUG:boto:Data: 2023-07-29T00:27:44.951 DEBUG:boto:Headers: {} 2023-07-29T00:27:44.952 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:27:44.952 DEBUG:boto:Port: 8001 2023-07-29T00:27:44.953 DEBUG:boto:Params: {} 2023-07-29T00:27:44.953 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-29T00:27:44.954 DEBUG:boto:Token: None 2023-07-29T00:27:44.954 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:27:44 GMT /hiiqos-59/ 2023-07-29T00:27:44.955 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1GUEa5xcffcRyYr5OA8J2lJwXmY= 2023-07-29T00:27:44.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:44 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1GUEa5xcffcRyYr5OA8J2lJwXmY=', 'Content-Length': '0'} 2023-07-29T00:27:49.959 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000759d495cfc038cc4-0064c45d05-1c08-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:27:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:49.960 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:27:49.960 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:27:49.960 DEBUG:boto:Method: HEAD 2023-07-29T00:27:49.961 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:27:49.961 DEBUG:boto:Data: 2023-07-29T00:27:49.962 DEBUG:boto:Headers: {} 2023-07-29T00:27:49.962 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:49.962 DEBUG:boto:Port: 8000 2023-07-29T00:27:49.963 DEBUG:boto:Params: {} 2023-07-29T00:27:49.963 DEBUG:boto:Token: None 2023-07-29T00:27:49.964 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:27:49 GMT /hiiqos-59/ 2023-07-29T00:27:49.964 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TOFekhfGZ8cvGUa6ftg2RhFmrBA= 2023-07-29T00:27:49.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TOFekhfGZ8cvGUa6ftg2RhFmrBA=', 'Content-Length': '0'} 2023-07-29T00:27:49.968 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000040e5824033ffafa9-0064c45d05-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:27:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:49.968 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-29T00:27:49.969 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:27:49.969 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:27:49.969 DEBUG:boto:path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:27:49.970 DEBUG:boto:auth_path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:27:49.970 DEBUG:boto:Method: GET 2023-07-29T00:27:49.970 DEBUG:boto:Path: /hiiqos-59/?versions&max-keys=999 2023-07-29T00:27:49.971 DEBUG:boto:Data: 2023-07-29T00:27:49.971 DEBUG:boto:Headers: {} 2023-07-29T00:27:49.971 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:27:49.972 DEBUG:boto:Port: 8001 2023-07-29T00:27:49.972 DEBUG:boto:Params: {} 2023-07-29T00:27:49.973 DEBUG:boto:Token: None 2023-07-29T00:27:49.973 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:49 GMT /hiiqos-59/?versions 2023-07-29T00:27:49.974 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2BA57JjwpnYQicLOtmB1iKuqgFU= 2023-07-29T00:27:49.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2BA57JjwpnYQicLOtmB1iKuqgFU=', 'Content-Length': '0'} 2023-07-29T00:27:49.977 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008adc2f26cfcbc26-0064c45d05-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:27:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:49.978 DEBUG:boto:b'hiiqos-59999falseobj11nulltrue2023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:27:49.979 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-07-29T00:27:49.979 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-07-29T00:27:49.979 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-07-29T00:27:49.980 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-07-29T00:27:49.980 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-07-29T00:27:49.980 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-29T00:27:49.981 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:27:49.981 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:27:49.982 DEBUG:boto:path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:27:49.982 DEBUG:boto:auth_path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:27:49.982 DEBUG:boto:Method: GET 2023-07-29T00:27:49.983 DEBUG:boto:Path: /hiiqos-59/?versions&max-keys=999 2023-07-29T00:27:49.983 DEBUG:boto:Data: 2023-07-29T00:27:49.983 DEBUG:boto:Headers: {} 2023-07-29T00:27:49.984 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:49.984 DEBUG:boto:Port: 8000 2023-07-29T00:27:49.984 DEBUG:boto:Params: {} 2023-07-29T00:27:49.985 DEBUG:boto:Token: None 2023-07-29T00:27:49.985 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:49 GMT /hiiqos-59/?versions 2023-07-29T00:27:49.986 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2BA57JjwpnYQicLOtmB1iKuqgFU= 2023-07-29T00:27:49.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2BA57JjwpnYQicLOtmB1iKuqgFU=', 'Content-Length': '0'} 2023-07-29T00:27:49.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059b22ceb1a7ee12d-0064c45d05-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:27:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:49.989 DEBUG:boto:b'hiiqos-59999falseobj11nulltrue2023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:27:49.990 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-07-29T00:27:49.990 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-07-29T00:27:49.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-07-29T00:27:49.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-07-29T00:27:49.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-07-29T00:27:49.993 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:27:49.993 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:27:49.993 DEBUG:boto:path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:27:49.994 DEBUG:boto:auth_path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:27:49.994 DEBUG:boto:Method: GET 2023-07-29T00:27:49.995 DEBUG:boto:Path: /hiiqos-59/?versions&max-keys=999 2023-07-29T00:27:49.995 DEBUG:boto:Data: 2023-07-29T00:27:49.995 DEBUG:boto:Headers: {} 2023-07-29T00:27:49.996 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:27:49.996 DEBUG:boto:Port: 8001 2023-07-29T00:27:49.996 DEBUG:boto:Params: {} 2023-07-29T00:27:49.997 DEBUG:boto:Token: None 2023-07-29T00:27:49.997 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:49 GMT /hiiqos-59/?versions 2023-07-29T00:27:49.998 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2BA57JjwpnYQicLOtmB1iKuqgFU= 2023-07-29T00:27:49.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2BA57JjwpnYQicLOtmB1iKuqgFU=', 'Content-Length': '0'} 2023-07-29T00:27:50.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052c2018db2e136e2-0064c45d05-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:27:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.002 DEBUG:boto:b'hiiqos-59999falseobj11nulltrue2023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:27:50.003 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:27:50.003 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:27:50.003 DEBUG:boto:path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:27:50.004 DEBUG:boto:auth_path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:27:50.004 DEBUG:boto:Method: GET 2023-07-29T00:27:50.005 DEBUG:boto:Path: /hiiqos-59/?versions&max-keys=999 2023-07-29T00:27:50.005 DEBUG:boto:Data: 2023-07-29T00:27:50.006 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.007 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:50.008 DEBUG:boto:Port: 8000 2023-07-29T00:27:50.008 DEBUG:boto:Params: {} 2023-07-29T00:27:50.008 DEBUG:boto:Token: None 2023-07-29T00:27:50.009 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/?versions 2023-07-29T00:27:50.009 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2FiMmUGaBxEFnXMx2ktATevOloA= 2023-07-29T00:27:50.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2FiMmUGaBxEFnXMx2ktATevOloA=', 'Content-Length': '0'} 2023-07-29T00:27:50.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f35cb508c973bb5d-0064c45d05-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:27:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.013 DEBUG:boto:b'hiiqos-59999falseobj11nulltrue2023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:27:50.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-07-29T00:27:50.014 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:27:50.015 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:27:50.015 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:27:50.015 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:27:50.016 DEBUG:boto:Method: GET 2023-07-29T00:27:50.016 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:27:50.017 DEBUG:boto:Data: 2023-07-29T00:27:50.017 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.017 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:27:50.018 DEBUG:boto:Port: 8001 2023-07-29T00:27:50.018 DEBUG:boto:Params: {} 2023-07-29T00:27:50.018 DEBUG:boto:Token: None 2023-07-29T00:27:50.019 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:27:50.019 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HJI3h2cvzwMVK7xvv8g3kq9KaDA= 2023-07-29T00:27:50.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HJI3h2cvzwMVK7xvv8g3kq9KaDA=', 'Content-Length': '0'} 2023-07-29T00:27:50.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c99576e1c12eb8e-0064c45d05-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.024 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:27:50.025 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:27:50.026 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:27:50.026 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:27:50.026 DEBUG:boto:Method: GET 2023-07-29T00:27:50.027 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:27:50.027 DEBUG:boto:Data: 2023-07-29T00:27:50.027 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.028 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:50.028 DEBUG:boto:Port: 8000 2023-07-29T00:27:50.029 DEBUG:boto:Params: {} 2023-07-29T00:27:50.029 DEBUG:boto:Token: None 2023-07-29T00:27:50.029 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:27:50.030 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HJI3h2cvzwMVK7xvv8g3kq9KaDA= 2023-07-29T00:27:50.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HJI3h2cvzwMVK7xvv8g3kq9KaDA=', 'Content-Length': '0'} 2023-07-29T00:27:50.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000db4e9de89f8c586-0064c45d05-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.076 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:27:50.077 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:27:50.077 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:27:50.078 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:27:50.078 DEBUG:boto:Method: HEAD 2023-07-29T00:27:50.079 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:27:50.079 DEBUG:boto:Data: 2023-07-29T00:27:50.079 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.080 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:27:50.080 DEBUG:boto:Port: 8001 2023-07-29T00:27:50.080 DEBUG:boto:Params: {} 2023-07-29T00:27:50.081 DEBUG:boto:Token: None 2023-07-29T00:27:50.081 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:27:50.082 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lfsdxXQlOqCsY4bXzXMuX0zTwzo= 2023-07-29T00:27:50.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lfsdxXQlOqCsY4bXzXMuX0zTwzo=', 'Content-Length': '0'} 2023-07-29T00:27:50.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b547a0fd8e1908e9-0064c45d06-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.085 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:27:50.085 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:27:50.086 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:27:50.086 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:27:50.086 DEBUG:boto:Method: HEAD 2023-07-29T00:27:50.087 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:27:50.087 DEBUG:boto:Data: 2023-07-29T00:27:50.087 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.088 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:50.088 DEBUG:boto:Port: 8000 2023-07-29T00:27:50.088 DEBUG:boto:Params: {} 2023-07-29T00:27:50.089 DEBUG:boto:Token: None 2023-07-29T00:27:50.089 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:27:50.089 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lfsdxXQlOqCsY4bXzXMuX0zTwzo= 2023-07-29T00:27:50.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lfsdxXQlOqCsY4bXzXMuX0zTwzo=', 'Content-Length': '0'} 2023-07-29T00:27:50.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092e6d0e2139e698d-0064c45d06-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-07-29T00:27:50.093 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:27:50.093 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:27:50.094 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:27:50.094 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:27:50.095 DEBUG:boto:Method: GET 2023-07-29T00:27:50.095 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:27:50.096 DEBUG:boto:Data: 2023-07-29T00:27:50.097 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.098 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:27:50.098 DEBUG:boto:Port: 8001 2023-07-29T00:27:50.099 DEBUG:boto:Params: {} 2023-07-29T00:27:50.099 DEBUG:boto:Token: None 2023-07-29T00:27:50.100 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:27:50.100 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HJI3h2cvzwMVK7xvv8g3kq9KaDA= 2023-07-29T00:27:50.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HJI3h2cvzwMVK7xvv8g3kq9KaDA=', 'Content-Length': '0'} 2023-07-29T00:27:50.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c4212161a5b2d0d-0064c45d06-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.104 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:27:50.104 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:27:50.105 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:27:50.105 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:27:50.106 DEBUG:boto:Method: GET 2023-07-29T00:27:50.106 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:27:50.107 DEBUG:boto:Data: 2023-07-29T00:27:50.107 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.107 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:50.108 DEBUG:boto:Port: 8000 2023-07-29T00:27:50.109 DEBUG:boto:Params: {} 2023-07-29T00:27:50.109 DEBUG:boto:Token: None 2023-07-29T00:27:50.110 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:27:50.110 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HJI3h2cvzwMVK7xvv8g3kq9KaDA= 2023-07-29T00:27:50.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HJI3h2cvzwMVK7xvv8g3kq9KaDA=', 'Content-Length': '0'} 2023-07-29T00:27:50.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013f40df9a7b30cbc-0064c45d06-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.114 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:27:50.114 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:27:50.114 DEBUG:boto:path=/hiiqos-59/?prefix=obj11 2023-07-29T00:27:50.115 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj11 2023-07-29T00:27:50.115 DEBUG:boto:Method: GET 2023-07-29T00:27:50.116 DEBUG:boto:Path: /hiiqos-59/?prefix=obj11 2023-07-29T00:27:50.117 DEBUG:boto:Data: 2023-07-29T00:27:50.117 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.118 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:27:50.118 DEBUG:boto:Port: 8001 2023-07-29T00:27:50.118 DEBUG:boto:Params: {} 2023-07-29T00:27:50.119 DEBUG:boto:Token: None 2023-07-29T00:27:50.119 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/ 2023-07-29T00:27:50.120 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qqCTonHJd3qVSeStZ2f5/FIiO0k= 2023-07-29T00:27:50.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qqCTonHJd3qVSeStZ2f5/FIiO0k=', 'Content-Length': '0'} 2023-07-29T00:27:50.124 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a4d143faba5078a-0064c45d06-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.125 DEBUG:boto:b'hiiqos-59obj111000falseobj112023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:27:50.126 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:27:50.127 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:27:50.128 DEBUG:boto:path=/hiiqos-59/?prefix=obj11 2023-07-29T00:27:50.129 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj11 2023-07-29T00:27:50.130 DEBUG:boto:Method: GET 2023-07-29T00:27:50.131 DEBUG:boto:Path: /hiiqos-59/?prefix=obj11 2023-07-29T00:27:50.132 DEBUG:boto:Data: 2023-07-29T00:27:50.133 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.134 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:50.134 DEBUG:boto:Port: 8000 2023-07-29T00:27:50.135 DEBUG:boto:Params: {} 2023-07-29T00:27:50.136 DEBUG:boto:Token: None 2023-07-29T00:27:50.137 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/ 2023-07-29T00:27:50.137 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qqCTonHJd3qVSeStZ2f5/FIiO0k= 2023-07-29T00:27:50.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qqCTonHJd3qVSeStZ2f5/FIiO0k=', 'Content-Length': '0'} 2023-07-29T00:27:50.141 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018540623876e4b5d-0064c45d06-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.141 DEBUG:boto:b'hiiqos-59obj111000falseobj112023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:27:50.142 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:27:50.143 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:27:50.143 DEBUG:boto:Method: HEAD 2023-07-29T00:27:50.143 DEBUG:boto:Path: /hiiqos-59/obj11 2023-07-29T00:27:50.144 DEBUG:boto:Data: 2023-07-29T00:27:50.144 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.145 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:27:50.145 DEBUG:boto:Port: 8001 2023-07-29T00:27:50.145 DEBUG:boto:Params: {} 2023-07-29T00:27:50.146 DEBUG:boto:Token: None 2023-07-29T00:27:50.146 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj11 2023-07-29T00:27:50.147 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gh6vP7ne3EbNhdCt+Q15mtlmEhI= 2023-07-29T00:27:50.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gh6vP7ne3EbNhdCt+Q15mtlmEhI=', 'Content-Length': '0'} 2023-07-29T00:27:50.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf1f96d76954b206-0064c45d06-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.150 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:27:50.150 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:27:50.151 DEBUG:boto:Method: HEAD 2023-07-29T00:27:50.151 DEBUG:boto:Path: /hiiqos-59/obj11 2023-07-29T00:27:50.152 DEBUG:boto:Data: 2023-07-29T00:27:50.152 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.153 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:50.153 DEBUG:boto:Port: 8000 2023-07-29T00:27:50.153 DEBUG:boto:Params: {} 2023-07-29T00:27:50.154 DEBUG:boto:Token: None 2023-07-29T00:27:50.154 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj11 2023-07-29T00:27:50.154 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gh6vP7ne3EbNhdCt+Q15mtlmEhI= 2023-07-29T00:27:50.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gh6vP7ne3EbNhdCt+Q15mtlmEhI=', 'Content-Length': '0'} 2023-07-29T00:27:50.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eea79d2fc1e68529-0064c45d06-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-07-29T00:27:50.159 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:27:50.160 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:27:50.161 DEBUG:boto:Method: GET 2023-07-29T00:27:50.162 DEBUG:boto:Path: /hiiqos-59/obj11 2023-07-29T00:27:50.163 DEBUG:boto:Data: 2023-07-29T00:27:50.164 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.165 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:27:50.165 DEBUG:boto:Port: 8001 2023-07-29T00:27:50.166 DEBUG:boto:Params: {} 2023-07-29T00:27:50.167 DEBUG:boto:Token: None 2023-07-29T00:27:50.168 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj11 2023-07-29T00:27:50.169 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OUJ6U86aq9t0lkUCaf2VlHDM1as= 2023-07-29T00:27:50.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OUJ6U86aq9t0lkUCaf2VlHDM1as=', 'Content-Length': '0'} 2023-07-29T00:27:50.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b2821fca30010b4-0064c45d06-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.212 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:27:50.213 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:27:50.213 DEBUG:boto:Method: GET 2023-07-29T00:27:50.214 DEBUG:boto:Path: /hiiqos-59/obj11 2023-07-29T00:27:50.214 DEBUG:boto:Data: 2023-07-29T00:27:50.215 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.215 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:50.215 DEBUG:boto:Port: 8000 2023-07-29T00:27:50.216 DEBUG:boto:Params: {} 2023-07-29T00:27:50.216 DEBUG:boto:Token: None 2023-07-29T00:27:50.217 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj11 2023-07-29T00:27:50.217 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OUJ6U86aq9t0lkUCaf2VlHDM1as= 2023-07-29T00:27:50.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OUJ6U86aq9t0lkUCaf2VlHDM1as=', 'Content-Length': '0'} 2023-07-29T00:27:50.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1562c65dba3ca93-0064c45d06-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.221 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:27:50.222 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:27:50.222 DEBUG:boto:path=/hiiqos-59/?prefix=obj11 2023-07-29T00:27:50.222 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj11 2023-07-29T00:27:50.223 DEBUG:boto:Method: GET 2023-07-29T00:27:50.223 DEBUG:boto:Path: /hiiqos-59/?prefix=obj11 2023-07-29T00:27:50.224 DEBUG:boto:Data: 2023-07-29T00:27:50.224 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.224 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:27:50.225 DEBUG:boto:Port: 8001 2023-07-29T00:27:50.225 DEBUG:boto:Params: {} 2023-07-29T00:27:50.226 DEBUG:boto:Token: None 2023-07-29T00:27:50.227 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/ 2023-07-29T00:27:50.228 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qqCTonHJd3qVSeStZ2f5/FIiO0k= 2023-07-29T00:27:50.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qqCTonHJd3qVSeStZ2f5/FIiO0k=', 'Content-Length': '0'} 2023-07-29T00:27:50.232 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0332b54ddfa6b9e-0064c45d06-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.233 DEBUG:boto:b'hiiqos-59obj111000falseobj112023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:27:50.234 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:27:50.235 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:27:50.236 DEBUG:boto:path=/hiiqos-59/?prefix=obj11 2023-07-29T00:27:50.236 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj11 2023-07-29T00:27:50.237 DEBUG:boto:Method: GET 2023-07-29T00:27:50.238 DEBUG:boto:Path: /hiiqos-59/?prefix=obj11 2023-07-29T00:27:50.238 DEBUG:boto:Data: 2023-07-29T00:27:50.239 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.239 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:50.239 DEBUG:boto:Port: 8000 2023-07-29T00:27:50.240 DEBUG:boto:Params: {} 2023-07-29T00:27:50.240 DEBUG:boto:Token: None 2023-07-29T00:27:50.241 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/ 2023-07-29T00:27:50.241 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qqCTonHJd3qVSeStZ2f5/FIiO0k= 2023-07-29T00:27:50.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qqCTonHJd3qVSeStZ2f5/FIiO0k=', 'Content-Length': '0'} 2023-07-29T00:27:50.244 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c306f136ed774b3-0064c45d06-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.245 DEBUG:boto:b'hiiqos-59obj111000falseobj112023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:27:50.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-07-29T00:27:50.247 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:27:50.247 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:27:50.248 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:27:50.248 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:27:50.248 DEBUG:boto:Method: GET 2023-07-29T00:27:50.249 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:27:50.249 DEBUG:boto:Data: 2023-07-29T00:27:50.250 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.250 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:27:50.250 DEBUG:boto:Port: 8001 2023-07-29T00:27:50.251 DEBUG:boto:Params: {} 2023-07-29T00:27:50.251 DEBUG:boto:Token: None 2023-07-29T00:27:50.252 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:27:50.252 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:o47abrhkboZEDUDjJJC/a2+ZNqc= 2023-07-29T00:27:50.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:o47abrhkboZEDUDjJJC/a2+ZNqc=', 'Content-Length': '0'} 2023-07-29T00:27:50.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a158b18706a17c3-0064c45d06-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.255 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:27:50.256 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:27:50.256 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:27:50.257 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:27:50.257 DEBUG:boto:Method: GET 2023-07-29T00:27:50.258 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:27:50.258 DEBUG:boto:Data: 2023-07-29T00:27:50.258 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.259 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:50.260 DEBUG:boto:Port: 8000 2023-07-29T00:27:50.260 DEBUG:boto:Params: {} 2023-07-29T00:27:50.261 DEBUG:boto:Token: None 2023-07-29T00:27:50.261 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:27:50.262 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:o47abrhkboZEDUDjJJC/a2+ZNqc= 2023-07-29T00:27:50.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:o47abrhkboZEDUDjJJC/a2+ZNqc=', 'Content-Length': '0'} 2023-07-29T00:27:50.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000349eaf24697f0752-0064c45d06-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.265 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:27:50.265 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:27:50.266 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:27:50.266 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:27:50.266 DEBUG:boto:Method: HEAD 2023-07-29T00:27:50.267 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:27:50.267 DEBUG:boto:Data: 2023-07-29T00:27:50.267 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.268 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:27:50.268 DEBUG:boto:Port: 8001 2023-07-29T00:27:50.268 DEBUG:boto:Params: {} 2023-07-29T00:27:50.269 DEBUG:boto:Token: None 2023-07-29T00:27:50.269 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:27:50.270 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DulwuBSJZ+2fvdPncDkrMFr9NHk= 2023-07-29T00:27:50.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DulwuBSJZ+2fvdPncDkrMFr9NHk=', 'Content-Length': '0'} 2023-07-29T00:27:50.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a51dfe6213f1fc8-0064c45d06-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.273 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:27:50.273 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:27:50.273 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:27:50.274 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:27:50.274 DEBUG:boto:Method: HEAD 2023-07-29T00:27:50.274 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:27:50.275 DEBUG:boto:Data: 2023-07-29T00:27:50.275 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.275 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:50.276 DEBUG:boto:Port: 8000 2023-07-29T00:27:50.276 DEBUG:boto:Params: {} 2023-07-29T00:27:50.277 DEBUG:boto:Token: None 2023-07-29T00:27:50.277 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:27:50.277 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DulwuBSJZ+2fvdPncDkrMFr9NHk= 2023-07-29T00:27:50.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DulwuBSJZ+2fvdPncDkrMFr9NHk=', 'Content-Length': '0'} 2023-07-29T00:27:50.281 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2c9a8f21fc214a7-0064c45d06-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-07-29T00:27:50.283 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:27:50.284 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:27:50.284 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:27:50.285 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:27:50.286 DEBUG:boto:Method: GET 2023-07-29T00:27:50.287 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:27:50.287 DEBUG:boto:Data: 2023-07-29T00:27:50.288 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.288 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:27:50.289 DEBUG:boto:Port: 8001 2023-07-29T00:27:50.290 DEBUG:boto:Params: {} 2023-07-29T00:27:50.291 DEBUG:boto:Token: None 2023-07-29T00:27:50.291 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:27:50.292 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:o47abrhkboZEDUDjJJC/a2+ZNqc= 2023-07-29T00:27:50.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:o47abrhkboZEDUDjJJC/a2+ZNqc=', 'Content-Length': '0'} 2023-07-29T00:27:50.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003dd28d119d0dbfea-0064c45d06-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.295 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:27:50.296 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:27:50.297 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:27:50.298 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:27:50.298 DEBUG:boto:Method: GET 2023-07-29T00:27:50.299 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:27:50.300 DEBUG:boto:Data: 2023-07-29T00:27:50.301 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.301 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:50.302 DEBUG:boto:Port: 8000 2023-07-29T00:27:50.303 DEBUG:boto:Params: {} 2023-07-29T00:27:50.303 DEBUG:boto:Token: None 2023-07-29T00:27:50.304 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:27:50.304 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:o47abrhkboZEDUDjJJC/a2+ZNqc= 2023-07-29T00:27:50.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:o47abrhkboZEDUDjJJC/a2+ZNqc=', 'Content-Length': '0'} 2023-07-29T00:27:50.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019c5a0b4945d2dc2-0064c45d06-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.352 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:27:50.353 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:27:50.353 DEBUG:boto:path=/hiiqos-59/?prefix=obj22 2023-07-29T00:27:50.354 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj22 2023-07-29T00:27:50.354 DEBUG:boto:Method: GET 2023-07-29T00:27:50.355 DEBUG:boto:Path: /hiiqos-59/?prefix=obj22 2023-07-29T00:27:50.355 DEBUG:boto:Data: 2023-07-29T00:27:50.356 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.356 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:27:50.357 DEBUG:boto:Port: 8001 2023-07-29T00:27:50.358 DEBUG:boto:Params: {} 2023-07-29T00:27:50.360 DEBUG:boto:Token: None 2023-07-29T00:27:50.360 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/ 2023-07-29T00:27:50.361 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qqCTonHJd3qVSeStZ2f5/FIiO0k= 2023-07-29T00:27:50.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qqCTonHJd3qVSeStZ2f5/FIiO0k=', 'Content-Length': '0'} 2023-07-29T00:27:50.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8545e8b54c8359f-0064c45d06-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.365 DEBUG:boto:b'hiiqos-59obj221000falseobj222023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:27:50.366 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:27:50.367 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:27:50.367 DEBUG:boto:path=/hiiqos-59/?prefix=obj22 2023-07-29T00:27:50.369 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj22 2023-07-29T00:27:50.370 DEBUG:boto:Method: GET 2023-07-29T00:27:50.370 DEBUG:boto:Path: /hiiqos-59/?prefix=obj22 2023-07-29T00:27:50.371 DEBUG:boto:Data: 2023-07-29T00:27:50.372 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.373 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:50.373 DEBUG:boto:Port: 8000 2023-07-29T00:27:50.374 DEBUG:boto:Params: {} 2023-07-29T00:27:50.375 DEBUG:boto:Token: None 2023-07-29T00:27:50.376 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/ 2023-07-29T00:27:50.377 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qqCTonHJd3qVSeStZ2f5/FIiO0k= 2023-07-29T00:27:50.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qqCTonHJd3qVSeStZ2f5/FIiO0k=', 'Content-Length': '0'} 2023-07-29T00:27:50.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c1e1a76083f6d31-0064c45d06-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.380 DEBUG:boto:b'hiiqos-59obj221000falseobj222023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:27:50.381 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:27:50.382 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:27:50.382 DEBUG:boto:Method: HEAD 2023-07-29T00:27:50.382 DEBUG:boto:Path: /hiiqos-59/obj22 2023-07-29T00:27:50.383 DEBUG:boto:Data: 2023-07-29T00:27:50.383 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.384 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:27:50.384 DEBUG:boto:Port: 8001 2023-07-29T00:27:50.385 DEBUG:boto:Params: {} 2023-07-29T00:27:50.385 DEBUG:boto:Token: None 2023-07-29T00:27:50.386 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj22 2023-07-29T00:27:50.387 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xpwl7B/ouUrH5ezmgJU9YsN7Yos= 2023-07-29T00:27:50.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xpwl7B/ouUrH5ezmgJU9YsN7Yos=', 'Content-Length': '0'} 2023-07-29T00:27:50.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f6a3047ab679128-0064c45d06-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.391 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:27:50.391 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:27:50.391 DEBUG:boto:Method: HEAD 2023-07-29T00:27:50.392 DEBUG:boto:Path: /hiiqos-59/obj22 2023-07-29T00:27:50.392 DEBUG:boto:Data: 2023-07-29T00:27:50.393 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.393 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:50.393 DEBUG:boto:Port: 8000 2023-07-29T00:27:50.394 DEBUG:boto:Params: {} 2023-07-29T00:27:50.395 DEBUG:boto:Token: None 2023-07-29T00:27:50.395 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj22 2023-07-29T00:27:50.395 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xpwl7B/ouUrH5ezmgJU9YsN7Yos= 2023-07-29T00:27:50.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xpwl7B/ouUrH5ezmgJU9YsN7Yos=', 'Content-Length': '0'} 2023-07-29T00:27:50.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024bcd0b13d5f8f24-0064c45d06-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-07-29T00:27:50.399 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:27:50.400 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:27:50.400 DEBUG:boto:Method: GET 2023-07-29T00:27:50.401 DEBUG:boto:Path: /hiiqos-59/obj22 2023-07-29T00:27:50.401 DEBUG:boto:Data: 2023-07-29T00:27:50.401 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.402 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:27:50.402 DEBUG:boto:Port: 8001 2023-07-29T00:27:50.402 DEBUG:boto:Params: {} 2023-07-29T00:27:50.403 DEBUG:boto:Token: None 2023-07-29T00:27:50.403 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj22 2023-07-29T00:27:50.404 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Q7bomV4k0wAVG5XAfxx76hpYDw4= 2023-07-29T00:27:50.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q7bomV4k0wAVG5XAfxx76hpYDw4=', 'Content-Length': '0'} 2023-07-29T00:27:50.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c66aeb329146301c-0064c45d06-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.407 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:27:50.407 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:27:50.408 DEBUG:boto:Method: GET 2023-07-29T00:27:50.408 DEBUG:boto:Path: /hiiqos-59/obj22 2023-07-29T00:27:50.408 DEBUG:boto:Data: 2023-07-29T00:27:50.409 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.409 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:50.410 DEBUG:boto:Port: 8000 2023-07-29T00:27:50.410 DEBUG:boto:Params: {} 2023-07-29T00:27:50.410 DEBUG:boto:Token: None 2023-07-29T00:27:50.411 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj22 2023-07-29T00:27:50.411 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Q7bomV4k0wAVG5XAfxx76hpYDw4= 2023-07-29T00:27:50.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q7bomV4k0wAVG5XAfxx76hpYDw4=', 'Content-Length': '0'} 2023-07-29T00:27:50.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aff611b9c8360f72-0064c45d06-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.414 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:27:50.415 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:27:50.415 DEBUG:boto:path=/hiiqos-59/?prefix=obj22 2023-07-29T00:27:50.415 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj22 2023-07-29T00:27:50.416 DEBUG:boto:Method: GET 2023-07-29T00:27:50.416 DEBUG:boto:Path: /hiiqos-59/?prefix=obj22 2023-07-29T00:27:50.416 DEBUG:boto:Data: 2023-07-29T00:27:50.417 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.417 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:27:50.417 DEBUG:boto:Port: 8001 2023-07-29T00:27:50.418 DEBUG:boto:Params: {} 2023-07-29T00:27:50.418 DEBUG:boto:Token: None 2023-07-29T00:27:50.419 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/ 2023-07-29T00:27:50.419 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qqCTonHJd3qVSeStZ2f5/FIiO0k= 2023-07-29T00:27:50.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qqCTonHJd3qVSeStZ2f5/FIiO0k=', 'Content-Length': '0'} 2023-07-29T00:27:50.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063c9e522289e506d-0064c45d06-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.422 DEBUG:boto:b'hiiqos-59obj221000falseobj222023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:27:50.423 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:27:50.424 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:27:50.424 DEBUG:boto:path=/hiiqos-59/?prefix=obj22 2023-07-29T00:27:50.424 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj22 2023-07-29T00:27:50.425 DEBUG:boto:Method: GET 2023-07-29T00:27:50.425 DEBUG:boto:Path: /hiiqos-59/?prefix=obj22 2023-07-29T00:27:50.425 DEBUG:boto:Data: 2023-07-29T00:27:50.426 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.426 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:50.426 DEBUG:boto:Port: 8000 2023-07-29T00:27:50.427 DEBUG:boto:Params: {} 2023-07-29T00:27:50.427 DEBUG:boto:Token: None 2023-07-29T00:27:50.428 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/ 2023-07-29T00:27:50.428 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qqCTonHJd3qVSeStZ2f5/FIiO0k= 2023-07-29T00:27:50.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qqCTonHJd3qVSeStZ2f5/FIiO0k=', 'Content-Length': '0'} 2023-07-29T00:27:50.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000106b22f511dbcc4f-0064c45d06-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.432 DEBUG:boto:b'hiiqos-59obj221000falseobj222023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:27:50.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-07-29T00:27:50.433 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:27:50.433 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:27:50.433 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:27:50.434 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:27:50.434 DEBUG:boto:Method: GET 2023-07-29T00:27:50.434 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:27:50.435 DEBUG:boto:Data: 2023-07-29T00:27:50.435 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.436 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:27:50.436 DEBUG:boto:Port: 8001 2023-07-29T00:27:50.436 DEBUG:boto:Params: {} 2023-07-29T00:27:50.437 DEBUG:boto:Token: None 2023-07-29T00:27:50.437 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:27:50.438 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hjPj4z4AQFNptAbfkCWTDV16EvA= 2023-07-29T00:27:50.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hjPj4z4AQFNptAbfkCWTDV16EvA=', 'Content-Length': '0'} 2023-07-29T00:27:50.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc9d1f1fec5c3480-0064c45d06-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.441 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:27:50.441 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:27:50.441 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:27:50.442 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:27:50.442 DEBUG:boto:Method: GET 2023-07-29T00:27:50.442 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:27:50.443 DEBUG:boto:Data: 2023-07-29T00:27:50.443 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.444 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:50.444 DEBUG:boto:Port: 8000 2023-07-29T00:27:50.444 DEBUG:boto:Params: {} 2023-07-29T00:27:50.445 DEBUG:boto:Token: None 2023-07-29T00:27:50.445 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:27:50.446 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hjPj4z4AQFNptAbfkCWTDV16EvA= 2023-07-29T00:27:50.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hjPj4z4AQFNptAbfkCWTDV16EvA=', 'Content-Length': '0'} 2023-07-29T00:27:50.448 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7b62bceb3171500-0064c45d06-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.449 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:27:50.450 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:27:50.450 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:27:50.451 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:27:50.451 DEBUG:boto:Method: HEAD 2023-07-29T00:27:50.451 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:27:50.452 DEBUG:boto:Data: 2023-07-29T00:27:50.452 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.453 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:27:50.453 DEBUG:boto:Port: 8001 2023-07-29T00:27:50.453 DEBUG:boto:Params: {} 2023-07-29T00:27:50.454 DEBUG:boto:Token: None 2023-07-29T00:27:50.454 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:27:50.454 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XvITCwOjOAbktyLyh4yLbGfs2KI= 2023-07-29T00:27:50.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XvITCwOjOAbktyLyh4yLbGfs2KI=', 'Content-Length': '0'} 2023-07-29T00:27:50.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed39be509f17fa3c-0064c45d06-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.457 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:27:50.458 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:27:50.458 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:27:50.459 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:27:50.459 DEBUG:boto:Method: HEAD 2023-07-29T00:27:50.460 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:27:50.461 DEBUG:boto:Data: 2023-07-29T00:27:50.461 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.462 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:50.463 DEBUG:boto:Port: 8000 2023-07-29T00:27:50.463 DEBUG:boto:Params: {} 2023-07-29T00:27:50.464 DEBUG:boto:Token: None 2023-07-29T00:27:50.465 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:27:50.466 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XvITCwOjOAbktyLyh4yLbGfs2KI= 2023-07-29T00:27:50.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XvITCwOjOAbktyLyh4yLbGfs2KI=', 'Content-Length': '0'} 2023-07-29T00:27:50.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000964e327a7671aca0-0064c45d06-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-07-29T00:27:50.471 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:27:50.472 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:27:50.473 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:27:50.474 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:27:50.475 DEBUG:boto:Method: GET 2023-07-29T00:27:50.475 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:27:50.476 DEBUG:boto:Data: 2023-07-29T00:27:50.476 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.477 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:27:50.477 DEBUG:boto:Port: 8001 2023-07-29T00:27:50.477 DEBUG:boto:Params: {} 2023-07-29T00:27:50.478 DEBUG:boto:Token: None 2023-07-29T00:27:50.478 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:27:50.479 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hjPj4z4AQFNptAbfkCWTDV16EvA= 2023-07-29T00:27:50.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hjPj4z4AQFNptAbfkCWTDV16EvA=', 'Content-Length': '0'} 2023-07-29T00:27:50.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9f3a6fe6ee65396-0064c45d06-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.528 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:27:50.529 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:27:50.529 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:27:50.530 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:27:50.530 DEBUG:boto:Method: GET 2023-07-29T00:27:50.530 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:27:50.531 DEBUG:boto:Data: 2023-07-29T00:27:50.531 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.531 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:50.532 DEBUG:boto:Port: 8000 2023-07-29T00:27:50.532 DEBUG:boto:Params: {} 2023-07-29T00:27:50.533 DEBUG:boto:Token: None 2023-07-29T00:27:50.533 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:27:50.534 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hjPj4z4AQFNptAbfkCWTDV16EvA= 2023-07-29T00:27:50.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hjPj4z4AQFNptAbfkCWTDV16EvA=', 'Content-Length': '0'} 2023-07-29T00:27:50.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e613eec8b4f75d97-0064c45d06-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.537 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:27:50.537 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:27:50.538 DEBUG:boto:path=/hiiqos-59/?prefix=obj33 2023-07-29T00:27:50.538 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj33 2023-07-29T00:27:50.538 DEBUG:boto:Method: GET 2023-07-29T00:27:50.539 DEBUG:boto:Path: /hiiqos-59/?prefix=obj33 2023-07-29T00:27:50.539 DEBUG:boto:Data: 2023-07-29T00:27:50.540 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.540 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:27:50.540 DEBUG:boto:Port: 8001 2023-07-29T00:27:50.541 DEBUG:boto:Params: {} 2023-07-29T00:27:50.541 DEBUG:boto:Token: None 2023-07-29T00:27:50.541 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/ 2023-07-29T00:27:50.542 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qqCTonHJd3qVSeStZ2f5/FIiO0k= 2023-07-29T00:27:50.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qqCTonHJd3qVSeStZ2f5/FIiO0k=', 'Content-Length': '0'} 2023-07-29T00:27:50.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e35fc0bbb951640f-0064c45d06-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.545 DEBUG:boto:b'hiiqos-59obj331000falseobj332023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:27:50.546 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:27:50.546 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:27:50.547 DEBUG:boto:path=/hiiqos-59/?prefix=obj33 2023-07-29T00:27:50.547 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj33 2023-07-29T00:27:50.547 DEBUG:boto:Method: GET 2023-07-29T00:27:50.548 DEBUG:boto:Path: /hiiqos-59/?prefix=obj33 2023-07-29T00:27:50.548 DEBUG:boto:Data: 2023-07-29T00:27:50.549 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.549 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:50.549 DEBUG:boto:Port: 8000 2023-07-29T00:27:50.550 DEBUG:boto:Params: {} 2023-07-29T00:27:50.550 DEBUG:boto:Token: None 2023-07-29T00:27:50.550 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/ 2023-07-29T00:27:50.551 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qqCTonHJd3qVSeStZ2f5/FIiO0k= 2023-07-29T00:27:50.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qqCTonHJd3qVSeStZ2f5/FIiO0k=', 'Content-Length': '0'} 2023-07-29T00:27:50.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000553042c6d2208080-0064c45d06-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.596 DEBUG:boto:b'hiiqos-59obj331000falseobj332023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:27:50.597 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:27:50.597 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:27:50.598 DEBUG:boto:Method: HEAD 2023-07-29T00:27:50.598 DEBUG:boto:Path: /hiiqos-59/obj33 2023-07-29T00:27:50.599 DEBUG:boto:Data: 2023-07-29T00:27:50.599 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.599 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:27:50.600 DEBUG:boto:Port: 8001 2023-07-29T00:27:50.600 DEBUG:boto:Params: {} 2023-07-29T00:27:50.600 DEBUG:boto:Token: None 2023-07-29T00:27:50.601 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj33 2023-07-29T00:27:50.601 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VtAPE1MbVSc7InyrV1GpEZx/gsM= 2023-07-29T00:27:50.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VtAPE1MbVSc7InyrV1GpEZx/gsM=', 'Content-Length': '0'} 2023-07-29T00:27:50.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2fe9b490713097c-0064c45d06-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.604 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:27:50.604 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:27:50.605 DEBUG:boto:Method: HEAD 2023-07-29T00:27:50.605 DEBUG:boto:Path: /hiiqos-59/obj33 2023-07-29T00:27:50.606 DEBUG:boto:Data: 2023-07-29T00:27:50.606 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.606 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:50.606 DEBUG:boto:Port: 8000 2023-07-29T00:27:50.607 DEBUG:boto:Params: {} 2023-07-29T00:27:50.607 DEBUG:boto:Token: None 2023-07-29T00:27:50.608 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj33 2023-07-29T00:27:50.608 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VtAPE1MbVSc7InyrV1GpEZx/gsM= 2023-07-29T00:27:50.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VtAPE1MbVSc7InyrV1GpEZx/gsM=', 'Content-Length': '0'} 2023-07-29T00:27:50.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033fe9436cd6fb587-0064c45d06-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-07-29T00:27:50.613 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:27:50.613 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:27:50.614 DEBUG:boto:Method: GET 2023-07-29T00:27:50.615 DEBUG:boto:Path: /hiiqos-59/obj33 2023-07-29T00:27:50.616 DEBUG:boto:Data: 2023-07-29T00:27:50.617 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.618 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:27:50.618 DEBUG:boto:Port: 8001 2023-07-29T00:27:50.618 DEBUG:boto:Params: {} 2023-07-29T00:27:50.619 DEBUG:boto:Token: None 2023-07-29T00:27:50.619 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj33 2023-07-29T00:27:50.620 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9URMLQk7CS3uXHuvCwJYnh1E3Iw= 2023-07-29T00:27:50.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9URMLQk7CS3uXHuvCwJYnh1E3Iw=', 'Content-Length': '0'} 2023-07-29T00:27:50.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006669bb0535454fbc-0064c45d06-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.623 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:27:50.623 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:27:50.624 DEBUG:boto:Method: GET 2023-07-29T00:27:50.624 DEBUG:boto:Path: /hiiqos-59/obj33 2023-07-29T00:27:50.625 DEBUG:boto:Data: 2023-07-29T00:27:50.625 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.625 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:50.626 DEBUG:boto:Port: 8000 2023-07-29T00:27:50.626 DEBUG:boto:Params: {} 2023-07-29T00:27:50.626 DEBUG:boto:Token: None 2023-07-29T00:27:50.627 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj33 2023-07-29T00:27:50.627 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9URMLQk7CS3uXHuvCwJYnh1E3Iw= 2023-07-29T00:27:50.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9URMLQk7CS3uXHuvCwJYnh1E3Iw=', 'Content-Length': '0'} 2023-07-29T00:27:50.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e080303bfdc01c1c-0064c45d06-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.630 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:27:50.631 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:27:50.631 DEBUG:boto:path=/hiiqos-59/?prefix=obj33 2023-07-29T00:27:50.631 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj33 2023-07-29T00:27:50.632 DEBUG:boto:Method: GET 2023-07-29T00:27:50.632 DEBUG:boto:Path: /hiiqos-59/?prefix=obj33 2023-07-29T00:27:50.632 DEBUG:boto:Data: 2023-07-29T00:27:50.633 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.633 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:27:50.633 DEBUG:boto:Port: 8001 2023-07-29T00:27:50.634 DEBUG:boto:Params: {} 2023-07-29T00:27:50.634 DEBUG:boto:Token: None 2023-07-29T00:27:50.635 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/ 2023-07-29T00:27:50.635 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qqCTonHJd3qVSeStZ2f5/FIiO0k= 2023-07-29T00:27:50.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qqCTonHJd3qVSeStZ2f5/FIiO0k=', 'Content-Length': '0'} 2023-07-29T00:27:50.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d82825c486f1e682-0064c45d06-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.638 DEBUG:boto:b'hiiqos-59obj331000falseobj332023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:27:50.639 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:27:50.640 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:27:50.640 DEBUG:boto:path=/hiiqos-59/?prefix=obj33 2023-07-29T00:27:50.640 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj33 2023-07-29T00:27:50.641 DEBUG:boto:Method: GET 2023-07-29T00:27:50.641 DEBUG:boto:Path: /hiiqos-59/?prefix=obj33 2023-07-29T00:27:50.641 DEBUG:boto:Data: 2023-07-29T00:27:50.642 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.642 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:50.643 DEBUG:boto:Port: 8000 2023-07-29T00:27:50.643 DEBUG:boto:Params: {} 2023-07-29T00:27:50.643 DEBUG:boto:Token: None 2023-07-29T00:27:50.644 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/ 2023-07-29T00:27:50.644 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qqCTonHJd3qVSeStZ2f5/FIiO0k= 2023-07-29T00:27:50.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qqCTonHJd3qVSeStZ2f5/FIiO0k=', 'Content-Length': '0'} 2023-07-29T00:27:50.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5011b5543511c02-0064c45d06-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.648 DEBUG:boto:b'hiiqos-59obj331000falseobj332023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:27:50.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-07-29T00:27:50.649 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:27:50.650 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:27:50.650 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:27:50.650 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:27:50.651 DEBUG:boto:Method: GET 2023-07-29T00:27:50.651 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:27:50.652 DEBUG:boto:Data: 2023-07-29T00:27:50.652 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.652 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:27:50.653 DEBUG:boto:Port: 8001 2023-07-29T00:27:50.653 DEBUG:boto:Params: {} 2023-07-29T00:27:50.653 DEBUG:boto:Token: None 2023-07-29T00:27:50.654 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:27:50.654 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:e9ORPlSn+hUgtHiWbjd7Votdinw= 2023-07-29T00:27:50.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:e9ORPlSn+hUgtHiWbjd7Votdinw=', 'Content-Length': '0'} 2023-07-29T00:27:50.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b37216d740d0c7e-0064c45d06-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.658 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:27:50.658 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:27:50.659 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:27:50.659 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:27:50.659 DEBUG:boto:Method: GET 2023-07-29T00:27:50.660 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:27:50.660 DEBUG:boto:Data: 2023-07-29T00:27:50.661 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.661 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:50.661 DEBUG:boto:Port: 8000 2023-07-29T00:27:50.662 DEBUG:boto:Params: {} 2023-07-29T00:27:50.662 DEBUG:boto:Token: None 2023-07-29T00:27:50.663 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:27:50.663 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:e9ORPlSn+hUgtHiWbjd7Votdinw= 2023-07-29T00:27:50.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:e9ORPlSn+hUgtHiWbjd7Votdinw=', 'Content-Length': '0'} 2023-07-29T00:27:50.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c05df2a72e56ef14-0064c45d06-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.666 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:27:50.667 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:27:50.667 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:27:50.667 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:27:50.668 DEBUG:boto:Method: HEAD 2023-07-29T00:27:50.668 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:27:50.669 DEBUG:boto:Data: 2023-07-29T00:27:50.669 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.669 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:27:50.670 DEBUG:boto:Port: 8001 2023-07-29T00:27:50.670 DEBUG:boto:Params: {} 2023-07-29T00:27:50.670 DEBUG:boto:Token: None 2023-07-29T00:27:50.671 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:27:50.671 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pcqCOn74XzYKNpxL8V+4TSaBxAE= 2023-07-29T00:27:50.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pcqCOn74XzYKNpxL8V+4TSaBxAE=', 'Content-Length': '0'} 2023-07-29T00:27:50.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dde5e968caf0d59-0064c45d06-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.674 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:27:50.674 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:27:50.674 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:27:50.675 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:27:50.675 DEBUG:boto:Method: HEAD 2023-07-29T00:27:50.675 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:27:50.676 DEBUG:boto:Data: 2023-07-29T00:27:50.676 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.677 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:50.677 DEBUG:boto:Port: 8000 2023-07-29T00:27:50.677 DEBUG:boto:Params: {} 2023-07-29T00:27:50.678 DEBUG:boto:Token: None 2023-07-29T00:27:50.678 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:27:50.679 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pcqCOn74XzYKNpxL8V+4TSaBxAE= 2023-07-29T00:27:50.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pcqCOn74XzYKNpxL8V+4TSaBxAE=', 'Content-Length': '0'} 2023-07-29T00:27:50.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eba045939ff25af6-0064c45d06-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-07-29T00:27:50.682 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:27:50.682 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:27:50.683 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:27:50.683 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:27:50.684 DEBUG:boto:Method: GET 2023-07-29T00:27:50.684 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:27:50.685 DEBUG:boto:Data: 2023-07-29T00:27:50.685 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.685 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:27:50.686 DEBUG:boto:Port: 8001 2023-07-29T00:27:50.686 DEBUG:boto:Params: {} 2023-07-29T00:27:50.686 DEBUG:boto:Token: None 2023-07-29T00:27:50.687 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:27:50.687 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:e9ORPlSn+hUgtHiWbjd7Votdinw= 2023-07-29T00:27:50.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:e9ORPlSn+hUgtHiWbjd7Votdinw=', 'Content-Length': '0'} 2023-07-29T00:27:50.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd1882bb33fe8bda-0064c45d06-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.690 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:27:50.691 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:27:50.691 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:27:50.692 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:27:50.692 DEBUG:boto:Method: GET 2023-07-29T00:27:50.692 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:27:50.693 DEBUG:boto:Data: 2023-07-29T00:27:50.693 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.694 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:50.694 DEBUG:boto:Port: 8000 2023-07-29T00:27:50.694 DEBUG:boto:Params: {} 2023-07-29T00:27:50.695 DEBUG:boto:Token: None 2023-07-29T00:27:50.695 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:27:50.696 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:e9ORPlSn+hUgtHiWbjd7Votdinw= 2023-07-29T00:27:50.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:e9ORPlSn+hUgtHiWbjd7Votdinw=', 'Content-Length': '0'} 2023-07-29T00:27:50.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098eeb7b75d0669f4-0064c45d06-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.699 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:27:50.699 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:27:50.699 DEBUG:boto:path=/hiiqos-59/?prefix=obj44 2023-07-29T00:27:50.700 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj44 2023-07-29T00:27:50.700 DEBUG:boto:Method: GET 2023-07-29T00:27:50.701 DEBUG:boto:Path: /hiiqos-59/?prefix=obj44 2023-07-29T00:27:50.701 DEBUG:boto:Data: 2023-07-29T00:27:50.701 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.702 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:27:50.702 DEBUG:boto:Port: 8001 2023-07-29T00:27:50.702 DEBUG:boto:Params: {} 2023-07-29T00:27:50.703 DEBUG:boto:Token: None 2023-07-29T00:27:50.703 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/ 2023-07-29T00:27:50.704 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qqCTonHJd3qVSeStZ2f5/FIiO0k= 2023-07-29T00:27:50.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qqCTonHJd3qVSeStZ2f5/FIiO0k=', 'Content-Length': '0'} 2023-07-29T00:27:50.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062dd5f4adbc258af-0064c45d06-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.707 DEBUG:boto:b'hiiqos-59obj441000falseobj442023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:27:50.708 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:27:50.708 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:27:50.709 DEBUG:boto:path=/hiiqos-59/?prefix=obj44 2023-07-29T00:27:50.709 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj44 2023-07-29T00:27:50.709 DEBUG:boto:Method: GET 2023-07-29T00:27:50.710 DEBUG:boto:Path: /hiiqos-59/?prefix=obj44 2023-07-29T00:27:50.710 DEBUG:boto:Data: 2023-07-29T00:27:50.711 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.711 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:50.711 DEBUG:boto:Port: 8000 2023-07-29T00:27:50.712 DEBUG:boto:Params: {} 2023-07-29T00:27:50.712 DEBUG:boto:Token: None 2023-07-29T00:27:50.713 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/ 2023-07-29T00:27:50.713 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qqCTonHJd3qVSeStZ2f5/FIiO0k= 2023-07-29T00:27:50.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qqCTonHJd3qVSeStZ2f5/FIiO0k=', 'Content-Length': '0'} 2023-07-29T00:27:50.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d27eb6e6303bf8e2-0064c45d06-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.718 DEBUG:boto:b'hiiqos-59obj441000falseobj442023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:27:50.719 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:27:50.720 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:27:50.721 DEBUG:boto:Method: HEAD 2023-07-29T00:27:50.722 DEBUG:boto:Path: /hiiqos-59/obj44 2023-07-29T00:27:50.722 DEBUG:boto:Data: 2023-07-29T00:27:50.723 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.724 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:27:50.724 DEBUG:boto:Port: 8001 2023-07-29T00:27:50.725 DEBUG:boto:Params: {} 2023-07-29T00:27:50.726 DEBUG:boto:Token: None 2023-07-29T00:27:50.727 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj44 2023-07-29T00:27:50.727 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P7z4yOxdYMo5lSqR87m1kQZfsBg= 2023-07-29T00:27:50.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P7z4yOxdYMo5lSqR87m1kQZfsBg=', 'Content-Length': '0'} 2023-07-29T00:27:50.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006506aa971d3a4484-0064c45d06-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.732 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:27:50.733 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:27:50.733 DEBUG:boto:Method: HEAD 2023-07-29T00:27:50.734 DEBUG:boto:Path: /hiiqos-59/obj44 2023-07-29T00:27:50.734 DEBUG:boto:Data: 2023-07-29T00:27:50.735 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.736 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:50.736 DEBUG:boto:Port: 8000 2023-07-29T00:27:50.737 DEBUG:boto:Params: {} 2023-07-29T00:27:50.737 DEBUG:boto:Token: None 2023-07-29T00:27:50.738 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj44 2023-07-29T00:27:50.738 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P7z4yOxdYMo5lSqR87m1kQZfsBg= 2023-07-29T00:27:50.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P7z4yOxdYMo5lSqR87m1kQZfsBg=', 'Content-Length': '0'} 2023-07-29T00:27:50.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1eeede572d09caa-0064c45d06-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-07-29T00:27:50.742 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:27:50.742 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:27:50.742 DEBUG:boto:Method: GET 2023-07-29T00:27:50.743 DEBUG:boto:Path: /hiiqos-59/obj44 2023-07-29T00:27:50.743 DEBUG:boto:Data: 2023-07-29T00:27:50.743 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.744 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:27:50.744 DEBUG:boto:Port: 8001 2023-07-29T00:27:50.745 DEBUG:boto:Params: {} 2023-07-29T00:27:50.745 DEBUG:boto:Token: None 2023-07-29T00:27:50.745 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj44 2023-07-29T00:27:50.746 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:o4WB+neLJZZaLjIiUU2U3wdqktY= 2023-07-29T00:27:50.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:o4WB+neLJZZaLjIiUU2U3wdqktY=', 'Content-Length': '0'} 2023-07-29T00:27:50.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecc8f78848c5ea53-0064c45d06-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.749 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:27:50.749 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:27:50.750 DEBUG:boto:Method: GET 2023-07-29T00:27:50.750 DEBUG:boto:Path: /hiiqos-59/obj44 2023-07-29T00:27:50.750 DEBUG:boto:Data: 2023-07-29T00:27:50.751 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.751 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:50.751 DEBUG:boto:Port: 8000 2023-07-29T00:27:50.752 DEBUG:boto:Params: {} 2023-07-29T00:27:50.752 DEBUG:boto:Token: None 2023-07-29T00:27:50.752 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj44 2023-07-29T00:27:50.753 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:o4WB+neLJZZaLjIiUU2U3wdqktY= 2023-07-29T00:27:50.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:o4WB+neLJZZaLjIiUU2U3wdqktY=', 'Content-Length': '0'} 2023-07-29T00:27:50.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060bdf81f2ce3416f-0064c45d06-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.796 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:27:50.797 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:27:50.798 DEBUG:boto:path=/hiiqos-59/?prefix=obj44 2023-07-29T00:27:50.798 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj44 2023-07-29T00:27:50.799 DEBUG:boto:Method: GET 2023-07-29T00:27:50.799 DEBUG:boto:Path: /hiiqos-59/?prefix=obj44 2023-07-29T00:27:50.799 DEBUG:boto:Data: 2023-07-29T00:27:50.800 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.801 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:27:50.802 DEBUG:boto:Port: 8001 2023-07-29T00:27:50.802 DEBUG:boto:Params: {} 2023-07-29T00:27:50.803 DEBUG:boto:Token: None 2023-07-29T00:27:50.803 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/ 2023-07-29T00:27:50.803 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qqCTonHJd3qVSeStZ2f5/FIiO0k= 2023-07-29T00:27:50.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qqCTonHJd3qVSeStZ2f5/FIiO0k=', 'Content-Length': '0'} 2023-07-29T00:27:50.807 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3d062b2bcc13870-0064c45d06-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.807 DEBUG:boto:b'hiiqos-59obj441000falseobj442023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:27:50.808 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:27:50.809 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:27:50.809 DEBUG:boto:path=/hiiqos-59/?prefix=obj44 2023-07-29T00:27:50.810 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj44 2023-07-29T00:27:50.810 DEBUG:boto:Method: GET 2023-07-29T00:27:50.810 DEBUG:boto:Path: /hiiqos-59/?prefix=obj44 2023-07-29T00:27:50.811 DEBUG:boto:Data: 2023-07-29T00:27:50.811 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.811 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:50.812 DEBUG:boto:Port: 8000 2023-07-29T00:27:50.812 DEBUG:boto:Params: {} 2023-07-29T00:27:50.812 DEBUG:boto:Token: None 2023-07-29T00:27:50.813 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/ 2023-07-29T00:27:50.813 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qqCTonHJd3qVSeStZ2f5/FIiO0k= 2023-07-29T00:27:50.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qqCTonHJd3qVSeStZ2f5/FIiO0k=', 'Content-Length': '0'} 2023-07-29T00:27:50.816 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e709e724da2b50f-0064c45d06-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.817 DEBUG:boto:b'hiiqos-59obj441000falseobj442023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:27:50.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-07-29T00:27:50.818 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:27:50.819 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:27:50.819 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:27:50.820 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:27:50.820 DEBUG:boto:Method: GET 2023-07-29T00:27:50.821 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:27:50.821 DEBUG:boto:Data: 2023-07-29T00:27:50.821 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.822 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:27:50.822 DEBUG:boto:Port: 8001 2023-07-29T00:27:50.822 DEBUG:boto:Params: {} 2023-07-29T00:27:50.823 DEBUG:boto:Token: None 2023-07-29T00:27:50.823 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:27:50.824 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5auvuNSUvq2v+2NCZQD2TnLpkls= 2023-07-29T00:27:50.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5auvuNSUvq2v+2NCZQD2TnLpkls=', 'Content-Length': '0'} 2023-07-29T00:27:50.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfab53881347d0c0-0064c45d06-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.827 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:27:50.827 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:27:50.828 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:27:50.828 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:27:50.828 DEBUG:boto:Method: GET 2023-07-29T00:27:50.829 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:27:50.829 DEBUG:boto:Data: 2023-07-29T00:27:50.829 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.830 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:50.831 DEBUG:boto:Port: 8000 2023-07-29T00:27:50.831 DEBUG:boto:Params: {} 2023-07-29T00:27:50.831 DEBUG:boto:Token: None 2023-07-29T00:27:50.832 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:27:50.833 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5auvuNSUvq2v+2NCZQD2TnLpkls= 2023-07-29T00:27:50.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5auvuNSUvq2v+2NCZQD2TnLpkls=', 'Content-Length': '0'} 2023-07-29T00:27:50.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d5f140304bbae6b-0064c45d06-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.836 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:27:50.837 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:27:50.837 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:27:50.837 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:27:50.838 DEBUG:boto:Method: HEAD 2023-07-29T00:27:50.838 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:27:50.838 DEBUG:boto:Data: 2023-07-29T00:27:50.839 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.839 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:27:50.839 DEBUG:boto:Port: 8001 2023-07-29T00:27:50.840 DEBUG:boto:Params: {} 2023-07-29T00:27:50.840 DEBUG:boto:Token: None 2023-07-29T00:27:50.841 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:27:50.841 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ui8AXtxNKxnQEIfgQyXbQhIToh8= 2023-07-29T00:27:50.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ui8AXtxNKxnQEIfgQyXbQhIToh8=', 'Content-Length': '0'} 2023-07-29T00:27:50.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000238b229878ecb156-0064c45d06-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.844 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:27:50.844 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:27:50.845 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:27:50.845 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:27:50.845 DEBUG:boto:Method: HEAD 2023-07-29T00:27:50.846 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:27:50.846 DEBUG:boto:Data: 2023-07-29T00:27:50.847 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.847 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:50.847 DEBUG:boto:Port: 8000 2023-07-29T00:27:50.848 DEBUG:boto:Params: {} 2023-07-29T00:27:50.848 DEBUG:boto:Token: None 2023-07-29T00:27:50.848 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:27:50.849 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ui8AXtxNKxnQEIfgQyXbQhIToh8= 2023-07-29T00:27:50.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ui8AXtxNKxnQEIfgQyXbQhIToh8=', 'Content-Length': '0'} 2023-07-29T00:27:50.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ddd27afad3c9719-0064c45d06-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-07-29T00:27:50.852 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:27:50.853 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:27:50.854 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:27:50.854 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:27:50.855 DEBUG:boto:Method: GET 2023-07-29T00:27:50.855 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:27:50.855 DEBUG:boto:Data: 2023-07-29T00:27:50.856 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.857 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:27:50.858 DEBUG:boto:Port: 8001 2023-07-29T00:27:50.859 DEBUG:boto:Params: {} 2023-07-29T00:27:50.860 DEBUG:boto:Token: None 2023-07-29T00:27:50.861 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:27:50.862 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5auvuNSUvq2v+2NCZQD2TnLpkls= 2023-07-29T00:27:50.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5auvuNSUvq2v+2NCZQD2TnLpkls=', 'Content-Length': '0'} 2023-07-29T00:27:50.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fd974417b5a8858-0064c45d06-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.865 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:27:50.867 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:27:50.868 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:27:50.868 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:27:50.869 DEBUG:boto:Method: GET 2023-07-29T00:27:50.869 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:27:50.869 DEBUG:boto:Data: 2023-07-29T00:27:50.870 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.870 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:50.870 DEBUG:boto:Port: 8000 2023-07-29T00:27:50.871 DEBUG:boto:Params: {} 2023-07-29T00:27:50.871 DEBUG:boto:Token: None 2023-07-29T00:27:50.872 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:27:50.873 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5auvuNSUvq2v+2NCZQD2TnLpkls= 2023-07-29T00:27:50.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5auvuNSUvq2v+2NCZQD2TnLpkls=', 'Content-Length': '0'} 2023-07-29T00:27:50.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000460dfde0e8f6d47e-0064c45d06-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.876 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:27:50.877 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:27:50.878 DEBUG:boto:path=/hiiqos-59/?prefix=obj55 2023-07-29T00:27:50.878 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj55 2023-07-29T00:27:50.879 DEBUG:boto:Method: GET 2023-07-29T00:27:50.880 DEBUG:boto:Path: /hiiqos-59/?prefix=obj55 2023-07-29T00:27:50.880 DEBUG:boto:Data: 2023-07-29T00:27:50.880 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.881 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:27:50.881 DEBUG:boto:Port: 8001 2023-07-29T00:27:50.882 DEBUG:boto:Params: {} 2023-07-29T00:27:50.883 DEBUG:boto:Token: None 2023-07-29T00:27:50.883 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/ 2023-07-29T00:27:50.884 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qqCTonHJd3qVSeStZ2f5/FIiO0k= 2023-07-29T00:27:50.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qqCTonHJd3qVSeStZ2f5/FIiO0k=', 'Content-Length': '0'} 2023-07-29T00:27:50.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f40768b054d0f20-0064c45d06-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.932 DEBUG:boto:b'hiiqos-59obj551000falseobj552023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:27:50.933 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:27:50.934 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:27:50.934 DEBUG:boto:path=/hiiqos-59/?prefix=obj55 2023-07-29T00:27:50.935 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj55 2023-07-29T00:27:50.935 DEBUG:boto:Method: GET 2023-07-29T00:27:50.935 DEBUG:boto:Path: /hiiqos-59/?prefix=obj55 2023-07-29T00:27:50.936 DEBUG:boto:Data: 2023-07-29T00:27:50.936 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.937 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:50.937 DEBUG:boto:Port: 8000 2023-07-29T00:27:50.937 DEBUG:boto:Params: {} 2023-07-29T00:27:50.938 DEBUG:boto:Token: None 2023-07-29T00:27:50.938 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/ 2023-07-29T00:27:50.939 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qqCTonHJd3qVSeStZ2f5/FIiO0k= 2023-07-29T00:27:50.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qqCTonHJd3qVSeStZ2f5/FIiO0k=', 'Content-Length': '0'} 2023-07-29T00:27:50.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003227a673c9a43046-0064c45d06-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.942 DEBUG:boto:b'hiiqos-59obj551000falseobj552023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:27:50.943 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:27:50.943 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:27:50.944 DEBUG:boto:Method: HEAD 2023-07-29T00:27:50.944 DEBUG:boto:Path: /hiiqos-59/obj55 2023-07-29T00:27:50.944 DEBUG:boto:Data: 2023-07-29T00:27:50.945 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.945 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:27:50.945 DEBUG:boto:Port: 8001 2023-07-29T00:27:50.946 DEBUG:boto:Params: {} 2023-07-29T00:27:50.946 DEBUG:boto:Token: None 2023-07-29T00:27:50.947 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj55 2023-07-29T00:27:50.947 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:X0Fuob6XHfvGubS40xvNsVw/E/4= 2023-07-29T00:27:50.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:X0Fuob6XHfvGubS40xvNsVw/E/4=', 'Content-Length': '0'} 2023-07-29T00:27:50.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaa70a1b4e787fa1-0064c45d06-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.950 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:27:50.951 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:27:50.951 DEBUG:boto:Method: HEAD 2023-07-29T00:27:50.951 DEBUG:boto:Path: /hiiqos-59/obj55 2023-07-29T00:27:50.952 DEBUG:boto:Data: 2023-07-29T00:27:50.952 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.953 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:50.953 DEBUG:boto:Port: 8000 2023-07-29T00:27:50.953 DEBUG:boto:Params: {} 2023-07-29T00:27:50.954 DEBUG:boto:Token: None 2023-07-29T00:27:50.954 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj55 2023-07-29T00:27:50.955 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:X0Fuob6XHfvGubS40xvNsVw/E/4= 2023-07-29T00:27:50.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:X0Fuob6XHfvGubS40xvNsVw/E/4=', 'Content-Length': '0'} 2023-07-29T00:27:50.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001608549da1b56646-0064c45d06-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-07-29T00:27:50.958 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:27:50.959 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:27:50.959 DEBUG:boto:Method: GET 2023-07-29T00:27:50.959 DEBUG:boto:Path: /hiiqos-59/obj55 2023-07-29T00:27:50.960 DEBUG:boto:Data: 2023-07-29T00:27:50.960 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.960 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:27:50.961 DEBUG:boto:Port: 8001 2023-07-29T00:27:50.961 DEBUG:boto:Params: {} 2023-07-29T00:27:50.962 DEBUG:boto:Token: None 2023-07-29T00:27:50.962 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj55 2023-07-29T00:27:50.962 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lJ6IPQO9mzHSq+ab279w4W+psDg= 2023-07-29T00:27:50.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lJ6IPQO9mzHSq+ab279w4W+psDg=', 'Content-Length': '0'} 2023-07-29T00:27:50.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069664df551789c2c-0064c45d06-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:50.965 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:27:50.966 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:27:50.966 DEBUG:boto:Method: GET 2023-07-29T00:27:50.967 DEBUG:boto:Path: /hiiqos-59/obj55 2023-07-29T00:27:50.967 DEBUG:boto:Data: 2023-07-29T00:27:50.967 DEBUG:boto:Headers: {} 2023-07-29T00:27:50.968 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:50.968 DEBUG:boto:Port: 8000 2023-07-29T00:27:50.968 DEBUG:boto:Params: {} 2023-07-29T00:27:50.969 DEBUG:boto:Token: None 2023-07-29T00:27:50.969 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:50 GMT /hiiqos-59/obj55 2023-07-29T00:27:50.970 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lJ6IPQO9mzHSq+ab279w4W+psDg= 2023-07-29T00:27:50.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lJ6IPQO9mzHSq+ab279w4W+psDg=', 'Content-Length': '0'} 2023-07-29T00:27:50.972 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5cbe365c25069af-0064c45d06-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:51.012 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:27:51.013 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:27:51.013 DEBUG:boto:path=/hiiqos-59/?prefix=obj55 2023-07-29T00:27:51.014 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj55 2023-07-29T00:27:51.014 DEBUG:boto:Method: GET 2023-07-29T00:27:51.014 DEBUG:boto:Path: /hiiqos-59/?prefix=obj55 2023-07-29T00:27:51.015 DEBUG:boto:Data: 2023-07-29T00:27:51.015 DEBUG:boto:Headers: {} 2023-07-29T00:27:51.015 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:27:51.016 DEBUG:boto:Port: 8001 2023-07-29T00:27:51.016 DEBUG:boto:Params: {} 2023-07-29T00:27:51.017 DEBUG:boto:Token: None 2023-07-29T00:27:51.017 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:51 GMT /hiiqos-59/ 2023-07-29T00:27:51.017 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9ZWdXMFszAdccb2o9HVfp1JgZds= 2023-07-29T00:27:51.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9ZWdXMFszAdccb2o9HVfp1JgZds=', 'Content-Length': '0'} 2023-07-29T00:27:51.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078c00ba7ad7c1aaa-0064c45d06-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:51.021 DEBUG:boto:b'hiiqos-59obj551000falseobj552023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:27:51.022 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:27:51.022 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:27:51.022 DEBUG:boto:path=/hiiqos-59/?prefix=obj55 2023-07-29T00:27:51.023 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj55 2023-07-29T00:27:51.023 DEBUG:boto:Method: GET 2023-07-29T00:27:51.023 DEBUG:boto:Path: /hiiqos-59/?prefix=obj55 2023-07-29T00:27:51.024 DEBUG:boto:Data: 2023-07-29T00:27:51.024 DEBUG:boto:Headers: {} 2023-07-29T00:27:51.025 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:51.025 DEBUG:boto:Port: 8000 2023-07-29T00:27:51.026 DEBUG:boto:Params: {} 2023-07-29T00:27:51.026 DEBUG:boto:Token: None 2023-07-29T00:27:51.026 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:27:51 GMT /hiiqos-59/ 2023-07-29T00:27:51.027 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9ZWdXMFszAdccb2o9HVfp1JgZds= 2023-07-29T00:27:51.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9ZWdXMFszAdccb2o9HVfp1JgZds=', 'Content-Length': '0'} 2023-07-29T00:27:51.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8db74871c137e6a-0064c45d06-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:27:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:51.030 DEBUG:boto:b'hiiqos-59obj551000falseobj552023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:27:51.031 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-59 zones={a2, a1} 2023-07-29T00:27:51.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:27:51.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:27:51.032 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:27:51.088 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:27:51.088 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:27:51.132 INFO:teuthology.orchestra.run.smithi101.stdout:tenant: 2023-07-29T00:27:51.133 INFO:teuthology.orchestra.run.smithi101.stdout:bucket name: hiiqos-59 2023-07-29T00:27:51.136 INFO:teuthology.orchestra.run.smithi101.stdout:total entries: 5 2023-07-29T00:27:51.172 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:27:51.173 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:27:51.173 DEBUG:boto:Method: HEAD 2023-07-29T00:27:51.174 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:27:51.174 DEBUG:boto:Data: 2023-07-29T00:27:51.174 DEBUG:boto:Headers: {} 2023-07-29T00:27:51.175 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:51.175 DEBUG:boto:Port: 8000 2023-07-29T00:27:51.175 DEBUG:boto:Params: {} 2023-07-29T00:27:51.176 DEBUG:boto:Token: None 2023-07-29T00:27:51.176 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:27:51 GMT /hiiqos-59/ 2023-07-29T00:27:51.177 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3/POKxhqIA1hpQW5NAky7pMScNE= 2023-07-29T00:27:51.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3/POKxhqIA1hpQW5NAky7pMScNE=', 'Content-Length': '0'} 2023-07-29T00:27:51.180 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000076da7b04281c9ef5-0064c45d07-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:51.181 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:27:51.181 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:27:51.182 DEBUG:boto:Method: PUT 2023-07-29T00:27:51.182 DEBUG:boto:Path: /hiiqos-59/obj17 2023-07-29T00:27:51.182 DEBUG:boto:Data: 2023-07-29T00:27:51.183 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:27:51.183 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:51.183 DEBUG:boto:Port: 8000 2023-07-29T00:27:51.184 DEBUG:boto:Params: {} 2023-07-29T00:27:51.184 DEBUG:boto:Token: None 2023-07-29T00:27:51.185 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:27:51 GMT /hiiqos-59/obj17 2023-07-29T00:27:51.185 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pPxt87s4BmcPms7LPMLiNlFNZq8= 2023-07-29T00:27:51.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:27:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pPxt87s4BmcPms7LPMLiNlFNZq8='} 2023-07-29T00:27:51.190 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000073feeb3cfa12378-0064c45d07-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:51.192 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:27:51.192 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:27:51.193 DEBUG:boto:Method: HEAD 2023-07-29T00:27:51.194 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:27:51.195 DEBUG:boto:Data: 2023-07-29T00:27:51.195 DEBUG:boto:Headers: {} 2023-07-29T00:27:51.196 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:51.197 DEBUG:boto:Port: 8000 2023-07-29T00:27:51.198 DEBUG:boto:Params: {} 2023-07-29T00:27:51.198 DEBUG:boto:Token: None 2023-07-29T00:27:51.199 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:27:51 GMT /hiiqos-59/ 2023-07-29T00:27:51.199 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3/POKxhqIA1hpQW5NAky7pMScNE= 2023-07-29T00:27:51.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3/POKxhqIA1hpQW5NAky7pMScNE=', 'Content-Length': '0'} 2023-07-29T00:27:51.204 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002652bf366bc4d3dc-0064c45d07-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:51.206 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:27:51.206 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:27:51.206 DEBUG:boto:Method: PUT 2023-07-29T00:27:51.207 DEBUG:boto:Path: /hiiqos-59/obj34 2023-07-29T00:27:51.207 DEBUG:boto:Data: 2023-07-29T00:27:51.207 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:27:51.208 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:51.208 DEBUG:boto:Port: 8000 2023-07-29T00:27:51.208 DEBUG:boto:Params: {} 2023-07-29T00:27:51.209 DEBUG:boto:Token: None 2023-07-29T00:27:51.209 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:27:51 GMT /hiiqos-59/obj34 2023-07-29T00:27:51.210 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:M69fObIH4sbOPxLWiBiSTIPeb74= 2023-07-29T00:27:51.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:27:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:M69fObIH4sbOPxLWiBiSTIPeb74='} 2023-07-29T00:27:51.216 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006aa9e69e46213952-0064c45d07-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:51.217 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:27:51.217 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:27:51.218 DEBUG:boto:Method: HEAD 2023-07-29T00:27:51.218 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:27:51.218 DEBUG:boto:Data: 2023-07-29T00:27:51.219 DEBUG:boto:Headers: {} 2023-07-29T00:27:51.219 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:51.219 DEBUG:boto:Port: 8000 2023-07-29T00:27:51.219 DEBUG:boto:Params: {} 2023-07-29T00:27:51.221 DEBUG:boto:Token: None 2023-07-29T00:27:51.222 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:27:51 GMT /hiiqos-59/ 2023-07-29T00:27:51.222 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3/POKxhqIA1hpQW5NAky7pMScNE= 2023-07-29T00:27:51.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3/POKxhqIA1hpQW5NAky7pMScNE=', 'Content-Length': '0'} 2023-07-29T00:27:51.226 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bc358729cba65e62-0064c45d07-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:51.226 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:27:51.227 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:27:51.227 DEBUG:boto:Method: PUT 2023-07-29T00:27:51.228 DEBUG:boto:Path: /hiiqos-59/obj51 2023-07-29T00:27:51.228 DEBUG:boto:Data: 2023-07-29T00:27:51.228 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:27:51.229 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:51.229 DEBUG:boto:Port: 8000 2023-07-29T00:27:51.230 DEBUG:boto:Params: {} 2023-07-29T00:27:51.231 DEBUG:boto:Token: None 2023-07-29T00:27:51.231 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:27:51 GMT /hiiqos-59/obj51 2023-07-29T00:27:51.232 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4zz25xqv9uErJOsfdLiRYnsob1I= 2023-07-29T00:27:51.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:27:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4zz25xqv9uErJOsfdLiRYnsob1I='} 2023-07-29T00:27:51.280 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000067012e4a6108f149-0064c45d07-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:51.281 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:27:51.281 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:27:51.282 DEBUG:boto:Method: HEAD 2023-07-29T00:27:51.282 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:27:51.282 DEBUG:boto:Data: 2023-07-29T00:27:51.283 DEBUG:boto:Headers: {} 2023-07-29T00:27:51.283 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:51.284 DEBUG:boto:Port: 8000 2023-07-29T00:27:51.284 DEBUG:boto:Params: {} 2023-07-29T00:27:51.285 DEBUG:boto:Token: None 2023-07-29T00:27:51.285 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:27:51 GMT /hiiqos-59/ 2023-07-29T00:27:51.286 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3/POKxhqIA1hpQW5NAky7pMScNE= 2023-07-29T00:27:51.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3/POKxhqIA1hpQW5NAky7pMScNE=', 'Content-Length': '0'} 2023-07-29T00:27:51.289 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000041629a92db9aef82-0064c45d07-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:51.290 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:27:51.291 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:27:51.292 DEBUG:boto:Method: PUT 2023-07-29T00:27:51.293 DEBUG:boto:Path: /hiiqos-59/obj68 2023-07-29T00:27:51.293 DEBUG:boto:Data: 2023-07-29T00:27:51.294 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:27:51.295 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:51.296 DEBUG:boto:Port: 8000 2023-07-29T00:27:51.297 DEBUG:boto:Params: {} 2023-07-29T00:27:51.298 DEBUG:boto:Token: None 2023-07-29T00:27:51.299 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:27:51 GMT /hiiqos-59/obj68 2023-07-29T00:27:51.300 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7wK5zXckUrs7SEmQKncP/3mDUy4= 2023-07-29T00:27:51.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:27:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7wK5zXckUrs7SEmQKncP/3mDUy4='} 2023-07-29T00:27:51.306 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000caf1d5f66a685d69-0064c45d07-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:51.307 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:27:51.308 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:27:51.309 DEBUG:boto:Method: HEAD 2023-07-29T00:27:51.309 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:27:51.310 DEBUG:boto:Data: 2023-07-29T00:27:51.311 DEBUG:boto:Headers: {} 2023-07-29T00:27:51.311 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:51.312 DEBUG:boto:Port: 8000 2023-07-29T00:27:51.313 DEBUG:boto:Params: {} 2023-07-29T00:27:51.314 DEBUG:boto:Token: None 2023-07-29T00:27:51.314 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:27:51 GMT /hiiqos-59/ 2023-07-29T00:27:51.315 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3/POKxhqIA1hpQW5NAky7pMScNE= 2023-07-29T00:27:51.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:27:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3/POKxhqIA1hpQW5NAky7pMScNE=', 'Content-Length': '0'} 2023-07-29T00:27:51.319 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f240d383b2fd3922-0064c45d07-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:51.320 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:27:51.320 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:27:51.321 DEBUG:boto:Method: PUT 2023-07-29T00:27:51.321 DEBUG:boto:Path: /hiiqos-59/obj85 2023-07-29T00:27:51.322 DEBUG:boto:Data: 2023-07-29T00:27:51.322 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:27:51.323 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:27:51.323 DEBUG:boto:Port: 8000 2023-07-29T00:27:51.324 DEBUG:boto:Params: {} 2023-07-29T00:27:51.324 DEBUG:boto:Token: None 2023-07-29T00:27:51.325 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:27:51 GMT /hiiqos-59/obj85 2023-07-29T00:27:51.326 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jPjRbtRBx4EbEYp7Dp+1bJ25VOA= 2023-07-29T00:27:51.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:27:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jPjRbtRBx4EbEYp7Dp+1bJ25VOA='} 2023-07-29T00:27:51.377 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d6812bfd9adcdc4a-0064c45d07-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:27:51.378 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-59 2023-07-29T00:27:51.379 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:27:51.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:27:51.380 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:27:51.432 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:27:51.433 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:27:51.466 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:27:51.411+0000 7fb315124880 1 waiting to reach latest gen 1, on 0.. 2023-07-29T00:27:56.466 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:27:56.412+0000 7fb315124880 1 waiting to reach latest gen 1, on 0.. 2023-07-29T00:28:01.468 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:28:01.413+0000 7fb315124880 1 waiting to reach latest gen 1, on 0.. 2023-07-29T00:28:06.470 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:28:06.414+0000 7fb315124880 1 waiting for incremental sync to catch up: 2023-07-29T00:28:06.470 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , , , , , , , , , ] 2023-07-29T00:28:06.470 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000001.74.6, , , , , , 00000000001.40.6, , 00000000001.74.6, , , , 00000000001.46.6, 00000000001.60.6, , , ] 2023-07-29T00:28:11.471 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:28:11.416+0000 7fb315124880 1 waiting for incremental sync to catch up: 2023-07-29T00:28:11.471 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , , , , , , , , , ] 2023-07-29T00:28:11.471 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000001.74.6, , , , , , 00000000001.40.6, , 00000000001.74.6, , , , 00000000001.46.6, 00000000001.60.6, , , ] 2023-07-29T00:28:16.472 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:28:16.417+0000 7fb315124880 1 waiting for incremental sync to catch up: 2023-07-29T00:28:16.473 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , , , , , , , , , ] 2023-07-29T00:28:16.473 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000001.74.6, , , , , , 00000000001.40.6, , 00000000001.74.6, , , , 00000000001.46.6, 00000000001.60.6, , , ] 2023-07-29T00:28:21.474 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:28:21.419+0000 7fb315124880 1 waiting for incremental sync to catch up: 2023-07-29T00:28:21.474 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , , , , , , , , , ] 2023-07-29T00:28:21.475 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000001.74.6, , , , , , 00000000001.40.6, , 00000000001.74.6, , , , 00000000001.46.6, 00000000001.60.6, , , ] 2023-07-29T00:28:26.476 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:28:26.420+0000 7fb315124880 1 waiting for incremental sync to catch up: 2023-07-29T00:28:26.476 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , , , , , , , , , ] 2023-07-29T00:28:26.477 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000001.74.6, , , , , , 00000000001.40.6, , 00000000001.74.6, , , , 00000000001.46.6, 00000000001.60.6, , , ] 2023-07-29T00:28:31.477 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:28:31.422+0000 7fb315124880 1 waiting for incremental sync to catch up: 2023-07-29T00:28:31.478 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , , , , , , , , , ] 2023-07-29T00:28:31.478 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000001.74.6, , , , , , 00000000001.40.6, , 00000000001.74.6, , , , 00000000001.46.6, 00000000001.60.6, , , ] 2023-07-29T00:28:36.479 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:28:36.424+0000 7fb315124880 1 waiting for incremental sync to catch up: 2023-07-29T00:28:36.480 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , , , , , , , , , ] 2023-07-29T00:28:36.480 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000001.74.6, , , , , , 00000000001.40.6, , 00000000001.74.6, , , , 00000000001.46.6, 00000000001.60.6, , , ] 2023-07-29T00:28:41.481 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:28:41.425+0000 7fb315124880 1 waiting for incremental sync to catch up: 2023-07-29T00:28:41.481 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , , , , , , , , , ] 2023-07-29T00:28:41.482 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000001.74.6, , , , , , 00000000001.40.6, , 00000000001.74.6, , , , 00000000001.46.6, 00000000001.60.6, , , ] 2023-07-29T00:28:46.483 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:28:46.427+0000 7fb315124880 1 waiting for incremental sync to catch up: 2023-07-29T00:28:46.483 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000001.40.6, , , , , , , 00000000001.60.6, , , ] 2023-07-29T00:28:46.483 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000001.74.6, , , , , , 00000000001.40.6, , 00000000001.74.6, , , , 00000000001.46.6, 00000000001.60.6, , , ] 2023-07-29T00:28:51.484 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:28:51.429+0000 7fb315124880 1 waiting for incremental sync to catch up: 2023-07-29T00:28:51.485 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000001.40.6, , , , , , , 00000000001.60.6, , , ] 2023-07-29T00:28:51.485 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000001.74.6, , , , , , 00000000001.40.6, , 00000000001.74.6, , , , 00000000001.46.6, 00000000001.60.6, , , ] 2023-07-29T00:28:56.486 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:28:56.430+0000 7fb315124880 1 waiting for incremental sync to catch up: 2023-07-29T00:28:56.486 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000001.40.6, , , , , , , 00000000001.60.6, , , ] 2023-07-29T00:28:56.487 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000001.74.6, , , , , , 00000000001.40.6, , 00000000001.74.6, , , , 00000000001.46.6, 00000000001.60.6, , , ] 2023-07-29T00:29:01.488 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:29:01.432+0000 7fb315124880 1 waiting for incremental sync to catch up: 2023-07-29T00:29:01.488 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000001.40.6, , , , , , , 00000000001.60.6, , , ] 2023-07-29T00:29:01.488 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000001.74.6, , , , , , 00000000001.40.6, , 00000000001.74.6, , , , 00000000001.46.6, 00000000001.60.6, , , ] 2023-07-29T00:29:03.108 INFO:tasks.rgw.c1.client.0.smithi101.stdout:2023-07-29T00:29:03.052+0000 7f0081a5e640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T00:29:06.489 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:29:06.433+0000 7fb315124880 1 waiting for incremental sync to catch up: 2023-07-29T00:29:06.490 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000001.40.6, , , , , , , 00000000001.60.6, , , ] 2023-07-29T00:29:06.490 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000001.74.6, , , , , , 00000000001.40.6, , 00000000001.74.6, , , , 00000000001.46.6, 00000000001.60.6, , , ] 2023-07-29T00:29:11.491 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:29:11.435+0000 7fb315124880 1 waiting for incremental sync to catch up: 2023-07-29T00:29:11.491 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000001.40.6, , , , , , , 00000000001.60.6, , , ] 2023-07-29T00:29:11.491 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000001.74.6, , , , , , 00000000001.40.6, , 00000000001.74.6, , , , 00000000001.46.6, 00000000001.60.6, , , ] 2023-07-29T00:29:16.493 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:29:16.436+0000 7fb315124880 1 waiting for incremental sync to catch up: 2023-07-29T00:29:16.493 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000001.40.6, , , , , , , 00000000001.60.6, , , ] 2023-07-29T00:29:16.493 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000001.74.6, , , , , , 00000000001.40.6, , 00000000001.74.6, , , , 00000000001.46.6, 00000000001.60.6, , , ] 2023-07-29T00:29:21.494 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:29:21.438+0000 7fb315124880 1 waiting for incremental sync to catch up: 2023-07-29T00:29:21.494 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000001.40.6, , , , , , , 00000000001.60.6, , , ] 2023-07-29T00:29:21.495 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000001.74.6, , , , , , 00000000001.40.6, , 00000000001.74.6, , , , 00000000001.46.6, 00000000001.60.6, , , ] 2023-07-29T00:29:26.496 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:29:26.439+0000 7fb315124880 1 bucket sync caught up with source: 2023-07-29T00:29:26.496 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000001.74.6, , , , , , 00000000001.40.6, , 00000000001.74.6, , , , 00000000001.46.6, 00000000001.60.6, , , ] 2023-07-29T00:29:26.497 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000001.74.6, , , , , , 00000000001.40.6, , 00000000001.74.6, , , , 00000000001.46.6, 00000000001.60.6, , , ] 2023-07-29T00:29:26.497 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:29:26.439+0000 7fb315124880 0 bucket checkpoint complete 2023-07-29T00:29:26.513 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-59 2023-07-29T00:29:26.514 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:29:26.515 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:29:26.515 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:29:26.573 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:29:26.573 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:29:26.610 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:29:26.553+0000 7f9af1a16880 1 waiting for incremental sync to catch up: 2023-07-29T00:29:26.610 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , 00000000001.690.6, , , , , , 00000000001.285.6, 00000000002.775.6, 00000000003.1363.6] 2023-07-29T00:29:26.611 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000001.844.6, 00000000001.690.6, , , 00000000001.729.6, , , 00000000001.285.6, 00000000003.783.6, 00000000003.1363.6] 2023-07-29T00:29:31.611 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:29:31.555+0000 7f9af1a16880 1 waiting for incremental sync to catch up: 2023-07-29T00:29:31.611 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , 00000000001.690.6, , , , , , 00000000001.285.6, 00000000002.775.6, 00000000003.1363.6] 2023-07-29T00:29:31.611 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000001.844.6, 00000000001.690.6, , , 00000000001.729.6, , , 00000000001.285.6, 00000000003.783.6, 00000000003.1363.6] 2023-07-29T00:29:36.612 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:29:36.556+0000 7f9af1a16880 1 waiting for incremental sync to catch up: 2023-07-29T00:29:36.613 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , 00000000001.690.6, , , , , , 00000000001.285.6, 00000000002.775.6, 00000000003.1363.6] 2023-07-29T00:29:36.613 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000001.844.6, 00000000001.690.6, , , 00000000001.729.6, , , 00000000001.285.6, 00000000003.783.6, 00000000003.1363.6] 2023-07-29T00:29:41.614 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:29:41.557+0000 7f9af1a16880 1 waiting for incremental sync to catch up: 2023-07-29T00:29:41.614 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , 00000000001.690.6, , , , , , 00000000001.285.6, 00000000002.775.6, 00000000003.1363.6] 2023-07-29T00:29:41.614 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000001.844.6, 00000000001.690.6, , , 00000000001.729.6, , , 00000000001.285.6, 00000000003.783.6, 00000000003.1363.6] 2023-07-29T00:29:46.615 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:29:46.559+0000 7f9af1a16880 1 bucket sync caught up with source: 2023-07-29T00:29:46.615 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000001.844.6, 00000000001.690.6, , , 00000000001.729.6, , , 00000000001.285.6, 00000000003.783.6, 00000000003.1363.6] 2023-07-29T00:29:46.616 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000001.844.6, 00000000001.690.6, , , 00000000001.729.6, , , 00000000001.285.6, 00000000003.783.6, 00000000003.1363.6] 2023-07-29T00:29:46.616 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:29:46.559+0000 7f9af1a16880 0 bucket checkpoint complete 2023-07-29T00:29:46.632 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-59 zones={a2, a1} 2023-07-29T00:29:46.633 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:46.633 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:46.634 DEBUG:boto:Method: HEAD 2023-07-29T00:29:46.634 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:29:46.635 DEBUG:boto:Data: 2023-07-29T00:29:46.635 DEBUG:boto:Headers: {} 2023-07-29T00:29:46.635 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:46.636 DEBUG:boto:Port: 8001 2023-07-29T00:29:46.636 DEBUG:boto:Params: {} 2023-07-29T00:29:46.636 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-29T00:29:46.637 DEBUG:boto:Token: None 2023-07-29T00:29:46.637 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:46 GMT /hiiqos-59/ 2023-07-29T00:29:46.638 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:x1xTjZe1JfoG2yGTe6NUBKxLfIE= 2023-07-29T00:29:46.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:x1xTjZe1JfoG2yGTe6NUBKxLfIE=', 'Content-Length': '0'} 2023-07-29T00:29:51.643 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a62eec5a0c9687c-0064c45d7f-1c08-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:29:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:51.644 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:51.644 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:51.645 DEBUG:boto:Method: HEAD 2023-07-29T00:29:51.645 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:29:51.645 DEBUG:boto:Data: 2023-07-29T00:29:51.646 DEBUG:boto:Headers: {} 2023-07-29T00:29:51.646 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:51.646 DEBUG:boto:Port: 8000 2023-07-29T00:29:51.647 DEBUG:boto:Params: {} 2023-07-29T00:29:51.647 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-29T00:29:51.648 DEBUG:boto:Token: None 2023-07-29T00:29:51.648 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:51 GMT /hiiqos-59/ 2023-07-29T00:29:51.649 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:T50GILXjnGmW2Q1eq0q7ZCwKcTo= 2023-07-29T00:29:51.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:T50GILXjnGmW2Q1eq0q7ZCwKcTo=', 'Content-Length': '0'} 2023-07-29T00:29:56.653 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d4fd00db8895fce8-0064c45d84-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:29:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:56.654 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-29T00:29:56.655 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:56.655 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:56.655 DEBUG:boto:path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:29:56.656 DEBUG:boto:auth_path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:29:56.656 DEBUG:boto:Method: GET 2023-07-29T00:29:56.657 DEBUG:boto:Path: /hiiqos-59/?versions&max-keys=999 2023-07-29T00:29:56.657 DEBUG:boto:Data: 2023-07-29T00:29:56.658 DEBUG:boto:Headers: {} 2023-07-29T00:29:56.658 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:56.658 DEBUG:boto:Port: 8001 2023-07-29T00:29:56.659 DEBUG:boto:Params: {} 2023-07-29T00:29:56.659 DEBUG:boto:Token: None 2023-07-29T00:29:56.660 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:56 GMT /hiiqos-59/?versions 2023-07-29T00:29:56.661 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5UiEYJbNGw3C8vUX2j3EjQucutM= 2023-07-29T00:29:56.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5UiEYJbNGw3C8vUX2j3EjQucutM=', 'Content-Length': '0'} 2023-07-29T00:29:56.665 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5916a69457f345e-0064c45d84-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:56.666 DEBUG:boto:b'hiiqos-59999falseobj11nulltrue2023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:56.668 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-07-29T00:29:56.669 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-07-29T00:29:56.669 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-07-29T00:29:56.669 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-07-29T00:29:56.670 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-07-29T00:29:56.670 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-07-29T00:29:56.671 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-07-29T00:29:56.671 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-07-29T00:29:56.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-07-29T00:29:56.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-07-29T00:29:56.673 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-29T00:29:56.674 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:56.674 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:56.675 DEBUG:boto:path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:29:56.676 DEBUG:boto:auth_path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:29:56.676 DEBUG:boto:Method: GET 2023-07-29T00:29:56.677 DEBUG:boto:Path: /hiiqos-59/?versions&max-keys=999 2023-07-29T00:29:56.678 DEBUG:boto:Data: 2023-07-29T00:29:56.679 DEBUG:boto:Headers: {} 2023-07-29T00:29:56.679 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:56.680 DEBUG:boto:Port: 8000 2023-07-29T00:29:56.681 DEBUG:boto:Params: {} 2023-07-29T00:29:56.682 DEBUG:boto:Token: None 2023-07-29T00:29:56.683 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:56 GMT /hiiqos-59/?versions 2023-07-29T00:29:56.684 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5UiEYJbNGw3C8vUX2j3EjQucutM= 2023-07-29T00:29:56.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5UiEYJbNGw3C8vUX2j3EjQucutM=', 'Content-Length': '0'} 2023-07-29T00:29:56.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fa8693b3488f45b-0064c45d84-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:56.689 DEBUG:boto:b'hiiqos-59999falseobj11nulltrue2023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:56.690 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-07-29T00:29:56.690 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-07-29T00:29:56.690 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-07-29T00:29:56.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-07-29T00:29:56.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-07-29T00:29:56.692 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-07-29T00:29:56.692 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-07-29T00:29:56.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-07-29T00:29:56.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-07-29T00:29:56.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-07-29T00:29:56.694 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:56.694 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:56.694 DEBUG:boto:path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:29:56.695 DEBUG:boto:auth_path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:29:56.695 DEBUG:boto:Method: GET 2023-07-29T00:29:56.695 DEBUG:boto:Path: /hiiqos-59/?versions&max-keys=999 2023-07-29T00:29:56.696 DEBUG:boto:Data: 2023-07-29T00:29:56.696 DEBUG:boto:Headers: {} 2023-07-29T00:29:56.697 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:56.697 DEBUG:boto:Port: 8001 2023-07-29T00:29:56.697 DEBUG:boto:Params: {} 2023-07-29T00:29:56.698 DEBUG:boto:Token: None 2023-07-29T00:29:56.698 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:56 GMT /hiiqos-59/?versions 2023-07-29T00:29:56.699 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5UiEYJbNGw3C8vUX2j3EjQucutM= 2023-07-29T00:29:56.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5UiEYJbNGw3C8vUX2j3EjQucutM=', 'Content-Length': '0'} 2023-07-29T00:29:56.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084bb726ec9bff321-0064c45d84-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:56.704 DEBUG:boto:b'hiiqos-59999falseobj11nulltrue2023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:56.705 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:56.705 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:56.706 DEBUG:boto:path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:29:56.706 DEBUG:boto:auth_path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:29:56.706 DEBUG:boto:Method: GET 2023-07-29T00:29:56.707 DEBUG:boto:Path: /hiiqos-59/?versions&max-keys=999 2023-07-29T00:29:56.707 DEBUG:boto:Data: 2023-07-29T00:29:56.708 DEBUG:boto:Headers: {} 2023-07-29T00:29:56.708 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:56.708 DEBUG:boto:Port: 8000 2023-07-29T00:29:56.709 DEBUG:boto:Params: {} 2023-07-29T00:29:56.709 DEBUG:boto:Token: None 2023-07-29T00:29:56.710 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:56 GMT /hiiqos-59/?versions 2023-07-29T00:29:56.710 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5UiEYJbNGw3C8vUX2j3EjQucutM= 2023-07-29T00:29:56.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5UiEYJbNGw3C8vUX2j3EjQucutM=', 'Content-Length': '0'} 2023-07-29T00:29:56.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2c20f9a7a1756b5-0064c45d84-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:56.714 DEBUG:boto:b'hiiqos-59999falseobj11nulltrue2023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:56.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-07-29T00:29:56.716 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:29:56.716 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:29:56.716 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:29:56.717 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:29:56.717 DEBUG:boto:Method: GET 2023-07-29T00:29:56.718 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:29:56.718 DEBUG:boto:Data: 2023-07-29T00:29:56.718 DEBUG:boto:Headers: {} 2023-07-29T00:29:56.719 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:56.719 DEBUG:boto:Port: 8001 2023-07-29T00:29:56.720 DEBUG:boto:Params: {} 2023-07-29T00:29:56.720 DEBUG:boto:Token: None 2023-07-29T00:29:56.720 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:56 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:29:56.721 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RWiGZqvFAgAwcg2OOHNtm27WQvY= 2023-07-29T00:29:56.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RWiGZqvFAgAwcg2OOHNtm27WQvY=', 'Content-Length': '0'} 2023-07-29T00:29:56.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f0cc31541862f47-0064c45d84-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:56.724 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:29:56.725 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:29:56.725 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:29:56.725 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:29:56.726 DEBUG:boto:Method: GET 2023-07-29T00:29:56.726 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:29:56.726 DEBUG:boto:Data: 2023-07-29T00:29:56.727 DEBUG:boto:Headers: {} 2023-07-29T00:29:56.727 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:56.727 DEBUG:boto:Port: 8000 2023-07-29T00:29:56.728 DEBUG:boto:Params: {} 2023-07-29T00:29:56.728 DEBUG:boto:Token: None 2023-07-29T00:29:56.729 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:56 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:29:56.729 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RWiGZqvFAgAwcg2OOHNtm27WQvY= 2023-07-29T00:29:56.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RWiGZqvFAgAwcg2OOHNtm27WQvY=', 'Content-Length': '0'} 2023-07-29T00:29:56.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000431f652177d4aba1-0064c45d84-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:56.733 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:29:56.733 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:29:56.733 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:29:56.734 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:29:56.734 DEBUG:boto:Method: HEAD 2023-07-29T00:29:56.734 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:29:56.735 DEBUG:boto:Data: 2023-07-29T00:29:56.735 DEBUG:boto:Headers: {} 2023-07-29T00:29:56.735 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:56.736 DEBUG:boto:Port: 8001 2023-07-29T00:29:56.736 DEBUG:boto:Params: {} 2023-07-29T00:29:56.737 DEBUG:boto:Token: None 2023-07-29T00:29:56.737 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:56 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:29:56.737 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WqE/nYl1w6vWhxC/c7h9ZTbZmOo= 2023-07-29T00:29:56.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WqE/nYl1w6vWhxC/c7h9ZTbZmOo=', 'Content-Length': '0'} 2023-07-29T00:29:56.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5abb4fde144198d-0064c45d84-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:56.740 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:29:56.741 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:29:56.741 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:29:56.741 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:29:56.742 DEBUG:boto:Method: HEAD 2023-07-29T00:29:56.742 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:29:56.743 DEBUG:boto:Data: 2023-07-29T00:29:56.743 DEBUG:boto:Headers: {} 2023-07-29T00:29:56.743 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:56.744 DEBUG:boto:Port: 8000 2023-07-29T00:29:56.744 DEBUG:boto:Params: {} 2023-07-29T00:29:56.744 DEBUG:boto:Token: None 2023-07-29T00:29:56.745 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:56 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:29:56.745 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WqE/nYl1w6vWhxC/c7h9ZTbZmOo= 2023-07-29T00:29:56.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WqE/nYl1w6vWhxC/c7h9ZTbZmOo=', 'Content-Length': '0'} 2023-07-29T00:29:56.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb73243f7ad8bc80-0064c45d84-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:56.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-07-29T00:29:56.750 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:29:56.751 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:29:56.751 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:29:56.752 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:29:56.753 DEBUG:boto:Method: GET 2023-07-29T00:29:56.753 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:29:56.753 DEBUG:boto:Data: 2023-07-29T00:29:56.754 DEBUG:boto:Headers: {} 2023-07-29T00:29:56.754 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:56.754 DEBUG:boto:Port: 8001 2023-07-29T00:29:56.755 DEBUG:boto:Params: {} 2023-07-29T00:29:56.755 DEBUG:boto:Token: None 2023-07-29T00:29:56.756 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:56 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:29:56.756 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RWiGZqvFAgAwcg2OOHNtm27WQvY= 2023-07-29T00:29:56.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RWiGZqvFAgAwcg2OOHNtm27WQvY=', 'Content-Length': '0'} 2023-07-29T00:29:56.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000842dec17351856c5-0064c45d84-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:56.759 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:29:56.759 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:29:56.760 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:29:56.760 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:29:56.761 DEBUG:boto:Method: GET 2023-07-29T00:29:56.761 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:29:56.761 DEBUG:boto:Data: 2023-07-29T00:29:56.762 DEBUG:boto:Headers: {} 2023-07-29T00:29:56.762 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:56.762 DEBUG:boto:Port: 8000 2023-07-29T00:29:56.763 DEBUG:boto:Params: {} 2023-07-29T00:29:56.763 DEBUG:boto:Token: None 2023-07-29T00:29:56.764 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:56 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:29:56.764 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RWiGZqvFAgAwcg2OOHNtm27WQvY= 2023-07-29T00:29:56.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RWiGZqvFAgAwcg2OOHNtm27WQvY=', 'Content-Length': '0'} 2023-07-29T00:29:56.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000876f8ea53c3fbf8-0064c45d84-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:56.767 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:56.767 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:56.768 DEBUG:boto:path=/hiiqos-59/?prefix=obj11 2023-07-29T00:29:56.768 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj11 2023-07-29T00:29:56.768 DEBUG:boto:Method: GET 2023-07-29T00:29:56.769 DEBUG:boto:Path: /hiiqos-59/?prefix=obj11 2023-07-29T00:29:56.769 DEBUG:boto:Data: 2023-07-29T00:29:56.770 DEBUG:boto:Headers: {} 2023-07-29T00:29:56.770 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:56.770 DEBUG:boto:Port: 8001 2023-07-29T00:29:56.771 DEBUG:boto:Params: {} 2023-07-29T00:29:56.771 DEBUG:boto:Token: None 2023-07-29T00:29:56.771 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:56 GMT /hiiqos-59/ 2023-07-29T00:29:56.772 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jPZk/c38GI5ourfaUaDgVDO9RZs= 2023-07-29T00:29:56.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jPZk/c38GI5ourfaUaDgVDO9RZs=', 'Content-Length': '0'} 2023-07-29T00:29:56.775 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c327fb2fdf2e7e8-0064c45d84-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:56.775 DEBUG:boto:b'hiiqos-59obj111000falseobj112023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:56.776 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:56.776 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:56.777 DEBUG:boto:path=/hiiqos-59/?prefix=obj11 2023-07-29T00:29:56.778 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj11 2023-07-29T00:29:56.779 DEBUG:boto:Method: GET 2023-07-29T00:29:56.780 DEBUG:boto:Path: /hiiqos-59/?prefix=obj11 2023-07-29T00:29:56.781 DEBUG:boto:Data: 2023-07-29T00:29:56.782 DEBUG:boto:Headers: {} 2023-07-29T00:29:56.782 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:56.783 DEBUG:boto:Port: 8000 2023-07-29T00:29:56.784 DEBUG:boto:Params: {} 2023-07-29T00:29:56.785 DEBUG:boto:Token: None 2023-07-29T00:29:56.785 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:56 GMT /hiiqos-59/ 2023-07-29T00:29:56.786 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jPZk/c38GI5ourfaUaDgVDO9RZs= 2023-07-29T00:29:56.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jPZk/c38GI5ourfaUaDgVDO9RZs=', 'Content-Length': '0'} 2023-07-29T00:29:56.790 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8ecec35315f796f-0064c45d84-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:56.791 DEBUG:boto:b'hiiqos-59obj111000falseobj112023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:56.792 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:29:56.792 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:29:56.793 DEBUG:boto:Method: HEAD 2023-07-29T00:29:56.793 DEBUG:boto:Path: /hiiqos-59/obj11 2023-07-29T00:29:56.793 DEBUG:boto:Data: 2023-07-29T00:29:56.794 DEBUG:boto:Headers: {} 2023-07-29T00:29:56.794 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:56.794 DEBUG:boto:Port: 8001 2023-07-29T00:29:56.795 DEBUG:boto:Params: {} 2023-07-29T00:29:56.795 DEBUG:boto:Token: None 2023-07-29T00:29:56.795 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:56 GMT /hiiqos-59/obj11 2023-07-29T00:29:56.796 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2X+LCZPGIy3M8QbLFrNkKzp3xN4= 2023-07-29T00:29:56.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2X+LCZPGIy3M8QbLFrNkKzp3xN4=', 'Content-Length': '0'} 2023-07-29T00:29:56.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000514f09caf2de999-0064c45d84-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:56.799 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:29:56.800 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:29:56.800 DEBUG:boto:Method: HEAD 2023-07-29T00:29:56.801 DEBUG:boto:Path: /hiiqos-59/obj11 2023-07-29T00:29:56.802 DEBUG:boto:Data: 2023-07-29T00:29:56.802 DEBUG:boto:Headers: {} 2023-07-29T00:29:56.803 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:56.803 DEBUG:boto:Port: 8000 2023-07-29T00:29:56.804 DEBUG:boto:Params: {} 2023-07-29T00:29:56.805 DEBUG:boto:Token: None 2023-07-29T00:29:56.805 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:56 GMT /hiiqos-59/obj11 2023-07-29T00:29:56.806 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2X+LCZPGIy3M8QbLFrNkKzp3xN4= 2023-07-29T00:29:56.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2X+LCZPGIy3M8QbLFrNkKzp3xN4=', 'Content-Length': '0'} 2023-07-29T00:29:56.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaeb5cf4589dbf47-0064c45d84-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:56.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-07-29T00:29:56.809 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:29:56.809 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:29:56.810 DEBUG:boto:Method: GET 2023-07-29T00:29:56.810 DEBUG:boto:Path: /hiiqos-59/obj11 2023-07-29T00:29:56.810 DEBUG:boto:Data: 2023-07-29T00:29:56.811 DEBUG:boto:Headers: {} 2023-07-29T00:29:56.811 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:56.812 DEBUG:boto:Port: 8001 2023-07-29T00:29:56.812 DEBUG:boto:Params: {} 2023-07-29T00:29:56.812 DEBUG:boto:Token: None 2023-07-29T00:29:56.813 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:56 GMT /hiiqos-59/obj11 2023-07-29T00:29:56.813 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+VzOlacaOsqjp3DVpa+cpNrItWY= 2023-07-29T00:29:56.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+VzOlacaOsqjp3DVpa+cpNrItWY=', 'Content-Length': '0'} 2023-07-29T00:29:56.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d939564921e4b5f-0064c45d84-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:56.816 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:29:56.817 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:29:56.817 DEBUG:boto:Method: GET 2023-07-29T00:29:56.817 DEBUG:boto:Path: /hiiqos-59/obj11 2023-07-29T00:29:56.818 DEBUG:boto:Data: 2023-07-29T00:29:56.818 DEBUG:boto:Headers: {} 2023-07-29T00:29:56.818 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:56.819 DEBUG:boto:Port: 8000 2023-07-29T00:29:56.819 DEBUG:boto:Params: {} 2023-07-29T00:29:56.819 DEBUG:boto:Token: None 2023-07-29T00:29:56.820 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:56 GMT /hiiqos-59/obj11 2023-07-29T00:29:56.820 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+VzOlacaOsqjp3DVpa+cpNrItWY= 2023-07-29T00:29:56.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+VzOlacaOsqjp3DVpa+cpNrItWY=', 'Content-Length': '0'} 2023-07-29T00:29:56.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f342f97b1709da36-0064c45d84-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:56.864 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:56.865 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:56.866 DEBUG:boto:path=/hiiqos-59/?prefix=obj11 2023-07-29T00:29:56.866 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj11 2023-07-29T00:29:56.866 DEBUG:boto:Method: GET 2023-07-29T00:29:56.867 DEBUG:boto:Path: /hiiqos-59/?prefix=obj11 2023-07-29T00:29:56.867 DEBUG:boto:Data: 2023-07-29T00:29:56.867 DEBUG:boto:Headers: {} 2023-07-29T00:29:56.868 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:56.868 DEBUG:boto:Port: 8001 2023-07-29T00:29:56.869 DEBUG:boto:Params: {} 2023-07-29T00:29:56.869 DEBUG:boto:Token: None 2023-07-29T00:29:56.870 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:56 GMT /hiiqos-59/ 2023-07-29T00:29:56.870 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jPZk/c38GI5ourfaUaDgVDO9RZs= 2023-07-29T00:29:56.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jPZk/c38GI5ourfaUaDgVDO9RZs=', 'Content-Length': '0'} 2023-07-29T00:29:56.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027bf0036ba529f51-0064c45d84-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:56.873 DEBUG:boto:b'hiiqos-59obj111000falseobj112023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:56.874 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:56.875 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:56.875 DEBUG:boto:path=/hiiqos-59/?prefix=obj11 2023-07-29T00:29:56.876 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj11 2023-07-29T00:29:56.876 DEBUG:boto:Method: GET 2023-07-29T00:29:56.877 DEBUG:boto:Path: /hiiqos-59/?prefix=obj11 2023-07-29T00:29:56.878 DEBUG:boto:Data: 2023-07-29T00:29:56.878 DEBUG:boto:Headers: {} 2023-07-29T00:29:56.879 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:56.880 DEBUG:boto:Port: 8000 2023-07-29T00:29:56.880 DEBUG:boto:Params: {} 2023-07-29T00:29:56.881 DEBUG:boto:Token: None 2023-07-29T00:29:56.881 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:56 GMT /hiiqos-59/ 2023-07-29T00:29:56.882 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jPZk/c38GI5ourfaUaDgVDO9RZs= 2023-07-29T00:29:56.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jPZk/c38GI5ourfaUaDgVDO9RZs=', 'Content-Length': '0'} 2023-07-29T00:29:56.885 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f004dbfdde07d6b-0064c45d84-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:56.886 DEBUG:boto:b'hiiqos-59obj111000falseobj112023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:56.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-07-29T00:29:56.887 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:29:56.887 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:29:56.888 DEBUG:boto:path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:29:56.888 DEBUG:boto:auth_path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:29:56.888 DEBUG:boto:Method: GET 2023-07-29T00:29:56.889 DEBUG:boto:Path: /hiiqos-59/obj17?versionId=null 2023-07-29T00:29:56.889 DEBUG:boto:Data: 2023-07-29T00:29:56.890 DEBUG:boto:Headers: {} 2023-07-29T00:29:56.890 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:56.890 DEBUG:boto:Port: 8001 2023-07-29T00:29:56.891 DEBUG:boto:Params: {} 2023-07-29T00:29:56.891 DEBUG:boto:Token: None 2023-07-29T00:29:56.892 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:56 GMT /hiiqos-59/obj17?versionId=null 2023-07-29T00:29:56.892 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0xYgBjSInkzb1ShbDXGkNBJeGCI= 2023-07-29T00:29:56.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0xYgBjSInkzb1ShbDXGkNBJeGCI=', 'Content-Length': '0'} 2023-07-29T00:29:56.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf0222a76e5d1fbe-0064c45d84-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:56.895 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:29:56.896 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:29:56.896 DEBUG:boto:path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:29:56.897 DEBUG:boto:auth_path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:29:56.897 DEBUG:boto:Method: GET 2023-07-29T00:29:56.897 DEBUG:boto:Path: /hiiqos-59/obj17?versionId=null 2023-07-29T00:29:56.898 DEBUG:boto:Data: 2023-07-29T00:29:56.898 DEBUG:boto:Headers: {} 2023-07-29T00:29:56.898 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:56.899 DEBUG:boto:Port: 8000 2023-07-29T00:29:56.899 DEBUG:boto:Params: {} 2023-07-29T00:29:56.899 DEBUG:boto:Token: None 2023-07-29T00:29:56.900 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:56 GMT /hiiqos-59/obj17?versionId=null 2023-07-29T00:29:56.900 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0xYgBjSInkzb1ShbDXGkNBJeGCI= 2023-07-29T00:29:56.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0xYgBjSInkzb1ShbDXGkNBJeGCI=', 'Content-Length': '0'} 2023-07-29T00:29:56.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b3093553770bb59-0064c45d84-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:56.903 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:29:56.904 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:29:56.904 DEBUG:boto:path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:29:56.904 DEBUG:boto:auth_path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:29:56.905 DEBUG:boto:Method: HEAD 2023-07-29T00:29:56.905 DEBUG:boto:Path: /hiiqos-59/obj17?versionId=null 2023-07-29T00:29:56.905 DEBUG:boto:Data: 2023-07-29T00:29:56.906 DEBUG:boto:Headers: {} 2023-07-29T00:29:56.906 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:56.907 DEBUG:boto:Port: 8001 2023-07-29T00:29:56.907 DEBUG:boto:Params: {} 2023-07-29T00:29:56.907 DEBUG:boto:Token: None 2023-07-29T00:29:56.908 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:56 GMT /hiiqos-59/obj17?versionId=null 2023-07-29T00:29:56.908 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Zv3GYnioww9kx9ej+ZCjV4n5K4Q= 2023-07-29T00:29:56.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Zv3GYnioww9kx9ej+ZCjV4n5K4Q=', 'Content-Length': '0'} 2023-07-29T00:29:56.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b797ddb5c85d021e-0064c45d84-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:56.911 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:29:56.912 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:29:56.913 DEBUG:boto:path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:29:56.913 DEBUG:boto:auth_path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:29:56.914 DEBUG:boto:Method: HEAD 2023-07-29T00:29:56.914 DEBUG:boto:Path: /hiiqos-59/obj17?versionId=null 2023-07-29T00:29:56.914 DEBUG:boto:Data: 2023-07-29T00:29:56.915 DEBUG:boto:Headers: {} 2023-07-29T00:29:56.915 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:56.915 DEBUG:boto:Port: 8000 2023-07-29T00:29:56.916 DEBUG:boto:Params: {} 2023-07-29T00:29:56.916 DEBUG:boto:Token: None 2023-07-29T00:29:56.917 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:56 GMT /hiiqos-59/obj17?versionId=null 2023-07-29T00:29:56.917 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Zv3GYnioww9kx9ej+ZCjV4n5K4Q= 2023-07-29T00:29:56.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Zv3GYnioww9kx9ej+ZCjV4n5K4Q=', 'Content-Length': '0'} 2023-07-29T00:29:56.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000789fcd26242dc49a-0064c45d84-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:56.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-07-29T00:29:56.920 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:29:56.921 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:29:56.921 DEBUG:boto:path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:29:56.922 DEBUG:boto:auth_path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:29:56.922 DEBUG:boto:Method: GET 2023-07-29T00:29:56.922 DEBUG:boto:Path: /hiiqos-59/obj17?versionId=null 2023-07-29T00:29:56.923 DEBUG:boto:Data: 2023-07-29T00:29:56.923 DEBUG:boto:Headers: {} 2023-07-29T00:29:56.923 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:56.924 DEBUG:boto:Port: 8001 2023-07-29T00:29:56.924 DEBUG:boto:Params: {} 2023-07-29T00:29:56.925 DEBUG:boto:Token: None 2023-07-29T00:29:56.925 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:56 GMT /hiiqos-59/obj17?versionId=null 2023-07-29T00:29:56.925 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0xYgBjSInkzb1ShbDXGkNBJeGCI= 2023-07-29T00:29:56.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0xYgBjSInkzb1ShbDXGkNBJeGCI=', 'Content-Length': '0'} 2023-07-29T00:29:56.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ca51c136519cfd2-0064c45d84-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:56.929 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:29:56.929 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:29:56.930 DEBUG:boto:path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:29:56.931 DEBUG:boto:auth_path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:29:56.931 DEBUG:boto:Method: GET 2023-07-29T00:29:56.932 DEBUG:boto:Path: /hiiqos-59/obj17?versionId=null 2023-07-29T00:29:56.932 DEBUG:boto:Data: 2023-07-29T00:29:56.932 DEBUG:boto:Headers: {} 2023-07-29T00:29:56.933 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:56.933 DEBUG:boto:Port: 8000 2023-07-29T00:29:56.934 DEBUG:boto:Params: {} 2023-07-29T00:29:56.934 DEBUG:boto:Token: None 2023-07-29T00:29:56.934 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:56 GMT /hiiqos-59/obj17?versionId=null 2023-07-29T00:29:56.935 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0xYgBjSInkzb1ShbDXGkNBJeGCI= 2023-07-29T00:29:56.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0xYgBjSInkzb1ShbDXGkNBJeGCI=', 'Content-Length': '0'} 2023-07-29T00:29:56.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018f441cbb446c437-0064c45d84-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:56.938 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:56.938 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:56.938 DEBUG:boto:path=/hiiqos-59/?prefix=obj17 2023-07-29T00:29:56.939 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj17 2023-07-29T00:29:56.939 DEBUG:boto:Method: GET 2023-07-29T00:29:56.940 DEBUG:boto:Path: /hiiqos-59/?prefix=obj17 2023-07-29T00:29:56.940 DEBUG:boto:Data: 2023-07-29T00:29:56.940 DEBUG:boto:Headers: {} 2023-07-29T00:29:56.941 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:56.941 DEBUG:boto:Port: 8001 2023-07-29T00:29:56.942 DEBUG:boto:Params: {} 2023-07-29T00:29:56.942 DEBUG:boto:Token: None 2023-07-29T00:29:56.943 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:56 GMT /hiiqos-59/ 2023-07-29T00:29:56.943 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jPZk/c38GI5ourfaUaDgVDO9RZs= 2023-07-29T00:29:56.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jPZk/c38GI5ourfaUaDgVDO9RZs=', 'Content-Length': '0'} 2023-07-29T00:29:56.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb0ccc4a9e414254-0064c45d84-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:56.947 DEBUG:boto:b'hiiqos-59obj171000falseobj172023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:56.947 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:56.948 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:56.948 DEBUG:boto:path=/hiiqos-59/?prefix=obj17 2023-07-29T00:29:56.949 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj17 2023-07-29T00:29:56.949 DEBUG:boto:Method: GET 2023-07-29T00:29:56.949 DEBUG:boto:Path: /hiiqos-59/?prefix=obj17 2023-07-29T00:29:56.950 DEBUG:boto:Data: 2023-07-29T00:29:56.950 DEBUG:boto:Headers: {} 2023-07-29T00:29:56.950 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:56.951 DEBUG:boto:Port: 8000 2023-07-29T00:29:56.951 DEBUG:boto:Params: {} 2023-07-29T00:29:56.951 DEBUG:boto:Token: None 2023-07-29T00:29:56.952 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:56 GMT /hiiqos-59/ 2023-07-29T00:29:56.952 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jPZk/c38GI5ourfaUaDgVDO9RZs= 2023-07-29T00:29:56.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:56 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jPZk/c38GI5ourfaUaDgVDO9RZs=', 'Content-Length': '0'} 2023-07-29T00:29:56.956 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e87e0523b132b71-0064c45d84-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:56.996 DEBUG:boto:b'hiiqos-59obj171000falseobj172023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:56.997 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:29:56.998 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:29:56.998 DEBUG:boto:Method: HEAD 2023-07-29T00:29:56.998 DEBUG:boto:Path: /hiiqos-59/obj17 2023-07-29T00:29:56.999 DEBUG:boto:Data: 2023-07-29T00:29:56.999 DEBUG:boto:Headers: {} 2023-07-29T00:29:56.999 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:57.000 DEBUG:boto:Port: 8001 2023-07-29T00:29:57.000 DEBUG:boto:Params: {} 2023-07-29T00:29:57.001 DEBUG:boto:Token: None 2023-07-29T00:29:57.001 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj17 2023-07-29T00:29:57.001 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7sbOaxgu2TEUNJov3ZQj6hCHE1E= 2023-07-29T00:29:57.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7sbOaxgu2TEUNJov3ZQj6hCHE1E=', 'Content-Length': '0'} 2023-07-29T00:29:57.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000703e9125f65e225c-0064c45d84-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.004 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:29:57.005 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:29:57.005 DEBUG:boto:Method: HEAD 2023-07-29T00:29:57.005 DEBUG:boto:Path: /hiiqos-59/obj17 2023-07-29T00:29:57.006 DEBUG:boto:Data: 2023-07-29T00:29:57.006 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.006 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:57.007 DEBUG:boto:Port: 8000 2023-07-29T00:29:57.007 DEBUG:boto:Params: {} 2023-07-29T00:29:57.007 DEBUG:boto:Token: None 2023-07-29T00:29:57.008 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj17 2023-07-29T00:29:57.008 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7sbOaxgu2TEUNJov3ZQj6hCHE1E= 2023-07-29T00:29:57.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7sbOaxgu2TEUNJov3ZQj6hCHE1E=', 'Content-Length': '0'} 2023-07-29T00:29:57.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b4d078c0f22992f-0064c45d84-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-07-29T00:29:57.012 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:29:57.013 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:29:57.014 DEBUG:boto:Method: GET 2023-07-29T00:29:57.014 DEBUG:boto:Path: /hiiqos-59/obj17 2023-07-29T00:29:57.015 DEBUG:boto:Data: 2023-07-29T00:29:57.015 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.015 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:57.016 DEBUG:boto:Port: 8001 2023-07-29T00:29:57.016 DEBUG:boto:Params: {} 2023-07-29T00:29:57.017 DEBUG:boto:Token: None 2023-07-29T00:29:57.017 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj17 2023-07-29T00:29:57.018 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w43PYgQz0N/l62UvEL47YoveQyo= 2023-07-29T00:29:57.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w43PYgQz0N/l62UvEL47YoveQyo=', 'Content-Length': '0'} 2023-07-29T00:29:57.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8061d1452a8af60-0064c45d84-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.021 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:29:57.021 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:29:57.021 DEBUG:boto:Method: GET 2023-07-29T00:29:57.022 DEBUG:boto:Path: /hiiqos-59/obj17 2023-07-29T00:29:57.022 DEBUG:boto:Data: 2023-07-29T00:29:57.022 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.023 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:57.023 DEBUG:boto:Port: 8000 2023-07-29T00:29:57.024 DEBUG:boto:Params: {} 2023-07-29T00:29:57.024 DEBUG:boto:Token: None 2023-07-29T00:29:57.024 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj17 2023-07-29T00:29:57.025 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w43PYgQz0N/l62UvEL47YoveQyo= 2023-07-29T00:29:57.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w43PYgQz0N/l62UvEL47YoveQyo=', 'Content-Length': '0'} 2023-07-29T00:29:57.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cecd6f72e76d74db-0064c45d84-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.028 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:57.029 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:57.030 DEBUG:boto:path=/hiiqos-59/?prefix=obj17 2023-07-29T00:29:57.031 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj17 2023-07-29T00:29:57.032 DEBUG:boto:Method: GET 2023-07-29T00:29:57.032 DEBUG:boto:Path: /hiiqos-59/?prefix=obj17 2023-07-29T00:29:57.033 DEBUG:boto:Data: 2023-07-29T00:29:57.034 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.035 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:57.035 DEBUG:boto:Port: 8001 2023-07-29T00:29:57.035 DEBUG:boto:Params: {} 2023-07-29T00:29:57.036 DEBUG:boto:Token: None 2023-07-29T00:29:57.036 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/ 2023-07-29T00:29:57.037 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0= 2023-07-29T00:29:57.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0=', 'Content-Length': '0'} 2023-07-29T00:29:57.040 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf096ef40c840152-0064c45d84-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.040 DEBUG:boto:b'hiiqos-59obj171000falseobj172023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:57.041 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:57.041 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:57.042 DEBUG:boto:path=/hiiqos-59/?prefix=obj17 2023-07-29T00:29:57.042 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj17 2023-07-29T00:29:57.042 DEBUG:boto:Method: GET 2023-07-29T00:29:57.043 DEBUG:boto:Path: /hiiqos-59/?prefix=obj17 2023-07-29T00:29:57.044 DEBUG:boto:Data: 2023-07-29T00:29:57.044 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.044 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:57.045 DEBUG:boto:Port: 8000 2023-07-29T00:29:57.045 DEBUG:boto:Params: {} 2023-07-29T00:29:57.045 DEBUG:boto:Token: None 2023-07-29T00:29:57.046 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/ 2023-07-29T00:29:57.046 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0= 2023-07-29T00:29:57.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0=', 'Content-Length': '0'} 2023-07-29T00:29:57.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032eddfbd72415cd0-0064c45d84-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:56 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.050 DEBUG:boto:b'hiiqos-59obj171000falseobj172023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:57.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-07-29T00:29:57.051 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:29:57.052 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:29:57.052 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:29:57.053 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:29:57.053 DEBUG:boto:Method: GET 2023-07-29T00:29:57.053 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:29:57.054 DEBUG:boto:Data: 2023-07-29T00:29:57.054 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.054 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:57.055 DEBUG:boto:Port: 8001 2023-07-29T00:29:57.055 DEBUG:boto:Params: {} 2023-07-29T00:29:57.056 DEBUG:boto:Token: None 2023-07-29T00:29:57.056 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:29:57.057 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rgqWFSZo4vdHGEJJlCNx4IPGcmc= 2023-07-29T00:29:57.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rgqWFSZo4vdHGEJJlCNx4IPGcmc=', 'Content-Length': '0'} 2023-07-29T00:29:57.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052fc7d7f31359f6d-0064c45d85-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.060 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:29:57.060 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:29:57.061 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:29:57.061 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:29:57.062 DEBUG:boto:Method: GET 2023-07-29T00:29:57.062 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:29:57.063 DEBUG:boto:Data: 2023-07-29T00:29:57.063 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.063 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:57.064 DEBUG:boto:Port: 8000 2023-07-29T00:29:57.064 DEBUG:boto:Params: {} 2023-07-29T00:29:57.065 DEBUG:boto:Token: None 2023-07-29T00:29:57.065 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:29:57.065 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rgqWFSZo4vdHGEJJlCNx4IPGcmc= 2023-07-29T00:29:57.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rgqWFSZo4vdHGEJJlCNx4IPGcmc=', 'Content-Length': '0'} 2023-07-29T00:29:57.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0566516eeb763d6-0064c45d85-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.069 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:29:57.069 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:29:57.069 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:29:57.070 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:29:57.070 DEBUG:boto:Method: HEAD 2023-07-29T00:29:57.070 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:29:57.071 DEBUG:boto:Data: 2023-07-29T00:29:57.071 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.071 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:57.072 DEBUG:boto:Port: 8001 2023-07-29T00:29:57.072 DEBUG:boto:Params: {} 2023-07-29T00:29:57.073 DEBUG:boto:Token: None 2023-07-29T00:29:57.073 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:29:57.073 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uYb5oMMkPhNKb/fJ2RCsyql0viU= 2023-07-29T00:29:57.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uYb5oMMkPhNKb/fJ2RCsyql0viU=', 'Content-Length': '0'} 2023-07-29T00:29:57.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008230409b53044ee2-0064c45d85-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.076 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:29:57.076 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:29:57.077 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:29:57.077 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:29:57.077 DEBUG:boto:Method: HEAD 2023-07-29T00:29:57.078 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:29:57.078 DEBUG:boto:Data: 2023-07-29T00:29:57.078 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.079 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:57.079 DEBUG:boto:Port: 8000 2023-07-29T00:29:57.079 DEBUG:boto:Params: {} 2023-07-29T00:29:57.080 DEBUG:boto:Token: None 2023-07-29T00:29:57.080 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:29:57.081 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uYb5oMMkPhNKb/fJ2RCsyql0viU= 2023-07-29T00:29:57.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uYb5oMMkPhNKb/fJ2RCsyql0viU=', 'Content-Length': '0'} 2023-07-29T00:29:57.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cfa55af7a37ffca-0064c45d85-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-07-29T00:29:57.084 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:29:57.085 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:29:57.085 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:29:57.086 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:29:57.087 DEBUG:boto:Method: GET 2023-07-29T00:29:57.088 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:29:57.088 DEBUG:boto:Data: 2023-07-29T00:29:57.089 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.090 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:57.091 DEBUG:boto:Port: 8001 2023-07-29T00:29:57.092 DEBUG:boto:Params: {} 2023-07-29T00:29:57.092 DEBUG:boto:Token: None 2023-07-29T00:29:57.093 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:29:57.093 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rgqWFSZo4vdHGEJJlCNx4IPGcmc= 2023-07-29T00:29:57.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rgqWFSZo4vdHGEJJlCNx4IPGcmc=', 'Content-Length': '0'} 2023-07-29T00:29:57.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c8959652873372c-0064c45d85-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.096 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:29:57.097 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:29:57.097 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:29:57.098 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:29:57.098 DEBUG:boto:Method: GET 2023-07-29T00:29:57.098 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:29:57.099 DEBUG:boto:Data: 2023-07-29T00:29:57.099 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.100 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:57.100 DEBUG:boto:Port: 8000 2023-07-29T00:29:57.100 DEBUG:boto:Params: {} 2023-07-29T00:29:57.101 DEBUG:boto:Token: None 2023-07-29T00:29:57.101 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:29:57.102 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rgqWFSZo4vdHGEJJlCNx4IPGcmc= 2023-07-29T00:29:57.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rgqWFSZo4vdHGEJJlCNx4IPGcmc=', 'Content-Length': '0'} 2023-07-29T00:29:57.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c7cfa92105b7fd9-0064c45d85-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.105 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:57.105 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:57.106 DEBUG:boto:path=/hiiqos-59/?prefix=obj22 2023-07-29T00:29:57.106 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj22 2023-07-29T00:29:57.107 DEBUG:boto:Method: GET 2023-07-29T00:29:57.107 DEBUG:boto:Path: /hiiqos-59/?prefix=obj22 2023-07-29T00:29:57.107 DEBUG:boto:Data: 2023-07-29T00:29:57.108 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.108 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:57.108 DEBUG:boto:Port: 8001 2023-07-29T00:29:57.109 DEBUG:boto:Params: {} 2023-07-29T00:29:57.109 DEBUG:boto:Token: None 2023-07-29T00:29:57.110 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/ 2023-07-29T00:29:57.110 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0= 2023-07-29T00:29:57.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0=', 'Content-Length': '0'} 2023-07-29T00:29:57.113 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0565268a523145d-0064c45d85-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.114 DEBUG:boto:b'hiiqos-59obj221000falseobj222023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:57.114 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:57.115 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:57.115 DEBUG:boto:path=/hiiqos-59/?prefix=obj22 2023-07-29T00:29:57.115 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj22 2023-07-29T00:29:57.116 DEBUG:boto:Method: GET 2023-07-29T00:29:57.116 DEBUG:boto:Path: /hiiqos-59/?prefix=obj22 2023-07-29T00:29:57.117 DEBUG:boto:Data: 2023-07-29T00:29:57.117 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.117 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:57.118 DEBUG:boto:Port: 8000 2023-07-29T00:29:57.119 DEBUG:boto:Params: {} 2023-07-29T00:29:57.120 DEBUG:boto:Token: None 2023-07-29T00:29:57.121 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/ 2023-07-29T00:29:57.122 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0= 2023-07-29T00:29:57.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0=', 'Content-Length': '0'} 2023-07-29T00:29:57.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abb6cef82ec37902-0064c45d85-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.168 DEBUG:boto:b'hiiqos-59obj221000falseobj222023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:57.169 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:29:57.169 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:29:57.170 DEBUG:boto:Method: HEAD 2023-07-29T00:29:57.170 DEBUG:boto:Path: /hiiqos-59/obj22 2023-07-29T00:29:57.171 DEBUG:boto:Data: 2023-07-29T00:29:57.171 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.171 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:57.172 DEBUG:boto:Port: 8001 2023-07-29T00:29:57.172 DEBUG:boto:Params: {} 2023-07-29T00:29:57.172 DEBUG:boto:Token: None 2023-07-29T00:29:57.173 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj22 2023-07-29T00:29:57.173 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2JZGIg5kpP9DunTZ19o5vSN4LvA= 2023-07-29T00:29:57.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2JZGIg5kpP9DunTZ19o5vSN4LvA=', 'Content-Length': '0'} 2023-07-29T00:29:57.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc894a4f80b99c2e-0064c45d85-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.176 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:29:57.177 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:29:57.177 DEBUG:boto:Method: HEAD 2023-07-29T00:29:57.177 DEBUG:boto:Path: /hiiqos-59/obj22 2023-07-29T00:29:57.178 DEBUG:boto:Data: 2023-07-29T00:29:57.178 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.178 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:57.179 DEBUG:boto:Port: 8000 2023-07-29T00:29:57.179 DEBUG:boto:Params: {} 2023-07-29T00:29:57.179 DEBUG:boto:Token: None 2023-07-29T00:29:57.180 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj22 2023-07-29T00:29:57.180 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2JZGIg5kpP9DunTZ19o5vSN4LvA= 2023-07-29T00:29:57.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2JZGIg5kpP9DunTZ19o5vSN4LvA=', 'Content-Length': '0'} 2023-07-29T00:29:57.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1bba1eba41bb831-0064c45d85-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-07-29T00:29:57.183 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:29:57.184 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:29:57.184 DEBUG:boto:Method: GET 2023-07-29T00:29:57.185 DEBUG:boto:Path: /hiiqos-59/obj22 2023-07-29T00:29:57.185 DEBUG:boto:Data: 2023-07-29T00:29:57.185 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.186 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:57.186 DEBUG:boto:Port: 8001 2023-07-29T00:29:57.187 DEBUG:boto:Params: {} 2023-07-29T00:29:57.187 DEBUG:boto:Token: None 2023-07-29T00:29:57.187 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj22 2023-07-29T00:29:57.188 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:iLNbkceGhKvVtETtsrqFmxV69PQ= 2023-07-29T00:29:57.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:iLNbkceGhKvVtETtsrqFmxV69PQ=', 'Content-Length': '0'} 2023-07-29T00:29:57.190 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c3b5c63c971a894-0064c45d85-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.191 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:29:57.191 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:29:57.192 DEBUG:boto:Method: GET 2023-07-29T00:29:57.192 DEBUG:boto:Path: /hiiqos-59/obj22 2023-07-29T00:29:57.192 DEBUG:boto:Data: 2023-07-29T00:29:57.193 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.193 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:57.194 DEBUG:boto:Port: 8000 2023-07-29T00:29:57.194 DEBUG:boto:Params: {} 2023-07-29T00:29:57.194 DEBUG:boto:Token: None 2023-07-29T00:29:57.195 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj22 2023-07-29T00:29:57.195 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:iLNbkceGhKvVtETtsrqFmxV69PQ= 2023-07-29T00:29:57.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:iLNbkceGhKvVtETtsrqFmxV69PQ=', 'Content-Length': '0'} 2023-07-29T00:29:57.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071a198f009fac22c-0064c45d85-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.199 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:57.199 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:57.200 DEBUG:boto:path=/hiiqos-59/?prefix=obj22 2023-07-29T00:29:57.200 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj22 2023-07-29T00:29:57.200 DEBUG:boto:Method: GET 2023-07-29T00:29:57.201 DEBUG:boto:Path: /hiiqos-59/?prefix=obj22 2023-07-29T00:29:57.201 DEBUG:boto:Data: 2023-07-29T00:29:57.202 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.202 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:57.202 DEBUG:boto:Port: 8001 2023-07-29T00:29:57.203 DEBUG:boto:Params: {} 2023-07-29T00:29:57.203 DEBUG:boto:Token: None 2023-07-29T00:29:57.204 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/ 2023-07-29T00:29:57.204 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0= 2023-07-29T00:29:57.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0=', 'Content-Length': '0'} 2023-07-29T00:29:57.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008985b961ae86d349-0064c45d85-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.208 DEBUG:boto:b'hiiqos-59obj221000falseobj222023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:57.209 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:57.209 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:57.209 DEBUG:boto:path=/hiiqos-59/?prefix=obj22 2023-07-29T00:29:57.210 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj22 2023-07-29T00:29:57.210 DEBUG:boto:Method: GET 2023-07-29T00:29:57.210 DEBUG:boto:Path: /hiiqos-59/?prefix=obj22 2023-07-29T00:29:57.211 DEBUG:boto:Data: 2023-07-29T00:29:57.211 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.212 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:57.212 DEBUG:boto:Port: 8000 2023-07-29T00:29:57.212 DEBUG:boto:Params: {} 2023-07-29T00:29:57.213 DEBUG:boto:Token: None 2023-07-29T00:29:57.213 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/ 2023-07-29T00:29:57.214 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0= 2023-07-29T00:29:57.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0=', 'Content-Length': '0'} 2023-07-29T00:29:57.217 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f40a1b81b7297ba-0064c45d85-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.217 DEBUG:boto:b'hiiqos-59obj221000falseobj222023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:57.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-07-29T00:29:57.218 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:29:57.219 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:29:57.219 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:29:57.219 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:29:57.220 DEBUG:boto:Method: GET 2023-07-29T00:29:57.220 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:29:57.221 DEBUG:boto:Data: 2023-07-29T00:29:57.221 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.221 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:57.222 DEBUG:boto:Port: 8001 2023-07-29T00:29:57.222 DEBUG:boto:Params: {} 2023-07-29T00:29:57.222 DEBUG:boto:Token: None 2023-07-29T00:29:57.223 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:29:57.223 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5W1V6OC6YYvsV4KZkCLu6eu9R5c= 2023-07-29T00:29:57.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5W1V6OC6YYvsV4KZkCLu6eu9R5c=', 'Content-Length': '0'} 2023-07-29T00:29:57.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000feceb123ec6877e8-0064c45d85-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.226 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:29:57.227 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:29:57.227 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:29:57.228 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:29:57.228 DEBUG:boto:Method: GET 2023-07-29T00:29:57.229 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:29:57.229 DEBUG:boto:Data: 2023-07-29T00:29:57.229 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.230 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:57.230 DEBUG:boto:Port: 8000 2023-07-29T00:29:57.230 DEBUG:boto:Params: {} 2023-07-29T00:29:57.231 DEBUG:boto:Token: None 2023-07-29T00:29:57.231 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:29:57.232 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5W1V6OC6YYvsV4KZkCLu6eu9R5c= 2023-07-29T00:29:57.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5W1V6OC6YYvsV4KZkCLu6eu9R5c=', 'Content-Length': '0'} 2023-07-29T00:29:57.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca2145dd7461ea59-0064c45d85-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.235 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:29:57.235 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:29:57.236 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:29:57.236 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:29:57.236 DEBUG:boto:Method: HEAD 2023-07-29T00:29:57.237 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:29:57.237 DEBUG:boto:Data: 2023-07-29T00:29:57.238 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.238 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:57.238 DEBUG:boto:Port: 8001 2023-07-29T00:29:57.239 DEBUG:boto:Params: {} 2023-07-29T00:29:57.239 DEBUG:boto:Token: None 2023-07-29T00:29:57.240 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:29:57.240 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oP3cQk/ftP4DL1ejXcQq0q9Kd+U= 2023-07-29T00:29:57.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oP3cQk/ftP4DL1ejXcQq0q9Kd+U=', 'Content-Length': '0'} 2023-07-29T00:29:57.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d69224664ee265b6-0064c45d85-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.243 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:29:57.243 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:29:57.243 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:29:57.244 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:29:57.244 DEBUG:boto:Method: HEAD 2023-07-29T00:29:57.244 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:29:57.245 DEBUG:boto:Data: 2023-07-29T00:29:57.245 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.245 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:57.246 DEBUG:boto:Port: 8000 2023-07-29T00:29:57.246 DEBUG:boto:Params: {} 2023-07-29T00:29:57.246 DEBUG:boto:Token: None 2023-07-29T00:29:57.247 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:29:57.247 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oP3cQk/ftP4DL1ejXcQq0q9Kd+U= 2023-07-29T00:29:57.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oP3cQk/ftP4DL1ejXcQq0q9Kd+U=', 'Content-Length': '0'} 2023-07-29T00:29:57.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd5aa7969ff98611-0064c45d85-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-07-29T00:29:57.251 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:29:57.251 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:29:57.251 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:29:57.252 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:29:57.252 DEBUG:boto:Method: GET 2023-07-29T00:29:57.252 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:29:57.253 DEBUG:boto:Data: 2023-07-29T00:29:57.253 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.254 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:57.254 DEBUG:boto:Port: 8001 2023-07-29T00:29:57.254 DEBUG:boto:Params: {} 2023-07-29T00:29:57.255 DEBUG:boto:Token: None 2023-07-29T00:29:57.255 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:29:57.255 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5W1V6OC6YYvsV4KZkCLu6eu9R5c= 2023-07-29T00:29:57.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5W1V6OC6YYvsV4KZkCLu6eu9R5c=', 'Content-Length': '0'} 2023-07-29T00:29:57.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003caab49c63c50c43-0064c45d85-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.258 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:29:57.259 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:29:57.259 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:29:57.260 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:29:57.260 DEBUG:boto:Method: GET 2023-07-29T00:29:57.260 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:29:57.261 DEBUG:boto:Data: 2023-07-29T00:29:57.261 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.261 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:57.262 DEBUG:boto:Port: 8000 2023-07-29T00:29:57.262 DEBUG:boto:Params: {} 2023-07-29T00:29:57.263 DEBUG:boto:Token: None 2023-07-29T00:29:57.263 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:29:57.263 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5W1V6OC6YYvsV4KZkCLu6eu9R5c= 2023-07-29T00:29:57.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5W1V6OC6YYvsV4KZkCLu6eu9R5c=', 'Content-Length': '0'} 2023-07-29T00:29:57.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000340546e2bccd5f46-0064c45d85-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.266 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:57.267 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:57.267 DEBUG:boto:path=/hiiqos-59/?prefix=obj33 2023-07-29T00:29:57.267 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj33 2023-07-29T00:29:57.268 DEBUG:boto:Method: GET 2023-07-29T00:29:57.268 DEBUG:boto:Path: /hiiqos-59/?prefix=obj33 2023-07-29T00:29:57.269 DEBUG:boto:Data: 2023-07-29T00:29:57.269 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.269 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:57.270 DEBUG:boto:Port: 8001 2023-07-29T00:29:57.270 DEBUG:boto:Params: {} 2023-07-29T00:29:57.271 DEBUG:boto:Token: None 2023-07-29T00:29:57.271 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/ 2023-07-29T00:29:57.271 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0= 2023-07-29T00:29:57.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0=', 'Content-Length': '0'} 2023-07-29T00:29:57.274 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052449f15653f57b2-0064c45d85-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.275 DEBUG:boto:b'hiiqos-59obj331000falseobj332023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:57.276 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:57.276 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:57.276 DEBUG:boto:path=/hiiqos-59/?prefix=obj33 2023-07-29T00:29:57.277 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj33 2023-07-29T00:29:57.277 DEBUG:boto:Method: GET 2023-07-29T00:29:57.277 DEBUG:boto:Path: /hiiqos-59/?prefix=obj33 2023-07-29T00:29:57.278 DEBUG:boto:Data: 2023-07-29T00:29:57.278 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.279 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:57.279 DEBUG:boto:Port: 8000 2023-07-29T00:29:57.279 DEBUG:boto:Params: {} 2023-07-29T00:29:57.280 DEBUG:boto:Token: None 2023-07-29T00:29:57.280 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/ 2023-07-29T00:29:57.281 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0= 2023-07-29T00:29:57.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0=', 'Content-Length': '0'} 2023-07-29T00:29:57.284 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df73da9ae05df77f-0064c45d85-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.284 DEBUG:boto:b'hiiqos-59obj331000falseobj332023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:57.285 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:29:57.285 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:29:57.286 DEBUG:boto:Method: HEAD 2023-07-29T00:29:57.286 DEBUG:boto:Path: /hiiqos-59/obj33 2023-07-29T00:29:57.286 DEBUG:boto:Data: 2023-07-29T00:29:57.287 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.287 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:57.287 DEBUG:boto:Port: 8001 2023-07-29T00:29:57.288 DEBUG:boto:Params: {} 2023-07-29T00:29:57.288 DEBUG:boto:Token: None 2023-07-29T00:29:57.289 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj33 2023-07-29T00:29:57.289 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:NU+omF0HMcQ+OMPLlyZDi85sgL4= 2023-07-29T00:29:57.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:NU+omF0HMcQ+OMPLlyZDi85sgL4=', 'Content-Length': '0'} 2023-07-29T00:29:57.291 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057f77c669233c9f4-0064c45d85-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.292 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:29:57.292 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:29:57.293 DEBUG:boto:Method: HEAD 2023-07-29T00:29:57.293 DEBUG:boto:Path: /hiiqos-59/obj33 2023-07-29T00:29:57.293 DEBUG:boto:Data: 2023-07-29T00:29:57.294 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.294 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:57.294 DEBUG:boto:Port: 8000 2023-07-29T00:29:57.295 DEBUG:boto:Params: {} 2023-07-29T00:29:57.295 DEBUG:boto:Token: None 2023-07-29T00:29:57.295 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj33 2023-07-29T00:29:57.296 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:NU+omF0HMcQ+OMPLlyZDi85sgL4= 2023-07-29T00:29:57.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:NU+omF0HMcQ+OMPLlyZDi85sgL4=', 'Content-Length': '0'} 2023-07-29T00:29:57.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000889fbff13d682658-0064c45d85-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-07-29T00:29:57.299 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:29:57.300 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:29:57.300 DEBUG:boto:Method: GET 2023-07-29T00:29:57.301 DEBUG:boto:Path: /hiiqos-59/obj33 2023-07-29T00:29:57.301 DEBUG:boto:Data: 2023-07-29T00:29:57.301 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.302 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:57.302 DEBUG:boto:Port: 8001 2023-07-29T00:29:57.302 DEBUG:boto:Params: {} 2023-07-29T00:29:57.303 DEBUG:boto:Token: None 2023-07-29T00:29:57.303 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj33 2023-07-29T00:29:57.304 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:unqqtKeSOQaF6u4nutihCYw8a8I= 2023-07-29T00:29:57.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:unqqtKeSOQaF6u4nutihCYw8a8I=', 'Content-Length': '0'} 2023-07-29T00:29:57.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a0f5a3be39d25b9-0064c45d85-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.306 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:29:57.307 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:29:57.307 DEBUG:boto:Method: GET 2023-07-29T00:29:57.308 DEBUG:boto:Path: /hiiqos-59/obj33 2023-07-29T00:29:57.308 DEBUG:boto:Data: 2023-07-29T00:29:57.308 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.309 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:57.309 DEBUG:boto:Port: 8000 2023-07-29T00:29:57.309 DEBUG:boto:Params: {} 2023-07-29T00:29:57.310 DEBUG:boto:Token: None 2023-07-29T00:29:57.310 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj33 2023-07-29T00:29:57.311 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:unqqtKeSOQaF6u4nutihCYw8a8I= 2023-07-29T00:29:57.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:unqqtKeSOQaF6u4nutihCYw8a8I=', 'Content-Length': '0'} 2023-07-29T00:29:57.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b449540da34bb97-0064c45d85-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.314 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:57.315 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:57.315 DEBUG:boto:path=/hiiqos-59/?prefix=obj33 2023-07-29T00:29:57.315 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj33 2023-07-29T00:29:57.316 DEBUG:boto:Method: GET 2023-07-29T00:29:57.316 DEBUG:boto:Path: /hiiqos-59/?prefix=obj33 2023-07-29T00:29:57.317 DEBUG:boto:Data: 2023-07-29T00:29:57.317 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.317 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:57.318 DEBUG:boto:Port: 8001 2023-07-29T00:29:57.318 DEBUG:boto:Params: {} 2023-07-29T00:29:57.318 DEBUG:boto:Token: None 2023-07-29T00:29:57.319 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/ 2023-07-29T00:29:57.319 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0= 2023-07-29T00:29:57.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0=', 'Content-Length': '0'} 2023-07-29T00:29:57.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea1de17677069208-0064c45d85-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.323 DEBUG:boto:b'hiiqos-59obj331000falseobj332023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:57.323 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:57.324 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:57.324 DEBUG:boto:path=/hiiqos-59/?prefix=obj33 2023-07-29T00:29:57.325 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj33 2023-07-29T00:29:57.325 DEBUG:boto:Method: GET 2023-07-29T00:29:57.325 DEBUG:boto:Path: /hiiqos-59/?prefix=obj33 2023-07-29T00:29:57.326 DEBUG:boto:Data: 2023-07-29T00:29:57.326 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.326 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:57.327 DEBUG:boto:Port: 8000 2023-07-29T00:29:57.327 DEBUG:boto:Params: {} 2023-07-29T00:29:57.327 DEBUG:boto:Token: None 2023-07-29T00:29:57.328 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/ 2023-07-29T00:29:57.328 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0= 2023-07-29T00:29:57.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0=', 'Content-Length': '0'} 2023-07-29T00:29:57.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ad9bf1c2fbe3786-0064c45d85-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.372 DEBUG:boto:b'hiiqos-59obj331000falseobj332023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:57.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-07-29T00:29:57.375 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:29:57.376 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:29:57.377 DEBUG:boto:path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:29:57.378 DEBUG:boto:auth_path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:29:57.379 DEBUG:boto:Method: GET 2023-07-29T00:29:57.379 DEBUG:boto:Path: /hiiqos-59/obj34?versionId=null 2023-07-29T00:29:57.380 DEBUG:boto:Data: 2023-07-29T00:29:57.381 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.381 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:57.382 DEBUG:boto:Port: 8001 2023-07-29T00:29:57.383 DEBUG:boto:Params: {} 2023-07-29T00:29:57.383 DEBUG:boto:Token: None 2023-07-29T00:29:57.384 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj34?versionId=null 2023-07-29T00:29:57.384 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CPgTnzakBtcgLr32l1vnWfV5aS0= 2023-07-29T00:29:57.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CPgTnzakBtcgLr32l1vnWfV5aS0=', 'Content-Length': '0'} 2023-07-29T00:29:57.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5f711e167950c0e-0064c45d85-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.388 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:29:57.388 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:29:57.389 DEBUG:boto:path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:29:57.389 DEBUG:boto:auth_path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:29:57.390 DEBUG:boto:Method: GET 2023-07-29T00:29:57.390 DEBUG:boto:Path: /hiiqos-59/obj34?versionId=null 2023-07-29T00:29:57.390 DEBUG:boto:Data: 2023-07-29T00:29:57.391 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.391 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:57.391 DEBUG:boto:Port: 8000 2023-07-29T00:29:57.392 DEBUG:boto:Params: {} 2023-07-29T00:29:57.392 DEBUG:boto:Token: None 2023-07-29T00:29:57.393 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj34?versionId=null 2023-07-29T00:29:57.393 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CPgTnzakBtcgLr32l1vnWfV5aS0= 2023-07-29T00:29:57.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CPgTnzakBtcgLr32l1vnWfV5aS0=', 'Content-Length': '0'} 2023-07-29T00:29:57.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018209abda89c4f52-0064c45d85-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.396 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:29:57.396 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:29:57.397 DEBUG:boto:path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:29:57.397 DEBUG:boto:auth_path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:29:57.397 DEBUG:boto:Method: HEAD 2023-07-29T00:29:57.398 DEBUG:boto:Path: /hiiqos-59/obj34?versionId=null 2023-07-29T00:29:57.398 DEBUG:boto:Data: 2023-07-29T00:29:57.398 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.399 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:57.399 DEBUG:boto:Port: 8001 2023-07-29T00:29:57.399 DEBUG:boto:Params: {} 2023-07-29T00:29:57.400 DEBUG:boto:Token: None 2023-07-29T00:29:57.400 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj34?versionId=null 2023-07-29T00:29:57.401 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Y/wnY71dab1Jj2XObVJupsq8CWA= 2023-07-29T00:29:57.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Y/wnY71dab1Jj2XObVJupsq8CWA=', 'Content-Length': '0'} 2023-07-29T00:29:57.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d44e0141c9348d56-0064c45d85-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.405 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:29:57.405 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:29:57.405 DEBUG:boto:path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:29:57.406 DEBUG:boto:auth_path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:29:57.406 DEBUG:boto:Method: HEAD 2023-07-29T00:29:57.406 DEBUG:boto:Path: /hiiqos-59/obj34?versionId=null 2023-07-29T00:29:57.407 DEBUG:boto:Data: 2023-07-29T00:29:57.407 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.407 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:57.408 DEBUG:boto:Port: 8000 2023-07-29T00:29:57.408 DEBUG:boto:Params: {} 2023-07-29T00:29:57.408 DEBUG:boto:Token: None 2023-07-29T00:29:57.409 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj34?versionId=null 2023-07-29T00:29:57.409 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Y/wnY71dab1Jj2XObVJupsq8CWA= 2023-07-29T00:29:57.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Y/wnY71dab1Jj2XObVJupsq8CWA=', 'Content-Length': '0'} 2023-07-29T00:29:57.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b8a474b987698ff-0064c45d85-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-07-29T00:29:57.412 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:29:57.413 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:29:57.413 DEBUG:boto:path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:29:57.413 DEBUG:boto:auth_path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:29:57.414 DEBUG:boto:Method: GET 2023-07-29T00:29:57.414 DEBUG:boto:Path: /hiiqos-59/obj34?versionId=null 2023-07-29T00:29:57.414 DEBUG:boto:Data: 2023-07-29T00:29:57.415 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.415 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:57.415 DEBUG:boto:Port: 8001 2023-07-29T00:29:57.416 DEBUG:boto:Params: {} 2023-07-29T00:29:57.416 DEBUG:boto:Token: None 2023-07-29T00:29:57.417 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj34?versionId=null 2023-07-29T00:29:57.417 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CPgTnzakBtcgLr32l1vnWfV5aS0= 2023-07-29T00:29:57.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CPgTnzakBtcgLr32l1vnWfV5aS0=', 'Content-Length': '0'} 2023-07-29T00:29:57.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4aa2e168eca40a1-0064c45d85-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.420 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:29:57.420 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:29:57.421 DEBUG:boto:path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:29:57.421 DEBUG:boto:auth_path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:29:57.422 DEBUG:boto:Method: GET 2023-07-29T00:29:57.422 DEBUG:boto:Path: /hiiqos-59/obj34?versionId=null 2023-07-29T00:29:57.422 DEBUG:boto:Data: 2023-07-29T00:29:57.423 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.423 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:57.423 DEBUG:boto:Port: 8000 2023-07-29T00:29:57.424 DEBUG:boto:Params: {} 2023-07-29T00:29:57.424 DEBUG:boto:Token: None 2023-07-29T00:29:57.424 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj34?versionId=null 2023-07-29T00:29:57.425 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CPgTnzakBtcgLr32l1vnWfV5aS0= 2023-07-29T00:29:57.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CPgTnzakBtcgLr32l1vnWfV5aS0=', 'Content-Length': '0'} 2023-07-29T00:29:57.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1a786667435be7c-0064c45d85-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.428 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:57.428 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:57.429 DEBUG:boto:path=/hiiqos-59/?prefix=obj34 2023-07-29T00:29:57.429 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj34 2023-07-29T00:29:57.429 DEBUG:boto:Method: GET 2023-07-29T00:29:57.430 DEBUG:boto:Path: /hiiqos-59/?prefix=obj34 2023-07-29T00:29:57.430 DEBUG:boto:Data: 2023-07-29T00:29:57.430 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.431 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:57.431 DEBUG:boto:Port: 8001 2023-07-29T00:29:57.432 DEBUG:boto:Params: {} 2023-07-29T00:29:57.432 DEBUG:boto:Token: None 2023-07-29T00:29:57.432 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/ 2023-07-29T00:29:57.433 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0= 2023-07-29T00:29:57.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0=', 'Content-Length': '0'} 2023-07-29T00:29:57.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090abeb43cb182d98-0064c45d85-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.436 DEBUG:boto:b'hiiqos-59obj341000falseobj342023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:57.437 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:57.437 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:57.437 DEBUG:boto:path=/hiiqos-59/?prefix=obj34 2023-07-29T00:29:57.438 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj34 2023-07-29T00:29:57.438 DEBUG:boto:Method: GET 2023-07-29T00:29:57.438 DEBUG:boto:Path: /hiiqos-59/?prefix=obj34 2023-07-29T00:29:57.439 DEBUG:boto:Data: 2023-07-29T00:29:57.439 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.440 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:57.440 DEBUG:boto:Port: 8000 2023-07-29T00:29:57.440 DEBUG:boto:Params: {} 2023-07-29T00:29:57.441 DEBUG:boto:Token: None 2023-07-29T00:29:57.441 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/ 2023-07-29T00:29:57.442 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0= 2023-07-29T00:29:57.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0=', 'Content-Length': '0'} 2023-07-29T00:29:57.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fd3d44378db3e7d-0064c45d85-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.445 DEBUG:boto:b'hiiqos-59obj341000falseobj342023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:57.446 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:29:57.446 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:29:57.447 DEBUG:boto:Method: HEAD 2023-07-29T00:29:57.447 DEBUG:boto:Path: /hiiqos-59/obj34 2023-07-29T00:29:57.448 DEBUG:boto:Data: 2023-07-29T00:29:57.448 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.448 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:57.449 DEBUG:boto:Port: 8001 2023-07-29T00:29:57.449 DEBUG:boto:Params: {} 2023-07-29T00:29:57.449 DEBUG:boto:Token: None 2023-07-29T00:29:57.450 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj34 2023-07-29T00:29:57.450 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wyg5KwBHOPe22skuIHqtFV6gfqk= 2023-07-29T00:29:57.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wyg5KwBHOPe22skuIHqtFV6gfqk=', 'Content-Length': '0'} 2023-07-29T00:29:57.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c40114cc35f7c081-0064c45d85-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.453 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:29:57.453 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:29:57.453 DEBUG:boto:Method: HEAD 2023-07-29T00:29:57.454 DEBUG:boto:Path: /hiiqos-59/obj34 2023-07-29T00:29:57.454 DEBUG:boto:Data: 2023-07-29T00:29:57.454 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.455 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:57.455 DEBUG:boto:Port: 8000 2023-07-29T00:29:57.455 DEBUG:boto:Params: {} 2023-07-29T00:29:57.456 DEBUG:boto:Token: None 2023-07-29T00:29:57.456 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj34 2023-07-29T00:29:57.457 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wyg5KwBHOPe22skuIHqtFV6gfqk= 2023-07-29T00:29:57.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wyg5KwBHOPe22skuIHqtFV6gfqk=', 'Content-Length': '0'} 2023-07-29T00:29:57.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b1b9899f51147f8-0064c45d85-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-07-29T00:29:57.460 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:29:57.460 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:29:57.461 DEBUG:boto:Method: GET 2023-07-29T00:29:57.461 DEBUG:boto:Path: /hiiqos-59/obj34 2023-07-29T00:29:57.461 DEBUG:boto:Data: 2023-07-29T00:29:57.462 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.462 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:57.462 DEBUG:boto:Port: 8001 2023-07-29T00:29:57.463 DEBUG:boto:Params: {} 2023-07-29T00:29:57.463 DEBUG:boto:Token: None 2023-07-29T00:29:57.464 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj34 2023-07-29T00:29:57.464 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:18MaaJ2loirhx4kvFreC+Oj3R2A= 2023-07-29T00:29:57.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:18MaaJ2loirhx4kvFreC+Oj3R2A=', 'Content-Length': '0'} 2023-07-29T00:29:57.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007830733e08fbcd72-0064c45d85-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.467 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:29:57.467 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:29:57.468 DEBUG:boto:Method: GET 2023-07-29T00:29:57.468 DEBUG:boto:Path: /hiiqos-59/obj34 2023-07-29T00:29:57.468 DEBUG:boto:Data: 2023-07-29T00:29:57.469 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.469 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:57.470 DEBUG:boto:Port: 8000 2023-07-29T00:29:57.470 DEBUG:boto:Params: {} 2023-07-29T00:29:57.470 DEBUG:boto:Token: None 2023-07-29T00:29:57.471 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj34 2023-07-29T00:29:57.471 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:18MaaJ2loirhx4kvFreC+Oj3R2A= 2023-07-29T00:29:57.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:18MaaJ2loirhx4kvFreC+Oj3R2A=', 'Content-Length': '0'} 2023-07-29T00:29:57.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003468d51bcb5ba726-0064c45d85-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.474 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:57.475 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:57.475 DEBUG:boto:path=/hiiqos-59/?prefix=obj34 2023-07-29T00:29:57.475 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj34 2023-07-29T00:29:57.476 DEBUG:boto:Method: GET 2023-07-29T00:29:57.476 DEBUG:boto:Path: /hiiqos-59/?prefix=obj34 2023-07-29T00:29:57.476 DEBUG:boto:Data: 2023-07-29T00:29:57.477 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.477 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:57.477 DEBUG:boto:Port: 8001 2023-07-29T00:29:57.478 DEBUG:boto:Params: {} 2023-07-29T00:29:57.478 DEBUG:boto:Token: None 2023-07-29T00:29:57.479 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/ 2023-07-29T00:29:57.479 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0= 2023-07-29T00:29:57.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0=', 'Content-Length': '0'} 2023-07-29T00:29:57.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bec1940632cb307-0064c45d85-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.482 DEBUG:boto:b'hiiqos-59obj341000falseobj342023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:57.483 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:57.483 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:57.484 DEBUG:boto:path=/hiiqos-59/?prefix=obj34 2023-07-29T00:29:57.484 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj34 2023-07-29T00:29:57.484 DEBUG:boto:Method: GET 2023-07-29T00:29:57.485 DEBUG:boto:Path: /hiiqos-59/?prefix=obj34 2023-07-29T00:29:57.485 DEBUG:boto:Data: 2023-07-29T00:29:57.485 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.486 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:57.486 DEBUG:boto:Port: 8000 2023-07-29T00:29:57.486 DEBUG:boto:Params: {} 2023-07-29T00:29:57.487 DEBUG:boto:Token: None 2023-07-29T00:29:57.487 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/ 2023-07-29T00:29:57.488 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0= 2023-07-29T00:29:57.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0=', 'Content-Length': '0'} 2023-07-29T00:29:57.491 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdfa99faa73ec39d-0064c45d85-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.491 DEBUG:boto:b'hiiqos-59obj341000falseobj342023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:57.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-07-29T00:29:57.492 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:29:57.493 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:29:57.493 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:29:57.493 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:29:57.494 DEBUG:boto:Method: GET 2023-07-29T00:29:57.494 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:29:57.494 DEBUG:boto:Data: 2023-07-29T00:29:57.495 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.495 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:57.495 DEBUG:boto:Port: 8001 2023-07-29T00:29:57.496 DEBUG:boto:Params: {} 2023-07-29T00:29:57.496 DEBUG:boto:Token: None 2023-07-29T00:29:57.497 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:29:57.497 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IWJFAZHGml8YSyHeoHG8jXVn/7A= 2023-07-29T00:29:57.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IWJFAZHGml8YSyHeoHG8jXVn/7A=', 'Content-Length': '0'} 2023-07-29T00:29:57.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000657b86b98ba78e75-0064c45d85-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.500 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:29:57.500 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:29:57.501 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:29:57.501 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:29:57.501 DEBUG:boto:Method: GET 2023-07-29T00:29:57.502 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:29:57.502 DEBUG:boto:Data: 2023-07-29T00:29:57.502 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.503 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:57.503 DEBUG:boto:Port: 8000 2023-07-29T00:29:57.503 DEBUG:boto:Params: {} 2023-07-29T00:29:57.504 DEBUG:boto:Token: None 2023-07-29T00:29:57.504 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:29:57.505 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IWJFAZHGml8YSyHeoHG8jXVn/7A= 2023-07-29T00:29:57.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IWJFAZHGml8YSyHeoHG8jXVn/7A=', 'Content-Length': '0'} 2023-07-29T00:29:57.507 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5a944b2a117fa9f-0064c45d85-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.507 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:29:57.508 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:29:57.508 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:29:57.508 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:29:57.509 DEBUG:boto:Method: HEAD 2023-07-29T00:29:57.509 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:29:57.509 DEBUG:boto:Data: 2023-07-29T00:29:57.510 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.510 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:57.510 DEBUG:boto:Port: 8001 2023-07-29T00:29:57.511 DEBUG:boto:Params: {} 2023-07-29T00:29:57.511 DEBUG:boto:Token: None 2023-07-29T00:29:57.512 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:29:57.512 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qlyk8nrP649g1lEH99xeKz4BT7Q= 2023-07-29T00:29:57.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qlyk8nrP649g1lEH99xeKz4BT7Q=', 'Content-Length': '0'} 2023-07-29T00:29:57.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e48b18bd3df4d5d-0064c45d85-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.515 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:29:57.515 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:29:57.515 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:29:57.516 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:29:57.516 DEBUG:boto:Method: HEAD 2023-07-29T00:29:57.516 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:29:57.517 DEBUG:boto:Data: 2023-07-29T00:29:57.517 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.517 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:57.518 DEBUG:boto:Port: 8000 2023-07-29T00:29:57.518 DEBUG:boto:Params: {} 2023-07-29T00:29:57.518 DEBUG:boto:Token: None 2023-07-29T00:29:57.519 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:29:57.519 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qlyk8nrP649g1lEH99xeKz4BT7Q= 2023-07-29T00:29:57.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qlyk8nrP649g1lEH99xeKz4BT7Q=', 'Content-Length': '0'} 2023-07-29T00:29:57.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c81451523b7fd56b-0064c45d85-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-07-29T00:29:57.522 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:29:57.523 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:29:57.523 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:29:57.523 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:29:57.524 DEBUG:boto:Method: GET 2023-07-29T00:29:57.524 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:29:57.524 DEBUG:boto:Data: 2023-07-29T00:29:57.525 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.525 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:57.525 DEBUG:boto:Port: 8001 2023-07-29T00:29:57.526 DEBUG:boto:Params: {} 2023-07-29T00:29:57.526 DEBUG:boto:Token: None 2023-07-29T00:29:57.527 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:29:57.527 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IWJFAZHGml8YSyHeoHG8jXVn/7A= 2023-07-29T00:29:57.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IWJFAZHGml8YSyHeoHG8jXVn/7A=', 'Content-Length': '0'} 2023-07-29T00:29:57.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee6b29c2c15c6069-0064c45d85-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.530 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:29:57.530 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:29:57.531 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:29:57.531 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:29:57.531 DEBUG:boto:Method: GET 2023-07-29T00:29:57.532 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:29:57.532 DEBUG:boto:Data: 2023-07-29T00:29:57.532 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.533 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:57.533 DEBUG:boto:Port: 8000 2023-07-29T00:29:57.533 DEBUG:boto:Params: {} 2023-07-29T00:29:57.534 DEBUG:boto:Token: None 2023-07-29T00:29:57.534 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:29:57.535 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IWJFAZHGml8YSyHeoHG8jXVn/7A= 2023-07-29T00:29:57.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IWJFAZHGml8YSyHeoHG8jXVn/7A=', 'Content-Length': '0'} 2023-07-29T00:29:57.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068f0b300012e71e2-0064c45d85-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.537 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:57.538 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:57.538 DEBUG:boto:path=/hiiqos-59/?prefix=obj44 2023-07-29T00:29:57.539 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj44 2023-07-29T00:29:57.539 DEBUG:boto:Method: GET 2023-07-29T00:29:57.539 DEBUG:boto:Path: /hiiqos-59/?prefix=obj44 2023-07-29T00:29:57.540 DEBUG:boto:Data: 2023-07-29T00:29:57.540 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.540 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:57.541 DEBUG:boto:Port: 8001 2023-07-29T00:29:57.541 DEBUG:boto:Params: {} 2023-07-29T00:29:57.541 DEBUG:boto:Token: None 2023-07-29T00:29:57.542 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/ 2023-07-29T00:29:57.542 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0= 2023-07-29T00:29:57.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0=', 'Content-Length': '0'} 2023-07-29T00:29:57.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000725ce9831a5b466a-0064c45d85-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.588 DEBUG:boto:b'hiiqos-59obj441000falseobj442023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:57.589 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:57.589 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:57.590 DEBUG:boto:path=/hiiqos-59/?prefix=obj44 2023-07-29T00:29:57.590 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj44 2023-07-29T00:29:57.590 DEBUG:boto:Method: GET 2023-07-29T00:29:57.591 DEBUG:boto:Path: /hiiqos-59/?prefix=obj44 2023-07-29T00:29:57.591 DEBUG:boto:Data: 2023-07-29T00:29:57.591 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.592 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:57.592 DEBUG:boto:Port: 8000 2023-07-29T00:29:57.593 DEBUG:boto:Params: {} 2023-07-29T00:29:57.593 DEBUG:boto:Token: None 2023-07-29T00:29:57.593 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/ 2023-07-29T00:29:57.594 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0= 2023-07-29T00:29:57.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0=', 'Content-Length': '0'} 2023-07-29T00:29:57.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000769a583443c21e93-0064c45d85-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.598 DEBUG:boto:b'hiiqos-59obj441000falseobj442023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:57.598 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:29:57.599 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:29:57.599 DEBUG:boto:Method: HEAD 2023-07-29T00:29:57.599 DEBUG:boto:Path: /hiiqos-59/obj44 2023-07-29T00:29:57.600 DEBUG:boto:Data: 2023-07-29T00:29:57.600 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.600 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:57.601 DEBUG:boto:Port: 8001 2023-07-29T00:29:57.601 DEBUG:boto:Params: {} 2023-07-29T00:29:57.601 DEBUG:boto:Token: None 2023-07-29T00:29:57.602 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj44 2023-07-29T00:29:57.602 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2/c83Iko+UwvkHPI1OPr+Dr4bJE= 2023-07-29T00:29:57.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2/c83Iko+UwvkHPI1OPr+Dr4bJE=', 'Content-Length': '0'} 2023-07-29T00:29:57.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035ed99182c068532-0064c45d85-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.605 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:29:57.605 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:29:57.606 DEBUG:boto:Method: HEAD 2023-07-29T00:29:57.606 DEBUG:boto:Path: /hiiqos-59/obj44 2023-07-29T00:29:57.606 DEBUG:boto:Data: 2023-07-29T00:29:57.607 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.607 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:57.607 DEBUG:boto:Port: 8000 2023-07-29T00:29:57.608 DEBUG:boto:Params: {} 2023-07-29T00:29:57.608 DEBUG:boto:Token: None 2023-07-29T00:29:57.608 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj44 2023-07-29T00:29:57.609 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2/c83Iko+UwvkHPI1OPr+Dr4bJE= 2023-07-29T00:29:57.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2/c83Iko+UwvkHPI1OPr+Dr4bJE=', 'Content-Length': '0'} 2023-07-29T00:29:57.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d32f9d99e96cc5a0-0064c45d85-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-07-29T00:29:57.612 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:29:57.612 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:29:57.613 DEBUG:boto:Method: GET 2023-07-29T00:29:57.613 DEBUG:boto:Path: /hiiqos-59/obj44 2023-07-29T00:29:57.613 DEBUG:boto:Data: 2023-07-29T00:29:57.614 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.614 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:57.614 DEBUG:boto:Port: 8001 2023-07-29T00:29:57.615 DEBUG:boto:Params: {} 2023-07-29T00:29:57.615 DEBUG:boto:Token: None 2023-07-29T00:29:57.615 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj44 2023-07-29T00:29:57.616 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/jpKcYSiy0HgwpVUAOXaJfFMPNc= 2023-07-29T00:29:57.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/jpKcYSiy0HgwpVUAOXaJfFMPNc=', 'Content-Length': '0'} 2023-07-29T00:29:57.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aba7dc04041e626b-0064c45d85-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.619 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:29:57.619 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:29:57.620 DEBUG:boto:Method: GET 2023-07-29T00:29:57.620 DEBUG:boto:Path: /hiiqos-59/obj44 2023-07-29T00:29:57.620 DEBUG:boto:Data: 2023-07-29T00:29:57.621 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.621 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:57.621 DEBUG:boto:Port: 8000 2023-07-29T00:29:57.622 DEBUG:boto:Params: {} 2023-07-29T00:29:57.622 DEBUG:boto:Token: None 2023-07-29T00:29:57.623 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj44 2023-07-29T00:29:57.623 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/jpKcYSiy0HgwpVUAOXaJfFMPNc= 2023-07-29T00:29:57.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/jpKcYSiy0HgwpVUAOXaJfFMPNc=', 'Content-Length': '0'} 2023-07-29T00:29:57.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb30c51370f5b584-0064c45d85-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.672 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:57.673 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:57.673 DEBUG:boto:path=/hiiqos-59/?prefix=obj44 2023-07-29T00:29:57.674 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj44 2023-07-29T00:29:57.674 DEBUG:boto:Method: GET 2023-07-29T00:29:57.675 DEBUG:boto:Path: /hiiqos-59/?prefix=obj44 2023-07-29T00:29:57.675 DEBUG:boto:Data: 2023-07-29T00:29:57.675 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.676 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:57.676 DEBUG:boto:Port: 8001 2023-07-29T00:29:57.676 DEBUG:boto:Params: {} 2023-07-29T00:29:57.677 DEBUG:boto:Token: None 2023-07-29T00:29:57.677 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/ 2023-07-29T00:29:57.678 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0= 2023-07-29T00:29:57.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0=', 'Content-Length': '0'} 2023-07-29T00:29:57.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa0e7430ada67bcb-0064c45d85-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.681 DEBUG:boto:b'hiiqos-59obj441000falseobj442023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:57.682 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:57.682 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:57.683 DEBUG:boto:path=/hiiqos-59/?prefix=obj44 2023-07-29T00:29:57.683 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj44 2023-07-29T00:29:57.683 DEBUG:boto:Method: GET 2023-07-29T00:29:57.684 DEBUG:boto:Path: /hiiqos-59/?prefix=obj44 2023-07-29T00:29:57.684 DEBUG:boto:Data: 2023-07-29T00:29:57.684 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.685 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:57.685 DEBUG:boto:Port: 8000 2023-07-29T00:29:57.686 DEBUG:boto:Params: {} 2023-07-29T00:29:57.686 DEBUG:boto:Token: None 2023-07-29T00:29:57.686 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/ 2023-07-29T00:29:57.687 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0= 2023-07-29T00:29:57.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0=', 'Content-Length': '0'} 2023-07-29T00:29:57.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b220bd51be5a739b-0064c45d85-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.690 DEBUG:boto:b'hiiqos-59obj441000falseobj442023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:57.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-07-29T00:29:57.691 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:29:57.692 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:29:57.692 DEBUG:boto:path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:29:57.692 DEBUG:boto:auth_path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:29:57.693 DEBUG:boto:Method: GET 2023-07-29T00:29:57.693 DEBUG:boto:Path: /hiiqos-59/obj51?versionId=null 2023-07-29T00:29:57.693 DEBUG:boto:Data: 2023-07-29T00:29:57.694 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.694 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:57.694 DEBUG:boto:Port: 8001 2023-07-29T00:29:57.695 DEBUG:boto:Params: {} 2023-07-29T00:29:57.695 DEBUG:boto:Token: None 2023-07-29T00:29:57.696 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj51?versionId=null 2023-07-29T00:29:57.696 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oSsFto+VTqGQRZvjfKD7yQp3GrM= 2023-07-29T00:29:57.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oSsFto+VTqGQRZvjfKD7yQp3GrM=', 'Content-Length': '0'} 2023-07-29T00:29:57.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000632301f3121f972c-0064c45d85-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.744 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:29:57.745 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:29:57.745 DEBUG:boto:path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:29:57.746 DEBUG:boto:auth_path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:29:57.746 DEBUG:boto:Method: GET 2023-07-29T00:29:57.746 DEBUG:boto:Path: /hiiqos-59/obj51?versionId=null 2023-07-29T00:29:57.747 DEBUG:boto:Data: 2023-07-29T00:29:57.747 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.747 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:57.748 DEBUG:boto:Port: 8000 2023-07-29T00:29:57.748 DEBUG:boto:Params: {} 2023-07-29T00:29:57.748 DEBUG:boto:Token: None 2023-07-29T00:29:57.749 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj51?versionId=null 2023-07-29T00:29:57.749 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oSsFto+VTqGQRZvjfKD7yQp3GrM= 2023-07-29T00:29:57.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oSsFto+VTqGQRZvjfKD7yQp3GrM=', 'Content-Length': '0'} 2023-07-29T00:29:57.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000345ddcce2b1ff68e-0064c45d85-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.753 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:29:57.753 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:29:57.754 DEBUG:boto:path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:29:57.754 DEBUG:boto:auth_path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:29:57.754 DEBUG:boto:Method: HEAD 2023-07-29T00:29:57.755 DEBUG:boto:Path: /hiiqos-59/obj51?versionId=null 2023-07-29T00:29:57.755 DEBUG:boto:Data: 2023-07-29T00:29:57.755 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.756 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:57.756 DEBUG:boto:Port: 8001 2023-07-29T00:29:57.756 DEBUG:boto:Params: {} 2023-07-29T00:29:57.757 DEBUG:boto:Token: None 2023-07-29T00:29:57.757 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj51?versionId=null 2023-07-29T00:29:57.757 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pT83cj0qZecznmkxda2B/pq+2rA= 2023-07-29T00:29:57.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pT83cj0qZecznmkxda2B/pq+2rA=', 'Content-Length': '0'} 2023-07-29T00:29:57.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e091f119189334b6-0064c45d85-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.760 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:29:57.761 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:29:57.761 DEBUG:boto:path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:29:57.761 DEBUG:boto:auth_path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:29:57.762 DEBUG:boto:Method: HEAD 2023-07-29T00:29:57.762 DEBUG:boto:Path: /hiiqos-59/obj51?versionId=null 2023-07-29T00:29:57.762 DEBUG:boto:Data: 2023-07-29T00:29:57.763 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.763 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:57.763 DEBUG:boto:Port: 8000 2023-07-29T00:29:57.764 DEBUG:boto:Params: {} 2023-07-29T00:29:57.764 DEBUG:boto:Token: None 2023-07-29T00:29:57.764 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj51?versionId=null 2023-07-29T00:29:57.765 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pT83cj0qZecznmkxda2B/pq+2rA= 2023-07-29T00:29:57.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pT83cj0qZecznmkxda2B/pq+2rA=', 'Content-Length': '0'} 2023-07-29T00:29:57.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e8d4615ffb5e21f-0064c45d85-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-07-29T00:29:57.768 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:29:57.768 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:29:57.768 DEBUG:boto:path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:29:57.769 DEBUG:boto:auth_path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:29:57.770 DEBUG:boto:Method: GET 2023-07-29T00:29:57.770 DEBUG:boto:Path: /hiiqos-59/obj51?versionId=null 2023-07-29T00:29:57.770 DEBUG:boto:Data: 2023-07-29T00:29:57.771 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.771 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:57.771 DEBUG:boto:Port: 8001 2023-07-29T00:29:57.772 DEBUG:boto:Params: {} 2023-07-29T00:29:57.772 DEBUG:boto:Token: None 2023-07-29T00:29:57.773 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj51?versionId=null 2023-07-29T00:29:57.773 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oSsFto+VTqGQRZvjfKD7yQp3GrM= 2023-07-29T00:29:57.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oSsFto+VTqGQRZvjfKD7yQp3GrM=', 'Content-Length': '0'} 2023-07-29T00:29:57.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065f60e45626f1fc5-0064c45d85-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.776 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:29:57.776 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:29:57.777 DEBUG:boto:path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:29:57.777 DEBUG:boto:auth_path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:29:57.777 DEBUG:boto:Method: GET 2023-07-29T00:29:57.778 DEBUG:boto:Path: /hiiqos-59/obj51?versionId=null 2023-07-29T00:29:57.778 DEBUG:boto:Data: 2023-07-29T00:29:57.778 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.779 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:57.779 DEBUG:boto:Port: 8000 2023-07-29T00:29:57.780 DEBUG:boto:Params: {} 2023-07-29T00:29:57.780 DEBUG:boto:Token: None 2023-07-29T00:29:57.780 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj51?versionId=null 2023-07-29T00:29:57.781 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oSsFto+VTqGQRZvjfKD7yQp3GrM= 2023-07-29T00:29:57.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oSsFto+VTqGQRZvjfKD7yQp3GrM=', 'Content-Length': '0'} 2023-07-29T00:29:57.783 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc7b4afe1306b797-0064c45d85-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.784 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:57.784 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:57.784 DEBUG:boto:path=/hiiqos-59/?prefix=obj51 2023-07-29T00:29:57.785 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj51 2023-07-29T00:29:57.785 DEBUG:boto:Method: GET 2023-07-29T00:29:57.785 DEBUG:boto:Path: /hiiqos-59/?prefix=obj51 2023-07-29T00:29:57.786 DEBUG:boto:Data: 2023-07-29T00:29:57.786 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.786 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:57.787 DEBUG:boto:Port: 8001 2023-07-29T00:29:57.787 DEBUG:boto:Params: {} 2023-07-29T00:29:57.788 DEBUG:boto:Token: None 2023-07-29T00:29:57.788 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/ 2023-07-29T00:29:57.788 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0= 2023-07-29T00:29:57.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0=', 'Content-Length': '0'} 2023-07-29T00:29:57.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f9addbb823cc56b-0064c45d85-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.792 DEBUG:boto:b'hiiqos-59obj511000falseobj512023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:57.792 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:57.793 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:57.793 DEBUG:boto:path=/hiiqos-59/?prefix=obj51 2023-07-29T00:29:57.793 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj51 2023-07-29T00:29:57.794 DEBUG:boto:Method: GET 2023-07-29T00:29:57.794 DEBUG:boto:Path: /hiiqos-59/?prefix=obj51 2023-07-29T00:29:57.794 DEBUG:boto:Data: 2023-07-29T00:29:57.795 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.795 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:57.795 DEBUG:boto:Port: 8000 2023-07-29T00:29:57.796 DEBUG:boto:Params: {} 2023-07-29T00:29:57.796 DEBUG:boto:Token: None 2023-07-29T00:29:57.797 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/ 2023-07-29T00:29:57.797 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0= 2023-07-29T00:29:57.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0=', 'Content-Length': '0'} 2023-07-29T00:29:57.800 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002aa4dd78b0fd5d5e-0064c45d85-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.800 DEBUG:boto:b'hiiqos-59obj511000falseobj512023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:57.801 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:29:57.801 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:29:57.802 DEBUG:boto:Method: HEAD 2023-07-29T00:29:57.802 DEBUG:boto:Path: /hiiqos-59/obj51 2023-07-29T00:29:57.802 DEBUG:boto:Data: 2023-07-29T00:29:57.803 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.803 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:57.803 DEBUG:boto:Port: 8001 2023-07-29T00:29:57.804 DEBUG:boto:Params: {} 2023-07-29T00:29:57.804 DEBUG:boto:Token: None 2023-07-29T00:29:57.804 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj51 2023-07-29T00:29:57.805 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LylagAs0GKgAdLd22MbhKkML6qg= 2023-07-29T00:29:57.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LylagAs0GKgAdLd22MbhKkML6qg=', 'Content-Length': '0'} 2023-07-29T00:29:57.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075bde9aab779cd72-0064c45d85-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.807 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:29:57.808 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:29:57.808 DEBUG:boto:Method: HEAD 2023-07-29T00:29:57.808 DEBUG:boto:Path: /hiiqos-59/obj51 2023-07-29T00:29:57.809 DEBUG:boto:Data: 2023-07-29T00:29:57.809 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.809 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:57.810 DEBUG:boto:Port: 8000 2023-07-29T00:29:57.810 DEBUG:boto:Params: {} 2023-07-29T00:29:57.810 DEBUG:boto:Token: None 2023-07-29T00:29:57.811 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj51 2023-07-29T00:29:57.811 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LylagAs0GKgAdLd22MbhKkML6qg= 2023-07-29T00:29:57.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LylagAs0GKgAdLd22MbhKkML6qg=', 'Content-Length': '0'} 2023-07-29T00:29:57.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a0881e7c0f15c8b-0064c45d85-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-07-29T00:29:57.814 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:29:57.815 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:29:57.815 DEBUG:boto:Method: GET 2023-07-29T00:29:57.815 DEBUG:boto:Path: /hiiqos-59/obj51 2023-07-29T00:29:57.816 DEBUG:boto:Data: 2023-07-29T00:29:57.816 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.816 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:57.817 DEBUG:boto:Port: 8001 2023-07-29T00:29:57.817 DEBUG:boto:Params: {} 2023-07-29T00:29:57.817 DEBUG:boto:Token: None 2023-07-29T00:29:57.818 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj51 2023-07-29T00:29:57.818 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:G+C/b5C+q3OTXnMZLgbpL/dxV+0= 2023-07-29T00:29:57.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:G+C/b5C+q3OTXnMZLgbpL/dxV+0=', 'Content-Length': '0'} 2023-07-29T00:29:57.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074d93f9872592bae-0064c45d85-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.821 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:29:57.821 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:29:57.822 DEBUG:boto:Method: GET 2023-07-29T00:29:57.822 DEBUG:boto:Path: /hiiqos-59/obj51 2023-07-29T00:29:57.822 DEBUG:boto:Data: 2023-07-29T00:29:57.823 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.823 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:57.824 DEBUG:boto:Port: 8000 2023-07-29T00:29:57.824 DEBUG:boto:Params: {} 2023-07-29T00:29:57.824 DEBUG:boto:Token: None 2023-07-29T00:29:57.825 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj51 2023-07-29T00:29:57.825 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:G+C/b5C+q3OTXnMZLgbpL/dxV+0= 2023-07-29T00:29:57.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:G+C/b5C+q3OTXnMZLgbpL/dxV+0=', 'Content-Length': '0'} 2023-07-29T00:29:57.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d15a1db0a1414dbf-0064c45d85-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.828 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:57.828 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:57.828 DEBUG:boto:path=/hiiqos-59/?prefix=obj51 2023-07-29T00:29:57.829 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj51 2023-07-29T00:29:57.829 DEBUG:boto:Method: GET 2023-07-29T00:29:57.830 DEBUG:boto:Path: /hiiqos-59/?prefix=obj51 2023-07-29T00:29:57.830 DEBUG:boto:Data: 2023-07-29T00:29:57.830 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.831 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:57.831 DEBUG:boto:Port: 8001 2023-07-29T00:29:57.831 DEBUG:boto:Params: {} 2023-07-29T00:29:57.832 DEBUG:boto:Token: None 2023-07-29T00:29:57.832 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/ 2023-07-29T00:29:57.832 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0= 2023-07-29T00:29:57.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0=', 'Content-Length': '0'} 2023-07-29T00:29:57.835 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f09f0558edce2f1-0064c45d85-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.835 DEBUG:boto:b'hiiqos-59obj511000falseobj512023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:57.836 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:57.837 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:57.837 DEBUG:boto:path=/hiiqos-59/?prefix=obj51 2023-07-29T00:29:57.837 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj51 2023-07-29T00:29:57.838 DEBUG:boto:Method: GET 2023-07-29T00:29:57.838 DEBUG:boto:Path: /hiiqos-59/?prefix=obj51 2023-07-29T00:29:57.838 DEBUG:boto:Data: 2023-07-29T00:29:57.839 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.839 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:57.839 DEBUG:boto:Port: 8000 2023-07-29T00:29:57.840 DEBUG:boto:Params: {} 2023-07-29T00:29:57.840 DEBUG:boto:Token: None 2023-07-29T00:29:57.841 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/ 2023-07-29T00:29:57.841 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0= 2023-07-29T00:29:57.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0=', 'Content-Length': '0'} 2023-07-29T00:29:57.844 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094af008087da4ab2-0064c45d85-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.844 DEBUG:boto:b'hiiqos-59obj511000falseobj512023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:57.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-07-29T00:29:57.845 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:29:57.846 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:29:57.846 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:29:57.846 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:29:57.847 DEBUG:boto:Method: GET 2023-07-29T00:29:57.847 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:29:57.847 DEBUG:boto:Data: 2023-07-29T00:29:57.848 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.848 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:57.849 DEBUG:boto:Port: 8001 2023-07-29T00:29:57.849 DEBUG:boto:Params: {} 2023-07-29T00:29:57.849 DEBUG:boto:Token: None 2023-07-29T00:29:57.850 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:29:57.850 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:blQ9V8AB7fbW4WL+xkejjh1VwcM= 2023-07-29T00:29:57.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:blQ9V8AB7fbW4WL+xkejjh1VwcM=', 'Content-Length': '0'} 2023-07-29T00:29:57.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016f89409579c0075-0064c45d85-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.853 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:29:57.853 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:29:57.854 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:29:57.854 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:29:57.854 DEBUG:boto:Method: GET 2023-07-29T00:29:57.855 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:29:57.855 DEBUG:boto:Data: 2023-07-29T00:29:57.855 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.856 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:57.856 DEBUG:boto:Port: 8000 2023-07-29T00:29:57.857 DEBUG:boto:Params: {} 2023-07-29T00:29:57.857 DEBUG:boto:Token: None 2023-07-29T00:29:57.857 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:29:57.858 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:blQ9V8AB7fbW4WL+xkejjh1VwcM= 2023-07-29T00:29:57.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:blQ9V8AB7fbW4WL+xkejjh1VwcM=', 'Content-Length': '0'} 2023-07-29T00:29:57.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009325fd46f998ee00-0064c45d85-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.861 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:29:57.862 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:29:57.862 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:29:57.863 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:29:57.864 DEBUG:boto:Method: HEAD 2023-07-29T00:29:57.864 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:29:57.865 DEBUG:boto:Data: 2023-07-29T00:29:57.866 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.866 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:57.867 DEBUG:boto:Port: 8001 2023-07-29T00:29:57.868 DEBUG:boto:Params: {} 2023-07-29T00:29:57.869 DEBUG:boto:Token: None 2023-07-29T00:29:57.869 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:29:57.870 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KnFy63i9HtT58IqJDgqsZ7bFrXY= 2023-07-29T00:29:57.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KnFy63i9HtT58IqJDgqsZ7bFrXY=', 'Content-Length': '0'} 2023-07-29T00:29:57.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040858b31fcd5d995-0064c45d85-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.873 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:29:57.874 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:29:57.875 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:29:57.875 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:29:57.876 DEBUG:boto:Method: HEAD 2023-07-29T00:29:57.877 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:29:57.877 DEBUG:boto:Data: 2023-07-29T00:29:57.878 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.878 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:57.879 DEBUG:boto:Port: 8000 2023-07-29T00:29:57.879 DEBUG:boto:Params: {} 2023-07-29T00:29:57.880 DEBUG:boto:Token: None 2023-07-29T00:29:57.880 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:29:57.880 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KnFy63i9HtT58IqJDgqsZ7bFrXY= 2023-07-29T00:29:57.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KnFy63i9HtT58IqJDgqsZ7bFrXY=', 'Content-Length': '0'} 2023-07-29T00:29:57.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009aa8e741232d518-0064c45d85-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-07-29T00:29:57.883 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:29:57.884 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:29:57.884 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:29:57.885 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:29:57.885 DEBUG:boto:Method: GET 2023-07-29T00:29:57.885 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:29:57.886 DEBUG:boto:Data: 2023-07-29T00:29:57.886 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.886 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:57.887 DEBUG:boto:Port: 8001 2023-07-29T00:29:57.887 DEBUG:boto:Params: {} 2023-07-29T00:29:57.888 DEBUG:boto:Token: None 2023-07-29T00:29:57.888 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:29:57.888 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:blQ9V8AB7fbW4WL+xkejjh1VwcM= 2023-07-29T00:29:57.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:blQ9V8AB7fbW4WL+xkejjh1VwcM=', 'Content-Length': '0'} 2023-07-29T00:29:57.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000372e6c4eb4d877bc-0064c45d85-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.936 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:29:57.937 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:29:57.937 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:29:57.938 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:29:57.938 DEBUG:boto:Method: GET 2023-07-29T00:29:57.938 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:29:57.939 DEBUG:boto:Data: 2023-07-29T00:29:57.939 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.940 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:57.940 DEBUG:boto:Port: 8000 2023-07-29T00:29:57.940 DEBUG:boto:Params: {} 2023-07-29T00:29:57.942 DEBUG:boto:Token: None 2023-07-29T00:29:57.942 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:29:57.943 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:blQ9V8AB7fbW4WL+xkejjh1VwcM= 2023-07-29T00:29:57.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:blQ9V8AB7fbW4WL+xkejjh1VwcM=', 'Content-Length': '0'} 2023-07-29T00:29:57.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6aa860b8fe670f9-0064c45d85-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.946 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:57.946 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:57.946 DEBUG:boto:path=/hiiqos-59/?prefix=obj55 2023-07-29T00:29:57.947 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj55 2023-07-29T00:29:57.947 DEBUG:boto:Method: GET 2023-07-29T00:29:57.948 DEBUG:boto:Path: /hiiqos-59/?prefix=obj55 2023-07-29T00:29:57.948 DEBUG:boto:Data: 2023-07-29T00:29:57.948 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.949 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:57.949 DEBUG:boto:Port: 8001 2023-07-29T00:29:57.950 DEBUG:boto:Params: {} 2023-07-29T00:29:57.950 DEBUG:boto:Token: None 2023-07-29T00:29:57.950 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/ 2023-07-29T00:29:57.951 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0= 2023-07-29T00:29:57.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0=', 'Content-Length': '0'} 2023-07-29T00:29:57.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000092cf51da86d3e16-0064c45d85-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:57.954 DEBUG:boto:b'hiiqos-59obj551000falseobj552023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:57.955 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:57.955 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:57.956 DEBUG:boto:path=/hiiqos-59/?prefix=obj55 2023-07-29T00:29:57.956 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj55 2023-07-29T00:29:57.956 DEBUG:boto:Method: GET 2023-07-29T00:29:57.957 DEBUG:boto:Path: /hiiqos-59/?prefix=obj55 2023-07-29T00:29:57.957 DEBUG:boto:Data: 2023-07-29T00:29:57.957 DEBUG:boto:Headers: {} 2023-07-29T00:29:57.958 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:57.958 DEBUG:boto:Port: 8000 2023-07-29T00:29:57.958 DEBUG:boto:Params: {} 2023-07-29T00:29:57.959 DEBUG:boto:Token: None 2023-07-29T00:29:57.959 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:57 GMT /hiiqos-59/ 2023-07-29T00:29:57.960 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0= 2023-07-29T00:29:57.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:b/UDByK9h/bobNEpMp2aP5zfLg0=', 'Content-Length': '0'} 2023-07-29T00:29:57.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086b357c635c406f1-0064c45d85-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.006 DEBUG:boto:b'hiiqos-59obj551000falseobj552023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:58.008 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:29:58.008 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:29:58.008 DEBUG:boto:Method: HEAD 2023-07-29T00:29:58.009 DEBUG:boto:Path: /hiiqos-59/obj55 2023-07-29T00:29:58.009 DEBUG:boto:Data: 2023-07-29T00:29:58.010 DEBUG:boto:Headers: {} 2023-07-29T00:29:58.010 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:58.011 DEBUG:boto:Port: 8001 2023-07-29T00:29:58.011 DEBUG:boto:Params: {} 2023-07-29T00:29:58.011 DEBUG:boto:Token: None 2023-07-29T00:29:58.012 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/obj55 2023-07-29T00:29:58.012 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/QguTtVGYyjDgChjZ59lQjoRSrc= 2023-07-29T00:29:58.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/QguTtVGYyjDgChjZ59lQjoRSrc=', 'Content-Length': '0'} 2023-07-29T00:29:58.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000736343fda67e1e60-0064c45d85-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.015 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:29:58.016 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:29:58.017 DEBUG:boto:Method: HEAD 2023-07-29T00:29:58.017 DEBUG:boto:Path: /hiiqos-59/obj55 2023-07-29T00:29:58.018 DEBUG:boto:Data: 2023-07-29T00:29:58.019 DEBUG:boto:Headers: {} 2023-07-29T00:29:58.019 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:58.020 DEBUG:boto:Port: 8000 2023-07-29T00:29:58.021 DEBUG:boto:Params: {} 2023-07-29T00:29:58.022 DEBUG:boto:Token: None 2023-07-29T00:29:58.022 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/obj55 2023-07-29T00:29:58.024 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/QguTtVGYyjDgChjZ59lQjoRSrc= 2023-07-29T00:29:58.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/QguTtVGYyjDgChjZ59lQjoRSrc=', 'Content-Length': '0'} 2023-07-29T00:29:58.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070bcaef12411a08b-0064c45d85-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-07-29T00:29:58.028 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:29:58.028 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:29:58.030 DEBUG:boto:Method: GET 2023-07-29T00:29:58.030 DEBUG:boto:Path: /hiiqos-59/obj55 2023-07-29T00:29:58.031 DEBUG:boto:Data: 2023-07-29T00:29:58.032 DEBUG:boto:Headers: {} 2023-07-29T00:29:58.033 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:58.033 DEBUG:boto:Port: 8001 2023-07-29T00:29:58.034 DEBUG:boto:Params: {} 2023-07-29T00:29:58.035 DEBUG:boto:Token: None 2023-07-29T00:29:58.035 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/obj55 2023-07-29T00:29:58.035 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8KYvJch7fBEWk8qfChk3Wx/x460= 2023-07-29T00:29:58.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8KYvJch7fBEWk8qfChk3Wx/x460=', 'Content-Length': '0'} 2023-07-29T00:29:58.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074519545def9b7ca-0064c45d85-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.039 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:29:58.039 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:29:58.040 DEBUG:boto:Method: GET 2023-07-29T00:29:58.040 DEBUG:boto:Path: /hiiqos-59/obj55 2023-07-29T00:29:58.040 DEBUG:boto:Data: 2023-07-29T00:29:58.041 DEBUG:boto:Headers: {} 2023-07-29T00:29:58.041 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:58.041 DEBUG:boto:Port: 8000 2023-07-29T00:29:58.042 DEBUG:boto:Params: {} 2023-07-29T00:29:58.042 DEBUG:boto:Token: None 2023-07-29T00:29:58.043 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/obj55 2023-07-29T00:29:58.044 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8KYvJch7fBEWk8qfChk3Wx/x460= 2023-07-29T00:29:58.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8KYvJch7fBEWk8qfChk3Wx/x460=', 'Content-Length': '0'} 2023-07-29T00:29:58.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ada082fd5395841-0064c45d85-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.048 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:58.048 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:58.049 DEBUG:boto:path=/hiiqos-59/?prefix=obj55 2023-07-29T00:29:58.050 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj55 2023-07-29T00:29:58.050 DEBUG:boto:Method: GET 2023-07-29T00:29:58.051 DEBUG:boto:Path: /hiiqos-59/?prefix=obj55 2023-07-29T00:29:58.051 DEBUG:boto:Data: 2023-07-29T00:29:58.052 DEBUG:boto:Headers: {} 2023-07-29T00:29:58.053 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:58.053 DEBUG:boto:Port: 8001 2023-07-29T00:29:58.054 DEBUG:boto:Params: {} 2023-07-29T00:29:58.055 DEBUG:boto:Token: None 2023-07-29T00:29:58.055 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/ 2023-07-29T00:29:58.056 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w4ZSCSpxxPOdJVvr3TC8eoHwN0A= 2023-07-29T00:29:58.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w4ZSCSpxxPOdJVvr3TC8eoHwN0A=', 'Content-Length': '0'} 2023-07-29T00:29:58.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000053207da1a7dec2-0064c45d86-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.059 DEBUG:boto:b'hiiqos-59obj551000falseobj552023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:58.060 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:58.061 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:58.061 DEBUG:boto:path=/hiiqos-59/?prefix=obj55 2023-07-29T00:29:58.061 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj55 2023-07-29T00:29:58.062 DEBUG:boto:Method: GET 2023-07-29T00:29:58.062 DEBUG:boto:Path: /hiiqos-59/?prefix=obj55 2023-07-29T00:29:58.063 DEBUG:boto:Data: 2023-07-29T00:29:58.063 DEBUG:boto:Headers: {} 2023-07-29T00:29:58.063 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:58.064 DEBUG:boto:Port: 8000 2023-07-29T00:29:58.064 DEBUG:boto:Params: {} 2023-07-29T00:29:58.065 DEBUG:boto:Token: None 2023-07-29T00:29:58.065 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/ 2023-07-29T00:29:58.066 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w4ZSCSpxxPOdJVvr3TC8eoHwN0A= 2023-07-29T00:29:58.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w4ZSCSpxxPOdJVvr3TC8eoHwN0A=', 'Content-Length': '0'} 2023-07-29T00:29:58.068 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac85f392ea5f5fa5-0064c45d86-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.069 DEBUG:boto:b'hiiqos-59obj551000falseobj552023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:58.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-07-29T00:29:58.070 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:29:58.071 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:29:58.071 DEBUG:boto:path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:29:58.071 DEBUG:boto:auth_path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:29:58.072 DEBUG:boto:Method: GET 2023-07-29T00:29:58.072 DEBUG:boto:Path: /hiiqos-59/obj68?versionId=null 2023-07-29T00:29:58.073 DEBUG:boto:Data: 2023-07-29T00:29:58.073 DEBUG:boto:Headers: {} 2023-07-29T00:29:58.073 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:58.074 DEBUG:boto:Port: 8001 2023-07-29T00:29:58.074 DEBUG:boto:Params: {} 2023-07-29T00:29:58.074 DEBUG:boto:Token: None 2023-07-29T00:29:58.075 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/obj68?versionId=null 2023-07-29T00:29:58.075 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8frHhBsLKUcy+fJC4xbQFSQPZPU= 2023-07-29T00:29:58.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8frHhBsLKUcy+fJC4xbQFSQPZPU=', 'Content-Length': '0'} 2023-07-29T00:29:58.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c9606a188c3f4bf-0064c45d86-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.078 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:29:58.079 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:29:58.079 DEBUG:boto:path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:29:58.079 DEBUG:boto:auth_path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:29:58.080 DEBUG:boto:Method: GET 2023-07-29T00:29:58.080 DEBUG:boto:Path: /hiiqos-59/obj68?versionId=null 2023-07-29T00:29:58.081 DEBUG:boto:Data: 2023-07-29T00:29:58.081 DEBUG:boto:Headers: {} 2023-07-29T00:29:58.081 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:58.082 DEBUG:boto:Port: 8000 2023-07-29T00:29:58.082 DEBUG:boto:Params: {} 2023-07-29T00:29:58.083 DEBUG:boto:Token: None 2023-07-29T00:29:58.084 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/obj68?versionId=null 2023-07-29T00:29:58.085 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8frHhBsLKUcy+fJC4xbQFSQPZPU= 2023-07-29T00:29:58.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8frHhBsLKUcy+fJC4xbQFSQPZPU=', 'Content-Length': '0'} 2023-07-29T00:29:58.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9c7c7cf8f024ff1-0064c45d86-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.090 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:29:58.090 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:29:58.091 DEBUG:boto:path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:29:58.091 DEBUG:boto:auth_path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:29:58.092 DEBUG:boto:Method: HEAD 2023-07-29T00:29:58.093 DEBUG:boto:Path: /hiiqos-59/obj68?versionId=null 2023-07-29T00:29:58.094 DEBUG:boto:Data: 2023-07-29T00:29:58.094 DEBUG:boto:Headers: {} 2023-07-29T00:29:58.094 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:58.095 DEBUG:boto:Port: 8001 2023-07-29T00:29:58.095 DEBUG:boto:Params: {} 2023-07-29T00:29:58.095 DEBUG:boto:Token: None 2023-07-29T00:29:58.096 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/obj68?versionId=null 2023-07-29T00:29:58.096 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J3REkiI/NbZIMvC7menpE96h7ro= 2023-07-29T00:29:58.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J3REkiI/NbZIMvC7menpE96h7ro=', 'Content-Length': '0'} 2023-07-29T00:29:58.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003702dab1dcbd092d-0064c45d86-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.099 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:29:58.100 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:29:58.100 DEBUG:boto:path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:29:58.101 DEBUG:boto:auth_path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:29:58.101 DEBUG:boto:Method: HEAD 2023-07-29T00:29:58.101 DEBUG:boto:Path: /hiiqos-59/obj68?versionId=null 2023-07-29T00:29:58.102 DEBUG:boto:Data: 2023-07-29T00:29:58.102 DEBUG:boto:Headers: {} 2023-07-29T00:29:58.102 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:58.103 DEBUG:boto:Port: 8000 2023-07-29T00:29:58.103 DEBUG:boto:Params: {} 2023-07-29T00:29:58.103 DEBUG:boto:Token: None 2023-07-29T00:29:58.104 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/obj68?versionId=null 2023-07-29T00:29:58.104 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J3REkiI/NbZIMvC7menpE96h7ro= 2023-07-29T00:29:58.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J3REkiI/NbZIMvC7menpE96h7ro=', 'Content-Length': '0'} 2023-07-29T00:29:58.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000211b444a85d6d12-0064c45d86-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-07-29T00:29:58.107 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:29:58.108 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:29:58.108 DEBUG:boto:path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:29:58.109 DEBUG:boto:auth_path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:29:58.109 DEBUG:boto:Method: GET 2023-07-29T00:29:58.109 DEBUG:boto:Path: /hiiqos-59/obj68?versionId=null 2023-07-29T00:29:58.110 DEBUG:boto:Data: 2023-07-29T00:29:58.110 DEBUG:boto:Headers: {} 2023-07-29T00:29:58.110 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:58.111 DEBUG:boto:Port: 8001 2023-07-29T00:29:58.111 DEBUG:boto:Params: {} 2023-07-29T00:29:58.112 DEBUG:boto:Token: None 2023-07-29T00:29:58.112 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/obj68?versionId=null 2023-07-29T00:29:58.113 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8frHhBsLKUcy+fJC4xbQFSQPZPU= 2023-07-29T00:29:58.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8frHhBsLKUcy+fJC4xbQFSQPZPU=', 'Content-Length': '0'} 2023-07-29T00:29:58.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a00595351d0fff6-0064c45d86-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.116 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:29:58.116 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:29:58.116 DEBUG:boto:path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:29:58.117 DEBUG:boto:auth_path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:29:58.117 DEBUG:boto:Method: GET 2023-07-29T00:29:58.118 DEBUG:boto:Path: /hiiqos-59/obj68?versionId=null 2023-07-29T00:29:58.118 DEBUG:boto:Data: 2023-07-29T00:29:58.119 DEBUG:boto:Headers: {} 2023-07-29T00:29:58.119 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:58.119 DEBUG:boto:Port: 8000 2023-07-29T00:29:58.120 DEBUG:boto:Params: {} 2023-07-29T00:29:58.120 DEBUG:boto:Token: None 2023-07-29T00:29:58.121 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/obj68?versionId=null 2023-07-29T00:29:58.121 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8frHhBsLKUcy+fJC4xbQFSQPZPU= 2023-07-29T00:29:58.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8frHhBsLKUcy+fJC4xbQFSQPZPU=', 'Content-Length': '0'} 2023-07-29T00:29:58.123 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ce52a9edf0f7750-0064c45d86-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.164 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:58.165 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:58.166 DEBUG:boto:path=/hiiqos-59/?prefix=obj68 2023-07-29T00:29:58.167 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj68 2023-07-29T00:29:58.167 DEBUG:boto:Method: GET 2023-07-29T00:29:58.168 DEBUG:boto:Path: /hiiqos-59/?prefix=obj68 2023-07-29T00:29:58.169 DEBUG:boto:Data: 2023-07-29T00:29:58.169 DEBUG:boto:Headers: {} 2023-07-29T00:29:58.170 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:58.171 DEBUG:boto:Port: 8001 2023-07-29T00:29:58.171 DEBUG:boto:Params: {} 2023-07-29T00:29:58.172 DEBUG:boto:Token: None 2023-07-29T00:29:58.173 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/ 2023-07-29T00:29:58.174 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w4ZSCSpxxPOdJVvr3TC8eoHwN0A= 2023-07-29T00:29:58.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w4ZSCSpxxPOdJVvr3TC8eoHwN0A=', 'Content-Length': '0'} 2023-07-29T00:29:58.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3138bd82439af7e-0064c45d86-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.178 DEBUG:boto:b'hiiqos-59obj681000falseobj682023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:58.179 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:58.180 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:58.181 DEBUG:boto:path=/hiiqos-59/?prefix=obj68 2023-07-29T00:29:58.182 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj68 2023-07-29T00:29:58.183 DEBUG:boto:Method: GET 2023-07-29T00:29:58.183 DEBUG:boto:Path: /hiiqos-59/?prefix=obj68 2023-07-29T00:29:58.183 DEBUG:boto:Data: 2023-07-29T00:29:58.184 DEBUG:boto:Headers: {} 2023-07-29T00:29:58.184 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:58.185 DEBUG:boto:Port: 8000 2023-07-29T00:29:58.185 DEBUG:boto:Params: {} 2023-07-29T00:29:58.185 DEBUG:boto:Token: None 2023-07-29T00:29:58.186 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/ 2023-07-29T00:29:58.186 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w4ZSCSpxxPOdJVvr3TC8eoHwN0A= 2023-07-29T00:29:58.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w4ZSCSpxxPOdJVvr3TC8eoHwN0A=', 'Content-Length': '0'} 2023-07-29T00:29:58.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043761b0e6ffd47d6-0064c45d86-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.190 DEBUG:boto:b'hiiqos-59obj681000falseobj682023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:58.190 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:29:58.191 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:29:58.191 DEBUG:boto:Method: HEAD 2023-07-29T00:29:58.192 DEBUG:boto:Path: /hiiqos-59/obj68 2023-07-29T00:29:58.192 DEBUG:boto:Data: 2023-07-29T00:29:58.192 DEBUG:boto:Headers: {} 2023-07-29T00:29:58.193 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:58.193 DEBUG:boto:Port: 8001 2023-07-29T00:29:58.194 DEBUG:boto:Params: {} 2023-07-29T00:29:58.194 DEBUG:boto:Token: None 2023-07-29T00:29:58.195 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/obj68 2023-07-29T00:29:58.195 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gkCAlBEUlyyiI1Lc5LMqkaxxyaY= 2023-07-29T00:29:58.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gkCAlBEUlyyiI1Lc5LMqkaxxyaY=', 'Content-Length': '0'} 2023-07-29T00:29:58.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7b5ad0f7216b4ca-0064c45d86-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.198 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:29:58.198 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:29:58.199 DEBUG:boto:Method: HEAD 2023-07-29T00:29:58.200 DEBUG:boto:Path: /hiiqos-59/obj68 2023-07-29T00:29:58.201 DEBUG:boto:Data: 2023-07-29T00:29:58.202 DEBUG:boto:Headers: {} 2023-07-29T00:29:58.203 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:58.204 DEBUG:boto:Port: 8000 2023-07-29T00:29:58.204 DEBUG:boto:Params: {} 2023-07-29T00:29:58.205 DEBUG:boto:Token: None 2023-07-29T00:29:58.206 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/obj68 2023-07-29T00:29:58.207 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gkCAlBEUlyyiI1Lc5LMqkaxxyaY= 2023-07-29T00:29:58.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gkCAlBEUlyyiI1Lc5LMqkaxxyaY=', 'Content-Length': '0'} 2023-07-29T00:29:58.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba9214352e940487-0064c45d86-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-07-29T00:29:58.211 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:29:58.211 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:29:58.211 DEBUG:boto:Method: GET 2023-07-29T00:29:58.212 DEBUG:boto:Path: /hiiqos-59/obj68 2023-07-29T00:29:58.213 DEBUG:boto:Data: 2023-07-29T00:29:58.214 DEBUG:boto:Headers: {} 2023-07-29T00:29:58.215 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:58.216 DEBUG:boto:Port: 8001 2023-07-29T00:29:58.217 DEBUG:boto:Params: {} 2023-07-29T00:29:58.218 DEBUG:boto:Token: None 2023-07-29T00:29:58.219 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/obj68 2023-07-29T00:29:58.220 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MyqIQUSvZGAwqm/qF1cS/dkR94E= 2023-07-29T00:29:58.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MyqIQUSvZGAwqm/qF1cS/dkR94E=', 'Content-Length': '0'} 2023-07-29T00:29:58.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b12dc5f6238ade09-0064c45d86-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.223 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:29:58.224 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:29:58.224 DEBUG:boto:Method: GET 2023-07-29T00:29:58.225 DEBUG:boto:Path: /hiiqos-59/obj68 2023-07-29T00:29:58.225 DEBUG:boto:Data: 2023-07-29T00:29:58.225 DEBUG:boto:Headers: {} 2023-07-29T00:29:58.226 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:58.226 DEBUG:boto:Port: 8000 2023-07-29T00:29:58.227 DEBUG:boto:Params: {} 2023-07-29T00:29:58.227 DEBUG:boto:Token: None 2023-07-29T00:29:58.228 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/obj68 2023-07-29T00:29:58.229 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MyqIQUSvZGAwqm/qF1cS/dkR94E= 2023-07-29T00:29:58.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MyqIQUSvZGAwqm/qF1cS/dkR94E=', 'Content-Length': '0'} 2023-07-29T00:29:58.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e12f14a62f33f88-0064c45d86-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.233 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:58.234 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:58.234 DEBUG:boto:path=/hiiqos-59/?prefix=obj68 2023-07-29T00:29:58.235 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj68 2023-07-29T00:29:58.235 DEBUG:boto:Method: GET 2023-07-29T00:29:58.235 DEBUG:boto:Path: /hiiqos-59/?prefix=obj68 2023-07-29T00:29:58.236 DEBUG:boto:Data: 2023-07-29T00:29:58.236 DEBUG:boto:Headers: {} 2023-07-29T00:29:58.236 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:58.237 DEBUG:boto:Port: 8001 2023-07-29T00:29:58.237 DEBUG:boto:Params: {} 2023-07-29T00:29:58.237 DEBUG:boto:Token: None 2023-07-29T00:29:58.238 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/ 2023-07-29T00:29:58.238 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w4ZSCSpxxPOdJVvr3TC8eoHwN0A= 2023-07-29T00:29:58.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w4ZSCSpxxPOdJVvr3TC8eoHwN0A=', 'Content-Length': '0'} 2023-07-29T00:29:58.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf9c63416a02072a-0064c45d86-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.242 DEBUG:boto:b'hiiqos-59obj681000falseobj682023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:58.242 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:58.243 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:58.243 DEBUG:boto:path=/hiiqos-59/?prefix=obj68 2023-07-29T00:29:58.243 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj68 2023-07-29T00:29:58.244 DEBUG:boto:Method: GET 2023-07-29T00:29:58.244 DEBUG:boto:Path: /hiiqos-59/?prefix=obj68 2023-07-29T00:29:58.245 DEBUG:boto:Data: 2023-07-29T00:29:58.245 DEBUG:boto:Headers: {} 2023-07-29T00:29:58.245 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:58.246 DEBUG:boto:Port: 8000 2023-07-29T00:29:58.246 DEBUG:boto:Params: {} 2023-07-29T00:29:58.246 DEBUG:boto:Token: None 2023-07-29T00:29:58.247 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/ 2023-07-29T00:29:58.247 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w4ZSCSpxxPOdJVvr3TC8eoHwN0A= 2023-07-29T00:29:58.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w4ZSCSpxxPOdJVvr3TC8eoHwN0A=', 'Content-Length': '0'} 2023-07-29T00:29:58.251 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057f84895faeb1e30-0064c45d86-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.294 DEBUG:boto:b'hiiqos-59obj681000falseobj682023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:58.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-07-29T00:29:58.297 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:29:58.297 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:29:58.298 DEBUG:boto:path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:29:58.299 DEBUG:boto:auth_path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:29:58.300 DEBUG:boto:Method: GET 2023-07-29T00:29:58.301 DEBUG:boto:Path: /hiiqos-59/obj85?versionId=null 2023-07-29T00:29:58.302 DEBUG:boto:Data: 2023-07-29T00:29:58.302 DEBUG:boto:Headers: {} 2023-07-29T00:29:58.304 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:58.305 DEBUG:boto:Port: 8001 2023-07-29T00:29:58.306 DEBUG:boto:Params: {} 2023-07-29T00:29:58.308 DEBUG:boto:Token: None 2023-07-29T00:29:58.308 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/obj85?versionId=null 2023-07-29T00:29:58.310 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jt3fScUf/yIUcP1qJYTTcl2ItWA= 2023-07-29T00:29:58.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jt3fScUf/yIUcP1qJYTTcl2ItWA=', 'Content-Length': '0'} 2023-07-29T00:29:58.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a6149fe08fd0c30-0064c45d86-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.315 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:29:58.315 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:29:58.316 DEBUG:boto:path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:29:58.316 DEBUG:boto:auth_path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:29:58.317 DEBUG:boto:Method: GET 2023-07-29T00:29:58.318 DEBUG:boto:Path: /hiiqos-59/obj85?versionId=null 2023-07-29T00:29:58.318 DEBUG:boto:Data: 2023-07-29T00:29:58.319 DEBUG:boto:Headers: {} 2023-07-29T00:29:58.320 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:58.321 DEBUG:boto:Port: 8000 2023-07-29T00:29:58.322 DEBUG:boto:Params: {} 2023-07-29T00:29:58.323 DEBUG:boto:Token: None 2023-07-29T00:29:58.324 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/obj85?versionId=null 2023-07-29T00:29:58.325 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jt3fScUf/yIUcP1qJYTTcl2ItWA= 2023-07-29T00:29:58.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jt3fScUf/yIUcP1qJYTTcl2ItWA=', 'Content-Length': '0'} 2023-07-29T00:29:58.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000459fa32350a15f20-0064c45d86-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.332 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:29:58.333 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:29:58.333 DEBUG:boto:path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:29:58.334 DEBUG:boto:auth_path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:29:58.334 DEBUG:boto:Method: HEAD 2023-07-29T00:29:58.334 DEBUG:boto:Path: /hiiqos-59/obj85?versionId=null 2023-07-29T00:29:58.335 DEBUG:boto:Data: 2023-07-29T00:29:58.335 DEBUG:boto:Headers: {} 2023-07-29T00:29:58.336 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:58.336 DEBUG:boto:Port: 8001 2023-07-29T00:29:58.336 DEBUG:boto:Params: {} 2023-07-29T00:29:58.337 DEBUG:boto:Token: None 2023-07-29T00:29:58.337 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/obj85?versionId=null 2023-07-29T00:29:58.338 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lu/aJHifYxqmLe4eOGRTSLUS05M= 2023-07-29T00:29:58.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lu/aJHifYxqmLe4eOGRTSLUS05M=', 'Content-Length': '0'} 2023-07-29T00:29:58.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e580c010b983b3d0-0064c45d86-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.344 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:29:58.344 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:29:58.344 DEBUG:boto:path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:29:58.345 DEBUG:boto:auth_path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:29:58.345 DEBUG:boto:Method: HEAD 2023-07-29T00:29:58.346 DEBUG:boto:Path: /hiiqos-59/obj85?versionId=null 2023-07-29T00:29:58.346 DEBUG:boto:Data: 2023-07-29T00:29:58.346 DEBUG:boto:Headers: {} 2023-07-29T00:29:58.347 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:58.347 DEBUG:boto:Port: 8000 2023-07-29T00:29:58.347 DEBUG:boto:Params: {} 2023-07-29T00:29:58.348 DEBUG:boto:Token: None 2023-07-29T00:29:58.348 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/obj85?versionId=null 2023-07-29T00:29:58.348 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lu/aJHifYxqmLe4eOGRTSLUS05M= 2023-07-29T00:29:58.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lu/aJHifYxqmLe4eOGRTSLUS05M=', 'Content-Length': '0'} 2023-07-29T00:29:58.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033f943cf93c18116-0064c45d86-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-07-29T00:29:58.353 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:29:58.353 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:29:58.354 DEBUG:boto:path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:29:58.354 DEBUG:boto:auth_path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:29:58.355 DEBUG:boto:Method: GET 2023-07-29T00:29:58.355 DEBUG:boto:Path: /hiiqos-59/obj85?versionId=null 2023-07-29T00:29:58.355 DEBUG:boto:Data: 2023-07-29T00:29:58.356 DEBUG:boto:Headers: {} 2023-07-29T00:29:58.356 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:58.357 DEBUG:boto:Port: 8001 2023-07-29T00:29:58.358 DEBUG:boto:Params: {} 2023-07-29T00:29:58.359 DEBUG:boto:Token: None 2023-07-29T00:29:58.360 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/obj85?versionId=null 2023-07-29T00:29:58.361 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jt3fScUf/yIUcP1qJYTTcl2ItWA= 2023-07-29T00:29:58.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jt3fScUf/yIUcP1qJYTTcl2ItWA=', 'Content-Length': '0'} 2023-07-29T00:29:58.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa5fd29b487aedc0-0064c45d86-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.408 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:29:58.409 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:29:58.409 DEBUG:boto:path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:29:58.410 DEBUG:boto:auth_path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:29:58.410 DEBUG:boto:Method: GET 2023-07-29T00:29:58.411 DEBUG:boto:Path: /hiiqos-59/obj85?versionId=null 2023-07-29T00:29:58.411 DEBUG:boto:Data: 2023-07-29T00:29:58.411 DEBUG:boto:Headers: {} 2023-07-29T00:29:58.412 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:58.412 DEBUG:boto:Port: 8000 2023-07-29T00:29:58.413 DEBUG:boto:Params: {} 2023-07-29T00:29:58.413 DEBUG:boto:Token: None 2023-07-29T00:29:58.413 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/obj85?versionId=null 2023-07-29T00:29:58.414 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jt3fScUf/yIUcP1qJYTTcl2ItWA= 2023-07-29T00:29:58.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jt3fScUf/yIUcP1qJYTTcl2ItWA=', 'Content-Length': '0'} 2023-07-29T00:29:58.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d1841a1f29a96f9-0064c45d86-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.418 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:58.418 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:58.419 DEBUG:boto:path=/hiiqos-59/?prefix=obj85 2023-07-29T00:29:58.419 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj85 2023-07-29T00:29:58.419 DEBUG:boto:Method: GET 2023-07-29T00:29:58.420 DEBUG:boto:Path: /hiiqos-59/?prefix=obj85 2023-07-29T00:29:58.420 DEBUG:boto:Data: 2023-07-29T00:29:58.421 DEBUG:boto:Headers: {} 2023-07-29T00:29:58.422 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:58.422 DEBUG:boto:Port: 8001 2023-07-29T00:29:58.423 DEBUG:boto:Params: {} 2023-07-29T00:29:58.423 DEBUG:boto:Token: None 2023-07-29T00:29:58.423 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/ 2023-07-29T00:29:58.424 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w4ZSCSpxxPOdJVvr3TC8eoHwN0A= 2023-07-29T00:29:58.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w4ZSCSpxxPOdJVvr3TC8eoHwN0A=', 'Content-Length': '0'} 2023-07-29T00:29:58.427 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002aee39ce5071cee0-0064c45d86-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.428 DEBUG:boto:b'hiiqos-59obj851000falseobj852023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:58.429 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:58.430 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:58.431 DEBUG:boto:path=/hiiqos-59/?prefix=obj85 2023-07-29T00:29:58.431 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj85 2023-07-29T00:29:58.432 DEBUG:boto:Method: GET 2023-07-29T00:29:58.433 DEBUG:boto:Path: /hiiqos-59/?prefix=obj85 2023-07-29T00:29:58.434 DEBUG:boto:Data: 2023-07-29T00:29:58.434 DEBUG:boto:Headers: {} 2023-07-29T00:29:58.435 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:58.435 DEBUG:boto:Port: 8000 2023-07-29T00:29:58.435 DEBUG:boto:Params: {} 2023-07-29T00:29:58.436 DEBUG:boto:Token: None 2023-07-29T00:29:58.436 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/ 2023-07-29T00:29:58.437 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w4ZSCSpxxPOdJVvr3TC8eoHwN0A= 2023-07-29T00:29:58.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w4ZSCSpxxPOdJVvr3TC8eoHwN0A=', 'Content-Length': '0'} 2023-07-29T00:29:58.441 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9ddcf545b398b7a-0064c45d86-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.484 DEBUG:boto:b'hiiqos-59obj851000falseobj852023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:58.485 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:29:58.486 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:29:58.486 DEBUG:boto:Method: HEAD 2023-07-29T00:29:58.486 DEBUG:boto:Path: /hiiqos-59/obj85 2023-07-29T00:29:58.487 DEBUG:boto:Data: 2023-07-29T00:29:58.487 DEBUG:boto:Headers: {} 2023-07-29T00:29:58.487 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:58.488 DEBUG:boto:Port: 8001 2023-07-29T00:29:58.488 DEBUG:boto:Params: {} 2023-07-29T00:29:58.488 DEBUG:boto:Token: None 2023-07-29T00:29:58.489 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/obj85 2023-07-29T00:29:58.489 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AxddvV0W4H9aq1XSaWRDNjWSceU= 2023-07-29T00:29:58.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AxddvV0W4H9aq1XSaWRDNjWSceU=', 'Content-Length': '0'} 2023-07-29T00:29:58.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad57ef9c9b0b17bb-0064c45d86-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.492 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:29:58.493 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:29:58.493 DEBUG:boto:Method: HEAD 2023-07-29T00:29:58.493 DEBUG:boto:Path: /hiiqos-59/obj85 2023-07-29T00:29:58.494 DEBUG:boto:Data: 2023-07-29T00:29:58.494 DEBUG:boto:Headers: {} 2023-07-29T00:29:58.494 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:58.495 DEBUG:boto:Port: 8000 2023-07-29T00:29:58.495 DEBUG:boto:Params: {} 2023-07-29T00:29:58.495 DEBUG:boto:Token: None 2023-07-29T00:29:58.496 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/obj85 2023-07-29T00:29:58.496 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AxddvV0W4H9aq1XSaWRDNjWSceU= 2023-07-29T00:29:58.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AxddvV0W4H9aq1XSaWRDNjWSceU=', 'Content-Length': '0'} 2023-07-29T00:29:58.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dbfd905e5837ff5-0064c45d86-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-07-29T00:29:58.500 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:29:58.500 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:29:58.500 DEBUG:boto:Method: GET 2023-07-29T00:29:58.501 DEBUG:boto:Path: /hiiqos-59/obj85 2023-07-29T00:29:58.501 DEBUG:boto:Data: 2023-07-29T00:29:58.502 DEBUG:boto:Headers: {} 2023-07-29T00:29:58.502 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:58.502 DEBUG:boto:Port: 8001 2023-07-29T00:29:58.503 DEBUG:boto:Params: {} 2023-07-29T00:29:58.503 DEBUG:boto:Token: None 2023-07-29T00:29:58.503 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/obj85 2023-07-29T00:29:58.504 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QCsa/ubgJRsC77oXVUKoUIfw9p0= 2023-07-29T00:29:58.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QCsa/ubgJRsC77oXVUKoUIfw9p0=', 'Content-Length': '0'} 2023-07-29T00:29:58.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cca478ee517062d7-0064c45d86-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.507 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:29:58.507 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:29:58.508 DEBUG:boto:Method: GET 2023-07-29T00:29:58.508 DEBUG:boto:Path: /hiiqos-59/obj85 2023-07-29T00:29:58.508 DEBUG:boto:Data: 2023-07-29T00:29:58.509 DEBUG:boto:Headers: {} 2023-07-29T00:29:58.509 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:58.509 DEBUG:boto:Port: 8000 2023-07-29T00:29:58.510 DEBUG:boto:Params: {} 2023-07-29T00:29:58.510 DEBUG:boto:Token: None 2023-07-29T00:29:58.511 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/obj85 2023-07-29T00:29:58.511 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QCsa/ubgJRsC77oXVUKoUIfw9p0= 2023-07-29T00:29:58.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QCsa/ubgJRsC77oXVUKoUIfw9p0=', 'Content-Length': '0'} 2023-07-29T00:29:58.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b83c22460703084-0064c45d86-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.514 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:58.515 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:58.515 DEBUG:boto:path=/hiiqos-59/?prefix=obj85 2023-07-29T00:29:58.515 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj85 2023-07-29T00:29:58.516 DEBUG:boto:Method: GET 2023-07-29T00:29:58.516 DEBUG:boto:Path: /hiiqos-59/?prefix=obj85 2023-07-29T00:29:58.517 DEBUG:boto:Data: 2023-07-29T00:29:58.517 DEBUG:boto:Headers: {} 2023-07-29T00:29:58.517 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:29:58.518 DEBUG:boto:Port: 8001 2023-07-29T00:29:58.518 DEBUG:boto:Params: {} 2023-07-29T00:29:58.518 DEBUG:boto:Token: None 2023-07-29T00:29:58.519 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/ 2023-07-29T00:29:58.519 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w4ZSCSpxxPOdJVvr3TC8eoHwN0A= 2023-07-29T00:29:58.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w4ZSCSpxxPOdJVvr3TC8eoHwN0A=', 'Content-Length': '0'} 2023-07-29T00:29:58.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d14293580ea80692-0064c45d86-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.523 DEBUG:boto:b'hiiqos-59obj851000falseobj852023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:58.523 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:58.524 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:58.524 DEBUG:boto:path=/hiiqos-59/?prefix=obj85 2023-07-29T00:29:58.524 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj85 2023-07-29T00:29:58.525 DEBUG:boto:Method: GET 2023-07-29T00:29:58.525 DEBUG:boto:Path: /hiiqos-59/?prefix=obj85 2023-07-29T00:29:58.525 DEBUG:boto:Data: 2023-07-29T00:29:58.526 DEBUG:boto:Headers: {} 2023-07-29T00:29:58.526 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:58.527 DEBUG:boto:Port: 8000 2023-07-29T00:29:58.527 DEBUG:boto:Params: {} 2023-07-29T00:29:58.527 DEBUG:boto:Token: None 2023-07-29T00:29:58.528 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/ 2023-07-29T00:29:58.528 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w4ZSCSpxxPOdJVvr3TC8eoHwN0A= 2023-07-29T00:29:58.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w4ZSCSpxxPOdJVvr3TC8eoHwN0A=', 'Content-Length': '0'} 2023-07-29T00:29:58.532 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b93b45d0dff1e69-0064c45d86-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.532 DEBUG:boto:b'hiiqos-59obj851000falseobj852023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:29:58.533 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-59 zones={a2, a1} 2023-07-29T00:29:58.533 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:29:58.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:29:58.534 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:29:58.589 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:29:58.590 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:29:58.634 INFO:teuthology.orchestra.run.smithi101.stdout:tenant: 2023-07-29T00:29:58.634 INFO:teuthology.orchestra.run.smithi101.stdout:bucket name: hiiqos-59 2023-07-29T00:29:58.640 INFO:teuthology.orchestra.run.smithi101.stdout:total entries: 10 2023-07-29T00:29:58.682 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:58.683 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:58.683 DEBUG:boto:Method: HEAD 2023-07-29T00:29:58.684 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:29:58.684 DEBUG:boto:Data: 2023-07-29T00:29:58.685 DEBUG:boto:Headers: {} 2023-07-29T00:29:58.685 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:58.685 DEBUG:boto:Port: 8000 2023-07-29T00:29:58.686 DEBUG:boto:Params: {} 2023-07-29T00:29:58.686 DEBUG:boto:Token: None 2023-07-29T00:29:58.686 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/ 2023-07-29T00:29:58.687 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mvVaLDhCWd5+vZkH9SAng0mYk0c= 2023-07-29T00:29:58.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mvVaLDhCWd5+vZkH9SAng0mYk0c=', 'Content-Length': '0'} 2023-07-29T00:29:58.690 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002eccebed63f76e74-0064c45d86-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.691 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:29:58.691 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:29:58.691 DEBUG:boto:Method: PUT 2023-07-29T00:29:58.692 DEBUG:boto:Path: /hiiqos-59/obj19 2023-07-29T00:29:58.692 DEBUG:boto:Data: 2023-07-29T00:29:58.692 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:29:58.693 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:58.693 DEBUG:boto:Port: 8000 2023-07-29T00:29:58.693 DEBUG:boto:Params: {} 2023-07-29T00:29:58.694 DEBUG:boto:Token: None 2023-07-29T00:29:58.694 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/obj19 2023-07-29T00:29:58.695 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HbpUsjqD+BBbz/9XMlNSlrRAoqM= 2023-07-29T00:29:58.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HbpUsjqD+BBbz/9XMlNSlrRAoqM='} 2023-07-29T00:29:58.700 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a927a2e12dd0f2bc-0064c45d86-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.700 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:58.701 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:58.701 DEBUG:boto:Method: HEAD 2023-07-29T00:29:58.701 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:29:58.702 DEBUG:boto:Data: 2023-07-29T00:29:58.702 DEBUG:boto:Headers: {} 2023-07-29T00:29:58.703 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:58.703 DEBUG:boto:Port: 8000 2023-07-29T00:29:58.704 DEBUG:boto:Params: {} 2023-07-29T00:29:58.704 DEBUG:boto:Token: None 2023-07-29T00:29:58.705 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/ 2023-07-29T00:29:58.706 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mvVaLDhCWd5+vZkH9SAng0mYk0c= 2023-07-29T00:29:58.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mvVaLDhCWd5+vZkH9SAng0mYk0c=', 'Content-Length': '0'} 2023-07-29T00:29:58.709 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003a77e0ea3ebfc28e-0064c45d86-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.710 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:29:58.710 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:29:58.711 DEBUG:boto:Method: PUT 2023-07-29T00:29:58.712 DEBUG:boto:Path: /hiiqos-59/obj38 2023-07-29T00:29:58.712 DEBUG:boto:Data: 2023-07-29T00:29:58.713 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:29:58.713 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:58.714 DEBUG:boto:Port: 8000 2023-07-29T00:29:58.714 DEBUG:boto:Params: {} 2023-07-29T00:29:58.715 DEBUG:boto:Token: None 2023-07-29T00:29:58.716 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/obj38 2023-07-29T00:29:58.716 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Wq9i5d2P/axcEEIxU+QYlO37deU= 2023-07-29T00:29:58.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Wq9i5d2P/axcEEIxU+QYlO37deU='} 2023-07-29T00:29:58.760 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002ca0b56223464566-0064c45d86-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.761 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:58.762 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:58.762 DEBUG:boto:Method: HEAD 2023-07-29T00:29:58.762 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:29:58.763 DEBUG:boto:Data: 2023-07-29T00:29:58.763 DEBUG:boto:Headers: {} 2023-07-29T00:29:58.763 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:58.763 DEBUG:boto:Port: 8000 2023-07-29T00:29:58.764 DEBUG:boto:Params: {} 2023-07-29T00:29:58.764 DEBUG:boto:Token: None 2023-07-29T00:29:58.765 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/ 2023-07-29T00:29:58.765 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mvVaLDhCWd5+vZkH9SAng0mYk0c= 2023-07-29T00:29:58.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mvVaLDhCWd5+vZkH9SAng0mYk0c=', 'Content-Length': '0'} 2023-07-29T00:29:58.768 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003ec7fc007bc3dd56-0064c45d86-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.769 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:29:58.769 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:29:58.770 DEBUG:boto:Method: PUT 2023-07-29T00:29:58.770 DEBUG:boto:Path: /hiiqos-59/obj57 2023-07-29T00:29:58.770 DEBUG:boto:Data: 2023-07-29T00:29:58.771 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:29:58.771 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:58.772 DEBUG:boto:Port: 8000 2023-07-29T00:29:58.772 DEBUG:boto:Params: {} 2023-07-29T00:29:58.772 DEBUG:boto:Token: None 2023-07-29T00:29:58.773 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/obj57 2023-07-29T00:29:58.773 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:s+OgVRQOMXw4VmykafJPJugvP/Q= 2023-07-29T00:29:58.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:s+OgVRQOMXw4VmykafJPJugvP/Q='} 2023-07-29T00:29:58.777 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dd953801e8ceaa5a-0064c45d86-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.778 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:58.778 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:58.778 DEBUG:boto:Method: HEAD 2023-07-29T00:29:58.779 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:29:58.779 DEBUG:boto:Data: 2023-07-29T00:29:58.779 DEBUG:boto:Headers: {} 2023-07-29T00:29:58.780 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:58.780 DEBUG:boto:Port: 8000 2023-07-29T00:29:58.780 DEBUG:boto:Params: {} 2023-07-29T00:29:58.781 DEBUG:boto:Token: None 2023-07-29T00:29:58.781 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/ 2023-07-29T00:29:58.782 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mvVaLDhCWd5+vZkH9SAng0mYk0c= 2023-07-29T00:29:58.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mvVaLDhCWd5+vZkH9SAng0mYk0c=', 'Content-Length': '0'} 2023-07-29T00:29:58.784 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004f3db2ed7a669907-0064c45d86-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.785 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:29:58.785 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:29:58.786 DEBUG:boto:Method: PUT 2023-07-29T00:29:58.786 DEBUG:boto:Path: /hiiqos-59/obj76 2023-07-29T00:29:58.786 DEBUG:boto:Data: 2023-07-29T00:29:58.787 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:29:58.787 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:58.787 DEBUG:boto:Port: 8000 2023-07-29T00:29:58.788 DEBUG:boto:Params: {} 2023-07-29T00:29:58.788 DEBUG:boto:Token: None 2023-07-29T00:29:58.789 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/obj76 2023-07-29T00:29:58.789 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LiZEhokZdd9XzJOyAqswEgM36ZU= 2023-07-29T00:29:58.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LiZEhokZdd9XzJOyAqswEgM36ZU='} 2023-07-29T00:29:58.832 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000626eac2c961a3792-0064c45d86-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.833 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:29:58.834 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:29:58.834 DEBUG:boto:Method: HEAD 2023-07-29T00:29:58.834 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:29:58.835 DEBUG:boto:Data: 2023-07-29T00:29:58.835 DEBUG:boto:Headers: {} 2023-07-29T00:29:58.835 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:58.836 DEBUG:boto:Port: 8000 2023-07-29T00:29:58.836 DEBUG:boto:Params: {} 2023-07-29T00:29:58.837 DEBUG:boto:Token: None 2023-07-29T00:29:58.837 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/ 2023-07-29T00:29:58.837 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mvVaLDhCWd5+vZkH9SAng0mYk0c= 2023-07-29T00:29:58.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mvVaLDhCWd5+vZkH9SAng0mYk0c=', 'Content-Length': '0'} 2023-07-29T00:29:58.840 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003f3b7abcb700f5ca-0064c45d86-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.841 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:29:58.841 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:29:58.842 DEBUG:boto:Method: PUT 2023-07-29T00:29:58.842 DEBUG:boto:Path: /hiiqos-59/obj95 2023-07-29T00:29:58.842 DEBUG:boto:Data: 2023-07-29T00:29:58.843 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:29:58.843 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:29:58.843 DEBUG:boto:Port: 8000 2023-07-29T00:29:58.844 DEBUG:boto:Params: {} 2023-07-29T00:29:58.844 DEBUG:boto:Token: None 2023-07-29T00:29:58.845 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:29:58 GMT /hiiqos-59/obj95 2023-07-29T00:29:58.845 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Jxs5rEuZIjt/44FM2ieQmHwfKKM= 2023-07-29T00:29:58.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:29:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Jxs5rEuZIjt/44FM2ieQmHwfKKM='} 2023-07-29T00:29:58.849 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001c2a0010222cd4cf-0064c45d86-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:29:58.850 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-59 2023-07-29T00:29:58.850 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:29:58.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:29:58.851 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:29:58.903 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:29:58.904 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:29:58.937 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:29:58.880+0000 7f5be5826880 1 waiting to reach latest gen 2, on 1.. 2023-07-29T00:30:03.938 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:30:03.881+0000 7f5be5826880 1 waiting to reach latest gen 2, on 1.. 2023-07-29T00:30:08.940 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:30:08.883+0000 7f5be5826880 1 waiting for incremental sync to catch up: 2023-07-29T00:30:08.940 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2023-07-29T00:30:08.940 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.78.6, 00000000004.86.6, , , , , , , , , , , ] 2023-07-29T00:30:13.941 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:30:13.885+0000 7f5be5826880 1 waiting for incremental sync to catch up: 2023-07-29T00:30:13.942 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2023-07-29T00:30:13.942 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.78.6, 00000000004.86.6, , , , , , , , , , , ] 2023-07-29T00:30:18.943 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:30:18.886+0000 7f5be5826880 1 waiting for incremental sync to catch up: 2023-07-29T00:30:18.943 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2023-07-29T00:30:18.944 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.78.6, 00000000004.86.6, , , , , , , , , , , ] 2023-07-29T00:30:23.945 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:30:23.888+0000 7f5be5826880 1 waiting for incremental sync to catch up: 2023-07-29T00:30:23.945 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , 00000000001.80.6, , , , , , , , , , , ] 2023-07-29T00:30:23.946 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.78.6, 00000000004.86.6, , , , , , , , , , , ] 2023-07-29T00:30:28.946 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:30:28.889+0000 7f5be5826880 1 waiting for incremental sync to catch up: 2023-07-29T00:30:28.947 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , 00000000001.80.6, , , , , , , , , , , ] 2023-07-29T00:30:28.947 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.78.6, 00000000004.86.6, , , , , , , , , , , ] 2023-07-29T00:30:33.948 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:30:33.891+0000 7f5be5826880 1 waiting for incremental sync to catch up: 2023-07-29T00:30:33.948 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , 00000000001.80.6, , , , , , , , , , , ] 2023-07-29T00:30:33.948 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.78.6, 00000000004.86.6, , , , , , , , , , , ] 2023-07-29T00:30:38.949 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:30:38.892+0000 7f5be5826880 1 waiting for incremental sync to catch up: 2023-07-29T00:30:38.950 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , 00000000001.80.6, , , , , , , , , , , ] 2023-07-29T00:30:38.950 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.78.6, 00000000004.86.6, , , , , , , , , , , ] 2023-07-29T00:30:43.951 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:30:43.894+0000 7f5be5826880 1 waiting for incremental sync to catch up: 2023-07-29T00:30:43.951 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , 00000000001.80.6, , , , , , , , , , , ] 2023-07-29T00:30:43.951 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.78.6, 00000000004.86.6, , , , , , , , , , , ] 2023-07-29T00:30:48.953 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:30:48.895+0000 7f5be5826880 1 waiting for incremental sync to catch up: 2023-07-29T00:30:48.953 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.78.6, 00000000001.80.6, , , , , , , , , , , ] 2023-07-29T00:30:48.953 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.78.6, 00000000004.86.6, , , , , , , , , , , ] 2023-07-29T00:30:53.954 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:30:53.897+0000 7f5be5826880 1 waiting for incremental sync to catch up: 2023-07-29T00:30:53.955 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.78.6, 00000000001.80.6, , , , , , , , , , , ] 2023-07-29T00:30:53.955 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.78.6, 00000000004.86.6, , , , , , , , , , , ] 2023-07-29T00:30:58.956 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:30:58.899+0000 7f5be5826880 1 waiting for incremental sync to catch up: 2023-07-29T00:30:58.957 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.78.6, 00000000001.80.6, , , , , , , , , , , ] 2023-07-29T00:30:58.957 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.78.6, 00000000004.86.6, , , , , , , , , , , ] 2023-07-29T00:31:03.958 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:31:03.900+0000 7f5be5826880 1 waiting for incremental sync to catch up: 2023-07-29T00:31:03.958 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.78.6, 00000000002.82.6, , , , , , , , , , , ] 2023-07-29T00:31:03.958 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.78.6, 00000000004.86.6, , , , , , , , , , , ] 2023-07-29T00:31:08.959 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:31:08.902+0000 7f5be5826880 1 waiting for incremental sync to catch up: 2023-07-29T00:31:08.960 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.78.6, 00000000002.82.6, , , , , , , , , , , ] 2023-07-29T00:31:08.960 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.78.6, 00000000004.86.6, , , , , , , , , , , ] 2023-07-29T00:31:13.961 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:31:13.903+0000 7f5be5826880 1 waiting for incremental sync to catch up: 2023-07-29T00:31:13.962 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.78.6, 00000000002.82.6, , , , , , , , , , , ] 2023-07-29T00:31:13.962 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.78.6, 00000000004.86.6, , , , , , , , , , , ] 2023-07-29T00:31:18.112 INFO:tasks.rgw.c2.client.1.smithi139.stdout:2023-07-29T00:31:18.054+0000 7f5731a6a640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T00:31:18.962 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:31:18.904+0000 7f5be5826880 1 waiting for incremental sync to catch up: 2023-07-29T00:31:18.962 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.78.6, 00000000002.82.6, , , , , , , , , , , ] 2023-07-29T00:31:18.962 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.78.6, 00000000004.86.6, , , , , , , , , , , ] 2023-07-29T00:31:22.919 INFO:tasks.rgw.c2.client.0.smithi139.stdout:2023-07-29T00:31:22.861+0000 7f5f2671b640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T00:31:23.963 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:31:23.906+0000 7f5be5826880 1 waiting for incremental sync to catch up: 2023-07-29T00:31:23.964 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.78.6, 00000000002.82.6, , , , , , , , , , , ] 2023-07-29T00:31:23.964 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.78.6, 00000000004.86.6, , , , , , , , , , , ] 2023-07-29T00:31:28.965 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:31:28.907+0000 7f5be5826880 1 waiting for incremental sync to catch up: 2023-07-29T00:31:28.966 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.78.6, 00000000002.82.6, , , , , , , , , , , ] 2023-07-29T00:31:28.966 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.78.6, 00000000004.86.6, , , , , , , , , , , ] 2023-07-29T00:31:33.967 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:31:33.909+0000 7f5be5826880 1 waiting for incremental sync to catch up: 2023-07-29T00:31:33.967 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.78.6, 00000000002.82.6, , , , , , , , , , , ] 2023-07-29T00:31:33.967 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.78.6, 00000000004.86.6, , , , , , , , , , , ] 2023-07-29T00:31:38.968 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:31:38.910+0000 7f5be5826880 1 waiting for incremental sync to catch up: 2023-07-29T00:31:38.969 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.78.6, 00000000002.82.6, , , , , , , , , , , ] 2023-07-29T00:31:38.969 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.78.6, 00000000004.86.6, , , , , , , , , , , ] 2023-07-29T00:31:43.970 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:31:43.912+0000 7f5be5826880 1 waiting for incremental sync to catch up: 2023-07-29T00:31:43.970 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.78.6, 00000000002.82.6, , , , , , , , , , , ] 2023-07-29T00:31:43.970 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.78.6, 00000000004.86.6, , , , , , , , , , , ] 2023-07-29T00:31:48.972 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:31:48.914+0000 7f5be5826880 1 waiting for incremental sync to catch up: 2023-07-29T00:31:48.972 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.78.6, 00000000002.82.6, , , , , , , , , , , ] 2023-07-29T00:31:48.972 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.78.6, 00000000004.86.6, , , , , , , , , , , ] 2023-07-29T00:31:53.973 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:31:53.915+0000 7f5be5826880 1 waiting for incremental sync to catch up: 2023-07-29T00:31:53.974 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.78.6, 00000000002.82.6, , , , , , , , , , , ] 2023-07-29T00:31:53.974 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.78.6, 00000000004.86.6, , , , , , , , , , , ] 2023-07-29T00:31:58.975 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:31:58.917+0000 7f5be5826880 1 waiting for incremental sync to catch up: 2023-07-29T00:31:58.975 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.78.6, 00000000002.82.6, , , , , , , , , , , ] 2023-07-29T00:31:58.976 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.78.6, 00000000004.86.6, , , , , , , , , , , ] 2023-07-29T00:32:03.977 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:32:03.918+0000 7f5be5826880 1 waiting for incremental sync to catch up: 2023-07-29T00:32:03.977 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.78.6, 00000000003.84.6, , , , , , , , , , , ] 2023-07-29T00:32:03.977 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.78.6, 00000000004.86.6, , , , , , , , , , , ] 2023-07-29T00:32:08.978 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:32:08.920+0000 7f5be5826880 1 waiting for incremental sync to catch up: 2023-07-29T00:32:08.979 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.78.6, 00000000003.84.6, , , , , , , , , , , ] 2023-07-29T00:32:08.979 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.78.6, 00000000004.86.6, , , , , , , , , , , ] 2023-07-29T00:32:13.980 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:32:13.921+0000 7f5be5826880 1 waiting for incremental sync to catch up: 2023-07-29T00:32:13.980 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.78.6, 00000000003.84.6, , , , , , , , , , , ] 2023-07-29T00:32:13.981 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.78.6, 00000000004.86.6, , , , , , , , , , , ] 2023-07-29T00:32:18.981 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:32:18.923+0000 7f5be5826880 1 waiting for incremental sync to catch up: 2023-07-29T00:32:18.982 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.78.6, 00000000003.84.6, , , , , , , , , , , ] 2023-07-29T00:32:18.982 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.78.6, 00000000004.86.6, , , , , , , , , , , ] 2023-07-29T00:32:23.983 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:32:23.925+0000 7f5be5826880 1 waiting for incremental sync to catch up: 2023-07-29T00:32:23.984 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.78.6, 00000000003.84.6, , , , , , , , , , , ] 2023-07-29T00:32:23.984 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.78.6, 00000000004.86.6, , , , , , , , , , , ] 2023-07-29T00:32:28.985 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:32:28.926+0000 7f5be5826880 1 waiting for incremental sync to catch up: 2023-07-29T00:32:28.985 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.78.6, 00000000003.84.6, , , , , , , , , , , ] 2023-07-29T00:32:28.986 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.78.6, 00000000004.86.6, , , , , , , , , , , ] 2023-07-29T00:32:33.987 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:32:33.928+0000 7f5be5826880 1 waiting for incremental sync to catch up: 2023-07-29T00:32:33.987 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.78.6, 00000000003.84.6, , , , , , , , , , , ] 2023-07-29T00:32:33.987 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.78.6, 00000000004.86.6, , , , , , , , , , , ] 2023-07-29T00:32:38.989 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:32:38.930+0000 7f5be5826880 1 waiting for incremental sync to catch up: 2023-07-29T00:32:38.989 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.78.6, 00000000003.84.6, , , , , , , , , , , ] 2023-07-29T00:32:38.990 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.78.6, 00000000004.86.6, , , , , , , , , , , ] 2023-07-29T00:32:43.990 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:32:43.931+0000 7f5be5826880 1 waiting for incremental sync to catch up: 2023-07-29T00:32:43.991 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.78.6, 00000000003.84.6, , , , , , , , , , , ] 2023-07-29T00:32:43.991 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.78.6, 00000000004.86.6, , , , , , , , , , , ] 2023-07-29T00:32:48.992 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:32:48.933+0000 7f5be5826880 1 waiting for incremental sync to catch up: 2023-07-29T00:32:48.992 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.78.6, 00000000003.84.6, , , , , , , , , , , ] 2023-07-29T00:32:48.992 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.78.6, 00000000004.86.6, , , , , , , , , , , ] 2023-07-29T00:32:53.994 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:32:53.935+0000 7f5be5826880 1 waiting for incremental sync to catch up: 2023-07-29T00:32:53.994 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.78.6, 00000000003.84.6, , , , , , , , , , , ] 2023-07-29T00:32:53.994 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.78.6, 00000000004.86.6, , , , , , , , , , , ] 2023-07-29T00:32:58.995 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:32:58.936+0000 7f5be5826880 1 waiting for incremental sync to catch up: 2023-07-29T00:32:58.996 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.78.6, 00000000003.84.6, , , , , , , , , , , ] 2023-07-29T00:32:58.996 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.78.6, 00000000004.86.6, , , , , , , , , , , ] 2023-07-29T00:33:03.997 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:33:03.938+0000 7f5be5826880 1 bucket sync caught up with source: 2023-07-29T00:33:03.997 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.78.6, 00000000004.86.6, , , , , , , , , , , ] 2023-07-29T00:33:03.997 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.78.6, 00000000004.86.6, , , , , , , , , , , ] 2023-07-29T00:33:03.998 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:33:03.938+0000 7f5be5826880 0 bucket checkpoint complete 2023-07-29T00:33:04.014 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-59 2023-07-29T00:33:04.015 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:33:04.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:33:04.016 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:33:04.075 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:33:04.076 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:33:04.112 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:33:04.053+0000 7fd89d814880 1 bucket sync caught up with source: 2023-07-29T00:33:04.113 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.848.6, 00000000001.690.6, 00000000001.1200.6, 00000000001.942.6, 00000000001.729.6, , , 00000000002.295.6, 00000000004.787.6, 00000000003.1363.6] 2023-07-29T00:33:04.113 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000002.848.6, 00000000001.690.6, 00000000001.1200.6, 00000000001.942.6, 00000000001.729.6, , , 00000000002.295.6, 00000000004.787.6, 00000000003.1363.6] 2023-07-29T00:33:04.113 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:33:04.053+0000 7fd89d814880 0 bucket checkpoint complete 2023-07-29T00:33:04.128 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-59 zones={a2, a1} 2023-07-29T00:33:04.129 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:04.130 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:04.131 DEBUG:boto:Method: HEAD 2023-07-29T00:33:04.131 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:33:04.131 DEBUG:boto:Data: 2023-07-29T00:33:04.132 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.132 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:04.132 DEBUG:boto:Port: 8001 2023-07-29T00:33:04.133 DEBUG:boto:Params: {} 2023-07-29T00:33:04.133 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-29T00:33:04.134 DEBUG:boto:Token: None 2023-07-29T00:33:04.134 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/ 2023-07-29T00:33:04.135 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8sDSgK5+6PBxtzEuwQo0U/U6n08= 2023-07-29T00:33:04.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8sDSgK5+6PBxtzEuwQo0U/U6n08=', 'Content-Length': '0'} 2023-07-29T00:33:04.139 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000005ade34e9f9527aa-0064c45e40-1c08-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.139 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:04.140 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:04.140 DEBUG:boto:Method: HEAD 2023-07-29T00:33:04.140 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:33:04.141 DEBUG:boto:Data: 2023-07-29T00:33:04.141 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.142 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:04.142 DEBUG:boto:Port: 8000 2023-07-29T00:33:04.142 DEBUG:boto:Params: {} 2023-07-29T00:33:04.143 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-29T00:33:04.143 DEBUG:boto:Token: None 2023-07-29T00:33:04.144 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/ 2023-07-29T00:33:04.144 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8sDSgK5+6PBxtzEuwQo0U/U6n08= 2023-07-29T00:33:04.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8sDSgK5+6PBxtzEuwQo0U/U6n08=', 'Content-Length': '0'} 2023-07-29T00:33:04.149 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c4891e07fa042065-0064c45e40-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.150 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-29T00:33:04.150 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:04.151 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:04.151 DEBUG:boto:path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:33:04.151 DEBUG:boto:auth_path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:33:04.152 DEBUG:boto:Method: GET 2023-07-29T00:33:04.152 DEBUG:boto:Path: /hiiqos-59/?versions&max-keys=999 2023-07-29T00:33:04.152 DEBUG:boto:Data: 2023-07-29T00:33:04.153 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.153 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:04.154 DEBUG:boto:Port: 8001 2023-07-29T00:33:04.154 DEBUG:boto:Params: {} 2023-07-29T00:33:04.154 DEBUG:boto:Token: None 2023-07-29T00:33:04.155 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/?versions 2023-07-29T00:33:04.155 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1H5cny3p4x1vZHlIRJCzg6d3uJ4= 2023-07-29T00:33:04.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1H5cny3p4x1vZHlIRJCzg6d3uJ4=', 'Content-Length': '0'} 2023-07-29T00:33:04.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000af4066a89d2a72b-0064c45e40-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.159 DEBUG:boto:b'hiiqos-59999falseobj11nulltrue2023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:04.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-07-29T00:33:04.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-07-29T00:33:04.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-07-29T00:33:04.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-07-29T00:33:04.162 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-07-29T00:33:04.162 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-07-29T00:33:04.162 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-07-29T00:33:04.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-07-29T00:33:04.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-07-29T00:33:04.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-07-29T00:33:04.164 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-07-29T00:33:04.164 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-07-29T00:33:04.164 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-07-29T00:33:04.165 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-07-29T00:33:04.165 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-07-29T00:33:04.165 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-29T00:33:04.166 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:04.166 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:04.167 DEBUG:boto:path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:33:04.167 DEBUG:boto:auth_path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:33:04.167 DEBUG:boto:Method: GET 2023-07-29T00:33:04.168 DEBUG:boto:Path: /hiiqos-59/?versions&max-keys=999 2023-07-29T00:33:04.168 DEBUG:boto:Data: 2023-07-29T00:33:04.169 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.169 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:04.169 DEBUG:boto:Port: 8000 2023-07-29T00:33:04.170 DEBUG:boto:Params: {} 2023-07-29T00:33:04.170 DEBUG:boto:Token: None 2023-07-29T00:33:04.171 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/?versions 2023-07-29T00:33:04.171 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1H5cny3p4x1vZHlIRJCzg6d3uJ4= 2023-07-29T00:33:04.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1H5cny3p4x1vZHlIRJCzg6d3uJ4=', 'Content-Length': '0'} 2023-07-29T00:33:04.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000613c5e3fffafaa7b-0064c45e40-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.175 DEBUG:boto:b'hiiqos-59999falseobj11nulltrue2023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:04.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-07-29T00:33:04.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-07-29T00:33:04.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-07-29T00:33:04.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-07-29T00:33:04.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-07-29T00:33:04.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-07-29T00:33:04.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-07-29T00:33:04.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-07-29T00:33:04.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-07-29T00:33:04.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-07-29T00:33:04.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-07-29T00:33:04.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-07-29T00:33:04.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-07-29T00:33:04.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-07-29T00:33:04.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-07-29T00:33:04.182 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:04.182 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:04.182 DEBUG:boto:path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:33:04.183 DEBUG:boto:auth_path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:33:04.183 DEBUG:boto:Method: GET 2023-07-29T00:33:04.183 DEBUG:boto:Path: /hiiqos-59/?versions&max-keys=999 2023-07-29T00:33:04.184 DEBUG:boto:Data: 2023-07-29T00:33:04.184 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.185 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:04.185 DEBUG:boto:Port: 8001 2023-07-29T00:33:04.185 DEBUG:boto:Params: {} 2023-07-29T00:33:04.186 DEBUG:boto:Token: None 2023-07-29T00:33:04.186 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/?versions 2023-07-29T00:33:04.187 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1H5cny3p4x1vZHlIRJCzg6d3uJ4= 2023-07-29T00:33:04.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1H5cny3p4x1vZHlIRJCzg6d3uJ4=', 'Content-Length': '0'} 2023-07-29T00:33:04.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c74c6b5f370c0360-0064c45e40-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.190 DEBUG:boto:b'hiiqos-59999falseobj11nulltrue2023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:04.191 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:04.191 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:04.192 DEBUG:boto:path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:33:04.192 DEBUG:boto:auth_path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:33:04.193 DEBUG:boto:Method: GET 2023-07-29T00:33:04.193 DEBUG:boto:Path: /hiiqos-59/?versions&max-keys=999 2023-07-29T00:33:04.193 DEBUG:boto:Data: 2023-07-29T00:33:04.194 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.194 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:04.195 DEBUG:boto:Port: 8000 2023-07-29T00:33:04.195 DEBUG:boto:Params: {} 2023-07-29T00:33:04.195 DEBUG:boto:Token: None 2023-07-29T00:33:04.196 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/?versions 2023-07-29T00:33:04.196 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1H5cny3p4x1vZHlIRJCzg6d3uJ4= 2023-07-29T00:33:04.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1H5cny3p4x1vZHlIRJCzg6d3uJ4=', 'Content-Length': '0'} 2023-07-29T00:33:04.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065c3b61b90c46713-0064c45e40-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.200 DEBUG:boto:b'hiiqos-59999falseobj11nulltrue2023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:04.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-07-29T00:33:04.202 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:33:04.202 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:33:04.202 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:33:04.203 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:33:04.203 DEBUG:boto:Method: GET 2023-07-29T00:33:04.203 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:33:04.204 DEBUG:boto:Data: 2023-07-29T00:33:04.204 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.205 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:04.205 DEBUG:boto:Port: 8001 2023-07-29T00:33:04.205 DEBUG:boto:Params: {} 2023-07-29T00:33:04.206 DEBUG:boto:Token: None 2023-07-29T00:33:04.206 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:33:04.207 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dHVH6M/VntV/gNqJKK30yqJzTZU= 2023-07-29T00:33:04.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dHVH6M/VntV/gNqJKK30yqJzTZU=', 'Content-Length': '0'} 2023-07-29T00:33:04.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5f883c4254720ae-0064c45e40-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.210 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:33:04.210 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:33:04.211 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:33:04.211 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:33:04.211 DEBUG:boto:Method: GET 2023-07-29T00:33:04.212 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:33:04.212 DEBUG:boto:Data: 2023-07-29T00:33:04.212 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.213 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:04.213 DEBUG:boto:Port: 8000 2023-07-29T00:33:04.214 DEBUG:boto:Params: {} 2023-07-29T00:33:04.214 DEBUG:boto:Token: None 2023-07-29T00:33:04.214 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:33:04.215 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dHVH6M/VntV/gNqJKK30yqJzTZU= 2023-07-29T00:33:04.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dHVH6M/VntV/gNqJKK30yqJzTZU=', 'Content-Length': '0'} 2023-07-29T00:33:04.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f8637f10b6e8b5c-0064c45e40-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.217 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:33:04.218 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:33:04.218 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:33:04.218 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:33:04.219 DEBUG:boto:Method: HEAD 2023-07-29T00:33:04.219 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:33:04.220 DEBUG:boto:Data: 2023-07-29T00:33:04.220 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.220 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:04.221 DEBUG:boto:Port: 8001 2023-07-29T00:33:04.221 DEBUG:boto:Params: {} 2023-07-29T00:33:04.222 DEBUG:boto:Token: None 2023-07-29T00:33:04.222 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:33:04.222 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qLiPheSsOhF+DIikSDhpj5t9uJE= 2023-07-29T00:33:04.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qLiPheSsOhF+DIikSDhpj5t9uJE=', 'Content-Length': '0'} 2023-07-29T00:33:04.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1607a9ea4a55072-0064c45e40-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.225 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:33:04.225 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:33:04.226 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:33:04.226 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:33:04.226 DEBUG:boto:Method: HEAD 2023-07-29T00:33:04.227 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:33:04.228 DEBUG:boto:Data: 2023-07-29T00:33:04.229 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.229 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:04.230 DEBUG:boto:Port: 8000 2023-07-29T00:33:04.230 DEBUG:boto:Params: {} 2023-07-29T00:33:04.230 DEBUG:boto:Token: None 2023-07-29T00:33:04.231 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:33:04.232 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qLiPheSsOhF+DIikSDhpj5t9uJE= 2023-07-29T00:33:04.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qLiPheSsOhF+DIikSDhpj5t9uJE=', 'Content-Length': '0'} 2023-07-29T00:33:04.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaeab7028382f152-0064c45e40-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-07-29T00:33:04.235 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:33:04.235 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:33:04.236 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:33:04.236 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:33:04.236 DEBUG:boto:Method: GET 2023-07-29T00:33:04.237 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:33:04.237 DEBUG:boto:Data: 2023-07-29T00:33:04.238 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.238 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:04.239 DEBUG:boto:Port: 8001 2023-07-29T00:33:04.239 DEBUG:boto:Params: {} 2023-07-29T00:33:04.239 DEBUG:boto:Token: None 2023-07-29T00:33:04.240 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:33:04.242 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dHVH6M/VntV/gNqJKK30yqJzTZU= 2023-07-29T00:33:04.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dHVH6M/VntV/gNqJKK30yqJzTZU=', 'Content-Length': '0'} 2023-07-29T00:33:04.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a2415cfc5987d0f-0064c45e40-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.284 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:33:04.285 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:33:04.285 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:33:04.285 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:33:04.286 DEBUG:boto:Method: GET 2023-07-29T00:33:04.286 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:33:04.287 DEBUG:boto:Data: 2023-07-29T00:33:04.287 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.287 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:04.288 DEBUG:boto:Port: 8000 2023-07-29T00:33:04.288 DEBUG:boto:Params: {} 2023-07-29T00:33:04.289 DEBUG:boto:Token: None 2023-07-29T00:33:04.289 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:33:04.290 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dHVH6M/VntV/gNqJKK30yqJzTZU= 2023-07-29T00:33:04.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dHVH6M/VntV/gNqJKK30yqJzTZU=', 'Content-Length': '0'} 2023-07-29T00:33:04.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000463cd93df0c2d490-0064c45e40-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.292 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:04.293 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:04.293 DEBUG:boto:path=/hiiqos-59/?prefix=obj11 2023-07-29T00:33:04.294 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj11 2023-07-29T00:33:04.294 DEBUG:boto:Method: GET 2023-07-29T00:33:04.294 DEBUG:boto:Path: /hiiqos-59/?prefix=obj11 2023-07-29T00:33:04.295 DEBUG:boto:Data: 2023-07-29T00:33:04.295 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.295 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:04.296 DEBUG:boto:Port: 8001 2023-07-29T00:33:04.296 DEBUG:boto:Params: {} 2023-07-29T00:33:04.297 DEBUG:boto:Token: None 2023-07-29T00:33:04.297 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/ 2023-07-29T00:33:04.297 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:f7Wu5FE5A79hIh9pWYI+VPq2TkA= 2023-07-29T00:33:04.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:f7Wu5FE5A79hIh9pWYI+VPq2TkA=', 'Content-Length': '0'} 2023-07-29T00:33:04.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9895825c9d26d05-0064c45e40-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.301 DEBUG:boto:b'hiiqos-59obj111000falseobj112023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:04.301 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:04.302 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:04.302 DEBUG:boto:path=/hiiqos-59/?prefix=obj11 2023-07-29T00:33:04.303 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj11 2023-07-29T00:33:04.303 DEBUG:boto:Method: GET 2023-07-29T00:33:04.303 DEBUG:boto:Path: /hiiqos-59/?prefix=obj11 2023-07-29T00:33:04.304 DEBUG:boto:Data: 2023-07-29T00:33:04.304 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.305 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:04.305 DEBUG:boto:Port: 8000 2023-07-29T00:33:04.305 DEBUG:boto:Params: {} 2023-07-29T00:33:04.306 DEBUG:boto:Token: None 2023-07-29T00:33:04.306 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/ 2023-07-29T00:33:04.307 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:f7Wu5FE5A79hIh9pWYI+VPq2TkA= 2023-07-29T00:33:04.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:f7Wu5FE5A79hIh9pWYI+VPq2TkA=', 'Content-Length': '0'} 2023-07-29T00:33:04.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051abf7e99b93805d-0064c45e40-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.310 DEBUG:boto:b'hiiqos-59obj111000falseobj112023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:04.311 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:33:04.311 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:33:04.312 DEBUG:boto:Method: HEAD 2023-07-29T00:33:04.312 DEBUG:boto:Path: /hiiqos-59/obj11 2023-07-29T00:33:04.312 DEBUG:boto:Data: 2023-07-29T00:33:04.313 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.313 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:04.313 DEBUG:boto:Port: 8001 2023-07-29T00:33:04.314 DEBUG:boto:Params: {} 2023-07-29T00:33:04.314 DEBUG:boto:Token: None 2023-07-29T00:33:04.315 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj11 2023-07-29T00:33:04.315 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xTlz05PCssrsqahgA+2soRINBHs= 2023-07-29T00:33:04.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xTlz05PCssrsqahgA+2soRINBHs=', 'Content-Length': '0'} 2023-07-29T00:33:04.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f91571a7795f628c-0064c45e40-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.317 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:33:04.318 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:33:04.318 DEBUG:boto:Method: HEAD 2023-07-29T00:33:04.319 DEBUG:boto:Path: /hiiqos-59/obj11 2023-07-29T00:33:04.319 DEBUG:boto:Data: 2023-07-29T00:33:04.320 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.320 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:04.320 DEBUG:boto:Port: 8000 2023-07-29T00:33:04.321 DEBUG:boto:Params: {} 2023-07-29T00:33:04.321 DEBUG:boto:Token: None 2023-07-29T00:33:04.321 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj11 2023-07-29T00:33:04.322 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xTlz05PCssrsqahgA+2soRINBHs= 2023-07-29T00:33:04.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xTlz05PCssrsqahgA+2soRINBHs=', 'Content-Length': '0'} 2023-07-29T00:33:04.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cd7e95bd2d4ee6c-0064c45e40-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-07-29T00:33:04.325 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:33:04.325 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:33:04.325 DEBUG:boto:Method: GET 2023-07-29T00:33:04.326 DEBUG:boto:Path: /hiiqos-59/obj11 2023-07-29T00:33:04.326 DEBUG:boto:Data: 2023-07-29T00:33:04.327 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.327 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:04.327 DEBUG:boto:Port: 8001 2023-07-29T00:33:04.328 DEBUG:boto:Params: {} 2023-07-29T00:33:04.328 DEBUG:boto:Token: None 2023-07-29T00:33:04.329 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj11 2023-07-29T00:33:04.329 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JKUxN3walkElOp29fuvR9RfI8aE= 2023-07-29T00:33:04.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JKUxN3walkElOp29fuvR9RfI8aE=', 'Content-Length': '0'} 2023-07-29T00:33:04.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d078e21c1eadb41-0064c45e40-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.332 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:33:04.332 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:33:04.333 DEBUG:boto:Method: GET 2023-07-29T00:33:04.333 DEBUG:boto:Path: /hiiqos-59/obj11 2023-07-29T00:33:04.333 DEBUG:boto:Data: 2023-07-29T00:33:04.334 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.334 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:04.335 DEBUG:boto:Port: 8000 2023-07-29T00:33:04.335 DEBUG:boto:Params: {} 2023-07-29T00:33:04.335 DEBUG:boto:Token: None 2023-07-29T00:33:04.336 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj11 2023-07-29T00:33:04.336 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JKUxN3walkElOp29fuvR9RfI8aE= 2023-07-29T00:33:04.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JKUxN3walkElOp29fuvR9RfI8aE=', 'Content-Length': '0'} 2023-07-29T00:33:04.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f27f550041519e85-0064c45e40-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.384 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:04.385 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:04.385 DEBUG:boto:path=/hiiqos-59/?prefix=obj11 2023-07-29T00:33:04.386 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj11 2023-07-29T00:33:04.386 DEBUG:boto:Method: GET 2023-07-29T00:33:04.387 DEBUG:boto:Path: /hiiqos-59/?prefix=obj11 2023-07-29T00:33:04.387 DEBUG:boto:Data: 2023-07-29T00:33:04.387 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.388 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:04.388 DEBUG:boto:Port: 8001 2023-07-29T00:33:04.388 DEBUG:boto:Params: {} 2023-07-29T00:33:04.390 DEBUG:boto:Token: None 2023-07-29T00:33:04.391 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/ 2023-07-29T00:33:04.391 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:f7Wu5FE5A79hIh9pWYI+VPq2TkA= 2023-07-29T00:33:04.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:f7Wu5FE5A79hIh9pWYI+VPq2TkA=', 'Content-Length': '0'} 2023-07-29T00:33:04.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066dcdee941eb5c1a-0064c45e40-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.394 DEBUG:boto:b'hiiqos-59obj111000falseobj112023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:04.395 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:04.395 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:04.396 DEBUG:boto:path=/hiiqos-59/?prefix=obj11 2023-07-29T00:33:04.396 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj11 2023-07-29T00:33:04.396 DEBUG:boto:Method: GET 2023-07-29T00:33:04.397 DEBUG:boto:Path: /hiiqos-59/?prefix=obj11 2023-07-29T00:33:04.397 DEBUG:boto:Data: 2023-07-29T00:33:04.397 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.398 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:04.398 DEBUG:boto:Port: 8000 2023-07-29T00:33:04.399 DEBUG:boto:Params: {} 2023-07-29T00:33:04.399 DEBUG:boto:Token: None 2023-07-29T00:33:04.399 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/ 2023-07-29T00:33:04.400 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:f7Wu5FE5A79hIh9pWYI+VPq2TkA= 2023-07-29T00:33:04.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:f7Wu5FE5A79hIh9pWYI+VPq2TkA=', 'Content-Length': '0'} 2023-07-29T00:33:04.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c9ebde941122379-0064c45e40-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.404 DEBUG:boto:b'hiiqos-59obj111000falseobj112023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:04.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-07-29T00:33:04.405 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:33:04.406 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:33:04.406 DEBUG:boto:path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:33:04.406 DEBUG:boto:auth_path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:33:04.407 DEBUG:boto:Method: GET 2023-07-29T00:33:04.407 DEBUG:boto:Path: /hiiqos-59/obj17?versionId=null 2023-07-29T00:33:04.408 DEBUG:boto:Data: 2023-07-29T00:33:04.408 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.408 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:04.409 DEBUG:boto:Port: 8001 2023-07-29T00:33:04.409 DEBUG:boto:Params: {} 2023-07-29T00:33:04.410 DEBUG:boto:Token: None 2023-07-29T00:33:04.410 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj17?versionId=null 2023-07-29T00:33:04.410 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Tl9/7uBNPvasEpKprIXdSND0Ui0= 2023-07-29T00:33:04.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Tl9/7uBNPvasEpKprIXdSND0Ui0=', 'Content-Length': '0'} 2023-07-29T00:33:04.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7e951ce0e51dff9-0064c45e40-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.413 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:33:04.414 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:33:04.414 DEBUG:boto:path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:33:04.414 DEBUG:boto:auth_path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:33:04.415 DEBUG:boto:Method: GET 2023-07-29T00:33:04.415 DEBUG:boto:Path: /hiiqos-59/obj17?versionId=null 2023-07-29T00:33:04.415 DEBUG:boto:Data: 2023-07-29T00:33:04.416 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.416 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:04.416 DEBUG:boto:Port: 8000 2023-07-29T00:33:04.417 DEBUG:boto:Params: {} 2023-07-29T00:33:04.417 DEBUG:boto:Token: None 2023-07-29T00:33:04.418 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj17?versionId=null 2023-07-29T00:33:04.418 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Tl9/7uBNPvasEpKprIXdSND0Ui0= 2023-07-29T00:33:04.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Tl9/7uBNPvasEpKprIXdSND0Ui0=', 'Content-Length': '0'} 2023-07-29T00:33:04.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd761e8035c3622c-0064c45e40-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.421 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:33:04.421 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:33:04.421 DEBUG:boto:path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:33:04.422 DEBUG:boto:auth_path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:33:04.422 DEBUG:boto:Method: HEAD 2023-07-29T00:33:04.422 DEBUG:boto:Path: /hiiqos-59/obj17?versionId=null 2023-07-29T00:33:04.423 DEBUG:boto:Data: 2023-07-29T00:33:04.423 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.423 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:04.424 DEBUG:boto:Port: 8001 2023-07-29T00:33:04.424 DEBUG:boto:Params: {} 2023-07-29T00:33:04.424 DEBUG:boto:Token: None 2023-07-29T00:33:04.425 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj17?versionId=null 2023-07-29T00:33:04.425 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DwiKZMguPBjgm+TukIC4N+GKdrQ= 2023-07-29T00:33:04.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DwiKZMguPBjgm+TukIC4N+GKdrQ=', 'Content-Length': '0'} 2023-07-29T00:33:04.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbada9a33e04d386-0064c45e40-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.428 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:33:04.428 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:33:04.429 DEBUG:boto:path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:33:04.429 DEBUG:boto:auth_path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:33:04.430 DEBUG:boto:Method: HEAD 2023-07-29T00:33:04.430 DEBUG:boto:Path: /hiiqos-59/obj17?versionId=null 2023-07-29T00:33:04.430 DEBUG:boto:Data: 2023-07-29T00:33:04.431 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.431 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:04.431 DEBUG:boto:Port: 8000 2023-07-29T00:33:04.432 DEBUG:boto:Params: {} 2023-07-29T00:33:04.432 DEBUG:boto:Token: None 2023-07-29T00:33:04.432 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj17?versionId=null 2023-07-29T00:33:04.433 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DwiKZMguPBjgm+TukIC4N+GKdrQ= 2023-07-29T00:33:04.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DwiKZMguPBjgm+TukIC4N+GKdrQ=', 'Content-Length': '0'} 2023-07-29T00:33:04.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a3f2ba18997f668-0064c45e40-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-07-29T00:33:04.436 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:33:04.436 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:33:04.436 DEBUG:boto:path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:33:04.437 DEBUG:boto:auth_path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:33:04.437 DEBUG:boto:Method: GET 2023-07-29T00:33:04.437 DEBUG:boto:Path: /hiiqos-59/obj17?versionId=null 2023-07-29T00:33:04.438 DEBUG:boto:Data: 2023-07-29T00:33:04.438 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.438 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:04.439 DEBUG:boto:Port: 8001 2023-07-29T00:33:04.439 DEBUG:boto:Params: {} 2023-07-29T00:33:04.440 DEBUG:boto:Token: None 2023-07-29T00:33:04.440 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj17?versionId=null 2023-07-29T00:33:04.440 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Tl9/7uBNPvasEpKprIXdSND0Ui0= 2023-07-29T00:33:04.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Tl9/7uBNPvasEpKprIXdSND0Ui0=', 'Content-Length': '0'} 2023-07-29T00:33:04.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003368290329191be4-0064c45e40-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.488 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:33:04.489 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:33:04.489 DEBUG:boto:path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:33:04.490 DEBUG:boto:auth_path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:33:04.490 DEBUG:boto:Method: GET 2023-07-29T00:33:04.491 DEBUG:boto:Path: /hiiqos-59/obj17?versionId=null 2023-07-29T00:33:04.491 DEBUG:boto:Data: 2023-07-29T00:33:04.491 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.492 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:04.492 DEBUG:boto:Port: 8000 2023-07-29T00:33:04.492 DEBUG:boto:Params: {} 2023-07-29T00:33:04.493 DEBUG:boto:Token: None 2023-07-29T00:33:04.493 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj17?versionId=null 2023-07-29T00:33:04.494 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Tl9/7uBNPvasEpKprIXdSND0Ui0= 2023-07-29T00:33:04.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Tl9/7uBNPvasEpKprIXdSND0Ui0=', 'Content-Length': '0'} 2023-07-29T00:33:04.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adcd624656ab3b58-0064c45e40-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.497 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:04.497 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:04.498 DEBUG:boto:path=/hiiqos-59/?prefix=obj17 2023-07-29T00:33:04.498 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj17 2023-07-29T00:33:04.498 DEBUG:boto:Method: GET 2023-07-29T00:33:04.499 DEBUG:boto:Path: /hiiqos-59/?prefix=obj17 2023-07-29T00:33:04.499 DEBUG:boto:Data: 2023-07-29T00:33:04.499 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.500 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:04.500 DEBUG:boto:Port: 8001 2023-07-29T00:33:04.500 DEBUG:boto:Params: {} 2023-07-29T00:33:04.501 DEBUG:boto:Token: None 2023-07-29T00:33:04.501 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/ 2023-07-29T00:33:04.502 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:f7Wu5FE5A79hIh9pWYI+VPq2TkA= 2023-07-29T00:33:04.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:f7Wu5FE5A79hIh9pWYI+VPq2TkA=', 'Content-Length': '0'} 2023-07-29T00:33:04.504 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b5f1ac7d432d602-0064c45e40-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.505 DEBUG:boto:b'hiiqos-59obj171000falseobj172023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:04.505 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:04.506 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:04.506 DEBUG:boto:path=/hiiqos-59/?prefix=obj17 2023-07-29T00:33:04.506 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj17 2023-07-29T00:33:04.507 DEBUG:boto:Method: GET 2023-07-29T00:33:04.507 DEBUG:boto:Path: /hiiqos-59/?prefix=obj17 2023-07-29T00:33:04.508 DEBUG:boto:Data: 2023-07-29T00:33:04.508 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.508 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:04.509 DEBUG:boto:Port: 8000 2023-07-29T00:33:04.509 DEBUG:boto:Params: {} 2023-07-29T00:33:04.509 DEBUG:boto:Token: None 2023-07-29T00:33:04.510 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/ 2023-07-29T00:33:04.510 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:f7Wu5FE5A79hIh9pWYI+VPq2TkA= 2023-07-29T00:33:04.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:f7Wu5FE5A79hIh9pWYI+VPq2TkA=', 'Content-Length': '0'} 2023-07-29T00:33:04.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cbe50577221c7fd-0064c45e40-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.514 DEBUG:boto:b'hiiqos-59obj171000falseobj172023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:04.514 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:33:04.515 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:33:04.515 DEBUG:boto:Method: HEAD 2023-07-29T00:33:04.515 DEBUG:boto:Path: /hiiqos-59/obj17 2023-07-29T00:33:04.516 DEBUG:boto:Data: 2023-07-29T00:33:04.516 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.516 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:04.517 DEBUG:boto:Port: 8001 2023-07-29T00:33:04.517 DEBUG:boto:Params: {} 2023-07-29T00:33:04.517 DEBUG:boto:Token: None 2023-07-29T00:33:04.518 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj17 2023-07-29T00:33:04.518 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5rjGO7RjEtpr6XB79rjbamsHTD8= 2023-07-29T00:33:04.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5rjGO7RjEtpr6XB79rjbamsHTD8=', 'Content-Length': '0'} 2023-07-29T00:33:04.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f6235fb74040119-0064c45e40-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.521 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:33:04.522 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:33:04.522 DEBUG:boto:Method: HEAD 2023-07-29T00:33:04.522 DEBUG:boto:Path: /hiiqos-59/obj17 2023-07-29T00:33:04.523 DEBUG:boto:Data: 2023-07-29T00:33:04.523 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.523 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:04.524 DEBUG:boto:Port: 8000 2023-07-29T00:33:04.524 DEBUG:boto:Params: {} 2023-07-29T00:33:04.524 DEBUG:boto:Token: None 2023-07-29T00:33:04.525 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj17 2023-07-29T00:33:04.525 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5rjGO7RjEtpr6XB79rjbamsHTD8= 2023-07-29T00:33:04.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5rjGO7RjEtpr6XB79rjbamsHTD8=', 'Content-Length': '0'} 2023-07-29T00:33:04.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccb9f76cd761aefe-0064c45e40-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-07-29T00:33:04.528 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:33:04.529 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:33:04.529 DEBUG:boto:Method: GET 2023-07-29T00:33:04.529 DEBUG:boto:Path: /hiiqos-59/obj17 2023-07-29T00:33:04.530 DEBUG:boto:Data: 2023-07-29T00:33:04.530 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.530 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:04.531 DEBUG:boto:Port: 8001 2023-07-29T00:33:04.531 DEBUG:boto:Params: {} 2023-07-29T00:33:04.532 DEBUG:boto:Token: None 2023-07-29T00:33:04.532 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj17 2023-07-29T00:33:04.532 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1iYMznpsbsUis5wvY8G7qxxmAfM= 2023-07-29T00:33:04.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1iYMznpsbsUis5wvY8G7qxxmAfM=', 'Content-Length': '0'} 2023-07-29T00:33:04.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039f57bd9df5719c3-0064c45e40-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.535 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:33:04.535 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:33:04.536 DEBUG:boto:Method: GET 2023-07-29T00:33:04.536 DEBUG:boto:Path: /hiiqos-59/obj17 2023-07-29T00:33:04.537 DEBUG:boto:Data: 2023-07-29T00:33:04.537 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.537 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:04.538 DEBUG:boto:Port: 8000 2023-07-29T00:33:04.538 DEBUG:boto:Params: {} 2023-07-29T00:33:04.538 DEBUG:boto:Token: None 2023-07-29T00:33:04.539 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj17 2023-07-29T00:33:04.539 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1iYMznpsbsUis5wvY8G7qxxmAfM= 2023-07-29T00:33:04.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1iYMznpsbsUis5wvY8G7qxxmAfM=', 'Content-Length': '0'} 2023-07-29T00:33:04.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe4047e7e36457fe-0064c45e40-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.584 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:04.585 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:04.585 DEBUG:boto:path=/hiiqos-59/?prefix=obj17 2023-07-29T00:33:04.586 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj17 2023-07-29T00:33:04.586 DEBUG:boto:Method: GET 2023-07-29T00:33:04.586 DEBUG:boto:Path: /hiiqos-59/?prefix=obj17 2023-07-29T00:33:04.587 DEBUG:boto:Data: 2023-07-29T00:33:04.587 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.587 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:04.588 DEBUG:boto:Port: 8001 2023-07-29T00:33:04.588 DEBUG:boto:Params: {} 2023-07-29T00:33:04.588 DEBUG:boto:Token: None 2023-07-29T00:33:04.589 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/ 2023-07-29T00:33:04.589 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:f7Wu5FE5A79hIh9pWYI+VPq2TkA= 2023-07-29T00:33:04.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:f7Wu5FE5A79hIh9pWYI+VPq2TkA=', 'Content-Length': '0'} 2023-07-29T00:33:04.592 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033f4fe2a62e1f060-0064c45e40-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.592 DEBUG:boto:b'hiiqos-59obj171000falseobj172023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:04.593 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:04.594 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:04.594 DEBUG:boto:path=/hiiqos-59/?prefix=obj17 2023-07-29T00:33:04.594 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj17 2023-07-29T00:33:04.595 DEBUG:boto:Method: GET 2023-07-29T00:33:04.595 DEBUG:boto:Path: /hiiqos-59/?prefix=obj17 2023-07-29T00:33:04.595 DEBUG:boto:Data: 2023-07-29T00:33:04.596 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.596 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:04.596 DEBUG:boto:Port: 8000 2023-07-29T00:33:04.597 DEBUG:boto:Params: {} 2023-07-29T00:33:04.597 DEBUG:boto:Token: None 2023-07-29T00:33:04.598 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/ 2023-07-29T00:33:04.598 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:f7Wu5FE5A79hIh9pWYI+VPq2TkA= 2023-07-29T00:33:04.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:f7Wu5FE5A79hIh9pWYI+VPq2TkA=', 'Content-Length': '0'} 2023-07-29T00:33:04.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9c13b901359b9f1-0064c45e40-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.601 DEBUG:boto:b'hiiqos-59obj171000falseobj172023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:04.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-07-29T00:33:04.602 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:33:04.603 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:33:04.603 DEBUG:boto:path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:33:04.604 DEBUG:boto:auth_path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:33:04.604 DEBUG:boto:Method: GET 2023-07-29T00:33:04.604 DEBUG:boto:Path: /hiiqos-59/obj19?versionId=null 2023-07-29T00:33:04.605 DEBUG:boto:Data: 2023-07-29T00:33:04.605 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.605 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:04.606 DEBUG:boto:Port: 8001 2023-07-29T00:33:04.606 DEBUG:boto:Params: {} 2023-07-29T00:33:04.606 DEBUG:boto:Token: None 2023-07-29T00:33:04.607 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj19?versionId=null 2023-07-29T00:33:04.607 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MDE0NAvBrLLhu+oo8fnB3EVTR2g= 2023-07-29T00:33:04.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MDE0NAvBrLLhu+oo8fnB3EVTR2g=', 'Content-Length': '0'} 2023-07-29T00:33:04.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043d4b2dd730458b3-0064c45e40-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.610 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:33:04.611 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:33:04.611 DEBUG:boto:path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:33:04.611 DEBUG:boto:auth_path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:33:04.612 DEBUG:boto:Method: GET 2023-07-29T00:33:04.612 DEBUG:boto:Path: /hiiqos-59/obj19?versionId=null 2023-07-29T00:33:04.612 DEBUG:boto:Data: 2023-07-29T00:33:04.613 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.613 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:04.614 DEBUG:boto:Port: 8000 2023-07-29T00:33:04.614 DEBUG:boto:Params: {} 2023-07-29T00:33:04.614 DEBUG:boto:Token: None 2023-07-29T00:33:04.615 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj19?versionId=null 2023-07-29T00:33:04.615 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MDE0NAvBrLLhu+oo8fnB3EVTR2g= 2023-07-29T00:33:04.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MDE0NAvBrLLhu+oo8fnB3EVTR2g=', 'Content-Length': '0'} 2023-07-29T00:33:04.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8bd3aa2bbb82d65-0064c45e40-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.618 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:33:04.618 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:33:04.618 DEBUG:boto:path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:33:04.619 DEBUG:boto:auth_path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:33:04.619 DEBUG:boto:Method: HEAD 2023-07-29T00:33:04.619 DEBUG:boto:Path: /hiiqos-59/obj19?versionId=null 2023-07-29T00:33:04.620 DEBUG:boto:Data: 2023-07-29T00:33:04.620 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.620 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:04.621 DEBUG:boto:Port: 8001 2023-07-29T00:33:04.621 DEBUG:boto:Params: {} 2023-07-29T00:33:04.621 DEBUG:boto:Token: None 2023-07-29T00:33:04.622 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj19?versionId=null 2023-07-29T00:33:04.622 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ctcUk48rRQ12VdinqyDlovgvL7c= 2023-07-29T00:33:04.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ctcUk48rRQ12VdinqyDlovgvL7c=', 'Content-Length': '0'} 2023-07-29T00:33:04.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac3c49ad232390dd-0064c45e40-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.624 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:33:04.625 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:33:04.625 DEBUG:boto:path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:33:04.626 DEBUG:boto:auth_path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:33:04.626 DEBUG:boto:Method: HEAD 2023-07-29T00:33:04.626 DEBUG:boto:Path: /hiiqos-59/obj19?versionId=null 2023-07-29T00:33:04.626 DEBUG:boto:Data: 2023-07-29T00:33:04.627 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.627 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:04.627 DEBUG:boto:Port: 8000 2023-07-29T00:33:04.628 DEBUG:boto:Params: {} 2023-07-29T00:33:04.628 DEBUG:boto:Token: None 2023-07-29T00:33:04.629 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj19?versionId=null 2023-07-29T00:33:04.629 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ctcUk48rRQ12VdinqyDlovgvL7c= 2023-07-29T00:33:04.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ctcUk48rRQ12VdinqyDlovgvL7c=', 'Content-Length': '0'} 2023-07-29T00:33:04.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038c1a7b180cb0b52-0064c45e40-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-07-29T00:33:04.632 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:33:04.632 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:33:04.633 DEBUG:boto:path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:33:04.633 DEBUG:boto:auth_path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:33:04.633 DEBUG:boto:Method: GET 2023-07-29T00:33:04.634 DEBUG:boto:Path: /hiiqos-59/obj19?versionId=null 2023-07-29T00:33:04.634 DEBUG:boto:Data: 2023-07-29T00:33:04.634 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.635 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:04.635 DEBUG:boto:Port: 8001 2023-07-29T00:33:04.635 DEBUG:boto:Params: {} 2023-07-29T00:33:04.636 DEBUG:boto:Token: None 2023-07-29T00:33:04.636 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj19?versionId=null 2023-07-29T00:33:04.637 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MDE0NAvBrLLhu+oo8fnB3EVTR2g= 2023-07-29T00:33:04.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MDE0NAvBrLLhu+oo8fnB3EVTR2g=', 'Content-Length': '0'} 2023-07-29T00:33:04.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f649e7d70305a37-0064c45e40-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.639 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:33:04.640 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:33:04.640 DEBUG:boto:path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:33:04.641 DEBUG:boto:auth_path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:33:04.641 DEBUG:boto:Method: GET 2023-07-29T00:33:04.641 DEBUG:boto:Path: /hiiqos-59/obj19?versionId=null 2023-07-29T00:33:04.642 DEBUG:boto:Data: 2023-07-29T00:33:04.642 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.642 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:04.643 DEBUG:boto:Port: 8000 2023-07-29T00:33:04.643 DEBUG:boto:Params: {} 2023-07-29T00:33:04.643 DEBUG:boto:Token: None 2023-07-29T00:33:04.644 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj19?versionId=null 2023-07-29T00:33:04.644 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MDE0NAvBrLLhu+oo8fnB3EVTR2g= 2023-07-29T00:33:04.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MDE0NAvBrLLhu+oo8fnB3EVTR2g=', 'Content-Length': '0'} 2023-07-29T00:33:04.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006adb64e2a80cc6d2-0064c45e40-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.647 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:04.647 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:04.648 DEBUG:boto:path=/hiiqos-59/?prefix=obj19 2023-07-29T00:33:04.648 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj19 2023-07-29T00:33:04.648 DEBUG:boto:Method: GET 2023-07-29T00:33:04.649 DEBUG:boto:Path: /hiiqos-59/?prefix=obj19 2023-07-29T00:33:04.649 DEBUG:boto:Data: 2023-07-29T00:33:04.649 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.650 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:04.650 DEBUG:boto:Port: 8001 2023-07-29T00:33:04.651 DEBUG:boto:Params: {} 2023-07-29T00:33:04.651 DEBUG:boto:Token: None 2023-07-29T00:33:04.652 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/ 2023-07-29T00:33:04.652 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:f7Wu5FE5A79hIh9pWYI+VPq2TkA= 2023-07-29T00:33:04.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:f7Wu5FE5A79hIh9pWYI+VPq2TkA=', 'Content-Length': '0'} 2023-07-29T00:33:04.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007272c46f4fb3945e-0064c45e40-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.655 DEBUG:boto:b'hiiqos-59obj191000falseobj192023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:04.656 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:04.656 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:04.656 DEBUG:boto:path=/hiiqos-59/?prefix=obj19 2023-07-29T00:33:04.657 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj19 2023-07-29T00:33:04.657 DEBUG:boto:Method: GET 2023-07-29T00:33:04.657 DEBUG:boto:Path: /hiiqos-59/?prefix=obj19 2023-07-29T00:33:04.658 DEBUG:boto:Data: 2023-07-29T00:33:04.658 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.659 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:04.659 DEBUG:boto:Port: 8000 2023-07-29T00:33:04.659 DEBUG:boto:Params: {} 2023-07-29T00:33:04.660 DEBUG:boto:Token: None 2023-07-29T00:33:04.660 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/ 2023-07-29T00:33:04.660 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:f7Wu5FE5A79hIh9pWYI+VPq2TkA= 2023-07-29T00:33:04.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:f7Wu5FE5A79hIh9pWYI+VPq2TkA=', 'Content-Length': '0'} 2023-07-29T00:33:04.663 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af0bd9e529efebef-0064c45e40-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.664 DEBUG:boto:b'hiiqos-59obj191000falseobj192023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:04.665 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:33:04.665 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:33:04.665 DEBUG:boto:Method: HEAD 2023-07-29T00:33:04.666 DEBUG:boto:Path: /hiiqos-59/obj19 2023-07-29T00:33:04.666 DEBUG:boto:Data: 2023-07-29T00:33:04.666 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.667 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:04.667 DEBUG:boto:Port: 8001 2023-07-29T00:33:04.667 DEBUG:boto:Params: {} 2023-07-29T00:33:04.668 DEBUG:boto:Token: None 2023-07-29T00:33:04.668 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj19 2023-07-29T00:33:04.669 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sWTp2MHvzKSB5VFWsu74TgeeP7o= 2023-07-29T00:33:04.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sWTp2MHvzKSB5VFWsu74TgeeP7o=', 'Content-Length': '0'} 2023-07-29T00:33:04.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1517ec3356fd48f-0064c45e40-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.671 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:33:04.671 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:33:04.672 DEBUG:boto:Method: HEAD 2023-07-29T00:33:04.672 DEBUG:boto:Path: /hiiqos-59/obj19 2023-07-29T00:33:04.672 DEBUG:boto:Data: 2023-07-29T00:33:04.673 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.673 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:04.673 DEBUG:boto:Port: 8000 2023-07-29T00:33:04.674 DEBUG:boto:Params: {} 2023-07-29T00:33:04.674 DEBUG:boto:Token: None 2023-07-29T00:33:04.674 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj19 2023-07-29T00:33:04.675 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sWTp2MHvzKSB5VFWsu74TgeeP7o= 2023-07-29T00:33:04.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sWTp2MHvzKSB5VFWsu74TgeeP7o=', 'Content-Length': '0'} 2023-07-29T00:33:04.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e4675faeebf44e9-0064c45e40-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-07-29T00:33:04.678 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:33:04.678 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:33:04.678 DEBUG:boto:Method: GET 2023-07-29T00:33:04.679 DEBUG:boto:Path: /hiiqos-59/obj19 2023-07-29T00:33:04.679 DEBUG:boto:Data: 2023-07-29T00:33:04.679 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.680 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:04.680 DEBUG:boto:Port: 8001 2023-07-29T00:33:04.680 DEBUG:boto:Params: {} 2023-07-29T00:33:04.681 DEBUG:boto:Token: None 2023-07-29T00:33:04.681 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj19 2023-07-29T00:33:04.682 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hiz2jorWlre5Vdf+GQQJsL9tFE0= 2023-07-29T00:33:04.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hiz2jorWlre5Vdf+GQQJsL9tFE0=', 'Content-Length': '0'} 2023-07-29T00:33:04.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006553f895427bca3f-0064c45e40-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.684 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:33:04.685 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:33:04.685 DEBUG:boto:Method: GET 2023-07-29T00:33:04.685 DEBUG:boto:Path: /hiiqos-59/obj19 2023-07-29T00:33:04.686 DEBUG:boto:Data: 2023-07-29T00:33:04.686 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.686 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:04.687 DEBUG:boto:Port: 8000 2023-07-29T00:33:04.687 DEBUG:boto:Params: {} 2023-07-29T00:33:04.687 DEBUG:boto:Token: None 2023-07-29T00:33:04.688 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj19 2023-07-29T00:33:04.688 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hiz2jorWlre5Vdf+GQQJsL9tFE0= 2023-07-29T00:33:04.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hiz2jorWlre5Vdf+GQQJsL9tFE0=', 'Content-Length': '0'} 2023-07-29T00:33:04.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9e3f889c416efb2-0064c45e40-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.732 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:04.733 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:04.733 DEBUG:boto:path=/hiiqos-59/?prefix=obj19 2023-07-29T00:33:04.733 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj19 2023-07-29T00:33:04.734 DEBUG:boto:Method: GET 2023-07-29T00:33:04.734 DEBUG:boto:Path: /hiiqos-59/?prefix=obj19 2023-07-29T00:33:04.735 DEBUG:boto:Data: 2023-07-29T00:33:04.735 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.735 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:04.736 DEBUG:boto:Port: 8001 2023-07-29T00:33:04.736 DEBUG:boto:Params: {} 2023-07-29T00:33:04.736 DEBUG:boto:Token: None 2023-07-29T00:33:04.737 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/ 2023-07-29T00:33:04.737 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:f7Wu5FE5A79hIh9pWYI+VPq2TkA= 2023-07-29T00:33:04.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:f7Wu5FE5A79hIh9pWYI+VPq2TkA=', 'Content-Length': '0'} 2023-07-29T00:33:04.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbaa099ec216f93d-0064c45e40-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.740 DEBUG:boto:b'hiiqos-59obj191000falseobj192023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:04.741 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:04.741 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:04.742 DEBUG:boto:path=/hiiqos-59/?prefix=obj19 2023-07-29T00:33:04.742 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj19 2023-07-29T00:33:04.742 DEBUG:boto:Method: GET 2023-07-29T00:33:04.743 DEBUG:boto:Path: /hiiqos-59/?prefix=obj19 2023-07-29T00:33:04.743 DEBUG:boto:Data: 2023-07-29T00:33:04.743 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.744 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:04.744 DEBUG:boto:Port: 8000 2023-07-29T00:33:04.744 DEBUG:boto:Params: {} 2023-07-29T00:33:04.745 DEBUG:boto:Token: None 2023-07-29T00:33:04.745 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/ 2023-07-29T00:33:04.746 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:f7Wu5FE5A79hIh9pWYI+VPq2TkA= 2023-07-29T00:33:04.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:f7Wu5FE5A79hIh9pWYI+VPq2TkA=', 'Content-Length': '0'} 2023-07-29T00:33:04.749 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048db8667e53d6364-0064c45e40-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.749 DEBUG:boto:b'hiiqos-59obj191000falseobj192023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:04.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-07-29T00:33:04.750 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:33:04.751 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:33:04.751 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:33:04.751 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:33:04.752 DEBUG:boto:Method: GET 2023-07-29T00:33:04.752 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:33:04.752 DEBUG:boto:Data: 2023-07-29T00:33:04.753 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.753 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:04.753 DEBUG:boto:Port: 8001 2023-07-29T00:33:04.754 DEBUG:boto:Params: {} 2023-07-29T00:33:04.754 DEBUG:boto:Token: None 2023-07-29T00:33:04.755 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:33:04.755 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rFfIDC87I266h7CIv4FZi2KAIwI= 2023-07-29T00:33:04.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rFfIDC87I266h7CIv4FZi2KAIwI=', 'Content-Length': '0'} 2023-07-29T00:33:04.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000116483d9f686fe81-0064c45e40-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.758 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:33:04.758 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:33:04.759 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:33:04.759 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:33:04.759 DEBUG:boto:Method: GET 2023-07-29T00:33:04.760 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:33:04.760 DEBUG:boto:Data: 2023-07-29T00:33:04.760 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.761 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:04.761 DEBUG:boto:Port: 8000 2023-07-29T00:33:04.761 DEBUG:boto:Params: {} 2023-07-29T00:33:04.762 DEBUG:boto:Token: None 2023-07-29T00:33:04.762 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:33:04.763 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rFfIDC87I266h7CIv4FZi2KAIwI= 2023-07-29T00:33:04.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rFfIDC87I266h7CIv4FZi2KAIwI=', 'Content-Length': '0'} 2023-07-29T00:33:04.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000258fe5227dd98d4c-0064c45e40-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.765 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:33:04.766 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:33:04.766 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:33:04.766 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:33:04.767 DEBUG:boto:Method: HEAD 2023-07-29T00:33:04.767 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:33:04.767 DEBUG:boto:Data: 2023-07-29T00:33:04.768 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.768 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:04.768 DEBUG:boto:Port: 8001 2023-07-29T00:33:04.769 DEBUG:boto:Params: {} 2023-07-29T00:33:04.769 DEBUG:boto:Token: None 2023-07-29T00:33:04.770 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:33:04.770 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TaUZEEa4M8cMDIXXQfekxmT0/jY= 2023-07-29T00:33:04.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TaUZEEa4M8cMDIXXQfekxmT0/jY=', 'Content-Length': '0'} 2023-07-29T00:33:04.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ac141eea09cc65e-0064c45e40-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.773 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:33:04.773 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:33:04.773 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:33:04.774 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:33:04.774 DEBUG:boto:Method: HEAD 2023-07-29T00:33:04.774 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:33:04.775 DEBUG:boto:Data: 2023-07-29T00:33:04.775 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.775 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:04.776 DEBUG:boto:Port: 8000 2023-07-29T00:33:04.776 DEBUG:boto:Params: {} 2023-07-29T00:33:04.776 DEBUG:boto:Token: None 2023-07-29T00:33:04.777 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:33:04.777 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TaUZEEa4M8cMDIXXQfekxmT0/jY= 2023-07-29T00:33:04.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TaUZEEa4M8cMDIXXQfekxmT0/jY=', 'Content-Length': '0'} 2023-07-29T00:33:04.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084cf856215400819-0064c45e40-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-07-29T00:33:04.780 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:33:04.780 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:33:04.781 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:33:04.781 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:33:04.781 DEBUG:boto:Method: GET 2023-07-29T00:33:04.782 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:33:04.782 DEBUG:boto:Data: 2023-07-29T00:33:04.782 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.783 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:04.783 DEBUG:boto:Port: 8001 2023-07-29T00:33:04.783 DEBUG:boto:Params: {} 2023-07-29T00:33:04.784 DEBUG:boto:Token: None 2023-07-29T00:33:04.784 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:33:04.785 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rFfIDC87I266h7CIv4FZi2KAIwI= 2023-07-29T00:33:04.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rFfIDC87I266h7CIv4FZi2KAIwI=', 'Content-Length': '0'} 2023-07-29T00:33:04.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a42eabf75788880-0064c45e40-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.828 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:33:04.829 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:33:04.829 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:33:04.829 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:33:04.830 DEBUG:boto:Method: GET 2023-07-29T00:33:04.830 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:33:04.831 DEBUG:boto:Data: 2023-07-29T00:33:04.831 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.831 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:04.832 DEBUG:boto:Port: 8000 2023-07-29T00:33:04.832 DEBUG:boto:Params: {} 2023-07-29T00:33:04.832 DEBUG:boto:Token: None 2023-07-29T00:33:04.833 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:33:04.833 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rFfIDC87I266h7CIv4FZi2KAIwI= 2023-07-29T00:33:04.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rFfIDC87I266h7CIv4FZi2KAIwI=', 'Content-Length': '0'} 2023-07-29T00:33:04.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbac77ddc134b1fd-0064c45e40-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.836 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:04.836 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:04.837 DEBUG:boto:path=/hiiqos-59/?prefix=obj22 2023-07-29T00:33:04.837 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj22 2023-07-29T00:33:04.837 DEBUG:boto:Method: GET 2023-07-29T00:33:04.838 DEBUG:boto:Path: /hiiqos-59/?prefix=obj22 2023-07-29T00:33:04.838 DEBUG:boto:Data: 2023-07-29T00:33:04.839 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.839 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:04.839 DEBUG:boto:Port: 8001 2023-07-29T00:33:04.840 DEBUG:boto:Params: {} 2023-07-29T00:33:04.840 DEBUG:boto:Token: None 2023-07-29T00:33:04.840 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/ 2023-07-29T00:33:04.841 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:f7Wu5FE5A79hIh9pWYI+VPq2TkA= 2023-07-29T00:33:04.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:f7Wu5FE5A79hIh9pWYI+VPq2TkA=', 'Content-Length': '0'} 2023-07-29T00:33:04.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042a400e5afdefdca-0064c45e40-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.844 DEBUG:boto:b'hiiqos-59obj221000falseobj222023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:04.844 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:04.845 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:04.845 DEBUG:boto:path=/hiiqos-59/?prefix=obj22 2023-07-29T00:33:04.845 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj22 2023-07-29T00:33:04.846 DEBUG:boto:Method: GET 2023-07-29T00:33:04.846 DEBUG:boto:Path: /hiiqos-59/?prefix=obj22 2023-07-29T00:33:04.847 DEBUG:boto:Data: 2023-07-29T00:33:04.847 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.847 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:04.848 DEBUG:boto:Port: 8000 2023-07-29T00:33:04.848 DEBUG:boto:Params: {} 2023-07-29T00:33:04.848 DEBUG:boto:Token: None 2023-07-29T00:33:04.849 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/ 2023-07-29T00:33:04.849 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:f7Wu5FE5A79hIh9pWYI+VPq2TkA= 2023-07-29T00:33:04.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:f7Wu5FE5A79hIh9pWYI+VPq2TkA=', 'Content-Length': '0'} 2023-07-29T00:33:04.852 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000381dd55dfccf8407-0064c45e40-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.853 DEBUG:boto:b'hiiqos-59obj221000falseobj222023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:04.853 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:33:04.854 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:33:04.854 DEBUG:boto:Method: HEAD 2023-07-29T00:33:04.854 DEBUG:boto:Path: /hiiqos-59/obj22 2023-07-29T00:33:04.855 DEBUG:boto:Data: 2023-07-29T00:33:04.855 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.855 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:04.856 DEBUG:boto:Port: 8001 2023-07-29T00:33:04.856 DEBUG:boto:Params: {} 2023-07-29T00:33:04.856 DEBUG:boto:Token: None 2023-07-29T00:33:04.857 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj22 2023-07-29T00:33:04.857 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:c6omcUzJIo7klDmJvwts9CGsP84= 2023-07-29T00:33:04.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c6omcUzJIo7klDmJvwts9CGsP84=', 'Content-Length': '0'} 2023-07-29T00:33:04.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004676d5142d6422a7-0064c45e40-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.860 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:33:04.860 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:33:04.860 DEBUG:boto:Method: HEAD 2023-07-29T00:33:04.861 DEBUG:boto:Path: /hiiqos-59/obj22 2023-07-29T00:33:04.861 DEBUG:boto:Data: 2023-07-29T00:33:04.861 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.862 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:04.862 DEBUG:boto:Port: 8000 2023-07-29T00:33:04.862 DEBUG:boto:Params: {} 2023-07-29T00:33:04.863 DEBUG:boto:Token: None 2023-07-29T00:33:04.863 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj22 2023-07-29T00:33:04.864 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:c6omcUzJIo7klDmJvwts9CGsP84= 2023-07-29T00:33:04.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c6omcUzJIo7klDmJvwts9CGsP84=', 'Content-Length': '0'} 2023-07-29T00:33:04.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000919ab042beeb9d43-0064c45e40-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-07-29T00:33:04.867 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:33:04.867 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:33:04.868 DEBUG:boto:Method: GET 2023-07-29T00:33:04.868 DEBUG:boto:Path: /hiiqos-59/obj22 2023-07-29T00:33:04.868 DEBUG:boto:Data: 2023-07-29T00:33:04.869 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.869 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:04.869 DEBUG:boto:Port: 8001 2023-07-29T00:33:04.870 DEBUG:boto:Params: {} 2023-07-29T00:33:04.870 DEBUG:boto:Token: None 2023-07-29T00:33:04.871 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj22 2023-07-29T00:33:04.871 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rirGyKurdY+SH4zDJFjMQwriPFU= 2023-07-29T00:33:04.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rirGyKurdY+SH4zDJFjMQwriPFU=', 'Content-Length': '0'} 2023-07-29T00:33:04.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b8f243236e6ac04-0064c45e40-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.874 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:33:04.874 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:33:04.874 DEBUG:boto:Method: GET 2023-07-29T00:33:04.875 DEBUG:boto:Path: /hiiqos-59/obj22 2023-07-29T00:33:04.875 DEBUG:boto:Data: 2023-07-29T00:33:04.876 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.876 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:04.876 DEBUG:boto:Port: 8000 2023-07-29T00:33:04.877 DEBUG:boto:Params: {} 2023-07-29T00:33:04.877 DEBUG:boto:Token: None 2023-07-29T00:33:04.877 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj22 2023-07-29T00:33:04.878 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rirGyKurdY+SH4zDJFjMQwriPFU= 2023-07-29T00:33:04.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rirGyKurdY+SH4zDJFjMQwriPFU=', 'Content-Length': '0'} 2023-07-29T00:33:04.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd05fbfe33b1e8bd-0064c45e40-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.880 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:04.881 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:04.881 DEBUG:boto:path=/hiiqos-59/?prefix=obj22 2023-07-29T00:33:04.882 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj22 2023-07-29T00:33:04.882 DEBUG:boto:Method: GET 2023-07-29T00:33:04.882 DEBUG:boto:Path: /hiiqos-59/?prefix=obj22 2023-07-29T00:33:04.883 DEBUG:boto:Data: 2023-07-29T00:33:04.883 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.883 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:04.884 DEBUG:boto:Port: 8001 2023-07-29T00:33:04.884 DEBUG:boto:Params: {} 2023-07-29T00:33:04.884 DEBUG:boto:Token: None 2023-07-29T00:33:04.885 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/ 2023-07-29T00:33:04.885 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:f7Wu5FE5A79hIh9pWYI+VPq2TkA= 2023-07-29T00:33:04.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:f7Wu5FE5A79hIh9pWYI+VPq2TkA=', 'Content-Length': '0'} 2023-07-29T00:33:04.888 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003404aabb6cb8055a-0064c45e40-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.888 DEBUG:boto:b'hiiqos-59obj221000falseobj222023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:04.889 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:04.889 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:04.889 DEBUG:boto:path=/hiiqos-59/?prefix=obj22 2023-07-29T00:33:04.890 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj22 2023-07-29T00:33:04.890 DEBUG:boto:Method: GET 2023-07-29T00:33:04.891 DEBUG:boto:Path: /hiiqos-59/?prefix=obj22 2023-07-29T00:33:04.891 DEBUG:boto:Data: 2023-07-29T00:33:04.891 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.892 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:04.892 DEBUG:boto:Port: 8000 2023-07-29T00:33:04.892 DEBUG:boto:Params: {} 2023-07-29T00:33:04.893 DEBUG:boto:Token: None 2023-07-29T00:33:04.893 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/ 2023-07-29T00:33:04.894 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:f7Wu5FE5A79hIh9pWYI+VPq2TkA= 2023-07-29T00:33:04.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:f7Wu5FE5A79hIh9pWYI+VPq2TkA=', 'Content-Length': '0'} 2023-07-29T00:33:04.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000021fdb236a5884f1-0064c45e40-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.897 DEBUG:boto:b'hiiqos-59obj221000falseobj222023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:04.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-07-29T00:33:04.898 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:33:04.899 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:33:04.899 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:33:04.899 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:33:04.900 DEBUG:boto:Method: GET 2023-07-29T00:33:04.900 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:33:04.900 DEBUG:boto:Data: 2023-07-29T00:33:04.901 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.901 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:04.901 DEBUG:boto:Port: 8001 2023-07-29T00:33:04.902 DEBUG:boto:Params: {} 2023-07-29T00:33:04.902 DEBUG:boto:Token: None 2023-07-29T00:33:04.903 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:33:04.903 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/DOYQcgWBKUymcOutyXZHka2osw= 2023-07-29T00:33:04.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/DOYQcgWBKUymcOutyXZHka2osw=', 'Content-Length': '0'} 2023-07-29T00:33:04.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061cc70326283b102-0064c45e40-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.906 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:33:04.906 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:33:04.907 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:33:04.907 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:33:04.907 DEBUG:boto:Method: GET 2023-07-29T00:33:04.908 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:33:04.908 DEBUG:boto:Data: 2023-07-29T00:33:04.908 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.909 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:04.909 DEBUG:boto:Port: 8000 2023-07-29T00:33:04.909 DEBUG:boto:Params: {} 2023-07-29T00:33:04.910 DEBUG:boto:Token: None 2023-07-29T00:33:04.910 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:33:04.911 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/DOYQcgWBKUymcOutyXZHka2osw= 2023-07-29T00:33:04.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/DOYQcgWBKUymcOutyXZHka2osw=', 'Content-Length': '0'} 2023-07-29T00:33:04.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008942cafc5b303ad3-0064c45e40-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.913 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:33:04.914 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:33:04.914 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:33:04.914 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:33:04.915 DEBUG:boto:Method: HEAD 2023-07-29T00:33:04.915 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:33:04.915 DEBUG:boto:Data: 2023-07-29T00:33:04.916 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.916 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:04.916 DEBUG:boto:Port: 8001 2023-07-29T00:33:04.917 DEBUG:boto:Params: {} 2023-07-29T00:33:04.917 DEBUG:boto:Token: None 2023-07-29T00:33:04.917 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:33:04.918 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:j9l6uFNBbTkXLIg2azaCeMlkUjk= 2023-07-29T00:33:04.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:j9l6uFNBbTkXLIg2azaCeMlkUjk=', 'Content-Length': '0'} 2023-07-29T00:33:04.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000101a2f3e8dbef5bb-0064c45e40-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.920 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:33:04.921 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:33:04.921 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:33:04.922 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:33:04.922 DEBUG:boto:Method: HEAD 2023-07-29T00:33:04.922 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:33:04.923 DEBUG:boto:Data: 2023-07-29T00:33:04.923 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.923 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:04.923 DEBUG:boto:Port: 8000 2023-07-29T00:33:04.924 DEBUG:boto:Params: {} 2023-07-29T00:33:04.924 DEBUG:boto:Token: None 2023-07-29T00:33:04.925 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:33:04.925 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:j9l6uFNBbTkXLIg2azaCeMlkUjk= 2023-07-29T00:33:04.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:j9l6uFNBbTkXLIg2azaCeMlkUjk=', 'Content-Length': '0'} 2023-07-29T00:33:04.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000235e7d65937bd76a-0064c45e40-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-07-29T00:33:04.928 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:33:04.928 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:33:04.929 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:33:04.929 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:33:04.929 DEBUG:boto:Method: GET 2023-07-29T00:33:04.930 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:33:04.930 DEBUG:boto:Data: 2023-07-29T00:33:04.930 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.931 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:04.931 DEBUG:boto:Port: 8001 2023-07-29T00:33:04.931 DEBUG:boto:Params: {} 2023-07-29T00:33:04.932 DEBUG:boto:Token: None 2023-07-29T00:33:04.932 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:33:04.933 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/DOYQcgWBKUymcOutyXZHka2osw= 2023-07-29T00:33:04.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/DOYQcgWBKUymcOutyXZHka2osw=', 'Content-Length': '0'} 2023-07-29T00:33:04.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007874a64c82d2fb69-0064c45e40-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.935 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:33:04.936 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:33:04.936 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:33:04.936 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:33:04.937 DEBUG:boto:Method: GET 2023-07-29T00:33:04.937 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:33:04.938 DEBUG:boto:Data: 2023-07-29T00:33:04.938 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.938 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:04.939 DEBUG:boto:Port: 8000 2023-07-29T00:33:04.939 DEBUG:boto:Params: {} 2023-07-29T00:33:04.939 DEBUG:boto:Token: None 2023-07-29T00:33:04.940 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:33:04.940 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/DOYQcgWBKUymcOutyXZHka2osw= 2023-07-29T00:33:04.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/DOYQcgWBKUymcOutyXZHka2osw=', 'Content-Length': '0'} 2023-07-29T00:33:04.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa4684f06e6d51df-0064c45e40-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.984 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:04.985 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:04.985 DEBUG:boto:path=/hiiqos-59/?prefix=obj33 2023-07-29T00:33:04.986 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj33 2023-07-29T00:33:04.986 DEBUG:boto:Method: GET 2023-07-29T00:33:04.986 DEBUG:boto:Path: /hiiqos-59/?prefix=obj33 2023-07-29T00:33:04.987 DEBUG:boto:Data: 2023-07-29T00:33:04.987 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.987 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:04.988 DEBUG:boto:Port: 8001 2023-07-29T00:33:04.988 DEBUG:boto:Params: {} 2023-07-29T00:33:04.989 DEBUG:boto:Token: None 2023-07-29T00:33:04.989 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/ 2023-07-29T00:33:04.989 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:f7Wu5FE5A79hIh9pWYI+VPq2TkA= 2023-07-29T00:33:04.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:f7Wu5FE5A79hIh9pWYI+VPq2TkA=', 'Content-Length': '0'} 2023-07-29T00:33:04.992 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d45d57fa2703c695-0064c45e40-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:04.993 DEBUG:boto:b'hiiqos-59obj331000falseobj332023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:04.993 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:04.994 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:04.994 DEBUG:boto:path=/hiiqos-59/?prefix=obj33 2023-07-29T00:33:04.995 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj33 2023-07-29T00:33:04.995 DEBUG:boto:Method: GET 2023-07-29T00:33:04.995 DEBUG:boto:Path: /hiiqos-59/?prefix=obj33 2023-07-29T00:33:04.996 DEBUG:boto:Data: 2023-07-29T00:33:04.996 DEBUG:boto:Headers: {} 2023-07-29T00:33:04.996 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:04.997 DEBUG:boto:Port: 8000 2023-07-29T00:33:04.997 DEBUG:boto:Params: {} 2023-07-29T00:33:04.997 DEBUG:boto:Token: None 2023-07-29T00:33:04.998 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:04 GMT /hiiqos-59/ 2023-07-29T00:33:04.998 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:f7Wu5FE5A79hIh9pWYI+VPq2TkA= 2023-07-29T00:33:04.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:f7Wu5FE5A79hIh9pWYI+VPq2TkA=', 'Content-Length': '0'} 2023-07-29T00:33:05.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d801cf275d8c518-0064c45e40-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.002 DEBUG:boto:b'hiiqos-59obj331000falseobj332023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:05.003 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:33:05.003 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:33:05.003 DEBUG:boto:Method: HEAD 2023-07-29T00:33:05.004 DEBUG:boto:Path: /hiiqos-59/obj33 2023-07-29T00:33:05.004 DEBUG:boto:Data: 2023-07-29T00:33:05.004 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.005 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:05.005 DEBUG:boto:Port: 8001 2023-07-29T00:33:05.005 DEBUG:boto:Params: {} 2023-07-29T00:33:05.006 DEBUG:boto:Token: None 2023-07-29T00:33:05.006 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj33 2023-07-29T00:33:05.007 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aLWvwEPYHZmbHO8jsIvNqfVEMB4= 2023-07-29T00:33:05.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aLWvwEPYHZmbHO8jsIvNqfVEMB4=', 'Content-Length': '0'} 2023-07-29T00:33:05.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2880a17788059bb-0064c45e40-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.009 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:33:05.010 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:33:05.010 DEBUG:boto:Method: HEAD 2023-07-29T00:33:05.010 DEBUG:boto:Path: /hiiqos-59/obj33 2023-07-29T00:33:05.011 DEBUG:boto:Data: 2023-07-29T00:33:05.011 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.011 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:05.012 DEBUG:boto:Port: 8000 2023-07-29T00:33:05.012 DEBUG:boto:Params: {} 2023-07-29T00:33:05.012 DEBUG:boto:Token: None 2023-07-29T00:33:05.013 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj33 2023-07-29T00:33:05.013 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aLWvwEPYHZmbHO8jsIvNqfVEMB4= 2023-07-29T00:33:05.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aLWvwEPYHZmbHO8jsIvNqfVEMB4=', 'Content-Length': '0'} 2023-07-29T00:33:05.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0f45bb3b19325ad-0064c45e40-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-07-29T00:33:05.016 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:33:05.016 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:33:05.017 DEBUG:boto:Method: GET 2023-07-29T00:33:05.017 DEBUG:boto:Path: /hiiqos-59/obj33 2023-07-29T00:33:05.017 DEBUG:boto:Data: 2023-07-29T00:33:05.018 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.018 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:05.018 DEBUG:boto:Port: 8001 2023-07-29T00:33:05.019 DEBUG:boto:Params: {} 2023-07-29T00:33:05.019 DEBUG:boto:Token: None 2023-07-29T00:33:05.020 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj33 2023-07-29T00:33:05.020 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:St1376neGCC/cZ9kRtUI+RpUBHY= 2023-07-29T00:33:05.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:St1376neGCC/cZ9kRtUI+RpUBHY=', 'Content-Length': '0'} 2023-07-29T00:33:05.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000914344c9e9a7a0ef-0064c45e40-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.064 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:33:05.065 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:33:05.065 DEBUG:boto:Method: GET 2023-07-29T00:33:05.066 DEBUG:boto:Path: /hiiqos-59/obj33 2023-07-29T00:33:05.066 DEBUG:boto:Data: 2023-07-29T00:33:05.066 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.067 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:05.067 DEBUG:boto:Port: 8000 2023-07-29T00:33:05.067 DEBUG:boto:Params: {} 2023-07-29T00:33:05.068 DEBUG:boto:Token: None 2023-07-29T00:33:05.068 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj33 2023-07-29T00:33:05.069 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:St1376neGCC/cZ9kRtUI+RpUBHY= 2023-07-29T00:33:05.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:St1376neGCC/cZ9kRtUI+RpUBHY=', 'Content-Length': '0'} 2023-07-29T00:33:05.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e79ad5102dcc8653-0064c45e41-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.072 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:05.072 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:05.073 DEBUG:boto:path=/hiiqos-59/?prefix=obj33 2023-07-29T00:33:05.073 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj33 2023-07-29T00:33:05.073 DEBUG:boto:Method: GET 2023-07-29T00:33:05.074 DEBUG:boto:Path: /hiiqos-59/?prefix=obj33 2023-07-29T00:33:05.074 DEBUG:boto:Data: 2023-07-29T00:33:05.074 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.075 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:05.075 DEBUG:boto:Port: 8001 2023-07-29T00:33:05.075 DEBUG:boto:Params: {} 2023-07-29T00:33:05.076 DEBUG:boto:Token: None 2023-07-29T00:33:05.076 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/ 2023-07-29T00:33:05.077 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs= 2023-07-29T00:33:05.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs=', 'Content-Length': '0'} 2023-07-29T00:33:05.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000044aac1d73a91f14-0064c45e41-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.080 DEBUG:boto:b'hiiqos-59obj331000falseobj332023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:05.081 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:05.081 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:05.081 DEBUG:boto:path=/hiiqos-59/?prefix=obj33 2023-07-29T00:33:05.082 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj33 2023-07-29T00:33:05.082 DEBUG:boto:Method: GET 2023-07-29T00:33:05.082 DEBUG:boto:Path: /hiiqos-59/?prefix=obj33 2023-07-29T00:33:05.083 DEBUG:boto:Data: 2023-07-29T00:33:05.083 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.084 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:05.084 DEBUG:boto:Port: 8000 2023-07-29T00:33:05.084 DEBUG:boto:Params: {} 2023-07-29T00:33:05.085 DEBUG:boto:Token: None 2023-07-29T00:33:05.085 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/ 2023-07-29T00:33:05.086 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs= 2023-07-29T00:33:05.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs=', 'Content-Length': '0'} 2023-07-29T00:33:05.089 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077082d619797c505-0064c45e41-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.089 DEBUG:boto:b'hiiqos-59obj331000falseobj332023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:05.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-07-29T00:33:05.090 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:33:05.091 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:33:05.091 DEBUG:boto:path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:33:05.091 DEBUG:boto:auth_path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:33:05.092 DEBUG:boto:Method: GET 2023-07-29T00:33:05.092 DEBUG:boto:Path: /hiiqos-59/obj34?versionId=null 2023-07-29T00:33:05.092 DEBUG:boto:Data: 2023-07-29T00:33:05.093 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.093 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:05.093 DEBUG:boto:Port: 8001 2023-07-29T00:33:05.094 DEBUG:boto:Params: {} 2023-07-29T00:33:05.094 DEBUG:boto:Token: None 2023-07-29T00:33:05.095 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj34?versionId=null 2023-07-29T00:33:05.095 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:C/v+FdPrwxaB9UiHibJvDvAP7S8= 2023-07-29T00:33:05.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:C/v+FdPrwxaB9UiHibJvDvAP7S8=', 'Content-Length': '0'} 2023-07-29T00:33:05.097 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc675c13da01a839-0064c45e41-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.098 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:33:05.098 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:33:05.099 DEBUG:boto:path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:33:05.099 DEBUG:boto:auth_path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:33:05.099 DEBUG:boto:Method: GET 2023-07-29T00:33:05.100 DEBUG:boto:Path: /hiiqos-59/obj34?versionId=null 2023-07-29T00:33:05.100 DEBUG:boto:Data: 2023-07-29T00:33:05.100 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.101 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:05.101 DEBUG:boto:Port: 8000 2023-07-29T00:33:05.101 DEBUG:boto:Params: {} 2023-07-29T00:33:05.102 DEBUG:boto:Token: None 2023-07-29T00:33:05.102 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj34?versionId=null 2023-07-29T00:33:05.103 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:C/v+FdPrwxaB9UiHibJvDvAP7S8= 2023-07-29T00:33:05.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:C/v+FdPrwxaB9UiHibJvDvAP7S8=', 'Content-Length': '0'} 2023-07-29T00:33:05.105 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cefa866c8737c5a8-0064c45e41-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.148 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:33:05.149 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:33:05.149 DEBUG:boto:path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:33:05.150 DEBUG:boto:auth_path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:33:05.150 DEBUG:boto:Method: HEAD 2023-07-29T00:33:05.150 DEBUG:boto:Path: /hiiqos-59/obj34?versionId=null 2023-07-29T00:33:05.151 DEBUG:boto:Data: 2023-07-29T00:33:05.151 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.152 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:05.152 DEBUG:boto:Port: 8001 2023-07-29T00:33:05.152 DEBUG:boto:Params: {} 2023-07-29T00:33:05.153 DEBUG:boto:Token: None 2023-07-29T00:33:05.153 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj34?versionId=null 2023-07-29T00:33:05.154 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OS4egfdE0IYG8LQqodKOo48wtJw= 2023-07-29T00:33:05.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OS4egfdE0IYG8LQqodKOo48wtJw=', 'Content-Length': '0'} 2023-07-29T00:33:05.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091fc7adde5dd7f75-0064c45e41-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.156 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:33:05.157 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:33:05.157 DEBUG:boto:path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:33:05.158 DEBUG:boto:auth_path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:33:05.158 DEBUG:boto:Method: HEAD 2023-07-29T00:33:05.158 DEBUG:boto:Path: /hiiqos-59/obj34?versionId=null 2023-07-29T00:33:05.159 DEBUG:boto:Data: 2023-07-29T00:33:05.159 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.159 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:05.160 DEBUG:boto:Port: 8000 2023-07-29T00:33:05.160 DEBUG:boto:Params: {} 2023-07-29T00:33:05.160 DEBUG:boto:Token: None 2023-07-29T00:33:05.161 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj34?versionId=null 2023-07-29T00:33:05.161 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OS4egfdE0IYG8LQqodKOo48wtJw= 2023-07-29T00:33:05.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OS4egfdE0IYG8LQqodKOo48wtJw=', 'Content-Length': '0'} 2023-07-29T00:33:05.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fb7de5021eb9215-0064c45e41-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-07-29T00:33:05.164 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:33:05.165 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:33:05.165 DEBUG:boto:path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:33:05.165 DEBUG:boto:auth_path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:33:05.166 DEBUG:boto:Method: GET 2023-07-29T00:33:05.166 DEBUG:boto:Path: /hiiqos-59/obj34?versionId=null 2023-07-29T00:33:05.166 DEBUG:boto:Data: 2023-07-29T00:33:05.167 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.167 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:05.167 DEBUG:boto:Port: 8001 2023-07-29T00:33:05.168 DEBUG:boto:Params: {} 2023-07-29T00:33:05.168 DEBUG:boto:Token: None 2023-07-29T00:33:05.169 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj34?versionId=null 2023-07-29T00:33:05.169 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:C/v+FdPrwxaB9UiHibJvDvAP7S8= 2023-07-29T00:33:05.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:C/v+FdPrwxaB9UiHibJvDvAP7S8=', 'Content-Length': '0'} 2023-07-29T00:33:05.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f41bd9c8d0829ba1-0064c45e41-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.172 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:33:05.172 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:33:05.172 DEBUG:boto:path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:33:05.173 DEBUG:boto:auth_path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:33:05.173 DEBUG:boto:Method: GET 2023-07-29T00:33:05.174 DEBUG:boto:Path: /hiiqos-59/obj34?versionId=null 2023-07-29T00:33:05.174 DEBUG:boto:Data: 2023-07-29T00:33:05.174 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.175 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:05.175 DEBUG:boto:Port: 8000 2023-07-29T00:33:05.175 DEBUG:boto:Params: {} 2023-07-29T00:33:05.176 DEBUG:boto:Token: None 2023-07-29T00:33:05.176 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj34?versionId=null 2023-07-29T00:33:05.177 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:C/v+FdPrwxaB9UiHibJvDvAP7S8= 2023-07-29T00:33:05.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:C/v+FdPrwxaB9UiHibJvDvAP7S8=', 'Content-Length': '0'} 2023-07-29T00:33:05.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007264c30620ee2f50-0064c45e41-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.179 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:05.180 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:05.180 DEBUG:boto:path=/hiiqos-59/?prefix=obj34 2023-07-29T00:33:05.180 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj34 2023-07-29T00:33:05.181 DEBUG:boto:Method: GET 2023-07-29T00:33:05.181 DEBUG:boto:Path: /hiiqos-59/?prefix=obj34 2023-07-29T00:33:05.182 DEBUG:boto:Data: 2023-07-29T00:33:05.182 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.182 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:05.183 DEBUG:boto:Port: 8001 2023-07-29T00:33:05.183 DEBUG:boto:Params: {} 2023-07-29T00:33:05.183 DEBUG:boto:Token: None 2023-07-29T00:33:05.184 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/ 2023-07-29T00:33:05.184 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs= 2023-07-29T00:33:05.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs=', 'Content-Length': '0'} 2023-07-29T00:33:05.187 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000020c671ff05b315a-0064c45e41-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.187 DEBUG:boto:b'hiiqos-59obj341000falseobj342023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:05.188 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:05.188 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:05.189 DEBUG:boto:path=/hiiqos-59/?prefix=obj34 2023-07-29T00:33:05.189 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj34 2023-07-29T00:33:05.190 DEBUG:boto:Method: GET 2023-07-29T00:33:05.190 DEBUG:boto:Path: /hiiqos-59/?prefix=obj34 2023-07-29T00:33:05.190 DEBUG:boto:Data: 2023-07-29T00:33:05.191 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.191 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:05.191 DEBUG:boto:Port: 8000 2023-07-29T00:33:05.192 DEBUG:boto:Params: {} 2023-07-29T00:33:05.192 DEBUG:boto:Token: None 2023-07-29T00:33:05.193 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/ 2023-07-29T00:33:05.193 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs= 2023-07-29T00:33:05.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs=', 'Content-Length': '0'} 2023-07-29T00:33:05.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4e2ad46e8348451-0064c45e41-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.196 DEBUG:boto:b'hiiqos-59obj341000falseobj342023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:05.197 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:33:05.197 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:33:05.198 DEBUG:boto:Method: HEAD 2023-07-29T00:33:05.198 DEBUG:boto:Path: /hiiqos-59/obj34 2023-07-29T00:33:05.198 DEBUG:boto:Data: 2023-07-29T00:33:05.199 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.199 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:05.199 DEBUG:boto:Port: 8001 2023-07-29T00:33:05.200 DEBUG:boto:Params: {} 2023-07-29T00:33:05.200 DEBUG:boto:Token: None 2023-07-29T00:33:05.201 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj34 2023-07-29T00:33:05.201 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EPiv76rjqMnPksUfvSYNe5vkmRU= 2023-07-29T00:33:05.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EPiv76rjqMnPksUfvSYNe5vkmRU=', 'Content-Length': '0'} 2023-07-29T00:33:05.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076edd3bd141aaddd-0064c45e41-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.204 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:33:05.204 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:33:05.204 DEBUG:boto:Method: HEAD 2023-07-29T00:33:05.205 DEBUG:boto:Path: /hiiqos-59/obj34 2023-07-29T00:33:05.205 DEBUG:boto:Data: 2023-07-29T00:33:05.205 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.206 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:05.206 DEBUG:boto:Port: 8000 2023-07-29T00:33:05.206 DEBUG:boto:Params: {} 2023-07-29T00:33:05.207 DEBUG:boto:Token: None 2023-07-29T00:33:05.207 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj34 2023-07-29T00:33:05.208 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EPiv76rjqMnPksUfvSYNe5vkmRU= 2023-07-29T00:33:05.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EPiv76rjqMnPksUfvSYNe5vkmRU=', 'Content-Length': '0'} 2023-07-29T00:33:05.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d371b4c3ce61f608-0064c45e41-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-07-29T00:33:05.210 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:33:05.211 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:33:05.211 DEBUG:boto:Method: GET 2023-07-29T00:33:05.211 DEBUG:boto:Path: /hiiqos-59/obj34 2023-07-29T00:33:05.212 DEBUG:boto:Data: 2023-07-29T00:33:05.212 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.213 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:05.213 DEBUG:boto:Port: 8001 2023-07-29T00:33:05.213 DEBUG:boto:Params: {} 2023-07-29T00:33:05.214 DEBUG:boto:Token: None 2023-07-29T00:33:05.214 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj34 2023-07-29T00:33:05.214 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fZG5rSGZrjANwI/p95+O+UfyY4o= 2023-07-29T00:33:05.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fZG5rSGZrjANwI/p95+O+UfyY4o=', 'Content-Length': '0'} 2023-07-29T00:33:05.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ee4c0285c749622-0064c45e41-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.217 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:33:05.217 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:33:05.218 DEBUG:boto:Method: GET 2023-07-29T00:33:05.218 DEBUG:boto:Path: /hiiqos-59/obj34 2023-07-29T00:33:05.218 DEBUG:boto:Data: 2023-07-29T00:33:05.219 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.219 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:05.220 DEBUG:boto:Port: 8000 2023-07-29T00:33:05.220 DEBUG:boto:Params: {} 2023-07-29T00:33:05.220 DEBUG:boto:Token: None 2023-07-29T00:33:05.221 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj34 2023-07-29T00:33:05.221 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fZG5rSGZrjANwI/p95+O+UfyY4o= 2023-07-29T00:33:05.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fZG5rSGZrjANwI/p95+O+UfyY4o=', 'Content-Length': '0'} 2023-07-29T00:33:05.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b070b9952f8fa63d-0064c45e41-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.224 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:05.224 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:05.225 DEBUG:boto:path=/hiiqos-59/?prefix=obj34 2023-07-29T00:33:05.225 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj34 2023-07-29T00:33:05.225 DEBUG:boto:Method: GET 2023-07-29T00:33:05.226 DEBUG:boto:Path: /hiiqos-59/?prefix=obj34 2023-07-29T00:33:05.226 DEBUG:boto:Data: 2023-07-29T00:33:05.226 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.227 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:05.227 DEBUG:boto:Port: 8001 2023-07-29T00:33:05.227 DEBUG:boto:Params: {} 2023-07-29T00:33:05.228 DEBUG:boto:Token: None 2023-07-29T00:33:05.228 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/ 2023-07-29T00:33:05.229 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs= 2023-07-29T00:33:05.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs=', 'Content-Length': '0'} 2023-07-29T00:33:05.231 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000562857ff8cb28d53-0064c45e41-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.232 DEBUG:boto:b'hiiqos-59obj341000falseobj342023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:05.232 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:05.233 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:05.233 DEBUG:boto:path=/hiiqos-59/?prefix=obj34 2023-07-29T00:33:05.233 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj34 2023-07-29T00:33:05.234 DEBUG:boto:Method: GET 2023-07-29T00:33:05.234 DEBUG:boto:Path: /hiiqos-59/?prefix=obj34 2023-07-29T00:33:05.234 DEBUG:boto:Data: 2023-07-29T00:33:05.235 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.235 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:05.235 DEBUG:boto:Port: 8000 2023-07-29T00:33:05.236 DEBUG:boto:Params: {} 2023-07-29T00:33:05.236 DEBUG:boto:Token: None 2023-07-29T00:33:05.237 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/ 2023-07-29T00:33:05.237 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs= 2023-07-29T00:33:05.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs=', 'Content-Length': '0'} 2023-07-29T00:33:05.240 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdf24ca08543aba1-0064c45e41-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.241 DEBUG:boto:b'hiiqos-59obj341000falseobj342023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:05.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-07-29T00:33:05.242 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:33:05.242 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:33:05.242 DEBUG:boto:path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:33:05.243 DEBUG:boto:auth_path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:33:05.243 DEBUG:boto:Method: GET 2023-07-29T00:33:05.243 DEBUG:boto:Path: /hiiqos-59/obj38?versionId=null 2023-07-29T00:33:05.244 DEBUG:boto:Data: 2023-07-29T00:33:05.244 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.245 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:05.245 DEBUG:boto:Port: 8001 2023-07-29T00:33:05.245 DEBUG:boto:Params: {} 2023-07-29T00:33:05.246 DEBUG:boto:Token: None 2023-07-29T00:33:05.246 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj38?versionId=null 2023-07-29T00:33:05.246 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dOnN6aQrPwjoUSLMaWoH3Ded9V0= 2023-07-29T00:33:05.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dOnN6aQrPwjoUSLMaWoH3Ded9V0=', 'Content-Length': '0'} 2023-07-29T00:33:05.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f32d59267fbe4433-0064c45e41-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.249 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:33:05.250 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:33:05.250 DEBUG:boto:path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:33:05.250 DEBUG:boto:auth_path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:33:05.251 DEBUG:boto:Method: GET 2023-07-29T00:33:05.251 DEBUG:boto:Path: /hiiqos-59/obj38?versionId=null 2023-07-29T00:33:05.251 DEBUG:boto:Data: 2023-07-29T00:33:05.252 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.252 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:05.252 DEBUG:boto:Port: 8000 2023-07-29T00:33:05.253 DEBUG:boto:Params: {} 2023-07-29T00:33:05.253 DEBUG:boto:Token: None 2023-07-29T00:33:05.254 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj38?versionId=null 2023-07-29T00:33:05.254 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dOnN6aQrPwjoUSLMaWoH3Ded9V0= 2023-07-29T00:33:05.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dOnN6aQrPwjoUSLMaWoH3Ded9V0=', 'Content-Length': '0'} 2023-07-29T00:33:05.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072ef63bfb93fd3a3-0064c45e41-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.256 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:33:05.257 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:33:05.257 DEBUG:boto:path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:33:05.258 DEBUG:boto:auth_path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:33:05.258 DEBUG:boto:Method: HEAD 2023-07-29T00:33:05.258 DEBUG:boto:Path: /hiiqos-59/obj38?versionId=null 2023-07-29T00:33:05.259 DEBUG:boto:Data: 2023-07-29T00:33:05.259 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.259 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:05.260 DEBUG:boto:Port: 8001 2023-07-29T00:33:05.260 DEBUG:boto:Params: {} 2023-07-29T00:33:05.260 DEBUG:boto:Token: None 2023-07-29T00:33:05.261 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj38?versionId=null 2023-07-29T00:33:05.261 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hZy3qjQBCoOKuXbffecLmq4aqvc= 2023-07-29T00:33:05.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hZy3qjQBCoOKuXbffecLmq4aqvc=', 'Content-Length': '0'} 2023-07-29T00:33:05.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007655d5bfa132897a-0064c45e41-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.264 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:33:05.264 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:33:05.264 DEBUG:boto:path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:33:05.265 DEBUG:boto:auth_path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:33:05.265 DEBUG:boto:Method: HEAD 2023-07-29T00:33:05.265 DEBUG:boto:Path: /hiiqos-59/obj38?versionId=null 2023-07-29T00:33:05.266 DEBUG:boto:Data: 2023-07-29T00:33:05.266 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.266 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:05.267 DEBUG:boto:Port: 8000 2023-07-29T00:33:05.267 DEBUG:boto:Params: {} 2023-07-29T00:33:05.267 DEBUG:boto:Token: None 2023-07-29T00:33:05.268 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj38?versionId=null 2023-07-29T00:33:05.268 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hZy3qjQBCoOKuXbffecLmq4aqvc= 2023-07-29T00:33:05.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hZy3qjQBCoOKuXbffecLmq4aqvc=', 'Content-Length': '0'} 2023-07-29T00:33:05.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c76d90beb6c0af9d-0064c45e41-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-07-29T00:33:05.271 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:33:05.271 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:33:05.272 DEBUG:boto:path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:33:05.272 DEBUG:boto:auth_path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:33:05.272 DEBUG:boto:Method: GET 2023-07-29T00:33:05.273 DEBUG:boto:Path: /hiiqos-59/obj38?versionId=null 2023-07-29T00:33:05.273 DEBUG:boto:Data: 2023-07-29T00:33:05.274 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.274 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:05.274 DEBUG:boto:Port: 8001 2023-07-29T00:33:05.275 DEBUG:boto:Params: {} 2023-07-29T00:33:05.275 DEBUG:boto:Token: None 2023-07-29T00:33:05.275 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj38?versionId=null 2023-07-29T00:33:05.276 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dOnN6aQrPwjoUSLMaWoH3Ded9V0= 2023-07-29T00:33:05.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dOnN6aQrPwjoUSLMaWoH3Ded9V0=', 'Content-Length': '0'} 2023-07-29T00:33:05.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ecec3b295ea4087-0064c45e41-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.324 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:33:05.325 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:33:05.325 DEBUG:boto:path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:33:05.325 DEBUG:boto:auth_path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:33:05.326 DEBUG:boto:Method: GET 2023-07-29T00:33:05.326 DEBUG:boto:Path: /hiiqos-59/obj38?versionId=null 2023-07-29T00:33:05.327 DEBUG:boto:Data: 2023-07-29T00:33:05.327 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.327 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:05.328 DEBUG:boto:Port: 8000 2023-07-29T00:33:05.328 DEBUG:boto:Params: {} 2023-07-29T00:33:05.328 DEBUG:boto:Token: None 2023-07-29T00:33:05.329 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj38?versionId=null 2023-07-29T00:33:05.329 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dOnN6aQrPwjoUSLMaWoH3Ded9V0= 2023-07-29T00:33:05.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dOnN6aQrPwjoUSLMaWoH3Ded9V0=', 'Content-Length': '0'} 2023-07-29T00:33:05.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029b832f15150bac0-0064c45e41-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.332 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:05.333 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:05.333 DEBUG:boto:path=/hiiqos-59/?prefix=obj38 2023-07-29T00:33:05.333 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj38 2023-07-29T00:33:05.334 DEBUG:boto:Method: GET 2023-07-29T00:33:05.334 DEBUG:boto:Path: /hiiqos-59/?prefix=obj38 2023-07-29T00:33:05.335 DEBUG:boto:Data: 2023-07-29T00:33:05.335 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.336 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:05.336 DEBUG:boto:Port: 8001 2023-07-29T00:33:05.336 DEBUG:boto:Params: {} 2023-07-29T00:33:05.337 DEBUG:boto:Token: None 2023-07-29T00:33:05.337 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/ 2023-07-29T00:33:05.338 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs= 2023-07-29T00:33:05.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs=', 'Content-Length': '0'} 2023-07-29T00:33:05.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000486727c7372da562-0064c45e41-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.341 DEBUG:boto:b'hiiqos-59obj381000falseobj382023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:05.341 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:05.342 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:05.342 DEBUG:boto:path=/hiiqos-59/?prefix=obj38 2023-07-29T00:33:05.343 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj38 2023-07-29T00:33:05.343 DEBUG:boto:Method: GET 2023-07-29T00:33:05.343 DEBUG:boto:Path: /hiiqos-59/?prefix=obj38 2023-07-29T00:33:05.344 DEBUG:boto:Data: 2023-07-29T00:33:05.344 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.344 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:05.345 DEBUG:boto:Port: 8000 2023-07-29T00:33:05.345 DEBUG:boto:Params: {} 2023-07-29T00:33:05.346 DEBUG:boto:Token: None 2023-07-29T00:33:05.346 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/ 2023-07-29T00:33:05.346 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs= 2023-07-29T00:33:05.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs=', 'Content-Length': '0'} 2023-07-29T00:33:05.350 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2a92b9f963beea4-0064c45e41-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.350 DEBUG:boto:b'hiiqos-59obj381000falseobj382023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:05.351 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:33:05.351 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:33:05.351 DEBUG:boto:Method: HEAD 2023-07-29T00:33:05.352 DEBUG:boto:Path: /hiiqos-59/obj38 2023-07-29T00:33:05.352 DEBUG:boto:Data: 2023-07-29T00:33:05.353 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.353 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:05.353 DEBUG:boto:Port: 8001 2023-07-29T00:33:05.354 DEBUG:boto:Params: {} 2023-07-29T00:33:05.354 DEBUG:boto:Token: None 2023-07-29T00:33:05.354 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj38 2023-07-29T00:33:05.355 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qPa8chm1slM4FZTK+NgZBnHFi5k= 2023-07-29T00:33:05.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qPa8chm1slM4FZTK+NgZBnHFi5k=', 'Content-Length': '0'} 2023-07-29T00:33:05.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000060cd0d14e8f9744-0064c45e41-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.357 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:33:05.358 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:33:05.358 DEBUG:boto:Method: HEAD 2023-07-29T00:33:05.358 DEBUG:boto:Path: /hiiqos-59/obj38 2023-07-29T00:33:05.359 DEBUG:boto:Data: 2023-07-29T00:33:05.359 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.359 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:05.360 DEBUG:boto:Port: 8000 2023-07-29T00:33:05.360 DEBUG:boto:Params: {} 2023-07-29T00:33:05.360 DEBUG:boto:Token: None 2023-07-29T00:33:05.361 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj38 2023-07-29T00:33:05.361 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qPa8chm1slM4FZTK+NgZBnHFi5k= 2023-07-29T00:33:05.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qPa8chm1slM4FZTK+NgZBnHFi5k=', 'Content-Length': '0'} 2023-07-29T00:33:05.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000893e49730b996c8f-0064c45e41-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-07-29T00:33:05.364 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:33:05.364 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:33:05.365 DEBUG:boto:Method: GET 2023-07-29T00:33:05.365 DEBUG:boto:Path: /hiiqos-59/obj38 2023-07-29T00:33:05.366 DEBUG:boto:Data: 2023-07-29T00:33:05.366 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.366 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:05.367 DEBUG:boto:Port: 8001 2023-07-29T00:33:05.367 DEBUG:boto:Params: {} 2023-07-29T00:33:05.367 DEBUG:boto:Token: None 2023-07-29T00:33:05.368 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj38 2023-07-29T00:33:05.368 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:voIedtuAEeZ48xiceFwxsS8tdVk= 2023-07-29T00:33:05.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:voIedtuAEeZ48xiceFwxsS8tdVk=', 'Content-Length': '0'} 2023-07-29T00:33:05.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb497c473dda570f-0064c45e41-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.412 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:33:05.413 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:33:05.413 DEBUG:boto:Method: GET 2023-07-29T00:33:05.413 DEBUG:boto:Path: /hiiqos-59/obj38 2023-07-29T00:33:05.414 DEBUG:boto:Data: 2023-07-29T00:33:05.414 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.414 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:05.415 DEBUG:boto:Port: 8000 2023-07-29T00:33:05.415 DEBUG:boto:Params: {} 2023-07-29T00:33:05.416 DEBUG:boto:Token: None 2023-07-29T00:33:05.416 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj38 2023-07-29T00:33:05.416 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:voIedtuAEeZ48xiceFwxsS8tdVk= 2023-07-29T00:33:05.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:voIedtuAEeZ48xiceFwxsS8tdVk=', 'Content-Length': '0'} 2023-07-29T00:33:05.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007989838c8012b105-0064c45e41-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.419 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:05.419 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:05.420 DEBUG:boto:path=/hiiqos-59/?prefix=obj38 2023-07-29T00:33:05.420 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj38 2023-07-29T00:33:05.421 DEBUG:boto:Method: GET 2023-07-29T00:33:05.421 DEBUG:boto:Path: /hiiqos-59/?prefix=obj38 2023-07-29T00:33:05.421 DEBUG:boto:Data: 2023-07-29T00:33:05.422 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.422 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:05.422 DEBUG:boto:Port: 8001 2023-07-29T00:33:05.423 DEBUG:boto:Params: {} 2023-07-29T00:33:05.423 DEBUG:boto:Token: None 2023-07-29T00:33:05.423 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/ 2023-07-29T00:33:05.424 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs= 2023-07-29T00:33:05.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs=', 'Content-Length': '0'} 2023-07-29T00:33:05.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042030cd424737b96-0064c45e41-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.429 DEBUG:boto:b'hiiqos-59obj381000falseobj382023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:05.430 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:05.430 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:05.431 DEBUG:boto:path=/hiiqos-59/?prefix=obj38 2023-07-29T00:33:05.431 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj38 2023-07-29T00:33:05.431 DEBUG:boto:Method: GET 2023-07-29T00:33:05.432 DEBUG:boto:Path: /hiiqos-59/?prefix=obj38 2023-07-29T00:33:05.432 DEBUG:boto:Data: 2023-07-29T00:33:05.433 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.433 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:05.433 DEBUG:boto:Port: 8000 2023-07-29T00:33:05.434 DEBUG:boto:Params: {} 2023-07-29T00:33:05.434 DEBUG:boto:Token: None 2023-07-29T00:33:05.435 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/ 2023-07-29T00:33:05.435 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs= 2023-07-29T00:33:05.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs=', 'Content-Length': '0'} 2023-07-29T00:33:05.438 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003686005f54edb554-0064c45e41-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.438 DEBUG:boto:b'hiiqos-59obj381000falseobj382023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:05.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-07-29T00:33:05.440 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:33:05.440 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:33:05.440 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:33:05.441 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:33:05.441 DEBUG:boto:Method: GET 2023-07-29T00:33:05.441 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:33:05.442 DEBUG:boto:Data: 2023-07-29T00:33:05.442 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.442 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:05.443 DEBUG:boto:Port: 8001 2023-07-29T00:33:05.443 DEBUG:boto:Params: {} 2023-07-29T00:33:05.444 DEBUG:boto:Token: None 2023-07-29T00:33:05.444 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:33:05.444 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:v9DdQp9Td2mwr405AwpYKM09ulM= 2023-07-29T00:33:05.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:v9DdQp9Td2mwr405AwpYKM09ulM=', 'Content-Length': '0'} 2023-07-29T00:33:05.446 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c33dbd2b07a9082f-0064c45e41-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.447 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:33:05.447 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:33:05.448 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:33:05.448 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:33:05.449 DEBUG:boto:Method: GET 2023-07-29T00:33:05.449 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:33:05.449 DEBUG:boto:Data: 2023-07-29T00:33:05.450 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.450 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:05.450 DEBUG:boto:Port: 8000 2023-07-29T00:33:05.451 DEBUG:boto:Params: {} 2023-07-29T00:33:05.451 DEBUG:boto:Token: None 2023-07-29T00:33:05.451 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:33:05.452 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:v9DdQp9Td2mwr405AwpYKM09ulM= 2023-07-29T00:33:05.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:v9DdQp9Td2mwr405AwpYKM09ulM=', 'Content-Length': '0'} 2023-07-29T00:33:05.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009285134c735a1b7-0064c45e41-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.455 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:33:05.455 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:33:05.455 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:33:05.456 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:33:05.456 DEBUG:boto:Method: HEAD 2023-07-29T00:33:05.456 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:33:05.457 DEBUG:boto:Data: 2023-07-29T00:33:05.457 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.457 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:05.458 DEBUG:boto:Port: 8001 2023-07-29T00:33:05.458 DEBUG:boto:Params: {} 2023-07-29T00:33:05.458 DEBUG:boto:Token: None 2023-07-29T00:33:05.459 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:33:05.459 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6TDdldyrhjfN40RrAmLRoNmuzm0= 2023-07-29T00:33:05.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6TDdldyrhjfN40RrAmLRoNmuzm0=', 'Content-Length': '0'} 2023-07-29T00:33:05.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fc304de99ddf81a-0064c45e41-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.462 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:33:05.462 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:33:05.463 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:33:05.463 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:33:05.463 DEBUG:boto:Method: HEAD 2023-07-29T00:33:05.464 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:33:05.464 DEBUG:boto:Data: 2023-07-29T00:33:05.464 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.465 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:05.465 DEBUG:boto:Port: 8000 2023-07-29T00:33:05.465 DEBUG:boto:Params: {} 2023-07-29T00:33:05.466 DEBUG:boto:Token: None 2023-07-29T00:33:05.466 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:33:05.467 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6TDdldyrhjfN40RrAmLRoNmuzm0= 2023-07-29T00:33:05.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6TDdldyrhjfN40RrAmLRoNmuzm0=', 'Content-Length': '0'} 2023-07-29T00:33:05.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5121792594e32e4-0064c45e41-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-07-29T00:33:05.469 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:33:05.470 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:33:05.470 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:33:05.470 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:33:05.471 DEBUG:boto:Method: GET 2023-07-29T00:33:05.471 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:33:05.471 DEBUG:boto:Data: 2023-07-29T00:33:05.472 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.472 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:05.473 DEBUG:boto:Port: 8001 2023-07-29T00:33:05.473 DEBUG:boto:Params: {} 2023-07-29T00:33:05.473 DEBUG:boto:Token: None 2023-07-29T00:33:05.474 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:33:05.474 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:v9DdQp9Td2mwr405AwpYKM09ulM= 2023-07-29T00:33:05.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:v9DdQp9Td2mwr405AwpYKM09ulM=', 'Content-Length': '0'} 2023-07-29T00:33:05.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049897c1a9c40c12f-0064c45e41-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.477 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:33:05.477 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:33:05.477 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:33:05.478 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:33:05.478 DEBUG:boto:Method: GET 2023-07-29T00:33:05.478 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:33:05.479 DEBUG:boto:Data: 2023-07-29T00:33:05.479 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.480 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:05.480 DEBUG:boto:Port: 8000 2023-07-29T00:33:05.480 DEBUG:boto:Params: {} 2023-07-29T00:33:05.481 DEBUG:boto:Token: None 2023-07-29T00:33:05.481 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:33:05.482 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:v9DdQp9Td2mwr405AwpYKM09ulM= 2023-07-29T00:33:05.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:v9DdQp9Td2mwr405AwpYKM09ulM=', 'Content-Length': '0'} 2023-07-29T00:33:05.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac5f05767345f5c0-0064c45e41-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.484 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:05.484 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:05.485 DEBUG:boto:path=/hiiqos-59/?prefix=obj44 2023-07-29T00:33:05.485 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj44 2023-07-29T00:33:05.486 DEBUG:boto:Method: GET 2023-07-29T00:33:05.486 DEBUG:boto:Path: /hiiqos-59/?prefix=obj44 2023-07-29T00:33:05.486 DEBUG:boto:Data: 2023-07-29T00:33:05.487 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.487 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:05.487 DEBUG:boto:Port: 8001 2023-07-29T00:33:05.488 DEBUG:boto:Params: {} 2023-07-29T00:33:05.488 DEBUG:boto:Token: None 2023-07-29T00:33:05.489 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/ 2023-07-29T00:33:05.489 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs= 2023-07-29T00:33:05.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs=', 'Content-Length': '0'} 2023-07-29T00:33:05.491 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067712617cd34bba3-0064c45e41-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.492 DEBUG:boto:b'hiiqos-59obj441000falseobj442023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:05.496 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:05.496 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:05.497 DEBUG:boto:path=/hiiqos-59/?prefix=obj44 2023-07-29T00:33:05.497 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj44 2023-07-29T00:33:05.497 DEBUG:boto:Method: GET 2023-07-29T00:33:05.498 DEBUG:boto:Path: /hiiqos-59/?prefix=obj44 2023-07-29T00:33:05.498 DEBUG:boto:Data: 2023-07-29T00:33:05.498 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.499 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:05.499 DEBUG:boto:Port: 8000 2023-07-29T00:33:05.499 DEBUG:boto:Params: {} 2023-07-29T00:33:05.500 DEBUG:boto:Token: None 2023-07-29T00:33:05.500 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/ 2023-07-29T00:33:05.501 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs= 2023-07-29T00:33:05.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs=', 'Content-Length': '0'} 2023-07-29T00:33:05.504 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e10f5fb4ec6ec0a9-0064c45e41-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.504 DEBUG:boto:b'hiiqos-59obj441000falseobj442023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:05.505 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:33:05.505 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:33:05.506 DEBUG:boto:Method: HEAD 2023-07-29T00:33:05.506 DEBUG:boto:Path: /hiiqos-59/obj44 2023-07-29T00:33:05.506 DEBUG:boto:Data: 2023-07-29T00:33:05.507 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.507 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:05.507 DEBUG:boto:Port: 8001 2023-07-29T00:33:05.508 DEBUG:boto:Params: {} 2023-07-29T00:33:05.508 DEBUG:boto:Token: None 2023-07-29T00:33:05.508 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj44 2023-07-29T00:33:05.509 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:seAnM8U0OuJMgXCwA4/YEsdg+1Y= 2023-07-29T00:33:05.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:seAnM8U0OuJMgXCwA4/YEsdg+1Y=', 'Content-Length': '0'} 2023-07-29T00:33:05.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e31b0ef3fe7ce99e-0064c45e41-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.511 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:33:05.512 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:33:05.512 DEBUG:boto:Method: HEAD 2023-07-29T00:33:05.512 DEBUG:boto:Path: /hiiqos-59/obj44 2023-07-29T00:33:05.513 DEBUG:boto:Data: 2023-07-29T00:33:05.513 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.513 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:05.514 DEBUG:boto:Port: 8000 2023-07-29T00:33:05.514 DEBUG:boto:Params: {} 2023-07-29T00:33:05.515 DEBUG:boto:Token: None 2023-07-29T00:33:05.515 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj44 2023-07-29T00:33:05.515 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:seAnM8U0OuJMgXCwA4/YEsdg+1Y= 2023-07-29T00:33:05.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:seAnM8U0OuJMgXCwA4/YEsdg+1Y=', 'Content-Length': '0'} 2023-07-29T00:33:05.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9f2c400fa7bf736-0064c45e41-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-07-29T00:33:05.518 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:33:05.518 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:33:05.519 DEBUG:boto:Method: GET 2023-07-29T00:33:05.519 DEBUG:boto:Path: /hiiqos-59/obj44 2023-07-29T00:33:05.520 DEBUG:boto:Data: 2023-07-29T00:33:05.520 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.520 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:05.521 DEBUG:boto:Port: 8001 2023-07-29T00:33:05.521 DEBUG:boto:Params: {} 2023-07-29T00:33:05.521 DEBUG:boto:Token: None 2023-07-29T00:33:05.522 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj44 2023-07-29T00:33:05.522 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:raDJJVwmaV5gwkKZ4YiOykpzFow= 2023-07-29T00:33:05.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:raDJJVwmaV5gwkKZ4YiOykpzFow=', 'Content-Length': '0'} 2023-07-29T00:33:05.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005275bafb14b7d8bb-0064c45e41-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.525 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:33:05.525 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:33:05.526 DEBUG:boto:Method: GET 2023-07-29T00:33:05.526 DEBUG:boto:Path: /hiiqos-59/obj44 2023-07-29T00:33:05.526 DEBUG:boto:Data: 2023-07-29T00:33:05.527 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.527 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:05.528 DEBUG:boto:Port: 8000 2023-07-29T00:33:05.528 DEBUG:boto:Params: {} 2023-07-29T00:33:05.529 DEBUG:boto:Token: None 2023-07-29T00:33:05.529 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj44 2023-07-29T00:33:05.529 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:raDJJVwmaV5gwkKZ4YiOykpzFow= 2023-07-29T00:33:05.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:raDJJVwmaV5gwkKZ4YiOykpzFow=', 'Content-Length': '0'} 2023-07-29T00:33:05.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff11de8b600f8e98-0064c45e41-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.532 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:05.533 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:05.533 DEBUG:boto:path=/hiiqos-59/?prefix=obj44 2023-07-29T00:33:05.533 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj44 2023-07-29T00:33:05.534 DEBUG:boto:Method: GET 2023-07-29T00:33:05.534 DEBUG:boto:Path: /hiiqos-59/?prefix=obj44 2023-07-29T00:33:05.534 DEBUG:boto:Data: 2023-07-29T00:33:05.535 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.535 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:05.536 DEBUG:boto:Port: 8001 2023-07-29T00:33:05.536 DEBUG:boto:Params: {} 2023-07-29T00:33:05.536 DEBUG:boto:Token: None 2023-07-29T00:33:05.537 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/ 2023-07-29T00:33:05.537 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs= 2023-07-29T00:33:05.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs=', 'Content-Length': '0'} 2023-07-29T00:33:05.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074d51a2e52011d06-0064c45e41-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.540 DEBUG:boto:b'hiiqos-59obj441000falseobj442023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:05.541 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:05.541 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:05.542 DEBUG:boto:path=/hiiqos-59/?prefix=obj44 2023-07-29T00:33:05.542 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj44 2023-07-29T00:33:05.542 DEBUG:boto:Method: GET 2023-07-29T00:33:05.543 DEBUG:boto:Path: /hiiqos-59/?prefix=obj44 2023-07-29T00:33:05.543 DEBUG:boto:Data: 2023-07-29T00:33:05.543 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.544 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:05.544 DEBUG:boto:Port: 8000 2023-07-29T00:33:05.545 DEBUG:boto:Params: {} 2023-07-29T00:33:05.545 DEBUG:boto:Token: None 2023-07-29T00:33:05.545 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/ 2023-07-29T00:33:05.546 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs= 2023-07-29T00:33:05.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs=', 'Content-Length': '0'} 2023-07-29T00:33:05.549 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6db48fa11bed59c-0064c45e41-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.549 DEBUG:boto:b'hiiqos-59obj441000falseobj442023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:05.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-07-29T00:33:05.550 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:33:05.551 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:33:05.551 DEBUG:boto:path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:33:05.551 DEBUG:boto:auth_path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:33:05.552 DEBUG:boto:Method: GET 2023-07-29T00:33:05.552 DEBUG:boto:Path: /hiiqos-59/obj51?versionId=null 2023-07-29T00:33:05.552 DEBUG:boto:Data: 2023-07-29T00:33:05.553 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.553 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:05.553 DEBUG:boto:Port: 8001 2023-07-29T00:33:05.554 DEBUG:boto:Params: {} 2023-07-29T00:33:05.554 DEBUG:boto:Token: None 2023-07-29T00:33:05.555 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj51?versionId=null 2023-07-29T00:33:05.555 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7eejyBlgaAC82bQ5PrQDvRmeP90= 2023-07-29T00:33:05.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7eejyBlgaAC82bQ5PrQDvRmeP90=', 'Content-Length': '0'} 2023-07-29T00:33:05.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b05fb829f70834d-0064c45e41-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.558 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:33:05.558 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:33:05.559 DEBUG:boto:path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:33:05.559 DEBUG:boto:auth_path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:33:05.559 DEBUG:boto:Method: GET 2023-07-29T00:33:05.560 DEBUG:boto:Path: /hiiqos-59/obj51?versionId=null 2023-07-29T00:33:05.560 DEBUG:boto:Data: 2023-07-29T00:33:05.561 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.561 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:05.561 DEBUG:boto:Port: 8000 2023-07-29T00:33:05.562 DEBUG:boto:Params: {} 2023-07-29T00:33:05.562 DEBUG:boto:Token: None 2023-07-29T00:33:05.562 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj51?versionId=null 2023-07-29T00:33:05.563 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7eejyBlgaAC82bQ5PrQDvRmeP90= 2023-07-29T00:33:05.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7eejyBlgaAC82bQ5PrQDvRmeP90=', 'Content-Length': '0'} 2023-07-29T00:33:05.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000465e95553dc2f858-0064c45e41-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.565 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:33:05.566 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:33:05.566 DEBUG:boto:path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:33:05.566 DEBUG:boto:auth_path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:33:05.567 DEBUG:boto:Method: HEAD 2023-07-29T00:33:05.567 DEBUG:boto:Path: /hiiqos-59/obj51?versionId=null 2023-07-29T00:33:05.567 DEBUG:boto:Data: 2023-07-29T00:33:05.568 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.568 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:05.569 DEBUG:boto:Port: 8001 2023-07-29T00:33:05.569 DEBUG:boto:Params: {} 2023-07-29T00:33:05.569 DEBUG:boto:Token: None 2023-07-29T00:33:05.570 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj51?versionId=null 2023-07-29T00:33:05.570 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/sceMqpyXtF8WmTuk416+9il3sY= 2023-07-29T00:33:05.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/sceMqpyXtF8WmTuk416+9il3sY=', 'Content-Length': '0'} 2023-07-29T00:33:05.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000487f254984d6b4b4-0064c45e41-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.573 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:33:05.573 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:33:05.573 DEBUG:boto:path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:33:05.574 DEBUG:boto:auth_path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:33:05.574 DEBUG:boto:Method: HEAD 2023-07-29T00:33:05.574 DEBUG:boto:Path: /hiiqos-59/obj51?versionId=null 2023-07-29T00:33:05.575 DEBUG:boto:Data: 2023-07-29T00:33:05.575 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.575 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:05.576 DEBUG:boto:Port: 8000 2023-07-29T00:33:05.576 DEBUG:boto:Params: {} 2023-07-29T00:33:05.576 DEBUG:boto:Token: None 2023-07-29T00:33:05.577 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj51?versionId=null 2023-07-29T00:33:05.577 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/sceMqpyXtF8WmTuk416+9il3sY= 2023-07-29T00:33:05.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/sceMqpyXtF8WmTuk416+9il3sY=', 'Content-Length': '0'} 2023-07-29T00:33:05.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f43815af4b5469c-0064c45e41-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-07-29T00:33:05.580 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:33:05.580 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:33:05.581 DEBUG:boto:path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:33:05.581 DEBUG:boto:auth_path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:33:05.581 DEBUG:boto:Method: GET 2023-07-29T00:33:05.582 DEBUG:boto:Path: /hiiqos-59/obj51?versionId=null 2023-07-29T00:33:05.582 DEBUG:boto:Data: 2023-07-29T00:33:05.582 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.583 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:05.583 DEBUG:boto:Port: 8001 2023-07-29T00:33:05.583 DEBUG:boto:Params: {} 2023-07-29T00:33:05.584 DEBUG:boto:Token: None 2023-07-29T00:33:05.584 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj51?versionId=null 2023-07-29T00:33:05.585 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7eejyBlgaAC82bQ5PrQDvRmeP90= 2023-07-29T00:33:05.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7eejyBlgaAC82bQ5PrQDvRmeP90=', 'Content-Length': '0'} 2023-07-29T00:33:05.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004438f2a85ca36922-0064c45e41-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.587 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:33:05.588 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:33:05.588 DEBUG:boto:path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:33:05.588 DEBUG:boto:auth_path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:33:05.589 DEBUG:boto:Method: GET 2023-07-29T00:33:05.589 DEBUG:boto:Path: /hiiqos-59/obj51?versionId=null 2023-07-29T00:33:05.589 DEBUG:boto:Data: 2023-07-29T00:33:05.590 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.590 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:05.590 DEBUG:boto:Port: 8000 2023-07-29T00:33:05.591 DEBUG:boto:Params: {} 2023-07-29T00:33:05.591 DEBUG:boto:Token: None 2023-07-29T00:33:05.592 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj51?versionId=null 2023-07-29T00:33:05.592 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7eejyBlgaAC82bQ5PrQDvRmeP90= 2023-07-29T00:33:05.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7eejyBlgaAC82bQ5PrQDvRmeP90=', 'Content-Length': '0'} 2023-07-29T00:33:05.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000611c85a72aa4c320-0064c45e41-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.636 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:05.637 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:05.637 DEBUG:boto:path=/hiiqos-59/?prefix=obj51 2023-07-29T00:33:05.637 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj51 2023-07-29T00:33:05.638 DEBUG:boto:Method: GET 2023-07-29T00:33:05.638 DEBUG:boto:Path: /hiiqos-59/?prefix=obj51 2023-07-29T00:33:05.639 DEBUG:boto:Data: 2023-07-29T00:33:05.639 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.639 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:05.640 DEBUG:boto:Port: 8001 2023-07-29T00:33:05.640 DEBUG:boto:Params: {} 2023-07-29T00:33:05.640 DEBUG:boto:Token: None 2023-07-29T00:33:05.641 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/ 2023-07-29T00:33:05.641 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs= 2023-07-29T00:33:05.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs=', 'Content-Length': '0'} 2023-07-29T00:33:05.644 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5a568bb866bf664-0064c45e41-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.644 DEBUG:boto:b'hiiqos-59obj511000falseobj512023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:05.645 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:05.645 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:05.646 DEBUG:boto:path=/hiiqos-59/?prefix=obj51 2023-07-29T00:33:05.646 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj51 2023-07-29T00:33:05.646 DEBUG:boto:Method: GET 2023-07-29T00:33:05.647 DEBUG:boto:Path: /hiiqos-59/?prefix=obj51 2023-07-29T00:33:05.647 DEBUG:boto:Data: 2023-07-29T00:33:05.647 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.648 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:05.648 DEBUG:boto:Port: 8000 2023-07-29T00:33:05.648 DEBUG:boto:Params: {} 2023-07-29T00:33:05.649 DEBUG:boto:Token: None 2023-07-29T00:33:05.649 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/ 2023-07-29T00:33:05.650 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs= 2023-07-29T00:33:05.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs=', 'Content-Length': '0'} 2023-07-29T00:33:05.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a14d989555850cb6-0064c45e41-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.653 DEBUG:boto:b'hiiqos-59obj511000falseobj512023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:05.654 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:33:05.654 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:33:05.655 DEBUG:boto:Method: HEAD 2023-07-29T00:33:05.655 DEBUG:boto:Path: /hiiqos-59/obj51 2023-07-29T00:33:05.655 DEBUG:boto:Data: 2023-07-29T00:33:05.656 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.656 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:05.656 DEBUG:boto:Port: 8001 2023-07-29T00:33:05.657 DEBUG:boto:Params: {} 2023-07-29T00:33:05.657 DEBUG:boto:Token: None 2023-07-29T00:33:05.658 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj51 2023-07-29T00:33:05.658 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kr17R8wXF4IiLrQv7pp4UvLlgOc= 2023-07-29T00:33:05.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kr17R8wXF4IiLrQv7pp4UvLlgOc=', 'Content-Length': '0'} 2023-07-29T00:33:05.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035d6579b3eca9ff8-0064c45e41-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.661 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:33:05.661 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:33:05.662 DEBUG:boto:Method: HEAD 2023-07-29T00:33:05.662 DEBUG:boto:Path: /hiiqos-59/obj51 2023-07-29T00:33:05.662 DEBUG:boto:Data: 2023-07-29T00:33:05.663 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.663 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:05.663 DEBUG:boto:Port: 8000 2023-07-29T00:33:05.664 DEBUG:boto:Params: {} 2023-07-29T00:33:05.664 DEBUG:boto:Token: None 2023-07-29T00:33:05.665 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj51 2023-07-29T00:33:05.665 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kr17R8wXF4IiLrQv7pp4UvLlgOc= 2023-07-29T00:33:05.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kr17R8wXF4IiLrQv7pp4UvLlgOc=', 'Content-Length': '0'} 2023-07-29T00:33:05.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4ef1d15e9041543-0064c45e41-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-07-29T00:33:05.668 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:33:05.668 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:33:05.669 DEBUG:boto:Method: GET 2023-07-29T00:33:05.669 DEBUG:boto:Path: /hiiqos-59/obj51 2023-07-29T00:33:05.669 DEBUG:boto:Data: 2023-07-29T00:33:05.670 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.670 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:05.670 DEBUG:boto:Port: 8001 2023-07-29T00:33:05.671 DEBUG:boto:Params: {} 2023-07-29T00:33:05.671 DEBUG:boto:Token: None 2023-07-29T00:33:05.672 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj51 2023-07-29T00:33:05.672 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DZceA41v6/M6OmxC9UB7OrkvmAM= 2023-07-29T00:33:05.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DZceA41v6/M6OmxC9UB7OrkvmAM=', 'Content-Length': '0'} 2023-07-29T00:33:05.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000325ba8b275d2d20c-0064c45e41-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.720 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:33:05.721 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:33:05.721 DEBUG:boto:Method: GET 2023-07-29T00:33:05.722 DEBUG:boto:Path: /hiiqos-59/obj51 2023-07-29T00:33:05.722 DEBUG:boto:Data: 2023-07-29T00:33:05.722 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.723 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:05.723 DEBUG:boto:Port: 8000 2023-07-29T00:33:05.723 DEBUG:boto:Params: {} 2023-07-29T00:33:05.724 DEBUG:boto:Token: None 2023-07-29T00:33:05.724 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj51 2023-07-29T00:33:05.725 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DZceA41v6/M6OmxC9UB7OrkvmAM= 2023-07-29T00:33:05.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DZceA41v6/M6OmxC9UB7OrkvmAM=', 'Content-Length': '0'} 2023-07-29T00:33:05.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4110ac40572d2ab-0064c45e41-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.727 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:05.728 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:05.728 DEBUG:boto:path=/hiiqos-59/?prefix=obj51 2023-07-29T00:33:05.729 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj51 2023-07-29T00:33:05.729 DEBUG:boto:Method: GET 2023-07-29T00:33:05.729 DEBUG:boto:Path: /hiiqos-59/?prefix=obj51 2023-07-29T00:33:05.730 DEBUG:boto:Data: 2023-07-29T00:33:05.730 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.730 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:05.731 DEBUG:boto:Port: 8001 2023-07-29T00:33:05.731 DEBUG:boto:Params: {} 2023-07-29T00:33:05.731 DEBUG:boto:Token: None 2023-07-29T00:33:05.732 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/ 2023-07-29T00:33:05.732 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs= 2023-07-29T00:33:05.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs=', 'Content-Length': '0'} 2023-07-29T00:33:05.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1d47a99822bb6a6-0064c45e41-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.735 DEBUG:boto:b'hiiqos-59obj511000falseobj512023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:05.736 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:05.736 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:05.737 DEBUG:boto:path=/hiiqos-59/?prefix=obj51 2023-07-29T00:33:05.737 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj51 2023-07-29T00:33:05.737 DEBUG:boto:Method: GET 2023-07-29T00:33:05.738 DEBUG:boto:Path: /hiiqos-59/?prefix=obj51 2023-07-29T00:33:05.738 DEBUG:boto:Data: 2023-07-29T00:33:05.738 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.739 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:05.739 DEBUG:boto:Port: 8000 2023-07-29T00:33:05.739 DEBUG:boto:Params: {} 2023-07-29T00:33:05.740 DEBUG:boto:Token: None 2023-07-29T00:33:05.740 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/ 2023-07-29T00:33:05.741 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs= 2023-07-29T00:33:05.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs=', 'Content-Length': '0'} 2023-07-29T00:33:05.744 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086cd980bac3e2ba3-0064c45e41-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.744 DEBUG:boto:b'hiiqos-59obj511000falseobj512023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:05.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-07-29T00:33:05.745 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:33:05.746 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:33:05.746 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:33:05.746 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:33:05.747 DEBUG:boto:Method: GET 2023-07-29T00:33:05.747 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:33:05.747 DEBUG:boto:Data: 2023-07-29T00:33:05.748 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.748 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:05.748 DEBUG:boto:Port: 8001 2023-07-29T00:33:05.749 DEBUG:boto:Params: {} 2023-07-29T00:33:05.749 DEBUG:boto:Token: None 2023-07-29T00:33:05.750 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:33:05.750 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xwtbb+EDXatE0+7Lkd1Q3K72UY4= 2023-07-29T00:33:05.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xwtbb+EDXatE0+7Lkd1Q3K72UY4=', 'Content-Length': '0'} 2023-07-29T00:33:05.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e15626d138c7e4af-0064c45e41-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.753 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:33:05.753 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:33:05.753 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:33:05.754 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:33:05.754 DEBUG:boto:Method: GET 2023-07-29T00:33:05.754 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:33:05.755 DEBUG:boto:Data: 2023-07-29T00:33:05.755 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.755 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:05.756 DEBUG:boto:Port: 8000 2023-07-29T00:33:05.756 DEBUG:boto:Params: {} 2023-07-29T00:33:05.757 DEBUG:boto:Token: None 2023-07-29T00:33:05.757 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:33:05.757 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xwtbb+EDXatE0+7Lkd1Q3K72UY4= 2023-07-29T00:33:05.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xwtbb+EDXatE0+7Lkd1Q3K72UY4=', 'Content-Length': '0'} 2023-07-29T00:33:05.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005561c45f35c10048-0064c45e41-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.760 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:33:05.760 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:33:05.761 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:33:05.761 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:33:05.761 DEBUG:boto:Method: HEAD 2023-07-29T00:33:05.762 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:33:05.762 DEBUG:boto:Data: 2023-07-29T00:33:05.762 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.763 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:05.763 DEBUG:boto:Port: 8001 2023-07-29T00:33:05.763 DEBUG:boto:Params: {} 2023-07-29T00:33:05.764 DEBUG:boto:Token: None 2023-07-29T00:33:05.764 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:33:05.765 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:g6oO+0vPD/ocshOlFeCg5KAJLmM= 2023-07-29T00:33:05.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:g6oO+0vPD/ocshOlFeCg5KAJLmM=', 'Content-Length': '0'} 2023-07-29T00:33:05.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095b5e814dc5a3a2d-0064c45e41-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.767 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:33:05.768 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:33:05.768 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:33:05.768 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:33:05.769 DEBUG:boto:Method: HEAD 2023-07-29T00:33:05.769 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:33:05.769 DEBUG:boto:Data: 2023-07-29T00:33:05.770 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.770 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:05.770 DEBUG:boto:Port: 8000 2023-07-29T00:33:05.771 DEBUG:boto:Params: {} 2023-07-29T00:33:05.771 DEBUG:boto:Token: None 2023-07-29T00:33:05.771 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:33:05.772 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:g6oO+0vPD/ocshOlFeCg5KAJLmM= 2023-07-29T00:33:05.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:g6oO+0vPD/ocshOlFeCg5KAJLmM=', 'Content-Length': '0'} 2023-07-29T00:33:05.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b3640f160ec20ef-0064c45e41-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-07-29T00:33:05.775 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:33:05.775 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:33:05.775 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:33:05.776 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:33:05.776 DEBUG:boto:Method: GET 2023-07-29T00:33:05.776 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:33:05.777 DEBUG:boto:Data: 2023-07-29T00:33:05.777 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.777 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:05.778 DEBUG:boto:Port: 8001 2023-07-29T00:33:05.778 DEBUG:boto:Params: {} 2023-07-29T00:33:05.778 DEBUG:boto:Token: None 2023-07-29T00:33:05.779 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:33:05.779 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xwtbb+EDXatE0+7Lkd1Q3K72UY4= 2023-07-29T00:33:05.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xwtbb+EDXatE0+7Lkd1Q3K72UY4=', 'Content-Length': '0'} 2023-07-29T00:33:05.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008267a19fb9aba7e8-0064c45e41-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.782 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:33:05.782 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:33:05.783 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:33:05.783 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:33:05.783 DEBUG:boto:Method: GET 2023-07-29T00:33:05.784 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:33:05.784 DEBUG:boto:Data: 2023-07-29T00:33:05.784 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.785 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:05.785 DEBUG:boto:Port: 8000 2023-07-29T00:33:05.785 DEBUG:boto:Params: {} 2023-07-29T00:33:05.786 DEBUG:boto:Token: None 2023-07-29T00:33:05.786 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:33:05.787 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xwtbb+EDXatE0+7Lkd1Q3K72UY4= 2023-07-29T00:33:05.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xwtbb+EDXatE0+7Lkd1Q3K72UY4=', 'Content-Length': '0'} 2023-07-29T00:33:05.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f7c9a827d16253a-0064c45e41-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.789 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:05.790 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:05.790 DEBUG:boto:path=/hiiqos-59/?prefix=obj55 2023-07-29T00:33:05.790 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj55 2023-07-29T00:33:05.791 DEBUG:boto:Method: GET 2023-07-29T00:33:05.791 DEBUG:boto:Path: /hiiqos-59/?prefix=obj55 2023-07-29T00:33:05.791 DEBUG:boto:Data: 2023-07-29T00:33:05.792 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.792 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:05.792 DEBUG:boto:Port: 8001 2023-07-29T00:33:05.793 DEBUG:boto:Params: {} 2023-07-29T00:33:05.793 DEBUG:boto:Token: None 2023-07-29T00:33:05.794 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/ 2023-07-29T00:33:05.794 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs= 2023-07-29T00:33:05.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs=', 'Content-Length': '0'} 2023-07-29T00:33:05.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000baeb99e317dcb32f-0064c45e41-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.797 DEBUG:boto:b'hiiqos-59obj551000falseobj552023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:05.797 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:05.798 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:05.798 DEBUG:boto:path=/hiiqos-59/?prefix=obj55 2023-07-29T00:33:05.799 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj55 2023-07-29T00:33:05.799 DEBUG:boto:Method: GET 2023-07-29T00:33:05.799 DEBUG:boto:Path: /hiiqos-59/?prefix=obj55 2023-07-29T00:33:05.800 DEBUG:boto:Data: 2023-07-29T00:33:05.800 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.800 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:05.801 DEBUG:boto:Port: 8000 2023-07-29T00:33:05.801 DEBUG:boto:Params: {} 2023-07-29T00:33:05.801 DEBUG:boto:Token: None 2023-07-29T00:33:05.802 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/ 2023-07-29T00:33:05.802 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs= 2023-07-29T00:33:05.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs=', 'Content-Length': '0'} 2023-07-29T00:33:05.805 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057540cb8c0cfb8a6-0064c45e41-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.848 DEBUG:boto:b'hiiqos-59obj551000falseobj552023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:05.849 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:33:05.849 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:33:05.850 DEBUG:boto:Method: HEAD 2023-07-29T00:33:05.850 DEBUG:boto:Path: /hiiqos-59/obj55 2023-07-29T00:33:05.850 DEBUG:boto:Data: 2023-07-29T00:33:05.851 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.851 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:05.851 DEBUG:boto:Port: 8001 2023-07-29T00:33:05.852 DEBUG:boto:Params: {} 2023-07-29T00:33:05.852 DEBUG:boto:Token: None 2023-07-29T00:33:05.852 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj55 2023-07-29T00:33:05.853 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cO1L8U7ucXyHckBuxFtYRW28LnE= 2023-07-29T00:33:05.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cO1L8U7ucXyHckBuxFtYRW28LnE=', 'Content-Length': '0'} 2023-07-29T00:33:05.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066bac99b472a496b-0064c45e41-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.855 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:33:05.856 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:33:05.856 DEBUG:boto:Method: HEAD 2023-07-29T00:33:05.856 DEBUG:boto:Path: /hiiqos-59/obj55 2023-07-29T00:33:05.857 DEBUG:boto:Data: 2023-07-29T00:33:05.857 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.857 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:05.858 DEBUG:boto:Port: 8000 2023-07-29T00:33:05.858 DEBUG:boto:Params: {} 2023-07-29T00:33:05.858 DEBUG:boto:Token: None 2023-07-29T00:33:05.859 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj55 2023-07-29T00:33:05.859 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cO1L8U7ucXyHckBuxFtYRW28LnE= 2023-07-29T00:33:05.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cO1L8U7ucXyHckBuxFtYRW28LnE=', 'Content-Length': '0'} 2023-07-29T00:33:05.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060efc7b1b881397b-0064c45e41-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-07-29T00:33:05.862 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:33:05.862 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:33:05.863 DEBUG:boto:Method: GET 2023-07-29T00:33:05.863 DEBUG:boto:Path: /hiiqos-59/obj55 2023-07-29T00:33:05.864 DEBUG:boto:Data: 2023-07-29T00:33:05.864 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.864 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:05.865 DEBUG:boto:Port: 8001 2023-07-29T00:33:05.865 DEBUG:boto:Params: {} 2023-07-29T00:33:05.865 DEBUG:boto:Token: None 2023-07-29T00:33:05.866 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj55 2023-07-29T00:33:05.866 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:A9MS06/SlsxWQM4My9ihF1wIDDs= 2023-07-29T00:33:05.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:A9MS06/SlsxWQM4My9ihF1wIDDs=', 'Content-Length': '0'} 2023-07-29T00:33:05.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023635a1911d37dd2-0064c45e41-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.869 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:33:05.869 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:33:05.869 DEBUG:boto:Method: GET 2023-07-29T00:33:05.870 DEBUG:boto:Path: /hiiqos-59/obj55 2023-07-29T00:33:05.870 DEBUG:boto:Data: 2023-07-29T00:33:05.870 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.871 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:05.871 DEBUG:boto:Port: 8000 2023-07-29T00:33:05.871 DEBUG:boto:Params: {} 2023-07-29T00:33:05.872 DEBUG:boto:Token: None 2023-07-29T00:33:05.872 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj55 2023-07-29T00:33:05.873 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:A9MS06/SlsxWQM4My9ihF1wIDDs= 2023-07-29T00:33:05.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:A9MS06/SlsxWQM4My9ihF1wIDDs=', 'Content-Length': '0'} 2023-07-29T00:33:05.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d05490997e73bf0d-0064c45e41-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.875 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:05.876 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:05.876 DEBUG:boto:path=/hiiqos-59/?prefix=obj55 2023-07-29T00:33:05.877 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj55 2023-07-29T00:33:05.877 DEBUG:boto:Method: GET 2023-07-29T00:33:05.877 DEBUG:boto:Path: /hiiqos-59/?prefix=obj55 2023-07-29T00:33:05.878 DEBUG:boto:Data: 2023-07-29T00:33:05.878 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.878 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:05.879 DEBUG:boto:Port: 8001 2023-07-29T00:33:05.879 DEBUG:boto:Params: {} 2023-07-29T00:33:05.879 DEBUG:boto:Token: None 2023-07-29T00:33:05.880 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/ 2023-07-29T00:33:05.880 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs= 2023-07-29T00:33:05.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs=', 'Content-Length': '0'} 2023-07-29T00:33:05.883 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c29423358cd9e60a-0064c45e41-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.883 DEBUG:boto:b'hiiqos-59obj551000falseobj552023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:05.884 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:05.884 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:05.885 DEBUG:boto:path=/hiiqos-59/?prefix=obj55 2023-07-29T00:33:05.885 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj55 2023-07-29T00:33:05.885 DEBUG:boto:Method: GET 2023-07-29T00:33:05.886 DEBUG:boto:Path: /hiiqos-59/?prefix=obj55 2023-07-29T00:33:05.886 DEBUG:boto:Data: 2023-07-29T00:33:05.886 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.887 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:05.887 DEBUG:boto:Port: 8000 2023-07-29T00:33:05.887 DEBUG:boto:Params: {} 2023-07-29T00:33:05.888 DEBUG:boto:Token: None 2023-07-29T00:33:05.888 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/ 2023-07-29T00:33:05.889 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs= 2023-07-29T00:33:05.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs=', 'Content-Length': '0'} 2023-07-29T00:33:05.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c864d1690450a164-0064c45e41-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.892 DEBUG:boto:b'hiiqos-59obj551000falseobj552023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:05.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-07-29T00:33:05.893 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:33:05.893 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:33:05.894 DEBUG:boto:path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:33:05.894 DEBUG:boto:auth_path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:33:05.894 DEBUG:boto:Method: GET 2023-07-29T00:33:05.895 DEBUG:boto:Path: /hiiqos-59/obj57?versionId=null 2023-07-29T00:33:05.895 DEBUG:boto:Data: 2023-07-29T00:33:05.895 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.896 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:05.896 DEBUG:boto:Port: 8001 2023-07-29T00:33:05.897 DEBUG:boto:Params: {} 2023-07-29T00:33:05.897 DEBUG:boto:Token: None 2023-07-29T00:33:05.897 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj57?versionId=null 2023-07-29T00:33:05.898 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lpEzDv6YihIBRyVkaUchUsdQCzI= 2023-07-29T00:33:05.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lpEzDv6YihIBRyVkaUchUsdQCzI=', 'Content-Length': '0'} 2023-07-29T00:33:05.900 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c26da779d068ed8a-0064c45e41-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.900 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:33:05.901 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:33:05.901 DEBUG:boto:path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:33:05.901 DEBUG:boto:auth_path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:33:05.902 DEBUG:boto:Method: GET 2023-07-29T00:33:05.902 DEBUG:boto:Path: /hiiqos-59/obj57?versionId=null 2023-07-29T00:33:05.903 DEBUG:boto:Data: 2023-07-29T00:33:05.903 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.903 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:05.904 DEBUG:boto:Port: 8000 2023-07-29T00:33:05.904 DEBUG:boto:Params: {} 2023-07-29T00:33:05.904 DEBUG:boto:Token: None 2023-07-29T00:33:05.905 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj57?versionId=null 2023-07-29T00:33:05.905 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lpEzDv6YihIBRyVkaUchUsdQCzI= 2023-07-29T00:33:05.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lpEzDv6YihIBRyVkaUchUsdQCzI=', 'Content-Length': '0'} 2023-07-29T00:33:05.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005be1fb25b2d86fd4-0064c45e41-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.952 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:33:05.953 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:33:05.953 DEBUG:boto:path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:33:05.953 DEBUG:boto:auth_path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:33:05.954 DEBUG:boto:Method: HEAD 2023-07-29T00:33:05.954 DEBUG:boto:Path: /hiiqos-59/obj57?versionId=null 2023-07-29T00:33:05.954 DEBUG:boto:Data: 2023-07-29T00:33:05.955 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.955 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:05.955 DEBUG:boto:Port: 8001 2023-07-29T00:33:05.956 DEBUG:boto:Params: {} 2023-07-29T00:33:05.956 DEBUG:boto:Token: None 2023-07-29T00:33:05.956 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj57?versionId=null 2023-07-29T00:33:05.957 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:v4hOCypiCdJDfr6zwyFEg/pPJ/0= 2023-07-29T00:33:05.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:v4hOCypiCdJDfr6zwyFEg/pPJ/0=', 'Content-Length': '0'} 2023-07-29T00:33:05.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005eb904bd8eea00ae-0064c45e41-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.959 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:33:05.960 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:33:05.960 DEBUG:boto:path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:33:05.960 DEBUG:boto:auth_path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:33:05.961 DEBUG:boto:Method: HEAD 2023-07-29T00:33:05.961 DEBUG:boto:Path: /hiiqos-59/obj57?versionId=null 2023-07-29T00:33:05.961 DEBUG:boto:Data: 2023-07-29T00:33:05.962 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.962 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:05.962 DEBUG:boto:Port: 8000 2023-07-29T00:33:05.963 DEBUG:boto:Params: {} 2023-07-29T00:33:05.963 DEBUG:boto:Token: None 2023-07-29T00:33:05.963 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj57?versionId=null 2023-07-29T00:33:05.964 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:v4hOCypiCdJDfr6zwyFEg/pPJ/0= 2023-07-29T00:33:05.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:v4hOCypiCdJDfr6zwyFEg/pPJ/0=', 'Content-Length': '0'} 2023-07-29T00:33:05.966 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000683b487df73d2194-0064c45e41-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-07-29T00:33:05.967 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:33:05.967 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:33:05.968 DEBUG:boto:path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:33:05.968 DEBUG:boto:auth_path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:33:05.968 DEBUG:boto:Method: GET 2023-07-29T00:33:05.969 DEBUG:boto:Path: /hiiqos-59/obj57?versionId=null 2023-07-29T00:33:05.969 DEBUG:boto:Data: 2023-07-29T00:33:05.969 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.970 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:05.970 DEBUG:boto:Port: 8001 2023-07-29T00:33:05.970 DEBUG:boto:Params: {} 2023-07-29T00:33:05.971 DEBUG:boto:Token: None 2023-07-29T00:33:05.971 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj57?versionId=null 2023-07-29T00:33:05.972 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lpEzDv6YihIBRyVkaUchUsdQCzI= 2023-07-29T00:33:05.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lpEzDv6YihIBRyVkaUchUsdQCzI=', 'Content-Length': '0'} 2023-07-29T00:33:05.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca8cd7839f01f6c3-0064c45e41-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.974 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:33:05.975 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:33:05.975 DEBUG:boto:path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:33:05.975 DEBUG:boto:auth_path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:33:05.976 DEBUG:boto:Method: GET 2023-07-29T00:33:05.976 DEBUG:boto:Path: /hiiqos-59/obj57?versionId=null 2023-07-29T00:33:05.976 DEBUG:boto:Data: 2023-07-29T00:33:05.977 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.977 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:05.977 DEBUG:boto:Port: 8000 2023-07-29T00:33:05.978 DEBUG:boto:Params: {} 2023-07-29T00:33:05.978 DEBUG:boto:Token: None 2023-07-29T00:33:05.979 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/obj57?versionId=null 2023-07-29T00:33:05.979 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lpEzDv6YihIBRyVkaUchUsdQCzI= 2023-07-29T00:33:05.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lpEzDv6YihIBRyVkaUchUsdQCzI=', 'Content-Length': '0'} 2023-07-29T00:33:05.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086c265586e0feada-0064c45e41-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.981 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:05.982 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:05.982 DEBUG:boto:path=/hiiqos-59/?prefix=obj57 2023-07-29T00:33:05.983 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj57 2023-07-29T00:33:05.983 DEBUG:boto:Method: GET 2023-07-29T00:33:05.983 DEBUG:boto:Path: /hiiqos-59/?prefix=obj57 2023-07-29T00:33:05.984 DEBUG:boto:Data: 2023-07-29T00:33:05.984 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.984 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:05.985 DEBUG:boto:Port: 8001 2023-07-29T00:33:05.985 DEBUG:boto:Params: {} 2023-07-29T00:33:05.985 DEBUG:boto:Token: None 2023-07-29T00:33:05.986 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/ 2023-07-29T00:33:05.986 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs= 2023-07-29T00:33:05.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs=', 'Content-Length': '0'} 2023-07-29T00:33:05.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000311b8cffcb1e77ab-0064c45e41-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.989 DEBUG:boto:b'hiiqos-59obj571000falseobj572023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:05.990 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:05.990 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:05.991 DEBUG:boto:path=/hiiqos-59/?prefix=obj57 2023-07-29T00:33:05.991 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj57 2023-07-29T00:33:05.991 DEBUG:boto:Method: GET 2023-07-29T00:33:05.992 DEBUG:boto:Path: /hiiqos-59/?prefix=obj57 2023-07-29T00:33:05.992 DEBUG:boto:Data: 2023-07-29T00:33:05.992 DEBUG:boto:Headers: {} 2023-07-29T00:33:05.993 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:05.993 DEBUG:boto:Port: 8000 2023-07-29T00:33:05.993 DEBUG:boto:Params: {} 2023-07-29T00:33:05.994 DEBUG:boto:Token: None 2023-07-29T00:33:05.994 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:05 GMT /hiiqos-59/ 2023-07-29T00:33:05.995 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs= 2023-07-29T00:33:05.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QXYTtdNQ7nWsWk2OpmOJmhgYDDs=', 'Content-Length': '0'} 2023-07-29T00:33:05.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024e3a3eca6b010ff-0064c45e41-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:05.998 DEBUG:boto:b'hiiqos-59obj571000falseobj572023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:05.999 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:33:05.999 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:33:06.000 DEBUG:boto:Method: HEAD 2023-07-29T00:33:06.000 DEBUG:boto:Path: /hiiqos-59/obj57 2023-07-29T00:33:06.000 DEBUG:boto:Data: 2023-07-29T00:33:06.001 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.001 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:06.001 DEBUG:boto:Port: 8001 2023-07-29T00:33:06.002 DEBUG:boto:Params: {} 2023-07-29T00:33:06.002 DEBUG:boto:Token: None 2023-07-29T00:33:06.002 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj57 2023-07-29T00:33:06.003 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qZMQbXNmZ4uhWkGMbcp5P9OMx/E= 2023-07-29T00:33:06.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qZMQbXNmZ4uhWkGMbcp5P9OMx/E=', 'Content-Length': '0'} 2023-07-29T00:33:06.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000998abb505ff6bfce-0064c45e41-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.005 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:33:06.006 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:33:06.006 DEBUG:boto:Method: HEAD 2023-07-29T00:33:06.006 DEBUG:boto:Path: /hiiqos-59/obj57 2023-07-29T00:33:06.007 DEBUG:boto:Data: 2023-07-29T00:33:06.007 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.007 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:06.008 DEBUG:boto:Port: 8000 2023-07-29T00:33:06.008 DEBUG:boto:Params: {} 2023-07-29T00:33:06.008 DEBUG:boto:Token: None 2023-07-29T00:33:06.009 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj57 2023-07-29T00:33:06.009 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qZMQbXNmZ4uhWkGMbcp5P9OMx/E= 2023-07-29T00:33:06.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qZMQbXNmZ4uhWkGMbcp5P9OMx/E=', 'Content-Length': '0'} 2023-07-29T00:33:06.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004635d2e6d67c38bd-0064c45e41-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-07-29T00:33:06.012 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:33:06.012 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:33:06.013 DEBUG:boto:Method: GET 2023-07-29T00:33:06.013 DEBUG:boto:Path: /hiiqos-59/obj57 2023-07-29T00:33:06.013 DEBUG:boto:Data: 2023-07-29T00:33:06.014 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.014 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:06.014 DEBUG:boto:Port: 8001 2023-07-29T00:33:06.015 DEBUG:boto:Params: {} 2023-07-29T00:33:06.015 DEBUG:boto:Token: None 2023-07-29T00:33:06.016 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj57 2023-07-29T00:33:06.016 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+0wJAp3/qJOHZQPta6CBmtVMqB8= 2023-07-29T00:33:06.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+0wJAp3/qJOHZQPta6CBmtVMqB8=', 'Content-Length': '0'} 2023-07-29T00:33:06.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082227b71c3251455-0064c45e41-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.060 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:33:06.061 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:33:06.061 DEBUG:boto:Method: GET 2023-07-29T00:33:06.061 DEBUG:boto:Path: /hiiqos-59/obj57 2023-07-29T00:33:06.062 DEBUG:boto:Data: 2023-07-29T00:33:06.062 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.062 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:06.063 DEBUG:boto:Port: 8000 2023-07-29T00:33:06.063 DEBUG:boto:Params: {} 2023-07-29T00:33:06.064 DEBUG:boto:Token: None 2023-07-29T00:33:06.064 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj57 2023-07-29T00:33:06.064 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+0wJAp3/qJOHZQPta6CBmtVMqB8= 2023-07-29T00:33:06.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+0wJAp3/qJOHZQPta6CBmtVMqB8=', 'Content-Length': '0'} 2023-07-29T00:33:06.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009a124e93e7831e0-0064c45e42-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.067 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:06.068 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:06.068 DEBUG:boto:path=/hiiqos-59/?prefix=obj57 2023-07-29T00:33:06.068 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj57 2023-07-29T00:33:06.069 DEBUG:boto:Method: GET 2023-07-29T00:33:06.069 DEBUG:boto:Path: /hiiqos-59/?prefix=obj57 2023-07-29T00:33:06.069 DEBUG:boto:Data: 2023-07-29T00:33:06.070 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.070 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:06.070 DEBUG:boto:Port: 8001 2023-07-29T00:33:06.071 DEBUG:boto:Params: {} 2023-07-29T00:33:06.071 DEBUG:boto:Token: None 2023-07-29T00:33:06.072 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/ 2023-07-29T00:33:06.072 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HeLMvBstJcmxLNdfMuXMmF6u0Vo= 2023-07-29T00:33:06.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HeLMvBstJcmxLNdfMuXMmF6u0Vo=', 'Content-Length': '0'} 2023-07-29T00:33:06.074 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b63f97edb9237ced-0064c45e42-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.075 DEBUG:boto:b'hiiqos-59obj571000falseobj572023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:06.075 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:06.076 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:06.076 DEBUG:boto:path=/hiiqos-59/?prefix=obj57 2023-07-29T00:33:06.077 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj57 2023-07-29T00:33:06.077 DEBUG:boto:Method: GET 2023-07-29T00:33:06.077 DEBUG:boto:Path: /hiiqos-59/?prefix=obj57 2023-07-29T00:33:06.078 DEBUG:boto:Data: 2023-07-29T00:33:06.078 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.078 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:06.079 DEBUG:boto:Port: 8000 2023-07-29T00:33:06.079 DEBUG:boto:Params: {} 2023-07-29T00:33:06.079 DEBUG:boto:Token: None 2023-07-29T00:33:06.080 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/ 2023-07-29T00:33:06.080 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HeLMvBstJcmxLNdfMuXMmF6u0Vo= 2023-07-29T00:33:06.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HeLMvBstJcmxLNdfMuXMmF6u0Vo=', 'Content-Length': '0'} 2023-07-29T00:33:06.084 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e334b54526027022-0064c45e42-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.084 DEBUG:boto:b'hiiqos-59obj571000falseobj572023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:06.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-07-29T00:33:06.085 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:33:06.086 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:33:06.086 DEBUG:boto:path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:33:06.086 DEBUG:boto:auth_path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:33:06.087 DEBUG:boto:Method: GET 2023-07-29T00:33:06.087 DEBUG:boto:Path: /hiiqos-59/obj68?versionId=null 2023-07-29T00:33:06.087 DEBUG:boto:Data: 2023-07-29T00:33:06.088 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.088 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:06.088 DEBUG:boto:Port: 8001 2023-07-29T00:33:06.089 DEBUG:boto:Params: {} 2023-07-29T00:33:06.089 DEBUG:boto:Token: None 2023-07-29T00:33:06.090 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj68?versionId=null 2023-07-29T00:33:06.090 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qVxNtvA02KBOjLYFKIMbZ9CrS0Q= 2023-07-29T00:33:06.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qVxNtvA02KBOjLYFKIMbZ9CrS0Q=', 'Content-Length': '0'} 2023-07-29T00:33:06.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c145be778ea38bc7-0064c45e42-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.093 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:33:06.093 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:33:06.093 DEBUG:boto:path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:33:06.094 DEBUG:boto:auth_path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:33:06.094 DEBUG:boto:Method: GET 2023-07-29T00:33:06.094 DEBUG:boto:Path: /hiiqos-59/obj68?versionId=null 2023-07-29T00:33:06.095 DEBUG:boto:Data: 2023-07-29T00:33:06.095 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.095 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:06.096 DEBUG:boto:Port: 8000 2023-07-29T00:33:06.096 DEBUG:boto:Params: {} 2023-07-29T00:33:06.097 DEBUG:boto:Token: None 2023-07-29T00:33:06.097 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj68?versionId=null 2023-07-29T00:33:06.097 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qVxNtvA02KBOjLYFKIMbZ9CrS0Q= 2023-07-29T00:33:06.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qVxNtvA02KBOjLYFKIMbZ9CrS0Q=', 'Content-Length': '0'} 2023-07-29T00:33:06.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5d58c455684e44b-0064c45e42-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.100 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:33:06.100 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:33:06.100 DEBUG:boto:path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:33:06.101 DEBUG:boto:auth_path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:33:06.101 DEBUG:boto:Method: HEAD 2023-07-29T00:33:06.101 DEBUG:boto:Path: /hiiqos-59/obj68?versionId=null 2023-07-29T00:33:06.102 DEBUG:boto:Data: 2023-07-29T00:33:06.102 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.102 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:06.103 DEBUG:boto:Port: 8001 2023-07-29T00:33:06.103 DEBUG:boto:Params: {} 2023-07-29T00:33:06.103 DEBUG:boto:Token: None 2023-07-29T00:33:06.104 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj68?versionId=null 2023-07-29T00:33:06.104 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IdovhVUQafb3N9iO5mDTegG+YJM= 2023-07-29T00:33:06.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IdovhVUQafb3N9iO5mDTegG+YJM=', 'Content-Length': '0'} 2023-07-29T00:33:06.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8aee551e1ea0a54-0064c45e42-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.107 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:33:06.107 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:33:06.107 DEBUG:boto:path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:33:06.108 DEBUG:boto:auth_path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:33:06.108 DEBUG:boto:Method: HEAD 2023-07-29T00:33:06.109 DEBUG:boto:Path: /hiiqos-59/obj68?versionId=null 2023-07-29T00:33:06.109 DEBUG:boto:Data: 2023-07-29T00:33:06.109 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.109 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:06.110 DEBUG:boto:Port: 8000 2023-07-29T00:33:06.110 DEBUG:boto:Params: {} 2023-07-29T00:33:06.110 DEBUG:boto:Token: None 2023-07-29T00:33:06.111 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj68?versionId=null 2023-07-29T00:33:06.111 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IdovhVUQafb3N9iO5mDTegG+YJM= 2023-07-29T00:33:06.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IdovhVUQafb3N9iO5mDTegG+YJM=', 'Content-Length': '0'} 2023-07-29T00:33:06.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000958f9784b60d4066-0064c45e42-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-07-29T00:33:06.114 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:33:06.114 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:33:06.115 DEBUG:boto:path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:33:06.115 DEBUG:boto:auth_path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:33:06.116 DEBUG:boto:Method: GET 2023-07-29T00:33:06.116 DEBUG:boto:Path: /hiiqos-59/obj68?versionId=null 2023-07-29T00:33:06.116 DEBUG:boto:Data: 2023-07-29T00:33:06.117 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.117 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:06.117 DEBUG:boto:Port: 8001 2023-07-29T00:33:06.118 DEBUG:boto:Params: {} 2023-07-29T00:33:06.118 DEBUG:boto:Token: None 2023-07-29T00:33:06.118 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj68?versionId=null 2023-07-29T00:33:06.119 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qVxNtvA02KBOjLYFKIMbZ9CrS0Q= 2023-07-29T00:33:06.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qVxNtvA02KBOjLYFKIMbZ9CrS0Q=', 'Content-Length': '0'} 2023-07-29T00:33:06.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008aeb5c6018502dc9-0064c45e42-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.121 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:33:06.122 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:33:06.122 DEBUG:boto:path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:33:06.122 DEBUG:boto:auth_path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:33:06.123 DEBUG:boto:Method: GET 2023-07-29T00:33:06.123 DEBUG:boto:Path: /hiiqos-59/obj68?versionId=null 2023-07-29T00:33:06.124 DEBUG:boto:Data: 2023-07-29T00:33:06.124 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.124 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:06.125 DEBUG:boto:Port: 8000 2023-07-29T00:33:06.125 DEBUG:boto:Params: {} 2023-07-29T00:33:06.125 DEBUG:boto:Token: None 2023-07-29T00:33:06.126 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj68?versionId=null 2023-07-29T00:33:06.126 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qVxNtvA02KBOjLYFKIMbZ9CrS0Q= 2023-07-29T00:33:06.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qVxNtvA02KBOjLYFKIMbZ9CrS0Q=', 'Content-Length': '0'} 2023-07-29T00:33:06.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f63d1fb674e95cf7-0064c45e42-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.129 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:06.129 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:06.130 DEBUG:boto:path=/hiiqos-59/?prefix=obj68 2023-07-29T00:33:06.130 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj68 2023-07-29T00:33:06.131 DEBUG:boto:Method: GET 2023-07-29T00:33:06.131 DEBUG:boto:Path: /hiiqos-59/?prefix=obj68 2023-07-29T00:33:06.131 DEBUG:boto:Data: 2023-07-29T00:33:06.132 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.132 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:06.132 DEBUG:boto:Port: 8001 2023-07-29T00:33:06.133 DEBUG:boto:Params: {} 2023-07-29T00:33:06.133 DEBUG:boto:Token: None 2023-07-29T00:33:06.134 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/ 2023-07-29T00:33:06.134 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HeLMvBstJcmxLNdfMuXMmF6u0Vo= 2023-07-29T00:33:06.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HeLMvBstJcmxLNdfMuXMmF6u0Vo=', 'Content-Length': '0'} 2023-07-29T00:33:06.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d908434f76d89296-0064c45e42-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.137 DEBUG:boto:b'hiiqos-59obj681000falseobj682023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:06.137 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:06.138 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:06.138 DEBUG:boto:path=/hiiqos-59/?prefix=obj68 2023-07-29T00:33:06.139 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj68 2023-07-29T00:33:06.139 DEBUG:boto:Method: GET 2023-07-29T00:33:06.139 DEBUG:boto:Path: /hiiqos-59/?prefix=obj68 2023-07-29T00:33:06.140 DEBUG:boto:Data: 2023-07-29T00:33:06.140 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.140 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:06.141 DEBUG:boto:Port: 8000 2023-07-29T00:33:06.141 DEBUG:boto:Params: {} 2023-07-29T00:33:06.141 DEBUG:boto:Token: None 2023-07-29T00:33:06.142 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/ 2023-07-29T00:33:06.142 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HeLMvBstJcmxLNdfMuXMmF6u0Vo= 2023-07-29T00:33:06.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HeLMvBstJcmxLNdfMuXMmF6u0Vo=', 'Content-Length': '0'} 2023-07-29T00:33:06.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071b3f97776afffea-0064c45e42-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.146 DEBUG:boto:b'hiiqos-59obj681000falseobj682023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:06.146 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:33:06.147 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:33:06.147 DEBUG:boto:Method: HEAD 2023-07-29T00:33:06.147 DEBUG:boto:Path: /hiiqos-59/obj68 2023-07-29T00:33:06.148 DEBUG:boto:Data: 2023-07-29T00:33:06.148 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.148 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:06.149 DEBUG:boto:Port: 8001 2023-07-29T00:33:06.149 DEBUG:boto:Params: {} 2023-07-29T00:33:06.149 DEBUG:boto:Token: None 2023-07-29T00:33:06.150 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj68 2023-07-29T00:33:06.150 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WBkyS/T2/oVBfGcM106+y2oIJLc= 2023-07-29T00:33:06.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WBkyS/T2/oVBfGcM106+y2oIJLc=', 'Content-Length': '0'} 2023-07-29T00:33:06.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b6b54b1795f063c-0064c45e42-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.153 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:33:06.153 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:33:06.154 DEBUG:boto:Method: HEAD 2023-07-29T00:33:06.154 DEBUG:boto:Path: /hiiqos-59/obj68 2023-07-29T00:33:06.154 DEBUG:boto:Data: 2023-07-29T00:33:06.155 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.155 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:06.155 DEBUG:boto:Port: 8000 2023-07-29T00:33:06.155 DEBUG:boto:Params: {} 2023-07-29T00:33:06.156 DEBUG:boto:Token: None 2023-07-29T00:33:06.156 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj68 2023-07-29T00:33:06.157 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WBkyS/T2/oVBfGcM106+y2oIJLc= 2023-07-29T00:33:06.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WBkyS/T2/oVBfGcM106+y2oIJLc=', 'Content-Length': '0'} 2023-07-29T00:33:06.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c8c199b719b5ef9-0064c45e42-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-07-29T00:33:06.160 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:33:06.160 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:33:06.160 DEBUG:boto:Method: GET 2023-07-29T00:33:06.161 DEBUG:boto:Path: /hiiqos-59/obj68 2023-07-29T00:33:06.161 DEBUG:boto:Data: 2023-07-29T00:33:06.161 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.162 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:06.162 DEBUG:boto:Port: 8001 2023-07-29T00:33:06.162 DEBUG:boto:Params: {} 2023-07-29T00:33:06.163 DEBUG:boto:Token: None 2023-07-29T00:33:06.163 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj68 2023-07-29T00:33:06.164 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IIPDsuxiXMeetDxVAT61VXq0cI8= 2023-07-29T00:33:06.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IIPDsuxiXMeetDxVAT61VXq0cI8=', 'Content-Length': '0'} 2023-07-29T00:33:06.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb3927f0fdf93f9b-0064c45e42-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.208 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:33:06.209 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:33:06.209 DEBUG:boto:Method: GET 2023-07-29T00:33:06.209 DEBUG:boto:Path: /hiiqos-59/obj68 2023-07-29T00:33:06.210 DEBUG:boto:Data: 2023-07-29T00:33:06.210 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.211 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:06.211 DEBUG:boto:Port: 8000 2023-07-29T00:33:06.211 DEBUG:boto:Params: {} 2023-07-29T00:33:06.212 DEBUG:boto:Token: None 2023-07-29T00:33:06.212 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj68 2023-07-29T00:33:06.213 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IIPDsuxiXMeetDxVAT61VXq0cI8= 2023-07-29T00:33:06.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IIPDsuxiXMeetDxVAT61VXq0cI8=', 'Content-Length': '0'} 2023-07-29T00:33:06.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060afe5f7f587cb75-0064c45e42-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.215 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:06.216 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:06.216 DEBUG:boto:path=/hiiqos-59/?prefix=obj68 2023-07-29T00:33:06.217 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj68 2023-07-29T00:33:06.217 DEBUG:boto:Method: GET 2023-07-29T00:33:06.217 DEBUG:boto:Path: /hiiqos-59/?prefix=obj68 2023-07-29T00:33:06.218 DEBUG:boto:Data: 2023-07-29T00:33:06.218 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.218 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:06.219 DEBUG:boto:Port: 8001 2023-07-29T00:33:06.219 DEBUG:boto:Params: {} 2023-07-29T00:33:06.219 DEBUG:boto:Token: None 2023-07-29T00:33:06.220 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/ 2023-07-29T00:33:06.220 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HeLMvBstJcmxLNdfMuXMmF6u0Vo= 2023-07-29T00:33:06.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HeLMvBstJcmxLNdfMuXMmF6u0Vo=', 'Content-Length': '0'} 2023-07-29T00:33:06.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c62f5a5b84f04cb9-0064c45e42-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.223 DEBUG:boto:b'hiiqos-59obj681000falseobj682023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:06.224 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:06.224 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:06.225 DEBUG:boto:path=/hiiqos-59/?prefix=obj68 2023-07-29T00:33:06.225 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj68 2023-07-29T00:33:06.225 DEBUG:boto:Method: GET 2023-07-29T00:33:06.226 DEBUG:boto:Path: /hiiqos-59/?prefix=obj68 2023-07-29T00:33:06.226 DEBUG:boto:Data: 2023-07-29T00:33:06.226 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.227 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:06.227 DEBUG:boto:Port: 8000 2023-07-29T00:33:06.228 DEBUG:boto:Params: {} 2023-07-29T00:33:06.228 DEBUG:boto:Token: None 2023-07-29T00:33:06.228 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/ 2023-07-29T00:33:06.229 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HeLMvBstJcmxLNdfMuXMmF6u0Vo= 2023-07-29T00:33:06.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HeLMvBstJcmxLNdfMuXMmF6u0Vo=', 'Content-Length': '0'} 2023-07-29T00:33:06.232 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007981d00d0cfe8fa2-0064c45e42-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.232 DEBUG:boto:b'hiiqos-59obj681000falseobj682023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:06.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-07-29T00:33:06.233 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:33:06.234 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:33:06.234 DEBUG:boto:path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:33:06.235 DEBUG:boto:auth_path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:33:06.235 DEBUG:boto:Method: GET 2023-07-29T00:33:06.235 DEBUG:boto:Path: /hiiqos-59/obj76?versionId=null 2023-07-29T00:33:06.236 DEBUG:boto:Data: 2023-07-29T00:33:06.236 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.236 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:06.237 DEBUG:boto:Port: 8001 2023-07-29T00:33:06.237 DEBUG:boto:Params: {} 2023-07-29T00:33:06.238 DEBUG:boto:Token: None 2023-07-29T00:33:06.238 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj76?versionId=null 2023-07-29T00:33:06.238 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CqOwrpJkhUiXEKnkaF7ZHDH9LRU= 2023-07-29T00:33:06.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CqOwrpJkhUiXEKnkaF7ZHDH9LRU=', 'Content-Length': '0'} 2023-07-29T00:33:06.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1b8e3293bb34e1b-0064c45e42-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.241 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:33:06.242 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:33:06.242 DEBUG:boto:path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:33:06.242 DEBUG:boto:auth_path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:33:06.243 DEBUG:boto:Method: GET 2023-07-29T00:33:06.243 DEBUG:boto:Path: /hiiqos-59/obj76?versionId=null 2023-07-29T00:33:06.243 DEBUG:boto:Data: 2023-07-29T00:33:06.244 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.244 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:06.244 DEBUG:boto:Port: 8000 2023-07-29T00:33:06.245 DEBUG:boto:Params: {} 2023-07-29T00:33:06.245 DEBUG:boto:Token: None 2023-07-29T00:33:06.246 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj76?versionId=null 2023-07-29T00:33:06.246 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CqOwrpJkhUiXEKnkaF7ZHDH9LRU= 2023-07-29T00:33:06.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CqOwrpJkhUiXEKnkaF7ZHDH9LRU=', 'Content-Length': '0'} 2023-07-29T00:33:06.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aaa5a63b17f4fa48-0064c45e42-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.249 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:33:06.249 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:33:06.250 DEBUG:boto:path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:33:06.250 DEBUG:boto:auth_path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:33:06.250 DEBUG:boto:Method: HEAD 2023-07-29T00:33:06.251 DEBUG:boto:Path: /hiiqos-59/obj76?versionId=null 2023-07-29T00:33:06.251 DEBUG:boto:Data: 2023-07-29T00:33:06.251 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.252 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:06.252 DEBUG:boto:Port: 8001 2023-07-29T00:33:06.252 DEBUG:boto:Params: {} 2023-07-29T00:33:06.253 DEBUG:boto:Token: None 2023-07-29T00:33:06.253 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj76?versionId=null 2023-07-29T00:33:06.253 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Wr8q/cNAwijbDvjAlpUNWDk7dsY= 2023-07-29T00:33:06.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Wr8q/cNAwijbDvjAlpUNWDk7dsY=', 'Content-Length': '0'} 2023-07-29T00:33:06.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9aa9a677856640d-0064c45e42-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.256 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:33:06.256 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:33:06.257 DEBUG:boto:path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:33:06.257 DEBUG:boto:auth_path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:33:06.257 DEBUG:boto:Method: HEAD 2023-07-29T00:33:06.258 DEBUG:boto:Path: /hiiqos-59/obj76?versionId=null 2023-07-29T00:33:06.258 DEBUG:boto:Data: 2023-07-29T00:33:06.258 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.258 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:06.259 DEBUG:boto:Port: 8000 2023-07-29T00:33:06.259 DEBUG:boto:Params: {} 2023-07-29T00:33:06.260 DEBUG:boto:Token: None 2023-07-29T00:33:06.260 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj76?versionId=null 2023-07-29T00:33:06.260 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Wr8q/cNAwijbDvjAlpUNWDk7dsY= 2023-07-29T00:33:06.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Wr8q/cNAwijbDvjAlpUNWDk7dsY=', 'Content-Length': '0'} 2023-07-29T00:33:06.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc90bbf092067bde-0064c45e42-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-07-29T00:33:06.263 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:33:06.264 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:33:06.264 DEBUG:boto:path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:33:06.264 DEBUG:boto:auth_path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:33:06.265 DEBUG:boto:Method: GET 2023-07-29T00:33:06.265 DEBUG:boto:Path: /hiiqos-59/obj76?versionId=null 2023-07-29T00:33:06.265 DEBUG:boto:Data: 2023-07-29T00:33:06.266 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.266 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:06.266 DEBUG:boto:Port: 8001 2023-07-29T00:33:06.267 DEBUG:boto:Params: {} 2023-07-29T00:33:06.267 DEBUG:boto:Token: None 2023-07-29T00:33:06.268 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj76?versionId=null 2023-07-29T00:33:06.268 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CqOwrpJkhUiXEKnkaF7ZHDH9LRU= 2023-07-29T00:33:06.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CqOwrpJkhUiXEKnkaF7ZHDH9LRU=', 'Content-Length': '0'} 2023-07-29T00:33:06.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a1d90978d899e51-0064c45e42-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.270 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:33:06.271 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:33:06.271 DEBUG:boto:path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:33:06.272 DEBUG:boto:auth_path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:33:06.272 DEBUG:boto:Method: GET 2023-07-29T00:33:06.272 DEBUG:boto:Path: /hiiqos-59/obj76?versionId=null 2023-07-29T00:33:06.273 DEBUG:boto:Data: 2023-07-29T00:33:06.273 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.273 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:06.274 DEBUG:boto:Port: 8000 2023-07-29T00:33:06.274 DEBUG:boto:Params: {} 2023-07-29T00:33:06.274 DEBUG:boto:Token: None 2023-07-29T00:33:06.275 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj76?versionId=null 2023-07-29T00:33:06.275 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CqOwrpJkhUiXEKnkaF7ZHDH9LRU= 2023-07-29T00:33:06.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CqOwrpJkhUiXEKnkaF7ZHDH9LRU=', 'Content-Length': '0'} 2023-07-29T00:33:06.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018356f22575121f4-0064c45e42-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.278 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:06.278 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:06.279 DEBUG:boto:path=/hiiqos-59/?prefix=obj76 2023-07-29T00:33:06.279 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj76 2023-07-29T00:33:06.279 DEBUG:boto:Method: GET 2023-07-29T00:33:06.280 DEBUG:boto:Path: /hiiqos-59/?prefix=obj76 2023-07-29T00:33:06.280 DEBUG:boto:Data: 2023-07-29T00:33:06.280 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.281 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:06.281 DEBUG:boto:Port: 8001 2023-07-29T00:33:06.281 DEBUG:boto:Params: {} 2023-07-29T00:33:06.282 DEBUG:boto:Token: None 2023-07-29T00:33:06.282 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/ 2023-07-29T00:33:06.283 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HeLMvBstJcmxLNdfMuXMmF6u0Vo= 2023-07-29T00:33:06.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HeLMvBstJcmxLNdfMuXMmF6u0Vo=', 'Content-Length': '0'} 2023-07-29T00:33:06.285 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f87991d788c93fcc-0064c45e42-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.286 DEBUG:boto:b'hiiqos-59obj761000falseobj762023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:06.286 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:06.287 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:06.287 DEBUG:boto:path=/hiiqos-59/?prefix=obj76 2023-07-29T00:33:06.288 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj76 2023-07-29T00:33:06.288 DEBUG:boto:Method: GET 2023-07-29T00:33:06.288 DEBUG:boto:Path: /hiiqos-59/?prefix=obj76 2023-07-29T00:33:06.289 DEBUG:boto:Data: 2023-07-29T00:33:06.289 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.289 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:06.290 DEBUG:boto:Port: 8000 2023-07-29T00:33:06.290 DEBUG:boto:Params: {} 2023-07-29T00:33:06.290 DEBUG:boto:Token: None 2023-07-29T00:33:06.291 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/ 2023-07-29T00:33:06.291 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HeLMvBstJcmxLNdfMuXMmF6u0Vo= 2023-07-29T00:33:06.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HeLMvBstJcmxLNdfMuXMmF6u0Vo=', 'Content-Length': '0'} 2023-07-29T00:33:06.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6b721e9e74d43d6-0064c45e42-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.295 DEBUG:boto:b'hiiqos-59obj761000falseobj762023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:06.296 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:33:06.296 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:33:06.296 DEBUG:boto:Method: HEAD 2023-07-29T00:33:06.297 DEBUG:boto:Path: /hiiqos-59/obj76 2023-07-29T00:33:06.297 DEBUG:boto:Data: 2023-07-29T00:33:06.297 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.298 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:06.298 DEBUG:boto:Port: 8001 2023-07-29T00:33:06.298 DEBUG:boto:Params: {} 2023-07-29T00:33:06.299 DEBUG:boto:Token: None 2023-07-29T00:33:06.299 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj76 2023-07-29T00:33:06.299 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2m5cVrh5xOhCsTMzoq/yaFn+4j4= 2023-07-29T00:33:06.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2m5cVrh5xOhCsTMzoq/yaFn+4j4=', 'Content-Length': '0'} 2023-07-29T00:33:06.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000246e5d4281a0a563-0064c45e42-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.302 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:33:06.302 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:33:06.303 DEBUG:boto:Method: HEAD 2023-07-29T00:33:06.303 DEBUG:boto:Path: /hiiqos-59/obj76 2023-07-29T00:33:06.303 DEBUG:boto:Data: 2023-07-29T00:33:06.304 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.304 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:06.304 DEBUG:boto:Port: 8000 2023-07-29T00:33:06.305 DEBUG:boto:Params: {} 2023-07-29T00:33:06.305 DEBUG:boto:Token: None 2023-07-29T00:33:06.305 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj76 2023-07-29T00:33:06.306 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2m5cVrh5xOhCsTMzoq/yaFn+4j4= 2023-07-29T00:33:06.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2m5cVrh5xOhCsTMzoq/yaFn+4j4=', 'Content-Length': '0'} 2023-07-29T00:33:06.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f54ce450c113402a-0064c45e42-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-07-29T00:33:06.309 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:33:06.309 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:33:06.310 DEBUG:boto:Method: GET 2023-07-29T00:33:06.310 DEBUG:boto:Path: /hiiqos-59/obj76 2023-07-29T00:33:06.310 DEBUG:boto:Data: 2023-07-29T00:33:06.311 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.311 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:06.312 DEBUG:boto:Port: 8001 2023-07-29T00:33:06.312 DEBUG:boto:Params: {} 2023-07-29T00:33:06.312 DEBUG:boto:Token: None 2023-07-29T00:33:06.313 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj76 2023-07-29T00:33:06.313 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:t3qXJ75TJLZX6It2XIIs1hYBDfM= 2023-07-29T00:33:06.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t3qXJ75TJLZX6It2XIIs1hYBDfM=', 'Content-Length': '0'} 2023-07-29T00:33:06.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001feb52ec25756041-0064c45e42-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.356 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:33:06.357 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:33:06.357 DEBUG:boto:Method: GET 2023-07-29T00:33:06.357 DEBUG:boto:Path: /hiiqos-59/obj76 2023-07-29T00:33:06.358 DEBUG:boto:Data: 2023-07-29T00:33:06.358 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.358 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:06.359 DEBUG:boto:Port: 8000 2023-07-29T00:33:06.359 DEBUG:boto:Params: {} 2023-07-29T00:33:06.360 DEBUG:boto:Token: None 2023-07-29T00:33:06.360 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj76 2023-07-29T00:33:06.360 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:t3qXJ75TJLZX6It2XIIs1hYBDfM= 2023-07-29T00:33:06.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t3qXJ75TJLZX6It2XIIs1hYBDfM=', 'Content-Length': '0'} 2023-07-29T00:33:06.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055228e414aca015b-0064c45e42-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.363 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:06.363 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:06.364 DEBUG:boto:path=/hiiqos-59/?prefix=obj76 2023-07-29T00:33:06.364 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj76 2023-07-29T00:33:06.365 DEBUG:boto:Method: GET 2023-07-29T00:33:06.365 DEBUG:boto:Path: /hiiqos-59/?prefix=obj76 2023-07-29T00:33:06.365 DEBUG:boto:Data: 2023-07-29T00:33:06.366 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.366 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:06.366 DEBUG:boto:Port: 8001 2023-07-29T00:33:06.367 DEBUG:boto:Params: {} 2023-07-29T00:33:06.367 DEBUG:boto:Token: None 2023-07-29T00:33:06.367 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/ 2023-07-29T00:33:06.368 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HeLMvBstJcmxLNdfMuXMmF6u0Vo= 2023-07-29T00:33:06.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HeLMvBstJcmxLNdfMuXMmF6u0Vo=', 'Content-Length': '0'} 2023-07-29T00:33:06.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057e16fa1434b49ec-0064c45e42-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.371 DEBUG:boto:b'hiiqos-59obj761000falseobj762023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:06.371 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:06.372 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:06.372 DEBUG:boto:path=/hiiqos-59/?prefix=obj76 2023-07-29T00:33:06.373 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj76 2023-07-29T00:33:06.373 DEBUG:boto:Method: GET 2023-07-29T00:33:06.373 DEBUG:boto:Path: /hiiqos-59/?prefix=obj76 2023-07-29T00:33:06.374 DEBUG:boto:Data: 2023-07-29T00:33:06.374 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.374 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:06.375 DEBUG:boto:Port: 8000 2023-07-29T00:33:06.375 DEBUG:boto:Params: {} 2023-07-29T00:33:06.376 DEBUG:boto:Token: None 2023-07-29T00:33:06.376 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/ 2023-07-29T00:33:06.376 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HeLMvBstJcmxLNdfMuXMmF6u0Vo= 2023-07-29T00:33:06.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HeLMvBstJcmxLNdfMuXMmF6u0Vo=', 'Content-Length': '0'} 2023-07-29T00:33:06.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee60941ed3ba749c-0064c45e42-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.380 DEBUG:boto:b'hiiqos-59obj761000falseobj762023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:06.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-07-29T00:33:06.381 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:33:06.381 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:33:06.381 DEBUG:boto:path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:33:06.382 DEBUG:boto:auth_path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:33:06.382 DEBUG:boto:Method: GET 2023-07-29T00:33:06.383 DEBUG:boto:Path: /hiiqos-59/obj85?versionId=null 2023-07-29T00:33:06.383 DEBUG:boto:Data: 2023-07-29T00:33:06.383 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.384 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:06.384 DEBUG:boto:Port: 8001 2023-07-29T00:33:06.384 DEBUG:boto:Params: {} 2023-07-29T00:33:06.385 DEBUG:boto:Token: None 2023-07-29T00:33:06.385 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj85?versionId=null 2023-07-29T00:33:06.386 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eBIs5/cIBMTRdn64HZVKWOZ1GKY= 2023-07-29T00:33:06.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eBIs5/cIBMTRdn64HZVKWOZ1GKY=', 'Content-Length': '0'} 2023-07-29T00:33:06.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068524d91457d5229-0064c45e42-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.388 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:33:06.389 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:33:06.389 DEBUG:boto:path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:33:06.389 DEBUG:boto:auth_path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:33:06.390 DEBUG:boto:Method: GET 2023-07-29T00:33:06.390 DEBUG:boto:Path: /hiiqos-59/obj85?versionId=null 2023-07-29T00:33:06.390 DEBUG:boto:Data: 2023-07-29T00:33:06.391 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.391 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:06.391 DEBUG:boto:Port: 8000 2023-07-29T00:33:06.392 DEBUG:boto:Params: {} 2023-07-29T00:33:06.392 DEBUG:boto:Token: None 2023-07-29T00:33:06.393 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj85?versionId=null 2023-07-29T00:33:06.393 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eBIs5/cIBMTRdn64HZVKWOZ1GKY= 2023-07-29T00:33:06.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eBIs5/cIBMTRdn64HZVKWOZ1GKY=', 'Content-Length': '0'} 2023-07-29T00:33:06.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073bee1daa2c1d7a0-0064c45e42-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.396 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:33:06.396 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:33:06.396 DEBUG:boto:path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:33:06.397 DEBUG:boto:auth_path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:33:06.397 DEBUG:boto:Method: HEAD 2023-07-29T00:33:06.397 DEBUG:boto:Path: /hiiqos-59/obj85?versionId=null 2023-07-29T00:33:06.398 DEBUG:boto:Data: 2023-07-29T00:33:06.398 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.398 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:06.399 DEBUG:boto:Port: 8001 2023-07-29T00:33:06.399 DEBUG:boto:Params: {} 2023-07-29T00:33:06.399 DEBUG:boto:Token: None 2023-07-29T00:33:06.400 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj85?versionId=null 2023-07-29T00:33:06.400 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kFKGlCudxt7R1Gym7Uusih4e/bU= 2023-07-29T00:33:06.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kFKGlCudxt7R1Gym7Uusih4e/bU=', 'Content-Length': '0'} 2023-07-29T00:33:06.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000c21c748cac199a-0064c45e42-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.403 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:33:06.403 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:33:06.404 DEBUG:boto:path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:33:06.404 DEBUG:boto:auth_path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:33:06.404 DEBUG:boto:Method: HEAD 2023-07-29T00:33:06.405 DEBUG:boto:Path: /hiiqos-59/obj85?versionId=null 2023-07-29T00:33:06.405 DEBUG:boto:Data: 2023-07-29T00:33:06.405 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.406 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:06.406 DEBUG:boto:Port: 8000 2023-07-29T00:33:06.406 DEBUG:boto:Params: {} 2023-07-29T00:33:06.407 DEBUG:boto:Token: None 2023-07-29T00:33:06.407 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj85?versionId=null 2023-07-29T00:33:06.407 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kFKGlCudxt7R1Gym7Uusih4e/bU= 2023-07-29T00:33:06.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kFKGlCudxt7R1Gym7Uusih4e/bU=', 'Content-Length': '0'} 2023-07-29T00:33:06.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eed86c1fc9e76861-0064c45e42-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-07-29T00:33:06.411 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:33:06.411 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:33:06.411 DEBUG:boto:path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:33:06.412 DEBUG:boto:auth_path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:33:06.412 DEBUG:boto:Method: GET 2023-07-29T00:33:06.412 DEBUG:boto:Path: /hiiqos-59/obj85?versionId=null 2023-07-29T00:33:06.413 DEBUG:boto:Data: 2023-07-29T00:33:06.413 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.413 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:06.414 DEBUG:boto:Port: 8001 2023-07-29T00:33:06.414 DEBUG:boto:Params: {} 2023-07-29T00:33:06.415 DEBUG:boto:Token: None 2023-07-29T00:33:06.415 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj85?versionId=null 2023-07-29T00:33:06.415 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eBIs5/cIBMTRdn64HZVKWOZ1GKY= 2023-07-29T00:33:06.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eBIs5/cIBMTRdn64HZVKWOZ1GKY=', 'Content-Length': '0'} 2023-07-29T00:33:06.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062cd1b57044919bc-0064c45e42-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.418 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:33:06.418 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:33:06.419 DEBUG:boto:path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:33:06.419 DEBUG:boto:auth_path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:33:06.420 DEBUG:boto:Method: GET 2023-07-29T00:33:06.420 DEBUG:boto:Path: /hiiqos-59/obj85?versionId=null 2023-07-29T00:33:06.420 DEBUG:boto:Data: 2023-07-29T00:33:06.421 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.421 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:06.421 DEBUG:boto:Port: 8000 2023-07-29T00:33:06.422 DEBUG:boto:Params: {} 2023-07-29T00:33:06.422 DEBUG:boto:Token: None 2023-07-29T00:33:06.422 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj85?versionId=null 2023-07-29T00:33:06.423 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eBIs5/cIBMTRdn64HZVKWOZ1GKY= 2023-07-29T00:33:06.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eBIs5/cIBMTRdn64HZVKWOZ1GKY=', 'Content-Length': '0'} 2023-07-29T00:33:06.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ad43625590d3482-0064c45e42-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.426 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:06.426 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:06.426 DEBUG:boto:path=/hiiqos-59/?prefix=obj85 2023-07-29T00:33:06.427 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj85 2023-07-29T00:33:06.427 DEBUG:boto:Method: GET 2023-07-29T00:33:06.427 DEBUG:boto:Path: /hiiqos-59/?prefix=obj85 2023-07-29T00:33:06.428 DEBUG:boto:Data: 2023-07-29T00:33:06.428 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.429 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:06.429 DEBUG:boto:Port: 8001 2023-07-29T00:33:06.429 DEBUG:boto:Params: {} 2023-07-29T00:33:06.430 DEBUG:boto:Token: None 2023-07-29T00:33:06.430 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/ 2023-07-29T00:33:06.431 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HeLMvBstJcmxLNdfMuXMmF6u0Vo= 2023-07-29T00:33:06.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HeLMvBstJcmxLNdfMuXMmF6u0Vo=', 'Content-Length': '0'} 2023-07-29T00:33:06.433 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1102448b13efd63-0064c45e42-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.434 DEBUG:boto:b'hiiqos-59obj851000falseobj852023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:06.434 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:06.435 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:06.435 DEBUG:boto:path=/hiiqos-59/?prefix=obj85 2023-07-29T00:33:06.435 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj85 2023-07-29T00:33:06.436 DEBUG:boto:Method: GET 2023-07-29T00:33:06.436 DEBUG:boto:Path: /hiiqos-59/?prefix=obj85 2023-07-29T00:33:06.436 DEBUG:boto:Data: 2023-07-29T00:33:06.437 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.437 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:06.437 DEBUG:boto:Port: 8000 2023-07-29T00:33:06.438 DEBUG:boto:Params: {} 2023-07-29T00:33:06.438 DEBUG:boto:Token: None 2023-07-29T00:33:06.439 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/ 2023-07-29T00:33:06.439 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HeLMvBstJcmxLNdfMuXMmF6u0Vo= 2023-07-29T00:33:06.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HeLMvBstJcmxLNdfMuXMmF6u0Vo=', 'Content-Length': '0'} 2023-07-29T00:33:06.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b25d9856740510a6-0064c45e42-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.488 DEBUG:boto:b'hiiqos-59obj851000falseobj852023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:06.489 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:33:06.489 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:33:06.490 DEBUG:boto:Method: HEAD 2023-07-29T00:33:06.490 DEBUG:boto:Path: /hiiqos-59/obj85 2023-07-29T00:33:06.490 DEBUG:boto:Data: 2023-07-29T00:33:06.491 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.491 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:06.491 DEBUG:boto:Port: 8001 2023-07-29T00:33:06.492 DEBUG:boto:Params: {} 2023-07-29T00:33:06.492 DEBUG:boto:Token: None 2023-07-29T00:33:06.492 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj85 2023-07-29T00:33:06.493 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I2mpoxiAcy+PZmkB2z7rHRUzRKY= 2023-07-29T00:33:06.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I2mpoxiAcy+PZmkB2z7rHRUzRKY=', 'Content-Length': '0'} 2023-07-29T00:33:06.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000002482de3985aff7-0064c45e42-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.495 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:33:06.496 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:33:06.496 DEBUG:boto:Method: HEAD 2023-07-29T00:33:06.496 DEBUG:boto:Path: /hiiqos-59/obj85 2023-07-29T00:33:06.497 DEBUG:boto:Data: 2023-07-29T00:33:06.497 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.497 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:06.498 DEBUG:boto:Port: 8000 2023-07-29T00:33:06.498 DEBUG:boto:Params: {} 2023-07-29T00:33:06.498 DEBUG:boto:Token: None 2023-07-29T00:33:06.499 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj85 2023-07-29T00:33:06.499 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I2mpoxiAcy+PZmkB2z7rHRUzRKY= 2023-07-29T00:33:06.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I2mpoxiAcy+PZmkB2z7rHRUzRKY=', 'Content-Length': '0'} 2023-07-29T00:33:06.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c660cb24e011d01-0064c45e42-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-07-29T00:33:06.502 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:33:06.503 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:33:06.503 DEBUG:boto:Method: GET 2023-07-29T00:33:06.503 DEBUG:boto:Path: /hiiqos-59/obj85 2023-07-29T00:33:06.504 DEBUG:boto:Data: 2023-07-29T00:33:06.504 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.504 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:06.505 DEBUG:boto:Port: 8001 2023-07-29T00:33:06.505 DEBUG:boto:Params: {} 2023-07-29T00:33:06.505 DEBUG:boto:Token: None 2023-07-29T00:33:06.506 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj85 2023-07-29T00:33:06.506 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6ITil5Za6Z7r3HCnGWmXw5BUgMA= 2023-07-29T00:33:06.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6ITil5Za6Z7r3HCnGWmXw5BUgMA=', 'Content-Length': '0'} 2023-07-29T00:33:06.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069952bb35a7d30ee-0064c45e42-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.509 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:33:06.509 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:33:06.510 DEBUG:boto:Method: GET 2023-07-29T00:33:06.510 DEBUG:boto:Path: /hiiqos-59/obj85 2023-07-29T00:33:06.510 DEBUG:boto:Data: 2023-07-29T00:33:06.511 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.511 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:06.511 DEBUG:boto:Port: 8000 2023-07-29T00:33:06.512 DEBUG:boto:Params: {} 2023-07-29T00:33:06.512 DEBUG:boto:Token: None 2023-07-29T00:33:06.513 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj85 2023-07-29T00:33:06.513 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6ITil5Za6Z7r3HCnGWmXw5BUgMA= 2023-07-29T00:33:06.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6ITil5Za6Z7r3HCnGWmXw5BUgMA=', 'Content-Length': '0'} 2023-07-29T00:33:06.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4e213ce797f2beb-0064c45e42-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.516 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:06.516 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:06.516 DEBUG:boto:path=/hiiqos-59/?prefix=obj85 2023-07-29T00:33:06.517 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj85 2023-07-29T00:33:06.517 DEBUG:boto:Method: GET 2023-07-29T00:33:06.517 DEBUG:boto:Path: /hiiqos-59/?prefix=obj85 2023-07-29T00:33:06.518 DEBUG:boto:Data: 2023-07-29T00:33:06.518 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.519 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:06.519 DEBUG:boto:Port: 8001 2023-07-29T00:33:06.519 DEBUG:boto:Params: {} 2023-07-29T00:33:06.520 DEBUG:boto:Token: None 2023-07-29T00:33:06.520 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/ 2023-07-29T00:33:06.520 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HeLMvBstJcmxLNdfMuXMmF6u0Vo= 2023-07-29T00:33:06.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HeLMvBstJcmxLNdfMuXMmF6u0Vo=', 'Content-Length': '0'} 2023-07-29T00:33:06.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4c587a55585acf4-0064c45e42-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.523 DEBUG:boto:b'hiiqos-59obj851000falseobj852023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:06.524 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:06.524 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:06.525 DEBUG:boto:path=/hiiqos-59/?prefix=obj85 2023-07-29T00:33:06.525 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj85 2023-07-29T00:33:06.525 DEBUG:boto:Method: GET 2023-07-29T00:33:06.526 DEBUG:boto:Path: /hiiqos-59/?prefix=obj85 2023-07-29T00:33:06.526 DEBUG:boto:Data: 2023-07-29T00:33:06.526 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.527 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:06.527 DEBUG:boto:Port: 8000 2023-07-29T00:33:06.528 DEBUG:boto:Params: {} 2023-07-29T00:33:06.528 DEBUG:boto:Token: None 2023-07-29T00:33:06.528 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/ 2023-07-29T00:33:06.529 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HeLMvBstJcmxLNdfMuXMmF6u0Vo= 2023-07-29T00:33:06.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HeLMvBstJcmxLNdfMuXMmF6u0Vo=', 'Content-Length': '0'} 2023-07-29T00:33:06.532 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbb42b403439f3fe-0064c45e42-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.532 DEBUG:boto:b'hiiqos-59obj851000falseobj852023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:06.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-07-29T00:33:06.533 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:33:06.534 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:33:06.534 DEBUG:boto:path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:33:06.534 DEBUG:boto:auth_path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:33:06.535 DEBUG:boto:Method: GET 2023-07-29T00:33:06.535 DEBUG:boto:Path: /hiiqos-59/obj95?versionId=null 2023-07-29T00:33:06.535 DEBUG:boto:Data: 2023-07-29T00:33:06.536 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.536 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:06.536 DEBUG:boto:Port: 8001 2023-07-29T00:33:06.537 DEBUG:boto:Params: {} 2023-07-29T00:33:06.537 DEBUG:boto:Token: None 2023-07-29T00:33:06.538 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj95?versionId=null 2023-07-29T00:33:06.538 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:84z8bL3VyedVaNLcoBNCSJeMdSw= 2023-07-29T00:33:06.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:84z8bL3VyedVaNLcoBNCSJeMdSw=', 'Content-Length': '0'} 2023-07-29T00:33:06.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bcdd78dbed01258-0064c45e42-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.541 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:33:06.541 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:33:06.542 DEBUG:boto:path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:33:06.542 DEBUG:boto:auth_path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:33:06.542 DEBUG:boto:Method: GET 2023-07-29T00:33:06.543 DEBUG:boto:Path: /hiiqos-59/obj95?versionId=null 2023-07-29T00:33:06.543 DEBUG:boto:Data: 2023-07-29T00:33:06.544 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.544 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:06.544 DEBUG:boto:Port: 8000 2023-07-29T00:33:06.545 DEBUG:boto:Params: {} 2023-07-29T00:33:06.545 DEBUG:boto:Token: None 2023-07-29T00:33:06.546 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj95?versionId=null 2023-07-29T00:33:06.546 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:84z8bL3VyedVaNLcoBNCSJeMdSw= 2023-07-29T00:33:06.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:84z8bL3VyedVaNLcoBNCSJeMdSw=', 'Content-Length': '0'} 2023-07-29T00:33:06.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025aba61bb99ec062-0064c45e42-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.549 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:33:06.549 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:33:06.549 DEBUG:boto:path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:33:06.550 DEBUG:boto:auth_path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:33:06.550 DEBUG:boto:Method: HEAD 2023-07-29T00:33:06.550 DEBUG:boto:Path: /hiiqos-59/obj95?versionId=null 2023-07-29T00:33:06.551 DEBUG:boto:Data: 2023-07-29T00:33:06.551 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.551 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:06.552 DEBUG:boto:Port: 8001 2023-07-29T00:33:06.552 DEBUG:boto:Params: {} 2023-07-29T00:33:06.553 DEBUG:boto:Token: None 2023-07-29T00:33:06.553 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj95?versionId=null 2023-07-29T00:33:06.553 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kt7fNDTtScg5vFx3G5jmmzr5ChE= 2023-07-29T00:33:06.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kt7fNDTtScg5vFx3G5jmmzr5ChE=', 'Content-Length': '0'} 2023-07-29T00:33:06.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000858b6eef7414f441-0064c45e42-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.556 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:33:06.556 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:33:06.557 DEBUG:boto:path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:33:06.557 DEBUG:boto:auth_path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:33:06.557 DEBUG:boto:Method: HEAD 2023-07-29T00:33:06.558 DEBUG:boto:Path: /hiiqos-59/obj95?versionId=null 2023-07-29T00:33:06.558 DEBUG:boto:Data: 2023-07-29T00:33:06.558 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.559 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:06.559 DEBUG:boto:Port: 8000 2023-07-29T00:33:06.559 DEBUG:boto:Params: {} 2023-07-29T00:33:06.560 DEBUG:boto:Token: None 2023-07-29T00:33:06.560 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj95?versionId=null 2023-07-29T00:33:06.560 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kt7fNDTtScg5vFx3G5jmmzr5ChE= 2023-07-29T00:33:06.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kt7fNDTtScg5vFx3G5jmmzr5ChE=', 'Content-Length': '0'} 2023-07-29T00:33:06.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004733b28c095196b-0064c45e42-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-07-29T00:33:06.563 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:33:06.563 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:33:06.564 DEBUG:boto:path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:33:06.564 DEBUG:boto:auth_path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:33:06.565 DEBUG:boto:Method: GET 2023-07-29T00:33:06.565 DEBUG:boto:Path: /hiiqos-59/obj95?versionId=null 2023-07-29T00:33:06.565 DEBUG:boto:Data: 2023-07-29T00:33:06.566 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.566 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:06.566 DEBUG:boto:Port: 8001 2023-07-29T00:33:06.567 DEBUG:boto:Params: {} 2023-07-29T00:33:06.567 DEBUG:boto:Token: None 2023-07-29T00:33:06.567 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj95?versionId=null 2023-07-29T00:33:06.568 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:84z8bL3VyedVaNLcoBNCSJeMdSw= 2023-07-29T00:33:06.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:84z8bL3VyedVaNLcoBNCSJeMdSw=', 'Content-Length': '0'} 2023-07-29T00:33:06.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008dcb79e89c3593b1-0064c45e42-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.612 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:33:06.613 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:33:06.613 DEBUG:boto:path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:33:06.613 DEBUG:boto:auth_path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:33:06.614 DEBUG:boto:Method: GET 2023-07-29T00:33:06.614 DEBUG:boto:Path: /hiiqos-59/obj95?versionId=null 2023-07-29T00:33:06.614 DEBUG:boto:Data: 2023-07-29T00:33:06.615 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.615 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:06.615 DEBUG:boto:Port: 8000 2023-07-29T00:33:06.616 DEBUG:boto:Params: {} 2023-07-29T00:33:06.616 DEBUG:boto:Token: None 2023-07-29T00:33:06.617 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj95?versionId=null 2023-07-29T00:33:06.617 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:84z8bL3VyedVaNLcoBNCSJeMdSw= 2023-07-29T00:33:06.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:84z8bL3VyedVaNLcoBNCSJeMdSw=', 'Content-Length': '0'} 2023-07-29T00:33:06.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9e65110f812e3f8-0064c45e42-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.620 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:06.620 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:06.621 DEBUG:boto:path=/hiiqos-59/?prefix=obj95 2023-07-29T00:33:06.621 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj95 2023-07-29T00:33:06.621 DEBUG:boto:Method: GET 2023-07-29T00:33:06.622 DEBUG:boto:Path: /hiiqos-59/?prefix=obj95 2023-07-29T00:33:06.622 DEBUG:boto:Data: 2023-07-29T00:33:06.622 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.623 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:06.623 DEBUG:boto:Port: 8001 2023-07-29T00:33:06.623 DEBUG:boto:Params: {} 2023-07-29T00:33:06.624 DEBUG:boto:Token: None 2023-07-29T00:33:06.624 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/ 2023-07-29T00:33:06.625 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HeLMvBstJcmxLNdfMuXMmF6u0Vo= 2023-07-29T00:33:06.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HeLMvBstJcmxLNdfMuXMmF6u0Vo=', 'Content-Length': '0'} 2023-07-29T00:33:06.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1b5981e8b58fbe9-0064c45e42-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.627 DEBUG:boto:b'hiiqos-59obj951000falseobj952023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:06.628 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:06.628 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:06.629 DEBUG:boto:path=/hiiqos-59/?prefix=obj95 2023-07-29T00:33:06.629 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj95 2023-07-29T00:33:06.629 DEBUG:boto:Method: GET 2023-07-29T00:33:06.630 DEBUG:boto:Path: /hiiqos-59/?prefix=obj95 2023-07-29T00:33:06.630 DEBUG:boto:Data: 2023-07-29T00:33:06.631 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.631 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:06.631 DEBUG:boto:Port: 8000 2023-07-29T00:33:06.632 DEBUG:boto:Params: {} 2023-07-29T00:33:06.632 DEBUG:boto:Token: None 2023-07-29T00:33:06.632 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/ 2023-07-29T00:33:06.633 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HeLMvBstJcmxLNdfMuXMmF6u0Vo= 2023-07-29T00:33:06.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HeLMvBstJcmxLNdfMuXMmF6u0Vo=', 'Content-Length': '0'} 2023-07-29T00:33:06.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013aa71f16793da21-0064c45e42-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.636 DEBUG:boto:b'hiiqos-59obj951000falseobj952023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:06.637 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:33:06.637 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:33:06.638 DEBUG:boto:Method: HEAD 2023-07-29T00:33:06.638 DEBUG:boto:Path: /hiiqos-59/obj95 2023-07-29T00:33:06.638 DEBUG:boto:Data: 2023-07-29T00:33:06.639 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.639 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:06.639 DEBUG:boto:Port: 8001 2023-07-29T00:33:06.640 DEBUG:boto:Params: {} 2023-07-29T00:33:06.640 DEBUG:boto:Token: None 2023-07-29T00:33:06.640 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj95 2023-07-29T00:33:06.641 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:s+vvf5HW/apBfuHwsP3OtBrrAY4= 2023-07-29T00:33:06.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:s+vvf5HW/apBfuHwsP3OtBrrAY4=', 'Content-Length': '0'} 2023-07-29T00:33:06.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad56360038905afc-0064c45e42-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.644 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:33:06.644 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:33:06.644 DEBUG:boto:Method: HEAD 2023-07-29T00:33:06.645 DEBUG:boto:Path: /hiiqos-59/obj95 2023-07-29T00:33:06.645 DEBUG:boto:Data: 2023-07-29T00:33:06.645 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.646 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:06.646 DEBUG:boto:Port: 8000 2023-07-29T00:33:06.646 DEBUG:boto:Params: {} 2023-07-29T00:33:06.647 DEBUG:boto:Token: None 2023-07-29T00:33:06.647 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj95 2023-07-29T00:33:06.647 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:s+vvf5HW/apBfuHwsP3OtBrrAY4= 2023-07-29T00:33:06.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:s+vvf5HW/apBfuHwsP3OtBrrAY4=', 'Content-Length': '0'} 2023-07-29T00:33:06.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3cf3ed2671053ea-0064c45e42-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-07-29T00:33:06.650 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:33:06.650 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:33:06.651 DEBUG:boto:Method: GET 2023-07-29T00:33:06.651 DEBUG:boto:Path: /hiiqos-59/obj95 2023-07-29T00:33:06.652 DEBUG:boto:Data: 2023-07-29T00:33:06.652 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.652 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:06.653 DEBUG:boto:Port: 8001 2023-07-29T00:33:06.653 DEBUG:boto:Params: {} 2023-07-29T00:33:06.653 DEBUG:boto:Token: None 2023-07-29T00:33:06.654 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj95 2023-07-29T00:33:06.654 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:s/w8xTsF2VrSMAPFNlo/v2oeJtE= 2023-07-29T00:33:06.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:s/w8xTsF2VrSMAPFNlo/v2oeJtE=', 'Content-Length': '0'} 2023-07-29T00:33:06.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055a63647cfd3e743-0064c45e42-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.657 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:33:06.657 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:33:06.658 DEBUG:boto:Method: GET 2023-07-29T00:33:06.658 DEBUG:boto:Path: /hiiqos-59/obj95 2023-07-29T00:33:06.658 DEBUG:boto:Data: 2023-07-29T00:33:06.659 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.659 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:06.659 DEBUG:boto:Port: 8000 2023-07-29T00:33:06.660 DEBUG:boto:Params: {} 2023-07-29T00:33:06.660 DEBUG:boto:Token: None 2023-07-29T00:33:06.661 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj95 2023-07-29T00:33:06.661 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:s/w8xTsF2VrSMAPFNlo/v2oeJtE= 2023-07-29T00:33:06.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:s/w8xTsF2VrSMAPFNlo/v2oeJtE=', 'Content-Length': '0'} 2023-07-29T00:33:06.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034a83810cd51a139-0064c45e42-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.664 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:06.664 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:06.665 DEBUG:boto:path=/hiiqos-59/?prefix=obj95 2023-07-29T00:33:06.665 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj95 2023-07-29T00:33:06.665 DEBUG:boto:Method: GET 2023-07-29T00:33:06.666 DEBUG:boto:Path: /hiiqos-59/?prefix=obj95 2023-07-29T00:33:06.666 DEBUG:boto:Data: 2023-07-29T00:33:06.666 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.667 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:33:06.667 DEBUG:boto:Port: 8001 2023-07-29T00:33:06.667 DEBUG:boto:Params: {} 2023-07-29T00:33:06.668 DEBUG:boto:Token: None 2023-07-29T00:33:06.668 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/ 2023-07-29T00:33:06.669 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HeLMvBstJcmxLNdfMuXMmF6u0Vo= 2023-07-29T00:33:06.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HeLMvBstJcmxLNdfMuXMmF6u0Vo=', 'Content-Length': '0'} 2023-07-29T00:33:06.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c379fd296dd281d8-0064c45e42-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.671 DEBUG:boto:b'hiiqos-59obj951000falseobj952023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:06.672 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:06.672 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:06.673 DEBUG:boto:path=/hiiqos-59/?prefix=obj95 2023-07-29T00:33:06.673 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj95 2023-07-29T00:33:06.673 DEBUG:boto:Method: GET 2023-07-29T00:33:06.674 DEBUG:boto:Path: /hiiqos-59/?prefix=obj95 2023-07-29T00:33:06.674 DEBUG:boto:Data: 2023-07-29T00:33:06.675 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.675 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:06.675 DEBUG:boto:Port: 8000 2023-07-29T00:33:06.676 DEBUG:boto:Params: {} 2023-07-29T00:33:06.676 DEBUG:boto:Token: None 2023-07-29T00:33:06.676 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/ 2023-07-29T00:33:06.677 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HeLMvBstJcmxLNdfMuXMmF6u0Vo= 2023-07-29T00:33:06.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HeLMvBstJcmxLNdfMuXMmF6u0Vo=', 'Content-Length': '0'} 2023-07-29T00:33:06.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000356fb31d1af2577d-0064c45e42-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.680 DEBUG:boto:b'hiiqos-59obj951000falseobj952023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:33:06.681 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-59 zones={a2, a1} 2023-07-29T00:33:06.681 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:33:06.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:33:06.683 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:33:06.733 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:33:06.733 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:33:06.776 INFO:teuthology.orchestra.run.smithi101.stdout:tenant: 2023-07-29T00:33:06.777 INFO:teuthology.orchestra.run.smithi101.stdout:bucket name: hiiqos-59 2023-07-29T00:33:06.783 INFO:teuthology.orchestra.run.smithi101.stdout:total entries: 15 2023-07-29T00:33:06.827 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:06.828 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:06.828 DEBUG:boto:Method: HEAD 2023-07-29T00:33:06.829 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:33:06.829 DEBUG:boto:Data: 2023-07-29T00:33:06.830 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.830 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:06.830 DEBUG:boto:Port: 8000 2023-07-29T00:33:06.831 DEBUG:boto:Params: {} 2023-07-29T00:33:06.831 DEBUG:boto:Token: None 2023-07-29T00:33:06.831 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/ 2023-07-29T00:33:06.832 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Q0qVzpK5QesBKcLAopwS4/E0arA= 2023-07-29T00:33:06.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q0qVzpK5QesBKcLAopwS4/E0arA=', 'Content-Length': '0'} 2023-07-29T00:33:06.835 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000940d6a5da95077a-0064c45e42-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.835 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:33:06.836 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:33:06.836 DEBUG:boto:Method: PUT 2023-07-29T00:33:06.837 DEBUG:boto:Path: /hiiqos-59/obj23 2023-07-29T00:33:06.837 DEBUG:boto:Data: 2023-07-29T00:33:06.837 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:33:06.838 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:06.838 DEBUG:boto:Port: 8000 2023-07-29T00:33:06.838 DEBUG:boto:Params: {} 2023-07-29T00:33:06.839 DEBUG:boto:Token: None 2023-07-29T00:33:06.839 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj23 2023-07-29T00:33:06.840 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IXHH1fE+niq89S9rjsmxHAP2Ytg= 2023-07-29T00:33:06.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IXHH1fE+niq89S9rjsmxHAP2Ytg='} 2023-07-29T00:33:06.845 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000080a559814c9b31c9-0064c45e42-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.846 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:06.846 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:06.846 DEBUG:boto:Method: HEAD 2023-07-29T00:33:06.847 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:33:06.847 DEBUG:boto:Data: 2023-07-29T00:33:06.847 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.848 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:06.848 DEBUG:boto:Port: 8000 2023-07-29T00:33:06.848 DEBUG:boto:Params: {} 2023-07-29T00:33:06.849 DEBUG:boto:Token: None 2023-07-29T00:33:06.849 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/ 2023-07-29T00:33:06.850 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Q0qVzpK5QesBKcLAopwS4/E0arA= 2023-07-29T00:33:06.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q0qVzpK5QesBKcLAopwS4/E0arA=', 'Content-Length': '0'} 2023-07-29T00:33:06.853 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002db26da003ad426b-0064c45e42-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.853 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:33:06.854 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:33:06.854 DEBUG:boto:Method: PUT 2023-07-29T00:33:06.855 DEBUG:boto:Path: /hiiqos-59/obj46 2023-07-29T00:33:06.855 DEBUG:boto:Data: 2023-07-29T00:33:06.855 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:33:06.856 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:06.856 DEBUG:boto:Port: 8000 2023-07-29T00:33:06.856 DEBUG:boto:Params: {} 2023-07-29T00:33:06.857 DEBUG:boto:Token: None 2023-07-29T00:33:06.857 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj46 2023-07-29T00:33:06.857 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3t9aTy4k0Ha0XmwvtAebqG4mN90= 2023-07-29T00:33:06.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3t9aTy4k0Ha0XmwvtAebqG4mN90='} 2023-07-29T00:33:06.900 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000a508a4432090fe2-0064c45e42-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.901 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:06.901 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:06.902 DEBUG:boto:Method: HEAD 2023-07-29T00:33:06.902 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:33:06.903 DEBUG:boto:Data: 2023-07-29T00:33:06.903 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.903 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:06.904 DEBUG:boto:Port: 8000 2023-07-29T00:33:06.904 DEBUG:boto:Params: {} 2023-07-29T00:33:06.904 DEBUG:boto:Token: None 2023-07-29T00:33:06.905 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/ 2023-07-29T00:33:06.905 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Q0qVzpK5QesBKcLAopwS4/E0arA= 2023-07-29T00:33:06.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q0qVzpK5QesBKcLAopwS4/E0arA=', 'Content-Length': '0'} 2023-07-29T00:33:06.908 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000079d31f1a3ff3525d-0064c45e42-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.909 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:33:06.909 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:33:06.910 DEBUG:boto:Method: PUT 2023-07-29T00:33:06.910 DEBUG:boto:Path: /hiiqos-59/obj69 2023-07-29T00:33:06.910 DEBUG:boto:Data: 2023-07-29T00:33:06.911 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:33:06.911 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:06.911 DEBUG:boto:Port: 8000 2023-07-29T00:33:06.911 DEBUG:boto:Params: {} 2023-07-29T00:33:06.912 DEBUG:boto:Token: None 2023-07-29T00:33:06.912 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj69 2023-07-29T00:33:06.913 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0Y77jjbKnHGu66QT2B4vIAbvhX0= 2023-07-29T00:33:06.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0Y77jjbKnHGu66QT2B4vIAbvhX0='} 2023-07-29T00:33:06.919 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dcd9d53a4f18056f-0064c45e42-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.919 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:06.920 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:06.920 DEBUG:boto:Method: HEAD 2023-07-29T00:33:06.920 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:33:06.921 DEBUG:boto:Data: 2023-07-29T00:33:06.921 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.921 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:06.922 DEBUG:boto:Port: 8000 2023-07-29T00:33:06.922 DEBUG:boto:Params: {} 2023-07-29T00:33:06.922 DEBUG:boto:Token: None 2023-07-29T00:33:06.923 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/ 2023-07-29T00:33:06.923 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Q0qVzpK5QesBKcLAopwS4/E0arA= 2023-07-29T00:33:06.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q0qVzpK5QesBKcLAopwS4/E0arA=', 'Content-Length': '0'} 2023-07-29T00:33:06.926 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000539915410021c422-0064c45e42-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.927 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:33:06.927 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:33:06.928 DEBUG:boto:Method: PUT 2023-07-29T00:33:06.928 DEBUG:boto:Path: /hiiqos-59/obj92 2023-07-29T00:33:06.928 DEBUG:boto:Data: 2023-07-29T00:33:06.929 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:33:06.929 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:06.929 DEBUG:boto:Port: 8000 2023-07-29T00:33:06.930 DEBUG:boto:Params: {} 2023-07-29T00:33:06.930 DEBUG:boto:Token: None 2023-07-29T00:33:06.930 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj92 2023-07-29T00:33:06.931 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uv75oWdKjGWdvZ/TVPcrKtfFqXs= 2023-07-29T00:33:06.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uv75oWdKjGWdvZ/TVPcrKtfFqXs='} 2023-07-29T00:33:06.981 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009a7701488c10e270-0064c45e42-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.982 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:33:06.982 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:33:06.983 DEBUG:boto:Method: HEAD 2023-07-29T00:33:06.984 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:33:06.985 DEBUG:boto:Data: 2023-07-29T00:33:06.986 DEBUG:boto:Headers: {} 2023-07-29T00:33:06.986 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:06.987 DEBUG:boto:Port: 8000 2023-07-29T00:33:06.987 DEBUG:boto:Params: {} 2023-07-29T00:33:06.987 DEBUG:boto:Token: None 2023-07-29T00:33:06.988 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/ 2023-07-29T00:33:06.988 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Q0qVzpK5QesBKcLAopwS4/E0arA= 2023-07-29T00:33:06.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q0qVzpK5QesBKcLAopwS4/E0arA=', 'Content-Length': '0'} 2023-07-29T00:33:06.991 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000090e80f9d23143a8f-0064c45e42-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:06.992 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:33:06.992 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:33:06.993 DEBUG:boto:Method: PUT 2023-07-29T00:33:06.993 DEBUG:boto:Path: /hiiqos-59/obj115 2023-07-29T00:33:06.994 DEBUG:boto:Data: 2023-07-29T00:33:06.994 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:33:06.994 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:33:06.995 DEBUG:boto:Port: 8000 2023-07-29T00:33:06.995 DEBUG:boto:Params: {} 2023-07-29T00:33:06.996 DEBUG:boto:Token: None 2023-07-29T00:33:06.996 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:33:06 GMT /hiiqos-59/obj115 2023-07-29T00:33:06.997 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yS1TKb6HZzvmmogtuOhEE6LyAYc= 2023-07-29T00:33:06.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:33:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yS1TKb6HZzvmmogtuOhEE6LyAYc='} 2023-07-29T00:33:07.002 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b2afdadc23d69c11-0064c45e42-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:33:07.003 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-59 2023-07-29T00:33:07.003 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:33:07.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:33:07.004 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:33:07.056 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:33:07.057 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:33:07.094 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:33:07.035+0000 7f32cc726880 1 waiting to reach latest gen 3, on 2.. 2023-07-29T00:33:12.094 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:33:12.035+0000 7f32cc726880 1 waiting to reach latest gen 3, on 2.. 2023-07-29T00:33:17.095 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:33:17.036+0000 7f32cc726880 1 waiting to reach latest gen 3, on 2.. 2023-07-29T00:33:22.096 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:33:22.037+0000 7f32cc726880 1 waiting to reach latest gen 3, on 2.. 2023-07-29T00:33:27.098 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:33:27.039+0000 7f32cc726880 1 waiting for incremental sync to catch up: 2023-07-29T00:33:27.099 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2023-07-29T00:33:27.099 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000001.80.6, , , 00000000002.54.6, , 00000000002.83.6, , , , , , 00000000001.71.6, , 00000000001.55.6, , , , ] 2023-07-29T00:33:32.100 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:33:32.041+0000 7f32cc726880 1 waiting for incremental sync to catch up: 2023-07-29T00:33:32.100 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2023-07-29T00:33:32.101 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000001.80.6, , , 00000000002.54.6, , 00000000002.83.6, , , , , , 00000000001.71.6, , 00000000001.55.6, , , , ] 2023-07-29T00:33:37.102 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:33:37.042+0000 7f32cc726880 1 waiting for incremental sync to catch up: 2023-07-29T00:33:37.102 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2023-07-29T00:33:37.103 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000001.80.6, , , 00000000002.54.6, , 00000000002.83.6, , , , , , 00000000001.71.6, , 00000000001.55.6, , , , ] 2023-07-29T00:33:42.104 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:33:42.044+0000 7f32cc726880 1 waiting for incremental sync to catch up: 2023-07-29T00:33:42.104 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2023-07-29T00:33:42.105 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000001.80.6, , , 00000000002.54.6, , 00000000002.83.6, , , , , , 00000000001.71.6, , 00000000001.55.6, , , , ] 2023-07-29T00:33:47.106 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:33:47.046+0000 7f32cc726880 1 waiting for incremental sync to catch up: 2023-07-29T00:33:47.106 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2023-07-29T00:33:47.106 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000001.80.6, , , 00000000002.54.6, , 00000000002.83.6, , , , , , 00000000001.71.6, , 00000000001.55.6, , , , ] 2023-07-29T00:33:52.108 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:33:52.048+0000 7f32cc726880 1 waiting for incremental sync to catch up: 2023-07-29T00:33:52.108 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2023-07-29T00:33:52.108 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000001.80.6, , , 00000000002.54.6, , 00000000002.83.6, , , , , , 00000000001.71.6, , 00000000001.55.6, , , , ] 2023-07-29T00:33:57.110 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:33:57.050+0000 7f32cc726880 1 waiting for incremental sync to catch up: 2023-07-29T00:33:57.110 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2023-07-29T00:33:57.110 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000001.80.6, , , 00000000002.54.6, , 00000000002.83.6, , , , , , 00000000001.71.6, , 00000000001.55.6, , , , ] 2023-07-29T00:34:02.112 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:34:02.052+0000 7f32cc726880 1 waiting for incremental sync to catch up: 2023-07-29T00:34:02.113 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2023-07-29T00:34:02.113 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000001.80.6, , , 00000000002.54.6, , 00000000002.83.6, , , , , , 00000000001.71.6, , 00000000001.55.6, , , , ] 2023-07-29T00:34:07.114 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:34:07.054+0000 7f32cc726880 1 waiting for incremental sync to catch up: 2023-07-29T00:34:07.114 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000001.80.6, , , , , 00000000002.83.6, , , , , , , , 00000000001.55.6, , , , ] 2023-07-29T00:34:07.114 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000001.80.6, , , 00000000002.54.6, , 00000000002.83.6, , , , , , 00000000001.71.6, , 00000000001.55.6, , , , ] 2023-07-29T00:34:12.115 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:34:12.055+0000 7f32cc726880 1 waiting for incremental sync to catch up: 2023-07-29T00:34:12.116 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000001.80.6, , , , , 00000000002.83.6, , , , , , , , 00000000001.55.6, , , , ] 2023-07-29T00:34:12.116 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000001.80.6, , , 00000000002.54.6, , 00000000002.83.6, , , , , , 00000000001.71.6, , 00000000001.55.6, , , , ] 2023-07-29T00:34:17.118 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:34:17.057+0000 7f32cc726880 1 waiting for incremental sync to catch up: 2023-07-29T00:34:17.118 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000001.80.6, , , , , 00000000002.83.6, , , , , , , , 00000000001.55.6, , , , ] 2023-07-29T00:34:17.118 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000001.80.6, , , 00000000002.54.6, , 00000000002.83.6, , , , , , 00000000001.71.6, , 00000000001.55.6, , , , ] 2023-07-29T00:34:22.119 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:34:22.059+0000 7f32cc726880 1 waiting for incremental sync to catch up: 2023-07-29T00:34:22.120 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000001.80.6, , , , , 00000000002.83.6, , , , , , , , 00000000001.55.6, , , , ] 2023-07-29T00:34:22.120 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000001.80.6, , , 00000000002.54.6, , 00000000002.83.6, , , , , , 00000000001.71.6, , 00000000001.55.6, , , , ] 2023-07-29T00:34:23.118 INFO:tasks.rgw.c1.client.0.smithi101.stdout:2023-07-29T00:34:23.058+0000 7f0081a5e640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T00:34:27.121 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:34:27.061+0000 7f32cc726880 1 bucket sync caught up with source: 2023-07-29T00:34:27.122 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000001.80.6, , , 00000000002.54.6, , 00000000002.83.6, , , , , , 00000000001.71.6, , 00000000001.55.6, , , , ] 2023-07-29T00:34:27.122 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000001.80.6, , , 00000000002.54.6, , 00000000002.83.6, , , , , , 00000000001.71.6, , 00000000001.55.6, , , , ] 2023-07-29T00:34:27.122 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:34:27.061+0000 7f32cc726880 0 bucket checkpoint complete 2023-07-29T00:34:27.138 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-59 2023-07-29T00:34:27.139 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:34:27.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:34:27.140 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:34:27.196 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:34:27.196 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:34:27.232 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:34:27.171+0000 7fcca689d880 1 waiting for incremental sync to catch up: 2023-07-29T00:34:27.232 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.848.6, 00000000001.690.6, 00000000001.1200.6, 00000000002.951.6, 00000000001.729.6, , , 00000000002.295.6, 00000000005.796.6, 00000000004.1374.6] 2023-07-29T00:34:27.232 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000002.848.6, 00000000002.701.6, 00000000001.1200.6, 00000000002.951.6, 00000000002.740.6, , , 00000000002.295.6, 00000000005.796.6, 00000000004.1374.6] 2023-07-29T00:34:32.233 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:34:32.173+0000 7fcca689d880 1 waiting for incremental sync to catch up: 2023-07-29T00:34:32.233 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.848.6, 00000000001.690.6, 00000000001.1200.6, 00000000002.951.6, 00000000001.729.6, , , 00000000002.295.6, 00000000005.796.6, 00000000004.1374.6] 2023-07-29T00:34:32.234 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000002.848.6, 00000000002.701.6, 00000000001.1200.6, 00000000002.951.6, 00000000002.740.6, , , 00000000002.295.6, 00000000005.796.6, 00000000004.1374.6] 2023-07-29T00:34:37.234 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:34:37.174+0000 7fcca689d880 1 waiting for incremental sync to catch up: 2023-07-29T00:34:37.235 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.848.6, 00000000001.690.6, 00000000001.1200.6, 00000000002.951.6, 00000000001.729.6, , , 00000000002.295.6, 00000000005.796.6, 00000000004.1374.6] 2023-07-29T00:34:37.235 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000002.848.6, 00000000002.701.6, 00000000001.1200.6, 00000000002.951.6, 00000000002.740.6, , , 00000000002.295.6, 00000000005.796.6, 00000000004.1374.6] 2023-07-29T00:34:42.236 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:34:42.175+0000 7fcca689d880 1 waiting for incremental sync to catch up: 2023-07-29T00:34:42.236 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.848.6, 00000000001.690.6, 00000000001.1200.6, 00000000002.951.6, 00000000001.729.6, , , 00000000002.295.6, 00000000005.796.6, 00000000004.1374.6] 2023-07-29T00:34:42.237 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000002.848.6, 00000000002.701.6, 00000000001.1200.6, 00000000002.951.6, 00000000002.740.6, , , 00000000002.295.6, 00000000005.796.6, 00000000004.1374.6] 2023-07-29T00:34:47.237 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:34:47.177+0000 7fcca689d880 1 bucket sync caught up with source: 2023-07-29T00:34:47.238 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.848.6, 00000000002.701.6, 00000000001.1200.6, 00000000002.951.6, 00000000002.740.6, , , 00000000002.295.6, 00000000005.796.6, 00000000004.1374.6] 2023-07-29T00:34:47.238 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000002.848.6, 00000000002.701.6, 00000000001.1200.6, 00000000002.951.6, 00000000002.740.6, , , 00000000002.295.6, 00000000005.796.6, 00000000004.1374.6] 2023-07-29T00:34:47.238 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:34:47.177+0000 7fcca689d880 0 bucket checkpoint complete 2023-07-29T00:34:47.255 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-59 zones={a2, a1} 2023-07-29T00:34:47.256 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:47.256 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:47.257 DEBUG:boto:Method: HEAD 2023-07-29T00:34:47.257 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:34:47.257 DEBUG:boto:Data: 2023-07-29T00:34:47.258 DEBUG:boto:Headers: {} 2023-07-29T00:34:47.258 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:47.259 DEBUG:boto:Port: 8001 2023-07-29T00:34:47.259 DEBUG:boto:Params: {} 2023-07-29T00:34:47.260 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-29T00:34:47.260 DEBUG:boto:Token: None 2023-07-29T00:34:47.261 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:47 GMT /hiiqos-59/ 2023-07-29T00:34:47.261 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:W8qMe+rbMEax7jLbqFB5HOJaaqY= 2023-07-29T00:34:47.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:W8qMe+rbMEax7jLbqFB5HOJaaqY=', 'Content-Length': '0'} 2023-07-29T00:34:52.266 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000054826135256f9599-0064c45eac-1c08-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:34:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:52.267 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:52.267 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:52.267 DEBUG:boto:Method: HEAD 2023-07-29T00:34:52.268 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:34:52.268 DEBUG:boto:Data: 2023-07-29T00:34:52.269 DEBUG:boto:Headers: {} 2023-07-29T00:34:52.269 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:52.269 DEBUG:boto:Port: 8000 2023-07-29T00:34:52.270 DEBUG:boto:Params: {} 2023-07-29T00:34:52.270 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-29T00:34:52.271 DEBUG:boto:Token: None 2023-07-29T00:34:52.271 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:52 GMT /hiiqos-59/ 2023-07-29T00:34:52.272 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dFnTsFFneBHAnlypvTIl4O7dG8M= 2023-07-29T00:34:52.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dFnTsFFneBHAnlypvTIl4O7dG8M=', 'Content-Length': '0'} 2023-07-29T00:34:57.277 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bb82c23d9d34ad73-0064c45eb1-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.278 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-29T00:34:57.278 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:57.279 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:57.279 DEBUG:boto:path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:34:57.279 DEBUG:boto:auth_path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:34:57.280 DEBUG:boto:Method: GET 2023-07-29T00:34:57.280 DEBUG:boto:Path: /hiiqos-59/?versions&max-keys=999 2023-07-29T00:34:57.280 DEBUG:boto:Data: 2023-07-29T00:34:57.281 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.281 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:57.282 DEBUG:boto:Port: 8001 2023-07-29T00:34:57.282 DEBUG:boto:Params: {} 2023-07-29T00:34:57.283 DEBUG:boto:Token: None 2023-07-29T00:34:57.284 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/?versions 2023-07-29T00:34:57.285 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bpeD9VX6P0BA9hmOqKJipsUj2ks= 2023-07-29T00:34:57.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bpeD9VX6P0BA9hmOqKJipsUj2ks=', 'Content-Length': '0'} 2023-07-29T00:34:57.289 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008628195a4c92b72a-0064c45eb1-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.289 DEBUG:boto:b'hiiqos-59999falseobj11nulltrue2023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-07-29T00:33:06.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-07-29T00:33:06.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-07-29T00:33:06.857Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-07-29T00:33:06.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:57.291 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-07-29T00:34:57.291 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-07-29T00:34:57.292 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-07-29T00:34:57.292 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-07-29T00:34:57.292 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-07-29T00:34:57.293 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-07-29T00:34:57.293 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-07-29T00:34:57.293 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-07-29T00:34:57.294 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-07-29T00:34:57.294 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-07-29T00:34:57.294 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-07-29T00:34:57.294 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-07-29T00:34:57.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-07-29T00:34:57.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-07-29T00:34:57.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-07-29T00:34:57.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-07-29T00:34:57.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-07-29T00:34:57.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-07-29T00:34:57.297 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-07-29T00:34:57.297 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-07-29T00:34:57.297 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-29T00:34:57.298 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:57.298 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:57.298 DEBUG:boto:path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:34:57.299 DEBUG:boto:auth_path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:34:57.299 DEBUG:boto:Method: GET 2023-07-29T00:34:57.299 DEBUG:boto:Path: /hiiqos-59/?versions&max-keys=999 2023-07-29T00:34:57.300 DEBUG:boto:Data: 2023-07-29T00:34:57.300 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.300 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:57.301 DEBUG:boto:Port: 8000 2023-07-29T00:34:57.301 DEBUG:boto:Params: {} 2023-07-29T00:34:57.301 DEBUG:boto:Token: None 2023-07-29T00:34:57.302 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/?versions 2023-07-29T00:34:57.303 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bpeD9VX6P0BA9hmOqKJipsUj2ks= 2023-07-29T00:34:57.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bpeD9VX6P0BA9hmOqKJipsUj2ks=', 'Content-Length': '0'} 2023-07-29T00:34:57.307 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5c7d7b84f2a8c7b-0064c45eb1-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.307 DEBUG:boto:b'hiiqos-59999falseobj11nulltrue2023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-07-29T00:33:06.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-07-29T00:33:06.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-07-29T00:33:06.857Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-07-29T00:33:06.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:57.309 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-07-29T00:34:57.309 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-07-29T00:34:57.309 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-07-29T00:34:57.310 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-07-29T00:34:57.310 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-07-29T00:34:57.310 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-07-29T00:34:57.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-07-29T00:34:57.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-07-29T00:34:57.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-07-29T00:34:57.312 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-07-29T00:34:57.312 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-07-29T00:34:57.312 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-07-29T00:34:57.313 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-07-29T00:34:57.313 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-07-29T00:34:57.313 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-07-29T00:34:57.313 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-07-29T00:34:57.314 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-07-29T00:34:57.314 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-07-29T00:34:57.314 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-07-29T00:34:57.315 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-07-29T00:34:57.315 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:57.315 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:57.316 DEBUG:boto:path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:34:57.316 DEBUG:boto:auth_path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:34:57.316 DEBUG:boto:Method: GET 2023-07-29T00:34:57.317 DEBUG:boto:Path: /hiiqos-59/?versions&max-keys=999 2023-07-29T00:34:57.317 DEBUG:boto:Data: 2023-07-29T00:34:57.317 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.318 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:57.318 DEBUG:boto:Port: 8001 2023-07-29T00:34:57.318 DEBUG:boto:Params: {} 2023-07-29T00:34:57.319 DEBUG:boto:Token: None 2023-07-29T00:34:57.320 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/?versions 2023-07-29T00:34:57.320 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bpeD9VX6P0BA9hmOqKJipsUj2ks= 2023-07-29T00:34:57.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bpeD9VX6P0BA9hmOqKJipsUj2ks=', 'Content-Length': '0'} 2023-07-29T00:34:57.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5b302619ff95c3f-0064c45eb1-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.325 DEBUG:boto:b'hiiqos-59999falseobj11nulltrue2023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-07-29T00:33:06.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-07-29T00:33:06.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-07-29T00:33:06.857Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-07-29T00:33:06.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:57.326 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:57.327 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:57.327 DEBUG:boto:path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:34:57.327 DEBUG:boto:auth_path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:34:57.328 DEBUG:boto:Method: GET 2023-07-29T00:34:57.328 DEBUG:boto:Path: /hiiqos-59/?versions&max-keys=999 2023-07-29T00:34:57.329 DEBUG:boto:Data: 2023-07-29T00:34:57.329 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.329 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:57.330 DEBUG:boto:Port: 8000 2023-07-29T00:34:57.330 DEBUG:boto:Params: {} 2023-07-29T00:34:57.331 DEBUG:boto:Token: None 2023-07-29T00:34:57.331 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/?versions 2023-07-29T00:34:57.332 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bpeD9VX6P0BA9hmOqKJipsUj2ks= 2023-07-29T00:34:57.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bpeD9VX6P0BA9hmOqKJipsUj2ks=', 'Content-Length': '0'} 2023-07-29T00:34:57.336 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9bc28c3d942cff3-0064c45eb1-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.337 DEBUG:boto:b'hiiqos-59999falseobj11nulltrue2023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-07-29T00:33:06.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-07-29T00:33:06.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-07-29T00:33:06.857Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-07-29T00:33:06.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:57.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-07-29T00:34:57.339 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:34:57.340 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:34:57.340 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:34:57.340 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:34:57.341 DEBUG:boto:Method: GET 2023-07-29T00:34:57.341 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:34:57.342 DEBUG:boto:Data: 2023-07-29T00:34:57.342 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.342 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:57.343 DEBUG:boto:Port: 8001 2023-07-29T00:34:57.343 DEBUG:boto:Params: {} 2023-07-29T00:34:57.343 DEBUG:boto:Token: None 2023-07-29T00:34:57.344 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:34:57.345 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0/CIvrbUIIiv/4ebONJbDfecnkM= 2023-07-29T00:34:57.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0/CIvrbUIIiv/4ebONJbDfecnkM=', 'Content-Length': '0'} 2023-07-29T00:34:57.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091591f059ea528b8-0064c45eb1-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.349 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:34:57.350 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:34:57.351 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:34:57.352 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:34:57.353 DEBUG:boto:Method: GET 2023-07-29T00:34:57.354 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:34:57.354 DEBUG:boto:Data: 2023-07-29T00:34:57.354 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.355 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:57.355 DEBUG:boto:Port: 8000 2023-07-29T00:34:57.356 DEBUG:boto:Params: {} 2023-07-29T00:34:57.356 DEBUG:boto:Token: None 2023-07-29T00:34:57.357 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:34:57.357 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0/CIvrbUIIiv/4ebONJbDfecnkM= 2023-07-29T00:34:57.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0/CIvrbUIIiv/4ebONJbDfecnkM=', 'Content-Length': '0'} 2023-07-29T00:34:57.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000574a3ef139ad2671-0064c45eb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.404 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:34:57.405 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:34:57.405 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:34:57.406 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:34:57.406 DEBUG:boto:Method: HEAD 2023-07-29T00:34:57.406 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:34:57.407 DEBUG:boto:Data: 2023-07-29T00:34:57.407 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.408 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:57.408 DEBUG:boto:Port: 8001 2023-07-29T00:34:57.408 DEBUG:boto:Params: {} 2023-07-29T00:34:57.409 DEBUG:boto:Token: None 2023-07-29T00:34:57.409 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:34:57.410 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IPwt9yoY9Z58w79aZEilP6fmQug= 2023-07-29T00:34:57.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IPwt9yoY9Z58w79aZEilP6fmQug=', 'Content-Length': '0'} 2023-07-29T00:34:57.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000949ef9e34fd60a40-0064c45eb1-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.413 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:34:57.413 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:34:57.413 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:34:57.414 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:34:57.414 DEBUG:boto:Method: HEAD 2023-07-29T00:34:57.414 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:34:57.415 DEBUG:boto:Data: 2023-07-29T00:34:57.415 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.415 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:57.416 DEBUG:boto:Port: 8000 2023-07-29T00:34:57.416 DEBUG:boto:Params: {} 2023-07-29T00:34:57.417 DEBUG:boto:Token: None 2023-07-29T00:34:57.417 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:34:57.417 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IPwt9yoY9Z58w79aZEilP6fmQug= 2023-07-29T00:34:57.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IPwt9yoY9Z58w79aZEilP6fmQug=', 'Content-Length': '0'} 2023-07-29T00:34:57.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a7785d110086746-0064c45eb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-07-29T00:34:57.421 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:34:57.421 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:34:57.421 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:34:57.422 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:34:57.422 DEBUG:boto:Method: GET 2023-07-29T00:34:57.422 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:34:57.423 DEBUG:boto:Data: 2023-07-29T00:34:57.423 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.424 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:57.424 DEBUG:boto:Port: 8001 2023-07-29T00:34:57.424 DEBUG:boto:Params: {} 2023-07-29T00:34:57.425 DEBUG:boto:Token: None 2023-07-29T00:34:57.425 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:34:57.426 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0/CIvrbUIIiv/4ebONJbDfecnkM= 2023-07-29T00:34:57.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0/CIvrbUIIiv/4ebONJbDfecnkM=', 'Content-Length': '0'} 2023-07-29T00:34:57.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025971b7f30c09990-0064c45eb1-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.429 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:34:57.429 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:34:57.430 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:34:57.430 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:34:57.430 DEBUG:boto:Method: GET 2023-07-29T00:34:57.431 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:34:57.431 DEBUG:boto:Data: 2023-07-29T00:34:57.432 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.432 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:57.432 DEBUG:boto:Port: 8000 2023-07-29T00:34:57.433 DEBUG:boto:Params: {} 2023-07-29T00:34:57.433 DEBUG:boto:Token: None 2023-07-29T00:34:57.434 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:34:57.434 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0/CIvrbUIIiv/4ebONJbDfecnkM= 2023-07-29T00:34:57.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0/CIvrbUIIiv/4ebONJbDfecnkM=', 'Content-Length': '0'} 2023-07-29T00:34:57.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce84c555a35f2f89-0064c45eb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.437 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:57.438 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:57.438 DEBUG:boto:path=/hiiqos-59/?prefix=obj11 2023-07-29T00:34:57.438 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj11 2023-07-29T00:34:57.439 DEBUG:boto:Method: GET 2023-07-29T00:34:57.439 DEBUG:boto:Path: /hiiqos-59/?prefix=obj11 2023-07-29T00:34:57.439 DEBUG:boto:Data: 2023-07-29T00:34:57.440 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.440 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:57.441 DEBUG:boto:Port: 8001 2023-07-29T00:34:57.441 DEBUG:boto:Params: {} 2023-07-29T00:34:57.441 DEBUG:boto:Token: None 2023-07-29T00:34:57.442 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/ 2023-07-29T00:34:57.442 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CdeTsNNlOP+SoOJZQmkiPsPal+A= 2023-07-29T00:34:57.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CdeTsNNlOP+SoOJZQmkiPsPal+A=', 'Content-Length': '0'} 2023-07-29T00:34:57.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4cb6de27c0c0552-0064c45eb1-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.488 DEBUG:boto:b'hiiqos-59obj111000falseobj112023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:57.489 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:57.490 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:57.490 DEBUG:boto:path=/hiiqos-59/?prefix=obj11 2023-07-29T00:34:57.491 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj11 2023-07-29T00:34:57.491 DEBUG:boto:Method: GET 2023-07-29T00:34:57.491 DEBUG:boto:Path: /hiiqos-59/?prefix=obj11 2023-07-29T00:34:57.492 DEBUG:boto:Data: 2023-07-29T00:34:57.492 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.493 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:57.493 DEBUG:boto:Port: 8000 2023-07-29T00:34:57.493 DEBUG:boto:Params: {} 2023-07-29T00:34:57.494 DEBUG:boto:Token: None 2023-07-29T00:34:57.494 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/ 2023-07-29T00:34:57.495 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CdeTsNNlOP+SoOJZQmkiPsPal+A= 2023-07-29T00:34:57.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CdeTsNNlOP+SoOJZQmkiPsPal+A=', 'Content-Length': '0'} 2023-07-29T00:34:57.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ba2e7fab9ce98ae-0064c45eb1-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.499 DEBUG:boto:b'hiiqos-59obj111000falseobj112023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:57.500 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:34:57.500 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:34:57.501 DEBUG:boto:Method: HEAD 2023-07-29T00:34:57.501 DEBUG:boto:Path: /hiiqos-59/obj11 2023-07-29T00:34:57.502 DEBUG:boto:Data: 2023-07-29T00:34:57.502 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.502 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:57.503 DEBUG:boto:Port: 8001 2023-07-29T00:34:57.503 DEBUG:boto:Params: {} 2023-07-29T00:34:57.504 DEBUG:boto:Token: None 2023-07-29T00:34:57.504 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/obj11 2023-07-29T00:34:57.505 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3kgsF54mS+zbRgNQtf3zqN70/fw= 2023-07-29T00:34:57.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3kgsF54mS+zbRgNQtf3zqN70/fw=', 'Content-Length': '0'} 2023-07-29T00:34:57.507 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf715185d37bf44b-0064c45eb1-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.507 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:34:57.508 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:34:57.508 DEBUG:boto:Method: HEAD 2023-07-29T00:34:57.509 DEBUG:boto:Path: /hiiqos-59/obj11 2023-07-29T00:34:57.509 DEBUG:boto:Data: 2023-07-29T00:34:57.509 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.510 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:57.510 DEBUG:boto:Port: 8000 2023-07-29T00:34:57.510 DEBUG:boto:Params: {} 2023-07-29T00:34:57.511 DEBUG:boto:Token: None 2023-07-29T00:34:57.511 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/obj11 2023-07-29T00:34:57.511 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3kgsF54mS+zbRgNQtf3zqN70/fw= 2023-07-29T00:34:57.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3kgsF54mS+zbRgNQtf3zqN70/fw=', 'Content-Length': '0'} 2023-07-29T00:34:57.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d810c2ca08891a81-0064c45eb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-07-29T00:34:57.515 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:34:57.515 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:34:57.515 DEBUG:boto:Method: GET 2023-07-29T00:34:57.516 DEBUG:boto:Path: /hiiqos-59/obj11 2023-07-29T00:34:57.516 DEBUG:boto:Data: 2023-07-29T00:34:57.517 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.517 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:57.517 DEBUG:boto:Port: 8001 2023-07-29T00:34:57.518 DEBUG:boto:Params: {} 2023-07-29T00:34:57.518 DEBUG:boto:Token: None 2023-07-29T00:34:57.518 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/obj11 2023-07-29T00:34:57.519 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pcTLFn8h380AlwXV1kob2DpdYQQ= 2023-07-29T00:34:57.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pcTLFn8h380AlwXV1kob2DpdYQQ=', 'Content-Length': '0'} 2023-07-29T00:34:57.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b76e7c6c0f9e048f-0064c45eb1-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.522 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:34:57.523 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:34:57.523 DEBUG:boto:Method: GET 2023-07-29T00:34:57.523 DEBUG:boto:Path: /hiiqos-59/obj11 2023-07-29T00:34:57.524 DEBUG:boto:Data: 2023-07-29T00:34:57.524 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.524 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:57.525 DEBUG:boto:Port: 8000 2023-07-29T00:34:57.525 DEBUG:boto:Params: {} 2023-07-29T00:34:57.526 DEBUG:boto:Token: None 2023-07-29T00:34:57.526 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/obj11 2023-07-29T00:34:57.527 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pcTLFn8h380AlwXV1kob2DpdYQQ= 2023-07-29T00:34:57.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pcTLFn8h380AlwXV1kob2DpdYQQ=', 'Content-Length': '0'} 2023-07-29T00:34:57.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023cebecb2565afb4-0064c45eb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.530 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:57.530 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:57.531 DEBUG:boto:path=/hiiqos-59/?prefix=obj11 2023-07-29T00:34:57.531 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj11 2023-07-29T00:34:57.531 DEBUG:boto:Method: GET 2023-07-29T00:34:57.532 DEBUG:boto:Path: /hiiqos-59/?prefix=obj11 2023-07-29T00:34:57.532 DEBUG:boto:Data: 2023-07-29T00:34:57.532 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.533 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:57.533 DEBUG:boto:Port: 8001 2023-07-29T00:34:57.534 DEBUG:boto:Params: {} 2023-07-29T00:34:57.534 DEBUG:boto:Token: None 2023-07-29T00:34:57.534 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/ 2023-07-29T00:34:57.535 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CdeTsNNlOP+SoOJZQmkiPsPal+A= 2023-07-29T00:34:57.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CdeTsNNlOP+SoOJZQmkiPsPal+A=', 'Content-Length': '0'} 2023-07-29T00:34:57.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000765060e954d54199-0064c45eb1-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.538 DEBUG:boto:b'hiiqos-59obj111000falseobj112023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:57.539 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:57.540 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:57.540 DEBUG:boto:path=/hiiqos-59/?prefix=obj11 2023-07-29T00:34:57.541 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj11 2023-07-29T00:34:57.541 DEBUG:boto:Method: GET 2023-07-29T00:34:57.541 DEBUG:boto:Path: /hiiqos-59/?prefix=obj11 2023-07-29T00:34:57.542 DEBUG:boto:Data: 2023-07-29T00:34:57.542 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.542 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:57.543 DEBUG:boto:Port: 8000 2023-07-29T00:34:57.543 DEBUG:boto:Params: {} 2023-07-29T00:34:57.544 DEBUG:boto:Token: None 2023-07-29T00:34:57.544 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/ 2023-07-29T00:34:57.545 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CdeTsNNlOP+SoOJZQmkiPsPal+A= 2023-07-29T00:34:57.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CdeTsNNlOP+SoOJZQmkiPsPal+A=', 'Content-Length': '0'} 2023-07-29T00:34:57.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000335576fd9298d5ba-0064c45eb1-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.588 DEBUG:boto:b'hiiqos-59obj111000falseobj112023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:57.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-07-29T00:34:57.589 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:34:57.590 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:34:57.590 DEBUG:boto:path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:34:57.591 DEBUG:boto:auth_path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:34:57.591 DEBUG:boto:Method: GET 2023-07-29T00:34:57.591 DEBUG:boto:Path: /hiiqos-59/obj115?versionId=null 2023-07-29T00:34:57.592 DEBUG:boto:Data: 2023-07-29T00:34:57.592 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.593 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:57.593 DEBUG:boto:Port: 8001 2023-07-29T00:34:57.593 DEBUG:boto:Params: {} 2023-07-29T00:34:57.594 DEBUG:boto:Token: None 2023-07-29T00:34:57.594 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/obj115?versionId=null 2023-07-29T00:34:57.595 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bZFEkxrwZHlvMpPm6UUTEFBcbIs= 2023-07-29T00:34:57.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bZFEkxrwZHlvMpPm6UUTEFBcbIs=', 'Content-Length': '0'} 2023-07-29T00:34:57.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0bdeba4467f1420-0064c45eb1-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.598 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:34:57.598 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:34:57.599 DEBUG:boto:path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:34:57.599 DEBUG:boto:auth_path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:34:57.599 DEBUG:boto:Method: GET 2023-07-29T00:34:57.600 DEBUG:boto:Path: /hiiqos-59/obj115?versionId=null 2023-07-29T00:34:57.600 DEBUG:boto:Data: 2023-07-29T00:34:57.601 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.601 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:57.601 DEBUG:boto:Port: 8000 2023-07-29T00:34:57.602 DEBUG:boto:Params: {} 2023-07-29T00:34:57.602 DEBUG:boto:Token: None 2023-07-29T00:34:57.603 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/obj115?versionId=null 2023-07-29T00:34:57.603 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bZFEkxrwZHlvMpPm6UUTEFBcbIs= 2023-07-29T00:34:57.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bZFEkxrwZHlvMpPm6UUTEFBcbIs=', 'Content-Length': '0'} 2023-07-29T00:34:57.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ef6a54ea9e5a46f-0064c45eb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.606 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:34:57.607 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:34:57.607 DEBUG:boto:path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:34:57.607 DEBUG:boto:auth_path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:34:57.608 DEBUG:boto:Method: HEAD 2023-07-29T00:34:57.608 DEBUG:boto:Path: /hiiqos-59/obj115?versionId=null 2023-07-29T00:34:57.609 DEBUG:boto:Data: 2023-07-29T00:34:57.609 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.609 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:57.610 DEBUG:boto:Port: 8001 2023-07-29T00:34:57.610 DEBUG:boto:Params: {} 2023-07-29T00:34:57.610 DEBUG:boto:Token: None 2023-07-29T00:34:57.611 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/obj115?versionId=null 2023-07-29T00:34:57.611 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GYDU9H1bpitSktSvinG7MEck4w4= 2023-07-29T00:34:57.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GYDU9H1bpitSktSvinG7MEck4w4=', 'Content-Length': '0'} 2023-07-29T00:34:57.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079e034d775db520c-0064c45eb1-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.614 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:34:57.614 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:34:57.614 DEBUG:boto:path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:34:57.615 DEBUG:boto:auth_path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:34:57.615 DEBUG:boto:Method: HEAD 2023-07-29T00:34:57.616 DEBUG:boto:Path: /hiiqos-59/obj115?versionId=null 2023-07-29T00:34:57.616 DEBUG:boto:Data: 2023-07-29T00:34:57.616 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.617 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:57.617 DEBUG:boto:Port: 8000 2023-07-29T00:34:57.617 DEBUG:boto:Params: {} 2023-07-29T00:34:57.618 DEBUG:boto:Token: None 2023-07-29T00:34:57.618 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/obj115?versionId=null 2023-07-29T00:34:57.618 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GYDU9H1bpitSktSvinG7MEck4w4= 2023-07-29T00:34:57.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GYDU9H1bpitSktSvinG7MEck4w4=', 'Content-Length': '0'} 2023-07-29T00:34:57.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac4e7777f5d3ae9e-0064c45eb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-07-29T00:34:57.622 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:34:57.622 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:34:57.623 DEBUG:boto:path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:34:57.623 DEBUG:boto:auth_path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:34:57.623 DEBUG:boto:Method: GET 2023-07-29T00:34:57.624 DEBUG:boto:Path: /hiiqos-59/obj115?versionId=null 2023-07-29T00:34:57.624 DEBUG:boto:Data: 2023-07-29T00:34:57.625 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.625 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:57.625 DEBUG:boto:Port: 8001 2023-07-29T00:34:57.626 DEBUG:boto:Params: {} 2023-07-29T00:34:57.626 DEBUG:boto:Token: None 2023-07-29T00:34:57.626 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/obj115?versionId=null 2023-07-29T00:34:57.627 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bZFEkxrwZHlvMpPm6UUTEFBcbIs= 2023-07-29T00:34:57.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bZFEkxrwZHlvMpPm6UUTEFBcbIs=', 'Content-Length': '0'} 2023-07-29T00:34:57.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000948b19939e4e430b-0064c45eb1-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.630 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:34:57.630 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:34:57.630 DEBUG:boto:path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:34:57.631 DEBUG:boto:auth_path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:34:57.631 DEBUG:boto:Method: GET 2023-07-29T00:34:57.632 DEBUG:boto:Path: /hiiqos-59/obj115?versionId=null 2023-07-29T00:34:57.632 DEBUG:boto:Data: 2023-07-29T00:34:57.632 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.633 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:57.633 DEBUG:boto:Port: 8000 2023-07-29T00:34:57.633 DEBUG:boto:Params: {} 2023-07-29T00:34:57.634 DEBUG:boto:Token: None 2023-07-29T00:34:57.634 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/obj115?versionId=null 2023-07-29T00:34:57.635 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bZFEkxrwZHlvMpPm6UUTEFBcbIs= 2023-07-29T00:34:57.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bZFEkxrwZHlvMpPm6UUTEFBcbIs=', 'Content-Length': '0'} 2023-07-29T00:34:57.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b892bee4721ce610-0064c45eb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.638 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:57.638 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:57.639 DEBUG:boto:path=/hiiqos-59/?prefix=obj115 2023-07-29T00:34:57.639 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj115 2023-07-29T00:34:57.639 DEBUG:boto:Method: GET 2023-07-29T00:34:57.640 DEBUG:boto:Path: /hiiqos-59/?prefix=obj115 2023-07-29T00:34:57.640 DEBUG:boto:Data: 2023-07-29T00:34:57.641 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.642 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:57.642 DEBUG:boto:Port: 8001 2023-07-29T00:34:57.643 DEBUG:boto:Params: {} 2023-07-29T00:34:57.644 DEBUG:boto:Token: None 2023-07-29T00:34:57.645 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/ 2023-07-29T00:34:57.646 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CdeTsNNlOP+SoOJZQmkiPsPal+A= 2023-07-29T00:34:57.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CdeTsNNlOP+SoOJZQmkiPsPal+A=', 'Content-Length': '0'} 2023-07-29T00:34:57.649 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000278a999d5aac6a3c-0064c45eb1-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.692 DEBUG:boto:b'hiiqos-59obj1151000falseobj1152023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:57.694 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:57.695 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:57.696 DEBUG:boto:path=/hiiqos-59/?prefix=obj115 2023-07-29T00:34:57.697 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj115 2023-07-29T00:34:57.697 DEBUG:boto:Method: GET 2023-07-29T00:34:57.698 DEBUG:boto:Path: /hiiqos-59/?prefix=obj115 2023-07-29T00:34:57.699 DEBUG:boto:Data: 2023-07-29T00:34:57.700 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.700 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:57.701 DEBUG:boto:Port: 8000 2023-07-29T00:34:57.702 DEBUG:boto:Params: {} 2023-07-29T00:34:57.702 DEBUG:boto:Token: None 2023-07-29T00:34:57.703 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/ 2023-07-29T00:34:57.703 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CdeTsNNlOP+SoOJZQmkiPsPal+A= 2023-07-29T00:34:57.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CdeTsNNlOP+SoOJZQmkiPsPal+A=', 'Content-Length': '0'} 2023-07-29T00:34:57.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d63edef31a46377d-0064c45eb1-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.708 DEBUG:boto:b'hiiqos-59obj1151000falseobj1152023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:57.708 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:34:57.709 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:34:57.709 DEBUG:boto:Method: HEAD 2023-07-29T00:34:57.710 DEBUG:boto:Path: /hiiqos-59/obj115 2023-07-29T00:34:57.711 DEBUG:boto:Data: 2023-07-29T00:34:57.711 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.712 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:57.713 DEBUG:boto:Port: 8001 2023-07-29T00:34:57.714 DEBUG:boto:Params: {} 2023-07-29T00:34:57.715 DEBUG:boto:Token: None 2023-07-29T00:34:57.716 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/obj115 2023-07-29T00:34:57.716 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lc6TmKTuoohdqaksMyuszdOZ//s= 2023-07-29T00:34:57.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lc6TmKTuoohdqaksMyuszdOZ//s=', 'Content-Length': '0'} 2023-07-29T00:34:57.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000826285a09ade132d-0064c45eb1-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.719 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:34:57.719 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:34:57.720 DEBUG:boto:Method: HEAD 2023-07-29T00:34:57.720 DEBUG:boto:Path: /hiiqos-59/obj115 2023-07-29T00:34:57.720 DEBUG:boto:Data: 2023-07-29T00:34:57.721 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.721 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:57.721 DEBUG:boto:Port: 8000 2023-07-29T00:34:57.722 DEBUG:boto:Params: {} 2023-07-29T00:34:57.722 DEBUG:boto:Token: None 2023-07-29T00:34:57.722 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/obj115 2023-07-29T00:34:57.723 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lc6TmKTuoohdqaksMyuszdOZ//s= 2023-07-29T00:34:57.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lc6TmKTuoohdqaksMyuszdOZ//s=', 'Content-Length': '0'} 2023-07-29T00:34:57.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b12231fc337929b9-0064c45eb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-07-29T00:34:57.726 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:34:57.726 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:34:57.727 DEBUG:boto:Method: GET 2023-07-29T00:34:57.727 DEBUG:boto:Path: /hiiqos-59/obj115 2023-07-29T00:34:57.728 DEBUG:boto:Data: 2023-07-29T00:34:57.728 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.728 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:57.729 DEBUG:boto:Port: 8001 2023-07-29T00:34:57.729 DEBUG:boto:Params: {} 2023-07-29T00:34:57.729 DEBUG:boto:Token: None 2023-07-29T00:34:57.730 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/obj115 2023-07-29T00:34:57.730 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nVfqA35Fd33dD/euAx0Crkqk6X0= 2023-07-29T00:34:57.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nVfqA35Fd33dD/euAx0Crkqk6X0=', 'Content-Length': '0'} 2023-07-29T00:34:57.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b934a65b7a7cc8c-0064c45eb1-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.733 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:34:57.734 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:34:57.734 DEBUG:boto:Method: GET 2023-07-29T00:34:57.735 DEBUG:boto:Path: /hiiqos-59/obj115 2023-07-29T00:34:57.735 DEBUG:boto:Data: 2023-07-29T00:34:57.735 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.736 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:57.736 DEBUG:boto:Port: 8000 2023-07-29T00:34:57.737 DEBUG:boto:Params: {} 2023-07-29T00:34:57.737 DEBUG:boto:Token: None 2023-07-29T00:34:57.738 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/obj115 2023-07-29T00:34:57.738 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nVfqA35Fd33dD/euAx0Crkqk6X0= 2023-07-29T00:34:57.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nVfqA35Fd33dD/euAx0Crkqk6X0=', 'Content-Length': '0'} 2023-07-29T00:34:57.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cf260b1458bdbaf-0064c45eb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.788 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:57.789 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:57.789 DEBUG:boto:path=/hiiqos-59/?prefix=obj115 2023-07-29T00:34:57.790 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj115 2023-07-29T00:34:57.790 DEBUG:boto:Method: GET 2023-07-29T00:34:57.790 DEBUG:boto:Path: /hiiqos-59/?prefix=obj115 2023-07-29T00:34:57.791 DEBUG:boto:Data: 2023-07-29T00:34:57.791 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.791 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:57.792 DEBUG:boto:Port: 8001 2023-07-29T00:34:57.792 DEBUG:boto:Params: {} 2023-07-29T00:34:57.793 DEBUG:boto:Token: None 2023-07-29T00:34:57.793 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/ 2023-07-29T00:34:57.793 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CdeTsNNlOP+SoOJZQmkiPsPal+A= 2023-07-29T00:34:57.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CdeTsNNlOP+SoOJZQmkiPsPal+A=', 'Content-Length': '0'} 2023-07-29T00:34:57.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe7a8e48c6405344-0064c45eb1-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.797 DEBUG:boto:b'hiiqos-59obj1151000falseobj1152023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:57.798 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:57.798 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:57.798 DEBUG:boto:path=/hiiqos-59/?prefix=obj115 2023-07-29T00:34:57.799 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj115 2023-07-29T00:34:57.799 DEBUG:boto:Method: GET 2023-07-29T00:34:57.799 DEBUG:boto:Path: /hiiqos-59/?prefix=obj115 2023-07-29T00:34:57.800 DEBUG:boto:Data: 2023-07-29T00:34:57.800 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.801 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:57.801 DEBUG:boto:Port: 8000 2023-07-29T00:34:57.801 DEBUG:boto:Params: {} 2023-07-29T00:34:57.802 DEBUG:boto:Token: None 2023-07-29T00:34:57.802 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/ 2023-07-29T00:34:57.803 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CdeTsNNlOP+SoOJZQmkiPsPal+A= 2023-07-29T00:34:57.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CdeTsNNlOP+SoOJZQmkiPsPal+A=', 'Content-Length': '0'} 2023-07-29T00:34:57.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044a7dfd5c012413b-0064c45eb1-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.807 DEBUG:boto:b'hiiqos-59obj1151000falseobj1152023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:57.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-07-29T00:34:57.808 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:34:57.808 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:34:57.809 DEBUG:boto:path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:34:57.809 DEBUG:boto:auth_path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:34:57.809 DEBUG:boto:Method: GET 2023-07-29T00:34:57.810 DEBUG:boto:Path: /hiiqos-59/obj17?versionId=null 2023-07-29T00:34:57.810 DEBUG:boto:Data: 2023-07-29T00:34:57.811 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.811 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:57.811 DEBUG:boto:Port: 8001 2023-07-29T00:34:57.812 DEBUG:boto:Params: {} 2023-07-29T00:34:57.812 DEBUG:boto:Token: None 2023-07-29T00:34:57.813 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/obj17?versionId=null 2023-07-29T00:34:57.813 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:X9IOJ/fgrBNNuZk3dczWh6x++Lg= 2023-07-29T00:34:57.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:X9IOJ/fgrBNNuZk3dczWh6x++Lg=', 'Content-Length': '0'} 2023-07-29T00:34:57.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9385cb6c1bff19c-0064c45eb1-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.816 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:34:57.817 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:34:57.817 DEBUG:boto:path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:34:57.817 DEBUG:boto:auth_path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:34:57.818 DEBUG:boto:Method: GET 2023-07-29T00:34:57.818 DEBUG:boto:Path: /hiiqos-59/obj17?versionId=null 2023-07-29T00:34:57.818 DEBUG:boto:Data: 2023-07-29T00:34:57.819 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.819 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:57.819 DEBUG:boto:Port: 8000 2023-07-29T00:34:57.820 DEBUG:boto:Params: {} 2023-07-29T00:34:57.820 DEBUG:boto:Token: None 2023-07-29T00:34:57.821 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/obj17?versionId=null 2023-07-29T00:34:57.821 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:X9IOJ/fgrBNNuZk3dczWh6x++Lg= 2023-07-29T00:34:57.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:X9IOJ/fgrBNNuZk3dczWh6x++Lg=', 'Content-Length': '0'} 2023-07-29T00:34:57.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000493371d7fc21c21e-0064c45eb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.824 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:34:57.825 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:34:57.825 DEBUG:boto:path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:34:57.826 DEBUG:boto:auth_path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:34:57.826 DEBUG:boto:Method: HEAD 2023-07-29T00:34:57.826 DEBUG:boto:Path: /hiiqos-59/obj17?versionId=null 2023-07-29T00:34:57.827 DEBUG:boto:Data: 2023-07-29T00:34:57.827 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.827 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:57.828 DEBUG:boto:Port: 8001 2023-07-29T00:34:57.828 DEBUG:boto:Params: {} 2023-07-29T00:34:57.828 DEBUG:boto:Token: None 2023-07-29T00:34:57.829 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/obj17?versionId=null 2023-07-29T00:34:57.829 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WUMTeGYhp1WdtTU7iTaWQ5S5NW4= 2023-07-29T00:34:57.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WUMTeGYhp1WdtTU7iTaWQ5S5NW4=', 'Content-Length': '0'} 2023-07-29T00:34:57.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003818823f41bce1b1-0064c45eb1-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.832 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:34:57.833 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:34:57.833 DEBUG:boto:path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:34:57.834 DEBUG:boto:auth_path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:34:57.834 DEBUG:boto:Method: HEAD 2023-07-29T00:34:57.834 DEBUG:boto:Path: /hiiqos-59/obj17?versionId=null 2023-07-29T00:34:57.835 DEBUG:boto:Data: 2023-07-29T00:34:57.835 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.835 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:57.836 DEBUG:boto:Port: 8000 2023-07-29T00:34:57.836 DEBUG:boto:Params: {} 2023-07-29T00:34:57.837 DEBUG:boto:Token: None 2023-07-29T00:34:57.837 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/obj17?versionId=null 2023-07-29T00:34:57.838 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WUMTeGYhp1WdtTU7iTaWQ5S5NW4= 2023-07-29T00:34:57.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WUMTeGYhp1WdtTU7iTaWQ5S5NW4=', 'Content-Length': '0'} 2023-07-29T00:34:57.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5a38ce692646bd0-0064c45eb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-07-29T00:34:57.841 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:34:57.841 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:34:57.842 DEBUG:boto:path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:34:57.842 DEBUG:boto:auth_path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:34:57.842 DEBUG:boto:Method: GET 2023-07-29T00:34:57.843 DEBUG:boto:Path: /hiiqos-59/obj17?versionId=null 2023-07-29T00:34:57.843 DEBUG:boto:Data: 2023-07-29T00:34:57.844 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.844 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:57.844 DEBUG:boto:Port: 8001 2023-07-29T00:34:57.845 DEBUG:boto:Params: {} 2023-07-29T00:34:57.845 DEBUG:boto:Token: None 2023-07-29T00:34:57.846 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/obj17?versionId=null 2023-07-29T00:34:57.846 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:X9IOJ/fgrBNNuZk3dczWh6x++Lg= 2023-07-29T00:34:57.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:X9IOJ/fgrBNNuZk3dczWh6x++Lg=', 'Content-Length': '0'} 2023-07-29T00:34:57.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021f68123f55724e4-0064c45eb1-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.849 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:34:57.849 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:34:57.850 DEBUG:boto:path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:34:57.850 DEBUG:boto:auth_path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:34:57.850 DEBUG:boto:Method: GET 2023-07-29T00:34:57.851 DEBUG:boto:Path: /hiiqos-59/obj17?versionId=null 2023-07-29T00:34:57.851 DEBUG:boto:Data: 2023-07-29T00:34:57.852 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.853 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:57.854 DEBUG:boto:Port: 8000 2023-07-29T00:34:57.854 DEBUG:boto:Params: {} 2023-07-29T00:34:57.855 DEBUG:boto:Token: None 2023-07-29T00:34:57.856 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/obj17?versionId=null 2023-07-29T00:34:57.857 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:X9IOJ/fgrBNNuZk3dczWh6x++Lg= 2023-07-29T00:34:57.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:X9IOJ/fgrBNNuZk3dczWh6x++Lg=', 'Content-Length': '0'} 2023-07-29T00:34:57.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4ae36b6405eaf4f-0064c45eb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.861 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:57.862 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:57.862 DEBUG:boto:path=/hiiqos-59/?prefix=obj17 2023-07-29T00:34:57.862 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj17 2023-07-29T00:34:57.863 DEBUG:boto:Method: GET 2023-07-29T00:34:57.863 DEBUG:boto:Path: /hiiqos-59/?prefix=obj17 2023-07-29T00:34:57.863 DEBUG:boto:Data: 2023-07-29T00:34:57.864 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.864 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:57.865 DEBUG:boto:Port: 8001 2023-07-29T00:34:57.865 DEBUG:boto:Params: {} 2023-07-29T00:34:57.865 DEBUG:boto:Token: None 2023-07-29T00:34:57.866 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/ 2023-07-29T00:34:57.867 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CdeTsNNlOP+SoOJZQmkiPsPal+A= 2023-07-29T00:34:57.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CdeTsNNlOP+SoOJZQmkiPsPal+A=', 'Content-Length': '0'} 2023-07-29T00:34:57.869 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fe71402f80abad5-0064c45eb1-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.912 DEBUG:boto:b'hiiqos-59obj171000falseobj172023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:57.913 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:57.914 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:57.914 DEBUG:boto:path=/hiiqos-59/?prefix=obj17 2023-07-29T00:34:57.914 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj17 2023-07-29T00:34:57.915 DEBUG:boto:Method: GET 2023-07-29T00:34:57.915 DEBUG:boto:Path: /hiiqos-59/?prefix=obj17 2023-07-29T00:34:57.916 DEBUG:boto:Data: 2023-07-29T00:34:57.916 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.916 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:57.917 DEBUG:boto:Port: 8000 2023-07-29T00:34:57.917 DEBUG:boto:Params: {} 2023-07-29T00:34:57.917 DEBUG:boto:Token: None 2023-07-29T00:34:57.918 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/ 2023-07-29T00:34:57.918 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CdeTsNNlOP+SoOJZQmkiPsPal+A= 2023-07-29T00:34:57.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CdeTsNNlOP+SoOJZQmkiPsPal+A=', 'Content-Length': '0'} 2023-07-29T00:34:57.922 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084ba231f9dc9994a-0064c45eb1-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.923 DEBUG:boto:b'hiiqos-59obj171000falseobj172023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:57.924 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:34:57.924 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:34:57.925 DEBUG:boto:Method: HEAD 2023-07-29T00:34:57.925 DEBUG:boto:Path: /hiiqos-59/obj17 2023-07-29T00:34:57.925 DEBUG:boto:Data: 2023-07-29T00:34:57.926 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.926 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:57.926 DEBUG:boto:Port: 8001 2023-07-29T00:34:57.927 DEBUG:boto:Params: {} 2023-07-29T00:34:57.927 DEBUG:boto:Token: None 2023-07-29T00:34:57.928 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/obj17 2023-07-29T00:34:57.928 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bUn5q/v7YrnojftKTo/NNCJfubw= 2023-07-29T00:34:57.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bUn5q/v7YrnojftKTo/NNCJfubw=', 'Content-Length': '0'} 2023-07-29T00:34:57.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b25b348a3a42d3ac-0064c45eb1-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.932 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:34:57.932 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:34:57.933 DEBUG:boto:Method: HEAD 2023-07-29T00:34:57.933 DEBUG:boto:Path: /hiiqos-59/obj17 2023-07-29T00:34:57.933 DEBUG:boto:Data: 2023-07-29T00:34:57.934 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.934 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:57.935 DEBUG:boto:Port: 8000 2023-07-29T00:34:57.935 DEBUG:boto:Params: {} 2023-07-29T00:34:57.935 DEBUG:boto:Token: None 2023-07-29T00:34:57.936 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/obj17 2023-07-29T00:34:57.936 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bUn5q/v7YrnojftKTo/NNCJfubw= 2023-07-29T00:34:57.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bUn5q/v7YrnojftKTo/NNCJfubw=', 'Content-Length': '0'} 2023-07-29T00:34:57.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007628807fe1786671-0064c45eb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-07-29T00:34:57.941 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:34:57.943 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:34:57.944 DEBUG:boto:Method: GET 2023-07-29T00:34:57.945 DEBUG:boto:Path: /hiiqos-59/obj17 2023-07-29T00:34:57.946 DEBUG:boto:Data: 2023-07-29T00:34:57.946 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.947 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:57.948 DEBUG:boto:Port: 8001 2023-07-29T00:34:57.949 DEBUG:boto:Params: {} 2023-07-29T00:34:57.950 DEBUG:boto:Token: None 2023-07-29T00:34:57.951 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/obj17 2023-07-29T00:34:57.952 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gYYCJKz3UmdJ1lvRDsLTtrCBrEU= 2023-07-29T00:34:57.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gYYCJKz3UmdJ1lvRDsLTtrCBrEU=', 'Content-Length': '0'} 2023-07-29T00:34:57.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014e630765556fcd7-0064c45eb1-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.956 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:34:57.957 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:34:57.957 DEBUG:boto:Method: GET 2023-07-29T00:34:57.958 DEBUG:boto:Path: /hiiqos-59/obj17 2023-07-29T00:34:57.959 DEBUG:boto:Data: 2023-07-29T00:34:57.960 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.960 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:57.961 DEBUG:boto:Port: 8000 2023-07-29T00:34:57.961 DEBUG:boto:Params: {} 2023-07-29T00:34:57.961 DEBUG:boto:Token: None 2023-07-29T00:34:57.962 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/obj17 2023-07-29T00:34:57.962 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gYYCJKz3UmdJ1lvRDsLTtrCBrEU= 2023-07-29T00:34:57.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gYYCJKz3UmdJ1lvRDsLTtrCBrEU=', 'Content-Length': '0'} 2023-07-29T00:34:57.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c97570595484d3f-0064c45eb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.965 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:57.966 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:57.966 DEBUG:boto:path=/hiiqos-59/?prefix=obj17 2023-07-29T00:34:57.967 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj17 2023-07-29T00:34:57.967 DEBUG:boto:Method: GET 2023-07-29T00:34:57.968 DEBUG:boto:Path: /hiiqos-59/?prefix=obj17 2023-07-29T00:34:57.968 DEBUG:boto:Data: 2023-07-29T00:34:57.969 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.969 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:57.969 DEBUG:boto:Port: 8001 2023-07-29T00:34:57.970 DEBUG:boto:Params: {} 2023-07-29T00:34:57.970 DEBUG:boto:Token: None 2023-07-29T00:34:57.970 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/ 2023-07-29T00:34:57.971 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CdeTsNNlOP+SoOJZQmkiPsPal+A= 2023-07-29T00:34:57.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CdeTsNNlOP+SoOJZQmkiPsPal+A=', 'Content-Length': '0'} 2023-07-29T00:34:57.974 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000246b17a4d40eabb3-0064c45eb1-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:57.974 DEBUG:boto:b'hiiqos-59obj171000falseobj172023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:57.975 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:57.975 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:57.976 DEBUG:boto:path=/hiiqos-59/?prefix=obj17 2023-07-29T00:34:57.976 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj17 2023-07-29T00:34:57.976 DEBUG:boto:Method: GET 2023-07-29T00:34:57.978 DEBUG:boto:Path: /hiiqos-59/?prefix=obj17 2023-07-29T00:34:57.978 DEBUG:boto:Data: 2023-07-29T00:34:57.978 DEBUG:boto:Headers: {} 2023-07-29T00:34:57.979 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:57.979 DEBUG:boto:Port: 8000 2023-07-29T00:34:57.979 DEBUG:boto:Params: {} 2023-07-29T00:34:57.980 DEBUG:boto:Token: None 2023-07-29T00:34:57.980 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:57 GMT /hiiqos-59/ 2023-07-29T00:34:57.981 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CdeTsNNlOP+SoOJZQmkiPsPal+A= 2023-07-29T00:34:57.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CdeTsNNlOP+SoOJZQmkiPsPal+A=', 'Content-Length': '0'} 2023-07-29T00:34:57.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d54f516abf9ab538-0064c45eb1-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.028 DEBUG:boto:b'hiiqos-59obj171000falseobj172023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:58.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-07-29T00:34:58.030 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:34:58.030 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:34:58.031 DEBUG:boto:path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:34:58.031 DEBUG:boto:auth_path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:34:58.031 DEBUG:boto:Method: GET 2023-07-29T00:34:58.032 DEBUG:boto:Path: /hiiqos-59/obj19?versionId=null 2023-07-29T00:34:58.032 DEBUG:boto:Data: 2023-07-29T00:34:58.032 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.033 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:58.033 DEBUG:boto:Port: 8001 2023-07-29T00:34:58.033 DEBUG:boto:Params: {} 2023-07-29T00:34:58.034 DEBUG:boto:Token: None 2023-07-29T00:34:58.034 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj19?versionId=null 2023-07-29T00:34:58.035 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:or60+8U0qVbmuceEa6Q5FdXufWg= 2023-07-29T00:34:58.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:or60+8U0qVbmuceEa6Q5FdXufWg=', 'Content-Length': '0'} 2023-07-29T00:34:58.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080eeead0873ae190-0064c45eb1-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.039 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:34:58.039 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:34:58.040 DEBUG:boto:path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:34:58.040 DEBUG:boto:auth_path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:34:58.041 DEBUG:boto:Method: GET 2023-07-29T00:34:58.041 DEBUG:boto:Path: /hiiqos-59/obj19?versionId=null 2023-07-29T00:34:58.041 DEBUG:boto:Data: 2023-07-29T00:34:58.042 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.042 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:58.042 DEBUG:boto:Port: 8000 2023-07-29T00:34:58.043 DEBUG:boto:Params: {} 2023-07-29T00:34:58.043 DEBUG:boto:Token: None 2023-07-29T00:34:58.044 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj19?versionId=null 2023-07-29T00:34:58.044 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:or60+8U0qVbmuceEa6Q5FdXufWg= 2023-07-29T00:34:58.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:or60+8U0qVbmuceEa6Q5FdXufWg=', 'Content-Length': '0'} 2023-07-29T00:34:58.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f27748f320d8196-0064c45eb1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.047 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:34:58.048 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:34:58.048 DEBUG:boto:path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:34:58.048 DEBUG:boto:auth_path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:34:58.049 DEBUG:boto:Method: HEAD 2023-07-29T00:34:58.049 DEBUG:boto:Path: /hiiqos-59/obj19?versionId=null 2023-07-29T00:34:58.050 DEBUG:boto:Data: 2023-07-29T00:34:58.050 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.050 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:58.051 DEBUG:boto:Port: 8001 2023-07-29T00:34:58.051 DEBUG:boto:Params: {} 2023-07-29T00:34:58.051 DEBUG:boto:Token: None 2023-07-29T00:34:58.052 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj19?versionId=null 2023-07-29T00:34:58.052 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:K5KLtY1+fR7RQ4+P8fl8YTDTkPw= 2023-07-29T00:34:58.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K5KLtY1+fR7RQ4+P8fl8YTDTkPw=', 'Content-Length': '0'} 2023-07-29T00:34:58.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e914fb879ccacce2-0064c45eb1-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:57 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.055 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:34:58.055 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:34:58.056 DEBUG:boto:path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:34:58.056 DEBUG:boto:auth_path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:34:58.056 DEBUG:boto:Method: HEAD 2023-07-29T00:34:58.057 DEBUG:boto:Path: /hiiqos-59/obj19?versionId=null 2023-07-29T00:34:58.057 DEBUG:boto:Data: 2023-07-29T00:34:58.057 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.058 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:58.058 DEBUG:boto:Port: 8000 2023-07-29T00:34:58.058 DEBUG:boto:Params: {} 2023-07-29T00:34:58.059 DEBUG:boto:Token: None 2023-07-29T00:34:58.059 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj19?versionId=null 2023-07-29T00:34:58.059 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:K5KLtY1+fR7RQ4+P8fl8YTDTkPw= 2023-07-29T00:34:58.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K5KLtY1+fR7RQ4+P8fl8YTDTkPw=', 'Content-Length': '0'} 2023-07-29T00:34:58.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d03b2c8278f5b67-0064c45eb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-07-29T00:34:58.063 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:34:58.063 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:34:58.063 DEBUG:boto:path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:34:58.064 DEBUG:boto:auth_path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:34:58.064 DEBUG:boto:Method: GET 2023-07-29T00:34:58.064 DEBUG:boto:Path: /hiiqos-59/obj19?versionId=null 2023-07-29T00:34:58.065 DEBUG:boto:Data: 2023-07-29T00:34:58.065 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.065 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:58.066 DEBUG:boto:Port: 8001 2023-07-29T00:34:58.066 DEBUG:boto:Params: {} 2023-07-29T00:34:58.067 DEBUG:boto:Token: None 2023-07-29T00:34:58.067 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj19?versionId=null 2023-07-29T00:34:58.067 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:or60+8U0qVbmuceEa6Q5FdXufWg= 2023-07-29T00:34:58.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:or60+8U0qVbmuceEa6Q5FdXufWg=', 'Content-Length': '0'} 2023-07-29T00:34:58.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0223718ec187a15-0064c45eb2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.070 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:34:58.071 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:34:58.071 DEBUG:boto:path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:34:58.071 DEBUG:boto:auth_path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:34:58.072 DEBUG:boto:Method: GET 2023-07-29T00:34:58.072 DEBUG:boto:Path: /hiiqos-59/obj19?versionId=null 2023-07-29T00:34:58.072 DEBUG:boto:Data: 2023-07-29T00:34:58.073 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.073 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:58.074 DEBUG:boto:Port: 8000 2023-07-29T00:34:58.074 DEBUG:boto:Params: {} 2023-07-29T00:34:58.074 DEBUG:boto:Token: None 2023-07-29T00:34:58.075 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj19?versionId=null 2023-07-29T00:34:58.075 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:or60+8U0qVbmuceEa6Q5FdXufWg= 2023-07-29T00:34:58.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:or60+8U0qVbmuceEa6Q5FdXufWg=', 'Content-Length': '0'} 2023-07-29T00:34:58.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028bc04057c68a72b-0064c45eb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.078 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:58.079 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:58.079 DEBUG:boto:path=/hiiqos-59/?prefix=obj19 2023-07-29T00:34:58.080 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj19 2023-07-29T00:34:58.080 DEBUG:boto:Method: GET 2023-07-29T00:34:58.080 DEBUG:boto:Path: /hiiqos-59/?prefix=obj19 2023-07-29T00:34:58.081 DEBUG:boto:Data: 2023-07-29T00:34:58.081 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.082 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:58.082 DEBUG:boto:Port: 8001 2023-07-29T00:34:58.082 DEBUG:boto:Params: {} 2023-07-29T00:34:58.083 DEBUG:boto:Token: None 2023-07-29T00:34:58.083 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/ 2023-07-29T00:34:58.083 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw= 2023-07-29T00:34:58.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw=', 'Content-Length': '0'} 2023-07-29T00:34:58.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000953b668d88a87204-0064c45eb2-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.087 DEBUG:boto:b'hiiqos-59obj191000falseobj192023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:58.087 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:58.088 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:58.088 DEBUG:boto:path=/hiiqos-59/?prefix=obj19 2023-07-29T00:34:58.088 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj19 2023-07-29T00:34:58.089 DEBUG:boto:Method: GET 2023-07-29T00:34:58.089 DEBUG:boto:Path: /hiiqos-59/?prefix=obj19 2023-07-29T00:34:58.090 DEBUG:boto:Data: 2023-07-29T00:34:58.090 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.090 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:58.091 DEBUG:boto:Port: 8000 2023-07-29T00:34:58.091 DEBUG:boto:Params: {} 2023-07-29T00:34:58.091 DEBUG:boto:Token: None 2023-07-29T00:34:58.092 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/ 2023-07-29T00:34:58.092 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw= 2023-07-29T00:34:58.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw=', 'Content-Length': '0'} 2023-07-29T00:34:58.096 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061677bbe1c3dab13-0064c45eb2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.096 DEBUG:boto:b'hiiqos-59obj191000falseobj192023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:58.097 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:34:58.097 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:34:58.098 DEBUG:boto:Method: HEAD 2023-07-29T00:34:58.098 DEBUG:boto:Path: /hiiqos-59/obj19 2023-07-29T00:34:58.099 DEBUG:boto:Data: 2023-07-29T00:34:58.099 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.099 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:58.100 DEBUG:boto:Port: 8001 2023-07-29T00:34:58.100 DEBUG:boto:Params: {} 2023-07-29T00:34:58.100 DEBUG:boto:Token: None 2023-07-29T00:34:58.101 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj19 2023-07-29T00:34:58.101 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:H7A8QSTMB9MzJqCnwUUSOmPUsWQ= 2023-07-29T00:34:58.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:H7A8QSTMB9MzJqCnwUUSOmPUsWQ=', 'Content-Length': '0'} 2023-07-29T00:34:58.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa5b910a39b41f79-0064c45eb2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.104 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:34:58.104 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:34:58.104 DEBUG:boto:Method: HEAD 2023-07-29T00:34:58.105 DEBUG:boto:Path: /hiiqos-59/obj19 2023-07-29T00:34:58.105 DEBUG:boto:Data: 2023-07-29T00:34:58.105 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.106 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:58.106 DEBUG:boto:Port: 8000 2023-07-29T00:34:58.106 DEBUG:boto:Params: {} 2023-07-29T00:34:58.107 DEBUG:boto:Token: None 2023-07-29T00:34:58.107 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj19 2023-07-29T00:34:58.108 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:H7A8QSTMB9MzJqCnwUUSOmPUsWQ= 2023-07-29T00:34:58.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:H7A8QSTMB9MzJqCnwUUSOmPUsWQ=', 'Content-Length': '0'} 2023-07-29T00:34:58.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000098cfa4266d0aa2b-0064c45eb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-07-29T00:34:58.111 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:34:58.111 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:34:58.111 DEBUG:boto:Method: GET 2023-07-29T00:34:58.112 DEBUG:boto:Path: /hiiqos-59/obj19 2023-07-29T00:34:58.112 DEBUG:boto:Data: 2023-07-29T00:34:58.113 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.113 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:58.113 DEBUG:boto:Port: 8001 2023-07-29T00:34:58.114 DEBUG:boto:Params: {} 2023-07-29T00:34:58.114 DEBUG:boto:Token: None 2023-07-29T00:34:58.114 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj19 2023-07-29T00:34:58.115 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rb3ONsvEu5yD53BJntvcFFqRNpo= 2023-07-29T00:34:58.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rb3ONsvEu5yD53BJntvcFFqRNpo=', 'Content-Length': '0'} 2023-07-29T00:34:58.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3e953466712f234-0064c45eb2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.118 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:34:58.118 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:34:58.118 DEBUG:boto:Method: GET 2023-07-29T00:34:58.119 DEBUG:boto:Path: /hiiqos-59/obj19 2023-07-29T00:34:58.119 DEBUG:boto:Data: 2023-07-29T00:34:58.120 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.120 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:58.120 DEBUG:boto:Port: 8000 2023-07-29T00:34:58.121 DEBUG:boto:Params: {} 2023-07-29T00:34:58.121 DEBUG:boto:Token: None 2023-07-29T00:34:58.122 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj19 2023-07-29T00:34:58.122 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rb3ONsvEu5yD53BJntvcFFqRNpo= 2023-07-29T00:34:58.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rb3ONsvEu5yD53BJntvcFFqRNpo=', 'Content-Length': '0'} 2023-07-29T00:34:58.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e407e80df624d9f-0064c45eb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.125 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:58.126 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:58.126 DEBUG:boto:path=/hiiqos-59/?prefix=obj19 2023-07-29T00:34:58.126 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj19 2023-07-29T00:34:58.127 DEBUG:boto:Method: GET 2023-07-29T00:34:58.127 DEBUG:boto:Path: /hiiqos-59/?prefix=obj19 2023-07-29T00:34:58.127 DEBUG:boto:Data: 2023-07-29T00:34:58.128 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.128 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:58.128 DEBUG:boto:Port: 8001 2023-07-29T00:34:58.129 DEBUG:boto:Params: {} 2023-07-29T00:34:58.129 DEBUG:boto:Token: None 2023-07-29T00:34:58.130 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/ 2023-07-29T00:34:58.130 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw= 2023-07-29T00:34:58.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw=', 'Content-Length': '0'} 2023-07-29T00:34:58.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005475623b7b3434ed-0064c45eb2-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.133 DEBUG:boto:b'hiiqos-59obj191000falseobj192023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:58.134 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:58.134 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:58.135 DEBUG:boto:path=/hiiqos-59/?prefix=obj19 2023-07-29T00:34:58.135 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj19 2023-07-29T00:34:58.135 DEBUG:boto:Method: GET 2023-07-29T00:34:58.136 DEBUG:boto:Path: /hiiqos-59/?prefix=obj19 2023-07-29T00:34:58.136 DEBUG:boto:Data: 2023-07-29T00:34:58.137 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.137 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:58.137 DEBUG:boto:Port: 8000 2023-07-29T00:34:58.138 DEBUG:boto:Params: {} 2023-07-29T00:34:58.138 DEBUG:boto:Token: None 2023-07-29T00:34:58.138 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/ 2023-07-29T00:34:58.139 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw= 2023-07-29T00:34:58.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw=', 'Content-Length': '0'} 2023-07-29T00:34:58.142 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009241fc8a0b5147b-0064c45eb2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.188 DEBUG:boto:b'hiiqos-59obj191000falseobj192023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:58.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-07-29T00:34:58.190 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:34:58.190 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:34:58.190 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:34:58.191 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:34:58.191 DEBUG:boto:Method: GET 2023-07-29T00:34:58.192 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:34:58.192 DEBUG:boto:Data: 2023-07-29T00:34:58.192 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.193 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:58.193 DEBUG:boto:Port: 8001 2023-07-29T00:34:58.193 DEBUG:boto:Params: {} 2023-07-29T00:34:58.194 DEBUG:boto:Token: None 2023-07-29T00:34:58.195 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:34:58.195 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qm8keAd24aFc706QhQD2h+L7SeU= 2023-07-29T00:34:58.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qm8keAd24aFc706QhQD2h+L7SeU=', 'Content-Length': '0'} 2023-07-29T00:34:58.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9859a5afe9f0162-0064c45eb2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.198 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:34:58.199 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:34:58.199 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:34:58.199 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:34:58.200 DEBUG:boto:Method: GET 2023-07-29T00:34:58.200 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:34:58.200 DEBUG:boto:Data: 2023-07-29T00:34:58.201 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.201 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:58.202 DEBUG:boto:Port: 8000 2023-07-29T00:34:58.202 DEBUG:boto:Params: {} 2023-07-29T00:34:58.202 DEBUG:boto:Token: None 2023-07-29T00:34:58.203 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:34:58.203 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qm8keAd24aFc706QhQD2h+L7SeU= 2023-07-29T00:34:58.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qm8keAd24aFc706QhQD2h+L7SeU=', 'Content-Length': '0'} 2023-07-29T00:34:58.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4ed4a4d79a08dcc-0064c45eb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.206 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:34:58.207 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:34:58.207 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:34:58.207 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:34:58.208 DEBUG:boto:Method: HEAD 2023-07-29T00:34:58.208 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:34:58.208 DEBUG:boto:Data: 2023-07-29T00:34:58.209 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.209 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:58.209 DEBUG:boto:Port: 8001 2023-07-29T00:34:58.210 DEBUG:boto:Params: {} 2023-07-29T00:34:58.210 DEBUG:boto:Token: None 2023-07-29T00:34:58.210 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:34:58.211 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CrTiu6BZka2M+/woH2BK80Bwd0s= 2023-07-29T00:34:58.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CrTiu6BZka2M+/woH2BK80Bwd0s=', 'Content-Length': '0'} 2023-07-29T00:34:58.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006427361384034b1c-0064c45eb2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.214 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:34:58.214 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:34:58.215 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:34:58.215 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:34:58.215 DEBUG:boto:Method: HEAD 2023-07-29T00:34:58.216 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:34:58.216 DEBUG:boto:Data: 2023-07-29T00:34:58.216 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.217 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:58.217 DEBUG:boto:Port: 8000 2023-07-29T00:34:58.217 DEBUG:boto:Params: {} 2023-07-29T00:34:58.218 DEBUG:boto:Token: None 2023-07-29T00:34:58.218 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:34:58.218 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CrTiu6BZka2M+/woH2BK80Bwd0s= 2023-07-29T00:34:58.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CrTiu6BZka2M+/woH2BK80Bwd0s=', 'Content-Length': '0'} 2023-07-29T00:34:58.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b29d2e9810533fa-0064c45eb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-07-29T00:34:58.221 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:34:58.222 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:34:58.222 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:34:58.222 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:34:58.223 DEBUG:boto:Method: GET 2023-07-29T00:34:58.223 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:34:58.224 DEBUG:boto:Data: 2023-07-29T00:34:58.224 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.224 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:58.225 DEBUG:boto:Port: 8001 2023-07-29T00:34:58.225 DEBUG:boto:Params: {} 2023-07-29T00:34:58.225 DEBUG:boto:Token: None 2023-07-29T00:34:58.226 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:34:58.226 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qm8keAd24aFc706QhQD2h+L7SeU= 2023-07-29T00:34:58.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qm8keAd24aFc706QhQD2h+L7SeU=', 'Content-Length': '0'} 2023-07-29T00:34:58.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b418193be3e3c543-0064c45eb2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.272 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:34:58.273 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:34:58.273 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:34:58.274 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:34:58.274 DEBUG:boto:Method: GET 2023-07-29T00:34:58.274 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:34:58.275 DEBUG:boto:Data: 2023-07-29T00:34:58.275 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.276 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:58.276 DEBUG:boto:Port: 8000 2023-07-29T00:34:58.276 DEBUG:boto:Params: {} 2023-07-29T00:34:58.277 DEBUG:boto:Token: None 2023-07-29T00:34:58.277 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:34:58.278 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qm8keAd24aFc706QhQD2h+L7SeU= 2023-07-29T00:34:58.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qm8keAd24aFc706QhQD2h+L7SeU=', 'Content-Length': '0'} 2023-07-29T00:34:58.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002496dfb34815eb91-0064c45eb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.281 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:58.281 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:58.281 DEBUG:boto:path=/hiiqos-59/?prefix=obj22 2023-07-29T00:34:58.282 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj22 2023-07-29T00:34:58.282 DEBUG:boto:Method: GET 2023-07-29T00:34:58.283 DEBUG:boto:Path: /hiiqos-59/?prefix=obj22 2023-07-29T00:34:58.283 DEBUG:boto:Data: 2023-07-29T00:34:58.283 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.284 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:58.284 DEBUG:boto:Port: 8001 2023-07-29T00:34:58.285 DEBUG:boto:Params: {} 2023-07-29T00:34:58.285 DEBUG:boto:Token: None 2023-07-29T00:34:58.286 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/ 2023-07-29T00:34:58.286 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw= 2023-07-29T00:34:58.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw=', 'Content-Length': '0'} 2023-07-29T00:34:58.289 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d234315f7881315-0064c45eb2-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.290 DEBUG:boto:b'hiiqos-59obj221000falseobj222023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:58.290 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:58.291 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:58.291 DEBUG:boto:path=/hiiqos-59/?prefix=obj22 2023-07-29T00:34:58.292 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj22 2023-07-29T00:34:58.292 DEBUG:boto:Method: GET 2023-07-29T00:34:58.293 DEBUG:boto:Path: /hiiqos-59/?prefix=obj22 2023-07-29T00:34:58.293 DEBUG:boto:Data: 2023-07-29T00:34:58.293 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.294 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:58.294 DEBUG:boto:Port: 8000 2023-07-29T00:34:58.294 DEBUG:boto:Params: {} 2023-07-29T00:34:58.295 DEBUG:boto:Token: None 2023-07-29T00:34:58.295 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/ 2023-07-29T00:34:58.296 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw= 2023-07-29T00:34:58.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw=', 'Content-Length': '0'} 2023-07-29T00:34:58.299 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f046bf482858eb68-0064c45eb2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.300 DEBUG:boto:b'hiiqos-59obj221000falseobj222023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:58.301 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:34:58.301 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:34:58.301 DEBUG:boto:Method: HEAD 2023-07-29T00:34:58.302 DEBUG:boto:Path: /hiiqos-59/obj22 2023-07-29T00:34:58.302 DEBUG:boto:Data: 2023-07-29T00:34:58.302 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.303 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:58.303 DEBUG:boto:Port: 8001 2023-07-29T00:34:58.303 DEBUG:boto:Params: {} 2023-07-29T00:34:58.304 DEBUG:boto:Token: None 2023-07-29T00:34:58.304 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj22 2023-07-29T00:34:58.305 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1oPGTAEfmlgiMZUzHIx8i/QDsnI= 2023-07-29T00:34:58.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1oPGTAEfmlgiMZUzHIx8i/QDsnI=', 'Content-Length': '0'} 2023-07-29T00:34:58.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e6128da1ffe0fe2-0064c45eb2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.307 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:34:58.308 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:34:58.308 DEBUG:boto:Method: HEAD 2023-07-29T00:34:58.309 DEBUG:boto:Path: /hiiqos-59/obj22 2023-07-29T00:34:58.309 DEBUG:boto:Data: 2023-07-29T00:34:58.309 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.310 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:58.310 DEBUG:boto:Port: 8000 2023-07-29T00:34:58.310 DEBUG:boto:Params: {} 2023-07-29T00:34:58.311 DEBUG:boto:Token: None 2023-07-29T00:34:58.311 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj22 2023-07-29T00:34:58.312 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1oPGTAEfmlgiMZUzHIx8i/QDsnI= 2023-07-29T00:34:58.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1oPGTAEfmlgiMZUzHIx8i/QDsnI=', 'Content-Length': '0'} 2023-07-29T00:34:58.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f82c570a6d5f593b-0064c45eb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-07-29T00:34:58.315 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:34:58.315 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:34:58.316 DEBUG:boto:Method: GET 2023-07-29T00:34:58.316 DEBUG:boto:Path: /hiiqos-59/obj22 2023-07-29T00:34:58.317 DEBUG:boto:Data: 2023-07-29T00:34:58.317 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.317 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:58.318 DEBUG:boto:Port: 8001 2023-07-29T00:34:58.318 DEBUG:boto:Params: {} 2023-07-29T00:34:58.319 DEBUG:boto:Token: None 2023-07-29T00:34:58.319 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj22 2023-07-29T00:34:58.320 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MqEmHhPdVtC1u5D+KyHVhbxWHFI= 2023-07-29T00:34:58.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MqEmHhPdVtC1u5D+KyHVhbxWHFI=', 'Content-Length': '0'} 2023-07-29T00:34:58.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002691244beb880cd3-0064c45eb2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.323 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:34:58.323 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:34:58.324 DEBUG:boto:Method: GET 2023-07-29T00:34:58.324 DEBUG:boto:Path: /hiiqos-59/obj22 2023-07-29T00:34:58.325 DEBUG:boto:Data: 2023-07-29T00:34:58.325 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.325 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:58.326 DEBUG:boto:Port: 8000 2023-07-29T00:34:58.326 DEBUG:boto:Params: {} 2023-07-29T00:34:58.327 DEBUG:boto:Token: None 2023-07-29T00:34:58.327 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj22 2023-07-29T00:34:58.328 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MqEmHhPdVtC1u5D+KyHVhbxWHFI= 2023-07-29T00:34:58.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MqEmHhPdVtC1u5D+KyHVhbxWHFI=', 'Content-Length': '0'} 2023-07-29T00:34:58.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba2fa85bb4f08ac3-0064c45eb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.331 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:58.331 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:58.332 DEBUG:boto:path=/hiiqos-59/?prefix=obj22 2023-07-29T00:34:58.332 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj22 2023-07-29T00:34:58.332 DEBUG:boto:Method: GET 2023-07-29T00:34:58.333 DEBUG:boto:Path: /hiiqos-59/?prefix=obj22 2023-07-29T00:34:58.333 DEBUG:boto:Data: 2023-07-29T00:34:58.334 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.334 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:58.334 DEBUG:boto:Port: 8001 2023-07-29T00:34:58.335 DEBUG:boto:Params: {} 2023-07-29T00:34:58.335 DEBUG:boto:Token: None 2023-07-29T00:34:58.336 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/ 2023-07-29T00:34:58.336 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw= 2023-07-29T00:34:58.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw=', 'Content-Length': '0'} 2023-07-29T00:34:58.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000baa2f9fed90c8d57-0064c45eb2-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.340 DEBUG:boto:b'hiiqos-59obj221000falseobj222023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:58.340 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:58.341 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:58.341 DEBUG:boto:path=/hiiqos-59/?prefix=obj22 2023-07-29T00:34:58.342 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj22 2023-07-29T00:34:58.342 DEBUG:boto:Method: GET 2023-07-29T00:34:58.342 DEBUG:boto:Path: /hiiqos-59/?prefix=obj22 2023-07-29T00:34:58.343 DEBUG:boto:Data: 2023-07-29T00:34:58.343 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.343 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:58.344 DEBUG:boto:Port: 8000 2023-07-29T00:34:58.344 DEBUG:boto:Params: {} 2023-07-29T00:34:58.345 DEBUG:boto:Token: None 2023-07-29T00:34:58.345 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/ 2023-07-29T00:34:58.346 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw= 2023-07-29T00:34:58.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw=', 'Content-Length': '0'} 2023-07-29T00:34:58.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a22de720de8d246f-0064c45eb2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.350 DEBUG:boto:b'hiiqos-59obj221000falseobj222023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:58.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-07-29T00:34:58.351 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:34:58.351 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:34:58.352 DEBUG:boto:path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:34:58.352 DEBUG:boto:auth_path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:34:58.352 DEBUG:boto:Method: GET 2023-07-29T00:34:58.353 DEBUG:boto:Path: /hiiqos-59/obj23?versionId=null 2023-07-29T00:34:58.353 DEBUG:boto:Data: 2023-07-29T00:34:58.353 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.354 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:58.354 DEBUG:boto:Port: 8001 2023-07-29T00:34:58.354 DEBUG:boto:Params: {} 2023-07-29T00:34:58.355 DEBUG:boto:Token: None 2023-07-29T00:34:58.355 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj23?versionId=null 2023-07-29T00:34:58.356 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QG9ZtINVg6BoKxupv1j5t+slo7U= 2023-07-29T00:34:58.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QG9ZtINVg6BoKxupv1j5t+slo7U=', 'Content-Length': '0'} 2023-07-29T00:34:58.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee769aa49f6ea7c9-0064c45eb2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.359 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:34:58.360 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:34:58.360 DEBUG:boto:path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:34:58.360 DEBUG:boto:auth_path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:34:58.361 DEBUG:boto:Method: GET 2023-07-29T00:34:58.361 DEBUG:boto:Path: /hiiqos-59/obj23?versionId=null 2023-07-29T00:34:58.361 DEBUG:boto:Data: 2023-07-29T00:34:58.362 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.362 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:58.363 DEBUG:boto:Port: 8000 2023-07-29T00:34:58.363 DEBUG:boto:Params: {} 2023-07-29T00:34:58.363 DEBUG:boto:Token: None 2023-07-29T00:34:58.364 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj23?versionId=null 2023-07-29T00:34:58.365 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QG9ZtINVg6BoKxupv1j5t+slo7U= 2023-07-29T00:34:58.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QG9ZtINVg6BoKxupv1j5t+slo7U=', 'Content-Length': '0'} 2023-07-29T00:34:58.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ae7fbe6bcf6c57e-0064c45eb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.368 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:34:58.368 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:34:58.369 DEBUG:boto:path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:34:58.369 DEBUG:boto:auth_path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:34:58.369 DEBUG:boto:Method: HEAD 2023-07-29T00:34:58.370 DEBUG:boto:Path: /hiiqos-59/obj23?versionId=null 2023-07-29T00:34:58.370 DEBUG:boto:Data: 2023-07-29T00:34:58.371 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.371 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:58.371 DEBUG:boto:Port: 8001 2023-07-29T00:34:58.372 DEBUG:boto:Params: {} 2023-07-29T00:34:58.372 DEBUG:boto:Token: None 2023-07-29T00:34:58.372 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj23?versionId=null 2023-07-29T00:34:58.373 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6lCCRDthSTLxe3B3CLpsvclkmLc= 2023-07-29T00:34:58.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6lCCRDthSTLxe3B3CLpsvclkmLc=', 'Content-Length': '0'} 2023-07-29T00:34:58.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9206830f8b1ebd5-0064c45eb2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.376 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:34:58.376 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:34:58.376 DEBUG:boto:path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:34:58.377 DEBUG:boto:auth_path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:34:58.377 DEBUG:boto:Method: HEAD 2023-07-29T00:34:58.377 DEBUG:boto:Path: /hiiqos-59/obj23?versionId=null 2023-07-29T00:34:58.378 DEBUG:boto:Data: 2023-07-29T00:34:58.378 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.378 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:58.379 DEBUG:boto:Port: 8000 2023-07-29T00:34:58.379 DEBUG:boto:Params: {} 2023-07-29T00:34:58.379 DEBUG:boto:Token: None 2023-07-29T00:34:58.380 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj23?versionId=null 2023-07-29T00:34:58.380 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6lCCRDthSTLxe3B3CLpsvclkmLc= 2023-07-29T00:34:58.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6lCCRDthSTLxe3B3CLpsvclkmLc=', 'Content-Length': '0'} 2023-07-29T00:34:58.383 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d1288764330232a-0064c45eb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-07-29T00:34:58.384 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:34:58.384 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:34:58.385 DEBUG:boto:path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:34:58.385 DEBUG:boto:auth_path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:34:58.385 DEBUG:boto:Method: GET 2023-07-29T00:34:58.386 DEBUG:boto:Path: /hiiqos-59/obj23?versionId=null 2023-07-29T00:34:58.386 DEBUG:boto:Data: 2023-07-29T00:34:58.386 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.387 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:58.387 DEBUG:boto:Port: 8001 2023-07-29T00:34:58.387 DEBUG:boto:Params: {} 2023-07-29T00:34:58.388 DEBUG:boto:Token: None 2023-07-29T00:34:58.392 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj23?versionId=null 2023-07-29T00:34:58.392 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QG9ZtINVg6BoKxupv1j5t+slo7U= 2023-07-29T00:34:58.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QG9ZtINVg6BoKxupv1j5t+slo7U=', 'Content-Length': '0'} 2023-07-29T00:34:58.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071e71e66315eb88f-0064c45eb2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.396 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:34:58.396 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:34:58.396 DEBUG:boto:path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:34:58.397 DEBUG:boto:auth_path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:34:58.397 DEBUG:boto:Method: GET 2023-07-29T00:34:58.398 DEBUG:boto:Path: /hiiqos-59/obj23?versionId=null 2023-07-29T00:34:58.398 DEBUG:boto:Data: 2023-07-29T00:34:58.398 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.399 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:58.399 DEBUG:boto:Port: 8000 2023-07-29T00:34:58.399 DEBUG:boto:Params: {} 2023-07-29T00:34:58.400 DEBUG:boto:Token: None 2023-07-29T00:34:58.400 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj23?versionId=null 2023-07-29T00:34:58.401 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QG9ZtINVg6BoKxupv1j5t+slo7U= 2023-07-29T00:34:58.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QG9ZtINVg6BoKxupv1j5t+slo7U=', 'Content-Length': '0'} 2023-07-29T00:34:58.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021e1ef7868da46bc-0064c45eb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.404 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:58.404 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:58.405 DEBUG:boto:path=/hiiqos-59/?prefix=obj23 2023-07-29T00:34:58.405 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj23 2023-07-29T00:34:58.405 DEBUG:boto:Method: GET 2023-07-29T00:34:58.406 DEBUG:boto:Path: /hiiqos-59/?prefix=obj23 2023-07-29T00:34:58.406 DEBUG:boto:Data: 2023-07-29T00:34:58.406 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.407 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:58.407 DEBUG:boto:Port: 8001 2023-07-29T00:34:58.408 DEBUG:boto:Params: {} 2023-07-29T00:34:58.408 DEBUG:boto:Token: None 2023-07-29T00:34:58.408 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/ 2023-07-29T00:34:58.409 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw= 2023-07-29T00:34:58.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw=', 'Content-Length': '0'} 2023-07-29T00:34:58.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f20a9c1764f4efb3-0064c45eb2-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.412 DEBUG:boto:b'hiiqos-59obj231000falseobj232023-07-29T00:33:06.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:58.413 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:58.413 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:58.414 DEBUG:boto:path=/hiiqos-59/?prefix=obj23 2023-07-29T00:34:58.414 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj23 2023-07-29T00:34:58.414 DEBUG:boto:Method: GET 2023-07-29T00:34:58.415 DEBUG:boto:Path: /hiiqos-59/?prefix=obj23 2023-07-29T00:34:58.415 DEBUG:boto:Data: 2023-07-29T00:34:58.415 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.416 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:58.416 DEBUG:boto:Port: 8000 2023-07-29T00:34:58.416 DEBUG:boto:Params: {} 2023-07-29T00:34:58.417 DEBUG:boto:Token: None 2023-07-29T00:34:58.417 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/ 2023-07-29T00:34:58.418 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw= 2023-07-29T00:34:58.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw=', 'Content-Length': '0'} 2023-07-29T00:34:58.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065fd0e66f9868fa1-0064c45eb2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.422 DEBUG:boto:b'hiiqos-59obj231000falseobj232023-07-29T00:33:06.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:58.423 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:34:58.423 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:34:58.423 DEBUG:boto:Method: HEAD 2023-07-29T00:34:58.424 DEBUG:boto:Path: /hiiqos-59/obj23 2023-07-29T00:34:58.424 DEBUG:boto:Data: 2023-07-29T00:34:58.425 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.425 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:58.425 DEBUG:boto:Port: 8001 2023-07-29T00:34:58.425 DEBUG:boto:Params: {} 2023-07-29T00:34:58.426 DEBUG:boto:Token: None 2023-07-29T00:34:58.426 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj23 2023-07-29T00:34:58.427 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xAv0EDUvs7zuz5rUb6e/dNZn38M= 2023-07-29T00:34:58.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xAv0EDUvs7zuz5rUb6e/dNZn38M=', 'Content-Length': '0'} 2023-07-29T00:34:58.429 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000489bc33bb1ef32c5-0064c45eb2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.430 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:34:58.430 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:34:58.430 DEBUG:boto:Method: HEAD 2023-07-29T00:34:58.431 DEBUG:boto:Path: /hiiqos-59/obj23 2023-07-29T00:34:58.431 DEBUG:boto:Data: 2023-07-29T00:34:58.431 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.432 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:58.432 DEBUG:boto:Port: 8000 2023-07-29T00:34:58.432 DEBUG:boto:Params: {} 2023-07-29T00:34:58.433 DEBUG:boto:Token: None 2023-07-29T00:34:58.433 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj23 2023-07-29T00:34:58.434 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xAv0EDUvs7zuz5rUb6e/dNZn38M= 2023-07-29T00:34:58.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xAv0EDUvs7zuz5rUb6e/dNZn38M=', 'Content-Length': '0'} 2023-07-29T00:34:58.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecb6899dd533669d-0064c45eb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-07-29T00:34:58.437 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:34:58.437 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:34:58.437 DEBUG:boto:Method: GET 2023-07-29T00:34:58.438 DEBUG:boto:Path: /hiiqos-59/obj23 2023-07-29T00:34:58.438 DEBUG:boto:Data: 2023-07-29T00:34:58.438 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.439 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:58.439 DEBUG:boto:Port: 8001 2023-07-29T00:34:58.439 DEBUG:boto:Params: {} 2023-07-29T00:34:58.440 DEBUG:boto:Token: None 2023-07-29T00:34:58.440 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj23 2023-07-29T00:34:58.441 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IvwJzPhou7OfjVp46kV4Vrcs8sk= 2023-07-29T00:34:58.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IvwJzPhou7OfjVp46kV4Vrcs8sk=', 'Content-Length': '0'} 2023-07-29T00:34:58.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de0dd824459e9c4f-0064c45eb2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.484 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:34:58.485 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:34:58.486 DEBUG:boto:Method: GET 2023-07-29T00:34:58.486 DEBUG:boto:Path: /hiiqos-59/obj23 2023-07-29T00:34:58.486 DEBUG:boto:Data: 2023-07-29T00:34:58.487 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.487 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:58.488 DEBUG:boto:Port: 8000 2023-07-29T00:34:58.488 DEBUG:boto:Params: {} 2023-07-29T00:34:58.488 DEBUG:boto:Token: None 2023-07-29T00:34:58.489 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj23 2023-07-29T00:34:58.489 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IvwJzPhou7OfjVp46kV4Vrcs8sk= 2023-07-29T00:34:58.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IvwJzPhou7OfjVp46kV4Vrcs8sk=', 'Content-Length': '0'} 2023-07-29T00:34:58.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8004630d4022717-0064c45eb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.493 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:58.493 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:58.494 DEBUG:boto:path=/hiiqos-59/?prefix=obj23 2023-07-29T00:34:58.494 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj23 2023-07-29T00:34:58.494 DEBUG:boto:Method: GET 2023-07-29T00:34:58.495 DEBUG:boto:Path: /hiiqos-59/?prefix=obj23 2023-07-29T00:34:58.495 DEBUG:boto:Data: 2023-07-29T00:34:58.495 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.496 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:58.496 DEBUG:boto:Port: 8001 2023-07-29T00:34:58.496 DEBUG:boto:Params: {} 2023-07-29T00:34:58.497 DEBUG:boto:Token: None 2023-07-29T00:34:58.497 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/ 2023-07-29T00:34:58.498 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw= 2023-07-29T00:34:58.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw=', 'Content-Length': '0'} 2023-07-29T00:34:58.500 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f2824bbec682a22-0064c45eb2-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.501 DEBUG:boto:b'hiiqos-59obj231000falseobj232023-07-29T00:33:06.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:58.502 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:58.502 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:58.502 DEBUG:boto:path=/hiiqos-59/?prefix=obj23 2023-07-29T00:34:58.503 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj23 2023-07-29T00:34:58.503 DEBUG:boto:Method: GET 2023-07-29T00:34:58.504 DEBUG:boto:Path: /hiiqos-59/?prefix=obj23 2023-07-29T00:34:58.504 DEBUG:boto:Data: 2023-07-29T00:34:58.504 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.505 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:58.505 DEBUG:boto:Port: 8000 2023-07-29T00:34:58.505 DEBUG:boto:Params: {} 2023-07-29T00:34:58.506 DEBUG:boto:Token: None 2023-07-29T00:34:58.506 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/ 2023-07-29T00:34:58.507 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw= 2023-07-29T00:34:58.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw=', 'Content-Length': '0'} 2023-07-29T00:34:58.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cfaea7e26869162-0064c45eb2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.511 DEBUG:boto:b'hiiqos-59obj231000falseobj232023-07-29T00:33:06.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:58.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-07-29T00:34:58.512 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:34:58.512 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:34:58.513 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:34:58.513 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:34:58.514 DEBUG:boto:Method: GET 2023-07-29T00:34:58.514 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:34:58.514 DEBUG:boto:Data: 2023-07-29T00:34:58.515 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.515 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:58.515 DEBUG:boto:Port: 8001 2023-07-29T00:34:58.516 DEBUG:boto:Params: {} 2023-07-29T00:34:58.516 DEBUG:boto:Token: None 2023-07-29T00:34:58.516 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:34:58.517 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UJKvspHUYCtDy8Nu0TpkouYf3n0= 2023-07-29T00:34:58.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UJKvspHUYCtDy8Nu0TpkouYf3n0=', 'Content-Length': '0'} 2023-07-29T00:34:58.519 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5900872aff811ab-0064c45eb2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.520 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:34:58.520 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:34:58.521 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:34:58.521 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:34:58.521 DEBUG:boto:Method: GET 2023-07-29T00:34:58.522 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:34:58.522 DEBUG:boto:Data: 2023-07-29T00:34:58.522 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.523 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:58.523 DEBUG:boto:Port: 8000 2023-07-29T00:34:58.523 DEBUG:boto:Params: {} 2023-07-29T00:34:58.524 DEBUG:boto:Token: None 2023-07-29T00:34:58.524 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:34:58.525 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UJKvspHUYCtDy8Nu0TpkouYf3n0= 2023-07-29T00:34:58.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UJKvspHUYCtDy8Nu0TpkouYf3n0=', 'Content-Length': '0'} 2023-07-29T00:34:58.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef3de6fa551abb72-0064c45eb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.528 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:34:58.528 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:34:58.528 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:34:58.529 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:34:58.529 DEBUG:boto:Method: HEAD 2023-07-29T00:34:58.529 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:34:58.530 DEBUG:boto:Data: 2023-07-29T00:34:58.530 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.530 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:58.531 DEBUG:boto:Port: 8001 2023-07-29T00:34:58.531 DEBUG:boto:Params: {} 2023-07-29T00:34:58.531 DEBUG:boto:Token: None 2023-07-29T00:34:58.532 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:34:58.532 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zT7ZtQsQEdQVPHh0EyDTOwHAl0w= 2023-07-29T00:34:58.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zT7ZtQsQEdQVPHh0EyDTOwHAl0w=', 'Content-Length': '0'} 2023-07-29T00:34:58.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095612a4766bb33ef-0064c45eb2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.535 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:34:58.535 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:34:58.536 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:34:58.536 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:34:58.536 DEBUG:boto:Method: HEAD 2023-07-29T00:34:58.537 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:34:58.537 DEBUG:boto:Data: 2023-07-29T00:34:58.537 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.538 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:58.538 DEBUG:boto:Port: 8000 2023-07-29T00:34:58.538 DEBUG:boto:Params: {} 2023-07-29T00:34:58.539 DEBUG:boto:Token: None 2023-07-29T00:34:58.539 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:34:58.539 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zT7ZtQsQEdQVPHh0EyDTOwHAl0w= 2023-07-29T00:34:58.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zT7ZtQsQEdQVPHh0EyDTOwHAl0w=', 'Content-Length': '0'} 2023-07-29T00:34:58.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2d7fd3e3909facc-0064c45eb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-07-29T00:34:58.543 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:34:58.543 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:34:58.543 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:34:58.544 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:34:58.544 DEBUG:boto:Method: GET 2023-07-29T00:34:58.544 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:34:58.545 DEBUG:boto:Data: 2023-07-29T00:34:58.545 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.546 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:58.546 DEBUG:boto:Port: 8001 2023-07-29T00:34:58.546 DEBUG:boto:Params: {} 2023-07-29T00:34:58.547 DEBUG:boto:Token: None 2023-07-29T00:34:58.547 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:34:58.548 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UJKvspHUYCtDy8Nu0TpkouYf3n0= 2023-07-29T00:34:58.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UJKvspHUYCtDy8Nu0TpkouYf3n0=', 'Content-Length': '0'} 2023-07-29T00:34:58.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fac8ac0f2e140ed-0064c45eb2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.550 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:34:58.551 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:34:58.551 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:34:58.552 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:34:58.552 DEBUG:boto:Method: GET 2023-07-29T00:34:58.552 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:34:58.553 DEBUG:boto:Data: 2023-07-29T00:34:58.553 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.553 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:58.554 DEBUG:boto:Port: 8000 2023-07-29T00:34:58.554 DEBUG:boto:Params: {} 2023-07-29T00:34:58.554 DEBUG:boto:Token: None 2023-07-29T00:34:58.555 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:34:58.555 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UJKvspHUYCtDy8Nu0TpkouYf3n0= 2023-07-29T00:34:58.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UJKvspHUYCtDy8Nu0TpkouYf3n0=', 'Content-Length': '0'} 2023-07-29T00:34:58.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffc57a93fcd30345-0064c45eb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.559 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:58.559 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:58.560 DEBUG:boto:path=/hiiqos-59/?prefix=obj33 2023-07-29T00:34:58.560 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj33 2023-07-29T00:34:58.560 DEBUG:boto:Method: GET 2023-07-29T00:34:58.561 DEBUG:boto:Path: /hiiqos-59/?prefix=obj33 2023-07-29T00:34:58.561 DEBUG:boto:Data: 2023-07-29T00:34:58.562 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.562 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:58.563 DEBUG:boto:Port: 8001 2023-07-29T00:34:58.563 DEBUG:boto:Params: {} 2023-07-29T00:34:58.563 DEBUG:boto:Token: None 2023-07-29T00:34:58.564 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/ 2023-07-29T00:34:58.564 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw= 2023-07-29T00:34:58.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw=', 'Content-Length': '0'} 2023-07-29T00:34:58.567 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003478ab86eaa86324-0064c45eb2-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.568 DEBUG:boto:b'hiiqos-59obj331000falseobj332023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:58.568 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:58.569 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:58.569 DEBUG:boto:path=/hiiqos-59/?prefix=obj33 2023-07-29T00:34:58.569 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj33 2023-07-29T00:34:58.570 DEBUG:boto:Method: GET 2023-07-29T00:34:58.570 DEBUG:boto:Path: /hiiqos-59/?prefix=obj33 2023-07-29T00:34:58.571 DEBUG:boto:Data: 2023-07-29T00:34:58.571 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.571 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:58.572 DEBUG:boto:Port: 8000 2023-07-29T00:34:58.572 DEBUG:boto:Params: {} 2023-07-29T00:34:58.572 DEBUG:boto:Token: None 2023-07-29T00:34:58.573 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/ 2023-07-29T00:34:58.573 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw= 2023-07-29T00:34:58.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw=', 'Content-Length': '0'} 2023-07-29T00:34:58.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fa3df322096d535-0064c45eb2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.578 DEBUG:boto:b'hiiqos-59obj331000falseobj332023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:58.578 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:34:58.579 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:34:58.579 DEBUG:boto:Method: HEAD 2023-07-29T00:34:58.580 DEBUG:boto:Path: /hiiqos-59/obj33 2023-07-29T00:34:58.580 DEBUG:boto:Data: 2023-07-29T00:34:58.580 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.581 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:58.581 DEBUG:boto:Port: 8001 2023-07-29T00:34:58.582 DEBUG:boto:Params: {} 2023-07-29T00:34:58.582 DEBUG:boto:Token: None 2023-07-29T00:34:58.582 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj33 2023-07-29T00:34:58.583 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MpnFRoQC0hBfabqC48y4POP8M/0= 2023-07-29T00:34:58.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MpnFRoQC0hBfabqC48y4POP8M/0=', 'Content-Length': '0'} 2023-07-29T00:34:58.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efa3503a5746e85a-0064c45eb2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.585 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:34:58.586 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:34:58.586 DEBUG:boto:Method: HEAD 2023-07-29T00:34:58.586 DEBUG:boto:Path: /hiiqos-59/obj33 2023-07-29T00:34:58.587 DEBUG:boto:Data: 2023-07-29T00:34:58.587 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.587 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:58.588 DEBUG:boto:Port: 8000 2023-07-29T00:34:58.588 DEBUG:boto:Params: {} 2023-07-29T00:34:58.589 DEBUG:boto:Token: None 2023-07-29T00:34:58.589 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj33 2023-07-29T00:34:58.589 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MpnFRoQC0hBfabqC48y4POP8M/0= 2023-07-29T00:34:58.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MpnFRoQC0hBfabqC48y4POP8M/0=', 'Content-Length': '0'} 2023-07-29T00:34:58.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df2a6a3bb6a2debe-0064c45eb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-07-29T00:34:58.592 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:34:58.593 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:34:58.593 DEBUG:boto:Method: GET 2023-07-29T00:34:58.594 DEBUG:boto:Path: /hiiqos-59/obj33 2023-07-29T00:34:58.594 DEBUG:boto:Data: 2023-07-29T00:34:58.594 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.595 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:58.595 DEBUG:boto:Port: 8001 2023-07-29T00:34:58.595 DEBUG:boto:Params: {} 2023-07-29T00:34:58.596 DEBUG:boto:Token: None 2023-07-29T00:34:58.596 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj33 2023-07-29T00:34:58.597 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Zt0FAjUI/4POWiijK7FWPIfDks4= 2023-07-29T00:34:58.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Zt0FAjUI/4POWiijK7FWPIfDks4=', 'Content-Length': '0'} 2023-07-29T00:34:58.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e18b659a874d27a5-0064c45eb2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.644 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:34:58.645 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:34:58.645 DEBUG:boto:Method: GET 2023-07-29T00:34:58.646 DEBUG:boto:Path: /hiiqos-59/obj33 2023-07-29T00:34:58.646 DEBUG:boto:Data: 2023-07-29T00:34:58.647 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.647 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:58.647 DEBUG:boto:Port: 8000 2023-07-29T00:34:58.648 DEBUG:boto:Params: {} 2023-07-29T00:34:58.648 DEBUG:boto:Token: None 2023-07-29T00:34:58.649 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj33 2023-07-29T00:34:58.649 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Zt0FAjUI/4POWiijK7FWPIfDks4= 2023-07-29T00:34:58.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Zt0FAjUI/4POWiijK7FWPIfDks4=', 'Content-Length': '0'} 2023-07-29T00:34:58.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008112e099f865c8ce-0064c45eb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.653 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:58.653 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:58.653 DEBUG:boto:path=/hiiqos-59/?prefix=obj33 2023-07-29T00:34:58.654 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj33 2023-07-29T00:34:58.654 DEBUG:boto:Method: GET 2023-07-29T00:34:58.654 DEBUG:boto:Path: /hiiqos-59/?prefix=obj33 2023-07-29T00:34:58.655 DEBUG:boto:Data: 2023-07-29T00:34:58.655 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.655 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:58.656 DEBUG:boto:Port: 8001 2023-07-29T00:34:58.656 DEBUG:boto:Params: {} 2023-07-29T00:34:58.657 DEBUG:boto:Token: None 2023-07-29T00:34:58.657 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/ 2023-07-29T00:34:58.657 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw= 2023-07-29T00:34:58.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw=', 'Content-Length': '0'} 2023-07-29T00:34:58.661 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6e91b632869a4aa-0064c45eb2-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.661 DEBUG:boto:b'hiiqos-59obj331000falseobj332023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:58.662 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:58.662 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:58.663 DEBUG:boto:path=/hiiqos-59/?prefix=obj33 2023-07-29T00:34:58.663 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj33 2023-07-29T00:34:58.663 DEBUG:boto:Method: GET 2023-07-29T00:34:58.664 DEBUG:boto:Path: /hiiqos-59/?prefix=obj33 2023-07-29T00:34:58.664 DEBUG:boto:Data: 2023-07-29T00:34:58.664 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.665 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:58.665 DEBUG:boto:Port: 8000 2023-07-29T00:34:58.666 DEBUG:boto:Params: {} 2023-07-29T00:34:58.666 DEBUG:boto:Token: None 2023-07-29T00:34:58.666 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/ 2023-07-29T00:34:58.667 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw= 2023-07-29T00:34:58.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw=', 'Content-Length': '0'} 2023-07-29T00:34:58.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006954568d1aace00a-0064c45eb2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.671 DEBUG:boto:b'hiiqos-59obj331000falseobj332023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:58.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-07-29T00:34:58.672 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:34:58.673 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:34:58.673 DEBUG:boto:path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:34:58.673 DEBUG:boto:auth_path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:34:58.674 DEBUG:boto:Method: GET 2023-07-29T00:34:58.674 DEBUG:boto:Path: /hiiqos-59/obj34?versionId=null 2023-07-29T00:34:58.675 DEBUG:boto:Data: 2023-07-29T00:34:58.675 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.675 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:58.676 DEBUG:boto:Port: 8001 2023-07-29T00:34:58.676 DEBUG:boto:Params: {} 2023-07-29T00:34:58.676 DEBUG:boto:Token: None 2023-07-29T00:34:58.677 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj34?versionId=null 2023-07-29T00:34:58.677 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vwNE7eeT0+JyxfJtR9a8WsTT8fU= 2023-07-29T00:34:58.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vwNE7eeT0+JyxfJtR9a8WsTT8fU=', 'Content-Length': '0'} 2023-07-29T00:34:58.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000716312a879315ac1-0064c45eb2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.680 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:34:58.681 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:34:58.682 DEBUG:boto:path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:34:58.682 DEBUG:boto:auth_path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:34:58.682 DEBUG:boto:Method: GET 2023-07-29T00:34:58.683 DEBUG:boto:Path: /hiiqos-59/obj34?versionId=null 2023-07-29T00:34:58.683 DEBUG:boto:Data: 2023-07-29T00:34:58.683 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.684 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:58.684 DEBUG:boto:Port: 8000 2023-07-29T00:34:58.684 DEBUG:boto:Params: {} 2023-07-29T00:34:58.685 DEBUG:boto:Token: None 2023-07-29T00:34:58.685 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj34?versionId=null 2023-07-29T00:34:58.686 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vwNE7eeT0+JyxfJtR9a8WsTT8fU= 2023-07-29T00:34:58.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vwNE7eeT0+JyxfJtR9a8WsTT8fU=', 'Content-Length': '0'} 2023-07-29T00:34:58.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ef38ca91361c6e5-0064c45eb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.728 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:34:58.729 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:34:58.729 DEBUG:boto:path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:34:58.729 DEBUG:boto:auth_path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:34:58.730 DEBUG:boto:Method: HEAD 2023-07-29T00:34:58.730 DEBUG:boto:Path: /hiiqos-59/obj34?versionId=null 2023-07-29T00:34:58.730 DEBUG:boto:Data: 2023-07-29T00:34:58.731 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.731 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:58.731 DEBUG:boto:Port: 8001 2023-07-29T00:34:58.732 DEBUG:boto:Params: {} 2023-07-29T00:34:58.732 DEBUG:boto:Token: None 2023-07-29T00:34:58.733 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj34?versionId=null 2023-07-29T00:34:58.733 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Mit0heJ9xSTkuQ84+RCsRKxwcwY= 2023-07-29T00:34:58.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Mit0heJ9xSTkuQ84+RCsRKxwcwY=', 'Content-Length': '0'} 2023-07-29T00:34:58.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019d4744caeed6167-0064c45eb2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.736 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:34:58.736 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:34:58.737 DEBUG:boto:path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:34:58.737 DEBUG:boto:auth_path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:34:58.737 DEBUG:boto:Method: HEAD 2023-07-29T00:34:58.738 DEBUG:boto:Path: /hiiqos-59/obj34?versionId=null 2023-07-29T00:34:58.738 DEBUG:boto:Data: 2023-07-29T00:34:58.738 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.739 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:58.739 DEBUG:boto:Port: 8000 2023-07-29T00:34:58.739 DEBUG:boto:Params: {} 2023-07-29T00:34:58.740 DEBUG:boto:Token: None 2023-07-29T00:34:58.740 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj34?versionId=null 2023-07-29T00:34:58.741 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Mit0heJ9xSTkuQ84+RCsRKxwcwY= 2023-07-29T00:34:58.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Mit0heJ9xSTkuQ84+RCsRKxwcwY=', 'Content-Length': '0'} 2023-07-29T00:34:58.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d51c5c691d7fdc6f-0064c45eb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-07-29T00:34:58.744 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:34:58.744 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:34:58.744 DEBUG:boto:path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:34:58.745 DEBUG:boto:auth_path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:34:58.745 DEBUG:boto:Method: GET 2023-07-29T00:34:58.745 DEBUG:boto:Path: /hiiqos-59/obj34?versionId=null 2023-07-29T00:34:58.746 DEBUG:boto:Data: 2023-07-29T00:34:58.746 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.746 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:58.747 DEBUG:boto:Port: 8001 2023-07-29T00:34:58.747 DEBUG:boto:Params: {} 2023-07-29T00:34:58.748 DEBUG:boto:Token: None 2023-07-29T00:34:58.748 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj34?versionId=null 2023-07-29T00:34:58.748 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vwNE7eeT0+JyxfJtR9a8WsTT8fU= 2023-07-29T00:34:58.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vwNE7eeT0+JyxfJtR9a8WsTT8fU=', 'Content-Length': '0'} 2023-07-29T00:34:58.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e92f1eca8464475b-0064c45eb2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.751 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:34:58.752 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:34:58.752 DEBUG:boto:path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:34:58.752 DEBUG:boto:auth_path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:34:58.753 DEBUG:boto:Method: GET 2023-07-29T00:34:58.753 DEBUG:boto:Path: /hiiqos-59/obj34?versionId=null 2023-07-29T00:34:58.754 DEBUG:boto:Data: 2023-07-29T00:34:58.754 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.754 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:58.755 DEBUG:boto:Port: 8000 2023-07-29T00:34:58.755 DEBUG:boto:Params: {} 2023-07-29T00:34:58.755 DEBUG:boto:Token: None 2023-07-29T00:34:58.756 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj34?versionId=null 2023-07-29T00:34:58.756 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vwNE7eeT0+JyxfJtR9a8WsTT8fU= 2023-07-29T00:34:58.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vwNE7eeT0+JyxfJtR9a8WsTT8fU=', 'Content-Length': '0'} 2023-07-29T00:34:58.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b529e82ecfeff19a-0064c45eb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.759 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:58.760 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:58.760 DEBUG:boto:path=/hiiqos-59/?prefix=obj34 2023-07-29T00:34:58.761 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj34 2023-07-29T00:34:58.761 DEBUG:boto:Method: GET 2023-07-29T00:34:58.761 DEBUG:boto:Path: /hiiqos-59/?prefix=obj34 2023-07-29T00:34:58.762 DEBUG:boto:Data: 2023-07-29T00:34:58.762 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.762 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:58.763 DEBUG:boto:Port: 8001 2023-07-29T00:34:58.763 DEBUG:boto:Params: {} 2023-07-29T00:34:58.763 DEBUG:boto:Token: None 2023-07-29T00:34:58.764 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/ 2023-07-29T00:34:58.764 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw= 2023-07-29T00:34:58.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw=', 'Content-Length': '0'} 2023-07-29T00:34:58.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078095e517dfabb6d-0064c45eb2-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.768 DEBUG:boto:b'hiiqos-59obj341000falseobj342023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:58.769 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:58.769 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:58.769 DEBUG:boto:path=/hiiqos-59/?prefix=obj34 2023-07-29T00:34:58.770 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj34 2023-07-29T00:34:58.770 DEBUG:boto:Method: GET 2023-07-29T00:34:58.770 DEBUG:boto:Path: /hiiqos-59/?prefix=obj34 2023-07-29T00:34:58.771 DEBUG:boto:Data: 2023-07-29T00:34:58.771 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.771 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:58.772 DEBUG:boto:Port: 8000 2023-07-29T00:34:58.772 DEBUG:boto:Params: {} 2023-07-29T00:34:58.773 DEBUG:boto:Token: None 2023-07-29T00:34:58.773 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/ 2023-07-29T00:34:58.773 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw= 2023-07-29T00:34:58.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw=', 'Content-Length': '0'} 2023-07-29T00:34:58.777 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009401245cddf27d3d-0064c45eb2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.778 DEBUG:boto:b'hiiqos-59obj341000falseobj342023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:58.778 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:34:58.779 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:34:58.779 DEBUG:boto:Method: HEAD 2023-07-29T00:34:58.780 DEBUG:boto:Path: /hiiqos-59/obj34 2023-07-29T00:34:58.780 DEBUG:boto:Data: 2023-07-29T00:34:58.780 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.781 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:58.781 DEBUG:boto:Port: 8001 2023-07-29T00:34:58.782 DEBUG:boto:Params: {} 2023-07-29T00:34:58.782 DEBUG:boto:Token: None 2023-07-29T00:34:58.782 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj34 2023-07-29T00:34:58.783 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rokXNahehqyOzIJcN4rLGmiClzg= 2023-07-29T00:34:58.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rokXNahehqyOzIJcN4rLGmiClzg=', 'Content-Length': '0'} 2023-07-29T00:34:58.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cb9d286e4745c53-0064c45eb2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.785 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:34:58.786 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:34:58.786 DEBUG:boto:Method: HEAD 2023-07-29T00:34:58.787 DEBUG:boto:Path: /hiiqos-59/obj34 2023-07-29T00:34:58.787 DEBUG:boto:Data: 2023-07-29T00:34:58.787 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.788 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:58.788 DEBUG:boto:Port: 8000 2023-07-29T00:34:58.788 DEBUG:boto:Params: {} 2023-07-29T00:34:58.789 DEBUG:boto:Token: None 2023-07-29T00:34:58.789 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj34 2023-07-29T00:34:58.789 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rokXNahehqyOzIJcN4rLGmiClzg= 2023-07-29T00:34:58.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rokXNahehqyOzIJcN4rLGmiClzg=', 'Content-Length': '0'} 2023-07-29T00:34:58.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e53d34b495c6fbc-0064c45eb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-07-29T00:34:58.793 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:34:58.793 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:34:58.793 DEBUG:boto:Method: GET 2023-07-29T00:34:58.794 DEBUG:boto:Path: /hiiqos-59/obj34 2023-07-29T00:34:58.794 DEBUG:boto:Data: 2023-07-29T00:34:58.794 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.795 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:58.795 DEBUG:boto:Port: 8001 2023-07-29T00:34:58.795 DEBUG:boto:Params: {} 2023-07-29T00:34:58.796 DEBUG:boto:Token: None 2023-07-29T00:34:58.796 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj34 2023-07-29T00:34:58.797 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wWoaTZ4VRq39jEtLGDqzh4Bayhs= 2023-07-29T00:34:58.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wWoaTZ4VRq39jEtLGDqzh4Bayhs=', 'Content-Length': '0'} 2023-07-29T00:34:58.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e77f22c5f98f1ed3-0064c45eb2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.840 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:34:58.841 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:34:58.841 DEBUG:boto:Method: GET 2023-07-29T00:34:58.842 DEBUG:boto:Path: /hiiqos-59/obj34 2023-07-29T00:34:58.842 DEBUG:boto:Data: 2023-07-29T00:34:58.842 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.843 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:58.843 DEBUG:boto:Port: 8000 2023-07-29T00:34:58.843 DEBUG:boto:Params: {} 2023-07-29T00:34:58.844 DEBUG:boto:Token: None 2023-07-29T00:34:58.844 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj34 2023-07-29T00:34:58.845 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wWoaTZ4VRq39jEtLGDqzh4Bayhs= 2023-07-29T00:34:58.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wWoaTZ4VRq39jEtLGDqzh4Bayhs=', 'Content-Length': '0'} 2023-07-29T00:34:58.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd9ef2fea64d8a67-0064c45eb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.848 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:58.848 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:58.849 DEBUG:boto:path=/hiiqos-59/?prefix=obj34 2023-07-29T00:34:58.849 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj34 2023-07-29T00:34:58.849 DEBUG:boto:Method: GET 2023-07-29T00:34:58.850 DEBUG:boto:Path: /hiiqos-59/?prefix=obj34 2023-07-29T00:34:58.850 DEBUG:boto:Data: 2023-07-29T00:34:58.850 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.851 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:58.851 DEBUG:boto:Port: 8001 2023-07-29T00:34:58.852 DEBUG:boto:Params: {} 2023-07-29T00:34:58.852 DEBUG:boto:Token: None 2023-07-29T00:34:58.852 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/ 2023-07-29T00:34:58.853 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw= 2023-07-29T00:34:58.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw=', 'Content-Length': '0'} 2023-07-29T00:34:58.856 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002791a7b178eaf9b1-0064c45eb2-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.856 DEBUG:boto:b'hiiqos-59obj341000falseobj342023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:58.857 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:58.857 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:58.858 DEBUG:boto:path=/hiiqos-59/?prefix=obj34 2023-07-29T00:34:58.858 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj34 2023-07-29T00:34:58.858 DEBUG:boto:Method: GET 2023-07-29T00:34:58.859 DEBUG:boto:Path: /hiiqos-59/?prefix=obj34 2023-07-29T00:34:58.859 DEBUG:boto:Data: 2023-07-29T00:34:58.859 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.860 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:58.860 DEBUG:boto:Port: 8000 2023-07-29T00:34:58.861 DEBUG:boto:Params: {} 2023-07-29T00:34:58.861 DEBUG:boto:Token: None 2023-07-29T00:34:58.861 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/ 2023-07-29T00:34:58.862 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw= 2023-07-29T00:34:58.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw=', 'Content-Length': '0'} 2023-07-29T00:34:58.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003144e3547e3ea9c2-0064c45eb2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.866 DEBUG:boto:b'hiiqos-59obj341000falseobj342023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:58.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-07-29T00:34:58.867 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:34:58.867 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:34:58.868 DEBUG:boto:path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:34:58.868 DEBUG:boto:auth_path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:34:58.869 DEBUG:boto:Method: GET 2023-07-29T00:34:58.869 DEBUG:boto:Path: /hiiqos-59/obj38?versionId=null 2023-07-29T00:34:58.869 DEBUG:boto:Data: 2023-07-29T00:34:58.870 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.870 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:58.870 DEBUG:boto:Port: 8001 2023-07-29T00:34:58.871 DEBUG:boto:Params: {} 2023-07-29T00:34:58.871 DEBUG:boto:Token: None 2023-07-29T00:34:58.871 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj38?versionId=null 2023-07-29T00:34:58.872 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sYewPAq39F21E9esWP03HJGh+u0= 2023-07-29T00:34:58.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sYewPAq39F21E9esWP03HJGh+u0=', 'Content-Length': '0'} 2023-07-29T00:34:58.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040b1523570542b45-0064c45eb2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.875 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:34:58.875 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:34:58.876 DEBUG:boto:path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:34:58.876 DEBUG:boto:auth_path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:34:58.876 DEBUG:boto:Method: GET 2023-07-29T00:34:58.877 DEBUG:boto:Path: /hiiqos-59/obj38?versionId=null 2023-07-29T00:34:58.877 DEBUG:boto:Data: 2023-07-29T00:34:58.877 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.878 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:58.878 DEBUG:boto:Port: 8000 2023-07-29T00:34:58.879 DEBUG:boto:Params: {} 2023-07-29T00:34:58.879 DEBUG:boto:Token: None 2023-07-29T00:34:58.879 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj38?versionId=null 2023-07-29T00:34:58.880 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sYewPAq39F21E9esWP03HJGh+u0= 2023-07-29T00:34:58.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sYewPAq39F21E9esWP03HJGh+u0=', 'Content-Length': '0'} 2023-07-29T00:34:58.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc693902ee9a6607-0064c45eb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.924 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:34:58.925 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:34:58.925 DEBUG:boto:path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:34:58.926 DEBUG:boto:auth_path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:34:58.926 DEBUG:boto:Method: HEAD 2023-07-29T00:34:58.926 DEBUG:boto:Path: /hiiqos-59/obj38?versionId=null 2023-07-29T00:34:58.927 DEBUG:boto:Data: 2023-07-29T00:34:58.927 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.927 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:58.928 DEBUG:boto:Port: 8001 2023-07-29T00:34:58.928 DEBUG:boto:Params: {} 2023-07-29T00:34:58.928 DEBUG:boto:Token: None 2023-07-29T00:34:58.929 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj38?versionId=null 2023-07-29T00:34:58.929 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3IFDjfO0wLhqEMEHPd/m4Nhy/Pk= 2023-07-29T00:34:58.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3IFDjfO0wLhqEMEHPd/m4Nhy/Pk=', 'Content-Length': '0'} 2023-07-29T00:34:58.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae6453c5bfe107a8-0064c45eb2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.932 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:34:58.933 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:34:58.933 DEBUG:boto:path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:34:58.933 DEBUG:boto:auth_path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:34:58.934 DEBUG:boto:Method: HEAD 2023-07-29T00:34:58.934 DEBUG:boto:Path: /hiiqos-59/obj38?versionId=null 2023-07-29T00:34:58.934 DEBUG:boto:Data: 2023-07-29T00:34:58.935 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.935 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:58.935 DEBUG:boto:Port: 8000 2023-07-29T00:34:58.936 DEBUG:boto:Params: {} 2023-07-29T00:34:58.936 DEBUG:boto:Token: None 2023-07-29T00:34:58.937 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj38?versionId=null 2023-07-29T00:34:58.937 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3IFDjfO0wLhqEMEHPd/m4Nhy/Pk= 2023-07-29T00:34:58.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3IFDjfO0wLhqEMEHPd/m4Nhy/Pk=', 'Content-Length': '0'} 2023-07-29T00:34:58.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7a4e30055d55725-0064c45eb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-07-29T00:34:58.940 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:34:58.940 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:34:58.941 DEBUG:boto:path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:34:58.941 DEBUG:boto:auth_path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:34:58.942 DEBUG:boto:Method: GET 2023-07-29T00:34:58.942 DEBUG:boto:Path: /hiiqos-59/obj38?versionId=null 2023-07-29T00:34:58.942 DEBUG:boto:Data: 2023-07-29T00:34:58.943 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.943 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:58.943 DEBUG:boto:Port: 8001 2023-07-29T00:34:58.944 DEBUG:boto:Params: {} 2023-07-29T00:34:58.944 DEBUG:boto:Token: None 2023-07-29T00:34:58.944 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj38?versionId=null 2023-07-29T00:34:58.945 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sYewPAq39F21E9esWP03HJGh+u0= 2023-07-29T00:34:58.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sYewPAq39F21E9esWP03HJGh+u0=', 'Content-Length': '0'} 2023-07-29T00:34:58.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8a2cc00ad7645de-0064c45eb2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.948 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:34:58.948 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:34:58.949 DEBUG:boto:path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:34:58.949 DEBUG:boto:auth_path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:34:58.949 DEBUG:boto:Method: GET 2023-07-29T00:34:58.950 DEBUG:boto:Path: /hiiqos-59/obj38?versionId=null 2023-07-29T00:34:58.950 DEBUG:boto:Data: 2023-07-29T00:34:58.950 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.951 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:58.951 DEBUG:boto:Port: 8000 2023-07-29T00:34:58.951 DEBUG:boto:Params: {} 2023-07-29T00:34:58.952 DEBUG:boto:Token: None 2023-07-29T00:34:58.952 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj38?versionId=null 2023-07-29T00:34:58.953 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sYewPAq39F21E9esWP03HJGh+u0= 2023-07-29T00:34:58.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sYewPAq39F21E9esWP03HJGh+u0=', 'Content-Length': '0'} 2023-07-29T00:34:58.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b273b1b449637fe9-0064c45eb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.955 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:58.956 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:58.956 DEBUG:boto:path=/hiiqos-59/?prefix=obj38 2023-07-29T00:34:58.957 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj38 2023-07-29T00:34:58.957 DEBUG:boto:Method: GET 2023-07-29T00:34:58.957 DEBUG:boto:Path: /hiiqos-59/?prefix=obj38 2023-07-29T00:34:58.958 DEBUG:boto:Data: 2023-07-29T00:34:58.958 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.958 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:58.959 DEBUG:boto:Port: 8001 2023-07-29T00:34:58.959 DEBUG:boto:Params: {} 2023-07-29T00:34:58.960 DEBUG:boto:Token: None 2023-07-29T00:34:58.960 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/ 2023-07-29T00:34:58.960 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw= 2023-07-29T00:34:58.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw=', 'Content-Length': '0'} 2023-07-29T00:34:58.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082e10b477dd52af5-0064c45eb2-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.964 DEBUG:boto:b'hiiqos-59obj381000falseobj382023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:58.965 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:58.965 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:58.965 DEBUG:boto:path=/hiiqos-59/?prefix=obj38 2023-07-29T00:34:58.966 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj38 2023-07-29T00:34:58.966 DEBUG:boto:Method: GET 2023-07-29T00:34:58.966 DEBUG:boto:Path: /hiiqos-59/?prefix=obj38 2023-07-29T00:34:58.967 DEBUG:boto:Data: 2023-07-29T00:34:58.967 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.968 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:58.968 DEBUG:boto:Port: 8000 2023-07-29T00:34:58.968 DEBUG:boto:Params: {} 2023-07-29T00:34:58.969 DEBUG:boto:Token: None 2023-07-29T00:34:58.969 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/ 2023-07-29T00:34:58.970 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw= 2023-07-29T00:34:58.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:x79zSh8fHfrIiUwgXcZmBtMkmJw=', 'Content-Length': '0'} 2023-07-29T00:34:58.973 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bce22a785531fae-0064c45eb2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.974 DEBUG:boto:b'hiiqos-59obj381000falseobj382023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:58.975 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:34:58.975 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:34:58.975 DEBUG:boto:Method: HEAD 2023-07-29T00:34:58.976 DEBUG:boto:Path: /hiiqos-59/obj38 2023-07-29T00:34:58.976 DEBUG:boto:Data: 2023-07-29T00:34:58.976 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.977 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:58.977 DEBUG:boto:Port: 8001 2023-07-29T00:34:58.977 DEBUG:boto:Params: {} 2023-07-29T00:34:58.978 DEBUG:boto:Token: None 2023-07-29T00:34:58.978 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj38 2023-07-29T00:34:58.979 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SziYrhsCsMicpNzXFczfCc+zJ/o= 2023-07-29T00:34:58.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SziYrhsCsMicpNzXFczfCc+zJ/o=', 'Content-Length': '0'} 2023-07-29T00:34:58.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060ad99e747f15d2e-0064c45eb2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.981 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:34:58.982 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:34:58.982 DEBUG:boto:Method: HEAD 2023-07-29T00:34:58.982 DEBUG:boto:Path: /hiiqos-59/obj38 2023-07-29T00:34:58.983 DEBUG:boto:Data: 2023-07-29T00:34:58.983 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.983 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:58.984 DEBUG:boto:Port: 8000 2023-07-29T00:34:58.984 DEBUG:boto:Params: {} 2023-07-29T00:34:58.985 DEBUG:boto:Token: None 2023-07-29T00:34:58.985 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj38 2023-07-29T00:34:58.985 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SziYrhsCsMicpNzXFczfCc+zJ/o= 2023-07-29T00:34:58.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SziYrhsCsMicpNzXFczfCc+zJ/o=', 'Content-Length': '0'} 2023-07-29T00:34:58.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cde027af3741500-0064c45eb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-07-29T00:34:58.989 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:34:58.989 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:34:58.989 DEBUG:boto:Method: GET 2023-07-29T00:34:58.990 DEBUG:boto:Path: /hiiqos-59/obj38 2023-07-29T00:34:58.990 DEBUG:boto:Data: 2023-07-29T00:34:58.990 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.991 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:58.991 DEBUG:boto:Port: 8001 2023-07-29T00:34:58.991 DEBUG:boto:Params: {} 2023-07-29T00:34:58.992 DEBUG:boto:Token: None 2023-07-29T00:34:58.992 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj38 2023-07-29T00:34:58.993 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:y76cPtltWhwlcmSTntMZRJPSRek= 2023-07-29T00:34:58.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:y76cPtltWhwlcmSTntMZRJPSRek=', 'Content-Length': '0'} 2023-07-29T00:34:58.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ac81f23e5ca3297-0064c45eb2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:58.996 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:34:58.996 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:34:58.996 DEBUG:boto:Method: GET 2023-07-29T00:34:58.997 DEBUG:boto:Path: /hiiqos-59/obj38 2023-07-29T00:34:58.997 DEBUG:boto:Data: 2023-07-29T00:34:58.998 DEBUG:boto:Headers: {} 2023-07-29T00:34:58.998 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:58.998 DEBUG:boto:Port: 8000 2023-07-29T00:34:58.999 DEBUG:boto:Params: {} 2023-07-29T00:34:58.999 DEBUG:boto:Token: None 2023-07-29T00:34:58.999 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:58 GMT /hiiqos-59/obj38 2023-07-29T00:34:59.000 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:y76cPtltWhwlcmSTntMZRJPSRek= 2023-07-29T00:34:59.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:y76cPtltWhwlcmSTntMZRJPSRek=', 'Content-Length': '0'} 2023-07-29T00:34:59.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000731534e71e353e0f-0064c45eb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.003 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:59.003 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:59.004 DEBUG:boto:path=/hiiqos-59/?prefix=obj38 2023-07-29T00:34:59.004 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj38 2023-07-29T00:34:59.004 DEBUG:boto:Method: GET 2023-07-29T00:34:59.005 DEBUG:boto:Path: /hiiqos-59/?prefix=obj38 2023-07-29T00:34:59.005 DEBUG:boto:Data: 2023-07-29T00:34:59.006 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.006 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:59.006 DEBUG:boto:Port: 8001 2023-07-29T00:34:59.007 DEBUG:boto:Params: {} 2023-07-29T00:34:59.007 DEBUG:boto:Token: None 2023-07-29T00:34:59.007 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/ 2023-07-29T00:34:59.008 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SnG8lOSwilMTlkBxsp4+W6VQZMA= 2023-07-29T00:34:59.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SnG8lOSwilMTlkBxsp4+W6VQZMA=', 'Content-Length': '0'} 2023-07-29T00:34:59.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5776f59528467d2-0064c45eb2-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.011 DEBUG:boto:b'hiiqos-59obj381000falseobj382023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:59.012 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:59.012 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:59.013 DEBUG:boto:path=/hiiqos-59/?prefix=obj38 2023-07-29T00:34:59.013 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj38 2023-07-29T00:34:59.013 DEBUG:boto:Method: GET 2023-07-29T00:34:59.014 DEBUG:boto:Path: /hiiqos-59/?prefix=obj38 2023-07-29T00:34:59.014 DEBUG:boto:Data: 2023-07-29T00:34:59.014 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.015 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:59.015 DEBUG:boto:Port: 8000 2023-07-29T00:34:59.016 DEBUG:boto:Params: {} 2023-07-29T00:34:59.016 DEBUG:boto:Token: None 2023-07-29T00:34:59.016 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/ 2023-07-29T00:34:59.017 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SnG8lOSwilMTlkBxsp4+W6VQZMA= 2023-07-29T00:34:59.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SnG8lOSwilMTlkBxsp4+W6VQZMA=', 'Content-Length': '0'} 2023-07-29T00:34:59.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8ad132652d6b530-0064c45eb2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.021 DEBUG:boto:b'hiiqos-59obj381000falseobj382023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:59.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-07-29T00:34:59.022 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:34:59.022 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:34:59.023 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:34:59.023 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:34:59.023 DEBUG:boto:Method: GET 2023-07-29T00:34:59.024 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:34:59.024 DEBUG:boto:Data: 2023-07-29T00:34:59.025 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.025 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:59.025 DEBUG:boto:Port: 8001 2023-07-29T00:34:59.026 DEBUG:boto:Params: {} 2023-07-29T00:34:59.026 DEBUG:boto:Token: None 2023-07-29T00:34:59.027 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:34:59.027 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XwyglEqviyFtbDAPbwMbEjrJE/4= 2023-07-29T00:34:59.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XwyglEqviyFtbDAPbwMbEjrJE/4=', 'Content-Length': '0'} 2023-07-29T00:34:59.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e25c9ec25406e466-0064c45eb2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.030 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:34:59.031 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:34:59.031 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:34:59.031 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:34:59.032 DEBUG:boto:Method: GET 2023-07-29T00:34:59.032 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:34:59.033 DEBUG:boto:Data: 2023-07-29T00:34:59.033 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.033 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:59.034 DEBUG:boto:Port: 8000 2023-07-29T00:34:59.034 DEBUG:boto:Params: {} 2023-07-29T00:34:59.034 DEBUG:boto:Token: None 2023-07-29T00:34:59.035 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:34:59.035 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XwyglEqviyFtbDAPbwMbEjrJE/4= 2023-07-29T00:34:59.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XwyglEqviyFtbDAPbwMbEjrJE/4=', 'Content-Length': '0'} 2023-07-29T00:34:59.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f57d95de3a0408f-0064c45eb2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.080 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:34:59.081 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:34:59.081 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:34:59.082 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:34:59.082 DEBUG:boto:Method: HEAD 2023-07-29T00:34:59.083 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:34:59.083 DEBUG:boto:Data: 2023-07-29T00:34:59.084 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.084 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:59.085 DEBUG:boto:Port: 8001 2023-07-29T00:34:59.085 DEBUG:boto:Params: {} 2023-07-29T00:34:59.086 DEBUG:boto:Token: None 2023-07-29T00:34:59.086 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:34:59.087 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zHgJlLKhwXOlwRtQ8Gc6j0pUY+M= 2023-07-29T00:34:59.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zHgJlLKhwXOlwRtQ8Gc6j0pUY+M=', 'Content-Length': '0'} 2023-07-29T00:34:59.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003753bf37d90faa3c-0064c45eb3-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.091 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:34:59.091 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:34:59.092 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:34:59.092 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:34:59.093 DEBUG:boto:Method: HEAD 2023-07-29T00:34:59.093 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:34:59.094 DEBUG:boto:Data: 2023-07-29T00:34:59.094 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.095 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:59.095 DEBUG:boto:Port: 8000 2023-07-29T00:34:59.096 DEBUG:boto:Params: {} 2023-07-29T00:34:59.096 DEBUG:boto:Token: None 2023-07-29T00:34:59.097 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:34:59.097 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zHgJlLKhwXOlwRtQ8Gc6j0pUY+M= 2023-07-29T00:34:59.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zHgJlLKhwXOlwRtQ8Gc6j0pUY+M=', 'Content-Length': '0'} 2023-07-29T00:34:59.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0bf3d15138cbd44-0064c45eb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-07-29T00:34:59.101 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:34:59.101 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:34:59.102 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:34:59.102 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:34:59.103 DEBUG:boto:Method: GET 2023-07-29T00:34:59.103 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:34:59.103 DEBUG:boto:Data: 2023-07-29T00:34:59.104 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.104 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:59.105 DEBUG:boto:Port: 8001 2023-07-29T00:34:59.105 DEBUG:boto:Params: {} 2023-07-29T00:34:59.105 DEBUG:boto:Token: None 2023-07-29T00:34:59.106 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:34:59.106 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XwyglEqviyFtbDAPbwMbEjrJE/4= 2023-07-29T00:34:59.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XwyglEqviyFtbDAPbwMbEjrJE/4=', 'Content-Length': '0'} 2023-07-29T00:34:59.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008db529eaaad37a2e-0064c45eb3-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.109 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:34:59.110 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:34:59.110 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:34:59.110 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:34:59.111 DEBUG:boto:Method: GET 2023-07-29T00:34:59.111 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:34:59.112 DEBUG:boto:Data: 2023-07-29T00:34:59.112 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.112 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:59.113 DEBUG:boto:Port: 8000 2023-07-29T00:34:59.113 DEBUG:boto:Params: {} 2023-07-29T00:34:59.113 DEBUG:boto:Token: None 2023-07-29T00:34:59.114 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:34:59.114 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XwyglEqviyFtbDAPbwMbEjrJE/4= 2023-07-29T00:34:59.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XwyglEqviyFtbDAPbwMbEjrJE/4=', 'Content-Length': '0'} 2023-07-29T00:34:59.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005804d37a699e31ca-0064c45eb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.117 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:59.118 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:59.118 DEBUG:boto:path=/hiiqos-59/?prefix=obj44 2023-07-29T00:34:59.118 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj44 2023-07-29T00:34:59.119 DEBUG:boto:Method: GET 2023-07-29T00:34:59.119 DEBUG:boto:Path: /hiiqos-59/?prefix=obj44 2023-07-29T00:34:59.119 DEBUG:boto:Data: 2023-07-29T00:34:59.120 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.120 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:59.121 DEBUG:boto:Port: 8001 2023-07-29T00:34:59.121 DEBUG:boto:Params: {} 2023-07-29T00:34:59.122 DEBUG:boto:Token: None 2023-07-29T00:34:59.122 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/ 2023-07-29T00:34:59.122 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SnG8lOSwilMTlkBxsp4+W6VQZMA= 2023-07-29T00:34:59.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SnG8lOSwilMTlkBxsp4+W6VQZMA=', 'Content-Length': '0'} 2023-07-29T00:34:59.125 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014d99bae6191a2fa-0064c45eb3-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.172 DEBUG:boto:b'hiiqos-59obj441000falseobj442023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:59.173 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:59.174 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:59.175 DEBUG:boto:path=/hiiqos-59/?prefix=obj44 2023-07-29T00:34:59.176 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj44 2023-07-29T00:34:59.176 DEBUG:boto:Method: GET 2023-07-29T00:34:59.177 DEBUG:boto:Path: /hiiqos-59/?prefix=obj44 2023-07-29T00:34:59.177 DEBUG:boto:Data: 2023-07-29T00:34:59.177 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.178 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:59.178 DEBUG:boto:Port: 8000 2023-07-29T00:34:59.178 DEBUG:boto:Params: {} 2023-07-29T00:34:59.179 DEBUG:boto:Token: None 2023-07-29T00:34:59.179 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/ 2023-07-29T00:34:59.180 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SnG8lOSwilMTlkBxsp4+W6VQZMA= 2023-07-29T00:34:59.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SnG8lOSwilMTlkBxsp4+W6VQZMA=', 'Content-Length': '0'} 2023-07-29T00:34:59.184 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfd5e0078579fe57-0064c45eb3-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.184 DEBUG:boto:b'hiiqos-59obj441000falseobj442023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:59.185 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:34:59.185 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:34:59.186 DEBUG:boto:Method: HEAD 2023-07-29T00:34:59.186 DEBUG:boto:Path: /hiiqos-59/obj44 2023-07-29T00:34:59.187 DEBUG:boto:Data: 2023-07-29T00:34:59.187 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.187 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:59.188 DEBUG:boto:Port: 8001 2023-07-29T00:34:59.188 DEBUG:boto:Params: {} 2023-07-29T00:34:59.189 DEBUG:boto:Token: None 2023-07-29T00:34:59.189 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj44 2023-07-29T00:34:59.190 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FYmOJQi2Ug+G/Gh1j8UorchwmN0= 2023-07-29T00:34:59.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FYmOJQi2Ug+G/Gh1j8UorchwmN0=', 'Content-Length': '0'} 2023-07-29T00:34:59.192 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1d5873b8bef3ce3-0064c45eb3-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.193 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:34:59.194 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:34:59.194 DEBUG:boto:Method: HEAD 2023-07-29T00:34:59.194 DEBUG:boto:Path: /hiiqos-59/obj44 2023-07-29T00:34:59.195 DEBUG:boto:Data: 2023-07-29T00:34:59.195 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.196 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:59.196 DEBUG:boto:Port: 8000 2023-07-29T00:34:59.197 DEBUG:boto:Params: {} 2023-07-29T00:34:59.197 DEBUG:boto:Token: None 2023-07-29T00:34:59.197 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj44 2023-07-29T00:34:59.198 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FYmOJQi2Ug+G/Gh1j8UorchwmN0= 2023-07-29T00:34:59.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FYmOJQi2Ug+G/Gh1j8UorchwmN0=', 'Content-Length': '0'} 2023-07-29T00:34:59.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009caae8c4be868430-0064c45eb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-07-29T00:34:59.201 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:34:59.201 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:34:59.202 DEBUG:boto:Method: GET 2023-07-29T00:34:59.202 DEBUG:boto:Path: /hiiqos-59/obj44 2023-07-29T00:34:59.202 DEBUG:boto:Data: 2023-07-29T00:34:59.203 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.203 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:59.203 DEBUG:boto:Port: 8001 2023-07-29T00:34:59.204 DEBUG:boto:Params: {} 2023-07-29T00:34:59.204 DEBUG:boto:Token: None 2023-07-29T00:34:59.205 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj44 2023-07-29T00:34:59.205 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CddvFod84WHjVmBTFbu37UPUE5A= 2023-07-29T00:34:59.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CddvFod84WHjVmBTFbu37UPUE5A=', 'Content-Length': '0'} 2023-07-29T00:34:59.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003248b36f3cd6a39b-0064c45eb3-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.208 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:34:59.209 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:34:59.209 DEBUG:boto:Method: GET 2023-07-29T00:34:59.210 DEBUG:boto:Path: /hiiqos-59/obj44 2023-07-29T00:34:59.210 DEBUG:boto:Data: 2023-07-29T00:34:59.210 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.211 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:59.211 DEBUG:boto:Port: 8000 2023-07-29T00:34:59.212 DEBUG:boto:Params: {} 2023-07-29T00:34:59.212 DEBUG:boto:Token: None 2023-07-29T00:34:59.213 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj44 2023-07-29T00:34:59.213 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CddvFod84WHjVmBTFbu37UPUE5A= 2023-07-29T00:34:59.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CddvFod84WHjVmBTFbu37UPUE5A=', 'Content-Length': '0'} 2023-07-29T00:34:59.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff4ac5818eb27594-0064c45eb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.216 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:59.217 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:59.217 DEBUG:boto:path=/hiiqos-59/?prefix=obj44 2023-07-29T00:34:59.218 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj44 2023-07-29T00:34:59.218 DEBUG:boto:Method: GET 2023-07-29T00:34:59.218 DEBUG:boto:Path: /hiiqos-59/?prefix=obj44 2023-07-29T00:34:59.219 DEBUG:boto:Data: 2023-07-29T00:34:59.219 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.219 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:59.220 DEBUG:boto:Port: 8001 2023-07-29T00:34:59.220 DEBUG:boto:Params: {} 2023-07-29T00:34:59.221 DEBUG:boto:Token: None 2023-07-29T00:34:59.221 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/ 2023-07-29T00:34:59.222 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SnG8lOSwilMTlkBxsp4+W6VQZMA= 2023-07-29T00:34:59.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SnG8lOSwilMTlkBxsp4+W6VQZMA=', 'Content-Length': '0'} 2023-07-29T00:34:59.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000296271b956cb1f-0064c45eb3-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.225 DEBUG:boto:b'hiiqos-59obj441000falseobj442023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:59.225 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:59.226 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:59.226 DEBUG:boto:path=/hiiqos-59/?prefix=obj44 2023-07-29T00:34:59.227 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj44 2023-07-29T00:34:59.227 DEBUG:boto:Method: GET 2023-07-29T00:34:59.228 DEBUG:boto:Path: /hiiqos-59/?prefix=obj44 2023-07-29T00:34:59.228 DEBUG:boto:Data: 2023-07-29T00:34:59.228 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.229 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:59.229 DEBUG:boto:Port: 8000 2023-07-29T00:34:59.229 DEBUG:boto:Params: {} 2023-07-29T00:34:59.230 DEBUG:boto:Token: None 2023-07-29T00:34:59.230 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/ 2023-07-29T00:34:59.231 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SnG8lOSwilMTlkBxsp4+W6VQZMA= 2023-07-29T00:34:59.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SnG8lOSwilMTlkBxsp4+W6VQZMA=', 'Content-Length': '0'} 2023-07-29T00:34:59.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c8825b3aaa6c36b-0064c45eb3-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.235 DEBUG:boto:b'hiiqos-59obj441000falseobj442023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:59.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-07-29T00:34:59.236 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:34:59.236 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:34:59.237 DEBUG:boto:path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:34:59.237 DEBUG:boto:auth_path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:34:59.237 DEBUG:boto:Method: GET 2023-07-29T00:34:59.238 DEBUG:boto:Path: /hiiqos-59/obj46?versionId=null 2023-07-29T00:34:59.238 DEBUG:boto:Data: 2023-07-29T00:34:59.239 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.239 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:59.239 DEBUG:boto:Port: 8001 2023-07-29T00:34:59.240 DEBUG:boto:Params: {} 2023-07-29T00:34:59.240 DEBUG:boto:Token: None 2023-07-29T00:34:59.241 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj46?versionId=null 2023-07-29T00:34:59.241 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5ck7MXNbMQOknGCTvM/+zzTHPrk= 2023-07-29T00:34:59.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5ck7MXNbMQOknGCTvM/+zzTHPrk=', 'Content-Length': '0'} 2023-07-29T00:34:59.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ae425a9853b09ea-0064c45eb3-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.244 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:34:59.245 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:34:59.245 DEBUG:boto:path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:34:59.245 DEBUG:boto:auth_path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:34:59.246 DEBUG:boto:Method: GET 2023-07-29T00:34:59.246 DEBUG:boto:Path: /hiiqos-59/obj46?versionId=null 2023-07-29T00:34:59.247 DEBUG:boto:Data: 2023-07-29T00:34:59.247 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.247 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:59.248 DEBUG:boto:Port: 8000 2023-07-29T00:34:59.248 DEBUG:boto:Params: {} 2023-07-29T00:34:59.249 DEBUG:boto:Token: None 2023-07-29T00:34:59.249 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj46?versionId=null 2023-07-29T00:34:59.250 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5ck7MXNbMQOknGCTvM/+zzTHPrk= 2023-07-29T00:34:59.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5ck7MXNbMQOknGCTvM/+zzTHPrk=', 'Content-Length': '0'} 2023-07-29T00:34:59.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a313bb743ca33ffe-0064c45eb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.252 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:34:59.253 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:34:59.253 DEBUG:boto:path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:34:59.254 DEBUG:boto:auth_path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:34:59.254 DEBUG:boto:Method: HEAD 2023-07-29T00:34:59.255 DEBUG:boto:Path: /hiiqos-59/obj46?versionId=null 2023-07-29T00:34:59.255 DEBUG:boto:Data: 2023-07-29T00:34:59.255 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.256 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:59.256 DEBUG:boto:Port: 8001 2023-07-29T00:34:59.256 DEBUG:boto:Params: {} 2023-07-29T00:34:59.257 DEBUG:boto:Token: None 2023-07-29T00:34:59.257 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj46?versionId=null 2023-07-29T00:34:59.258 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:W5BCuG5+LH5CRgHs23n8tkYM3Yk= 2023-07-29T00:34:59.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:W5BCuG5+LH5CRgHs23n8tkYM3Yk=', 'Content-Length': '0'} 2023-07-29T00:34:59.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac3e97bad1bf4aec-0064c45eb3-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.261 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:34:59.261 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:34:59.261 DEBUG:boto:path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:34:59.262 DEBUG:boto:auth_path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:34:59.262 DEBUG:boto:Method: HEAD 2023-07-29T00:34:59.262 DEBUG:boto:Path: /hiiqos-59/obj46?versionId=null 2023-07-29T00:34:59.263 DEBUG:boto:Data: 2023-07-29T00:34:59.263 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.263 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:59.264 DEBUG:boto:Port: 8000 2023-07-29T00:34:59.264 DEBUG:boto:Params: {} 2023-07-29T00:34:59.264 DEBUG:boto:Token: None 2023-07-29T00:34:59.265 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj46?versionId=null 2023-07-29T00:34:59.265 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:W5BCuG5+LH5CRgHs23n8tkYM3Yk= 2023-07-29T00:34:59.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:W5BCuG5+LH5CRgHs23n8tkYM3Yk=', 'Content-Length': '0'} 2023-07-29T00:34:59.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001af19bd0ec935ab3-0064c45eb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-07-29T00:34:59.269 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:34:59.269 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:34:59.270 DEBUG:boto:path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:34:59.270 DEBUG:boto:auth_path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:34:59.270 DEBUG:boto:Method: GET 2023-07-29T00:34:59.271 DEBUG:boto:Path: /hiiqos-59/obj46?versionId=null 2023-07-29T00:34:59.271 DEBUG:boto:Data: 2023-07-29T00:34:59.271 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.272 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:59.272 DEBUG:boto:Port: 8001 2023-07-29T00:34:59.272 DEBUG:boto:Params: {} 2023-07-29T00:34:59.273 DEBUG:boto:Token: None 2023-07-29T00:34:59.273 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj46?versionId=null 2023-07-29T00:34:59.274 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5ck7MXNbMQOknGCTvM/+zzTHPrk= 2023-07-29T00:34:59.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5ck7MXNbMQOknGCTvM/+zzTHPrk=', 'Content-Length': '0'} 2023-07-29T00:34:59.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c86d2539499c4b8c-0064c45eb3-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.277 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:34:59.277 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:34:59.277 DEBUG:boto:path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:34:59.278 DEBUG:boto:auth_path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:34:59.278 DEBUG:boto:Method: GET 2023-07-29T00:34:59.278 DEBUG:boto:Path: /hiiqos-59/obj46?versionId=null 2023-07-29T00:34:59.279 DEBUG:boto:Data: 2023-07-29T00:34:59.279 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.280 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:59.280 DEBUG:boto:Port: 8000 2023-07-29T00:34:59.280 DEBUG:boto:Params: {} 2023-07-29T00:34:59.281 DEBUG:boto:Token: None 2023-07-29T00:34:59.281 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj46?versionId=null 2023-07-29T00:34:59.282 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5ck7MXNbMQOknGCTvM/+zzTHPrk= 2023-07-29T00:34:59.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5ck7MXNbMQOknGCTvM/+zzTHPrk=', 'Content-Length': '0'} 2023-07-29T00:34:59.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6bc261b99e6e5ab-0064c45eb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.285 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:59.285 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:59.286 DEBUG:boto:path=/hiiqos-59/?prefix=obj46 2023-07-29T00:34:59.286 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj46 2023-07-29T00:34:59.286 DEBUG:boto:Method: GET 2023-07-29T00:34:59.287 DEBUG:boto:Path: /hiiqos-59/?prefix=obj46 2023-07-29T00:34:59.287 DEBUG:boto:Data: 2023-07-29T00:34:59.288 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.288 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:59.288 DEBUG:boto:Port: 8001 2023-07-29T00:34:59.289 DEBUG:boto:Params: {} 2023-07-29T00:34:59.289 DEBUG:boto:Token: None 2023-07-29T00:34:59.290 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/ 2023-07-29T00:34:59.290 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SnG8lOSwilMTlkBxsp4+W6VQZMA= 2023-07-29T00:34:59.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SnG8lOSwilMTlkBxsp4+W6VQZMA=', 'Content-Length': '0'} 2023-07-29T00:34:59.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d578b8d47b7d9661-0064c45eb3-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.336 DEBUG:boto:b'hiiqos-59obj461000falseobj462023-07-29T00:33:06.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:59.337 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:59.338 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:59.338 DEBUG:boto:path=/hiiqos-59/?prefix=obj46 2023-07-29T00:34:59.338 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj46 2023-07-29T00:34:59.339 DEBUG:boto:Method: GET 2023-07-29T00:34:59.339 DEBUG:boto:Path: /hiiqos-59/?prefix=obj46 2023-07-29T00:34:59.339 DEBUG:boto:Data: 2023-07-29T00:34:59.340 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.340 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:59.340 DEBUG:boto:Port: 8000 2023-07-29T00:34:59.341 DEBUG:boto:Params: {} 2023-07-29T00:34:59.341 DEBUG:boto:Token: None 2023-07-29T00:34:59.342 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/ 2023-07-29T00:34:59.342 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SnG8lOSwilMTlkBxsp4+W6VQZMA= 2023-07-29T00:34:59.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SnG8lOSwilMTlkBxsp4+W6VQZMA=', 'Content-Length': '0'} 2023-07-29T00:34:59.347 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000034dd856ca35118a-0064c45eb3-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.347 DEBUG:boto:b'hiiqos-59obj461000falseobj462023-07-29T00:33:06.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:59.348 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:34:59.348 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:34:59.349 DEBUG:boto:Method: HEAD 2023-07-29T00:34:59.349 DEBUG:boto:Path: /hiiqos-59/obj46 2023-07-29T00:34:59.349 DEBUG:boto:Data: 2023-07-29T00:34:59.350 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.350 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:59.350 DEBUG:boto:Port: 8001 2023-07-29T00:34:59.351 DEBUG:boto:Params: {} 2023-07-29T00:34:59.351 DEBUG:boto:Token: None 2023-07-29T00:34:59.352 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj46 2023-07-29T00:34:59.352 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PiQkb6u7HnIkimSd482fdCjcO4Q= 2023-07-29T00:34:59.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PiQkb6u7HnIkimSd482fdCjcO4Q=', 'Content-Length': '0'} 2023-07-29T00:34:59.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f7dd4f79e5aea0d-0064c45eb3-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.355 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:34:59.355 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:34:59.356 DEBUG:boto:Method: HEAD 2023-07-29T00:34:59.356 DEBUG:boto:Path: /hiiqos-59/obj46 2023-07-29T00:34:59.356 DEBUG:boto:Data: 2023-07-29T00:34:59.357 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.357 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:59.357 DEBUG:boto:Port: 8000 2023-07-29T00:34:59.358 DEBUG:boto:Params: {} 2023-07-29T00:34:59.358 DEBUG:boto:Token: None 2023-07-29T00:34:59.358 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj46 2023-07-29T00:34:59.359 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PiQkb6u7HnIkimSd482fdCjcO4Q= 2023-07-29T00:34:59.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PiQkb6u7HnIkimSd482fdCjcO4Q=', 'Content-Length': '0'} 2023-07-29T00:34:59.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004dc412084fe8e652-0064c45eb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-07-29T00:34:59.362 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:34:59.363 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:34:59.363 DEBUG:boto:Method: GET 2023-07-29T00:34:59.363 DEBUG:boto:Path: /hiiqos-59/obj46 2023-07-29T00:34:59.364 DEBUG:boto:Data: 2023-07-29T00:34:59.364 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.365 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:59.365 DEBUG:boto:Port: 8001 2023-07-29T00:34:59.365 DEBUG:boto:Params: {} 2023-07-29T00:34:59.366 DEBUG:boto:Token: None 2023-07-29T00:34:59.366 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj46 2023-07-29T00:34:59.367 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:biweoRvQBU+Ve8XsQCaVoEMkz3A= 2023-07-29T00:34:59.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:biweoRvQBU+Ve8XsQCaVoEMkz3A=', 'Content-Length': '0'} 2023-07-29T00:34:59.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000222f68f8246be60c-0064c45eb3-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.370 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:34:59.371 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:34:59.371 DEBUG:boto:Method: GET 2023-07-29T00:34:59.372 DEBUG:boto:Path: /hiiqos-59/obj46 2023-07-29T00:34:59.372 DEBUG:boto:Data: 2023-07-29T00:34:59.372 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.373 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:59.373 DEBUG:boto:Port: 8000 2023-07-29T00:34:59.373 DEBUG:boto:Params: {} 2023-07-29T00:34:59.374 DEBUG:boto:Token: None 2023-07-29T00:34:59.374 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj46 2023-07-29T00:34:59.375 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:biweoRvQBU+Ve8XsQCaVoEMkz3A= 2023-07-29T00:34:59.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:biweoRvQBU+Ve8XsQCaVoEMkz3A=', 'Content-Length': '0'} 2023-07-29T00:34:59.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efd47da90fbd95fd-0064c45eb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.420 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:59.421 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:59.421 DEBUG:boto:path=/hiiqos-59/?prefix=obj46 2023-07-29T00:34:59.422 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj46 2023-07-29T00:34:59.422 DEBUG:boto:Method: GET 2023-07-29T00:34:59.422 DEBUG:boto:Path: /hiiqos-59/?prefix=obj46 2023-07-29T00:34:59.423 DEBUG:boto:Data: 2023-07-29T00:34:59.423 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.423 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:59.424 DEBUG:boto:Port: 8001 2023-07-29T00:34:59.424 DEBUG:boto:Params: {} 2023-07-29T00:34:59.425 DEBUG:boto:Token: None 2023-07-29T00:34:59.425 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/ 2023-07-29T00:34:59.425 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SnG8lOSwilMTlkBxsp4+W6VQZMA= 2023-07-29T00:34:59.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SnG8lOSwilMTlkBxsp4+W6VQZMA=', 'Content-Length': '0'} 2023-07-29T00:34:59.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d531b1ce0e809e3-0064c45eb3-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.429 DEBUG:boto:b'hiiqos-59obj461000falseobj462023-07-29T00:33:06.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:59.429 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:59.430 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:59.430 DEBUG:boto:path=/hiiqos-59/?prefix=obj46 2023-07-29T00:34:59.431 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj46 2023-07-29T00:34:59.431 DEBUG:boto:Method: GET 2023-07-29T00:34:59.431 DEBUG:boto:Path: /hiiqos-59/?prefix=obj46 2023-07-29T00:34:59.432 DEBUG:boto:Data: 2023-07-29T00:34:59.432 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.432 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:59.433 DEBUG:boto:Port: 8000 2023-07-29T00:34:59.433 DEBUG:boto:Params: {} 2023-07-29T00:34:59.433 DEBUG:boto:Token: None 2023-07-29T00:34:59.434 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/ 2023-07-29T00:34:59.434 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SnG8lOSwilMTlkBxsp4+W6VQZMA= 2023-07-29T00:34:59.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SnG8lOSwilMTlkBxsp4+W6VQZMA=', 'Content-Length': '0'} 2023-07-29T00:34:59.438 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee2a1f8c6655ca8b-0064c45eb3-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.439 DEBUG:boto:b'hiiqos-59obj461000falseobj462023-07-29T00:33:06.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:59.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-07-29T00:34:59.440 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:34:59.440 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:34:59.441 DEBUG:boto:path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:34:59.441 DEBUG:boto:auth_path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:34:59.441 DEBUG:boto:Method: GET 2023-07-29T00:34:59.442 DEBUG:boto:Path: /hiiqos-59/obj51?versionId=null 2023-07-29T00:34:59.442 DEBUG:boto:Data: 2023-07-29T00:34:59.442 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.443 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:59.443 DEBUG:boto:Port: 8001 2023-07-29T00:34:59.443 DEBUG:boto:Params: {} 2023-07-29T00:34:59.444 DEBUG:boto:Token: None 2023-07-29T00:34:59.444 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj51?versionId=null 2023-07-29T00:34:59.445 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Jix5eocE4wwY11abO1EKcHr/7fc= 2023-07-29T00:34:59.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Jix5eocE4wwY11abO1EKcHr/7fc=', 'Content-Length': '0'} 2023-07-29T00:34:59.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048ba609a65504d5b-0064c45eb3-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.448 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:34:59.448 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:34:59.448 DEBUG:boto:path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:34:59.449 DEBUG:boto:auth_path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:34:59.449 DEBUG:boto:Method: GET 2023-07-29T00:34:59.450 DEBUG:boto:Path: /hiiqos-59/obj51?versionId=null 2023-07-29T00:34:59.450 DEBUG:boto:Data: 2023-07-29T00:34:59.450 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.451 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:59.451 DEBUG:boto:Port: 8000 2023-07-29T00:34:59.451 DEBUG:boto:Params: {} 2023-07-29T00:34:59.452 DEBUG:boto:Token: None 2023-07-29T00:34:59.452 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj51?versionId=null 2023-07-29T00:34:59.453 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Jix5eocE4wwY11abO1EKcHr/7fc= 2023-07-29T00:34:59.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Jix5eocE4wwY11abO1EKcHr/7fc=', 'Content-Length': '0'} 2023-07-29T00:34:59.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000437f78f8bdc16d29-0064c45eb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.456 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:34:59.456 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:34:59.456 DEBUG:boto:path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:34:59.457 DEBUG:boto:auth_path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:34:59.457 DEBUG:boto:Method: HEAD 2023-07-29T00:34:59.457 DEBUG:boto:Path: /hiiqos-59/obj51?versionId=null 2023-07-29T00:34:59.458 DEBUG:boto:Data: 2023-07-29T00:34:59.458 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.458 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:59.459 DEBUG:boto:Port: 8001 2023-07-29T00:34:59.459 DEBUG:boto:Params: {} 2023-07-29T00:34:59.459 DEBUG:boto:Token: None 2023-07-29T00:34:59.460 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj51?versionId=null 2023-07-29T00:34:59.460 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:K1AtXOEyNwQgp0t0EZG+ryEdc5g= 2023-07-29T00:34:59.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K1AtXOEyNwQgp0t0EZG+ryEdc5g=', 'Content-Length': '0'} 2023-07-29T00:34:59.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3db433f30cebb20-0064c45eb3-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.463 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:34:59.463 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:34:59.463 DEBUG:boto:path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:34:59.464 DEBUG:boto:auth_path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:34:59.464 DEBUG:boto:Method: HEAD 2023-07-29T00:34:59.464 DEBUG:boto:Path: /hiiqos-59/obj51?versionId=null 2023-07-29T00:34:59.465 DEBUG:boto:Data: 2023-07-29T00:34:59.465 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.465 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:59.466 DEBUG:boto:Port: 8000 2023-07-29T00:34:59.466 DEBUG:boto:Params: {} 2023-07-29T00:34:59.466 DEBUG:boto:Token: None 2023-07-29T00:34:59.467 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj51?versionId=null 2023-07-29T00:34:59.467 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:K1AtXOEyNwQgp0t0EZG+ryEdc5g= 2023-07-29T00:34:59.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K1AtXOEyNwQgp0t0EZG+ryEdc5g=', 'Content-Length': '0'} 2023-07-29T00:34:59.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3305bda0afee699-0064c45eb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-07-29T00:34:59.470 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:34:59.470 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:34:59.471 DEBUG:boto:path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:34:59.471 DEBUG:boto:auth_path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:34:59.472 DEBUG:boto:Method: GET 2023-07-29T00:34:59.472 DEBUG:boto:Path: /hiiqos-59/obj51?versionId=null 2023-07-29T00:34:59.472 DEBUG:boto:Data: 2023-07-29T00:34:59.473 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.473 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:59.473 DEBUG:boto:Port: 8001 2023-07-29T00:34:59.474 DEBUG:boto:Params: {} 2023-07-29T00:34:59.474 DEBUG:boto:Token: None 2023-07-29T00:34:59.474 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj51?versionId=null 2023-07-29T00:34:59.475 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Jix5eocE4wwY11abO1EKcHr/7fc= 2023-07-29T00:34:59.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Jix5eocE4wwY11abO1EKcHr/7fc=', 'Content-Length': '0'} 2023-07-29T00:34:59.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbed112722520077-0064c45eb3-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.478 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:34:59.478 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:34:59.479 DEBUG:boto:path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:34:59.479 DEBUG:boto:auth_path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:34:59.479 DEBUG:boto:Method: GET 2023-07-29T00:34:59.480 DEBUG:boto:Path: /hiiqos-59/obj51?versionId=null 2023-07-29T00:34:59.480 DEBUG:boto:Data: 2023-07-29T00:34:59.480 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.481 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:59.481 DEBUG:boto:Port: 8000 2023-07-29T00:34:59.482 DEBUG:boto:Params: {} 2023-07-29T00:34:59.482 DEBUG:boto:Token: None 2023-07-29T00:34:59.482 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj51?versionId=null 2023-07-29T00:34:59.483 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Jix5eocE4wwY11abO1EKcHr/7fc= 2023-07-29T00:34:59.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Jix5eocE4wwY11abO1EKcHr/7fc=', 'Content-Length': '0'} 2023-07-29T00:34:59.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcfaab6a26307f71-0064c45eb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.486 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:59.487 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:59.487 DEBUG:boto:path=/hiiqos-59/?prefix=obj51 2023-07-29T00:34:59.487 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj51 2023-07-29T00:34:59.488 DEBUG:boto:Method: GET 2023-07-29T00:34:59.488 DEBUG:boto:Path: /hiiqos-59/?prefix=obj51 2023-07-29T00:34:59.488 DEBUG:boto:Data: 2023-07-29T00:34:59.489 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.489 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:59.489 DEBUG:boto:Port: 8001 2023-07-29T00:34:59.490 DEBUG:boto:Params: {} 2023-07-29T00:34:59.490 DEBUG:boto:Token: None 2023-07-29T00:34:59.491 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/ 2023-07-29T00:34:59.491 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SnG8lOSwilMTlkBxsp4+W6VQZMA= 2023-07-29T00:34:59.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SnG8lOSwilMTlkBxsp4+W6VQZMA=', 'Content-Length': '0'} 2023-07-29T00:34:59.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002809326dc7774883-0064c45eb3-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.536 DEBUG:boto:b'hiiqos-59obj511000falseobj512023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:59.538 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:59.538 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:59.539 DEBUG:boto:path=/hiiqos-59/?prefix=obj51 2023-07-29T00:34:59.539 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj51 2023-07-29T00:34:59.539 DEBUG:boto:Method: GET 2023-07-29T00:34:59.540 DEBUG:boto:Path: /hiiqos-59/?prefix=obj51 2023-07-29T00:34:59.540 DEBUG:boto:Data: 2023-07-29T00:34:59.541 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.542 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:59.542 DEBUG:boto:Port: 8000 2023-07-29T00:34:59.542 DEBUG:boto:Params: {} 2023-07-29T00:34:59.543 DEBUG:boto:Token: None 2023-07-29T00:34:59.543 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/ 2023-07-29T00:34:59.544 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SnG8lOSwilMTlkBxsp4+W6VQZMA= 2023-07-29T00:34:59.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SnG8lOSwilMTlkBxsp4+W6VQZMA=', 'Content-Length': '0'} 2023-07-29T00:34:59.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9032484f69c5e97-0064c45eb3-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.548 DEBUG:boto:b'hiiqos-59obj511000falseobj512023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:59.549 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:34:59.549 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:34:59.550 DEBUG:boto:Method: HEAD 2023-07-29T00:34:59.550 DEBUG:boto:Path: /hiiqos-59/obj51 2023-07-29T00:34:59.551 DEBUG:boto:Data: 2023-07-29T00:34:59.551 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.552 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:59.552 DEBUG:boto:Port: 8001 2023-07-29T00:34:59.552 DEBUG:boto:Params: {} 2023-07-29T00:34:59.553 DEBUG:boto:Token: None 2023-07-29T00:34:59.553 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj51 2023-07-29T00:34:59.554 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Iy/FkvhSLd321g1+97eZ2u955UU= 2023-07-29T00:34:59.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Iy/FkvhSLd321g1+97eZ2u955UU=', 'Content-Length': '0'} 2023-07-29T00:34:59.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000167e48ec9fd27295-0064c45eb3-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.557 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:34:59.557 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:34:59.557 DEBUG:boto:Method: HEAD 2023-07-29T00:34:59.558 DEBUG:boto:Path: /hiiqos-59/obj51 2023-07-29T00:34:59.558 DEBUG:boto:Data: 2023-07-29T00:34:59.558 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.559 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:59.559 DEBUG:boto:Port: 8000 2023-07-29T00:34:59.559 DEBUG:boto:Params: {} 2023-07-29T00:34:59.560 DEBUG:boto:Token: None 2023-07-29T00:34:59.560 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj51 2023-07-29T00:34:59.561 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Iy/FkvhSLd321g1+97eZ2u955UU= 2023-07-29T00:34:59.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Iy/FkvhSLd321g1+97eZ2u955UU=', 'Content-Length': '0'} 2023-07-29T00:34:59.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003485e0a7b0a9a1bc-0064c45eb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-07-29T00:34:59.564 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:34:59.564 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:34:59.565 DEBUG:boto:Method: GET 2023-07-29T00:34:59.565 DEBUG:boto:Path: /hiiqos-59/obj51 2023-07-29T00:34:59.566 DEBUG:boto:Data: 2023-07-29T00:34:59.566 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.566 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:59.567 DEBUG:boto:Port: 8001 2023-07-29T00:34:59.567 DEBUG:boto:Params: {} 2023-07-29T00:34:59.568 DEBUG:boto:Token: None 2023-07-29T00:34:59.568 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj51 2023-07-29T00:34:59.568 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zL9s7VMyoKrdEOKsP6+GhUcksWo= 2023-07-29T00:34:59.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zL9s7VMyoKrdEOKsP6+GhUcksWo=', 'Content-Length': '0'} 2023-07-29T00:34:59.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d5d852ab82f9a54-0064c45eb3-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.571 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:34:59.572 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:34:59.572 DEBUG:boto:Method: GET 2023-07-29T00:34:59.572 DEBUG:boto:Path: /hiiqos-59/obj51 2023-07-29T00:34:59.573 DEBUG:boto:Data: 2023-07-29T00:34:59.573 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.574 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:59.574 DEBUG:boto:Port: 8000 2023-07-29T00:34:59.574 DEBUG:boto:Params: {} 2023-07-29T00:34:59.575 DEBUG:boto:Token: None 2023-07-29T00:34:59.575 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj51 2023-07-29T00:34:59.576 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zL9s7VMyoKrdEOKsP6+GhUcksWo= 2023-07-29T00:34:59.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zL9s7VMyoKrdEOKsP6+GhUcksWo=', 'Content-Length': '0'} 2023-07-29T00:34:59.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058c95dc7353393d6-0064c45eb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.579 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:59.579 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:59.579 DEBUG:boto:path=/hiiqos-59/?prefix=obj51 2023-07-29T00:34:59.580 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj51 2023-07-29T00:34:59.580 DEBUG:boto:Method: GET 2023-07-29T00:34:59.581 DEBUG:boto:Path: /hiiqos-59/?prefix=obj51 2023-07-29T00:34:59.581 DEBUG:boto:Data: 2023-07-29T00:34:59.581 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.582 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:59.582 DEBUG:boto:Port: 8001 2023-07-29T00:34:59.583 DEBUG:boto:Params: {} 2023-07-29T00:34:59.583 DEBUG:boto:Token: None 2023-07-29T00:34:59.583 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/ 2023-07-29T00:34:59.584 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SnG8lOSwilMTlkBxsp4+W6VQZMA= 2023-07-29T00:34:59.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SnG8lOSwilMTlkBxsp4+W6VQZMA=', 'Content-Length': '0'} 2023-07-29T00:34:59.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001da433e79807191e-0064c45eb3-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.587 DEBUG:boto:b'hiiqos-59obj511000falseobj512023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:59.588 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:59.588 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:59.589 DEBUG:boto:path=/hiiqos-59/?prefix=obj51 2023-07-29T00:34:59.589 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj51 2023-07-29T00:34:59.590 DEBUG:boto:Method: GET 2023-07-29T00:34:59.590 DEBUG:boto:Path: /hiiqos-59/?prefix=obj51 2023-07-29T00:34:59.590 DEBUG:boto:Data: 2023-07-29T00:34:59.591 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.591 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:59.592 DEBUG:boto:Port: 8000 2023-07-29T00:34:59.592 DEBUG:boto:Params: {} 2023-07-29T00:34:59.593 DEBUG:boto:Token: None 2023-07-29T00:34:59.593 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/ 2023-07-29T00:34:59.594 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SnG8lOSwilMTlkBxsp4+W6VQZMA= 2023-07-29T00:34:59.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SnG8lOSwilMTlkBxsp4+W6VQZMA=', 'Content-Length': '0'} 2023-07-29T00:34:59.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4bd704d8f3cd512-0064c45eb3-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.598 DEBUG:boto:b'hiiqos-59obj511000falseobj512023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:59.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-07-29T00:34:59.599 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:34:59.600 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:34:59.600 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:34:59.600 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:34:59.601 DEBUG:boto:Method: GET 2023-07-29T00:34:59.601 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:34:59.602 DEBUG:boto:Data: 2023-07-29T00:34:59.602 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.602 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:59.603 DEBUG:boto:Port: 8001 2023-07-29T00:34:59.603 DEBUG:boto:Params: {} 2023-07-29T00:34:59.604 DEBUG:boto:Token: None 2023-07-29T00:34:59.604 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:34:59.605 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hUY/56XCMAl/YYozIRmGqJ9XWJ4= 2023-07-29T00:34:59.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hUY/56XCMAl/YYozIRmGqJ9XWJ4=', 'Content-Length': '0'} 2023-07-29T00:34:59.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1e63bdf1d331eb7-0064c45eb3-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.608 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:34:59.608 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:34:59.608 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:34:59.609 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:34:59.609 DEBUG:boto:Method: GET 2023-07-29T00:34:59.610 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:34:59.610 DEBUG:boto:Data: 2023-07-29T00:34:59.610 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.611 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:59.611 DEBUG:boto:Port: 8000 2023-07-29T00:34:59.611 DEBUG:boto:Params: {} 2023-07-29T00:34:59.612 DEBUG:boto:Token: None 2023-07-29T00:34:59.612 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:34:59.613 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hUY/56XCMAl/YYozIRmGqJ9XWJ4= 2023-07-29T00:34:59.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hUY/56XCMAl/YYozIRmGqJ9XWJ4=', 'Content-Length': '0'} 2023-07-29T00:34:59.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006204ee31db51db97-0064c45eb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.656 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:34:59.657 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:34:59.657 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:34:59.657 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:34:59.658 DEBUG:boto:Method: HEAD 2023-07-29T00:34:59.658 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:34:59.658 DEBUG:boto:Data: 2023-07-29T00:34:59.659 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.659 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:59.659 DEBUG:boto:Port: 8001 2023-07-29T00:34:59.660 DEBUG:boto:Params: {} 2023-07-29T00:34:59.660 DEBUG:boto:Token: None 2023-07-29T00:34:59.661 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:34:59.661 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DrdnXdMuniJGU2+mb4DSD3EqMC4= 2023-07-29T00:34:59.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DrdnXdMuniJGU2+mb4DSD3EqMC4=', 'Content-Length': '0'} 2023-07-29T00:34:59.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f7f7bd8b739e7a0-0064c45eb3-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.664 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:34:59.665 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:34:59.665 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:34:59.665 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:34:59.666 DEBUG:boto:Method: HEAD 2023-07-29T00:34:59.666 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:34:59.666 DEBUG:boto:Data: 2023-07-29T00:34:59.667 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.667 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:59.667 DEBUG:boto:Port: 8000 2023-07-29T00:34:59.668 DEBUG:boto:Params: {} 2023-07-29T00:34:59.668 DEBUG:boto:Token: None 2023-07-29T00:34:59.669 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:34:59.669 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DrdnXdMuniJGU2+mb4DSD3EqMC4= 2023-07-29T00:34:59.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DrdnXdMuniJGU2+mb4DSD3EqMC4=', 'Content-Length': '0'} 2023-07-29T00:34:59.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001aa7d8941858f56b-0064c45eb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-07-29T00:34:59.673 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:34:59.673 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:34:59.673 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:34:59.674 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:34:59.674 DEBUG:boto:Method: GET 2023-07-29T00:34:59.675 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:34:59.675 DEBUG:boto:Data: 2023-07-29T00:34:59.676 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.676 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:59.677 DEBUG:boto:Port: 8001 2023-07-29T00:34:59.677 DEBUG:boto:Params: {} 2023-07-29T00:34:59.677 DEBUG:boto:Token: None 2023-07-29T00:34:59.678 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:34:59.679 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hUY/56XCMAl/YYozIRmGqJ9XWJ4= 2023-07-29T00:34:59.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hUY/56XCMAl/YYozIRmGqJ9XWJ4=', 'Content-Length': '0'} 2023-07-29T00:34:59.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b83f9ad6cea19154-0064c45eb3-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.682 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:34:59.683 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:34:59.683 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:34:59.683 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:34:59.684 DEBUG:boto:Method: GET 2023-07-29T00:34:59.684 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:34:59.685 DEBUG:boto:Data: 2023-07-29T00:34:59.685 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.686 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:59.686 DEBUG:boto:Port: 8000 2023-07-29T00:34:59.686 DEBUG:boto:Params: {} 2023-07-29T00:34:59.687 DEBUG:boto:Token: None 2023-07-29T00:34:59.688 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:34:59.688 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hUY/56XCMAl/YYozIRmGqJ9XWJ4= 2023-07-29T00:34:59.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hUY/56XCMAl/YYozIRmGqJ9XWJ4=', 'Content-Length': '0'} 2023-07-29T00:34:59.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b3eebc772d9988a-0064c45eb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.692 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:59.692 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:59.693 DEBUG:boto:path=/hiiqos-59/?prefix=obj55 2023-07-29T00:34:59.693 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj55 2023-07-29T00:34:59.693 DEBUG:boto:Method: GET 2023-07-29T00:34:59.694 DEBUG:boto:Path: /hiiqos-59/?prefix=obj55 2023-07-29T00:34:59.694 DEBUG:boto:Data: 2023-07-29T00:34:59.695 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.695 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:59.696 DEBUG:boto:Port: 8001 2023-07-29T00:34:59.697 DEBUG:boto:Params: {} 2023-07-29T00:34:59.698 DEBUG:boto:Token: None 2023-07-29T00:34:59.698 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/ 2023-07-29T00:34:59.699 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SnG8lOSwilMTlkBxsp4+W6VQZMA= 2023-07-29T00:34:59.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SnG8lOSwilMTlkBxsp4+W6VQZMA=', 'Content-Length': '0'} 2023-07-29T00:34:59.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cac7d0d7dda446ed-0064c45eb3-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.704 DEBUG:boto:b'hiiqos-59obj551000falseobj552023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:59.704 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:59.705 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:59.705 DEBUG:boto:path=/hiiqos-59/?prefix=obj55 2023-07-29T00:34:59.706 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj55 2023-07-29T00:34:59.706 DEBUG:boto:Method: GET 2023-07-29T00:34:59.707 DEBUG:boto:Path: /hiiqos-59/?prefix=obj55 2023-07-29T00:34:59.707 DEBUG:boto:Data: 2023-07-29T00:34:59.707 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.708 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:59.708 DEBUG:boto:Port: 8000 2023-07-29T00:34:59.708 DEBUG:boto:Params: {} 2023-07-29T00:34:59.709 DEBUG:boto:Token: None 2023-07-29T00:34:59.709 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/ 2023-07-29T00:34:59.710 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SnG8lOSwilMTlkBxsp4+W6VQZMA= 2023-07-29T00:34:59.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SnG8lOSwilMTlkBxsp4+W6VQZMA=', 'Content-Length': '0'} 2023-07-29T00:34:59.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7c915b0063c1e94-0064c45eb3-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.714 DEBUG:boto:b'hiiqos-59obj551000falseobj552023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:59.715 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:34:59.715 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:34:59.715 DEBUG:boto:Method: HEAD 2023-07-29T00:34:59.716 DEBUG:boto:Path: /hiiqos-59/obj55 2023-07-29T00:34:59.716 DEBUG:boto:Data: 2023-07-29T00:34:59.717 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.717 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:59.717 DEBUG:boto:Port: 8001 2023-07-29T00:34:59.718 DEBUG:boto:Params: {} 2023-07-29T00:34:59.718 DEBUG:boto:Token: None 2023-07-29T00:34:59.718 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj55 2023-07-29T00:34:59.719 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IphSNs1JqX62mnWIfJdGafpzsMU= 2023-07-29T00:34:59.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IphSNs1JqX62mnWIfJdGafpzsMU=', 'Content-Length': '0'} 2023-07-29T00:34:59.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b946fca861d20d71-0064c45eb3-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.722 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:34:59.722 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:34:59.722 DEBUG:boto:Method: HEAD 2023-07-29T00:34:59.723 DEBUG:boto:Path: /hiiqos-59/obj55 2023-07-29T00:34:59.723 DEBUG:boto:Data: 2023-07-29T00:34:59.723 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.724 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:59.724 DEBUG:boto:Port: 8000 2023-07-29T00:34:59.724 DEBUG:boto:Params: {} 2023-07-29T00:34:59.725 DEBUG:boto:Token: None 2023-07-29T00:34:59.725 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj55 2023-07-29T00:34:59.726 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IphSNs1JqX62mnWIfJdGafpzsMU= 2023-07-29T00:34:59.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IphSNs1JqX62mnWIfJdGafpzsMU=', 'Content-Length': '0'} 2023-07-29T00:34:59.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a15cb9ae17d46900-0064c45eb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-07-29T00:34:59.730 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:34:59.730 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:34:59.731 DEBUG:boto:Method: GET 2023-07-29T00:34:59.732 DEBUG:boto:Path: /hiiqos-59/obj55 2023-07-29T00:34:59.733 DEBUG:boto:Data: 2023-07-29T00:34:59.734 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.735 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:59.735 DEBUG:boto:Port: 8001 2023-07-29T00:34:59.736 DEBUG:boto:Params: {} 2023-07-29T00:34:59.736 DEBUG:boto:Token: None 2023-07-29T00:34:59.737 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj55 2023-07-29T00:34:59.737 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rr+B4HONmzrxqVs9YGYnZ7BzxBI= 2023-07-29T00:34:59.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rr+B4HONmzrxqVs9YGYnZ7BzxBI=', 'Content-Length': '0'} 2023-07-29T00:34:59.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e2c61848ebcb70d-0064c45eb3-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.741 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:34:59.741 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:34:59.742 DEBUG:boto:Method: GET 2023-07-29T00:34:59.742 DEBUG:boto:Path: /hiiqos-59/obj55 2023-07-29T00:34:59.742 DEBUG:boto:Data: 2023-07-29T00:34:59.743 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.743 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:59.743 DEBUG:boto:Port: 8000 2023-07-29T00:34:59.744 DEBUG:boto:Params: {} 2023-07-29T00:34:59.744 DEBUG:boto:Token: None 2023-07-29T00:34:59.745 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj55 2023-07-29T00:34:59.745 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rr+B4HONmzrxqVs9YGYnZ7BzxBI= 2023-07-29T00:34:59.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rr+B4HONmzrxqVs9YGYnZ7BzxBI=', 'Content-Length': '0'} 2023-07-29T00:34:59.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091f91dcd2c44e587-0064c45eb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.788 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:59.789 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:59.791 DEBUG:boto:path=/hiiqos-59/?prefix=obj55 2023-07-29T00:34:59.791 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj55 2023-07-29T00:34:59.792 DEBUG:boto:Method: GET 2023-07-29T00:34:59.793 DEBUG:boto:Path: /hiiqos-59/?prefix=obj55 2023-07-29T00:34:59.794 DEBUG:boto:Data: 2023-07-29T00:34:59.794 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.795 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:59.796 DEBUG:boto:Port: 8001 2023-07-29T00:34:59.797 DEBUG:boto:Params: {} 2023-07-29T00:34:59.797 DEBUG:boto:Token: None 2023-07-29T00:34:59.798 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/ 2023-07-29T00:34:59.798 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SnG8lOSwilMTlkBxsp4+W6VQZMA= 2023-07-29T00:34:59.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SnG8lOSwilMTlkBxsp4+W6VQZMA=', 'Content-Length': '0'} 2023-07-29T00:34:59.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a38a3f49045deb8-0064c45eb3-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.801 DEBUG:boto:b'hiiqos-59obj551000falseobj552023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:59.802 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:59.803 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:59.803 DEBUG:boto:path=/hiiqos-59/?prefix=obj55 2023-07-29T00:34:59.803 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj55 2023-07-29T00:34:59.804 DEBUG:boto:Method: GET 2023-07-29T00:34:59.804 DEBUG:boto:Path: /hiiqos-59/?prefix=obj55 2023-07-29T00:34:59.804 DEBUG:boto:Data: 2023-07-29T00:34:59.805 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.805 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:59.806 DEBUG:boto:Port: 8000 2023-07-29T00:34:59.806 DEBUG:boto:Params: {} 2023-07-29T00:34:59.807 DEBUG:boto:Token: None 2023-07-29T00:34:59.807 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/ 2023-07-29T00:34:59.808 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SnG8lOSwilMTlkBxsp4+W6VQZMA= 2023-07-29T00:34:59.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SnG8lOSwilMTlkBxsp4+W6VQZMA=', 'Content-Length': '0'} 2023-07-29T00:34:59.812 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023aabdbfae747fbc-0064c45eb3-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.812 DEBUG:boto:b'hiiqos-59obj551000falseobj552023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:59.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-07-29T00:34:59.814 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:34:59.814 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:34:59.814 DEBUG:boto:path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:34:59.815 DEBUG:boto:auth_path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:34:59.815 DEBUG:boto:Method: GET 2023-07-29T00:34:59.816 DEBUG:boto:Path: /hiiqos-59/obj57?versionId=null 2023-07-29T00:34:59.816 DEBUG:boto:Data: 2023-07-29T00:34:59.817 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.817 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:59.818 DEBUG:boto:Port: 8001 2023-07-29T00:34:59.818 DEBUG:boto:Params: {} 2023-07-29T00:34:59.819 DEBUG:boto:Token: None 2023-07-29T00:34:59.819 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj57?versionId=null 2023-07-29T00:34:59.820 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1hdCvC739gywutMUwYQHNTswg7I= 2023-07-29T00:34:59.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1hdCvC739gywutMUwYQHNTswg7I=', 'Content-Length': '0'} 2023-07-29T00:34:59.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ca2eb3d257ae47d-0064c45eb3-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.864 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:34:59.865 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:34:59.866 DEBUG:boto:path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:34:59.866 DEBUG:boto:auth_path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:34:59.866 DEBUG:boto:Method: GET 2023-07-29T00:34:59.867 DEBUG:boto:Path: /hiiqos-59/obj57?versionId=null 2023-07-29T00:34:59.868 DEBUG:boto:Data: 2023-07-29T00:34:59.868 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.869 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:59.870 DEBUG:boto:Port: 8000 2023-07-29T00:34:59.870 DEBUG:boto:Params: {} 2023-07-29T00:34:59.871 DEBUG:boto:Token: None 2023-07-29T00:34:59.871 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj57?versionId=null 2023-07-29T00:34:59.871 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1hdCvC739gywutMUwYQHNTswg7I= 2023-07-29T00:34:59.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1hdCvC739gywutMUwYQHNTswg7I=', 'Content-Length': '0'} 2023-07-29T00:34:59.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aec23bc6e8bedcf8-0064c45eb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.875 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:34:59.875 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:34:59.875 DEBUG:boto:path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:34:59.876 DEBUG:boto:auth_path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:34:59.876 DEBUG:boto:Method: HEAD 2023-07-29T00:34:59.876 DEBUG:boto:Path: /hiiqos-59/obj57?versionId=null 2023-07-29T00:34:59.877 DEBUG:boto:Data: 2023-07-29T00:34:59.877 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.877 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:59.878 DEBUG:boto:Port: 8001 2023-07-29T00:34:59.878 DEBUG:boto:Params: {} 2023-07-29T00:34:59.878 DEBUG:boto:Token: None 2023-07-29T00:34:59.879 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj57?versionId=null 2023-07-29T00:34:59.879 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Qy444SxddyBJx6FjTEeFzpCnLQo= 2023-07-29T00:34:59.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Qy444SxddyBJx6FjTEeFzpCnLQo=', 'Content-Length': '0'} 2023-07-29T00:34:59.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ddf92ed0ad06172-0064c45eb3-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.882 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:34:59.883 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:34:59.883 DEBUG:boto:path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:34:59.883 DEBUG:boto:auth_path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:34:59.884 DEBUG:boto:Method: HEAD 2023-07-29T00:34:59.884 DEBUG:boto:Path: /hiiqos-59/obj57?versionId=null 2023-07-29T00:34:59.884 DEBUG:boto:Data: 2023-07-29T00:34:59.885 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.885 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:59.885 DEBUG:boto:Port: 8000 2023-07-29T00:34:59.886 DEBUG:boto:Params: {} 2023-07-29T00:34:59.886 DEBUG:boto:Token: None 2023-07-29T00:34:59.886 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj57?versionId=null 2023-07-29T00:34:59.887 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Qy444SxddyBJx6FjTEeFzpCnLQo= 2023-07-29T00:34:59.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Qy444SxddyBJx6FjTEeFzpCnLQo=', 'Content-Length': '0'} 2023-07-29T00:34:59.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000350a945bbefc3a3-0064c45eb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-07-29T00:34:59.890 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:34:59.891 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:34:59.891 DEBUG:boto:path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:34:59.891 DEBUG:boto:auth_path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:34:59.892 DEBUG:boto:Method: GET 2023-07-29T00:34:59.892 DEBUG:boto:Path: /hiiqos-59/obj57?versionId=null 2023-07-29T00:34:59.892 DEBUG:boto:Data: 2023-07-29T00:34:59.893 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.893 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:59.893 DEBUG:boto:Port: 8001 2023-07-29T00:34:59.894 DEBUG:boto:Params: {} 2023-07-29T00:34:59.894 DEBUG:boto:Token: None 2023-07-29T00:34:59.895 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj57?versionId=null 2023-07-29T00:34:59.895 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1hdCvC739gywutMUwYQHNTswg7I= 2023-07-29T00:34:59.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1hdCvC739gywutMUwYQHNTswg7I=', 'Content-Length': '0'} 2023-07-29T00:34:59.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f1be21282c09b78-0064c45eb3-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.898 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:34:59.898 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:34:59.899 DEBUG:boto:path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:34:59.899 DEBUG:boto:auth_path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:34:59.899 DEBUG:boto:Method: GET 2023-07-29T00:34:59.900 DEBUG:boto:Path: /hiiqos-59/obj57?versionId=null 2023-07-29T00:34:59.900 DEBUG:boto:Data: 2023-07-29T00:34:59.901 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.901 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:59.901 DEBUG:boto:Port: 8000 2023-07-29T00:34:59.902 DEBUG:boto:Params: {} 2023-07-29T00:34:59.902 DEBUG:boto:Token: None 2023-07-29T00:34:59.902 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj57?versionId=null 2023-07-29T00:34:59.903 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1hdCvC739gywutMUwYQHNTswg7I= 2023-07-29T00:34:59.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1hdCvC739gywutMUwYQHNTswg7I=', 'Content-Length': '0'} 2023-07-29T00:34:59.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4061d61aad12f1f-0064c45eb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.906 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:59.906 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:59.907 DEBUG:boto:path=/hiiqos-59/?prefix=obj57 2023-07-29T00:34:59.907 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj57 2023-07-29T00:34:59.907 DEBUG:boto:Method: GET 2023-07-29T00:34:59.908 DEBUG:boto:Path: /hiiqos-59/?prefix=obj57 2023-07-29T00:34:59.908 DEBUG:boto:Data: 2023-07-29T00:34:59.908 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.909 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:59.909 DEBUG:boto:Port: 8001 2023-07-29T00:34:59.909 DEBUG:boto:Params: {} 2023-07-29T00:34:59.910 DEBUG:boto:Token: None 2023-07-29T00:34:59.910 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/ 2023-07-29T00:34:59.911 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SnG8lOSwilMTlkBxsp4+W6VQZMA= 2023-07-29T00:34:59.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SnG8lOSwilMTlkBxsp4+W6VQZMA=', 'Content-Length': '0'} 2023-07-29T00:34:59.913 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d533ac4b121d99db-0064c45eb3-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.914 DEBUG:boto:b'hiiqos-59obj571000falseobj572023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:59.915 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:59.915 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:59.915 DEBUG:boto:path=/hiiqos-59/?prefix=obj57 2023-07-29T00:34:59.916 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj57 2023-07-29T00:34:59.916 DEBUG:boto:Method: GET 2023-07-29T00:34:59.916 DEBUG:boto:Path: /hiiqos-59/?prefix=obj57 2023-07-29T00:34:59.917 DEBUG:boto:Data: 2023-07-29T00:34:59.917 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.918 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:59.918 DEBUG:boto:Port: 8000 2023-07-29T00:34:59.918 DEBUG:boto:Params: {} 2023-07-29T00:34:59.919 DEBUG:boto:Token: None 2023-07-29T00:34:59.919 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/ 2023-07-29T00:34:59.919 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SnG8lOSwilMTlkBxsp4+W6VQZMA= 2023-07-29T00:34:59.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SnG8lOSwilMTlkBxsp4+W6VQZMA=', 'Content-Length': '0'} 2023-07-29T00:34:59.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fc19bfdc0ca73a5-0064c45eb3-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.964 DEBUG:boto:b'hiiqos-59obj571000falseobj572023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:34:59.965 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:34:59.965 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:34:59.966 DEBUG:boto:Method: HEAD 2023-07-29T00:34:59.966 DEBUG:boto:Path: /hiiqos-59/obj57 2023-07-29T00:34:59.967 DEBUG:boto:Data: 2023-07-29T00:34:59.967 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.967 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:59.968 DEBUG:boto:Port: 8001 2023-07-29T00:34:59.968 DEBUG:boto:Params: {} 2023-07-29T00:34:59.968 DEBUG:boto:Token: None 2023-07-29T00:34:59.969 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj57 2023-07-29T00:34:59.969 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uBnX2sud1aQPRd8t0CQ9Zw/FO6Y= 2023-07-29T00:34:59.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uBnX2sud1aQPRd8t0CQ9Zw/FO6Y=', 'Content-Length': '0'} 2023-07-29T00:34:59.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087ff29d0e22a6074-0064c45eb3-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.972 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:34:59.972 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:34:59.973 DEBUG:boto:Method: HEAD 2023-07-29T00:34:59.973 DEBUG:boto:Path: /hiiqos-59/obj57 2023-07-29T00:34:59.973 DEBUG:boto:Data: 2023-07-29T00:34:59.974 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.974 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:59.974 DEBUG:boto:Port: 8000 2023-07-29T00:34:59.975 DEBUG:boto:Params: {} 2023-07-29T00:34:59.975 DEBUG:boto:Token: None 2023-07-29T00:34:59.976 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj57 2023-07-29T00:34:59.976 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uBnX2sud1aQPRd8t0CQ9Zw/FO6Y= 2023-07-29T00:34:59.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uBnX2sud1aQPRd8t0CQ9Zw/FO6Y=', 'Content-Length': '0'} 2023-07-29T00:34:59.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7fbf15c9b59eabd-0064c45eb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-07-29T00:34:59.979 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:34:59.980 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:34:59.980 DEBUG:boto:Method: GET 2023-07-29T00:34:59.980 DEBUG:boto:Path: /hiiqos-59/obj57 2023-07-29T00:34:59.981 DEBUG:boto:Data: 2023-07-29T00:34:59.981 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.982 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:59.982 DEBUG:boto:Port: 8001 2023-07-29T00:34:59.982 DEBUG:boto:Params: {} 2023-07-29T00:34:59.983 DEBUG:boto:Token: None 2023-07-29T00:34:59.983 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj57 2023-07-29T00:34:59.984 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pvcuJkHxBz7wN0QtZEQBG10Rbd0= 2023-07-29T00:34:59.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pvcuJkHxBz7wN0QtZEQBG10Rbd0=', 'Content-Length': '0'} 2023-07-29T00:34:59.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014ff7b24cddddee2-0064c45eb3-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.987 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:34:59.987 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:34:59.988 DEBUG:boto:Method: GET 2023-07-29T00:34:59.988 DEBUG:boto:Path: /hiiqos-59/obj57 2023-07-29T00:34:59.989 DEBUG:boto:Data: 2023-07-29T00:34:59.989 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.989 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:34:59.990 DEBUG:boto:Port: 8000 2023-07-29T00:34:59.990 DEBUG:boto:Params: {} 2023-07-29T00:34:59.991 DEBUG:boto:Token: None 2023-07-29T00:34:59.991 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/obj57 2023-07-29T00:34:59.991 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pvcuJkHxBz7wN0QtZEQBG10Rbd0= 2023-07-29T00:34:59.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pvcuJkHxBz7wN0QtZEQBG10Rbd0=', 'Content-Length': '0'} 2023-07-29T00:34:59.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cfcd9fd24a4cfb2-0064c45eb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:34:59.995 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:34:59.995 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:34:59.995 DEBUG:boto:path=/hiiqos-59/?prefix=obj57 2023-07-29T00:34:59.996 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj57 2023-07-29T00:34:59.996 DEBUG:boto:Method: GET 2023-07-29T00:34:59.996 DEBUG:boto:Path: /hiiqos-59/?prefix=obj57 2023-07-29T00:34:59.997 DEBUG:boto:Data: 2023-07-29T00:34:59.997 DEBUG:boto:Headers: {} 2023-07-29T00:34:59.998 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:34:59.998 DEBUG:boto:Port: 8001 2023-07-29T00:34:59.998 DEBUG:boto:Params: {} 2023-07-29T00:34:59.999 DEBUG:boto:Token: None 2023-07-29T00:34:59.999 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:34:59 GMT /hiiqos-59/ 2023-07-29T00:34:59.999 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SnG8lOSwilMTlkBxsp4+W6VQZMA= 2023-07-29T00:35:00.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:34:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SnG8lOSwilMTlkBxsp4+W6VQZMA=', 'Content-Length': '0'} 2023-07-29T00:35:00.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b0d61752abfb70d-0064c45eb3-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.003 DEBUG:boto:b'hiiqos-59obj571000falseobj572023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:00.003 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:00.004 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:00.004 DEBUG:boto:path=/hiiqos-59/?prefix=obj57 2023-07-29T00:35:00.005 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj57 2023-07-29T00:35:00.005 DEBUG:boto:Method: GET 2023-07-29T00:35:00.006 DEBUG:boto:Path: /hiiqos-59/?prefix=obj57 2023-07-29T00:35:00.006 DEBUG:boto:Data: 2023-07-29T00:35:00.006 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.007 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:00.007 DEBUG:boto:Port: 8000 2023-07-29T00:35:00.007 DEBUG:boto:Params: {} 2023-07-29T00:35:00.008 DEBUG:boto:Token: None 2023-07-29T00:35:00.008 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/ 2023-07-29T00:35:00.009 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0N+PAhtHIs4KHe60wn+BGQJADio= 2023-07-29T00:35:00.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0N+PAhtHIs4KHe60wn+BGQJADio=', 'Content-Length': '0'} 2023-07-29T00:35:00.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a992b983195a63fe-0064c45eb3-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.013 DEBUG:boto:b'hiiqos-59obj571000falseobj572023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:00.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-07-29T00:35:00.016 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:35:00.017 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:35:00.017 DEBUG:boto:path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:35:00.017 DEBUG:boto:auth_path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:35:00.018 DEBUG:boto:Method: GET 2023-07-29T00:35:00.018 DEBUG:boto:Path: /hiiqos-59/obj68?versionId=null 2023-07-29T00:35:00.019 DEBUG:boto:Data: 2023-07-29T00:35:00.019 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.019 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:00.020 DEBUG:boto:Port: 8001 2023-07-29T00:35:00.020 DEBUG:boto:Params: {} 2023-07-29T00:35:00.021 DEBUG:boto:Token: None 2023-07-29T00:35:00.021 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj68?versionId=null 2023-07-29T00:35:00.021 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CzxfPPbhNW+NxhfONiyFMBlPh0Q= 2023-07-29T00:35:00.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CzxfPPbhNW+NxhfONiyFMBlPh0Q=', 'Content-Length': '0'} 2023-07-29T00:35:00.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008da34b6c00ce6e0b-0064c45eb3-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.025 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:35:00.025 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:35:00.025 DEBUG:boto:path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:35:00.026 DEBUG:boto:auth_path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:35:00.026 DEBUG:boto:Method: GET 2023-07-29T00:35:00.026 DEBUG:boto:Path: /hiiqos-59/obj68?versionId=null 2023-07-29T00:35:00.027 DEBUG:boto:Data: 2023-07-29T00:35:00.027 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.028 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:00.028 DEBUG:boto:Port: 8000 2023-07-29T00:35:00.028 DEBUG:boto:Params: {} 2023-07-29T00:35:00.029 DEBUG:boto:Token: None 2023-07-29T00:35:00.029 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj68?versionId=null 2023-07-29T00:35:00.030 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CzxfPPbhNW+NxhfONiyFMBlPh0Q= 2023-07-29T00:35:00.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CzxfPPbhNW+NxhfONiyFMBlPh0Q=', 'Content-Length': '0'} 2023-07-29T00:35:00.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049439816c45ecbbf-0064c45eb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.033 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:35:00.033 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:35:00.033 DEBUG:boto:path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:35:00.034 DEBUG:boto:auth_path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:35:00.034 DEBUG:boto:Method: HEAD 2023-07-29T00:35:00.034 DEBUG:boto:Path: /hiiqos-59/obj68?versionId=null 2023-07-29T00:35:00.035 DEBUG:boto:Data: 2023-07-29T00:35:00.035 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.035 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:00.036 DEBUG:boto:Port: 8001 2023-07-29T00:35:00.036 DEBUG:boto:Params: {} 2023-07-29T00:35:00.036 DEBUG:boto:Token: None 2023-07-29T00:35:00.037 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj68?versionId=null 2023-07-29T00:35:00.037 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:snjw+EXEpp7erRWcedeW8wcWfG4= 2023-07-29T00:35:00.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:snjw+EXEpp7erRWcedeW8wcWfG4=', 'Content-Length': '0'} 2023-07-29T00:35:00.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a33d6a941d4d3f69-0064c45eb3-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.040 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:35:00.040 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:35:00.041 DEBUG:boto:path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:35:00.041 DEBUG:boto:auth_path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:35:00.041 DEBUG:boto:Method: HEAD 2023-07-29T00:35:00.042 DEBUG:boto:Path: /hiiqos-59/obj68?versionId=null 2023-07-29T00:35:00.042 DEBUG:boto:Data: 2023-07-29T00:35:00.042 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.043 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:00.043 DEBUG:boto:Port: 8000 2023-07-29T00:35:00.043 DEBUG:boto:Params: {} 2023-07-29T00:35:00.044 DEBUG:boto:Token: None 2023-07-29T00:35:00.044 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj68?versionId=null 2023-07-29T00:35:00.045 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:snjw+EXEpp7erRWcedeW8wcWfG4= 2023-07-29T00:35:00.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:snjw+EXEpp7erRWcedeW8wcWfG4=', 'Content-Length': '0'} 2023-07-29T00:35:00.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efa8cc43a7555df9-0064c45eb3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-07-29T00:35:00.048 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:35:00.048 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:35:00.049 DEBUG:boto:path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:35:00.049 DEBUG:boto:auth_path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:35:00.049 DEBUG:boto:Method: GET 2023-07-29T00:35:00.050 DEBUG:boto:Path: /hiiqos-59/obj68?versionId=null 2023-07-29T00:35:00.050 DEBUG:boto:Data: 2023-07-29T00:35:00.050 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.051 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:00.051 DEBUG:boto:Port: 8001 2023-07-29T00:35:00.052 DEBUG:boto:Params: {} 2023-07-29T00:35:00.052 DEBUG:boto:Token: None 2023-07-29T00:35:00.052 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj68?versionId=null 2023-07-29T00:35:00.053 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CzxfPPbhNW+NxhfONiyFMBlPh0Q= 2023-07-29T00:35:00.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CzxfPPbhNW+NxhfONiyFMBlPh0Q=', 'Content-Length': '0'} 2023-07-29T00:35:00.055 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008313231f2e8355f3-0064c45eb3-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:34:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.056 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:35:00.056 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:35:00.057 DEBUG:boto:path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:35:00.057 DEBUG:boto:auth_path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:35:00.057 DEBUG:boto:Method: GET 2023-07-29T00:35:00.058 DEBUG:boto:Path: /hiiqos-59/obj68?versionId=null 2023-07-29T00:35:00.058 DEBUG:boto:Data: 2023-07-29T00:35:00.059 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.059 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:00.059 DEBUG:boto:Port: 8000 2023-07-29T00:35:00.060 DEBUG:boto:Params: {} 2023-07-29T00:35:00.060 DEBUG:boto:Token: None 2023-07-29T00:35:00.061 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj68?versionId=null 2023-07-29T00:35:00.061 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CzxfPPbhNW+NxhfONiyFMBlPh0Q= 2023-07-29T00:35:00.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CzxfPPbhNW+NxhfONiyFMBlPh0Q=', 'Content-Length': '0'} 2023-07-29T00:35:00.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000700bbfbd66852073-0064c45eb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.064 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:00.065 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:00.065 DEBUG:boto:path=/hiiqos-59/?prefix=obj68 2023-07-29T00:35:00.065 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj68 2023-07-29T00:35:00.066 DEBUG:boto:Method: GET 2023-07-29T00:35:00.066 DEBUG:boto:Path: /hiiqos-59/?prefix=obj68 2023-07-29T00:35:00.066 DEBUG:boto:Data: 2023-07-29T00:35:00.067 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.067 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:00.067 DEBUG:boto:Port: 8001 2023-07-29T00:35:00.068 DEBUG:boto:Params: {} 2023-07-29T00:35:00.068 DEBUG:boto:Token: None 2023-07-29T00:35:00.069 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/ 2023-07-29T00:35:00.069 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0N+PAhtHIs4KHe60wn+BGQJADio= 2023-07-29T00:35:00.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0N+PAhtHIs4KHe60wn+BGQJADio=', 'Content-Length': '0'} 2023-07-29T00:35:00.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021cb2f7a3e07cd39-0064c45eb4-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.072 DEBUG:boto:b'hiiqos-59obj681000falseobj682023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:00.073 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:00.073 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:00.074 DEBUG:boto:path=/hiiqos-59/?prefix=obj68 2023-07-29T00:35:00.074 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj68 2023-07-29T00:35:00.075 DEBUG:boto:Method: GET 2023-07-29T00:35:00.075 DEBUG:boto:Path: /hiiqos-59/?prefix=obj68 2023-07-29T00:35:00.075 DEBUG:boto:Data: 2023-07-29T00:35:00.076 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.076 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:00.076 DEBUG:boto:Port: 8000 2023-07-29T00:35:00.077 DEBUG:boto:Params: {} 2023-07-29T00:35:00.077 DEBUG:boto:Token: None 2023-07-29T00:35:00.078 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/ 2023-07-29T00:35:00.078 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0N+PAhtHIs4KHe60wn+BGQJADio= 2023-07-29T00:35:00.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0N+PAhtHIs4KHe60wn+BGQJADio=', 'Content-Length': '0'} 2023-07-29T00:35:00.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000039e7445214d304e-0064c45eb4-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.082 DEBUG:boto:b'hiiqos-59obj681000falseobj682023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:00.083 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:35:00.084 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:35:00.084 DEBUG:boto:Method: HEAD 2023-07-29T00:35:00.084 DEBUG:boto:Path: /hiiqos-59/obj68 2023-07-29T00:35:00.085 DEBUG:boto:Data: 2023-07-29T00:35:00.085 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.085 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:00.086 DEBUG:boto:Port: 8001 2023-07-29T00:35:00.086 DEBUG:boto:Params: {} 2023-07-29T00:35:00.086 DEBUG:boto:Token: None 2023-07-29T00:35:00.087 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj68 2023-07-29T00:35:00.087 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IeaDBypEhAGyhIcsIIw1pR70HC0= 2023-07-29T00:35:00.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IeaDBypEhAGyhIcsIIw1pR70HC0=', 'Content-Length': '0'} 2023-07-29T00:35:00.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029c4f5125c57fcfd-0064c45eb4-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.090 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:35:00.090 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:35:00.091 DEBUG:boto:Method: HEAD 2023-07-29T00:35:00.091 DEBUG:boto:Path: /hiiqos-59/obj68 2023-07-29T00:35:00.092 DEBUG:boto:Data: 2023-07-29T00:35:00.092 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.092 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:00.093 DEBUG:boto:Port: 8000 2023-07-29T00:35:00.093 DEBUG:boto:Params: {} 2023-07-29T00:35:00.093 DEBUG:boto:Token: None 2023-07-29T00:35:00.094 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj68 2023-07-29T00:35:00.094 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IeaDBypEhAGyhIcsIIw1pR70HC0= 2023-07-29T00:35:00.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IeaDBypEhAGyhIcsIIw1pR70HC0=', 'Content-Length': '0'} 2023-07-29T00:35:00.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfb4fd7628080ba6-0064c45eb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-07-29T00:35:00.097 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:35:00.098 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:35:00.098 DEBUG:boto:Method: GET 2023-07-29T00:35:00.099 DEBUG:boto:Path: /hiiqos-59/obj68 2023-07-29T00:35:00.099 DEBUG:boto:Data: 2023-07-29T00:35:00.099 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.100 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:00.100 DEBUG:boto:Port: 8001 2023-07-29T00:35:00.100 DEBUG:boto:Params: {} 2023-07-29T00:35:00.101 DEBUG:boto:Token: None 2023-07-29T00:35:00.101 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj68 2023-07-29T00:35:00.102 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mY6S2/Iw1ehY/cbE2z4Oa3O/oEo= 2023-07-29T00:35:00.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mY6S2/Iw1ehY/cbE2z4Oa3O/oEo=', 'Content-Length': '0'} 2023-07-29T00:35:00.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000770d0d8497b27828-0064c45eb4-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.105 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:35:00.105 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:35:00.105 DEBUG:boto:Method: GET 2023-07-29T00:35:00.106 DEBUG:boto:Path: /hiiqos-59/obj68 2023-07-29T00:35:00.106 DEBUG:boto:Data: 2023-07-29T00:35:00.106 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.107 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:00.107 DEBUG:boto:Port: 8000 2023-07-29T00:35:00.107 DEBUG:boto:Params: {} 2023-07-29T00:35:00.108 DEBUG:boto:Token: None 2023-07-29T00:35:00.108 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj68 2023-07-29T00:35:00.109 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mY6S2/Iw1ehY/cbE2z4Oa3O/oEo= 2023-07-29T00:35:00.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mY6S2/Iw1ehY/cbE2z4Oa3O/oEo=', 'Content-Length': '0'} 2023-07-29T00:35:00.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c00b629f56d8b1b-0064c45eb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.152 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:00.153 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:00.153 DEBUG:boto:path=/hiiqos-59/?prefix=obj68 2023-07-29T00:35:00.154 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj68 2023-07-29T00:35:00.154 DEBUG:boto:Method: GET 2023-07-29T00:35:00.154 DEBUG:boto:Path: /hiiqos-59/?prefix=obj68 2023-07-29T00:35:00.155 DEBUG:boto:Data: 2023-07-29T00:35:00.155 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.155 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:00.156 DEBUG:boto:Port: 8001 2023-07-29T00:35:00.156 DEBUG:boto:Params: {} 2023-07-29T00:35:00.156 DEBUG:boto:Token: None 2023-07-29T00:35:00.157 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/ 2023-07-29T00:35:00.157 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0N+PAhtHIs4KHe60wn+BGQJADio= 2023-07-29T00:35:00.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0N+PAhtHIs4KHe60wn+BGQJADio=', 'Content-Length': '0'} 2023-07-29T00:35:00.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d6f48b494611a2b-0064c45eb4-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.161 DEBUG:boto:b'hiiqos-59obj681000falseobj682023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:00.161 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:00.162 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:00.162 DEBUG:boto:path=/hiiqos-59/?prefix=obj68 2023-07-29T00:35:00.163 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj68 2023-07-29T00:35:00.163 DEBUG:boto:Method: GET 2023-07-29T00:35:00.163 DEBUG:boto:Path: /hiiqos-59/?prefix=obj68 2023-07-29T00:35:00.164 DEBUG:boto:Data: 2023-07-29T00:35:00.164 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.164 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:00.165 DEBUG:boto:Port: 8000 2023-07-29T00:35:00.165 DEBUG:boto:Params: {} 2023-07-29T00:35:00.166 DEBUG:boto:Token: None 2023-07-29T00:35:00.166 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/ 2023-07-29T00:35:00.167 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0N+PAhtHIs4KHe60wn+BGQJADio= 2023-07-29T00:35:00.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0N+PAhtHIs4KHe60wn+BGQJADio=', 'Content-Length': '0'} 2023-07-29T00:35:00.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f725b4653c39e57c-0064c45eb4-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.171 DEBUG:boto:b'hiiqos-59obj681000falseobj682023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:00.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-07-29T00:35:00.172 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:35:00.173 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:35:00.173 DEBUG:boto:path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:35:00.174 DEBUG:boto:auth_path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:35:00.174 DEBUG:boto:Method: GET 2023-07-29T00:35:00.174 DEBUG:boto:Path: /hiiqos-59/obj69?versionId=null 2023-07-29T00:35:00.175 DEBUG:boto:Data: 2023-07-29T00:35:00.175 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.175 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:00.176 DEBUG:boto:Port: 8001 2023-07-29T00:35:00.176 DEBUG:boto:Params: {} 2023-07-29T00:35:00.176 DEBUG:boto:Token: None 2023-07-29T00:35:00.177 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj69?versionId=null 2023-07-29T00:35:00.177 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DQ14PCtH+S86O9VjTVACxB+Cd5A= 2023-07-29T00:35:00.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DQ14PCtH+S86O9VjTVACxB+Cd5A=', 'Content-Length': '0'} 2023-07-29T00:35:00.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb2cd0e989dead2d-0064c45eb4-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.181 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:35:00.181 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:35:00.181 DEBUG:boto:path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:35:00.182 DEBUG:boto:auth_path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:35:00.182 DEBUG:boto:Method: GET 2023-07-29T00:35:00.182 DEBUG:boto:Path: /hiiqos-59/obj69?versionId=null 2023-07-29T00:35:00.183 DEBUG:boto:Data: 2023-07-29T00:35:00.183 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.183 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:00.184 DEBUG:boto:Port: 8000 2023-07-29T00:35:00.184 DEBUG:boto:Params: {} 2023-07-29T00:35:00.185 DEBUG:boto:Token: None 2023-07-29T00:35:00.185 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj69?versionId=null 2023-07-29T00:35:00.185 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DQ14PCtH+S86O9VjTVACxB+Cd5A= 2023-07-29T00:35:00.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DQ14PCtH+S86O9VjTVACxB+Cd5A=', 'Content-Length': '0'} 2023-07-29T00:35:00.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f9747850b566c55-0064c45eb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.189 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:35:00.189 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:35:00.189 DEBUG:boto:path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:35:00.190 DEBUG:boto:auth_path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:35:00.190 DEBUG:boto:Method: HEAD 2023-07-29T00:35:00.190 DEBUG:boto:Path: /hiiqos-59/obj69?versionId=null 2023-07-29T00:35:00.191 DEBUG:boto:Data: 2023-07-29T00:35:00.191 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.191 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:00.192 DEBUG:boto:Port: 8001 2023-07-29T00:35:00.192 DEBUG:boto:Params: {} 2023-07-29T00:35:00.192 DEBUG:boto:Token: None 2023-07-29T00:35:00.193 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj69?versionId=null 2023-07-29T00:35:00.193 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dse++BdAqEMaNlF6nE3860CenJo= 2023-07-29T00:35:00.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dse++BdAqEMaNlF6nE3860CenJo=', 'Content-Length': '0'} 2023-07-29T00:35:00.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f43b17e8d67688f-0064c45eb4-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.196 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:35:00.197 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:35:00.197 DEBUG:boto:path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:35:00.197 DEBUG:boto:auth_path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:35:00.198 DEBUG:boto:Method: HEAD 2023-07-29T00:35:00.198 DEBUG:boto:Path: /hiiqos-59/obj69?versionId=null 2023-07-29T00:35:00.198 DEBUG:boto:Data: 2023-07-29T00:35:00.199 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.199 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:00.199 DEBUG:boto:Port: 8000 2023-07-29T00:35:00.200 DEBUG:boto:Params: {} 2023-07-29T00:35:00.200 DEBUG:boto:Token: None 2023-07-29T00:35:00.201 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj69?versionId=null 2023-07-29T00:35:00.201 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dse++BdAqEMaNlF6nE3860CenJo= 2023-07-29T00:35:00.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dse++BdAqEMaNlF6nE3860CenJo=', 'Content-Length': '0'} 2023-07-29T00:35:00.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc8f38720b4c19ad-0064c45eb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-07-29T00:35:00.204 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:35:00.205 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:35:00.205 DEBUG:boto:path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:35:00.205 DEBUG:boto:auth_path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:35:00.206 DEBUG:boto:Method: GET 2023-07-29T00:35:00.206 DEBUG:boto:Path: /hiiqos-59/obj69?versionId=null 2023-07-29T00:35:00.206 DEBUG:boto:Data: 2023-07-29T00:35:00.207 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.207 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:00.208 DEBUG:boto:Port: 8001 2023-07-29T00:35:00.208 DEBUG:boto:Params: {} 2023-07-29T00:35:00.208 DEBUG:boto:Token: None 2023-07-29T00:35:00.209 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj69?versionId=null 2023-07-29T00:35:00.209 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DQ14PCtH+S86O9VjTVACxB+Cd5A= 2023-07-29T00:35:00.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DQ14PCtH+S86O9VjTVACxB+Cd5A=', 'Content-Length': '0'} 2023-07-29T00:35:00.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e44634bfe9661ba-0064c45eb4-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.212 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:35:00.213 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:35:00.213 DEBUG:boto:path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:35:00.213 DEBUG:boto:auth_path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:35:00.214 DEBUG:boto:Method: GET 2023-07-29T00:35:00.214 DEBUG:boto:Path: /hiiqos-59/obj69?versionId=null 2023-07-29T00:35:00.215 DEBUG:boto:Data: 2023-07-29T00:35:00.215 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.215 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:00.216 DEBUG:boto:Port: 8000 2023-07-29T00:35:00.216 DEBUG:boto:Params: {} 2023-07-29T00:35:00.216 DEBUG:boto:Token: None 2023-07-29T00:35:00.217 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj69?versionId=null 2023-07-29T00:35:00.217 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DQ14PCtH+S86O9VjTVACxB+Cd5A= 2023-07-29T00:35:00.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DQ14PCtH+S86O9VjTVACxB+Cd5A=', 'Content-Length': '0'} 2023-07-29T00:35:00.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c30a6627334ad4d2-0064c45eb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.220 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:00.221 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:00.221 DEBUG:boto:path=/hiiqos-59/?prefix=obj69 2023-07-29T00:35:00.222 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj69 2023-07-29T00:35:00.222 DEBUG:boto:Method: GET 2023-07-29T00:35:00.222 DEBUG:boto:Path: /hiiqos-59/?prefix=obj69 2023-07-29T00:35:00.223 DEBUG:boto:Data: 2023-07-29T00:35:00.223 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.223 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:00.224 DEBUG:boto:Port: 8001 2023-07-29T00:35:00.224 DEBUG:boto:Params: {} 2023-07-29T00:35:00.225 DEBUG:boto:Token: None 2023-07-29T00:35:00.225 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/ 2023-07-29T00:35:00.225 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0N+PAhtHIs4KHe60wn+BGQJADio= 2023-07-29T00:35:00.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0N+PAhtHIs4KHe60wn+BGQJADio=', 'Content-Length': '0'} 2023-07-29T00:35:00.228 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b1d315a66b349c6-0064c45eb4-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.229 DEBUG:boto:b'hiiqos-59obj691000falseobj692023-07-29T00:33:06.857Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:00.229 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:00.230 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:00.230 DEBUG:boto:path=/hiiqos-59/?prefix=obj69 2023-07-29T00:35:00.231 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj69 2023-07-29T00:35:00.231 DEBUG:boto:Method: GET 2023-07-29T00:35:00.231 DEBUG:boto:Path: /hiiqos-59/?prefix=obj69 2023-07-29T00:35:00.232 DEBUG:boto:Data: 2023-07-29T00:35:00.232 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.232 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:00.233 DEBUG:boto:Port: 8000 2023-07-29T00:35:00.233 DEBUG:boto:Params: {} 2023-07-29T00:35:00.233 DEBUG:boto:Token: None 2023-07-29T00:35:00.234 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/ 2023-07-29T00:35:00.234 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0N+PAhtHIs4KHe60wn+BGQJADio= 2023-07-29T00:35:00.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0N+PAhtHIs4KHe60wn+BGQJADio=', 'Content-Length': '0'} 2023-07-29T00:35:00.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a74c7f0d81235b32-0064c45eb4-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.238 DEBUG:boto:b'hiiqos-59obj691000falseobj692023-07-29T00:33:06.857Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:00.239 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:35:00.239 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:35:00.240 DEBUG:boto:Method: HEAD 2023-07-29T00:35:00.240 DEBUG:boto:Path: /hiiqos-59/obj69 2023-07-29T00:35:00.240 DEBUG:boto:Data: 2023-07-29T00:35:00.241 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.241 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:00.241 DEBUG:boto:Port: 8001 2023-07-29T00:35:00.242 DEBUG:boto:Params: {} 2023-07-29T00:35:00.242 DEBUG:boto:Token: None 2023-07-29T00:35:00.243 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj69 2023-07-29T00:35:00.243 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tRNp8Ay7BEeshqs+B2mGTRLQ0yM= 2023-07-29T00:35:00.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tRNp8Ay7BEeshqs+B2mGTRLQ0yM=', 'Content-Length': '0'} 2023-07-29T00:35:00.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c6d09c5c385e683-0064c45eb4-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.246 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:35:00.246 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:35:00.246 DEBUG:boto:Method: HEAD 2023-07-29T00:35:00.247 DEBUG:boto:Path: /hiiqos-59/obj69 2023-07-29T00:35:00.247 DEBUG:boto:Data: 2023-07-29T00:35:00.247 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.248 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:00.248 DEBUG:boto:Port: 8000 2023-07-29T00:35:00.248 DEBUG:boto:Params: {} 2023-07-29T00:35:00.249 DEBUG:boto:Token: None 2023-07-29T00:35:00.249 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj69 2023-07-29T00:35:00.250 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tRNp8Ay7BEeshqs+B2mGTRLQ0yM= 2023-07-29T00:35:00.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tRNp8Ay7BEeshqs+B2mGTRLQ0yM=', 'Content-Length': '0'} 2023-07-29T00:35:00.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072178d67882c93b0-0064c45eb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-07-29T00:35:00.253 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:35:00.253 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:35:00.254 DEBUG:boto:Method: GET 2023-07-29T00:35:00.254 DEBUG:boto:Path: /hiiqos-59/obj69 2023-07-29T00:35:00.255 DEBUG:boto:Data: 2023-07-29T00:35:00.255 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.255 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:00.256 DEBUG:boto:Port: 8001 2023-07-29T00:35:00.256 DEBUG:boto:Params: {} 2023-07-29T00:35:00.256 DEBUG:boto:Token: None 2023-07-29T00:35:00.257 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj69 2023-07-29T00:35:00.257 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xTtNePBxr6hcS5AuzhX6XRrwxLE= 2023-07-29T00:35:00.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xTtNePBxr6hcS5AuzhX6XRrwxLE=', 'Content-Length': '0'} 2023-07-29T00:35:00.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009436547324eda5cf-0064c45eb4-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.304 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:35:00.305 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:35:00.305 DEBUG:boto:Method: GET 2023-07-29T00:35:00.306 DEBUG:boto:Path: /hiiqos-59/obj69 2023-07-29T00:35:00.306 DEBUG:boto:Data: 2023-07-29T00:35:00.306 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.307 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:00.307 DEBUG:boto:Port: 8000 2023-07-29T00:35:00.307 DEBUG:boto:Params: {} 2023-07-29T00:35:00.308 DEBUG:boto:Token: None 2023-07-29T00:35:00.308 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj69 2023-07-29T00:35:00.309 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xTtNePBxr6hcS5AuzhX6XRrwxLE= 2023-07-29T00:35:00.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xTtNePBxr6hcS5AuzhX6XRrwxLE=', 'Content-Length': '0'} 2023-07-29T00:35:00.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042c7f09d9ded711f-0064c45eb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.312 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:00.312 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:00.313 DEBUG:boto:path=/hiiqos-59/?prefix=obj69 2023-07-29T00:35:00.313 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj69 2023-07-29T00:35:00.313 DEBUG:boto:Method: GET 2023-07-29T00:35:00.314 DEBUG:boto:Path: /hiiqos-59/?prefix=obj69 2023-07-29T00:35:00.314 DEBUG:boto:Data: 2023-07-29T00:35:00.315 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.315 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:00.315 DEBUG:boto:Port: 8001 2023-07-29T00:35:00.316 DEBUG:boto:Params: {} 2023-07-29T00:35:00.316 DEBUG:boto:Token: None 2023-07-29T00:35:00.316 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/ 2023-07-29T00:35:00.317 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0N+PAhtHIs4KHe60wn+BGQJADio= 2023-07-29T00:35:00.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0N+PAhtHIs4KHe60wn+BGQJADio=', 'Content-Length': '0'} 2023-07-29T00:35:00.320 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbe074e84ab7e4ac-0064c45eb4-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.320 DEBUG:boto:b'hiiqos-59obj691000falseobj692023-07-29T00:33:06.857Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:00.321 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:00.321 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:00.322 DEBUG:boto:path=/hiiqos-59/?prefix=obj69 2023-07-29T00:35:00.322 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj69 2023-07-29T00:35:00.323 DEBUG:boto:Method: GET 2023-07-29T00:35:00.323 DEBUG:boto:Path: /hiiqos-59/?prefix=obj69 2023-07-29T00:35:00.323 DEBUG:boto:Data: 2023-07-29T00:35:00.324 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.324 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:00.324 DEBUG:boto:Port: 8000 2023-07-29T00:35:00.325 DEBUG:boto:Params: {} 2023-07-29T00:35:00.325 DEBUG:boto:Token: None 2023-07-29T00:35:00.326 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/ 2023-07-29T00:35:00.326 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0N+PAhtHIs4KHe60wn+BGQJADio= 2023-07-29T00:35:00.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0N+PAhtHIs4KHe60wn+BGQJADio=', 'Content-Length': '0'} 2023-07-29T00:35:00.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd7e37907cee648a-0064c45eb4-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.331 DEBUG:boto:b'hiiqos-59obj691000falseobj692023-07-29T00:33:06.857Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:00.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-07-29T00:35:00.332 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:35:00.332 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:35:00.333 DEBUG:boto:path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:35:00.333 DEBUG:boto:auth_path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:35:00.333 DEBUG:boto:Method: GET 2023-07-29T00:35:00.334 DEBUG:boto:Path: /hiiqos-59/obj76?versionId=null 2023-07-29T00:35:00.334 DEBUG:boto:Data: 2023-07-29T00:35:00.335 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.335 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:00.335 DEBUG:boto:Port: 8001 2023-07-29T00:35:00.336 DEBUG:boto:Params: {} 2023-07-29T00:35:00.336 DEBUG:boto:Token: None 2023-07-29T00:35:00.336 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj76?versionId=null 2023-07-29T00:35:00.337 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bzEreD5Sd7wCVhCnatK5ZjU3W44= 2023-07-29T00:35:00.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bzEreD5Sd7wCVhCnatK5ZjU3W44=', 'Content-Length': '0'} 2023-07-29T00:35:00.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad5b0990323832a1-0064c45eb4-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.341 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:35:00.341 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:35:00.342 DEBUG:boto:path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:35:00.342 DEBUG:boto:auth_path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:35:00.343 DEBUG:boto:Method: GET 2023-07-29T00:35:00.343 DEBUG:boto:Path: /hiiqos-59/obj76?versionId=null 2023-07-29T00:35:00.343 DEBUG:boto:Data: 2023-07-29T00:35:00.344 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.344 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:00.344 DEBUG:boto:Port: 8000 2023-07-29T00:35:00.345 DEBUG:boto:Params: {} 2023-07-29T00:35:00.345 DEBUG:boto:Token: None 2023-07-29T00:35:00.346 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj76?versionId=null 2023-07-29T00:35:00.346 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bzEreD5Sd7wCVhCnatK5ZjU3W44= 2023-07-29T00:35:00.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bzEreD5Sd7wCVhCnatK5ZjU3W44=', 'Content-Length': '0'} 2023-07-29T00:35:00.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004a5c5162bc580c6-0064c45eb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.349 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:35:00.350 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:35:00.350 DEBUG:boto:path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:35:00.350 DEBUG:boto:auth_path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:35:00.351 DEBUG:boto:Method: HEAD 2023-07-29T00:35:00.351 DEBUG:boto:Path: /hiiqos-59/obj76?versionId=null 2023-07-29T00:35:00.352 DEBUG:boto:Data: 2023-07-29T00:35:00.352 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.352 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:00.353 DEBUG:boto:Port: 8001 2023-07-29T00:35:00.353 DEBUG:boto:Params: {} 2023-07-29T00:35:00.353 DEBUG:boto:Token: None 2023-07-29T00:35:00.354 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj76?versionId=null 2023-07-29T00:35:00.354 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QZTpl6EIuzBj5kjFUjzw4Q0szzg= 2023-07-29T00:35:00.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QZTpl6EIuzBj5kjFUjzw4Q0szzg=', 'Content-Length': '0'} 2023-07-29T00:35:00.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2470d4c645ac1ab-0064c45eb4-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.357 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:35:00.357 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:35:00.358 DEBUG:boto:path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:35:00.358 DEBUG:boto:auth_path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:35:00.358 DEBUG:boto:Method: HEAD 2023-07-29T00:35:00.359 DEBUG:boto:Path: /hiiqos-59/obj76?versionId=null 2023-07-29T00:35:00.359 DEBUG:boto:Data: 2023-07-29T00:35:00.359 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.360 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:00.360 DEBUG:boto:Port: 8000 2023-07-29T00:35:00.361 DEBUG:boto:Params: {} 2023-07-29T00:35:00.361 DEBUG:boto:Token: None 2023-07-29T00:35:00.361 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj76?versionId=null 2023-07-29T00:35:00.362 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QZTpl6EIuzBj5kjFUjzw4Q0szzg= 2023-07-29T00:35:00.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QZTpl6EIuzBj5kjFUjzw4Q0szzg=', 'Content-Length': '0'} 2023-07-29T00:35:00.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cec213ec944f2c6e-0064c45eb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-07-29T00:35:00.365 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:35:00.366 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:35:00.366 DEBUG:boto:path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:35:00.366 DEBUG:boto:auth_path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:35:00.367 DEBUG:boto:Method: GET 2023-07-29T00:35:00.367 DEBUG:boto:Path: /hiiqos-59/obj76?versionId=null 2023-07-29T00:35:00.368 DEBUG:boto:Data: 2023-07-29T00:35:00.368 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.368 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:00.369 DEBUG:boto:Port: 8001 2023-07-29T00:35:00.369 DEBUG:boto:Params: {} 2023-07-29T00:35:00.370 DEBUG:boto:Token: None 2023-07-29T00:35:00.370 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj76?versionId=null 2023-07-29T00:35:00.371 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bzEreD5Sd7wCVhCnatK5ZjU3W44= 2023-07-29T00:35:00.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bzEreD5Sd7wCVhCnatK5ZjU3W44=', 'Content-Length': '0'} 2023-07-29T00:35:00.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e467caa522b6bf1d-0064c45eb4-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.374 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:35:00.374 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:35:00.375 DEBUG:boto:path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:35:00.375 DEBUG:boto:auth_path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:35:00.375 DEBUG:boto:Method: GET 2023-07-29T00:35:00.376 DEBUG:boto:Path: /hiiqos-59/obj76?versionId=null 2023-07-29T00:35:00.376 DEBUG:boto:Data: 2023-07-29T00:35:00.376 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.377 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:00.377 DEBUG:boto:Port: 8000 2023-07-29T00:35:00.378 DEBUG:boto:Params: {} 2023-07-29T00:35:00.378 DEBUG:boto:Token: None 2023-07-29T00:35:00.378 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj76?versionId=null 2023-07-29T00:35:00.379 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bzEreD5Sd7wCVhCnatK5ZjU3W44= 2023-07-29T00:35:00.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bzEreD5Sd7wCVhCnatK5ZjU3W44=', 'Content-Length': '0'} 2023-07-29T00:35:00.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089886cd71123d3c3-0064c45eb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.382 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:00.382 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:00.383 DEBUG:boto:path=/hiiqos-59/?prefix=obj76 2023-07-29T00:35:00.383 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj76 2023-07-29T00:35:00.383 DEBUG:boto:Method: GET 2023-07-29T00:35:00.384 DEBUG:boto:Path: /hiiqos-59/?prefix=obj76 2023-07-29T00:35:00.384 DEBUG:boto:Data: 2023-07-29T00:35:00.385 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.385 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:00.385 DEBUG:boto:Port: 8001 2023-07-29T00:35:00.386 DEBUG:boto:Params: {} 2023-07-29T00:35:00.386 DEBUG:boto:Token: None 2023-07-29T00:35:00.386 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/ 2023-07-29T00:35:00.387 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0N+PAhtHIs4KHe60wn+BGQJADio= 2023-07-29T00:35:00.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0N+PAhtHIs4KHe60wn+BGQJADio=', 'Content-Length': '0'} 2023-07-29T00:35:00.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d0df9ee2cefba4e-0064c45eb4-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.432 DEBUG:boto:b'hiiqos-59obj761000falseobj762023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:00.433 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:00.434 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:00.434 DEBUG:boto:path=/hiiqos-59/?prefix=obj76 2023-07-29T00:35:00.434 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj76 2023-07-29T00:35:00.435 DEBUG:boto:Method: GET 2023-07-29T00:35:00.435 DEBUG:boto:Path: /hiiqos-59/?prefix=obj76 2023-07-29T00:35:00.436 DEBUG:boto:Data: 2023-07-29T00:35:00.436 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.436 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:00.437 DEBUG:boto:Port: 8000 2023-07-29T00:35:00.437 DEBUG:boto:Params: {} 2023-07-29T00:35:00.438 DEBUG:boto:Token: None 2023-07-29T00:35:00.438 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/ 2023-07-29T00:35:00.438 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0N+PAhtHIs4KHe60wn+BGQJADio= 2023-07-29T00:35:00.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0N+PAhtHIs4KHe60wn+BGQJADio=', 'Content-Length': '0'} 2023-07-29T00:35:00.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f48933a3651964ba-0064c45eb4-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.443 DEBUG:boto:b'hiiqos-59obj761000falseobj762023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:00.444 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:35:00.444 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:35:00.445 DEBUG:boto:Method: HEAD 2023-07-29T00:35:00.445 DEBUG:boto:Path: /hiiqos-59/obj76 2023-07-29T00:35:00.445 DEBUG:boto:Data: 2023-07-29T00:35:00.446 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.447 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:00.448 DEBUG:boto:Port: 8001 2023-07-29T00:35:00.449 DEBUG:boto:Params: {} 2023-07-29T00:35:00.450 DEBUG:boto:Token: None 2023-07-29T00:35:00.450 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj76 2023-07-29T00:35:00.451 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:NVT0yQacEZCA85JBvAqqDtjP3Do= 2023-07-29T00:35:00.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:NVT0yQacEZCA85JBvAqqDtjP3Do=', 'Content-Length': '0'} 2023-07-29T00:35:00.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7340a9216768d4f-0064c45eb4-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.456 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:35:00.456 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:35:00.457 DEBUG:boto:Method: HEAD 2023-07-29T00:35:00.457 DEBUG:boto:Path: /hiiqos-59/obj76 2023-07-29T00:35:00.458 DEBUG:boto:Data: 2023-07-29T00:35:00.458 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.458 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:00.459 DEBUG:boto:Port: 8000 2023-07-29T00:35:00.459 DEBUG:boto:Params: {} 2023-07-29T00:35:00.459 DEBUG:boto:Token: None 2023-07-29T00:35:00.460 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj76 2023-07-29T00:35:00.460 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:NVT0yQacEZCA85JBvAqqDtjP3Do= 2023-07-29T00:35:00.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:NVT0yQacEZCA85JBvAqqDtjP3Do=', 'Content-Length': '0'} 2023-07-29T00:35:00.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003735eb888d1daae2-0064c45eb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-07-29T00:35:00.463 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:35:00.464 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:35:00.464 DEBUG:boto:Method: GET 2023-07-29T00:35:00.465 DEBUG:boto:Path: /hiiqos-59/obj76 2023-07-29T00:35:00.465 DEBUG:boto:Data: 2023-07-29T00:35:00.465 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.466 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:00.466 DEBUG:boto:Port: 8001 2023-07-29T00:35:00.466 DEBUG:boto:Params: {} 2023-07-29T00:35:00.467 DEBUG:boto:Token: None 2023-07-29T00:35:00.467 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj76 2023-07-29T00:35:00.468 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1RAZ6JtOHeuf+HiM4oBdH/b/Ex8= 2023-07-29T00:35:00.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1RAZ6JtOHeuf+HiM4oBdH/b/Ex8=', 'Content-Length': '0'} 2023-07-29T00:35:00.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000244e2fdd0118956b-0064c45eb4-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.471 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:35:00.471 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:35:00.472 DEBUG:boto:Method: GET 2023-07-29T00:35:00.472 DEBUG:boto:Path: /hiiqos-59/obj76 2023-07-29T00:35:00.473 DEBUG:boto:Data: 2023-07-29T00:35:00.473 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.473 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:00.474 DEBUG:boto:Port: 8000 2023-07-29T00:35:00.474 DEBUG:boto:Params: {} 2023-07-29T00:35:00.475 DEBUG:boto:Token: None 2023-07-29T00:35:00.475 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj76 2023-07-29T00:35:00.476 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1RAZ6JtOHeuf+HiM4oBdH/b/Ex8= 2023-07-29T00:35:00.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1RAZ6JtOHeuf+HiM4oBdH/b/Ex8=', 'Content-Length': '0'} 2023-07-29T00:35:00.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029beaf1a24b999a1-0064c45eb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.520 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:00.521 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:00.521 DEBUG:boto:path=/hiiqos-59/?prefix=obj76 2023-07-29T00:35:00.522 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj76 2023-07-29T00:35:00.522 DEBUG:boto:Method: GET 2023-07-29T00:35:00.522 DEBUG:boto:Path: /hiiqos-59/?prefix=obj76 2023-07-29T00:35:00.523 DEBUG:boto:Data: 2023-07-29T00:35:00.523 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.523 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:00.524 DEBUG:boto:Port: 8001 2023-07-29T00:35:00.524 DEBUG:boto:Params: {} 2023-07-29T00:35:00.525 DEBUG:boto:Token: None 2023-07-29T00:35:00.525 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/ 2023-07-29T00:35:00.526 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0N+PAhtHIs4KHe60wn+BGQJADio= 2023-07-29T00:35:00.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0N+PAhtHIs4KHe60wn+BGQJADio=', 'Content-Length': '0'} 2023-07-29T00:35:00.529 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a59ab41008def3ae-0064c45eb4-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.530 DEBUG:boto:b'hiiqos-59obj761000falseobj762023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:00.531 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:00.532 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:00.533 DEBUG:boto:path=/hiiqos-59/?prefix=obj76 2023-07-29T00:35:00.534 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj76 2023-07-29T00:35:00.535 DEBUG:boto:Method: GET 2023-07-29T00:35:00.536 DEBUG:boto:Path: /hiiqos-59/?prefix=obj76 2023-07-29T00:35:00.537 DEBUG:boto:Data: 2023-07-29T00:35:00.537 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.538 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:00.539 DEBUG:boto:Port: 8000 2023-07-29T00:35:00.540 DEBUG:boto:Params: {} 2023-07-29T00:35:00.540 DEBUG:boto:Token: None 2023-07-29T00:35:00.541 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/ 2023-07-29T00:35:00.542 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0N+PAhtHIs4KHe60wn+BGQJADio= 2023-07-29T00:35:00.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0N+PAhtHIs4KHe60wn+BGQJADio=', 'Content-Length': '0'} 2023-07-29T00:35:00.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c5f88f856b68926-0064c45eb4-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.546 DEBUG:boto:b'hiiqos-59obj761000falseobj762023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:00.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-07-29T00:35:00.547 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:35:00.548 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:35:00.548 DEBUG:boto:path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:35:00.548 DEBUG:boto:auth_path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:35:00.549 DEBUG:boto:Method: GET 2023-07-29T00:35:00.549 DEBUG:boto:Path: /hiiqos-59/obj85?versionId=null 2023-07-29T00:35:00.549 DEBUG:boto:Data: 2023-07-29T00:35:00.550 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.550 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:00.551 DEBUG:boto:Port: 8001 2023-07-29T00:35:00.551 DEBUG:boto:Params: {} 2023-07-29T00:35:00.551 DEBUG:boto:Token: None 2023-07-29T00:35:00.552 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj85?versionId=null 2023-07-29T00:35:00.552 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:q9cAK0HrTandY85rduPBeCT3mdg= 2023-07-29T00:35:00.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:q9cAK0HrTandY85rduPBeCT3mdg=', 'Content-Length': '0'} 2023-07-29T00:35:00.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059925e044a6cdd46-0064c45eb4-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.555 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:35:00.556 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:35:00.556 DEBUG:boto:path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:35:00.557 DEBUG:boto:auth_path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:35:00.557 DEBUG:boto:Method: GET 2023-07-29T00:35:00.557 DEBUG:boto:Path: /hiiqos-59/obj85?versionId=null 2023-07-29T00:35:00.558 DEBUG:boto:Data: 2023-07-29T00:35:00.558 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.558 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:00.559 DEBUG:boto:Port: 8000 2023-07-29T00:35:00.559 DEBUG:boto:Params: {} 2023-07-29T00:35:00.560 DEBUG:boto:Token: None 2023-07-29T00:35:00.560 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj85?versionId=null 2023-07-29T00:35:00.561 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:q9cAK0HrTandY85rduPBeCT3mdg= 2023-07-29T00:35:00.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:q9cAK0HrTandY85rduPBeCT3mdg=', 'Content-Length': '0'} 2023-07-29T00:35:00.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5f578a1b5b28ca9-0064c45eb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.563 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:35:00.564 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:35:00.564 DEBUG:boto:path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:35:00.565 DEBUG:boto:auth_path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:35:00.565 DEBUG:boto:Method: HEAD 2023-07-29T00:35:00.565 DEBUG:boto:Path: /hiiqos-59/obj85?versionId=null 2023-07-29T00:35:00.566 DEBUG:boto:Data: 2023-07-29T00:35:00.566 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.567 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:00.567 DEBUG:boto:Port: 8001 2023-07-29T00:35:00.567 DEBUG:boto:Params: {} 2023-07-29T00:35:00.568 DEBUG:boto:Token: None 2023-07-29T00:35:00.568 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj85?versionId=null 2023-07-29T00:35:00.569 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MGLhBEk9GIW+mzpk53QvFD+DGX0= 2023-07-29T00:35:00.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MGLhBEk9GIW+mzpk53QvFD+DGX0=', 'Content-Length': '0'} 2023-07-29T00:35:00.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fa8bafcbe7648ba-0064c45eb4-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.572 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:35:00.572 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:35:00.573 DEBUG:boto:path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:35:00.573 DEBUG:boto:auth_path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:35:00.574 DEBUG:boto:Method: HEAD 2023-07-29T00:35:00.574 DEBUG:boto:Path: /hiiqos-59/obj85?versionId=null 2023-07-29T00:35:00.574 DEBUG:boto:Data: 2023-07-29T00:35:00.575 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.575 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:00.575 DEBUG:boto:Port: 8000 2023-07-29T00:35:00.576 DEBUG:boto:Params: {} 2023-07-29T00:35:00.576 DEBUG:boto:Token: None 2023-07-29T00:35:00.576 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj85?versionId=null 2023-07-29T00:35:00.577 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MGLhBEk9GIW+mzpk53QvFD+DGX0= 2023-07-29T00:35:00.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MGLhBEk9GIW+mzpk53QvFD+DGX0=', 'Content-Length': '0'} 2023-07-29T00:35:00.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000453e0a75434a3270-0064c45eb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-07-29T00:35:00.580 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:35:00.581 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:35:00.581 DEBUG:boto:path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:35:00.582 DEBUG:boto:auth_path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:35:00.582 DEBUG:boto:Method: GET 2023-07-29T00:35:00.583 DEBUG:boto:Path: /hiiqos-59/obj85?versionId=null 2023-07-29T00:35:00.583 DEBUG:boto:Data: 2023-07-29T00:35:00.583 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.584 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:00.584 DEBUG:boto:Port: 8001 2023-07-29T00:35:00.584 DEBUG:boto:Params: {} 2023-07-29T00:35:00.585 DEBUG:boto:Token: None 2023-07-29T00:35:00.585 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj85?versionId=null 2023-07-29T00:35:00.586 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:q9cAK0HrTandY85rduPBeCT3mdg= 2023-07-29T00:35:00.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:q9cAK0HrTandY85rduPBeCT3mdg=', 'Content-Length': '0'} 2023-07-29T00:35:00.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ac3d96574dc24f1-0064c45eb4-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.589 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:35:00.589 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:35:00.590 DEBUG:boto:path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:35:00.590 DEBUG:boto:auth_path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:35:00.590 DEBUG:boto:Method: GET 2023-07-29T00:35:00.591 DEBUG:boto:Path: /hiiqos-59/obj85?versionId=null 2023-07-29T00:35:00.591 DEBUG:boto:Data: 2023-07-29T00:35:00.591 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.592 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:00.592 DEBUG:boto:Port: 8000 2023-07-29T00:35:00.593 DEBUG:boto:Params: {} 2023-07-29T00:35:00.593 DEBUG:boto:Token: None 2023-07-29T00:35:00.594 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj85?versionId=null 2023-07-29T00:35:00.594 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:q9cAK0HrTandY85rduPBeCT3mdg= 2023-07-29T00:35:00.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:q9cAK0HrTandY85rduPBeCT3mdg=', 'Content-Length': '0'} 2023-07-29T00:35:00.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee3b5837bb4fad0a-0064c45eb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.597 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:00.597 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:00.598 DEBUG:boto:path=/hiiqos-59/?prefix=obj85 2023-07-29T00:35:00.598 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj85 2023-07-29T00:35:00.599 DEBUG:boto:Method: GET 2023-07-29T00:35:00.599 DEBUG:boto:Path: /hiiqos-59/?prefix=obj85 2023-07-29T00:35:00.599 DEBUG:boto:Data: 2023-07-29T00:35:00.600 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.600 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:00.601 DEBUG:boto:Port: 8001 2023-07-29T00:35:00.601 DEBUG:boto:Params: {} 2023-07-29T00:35:00.602 DEBUG:boto:Token: None 2023-07-29T00:35:00.602 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/ 2023-07-29T00:35:00.603 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0N+PAhtHIs4KHe60wn+BGQJADio= 2023-07-29T00:35:00.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0N+PAhtHIs4KHe60wn+BGQJADio=', 'Content-Length': '0'} 2023-07-29T00:35:00.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcd783eaded031d2-0064c45eb4-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.648 DEBUG:boto:b'hiiqos-59obj851000falseobj852023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:00.650 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:00.650 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:00.651 DEBUG:boto:path=/hiiqos-59/?prefix=obj85 2023-07-29T00:35:00.652 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj85 2023-07-29T00:35:00.653 DEBUG:boto:Method: GET 2023-07-29T00:35:00.653 DEBUG:boto:Path: /hiiqos-59/?prefix=obj85 2023-07-29T00:35:00.654 DEBUG:boto:Data: 2023-07-29T00:35:00.654 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.655 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:00.656 DEBUG:boto:Port: 8000 2023-07-29T00:35:00.656 DEBUG:boto:Params: {} 2023-07-29T00:35:00.657 DEBUG:boto:Token: None 2023-07-29T00:35:00.658 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/ 2023-07-29T00:35:00.659 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0N+PAhtHIs4KHe60wn+BGQJADio= 2023-07-29T00:35:00.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0N+PAhtHIs4KHe60wn+BGQJADio=', 'Content-Length': '0'} 2023-07-29T00:35:00.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b5d1e0a4f4bd551-0064c45eb4-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.666 DEBUG:boto:b'hiiqos-59obj851000falseobj852023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:00.667 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:35:00.667 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:35:00.668 DEBUG:boto:Method: HEAD 2023-07-29T00:35:00.668 DEBUG:boto:Path: /hiiqos-59/obj85 2023-07-29T00:35:00.668 DEBUG:boto:Data: 2023-07-29T00:35:00.669 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.669 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:00.670 DEBUG:boto:Port: 8001 2023-07-29T00:35:00.670 DEBUG:boto:Params: {} 2023-07-29T00:35:00.671 DEBUG:boto:Token: None 2023-07-29T00:35:00.671 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj85 2023-07-29T00:35:00.671 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QWRb0uPFB2b2ipkLKKOCqx870No= 2023-07-29T00:35:00.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QWRb0uPFB2b2ipkLKKOCqx870No=', 'Content-Length': '0'} 2023-07-29T00:35:00.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e53bdadbe46560f-0064c45eb4-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.674 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:35:00.675 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:35:00.675 DEBUG:boto:Method: HEAD 2023-07-29T00:35:00.675 DEBUG:boto:Path: /hiiqos-59/obj85 2023-07-29T00:35:00.676 DEBUG:boto:Data: 2023-07-29T00:35:00.676 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.677 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:00.677 DEBUG:boto:Port: 8000 2023-07-29T00:35:00.678 DEBUG:boto:Params: {} 2023-07-29T00:35:00.678 DEBUG:boto:Token: None 2023-07-29T00:35:00.679 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj85 2023-07-29T00:35:00.679 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QWRb0uPFB2b2ipkLKKOCqx870No= 2023-07-29T00:35:00.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QWRb0uPFB2b2ipkLKKOCqx870No=', 'Content-Length': '0'} 2023-07-29T00:35:00.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa0b93c413f8be44-0064c45eb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-07-29T00:35:00.682 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:35:00.682 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:35:00.683 DEBUG:boto:Method: GET 2023-07-29T00:35:00.683 DEBUG:boto:Path: /hiiqos-59/obj85 2023-07-29T00:35:00.684 DEBUG:boto:Data: 2023-07-29T00:35:00.684 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.684 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:00.685 DEBUG:boto:Port: 8001 2023-07-29T00:35:00.685 DEBUG:boto:Params: {} 2023-07-29T00:35:00.685 DEBUG:boto:Token: None 2023-07-29T00:35:00.686 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj85 2023-07-29T00:35:00.686 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uCn401DguK98/QsOCeArLV5MXFE= 2023-07-29T00:35:00.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uCn401DguK98/QsOCeArLV5MXFE=', 'Content-Length': '0'} 2023-07-29T00:35:00.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db01fa7751c0fe2d-0064c45eb4-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.689 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:35:00.690 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:35:00.690 DEBUG:boto:Method: GET 2023-07-29T00:35:00.691 DEBUG:boto:Path: /hiiqos-59/obj85 2023-07-29T00:35:00.691 DEBUG:boto:Data: 2023-07-29T00:35:00.691 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.692 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:00.692 DEBUG:boto:Port: 8000 2023-07-29T00:35:00.693 DEBUG:boto:Params: {} 2023-07-29T00:35:00.693 DEBUG:boto:Token: None 2023-07-29T00:35:00.694 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj85 2023-07-29T00:35:00.694 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uCn401DguK98/QsOCeArLV5MXFE= 2023-07-29T00:35:00.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uCn401DguK98/QsOCeArLV5MXFE=', 'Content-Length': '0'} 2023-07-29T00:35:00.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc2077f57ec92237-0064c45eb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.698 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:00.698 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:00.699 DEBUG:boto:path=/hiiqos-59/?prefix=obj85 2023-07-29T00:35:00.699 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj85 2023-07-29T00:35:00.700 DEBUG:boto:Method: GET 2023-07-29T00:35:00.700 DEBUG:boto:Path: /hiiqos-59/?prefix=obj85 2023-07-29T00:35:00.701 DEBUG:boto:Data: 2023-07-29T00:35:00.701 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.702 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:00.702 DEBUG:boto:Port: 8001 2023-07-29T00:35:00.702 DEBUG:boto:Params: {} 2023-07-29T00:35:00.703 DEBUG:boto:Token: None 2023-07-29T00:35:00.703 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/ 2023-07-29T00:35:00.703 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0N+PAhtHIs4KHe60wn+BGQJADio= 2023-07-29T00:35:00.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0N+PAhtHIs4KHe60wn+BGQJADio=', 'Content-Length': '0'} 2023-07-29T00:35:00.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029b9d2a0a67fb1fd-0064c45eb4-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.707 DEBUG:boto:b'hiiqos-59obj851000falseobj852023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:00.708 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:00.708 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:00.709 DEBUG:boto:path=/hiiqos-59/?prefix=obj85 2023-07-29T00:35:00.709 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj85 2023-07-29T00:35:00.709 DEBUG:boto:Method: GET 2023-07-29T00:35:00.710 DEBUG:boto:Path: /hiiqos-59/?prefix=obj85 2023-07-29T00:35:00.710 DEBUG:boto:Data: 2023-07-29T00:35:00.711 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.711 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:00.711 DEBUG:boto:Port: 8000 2023-07-29T00:35:00.712 DEBUG:boto:Params: {} 2023-07-29T00:35:00.713 DEBUG:boto:Token: None 2023-07-29T00:35:00.714 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/ 2023-07-29T00:35:00.715 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0N+PAhtHIs4KHe60wn+BGQJADio= 2023-07-29T00:35:00.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0N+PAhtHIs4KHe60wn+BGQJADio=', 'Content-Length': '0'} 2023-07-29T00:35:00.720 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029d86fdc59ced975-0064c45eb4-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.760 DEBUG:boto:b'hiiqos-59obj851000falseobj852023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:00.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-07-29T00:35:00.762 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:35:00.763 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:35:00.763 DEBUG:boto:path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:35:00.763 DEBUG:boto:auth_path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:35:00.764 DEBUG:boto:Method: GET 2023-07-29T00:35:00.764 DEBUG:boto:Path: /hiiqos-59/obj92?versionId=null 2023-07-29T00:35:00.765 DEBUG:boto:Data: 2023-07-29T00:35:00.765 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.766 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:00.766 DEBUG:boto:Port: 8001 2023-07-29T00:35:00.766 DEBUG:boto:Params: {} 2023-07-29T00:35:00.767 DEBUG:boto:Token: None 2023-07-29T00:35:00.767 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj92?versionId=null 2023-07-29T00:35:00.768 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cJnghs8cggrm7N/2AFVcwLBGMKc= 2023-07-29T00:35:00.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cJnghs8cggrm7N/2AFVcwLBGMKc=', 'Content-Length': '0'} 2023-07-29T00:35:00.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da9ee45880cabbd1-0064c45eb4-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.772 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:35:00.772 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:35:00.773 DEBUG:boto:path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:35:00.773 DEBUG:boto:auth_path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:35:00.773 DEBUG:boto:Method: GET 2023-07-29T00:35:00.774 DEBUG:boto:Path: /hiiqos-59/obj92?versionId=null 2023-07-29T00:35:00.774 DEBUG:boto:Data: 2023-07-29T00:35:00.774 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.775 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:00.775 DEBUG:boto:Port: 8000 2023-07-29T00:35:00.775 DEBUG:boto:Params: {} 2023-07-29T00:35:00.776 DEBUG:boto:Token: None 2023-07-29T00:35:00.776 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj92?versionId=null 2023-07-29T00:35:00.777 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cJnghs8cggrm7N/2AFVcwLBGMKc= 2023-07-29T00:35:00.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cJnghs8cggrm7N/2AFVcwLBGMKc=', 'Content-Length': '0'} 2023-07-29T00:35:00.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1f01e830f0942b2-0064c45eb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.780 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:35:00.780 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:35:00.781 DEBUG:boto:path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:35:00.781 DEBUG:boto:auth_path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:35:00.781 DEBUG:boto:Method: HEAD 2023-07-29T00:35:00.782 DEBUG:boto:Path: /hiiqos-59/obj92?versionId=null 2023-07-29T00:35:00.782 DEBUG:boto:Data: 2023-07-29T00:35:00.782 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.783 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:00.783 DEBUG:boto:Port: 8001 2023-07-29T00:35:00.784 DEBUG:boto:Params: {} 2023-07-29T00:35:00.784 DEBUG:boto:Token: None 2023-07-29T00:35:00.785 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj92?versionId=null 2023-07-29T00:35:00.785 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cvjzGx/rRn2V2Df2cqNM87KWa1Y= 2023-07-29T00:35:00.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cvjzGx/rRn2V2Df2cqNM87KWa1Y=', 'Content-Length': '0'} 2023-07-29T00:35:00.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da8d2f4b6c06befd-0064c45eb4-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.788 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:35:00.789 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:35:00.789 DEBUG:boto:path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:35:00.790 DEBUG:boto:auth_path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:35:00.790 DEBUG:boto:Method: HEAD 2023-07-29T00:35:00.791 DEBUG:boto:Path: /hiiqos-59/obj92?versionId=null 2023-07-29T00:35:00.791 DEBUG:boto:Data: 2023-07-29T00:35:00.791 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.792 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:00.792 DEBUG:boto:Port: 8000 2023-07-29T00:35:00.793 DEBUG:boto:Params: {} 2023-07-29T00:35:00.793 DEBUG:boto:Token: None 2023-07-29T00:35:00.794 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj92?versionId=null 2023-07-29T00:35:00.794 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cvjzGx/rRn2V2Df2cqNM87KWa1Y= 2023-07-29T00:35:00.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cvjzGx/rRn2V2Df2cqNM87KWa1Y=', 'Content-Length': '0'} 2023-07-29T00:35:00.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009674ec9f52e46a7a-0064c45eb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-07-29T00:35:00.797 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:35:00.798 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:35:00.798 DEBUG:boto:path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:35:00.798 DEBUG:boto:auth_path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:35:00.799 DEBUG:boto:Method: GET 2023-07-29T00:35:00.799 DEBUG:boto:Path: /hiiqos-59/obj92?versionId=null 2023-07-29T00:35:00.799 DEBUG:boto:Data: 2023-07-29T00:35:00.800 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.800 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:00.801 DEBUG:boto:Port: 8001 2023-07-29T00:35:00.801 DEBUG:boto:Params: {} 2023-07-29T00:35:00.801 DEBUG:boto:Token: None 2023-07-29T00:35:00.802 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj92?versionId=null 2023-07-29T00:35:00.802 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cJnghs8cggrm7N/2AFVcwLBGMKc= 2023-07-29T00:35:00.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cJnghs8cggrm7N/2AFVcwLBGMKc=', 'Content-Length': '0'} 2023-07-29T00:35:00.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b48157f25d680ee-0064c45eb4-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.805 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:35:00.805 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:35:00.806 DEBUG:boto:path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:35:00.806 DEBUG:boto:auth_path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:35:00.806 DEBUG:boto:Method: GET 2023-07-29T00:35:00.807 DEBUG:boto:Path: /hiiqos-59/obj92?versionId=null 2023-07-29T00:35:00.807 DEBUG:boto:Data: 2023-07-29T00:35:00.808 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.808 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:00.808 DEBUG:boto:Port: 8000 2023-07-29T00:35:00.809 DEBUG:boto:Params: {} 2023-07-29T00:35:00.809 DEBUG:boto:Token: None 2023-07-29T00:35:00.809 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj92?versionId=null 2023-07-29T00:35:00.810 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cJnghs8cggrm7N/2AFVcwLBGMKc= 2023-07-29T00:35:00.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cJnghs8cggrm7N/2AFVcwLBGMKc=', 'Content-Length': '0'} 2023-07-29T00:35:00.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059e49af477614afa-0064c45eb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.813 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:00.814 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:00.814 DEBUG:boto:path=/hiiqos-59/?prefix=obj92 2023-07-29T00:35:00.814 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj92 2023-07-29T00:35:00.815 DEBUG:boto:Method: GET 2023-07-29T00:35:00.815 DEBUG:boto:Path: /hiiqos-59/?prefix=obj92 2023-07-29T00:35:00.816 DEBUG:boto:Data: 2023-07-29T00:35:00.816 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.817 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:00.817 DEBUG:boto:Port: 8001 2023-07-29T00:35:00.817 DEBUG:boto:Params: {} 2023-07-29T00:35:00.818 DEBUG:boto:Token: None 2023-07-29T00:35:00.818 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/ 2023-07-29T00:35:00.819 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0N+PAhtHIs4KHe60wn+BGQJADio= 2023-07-29T00:35:00.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0N+PAhtHIs4KHe60wn+BGQJADio=', 'Content-Length': '0'} 2023-07-29T00:35:00.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f16883444380747-0064c45eb4-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.864 DEBUG:boto:b'hiiqos-59obj921000falseobj922023-07-29T00:33:06.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:00.865 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:00.865 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:00.866 DEBUG:boto:path=/hiiqos-59/?prefix=obj92 2023-07-29T00:35:00.866 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj92 2023-07-29T00:35:00.867 DEBUG:boto:Method: GET 2023-07-29T00:35:00.867 DEBUG:boto:Path: /hiiqos-59/?prefix=obj92 2023-07-29T00:35:00.867 DEBUG:boto:Data: 2023-07-29T00:35:00.868 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.868 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:00.868 DEBUG:boto:Port: 8000 2023-07-29T00:35:00.869 DEBUG:boto:Params: {} 2023-07-29T00:35:00.869 DEBUG:boto:Token: None 2023-07-29T00:35:00.870 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/ 2023-07-29T00:35:00.870 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0N+PAhtHIs4KHe60wn+BGQJADio= 2023-07-29T00:35:00.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0N+PAhtHIs4KHe60wn+BGQJADio=', 'Content-Length': '0'} 2023-07-29T00:35:00.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e491332e5f2d11e1-0064c45eb4-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.875 DEBUG:boto:b'hiiqos-59obj921000falseobj922023-07-29T00:33:06.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:00.875 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:35:00.876 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:35:00.876 DEBUG:boto:Method: HEAD 2023-07-29T00:35:00.877 DEBUG:boto:Path: /hiiqos-59/obj92 2023-07-29T00:35:00.877 DEBUG:boto:Data: 2023-07-29T00:35:00.877 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.878 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:00.878 DEBUG:boto:Port: 8001 2023-07-29T00:35:00.879 DEBUG:boto:Params: {} 2023-07-29T00:35:00.879 DEBUG:boto:Token: None 2023-07-29T00:35:00.880 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj92 2023-07-29T00:35:00.880 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:v9fJR6n7D5UrJ4nG8amDh5QsbwI= 2023-07-29T00:35:00.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:v9fJR6n7D5UrJ4nG8amDh5QsbwI=', 'Content-Length': '0'} 2023-07-29T00:35:00.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008179b4a0b49ea83e-0064c45eb4-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.883 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:35:00.884 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:35:00.884 DEBUG:boto:Method: HEAD 2023-07-29T00:35:00.885 DEBUG:boto:Path: /hiiqos-59/obj92 2023-07-29T00:35:00.885 DEBUG:boto:Data: 2023-07-29T00:35:00.885 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.886 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:00.886 DEBUG:boto:Port: 8000 2023-07-29T00:35:00.886 DEBUG:boto:Params: {} 2023-07-29T00:35:00.887 DEBUG:boto:Token: None 2023-07-29T00:35:00.887 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj92 2023-07-29T00:35:00.887 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:v9fJR6n7D5UrJ4nG8amDh5QsbwI= 2023-07-29T00:35:00.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:v9fJR6n7D5UrJ4nG8amDh5QsbwI=', 'Content-Length': '0'} 2023-07-29T00:35:00.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bf4ecf44a0fb5a1-0064c45eb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-07-29T00:35:00.891 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:35:00.891 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:35:00.891 DEBUG:boto:Method: GET 2023-07-29T00:35:00.892 DEBUG:boto:Path: /hiiqos-59/obj92 2023-07-29T00:35:00.892 DEBUG:boto:Data: 2023-07-29T00:35:00.893 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.893 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:00.894 DEBUG:boto:Port: 8001 2023-07-29T00:35:00.894 DEBUG:boto:Params: {} 2023-07-29T00:35:00.895 DEBUG:boto:Token: None 2023-07-29T00:35:00.895 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj92 2023-07-29T00:35:00.896 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5wLuYUaBH7tb/9hnljAjfi6gxpk= 2023-07-29T00:35:00.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5wLuYUaBH7tb/9hnljAjfi6gxpk=', 'Content-Length': '0'} 2023-07-29T00:35:00.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000478d577a82a048e6-0064c45eb4-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.898 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:35:00.899 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:35:00.899 DEBUG:boto:Method: GET 2023-07-29T00:35:00.900 DEBUG:boto:Path: /hiiqos-59/obj92 2023-07-29T00:35:00.900 DEBUG:boto:Data: 2023-07-29T00:35:00.900 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.901 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:00.901 DEBUG:boto:Port: 8000 2023-07-29T00:35:00.902 DEBUG:boto:Params: {} 2023-07-29T00:35:00.902 DEBUG:boto:Token: None 2023-07-29T00:35:00.903 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj92 2023-07-29T00:35:00.903 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5wLuYUaBH7tb/9hnljAjfi6gxpk= 2023-07-29T00:35:00.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5wLuYUaBH7tb/9hnljAjfi6gxpk=', 'Content-Length': '0'} 2023-07-29T00:35:00.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d119bab31bfcf7b6-0064c45eb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.948 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:00.949 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:00.949 DEBUG:boto:path=/hiiqos-59/?prefix=obj92 2023-07-29T00:35:00.950 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj92 2023-07-29T00:35:00.950 DEBUG:boto:Method: GET 2023-07-29T00:35:00.950 DEBUG:boto:Path: /hiiqos-59/?prefix=obj92 2023-07-29T00:35:00.951 DEBUG:boto:Data: 2023-07-29T00:35:00.951 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.951 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:00.952 DEBUG:boto:Port: 8001 2023-07-29T00:35:00.952 DEBUG:boto:Params: {} 2023-07-29T00:35:00.953 DEBUG:boto:Token: None 2023-07-29T00:35:00.953 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/ 2023-07-29T00:35:00.953 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0N+PAhtHIs4KHe60wn+BGQJADio= 2023-07-29T00:35:00.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0N+PAhtHIs4KHe60wn+BGQJADio=', 'Content-Length': '0'} 2023-07-29T00:35:00.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d655797e31e44668-0064c45eb4-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.958 DEBUG:boto:b'hiiqos-59obj921000falseobj922023-07-29T00:33:06.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:00.958 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:00.959 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:00.959 DEBUG:boto:path=/hiiqos-59/?prefix=obj92 2023-07-29T00:35:00.960 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj92 2023-07-29T00:35:00.960 DEBUG:boto:Method: GET 2023-07-29T00:35:00.961 DEBUG:boto:Path: /hiiqos-59/?prefix=obj92 2023-07-29T00:35:00.961 DEBUG:boto:Data: 2023-07-29T00:35:00.962 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.962 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:00.963 DEBUG:boto:Port: 8000 2023-07-29T00:35:00.963 DEBUG:boto:Params: {} 2023-07-29T00:35:00.963 DEBUG:boto:Token: None 2023-07-29T00:35:00.964 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/ 2023-07-29T00:35:00.964 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0N+PAhtHIs4KHe60wn+BGQJADio= 2023-07-29T00:35:00.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0N+PAhtHIs4KHe60wn+BGQJADio=', 'Content-Length': '0'} 2023-07-29T00:35:00.968 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036e735f1af712991-0064c45eb4-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.969 DEBUG:boto:b'hiiqos-59obj921000falseobj922023-07-29T00:33:06.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:00.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-07-29T00:35:00.970 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:35:00.970 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:35:00.970 DEBUG:boto:path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:35:00.971 DEBUG:boto:auth_path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:35:00.971 DEBUG:boto:Method: GET 2023-07-29T00:35:00.972 DEBUG:boto:Path: /hiiqos-59/obj95?versionId=null 2023-07-29T00:35:00.972 DEBUG:boto:Data: 2023-07-29T00:35:00.972 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.973 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:00.973 DEBUG:boto:Port: 8001 2023-07-29T00:35:00.973 DEBUG:boto:Params: {} 2023-07-29T00:35:00.974 DEBUG:boto:Token: None 2023-07-29T00:35:00.974 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj95?versionId=null 2023-07-29T00:35:00.975 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tMcAMQfKbZo/2XHX/r2TL9Fvr0s= 2023-07-29T00:35:00.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tMcAMQfKbZo/2XHX/r2TL9Fvr0s=', 'Content-Length': '0'} 2023-07-29T00:35:00.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7b1b7e0aa7a65b7-0064c45eb4-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.978 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:35:00.978 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:35:00.978 DEBUG:boto:path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:35:00.979 DEBUG:boto:auth_path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:35:00.979 DEBUG:boto:Method: GET 2023-07-29T00:35:00.980 DEBUG:boto:Path: /hiiqos-59/obj95?versionId=null 2023-07-29T00:35:00.980 DEBUG:boto:Data: 2023-07-29T00:35:00.981 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.981 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:00.981 DEBUG:boto:Port: 8000 2023-07-29T00:35:00.982 DEBUG:boto:Params: {} 2023-07-29T00:35:00.982 DEBUG:boto:Token: None 2023-07-29T00:35:00.983 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj95?versionId=null 2023-07-29T00:35:00.983 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tMcAMQfKbZo/2XHX/r2TL9Fvr0s= 2023-07-29T00:35:00.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tMcAMQfKbZo/2XHX/r2TL9Fvr0s=', 'Content-Length': '0'} 2023-07-29T00:35:00.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9e98aa51f773e2b-0064c45eb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.986 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:35:00.987 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:35:00.987 DEBUG:boto:path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:35:00.988 DEBUG:boto:auth_path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:35:00.988 DEBUG:boto:Method: HEAD 2023-07-29T00:35:00.988 DEBUG:boto:Path: /hiiqos-59/obj95?versionId=null 2023-07-29T00:35:00.989 DEBUG:boto:Data: 2023-07-29T00:35:00.989 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.989 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:00.990 DEBUG:boto:Port: 8001 2023-07-29T00:35:00.990 DEBUG:boto:Params: {} 2023-07-29T00:35:00.991 DEBUG:boto:Token: None 2023-07-29T00:35:00.991 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj95?versionId=null 2023-07-29T00:35:00.992 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GG3M8jLhZVMKSQVhldj7TXj9Kzo= 2023-07-29T00:35:00.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GG3M8jLhZVMKSQVhldj7TXj9Kzo=', 'Content-Length': '0'} 2023-07-29T00:35:00.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7c0e072909ee664-0064c45eb4-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:00.995 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:35:00.995 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:35:00.996 DEBUG:boto:path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:35:00.996 DEBUG:boto:auth_path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:35:00.996 DEBUG:boto:Method: HEAD 2023-07-29T00:35:00.997 DEBUG:boto:Path: /hiiqos-59/obj95?versionId=null 2023-07-29T00:35:00.997 DEBUG:boto:Data: 2023-07-29T00:35:00.997 DEBUG:boto:Headers: {} 2023-07-29T00:35:00.998 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:00.998 DEBUG:boto:Port: 8000 2023-07-29T00:35:00.998 DEBUG:boto:Params: {} 2023-07-29T00:35:00.999 DEBUG:boto:Token: None 2023-07-29T00:35:00.999 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:00 GMT /hiiqos-59/obj95?versionId=null 2023-07-29T00:35:01.000 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GG3M8jLhZVMKSQVhldj7TXj9Kzo= 2023-07-29T00:35:01.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GG3M8jLhZVMKSQVhldj7TXj9Kzo=', 'Content-Length': '0'} 2023-07-29T00:35:01.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006009ec5f56d49b36-0064c45eb4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:01.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-07-29T00:35:01.003 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:35:01.003 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:35:01.004 DEBUG:boto:path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:35:01.004 DEBUG:boto:auth_path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:35:01.004 DEBUG:boto:Method: GET 2023-07-29T00:35:01.005 DEBUG:boto:Path: /hiiqos-59/obj95?versionId=null 2023-07-29T00:35:01.005 DEBUG:boto:Data: 2023-07-29T00:35:01.006 DEBUG:boto:Headers: {} 2023-07-29T00:35:01.007 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:01.008 DEBUG:boto:Port: 8001 2023-07-29T00:35:01.008 DEBUG:boto:Params: {} 2023-07-29T00:35:01.009 DEBUG:boto:Token: None 2023-07-29T00:35:01.010 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:01 GMT /hiiqos-59/obj95?versionId=null 2023-07-29T00:35:01.011 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JVnTp0DJueOtYghGwI4zk+XZrj8= 2023-07-29T00:35:01.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JVnTp0DJueOtYghGwI4zk+XZrj8=', 'Content-Length': '0'} 2023-07-29T00:35:01.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075e56995f95a0888-0064c45eb4-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:01.056 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:35:01.057 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:35:01.058 DEBUG:boto:path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:35:01.058 DEBUG:boto:auth_path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:35:01.058 DEBUG:boto:Method: GET 2023-07-29T00:35:01.059 DEBUG:boto:Path: /hiiqos-59/obj95?versionId=null 2023-07-29T00:35:01.059 DEBUG:boto:Data: 2023-07-29T00:35:01.060 DEBUG:boto:Headers: {} 2023-07-29T00:35:01.060 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:01.060 DEBUG:boto:Port: 8000 2023-07-29T00:35:01.061 DEBUG:boto:Params: {} 2023-07-29T00:35:01.061 DEBUG:boto:Token: None 2023-07-29T00:35:01.062 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:01 GMT /hiiqos-59/obj95?versionId=null 2023-07-29T00:35:01.062 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JVnTp0DJueOtYghGwI4zk+XZrj8= 2023-07-29T00:35:01.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JVnTp0DJueOtYghGwI4zk+XZrj8=', 'Content-Length': '0'} 2023-07-29T00:35:01.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c1597f62b016f36-0064c45eb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:01.065 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:01.066 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:01.066 DEBUG:boto:path=/hiiqos-59/?prefix=obj95 2023-07-29T00:35:01.066 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj95 2023-07-29T00:35:01.067 DEBUG:boto:Method: GET 2023-07-29T00:35:01.067 DEBUG:boto:Path: /hiiqos-59/?prefix=obj95 2023-07-29T00:35:01.067 DEBUG:boto:Data: 2023-07-29T00:35:01.068 DEBUG:boto:Headers: {} 2023-07-29T00:35:01.068 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:01.069 DEBUG:boto:Port: 8001 2023-07-29T00:35:01.069 DEBUG:boto:Params: {} 2023-07-29T00:35:01.069 DEBUG:boto:Token: None 2023-07-29T00:35:01.070 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:01 GMT /hiiqos-59/ 2023-07-29T00:35:01.070 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rTjF1eljIqMIWqkjwHViEYOR4Bg= 2023-07-29T00:35:01.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rTjF1eljIqMIWqkjwHViEYOR4Bg=', 'Content-Length': '0'} 2023-07-29T00:35:01.073 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057970edd1e46c352-0064c45eb5-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:01.074 DEBUG:boto:b'hiiqos-59obj951000falseobj952023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:01.075 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:01.075 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:01.075 DEBUG:boto:path=/hiiqos-59/?prefix=obj95 2023-07-29T00:35:01.076 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj95 2023-07-29T00:35:01.076 DEBUG:boto:Method: GET 2023-07-29T00:35:01.076 DEBUG:boto:Path: /hiiqos-59/?prefix=obj95 2023-07-29T00:35:01.077 DEBUG:boto:Data: 2023-07-29T00:35:01.077 DEBUG:boto:Headers: {} 2023-07-29T00:35:01.078 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:01.078 DEBUG:boto:Port: 8000 2023-07-29T00:35:01.078 DEBUG:boto:Params: {} 2023-07-29T00:35:01.079 DEBUG:boto:Token: None 2023-07-29T00:35:01.079 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:01 GMT /hiiqos-59/ 2023-07-29T00:35:01.080 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rTjF1eljIqMIWqkjwHViEYOR4Bg= 2023-07-29T00:35:01.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rTjF1eljIqMIWqkjwHViEYOR4Bg=', 'Content-Length': '0'} 2023-07-29T00:35:01.083 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c35b4ff03251ce2-0064c45eb5-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:01.124 DEBUG:boto:b'hiiqos-59obj951000falseobj952023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:01.125 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:35:01.126 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:35:01.126 DEBUG:boto:Method: HEAD 2023-07-29T00:35:01.126 DEBUG:boto:Path: /hiiqos-59/obj95 2023-07-29T00:35:01.127 DEBUG:boto:Data: 2023-07-29T00:35:01.127 DEBUG:boto:Headers: {} 2023-07-29T00:35:01.127 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:01.128 DEBUG:boto:Port: 8001 2023-07-29T00:35:01.128 DEBUG:boto:Params: {} 2023-07-29T00:35:01.128 DEBUG:boto:Token: None 2023-07-29T00:35:01.129 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:01 GMT /hiiqos-59/obj95 2023-07-29T00:35:01.129 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Yt4FhThrggms7e6SpPXTWxk8EMY= 2023-07-29T00:35:01.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Yt4FhThrggms7e6SpPXTWxk8EMY=', 'Content-Length': '0'} 2023-07-29T00:35:01.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004241e3dedea8045f-0064c45eb5-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:01.132 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:35:01.133 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:35:01.133 DEBUG:boto:Method: HEAD 2023-07-29T00:35:01.133 DEBUG:boto:Path: /hiiqos-59/obj95 2023-07-29T00:35:01.134 DEBUG:boto:Data: 2023-07-29T00:35:01.134 DEBUG:boto:Headers: {} 2023-07-29T00:35:01.135 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:01.135 DEBUG:boto:Port: 8000 2023-07-29T00:35:01.135 DEBUG:boto:Params: {} 2023-07-29T00:35:01.136 DEBUG:boto:Token: None 2023-07-29T00:35:01.136 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:01 GMT /hiiqos-59/obj95 2023-07-29T00:35:01.137 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Yt4FhThrggms7e6SpPXTWxk8EMY= 2023-07-29T00:35:01.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Yt4FhThrggms7e6SpPXTWxk8EMY=', 'Content-Length': '0'} 2023-07-29T00:35:01.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000465689881d65a80e-0064c45eb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:01.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-07-29T00:35:01.140 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:35:01.141 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:35:01.141 DEBUG:boto:Method: GET 2023-07-29T00:35:01.142 DEBUG:boto:Path: /hiiqos-59/obj95 2023-07-29T00:35:01.142 DEBUG:boto:Data: 2023-07-29T00:35:01.143 DEBUG:boto:Headers: {} 2023-07-29T00:35:01.143 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:01.143 DEBUG:boto:Port: 8001 2023-07-29T00:35:01.144 DEBUG:boto:Params: {} 2023-07-29T00:35:01.144 DEBUG:boto:Token: None 2023-07-29T00:35:01.145 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:01 GMT /hiiqos-59/obj95 2023-07-29T00:35:01.146 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Fnp62NvmdD6/FQYIb7nZIijmbGs= 2023-07-29T00:35:01.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Fnp62NvmdD6/FQYIb7nZIijmbGs=', 'Content-Length': '0'} 2023-07-29T00:35:01.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ae901d6434c0b3a-0064c45eb5-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:01.149 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:35:01.150 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:35:01.151 DEBUG:boto:Method: GET 2023-07-29T00:35:01.152 DEBUG:boto:Path: /hiiqos-59/obj95 2023-07-29T00:35:01.153 DEBUG:boto:Data: 2023-07-29T00:35:01.153 DEBUG:boto:Headers: {} 2023-07-29T00:35:01.154 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:01.154 DEBUG:boto:Port: 8000 2023-07-29T00:35:01.155 DEBUG:boto:Params: {} 2023-07-29T00:35:01.155 DEBUG:boto:Token: None 2023-07-29T00:35:01.155 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:01 GMT /hiiqos-59/obj95 2023-07-29T00:35:01.156 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Fnp62NvmdD6/FQYIb7nZIijmbGs= 2023-07-29T00:35:01.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Fnp62NvmdD6/FQYIb7nZIijmbGs=', 'Content-Length': '0'} 2023-07-29T00:35:01.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c212725389ef6d9-0064c45eb5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:01.159 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:01.159 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:01.160 DEBUG:boto:path=/hiiqos-59/?prefix=obj95 2023-07-29T00:35:01.160 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj95 2023-07-29T00:35:01.160 DEBUG:boto:Method: GET 2023-07-29T00:35:01.161 DEBUG:boto:Path: /hiiqos-59/?prefix=obj95 2023-07-29T00:35:01.161 DEBUG:boto:Data: 2023-07-29T00:35:01.162 DEBUG:boto:Headers: {} 2023-07-29T00:35:01.162 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:01.162 DEBUG:boto:Port: 8001 2023-07-29T00:35:01.163 DEBUG:boto:Params: {} 2023-07-29T00:35:01.163 DEBUG:boto:Token: None 2023-07-29T00:35:01.163 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:01 GMT /hiiqos-59/ 2023-07-29T00:35:01.164 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rTjF1eljIqMIWqkjwHViEYOR4Bg= 2023-07-29T00:35:01.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rTjF1eljIqMIWqkjwHViEYOR4Bg=', 'Content-Length': '0'} 2023-07-29T00:35:01.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000805e5fdb66571584-0064c45eb5-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:01.168 DEBUG:boto:b'hiiqos-59obj951000falseobj952023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:01.168 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:01.169 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:01.169 DEBUG:boto:path=/hiiqos-59/?prefix=obj95 2023-07-29T00:35:01.170 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj95 2023-07-29T00:35:01.170 DEBUG:boto:Method: GET 2023-07-29T00:35:01.170 DEBUG:boto:Path: /hiiqos-59/?prefix=obj95 2023-07-29T00:35:01.171 DEBUG:boto:Data: 2023-07-29T00:35:01.171 DEBUG:boto:Headers: {} 2023-07-29T00:35:01.171 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:01.172 DEBUG:boto:Port: 8000 2023-07-29T00:35:01.172 DEBUG:boto:Params: {} 2023-07-29T00:35:01.173 DEBUG:boto:Token: None 2023-07-29T00:35:01.173 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:01 GMT /hiiqos-59/ 2023-07-29T00:35:01.174 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rTjF1eljIqMIWqkjwHViEYOR4Bg= 2023-07-29T00:35:01.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rTjF1eljIqMIWqkjwHViEYOR4Bg=', 'Content-Length': '0'} 2023-07-29T00:35:01.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003c8bea1d19404b6-0064c45eb5-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:01.178 DEBUG:boto:b'hiiqos-59obj951000falseobj952023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:01.179 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-59 zones={a2, a1} 2023-07-29T00:35:01.179 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:35:01.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:35:01.180 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:35:01.236 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:35:01.237 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:35:01.286 INFO:teuthology.orchestra.run.smithi101.stdout:tenant: 2023-07-29T00:35:01.287 INFO:teuthology.orchestra.run.smithi101.stdout:bucket name: hiiqos-59 2023-07-29T00:35:01.295 INFO:teuthology.orchestra.run.smithi101.stdout:total entries: 20 2023-07-29T00:35:01.350 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:01.350 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:01.351 DEBUG:boto:Method: HEAD 2023-07-29T00:35:01.351 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:35:01.352 DEBUG:boto:Data: 2023-07-29T00:35:01.352 DEBUG:boto:Headers: {} 2023-07-29T00:35:01.352 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:01.353 DEBUG:boto:Port: 8000 2023-07-29T00:35:01.353 DEBUG:boto:Params: {} 2023-07-29T00:35:01.353 DEBUG:boto:Token: None 2023-07-29T00:35:01.354 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:01 GMT /hiiqos-59/ 2023-07-29T00:35:01.354 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vav+RHIS5e6wzMResBjHVyC3B6A= 2023-07-29T00:35:01.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vav+RHIS5e6wzMResBjHVyC3B6A=', 'Content-Length': '0'} 2023-07-29T00:35:01.358 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000606e634f5cbbccfd-0064c45eb5-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:01.359 DEBUG:boto:path=/hiiqos-59/obj29 2023-07-29T00:35:01.360 DEBUG:boto:auth_path=/hiiqos-59/obj29 2023-07-29T00:35:01.361 DEBUG:boto:Method: PUT 2023-07-29T00:35:01.361 DEBUG:boto:Path: /hiiqos-59/obj29 2023-07-29T00:35:01.361 DEBUG:boto:Data: 2023-07-29T00:35:01.362 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:35:01.362 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:01.362 DEBUG:boto:Port: 8000 2023-07-29T00:35:01.363 DEBUG:boto:Params: {} 2023-07-29T00:35:01.363 DEBUG:boto:Token: None 2023-07-29T00:35:01.364 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:35:01 GMT /hiiqos-59/obj29 2023-07-29T00:35:01.364 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:31oGI86UD2fMkbL50Hwu3Q1jmbI= 2023-07-29T00:35:01.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:35:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:31oGI86UD2fMkbL50Hwu3Q1jmbI='} 2023-07-29T00:35:01.370 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009d45bcb725bce068-0064c45eb5-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:01.371 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:01.372 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:01.372 DEBUG:boto:Method: HEAD 2023-07-29T00:35:01.373 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:35:01.373 DEBUG:boto:Data: 2023-07-29T00:35:01.374 DEBUG:boto:Headers: {} 2023-07-29T00:35:01.374 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:01.374 DEBUG:boto:Port: 8000 2023-07-29T00:35:01.375 DEBUG:boto:Params: {} 2023-07-29T00:35:01.375 DEBUG:boto:Token: None 2023-07-29T00:35:01.376 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:01 GMT /hiiqos-59/ 2023-07-29T00:35:01.376 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vav+RHIS5e6wzMResBjHVyC3B6A= 2023-07-29T00:35:01.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vav+RHIS5e6wzMResBjHVyC3B6A=', 'Content-Length': '0'} 2023-07-29T00:35:01.380 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000003205bab764386ee-0064c45eb5-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:01.380 DEBUG:boto:path=/hiiqos-59/obj58 2023-07-29T00:35:01.381 DEBUG:boto:auth_path=/hiiqos-59/obj58 2023-07-29T00:35:01.381 DEBUG:boto:Method: PUT 2023-07-29T00:35:01.382 DEBUG:boto:Path: /hiiqos-59/obj58 2023-07-29T00:35:01.382 DEBUG:boto:Data: 2023-07-29T00:35:01.383 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:35:01.383 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:01.384 DEBUG:boto:Port: 8000 2023-07-29T00:35:01.384 DEBUG:boto:Params: {} 2023-07-29T00:35:01.384 DEBUG:boto:Token: None 2023-07-29T00:35:01.385 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:35:01 GMT /hiiqos-59/obj58 2023-07-29T00:35:01.385 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZDhDvy082hmPk1/1xLlOtaEvbTw= 2023-07-29T00:35:01.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:35:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZDhDvy082hmPk1/1xLlOtaEvbTw='} 2023-07-29T00:35:01.428 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e6d43c71d44b4c36-0064c45eb5-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:01.429 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:01.430 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:01.430 DEBUG:boto:Method: HEAD 2023-07-29T00:35:01.430 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:35:01.431 DEBUG:boto:Data: 2023-07-29T00:35:01.431 DEBUG:boto:Headers: {} 2023-07-29T00:35:01.431 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:01.432 DEBUG:boto:Port: 8000 2023-07-29T00:35:01.432 DEBUG:boto:Params: {} 2023-07-29T00:35:01.433 DEBUG:boto:Token: None 2023-07-29T00:35:01.433 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:01 GMT /hiiqos-59/ 2023-07-29T00:35:01.433 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vav+RHIS5e6wzMResBjHVyC3B6A= 2023-07-29T00:35:01.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vav+RHIS5e6wzMResBjHVyC3B6A=', 'Content-Length': '0'} 2023-07-29T00:35:01.437 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e44f89acb274cdaf-0064c45eb5-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:01.438 DEBUG:boto:path=/hiiqos-59/obj87 2023-07-29T00:35:01.438 DEBUG:boto:auth_path=/hiiqos-59/obj87 2023-07-29T00:35:01.439 DEBUG:boto:Method: PUT 2023-07-29T00:35:01.439 DEBUG:boto:Path: /hiiqos-59/obj87 2023-07-29T00:35:01.439 DEBUG:boto:Data: 2023-07-29T00:35:01.440 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:35:01.440 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:01.441 DEBUG:boto:Port: 8000 2023-07-29T00:35:01.441 DEBUG:boto:Params: {} 2023-07-29T00:35:01.441 DEBUG:boto:Token: None 2023-07-29T00:35:01.442 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:35:01 GMT /hiiqos-59/obj87 2023-07-29T00:35:01.442 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:awDvtlb4XyHsCrDibRilpOHbOds= 2023-07-29T00:35:01.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:35:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:awDvtlb4XyHsCrDibRilpOHbOds='} 2023-07-29T00:35:01.447 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000754b39f366681b71-0064c45eb5-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:01.448 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:01.448 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:01.449 DEBUG:boto:Method: HEAD 2023-07-29T00:35:01.449 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:35:01.449 DEBUG:boto:Data: 2023-07-29T00:35:01.450 DEBUG:boto:Headers: {} 2023-07-29T00:35:01.450 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:01.451 DEBUG:boto:Port: 8000 2023-07-29T00:35:01.451 DEBUG:boto:Params: {} 2023-07-29T00:35:01.452 DEBUG:boto:Token: None 2023-07-29T00:35:01.452 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:01 GMT /hiiqos-59/ 2023-07-29T00:35:01.453 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vav+RHIS5e6wzMResBjHVyC3B6A= 2023-07-29T00:35:01.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vav+RHIS5e6wzMResBjHVyC3B6A=', 'Content-Length': '0'} 2023-07-29T00:35:01.456 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b0992fd36fc3cff8-0064c45eb5-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:01.457 DEBUG:boto:path=/hiiqos-59/obj116 2023-07-29T00:35:01.457 DEBUG:boto:auth_path=/hiiqos-59/obj116 2023-07-29T00:35:01.457 DEBUG:boto:Method: PUT 2023-07-29T00:35:01.458 DEBUG:boto:Path: /hiiqos-59/obj116 2023-07-29T00:35:01.458 DEBUG:boto:Data: 2023-07-29T00:35:01.458 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:35:01.459 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:01.459 DEBUG:boto:Port: 8000 2023-07-29T00:35:01.460 DEBUG:boto:Params: {} 2023-07-29T00:35:01.460 DEBUG:boto:Token: None 2023-07-29T00:35:01.461 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:35:01 GMT /hiiqos-59/obj116 2023-07-29T00:35:01.461 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:03EpGhLiFCerlpUjnXXYWieNG1c= 2023-07-29T00:35:01.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:35:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:03EpGhLiFCerlpUjnXXYWieNG1c='} 2023-07-29T00:35:01.504 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000072cefed39c3da1a4-0064c45eb5-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:01.505 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:01.505 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:01.506 DEBUG:boto:Method: HEAD 2023-07-29T00:35:01.506 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:35:01.506 DEBUG:boto:Data: 2023-07-29T00:35:01.507 DEBUG:boto:Headers: {} 2023-07-29T00:35:01.507 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:01.507 DEBUG:boto:Port: 8000 2023-07-29T00:35:01.508 DEBUG:boto:Params: {} 2023-07-29T00:35:01.508 DEBUG:boto:Token: None 2023-07-29T00:35:01.509 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:01 GMT /hiiqos-59/ 2023-07-29T00:35:01.509 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vav+RHIS5e6wzMResBjHVyC3B6A= 2023-07-29T00:35:01.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vav+RHIS5e6wzMResBjHVyC3B6A=', 'Content-Length': '0'} 2023-07-29T00:35:01.512 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c0b2f9d44997096d-0064c45eb5-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:01.513 DEBUG:boto:path=/hiiqos-59/obj145 2023-07-29T00:35:01.513 DEBUG:boto:auth_path=/hiiqos-59/obj145 2023-07-29T00:35:01.513 DEBUG:boto:Method: PUT 2023-07-29T00:35:01.514 DEBUG:boto:Path: /hiiqos-59/obj145 2023-07-29T00:35:01.514 DEBUG:boto:Data: 2023-07-29T00:35:01.514 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:35:01.515 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:01.515 DEBUG:boto:Port: 8000 2023-07-29T00:35:01.515 DEBUG:boto:Params: {} 2023-07-29T00:35:01.516 DEBUG:boto:Token: None 2023-07-29T00:35:01.516 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:35:01 GMT /hiiqos-59/obj145 2023-07-29T00:35:01.517 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rI0T2ETPsfhLlwEZ6S2Nv/c2Znk= 2023-07-29T00:35:01.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:35:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rI0T2ETPsfhLlwEZ6S2Nv/c2Znk='} 2023-07-29T00:35:01.522 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004f43118d3b5f91a7-0064c45eb5-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:01.522 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-59 2023-07-29T00:35:01.523 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:35:01.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:35:01.523 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:35:01.577 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:35:01.578 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:35:01.613 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:35:01.552+0000 7f95ef867880 1 waiting to reach latest gen 4, on 3.. 2023-07-29T00:35:06.615 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:35:06.554+0000 7f95ef867880 1 bucket sync caught up with source: 2023-07-29T00:35:06.616 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [00000000002.92.6, 00000000001.63.6, , , , , 00000000001.72.6, , , , , , , , , , , 00000000002.54.6, 00000000001.64.6, , , , , , , , , , ] 2023-07-29T00:35:06.616 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [00000000002.92.6, 00000000001.63.6, , , , , 00000000001.72.6, , , , , , , , , , , 00000000002.54.6, 00000000001.64.6, , , , , , , , , , ] 2023-07-29T00:35:06.616 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:35:06.554+0000 7f95ef867880 0 bucket checkpoint complete 2023-07-29T00:35:06.632 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-59 2023-07-29T00:35:06.633 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:35:06.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:35:06.634 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:35:06.693 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:35:06.693 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:35:06.729 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:35:06.669+0000 7fcfbed4b880 1 waiting for incremental sync to catch up: 2023-07-29T00:35:06.730 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.848.6, 00000000002.701.6, 00000000001.1200.6, 00000000002.951.6, 00000000002.740.6, , , 00000000002.295.6, 00000000005.796.6, 00000000004.1374.6] 2023-07-29T00:35:06.730 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000002.848.6, 00000000002.701.6, 00000000001.1200.6, 00000000003.954.6, 00000000004.745.6, 00000000001.1016.6, 00000000001.432.6, 00000000002.295.6, 00000000005.796.6, 00000000004.1374.6] 2023-07-29T00:35:11.731 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:35:11.670+0000 7fcfbed4b880 1 waiting for incremental sync to catch up: 2023-07-29T00:35:11.731 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.848.6, 00000000002.701.6, 00000000001.1200.6, 00000000002.951.6, 00000000002.740.6, , , 00000000002.295.6, 00000000005.796.6, 00000000004.1374.6] 2023-07-29T00:35:11.731 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000002.848.6, 00000000002.701.6, 00000000001.1200.6, 00000000003.954.6, 00000000004.745.6, 00000000001.1016.6, 00000000001.432.6, 00000000002.295.6, 00000000005.796.6, 00000000004.1374.6] 2023-07-29T00:35:16.732 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:35:16.671+0000 7fcfbed4b880 1 waiting for incremental sync to catch up: 2023-07-29T00:35:16.732 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.848.6, 00000000002.701.6, 00000000001.1200.6, 00000000002.951.6, 00000000002.740.6, , , 00000000002.295.6, 00000000005.796.6, 00000000004.1374.6] 2023-07-29T00:35:16.733 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000002.848.6, 00000000002.701.6, 00000000001.1200.6, 00000000003.954.6, 00000000004.745.6, 00000000001.1016.6, 00000000001.432.6, 00000000002.295.6, 00000000005.796.6, 00000000004.1374.6] 2023-07-29T00:35:21.733 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:35:21.672+0000 7fcfbed4b880 1 waiting for incremental sync to catch up: 2023-07-29T00:35:21.734 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.848.6, 00000000002.701.6, 00000000001.1200.6, 00000000002.951.6, 00000000002.740.6, , , 00000000002.295.6, 00000000005.796.6, 00000000004.1374.6] 2023-07-29T00:35:21.734 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000002.848.6, 00000000002.701.6, 00000000001.1200.6, 00000000003.954.6, 00000000004.745.6, 00000000001.1016.6, 00000000001.432.6, 00000000002.295.6, 00000000005.796.6, 00000000004.1374.6] 2023-07-29T00:35:26.735 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:35:26.674+0000 7fcfbed4b880 1 bucket sync caught up with source: 2023-07-29T00:35:26.735 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000002.848.6, 00000000002.701.6, 00000000001.1200.6, 00000000003.954.6, 00000000004.745.6, 00000000001.1016.6, 00000000001.432.6, 00000000002.295.6, 00000000005.796.6, 00000000004.1374.6] 2023-07-29T00:35:26.736 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000002.848.6, 00000000002.701.6, 00000000001.1200.6, 00000000003.954.6, 00000000004.745.6, 00000000001.1016.6, 00000000001.432.6, 00000000002.295.6, 00000000005.796.6, 00000000004.1374.6] 2023-07-29T00:35:26.736 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:35:26.674+0000 7fcfbed4b880 0 bucket checkpoint complete 2023-07-29T00:35:26.752 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-59 zones={a2, a1} 2023-07-29T00:35:26.753 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:26.753 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:26.754 DEBUG:boto:Method: HEAD 2023-07-29T00:35:26.754 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:35:26.754 DEBUG:boto:Data: 2023-07-29T00:35:26.755 DEBUG:boto:Headers: {} 2023-07-29T00:35:26.755 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:26.755 DEBUG:boto:Port: 8001 2023-07-29T00:35:26.756 DEBUG:boto:Params: {} 2023-07-29T00:35:26.756 DEBUG:boto:Token: None 2023-07-29T00:35:26.757 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:26 GMT /hiiqos-59/ 2023-07-29T00:35:26.757 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yajmp5tTNXaFcJBsXUF2t22nzjc= 2023-07-29T00:35:26.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:26 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yajmp5tTNXaFcJBsXUF2t22nzjc=', 'Content-Length': '0'} 2023-07-29T00:35:26.760 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef4a3ec72999c21b-0064c45ece-1c08-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:35:26 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:26.761 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:26.761 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:26.761 DEBUG:boto:Method: HEAD 2023-07-29T00:35:26.762 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:35:26.762 DEBUG:boto:Data: 2023-07-29T00:35:26.762 DEBUG:boto:Headers: {} 2023-07-29T00:35:26.763 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:26.763 DEBUG:boto:Port: 8000 2023-07-29T00:35:26.763 DEBUG:boto:Params: {} 2023-07-29T00:35:26.764 DEBUG:boto:Token: None 2023-07-29T00:35:26.764 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:26 GMT /hiiqos-59/ 2023-07-29T00:35:26.765 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yajmp5tTNXaFcJBsXUF2t22nzjc= 2023-07-29T00:35:26.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:26 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yajmp5tTNXaFcJBsXUF2t22nzjc=', 'Content-Length': '0'} 2023-07-29T00:35:26.769 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000caca7c3abc1d38e8-0064c45ece-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:35:26 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:26.769 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-29T00:35:26.770 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:26.770 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:26.770 DEBUG:boto:path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:35:26.771 DEBUG:boto:auth_path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:35:26.771 DEBUG:boto:Method: GET 2023-07-29T00:35:26.771 DEBUG:boto:Path: /hiiqos-59/?versions&max-keys=999 2023-07-29T00:35:26.772 DEBUG:boto:Data: 2023-07-29T00:35:26.772 DEBUG:boto:Headers: {} 2023-07-29T00:35:26.772 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:26.773 DEBUG:boto:Port: 8001 2023-07-29T00:35:26.773 DEBUG:boto:Params: {} 2023-07-29T00:35:26.773 DEBUG:boto:Token: None 2023-07-29T00:35:26.774 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:26 GMT /hiiqos-59/?versions 2023-07-29T00:35:26.774 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:57CHT6ureQXsIuHHgVgWxEUDYMY= 2023-07-29T00:35:26.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:26 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:57CHT6ureQXsIuHHgVgWxEUDYMY=', 'Content-Length': '0'} 2023-07-29T00:35:26.777 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0d64fd388cfb456-0064c45ece-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:26 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:26.778 DEBUG:boto:b'hiiqos-59999falseobj11nulltrue2023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-07-29T00:35:01.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-07-29T00:33:06.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-07-29T00:35:01.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-07-29T00:33:06.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-07-29T00:35:01.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-07-29T00:33:06.857Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-07-29T00:35:01.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-07-29T00:33:06.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:26.780 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-07-29T00:35:26.780 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-07-29T00:35:26.780 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-07-29T00:35:26.781 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-07-29T00:35:26.781 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-07-29T00:35:26.781 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-07-29T00:35:26.782 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-07-29T00:35:26.782 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-07-29T00:35:26.782 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-07-29T00:35:26.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-07-29T00:35:26.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-07-29T00:35:26.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-07-29T00:35:26.784 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-07-29T00:35:26.784 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-07-29T00:35:26.784 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-07-29T00:35:26.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-07-29T00:35:26.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-07-29T00:35:26.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-07-29T00:35:26.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-07-29T00:35:26.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-07-29T00:35:26.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-07-29T00:35:26.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-07-29T00:35:26.787 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-07-29T00:35:26.787 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-07-29T00:35:26.787 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-07-29T00:35:26.788 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-29T00:35:26.788 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:26.788 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:26.789 DEBUG:boto:path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:35:26.789 DEBUG:boto:auth_path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:35:26.790 DEBUG:boto:Method: GET 2023-07-29T00:35:26.790 DEBUG:boto:Path: /hiiqos-59/?versions&max-keys=999 2023-07-29T00:35:26.790 DEBUG:boto:Data: 2023-07-29T00:35:26.791 DEBUG:boto:Headers: {} 2023-07-29T00:35:26.791 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:26.791 DEBUG:boto:Port: 8000 2023-07-29T00:35:26.792 DEBUG:boto:Params: {} 2023-07-29T00:35:26.792 DEBUG:boto:Token: None 2023-07-29T00:35:26.792 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:26 GMT /hiiqos-59/?versions 2023-07-29T00:35:26.793 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:57CHT6ureQXsIuHHgVgWxEUDYMY= 2023-07-29T00:35:26.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:26 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:57CHT6ureQXsIuHHgVgWxEUDYMY=', 'Content-Length': '0'} 2023-07-29T00:35:26.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008e40e4ac53c8bcd-0064c45ece-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:26 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:26.798 DEBUG:boto:b'hiiqos-59999falseobj11nulltrue2023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-07-29T00:35:01.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-07-29T00:33:06.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-07-29T00:35:01.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-07-29T00:33:06.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-07-29T00:35:01.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-07-29T00:33:06.857Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-07-29T00:35:01.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-07-29T00:33:06.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:26.799 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-07-29T00:35:26.800 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-07-29T00:35:26.800 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-07-29T00:35:26.800 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-07-29T00:35:26.801 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-07-29T00:35:26.801 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-07-29T00:35:26.801 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-07-29T00:35:26.802 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-07-29T00:35:26.802 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-07-29T00:35:26.802 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-07-29T00:35:26.802 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-07-29T00:35:26.803 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-07-29T00:35:26.803 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-07-29T00:35:26.803 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-07-29T00:35:26.804 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-07-29T00:35:26.804 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-07-29T00:35:26.804 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-07-29T00:35:26.804 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-07-29T00:35:26.805 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-07-29T00:35:26.805 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-07-29T00:35:26.805 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-07-29T00:35:26.806 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-07-29T00:35:26.806 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-07-29T00:35:26.806 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-07-29T00:35:26.806 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-07-29T00:35:26.807 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:26.807 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:26.808 DEBUG:boto:path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:35:26.808 DEBUG:boto:auth_path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:35:26.808 DEBUG:boto:Method: GET 2023-07-29T00:35:26.809 DEBUG:boto:Path: /hiiqos-59/?versions&max-keys=999 2023-07-29T00:35:26.809 DEBUG:boto:Data: 2023-07-29T00:35:26.809 DEBUG:boto:Headers: {} 2023-07-29T00:35:26.810 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:26.810 DEBUG:boto:Port: 8001 2023-07-29T00:35:26.810 DEBUG:boto:Params: {} 2023-07-29T00:35:26.811 DEBUG:boto:Token: None 2023-07-29T00:35:26.811 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:26 GMT /hiiqos-59/?versions 2023-07-29T00:35:26.812 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:57CHT6ureQXsIuHHgVgWxEUDYMY= 2023-07-29T00:35:26.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:26 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:57CHT6ureQXsIuHHgVgWxEUDYMY=', 'Content-Length': '0'} 2023-07-29T00:35:26.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a860d50439067a0-0064c45ece-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:26 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:26.815 DEBUG:boto:b'hiiqos-59999falseobj11nulltrue2023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-07-29T00:35:01.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-07-29T00:33:06.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-07-29T00:35:01.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-07-29T00:33:06.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-07-29T00:35:01.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-07-29T00:33:06.857Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-07-29T00:35:01.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-07-29T00:33:06.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:26.817 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:26.817 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:26.818 DEBUG:boto:path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:35:26.818 DEBUG:boto:auth_path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:35:26.818 DEBUG:boto:Method: GET 2023-07-29T00:35:26.819 DEBUG:boto:Path: /hiiqos-59/?versions&max-keys=999 2023-07-29T00:35:26.819 DEBUG:boto:Data: 2023-07-29T00:35:26.819 DEBUG:boto:Headers: {} 2023-07-29T00:35:26.820 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:26.820 DEBUG:boto:Port: 8000 2023-07-29T00:35:26.820 DEBUG:boto:Params: {} 2023-07-29T00:35:26.821 DEBUG:boto:Token: None 2023-07-29T00:35:26.821 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:26 GMT /hiiqos-59/?versions 2023-07-29T00:35:26.822 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:57CHT6ureQXsIuHHgVgWxEUDYMY= 2023-07-29T00:35:26.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:26 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:57CHT6ureQXsIuHHgVgWxEUDYMY=', 'Content-Length': '0'} 2023-07-29T00:35:26.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab16dda3280936cc-0064c45ece-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:26 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:26.868 DEBUG:boto:b'hiiqos-59999falseobj11nulltrue2023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-07-29T00:35:01.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-07-29T00:33:06.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-07-29T00:35:01.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-07-29T00:33:06.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-07-29T00:35:01.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-07-29T00:33:06.857Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-07-29T00:35:01.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-07-29T00:33:06.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:26.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-07-29T00:35:26.870 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:35:26.871 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:35:26.871 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:35:26.871 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:35:26.872 DEBUG:boto:Method: GET 2023-07-29T00:35:26.872 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:35:26.873 DEBUG:boto:Data: 2023-07-29T00:35:26.873 DEBUG:boto:Headers: {} 2023-07-29T00:35:26.873 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:26.874 DEBUG:boto:Port: 8001 2023-07-29T00:35:26.874 DEBUG:boto:Params: {} 2023-07-29T00:35:26.874 DEBUG:boto:Token: None 2023-07-29T00:35:26.875 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:26 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:35:26.875 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sbC407nlCzulwDRoP6UG3P1p3N0= 2023-07-29T00:35:26.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:26 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sbC407nlCzulwDRoP6UG3P1p3N0=', 'Content-Length': '0'} 2023-07-29T00:35:26.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d66cfb85c039b32b-0064c45ece-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:26 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:26.878 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:35:26.879 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:35:26.879 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:35:26.879 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:35:26.880 DEBUG:boto:Method: GET 2023-07-29T00:35:26.880 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:35:26.880 DEBUG:boto:Data: 2023-07-29T00:35:26.881 DEBUG:boto:Headers: {} 2023-07-29T00:35:26.881 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:26.882 DEBUG:boto:Port: 8000 2023-07-29T00:35:26.882 DEBUG:boto:Params: {} 2023-07-29T00:35:26.882 DEBUG:boto:Token: None 2023-07-29T00:35:26.883 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:26 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:35:26.883 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sbC407nlCzulwDRoP6UG3P1p3N0= 2023-07-29T00:35:26.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:26 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sbC407nlCzulwDRoP6UG3P1p3N0=', 'Content-Length': '0'} 2023-07-29T00:35:26.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9f474b8d8683b00-0064c45ece-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:26 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:26.886 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:35:26.887 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:35:26.887 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:35:26.887 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:35:26.888 DEBUG:boto:Method: HEAD 2023-07-29T00:35:26.888 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:35:26.888 DEBUG:boto:Data: 2023-07-29T00:35:26.889 DEBUG:boto:Headers: {} 2023-07-29T00:35:26.889 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:26.889 DEBUG:boto:Port: 8001 2023-07-29T00:35:26.890 DEBUG:boto:Params: {} 2023-07-29T00:35:26.890 DEBUG:boto:Token: None 2023-07-29T00:35:26.890 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:26 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:35:26.891 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SYVWbSWT7OdFWuTLprelaOd6zQg= 2023-07-29T00:35:26.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:26 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SYVWbSWT7OdFWuTLprelaOd6zQg=', 'Content-Length': '0'} 2023-07-29T00:35:26.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008237102e51348867-0064c45ece-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:26 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:26.894 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:35:26.894 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:35:26.894 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:35:26.895 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:35:26.895 DEBUG:boto:Method: HEAD 2023-07-29T00:35:26.895 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:35:26.896 DEBUG:boto:Data: 2023-07-29T00:35:26.896 DEBUG:boto:Headers: {} 2023-07-29T00:35:26.897 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:26.897 DEBUG:boto:Port: 8000 2023-07-29T00:35:26.897 DEBUG:boto:Params: {} 2023-07-29T00:35:26.898 DEBUG:boto:Token: None 2023-07-29T00:35:26.898 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:26 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:35:26.898 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SYVWbSWT7OdFWuTLprelaOd6zQg= 2023-07-29T00:35:26.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:26 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SYVWbSWT7OdFWuTLprelaOd6zQg=', 'Content-Length': '0'} 2023-07-29T00:35:26.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000838bc4b80c1357a9-0064c45ece-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:26 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:26.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-07-29T00:35:26.901 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:35:26.902 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:35:26.902 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:35:26.903 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:35:26.903 DEBUG:boto:Method: GET 2023-07-29T00:35:26.903 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:35:26.904 DEBUG:boto:Data: 2023-07-29T00:35:26.904 DEBUG:boto:Headers: {} 2023-07-29T00:35:26.904 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:26.905 DEBUG:boto:Port: 8001 2023-07-29T00:35:26.905 DEBUG:boto:Params: {} 2023-07-29T00:35:26.905 DEBUG:boto:Token: None 2023-07-29T00:35:26.906 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:26 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:35:26.906 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sbC407nlCzulwDRoP6UG3P1p3N0= 2023-07-29T00:35:26.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:26 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sbC407nlCzulwDRoP6UG3P1p3N0=', 'Content-Length': '0'} 2023-07-29T00:35:26.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089010f96a54add13-0064c45ece-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:26 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:26.909 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:35:26.909 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:35:26.910 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:35:26.910 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:35:26.910 DEBUG:boto:Method: GET 2023-07-29T00:35:26.911 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:35:26.911 DEBUG:boto:Data: 2023-07-29T00:35:26.912 DEBUG:boto:Headers: {} 2023-07-29T00:35:26.912 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:26.912 DEBUG:boto:Port: 8000 2023-07-29T00:35:26.913 DEBUG:boto:Params: {} 2023-07-29T00:35:26.913 DEBUG:boto:Token: None 2023-07-29T00:35:26.913 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:26 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:35:26.914 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sbC407nlCzulwDRoP6UG3P1p3N0= 2023-07-29T00:35:26.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:26 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sbC407nlCzulwDRoP6UG3P1p3N0=', 'Content-Length': '0'} 2023-07-29T00:35:26.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7a303999ff9f422-0064c45ece-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:26 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:26.917 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:26.918 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:26.918 DEBUG:boto:path=/hiiqos-59/?prefix=obj11 2023-07-29T00:35:26.918 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj11 2023-07-29T00:35:26.919 DEBUG:boto:Method: GET 2023-07-29T00:35:26.919 DEBUG:boto:Path: /hiiqos-59/?prefix=obj11 2023-07-29T00:35:26.920 DEBUG:boto:Data: 2023-07-29T00:35:26.920 DEBUG:boto:Headers: {} 2023-07-29T00:35:26.920 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:26.921 DEBUG:boto:Port: 8001 2023-07-29T00:35:26.921 DEBUG:boto:Params: {} 2023-07-29T00:35:26.921 DEBUG:boto:Token: None 2023-07-29T00:35:26.922 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:26 GMT /hiiqos-59/ 2023-07-29T00:35:26.922 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uIoOGIwXIiBKLGm+vUUTgCciLNs= 2023-07-29T00:35:26.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:26 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uIoOGIwXIiBKLGm+vUUTgCciLNs=', 'Content-Length': '0'} 2023-07-29T00:35:26.925 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c4765801594253e-0064c45ece-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:26 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:26.968 DEBUG:boto:b'hiiqos-59obj111000falseobj112023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:26.969 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:26.969 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:26.970 DEBUG:boto:path=/hiiqos-59/?prefix=obj11 2023-07-29T00:35:26.970 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj11 2023-07-29T00:35:26.971 DEBUG:boto:Method: GET 2023-07-29T00:35:26.971 DEBUG:boto:Path: /hiiqos-59/?prefix=obj11 2023-07-29T00:35:26.971 DEBUG:boto:Data: 2023-07-29T00:35:26.972 DEBUG:boto:Headers: {} 2023-07-29T00:35:26.972 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:26.972 DEBUG:boto:Port: 8000 2023-07-29T00:35:26.973 DEBUG:boto:Params: {} 2023-07-29T00:35:26.973 DEBUG:boto:Token: None 2023-07-29T00:35:26.974 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:26 GMT /hiiqos-59/ 2023-07-29T00:35:26.974 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uIoOGIwXIiBKLGm+vUUTgCciLNs= 2023-07-29T00:35:26.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:26 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uIoOGIwXIiBKLGm+vUUTgCciLNs=', 'Content-Length': '0'} 2023-07-29T00:35:26.978 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7af03c9e78ae41d-0064c45ece-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:26 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:26.979 DEBUG:boto:b'hiiqos-59obj111000falseobj112023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:26.979 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:35:26.980 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:35:26.980 DEBUG:boto:Method: HEAD 2023-07-29T00:35:26.980 DEBUG:boto:Path: /hiiqos-59/obj11 2023-07-29T00:35:26.981 DEBUG:boto:Data: 2023-07-29T00:35:26.981 DEBUG:boto:Headers: {} 2023-07-29T00:35:26.981 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:26.982 DEBUG:boto:Port: 8001 2023-07-29T00:35:26.982 DEBUG:boto:Params: {} 2023-07-29T00:35:26.982 DEBUG:boto:Token: None 2023-07-29T00:35:26.983 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:26 GMT /hiiqos-59/obj11 2023-07-29T00:35:26.983 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:itAmyKEAOJT5um5V+hpNhhzqPQc= 2023-07-29T00:35:26.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:26 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:itAmyKEAOJT5um5V+hpNhhzqPQc=', 'Content-Length': '0'} 2023-07-29T00:35:26.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000188ae3bcaaafe675-0064c45ece-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:26 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:26.986 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:35:26.986 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:35:26.987 DEBUG:boto:Method: HEAD 2023-07-29T00:35:26.987 DEBUG:boto:Path: /hiiqos-59/obj11 2023-07-29T00:35:26.987 DEBUG:boto:Data: 2023-07-29T00:35:26.988 DEBUG:boto:Headers: {} 2023-07-29T00:35:26.988 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:26.988 DEBUG:boto:Port: 8000 2023-07-29T00:35:26.989 DEBUG:boto:Params: {} 2023-07-29T00:35:26.989 DEBUG:boto:Token: None 2023-07-29T00:35:26.990 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:26 GMT /hiiqos-59/obj11 2023-07-29T00:35:26.990 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:itAmyKEAOJT5um5V+hpNhhzqPQc= 2023-07-29T00:35:26.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:26 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:itAmyKEAOJT5um5V+hpNhhzqPQc=', 'Content-Length': '0'} 2023-07-29T00:35:26.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5f7170cca3bbcb8-0064c45ece-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:26 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:26.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-07-29T00:35:26.993 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:35:26.994 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:35:26.994 DEBUG:boto:Method: GET 2023-07-29T00:35:26.994 DEBUG:boto:Path: /hiiqos-59/obj11 2023-07-29T00:35:26.995 DEBUG:boto:Data: 2023-07-29T00:35:26.995 DEBUG:boto:Headers: {} 2023-07-29T00:35:26.995 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:26.996 DEBUG:boto:Port: 8001 2023-07-29T00:35:26.996 DEBUG:boto:Params: {} 2023-07-29T00:35:26.997 DEBUG:boto:Token: None 2023-07-29T00:35:26.997 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:26 GMT /hiiqos-59/obj11 2023-07-29T00:35:26.997 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EEu+o+sDmgGioSe3QEp9pOh2loE= 2023-07-29T00:35:26.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:26 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EEu+o+sDmgGioSe3QEp9pOh2loE=', 'Content-Length': '0'} 2023-07-29T00:35:26.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b87a80ed036555ec-0064c45ece-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:26 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.000 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:35:27.001 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:35:27.001 DEBUG:boto:Method: GET 2023-07-29T00:35:27.001 DEBUG:boto:Path: /hiiqos-59/obj11 2023-07-29T00:35:27.002 DEBUG:boto:Data: 2023-07-29T00:35:27.002 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.002 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:27.003 DEBUG:boto:Port: 8000 2023-07-29T00:35:27.003 DEBUG:boto:Params: {} 2023-07-29T00:35:27.003 DEBUG:boto:Token: None 2023-07-29T00:35:27.004 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj11 2023-07-29T00:35:27.004 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7z/rJGESZsWpxubmCW2Mv+ZBI2o= 2023-07-29T00:35:27.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7z/rJGESZsWpxubmCW2Mv+ZBI2o=', 'Content-Length': '0'} 2023-07-29T00:35:27.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf66756c41b74861-0064c45ece-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:26 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.048 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:27.049 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:27.049 DEBUG:boto:path=/hiiqos-59/?prefix=obj11 2023-07-29T00:35:27.049 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj11 2023-07-29T00:35:27.050 DEBUG:boto:Method: GET 2023-07-29T00:35:27.050 DEBUG:boto:Path: /hiiqos-59/?prefix=obj11 2023-07-29T00:35:27.050 DEBUG:boto:Data: 2023-07-29T00:35:27.051 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.051 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:27.052 DEBUG:boto:Port: 8001 2023-07-29T00:35:27.052 DEBUG:boto:Params: {} 2023-07-29T00:35:27.052 DEBUG:boto:Token: None 2023-07-29T00:35:27.053 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/ 2023-07-29T00:35:27.053 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A= 2023-07-29T00:35:27.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A=', 'Content-Length': '0'} 2023-07-29T00:35:27.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000962875ab1beae9fe-0064c45ece-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:26 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.057 DEBUG:boto:b'hiiqos-59obj111000falseobj112023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:27.057 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:27.058 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:27.058 DEBUG:boto:path=/hiiqos-59/?prefix=obj11 2023-07-29T00:35:27.058 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj11 2023-07-29T00:35:27.059 DEBUG:boto:Method: GET 2023-07-29T00:35:27.059 DEBUG:boto:Path: /hiiqos-59/?prefix=obj11 2023-07-29T00:35:27.059 DEBUG:boto:Data: 2023-07-29T00:35:27.060 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.060 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:27.061 DEBUG:boto:Port: 8000 2023-07-29T00:35:27.061 DEBUG:boto:Params: {} 2023-07-29T00:35:27.061 DEBUG:boto:Token: None 2023-07-29T00:35:27.062 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/ 2023-07-29T00:35:27.062 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A= 2023-07-29T00:35:27.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A=', 'Content-Length': '0'} 2023-07-29T00:35:27.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2c1cb1a16f66c2f-0064c45ecf-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.067 DEBUG:boto:b'hiiqos-59obj111000falseobj112023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:27.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-07-29T00:35:27.068 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:35:27.068 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:35:27.069 DEBUG:boto:path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:35:27.069 DEBUG:boto:auth_path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:35:27.069 DEBUG:boto:Method: GET 2023-07-29T00:35:27.070 DEBUG:boto:Path: /hiiqos-59/obj115?versionId=null 2023-07-29T00:35:27.070 DEBUG:boto:Data: 2023-07-29T00:35:27.070 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.071 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:27.071 DEBUG:boto:Port: 8001 2023-07-29T00:35:27.071 DEBUG:boto:Params: {} 2023-07-29T00:35:27.072 DEBUG:boto:Token: None 2023-07-29T00:35:27.072 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj115?versionId=null 2023-07-29T00:35:27.073 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fkWEbtZW/8+Q614E4Tp5/BQU/BM= 2023-07-29T00:35:27.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fkWEbtZW/8+Q614E4Tp5/BQU/BM=', 'Content-Length': '0'} 2023-07-29T00:35:27.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028c05a0035a8b118-0064c45ecf-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.076 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:35:27.076 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:35:27.076 DEBUG:boto:path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:35:27.077 DEBUG:boto:auth_path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:35:27.077 DEBUG:boto:Method: GET 2023-07-29T00:35:27.078 DEBUG:boto:Path: /hiiqos-59/obj115?versionId=null 2023-07-29T00:35:27.078 DEBUG:boto:Data: 2023-07-29T00:35:27.078 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.079 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:27.079 DEBUG:boto:Port: 8000 2023-07-29T00:35:27.079 DEBUG:boto:Params: {} 2023-07-29T00:35:27.080 DEBUG:boto:Token: None 2023-07-29T00:35:27.080 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj115?versionId=null 2023-07-29T00:35:27.081 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fkWEbtZW/8+Q614E4Tp5/BQU/BM= 2023-07-29T00:35:27.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fkWEbtZW/8+Q614E4Tp5/BQU/BM=', 'Content-Length': '0'} 2023-07-29T00:35:27.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a9c765101c76c6c-0064c45ecf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.083 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:35:27.084 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:35:27.084 DEBUG:boto:path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:35:27.084 DEBUG:boto:auth_path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:35:27.085 DEBUG:boto:Method: HEAD 2023-07-29T00:35:27.085 DEBUG:boto:Path: /hiiqos-59/obj115?versionId=null 2023-07-29T00:35:27.085 DEBUG:boto:Data: 2023-07-29T00:35:27.086 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.086 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:27.086 DEBUG:boto:Port: 8001 2023-07-29T00:35:27.087 DEBUG:boto:Params: {} 2023-07-29T00:35:27.087 DEBUG:boto:Token: None 2023-07-29T00:35:27.088 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj115?versionId=null 2023-07-29T00:35:27.088 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UWS+r9SJIBWEfksn2ewEUMHhQZs= 2023-07-29T00:35:27.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UWS+r9SJIBWEfksn2ewEUMHhQZs=', 'Content-Length': '0'} 2023-07-29T00:35:27.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b88a739f3132d516-0064c45ecf-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.091 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:35:27.091 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:35:27.091 DEBUG:boto:path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:35:27.092 DEBUG:boto:auth_path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:35:27.092 DEBUG:boto:Method: HEAD 2023-07-29T00:35:27.092 DEBUG:boto:Path: /hiiqos-59/obj115?versionId=null 2023-07-29T00:35:27.093 DEBUG:boto:Data: 2023-07-29T00:35:27.093 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.093 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:27.094 DEBUG:boto:Port: 8000 2023-07-29T00:35:27.094 DEBUG:boto:Params: {} 2023-07-29T00:35:27.094 DEBUG:boto:Token: None 2023-07-29T00:35:27.095 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj115?versionId=null 2023-07-29T00:35:27.095 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UWS+r9SJIBWEfksn2ewEUMHhQZs= 2023-07-29T00:35:27.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UWS+r9SJIBWEfksn2ewEUMHhQZs=', 'Content-Length': '0'} 2023-07-29T00:35:27.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002809ce6817758e6d-0064c45ecf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-07-29T00:35:27.098 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:35:27.099 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:35:27.099 DEBUG:boto:path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:35:27.100 DEBUG:boto:auth_path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:35:27.100 DEBUG:boto:Method: GET 2023-07-29T00:35:27.100 DEBUG:boto:Path: /hiiqos-59/obj115?versionId=null 2023-07-29T00:35:27.101 DEBUG:boto:Data: 2023-07-29T00:35:27.101 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.101 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:27.102 DEBUG:boto:Port: 8001 2023-07-29T00:35:27.102 DEBUG:boto:Params: {} 2023-07-29T00:35:27.102 DEBUG:boto:Token: None 2023-07-29T00:35:27.103 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj115?versionId=null 2023-07-29T00:35:27.103 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fkWEbtZW/8+Q614E4Tp5/BQU/BM= 2023-07-29T00:35:27.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fkWEbtZW/8+Q614E4Tp5/BQU/BM=', 'Content-Length': '0'} 2023-07-29T00:35:27.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046d588f13ece37ea-0064c45ecf-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.106 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:35:27.107 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:35:27.107 DEBUG:boto:path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:35:27.107 DEBUG:boto:auth_path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:35:27.108 DEBUG:boto:Method: GET 2023-07-29T00:35:27.108 DEBUG:boto:Path: /hiiqos-59/obj115?versionId=null 2023-07-29T00:35:27.109 DEBUG:boto:Data: 2023-07-29T00:35:27.109 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.109 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:27.110 DEBUG:boto:Port: 8000 2023-07-29T00:35:27.110 DEBUG:boto:Params: {} 2023-07-29T00:35:27.110 DEBUG:boto:Token: None 2023-07-29T00:35:27.111 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj115?versionId=null 2023-07-29T00:35:27.111 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fkWEbtZW/8+Q614E4Tp5/BQU/BM= 2023-07-29T00:35:27.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fkWEbtZW/8+Q614E4Tp5/BQU/BM=', 'Content-Length': '0'} 2023-07-29T00:35:27.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000209acb6522d41eef-0064c45ecf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.115 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:27.115 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:27.115 DEBUG:boto:path=/hiiqos-59/?prefix=obj115 2023-07-29T00:35:27.116 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj115 2023-07-29T00:35:27.116 DEBUG:boto:Method: GET 2023-07-29T00:35:27.116 DEBUG:boto:Path: /hiiqos-59/?prefix=obj115 2023-07-29T00:35:27.117 DEBUG:boto:Data: 2023-07-29T00:35:27.117 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.117 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:27.118 DEBUG:boto:Port: 8001 2023-07-29T00:35:27.118 DEBUG:boto:Params: {} 2023-07-29T00:35:27.119 DEBUG:boto:Token: None 2023-07-29T00:35:27.119 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/ 2023-07-29T00:35:27.119 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A= 2023-07-29T00:35:27.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A=', 'Content-Length': '0'} 2023-07-29T00:35:27.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c750a7f5e360f42-0064c45ecf-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.123 DEBUG:boto:b'hiiqos-59obj1151000falseobj1152023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:27.123 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:27.124 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:27.124 DEBUG:boto:path=/hiiqos-59/?prefix=obj115 2023-07-29T00:35:27.124 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj115 2023-07-29T00:35:27.125 DEBUG:boto:Method: GET 2023-07-29T00:35:27.125 DEBUG:boto:Path: /hiiqos-59/?prefix=obj115 2023-07-29T00:35:27.125 DEBUG:boto:Data: 2023-07-29T00:35:27.126 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.126 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:27.126 DEBUG:boto:Port: 8000 2023-07-29T00:35:27.127 DEBUG:boto:Params: {} 2023-07-29T00:35:27.127 DEBUG:boto:Token: None 2023-07-29T00:35:27.128 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/ 2023-07-29T00:35:27.128 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A= 2023-07-29T00:35:27.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A=', 'Content-Length': '0'} 2023-07-29T00:35:27.132 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e43b3faa4ad4150-0064c45ecf-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.133 DEBUG:boto:b'hiiqos-59obj1151000falseobj1152023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:27.133 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:35:27.134 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:35:27.134 DEBUG:boto:Method: HEAD 2023-07-29T00:35:27.134 DEBUG:boto:Path: /hiiqos-59/obj115 2023-07-29T00:35:27.135 DEBUG:boto:Data: 2023-07-29T00:35:27.135 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.135 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:27.136 DEBUG:boto:Port: 8001 2023-07-29T00:35:27.136 DEBUG:boto:Params: {} 2023-07-29T00:35:27.136 DEBUG:boto:Token: None 2023-07-29T00:35:27.137 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj115 2023-07-29T00:35:27.137 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yBfx7nCeyU5fXLfkZ1w4fmmSl0E= 2023-07-29T00:35:27.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yBfx7nCeyU5fXLfkZ1w4fmmSl0E=', 'Content-Length': '0'} 2023-07-29T00:35:27.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017e567c3cc69a4d1-0064c45ecf-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.140 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:35:27.140 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:35:27.141 DEBUG:boto:Method: HEAD 2023-07-29T00:35:27.141 DEBUG:boto:Path: /hiiqos-59/obj115 2023-07-29T00:35:27.141 DEBUG:boto:Data: 2023-07-29T00:35:27.142 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.142 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:27.142 DEBUG:boto:Port: 8000 2023-07-29T00:35:27.143 DEBUG:boto:Params: {} 2023-07-29T00:35:27.143 DEBUG:boto:Token: None 2023-07-29T00:35:27.143 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj115 2023-07-29T00:35:27.144 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yBfx7nCeyU5fXLfkZ1w4fmmSl0E= 2023-07-29T00:35:27.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yBfx7nCeyU5fXLfkZ1w4fmmSl0E=', 'Content-Length': '0'} 2023-07-29T00:35:27.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b333632e989e3bf1-0064c45ecf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-07-29T00:35:27.147 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:35:27.147 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:35:27.147 DEBUG:boto:Method: GET 2023-07-29T00:35:27.148 DEBUG:boto:Path: /hiiqos-59/obj115 2023-07-29T00:35:27.148 DEBUG:boto:Data: 2023-07-29T00:35:27.149 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.149 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:27.149 DEBUG:boto:Port: 8001 2023-07-29T00:35:27.150 DEBUG:boto:Params: {} 2023-07-29T00:35:27.150 DEBUG:boto:Token: None 2023-07-29T00:35:27.151 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj115 2023-07-29T00:35:27.151 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tXGNeIQNTK5WI3KTxPTNHQoFO7Y= 2023-07-29T00:35:27.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tXGNeIQNTK5WI3KTxPTNHQoFO7Y=', 'Content-Length': '0'} 2023-07-29T00:35:27.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f706e6e470297b12-0064c45ecf-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.154 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:35:27.154 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:35:27.154 DEBUG:boto:Method: GET 2023-07-29T00:35:27.155 DEBUG:boto:Path: /hiiqos-59/obj115 2023-07-29T00:35:27.155 DEBUG:boto:Data: 2023-07-29T00:35:27.156 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.156 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:27.156 DEBUG:boto:Port: 8000 2023-07-29T00:35:27.157 DEBUG:boto:Params: {} 2023-07-29T00:35:27.157 DEBUG:boto:Token: None 2023-07-29T00:35:27.157 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj115 2023-07-29T00:35:27.158 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tXGNeIQNTK5WI3KTxPTNHQoFO7Y= 2023-07-29T00:35:27.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tXGNeIQNTK5WI3KTxPTNHQoFO7Y=', 'Content-Length': '0'} 2023-07-29T00:35:27.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3a0eca2059eb664-0064c45ecf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.200 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:27.201 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:27.201 DEBUG:boto:path=/hiiqos-59/?prefix=obj115 2023-07-29T00:35:27.201 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj115 2023-07-29T00:35:27.202 DEBUG:boto:Method: GET 2023-07-29T00:35:27.202 DEBUG:boto:Path: /hiiqos-59/?prefix=obj115 2023-07-29T00:35:27.202 DEBUG:boto:Data: 2023-07-29T00:35:27.203 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.203 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:27.203 DEBUG:boto:Port: 8001 2023-07-29T00:35:27.204 DEBUG:boto:Params: {} 2023-07-29T00:35:27.204 DEBUG:boto:Token: None 2023-07-29T00:35:27.205 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/ 2023-07-29T00:35:27.205 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A= 2023-07-29T00:35:27.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A=', 'Content-Length': '0'} 2023-07-29T00:35:27.208 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a6e19e89b172c1e-0064c45ecf-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.208 DEBUG:boto:b'hiiqos-59obj1151000falseobj1152023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:27.209 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:27.209 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:27.210 DEBUG:boto:path=/hiiqos-59/?prefix=obj115 2023-07-29T00:35:27.210 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj115 2023-07-29T00:35:27.211 DEBUG:boto:Method: GET 2023-07-29T00:35:27.211 DEBUG:boto:Path: /hiiqos-59/?prefix=obj115 2023-07-29T00:35:27.211 DEBUG:boto:Data: 2023-07-29T00:35:27.212 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.212 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:27.212 DEBUG:boto:Port: 8000 2023-07-29T00:35:27.213 DEBUG:boto:Params: {} 2023-07-29T00:35:27.213 DEBUG:boto:Token: None 2023-07-29T00:35:27.213 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/ 2023-07-29T00:35:27.214 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A= 2023-07-29T00:35:27.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A=', 'Content-Length': '0'} 2023-07-29T00:35:27.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c1cac7c8c8f5909-0064c45ecf-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.219 DEBUG:boto:b'hiiqos-59obj1151000falseobj1152023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:27.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-07-29T00:35:27.220 DEBUG:boto:path=/hiiqos-59/obj116 2023-07-29T00:35:27.220 DEBUG:boto:auth_path=/hiiqos-59/obj116 2023-07-29T00:35:27.220 DEBUG:boto:path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:35:27.221 DEBUG:boto:auth_path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:35:27.221 DEBUG:boto:Method: GET 2023-07-29T00:35:27.221 DEBUG:boto:Path: /hiiqos-59/obj116?versionId=null 2023-07-29T00:35:27.222 DEBUG:boto:Data: 2023-07-29T00:35:27.222 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.223 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:27.223 DEBUG:boto:Port: 8001 2023-07-29T00:35:27.223 DEBUG:boto:Params: {} 2023-07-29T00:35:27.224 DEBUG:boto:Token: None 2023-07-29T00:35:27.224 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj116?versionId=null 2023-07-29T00:35:27.224 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2dcFYq250K7I4ydmdDDdg2QHMyY= 2023-07-29T00:35:27.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2dcFYq250K7I4ydmdDDdg2QHMyY=', 'Content-Length': '0'} 2023-07-29T00:35:27.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c6d55ac2a736ec4-0064c45ecf-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.272 DEBUG:boto:path=/hiiqos-59/obj116 2023-07-29T00:35:27.273 DEBUG:boto:auth_path=/hiiqos-59/obj116 2023-07-29T00:35:27.273 DEBUG:boto:path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:35:27.274 DEBUG:boto:auth_path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:35:27.274 DEBUG:boto:Method: GET 2023-07-29T00:35:27.274 DEBUG:boto:Path: /hiiqos-59/obj116?versionId=null 2023-07-29T00:35:27.275 DEBUG:boto:Data: 2023-07-29T00:35:27.275 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.275 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:27.276 DEBUG:boto:Port: 8000 2023-07-29T00:35:27.276 DEBUG:boto:Params: {} 2023-07-29T00:35:27.277 DEBUG:boto:Token: None 2023-07-29T00:35:27.277 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj116?versionId=null 2023-07-29T00:35:27.277 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2dcFYq250K7I4ydmdDDdg2QHMyY= 2023-07-29T00:35:27.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2dcFYq250K7I4ydmdDDdg2QHMyY=', 'Content-Length': '0'} 2023-07-29T00:35:27.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002aba0f398f96546-0064c45ecf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.281 DEBUG:boto:path=/hiiqos-59/obj116 2023-07-29T00:35:27.281 DEBUG:boto:auth_path=/hiiqos-59/obj116 2023-07-29T00:35:27.281 DEBUG:boto:path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:35:27.282 DEBUG:boto:auth_path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:35:27.282 DEBUG:boto:Method: HEAD 2023-07-29T00:35:27.282 DEBUG:boto:Path: /hiiqos-59/obj116?versionId=null 2023-07-29T00:35:27.283 DEBUG:boto:Data: 2023-07-29T00:35:27.283 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.283 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:27.284 DEBUG:boto:Port: 8001 2023-07-29T00:35:27.284 DEBUG:boto:Params: {} 2023-07-29T00:35:27.284 DEBUG:boto:Token: None 2023-07-29T00:35:27.285 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj116?versionId=null 2023-07-29T00:35:27.285 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7HF3PL5dmesM9CZkaC5CsIT6jks= 2023-07-29T00:35:27.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7HF3PL5dmesM9CZkaC5CsIT6jks=', 'Content-Length': '0'} 2023-07-29T00:35:27.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f6456854b49b1b7-0064c45ecf-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.288 DEBUG:boto:path=/hiiqos-59/obj116 2023-07-29T00:35:27.289 DEBUG:boto:auth_path=/hiiqos-59/obj116 2023-07-29T00:35:27.289 DEBUG:boto:path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:35:27.289 DEBUG:boto:auth_path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:35:27.290 DEBUG:boto:Method: HEAD 2023-07-29T00:35:27.290 DEBUG:boto:Path: /hiiqos-59/obj116?versionId=null 2023-07-29T00:35:27.290 DEBUG:boto:Data: 2023-07-29T00:35:27.291 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.291 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:27.291 DEBUG:boto:Port: 8000 2023-07-29T00:35:27.292 DEBUG:boto:Params: {} 2023-07-29T00:35:27.292 DEBUG:boto:Token: None 2023-07-29T00:35:27.292 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj116?versionId=null 2023-07-29T00:35:27.293 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7HF3PL5dmesM9CZkaC5CsIT6jks= 2023-07-29T00:35:27.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7HF3PL5dmesM9CZkaC5CsIT6jks=', 'Content-Length': '0'} 2023-07-29T00:35:27.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c9a03123418d0d7-0064c45ecf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-07-29T00:35:27.296 DEBUG:boto:path=/hiiqos-59/obj116 2023-07-29T00:35:27.296 DEBUG:boto:auth_path=/hiiqos-59/obj116 2023-07-29T00:35:27.297 DEBUG:boto:path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:35:27.297 DEBUG:boto:auth_path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:35:27.297 DEBUG:boto:Method: GET 2023-07-29T00:35:27.298 DEBUG:boto:Path: /hiiqos-59/obj116?versionId=null 2023-07-29T00:35:27.298 DEBUG:boto:Data: 2023-07-29T00:35:27.298 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.299 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:27.299 DEBUG:boto:Port: 8001 2023-07-29T00:35:27.300 DEBUG:boto:Params: {} 2023-07-29T00:35:27.300 DEBUG:boto:Token: None 2023-07-29T00:35:27.300 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj116?versionId=null 2023-07-29T00:35:27.301 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2dcFYq250K7I4ydmdDDdg2QHMyY= 2023-07-29T00:35:27.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2dcFYq250K7I4ydmdDDdg2QHMyY=', 'Content-Length': '0'} 2023-07-29T00:35:27.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000353033f90a882ad4-0064c45ecf-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.304 DEBUG:boto:path=/hiiqos-59/obj116 2023-07-29T00:35:27.304 DEBUG:boto:auth_path=/hiiqos-59/obj116 2023-07-29T00:35:27.304 DEBUG:boto:path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:35:27.305 DEBUG:boto:auth_path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:35:27.305 DEBUG:boto:Method: GET 2023-07-29T00:35:27.306 DEBUG:boto:Path: /hiiqos-59/obj116?versionId=null 2023-07-29T00:35:27.306 DEBUG:boto:Data: 2023-07-29T00:35:27.306 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.307 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:27.307 DEBUG:boto:Port: 8000 2023-07-29T00:35:27.308 DEBUG:boto:Params: {} 2023-07-29T00:35:27.308 DEBUG:boto:Token: None 2023-07-29T00:35:27.308 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj116?versionId=null 2023-07-29T00:35:27.309 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2dcFYq250K7I4ydmdDDdg2QHMyY= 2023-07-29T00:35:27.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2dcFYq250K7I4ydmdDDdg2QHMyY=', 'Content-Length': '0'} 2023-07-29T00:35:27.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000568ce1951c29d371-0064c45ecf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.312 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:27.313 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:27.313 DEBUG:boto:path=/hiiqos-59/?prefix=obj116 2023-07-29T00:35:27.313 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj116 2023-07-29T00:35:27.314 DEBUG:boto:Method: GET 2023-07-29T00:35:27.314 DEBUG:boto:Path: /hiiqos-59/?prefix=obj116 2023-07-29T00:35:27.315 DEBUG:boto:Data: 2023-07-29T00:35:27.315 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.315 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:27.316 DEBUG:boto:Port: 8001 2023-07-29T00:35:27.316 DEBUG:boto:Params: {} 2023-07-29T00:35:27.316 DEBUG:boto:Token: None 2023-07-29T00:35:27.317 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/ 2023-07-29T00:35:27.317 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A= 2023-07-29T00:35:27.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A=', 'Content-Length': '0'} 2023-07-29T00:35:27.320 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000042489a9e501cd64-0064c45ecf-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.321 DEBUG:boto:b'hiiqos-59obj1161000falseobj1162023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:27.322 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:27.322 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:27.323 DEBUG:boto:path=/hiiqos-59/?prefix=obj116 2023-07-29T00:35:27.324 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj116 2023-07-29T00:35:27.324 DEBUG:boto:Method: GET 2023-07-29T00:35:27.325 DEBUG:boto:Path: /hiiqos-59/?prefix=obj116 2023-07-29T00:35:27.325 DEBUG:boto:Data: 2023-07-29T00:35:27.325 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.326 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:27.326 DEBUG:boto:Port: 8000 2023-07-29T00:35:27.326 DEBUG:boto:Params: {} 2023-07-29T00:35:27.327 DEBUG:boto:Token: None 2023-07-29T00:35:27.327 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/ 2023-07-29T00:35:27.328 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A= 2023-07-29T00:35:27.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A=', 'Content-Length': '0'} 2023-07-29T00:35:27.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ffc31ffcc7ddd0d-0064c45ecf-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.332 DEBUG:boto:b'hiiqos-59obj1161000falseobj1162023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:27.333 DEBUG:boto:path=/hiiqos-59/obj116 2023-07-29T00:35:27.333 DEBUG:boto:auth_path=/hiiqos-59/obj116 2023-07-29T00:35:27.334 DEBUG:boto:Method: HEAD 2023-07-29T00:35:27.334 DEBUG:boto:Path: /hiiqos-59/obj116 2023-07-29T00:35:27.334 DEBUG:boto:Data: 2023-07-29T00:35:27.335 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.335 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:27.335 DEBUG:boto:Port: 8001 2023-07-29T00:35:27.336 DEBUG:boto:Params: {} 2023-07-29T00:35:27.336 DEBUG:boto:Token: None 2023-07-29T00:35:27.336 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj116 2023-07-29T00:35:27.337 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TOCMnSIhL1VgP1o5mDEyHt/SVQ8= 2023-07-29T00:35:27.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TOCMnSIhL1VgP1o5mDEyHt/SVQ8=', 'Content-Length': '0'} 2023-07-29T00:35:27.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d7f7faa852fe929-0064c45ecf-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.340 DEBUG:boto:path=/hiiqos-59/obj116 2023-07-29T00:35:27.340 DEBUG:boto:auth_path=/hiiqos-59/obj116 2023-07-29T00:35:27.340 DEBUG:boto:Method: HEAD 2023-07-29T00:35:27.341 DEBUG:boto:Path: /hiiqos-59/obj116 2023-07-29T00:35:27.341 DEBUG:boto:Data: 2023-07-29T00:35:27.341 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.342 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:27.342 DEBUG:boto:Port: 8000 2023-07-29T00:35:27.342 DEBUG:boto:Params: {} 2023-07-29T00:35:27.343 DEBUG:boto:Token: None 2023-07-29T00:35:27.343 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj116 2023-07-29T00:35:27.344 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TOCMnSIhL1VgP1o5mDEyHt/SVQ8= 2023-07-29T00:35:27.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TOCMnSIhL1VgP1o5mDEyHt/SVQ8=', 'Content-Length': '0'} 2023-07-29T00:35:27.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b957cdb781a928d8-0064c45ecf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-07-29T00:35:27.347 DEBUG:boto:path=/hiiqos-59/obj116 2023-07-29T00:35:27.347 DEBUG:boto:auth_path=/hiiqos-59/obj116 2023-07-29T00:35:27.347 DEBUG:boto:Method: GET 2023-07-29T00:35:27.348 DEBUG:boto:Path: /hiiqos-59/obj116 2023-07-29T00:35:27.348 DEBUG:boto:Data: 2023-07-29T00:35:27.349 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.349 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:27.349 DEBUG:boto:Port: 8001 2023-07-29T00:35:27.350 DEBUG:boto:Params: {} 2023-07-29T00:35:27.350 DEBUG:boto:Token: None 2023-07-29T00:35:27.350 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj116 2023-07-29T00:35:27.351 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:65EiehGc/xERloQIjkPiWJh8QjE= 2023-07-29T00:35:27.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:65EiehGc/xERloQIjkPiWJh8QjE=', 'Content-Length': '0'} 2023-07-29T00:35:27.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011d28feae6832edc-0064c45ecf-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.353 DEBUG:boto:path=/hiiqos-59/obj116 2023-07-29T00:35:27.354 DEBUG:boto:auth_path=/hiiqos-59/obj116 2023-07-29T00:35:27.354 DEBUG:boto:Method: GET 2023-07-29T00:35:27.355 DEBUG:boto:Path: /hiiqos-59/obj116 2023-07-29T00:35:27.355 DEBUG:boto:Data: 2023-07-29T00:35:27.355 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.356 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:27.356 DEBUG:boto:Port: 8000 2023-07-29T00:35:27.356 DEBUG:boto:Params: {} 2023-07-29T00:35:27.357 DEBUG:boto:Token: None 2023-07-29T00:35:27.357 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj116 2023-07-29T00:35:27.358 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:65EiehGc/xERloQIjkPiWJh8QjE= 2023-07-29T00:35:27.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:65EiehGc/xERloQIjkPiWJh8QjE=', 'Content-Length': '0'} 2023-07-29T00:35:27.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf78f5f6ded7e3b3-0064c45ecf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.360 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:27.361 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:27.361 DEBUG:boto:path=/hiiqos-59/?prefix=obj116 2023-07-29T00:35:27.361 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj116 2023-07-29T00:35:27.362 DEBUG:boto:Method: GET 2023-07-29T00:35:27.362 DEBUG:boto:Path: /hiiqos-59/?prefix=obj116 2023-07-29T00:35:27.362 DEBUG:boto:Data: 2023-07-29T00:35:27.363 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.363 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:27.364 DEBUG:boto:Port: 8001 2023-07-29T00:35:27.364 DEBUG:boto:Params: {} 2023-07-29T00:35:27.364 DEBUG:boto:Token: None 2023-07-29T00:35:27.365 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/ 2023-07-29T00:35:27.365 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A= 2023-07-29T00:35:27.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A=', 'Content-Length': '0'} 2023-07-29T00:35:27.368 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8a643f8d4da8982-0064c45ecf-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.408 DEBUG:boto:b'hiiqos-59obj1161000falseobj1162023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:27.409 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:27.409 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:27.410 DEBUG:boto:path=/hiiqos-59/?prefix=obj116 2023-07-29T00:35:27.410 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj116 2023-07-29T00:35:27.411 DEBUG:boto:Method: GET 2023-07-29T00:35:27.411 DEBUG:boto:Path: /hiiqos-59/?prefix=obj116 2023-07-29T00:35:27.411 DEBUG:boto:Data: 2023-07-29T00:35:27.412 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.412 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:27.412 DEBUG:boto:Port: 8000 2023-07-29T00:35:27.413 DEBUG:boto:Params: {} 2023-07-29T00:35:27.413 DEBUG:boto:Token: None 2023-07-29T00:35:27.413 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/ 2023-07-29T00:35:27.414 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A= 2023-07-29T00:35:27.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A=', 'Content-Length': '0'} 2023-07-29T00:35:27.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebbe49e784d6975d-0064c45ecf-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.418 DEBUG:boto:b'hiiqos-59obj1161000falseobj1162023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:27.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-07-29T00:35:27.419 DEBUG:boto:path=/hiiqos-59/obj145 2023-07-29T00:35:27.420 DEBUG:boto:auth_path=/hiiqos-59/obj145 2023-07-29T00:35:27.420 DEBUG:boto:path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:35:27.420 DEBUG:boto:auth_path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:35:27.421 DEBUG:boto:Method: GET 2023-07-29T00:35:27.421 DEBUG:boto:Path: /hiiqos-59/obj145?versionId=null 2023-07-29T00:35:27.421 DEBUG:boto:Data: 2023-07-29T00:35:27.422 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.422 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:27.423 DEBUG:boto:Port: 8001 2023-07-29T00:35:27.423 DEBUG:boto:Params: {} 2023-07-29T00:35:27.423 DEBUG:boto:Token: None 2023-07-29T00:35:27.424 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj145?versionId=null 2023-07-29T00:35:27.424 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PMq8AJ6nqBArniUSGd8Zc5so+/o= 2023-07-29T00:35:27.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PMq8AJ6nqBArniUSGd8Zc5so+/o=', 'Content-Length': '0'} 2023-07-29T00:35:27.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a489fa502b053c6-0064c45ecf-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.427 DEBUG:boto:path=/hiiqos-59/obj145 2023-07-29T00:35:27.428 DEBUG:boto:auth_path=/hiiqos-59/obj145 2023-07-29T00:35:27.428 DEBUG:boto:path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:35:27.428 DEBUG:boto:auth_path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:35:27.429 DEBUG:boto:Method: GET 2023-07-29T00:35:27.429 DEBUG:boto:Path: /hiiqos-59/obj145?versionId=null 2023-07-29T00:35:27.430 DEBUG:boto:Data: 2023-07-29T00:35:27.430 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.430 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:27.431 DEBUG:boto:Port: 8000 2023-07-29T00:35:27.431 DEBUG:boto:Params: {} 2023-07-29T00:35:27.431 DEBUG:boto:Token: None 2023-07-29T00:35:27.432 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj145?versionId=null 2023-07-29T00:35:27.432 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PMq8AJ6nqBArniUSGd8Zc5so+/o= 2023-07-29T00:35:27.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PMq8AJ6nqBArniUSGd8Zc5so+/o=', 'Content-Length': '0'} 2023-07-29T00:35:27.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f870fb122126dbd2-0064c45ecf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.476 DEBUG:boto:path=/hiiqos-59/obj145 2023-07-29T00:35:27.477 DEBUG:boto:auth_path=/hiiqos-59/obj145 2023-07-29T00:35:27.477 DEBUG:boto:path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:35:27.477 DEBUG:boto:auth_path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:35:27.478 DEBUG:boto:Method: HEAD 2023-07-29T00:35:27.478 DEBUG:boto:Path: /hiiqos-59/obj145?versionId=null 2023-07-29T00:35:27.478 DEBUG:boto:Data: 2023-07-29T00:35:27.479 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.479 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:27.479 DEBUG:boto:Port: 8001 2023-07-29T00:35:27.480 DEBUG:boto:Params: {} 2023-07-29T00:35:27.480 DEBUG:boto:Token: None 2023-07-29T00:35:27.480 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj145?versionId=null 2023-07-29T00:35:27.481 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DuKO10ehIG2Lhr0R3eMHz0aUGmE= 2023-07-29T00:35:27.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DuKO10ehIG2Lhr0R3eMHz0aUGmE=', 'Content-Length': '0'} 2023-07-29T00:35:27.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d77d126fca9a11e6-0064c45ecf-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.484 DEBUG:boto:path=/hiiqos-59/obj145 2023-07-29T00:35:27.484 DEBUG:boto:auth_path=/hiiqos-59/obj145 2023-07-29T00:35:27.484 DEBUG:boto:path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:35:27.485 DEBUG:boto:auth_path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:35:27.485 DEBUG:boto:Method: HEAD 2023-07-29T00:35:27.485 DEBUG:boto:Path: /hiiqos-59/obj145?versionId=null 2023-07-29T00:35:27.486 DEBUG:boto:Data: 2023-07-29T00:35:27.486 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.486 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:27.487 DEBUG:boto:Port: 8000 2023-07-29T00:35:27.487 DEBUG:boto:Params: {} 2023-07-29T00:35:27.487 DEBUG:boto:Token: None 2023-07-29T00:35:27.488 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj145?versionId=null 2023-07-29T00:35:27.488 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DuKO10ehIG2Lhr0R3eMHz0aUGmE= 2023-07-29T00:35:27.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DuKO10ehIG2Lhr0R3eMHz0aUGmE=', 'Content-Length': '0'} 2023-07-29T00:35:27.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b180755d8b3388c9-0064c45ecf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-07-29T00:35:27.491 DEBUG:boto:path=/hiiqos-59/obj145 2023-07-29T00:35:27.491 DEBUG:boto:auth_path=/hiiqos-59/obj145 2023-07-29T00:35:27.492 DEBUG:boto:path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:35:27.492 DEBUG:boto:auth_path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:35:27.493 DEBUG:boto:Method: GET 2023-07-29T00:35:27.493 DEBUG:boto:Path: /hiiqos-59/obj145?versionId=null 2023-07-29T00:35:27.493 DEBUG:boto:Data: 2023-07-29T00:35:27.494 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.494 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:27.494 DEBUG:boto:Port: 8001 2023-07-29T00:35:27.495 DEBUG:boto:Params: {} 2023-07-29T00:35:27.495 DEBUG:boto:Token: None 2023-07-29T00:35:27.495 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj145?versionId=null 2023-07-29T00:35:27.496 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PMq8AJ6nqBArniUSGd8Zc5so+/o= 2023-07-29T00:35:27.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PMq8AJ6nqBArniUSGd8Zc5so+/o=', 'Content-Length': '0'} 2023-07-29T00:35:27.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005048c1126fd59fcb-0064c45ecf-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.499 DEBUG:boto:path=/hiiqos-59/obj145 2023-07-29T00:35:27.499 DEBUG:boto:auth_path=/hiiqos-59/obj145 2023-07-29T00:35:27.499 DEBUG:boto:path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:35:27.500 DEBUG:boto:auth_path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:35:27.500 DEBUG:boto:Method: GET 2023-07-29T00:35:27.500 DEBUG:boto:Path: /hiiqos-59/obj145?versionId=null 2023-07-29T00:35:27.501 DEBUG:boto:Data: 2023-07-29T00:35:27.501 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.502 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:27.502 DEBUG:boto:Port: 8000 2023-07-29T00:35:27.502 DEBUG:boto:Params: {} 2023-07-29T00:35:27.503 DEBUG:boto:Token: None 2023-07-29T00:35:27.503 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj145?versionId=null 2023-07-29T00:35:27.503 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PMq8AJ6nqBArniUSGd8Zc5so+/o= 2023-07-29T00:35:27.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PMq8AJ6nqBArniUSGd8Zc5so+/o=', 'Content-Length': '0'} 2023-07-29T00:35:27.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000928faf7564543a57-0064c45ecf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.506 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:27.507 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:27.507 DEBUG:boto:path=/hiiqos-59/?prefix=obj145 2023-07-29T00:35:27.508 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj145 2023-07-29T00:35:27.508 DEBUG:boto:Method: GET 2023-07-29T00:35:27.508 DEBUG:boto:Path: /hiiqos-59/?prefix=obj145 2023-07-29T00:35:27.509 DEBUG:boto:Data: 2023-07-29T00:35:27.509 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.510 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:27.510 DEBUG:boto:Port: 8001 2023-07-29T00:35:27.510 DEBUG:boto:Params: {} 2023-07-29T00:35:27.511 DEBUG:boto:Token: None 2023-07-29T00:35:27.511 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/ 2023-07-29T00:35:27.512 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A= 2023-07-29T00:35:27.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A=', 'Content-Length': '0'} 2023-07-29T00:35:27.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fbd8c1c0b7d25e3-0064c45ecf-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.515 DEBUG:boto:b'hiiqos-59obj1451000falseobj1452023-07-29T00:35:01.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:27.516 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:27.516 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:27.516 DEBUG:boto:path=/hiiqos-59/?prefix=obj145 2023-07-29T00:35:27.517 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj145 2023-07-29T00:35:27.517 DEBUG:boto:Method: GET 2023-07-29T00:35:27.517 DEBUG:boto:Path: /hiiqos-59/?prefix=obj145 2023-07-29T00:35:27.518 DEBUG:boto:Data: 2023-07-29T00:35:27.518 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.518 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:27.519 DEBUG:boto:Port: 8000 2023-07-29T00:35:27.519 DEBUG:boto:Params: {} 2023-07-29T00:35:27.520 DEBUG:boto:Token: None 2023-07-29T00:35:27.520 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/ 2023-07-29T00:35:27.520 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A= 2023-07-29T00:35:27.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A=', 'Content-Length': '0'} 2023-07-29T00:35:27.524 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005be28c737d5d18bb-0064c45ecf-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.525 DEBUG:boto:b'hiiqos-59obj1451000falseobj1452023-07-29T00:35:01.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:27.525 DEBUG:boto:path=/hiiqos-59/obj145 2023-07-29T00:35:27.526 DEBUG:boto:auth_path=/hiiqos-59/obj145 2023-07-29T00:35:27.526 DEBUG:boto:Method: HEAD 2023-07-29T00:35:27.526 DEBUG:boto:Path: /hiiqos-59/obj145 2023-07-29T00:35:27.527 DEBUG:boto:Data: 2023-07-29T00:35:27.527 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.527 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:27.528 DEBUG:boto:Port: 8001 2023-07-29T00:35:27.528 DEBUG:boto:Params: {} 2023-07-29T00:35:27.528 DEBUG:boto:Token: None 2023-07-29T00:35:27.529 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj145 2023-07-29T00:35:27.529 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:plrmBw+a//ljuWrTjdEGXpU2YDs= 2023-07-29T00:35:27.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:plrmBw+a//ljuWrTjdEGXpU2YDs=', 'Content-Length': '0'} 2023-07-29T00:35:27.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8a4487ed09970d4-0064c45ecf-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.532 DEBUG:boto:path=/hiiqos-59/obj145 2023-07-29T00:35:27.532 DEBUG:boto:auth_path=/hiiqos-59/obj145 2023-07-29T00:35:27.533 DEBUG:boto:Method: HEAD 2023-07-29T00:35:27.533 DEBUG:boto:Path: /hiiqos-59/obj145 2023-07-29T00:35:27.533 DEBUG:boto:Data: 2023-07-29T00:35:27.534 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.534 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:27.534 DEBUG:boto:Port: 8000 2023-07-29T00:35:27.535 DEBUG:boto:Params: {} 2023-07-29T00:35:27.535 DEBUG:boto:Token: None 2023-07-29T00:35:27.535 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj145 2023-07-29T00:35:27.536 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:plrmBw+a//ljuWrTjdEGXpU2YDs= 2023-07-29T00:35:27.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:plrmBw+a//ljuWrTjdEGXpU2YDs=', 'Content-Length': '0'} 2023-07-29T00:35:27.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9ac40cdb0f7a5b9-0064c45ecf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-07-29T00:35:27.539 DEBUG:boto:path=/hiiqos-59/obj145 2023-07-29T00:35:27.539 DEBUG:boto:auth_path=/hiiqos-59/obj145 2023-07-29T00:35:27.539 DEBUG:boto:Method: GET 2023-07-29T00:35:27.540 DEBUG:boto:Path: /hiiqos-59/obj145 2023-07-29T00:35:27.540 DEBUG:boto:Data: 2023-07-29T00:35:27.541 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.541 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:27.541 DEBUG:boto:Port: 8001 2023-07-29T00:35:27.542 DEBUG:boto:Params: {} 2023-07-29T00:35:27.542 DEBUG:boto:Token: None 2023-07-29T00:35:27.542 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj145 2023-07-29T00:35:27.543 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LnqjdLKdn7v/Fa4nRv3mxDyaVO0= 2023-07-29T00:35:27.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LnqjdLKdn7v/Fa4nRv3mxDyaVO0=', 'Content-Length': '0'} 2023-07-29T00:35:27.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1f05812cc8ea6bc-0064c45ecf-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.546 DEBUG:boto:path=/hiiqos-59/obj145 2023-07-29T00:35:27.546 DEBUG:boto:auth_path=/hiiqos-59/obj145 2023-07-29T00:35:27.546 DEBUG:boto:Method: GET 2023-07-29T00:35:27.547 DEBUG:boto:Path: /hiiqos-59/obj145 2023-07-29T00:35:27.547 DEBUG:boto:Data: 2023-07-29T00:35:27.547 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.548 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:27.548 DEBUG:boto:Port: 8000 2023-07-29T00:35:27.549 DEBUG:boto:Params: {} 2023-07-29T00:35:27.549 DEBUG:boto:Token: None 2023-07-29T00:35:27.549 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj145 2023-07-29T00:35:27.550 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LnqjdLKdn7v/Fa4nRv3mxDyaVO0= 2023-07-29T00:35:27.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LnqjdLKdn7v/Fa4nRv3mxDyaVO0=', 'Content-Length': '0'} 2023-07-29T00:35:27.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8d38fe20fcbfe48-0064c45ecf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.552 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:27.553 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:27.553 DEBUG:boto:path=/hiiqos-59/?prefix=obj145 2023-07-29T00:35:27.553 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj145 2023-07-29T00:35:27.554 DEBUG:boto:Method: GET 2023-07-29T00:35:27.554 DEBUG:boto:Path: /hiiqos-59/?prefix=obj145 2023-07-29T00:35:27.555 DEBUG:boto:Data: 2023-07-29T00:35:27.555 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.555 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:27.556 DEBUG:boto:Port: 8001 2023-07-29T00:35:27.556 DEBUG:boto:Params: {} 2023-07-29T00:35:27.556 DEBUG:boto:Token: None 2023-07-29T00:35:27.557 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/ 2023-07-29T00:35:27.557 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A= 2023-07-29T00:35:27.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A=', 'Content-Length': '0'} 2023-07-29T00:35:27.560 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5cffdb0dcf35dc9-0064c45ecf-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.561 DEBUG:boto:b'hiiqos-59obj1451000falseobj1452023-07-29T00:35:01.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:27.561 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:27.562 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:27.562 DEBUG:boto:path=/hiiqos-59/?prefix=obj145 2023-07-29T00:35:27.562 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj145 2023-07-29T00:35:27.563 DEBUG:boto:Method: GET 2023-07-29T00:35:27.563 DEBUG:boto:Path: /hiiqos-59/?prefix=obj145 2023-07-29T00:35:27.563 DEBUG:boto:Data: 2023-07-29T00:35:27.564 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.564 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:27.564 DEBUG:boto:Port: 8000 2023-07-29T00:35:27.565 DEBUG:boto:Params: {} 2023-07-29T00:35:27.565 DEBUG:boto:Token: None 2023-07-29T00:35:27.566 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/ 2023-07-29T00:35:27.566 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A= 2023-07-29T00:35:27.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A=', 'Content-Length': '0'} 2023-07-29T00:35:27.569 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005357b57a793980fe-0064c45ecf-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.570 DEBUG:boto:b'hiiqos-59obj1451000falseobj1452023-07-29T00:35:01.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:27.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-07-29T00:35:27.571 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:35:27.571 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:35:27.572 DEBUG:boto:path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:35:27.572 DEBUG:boto:auth_path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:35:27.572 DEBUG:boto:Method: GET 2023-07-29T00:35:27.573 DEBUG:boto:Path: /hiiqos-59/obj17?versionId=null 2023-07-29T00:35:27.573 DEBUG:boto:Data: 2023-07-29T00:35:27.573 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.574 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:27.574 DEBUG:boto:Port: 8001 2023-07-29T00:35:27.574 DEBUG:boto:Params: {} 2023-07-29T00:35:27.575 DEBUG:boto:Token: None 2023-07-29T00:35:27.575 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj17?versionId=null 2023-07-29T00:35:27.576 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vaoycmpMzNwozKCyJH8bbKfaXKs= 2023-07-29T00:35:27.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vaoycmpMzNwozKCyJH8bbKfaXKs=', 'Content-Length': '0'} 2023-07-29T00:35:27.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037f41fb798ccce3b-0064c45ecf-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.620 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:35:27.621 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:35:27.621 DEBUG:boto:path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:35:27.621 DEBUG:boto:auth_path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:35:27.622 DEBUG:boto:Method: GET 2023-07-29T00:35:27.622 DEBUG:boto:Path: /hiiqos-59/obj17?versionId=null 2023-07-29T00:35:27.622 DEBUG:boto:Data: 2023-07-29T00:35:27.623 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.623 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:27.623 DEBUG:boto:Port: 8000 2023-07-29T00:35:27.624 DEBUG:boto:Params: {} 2023-07-29T00:35:27.624 DEBUG:boto:Token: None 2023-07-29T00:35:27.625 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj17?versionId=null 2023-07-29T00:35:27.625 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vaoycmpMzNwozKCyJH8bbKfaXKs= 2023-07-29T00:35:27.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vaoycmpMzNwozKCyJH8bbKfaXKs=', 'Content-Length': '0'} 2023-07-29T00:35:27.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003196bdd9f8ba252d-0064c45ecf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.628 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:35:27.629 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:35:27.629 DEBUG:boto:path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:35:27.629 DEBUG:boto:auth_path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:35:27.630 DEBUG:boto:Method: HEAD 2023-07-29T00:35:27.630 DEBUG:boto:Path: /hiiqos-59/obj17?versionId=null 2023-07-29T00:35:27.630 DEBUG:boto:Data: 2023-07-29T00:35:27.631 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.631 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:27.631 DEBUG:boto:Port: 8001 2023-07-29T00:35:27.632 DEBUG:boto:Params: {} 2023-07-29T00:35:27.632 DEBUG:boto:Token: None 2023-07-29T00:35:27.632 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj17?versionId=null 2023-07-29T00:35:27.633 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2Dsgm+BPeTMBqQwVUtYzLeLQJpk= 2023-07-29T00:35:27.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2Dsgm+BPeTMBqQwVUtYzLeLQJpk=', 'Content-Length': '0'} 2023-07-29T00:35:27.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2faaa7581b22b3d-0064c45ecf-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.635 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:35:27.636 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:35:27.636 DEBUG:boto:path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:35:27.636 DEBUG:boto:auth_path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:35:27.637 DEBUG:boto:Method: HEAD 2023-07-29T00:35:27.637 DEBUG:boto:Path: /hiiqos-59/obj17?versionId=null 2023-07-29T00:35:27.637 DEBUG:boto:Data: 2023-07-29T00:35:27.638 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.638 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:27.638 DEBUG:boto:Port: 8000 2023-07-29T00:35:27.639 DEBUG:boto:Params: {} 2023-07-29T00:35:27.639 DEBUG:boto:Token: None 2023-07-29T00:35:27.640 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj17?versionId=null 2023-07-29T00:35:27.640 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2Dsgm+BPeTMBqQwVUtYzLeLQJpk= 2023-07-29T00:35:27.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2Dsgm+BPeTMBqQwVUtYzLeLQJpk=', 'Content-Length': '0'} 2023-07-29T00:35:27.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009582db4bae0dacdf-0064c45ecf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-07-29T00:35:27.643 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:35:27.643 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:35:27.644 DEBUG:boto:path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:35:27.644 DEBUG:boto:auth_path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:35:27.644 DEBUG:boto:Method: GET 2023-07-29T00:35:27.645 DEBUG:boto:Path: /hiiqos-59/obj17?versionId=null 2023-07-29T00:35:27.645 DEBUG:boto:Data: 2023-07-29T00:35:27.646 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.646 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:27.646 DEBUG:boto:Port: 8001 2023-07-29T00:35:27.647 DEBUG:boto:Params: {} 2023-07-29T00:35:27.647 DEBUG:boto:Token: None 2023-07-29T00:35:27.647 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj17?versionId=null 2023-07-29T00:35:27.648 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vaoycmpMzNwozKCyJH8bbKfaXKs= 2023-07-29T00:35:27.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vaoycmpMzNwozKCyJH8bbKfaXKs=', 'Content-Length': '0'} 2023-07-29T00:35:27.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000854b1c0ef7141d9a-0064c45ecf-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.650 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:35:27.651 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:35:27.651 DEBUG:boto:path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:35:27.651 DEBUG:boto:auth_path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:35:27.652 DEBUG:boto:Method: GET 2023-07-29T00:35:27.652 DEBUG:boto:Path: /hiiqos-59/obj17?versionId=null 2023-07-29T00:35:27.653 DEBUG:boto:Data: 2023-07-29T00:35:27.653 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.653 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:27.654 DEBUG:boto:Port: 8000 2023-07-29T00:35:27.654 DEBUG:boto:Params: {} 2023-07-29T00:35:27.654 DEBUG:boto:Token: None 2023-07-29T00:35:27.655 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj17?versionId=null 2023-07-29T00:35:27.655 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vaoycmpMzNwozKCyJH8bbKfaXKs= 2023-07-29T00:35:27.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vaoycmpMzNwozKCyJH8bbKfaXKs=', 'Content-Length': '0'} 2023-07-29T00:35:27.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a5a34ae07090372-0064c45ecf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.658 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:27.658 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:27.659 DEBUG:boto:path=/hiiqos-59/?prefix=obj17 2023-07-29T00:35:27.659 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj17 2023-07-29T00:35:27.660 DEBUG:boto:Method: GET 2023-07-29T00:35:27.660 DEBUG:boto:Path: /hiiqos-59/?prefix=obj17 2023-07-29T00:35:27.660 DEBUG:boto:Data: 2023-07-29T00:35:27.661 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.661 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:27.661 DEBUG:boto:Port: 8001 2023-07-29T00:35:27.662 DEBUG:boto:Params: {} 2023-07-29T00:35:27.662 DEBUG:boto:Token: None 2023-07-29T00:35:27.662 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/ 2023-07-29T00:35:27.663 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A= 2023-07-29T00:35:27.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A=', 'Content-Length': '0'} 2023-07-29T00:35:27.665 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e5c524ce4cf8ed3-0064c45ecf-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.666 DEBUG:boto:b'hiiqos-59obj171000falseobj172023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:27.666 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:27.667 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:27.667 DEBUG:boto:path=/hiiqos-59/?prefix=obj17 2023-07-29T00:35:27.668 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj17 2023-07-29T00:35:27.668 DEBUG:boto:Method: GET 2023-07-29T00:35:27.668 DEBUG:boto:Path: /hiiqos-59/?prefix=obj17 2023-07-29T00:35:27.669 DEBUG:boto:Data: 2023-07-29T00:35:27.669 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.669 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:27.670 DEBUG:boto:Port: 8000 2023-07-29T00:35:27.670 DEBUG:boto:Params: {} 2023-07-29T00:35:27.670 DEBUG:boto:Token: None 2023-07-29T00:35:27.671 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/ 2023-07-29T00:35:27.671 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A= 2023-07-29T00:35:27.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A=', 'Content-Length': '0'} 2023-07-29T00:35:27.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d156b37be0fba34-0064c45ecf-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.675 DEBUG:boto:b'hiiqos-59obj171000falseobj172023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:27.676 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:35:27.676 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:35:27.677 DEBUG:boto:Method: HEAD 2023-07-29T00:35:27.677 DEBUG:boto:Path: /hiiqos-59/obj17 2023-07-29T00:35:27.677 DEBUG:boto:Data: 2023-07-29T00:35:27.678 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.678 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:27.678 DEBUG:boto:Port: 8001 2023-07-29T00:35:27.679 DEBUG:boto:Params: {} 2023-07-29T00:35:27.679 DEBUG:boto:Token: None 2023-07-29T00:35:27.679 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj17 2023-07-29T00:35:27.680 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vcPzu9mHot/0iDya8SiOyB2s2MU= 2023-07-29T00:35:27.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vcPzu9mHot/0iDya8SiOyB2s2MU=', 'Content-Length': '0'} 2023-07-29T00:35:27.682 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd3a42ba82ff88b2-0064c45ecf-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.682 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:35:27.683 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:35:27.683 DEBUG:boto:Method: HEAD 2023-07-29T00:35:27.683 DEBUG:boto:Path: /hiiqos-59/obj17 2023-07-29T00:35:27.684 DEBUG:boto:Data: 2023-07-29T00:35:27.684 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.684 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:27.685 DEBUG:boto:Port: 8000 2023-07-29T00:35:27.685 DEBUG:boto:Params: {} 2023-07-29T00:35:27.685 DEBUG:boto:Token: None 2023-07-29T00:35:27.686 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj17 2023-07-29T00:35:27.686 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vcPzu9mHot/0iDya8SiOyB2s2MU= 2023-07-29T00:35:27.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vcPzu9mHot/0iDya8SiOyB2s2MU=', 'Content-Length': '0'} 2023-07-29T00:35:27.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e6deb8e0b50638e-0064c45ecf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-07-29T00:35:27.689 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:35:27.689 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:35:27.690 DEBUG:boto:Method: GET 2023-07-29T00:35:27.690 DEBUG:boto:Path: /hiiqos-59/obj17 2023-07-29T00:35:27.690 DEBUG:boto:Data: 2023-07-29T00:35:27.691 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.691 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:27.691 DEBUG:boto:Port: 8001 2023-07-29T00:35:27.692 DEBUG:boto:Params: {} 2023-07-29T00:35:27.692 DEBUG:boto:Token: None 2023-07-29T00:35:27.693 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj17 2023-07-29T00:35:27.693 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:c5lw1MGlBgmZWadPXBHHdHq400A= 2023-07-29T00:35:27.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c5lw1MGlBgmZWadPXBHHdHq400A=', 'Content-Length': '0'} 2023-07-29T00:35:27.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3369cc02002441a-0064c45ecf-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.696 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:35:27.696 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:35:27.696 DEBUG:boto:Method: GET 2023-07-29T00:35:27.697 DEBUG:boto:Path: /hiiqos-59/obj17 2023-07-29T00:35:27.697 DEBUG:boto:Data: 2023-07-29T00:35:27.698 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.698 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:27.698 DEBUG:boto:Port: 8000 2023-07-29T00:35:27.699 DEBUG:boto:Params: {} 2023-07-29T00:35:27.699 DEBUG:boto:Token: None 2023-07-29T00:35:27.699 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj17 2023-07-29T00:35:27.700 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:c5lw1MGlBgmZWadPXBHHdHq400A= 2023-07-29T00:35:27.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c5lw1MGlBgmZWadPXBHHdHq400A=', 'Content-Length': '0'} 2023-07-29T00:35:27.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000add00d6f35066e8f-0064c45ecf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.703 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:27.703 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:27.703 DEBUG:boto:path=/hiiqos-59/?prefix=obj17 2023-07-29T00:35:27.704 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj17 2023-07-29T00:35:27.704 DEBUG:boto:Method: GET 2023-07-29T00:35:27.704 DEBUG:boto:Path: /hiiqos-59/?prefix=obj17 2023-07-29T00:35:27.705 DEBUG:boto:Data: 2023-07-29T00:35:27.705 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.705 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:27.706 DEBUG:boto:Port: 8001 2023-07-29T00:35:27.706 DEBUG:boto:Params: {} 2023-07-29T00:35:27.707 DEBUG:boto:Token: None 2023-07-29T00:35:27.707 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/ 2023-07-29T00:35:27.707 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A= 2023-07-29T00:35:27.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A=', 'Content-Length': '0'} 2023-07-29T00:35:27.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd49e193a5cd1232-0064c45ecf-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.711 DEBUG:boto:b'hiiqos-59obj171000falseobj172023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:27.711 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:27.712 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:27.712 DEBUG:boto:path=/hiiqos-59/?prefix=obj17 2023-07-29T00:35:27.712 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj17 2023-07-29T00:35:27.713 DEBUG:boto:Method: GET 2023-07-29T00:35:27.713 DEBUG:boto:Path: /hiiqos-59/?prefix=obj17 2023-07-29T00:35:27.714 DEBUG:boto:Data: 2023-07-29T00:35:27.714 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.714 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:27.715 DEBUG:boto:Port: 8000 2023-07-29T00:35:27.715 DEBUG:boto:Params: {} 2023-07-29T00:35:27.715 DEBUG:boto:Token: None 2023-07-29T00:35:27.716 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/ 2023-07-29T00:35:27.716 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A= 2023-07-29T00:35:27.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A=', 'Content-Length': '0'} 2023-07-29T00:35:27.719 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bc0e4d3badc05fe-0064c45ecf-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.720 DEBUG:boto:b'hiiqos-59obj171000falseobj172023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:27.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-07-29T00:35:27.721 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:35:27.721 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:35:27.722 DEBUG:boto:path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:35:27.722 DEBUG:boto:auth_path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:35:27.722 DEBUG:boto:Method: GET 2023-07-29T00:35:27.723 DEBUG:boto:Path: /hiiqos-59/obj19?versionId=null 2023-07-29T00:35:27.723 DEBUG:boto:Data: 2023-07-29T00:35:27.723 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.724 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:27.724 DEBUG:boto:Port: 8001 2023-07-29T00:35:27.725 DEBUG:boto:Params: {} 2023-07-29T00:35:27.725 DEBUG:boto:Token: None 2023-07-29T00:35:27.725 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj19?versionId=null 2023-07-29T00:35:27.726 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6xRNJdf/C+2ILoB+dxH8jRmuiHY= 2023-07-29T00:35:27.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6xRNJdf/C+2ILoB+dxH8jRmuiHY=', 'Content-Length': '0'} 2023-07-29T00:35:27.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5e353d4479f7538-0064c45ecf-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.728 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:35:27.729 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:35:27.729 DEBUG:boto:path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:35:27.730 DEBUG:boto:auth_path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:35:27.730 DEBUG:boto:Method: GET 2023-07-29T00:35:27.730 DEBUG:boto:Path: /hiiqos-59/obj19?versionId=null 2023-07-29T00:35:27.731 DEBUG:boto:Data: 2023-07-29T00:35:27.731 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.731 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:27.732 DEBUG:boto:Port: 8000 2023-07-29T00:35:27.732 DEBUG:boto:Params: {} 2023-07-29T00:35:27.732 DEBUG:boto:Token: None 2023-07-29T00:35:27.733 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj19?versionId=null 2023-07-29T00:35:27.733 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6xRNJdf/C+2ILoB+dxH8jRmuiHY= 2023-07-29T00:35:27.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6xRNJdf/C+2ILoB+dxH8jRmuiHY=', 'Content-Length': '0'} 2023-07-29T00:35:27.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000604534ebbe177990-0064c45ecf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.736 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:35:27.736 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:35:27.737 DEBUG:boto:path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:35:27.737 DEBUG:boto:auth_path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:35:27.737 DEBUG:boto:Method: HEAD 2023-07-29T00:35:27.738 DEBUG:boto:Path: /hiiqos-59/obj19?versionId=null 2023-07-29T00:35:27.738 DEBUG:boto:Data: 2023-07-29T00:35:27.738 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.739 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:27.739 DEBUG:boto:Port: 8001 2023-07-29T00:35:27.739 DEBUG:boto:Params: {} 2023-07-29T00:35:27.740 DEBUG:boto:Token: None 2023-07-29T00:35:27.740 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj19?versionId=null 2023-07-29T00:35:27.741 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OCbLI4hG/7UZDmNirGDWIkeInTc= 2023-07-29T00:35:27.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OCbLI4hG/7UZDmNirGDWIkeInTc=', 'Content-Length': '0'} 2023-07-29T00:35:27.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082a887db023ea7de-0064c45ecf-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.743 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:35:27.743 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:35:27.744 DEBUG:boto:path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:35:27.744 DEBUG:boto:auth_path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:35:27.744 DEBUG:boto:Method: HEAD 2023-07-29T00:35:27.745 DEBUG:boto:Path: /hiiqos-59/obj19?versionId=null 2023-07-29T00:35:27.745 DEBUG:boto:Data: 2023-07-29T00:35:27.745 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.746 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:27.746 DEBUG:boto:Port: 8000 2023-07-29T00:35:27.746 DEBUG:boto:Params: {} 2023-07-29T00:35:27.747 DEBUG:boto:Token: None 2023-07-29T00:35:27.747 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj19?versionId=null 2023-07-29T00:35:27.747 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OCbLI4hG/7UZDmNirGDWIkeInTc= 2023-07-29T00:35:27.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OCbLI4hG/7UZDmNirGDWIkeInTc=', 'Content-Length': '0'} 2023-07-29T00:35:27.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001af68676f21220b4-0064c45ecf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-07-29T00:35:27.750 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:35:27.751 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:35:27.751 DEBUG:boto:path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:35:27.751 DEBUG:boto:auth_path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:35:27.752 DEBUG:boto:Method: GET 2023-07-29T00:35:27.752 DEBUG:boto:Path: /hiiqos-59/obj19?versionId=null 2023-07-29T00:35:27.752 DEBUG:boto:Data: 2023-07-29T00:35:27.753 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.753 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:27.753 DEBUG:boto:Port: 8001 2023-07-29T00:35:27.754 DEBUG:boto:Params: {} 2023-07-29T00:35:27.754 DEBUG:boto:Token: None 2023-07-29T00:35:27.755 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj19?versionId=null 2023-07-29T00:35:27.755 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6xRNJdf/C+2ILoB+dxH8jRmuiHY= 2023-07-29T00:35:27.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6xRNJdf/C+2ILoB+dxH8jRmuiHY=', 'Content-Length': '0'} 2023-07-29T00:35:27.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff5ae60353fe1f93-0064c45ecf-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.758 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:35:27.758 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:35:27.758 DEBUG:boto:path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:35:27.759 DEBUG:boto:auth_path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:35:27.759 DEBUG:boto:Method: GET 2023-07-29T00:35:27.759 DEBUG:boto:Path: /hiiqos-59/obj19?versionId=null 2023-07-29T00:35:27.760 DEBUG:boto:Data: 2023-07-29T00:35:27.760 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.760 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:27.761 DEBUG:boto:Port: 8000 2023-07-29T00:35:27.761 DEBUG:boto:Params: {} 2023-07-29T00:35:27.762 DEBUG:boto:Token: None 2023-07-29T00:35:27.762 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj19?versionId=null 2023-07-29T00:35:27.762 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6xRNJdf/C+2ILoB+dxH8jRmuiHY= 2023-07-29T00:35:27.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6xRNJdf/C+2ILoB+dxH8jRmuiHY=', 'Content-Length': '0'} 2023-07-29T00:35:27.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea23be3578ae1e44-0064c45ecf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.765 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:27.766 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:27.766 DEBUG:boto:path=/hiiqos-59/?prefix=obj19 2023-07-29T00:35:27.766 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj19 2023-07-29T00:35:27.767 DEBUG:boto:Method: GET 2023-07-29T00:35:27.767 DEBUG:boto:Path: /hiiqos-59/?prefix=obj19 2023-07-29T00:35:27.767 DEBUG:boto:Data: 2023-07-29T00:35:27.768 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.768 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:27.769 DEBUG:boto:Port: 8001 2023-07-29T00:35:27.769 DEBUG:boto:Params: {} 2023-07-29T00:35:27.769 DEBUG:boto:Token: None 2023-07-29T00:35:27.770 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/ 2023-07-29T00:35:27.770 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A= 2023-07-29T00:35:27.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A=', 'Content-Length': '0'} 2023-07-29T00:35:27.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091f3f09204a0a174-0064c45ecf-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.773 DEBUG:boto:b'hiiqos-59obj191000falseobj192023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:27.774 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:27.774 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:27.774 DEBUG:boto:path=/hiiqos-59/?prefix=obj19 2023-07-29T00:35:27.775 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj19 2023-07-29T00:35:27.775 DEBUG:boto:Method: GET 2023-07-29T00:35:27.775 DEBUG:boto:Path: /hiiqos-59/?prefix=obj19 2023-07-29T00:35:27.776 DEBUG:boto:Data: 2023-07-29T00:35:27.776 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.776 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:27.777 DEBUG:boto:Port: 8000 2023-07-29T00:35:27.777 DEBUG:boto:Params: {} 2023-07-29T00:35:27.778 DEBUG:boto:Token: None 2023-07-29T00:35:27.778 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/ 2023-07-29T00:35:27.778 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A= 2023-07-29T00:35:27.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A=', 'Content-Length': '0'} 2023-07-29T00:35:27.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee9cfebacfbf19f2-0064c45ecf-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.824 DEBUG:boto:b'hiiqos-59obj191000falseobj192023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:27.825 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:35:27.825 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:35:27.825 DEBUG:boto:Method: HEAD 2023-07-29T00:35:27.826 DEBUG:boto:Path: /hiiqos-59/obj19 2023-07-29T00:35:27.826 DEBUG:boto:Data: 2023-07-29T00:35:27.826 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.827 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:27.827 DEBUG:boto:Port: 8001 2023-07-29T00:35:27.827 DEBUG:boto:Params: {} 2023-07-29T00:35:27.828 DEBUG:boto:Token: None 2023-07-29T00:35:27.828 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj19 2023-07-29T00:35:27.829 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S2mwlICmqbkd47RtdA6YVbttKCA= 2023-07-29T00:35:27.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S2mwlICmqbkd47RtdA6YVbttKCA=', 'Content-Length': '0'} 2023-07-29T00:35:27.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0e8ca1b62b9932a-0064c45ecf-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.831 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:35:27.832 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:35:27.832 DEBUG:boto:Method: HEAD 2023-07-29T00:35:27.832 DEBUG:boto:Path: /hiiqos-59/obj19 2023-07-29T00:35:27.833 DEBUG:boto:Data: 2023-07-29T00:35:27.833 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.833 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:27.834 DEBUG:boto:Port: 8000 2023-07-29T00:35:27.834 DEBUG:boto:Params: {} 2023-07-29T00:35:27.834 DEBUG:boto:Token: None 2023-07-29T00:35:27.835 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj19 2023-07-29T00:35:27.835 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S2mwlICmqbkd47RtdA6YVbttKCA= 2023-07-29T00:35:27.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S2mwlICmqbkd47RtdA6YVbttKCA=', 'Content-Length': '0'} 2023-07-29T00:35:27.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9553ee411d69b61-0064c45ecf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-07-29T00:35:27.838 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:35:27.838 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:35:27.839 DEBUG:boto:Method: GET 2023-07-29T00:35:27.839 DEBUG:boto:Path: /hiiqos-59/obj19 2023-07-29T00:35:27.840 DEBUG:boto:Data: 2023-07-29T00:35:27.840 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.840 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:27.841 DEBUG:boto:Port: 8001 2023-07-29T00:35:27.841 DEBUG:boto:Params: {} 2023-07-29T00:35:27.841 DEBUG:boto:Token: None 2023-07-29T00:35:27.842 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj19 2023-07-29T00:35:27.842 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OsR0fJiAeouJqJrmk1Fqu6Vizn4= 2023-07-29T00:35:27.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OsR0fJiAeouJqJrmk1Fqu6Vizn4=', 'Content-Length': '0'} 2023-07-29T00:35:27.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009399dec3e77b1399-0064c45ecf-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.845 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:35:27.845 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:35:27.846 DEBUG:boto:Method: GET 2023-07-29T00:35:27.846 DEBUG:boto:Path: /hiiqos-59/obj19 2023-07-29T00:35:27.846 DEBUG:boto:Data: 2023-07-29T00:35:27.847 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.847 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:27.847 DEBUG:boto:Port: 8000 2023-07-29T00:35:27.848 DEBUG:boto:Params: {} 2023-07-29T00:35:27.848 DEBUG:boto:Token: None 2023-07-29T00:35:27.849 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj19 2023-07-29T00:35:27.849 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OsR0fJiAeouJqJrmk1Fqu6Vizn4= 2023-07-29T00:35:27.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OsR0fJiAeouJqJrmk1Fqu6Vizn4=', 'Content-Length': '0'} 2023-07-29T00:35:27.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0df5f70ce6be0d0-0064c45ecf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.852 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:27.852 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:27.853 DEBUG:boto:path=/hiiqos-59/?prefix=obj19 2023-07-29T00:35:27.853 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj19 2023-07-29T00:35:27.853 DEBUG:boto:Method: GET 2023-07-29T00:35:27.854 DEBUG:boto:Path: /hiiqos-59/?prefix=obj19 2023-07-29T00:35:27.854 DEBUG:boto:Data: 2023-07-29T00:35:27.854 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.855 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:27.855 DEBUG:boto:Port: 8001 2023-07-29T00:35:27.855 DEBUG:boto:Params: {} 2023-07-29T00:35:27.856 DEBUG:boto:Token: None 2023-07-29T00:35:27.856 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/ 2023-07-29T00:35:27.857 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A= 2023-07-29T00:35:27.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A=', 'Content-Length': '0'} 2023-07-29T00:35:27.859 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072fac0818cffa81d-0064c45ecf-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.900 DEBUG:boto:b'hiiqos-59obj191000falseobj192023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:27.901 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:27.901 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:27.902 DEBUG:boto:path=/hiiqos-59/?prefix=obj19 2023-07-29T00:35:27.902 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj19 2023-07-29T00:35:27.902 DEBUG:boto:Method: GET 2023-07-29T00:35:27.903 DEBUG:boto:Path: /hiiqos-59/?prefix=obj19 2023-07-29T00:35:27.903 DEBUG:boto:Data: 2023-07-29T00:35:27.903 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.904 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:27.904 DEBUG:boto:Port: 8000 2023-07-29T00:35:27.904 DEBUG:boto:Params: {} 2023-07-29T00:35:27.905 DEBUG:boto:Token: None 2023-07-29T00:35:27.905 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/ 2023-07-29T00:35:27.906 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A= 2023-07-29T00:35:27.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P2ujyF15pgHq1Rp3s58g9BlUX8A=', 'Content-Length': '0'} 2023-07-29T00:35:27.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025ea32574d75bc55-0064c45ecf-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.910 DEBUG:boto:b'hiiqos-59obj191000falseobj192023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:27.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-07-29T00:35:27.911 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:35:27.911 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:35:27.911 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:35:27.912 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:35:27.912 DEBUG:boto:Method: GET 2023-07-29T00:35:27.912 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:35:27.913 DEBUG:boto:Data: 2023-07-29T00:35:27.913 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.913 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:27.914 DEBUG:boto:Port: 8001 2023-07-29T00:35:27.914 DEBUG:boto:Params: {} 2023-07-29T00:35:27.915 DEBUG:boto:Token: None 2023-07-29T00:35:27.915 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:35:27.915 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AWEuhdBOtFvI8Uv0dIkSt0GBVVY= 2023-07-29T00:35:27.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AWEuhdBOtFvI8Uv0dIkSt0GBVVY=', 'Content-Length': '0'} 2023-07-29T00:35:27.917 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a37c5350e094fba6-0064c45ecf-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.918 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:35:27.918 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:35:27.919 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:35:27.919 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:35:27.919 DEBUG:boto:Method: GET 2023-07-29T00:35:27.920 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:35:27.920 DEBUG:boto:Data: 2023-07-29T00:35:27.921 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.921 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:27.921 DEBUG:boto:Port: 8000 2023-07-29T00:35:27.922 DEBUG:boto:Params: {} 2023-07-29T00:35:27.922 DEBUG:boto:Token: None 2023-07-29T00:35:27.922 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:35:27.923 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AWEuhdBOtFvI8Uv0dIkSt0GBVVY= 2023-07-29T00:35:27.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AWEuhdBOtFvI8Uv0dIkSt0GBVVY=', 'Content-Length': '0'} 2023-07-29T00:35:27.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4652a5689ebda71-0064c45ecf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.925 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:35:27.926 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:35:27.926 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:35:27.926 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:35:27.927 DEBUG:boto:Method: HEAD 2023-07-29T00:35:27.927 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:35:27.927 DEBUG:boto:Data: 2023-07-29T00:35:27.928 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.928 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:27.928 DEBUG:boto:Port: 8001 2023-07-29T00:35:27.929 DEBUG:boto:Params: {} 2023-07-29T00:35:27.929 DEBUG:boto:Token: None 2023-07-29T00:35:27.929 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:35:27.930 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+CMKVXl313WFekfreF/ljF97DtE= 2023-07-29T00:35:27.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+CMKVXl313WFekfreF/ljF97DtE=', 'Content-Length': '0'} 2023-07-29T00:35:27.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7ac87b3c152cd90-0064c45ecf-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.932 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:35:27.933 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:35:27.933 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:35:27.933 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:35:27.934 DEBUG:boto:Method: HEAD 2023-07-29T00:35:27.934 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:35:27.934 DEBUG:boto:Data: 2023-07-29T00:35:27.935 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.935 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:27.935 DEBUG:boto:Port: 8000 2023-07-29T00:35:27.936 DEBUG:boto:Params: {} 2023-07-29T00:35:27.936 DEBUG:boto:Token: None 2023-07-29T00:35:27.936 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:35:27.937 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+CMKVXl313WFekfreF/ljF97DtE= 2023-07-29T00:35:27.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+CMKVXl313WFekfreF/ljF97DtE=', 'Content-Length': '0'} 2023-07-29T00:35:27.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000265f12503ddae5c7-0064c45ecf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-07-29T00:35:27.940 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:35:27.940 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:35:27.940 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:35:27.941 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:35:27.941 DEBUG:boto:Method: GET 2023-07-29T00:35:27.941 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:35:27.942 DEBUG:boto:Data: 2023-07-29T00:35:27.942 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.942 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:27.943 DEBUG:boto:Port: 8001 2023-07-29T00:35:27.943 DEBUG:boto:Params: {} 2023-07-29T00:35:27.944 DEBUG:boto:Token: None 2023-07-29T00:35:27.944 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:35:27.944 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AWEuhdBOtFvI8Uv0dIkSt0GBVVY= 2023-07-29T00:35:27.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AWEuhdBOtFvI8Uv0dIkSt0GBVVY=', 'Content-Length': '0'} 2023-07-29T00:35:27.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002554a2aa19b4acba-0064c45ecf-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.947 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:35:27.947 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:35:27.948 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:35:27.948 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:35:27.949 DEBUG:boto:Method: GET 2023-07-29T00:35:27.949 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:35:27.949 DEBUG:boto:Data: 2023-07-29T00:35:27.950 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.950 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:27.950 DEBUG:boto:Port: 8000 2023-07-29T00:35:27.951 DEBUG:boto:Params: {} 2023-07-29T00:35:27.951 DEBUG:boto:Token: None 2023-07-29T00:35:27.952 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:27 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:35:27.952 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AWEuhdBOtFvI8Uv0dIkSt0GBVVY= 2023-07-29T00:35:27.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AWEuhdBOtFvI8Uv0dIkSt0GBVVY=', 'Content-Length': '0'} 2023-07-29T00:35:27.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bae990838c8a889-0064c45ecf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:27.996 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:27.997 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:27.997 DEBUG:boto:path=/hiiqos-59/?prefix=obj22 2023-07-29T00:35:27.997 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj22 2023-07-29T00:35:27.998 DEBUG:boto:Method: GET 2023-07-29T00:35:27.998 DEBUG:boto:Path: /hiiqos-59/?prefix=obj22 2023-07-29T00:35:27.998 DEBUG:boto:Data: 2023-07-29T00:35:27.999 DEBUG:boto:Headers: {} 2023-07-29T00:35:27.999 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:27.999 DEBUG:boto:Port: 8001 2023-07-29T00:35:28.000 DEBUG:boto:Params: {} 2023-07-29T00:35:28.000 DEBUG:boto:Token: None 2023-07-29T00:35:28.001 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/ 2023-07-29T00:35:28.001 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw= 2023-07-29T00:35:28.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw=', 'Content-Length': '0'} 2023-07-29T00:35:28.004 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058ce7518d19b05ea-0064c45ecf-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.004 DEBUG:boto:b'hiiqos-59obj221000falseobj222023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:28.005 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:28.005 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:28.006 DEBUG:boto:path=/hiiqos-59/?prefix=obj22 2023-07-29T00:35:28.006 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj22 2023-07-29T00:35:28.006 DEBUG:boto:Method: GET 2023-07-29T00:35:28.007 DEBUG:boto:Path: /hiiqos-59/?prefix=obj22 2023-07-29T00:35:28.007 DEBUG:boto:Data: 2023-07-29T00:35:28.007 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.008 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:28.008 DEBUG:boto:Port: 8000 2023-07-29T00:35:28.008 DEBUG:boto:Params: {} 2023-07-29T00:35:28.009 DEBUG:boto:Token: None 2023-07-29T00:35:28.009 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/ 2023-07-29T00:35:28.010 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw= 2023-07-29T00:35:28.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw=', 'Content-Length': '0'} 2023-07-29T00:35:28.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078779bad073e4ad1-0064c45ecf-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.014 DEBUG:boto:b'hiiqos-59obj221000falseobj222023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:28.014 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:35:28.014 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:35:28.015 DEBUG:boto:Method: HEAD 2023-07-29T00:35:28.015 DEBUG:boto:Path: /hiiqos-59/obj22 2023-07-29T00:35:28.016 DEBUG:boto:Data: 2023-07-29T00:35:28.016 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.016 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:28.017 DEBUG:boto:Port: 8001 2023-07-29T00:35:28.017 DEBUG:boto:Params: {} 2023-07-29T00:35:28.017 DEBUG:boto:Token: None 2023-07-29T00:35:28.018 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj22 2023-07-29T00:35:28.018 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BSDbPTlRqM6BrhdK5rIgLe8lhKg= 2023-07-29T00:35:28.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BSDbPTlRqM6BrhdK5rIgLe8lhKg=', 'Content-Length': '0'} 2023-07-29T00:35:28.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000728554c354ed4f5-0064c45ecf-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.020 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:35:28.021 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:35:28.021 DEBUG:boto:Method: HEAD 2023-07-29T00:35:28.022 DEBUG:boto:Path: /hiiqos-59/obj22 2023-07-29T00:35:28.022 DEBUG:boto:Data: 2023-07-29T00:35:28.022 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.023 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:28.023 DEBUG:boto:Port: 8000 2023-07-29T00:35:28.023 DEBUG:boto:Params: {} 2023-07-29T00:35:28.024 DEBUG:boto:Token: None 2023-07-29T00:35:28.024 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj22 2023-07-29T00:35:28.024 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BSDbPTlRqM6BrhdK5rIgLe8lhKg= 2023-07-29T00:35:28.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BSDbPTlRqM6BrhdK5rIgLe8lhKg=', 'Content-Length': '0'} 2023-07-29T00:35:28.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b97f226dad6a449-0064c45ecf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-07-29T00:35:28.027 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:35:28.028 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:35:28.028 DEBUG:boto:Method: GET 2023-07-29T00:35:28.028 DEBUG:boto:Path: /hiiqos-59/obj22 2023-07-29T00:35:28.029 DEBUG:boto:Data: 2023-07-29T00:35:28.029 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.029 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:28.030 DEBUG:boto:Port: 8001 2023-07-29T00:35:28.030 DEBUG:boto:Params: {} 2023-07-29T00:35:28.030 DEBUG:boto:Token: None 2023-07-29T00:35:28.031 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj22 2023-07-29T00:35:28.031 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TFW7K9DYNciUzm5f44uVfYtpRzs= 2023-07-29T00:35:28.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TFW7K9DYNciUzm5f44uVfYtpRzs=', 'Content-Length': '0'} 2023-07-29T00:35:28.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d94afd102f3007a5-0064c45ecf-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.034 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:35:28.034 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:35:28.035 DEBUG:boto:Method: GET 2023-07-29T00:35:28.035 DEBUG:boto:Path: /hiiqos-59/obj22 2023-07-29T00:35:28.035 DEBUG:boto:Data: 2023-07-29T00:35:28.036 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.036 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:28.036 DEBUG:boto:Port: 8000 2023-07-29T00:35:28.037 DEBUG:boto:Params: {} 2023-07-29T00:35:28.037 DEBUG:boto:Token: None 2023-07-29T00:35:28.037 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj22 2023-07-29T00:35:28.038 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TFW7K9DYNciUzm5f44uVfYtpRzs= 2023-07-29T00:35:28.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TFW7K9DYNciUzm5f44uVfYtpRzs=', 'Content-Length': '0'} 2023-07-29T00:35:28.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e92ad9c853d80d1d-0064c45ecf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.041 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:28.041 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:28.042 DEBUG:boto:path=/hiiqos-59/?prefix=obj22 2023-07-29T00:35:28.042 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj22 2023-07-29T00:35:28.042 DEBUG:boto:Method: GET 2023-07-29T00:35:28.043 DEBUG:boto:Path: /hiiqos-59/?prefix=obj22 2023-07-29T00:35:28.043 DEBUG:boto:Data: 2023-07-29T00:35:28.043 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.044 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:28.044 DEBUG:boto:Port: 8001 2023-07-29T00:35:28.044 DEBUG:boto:Params: {} 2023-07-29T00:35:28.045 DEBUG:boto:Token: None 2023-07-29T00:35:28.045 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/ 2023-07-29T00:35:28.046 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw= 2023-07-29T00:35:28.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw=', 'Content-Length': '0'} 2023-07-29T00:35:28.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afebc93b82245f56-0064c45ecf-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.049 DEBUG:boto:b'hiiqos-59obj221000falseobj222023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:28.050 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:28.050 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:28.051 DEBUG:boto:path=/hiiqos-59/?prefix=obj22 2023-07-29T00:35:28.051 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj22 2023-07-29T00:35:28.051 DEBUG:boto:Method: GET 2023-07-29T00:35:28.052 DEBUG:boto:Path: /hiiqos-59/?prefix=obj22 2023-07-29T00:35:28.052 DEBUG:boto:Data: 2023-07-29T00:35:28.052 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.053 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:28.053 DEBUG:boto:Port: 8000 2023-07-29T00:35:28.053 DEBUG:boto:Params: {} 2023-07-29T00:35:28.054 DEBUG:boto:Token: None 2023-07-29T00:35:28.054 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/ 2023-07-29T00:35:28.055 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw= 2023-07-29T00:35:28.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw=', 'Content-Length': '0'} 2023-07-29T00:35:28.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdeb6ef05fd5f304-0064c45ecf-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.100 DEBUG:boto:b'hiiqos-59obj221000falseobj222023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:28.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-07-29T00:35:28.101 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:35:28.102 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:35:28.102 DEBUG:boto:path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:35:28.103 DEBUG:boto:auth_path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:35:28.103 DEBUG:boto:Method: GET 2023-07-29T00:35:28.103 DEBUG:boto:Path: /hiiqos-59/obj23?versionId=null 2023-07-29T00:35:28.104 DEBUG:boto:Data: 2023-07-29T00:35:28.104 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.104 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:28.105 DEBUG:boto:Port: 8001 2023-07-29T00:35:28.105 DEBUG:boto:Params: {} 2023-07-29T00:35:28.105 DEBUG:boto:Token: None 2023-07-29T00:35:28.106 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj23?versionId=null 2023-07-29T00:35:28.106 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/lDaW9NZNTTE0JvtFUXpwI6Y1Rk= 2023-07-29T00:35:28.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/lDaW9NZNTTE0JvtFUXpwI6Y1Rk=', 'Content-Length': '0'} 2023-07-29T00:35:28.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000242fe89579de6421-0064c45ed0-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.109 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:35:28.110 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:35:28.110 DEBUG:boto:path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:35:28.110 DEBUG:boto:auth_path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:35:28.111 DEBUG:boto:Method: GET 2023-07-29T00:35:28.111 DEBUG:boto:Path: /hiiqos-59/obj23?versionId=null 2023-07-29T00:35:28.112 DEBUG:boto:Data: 2023-07-29T00:35:28.112 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.113 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:28.113 DEBUG:boto:Port: 8000 2023-07-29T00:35:28.114 DEBUG:boto:Params: {} 2023-07-29T00:35:28.115 DEBUG:boto:Token: None 2023-07-29T00:35:28.116 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj23?versionId=null 2023-07-29T00:35:28.116 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/lDaW9NZNTTE0JvtFUXpwI6Y1Rk= 2023-07-29T00:35:28.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/lDaW9NZNTTE0JvtFUXpwI6Y1Rk=', 'Content-Length': '0'} 2023-07-29T00:35:28.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3cd5b234f02ec81-0064c45ed0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.120 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:35:28.120 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:35:28.121 DEBUG:boto:path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:35:28.121 DEBUG:boto:auth_path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:35:28.122 DEBUG:boto:Method: HEAD 2023-07-29T00:35:28.122 DEBUG:boto:Path: /hiiqos-59/obj23?versionId=null 2023-07-29T00:35:28.123 DEBUG:boto:Data: 2023-07-29T00:35:28.123 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.124 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:28.124 DEBUG:boto:Port: 8001 2023-07-29T00:35:28.125 DEBUG:boto:Params: {} 2023-07-29T00:35:28.125 DEBUG:boto:Token: None 2023-07-29T00:35:28.126 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj23?versionId=null 2023-07-29T00:35:28.127 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9Nkt2cOzWm2C7YTqnxxIstVNdKo= 2023-07-29T00:35:28.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9Nkt2cOzWm2C7YTqnxxIstVNdKo=', 'Content-Length': '0'} 2023-07-29T00:35:28.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fd0fd9ee433784f-0064c45ed0-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.130 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:35:28.131 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:35:28.132 DEBUG:boto:path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:35:28.132 DEBUG:boto:auth_path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:35:28.133 DEBUG:boto:Method: HEAD 2023-07-29T00:35:28.133 DEBUG:boto:Path: /hiiqos-59/obj23?versionId=null 2023-07-29T00:35:28.134 DEBUG:boto:Data: 2023-07-29T00:35:28.134 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.135 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:28.135 DEBUG:boto:Port: 8000 2023-07-29T00:35:28.136 DEBUG:boto:Params: {} 2023-07-29T00:35:28.136 DEBUG:boto:Token: None 2023-07-29T00:35:28.137 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj23?versionId=null 2023-07-29T00:35:28.137 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9Nkt2cOzWm2C7YTqnxxIstVNdKo= 2023-07-29T00:35:28.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9Nkt2cOzWm2C7YTqnxxIstVNdKo=', 'Content-Length': '0'} 2023-07-29T00:35:28.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c537cf2a9127fc91-0064c45ed0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-07-29T00:35:28.142 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:35:28.142 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:35:28.143 DEBUG:boto:path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:35:28.143 DEBUG:boto:auth_path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:35:28.144 DEBUG:boto:Method: GET 2023-07-29T00:35:28.144 DEBUG:boto:Path: /hiiqos-59/obj23?versionId=null 2023-07-29T00:35:28.145 DEBUG:boto:Data: 2023-07-29T00:35:28.145 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.146 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:28.147 DEBUG:boto:Port: 8001 2023-07-29T00:35:28.147 DEBUG:boto:Params: {} 2023-07-29T00:35:28.148 DEBUG:boto:Token: None 2023-07-29T00:35:28.149 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj23?versionId=null 2023-07-29T00:35:28.149 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/lDaW9NZNTTE0JvtFUXpwI6Y1Rk= 2023-07-29T00:35:28.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/lDaW9NZNTTE0JvtFUXpwI6Y1Rk=', 'Content-Length': '0'} 2023-07-29T00:35:28.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a01690b3d2aa2be-0064c45ed0-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.153 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:35:28.153 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:35:28.154 DEBUG:boto:path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:35:28.155 DEBUG:boto:auth_path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:35:28.156 DEBUG:boto:Method: GET 2023-07-29T00:35:28.156 DEBUG:boto:Path: /hiiqos-59/obj23?versionId=null 2023-07-29T00:35:28.157 DEBUG:boto:Data: 2023-07-29T00:35:28.158 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.158 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:28.159 DEBUG:boto:Port: 8000 2023-07-29T00:35:28.160 DEBUG:boto:Params: {} 2023-07-29T00:35:28.160 DEBUG:boto:Token: None 2023-07-29T00:35:28.161 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj23?versionId=null 2023-07-29T00:35:28.162 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/lDaW9NZNTTE0JvtFUXpwI6Y1Rk= 2023-07-29T00:35:28.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/lDaW9NZNTTE0JvtFUXpwI6Y1Rk=', 'Content-Length': '0'} 2023-07-29T00:35:28.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024b1e465234c5d21-0064c45ed0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.165 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:28.166 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:28.167 DEBUG:boto:path=/hiiqos-59/?prefix=obj23 2023-07-29T00:35:28.167 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj23 2023-07-29T00:35:28.167 DEBUG:boto:Method: GET 2023-07-29T00:35:28.168 DEBUG:boto:Path: /hiiqos-59/?prefix=obj23 2023-07-29T00:35:28.168 DEBUG:boto:Data: 2023-07-29T00:35:28.168 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.169 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:28.169 DEBUG:boto:Port: 8001 2023-07-29T00:35:28.170 DEBUG:boto:Params: {} 2023-07-29T00:35:28.170 DEBUG:boto:Token: None 2023-07-29T00:35:28.170 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/ 2023-07-29T00:35:28.171 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw= 2023-07-29T00:35:28.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw=', 'Content-Length': '0'} 2023-07-29T00:35:28.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c33d9f72f86b48ff-0064c45ed0-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.174 DEBUG:boto:b'hiiqos-59obj231000falseobj232023-07-29T00:33:06.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:28.174 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:28.175 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:28.175 DEBUG:boto:path=/hiiqos-59/?prefix=obj23 2023-07-29T00:35:28.176 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj23 2023-07-29T00:35:28.176 DEBUG:boto:Method: GET 2023-07-29T00:35:28.176 DEBUG:boto:Path: /hiiqos-59/?prefix=obj23 2023-07-29T00:35:28.177 DEBUG:boto:Data: 2023-07-29T00:35:28.177 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.177 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:28.178 DEBUG:boto:Port: 8000 2023-07-29T00:35:28.178 DEBUG:boto:Params: {} 2023-07-29T00:35:28.178 DEBUG:boto:Token: None 2023-07-29T00:35:28.179 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/ 2023-07-29T00:35:28.179 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw= 2023-07-29T00:35:28.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw=', 'Content-Length': '0'} 2023-07-29T00:35:28.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001359ac987b3e4314-0064c45ed0-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.184 DEBUG:boto:b'hiiqos-59obj231000falseobj232023-07-29T00:33:06.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:28.184 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:35:28.185 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:35:28.185 DEBUG:boto:Method: HEAD 2023-07-29T00:35:28.185 DEBUG:boto:Path: /hiiqos-59/obj23 2023-07-29T00:35:28.186 DEBUG:boto:Data: 2023-07-29T00:35:28.186 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.186 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:28.187 DEBUG:boto:Port: 8001 2023-07-29T00:35:28.187 DEBUG:boto:Params: {} 2023-07-29T00:35:28.187 DEBUG:boto:Token: None 2023-07-29T00:35:28.188 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj23 2023-07-29T00:35:28.188 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/0EzkEJE4s9MwcjLl53eq03VMjo= 2023-07-29T00:35:28.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/0EzkEJE4s9MwcjLl53eq03VMjo=', 'Content-Length': '0'} 2023-07-29T00:35:28.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e2be47dae9ef435-0064c45ed0-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.191 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:35:28.191 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:35:28.192 DEBUG:boto:Method: HEAD 2023-07-29T00:35:28.192 DEBUG:boto:Path: /hiiqos-59/obj23 2023-07-29T00:35:28.193 DEBUG:boto:Data: 2023-07-29T00:35:28.193 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.193 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:28.194 DEBUG:boto:Port: 8000 2023-07-29T00:35:28.194 DEBUG:boto:Params: {} 2023-07-29T00:35:28.194 DEBUG:boto:Token: None 2023-07-29T00:35:28.195 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj23 2023-07-29T00:35:28.195 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/0EzkEJE4s9MwcjLl53eq03VMjo= 2023-07-29T00:35:28.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/0EzkEJE4s9MwcjLl53eq03VMjo=', 'Content-Length': '0'} 2023-07-29T00:35:28.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa5c76302f38ce76-0064c45ed0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-07-29T00:35:28.198 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:35:28.199 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:35:28.199 DEBUG:boto:Method: GET 2023-07-29T00:35:28.199 DEBUG:boto:Path: /hiiqos-59/obj23 2023-07-29T00:35:28.200 DEBUG:boto:Data: 2023-07-29T00:35:28.200 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.200 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:28.201 DEBUG:boto:Port: 8001 2023-07-29T00:35:28.201 DEBUG:boto:Params: {} 2023-07-29T00:35:28.202 DEBUG:boto:Token: None 2023-07-29T00:35:28.202 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj23 2023-07-29T00:35:28.202 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZlSlDI2fO99oWm9A4JwkQRZCW2s= 2023-07-29T00:35:28.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZlSlDI2fO99oWm9A4JwkQRZCW2s=', 'Content-Length': '0'} 2023-07-29T00:35:28.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004495df30048960d2-0064c45ed0-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.248 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:35:28.249 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:35:28.249 DEBUG:boto:Method: GET 2023-07-29T00:35:28.250 DEBUG:boto:Path: /hiiqos-59/obj23 2023-07-29T00:35:28.250 DEBUG:boto:Data: 2023-07-29T00:35:28.251 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.251 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:28.252 DEBUG:boto:Port: 8000 2023-07-29T00:35:28.252 DEBUG:boto:Params: {} 2023-07-29T00:35:28.253 DEBUG:boto:Token: None 2023-07-29T00:35:28.253 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj23 2023-07-29T00:35:28.253 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZlSlDI2fO99oWm9A4JwkQRZCW2s= 2023-07-29T00:35:28.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZlSlDI2fO99oWm9A4JwkQRZCW2s=', 'Content-Length': '0'} 2023-07-29T00:35:28.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca4092b601c0ea8e-0064c45ed0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.256 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:28.257 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:28.257 DEBUG:boto:path=/hiiqos-59/?prefix=obj23 2023-07-29T00:35:28.258 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj23 2023-07-29T00:35:28.258 DEBUG:boto:Method: GET 2023-07-29T00:35:28.258 DEBUG:boto:Path: /hiiqos-59/?prefix=obj23 2023-07-29T00:35:28.259 DEBUG:boto:Data: 2023-07-29T00:35:28.259 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.259 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:28.260 DEBUG:boto:Port: 8001 2023-07-29T00:35:28.260 DEBUG:boto:Params: {} 2023-07-29T00:35:28.260 DEBUG:boto:Token: None 2023-07-29T00:35:28.261 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/ 2023-07-29T00:35:28.261 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw= 2023-07-29T00:35:28.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw=', 'Content-Length': '0'} 2023-07-29T00:35:28.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc3a2c3ecd782789-0064c45ed0-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.264 DEBUG:boto:b'hiiqos-59obj231000falseobj232023-07-29T00:33:06.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:28.265 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:28.265 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:28.266 DEBUG:boto:path=/hiiqos-59/?prefix=obj23 2023-07-29T00:35:28.266 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj23 2023-07-29T00:35:28.266 DEBUG:boto:Method: GET 2023-07-29T00:35:28.267 DEBUG:boto:Path: /hiiqos-59/?prefix=obj23 2023-07-29T00:35:28.267 DEBUG:boto:Data: 2023-07-29T00:35:28.267 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.268 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:28.268 DEBUG:boto:Port: 8000 2023-07-29T00:35:28.269 DEBUG:boto:Params: {} 2023-07-29T00:35:28.269 DEBUG:boto:Token: None 2023-07-29T00:35:28.269 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/ 2023-07-29T00:35:28.270 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw= 2023-07-29T00:35:28.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw=', 'Content-Length': '0'} 2023-07-29T00:35:28.273 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067ba6e526259cdfd-0064c45ed0-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.274 DEBUG:boto:b'hiiqos-59obj231000falseobj232023-07-29T00:33:06.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:28.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-07-29T00:35:28.275 DEBUG:boto:path=/hiiqos-59/obj29 2023-07-29T00:35:28.275 DEBUG:boto:auth_path=/hiiqos-59/obj29 2023-07-29T00:35:28.276 DEBUG:boto:path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:35:28.276 DEBUG:boto:auth_path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:35:28.277 DEBUG:boto:Method: GET 2023-07-29T00:35:28.277 DEBUG:boto:Path: /hiiqos-59/obj29?versionId=null 2023-07-29T00:35:28.278 DEBUG:boto:Data: 2023-07-29T00:35:28.278 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.278 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:28.279 DEBUG:boto:Port: 8001 2023-07-29T00:35:28.279 DEBUG:boto:Params: {} 2023-07-29T00:35:28.280 DEBUG:boto:Token: None 2023-07-29T00:35:28.280 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj29?versionId=null 2023-07-29T00:35:28.280 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nbYqE6glvsJI9DF85gGZO9abCrI= 2023-07-29T00:35:28.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nbYqE6glvsJI9DF85gGZO9abCrI=', 'Content-Length': '0'} 2023-07-29T00:35:28.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083d3004994260481-0064c45ed0-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.283 DEBUG:boto:path=/hiiqos-59/obj29 2023-07-29T00:35:28.284 DEBUG:boto:auth_path=/hiiqos-59/obj29 2023-07-29T00:35:28.284 DEBUG:boto:path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:35:28.285 DEBUG:boto:auth_path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:35:28.285 DEBUG:boto:Method: GET 2023-07-29T00:35:28.285 DEBUG:boto:Path: /hiiqos-59/obj29?versionId=null 2023-07-29T00:35:28.286 DEBUG:boto:Data: 2023-07-29T00:35:28.286 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.286 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:28.287 DEBUG:boto:Port: 8000 2023-07-29T00:35:28.287 DEBUG:boto:Params: {} 2023-07-29T00:35:28.287 DEBUG:boto:Token: None 2023-07-29T00:35:28.288 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj29?versionId=null 2023-07-29T00:35:28.288 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nbYqE6glvsJI9DF85gGZO9abCrI= 2023-07-29T00:35:28.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nbYqE6glvsJI9DF85gGZO9abCrI=', 'Content-Length': '0'} 2023-07-29T00:35:28.291 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062b49a2cf31b03d9-0064c45ed0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.332 DEBUG:boto:path=/hiiqos-59/obj29 2023-07-29T00:35:28.333 DEBUG:boto:auth_path=/hiiqos-59/obj29 2023-07-29T00:35:28.333 DEBUG:boto:path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:35:28.333 DEBUG:boto:auth_path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:35:28.334 DEBUG:boto:Method: HEAD 2023-07-29T00:35:28.334 DEBUG:boto:Path: /hiiqos-59/obj29?versionId=null 2023-07-29T00:35:28.334 DEBUG:boto:Data: 2023-07-29T00:35:28.335 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.335 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:28.335 DEBUG:boto:Port: 8001 2023-07-29T00:35:28.336 DEBUG:boto:Params: {} 2023-07-29T00:35:28.336 DEBUG:boto:Token: None 2023-07-29T00:35:28.337 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj29?versionId=null 2023-07-29T00:35:28.337 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rbwP7bU68NzutrjcyXMI5DB2P1o= 2023-07-29T00:35:28.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rbwP7bU68NzutrjcyXMI5DB2P1o=', 'Content-Length': '0'} 2023-07-29T00:35:28.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000282628c421e4c122-0064c45ed0-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.340 DEBUG:boto:path=/hiiqos-59/obj29 2023-07-29T00:35:28.341 DEBUG:boto:auth_path=/hiiqos-59/obj29 2023-07-29T00:35:28.341 DEBUG:boto:path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:35:28.341 DEBUG:boto:auth_path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:35:28.342 DEBUG:boto:Method: HEAD 2023-07-29T00:35:28.342 DEBUG:boto:Path: /hiiqos-59/obj29?versionId=null 2023-07-29T00:35:28.342 DEBUG:boto:Data: 2023-07-29T00:35:28.343 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.343 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:28.343 DEBUG:boto:Port: 8000 2023-07-29T00:35:28.343 DEBUG:boto:Params: {} 2023-07-29T00:35:28.344 DEBUG:boto:Token: None 2023-07-29T00:35:28.344 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj29?versionId=null 2023-07-29T00:35:28.345 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rbwP7bU68NzutrjcyXMI5DB2P1o= 2023-07-29T00:35:28.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rbwP7bU68NzutrjcyXMI5DB2P1o=', 'Content-Length': '0'} 2023-07-29T00:35:28.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0023f558711d0f1-0064c45ed0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-07-29T00:35:28.348 DEBUG:boto:path=/hiiqos-59/obj29 2023-07-29T00:35:28.348 DEBUG:boto:auth_path=/hiiqos-59/obj29 2023-07-29T00:35:28.349 DEBUG:boto:path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:35:28.349 DEBUG:boto:auth_path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:35:28.349 DEBUG:boto:Method: GET 2023-07-29T00:35:28.350 DEBUG:boto:Path: /hiiqos-59/obj29?versionId=null 2023-07-29T00:35:28.350 DEBUG:boto:Data: 2023-07-29T00:35:28.350 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.351 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:28.351 DEBUG:boto:Port: 8001 2023-07-29T00:35:28.351 DEBUG:boto:Params: {} 2023-07-29T00:35:28.352 DEBUG:boto:Token: None 2023-07-29T00:35:28.353 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj29?versionId=null 2023-07-29T00:35:28.353 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nbYqE6glvsJI9DF85gGZO9abCrI= 2023-07-29T00:35:28.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nbYqE6glvsJI9DF85gGZO9abCrI=', 'Content-Length': '0'} 2023-07-29T00:35:28.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c94df3ee945eed6a-0064c45ed0-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.356 DEBUG:boto:path=/hiiqos-59/obj29 2023-07-29T00:35:28.357 DEBUG:boto:auth_path=/hiiqos-59/obj29 2023-07-29T00:35:28.357 DEBUG:boto:path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:35:28.357 DEBUG:boto:auth_path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:35:28.358 DEBUG:boto:Method: GET 2023-07-29T00:35:28.358 DEBUG:boto:Path: /hiiqos-59/obj29?versionId=null 2023-07-29T00:35:28.358 DEBUG:boto:Data: 2023-07-29T00:35:28.359 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.359 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:28.359 DEBUG:boto:Port: 8000 2023-07-29T00:35:28.360 DEBUG:boto:Params: {} 2023-07-29T00:35:28.360 DEBUG:boto:Token: None 2023-07-29T00:35:28.361 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj29?versionId=null 2023-07-29T00:35:28.361 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nbYqE6glvsJI9DF85gGZO9abCrI= 2023-07-29T00:35:28.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nbYqE6glvsJI9DF85gGZO9abCrI=', 'Content-Length': '0'} 2023-07-29T00:35:28.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdcfe7c7c1b82e4b-0064c45ed0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.364 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:28.364 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:28.365 DEBUG:boto:path=/hiiqos-59/?prefix=obj29 2023-07-29T00:35:28.365 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj29 2023-07-29T00:35:28.365 DEBUG:boto:Method: GET 2023-07-29T00:35:28.366 DEBUG:boto:Path: /hiiqos-59/?prefix=obj29 2023-07-29T00:35:28.366 DEBUG:boto:Data: 2023-07-29T00:35:28.366 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.367 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:28.367 DEBUG:boto:Port: 8001 2023-07-29T00:35:28.367 DEBUG:boto:Params: {} 2023-07-29T00:35:28.368 DEBUG:boto:Token: None 2023-07-29T00:35:28.368 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/ 2023-07-29T00:35:28.369 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw= 2023-07-29T00:35:28.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw=', 'Content-Length': '0'} 2023-07-29T00:35:28.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048ae80a0ce44a10a-0064c45ed0-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.372 DEBUG:boto:b'hiiqos-59obj291000falseobj292023-07-29T00:35:01.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:28.373 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:28.373 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:28.373 DEBUG:boto:path=/hiiqos-59/?prefix=obj29 2023-07-29T00:35:28.374 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj29 2023-07-29T00:35:28.374 DEBUG:boto:Method: GET 2023-07-29T00:35:28.375 DEBUG:boto:Path: /hiiqos-59/?prefix=obj29 2023-07-29T00:35:28.375 DEBUG:boto:Data: 2023-07-29T00:35:28.375 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.376 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:28.376 DEBUG:boto:Port: 8000 2023-07-29T00:35:28.376 DEBUG:boto:Params: {} 2023-07-29T00:35:28.377 DEBUG:boto:Token: None 2023-07-29T00:35:28.377 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/ 2023-07-29T00:35:28.378 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw= 2023-07-29T00:35:28.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw=', 'Content-Length': '0'} 2023-07-29T00:35:28.381 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef520e8fab2f6b98-0064c45ed0-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.382 DEBUG:boto:b'hiiqos-59obj291000falseobj292023-07-29T00:35:01.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:28.383 DEBUG:boto:path=/hiiqos-59/obj29 2023-07-29T00:35:28.383 DEBUG:boto:auth_path=/hiiqos-59/obj29 2023-07-29T00:35:28.383 DEBUG:boto:Method: HEAD 2023-07-29T00:35:28.384 DEBUG:boto:Path: /hiiqos-59/obj29 2023-07-29T00:35:28.384 DEBUG:boto:Data: 2023-07-29T00:35:28.384 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.385 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:28.385 DEBUG:boto:Port: 8001 2023-07-29T00:35:28.385 DEBUG:boto:Params: {} 2023-07-29T00:35:28.386 DEBUG:boto:Token: None 2023-07-29T00:35:28.386 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj29 2023-07-29T00:35:28.386 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P63LgRbisB00/KG81CsNkvBo1o4= 2023-07-29T00:35:28.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P63LgRbisB00/KG81CsNkvBo1o4=', 'Content-Length': '0'} 2023-07-29T00:35:28.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e877e3630c072434-0064c45ed0-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.389 DEBUG:boto:path=/hiiqos-59/obj29 2023-07-29T00:35:28.389 DEBUG:boto:auth_path=/hiiqos-59/obj29 2023-07-29T00:35:28.390 DEBUG:boto:Method: HEAD 2023-07-29T00:35:28.390 DEBUG:boto:Path: /hiiqos-59/obj29 2023-07-29T00:35:28.391 DEBUG:boto:Data: 2023-07-29T00:35:28.391 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.391 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:28.392 DEBUG:boto:Port: 8000 2023-07-29T00:35:28.392 DEBUG:boto:Params: {} 2023-07-29T00:35:28.392 DEBUG:boto:Token: None 2023-07-29T00:35:28.393 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj29 2023-07-29T00:35:28.393 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P63LgRbisB00/KG81CsNkvBo1o4= 2023-07-29T00:35:28.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P63LgRbisB00/KG81CsNkvBo1o4=', 'Content-Length': '0'} 2023-07-29T00:35:28.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b46094958fffb6e-0064c45ed0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-07-29T00:35:28.396 DEBUG:boto:path=/hiiqos-59/obj29 2023-07-29T00:35:28.397 DEBUG:boto:auth_path=/hiiqos-59/obj29 2023-07-29T00:35:28.397 DEBUG:boto:Method: GET 2023-07-29T00:35:28.398 DEBUG:boto:Path: /hiiqos-59/obj29 2023-07-29T00:35:28.398 DEBUG:boto:Data: 2023-07-29T00:35:28.398 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.399 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:28.399 DEBUG:boto:Port: 8001 2023-07-29T00:35:28.399 DEBUG:boto:Params: {} 2023-07-29T00:35:28.400 DEBUG:boto:Token: None 2023-07-29T00:35:28.400 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj29 2023-07-29T00:35:28.401 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aW7ZjeAEGCeIigubiaUIoXL/i9g= 2023-07-29T00:35:28.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aW7ZjeAEGCeIigubiaUIoXL/i9g=', 'Content-Length': '0'} 2023-07-29T00:35:28.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000727fee8f901021ab-0064c45ed0-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.404 DEBUG:boto:path=/hiiqos-59/obj29 2023-07-29T00:35:28.404 DEBUG:boto:auth_path=/hiiqos-59/obj29 2023-07-29T00:35:28.405 DEBUG:boto:Method: GET 2023-07-29T00:35:28.405 DEBUG:boto:Path: /hiiqos-59/obj29 2023-07-29T00:35:28.405 DEBUG:boto:Data: 2023-07-29T00:35:28.406 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.406 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:28.407 DEBUG:boto:Port: 8000 2023-07-29T00:35:28.408 DEBUG:boto:Params: {} 2023-07-29T00:35:28.409 DEBUG:boto:Token: None 2023-07-29T00:35:28.409 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj29 2023-07-29T00:35:28.410 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aW7ZjeAEGCeIigubiaUIoXL/i9g= 2023-07-29T00:35:28.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aW7ZjeAEGCeIigubiaUIoXL/i9g=', 'Content-Length': '0'} 2023-07-29T00:35:28.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b95e661363884777-0064c45ed0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.413 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:28.414 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:28.414 DEBUG:boto:path=/hiiqos-59/?prefix=obj29 2023-07-29T00:35:28.415 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj29 2023-07-29T00:35:28.415 DEBUG:boto:Method: GET 2023-07-29T00:35:28.415 DEBUG:boto:Path: /hiiqos-59/?prefix=obj29 2023-07-29T00:35:28.416 DEBUG:boto:Data: 2023-07-29T00:35:28.416 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.416 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:28.417 DEBUG:boto:Port: 8001 2023-07-29T00:35:28.417 DEBUG:boto:Params: {} 2023-07-29T00:35:28.418 DEBUG:boto:Token: None 2023-07-29T00:35:28.418 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/ 2023-07-29T00:35:28.418 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw= 2023-07-29T00:35:28.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw=', 'Content-Length': '0'} 2023-07-29T00:35:28.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036fa036ad405799e-0064c45ed0-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.422 DEBUG:boto:b'hiiqos-59obj291000falseobj292023-07-29T00:35:01.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:28.422 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:28.423 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:28.423 DEBUG:boto:path=/hiiqos-59/?prefix=obj29 2023-07-29T00:35:28.423 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj29 2023-07-29T00:35:28.424 DEBUG:boto:Method: GET 2023-07-29T00:35:28.424 DEBUG:boto:Path: /hiiqos-59/?prefix=obj29 2023-07-29T00:35:28.424 DEBUG:boto:Data: 2023-07-29T00:35:28.425 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.425 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:28.426 DEBUG:boto:Port: 8000 2023-07-29T00:35:28.426 DEBUG:boto:Params: {} 2023-07-29T00:35:28.426 DEBUG:boto:Token: None 2023-07-29T00:35:28.427 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/ 2023-07-29T00:35:28.427 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw= 2023-07-29T00:35:28.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw=', 'Content-Length': '0'} 2023-07-29T00:35:28.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8a4a702c111ec84-0064c45ed0-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.476 DEBUG:boto:b'hiiqos-59obj291000falseobj292023-07-29T00:35:01.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:28.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-07-29T00:35:28.477 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:35:28.478 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:35:28.478 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:35:28.479 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:35:28.479 DEBUG:boto:Method: GET 2023-07-29T00:35:28.479 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:35:28.480 DEBUG:boto:Data: 2023-07-29T00:35:28.480 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.480 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:28.481 DEBUG:boto:Port: 8001 2023-07-29T00:35:28.481 DEBUG:boto:Params: {} 2023-07-29T00:35:28.482 DEBUG:boto:Token: None 2023-07-29T00:35:28.482 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:35:28.483 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZP/kioHPArAma+knS10I8GG6ObY= 2023-07-29T00:35:28.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZP/kioHPArAma+knS10I8GG6ObY=', 'Content-Length': '0'} 2023-07-29T00:35:28.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046736b79ed5dce20-0064c45ed0-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.486 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:35:28.486 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:35:28.486 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:35:28.487 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:35:28.487 DEBUG:boto:Method: GET 2023-07-29T00:35:28.488 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:35:28.488 DEBUG:boto:Data: 2023-07-29T00:35:28.488 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.489 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:28.489 DEBUG:boto:Port: 8000 2023-07-29T00:35:28.489 DEBUG:boto:Params: {} 2023-07-29T00:35:28.490 DEBUG:boto:Token: None 2023-07-29T00:35:28.490 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:35:28.491 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZP/kioHPArAma+knS10I8GG6ObY= 2023-07-29T00:35:28.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZP/kioHPArAma+knS10I8GG6ObY=', 'Content-Length': '0'} 2023-07-29T00:35:28.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbf1b09ea1418aff-0064c45ed0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.494 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:35:28.494 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:35:28.494 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:35:28.495 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:35:28.495 DEBUG:boto:Method: HEAD 2023-07-29T00:35:28.496 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:35:28.496 DEBUG:boto:Data: 2023-07-29T00:35:28.497 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.497 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:28.497 DEBUG:boto:Port: 8001 2023-07-29T00:35:28.498 DEBUG:boto:Params: {} 2023-07-29T00:35:28.498 DEBUG:boto:Token: None 2023-07-29T00:35:28.499 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:35:28.499 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kJxsYSDtSv8upqwGlAcDUp+eF+E= 2023-07-29T00:35:28.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kJxsYSDtSv8upqwGlAcDUp+eF+E=', 'Content-Length': '0'} 2023-07-29T00:35:28.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe86f5f3a2e2cbbf-0064c45ed0-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.502 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:35:28.502 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:35:28.503 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:35:28.503 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:35:28.503 DEBUG:boto:Method: HEAD 2023-07-29T00:35:28.504 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:35:28.504 DEBUG:boto:Data: 2023-07-29T00:35:28.504 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.505 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:28.505 DEBUG:boto:Port: 8000 2023-07-29T00:35:28.505 DEBUG:boto:Params: {} 2023-07-29T00:35:28.506 DEBUG:boto:Token: None 2023-07-29T00:35:28.506 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:35:28.507 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kJxsYSDtSv8upqwGlAcDUp+eF+E= 2023-07-29T00:35:28.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kJxsYSDtSv8upqwGlAcDUp+eF+E=', 'Content-Length': '0'} 2023-07-29T00:35:28.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e32bbd9d8cb0232e-0064c45ed0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-07-29T00:35:28.510 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:35:28.510 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:35:28.510 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:35:28.511 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:35:28.511 DEBUG:boto:Method: GET 2023-07-29T00:35:28.511 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:35:28.512 DEBUG:boto:Data: 2023-07-29T00:35:28.512 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.513 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:28.513 DEBUG:boto:Port: 8001 2023-07-29T00:35:28.513 DEBUG:boto:Params: {} 2023-07-29T00:35:28.514 DEBUG:boto:Token: None 2023-07-29T00:35:28.514 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:35:28.515 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZP/kioHPArAma+knS10I8GG6ObY= 2023-07-29T00:35:28.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZP/kioHPArAma+knS10I8GG6ObY=', 'Content-Length': '0'} 2023-07-29T00:35:28.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007edb80d0fe5ff9c7-0064c45ed0-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.560 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:35:28.561 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:35:28.561 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:35:28.562 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:35:28.562 DEBUG:boto:Method: GET 2023-07-29T00:35:28.562 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:35:28.563 DEBUG:boto:Data: 2023-07-29T00:35:28.563 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.563 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:28.564 DEBUG:boto:Port: 8000 2023-07-29T00:35:28.565 DEBUG:boto:Params: {} 2023-07-29T00:35:28.565 DEBUG:boto:Token: None 2023-07-29T00:35:28.566 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:35:28.567 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZP/kioHPArAma+knS10I8GG6ObY= 2023-07-29T00:35:28.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZP/kioHPArAma+knS10I8GG6ObY=', 'Content-Length': '0'} 2023-07-29T00:35:28.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b7bd8cdd1be7363-0064c45ed0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.570 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:28.570 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:28.570 DEBUG:boto:path=/hiiqos-59/?prefix=obj33 2023-07-29T00:35:28.571 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj33 2023-07-29T00:35:28.571 DEBUG:boto:Method: GET 2023-07-29T00:35:28.571 DEBUG:boto:Path: /hiiqos-59/?prefix=obj33 2023-07-29T00:35:28.572 DEBUG:boto:Data: 2023-07-29T00:35:28.572 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.573 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:28.573 DEBUG:boto:Port: 8001 2023-07-29T00:35:28.573 DEBUG:boto:Params: {} 2023-07-29T00:35:28.574 DEBUG:boto:Token: None 2023-07-29T00:35:28.574 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/ 2023-07-29T00:35:28.575 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw= 2023-07-29T00:35:28.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw=', 'Content-Length': '0'} 2023-07-29T00:35:28.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd8dbe5753629b6a-0064c45ed0-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.578 DEBUG:boto:b'hiiqos-59obj331000falseobj332023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:28.579 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:28.579 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:28.579 DEBUG:boto:path=/hiiqos-59/?prefix=obj33 2023-07-29T00:35:28.580 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj33 2023-07-29T00:35:28.580 DEBUG:boto:Method: GET 2023-07-29T00:35:28.581 DEBUG:boto:Path: /hiiqos-59/?prefix=obj33 2023-07-29T00:35:28.581 DEBUG:boto:Data: 2023-07-29T00:35:28.581 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.582 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:28.582 DEBUG:boto:Port: 8000 2023-07-29T00:35:28.582 DEBUG:boto:Params: {} 2023-07-29T00:35:28.583 DEBUG:boto:Token: None 2023-07-29T00:35:28.583 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/ 2023-07-29T00:35:28.584 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw= 2023-07-29T00:35:28.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw=', 'Content-Length': '0'} 2023-07-29T00:35:28.588 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a4d900e0b2493bd-0064c45ed0-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.632 DEBUG:boto:b'hiiqos-59obj331000falseobj332023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:28.633 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:35:28.634 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:35:28.635 DEBUG:boto:Method: HEAD 2023-07-29T00:35:28.635 DEBUG:boto:Path: /hiiqos-59/obj33 2023-07-29T00:35:28.635 DEBUG:boto:Data: 2023-07-29T00:35:28.636 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.636 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:28.636 DEBUG:boto:Port: 8001 2023-07-29T00:35:28.637 DEBUG:boto:Params: {} 2023-07-29T00:35:28.637 DEBUG:boto:Token: None 2023-07-29T00:35:28.638 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj33 2023-07-29T00:35:28.638 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EsQV5h9uT9dH8LOxpHhwYJRRkKI= 2023-07-29T00:35:28.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EsQV5h9uT9dH8LOxpHhwYJRRkKI=', 'Content-Length': '0'} 2023-07-29T00:35:28.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009af989d27daaf8da-0064c45ed0-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.642 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:35:28.642 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:35:28.642 DEBUG:boto:Method: HEAD 2023-07-29T00:35:28.643 DEBUG:boto:Path: /hiiqos-59/obj33 2023-07-29T00:35:28.643 DEBUG:boto:Data: 2023-07-29T00:35:28.643 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.644 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:28.644 DEBUG:boto:Port: 8000 2023-07-29T00:35:28.644 DEBUG:boto:Params: {} 2023-07-29T00:35:28.645 DEBUG:boto:Token: None 2023-07-29T00:35:28.645 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj33 2023-07-29T00:35:28.646 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EsQV5h9uT9dH8LOxpHhwYJRRkKI= 2023-07-29T00:35:28.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EsQV5h9uT9dH8LOxpHhwYJRRkKI=', 'Content-Length': '0'} 2023-07-29T00:35:28.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000947bcb06c4afa8df-0064c45ed0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-07-29T00:35:28.649 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:35:28.649 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:35:28.650 DEBUG:boto:Method: GET 2023-07-29T00:35:28.650 DEBUG:boto:Path: /hiiqos-59/obj33 2023-07-29T00:35:28.650 DEBUG:boto:Data: 2023-07-29T00:35:28.651 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.651 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:28.652 DEBUG:boto:Port: 8001 2023-07-29T00:35:28.652 DEBUG:boto:Params: {} 2023-07-29T00:35:28.652 DEBUG:boto:Token: None 2023-07-29T00:35:28.653 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj33 2023-07-29T00:35:28.653 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RoaOaTMDNbr9S73ddWmBA5Q9HdY= 2023-07-29T00:35:28.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RoaOaTMDNbr9S73ddWmBA5Q9HdY=', 'Content-Length': '0'} 2023-07-29T00:35:28.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024b4e2f52c6da2c4-0064c45ed0-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.656 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:35:28.657 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:35:28.657 DEBUG:boto:Method: GET 2023-07-29T00:35:28.657 DEBUG:boto:Path: /hiiqos-59/obj33 2023-07-29T00:35:28.658 DEBUG:boto:Data: 2023-07-29T00:35:28.658 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.658 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:28.659 DEBUG:boto:Port: 8000 2023-07-29T00:35:28.659 DEBUG:boto:Params: {} 2023-07-29T00:35:28.660 DEBUG:boto:Token: None 2023-07-29T00:35:28.660 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj33 2023-07-29T00:35:28.660 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RoaOaTMDNbr9S73ddWmBA5Q9HdY= 2023-07-29T00:35:28.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RoaOaTMDNbr9S73ddWmBA5Q9HdY=', 'Content-Length': '0'} 2023-07-29T00:35:28.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b85df74160ba378-0064c45ed0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.664 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:28.664 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:28.664 DEBUG:boto:path=/hiiqos-59/?prefix=obj33 2023-07-29T00:35:28.665 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj33 2023-07-29T00:35:28.665 DEBUG:boto:Method: GET 2023-07-29T00:35:28.666 DEBUG:boto:Path: /hiiqos-59/?prefix=obj33 2023-07-29T00:35:28.666 DEBUG:boto:Data: 2023-07-29T00:35:28.666 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.667 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:28.667 DEBUG:boto:Port: 8001 2023-07-29T00:35:28.667 DEBUG:boto:Params: {} 2023-07-29T00:35:28.668 DEBUG:boto:Token: None 2023-07-29T00:35:28.668 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/ 2023-07-29T00:35:28.669 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw= 2023-07-29T00:35:28.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw=', 'Content-Length': '0'} 2023-07-29T00:35:28.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023d4499efd0a4d02-0064c45ed0-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.672 DEBUG:boto:b'hiiqos-59obj331000falseobj332023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:28.673 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:28.673 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:28.674 DEBUG:boto:path=/hiiqos-59/?prefix=obj33 2023-07-29T00:35:28.674 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj33 2023-07-29T00:35:28.674 DEBUG:boto:Method: GET 2023-07-29T00:35:28.675 DEBUG:boto:Path: /hiiqos-59/?prefix=obj33 2023-07-29T00:35:28.675 DEBUG:boto:Data: 2023-07-29T00:35:28.675 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.676 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:28.676 DEBUG:boto:Port: 8000 2023-07-29T00:35:28.676 DEBUG:boto:Params: {} 2023-07-29T00:35:28.677 DEBUG:boto:Token: None 2023-07-29T00:35:28.677 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/ 2023-07-29T00:35:28.678 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw= 2023-07-29T00:35:28.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw=', 'Content-Length': '0'} 2023-07-29T00:35:28.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015f0b4f8042f8fd7-0064c45ed0-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.682 DEBUG:boto:b'hiiqos-59obj331000falseobj332023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:28.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-07-29T00:35:28.683 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:35:28.684 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:35:28.684 DEBUG:boto:path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:35:28.685 DEBUG:boto:auth_path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:35:28.685 DEBUG:boto:Method: GET 2023-07-29T00:35:28.685 DEBUG:boto:Path: /hiiqos-59/obj34?versionId=null 2023-07-29T00:35:28.686 DEBUG:boto:Data: 2023-07-29T00:35:28.686 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.686 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:28.687 DEBUG:boto:Port: 8001 2023-07-29T00:35:28.687 DEBUG:boto:Params: {} 2023-07-29T00:35:28.688 DEBUG:boto:Token: None 2023-07-29T00:35:28.688 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj34?versionId=null 2023-07-29T00:35:28.688 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1FDRRjvUs4+on7BujrSAsX7kxC4= 2023-07-29T00:35:28.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1FDRRjvUs4+on7BujrSAsX7kxC4=', 'Content-Length': '0'} 2023-07-29T00:35:28.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039fd13efc45c5227-0064c45ed0-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.691 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:35:28.692 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:35:28.692 DEBUG:boto:path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:35:28.693 DEBUG:boto:auth_path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:35:28.693 DEBUG:boto:Method: GET 2023-07-29T00:35:28.693 DEBUG:boto:Path: /hiiqos-59/obj34?versionId=null 2023-07-29T00:35:28.694 DEBUG:boto:Data: 2023-07-29T00:35:28.694 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.694 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:28.695 DEBUG:boto:Port: 8000 2023-07-29T00:35:28.695 DEBUG:boto:Params: {} 2023-07-29T00:35:28.695 DEBUG:boto:Token: None 2023-07-29T00:35:28.696 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj34?versionId=null 2023-07-29T00:35:28.696 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1FDRRjvUs4+on7BujrSAsX7kxC4= 2023-07-29T00:35:28.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1FDRRjvUs4+on7BujrSAsX7kxC4=', 'Content-Length': '0'} 2023-07-29T00:35:28.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecec97540dd77f70-0064c45ed0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.699 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:35:28.699 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:35:28.700 DEBUG:boto:path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:35:28.700 DEBUG:boto:auth_path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:35:28.701 DEBUG:boto:Method: HEAD 2023-07-29T00:35:28.701 DEBUG:boto:Path: /hiiqos-59/obj34?versionId=null 2023-07-29T00:35:28.701 DEBUG:boto:Data: 2023-07-29T00:35:28.702 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.702 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:28.702 DEBUG:boto:Port: 8001 2023-07-29T00:35:28.702 DEBUG:boto:Params: {} 2023-07-29T00:35:28.703 DEBUG:boto:Token: None 2023-07-29T00:35:28.703 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj34?versionId=null 2023-07-29T00:35:28.704 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LYDz33ng0KGtazgkbZBRbb/7NPs= 2023-07-29T00:35:28.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LYDz33ng0KGtazgkbZBRbb/7NPs=', 'Content-Length': '0'} 2023-07-29T00:35:28.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6a1e346821b7dfd-0064c45ed0-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.706 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:35:28.707 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:35:28.707 DEBUG:boto:path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:35:28.707 DEBUG:boto:auth_path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:35:28.708 DEBUG:boto:Method: HEAD 2023-07-29T00:35:28.708 DEBUG:boto:Path: /hiiqos-59/obj34?versionId=null 2023-07-29T00:35:28.708 DEBUG:boto:Data: 2023-07-29T00:35:28.709 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.709 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:28.709 DEBUG:boto:Port: 8000 2023-07-29T00:35:28.710 DEBUG:boto:Params: {} 2023-07-29T00:35:28.710 DEBUG:boto:Token: None 2023-07-29T00:35:28.710 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj34?versionId=null 2023-07-29T00:35:28.711 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LYDz33ng0KGtazgkbZBRbb/7NPs= 2023-07-29T00:35:28.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LYDz33ng0KGtazgkbZBRbb/7NPs=', 'Content-Length': '0'} 2023-07-29T00:35:28.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000614a81b53feac95b-0064c45ed0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-07-29T00:35:28.714 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:35:28.714 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:35:28.715 DEBUG:boto:path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:35:28.715 DEBUG:boto:auth_path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:35:28.715 DEBUG:boto:Method: GET 2023-07-29T00:35:28.716 DEBUG:boto:Path: /hiiqos-59/obj34?versionId=null 2023-07-29T00:35:28.716 DEBUG:boto:Data: 2023-07-29T00:35:28.716 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.717 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:28.717 DEBUG:boto:Port: 8001 2023-07-29T00:35:28.717 DEBUG:boto:Params: {} 2023-07-29T00:35:28.718 DEBUG:boto:Token: None 2023-07-29T00:35:28.718 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj34?versionId=null 2023-07-29T00:35:28.719 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1FDRRjvUs4+on7BujrSAsX7kxC4= 2023-07-29T00:35:28.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1FDRRjvUs4+on7BujrSAsX7kxC4=', 'Content-Length': '0'} 2023-07-29T00:35:28.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f4121729cf2486c-0064c45ed0-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.722 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:35:28.722 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:35:28.722 DEBUG:boto:path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:35:28.723 DEBUG:boto:auth_path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:35:28.723 DEBUG:boto:Method: GET 2023-07-29T00:35:28.723 DEBUG:boto:Path: /hiiqos-59/obj34?versionId=null 2023-07-29T00:35:28.724 DEBUG:boto:Data: 2023-07-29T00:35:28.724 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.724 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:28.725 DEBUG:boto:Port: 8000 2023-07-29T00:35:28.725 DEBUG:boto:Params: {} 2023-07-29T00:35:28.726 DEBUG:boto:Token: None 2023-07-29T00:35:28.726 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj34?versionId=null 2023-07-29T00:35:28.726 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1FDRRjvUs4+on7BujrSAsX7kxC4= 2023-07-29T00:35:28.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1FDRRjvUs4+on7BujrSAsX7kxC4=', 'Content-Length': '0'} 2023-07-29T00:35:28.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000214a62092fab970b-0064c45ed0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.772 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:28.773 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:28.773 DEBUG:boto:path=/hiiqos-59/?prefix=obj34 2023-07-29T00:35:28.773 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj34 2023-07-29T00:35:28.774 DEBUG:boto:Method: GET 2023-07-29T00:35:28.774 DEBUG:boto:Path: /hiiqos-59/?prefix=obj34 2023-07-29T00:35:28.774 DEBUG:boto:Data: 2023-07-29T00:35:28.775 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.775 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:28.776 DEBUG:boto:Port: 8001 2023-07-29T00:35:28.776 DEBUG:boto:Params: {} 2023-07-29T00:35:28.776 DEBUG:boto:Token: None 2023-07-29T00:35:28.777 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/ 2023-07-29T00:35:28.777 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw= 2023-07-29T00:35:28.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw=', 'Content-Length': '0'} 2023-07-29T00:35:28.780 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000862e0461558d3c36-0064c45ed0-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.780 DEBUG:boto:b'hiiqos-59obj341000falseobj342023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:28.781 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:28.781 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:28.782 DEBUG:boto:path=/hiiqos-59/?prefix=obj34 2023-07-29T00:35:28.782 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj34 2023-07-29T00:35:28.782 DEBUG:boto:Method: GET 2023-07-29T00:35:28.783 DEBUG:boto:Path: /hiiqos-59/?prefix=obj34 2023-07-29T00:35:28.783 DEBUG:boto:Data: 2023-07-29T00:35:28.783 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.784 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:28.784 DEBUG:boto:Port: 8000 2023-07-29T00:35:28.785 DEBUG:boto:Params: {} 2023-07-29T00:35:28.785 DEBUG:boto:Token: None 2023-07-29T00:35:28.785 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/ 2023-07-29T00:35:28.786 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw= 2023-07-29T00:35:28.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw=', 'Content-Length': '0'} 2023-07-29T00:35:28.790 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c20bb4a094d3af6c-0064c45ed0-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.790 DEBUG:boto:b'hiiqos-59obj341000falseobj342023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:28.791 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:35:28.791 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:35:28.792 DEBUG:boto:Method: HEAD 2023-07-29T00:35:28.792 DEBUG:boto:Path: /hiiqos-59/obj34 2023-07-29T00:35:28.792 DEBUG:boto:Data: 2023-07-29T00:35:28.793 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.793 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:28.793 DEBUG:boto:Port: 8001 2023-07-29T00:35:28.794 DEBUG:boto:Params: {} 2023-07-29T00:35:28.794 DEBUG:boto:Token: None 2023-07-29T00:35:28.795 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj34 2023-07-29T00:35:28.795 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qTyyIizCszztSKDnU+6da0oYzh8= 2023-07-29T00:35:28.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qTyyIizCszztSKDnU+6da0oYzh8=', 'Content-Length': '0'} 2023-07-29T00:35:28.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b7578d3ccca2991-0064c45ed0-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.798 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:35:28.798 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:35:28.799 DEBUG:boto:Method: HEAD 2023-07-29T00:35:28.799 DEBUG:boto:Path: /hiiqos-59/obj34 2023-07-29T00:35:28.799 DEBUG:boto:Data: 2023-07-29T00:35:28.800 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.800 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:28.800 DEBUG:boto:Port: 8000 2023-07-29T00:35:28.801 DEBUG:boto:Params: {} 2023-07-29T00:35:28.801 DEBUG:boto:Token: None 2023-07-29T00:35:28.801 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj34 2023-07-29T00:35:28.802 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qTyyIizCszztSKDnU+6da0oYzh8= 2023-07-29T00:35:28.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qTyyIizCszztSKDnU+6da0oYzh8=', 'Content-Length': '0'} 2023-07-29T00:35:28.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d023b9967261c75-0064c45ed0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-07-29T00:35:28.805 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:35:28.805 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:35:28.805 DEBUG:boto:Method: GET 2023-07-29T00:35:28.806 DEBUG:boto:Path: /hiiqos-59/obj34 2023-07-29T00:35:28.806 DEBUG:boto:Data: 2023-07-29T00:35:28.807 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.807 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:28.807 DEBUG:boto:Port: 8001 2023-07-29T00:35:28.808 DEBUG:boto:Params: {} 2023-07-29T00:35:28.808 DEBUG:boto:Token: None 2023-07-29T00:35:28.808 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj34 2023-07-29T00:35:28.809 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KeHKkBgbTIEusbrniKmDBnBUcyY= 2023-07-29T00:35:28.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KeHKkBgbTIEusbrniKmDBnBUcyY=', 'Content-Length': '0'} 2023-07-29T00:35:28.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096d76a44d5a86d36-0064c45ed0-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.852 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:35:28.853 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:35:28.853 DEBUG:boto:Method: GET 2023-07-29T00:35:28.853 DEBUG:boto:Path: /hiiqos-59/obj34 2023-07-29T00:35:28.854 DEBUG:boto:Data: 2023-07-29T00:35:28.854 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.854 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:28.855 DEBUG:boto:Port: 8000 2023-07-29T00:35:28.855 DEBUG:boto:Params: {} 2023-07-29T00:35:28.856 DEBUG:boto:Token: None 2023-07-29T00:35:28.856 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj34 2023-07-29T00:35:28.857 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KeHKkBgbTIEusbrniKmDBnBUcyY= 2023-07-29T00:35:28.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KeHKkBgbTIEusbrniKmDBnBUcyY=', 'Content-Length': '0'} 2023-07-29T00:35:28.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d40a361b04a6deca-0064c45ed0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.860 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:28.860 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:28.860 DEBUG:boto:path=/hiiqos-59/?prefix=obj34 2023-07-29T00:35:28.861 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj34 2023-07-29T00:35:28.861 DEBUG:boto:Method: GET 2023-07-29T00:35:28.861 DEBUG:boto:Path: /hiiqos-59/?prefix=obj34 2023-07-29T00:35:28.862 DEBUG:boto:Data: 2023-07-29T00:35:28.862 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.862 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:28.863 DEBUG:boto:Port: 8001 2023-07-29T00:35:28.863 DEBUG:boto:Params: {} 2023-07-29T00:35:28.864 DEBUG:boto:Token: None 2023-07-29T00:35:28.864 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/ 2023-07-29T00:35:28.864 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw= 2023-07-29T00:35:28.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw=', 'Content-Length': '0'} 2023-07-29T00:35:28.867 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019dda481e5406fc2-0064c45ed0-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.867 DEBUG:boto:b'hiiqos-59obj341000falseobj342023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:28.868 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:28.869 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:28.869 DEBUG:boto:path=/hiiqos-59/?prefix=obj34 2023-07-29T00:35:28.869 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj34 2023-07-29T00:35:28.870 DEBUG:boto:Method: GET 2023-07-29T00:35:28.870 DEBUG:boto:Path: /hiiqos-59/?prefix=obj34 2023-07-29T00:35:28.870 DEBUG:boto:Data: 2023-07-29T00:35:28.871 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.871 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:28.871 DEBUG:boto:Port: 8000 2023-07-29T00:35:28.872 DEBUG:boto:Params: {} 2023-07-29T00:35:28.872 DEBUG:boto:Token: None 2023-07-29T00:35:28.872 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/ 2023-07-29T00:35:28.873 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw= 2023-07-29T00:35:28.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw=', 'Content-Length': '0'} 2023-07-29T00:35:28.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4480e509c2aae97-0064c45ed0-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.877 DEBUG:boto:b'hiiqos-59obj341000falseobj342023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:28.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-07-29T00:35:28.878 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:35:28.879 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:35:28.879 DEBUG:boto:path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:35:28.879 DEBUG:boto:auth_path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:35:28.880 DEBUG:boto:Method: GET 2023-07-29T00:35:28.880 DEBUG:boto:Path: /hiiqos-59/obj38?versionId=null 2023-07-29T00:35:28.880 DEBUG:boto:Data: 2023-07-29T00:35:28.881 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.881 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:28.881 DEBUG:boto:Port: 8001 2023-07-29T00:35:28.882 DEBUG:boto:Params: {} 2023-07-29T00:35:28.882 DEBUG:boto:Token: None 2023-07-29T00:35:28.883 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj38?versionId=null 2023-07-29T00:35:28.883 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9my65rzhoh0gkFp2Rx8pxcjg7sQ= 2023-07-29T00:35:28.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9my65rzhoh0gkFp2Rx8pxcjg7sQ=', 'Content-Length': '0'} 2023-07-29T00:35:28.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d580f602b544a26-0064c45ed0-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.886 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:35:28.887 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:35:28.887 DEBUG:boto:path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:35:28.887 DEBUG:boto:auth_path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:35:28.888 DEBUG:boto:Method: GET 2023-07-29T00:35:28.888 DEBUG:boto:Path: /hiiqos-59/obj38?versionId=null 2023-07-29T00:35:28.889 DEBUG:boto:Data: 2023-07-29T00:35:28.889 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.889 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:28.890 DEBUG:boto:Port: 8000 2023-07-29T00:35:28.890 DEBUG:boto:Params: {} 2023-07-29T00:35:28.890 DEBUG:boto:Token: None 2023-07-29T00:35:28.891 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj38?versionId=null 2023-07-29T00:35:28.891 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9my65rzhoh0gkFp2Rx8pxcjg7sQ= 2023-07-29T00:35:28.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9my65rzhoh0gkFp2Rx8pxcjg7sQ=', 'Content-Length': '0'} 2023-07-29T00:35:28.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6f9819e3176ae41-0064c45ed0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.936 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:35:28.937 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:35:28.937 DEBUG:boto:path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:35:28.937 DEBUG:boto:auth_path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:35:28.938 DEBUG:boto:Method: HEAD 2023-07-29T00:35:28.938 DEBUG:boto:Path: /hiiqos-59/obj38?versionId=null 2023-07-29T00:35:28.938 DEBUG:boto:Data: 2023-07-29T00:35:28.939 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.939 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:28.939 DEBUG:boto:Port: 8001 2023-07-29T00:35:28.940 DEBUG:boto:Params: {} 2023-07-29T00:35:28.940 DEBUG:boto:Token: None 2023-07-29T00:35:28.940 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj38?versionId=null 2023-07-29T00:35:28.941 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Z9s92GStDIh9rekJ7dzykTQOXpc= 2023-07-29T00:35:28.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Z9s92GStDIh9rekJ7dzykTQOXpc=', 'Content-Length': '0'} 2023-07-29T00:35:28.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc275d60482cd1b4-0064c45ed0-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.944 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:35:28.944 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:35:28.945 DEBUG:boto:path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:35:28.945 DEBUG:boto:auth_path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:35:28.945 DEBUG:boto:Method: HEAD 2023-07-29T00:35:28.946 DEBUG:boto:Path: /hiiqos-59/obj38?versionId=null 2023-07-29T00:35:28.946 DEBUG:boto:Data: 2023-07-29T00:35:28.946 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.947 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:28.947 DEBUG:boto:Port: 8000 2023-07-29T00:35:28.947 DEBUG:boto:Params: {} 2023-07-29T00:35:28.948 DEBUG:boto:Token: None 2023-07-29T00:35:28.948 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj38?versionId=null 2023-07-29T00:35:28.949 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Z9s92GStDIh9rekJ7dzykTQOXpc= 2023-07-29T00:35:28.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Z9s92GStDIh9rekJ7dzykTQOXpc=', 'Content-Length': '0'} 2023-07-29T00:35:28.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f91f0086e0576a4-0064c45ed0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-07-29T00:35:28.952 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:35:28.952 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:35:28.953 DEBUG:boto:path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:35:28.953 DEBUG:boto:auth_path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:35:28.953 DEBUG:boto:Method: GET 2023-07-29T00:35:28.954 DEBUG:boto:Path: /hiiqos-59/obj38?versionId=null 2023-07-29T00:35:28.954 DEBUG:boto:Data: 2023-07-29T00:35:28.954 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.955 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:28.955 DEBUG:boto:Port: 8001 2023-07-29T00:35:28.955 DEBUG:boto:Params: {} 2023-07-29T00:35:28.956 DEBUG:boto:Token: None 2023-07-29T00:35:28.956 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj38?versionId=null 2023-07-29T00:35:28.957 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9my65rzhoh0gkFp2Rx8pxcjg7sQ= 2023-07-29T00:35:28.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9my65rzhoh0gkFp2Rx8pxcjg7sQ=', 'Content-Length': '0'} 2023-07-29T00:35:28.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b7c9e1796ed0d51-0064c45ed0-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.960 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:35:28.960 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:35:28.960 DEBUG:boto:path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:35:28.961 DEBUG:boto:auth_path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:35:28.961 DEBUG:boto:Method: GET 2023-07-29T00:35:28.961 DEBUG:boto:Path: /hiiqos-59/obj38?versionId=null 2023-07-29T00:35:28.962 DEBUG:boto:Data: 2023-07-29T00:35:28.962 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.962 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:28.963 DEBUG:boto:Port: 8000 2023-07-29T00:35:28.963 DEBUG:boto:Params: {} 2023-07-29T00:35:28.964 DEBUG:boto:Token: None 2023-07-29T00:35:28.964 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj38?versionId=null 2023-07-29T00:35:28.964 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9my65rzhoh0gkFp2Rx8pxcjg7sQ= 2023-07-29T00:35:28.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9my65rzhoh0gkFp2Rx8pxcjg7sQ=', 'Content-Length': '0'} 2023-07-29T00:35:28.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ad3d8da4935f2d5-0064c45ed0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.967 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:28.968 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:28.968 DEBUG:boto:path=/hiiqos-59/?prefix=obj38 2023-07-29T00:35:28.969 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj38 2023-07-29T00:35:28.969 DEBUG:boto:Method: GET 2023-07-29T00:35:28.969 DEBUG:boto:Path: /hiiqos-59/?prefix=obj38 2023-07-29T00:35:28.970 DEBUG:boto:Data: 2023-07-29T00:35:28.970 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.970 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:28.971 DEBUG:boto:Port: 8001 2023-07-29T00:35:28.971 DEBUG:boto:Params: {} 2023-07-29T00:35:28.971 DEBUG:boto:Token: None 2023-07-29T00:35:28.972 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/ 2023-07-29T00:35:28.972 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw= 2023-07-29T00:35:28.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw=', 'Content-Length': '0'} 2023-07-29T00:35:28.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2142225474ed9c9-0064c45ed0-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.975 DEBUG:boto:b'hiiqos-59obj381000falseobj382023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:28.976 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:28.977 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:28.977 DEBUG:boto:path=/hiiqos-59/?prefix=obj38 2023-07-29T00:35:28.977 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj38 2023-07-29T00:35:28.978 DEBUG:boto:Method: GET 2023-07-29T00:35:28.978 DEBUG:boto:Path: /hiiqos-59/?prefix=obj38 2023-07-29T00:35:28.978 DEBUG:boto:Data: 2023-07-29T00:35:28.979 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.979 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:28.979 DEBUG:boto:Port: 8000 2023-07-29T00:35:28.980 DEBUG:boto:Params: {} 2023-07-29T00:35:28.980 DEBUG:boto:Token: None 2023-07-29T00:35:28.981 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/ 2023-07-29T00:35:28.981 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw= 2023-07-29T00:35:28.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:801JmVKD+pvsfllqEUpRsoyTeIw=', 'Content-Length': '0'} 2023-07-29T00:35:28.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb8bb2d892ea3fd3-0064c45ed0-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.985 DEBUG:boto:b'hiiqos-59obj381000falseobj382023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:28.986 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:35:28.987 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:35:28.987 DEBUG:boto:Method: HEAD 2023-07-29T00:35:28.987 DEBUG:boto:Path: /hiiqos-59/obj38 2023-07-29T00:35:28.988 DEBUG:boto:Data: 2023-07-29T00:35:28.988 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.988 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:28.989 DEBUG:boto:Port: 8001 2023-07-29T00:35:28.989 DEBUG:boto:Params: {} 2023-07-29T00:35:28.989 DEBUG:boto:Token: None 2023-07-29T00:35:28.990 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj38 2023-07-29T00:35:28.990 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S/81kDrWSS2q6QIZD/gwgtLaWF4= 2023-07-29T00:35:28.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S/81kDrWSS2q6QIZD/gwgtLaWF4=', 'Content-Length': '0'} 2023-07-29T00:35:28.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e3e6c526f7a4a7d-0064c45ed0-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.993 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:35:28.993 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:35:28.993 DEBUG:boto:Method: HEAD 2023-07-29T00:35:28.994 DEBUG:boto:Path: /hiiqos-59/obj38 2023-07-29T00:35:28.994 DEBUG:boto:Data: 2023-07-29T00:35:28.994 DEBUG:boto:Headers: {} 2023-07-29T00:35:28.995 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:28.995 DEBUG:boto:Port: 8000 2023-07-29T00:35:28.995 DEBUG:boto:Params: {} 2023-07-29T00:35:28.996 DEBUG:boto:Token: None 2023-07-29T00:35:28.996 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:28 GMT /hiiqos-59/obj38 2023-07-29T00:35:28.997 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S/81kDrWSS2q6QIZD/gwgtLaWF4= 2023-07-29T00:35:28.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S/81kDrWSS2q6QIZD/gwgtLaWF4=', 'Content-Length': '0'} 2023-07-29T00:35:28.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8afecae48fb351b-0064c45ed0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:28.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-07-29T00:35:29.000 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:35:29.000 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:35:29.001 DEBUG:boto:Method: GET 2023-07-29T00:35:29.001 DEBUG:boto:Path: /hiiqos-59/obj38 2023-07-29T00:35:29.001 DEBUG:boto:Data: 2023-07-29T00:35:29.002 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.002 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:29.002 DEBUG:boto:Port: 8001 2023-07-29T00:35:29.003 DEBUG:boto:Params: {} 2023-07-29T00:35:29.003 DEBUG:boto:Token: None 2023-07-29T00:35:29.004 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj38 2023-07-29T00:35:29.004 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7lkfu/iBweraRQ9x2t5wPjq8qGY= 2023-07-29T00:35:29.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7lkfu/iBweraRQ9x2t5wPjq8qGY=', 'Content-Length': '0'} 2023-07-29T00:35:29.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024df0ccf4c0849f3-0064c45ed0-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.007 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:35:29.008 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:35:29.008 DEBUG:boto:Method: GET 2023-07-29T00:35:29.008 DEBUG:boto:Path: /hiiqos-59/obj38 2023-07-29T00:35:29.009 DEBUG:boto:Data: 2023-07-29T00:35:29.009 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.009 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:29.010 DEBUG:boto:Port: 8000 2023-07-29T00:35:29.010 DEBUG:boto:Params: {} 2023-07-29T00:35:29.010 DEBUG:boto:Token: None 2023-07-29T00:35:29.011 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj38 2023-07-29T00:35:29.011 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7lkfu/iBweraRQ9x2t5wPjq8qGY= 2023-07-29T00:35:29.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7lkfu/iBweraRQ9x2t5wPjq8qGY=', 'Content-Length': '0'} 2023-07-29T00:35:29.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041a2b2093a235b85-0064c45ed0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.014 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:29.015 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:29.015 DEBUG:boto:path=/hiiqos-59/?prefix=obj38 2023-07-29T00:35:29.015 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj38 2023-07-29T00:35:29.016 DEBUG:boto:Method: GET 2023-07-29T00:35:29.016 DEBUG:boto:Path: /hiiqos-59/?prefix=obj38 2023-07-29T00:35:29.016 DEBUG:boto:Data: 2023-07-29T00:35:29.017 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.017 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:29.017 DEBUG:boto:Port: 8001 2023-07-29T00:35:29.018 DEBUG:boto:Params: {} 2023-07-29T00:35:29.018 DEBUG:boto:Token: None 2023-07-29T00:35:29.019 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/ 2023-07-29T00:35:29.019 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O4RRBqFCnYe7CW88crqf26I4JfM= 2023-07-29T00:35:29.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O4RRBqFCnYe7CW88crqf26I4JfM=', 'Content-Length': '0'} 2023-07-29T00:35:29.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfdbb33ce23cd89e-0064c45ed0-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.022 DEBUG:boto:b'hiiqos-59obj381000falseobj382023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:29.023 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:29.023 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:29.023 DEBUG:boto:path=/hiiqos-59/?prefix=obj38 2023-07-29T00:35:29.024 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj38 2023-07-29T00:35:29.024 DEBUG:boto:Method: GET 2023-07-29T00:35:29.025 DEBUG:boto:Path: /hiiqos-59/?prefix=obj38 2023-07-29T00:35:29.025 DEBUG:boto:Data: 2023-07-29T00:35:29.025 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.026 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:29.026 DEBUG:boto:Port: 8000 2023-07-29T00:35:29.026 DEBUG:boto:Params: {} 2023-07-29T00:35:29.027 DEBUG:boto:Token: None 2023-07-29T00:35:29.027 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/ 2023-07-29T00:35:29.027 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O4RRBqFCnYe7CW88crqf26I4JfM= 2023-07-29T00:35:29.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O4RRBqFCnYe7CW88crqf26I4JfM=', 'Content-Length': '0'} 2023-07-29T00:35:29.031 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f7d0bc3572b356e-0064c45ed0-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.032 DEBUG:boto:b'hiiqos-59obj381000falseobj382023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:29.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-07-29T00:35:29.033 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:35:29.033 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:35:29.034 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:35:29.034 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:35:29.034 DEBUG:boto:Method: GET 2023-07-29T00:35:29.035 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:35:29.035 DEBUG:boto:Data: 2023-07-29T00:35:29.035 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.036 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:29.036 DEBUG:boto:Port: 8001 2023-07-29T00:35:29.037 DEBUG:boto:Params: {} 2023-07-29T00:35:29.037 DEBUG:boto:Token: None 2023-07-29T00:35:29.038 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:35:29.038 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LozYKXUUVVMQsbCK/MDq51uV1bE= 2023-07-29T00:35:29.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LozYKXUUVVMQsbCK/MDq51uV1bE=', 'Content-Length': '0'} 2023-07-29T00:35:29.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fce726e8e3be4fe6-0064c45ed0-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.084 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:35:29.085 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:35:29.085 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:35:29.085 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:35:29.086 DEBUG:boto:Method: GET 2023-07-29T00:35:29.086 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:35:29.087 DEBUG:boto:Data: 2023-07-29T00:35:29.087 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.087 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:29.088 DEBUG:boto:Port: 8000 2023-07-29T00:35:29.088 DEBUG:boto:Params: {} 2023-07-29T00:35:29.088 DEBUG:boto:Token: None 2023-07-29T00:35:29.089 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:35:29.089 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LozYKXUUVVMQsbCK/MDq51uV1bE= 2023-07-29T00:35:29.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LozYKXUUVVMQsbCK/MDq51uV1bE=', 'Content-Length': '0'} 2023-07-29T00:35:29.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5e43f9a4088e7dd-0064c45ed1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.092 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:35:29.093 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:35:29.093 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:35:29.093 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:35:29.094 DEBUG:boto:Method: HEAD 2023-07-29T00:35:29.094 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:35:29.094 DEBUG:boto:Data: 2023-07-29T00:35:29.095 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.095 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:29.095 DEBUG:boto:Port: 8001 2023-07-29T00:35:29.096 DEBUG:boto:Params: {} 2023-07-29T00:35:29.096 DEBUG:boto:Token: None 2023-07-29T00:35:29.097 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:35:29.097 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Vpp4TnQa1I5hU4mWFk3v83TqV/8= 2023-07-29T00:35:29.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Vpp4TnQa1I5hU4mWFk3v83TqV/8=', 'Content-Length': '0'} 2023-07-29T00:35:29.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006afa577607206015-0064c45ed1-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.100 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:35:29.100 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:35:29.101 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:35:29.101 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:35:29.101 DEBUG:boto:Method: HEAD 2023-07-29T00:35:29.102 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:35:29.102 DEBUG:boto:Data: 2023-07-29T00:35:29.102 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.103 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:29.103 DEBUG:boto:Port: 8000 2023-07-29T00:35:29.103 DEBUG:boto:Params: {} 2023-07-29T00:35:29.104 DEBUG:boto:Token: None 2023-07-29T00:35:29.104 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:35:29.105 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Vpp4TnQa1I5hU4mWFk3v83TqV/8= 2023-07-29T00:35:29.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Vpp4TnQa1I5hU4mWFk3v83TqV/8=', 'Content-Length': '0'} 2023-07-29T00:35:29.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f45c3965540b54a0-0064c45ed1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-07-29T00:35:29.108 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:35:29.108 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:35:29.109 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:35:29.109 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:35:29.109 DEBUG:boto:Method: GET 2023-07-29T00:35:29.110 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:35:29.110 DEBUG:boto:Data: 2023-07-29T00:35:29.110 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.111 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:29.111 DEBUG:boto:Port: 8001 2023-07-29T00:35:29.111 DEBUG:boto:Params: {} 2023-07-29T00:35:29.112 DEBUG:boto:Token: None 2023-07-29T00:35:29.112 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:35:29.113 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LozYKXUUVVMQsbCK/MDq51uV1bE= 2023-07-29T00:35:29.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LozYKXUUVVMQsbCK/MDq51uV1bE=', 'Content-Length': '0'} 2023-07-29T00:35:29.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f6731d0c2526937-0064c45ed1-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.115 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:35:29.116 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:35:29.116 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:35:29.117 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:35:29.117 DEBUG:boto:Method: GET 2023-07-29T00:35:29.117 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:35:29.118 DEBUG:boto:Data: 2023-07-29T00:35:29.118 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.118 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:29.119 DEBUG:boto:Port: 8000 2023-07-29T00:35:29.119 DEBUG:boto:Params: {} 2023-07-29T00:35:29.119 DEBUG:boto:Token: None 2023-07-29T00:35:29.120 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:35:29.120 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LozYKXUUVVMQsbCK/MDq51uV1bE= 2023-07-29T00:35:29.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LozYKXUUVVMQsbCK/MDq51uV1bE=', 'Content-Length': '0'} 2023-07-29T00:35:29.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bec5b2ec4fb78ebc-0064c45ed1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.123 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:29.123 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:29.124 DEBUG:boto:path=/hiiqos-59/?prefix=obj44 2023-07-29T00:35:29.124 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj44 2023-07-29T00:35:29.124 DEBUG:boto:Method: GET 2023-07-29T00:35:29.125 DEBUG:boto:Path: /hiiqos-59/?prefix=obj44 2023-07-29T00:35:29.125 DEBUG:boto:Data: 2023-07-29T00:35:29.125 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.126 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:29.126 DEBUG:boto:Port: 8001 2023-07-29T00:35:29.127 DEBUG:boto:Params: {} 2023-07-29T00:35:29.127 DEBUG:boto:Token: None 2023-07-29T00:35:29.127 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/ 2023-07-29T00:35:29.128 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O4RRBqFCnYe7CW88crqf26I4JfM= 2023-07-29T00:35:29.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O4RRBqFCnYe7CW88crqf26I4JfM=', 'Content-Length': '0'} 2023-07-29T00:35:29.131 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000619077e0a11c34b6-0064c45ed1-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.131 DEBUG:boto:b'hiiqos-59obj441000falseobj442023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:29.132 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:29.132 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:29.133 DEBUG:boto:path=/hiiqos-59/?prefix=obj44 2023-07-29T00:35:29.133 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj44 2023-07-29T00:35:29.133 DEBUG:boto:Method: GET 2023-07-29T00:35:29.134 DEBUG:boto:Path: /hiiqos-59/?prefix=obj44 2023-07-29T00:35:29.134 DEBUG:boto:Data: 2023-07-29T00:35:29.135 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.135 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:29.135 DEBUG:boto:Port: 8000 2023-07-29T00:35:29.136 DEBUG:boto:Params: {} 2023-07-29T00:35:29.136 DEBUG:boto:Token: None 2023-07-29T00:35:29.137 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/ 2023-07-29T00:35:29.137 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O4RRBqFCnYe7CW88crqf26I4JfM= 2023-07-29T00:35:29.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O4RRBqFCnYe7CW88crqf26I4JfM=', 'Content-Length': '0'} 2023-07-29T00:35:29.141 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a2847db6c6ad6e6-0064c45ed1-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.142 DEBUG:boto:b'hiiqos-59obj441000falseobj442023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:29.143 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:35:29.143 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:35:29.143 DEBUG:boto:Method: HEAD 2023-07-29T00:35:29.144 DEBUG:boto:Path: /hiiqos-59/obj44 2023-07-29T00:35:29.144 DEBUG:boto:Data: 2023-07-29T00:35:29.144 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.145 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:29.145 DEBUG:boto:Port: 8001 2023-07-29T00:35:29.145 DEBUG:boto:Params: {} 2023-07-29T00:35:29.146 DEBUG:boto:Token: None 2023-07-29T00:35:29.146 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj44 2023-07-29T00:35:29.146 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:i0nbmIRFirrYyeYUlEUvZH2yEyg= 2023-07-29T00:35:29.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:i0nbmIRFirrYyeYUlEUvZH2yEyg=', 'Content-Length': '0'} 2023-07-29T00:35:29.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000282263a4b2eea783-0064c45ed1-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.149 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:35:29.150 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:35:29.150 DEBUG:boto:Method: HEAD 2023-07-29T00:35:29.150 DEBUG:boto:Path: /hiiqos-59/obj44 2023-07-29T00:35:29.151 DEBUG:boto:Data: 2023-07-29T00:35:29.151 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.151 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:29.152 DEBUG:boto:Port: 8000 2023-07-29T00:35:29.152 DEBUG:boto:Params: {} 2023-07-29T00:35:29.152 DEBUG:boto:Token: None 2023-07-29T00:35:29.153 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj44 2023-07-29T00:35:29.153 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:i0nbmIRFirrYyeYUlEUvZH2yEyg= 2023-07-29T00:35:29.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:i0nbmIRFirrYyeYUlEUvZH2yEyg=', 'Content-Length': '0'} 2023-07-29T00:35:29.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5c5d1c8e58b98ce-0064c45ed1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-07-29T00:35:29.156 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:35:29.157 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:35:29.157 DEBUG:boto:Method: GET 2023-07-29T00:35:29.157 DEBUG:boto:Path: /hiiqos-59/obj44 2023-07-29T00:35:29.158 DEBUG:boto:Data: 2023-07-29T00:35:29.158 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.158 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:29.159 DEBUG:boto:Port: 8001 2023-07-29T00:35:29.159 DEBUG:boto:Params: {} 2023-07-29T00:35:29.159 DEBUG:boto:Token: None 2023-07-29T00:35:29.160 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj44 2023-07-29T00:35:29.160 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:f0x7wI0n8KIu9PJIlTYjUzNZ/xo= 2023-07-29T00:35:29.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:f0x7wI0n8KIu9PJIlTYjUzNZ/xo=', 'Content-Length': '0'} 2023-07-29T00:35:29.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000712985aa75689f1a-0064c45ed1-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.163 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:35:29.163 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:35:29.164 DEBUG:boto:Method: GET 2023-07-29T00:35:29.164 DEBUG:boto:Path: /hiiqos-59/obj44 2023-07-29T00:35:29.165 DEBUG:boto:Data: 2023-07-29T00:35:29.165 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.165 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:29.166 DEBUG:boto:Port: 8000 2023-07-29T00:35:29.166 DEBUG:boto:Params: {} 2023-07-29T00:35:29.166 DEBUG:boto:Token: None 2023-07-29T00:35:29.167 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj44 2023-07-29T00:35:29.167 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:f0x7wI0n8KIu9PJIlTYjUzNZ/xo= 2023-07-29T00:35:29.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:f0x7wI0n8KIu9PJIlTYjUzNZ/xo=', 'Content-Length': '0'} 2023-07-29T00:35:29.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058bf69a11a634d2a-0064c45ed1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.212 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:29.213 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:29.213 DEBUG:boto:path=/hiiqos-59/?prefix=obj44 2023-07-29T00:35:29.213 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj44 2023-07-29T00:35:29.214 DEBUG:boto:Method: GET 2023-07-29T00:35:29.214 DEBUG:boto:Path: /hiiqos-59/?prefix=obj44 2023-07-29T00:35:29.214 DEBUG:boto:Data: 2023-07-29T00:35:29.215 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.215 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:29.216 DEBUG:boto:Port: 8001 2023-07-29T00:35:29.216 DEBUG:boto:Params: {} 2023-07-29T00:35:29.216 DEBUG:boto:Token: None 2023-07-29T00:35:29.217 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/ 2023-07-29T00:35:29.217 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O4RRBqFCnYe7CW88crqf26I4JfM= 2023-07-29T00:35:29.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O4RRBqFCnYe7CW88crqf26I4JfM=', 'Content-Length': '0'} 2023-07-29T00:35:29.220 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2371aa1d691aee2-0064c45ed1-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.221 DEBUG:boto:b'hiiqos-59obj441000falseobj442023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:29.221 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:29.222 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:29.222 DEBUG:boto:path=/hiiqos-59/?prefix=obj44 2023-07-29T00:35:29.222 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj44 2023-07-29T00:35:29.223 DEBUG:boto:Method: GET 2023-07-29T00:35:29.223 DEBUG:boto:Path: /hiiqos-59/?prefix=obj44 2023-07-29T00:35:29.223 DEBUG:boto:Data: 2023-07-29T00:35:29.224 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.224 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:29.225 DEBUG:boto:Port: 8000 2023-07-29T00:35:29.225 DEBUG:boto:Params: {} 2023-07-29T00:35:29.225 DEBUG:boto:Token: None 2023-07-29T00:35:29.226 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/ 2023-07-29T00:35:29.226 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O4RRBqFCnYe7CW88crqf26I4JfM= 2023-07-29T00:35:29.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O4RRBqFCnYe7CW88crqf26I4JfM=', 'Content-Length': '0'} 2023-07-29T00:35:29.230 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085ae7b736d9e1e8d-0064c45ed1-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.231 DEBUG:boto:b'hiiqos-59obj441000falseobj442023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:29.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-07-29T00:35:29.232 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:35:29.232 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:35:29.232 DEBUG:boto:path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:35:29.233 DEBUG:boto:auth_path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:35:29.233 DEBUG:boto:Method: GET 2023-07-29T00:35:29.233 DEBUG:boto:Path: /hiiqos-59/obj46?versionId=null 2023-07-29T00:35:29.234 DEBUG:boto:Data: 2023-07-29T00:35:29.234 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.234 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:29.235 DEBUG:boto:Port: 8001 2023-07-29T00:35:29.235 DEBUG:boto:Params: {} 2023-07-29T00:35:29.236 DEBUG:boto:Token: None 2023-07-29T00:35:29.236 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj46?versionId=null 2023-07-29T00:35:29.236 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rCWxAmHY1UzfI9pb3f9ASBSxqBA= 2023-07-29T00:35:29.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rCWxAmHY1UzfI9pb3f9ASBSxqBA=', 'Content-Length': '0'} 2023-07-29T00:35:29.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d32e84dad971e169-0064c45ed1-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.280 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:35:29.281 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:35:29.281 DEBUG:boto:path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:35:29.281 DEBUG:boto:auth_path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:35:29.282 DEBUG:boto:Method: GET 2023-07-29T00:35:29.282 DEBUG:boto:Path: /hiiqos-59/obj46?versionId=null 2023-07-29T00:35:29.282 DEBUG:boto:Data: 2023-07-29T00:35:29.283 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.283 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:29.283 DEBUG:boto:Port: 8000 2023-07-29T00:35:29.284 DEBUG:boto:Params: {} 2023-07-29T00:35:29.284 DEBUG:boto:Token: None 2023-07-29T00:35:29.285 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj46?versionId=null 2023-07-29T00:35:29.285 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rCWxAmHY1UzfI9pb3f9ASBSxqBA= 2023-07-29T00:35:29.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rCWxAmHY1UzfI9pb3f9ASBSxqBA=', 'Content-Length': '0'} 2023-07-29T00:35:29.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002393e02374719c8d-0064c45ed1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.288 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:35:29.288 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:35:29.289 DEBUG:boto:path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:35:29.289 DEBUG:boto:auth_path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:35:29.289 DEBUG:boto:Method: HEAD 2023-07-29T00:35:29.290 DEBUG:boto:Path: /hiiqos-59/obj46?versionId=null 2023-07-29T00:35:29.290 DEBUG:boto:Data: 2023-07-29T00:35:29.290 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.291 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:29.291 DEBUG:boto:Port: 8001 2023-07-29T00:35:29.291 DEBUG:boto:Params: {} 2023-07-29T00:35:29.292 DEBUG:boto:Token: None 2023-07-29T00:35:29.292 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj46?versionId=null 2023-07-29T00:35:29.293 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rKiC4fgJFj/8ttFYzCbs/Wbq9ac= 2023-07-29T00:35:29.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rKiC4fgJFj/8ttFYzCbs/Wbq9ac=', 'Content-Length': '0'} 2023-07-29T00:35:29.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000557ca7b87fdd431d-0064c45ed1-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.295 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:35:29.296 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:35:29.296 DEBUG:boto:path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:35:29.296 DEBUG:boto:auth_path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:35:29.297 DEBUG:boto:Method: HEAD 2023-07-29T00:35:29.297 DEBUG:boto:Path: /hiiqos-59/obj46?versionId=null 2023-07-29T00:35:29.297 DEBUG:boto:Data: 2023-07-29T00:35:29.298 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.298 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:29.298 DEBUG:boto:Port: 8000 2023-07-29T00:35:29.299 DEBUG:boto:Params: {} 2023-07-29T00:35:29.299 DEBUG:boto:Token: None 2023-07-29T00:35:29.299 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj46?versionId=null 2023-07-29T00:35:29.300 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rKiC4fgJFj/8ttFYzCbs/Wbq9ac= 2023-07-29T00:35:29.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rKiC4fgJFj/8ttFYzCbs/Wbq9ac=', 'Content-Length': '0'} 2023-07-29T00:35:29.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009580c6ac9ce61841-0064c45ed1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-07-29T00:35:29.303 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:35:29.303 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:35:29.304 DEBUG:boto:path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:35:29.304 DEBUG:boto:auth_path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:35:29.304 DEBUG:boto:Method: GET 2023-07-29T00:35:29.305 DEBUG:boto:Path: /hiiqos-59/obj46?versionId=null 2023-07-29T00:35:29.305 DEBUG:boto:Data: 2023-07-29T00:35:29.305 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.306 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:29.306 DEBUG:boto:Port: 8001 2023-07-29T00:35:29.307 DEBUG:boto:Params: {} 2023-07-29T00:35:29.307 DEBUG:boto:Token: None 2023-07-29T00:35:29.307 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj46?versionId=null 2023-07-29T00:35:29.308 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rCWxAmHY1UzfI9pb3f9ASBSxqBA= 2023-07-29T00:35:29.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rCWxAmHY1UzfI9pb3f9ASBSxqBA=', 'Content-Length': '0'} 2023-07-29T00:35:29.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e1f8d76ba82d7e3-0064c45ed1-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.311 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:35:29.311 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:35:29.311 DEBUG:boto:path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:35:29.312 DEBUG:boto:auth_path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:35:29.312 DEBUG:boto:Method: GET 2023-07-29T00:35:29.313 DEBUG:boto:Path: /hiiqos-59/obj46?versionId=null 2023-07-29T00:35:29.313 DEBUG:boto:Data: 2023-07-29T00:35:29.313 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.314 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:29.314 DEBUG:boto:Port: 8000 2023-07-29T00:35:29.314 DEBUG:boto:Params: {} 2023-07-29T00:35:29.315 DEBUG:boto:Token: None 2023-07-29T00:35:29.315 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj46?versionId=null 2023-07-29T00:35:29.316 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rCWxAmHY1UzfI9pb3f9ASBSxqBA= 2023-07-29T00:35:29.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rCWxAmHY1UzfI9pb3f9ASBSxqBA=', 'Content-Length': '0'} 2023-07-29T00:35:29.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009280fd01dac38a65-0064c45ed1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.319 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:29.319 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:29.319 DEBUG:boto:path=/hiiqos-59/?prefix=obj46 2023-07-29T00:35:29.320 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj46 2023-07-29T00:35:29.320 DEBUG:boto:Method: GET 2023-07-29T00:35:29.320 DEBUG:boto:Path: /hiiqos-59/?prefix=obj46 2023-07-29T00:35:29.321 DEBUG:boto:Data: 2023-07-29T00:35:29.321 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.321 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:29.322 DEBUG:boto:Port: 8001 2023-07-29T00:35:29.322 DEBUG:boto:Params: {} 2023-07-29T00:35:29.323 DEBUG:boto:Token: None 2023-07-29T00:35:29.323 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/ 2023-07-29T00:35:29.323 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O4RRBqFCnYe7CW88crqf26I4JfM= 2023-07-29T00:35:29.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O4RRBqFCnYe7CW88crqf26I4JfM=', 'Content-Length': '0'} 2023-07-29T00:35:29.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a912db2f859f64d8-0064c45ed1-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.329 DEBUG:boto:b'hiiqos-59obj461000falseobj462023-07-29T00:33:06.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:29.330 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:29.330 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:29.330 DEBUG:boto:path=/hiiqos-59/?prefix=obj46 2023-07-29T00:35:29.331 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj46 2023-07-29T00:35:29.331 DEBUG:boto:Method: GET 2023-07-29T00:35:29.332 DEBUG:boto:Path: /hiiqos-59/?prefix=obj46 2023-07-29T00:35:29.332 DEBUG:boto:Data: 2023-07-29T00:35:29.332 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.333 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:29.333 DEBUG:boto:Port: 8000 2023-07-29T00:35:29.333 DEBUG:boto:Params: {} 2023-07-29T00:35:29.334 DEBUG:boto:Token: None 2023-07-29T00:35:29.334 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/ 2023-07-29T00:35:29.335 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O4RRBqFCnYe7CW88crqf26I4JfM= 2023-07-29T00:35:29.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O4RRBqFCnYe7CW88crqf26I4JfM=', 'Content-Length': '0'} 2023-07-29T00:35:29.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f0b9161e4bf9b02-0064c45ed1-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.339 DEBUG:boto:b'hiiqos-59obj461000falseobj462023-07-29T00:33:06.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:29.340 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:35:29.340 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:35:29.341 DEBUG:boto:Method: HEAD 2023-07-29T00:35:29.341 DEBUG:boto:Path: /hiiqos-59/obj46 2023-07-29T00:35:29.341 DEBUG:boto:Data: 2023-07-29T00:35:29.342 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.342 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:29.342 DEBUG:boto:Port: 8001 2023-07-29T00:35:29.343 DEBUG:boto:Params: {} 2023-07-29T00:35:29.343 DEBUG:boto:Token: None 2023-07-29T00:35:29.343 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj46 2023-07-29T00:35:29.344 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dS9i7igdB9GdEgm0sZXZH/eiXKE= 2023-07-29T00:35:29.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dS9i7igdB9GdEgm0sZXZH/eiXKE=', 'Content-Length': '0'} 2023-07-29T00:35:29.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000177e8eb00d99a89f-0064c45ed1-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.346 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:35:29.347 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:35:29.347 DEBUG:boto:Method: HEAD 2023-07-29T00:35:29.348 DEBUG:boto:Path: /hiiqos-59/obj46 2023-07-29T00:35:29.348 DEBUG:boto:Data: 2023-07-29T00:35:29.348 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.349 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:29.349 DEBUG:boto:Port: 8000 2023-07-29T00:35:29.349 DEBUG:boto:Params: {} 2023-07-29T00:35:29.350 DEBUG:boto:Token: None 2023-07-29T00:35:29.350 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj46 2023-07-29T00:35:29.350 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dS9i7igdB9GdEgm0sZXZH/eiXKE= 2023-07-29T00:35:29.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dS9i7igdB9GdEgm0sZXZH/eiXKE=', 'Content-Length': '0'} 2023-07-29T00:35:29.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008040a82928bfe5f2-0064c45ed1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-07-29T00:35:29.353 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:35:29.354 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:35:29.354 DEBUG:boto:Method: GET 2023-07-29T00:35:29.354 DEBUG:boto:Path: /hiiqos-59/obj46 2023-07-29T00:35:29.355 DEBUG:boto:Data: 2023-07-29T00:35:29.355 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.355 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:29.356 DEBUG:boto:Port: 8001 2023-07-29T00:35:29.356 DEBUG:boto:Params: {} 2023-07-29T00:35:29.357 DEBUG:boto:Token: None 2023-07-29T00:35:29.357 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj46 2023-07-29T00:35:29.357 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AcwdNRRBPNpCO1daXa4h6Rohi2U= 2023-07-29T00:35:29.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AcwdNRRBPNpCO1daXa4h6Rohi2U=', 'Content-Length': '0'} 2023-07-29T00:35:29.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e1aa1009417d6e2-0064c45ed1-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.360 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:35:29.361 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:35:29.361 DEBUG:boto:Method: GET 2023-07-29T00:35:29.362 DEBUG:boto:Path: /hiiqos-59/obj46 2023-07-29T00:35:29.362 DEBUG:boto:Data: 2023-07-29T00:35:29.362 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.363 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:29.363 DEBUG:boto:Port: 8000 2023-07-29T00:35:29.363 DEBUG:boto:Params: {} 2023-07-29T00:35:29.364 DEBUG:boto:Token: None 2023-07-29T00:35:29.364 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj46 2023-07-29T00:35:29.365 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AcwdNRRBPNpCO1daXa4h6Rohi2U= 2023-07-29T00:35:29.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AcwdNRRBPNpCO1daXa4h6Rohi2U=', 'Content-Length': '0'} 2023-07-29T00:35:29.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddf3647434c61b54-0064c45ed1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.408 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:29.409 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:29.409 DEBUG:boto:path=/hiiqos-59/?prefix=obj46 2023-07-29T00:35:29.410 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj46 2023-07-29T00:35:29.410 DEBUG:boto:Method: GET 2023-07-29T00:35:29.410 DEBUG:boto:Path: /hiiqos-59/?prefix=obj46 2023-07-29T00:35:29.411 DEBUG:boto:Data: 2023-07-29T00:35:29.411 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.411 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:29.412 DEBUG:boto:Port: 8001 2023-07-29T00:35:29.412 DEBUG:boto:Params: {} 2023-07-29T00:35:29.413 DEBUG:boto:Token: None 2023-07-29T00:35:29.413 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/ 2023-07-29T00:35:29.413 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O4RRBqFCnYe7CW88crqf26I4JfM= 2023-07-29T00:35:29.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O4RRBqFCnYe7CW88crqf26I4JfM=', 'Content-Length': '0'} 2023-07-29T00:35:29.416 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007de670f1506c10e-0064c45ed1-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.417 DEBUG:boto:b'hiiqos-59obj461000falseobj462023-07-29T00:33:06.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:29.418 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:29.418 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:29.418 DEBUG:boto:path=/hiiqos-59/?prefix=obj46 2023-07-29T00:35:29.419 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj46 2023-07-29T00:35:29.419 DEBUG:boto:Method: GET 2023-07-29T00:35:29.419 DEBUG:boto:Path: /hiiqos-59/?prefix=obj46 2023-07-29T00:35:29.420 DEBUG:boto:Data: 2023-07-29T00:35:29.420 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.421 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:29.421 DEBUG:boto:Port: 8000 2023-07-29T00:35:29.421 DEBUG:boto:Params: {} 2023-07-29T00:35:29.422 DEBUG:boto:Token: None 2023-07-29T00:35:29.422 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/ 2023-07-29T00:35:29.423 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O4RRBqFCnYe7CW88crqf26I4JfM= 2023-07-29T00:35:29.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O4RRBqFCnYe7CW88crqf26I4JfM=', 'Content-Length': '0'} 2023-07-29T00:35:29.427 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e03ee65602104f03-0064c45ed1-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.427 DEBUG:boto:b'hiiqos-59obj461000falseobj462023-07-29T00:33:06.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:29.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-07-29T00:35:29.429 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:35:29.429 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:35:29.429 DEBUG:boto:path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:35:29.430 DEBUG:boto:auth_path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:35:29.430 DEBUG:boto:Method: GET 2023-07-29T00:35:29.430 DEBUG:boto:Path: /hiiqos-59/obj51?versionId=null 2023-07-29T00:35:29.431 DEBUG:boto:Data: 2023-07-29T00:35:29.431 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.432 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:29.432 DEBUG:boto:Port: 8001 2023-07-29T00:35:29.432 DEBUG:boto:Params: {} 2023-07-29T00:35:29.433 DEBUG:boto:Token: None 2023-07-29T00:35:29.433 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj51?versionId=null 2023-07-29T00:35:29.434 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:adAPGabaRkVsWCPDKdl49T6BU4M= 2023-07-29T00:35:29.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:adAPGabaRkVsWCPDKdl49T6BU4M=', 'Content-Length': '0'} 2023-07-29T00:35:29.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc6a91b97f62da93-0064c45ed1-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.476 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:35:29.477 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:35:29.477 DEBUG:boto:path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:35:29.478 DEBUG:boto:auth_path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:35:29.478 DEBUG:boto:Method: GET 2023-07-29T00:35:29.478 DEBUG:boto:Path: /hiiqos-59/obj51?versionId=null 2023-07-29T00:35:29.479 DEBUG:boto:Data: 2023-07-29T00:35:29.479 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.479 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:29.480 DEBUG:boto:Port: 8000 2023-07-29T00:35:29.480 DEBUG:boto:Params: {} 2023-07-29T00:35:29.481 DEBUG:boto:Token: None 2023-07-29T00:35:29.481 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj51?versionId=null 2023-07-29T00:35:29.482 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:adAPGabaRkVsWCPDKdl49T6BU4M= 2023-07-29T00:35:29.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:adAPGabaRkVsWCPDKdl49T6BU4M=', 'Content-Length': '0'} 2023-07-29T00:35:29.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000931a79d111ff8bd8-0064c45ed1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.485 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:35:29.485 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:35:29.486 DEBUG:boto:path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:35:29.486 DEBUG:boto:auth_path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:35:29.486 DEBUG:boto:Method: HEAD 2023-07-29T00:35:29.486 DEBUG:boto:Path: /hiiqos-59/obj51?versionId=null 2023-07-29T00:35:29.487 DEBUG:boto:Data: 2023-07-29T00:35:29.487 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.488 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:29.488 DEBUG:boto:Port: 8001 2023-07-29T00:35:29.488 DEBUG:boto:Params: {} 2023-07-29T00:35:29.489 DEBUG:boto:Token: None 2023-07-29T00:35:29.489 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj51?versionId=null 2023-07-29T00:35:29.490 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fdxPAmSDVQ0acEKFd8MpM40XSW8= 2023-07-29T00:35:29.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fdxPAmSDVQ0acEKFd8MpM40XSW8=', 'Content-Length': '0'} 2023-07-29T00:35:29.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003071724f8e4a6e77-0064c45ed1-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.492 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:35:29.493 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:35:29.493 DEBUG:boto:path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:35:29.493 DEBUG:boto:auth_path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:35:29.494 DEBUG:boto:Method: HEAD 2023-07-29T00:35:29.494 DEBUG:boto:Path: /hiiqos-59/obj51?versionId=null 2023-07-29T00:35:29.494 DEBUG:boto:Data: 2023-07-29T00:35:29.495 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.495 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:29.495 DEBUG:boto:Port: 8000 2023-07-29T00:35:29.496 DEBUG:boto:Params: {} 2023-07-29T00:35:29.496 DEBUG:boto:Token: None 2023-07-29T00:35:29.496 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj51?versionId=null 2023-07-29T00:35:29.497 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fdxPAmSDVQ0acEKFd8MpM40XSW8= 2023-07-29T00:35:29.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fdxPAmSDVQ0acEKFd8MpM40XSW8=', 'Content-Length': '0'} 2023-07-29T00:35:29.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e256ccfbd81ffd3-0064c45ed1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-07-29T00:35:29.500 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:35:29.500 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:35:29.501 DEBUG:boto:path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:35:29.501 DEBUG:boto:auth_path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:35:29.501 DEBUG:boto:Method: GET 2023-07-29T00:35:29.502 DEBUG:boto:Path: /hiiqos-59/obj51?versionId=null 2023-07-29T00:35:29.502 DEBUG:boto:Data: 2023-07-29T00:35:29.502 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.503 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:29.503 DEBUG:boto:Port: 8001 2023-07-29T00:35:29.503 DEBUG:boto:Params: {} 2023-07-29T00:35:29.504 DEBUG:boto:Token: None 2023-07-29T00:35:29.504 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj51?versionId=null 2023-07-29T00:35:29.505 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:adAPGabaRkVsWCPDKdl49T6BU4M= 2023-07-29T00:35:29.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:adAPGabaRkVsWCPDKdl49T6BU4M=', 'Content-Length': '0'} 2023-07-29T00:35:29.507 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b0749082a7ae3be-0064c45ed1-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.508 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:35:29.508 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:35:29.508 DEBUG:boto:path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:35:29.509 DEBUG:boto:auth_path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:35:29.509 DEBUG:boto:Method: GET 2023-07-29T00:35:29.510 DEBUG:boto:Path: /hiiqos-59/obj51?versionId=null 2023-07-29T00:35:29.510 DEBUG:boto:Data: 2023-07-29T00:35:29.510 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.511 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:29.511 DEBUG:boto:Port: 8000 2023-07-29T00:35:29.511 DEBUG:boto:Params: {} 2023-07-29T00:35:29.512 DEBUG:boto:Token: None 2023-07-29T00:35:29.512 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj51?versionId=null 2023-07-29T00:35:29.513 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:adAPGabaRkVsWCPDKdl49T6BU4M= 2023-07-29T00:35:29.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:adAPGabaRkVsWCPDKdl49T6BU4M=', 'Content-Length': '0'} 2023-07-29T00:35:29.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030b46008e14912db-0064c45ed1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.516 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:29.516 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:29.517 DEBUG:boto:path=/hiiqos-59/?prefix=obj51 2023-07-29T00:35:29.517 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj51 2023-07-29T00:35:29.517 DEBUG:boto:Method: GET 2023-07-29T00:35:29.518 DEBUG:boto:Path: /hiiqos-59/?prefix=obj51 2023-07-29T00:35:29.518 DEBUG:boto:Data: 2023-07-29T00:35:29.518 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.519 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:29.519 DEBUG:boto:Port: 8001 2023-07-29T00:35:29.519 DEBUG:boto:Params: {} 2023-07-29T00:35:29.520 DEBUG:boto:Token: None 2023-07-29T00:35:29.520 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/ 2023-07-29T00:35:29.521 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O4RRBqFCnYe7CW88crqf26I4JfM= 2023-07-29T00:35:29.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O4RRBqFCnYe7CW88crqf26I4JfM=', 'Content-Length': '0'} 2023-07-29T00:35:29.524 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004da6fcd5e5a7ee1b-0064c45ed1-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.524 DEBUG:boto:b'hiiqos-59obj511000falseobj512023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:29.525 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:29.525 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:29.526 DEBUG:boto:path=/hiiqos-59/?prefix=obj51 2023-07-29T00:35:29.526 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj51 2023-07-29T00:35:29.526 DEBUG:boto:Method: GET 2023-07-29T00:35:29.527 DEBUG:boto:Path: /hiiqos-59/?prefix=obj51 2023-07-29T00:35:29.527 DEBUG:boto:Data: 2023-07-29T00:35:29.527 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.528 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:29.528 DEBUG:boto:Port: 8000 2023-07-29T00:35:29.529 DEBUG:boto:Params: {} 2023-07-29T00:35:29.529 DEBUG:boto:Token: None 2023-07-29T00:35:29.529 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/ 2023-07-29T00:35:29.530 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O4RRBqFCnYe7CW88crqf26I4JfM= 2023-07-29T00:35:29.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O4RRBqFCnYe7CW88crqf26I4JfM=', 'Content-Length': '0'} 2023-07-29T00:35:29.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009851d378e6e073c2-0064c45ed1-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.534 DEBUG:boto:b'hiiqos-59obj511000falseobj512023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:29.535 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:35:29.535 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:35:29.536 DEBUG:boto:Method: HEAD 2023-07-29T00:35:29.536 DEBUG:boto:Path: /hiiqos-59/obj51 2023-07-29T00:35:29.536 DEBUG:boto:Data: 2023-07-29T00:35:29.537 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.537 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:29.537 DEBUG:boto:Port: 8001 2023-07-29T00:35:29.538 DEBUG:boto:Params: {} 2023-07-29T00:35:29.538 DEBUG:boto:Token: None 2023-07-29T00:35:29.538 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj51 2023-07-29T00:35:29.539 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JRPRtDAkt6GxVdMtbqMLo3f3h78= 2023-07-29T00:35:29.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JRPRtDAkt6GxVdMtbqMLo3f3h78=', 'Content-Length': '0'} 2023-07-29T00:35:29.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056db5136d5f5fe96-0064c45ed1-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.542 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:35:29.542 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:35:29.542 DEBUG:boto:Method: HEAD 2023-07-29T00:35:29.543 DEBUG:boto:Path: /hiiqos-59/obj51 2023-07-29T00:35:29.543 DEBUG:boto:Data: 2023-07-29T00:35:29.543 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.544 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:29.544 DEBUG:boto:Port: 8000 2023-07-29T00:35:29.544 DEBUG:boto:Params: {} 2023-07-29T00:35:29.545 DEBUG:boto:Token: None 2023-07-29T00:35:29.545 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj51 2023-07-29T00:35:29.546 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JRPRtDAkt6GxVdMtbqMLo3f3h78= 2023-07-29T00:35:29.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JRPRtDAkt6GxVdMtbqMLo3f3h78=', 'Content-Length': '0'} 2023-07-29T00:35:29.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d761989c3351a027-0064c45ed1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-07-29T00:35:29.549 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:35:29.549 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:35:29.549 DEBUG:boto:Method: GET 2023-07-29T00:35:29.550 DEBUG:boto:Path: /hiiqos-59/obj51 2023-07-29T00:35:29.550 DEBUG:boto:Data: 2023-07-29T00:35:29.550 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.551 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:29.551 DEBUG:boto:Port: 8001 2023-07-29T00:35:29.552 DEBUG:boto:Params: {} 2023-07-29T00:35:29.552 DEBUG:boto:Token: None 2023-07-29T00:35:29.552 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj51 2023-07-29T00:35:29.553 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/+12eejqgclw9HsA48JwloZbm/s= 2023-07-29T00:35:29.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/+12eejqgclw9HsA48JwloZbm/s=', 'Content-Length': '0'} 2023-07-29T00:35:29.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071e0562c36f329c4-0064c45ed1-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.556 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:35:29.556 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:35:29.556 DEBUG:boto:Method: GET 2023-07-29T00:35:29.557 DEBUG:boto:Path: /hiiqos-59/obj51 2023-07-29T00:35:29.557 DEBUG:boto:Data: 2023-07-29T00:35:29.558 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.558 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:29.558 DEBUG:boto:Port: 8000 2023-07-29T00:35:29.559 DEBUG:boto:Params: {} 2023-07-29T00:35:29.559 DEBUG:boto:Token: None 2023-07-29T00:35:29.559 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj51 2023-07-29T00:35:29.560 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/+12eejqgclw9HsA48JwloZbm/s= 2023-07-29T00:35:29.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/+12eejqgclw9HsA48JwloZbm/s=', 'Content-Length': '0'} 2023-07-29T00:35:29.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000727c54a1b7411a81-0064c45ed1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.604 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:29.605 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:29.605 DEBUG:boto:path=/hiiqos-59/?prefix=obj51 2023-07-29T00:35:29.605 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj51 2023-07-29T00:35:29.606 DEBUG:boto:Method: GET 2023-07-29T00:35:29.606 DEBUG:boto:Path: /hiiqos-59/?prefix=obj51 2023-07-29T00:35:29.606 DEBUG:boto:Data: 2023-07-29T00:35:29.607 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.607 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:29.607 DEBUG:boto:Port: 8001 2023-07-29T00:35:29.608 DEBUG:boto:Params: {} 2023-07-29T00:35:29.608 DEBUG:boto:Token: None 2023-07-29T00:35:29.609 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/ 2023-07-29T00:35:29.609 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O4RRBqFCnYe7CW88crqf26I4JfM= 2023-07-29T00:35:29.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O4RRBqFCnYe7CW88crqf26I4JfM=', 'Content-Length': '0'} 2023-07-29T00:35:29.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b38b7b73c680d0e3-0064c45ed1-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.612 DEBUG:boto:b'hiiqos-59obj511000falseobj512023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:29.613 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:29.613 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:29.614 DEBUG:boto:path=/hiiqos-59/?prefix=obj51 2023-07-29T00:35:29.614 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj51 2023-07-29T00:35:29.614 DEBUG:boto:Method: GET 2023-07-29T00:35:29.615 DEBUG:boto:Path: /hiiqos-59/?prefix=obj51 2023-07-29T00:35:29.615 DEBUG:boto:Data: 2023-07-29T00:35:29.615 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.616 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:29.616 DEBUG:boto:Port: 8000 2023-07-29T00:35:29.616 DEBUG:boto:Params: {} 2023-07-29T00:35:29.617 DEBUG:boto:Token: None 2023-07-29T00:35:29.617 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/ 2023-07-29T00:35:29.618 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O4RRBqFCnYe7CW88crqf26I4JfM= 2023-07-29T00:35:29.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O4RRBqFCnYe7CW88crqf26I4JfM=', 'Content-Length': '0'} 2023-07-29T00:35:29.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a643d9dac81a8064-0064c45ed1-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.622 DEBUG:boto:b'hiiqos-59obj511000falseobj512023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:29.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-07-29T00:35:29.623 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:35:29.624 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:35:29.624 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:35:29.625 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:35:29.625 DEBUG:boto:Method: GET 2023-07-29T00:35:29.625 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:35:29.626 DEBUG:boto:Data: 2023-07-29T00:35:29.626 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.626 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:29.627 DEBUG:boto:Port: 8001 2023-07-29T00:35:29.627 DEBUG:boto:Params: {} 2023-07-29T00:35:29.627 DEBUG:boto:Token: None 2023-07-29T00:35:29.628 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:35:29.628 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qtE3dsWvCk67Quh8Dqko7bhHAok= 2023-07-29T00:35:29.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qtE3dsWvCk67Quh8Dqko7bhHAok=', 'Content-Length': '0'} 2023-07-29T00:35:29.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000709b24a6f09bc905-0064c45ed1-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.631 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:35:29.632 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:35:29.632 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:35:29.632 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:35:29.633 DEBUG:boto:Method: GET 2023-07-29T00:35:29.633 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:35:29.633 DEBUG:boto:Data: 2023-07-29T00:35:29.634 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.634 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:29.634 DEBUG:boto:Port: 8000 2023-07-29T00:35:29.635 DEBUG:boto:Params: {} 2023-07-29T00:35:29.635 DEBUG:boto:Token: None 2023-07-29T00:35:29.636 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:35:29.636 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qtE3dsWvCk67Quh8Dqko7bhHAok= 2023-07-29T00:35:29.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qtE3dsWvCk67Quh8Dqko7bhHAok=', 'Content-Length': '0'} 2023-07-29T00:35:29.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9c4d9cc2e5e0a7a-0064c45ed1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.639 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:35:29.639 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:35:29.640 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:35:29.640 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:35:29.640 DEBUG:boto:Method: HEAD 2023-07-29T00:35:29.641 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:35:29.641 DEBUG:boto:Data: 2023-07-29T00:35:29.641 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.642 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:29.642 DEBUG:boto:Port: 8001 2023-07-29T00:35:29.642 DEBUG:boto:Params: {} 2023-07-29T00:35:29.643 DEBUG:boto:Token: None 2023-07-29T00:35:29.643 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:35:29.643 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ruW5qqyO0eRq9y5qrNhjorWPoi8= 2023-07-29T00:35:29.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ruW5qqyO0eRq9y5qrNhjorWPoi8=', 'Content-Length': '0'} 2023-07-29T00:35:29.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1850fab277e6fd4-0064c45ed1-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.646 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:35:29.647 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:35:29.647 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:35:29.647 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:35:29.648 DEBUG:boto:Method: HEAD 2023-07-29T00:35:29.648 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:35:29.648 DEBUG:boto:Data: 2023-07-29T00:35:29.649 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.649 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:29.649 DEBUG:boto:Port: 8000 2023-07-29T00:35:29.650 DEBUG:boto:Params: {} 2023-07-29T00:35:29.650 DEBUG:boto:Token: None 2023-07-29T00:35:29.650 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:35:29.651 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ruW5qqyO0eRq9y5qrNhjorWPoi8= 2023-07-29T00:35:29.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ruW5qqyO0eRq9y5qrNhjorWPoi8=', 'Content-Length': '0'} 2023-07-29T00:35:29.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e573571f58efc973-0064c45ed1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-07-29T00:35:29.654 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:35:29.654 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:35:29.655 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:35:29.655 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:35:29.655 DEBUG:boto:Method: GET 2023-07-29T00:35:29.656 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:35:29.656 DEBUG:boto:Data: 2023-07-29T00:35:29.656 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.657 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:29.657 DEBUG:boto:Port: 8001 2023-07-29T00:35:29.657 DEBUG:boto:Params: {} 2023-07-29T00:35:29.658 DEBUG:boto:Token: None 2023-07-29T00:35:29.658 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:35:29.659 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qtE3dsWvCk67Quh8Dqko7bhHAok= 2023-07-29T00:35:29.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qtE3dsWvCk67Quh8Dqko7bhHAok=', 'Content-Length': '0'} 2023-07-29T00:35:29.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b680fddc4333ffa1-0064c45ed1-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.661 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:35:29.662 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:35:29.662 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:35:29.663 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:35:29.663 DEBUG:boto:Method: GET 2023-07-29T00:35:29.663 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:35:29.664 DEBUG:boto:Data: 2023-07-29T00:35:29.664 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.664 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:29.665 DEBUG:boto:Port: 8000 2023-07-29T00:35:29.665 DEBUG:boto:Params: {} 2023-07-29T00:35:29.665 DEBUG:boto:Token: None 2023-07-29T00:35:29.666 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:35:29.666 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qtE3dsWvCk67Quh8Dqko7bhHAok= 2023-07-29T00:35:29.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qtE3dsWvCk67Quh8Dqko7bhHAok=', 'Content-Length': '0'} 2023-07-29T00:35:29.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e8e99a02dd598d4-0064c45ed1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.716 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:29.717 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:29.717 DEBUG:boto:path=/hiiqos-59/?prefix=obj55 2023-07-29T00:35:29.718 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj55 2023-07-29T00:35:29.718 DEBUG:boto:Method: GET 2023-07-29T00:35:29.718 DEBUG:boto:Path: /hiiqos-59/?prefix=obj55 2023-07-29T00:35:29.719 DEBUG:boto:Data: 2023-07-29T00:35:29.719 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.719 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:29.720 DEBUG:boto:Port: 8001 2023-07-29T00:35:29.720 DEBUG:boto:Params: {} 2023-07-29T00:35:29.721 DEBUG:boto:Token: None 2023-07-29T00:35:29.721 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/ 2023-07-29T00:35:29.721 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O4RRBqFCnYe7CW88crqf26I4JfM= 2023-07-29T00:35:29.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O4RRBqFCnYe7CW88crqf26I4JfM=', 'Content-Length': '0'} 2023-07-29T00:35:29.724 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089a9770282e506f8-0064c45ed1-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.725 DEBUG:boto:b'hiiqos-59obj551000falseobj552023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:29.726 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:29.726 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:29.726 DEBUG:boto:path=/hiiqos-59/?prefix=obj55 2023-07-29T00:35:29.727 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj55 2023-07-29T00:35:29.727 DEBUG:boto:Method: GET 2023-07-29T00:35:29.727 DEBUG:boto:Path: /hiiqos-59/?prefix=obj55 2023-07-29T00:35:29.728 DEBUG:boto:Data: 2023-07-29T00:35:29.728 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.729 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:29.729 DEBUG:boto:Port: 8000 2023-07-29T00:35:29.729 DEBUG:boto:Params: {} 2023-07-29T00:35:29.730 DEBUG:boto:Token: None 2023-07-29T00:35:29.731 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/ 2023-07-29T00:35:29.731 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O4RRBqFCnYe7CW88crqf26I4JfM= 2023-07-29T00:35:29.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O4RRBqFCnYe7CW88crqf26I4JfM=', 'Content-Length': '0'} 2023-07-29T00:35:29.736 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e26bd351a0518471-0064c45ed1-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.737 DEBUG:boto:b'hiiqos-59obj551000falseobj552023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:29.737 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:35:29.738 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:35:29.738 DEBUG:boto:Method: HEAD 2023-07-29T00:35:29.738 DEBUG:boto:Path: /hiiqos-59/obj55 2023-07-29T00:35:29.739 DEBUG:boto:Data: 2023-07-29T00:35:29.739 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.739 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:29.740 DEBUG:boto:Port: 8001 2023-07-29T00:35:29.740 DEBUG:boto:Params: {} 2023-07-29T00:35:29.740 DEBUG:boto:Token: None 2023-07-29T00:35:29.741 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj55 2023-07-29T00:35:29.741 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bQosPvU571xQoDjgttHUM3QrBSk= 2023-07-29T00:35:29.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bQosPvU571xQoDjgttHUM3QrBSk=', 'Content-Length': '0'} 2023-07-29T00:35:29.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb8b09ef7c49cf00-0064c45ed1-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.744 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:35:29.744 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:35:29.745 DEBUG:boto:Method: HEAD 2023-07-29T00:35:29.745 DEBUG:boto:Path: /hiiqos-59/obj55 2023-07-29T00:35:29.746 DEBUG:boto:Data: 2023-07-29T00:35:29.746 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.746 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:29.747 DEBUG:boto:Port: 8000 2023-07-29T00:35:29.747 DEBUG:boto:Params: {} 2023-07-29T00:35:29.747 DEBUG:boto:Token: None 2023-07-29T00:35:29.748 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj55 2023-07-29T00:35:29.748 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bQosPvU571xQoDjgttHUM3QrBSk= 2023-07-29T00:35:29.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bQosPvU571xQoDjgttHUM3QrBSk=', 'Content-Length': '0'} 2023-07-29T00:35:29.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7d1f5d2137e4e71-0064c45ed1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-07-29T00:35:29.751 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:35:29.752 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:35:29.752 DEBUG:boto:Method: GET 2023-07-29T00:35:29.752 DEBUG:boto:Path: /hiiqos-59/obj55 2023-07-29T00:35:29.753 DEBUG:boto:Data: 2023-07-29T00:35:29.753 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.754 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:29.754 DEBUG:boto:Port: 8001 2023-07-29T00:35:29.754 DEBUG:boto:Params: {} 2023-07-29T00:35:29.755 DEBUG:boto:Token: None 2023-07-29T00:35:29.755 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj55 2023-07-29T00:35:29.756 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:r1fbtZ83EaLVmoUJm9Hha4xIBqA= 2023-07-29T00:35:29.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:r1fbtZ83EaLVmoUJm9Hha4xIBqA=', 'Content-Length': '0'} 2023-07-29T00:35:29.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b318bde9ee586889-0064c45ed1-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.800 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:35:29.801 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:35:29.801 DEBUG:boto:Method: GET 2023-07-29T00:35:29.802 DEBUG:boto:Path: /hiiqos-59/obj55 2023-07-29T00:35:29.802 DEBUG:boto:Data: 2023-07-29T00:35:29.802 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.803 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:29.803 DEBUG:boto:Port: 8000 2023-07-29T00:35:29.803 DEBUG:boto:Params: {} 2023-07-29T00:35:29.804 DEBUG:boto:Token: None 2023-07-29T00:35:29.804 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj55 2023-07-29T00:35:29.805 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:r1fbtZ83EaLVmoUJm9Hha4xIBqA= 2023-07-29T00:35:29.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:r1fbtZ83EaLVmoUJm9Hha4xIBqA=', 'Content-Length': '0'} 2023-07-29T00:35:29.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1f2a25cd93bedb9-0064c45ed1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.808 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:29.808 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:29.809 DEBUG:boto:path=/hiiqos-59/?prefix=obj55 2023-07-29T00:35:29.809 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj55 2023-07-29T00:35:29.809 DEBUG:boto:Method: GET 2023-07-29T00:35:29.810 DEBUG:boto:Path: /hiiqos-59/?prefix=obj55 2023-07-29T00:35:29.810 DEBUG:boto:Data: 2023-07-29T00:35:29.810 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.811 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:29.811 DEBUG:boto:Port: 8001 2023-07-29T00:35:29.811 DEBUG:boto:Params: {} 2023-07-29T00:35:29.812 DEBUG:boto:Token: None 2023-07-29T00:35:29.812 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/ 2023-07-29T00:35:29.813 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O4RRBqFCnYe7CW88crqf26I4JfM= 2023-07-29T00:35:29.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O4RRBqFCnYe7CW88crqf26I4JfM=', 'Content-Length': '0'} 2023-07-29T00:35:29.816 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b85da88e38566efe-0064c45ed1-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.817 DEBUG:boto:b'hiiqos-59obj551000falseobj552023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:29.819 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:29.820 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:29.820 DEBUG:boto:path=/hiiqos-59/?prefix=obj55 2023-07-29T00:35:29.821 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj55 2023-07-29T00:35:29.822 DEBUG:boto:Method: GET 2023-07-29T00:35:29.822 DEBUG:boto:Path: /hiiqos-59/?prefix=obj55 2023-07-29T00:35:29.823 DEBUG:boto:Data: 2023-07-29T00:35:29.824 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.824 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:29.825 DEBUG:boto:Port: 8000 2023-07-29T00:35:29.825 DEBUG:boto:Params: {} 2023-07-29T00:35:29.826 DEBUG:boto:Token: None 2023-07-29T00:35:29.826 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/ 2023-07-29T00:35:29.826 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O4RRBqFCnYe7CW88crqf26I4JfM= 2023-07-29T00:35:29.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O4RRBqFCnYe7CW88crqf26I4JfM=', 'Content-Length': '0'} 2023-07-29T00:35:29.831 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a056c99fc2459842-0064c45ed1-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.831 DEBUG:boto:b'hiiqos-59obj551000falseobj552023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:29.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-07-29T00:35:29.832 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:35:29.833 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:35:29.833 DEBUG:boto:path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:35:29.833 DEBUG:boto:auth_path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:35:29.834 DEBUG:boto:Method: GET 2023-07-29T00:35:29.834 DEBUG:boto:Path: /hiiqos-59/obj57?versionId=null 2023-07-29T00:35:29.835 DEBUG:boto:Data: 2023-07-29T00:35:29.835 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.835 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:29.836 DEBUG:boto:Port: 8001 2023-07-29T00:35:29.836 DEBUG:boto:Params: {} 2023-07-29T00:35:29.837 DEBUG:boto:Token: None 2023-07-29T00:35:29.837 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj57?versionId=null 2023-07-29T00:35:29.837 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oRhswOOmql+4KlehT6KlFp4130o= 2023-07-29T00:35:29.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oRhswOOmql+4KlehT6KlFp4130o=', 'Content-Length': '0'} 2023-07-29T00:35:29.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f276688badbd65a9-0064c45ed1-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.840 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:35:29.841 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:35:29.841 DEBUG:boto:path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:35:29.842 DEBUG:boto:auth_path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:35:29.843 DEBUG:boto:Method: GET 2023-07-29T00:35:29.843 DEBUG:boto:Path: /hiiqos-59/obj57?versionId=null 2023-07-29T00:35:29.844 DEBUG:boto:Data: 2023-07-29T00:35:29.845 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.845 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:29.846 DEBUG:boto:Port: 8000 2023-07-29T00:35:29.846 DEBUG:boto:Params: {} 2023-07-29T00:35:29.847 DEBUG:boto:Token: None 2023-07-29T00:35:29.848 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj57?versionId=null 2023-07-29T00:35:29.848 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oRhswOOmql+4KlehT6KlFp4130o= 2023-07-29T00:35:29.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oRhswOOmql+4KlehT6KlFp4130o=', 'Content-Length': '0'} 2023-07-29T00:35:29.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006567e9047c48590f-0064c45ed1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.892 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:35:29.893 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:35:29.893 DEBUG:boto:path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:35:29.894 DEBUG:boto:auth_path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:35:29.894 DEBUG:boto:Method: HEAD 2023-07-29T00:35:29.894 DEBUG:boto:Path: /hiiqos-59/obj57?versionId=null 2023-07-29T00:35:29.895 DEBUG:boto:Data: 2023-07-29T00:35:29.895 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.895 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:29.896 DEBUG:boto:Port: 8001 2023-07-29T00:35:29.896 DEBUG:boto:Params: {} 2023-07-29T00:35:29.896 DEBUG:boto:Token: None 2023-07-29T00:35:29.897 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj57?versionId=null 2023-07-29T00:35:29.897 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mTaAeahChMdsc+WyB16t+wXY21k= 2023-07-29T00:35:29.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mTaAeahChMdsc+WyB16t+wXY21k=', 'Content-Length': '0'} 2023-07-29T00:35:29.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe03d85764d5267b-0064c45ed1-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.900 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:35:29.900 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:35:29.901 DEBUG:boto:path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:35:29.901 DEBUG:boto:auth_path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:35:29.901 DEBUG:boto:Method: HEAD 2023-07-29T00:35:29.902 DEBUG:boto:Path: /hiiqos-59/obj57?versionId=null 2023-07-29T00:35:29.902 DEBUG:boto:Data: 2023-07-29T00:35:29.902 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.903 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:29.903 DEBUG:boto:Port: 8000 2023-07-29T00:35:29.903 DEBUG:boto:Params: {} 2023-07-29T00:35:29.904 DEBUG:boto:Token: None 2023-07-29T00:35:29.904 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj57?versionId=null 2023-07-29T00:35:29.905 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mTaAeahChMdsc+WyB16t+wXY21k= 2023-07-29T00:35:29.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mTaAeahChMdsc+WyB16t+wXY21k=', 'Content-Length': '0'} 2023-07-29T00:35:29.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c093b7d441c7f54d-0064c45ed1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-07-29T00:35:29.908 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:35:29.908 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:35:29.909 DEBUG:boto:path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:35:29.909 DEBUG:boto:auth_path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:35:29.909 DEBUG:boto:Method: GET 2023-07-29T00:35:29.910 DEBUG:boto:Path: /hiiqos-59/obj57?versionId=null 2023-07-29T00:35:29.910 DEBUG:boto:Data: 2023-07-29T00:35:29.911 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.911 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:29.911 DEBUG:boto:Port: 8001 2023-07-29T00:35:29.912 DEBUG:boto:Params: {} 2023-07-29T00:35:29.912 DEBUG:boto:Token: None 2023-07-29T00:35:29.912 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj57?versionId=null 2023-07-29T00:35:29.913 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oRhswOOmql+4KlehT6KlFp4130o= 2023-07-29T00:35:29.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oRhswOOmql+4KlehT6KlFp4130o=', 'Content-Length': '0'} 2023-07-29T00:35:29.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d97854176fd3c4d9-0064c45ed1-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.916 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:35:29.916 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:35:29.917 DEBUG:boto:path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:35:29.917 DEBUG:boto:auth_path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:35:29.917 DEBUG:boto:Method: GET 2023-07-29T00:35:29.918 DEBUG:boto:Path: /hiiqos-59/obj57?versionId=null 2023-07-29T00:35:29.918 DEBUG:boto:Data: 2023-07-29T00:35:29.918 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.919 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:29.919 DEBUG:boto:Port: 8000 2023-07-29T00:35:29.919 DEBUG:boto:Params: {} 2023-07-29T00:35:29.920 DEBUG:boto:Token: None 2023-07-29T00:35:29.920 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj57?versionId=null 2023-07-29T00:35:29.921 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oRhswOOmql+4KlehT6KlFp4130o= 2023-07-29T00:35:29.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oRhswOOmql+4KlehT6KlFp4130o=', 'Content-Length': '0'} 2023-07-29T00:35:29.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084799c2906127247-0064c45ed1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.924 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:29.924 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:29.924 DEBUG:boto:path=/hiiqos-59/?prefix=obj57 2023-07-29T00:35:29.925 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj57 2023-07-29T00:35:29.925 DEBUG:boto:Method: GET 2023-07-29T00:35:29.925 DEBUG:boto:Path: /hiiqos-59/?prefix=obj57 2023-07-29T00:35:29.926 DEBUG:boto:Data: 2023-07-29T00:35:29.926 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.927 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:29.927 DEBUG:boto:Port: 8001 2023-07-29T00:35:29.927 DEBUG:boto:Params: {} 2023-07-29T00:35:29.928 DEBUG:boto:Token: None 2023-07-29T00:35:29.928 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/ 2023-07-29T00:35:29.929 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O4RRBqFCnYe7CW88crqf26I4JfM= 2023-07-29T00:35:29.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O4RRBqFCnYe7CW88crqf26I4JfM=', 'Content-Length': '0'} 2023-07-29T00:35:29.931 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8cfe0c4cc1f994c-0064c45ed1-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.932 DEBUG:boto:b'hiiqos-59obj571000falseobj572023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:29.933 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:29.934 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:29.935 DEBUG:boto:path=/hiiqos-59/?prefix=obj57 2023-07-29T00:35:29.935 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj57 2023-07-29T00:35:29.936 DEBUG:boto:Method: GET 2023-07-29T00:35:29.937 DEBUG:boto:Path: /hiiqos-59/?prefix=obj57 2023-07-29T00:35:29.937 DEBUG:boto:Data: 2023-07-29T00:35:29.938 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.939 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:29.939 DEBUG:boto:Port: 8000 2023-07-29T00:35:29.940 DEBUG:boto:Params: {} 2023-07-29T00:35:29.941 DEBUG:boto:Token: None 2023-07-29T00:35:29.941 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/ 2023-07-29T00:35:29.942 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O4RRBqFCnYe7CW88crqf26I4JfM= 2023-07-29T00:35:29.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O4RRBqFCnYe7CW88crqf26I4JfM=', 'Content-Length': '0'} 2023-07-29T00:35:29.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bf58db0fc7e4f8e-0064c45ed1-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.948 DEBUG:boto:b'hiiqos-59obj571000falseobj572023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:29.949 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:35:29.950 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:35:29.950 DEBUG:boto:Method: HEAD 2023-07-29T00:35:29.951 DEBUG:boto:Path: /hiiqos-59/obj57 2023-07-29T00:35:29.952 DEBUG:boto:Data: 2023-07-29T00:35:29.952 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.953 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:29.953 DEBUG:boto:Port: 8001 2023-07-29T00:35:29.954 DEBUG:boto:Params: {} 2023-07-29T00:35:29.955 DEBUG:boto:Token: None 2023-07-29T00:35:29.955 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj57 2023-07-29T00:35:29.956 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Sw9mpIVNbicrRkp8K5OJ70M9ueA= 2023-07-29T00:35:29.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Sw9mpIVNbicrRkp8K5OJ70M9ueA=', 'Content-Length': '0'} 2023-07-29T00:35:29.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e018fee591af675-0064c45ed1-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.960 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:35:29.961 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:35:29.961 DEBUG:boto:Method: HEAD 2023-07-29T00:35:29.962 DEBUG:boto:Path: /hiiqos-59/obj57 2023-07-29T00:35:29.963 DEBUG:boto:Data: 2023-07-29T00:35:29.963 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.964 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:29.964 DEBUG:boto:Port: 8000 2023-07-29T00:35:29.965 DEBUG:boto:Params: {} 2023-07-29T00:35:29.966 DEBUG:boto:Token: None 2023-07-29T00:35:29.967 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj57 2023-07-29T00:35:29.967 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Sw9mpIVNbicrRkp8K5OJ70M9ueA= 2023-07-29T00:35:29.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Sw9mpIVNbicrRkp8K5OJ70M9ueA=', 'Content-Length': '0'} 2023-07-29T00:35:29.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b69f67225bdd3ffc-0064c45ed1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-07-29T00:35:29.972 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:35:29.973 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:35:29.973 DEBUG:boto:Method: GET 2023-07-29T00:35:29.974 DEBUG:boto:Path: /hiiqos-59/obj57 2023-07-29T00:35:29.975 DEBUG:boto:Data: 2023-07-29T00:35:29.975 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.976 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:29.977 DEBUG:boto:Port: 8001 2023-07-29T00:35:29.977 DEBUG:boto:Params: {} 2023-07-29T00:35:29.978 DEBUG:boto:Token: None 2023-07-29T00:35:29.979 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj57 2023-07-29T00:35:29.980 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qC/K4wO+5XYH2XOS24TWN/vWtTM= 2023-07-29T00:35:29.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qC/K4wO+5XYH2XOS24TWN/vWtTM=', 'Content-Length': '0'} 2023-07-29T00:35:29.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000045e6a25939d2c1-0064c45ed1-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.983 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:35:29.984 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:35:29.985 DEBUG:boto:Method: GET 2023-07-29T00:35:29.985 DEBUG:boto:Path: /hiiqos-59/obj57 2023-07-29T00:35:29.985 DEBUG:boto:Data: 2023-07-29T00:35:29.986 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.986 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:29.986 DEBUG:boto:Port: 8000 2023-07-29T00:35:29.987 DEBUG:boto:Params: {} 2023-07-29T00:35:29.987 DEBUG:boto:Token: None 2023-07-29T00:35:29.987 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/obj57 2023-07-29T00:35:29.988 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qC/K4wO+5XYH2XOS24TWN/vWtTM= 2023-07-29T00:35:29.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qC/K4wO+5XYH2XOS24TWN/vWtTM=', 'Content-Length': '0'} 2023-07-29T00:35:29.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000917497e6b5713cde-0064c45ed1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:29.991 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:29.991 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:29.992 DEBUG:boto:path=/hiiqos-59/?prefix=obj57 2023-07-29T00:35:29.992 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj57 2023-07-29T00:35:29.992 DEBUG:boto:Method: GET 2023-07-29T00:35:29.993 DEBUG:boto:Path: /hiiqos-59/?prefix=obj57 2023-07-29T00:35:29.993 DEBUG:boto:Data: 2023-07-29T00:35:29.993 DEBUG:boto:Headers: {} 2023-07-29T00:35:29.994 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:29.994 DEBUG:boto:Port: 8001 2023-07-29T00:35:29.995 DEBUG:boto:Params: {} 2023-07-29T00:35:29.996 DEBUG:boto:Token: None 2023-07-29T00:35:29.997 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:29 GMT /hiiqos-59/ 2023-07-29T00:35:29.998 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O4RRBqFCnYe7CW88crqf26I4JfM= 2023-07-29T00:35:29.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:29 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O4RRBqFCnYe7CW88crqf26I4JfM=', 'Content-Length': '0'} 2023-07-29T00:35:30.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000591b28d827c1fa14-0064c45ed1-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.003 DEBUG:boto:b'hiiqos-59obj571000falseobj572023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:30.005 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:30.005 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:30.005 DEBUG:boto:path=/hiiqos-59/?prefix=obj57 2023-07-29T00:35:30.006 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj57 2023-07-29T00:35:30.006 DEBUG:boto:Method: GET 2023-07-29T00:35:30.006 DEBUG:boto:Path: /hiiqos-59/?prefix=obj57 2023-07-29T00:35:30.007 DEBUG:boto:Data: 2023-07-29T00:35:30.007 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.007 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:30.008 DEBUG:boto:Port: 8000 2023-07-29T00:35:30.008 DEBUG:boto:Params: {} 2023-07-29T00:35:30.009 DEBUG:boto:Token: None 2023-07-29T00:35:30.009 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/ 2023-07-29T00:35:30.010 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qm+SsO8XwO9SXjVIZFoLoF0u/VU= 2023-07-29T00:35:30.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qm+SsO8XwO9SXjVIZFoLoF0u/VU=', 'Content-Length': '0'} 2023-07-29T00:35:30.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b0ff104faf197da-0064c45ed1-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.014 DEBUG:boto:b'hiiqos-59obj571000falseobj572023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:30.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-07-29T00:35:30.015 DEBUG:boto:path=/hiiqos-59/obj58 2023-07-29T00:35:30.016 DEBUG:boto:auth_path=/hiiqos-59/obj58 2023-07-29T00:35:30.016 DEBUG:boto:path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:35:30.017 DEBUG:boto:auth_path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:35:30.017 DEBUG:boto:Method: GET 2023-07-29T00:35:30.017 DEBUG:boto:Path: /hiiqos-59/obj58?versionId=null 2023-07-29T00:35:30.018 DEBUG:boto:Data: 2023-07-29T00:35:30.018 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.018 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:30.019 DEBUG:boto:Port: 8001 2023-07-29T00:35:30.019 DEBUG:boto:Params: {} 2023-07-29T00:35:30.019 DEBUG:boto:Token: None 2023-07-29T00:35:30.020 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj58?versionId=null 2023-07-29T00:35:30.020 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VVbN5vkmBprQpTjYxhaJ/RsSAj4= 2023-07-29T00:35:30.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VVbN5vkmBprQpTjYxhaJ/RsSAj4=', 'Content-Length': '0'} 2023-07-29T00:35:30.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000faf1aa8734227535-0064c45ed1-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.023 DEBUG:boto:path=/hiiqos-59/obj58 2023-07-29T00:35:30.024 DEBUG:boto:auth_path=/hiiqos-59/obj58 2023-07-29T00:35:30.024 DEBUG:boto:path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:35:30.025 DEBUG:boto:auth_path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:35:30.025 DEBUG:boto:Method: GET 2023-07-29T00:35:30.025 DEBUG:boto:Path: /hiiqos-59/obj58?versionId=null 2023-07-29T00:35:30.026 DEBUG:boto:Data: 2023-07-29T00:35:30.026 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.026 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:30.027 DEBUG:boto:Port: 8000 2023-07-29T00:35:30.027 DEBUG:boto:Params: {} 2023-07-29T00:35:30.028 DEBUG:boto:Token: None 2023-07-29T00:35:30.028 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj58?versionId=null 2023-07-29T00:35:30.028 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VVbN5vkmBprQpTjYxhaJ/RsSAj4= 2023-07-29T00:35:30.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VVbN5vkmBprQpTjYxhaJ/RsSAj4=', 'Content-Length': '0'} 2023-07-29T00:35:30.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000090fe7efc248e4ac-0064c45ed1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:29 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.072 DEBUG:boto:path=/hiiqos-59/obj58 2023-07-29T00:35:30.073 DEBUG:boto:auth_path=/hiiqos-59/obj58 2023-07-29T00:35:30.073 DEBUG:boto:path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:35:30.073 DEBUG:boto:auth_path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:35:30.074 DEBUG:boto:Method: HEAD 2023-07-29T00:35:30.074 DEBUG:boto:Path: /hiiqos-59/obj58?versionId=null 2023-07-29T00:35:30.074 DEBUG:boto:Data: 2023-07-29T00:35:30.075 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.075 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:30.075 DEBUG:boto:Port: 8001 2023-07-29T00:35:30.076 DEBUG:boto:Params: {} 2023-07-29T00:35:30.076 DEBUG:boto:Token: None 2023-07-29T00:35:30.077 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj58?versionId=null 2023-07-29T00:35:30.077 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UR4XNprrNA5uvwDmYi1c++nyiP8= 2023-07-29T00:35:30.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UR4XNprrNA5uvwDmYi1c++nyiP8=', 'Content-Length': '0'} 2023-07-29T00:35:30.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006353ed39a8a8ad3f-0064c45ed2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.080 DEBUG:boto:path=/hiiqos-59/obj58 2023-07-29T00:35:30.080 DEBUG:boto:auth_path=/hiiqos-59/obj58 2023-07-29T00:35:30.081 DEBUG:boto:path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:35:30.081 DEBUG:boto:auth_path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:35:30.081 DEBUG:boto:Method: HEAD 2023-07-29T00:35:30.082 DEBUG:boto:Path: /hiiqos-59/obj58?versionId=null 2023-07-29T00:35:30.082 DEBUG:boto:Data: 2023-07-29T00:35:30.083 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.083 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:30.083 DEBUG:boto:Port: 8000 2023-07-29T00:35:30.084 DEBUG:boto:Params: {} 2023-07-29T00:35:30.084 DEBUG:boto:Token: None 2023-07-29T00:35:30.085 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj58?versionId=null 2023-07-29T00:35:30.085 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UR4XNprrNA5uvwDmYi1c++nyiP8= 2023-07-29T00:35:30.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UR4XNprrNA5uvwDmYi1c++nyiP8=', 'Content-Length': '0'} 2023-07-29T00:35:30.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba59d54cd620db98-0064c45ed2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-07-29T00:35:30.088 DEBUG:boto:path=/hiiqos-59/obj58 2023-07-29T00:35:30.088 DEBUG:boto:auth_path=/hiiqos-59/obj58 2023-07-29T00:35:30.089 DEBUG:boto:path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:35:30.089 DEBUG:boto:auth_path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:35:30.089 DEBUG:boto:Method: GET 2023-07-29T00:35:30.090 DEBUG:boto:Path: /hiiqos-59/obj58?versionId=null 2023-07-29T00:35:30.090 DEBUG:boto:Data: 2023-07-29T00:35:30.091 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.091 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:30.091 DEBUG:boto:Port: 8001 2023-07-29T00:35:30.092 DEBUG:boto:Params: {} 2023-07-29T00:35:30.092 DEBUG:boto:Token: None 2023-07-29T00:35:30.093 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj58?versionId=null 2023-07-29T00:35:30.093 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VVbN5vkmBprQpTjYxhaJ/RsSAj4= 2023-07-29T00:35:30.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VVbN5vkmBprQpTjYxhaJ/RsSAj4=', 'Content-Length': '0'} 2023-07-29T00:35:30.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a78d69344bb8be15-0064c45ed2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.096 DEBUG:boto:path=/hiiqos-59/obj58 2023-07-29T00:35:30.096 DEBUG:boto:auth_path=/hiiqos-59/obj58 2023-07-29T00:35:30.097 DEBUG:boto:path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:35:30.097 DEBUG:boto:auth_path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:35:30.097 DEBUG:boto:Method: GET 2023-07-29T00:35:30.098 DEBUG:boto:Path: /hiiqos-59/obj58?versionId=null 2023-07-29T00:35:30.098 DEBUG:boto:Data: 2023-07-29T00:35:30.099 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.099 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:30.099 DEBUG:boto:Port: 8000 2023-07-29T00:35:30.100 DEBUG:boto:Params: {} 2023-07-29T00:35:30.100 DEBUG:boto:Token: None 2023-07-29T00:35:30.100 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj58?versionId=null 2023-07-29T00:35:30.101 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VVbN5vkmBprQpTjYxhaJ/RsSAj4= 2023-07-29T00:35:30.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VVbN5vkmBprQpTjYxhaJ/RsSAj4=', 'Content-Length': '0'} 2023-07-29T00:35:30.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003a9164795474c92-0064c45ed2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.104 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:30.104 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:30.105 DEBUG:boto:path=/hiiqos-59/?prefix=obj58 2023-07-29T00:35:30.105 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj58 2023-07-29T00:35:30.105 DEBUG:boto:Method: GET 2023-07-29T00:35:30.106 DEBUG:boto:Path: /hiiqos-59/?prefix=obj58 2023-07-29T00:35:30.106 DEBUG:boto:Data: 2023-07-29T00:35:30.106 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.107 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:30.107 DEBUG:boto:Port: 8001 2023-07-29T00:35:30.107 DEBUG:boto:Params: {} 2023-07-29T00:35:30.108 DEBUG:boto:Token: None 2023-07-29T00:35:30.108 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/ 2023-07-29T00:35:30.109 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qm+SsO8XwO9SXjVIZFoLoF0u/VU= 2023-07-29T00:35:30.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qm+SsO8XwO9SXjVIZFoLoF0u/VU=', 'Content-Length': '0'} 2023-07-29T00:35:30.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061cdf419a973486f-0064c45ed2-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.112 DEBUG:boto:b'hiiqos-59obj581000falseobj582023-07-29T00:35:01.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:30.113 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:30.113 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:30.113 DEBUG:boto:path=/hiiqos-59/?prefix=obj58 2023-07-29T00:35:30.114 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj58 2023-07-29T00:35:30.114 DEBUG:boto:Method: GET 2023-07-29T00:35:30.114 DEBUG:boto:Path: /hiiqos-59/?prefix=obj58 2023-07-29T00:35:30.115 DEBUG:boto:Data: 2023-07-29T00:35:30.115 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.115 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:30.116 DEBUG:boto:Port: 8000 2023-07-29T00:35:30.116 DEBUG:boto:Params: {} 2023-07-29T00:35:30.117 DEBUG:boto:Token: None 2023-07-29T00:35:30.117 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/ 2023-07-29T00:35:30.117 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qm+SsO8XwO9SXjVIZFoLoF0u/VU= 2023-07-29T00:35:30.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qm+SsO8XwO9SXjVIZFoLoF0u/VU=', 'Content-Length': '0'} 2023-07-29T00:35:30.121 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f09364b4f7307de-0064c45ed2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.122 DEBUG:boto:b'hiiqos-59obj581000falseobj582023-07-29T00:35:01.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:30.123 DEBUG:boto:path=/hiiqos-59/obj58 2023-07-29T00:35:30.123 DEBUG:boto:auth_path=/hiiqos-59/obj58 2023-07-29T00:35:30.123 DEBUG:boto:Method: HEAD 2023-07-29T00:35:30.124 DEBUG:boto:Path: /hiiqos-59/obj58 2023-07-29T00:35:30.124 DEBUG:boto:Data: 2023-07-29T00:35:30.124 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.125 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:30.125 DEBUG:boto:Port: 8001 2023-07-29T00:35:30.125 DEBUG:boto:Params: {} 2023-07-29T00:35:30.126 DEBUG:boto:Token: None 2023-07-29T00:35:30.126 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj58 2023-07-29T00:35:30.126 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LljtXOpTOOcP0+2p8AFqSiQTLtI= 2023-07-29T00:35:30.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LljtXOpTOOcP0+2p8AFqSiQTLtI=', 'Content-Length': '0'} 2023-07-29T00:35:30.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d8b2e11c2cfedc0-0064c45ed2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.129 DEBUG:boto:path=/hiiqos-59/obj58 2023-07-29T00:35:30.129 DEBUG:boto:auth_path=/hiiqos-59/obj58 2023-07-29T00:35:30.130 DEBUG:boto:Method: HEAD 2023-07-29T00:35:30.130 DEBUG:boto:Path: /hiiqos-59/obj58 2023-07-29T00:35:30.130 DEBUG:boto:Data: 2023-07-29T00:35:30.131 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.131 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:30.131 DEBUG:boto:Port: 8000 2023-07-29T00:35:30.132 DEBUG:boto:Params: {} 2023-07-29T00:35:30.132 DEBUG:boto:Token: None 2023-07-29T00:35:30.133 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj58 2023-07-29T00:35:30.133 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LljtXOpTOOcP0+2p8AFqSiQTLtI= 2023-07-29T00:35:30.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LljtXOpTOOcP0+2p8AFqSiQTLtI=', 'Content-Length': '0'} 2023-07-29T00:35:30.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064e90f40ee2e5b8a-0064c45ed2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-07-29T00:35:30.136 DEBUG:boto:path=/hiiqos-59/obj58 2023-07-29T00:35:30.136 DEBUG:boto:auth_path=/hiiqos-59/obj58 2023-07-29T00:35:30.137 DEBUG:boto:Method: GET 2023-07-29T00:35:30.137 DEBUG:boto:Path: /hiiqos-59/obj58 2023-07-29T00:35:30.138 DEBUG:boto:Data: 2023-07-29T00:35:30.138 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.138 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:30.139 DEBUG:boto:Port: 8001 2023-07-29T00:35:30.139 DEBUG:boto:Params: {} 2023-07-29T00:35:30.139 DEBUG:boto:Token: None 2023-07-29T00:35:30.140 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj58 2023-07-29T00:35:30.140 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZvYkBGAMXq9ikFNmG7iRA4ivt3w= 2023-07-29T00:35:30.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZvYkBGAMXq9ikFNmG7iRA4ivt3w=', 'Content-Length': '0'} 2023-07-29T00:35:30.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbc43e023279089b-0064c45ed2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.184 DEBUG:boto:path=/hiiqos-59/obj58 2023-07-29T00:35:30.185 DEBUG:boto:auth_path=/hiiqos-59/obj58 2023-07-29T00:35:30.185 DEBUG:boto:Method: GET 2023-07-29T00:35:30.186 DEBUG:boto:Path: /hiiqos-59/obj58 2023-07-29T00:35:30.186 DEBUG:boto:Data: 2023-07-29T00:35:30.186 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.187 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:30.187 DEBUG:boto:Port: 8000 2023-07-29T00:35:30.187 DEBUG:boto:Params: {} 2023-07-29T00:35:30.188 DEBUG:boto:Token: None 2023-07-29T00:35:30.188 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj58 2023-07-29T00:35:30.189 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZvYkBGAMXq9ikFNmG7iRA4ivt3w= 2023-07-29T00:35:30.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZvYkBGAMXq9ikFNmG7iRA4ivt3w=', 'Content-Length': '0'} 2023-07-29T00:35:30.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003928f08878ef745-0064c45ed2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.192 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:30.192 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:30.192 DEBUG:boto:path=/hiiqos-59/?prefix=obj58 2023-07-29T00:35:30.193 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj58 2023-07-29T00:35:30.193 DEBUG:boto:Method: GET 2023-07-29T00:35:30.193 DEBUG:boto:Path: /hiiqos-59/?prefix=obj58 2023-07-29T00:35:30.194 DEBUG:boto:Data: 2023-07-29T00:35:30.194 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.194 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:30.195 DEBUG:boto:Port: 8001 2023-07-29T00:35:30.195 DEBUG:boto:Params: {} 2023-07-29T00:35:30.196 DEBUG:boto:Token: None 2023-07-29T00:35:30.196 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/ 2023-07-29T00:35:30.196 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qm+SsO8XwO9SXjVIZFoLoF0u/VU= 2023-07-29T00:35:30.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qm+SsO8XwO9SXjVIZFoLoF0u/VU=', 'Content-Length': '0'} 2023-07-29T00:35:30.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098688b3a0d7954a3-0064c45ed2-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.200 DEBUG:boto:b'hiiqos-59obj581000falseobj582023-07-29T00:35:01.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:30.200 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:30.201 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:30.201 DEBUG:boto:path=/hiiqos-59/?prefix=obj58 2023-07-29T00:35:30.201 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj58 2023-07-29T00:35:30.202 DEBUG:boto:Method: GET 2023-07-29T00:35:30.202 DEBUG:boto:Path: /hiiqos-59/?prefix=obj58 2023-07-29T00:35:30.203 DEBUG:boto:Data: 2023-07-29T00:35:30.204 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.205 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:30.205 DEBUG:boto:Port: 8000 2023-07-29T00:35:30.206 DEBUG:boto:Params: {} 2023-07-29T00:35:30.207 DEBUG:boto:Token: None 2023-07-29T00:35:30.208 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/ 2023-07-29T00:35:30.209 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qm+SsO8XwO9SXjVIZFoLoF0u/VU= 2023-07-29T00:35:30.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qm+SsO8XwO9SXjVIZFoLoF0u/VU=', 'Content-Length': '0'} 2023-07-29T00:35:30.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000733dd4e873d44328-0064c45ed2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.215 DEBUG:boto:b'hiiqos-59obj581000falseobj582023-07-29T00:35:01.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:30.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-07-29T00:35:30.216 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:35:30.216 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:35:30.217 DEBUG:boto:path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:35:30.217 DEBUG:boto:auth_path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:35:30.217 DEBUG:boto:Method: GET 2023-07-29T00:35:30.218 DEBUG:boto:Path: /hiiqos-59/obj68?versionId=null 2023-07-29T00:35:30.218 DEBUG:boto:Data: 2023-07-29T00:35:30.218 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.219 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:30.219 DEBUG:boto:Port: 8001 2023-07-29T00:35:30.219 DEBUG:boto:Params: {} 2023-07-29T00:35:30.220 DEBUG:boto:Token: None 2023-07-29T00:35:30.220 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj68?versionId=null 2023-07-29T00:35:30.221 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vHEGFDawHs9MT4Lz3i5IzWHv8qs= 2023-07-29T00:35:30.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vHEGFDawHs9MT4Lz3i5IzWHv8qs=', 'Content-Length': '0'} 2023-07-29T00:35:30.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005995bf8006493d7c-0064c45ed2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.224 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:35:30.224 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:35:30.224 DEBUG:boto:path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:35:30.225 DEBUG:boto:auth_path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:35:30.225 DEBUG:boto:Method: GET 2023-07-29T00:35:30.225 DEBUG:boto:Path: /hiiqos-59/obj68?versionId=null 2023-07-29T00:35:30.226 DEBUG:boto:Data: 2023-07-29T00:35:30.226 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.227 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:30.227 DEBUG:boto:Port: 8000 2023-07-29T00:35:30.227 DEBUG:boto:Params: {} 2023-07-29T00:35:30.228 DEBUG:boto:Token: None 2023-07-29T00:35:30.228 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj68?versionId=null 2023-07-29T00:35:30.229 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vHEGFDawHs9MT4Lz3i5IzWHv8qs= 2023-07-29T00:35:30.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vHEGFDawHs9MT4Lz3i5IzWHv8qs=', 'Content-Length': '0'} 2023-07-29T00:35:30.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dc23dcaffc62472-0064c45ed2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.272 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:35:30.273 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:35:30.273 DEBUG:boto:path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:35:30.273 DEBUG:boto:auth_path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:35:30.274 DEBUG:boto:Method: HEAD 2023-07-29T00:35:30.274 DEBUG:boto:Path: /hiiqos-59/obj68?versionId=null 2023-07-29T00:35:30.274 DEBUG:boto:Data: 2023-07-29T00:35:30.275 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.275 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:30.275 DEBUG:boto:Port: 8001 2023-07-29T00:35:30.276 DEBUG:boto:Params: {} 2023-07-29T00:35:30.276 DEBUG:boto:Token: None 2023-07-29T00:35:30.277 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj68?versionId=null 2023-07-29T00:35:30.277 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eMwSqlCBjrq5Dr6hVjt1Lkw2M1Q= 2023-07-29T00:35:30.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eMwSqlCBjrq5Dr6hVjt1Lkw2M1Q=', 'Content-Length': '0'} 2023-07-29T00:35:30.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069a0be88c9f8a18b-0064c45ed2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.280 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:35:30.280 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:35:30.281 DEBUG:boto:path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:35:30.281 DEBUG:boto:auth_path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:35:30.281 DEBUG:boto:Method: HEAD 2023-07-29T00:35:30.282 DEBUG:boto:Path: /hiiqos-59/obj68?versionId=null 2023-07-29T00:35:30.282 DEBUG:boto:Data: 2023-07-29T00:35:30.282 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.283 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:30.283 DEBUG:boto:Port: 8000 2023-07-29T00:35:30.283 DEBUG:boto:Params: {} 2023-07-29T00:35:30.284 DEBUG:boto:Token: None 2023-07-29T00:35:30.284 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj68?versionId=null 2023-07-29T00:35:30.285 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eMwSqlCBjrq5Dr6hVjt1Lkw2M1Q= 2023-07-29T00:35:30.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eMwSqlCBjrq5Dr6hVjt1Lkw2M1Q=', 'Content-Length': '0'} 2023-07-29T00:35:30.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006d929349adc0f59-0064c45ed2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-07-29T00:35:30.288 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:35:30.288 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:35:30.289 DEBUG:boto:path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:35:30.289 DEBUG:boto:auth_path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:35:30.289 DEBUG:boto:Method: GET 2023-07-29T00:35:30.290 DEBUG:boto:Path: /hiiqos-59/obj68?versionId=null 2023-07-29T00:35:30.290 DEBUG:boto:Data: 2023-07-29T00:35:30.290 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.291 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:30.291 DEBUG:boto:Port: 8001 2023-07-29T00:35:30.292 DEBUG:boto:Params: {} 2023-07-29T00:35:30.292 DEBUG:boto:Token: None 2023-07-29T00:35:30.292 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj68?versionId=null 2023-07-29T00:35:30.293 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vHEGFDawHs9MT4Lz3i5IzWHv8qs= 2023-07-29T00:35:30.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vHEGFDawHs9MT4Lz3i5IzWHv8qs=', 'Content-Length': '0'} 2023-07-29T00:35:30.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba06a487ac1ff351-0064c45ed2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.296 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:35:30.296 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:35:30.297 DEBUG:boto:path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:35:30.297 DEBUG:boto:auth_path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:35:30.297 DEBUG:boto:Method: GET 2023-07-29T00:35:30.298 DEBUG:boto:Path: /hiiqos-59/obj68?versionId=null 2023-07-29T00:35:30.298 DEBUG:boto:Data: 2023-07-29T00:35:30.298 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.299 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:30.299 DEBUG:boto:Port: 8000 2023-07-29T00:35:30.299 DEBUG:boto:Params: {} 2023-07-29T00:35:30.300 DEBUG:boto:Token: None 2023-07-29T00:35:30.300 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj68?versionId=null 2023-07-29T00:35:30.301 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vHEGFDawHs9MT4Lz3i5IzWHv8qs= 2023-07-29T00:35:30.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vHEGFDawHs9MT4Lz3i5IzWHv8qs=', 'Content-Length': '0'} 2023-07-29T00:35:30.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068d32b7adc4bf4ac-0064c45ed2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.303 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:30.304 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:30.304 DEBUG:boto:path=/hiiqos-59/?prefix=obj68 2023-07-29T00:35:30.304 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj68 2023-07-29T00:35:30.305 DEBUG:boto:Method: GET 2023-07-29T00:35:30.305 DEBUG:boto:Path: /hiiqos-59/?prefix=obj68 2023-07-29T00:35:30.306 DEBUG:boto:Data: 2023-07-29T00:35:30.306 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.306 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:30.307 DEBUG:boto:Port: 8001 2023-07-29T00:35:30.307 DEBUG:boto:Params: {} 2023-07-29T00:35:30.307 DEBUG:boto:Token: None 2023-07-29T00:35:30.308 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/ 2023-07-29T00:35:30.308 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qm+SsO8XwO9SXjVIZFoLoF0u/VU= 2023-07-29T00:35:30.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qm+SsO8XwO9SXjVIZFoLoF0u/VU=', 'Content-Length': '0'} 2023-07-29T00:35:30.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec2902fcc065d6dc-0064c45ed2-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.311 DEBUG:boto:b'hiiqos-59obj681000falseobj682023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:30.312 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:30.313 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:30.313 DEBUG:boto:path=/hiiqos-59/?prefix=obj68 2023-07-29T00:35:30.313 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj68 2023-07-29T00:35:30.314 DEBUG:boto:Method: GET 2023-07-29T00:35:30.314 DEBUG:boto:Path: /hiiqos-59/?prefix=obj68 2023-07-29T00:35:30.314 DEBUG:boto:Data: 2023-07-29T00:35:30.315 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.315 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:30.315 DEBUG:boto:Port: 8000 2023-07-29T00:35:30.316 DEBUG:boto:Params: {} 2023-07-29T00:35:30.316 DEBUG:boto:Token: None 2023-07-29T00:35:30.317 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/ 2023-07-29T00:35:30.317 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qm+SsO8XwO9SXjVIZFoLoF0u/VU= 2023-07-29T00:35:30.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qm+SsO8XwO9SXjVIZFoLoF0u/VU=', 'Content-Length': '0'} 2023-07-29T00:35:30.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd50fa3de979836c-0064c45ed2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.322 DEBUG:boto:b'hiiqos-59obj681000falseobj682023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:30.322 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:35:30.323 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:35:30.323 DEBUG:boto:Method: HEAD 2023-07-29T00:35:30.324 DEBUG:boto:Path: /hiiqos-59/obj68 2023-07-29T00:35:30.324 DEBUG:boto:Data: 2023-07-29T00:35:30.325 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.326 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:30.326 DEBUG:boto:Port: 8001 2023-07-29T00:35:30.327 DEBUG:boto:Params: {} 2023-07-29T00:35:30.328 DEBUG:boto:Token: None 2023-07-29T00:35:30.329 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj68 2023-07-29T00:35:30.330 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:B+kZJF77cXGe+zL/c3Gmb5iJd+Q= 2023-07-29T00:35:30.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:B+kZJF77cXGe+zL/c3Gmb5iJd+Q=', 'Content-Length': '0'} 2023-07-29T00:35:30.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000faf0d633c73e5a91-0064c45ed2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.334 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:35:30.335 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:35:30.335 DEBUG:boto:Method: HEAD 2023-07-29T00:35:30.336 DEBUG:boto:Path: /hiiqos-59/obj68 2023-07-29T00:35:30.336 DEBUG:boto:Data: 2023-07-29T00:35:30.336 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.337 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:30.337 DEBUG:boto:Port: 8000 2023-07-29T00:35:30.337 DEBUG:boto:Params: {} 2023-07-29T00:35:30.338 DEBUG:boto:Token: None 2023-07-29T00:35:30.338 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj68 2023-07-29T00:35:30.339 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:B+kZJF77cXGe+zL/c3Gmb5iJd+Q= 2023-07-29T00:35:30.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:B+kZJF77cXGe+zL/c3Gmb5iJd+Q=', 'Content-Length': '0'} 2023-07-29T00:35:30.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000674bf356b97b4cb3-0064c45ed2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-07-29T00:35:30.342 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:35:30.342 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:35:30.343 DEBUG:boto:Method: GET 2023-07-29T00:35:30.343 DEBUG:boto:Path: /hiiqos-59/obj68 2023-07-29T00:35:30.343 DEBUG:boto:Data: 2023-07-29T00:35:30.344 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.344 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:30.344 DEBUG:boto:Port: 8001 2023-07-29T00:35:30.345 DEBUG:boto:Params: {} 2023-07-29T00:35:30.345 DEBUG:boto:Token: None 2023-07-29T00:35:30.346 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj68 2023-07-29T00:35:30.346 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lClq/19ushds9YXzT5SzJ3OZABM= 2023-07-29T00:35:30.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lClq/19ushds9YXzT5SzJ3OZABM=', 'Content-Length': '0'} 2023-07-29T00:35:30.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004dc04022da65c977-0064c45ed2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.349 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:35:30.349 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:35:30.350 DEBUG:boto:Method: GET 2023-07-29T00:35:30.350 DEBUG:boto:Path: /hiiqos-59/obj68 2023-07-29T00:35:30.350 DEBUG:boto:Data: 2023-07-29T00:35:30.351 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.351 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:30.351 DEBUG:boto:Port: 8000 2023-07-29T00:35:30.352 DEBUG:boto:Params: {} 2023-07-29T00:35:30.352 DEBUG:boto:Token: None 2023-07-29T00:35:30.353 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj68 2023-07-29T00:35:30.353 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lClq/19ushds9YXzT5SzJ3OZABM= 2023-07-29T00:35:30.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lClq/19ushds9YXzT5SzJ3OZABM=', 'Content-Length': '0'} 2023-07-29T00:35:30.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008054b315f02213e3-0064c45ed2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.400 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:30.401 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:30.401 DEBUG:boto:path=/hiiqos-59/?prefix=obj68 2023-07-29T00:35:30.401 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj68 2023-07-29T00:35:30.402 DEBUG:boto:Method: GET 2023-07-29T00:35:30.402 DEBUG:boto:Path: /hiiqos-59/?prefix=obj68 2023-07-29T00:35:30.403 DEBUG:boto:Data: 2023-07-29T00:35:30.403 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.403 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:30.404 DEBUG:boto:Port: 8001 2023-07-29T00:35:30.404 DEBUG:boto:Params: {} 2023-07-29T00:35:30.405 DEBUG:boto:Token: None 2023-07-29T00:35:30.406 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/ 2023-07-29T00:35:30.407 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qm+SsO8XwO9SXjVIZFoLoF0u/VU= 2023-07-29T00:35:30.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qm+SsO8XwO9SXjVIZFoLoF0u/VU=', 'Content-Length': '0'} 2023-07-29T00:35:30.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015b8afc9495744ad-0064c45ed2-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.411 DEBUG:boto:b'hiiqos-59obj681000falseobj682023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:30.412 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:30.412 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:30.413 DEBUG:boto:path=/hiiqos-59/?prefix=obj68 2023-07-29T00:35:30.413 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj68 2023-07-29T00:35:30.413 DEBUG:boto:Method: GET 2023-07-29T00:35:30.414 DEBUG:boto:Path: /hiiqos-59/?prefix=obj68 2023-07-29T00:35:30.414 DEBUG:boto:Data: 2023-07-29T00:35:30.414 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.415 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:30.415 DEBUG:boto:Port: 8000 2023-07-29T00:35:30.416 DEBUG:boto:Params: {} 2023-07-29T00:35:30.416 DEBUG:boto:Token: None 2023-07-29T00:35:30.416 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/ 2023-07-29T00:35:30.417 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qm+SsO8XwO9SXjVIZFoLoF0u/VU= 2023-07-29T00:35:30.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qm+SsO8XwO9SXjVIZFoLoF0u/VU=', 'Content-Length': '0'} 2023-07-29T00:35:30.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098d2109d2b3bce53-0064c45ed2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.424 DEBUG:boto:b'hiiqos-59obj681000falseobj682023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:30.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-07-29T00:35:30.425 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:35:30.425 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:35:30.425 DEBUG:boto:path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:35:30.426 DEBUG:boto:auth_path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:35:30.426 DEBUG:boto:Method: GET 2023-07-29T00:35:30.427 DEBUG:boto:Path: /hiiqos-59/obj69?versionId=null 2023-07-29T00:35:30.427 DEBUG:boto:Data: 2023-07-29T00:35:30.427 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.428 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:30.428 DEBUG:boto:Port: 8001 2023-07-29T00:35:30.428 DEBUG:boto:Params: {} 2023-07-29T00:35:30.429 DEBUG:boto:Token: None 2023-07-29T00:35:30.429 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj69?versionId=null 2023-07-29T00:35:30.430 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:357hHiPFbfLHu18CnsTvGxn4L18= 2023-07-29T00:35:30.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:357hHiPFbfLHu18CnsTvGxn4L18=', 'Content-Length': '0'} 2023-07-29T00:35:30.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000562f12117fdb8c8e-0064c45ed2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.433 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:35:30.433 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:35:30.433 DEBUG:boto:path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:35:30.434 DEBUG:boto:auth_path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:35:30.434 DEBUG:boto:Method: GET 2023-07-29T00:35:30.435 DEBUG:boto:Path: /hiiqos-59/obj69?versionId=null 2023-07-29T00:35:30.435 DEBUG:boto:Data: 2023-07-29T00:35:30.436 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.436 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:30.437 DEBUG:boto:Port: 8000 2023-07-29T00:35:30.438 DEBUG:boto:Params: {} 2023-07-29T00:35:30.439 DEBUG:boto:Token: None 2023-07-29T00:35:30.439 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj69?versionId=null 2023-07-29T00:35:30.440 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:357hHiPFbfLHu18CnsTvGxn4L18= 2023-07-29T00:35:30.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:357hHiPFbfLHu18CnsTvGxn4L18=', 'Content-Length': '0'} 2023-07-29T00:35:30.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f8f576ed92d64b2-0064c45ed2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.443 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:35:30.443 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:35:30.443 DEBUG:boto:path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:35:30.444 DEBUG:boto:auth_path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:35:30.444 DEBUG:boto:Method: HEAD 2023-07-29T00:35:30.444 DEBUG:boto:Path: /hiiqos-59/obj69?versionId=null 2023-07-29T00:35:30.445 DEBUG:boto:Data: 2023-07-29T00:35:30.445 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.445 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:30.446 DEBUG:boto:Port: 8001 2023-07-29T00:35:30.446 DEBUG:boto:Params: {} 2023-07-29T00:35:30.446 DEBUG:boto:Token: None 2023-07-29T00:35:30.447 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj69?versionId=null 2023-07-29T00:35:30.447 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S5AG9iysXBZLhaIxrrJwHrCPwRE= 2023-07-29T00:35:30.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S5AG9iysXBZLhaIxrrJwHrCPwRE=', 'Content-Length': '0'} 2023-07-29T00:35:30.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000275dce16e193c1ad-0064c45ed2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.450 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:35:30.450 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:35:30.451 DEBUG:boto:path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:35:30.451 DEBUG:boto:auth_path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:35:30.451 DEBUG:boto:Method: HEAD 2023-07-29T00:35:30.452 DEBUG:boto:Path: /hiiqos-59/obj69?versionId=null 2023-07-29T00:35:30.452 DEBUG:boto:Data: 2023-07-29T00:35:30.452 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.453 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:30.453 DEBUG:boto:Port: 8000 2023-07-29T00:35:30.453 DEBUG:boto:Params: {} 2023-07-29T00:35:30.454 DEBUG:boto:Token: None 2023-07-29T00:35:30.454 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj69?versionId=null 2023-07-29T00:35:30.455 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S5AG9iysXBZLhaIxrrJwHrCPwRE= 2023-07-29T00:35:30.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S5AG9iysXBZLhaIxrrJwHrCPwRE=', 'Content-Length': '0'} 2023-07-29T00:35:30.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078be7709ddfa4afc-0064c45ed2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-07-29T00:35:30.458 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:35:30.458 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:35:30.458 DEBUG:boto:path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:35:30.459 DEBUG:boto:auth_path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:35:30.459 DEBUG:boto:Method: GET 2023-07-29T00:35:30.459 DEBUG:boto:Path: /hiiqos-59/obj69?versionId=null 2023-07-29T00:35:30.460 DEBUG:boto:Data: 2023-07-29T00:35:30.460 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.461 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:30.461 DEBUG:boto:Port: 8001 2023-07-29T00:35:30.461 DEBUG:boto:Params: {} 2023-07-29T00:35:30.462 DEBUG:boto:Token: None 2023-07-29T00:35:30.462 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj69?versionId=null 2023-07-29T00:35:30.463 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:357hHiPFbfLHu18CnsTvGxn4L18= 2023-07-29T00:35:30.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:357hHiPFbfLHu18CnsTvGxn4L18=', 'Content-Length': '0'} 2023-07-29T00:35:30.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010cca51413a146a9-0064c45ed2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.508 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:35:30.509 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:35:30.509 DEBUG:boto:path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:35:30.510 DEBUG:boto:auth_path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:35:30.510 DEBUG:boto:Method: GET 2023-07-29T00:35:30.511 DEBUG:boto:Path: /hiiqos-59/obj69?versionId=null 2023-07-29T00:35:30.511 DEBUG:boto:Data: 2023-07-29T00:35:30.511 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.512 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:30.512 DEBUG:boto:Port: 8000 2023-07-29T00:35:30.512 DEBUG:boto:Params: {} 2023-07-29T00:35:30.513 DEBUG:boto:Token: None 2023-07-29T00:35:30.513 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj69?versionId=null 2023-07-29T00:35:30.514 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:357hHiPFbfLHu18CnsTvGxn4L18= 2023-07-29T00:35:30.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:357hHiPFbfLHu18CnsTvGxn4L18=', 'Content-Length': '0'} 2023-07-29T00:35:30.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d2ac5f651acc982-0064c45ed2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.517 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:30.517 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:30.518 DEBUG:boto:path=/hiiqos-59/?prefix=obj69 2023-07-29T00:35:30.518 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj69 2023-07-29T00:35:30.518 DEBUG:boto:Method: GET 2023-07-29T00:35:30.519 DEBUG:boto:Path: /hiiqos-59/?prefix=obj69 2023-07-29T00:35:30.519 DEBUG:boto:Data: 2023-07-29T00:35:30.519 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.520 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:30.520 DEBUG:boto:Port: 8001 2023-07-29T00:35:30.520 DEBUG:boto:Params: {} 2023-07-29T00:35:30.521 DEBUG:boto:Token: None 2023-07-29T00:35:30.521 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/ 2023-07-29T00:35:30.522 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qm+SsO8XwO9SXjVIZFoLoF0u/VU= 2023-07-29T00:35:30.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qm+SsO8XwO9SXjVIZFoLoF0u/VU=', 'Content-Length': '0'} 2023-07-29T00:35:30.524 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006dc7db1000e03b79-0064c45ed2-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.525 DEBUG:boto:b'hiiqos-59obj691000falseobj692023-07-29T00:33:06.857Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:30.526 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:30.526 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:30.526 DEBUG:boto:path=/hiiqos-59/?prefix=obj69 2023-07-29T00:35:30.527 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj69 2023-07-29T00:35:30.527 DEBUG:boto:Method: GET 2023-07-29T00:35:30.527 DEBUG:boto:Path: /hiiqos-59/?prefix=obj69 2023-07-29T00:35:30.528 DEBUG:boto:Data: 2023-07-29T00:35:30.528 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.528 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:30.529 DEBUG:boto:Port: 8000 2023-07-29T00:35:30.529 DEBUG:boto:Params: {} 2023-07-29T00:35:30.530 DEBUG:boto:Token: None 2023-07-29T00:35:30.530 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/ 2023-07-29T00:35:30.530 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qm+SsO8XwO9SXjVIZFoLoF0u/VU= 2023-07-29T00:35:30.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qm+SsO8XwO9SXjVIZFoLoF0u/VU=', 'Content-Length': '0'} 2023-07-29T00:35:30.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000accd5d4a8a1cd579-0064c45ed2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.580 DEBUG:boto:b'hiiqos-59obj691000falseobj692023-07-29T00:33:06.857Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:30.581 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:35:30.581 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:35:30.582 DEBUG:boto:Method: HEAD 2023-07-29T00:35:30.582 DEBUG:boto:Path: /hiiqos-59/obj69 2023-07-29T00:35:30.582 DEBUG:boto:Data: 2023-07-29T00:35:30.583 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.583 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:30.583 DEBUG:boto:Port: 8001 2023-07-29T00:35:30.584 DEBUG:boto:Params: {} 2023-07-29T00:35:30.584 DEBUG:boto:Token: None 2023-07-29T00:35:30.585 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj69 2023-07-29T00:35:30.585 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LIa6xi3dAO/oorFWCGT1o0tcCLk= 2023-07-29T00:35:30.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LIa6xi3dAO/oorFWCGT1o0tcCLk=', 'Content-Length': '0'} 2023-07-29T00:35:30.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b687cc2140b8ee9a-0064c45ed2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.588 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:35:30.588 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:35:30.589 DEBUG:boto:Method: HEAD 2023-07-29T00:35:30.589 DEBUG:boto:Path: /hiiqos-59/obj69 2023-07-29T00:35:30.589 DEBUG:boto:Data: 2023-07-29T00:35:30.590 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.590 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:30.590 DEBUG:boto:Port: 8000 2023-07-29T00:35:30.591 DEBUG:boto:Params: {} 2023-07-29T00:35:30.591 DEBUG:boto:Token: None 2023-07-29T00:35:30.592 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj69 2023-07-29T00:35:30.592 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LIa6xi3dAO/oorFWCGT1o0tcCLk= 2023-07-29T00:35:30.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LIa6xi3dAO/oorFWCGT1o0tcCLk=', 'Content-Length': '0'} 2023-07-29T00:35:30.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005932526488b6a0a3-0064c45ed2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-07-29T00:35:30.595 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:35:30.595 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:35:30.596 DEBUG:boto:Method: GET 2023-07-29T00:35:30.596 DEBUG:boto:Path: /hiiqos-59/obj69 2023-07-29T00:35:30.597 DEBUG:boto:Data: 2023-07-29T00:35:30.597 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.597 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:30.598 DEBUG:boto:Port: 8001 2023-07-29T00:35:30.598 DEBUG:boto:Params: {} 2023-07-29T00:35:30.599 DEBUG:boto:Token: None 2023-07-29T00:35:30.599 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj69 2023-07-29T00:35:30.599 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lmmghRUSduIjbnqi6ScbfuZs7iA= 2023-07-29T00:35:30.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lmmghRUSduIjbnqi6ScbfuZs7iA=', 'Content-Length': '0'} 2023-07-29T00:35:30.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c625c6e51bb608ee-0064c45ed2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.602 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:35:30.603 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:35:30.603 DEBUG:boto:Method: GET 2023-07-29T00:35:30.603 DEBUG:boto:Path: /hiiqos-59/obj69 2023-07-29T00:35:30.604 DEBUG:boto:Data: 2023-07-29T00:35:30.604 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.605 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:30.605 DEBUG:boto:Port: 8000 2023-07-29T00:35:30.605 DEBUG:boto:Params: {} 2023-07-29T00:35:30.606 DEBUG:boto:Token: None 2023-07-29T00:35:30.606 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj69 2023-07-29T00:35:30.607 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lmmghRUSduIjbnqi6ScbfuZs7iA= 2023-07-29T00:35:30.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lmmghRUSduIjbnqi6ScbfuZs7iA=', 'Content-Length': '0'} 2023-07-29T00:35:30.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3a6cda46975b694-0064c45ed2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.609 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:30.610 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:30.610 DEBUG:boto:path=/hiiqos-59/?prefix=obj69 2023-07-29T00:35:30.611 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj69 2023-07-29T00:35:30.611 DEBUG:boto:Method: GET 2023-07-29T00:35:30.611 DEBUG:boto:Path: /hiiqos-59/?prefix=obj69 2023-07-29T00:35:30.612 DEBUG:boto:Data: 2023-07-29T00:35:30.612 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.612 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:30.613 DEBUG:boto:Port: 8001 2023-07-29T00:35:30.613 DEBUG:boto:Params: {} 2023-07-29T00:35:30.613 DEBUG:boto:Token: None 2023-07-29T00:35:30.614 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/ 2023-07-29T00:35:30.614 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qm+SsO8XwO9SXjVIZFoLoF0u/VU= 2023-07-29T00:35:30.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qm+SsO8XwO9SXjVIZFoLoF0u/VU=', 'Content-Length': '0'} 2023-07-29T00:35:30.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000344377ed21dfaa69-0064c45ed2-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.618 DEBUG:boto:b'hiiqos-59obj691000falseobj692023-07-29T00:33:06.857Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:30.618 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:30.619 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:30.619 DEBUG:boto:path=/hiiqos-59/?prefix=obj69 2023-07-29T00:35:30.620 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj69 2023-07-29T00:35:30.620 DEBUG:boto:Method: GET 2023-07-29T00:35:30.620 DEBUG:boto:Path: /hiiqos-59/?prefix=obj69 2023-07-29T00:35:30.621 DEBUG:boto:Data: 2023-07-29T00:35:30.621 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.621 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:30.622 DEBUG:boto:Port: 8000 2023-07-29T00:35:30.622 DEBUG:boto:Params: {} 2023-07-29T00:35:30.622 DEBUG:boto:Token: None 2023-07-29T00:35:30.623 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/ 2023-07-29T00:35:30.623 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qm+SsO8XwO9SXjVIZFoLoF0u/VU= 2023-07-29T00:35:30.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qm+SsO8XwO9SXjVIZFoLoF0u/VU=', 'Content-Length': '0'} 2023-07-29T00:35:30.628 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e7542d3ef18803b-0064c45ed2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.628 DEBUG:boto:b'hiiqos-59obj691000falseobj692023-07-29T00:33:06.857Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:30.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-07-29T00:35:30.629 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:35:30.630 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:35:30.630 DEBUG:boto:path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:35:30.630 DEBUG:boto:auth_path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:35:30.631 DEBUG:boto:Method: GET 2023-07-29T00:35:30.631 DEBUG:boto:Path: /hiiqos-59/obj76?versionId=null 2023-07-29T00:35:30.631 DEBUG:boto:Data: 2023-07-29T00:35:30.632 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.632 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:30.632 DEBUG:boto:Port: 8001 2023-07-29T00:35:30.633 DEBUG:boto:Params: {} 2023-07-29T00:35:30.633 DEBUG:boto:Token: None 2023-07-29T00:35:30.634 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj76?versionId=null 2023-07-29T00:35:30.634 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZHOiOzGDseKn1akbJfITa2dnSfg= 2023-07-29T00:35:30.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZHOiOzGDseKn1akbJfITa2dnSfg=', 'Content-Length': '0'} 2023-07-29T00:35:30.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014a965cba80ee62a-0064c45ed2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.637 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:35:30.637 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:35:30.638 DEBUG:boto:path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:35:30.638 DEBUG:boto:auth_path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:35:30.639 DEBUG:boto:Method: GET 2023-07-29T00:35:30.639 DEBUG:boto:Path: /hiiqos-59/obj76?versionId=null 2023-07-29T00:35:30.639 DEBUG:boto:Data: 2023-07-29T00:35:30.640 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.640 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:30.640 DEBUG:boto:Port: 8000 2023-07-29T00:35:30.644 DEBUG:boto:Params: {} 2023-07-29T00:35:30.645 DEBUG:boto:Token: None 2023-07-29T00:35:30.645 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj76?versionId=null 2023-07-29T00:35:30.645 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZHOiOzGDseKn1akbJfITa2dnSfg= 2023-07-29T00:35:30.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZHOiOzGDseKn1akbJfITa2dnSfg=', 'Content-Length': '0'} 2023-07-29T00:35:30.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7fb59ea4dadc731-0064c45ed2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.648 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:35:30.649 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:35:30.649 DEBUG:boto:path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:35:30.649 DEBUG:boto:auth_path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:35:30.650 DEBUG:boto:Method: HEAD 2023-07-29T00:35:30.650 DEBUG:boto:Path: /hiiqos-59/obj76?versionId=null 2023-07-29T00:35:30.650 DEBUG:boto:Data: 2023-07-29T00:35:30.651 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.651 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:30.651 DEBUG:boto:Port: 8001 2023-07-29T00:35:30.652 DEBUG:boto:Params: {} 2023-07-29T00:35:30.652 DEBUG:boto:Token: None 2023-07-29T00:35:30.652 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj76?versionId=null 2023-07-29T00:35:30.653 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:epHO0xY1fYQJBrlcAQZXRQkZAMQ= 2023-07-29T00:35:30.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:epHO0xY1fYQJBrlcAQZXRQkZAMQ=', 'Content-Length': '0'} 2023-07-29T00:35:30.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000248dd68ca5ca92e7-0064c45ed2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.655 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:35:30.656 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:35:30.656 DEBUG:boto:path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:35:30.656 DEBUG:boto:auth_path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:35:30.657 DEBUG:boto:Method: HEAD 2023-07-29T00:35:30.657 DEBUG:boto:Path: /hiiqos-59/obj76?versionId=null 2023-07-29T00:35:30.657 DEBUG:boto:Data: 2023-07-29T00:35:30.658 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.658 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:30.658 DEBUG:boto:Port: 8000 2023-07-29T00:35:30.659 DEBUG:boto:Params: {} 2023-07-29T00:35:30.659 DEBUG:boto:Token: None 2023-07-29T00:35:30.659 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj76?versionId=null 2023-07-29T00:35:30.660 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:epHO0xY1fYQJBrlcAQZXRQkZAMQ= 2023-07-29T00:35:30.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:epHO0xY1fYQJBrlcAQZXRQkZAMQ=', 'Content-Length': '0'} 2023-07-29T00:35:30.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bca501f4f7d0ed3-0064c45ed2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-07-29T00:35:30.663 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:35:30.663 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:35:30.663 DEBUG:boto:path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:35:30.664 DEBUG:boto:auth_path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:35:30.664 DEBUG:boto:Method: GET 2023-07-29T00:35:30.665 DEBUG:boto:Path: /hiiqos-59/obj76?versionId=null 2023-07-29T00:35:30.665 DEBUG:boto:Data: 2023-07-29T00:35:30.665 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.666 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:30.666 DEBUG:boto:Port: 8001 2023-07-29T00:35:30.666 DEBUG:boto:Params: {} 2023-07-29T00:35:30.667 DEBUG:boto:Token: None 2023-07-29T00:35:30.667 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj76?versionId=null 2023-07-29T00:35:30.668 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZHOiOzGDseKn1akbJfITa2dnSfg= 2023-07-29T00:35:30.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZHOiOzGDseKn1akbJfITa2dnSfg=', 'Content-Length': '0'} 2023-07-29T00:35:30.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033772a3edb112b4c-0064c45ed2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.712 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:35:30.713 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:35:30.713 DEBUG:boto:path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:35:30.713 DEBUG:boto:auth_path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:35:30.714 DEBUG:boto:Method: GET 2023-07-29T00:35:30.714 DEBUG:boto:Path: /hiiqos-59/obj76?versionId=null 2023-07-29T00:35:30.715 DEBUG:boto:Data: 2023-07-29T00:35:30.715 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.715 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:30.716 DEBUG:boto:Port: 8000 2023-07-29T00:35:30.716 DEBUG:boto:Params: {} 2023-07-29T00:35:30.716 DEBUG:boto:Token: None 2023-07-29T00:35:30.717 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj76?versionId=null 2023-07-29T00:35:30.717 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZHOiOzGDseKn1akbJfITa2dnSfg= 2023-07-29T00:35:30.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZHOiOzGDseKn1akbJfITa2dnSfg=', 'Content-Length': '0'} 2023-07-29T00:35:30.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c7e3b7154e9a8c7-0064c45ed2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.721 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:30.721 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:30.721 DEBUG:boto:path=/hiiqos-59/?prefix=obj76 2023-07-29T00:35:30.722 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj76 2023-07-29T00:35:30.722 DEBUG:boto:Method: GET 2023-07-29T00:35:30.722 DEBUG:boto:Path: /hiiqos-59/?prefix=obj76 2023-07-29T00:35:30.723 DEBUG:boto:Data: 2023-07-29T00:35:30.723 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.723 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:30.724 DEBUG:boto:Port: 8001 2023-07-29T00:35:30.724 DEBUG:boto:Params: {} 2023-07-29T00:35:30.725 DEBUG:boto:Token: None 2023-07-29T00:35:30.725 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/ 2023-07-29T00:35:30.725 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qm+SsO8XwO9SXjVIZFoLoF0u/VU= 2023-07-29T00:35:30.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qm+SsO8XwO9SXjVIZFoLoF0u/VU=', 'Content-Length': '0'} 2023-07-29T00:35:30.728 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ce8f254ef57da76-0064c45ed2-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.729 DEBUG:boto:b'hiiqos-59obj761000falseobj762023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:30.729 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:30.730 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:30.730 DEBUG:boto:path=/hiiqos-59/?prefix=obj76 2023-07-29T00:35:30.730 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj76 2023-07-29T00:35:30.731 DEBUG:boto:Method: GET 2023-07-29T00:35:30.731 DEBUG:boto:Path: /hiiqos-59/?prefix=obj76 2023-07-29T00:35:30.731 DEBUG:boto:Data: 2023-07-29T00:35:30.732 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.732 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:30.733 DEBUG:boto:Port: 8000 2023-07-29T00:35:30.733 DEBUG:boto:Params: {} 2023-07-29T00:35:30.733 DEBUG:boto:Token: None 2023-07-29T00:35:30.734 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/ 2023-07-29T00:35:30.734 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qm+SsO8XwO9SXjVIZFoLoF0u/VU= 2023-07-29T00:35:30.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qm+SsO8XwO9SXjVIZFoLoF0u/VU=', 'Content-Length': '0'} 2023-07-29T00:35:30.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c839e09afead9bac-0064c45ed2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.739 DEBUG:boto:b'hiiqos-59obj761000falseobj762023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:30.739 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:35:30.740 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:35:30.740 DEBUG:boto:Method: HEAD 2023-07-29T00:35:30.740 DEBUG:boto:Path: /hiiqos-59/obj76 2023-07-29T00:35:30.741 DEBUG:boto:Data: 2023-07-29T00:35:30.741 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.741 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:30.742 DEBUG:boto:Port: 8001 2023-07-29T00:35:30.742 DEBUG:boto:Params: {} 2023-07-29T00:35:30.742 DEBUG:boto:Token: None 2023-07-29T00:35:30.743 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj76 2023-07-29T00:35:30.743 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5wDO7FbTQeIyX5UxxKmsaG8ySC8= 2023-07-29T00:35:30.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5wDO7FbTQeIyX5UxxKmsaG8ySC8=', 'Content-Length': '0'} 2023-07-29T00:35:30.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecb69fba608a605a-0064c45ed2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.746 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:35:30.746 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:35:30.747 DEBUG:boto:Method: HEAD 2023-07-29T00:35:30.747 DEBUG:boto:Path: /hiiqos-59/obj76 2023-07-29T00:35:30.747 DEBUG:boto:Data: 2023-07-29T00:35:30.748 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.748 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:30.748 DEBUG:boto:Port: 8000 2023-07-29T00:35:30.749 DEBUG:boto:Params: {} 2023-07-29T00:35:30.749 DEBUG:boto:Token: None 2023-07-29T00:35:30.750 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj76 2023-07-29T00:35:30.750 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5wDO7FbTQeIyX5UxxKmsaG8ySC8= 2023-07-29T00:35:30.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5wDO7FbTQeIyX5UxxKmsaG8ySC8=', 'Content-Length': '0'} 2023-07-29T00:35:30.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b70d3e90b0ed054-0064c45ed2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-07-29T00:35:30.753 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:35:30.753 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:35:30.754 DEBUG:boto:Method: GET 2023-07-29T00:35:30.754 DEBUG:boto:Path: /hiiqos-59/obj76 2023-07-29T00:35:30.755 DEBUG:boto:Data: 2023-07-29T00:35:30.755 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.755 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:30.756 DEBUG:boto:Port: 8001 2023-07-29T00:35:30.756 DEBUG:boto:Params: {} 2023-07-29T00:35:30.756 DEBUG:boto:Token: None 2023-07-29T00:35:30.757 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj76 2023-07-29T00:35:30.757 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:feEbX7Mj4yJE+IpbaG8Y8sNmiAc= 2023-07-29T00:35:30.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:feEbX7Mj4yJE+IpbaG8Y8sNmiAc=', 'Content-Length': '0'} 2023-07-29T00:35:30.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2c1762aeb679219-0064c45ed2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.760 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:35:30.760 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:35:30.761 DEBUG:boto:Method: GET 2023-07-29T00:35:30.761 DEBUG:boto:Path: /hiiqos-59/obj76 2023-07-29T00:35:30.761 DEBUG:boto:Data: 2023-07-29T00:35:30.762 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.762 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:30.762 DEBUG:boto:Port: 8000 2023-07-29T00:35:30.763 DEBUG:boto:Params: {} 2023-07-29T00:35:30.763 DEBUG:boto:Token: None 2023-07-29T00:35:30.763 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj76 2023-07-29T00:35:30.764 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:feEbX7Mj4yJE+IpbaG8Y8sNmiAc= 2023-07-29T00:35:30.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:feEbX7Mj4yJE+IpbaG8Y8sNmiAc=', 'Content-Length': '0'} 2023-07-29T00:35:30.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8b5f67961bd2c75-0064c45ed2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.808 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:30.809 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:30.809 DEBUG:boto:path=/hiiqos-59/?prefix=obj76 2023-07-29T00:35:30.809 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj76 2023-07-29T00:35:30.810 DEBUG:boto:Method: GET 2023-07-29T00:35:30.810 DEBUG:boto:Path: /hiiqos-59/?prefix=obj76 2023-07-29T00:35:30.810 DEBUG:boto:Data: 2023-07-29T00:35:30.811 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.811 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:30.812 DEBUG:boto:Port: 8001 2023-07-29T00:35:30.812 DEBUG:boto:Params: {} 2023-07-29T00:35:30.812 DEBUG:boto:Token: None 2023-07-29T00:35:30.813 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/ 2023-07-29T00:35:30.813 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qm+SsO8XwO9SXjVIZFoLoF0u/VU= 2023-07-29T00:35:30.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qm+SsO8XwO9SXjVIZFoLoF0u/VU=', 'Content-Length': '0'} 2023-07-29T00:35:30.816 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d821e505d21bdc0-0064c45ed2-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.816 DEBUG:boto:b'hiiqos-59obj761000falseobj762023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:30.817 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:30.818 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:30.818 DEBUG:boto:path=/hiiqos-59/?prefix=obj76 2023-07-29T00:35:30.818 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj76 2023-07-29T00:35:30.819 DEBUG:boto:Method: GET 2023-07-29T00:35:30.819 DEBUG:boto:Path: /hiiqos-59/?prefix=obj76 2023-07-29T00:35:30.819 DEBUG:boto:Data: 2023-07-29T00:35:30.820 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.820 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:30.820 DEBUG:boto:Port: 8000 2023-07-29T00:35:30.821 DEBUG:boto:Params: {} 2023-07-29T00:35:30.821 DEBUG:boto:Token: None 2023-07-29T00:35:30.822 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/ 2023-07-29T00:35:30.822 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qm+SsO8XwO9SXjVIZFoLoF0u/VU= 2023-07-29T00:35:30.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qm+SsO8XwO9SXjVIZFoLoF0u/VU=', 'Content-Length': '0'} 2023-07-29T00:35:30.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fed6878ff165b728-0064c45ed2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.827 DEBUG:boto:b'hiiqos-59obj761000falseobj762023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:30.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-07-29T00:35:30.828 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:35:30.828 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:35:30.828 DEBUG:boto:path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:35:30.829 DEBUG:boto:auth_path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:35:30.829 DEBUG:boto:Method: GET 2023-07-29T00:35:30.829 DEBUG:boto:Path: /hiiqos-59/obj85?versionId=null 2023-07-29T00:35:30.830 DEBUG:boto:Data: 2023-07-29T00:35:30.830 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.830 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:30.831 DEBUG:boto:Port: 8001 2023-07-29T00:35:30.831 DEBUG:boto:Params: {} 2023-07-29T00:35:30.832 DEBUG:boto:Token: None 2023-07-29T00:35:30.832 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj85?versionId=null 2023-07-29T00:35:30.832 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1HAXfmEBv0FG/ciuXHAC/QujSms= 2023-07-29T00:35:30.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1HAXfmEBv0FG/ciuXHAC/QujSms=', 'Content-Length': '0'} 2023-07-29T00:35:30.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002259748eb4814338-0064c45ed2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.835 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:35:30.836 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:35:30.836 DEBUG:boto:path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:35:30.836 DEBUG:boto:auth_path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:35:30.837 DEBUG:boto:Method: GET 2023-07-29T00:35:30.837 DEBUG:boto:Path: /hiiqos-59/obj85?versionId=null 2023-07-29T00:35:30.838 DEBUG:boto:Data: 2023-07-29T00:35:30.838 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.838 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:30.839 DEBUG:boto:Port: 8000 2023-07-29T00:35:30.839 DEBUG:boto:Params: {} 2023-07-29T00:35:30.839 DEBUG:boto:Token: None 2023-07-29T00:35:30.840 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj85?versionId=null 2023-07-29T00:35:30.840 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1HAXfmEBv0FG/ciuXHAC/QujSms= 2023-07-29T00:35:30.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1HAXfmEBv0FG/ciuXHAC/QujSms=', 'Content-Length': '0'} 2023-07-29T00:35:30.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b7be4c077e057a3-0064c45ed2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.843 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:35:30.843 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:35:30.844 DEBUG:boto:path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:35:30.844 DEBUG:boto:auth_path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:35:30.844 DEBUG:boto:Method: HEAD 2023-07-29T00:35:30.845 DEBUG:boto:Path: /hiiqos-59/obj85?versionId=null 2023-07-29T00:35:30.845 DEBUG:boto:Data: 2023-07-29T00:35:30.845 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.846 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:30.846 DEBUG:boto:Port: 8001 2023-07-29T00:35:30.846 DEBUG:boto:Params: {} 2023-07-29T00:35:30.847 DEBUG:boto:Token: None 2023-07-29T00:35:30.847 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj85?versionId=null 2023-07-29T00:35:30.848 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5JSVIA9T3PB5GoxCY8YU+M0htzA= 2023-07-29T00:35:30.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5JSVIA9T3PB5GoxCY8YU+M0htzA=', 'Content-Length': '0'} 2023-07-29T00:35:30.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c6d70e0b86622d9-0064c45ed2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.850 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:35:30.851 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:35:30.851 DEBUG:boto:path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:35:30.851 DEBUG:boto:auth_path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:35:30.852 DEBUG:boto:Method: HEAD 2023-07-29T00:35:30.852 DEBUG:boto:Path: /hiiqos-59/obj85?versionId=null 2023-07-29T00:35:30.852 DEBUG:boto:Data: 2023-07-29T00:35:30.853 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.853 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:30.853 DEBUG:boto:Port: 8000 2023-07-29T00:35:30.854 DEBUG:boto:Params: {} 2023-07-29T00:35:30.854 DEBUG:boto:Token: None 2023-07-29T00:35:30.855 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj85?versionId=null 2023-07-29T00:35:30.855 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5JSVIA9T3PB5GoxCY8YU+M0htzA= 2023-07-29T00:35:30.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5JSVIA9T3PB5GoxCY8YU+M0htzA=', 'Content-Length': '0'} 2023-07-29T00:35:30.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008346d990b3d22777-0064c45ed2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-07-29T00:35:30.858 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:35:30.858 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:35:30.858 DEBUG:boto:path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:35:30.859 DEBUG:boto:auth_path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:35:30.859 DEBUG:boto:Method: GET 2023-07-29T00:35:30.860 DEBUG:boto:Path: /hiiqos-59/obj85?versionId=null 2023-07-29T00:35:30.860 DEBUG:boto:Data: 2023-07-29T00:35:30.860 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.861 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:30.861 DEBUG:boto:Port: 8001 2023-07-29T00:35:30.861 DEBUG:boto:Params: {} 2023-07-29T00:35:30.862 DEBUG:boto:Token: None 2023-07-29T00:35:30.862 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj85?versionId=null 2023-07-29T00:35:30.863 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1HAXfmEBv0FG/ciuXHAC/QujSms= 2023-07-29T00:35:30.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1HAXfmEBv0FG/ciuXHAC/QujSms=', 'Content-Length': '0'} 2023-07-29T00:35:30.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000765984dd04df4085-0064c45ed2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.865 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:35:30.866 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:35:30.866 DEBUG:boto:path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:35:30.866 DEBUG:boto:auth_path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:35:30.867 DEBUG:boto:Method: GET 2023-07-29T00:35:30.867 DEBUG:boto:Path: /hiiqos-59/obj85?versionId=null 2023-07-29T00:35:30.867 DEBUG:boto:Data: 2023-07-29T00:35:30.868 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.868 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:30.869 DEBUG:boto:Port: 8000 2023-07-29T00:35:30.869 DEBUG:boto:Params: {} 2023-07-29T00:35:30.869 DEBUG:boto:Token: None 2023-07-29T00:35:30.870 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj85?versionId=null 2023-07-29T00:35:30.870 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1HAXfmEBv0FG/ciuXHAC/QujSms= 2023-07-29T00:35:30.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1HAXfmEBv0FG/ciuXHAC/QujSms=', 'Content-Length': '0'} 2023-07-29T00:35:30.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f3c1c69e9eb2863-0064c45ed2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.916 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:30.917 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:30.917 DEBUG:boto:path=/hiiqos-59/?prefix=obj85 2023-07-29T00:35:30.917 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj85 2023-07-29T00:35:30.918 DEBUG:boto:Method: GET 2023-07-29T00:35:30.918 DEBUG:boto:Path: /hiiqos-59/?prefix=obj85 2023-07-29T00:35:30.918 DEBUG:boto:Data: 2023-07-29T00:35:30.919 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.919 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:30.919 DEBUG:boto:Port: 8001 2023-07-29T00:35:30.920 DEBUG:boto:Params: {} 2023-07-29T00:35:30.920 DEBUG:boto:Token: None 2023-07-29T00:35:30.921 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/ 2023-07-29T00:35:30.921 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qm+SsO8XwO9SXjVIZFoLoF0u/VU= 2023-07-29T00:35:30.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qm+SsO8XwO9SXjVIZFoLoF0u/VU=', 'Content-Length': '0'} 2023-07-29T00:35:30.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f50f40eca4a5975d-0064c45ed2-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.924 DEBUG:boto:b'hiiqos-59obj851000falseobj852023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:30.925 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:30.925 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:30.926 DEBUG:boto:path=/hiiqos-59/?prefix=obj85 2023-07-29T00:35:30.926 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj85 2023-07-29T00:35:30.926 DEBUG:boto:Method: GET 2023-07-29T00:35:30.927 DEBUG:boto:Path: /hiiqos-59/?prefix=obj85 2023-07-29T00:35:30.927 DEBUG:boto:Data: 2023-07-29T00:35:30.928 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.928 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:30.928 DEBUG:boto:Port: 8000 2023-07-29T00:35:30.929 DEBUG:boto:Params: {} 2023-07-29T00:35:30.929 DEBUG:boto:Token: None 2023-07-29T00:35:30.929 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/ 2023-07-29T00:35:30.930 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qm+SsO8XwO9SXjVIZFoLoF0u/VU= 2023-07-29T00:35:30.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qm+SsO8XwO9SXjVIZFoLoF0u/VU=', 'Content-Length': '0'} 2023-07-29T00:35:30.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ec93a723f14925c-0064c45ed2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.935 DEBUG:boto:b'hiiqos-59obj851000falseobj852023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:30.935 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:35:30.935 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:35:30.936 DEBUG:boto:Method: HEAD 2023-07-29T00:35:30.936 DEBUG:boto:Path: /hiiqos-59/obj85 2023-07-29T00:35:30.937 DEBUG:boto:Data: 2023-07-29T00:35:30.937 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.937 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:30.938 DEBUG:boto:Port: 8001 2023-07-29T00:35:30.938 DEBUG:boto:Params: {} 2023-07-29T00:35:30.938 DEBUG:boto:Token: None 2023-07-29T00:35:30.939 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj85 2023-07-29T00:35:30.939 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qU+lwjkjtX97CupdBOtRbPBBIMM= 2023-07-29T00:35:30.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qU+lwjkjtX97CupdBOtRbPBBIMM=', 'Content-Length': '0'} 2023-07-29T00:35:30.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000934c2a8f0dcddcf7-0064c45ed2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.942 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:35:30.942 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:35:30.942 DEBUG:boto:Method: HEAD 2023-07-29T00:35:30.943 DEBUG:boto:Path: /hiiqos-59/obj85 2023-07-29T00:35:30.943 DEBUG:boto:Data: 2023-07-29T00:35:30.943 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.944 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:30.944 DEBUG:boto:Port: 8000 2023-07-29T00:35:30.944 DEBUG:boto:Params: {} 2023-07-29T00:35:30.945 DEBUG:boto:Token: None 2023-07-29T00:35:30.945 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj85 2023-07-29T00:35:30.946 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qU+lwjkjtX97CupdBOtRbPBBIMM= 2023-07-29T00:35:30.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qU+lwjkjtX97CupdBOtRbPBBIMM=', 'Content-Length': '0'} 2023-07-29T00:35:30.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000688b5bd00845ad9a-0064c45ed2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-07-29T00:35:30.949 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:35:30.949 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:35:30.949 DEBUG:boto:Method: GET 2023-07-29T00:35:30.950 DEBUG:boto:Path: /hiiqos-59/obj85 2023-07-29T00:35:30.950 DEBUG:boto:Data: 2023-07-29T00:35:30.950 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.951 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:30.951 DEBUG:boto:Port: 8001 2023-07-29T00:35:30.951 DEBUG:boto:Params: {} 2023-07-29T00:35:30.952 DEBUG:boto:Token: None 2023-07-29T00:35:30.952 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:30 GMT /hiiqos-59/obj85 2023-07-29T00:35:30.953 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:k6jyUdjJt/RwfuJhh36KyZpUPuA= 2023-07-29T00:35:30.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:k6jyUdjJt/RwfuJhh36KyZpUPuA=', 'Content-Length': '0'} 2023-07-29T00:35:30.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000833527cc20bd3718-0064c45ed2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:30.996 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:35:30.997 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:35:30.997 DEBUG:boto:Method: GET 2023-07-29T00:35:30.997 DEBUG:boto:Path: /hiiqos-59/obj85 2023-07-29T00:35:30.998 DEBUG:boto:Data: 2023-07-29T00:35:30.998 DEBUG:boto:Headers: {} 2023-07-29T00:35:30.999 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:30.999 DEBUG:boto:Port: 8000 2023-07-29T00:35:30.999 DEBUG:boto:Params: {} 2023-07-29T00:35:31.000 DEBUG:boto:Token: None 2023-07-29T00:35:31.000 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/obj85 2023-07-29T00:35:31.001 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UqBjI1i4lSKnZ5+v8qd092Gy870= 2023-07-29T00:35:31.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UqBjI1i4lSKnZ5+v8qd092Gy870=', 'Content-Length': '0'} 2023-07-29T00:35:31.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c91007bfd29c9a94-0064c45ed2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.003 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:31.004 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:31.004 DEBUG:boto:path=/hiiqos-59/?prefix=obj85 2023-07-29T00:35:31.004 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj85 2023-07-29T00:35:31.005 DEBUG:boto:Method: GET 2023-07-29T00:35:31.005 DEBUG:boto:Path: /hiiqos-59/?prefix=obj85 2023-07-29T00:35:31.005 DEBUG:boto:Data: 2023-07-29T00:35:31.006 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.006 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:31.007 DEBUG:boto:Port: 8001 2023-07-29T00:35:31.007 DEBUG:boto:Params: {} 2023-07-29T00:35:31.007 DEBUG:boto:Token: None 2023-07-29T00:35:31.008 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/ 2023-07-29T00:35:31.008 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kHyDn0lT4eNZCdvQe0CQ0o0BiG0= 2023-07-29T00:35:31.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kHyDn0lT4eNZCdvQe0CQ0o0BiG0=', 'Content-Length': '0'} 2023-07-29T00:35:31.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed923d5f81b161ed-0064c45ed2-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.012 DEBUG:boto:b'hiiqos-59obj851000falseobj852023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:31.012 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:31.013 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:31.013 DEBUG:boto:path=/hiiqos-59/?prefix=obj85 2023-07-29T00:35:31.013 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj85 2023-07-29T00:35:31.014 DEBUG:boto:Method: GET 2023-07-29T00:35:31.014 DEBUG:boto:Path: /hiiqos-59/?prefix=obj85 2023-07-29T00:35:31.014 DEBUG:boto:Data: 2023-07-29T00:35:31.015 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.015 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:31.015 DEBUG:boto:Port: 8000 2023-07-29T00:35:31.016 DEBUG:boto:Params: {} 2023-07-29T00:35:31.016 DEBUG:boto:Token: None 2023-07-29T00:35:31.017 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/ 2023-07-29T00:35:31.017 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kHyDn0lT4eNZCdvQe0CQ0o0BiG0= 2023-07-29T00:35:31.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kHyDn0lT4eNZCdvQe0CQ0o0BiG0=', 'Content-Length': '0'} 2023-07-29T00:35:31.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e48207629761467f-0064c45ed2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.022 DEBUG:boto:b'hiiqos-59obj851000falseobj852023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:31.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-07-29T00:35:31.023 DEBUG:boto:path=/hiiqos-59/obj87 2023-07-29T00:35:31.023 DEBUG:boto:auth_path=/hiiqos-59/obj87 2023-07-29T00:35:31.024 DEBUG:boto:path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:35:31.024 DEBUG:boto:auth_path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:35:31.024 DEBUG:boto:Method: GET 2023-07-29T00:35:31.025 DEBUG:boto:Path: /hiiqos-59/obj87?versionId=null 2023-07-29T00:35:31.025 DEBUG:boto:Data: 2023-07-29T00:35:31.025 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.026 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:31.026 DEBUG:boto:Port: 8001 2023-07-29T00:35:31.026 DEBUG:boto:Params: {} 2023-07-29T00:35:31.027 DEBUG:boto:Token: None 2023-07-29T00:35:31.027 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/obj87?versionId=null 2023-07-29T00:35:31.028 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pfUnwlLii/y4Zw/+Di8IUlrnaE4= 2023-07-29T00:35:31.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pfUnwlLii/y4Zw/+Di8IUlrnaE4=', 'Content-Length': '0'} 2023-07-29T00:35:31.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cf59b388d6d68d5-0064c45ed2-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.031 DEBUG:boto:path=/hiiqos-59/obj87 2023-07-29T00:35:31.031 DEBUG:boto:auth_path=/hiiqos-59/obj87 2023-07-29T00:35:31.032 DEBUG:boto:path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:35:31.032 DEBUG:boto:auth_path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:35:31.032 DEBUG:boto:Method: GET 2023-07-29T00:35:31.033 DEBUG:boto:Path: /hiiqos-59/obj87?versionId=null 2023-07-29T00:35:31.033 DEBUG:boto:Data: 2023-07-29T00:35:31.033 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.034 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:31.034 DEBUG:boto:Port: 8000 2023-07-29T00:35:31.034 DEBUG:boto:Params: {} 2023-07-29T00:35:31.035 DEBUG:boto:Token: None 2023-07-29T00:35:31.035 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/obj87?versionId=null 2023-07-29T00:35:31.036 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pfUnwlLii/y4Zw/+Di8IUlrnaE4= 2023-07-29T00:35:31.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pfUnwlLii/y4Zw/+Di8IUlrnaE4=', 'Content-Length': '0'} 2023-07-29T00:35:31.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a347c8ae6e73c0f7-0064c45ed2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:30 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.080 DEBUG:boto:path=/hiiqos-59/obj87 2023-07-29T00:35:31.081 DEBUG:boto:auth_path=/hiiqos-59/obj87 2023-07-29T00:35:31.081 DEBUG:boto:path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:35:31.081 DEBUG:boto:auth_path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:35:31.082 DEBUG:boto:Method: HEAD 2023-07-29T00:35:31.082 DEBUG:boto:Path: /hiiqos-59/obj87?versionId=null 2023-07-29T00:35:31.082 DEBUG:boto:Data: 2023-07-29T00:35:31.083 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.083 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:31.084 DEBUG:boto:Port: 8001 2023-07-29T00:35:31.084 DEBUG:boto:Params: {} 2023-07-29T00:35:31.084 DEBUG:boto:Token: None 2023-07-29T00:35:31.085 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/obj87?versionId=null 2023-07-29T00:35:31.085 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:R0ZDmwlOkqpByGhEBWvrYPUdDWo= 2023-07-29T00:35:31.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:R0ZDmwlOkqpByGhEBWvrYPUdDWo=', 'Content-Length': '0'} 2023-07-29T00:35:31.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8beea3e99107c41-0064c45ed3-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.089 DEBUG:boto:path=/hiiqos-59/obj87 2023-07-29T00:35:31.090 DEBUG:boto:auth_path=/hiiqos-59/obj87 2023-07-29T00:35:31.091 DEBUG:boto:path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:35:31.091 DEBUG:boto:auth_path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:35:31.092 DEBUG:boto:Method: HEAD 2023-07-29T00:35:31.092 DEBUG:boto:Path: /hiiqos-59/obj87?versionId=null 2023-07-29T00:35:31.093 DEBUG:boto:Data: 2023-07-29T00:35:31.094 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.094 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:31.095 DEBUG:boto:Port: 8000 2023-07-29T00:35:31.096 DEBUG:boto:Params: {} 2023-07-29T00:35:31.096 DEBUG:boto:Token: None 2023-07-29T00:35:31.097 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/obj87?versionId=null 2023-07-29T00:35:31.098 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:R0ZDmwlOkqpByGhEBWvrYPUdDWo= 2023-07-29T00:35:31.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:R0ZDmwlOkqpByGhEBWvrYPUdDWo=', 'Content-Length': '0'} 2023-07-29T00:35:31.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8237b4ea25e8007-0064c45ed3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-07-29T00:35:31.104 DEBUG:boto:path=/hiiqos-59/obj87 2023-07-29T00:35:31.105 DEBUG:boto:auth_path=/hiiqos-59/obj87 2023-07-29T00:35:31.105 DEBUG:boto:path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:35:31.105 DEBUG:boto:auth_path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:35:31.106 DEBUG:boto:Method: GET 2023-07-29T00:35:31.106 DEBUG:boto:Path: /hiiqos-59/obj87?versionId=null 2023-07-29T00:35:31.106 DEBUG:boto:Data: 2023-07-29T00:35:31.107 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.107 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:31.108 DEBUG:boto:Port: 8001 2023-07-29T00:35:31.108 DEBUG:boto:Params: {} 2023-07-29T00:35:31.108 DEBUG:boto:Token: None 2023-07-29T00:35:31.109 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/obj87?versionId=null 2023-07-29T00:35:31.109 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pfUnwlLii/y4Zw/+Di8IUlrnaE4= 2023-07-29T00:35:31.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pfUnwlLii/y4Zw/+Di8IUlrnaE4=', 'Content-Length': '0'} 2023-07-29T00:35:31.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cab1ae6983bc3e5-0064c45ed3-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.115 DEBUG:boto:path=/hiiqos-59/obj87 2023-07-29T00:35:31.116 DEBUG:boto:auth_path=/hiiqos-59/obj87 2023-07-29T00:35:31.116 DEBUG:boto:path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:35:31.116 DEBUG:boto:auth_path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:35:31.117 DEBUG:boto:Method: GET 2023-07-29T00:35:31.117 DEBUG:boto:Path: /hiiqos-59/obj87?versionId=null 2023-07-29T00:35:31.118 DEBUG:boto:Data: 2023-07-29T00:35:31.118 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.118 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:31.119 DEBUG:boto:Port: 8000 2023-07-29T00:35:31.119 DEBUG:boto:Params: {} 2023-07-29T00:35:31.120 DEBUG:boto:Token: None 2023-07-29T00:35:31.120 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/obj87?versionId=null 2023-07-29T00:35:31.121 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pfUnwlLii/y4Zw/+Di8IUlrnaE4= 2023-07-29T00:35:31.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pfUnwlLii/y4Zw/+Di8IUlrnaE4=', 'Content-Length': '0'} 2023-07-29T00:35:31.123 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018a8330732bbf19c-0064c45ed3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.124 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:31.124 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:31.125 DEBUG:boto:path=/hiiqos-59/?prefix=obj87 2023-07-29T00:35:31.125 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj87 2023-07-29T00:35:31.125 DEBUG:boto:Method: GET 2023-07-29T00:35:31.126 DEBUG:boto:Path: /hiiqos-59/?prefix=obj87 2023-07-29T00:35:31.126 DEBUG:boto:Data: 2023-07-29T00:35:31.127 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.127 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:31.127 DEBUG:boto:Port: 8001 2023-07-29T00:35:31.128 DEBUG:boto:Params: {} 2023-07-29T00:35:31.128 DEBUG:boto:Token: None 2023-07-29T00:35:31.129 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/ 2023-07-29T00:35:31.129 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kHyDn0lT4eNZCdvQe0CQ0o0BiG0= 2023-07-29T00:35:31.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kHyDn0lT4eNZCdvQe0CQ0o0BiG0=', 'Content-Length': '0'} 2023-07-29T00:35:31.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9da8d810f98b61d-0064c45ed3-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.133 DEBUG:boto:b'hiiqos-59obj871000falseobj872023-07-29T00:35:01.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:31.134 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:31.135 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:31.135 DEBUG:boto:path=/hiiqos-59/?prefix=obj87 2023-07-29T00:35:31.135 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj87 2023-07-29T00:35:31.136 DEBUG:boto:Method: GET 2023-07-29T00:35:31.136 DEBUG:boto:Path: /hiiqos-59/?prefix=obj87 2023-07-29T00:35:31.137 DEBUG:boto:Data: 2023-07-29T00:35:31.137 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.137 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:31.138 DEBUG:boto:Port: 8000 2023-07-29T00:35:31.138 DEBUG:boto:Params: {} 2023-07-29T00:35:31.139 DEBUG:boto:Token: None 2023-07-29T00:35:31.139 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/ 2023-07-29T00:35:31.140 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kHyDn0lT4eNZCdvQe0CQ0o0BiG0= 2023-07-29T00:35:31.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kHyDn0lT4eNZCdvQe0CQ0o0BiG0=', 'Content-Length': '0'} 2023-07-29T00:35:31.144 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d80e95ac6ad1f778-0064c45ed3-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.144 DEBUG:boto:b'hiiqos-59obj871000falseobj872023-07-29T00:35:01.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:31.145 DEBUG:boto:path=/hiiqos-59/obj87 2023-07-29T00:35:31.145 DEBUG:boto:auth_path=/hiiqos-59/obj87 2023-07-29T00:35:31.146 DEBUG:boto:Method: HEAD 2023-07-29T00:35:31.146 DEBUG:boto:Path: /hiiqos-59/obj87 2023-07-29T00:35:31.147 DEBUG:boto:Data: 2023-07-29T00:35:31.147 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.147 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:31.148 DEBUG:boto:Port: 8001 2023-07-29T00:35:31.148 DEBUG:boto:Params: {} 2023-07-29T00:35:31.149 DEBUG:boto:Token: None 2023-07-29T00:35:31.149 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/obj87 2023-07-29T00:35:31.149 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4EY1bgX/dAJKtTRZZi4uadd02kI= 2023-07-29T00:35:31.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4EY1bgX/dAJKtTRZZi4uadd02kI=', 'Content-Length': '0'} 2023-07-29T00:35:31.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a66e8f08af42cd0-0064c45ed3-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.154 DEBUG:boto:path=/hiiqos-59/obj87 2023-07-29T00:35:31.154 DEBUG:boto:auth_path=/hiiqos-59/obj87 2023-07-29T00:35:31.155 DEBUG:boto:Method: HEAD 2023-07-29T00:35:31.155 DEBUG:boto:Path: /hiiqos-59/obj87 2023-07-29T00:35:31.155 DEBUG:boto:Data: 2023-07-29T00:35:31.156 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.156 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:31.157 DEBUG:boto:Port: 8000 2023-07-29T00:35:31.157 DEBUG:boto:Params: {} 2023-07-29T00:35:31.158 DEBUG:boto:Token: None 2023-07-29T00:35:31.158 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/obj87 2023-07-29T00:35:31.158 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4EY1bgX/dAJKtTRZZi4uadd02kI= 2023-07-29T00:35:31.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4EY1bgX/dAJKtTRZZi4uadd02kI=', 'Content-Length': '0'} 2023-07-29T00:35:31.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b46f5bb3241fe102-0064c45ed3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-07-29T00:35:31.163 DEBUG:boto:path=/hiiqos-59/obj87 2023-07-29T00:35:31.164 DEBUG:boto:auth_path=/hiiqos-59/obj87 2023-07-29T00:35:31.165 DEBUG:boto:Method: GET 2023-07-29T00:35:31.165 DEBUG:boto:Path: /hiiqos-59/obj87 2023-07-29T00:35:31.166 DEBUG:boto:Data: 2023-07-29T00:35:31.166 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.167 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:31.167 DEBUG:boto:Port: 8001 2023-07-29T00:35:31.168 DEBUG:boto:Params: {} 2023-07-29T00:35:31.168 DEBUG:boto:Token: None 2023-07-29T00:35:31.169 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/obj87 2023-07-29T00:35:31.169 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZE95h5B1YKa60D4gt78YgoR1TYU= 2023-07-29T00:35:31.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZE95h5B1YKa60D4gt78YgoR1TYU=', 'Content-Length': '0'} 2023-07-29T00:35:31.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030000b6ee5ddcf9e-0064c45ed3-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.212 DEBUG:boto:path=/hiiqos-59/obj87 2023-07-29T00:35:31.213 DEBUG:boto:auth_path=/hiiqos-59/obj87 2023-07-29T00:35:31.213 DEBUG:boto:Method: GET 2023-07-29T00:35:31.214 DEBUG:boto:Path: /hiiqos-59/obj87 2023-07-29T00:35:31.214 DEBUG:boto:Data: 2023-07-29T00:35:31.215 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.215 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:31.216 DEBUG:boto:Port: 8000 2023-07-29T00:35:31.216 DEBUG:boto:Params: {} 2023-07-29T00:35:31.217 DEBUG:boto:Token: None 2023-07-29T00:35:31.218 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/obj87 2023-07-29T00:35:31.218 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZE95h5B1YKa60D4gt78YgoR1TYU= 2023-07-29T00:35:31.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZE95h5B1YKa60D4gt78YgoR1TYU=', 'Content-Length': '0'} 2023-07-29T00:35:31.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089891f746cbe0dcb-0064c45ed3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.223 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:31.223 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:31.224 DEBUG:boto:path=/hiiqos-59/?prefix=obj87 2023-07-29T00:35:31.224 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj87 2023-07-29T00:35:31.224 DEBUG:boto:Method: GET 2023-07-29T00:35:31.225 DEBUG:boto:Path: /hiiqos-59/?prefix=obj87 2023-07-29T00:35:31.225 DEBUG:boto:Data: 2023-07-29T00:35:31.226 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.226 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:31.226 DEBUG:boto:Port: 8001 2023-07-29T00:35:31.227 DEBUG:boto:Params: {} 2023-07-29T00:35:31.227 DEBUG:boto:Token: None 2023-07-29T00:35:31.227 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/ 2023-07-29T00:35:31.228 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kHyDn0lT4eNZCdvQe0CQ0o0BiG0= 2023-07-29T00:35:31.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kHyDn0lT4eNZCdvQe0CQ0o0BiG0=', 'Content-Length': '0'} 2023-07-29T00:35:31.231 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ccc618a28fed332-0064c45ed3-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.232 DEBUG:boto:b'hiiqos-59obj871000falseobj872023-07-29T00:35:01.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:31.232 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:31.233 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:31.233 DEBUG:boto:path=/hiiqos-59/?prefix=obj87 2023-07-29T00:35:31.234 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj87 2023-07-29T00:35:31.234 DEBUG:boto:Method: GET 2023-07-29T00:35:31.234 DEBUG:boto:Path: /hiiqos-59/?prefix=obj87 2023-07-29T00:35:31.235 DEBUG:boto:Data: 2023-07-29T00:35:31.235 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.235 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:31.236 DEBUG:boto:Port: 8000 2023-07-29T00:35:31.236 DEBUG:boto:Params: {} 2023-07-29T00:35:31.237 DEBUG:boto:Token: None 2023-07-29T00:35:31.237 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/ 2023-07-29T00:35:31.237 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kHyDn0lT4eNZCdvQe0CQ0o0BiG0= 2023-07-29T00:35:31.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kHyDn0lT4eNZCdvQe0CQ0o0BiG0=', 'Content-Length': '0'} 2023-07-29T00:35:31.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004c4259e3ff30035-0064c45ed3-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.243 DEBUG:boto:b'hiiqos-59obj871000falseobj872023-07-29T00:35:01.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:31.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-07-29T00:35:31.244 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:35:31.245 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:35:31.245 DEBUG:boto:path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:35:31.245 DEBUG:boto:auth_path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:35:31.246 DEBUG:boto:Method: GET 2023-07-29T00:35:31.246 DEBUG:boto:Path: /hiiqos-59/obj92?versionId=null 2023-07-29T00:35:31.246 DEBUG:boto:Data: 2023-07-29T00:35:31.247 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.247 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:31.248 DEBUG:boto:Port: 8001 2023-07-29T00:35:31.248 DEBUG:boto:Params: {} 2023-07-29T00:35:31.248 DEBUG:boto:Token: None 2023-07-29T00:35:31.249 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/obj92?versionId=null 2023-07-29T00:35:31.249 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XmiNpyjUHFBYQyHZlvyoKt/Tkgo= 2023-07-29T00:35:31.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XmiNpyjUHFBYQyHZlvyoKt/Tkgo=', 'Content-Length': '0'} 2023-07-29T00:35:31.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff79ec07bd0a2a80-0064c45ed3-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.253 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:35:31.253 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:35:31.253 DEBUG:boto:path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:35:31.254 DEBUG:boto:auth_path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:35:31.254 DEBUG:boto:Method: GET 2023-07-29T00:35:31.255 DEBUG:boto:Path: /hiiqos-59/obj92?versionId=null 2023-07-29T00:35:31.255 DEBUG:boto:Data: 2023-07-29T00:35:31.256 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.256 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:31.256 DEBUG:boto:Port: 8000 2023-07-29T00:35:31.257 DEBUG:boto:Params: {} 2023-07-29T00:35:31.257 DEBUG:boto:Token: None 2023-07-29T00:35:31.258 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/obj92?versionId=null 2023-07-29T00:35:31.258 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XmiNpyjUHFBYQyHZlvyoKt/Tkgo= 2023-07-29T00:35:31.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XmiNpyjUHFBYQyHZlvyoKt/Tkgo=', 'Content-Length': '0'} 2023-07-29T00:35:31.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001944f82498eac39e-0064c45ed3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.262 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:35:31.262 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:35:31.263 DEBUG:boto:path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:35:31.263 DEBUG:boto:auth_path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:35:31.264 DEBUG:boto:Method: HEAD 2023-07-29T00:35:31.264 DEBUG:boto:Path: /hiiqos-59/obj92?versionId=null 2023-07-29T00:35:31.265 DEBUG:boto:Data: 2023-07-29T00:35:31.265 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.265 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:31.266 DEBUG:boto:Port: 8001 2023-07-29T00:35:31.266 DEBUG:boto:Params: {} 2023-07-29T00:35:31.267 DEBUG:boto:Token: None 2023-07-29T00:35:31.267 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/obj92?versionId=null 2023-07-29T00:35:31.268 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P8vS7IyKdyKBTJo4fWN0iN8DNno= 2023-07-29T00:35:31.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P8vS7IyKdyKBTJo4fWN0iN8DNno=', 'Content-Length': '0'} 2023-07-29T00:35:31.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a77b270e6e986eac-0064c45ed3-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.271 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:35:31.272 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:35:31.272 DEBUG:boto:path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:35:31.272 DEBUG:boto:auth_path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:35:31.273 DEBUG:boto:Method: HEAD 2023-07-29T00:35:31.273 DEBUG:boto:Path: /hiiqos-59/obj92?versionId=null 2023-07-29T00:35:31.274 DEBUG:boto:Data: 2023-07-29T00:35:31.274 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.274 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:31.275 DEBUG:boto:Port: 8000 2023-07-29T00:35:31.275 DEBUG:boto:Params: {} 2023-07-29T00:35:31.276 DEBUG:boto:Token: None 2023-07-29T00:35:31.276 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/obj92?versionId=null 2023-07-29T00:35:31.276 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P8vS7IyKdyKBTJo4fWN0iN8DNno= 2023-07-29T00:35:31.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P8vS7IyKdyKBTJo4fWN0iN8DNno=', 'Content-Length': '0'} 2023-07-29T00:35:31.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000375e7f3c5c6d7914-0064c45ed3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-07-29T00:35:31.280 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:35:31.280 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:35:31.281 DEBUG:boto:path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:35:31.281 DEBUG:boto:auth_path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:35:31.281 DEBUG:boto:Method: GET 2023-07-29T00:35:31.282 DEBUG:boto:Path: /hiiqos-59/obj92?versionId=null 2023-07-29T00:35:31.282 DEBUG:boto:Data: 2023-07-29T00:35:31.283 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.283 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:31.283 DEBUG:boto:Port: 8001 2023-07-29T00:35:31.284 DEBUG:boto:Params: {} 2023-07-29T00:35:31.284 DEBUG:boto:Token: None 2023-07-29T00:35:31.285 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/obj92?versionId=null 2023-07-29T00:35:31.285 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XmiNpyjUHFBYQyHZlvyoKt/Tkgo= 2023-07-29T00:35:31.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XmiNpyjUHFBYQyHZlvyoKt/Tkgo=', 'Content-Length': '0'} 2023-07-29T00:35:31.288 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbe6fe4604e62029-0064c45ed3-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.288 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:35:31.289 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:35:31.289 DEBUG:boto:path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:35:31.289 DEBUG:boto:auth_path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:35:31.290 DEBUG:boto:Method: GET 2023-07-29T00:35:31.290 DEBUG:boto:Path: /hiiqos-59/obj92?versionId=null 2023-07-29T00:35:31.290 DEBUG:boto:Data: 2023-07-29T00:35:31.291 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.291 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:31.291 DEBUG:boto:Port: 8000 2023-07-29T00:35:31.292 DEBUG:boto:Params: {} 2023-07-29T00:35:31.292 DEBUG:boto:Token: None 2023-07-29T00:35:31.293 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/obj92?versionId=null 2023-07-29T00:35:31.293 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XmiNpyjUHFBYQyHZlvyoKt/Tkgo= 2023-07-29T00:35:31.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XmiNpyjUHFBYQyHZlvyoKt/Tkgo=', 'Content-Length': '0'} 2023-07-29T00:35:31.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec527432014ef0fe-0064c45ed3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.336 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:31.337 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:31.337 DEBUG:boto:path=/hiiqos-59/?prefix=obj92 2023-07-29T00:35:31.338 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj92 2023-07-29T00:35:31.338 DEBUG:boto:Method: GET 2023-07-29T00:35:31.339 DEBUG:boto:Path: /hiiqos-59/?prefix=obj92 2023-07-29T00:35:31.339 DEBUG:boto:Data: 2023-07-29T00:35:31.339 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.340 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:31.340 DEBUG:boto:Port: 8001 2023-07-29T00:35:31.341 DEBUG:boto:Params: {} 2023-07-29T00:35:31.341 DEBUG:boto:Token: None 2023-07-29T00:35:31.341 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/ 2023-07-29T00:35:31.342 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kHyDn0lT4eNZCdvQe0CQ0o0BiG0= 2023-07-29T00:35:31.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kHyDn0lT4eNZCdvQe0CQ0o0BiG0=', 'Content-Length': '0'} 2023-07-29T00:35:31.347 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004aad0e45a45503c8-0064c45ed3-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.348 DEBUG:boto:b'hiiqos-59obj921000falseobj922023-07-29T00:33:06.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:31.349 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:31.349 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:31.350 DEBUG:boto:path=/hiiqos-59/?prefix=obj92 2023-07-29T00:35:31.350 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj92 2023-07-29T00:35:31.350 DEBUG:boto:Method: GET 2023-07-29T00:35:31.351 DEBUG:boto:Path: /hiiqos-59/?prefix=obj92 2023-07-29T00:35:31.351 DEBUG:boto:Data: 2023-07-29T00:35:31.351 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.352 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:31.352 DEBUG:boto:Port: 8000 2023-07-29T00:35:31.352 DEBUG:boto:Params: {} 2023-07-29T00:35:31.353 DEBUG:boto:Token: None 2023-07-29T00:35:31.354 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/ 2023-07-29T00:35:31.354 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kHyDn0lT4eNZCdvQe0CQ0o0BiG0= 2023-07-29T00:35:31.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kHyDn0lT4eNZCdvQe0CQ0o0BiG0=', 'Content-Length': '0'} 2023-07-29T00:35:31.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1120b5c4b0a0eff-0064c45ed3-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.359 DEBUG:boto:b'hiiqos-59obj921000falseobj922023-07-29T00:33:06.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:31.359 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:35:31.360 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:35:31.360 DEBUG:boto:Method: HEAD 2023-07-29T00:35:31.361 DEBUG:boto:Path: /hiiqos-59/obj92 2023-07-29T00:35:31.361 DEBUG:boto:Data: 2023-07-29T00:35:31.361 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.362 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:31.362 DEBUG:boto:Port: 8001 2023-07-29T00:35:31.362 DEBUG:boto:Params: {} 2023-07-29T00:35:31.363 DEBUG:boto:Token: None 2023-07-29T00:35:31.363 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/obj92 2023-07-29T00:35:31.363 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:k7TQpCZLMp9xQWPVyqhfwKrvM5M= 2023-07-29T00:35:31.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:k7TQpCZLMp9xQWPVyqhfwKrvM5M=', 'Content-Length': '0'} 2023-07-29T00:35:31.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000441f6e4846860548-0064c45ed3-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.367 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:35:31.367 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:35:31.367 DEBUG:boto:Method: HEAD 2023-07-29T00:35:31.368 DEBUG:boto:Path: /hiiqos-59/obj92 2023-07-29T00:35:31.368 DEBUG:boto:Data: 2023-07-29T00:35:31.369 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.369 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:31.369 DEBUG:boto:Port: 8000 2023-07-29T00:35:31.370 DEBUG:boto:Params: {} 2023-07-29T00:35:31.370 DEBUG:boto:Token: None 2023-07-29T00:35:31.370 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/obj92 2023-07-29T00:35:31.371 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:k7TQpCZLMp9xQWPVyqhfwKrvM5M= 2023-07-29T00:35:31.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:k7TQpCZLMp9xQWPVyqhfwKrvM5M=', 'Content-Length': '0'} 2023-07-29T00:35:31.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7d9499a957cc40d-0064c45ed3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-07-29T00:35:31.375 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:35:31.375 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:35:31.375 DEBUG:boto:Method: GET 2023-07-29T00:35:31.376 DEBUG:boto:Path: /hiiqos-59/obj92 2023-07-29T00:35:31.376 DEBUG:boto:Data: 2023-07-29T00:35:31.377 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.377 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:31.377 DEBUG:boto:Port: 8001 2023-07-29T00:35:31.378 DEBUG:boto:Params: {} 2023-07-29T00:35:31.378 DEBUG:boto:Token: None 2023-07-29T00:35:31.378 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/obj92 2023-07-29T00:35:31.379 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ftIxwsDPfsit5WX/Q11dRHpXfhw= 2023-07-29T00:35:31.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ftIxwsDPfsit5WX/Q11dRHpXfhw=', 'Content-Length': '0'} 2023-07-29T00:35:31.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ab83a079ef34bc8-0064c45ed3-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.424 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:35:31.425 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:35:31.425 DEBUG:boto:Method: GET 2023-07-29T00:35:31.426 DEBUG:boto:Path: /hiiqos-59/obj92 2023-07-29T00:35:31.426 DEBUG:boto:Data: 2023-07-29T00:35:31.427 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.427 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:31.427 DEBUG:boto:Port: 8000 2023-07-29T00:35:31.428 DEBUG:boto:Params: {} 2023-07-29T00:35:31.428 DEBUG:boto:Token: None 2023-07-29T00:35:31.428 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/obj92 2023-07-29T00:35:31.429 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ftIxwsDPfsit5WX/Q11dRHpXfhw= 2023-07-29T00:35:31.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ftIxwsDPfsit5WX/Q11dRHpXfhw=', 'Content-Length': '0'} 2023-07-29T00:35:31.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088cbceee55cff2a7-0064c45ed3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.432 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:31.432 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:31.433 DEBUG:boto:path=/hiiqos-59/?prefix=obj92 2023-07-29T00:35:31.433 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj92 2023-07-29T00:35:31.434 DEBUG:boto:Method: GET 2023-07-29T00:35:31.434 DEBUG:boto:Path: /hiiqos-59/?prefix=obj92 2023-07-29T00:35:31.434 DEBUG:boto:Data: 2023-07-29T00:35:31.435 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.435 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:31.435 DEBUG:boto:Port: 8001 2023-07-29T00:35:31.436 DEBUG:boto:Params: {} 2023-07-29T00:35:31.436 DEBUG:boto:Token: None 2023-07-29T00:35:31.437 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/ 2023-07-29T00:35:31.437 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kHyDn0lT4eNZCdvQe0CQ0o0BiG0= 2023-07-29T00:35:31.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kHyDn0lT4eNZCdvQe0CQ0o0BiG0=', 'Content-Length': '0'} 2023-07-29T00:35:31.440 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d8800b2a1ac2caf-0064c45ed3-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.441 DEBUG:boto:b'hiiqos-59obj921000falseobj922023-07-29T00:33:06.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:31.442 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:31.442 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:31.442 DEBUG:boto:path=/hiiqos-59/?prefix=obj92 2023-07-29T00:35:31.443 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj92 2023-07-29T00:35:31.443 DEBUG:boto:Method: GET 2023-07-29T00:35:31.443 DEBUG:boto:Path: /hiiqos-59/?prefix=obj92 2023-07-29T00:35:31.444 DEBUG:boto:Data: 2023-07-29T00:35:31.444 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.444 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:31.445 DEBUG:boto:Port: 8000 2023-07-29T00:35:31.445 DEBUG:boto:Params: {} 2023-07-29T00:35:31.446 DEBUG:boto:Token: None 2023-07-29T00:35:31.446 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/ 2023-07-29T00:35:31.446 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kHyDn0lT4eNZCdvQe0CQ0o0BiG0= 2023-07-29T00:35:31.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kHyDn0lT4eNZCdvQe0CQ0o0BiG0=', 'Content-Length': '0'} 2023-07-29T00:35:31.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5d449bcf11c7374-0064c45ed3-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.451 DEBUG:boto:b'hiiqos-59obj921000falseobj922023-07-29T00:33:06.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:31.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-07-29T00:35:31.452 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:35:31.453 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:35:31.453 DEBUG:boto:path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:35:31.453 DEBUG:boto:auth_path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:35:31.454 DEBUG:boto:Method: GET 2023-07-29T00:35:31.454 DEBUG:boto:Path: /hiiqos-59/obj95?versionId=null 2023-07-29T00:35:31.454 DEBUG:boto:Data: 2023-07-29T00:35:31.455 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.455 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:31.455 DEBUG:boto:Port: 8001 2023-07-29T00:35:31.456 DEBUG:boto:Params: {} 2023-07-29T00:35:31.456 DEBUG:boto:Token: None 2023-07-29T00:35:31.457 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/obj95?versionId=null 2023-07-29T00:35:31.457 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oqxjWDfgontyup7Jd2t0Z+h8bG8= 2023-07-29T00:35:31.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oqxjWDfgontyup7Jd2t0Z+h8bG8=', 'Content-Length': '0'} 2023-07-29T00:35:31.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ebe247c8fa49f72-0064c45ed3-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.460 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:35:31.461 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:35:31.461 DEBUG:boto:path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:35:31.461 DEBUG:boto:auth_path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:35:31.462 DEBUG:boto:Method: GET 2023-07-29T00:35:31.462 DEBUG:boto:Path: /hiiqos-59/obj95?versionId=null 2023-07-29T00:35:31.462 DEBUG:boto:Data: 2023-07-29T00:35:31.463 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.463 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:31.464 DEBUG:boto:Port: 8000 2023-07-29T00:35:31.464 DEBUG:boto:Params: {} 2023-07-29T00:35:31.464 DEBUG:boto:Token: None 2023-07-29T00:35:31.465 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/obj95?versionId=null 2023-07-29T00:35:31.465 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oqxjWDfgontyup7Jd2t0Z+h8bG8= 2023-07-29T00:35:31.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oqxjWDfgontyup7Jd2t0Z+h8bG8=', 'Content-Length': '0'} 2023-07-29T00:35:31.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ac0f8a75b07f0c9-0064c45ed3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.512 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:35:31.513 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:35:31.513 DEBUG:boto:path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:35:31.514 DEBUG:boto:auth_path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:35:31.514 DEBUG:boto:Method: HEAD 2023-07-29T00:35:31.514 DEBUG:boto:Path: /hiiqos-59/obj95?versionId=null 2023-07-29T00:35:31.515 DEBUG:boto:Data: 2023-07-29T00:35:31.515 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.515 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:31.516 DEBUG:boto:Port: 8001 2023-07-29T00:35:31.516 DEBUG:boto:Params: {} 2023-07-29T00:35:31.516 DEBUG:boto:Token: None 2023-07-29T00:35:31.517 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/obj95?versionId=null 2023-07-29T00:35:31.517 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:G93dm0y0tnnvgJwy71JAg48GtAk= 2023-07-29T00:35:31.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:G93dm0y0tnnvgJwy71JAg48GtAk=', 'Content-Length': '0'} 2023-07-29T00:35:31.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e97e836b3bebcef-0064c45ed3-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.520 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:35:31.521 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:35:31.521 DEBUG:boto:path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:35:31.521 DEBUG:boto:auth_path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:35:31.522 DEBUG:boto:Method: HEAD 2023-07-29T00:35:31.522 DEBUG:boto:Path: /hiiqos-59/obj95?versionId=null 2023-07-29T00:35:31.522 DEBUG:boto:Data: 2023-07-29T00:35:31.523 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.523 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:31.523 DEBUG:boto:Port: 8000 2023-07-29T00:35:31.524 DEBUG:boto:Params: {} 2023-07-29T00:35:31.524 DEBUG:boto:Token: None 2023-07-29T00:35:31.524 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/obj95?versionId=null 2023-07-29T00:35:31.525 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:G93dm0y0tnnvgJwy71JAg48GtAk= 2023-07-29T00:35:31.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:G93dm0y0tnnvgJwy71JAg48GtAk=', 'Content-Length': '0'} 2023-07-29T00:35:31.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000283207d73879f5ac-0064c45ed3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-07-29T00:35:31.528 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:35:31.529 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:35:31.529 DEBUG:boto:path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:35:31.529 DEBUG:boto:auth_path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:35:31.530 DEBUG:boto:Method: GET 2023-07-29T00:35:31.530 DEBUG:boto:Path: /hiiqos-59/obj95?versionId=null 2023-07-29T00:35:31.530 DEBUG:boto:Data: 2023-07-29T00:35:31.531 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.531 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:31.531 DEBUG:boto:Port: 8001 2023-07-29T00:35:31.532 DEBUG:boto:Params: {} 2023-07-29T00:35:31.532 DEBUG:boto:Token: None 2023-07-29T00:35:31.533 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/obj95?versionId=null 2023-07-29T00:35:31.533 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oqxjWDfgontyup7Jd2t0Z+h8bG8= 2023-07-29T00:35:31.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oqxjWDfgontyup7Jd2t0Z+h8bG8=', 'Content-Length': '0'} 2023-07-29T00:35:31.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008dc50ad54f82bf2d-0064c45ed3-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.536 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:35:31.536 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:35:31.537 DEBUG:boto:path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:35:31.537 DEBUG:boto:auth_path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:35:31.538 DEBUG:boto:Method: GET 2023-07-29T00:35:31.538 DEBUG:boto:Path: /hiiqos-59/obj95?versionId=null 2023-07-29T00:35:31.538 DEBUG:boto:Data: 2023-07-29T00:35:31.539 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.539 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:31.539 DEBUG:boto:Port: 8000 2023-07-29T00:35:31.540 DEBUG:boto:Params: {} 2023-07-29T00:35:31.540 DEBUG:boto:Token: None 2023-07-29T00:35:31.540 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/obj95?versionId=null 2023-07-29T00:35:31.541 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oqxjWDfgontyup7Jd2t0Z+h8bG8= 2023-07-29T00:35:31.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oqxjWDfgontyup7Jd2t0Z+h8bG8=', 'Content-Length': '0'} 2023-07-29T00:35:31.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043ee1c807b6b4aeb-0064c45ed3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.544 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:31.544 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:31.544 DEBUG:boto:path=/hiiqos-59/?prefix=obj95 2023-07-29T00:35:31.545 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj95 2023-07-29T00:35:31.545 DEBUG:boto:Method: GET 2023-07-29T00:35:31.546 DEBUG:boto:Path: /hiiqos-59/?prefix=obj95 2023-07-29T00:35:31.546 DEBUG:boto:Data: 2023-07-29T00:35:31.546 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.547 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:31.547 DEBUG:boto:Port: 8001 2023-07-29T00:35:31.547 DEBUG:boto:Params: {} 2023-07-29T00:35:31.548 DEBUG:boto:Token: None 2023-07-29T00:35:31.548 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/ 2023-07-29T00:35:31.549 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kHyDn0lT4eNZCdvQe0CQ0o0BiG0= 2023-07-29T00:35:31.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kHyDn0lT4eNZCdvQe0CQ0o0BiG0=', 'Content-Length': '0'} 2023-07-29T00:35:31.551 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012d376c926e9860f-0064c45ed3-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.552 DEBUG:boto:b'hiiqos-59obj951000falseobj952023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:31.553 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:31.553 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:31.553 DEBUG:boto:path=/hiiqos-59/?prefix=obj95 2023-07-29T00:35:31.554 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj95 2023-07-29T00:35:31.554 DEBUG:boto:Method: GET 2023-07-29T00:35:31.554 DEBUG:boto:Path: /hiiqos-59/?prefix=obj95 2023-07-29T00:35:31.555 DEBUG:boto:Data: 2023-07-29T00:35:31.555 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.555 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:31.556 DEBUG:boto:Port: 8000 2023-07-29T00:35:31.556 DEBUG:boto:Params: {} 2023-07-29T00:35:31.557 DEBUG:boto:Token: None 2023-07-29T00:35:31.557 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/ 2023-07-29T00:35:31.557 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kHyDn0lT4eNZCdvQe0CQ0o0BiG0= 2023-07-29T00:35:31.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kHyDn0lT4eNZCdvQe0CQ0o0BiG0=', 'Content-Length': '0'} 2023-07-29T00:35:31.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004393be1ba2fe525c-0064c45ed3-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.562 DEBUG:boto:b'hiiqos-59obj951000falseobj952023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:31.563 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:35:31.563 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:35:31.563 DEBUG:boto:Method: HEAD 2023-07-29T00:35:31.564 DEBUG:boto:Path: /hiiqos-59/obj95 2023-07-29T00:35:31.564 DEBUG:boto:Data: 2023-07-29T00:35:31.565 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.565 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:31.565 DEBUG:boto:Port: 8001 2023-07-29T00:35:31.566 DEBUG:boto:Params: {} 2023-07-29T00:35:31.566 DEBUG:boto:Token: None 2023-07-29T00:35:31.566 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/obj95 2023-07-29T00:35:31.567 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YBopUkhzxzUHNlfvqNm6yDqBCI8= 2023-07-29T00:35:31.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YBopUkhzxzUHNlfvqNm6yDqBCI8=', 'Content-Length': '0'} 2023-07-29T00:35:31.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1e67f22dfb37361-0064c45ed3-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.569 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:35:31.570 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:35:31.570 DEBUG:boto:Method: HEAD 2023-07-29T00:35:31.570 DEBUG:boto:Path: /hiiqos-59/obj95 2023-07-29T00:35:31.571 DEBUG:boto:Data: 2023-07-29T00:35:31.571 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.571 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:31.572 DEBUG:boto:Port: 8000 2023-07-29T00:35:31.572 DEBUG:boto:Params: {} 2023-07-29T00:35:31.573 DEBUG:boto:Token: None 2023-07-29T00:35:31.573 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/obj95 2023-07-29T00:35:31.573 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YBopUkhzxzUHNlfvqNm6yDqBCI8= 2023-07-29T00:35:31.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YBopUkhzxzUHNlfvqNm6yDqBCI8=', 'Content-Length': '0'} 2023-07-29T00:35:31.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000356156093d2a754c-0064c45ed3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-07-29T00:35:31.576 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:35:31.577 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:35:31.577 DEBUG:boto:Method: GET 2023-07-29T00:35:31.578 DEBUG:boto:Path: /hiiqos-59/obj95 2023-07-29T00:35:31.578 DEBUG:boto:Data: 2023-07-29T00:35:31.578 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.579 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:31.579 DEBUG:boto:Port: 8001 2023-07-29T00:35:31.579 DEBUG:boto:Params: {} 2023-07-29T00:35:31.580 DEBUG:boto:Token: None 2023-07-29T00:35:31.580 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/obj95 2023-07-29T00:35:31.581 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2kkL0yWSTssNxqZ+wYI4Z08TXRE= 2023-07-29T00:35:31.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2kkL0yWSTssNxqZ+wYI4Z08TXRE=', 'Content-Length': '0'} 2023-07-29T00:35:31.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7ee40da3611aa94-0064c45ed3-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.624 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:35:31.625 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:35:31.625 DEBUG:boto:Method: GET 2023-07-29T00:35:31.626 DEBUG:boto:Path: /hiiqos-59/obj95 2023-07-29T00:35:31.626 DEBUG:boto:Data: 2023-07-29T00:35:31.626 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.627 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:31.627 DEBUG:boto:Port: 8000 2023-07-29T00:35:31.627 DEBUG:boto:Params: {} 2023-07-29T00:35:31.628 DEBUG:boto:Token: None 2023-07-29T00:35:31.628 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/obj95 2023-07-29T00:35:31.629 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2kkL0yWSTssNxqZ+wYI4Z08TXRE= 2023-07-29T00:35:31.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2kkL0yWSTssNxqZ+wYI4Z08TXRE=', 'Content-Length': '0'} 2023-07-29T00:35:31.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5c8a60698f8089e-0064c45ed3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.632 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:31.632 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:31.632 DEBUG:boto:path=/hiiqos-59/?prefix=obj95 2023-07-29T00:35:31.633 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj95 2023-07-29T00:35:31.633 DEBUG:boto:Method: GET 2023-07-29T00:35:31.633 DEBUG:boto:Path: /hiiqos-59/?prefix=obj95 2023-07-29T00:35:31.634 DEBUG:boto:Data: 2023-07-29T00:35:31.634 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.634 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:35:31.635 DEBUG:boto:Port: 8001 2023-07-29T00:35:31.635 DEBUG:boto:Params: {} 2023-07-29T00:35:31.636 DEBUG:boto:Token: None 2023-07-29T00:35:31.636 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/ 2023-07-29T00:35:31.636 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kHyDn0lT4eNZCdvQe0CQ0o0BiG0= 2023-07-29T00:35:31.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kHyDn0lT4eNZCdvQe0CQ0o0BiG0=', 'Content-Length': '0'} 2023-07-29T00:35:31.639 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e91e03d9beafd57-0064c45ed3-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.640 DEBUG:boto:b'hiiqos-59obj951000falseobj952023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:31.640 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:31.641 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:31.641 DEBUG:boto:path=/hiiqos-59/?prefix=obj95 2023-07-29T00:35:31.641 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj95 2023-07-29T00:35:31.642 DEBUG:boto:Method: GET 2023-07-29T00:35:31.642 DEBUG:boto:Path: /hiiqos-59/?prefix=obj95 2023-07-29T00:35:31.642 DEBUG:boto:Data: 2023-07-29T00:35:31.643 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.643 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:31.643 DEBUG:boto:Port: 8000 2023-07-29T00:35:31.644 DEBUG:boto:Params: {} 2023-07-29T00:35:31.644 DEBUG:boto:Token: None 2023-07-29T00:35:31.645 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/ 2023-07-29T00:35:31.645 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kHyDn0lT4eNZCdvQe0CQ0o0BiG0= 2023-07-29T00:35:31.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kHyDn0lT4eNZCdvQe0CQ0o0BiG0=', 'Content-Length': '0'} 2023-07-29T00:35:31.649 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085ac7cd3e7bea44a-0064c45ed3-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.650 DEBUG:boto:b'hiiqos-59obj951000falseobj952023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:35:31.650 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-59 zones={a2, a1} 2023-07-29T00:35:31.651 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:35:31.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:35:31.652 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:35:31.705 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:35:31.705 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:35:31.753 INFO:teuthology.orchestra.run.smithi101.stdout:tenant: 2023-07-29T00:35:31.753 INFO:teuthology.orchestra.run.smithi101.stdout:bucket name: hiiqos-59 2023-07-29T00:35:31.763 INFO:teuthology.orchestra.run.smithi101.stdout:total entries: 25 2023-07-29T00:35:31.817 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:31.818 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:31.818 DEBUG:boto:Method: HEAD 2023-07-29T00:35:31.818 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:35:31.819 DEBUG:boto:Data: 2023-07-29T00:35:31.819 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.819 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:31.820 DEBUG:boto:Port: 8000 2023-07-29T00:35:31.820 DEBUG:boto:Params: {} 2023-07-29T00:35:31.820 DEBUG:boto:Token: None 2023-07-29T00:35:31.821 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/ 2023-07-29T00:35:31.821 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8TzDD9BiubrLTy80Wf5IvyZ/Tbo= 2023-07-29T00:35:31.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8TzDD9BiubrLTy80Wf5IvyZ/Tbo=', 'Content-Length': '0'} 2023-07-29T00:35:31.825 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001e390c8aede921d4-0064c45ed3-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.825 DEBUG:boto:path=/hiiqos-59/obj31 2023-07-29T00:35:31.826 DEBUG:boto:auth_path=/hiiqos-59/obj31 2023-07-29T00:35:31.826 DEBUG:boto:Method: PUT 2023-07-29T00:35:31.826 DEBUG:boto:Path: /hiiqos-59/obj31 2023-07-29T00:35:31.827 DEBUG:boto:Data: 2023-07-29T00:35:31.827 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:35:31.827 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:31.828 DEBUG:boto:Port: 8000 2023-07-29T00:35:31.828 DEBUG:boto:Params: {} 2023-07-29T00:35:31.828 DEBUG:boto:Token: None 2023-07-29T00:35:31.829 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/obj31 2023-07-29T00:35:31.829 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:us+F2JZMMvGxZcsAub9DaUSHKbY= 2023-07-29T00:35:31.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:us+F2JZMMvGxZcsAub9DaUSHKbY='} 2023-07-29T00:35:31.872 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f2f5e0ff0d39520f-0064c45ed3-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.873 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:31.873 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:31.874 DEBUG:boto:Method: HEAD 2023-07-29T00:35:31.874 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:35:31.874 DEBUG:boto:Data: 2023-07-29T00:35:31.875 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.875 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:31.875 DEBUG:boto:Port: 8000 2023-07-29T00:35:31.876 DEBUG:boto:Params: {} 2023-07-29T00:35:31.876 DEBUG:boto:Token: None 2023-07-29T00:35:31.876 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/ 2023-07-29T00:35:31.877 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8TzDD9BiubrLTy80Wf5IvyZ/Tbo= 2023-07-29T00:35:31.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8TzDD9BiubrLTy80Wf5IvyZ/Tbo=', 'Content-Length': '0'} 2023-07-29T00:35:31.880 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000047cad4318b3a49e-0064c45ed3-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.881 DEBUG:boto:path=/hiiqos-59/obj62 2023-07-29T00:35:31.881 DEBUG:boto:auth_path=/hiiqos-59/obj62 2023-07-29T00:35:31.881 DEBUG:boto:Method: PUT 2023-07-29T00:35:31.882 DEBUG:boto:Path: /hiiqos-59/obj62 2023-07-29T00:35:31.882 DEBUG:boto:Data: 2023-07-29T00:35:31.882 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:35:31.883 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:31.883 DEBUG:boto:Port: 8000 2023-07-29T00:35:31.883 DEBUG:boto:Params: {} 2023-07-29T00:35:31.884 DEBUG:boto:Token: None 2023-07-29T00:35:31.884 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/obj62 2023-07-29T00:35:31.885 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:F6F277cVt7cXqXDfDHDpXuEK8oI= 2023-07-29T00:35:31.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:F6F277cVt7cXqXDfDHDpXuEK8oI='} 2023-07-29T00:35:31.890 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000022fab85568cd6ebe-0064c45ed3-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.890 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:31.891 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:31.891 DEBUG:boto:Method: HEAD 2023-07-29T00:35:31.891 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:35:31.892 DEBUG:boto:Data: 2023-07-29T00:35:31.892 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.892 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:31.893 DEBUG:boto:Port: 8000 2023-07-29T00:35:31.893 DEBUG:boto:Params: {} 2023-07-29T00:35:31.893 DEBUG:boto:Token: None 2023-07-29T00:35:31.894 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/ 2023-07-29T00:35:31.894 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8TzDD9BiubrLTy80Wf5IvyZ/Tbo= 2023-07-29T00:35:31.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8TzDD9BiubrLTy80Wf5IvyZ/Tbo=', 'Content-Length': '0'} 2023-07-29T00:35:31.898 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d3d43047d00e3695-0064c45ed3-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.898 DEBUG:boto:path=/hiiqos-59/obj93 2023-07-29T00:35:31.899 DEBUG:boto:auth_path=/hiiqos-59/obj93 2023-07-29T00:35:31.899 DEBUG:boto:Method: PUT 2023-07-29T00:35:31.899 DEBUG:boto:Path: /hiiqos-59/obj93 2023-07-29T00:35:31.900 DEBUG:boto:Data: 2023-07-29T00:35:31.900 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:35:31.900 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:31.901 DEBUG:boto:Port: 8000 2023-07-29T00:35:31.901 DEBUG:boto:Params: {} 2023-07-29T00:35:31.901 DEBUG:boto:Token: None 2023-07-29T00:35:31.902 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/obj93 2023-07-29T00:35:31.902 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4USoEcfWvC1AOyrfgqPTe9NjB44= 2023-07-29T00:35:31.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4USoEcfWvC1AOyrfgqPTe9NjB44='} 2023-07-29T00:35:31.944 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009a6f3a640f0a76db-0064c45ed3-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.945 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:31.945 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:31.946 DEBUG:boto:Method: HEAD 2023-07-29T00:35:31.946 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:35:31.947 DEBUG:boto:Data: 2023-07-29T00:35:31.948 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.948 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:31.949 DEBUG:boto:Port: 8000 2023-07-29T00:35:31.949 DEBUG:boto:Params: {} 2023-07-29T00:35:31.950 DEBUG:boto:Token: None 2023-07-29T00:35:31.950 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/ 2023-07-29T00:35:31.951 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8TzDD9BiubrLTy80Wf5IvyZ/Tbo= 2023-07-29T00:35:31.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8TzDD9BiubrLTy80Wf5IvyZ/Tbo=', 'Content-Length': '0'} 2023-07-29T00:35:31.954 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000099c8716205f8559c-0064c45ed3-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.955 DEBUG:boto:path=/hiiqos-59/obj124 2023-07-29T00:35:31.955 DEBUG:boto:auth_path=/hiiqos-59/obj124 2023-07-29T00:35:31.956 DEBUG:boto:Method: PUT 2023-07-29T00:35:31.956 DEBUG:boto:Path: /hiiqos-59/obj124 2023-07-29T00:35:31.957 DEBUG:boto:Data: 2023-07-29T00:35:31.957 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:35:31.957 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:31.958 DEBUG:boto:Port: 8000 2023-07-29T00:35:31.958 DEBUG:boto:Params: {} 2023-07-29T00:35:31.958 DEBUG:boto:Token: None 2023-07-29T00:35:31.959 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/obj124 2023-07-29T00:35:31.959 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ns+qDM5Nb3zOADE0LeYlUbdoQOQ= 2023-07-29T00:35:31.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ns+qDM5Nb3zOADE0LeYlUbdoQOQ='} 2023-07-29T00:35:31.964 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002369dfab6a0fd00c-0064c45ed3-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.965 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:35:31.965 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:35:31.966 DEBUG:boto:Method: HEAD 2023-07-29T00:35:31.966 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:35:31.966 DEBUG:boto:Data: 2023-07-29T00:35:31.967 DEBUG:boto:Headers: {} 2023-07-29T00:35:31.967 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:31.967 DEBUG:boto:Port: 8000 2023-07-29T00:35:31.968 DEBUG:boto:Params: {} 2023-07-29T00:35:31.968 DEBUG:boto:Token: None 2023-07-29T00:35:31.969 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/ 2023-07-29T00:35:31.969 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8TzDD9BiubrLTy80Wf5IvyZ/Tbo= 2023-07-29T00:35:31.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8TzDD9BiubrLTy80Wf5IvyZ/Tbo=', 'Content-Length': '0'} 2023-07-29T00:35:31.973 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000db6586f15f6a07d7-0064c45ed3-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:31.973 DEBUG:boto:path=/hiiqos-59/obj155 2023-07-29T00:35:31.974 DEBUG:boto:auth_path=/hiiqos-59/obj155 2023-07-29T00:35:31.974 DEBUG:boto:Method: PUT 2023-07-29T00:35:31.974 DEBUG:boto:Path: /hiiqos-59/obj155 2023-07-29T00:35:31.975 DEBUG:boto:Data: 2023-07-29T00:35:31.975 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:35:31.975 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:35:31.976 DEBUG:boto:Port: 8000 2023-07-29T00:35:31.976 DEBUG:boto:Params: {} 2023-07-29T00:35:31.976 DEBUG:boto:Token: None 2023-07-29T00:35:31.977 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:35:31 GMT /hiiqos-59/obj155 2023-07-29T00:35:31.977 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:s3q4I/Hp3yewnxEfAkufEooTX7Q= 2023-07-29T00:35:31.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:35:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:s3q4I/Hp3yewnxEfAkufEooTX7Q='} 2023-07-29T00:35:32.024 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005b26be67aca3a628-0064c45ed3-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:35:32.025 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-59 2023-07-29T00:35:32.026 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:35:32.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:35:32.026 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:35:32.079 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:35:32.079 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:35:32.113 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:35:32.052+0000 7f67d5c14880 1 waiting to reach latest gen 5, on 4.. 2023-07-29T00:35:37.114 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:35:37.053+0000 7f67d5c14880 1 waiting to reach latest gen 5, on 4.. 2023-07-29T00:35:42.115 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:35:42.053+0000 7f67d5c14880 1 waiting to reach latest gen 5, on 4.. 2023-07-29T00:35:47.118 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:35:47.056+0000 7f67d5c14880 1 waiting for incremental sync to catch up: 2023-07-29T00:35:47.118 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-07-29T00:35:47.118 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , 00000000002.125.6, , 00000000001.138.6, , 00000000001.68.6, , , , , , , , , , 00000000001.70.6, , , , , , , , , , , , , , 00000000001.55.6] 2023-07-29T00:35:52.120 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:35:52.059+0000 7f67d5c14880 1 waiting for incremental sync to catch up: 2023-07-29T00:35:52.121 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-07-29T00:35:52.121 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , 00000000002.125.6, , 00000000001.138.6, , 00000000001.68.6, , , , , , , , , , 00000000001.70.6, , , , , , , , , , , , , , 00000000001.55.6] 2023-07-29T00:35:57.123 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:35:57.061+0000 7f67d5c14880 1 waiting for incremental sync to catch up: 2023-07-29T00:35:57.123 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-07-29T00:35:57.124 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , 00000000002.125.6, , 00000000001.138.6, , 00000000001.68.6, , , , , , , , , , 00000000001.70.6, , , , , , , , , , , , , , 00000000001.55.6] 2023-07-29T00:36:02.125 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:36:02.063+0000 7f67d5c14880 1 waiting for incremental sync to catch up: 2023-07-29T00:36:02.125 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-07-29T00:36:02.125 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , 00000000002.125.6, , 00000000001.138.6, , 00000000001.68.6, , , , , , , , , , 00000000001.70.6, , , , , , , , , , , , , , 00000000001.55.6] 2023-07-29T00:36:07.127 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:36:07.065+0000 7f67d5c14880 1 waiting for incremental sync to catch up: 2023-07-29T00:36:07.127 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-07-29T00:36:07.128 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , 00000000002.125.6, , 00000000001.138.6, , 00000000001.68.6, , , , , , , , , , 00000000001.70.6, , , , , , , , , , , , , , 00000000001.55.6] 2023-07-29T00:36:12.129 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:36:12.067+0000 7f67d5c14880 1 waiting for incremental sync to catch up: 2023-07-29T00:36:12.130 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-07-29T00:36:12.130 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , 00000000002.125.6, , 00000000001.138.6, , 00000000001.68.6, , , , , , , , , , 00000000001.70.6, , , , , , , , , , , , , , 00000000001.55.6] 2023-07-29T00:36:17.131 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:36:17.070+0000 7f67d5c14880 1 waiting for incremental sync to catch up: 2023-07-29T00:36:17.132 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-07-29T00:36:17.132 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , 00000000002.125.6, , 00000000001.138.6, , 00000000001.68.6, , , , , , , , , , 00000000001.70.6, , , , , , , , , , , , , , 00000000001.55.6] 2023-07-29T00:36:22.134 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:36:22.072+0000 7f67d5c14880 1 waiting for incremental sync to catch up: 2023-07-29T00:36:22.134 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-07-29T00:36:22.134 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , 00000000002.125.6, , 00000000001.138.6, , 00000000001.68.6, , , , , , , , , , 00000000001.70.6, , , , , , , , , , , , , , 00000000001.55.6] 2023-07-29T00:36:27.136 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:36:27.074+0000 7f67d5c14880 1 waiting for incremental sync to catch up: 2023-07-29T00:36:27.137 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , 00000000001.138.6, , 00000000001.68.6, , , , , , , , , , 00000000001.70.6, , , , , , , , , , , , , , 00000000001.55.6] 2023-07-29T00:36:27.137 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , 00000000002.125.6, , 00000000001.138.6, , 00000000001.68.6, , , , , , , , , , 00000000001.70.6, , , , , , , , , , , , , , 00000000001.55.6] 2023-07-29T00:36:32.139 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:36:32.077+0000 7f67d5c14880 1 waiting for incremental sync to catch up: 2023-07-29T00:36:32.140 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , 00000000001.138.6, , 00000000001.68.6, , , , , , , , , , 00000000001.70.6, , , , , , , , , , , , , , 00000000001.55.6] 2023-07-29T00:36:32.140 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , 00000000002.125.6, , 00000000001.138.6, , 00000000001.68.6, , , , , , , , , , 00000000001.70.6, , , , , , , , , , , , , , 00000000001.55.6] 2023-07-29T00:36:37.141 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:36:37.079+0000 7f67d5c14880 1 waiting for incremental sync to catch up: 2023-07-29T00:36:37.142 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , 00000000001.138.6, , 00000000001.68.6, , , , , , , , , , 00000000001.70.6, , , , , , , , , , , , , , 00000000001.55.6] 2023-07-29T00:36:37.142 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , 00000000002.125.6, , 00000000001.138.6, , 00000000001.68.6, , , , , , , , , , 00000000001.70.6, , , , , , , , , , , , , , 00000000001.55.6] 2023-07-29T00:36:41.837 INFO:tasks.rgw.c2.client.0.smithi139.stdout:2023-07-29T00:36:41.775+0000 7f5f2671b640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T00:36:42.143 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:36:42.081+0000 7f67d5c14880 1 waiting for incremental sync to catch up: 2023-07-29T00:36:42.144 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , 00000000001.138.6, , 00000000001.68.6, , , , , , , , , , 00000000001.70.6, , , , , , , , , , , , , , 00000000001.55.6] 2023-07-29T00:36:42.144 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , 00000000002.125.6, , 00000000001.138.6, , 00000000001.68.6, , , , , , , , , , 00000000001.70.6, , , , , , , , , , , , , , 00000000001.55.6] 2023-07-29T00:36:47.146 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:36:47.084+0000 7f67d5c14880 1 waiting for incremental sync to catch up: 2023-07-29T00:36:47.146 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , 00000000001.138.6, , 00000000001.68.6, , , , , , , , , , 00000000001.70.6, , , , , , , , , , , , , , 00000000001.55.6] 2023-07-29T00:36:47.147 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , 00000000002.125.6, , 00000000001.138.6, , 00000000001.68.6, , , , , , , , , , 00000000001.70.6, , , , , , , , , , , , , , 00000000001.55.6] 2023-07-29T00:36:52.149 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:36:52.086+0000 7f67d5c14880 1 waiting for incremental sync to catch up: 2023-07-29T00:36:52.149 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , 00000000001.138.6, , 00000000001.68.6, , , , , , , , , , 00000000001.70.6, , , , , , , , , , , , , , 00000000001.55.6] 2023-07-29T00:36:52.149 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , 00000000002.125.6, , 00000000001.138.6, , 00000000001.68.6, , , , , , , , , , 00000000001.70.6, , , , , , , , , , , , , , 00000000001.55.6] 2023-07-29T00:36:57.151 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:36:57.089+0000 7f67d5c14880 1 waiting for incremental sync to catch up: 2023-07-29T00:36:57.151 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , 00000000001.138.6, , 00000000001.68.6, , , , , , , , , , 00000000001.70.6, , , , , , , , , , , , , , 00000000001.55.6] 2023-07-29T00:36:57.152 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , 00000000002.125.6, , 00000000001.138.6, , 00000000001.68.6, , , , , , , , , , 00000000001.70.6, , , , , , , , , , , , , , 00000000001.55.6] 2023-07-29T00:37:02.153 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:37:02.091+0000 7f67d5c14880 1 waiting for incremental sync to catch up: 2023-07-29T00:37:02.153 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , 00000000001.138.6, , 00000000001.68.6, , , , , , , , , , 00000000001.70.6, , , , , , , , , , , , , , 00000000001.55.6] 2023-07-29T00:37:02.154 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , 00000000002.125.6, , 00000000001.138.6, , 00000000001.68.6, , , , , , , , , , 00000000001.70.6, , , , , , , , , , , , , , 00000000001.55.6] 2023-07-29T00:37:07.156 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:37:07.093+0000 7f67d5c14880 1 waiting for incremental sync to catch up: 2023-07-29T00:37:07.156 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , 00000000001.138.6, , 00000000001.68.6, , , , , , , , , , 00000000001.70.6, , , , , , , , , , , , , , 00000000001.55.6] 2023-07-29T00:37:07.156 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , 00000000002.125.6, , 00000000001.138.6, , 00000000001.68.6, , , , , , , , , , 00000000001.70.6, , , , , , , , , , , , , , 00000000001.55.6] 2023-07-29T00:37:12.158 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:37:12.095+0000 7f67d5c14880 1 waiting for incremental sync to catch up: 2023-07-29T00:37:12.158 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , 00000000001.138.6, , 00000000001.68.6, , , , , , , , , , 00000000001.70.6, , , , , , , , , , , , , , 00000000001.55.6] 2023-07-29T00:37:12.159 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , 00000000002.125.6, , 00000000001.138.6, , 00000000001.68.6, , , , , , , , , , 00000000001.70.6, , , , , , , , , , , , , , 00000000001.55.6] 2023-07-29T00:37:17.160 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:37:17.098+0000 7f67d5c14880 1 waiting for incremental sync to catch up: 2023-07-29T00:37:17.161 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , 00000000001.138.6, , 00000000001.68.6, , , , , , , , , , 00000000001.70.6, , , , , , , , , , , , , , 00000000001.55.6] 2023-07-29T00:37:17.161 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , 00000000002.125.6, , 00000000001.138.6, , 00000000001.68.6, , , , , , , , , , 00000000001.70.6, , , , , , , , , , , , , , 00000000001.55.6] 2023-07-29T00:37:17.759 INFO:tasks.rgw.c2.client.1.smithi139.stdout:2023-07-29T00:37:17.695+0000 7f5731a6a640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T00:37:22.163 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:37:22.100+0000 7f67d5c14880 1 waiting for incremental sync to catch up: 2023-07-29T00:37:22.163 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , 00000000001.138.6, , 00000000001.68.6, , , , , , , , , , 00000000001.70.6, , , , , , , , , , , , , , 00000000001.55.6] 2023-07-29T00:37:22.163 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , 00000000002.125.6, , 00000000001.138.6, , 00000000001.68.6, , , , , , , , , , 00000000001.70.6, , , , , , , , , , , , , , 00000000001.55.6] 2023-07-29T00:37:27.165 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:37:27.102+0000 7f67d5c14880 1 bucket sync caught up with source: 2023-07-29T00:37:27.165 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , 00000000002.125.6, , 00000000001.138.6, , 00000000001.68.6, , , , , , , , , , 00000000001.70.6, , , , , , , , , , , , , , 00000000001.55.6] 2023-07-29T00:37:27.165 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , 00000000002.125.6, , 00000000001.138.6, , 00000000001.68.6, , , , , , , , , , 00000000001.70.6, , , , , , , , , , , , , , 00000000001.55.6] 2023-07-29T00:37:27.165 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:37:27.102+0000 7f67d5c14880 0 bucket checkpoint complete 2023-07-29T00:37:27.182 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-59 2023-07-29T00:37:27.183 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:37:27.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:37:27.184 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:37:27.241 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:37:27.241 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:37:27.277 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:37:27.214+0000 7fd81e28b880 1 waiting for incremental sync to catch up: 2023-07-29T00:37:27.277 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000003.862.6, 00000000002.701.6, 00000000002.1214.6, 00000000003.954.6, 00000000004.745.6, 00000000001.1016.6, 00000000001.432.6, 00000000003.309.6, 00000000006.803.6, 00000000004.1374.6] 2023-07-29T00:37:27.278 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000003.862.6, 00000000002.701.6, 00000000002.1214.6, 00000000004.960.6, 00000000004.745.6, 00000000001.1016.6, 00000000001.432.6, 00000000003.309.6, 00000000006.803.6, 00000000004.1374.6] 2023-07-29T00:37:32.279 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:37:32.216+0000 7fd81e28b880 1 waiting for incremental sync to catch up: 2023-07-29T00:37:32.279 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000003.862.6, 00000000002.701.6, 00000000002.1214.6, 00000000003.954.6, 00000000004.745.6, 00000000001.1016.6, 00000000001.432.6, 00000000003.309.6, 00000000006.803.6, 00000000004.1374.6] 2023-07-29T00:37:32.279 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000003.862.6, 00000000002.701.6, 00000000002.1214.6, 00000000004.960.6, 00000000004.745.6, 00000000001.1016.6, 00000000001.432.6, 00000000003.309.6, 00000000006.803.6, 00000000004.1374.6] 2023-07-29T00:37:37.280 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:37:37.217+0000 7fd81e28b880 1 waiting for incremental sync to catch up: 2023-07-29T00:37:37.281 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000003.862.6, 00000000002.701.6, 00000000002.1214.6, 00000000003.954.6, 00000000004.745.6, 00000000001.1016.6, 00000000001.432.6, 00000000003.309.6, 00000000006.803.6, 00000000004.1374.6] 2023-07-29T00:37:37.281 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000003.862.6, 00000000002.701.6, 00000000002.1214.6, 00000000004.960.6, 00000000004.745.6, 00000000001.1016.6, 00000000001.432.6, 00000000003.309.6, 00000000006.803.6, 00000000004.1374.6] 2023-07-29T00:37:42.282 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:37:42.219+0000 7fd81e28b880 1 waiting for incremental sync to catch up: 2023-07-29T00:37:42.282 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000003.862.6, 00000000002.701.6, 00000000002.1214.6, 00000000003.954.6, 00000000004.745.6, 00000000001.1016.6, 00000000001.432.6, 00000000003.309.6, 00000000006.803.6, 00000000004.1374.6] 2023-07-29T00:37:42.282 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000003.862.6, 00000000002.701.6, 00000000002.1214.6, 00000000004.960.6, 00000000004.745.6, 00000000001.1016.6, 00000000001.432.6, 00000000003.309.6, 00000000006.803.6, 00000000004.1374.6] 2023-07-29T00:37:47.283 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:37:47.220+0000 7fd81e28b880 1 bucket sync caught up with source: 2023-07-29T00:37:47.284 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000003.862.6, 00000000002.701.6, 00000000002.1214.6, 00000000004.960.6, 00000000004.745.6, 00000000001.1016.6, 00000000001.432.6, 00000000003.309.6, 00000000006.803.6, 00000000004.1374.6] 2023-07-29T00:37:47.284 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000003.862.6, 00000000002.701.6, 00000000002.1214.6, 00000000004.960.6, 00000000004.745.6, 00000000001.1016.6, 00000000001.432.6, 00000000003.309.6, 00000000006.803.6, 00000000004.1374.6] 2023-07-29T00:37:47.284 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:37:47.220+0000 7fd81e28b880 0 bucket checkpoint complete 2023-07-29T00:37:47.300 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-59 zones={a2, a1} 2023-07-29T00:37:47.301 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:47.301 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:47.302 DEBUG:boto:Method: HEAD 2023-07-29T00:37:47.302 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:37:47.302 DEBUG:boto:Data: 2023-07-29T00:37:47.303 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.303 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:47.304 DEBUG:boto:Port: 8001 2023-07-29T00:37:47.304 DEBUG:boto:Params: {} 2023-07-29T00:37:47.304 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-29T00:37:47.305 DEBUG:boto:Token: None 2023-07-29T00:37:47.306 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/ 2023-07-29T00:37:47.306 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w9T5Spf+LwhOA1ovFDohKQv+sBo= 2023-07-29T00:37:47.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w9T5Spf+LwhOA1ovFDohKQv+sBo=', 'Content-Length': '0'} 2023-07-29T00:37:47.311 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009e8cf3dcf58b030e-0064c45f5b-1c08-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.312 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:47.312 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:47.313 DEBUG:boto:Method: HEAD 2023-07-29T00:37:47.313 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:37:47.313 DEBUG:boto:Data: 2023-07-29T00:37:47.314 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.314 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:47.314 DEBUG:boto:Port: 8000 2023-07-29T00:37:47.315 DEBUG:boto:Params: {} 2023-07-29T00:37:47.315 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-29T00:37:47.316 DEBUG:boto:Token: None 2023-07-29T00:37:47.316 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/ 2023-07-29T00:37:47.316 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w9T5Spf+LwhOA1ovFDohKQv+sBo= 2023-07-29T00:37:47.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w9T5Spf+LwhOA1ovFDohKQv+sBo=', 'Content-Length': '0'} 2023-07-29T00:37:47.322 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000721e6c738e4c312f-0064c45f5b-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.322 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-29T00:37:47.323 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:47.323 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:47.324 DEBUG:boto:path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:37:47.324 DEBUG:boto:auth_path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:37:47.324 DEBUG:boto:Method: GET 2023-07-29T00:37:47.325 DEBUG:boto:Path: /hiiqos-59/?versions&max-keys=999 2023-07-29T00:37:47.325 DEBUG:boto:Data: 2023-07-29T00:37:47.325 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.326 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:47.326 DEBUG:boto:Port: 8001 2023-07-29T00:37:47.326 DEBUG:boto:Params: {} 2023-07-29T00:37:47.327 DEBUG:boto:Token: None 2023-07-29T00:37:47.327 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/?versions 2023-07-29T00:37:47.328 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cO9JtLLeVqPUn0UxKSzj71q1Tao= 2023-07-29T00:37:47.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cO9JtLLeVqPUn0UxKSzj71q1Tao=', 'Content-Length': '0'} 2023-07-29T00:37:47.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033e7d48022c1a589-0064c45f5b-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.332 DEBUG:boto:b'hiiqos-59999falseobj11nulltrue2023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-07-29T00:35:31.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-07-29T00:35:01.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-07-29T00:35:31.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-07-29T00:33:06.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-07-29T00:35:01.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-07-29T00:35:31.771Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-07-29T00:33:06.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-07-29T00:35:01.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-07-29T00:35:31.826Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-07-29T00:33:06.857Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-07-29T00:35:01.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-07-29T00:33:06.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-07-29T00:35:31.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:47.334 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-07-29T00:37:47.334 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-07-29T00:37:47.334 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-07-29T00:37:47.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-07-29T00:37:47.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-07-29T00:37:47.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-07-29T00:37:47.336 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-07-29T00:37:47.336 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-07-29T00:37:47.336 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-07-29T00:37:47.336 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-07-29T00:37:47.337 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-07-29T00:37:47.337 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-07-29T00:37:47.337 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-07-29T00:37:47.338 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-07-29T00:37:47.338 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-07-29T00:37:47.338 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-07-29T00:37:47.339 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-07-29T00:37:47.339 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-07-29T00:37:47.339 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-07-29T00:37:47.339 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-07-29T00:37:47.340 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-07-29T00:37:47.340 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-07-29T00:37:47.340 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-07-29T00:37:47.341 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-07-29T00:37:47.341 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-07-29T00:37:47.341 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-07-29T00:37:47.341 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-07-29T00:37:47.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-07-29T00:37:47.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-07-29T00:37:47.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-07-29T00:37:47.343 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-29T00:37:47.343 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:47.343 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:47.344 DEBUG:boto:path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:37:47.344 DEBUG:boto:auth_path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:37:47.345 DEBUG:boto:Method: GET 2023-07-29T00:37:47.345 DEBUG:boto:Path: /hiiqos-59/?versions&max-keys=999 2023-07-29T00:37:47.346 DEBUG:boto:Data: 2023-07-29T00:37:47.346 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.346 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:47.347 DEBUG:boto:Port: 8000 2023-07-29T00:37:47.347 DEBUG:boto:Params: {} 2023-07-29T00:37:47.347 DEBUG:boto:Token: None 2023-07-29T00:37:47.348 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/?versions 2023-07-29T00:37:47.348 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cO9JtLLeVqPUn0UxKSzj71q1Tao= 2023-07-29T00:37:47.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cO9JtLLeVqPUn0UxKSzj71q1Tao=', 'Content-Length': '0'} 2023-07-29T00:37:47.356 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d4f9f48b788d301-0064c45f5b-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.357 DEBUG:boto:b'hiiqos-59999falseobj11nulltrue2023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-07-29T00:35:31.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-07-29T00:35:01.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-07-29T00:35:31.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-07-29T00:33:06.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-07-29T00:35:01.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-07-29T00:35:31.771Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-07-29T00:33:06.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-07-29T00:35:01.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-07-29T00:35:31.826Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-07-29T00:33:06.857Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-07-29T00:35:01.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-07-29T00:33:06.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-07-29T00:35:31.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:47.359 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-07-29T00:37:47.359 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-07-29T00:37:47.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-07-29T00:37:47.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-07-29T00:37:47.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-07-29T00:37:47.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-07-29T00:37:47.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-07-29T00:37:47.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-07-29T00:37:47.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-07-29T00:37:47.362 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-07-29T00:37:47.362 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-07-29T00:37:47.362 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-07-29T00:37:47.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-07-29T00:37:47.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-07-29T00:37:47.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-07-29T00:37:47.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-07-29T00:37:47.364 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-07-29T00:37:47.364 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-07-29T00:37:47.364 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-07-29T00:37:47.365 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-07-29T00:37:47.365 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-07-29T00:37:47.365 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-07-29T00:37:47.366 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-07-29T00:37:47.366 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-07-29T00:37:47.366 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-07-29T00:37:47.366 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-07-29T00:37:47.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-07-29T00:37:47.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-07-29T00:37:47.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-07-29T00:37:47.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-07-29T00:37:47.368 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:47.369 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:47.369 DEBUG:boto:path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:37:47.369 DEBUG:boto:auth_path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:37:47.370 DEBUG:boto:Method: GET 2023-07-29T00:37:47.370 DEBUG:boto:Path: /hiiqos-59/?versions&max-keys=999 2023-07-29T00:37:47.370 DEBUG:boto:Data: 2023-07-29T00:37:47.371 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.371 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:47.371 DEBUG:boto:Port: 8001 2023-07-29T00:37:47.372 DEBUG:boto:Params: {} 2023-07-29T00:37:47.372 DEBUG:boto:Token: None 2023-07-29T00:37:47.373 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/?versions 2023-07-29T00:37:47.373 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cO9JtLLeVqPUn0UxKSzj71q1Tao= 2023-07-29T00:37:47.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cO9JtLLeVqPUn0UxKSzj71q1Tao=', 'Content-Length': '0'} 2023-07-29T00:37:47.376 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b8e2afd371883b1-0064c45f5b-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.377 DEBUG:boto:b'hiiqos-59999falseobj11nulltrue2023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-07-29T00:35:31.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-07-29T00:35:01.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-07-29T00:35:31.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-07-29T00:33:06.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-07-29T00:35:01.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-07-29T00:35:31.771Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-07-29T00:33:06.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-07-29T00:35:01.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-07-29T00:35:31.826Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-07-29T00:33:06.857Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-07-29T00:35:01.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-07-29T00:33:06.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-07-29T00:35:31.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:47.379 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:47.379 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:47.380 DEBUG:boto:path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:37:47.380 DEBUG:boto:auth_path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:37:47.381 DEBUG:boto:Method: GET 2023-07-29T00:37:47.381 DEBUG:boto:Path: /hiiqos-59/?versions&max-keys=999 2023-07-29T00:37:47.381 DEBUG:boto:Data: 2023-07-29T00:37:47.382 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.382 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:47.382 DEBUG:boto:Port: 8000 2023-07-29T00:37:47.383 DEBUG:boto:Params: {} 2023-07-29T00:37:47.383 DEBUG:boto:Token: None 2023-07-29T00:37:47.384 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/?versions 2023-07-29T00:37:47.384 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cO9JtLLeVqPUn0UxKSzj71q1Tao= 2023-07-29T00:37:47.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cO9JtLLeVqPUn0UxKSzj71q1Tao=', 'Content-Length': '0'} 2023-07-29T00:37:47.388 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000540008e04c8ecf27-0064c45f5b-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.389 DEBUG:boto:b'hiiqos-59999falseobj11nulltrue2023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-07-29T00:35:31.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-07-29T00:35:01.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-07-29T00:35:31.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-07-29T00:33:06.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-07-29T00:35:01.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-07-29T00:35:31.771Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-07-29T00:33:06.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-07-29T00:35:01.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-07-29T00:35:31.826Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-07-29T00:33:06.857Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-07-29T00:35:01.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-07-29T00:33:06.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-07-29T00:35:31.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:47.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-07-29T00:37:47.391 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:37:47.392 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:37:47.392 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:37:47.392 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:37:47.393 DEBUG:boto:Method: GET 2023-07-29T00:37:47.393 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:37:47.393 DEBUG:boto:Data: 2023-07-29T00:37:47.394 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.394 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:47.395 DEBUG:boto:Port: 8001 2023-07-29T00:37:47.395 DEBUG:boto:Params: {} 2023-07-29T00:37:47.395 DEBUG:boto:Token: None 2023-07-29T00:37:47.396 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:37:47.396 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+aVplbhn04+7tVhUQ1IzokY9Oq8= 2023-07-29T00:37:47.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+aVplbhn04+7tVhUQ1IzokY9Oq8=', 'Content-Length': '0'} 2023-07-29T00:37:47.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e7b6f8879c27e77-0064c45f5b-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.400 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:37:47.400 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:37:47.400 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:37:47.401 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:37:47.401 DEBUG:boto:Method: GET 2023-07-29T00:37:47.402 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:37:47.402 DEBUG:boto:Data: 2023-07-29T00:37:47.403 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.403 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:47.404 DEBUG:boto:Port: 8000 2023-07-29T00:37:47.404 DEBUG:boto:Params: {} 2023-07-29T00:37:47.405 DEBUG:boto:Token: None 2023-07-29T00:37:47.406 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:37:47.406 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+aVplbhn04+7tVhUQ1IzokY9Oq8= 2023-07-29T00:37:47.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+aVplbhn04+7tVhUQ1IzokY9Oq8=', 'Content-Length': '0'} 2023-07-29T00:37:47.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5d89f8eca47c896-0064c45f5b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.452 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:37:47.453 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:37:47.453 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:37:47.453 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:37:47.454 DEBUG:boto:Method: HEAD 2023-07-29T00:37:47.454 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:37:47.454 DEBUG:boto:Data: 2023-07-29T00:37:47.455 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.455 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:47.455 DEBUG:boto:Port: 8001 2023-07-29T00:37:47.456 DEBUG:boto:Params: {} 2023-07-29T00:37:47.456 DEBUG:boto:Token: None 2023-07-29T00:37:47.457 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:37:47.457 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tR9H/afbwQtDsm7Se633bcF3JJ4= 2023-07-29T00:37:47.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tR9H/afbwQtDsm7Se633bcF3JJ4=', 'Content-Length': '0'} 2023-07-29T00:37:47.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000038cf41227a4087e-0064c45f5b-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.460 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:37:47.460 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:37:47.461 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:37:47.461 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:37:47.461 DEBUG:boto:Method: HEAD 2023-07-29T00:37:47.462 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:37:47.462 DEBUG:boto:Data: 2023-07-29T00:37:47.462 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.463 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:47.463 DEBUG:boto:Port: 8000 2023-07-29T00:37:47.463 DEBUG:boto:Params: {} 2023-07-29T00:37:47.464 DEBUG:boto:Token: None 2023-07-29T00:37:47.464 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:37:47.465 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tR9H/afbwQtDsm7Se633bcF3JJ4= 2023-07-29T00:37:47.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tR9H/afbwQtDsm7Se633bcF3JJ4=', 'Content-Length': '0'} 2023-07-29T00:37:47.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9d452a41dad80fa-0064c45f5b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-07-29T00:37:47.468 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:37:47.468 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:37:47.469 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:37:47.469 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:37:47.470 DEBUG:boto:Method: GET 2023-07-29T00:37:47.470 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:37:47.470 DEBUG:boto:Data: 2023-07-29T00:37:47.471 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.471 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:47.471 DEBUG:boto:Port: 8001 2023-07-29T00:37:47.472 DEBUG:boto:Params: {} 2023-07-29T00:37:47.472 DEBUG:boto:Token: None 2023-07-29T00:37:47.473 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:37:47.473 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+aVplbhn04+7tVhUQ1IzokY9Oq8= 2023-07-29T00:37:47.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+aVplbhn04+7tVhUQ1IzokY9Oq8=', 'Content-Length': '0'} 2023-07-29T00:37:47.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000220d2c71958d9388-0064c45f5b-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.476 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:37:47.476 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:37:47.477 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:37:47.477 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:37:47.478 DEBUG:boto:Method: GET 2023-07-29T00:37:47.478 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:37:47.478 DEBUG:boto:Data: 2023-07-29T00:37:47.479 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.479 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:47.479 DEBUG:boto:Port: 8000 2023-07-29T00:37:47.480 DEBUG:boto:Params: {} 2023-07-29T00:37:47.480 DEBUG:boto:Token: None 2023-07-29T00:37:47.481 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:37:47.481 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+aVplbhn04+7tVhUQ1IzokY9Oq8= 2023-07-29T00:37:47.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+aVplbhn04+7tVhUQ1IzokY9Oq8=', 'Content-Length': '0'} 2023-07-29T00:37:47.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000dd7bfe2678eaed2-0064c45f5b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.484 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:47.484 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:47.485 DEBUG:boto:path=/hiiqos-59/?prefix=obj11 2023-07-29T00:37:47.485 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj11 2023-07-29T00:37:47.485 DEBUG:boto:Method: GET 2023-07-29T00:37:47.486 DEBUG:boto:Path: /hiiqos-59/?prefix=obj11 2023-07-29T00:37:47.486 DEBUG:boto:Data: 2023-07-29T00:37:47.486 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.487 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:47.487 DEBUG:boto:Port: 8001 2023-07-29T00:37:47.488 DEBUG:boto:Params: {} 2023-07-29T00:37:47.488 DEBUG:boto:Token: None 2023-07-29T00:37:47.488 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/ 2023-07-29T00:37:47.489 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ocIXDkViifNqT9g/BGplHu9r558= 2023-07-29T00:37:47.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ocIXDkViifNqT9g/BGplHu9r558=', 'Content-Length': '0'} 2023-07-29T00:37:47.491 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003577a8a5afe46c20-0064c45f5b-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.492 DEBUG:boto:b'hiiqos-59obj111000falseobj112023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:47.493 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:47.493 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:47.494 DEBUG:boto:path=/hiiqos-59/?prefix=obj11 2023-07-29T00:37:47.494 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj11 2023-07-29T00:37:47.494 DEBUG:boto:Method: GET 2023-07-29T00:37:47.495 DEBUG:boto:Path: /hiiqos-59/?prefix=obj11 2023-07-29T00:37:47.495 DEBUG:boto:Data: 2023-07-29T00:37:47.495 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.496 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:47.496 DEBUG:boto:Port: 8000 2023-07-29T00:37:47.497 DEBUG:boto:Params: {} 2023-07-29T00:37:47.497 DEBUG:boto:Token: None 2023-07-29T00:37:47.498 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/ 2023-07-29T00:37:47.498 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ocIXDkViifNqT9g/BGplHu9r558= 2023-07-29T00:37:47.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ocIXDkViifNqT9g/BGplHu9r558=', 'Content-Length': '0'} 2023-07-29T00:37:47.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094319ce7b9a4769d-0064c45f5b-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.503 DEBUG:boto:b'hiiqos-59obj111000falseobj112023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:47.504 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:37:47.504 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:37:47.504 DEBUG:boto:Method: HEAD 2023-07-29T00:37:47.505 DEBUG:boto:Path: /hiiqos-59/obj11 2023-07-29T00:37:47.505 DEBUG:boto:Data: 2023-07-29T00:37:47.505 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.506 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:47.506 DEBUG:boto:Port: 8001 2023-07-29T00:37:47.506 DEBUG:boto:Params: {} 2023-07-29T00:37:47.507 DEBUG:boto:Token: None 2023-07-29T00:37:47.507 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/obj11 2023-07-29T00:37:47.508 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8mE2N9uOEzN/CHO3RHovfPy+kYo= 2023-07-29T00:37:47.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8mE2N9uOEzN/CHO3RHovfPy+kYo=', 'Content-Length': '0'} 2023-07-29T00:37:47.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000624692ed05976c8e-0064c45f5b-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.510 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:37:47.511 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:37:47.511 DEBUG:boto:Method: HEAD 2023-07-29T00:37:47.512 DEBUG:boto:Path: /hiiqos-59/obj11 2023-07-29T00:37:47.512 DEBUG:boto:Data: 2023-07-29T00:37:47.512 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.513 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:47.513 DEBUG:boto:Port: 8000 2023-07-29T00:37:47.513 DEBUG:boto:Params: {} 2023-07-29T00:37:47.514 DEBUG:boto:Token: None 2023-07-29T00:37:47.514 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/obj11 2023-07-29T00:37:47.515 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8mE2N9uOEzN/CHO3RHovfPy+kYo= 2023-07-29T00:37:47.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8mE2N9uOEzN/CHO3RHovfPy+kYo=', 'Content-Length': '0'} 2023-07-29T00:37:47.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a14a55132908031f-0064c45f5b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-07-29T00:37:47.518 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:37:47.519 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:37:47.519 DEBUG:boto:Method: GET 2023-07-29T00:37:47.519 DEBUG:boto:Path: /hiiqos-59/obj11 2023-07-29T00:37:47.520 DEBUG:boto:Data: 2023-07-29T00:37:47.520 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.520 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:47.521 DEBUG:boto:Port: 8001 2023-07-29T00:37:47.521 DEBUG:boto:Params: {} 2023-07-29T00:37:47.521 DEBUG:boto:Token: None 2023-07-29T00:37:47.522 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/obj11 2023-07-29T00:37:47.522 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6UT2UoRwBle0VCziJENW3geFfa4= 2023-07-29T00:37:47.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6UT2UoRwBle0VCziJENW3geFfa4=', 'Content-Length': '0'} 2023-07-29T00:37:47.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c228c94a7ecea6ae-0064c45f5b-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.525 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:37:47.526 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:37:47.526 DEBUG:boto:Method: GET 2023-07-29T00:37:47.526 DEBUG:boto:Path: /hiiqos-59/obj11 2023-07-29T00:37:47.527 DEBUG:boto:Data: 2023-07-29T00:37:47.527 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.527 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:47.528 DEBUG:boto:Port: 8000 2023-07-29T00:37:47.528 DEBUG:boto:Params: {} 2023-07-29T00:37:47.529 DEBUG:boto:Token: None 2023-07-29T00:37:47.529 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/obj11 2023-07-29T00:37:47.529 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6UT2UoRwBle0VCziJENW3geFfa4= 2023-07-29T00:37:47.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6UT2UoRwBle0VCziJENW3geFfa4=', 'Content-Length': '0'} 2023-07-29T00:37:47.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a3a6330d5a2c9a8-0064c45f5b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.572 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:47.573 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:47.573 DEBUG:boto:path=/hiiqos-59/?prefix=obj11 2023-07-29T00:37:47.574 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj11 2023-07-29T00:37:47.574 DEBUG:boto:Method: GET 2023-07-29T00:37:47.574 DEBUG:boto:Path: /hiiqos-59/?prefix=obj11 2023-07-29T00:37:47.575 DEBUG:boto:Data: 2023-07-29T00:37:47.575 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.575 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:47.576 DEBUG:boto:Port: 8001 2023-07-29T00:37:47.576 DEBUG:boto:Params: {} 2023-07-29T00:37:47.577 DEBUG:boto:Token: None 2023-07-29T00:37:47.577 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/ 2023-07-29T00:37:47.577 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ocIXDkViifNqT9g/BGplHu9r558= 2023-07-29T00:37:47.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ocIXDkViifNqT9g/BGplHu9r558=', 'Content-Length': '0'} 2023-07-29T00:37:47.580 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e5a45b137c3c3a8-0064c45f5b-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.581 DEBUG:boto:b'hiiqos-59obj111000falseobj112023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:47.582 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:47.582 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:47.582 DEBUG:boto:path=/hiiqos-59/?prefix=obj11 2023-07-29T00:37:47.583 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj11 2023-07-29T00:37:47.583 DEBUG:boto:Method: GET 2023-07-29T00:37:47.583 DEBUG:boto:Path: /hiiqos-59/?prefix=obj11 2023-07-29T00:37:47.584 DEBUG:boto:Data: 2023-07-29T00:37:47.584 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.584 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:47.585 DEBUG:boto:Port: 8000 2023-07-29T00:37:47.585 DEBUG:boto:Params: {} 2023-07-29T00:37:47.586 DEBUG:boto:Token: None 2023-07-29T00:37:47.586 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/ 2023-07-29T00:37:47.586 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ocIXDkViifNqT9g/BGplHu9r558= 2023-07-29T00:37:47.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ocIXDkViifNqT9g/BGplHu9r558=', 'Content-Length': '0'} 2023-07-29T00:37:47.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003de64bc13d56ad39-0064c45f5b-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.591 DEBUG:boto:b'hiiqos-59obj111000falseobj112023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:47.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-07-29T00:37:47.593 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:37:47.593 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:37:47.593 DEBUG:boto:path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:37:47.594 DEBUG:boto:auth_path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:37:47.594 DEBUG:boto:Method: GET 2023-07-29T00:37:47.594 DEBUG:boto:Path: /hiiqos-59/obj115?versionId=null 2023-07-29T00:37:47.595 DEBUG:boto:Data: 2023-07-29T00:37:47.595 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.595 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:47.596 DEBUG:boto:Port: 8001 2023-07-29T00:37:47.596 DEBUG:boto:Params: {} 2023-07-29T00:37:47.597 DEBUG:boto:Token: None 2023-07-29T00:37:47.597 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/obj115?versionId=null 2023-07-29T00:37:47.597 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lP/72vpkJ/XFh2u0jO3mNM08mrw= 2023-07-29T00:37:47.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lP/72vpkJ/XFh2u0jO3mNM08mrw=', 'Content-Length': '0'} 2023-07-29T00:37:47.600 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020ff7d9fc72393bf-0064c45f5b-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.600 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:37:47.601 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:37:47.601 DEBUG:boto:path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:37:47.602 DEBUG:boto:auth_path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:37:47.602 DEBUG:boto:Method: GET 2023-07-29T00:37:47.602 DEBUG:boto:Path: /hiiqos-59/obj115?versionId=null 2023-07-29T00:37:47.603 DEBUG:boto:Data: 2023-07-29T00:37:47.603 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.603 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:47.604 DEBUG:boto:Port: 8000 2023-07-29T00:37:47.604 DEBUG:boto:Params: {} 2023-07-29T00:37:47.604 DEBUG:boto:Token: None 2023-07-29T00:37:47.605 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/obj115?versionId=null 2023-07-29T00:37:47.605 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lP/72vpkJ/XFh2u0jO3mNM08mrw= 2023-07-29T00:37:47.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lP/72vpkJ/XFh2u0jO3mNM08mrw=', 'Content-Length': '0'} 2023-07-29T00:37:47.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059a04a766e718c48-0064c45f5b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.608 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:37:47.609 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:37:47.609 DEBUG:boto:path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:37:47.609 DEBUG:boto:auth_path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:37:47.610 DEBUG:boto:Method: HEAD 2023-07-29T00:37:47.610 DEBUG:boto:Path: /hiiqos-59/obj115?versionId=null 2023-07-29T00:37:47.610 DEBUG:boto:Data: 2023-07-29T00:37:47.611 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.611 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:47.611 DEBUG:boto:Port: 8001 2023-07-29T00:37:47.612 DEBUG:boto:Params: {} 2023-07-29T00:37:47.612 DEBUG:boto:Token: None 2023-07-29T00:37:47.612 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/obj115?versionId=null 2023-07-29T00:37:47.613 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/GOeYyZY1t6POiXocxyKOStKPgo= 2023-07-29T00:37:47.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/GOeYyZY1t6POiXocxyKOStKPgo=', 'Content-Length': '0'} 2023-07-29T00:37:47.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8da0aaf62819f9e-0064c45f5b-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.615 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:37:47.616 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:37:47.616 DEBUG:boto:path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:37:47.617 DEBUG:boto:auth_path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:37:47.617 DEBUG:boto:Method: HEAD 2023-07-29T00:37:47.617 DEBUG:boto:Path: /hiiqos-59/obj115?versionId=null 2023-07-29T00:37:47.618 DEBUG:boto:Data: 2023-07-29T00:37:47.618 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.618 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:47.619 DEBUG:boto:Port: 8000 2023-07-29T00:37:47.619 DEBUG:boto:Params: {} 2023-07-29T00:37:47.619 DEBUG:boto:Token: None 2023-07-29T00:37:47.620 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/obj115?versionId=null 2023-07-29T00:37:47.620 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/GOeYyZY1t6POiXocxyKOStKPgo= 2023-07-29T00:37:47.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/GOeYyZY1t6POiXocxyKOStKPgo=', 'Content-Length': '0'} 2023-07-29T00:37:47.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005897fbd061cbdd3d-0064c45f5b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-07-29T00:37:47.623 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:37:47.624 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:37:47.624 DEBUG:boto:path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:37:47.624 DEBUG:boto:auth_path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:37:47.625 DEBUG:boto:Method: GET 2023-07-29T00:37:47.625 DEBUG:boto:Path: /hiiqos-59/obj115?versionId=null 2023-07-29T00:37:47.625 DEBUG:boto:Data: 2023-07-29T00:37:47.626 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.626 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:47.627 DEBUG:boto:Port: 8001 2023-07-29T00:37:47.627 DEBUG:boto:Params: {} 2023-07-29T00:37:47.627 DEBUG:boto:Token: None 2023-07-29T00:37:47.628 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/obj115?versionId=null 2023-07-29T00:37:47.628 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lP/72vpkJ/XFh2u0jO3mNM08mrw= 2023-07-29T00:37:47.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lP/72vpkJ/XFh2u0jO3mNM08mrw=', 'Content-Length': '0'} 2023-07-29T00:37:47.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af550eb9b521ef27-0064c45f5b-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.631 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:37:47.631 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:37:47.632 DEBUG:boto:path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:37:47.632 DEBUG:boto:auth_path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:37:47.633 DEBUG:boto:Method: GET 2023-07-29T00:37:47.633 DEBUG:boto:Path: /hiiqos-59/obj115?versionId=null 2023-07-29T00:37:47.633 DEBUG:boto:Data: 2023-07-29T00:37:47.634 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.634 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:47.634 DEBUG:boto:Port: 8000 2023-07-29T00:37:47.635 DEBUG:boto:Params: {} 2023-07-29T00:37:47.635 DEBUG:boto:Token: None 2023-07-29T00:37:47.635 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/obj115?versionId=null 2023-07-29T00:37:47.636 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lP/72vpkJ/XFh2u0jO3mNM08mrw= 2023-07-29T00:37:47.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lP/72vpkJ/XFh2u0jO3mNM08mrw=', 'Content-Length': '0'} 2023-07-29T00:37:47.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040ce76ac62699ac9-0064c45f5b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.639 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:47.639 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:47.640 DEBUG:boto:path=/hiiqos-59/?prefix=obj115 2023-07-29T00:37:47.640 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj115 2023-07-29T00:37:47.641 DEBUG:boto:Method: GET 2023-07-29T00:37:47.641 DEBUG:boto:Path: /hiiqos-59/?prefix=obj115 2023-07-29T00:37:47.641 DEBUG:boto:Data: 2023-07-29T00:37:47.642 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.642 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:47.642 DEBUG:boto:Port: 8001 2023-07-29T00:37:47.643 DEBUG:boto:Params: {} 2023-07-29T00:37:47.643 DEBUG:boto:Token: None 2023-07-29T00:37:47.643 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/ 2023-07-29T00:37:47.644 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ocIXDkViifNqT9g/BGplHu9r558= 2023-07-29T00:37:47.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ocIXDkViifNqT9g/BGplHu9r558=', 'Content-Length': '0'} 2023-07-29T00:37:47.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f4ebc50d722515d-0064c45f5b-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.647 DEBUG:boto:b'hiiqos-59obj1151000falseobj1152023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:47.648 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:47.648 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:47.649 DEBUG:boto:path=/hiiqos-59/?prefix=obj115 2023-07-29T00:37:47.649 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj115 2023-07-29T00:37:47.649 DEBUG:boto:Method: GET 2023-07-29T00:37:47.650 DEBUG:boto:Path: /hiiqos-59/?prefix=obj115 2023-07-29T00:37:47.650 DEBUG:boto:Data: 2023-07-29T00:37:47.650 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.651 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:47.651 DEBUG:boto:Port: 8000 2023-07-29T00:37:47.651 DEBUG:boto:Params: {} 2023-07-29T00:37:47.652 DEBUG:boto:Token: None 2023-07-29T00:37:47.652 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/ 2023-07-29T00:37:47.653 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ocIXDkViifNqT9g/BGplHu9r558= 2023-07-29T00:37:47.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ocIXDkViifNqT9g/BGplHu9r558=', 'Content-Length': '0'} 2023-07-29T00:37:47.657 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c09ab818d903cef-0064c45f5b-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.658 DEBUG:boto:b'hiiqos-59obj1151000falseobj1152023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:47.658 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:37:47.659 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:37:47.659 DEBUG:boto:Method: HEAD 2023-07-29T00:37:47.660 DEBUG:boto:Path: /hiiqos-59/obj115 2023-07-29T00:37:47.660 DEBUG:boto:Data: 2023-07-29T00:37:47.660 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.661 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:47.661 DEBUG:boto:Port: 8001 2023-07-29T00:37:47.661 DEBUG:boto:Params: {} 2023-07-29T00:37:47.662 DEBUG:boto:Token: None 2023-07-29T00:37:47.662 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/obj115 2023-07-29T00:37:47.663 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:M1wNfe6GcwUWL9K/Ke5dCtu9P2Q= 2023-07-29T00:37:47.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:M1wNfe6GcwUWL9K/Ke5dCtu9P2Q=', 'Content-Length': '0'} 2023-07-29T00:37:47.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008196277f4b9fb82c-0064c45f5b-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.665 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:37:47.666 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:37:47.666 DEBUG:boto:Method: HEAD 2023-07-29T00:37:47.667 DEBUG:boto:Path: /hiiqos-59/obj115 2023-07-29T00:37:47.667 DEBUG:boto:Data: 2023-07-29T00:37:47.667 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.668 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:47.668 DEBUG:boto:Port: 8000 2023-07-29T00:37:47.668 DEBUG:boto:Params: {} 2023-07-29T00:37:47.669 DEBUG:boto:Token: None 2023-07-29T00:37:47.669 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/obj115 2023-07-29T00:37:47.669 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:M1wNfe6GcwUWL9K/Ke5dCtu9P2Q= 2023-07-29T00:37:47.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:M1wNfe6GcwUWL9K/Ke5dCtu9P2Q=', 'Content-Length': '0'} 2023-07-29T00:37:47.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f3a98fce33b07ff-0064c45f5b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-07-29T00:37:47.673 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:37:47.673 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:37:47.673 DEBUG:boto:Method: GET 2023-07-29T00:37:47.674 DEBUG:boto:Path: /hiiqos-59/obj115 2023-07-29T00:37:47.674 DEBUG:boto:Data: 2023-07-29T00:37:47.674 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.675 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:47.675 DEBUG:boto:Port: 8001 2023-07-29T00:37:47.675 DEBUG:boto:Params: {} 2023-07-29T00:37:47.676 DEBUG:boto:Token: None 2023-07-29T00:37:47.676 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/obj115 2023-07-29T00:37:47.677 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZlFroIs0XVvUJabrdefeZQt01QU= 2023-07-29T00:37:47.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZlFroIs0XVvUJabrdefeZQt01QU=', 'Content-Length': '0'} 2023-07-29T00:37:47.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb1ae5bcb959a147-0064c45f5b-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.720 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:37:47.721 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:37:47.721 DEBUG:boto:Method: GET 2023-07-29T00:37:47.722 DEBUG:boto:Path: /hiiqos-59/obj115 2023-07-29T00:37:47.722 DEBUG:boto:Data: 2023-07-29T00:37:47.722 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.723 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:47.723 DEBUG:boto:Port: 8000 2023-07-29T00:37:47.723 DEBUG:boto:Params: {} 2023-07-29T00:37:47.724 DEBUG:boto:Token: None 2023-07-29T00:37:47.724 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/obj115 2023-07-29T00:37:47.725 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZlFroIs0XVvUJabrdefeZQt01QU= 2023-07-29T00:37:47.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZlFroIs0XVvUJabrdefeZQt01QU=', 'Content-Length': '0'} 2023-07-29T00:37:47.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5e1603967670b77-0064c45f5b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.728 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:47.728 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:47.728 DEBUG:boto:path=/hiiqos-59/?prefix=obj115 2023-07-29T00:37:47.729 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj115 2023-07-29T00:37:47.729 DEBUG:boto:Method: GET 2023-07-29T00:37:47.731 DEBUG:boto:Path: /hiiqos-59/?prefix=obj115 2023-07-29T00:37:47.732 DEBUG:boto:Data: 2023-07-29T00:37:47.732 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.733 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:47.733 DEBUG:boto:Port: 8001 2023-07-29T00:37:47.734 DEBUG:boto:Params: {} 2023-07-29T00:37:47.734 DEBUG:boto:Token: None 2023-07-29T00:37:47.735 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/ 2023-07-29T00:37:47.735 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ocIXDkViifNqT9g/BGplHu9r558= 2023-07-29T00:37:47.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ocIXDkViifNqT9g/BGplHu9r558=', 'Content-Length': '0'} 2023-07-29T00:37:47.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd7439f27d434b3d-0064c45f5b-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.738 DEBUG:boto:b'hiiqos-59obj1151000falseobj1152023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:47.739 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:47.739 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:47.740 DEBUG:boto:path=/hiiqos-59/?prefix=obj115 2023-07-29T00:37:47.740 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj115 2023-07-29T00:37:47.741 DEBUG:boto:Method: GET 2023-07-29T00:37:47.741 DEBUG:boto:Path: /hiiqos-59/?prefix=obj115 2023-07-29T00:37:47.741 DEBUG:boto:Data: 2023-07-29T00:37:47.742 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.742 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:47.742 DEBUG:boto:Port: 8000 2023-07-29T00:37:47.743 DEBUG:boto:Params: {} 2023-07-29T00:37:47.743 DEBUG:boto:Token: None 2023-07-29T00:37:47.743 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/ 2023-07-29T00:37:47.744 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ocIXDkViifNqT9g/BGplHu9r558= 2023-07-29T00:37:47.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ocIXDkViifNqT9g/BGplHu9r558=', 'Content-Length': '0'} 2023-07-29T00:37:47.748 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042e541d026f3c156-0064c45f5b-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.749 DEBUG:boto:b'hiiqos-59obj1151000falseobj1152023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:47.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-07-29T00:37:47.750 DEBUG:boto:path=/hiiqos-59/obj116 2023-07-29T00:37:47.750 DEBUG:boto:auth_path=/hiiqos-59/obj116 2023-07-29T00:37:47.750 DEBUG:boto:path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:37:47.751 DEBUG:boto:auth_path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:37:47.751 DEBUG:boto:Method: GET 2023-07-29T00:37:47.751 DEBUG:boto:Path: /hiiqos-59/obj116?versionId=null 2023-07-29T00:37:47.752 DEBUG:boto:Data: 2023-07-29T00:37:47.752 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.753 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:47.753 DEBUG:boto:Port: 8001 2023-07-29T00:37:47.753 DEBUG:boto:Params: {} 2023-07-29T00:37:47.754 DEBUG:boto:Token: None 2023-07-29T00:37:47.754 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/obj116?versionId=null 2023-07-29T00:37:47.755 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JY5ZK0/AWkWtwwfI+QIpvcG+in0= 2023-07-29T00:37:47.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JY5ZK0/AWkWtwwfI+QIpvcG+in0=', 'Content-Length': '0'} 2023-07-29T00:37:47.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000873ca7ab215813a-0064c45f5b-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.758 DEBUG:boto:path=/hiiqos-59/obj116 2023-07-29T00:37:47.758 DEBUG:boto:auth_path=/hiiqos-59/obj116 2023-07-29T00:37:47.758 DEBUG:boto:path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:37:47.759 DEBUG:boto:auth_path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:37:47.759 DEBUG:boto:Method: GET 2023-07-29T00:37:47.759 DEBUG:boto:Path: /hiiqos-59/obj116?versionId=null 2023-07-29T00:37:47.760 DEBUG:boto:Data: 2023-07-29T00:37:47.760 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.761 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:47.761 DEBUG:boto:Port: 8000 2023-07-29T00:37:47.761 DEBUG:boto:Params: {} 2023-07-29T00:37:47.762 DEBUG:boto:Token: None 2023-07-29T00:37:47.762 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/obj116?versionId=null 2023-07-29T00:37:47.763 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JY5ZK0/AWkWtwwfI+QIpvcG+in0= 2023-07-29T00:37:47.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JY5ZK0/AWkWtwwfI+QIpvcG+in0=', 'Content-Length': '0'} 2023-07-29T00:37:47.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093c2f24678567efc-0064c45f5b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.766 DEBUG:boto:path=/hiiqos-59/obj116 2023-07-29T00:37:47.766 DEBUG:boto:auth_path=/hiiqos-59/obj116 2023-07-29T00:37:47.766 DEBUG:boto:path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:37:47.767 DEBUG:boto:auth_path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:37:47.767 DEBUG:boto:Method: HEAD 2023-07-29T00:37:47.767 DEBUG:boto:Path: /hiiqos-59/obj116?versionId=null 2023-07-29T00:37:47.768 DEBUG:boto:Data: 2023-07-29T00:37:47.768 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.769 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:47.769 DEBUG:boto:Port: 8001 2023-07-29T00:37:47.769 DEBUG:boto:Params: {} 2023-07-29T00:37:47.770 DEBUG:boto:Token: None 2023-07-29T00:37:47.770 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/obj116?versionId=null 2023-07-29T00:37:47.770 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mV5V11AQg+FulndeLm+wQBiLRSQ= 2023-07-29T00:37:47.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mV5V11AQg+FulndeLm+wQBiLRSQ=', 'Content-Length': '0'} 2023-07-29T00:37:47.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e308b04b92bd5356-0064c45f5b-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.773 DEBUG:boto:path=/hiiqos-59/obj116 2023-07-29T00:37:47.773 DEBUG:boto:auth_path=/hiiqos-59/obj116 2023-07-29T00:37:47.774 DEBUG:boto:path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:37:47.774 DEBUG:boto:auth_path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:37:47.774 DEBUG:boto:Method: HEAD 2023-07-29T00:37:47.775 DEBUG:boto:Path: /hiiqos-59/obj116?versionId=null 2023-07-29T00:37:47.775 DEBUG:boto:Data: 2023-07-29T00:37:47.775 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.776 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:47.776 DEBUG:boto:Port: 8000 2023-07-29T00:37:47.776 DEBUG:boto:Params: {} 2023-07-29T00:37:47.777 DEBUG:boto:Token: None 2023-07-29T00:37:47.777 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/obj116?versionId=null 2023-07-29T00:37:47.777 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mV5V11AQg+FulndeLm+wQBiLRSQ= 2023-07-29T00:37:47.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mV5V11AQg+FulndeLm+wQBiLRSQ=', 'Content-Length': '0'} 2023-07-29T00:37:47.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e52e2a225eeb510a-0064c45f5b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-07-29T00:37:47.780 DEBUG:boto:path=/hiiqos-59/obj116 2023-07-29T00:37:47.781 DEBUG:boto:auth_path=/hiiqos-59/obj116 2023-07-29T00:37:47.781 DEBUG:boto:path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:37:47.781 DEBUG:boto:auth_path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:37:47.782 DEBUG:boto:Method: GET 2023-07-29T00:37:47.782 DEBUG:boto:Path: /hiiqos-59/obj116?versionId=null 2023-07-29T00:37:47.783 DEBUG:boto:Data: 2023-07-29T00:37:47.783 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.783 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:47.784 DEBUG:boto:Port: 8001 2023-07-29T00:37:47.784 DEBUG:boto:Params: {} 2023-07-29T00:37:47.784 DEBUG:boto:Token: None 2023-07-29T00:37:47.785 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/obj116?versionId=null 2023-07-29T00:37:47.785 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JY5ZK0/AWkWtwwfI+QIpvcG+in0= 2023-07-29T00:37:47.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JY5ZK0/AWkWtwwfI+QIpvcG+in0=', 'Content-Length': '0'} 2023-07-29T00:37:47.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ebcfb899758b512-0064c45f5b-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.828 DEBUG:boto:path=/hiiqos-59/obj116 2023-07-29T00:37:47.829 DEBUG:boto:auth_path=/hiiqos-59/obj116 2023-07-29T00:37:47.829 DEBUG:boto:path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:37:47.830 DEBUG:boto:auth_path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:37:47.830 DEBUG:boto:Method: GET 2023-07-29T00:37:47.830 DEBUG:boto:Path: /hiiqos-59/obj116?versionId=null 2023-07-29T00:37:47.831 DEBUG:boto:Data: 2023-07-29T00:37:47.831 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.831 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:47.832 DEBUG:boto:Port: 8000 2023-07-29T00:37:47.832 DEBUG:boto:Params: {} 2023-07-29T00:37:47.832 DEBUG:boto:Token: None 2023-07-29T00:37:47.833 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/obj116?versionId=null 2023-07-29T00:37:47.833 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JY5ZK0/AWkWtwwfI+QIpvcG+in0= 2023-07-29T00:37:47.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JY5ZK0/AWkWtwwfI+QIpvcG+in0=', 'Content-Length': '0'} 2023-07-29T00:37:47.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6ed365ca365c4a6-0064c45f5b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.836 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:47.837 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:47.837 DEBUG:boto:path=/hiiqos-59/?prefix=obj116 2023-07-29T00:37:47.837 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj116 2023-07-29T00:37:47.838 DEBUG:boto:Method: GET 2023-07-29T00:37:47.838 DEBUG:boto:Path: /hiiqos-59/?prefix=obj116 2023-07-29T00:37:47.839 DEBUG:boto:Data: 2023-07-29T00:37:47.839 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.839 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:47.840 DEBUG:boto:Port: 8001 2023-07-29T00:37:47.840 DEBUG:boto:Params: {} 2023-07-29T00:37:47.841 DEBUG:boto:Token: None 2023-07-29T00:37:47.841 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/ 2023-07-29T00:37:47.841 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ocIXDkViifNqT9g/BGplHu9r558= 2023-07-29T00:37:47.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ocIXDkViifNqT9g/BGplHu9r558=', 'Content-Length': '0'} 2023-07-29T00:37:47.844 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bdbeca737ad9426-0064c45f5b-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.845 DEBUG:boto:b'hiiqos-59obj1161000falseobj1162023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:47.846 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:47.846 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:47.846 DEBUG:boto:path=/hiiqos-59/?prefix=obj116 2023-07-29T00:37:47.847 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj116 2023-07-29T00:37:47.847 DEBUG:boto:Method: GET 2023-07-29T00:37:47.847 DEBUG:boto:Path: /hiiqos-59/?prefix=obj116 2023-07-29T00:37:47.848 DEBUG:boto:Data: 2023-07-29T00:37:47.848 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.849 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:47.849 DEBUG:boto:Port: 8000 2023-07-29T00:37:47.849 DEBUG:boto:Params: {} 2023-07-29T00:37:47.850 DEBUG:boto:Token: None 2023-07-29T00:37:47.850 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/ 2023-07-29T00:37:47.850 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ocIXDkViifNqT9g/BGplHu9r558= 2023-07-29T00:37:47.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ocIXDkViifNqT9g/BGplHu9r558=', 'Content-Length': '0'} 2023-07-29T00:37:47.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000192592df7f052e8c-0064c45f5b-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.855 DEBUG:boto:b'hiiqos-59obj1161000falseobj1162023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:47.856 DEBUG:boto:path=/hiiqos-59/obj116 2023-07-29T00:37:47.856 DEBUG:boto:auth_path=/hiiqos-59/obj116 2023-07-29T00:37:47.857 DEBUG:boto:Method: HEAD 2023-07-29T00:37:47.857 DEBUG:boto:Path: /hiiqos-59/obj116 2023-07-29T00:37:47.857 DEBUG:boto:Data: 2023-07-29T00:37:47.858 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.858 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:47.859 DEBUG:boto:Port: 8001 2023-07-29T00:37:47.859 DEBUG:boto:Params: {} 2023-07-29T00:37:47.860 DEBUG:boto:Token: None 2023-07-29T00:37:47.860 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/obj116 2023-07-29T00:37:47.860 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YaG3WbQDbdEzvwN8cNJNF+hW1pA= 2023-07-29T00:37:47.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YaG3WbQDbdEzvwN8cNJNF+hW1pA=', 'Content-Length': '0'} 2023-07-29T00:37:47.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e86d0c53fcd84cd3-0064c45f5b-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.863 DEBUG:boto:path=/hiiqos-59/obj116 2023-07-29T00:37:47.863 DEBUG:boto:auth_path=/hiiqos-59/obj116 2023-07-29T00:37:47.864 DEBUG:boto:Method: HEAD 2023-07-29T00:37:47.864 DEBUG:boto:Path: /hiiqos-59/obj116 2023-07-29T00:37:47.864 DEBUG:boto:Data: 2023-07-29T00:37:47.865 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.865 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:47.865 DEBUG:boto:Port: 8000 2023-07-29T00:37:47.866 DEBUG:boto:Params: {} 2023-07-29T00:37:47.866 DEBUG:boto:Token: None 2023-07-29T00:37:47.866 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/obj116 2023-07-29T00:37:47.867 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YaG3WbQDbdEzvwN8cNJNF+hW1pA= 2023-07-29T00:37:47.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YaG3WbQDbdEzvwN8cNJNF+hW1pA=', 'Content-Length': '0'} 2023-07-29T00:37:47.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045ca99ead954cb77-0064c45f5b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-07-29T00:37:47.870 DEBUG:boto:path=/hiiqos-59/obj116 2023-07-29T00:37:47.870 DEBUG:boto:auth_path=/hiiqos-59/obj116 2023-07-29T00:37:47.871 DEBUG:boto:Method: GET 2023-07-29T00:37:47.871 DEBUG:boto:Path: /hiiqos-59/obj116 2023-07-29T00:37:47.872 DEBUG:boto:Data: 2023-07-29T00:37:47.872 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.872 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:47.873 DEBUG:boto:Port: 8001 2023-07-29T00:37:47.873 DEBUG:boto:Params: {} 2023-07-29T00:37:47.873 DEBUG:boto:Token: None 2023-07-29T00:37:47.874 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/obj116 2023-07-29T00:37:47.874 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/klyI9jEqBGHc2QN2KnPI+WZyqc= 2023-07-29T00:37:47.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/klyI9jEqBGHc2QN2KnPI+WZyqc=', 'Content-Length': '0'} 2023-07-29T00:37:47.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000089961567925b0f9-0064c45f5b-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.877 DEBUG:boto:path=/hiiqos-59/obj116 2023-07-29T00:37:47.877 DEBUG:boto:auth_path=/hiiqos-59/obj116 2023-07-29T00:37:47.878 DEBUG:boto:Method: GET 2023-07-29T00:37:47.878 DEBUG:boto:Path: /hiiqos-59/obj116 2023-07-29T00:37:47.879 DEBUG:boto:Data: 2023-07-29T00:37:47.879 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.879 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:47.880 DEBUG:boto:Port: 8000 2023-07-29T00:37:47.880 DEBUG:boto:Params: {} 2023-07-29T00:37:47.880 DEBUG:boto:Token: None 2023-07-29T00:37:47.881 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/obj116 2023-07-29T00:37:47.881 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/klyI9jEqBGHc2QN2KnPI+WZyqc= 2023-07-29T00:37:47.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/klyI9jEqBGHc2QN2KnPI+WZyqc=', 'Content-Length': '0'} 2023-07-29T00:37:47.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025fe7cbbb564aaf7-0064c45f5b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.924 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:47.925 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:47.925 DEBUG:boto:path=/hiiqos-59/?prefix=obj116 2023-07-29T00:37:47.926 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj116 2023-07-29T00:37:47.926 DEBUG:boto:Method: GET 2023-07-29T00:37:47.926 DEBUG:boto:Path: /hiiqos-59/?prefix=obj116 2023-07-29T00:37:47.927 DEBUG:boto:Data: 2023-07-29T00:37:47.927 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.927 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:47.928 DEBUG:boto:Port: 8001 2023-07-29T00:37:47.928 DEBUG:boto:Params: {} 2023-07-29T00:37:47.929 DEBUG:boto:Token: None 2023-07-29T00:37:47.929 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/ 2023-07-29T00:37:47.929 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ocIXDkViifNqT9g/BGplHu9r558= 2023-07-29T00:37:47.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ocIXDkViifNqT9g/BGplHu9r558=', 'Content-Length': '0'} 2023-07-29T00:37:47.932 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce2737a1a8d11d21-0064c45f5b-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.933 DEBUG:boto:b'hiiqos-59obj1161000falseobj1162023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:47.934 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:47.934 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:47.934 DEBUG:boto:path=/hiiqos-59/?prefix=obj116 2023-07-29T00:37:47.935 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj116 2023-07-29T00:37:47.935 DEBUG:boto:Method: GET 2023-07-29T00:37:47.935 DEBUG:boto:Path: /hiiqos-59/?prefix=obj116 2023-07-29T00:37:47.936 DEBUG:boto:Data: 2023-07-29T00:37:47.936 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.936 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:47.937 DEBUG:boto:Port: 8000 2023-07-29T00:37:47.937 DEBUG:boto:Params: {} 2023-07-29T00:37:47.938 DEBUG:boto:Token: None 2023-07-29T00:37:47.938 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/ 2023-07-29T00:37:47.938 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ocIXDkViifNqT9g/BGplHu9r558= 2023-07-29T00:37:47.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ocIXDkViifNqT9g/BGplHu9r558=', 'Content-Length': '0'} 2023-07-29T00:37:47.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032c530de9502b330-0064c45f5b-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.943 DEBUG:boto:b'hiiqos-59obj1161000falseobj1162023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:47.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-07-29T00:37:47.944 DEBUG:boto:path=/hiiqos-59/obj124 2023-07-29T00:37:47.944 DEBUG:boto:auth_path=/hiiqos-59/obj124 2023-07-29T00:37:47.945 DEBUG:boto:path=/hiiqos-59/obj124?versionId=null 2023-07-29T00:37:47.945 DEBUG:boto:auth_path=/hiiqos-59/obj124?versionId=null 2023-07-29T00:37:47.946 DEBUG:boto:Method: GET 2023-07-29T00:37:47.946 DEBUG:boto:Path: /hiiqos-59/obj124?versionId=null 2023-07-29T00:37:47.946 DEBUG:boto:Data: 2023-07-29T00:37:47.947 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.947 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:47.947 DEBUG:boto:Port: 8001 2023-07-29T00:37:47.948 DEBUG:boto:Params: {} 2023-07-29T00:37:47.948 DEBUG:boto:Token: None 2023-07-29T00:37:47.949 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/obj124?versionId=null 2023-07-29T00:37:47.949 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6OxnC4Rxfqu3LBuNiBu3Ci9Dlzo= 2023-07-29T00:37:47.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6OxnC4Rxfqu3LBuNiBu3Ci9Dlzo=', 'Content-Length': '0'} 2023-07-29T00:37:47.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f56ce3d32a12a1de-0064c45f5b-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:47.992 DEBUG:boto:path=/hiiqos-59/obj124 2023-07-29T00:37:47.993 DEBUG:boto:auth_path=/hiiqos-59/obj124 2023-07-29T00:37:47.993 DEBUG:boto:path=/hiiqos-59/obj124?versionId=null 2023-07-29T00:37:47.993 DEBUG:boto:auth_path=/hiiqos-59/obj124?versionId=null 2023-07-29T00:37:47.994 DEBUG:boto:Method: GET 2023-07-29T00:37:47.994 DEBUG:boto:Path: /hiiqos-59/obj124?versionId=null 2023-07-29T00:37:47.995 DEBUG:boto:Data: 2023-07-29T00:37:47.995 DEBUG:boto:Headers: {} 2023-07-29T00:37:47.995 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:47.996 DEBUG:boto:Port: 8000 2023-07-29T00:37:47.996 DEBUG:boto:Params: {} 2023-07-29T00:37:47.996 DEBUG:boto:Token: None 2023-07-29T00:37:47.997 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:47 GMT /hiiqos-59/obj124?versionId=null 2023-07-29T00:37:47.997 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6OxnC4Rxfqu3LBuNiBu3Ci9Dlzo= 2023-07-29T00:37:47.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6OxnC4Rxfqu3LBuNiBu3Ci9Dlzo=', 'Content-Length': '0'} 2023-07-29T00:37:48.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e67cd4f081105d93-0064c45f5b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.000 DEBUG:boto:path=/hiiqos-59/obj124 2023-07-29T00:37:48.001 DEBUG:boto:auth_path=/hiiqos-59/obj124 2023-07-29T00:37:48.001 DEBUG:boto:path=/hiiqos-59/obj124?versionId=null 2023-07-29T00:37:48.001 DEBUG:boto:auth_path=/hiiqos-59/obj124?versionId=null 2023-07-29T00:37:48.002 DEBUG:boto:Method: HEAD 2023-07-29T00:37:48.002 DEBUG:boto:Path: /hiiqos-59/obj124?versionId=null 2023-07-29T00:37:48.003 DEBUG:boto:Data: 2023-07-29T00:37:48.003 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.003 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:48.003 DEBUG:boto:Port: 8001 2023-07-29T00:37:48.004 DEBUG:boto:Params: {} 2023-07-29T00:37:48.004 DEBUG:boto:Token: None 2023-07-29T00:37:48.005 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj124?versionId=null 2023-07-29T00:37:48.005 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WoZ/xBK1NPM3hTIpoqEwyoCBas0= 2023-07-29T00:37:48.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WoZ/xBK1NPM3hTIpoqEwyoCBas0=', 'Content-Length': '0'} 2023-07-29T00:37:48.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a011134901f402f-0064c45f5b-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.008 DEBUG:boto:path=/hiiqos-59/obj124 2023-07-29T00:37:48.008 DEBUG:boto:auth_path=/hiiqos-59/obj124 2023-07-29T00:37:48.009 DEBUG:boto:path=/hiiqos-59/obj124?versionId=null 2023-07-29T00:37:48.009 DEBUG:boto:auth_path=/hiiqos-59/obj124?versionId=null 2023-07-29T00:37:48.009 DEBUG:boto:Method: HEAD 2023-07-29T00:37:48.010 DEBUG:boto:Path: /hiiqos-59/obj124?versionId=null 2023-07-29T00:37:48.010 DEBUG:boto:Data: 2023-07-29T00:37:48.010 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.011 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:48.011 DEBUG:boto:Port: 8000 2023-07-29T00:37:48.011 DEBUG:boto:Params: {} 2023-07-29T00:37:48.012 DEBUG:boto:Token: None 2023-07-29T00:37:48.012 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj124?versionId=null 2023-07-29T00:37:48.012 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WoZ/xBK1NPM3hTIpoqEwyoCBas0= 2023-07-29T00:37:48.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WoZ/xBK1NPM3hTIpoqEwyoCBas0=', 'Content-Length': '0'} 2023-07-29T00:37:48.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005df7263d2203ed42-0064c45f5b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-07-29T00:37:48.016 DEBUG:boto:path=/hiiqos-59/obj124 2023-07-29T00:37:48.016 DEBUG:boto:auth_path=/hiiqos-59/obj124 2023-07-29T00:37:48.016 DEBUG:boto:path=/hiiqos-59/obj124?versionId=null 2023-07-29T00:37:48.017 DEBUG:boto:auth_path=/hiiqos-59/obj124?versionId=null 2023-07-29T00:37:48.017 DEBUG:boto:Method: GET 2023-07-29T00:37:48.018 DEBUG:boto:Path: /hiiqos-59/obj124?versionId=null 2023-07-29T00:37:48.018 DEBUG:boto:Data: 2023-07-29T00:37:48.018 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.019 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:48.019 DEBUG:boto:Port: 8001 2023-07-29T00:37:48.019 DEBUG:boto:Params: {} 2023-07-29T00:37:48.020 DEBUG:boto:Token: None 2023-07-29T00:37:48.020 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj124?versionId=null 2023-07-29T00:37:48.021 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:N+MHMhoaY2nOcVZqOOgMD3cOVHo= 2023-07-29T00:37:48.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:N+MHMhoaY2nOcVZqOOgMD3cOVHo=', 'Content-Length': '0'} 2023-07-29T00:37:48.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ff7dcca33a1e4e3-0064c45f5b-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.024 DEBUG:boto:path=/hiiqos-59/obj124 2023-07-29T00:37:48.024 DEBUG:boto:auth_path=/hiiqos-59/obj124 2023-07-29T00:37:48.024 DEBUG:boto:path=/hiiqos-59/obj124?versionId=null 2023-07-29T00:37:48.025 DEBUG:boto:auth_path=/hiiqos-59/obj124?versionId=null 2023-07-29T00:37:48.025 DEBUG:boto:Method: GET 2023-07-29T00:37:48.026 DEBUG:boto:Path: /hiiqos-59/obj124?versionId=null 2023-07-29T00:37:48.026 DEBUG:boto:Data: 2023-07-29T00:37:48.026 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.027 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:48.027 DEBUG:boto:Port: 8000 2023-07-29T00:37:48.027 DEBUG:boto:Params: {} 2023-07-29T00:37:48.028 DEBUG:boto:Token: None 2023-07-29T00:37:48.028 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj124?versionId=null 2023-07-29T00:37:48.028 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:N+MHMhoaY2nOcVZqOOgMD3cOVHo= 2023-07-29T00:37:48.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:N+MHMhoaY2nOcVZqOOgMD3cOVHo=', 'Content-Length': '0'} 2023-07-29T00:37:48.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008b8b0b507cffdea-0064c45f5b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.032 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:48.032 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:48.032 DEBUG:boto:path=/hiiqos-59/?prefix=obj124 2023-07-29T00:37:48.033 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj124 2023-07-29T00:37:48.033 DEBUG:boto:Method: GET 2023-07-29T00:37:48.033 DEBUG:boto:Path: /hiiqos-59/?prefix=obj124 2023-07-29T00:37:48.034 DEBUG:boto:Data: 2023-07-29T00:37:48.034 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.034 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:48.035 DEBUG:boto:Port: 8001 2023-07-29T00:37:48.035 DEBUG:boto:Params: {} 2023-07-29T00:37:48.036 DEBUG:boto:Token: None 2023-07-29T00:37:48.036 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/ 2023-07-29T00:37:48.036 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28= 2023-07-29T00:37:48.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28=', 'Content-Length': '0'} 2023-07-29T00:37:48.039 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021c150bf2e33ba21-0064c45f5b-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.040 DEBUG:boto:b'hiiqos-59obj1241000falseobj1242023-07-29T00:35:31.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:48.041 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:48.041 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:48.041 DEBUG:boto:path=/hiiqos-59/?prefix=obj124 2023-07-29T00:37:48.042 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj124 2023-07-29T00:37:48.042 DEBUG:boto:Method: GET 2023-07-29T00:37:48.042 DEBUG:boto:Path: /hiiqos-59/?prefix=obj124 2023-07-29T00:37:48.043 DEBUG:boto:Data: 2023-07-29T00:37:48.043 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.044 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:48.044 DEBUG:boto:Port: 8000 2023-07-29T00:37:48.044 DEBUG:boto:Params: {} 2023-07-29T00:37:48.045 DEBUG:boto:Token: None 2023-07-29T00:37:48.045 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/ 2023-07-29T00:37:48.045 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28= 2023-07-29T00:37:48.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28=', 'Content-Length': '0'} 2023-07-29T00:37:48.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a8eee0a9120ede5-0064c45f5b-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.050 DEBUG:boto:b'hiiqos-59obj1241000falseobj1242023-07-29T00:35:31.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:48.051 DEBUG:boto:path=/hiiqos-59/obj124 2023-07-29T00:37:48.051 DEBUG:boto:auth_path=/hiiqos-59/obj124 2023-07-29T00:37:48.052 DEBUG:boto:Method: HEAD 2023-07-29T00:37:48.052 DEBUG:boto:Path: /hiiqos-59/obj124 2023-07-29T00:37:48.053 DEBUG:boto:Data: 2023-07-29T00:37:48.053 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.053 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:48.054 DEBUG:boto:Port: 8001 2023-07-29T00:37:48.054 DEBUG:boto:Params: {} 2023-07-29T00:37:48.054 DEBUG:boto:Token: None 2023-07-29T00:37:48.055 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj124 2023-07-29T00:37:48.055 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UKWJ7Ltx5J/cVMLuSxpnV35kIGc= 2023-07-29T00:37:48.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UKWJ7Ltx5J/cVMLuSxpnV35kIGc=', 'Content-Length': '0'} 2023-07-29T00:37:48.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065e146d0ba5cda83-0064c45f5b-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.058 DEBUG:boto:path=/hiiqos-59/obj124 2023-07-29T00:37:48.058 DEBUG:boto:auth_path=/hiiqos-59/obj124 2023-07-29T00:37:48.058 DEBUG:boto:Method: HEAD 2023-07-29T00:37:48.059 DEBUG:boto:Path: /hiiqos-59/obj124 2023-07-29T00:37:48.059 DEBUG:boto:Data: 2023-07-29T00:37:48.059 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.060 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:48.060 DEBUG:boto:Port: 8000 2023-07-29T00:37:48.061 DEBUG:boto:Params: {} 2023-07-29T00:37:48.061 DEBUG:boto:Token: None 2023-07-29T00:37:48.061 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj124 2023-07-29T00:37:48.062 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UKWJ7Ltx5J/cVMLuSxpnV35kIGc= 2023-07-29T00:37:48.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UKWJ7Ltx5J/cVMLuSxpnV35kIGc=', 'Content-Length': '0'} 2023-07-29T00:37:48.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fc632c8fb48e186-0064c45f5c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-07-29T00:37:48.065 DEBUG:boto:path=/hiiqos-59/obj124 2023-07-29T00:37:48.065 DEBUG:boto:auth_path=/hiiqos-59/obj124 2023-07-29T00:37:48.066 DEBUG:boto:Method: GET 2023-07-29T00:37:48.066 DEBUG:boto:Path: /hiiqos-59/obj124 2023-07-29T00:37:48.066 DEBUG:boto:Data: 2023-07-29T00:37:48.067 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.067 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:48.067 DEBUG:boto:Port: 8001 2023-07-29T00:37:48.068 DEBUG:boto:Params: {} 2023-07-29T00:37:48.068 DEBUG:boto:Token: None 2023-07-29T00:37:48.069 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj124 2023-07-29T00:37:48.069 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sjYrYZUcSOzAGM/GlSIJlx1+USw= 2023-07-29T00:37:48.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sjYrYZUcSOzAGM/GlSIJlx1+USw=', 'Content-Length': '0'} 2023-07-29T00:37:48.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f808962f975988c8-0064c45f5c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.072 DEBUG:boto:path=/hiiqos-59/obj124 2023-07-29T00:37:48.072 DEBUG:boto:auth_path=/hiiqos-59/obj124 2023-07-29T00:37:48.073 DEBUG:boto:Method: GET 2023-07-29T00:37:48.073 DEBUG:boto:Path: /hiiqos-59/obj124 2023-07-29T00:37:48.074 DEBUG:boto:Data: 2023-07-29T00:37:48.074 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.074 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:48.075 DEBUG:boto:Port: 8000 2023-07-29T00:37:48.075 DEBUG:boto:Params: {} 2023-07-29T00:37:48.075 DEBUG:boto:Token: None 2023-07-29T00:37:48.076 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj124 2023-07-29T00:37:48.076 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sjYrYZUcSOzAGM/GlSIJlx1+USw= 2023-07-29T00:37:48.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sjYrYZUcSOzAGM/GlSIJlx1+USw=', 'Content-Length': '0'} 2023-07-29T00:37:48.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff94d18c8843920e-0064c45f5c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.079 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:48.080 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:48.080 DEBUG:boto:path=/hiiqos-59/?prefix=obj124 2023-07-29T00:37:48.080 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj124 2023-07-29T00:37:48.081 DEBUG:boto:Method: GET 2023-07-29T00:37:48.081 DEBUG:boto:Path: /hiiqos-59/?prefix=obj124 2023-07-29T00:37:48.081 DEBUG:boto:Data: 2023-07-29T00:37:48.082 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.082 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:48.082 DEBUG:boto:Port: 8001 2023-07-29T00:37:48.083 DEBUG:boto:Params: {} 2023-07-29T00:37:48.083 DEBUG:boto:Token: None 2023-07-29T00:37:48.084 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/ 2023-07-29T00:37:48.084 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28= 2023-07-29T00:37:48.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28=', 'Content-Length': '0'} 2023-07-29T00:37:48.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cab14eebd2a6b2f8-0064c45f5c-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.087 DEBUG:boto:b'hiiqos-59obj1241000falseobj1242023-07-29T00:35:31.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:48.088 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:48.088 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:48.089 DEBUG:boto:path=/hiiqos-59/?prefix=obj124 2023-07-29T00:37:48.089 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj124 2023-07-29T00:37:48.090 DEBUG:boto:Method: GET 2023-07-29T00:37:48.090 DEBUG:boto:Path: /hiiqos-59/?prefix=obj124 2023-07-29T00:37:48.090 DEBUG:boto:Data: 2023-07-29T00:37:48.091 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.091 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:48.091 DEBUG:boto:Port: 8000 2023-07-29T00:37:48.092 DEBUG:boto:Params: {} 2023-07-29T00:37:48.092 DEBUG:boto:Token: None 2023-07-29T00:37:48.092 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/ 2023-07-29T00:37:48.093 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28= 2023-07-29T00:37:48.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28=', 'Content-Length': '0'} 2023-07-29T00:37:48.097 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd389be16a24e716-0064c45f5c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.097 DEBUG:boto:b'hiiqos-59obj1241000falseobj1242023-07-29T00:35:31.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:48.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-07-29T00:37:48.098 DEBUG:boto:path=/hiiqos-59/obj145 2023-07-29T00:37:48.099 DEBUG:boto:auth_path=/hiiqos-59/obj145 2023-07-29T00:37:48.099 DEBUG:boto:path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:37:48.099 DEBUG:boto:auth_path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:37:48.100 DEBUG:boto:Method: GET 2023-07-29T00:37:48.100 DEBUG:boto:Path: /hiiqos-59/obj145?versionId=null 2023-07-29T00:37:48.101 DEBUG:boto:Data: 2023-07-29T00:37:48.101 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.101 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:48.102 DEBUG:boto:Port: 8001 2023-07-29T00:37:48.102 DEBUG:boto:Params: {} 2023-07-29T00:37:48.102 DEBUG:boto:Token: None 2023-07-29T00:37:48.103 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj145?versionId=null 2023-07-29T00:37:48.103 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VYsOHfpyy/yJE3xxSKOEDBYvay0= 2023-07-29T00:37:48.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VYsOHfpyy/yJE3xxSKOEDBYvay0=', 'Content-Length': '0'} 2023-07-29T00:37:48.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009746cde7f5d7a3c6-0064c45f5c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.106 DEBUG:boto:path=/hiiqos-59/obj145 2023-07-29T00:37:48.107 DEBUG:boto:auth_path=/hiiqos-59/obj145 2023-07-29T00:37:48.107 DEBUG:boto:path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:37:48.107 DEBUG:boto:auth_path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:37:48.108 DEBUG:boto:Method: GET 2023-07-29T00:37:48.108 DEBUG:boto:Path: /hiiqos-59/obj145?versionId=null 2023-07-29T00:37:48.108 DEBUG:boto:Data: 2023-07-29T00:37:48.109 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.109 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:48.109 DEBUG:boto:Port: 8000 2023-07-29T00:37:48.110 DEBUG:boto:Params: {} 2023-07-29T00:37:48.110 DEBUG:boto:Token: None 2023-07-29T00:37:48.111 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj145?versionId=null 2023-07-29T00:37:48.111 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VYsOHfpyy/yJE3xxSKOEDBYvay0= 2023-07-29T00:37:48.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VYsOHfpyy/yJE3xxSKOEDBYvay0=', 'Content-Length': '0'} 2023-07-29T00:37:48.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000424b87f0506665f8-0064c45f5c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.156 DEBUG:boto:path=/hiiqos-59/obj145 2023-07-29T00:37:48.157 DEBUG:boto:auth_path=/hiiqos-59/obj145 2023-07-29T00:37:48.157 DEBUG:boto:path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:37:48.157 DEBUG:boto:auth_path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:37:48.158 DEBUG:boto:Method: HEAD 2023-07-29T00:37:48.158 DEBUG:boto:Path: /hiiqos-59/obj145?versionId=null 2023-07-29T00:37:48.158 DEBUG:boto:Data: 2023-07-29T00:37:48.159 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.159 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:48.160 DEBUG:boto:Port: 8001 2023-07-29T00:37:48.160 DEBUG:boto:Params: {} 2023-07-29T00:37:48.160 DEBUG:boto:Token: None 2023-07-29T00:37:48.161 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj145?versionId=null 2023-07-29T00:37:48.161 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P3um2bStqJ95FVy5+KW105u4+b4= 2023-07-29T00:37:48.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P3um2bStqJ95FVy5+KW105u4+b4=', 'Content-Length': '0'} 2023-07-29T00:37:48.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002c550fb457a1e0b-0064c45f5c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.164 DEBUG:boto:path=/hiiqos-59/obj145 2023-07-29T00:37:48.164 DEBUG:boto:auth_path=/hiiqos-59/obj145 2023-07-29T00:37:48.165 DEBUG:boto:path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:37:48.165 DEBUG:boto:auth_path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:37:48.165 DEBUG:boto:Method: HEAD 2023-07-29T00:37:48.166 DEBUG:boto:Path: /hiiqos-59/obj145?versionId=null 2023-07-29T00:37:48.166 DEBUG:boto:Data: 2023-07-29T00:37:48.166 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.167 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:48.167 DEBUG:boto:Port: 8000 2023-07-29T00:37:48.167 DEBUG:boto:Params: {} 2023-07-29T00:37:48.168 DEBUG:boto:Token: None 2023-07-29T00:37:48.168 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj145?versionId=null 2023-07-29T00:37:48.169 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P3um2bStqJ95FVy5+KW105u4+b4= 2023-07-29T00:37:48.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P3um2bStqJ95FVy5+KW105u4+b4=', 'Content-Length': '0'} 2023-07-29T00:37:48.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000710609fec2e1725-0064c45f5c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-07-29T00:37:48.172 DEBUG:boto:path=/hiiqos-59/obj145 2023-07-29T00:37:48.172 DEBUG:boto:auth_path=/hiiqos-59/obj145 2023-07-29T00:37:48.173 DEBUG:boto:path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:37:48.173 DEBUG:boto:auth_path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:37:48.173 DEBUG:boto:Method: GET 2023-07-29T00:37:48.174 DEBUG:boto:Path: /hiiqos-59/obj145?versionId=null 2023-07-29T00:37:48.174 DEBUG:boto:Data: 2023-07-29T00:37:48.174 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.175 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:48.175 DEBUG:boto:Port: 8001 2023-07-29T00:37:48.176 DEBUG:boto:Params: {} 2023-07-29T00:37:48.176 DEBUG:boto:Token: None 2023-07-29T00:37:48.176 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj145?versionId=null 2023-07-29T00:37:48.177 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VYsOHfpyy/yJE3xxSKOEDBYvay0= 2023-07-29T00:37:48.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VYsOHfpyy/yJE3xxSKOEDBYvay0=', 'Content-Length': '0'} 2023-07-29T00:37:48.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae7c73299eea46a9-0064c45f5c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.180 DEBUG:boto:path=/hiiqos-59/obj145 2023-07-29T00:37:48.180 DEBUG:boto:auth_path=/hiiqos-59/obj145 2023-07-29T00:37:48.181 DEBUG:boto:path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:37:48.181 DEBUG:boto:auth_path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:37:48.181 DEBUG:boto:Method: GET 2023-07-29T00:37:48.182 DEBUG:boto:Path: /hiiqos-59/obj145?versionId=null 2023-07-29T00:37:48.182 DEBUG:boto:Data: 2023-07-29T00:37:48.182 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.183 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:48.183 DEBUG:boto:Port: 8000 2023-07-29T00:37:48.184 DEBUG:boto:Params: {} 2023-07-29T00:37:48.184 DEBUG:boto:Token: None 2023-07-29T00:37:48.184 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj145?versionId=null 2023-07-29T00:37:48.185 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VYsOHfpyy/yJE3xxSKOEDBYvay0= 2023-07-29T00:37:48.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VYsOHfpyy/yJE3xxSKOEDBYvay0=', 'Content-Length': '0'} 2023-07-29T00:37:48.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acf5a3543197f0e1-0064c45f5c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.188 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:48.188 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:48.189 DEBUG:boto:path=/hiiqos-59/?prefix=obj145 2023-07-29T00:37:48.189 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj145 2023-07-29T00:37:48.189 DEBUG:boto:Method: GET 2023-07-29T00:37:48.190 DEBUG:boto:Path: /hiiqos-59/?prefix=obj145 2023-07-29T00:37:48.190 DEBUG:boto:Data: 2023-07-29T00:37:48.190 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.191 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:48.191 DEBUG:boto:Port: 8001 2023-07-29T00:37:48.191 DEBUG:boto:Params: {} 2023-07-29T00:37:48.192 DEBUG:boto:Token: None 2023-07-29T00:37:48.192 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/ 2023-07-29T00:37:48.193 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28= 2023-07-29T00:37:48.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28=', 'Content-Length': '0'} 2023-07-29T00:37:48.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005206a4b11c36196-0064c45f5c-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.196 DEBUG:boto:b'hiiqos-59obj1451000falseobj1452023-07-29T00:35:01.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:48.197 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:48.197 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:48.197 DEBUG:boto:path=/hiiqos-59/?prefix=obj145 2023-07-29T00:37:48.198 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj145 2023-07-29T00:37:48.198 DEBUG:boto:Method: GET 2023-07-29T00:37:48.198 DEBUG:boto:Path: /hiiqos-59/?prefix=obj145 2023-07-29T00:37:48.199 DEBUG:boto:Data: 2023-07-29T00:37:48.199 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.200 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:48.200 DEBUG:boto:Port: 8000 2023-07-29T00:37:48.200 DEBUG:boto:Params: {} 2023-07-29T00:37:48.201 DEBUG:boto:Token: None 2023-07-29T00:37:48.201 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/ 2023-07-29T00:37:48.201 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28= 2023-07-29T00:37:48.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28=', 'Content-Length': '0'} 2023-07-29T00:37:48.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c8ed8971a21ea4d-0064c45f5c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.206 DEBUG:boto:b'hiiqos-59obj1451000falseobj1452023-07-29T00:35:01.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:48.207 DEBUG:boto:path=/hiiqos-59/obj145 2023-07-29T00:37:48.207 DEBUG:boto:auth_path=/hiiqos-59/obj145 2023-07-29T00:37:48.207 DEBUG:boto:Method: HEAD 2023-07-29T00:37:48.208 DEBUG:boto:Path: /hiiqos-59/obj145 2023-07-29T00:37:48.208 DEBUG:boto:Data: 2023-07-29T00:37:48.208 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.209 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:48.209 DEBUG:boto:Port: 8001 2023-07-29T00:37:48.209 DEBUG:boto:Params: {} 2023-07-29T00:37:48.210 DEBUG:boto:Token: None 2023-07-29T00:37:48.210 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj145 2023-07-29T00:37:48.211 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gz6gFQse6ESvE3tLdByxAvClIr4= 2023-07-29T00:37:48.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gz6gFQse6ESvE3tLdByxAvClIr4=', 'Content-Length': '0'} 2023-07-29T00:37:48.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000460a0e1f64234065-0064c45f5c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.214 DEBUG:boto:path=/hiiqos-59/obj145 2023-07-29T00:37:48.214 DEBUG:boto:auth_path=/hiiqos-59/obj145 2023-07-29T00:37:48.214 DEBUG:boto:Method: HEAD 2023-07-29T00:37:48.215 DEBUG:boto:Path: /hiiqos-59/obj145 2023-07-29T00:37:48.215 DEBUG:boto:Data: 2023-07-29T00:37:48.215 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.216 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:48.216 DEBUG:boto:Port: 8000 2023-07-29T00:37:48.216 DEBUG:boto:Params: {} 2023-07-29T00:37:48.217 DEBUG:boto:Token: None 2023-07-29T00:37:48.217 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj145 2023-07-29T00:37:48.217 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gz6gFQse6ESvE3tLdByxAvClIr4= 2023-07-29T00:37:48.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gz6gFQse6ESvE3tLdByxAvClIr4=', 'Content-Length': '0'} 2023-07-29T00:37:48.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000291b5ecae36bf524-0064c45f5c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-07-29T00:37:48.221 DEBUG:boto:path=/hiiqos-59/obj145 2023-07-29T00:37:48.221 DEBUG:boto:auth_path=/hiiqos-59/obj145 2023-07-29T00:37:48.221 DEBUG:boto:Method: GET 2023-07-29T00:37:48.222 DEBUG:boto:Path: /hiiqos-59/obj145 2023-07-29T00:37:48.222 DEBUG:boto:Data: 2023-07-29T00:37:48.222 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.223 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:48.223 DEBUG:boto:Port: 8001 2023-07-29T00:37:48.223 DEBUG:boto:Params: {} 2023-07-29T00:37:48.224 DEBUG:boto:Token: None 2023-07-29T00:37:48.224 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj145 2023-07-29T00:37:48.225 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:A3rufu1FSqOoEKKdo7+UOvv8680= 2023-07-29T00:37:48.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:A3rufu1FSqOoEKKdo7+UOvv8680=', 'Content-Length': '0'} 2023-07-29T00:37:48.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005561ae492a6d1a46-0064c45f5c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.268 DEBUG:boto:path=/hiiqos-59/obj145 2023-07-29T00:37:48.269 DEBUG:boto:auth_path=/hiiqos-59/obj145 2023-07-29T00:37:48.269 DEBUG:boto:Method: GET 2023-07-29T00:37:48.270 DEBUG:boto:Path: /hiiqos-59/obj145 2023-07-29T00:37:48.270 DEBUG:boto:Data: 2023-07-29T00:37:48.270 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.271 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:48.271 DEBUG:boto:Port: 8000 2023-07-29T00:37:48.271 DEBUG:boto:Params: {} 2023-07-29T00:37:48.272 DEBUG:boto:Token: None 2023-07-29T00:37:48.272 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj145 2023-07-29T00:37:48.273 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:A3rufu1FSqOoEKKdo7+UOvv8680= 2023-07-29T00:37:48.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:A3rufu1FSqOoEKKdo7+UOvv8680=', 'Content-Length': '0'} 2023-07-29T00:37:48.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ca22b4ee30ba982-0064c45f5c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.276 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:48.276 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:48.276 DEBUG:boto:path=/hiiqos-59/?prefix=obj145 2023-07-29T00:37:48.277 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj145 2023-07-29T00:37:48.277 DEBUG:boto:Method: GET 2023-07-29T00:37:48.278 DEBUG:boto:Path: /hiiqos-59/?prefix=obj145 2023-07-29T00:37:48.278 DEBUG:boto:Data: 2023-07-29T00:37:48.278 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.279 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:48.279 DEBUG:boto:Port: 8001 2023-07-29T00:37:48.279 DEBUG:boto:Params: {} 2023-07-29T00:37:48.280 DEBUG:boto:Token: None 2023-07-29T00:37:48.280 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/ 2023-07-29T00:37:48.280 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28= 2023-07-29T00:37:48.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28=', 'Content-Length': '0'} 2023-07-29T00:37:48.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082aeb4732cd953d4-0064c45f5c-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.284 DEBUG:boto:b'hiiqos-59obj1451000falseobj1452023-07-29T00:35:01.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:48.285 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:48.285 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:48.285 DEBUG:boto:path=/hiiqos-59/?prefix=obj145 2023-07-29T00:37:48.286 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj145 2023-07-29T00:37:48.286 DEBUG:boto:Method: GET 2023-07-29T00:37:48.286 DEBUG:boto:Path: /hiiqos-59/?prefix=obj145 2023-07-29T00:37:48.287 DEBUG:boto:Data: 2023-07-29T00:37:48.287 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.287 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:48.288 DEBUG:boto:Port: 8000 2023-07-29T00:37:48.288 DEBUG:boto:Params: {} 2023-07-29T00:37:48.288 DEBUG:boto:Token: None 2023-07-29T00:37:48.289 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/ 2023-07-29T00:37:48.289 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28= 2023-07-29T00:37:48.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28=', 'Content-Length': '0'} 2023-07-29T00:37:48.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efd2790f06cd7377-0064c45f5c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.294 DEBUG:boto:b'hiiqos-59obj1451000falseobj1452023-07-29T00:35:01.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:48.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-07-29T00:37:48.295 DEBUG:boto:path=/hiiqos-59/obj155 2023-07-29T00:37:48.295 DEBUG:boto:auth_path=/hiiqos-59/obj155 2023-07-29T00:37:48.296 DEBUG:boto:path=/hiiqos-59/obj155?versionId=null 2023-07-29T00:37:48.296 DEBUG:boto:auth_path=/hiiqos-59/obj155?versionId=null 2023-07-29T00:37:48.296 DEBUG:boto:Method: GET 2023-07-29T00:37:48.297 DEBUG:boto:Path: /hiiqos-59/obj155?versionId=null 2023-07-29T00:37:48.297 DEBUG:boto:Data: 2023-07-29T00:37:48.298 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.298 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:48.298 DEBUG:boto:Port: 8001 2023-07-29T00:37:48.299 DEBUG:boto:Params: {} 2023-07-29T00:37:48.299 DEBUG:boto:Token: None 2023-07-29T00:37:48.299 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj155?versionId=null 2023-07-29T00:37:48.300 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:36KJYZaVpkeEAFnQHYz/zx9exb0= 2023-07-29T00:37:48.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:36KJYZaVpkeEAFnQHYz/zx9exb0=', 'Content-Length': '0'} 2023-07-29T00:37:48.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf05f82bc5bb11d9-0064c45f5c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.303 DEBUG:boto:path=/hiiqos-59/obj155 2023-07-29T00:37:48.303 DEBUG:boto:auth_path=/hiiqos-59/obj155 2023-07-29T00:37:48.304 DEBUG:boto:path=/hiiqos-59/obj155?versionId=null 2023-07-29T00:37:48.304 DEBUG:boto:auth_path=/hiiqos-59/obj155?versionId=null 2023-07-29T00:37:48.304 DEBUG:boto:Method: GET 2023-07-29T00:37:48.305 DEBUG:boto:Path: /hiiqos-59/obj155?versionId=null 2023-07-29T00:37:48.305 DEBUG:boto:Data: 2023-07-29T00:37:48.306 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.306 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:48.306 DEBUG:boto:Port: 8000 2023-07-29T00:37:48.307 DEBUG:boto:Params: {} 2023-07-29T00:37:48.307 DEBUG:boto:Token: None 2023-07-29T00:37:48.307 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj155?versionId=null 2023-07-29T00:37:48.308 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:36KJYZaVpkeEAFnQHYz/zx9exb0= 2023-07-29T00:37:48.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:36KJYZaVpkeEAFnQHYz/zx9exb0=', 'Content-Length': '0'} 2023-07-29T00:37:48.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6f91e733e9a5eb3-0064c45f5c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.352 DEBUG:boto:path=/hiiqos-59/obj155 2023-07-29T00:37:48.353 DEBUG:boto:auth_path=/hiiqos-59/obj155 2023-07-29T00:37:48.353 DEBUG:boto:path=/hiiqos-59/obj155?versionId=null 2023-07-29T00:37:48.354 DEBUG:boto:auth_path=/hiiqos-59/obj155?versionId=null 2023-07-29T00:37:48.354 DEBUG:boto:Method: HEAD 2023-07-29T00:37:48.354 DEBUG:boto:Path: /hiiqos-59/obj155?versionId=null 2023-07-29T00:37:48.355 DEBUG:boto:Data: 2023-07-29T00:37:48.355 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.355 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:48.356 DEBUG:boto:Port: 8001 2023-07-29T00:37:48.356 DEBUG:boto:Params: {} 2023-07-29T00:37:48.357 DEBUG:boto:Token: None 2023-07-29T00:37:48.357 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj155?versionId=null 2023-07-29T00:37:48.357 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:H0An6pp1FjgqtsE9hPdBvWzo4ZI= 2023-07-29T00:37:48.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:H0An6pp1FjgqtsE9hPdBvWzo4ZI=', 'Content-Length': '0'} 2023-07-29T00:37:48.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4bf3fe03cfb710c-0064c45f5c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.360 DEBUG:boto:path=/hiiqos-59/obj155 2023-07-29T00:37:48.361 DEBUG:boto:auth_path=/hiiqos-59/obj155 2023-07-29T00:37:48.361 DEBUG:boto:path=/hiiqos-59/obj155?versionId=null 2023-07-29T00:37:48.361 DEBUG:boto:auth_path=/hiiqos-59/obj155?versionId=null 2023-07-29T00:37:48.362 DEBUG:boto:Method: HEAD 2023-07-29T00:37:48.362 DEBUG:boto:Path: /hiiqos-59/obj155?versionId=null 2023-07-29T00:37:48.362 DEBUG:boto:Data: 2023-07-29T00:37:48.363 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.363 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:48.363 DEBUG:boto:Port: 8000 2023-07-29T00:37:48.364 DEBUG:boto:Params: {} 2023-07-29T00:37:48.364 DEBUG:boto:Token: None 2023-07-29T00:37:48.365 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj155?versionId=null 2023-07-29T00:37:48.365 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:H0An6pp1FjgqtsE9hPdBvWzo4ZI= 2023-07-29T00:37:48.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:H0An6pp1FjgqtsE9hPdBvWzo4ZI=', 'Content-Length': '0'} 2023-07-29T00:37:48.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d20cf8eb5b39275-0064c45f5c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-07-29T00:37:48.368 DEBUG:boto:path=/hiiqos-59/obj155 2023-07-29T00:37:48.368 DEBUG:boto:auth_path=/hiiqos-59/obj155 2023-07-29T00:37:48.369 DEBUG:boto:path=/hiiqos-59/obj155?versionId=null 2023-07-29T00:37:48.369 DEBUG:boto:auth_path=/hiiqos-59/obj155?versionId=null 2023-07-29T00:37:48.370 DEBUG:boto:Method: GET 2023-07-29T00:37:48.370 DEBUG:boto:Path: /hiiqos-59/obj155?versionId=null 2023-07-29T00:37:48.370 DEBUG:boto:Data: 2023-07-29T00:37:48.371 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.371 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:48.371 DEBUG:boto:Port: 8001 2023-07-29T00:37:48.372 DEBUG:boto:Params: {} 2023-07-29T00:37:48.372 DEBUG:boto:Token: None 2023-07-29T00:37:48.372 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj155?versionId=null 2023-07-29T00:37:48.373 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:36KJYZaVpkeEAFnQHYz/zx9exb0= 2023-07-29T00:37:48.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:36KJYZaVpkeEAFnQHYz/zx9exb0=', 'Content-Length': '0'} 2023-07-29T00:37:48.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072ea368f11a0b073-0064c45f5c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.376 DEBUG:boto:path=/hiiqos-59/obj155 2023-07-29T00:37:48.376 DEBUG:boto:auth_path=/hiiqos-59/obj155 2023-07-29T00:37:48.377 DEBUG:boto:path=/hiiqos-59/obj155?versionId=null 2023-07-29T00:37:48.377 DEBUG:boto:auth_path=/hiiqos-59/obj155?versionId=null 2023-07-29T00:37:48.377 DEBUG:boto:Method: GET 2023-07-29T00:37:48.378 DEBUG:boto:Path: /hiiqos-59/obj155?versionId=null 2023-07-29T00:37:48.378 DEBUG:boto:Data: 2023-07-29T00:37:48.378 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.379 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:48.379 DEBUG:boto:Port: 8000 2023-07-29T00:37:48.379 DEBUG:boto:Params: {} 2023-07-29T00:37:48.380 DEBUG:boto:Token: None 2023-07-29T00:37:48.380 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj155?versionId=null 2023-07-29T00:37:48.381 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:36KJYZaVpkeEAFnQHYz/zx9exb0= 2023-07-29T00:37:48.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:36KJYZaVpkeEAFnQHYz/zx9exb0=', 'Content-Length': '0'} 2023-07-29T00:37:48.383 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063173d97093dc088-0064c45f5c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.384 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:48.384 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:48.385 DEBUG:boto:path=/hiiqos-59/?prefix=obj155 2023-07-29T00:37:48.385 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj155 2023-07-29T00:37:48.385 DEBUG:boto:Method: GET 2023-07-29T00:37:48.386 DEBUG:boto:Path: /hiiqos-59/?prefix=obj155 2023-07-29T00:37:48.386 DEBUG:boto:Data: 2023-07-29T00:37:48.386 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.387 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:48.387 DEBUG:boto:Port: 8001 2023-07-29T00:37:48.387 DEBUG:boto:Params: {} 2023-07-29T00:37:48.388 DEBUG:boto:Token: None 2023-07-29T00:37:48.388 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/ 2023-07-29T00:37:48.389 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28= 2023-07-29T00:37:48.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28=', 'Content-Length': '0'} 2023-07-29T00:37:48.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d365796a088aacc-0064c45f5c-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.392 DEBUG:boto:b'hiiqos-59obj1551000falseobj1552023-07-29T00:35:31.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:48.393 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:48.393 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:48.393 DEBUG:boto:path=/hiiqos-59/?prefix=obj155 2023-07-29T00:37:48.394 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj155 2023-07-29T00:37:48.394 DEBUG:boto:Method: GET 2023-07-29T00:37:48.394 DEBUG:boto:Path: /hiiqos-59/?prefix=obj155 2023-07-29T00:37:48.395 DEBUG:boto:Data: 2023-07-29T00:37:48.395 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.396 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:48.396 DEBUG:boto:Port: 8000 2023-07-29T00:37:48.396 DEBUG:boto:Params: {} 2023-07-29T00:37:48.397 DEBUG:boto:Token: None 2023-07-29T00:37:48.397 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/ 2023-07-29T00:37:48.397 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28= 2023-07-29T00:37:48.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28=', 'Content-Length': '0'} 2023-07-29T00:37:48.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008d2f647810211de-0064c45f5c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.402 DEBUG:boto:b'hiiqos-59obj1551000falseobj1552023-07-29T00:35:31.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:48.403 DEBUG:boto:path=/hiiqos-59/obj155 2023-07-29T00:37:48.404 DEBUG:boto:auth_path=/hiiqos-59/obj155 2023-07-29T00:37:48.404 DEBUG:boto:Method: HEAD 2023-07-29T00:37:48.404 DEBUG:boto:Path: /hiiqos-59/obj155 2023-07-29T00:37:48.405 DEBUG:boto:Data: 2023-07-29T00:37:48.405 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.405 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:48.406 DEBUG:boto:Port: 8001 2023-07-29T00:37:48.406 DEBUG:boto:Params: {} 2023-07-29T00:37:48.407 DEBUG:boto:Token: None 2023-07-29T00:37:48.407 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj155 2023-07-29T00:37:48.407 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:75YwC5fCs3yXXweUoYys9WvMvpg= 2023-07-29T00:37:48.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:75YwC5fCs3yXXweUoYys9WvMvpg=', 'Content-Length': '0'} 2023-07-29T00:37:48.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bca49d99443de98f-0064c45f5c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.410 DEBUG:boto:path=/hiiqos-59/obj155 2023-07-29T00:37:48.410 DEBUG:boto:auth_path=/hiiqos-59/obj155 2023-07-29T00:37:48.411 DEBUG:boto:Method: HEAD 2023-07-29T00:37:48.411 DEBUG:boto:Path: /hiiqos-59/obj155 2023-07-29T00:37:48.412 DEBUG:boto:Data: 2023-07-29T00:37:48.412 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.412 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:48.413 DEBUG:boto:Port: 8000 2023-07-29T00:37:48.413 DEBUG:boto:Params: {} 2023-07-29T00:37:48.413 DEBUG:boto:Token: None 2023-07-29T00:37:48.414 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj155 2023-07-29T00:37:48.414 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:75YwC5fCs3yXXweUoYys9WvMvpg= 2023-07-29T00:37:48.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:75YwC5fCs3yXXweUoYys9WvMvpg=', 'Content-Length': '0'} 2023-07-29T00:37:48.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ed9378992360d2f-0064c45f5c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-07-29T00:37:48.417 DEBUG:boto:path=/hiiqos-59/obj155 2023-07-29T00:37:48.418 DEBUG:boto:auth_path=/hiiqos-59/obj155 2023-07-29T00:37:48.418 DEBUG:boto:Method: GET 2023-07-29T00:37:48.418 DEBUG:boto:Path: /hiiqos-59/obj155 2023-07-29T00:37:48.419 DEBUG:boto:Data: 2023-07-29T00:37:48.419 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.419 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:48.420 DEBUG:boto:Port: 8001 2023-07-29T00:37:48.420 DEBUG:boto:Params: {} 2023-07-29T00:37:48.421 DEBUG:boto:Token: None 2023-07-29T00:37:48.421 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj155 2023-07-29T00:37:48.421 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vs++qEo0q1nRxXoLm1xo8TklPII= 2023-07-29T00:37:48.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vs++qEo0q1nRxXoLm1xo8TklPII=', 'Content-Length': '0'} 2023-07-29T00:37:48.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040cdd54c0fa6d509-0064c45f5c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.424 DEBUG:boto:path=/hiiqos-59/obj155 2023-07-29T00:37:48.425 DEBUG:boto:auth_path=/hiiqos-59/obj155 2023-07-29T00:37:48.425 DEBUG:boto:Method: GET 2023-07-29T00:37:48.426 DEBUG:boto:Path: /hiiqos-59/obj155 2023-07-29T00:37:48.426 DEBUG:boto:Data: 2023-07-29T00:37:48.426 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.427 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:48.427 DEBUG:boto:Port: 8000 2023-07-29T00:37:48.427 DEBUG:boto:Params: {} 2023-07-29T00:37:48.428 DEBUG:boto:Token: None 2023-07-29T00:37:48.428 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj155 2023-07-29T00:37:48.429 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vs++qEo0q1nRxXoLm1xo8TklPII= 2023-07-29T00:37:48.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vs++qEo0q1nRxXoLm1xo8TklPII=', 'Content-Length': '0'} 2023-07-29T00:37:48.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc5d3adec81641ed-0064c45f5c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.472 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:48.473 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:48.473 DEBUG:boto:path=/hiiqos-59/?prefix=obj155 2023-07-29T00:37:48.473 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj155 2023-07-29T00:37:48.474 DEBUG:boto:Method: GET 2023-07-29T00:37:48.474 DEBUG:boto:Path: /hiiqos-59/?prefix=obj155 2023-07-29T00:37:48.475 DEBUG:boto:Data: 2023-07-29T00:37:48.475 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.475 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:48.476 DEBUG:boto:Port: 8001 2023-07-29T00:37:48.476 DEBUG:boto:Params: {} 2023-07-29T00:37:48.476 DEBUG:boto:Token: None 2023-07-29T00:37:48.477 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/ 2023-07-29T00:37:48.477 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28= 2023-07-29T00:37:48.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28=', 'Content-Length': '0'} 2023-07-29T00:37:48.480 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002b11dc9e970fd27-0064c45f5c-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.481 DEBUG:boto:b'hiiqos-59obj1551000falseobj1552023-07-29T00:35:31.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:48.481 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:48.482 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:48.482 DEBUG:boto:path=/hiiqos-59/?prefix=obj155 2023-07-29T00:37:48.482 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj155 2023-07-29T00:37:48.483 DEBUG:boto:Method: GET 2023-07-29T00:37:48.483 DEBUG:boto:Path: /hiiqos-59/?prefix=obj155 2023-07-29T00:37:48.484 DEBUG:boto:Data: 2023-07-29T00:37:48.484 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.484 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:48.485 DEBUG:boto:Port: 8000 2023-07-29T00:37:48.485 DEBUG:boto:Params: {} 2023-07-29T00:37:48.485 DEBUG:boto:Token: None 2023-07-29T00:37:48.486 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/ 2023-07-29T00:37:48.486 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28= 2023-07-29T00:37:48.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28=', 'Content-Length': '0'} 2023-07-29T00:37:48.491 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f9162ab2584ba84-0064c45f5c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.492 DEBUG:boto:b'hiiqos-59obj1551000falseobj1552023-07-29T00:35:31.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:48.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-07-29T00:37:48.493 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:37:48.494 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:37:48.494 DEBUG:boto:path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:37:48.494 DEBUG:boto:auth_path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:37:48.495 DEBUG:boto:Method: GET 2023-07-29T00:37:48.495 DEBUG:boto:Path: /hiiqos-59/obj17?versionId=null 2023-07-29T00:37:48.495 DEBUG:boto:Data: 2023-07-29T00:37:48.496 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.496 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:48.496 DEBUG:boto:Port: 8001 2023-07-29T00:37:48.497 DEBUG:boto:Params: {} 2023-07-29T00:37:48.497 DEBUG:boto:Token: None 2023-07-29T00:37:48.498 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj17?versionId=null 2023-07-29T00:37:48.498 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Shk9sVrZxGxlttVg4S7LPcbFfoQ= 2023-07-29T00:37:48.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Shk9sVrZxGxlttVg4S7LPcbFfoQ=', 'Content-Length': '0'} 2023-07-29T00:37:48.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d7e73871badd6b3-0064c45f5c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.501 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:37:48.501 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:37:48.502 DEBUG:boto:path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:37:48.502 DEBUG:boto:auth_path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:37:48.502 DEBUG:boto:Method: GET 2023-07-29T00:37:48.503 DEBUG:boto:Path: /hiiqos-59/obj17?versionId=null 2023-07-29T00:37:48.503 DEBUG:boto:Data: 2023-07-29T00:37:48.503 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.504 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:48.504 DEBUG:boto:Port: 8000 2023-07-29T00:37:48.505 DEBUG:boto:Params: {} 2023-07-29T00:37:48.505 DEBUG:boto:Token: None 2023-07-29T00:37:48.505 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj17?versionId=null 2023-07-29T00:37:48.506 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Shk9sVrZxGxlttVg4S7LPcbFfoQ= 2023-07-29T00:37:48.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Shk9sVrZxGxlttVg4S7LPcbFfoQ=', 'Content-Length': '0'} 2023-07-29T00:37:48.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f342b9d876674dbd-0064c45f5c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.509 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:37:48.509 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:37:48.509 DEBUG:boto:path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:37:48.510 DEBUG:boto:auth_path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:37:48.510 DEBUG:boto:Method: HEAD 2023-07-29T00:37:48.510 DEBUG:boto:Path: /hiiqos-59/obj17?versionId=null 2023-07-29T00:37:48.511 DEBUG:boto:Data: 2023-07-29T00:37:48.511 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.511 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:48.512 DEBUG:boto:Port: 8001 2023-07-29T00:37:48.512 DEBUG:boto:Params: {} 2023-07-29T00:37:48.512 DEBUG:boto:Token: None 2023-07-29T00:37:48.513 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj17?versionId=null 2023-07-29T00:37:48.513 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:20L/BdXmU1lAdTf6UQT26K1jaLQ= 2023-07-29T00:37:48.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:20L/BdXmU1lAdTf6UQT26K1jaLQ=', 'Content-Length': '0'} 2023-07-29T00:37:48.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf6212f9a2128dd4-0064c45f5c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.516 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:37:48.516 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:37:48.517 DEBUG:boto:path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:37:48.517 DEBUG:boto:auth_path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:37:48.517 DEBUG:boto:Method: HEAD 2023-07-29T00:37:48.518 DEBUG:boto:Path: /hiiqos-59/obj17?versionId=null 2023-07-29T00:37:48.518 DEBUG:boto:Data: 2023-07-29T00:37:48.518 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.519 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:48.519 DEBUG:boto:Port: 8000 2023-07-29T00:37:48.519 DEBUG:boto:Params: {} 2023-07-29T00:37:48.520 DEBUG:boto:Token: None 2023-07-29T00:37:48.520 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj17?versionId=null 2023-07-29T00:37:48.521 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:20L/BdXmU1lAdTf6UQT26K1jaLQ= 2023-07-29T00:37:48.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:20L/BdXmU1lAdTf6UQT26K1jaLQ=', 'Content-Length': '0'} 2023-07-29T00:37:48.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000217d7bcf64a15c7b-0064c45f5c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-07-29T00:37:48.524 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:37:48.524 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:37:48.525 DEBUG:boto:path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:37:48.525 DEBUG:boto:auth_path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:37:48.525 DEBUG:boto:Method: GET 2023-07-29T00:37:48.526 DEBUG:boto:Path: /hiiqos-59/obj17?versionId=null 2023-07-29T00:37:48.526 DEBUG:boto:Data: 2023-07-29T00:37:48.526 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.527 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:48.527 DEBUG:boto:Port: 8001 2023-07-29T00:37:48.527 DEBUG:boto:Params: {} 2023-07-29T00:37:48.528 DEBUG:boto:Token: None 2023-07-29T00:37:48.528 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj17?versionId=null 2023-07-29T00:37:48.529 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Shk9sVrZxGxlttVg4S7LPcbFfoQ= 2023-07-29T00:37:48.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Shk9sVrZxGxlttVg4S7LPcbFfoQ=', 'Content-Length': '0'} 2023-07-29T00:37:48.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b11aa3dd53907961-0064c45f5c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.572 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:37:48.573 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:37:48.573 DEBUG:boto:path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:37:48.574 DEBUG:boto:auth_path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:37:48.574 DEBUG:boto:Method: GET 2023-07-29T00:37:48.575 DEBUG:boto:Path: /hiiqos-59/obj17?versionId=null 2023-07-29T00:37:48.575 DEBUG:boto:Data: 2023-07-29T00:37:48.575 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.576 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:48.576 DEBUG:boto:Port: 8000 2023-07-29T00:37:48.576 DEBUG:boto:Params: {} 2023-07-29T00:37:48.577 DEBUG:boto:Token: None 2023-07-29T00:37:48.577 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj17?versionId=null 2023-07-29T00:37:48.578 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Shk9sVrZxGxlttVg4S7LPcbFfoQ= 2023-07-29T00:37:48.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Shk9sVrZxGxlttVg4S7LPcbFfoQ=', 'Content-Length': '0'} 2023-07-29T00:37:48.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007977589b2ed8f449-0064c45f5c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.581 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:48.581 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:48.582 DEBUG:boto:path=/hiiqos-59/?prefix=obj17 2023-07-29T00:37:48.582 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj17 2023-07-29T00:37:48.582 DEBUG:boto:Method: GET 2023-07-29T00:37:48.583 DEBUG:boto:Path: /hiiqos-59/?prefix=obj17 2023-07-29T00:37:48.583 DEBUG:boto:Data: 2023-07-29T00:37:48.583 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.584 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:48.584 DEBUG:boto:Port: 8001 2023-07-29T00:37:48.585 DEBUG:boto:Params: {} 2023-07-29T00:37:48.585 DEBUG:boto:Token: None 2023-07-29T00:37:48.586 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/ 2023-07-29T00:37:48.586 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28= 2023-07-29T00:37:48.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28=', 'Content-Length': '0'} 2023-07-29T00:37:48.592 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1ef718656521c9e-0064c45f5c-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.593 DEBUG:boto:b'hiiqos-59obj171000falseobj172023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:48.594 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:48.594 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:48.595 DEBUG:boto:path=/hiiqos-59/?prefix=obj17 2023-07-29T00:37:48.595 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj17 2023-07-29T00:37:48.596 DEBUG:boto:Method: GET 2023-07-29T00:37:48.596 DEBUG:boto:Path: /hiiqos-59/?prefix=obj17 2023-07-29T00:37:48.597 DEBUG:boto:Data: 2023-07-29T00:37:48.597 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.598 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:48.598 DEBUG:boto:Port: 8000 2023-07-29T00:37:48.598 DEBUG:boto:Params: {} 2023-07-29T00:37:48.599 DEBUG:boto:Token: None 2023-07-29T00:37:48.599 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/ 2023-07-29T00:37:48.600 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28= 2023-07-29T00:37:48.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28=', 'Content-Length': '0'} 2023-07-29T00:37:48.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005774ffb8a6aef6c8-0064c45f5c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.648 DEBUG:boto:b'hiiqos-59obj171000falseobj172023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:48.649 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:37:48.649 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:37:48.650 DEBUG:boto:Method: HEAD 2023-07-29T00:37:48.650 DEBUG:boto:Path: /hiiqos-59/obj17 2023-07-29T00:37:48.651 DEBUG:boto:Data: 2023-07-29T00:37:48.651 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.651 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:48.652 DEBUG:boto:Port: 8001 2023-07-29T00:37:48.652 DEBUG:boto:Params: {} 2023-07-29T00:37:48.653 DEBUG:boto:Token: None 2023-07-29T00:37:48.653 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj17 2023-07-29T00:37:48.653 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AndXgherohkSxb2hvdTP6DX0ZVQ= 2023-07-29T00:37:48.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AndXgherohkSxb2hvdTP6DX0ZVQ=', 'Content-Length': '0'} 2023-07-29T00:37:48.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018558a294e6d871b-0064c45f5c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.657 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:37:48.657 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:37:48.657 DEBUG:boto:Method: HEAD 2023-07-29T00:37:48.658 DEBUG:boto:Path: /hiiqos-59/obj17 2023-07-29T00:37:48.658 DEBUG:boto:Data: 2023-07-29T00:37:48.658 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.659 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:48.659 DEBUG:boto:Port: 8000 2023-07-29T00:37:48.659 DEBUG:boto:Params: {} 2023-07-29T00:37:48.660 DEBUG:boto:Token: None 2023-07-29T00:37:48.660 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj17 2023-07-29T00:37:48.661 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AndXgherohkSxb2hvdTP6DX0ZVQ= 2023-07-29T00:37:48.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AndXgherohkSxb2hvdTP6DX0ZVQ=', 'Content-Length': '0'} 2023-07-29T00:37:48.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d9128f9b3edd42f-0064c45f5c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-07-29T00:37:48.664 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:37:48.665 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:37:48.666 DEBUG:boto:Method: GET 2023-07-29T00:37:48.666 DEBUG:boto:Path: /hiiqos-59/obj17 2023-07-29T00:37:48.667 DEBUG:boto:Data: 2023-07-29T00:37:48.667 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.667 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:48.668 DEBUG:boto:Port: 8001 2023-07-29T00:37:48.668 DEBUG:boto:Params: {} 2023-07-29T00:37:48.668 DEBUG:boto:Token: None 2023-07-29T00:37:48.669 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj17 2023-07-29T00:37:48.669 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2/Q92U3rgEyByJvJxFjP2XL1LSM= 2023-07-29T00:37:48.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2/Q92U3rgEyByJvJxFjP2XL1LSM=', 'Content-Length': '0'} 2023-07-29T00:37:48.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000067ad25cb9266c88-0064c45f5c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.673 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:37:48.673 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:37:48.673 DEBUG:boto:Method: GET 2023-07-29T00:37:48.674 DEBUG:boto:Path: /hiiqos-59/obj17 2023-07-29T00:37:48.674 DEBUG:boto:Data: 2023-07-29T00:37:48.675 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.675 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:48.676 DEBUG:boto:Port: 8000 2023-07-29T00:37:48.676 DEBUG:boto:Params: {} 2023-07-29T00:37:48.676 DEBUG:boto:Token: None 2023-07-29T00:37:48.677 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj17 2023-07-29T00:37:48.677 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2/Q92U3rgEyByJvJxFjP2XL1LSM= 2023-07-29T00:37:48.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2/Q92U3rgEyByJvJxFjP2XL1LSM=', 'Content-Length': '0'} 2023-07-29T00:37:48.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028949cc4472e4993-0064c45f5c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.681 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:48.681 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:48.681 DEBUG:boto:path=/hiiqos-59/?prefix=obj17 2023-07-29T00:37:48.682 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj17 2023-07-29T00:37:48.682 DEBUG:boto:Method: GET 2023-07-29T00:37:48.683 DEBUG:boto:Path: /hiiqos-59/?prefix=obj17 2023-07-29T00:37:48.683 DEBUG:boto:Data: 2023-07-29T00:37:48.684 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.684 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:48.684 DEBUG:boto:Port: 8001 2023-07-29T00:37:48.685 DEBUG:boto:Params: {} 2023-07-29T00:37:48.685 DEBUG:boto:Token: None 2023-07-29T00:37:48.686 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/ 2023-07-29T00:37:48.686 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28= 2023-07-29T00:37:48.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28=', 'Content-Length': '0'} 2023-07-29T00:37:48.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0b07486bea63afc-0064c45f5c-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.690 DEBUG:boto:b'hiiqos-59obj171000falseobj172023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:48.690 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:48.691 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:48.691 DEBUG:boto:path=/hiiqos-59/?prefix=obj17 2023-07-29T00:37:48.692 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj17 2023-07-29T00:37:48.692 DEBUG:boto:Method: GET 2023-07-29T00:37:48.693 DEBUG:boto:Path: /hiiqos-59/?prefix=obj17 2023-07-29T00:37:48.693 DEBUG:boto:Data: 2023-07-29T00:37:48.693 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.694 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:48.694 DEBUG:boto:Port: 8000 2023-07-29T00:37:48.695 DEBUG:boto:Params: {} 2023-07-29T00:37:48.695 DEBUG:boto:Token: None 2023-07-29T00:37:48.696 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/ 2023-07-29T00:37:48.696 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28= 2023-07-29T00:37:48.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28=', 'Content-Length': '0'} 2023-07-29T00:37:48.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051ff92fd7c19907b-0064c45f5c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.701 DEBUG:boto:b'hiiqos-59obj171000falseobj172023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:48.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-07-29T00:37:48.703 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:37:48.704 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:37:48.704 DEBUG:boto:path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:37:48.704 DEBUG:boto:auth_path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:37:48.705 DEBUG:boto:Method: GET 2023-07-29T00:37:48.705 DEBUG:boto:Path: /hiiqos-59/obj19?versionId=null 2023-07-29T00:37:48.706 DEBUG:boto:Data: 2023-07-29T00:37:48.706 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.706 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:48.707 DEBUG:boto:Port: 8001 2023-07-29T00:37:48.707 DEBUG:boto:Params: {} 2023-07-29T00:37:48.707 DEBUG:boto:Token: None 2023-07-29T00:37:48.708 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj19?versionId=null 2023-07-29T00:37:48.709 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:K1EuSslukyT1qvHFmei0xPbYEHA= 2023-07-29T00:37:48.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K1EuSslukyT1qvHFmei0xPbYEHA=', 'Content-Length': '0'} 2023-07-29T00:37:48.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f928d5426af27e6e-0064c45f5c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.712 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:37:48.712 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:37:48.713 DEBUG:boto:path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:37:48.713 DEBUG:boto:auth_path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:37:48.713 DEBUG:boto:Method: GET 2023-07-29T00:37:48.714 DEBUG:boto:Path: /hiiqos-59/obj19?versionId=null 2023-07-29T00:37:48.714 DEBUG:boto:Data: 2023-07-29T00:37:48.715 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.715 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:48.715 DEBUG:boto:Port: 8000 2023-07-29T00:37:48.716 DEBUG:boto:Params: {} 2023-07-29T00:37:48.716 DEBUG:boto:Token: None 2023-07-29T00:37:48.717 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj19?versionId=null 2023-07-29T00:37:48.718 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:K1EuSslukyT1qvHFmei0xPbYEHA= 2023-07-29T00:37:48.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K1EuSslukyT1qvHFmei0xPbYEHA=', 'Content-Length': '0'} 2023-07-29T00:37:48.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d03ee96863bb311-0064c45f5c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.721 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:37:48.722 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:37:48.722 DEBUG:boto:path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:37:48.723 DEBUG:boto:auth_path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:37:48.723 DEBUG:boto:Method: HEAD 2023-07-29T00:37:48.723 DEBUG:boto:Path: /hiiqos-59/obj19?versionId=null 2023-07-29T00:37:48.724 DEBUG:boto:Data: 2023-07-29T00:37:48.724 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.724 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:48.725 DEBUG:boto:Port: 8001 2023-07-29T00:37:48.725 DEBUG:boto:Params: {} 2023-07-29T00:37:48.726 DEBUG:boto:Token: None 2023-07-29T00:37:48.726 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj19?versionId=null 2023-07-29T00:37:48.726 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KS+j4G+q4lIV6AGgO1tpXt75F2Q= 2023-07-29T00:37:48.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KS+j4G+q4lIV6AGgO1tpXt75F2Q=', 'Content-Length': '0'} 2023-07-29T00:37:48.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023111ec6e5b32024-0064c45f5c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.729 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:37:48.729 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:37:48.730 DEBUG:boto:path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:37:48.730 DEBUG:boto:auth_path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:37:48.730 DEBUG:boto:Method: HEAD 2023-07-29T00:37:48.731 DEBUG:boto:Path: /hiiqos-59/obj19?versionId=null 2023-07-29T00:37:48.731 DEBUG:boto:Data: 2023-07-29T00:37:48.731 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.732 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:48.732 DEBUG:boto:Port: 8000 2023-07-29T00:37:48.732 DEBUG:boto:Params: {} 2023-07-29T00:37:48.733 DEBUG:boto:Token: None 2023-07-29T00:37:48.733 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj19?versionId=null 2023-07-29T00:37:48.734 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KS+j4G+q4lIV6AGgO1tpXt75F2Q= 2023-07-29T00:37:48.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KS+j4G+q4lIV6AGgO1tpXt75F2Q=', 'Content-Length': '0'} 2023-07-29T00:37:48.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027dc42943bbda6f6-0064c45f5c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-07-29T00:37:48.737 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:37:48.737 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:37:48.738 DEBUG:boto:path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:37:48.738 DEBUG:boto:auth_path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:37:48.738 DEBUG:boto:Method: GET 2023-07-29T00:37:48.739 DEBUG:boto:Path: /hiiqos-59/obj19?versionId=null 2023-07-29T00:37:48.739 DEBUG:boto:Data: 2023-07-29T00:37:48.739 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.740 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:48.740 DEBUG:boto:Port: 8001 2023-07-29T00:37:48.740 DEBUG:boto:Params: {} 2023-07-29T00:37:48.741 DEBUG:boto:Token: None 2023-07-29T00:37:48.741 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj19?versionId=null 2023-07-29T00:37:48.742 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:K1EuSslukyT1qvHFmei0xPbYEHA= 2023-07-29T00:37:48.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K1EuSslukyT1qvHFmei0xPbYEHA=', 'Content-Length': '0'} 2023-07-29T00:37:48.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f8d1c60b451f42e-0064c45f5c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.784 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:37:48.785 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:37:48.785 DEBUG:boto:path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:37:48.786 DEBUG:boto:auth_path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:37:48.786 DEBUG:boto:Method: GET 2023-07-29T00:37:48.787 DEBUG:boto:Path: /hiiqos-59/obj19?versionId=null 2023-07-29T00:37:48.787 DEBUG:boto:Data: 2023-07-29T00:37:48.787 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.788 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:48.788 DEBUG:boto:Port: 8000 2023-07-29T00:37:48.788 DEBUG:boto:Params: {} 2023-07-29T00:37:48.789 DEBUG:boto:Token: None 2023-07-29T00:37:48.789 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj19?versionId=null 2023-07-29T00:37:48.790 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:K1EuSslukyT1qvHFmei0xPbYEHA= 2023-07-29T00:37:48.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K1EuSslukyT1qvHFmei0xPbYEHA=', 'Content-Length': '0'} 2023-07-29T00:37:48.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3d57df37b2d6ca9-0064c45f5c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.793 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:48.793 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:48.794 DEBUG:boto:path=/hiiqos-59/?prefix=obj19 2023-07-29T00:37:48.794 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj19 2023-07-29T00:37:48.794 DEBUG:boto:Method: GET 2023-07-29T00:37:48.795 DEBUG:boto:Path: /hiiqos-59/?prefix=obj19 2023-07-29T00:37:48.795 DEBUG:boto:Data: 2023-07-29T00:37:48.795 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.796 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:48.796 DEBUG:boto:Port: 8001 2023-07-29T00:37:48.796 DEBUG:boto:Params: {} 2023-07-29T00:37:48.797 DEBUG:boto:Token: None 2023-07-29T00:37:48.797 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/ 2023-07-29T00:37:48.798 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28= 2023-07-29T00:37:48.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28=', 'Content-Length': '0'} 2023-07-29T00:37:48.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006586d3bfd755e43c-0064c45f5c-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.801 DEBUG:boto:b'hiiqos-59obj191000falseobj192023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:48.802 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:48.803 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:48.803 DEBUG:boto:path=/hiiqos-59/?prefix=obj19 2023-07-29T00:37:48.803 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj19 2023-07-29T00:37:48.804 DEBUG:boto:Method: GET 2023-07-29T00:37:48.804 DEBUG:boto:Path: /hiiqos-59/?prefix=obj19 2023-07-29T00:37:48.805 DEBUG:boto:Data: 2023-07-29T00:37:48.805 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.805 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:48.806 DEBUG:boto:Port: 8000 2023-07-29T00:37:48.806 DEBUG:boto:Params: {} 2023-07-29T00:37:48.807 DEBUG:boto:Token: None 2023-07-29T00:37:48.807 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/ 2023-07-29T00:37:48.807 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28= 2023-07-29T00:37:48.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28=', 'Content-Length': '0'} 2023-07-29T00:37:48.812 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038bcef6b1521b36c-0064c45f5c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.813 DEBUG:boto:b'hiiqos-59obj191000falseobj192023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:48.813 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:37:48.814 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:37:48.814 DEBUG:boto:Method: HEAD 2023-07-29T00:37:48.815 DEBUG:boto:Path: /hiiqos-59/obj19 2023-07-29T00:37:48.815 DEBUG:boto:Data: 2023-07-29T00:37:48.815 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.816 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:48.816 DEBUG:boto:Port: 8001 2023-07-29T00:37:48.816 DEBUG:boto:Params: {} 2023-07-29T00:37:48.817 DEBUG:boto:Token: None 2023-07-29T00:37:48.817 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj19 2023-07-29T00:37:48.817 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pzI8lHe+h6NOM7HmS+RbC0sP5Aw= 2023-07-29T00:37:48.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pzI8lHe+h6NOM7HmS+RbC0sP5Aw=', 'Content-Length': '0'} 2023-07-29T00:37:48.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8a8de592a9a8aea-0064c45f5c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.820 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:37:48.821 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:37:48.821 DEBUG:boto:Method: HEAD 2023-07-29T00:37:48.821 DEBUG:boto:Path: /hiiqos-59/obj19 2023-07-29T00:37:48.822 DEBUG:boto:Data: 2023-07-29T00:37:48.822 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.822 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:48.823 DEBUG:boto:Port: 8000 2023-07-29T00:37:48.823 DEBUG:boto:Params: {} 2023-07-29T00:37:48.823 DEBUG:boto:Token: None 2023-07-29T00:37:48.824 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj19 2023-07-29T00:37:48.824 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pzI8lHe+h6NOM7HmS+RbC0sP5Aw= 2023-07-29T00:37:48.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pzI8lHe+h6NOM7HmS+RbC0sP5Aw=', 'Content-Length': '0'} 2023-07-29T00:37:48.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2660591265982ae-0064c45f5c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-07-29T00:37:48.827 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:37:48.828 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:37:48.828 DEBUG:boto:Method: GET 2023-07-29T00:37:48.829 DEBUG:boto:Path: /hiiqos-59/obj19 2023-07-29T00:37:48.829 DEBUG:boto:Data: 2023-07-29T00:37:48.829 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.830 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:48.830 DEBUG:boto:Port: 8001 2023-07-29T00:37:48.830 DEBUG:boto:Params: {} 2023-07-29T00:37:48.831 DEBUG:boto:Token: None 2023-07-29T00:37:48.831 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj19 2023-07-29T00:37:48.832 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Z1yTpJ6qfo7FSBnafBF62a+3/gU= 2023-07-29T00:37:48.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Z1yTpJ6qfo7FSBnafBF62a+3/gU=', 'Content-Length': '0'} 2023-07-29T00:37:48.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000981b459bec835aa8-0064c45f5c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.835 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:37:48.835 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:37:48.836 DEBUG:boto:Method: GET 2023-07-29T00:37:48.836 DEBUG:boto:Path: /hiiqos-59/obj19 2023-07-29T00:37:48.836 DEBUG:boto:Data: 2023-07-29T00:37:48.837 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.837 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:48.838 DEBUG:boto:Port: 8000 2023-07-29T00:37:48.838 DEBUG:boto:Params: {} 2023-07-29T00:37:48.838 DEBUG:boto:Token: None 2023-07-29T00:37:48.839 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj19 2023-07-29T00:37:48.839 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Z1yTpJ6qfo7FSBnafBF62a+3/gU= 2023-07-29T00:37:48.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Z1yTpJ6qfo7FSBnafBF62a+3/gU=', 'Content-Length': '0'} 2023-07-29T00:37:48.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000649b23d9783258cf-0064c45f5c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.842 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:48.842 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:48.843 DEBUG:boto:path=/hiiqos-59/?prefix=obj19 2023-07-29T00:37:48.843 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj19 2023-07-29T00:37:48.843 DEBUG:boto:Method: GET 2023-07-29T00:37:48.844 DEBUG:boto:Path: /hiiqos-59/?prefix=obj19 2023-07-29T00:37:48.844 DEBUG:boto:Data: 2023-07-29T00:37:48.844 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.845 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:48.845 DEBUG:boto:Port: 8001 2023-07-29T00:37:48.846 DEBUG:boto:Params: {} 2023-07-29T00:37:48.846 DEBUG:boto:Token: None 2023-07-29T00:37:48.847 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/ 2023-07-29T00:37:48.847 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28= 2023-07-29T00:37:48.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28=', 'Content-Length': '0'} 2023-07-29T00:37:48.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e032bf42c68e08d3-0064c45f5c-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.850 DEBUG:boto:b'hiiqos-59obj191000falseobj192023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:48.851 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:48.851 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:48.852 DEBUG:boto:path=/hiiqos-59/?prefix=obj19 2023-07-29T00:37:48.852 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj19 2023-07-29T00:37:48.852 DEBUG:boto:Method: GET 2023-07-29T00:37:48.853 DEBUG:boto:Path: /hiiqos-59/?prefix=obj19 2023-07-29T00:37:48.853 DEBUG:boto:Data: 2023-07-29T00:37:48.854 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.854 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:48.854 DEBUG:boto:Port: 8000 2023-07-29T00:37:48.855 DEBUG:boto:Params: {} 2023-07-29T00:37:48.855 DEBUG:boto:Token: None 2023-07-29T00:37:48.856 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/ 2023-07-29T00:37:48.856 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28= 2023-07-29T00:37:48.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28=', 'Content-Length': '0'} 2023-07-29T00:37:48.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abe84fbc91f73f82-0064c45f5c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.861 DEBUG:boto:b'hiiqos-59obj191000falseobj192023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:48.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-07-29T00:37:48.862 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:37:48.862 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:37:48.863 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:37:48.863 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:37:48.863 DEBUG:boto:Method: GET 2023-07-29T00:37:48.864 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:37:48.864 DEBUG:boto:Data: 2023-07-29T00:37:48.864 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.865 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:48.865 DEBUG:boto:Port: 8001 2023-07-29T00:37:48.866 DEBUG:boto:Params: {} 2023-07-29T00:37:48.866 DEBUG:boto:Token: None 2023-07-29T00:37:48.867 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:37:48.867 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5TJIyeDFYt4DbzPOiBAzob9e2aw= 2023-07-29T00:37:48.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5TJIyeDFYt4DbzPOiBAzob9e2aw=', 'Content-Length': '0'} 2023-07-29T00:37:48.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000611a623891e4f9ad-0064c45f5c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.870 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:37:48.870 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:37:48.871 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:37:48.871 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:37:48.871 DEBUG:boto:Method: GET 2023-07-29T00:37:48.872 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:37:48.872 DEBUG:boto:Data: 2023-07-29T00:37:48.872 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.873 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:48.873 DEBUG:boto:Port: 8000 2023-07-29T00:37:48.873 DEBUG:boto:Params: {} 2023-07-29T00:37:48.874 DEBUG:boto:Token: None 2023-07-29T00:37:48.874 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:37:48.875 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5TJIyeDFYt4DbzPOiBAzob9e2aw= 2023-07-29T00:37:48.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5TJIyeDFYt4DbzPOiBAzob9e2aw=', 'Content-Length': '0'} 2023-07-29T00:37:48.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ab5c9933eb422c7-0064c45f5c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.878 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:37:48.879 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:37:48.879 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:37:48.879 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:37:48.880 DEBUG:boto:Method: HEAD 2023-07-29T00:37:48.880 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:37:48.880 DEBUG:boto:Data: 2023-07-29T00:37:48.881 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.881 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:48.881 DEBUG:boto:Port: 8001 2023-07-29T00:37:48.882 DEBUG:boto:Params: {} 2023-07-29T00:37:48.882 DEBUG:boto:Token: None 2023-07-29T00:37:48.883 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:37:48.883 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sEsqLclizFcvaUUC/3La0ID4ugk= 2023-07-29T00:37:48.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sEsqLclizFcvaUUC/3La0ID4ugk=', 'Content-Length': '0'} 2023-07-29T00:37:48.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040801fd0b5522c86-0064c45f5c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.886 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:37:48.886 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:37:48.886 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:37:48.887 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:37:48.887 DEBUG:boto:Method: HEAD 2023-07-29T00:37:48.888 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:37:48.888 DEBUG:boto:Data: 2023-07-29T00:37:48.888 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.889 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:48.889 DEBUG:boto:Port: 8000 2023-07-29T00:37:48.889 DEBUG:boto:Params: {} 2023-07-29T00:37:48.890 DEBUG:boto:Token: None 2023-07-29T00:37:48.890 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:37:48.890 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sEsqLclizFcvaUUC/3La0ID4ugk= 2023-07-29T00:37:48.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sEsqLclizFcvaUUC/3La0ID4ugk=', 'Content-Length': '0'} 2023-07-29T00:37:48.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9bbf37eddf12b6a-0064c45f5c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-07-29T00:37:48.893 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:37:48.894 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:37:48.894 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:37:48.895 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:37:48.895 DEBUG:boto:Method: GET 2023-07-29T00:37:48.895 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:37:48.896 DEBUG:boto:Data: 2023-07-29T00:37:48.896 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.896 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:48.897 DEBUG:boto:Port: 8001 2023-07-29T00:37:48.897 DEBUG:boto:Params: {} 2023-07-29T00:37:48.898 DEBUG:boto:Token: None 2023-07-29T00:37:48.898 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:37:48.898 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5TJIyeDFYt4DbzPOiBAzob9e2aw= 2023-07-29T00:37:48.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5TJIyeDFYt4DbzPOiBAzob9e2aw=', 'Content-Length': '0'} 2023-07-29T00:37:48.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fbde5b2b9711c14-0064c45f5c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.944 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:37:48.945 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:37:48.945 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:37:48.946 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:37:48.946 DEBUG:boto:Method: GET 2023-07-29T00:37:48.946 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:37:48.947 DEBUG:boto:Data: 2023-07-29T00:37:48.947 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.947 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:48.948 DEBUG:boto:Port: 8000 2023-07-29T00:37:48.948 DEBUG:boto:Params: {} 2023-07-29T00:37:48.949 DEBUG:boto:Token: None 2023-07-29T00:37:48.949 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:37:48.950 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5TJIyeDFYt4DbzPOiBAzob9e2aw= 2023-07-29T00:37:48.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5TJIyeDFYt4DbzPOiBAzob9e2aw=', 'Content-Length': '0'} 2023-07-29T00:37:48.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fdccba6120b2f17-0064c45f5c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.952 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:48.953 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:48.953 DEBUG:boto:path=/hiiqos-59/?prefix=obj22 2023-07-29T00:37:48.953 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj22 2023-07-29T00:37:48.954 DEBUG:boto:Method: GET 2023-07-29T00:37:48.954 DEBUG:boto:Path: /hiiqos-59/?prefix=obj22 2023-07-29T00:37:48.955 DEBUG:boto:Data: 2023-07-29T00:37:48.956 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.956 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:48.957 DEBUG:boto:Port: 8001 2023-07-29T00:37:48.957 DEBUG:boto:Params: {} 2023-07-29T00:37:48.958 DEBUG:boto:Token: None 2023-07-29T00:37:48.958 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/ 2023-07-29T00:37:48.958 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28= 2023-07-29T00:37:48.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28=', 'Content-Length': '0'} 2023-07-29T00:37:48.961 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000daca5dffcc9c9ce9-0064c45f5c-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.962 DEBUG:boto:b'hiiqos-59obj221000falseobj222023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:48.962 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:48.963 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:48.963 DEBUG:boto:path=/hiiqos-59/?prefix=obj22 2023-07-29T00:37:48.964 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj22 2023-07-29T00:37:48.964 DEBUG:boto:Method: GET 2023-07-29T00:37:48.964 DEBUG:boto:Path: /hiiqos-59/?prefix=obj22 2023-07-29T00:37:48.965 DEBUG:boto:Data: 2023-07-29T00:37:48.965 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.965 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:48.966 DEBUG:boto:Port: 8000 2023-07-29T00:37:48.966 DEBUG:boto:Params: {} 2023-07-29T00:37:48.967 DEBUG:boto:Token: None 2023-07-29T00:37:48.967 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/ 2023-07-29T00:37:48.968 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28= 2023-07-29T00:37:48.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6dtvw0zg+OTTPLjIc7eStHxof28=', 'Content-Length': '0'} 2023-07-29T00:37:48.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d9f2de0c47e288b-0064c45f5c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.972 DEBUG:boto:b'hiiqos-59obj221000falseobj222023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:48.973 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:37:48.973 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:37:48.974 DEBUG:boto:Method: HEAD 2023-07-29T00:37:48.974 DEBUG:boto:Path: /hiiqos-59/obj22 2023-07-29T00:37:48.974 DEBUG:boto:Data: 2023-07-29T00:37:48.975 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.975 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:48.975 DEBUG:boto:Port: 8001 2023-07-29T00:37:48.976 DEBUG:boto:Params: {} 2023-07-29T00:37:48.977 DEBUG:boto:Token: None 2023-07-29T00:37:48.977 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj22 2023-07-29T00:37:48.978 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VH45qnJD9rVmwbCJPKmHASt4UD0= 2023-07-29T00:37:48.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VH45qnJD9rVmwbCJPKmHASt4UD0=', 'Content-Length': '0'} 2023-07-29T00:37:48.980 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de4551b6b8ddb861-0064c45f5c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.980 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:37:48.981 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:37:48.981 DEBUG:boto:Method: HEAD 2023-07-29T00:37:48.982 DEBUG:boto:Path: /hiiqos-59/obj22 2023-07-29T00:37:48.982 DEBUG:boto:Data: 2023-07-29T00:37:48.982 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.983 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:48.983 DEBUG:boto:Port: 8000 2023-07-29T00:37:48.983 DEBUG:boto:Params: {} 2023-07-29T00:37:48.984 DEBUG:boto:Token: None 2023-07-29T00:37:48.984 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj22 2023-07-29T00:37:48.985 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VH45qnJD9rVmwbCJPKmHASt4UD0= 2023-07-29T00:37:48.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VH45qnJD9rVmwbCJPKmHASt4UD0=', 'Content-Length': '0'} 2023-07-29T00:37:48.987 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4b7f2143e0eebed-0064c45f5c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-07-29T00:37:48.988 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:37:48.988 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:37:48.989 DEBUG:boto:Method: GET 2023-07-29T00:37:48.989 DEBUG:boto:Path: /hiiqos-59/obj22 2023-07-29T00:37:48.990 DEBUG:boto:Data: 2023-07-29T00:37:48.990 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.990 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:48.991 DEBUG:boto:Port: 8001 2023-07-29T00:37:48.991 DEBUG:boto:Params: {} 2023-07-29T00:37:48.991 DEBUG:boto:Token: None 2023-07-29T00:37:48.992 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:48 GMT /hiiqos-59/obj22 2023-07-29T00:37:48.992 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BZDkRheYj/S6IB2GhkvhtV0xrc8= 2023-07-29T00:37:48.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BZDkRheYj/S6IB2GhkvhtV0xrc8=', 'Content-Length': '0'} 2023-07-29T00:37:48.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f01eb2009df97e14-0064c45f5c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:48.995 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:37:48.996 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:37:48.996 DEBUG:boto:Method: GET 2023-07-29T00:37:48.997 DEBUG:boto:Path: /hiiqos-59/obj22 2023-07-29T00:37:48.997 DEBUG:boto:Data: 2023-07-29T00:37:48.998 DEBUG:boto:Headers: {} 2023-07-29T00:37:48.998 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:48.999 DEBUG:boto:Port: 8000 2023-07-29T00:37:48.999 DEBUG:boto:Params: {} 2023-07-29T00:37:48.999 DEBUG:boto:Token: None 2023-07-29T00:37:49.000 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj22 2023-07-29T00:37:49.001 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5lT2IZdrfy/hAsrG86jqtuNnr5A= 2023-07-29T00:37:49.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5lT2IZdrfy/hAsrG86jqtuNnr5A=', 'Content-Length': '0'} 2023-07-29T00:37:49.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a40018a374a57f2c-0064c45f5c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.004 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:49.004 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:49.004 DEBUG:boto:path=/hiiqos-59/?prefix=obj22 2023-07-29T00:37:49.005 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj22 2023-07-29T00:37:49.005 DEBUG:boto:Method: GET 2023-07-29T00:37:49.006 DEBUG:boto:Path: /hiiqos-59/?prefix=obj22 2023-07-29T00:37:49.006 DEBUG:boto:Data: 2023-07-29T00:37:49.006 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.007 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:49.007 DEBUG:boto:Port: 8001 2023-07-29T00:37:49.008 DEBUG:boto:Params: {} 2023-07-29T00:37:49.008 DEBUG:boto:Token: None 2023-07-29T00:37:49.008 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/ 2023-07-29T00:37:49.009 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk= 2023-07-29T00:37:49.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk=', 'Content-Length': '0'} 2023-07-29T00:37:49.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025a76ab38dff5c4e-0064c45f5c-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.012 DEBUG:boto:b'hiiqos-59obj221000falseobj222023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:49.013 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:49.013 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:49.014 DEBUG:boto:path=/hiiqos-59/?prefix=obj22 2023-07-29T00:37:49.014 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj22 2023-07-29T00:37:49.014 DEBUG:boto:Method: GET 2023-07-29T00:37:49.015 DEBUG:boto:Path: /hiiqos-59/?prefix=obj22 2023-07-29T00:37:49.015 DEBUG:boto:Data: 2023-07-29T00:37:49.015 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.016 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:49.016 DEBUG:boto:Port: 8000 2023-07-29T00:37:49.017 DEBUG:boto:Params: {} 2023-07-29T00:37:49.017 DEBUG:boto:Token: None 2023-07-29T00:37:49.018 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/ 2023-07-29T00:37:49.018 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk= 2023-07-29T00:37:49.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk=', 'Content-Length': '0'} 2023-07-29T00:37:49.023 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a51c7ecf4f47b57-0064c45f5c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.023 DEBUG:boto:b'hiiqos-59obj221000falseobj222023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:49.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-07-29T00:37:49.024 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:37:49.025 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:37:49.025 DEBUG:boto:path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:37:49.026 DEBUG:boto:auth_path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:37:49.026 DEBUG:boto:Method: GET 2023-07-29T00:37:49.027 DEBUG:boto:Path: /hiiqos-59/obj23?versionId=null 2023-07-29T00:37:49.027 DEBUG:boto:Data: 2023-07-29T00:37:49.027 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.028 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:49.028 DEBUG:boto:Port: 8001 2023-07-29T00:37:49.028 DEBUG:boto:Params: {} 2023-07-29T00:37:49.029 DEBUG:boto:Token: None 2023-07-29T00:37:49.029 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj23?versionId=null 2023-07-29T00:37:49.030 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SFC4ezqgiHdJxVqjPUMPz8QlnH8= 2023-07-29T00:37:49.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SFC4ezqgiHdJxVqjPUMPz8QlnH8=', 'Content-Length': '0'} 2023-07-29T00:37:49.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a258f89e74fca6b-0064c45f5c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.033 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:37:49.034 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:37:49.034 DEBUG:boto:path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:37:49.034 DEBUG:boto:auth_path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:37:49.035 DEBUG:boto:Method: GET 2023-07-29T00:37:49.036 DEBUG:boto:Path: /hiiqos-59/obj23?versionId=null 2023-07-29T00:37:49.036 DEBUG:boto:Data: 2023-07-29T00:37:49.037 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.038 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:49.039 DEBUG:boto:Port: 8000 2023-07-29T00:37:49.040 DEBUG:boto:Params: {} 2023-07-29T00:37:49.041 DEBUG:boto:Token: None 2023-07-29T00:37:49.041 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj23?versionId=null 2023-07-29T00:37:49.042 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SFC4ezqgiHdJxVqjPUMPz8QlnH8= 2023-07-29T00:37:49.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SFC4ezqgiHdJxVqjPUMPz8QlnH8=', 'Content-Length': '0'} 2023-07-29T00:37:49.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007dfda139b4d3123e-0064c45f5c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.045 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:37:49.045 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:37:49.046 DEBUG:boto:path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:37:49.046 DEBUG:boto:auth_path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:37:49.047 DEBUG:boto:Method: HEAD 2023-07-29T00:37:49.047 DEBUG:boto:Path: /hiiqos-59/obj23?versionId=null 2023-07-29T00:37:49.047 DEBUG:boto:Data: 2023-07-29T00:37:49.048 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.048 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:49.049 DEBUG:boto:Port: 8001 2023-07-29T00:37:49.049 DEBUG:boto:Params: {} 2023-07-29T00:37:49.049 DEBUG:boto:Token: None 2023-07-29T00:37:49.050 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj23?versionId=null 2023-07-29T00:37:49.050 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4QCThDwSBIoFJN7SVgdz9Mu/TZE= 2023-07-29T00:37:49.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4QCThDwSBIoFJN7SVgdz9Mu/TZE=', 'Content-Length': '0'} 2023-07-29T00:37:49.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009c4f9761e97c9c7-0064c45f5c-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.053 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:37:49.053 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:37:49.053 DEBUG:boto:path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:37:49.054 DEBUG:boto:auth_path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:37:49.054 DEBUG:boto:Method: HEAD 2023-07-29T00:37:49.054 DEBUG:boto:Path: /hiiqos-59/obj23?versionId=null 2023-07-29T00:37:49.055 DEBUG:boto:Data: 2023-07-29T00:37:49.055 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.055 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:49.056 DEBUG:boto:Port: 8000 2023-07-29T00:37:49.056 DEBUG:boto:Params: {} 2023-07-29T00:37:49.057 DEBUG:boto:Token: None 2023-07-29T00:37:49.057 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj23?versionId=null 2023-07-29T00:37:49.057 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4QCThDwSBIoFJN7SVgdz9Mu/TZE= 2023-07-29T00:37:49.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4QCThDwSBIoFJN7SVgdz9Mu/TZE=', 'Content-Length': '0'} 2023-07-29T00:37:49.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000733461ad1b21816b-0064c45f5c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-07-29T00:37:49.060 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:37:49.061 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:37:49.061 DEBUG:boto:path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:37:49.062 DEBUG:boto:auth_path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:37:49.062 DEBUG:boto:Method: GET 2023-07-29T00:37:49.063 DEBUG:boto:Path: /hiiqos-59/obj23?versionId=null 2023-07-29T00:37:49.063 DEBUG:boto:Data: 2023-07-29T00:37:49.063 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.064 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:49.064 DEBUG:boto:Port: 8001 2023-07-29T00:37:49.064 DEBUG:boto:Params: {} 2023-07-29T00:37:49.065 DEBUG:boto:Token: None 2023-07-29T00:37:49.065 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj23?versionId=null 2023-07-29T00:37:49.066 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SFC4ezqgiHdJxVqjPUMPz8QlnH8= 2023-07-29T00:37:49.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SFC4ezqgiHdJxVqjPUMPz8QlnH8=', 'Content-Length': '0'} 2023-07-29T00:37:49.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078adde1e5aed9458-0064c45f5d-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.108 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:37:49.109 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:37:49.109 DEBUG:boto:path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:37:49.109 DEBUG:boto:auth_path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:37:49.110 DEBUG:boto:Method: GET 2023-07-29T00:37:49.110 DEBUG:boto:Path: /hiiqos-59/obj23?versionId=null 2023-07-29T00:37:49.110 DEBUG:boto:Data: 2023-07-29T00:37:49.111 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.111 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:49.111 DEBUG:boto:Port: 8000 2023-07-29T00:37:49.112 DEBUG:boto:Params: {} 2023-07-29T00:37:49.112 DEBUG:boto:Token: None 2023-07-29T00:37:49.113 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj23?versionId=null 2023-07-29T00:37:49.113 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SFC4ezqgiHdJxVqjPUMPz8QlnH8= 2023-07-29T00:37:49.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SFC4ezqgiHdJxVqjPUMPz8QlnH8=', 'Content-Length': '0'} 2023-07-29T00:37:49.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014de399dbfb8d8ba-0064c45f5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.116 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:49.116 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:49.117 DEBUG:boto:path=/hiiqos-59/?prefix=obj23 2023-07-29T00:37:49.117 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj23 2023-07-29T00:37:49.117 DEBUG:boto:Method: GET 2023-07-29T00:37:49.118 DEBUG:boto:Path: /hiiqos-59/?prefix=obj23 2023-07-29T00:37:49.118 DEBUG:boto:Data: 2023-07-29T00:37:49.118 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.119 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:49.119 DEBUG:boto:Port: 8001 2023-07-29T00:37:49.120 DEBUG:boto:Params: {} 2023-07-29T00:37:49.120 DEBUG:boto:Token: None 2023-07-29T00:37:49.121 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/ 2023-07-29T00:37:49.121 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk= 2023-07-29T00:37:49.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk=', 'Content-Length': '0'} 2023-07-29T00:37:49.124 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcca17aa7c8aec37-0064c45f5d-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.124 DEBUG:boto:b'hiiqos-59obj231000falseobj232023-07-29T00:33:06.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:49.125 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:49.125 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:49.126 DEBUG:boto:path=/hiiqos-59/?prefix=obj23 2023-07-29T00:37:49.126 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj23 2023-07-29T00:37:49.126 DEBUG:boto:Method: GET 2023-07-29T00:37:49.127 DEBUG:boto:Path: /hiiqos-59/?prefix=obj23 2023-07-29T00:37:49.127 DEBUG:boto:Data: 2023-07-29T00:37:49.128 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.128 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:49.128 DEBUG:boto:Port: 8000 2023-07-29T00:37:49.129 DEBUG:boto:Params: {} 2023-07-29T00:37:49.129 DEBUG:boto:Token: None 2023-07-29T00:37:49.129 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/ 2023-07-29T00:37:49.130 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk= 2023-07-29T00:37:49.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk=', 'Content-Length': '0'} 2023-07-29T00:37:49.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052b8e72087b9cc57-0064c45f5d-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.135 DEBUG:boto:b'hiiqos-59obj231000falseobj232023-07-29T00:33:06.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:49.135 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:37:49.136 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:37:49.136 DEBUG:boto:Method: HEAD 2023-07-29T00:37:49.136 DEBUG:boto:Path: /hiiqos-59/obj23 2023-07-29T00:37:49.137 DEBUG:boto:Data: 2023-07-29T00:37:49.137 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.137 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:49.138 DEBUG:boto:Port: 8001 2023-07-29T00:37:49.138 DEBUG:boto:Params: {} 2023-07-29T00:37:49.138 DEBUG:boto:Token: None 2023-07-29T00:37:49.139 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj23 2023-07-29T00:37:49.139 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SLIZJr43NamjwFDS01492lctGPo= 2023-07-29T00:37:49.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SLIZJr43NamjwFDS01492lctGPo=', 'Content-Length': '0'} 2023-07-29T00:37:49.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000374990a8204a8491-0064c45f5d-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.142 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:37:49.142 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:37:49.143 DEBUG:boto:Method: HEAD 2023-07-29T00:37:49.143 DEBUG:boto:Path: /hiiqos-59/obj23 2023-07-29T00:37:49.143 DEBUG:boto:Data: 2023-07-29T00:37:49.144 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.144 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:49.144 DEBUG:boto:Port: 8000 2023-07-29T00:37:49.145 DEBUG:boto:Params: {} 2023-07-29T00:37:49.145 DEBUG:boto:Token: None 2023-07-29T00:37:49.146 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj23 2023-07-29T00:37:49.146 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SLIZJr43NamjwFDS01492lctGPo= 2023-07-29T00:37:49.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SLIZJr43NamjwFDS01492lctGPo=', 'Content-Length': '0'} 2023-07-29T00:37:49.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036e9ea32927645b8-0064c45f5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-07-29T00:37:49.149 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:37:49.149 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:37:49.150 DEBUG:boto:Method: GET 2023-07-29T00:37:49.150 DEBUG:boto:Path: /hiiqos-59/obj23 2023-07-29T00:37:49.150 DEBUG:boto:Data: 2023-07-29T00:37:49.151 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.151 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:49.151 DEBUG:boto:Port: 8001 2023-07-29T00:37:49.152 DEBUG:boto:Params: {} 2023-07-29T00:37:49.152 DEBUG:boto:Token: None 2023-07-29T00:37:49.153 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj23 2023-07-29T00:37:49.153 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qj5EMxGSHNJNyQTGKP32a05gBWs= 2023-07-29T00:37:49.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qj5EMxGSHNJNyQTGKP32a05gBWs=', 'Content-Length': '0'} 2023-07-29T00:37:49.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfb9bc8cb7301dbb-0064c45f5d-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.156 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:37:49.156 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:37:49.157 DEBUG:boto:Method: GET 2023-07-29T00:37:49.157 DEBUG:boto:Path: /hiiqos-59/obj23 2023-07-29T00:37:49.158 DEBUG:boto:Data: 2023-07-29T00:37:49.158 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.158 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:49.159 DEBUG:boto:Port: 8000 2023-07-29T00:37:49.159 DEBUG:boto:Params: {} 2023-07-29T00:37:49.159 DEBUG:boto:Token: None 2023-07-29T00:37:49.160 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj23 2023-07-29T00:37:49.160 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qj5EMxGSHNJNyQTGKP32a05gBWs= 2023-07-29T00:37:49.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qj5EMxGSHNJNyQTGKP32a05gBWs=', 'Content-Length': '0'} 2023-07-29T00:37:49.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ac46a303f1de0a0-0064c45f5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.163 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:49.163 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:49.164 DEBUG:boto:path=/hiiqos-59/?prefix=obj23 2023-07-29T00:37:49.164 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj23 2023-07-29T00:37:49.165 DEBUG:boto:Method: GET 2023-07-29T00:37:49.165 DEBUG:boto:Path: /hiiqos-59/?prefix=obj23 2023-07-29T00:37:49.165 DEBUG:boto:Data: 2023-07-29T00:37:49.166 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.166 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:49.166 DEBUG:boto:Port: 8001 2023-07-29T00:37:49.167 DEBUG:boto:Params: {} 2023-07-29T00:37:49.167 DEBUG:boto:Token: None 2023-07-29T00:37:49.167 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/ 2023-07-29T00:37:49.168 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk= 2023-07-29T00:37:49.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk=', 'Content-Length': '0'} 2023-07-29T00:37:49.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7a395070a8f16b3-0064c45f5d-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.171 DEBUG:boto:b'hiiqos-59obj231000falseobj232023-07-29T00:33:06.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:49.172 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:49.172 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:49.172 DEBUG:boto:path=/hiiqos-59/?prefix=obj23 2023-07-29T00:37:49.173 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj23 2023-07-29T00:37:49.173 DEBUG:boto:Method: GET 2023-07-29T00:37:49.173 DEBUG:boto:Path: /hiiqos-59/?prefix=obj23 2023-07-29T00:37:49.174 DEBUG:boto:Data: 2023-07-29T00:37:49.174 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.174 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:49.175 DEBUG:boto:Port: 8000 2023-07-29T00:37:49.175 DEBUG:boto:Params: {} 2023-07-29T00:37:49.176 DEBUG:boto:Token: None 2023-07-29T00:37:49.176 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/ 2023-07-29T00:37:49.176 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk= 2023-07-29T00:37:49.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk=', 'Content-Length': '0'} 2023-07-29T00:37:49.181 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000790d03132657a28c-0064c45f5d-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.181 DEBUG:boto:b'hiiqos-59obj231000falseobj232023-07-29T00:33:06.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:49.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-07-29T00:37:49.182 DEBUG:boto:path=/hiiqos-59/obj29 2023-07-29T00:37:49.183 DEBUG:boto:auth_path=/hiiqos-59/obj29 2023-07-29T00:37:49.183 DEBUG:boto:path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:37:49.184 DEBUG:boto:auth_path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:37:49.184 DEBUG:boto:Method: GET 2023-07-29T00:37:49.184 DEBUG:boto:Path: /hiiqos-59/obj29?versionId=null 2023-07-29T00:37:49.185 DEBUG:boto:Data: 2023-07-29T00:37:49.185 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.186 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:49.186 DEBUG:boto:Port: 8001 2023-07-29T00:37:49.186 DEBUG:boto:Params: {} 2023-07-29T00:37:49.187 DEBUG:boto:Token: None 2023-07-29T00:37:49.187 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj29?versionId=null 2023-07-29T00:37:49.187 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1qiqBjf3CKILWzFjotWBlSzKzS0= 2023-07-29T00:37:49.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1qiqBjf3CKILWzFjotWBlSzKzS0=', 'Content-Length': '0'} 2023-07-29T00:37:49.190 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daf8d2bb9fc68f71-0064c45f5d-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.190 DEBUG:boto:path=/hiiqos-59/obj29 2023-07-29T00:37:49.191 DEBUG:boto:auth_path=/hiiqos-59/obj29 2023-07-29T00:37:49.191 DEBUG:boto:path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:37:49.191 DEBUG:boto:auth_path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:37:49.192 DEBUG:boto:Method: GET 2023-07-29T00:37:49.192 DEBUG:boto:Path: /hiiqos-59/obj29?versionId=null 2023-07-29T00:37:49.192 DEBUG:boto:Data: 2023-07-29T00:37:49.193 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.193 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:49.193 DEBUG:boto:Port: 8000 2023-07-29T00:37:49.194 DEBUG:boto:Params: {} 2023-07-29T00:37:49.194 DEBUG:boto:Token: None 2023-07-29T00:37:49.195 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj29?versionId=null 2023-07-29T00:37:49.195 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1qiqBjf3CKILWzFjotWBlSzKzS0= 2023-07-29T00:37:49.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1qiqBjf3CKILWzFjotWBlSzKzS0=', 'Content-Length': '0'} 2023-07-29T00:37:49.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057d9e57665695876-0064c45f5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.198 DEBUG:boto:path=/hiiqos-59/obj29 2023-07-29T00:37:49.198 DEBUG:boto:auth_path=/hiiqos-59/obj29 2023-07-29T00:37:49.198 DEBUG:boto:path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:37:49.199 DEBUG:boto:auth_path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:37:49.199 DEBUG:boto:Method: HEAD 2023-07-29T00:37:49.199 DEBUG:boto:Path: /hiiqos-59/obj29?versionId=null 2023-07-29T00:37:49.200 DEBUG:boto:Data: 2023-07-29T00:37:49.200 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.200 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:49.201 DEBUG:boto:Port: 8001 2023-07-29T00:37:49.201 DEBUG:boto:Params: {} 2023-07-29T00:37:49.201 DEBUG:boto:Token: None 2023-07-29T00:37:49.202 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj29?versionId=null 2023-07-29T00:37:49.202 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O+qorCw5x65gO0iffX1fMydw+94= 2023-07-29T00:37:49.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O+qorCw5x65gO0iffX1fMydw+94=', 'Content-Length': '0'} 2023-07-29T00:37:49.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3681bb3624845b9-0064c45f5d-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.205 DEBUG:boto:path=/hiiqos-59/obj29 2023-07-29T00:37:49.206 DEBUG:boto:auth_path=/hiiqos-59/obj29 2023-07-29T00:37:49.206 DEBUG:boto:path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:37:49.206 DEBUG:boto:auth_path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:37:49.207 DEBUG:boto:Method: HEAD 2023-07-29T00:37:49.207 DEBUG:boto:Path: /hiiqos-59/obj29?versionId=null 2023-07-29T00:37:49.207 DEBUG:boto:Data: 2023-07-29T00:37:49.208 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.208 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:49.208 DEBUG:boto:Port: 8000 2023-07-29T00:37:49.209 DEBUG:boto:Params: {} 2023-07-29T00:37:49.209 DEBUG:boto:Token: None 2023-07-29T00:37:49.209 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj29?versionId=null 2023-07-29T00:37:49.210 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O+qorCw5x65gO0iffX1fMydw+94= 2023-07-29T00:37:49.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O+qorCw5x65gO0iffX1fMydw+94=', 'Content-Length': '0'} 2023-07-29T00:37:49.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000449d7e7c6d4150d1-0064c45f5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-07-29T00:37:49.213 DEBUG:boto:path=/hiiqos-59/obj29 2023-07-29T00:37:49.213 DEBUG:boto:auth_path=/hiiqos-59/obj29 2023-07-29T00:37:49.214 DEBUG:boto:path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:37:49.214 DEBUG:boto:auth_path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:37:49.214 DEBUG:boto:Method: GET 2023-07-29T00:37:49.215 DEBUG:boto:Path: /hiiqos-59/obj29?versionId=null 2023-07-29T00:37:49.215 DEBUG:boto:Data: 2023-07-29T00:37:49.216 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.216 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:49.216 DEBUG:boto:Port: 8001 2023-07-29T00:37:49.217 DEBUG:boto:Params: {} 2023-07-29T00:37:49.217 DEBUG:boto:Token: None 2023-07-29T00:37:49.217 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj29?versionId=null 2023-07-29T00:37:49.218 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1qiqBjf3CKILWzFjotWBlSzKzS0= 2023-07-29T00:37:49.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1qiqBjf3CKILWzFjotWBlSzKzS0=', 'Content-Length': '0'} 2023-07-29T00:37:49.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e232b393d4901d4a-0064c45f5d-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.220 DEBUG:boto:path=/hiiqos-59/obj29 2023-07-29T00:37:49.221 DEBUG:boto:auth_path=/hiiqos-59/obj29 2023-07-29T00:37:49.221 DEBUG:boto:path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:37:49.222 DEBUG:boto:auth_path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:37:49.222 DEBUG:boto:Method: GET 2023-07-29T00:37:49.222 DEBUG:boto:Path: /hiiqos-59/obj29?versionId=null 2023-07-29T00:37:49.223 DEBUG:boto:Data: 2023-07-29T00:37:49.223 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.223 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:49.224 DEBUG:boto:Port: 8000 2023-07-29T00:37:49.224 DEBUG:boto:Params: {} 2023-07-29T00:37:49.224 DEBUG:boto:Token: None 2023-07-29T00:37:49.225 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj29?versionId=null 2023-07-29T00:37:49.225 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1qiqBjf3CKILWzFjotWBlSzKzS0= 2023-07-29T00:37:49.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1qiqBjf3CKILWzFjotWBlSzKzS0=', 'Content-Length': '0'} 2023-07-29T00:37:49.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000171dd61bf3309174-0064c45f5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.268 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:49.269 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:49.269 DEBUG:boto:path=/hiiqos-59/?prefix=obj29 2023-07-29T00:37:49.270 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj29 2023-07-29T00:37:49.270 DEBUG:boto:Method: GET 2023-07-29T00:37:49.270 DEBUG:boto:Path: /hiiqos-59/?prefix=obj29 2023-07-29T00:37:49.271 DEBUG:boto:Data: 2023-07-29T00:37:49.271 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.271 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:49.272 DEBUG:boto:Port: 8001 2023-07-29T00:37:49.272 DEBUG:boto:Params: {} 2023-07-29T00:37:49.273 DEBUG:boto:Token: None 2023-07-29T00:37:49.273 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/ 2023-07-29T00:37:49.273 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk= 2023-07-29T00:37:49.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk=', 'Content-Length': '0'} 2023-07-29T00:37:49.276 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da13b7a9a0b28552-0064c45f5d-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.277 DEBUG:boto:b'hiiqos-59obj291000falseobj292023-07-29T00:35:01.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:49.277 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:49.278 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:49.278 DEBUG:boto:path=/hiiqos-59/?prefix=obj29 2023-07-29T00:37:49.278 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj29 2023-07-29T00:37:49.279 DEBUG:boto:Method: GET 2023-07-29T00:37:49.279 DEBUG:boto:Path: /hiiqos-59/?prefix=obj29 2023-07-29T00:37:49.279 DEBUG:boto:Data: 2023-07-29T00:37:49.280 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.280 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:49.281 DEBUG:boto:Port: 8000 2023-07-29T00:37:49.281 DEBUG:boto:Params: {} 2023-07-29T00:37:49.281 DEBUG:boto:Token: None 2023-07-29T00:37:49.282 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/ 2023-07-29T00:37:49.282 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk= 2023-07-29T00:37:49.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk=', 'Content-Length': '0'} 2023-07-29T00:37:49.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c17b469c82c7d5b0-0064c45f5d-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.287 DEBUG:boto:b'hiiqos-59obj291000falseobj292023-07-29T00:35:01.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:49.287 DEBUG:boto:path=/hiiqos-59/obj29 2023-07-29T00:37:49.288 DEBUG:boto:auth_path=/hiiqos-59/obj29 2023-07-29T00:37:49.288 DEBUG:boto:Method: HEAD 2023-07-29T00:37:49.288 DEBUG:boto:Path: /hiiqos-59/obj29 2023-07-29T00:37:49.289 DEBUG:boto:Data: 2023-07-29T00:37:49.289 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.289 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:49.290 DEBUG:boto:Port: 8001 2023-07-29T00:37:49.290 DEBUG:boto:Params: {} 2023-07-29T00:37:49.290 DEBUG:boto:Token: None 2023-07-29T00:37:49.291 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj29 2023-07-29T00:37:49.291 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rJzLxjZFySaWxPGIdq7wKI+MXnw= 2023-07-29T00:37:49.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rJzLxjZFySaWxPGIdq7wKI+MXnw=', 'Content-Length': '0'} 2023-07-29T00:37:49.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014da9a5ee42f9380-0064c45f5d-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.294 DEBUG:boto:path=/hiiqos-59/obj29 2023-07-29T00:37:49.294 DEBUG:boto:auth_path=/hiiqos-59/obj29 2023-07-29T00:37:49.294 DEBUG:boto:Method: HEAD 2023-07-29T00:37:49.295 DEBUG:boto:Path: /hiiqos-59/obj29 2023-07-29T00:37:49.295 DEBUG:boto:Data: 2023-07-29T00:37:49.295 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.296 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:49.296 DEBUG:boto:Port: 8000 2023-07-29T00:37:49.296 DEBUG:boto:Params: {} 2023-07-29T00:37:49.297 DEBUG:boto:Token: None 2023-07-29T00:37:49.297 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj29 2023-07-29T00:37:49.298 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rJzLxjZFySaWxPGIdq7wKI+MXnw= 2023-07-29T00:37:49.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rJzLxjZFySaWxPGIdq7wKI+MXnw=', 'Content-Length': '0'} 2023-07-29T00:37:49.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c21ce023630e0ab6-0064c45f5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-07-29T00:37:49.301 DEBUG:boto:path=/hiiqos-59/obj29 2023-07-29T00:37:49.301 DEBUG:boto:auth_path=/hiiqos-59/obj29 2023-07-29T00:37:49.301 DEBUG:boto:Method: GET 2023-07-29T00:37:49.302 DEBUG:boto:Path: /hiiqos-59/obj29 2023-07-29T00:37:49.302 DEBUG:boto:Data: 2023-07-29T00:37:49.303 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.303 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:49.303 DEBUG:boto:Port: 8001 2023-07-29T00:37:49.304 DEBUG:boto:Params: {} 2023-07-29T00:37:49.304 DEBUG:boto:Token: None 2023-07-29T00:37:49.304 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj29 2023-07-29T00:37:49.305 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GkChkQPcT9m87n2KlTxew8aybp8= 2023-07-29T00:37:49.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GkChkQPcT9m87n2KlTxew8aybp8=', 'Content-Length': '0'} 2023-07-29T00:37:49.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000070a3e53c6db0781-0064c45f5d-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.348 DEBUG:boto:path=/hiiqos-59/obj29 2023-07-29T00:37:49.349 DEBUG:boto:auth_path=/hiiqos-59/obj29 2023-07-29T00:37:49.350 DEBUG:boto:Method: GET 2023-07-29T00:37:49.350 DEBUG:boto:Path: /hiiqos-59/obj29 2023-07-29T00:37:49.351 DEBUG:boto:Data: 2023-07-29T00:37:49.351 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.352 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:49.353 DEBUG:boto:Port: 8000 2023-07-29T00:37:49.353 DEBUG:boto:Params: {} 2023-07-29T00:37:49.354 DEBUG:boto:Token: None 2023-07-29T00:37:49.354 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj29 2023-07-29T00:37:49.354 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GkChkQPcT9m87n2KlTxew8aybp8= 2023-07-29T00:37:49.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GkChkQPcT9m87n2KlTxew8aybp8=', 'Content-Length': '0'} 2023-07-29T00:37:49.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071e9ecf3a1e0d5f9-0064c45f5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.363 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:49.363 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:49.364 DEBUG:boto:path=/hiiqos-59/?prefix=obj29 2023-07-29T00:37:49.365 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj29 2023-07-29T00:37:49.365 DEBUG:boto:Method: GET 2023-07-29T00:37:49.366 DEBUG:boto:Path: /hiiqos-59/?prefix=obj29 2023-07-29T00:37:49.367 DEBUG:boto:Data: 2023-07-29T00:37:49.368 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.368 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:49.369 DEBUG:boto:Port: 8001 2023-07-29T00:37:49.369 DEBUG:boto:Params: {} 2023-07-29T00:37:49.370 DEBUG:boto:Token: None 2023-07-29T00:37:49.370 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/ 2023-07-29T00:37:49.371 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk= 2023-07-29T00:37:49.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk=', 'Content-Length': '0'} 2023-07-29T00:37:49.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000931cd91f4974440c-0064c45f5d-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.376 DEBUG:boto:b'hiiqos-59obj291000falseobj292023-07-29T00:35:01.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:49.377 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:49.377 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:49.378 DEBUG:boto:path=/hiiqos-59/?prefix=obj29 2023-07-29T00:37:49.378 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj29 2023-07-29T00:37:49.379 DEBUG:boto:Method: GET 2023-07-29T00:37:49.379 DEBUG:boto:Path: /hiiqos-59/?prefix=obj29 2023-07-29T00:37:49.380 DEBUG:boto:Data: 2023-07-29T00:37:49.380 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.380 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:49.381 DEBUG:boto:Port: 8000 2023-07-29T00:37:49.381 DEBUG:boto:Params: {} 2023-07-29T00:37:49.382 DEBUG:boto:Token: None 2023-07-29T00:37:49.382 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/ 2023-07-29T00:37:49.383 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk= 2023-07-29T00:37:49.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk=', 'Content-Length': '0'} 2023-07-29T00:37:49.389 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e68aecf7bb6ee4a7-0064c45f5d-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.432 DEBUG:boto:b'hiiqos-59obj291000falseobj292023-07-29T00:35:01.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:49.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-07-29T00:37:49.434 DEBUG:boto:path=/hiiqos-59/obj31 2023-07-29T00:37:49.435 DEBUG:boto:auth_path=/hiiqos-59/obj31 2023-07-29T00:37:49.435 DEBUG:boto:path=/hiiqos-59/obj31?versionId=null 2023-07-29T00:37:49.435 DEBUG:boto:auth_path=/hiiqos-59/obj31?versionId=null 2023-07-29T00:37:49.436 DEBUG:boto:Method: GET 2023-07-29T00:37:49.436 DEBUG:boto:Path: /hiiqos-59/obj31?versionId=null 2023-07-29T00:37:49.436 DEBUG:boto:Data: 2023-07-29T00:37:49.437 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.437 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:49.437 DEBUG:boto:Port: 8001 2023-07-29T00:37:49.438 DEBUG:boto:Params: {} 2023-07-29T00:37:49.438 DEBUG:boto:Token: None 2023-07-29T00:37:49.439 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj31?versionId=null 2023-07-29T00:37:49.439 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nvr4Eyr7cpp6VO/uoX8cZS3h2iE= 2023-07-29T00:37:49.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nvr4Eyr7cpp6VO/uoX8cZS3h2iE=', 'Content-Length': '0'} 2023-07-29T00:37:49.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000082e08ed6be06731-0064c45f5d-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.444 DEBUG:boto:path=/hiiqos-59/obj31 2023-07-29T00:37:49.445 DEBUG:boto:auth_path=/hiiqos-59/obj31 2023-07-29T00:37:49.445 DEBUG:boto:path=/hiiqos-59/obj31?versionId=null 2023-07-29T00:37:49.445 DEBUG:boto:auth_path=/hiiqos-59/obj31?versionId=null 2023-07-29T00:37:49.446 DEBUG:boto:Method: GET 2023-07-29T00:37:49.446 DEBUG:boto:Path: /hiiqos-59/obj31?versionId=null 2023-07-29T00:37:49.447 DEBUG:boto:Data: 2023-07-29T00:37:49.447 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.447 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:49.448 DEBUG:boto:Port: 8000 2023-07-29T00:37:49.448 DEBUG:boto:Params: {} 2023-07-29T00:37:49.448 DEBUG:boto:Token: None 2023-07-29T00:37:49.449 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj31?versionId=null 2023-07-29T00:37:49.449 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nvr4Eyr7cpp6VO/uoX8cZS3h2iE= 2023-07-29T00:37:49.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nvr4Eyr7cpp6VO/uoX8cZS3h2iE=', 'Content-Length': '0'} 2023-07-29T00:37:49.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f485913a87f0b75c-0064c45f5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.453 DEBUG:boto:path=/hiiqos-59/obj31 2023-07-29T00:37:49.453 DEBUG:boto:auth_path=/hiiqos-59/obj31 2023-07-29T00:37:49.454 DEBUG:boto:path=/hiiqos-59/obj31?versionId=null 2023-07-29T00:37:49.454 DEBUG:boto:auth_path=/hiiqos-59/obj31?versionId=null 2023-07-29T00:37:49.454 DEBUG:boto:Method: HEAD 2023-07-29T00:37:49.455 DEBUG:boto:Path: /hiiqos-59/obj31?versionId=null 2023-07-29T00:37:49.455 DEBUG:boto:Data: 2023-07-29T00:37:49.455 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.456 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:49.456 DEBUG:boto:Port: 8001 2023-07-29T00:37:49.457 DEBUG:boto:Params: {} 2023-07-29T00:37:49.457 DEBUG:boto:Token: None 2023-07-29T00:37:49.457 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj31?versionId=null 2023-07-29T00:37:49.458 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Fbdt62iGsbHosTUa1H/3SMUOYuA= 2023-07-29T00:37:49.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Fbdt62iGsbHosTUa1H/3SMUOYuA=', 'Content-Length': '0'} 2023-07-29T00:37:49.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033cf50d5904b56a7-0064c45f5d-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.462 DEBUG:boto:path=/hiiqos-59/obj31 2023-07-29T00:37:49.462 DEBUG:boto:auth_path=/hiiqos-59/obj31 2023-07-29T00:37:49.462 DEBUG:boto:path=/hiiqos-59/obj31?versionId=null 2023-07-29T00:37:49.463 DEBUG:boto:auth_path=/hiiqos-59/obj31?versionId=null 2023-07-29T00:37:49.463 DEBUG:boto:Method: HEAD 2023-07-29T00:37:49.463 DEBUG:boto:Path: /hiiqos-59/obj31?versionId=null 2023-07-29T00:37:49.464 DEBUG:boto:Data: 2023-07-29T00:37:49.464 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.464 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:49.465 DEBUG:boto:Port: 8000 2023-07-29T00:37:49.465 DEBUG:boto:Params: {} 2023-07-29T00:37:49.465 DEBUG:boto:Token: None 2023-07-29T00:37:49.466 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj31?versionId=null 2023-07-29T00:37:49.466 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Fbdt62iGsbHosTUa1H/3SMUOYuA= 2023-07-29T00:37:49.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Fbdt62iGsbHosTUa1H/3SMUOYuA=', 'Content-Length': '0'} 2023-07-29T00:37:49.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028c4dac584dafab4-0064c45f5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-07-29T00:37:49.470 DEBUG:boto:path=/hiiqos-59/obj31 2023-07-29T00:37:49.471 DEBUG:boto:auth_path=/hiiqos-59/obj31 2023-07-29T00:37:49.471 DEBUG:boto:path=/hiiqos-59/obj31?versionId=null 2023-07-29T00:37:49.471 DEBUG:boto:auth_path=/hiiqos-59/obj31?versionId=null 2023-07-29T00:37:49.472 DEBUG:boto:Method: GET 2023-07-29T00:37:49.472 DEBUG:boto:Path: /hiiqos-59/obj31?versionId=null 2023-07-29T00:37:49.473 DEBUG:boto:Data: 2023-07-29T00:37:49.473 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.473 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:49.474 DEBUG:boto:Port: 8001 2023-07-29T00:37:49.474 DEBUG:boto:Params: {} 2023-07-29T00:37:49.474 DEBUG:boto:Token: None 2023-07-29T00:37:49.475 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj31?versionId=null 2023-07-29T00:37:49.475 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nvr4Eyr7cpp6VO/uoX8cZS3h2iE= 2023-07-29T00:37:49.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nvr4Eyr7cpp6VO/uoX8cZS3h2iE=', 'Content-Length': '0'} 2023-07-29T00:37:49.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a667c820b56b32c5-0064c45f5d-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.480 DEBUG:boto:path=/hiiqos-59/obj31 2023-07-29T00:37:49.480 DEBUG:boto:auth_path=/hiiqos-59/obj31 2023-07-29T00:37:49.481 DEBUG:boto:path=/hiiqos-59/obj31?versionId=null 2023-07-29T00:37:49.481 DEBUG:boto:auth_path=/hiiqos-59/obj31?versionId=null 2023-07-29T00:37:49.481 DEBUG:boto:Method: GET 2023-07-29T00:37:49.482 DEBUG:boto:Path: /hiiqos-59/obj31?versionId=null 2023-07-29T00:37:49.482 DEBUG:boto:Data: 2023-07-29T00:37:49.482 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.483 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:49.483 DEBUG:boto:Port: 8000 2023-07-29T00:37:49.483 DEBUG:boto:Params: {} 2023-07-29T00:37:49.484 DEBUG:boto:Token: None 2023-07-29T00:37:49.484 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj31?versionId=null 2023-07-29T00:37:49.485 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nvr4Eyr7cpp6VO/uoX8cZS3h2iE= 2023-07-29T00:37:49.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nvr4Eyr7cpp6VO/uoX8cZS3h2iE=', 'Content-Length': '0'} 2023-07-29T00:37:49.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004dcf5e761b490684-0064c45f5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.532 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:49.533 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:49.534 DEBUG:boto:path=/hiiqos-59/?prefix=obj31 2023-07-29T00:37:49.534 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj31 2023-07-29T00:37:49.535 DEBUG:boto:Method: GET 2023-07-29T00:37:49.535 DEBUG:boto:Path: /hiiqos-59/?prefix=obj31 2023-07-29T00:37:49.535 DEBUG:boto:Data: 2023-07-29T00:37:49.536 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.536 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:49.536 DEBUG:boto:Port: 8001 2023-07-29T00:37:49.537 DEBUG:boto:Params: {} 2023-07-29T00:37:49.537 DEBUG:boto:Token: None 2023-07-29T00:37:49.538 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/ 2023-07-29T00:37:49.538 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk= 2023-07-29T00:37:49.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk=', 'Content-Length': '0'} 2023-07-29T00:37:49.541 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e09124972fb9843-0064c45f5d-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.542 DEBUG:boto:b'hiiqos-59obj311000falseobj312023-07-29T00:35:31.771Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:49.543 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:49.543 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:49.543 DEBUG:boto:path=/hiiqos-59/?prefix=obj31 2023-07-29T00:37:49.544 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj31 2023-07-29T00:37:49.544 DEBUG:boto:Method: GET 2023-07-29T00:37:49.545 DEBUG:boto:Path: /hiiqos-59/?prefix=obj31 2023-07-29T00:37:49.545 DEBUG:boto:Data: 2023-07-29T00:37:49.545 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.546 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:49.546 DEBUG:boto:Port: 8000 2023-07-29T00:37:49.546 DEBUG:boto:Params: {} 2023-07-29T00:37:49.547 DEBUG:boto:Token: None 2023-07-29T00:37:49.547 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/ 2023-07-29T00:37:49.548 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk= 2023-07-29T00:37:49.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk=', 'Content-Length': '0'} 2023-07-29T00:37:49.552 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f052f3b1eac81c3-0064c45f5d-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.553 DEBUG:boto:b'hiiqos-59obj311000falseobj312023-07-29T00:35:31.771Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:49.553 DEBUG:boto:path=/hiiqos-59/obj31 2023-07-29T00:37:49.554 DEBUG:boto:auth_path=/hiiqos-59/obj31 2023-07-29T00:37:49.554 DEBUG:boto:Method: HEAD 2023-07-29T00:37:49.554 DEBUG:boto:Path: /hiiqos-59/obj31 2023-07-29T00:37:49.555 DEBUG:boto:Data: 2023-07-29T00:37:49.555 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.555 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:49.556 DEBUG:boto:Port: 8001 2023-07-29T00:37:49.556 DEBUG:boto:Params: {} 2023-07-29T00:37:49.556 DEBUG:boto:Token: None 2023-07-29T00:37:49.557 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj31 2023-07-29T00:37:49.557 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FVGLFrkkr+G9f2F6JJkUDSno/RE= 2023-07-29T00:37:49.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FVGLFrkkr+G9f2F6JJkUDSno/RE=', 'Content-Length': '0'} 2023-07-29T00:37:49.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000933d0ae3cc3a347a-0064c45f5d-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.561 DEBUG:boto:path=/hiiqos-59/obj31 2023-07-29T00:37:49.561 DEBUG:boto:auth_path=/hiiqos-59/obj31 2023-07-29T00:37:49.561 DEBUG:boto:Method: HEAD 2023-07-29T00:37:49.562 DEBUG:boto:Path: /hiiqos-59/obj31 2023-07-29T00:37:49.562 DEBUG:boto:Data: 2023-07-29T00:37:49.562 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.563 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:49.563 DEBUG:boto:Port: 8000 2023-07-29T00:37:49.563 DEBUG:boto:Params: {} 2023-07-29T00:37:49.564 DEBUG:boto:Token: None 2023-07-29T00:37:49.564 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj31 2023-07-29T00:37:49.565 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FVGLFrkkr+G9f2F6JJkUDSno/RE= 2023-07-29T00:37:49.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FVGLFrkkr+G9f2F6JJkUDSno/RE=', 'Content-Length': '0'} 2023-07-29T00:37:49.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021f209f02b0d557d-0064c45f5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-07-29T00:37:49.568 DEBUG:boto:path=/hiiqos-59/obj31 2023-07-29T00:37:49.569 DEBUG:boto:auth_path=/hiiqos-59/obj31 2023-07-29T00:37:49.569 DEBUG:boto:Method: GET 2023-07-29T00:37:49.570 DEBUG:boto:Path: /hiiqos-59/obj31 2023-07-29T00:37:49.570 DEBUG:boto:Data: 2023-07-29T00:37:49.570 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.571 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:49.571 DEBUG:boto:Port: 8001 2023-07-29T00:37:49.571 DEBUG:boto:Params: {} 2023-07-29T00:37:49.572 DEBUG:boto:Token: None 2023-07-29T00:37:49.572 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj31 2023-07-29T00:37:49.573 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3XkIPBW1Nn86P6yotdKwIcaNsSc= 2023-07-29T00:37:49.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3XkIPBW1Nn86P6yotdKwIcaNsSc=', 'Content-Length': '0'} 2023-07-29T00:37:49.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1fea959a4497a51-0064c45f5d-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.616 DEBUG:boto:path=/hiiqos-59/obj31 2023-07-29T00:37:49.617 DEBUG:boto:auth_path=/hiiqos-59/obj31 2023-07-29T00:37:49.617 DEBUG:boto:Method: GET 2023-07-29T00:37:49.618 DEBUG:boto:Path: /hiiqos-59/obj31 2023-07-29T00:37:49.618 DEBUG:boto:Data: 2023-07-29T00:37:49.619 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.619 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:49.619 DEBUG:boto:Port: 8000 2023-07-29T00:37:49.620 DEBUG:boto:Params: {} 2023-07-29T00:37:49.620 DEBUG:boto:Token: None 2023-07-29T00:37:49.621 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj31 2023-07-29T00:37:49.621 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3XkIPBW1Nn86P6yotdKwIcaNsSc= 2023-07-29T00:37:49.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3XkIPBW1Nn86P6yotdKwIcaNsSc=', 'Content-Length': '0'} 2023-07-29T00:37:49.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003defdd88a5b3e76e-0064c45f5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.625 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:49.625 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:49.625 DEBUG:boto:path=/hiiqos-59/?prefix=obj31 2023-07-29T00:37:49.626 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj31 2023-07-29T00:37:49.626 DEBUG:boto:Method: GET 2023-07-29T00:37:49.627 DEBUG:boto:Path: /hiiqos-59/?prefix=obj31 2023-07-29T00:37:49.627 DEBUG:boto:Data: 2023-07-29T00:37:49.627 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.628 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:49.628 DEBUG:boto:Port: 8001 2023-07-29T00:37:49.629 DEBUG:boto:Params: {} 2023-07-29T00:37:49.629 DEBUG:boto:Token: None 2023-07-29T00:37:49.630 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/ 2023-07-29T00:37:49.630 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk= 2023-07-29T00:37:49.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk=', 'Content-Length': '0'} 2023-07-29T00:37:49.633 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f3ad2d780e79740-0064c45f5d-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.633 DEBUG:boto:b'hiiqos-59obj311000falseobj312023-07-29T00:35:31.771Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:49.634 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:49.634 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:49.635 DEBUG:boto:path=/hiiqos-59/?prefix=obj31 2023-07-29T00:37:49.635 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj31 2023-07-29T00:37:49.636 DEBUG:boto:Method: GET 2023-07-29T00:37:49.636 DEBUG:boto:Path: /hiiqos-59/?prefix=obj31 2023-07-29T00:37:49.636 DEBUG:boto:Data: 2023-07-29T00:37:49.637 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.637 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:49.637 DEBUG:boto:Port: 8000 2023-07-29T00:37:49.638 DEBUG:boto:Params: {} 2023-07-29T00:37:49.638 DEBUG:boto:Token: None 2023-07-29T00:37:49.639 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/ 2023-07-29T00:37:49.639 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk= 2023-07-29T00:37:49.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk=', 'Content-Length': '0'} 2023-07-29T00:37:49.644 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbbe5a452d3a765b-0064c45f5d-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.644 DEBUG:boto:b'hiiqos-59obj311000falseobj312023-07-29T00:35:31.771Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:49.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-07-29T00:37:49.646 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:37:49.646 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:37:49.646 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:37:49.647 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:37:49.647 DEBUG:boto:Method: GET 2023-07-29T00:37:49.647 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:37:49.648 DEBUG:boto:Data: 2023-07-29T00:37:49.648 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.649 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:49.649 DEBUG:boto:Port: 8001 2023-07-29T00:37:49.649 DEBUG:boto:Params: {} 2023-07-29T00:37:49.650 DEBUG:boto:Token: None 2023-07-29T00:37:49.650 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:37:49.651 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2ZLeLITEgvoo1hHJDz/8Yk9doVs= 2023-07-29T00:37:49.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2ZLeLITEgvoo1hHJDz/8Yk9doVs=', 'Content-Length': '0'} 2023-07-29T00:37:49.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f3f957bd1e1749c-0064c45f5d-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.654 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:37:49.654 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:37:49.654 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:37:49.655 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:37:49.655 DEBUG:boto:Method: GET 2023-07-29T00:37:49.655 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:37:49.656 DEBUG:boto:Data: 2023-07-29T00:37:49.656 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.657 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:49.657 DEBUG:boto:Port: 8000 2023-07-29T00:37:49.657 DEBUG:boto:Params: {} 2023-07-29T00:37:49.658 DEBUG:boto:Token: None 2023-07-29T00:37:49.658 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:37:49.659 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2ZLeLITEgvoo1hHJDz/8Yk9doVs= 2023-07-29T00:37:49.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2ZLeLITEgvoo1hHJDz/8Yk9doVs=', 'Content-Length': '0'} 2023-07-29T00:37:49.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006702e6f74308da4d-0064c45f5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.704 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:37:49.705 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:37:49.705 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:37:49.705 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:37:49.706 DEBUG:boto:Method: HEAD 2023-07-29T00:37:49.706 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:37:49.706 DEBUG:boto:Data: 2023-07-29T00:37:49.707 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.707 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:49.707 DEBUG:boto:Port: 8001 2023-07-29T00:37:49.708 DEBUG:boto:Params: {} 2023-07-29T00:37:49.708 DEBUG:boto:Token: None 2023-07-29T00:37:49.708 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:37:49.709 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w/vPyoFY0FF6FrRnNRid11ERdBg= 2023-07-29T00:37:49.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w/vPyoFY0FF6FrRnNRid11ERdBg=', 'Content-Length': '0'} 2023-07-29T00:37:49.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092d0cbf448bb7450-0064c45f5d-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.711 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:37:49.712 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:37:49.712 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:37:49.713 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:37:49.713 DEBUG:boto:Method: HEAD 2023-07-29T00:37:49.713 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:37:49.714 DEBUG:boto:Data: 2023-07-29T00:37:49.714 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.714 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:49.715 DEBUG:boto:Port: 8000 2023-07-29T00:37:49.715 DEBUG:boto:Params: {} 2023-07-29T00:37:49.715 DEBUG:boto:Token: None 2023-07-29T00:37:49.716 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:37:49.716 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w/vPyoFY0FF6FrRnNRid11ERdBg= 2023-07-29T00:37:49.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w/vPyoFY0FF6FrRnNRid11ERdBg=', 'Content-Length': '0'} 2023-07-29T00:37:49.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b386b783d4dc5d7e-0064c45f5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-07-29T00:37:49.720 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:37:49.720 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:37:49.720 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:37:49.721 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:37:49.721 DEBUG:boto:Method: GET 2023-07-29T00:37:49.722 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:37:49.722 DEBUG:boto:Data: 2023-07-29T00:37:49.722 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.723 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:49.723 DEBUG:boto:Port: 8001 2023-07-29T00:37:49.723 DEBUG:boto:Params: {} 2023-07-29T00:37:49.724 DEBUG:boto:Token: None 2023-07-29T00:37:49.724 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:37:49.725 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2ZLeLITEgvoo1hHJDz/8Yk9doVs= 2023-07-29T00:37:49.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2ZLeLITEgvoo1hHJDz/8Yk9doVs=', 'Content-Length': '0'} 2023-07-29T00:37:49.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000869bd931293330c7-0064c45f5d-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.728 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:37:49.728 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:37:49.728 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:37:49.729 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:37:49.729 DEBUG:boto:Method: GET 2023-07-29T00:37:49.729 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:37:49.730 DEBUG:boto:Data: 2023-07-29T00:37:49.730 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.730 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:49.731 DEBUG:boto:Port: 8000 2023-07-29T00:37:49.731 DEBUG:boto:Params: {} 2023-07-29T00:37:49.732 DEBUG:boto:Token: None 2023-07-29T00:37:49.732 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:37:49.732 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2ZLeLITEgvoo1hHJDz/8Yk9doVs= 2023-07-29T00:37:49.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2ZLeLITEgvoo1hHJDz/8Yk9doVs=', 'Content-Length': '0'} 2023-07-29T00:37:49.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025645ba129e68930-0064c45f5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.735 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:49.736 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:49.736 DEBUG:boto:path=/hiiqos-59/?prefix=obj33 2023-07-29T00:37:49.736 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj33 2023-07-29T00:37:49.737 DEBUG:boto:Method: GET 2023-07-29T00:37:49.737 DEBUG:boto:Path: /hiiqos-59/?prefix=obj33 2023-07-29T00:37:49.738 DEBUG:boto:Data: 2023-07-29T00:37:49.738 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.738 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:49.739 DEBUG:boto:Port: 8001 2023-07-29T00:37:49.739 DEBUG:boto:Params: {} 2023-07-29T00:37:49.739 DEBUG:boto:Token: None 2023-07-29T00:37:49.740 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/ 2023-07-29T00:37:49.740 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk= 2023-07-29T00:37:49.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk=', 'Content-Length': '0'} 2023-07-29T00:37:49.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f29811b7f122f20-0064c45f5d-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.744 DEBUG:boto:b'hiiqos-59obj331000falseobj332023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:49.744 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:49.745 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:49.747 DEBUG:boto:path=/hiiqos-59/?prefix=obj33 2023-07-29T00:37:49.747 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj33 2023-07-29T00:37:49.748 DEBUG:boto:Method: GET 2023-07-29T00:37:49.748 DEBUG:boto:Path: /hiiqos-59/?prefix=obj33 2023-07-29T00:37:49.749 DEBUG:boto:Data: 2023-07-29T00:37:49.749 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.749 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:49.750 DEBUG:boto:Port: 8000 2023-07-29T00:37:49.750 DEBUG:boto:Params: {} 2023-07-29T00:37:49.750 DEBUG:boto:Token: None 2023-07-29T00:37:49.751 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/ 2023-07-29T00:37:49.751 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk= 2023-07-29T00:37:49.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk=', 'Content-Length': '0'} 2023-07-29T00:37:49.755 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5ed05d4c082244f-0064c45f5d-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.756 DEBUG:boto:b'hiiqos-59obj331000falseobj332023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:49.756 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:37:49.757 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:37:49.757 DEBUG:boto:Method: HEAD 2023-07-29T00:37:49.758 DEBUG:boto:Path: /hiiqos-59/obj33 2023-07-29T00:37:49.758 DEBUG:boto:Data: 2023-07-29T00:37:49.758 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.759 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:49.759 DEBUG:boto:Port: 8001 2023-07-29T00:37:49.759 DEBUG:boto:Params: {} 2023-07-29T00:37:49.760 DEBUG:boto:Token: None 2023-07-29T00:37:49.760 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj33 2023-07-29T00:37:49.761 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LCvTPFBAuwT8/MaRgI7qplsju/k= 2023-07-29T00:37:49.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LCvTPFBAuwT8/MaRgI7qplsju/k=', 'Content-Length': '0'} 2023-07-29T00:37:49.763 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028dd4d1e0e6f51e2-0064c45f5d-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.764 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:37:49.764 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:37:49.765 DEBUG:boto:Method: HEAD 2023-07-29T00:37:49.765 DEBUG:boto:Path: /hiiqos-59/obj33 2023-07-29T00:37:49.765 DEBUG:boto:Data: 2023-07-29T00:37:49.766 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.766 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:49.766 DEBUG:boto:Port: 8000 2023-07-29T00:37:49.767 DEBUG:boto:Params: {} 2023-07-29T00:37:49.767 DEBUG:boto:Token: None 2023-07-29T00:37:49.767 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj33 2023-07-29T00:37:49.768 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LCvTPFBAuwT8/MaRgI7qplsju/k= 2023-07-29T00:37:49.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LCvTPFBAuwT8/MaRgI7qplsju/k=', 'Content-Length': '0'} 2023-07-29T00:37:49.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2610827173b0405-0064c45f5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-07-29T00:37:49.771 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:37:49.772 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:37:49.772 DEBUG:boto:Method: GET 2023-07-29T00:37:49.773 DEBUG:boto:Path: /hiiqos-59/obj33 2023-07-29T00:37:49.773 DEBUG:boto:Data: 2023-07-29T00:37:49.774 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.774 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:49.775 DEBUG:boto:Port: 8001 2023-07-29T00:37:49.775 DEBUG:boto:Params: {} 2023-07-29T00:37:49.776 DEBUG:boto:Token: None 2023-07-29T00:37:49.776 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj33 2023-07-29T00:37:49.777 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2V075UR1ULMSZtV3TI6F1W+dqL8= 2023-07-29T00:37:49.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2V075UR1ULMSZtV3TI6F1W+dqL8=', 'Content-Length': '0'} 2023-07-29T00:37:49.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093229bdcbdc58759-0064c45f5d-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.780 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:37:49.781 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:37:49.781 DEBUG:boto:Method: GET 2023-07-29T00:37:49.782 DEBUG:boto:Path: /hiiqos-59/obj33 2023-07-29T00:37:49.782 DEBUG:boto:Data: 2023-07-29T00:37:49.782 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.783 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:49.783 DEBUG:boto:Port: 8000 2023-07-29T00:37:49.783 DEBUG:boto:Params: {} 2023-07-29T00:37:49.784 DEBUG:boto:Token: None 2023-07-29T00:37:49.784 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj33 2023-07-29T00:37:49.785 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2V075UR1ULMSZtV3TI6F1W+dqL8= 2023-07-29T00:37:49.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2V075UR1ULMSZtV3TI6F1W+dqL8=', 'Content-Length': '0'} 2023-07-29T00:37:49.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a01078db329db1ad-0064c45f5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.788 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:49.789 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:49.789 DEBUG:boto:path=/hiiqos-59/?prefix=obj33 2023-07-29T00:37:49.789 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj33 2023-07-29T00:37:49.790 DEBUG:boto:Method: GET 2023-07-29T00:37:49.790 DEBUG:boto:Path: /hiiqos-59/?prefix=obj33 2023-07-29T00:37:49.790 DEBUG:boto:Data: 2023-07-29T00:37:49.791 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.791 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:49.791 DEBUG:boto:Port: 8001 2023-07-29T00:37:49.792 DEBUG:boto:Params: {} 2023-07-29T00:37:49.792 DEBUG:boto:Token: None 2023-07-29T00:37:49.793 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/ 2023-07-29T00:37:49.793 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk= 2023-07-29T00:37:49.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk=', 'Content-Length': '0'} 2023-07-29T00:37:49.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004dbed4fda028a01e-0064c45f5d-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.796 DEBUG:boto:b'hiiqos-59obj331000falseobj332023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:49.797 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:49.798 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:49.798 DEBUG:boto:path=/hiiqos-59/?prefix=obj33 2023-07-29T00:37:49.798 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj33 2023-07-29T00:37:49.799 DEBUG:boto:Method: GET 2023-07-29T00:37:49.799 DEBUG:boto:Path: /hiiqos-59/?prefix=obj33 2023-07-29T00:37:49.799 DEBUG:boto:Data: 2023-07-29T00:37:49.800 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.800 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:49.801 DEBUG:boto:Port: 8000 2023-07-29T00:37:49.801 DEBUG:boto:Params: {} 2023-07-29T00:37:49.801 DEBUG:boto:Token: None 2023-07-29T00:37:49.802 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/ 2023-07-29T00:37:49.802 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk= 2023-07-29T00:37:49.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk=', 'Content-Length': '0'} 2023-07-29T00:37:49.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e92544b376f6155-0064c45f5d-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.807 DEBUG:boto:b'hiiqos-59obj331000falseobj332023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:49.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-07-29T00:37:49.808 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:37:49.809 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:37:49.809 DEBUG:boto:path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:37:49.809 DEBUG:boto:auth_path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:37:49.810 DEBUG:boto:Method: GET 2023-07-29T00:37:49.810 DEBUG:boto:Path: /hiiqos-59/obj34?versionId=null 2023-07-29T00:37:49.810 DEBUG:boto:Data: 2023-07-29T00:37:49.811 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.811 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:49.811 DEBUG:boto:Port: 8001 2023-07-29T00:37:49.812 DEBUG:boto:Params: {} 2023-07-29T00:37:49.812 DEBUG:boto:Token: None 2023-07-29T00:37:49.813 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj34?versionId=null 2023-07-29T00:37:49.813 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WXPlTBcnZ+gl9vH8lw/p5yNPndc= 2023-07-29T00:37:49.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WXPlTBcnZ+gl9vH8lw/p5yNPndc=', 'Content-Length': '0'} 2023-07-29T00:37:49.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091065e0c2df9b3c4-0064c45f5d-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.816 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:37:49.816 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:37:49.817 DEBUG:boto:path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:37:49.817 DEBUG:boto:auth_path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:37:49.817 DEBUG:boto:Method: GET 2023-07-29T00:37:49.818 DEBUG:boto:Path: /hiiqos-59/obj34?versionId=null 2023-07-29T00:37:49.818 DEBUG:boto:Data: 2023-07-29T00:37:49.819 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.819 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:49.819 DEBUG:boto:Port: 8000 2023-07-29T00:37:49.820 DEBUG:boto:Params: {} 2023-07-29T00:37:49.820 DEBUG:boto:Token: None 2023-07-29T00:37:49.820 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj34?versionId=null 2023-07-29T00:37:49.821 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WXPlTBcnZ+gl9vH8lw/p5yNPndc= 2023-07-29T00:37:49.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WXPlTBcnZ+gl9vH8lw/p5yNPndc=', 'Content-Length': '0'} 2023-07-29T00:37:49.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092c1d03f999912ee-0064c45f5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.824 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:37:49.824 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:37:49.825 DEBUG:boto:path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:37:49.825 DEBUG:boto:auth_path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:37:49.826 DEBUG:boto:Method: HEAD 2023-07-29T00:37:49.826 DEBUG:boto:Path: /hiiqos-59/obj34?versionId=null 2023-07-29T00:37:49.826 DEBUG:boto:Data: 2023-07-29T00:37:49.827 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.827 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:49.827 DEBUG:boto:Port: 8001 2023-07-29T00:37:49.828 DEBUG:boto:Params: {} 2023-07-29T00:37:49.828 DEBUG:boto:Token: None 2023-07-29T00:37:49.828 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj34?versionId=null 2023-07-29T00:37:49.829 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IjxWMQ+gXNKH4sDBIX/ny3Sx2QM= 2023-07-29T00:37:49.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IjxWMQ+gXNKH4sDBIX/ny3Sx2QM=', 'Content-Length': '0'} 2023-07-29T00:37:49.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062f2b40c4c4bdd34-0064c45f5d-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.831 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:37:49.832 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:37:49.832 DEBUG:boto:path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:37:49.832 DEBUG:boto:auth_path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:37:49.833 DEBUG:boto:Method: HEAD 2023-07-29T00:37:49.833 DEBUG:boto:Path: /hiiqos-59/obj34?versionId=null 2023-07-29T00:37:49.833 DEBUG:boto:Data: 2023-07-29T00:37:49.834 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.834 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:49.834 DEBUG:boto:Port: 8000 2023-07-29T00:37:49.835 DEBUG:boto:Params: {} 2023-07-29T00:37:49.835 DEBUG:boto:Token: None 2023-07-29T00:37:49.836 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj34?versionId=null 2023-07-29T00:37:49.836 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IjxWMQ+gXNKH4sDBIX/ny3Sx2QM= 2023-07-29T00:37:49.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IjxWMQ+gXNKH4sDBIX/ny3Sx2QM=', 'Content-Length': '0'} 2023-07-29T00:37:49.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1a0c5973d98b9bf-0064c45f5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-07-29T00:37:49.839 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:37:49.840 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:37:49.840 DEBUG:boto:path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:37:49.840 DEBUG:boto:auth_path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:37:49.841 DEBUG:boto:Method: GET 2023-07-29T00:37:49.841 DEBUG:boto:Path: /hiiqos-59/obj34?versionId=null 2023-07-29T00:37:49.841 DEBUG:boto:Data: 2023-07-29T00:37:49.842 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.842 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:49.842 DEBUG:boto:Port: 8001 2023-07-29T00:37:49.843 DEBUG:boto:Params: {} 2023-07-29T00:37:49.843 DEBUG:boto:Token: None 2023-07-29T00:37:49.844 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj34?versionId=null 2023-07-29T00:37:49.844 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WXPlTBcnZ+gl9vH8lw/p5yNPndc= 2023-07-29T00:37:49.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WXPlTBcnZ+gl9vH8lw/p5yNPndc=', 'Content-Length': '0'} 2023-07-29T00:37:49.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b66086961ce31d0-0064c45f5d-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.847 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:37:49.847 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:37:49.848 DEBUG:boto:path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:37:49.848 DEBUG:boto:auth_path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:37:49.848 DEBUG:boto:Method: GET 2023-07-29T00:37:49.849 DEBUG:boto:Path: /hiiqos-59/obj34?versionId=null 2023-07-29T00:37:49.849 DEBUG:boto:Data: 2023-07-29T00:37:49.849 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.850 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:49.850 DEBUG:boto:Port: 8000 2023-07-29T00:37:49.851 DEBUG:boto:Params: {} 2023-07-29T00:37:49.851 DEBUG:boto:Token: None 2023-07-29T00:37:49.851 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj34?versionId=null 2023-07-29T00:37:49.852 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WXPlTBcnZ+gl9vH8lw/p5yNPndc= 2023-07-29T00:37:49.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WXPlTBcnZ+gl9vH8lw/p5yNPndc=', 'Content-Length': '0'} 2023-07-29T00:37:49.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b4ffb0e52975b5b-0064c45f5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.855 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:49.855 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:49.856 DEBUG:boto:path=/hiiqos-59/?prefix=obj34 2023-07-29T00:37:49.856 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj34 2023-07-29T00:37:49.856 DEBUG:boto:Method: GET 2023-07-29T00:37:49.857 DEBUG:boto:Path: /hiiqos-59/?prefix=obj34 2023-07-29T00:37:49.857 DEBUG:boto:Data: 2023-07-29T00:37:49.857 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.858 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:49.858 DEBUG:boto:Port: 8001 2023-07-29T00:37:49.859 DEBUG:boto:Params: {} 2023-07-29T00:37:49.859 DEBUG:boto:Token: None 2023-07-29T00:37:49.859 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/ 2023-07-29T00:37:49.860 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk= 2023-07-29T00:37:49.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk=', 'Content-Length': '0'} 2023-07-29T00:37:49.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005a04101a196d65a-0064c45f5d-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.863 DEBUG:boto:b'hiiqos-59obj341000falseobj342023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:49.864 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:49.864 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:49.865 DEBUG:boto:path=/hiiqos-59/?prefix=obj34 2023-07-29T00:37:49.865 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj34 2023-07-29T00:37:49.865 DEBUG:boto:Method: GET 2023-07-29T00:37:49.866 DEBUG:boto:Path: /hiiqos-59/?prefix=obj34 2023-07-29T00:37:49.866 DEBUG:boto:Data: 2023-07-29T00:37:49.866 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.867 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:49.867 DEBUG:boto:Port: 8000 2023-07-29T00:37:49.867 DEBUG:boto:Params: {} 2023-07-29T00:37:49.868 DEBUG:boto:Token: None 2023-07-29T00:37:49.868 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/ 2023-07-29T00:37:49.869 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk= 2023-07-29T00:37:49.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk=', 'Content-Length': '0'} 2023-07-29T00:37:49.872 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d05ae701be7ff123-0064c45f5d-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.873 DEBUG:boto:b'hiiqos-59obj341000falseobj342023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:49.874 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:37:49.874 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:37:49.874 DEBUG:boto:Method: HEAD 2023-07-29T00:37:49.875 DEBUG:boto:Path: /hiiqos-59/obj34 2023-07-29T00:37:49.875 DEBUG:boto:Data: 2023-07-29T00:37:49.875 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.876 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:49.876 DEBUG:boto:Port: 8001 2023-07-29T00:37:49.876 DEBUG:boto:Params: {} 2023-07-29T00:37:49.877 DEBUG:boto:Token: None 2023-07-29T00:37:49.877 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj34 2023-07-29T00:37:49.878 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:W+ivgb1IQYxNH3rsNWiSmNIS4cM= 2023-07-29T00:37:49.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:W+ivgb1IQYxNH3rsNWiSmNIS4cM=', 'Content-Length': '0'} 2023-07-29T00:37:49.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4b6f4027bc32a7d-0064c45f5d-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.881 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:37:49.881 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:37:49.881 DEBUG:boto:Method: HEAD 2023-07-29T00:37:49.882 DEBUG:boto:Path: /hiiqos-59/obj34 2023-07-29T00:37:49.882 DEBUG:boto:Data: 2023-07-29T00:37:49.882 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.883 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:49.883 DEBUG:boto:Port: 8000 2023-07-29T00:37:49.883 DEBUG:boto:Params: {} 2023-07-29T00:37:49.884 DEBUG:boto:Token: None 2023-07-29T00:37:49.884 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj34 2023-07-29T00:37:49.884 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:W+ivgb1IQYxNH3rsNWiSmNIS4cM= 2023-07-29T00:37:49.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:W+ivgb1IQYxNH3rsNWiSmNIS4cM=', 'Content-Length': '0'} 2023-07-29T00:37:49.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000717504d22d4c9752-0064c45f5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-07-29T00:37:49.888 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:37:49.888 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:37:49.888 DEBUG:boto:Method: GET 2023-07-29T00:37:49.889 DEBUG:boto:Path: /hiiqos-59/obj34 2023-07-29T00:37:49.889 DEBUG:boto:Data: 2023-07-29T00:37:49.890 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.890 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:49.890 DEBUG:boto:Port: 8001 2023-07-29T00:37:49.891 DEBUG:boto:Params: {} 2023-07-29T00:37:49.891 DEBUG:boto:Token: None 2023-07-29T00:37:49.891 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj34 2023-07-29T00:37:49.892 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J3UNIf71qe3raECETwi8mEJ9JHs= 2023-07-29T00:37:49.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J3UNIf71qe3raECETwi8mEJ9JHs=', 'Content-Length': '0'} 2023-07-29T00:37:49.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000238063c7762c95e1-0064c45f5d-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.895 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:37:49.895 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:37:49.895 DEBUG:boto:Method: GET 2023-07-29T00:37:49.896 DEBUG:boto:Path: /hiiqos-59/obj34 2023-07-29T00:37:49.896 DEBUG:boto:Data: 2023-07-29T00:37:49.896 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.897 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:49.897 DEBUG:boto:Port: 8000 2023-07-29T00:37:49.898 DEBUG:boto:Params: {} 2023-07-29T00:37:49.898 DEBUG:boto:Token: None 2023-07-29T00:37:49.898 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj34 2023-07-29T00:37:49.899 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J3UNIf71qe3raECETwi8mEJ9JHs= 2023-07-29T00:37:49.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J3UNIf71qe3raECETwi8mEJ9JHs=', 'Content-Length': '0'} 2023-07-29T00:37:49.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be0962e63c555a3f-0064c45f5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.944 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:49.945 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:49.945 DEBUG:boto:path=/hiiqos-59/?prefix=obj34 2023-07-29T00:37:49.945 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj34 2023-07-29T00:37:49.946 DEBUG:boto:Method: GET 2023-07-29T00:37:49.946 DEBUG:boto:Path: /hiiqos-59/?prefix=obj34 2023-07-29T00:37:49.947 DEBUG:boto:Data: 2023-07-29T00:37:49.947 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.947 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:49.948 DEBUG:boto:Port: 8001 2023-07-29T00:37:49.948 DEBUG:boto:Params: {} 2023-07-29T00:37:49.948 DEBUG:boto:Token: None 2023-07-29T00:37:49.949 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/ 2023-07-29T00:37:49.949 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk= 2023-07-29T00:37:49.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk=', 'Content-Length': '0'} 2023-07-29T00:37:49.952 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f308cdabc1cd7e3-0064c45f5d-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.953 DEBUG:boto:b'hiiqos-59obj341000falseobj342023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:49.953 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:49.954 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:49.954 DEBUG:boto:path=/hiiqos-59/?prefix=obj34 2023-07-29T00:37:49.955 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj34 2023-07-29T00:37:49.955 DEBUG:boto:Method: GET 2023-07-29T00:37:49.955 DEBUG:boto:Path: /hiiqos-59/?prefix=obj34 2023-07-29T00:37:49.956 DEBUG:boto:Data: 2023-07-29T00:37:49.956 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.956 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:49.957 DEBUG:boto:Port: 8000 2023-07-29T00:37:49.957 DEBUG:boto:Params: {} 2023-07-29T00:37:49.958 DEBUG:boto:Token: None 2023-07-29T00:37:49.958 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/ 2023-07-29T00:37:49.958 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk= 2023-07-29T00:37:49.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gymLLk1gPpdt7cbijhTTa+SzHNk=', 'Content-Length': '0'} 2023-07-29T00:37:49.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089f1f7dd31ca15eb-0064c45f5d-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:49.963 DEBUG:boto:b'hiiqos-59obj341000falseobj342023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:49.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-07-29T00:37:49.965 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:37:49.965 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:37:49.965 DEBUG:boto:path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:37:49.966 DEBUG:boto:auth_path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:37:49.966 DEBUG:boto:Method: GET 2023-07-29T00:37:49.966 DEBUG:boto:Path: /hiiqos-59/obj38?versionId=null 2023-07-29T00:37:49.967 DEBUG:boto:Data: 2023-07-29T00:37:49.967 DEBUG:boto:Headers: {} 2023-07-29T00:37:49.967 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:49.968 DEBUG:boto:Port: 8001 2023-07-29T00:37:49.968 DEBUG:boto:Params: {} 2023-07-29T00:37:49.968 DEBUG:boto:Token: None 2023-07-29T00:37:49.969 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:49 GMT /hiiqos-59/obj38?versionId=null 2023-07-29T00:37:49.969 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UAcm8cq0wM13VAtynX5KkW62xHo= 2023-07-29T00:37:49.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UAcm8cq0wM13VAtynX5KkW62xHo=', 'Content-Length': '0'} 2023-07-29T00:37:49.972 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000746822fcf00353c7-0064c45f5d-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.012 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:37:50.013 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:37:50.013 DEBUG:boto:path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:37:50.014 DEBUG:boto:auth_path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:37:50.014 DEBUG:boto:Method: GET 2023-07-29T00:37:50.014 DEBUG:boto:Path: /hiiqos-59/obj38?versionId=null 2023-07-29T00:37:50.015 DEBUG:boto:Data: 2023-07-29T00:37:50.015 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.015 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:50.016 DEBUG:boto:Port: 8000 2023-07-29T00:37:50.016 DEBUG:boto:Params: {} 2023-07-29T00:37:50.017 DEBUG:boto:Token: None 2023-07-29T00:37:50.017 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj38?versionId=null 2023-07-29T00:37:50.018 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3oQa7IOYoLFeIVvnl8jPwLjAGVE= 2023-07-29T00:37:50.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3oQa7IOYoLFeIVvnl8jPwLjAGVE=', 'Content-Length': '0'} 2023-07-29T00:37:50.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c8a0a7f6346a647-0064c45f5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.021 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:37:50.021 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:37:50.022 DEBUG:boto:path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:37:50.022 DEBUG:boto:auth_path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:37:50.022 DEBUG:boto:Method: HEAD 2023-07-29T00:37:50.023 DEBUG:boto:Path: /hiiqos-59/obj38?versionId=null 2023-07-29T00:37:50.023 DEBUG:boto:Data: 2023-07-29T00:37:50.023 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.024 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:50.024 DEBUG:boto:Port: 8001 2023-07-29T00:37:50.024 DEBUG:boto:Params: {} 2023-07-29T00:37:50.025 DEBUG:boto:Token: None 2023-07-29T00:37:50.025 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj38?versionId=null 2023-07-29T00:37:50.025 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0ZePzFwysSrfBp0ERE4JhRXZHC0= 2023-07-29T00:37:50.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0ZePzFwysSrfBp0ERE4JhRXZHC0=', 'Content-Length': '0'} 2023-07-29T00:37:50.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de6d9eb963ec42c3-0064c45f5d-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.028 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:37:50.029 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:37:50.029 DEBUG:boto:path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:37:50.029 DEBUG:boto:auth_path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:37:50.030 DEBUG:boto:Method: HEAD 2023-07-29T00:37:50.030 DEBUG:boto:Path: /hiiqos-59/obj38?versionId=null 2023-07-29T00:37:50.030 DEBUG:boto:Data: 2023-07-29T00:37:50.031 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.031 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:50.031 DEBUG:boto:Port: 8000 2023-07-29T00:37:50.032 DEBUG:boto:Params: {} 2023-07-29T00:37:50.032 DEBUG:boto:Token: None 2023-07-29T00:37:50.032 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj38?versionId=null 2023-07-29T00:37:50.033 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0ZePzFwysSrfBp0ERE4JhRXZHC0= 2023-07-29T00:37:50.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0ZePzFwysSrfBp0ERE4JhRXZHC0=', 'Content-Length': '0'} 2023-07-29T00:37:50.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ec0a8ab9a023deb-0064c45f5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-07-29T00:37:50.036 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:37:50.037 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:37:50.037 DEBUG:boto:path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:37:50.037 DEBUG:boto:auth_path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:37:50.038 DEBUG:boto:Method: GET 2023-07-29T00:37:50.038 DEBUG:boto:Path: /hiiqos-59/obj38?versionId=null 2023-07-29T00:37:50.038 DEBUG:boto:Data: 2023-07-29T00:37:50.039 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.039 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:50.039 DEBUG:boto:Port: 8001 2023-07-29T00:37:50.040 DEBUG:boto:Params: {} 2023-07-29T00:37:50.040 DEBUG:boto:Token: None 2023-07-29T00:37:50.041 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj38?versionId=null 2023-07-29T00:37:50.041 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3oQa7IOYoLFeIVvnl8jPwLjAGVE= 2023-07-29T00:37:50.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3oQa7IOYoLFeIVvnl8jPwLjAGVE=', 'Content-Length': '0'} 2023-07-29T00:37:50.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015909ba9bb7554e9-0064c45f5d-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.044 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:37:50.045 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:37:50.045 DEBUG:boto:path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:37:50.045 DEBUG:boto:auth_path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:37:50.046 DEBUG:boto:Method: GET 2023-07-29T00:37:50.046 DEBUG:boto:Path: /hiiqos-59/obj38?versionId=null 2023-07-29T00:37:50.046 DEBUG:boto:Data: 2023-07-29T00:37:50.047 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.047 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:50.047 DEBUG:boto:Port: 8000 2023-07-29T00:37:50.048 DEBUG:boto:Params: {} 2023-07-29T00:37:50.048 DEBUG:boto:Token: None 2023-07-29T00:37:50.049 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj38?versionId=null 2023-07-29T00:37:50.049 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3oQa7IOYoLFeIVvnl8jPwLjAGVE= 2023-07-29T00:37:50.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3oQa7IOYoLFeIVvnl8jPwLjAGVE=', 'Content-Length': '0'} 2023-07-29T00:37:50.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070267138170e1d77-0064c45f5d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.052 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:50.052 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:50.053 DEBUG:boto:path=/hiiqos-59/?prefix=obj38 2023-07-29T00:37:50.053 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj38 2023-07-29T00:37:50.053 DEBUG:boto:Method: GET 2023-07-29T00:37:50.054 DEBUG:boto:Path: /hiiqos-59/?prefix=obj38 2023-07-29T00:37:50.054 DEBUG:boto:Data: 2023-07-29T00:37:50.055 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.055 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:50.055 DEBUG:boto:Port: 8001 2023-07-29T00:37:50.056 DEBUG:boto:Params: {} 2023-07-29T00:37:50.056 DEBUG:boto:Token: None 2023-07-29T00:37:50.056 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/ 2023-07-29T00:37:50.057 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU= 2023-07-29T00:37:50.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU=', 'Content-Length': '0'} 2023-07-29T00:37:50.060 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4dd7d145f0b1042-0064c45f5d-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.060 DEBUG:boto:b'hiiqos-59obj381000falseobj382023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:50.061 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:50.062 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:50.062 DEBUG:boto:path=/hiiqos-59/?prefix=obj38 2023-07-29T00:37:50.062 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj38 2023-07-29T00:37:50.063 DEBUG:boto:Method: GET 2023-07-29T00:37:50.063 DEBUG:boto:Path: /hiiqos-59/?prefix=obj38 2023-07-29T00:37:50.063 DEBUG:boto:Data: 2023-07-29T00:37:50.064 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.064 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:50.064 DEBUG:boto:Port: 8000 2023-07-29T00:37:50.065 DEBUG:boto:Params: {} 2023-07-29T00:37:50.065 DEBUG:boto:Token: None 2023-07-29T00:37:50.066 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/ 2023-07-29T00:37:50.066 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU= 2023-07-29T00:37:50.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU=', 'Content-Length': '0'} 2023-07-29T00:37:50.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de2ab87894c2b25a-0064c45f5e-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.071 DEBUG:boto:b'hiiqos-59obj381000falseobj382023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:50.071 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:37:50.072 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:37:50.072 DEBUG:boto:Method: HEAD 2023-07-29T00:37:50.072 DEBUG:boto:Path: /hiiqos-59/obj38 2023-07-29T00:37:50.073 DEBUG:boto:Data: 2023-07-29T00:37:50.073 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.073 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:50.074 DEBUG:boto:Port: 8001 2023-07-29T00:37:50.074 DEBUG:boto:Params: {} 2023-07-29T00:37:50.074 DEBUG:boto:Token: None 2023-07-29T00:37:50.075 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj38 2023-07-29T00:37:50.075 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VUOWEa1d0sRkJbiJBbbPtxfaMU8= 2023-07-29T00:37:50.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VUOWEa1d0sRkJbiJBbbPtxfaMU8=', 'Content-Length': '0'} 2023-07-29T00:37:50.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de796a17706625fb-0064c45f5e-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.078 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:37:50.078 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:37:50.079 DEBUG:boto:Method: HEAD 2023-07-29T00:37:50.079 DEBUG:boto:Path: /hiiqos-59/obj38 2023-07-29T00:37:50.079 DEBUG:boto:Data: 2023-07-29T00:37:50.080 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.080 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:50.080 DEBUG:boto:Port: 8000 2023-07-29T00:37:50.081 DEBUG:boto:Params: {} 2023-07-29T00:37:50.081 DEBUG:boto:Token: None 2023-07-29T00:37:50.081 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj38 2023-07-29T00:37:50.082 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VUOWEa1d0sRkJbiJBbbPtxfaMU8= 2023-07-29T00:37:50.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VUOWEa1d0sRkJbiJBbbPtxfaMU8=', 'Content-Length': '0'} 2023-07-29T00:37:50.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd11f0f0957e6bea-0064c45f5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-07-29T00:37:50.085 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:37:50.086 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:37:50.086 DEBUG:boto:Method: GET 2023-07-29T00:37:50.086 DEBUG:boto:Path: /hiiqos-59/obj38 2023-07-29T00:37:50.087 DEBUG:boto:Data: 2023-07-29T00:37:50.087 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.088 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:50.088 DEBUG:boto:Port: 8001 2023-07-29T00:37:50.088 DEBUG:boto:Params: {} 2023-07-29T00:37:50.089 DEBUG:boto:Token: None 2023-07-29T00:37:50.089 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj38 2023-07-29T00:37:50.089 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xRdATkezFNFs4rQL0Id0M6D1qnU= 2023-07-29T00:37:50.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xRdATkezFNFs4rQL0Id0M6D1qnU=', 'Content-Length': '0'} 2023-07-29T00:37:50.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de0656696c106b39-0064c45f5e-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.092 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:37:50.093 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:37:50.093 DEBUG:boto:Method: GET 2023-07-29T00:37:50.093 DEBUG:boto:Path: /hiiqos-59/obj38 2023-07-29T00:37:50.094 DEBUG:boto:Data: 2023-07-29T00:37:50.094 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.095 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:50.095 DEBUG:boto:Port: 8000 2023-07-29T00:37:50.095 DEBUG:boto:Params: {} 2023-07-29T00:37:50.096 DEBUG:boto:Token: None 2023-07-29T00:37:50.096 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj38 2023-07-29T00:37:50.096 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xRdATkezFNFs4rQL0Id0M6D1qnU= 2023-07-29T00:37:50.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xRdATkezFNFs4rQL0Id0M6D1qnU=', 'Content-Length': '0'} 2023-07-29T00:37:50.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000606ff9b22d25ee9d-0064c45f5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.140 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:50.141 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:50.141 DEBUG:boto:path=/hiiqos-59/?prefix=obj38 2023-07-29T00:37:50.142 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj38 2023-07-29T00:37:50.142 DEBUG:boto:Method: GET 2023-07-29T00:37:50.142 DEBUG:boto:Path: /hiiqos-59/?prefix=obj38 2023-07-29T00:37:50.143 DEBUG:boto:Data: 2023-07-29T00:37:50.143 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.143 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:50.144 DEBUG:boto:Port: 8001 2023-07-29T00:37:50.144 DEBUG:boto:Params: {} 2023-07-29T00:37:50.144 DEBUG:boto:Token: None 2023-07-29T00:37:50.145 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/ 2023-07-29T00:37:50.145 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU= 2023-07-29T00:37:50.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU=', 'Content-Length': '0'} 2023-07-29T00:37:50.148 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076eccfeb0edb1d28-0064c45f5e-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.149 DEBUG:boto:b'hiiqos-59obj381000falseobj382023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:50.149 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:50.150 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:50.150 DEBUG:boto:path=/hiiqos-59/?prefix=obj38 2023-07-29T00:37:50.151 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj38 2023-07-29T00:37:50.151 DEBUG:boto:Method: GET 2023-07-29T00:37:50.151 DEBUG:boto:Path: /hiiqos-59/?prefix=obj38 2023-07-29T00:37:50.152 DEBUG:boto:Data: 2023-07-29T00:37:50.152 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.152 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:50.153 DEBUG:boto:Port: 8000 2023-07-29T00:37:50.153 DEBUG:boto:Params: {} 2023-07-29T00:37:50.153 DEBUG:boto:Token: None 2023-07-29T00:37:50.154 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/ 2023-07-29T00:37:50.154 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU= 2023-07-29T00:37:50.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU=', 'Content-Length': '0'} 2023-07-29T00:37:50.159 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e942f18aaef772ef-0064c45f5e-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.160 DEBUG:boto:b'hiiqos-59obj381000falseobj382023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:50.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-07-29T00:37:50.161 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:37:50.161 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:37:50.161 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:37:50.162 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:37:50.162 DEBUG:boto:Method: GET 2023-07-29T00:37:50.162 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:37:50.163 DEBUG:boto:Data: 2023-07-29T00:37:50.163 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.164 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:50.164 DEBUG:boto:Port: 8001 2023-07-29T00:37:50.165 DEBUG:boto:Params: {} 2023-07-29T00:37:50.165 DEBUG:boto:Token: None 2023-07-29T00:37:50.166 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:37:50.166 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vBvd/LzeveaYIn6GJqZGWAR3kgw= 2023-07-29T00:37:50.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vBvd/LzeveaYIn6GJqZGWAR3kgw=', 'Content-Length': '0'} 2023-07-29T00:37:50.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001688a1866e26dd24-0064c45f5e-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.169 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:37:50.170 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:37:50.170 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:37:50.171 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:37:50.171 DEBUG:boto:Method: GET 2023-07-29T00:37:50.171 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:37:50.172 DEBUG:boto:Data: 2023-07-29T00:37:50.172 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.172 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:50.173 DEBUG:boto:Port: 8000 2023-07-29T00:37:50.173 DEBUG:boto:Params: {} 2023-07-29T00:37:50.173 DEBUG:boto:Token: None 2023-07-29T00:37:50.174 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:37:50.174 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vBvd/LzeveaYIn6GJqZGWAR3kgw= 2023-07-29T00:37:50.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vBvd/LzeveaYIn6GJqZGWAR3kgw=', 'Content-Length': '0'} 2023-07-29T00:37:50.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000604fe27337f9f946-0064c45f5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.178 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:37:50.178 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:37:50.178 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:37:50.179 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:37:50.179 DEBUG:boto:Method: HEAD 2023-07-29T00:37:50.179 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:37:50.180 DEBUG:boto:Data: 2023-07-29T00:37:50.180 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.180 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:50.181 DEBUG:boto:Port: 8001 2023-07-29T00:37:50.181 DEBUG:boto:Params: {} 2023-07-29T00:37:50.181 DEBUG:boto:Token: None 2023-07-29T00:37:50.182 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:37:50.182 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EuVAL8/o8P08p6Cz3K4No8zFfL4= 2023-07-29T00:37:50.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EuVAL8/o8P08p6Cz3K4No8zFfL4=', 'Content-Length': '0'} 2023-07-29T00:37:50.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000817ed613d6382c81-0064c45f5e-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.185 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:37:50.186 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:37:50.186 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:37:50.186 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:37:50.186 DEBUG:boto:Method: HEAD 2023-07-29T00:37:50.187 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:37:50.187 DEBUG:boto:Data: 2023-07-29T00:37:50.187 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.188 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:50.188 DEBUG:boto:Port: 8000 2023-07-29T00:37:50.188 DEBUG:boto:Params: {} 2023-07-29T00:37:50.189 DEBUG:boto:Token: None 2023-07-29T00:37:50.189 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:37:50.190 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EuVAL8/o8P08p6Cz3K4No8zFfL4= 2023-07-29T00:37:50.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EuVAL8/o8P08p6Cz3K4No8zFfL4=', 'Content-Length': '0'} 2023-07-29T00:37:50.192 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000346c0c81f1a2dca5-0064c45f5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-07-29T00:37:50.193 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:37:50.193 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:37:50.194 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:37:50.194 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:37:50.195 DEBUG:boto:Method: GET 2023-07-29T00:37:50.195 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:37:50.195 DEBUG:boto:Data: 2023-07-29T00:37:50.196 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.196 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:50.196 DEBUG:boto:Port: 8001 2023-07-29T00:37:50.197 DEBUG:boto:Params: {} 2023-07-29T00:37:50.197 DEBUG:boto:Token: None 2023-07-29T00:37:50.198 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:37:50.198 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vBvd/LzeveaYIn6GJqZGWAR3kgw= 2023-07-29T00:37:50.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vBvd/LzeveaYIn6GJqZGWAR3kgw=', 'Content-Length': '0'} 2023-07-29T00:37:50.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048f7f343f9296f19-0064c45f5e-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.244 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:37:50.245 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:37:50.245 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:37:50.246 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:37:50.246 DEBUG:boto:Method: GET 2023-07-29T00:37:50.247 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:37:50.247 DEBUG:boto:Data: 2023-07-29T00:37:50.247 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.248 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:50.248 DEBUG:boto:Port: 8000 2023-07-29T00:37:50.248 DEBUG:boto:Params: {} 2023-07-29T00:37:50.249 DEBUG:boto:Token: None 2023-07-29T00:37:50.249 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:37:50.250 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vBvd/LzeveaYIn6GJqZGWAR3kgw= 2023-07-29T00:37:50.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vBvd/LzeveaYIn6GJqZGWAR3kgw=', 'Content-Length': '0'} 2023-07-29T00:37:50.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000712ed8cab07c065f-0064c45f5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.253 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:50.253 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:50.254 DEBUG:boto:path=/hiiqos-59/?prefix=obj44 2023-07-29T00:37:50.254 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj44 2023-07-29T00:37:50.254 DEBUG:boto:Method: GET 2023-07-29T00:37:50.255 DEBUG:boto:Path: /hiiqos-59/?prefix=obj44 2023-07-29T00:37:50.255 DEBUG:boto:Data: 2023-07-29T00:37:50.255 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.256 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:50.256 DEBUG:boto:Port: 8001 2023-07-29T00:37:50.256 DEBUG:boto:Params: {} 2023-07-29T00:37:50.257 DEBUG:boto:Token: None 2023-07-29T00:37:50.257 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/ 2023-07-29T00:37:50.258 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU= 2023-07-29T00:37:50.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU=', 'Content-Length': '0'} 2023-07-29T00:37:50.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c185470623e1f0b9-0064c45f5e-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.261 DEBUG:boto:b'hiiqos-59obj441000falseobj442023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:50.262 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:50.262 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:50.262 DEBUG:boto:path=/hiiqos-59/?prefix=obj44 2023-07-29T00:37:50.263 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj44 2023-07-29T00:37:50.263 DEBUG:boto:Method: GET 2023-07-29T00:37:50.264 DEBUG:boto:Path: /hiiqos-59/?prefix=obj44 2023-07-29T00:37:50.264 DEBUG:boto:Data: 2023-07-29T00:37:50.264 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.265 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:50.265 DEBUG:boto:Port: 8000 2023-07-29T00:37:50.265 DEBUG:boto:Params: {} 2023-07-29T00:37:50.266 DEBUG:boto:Token: None 2023-07-29T00:37:50.266 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/ 2023-07-29T00:37:50.267 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU= 2023-07-29T00:37:50.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU=', 'Content-Length': '0'} 2023-07-29T00:37:50.271 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae8ddc110ca11026-0064c45f5e-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.272 DEBUG:boto:b'hiiqos-59obj441000falseobj442023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:50.272 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:37:50.273 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:37:50.273 DEBUG:boto:Method: HEAD 2023-07-29T00:37:50.273 DEBUG:boto:Path: /hiiqos-59/obj44 2023-07-29T00:37:50.274 DEBUG:boto:Data: 2023-07-29T00:37:50.274 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.274 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:50.275 DEBUG:boto:Port: 8001 2023-07-29T00:37:50.275 DEBUG:boto:Params: {} 2023-07-29T00:37:50.275 DEBUG:boto:Token: None 2023-07-29T00:37:50.276 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj44 2023-07-29T00:37:50.276 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8vbM5VQRKFZ5t9ph32wLEPIjthI= 2023-07-29T00:37:50.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8vbM5VQRKFZ5t9ph32wLEPIjthI=', 'Content-Length': '0'} 2023-07-29T00:37:50.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000620e75293e2385ac-0064c45f5e-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.279 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:37:50.279 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:37:50.279 DEBUG:boto:Method: HEAD 2023-07-29T00:37:50.280 DEBUG:boto:Path: /hiiqos-59/obj44 2023-07-29T00:37:50.280 DEBUG:boto:Data: 2023-07-29T00:37:50.281 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.281 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:50.281 DEBUG:boto:Port: 8000 2023-07-29T00:37:50.281 DEBUG:boto:Params: {} 2023-07-29T00:37:50.282 DEBUG:boto:Token: None 2023-07-29T00:37:50.282 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj44 2023-07-29T00:37:50.283 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8vbM5VQRKFZ5t9ph32wLEPIjthI= 2023-07-29T00:37:50.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8vbM5VQRKFZ5t9ph32wLEPIjthI=', 'Content-Length': '0'} 2023-07-29T00:37:50.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000141e00b6ab09ee89-0064c45f5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-07-29T00:37:50.286 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:37:50.286 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:37:50.286 DEBUG:boto:Method: GET 2023-07-29T00:37:50.287 DEBUG:boto:Path: /hiiqos-59/obj44 2023-07-29T00:37:50.287 DEBUG:boto:Data: 2023-07-29T00:37:50.288 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.288 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:50.289 DEBUG:boto:Port: 8001 2023-07-29T00:37:50.289 DEBUG:boto:Params: {} 2023-07-29T00:37:50.289 DEBUG:boto:Token: None 2023-07-29T00:37:50.290 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj44 2023-07-29T00:37:50.290 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JNJ8fzDsQIaShtXu+zHX9NgxVu4= 2023-07-29T00:37:50.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JNJ8fzDsQIaShtXu+zHX9NgxVu4=', 'Content-Length': '0'} 2023-07-29T00:37:50.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1eea8a18f4a26e2-0064c45f5e-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.293 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:37:50.293 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:37:50.294 DEBUG:boto:Method: GET 2023-07-29T00:37:50.294 DEBUG:boto:Path: /hiiqos-59/obj44 2023-07-29T00:37:50.294 DEBUG:boto:Data: 2023-07-29T00:37:50.295 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.295 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:50.295 DEBUG:boto:Port: 8000 2023-07-29T00:37:50.296 DEBUG:boto:Params: {} 2023-07-29T00:37:50.296 DEBUG:boto:Token: None 2023-07-29T00:37:50.297 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj44 2023-07-29T00:37:50.297 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JNJ8fzDsQIaShtXu+zHX9NgxVu4= 2023-07-29T00:37:50.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JNJ8fzDsQIaShtXu+zHX9NgxVu4=', 'Content-Length': '0'} 2023-07-29T00:37:50.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ee2ce08bcfeed67-0064c45f5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.340 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:50.341 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:50.341 DEBUG:boto:path=/hiiqos-59/?prefix=obj44 2023-07-29T00:37:50.341 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj44 2023-07-29T00:37:50.342 DEBUG:boto:Method: GET 2023-07-29T00:37:50.342 DEBUG:boto:Path: /hiiqos-59/?prefix=obj44 2023-07-29T00:37:50.343 DEBUG:boto:Data: 2023-07-29T00:37:50.343 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.344 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:50.344 DEBUG:boto:Port: 8001 2023-07-29T00:37:50.344 DEBUG:boto:Params: {} 2023-07-29T00:37:50.345 DEBUG:boto:Token: None 2023-07-29T00:37:50.345 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/ 2023-07-29T00:37:50.346 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU= 2023-07-29T00:37:50.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU=', 'Content-Length': '0'} 2023-07-29T00:37:50.348 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db91d311972caaea-0064c45f5e-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.349 DEBUG:boto:b'hiiqos-59obj441000falseobj442023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:50.350 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:50.350 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:50.350 DEBUG:boto:path=/hiiqos-59/?prefix=obj44 2023-07-29T00:37:50.351 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj44 2023-07-29T00:37:50.351 DEBUG:boto:Method: GET 2023-07-29T00:37:50.351 DEBUG:boto:Path: /hiiqos-59/?prefix=obj44 2023-07-29T00:37:50.352 DEBUG:boto:Data: 2023-07-29T00:37:50.352 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.352 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:50.353 DEBUG:boto:Port: 8000 2023-07-29T00:37:50.353 DEBUG:boto:Params: {} 2023-07-29T00:37:50.354 DEBUG:boto:Token: None 2023-07-29T00:37:50.354 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/ 2023-07-29T00:37:50.354 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU= 2023-07-29T00:37:50.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU=', 'Content-Length': '0'} 2023-07-29T00:37:50.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e93d2c6d396e5fe2-0064c45f5e-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.359 DEBUG:boto:b'hiiqos-59obj441000falseobj442023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:50.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-07-29T00:37:50.360 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:37:50.360 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:37:50.361 DEBUG:boto:path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:37:50.361 DEBUG:boto:auth_path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:37:50.362 DEBUG:boto:Method: GET 2023-07-29T00:37:50.362 DEBUG:boto:Path: /hiiqos-59/obj46?versionId=null 2023-07-29T00:37:50.363 DEBUG:boto:Data: 2023-07-29T00:37:50.363 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.363 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:50.364 DEBUG:boto:Port: 8001 2023-07-29T00:37:50.364 DEBUG:boto:Params: {} 2023-07-29T00:37:50.364 DEBUG:boto:Token: None 2023-07-29T00:37:50.365 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj46?versionId=null 2023-07-29T00:37:50.365 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:z2WXDufOLLyT6OtZm/tbQDtkjXU= 2023-07-29T00:37:50.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:z2WXDufOLLyT6OtZm/tbQDtkjXU=', 'Content-Length': '0'} 2023-07-29T00:37:50.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001da084d734610cd5-0064c45f5e-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.408 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:37:50.409 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:37:50.409 DEBUG:boto:path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:37:50.409 DEBUG:boto:auth_path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:37:50.410 DEBUG:boto:Method: GET 2023-07-29T00:37:50.410 DEBUG:boto:Path: /hiiqos-59/obj46?versionId=null 2023-07-29T00:37:50.411 DEBUG:boto:Data: 2023-07-29T00:37:50.411 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.411 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:50.412 DEBUG:boto:Port: 8000 2023-07-29T00:37:50.412 DEBUG:boto:Params: {} 2023-07-29T00:37:50.412 DEBUG:boto:Token: None 2023-07-29T00:37:50.413 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj46?versionId=null 2023-07-29T00:37:50.413 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:z2WXDufOLLyT6OtZm/tbQDtkjXU= 2023-07-29T00:37:50.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:z2WXDufOLLyT6OtZm/tbQDtkjXU=', 'Content-Length': '0'} 2023-07-29T00:37:50.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b794a5d221ecc5d-0064c45f5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.416 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:37:50.417 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:37:50.417 DEBUG:boto:path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:37:50.418 DEBUG:boto:auth_path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:37:50.418 DEBUG:boto:Method: HEAD 2023-07-29T00:37:50.419 DEBUG:boto:Path: /hiiqos-59/obj46?versionId=null 2023-07-29T00:37:50.419 DEBUG:boto:Data: 2023-07-29T00:37:50.419 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.420 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:50.420 DEBUG:boto:Port: 8001 2023-07-29T00:37:50.420 DEBUG:boto:Params: {} 2023-07-29T00:37:50.421 DEBUG:boto:Token: None 2023-07-29T00:37:50.421 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj46?versionId=null 2023-07-29T00:37:50.421 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0Elkhfjpl/tGfwu27/S3fenvFP0= 2023-07-29T00:37:50.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0Elkhfjpl/tGfwu27/S3fenvFP0=', 'Content-Length': '0'} 2023-07-29T00:37:50.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095d3310a6126f1c2-0064c45f5e-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.424 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:37:50.424 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:37:50.425 DEBUG:boto:path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:37:50.425 DEBUG:boto:auth_path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:37:50.425 DEBUG:boto:Method: HEAD 2023-07-29T00:37:50.426 DEBUG:boto:Path: /hiiqos-59/obj46?versionId=null 2023-07-29T00:37:50.426 DEBUG:boto:Data: 2023-07-29T00:37:50.426 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.427 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:50.427 DEBUG:boto:Port: 8000 2023-07-29T00:37:50.427 DEBUG:boto:Params: {} 2023-07-29T00:37:50.428 DEBUG:boto:Token: None 2023-07-29T00:37:50.428 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj46?versionId=null 2023-07-29T00:37:50.429 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0Elkhfjpl/tGfwu27/S3fenvFP0= 2023-07-29T00:37:50.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0Elkhfjpl/tGfwu27/S3fenvFP0=', 'Content-Length': '0'} 2023-07-29T00:37:50.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009715b672c8a5fcc2-0064c45f5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-07-29T00:37:50.432 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:37:50.432 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:37:50.432 DEBUG:boto:path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:37:50.433 DEBUG:boto:auth_path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:37:50.433 DEBUG:boto:Method: GET 2023-07-29T00:37:50.433 DEBUG:boto:Path: /hiiqos-59/obj46?versionId=null 2023-07-29T00:37:50.434 DEBUG:boto:Data: 2023-07-29T00:37:50.434 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.435 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:50.435 DEBUG:boto:Port: 8001 2023-07-29T00:37:50.435 DEBUG:boto:Params: {} 2023-07-29T00:37:50.436 DEBUG:boto:Token: None 2023-07-29T00:37:50.436 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj46?versionId=null 2023-07-29T00:37:50.437 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:z2WXDufOLLyT6OtZm/tbQDtkjXU= 2023-07-29T00:37:50.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:z2WXDufOLLyT6OtZm/tbQDtkjXU=', 'Content-Length': '0'} 2023-07-29T00:37:50.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a69232b4f637ff5e-0064c45f5e-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.439 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:37:50.440 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:37:50.440 DEBUG:boto:path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:37:50.441 DEBUG:boto:auth_path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:37:50.441 DEBUG:boto:Method: GET 2023-07-29T00:37:50.441 DEBUG:boto:Path: /hiiqos-59/obj46?versionId=null 2023-07-29T00:37:50.442 DEBUG:boto:Data: 2023-07-29T00:37:50.442 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.442 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:50.443 DEBUG:boto:Port: 8000 2023-07-29T00:37:50.443 DEBUG:boto:Params: {} 2023-07-29T00:37:50.443 DEBUG:boto:Token: None 2023-07-29T00:37:50.444 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj46?versionId=null 2023-07-29T00:37:50.444 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:z2WXDufOLLyT6OtZm/tbQDtkjXU= 2023-07-29T00:37:50.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:z2WXDufOLLyT6OtZm/tbQDtkjXU=', 'Content-Length': '0'} 2023-07-29T00:37:50.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000148517bc899ea8af-0064c45f5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.447 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:50.448 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:50.448 DEBUG:boto:path=/hiiqos-59/?prefix=obj46 2023-07-29T00:37:50.448 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj46 2023-07-29T00:37:50.449 DEBUG:boto:Method: GET 2023-07-29T00:37:50.449 DEBUG:boto:Path: /hiiqos-59/?prefix=obj46 2023-07-29T00:37:50.450 DEBUG:boto:Data: 2023-07-29T00:37:50.450 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.451 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:50.451 DEBUG:boto:Port: 8001 2023-07-29T00:37:50.451 DEBUG:boto:Params: {} 2023-07-29T00:37:50.452 DEBUG:boto:Token: None 2023-07-29T00:37:50.452 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/ 2023-07-29T00:37:50.453 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU= 2023-07-29T00:37:50.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU=', 'Content-Length': '0'} 2023-07-29T00:37:50.455 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003eb4c01e9e902994-0064c45f5e-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.456 DEBUG:boto:b'hiiqos-59obj461000falseobj462023-07-29T00:33:06.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:50.456 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:50.457 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:50.457 DEBUG:boto:path=/hiiqos-59/?prefix=obj46 2023-07-29T00:37:50.458 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj46 2023-07-29T00:37:50.458 DEBUG:boto:Method: GET 2023-07-29T00:37:50.458 DEBUG:boto:Path: /hiiqos-59/?prefix=obj46 2023-07-29T00:37:50.459 DEBUG:boto:Data: 2023-07-29T00:37:50.459 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.459 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:50.460 DEBUG:boto:Port: 8000 2023-07-29T00:37:50.460 DEBUG:boto:Params: {} 2023-07-29T00:37:50.460 DEBUG:boto:Token: None 2023-07-29T00:37:50.461 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/ 2023-07-29T00:37:50.461 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU= 2023-07-29T00:37:50.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU=', 'Content-Length': '0'} 2023-07-29T00:37:50.465 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c43a63a599e1d81b-0064c45f5e-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.466 DEBUG:boto:b'hiiqos-59obj461000falseobj462023-07-29T00:33:06.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:50.467 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:37:50.467 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:37:50.467 DEBUG:boto:Method: HEAD 2023-07-29T00:37:50.468 DEBUG:boto:Path: /hiiqos-59/obj46 2023-07-29T00:37:50.468 DEBUG:boto:Data: 2023-07-29T00:37:50.468 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.469 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:50.469 DEBUG:boto:Port: 8001 2023-07-29T00:37:50.469 DEBUG:boto:Params: {} 2023-07-29T00:37:50.470 DEBUG:boto:Token: None 2023-07-29T00:37:50.470 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj46 2023-07-29T00:37:50.470 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mdXCHl53eiDUO+SJuaEtciTTpbg= 2023-07-29T00:37:50.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mdXCHl53eiDUO+SJuaEtciTTpbg=', 'Content-Length': '0'} 2023-07-29T00:37:50.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0ae9f736e15fae6-0064c45f5e-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.473 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:37:50.473 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:37:50.474 DEBUG:boto:Method: HEAD 2023-07-29T00:37:50.474 DEBUG:boto:Path: /hiiqos-59/obj46 2023-07-29T00:37:50.475 DEBUG:boto:Data: 2023-07-29T00:37:50.475 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.475 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:50.475 DEBUG:boto:Port: 8000 2023-07-29T00:37:50.476 DEBUG:boto:Params: {} 2023-07-29T00:37:50.476 DEBUG:boto:Token: None 2023-07-29T00:37:50.477 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj46 2023-07-29T00:37:50.477 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mdXCHl53eiDUO+SJuaEtciTTpbg= 2023-07-29T00:37:50.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mdXCHl53eiDUO+SJuaEtciTTpbg=', 'Content-Length': '0'} 2023-07-29T00:37:50.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007065e6ba22879b4c-0064c45f5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.480 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-07-29T00:37:50.480 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:37:50.481 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:37:50.481 DEBUG:boto:Method: GET 2023-07-29T00:37:50.482 DEBUG:boto:Path: /hiiqos-59/obj46 2023-07-29T00:37:50.482 DEBUG:boto:Data: 2023-07-29T00:37:50.482 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.483 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:50.483 DEBUG:boto:Port: 8001 2023-07-29T00:37:50.483 DEBUG:boto:Params: {} 2023-07-29T00:37:50.484 DEBUG:boto:Token: None 2023-07-29T00:37:50.484 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj46 2023-07-29T00:37:50.485 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9xPhoBKY7RxWulvkrTgy3aHpfGs= 2023-07-29T00:37:50.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9xPhoBKY7RxWulvkrTgy3aHpfGs=', 'Content-Length': '0'} 2023-07-29T00:37:50.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000549ccafeb1348819-0064c45f5e-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.487 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:37:50.488 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:37:50.488 DEBUG:boto:Method: GET 2023-07-29T00:37:50.488 DEBUG:boto:Path: /hiiqos-59/obj46 2023-07-29T00:37:50.489 DEBUG:boto:Data: 2023-07-29T00:37:50.489 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.490 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:50.490 DEBUG:boto:Port: 8000 2023-07-29T00:37:50.490 DEBUG:boto:Params: {} 2023-07-29T00:37:50.491 DEBUG:boto:Token: None 2023-07-29T00:37:50.491 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj46 2023-07-29T00:37:50.491 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9xPhoBKY7RxWulvkrTgy3aHpfGs= 2023-07-29T00:37:50.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9xPhoBKY7RxWulvkrTgy3aHpfGs=', 'Content-Length': '0'} 2023-07-29T00:37:50.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da4f6564bbd96e9d-0064c45f5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.494 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:50.495 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:50.495 DEBUG:boto:path=/hiiqos-59/?prefix=obj46 2023-07-29T00:37:50.496 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj46 2023-07-29T00:37:50.496 DEBUG:boto:Method: GET 2023-07-29T00:37:50.496 DEBUG:boto:Path: /hiiqos-59/?prefix=obj46 2023-07-29T00:37:50.497 DEBUG:boto:Data: 2023-07-29T00:37:50.497 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.497 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:50.498 DEBUG:boto:Port: 8001 2023-07-29T00:37:50.498 DEBUG:boto:Params: {} 2023-07-29T00:37:50.498 DEBUG:boto:Token: None 2023-07-29T00:37:50.499 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/ 2023-07-29T00:37:50.499 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU= 2023-07-29T00:37:50.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU=', 'Content-Length': '0'} 2023-07-29T00:37:50.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044dc894441fec6e2-0064c45f5e-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.502 DEBUG:boto:b'hiiqos-59obj461000falseobj462023-07-29T00:33:06.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:50.503 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:50.503 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:50.504 DEBUG:boto:path=/hiiqos-59/?prefix=obj46 2023-07-29T00:37:50.504 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj46 2023-07-29T00:37:50.505 DEBUG:boto:Method: GET 2023-07-29T00:37:50.505 DEBUG:boto:Path: /hiiqos-59/?prefix=obj46 2023-07-29T00:37:50.505 DEBUG:boto:Data: 2023-07-29T00:37:50.506 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.506 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:50.506 DEBUG:boto:Port: 8000 2023-07-29T00:37:50.507 DEBUG:boto:Params: {} 2023-07-29T00:37:50.507 DEBUG:boto:Token: None 2023-07-29T00:37:50.508 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/ 2023-07-29T00:37:50.508 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU= 2023-07-29T00:37:50.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU=', 'Content-Length': '0'} 2023-07-29T00:37:50.512 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6e33550e1f94952-0064c45f5e-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.512 DEBUG:boto:b'hiiqos-59obj461000falseobj462023-07-29T00:33:06.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:50.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-07-29T00:37:50.513 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:37:50.514 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:37:50.514 DEBUG:boto:path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:37:50.514 DEBUG:boto:auth_path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:37:50.515 DEBUG:boto:Method: GET 2023-07-29T00:37:50.515 DEBUG:boto:Path: /hiiqos-59/obj51?versionId=null 2023-07-29T00:37:50.515 DEBUG:boto:Data: 2023-07-29T00:37:50.516 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.516 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:50.516 DEBUG:boto:Port: 8001 2023-07-29T00:37:50.517 DEBUG:boto:Params: {} 2023-07-29T00:37:50.517 DEBUG:boto:Token: None 2023-07-29T00:37:50.518 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj51?versionId=null 2023-07-29T00:37:50.518 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BIfBYJ0VRljvwE7oNfoRq1eAEJI= 2023-07-29T00:37:50.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BIfBYJ0VRljvwE7oNfoRq1eAEJI=', 'Content-Length': '0'} 2023-07-29T00:37:50.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e57d5147f6d51458-0064c45f5e-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.521 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:37:50.521 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:37:50.522 DEBUG:boto:path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:37:50.522 DEBUG:boto:auth_path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:37:50.522 DEBUG:boto:Method: GET 2023-07-29T00:37:50.523 DEBUG:boto:Path: /hiiqos-59/obj51?versionId=null 2023-07-29T00:37:50.523 DEBUG:boto:Data: 2023-07-29T00:37:50.523 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.524 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:50.524 DEBUG:boto:Port: 8000 2023-07-29T00:37:50.524 DEBUG:boto:Params: {} 2023-07-29T00:37:50.525 DEBUG:boto:Token: None 2023-07-29T00:37:50.525 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj51?versionId=null 2023-07-29T00:37:50.526 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BIfBYJ0VRljvwE7oNfoRq1eAEJI= 2023-07-29T00:37:50.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BIfBYJ0VRljvwE7oNfoRq1eAEJI=', 'Content-Length': '0'} 2023-07-29T00:37:50.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b51361cdc2a97679-0064c45f5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.529 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:37:50.529 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:37:50.530 DEBUG:boto:path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:37:50.530 DEBUG:boto:auth_path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:37:50.530 DEBUG:boto:Method: HEAD 2023-07-29T00:37:50.531 DEBUG:boto:Path: /hiiqos-59/obj51?versionId=null 2023-07-29T00:37:50.531 DEBUG:boto:Data: 2023-07-29T00:37:50.531 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.532 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:50.532 DEBUG:boto:Port: 8001 2023-07-29T00:37:50.532 DEBUG:boto:Params: {} 2023-07-29T00:37:50.533 DEBUG:boto:Token: None 2023-07-29T00:37:50.533 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj51?versionId=null 2023-07-29T00:37:50.534 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6VfJLxyNYnek9n7kiHDmtzrGApg= 2023-07-29T00:37:50.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6VfJLxyNYnek9n7kiHDmtzrGApg=', 'Content-Length': '0'} 2023-07-29T00:37:50.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000093ff0666480bf9-0064c45f5e-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.536 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:37:50.537 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:37:50.537 DEBUG:boto:path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:37:50.537 DEBUG:boto:auth_path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:37:50.538 DEBUG:boto:Method: HEAD 2023-07-29T00:37:50.538 DEBUG:boto:Path: /hiiqos-59/obj51?versionId=null 2023-07-29T00:37:50.538 DEBUG:boto:Data: 2023-07-29T00:37:50.539 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.539 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:50.539 DEBUG:boto:Port: 8000 2023-07-29T00:37:50.540 DEBUG:boto:Params: {} 2023-07-29T00:37:50.540 DEBUG:boto:Token: None 2023-07-29T00:37:50.540 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj51?versionId=null 2023-07-29T00:37:50.541 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6VfJLxyNYnek9n7kiHDmtzrGApg= 2023-07-29T00:37:50.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6VfJLxyNYnek9n7kiHDmtzrGApg=', 'Content-Length': '0'} 2023-07-29T00:37:50.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019b87c0b56ce19db-0064c45f5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-07-29T00:37:50.545 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:37:50.545 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:37:50.546 DEBUG:boto:path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:37:50.546 DEBUG:boto:auth_path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:37:50.547 DEBUG:boto:Method: GET 2023-07-29T00:37:50.547 DEBUG:boto:Path: /hiiqos-59/obj51?versionId=null 2023-07-29T00:37:50.548 DEBUG:boto:Data: 2023-07-29T00:37:50.548 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.549 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:50.550 DEBUG:boto:Port: 8001 2023-07-29T00:37:50.550 DEBUG:boto:Params: {} 2023-07-29T00:37:50.551 DEBUG:boto:Token: None 2023-07-29T00:37:50.552 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj51?versionId=null 2023-07-29T00:37:50.552 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BIfBYJ0VRljvwE7oNfoRq1eAEJI= 2023-07-29T00:37:50.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BIfBYJ0VRljvwE7oNfoRq1eAEJI=', 'Content-Length': '0'} 2023-07-29T00:37:50.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f19cb368871e1fac-0064c45f5e-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.596 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:37:50.597 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:37:50.597 DEBUG:boto:path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:37:50.598 DEBUG:boto:auth_path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:37:50.598 DEBUG:boto:Method: GET 2023-07-29T00:37:50.598 DEBUG:boto:Path: /hiiqos-59/obj51?versionId=null 2023-07-29T00:37:50.599 DEBUG:boto:Data: 2023-07-29T00:37:50.599 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.599 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:50.600 DEBUG:boto:Port: 8000 2023-07-29T00:37:50.600 DEBUG:boto:Params: {} 2023-07-29T00:37:50.601 DEBUG:boto:Token: None 2023-07-29T00:37:50.601 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj51?versionId=null 2023-07-29T00:37:50.601 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BIfBYJ0VRljvwE7oNfoRq1eAEJI= 2023-07-29T00:37:50.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BIfBYJ0VRljvwE7oNfoRq1eAEJI=', 'Content-Length': '0'} 2023-07-29T00:37:50.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaa55015ebc82b2e-0064c45f5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.605 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:50.605 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:50.605 DEBUG:boto:path=/hiiqos-59/?prefix=obj51 2023-07-29T00:37:50.606 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj51 2023-07-29T00:37:50.606 DEBUG:boto:Method: GET 2023-07-29T00:37:50.606 DEBUG:boto:Path: /hiiqos-59/?prefix=obj51 2023-07-29T00:37:50.607 DEBUG:boto:Data: 2023-07-29T00:37:50.607 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.608 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:50.608 DEBUG:boto:Port: 8001 2023-07-29T00:37:50.608 DEBUG:boto:Params: {} 2023-07-29T00:37:50.609 DEBUG:boto:Token: None 2023-07-29T00:37:50.609 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/ 2023-07-29T00:37:50.610 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU= 2023-07-29T00:37:50.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU=', 'Content-Length': '0'} 2023-07-29T00:37:50.613 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030241f02e650348b-0064c45f5e-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.613 DEBUG:boto:b'hiiqos-59obj511000falseobj512023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:50.614 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:50.615 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:50.615 DEBUG:boto:path=/hiiqos-59/?prefix=obj51 2023-07-29T00:37:50.615 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj51 2023-07-29T00:37:50.616 DEBUG:boto:Method: GET 2023-07-29T00:37:50.616 DEBUG:boto:Path: /hiiqos-59/?prefix=obj51 2023-07-29T00:37:50.616 DEBUG:boto:Data: 2023-07-29T00:37:50.617 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.617 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:50.618 DEBUG:boto:Port: 8000 2023-07-29T00:37:50.618 DEBUG:boto:Params: {} 2023-07-29T00:37:50.618 DEBUG:boto:Token: None 2023-07-29T00:37:50.619 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/ 2023-07-29T00:37:50.619 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU= 2023-07-29T00:37:50.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU=', 'Content-Length': '0'} 2023-07-29T00:37:50.623 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063ba0b1d8778fd90-0064c45f5e-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.624 DEBUG:boto:b'hiiqos-59obj511000falseobj512023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:50.625 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:37:50.625 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:37:50.625 DEBUG:boto:Method: HEAD 2023-07-29T00:37:50.626 DEBUG:boto:Path: /hiiqos-59/obj51 2023-07-29T00:37:50.626 DEBUG:boto:Data: 2023-07-29T00:37:50.626 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.627 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:50.627 DEBUG:boto:Port: 8001 2023-07-29T00:37:50.628 DEBUG:boto:Params: {} 2023-07-29T00:37:50.628 DEBUG:boto:Token: None 2023-07-29T00:37:50.629 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj51 2023-07-29T00:37:50.629 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kKLhd+u9E7LrddgEnxST82xy7Fw= 2023-07-29T00:37:50.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kKLhd+u9E7LrddgEnxST82xy7Fw=', 'Content-Length': '0'} 2023-07-29T00:37:50.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000add231374df46007-0064c45f5e-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.632 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:37:50.632 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:37:50.633 DEBUG:boto:Method: HEAD 2023-07-29T00:37:50.633 DEBUG:boto:Path: /hiiqos-59/obj51 2023-07-29T00:37:50.633 DEBUG:boto:Data: 2023-07-29T00:37:50.634 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.634 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:50.634 DEBUG:boto:Port: 8000 2023-07-29T00:37:50.635 DEBUG:boto:Params: {} 2023-07-29T00:37:50.635 DEBUG:boto:Token: None 2023-07-29T00:37:50.636 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj51 2023-07-29T00:37:50.636 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kKLhd+u9E7LrddgEnxST82xy7Fw= 2023-07-29T00:37:50.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kKLhd+u9E7LrddgEnxST82xy7Fw=', 'Content-Length': '0'} 2023-07-29T00:37:50.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da292c03a170356c-0064c45f5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-07-29T00:37:50.639 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:37:50.639 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:37:50.640 DEBUG:boto:Method: GET 2023-07-29T00:37:50.640 DEBUG:boto:Path: /hiiqos-59/obj51 2023-07-29T00:37:50.641 DEBUG:boto:Data: 2023-07-29T00:37:50.641 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.641 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:50.642 DEBUG:boto:Port: 8001 2023-07-29T00:37:50.642 DEBUG:boto:Params: {} 2023-07-29T00:37:50.642 DEBUG:boto:Token: None 2023-07-29T00:37:50.643 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj51 2023-07-29T00:37:50.643 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8AXYtwPcosFs0MaPclPe30OCDiQ= 2023-07-29T00:37:50.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8AXYtwPcosFs0MaPclPe30OCDiQ=', 'Content-Length': '0'} 2023-07-29T00:37:50.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a8651da46b68375-0064c45f5e-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.646 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:37:50.646 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:37:50.647 DEBUG:boto:Method: GET 2023-07-29T00:37:50.647 DEBUG:boto:Path: /hiiqos-59/obj51 2023-07-29T00:37:50.647 DEBUG:boto:Data: 2023-07-29T00:37:50.648 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.648 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:50.648 DEBUG:boto:Port: 8000 2023-07-29T00:37:50.649 DEBUG:boto:Params: {} 2023-07-29T00:37:50.649 DEBUG:boto:Token: None 2023-07-29T00:37:50.650 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj51 2023-07-29T00:37:50.650 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8AXYtwPcosFs0MaPclPe30OCDiQ= 2023-07-29T00:37:50.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8AXYtwPcosFs0MaPclPe30OCDiQ=', 'Content-Length': '0'} 2023-07-29T00:37:50.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028e6ee3e7eab86d1-0064c45f5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.653 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:50.654 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:50.654 DEBUG:boto:path=/hiiqos-59/?prefix=obj51 2023-07-29T00:37:50.654 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj51 2023-07-29T00:37:50.655 DEBUG:boto:Method: GET 2023-07-29T00:37:50.655 DEBUG:boto:Path: /hiiqos-59/?prefix=obj51 2023-07-29T00:37:50.656 DEBUG:boto:Data: 2023-07-29T00:37:50.656 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.656 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:50.657 DEBUG:boto:Port: 8001 2023-07-29T00:37:50.657 DEBUG:boto:Params: {} 2023-07-29T00:37:50.657 DEBUG:boto:Token: None 2023-07-29T00:37:50.658 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/ 2023-07-29T00:37:50.658 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU= 2023-07-29T00:37:50.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU=', 'Content-Length': '0'} 2023-07-29T00:37:50.661 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b9014cc924bfcb7-0064c45f5e-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.661 DEBUG:boto:b'hiiqos-59obj511000falseobj512023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:50.662 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:50.662 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:50.663 DEBUG:boto:path=/hiiqos-59/?prefix=obj51 2023-07-29T00:37:50.663 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj51 2023-07-29T00:37:50.663 DEBUG:boto:Method: GET 2023-07-29T00:37:50.664 DEBUG:boto:Path: /hiiqos-59/?prefix=obj51 2023-07-29T00:37:50.664 DEBUG:boto:Data: 2023-07-29T00:37:50.665 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.665 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:50.665 DEBUG:boto:Port: 8000 2023-07-29T00:37:50.666 DEBUG:boto:Params: {} 2023-07-29T00:37:50.666 DEBUG:boto:Token: None 2023-07-29T00:37:50.667 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/ 2023-07-29T00:37:50.667 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU= 2023-07-29T00:37:50.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU=', 'Content-Length': '0'} 2023-07-29T00:37:50.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000059156f2fb84117-0064c45f5e-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.672 DEBUG:boto:b'hiiqos-59obj511000falseobj512023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:50.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-07-29T00:37:50.673 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:37:50.673 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:37:50.674 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:37:50.674 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:37:50.674 DEBUG:boto:Method: GET 2023-07-29T00:37:50.675 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:37:50.675 DEBUG:boto:Data: 2023-07-29T00:37:50.675 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.676 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:50.676 DEBUG:boto:Port: 8001 2023-07-29T00:37:50.676 DEBUG:boto:Params: {} 2023-07-29T00:37:50.677 DEBUG:boto:Token: None 2023-07-29T00:37:50.677 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:37:50.678 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QNpCSbIYsHhb8Mg1ich8GHE9wEk= 2023-07-29T00:37:50.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QNpCSbIYsHhb8Mg1ich8GHE9wEk=', 'Content-Length': '0'} 2023-07-29T00:37:50.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051818b5459bfde20-0064c45f5e-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.720 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:37:50.721 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:37:50.721 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:37:50.722 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:37:50.722 DEBUG:boto:Method: GET 2023-07-29T00:37:50.722 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:37:50.723 DEBUG:boto:Data: 2023-07-29T00:37:50.723 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.723 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:50.724 DEBUG:boto:Port: 8000 2023-07-29T00:37:50.724 DEBUG:boto:Params: {} 2023-07-29T00:37:50.725 DEBUG:boto:Token: None 2023-07-29T00:37:50.725 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:37:50.726 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QNpCSbIYsHhb8Mg1ich8GHE9wEk= 2023-07-29T00:37:50.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QNpCSbIYsHhb8Mg1ich8GHE9wEk=', 'Content-Length': '0'} 2023-07-29T00:37:50.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3b2e1515b05501b-0064c45f5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.729 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:37:50.729 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:37:50.730 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:37:50.730 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:37:50.730 DEBUG:boto:Method: HEAD 2023-07-29T00:37:50.731 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:37:50.731 DEBUG:boto:Data: 2023-07-29T00:37:50.732 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.732 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:50.732 DEBUG:boto:Port: 8001 2023-07-29T00:37:50.733 DEBUG:boto:Params: {} 2023-07-29T00:37:50.733 DEBUG:boto:Token: None 2023-07-29T00:37:50.734 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:37:50.734 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1VMvRjWGJEXr5s4KF5W4/A5pAYY= 2023-07-29T00:37:50.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1VMvRjWGJEXr5s4KF5W4/A5pAYY=', 'Content-Length': '0'} 2023-07-29T00:37:50.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb89baa09053d7a1-0064c45f5e-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.737 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:37:50.737 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:37:50.737 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:37:50.738 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:37:50.738 DEBUG:boto:Method: HEAD 2023-07-29T00:37:50.738 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:37:50.739 DEBUG:boto:Data: 2023-07-29T00:37:50.739 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.739 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:50.740 DEBUG:boto:Port: 8000 2023-07-29T00:37:50.740 DEBUG:boto:Params: {} 2023-07-29T00:37:50.740 DEBUG:boto:Token: None 2023-07-29T00:37:50.741 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:37:50.741 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1VMvRjWGJEXr5s4KF5W4/A5pAYY= 2023-07-29T00:37:50.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1VMvRjWGJEXr5s4KF5W4/A5pAYY=', 'Content-Length': '0'} 2023-07-29T00:37:50.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d58dc6f147cc81d9-0064c45f5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-07-29T00:37:50.745 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:37:50.745 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:37:50.745 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:37:50.746 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:37:50.746 DEBUG:boto:Method: GET 2023-07-29T00:37:50.746 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:37:50.747 DEBUG:boto:Data: 2023-07-29T00:37:50.747 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.747 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:50.748 DEBUG:boto:Port: 8001 2023-07-29T00:37:50.748 DEBUG:boto:Params: {} 2023-07-29T00:37:50.749 DEBUG:boto:Token: None 2023-07-29T00:37:50.749 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:37:50.749 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QNpCSbIYsHhb8Mg1ich8GHE9wEk= 2023-07-29T00:37:50.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QNpCSbIYsHhb8Mg1ich8GHE9wEk=', 'Content-Length': '0'} 2023-07-29T00:37:50.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a25109f56387f380-0064c45f5e-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.752 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:37:50.753 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:37:50.753 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:37:50.753 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:37:50.754 DEBUG:boto:Method: GET 2023-07-29T00:37:50.754 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:37:50.754 DEBUG:boto:Data: 2023-07-29T00:37:50.755 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.755 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:50.755 DEBUG:boto:Port: 8000 2023-07-29T00:37:50.756 DEBUG:boto:Params: {} 2023-07-29T00:37:50.756 DEBUG:boto:Token: None 2023-07-29T00:37:50.757 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:37:50.757 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QNpCSbIYsHhb8Mg1ich8GHE9wEk= 2023-07-29T00:37:50.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QNpCSbIYsHhb8Mg1ich8GHE9wEk=', 'Content-Length': '0'} 2023-07-29T00:37:50.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efaaa1dd982aea7f-0064c45f5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.760 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:50.760 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:50.761 DEBUG:boto:path=/hiiqos-59/?prefix=obj55 2023-07-29T00:37:50.761 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj55 2023-07-29T00:37:50.762 DEBUG:boto:Method: GET 2023-07-29T00:37:50.762 DEBUG:boto:Path: /hiiqos-59/?prefix=obj55 2023-07-29T00:37:50.762 DEBUG:boto:Data: 2023-07-29T00:37:50.763 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.763 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:50.763 DEBUG:boto:Port: 8001 2023-07-29T00:37:50.764 DEBUG:boto:Params: {} 2023-07-29T00:37:50.764 DEBUG:boto:Token: None 2023-07-29T00:37:50.765 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/ 2023-07-29T00:37:50.765 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU= 2023-07-29T00:37:50.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU=', 'Content-Length': '0'} 2023-07-29T00:37:50.768 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000793ecaf42e038cc9-0064c45f5e-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.768 DEBUG:boto:b'hiiqos-59obj551000falseobj552023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:50.769 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:50.770 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:50.770 DEBUG:boto:path=/hiiqos-59/?prefix=obj55 2023-07-29T00:37:50.770 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj55 2023-07-29T00:37:50.771 DEBUG:boto:Method: GET 2023-07-29T00:37:50.771 DEBUG:boto:Path: /hiiqos-59/?prefix=obj55 2023-07-29T00:37:50.771 DEBUG:boto:Data: 2023-07-29T00:37:50.772 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.772 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:50.773 DEBUG:boto:Port: 8000 2023-07-29T00:37:50.773 DEBUG:boto:Params: {} 2023-07-29T00:37:50.773 DEBUG:boto:Token: None 2023-07-29T00:37:50.774 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/ 2023-07-29T00:37:50.774 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU= 2023-07-29T00:37:50.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU=', 'Content-Length': '0'} 2023-07-29T00:37:50.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004050f72b85e3fda4-0064c45f5e-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.779 DEBUG:boto:b'hiiqos-59obj551000falseobj552023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:50.779 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:37:50.780 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:37:50.780 DEBUG:boto:Method: HEAD 2023-07-29T00:37:50.780 DEBUG:boto:Path: /hiiqos-59/obj55 2023-07-29T00:37:50.781 DEBUG:boto:Data: 2023-07-29T00:37:50.781 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.781 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:50.782 DEBUG:boto:Port: 8001 2023-07-29T00:37:50.782 DEBUG:boto:Params: {} 2023-07-29T00:37:50.782 DEBUG:boto:Token: None 2023-07-29T00:37:50.783 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj55 2023-07-29T00:37:50.784 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uS9PKfHfxbXdRkDPKsb7+GZdFxI= 2023-07-29T00:37:50.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uS9PKfHfxbXdRkDPKsb7+GZdFxI=', 'Content-Length': '0'} 2023-07-29T00:37:50.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c06f59929b0931b9-0064c45f5e-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.786 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:37:50.787 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:37:50.787 DEBUG:boto:Method: HEAD 2023-07-29T00:37:50.787 DEBUG:boto:Path: /hiiqos-59/obj55 2023-07-29T00:37:50.788 DEBUG:boto:Data: 2023-07-29T00:37:50.788 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.788 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:50.789 DEBUG:boto:Port: 8000 2023-07-29T00:37:50.789 DEBUG:boto:Params: {} 2023-07-29T00:37:50.789 DEBUG:boto:Token: None 2023-07-29T00:37:50.790 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj55 2023-07-29T00:37:50.790 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uS9PKfHfxbXdRkDPKsb7+GZdFxI= 2023-07-29T00:37:50.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uS9PKfHfxbXdRkDPKsb7+GZdFxI=', 'Content-Length': '0'} 2023-07-29T00:37:50.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000644b1fa06ee7f503-0064c45f5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-07-29T00:37:50.793 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:37:50.794 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:37:50.794 DEBUG:boto:Method: GET 2023-07-29T00:37:50.795 DEBUG:boto:Path: /hiiqos-59/obj55 2023-07-29T00:37:50.795 DEBUG:boto:Data: 2023-07-29T00:37:50.795 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.796 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:50.796 DEBUG:boto:Port: 8001 2023-07-29T00:37:50.796 DEBUG:boto:Params: {} 2023-07-29T00:37:50.797 DEBUG:boto:Token: None 2023-07-29T00:37:50.797 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj55 2023-07-29T00:37:50.798 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RiwZUVd1hVm8MmfgWIpA/uX+bqY= 2023-07-29T00:37:50.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RiwZUVd1hVm8MmfgWIpA/uX+bqY=', 'Content-Length': '0'} 2023-07-29T00:37:50.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bfc25d8be427116-0064c45f5e-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.840 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:37:50.841 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:37:50.841 DEBUG:boto:Method: GET 2023-07-29T00:37:50.842 DEBUG:boto:Path: /hiiqos-59/obj55 2023-07-29T00:37:50.842 DEBUG:boto:Data: 2023-07-29T00:37:50.842 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.843 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:50.843 DEBUG:boto:Port: 8000 2023-07-29T00:37:50.843 DEBUG:boto:Params: {} 2023-07-29T00:37:50.844 DEBUG:boto:Token: None 2023-07-29T00:37:50.844 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj55 2023-07-29T00:37:50.845 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RiwZUVd1hVm8MmfgWIpA/uX+bqY= 2023-07-29T00:37:50.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RiwZUVd1hVm8MmfgWIpA/uX+bqY=', 'Content-Length': '0'} 2023-07-29T00:37:50.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a53c49931d16342b-0064c45f5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.848 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:50.848 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:50.848 DEBUG:boto:path=/hiiqos-59/?prefix=obj55 2023-07-29T00:37:50.849 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj55 2023-07-29T00:37:50.849 DEBUG:boto:Method: GET 2023-07-29T00:37:50.850 DEBUG:boto:Path: /hiiqos-59/?prefix=obj55 2023-07-29T00:37:50.850 DEBUG:boto:Data: 2023-07-29T00:37:50.850 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.851 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:50.851 DEBUG:boto:Port: 8001 2023-07-29T00:37:50.851 DEBUG:boto:Params: {} 2023-07-29T00:37:50.852 DEBUG:boto:Token: None 2023-07-29T00:37:50.852 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/ 2023-07-29T00:37:50.853 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU= 2023-07-29T00:37:50.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU=', 'Content-Length': '0'} 2023-07-29T00:37:50.856 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c701a2315c81795-0064c45f5e-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.856 DEBUG:boto:b'hiiqos-59obj551000falseobj552023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:50.857 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:50.857 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:50.858 DEBUG:boto:path=/hiiqos-59/?prefix=obj55 2023-07-29T00:37:50.858 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj55 2023-07-29T00:37:50.858 DEBUG:boto:Method: GET 2023-07-29T00:37:50.859 DEBUG:boto:Path: /hiiqos-59/?prefix=obj55 2023-07-29T00:37:50.859 DEBUG:boto:Data: 2023-07-29T00:37:50.859 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.860 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:50.860 DEBUG:boto:Port: 8000 2023-07-29T00:37:50.860 DEBUG:boto:Params: {} 2023-07-29T00:37:50.861 DEBUG:boto:Token: None 2023-07-29T00:37:50.861 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/ 2023-07-29T00:37:50.862 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU= 2023-07-29T00:37:50.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU=', 'Content-Length': '0'} 2023-07-29T00:37:50.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069df8bc27f24a290-0064c45f5e-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.866 DEBUG:boto:b'hiiqos-59obj551000falseobj552023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:50.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-07-29T00:37:50.867 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:37:50.867 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:37:50.868 DEBUG:boto:path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:37:50.868 DEBUG:boto:auth_path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:37:50.869 DEBUG:boto:Method: GET 2023-07-29T00:37:50.869 DEBUG:boto:Path: /hiiqos-59/obj57?versionId=null 2023-07-29T00:37:50.869 DEBUG:boto:Data: 2023-07-29T00:37:50.870 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.870 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:50.870 DEBUG:boto:Port: 8001 2023-07-29T00:37:50.871 DEBUG:boto:Params: {} 2023-07-29T00:37:50.871 DEBUG:boto:Token: None 2023-07-29T00:37:50.871 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj57?versionId=null 2023-07-29T00:37:50.872 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:abJASGM39UxOqg1ekLaNaY/7wsw= 2023-07-29T00:37:50.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:abJASGM39UxOqg1ekLaNaY/7wsw=', 'Content-Length': '0'} 2023-07-29T00:37:50.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f03832de32528d1-0064c45f5e-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.875 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:37:50.875 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:37:50.876 DEBUG:boto:path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:37:50.876 DEBUG:boto:auth_path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:37:50.876 DEBUG:boto:Method: GET 2023-07-29T00:37:50.877 DEBUG:boto:Path: /hiiqos-59/obj57?versionId=null 2023-07-29T00:37:50.877 DEBUG:boto:Data: 2023-07-29T00:37:50.877 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.878 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:50.878 DEBUG:boto:Port: 8000 2023-07-29T00:37:50.878 DEBUG:boto:Params: {} 2023-07-29T00:37:50.879 DEBUG:boto:Token: None 2023-07-29T00:37:50.879 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj57?versionId=null 2023-07-29T00:37:50.880 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:abJASGM39UxOqg1ekLaNaY/7wsw= 2023-07-29T00:37:50.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:abJASGM39UxOqg1ekLaNaY/7wsw=', 'Content-Length': '0'} 2023-07-29T00:37:50.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012ea84f3f9d39f95-0064c45f5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.882 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:37:50.883 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:37:50.883 DEBUG:boto:path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:37:50.884 DEBUG:boto:auth_path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:37:50.884 DEBUG:boto:Method: HEAD 2023-07-29T00:37:50.884 DEBUG:boto:Path: /hiiqos-59/obj57?versionId=null 2023-07-29T00:37:50.885 DEBUG:boto:Data: 2023-07-29T00:37:50.885 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.885 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:50.886 DEBUG:boto:Port: 8001 2023-07-29T00:37:50.886 DEBUG:boto:Params: {} 2023-07-29T00:37:50.886 DEBUG:boto:Token: None 2023-07-29T00:37:50.887 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj57?versionId=null 2023-07-29T00:37:50.887 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:raCW/KBTufxHVBz4R1bFkyWDYJQ= 2023-07-29T00:37:50.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:raCW/KBTufxHVBz4R1bFkyWDYJQ=', 'Content-Length': '0'} 2023-07-29T00:37:50.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000567a0580f017f996-0064c45f5e-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.890 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:37:50.890 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:37:50.891 DEBUG:boto:path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:37:50.891 DEBUG:boto:auth_path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:37:50.891 DEBUG:boto:Method: HEAD 2023-07-29T00:37:50.892 DEBUG:boto:Path: /hiiqos-59/obj57?versionId=null 2023-07-29T00:37:50.892 DEBUG:boto:Data: 2023-07-29T00:37:50.892 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.893 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:50.893 DEBUG:boto:Port: 8000 2023-07-29T00:37:50.893 DEBUG:boto:Params: {} 2023-07-29T00:37:50.894 DEBUG:boto:Token: None 2023-07-29T00:37:50.894 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj57?versionId=null 2023-07-29T00:37:50.895 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:raCW/KBTufxHVBz4R1bFkyWDYJQ= 2023-07-29T00:37:50.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:raCW/KBTufxHVBz4R1bFkyWDYJQ=', 'Content-Length': '0'} 2023-07-29T00:37:50.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fad45379a81b62b0-0064c45f5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-07-29T00:37:50.898 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:37:50.898 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:37:50.899 DEBUG:boto:path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:37:50.899 DEBUG:boto:auth_path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:37:50.899 DEBUG:boto:Method: GET 2023-07-29T00:37:50.900 DEBUG:boto:Path: /hiiqos-59/obj57?versionId=null 2023-07-29T00:37:50.900 DEBUG:boto:Data: 2023-07-29T00:37:50.900 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.901 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:50.901 DEBUG:boto:Port: 8001 2023-07-29T00:37:50.901 DEBUG:boto:Params: {} 2023-07-29T00:37:50.902 DEBUG:boto:Token: None 2023-07-29T00:37:50.902 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj57?versionId=null 2023-07-29T00:37:50.903 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:abJASGM39UxOqg1ekLaNaY/7wsw= 2023-07-29T00:37:50.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:abJASGM39UxOqg1ekLaNaY/7wsw=', 'Content-Length': '0'} 2023-07-29T00:37:50.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b662fa265ea0cae6-0064c45f5e-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.905 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:37:50.906 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:37:50.906 DEBUG:boto:path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:37:50.907 DEBUG:boto:auth_path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:37:50.907 DEBUG:boto:Method: GET 2023-07-29T00:37:50.907 DEBUG:boto:Path: /hiiqos-59/obj57?versionId=null 2023-07-29T00:37:50.908 DEBUG:boto:Data: 2023-07-29T00:37:50.908 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.908 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:50.909 DEBUG:boto:Port: 8000 2023-07-29T00:37:50.909 DEBUG:boto:Params: {} 2023-07-29T00:37:50.909 DEBUG:boto:Token: None 2023-07-29T00:37:50.910 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj57?versionId=null 2023-07-29T00:37:50.910 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:abJASGM39UxOqg1ekLaNaY/7wsw= 2023-07-29T00:37:50.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:abJASGM39UxOqg1ekLaNaY/7wsw=', 'Content-Length': '0'} 2023-07-29T00:37:50.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd5b7a788c2127fe-0064c45f5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.913 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:50.913 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:50.914 DEBUG:boto:path=/hiiqos-59/?prefix=obj57 2023-07-29T00:37:50.914 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj57 2023-07-29T00:37:50.915 DEBUG:boto:Method: GET 2023-07-29T00:37:50.915 DEBUG:boto:Path: /hiiqos-59/?prefix=obj57 2023-07-29T00:37:50.915 DEBUG:boto:Data: 2023-07-29T00:37:50.916 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.916 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:50.916 DEBUG:boto:Port: 8001 2023-07-29T00:37:50.917 DEBUG:boto:Params: {} 2023-07-29T00:37:50.917 DEBUG:boto:Token: None 2023-07-29T00:37:50.917 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/ 2023-07-29T00:37:50.918 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU= 2023-07-29T00:37:50.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU=', 'Content-Length': '0'} 2023-07-29T00:37:50.921 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002bcea15e664eb10c-0064c45f5e-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.921 DEBUG:boto:b'hiiqos-59obj571000falseobj572023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:50.922 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:50.922 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:50.923 DEBUG:boto:path=/hiiqos-59/?prefix=obj57 2023-07-29T00:37:50.923 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj57 2023-07-29T00:37:50.923 DEBUG:boto:Method: GET 2023-07-29T00:37:50.924 DEBUG:boto:Path: /hiiqos-59/?prefix=obj57 2023-07-29T00:37:50.924 DEBUG:boto:Data: 2023-07-29T00:37:50.924 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.925 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:50.925 DEBUG:boto:Port: 8000 2023-07-29T00:37:50.925 DEBUG:boto:Params: {} 2023-07-29T00:37:50.926 DEBUG:boto:Token: None 2023-07-29T00:37:50.926 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/ 2023-07-29T00:37:50.927 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU= 2023-07-29T00:37:50.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KxHMp+53sc5zLPb/v+0Qe6KRcJU=', 'Content-Length': '0'} 2023-07-29T00:37:50.931 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d18e0cfdc3b922dc-0064c45f5e-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.972 DEBUG:boto:b'hiiqos-59obj571000falseobj572023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:50.973 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:37:50.973 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:37:50.974 DEBUG:boto:Method: HEAD 2023-07-29T00:37:50.974 DEBUG:boto:Path: /hiiqos-59/obj57 2023-07-29T00:37:50.975 DEBUG:boto:Data: 2023-07-29T00:37:50.975 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.975 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:50.975 DEBUG:boto:Port: 8001 2023-07-29T00:37:50.976 DEBUG:boto:Params: {} 2023-07-29T00:37:50.976 DEBUG:boto:Token: None 2023-07-29T00:37:50.977 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj57 2023-07-29T00:37:50.977 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:E6Qd4Lt0YozZSOPWPs6wJzUuFHs= 2023-07-29T00:37:50.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:E6Qd4Lt0YozZSOPWPs6wJzUuFHs=', 'Content-Length': '0'} 2023-07-29T00:37:50.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aae23cceaa85fba9-0064c45f5e-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.980 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:37:50.980 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:37:50.981 DEBUG:boto:Method: HEAD 2023-07-29T00:37:50.981 DEBUG:boto:Path: /hiiqos-59/obj57 2023-07-29T00:37:50.981 DEBUG:boto:Data: 2023-07-29T00:37:50.982 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.982 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:50.982 DEBUG:boto:Port: 8000 2023-07-29T00:37:50.983 DEBUG:boto:Params: {} 2023-07-29T00:37:50.983 DEBUG:boto:Token: None 2023-07-29T00:37:50.983 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj57 2023-07-29T00:37:50.984 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:E6Qd4Lt0YozZSOPWPs6wJzUuFHs= 2023-07-29T00:37:50.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:E6Qd4Lt0YozZSOPWPs6wJzUuFHs=', 'Content-Length': '0'} 2023-07-29T00:37:50.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f95c0e1a2fe7de9-0064c45f5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-07-29T00:37:50.987 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:37:50.987 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:37:50.988 DEBUG:boto:Method: GET 2023-07-29T00:37:50.988 DEBUG:boto:Path: /hiiqos-59/obj57 2023-07-29T00:37:50.988 DEBUG:boto:Data: 2023-07-29T00:37:50.989 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.989 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:50.989 DEBUG:boto:Port: 8001 2023-07-29T00:37:50.990 DEBUG:boto:Params: {} 2023-07-29T00:37:50.990 DEBUG:boto:Token: None 2023-07-29T00:37:50.991 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj57 2023-07-29T00:37:50.991 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0MZAS4GealfHQ5bGnarhgF+qn7M= 2023-07-29T00:37:50.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0MZAS4GealfHQ5bGnarhgF+qn7M=', 'Content-Length': '0'} 2023-07-29T00:37:50.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bc49dcb839613a3-0064c45f5e-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:50.994 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:37:50.994 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:37:50.995 DEBUG:boto:Method: GET 2023-07-29T00:37:50.995 DEBUG:boto:Path: /hiiqos-59/obj57 2023-07-29T00:37:50.995 DEBUG:boto:Data: 2023-07-29T00:37:50.996 DEBUG:boto:Headers: {} 2023-07-29T00:37:50.996 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:50.996 DEBUG:boto:Port: 8000 2023-07-29T00:37:50.997 DEBUG:boto:Params: {} 2023-07-29T00:37:50.997 DEBUG:boto:Token: None 2023-07-29T00:37:50.998 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:50 GMT /hiiqos-59/obj57 2023-07-29T00:37:50.998 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0MZAS4GealfHQ5bGnarhgF+qn7M= 2023-07-29T00:37:50.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:50 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0MZAS4GealfHQ5bGnarhgF+qn7M=', 'Content-Length': '0'} 2023-07-29T00:37:51.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f68a8f6c2b8f6eb-0064c45f5e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.001 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:51.001 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:51.002 DEBUG:boto:path=/hiiqos-59/?prefix=obj57 2023-07-29T00:37:51.002 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj57 2023-07-29T00:37:51.003 DEBUG:boto:Method: GET 2023-07-29T00:37:51.003 DEBUG:boto:Path: /hiiqos-59/?prefix=obj57 2023-07-29T00:37:51.003 DEBUG:boto:Data: 2023-07-29T00:37:51.004 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.004 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:51.005 DEBUG:boto:Port: 8001 2023-07-29T00:37:51.005 DEBUG:boto:Params: {} 2023-07-29T00:37:51.005 DEBUG:boto:Token: None 2023-07-29T00:37:51.006 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/ 2023-07-29T00:37:51.006 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vNM5Y5SCpOQf7IWALA8ChXbEFm8= 2023-07-29T00:37:51.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vNM5Y5SCpOQf7IWALA8ChXbEFm8=', 'Content-Length': '0'} 2023-07-29T00:37:51.009 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008194dc5c21cd170d-0064c45f5e-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.009 DEBUG:boto:b'hiiqos-59obj571000falseobj572023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:51.010 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:51.011 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:51.011 DEBUG:boto:path=/hiiqos-59/?prefix=obj57 2023-07-29T00:37:51.011 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj57 2023-07-29T00:37:51.012 DEBUG:boto:Method: GET 2023-07-29T00:37:51.012 DEBUG:boto:Path: /hiiqos-59/?prefix=obj57 2023-07-29T00:37:51.012 DEBUG:boto:Data: 2023-07-29T00:37:51.013 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.013 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:51.013 DEBUG:boto:Port: 8000 2023-07-29T00:37:51.014 DEBUG:boto:Params: {} 2023-07-29T00:37:51.014 DEBUG:boto:Token: None 2023-07-29T00:37:51.015 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/ 2023-07-29T00:37:51.015 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vNM5Y5SCpOQf7IWALA8ChXbEFm8= 2023-07-29T00:37:51.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vNM5Y5SCpOQf7IWALA8ChXbEFm8=', 'Content-Length': '0'} 2023-07-29T00:37:51.019 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3a181d972248a67-0064c45f5e-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.020 DEBUG:boto:b'hiiqos-59obj571000falseobj572023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:51.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-07-29T00:37:51.021 DEBUG:boto:path=/hiiqos-59/obj58 2023-07-29T00:37:51.021 DEBUG:boto:auth_path=/hiiqos-59/obj58 2023-07-29T00:37:51.022 DEBUG:boto:path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:37:51.022 DEBUG:boto:auth_path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:37:51.022 DEBUG:boto:Method: GET 2023-07-29T00:37:51.023 DEBUG:boto:Path: /hiiqos-59/obj58?versionId=null 2023-07-29T00:37:51.023 DEBUG:boto:Data: 2023-07-29T00:37:51.023 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.024 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:51.024 DEBUG:boto:Port: 8001 2023-07-29T00:37:51.024 DEBUG:boto:Params: {} 2023-07-29T00:37:51.025 DEBUG:boto:Token: None 2023-07-29T00:37:51.025 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj58?versionId=null 2023-07-29T00:37:51.026 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DQt2pmSC5CmplWHO+0PtPISZmPc= 2023-07-29T00:37:51.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DQt2pmSC5CmplWHO+0PtPISZmPc=', 'Content-Length': '0'} 2023-07-29T00:37:51.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf4300c992c869c4-0064c45f5e-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:50 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.068 DEBUG:boto:path=/hiiqos-59/obj58 2023-07-29T00:37:51.069 DEBUG:boto:auth_path=/hiiqos-59/obj58 2023-07-29T00:37:51.069 DEBUG:boto:path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:37:51.070 DEBUG:boto:auth_path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:37:51.070 DEBUG:boto:Method: GET 2023-07-29T00:37:51.070 DEBUG:boto:Path: /hiiqos-59/obj58?versionId=null 2023-07-29T00:37:51.071 DEBUG:boto:Data: 2023-07-29T00:37:51.071 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.071 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:51.072 DEBUG:boto:Port: 8000 2023-07-29T00:37:51.072 DEBUG:boto:Params: {} 2023-07-29T00:37:51.073 DEBUG:boto:Token: None 2023-07-29T00:37:51.073 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj58?versionId=null 2023-07-29T00:37:51.073 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DQt2pmSC5CmplWHO+0PtPISZmPc= 2023-07-29T00:37:51.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DQt2pmSC5CmplWHO+0PtPISZmPc=', 'Content-Length': '0'} 2023-07-29T00:37:51.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba2cffded7f3df9e-0064c45f5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.077 DEBUG:boto:path=/hiiqos-59/obj58 2023-07-29T00:37:51.077 DEBUG:boto:auth_path=/hiiqos-59/obj58 2023-07-29T00:37:51.077 DEBUG:boto:path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:37:51.078 DEBUG:boto:auth_path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:37:51.078 DEBUG:boto:Method: HEAD 2023-07-29T00:37:51.078 DEBUG:boto:Path: /hiiqos-59/obj58?versionId=null 2023-07-29T00:37:51.079 DEBUG:boto:Data: 2023-07-29T00:37:51.079 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.079 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:51.080 DEBUG:boto:Port: 8001 2023-07-29T00:37:51.080 DEBUG:boto:Params: {} 2023-07-29T00:37:51.080 DEBUG:boto:Token: None 2023-07-29T00:37:51.081 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj58?versionId=null 2023-07-29T00:37:51.081 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S42oYmkzCJfDLJLtSOo4WinYwWY= 2023-07-29T00:37:51.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S42oYmkzCJfDLJLtSOo4WinYwWY=', 'Content-Length': '0'} 2023-07-29T00:37:51.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2050da2eb73d271-0064c45f5f-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.084 DEBUG:boto:path=/hiiqos-59/obj58 2023-07-29T00:37:51.084 DEBUG:boto:auth_path=/hiiqos-59/obj58 2023-07-29T00:37:51.085 DEBUG:boto:path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:37:51.085 DEBUG:boto:auth_path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:37:51.085 DEBUG:boto:Method: HEAD 2023-07-29T00:37:51.086 DEBUG:boto:Path: /hiiqos-59/obj58?versionId=null 2023-07-29T00:37:51.086 DEBUG:boto:Data: 2023-07-29T00:37:51.086 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.087 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:51.087 DEBUG:boto:Port: 8000 2023-07-29T00:37:51.087 DEBUG:boto:Params: {} 2023-07-29T00:37:51.088 DEBUG:boto:Token: None 2023-07-29T00:37:51.088 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj58?versionId=null 2023-07-29T00:37:51.088 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S42oYmkzCJfDLJLtSOo4WinYwWY= 2023-07-29T00:37:51.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S42oYmkzCJfDLJLtSOo4WinYwWY=', 'Content-Length': '0'} 2023-07-29T00:37:51.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2c3a2ccc8dfc168-0064c45f5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-07-29T00:37:51.092 DEBUG:boto:path=/hiiqos-59/obj58 2023-07-29T00:37:51.092 DEBUG:boto:auth_path=/hiiqos-59/obj58 2023-07-29T00:37:51.092 DEBUG:boto:path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:37:51.093 DEBUG:boto:auth_path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:37:51.093 DEBUG:boto:Method: GET 2023-07-29T00:37:51.093 DEBUG:boto:Path: /hiiqos-59/obj58?versionId=null 2023-07-29T00:37:51.094 DEBUG:boto:Data: 2023-07-29T00:37:51.094 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.095 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:51.095 DEBUG:boto:Port: 8001 2023-07-29T00:37:51.095 DEBUG:boto:Params: {} 2023-07-29T00:37:51.096 DEBUG:boto:Token: None 2023-07-29T00:37:51.096 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj58?versionId=null 2023-07-29T00:37:51.097 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DQt2pmSC5CmplWHO+0PtPISZmPc= 2023-07-29T00:37:51.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DQt2pmSC5CmplWHO+0PtPISZmPc=', 'Content-Length': '0'} 2023-07-29T00:37:51.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000019e0f5ed98e101f-0064c45f5f-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.099 DEBUG:boto:path=/hiiqos-59/obj58 2023-07-29T00:37:51.100 DEBUG:boto:auth_path=/hiiqos-59/obj58 2023-07-29T00:37:51.100 DEBUG:boto:path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:37:51.100 DEBUG:boto:auth_path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:37:51.101 DEBUG:boto:Method: GET 2023-07-29T00:37:51.101 DEBUG:boto:Path: /hiiqos-59/obj58?versionId=null 2023-07-29T00:37:51.102 DEBUG:boto:Data: 2023-07-29T00:37:51.102 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.102 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:51.103 DEBUG:boto:Port: 8000 2023-07-29T00:37:51.103 DEBUG:boto:Params: {} 2023-07-29T00:37:51.103 DEBUG:boto:Token: None 2023-07-29T00:37:51.104 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj58?versionId=null 2023-07-29T00:37:51.104 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DQt2pmSC5CmplWHO+0PtPISZmPc= 2023-07-29T00:37:51.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DQt2pmSC5CmplWHO+0PtPISZmPc=', 'Content-Length': '0'} 2023-07-29T00:37:51.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a448050e5785012-0064c45f5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.107 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:51.108 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:51.108 DEBUG:boto:path=/hiiqos-59/?prefix=obj58 2023-07-29T00:37:51.108 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj58 2023-07-29T00:37:51.109 DEBUG:boto:Method: GET 2023-07-29T00:37:51.109 DEBUG:boto:Path: /hiiqos-59/?prefix=obj58 2023-07-29T00:37:51.109 DEBUG:boto:Data: 2023-07-29T00:37:51.110 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.110 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:51.110 DEBUG:boto:Port: 8001 2023-07-29T00:37:51.111 DEBUG:boto:Params: {} 2023-07-29T00:37:51.111 DEBUG:boto:Token: None 2023-07-29T00:37:51.112 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/ 2023-07-29T00:37:51.112 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vNM5Y5SCpOQf7IWALA8ChXbEFm8= 2023-07-29T00:37:51.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vNM5Y5SCpOQf7IWALA8ChXbEFm8=', 'Content-Length': '0'} 2023-07-29T00:37:51.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5830985e5c3cea4-0064c45f5f-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.115 DEBUG:boto:b'hiiqos-59obj581000falseobj582023-07-29T00:35:01.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:51.116 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:51.116 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:51.117 DEBUG:boto:path=/hiiqos-59/?prefix=obj58 2023-07-29T00:37:51.117 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj58 2023-07-29T00:37:51.117 DEBUG:boto:Method: GET 2023-07-29T00:37:51.118 DEBUG:boto:Path: /hiiqos-59/?prefix=obj58 2023-07-29T00:37:51.118 DEBUG:boto:Data: 2023-07-29T00:37:51.118 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.119 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:51.119 DEBUG:boto:Port: 8000 2023-07-29T00:37:51.120 DEBUG:boto:Params: {} 2023-07-29T00:37:51.120 DEBUG:boto:Token: None 2023-07-29T00:37:51.120 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/ 2023-07-29T00:37:51.121 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vNM5Y5SCpOQf7IWALA8ChXbEFm8= 2023-07-29T00:37:51.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vNM5Y5SCpOQf7IWALA8ChXbEFm8=', 'Content-Length': '0'} 2023-07-29T00:37:51.125 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020e8faeb3bbf6dfb-0064c45f5f-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.125 DEBUG:boto:b'hiiqos-59obj581000falseobj582023-07-29T00:35:01.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:51.126 DEBUG:boto:path=/hiiqos-59/obj58 2023-07-29T00:37:51.126 DEBUG:boto:auth_path=/hiiqos-59/obj58 2023-07-29T00:37:51.127 DEBUG:boto:Method: HEAD 2023-07-29T00:37:51.127 DEBUG:boto:Path: /hiiqos-59/obj58 2023-07-29T00:37:51.127 DEBUG:boto:Data: 2023-07-29T00:37:51.128 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.128 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:51.128 DEBUG:boto:Port: 8001 2023-07-29T00:37:51.129 DEBUG:boto:Params: {} 2023-07-29T00:37:51.129 DEBUG:boto:Token: None 2023-07-29T00:37:51.129 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj58 2023-07-29T00:37:51.130 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eus6KFulPAmxbS7bkIF9K2Amojw= 2023-07-29T00:37:51.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eus6KFulPAmxbS7bkIF9K2Amojw=', 'Content-Length': '0'} 2023-07-29T00:37:51.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ee62652142567d2-0064c45f5f-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.132 DEBUG:boto:path=/hiiqos-59/obj58 2023-07-29T00:37:51.133 DEBUG:boto:auth_path=/hiiqos-59/obj58 2023-07-29T00:37:51.133 DEBUG:boto:Method: HEAD 2023-07-29T00:37:51.133 DEBUG:boto:Path: /hiiqos-59/obj58 2023-07-29T00:37:51.134 DEBUG:boto:Data: 2023-07-29T00:37:51.134 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.135 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:51.135 DEBUG:boto:Port: 8000 2023-07-29T00:37:51.135 DEBUG:boto:Params: {} 2023-07-29T00:37:51.136 DEBUG:boto:Token: None 2023-07-29T00:37:51.136 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj58 2023-07-29T00:37:51.136 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eus6KFulPAmxbS7bkIF9K2Amojw= 2023-07-29T00:37:51.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eus6KFulPAmxbS7bkIF9K2Amojw=', 'Content-Length': '0'} 2023-07-29T00:37:51.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e0d2fb7ef3b53a9-0064c45f5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-07-29T00:37:51.139 DEBUG:boto:path=/hiiqos-59/obj58 2023-07-29T00:37:51.140 DEBUG:boto:auth_path=/hiiqos-59/obj58 2023-07-29T00:37:51.140 DEBUG:boto:Method: GET 2023-07-29T00:37:51.141 DEBUG:boto:Path: /hiiqos-59/obj58 2023-07-29T00:37:51.141 DEBUG:boto:Data: 2023-07-29T00:37:51.141 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.142 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:51.142 DEBUG:boto:Port: 8001 2023-07-29T00:37:51.142 DEBUG:boto:Params: {} 2023-07-29T00:37:51.143 DEBUG:boto:Token: None 2023-07-29T00:37:51.143 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj58 2023-07-29T00:37:51.144 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:K9JfYmWDP6UXmx2pzczNs2a8YHk= 2023-07-29T00:37:51.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K9JfYmWDP6UXmx2pzczNs2a8YHk=', 'Content-Length': '0'} 2023-07-29T00:37:51.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a11a3fa8002c1100-0064c45f5f-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.147 DEBUG:boto:path=/hiiqos-59/obj58 2023-07-29T00:37:51.147 DEBUG:boto:auth_path=/hiiqos-59/obj58 2023-07-29T00:37:51.147 DEBUG:boto:Method: GET 2023-07-29T00:37:51.148 DEBUG:boto:Path: /hiiqos-59/obj58 2023-07-29T00:37:51.148 DEBUG:boto:Data: 2023-07-29T00:37:51.148 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.149 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:51.149 DEBUG:boto:Port: 8000 2023-07-29T00:37:51.150 DEBUG:boto:Params: {} 2023-07-29T00:37:51.150 DEBUG:boto:Token: None 2023-07-29T00:37:51.150 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj58 2023-07-29T00:37:51.151 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:K9JfYmWDP6UXmx2pzczNs2a8YHk= 2023-07-29T00:37:51.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K9JfYmWDP6UXmx2pzczNs2a8YHk=', 'Content-Length': '0'} 2023-07-29T00:37:51.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000067a163cd8abab47-0064c45f5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.196 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:51.197 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:51.197 DEBUG:boto:path=/hiiqos-59/?prefix=obj58 2023-07-29T00:37:51.197 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj58 2023-07-29T00:37:51.198 DEBUG:boto:Method: GET 2023-07-29T00:37:51.198 DEBUG:boto:Path: /hiiqos-59/?prefix=obj58 2023-07-29T00:37:51.199 DEBUG:boto:Data: 2023-07-29T00:37:51.199 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.199 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:51.200 DEBUG:boto:Port: 8001 2023-07-29T00:37:51.200 DEBUG:boto:Params: {} 2023-07-29T00:37:51.200 DEBUG:boto:Token: None 2023-07-29T00:37:51.201 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/ 2023-07-29T00:37:51.201 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vNM5Y5SCpOQf7IWALA8ChXbEFm8= 2023-07-29T00:37:51.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vNM5Y5SCpOQf7IWALA8ChXbEFm8=', 'Content-Length': '0'} 2023-07-29T00:37:51.204 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000892698ae654bc62a-0064c45f5f-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.205 DEBUG:boto:b'hiiqos-59obj581000falseobj582023-07-29T00:35:01.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:51.205 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:51.206 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:51.206 DEBUG:boto:path=/hiiqos-59/?prefix=obj58 2023-07-29T00:37:51.206 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj58 2023-07-29T00:37:51.207 DEBUG:boto:Method: GET 2023-07-29T00:37:51.207 DEBUG:boto:Path: /hiiqos-59/?prefix=obj58 2023-07-29T00:37:51.207 DEBUG:boto:Data: 2023-07-29T00:37:51.208 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.208 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:51.208 DEBUG:boto:Port: 8000 2023-07-29T00:37:51.209 DEBUG:boto:Params: {} 2023-07-29T00:37:51.209 DEBUG:boto:Token: None 2023-07-29T00:37:51.210 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/ 2023-07-29T00:37:51.210 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vNM5Y5SCpOQf7IWALA8ChXbEFm8= 2023-07-29T00:37:51.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vNM5Y5SCpOQf7IWALA8ChXbEFm8=', 'Content-Length': '0'} 2023-07-29T00:37:51.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000faad900e46e6e2f5-0064c45f5f-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.215 DEBUG:boto:b'hiiqos-59obj581000falseobj582023-07-29T00:35:01.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:51.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-07-29T00:37:51.216 DEBUG:boto:path=/hiiqos-59/obj62 2023-07-29T00:37:51.216 DEBUG:boto:auth_path=/hiiqos-59/obj62 2023-07-29T00:37:51.217 DEBUG:boto:path=/hiiqos-59/obj62?versionId=null 2023-07-29T00:37:51.217 DEBUG:boto:auth_path=/hiiqos-59/obj62?versionId=null 2023-07-29T00:37:51.217 DEBUG:boto:Method: GET 2023-07-29T00:37:51.218 DEBUG:boto:Path: /hiiqos-59/obj62?versionId=null 2023-07-29T00:37:51.218 DEBUG:boto:Data: 2023-07-29T00:37:51.218 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.219 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:51.219 DEBUG:boto:Port: 8001 2023-07-29T00:37:51.219 DEBUG:boto:Params: {} 2023-07-29T00:37:51.220 DEBUG:boto:Token: None 2023-07-29T00:37:51.220 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj62?versionId=null 2023-07-29T00:37:51.221 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pCUkrfR2mzpsIbp8XwUJ6iege08= 2023-07-29T00:37:51.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pCUkrfR2mzpsIbp8XwUJ6iege08=', 'Content-Length': '0'} 2023-07-29T00:37:51.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcf4f47ff7eefc02-0064c45f5f-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.224 DEBUG:boto:path=/hiiqos-59/obj62 2023-07-29T00:37:51.224 DEBUG:boto:auth_path=/hiiqos-59/obj62 2023-07-29T00:37:51.225 DEBUG:boto:path=/hiiqos-59/obj62?versionId=null 2023-07-29T00:37:51.225 DEBUG:boto:auth_path=/hiiqos-59/obj62?versionId=null 2023-07-29T00:37:51.225 DEBUG:boto:Method: GET 2023-07-29T00:37:51.226 DEBUG:boto:Path: /hiiqos-59/obj62?versionId=null 2023-07-29T00:37:51.226 DEBUG:boto:Data: 2023-07-29T00:37:51.226 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.227 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:51.227 DEBUG:boto:Port: 8000 2023-07-29T00:37:51.227 DEBUG:boto:Params: {} 2023-07-29T00:37:51.228 DEBUG:boto:Token: None 2023-07-29T00:37:51.228 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj62?versionId=null 2023-07-29T00:37:51.229 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pCUkrfR2mzpsIbp8XwUJ6iege08= 2023-07-29T00:37:51.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pCUkrfR2mzpsIbp8XwUJ6iege08=', 'Content-Length': '0'} 2023-07-29T00:37:51.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e1d8d46b3d74099-0064c45f5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.232 DEBUG:boto:path=/hiiqos-59/obj62 2023-07-29T00:37:51.232 DEBUG:boto:auth_path=/hiiqos-59/obj62 2023-07-29T00:37:51.232 DEBUG:boto:path=/hiiqos-59/obj62?versionId=null 2023-07-29T00:37:51.233 DEBUG:boto:auth_path=/hiiqos-59/obj62?versionId=null 2023-07-29T00:37:51.233 DEBUG:boto:Method: HEAD 2023-07-29T00:37:51.233 DEBUG:boto:Path: /hiiqos-59/obj62?versionId=null 2023-07-29T00:37:51.234 DEBUG:boto:Data: 2023-07-29T00:37:51.234 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.234 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:51.235 DEBUG:boto:Port: 8001 2023-07-29T00:37:51.235 DEBUG:boto:Params: {} 2023-07-29T00:37:51.235 DEBUG:boto:Token: None 2023-07-29T00:37:51.236 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj62?versionId=null 2023-07-29T00:37:51.236 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uPuMRvvd9Ekqi9KhKOosPRkjho4= 2023-07-29T00:37:51.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uPuMRvvd9Ekqi9KhKOosPRkjho4=', 'Content-Length': '0'} 2023-07-29T00:37:51.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002270fc4345bfdafd-0064c45f5f-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.239 DEBUG:boto:path=/hiiqos-59/obj62 2023-07-29T00:37:51.239 DEBUG:boto:auth_path=/hiiqos-59/obj62 2023-07-29T00:37:51.239 DEBUG:boto:path=/hiiqos-59/obj62?versionId=null 2023-07-29T00:37:51.240 DEBUG:boto:auth_path=/hiiqos-59/obj62?versionId=null 2023-07-29T00:37:51.240 DEBUG:boto:Method: HEAD 2023-07-29T00:37:51.240 DEBUG:boto:Path: /hiiqos-59/obj62?versionId=null 2023-07-29T00:37:51.241 DEBUG:boto:Data: 2023-07-29T00:37:51.241 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.241 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:51.242 DEBUG:boto:Port: 8000 2023-07-29T00:37:51.242 DEBUG:boto:Params: {} 2023-07-29T00:37:51.242 DEBUG:boto:Token: None 2023-07-29T00:37:51.243 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj62?versionId=null 2023-07-29T00:37:51.243 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uPuMRvvd9Ekqi9KhKOosPRkjho4= 2023-07-29T00:37:51.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uPuMRvvd9Ekqi9KhKOosPRkjho4=', 'Content-Length': '0'} 2023-07-29T00:37:51.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7911843af5f4dac-0064c45f5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-07-29T00:37:51.246 DEBUG:boto:path=/hiiqos-59/obj62 2023-07-29T00:37:51.247 DEBUG:boto:auth_path=/hiiqos-59/obj62 2023-07-29T00:37:51.247 DEBUG:boto:path=/hiiqos-59/obj62?versionId=null 2023-07-29T00:37:51.247 DEBUG:boto:auth_path=/hiiqos-59/obj62?versionId=null 2023-07-29T00:37:51.248 DEBUG:boto:Method: GET 2023-07-29T00:37:51.248 DEBUG:boto:Path: /hiiqos-59/obj62?versionId=null 2023-07-29T00:37:51.248 DEBUG:boto:Data: 2023-07-29T00:37:51.249 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.249 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:51.249 DEBUG:boto:Port: 8001 2023-07-29T00:37:51.250 DEBUG:boto:Params: {} 2023-07-29T00:37:51.250 DEBUG:boto:Token: None 2023-07-29T00:37:51.251 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj62?versionId=null 2023-07-29T00:37:51.251 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pCUkrfR2mzpsIbp8XwUJ6iege08= 2023-07-29T00:37:51.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pCUkrfR2mzpsIbp8XwUJ6iege08=', 'Content-Length': '0'} 2023-07-29T00:37:51.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005781b4a5aaf08a57-0064c45f5f-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.254 DEBUG:boto:path=/hiiqos-59/obj62 2023-07-29T00:37:51.254 DEBUG:boto:auth_path=/hiiqos-59/obj62 2023-07-29T00:37:51.255 DEBUG:boto:path=/hiiqos-59/obj62?versionId=null 2023-07-29T00:37:51.255 DEBUG:boto:auth_path=/hiiqos-59/obj62?versionId=null 2023-07-29T00:37:51.255 DEBUG:boto:Method: GET 2023-07-29T00:37:51.256 DEBUG:boto:Path: /hiiqos-59/obj62?versionId=null 2023-07-29T00:37:51.256 DEBUG:boto:Data: 2023-07-29T00:37:51.257 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.257 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:51.257 DEBUG:boto:Port: 8000 2023-07-29T00:37:51.258 DEBUG:boto:Params: {} 2023-07-29T00:37:51.258 DEBUG:boto:Token: None 2023-07-29T00:37:51.258 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj62?versionId=null 2023-07-29T00:37:51.259 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pCUkrfR2mzpsIbp8XwUJ6iege08= 2023-07-29T00:37:51.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pCUkrfR2mzpsIbp8XwUJ6iege08=', 'Content-Length': '0'} 2023-07-29T00:37:51.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d84a26c07747237d-0064c45f5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.262 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:51.262 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:51.262 DEBUG:boto:path=/hiiqos-59/?prefix=obj62 2023-07-29T00:37:51.263 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj62 2023-07-29T00:37:51.263 DEBUG:boto:Method: GET 2023-07-29T00:37:51.264 DEBUG:boto:Path: /hiiqos-59/?prefix=obj62 2023-07-29T00:37:51.264 DEBUG:boto:Data: 2023-07-29T00:37:51.264 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.265 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:51.265 DEBUG:boto:Port: 8001 2023-07-29T00:37:51.265 DEBUG:boto:Params: {} 2023-07-29T00:37:51.266 DEBUG:boto:Token: None 2023-07-29T00:37:51.266 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/ 2023-07-29T00:37:51.267 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vNM5Y5SCpOQf7IWALA8ChXbEFm8= 2023-07-29T00:37:51.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vNM5Y5SCpOQf7IWALA8ChXbEFm8=', 'Content-Length': '0'} 2023-07-29T00:37:51.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091ae912df0c2e245-0064c45f5f-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.312 DEBUG:boto:b'hiiqos-59obj621000falseobj622023-07-29T00:35:31.826Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:51.313 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:51.313 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:51.314 DEBUG:boto:path=/hiiqos-59/?prefix=obj62 2023-07-29T00:37:51.314 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj62 2023-07-29T00:37:51.315 DEBUG:boto:Method: GET 2023-07-29T00:37:51.315 DEBUG:boto:Path: /hiiqos-59/?prefix=obj62 2023-07-29T00:37:51.315 DEBUG:boto:Data: 2023-07-29T00:37:51.316 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.316 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:51.316 DEBUG:boto:Port: 8000 2023-07-29T00:37:51.317 DEBUG:boto:Params: {} 2023-07-29T00:37:51.317 DEBUG:boto:Token: None 2023-07-29T00:37:51.318 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/ 2023-07-29T00:37:51.318 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vNM5Y5SCpOQf7IWALA8ChXbEFm8= 2023-07-29T00:37:51.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vNM5Y5SCpOQf7IWALA8ChXbEFm8=', 'Content-Length': '0'} 2023-07-29T00:37:51.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bab54efb3fd653e7-0064c45f5f-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.323 DEBUG:boto:b'hiiqos-59obj621000falseobj622023-07-29T00:35:31.826Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:51.324 DEBUG:boto:path=/hiiqos-59/obj62 2023-07-29T00:37:51.324 DEBUG:boto:auth_path=/hiiqos-59/obj62 2023-07-29T00:37:51.324 DEBUG:boto:Method: HEAD 2023-07-29T00:37:51.325 DEBUG:boto:Path: /hiiqos-59/obj62 2023-07-29T00:37:51.325 DEBUG:boto:Data: 2023-07-29T00:37:51.325 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.326 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:51.326 DEBUG:boto:Port: 8001 2023-07-29T00:37:51.326 DEBUG:boto:Params: {} 2023-07-29T00:37:51.327 DEBUG:boto:Token: None 2023-07-29T00:37:51.327 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj62 2023-07-29T00:37:51.328 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ms++xK3OzDRfQ9Ja/cwd8N7Jms0= 2023-07-29T00:37:51.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ms++xK3OzDRfQ9Ja/cwd8N7Jms0=', 'Content-Length': '0'} 2023-07-29T00:37:51.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000616608195da4ce5f-0064c45f5f-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.330 DEBUG:boto:path=/hiiqos-59/obj62 2023-07-29T00:37:51.331 DEBUG:boto:auth_path=/hiiqos-59/obj62 2023-07-29T00:37:51.331 DEBUG:boto:Method: HEAD 2023-07-29T00:37:51.332 DEBUG:boto:Path: /hiiqos-59/obj62 2023-07-29T00:37:51.332 DEBUG:boto:Data: 2023-07-29T00:37:51.332 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.333 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:51.333 DEBUG:boto:Port: 8000 2023-07-29T00:37:51.333 DEBUG:boto:Params: {} 2023-07-29T00:37:51.334 DEBUG:boto:Token: None 2023-07-29T00:37:51.334 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj62 2023-07-29T00:37:51.334 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ms++xK3OzDRfQ9Ja/cwd8N7Jms0= 2023-07-29T00:37:51.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ms++xK3OzDRfQ9Ja/cwd8N7Jms0=', 'Content-Length': '0'} 2023-07-29T00:37:51.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6d5dc23297a436f-0064c45f5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-07-29T00:37:51.337 DEBUG:boto:path=/hiiqos-59/obj62 2023-07-29T00:37:51.338 DEBUG:boto:auth_path=/hiiqos-59/obj62 2023-07-29T00:37:51.338 DEBUG:boto:Method: GET 2023-07-29T00:37:51.339 DEBUG:boto:Path: /hiiqos-59/obj62 2023-07-29T00:37:51.339 DEBUG:boto:Data: 2023-07-29T00:37:51.339 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.340 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:51.340 DEBUG:boto:Port: 8001 2023-07-29T00:37:51.340 DEBUG:boto:Params: {} 2023-07-29T00:37:51.341 DEBUG:boto:Token: None 2023-07-29T00:37:51.341 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj62 2023-07-29T00:37:51.342 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VOhfeIEw3hmw110rKxUN1RaszRk= 2023-07-29T00:37:51.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VOhfeIEw3hmw110rKxUN1RaszRk=', 'Content-Length': '0'} 2023-07-29T00:37:51.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd0aa437971ff5e1-0064c45f5f-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.344 DEBUG:boto:path=/hiiqos-59/obj62 2023-07-29T00:37:51.345 DEBUG:boto:auth_path=/hiiqos-59/obj62 2023-07-29T00:37:51.345 DEBUG:boto:Method: GET 2023-07-29T00:37:51.346 DEBUG:boto:Path: /hiiqos-59/obj62 2023-07-29T00:37:51.346 DEBUG:boto:Data: 2023-07-29T00:37:51.346 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.347 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:51.347 DEBUG:boto:Port: 8000 2023-07-29T00:37:51.347 DEBUG:boto:Params: {} 2023-07-29T00:37:51.348 DEBUG:boto:Token: None 2023-07-29T00:37:51.348 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj62 2023-07-29T00:37:51.349 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VOhfeIEw3hmw110rKxUN1RaszRk= 2023-07-29T00:37:51.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VOhfeIEw3hmw110rKxUN1RaszRk=', 'Content-Length': '0'} 2023-07-29T00:37:51.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fcedda7cd39cf4f-0064c45f5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.392 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:51.393 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:51.393 DEBUG:boto:path=/hiiqos-59/?prefix=obj62 2023-07-29T00:37:51.394 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj62 2023-07-29T00:37:51.394 DEBUG:boto:Method: GET 2023-07-29T00:37:51.394 DEBUG:boto:Path: /hiiqos-59/?prefix=obj62 2023-07-29T00:37:51.395 DEBUG:boto:Data: 2023-07-29T00:37:51.395 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.395 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:51.396 DEBUG:boto:Port: 8001 2023-07-29T00:37:51.396 DEBUG:boto:Params: {} 2023-07-29T00:37:51.397 DEBUG:boto:Token: None 2023-07-29T00:37:51.397 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/ 2023-07-29T00:37:51.397 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vNM5Y5SCpOQf7IWALA8ChXbEFm8= 2023-07-29T00:37:51.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vNM5Y5SCpOQf7IWALA8ChXbEFm8=', 'Content-Length': '0'} 2023-07-29T00:37:51.400 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f1aec81c4cf4522-0064c45f5f-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.401 DEBUG:boto:b'hiiqos-59obj621000falseobj622023-07-29T00:35:31.826Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:51.402 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:51.402 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:51.403 DEBUG:boto:path=/hiiqos-59/?prefix=obj62 2023-07-29T00:37:51.403 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj62 2023-07-29T00:37:51.403 DEBUG:boto:Method: GET 2023-07-29T00:37:51.404 DEBUG:boto:Path: /hiiqos-59/?prefix=obj62 2023-07-29T00:37:51.404 DEBUG:boto:Data: 2023-07-29T00:37:51.405 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.405 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:51.405 DEBUG:boto:Port: 8000 2023-07-29T00:37:51.406 DEBUG:boto:Params: {} 2023-07-29T00:37:51.406 DEBUG:boto:Token: None 2023-07-29T00:37:51.406 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/ 2023-07-29T00:37:51.407 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vNM5Y5SCpOQf7IWALA8ChXbEFm8= 2023-07-29T00:37:51.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vNM5Y5SCpOQf7IWALA8ChXbEFm8=', 'Content-Length': '0'} 2023-07-29T00:37:51.413 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e7bf401ddb2418e-0064c45f5f-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.413 DEBUG:boto:b'hiiqos-59obj621000falseobj622023-07-29T00:35:31.826Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:51.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-07-29T00:37:51.414 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:37:51.415 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:37:51.415 DEBUG:boto:path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:37:51.415 DEBUG:boto:auth_path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:37:51.416 DEBUG:boto:Method: GET 2023-07-29T00:37:51.416 DEBUG:boto:Path: /hiiqos-59/obj68?versionId=null 2023-07-29T00:37:51.416 DEBUG:boto:Data: 2023-07-29T00:37:51.417 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.417 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:51.418 DEBUG:boto:Port: 8001 2023-07-29T00:37:51.418 DEBUG:boto:Params: {} 2023-07-29T00:37:51.418 DEBUG:boto:Token: None 2023-07-29T00:37:51.419 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj68?versionId=null 2023-07-29T00:37:51.419 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w6Poe6eDwGY73hHnOqfnGnEOWoY= 2023-07-29T00:37:51.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w6Poe6eDwGY73hHnOqfnGnEOWoY=', 'Content-Length': '0'} 2023-07-29T00:37:51.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b49e54473b01a01-0064c45f5f-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.422 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:37:51.423 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:37:51.423 DEBUG:boto:path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:37:51.423 DEBUG:boto:auth_path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:37:51.424 DEBUG:boto:Method: GET 2023-07-29T00:37:51.424 DEBUG:boto:Path: /hiiqos-59/obj68?versionId=null 2023-07-29T00:37:51.425 DEBUG:boto:Data: 2023-07-29T00:37:51.425 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.425 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:51.426 DEBUG:boto:Port: 8000 2023-07-29T00:37:51.426 DEBUG:boto:Params: {} 2023-07-29T00:37:51.426 DEBUG:boto:Token: None 2023-07-29T00:37:51.427 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj68?versionId=null 2023-07-29T00:37:51.427 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w6Poe6eDwGY73hHnOqfnGnEOWoY= 2023-07-29T00:37:51.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w6Poe6eDwGY73hHnOqfnGnEOWoY=', 'Content-Length': '0'} 2023-07-29T00:37:51.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c71bba0fbe3d13b3-0064c45f5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.430 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:37:51.431 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:37:51.431 DEBUG:boto:path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:37:51.431 DEBUG:boto:auth_path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:37:51.432 DEBUG:boto:Method: HEAD 2023-07-29T00:37:51.432 DEBUG:boto:Path: /hiiqos-59/obj68?versionId=null 2023-07-29T00:37:51.432 DEBUG:boto:Data: 2023-07-29T00:37:51.433 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.433 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:51.433 DEBUG:boto:Port: 8001 2023-07-29T00:37:51.434 DEBUG:boto:Params: {} 2023-07-29T00:37:51.434 DEBUG:boto:Token: None 2023-07-29T00:37:51.434 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj68?versionId=null 2023-07-29T00:37:51.435 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9NKpqEvghn43F6gvrI3DbL+896Y= 2023-07-29T00:37:51.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9NKpqEvghn43F6gvrI3DbL+896Y=', 'Content-Length': '0'} 2023-07-29T00:37:51.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0608daa046a46f5-0064c45f5f-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.438 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:37:51.438 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:37:51.438 DEBUG:boto:path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:37:51.439 DEBUG:boto:auth_path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:37:51.439 DEBUG:boto:Method: HEAD 2023-07-29T00:37:51.439 DEBUG:boto:Path: /hiiqos-59/obj68?versionId=null 2023-07-29T00:37:51.440 DEBUG:boto:Data: 2023-07-29T00:37:51.440 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.440 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:51.441 DEBUG:boto:Port: 8000 2023-07-29T00:37:51.441 DEBUG:boto:Params: {} 2023-07-29T00:37:51.441 DEBUG:boto:Token: None 2023-07-29T00:37:51.442 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj68?versionId=null 2023-07-29T00:37:51.442 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9NKpqEvghn43F6gvrI3DbL+896Y= 2023-07-29T00:37:51.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9NKpqEvghn43F6gvrI3DbL+896Y=', 'Content-Length': '0'} 2023-07-29T00:37:51.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8e9786fec21f1d4-0064c45f5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-07-29T00:37:51.445 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:37:51.446 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:37:51.446 DEBUG:boto:path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:37:51.446 DEBUG:boto:auth_path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:37:51.447 DEBUG:boto:Method: GET 2023-07-29T00:37:51.447 DEBUG:boto:Path: /hiiqos-59/obj68?versionId=null 2023-07-29T00:37:51.447 DEBUG:boto:Data: 2023-07-29T00:37:51.448 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.448 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:51.449 DEBUG:boto:Port: 8001 2023-07-29T00:37:51.449 DEBUG:boto:Params: {} 2023-07-29T00:37:51.449 DEBUG:boto:Token: None 2023-07-29T00:37:51.450 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj68?versionId=null 2023-07-29T00:37:51.450 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w6Poe6eDwGY73hHnOqfnGnEOWoY= 2023-07-29T00:37:51.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w6Poe6eDwGY73hHnOqfnGnEOWoY=', 'Content-Length': '0'} 2023-07-29T00:37:51.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018eb7c8c34ce4153-0064c45f5f-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.496 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:37:51.497 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:37:51.497 DEBUG:boto:path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:37:51.498 DEBUG:boto:auth_path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:37:51.498 DEBUG:boto:Method: GET 2023-07-29T00:37:51.499 DEBUG:boto:Path: /hiiqos-59/obj68?versionId=null 2023-07-29T00:37:51.499 DEBUG:boto:Data: 2023-07-29T00:37:51.499 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.500 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:51.500 DEBUG:boto:Port: 8000 2023-07-29T00:37:51.501 DEBUG:boto:Params: {} 2023-07-29T00:37:51.502 DEBUG:boto:Token: None 2023-07-29T00:37:51.503 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj68?versionId=null 2023-07-29T00:37:51.503 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w6Poe6eDwGY73hHnOqfnGnEOWoY= 2023-07-29T00:37:51.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w6Poe6eDwGY73hHnOqfnGnEOWoY=', 'Content-Length': '0'} 2023-07-29T00:37:51.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038673fcba37e921f-0064c45f5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.506 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:51.507 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:51.507 DEBUG:boto:path=/hiiqos-59/?prefix=obj68 2023-07-29T00:37:51.507 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj68 2023-07-29T00:37:51.508 DEBUG:boto:Method: GET 2023-07-29T00:37:51.508 DEBUG:boto:Path: /hiiqos-59/?prefix=obj68 2023-07-29T00:37:51.509 DEBUG:boto:Data: 2023-07-29T00:37:51.509 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.509 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:51.510 DEBUG:boto:Port: 8001 2023-07-29T00:37:51.510 DEBUG:boto:Params: {} 2023-07-29T00:37:51.510 DEBUG:boto:Token: None 2023-07-29T00:37:51.511 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/ 2023-07-29T00:37:51.511 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vNM5Y5SCpOQf7IWALA8ChXbEFm8= 2023-07-29T00:37:51.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vNM5Y5SCpOQf7IWALA8ChXbEFm8=', 'Content-Length': '0'} 2023-07-29T00:37:51.515 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d80ee3ef92f4dfd6-0064c45f5f-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.515 DEBUG:boto:b'hiiqos-59obj681000falseobj682023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:51.516 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:51.516 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:51.517 DEBUG:boto:path=/hiiqos-59/?prefix=obj68 2023-07-29T00:37:51.517 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj68 2023-07-29T00:37:51.517 DEBUG:boto:Method: GET 2023-07-29T00:37:51.518 DEBUG:boto:Path: /hiiqos-59/?prefix=obj68 2023-07-29T00:37:51.519 DEBUG:boto:Data: 2023-07-29T00:37:51.519 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.520 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:51.521 DEBUG:boto:Port: 8000 2023-07-29T00:37:51.522 DEBUG:boto:Params: {} 2023-07-29T00:37:51.523 DEBUG:boto:Token: None 2023-07-29T00:37:51.524 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/ 2023-07-29T00:37:51.524 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vNM5Y5SCpOQf7IWALA8ChXbEFm8= 2023-07-29T00:37:51.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vNM5Y5SCpOQf7IWALA8ChXbEFm8=', 'Content-Length': '0'} 2023-07-29T00:37:51.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000824c5f103ebafbd1-0064c45f5f-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.529 DEBUG:boto:b'hiiqos-59obj681000falseobj682023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:51.530 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:37:51.530 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:37:51.530 DEBUG:boto:Method: HEAD 2023-07-29T00:37:51.531 DEBUG:boto:Path: /hiiqos-59/obj68 2023-07-29T00:37:51.531 DEBUG:boto:Data: 2023-07-29T00:37:51.532 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.532 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:51.532 DEBUG:boto:Port: 8001 2023-07-29T00:37:51.533 DEBUG:boto:Params: {} 2023-07-29T00:37:51.533 DEBUG:boto:Token: None 2023-07-29T00:37:51.533 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj68 2023-07-29T00:37:51.534 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nMhi8lGvxW8IxobKDN3fqV2eXLQ= 2023-07-29T00:37:51.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nMhi8lGvxW8IxobKDN3fqV2eXLQ=', 'Content-Length': '0'} 2023-07-29T00:37:51.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a69bbbffda54b4e-0064c45f5f-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.537 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:37:51.537 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:37:51.538 DEBUG:boto:Method: HEAD 2023-07-29T00:37:51.538 DEBUG:boto:Path: /hiiqos-59/obj68 2023-07-29T00:37:51.539 DEBUG:boto:Data: 2023-07-29T00:37:51.539 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.539 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:51.540 DEBUG:boto:Port: 8000 2023-07-29T00:37:51.540 DEBUG:boto:Params: {} 2023-07-29T00:37:51.540 DEBUG:boto:Token: None 2023-07-29T00:37:51.541 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj68 2023-07-29T00:37:51.541 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nMhi8lGvxW8IxobKDN3fqV2eXLQ= 2023-07-29T00:37:51.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nMhi8lGvxW8IxobKDN3fqV2eXLQ=', 'Content-Length': '0'} 2023-07-29T00:37:51.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7d803c04e5fb2f9-0064c45f5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-07-29T00:37:51.544 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:37:51.545 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:37:51.545 DEBUG:boto:Method: GET 2023-07-29T00:37:51.546 DEBUG:boto:Path: /hiiqos-59/obj68 2023-07-29T00:37:51.546 DEBUG:boto:Data: 2023-07-29T00:37:51.546 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.547 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:51.547 DEBUG:boto:Port: 8001 2023-07-29T00:37:51.547 DEBUG:boto:Params: {} 2023-07-29T00:37:51.548 DEBUG:boto:Token: None 2023-07-29T00:37:51.549 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj68 2023-07-29T00:37:51.549 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dtbd2Fv9gtqnwrgfn+D+/rfLUAw= 2023-07-29T00:37:51.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dtbd2Fv9gtqnwrgfn+D+/rfLUAw=', 'Content-Length': '0'} 2023-07-29T00:37:51.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afa0a90bfaf4c2ce-0064c45f5f-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.553 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:37:51.554 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:37:51.555 DEBUG:boto:Method: GET 2023-07-29T00:37:51.556 DEBUG:boto:Path: /hiiqos-59/obj68 2023-07-29T00:37:51.557 DEBUG:boto:Data: 2023-07-29T00:37:51.558 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.559 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:51.560 DEBUG:boto:Port: 8000 2023-07-29T00:37:51.560 DEBUG:boto:Params: {} 2023-07-29T00:37:51.561 DEBUG:boto:Token: None 2023-07-29T00:37:51.562 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj68 2023-07-29T00:37:51.563 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dtbd2Fv9gtqnwrgfn+D+/rfLUAw= 2023-07-29T00:37:51.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dtbd2Fv9gtqnwrgfn+D+/rfLUAw=', 'Content-Length': '0'} 2023-07-29T00:37:51.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000970680c87bca39e9-0064c45f5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.608 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:51.609 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:51.609 DEBUG:boto:path=/hiiqos-59/?prefix=obj68 2023-07-29T00:37:51.610 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj68 2023-07-29T00:37:51.610 DEBUG:boto:Method: GET 2023-07-29T00:37:51.610 DEBUG:boto:Path: /hiiqos-59/?prefix=obj68 2023-07-29T00:37:51.611 DEBUG:boto:Data: 2023-07-29T00:37:51.611 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.612 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:51.612 DEBUG:boto:Port: 8001 2023-07-29T00:37:51.612 DEBUG:boto:Params: {} 2023-07-29T00:37:51.613 DEBUG:boto:Token: None 2023-07-29T00:37:51.613 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/ 2023-07-29T00:37:51.614 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vNM5Y5SCpOQf7IWALA8ChXbEFm8= 2023-07-29T00:37:51.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vNM5Y5SCpOQf7IWALA8ChXbEFm8=', 'Content-Length': '0'} 2023-07-29T00:37:51.616 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9dc0ce314375292-0064c45f5f-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.617 DEBUG:boto:b'hiiqos-59obj681000falseobj682023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:51.618 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:51.618 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:51.618 DEBUG:boto:path=/hiiqos-59/?prefix=obj68 2023-07-29T00:37:51.619 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj68 2023-07-29T00:37:51.619 DEBUG:boto:Method: GET 2023-07-29T00:37:51.620 DEBUG:boto:Path: /hiiqos-59/?prefix=obj68 2023-07-29T00:37:51.620 DEBUG:boto:Data: 2023-07-29T00:37:51.621 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.621 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:51.621 DEBUG:boto:Port: 8000 2023-07-29T00:37:51.622 DEBUG:boto:Params: {} 2023-07-29T00:37:51.622 DEBUG:boto:Token: None 2023-07-29T00:37:51.622 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/ 2023-07-29T00:37:51.623 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vNM5Y5SCpOQf7IWALA8ChXbEFm8= 2023-07-29T00:37:51.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vNM5Y5SCpOQf7IWALA8ChXbEFm8=', 'Content-Length': '0'} 2023-07-29T00:37:51.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086c53029dca77e46-0064c45f5f-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.628 DEBUG:boto:b'hiiqos-59obj681000falseobj682023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:51.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-07-29T00:37:51.629 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:37:51.629 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:37:51.630 DEBUG:boto:path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:37:51.630 DEBUG:boto:auth_path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:37:51.630 DEBUG:boto:Method: GET 2023-07-29T00:37:51.631 DEBUG:boto:Path: /hiiqos-59/obj69?versionId=null 2023-07-29T00:37:51.631 DEBUG:boto:Data: 2023-07-29T00:37:51.632 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.632 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:51.632 DEBUG:boto:Port: 8001 2023-07-29T00:37:51.633 DEBUG:boto:Params: {} 2023-07-29T00:37:51.633 DEBUG:boto:Token: None 2023-07-29T00:37:51.634 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj69?versionId=null 2023-07-29T00:37:51.634 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S/AULaY8Av0SpSoGjqkSa/laN8w= 2023-07-29T00:37:51.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S/AULaY8Av0SpSoGjqkSa/laN8w=', 'Content-Length': '0'} 2023-07-29T00:37:51.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f907e93788a4a21e-0064c45f5f-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.680 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:37:51.681 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:37:51.681 DEBUG:boto:path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:37:51.682 DEBUG:boto:auth_path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:37:51.682 DEBUG:boto:Method: GET 2023-07-29T00:37:51.682 DEBUG:boto:Path: /hiiqos-59/obj69?versionId=null 2023-07-29T00:37:51.683 DEBUG:boto:Data: 2023-07-29T00:37:51.683 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.683 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:51.684 DEBUG:boto:Port: 8000 2023-07-29T00:37:51.684 DEBUG:boto:Params: {} 2023-07-29T00:37:51.685 DEBUG:boto:Token: None 2023-07-29T00:37:51.685 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj69?versionId=null 2023-07-29T00:37:51.685 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S/AULaY8Av0SpSoGjqkSa/laN8w= 2023-07-29T00:37:51.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S/AULaY8Av0SpSoGjqkSa/laN8w=', 'Content-Length': '0'} 2023-07-29T00:37:51.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008607d52bc05b3618-0064c45f5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.689 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:37:51.689 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:37:51.689 DEBUG:boto:path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:37:51.690 DEBUG:boto:auth_path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:37:51.690 DEBUG:boto:Method: HEAD 2023-07-29T00:37:51.691 DEBUG:boto:Path: /hiiqos-59/obj69?versionId=null 2023-07-29T00:37:51.691 DEBUG:boto:Data: 2023-07-29T00:37:51.691 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.692 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:51.692 DEBUG:boto:Port: 8001 2023-07-29T00:37:51.692 DEBUG:boto:Params: {} 2023-07-29T00:37:51.693 DEBUG:boto:Token: None 2023-07-29T00:37:51.693 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj69?versionId=null 2023-07-29T00:37:51.693 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:75X+H+vkfHvZ+z4W0KOytPJr+To= 2023-07-29T00:37:51.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:75X+H+vkfHvZ+z4W0KOytPJr+To=', 'Content-Length': '0'} 2023-07-29T00:37:51.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e134a7980afef313-0064c45f5f-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.696 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:37:51.697 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:37:51.697 DEBUG:boto:path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:37:51.697 DEBUG:boto:auth_path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:37:51.698 DEBUG:boto:Method: HEAD 2023-07-29T00:37:51.698 DEBUG:boto:Path: /hiiqos-59/obj69?versionId=null 2023-07-29T00:37:51.698 DEBUG:boto:Data: 2023-07-29T00:37:51.699 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.699 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:51.699 DEBUG:boto:Port: 8000 2023-07-29T00:37:51.700 DEBUG:boto:Params: {} 2023-07-29T00:37:51.700 DEBUG:boto:Token: None 2023-07-29T00:37:51.701 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj69?versionId=null 2023-07-29T00:37:51.701 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:75X+H+vkfHvZ+z4W0KOytPJr+To= 2023-07-29T00:37:51.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:75X+H+vkfHvZ+z4W0KOytPJr+To=', 'Content-Length': '0'} 2023-07-29T00:37:51.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a341e540695eb925-0064c45f5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-07-29T00:37:51.704 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:37:51.705 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:37:51.705 DEBUG:boto:path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:37:51.705 DEBUG:boto:auth_path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:37:51.706 DEBUG:boto:Method: GET 2023-07-29T00:37:51.706 DEBUG:boto:Path: /hiiqos-59/obj69?versionId=null 2023-07-29T00:37:51.706 DEBUG:boto:Data: 2023-07-29T00:37:51.707 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.707 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:51.707 DEBUG:boto:Port: 8001 2023-07-29T00:37:51.708 DEBUG:boto:Params: {} 2023-07-29T00:37:51.708 DEBUG:boto:Token: None 2023-07-29T00:37:51.709 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj69?versionId=null 2023-07-29T00:37:51.709 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S/AULaY8Av0SpSoGjqkSa/laN8w= 2023-07-29T00:37:51.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S/AULaY8Av0SpSoGjqkSa/laN8w=', 'Content-Length': '0'} 2023-07-29T00:37:51.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023bcd0f97384bc93-0064c45f5f-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.712 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:37:51.713 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:37:51.713 DEBUG:boto:path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:37:51.713 DEBUG:boto:auth_path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:37:51.714 DEBUG:boto:Method: GET 2023-07-29T00:37:51.714 DEBUG:boto:Path: /hiiqos-59/obj69?versionId=null 2023-07-29T00:37:51.714 DEBUG:boto:Data: 2023-07-29T00:37:51.715 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.715 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:51.715 DEBUG:boto:Port: 8000 2023-07-29T00:37:51.716 DEBUG:boto:Params: {} 2023-07-29T00:37:51.716 DEBUG:boto:Token: None 2023-07-29T00:37:51.717 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj69?versionId=null 2023-07-29T00:37:51.717 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S/AULaY8Av0SpSoGjqkSa/laN8w= 2023-07-29T00:37:51.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S/AULaY8Av0SpSoGjqkSa/laN8w=', 'Content-Length': '0'} 2023-07-29T00:37:51.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082e8d5d87cbdebe7-0064c45f5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.720 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:51.720 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:51.721 DEBUG:boto:path=/hiiqos-59/?prefix=obj69 2023-07-29T00:37:51.721 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj69 2023-07-29T00:37:51.722 DEBUG:boto:Method: GET 2023-07-29T00:37:51.722 DEBUG:boto:Path: /hiiqos-59/?prefix=obj69 2023-07-29T00:37:51.722 DEBUG:boto:Data: 2023-07-29T00:37:51.723 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.723 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:51.723 DEBUG:boto:Port: 8001 2023-07-29T00:37:51.724 DEBUG:boto:Params: {} 2023-07-29T00:37:51.724 DEBUG:boto:Token: None 2023-07-29T00:37:51.724 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/ 2023-07-29T00:37:51.725 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vNM5Y5SCpOQf7IWALA8ChXbEFm8= 2023-07-29T00:37:51.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vNM5Y5SCpOQf7IWALA8ChXbEFm8=', 'Content-Length': '0'} 2023-07-29T00:37:51.728 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006917d16a110d7aae-0064c45f5f-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.729 DEBUG:boto:b'hiiqos-59obj691000falseobj692023-07-29T00:33:06.857Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:51.729 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:51.730 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:51.730 DEBUG:boto:path=/hiiqos-59/?prefix=obj69 2023-07-29T00:37:51.730 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj69 2023-07-29T00:37:51.731 DEBUG:boto:Method: GET 2023-07-29T00:37:51.731 DEBUG:boto:Path: /hiiqos-59/?prefix=obj69 2023-07-29T00:37:51.732 DEBUG:boto:Data: 2023-07-29T00:37:51.732 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.732 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:51.733 DEBUG:boto:Port: 8000 2023-07-29T00:37:51.733 DEBUG:boto:Params: {} 2023-07-29T00:37:51.733 DEBUG:boto:Token: None 2023-07-29T00:37:51.734 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/ 2023-07-29T00:37:51.734 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vNM5Y5SCpOQf7IWALA8ChXbEFm8= 2023-07-29T00:37:51.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vNM5Y5SCpOQf7IWALA8ChXbEFm8=', 'Content-Length': '0'} 2023-07-29T00:37:51.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0b1fa6746986cea-0064c45f5f-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.739 DEBUG:boto:b'hiiqos-59obj691000falseobj692023-07-29T00:33:06.857Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:51.740 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:37:51.740 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:37:51.741 DEBUG:boto:Method: HEAD 2023-07-29T00:37:51.741 DEBUG:boto:Path: /hiiqos-59/obj69 2023-07-29T00:37:51.741 DEBUG:boto:Data: 2023-07-29T00:37:51.742 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.742 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:51.742 DEBUG:boto:Port: 8001 2023-07-29T00:37:51.743 DEBUG:boto:Params: {} 2023-07-29T00:37:51.743 DEBUG:boto:Token: None 2023-07-29T00:37:51.743 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj69 2023-07-29T00:37:51.744 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DkoNtgbdw810ewWE2IktV4MxPaQ= 2023-07-29T00:37:51.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DkoNtgbdw810ewWE2IktV4MxPaQ=', 'Content-Length': '0'} 2023-07-29T00:37:51.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020a039936bc78240-0064c45f5f-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.747 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:37:51.747 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:37:51.748 DEBUG:boto:Method: HEAD 2023-07-29T00:37:51.748 DEBUG:boto:Path: /hiiqos-59/obj69 2023-07-29T00:37:51.748 DEBUG:boto:Data: 2023-07-29T00:37:51.749 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.749 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:51.749 DEBUG:boto:Port: 8000 2023-07-29T00:37:51.750 DEBUG:boto:Params: {} 2023-07-29T00:37:51.750 DEBUG:boto:Token: None 2023-07-29T00:37:51.750 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj69 2023-07-29T00:37:51.751 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DkoNtgbdw810ewWE2IktV4MxPaQ= 2023-07-29T00:37:51.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DkoNtgbdw810ewWE2IktV4MxPaQ=', 'Content-Length': '0'} 2023-07-29T00:37:51.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c231fba1b639b8dd-0064c45f5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-07-29T00:37:51.754 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:37:51.755 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:37:51.755 DEBUG:boto:Method: GET 2023-07-29T00:37:51.755 DEBUG:boto:Path: /hiiqos-59/obj69 2023-07-29T00:37:51.756 DEBUG:boto:Data: 2023-07-29T00:37:51.756 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.757 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:51.757 DEBUG:boto:Port: 8001 2023-07-29T00:37:51.757 DEBUG:boto:Params: {} 2023-07-29T00:37:51.758 DEBUG:boto:Token: None 2023-07-29T00:37:51.758 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj69 2023-07-29T00:37:51.759 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WBFboThDYlWvrnaGNXchvyuaut4= 2023-07-29T00:37:51.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WBFboThDYlWvrnaGNXchvyuaut4=', 'Content-Length': '0'} 2023-07-29T00:37:51.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086d8b6ba6028559f-0064c45f5f-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.764 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:37:51.764 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:37:51.765 DEBUG:boto:Method: GET 2023-07-29T00:37:51.765 DEBUG:boto:Path: /hiiqos-59/obj69 2023-07-29T00:37:51.765 DEBUG:boto:Data: 2023-07-29T00:37:51.766 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.766 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:51.766 DEBUG:boto:Port: 8000 2023-07-29T00:37:51.767 DEBUG:boto:Params: {} 2023-07-29T00:37:51.767 DEBUG:boto:Token: None 2023-07-29T00:37:51.768 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj69 2023-07-29T00:37:51.768 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WBFboThDYlWvrnaGNXchvyuaut4= 2023-07-29T00:37:51.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WBFboThDYlWvrnaGNXchvyuaut4=', 'Content-Length': '0'} 2023-07-29T00:37:51.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fcf3dc89e86cdea-0064c45f5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.812 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:51.813 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:51.813 DEBUG:boto:path=/hiiqos-59/?prefix=obj69 2023-07-29T00:37:51.814 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj69 2023-07-29T00:37:51.814 DEBUG:boto:Method: GET 2023-07-29T00:37:51.814 DEBUG:boto:Path: /hiiqos-59/?prefix=obj69 2023-07-29T00:37:51.815 DEBUG:boto:Data: 2023-07-29T00:37:51.815 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.815 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:51.816 DEBUG:boto:Port: 8001 2023-07-29T00:37:51.816 DEBUG:boto:Params: {} 2023-07-29T00:37:51.817 DEBUG:boto:Token: None 2023-07-29T00:37:51.817 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/ 2023-07-29T00:37:51.818 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vNM5Y5SCpOQf7IWALA8ChXbEFm8= 2023-07-29T00:37:51.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vNM5Y5SCpOQf7IWALA8ChXbEFm8=', 'Content-Length': '0'} 2023-07-29T00:37:51.821 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f324dce025cb242-0064c45f5f-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.821 DEBUG:boto:b'hiiqos-59obj691000falseobj692023-07-29T00:33:06.857Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:51.822 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:51.822 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:51.823 DEBUG:boto:path=/hiiqos-59/?prefix=obj69 2023-07-29T00:37:51.823 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj69 2023-07-29T00:37:51.823 DEBUG:boto:Method: GET 2023-07-29T00:37:51.824 DEBUG:boto:Path: /hiiqos-59/?prefix=obj69 2023-07-29T00:37:51.824 DEBUG:boto:Data: 2023-07-29T00:37:51.825 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.825 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:51.825 DEBUG:boto:Port: 8000 2023-07-29T00:37:51.826 DEBUG:boto:Params: {} 2023-07-29T00:37:51.826 DEBUG:boto:Token: None 2023-07-29T00:37:51.826 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/ 2023-07-29T00:37:51.827 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vNM5Y5SCpOQf7IWALA8ChXbEFm8= 2023-07-29T00:37:51.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vNM5Y5SCpOQf7IWALA8ChXbEFm8=', 'Content-Length': '0'} 2023-07-29T00:37:51.831 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000458f95b8151fbd22-0064c45f5f-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.832 DEBUG:boto:b'hiiqos-59obj691000falseobj692023-07-29T00:33:06.857Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:51.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-07-29T00:37:51.833 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:37:51.833 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:37:51.834 DEBUG:boto:path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:37:51.834 DEBUG:boto:auth_path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:37:51.835 DEBUG:boto:Method: GET 2023-07-29T00:37:51.835 DEBUG:boto:Path: /hiiqos-59/obj76?versionId=null 2023-07-29T00:37:51.835 DEBUG:boto:Data: 2023-07-29T00:37:51.836 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.836 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:51.837 DEBUG:boto:Port: 8001 2023-07-29T00:37:51.837 DEBUG:boto:Params: {} 2023-07-29T00:37:51.837 DEBUG:boto:Token: None 2023-07-29T00:37:51.838 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj76?versionId=null 2023-07-29T00:37:51.838 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:e0rsc4op5jiSupyOYqxILtUOb1s= 2023-07-29T00:37:51.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:e0rsc4op5jiSupyOYqxILtUOb1s=', 'Content-Length': '0'} 2023-07-29T00:37:51.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3510ade2e007979-0064c45f5f-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.841 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:37:51.842 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:37:51.842 DEBUG:boto:path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:37:51.842 DEBUG:boto:auth_path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:37:51.843 DEBUG:boto:Method: GET 2023-07-29T00:37:51.843 DEBUG:boto:Path: /hiiqos-59/obj76?versionId=null 2023-07-29T00:37:51.843 DEBUG:boto:Data: 2023-07-29T00:37:51.844 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.844 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:51.845 DEBUG:boto:Port: 8000 2023-07-29T00:37:51.845 DEBUG:boto:Params: {} 2023-07-29T00:37:51.845 DEBUG:boto:Token: None 2023-07-29T00:37:51.846 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj76?versionId=null 2023-07-29T00:37:51.846 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:e0rsc4op5jiSupyOYqxILtUOb1s= 2023-07-29T00:37:51.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:e0rsc4op5jiSupyOYqxILtUOb1s=', 'Content-Length': '0'} 2023-07-29T00:37:51.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005659f7787fa59180-0064c45f5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.850 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:37:51.850 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:37:51.850 DEBUG:boto:path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:37:51.851 DEBUG:boto:auth_path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:37:51.851 DEBUG:boto:Method: HEAD 2023-07-29T00:37:51.851 DEBUG:boto:Path: /hiiqos-59/obj76?versionId=null 2023-07-29T00:37:51.852 DEBUG:boto:Data: 2023-07-29T00:37:51.852 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.852 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:51.853 DEBUG:boto:Port: 8001 2023-07-29T00:37:51.853 DEBUG:boto:Params: {} 2023-07-29T00:37:51.853 DEBUG:boto:Token: None 2023-07-29T00:37:51.854 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj76?versionId=null 2023-07-29T00:37:51.854 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:NqGQlGNdqdBHVtIgZ1wiNIOiOZs= 2023-07-29T00:37:51.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:NqGQlGNdqdBHVtIgZ1wiNIOiOZs=', 'Content-Length': '0'} 2023-07-29T00:37:51.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3d25ea8014209ca-0064c45f5f-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.857 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:37:51.857 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:37:51.858 DEBUG:boto:path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:37:51.858 DEBUG:boto:auth_path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:37:51.858 DEBUG:boto:Method: HEAD 2023-07-29T00:37:51.859 DEBUG:boto:Path: /hiiqos-59/obj76?versionId=null 2023-07-29T00:37:51.859 DEBUG:boto:Data: 2023-07-29T00:37:51.859 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.860 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:51.860 DEBUG:boto:Port: 8000 2023-07-29T00:37:51.860 DEBUG:boto:Params: {} 2023-07-29T00:37:51.861 DEBUG:boto:Token: None 2023-07-29T00:37:51.861 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj76?versionId=null 2023-07-29T00:37:51.862 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:NqGQlGNdqdBHVtIgZ1wiNIOiOZs= 2023-07-29T00:37:51.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:NqGQlGNdqdBHVtIgZ1wiNIOiOZs=', 'Content-Length': '0'} 2023-07-29T00:37:51.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fa9803cc3a0a789-0064c45f5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-07-29T00:37:51.865 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:37:51.865 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:37:51.865 DEBUG:boto:path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:37:51.866 DEBUG:boto:auth_path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:37:51.866 DEBUG:boto:Method: GET 2023-07-29T00:37:51.867 DEBUG:boto:Path: /hiiqos-59/obj76?versionId=null 2023-07-29T00:37:51.867 DEBUG:boto:Data: 2023-07-29T00:37:51.867 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.868 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:51.868 DEBUG:boto:Port: 8001 2023-07-29T00:37:51.869 DEBUG:boto:Params: {} 2023-07-29T00:37:51.869 DEBUG:boto:Token: None 2023-07-29T00:37:51.869 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj76?versionId=null 2023-07-29T00:37:51.870 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:e0rsc4op5jiSupyOYqxILtUOb1s= 2023-07-29T00:37:51.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:e0rsc4op5jiSupyOYqxILtUOb1s=', 'Content-Length': '0'} 2023-07-29T00:37:51.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000203cfa7e05cf86c9-0064c45f5f-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.912 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:37:51.913 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:37:51.913 DEBUG:boto:path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:37:51.914 DEBUG:boto:auth_path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:37:51.914 DEBUG:boto:Method: GET 2023-07-29T00:37:51.915 DEBUG:boto:Path: /hiiqos-59/obj76?versionId=null 2023-07-29T00:37:51.915 DEBUG:boto:Data: 2023-07-29T00:37:51.915 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.916 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:51.916 DEBUG:boto:Port: 8000 2023-07-29T00:37:51.917 DEBUG:boto:Params: {} 2023-07-29T00:37:51.917 DEBUG:boto:Token: None 2023-07-29T00:37:51.918 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj76?versionId=null 2023-07-29T00:37:51.918 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:e0rsc4op5jiSupyOYqxILtUOb1s= 2023-07-29T00:37:51.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:e0rsc4op5jiSupyOYqxILtUOb1s=', 'Content-Length': '0'} 2023-07-29T00:37:51.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c4d355d70a106ad-0064c45f5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.922 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:51.922 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:51.922 DEBUG:boto:path=/hiiqos-59/?prefix=obj76 2023-07-29T00:37:51.923 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj76 2023-07-29T00:37:51.923 DEBUG:boto:Method: GET 2023-07-29T00:37:51.923 DEBUG:boto:Path: /hiiqos-59/?prefix=obj76 2023-07-29T00:37:51.924 DEBUG:boto:Data: 2023-07-29T00:37:51.924 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.925 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:51.925 DEBUG:boto:Port: 8001 2023-07-29T00:37:51.925 DEBUG:boto:Params: {} 2023-07-29T00:37:51.926 DEBUG:boto:Token: None 2023-07-29T00:37:51.926 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/ 2023-07-29T00:37:51.927 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vNM5Y5SCpOQf7IWALA8ChXbEFm8= 2023-07-29T00:37:51.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vNM5Y5SCpOQf7IWALA8ChXbEFm8=', 'Content-Length': '0'} 2023-07-29T00:37:51.930 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028d8d81845436c21-0064c45f5f-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.930 DEBUG:boto:b'hiiqos-59obj761000falseobj762023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:51.931 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:51.931 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:51.932 DEBUG:boto:path=/hiiqos-59/?prefix=obj76 2023-07-29T00:37:51.932 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj76 2023-07-29T00:37:51.933 DEBUG:boto:Method: GET 2023-07-29T00:37:51.933 DEBUG:boto:Path: /hiiqos-59/?prefix=obj76 2023-07-29T00:37:51.933 DEBUG:boto:Data: 2023-07-29T00:37:51.934 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.934 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:51.934 DEBUG:boto:Port: 8000 2023-07-29T00:37:51.935 DEBUG:boto:Params: {} 2023-07-29T00:37:51.935 DEBUG:boto:Token: None 2023-07-29T00:37:51.936 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/ 2023-07-29T00:37:51.936 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vNM5Y5SCpOQf7IWALA8ChXbEFm8= 2023-07-29T00:37:51.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vNM5Y5SCpOQf7IWALA8ChXbEFm8=', 'Content-Length': '0'} 2023-07-29T00:37:51.941 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002083a4e943718aff-0064c45f5f-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.942 DEBUG:boto:b'hiiqos-59obj761000falseobj762023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:51.942 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:37:51.943 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:37:51.943 DEBUG:boto:Method: HEAD 2023-07-29T00:37:51.943 DEBUG:boto:Path: /hiiqos-59/obj76 2023-07-29T00:37:51.944 DEBUG:boto:Data: 2023-07-29T00:37:51.944 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.944 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:51.945 DEBUG:boto:Port: 8001 2023-07-29T00:37:51.945 DEBUG:boto:Params: {} 2023-07-29T00:37:51.945 DEBUG:boto:Token: None 2023-07-29T00:37:51.946 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj76 2023-07-29T00:37:51.946 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rmtAM9VQEkmpF0LmmB5dek2z9ZY= 2023-07-29T00:37:51.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rmtAM9VQEkmpF0LmmB5dek2z9ZY=', 'Content-Length': '0'} 2023-07-29T00:37:51.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085097294ca5f706d-0064c45f5f-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.949 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:37:51.950 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:37:51.950 DEBUG:boto:Method: HEAD 2023-07-29T00:37:51.950 DEBUG:boto:Path: /hiiqos-59/obj76 2023-07-29T00:37:51.951 DEBUG:boto:Data: 2023-07-29T00:37:51.951 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.951 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:51.952 DEBUG:boto:Port: 8000 2023-07-29T00:37:51.952 DEBUG:boto:Params: {} 2023-07-29T00:37:51.953 DEBUG:boto:Token: None 2023-07-29T00:37:51.953 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj76 2023-07-29T00:37:51.953 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rmtAM9VQEkmpF0LmmB5dek2z9ZY= 2023-07-29T00:37:51.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rmtAM9VQEkmpF0LmmB5dek2z9ZY=', 'Content-Length': '0'} 2023-07-29T00:37:51.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ec7b6a098319bf3-0064c45f5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:51.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-07-29T00:37:51.957 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:37:51.957 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:37:51.958 DEBUG:boto:Method: GET 2023-07-29T00:37:51.959 DEBUG:boto:Path: /hiiqos-59/obj76 2023-07-29T00:37:51.960 DEBUG:boto:Data: 2023-07-29T00:37:51.961 DEBUG:boto:Headers: {} 2023-07-29T00:37:51.961 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:51.962 DEBUG:boto:Port: 8001 2023-07-29T00:37:51.963 DEBUG:boto:Params: {} 2023-07-29T00:37:51.964 DEBUG:boto:Token: None 2023-07-29T00:37:51.965 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:51 GMT /hiiqos-59/obj76 2023-07-29T00:37:51.966 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:43oa8fzL305Q7JxWHNWEUnFiFMg= 2023-07-29T00:37:51.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:43oa8fzL305Q7JxWHNWEUnFiFMg=', 'Content-Length': '0'} 2023-07-29T00:37:51.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c01eace13b7f6c2-0064c45f5f-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.012 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:37:52.013 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:37:52.013 DEBUG:boto:Method: GET 2023-07-29T00:37:52.014 DEBUG:boto:Path: /hiiqos-59/obj76 2023-07-29T00:37:52.014 DEBUG:boto:Data: 2023-07-29T00:37:52.014 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.015 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:52.015 DEBUG:boto:Port: 8000 2023-07-29T00:37:52.016 DEBUG:boto:Params: {} 2023-07-29T00:37:52.016 DEBUG:boto:Token: None 2023-07-29T00:37:52.016 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj76 2023-07-29T00:37:52.017 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:F5U0DdDwTeOZCdZ2UlUFJ+nH3gc= 2023-07-29T00:37:52.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:F5U0DdDwTeOZCdZ2UlUFJ+nH3gc=', 'Content-Length': '0'} 2023-07-29T00:37:52.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ae316735a23f1e4-0064c45f5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.020 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:52.021 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:52.021 DEBUG:boto:path=/hiiqos-59/?prefix=obj76 2023-07-29T00:37:52.021 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj76 2023-07-29T00:37:52.022 DEBUG:boto:Method: GET 2023-07-29T00:37:52.022 DEBUG:boto:Path: /hiiqos-59/?prefix=obj76 2023-07-29T00:37:52.023 DEBUG:boto:Data: 2023-07-29T00:37:52.023 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.023 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:52.024 DEBUG:boto:Port: 8001 2023-07-29T00:37:52.024 DEBUG:boto:Params: {} 2023-07-29T00:37:52.024 DEBUG:boto:Token: None 2023-07-29T00:37:52.025 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/ 2023-07-29T00:37:52.025 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2JwKdK0Oo0hX02BDmdjOqg50fzY= 2023-07-29T00:37:52.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2JwKdK0Oo0hX02BDmdjOqg50fzY=', 'Content-Length': '0'} 2023-07-29T00:37:52.028 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8e8da981b2ec963-0064c45f5f-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.029 DEBUG:boto:b'hiiqos-59obj761000falseobj762023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:52.029 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:52.030 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:52.030 DEBUG:boto:path=/hiiqos-59/?prefix=obj76 2023-07-29T00:37:52.030 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj76 2023-07-29T00:37:52.031 DEBUG:boto:Method: GET 2023-07-29T00:37:52.031 DEBUG:boto:Path: /hiiqos-59/?prefix=obj76 2023-07-29T00:37:52.032 DEBUG:boto:Data: 2023-07-29T00:37:52.032 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.032 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:52.033 DEBUG:boto:Port: 8000 2023-07-29T00:37:52.033 DEBUG:boto:Params: {} 2023-07-29T00:37:52.033 DEBUG:boto:Token: None 2023-07-29T00:37:52.034 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/ 2023-07-29T00:37:52.034 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2JwKdK0Oo0hX02BDmdjOqg50fzY= 2023-07-29T00:37:52.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2JwKdK0Oo0hX02BDmdjOqg50fzY=', 'Content-Length': '0'} 2023-07-29T00:37:52.039 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9ef53eeb7245bff-0064c45f5f-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.039 DEBUG:boto:b'hiiqos-59obj761000falseobj762023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:52.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-07-29T00:37:52.040 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:37:52.041 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:37:52.041 DEBUG:boto:path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:37:52.041 DEBUG:boto:auth_path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:37:52.042 DEBUG:boto:Method: GET 2023-07-29T00:37:52.042 DEBUG:boto:Path: /hiiqos-59/obj85?versionId=null 2023-07-29T00:37:52.042 DEBUG:boto:Data: 2023-07-29T00:37:52.043 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.043 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:52.043 DEBUG:boto:Port: 8001 2023-07-29T00:37:52.044 DEBUG:boto:Params: {} 2023-07-29T00:37:52.044 DEBUG:boto:Token: None 2023-07-29T00:37:52.045 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj85?versionId=null 2023-07-29T00:37:52.045 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9sqndIzzHBkCyow3Rlp6H3FVgB0= 2023-07-29T00:37:52.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9sqndIzzHBkCyow3Rlp6H3FVgB0=', 'Content-Length': '0'} 2023-07-29T00:37:52.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daa373eb57947554-0064c45f5f-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.048 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:37:52.048 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:37:52.049 DEBUG:boto:path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:37:52.049 DEBUG:boto:auth_path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:37:52.049 DEBUG:boto:Method: GET 2023-07-29T00:37:52.050 DEBUG:boto:Path: /hiiqos-59/obj85?versionId=null 2023-07-29T00:37:52.050 DEBUG:boto:Data: 2023-07-29T00:37:52.051 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.051 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:52.051 DEBUG:boto:Port: 8000 2023-07-29T00:37:52.052 DEBUG:boto:Params: {} 2023-07-29T00:37:52.052 DEBUG:boto:Token: None 2023-07-29T00:37:52.052 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj85?versionId=null 2023-07-29T00:37:52.053 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9sqndIzzHBkCyow3Rlp6H3FVgB0= 2023-07-29T00:37:52.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9sqndIzzHBkCyow3Rlp6H3FVgB0=', 'Content-Length': '0'} 2023-07-29T00:37:52.055 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6a86af3bb6d3334-0064c45f5f-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.096 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:37:52.097 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:37:52.097 DEBUG:boto:path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:37:52.097 DEBUG:boto:auth_path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:37:52.098 DEBUG:boto:Method: HEAD 2023-07-29T00:37:52.098 DEBUG:boto:Path: /hiiqos-59/obj85?versionId=null 2023-07-29T00:37:52.098 DEBUG:boto:Data: 2023-07-29T00:37:52.099 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.099 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:52.099 DEBUG:boto:Port: 8001 2023-07-29T00:37:52.100 DEBUG:boto:Params: {} 2023-07-29T00:37:52.100 DEBUG:boto:Token: None 2023-07-29T00:37:52.101 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj85?versionId=null 2023-07-29T00:37:52.101 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Jqlmx0GQe6tABtVm5L7ViIXHnSA= 2023-07-29T00:37:52.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Jqlmx0GQe6tABtVm5L7ViIXHnSA=', 'Content-Length': '0'} 2023-07-29T00:37:52.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095b582c22863772e-0064c45f60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.104 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:37:52.104 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:37:52.104 DEBUG:boto:path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:37:52.105 DEBUG:boto:auth_path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:37:52.105 DEBUG:boto:Method: HEAD 2023-07-29T00:37:52.105 DEBUG:boto:Path: /hiiqos-59/obj85?versionId=null 2023-07-29T00:37:52.106 DEBUG:boto:Data: 2023-07-29T00:37:52.106 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.106 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:52.107 DEBUG:boto:Port: 8000 2023-07-29T00:37:52.107 DEBUG:boto:Params: {} 2023-07-29T00:37:52.108 DEBUG:boto:Token: None 2023-07-29T00:37:52.108 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj85?versionId=null 2023-07-29T00:37:52.108 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Jqlmx0GQe6tABtVm5L7ViIXHnSA= 2023-07-29T00:37:52.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Jqlmx0GQe6tABtVm5L7ViIXHnSA=', 'Content-Length': '0'} 2023-07-29T00:37:52.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000662547751a5937d6-0064c45f60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-07-29T00:37:52.111 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:37:52.112 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:37:52.112 DEBUG:boto:path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:37:52.113 DEBUG:boto:auth_path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:37:52.113 DEBUG:boto:Method: GET 2023-07-29T00:37:52.113 DEBUG:boto:Path: /hiiqos-59/obj85?versionId=null 2023-07-29T00:37:52.114 DEBUG:boto:Data: 2023-07-29T00:37:52.114 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.115 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:52.115 DEBUG:boto:Port: 8001 2023-07-29T00:37:52.115 DEBUG:boto:Params: {} 2023-07-29T00:37:52.116 DEBUG:boto:Token: None 2023-07-29T00:37:52.116 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj85?versionId=null 2023-07-29T00:37:52.117 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9sqndIzzHBkCyow3Rlp6H3FVgB0= 2023-07-29T00:37:52.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9sqndIzzHBkCyow3Rlp6H3FVgB0=', 'Content-Length': '0'} 2023-07-29T00:37:52.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdf9842f18c85ae1-0064c45f60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.120 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:37:52.120 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:37:52.120 DEBUG:boto:path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:37:52.121 DEBUG:boto:auth_path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:37:52.121 DEBUG:boto:Method: GET 2023-07-29T00:37:52.122 DEBUG:boto:Path: /hiiqos-59/obj85?versionId=null 2023-07-29T00:37:52.122 DEBUG:boto:Data: 2023-07-29T00:37:52.122 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.123 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:52.123 DEBUG:boto:Port: 8000 2023-07-29T00:37:52.123 DEBUG:boto:Params: {} 2023-07-29T00:37:52.124 DEBUG:boto:Token: None 2023-07-29T00:37:52.124 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj85?versionId=null 2023-07-29T00:37:52.125 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9sqndIzzHBkCyow3Rlp6H3FVgB0= 2023-07-29T00:37:52.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9sqndIzzHBkCyow3Rlp6H3FVgB0=', 'Content-Length': '0'} 2023-07-29T00:37:52.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037c0cf0447abe149-0064c45f60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.127 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:52.128 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:52.128 DEBUG:boto:path=/hiiqos-59/?prefix=obj85 2023-07-29T00:37:52.129 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj85 2023-07-29T00:37:52.129 DEBUG:boto:Method: GET 2023-07-29T00:37:52.129 DEBUG:boto:Path: /hiiqos-59/?prefix=obj85 2023-07-29T00:37:52.130 DEBUG:boto:Data: 2023-07-29T00:37:52.130 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.131 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:52.131 DEBUG:boto:Port: 8001 2023-07-29T00:37:52.131 DEBUG:boto:Params: {} 2023-07-29T00:37:52.132 DEBUG:boto:Token: None 2023-07-29T00:37:52.132 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/ 2023-07-29T00:37:52.132 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2JwKdK0Oo0hX02BDmdjOqg50fzY= 2023-07-29T00:37:52.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2JwKdK0Oo0hX02BDmdjOqg50fzY=', 'Content-Length': '0'} 2023-07-29T00:37:52.135 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000034026924448212f-0064c45f60-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.136 DEBUG:boto:b'hiiqos-59obj851000falseobj852023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:52.136 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:52.137 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:52.137 DEBUG:boto:path=/hiiqos-59/?prefix=obj85 2023-07-29T00:37:52.137 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj85 2023-07-29T00:37:52.138 DEBUG:boto:Method: GET 2023-07-29T00:37:52.138 DEBUG:boto:Path: /hiiqos-59/?prefix=obj85 2023-07-29T00:37:52.138 DEBUG:boto:Data: 2023-07-29T00:37:52.139 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.139 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:52.140 DEBUG:boto:Port: 8000 2023-07-29T00:37:52.140 DEBUG:boto:Params: {} 2023-07-29T00:37:52.140 DEBUG:boto:Token: None 2023-07-29T00:37:52.141 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/ 2023-07-29T00:37:52.141 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2JwKdK0Oo0hX02BDmdjOqg50fzY= 2023-07-29T00:37:52.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2JwKdK0Oo0hX02BDmdjOqg50fzY=', 'Content-Length': '0'} 2023-07-29T00:37:52.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027bfc8d821374281-0064c45f60-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.146 DEBUG:boto:b'hiiqos-59obj851000falseobj852023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:52.146 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:37:52.147 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:37:52.147 DEBUG:boto:Method: HEAD 2023-07-29T00:37:52.147 DEBUG:boto:Path: /hiiqos-59/obj85 2023-07-29T00:37:52.148 DEBUG:boto:Data: 2023-07-29T00:37:52.148 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.148 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:52.149 DEBUG:boto:Port: 8001 2023-07-29T00:37:52.149 DEBUG:boto:Params: {} 2023-07-29T00:37:52.149 DEBUG:boto:Token: None 2023-07-29T00:37:52.150 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj85 2023-07-29T00:37:52.150 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S/aC+Lh9rxNblvWeWnAmfsRKjAA= 2023-07-29T00:37:52.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S/aC+Lh9rxNblvWeWnAmfsRKjAA=', 'Content-Length': '0'} 2023-07-29T00:37:52.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dafde4ebb98c6156-0064c45f60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.153 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:37:52.153 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:37:52.154 DEBUG:boto:Method: HEAD 2023-07-29T00:37:52.154 DEBUG:boto:Path: /hiiqos-59/obj85 2023-07-29T00:37:52.154 DEBUG:boto:Data: 2023-07-29T00:37:52.155 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.155 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:52.156 DEBUG:boto:Port: 8000 2023-07-29T00:37:52.156 DEBUG:boto:Params: {} 2023-07-29T00:37:52.156 DEBUG:boto:Token: None 2023-07-29T00:37:52.157 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj85 2023-07-29T00:37:52.157 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S/aC+Lh9rxNblvWeWnAmfsRKjAA= 2023-07-29T00:37:52.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S/aC+Lh9rxNblvWeWnAmfsRKjAA=', 'Content-Length': '0'} 2023-07-29T00:37:52.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8bbf8817682d433-0064c45f60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-07-29T00:37:52.160 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:37:52.161 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:37:52.161 DEBUG:boto:Method: GET 2023-07-29T00:37:52.161 DEBUG:boto:Path: /hiiqos-59/obj85 2023-07-29T00:37:52.162 DEBUG:boto:Data: 2023-07-29T00:37:52.162 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.163 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:52.163 DEBUG:boto:Port: 8001 2023-07-29T00:37:52.163 DEBUG:boto:Params: {} 2023-07-29T00:37:52.164 DEBUG:boto:Token: None 2023-07-29T00:37:52.164 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj85 2023-07-29T00:37:52.165 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UE/tYehIQrE4tJWvjn8/ZblAGqc= 2023-07-29T00:37:52.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UE/tYehIQrE4tJWvjn8/ZblAGqc=', 'Content-Length': '0'} 2023-07-29T00:37:52.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000595fa29945daef24-0064c45f60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.208 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:37:52.209 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:37:52.209 DEBUG:boto:Method: GET 2023-07-29T00:37:52.210 DEBUG:boto:Path: /hiiqos-59/obj85 2023-07-29T00:37:52.210 DEBUG:boto:Data: 2023-07-29T00:37:52.210 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.211 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:52.211 DEBUG:boto:Port: 8000 2023-07-29T00:37:52.211 DEBUG:boto:Params: {} 2023-07-29T00:37:52.212 DEBUG:boto:Token: None 2023-07-29T00:37:52.212 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj85 2023-07-29T00:37:52.213 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UE/tYehIQrE4tJWvjn8/ZblAGqc= 2023-07-29T00:37:52.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UE/tYehIQrE4tJWvjn8/ZblAGqc=', 'Content-Length': '0'} 2023-07-29T00:37:52.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7f1000cdeb32290-0064c45f60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.216 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:52.216 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:52.217 DEBUG:boto:path=/hiiqos-59/?prefix=obj85 2023-07-29T00:37:52.217 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj85 2023-07-29T00:37:52.217 DEBUG:boto:Method: GET 2023-07-29T00:37:52.218 DEBUG:boto:Path: /hiiqos-59/?prefix=obj85 2023-07-29T00:37:52.218 DEBUG:boto:Data: 2023-07-29T00:37:52.218 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.219 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:52.219 DEBUG:boto:Port: 8001 2023-07-29T00:37:52.219 DEBUG:boto:Params: {} 2023-07-29T00:37:52.220 DEBUG:boto:Token: None 2023-07-29T00:37:52.220 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/ 2023-07-29T00:37:52.221 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2JwKdK0Oo0hX02BDmdjOqg50fzY= 2023-07-29T00:37:52.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2JwKdK0Oo0hX02BDmdjOqg50fzY=', 'Content-Length': '0'} 2023-07-29T00:37:52.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001efdf524e9db2d22-0064c45f60-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.224 DEBUG:boto:b'hiiqos-59obj851000falseobj852023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:52.225 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:52.225 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:52.226 DEBUG:boto:path=/hiiqos-59/?prefix=obj85 2023-07-29T00:37:52.226 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj85 2023-07-29T00:37:52.227 DEBUG:boto:Method: GET 2023-07-29T00:37:52.227 DEBUG:boto:Path: /hiiqos-59/?prefix=obj85 2023-07-29T00:37:52.228 DEBUG:boto:Data: 2023-07-29T00:37:52.229 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.230 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:52.231 DEBUG:boto:Port: 8000 2023-07-29T00:37:52.232 DEBUG:boto:Params: {} 2023-07-29T00:37:52.233 DEBUG:boto:Token: None 2023-07-29T00:37:52.233 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/ 2023-07-29T00:37:52.234 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2JwKdK0Oo0hX02BDmdjOqg50fzY= 2023-07-29T00:37:52.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2JwKdK0Oo0hX02BDmdjOqg50fzY=', 'Content-Length': '0'} 2023-07-29T00:37:52.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d234517b88d5284d-0064c45f60-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.239 DEBUG:boto:b'hiiqos-59obj851000falseobj852023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:52.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-07-29T00:37:52.240 DEBUG:boto:path=/hiiqos-59/obj87 2023-07-29T00:37:52.241 DEBUG:boto:auth_path=/hiiqos-59/obj87 2023-07-29T00:37:52.241 DEBUG:boto:path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:37:52.242 DEBUG:boto:auth_path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:37:52.242 DEBUG:boto:Method: GET 2023-07-29T00:37:52.243 DEBUG:boto:Path: /hiiqos-59/obj87?versionId=null 2023-07-29T00:37:52.243 DEBUG:boto:Data: 2023-07-29T00:37:52.244 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.244 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:52.244 DEBUG:boto:Port: 8001 2023-07-29T00:37:52.245 DEBUG:boto:Params: {} 2023-07-29T00:37:52.245 DEBUG:boto:Token: None 2023-07-29T00:37:52.246 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj87?versionId=null 2023-07-29T00:37:52.246 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yRpCLx/fqrDnC7s3Wgnqd8dBvhI= 2023-07-29T00:37:52.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yRpCLx/fqrDnC7s3Wgnqd8dBvhI=', 'Content-Length': '0'} 2023-07-29T00:37:52.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac72fed431a6e35b-0064c45f60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.249 DEBUG:boto:path=/hiiqos-59/obj87 2023-07-29T00:37:52.249 DEBUG:boto:auth_path=/hiiqos-59/obj87 2023-07-29T00:37:52.250 DEBUG:boto:path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:37:52.250 DEBUG:boto:auth_path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:37:52.251 DEBUG:boto:Method: GET 2023-07-29T00:37:52.251 DEBUG:boto:Path: /hiiqos-59/obj87?versionId=null 2023-07-29T00:37:52.251 DEBUG:boto:Data: 2023-07-29T00:37:52.252 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.252 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:52.253 DEBUG:boto:Port: 8000 2023-07-29T00:37:52.253 DEBUG:boto:Params: {} 2023-07-29T00:37:52.253 DEBUG:boto:Token: None 2023-07-29T00:37:52.254 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj87?versionId=null 2023-07-29T00:37:52.254 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yRpCLx/fqrDnC7s3Wgnqd8dBvhI= 2023-07-29T00:37:52.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yRpCLx/fqrDnC7s3Wgnqd8dBvhI=', 'Content-Length': '0'} 2023-07-29T00:37:52.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6f185c6794807b8-0064c45f60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.300 DEBUG:boto:path=/hiiqos-59/obj87 2023-07-29T00:37:52.301 DEBUG:boto:auth_path=/hiiqos-59/obj87 2023-07-29T00:37:52.301 DEBUG:boto:path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:37:52.302 DEBUG:boto:auth_path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:37:52.302 DEBUG:boto:Method: HEAD 2023-07-29T00:37:52.302 DEBUG:boto:Path: /hiiqos-59/obj87?versionId=null 2023-07-29T00:37:52.303 DEBUG:boto:Data: 2023-07-29T00:37:52.303 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.303 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:52.304 DEBUG:boto:Port: 8001 2023-07-29T00:37:52.304 DEBUG:boto:Params: {} 2023-07-29T00:37:52.304 DEBUG:boto:Token: None 2023-07-29T00:37:52.305 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj87?versionId=null 2023-07-29T00:37:52.305 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eg7lSE7NlbNYlDGA6lJB76NFdl8= 2023-07-29T00:37:52.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eg7lSE7NlbNYlDGA6lJB76NFdl8=', 'Content-Length': '0'} 2023-07-29T00:37:52.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c87116cecf5fa147-0064c45f60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.308 DEBUG:boto:path=/hiiqos-59/obj87 2023-07-29T00:37:52.309 DEBUG:boto:auth_path=/hiiqos-59/obj87 2023-07-29T00:37:52.309 DEBUG:boto:path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:37:52.309 DEBUG:boto:auth_path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:37:52.310 DEBUG:boto:Method: HEAD 2023-07-29T00:37:52.310 DEBUG:boto:Path: /hiiqos-59/obj87?versionId=null 2023-07-29T00:37:52.310 DEBUG:boto:Data: 2023-07-29T00:37:52.311 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.311 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:52.311 DEBUG:boto:Port: 8000 2023-07-29T00:37:52.312 DEBUG:boto:Params: {} 2023-07-29T00:37:52.312 DEBUG:boto:Token: None 2023-07-29T00:37:52.313 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj87?versionId=null 2023-07-29T00:37:52.313 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eg7lSE7NlbNYlDGA6lJB76NFdl8= 2023-07-29T00:37:52.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eg7lSE7NlbNYlDGA6lJB76NFdl8=', 'Content-Length': '0'} 2023-07-29T00:37:52.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e90b06a18198505c-0064c45f60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-07-29T00:37:52.316 DEBUG:boto:path=/hiiqos-59/obj87 2023-07-29T00:37:52.317 DEBUG:boto:auth_path=/hiiqos-59/obj87 2023-07-29T00:37:52.317 DEBUG:boto:path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:37:52.318 DEBUG:boto:auth_path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:37:52.318 DEBUG:boto:Method: GET 2023-07-29T00:37:52.318 DEBUG:boto:Path: /hiiqos-59/obj87?versionId=null 2023-07-29T00:37:52.319 DEBUG:boto:Data: 2023-07-29T00:37:52.319 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.319 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:52.320 DEBUG:boto:Port: 8001 2023-07-29T00:37:52.320 DEBUG:boto:Params: {} 2023-07-29T00:37:52.321 DEBUG:boto:Token: None 2023-07-29T00:37:52.321 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj87?versionId=null 2023-07-29T00:37:52.321 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yRpCLx/fqrDnC7s3Wgnqd8dBvhI= 2023-07-29T00:37:52.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yRpCLx/fqrDnC7s3Wgnqd8dBvhI=', 'Content-Length': '0'} 2023-07-29T00:37:52.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b411ab4597c2d87b-0064c45f60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.324 DEBUG:boto:path=/hiiqos-59/obj87 2023-07-29T00:37:52.325 DEBUG:boto:auth_path=/hiiqos-59/obj87 2023-07-29T00:37:52.325 DEBUG:boto:path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:37:52.326 DEBUG:boto:auth_path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:37:52.326 DEBUG:boto:Method: GET 2023-07-29T00:37:52.326 DEBUG:boto:Path: /hiiqos-59/obj87?versionId=null 2023-07-29T00:37:52.327 DEBUG:boto:Data: 2023-07-29T00:37:52.327 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.328 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:52.328 DEBUG:boto:Port: 8000 2023-07-29T00:37:52.328 DEBUG:boto:Params: {} 2023-07-29T00:37:52.329 DEBUG:boto:Token: None 2023-07-29T00:37:52.329 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj87?versionId=null 2023-07-29T00:37:52.330 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yRpCLx/fqrDnC7s3Wgnqd8dBvhI= 2023-07-29T00:37:52.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yRpCLx/fqrDnC7s3Wgnqd8dBvhI=', 'Content-Length': '0'} 2023-07-29T00:37:52.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a386b62d342be1a-0064c45f60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.333 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:52.333 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:52.333 DEBUG:boto:path=/hiiqos-59/?prefix=obj87 2023-07-29T00:37:52.334 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj87 2023-07-29T00:37:52.334 DEBUG:boto:Method: GET 2023-07-29T00:37:52.335 DEBUG:boto:Path: /hiiqos-59/?prefix=obj87 2023-07-29T00:37:52.335 DEBUG:boto:Data: 2023-07-29T00:37:52.335 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.336 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:52.336 DEBUG:boto:Port: 8001 2023-07-29T00:37:52.336 DEBUG:boto:Params: {} 2023-07-29T00:37:52.337 DEBUG:boto:Token: None 2023-07-29T00:37:52.337 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/ 2023-07-29T00:37:52.338 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2JwKdK0Oo0hX02BDmdjOqg50fzY= 2023-07-29T00:37:52.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2JwKdK0Oo0hX02BDmdjOqg50fzY=', 'Content-Length': '0'} 2023-07-29T00:37:52.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049684aca966c8fed-0064c45f60-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.341 DEBUG:boto:b'hiiqos-59obj871000falseobj872023-07-29T00:35:01.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:52.342 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:52.342 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:52.343 DEBUG:boto:path=/hiiqos-59/?prefix=obj87 2023-07-29T00:37:52.343 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj87 2023-07-29T00:37:52.343 DEBUG:boto:Method: GET 2023-07-29T00:37:52.344 DEBUG:boto:Path: /hiiqos-59/?prefix=obj87 2023-07-29T00:37:52.344 DEBUG:boto:Data: 2023-07-29T00:37:52.344 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.345 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:52.345 DEBUG:boto:Port: 8000 2023-07-29T00:37:52.346 DEBUG:boto:Params: {} 2023-07-29T00:37:52.346 DEBUG:boto:Token: None 2023-07-29T00:37:52.346 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/ 2023-07-29T00:37:52.347 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2JwKdK0Oo0hX02BDmdjOqg50fzY= 2023-07-29T00:37:52.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2JwKdK0Oo0hX02BDmdjOqg50fzY=', 'Content-Length': '0'} 2023-07-29T00:37:52.352 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f02817a1efb2834d-0064c45f60-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.352 DEBUG:boto:b'hiiqos-59obj871000falseobj872023-07-29T00:35:01.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:52.353 DEBUG:boto:path=/hiiqos-59/obj87 2023-07-29T00:37:52.353 DEBUG:boto:auth_path=/hiiqos-59/obj87 2023-07-29T00:37:52.354 DEBUG:boto:Method: HEAD 2023-07-29T00:37:52.354 DEBUG:boto:Path: /hiiqos-59/obj87 2023-07-29T00:37:52.355 DEBUG:boto:Data: 2023-07-29T00:37:52.355 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.355 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:52.356 DEBUG:boto:Port: 8001 2023-07-29T00:37:52.356 DEBUG:boto:Params: {} 2023-07-29T00:37:52.356 DEBUG:boto:Token: None 2023-07-29T00:37:52.357 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj87 2023-07-29T00:37:52.357 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LAzI7qZxhoL9N53IUKhaQTfSQ9U= 2023-07-29T00:37:52.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LAzI7qZxhoL9N53IUKhaQTfSQ9U=', 'Content-Length': '0'} 2023-07-29T00:37:52.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f3695a4345c39c8-0064c45f60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.360 DEBUG:boto:path=/hiiqos-59/obj87 2023-07-29T00:37:52.361 DEBUG:boto:auth_path=/hiiqos-59/obj87 2023-07-29T00:37:52.361 DEBUG:boto:Method: HEAD 2023-07-29T00:37:52.361 DEBUG:boto:Path: /hiiqos-59/obj87 2023-07-29T00:37:52.362 DEBUG:boto:Data: 2023-07-29T00:37:52.362 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.363 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:52.363 DEBUG:boto:Port: 8000 2023-07-29T00:37:52.363 DEBUG:boto:Params: {} 2023-07-29T00:37:52.364 DEBUG:boto:Token: None 2023-07-29T00:37:52.364 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj87 2023-07-29T00:37:52.365 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LAzI7qZxhoL9N53IUKhaQTfSQ9U= 2023-07-29T00:37:52.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LAzI7qZxhoL9N53IUKhaQTfSQ9U=', 'Content-Length': '0'} 2023-07-29T00:37:52.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040d4f377b625d023-0064c45f60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-07-29T00:37:52.368 DEBUG:boto:path=/hiiqos-59/obj87 2023-07-29T00:37:52.369 DEBUG:boto:auth_path=/hiiqos-59/obj87 2023-07-29T00:37:52.369 DEBUG:boto:Method: GET 2023-07-29T00:37:52.369 DEBUG:boto:Path: /hiiqos-59/obj87 2023-07-29T00:37:52.370 DEBUG:boto:Data: 2023-07-29T00:37:52.370 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.371 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:52.371 DEBUG:boto:Port: 8001 2023-07-29T00:37:52.371 DEBUG:boto:Params: {} 2023-07-29T00:37:52.372 DEBUG:boto:Token: None 2023-07-29T00:37:52.372 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj87 2023-07-29T00:37:52.373 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/lO7NQiHwdOqGA2LvZwyQwNnMao= 2023-07-29T00:37:52.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/lO7NQiHwdOqGA2LvZwyQwNnMao=', 'Content-Length': '0'} 2023-07-29T00:37:52.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a87794e6e3c5f9c4-0064c45f60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.376 DEBUG:boto:path=/hiiqos-59/obj87 2023-07-29T00:37:52.376 DEBUG:boto:auth_path=/hiiqos-59/obj87 2023-07-29T00:37:52.377 DEBUG:boto:Method: GET 2023-07-29T00:37:52.377 DEBUG:boto:Path: /hiiqos-59/obj87 2023-07-29T00:37:52.377 DEBUG:boto:Data: 2023-07-29T00:37:52.378 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.378 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:52.378 DEBUG:boto:Port: 8000 2023-07-29T00:37:52.379 DEBUG:boto:Params: {} 2023-07-29T00:37:52.379 DEBUG:boto:Token: None 2023-07-29T00:37:52.380 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj87 2023-07-29T00:37:52.380 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/lO7NQiHwdOqGA2LvZwyQwNnMao= 2023-07-29T00:37:52.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/lO7NQiHwdOqGA2LvZwyQwNnMao=', 'Content-Length': '0'} 2023-07-29T00:37:52.383 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f93b72137697639-0064c45f60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.383 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:52.384 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:52.384 DEBUG:boto:path=/hiiqos-59/?prefix=obj87 2023-07-29T00:37:52.385 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj87 2023-07-29T00:37:52.385 DEBUG:boto:Method: GET 2023-07-29T00:37:52.385 DEBUG:boto:Path: /hiiqos-59/?prefix=obj87 2023-07-29T00:37:52.386 DEBUG:boto:Data: 2023-07-29T00:37:52.386 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.387 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:52.387 DEBUG:boto:Port: 8001 2023-07-29T00:37:52.387 DEBUG:boto:Params: {} 2023-07-29T00:37:52.388 DEBUG:boto:Token: None 2023-07-29T00:37:52.388 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/ 2023-07-29T00:37:52.389 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2JwKdK0Oo0hX02BDmdjOqg50fzY= 2023-07-29T00:37:52.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2JwKdK0Oo0hX02BDmdjOqg50fzY=', 'Content-Length': '0'} 2023-07-29T00:37:52.392 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c5568e3655e2191-0064c45f60-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.392 DEBUG:boto:b'hiiqos-59obj871000falseobj872023-07-29T00:35:01.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:52.393 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:52.394 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:52.394 DEBUG:boto:path=/hiiqos-59/?prefix=obj87 2023-07-29T00:37:52.395 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj87 2023-07-29T00:37:52.396 DEBUG:boto:Method: GET 2023-07-29T00:37:52.396 DEBUG:boto:Path: /hiiqos-59/?prefix=obj87 2023-07-29T00:37:52.397 DEBUG:boto:Data: 2023-07-29T00:37:52.398 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.399 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:52.399 DEBUG:boto:Port: 8000 2023-07-29T00:37:52.400 DEBUG:boto:Params: {} 2023-07-29T00:37:52.401 DEBUG:boto:Token: None 2023-07-29T00:37:52.402 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/ 2023-07-29T00:37:52.402 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2JwKdK0Oo0hX02BDmdjOqg50fzY= 2023-07-29T00:37:52.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2JwKdK0Oo0hX02BDmdjOqg50fzY=', 'Content-Length': '0'} 2023-07-29T00:37:52.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c37d275f8d6ffc4-0064c45f60-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.407 DEBUG:boto:b'hiiqos-59obj871000falseobj872023-07-29T00:35:01.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:52.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-07-29T00:37:52.408 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:37:52.409 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:37:52.409 DEBUG:boto:path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:37:52.409 DEBUG:boto:auth_path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:37:52.410 DEBUG:boto:Method: GET 2023-07-29T00:37:52.410 DEBUG:boto:Path: /hiiqos-59/obj92?versionId=null 2023-07-29T00:37:52.411 DEBUG:boto:Data: 2023-07-29T00:37:52.411 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.411 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:52.412 DEBUG:boto:Port: 8001 2023-07-29T00:37:52.412 DEBUG:boto:Params: {} 2023-07-29T00:37:52.412 DEBUG:boto:Token: None 2023-07-29T00:37:52.413 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj92?versionId=null 2023-07-29T00:37:52.413 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RDQ7ZS93tD5aYt4bFHIKSZ6iQxE= 2023-07-29T00:37:52.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RDQ7ZS93tD5aYt4bFHIKSZ6iQxE=', 'Content-Length': '0'} 2023-07-29T00:37:52.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cead173e50f366e-0064c45f60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.416 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:37:52.417 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:37:52.417 DEBUG:boto:path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:37:52.418 DEBUG:boto:auth_path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:37:52.418 DEBUG:boto:Method: GET 2023-07-29T00:37:52.419 DEBUG:boto:Path: /hiiqos-59/obj92?versionId=null 2023-07-29T00:37:52.419 DEBUG:boto:Data: 2023-07-29T00:37:52.420 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.421 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:52.421 DEBUG:boto:Port: 8000 2023-07-29T00:37:52.422 DEBUG:boto:Params: {} 2023-07-29T00:37:52.422 DEBUG:boto:Token: None 2023-07-29T00:37:52.422 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj92?versionId=null 2023-07-29T00:37:52.423 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RDQ7ZS93tD5aYt4bFHIKSZ6iQxE= 2023-07-29T00:37:52.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RDQ7ZS93tD5aYt4bFHIKSZ6iQxE=', 'Content-Length': '0'} 2023-07-29T00:37:52.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060af64efb35b6725-0064c45f60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.468 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:37:52.469 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:37:52.469 DEBUG:boto:path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:37:52.470 DEBUG:boto:auth_path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:37:52.470 DEBUG:boto:Method: HEAD 2023-07-29T00:37:52.470 DEBUG:boto:Path: /hiiqos-59/obj92?versionId=null 2023-07-29T00:37:52.471 DEBUG:boto:Data: 2023-07-29T00:37:52.471 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.471 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:52.472 DEBUG:boto:Port: 8001 2023-07-29T00:37:52.472 DEBUG:boto:Params: {} 2023-07-29T00:37:52.472 DEBUG:boto:Token: None 2023-07-29T00:37:52.473 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj92?versionId=null 2023-07-29T00:37:52.473 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kEKhADnMPuYBRiDFNPHZfZvxVzo= 2023-07-29T00:37:52.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kEKhADnMPuYBRiDFNPHZfZvxVzo=', 'Content-Length': '0'} 2023-07-29T00:37:52.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba15d135f45fe7ac-0064c45f60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.476 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:37:52.477 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:37:52.477 DEBUG:boto:path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:37:52.477 DEBUG:boto:auth_path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:37:52.478 DEBUG:boto:Method: HEAD 2023-07-29T00:37:52.478 DEBUG:boto:Path: /hiiqos-59/obj92?versionId=null 2023-07-29T00:37:52.478 DEBUG:boto:Data: 2023-07-29T00:37:52.479 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.479 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:52.479 DEBUG:boto:Port: 8000 2023-07-29T00:37:52.480 DEBUG:boto:Params: {} 2023-07-29T00:37:52.480 DEBUG:boto:Token: None 2023-07-29T00:37:52.480 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj92?versionId=null 2023-07-29T00:37:52.481 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kEKhADnMPuYBRiDFNPHZfZvxVzo= 2023-07-29T00:37:52.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kEKhADnMPuYBRiDFNPHZfZvxVzo=', 'Content-Length': '0'} 2023-07-29T00:37:52.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccff803c807d69c0-0064c45f60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-07-29T00:37:52.484 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:37:52.484 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:37:52.485 DEBUG:boto:path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:37:52.485 DEBUG:boto:auth_path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:37:52.485 DEBUG:boto:Method: GET 2023-07-29T00:37:52.486 DEBUG:boto:Path: /hiiqos-59/obj92?versionId=null 2023-07-29T00:37:52.486 DEBUG:boto:Data: 2023-07-29T00:37:52.487 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.487 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:52.487 DEBUG:boto:Port: 8001 2023-07-29T00:37:52.488 DEBUG:boto:Params: {} 2023-07-29T00:37:52.488 DEBUG:boto:Token: None 2023-07-29T00:37:52.488 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj92?versionId=null 2023-07-29T00:37:52.489 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RDQ7ZS93tD5aYt4bFHIKSZ6iQxE= 2023-07-29T00:37:52.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RDQ7ZS93tD5aYt4bFHIKSZ6iQxE=', 'Content-Length': '0'} 2023-07-29T00:37:52.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9ca336b0cc55947-0064c45f60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.492 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:37:52.492 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:37:52.493 DEBUG:boto:path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:37:52.493 DEBUG:boto:auth_path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:37:52.494 DEBUG:boto:Method: GET 2023-07-29T00:37:52.494 DEBUG:boto:Path: /hiiqos-59/obj92?versionId=null 2023-07-29T00:37:52.494 DEBUG:boto:Data: 2023-07-29T00:37:52.495 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.495 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:52.495 DEBUG:boto:Port: 8000 2023-07-29T00:37:52.496 DEBUG:boto:Params: {} 2023-07-29T00:37:52.496 DEBUG:boto:Token: None 2023-07-29T00:37:52.497 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj92?versionId=null 2023-07-29T00:37:52.497 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RDQ7ZS93tD5aYt4bFHIKSZ6iQxE= 2023-07-29T00:37:52.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RDQ7ZS93tD5aYt4bFHIKSZ6iQxE=', 'Content-Length': '0'} 2023-07-29T00:37:52.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1248538e4be76f6-0064c45f60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.500 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:52.501 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:52.501 DEBUG:boto:path=/hiiqos-59/?prefix=obj92 2023-07-29T00:37:52.501 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj92 2023-07-29T00:37:52.502 DEBUG:boto:Method: GET 2023-07-29T00:37:52.502 DEBUG:boto:Path: /hiiqos-59/?prefix=obj92 2023-07-29T00:37:52.502 DEBUG:boto:Data: 2023-07-29T00:37:52.503 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.503 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:52.504 DEBUG:boto:Port: 8001 2023-07-29T00:37:52.504 DEBUG:boto:Params: {} 2023-07-29T00:37:52.504 DEBUG:boto:Token: None 2023-07-29T00:37:52.505 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/ 2023-07-29T00:37:52.505 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2JwKdK0Oo0hX02BDmdjOqg50fzY= 2023-07-29T00:37:52.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2JwKdK0Oo0hX02BDmdjOqg50fzY=', 'Content-Length': '0'} 2023-07-29T00:37:52.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a39d92306d5d86e3-0064c45f60-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.509 DEBUG:boto:b'hiiqos-59obj921000falseobj922023-07-29T00:33:06.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:52.509 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:52.510 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:52.510 DEBUG:boto:path=/hiiqos-59/?prefix=obj92 2023-07-29T00:37:52.510 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj92 2023-07-29T00:37:52.511 DEBUG:boto:Method: GET 2023-07-29T00:37:52.511 DEBUG:boto:Path: /hiiqos-59/?prefix=obj92 2023-07-29T00:37:52.511 DEBUG:boto:Data: 2023-07-29T00:37:52.512 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.512 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:52.513 DEBUG:boto:Port: 8000 2023-07-29T00:37:52.513 DEBUG:boto:Params: {} 2023-07-29T00:37:52.513 DEBUG:boto:Token: None 2023-07-29T00:37:52.514 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/ 2023-07-29T00:37:52.514 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2JwKdK0Oo0hX02BDmdjOqg50fzY= 2023-07-29T00:37:52.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2JwKdK0Oo0hX02BDmdjOqg50fzY=', 'Content-Length': '0'} 2023-07-29T00:37:52.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fad4e3f37233524f-0064c45f60-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.519 DEBUG:boto:b'hiiqos-59obj921000falseobj922023-07-29T00:33:06.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:52.519 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:37:52.520 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:37:52.520 DEBUG:boto:Method: HEAD 2023-07-29T00:37:52.520 DEBUG:boto:Path: /hiiqos-59/obj92 2023-07-29T00:37:52.521 DEBUG:boto:Data: 2023-07-29T00:37:52.521 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.521 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:52.522 DEBUG:boto:Port: 8001 2023-07-29T00:37:52.522 DEBUG:boto:Params: {} 2023-07-29T00:37:52.522 DEBUG:boto:Token: None 2023-07-29T00:37:52.523 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj92 2023-07-29T00:37:52.523 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:U/dJDVydxuGvLvg4pq2PK+n7FCQ= 2023-07-29T00:37:52.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:U/dJDVydxuGvLvg4pq2PK+n7FCQ=', 'Content-Length': '0'} 2023-07-29T00:37:52.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074fb6efefa7c52f2-0064c45f60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.526 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:37:52.526 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:37:52.527 DEBUG:boto:Method: HEAD 2023-07-29T00:37:52.527 DEBUG:boto:Path: /hiiqos-59/obj92 2023-07-29T00:37:52.527 DEBUG:boto:Data: 2023-07-29T00:37:52.528 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.528 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:52.528 DEBUG:boto:Port: 8000 2023-07-29T00:37:52.529 DEBUG:boto:Params: {} 2023-07-29T00:37:52.529 DEBUG:boto:Token: None 2023-07-29T00:37:52.529 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj92 2023-07-29T00:37:52.530 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:U/dJDVydxuGvLvg4pq2PK+n7FCQ= 2023-07-29T00:37:52.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:U/dJDVydxuGvLvg4pq2PK+n7FCQ=', 'Content-Length': '0'} 2023-07-29T00:37:52.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e00678361179d87-0064c45f60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.532 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-07-29T00:37:52.533 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:37:52.533 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:37:52.534 DEBUG:boto:Method: GET 2023-07-29T00:37:52.534 DEBUG:boto:Path: /hiiqos-59/obj92 2023-07-29T00:37:52.534 DEBUG:boto:Data: 2023-07-29T00:37:52.535 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.535 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:52.535 DEBUG:boto:Port: 8001 2023-07-29T00:37:52.536 DEBUG:boto:Params: {} 2023-07-29T00:37:52.536 DEBUG:boto:Token: None 2023-07-29T00:37:52.537 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj92 2023-07-29T00:37:52.537 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8tu/7qCDNaaZB45oCt3EQrc74Pk= 2023-07-29T00:37:52.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8tu/7qCDNaaZB45oCt3EQrc74Pk=', 'Content-Length': '0'} 2023-07-29T00:37:52.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcbcefed3d67b888-0064c45f60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.580 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:37:52.581 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:37:52.581 DEBUG:boto:Method: GET 2023-07-29T00:37:52.582 DEBUG:boto:Path: /hiiqos-59/obj92 2023-07-29T00:37:52.582 DEBUG:boto:Data: 2023-07-29T00:37:52.582 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.583 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:52.583 DEBUG:boto:Port: 8000 2023-07-29T00:37:52.583 DEBUG:boto:Params: {} 2023-07-29T00:37:52.584 DEBUG:boto:Token: None 2023-07-29T00:37:52.584 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj92 2023-07-29T00:37:52.585 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8tu/7qCDNaaZB45oCt3EQrc74Pk= 2023-07-29T00:37:52.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8tu/7qCDNaaZB45oCt3EQrc74Pk=', 'Content-Length': '0'} 2023-07-29T00:37:52.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a295c348a23b406a-0064c45f60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.588 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:52.588 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:52.589 DEBUG:boto:path=/hiiqos-59/?prefix=obj92 2023-07-29T00:37:52.589 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj92 2023-07-29T00:37:52.590 DEBUG:boto:Method: GET 2023-07-29T00:37:52.590 DEBUG:boto:Path: /hiiqos-59/?prefix=obj92 2023-07-29T00:37:52.590 DEBUG:boto:Data: 2023-07-29T00:37:52.591 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.591 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:52.591 DEBUG:boto:Port: 8001 2023-07-29T00:37:52.592 DEBUG:boto:Params: {} 2023-07-29T00:37:52.592 DEBUG:boto:Token: None 2023-07-29T00:37:52.593 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/ 2023-07-29T00:37:52.593 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2JwKdK0Oo0hX02BDmdjOqg50fzY= 2023-07-29T00:37:52.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2JwKdK0Oo0hX02BDmdjOqg50fzY=', 'Content-Length': '0'} 2023-07-29T00:37:52.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5a5f8f11060a3e3-0064c45f60-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.596 DEBUG:boto:b'hiiqos-59obj921000falseobj922023-07-29T00:33:06.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:52.597 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:52.597 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:52.598 DEBUG:boto:path=/hiiqos-59/?prefix=obj92 2023-07-29T00:37:52.598 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj92 2023-07-29T00:37:52.598 DEBUG:boto:Method: GET 2023-07-29T00:37:52.599 DEBUG:boto:Path: /hiiqos-59/?prefix=obj92 2023-07-29T00:37:52.599 DEBUG:boto:Data: 2023-07-29T00:37:52.600 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.600 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:52.600 DEBUG:boto:Port: 8000 2023-07-29T00:37:52.601 DEBUG:boto:Params: {} 2023-07-29T00:37:52.601 DEBUG:boto:Token: None 2023-07-29T00:37:52.601 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/ 2023-07-29T00:37:52.602 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2JwKdK0Oo0hX02BDmdjOqg50fzY= 2023-07-29T00:37:52.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2JwKdK0Oo0hX02BDmdjOqg50fzY=', 'Content-Length': '0'} 2023-07-29T00:37:52.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c733f788acb3c3c6-0064c45f60-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.607 DEBUG:boto:b'hiiqos-59obj921000falseobj922023-07-29T00:33:06.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:52.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-07-29T00:37:52.608 DEBUG:boto:path=/hiiqos-59/obj93 2023-07-29T00:37:52.608 DEBUG:boto:auth_path=/hiiqos-59/obj93 2023-07-29T00:37:52.608 DEBUG:boto:path=/hiiqos-59/obj93?versionId=null 2023-07-29T00:37:52.609 DEBUG:boto:auth_path=/hiiqos-59/obj93?versionId=null 2023-07-29T00:37:52.609 DEBUG:boto:Method: GET 2023-07-29T00:37:52.610 DEBUG:boto:Path: /hiiqos-59/obj93?versionId=null 2023-07-29T00:37:52.610 DEBUG:boto:Data: 2023-07-29T00:37:52.610 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.611 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:52.611 DEBUG:boto:Port: 8001 2023-07-29T00:37:52.611 DEBUG:boto:Params: {} 2023-07-29T00:37:52.612 DEBUG:boto:Token: None 2023-07-29T00:37:52.612 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj93?versionId=null 2023-07-29T00:37:52.613 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0fSpk0XrZiavTxIwQVLVl28JJs8= 2023-07-29T00:37:52.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0fSpk0XrZiavTxIwQVLVl28JJs8=', 'Content-Length': '0'} 2023-07-29T00:37:52.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad06a480ea1c4ef5-0064c45f60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.616 DEBUG:boto:path=/hiiqos-59/obj93 2023-07-29T00:37:52.616 DEBUG:boto:auth_path=/hiiqos-59/obj93 2023-07-29T00:37:52.617 DEBUG:boto:path=/hiiqos-59/obj93?versionId=null 2023-07-29T00:37:52.617 DEBUG:boto:auth_path=/hiiqos-59/obj93?versionId=null 2023-07-29T00:37:52.617 DEBUG:boto:Method: GET 2023-07-29T00:37:52.618 DEBUG:boto:Path: /hiiqos-59/obj93?versionId=null 2023-07-29T00:37:52.618 DEBUG:boto:Data: 2023-07-29T00:37:52.618 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.619 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:52.619 DEBUG:boto:Port: 8000 2023-07-29T00:37:52.619 DEBUG:boto:Params: {} 2023-07-29T00:37:52.620 DEBUG:boto:Token: None 2023-07-29T00:37:52.620 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj93?versionId=null 2023-07-29T00:37:52.621 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0fSpk0XrZiavTxIwQVLVl28JJs8= 2023-07-29T00:37:52.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0fSpk0XrZiavTxIwQVLVl28JJs8=', 'Content-Length': '0'} 2023-07-29T00:37:52.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea0fc02637e4bf34-0064c45f60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.664 DEBUG:boto:path=/hiiqos-59/obj93 2023-07-29T00:37:52.665 DEBUG:boto:auth_path=/hiiqos-59/obj93 2023-07-29T00:37:52.665 DEBUG:boto:path=/hiiqos-59/obj93?versionId=null 2023-07-29T00:37:52.666 DEBUG:boto:auth_path=/hiiqos-59/obj93?versionId=null 2023-07-29T00:37:52.666 DEBUG:boto:Method: HEAD 2023-07-29T00:37:52.666 DEBUG:boto:Path: /hiiqos-59/obj93?versionId=null 2023-07-29T00:37:52.667 DEBUG:boto:Data: 2023-07-29T00:37:52.667 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.667 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:52.668 DEBUG:boto:Port: 8001 2023-07-29T00:37:52.668 DEBUG:boto:Params: {} 2023-07-29T00:37:52.668 DEBUG:boto:Token: None 2023-07-29T00:37:52.669 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj93?versionId=null 2023-07-29T00:37:52.669 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KyeFCOtFA86gm5tfGjcuZSrYKeY= 2023-07-29T00:37:52.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KyeFCOtFA86gm5tfGjcuZSrYKeY=', 'Content-Length': '0'} 2023-07-29T00:37:52.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff804ac5815c85cd-0064c45f60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.672 DEBUG:boto:path=/hiiqos-59/obj93 2023-07-29T00:37:52.672 DEBUG:boto:auth_path=/hiiqos-59/obj93 2023-07-29T00:37:52.673 DEBUG:boto:path=/hiiqos-59/obj93?versionId=null 2023-07-29T00:37:52.673 DEBUG:boto:auth_path=/hiiqos-59/obj93?versionId=null 2023-07-29T00:37:52.673 DEBUG:boto:Method: HEAD 2023-07-29T00:37:52.674 DEBUG:boto:Path: /hiiqos-59/obj93?versionId=null 2023-07-29T00:37:52.674 DEBUG:boto:Data: 2023-07-29T00:37:52.674 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.675 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:52.675 DEBUG:boto:Port: 8000 2023-07-29T00:37:52.675 DEBUG:boto:Params: {} 2023-07-29T00:37:52.676 DEBUG:boto:Token: None 2023-07-29T00:37:52.676 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj93?versionId=null 2023-07-29T00:37:52.677 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KyeFCOtFA86gm5tfGjcuZSrYKeY= 2023-07-29T00:37:52.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KyeFCOtFA86gm5tfGjcuZSrYKeY=', 'Content-Length': '0'} 2023-07-29T00:37:52.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b6a03912c78da54-0064c45f60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-07-29T00:37:52.680 DEBUG:boto:path=/hiiqos-59/obj93 2023-07-29T00:37:52.681 DEBUG:boto:auth_path=/hiiqos-59/obj93 2023-07-29T00:37:52.681 DEBUG:boto:path=/hiiqos-59/obj93?versionId=null 2023-07-29T00:37:52.681 DEBUG:boto:auth_path=/hiiqos-59/obj93?versionId=null 2023-07-29T00:37:52.682 DEBUG:boto:Method: GET 2023-07-29T00:37:52.682 DEBUG:boto:Path: /hiiqos-59/obj93?versionId=null 2023-07-29T00:37:52.682 DEBUG:boto:Data: 2023-07-29T00:37:52.683 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.683 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:52.683 DEBUG:boto:Port: 8001 2023-07-29T00:37:52.684 DEBUG:boto:Params: {} 2023-07-29T00:37:52.684 DEBUG:boto:Token: None 2023-07-29T00:37:52.685 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj93?versionId=null 2023-07-29T00:37:52.685 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0fSpk0XrZiavTxIwQVLVl28JJs8= 2023-07-29T00:37:52.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0fSpk0XrZiavTxIwQVLVl28JJs8=', 'Content-Length': '0'} 2023-07-29T00:37:52.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024925fde795f3aa0-0064c45f60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.688 DEBUG:boto:path=/hiiqos-59/obj93 2023-07-29T00:37:52.689 DEBUG:boto:auth_path=/hiiqos-59/obj93 2023-07-29T00:37:52.689 DEBUG:boto:path=/hiiqos-59/obj93?versionId=null 2023-07-29T00:37:52.690 DEBUG:boto:auth_path=/hiiqos-59/obj93?versionId=null 2023-07-29T00:37:52.690 DEBUG:boto:Method: GET 2023-07-29T00:37:52.690 DEBUG:boto:Path: /hiiqos-59/obj93?versionId=null 2023-07-29T00:37:52.691 DEBUG:boto:Data: 2023-07-29T00:37:52.691 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.691 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:52.692 DEBUG:boto:Port: 8000 2023-07-29T00:37:52.692 DEBUG:boto:Params: {} 2023-07-29T00:37:52.692 DEBUG:boto:Token: None 2023-07-29T00:37:52.693 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj93?versionId=null 2023-07-29T00:37:52.693 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0fSpk0XrZiavTxIwQVLVl28JJs8= 2023-07-29T00:37:52.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0fSpk0XrZiavTxIwQVLVl28JJs8=', 'Content-Length': '0'} 2023-07-29T00:37:52.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1c9e403d40d0ac6-0064c45f60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.696 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:52.697 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:52.697 DEBUG:boto:path=/hiiqos-59/?prefix=obj93 2023-07-29T00:37:52.698 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj93 2023-07-29T00:37:52.698 DEBUG:boto:Method: GET 2023-07-29T00:37:52.698 DEBUG:boto:Path: /hiiqos-59/?prefix=obj93 2023-07-29T00:37:52.699 DEBUG:boto:Data: 2023-07-29T00:37:52.699 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.699 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:52.700 DEBUG:boto:Port: 8001 2023-07-29T00:37:52.700 DEBUG:boto:Params: {} 2023-07-29T00:37:52.701 DEBUG:boto:Token: None 2023-07-29T00:37:52.701 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/ 2023-07-29T00:37:52.701 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2JwKdK0Oo0hX02BDmdjOqg50fzY= 2023-07-29T00:37:52.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2JwKdK0Oo0hX02BDmdjOqg50fzY=', 'Content-Length': '0'} 2023-07-29T00:37:52.704 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d39faf5fe2ea3b4f-0064c45f60-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.705 DEBUG:boto:b'hiiqos-59obj931000falseobj932023-07-29T00:35:31.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:52.706 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:52.706 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:52.706 DEBUG:boto:path=/hiiqos-59/?prefix=obj93 2023-07-29T00:37:52.707 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj93 2023-07-29T00:37:52.707 DEBUG:boto:Method: GET 2023-07-29T00:37:52.707 DEBUG:boto:Path: /hiiqos-59/?prefix=obj93 2023-07-29T00:37:52.708 DEBUG:boto:Data: 2023-07-29T00:37:52.708 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.709 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:52.709 DEBUG:boto:Port: 8000 2023-07-29T00:37:52.709 DEBUG:boto:Params: {} 2023-07-29T00:37:52.710 DEBUG:boto:Token: None 2023-07-29T00:37:52.710 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/ 2023-07-29T00:37:52.710 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2JwKdK0Oo0hX02BDmdjOqg50fzY= 2023-07-29T00:37:52.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2JwKdK0Oo0hX02BDmdjOqg50fzY=', 'Content-Length': '0'} 2023-07-29T00:37:52.715 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000186bd7bfbe93f833-0064c45f60-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.715 DEBUG:boto:b'hiiqos-59obj931000falseobj932023-07-29T00:35:31.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:52.716 DEBUG:boto:path=/hiiqos-59/obj93 2023-07-29T00:37:52.717 DEBUG:boto:auth_path=/hiiqos-59/obj93 2023-07-29T00:37:52.717 DEBUG:boto:Method: HEAD 2023-07-29T00:37:52.717 DEBUG:boto:Path: /hiiqos-59/obj93 2023-07-29T00:37:52.718 DEBUG:boto:Data: 2023-07-29T00:37:52.718 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.718 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:52.719 DEBUG:boto:Port: 8001 2023-07-29T00:37:52.719 DEBUG:boto:Params: {} 2023-07-29T00:37:52.719 DEBUG:boto:Token: None 2023-07-29T00:37:52.720 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj93 2023-07-29T00:37:52.720 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HTNmXkVFMON54nBpgukrjoN8dg0= 2023-07-29T00:37:52.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HTNmXkVFMON54nBpgukrjoN8dg0=', 'Content-Length': '0'} 2023-07-29T00:37:52.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000374d0fe5eb6a0681-0064c45f60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.723 DEBUG:boto:path=/hiiqos-59/obj93 2023-07-29T00:37:52.723 DEBUG:boto:auth_path=/hiiqos-59/obj93 2023-07-29T00:37:52.724 DEBUG:boto:Method: HEAD 2023-07-29T00:37:52.724 DEBUG:boto:Path: /hiiqos-59/obj93 2023-07-29T00:37:52.724 DEBUG:boto:Data: 2023-07-29T00:37:52.725 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.725 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:52.725 DEBUG:boto:Port: 8000 2023-07-29T00:37:52.726 DEBUG:boto:Params: {} 2023-07-29T00:37:52.726 DEBUG:boto:Token: None 2023-07-29T00:37:52.727 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj93 2023-07-29T00:37:52.727 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HTNmXkVFMON54nBpgukrjoN8dg0= 2023-07-29T00:37:52.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HTNmXkVFMON54nBpgukrjoN8dg0=', 'Content-Length': '0'} 2023-07-29T00:37:52.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a16a0d49ed9e62b-0064c45f60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-07-29T00:37:52.730 DEBUG:boto:path=/hiiqos-59/obj93 2023-07-29T00:37:52.731 DEBUG:boto:auth_path=/hiiqos-59/obj93 2023-07-29T00:37:52.731 DEBUG:boto:Method: GET 2023-07-29T00:37:52.731 DEBUG:boto:Path: /hiiqos-59/obj93 2023-07-29T00:37:52.732 DEBUG:boto:Data: 2023-07-29T00:37:52.732 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.732 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:52.733 DEBUG:boto:Port: 8001 2023-07-29T00:37:52.733 DEBUG:boto:Params: {} 2023-07-29T00:37:52.734 DEBUG:boto:Token: None 2023-07-29T00:37:52.734 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj93 2023-07-29T00:37:52.735 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BX3mKExNUF2EGdgTsjgG+1E1jL0= 2023-07-29T00:37:52.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BX3mKExNUF2EGdgTsjgG+1E1jL0=', 'Content-Length': '0'} 2023-07-29T00:37:52.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e8de64b71dfa280-0064c45f60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.738 DEBUG:boto:path=/hiiqos-59/obj93 2023-07-29T00:37:52.738 DEBUG:boto:auth_path=/hiiqos-59/obj93 2023-07-29T00:37:52.739 DEBUG:boto:Method: GET 2023-07-29T00:37:52.739 DEBUG:boto:Path: /hiiqos-59/obj93 2023-07-29T00:37:52.739 DEBUG:boto:Data: 2023-07-29T00:37:52.740 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.740 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:52.741 DEBUG:boto:Port: 8000 2023-07-29T00:37:52.741 DEBUG:boto:Params: {} 2023-07-29T00:37:52.741 DEBUG:boto:Token: None 2023-07-29T00:37:52.742 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj93 2023-07-29T00:37:52.742 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BX3mKExNUF2EGdgTsjgG+1E1jL0= 2023-07-29T00:37:52.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BX3mKExNUF2EGdgTsjgG+1E1jL0=', 'Content-Length': '0'} 2023-07-29T00:37:52.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009be1beb213d8e289-0064c45f60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.745 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:52.746 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:52.746 DEBUG:boto:path=/hiiqos-59/?prefix=obj93 2023-07-29T00:37:52.746 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj93 2023-07-29T00:37:52.747 DEBUG:boto:Method: GET 2023-07-29T00:37:52.747 DEBUG:boto:Path: /hiiqos-59/?prefix=obj93 2023-07-29T00:37:52.747 DEBUG:boto:Data: 2023-07-29T00:37:52.748 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.748 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:52.748 DEBUG:boto:Port: 8001 2023-07-29T00:37:52.749 DEBUG:boto:Params: {} 2023-07-29T00:37:52.749 DEBUG:boto:Token: None 2023-07-29T00:37:52.750 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/ 2023-07-29T00:37:52.750 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2JwKdK0Oo0hX02BDmdjOqg50fzY= 2023-07-29T00:37:52.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2JwKdK0Oo0hX02BDmdjOqg50fzY=', 'Content-Length': '0'} 2023-07-29T00:37:52.753 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007adf29edb6f8af13-0064c45f60-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.796 DEBUG:boto:b'hiiqos-59obj931000falseobj932023-07-29T00:35:31.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:52.797 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:52.797 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:52.798 DEBUG:boto:path=/hiiqos-59/?prefix=obj93 2023-07-29T00:37:52.798 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj93 2023-07-29T00:37:52.799 DEBUG:boto:Method: GET 2023-07-29T00:37:52.799 DEBUG:boto:Path: /hiiqos-59/?prefix=obj93 2023-07-29T00:37:52.799 DEBUG:boto:Data: 2023-07-29T00:37:52.800 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.800 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:52.800 DEBUG:boto:Port: 8000 2023-07-29T00:37:52.801 DEBUG:boto:Params: {} 2023-07-29T00:37:52.802 DEBUG:boto:Token: None 2023-07-29T00:37:52.802 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/ 2023-07-29T00:37:52.802 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2JwKdK0Oo0hX02BDmdjOqg50fzY= 2023-07-29T00:37:52.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2JwKdK0Oo0hX02BDmdjOqg50fzY=', 'Content-Length': '0'} 2023-07-29T00:37:52.807 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004488502d9070043-0064c45f60-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.807 DEBUG:boto:b'hiiqos-59obj931000falseobj932023-07-29T00:35:31.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:52.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-07-29T00:37:52.808 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:37:52.809 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:37:52.809 DEBUG:boto:path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:37:52.810 DEBUG:boto:auth_path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:37:52.810 DEBUG:boto:Method: GET 2023-07-29T00:37:52.810 DEBUG:boto:Path: /hiiqos-59/obj95?versionId=null 2023-07-29T00:37:52.811 DEBUG:boto:Data: 2023-07-29T00:37:52.811 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.811 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:52.812 DEBUG:boto:Port: 8001 2023-07-29T00:37:52.812 DEBUG:boto:Params: {} 2023-07-29T00:37:52.813 DEBUG:boto:Token: None 2023-07-29T00:37:52.813 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj95?versionId=null 2023-07-29T00:37:52.814 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:caBxH9/EPoRtQefYrlI7Ea4DnoA= 2023-07-29T00:37:52.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:caBxH9/EPoRtQefYrlI7Ea4DnoA=', 'Content-Length': '0'} 2023-07-29T00:37:52.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000359f2953722866ef-0064c45f60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.817 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:37:52.817 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:37:52.818 DEBUG:boto:path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:37:52.818 DEBUG:boto:auth_path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:37:52.818 DEBUG:boto:Method: GET 2023-07-29T00:37:52.819 DEBUG:boto:Path: /hiiqos-59/obj95?versionId=null 2023-07-29T00:37:52.819 DEBUG:boto:Data: 2023-07-29T00:37:52.819 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.820 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:52.820 DEBUG:boto:Port: 8000 2023-07-29T00:37:52.821 DEBUG:boto:Params: {} 2023-07-29T00:37:52.821 DEBUG:boto:Token: None 2023-07-29T00:37:52.821 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj95?versionId=null 2023-07-29T00:37:52.822 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:caBxH9/EPoRtQefYrlI7Ea4DnoA= 2023-07-29T00:37:52.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:caBxH9/EPoRtQefYrlI7Ea4DnoA=', 'Content-Length': '0'} 2023-07-29T00:37:52.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c46b2cf91f01c4f0-0064c45f60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.868 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:37:52.869 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:37:52.869 DEBUG:boto:path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:37:52.870 DEBUG:boto:auth_path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:37:52.870 DEBUG:boto:Method: HEAD 2023-07-29T00:37:52.870 DEBUG:boto:Path: /hiiqos-59/obj95?versionId=null 2023-07-29T00:37:52.871 DEBUG:boto:Data: 2023-07-29T00:37:52.871 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.871 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:52.872 DEBUG:boto:Port: 8001 2023-07-29T00:37:52.872 DEBUG:boto:Params: {} 2023-07-29T00:37:52.873 DEBUG:boto:Token: None 2023-07-29T00:37:52.873 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj95?versionId=null 2023-07-29T00:37:52.874 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gV7g2Nff6TZYohQSLSj5IYw0FHk= 2023-07-29T00:37:52.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gV7g2Nff6TZYohQSLSj5IYw0FHk=', 'Content-Length': '0'} 2023-07-29T00:37:52.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d60dbc2b67461320-0064c45f60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.876 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:37:52.877 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:37:52.877 DEBUG:boto:path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:37:52.878 DEBUG:boto:auth_path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:37:52.878 DEBUG:boto:Method: HEAD 2023-07-29T00:37:52.878 DEBUG:boto:Path: /hiiqos-59/obj95?versionId=null 2023-07-29T00:37:52.879 DEBUG:boto:Data: 2023-07-29T00:37:52.879 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.879 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:52.880 DEBUG:boto:Port: 8000 2023-07-29T00:37:52.880 DEBUG:boto:Params: {} 2023-07-29T00:37:52.881 DEBUG:boto:Token: None 2023-07-29T00:37:52.881 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj95?versionId=null 2023-07-29T00:37:52.882 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gV7g2Nff6TZYohQSLSj5IYw0FHk= 2023-07-29T00:37:52.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gV7g2Nff6TZYohQSLSj5IYw0FHk=', 'Content-Length': '0'} 2023-07-29T00:37:52.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095c28341d3671c3c-0064c45f60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-07-29T00:37:52.885 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:37:52.885 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:37:52.886 DEBUG:boto:path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:37:52.886 DEBUG:boto:auth_path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:37:52.886 DEBUG:boto:Method: GET 2023-07-29T00:37:52.887 DEBUG:boto:Path: /hiiqos-59/obj95?versionId=null 2023-07-29T00:37:52.887 DEBUG:boto:Data: 2023-07-29T00:37:52.888 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.888 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:52.888 DEBUG:boto:Port: 8001 2023-07-29T00:37:52.889 DEBUG:boto:Params: {} 2023-07-29T00:37:52.889 DEBUG:boto:Token: None 2023-07-29T00:37:52.889 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj95?versionId=null 2023-07-29T00:37:52.890 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:caBxH9/EPoRtQefYrlI7Ea4DnoA= 2023-07-29T00:37:52.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:caBxH9/EPoRtQefYrlI7Ea4DnoA=', 'Content-Length': '0'} 2023-07-29T00:37:52.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095a7c8b81e38923c-0064c45f60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.893 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:37:52.893 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:37:52.893 DEBUG:boto:path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:37:52.894 DEBUG:boto:auth_path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:37:52.894 DEBUG:boto:Method: GET 2023-07-29T00:37:52.894 DEBUG:boto:Path: /hiiqos-59/obj95?versionId=null 2023-07-29T00:37:52.895 DEBUG:boto:Data: 2023-07-29T00:37:52.895 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.896 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:52.896 DEBUG:boto:Port: 8000 2023-07-29T00:37:52.897 DEBUG:boto:Params: {} 2023-07-29T00:37:52.897 DEBUG:boto:Token: None 2023-07-29T00:37:52.897 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj95?versionId=null 2023-07-29T00:37:52.898 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:caBxH9/EPoRtQefYrlI7Ea4DnoA= 2023-07-29T00:37:52.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:caBxH9/EPoRtQefYrlI7Ea4DnoA=', 'Content-Length': '0'} 2023-07-29T00:37:52.900 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0ef46ead67827bb-0064c45f60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.901 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:52.901 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:52.902 DEBUG:boto:path=/hiiqos-59/?prefix=obj95 2023-07-29T00:37:52.902 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj95 2023-07-29T00:37:52.902 DEBUG:boto:Method: GET 2023-07-29T00:37:52.903 DEBUG:boto:Path: /hiiqos-59/?prefix=obj95 2023-07-29T00:37:52.903 DEBUG:boto:Data: 2023-07-29T00:37:52.903 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.904 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:52.904 DEBUG:boto:Port: 8001 2023-07-29T00:37:52.904 DEBUG:boto:Params: {} 2023-07-29T00:37:52.905 DEBUG:boto:Token: None 2023-07-29T00:37:52.905 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/ 2023-07-29T00:37:52.906 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2JwKdK0Oo0hX02BDmdjOqg50fzY= 2023-07-29T00:37:52.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2JwKdK0Oo0hX02BDmdjOqg50fzY=', 'Content-Length': '0'} 2023-07-29T00:37:52.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000537170b32b7788ed-0064c45f60-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.909 DEBUG:boto:b'hiiqos-59obj951000falseobj952023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:52.910 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:52.911 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:52.911 DEBUG:boto:path=/hiiqos-59/?prefix=obj95 2023-07-29T00:37:52.911 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj95 2023-07-29T00:37:52.912 DEBUG:boto:Method: GET 2023-07-29T00:37:52.912 DEBUG:boto:Path: /hiiqos-59/?prefix=obj95 2023-07-29T00:37:52.912 DEBUG:boto:Data: 2023-07-29T00:37:52.913 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.913 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:52.913 DEBUG:boto:Port: 8000 2023-07-29T00:37:52.914 DEBUG:boto:Params: {} 2023-07-29T00:37:52.914 DEBUG:boto:Token: None 2023-07-29T00:37:52.915 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/ 2023-07-29T00:37:52.915 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2JwKdK0Oo0hX02BDmdjOqg50fzY= 2023-07-29T00:37:52.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2JwKdK0Oo0hX02BDmdjOqg50fzY=', 'Content-Length': '0'} 2023-07-29T00:37:52.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a62dbce570fea8f-0064c45f60-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.920 DEBUG:boto:b'hiiqos-59obj951000falseobj952023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:52.921 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:37:52.921 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:37:52.921 DEBUG:boto:Method: HEAD 2023-07-29T00:37:52.922 DEBUG:boto:Path: /hiiqos-59/obj95 2023-07-29T00:37:52.922 DEBUG:boto:Data: 2023-07-29T00:37:52.922 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.923 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:52.923 DEBUG:boto:Port: 8001 2023-07-29T00:37:52.923 DEBUG:boto:Params: {} 2023-07-29T00:37:52.924 DEBUG:boto:Token: None 2023-07-29T00:37:52.924 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj95 2023-07-29T00:37:52.925 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fG4nLYbQUq3CIh678hAKO4di7qk= 2023-07-29T00:37:52.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fG4nLYbQUq3CIh678hAKO4di7qk=', 'Content-Length': '0'} 2023-07-29T00:37:52.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e45c3eab36ca61a-0064c45f60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.927 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:37:52.928 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:37:52.928 DEBUG:boto:Method: HEAD 2023-07-29T00:37:52.928 DEBUG:boto:Path: /hiiqos-59/obj95 2023-07-29T00:37:52.929 DEBUG:boto:Data: 2023-07-29T00:37:52.929 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.929 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:52.930 DEBUG:boto:Port: 8000 2023-07-29T00:37:52.930 DEBUG:boto:Params: {} 2023-07-29T00:37:52.930 DEBUG:boto:Token: None 2023-07-29T00:37:52.931 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj95 2023-07-29T00:37:52.931 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fG4nLYbQUq3CIh678hAKO4di7qk= 2023-07-29T00:37:52.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fG4nLYbQUq3CIh678hAKO4di7qk=', 'Content-Length': '0'} 2023-07-29T00:37:52.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1e1ee04881dde73-0064c45f60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-07-29T00:37:52.934 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:37:52.935 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:37:52.935 DEBUG:boto:Method: GET 2023-07-29T00:37:52.936 DEBUG:boto:Path: /hiiqos-59/obj95 2023-07-29T00:37:52.936 DEBUG:boto:Data: 2023-07-29T00:37:52.936 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.937 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:52.937 DEBUG:boto:Port: 8001 2023-07-29T00:37:52.937 DEBUG:boto:Params: {} 2023-07-29T00:37:52.938 DEBUG:boto:Token: None 2023-07-29T00:37:52.938 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj95 2023-07-29T00:37:52.939 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1lNTaQJP2rUnOH1llX3xa0or6e8= 2023-07-29T00:37:52.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1lNTaQJP2rUnOH1llX3xa0or6e8=', 'Content-Length': '0'} 2023-07-29T00:37:52.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cb0a456cf29bde1-0064c45f60-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.941 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:37:52.942 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:37:52.942 DEBUG:boto:Method: GET 2023-07-29T00:37:52.943 DEBUG:boto:Path: /hiiqos-59/obj95 2023-07-29T00:37:52.943 DEBUG:boto:Data: 2023-07-29T00:37:52.943 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.944 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:52.944 DEBUG:boto:Port: 8000 2023-07-29T00:37:52.944 DEBUG:boto:Params: {} 2023-07-29T00:37:52.945 DEBUG:boto:Token: None 2023-07-29T00:37:52.945 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/obj95 2023-07-29T00:37:52.946 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1lNTaQJP2rUnOH1llX3xa0or6e8= 2023-07-29T00:37:52.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1lNTaQJP2rUnOH1llX3xa0or6e8=', 'Content-Length': '0'} 2023-07-29T00:37:52.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000523c70edb3438819-0064c45f60-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:52.949 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:52.949 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:52.949 DEBUG:boto:path=/hiiqos-59/?prefix=obj95 2023-07-29T00:37:52.950 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj95 2023-07-29T00:37:52.950 DEBUG:boto:Method: GET 2023-07-29T00:37:52.950 DEBUG:boto:Path: /hiiqos-59/?prefix=obj95 2023-07-29T00:37:52.951 DEBUG:boto:Data: 2023-07-29T00:37:52.951 DEBUG:boto:Headers: {} 2023-07-29T00:37:52.951 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:37:52.952 DEBUG:boto:Port: 8001 2023-07-29T00:37:52.952 DEBUG:boto:Params: {} 2023-07-29T00:37:52.953 DEBUG:boto:Token: None 2023-07-29T00:37:52.953 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:52 GMT /hiiqos-59/ 2023-07-29T00:37:52.953 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2JwKdK0Oo0hX02BDmdjOqg50fzY= 2023-07-29T00:37:52.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2JwKdK0Oo0hX02BDmdjOqg50fzY=', 'Content-Length': '0'} 2023-07-29T00:37:52.956 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048454c2059fdb1f9-0064c45f60-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:53.000 DEBUG:boto:b'hiiqos-59obj951000falseobj952023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:53.001 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:53.002 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:53.002 DEBUG:boto:path=/hiiqos-59/?prefix=obj95 2023-07-29T00:37:53.002 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj95 2023-07-29T00:37:53.003 DEBUG:boto:Method: GET 2023-07-29T00:37:53.003 DEBUG:boto:Path: /hiiqos-59/?prefix=obj95 2023-07-29T00:37:53.003 DEBUG:boto:Data: 2023-07-29T00:37:53.004 DEBUG:boto:Headers: {} 2023-07-29T00:37:53.004 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:53.004 DEBUG:boto:Port: 8000 2023-07-29T00:37:53.005 DEBUG:boto:Params: {} 2023-07-29T00:37:53.005 DEBUG:boto:Token: None 2023-07-29T00:37:53.006 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:37:53 GMT /hiiqos-59/ 2023-07-29T00:37:53.006 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zbUKLh2FHDT1/XnGpS4Kau4xoM0= 2023-07-29T00:37:53.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zbUKLh2FHDT1/XnGpS4Kau4xoM0=', 'Content-Length': '0'} 2023-07-29T00:37:53.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdc3a573983b516f-0064c45f60-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:37:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:53.011 DEBUG:boto:b'hiiqos-59obj951000falseobj952023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:37:53.012 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-59 zones={a2, a1} 2023-07-29T00:37:53.012 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:37:53.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:37:53.013 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:37:53.066 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:37:53.066 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:37:53.114 INFO:teuthology.orchestra.run.smithi101.stdout:tenant: 2023-07-29T00:37:53.115 INFO:teuthology.orchestra.run.smithi101.stdout:bucket name: hiiqos-59 2023-07-29T00:37:53.125 INFO:teuthology.orchestra.run.smithi101.stdout:total entries: 30 2023-07-29T00:37:53.182 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:53.183 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:53.183 DEBUG:boto:Method: HEAD 2023-07-29T00:37:53.184 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:37:53.184 DEBUG:boto:Data: 2023-07-29T00:37:53.184 DEBUG:boto:Headers: {} 2023-07-29T00:37:53.185 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:53.185 DEBUG:boto:Port: 8000 2023-07-29T00:37:53.185 DEBUG:boto:Params: {} 2023-07-29T00:37:53.186 DEBUG:boto:Token: None 2023-07-29T00:37:53.186 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:53 GMT /hiiqos-59/ 2023-07-29T00:37:53.187 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5Sv/B+tCXNmSD8TlbOALk5dHAf4= 2023-07-29T00:37:53.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5Sv/B+tCXNmSD8TlbOALk5dHAf4=', 'Content-Length': '0'} 2023-07-29T00:37:53.191 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000079176297f8380c63-0064c45f61-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:53.191 DEBUG:boto:path=/hiiqos-59/obj37 2023-07-29T00:37:53.192 DEBUG:boto:auth_path=/hiiqos-59/obj37 2023-07-29T00:37:53.193 DEBUG:boto:Method: PUT 2023-07-29T00:37:53.193 DEBUG:boto:Path: /hiiqos-59/obj37 2023-07-29T00:37:53.194 DEBUG:boto:Data: 2023-07-29T00:37:53.194 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:37:53.194 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:53.195 DEBUG:boto:Port: 8000 2023-07-29T00:37:53.195 DEBUG:boto:Params: {} 2023-07-29T00:37:53.196 DEBUG:boto:Token: None 2023-07-29T00:37:53.196 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:37:53 GMT /hiiqos-59/obj37 2023-07-29T00:37:53.196 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1zuSS3zSboJcCpCPEaZRE4+b2Gc= 2023-07-29T00:37:53.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:37:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1zuSS3zSboJcCpCPEaZRE4+b2Gc='} 2023-07-29T00:37:53.240 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000da6775791308507d-0064c45f61-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:53.241 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:53.242 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:53.242 DEBUG:boto:Method: HEAD 2023-07-29T00:37:53.242 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:37:53.243 DEBUG:boto:Data: 2023-07-29T00:37:53.243 DEBUG:boto:Headers: {} 2023-07-29T00:37:53.244 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:53.244 DEBUG:boto:Port: 8000 2023-07-29T00:37:53.244 DEBUG:boto:Params: {} 2023-07-29T00:37:53.245 DEBUG:boto:Token: None 2023-07-29T00:37:53.245 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:53 GMT /hiiqos-59/ 2023-07-29T00:37:53.246 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5Sv/B+tCXNmSD8TlbOALk5dHAf4= 2023-07-29T00:37:53.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5Sv/B+tCXNmSD8TlbOALk5dHAf4=', 'Content-Length': '0'} 2023-07-29T00:37:53.250 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e296c8b8a8502963-0064c45f61-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:53.250 DEBUG:boto:path=/hiiqos-59/obj74 2023-07-29T00:37:53.251 DEBUG:boto:auth_path=/hiiqos-59/obj74 2023-07-29T00:37:53.251 DEBUG:boto:Method: PUT 2023-07-29T00:37:53.251 DEBUG:boto:Path: /hiiqos-59/obj74 2023-07-29T00:37:53.252 DEBUG:boto:Data: 2023-07-29T00:37:53.252 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:37:53.252 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:53.253 DEBUG:boto:Port: 8000 2023-07-29T00:37:53.253 DEBUG:boto:Params: {} 2023-07-29T00:37:53.254 DEBUG:boto:Token: None 2023-07-29T00:37:53.254 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:37:53 GMT /hiiqos-59/obj74 2023-07-29T00:37:53.255 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nkHqT8iY0OnUkmROijgNRXfgM7c= 2023-07-29T00:37:53.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:37:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nkHqT8iY0OnUkmROijgNRXfgM7c='} 2023-07-29T00:37:53.260 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aa40513720295ab8-0064c45f61-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:53.261 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:53.261 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:53.261 DEBUG:boto:Method: HEAD 2023-07-29T00:37:53.262 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:37:53.262 DEBUG:boto:Data: 2023-07-29T00:37:53.262 DEBUG:boto:Headers: {} 2023-07-29T00:37:53.263 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:53.263 DEBUG:boto:Port: 8000 2023-07-29T00:37:53.263 DEBUG:boto:Params: {} 2023-07-29T00:37:53.264 DEBUG:boto:Token: None 2023-07-29T00:37:53.264 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:53 GMT /hiiqos-59/ 2023-07-29T00:37:53.265 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5Sv/B+tCXNmSD8TlbOALk5dHAf4= 2023-07-29T00:37:53.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5Sv/B+tCXNmSD8TlbOALk5dHAf4=', 'Content-Length': '0'} 2023-07-29T00:37:53.268 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b71b828f81c5f4fd-0064c45f61-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:53.269 DEBUG:boto:path=/hiiqos-59/obj111 2023-07-29T00:37:53.269 DEBUG:boto:auth_path=/hiiqos-59/obj111 2023-07-29T00:37:53.270 DEBUG:boto:Method: PUT 2023-07-29T00:37:53.270 DEBUG:boto:Path: /hiiqos-59/obj111 2023-07-29T00:37:53.270 DEBUG:boto:Data: 2023-07-29T00:37:53.271 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:37:53.271 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:53.271 DEBUG:boto:Port: 8000 2023-07-29T00:37:53.272 DEBUG:boto:Params: {} 2023-07-29T00:37:53.272 DEBUG:boto:Token: None 2023-07-29T00:37:53.273 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:37:53 GMT /hiiqos-59/obj111 2023-07-29T00:37:53.273 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pg74vK3darGdbHDzuMkSCluhxSc= 2023-07-29T00:37:53.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:37:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pg74vK3darGdbHDzuMkSCluhxSc='} 2023-07-29T00:37:53.316 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006549b1b27d3d653c-0064c45f61-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:53.317 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:53.317 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:53.318 DEBUG:boto:Method: HEAD 2023-07-29T00:37:53.318 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:37:53.318 DEBUG:boto:Data: 2023-07-29T00:37:53.319 DEBUG:boto:Headers: {} 2023-07-29T00:37:53.319 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:53.320 DEBUG:boto:Port: 8000 2023-07-29T00:37:53.320 DEBUG:boto:Params: {} 2023-07-29T00:37:53.320 DEBUG:boto:Token: None 2023-07-29T00:37:53.321 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:53 GMT /hiiqos-59/ 2023-07-29T00:37:53.321 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5Sv/B+tCXNmSD8TlbOALk5dHAf4= 2023-07-29T00:37:53.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5Sv/B+tCXNmSD8TlbOALk5dHAf4=', 'Content-Length': '0'} 2023-07-29T00:37:53.325 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000be34cb707f60304f-0064c45f61-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:53.326 DEBUG:boto:path=/hiiqos-59/obj148 2023-07-29T00:37:53.326 DEBUG:boto:auth_path=/hiiqos-59/obj148 2023-07-29T00:37:53.326 DEBUG:boto:Method: PUT 2023-07-29T00:37:53.327 DEBUG:boto:Path: /hiiqos-59/obj148 2023-07-29T00:37:53.327 DEBUG:boto:Data: 2023-07-29T00:37:53.327 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:37:53.328 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:53.328 DEBUG:boto:Port: 8000 2023-07-29T00:37:53.329 DEBUG:boto:Params: {} 2023-07-29T00:37:53.329 DEBUG:boto:Token: None 2023-07-29T00:37:53.329 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:37:53 GMT /hiiqos-59/obj148 2023-07-29T00:37:53.330 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sjKAYEBFYVaeCzuvR68LtxWPl34= 2023-07-29T00:37:53.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:37:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sjKAYEBFYVaeCzuvR68LtxWPl34='} 2023-07-29T00:37:53.338 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003a0ae222d8de05f3-0064c45f61-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:53.339 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:37:53.339 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:37:53.339 DEBUG:boto:Method: HEAD 2023-07-29T00:37:53.340 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:37:53.340 DEBUG:boto:Data: 2023-07-29T00:37:53.341 DEBUG:boto:Headers: {} 2023-07-29T00:37:53.341 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:53.341 DEBUG:boto:Port: 8000 2023-07-29T00:37:53.342 DEBUG:boto:Params: {} 2023-07-29T00:37:53.342 DEBUG:boto:Token: None 2023-07-29T00:37:53.343 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:37:53 GMT /hiiqos-59/ 2023-07-29T00:37:53.343 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5Sv/B+tCXNmSD8TlbOALk5dHAf4= 2023-07-29T00:37:53.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:37:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5Sv/B+tCXNmSD8TlbOALk5dHAf4=', 'Content-Length': '0'} 2023-07-29T00:37:53.347 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005683f0e6866af577-0064c45f61-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:53.347 DEBUG:boto:path=/hiiqos-59/obj185 2023-07-29T00:37:53.347 DEBUG:boto:auth_path=/hiiqos-59/obj185 2023-07-29T00:37:53.348 DEBUG:boto:Method: PUT 2023-07-29T00:37:53.348 DEBUG:boto:Path: /hiiqos-59/obj185 2023-07-29T00:37:53.349 DEBUG:boto:Data: 2023-07-29T00:37:53.349 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:37:53.349 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:37:53.350 DEBUG:boto:Port: 8000 2023-07-29T00:37:53.350 DEBUG:boto:Params: {} 2023-07-29T00:37:53.350 DEBUG:boto:Token: None 2023-07-29T00:37:53.351 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:37:53 GMT /hiiqos-59/obj185 2023-07-29T00:37:53.351 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mNgftTG2MOdpT22J7PkOV8BlZGw= 2023-07-29T00:37:53.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:37:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mNgftTG2MOdpT22J7PkOV8BlZGw='} 2023-07-29T00:37:53.396 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000056d34ff13aff0ff7-0064c45f61-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:37:53.397 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-59 2023-07-29T00:37:53.397 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:37:53.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:37:53.398 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:37:53.450 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:37:53.451 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:37:53.484 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:37:53.421+0000 7fae14c8b880 1 waiting to reach latest gen 6, on 5.. 2023-07-29T00:37:58.485 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:37:58.422+0000 7fae14c8b880 1 waiting to reach latest gen 6, on 5.. 2023-07-29T00:38:03.486 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:38:03.422+0000 7fae14c8b880 1 waiting to reach latest gen 6, on 5.. 2023-07-29T00:38:08.489 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:38:08.425+0000 7fae14c8b880 1 waiting for incremental sync to catch up: 2023-07-29T00:38:08.489 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.101.6, , , , , , 00000000002.70.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.74.6] 2023-07-29T00:38:08.490 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.101.6, , , , , , 00000000002.70.6, , , , , 00000000002.62.6, , , , , , , , , , , , , , , 00000000001.88.6, , , , 00000000001.74.6] 2023-07-29T00:38:13.491 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:38:13.428+0000 7fae14c8b880 1 waiting for incremental sync to catch up: 2023-07-29T00:38:13.492 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.101.6, , , , , , 00000000002.70.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.74.6] 2023-07-29T00:38:13.492 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.101.6, , , , , , 00000000002.70.6, , , , , 00000000002.62.6, , , , , , , , , , , , , , , 00000000001.88.6, , , , 00000000001.74.6] 2023-07-29T00:38:18.494 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:38:18.431+0000 7fae14c8b880 1 waiting for incremental sync to catch up: 2023-07-29T00:38:18.495 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.101.6, , , , , , 00000000002.70.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.74.6] 2023-07-29T00:38:18.495 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.101.6, , , , , , 00000000002.70.6, , , , , 00000000002.62.6, , , , , , , , , , , , , , , 00000000001.88.6, , , , 00000000001.74.6] 2023-07-29T00:38:23.497 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:38:23.433+0000 7fae14c8b880 1 waiting for incremental sync to catch up: 2023-07-29T00:38:23.497 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.101.6, , , , , , 00000000002.70.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.74.6] 2023-07-29T00:38:23.498 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.101.6, , , , , , 00000000002.70.6, , , , , 00000000002.62.6, , , , , , , , , , , , , , , 00000000001.88.6, , , , 00000000001.74.6] 2023-07-29T00:38:28.500 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:38:28.436+0000 7fae14c8b880 1 waiting for incremental sync to catch up: 2023-07-29T00:38:28.500 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.101.6, , , , , , 00000000002.70.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.74.6] 2023-07-29T00:38:28.500 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.101.6, , , , , , 00000000002.70.6, , , , , 00000000002.62.6, , , , , , , , , , , , , , , 00000000001.88.6, , , , 00000000001.74.6] 2023-07-29T00:38:33.502 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:38:33.439+0000 7fae14c8b880 1 waiting for incremental sync to catch up: 2023-07-29T00:38:33.503 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.101.6, , , , , , 00000000002.70.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.74.6] 2023-07-29T00:38:33.503 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.101.6, , , , , , 00000000002.70.6, , , , , 00000000002.62.6, , , , , , , , , , , , , , , 00000000001.88.6, , , , 00000000001.74.6] 2023-07-29T00:38:38.505 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:38:38.441+0000 7fae14c8b880 1 waiting for incremental sync to catch up: 2023-07-29T00:38:38.506 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.101.6, , , , , , 00000000002.70.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.74.6] 2023-07-29T00:38:38.506 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.101.6, , , , , , 00000000002.70.6, , , , , 00000000002.62.6, , , , , , , , , , , , , , , 00000000001.88.6, , , , 00000000001.74.6] 2023-07-29T00:38:43.508 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:38:43.444+0000 7fae14c8b880 1 waiting for incremental sync to catch up: 2023-07-29T00:38:43.509 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.101.6, , , , , , 00000000002.70.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.74.6] 2023-07-29T00:38:43.509 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.101.6, , , , , , 00000000002.70.6, , , , , 00000000002.62.6, , , , , , , , , , , , , , , 00000000001.88.6, , , , 00000000001.74.6] 2023-07-29T00:38:48.511 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:38:48.447+0000 7fae14c8b880 1 waiting for incremental sync to catch up: 2023-07-29T00:38:48.511 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.101.6, , , , , , 00000000002.70.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.74.6] 2023-07-29T00:38:48.511 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.101.6, , , , , , 00000000002.70.6, , , , , 00000000002.62.6, , , , , , , , , , , , , , , 00000000001.88.6, , , , 00000000001.74.6] 2023-07-29T00:38:53.513 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:38:53.449+0000 7fae14c8b880 1 waiting for incremental sync to catch up: 2023-07-29T00:38:53.514 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.101.6, , , , , , 00000000002.70.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.74.6] 2023-07-29T00:38:53.514 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.101.6, , , , , , 00000000002.70.6, , , , , 00000000002.62.6, , , , , , , , , , , , , , , 00000000001.88.6, , , , 00000000001.74.6] 2023-07-29T00:38:58.516 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:38:58.452+0000 7fae14c8b880 1 waiting for incremental sync to catch up: 2023-07-29T00:38:58.517 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.101.6, , , , , , 00000000002.70.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.74.6] 2023-07-29T00:38:58.517 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.101.6, , , , , , 00000000002.70.6, , , , , 00000000002.62.6, , , , , , , , , , , , , , , 00000000001.88.6, , , , 00000000001.74.6] 2023-07-29T00:39:03.519 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:03.455+0000 7fae14c8b880 1 waiting for incremental sync to catch up: 2023-07-29T00:39:03.519 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.101.6, , , , , , 00000000002.70.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.74.6] 2023-07-29T00:39:03.520 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.101.6, , , , , , 00000000002.70.6, , , , , 00000000002.62.6, , , , , , , , , , , , , , , 00000000001.88.6, , , , 00000000001.74.6] 2023-07-29T00:39:08.522 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:08.458+0000 7fae14c8b880 1 bucket sync caught up with source: 2023-07-29T00:39:08.522 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.101.6, , , , , , 00000000002.70.6, , , , , 00000000002.62.6, , , , , , , , , , , , , , , 00000000001.88.6, , , , 00000000001.74.6] 2023-07-29T00:39:08.522 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.101.6, , , , , , 00000000002.70.6, , , , , 00000000002.62.6, , , , , , , , , , , , , , , 00000000001.88.6, , , , 00000000001.74.6] 2023-07-29T00:39:08.522 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:08.458+0000 7fae14c8b880 0 bucket checkpoint complete 2023-07-29T00:39:08.539 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-59 2023-07-29T00:39:08.540 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:39:08.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:39:08.542 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:39:08.598 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:08.599 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:08.634 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:08.570+0000 7fad4a220880 1 waiting for incremental sync to catch up: 2023-07-29T00:39:08.634 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000003.862.6, 00000000002.701.6, 00000000002.1214.6, 00000000004.960.6, 00000000004.745.6, 00000000001.1016.6, 00000000001.432.6, 00000000004.313.6, 00000000006.803.6, 00000000006.1385.6] 2023-07-29T00:39:08.635 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000003.862.6, 00000000002.701.6, 00000000002.1214.6, 00000000005.964.6, 00000000004.745.6, 00000000001.1016.6, 00000000001.432.6, 00000000004.313.6, 00000000007.807.6, 00000000006.1385.6] 2023-07-29T00:39:13.635 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:13.571+0000 7fad4a220880 1 waiting for incremental sync to catch up: 2023-07-29T00:39:13.636 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000003.862.6, 00000000002.701.6, 00000000002.1214.6, 00000000004.960.6, 00000000004.745.6, 00000000001.1016.6, 00000000001.432.6, 00000000004.313.6, 00000000006.803.6, 00000000006.1385.6] 2023-07-29T00:39:13.636 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000003.862.6, 00000000002.701.6, 00000000002.1214.6, 00000000005.964.6, 00000000004.745.6, 00000000001.1016.6, 00000000001.432.6, 00000000004.313.6, 00000000007.807.6, 00000000006.1385.6] 2023-07-29T00:39:18.637 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:18.573+0000 7fad4a220880 1 waiting for incremental sync to catch up: 2023-07-29T00:39:18.637 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000003.862.6, 00000000002.701.6, 00000000002.1214.6, 00000000004.960.6, 00000000004.745.6, 00000000001.1016.6, 00000000001.432.6, 00000000004.313.6, 00000000006.803.6, 00000000006.1385.6] 2023-07-29T00:39:18.638 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000003.862.6, 00000000002.701.6, 00000000002.1214.6, 00000000005.964.6, 00000000004.745.6, 00000000001.1016.6, 00000000001.432.6, 00000000004.313.6, 00000000007.807.6, 00000000006.1385.6] 2023-07-29T00:39:23.388 INFO:tasks.rgw.c1.client.0.smithi101.stdout:2023-07-29T00:39:23.323+0000 7f0081a5e640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T00:39:23.638 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:23.574+0000 7fad4a220880 1 bucket sync caught up with source: 2023-07-29T00:39:23.639 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000003.862.6, 00000000002.701.6, 00000000002.1214.6, 00000000005.964.6, 00000000004.745.6, 00000000001.1016.6, 00000000001.432.6, 00000000004.313.6, 00000000007.807.6, 00000000006.1385.6] 2023-07-29T00:39:23.639 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000003.862.6, 00000000002.701.6, 00000000002.1214.6, 00000000005.964.6, 00000000004.745.6, 00000000001.1016.6, 00000000001.432.6, 00000000004.313.6, 00000000007.807.6, 00000000006.1385.6] 2023-07-29T00:39:23.639 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:23.574+0000 7fad4a220880 0 bucket checkpoint complete 2023-07-29T00:39:23.656 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-59 zones={a2, a1} 2023-07-29T00:39:23.657 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:23.657 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:23.658 DEBUG:boto:Method: HEAD 2023-07-29T00:39:23.658 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:39:23.658 DEBUG:boto:Data: 2023-07-29T00:39:23.659 DEBUG:boto:Headers: {} 2023-07-29T00:39:23.659 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:23.660 DEBUG:boto:Port: 8001 2023-07-29T00:39:23.660 DEBUG:boto:Params: {} 2023-07-29T00:39:23.660 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-29T00:39:23.661 DEBUG:boto:Token: None 2023-07-29T00:39:23.661 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:23 GMT /hiiqos-59/ 2023-07-29T00:39:23.662 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:InnQM4bAa8Nq15VQ4GZedCv0jWE= 2023-07-29T00:39:23.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:InnQM4bAa8Nq15VQ4GZedCv0jWE=', 'Content-Length': '0'} 2023-07-29T00:39:28.667 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000927db7142d088995-0064c45fc0-1c08-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:39:28 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:28.668 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:28.668 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:28.669 DEBUG:boto:Method: HEAD 2023-07-29T00:39:28.669 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:39:28.670 DEBUG:boto:Data: 2023-07-29T00:39:28.670 DEBUG:boto:Headers: {} 2023-07-29T00:39:28.670 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:28.671 DEBUG:boto:Port: 8000 2023-07-29T00:39:28.671 DEBUG:boto:Params: {} 2023-07-29T00:39:28.671 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-29T00:39:28.672 DEBUG:boto:Token: None 2023-07-29T00:39:28.672 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:28 GMT /hiiqos-59/ 2023-07-29T00:39:28.673 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vXM52Gcp8cYsoBOM5eJ/ock2WHs= 2023-07-29T00:39:28.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:28 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vXM52Gcp8cYsoBOM5eJ/ock2WHs=', 'Content-Length': '0'} 2023-07-29T00:39:33.679 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004f72c76633382d92-0064c45fc5-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:39:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:33.680 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-29T00:39:33.680 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:33.681 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:33.681 DEBUG:boto:path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:39:33.682 DEBUG:boto:auth_path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:39:33.682 DEBUG:boto:Method: GET 2023-07-29T00:39:33.682 DEBUG:boto:Path: /hiiqos-59/?versions&max-keys=999 2023-07-29T00:39:33.683 DEBUG:boto:Data: 2023-07-29T00:39:33.683 DEBUG:boto:Headers: {} 2023-07-29T00:39:33.683 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:33.684 DEBUG:boto:Port: 8001 2023-07-29T00:39:33.684 DEBUG:boto:Params: {} 2023-07-29T00:39:33.684 DEBUG:boto:Token: None 2023-07-29T00:39:33.685 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:33 GMT /hiiqos-59/?versions 2023-07-29T00:39:33.685 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2y1wSmq7pEnxCDf3JBoPgVEVo2A= 2023-07-29T00:39:33.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2y1wSmq7pEnxCDf3JBoPgVEVo2A=', 'Content-Length': '0'} 2023-07-29T00:39:33.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9e5aedb64e23d88-0064c45fc5-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:33.689 DEBUG:boto:b'hiiqos-59999falseobj11nulltrue2023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-07-29T00:37:53.213Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-07-29T00:35:31.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-07-29T00:35:01.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-07-29T00:37:53.270Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-07-29T00:35:31.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-07-29T00:37:53.291Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-07-29T00:33:06.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-07-29T00:35:01.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-07-29T00:35:31.771Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-07-29T00:37:53.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-07-29T00:33:06.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-07-29T00:35:01.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-07-29T00:35:31.826Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-07-29T00:33:06.857Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-07-29T00:37:53.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-07-29T00:35:01.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-07-29T00:33:06.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-07-29T00:35:31.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:33.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-07-29T00:39:33.692 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-07-29T00:39:33.692 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-07-29T00:39:33.692 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-07-29T00:39:33.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-07-29T00:39:33.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-07-29T00:39:33.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-07-29T00:39:33.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-07-29T00:39:33.694 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-07-29T00:39:33.694 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-07-29T00:39:33.694 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-07-29T00:39:33.695 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-07-29T00:39:33.695 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-07-29T00:39:33.695 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-07-29T00:39:33.696 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-07-29T00:39:33.696 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-07-29T00:39:33.696 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-07-29T00:39:33.696 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-07-29T00:39:33.697 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-07-29T00:39:33.697 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-07-29T00:39:33.697 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-07-29T00:39:33.698 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-07-29T00:39:33.698 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-07-29T00:39:33.698 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-07-29T00:39:33.698 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-07-29T00:39:33.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-07-29T00:39:33.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-07-29T00:39:33.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-07-29T00:39:33.700 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-07-29T00:39:33.700 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-07-29T00:39:33.700 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-07-29T00:39:33.701 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-07-29T00:39:33.701 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-07-29T00:39:33.701 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-07-29T00:39:33.701 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-07-29T00:39:33.702 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-29T00:39:33.702 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:33.703 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:33.703 DEBUG:boto:path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:39:33.703 DEBUG:boto:auth_path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:39:33.704 DEBUG:boto:Method: GET 2023-07-29T00:39:33.704 DEBUG:boto:Path: /hiiqos-59/?versions&max-keys=999 2023-07-29T00:39:33.704 DEBUG:boto:Data: 2023-07-29T00:39:33.705 DEBUG:boto:Headers: {} 2023-07-29T00:39:33.705 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:33.705 DEBUG:boto:Port: 8000 2023-07-29T00:39:33.706 DEBUG:boto:Params: {} 2023-07-29T00:39:33.706 DEBUG:boto:Token: None 2023-07-29T00:39:33.707 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:33 GMT /hiiqos-59/?versions 2023-07-29T00:39:33.707 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2y1wSmq7pEnxCDf3JBoPgVEVo2A= 2023-07-29T00:39:33.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2y1wSmq7pEnxCDf3JBoPgVEVo2A=', 'Content-Length': '0'} 2023-07-29T00:39:33.712 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a01e06f2263e8961-0064c45fc5-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:33.712 DEBUG:boto:b'hiiqos-59999falseobj11nulltrue2023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-07-29T00:37:53.213Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-07-29T00:35:31.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-07-29T00:35:01.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-07-29T00:37:53.270Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-07-29T00:35:31.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-07-29T00:37:53.291Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-07-29T00:33:06.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-07-29T00:35:01.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-07-29T00:35:31.771Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-07-29T00:37:53.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-07-29T00:33:06.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-07-29T00:35:01.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-07-29T00:35:31.826Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-07-29T00:33:06.857Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-07-29T00:37:53.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-07-29T00:35:01.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-07-29T00:33:06.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-07-29T00:35:31.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:33.714 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-07-29T00:39:33.715 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-07-29T00:39:33.715 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-07-29T00:39:33.716 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-07-29T00:39:33.716 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-07-29T00:39:33.716 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-07-29T00:39:33.716 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-07-29T00:39:33.717 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-07-29T00:39:33.717 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-07-29T00:39:33.717 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-07-29T00:39:33.718 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-07-29T00:39:33.718 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-07-29T00:39:33.718 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-07-29T00:39:33.718 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-07-29T00:39:33.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-07-29T00:39:33.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-07-29T00:39:33.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-07-29T00:39:33.720 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-07-29T00:39:33.720 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-07-29T00:39:33.720 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-07-29T00:39:33.721 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-07-29T00:39:33.721 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-07-29T00:39:33.721 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-07-29T00:39:33.721 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-07-29T00:39:33.722 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-07-29T00:39:33.722 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-07-29T00:39:33.722 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-07-29T00:39:33.723 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-07-29T00:39:33.723 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-07-29T00:39:33.723 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-07-29T00:39:33.723 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-07-29T00:39:33.724 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-07-29T00:39:33.724 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-07-29T00:39:33.724 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-07-29T00:39:33.725 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-07-29T00:39:33.725 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:33.725 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:33.726 DEBUG:boto:path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:39:33.726 DEBUG:boto:auth_path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:39:33.727 DEBUG:boto:Method: GET 2023-07-29T00:39:33.727 DEBUG:boto:Path: /hiiqos-59/?versions&max-keys=999 2023-07-29T00:39:33.727 DEBUG:boto:Data: 2023-07-29T00:39:33.728 DEBUG:boto:Headers: {} 2023-07-29T00:39:33.728 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:33.728 DEBUG:boto:Port: 8001 2023-07-29T00:39:33.729 DEBUG:boto:Params: {} 2023-07-29T00:39:33.729 DEBUG:boto:Token: None 2023-07-29T00:39:33.729 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:33 GMT /hiiqos-59/?versions 2023-07-29T00:39:33.730 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2y1wSmq7pEnxCDf3JBoPgVEVo2A= 2023-07-29T00:39:33.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2y1wSmq7pEnxCDf3JBoPgVEVo2A=', 'Content-Length': '0'} 2023-07-29T00:39:33.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007889db7de32cb04e-0064c45fc5-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:33.734 DEBUG:boto:b'hiiqos-59999falseobj11nulltrue2023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-07-29T00:37:53.213Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-07-29T00:35:31.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-07-29T00:35:01.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-07-29T00:37:53.270Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-07-29T00:35:31.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-07-29T00:37:53.291Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-07-29T00:33:06.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-07-29T00:35:01.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-07-29T00:35:31.771Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-07-29T00:37:53.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-07-29T00:33:06.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-07-29T00:35:01.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-07-29T00:35:31.826Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-07-29T00:33:06.857Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-07-29T00:37:53.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-07-29T00:35:01.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-07-29T00:33:06.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-07-29T00:35:31.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:33.736 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:33.736 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:33.737 DEBUG:boto:path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:39:33.737 DEBUG:boto:auth_path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:39:33.737 DEBUG:boto:Method: GET 2023-07-29T00:39:33.738 DEBUG:boto:Path: /hiiqos-59/?versions&max-keys=999 2023-07-29T00:39:33.738 DEBUG:boto:Data: 2023-07-29T00:39:33.738 DEBUG:boto:Headers: {} 2023-07-29T00:39:33.739 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:33.739 DEBUG:boto:Port: 8000 2023-07-29T00:39:33.739 DEBUG:boto:Params: {} 2023-07-29T00:39:33.740 DEBUG:boto:Token: None 2023-07-29T00:39:33.740 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:33 GMT /hiiqos-59/?versions 2023-07-29T00:39:33.741 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2y1wSmq7pEnxCDf3JBoPgVEVo2A= 2023-07-29T00:39:33.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2y1wSmq7pEnxCDf3JBoPgVEVo2A=', 'Content-Length': '0'} 2023-07-29T00:39:33.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd917bd0be6c8d0d-0064c45fc5-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:33.747 DEBUG:boto:b'hiiqos-59999falseobj11nulltrue2023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-07-29T00:37:53.213Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-07-29T00:35:31.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-07-29T00:35:01.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-07-29T00:37:53.270Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-07-29T00:35:31.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-07-29T00:37:53.291Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-07-29T00:33:06.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-07-29T00:35:01.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-07-29T00:35:31.771Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-07-29T00:37:53.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-07-29T00:33:06.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-07-29T00:35:01.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-07-29T00:35:31.826Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-07-29T00:33:06.857Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-07-29T00:37:53.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-07-29T00:35:01.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-07-29T00:33:06.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-07-29T00:35:31.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:33.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-07-29T00:39:33.750 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:39:33.750 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:39:33.751 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:39:33.751 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:39:33.751 DEBUG:boto:Method: GET 2023-07-29T00:39:33.752 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:39:33.752 DEBUG:boto:Data: 2023-07-29T00:39:33.753 DEBUG:boto:Headers: {} 2023-07-29T00:39:33.753 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:33.753 DEBUG:boto:Port: 8001 2023-07-29T00:39:33.754 DEBUG:boto:Params: {} 2023-07-29T00:39:33.754 DEBUG:boto:Token: None 2023-07-29T00:39:33.754 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:33 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:39:33.755 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YprPbE6SDFbUl1EA4jTlibDQiw4= 2023-07-29T00:39:33.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YprPbE6SDFbUl1EA4jTlibDQiw4=', 'Content-Length': '0'} 2023-07-29T00:39:33.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3e39c33b8bd2cdd-0064c45fc5-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:33.763 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:39:33.764 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:39:33.764 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:39:33.764 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:39:33.765 DEBUG:boto:Method: GET 2023-07-29T00:39:33.765 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:39:33.766 DEBUG:boto:Data: 2023-07-29T00:39:33.766 DEBUG:boto:Headers: {} 2023-07-29T00:39:33.766 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:33.767 DEBUG:boto:Port: 8000 2023-07-29T00:39:33.767 DEBUG:boto:Params: {} 2023-07-29T00:39:33.768 DEBUG:boto:Token: None 2023-07-29T00:39:33.768 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:33 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:39:33.769 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YprPbE6SDFbUl1EA4jTlibDQiw4= 2023-07-29T00:39:33.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YprPbE6SDFbUl1EA4jTlibDQiw4=', 'Content-Length': '0'} 2023-07-29T00:39:33.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001b578375af9284c-0064c45fc5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:33.812 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:39:33.813 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:39:33.813 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:39:33.814 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:39:33.814 DEBUG:boto:Method: HEAD 2023-07-29T00:39:33.814 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:39:33.815 DEBUG:boto:Data: 2023-07-29T00:39:33.815 DEBUG:boto:Headers: {} 2023-07-29T00:39:33.815 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:33.816 DEBUG:boto:Port: 8001 2023-07-29T00:39:33.816 DEBUG:boto:Params: {} 2023-07-29T00:39:33.817 DEBUG:boto:Token: None 2023-07-29T00:39:33.817 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:33 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:39:33.817 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HIrDTqMJxWse8nuTxqgKsf0s42A= 2023-07-29T00:39:33.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HIrDTqMJxWse8nuTxqgKsf0s42A=', 'Content-Length': '0'} 2023-07-29T00:39:33.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000712a98d28285a0ab-0064c45fc5-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:33.820 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:39:33.821 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:39:33.821 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:39:33.822 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:39:33.822 DEBUG:boto:Method: HEAD 2023-07-29T00:39:33.822 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:39:33.823 DEBUG:boto:Data: 2023-07-29T00:39:33.823 DEBUG:boto:Headers: {} 2023-07-29T00:39:33.823 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:33.824 DEBUG:boto:Port: 8000 2023-07-29T00:39:33.824 DEBUG:boto:Params: {} 2023-07-29T00:39:33.824 DEBUG:boto:Token: None 2023-07-29T00:39:33.825 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:33 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:39:33.825 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HIrDTqMJxWse8nuTxqgKsf0s42A= 2023-07-29T00:39:33.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HIrDTqMJxWse8nuTxqgKsf0s42A=', 'Content-Length': '0'} 2023-07-29T00:39:33.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ca14e76b1c9ad8b-0064c45fc5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:33.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-07-29T00:39:33.829 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:39:33.829 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:39:33.829 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:39:33.830 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:39:33.830 DEBUG:boto:Method: GET 2023-07-29T00:39:33.830 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:39:33.831 DEBUG:boto:Data: 2023-07-29T00:39:33.831 DEBUG:boto:Headers: {} 2023-07-29T00:39:33.831 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:33.832 DEBUG:boto:Port: 8001 2023-07-29T00:39:33.832 DEBUG:boto:Params: {} 2023-07-29T00:39:33.833 DEBUG:boto:Token: None 2023-07-29T00:39:33.833 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:33 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:39:33.834 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YprPbE6SDFbUl1EA4jTlibDQiw4= 2023-07-29T00:39:33.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YprPbE6SDFbUl1EA4jTlibDQiw4=', 'Content-Length': '0'} 2023-07-29T00:39:33.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000648d46d65c17ab3d-0064c45fc5-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:33.836 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:39:33.837 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:39:33.837 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:39:33.837 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:39:33.838 DEBUG:boto:Method: GET 2023-07-29T00:39:33.838 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:39:33.838 DEBUG:boto:Data: 2023-07-29T00:39:33.839 DEBUG:boto:Headers: {} 2023-07-29T00:39:33.839 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:33.840 DEBUG:boto:Port: 8000 2023-07-29T00:39:33.840 DEBUG:boto:Params: {} 2023-07-29T00:39:33.840 DEBUG:boto:Token: None 2023-07-29T00:39:33.841 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:33 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:39:33.841 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YprPbE6SDFbUl1EA4jTlibDQiw4= 2023-07-29T00:39:33.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YprPbE6SDFbUl1EA4jTlibDQiw4=', 'Content-Length': '0'} 2023-07-29T00:39:33.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c999d33a7d915db0-0064c45fc5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:33.844 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:33.845 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:33.845 DEBUG:boto:path=/hiiqos-59/?prefix=obj11 2023-07-29T00:39:33.845 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj11 2023-07-29T00:39:33.846 DEBUG:boto:Method: GET 2023-07-29T00:39:33.846 DEBUG:boto:Path: /hiiqos-59/?prefix=obj11 2023-07-29T00:39:33.846 DEBUG:boto:Data: 2023-07-29T00:39:33.847 DEBUG:boto:Headers: {} 2023-07-29T00:39:33.847 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:33.848 DEBUG:boto:Port: 8001 2023-07-29T00:39:33.848 DEBUG:boto:Params: {} 2023-07-29T00:39:33.848 DEBUG:boto:Token: None 2023-07-29T00:39:33.849 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:33 GMT /hiiqos-59/ 2023-07-29T00:39:33.849 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AG7Y6krVy1uPXJPy7DfwAZzcVGw= 2023-07-29T00:39:33.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AG7Y6krVy1uPXJPy7DfwAZzcVGw=', 'Content-Length': '0'} 2023-07-29T00:39:33.852 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d847c7f34014939-0064c45fc5-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:33.853 DEBUG:boto:b'hiiqos-59obj111000falseobj112023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-07-29T00:37:53.213Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:33.853 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:33.854 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:33.854 DEBUG:boto:path=/hiiqos-59/?prefix=obj11 2023-07-29T00:39:33.855 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj11 2023-07-29T00:39:33.855 DEBUG:boto:Method: GET 2023-07-29T00:39:33.855 DEBUG:boto:Path: /hiiqos-59/?prefix=obj11 2023-07-29T00:39:33.856 DEBUG:boto:Data: 2023-07-29T00:39:33.856 DEBUG:boto:Headers: {} 2023-07-29T00:39:33.857 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:33.857 DEBUG:boto:Port: 8000 2023-07-29T00:39:33.857 DEBUG:boto:Params: {} 2023-07-29T00:39:33.858 DEBUG:boto:Token: None 2023-07-29T00:39:33.858 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:33 GMT /hiiqos-59/ 2023-07-29T00:39:33.859 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AG7Y6krVy1uPXJPy7DfwAZzcVGw= 2023-07-29T00:39:33.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AG7Y6krVy1uPXJPy7DfwAZzcVGw=', 'Content-Length': '0'} 2023-07-29T00:39:33.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b732d22a763f48e3-0064c45fc5-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:33.863 DEBUG:boto:b'hiiqos-59obj111000falseobj112023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-07-29T00:37:53.213Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:33.864 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:39:33.865 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:39:33.865 DEBUG:boto:Method: HEAD 2023-07-29T00:39:33.865 DEBUG:boto:Path: /hiiqos-59/obj11 2023-07-29T00:39:33.866 DEBUG:boto:Data: 2023-07-29T00:39:33.866 DEBUG:boto:Headers: {} 2023-07-29T00:39:33.866 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:33.867 DEBUG:boto:Port: 8001 2023-07-29T00:39:33.867 DEBUG:boto:Params: {} 2023-07-29T00:39:33.867 DEBUG:boto:Token: None 2023-07-29T00:39:33.868 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:33 GMT /hiiqos-59/obj11 2023-07-29T00:39:33.868 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nikvWWVy2gE8bd3WC2l7Do8ACRw= 2023-07-29T00:39:33.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nikvWWVy2gE8bd3WC2l7Do8ACRw=', 'Content-Length': '0'} 2023-07-29T00:39:33.871 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff6800e48c894950-0064c45fc5-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:33.871 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:39:33.872 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:39:33.872 DEBUG:boto:Method: HEAD 2023-07-29T00:39:33.872 DEBUG:boto:Path: /hiiqos-59/obj11 2023-07-29T00:39:33.873 DEBUG:boto:Data: 2023-07-29T00:39:33.873 DEBUG:boto:Headers: {} 2023-07-29T00:39:33.873 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:33.874 DEBUG:boto:Port: 8000 2023-07-29T00:39:33.874 DEBUG:boto:Params: {} 2023-07-29T00:39:33.874 DEBUG:boto:Token: None 2023-07-29T00:39:33.875 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:33 GMT /hiiqos-59/obj11 2023-07-29T00:39:33.875 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nikvWWVy2gE8bd3WC2l7Do8ACRw= 2023-07-29T00:39:33.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nikvWWVy2gE8bd3WC2l7Do8ACRw=', 'Content-Length': '0'} 2023-07-29T00:39:33.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8ec69cfce7f8588-0064c45fc5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:33.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-07-29T00:39:33.879 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:39:33.879 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:39:33.879 DEBUG:boto:Method: GET 2023-07-29T00:39:33.880 DEBUG:boto:Path: /hiiqos-59/obj11 2023-07-29T00:39:33.880 DEBUG:boto:Data: 2023-07-29T00:39:33.881 DEBUG:boto:Headers: {} 2023-07-29T00:39:33.881 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:33.881 DEBUG:boto:Port: 8001 2023-07-29T00:39:33.882 DEBUG:boto:Params: {} 2023-07-29T00:39:33.882 DEBUG:boto:Token: None 2023-07-29T00:39:33.882 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:33 GMT /hiiqos-59/obj11 2023-07-29T00:39:33.883 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UGbq25H7BFACJWhL3OH0HzwASVc= 2023-07-29T00:39:33.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UGbq25H7BFACJWhL3OH0HzwASVc=', 'Content-Length': '0'} 2023-07-29T00:39:33.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e628574e2c59e4c3-0064c45fc5-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:33.886 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:39:33.886 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:39:33.887 DEBUG:boto:Method: GET 2023-07-29T00:39:33.887 DEBUG:boto:Path: /hiiqos-59/obj11 2023-07-29T00:39:33.887 DEBUG:boto:Data: 2023-07-29T00:39:33.888 DEBUG:boto:Headers: {} 2023-07-29T00:39:33.888 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:33.888 DEBUG:boto:Port: 8000 2023-07-29T00:39:33.889 DEBUG:boto:Params: {} 2023-07-29T00:39:33.889 DEBUG:boto:Token: None 2023-07-29T00:39:33.890 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:33 GMT /hiiqos-59/obj11 2023-07-29T00:39:33.890 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UGbq25H7BFACJWhL3OH0HzwASVc= 2023-07-29T00:39:33.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UGbq25H7BFACJWhL3OH0HzwASVc=', 'Content-Length': '0'} 2023-07-29T00:39:33.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af2e932178c36d10-0064c45fc5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:33.893 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:33.893 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:33.894 DEBUG:boto:path=/hiiqos-59/?prefix=obj11 2023-07-29T00:39:33.894 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj11 2023-07-29T00:39:33.894 DEBUG:boto:Method: GET 2023-07-29T00:39:33.895 DEBUG:boto:Path: /hiiqos-59/?prefix=obj11 2023-07-29T00:39:33.895 DEBUG:boto:Data: 2023-07-29T00:39:33.896 DEBUG:boto:Headers: {} 2023-07-29T00:39:33.896 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:33.896 DEBUG:boto:Port: 8001 2023-07-29T00:39:33.897 DEBUG:boto:Params: {} 2023-07-29T00:39:33.897 DEBUG:boto:Token: None 2023-07-29T00:39:33.897 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:33 GMT /hiiqos-59/ 2023-07-29T00:39:33.898 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AG7Y6krVy1uPXJPy7DfwAZzcVGw= 2023-07-29T00:39:33.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AG7Y6krVy1uPXJPy7DfwAZzcVGw=', 'Content-Length': '0'} 2023-07-29T00:39:33.900 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee3e6b78cf0dc27d-0064c45fc5-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:33.901 DEBUG:boto:b'hiiqos-59obj111000falseobj112023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-07-29T00:37:53.213Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:33.902 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:33.902 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:33.902 DEBUG:boto:path=/hiiqos-59/?prefix=obj11 2023-07-29T00:39:33.903 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj11 2023-07-29T00:39:33.903 DEBUG:boto:Method: GET 2023-07-29T00:39:33.904 DEBUG:boto:Path: /hiiqos-59/?prefix=obj11 2023-07-29T00:39:33.904 DEBUG:boto:Data: 2023-07-29T00:39:33.904 DEBUG:boto:Headers: {} 2023-07-29T00:39:33.905 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:33.905 DEBUG:boto:Port: 8000 2023-07-29T00:39:33.905 DEBUG:boto:Params: {} 2023-07-29T00:39:33.906 DEBUG:boto:Token: None 2023-07-29T00:39:33.906 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:33 GMT /hiiqos-59/ 2023-07-29T00:39:33.907 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AG7Y6krVy1uPXJPy7DfwAZzcVGw= 2023-07-29T00:39:33.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AG7Y6krVy1uPXJPy7DfwAZzcVGw=', 'Content-Length': '0'} 2023-07-29T00:39:33.911 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae97b8c31b69f1a4-0064c45fc5-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:33.956 DEBUG:boto:b'hiiqos-59obj111000falseobj112023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-07-29T00:37:53.213Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:33.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-07-29T00:39:33.959 DEBUG:boto:path=/hiiqos-59/obj111 2023-07-29T00:39:33.960 DEBUG:boto:auth_path=/hiiqos-59/obj111 2023-07-29T00:39:33.961 DEBUG:boto:path=/hiiqos-59/obj111?versionId=null 2023-07-29T00:39:33.961 DEBUG:boto:auth_path=/hiiqos-59/obj111?versionId=null 2023-07-29T00:39:33.962 DEBUG:boto:Method: GET 2023-07-29T00:39:33.963 DEBUG:boto:Path: /hiiqos-59/obj111?versionId=null 2023-07-29T00:39:33.964 DEBUG:boto:Data: 2023-07-29T00:39:33.965 DEBUG:boto:Headers: {} 2023-07-29T00:39:33.965 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:33.966 DEBUG:boto:Port: 8001 2023-07-29T00:39:33.967 DEBUG:boto:Params: {} 2023-07-29T00:39:33.968 DEBUG:boto:Token: None 2023-07-29T00:39:33.969 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:33 GMT /hiiqos-59/obj111?versionId=null 2023-07-29T00:39:33.970 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:B6EbJafF4DvD9v+0hq0g1x1S5/4= 2023-07-29T00:39:33.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:B6EbJafF4DvD9v+0hq0g1x1S5/4=', 'Content-Length': '0'} 2023-07-29T00:39:33.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad00ffc3052d7603-0064c45fc5-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:33.975 DEBUG:boto:path=/hiiqos-59/obj111 2023-07-29T00:39:33.976 DEBUG:boto:auth_path=/hiiqos-59/obj111 2023-07-29T00:39:33.977 DEBUG:boto:path=/hiiqos-59/obj111?versionId=null 2023-07-29T00:39:33.978 DEBUG:boto:auth_path=/hiiqos-59/obj111?versionId=null 2023-07-29T00:39:33.979 DEBUG:boto:Method: GET 2023-07-29T00:39:33.980 DEBUG:boto:Path: /hiiqos-59/obj111?versionId=null 2023-07-29T00:39:33.980 DEBUG:boto:Data: 2023-07-29T00:39:33.981 DEBUG:boto:Headers: {} 2023-07-29T00:39:33.982 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:33.982 DEBUG:boto:Port: 8000 2023-07-29T00:39:33.983 DEBUG:boto:Params: {} 2023-07-29T00:39:33.984 DEBUG:boto:Token: None 2023-07-29T00:39:33.985 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:33 GMT /hiiqos-59/obj111?versionId=null 2023-07-29T00:39:33.985 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:B6EbJafF4DvD9v+0hq0g1x1S5/4= 2023-07-29T00:39:33.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:B6EbJafF4DvD9v+0hq0g1x1S5/4=', 'Content-Length': '0'} 2023-07-29T00:39:33.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d13a602f5a2ebef-0064c45fc5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:33.988 DEBUG:boto:path=/hiiqos-59/obj111 2023-07-29T00:39:33.989 DEBUG:boto:auth_path=/hiiqos-59/obj111 2023-07-29T00:39:33.989 DEBUG:boto:path=/hiiqos-59/obj111?versionId=null 2023-07-29T00:39:33.990 DEBUG:boto:auth_path=/hiiqos-59/obj111?versionId=null 2023-07-29T00:39:33.991 DEBUG:boto:Method: HEAD 2023-07-29T00:39:33.992 DEBUG:boto:Path: /hiiqos-59/obj111?versionId=null 2023-07-29T00:39:33.992 DEBUG:boto:Data: 2023-07-29T00:39:33.993 DEBUG:boto:Headers: {} 2023-07-29T00:39:33.994 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:33.995 DEBUG:boto:Port: 8001 2023-07-29T00:39:33.995 DEBUG:boto:Params: {} 2023-07-29T00:39:33.995 DEBUG:boto:Token: None 2023-07-29T00:39:33.996 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:33 GMT /hiiqos-59/obj111?versionId=null 2023-07-29T00:39:33.996 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZMa7SUPY4guhg8ekliyDl8AVpm0= 2023-07-29T00:39:33.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZMa7SUPY4guhg8ekliyDl8AVpm0=', 'Content-Length': '0'} 2023-07-29T00:39:33.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000636bde34efe4c6e5-0064c45fc5-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:33.999 DEBUG:boto:path=/hiiqos-59/obj111 2023-07-29T00:39:33.999 DEBUG:boto:auth_path=/hiiqos-59/obj111 2023-07-29T00:39:34.000 DEBUG:boto:path=/hiiqos-59/obj111?versionId=null 2023-07-29T00:39:34.000 DEBUG:boto:auth_path=/hiiqos-59/obj111?versionId=null 2023-07-29T00:39:34.001 DEBUG:boto:Method: HEAD 2023-07-29T00:39:34.001 DEBUG:boto:Path: /hiiqos-59/obj111?versionId=null 2023-07-29T00:39:34.001 DEBUG:boto:Data: 2023-07-29T00:39:34.002 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.002 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:34.003 DEBUG:boto:Port: 8000 2023-07-29T00:39:34.003 DEBUG:boto:Params: {} 2023-07-29T00:39:34.003 DEBUG:boto:Token: None 2023-07-29T00:39:34.004 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj111?versionId=null 2023-07-29T00:39:34.004 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ffTHE5xW9drjoRJa7fquddinxN8= 2023-07-29T00:39:34.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ffTHE5xW9drjoRJa7fquddinxN8=', 'Content-Length': '0'} 2023-07-29T00:39:34.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029fdfaec1ad1fa96-0064c45fc5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-07-29T00:39:34.009 DEBUG:boto:path=/hiiqos-59/obj111 2023-07-29T00:39:34.010 DEBUG:boto:auth_path=/hiiqos-59/obj111 2023-07-29T00:39:34.011 DEBUG:boto:path=/hiiqos-59/obj111?versionId=null 2023-07-29T00:39:34.011 DEBUG:boto:auth_path=/hiiqos-59/obj111?versionId=null 2023-07-29T00:39:34.012 DEBUG:boto:Method: GET 2023-07-29T00:39:34.013 DEBUG:boto:Path: /hiiqos-59/obj111?versionId=null 2023-07-29T00:39:34.013 DEBUG:boto:Data: 2023-07-29T00:39:34.014 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.015 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:34.015 DEBUG:boto:Port: 8001 2023-07-29T00:39:34.016 DEBUG:boto:Params: {} 2023-07-29T00:39:34.016 DEBUG:boto:Token: None 2023-07-29T00:39:34.017 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj111?versionId=null 2023-07-29T00:39:34.017 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RN+gIEZu+Hc/AM5rV3HQtscaAP8= 2023-07-29T00:39:34.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RN+gIEZu+Hc/AM5rV3HQtscaAP8=', 'Content-Length': '0'} 2023-07-29T00:39:34.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ced0e16cabd4df9-0064c45fc5-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.020 DEBUG:boto:path=/hiiqos-59/obj111 2023-07-29T00:39:34.021 DEBUG:boto:auth_path=/hiiqos-59/obj111 2023-07-29T00:39:34.021 DEBUG:boto:path=/hiiqos-59/obj111?versionId=null 2023-07-29T00:39:34.021 DEBUG:boto:auth_path=/hiiqos-59/obj111?versionId=null 2023-07-29T00:39:34.022 DEBUG:boto:Method: GET 2023-07-29T00:39:34.022 DEBUG:boto:Path: /hiiqos-59/obj111?versionId=null 2023-07-29T00:39:34.022 DEBUG:boto:Data: 2023-07-29T00:39:34.023 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.023 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:34.023 DEBUG:boto:Port: 8000 2023-07-29T00:39:34.024 DEBUG:boto:Params: {} 2023-07-29T00:39:34.024 DEBUG:boto:Token: None 2023-07-29T00:39:34.025 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj111?versionId=null 2023-07-29T00:39:34.025 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RN+gIEZu+Hc/AM5rV3HQtscaAP8= 2023-07-29T00:39:34.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RN+gIEZu+Hc/AM5rV3HQtscaAP8=', 'Content-Length': '0'} 2023-07-29T00:39:34.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064ffebbf279e920d-0064c45fc5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.028 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:34.029 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:34.029 DEBUG:boto:path=/hiiqos-59/?prefix=obj111 2023-07-29T00:39:34.029 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj111 2023-07-29T00:39:34.030 DEBUG:boto:Method: GET 2023-07-29T00:39:34.030 DEBUG:boto:Path: /hiiqos-59/?prefix=obj111 2023-07-29T00:39:34.030 DEBUG:boto:Data: 2023-07-29T00:39:34.031 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.031 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:34.032 DEBUG:boto:Port: 8001 2023-07-29T00:39:34.032 DEBUG:boto:Params: {} 2023-07-29T00:39:34.032 DEBUG:boto:Token: None 2023-07-29T00:39:34.033 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/ 2023-07-29T00:39:34.033 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S+Edt2AVsKtdLg5f775TAKithZ8= 2023-07-29T00:39:34.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S+Edt2AVsKtdLg5f775TAKithZ8=', 'Content-Length': '0'} 2023-07-29T00:39:34.036 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2a8791dbaafcf0f-0064c45fc5-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.037 DEBUG:boto:b'hiiqos-59obj1111000falseobj1112023-07-29T00:37:53.213Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:34.037 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:34.038 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:34.038 DEBUG:boto:path=/hiiqos-59/?prefix=obj111 2023-07-29T00:39:34.038 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj111 2023-07-29T00:39:34.039 DEBUG:boto:Method: GET 2023-07-29T00:39:34.039 DEBUG:boto:Path: /hiiqos-59/?prefix=obj111 2023-07-29T00:39:34.039 DEBUG:boto:Data: 2023-07-29T00:39:34.040 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.040 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:34.041 DEBUG:boto:Port: 8000 2023-07-29T00:39:34.041 DEBUG:boto:Params: {} 2023-07-29T00:39:34.041 DEBUG:boto:Token: None 2023-07-29T00:39:34.042 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/ 2023-07-29T00:39:34.042 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S+Edt2AVsKtdLg5f775TAKithZ8= 2023-07-29T00:39:34.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S+Edt2AVsKtdLg5f775TAKithZ8=', 'Content-Length': '0'} 2023-07-29T00:39:34.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a2a323727e64da0-0064c45fc5-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.047 DEBUG:boto:b'hiiqos-59obj1111000falseobj1112023-07-29T00:37:53.213Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:34.048 DEBUG:boto:path=/hiiqos-59/obj111 2023-07-29T00:39:34.048 DEBUG:boto:auth_path=/hiiqos-59/obj111 2023-07-29T00:39:34.049 DEBUG:boto:Method: HEAD 2023-07-29T00:39:34.049 DEBUG:boto:Path: /hiiqos-59/obj111 2023-07-29T00:39:34.049 DEBUG:boto:Data: 2023-07-29T00:39:34.050 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.050 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:34.050 DEBUG:boto:Port: 8001 2023-07-29T00:39:34.051 DEBUG:boto:Params: {} 2023-07-29T00:39:34.051 DEBUG:boto:Token: None 2023-07-29T00:39:34.052 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj111 2023-07-29T00:39:34.052 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DWdpCZUUGp8OY77Ji8T8Lcoo66E= 2023-07-29T00:39:34.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DWdpCZUUGp8OY77Ji8T8Lcoo66E=', 'Content-Length': '0'} 2023-07-29T00:39:34.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d485e295f1cd0f5-0064c45fc5-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.055 DEBUG:boto:path=/hiiqos-59/obj111 2023-07-29T00:39:34.055 DEBUG:boto:auth_path=/hiiqos-59/obj111 2023-07-29T00:39:34.056 DEBUG:boto:Method: HEAD 2023-07-29T00:39:34.056 DEBUG:boto:Path: /hiiqos-59/obj111 2023-07-29T00:39:34.056 DEBUG:boto:Data: 2023-07-29T00:39:34.057 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.057 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:34.058 DEBUG:boto:Port: 8000 2023-07-29T00:39:34.058 DEBUG:boto:Params: {} 2023-07-29T00:39:34.058 DEBUG:boto:Token: None 2023-07-29T00:39:34.059 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj111 2023-07-29T00:39:34.059 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DWdpCZUUGp8OY77Ji8T8Lcoo66E= 2023-07-29T00:39:34.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DWdpCZUUGp8OY77Ji8T8Lcoo66E=', 'Content-Length': '0'} 2023-07-29T00:39:34.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8e7ad34e09ceac3-0064c45fc5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-07-29T00:39:34.062 DEBUG:boto:path=/hiiqos-59/obj111 2023-07-29T00:39:34.063 DEBUG:boto:auth_path=/hiiqos-59/obj111 2023-07-29T00:39:34.063 DEBUG:boto:Method: GET 2023-07-29T00:39:34.064 DEBUG:boto:Path: /hiiqos-59/obj111 2023-07-29T00:39:34.064 DEBUG:boto:Data: 2023-07-29T00:39:34.064 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.065 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:34.065 DEBUG:boto:Port: 8001 2023-07-29T00:39:34.065 DEBUG:boto:Params: {} 2023-07-29T00:39:34.066 DEBUG:boto:Token: None 2023-07-29T00:39:34.066 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj111 2023-07-29T00:39:34.067 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SSlKTS5nsrxzXhoRUPLkRDQgkWY= 2023-07-29T00:39:34.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SSlKTS5nsrxzXhoRUPLkRDQgkWY=', 'Content-Length': '0'} 2023-07-29T00:39:34.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf2ff936d0680032-0064c45fc6-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.070 DEBUG:boto:path=/hiiqos-59/obj111 2023-07-29T00:39:34.070 DEBUG:boto:auth_path=/hiiqos-59/obj111 2023-07-29T00:39:34.071 DEBUG:boto:Method: GET 2023-07-29T00:39:34.071 DEBUG:boto:Path: /hiiqos-59/obj111 2023-07-29T00:39:34.071 DEBUG:boto:Data: 2023-07-29T00:39:34.072 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.072 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:34.073 DEBUG:boto:Port: 8000 2023-07-29T00:39:34.073 DEBUG:boto:Params: {} 2023-07-29T00:39:34.074 DEBUG:boto:Token: None 2023-07-29T00:39:34.074 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj111 2023-07-29T00:39:34.075 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SSlKTS5nsrxzXhoRUPLkRDQgkWY= 2023-07-29T00:39:34.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SSlKTS5nsrxzXhoRUPLkRDQgkWY=', 'Content-Length': '0'} 2023-07-29T00:39:34.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000700306346d0e63d0-0064c45fc6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.078 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:34.078 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:34.078 DEBUG:boto:path=/hiiqos-59/?prefix=obj111 2023-07-29T00:39:34.079 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj111 2023-07-29T00:39:34.079 DEBUG:boto:Method: GET 2023-07-29T00:39:34.079 DEBUG:boto:Path: /hiiqos-59/?prefix=obj111 2023-07-29T00:39:34.080 DEBUG:boto:Data: 2023-07-29T00:39:34.080 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.081 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:34.081 DEBUG:boto:Port: 8001 2023-07-29T00:39:34.081 DEBUG:boto:Params: {} 2023-07-29T00:39:34.082 DEBUG:boto:Token: None 2023-07-29T00:39:34.082 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/ 2023-07-29T00:39:34.083 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S+Edt2AVsKtdLg5f775TAKithZ8= 2023-07-29T00:39:34.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S+Edt2AVsKtdLg5f775TAKithZ8=', 'Content-Length': '0'} 2023-07-29T00:39:34.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cea68fb2f7b18733-0064c45fc6-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.128 DEBUG:boto:b'hiiqos-59obj1111000falseobj1112023-07-29T00:37:53.213Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:34.129 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:34.130 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:34.130 DEBUG:boto:path=/hiiqos-59/?prefix=obj111 2023-07-29T00:39:34.130 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj111 2023-07-29T00:39:34.131 DEBUG:boto:Method: GET 2023-07-29T00:39:34.131 DEBUG:boto:Path: /hiiqos-59/?prefix=obj111 2023-07-29T00:39:34.131 DEBUG:boto:Data: 2023-07-29T00:39:34.132 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.132 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:34.132 DEBUG:boto:Port: 8000 2023-07-29T00:39:34.133 DEBUG:boto:Params: {} 2023-07-29T00:39:34.133 DEBUG:boto:Token: None 2023-07-29T00:39:34.134 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/ 2023-07-29T00:39:34.134 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S+Edt2AVsKtdLg5f775TAKithZ8= 2023-07-29T00:39:34.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S+Edt2AVsKtdLg5f775TAKithZ8=', 'Content-Length': '0'} 2023-07-29T00:39:34.139 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbdca84f37ed29f6-0064c45fc6-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.139 DEBUG:boto:b'hiiqos-59obj1111000falseobj1112023-07-29T00:37:53.213Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:34.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-07-29T00:39:34.140 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:39:34.141 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:39:34.141 DEBUG:boto:path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:39:34.141 DEBUG:boto:auth_path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:39:34.142 DEBUG:boto:Method: GET 2023-07-29T00:39:34.142 DEBUG:boto:Path: /hiiqos-59/obj115?versionId=null 2023-07-29T00:39:34.142 DEBUG:boto:Data: 2023-07-29T00:39:34.143 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.143 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:34.144 DEBUG:boto:Port: 8001 2023-07-29T00:39:34.144 DEBUG:boto:Params: {} 2023-07-29T00:39:34.144 DEBUG:boto:Token: None 2023-07-29T00:39:34.145 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj115?versionId=null 2023-07-29T00:39:34.145 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ox94mz1UqG4j+XjUpuEn4nIxYpw= 2023-07-29T00:39:34.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ox94mz1UqG4j+XjUpuEn4nIxYpw=', 'Content-Length': '0'} 2023-07-29T00:39:34.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003504fb45e0062dea-0064c45fc6-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.148 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:39:34.149 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:39:34.149 DEBUG:boto:path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:39:34.150 DEBUG:boto:auth_path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:39:34.150 DEBUG:boto:Method: GET 2023-07-29T00:39:34.150 DEBUG:boto:Path: /hiiqos-59/obj115?versionId=null 2023-07-29T00:39:34.151 DEBUG:boto:Data: 2023-07-29T00:39:34.151 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.151 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:34.152 DEBUG:boto:Port: 8000 2023-07-29T00:39:34.152 DEBUG:boto:Params: {} 2023-07-29T00:39:34.153 DEBUG:boto:Token: None 2023-07-29T00:39:34.153 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj115?versionId=null 2023-07-29T00:39:34.153 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ox94mz1UqG4j+XjUpuEn4nIxYpw= 2023-07-29T00:39:34.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ox94mz1UqG4j+XjUpuEn4nIxYpw=', 'Content-Length': '0'} 2023-07-29T00:39:34.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b9b6dd9c4250f74-0064c45fc6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.196 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:39:34.197 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:39:34.197 DEBUG:boto:path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:39:34.198 DEBUG:boto:auth_path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:39:34.198 DEBUG:boto:Method: HEAD 2023-07-29T00:39:34.198 DEBUG:boto:Path: /hiiqos-59/obj115?versionId=null 2023-07-29T00:39:34.199 DEBUG:boto:Data: 2023-07-29T00:39:34.199 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.199 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:34.200 DEBUG:boto:Port: 8001 2023-07-29T00:39:34.200 DEBUG:boto:Params: {} 2023-07-29T00:39:34.200 DEBUG:boto:Token: None 2023-07-29T00:39:34.201 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj115?versionId=null 2023-07-29T00:39:34.201 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MuBhAUFnBNMFYloYNlDajn0+j1w= 2023-07-29T00:39:34.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MuBhAUFnBNMFYloYNlDajn0+j1w=', 'Content-Length': '0'} 2023-07-29T00:39:34.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ccf587e6e3abc74-0064c45fc6-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.204 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:39:34.205 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:39:34.205 DEBUG:boto:path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:39:34.205 DEBUG:boto:auth_path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:39:34.206 DEBUG:boto:Method: HEAD 2023-07-29T00:39:34.206 DEBUG:boto:Path: /hiiqos-59/obj115?versionId=null 2023-07-29T00:39:34.206 DEBUG:boto:Data: 2023-07-29T00:39:34.207 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.207 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:34.207 DEBUG:boto:Port: 8000 2023-07-29T00:39:34.208 DEBUG:boto:Params: {} 2023-07-29T00:39:34.208 DEBUG:boto:Token: None 2023-07-29T00:39:34.208 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj115?versionId=null 2023-07-29T00:39:34.209 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MuBhAUFnBNMFYloYNlDajn0+j1w= 2023-07-29T00:39:34.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MuBhAUFnBNMFYloYNlDajn0+j1w=', 'Content-Length': '0'} 2023-07-29T00:39:34.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000205d1661fd5d012c-0064c45fc6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-07-29T00:39:34.212 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:39:34.213 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:39:34.213 DEBUG:boto:path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:39:34.213 DEBUG:boto:auth_path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:39:34.214 DEBUG:boto:Method: GET 2023-07-29T00:39:34.214 DEBUG:boto:Path: /hiiqos-59/obj115?versionId=null 2023-07-29T00:39:34.214 DEBUG:boto:Data: 2023-07-29T00:39:34.215 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.215 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:34.216 DEBUG:boto:Port: 8001 2023-07-29T00:39:34.216 DEBUG:boto:Params: {} 2023-07-29T00:39:34.217 DEBUG:boto:Token: None 2023-07-29T00:39:34.217 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj115?versionId=null 2023-07-29T00:39:34.217 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ox94mz1UqG4j+XjUpuEn4nIxYpw= 2023-07-29T00:39:34.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ox94mz1UqG4j+XjUpuEn4nIxYpw=', 'Content-Length': '0'} 2023-07-29T00:39:34.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e38023965543c6d9-0064c45fc6-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.220 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:39:34.221 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:39:34.221 DEBUG:boto:path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:39:34.221 DEBUG:boto:auth_path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:39:34.222 DEBUG:boto:Method: GET 2023-07-29T00:39:34.222 DEBUG:boto:Path: /hiiqos-59/obj115?versionId=null 2023-07-29T00:39:34.223 DEBUG:boto:Data: 2023-07-29T00:39:34.223 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.223 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:34.224 DEBUG:boto:Port: 8000 2023-07-29T00:39:34.224 DEBUG:boto:Params: {} 2023-07-29T00:39:34.224 DEBUG:boto:Token: None 2023-07-29T00:39:34.225 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj115?versionId=null 2023-07-29T00:39:34.225 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ox94mz1UqG4j+XjUpuEn4nIxYpw= 2023-07-29T00:39:34.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ox94mz1UqG4j+XjUpuEn4nIxYpw=', 'Content-Length': '0'} 2023-07-29T00:39:34.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f12be78d2e4c490-0064c45fc6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.228 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:34.229 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:34.229 DEBUG:boto:path=/hiiqos-59/?prefix=obj115 2023-07-29T00:39:34.230 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj115 2023-07-29T00:39:34.231 DEBUG:boto:Method: GET 2023-07-29T00:39:34.232 DEBUG:boto:Path: /hiiqos-59/?prefix=obj115 2023-07-29T00:39:34.232 DEBUG:boto:Data: 2023-07-29T00:39:34.233 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.234 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:34.235 DEBUG:boto:Port: 8001 2023-07-29T00:39:34.236 DEBUG:boto:Params: {} 2023-07-29T00:39:34.237 DEBUG:boto:Token: None 2023-07-29T00:39:34.237 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/ 2023-07-29T00:39:34.237 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S+Edt2AVsKtdLg5f775TAKithZ8= 2023-07-29T00:39:34.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S+Edt2AVsKtdLg5f775TAKithZ8=', 'Content-Length': '0'} 2023-07-29T00:39:34.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a71d8859eb52309f-0064c45fc6-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.241 DEBUG:boto:b'hiiqos-59obj1151000falseobj1152023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:34.242 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:34.242 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:34.243 DEBUG:boto:path=/hiiqos-59/?prefix=obj115 2023-07-29T00:39:34.243 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj115 2023-07-29T00:39:34.243 DEBUG:boto:Method: GET 2023-07-29T00:39:34.244 DEBUG:boto:Path: /hiiqos-59/?prefix=obj115 2023-07-29T00:39:34.244 DEBUG:boto:Data: 2023-07-29T00:39:34.245 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.245 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:34.246 DEBUG:boto:Port: 8000 2023-07-29T00:39:34.246 DEBUG:boto:Params: {} 2023-07-29T00:39:34.246 DEBUG:boto:Token: None 2023-07-29T00:39:34.247 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/ 2023-07-29T00:39:34.247 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S+Edt2AVsKtdLg5f775TAKithZ8= 2023-07-29T00:39:34.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S+Edt2AVsKtdLg5f775TAKithZ8=', 'Content-Length': '0'} 2023-07-29T00:39:34.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000168c0999ad64ed4b-0064c45fc6-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.253 DEBUG:boto:b'hiiqos-59obj1151000falseobj1152023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:34.253 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:39:34.254 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:39:34.254 DEBUG:boto:Method: HEAD 2023-07-29T00:39:34.254 DEBUG:boto:Path: /hiiqos-59/obj115 2023-07-29T00:39:34.255 DEBUG:boto:Data: 2023-07-29T00:39:34.255 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.255 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:34.256 DEBUG:boto:Port: 8001 2023-07-29T00:39:34.256 DEBUG:boto:Params: {} 2023-07-29T00:39:34.256 DEBUG:boto:Token: None 2023-07-29T00:39:34.257 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj115 2023-07-29T00:39:34.257 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1RSgBi4zl1K2N2aGgrO/xqKIgIw= 2023-07-29T00:39:34.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1RSgBi4zl1K2N2aGgrO/xqKIgIw=', 'Content-Length': '0'} 2023-07-29T00:39:34.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4ea1ccab86b4c81-0064c45fc6-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.260 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:39:34.261 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:39:34.261 DEBUG:boto:Method: HEAD 2023-07-29T00:39:34.261 DEBUG:boto:Path: /hiiqos-59/obj115 2023-07-29T00:39:34.262 DEBUG:boto:Data: 2023-07-29T00:39:34.262 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.263 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:34.263 DEBUG:boto:Port: 8000 2023-07-29T00:39:34.263 DEBUG:boto:Params: {} 2023-07-29T00:39:34.264 DEBUG:boto:Token: None 2023-07-29T00:39:34.264 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj115 2023-07-29T00:39:34.264 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1RSgBi4zl1K2N2aGgrO/xqKIgIw= 2023-07-29T00:39:34.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1RSgBi4zl1K2N2aGgrO/xqKIgIw=', 'Content-Length': '0'} 2023-07-29T00:39:34.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f09f8f68b485b6a8-0064c45fc6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-07-29T00:39:34.268 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:39:34.268 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:39:34.269 DEBUG:boto:Method: GET 2023-07-29T00:39:34.269 DEBUG:boto:Path: /hiiqos-59/obj115 2023-07-29T00:39:34.270 DEBUG:boto:Data: 2023-07-29T00:39:34.270 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.270 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:34.271 DEBUG:boto:Port: 8001 2023-07-29T00:39:34.271 DEBUG:boto:Params: {} 2023-07-29T00:39:34.271 DEBUG:boto:Token: None 2023-07-29T00:39:34.272 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj115 2023-07-29T00:39:34.272 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4IC4j0N29AGZzVAwYAr+IxLYCJs= 2023-07-29T00:39:34.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4IC4j0N29AGZzVAwYAr+IxLYCJs=', 'Content-Length': '0'} 2023-07-29T00:39:34.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fd8d4c76991638c-0064c45fc6-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.275 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:39:34.276 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:39:34.276 DEBUG:boto:Method: GET 2023-07-29T00:39:34.277 DEBUG:boto:Path: /hiiqos-59/obj115 2023-07-29T00:39:34.277 DEBUG:boto:Data: 2023-07-29T00:39:34.277 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.278 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:34.278 DEBUG:boto:Port: 8000 2023-07-29T00:39:34.278 DEBUG:boto:Params: {} 2023-07-29T00:39:34.279 DEBUG:boto:Token: None 2023-07-29T00:39:34.279 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj115 2023-07-29T00:39:34.280 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4IC4j0N29AGZzVAwYAr+IxLYCJs= 2023-07-29T00:39:34.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4IC4j0N29AGZzVAwYAr+IxLYCJs=', 'Content-Length': '0'} 2023-07-29T00:39:34.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040a2cd867e3c8546-0064c45fc6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.284 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:34.285 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:34.285 DEBUG:boto:path=/hiiqos-59/?prefix=obj115 2023-07-29T00:39:34.286 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj115 2023-07-29T00:39:34.287 DEBUG:boto:Method: GET 2023-07-29T00:39:34.288 DEBUG:boto:Path: /hiiqos-59/?prefix=obj115 2023-07-29T00:39:34.289 DEBUG:boto:Data: 2023-07-29T00:39:34.289 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.290 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:34.291 DEBUG:boto:Port: 8001 2023-07-29T00:39:34.292 DEBUG:boto:Params: {} 2023-07-29T00:39:34.293 DEBUG:boto:Token: None 2023-07-29T00:39:34.294 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/ 2023-07-29T00:39:34.295 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S+Edt2AVsKtdLg5f775TAKithZ8= 2023-07-29T00:39:34.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S+Edt2AVsKtdLg5f775TAKithZ8=', 'Content-Length': '0'} 2023-07-29T00:39:34.299 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001522a83c581bee51-0064c45fc6-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.299 DEBUG:boto:b'hiiqos-59obj1151000falseobj1152023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:34.301 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:34.301 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:34.302 DEBUG:boto:path=/hiiqos-59/?prefix=obj115 2023-07-29T00:39:34.303 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj115 2023-07-29T00:39:34.303 DEBUG:boto:Method: GET 2023-07-29T00:39:34.304 DEBUG:boto:Path: /hiiqos-59/?prefix=obj115 2023-07-29T00:39:34.305 DEBUG:boto:Data: 2023-07-29T00:39:34.306 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.307 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:34.308 DEBUG:boto:Port: 8000 2023-07-29T00:39:34.309 DEBUG:boto:Params: {} 2023-07-29T00:39:34.309 DEBUG:boto:Token: None 2023-07-29T00:39:34.310 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/ 2023-07-29T00:39:34.310 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S+Edt2AVsKtdLg5f775TAKithZ8= 2023-07-29T00:39:34.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S+Edt2AVsKtdLg5f775TAKithZ8=', 'Content-Length': '0'} 2023-07-29T00:39:34.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045b289e1022b4c20-0064c45fc6-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.316 DEBUG:boto:b'hiiqos-59obj1151000falseobj1152023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:34.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-07-29T00:39:34.317 DEBUG:boto:path=/hiiqos-59/obj116 2023-07-29T00:39:34.317 DEBUG:boto:auth_path=/hiiqos-59/obj116 2023-07-29T00:39:34.318 DEBUG:boto:path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:39:34.318 DEBUG:boto:auth_path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:39:34.318 DEBUG:boto:Method: GET 2023-07-29T00:39:34.319 DEBUG:boto:Path: /hiiqos-59/obj116?versionId=null 2023-07-29T00:39:34.319 DEBUG:boto:Data: 2023-07-29T00:39:34.319 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.320 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:34.320 DEBUG:boto:Port: 8001 2023-07-29T00:39:34.320 DEBUG:boto:Params: {} 2023-07-29T00:39:34.321 DEBUG:boto:Token: None 2023-07-29T00:39:34.321 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj116?versionId=null 2023-07-29T00:39:34.322 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pXfx3H1gDHJIPTxWS+X+2BZIT6g= 2023-07-29T00:39:34.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pXfx3H1gDHJIPTxWS+X+2BZIT6g=', 'Content-Length': '0'} 2023-07-29T00:39:34.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc573eab564d5293-0064c45fc6-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.325 DEBUG:boto:path=/hiiqos-59/obj116 2023-07-29T00:39:34.325 DEBUG:boto:auth_path=/hiiqos-59/obj116 2023-07-29T00:39:34.326 DEBUG:boto:path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:39:34.326 DEBUG:boto:auth_path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:39:34.326 DEBUG:boto:Method: GET 2023-07-29T00:39:34.327 DEBUG:boto:Path: /hiiqos-59/obj116?versionId=null 2023-07-29T00:39:34.327 DEBUG:boto:Data: 2023-07-29T00:39:34.327 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.328 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:34.328 DEBUG:boto:Port: 8000 2023-07-29T00:39:34.329 DEBUG:boto:Params: {} 2023-07-29T00:39:34.330 DEBUG:boto:Token: None 2023-07-29T00:39:34.330 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj116?versionId=null 2023-07-29T00:39:34.331 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pXfx3H1gDHJIPTxWS+X+2BZIT6g= 2023-07-29T00:39:34.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pXfx3H1gDHJIPTxWS+X+2BZIT6g=', 'Content-Length': '0'} 2023-07-29T00:39:34.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfde416831ba33b4-0064c45fc6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.376 DEBUG:boto:path=/hiiqos-59/obj116 2023-07-29T00:39:34.377 DEBUG:boto:auth_path=/hiiqos-59/obj116 2023-07-29T00:39:34.377 DEBUG:boto:path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:39:34.378 DEBUG:boto:auth_path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:39:34.378 DEBUG:boto:Method: HEAD 2023-07-29T00:39:34.379 DEBUG:boto:Path: /hiiqos-59/obj116?versionId=null 2023-07-29T00:39:34.379 DEBUG:boto:Data: 2023-07-29T00:39:34.380 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.381 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:34.382 DEBUG:boto:Port: 8001 2023-07-29T00:39:34.382 DEBUG:boto:Params: {} 2023-07-29T00:39:34.383 DEBUG:boto:Token: None 2023-07-29T00:39:34.384 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj116?versionId=null 2023-07-29T00:39:34.384 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kqr5SiRj+ocrPdhXPvdcqLz3gJU= 2023-07-29T00:39:34.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kqr5SiRj+ocrPdhXPvdcqLz3gJU=', 'Content-Length': '0'} 2023-07-29T00:39:34.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed4007f6d89977e4-0064c45fc6-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.388 DEBUG:boto:path=/hiiqos-59/obj116 2023-07-29T00:39:34.389 DEBUG:boto:auth_path=/hiiqos-59/obj116 2023-07-29T00:39:34.389 DEBUG:boto:path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:39:34.389 DEBUG:boto:auth_path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:39:34.390 DEBUG:boto:Method: HEAD 2023-07-29T00:39:34.390 DEBUG:boto:Path: /hiiqos-59/obj116?versionId=null 2023-07-29T00:39:34.390 DEBUG:boto:Data: 2023-07-29T00:39:34.391 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.392 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:34.392 DEBUG:boto:Port: 8000 2023-07-29T00:39:34.393 DEBUG:boto:Params: {} 2023-07-29T00:39:34.394 DEBUG:boto:Token: None 2023-07-29T00:39:34.395 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj116?versionId=null 2023-07-29T00:39:34.396 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kqr5SiRj+ocrPdhXPvdcqLz3gJU= 2023-07-29T00:39:34.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kqr5SiRj+ocrPdhXPvdcqLz3gJU=', 'Content-Length': '0'} 2023-07-29T00:39:34.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006363ddeed570e0a1-0064c45fc6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-07-29T00:39:34.400 DEBUG:boto:path=/hiiqos-59/obj116 2023-07-29T00:39:34.400 DEBUG:boto:auth_path=/hiiqos-59/obj116 2023-07-29T00:39:34.401 DEBUG:boto:path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:39:34.401 DEBUG:boto:auth_path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:39:34.401 DEBUG:boto:Method: GET 2023-07-29T00:39:34.402 DEBUG:boto:Path: /hiiqos-59/obj116?versionId=null 2023-07-29T00:39:34.402 DEBUG:boto:Data: 2023-07-29T00:39:34.403 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.403 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:34.404 DEBUG:boto:Port: 8001 2023-07-29T00:39:34.404 DEBUG:boto:Params: {} 2023-07-29T00:39:34.405 DEBUG:boto:Token: None 2023-07-29T00:39:34.405 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj116?versionId=null 2023-07-29T00:39:34.406 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pXfx3H1gDHJIPTxWS+X+2BZIT6g= 2023-07-29T00:39:34.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pXfx3H1gDHJIPTxWS+X+2BZIT6g=', 'Content-Length': '0'} 2023-07-29T00:39:34.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003856772287f5d48e-0064c45fc6-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.409 DEBUG:boto:path=/hiiqos-59/obj116 2023-07-29T00:39:34.409 DEBUG:boto:auth_path=/hiiqos-59/obj116 2023-07-29T00:39:34.410 DEBUG:boto:path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:39:34.410 DEBUG:boto:auth_path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:39:34.410 DEBUG:boto:Method: GET 2023-07-29T00:39:34.411 DEBUG:boto:Path: /hiiqos-59/obj116?versionId=null 2023-07-29T00:39:34.411 DEBUG:boto:Data: 2023-07-29T00:39:34.411 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.412 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:34.412 DEBUG:boto:Port: 8000 2023-07-29T00:39:34.412 DEBUG:boto:Params: {} 2023-07-29T00:39:34.413 DEBUG:boto:Token: None 2023-07-29T00:39:34.413 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj116?versionId=null 2023-07-29T00:39:34.414 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pXfx3H1gDHJIPTxWS+X+2BZIT6g= 2023-07-29T00:39:34.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pXfx3H1gDHJIPTxWS+X+2BZIT6g=', 'Content-Length': '0'} 2023-07-29T00:39:34.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4c0ab645768bf25-0064c45fc6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.417 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:34.417 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:34.418 DEBUG:boto:path=/hiiqos-59/?prefix=obj116 2023-07-29T00:39:34.418 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj116 2023-07-29T00:39:34.418 DEBUG:boto:Method: GET 2023-07-29T00:39:34.419 DEBUG:boto:Path: /hiiqos-59/?prefix=obj116 2023-07-29T00:39:34.419 DEBUG:boto:Data: 2023-07-29T00:39:34.420 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.420 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:34.420 DEBUG:boto:Port: 8001 2023-07-29T00:39:34.421 DEBUG:boto:Params: {} 2023-07-29T00:39:34.421 DEBUG:boto:Token: None 2023-07-29T00:39:34.422 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/ 2023-07-29T00:39:34.422 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S+Edt2AVsKtdLg5f775TAKithZ8= 2023-07-29T00:39:34.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S+Edt2AVsKtdLg5f775TAKithZ8=', 'Content-Length': '0'} 2023-07-29T00:39:34.426 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f2d1b1897e90873-0064c45fc6-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.468 DEBUG:boto:b'hiiqos-59obj1161000falseobj1162023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:34.469 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:34.469 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:34.470 DEBUG:boto:path=/hiiqos-59/?prefix=obj116 2023-07-29T00:39:34.470 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj116 2023-07-29T00:39:34.470 DEBUG:boto:Method: GET 2023-07-29T00:39:34.471 DEBUG:boto:Path: /hiiqos-59/?prefix=obj116 2023-07-29T00:39:34.471 DEBUG:boto:Data: 2023-07-29T00:39:34.472 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.472 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:34.472 DEBUG:boto:Port: 8000 2023-07-29T00:39:34.473 DEBUG:boto:Params: {} 2023-07-29T00:39:34.473 DEBUG:boto:Token: None 2023-07-29T00:39:34.474 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/ 2023-07-29T00:39:34.474 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S+Edt2AVsKtdLg5f775TAKithZ8= 2023-07-29T00:39:34.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S+Edt2AVsKtdLg5f775TAKithZ8=', 'Content-Length': '0'} 2023-07-29T00:39:34.479 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec2e61c30172cfc3-0064c45fc6-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.479 DEBUG:boto:b'hiiqos-59obj1161000falseobj1162023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:34.480 DEBUG:boto:path=/hiiqos-59/obj116 2023-07-29T00:39:34.481 DEBUG:boto:auth_path=/hiiqos-59/obj116 2023-07-29T00:39:34.481 DEBUG:boto:Method: HEAD 2023-07-29T00:39:34.481 DEBUG:boto:Path: /hiiqos-59/obj116 2023-07-29T00:39:34.482 DEBUG:boto:Data: 2023-07-29T00:39:34.482 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.482 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:34.483 DEBUG:boto:Port: 8001 2023-07-29T00:39:34.483 DEBUG:boto:Params: {} 2023-07-29T00:39:34.483 DEBUG:boto:Token: None 2023-07-29T00:39:34.484 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj116 2023-07-29T00:39:34.484 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nOpRAAZNPAQoQvnbtZvyct256qg= 2023-07-29T00:39:34.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nOpRAAZNPAQoQvnbtZvyct256qg=', 'Content-Length': '0'} 2023-07-29T00:39:34.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ac0560907a0ccdf-0064c45fc6-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.487 DEBUG:boto:path=/hiiqos-59/obj116 2023-07-29T00:39:34.488 DEBUG:boto:auth_path=/hiiqos-59/obj116 2023-07-29T00:39:34.488 DEBUG:boto:Method: HEAD 2023-07-29T00:39:34.489 DEBUG:boto:Path: /hiiqos-59/obj116 2023-07-29T00:39:34.489 DEBUG:boto:Data: 2023-07-29T00:39:34.489 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.490 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:34.490 DEBUG:boto:Port: 8000 2023-07-29T00:39:34.490 DEBUG:boto:Params: {} 2023-07-29T00:39:34.491 DEBUG:boto:Token: None 2023-07-29T00:39:34.491 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj116 2023-07-29T00:39:34.491 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nOpRAAZNPAQoQvnbtZvyct256qg= 2023-07-29T00:39:34.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nOpRAAZNPAQoQvnbtZvyct256qg=', 'Content-Length': '0'} 2023-07-29T00:39:34.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dd14287b38a3a24-0064c45fc6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-07-29T00:39:34.495 DEBUG:boto:path=/hiiqos-59/obj116 2023-07-29T00:39:34.495 DEBUG:boto:auth_path=/hiiqos-59/obj116 2023-07-29T00:39:34.496 DEBUG:boto:Method: GET 2023-07-29T00:39:34.496 DEBUG:boto:Path: /hiiqos-59/obj116 2023-07-29T00:39:34.497 DEBUG:boto:Data: 2023-07-29T00:39:34.497 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.498 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:34.498 DEBUG:boto:Port: 8001 2023-07-29T00:39:34.498 DEBUG:boto:Params: {} 2023-07-29T00:39:34.499 DEBUG:boto:Token: None 2023-07-29T00:39:34.499 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj116 2023-07-29T00:39:34.499 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jmhHHQI5Lr8cW4KZEkV2gNS61CI= 2023-07-29T00:39:34.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jmhHHQI5Lr8cW4KZEkV2gNS61CI=', 'Content-Length': '0'} 2023-07-29T00:39:34.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d8180c38c554da3-0064c45fc6-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.502 DEBUG:boto:path=/hiiqos-59/obj116 2023-07-29T00:39:34.503 DEBUG:boto:auth_path=/hiiqos-59/obj116 2023-07-29T00:39:34.503 DEBUG:boto:Method: GET 2023-07-29T00:39:34.504 DEBUG:boto:Path: /hiiqos-59/obj116 2023-07-29T00:39:34.504 DEBUG:boto:Data: 2023-07-29T00:39:34.504 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.505 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:34.505 DEBUG:boto:Port: 8000 2023-07-29T00:39:34.506 DEBUG:boto:Params: {} 2023-07-29T00:39:34.506 DEBUG:boto:Token: None 2023-07-29T00:39:34.506 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj116 2023-07-29T00:39:34.507 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jmhHHQI5Lr8cW4KZEkV2gNS61CI= 2023-07-29T00:39:34.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jmhHHQI5Lr8cW4KZEkV2gNS61CI=', 'Content-Length': '0'} 2023-07-29T00:39:34.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1205510bee41362-0064c45fc6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.510 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:34.511 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:34.511 DEBUG:boto:path=/hiiqos-59/?prefix=obj116 2023-07-29T00:39:34.511 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj116 2023-07-29T00:39:34.512 DEBUG:boto:Method: GET 2023-07-29T00:39:34.512 DEBUG:boto:Path: /hiiqos-59/?prefix=obj116 2023-07-29T00:39:34.512 DEBUG:boto:Data: 2023-07-29T00:39:34.513 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.513 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:34.514 DEBUG:boto:Port: 8001 2023-07-29T00:39:34.514 DEBUG:boto:Params: {} 2023-07-29T00:39:34.514 DEBUG:boto:Token: None 2023-07-29T00:39:34.515 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/ 2023-07-29T00:39:34.515 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S+Edt2AVsKtdLg5f775TAKithZ8= 2023-07-29T00:39:34.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S+Edt2AVsKtdLg5f775TAKithZ8=', 'Content-Length': '0'} 2023-07-29T00:39:34.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c496da0805307873-0064c45fc6-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.518 DEBUG:boto:b'hiiqos-59obj1161000falseobj1162023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:34.519 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:34.520 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:34.520 DEBUG:boto:path=/hiiqos-59/?prefix=obj116 2023-07-29T00:39:34.520 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj116 2023-07-29T00:39:34.521 DEBUG:boto:Method: GET 2023-07-29T00:39:34.521 DEBUG:boto:Path: /hiiqos-59/?prefix=obj116 2023-07-29T00:39:34.521 DEBUG:boto:Data: 2023-07-29T00:39:34.522 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.522 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:34.523 DEBUG:boto:Port: 8000 2023-07-29T00:39:34.523 DEBUG:boto:Params: {} 2023-07-29T00:39:34.523 DEBUG:boto:Token: None 2023-07-29T00:39:34.524 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/ 2023-07-29T00:39:34.524 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S+Edt2AVsKtdLg5f775TAKithZ8= 2023-07-29T00:39:34.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S+Edt2AVsKtdLg5f775TAKithZ8=', 'Content-Length': '0'} 2023-07-29T00:39:34.529 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000002de9180b8ceac5-0064c45fc6-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.576 DEBUG:boto:b'hiiqos-59obj1161000falseobj1162023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:34.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-07-29T00:39:34.581 DEBUG:boto:path=/hiiqos-59/obj124 2023-07-29T00:39:34.581 DEBUG:boto:auth_path=/hiiqos-59/obj124 2023-07-29T00:39:34.582 DEBUG:boto:path=/hiiqos-59/obj124?versionId=null 2023-07-29T00:39:34.582 DEBUG:boto:auth_path=/hiiqos-59/obj124?versionId=null 2023-07-29T00:39:34.582 DEBUG:boto:Method: GET 2023-07-29T00:39:34.583 DEBUG:boto:Path: /hiiqos-59/obj124?versionId=null 2023-07-29T00:39:34.583 DEBUG:boto:Data: 2023-07-29T00:39:34.583 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.584 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:34.584 DEBUG:boto:Port: 8001 2023-07-29T00:39:34.585 DEBUG:boto:Params: {} 2023-07-29T00:39:34.585 DEBUG:boto:Token: None 2023-07-29T00:39:34.586 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj124?versionId=null 2023-07-29T00:39:34.586 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kgcnyo0W2DH5pW5vlgA//fVrFuE= 2023-07-29T00:39:34.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kgcnyo0W2DH5pW5vlgA//fVrFuE=', 'Content-Length': '0'} 2023-07-29T00:39:34.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000407fff8fa4c796bc-0064c45fc6-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.590 DEBUG:boto:path=/hiiqos-59/obj124 2023-07-29T00:39:34.591 DEBUG:boto:auth_path=/hiiqos-59/obj124 2023-07-29T00:39:34.591 DEBUG:boto:path=/hiiqos-59/obj124?versionId=null 2023-07-29T00:39:34.592 DEBUG:boto:auth_path=/hiiqos-59/obj124?versionId=null 2023-07-29T00:39:34.593 DEBUG:boto:Method: GET 2023-07-29T00:39:34.593 DEBUG:boto:Path: /hiiqos-59/obj124?versionId=null 2023-07-29T00:39:34.594 DEBUG:boto:Data: 2023-07-29T00:39:34.594 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.595 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:34.596 DEBUG:boto:Port: 8000 2023-07-29T00:39:34.596 DEBUG:boto:Params: {} 2023-07-29T00:39:34.597 DEBUG:boto:Token: None 2023-07-29T00:39:34.598 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj124?versionId=null 2023-07-29T00:39:34.598 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kgcnyo0W2DH5pW5vlgA//fVrFuE= 2023-07-29T00:39:34.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kgcnyo0W2DH5pW5vlgA//fVrFuE=', 'Content-Length': '0'} 2023-07-29T00:39:34.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2a43e572f634e93-0064c45fc6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.602 DEBUG:boto:path=/hiiqos-59/obj124 2023-07-29T00:39:34.603 DEBUG:boto:auth_path=/hiiqos-59/obj124 2023-07-29T00:39:34.603 DEBUG:boto:path=/hiiqos-59/obj124?versionId=null 2023-07-29T00:39:34.604 DEBUG:boto:auth_path=/hiiqos-59/obj124?versionId=null 2023-07-29T00:39:34.604 DEBUG:boto:Method: HEAD 2023-07-29T00:39:34.605 DEBUG:boto:Path: /hiiqos-59/obj124?versionId=null 2023-07-29T00:39:34.605 DEBUG:boto:Data: 2023-07-29T00:39:34.606 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.606 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:34.606 DEBUG:boto:Port: 8001 2023-07-29T00:39:34.607 DEBUG:boto:Params: {} 2023-07-29T00:39:34.607 DEBUG:boto:Token: None 2023-07-29T00:39:34.608 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj124?versionId=null 2023-07-29T00:39:34.608 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:n3jGEK+EmivA6HbSmfJo5D9VkyQ= 2023-07-29T00:39:34.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:n3jGEK+EmivA6HbSmfJo5D9VkyQ=', 'Content-Length': '0'} 2023-07-29T00:39:34.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ead684ef280c7d1a-0064c45fc6-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.611 DEBUG:boto:path=/hiiqos-59/obj124 2023-07-29T00:39:34.611 DEBUG:boto:auth_path=/hiiqos-59/obj124 2023-07-29T00:39:34.612 DEBUG:boto:path=/hiiqos-59/obj124?versionId=null 2023-07-29T00:39:34.612 DEBUG:boto:auth_path=/hiiqos-59/obj124?versionId=null 2023-07-29T00:39:34.613 DEBUG:boto:Method: HEAD 2023-07-29T00:39:34.613 DEBUG:boto:Path: /hiiqos-59/obj124?versionId=null 2023-07-29T00:39:34.613 DEBUG:boto:Data: 2023-07-29T00:39:34.614 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.614 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:34.615 DEBUG:boto:Port: 8000 2023-07-29T00:39:34.615 DEBUG:boto:Params: {} 2023-07-29T00:39:34.615 DEBUG:boto:Token: None 2023-07-29T00:39:34.616 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj124?versionId=null 2023-07-29T00:39:34.616 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:n3jGEK+EmivA6HbSmfJo5D9VkyQ= 2023-07-29T00:39:34.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:n3jGEK+EmivA6HbSmfJo5D9VkyQ=', 'Content-Length': '0'} 2023-07-29T00:39:34.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000442c9322842e7bff-0064c45fc6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-07-29T00:39:34.620 DEBUG:boto:path=/hiiqos-59/obj124 2023-07-29T00:39:34.620 DEBUG:boto:auth_path=/hiiqos-59/obj124 2023-07-29T00:39:34.620 DEBUG:boto:path=/hiiqos-59/obj124?versionId=null 2023-07-29T00:39:34.621 DEBUG:boto:auth_path=/hiiqos-59/obj124?versionId=null 2023-07-29T00:39:34.621 DEBUG:boto:Method: GET 2023-07-29T00:39:34.622 DEBUG:boto:Path: /hiiqos-59/obj124?versionId=null 2023-07-29T00:39:34.622 DEBUG:boto:Data: 2023-07-29T00:39:34.622 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.623 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:34.624 DEBUG:boto:Port: 8001 2023-07-29T00:39:34.624 DEBUG:boto:Params: {} 2023-07-29T00:39:34.625 DEBUG:boto:Token: None 2023-07-29T00:39:34.625 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj124?versionId=null 2023-07-29T00:39:34.625 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kgcnyo0W2DH5pW5vlgA//fVrFuE= 2023-07-29T00:39:34.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kgcnyo0W2DH5pW5vlgA//fVrFuE=', 'Content-Length': '0'} 2023-07-29T00:39:34.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f670f36d093ecc2-0064c45fc6-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.628 DEBUG:boto:path=/hiiqos-59/obj124 2023-07-29T00:39:34.629 DEBUG:boto:auth_path=/hiiqos-59/obj124 2023-07-29T00:39:34.630 DEBUG:boto:path=/hiiqos-59/obj124?versionId=null 2023-07-29T00:39:34.630 DEBUG:boto:auth_path=/hiiqos-59/obj124?versionId=null 2023-07-29T00:39:34.631 DEBUG:boto:Method: GET 2023-07-29T00:39:34.631 DEBUG:boto:Path: /hiiqos-59/obj124?versionId=null 2023-07-29T00:39:34.632 DEBUG:boto:Data: 2023-07-29T00:39:34.632 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.633 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:34.633 DEBUG:boto:Port: 8000 2023-07-29T00:39:34.634 DEBUG:boto:Params: {} 2023-07-29T00:39:34.635 DEBUG:boto:Token: None 2023-07-29T00:39:34.635 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj124?versionId=null 2023-07-29T00:39:34.636 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kgcnyo0W2DH5pW5vlgA//fVrFuE= 2023-07-29T00:39:34.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kgcnyo0W2DH5pW5vlgA//fVrFuE=', 'Content-Length': '0'} 2023-07-29T00:39:34.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c4d9a62446f1498-0064c45fc6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.639 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:34.640 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:34.641 DEBUG:boto:path=/hiiqos-59/?prefix=obj124 2023-07-29T00:39:34.641 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj124 2023-07-29T00:39:34.642 DEBUG:boto:Method: GET 2023-07-29T00:39:34.642 DEBUG:boto:Path: /hiiqos-59/?prefix=obj124 2023-07-29T00:39:34.642 DEBUG:boto:Data: 2023-07-29T00:39:34.643 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.643 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:34.643 DEBUG:boto:Port: 8001 2023-07-29T00:39:34.644 DEBUG:boto:Params: {} 2023-07-29T00:39:34.644 DEBUG:boto:Token: None 2023-07-29T00:39:34.645 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/ 2023-07-29T00:39:34.645 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S+Edt2AVsKtdLg5f775TAKithZ8= 2023-07-29T00:39:34.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S+Edt2AVsKtdLg5f775TAKithZ8=', 'Content-Length': '0'} 2023-07-29T00:39:34.648 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023d4558103226256-0064c45fc6-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.649 DEBUG:boto:b'hiiqos-59obj1241000falseobj1242023-07-29T00:35:31.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:34.650 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:34.650 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:34.650 DEBUG:boto:path=/hiiqos-59/?prefix=obj124 2023-07-29T00:39:34.651 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj124 2023-07-29T00:39:34.652 DEBUG:boto:Method: GET 2023-07-29T00:39:34.652 DEBUG:boto:Path: /hiiqos-59/?prefix=obj124 2023-07-29T00:39:34.653 DEBUG:boto:Data: 2023-07-29T00:39:34.653 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.653 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:34.654 DEBUG:boto:Port: 8000 2023-07-29T00:39:34.654 DEBUG:boto:Params: {} 2023-07-29T00:39:34.655 DEBUG:boto:Token: None 2023-07-29T00:39:34.655 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/ 2023-07-29T00:39:34.656 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S+Edt2AVsKtdLg5f775TAKithZ8= 2023-07-29T00:39:34.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S+Edt2AVsKtdLg5f775TAKithZ8=', 'Content-Length': '0'} 2023-07-29T00:39:34.661 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000534b917dbf4ce475-0064c45fc6-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.661 DEBUG:boto:b'hiiqos-59obj1241000falseobj1242023-07-29T00:35:31.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:34.662 DEBUG:boto:path=/hiiqos-59/obj124 2023-07-29T00:39:34.663 DEBUG:boto:auth_path=/hiiqos-59/obj124 2023-07-29T00:39:34.664 DEBUG:boto:Method: HEAD 2023-07-29T00:39:34.665 DEBUG:boto:Path: /hiiqos-59/obj124 2023-07-29T00:39:34.666 DEBUG:boto:Data: 2023-07-29T00:39:34.667 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.668 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:34.668 DEBUG:boto:Port: 8001 2023-07-29T00:39:34.668 DEBUG:boto:Params: {} 2023-07-29T00:39:34.669 DEBUG:boto:Token: None 2023-07-29T00:39:34.669 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj124 2023-07-29T00:39:34.670 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RyktTPiCzR5OFF1abRYiU4KN/Eo= 2023-07-29T00:39:34.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RyktTPiCzR5OFF1abRYiU4KN/Eo=', 'Content-Length': '0'} 2023-07-29T00:39:34.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ea97f1a990bfd18-0064c45fc6-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.673 DEBUG:boto:path=/hiiqos-59/obj124 2023-07-29T00:39:34.674 DEBUG:boto:auth_path=/hiiqos-59/obj124 2023-07-29T00:39:34.674 DEBUG:boto:Method: HEAD 2023-07-29T00:39:34.675 DEBUG:boto:Path: /hiiqos-59/obj124 2023-07-29T00:39:34.675 DEBUG:boto:Data: 2023-07-29T00:39:34.676 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.676 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:34.676 DEBUG:boto:Port: 8000 2023-07-29T00:39:34.677 DEBUG:boto:Params: {} 2023-07-29T00:39:34.677 DEBUG:boto:Token: None 2023-07-29T00:39:34.678 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj124 2023-07-29T00:39:34.678 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RyktTPiCzR5OFF1abRYiU4KN/Eo= 2023-07-29T00:39:34.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RyktTPiCzR5OFF1abRYiU4KN/Eo=', 'Content-Length': '0'} 2023-07-29T00:39:34.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066606f0970b3a680-0064c45fc6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-07-29T00:39:34.682 DEBUG:boto:path=/hiiqos-59/obj124 2023-07-29T00:39:34.682 DEBUG:boto:auth_path=/hiiqos-59/obj124 2023-07-29T00:39:34.682 DEBUG:boto:Method: GET 2023-07-29T00:39:34.683 DEBUG:boto:Path: /hiiqos-59/obj124 2023-07-29T00:39:34.683 DEBUG:boto:Data: 2023-07-29T00:39:34.683 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.684 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:34.684 DEBUG:boto:Port: 8001 2023-07-29T00:39:34.685 DEBUG:boto:Params: {} 2023-07-29T00:39:34.685 DEBUG:boto:Token: None 2023-07-29T00:39:34.685 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj124 2023-07-29T00:39:34.686 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:b5eiSvOvXSs1D3C1K1Gq5kCnVnk= 2023-07-29T00:39:34.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:b5eiSvOvXSs1D3C1K1Gq5kCnVnk=', 'Content-Length': '0'} 2023-07-29T00:39:34.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cd2739505fc0de0-0064c45fc6-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.732 DEBUG:boto:path=/hiiqos-59/obj124 2023-07-29T00:39:34.733 DEBUG:boto:auth_path=/hiiqos-59/obj124 2023-07-29T00:39:34.733 DEBUG:boto:Method: GET 2023-07-29T00:39:34.734 DEBUG:boto:Path: /hiiqos-59/obj124 2023-07-29T00:39:34.734 DEBUG:boto:Data: 2023-07-29T00:39:34.734 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.735 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:34.735 DEBUG:boto:Port: 8000 2023-07-29T00:39:34.735 DEBUG:boto:Params: {} 2023-07-29T00:39:34.736 DEBUG:boto:Token: None 2023-07-29T00:39:34.736 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj124 2023-07-29T00:39:34.737 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:b5eiSvOvXSs1D3C1K1Gq5kCnVnk= 2023-07-29T00:39:34.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:b5eiSvOvXSs1D3C1K1Gq5kCnVnk=', 'Content-Length': '0'} 2023-07-29T00:39:34.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c33a8c31e4ad338c-0064c45fc6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.740 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:34.740 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:34.741 DEBUG:boto:path=/hiiqos-59/?prefix=obj124 2023-07-29T00:39:34.741 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj124 2023-07-29T00:39:34.742 DEBUG:boto:Method: GET 2023-07-29T00:39:34.742 DEBUG:boto:Path: /hiiqos-59/?prefix=obj124 2023-07-29T00:39:34.742 DEBUG:boto:Data: 2023-07-29T00:39:34.743 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.743 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:34.743 DEBUG:boto:Port: 8001 2023-07-29T00:39:34.744 DEBUG:boto:Params: {} 2023-07-29T00:39:34.744 DEBUG:boto:Token: None 2023-07-29T00:39:34.744 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/ 2023-07-29T00:39:34.745 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S+Edt2AVsKtdLg5f775TAKithZ8= 2023-07-29T00:39:34.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S+Edt2AVsKtdLg5f775TAKithZ8=', 'Content-Length': '0'} 2023-07-29T00:39:34.748 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3999f1806d510f1-0064c45fc6-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.748 DEBUG:boto:b'hiiqos-59obj1241000falseobj1242023-07-29T00:35:31.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:34.749 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:34.749 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:34.750 DEBUG:boto:path=/hiiqos-59/?prefix=obj124 2023-07-29T00:39:34.750 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj124 2023-07-29T00:39:34.751 DEBUG:boto:Method: GET 2023-07-29T00:39:34.751 DEBUG:boto:Path: /hiiqos-59/?prefix=obj124 2023-07-29T00:39:34.751 DEBUG:boto:Data: 2023-07-29T00:39:34.752 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.752 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:34.752 DEBUG:boto:Port: 8000 2023-07-29T00:39:34.753 DEBUG:boto:Params: {} 2023-07-29T00:39:34.753 DEBUG:boto:Token: None 2023-07-29T00:39:34.754 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/ 2023-07-29T00:39:34.754 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S+Edt2AVsKtdLg5f775TAKithZ8= 2023-07-29T00:39:34.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S+Edt2AVsKtdLg5f775TAKithZ8=', 'Content-Length': '0'} 2023-07-29T00:39:34.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f3f01c6462185c4-0064c45fc6-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.760 DEBUG:boto:b'hiiqos-59obj1241000falseobj1242023-07-29T00:35:31.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:34.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-07-29T00:39:34.761 DEBUG:boto:path=/hiiqos-59/obj145 2023-07-29T00:39:34.762 DEBUG:boto:auth_path=/hiiqos-59/obj145 2023-07-29T00:39:34.762 DEBUG:boto:path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:39:34.763 DEBUG:boto:auth_path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:39:34.763 DEBUG:boto:Method: GET 2023-07-29T00:39:34.764 DEBUG:boto:Path: /hiiqos-59/obj145?versionId=null 2023-07-29T00:39:34.765 DEBUG:boto:Data: 2023-07-29T00:39:34.766 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.766 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:34.767 DEBUG:boto:Port: 8001 2023-07-29T00:39:34.768 DEBUG:boto:Params: {} 2023-07-29T00:39:34.769 DEBUG:boto:Token: None 2023-07-29T00:39:34.770 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj145?versionId=null 2023-07-29T00:39:34.770 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0zeoXHMsFyzx6Ojj2SKkBwwozbU= 2023-07-29T00:39:34.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0zeoXHMsFyzx6Ojj2SKkBwwozbU=', 'Content-Length': '0'} 2023-07-29T00:39:34.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b4fdbaa627f751a-0064c45fc6-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.774 DEBUG:boto:path=/hiiqos-59/obj145 2023-07-29T00:39:34.774 DEBUG:boto:auth_path=/hiiqos-59/obj145 2023-07-29T00:39:34.775 DEBUG:boto:path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:39:34.775 DEBUG:boto:auth_path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:39:34.776 DEBUG:boto:Method: GET 2023-07-29T00:39:34.776 DEBUG:boto:Path: /hiiqos-59/obj145?versionId=null 2023-07-29T00:39:34.777 DEBUG:boto:Data: 2023-07-29T00:39:34.778 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.778 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:34.779 DEBUG:boto:Port: 8000 2023-07-29T00:39:34.779 DEBUG:boto:Params: {} 2023-07-29T00:39:34.780 DEBUG:boto:Token: None 2023-07-29T00:39:34.780 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj145?versionId=null 2023-07-29T00:39:34.781 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0zeoXHMsFyzx6Ojj2SKkBwwozbU= 2023-07-29T00:39:34.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0zeoXHMsFyzx6Ojj2SKkBwwozbU=', 'Content-Length': '0'} 2023-07-29T00:39:34.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b98b1e3052ac5707-0064c45fc6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.785 DEBUG:boto:path=/hiiqos-59/obj145 2023-07-29T00:39:34.786 DEBUG:boto:auth_path=/hiiqos-59/obj145 2023-07-29T00:39:34.786 DEBUG:boto:path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:39:34.787 DEBUG:boto:auth_path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:39:34.787 DEBUG:boto:Method: HEAD 2023-07-29T00:39:34.787 DEBUG:boto:Path: /hiiqos-59/obj145?versionId=null 2023-07-29T00:39:34.788 DEBUG:boto:Data: 2023-07-29T00:39:34.788 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.789 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:34.789 DEBUG:boto:Port: 8001 2023-07-29T00:39:34.789 DEBUG:boto:Params: {} 2023-07-29T00:39:34.790 DEBUG:boto:Token: None 2023-07-29T00:39:34.790 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj145?versionId=null 2023-07-29T00:39:34.791 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:c5XrUeNRoarWLp10Wo3sO94K44A= 2023-07-29T00:39:34.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c5XrUeNRoarWLp10Wo3sO94K44A=', 'Content-Length': '0'} 2023-07-29T00:39:34.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006799c98424947eaa-0064c45fc6-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.794 DEBUG:boto:path=/hiiqos-59/obj145 2023-07-29T00:39:34.794 DEBUG:boto:auth_path=/hiiqos-59/obj145 2023-07-29T00:39:34.795 DEBUG:boto:path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:39:34.795 DEBUG:boto:auth_path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:39:34.795 DEBUG:boto:Method: HEAD 2023-07-29T00:39:34.796 DEBUG:boto:Path: /hiiqos-59/obj145?versionId=null 2023-07-29T00:39:34.796 DEBUG:boto:Data: 2023-07-29T00:39:34.797 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.797 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:34.797 DEBUG:boto:Port: 8000 2023-07-29T00:39:34.798 DEBUG:boto:Params: {} 2023-07-29T00:39:34.798 DEBUG:boto:Token: None 2023-07-29T00:39:34.799 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj145?versionId=null 2023-07-29T00:39:34.799 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:c5XrUeNRoarWLp10Wo3sO94K44A= 2023-07-29T00:39:34.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c5XrUeNRoarWLp10Wo3sO94K44A=', 'Content-Length': '0'} 2023-07-29T00:39:34.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f370ef4529223e7-0064c45fc6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-07-29T00:39:34.803 DEBUG:boto:path=/hiiqos-59/obj145 2023-07-29T00:39:34.803 DEBUG:boto:auth_path=/hiiqos-59/obj145 2023-07-29T00:39:34.804 DEBUG:boto:path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:39:34.804 DEBUG:boto:auth_path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:39:34.804 DEBUG:boto:Method: GET 2023-07-29T00:39:34.805 DEBUG:boto:Path: /hiiqos-59/obj145?versionId=null 2023-07-29T00:39:34.805 DEBUG:boto:Data: 2023-07-29T00:39:34.806 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.806 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:34.806 DEBUG:boto:Port: 8001 2023-07-29T00:39:34.807 DEBUG:boto:Params: {} 2023-07-29T00:39:34.807 DEBUG:boto:Token: None 2023-07-29T00:39:34.807 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj145?versionId=null 2023-07-29T00:39:34.808 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0zeoXHMsFyzx6Ojj2SKkBwwozbU= 2023-07-29T00:39:34.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0zeoXHMsFyzx6Ojj2SKkBwwozbU=', 'Content-Length': '0'} 2023-07-29T00:39:34.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001aafe5c0396a482b-0064c45fc6-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.852 DEBUG:boto:path=/hiiqos-59/obj145 2023-07-29T00:39:34.853 DEBUG:boto:auth_path=/hiiqos-59/obj145 2023-07-29T00:39:34.853 DEBUG:boto:path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:39:34.854 DEBUG:boto:auth_path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:39:34.854 DEBUG:boto:Method: GET 2023-07-29T00:39:34.854 DEBUG:boto:Path: /hiiqos-59/obj145?versionId=null 2023-07-29T00:39:34.855 DEBUG:boto:Data: 2023-07-29T00:39:34.855 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.855 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:34.856 DEBUG:boto:Port: 8000 2023-07-29T00:39:34.856 DEBUG:boto:Params: {} 2023-07-29T00:39:34.857 DEBUG:boto:Token: None 2023-07-29T00:39:34.857 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj145?versionId=null 2023-07-29T00:39:34.858 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0zeoXHMsFyzx6Ojj2SKkBwwozbU= 2023-07-29T00:39:34.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0zeoXHMsFyzx6Ojj2SKkBwwozbU=', 'Content-Length': '0'} 2023-07-29T00:39:34.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000487a3f6e01ccf0bc-0064c45fc6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.861 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:34.862 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:34.862 DEBUG:boto:path=/hiiqos-59/?prefix=obj145 2023-07-29T00:39:34.862 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj145 2023-07-29T00:39:34.863 DEBUG:boto:Method: GET 2023-07-29T00:39:34.863 DEBUG:boto:Path: /hiiqos-59/?prefix=obj145 2023-07-29T00:39:34.863 DEBUG:boto:Data: 2023-07-29T00:39:34.864 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.864 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:34.865 DEBUG:boto:Port: 8001 2023-07-29T00:39:34.865 DEBUG:boto:Params: {} 2023-07-29T00:39:34.865 DEBUG:boto:Token: None 2023-07-29T00:39:34.866 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/ 2023-07-29T00:39:34.866 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S+Edt2AVsKtdLg5f775TAKithZ8= 2023-07-29T00:39:34.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S+Edt2AVsKtdLg5f775TAKithZ8=', 'Content-Length': '0'} 2023-07-29T00:39:34.869 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec2846955fc6ac16-0064c45fc6-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.870 DEBUG:boto:b'hiiqos-59obj1451000falseobj1452023-07-29T00:35:01.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:34.871 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:34.871 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:34.872 DEBUG:boto:path=/hiiqos-59/?prefix=obj145 2023-07-29T00:39:34.872 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj145 2023-07-29T00:39:34.872 DEBUG:boto:Method: GET 2023-07-29T00:39:34.873 DEBUG:boto:Path: /hiiqos-59/?prefix=obj145 2023-07-29T00:39:34.873 DEBUG:boto:Data: 2023-07-29T00:39:34.873 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.874 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:34.874 DEBUG:boto:Port: 8000 2023-07-29T00:39:34.874 DEBUG:boto:Params: {} 2023-07-29T00:39:34.875 DEBUG:boto:Token: None 2023-07-29T00:39:34.875 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/ 2023-07-29T00:39:34.876 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S+Edt2AVsKtdLg5f775TAKithZ8= 2023-07-29T00:39:34.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S+Edt2AVsKtdLg5f775TAKithZ8=', 'Content-Length': '0'} 2023-07-29T00:39:34.880 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054316ab79fe61c7f-0064c45fc6-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.924 DEBUG:boto:b'hiiqos-59obj1451000falseobj1452023-07-29T00:35:01.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:34.926 DEBUG:boto:path=/hiiqos-59/obj145 2023-07-29T00:39:34.926 DEBUG:boto:auth_path=/hiiqos-59/obj145 2023-07-29T00:39:34.927 DEBUG:boto:Method: HEAD 2023-07-29T00:39:34.927 DEBUG:boto:Path: /hiiqos-59/obj145 2023-07-29T00:39:34.928 DEBUG:boto:Data: 2023-07-29T00:39:34.928 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.929 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:34.929 DEBUG:boto:Port: 8001 2023-07-29T00:39:34.929 DEBUG:boto:Params: {} 2023-07-29T00:39:34.930 DEBUG:boto:Token: None 2023-07-29T00:39:34.930 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj145 2023-07-29T00:39:34.931 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:accxJGZ75z/VPtm3cCh0HDklki8= 2023-07-29T00:39:34.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:accxJGZ75z/VPtm3cCh0HDklki8=', 'Content-Length': '0'} 2023-07-29T00:39:34.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077129569edd63105-0064c45fc6-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.935 DEBUG:boto:path=/hiiqos-59/obj145 2023-07-29T00:39:34.936 DEBUG:boto:auth_path=/hiiqos-59/obj145 2023-07-29T00:39:34.936 DEBUG:boto:Method: HEAD 2023-07-29T00:39:34.937 DEBUG:boto:Path: /hiiqos-59/obj145 2023-07-29T00:39:34.938 DEBUG:boto:Data: 2023-07-29T00:39:34.938 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.939 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:34.939 DEBUG:boto:Port: 8000 2023-07-29T00:39:34.940 DEBUG:boto:Params: {} 2023-07-29T00:39:34.941 DEBUG:boto:Token: None 2023-07-29T00:39:34.941 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj145 2023-07-29T00:39:34.942 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:accxJGZ75z/VPtm3cCh0HDklki8= 2023-07-29T00:39:34.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:accxJGZ75z/VPtm3cCh0HDklki8=', 'Content-Length': '0'} 2023-07-29T00:39:34.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028c098d755e2e03e-0064c45fc6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-07-29T00:39:34.947 DEBUG:boto:path=/hiiqos-59/obj145 2023-07-29T00:39:34.948 DEBUG:boto:auth_path=/hiiqos-59/obj145 2023-07-29T00:39:34.949 DEBUG:boto:Method: GET 2023-07-29T00:39:34.950 DEBUG:boto:Path: /hiiqos-59/obj145 2023-07-29T00:39:34.951 DEBUG:boto:Data: 2023-07-29T00:39:34.952 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.953 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:34.954 DEBUG:boto:Port: 8001 2023-07-29T00:39:34.954 DEBUG:boto:Params: {} 2023-07-29T00:39:34.955 DEBUG:boto:Token: None 2023-07-29T00:39:34.955 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj145 2023-07-29T00:39:34.956 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QrsYohPFVinTTpafH3sqT0bx2B8= 2023-07-29T00:39:34.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QrsYohPFVinTTpafH3sqT0bx2B8=', 'Content-Length': '0'} 2023-07-29T00:39:34.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb1ccc48b2dfe9c1-0064c45fc6-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.960 DEBUG:boto:path=/hiiqos-59/obj145 2023-07-29T00:39:34.961 DEBUG:boto:auth_path=/hiiqos-59/obj145 2023-07-29T00:39:34.961 DEBUG:boto:Method: GET 2023-07-29T00:39:34.961 DEBUG:boto:Path: /hiiqos-59/obj145 2023-07-29T00:39:34.962 DEBUG:boto:Data: 2023-07-29T00:39:34.962 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.963 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:34.963 DEBUG:boto:Port: 8000 2023-07-29T00:39:34.963 DEBUG:boto:Params: {} 2023-07-29T00:39:34.964 DEBUG:boto:Token: None 2023-07-29T00:39:34.964 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj145 2023-07-29T00:39:34.965 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QrsYohPFVinTTpafH3sqT0bx2B8= 2023-07-29T00:39:34.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QrsYohPFVinTTpafH3sqT0bx2B8=', 'Content-Length': '0'} 2023-07-29T00:39:34.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce98d2cd6c0f7c4e-0064c45fc6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.968 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:34.969 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:34.969 DEBUG:boto:path=/hiiqos-59/?prefix=obj145 2023-07-29T00:39:34.969 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj145 2023-07-29T00:39:34.970 DEBUG:boto:Method: GET 2023-07-29T00:39:34.970 DEBUG:boto:Path: /hiiqos-59/?prefix=obj145 2023-07-29T00:39:34.970 DEBUG:boto:Data: 2023-07-29T00:39:34.971 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.971 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:34.971 DEBUG:boto:Port: 8001 2023-07-29T00:39:34.972 DEBUG:boto:Params: {} 2023-07-29T00:39:34.972 DEBUG:boto:Token: None 2023-07-29T00:39:34.973 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/ 2023-07-29T00:39:34.973 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S+Edt2AVsKtdLg5f775TAKithZ8= 2023-07-29T00:39:34.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S+Edt2AVsKtdLg5f775TAKithZ8=', 'Content-Length': '0'} 2023-07-29T00:39:34.976 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051dfdf43b9f7b03c-0064c45fc6-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.977 DEBUG:boto:b'hiiqos-59obj1451000falseobj1452023-07-29T00:35:01.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:34.977 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:34.978 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:34.978 DEBUG:boto:path=/hiiqos-59/?prefix=obj145 2023-07-29T00:39:34.979 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj145 2023-07-29T00:39:34.979 DEBUG:boto:Method: GET 2023-07-29T00:39:34.979 DEBUG:boto:Path: /hiiqos-59/?prefix=obj145 2023-07-29T00:39:34.980 DEBUG:boto:Data: 2023-07-29T00:39:34.980 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.980 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:34.981 DEBUG:boto:Port: 8000 2023-07-29T00:39:34.981 DEBUG:boto:Params: {} 2023-07-29T00:39:34.982 DEBUG:boto:Token: None 2023-07-29T00:39:34.982 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/ 2023-07-29T00:39:34.982 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:S+Edt2AVsKtdLg5f775TAKithZ8= 2023-07-29T00:39:34.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:S+Edt2AVsKtdLg5f775TAKithZ8=', 'Content-Length': '0'} 2023-07-29T00:39:34.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1bbacf8e9c62e79-0064c45fc6-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:34.988 DEBUG:boto:b'hiiqos-59obj1451000falseobj1452023-07-29T00:35:01.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:34.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-07-29T00:39:34.989 DEBUG:boto:path=/hiiqos-59/obj148 2023-07-29T00:39:34.990 DEBUG:boto:auth_path=/hiiqos-59/obj148 2023-07-29T00:39:34.990 DEBUG:boto:path=/hiiqos-59/obj148?versionId=null 2023-07-29T00:39:34.990 DEBUG:boto:auth_path=/hiiqos-59/obj148?versionId=null 2023-07-29T00:39:34.991 DEBUG:boto:Method: GET 2023-07-29T00:39:34.992 DEBUG:boto:Path: /hiiqos-59/obj148?versionId=null 2023-07-29T00:39:34.993 DEBUG:boto:Data: 2023-07-29T00:39:34.994 DEBUG:boto:Headers: {} 2023-07-29T00:39:34.995 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:34.996 DEBUG:boto:Port: 8001 2023-07-29T00:39:34.997 DEBUG:boto:Params: {} 2023-07-29T00:39:34.997 DEBUG:boto:Token: None 2023-07-29T00:39:34.998 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:34 GMT /hiiqos-59/obj148?versionId=null 2023-07-29T00:39:34.999 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sjdNtQtXVh0FjVc8BLEwMHa+1ZI= 2023-07-29T00:39:35.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:34 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sjdNtQtXVh0FjVc8BLEwMHa+1ZI=', 'Content-Length': '0'} 2023-07-29T00:39:35.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072fb1aad3b3e3cd6-0064c45fc6-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.003 DEBUG:boto:path=/hiiqos-59/obj148 2023-07-29T00:39:35.003 DEBUG:boto:auth_path=/hiiqos-59/obj148 2023-07-29T00:39:35.003 DEBUG:boto:path=/hiiqos-59/obj148?versionId=null 2023-07-29T00:39:35.004 DEBUG:boto:auth_path=/hiiqos-59/obj148?versionId=null 2023-07-29T00:39:35.004 DEBUG:boto:Method: GET 2023-07-29T00:39:35.005 DEBUG:boto:Path: /hiiqos-59/obj148?versionId=null 2023-07-29T00:39:35.005 DEBUG:boto:Data: 2023-07-29T00:39:35.005 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.006 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:35.006 DEBUG:boto:Port: 8000 2023-07-29T00:39:35.006 DEBUG:boto:Params: {} 2023-07-29T00:39:35.007 DEBUG:boto:Token: None 2023-07-29T00:39:35.007 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj148?versionId=null 2023-07-29T00:39:35.008 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mjuCPUhTWHxoAvObWrbM+T8XnBU= 2023-07-29T00:39:35.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mjuCPUhTWHxoAvObWrbM+T8XnBU=', 'Content-Length': '0'} 2023-07-29T00:39:35.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074951dd0815e0f98-0064c45fc6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.011 DEBUG:boto:path=/hiiqos-59/obj148 2023-07-29T00:39:35.012 DEBUG:boto:auth_path=/hiiqos-59/obj148 2023-07-29T00:39:35.012 DEBUG:boto:path=/hiiqos-59/obj148?versionId=null 2023-07-29T00:39:35.013 DEBUG:boto:auth_path=/hiiqos-59/obj148?versionId=null 2023-07-29T00:39:35.013 DEBUG:boto:Method: HEAD 2023-07-29T00:39:35.013 DEBUG:boto:Path: /hiiqos-59/obj148?versionId=null 2023-07-29T00:39:35.014 DEBUG:boto:Data: 2023-07-29T00:39:35.014 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.014 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:35.015 DEBUG:boto:Port: 8001 2023-07-29T00:39:35.015 DEBUG:boto:Params: {} 2023-07-29T00:39:35.015 DEBUG:boto:Token: None 2023-07-29T00:39:35.016 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj148?versionId=null 2023-07-29T00:39:35.016 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Ex953fteSd93yP3zdH4BN2avBAI= 2023-07-29T00:39:35.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Ex953fteSd93yP3zdH4BN2avBAI=', 'Content-Length': '0'} 2023-07-29T00:39:35.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000378a50a8d542881e-0064c45fc6-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.019 DEBUG:boto:path=/hiiqos-59/obj148 2023-07-29T00:39:35.020 DEBUG:boto:auth_path=/hiiqos-59/obj148 2023-07-29T00:39:35.020 DEBUG:boto:path=/hiiqos-59/obj148?versionId=null 2023-07-29T00:39:35.020 DEBUG:boto:auth_path=/hiiqos-59/obj148?versionId=null 2023-07-29T00:39:35.021 DEBUG:boto:Method: HEAD 2023-07-29T00:39:35.021 DEBUG:boto:Path: /hiiqos-59/obj148?versionId=null 2023-07-29T00:39:35.021 DEBUG:boto:Data: 2023-07-29T00:39:35.022 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.022 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:35.022 DEBUG:boto:Port: 8000 2023-07-29T00:39:35.023 DEBUG:boto:Params: {} 2023-07-29T00:39:35.023 DEBUG:boto:Token: None 2023-07-29T00:39:35.024 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj148?versionId=null 2023-07-29T00:39:35.024 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Ex953fteSd93yP3zdH4BN2avBAI= 2023-07-29T00:39:35.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Ex953fteSd93yP3zdH4BN2avBAI=', 'Content-Length': '0'} 2023-07-29T00:39:35.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d363b427a005b33c-0064c45fc6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-07-29T00:39:35.027 DEBUG:boto:path=/hiiqos-59/obj148 2023-07-29T00:39:35.027 DEBUG:boto:auth_path=/hiiqos-59/obj148 2023-07-29T00:39:35.028 DEBUG:boto:path=/hiiqos-59/obj148?versionId=null 2023-07-29T00:39:35.028 DEBUG:boto:auth_path=/hiiqos-59/obj148?versionId=null 2023-07-29T00:39:35.029 DEBUG:boto:Method: GET 2023-07-29T00:39:35.029 DEBUG:boto:Path: /hiiqos-59/obj148?versionId=null 2023-07-29T00:39:35.029 DEBUG:boto:Data: 2023-07-29T00:39:35.030 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.030 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:35.031 DEBUG:boto:Port: 8001 2023-07-29T00:39:35.031 DEBUG:boto:Params: {} 2023-07-29T00:39:35.031 DEBUG:boto:Token: None 2023-07-29T00:39:35.032 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj148?versionId=null 2023-07-29T00:39:35.032 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mjuCPUhTWHxoAvObWrbM+T8XnBU= 2023-07-29T00:39:35.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mjuCPUhTWHxoAvObWrbM+T8XnBU=', 'Content-Length': '0'} 2023-07-29T00:39:35.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077e5f41ed4663130-0064c45fc6-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.036 DEBUG:boto:path=/hiiqos-59/obj148 2023-07-29T00:39:35.036 DEBUG:boto:auth_path=/hiiqos-59/obj148 2023-07-29T00:39:35.037 DEBUG:boto:path=/hiiqos-59/obj148?versionId=null 2023-07-29T00:39:35.038 DEBUG:boto:auth_path=/hiiqos-59/obj148?versionId=null 2023-07-29T00:39:35.039 DEBUG:boto:Method: GET 2023-07-29T00:39:35.040 DEBUG:boto:Path: /hiiqos-59/obj148?versionId=null 2023-07-29T00:39:35.041 DEBUG:boto:Data: 2023-07-29T00:39:35.041 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.041 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:35.042 DEBUG:boto:Port: 8000 2023-07-29T00:39:35.042 DEBUG:boto:Params: {} 2023-07-29T00:39:35.043 DEBUG:boto:Token: None 2023-07-29T00:39:35.043 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj148?versionId=null 2023-07-29T00:39:35.043 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mjuCPUhTWHxoAvObWrbM+T8XnBU= 2023-07-29T00:39:35.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mjuCPUhTWHxoAvObWrbM+T8XnBU=', 'Content-Length': '0'} 2023-07-29T00:39:35.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000498c548291d08742-0064c45fc6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.047 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:35.047 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:35.047 DEBUG:boto:path=/hiiqos-59/?prefix=obj148 2023-07-29T00:39:35.048 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj148 2023-07-29T00:39:35.048 DEBUG:boto:Method: GET 2023-07-29T00:39:35.049 DEBUG:boto:Path: /hiiqos-59/?prefix=obj148 2023-07-29T00:39:35.050 DEBUG:boto:Data: 2023-07-29T00:39:35.051 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.052 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:35.053 DEBUG:boto:Port: 8001 2023-07-29T00:39:35.054 DEBUG:boto:Params: {} 2023-07-29T00:39:35.055 DEBUG:boto:Token: None 2023-07-29T00:39:35.056 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/ 2023-07-29T00:39:35.057 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Svl+Yg6zeuZObtmqp4jIascQLG4= 2023-07-29T00:39:35.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Svl+Yg6zeuZObtmqp4jIascQLG4=', 'Content-Length': '0'} 2023-07-29T00:39:35.060 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024751f9e7909e802-0064c45fc6-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:34 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.061 DEBUG:boto:b'hiiqos-59obj1481000falseobj1482023-07-29T00:37:53.270Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:35.061 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:35.062 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:35.062 DEBUG:boto:path=/hiiqos-59/?prefix=obj148 2023-07-29T00:39:35.063 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj148 2023-07-29T00:39:35.063 DEBUG:boto:Method: GET 2023-07-29T00:39:35.064 DEBUG:boto:Path: /hiiqos-59/?prefix=obj148 2023-07-29T00:39:35.064 DEBUG:boto:Data: 2023-07-29T00:39:35.064 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.065 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:35.065 DEBUG:boto:Port: 8000 2023-07-29T00:39:35.065 DEBUG:boto:Params: {} 2023-07-29T00:39:35.066 DEBUG:boto:Token: None 2023-07-29T00:39:35.066 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/ 2023-07-29T00:39:35.067 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Svl+Yg6zeuZObtmqp4jIascQLG4= 2023-07-29T00:39:35.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Svl+Yg6zeuZObtmqp4jIascQLG4=', 'Content-Length': '0'} 2023-07-29T00:39:35.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070946091f8505bec-0064c45fc7-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.073 DEBUG:boto:b'hiiqos-59obj1481000falseobj1482023-07-29T00:37:53.270Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:35.074 DEBUG:boto:path=/hiiqos-59/obj148 2023-07-29T00:39:35.075 DEBUG:boto:auth_path=/hiiqos-59/obj148 2023-07-29T00:39:35.076 DEBUG:boto:Method: HEAD 2023-07-29T00:39:35.076 DEBUG:boto:Path: /hiiqos-59/obj148 2023-07-29T00:39:35.076 DEBUG:boto:Data: 2023-07-29T00:39:35.077 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.077 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:35.077 DEBUG:boto:Port: 8001 2023-07-29T00:39:35.078 DEBUG:boto:Params: {} 2023-07-29T00:39:35.078 DEBUG:boto:Token: None 2023-07-29T00:39:35.079 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj148 2023-07-29T00:39:35.079 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:C4nnAya2lj8XbJF8Wf8ssoZlcjU= 2023-07-29T00:39:35.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:C4nnAya2lj8XbJF8Wf8ssoZlcjU=', 'Content-Length': '0'} 2023-07-29T00:39:35.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043791b12f9ddd84c-0064c45fc7-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.082 DEBUG:boto:path=/hiiqos-59/obj148 2023-07-29T00:39:35.083 DEBUG:boto:auth_path=/hiiqos-59/obj148 2023-07-29T00:39:35.083 DEBUG:boto:Method: HEAD 2023-07-29T00:39:35.084 DEBUG:boto:Path: /hiiqos-59/obj148 2023-07-29T00:39:35.084 DEBUG:boto:Data: 2023-07-29T00:39:35.084 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.085 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:35.085 DEBUG:boto:Port: 8000 2023-07-29T00:39:35.085 DEBUG:boto:Params: {} 2023-07-29T00:39:35.086 DEBUG:boto:Token: None 2023-07-29T00:39:35.086 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj148 2023-07-29T00:39:35.087 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:C4nnAya2lj8XbJF8Wf8ssoZlcjU= 2023-07-29T00:39:35.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:C4nnAya2lj8XbJF8Wf8ssoZlcjU=', 'Content-Length': '0'} 2023-07-29T00:39:35.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007eab2ed4b67452be-0064c45fc7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-07-29T00:39:35.090 DEBUG:boto:path=/hiiqos-59/obj148 2023-07-29T00:39:35.090 DEBUG:boto:auth_path=/hiiqos-59/obj148 2023-07-29T00:39:35.090 DEBUG:boto:Method: GET 2023-07-29T00:39:35.091 DEBUG:boto:Path: /hiiqos-59/obj148 2023-07-29T00:39:35.091 DEBUG:boto:Data: 2023-07-29T00:39:35.091 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.092 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:35.092 DEBUG:boto:Port: 8001 2023-07-29T00:39:35.093 DEBUG:boto:Params: {} 2023-07-29T00:39:35.093 DEBUG:boto:Token: None 2023-07-29T00:39:35.093 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj148 2023-07-29T00:39:35.094 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Vpy2P48QBJchGwM7S2NAL8F+ifo= 2023-07-29T00:39:35.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Vpy2P48QBJchGwM7S2NAL8F+ifo=', 'Content-Length': '0'} 2023-07-29T00:39:35.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004020eb62e1f8ba58-0064c45fc7-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.140 DEBUG:boto:path=/hiiqos-59/obj148 2023-07-29T00:39:35.141 DEBUG:boto:auth_path=/hiiqos-59/obj148 2023-07-29T00:39:35.141 DEBUG:boto:Method: GET 2023-07-29T00:39:35.142 DEBUG:boto:Path: /hiiqos-59/obj148 2023-07-29T00:39:35.142 DEBUG:boto:Data: 2023-07-29T00:39:35.142 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.143 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:35.143 DEBUG:boto:Port: 8000 2023-07-29T00:39:35.144 DEBUG:boto:Params: {} 2023-07-29T00:39:35.144 DEBUG:boto:Token: None 2023-07-29T00:39:35.145 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj148 2023-07-29T00:39:35.146 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Vpy2P48QBJchGwM7S2NAL8F+ifo= 2023-07-29T00:39:35.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Vpy2P48QBJchGwM7S2NAL8F+ifo=', 'Content-Length': '0'} 2023-07-29T00:39:35.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b6d890efacca91e-0064c45fc7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.150 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:35.150 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:35.151 DEBUG:boto:path=/hiiqos-59/?prefix=obj148 2023-07-29T00:39:35.151 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj148 2023-07-29T00:39:35.151 DEBUG:boto:Method: GET 2023-07-29T00:39:35.152 DEBUG:boto:Path: /hiiqos-59/?prefix=obj148 2023-07-29T00:39:35.152 DEBUG:boto:Data: 2023-07-29T00:39:35.153 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.153 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:35.153 DEBUG:boto:Port: 8001 2023-07-29T00:39:35.154 DEBUG:boto:Params: {} 2023-07-29T00:39:35.155 DEBUG:boto:Token: None 2023-07-29T00:39:35.155 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/ 2023-07-29T00:39:35.156 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Svl+Yg6zeuZObtmqp4jIascQLG4= 2023-07-29T00:39:35.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Svl+Yg6zeuZObtmqp4jIascQLG4=', 'Content-Length': '0'} 2023-07-29T00:39:35.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd7e329bda644b3f-0064c45fc7-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.159 DEBUG:boto:b'hiiqos-59obj1481000falseobj1482023-07-29T00:37:53.270Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:35.160 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:35.160 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:35.161 DEBUG:boto:path=/hiiqos-59/?prefix=obj148 2023-07-29T00:39:35.161 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj148 2023-07-29T00:39:35.161 DEBUG:boto:Method: GET 2023-07-29T00:39:35.162 DEBUG:boto:Path: /hiiqos-59/?prefix=obj148 2023-07-29T00:39:35.162 DEBUG:boto:Data: 2023-07-29T00:39:35.162 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.163 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:35.163 DEBUG:boto:Port: 8000 2023-07-29T00:39:35.163 DEBUG:boto:Params: {} 2023-07-29T00:39:35.164 DEBUG:boto:Token: None 2023-07-29T00:39:35.164 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/ 2023-07-29T00:39:35.165 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Svl+Yg6zeuZObtmqp4jIascQLG4= 2023-07-29T00:39:35.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Svl+Yg6zeuZObtmqp4jIascQLG4=', 'Content-Length': '0'} 2023-07-29T00:39:35.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083f2bfec5c63911a-0064c45fc7-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.212 DEBUG:boto:b'hiiqos-59obj1481000falseobj1482023-07-29T00:37:53.270Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:35.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-07-29T00:39:35.214 DEBUG:boto:path=/hiiqos-59/obj155 2023-07-29T00:39:35.215 DEBUG:boto:auth_path=/hiiqos-59/obj155 2023-07-29T00:39:35.215 DEBUG:boto:path=/hiiqos-59/obj155?versionId=null 2023-07-29T00:39:35.215 DEBUG:boto:auth_path=/hiiqos-59/obj155?versionId=null 2023-07-29T00:39:35.216 DEBUG:boto:Method: GET 2023-07-29T00:39:35.216 DEBUG:boto:Path: /hiiqos-59/obj155?versionId=null 2023-07-29T00:39:35.216 DEBUG:boto:Data: 2023-07-29T00:39:35.217 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.217 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:35.218 DEBUG:boto:Port: 8001 2023-07-29T00:39:35.218 DEBUG:boto:Params: {} 2023-07-29T00:39:35.218 DEBUG:boto:Token: None 2023-07-29T00:39:35.219 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj155?versionId=null 2023-07-29T00:39:35.219 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gyEKDCXPjo5KH1xigi3ZHyKU3Jk= 2023-07-29T00:39:35.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gyEKDCXPjo5KH1xigi3ZHyKU3Jk=', 'Content-Length': '0'} 2023-07-29T00:39:35.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcdc89b5f23abf85-0064c45fc7-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.222 DEBUG:boto:path=/hiiqos-59/obj155 2023-07-29T00:39:35.223 DEBUG:boto:auth_path=/hiiqos-59/obj155 2023-07-29T00:39:35.223 DEBUG:boto:path=/hiiqos-59/obj155?versionId=null 2023-07-29T00:39:35.224 DEBUG:boto:auth_path=/hiiqos-59/obj155?versionId=null 2023-07-29T00:39:35.224 DEBUG:boto:Method: GET 2023-07-29T00:39:35.225 DEBUG:boto:Path: /hiiqos-59/obj155?versionId=null 2023-07-29T00:39:35.225 DEBUG:boto:Data: 2023-07-29T00:39:35.225 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.226 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:35.226 DEBUG:boto:Port: 8000 2023-07-29T00:39:35.227 DEBUG:boto:Params: {} 2023-07-29T00:39:35.227 DEBUG:boto:Token: None 2023-07-29T00:39:35.228 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj155?versionId=null 2023-07-29T00:39:35.228 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gyEKDCXPjo5KH1xigi3ZHyKU3Jk= 2023-07-29T00:39:35.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gyEKDCXPjo5KH1xigi3ZHyKU3Jk=', 'Content-Length': '0'} 2023-07-29T00:39:35.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000680be369a8d06bf6-0064c45fc7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.231 DEBUG:boto:path=/hiiqos-59/obj155 2023-07-29T00:39:35.232 DEBUG:boto:auth_path=/hiiqos-59/obj155 2023-07-29T00:39:35.232 DEBUG:boto:path=/hiiqos-59/obj155?versionId=null 2023-07-29T00:39:35.233 DEBUG:boto:auth_path=/hiiqos-59/obj155?versionId=null 2023-07-29T00:39:35.233 DEBUG:boto:Method: HEAD 2023-07-29T00:39:35.233 DEBUG:boto:Path: /hiiqos-59/obj155?versionId=null 2023-07-29T00:39:35.234 DEBUG:boto:Data: 2023-07-29T00:39:35.234 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.234 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:35.235 DEBUG:boto:Port: 8001 2023-07-29T00:39:35.235 DEBUG:boto:Params: {} 2023-07-29T00:39:35.235 DEBUG:boto:Token: None 2023-07-29T00:39:35.236 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj155?versionId=null 2023-07-29T00:39:35.237 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:G7T+7lyYY4URfK8qKF9tlrtmjVA= 2023-07-29T00:39:35.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:G7T+7lyYY4URfK8qKF9tlrtmjVA=', 'Content-Length': '0'} 2023-07-29T00:39:35.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000924f4b245e61c7ab-0064c45fc7-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.240 DEBUG:boto:path=/hiiqos-59/obj155 2023-07-29T00:39:35.240 DEBUG:boto:auth_path=/hiiqos-59/obj155 2023-07-29T00:39:35.240 DEBUG:boto:path=/hiiqos-59/obj155?versionId=null 2023-07-29T00:39:35.241 DEBUG:boto:auth_path=/hiiqos-59/obj155?versionId=null 2023-07-29T00:39:35.241 DEBUG:boto:Method: HEAD 2023-07-29T00:39:35.242 DEBUG:boto:Path: /hiiqos-59/obj155?versionId=null 2023-07-29T00:39:35.242 DEBUG:boto:Data: 2023-07-29T00:39:35.242 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.243 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:35.243 DEBUG:boto:Port: 8000 2023-07-29T00:39:35.243 DEBUG:boto:Params: {} 2023-07-29T00:39:35.244 DEBUG:boto:Token: None 2023-07-29T00:39:35.244 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj155?versionId=null 2023-07-29T00:39:35.245 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:G7T+7lyYY4URfK8qKF9tlrtmjVA= 2023-07-29T00:39:35.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:G7T+7lyYY4URfK8qKF9tlrtmjVA=', 'Content-Length': '0'} 2023-07-29T00:39:35.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000069ee7713a5e021a-0064c45fc7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-07-29T00:39:35.250 DEBUG:boto:path=/hiiqos-59/obj155 2023-07-29T00:39:35.250 DEBUG:boto:auth_path=/hiiqos-59/obj155 2023-07-29T00:39:35.251 DEBUG:boto:path=/hiiqos-59/obj155?versionId=null 2023-07-29T00:39:35.251 DEBUG:boto:auth_path=/hiiqos-59/obj155?versionId=null 2023-07-29T00:39:35.251 DEBUG:boto:Method: GET 2023-07-29T00:39:35.252 DEBUG:boto:Path: /hiiqos-59/obj155?versionId=null 2023-07-29T00:39:35.253 DEBUG:boto:Data: 2023-07-29T00:39:35.254 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.255 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:35.256 DEBUG:boto:Port: 8001 2023-07-29T00:39:35.257 DEBUG:boto:Params: {} 2023-07-29T00:39:35.257 DEBUG:boto:Token: None 2023-07-29T00:39:35.258 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj155?versionId=null 2023-07-29T00:39:35.258 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gyEKDCXPjo5KH1xigi3ZHyKU3Jk= 2023-07-29T00:39:35.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gyEKDCXPjo5KH1xigi3ZHyKU3Jk=', 'Content-Length': '0'} 2023-07-29T00:39:35.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e36e95add492c812-0064c45fc7-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.261 DEBUG:boto:path=/hiiqos-59/obj155 2023-07-29T00:39:35.262 DEBUG:boto:auth_path=/hiiqos-59/obj155 2023-07-29T00:39:35.263 DEBUG:boto:path=/hiiqos-59/obj155?versionId=null 2023-07-29T00:39:35.264 DEBUG:boto:auth_path=/hiiqos-59/obj155?versionId=null 2023-07-29T00:39:35.264 DEBUG:boto:Method: GET 2023-07-29T00:39:35.264 DEBUG:boto:Path: /hiiqos-59/obj155?versionId=null 2023-07-29T00:39:35.265 DEBUG:boto:Data: 2023-07-29T00:39:35.265 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.266 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:35.267 DEBUG:boto:Port: 8000 2023-07-29T00:39:35.268 DEBUG:boto:Params: {} 2023-07-29T00:39:35.269 DEBUG:boto:Token: None 2023-07-29T00:39:35.270 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj155?versionId=null 2023-07-29T00:39:35.270 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gyEKDCXPjo5KH1xigi3ZHyKU3Jk= 2023-07-29T00:39:35.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gyEKDCXPjo5KH1xigi3ZHyKU3Jk=', 'Content-Length': '0'} 2023-07-29T00:39:35.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000194c7a53f2b4b3e1-0064c45fc7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.274 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:35.274 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:35.275 DEBUG:boto:path=/hiiqos-59/?prefix=obj155 2023-07-29T00:39:35.275 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj155 2023-07-29T00:39:35.275 DEBUG:boto:Method: GET 2023-07-29T00:39:35.276 DEBUG:boto:Path: /hiiqos-59/?prefix=obj155 2023-07-29T00:39:35.276 DEBUG:boto:Data: 2023-07-29T00:39:35.276 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.277 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:35.277 DEBUG:boto:Port: 8001 2023-07-29T00:39:35.277 DEBUG:boto:Params: {} 2023-07-29T00:39:35.278 DEBUG:boto:Token: None 2023-07-29T00:39:35.279 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/ 2023-07-29T00:39:35.279 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Svl+Yg6zeuZObtmqp4jIascQLG4= 2023-07-29T00:39:35.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Svl+Yg6zeuZObtmqp4jIascQLG4=', 'Content-Length': '0'} 2023-07-29T00:39:35.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095d97f989d2c50d4-0064c45fc7-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.284 DEBUG:boto:b'hiiqos-59obj1551000falseobj1552023-07-29T00:35:31.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:35.285 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:35.286 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:35.286 DEBUG:boto:path=/hiiqos-59/?prefix=obj155 2023-07-29T00:39:35.287 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj155 2023-07-29T00:39:35.287 DEBUG:boto:Method: GET 2023-07-29T00:39:35.287 DEBUG:boto:Path: /hiiqos-59/?prefix=obj155 2023-07-29T00:39:35.288 DEBUG:boto:Data: 2023-07-29T00:39:35.288 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.288 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:35.289 DEBUG:boto:Port: 8000 2023-07-29T00:39:35.289 DEBUG:boto:Params: {} 2023-07-29T00:39:35.290 DEBUG:boto:Token: None 2023-07-29T00:39:35.290 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/ 2023-07-29T00:39:35.290 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Svl+Yg6zeuZObtmqp4jIascQLG4= 2023-07-29T00:39:35.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Svl+Yg6zeuZObtmqp4jIascQLG4=', 'Content-Length': '0'} 2023-07-29T00:39:35.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4c898bde828278c-0064c45fc7-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.296 DEBUG:boto:b'hiiqos-59obj1551000falseobj1552023-07-29T00:35:31.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:35.296 DEBUG:boto:path=/hiiqos-59/obj155 2023-07-29T00:39:35.297 DEBUG:boto:auth_path=/hiiqos-59/obj155 2023-07-29T00:39:35.297 DEBUG:boto:Method: HEAD 2023-07-29T00:39:35.297 DEBUG:boto:Path: /hiiqos-59/obj155 2023-07-29T00:39:35.298 DEBUG:boto:Data: 2023-07-29T00:39:35.298 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.298 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:35.299 DEBUG:boto:Port: 8001 2023-07-29T00:39:35.299 DEBUG:boto:Params: {} 2023-07-29T00:39:35.299 DEBUG:boto:Token: None 2023-07-29T00:39:35.300 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj155 2023-07-29T00:39:35.300 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O1FPYhdzqbrly9hE9w7aEyegRI0= 2023-07-29T00:39:35.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O1FPYhdzqbrly9hE9w7aEyegRI0=', 'Content-Length': '0'} 2023-07-29T00:39:35.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e096d7be9c049112-0064c45fc7-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.303 DEBUG:boto:path=/hiiqos-59/obj155 2023-07-29T00:39:35.303 DEBUG:boto:auth_path=/hiiqos-59/obj155 2023-07-29T00:39:35.304 DEBUG:boto:Method: HEAD 2023-07-29T00:39:35.304 DEBUG:boto:Path: /hiiqos-59/obj155 2023-07-29T00:39:35.305 DEBUG:boto:Data: 2023-07-29T00:39:35.305 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.305 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:35.305 DEBUG:boto:Port: 8000 2023-07-29T00:39:35.306 DEBUG:boto:Params: {} 2023-07-29T00:39:35.306 DEBUG:boto:Token: None 2023-07-29T00:39:35.307 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj155 2023-07-29T00:39:35.307 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O1FPYhdzqbrly9hE9w7aEyegRI0= 2023-07-29T00:39:35.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O1FPYhdzqbrly9hE9w7aEyegRI0=', 'Content-Length': '0'} 2023-07-29T00:39:35.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085b756faf883ea41-0064c45fc7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-07-29T00:39:35.310 DEBUG:boto:path=/hiiqos-59/obj155 2023-07-29T00:39:35.311 DEBUG:boto:auth_path=/hiiqos-59/obj155 2023-07-29T00:39:35.311 DEBUG:boto:Method: GET 2023-07-29T00:39:35.311 DEBUG:boto:Path: /hiiqos-59/obj155 2023-07-29T00:39:35.312 DEBUG:boto:Data: 2023-07-29T00:39:35.312 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.312 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:35.313 DEBUG:boto:Port: 8001 2023-07-29T00:39:35.313 DEBUG:boto:Params: {} 2023-07-29T00:39:35.314 DEBUG:boto:Token: None 2023-07-29T00:39:35.314 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj155 2023-07-29T00:39:35.315 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CrViYErmx0hUA4+aoG1lMQSCX4g= 2023-07-29T00:39:35.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CrViYErmx0hUA4+aoG1lMQSCX4g=', 'Content-Length': '0'} 2023-07-29T00:39:35.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012010d9c3e4558db-0064c45fc7-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.318 DEBUG:boto:path=/hiiqos-59/obj155 2023-07-29T00:39:35.319 DEBUG:boto:auth_path=/hiiqos-59/obj155 2023-07-29T00:39:35.319 DEBUG:boto:Method: GET 2023-07-29T00:39:35.319 DEBUG:boto:Path: /hiiqos-59/obj155 2023-07-29T00:39:35.320 DEBUG:boto:Data: 2023-07-29T00:39:35.320 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.320 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:35.321 DEBUG:boto:Port: 8000 2023-07-29T00:39:35.321 DEBUG:boto:Params: {} 2023-07-29T00:39:35.322 DEBUG:boto:Token: None 2023-07-29T00:39:35.322 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj155 2023-07-29T00:39:35.322 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CrViYErmx0hUA4+aoG1lMQSCX4g= 2023-07-29T00:39:35.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CrViYErmx0hUA4+aoG1lMQSCX4g=', 'Content-Length': '0'} 2023-07-29T00:39:35.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa1723cc2205d147-0064c45fc7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.372 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:35.373 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:35.374 DEBUG:boto:path=/hiiqos-59/?prefix=obj155 2023-07-29T00:39:35.374 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj155 2023-07-29T00:39:35.374 DEBUG:boto:Method: GET 2023-07-29T00:39:35.375 DEBUG:boto:Path: /hiiqos-59/?prefix=obj155 2023-07-29T00:39:35.375 DEBUG:boto:Data: 2023-07-29T00:39:35.376 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.376 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:35.377 DEBUG:boto:Port: 8001 2023-07-29T00:39:35.377 DEBUG:boto:Params: {} 2023-07-29T00:39:35.377 DEBUG:boto:Token: None 2023-07-29T00:39:35.378 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/ 2023-07-29T00:39:35.378 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Svl+Yg6zeuZObtmqp4jIascQLG4= 2023-07-29T00:39:35.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Svl+Yg6zeuZObtmqp4jIascQLG4=', 'Content-Length': '0'} 2023-07-29T00:39:35.381 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa14e8d27d32d700-0064c45fc7-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.382 DEBUG:boto:b'hiiqos-59obj1551000falseobj1552023-07-29T00:35:31.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:35.383 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:35.383 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:35.383 DEBUG:boto:path=/hiiqos-59/?prefix=obj155 2023-07-29T00:39:35.384 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj155 2023-07-29T00:39:35.384 DEBUG:boto:Method: GET 2023-07-29T00:39:35.385 DEBUG:boto:Path: /hiiqos-59/?prefix=obj155 2023-07-29T00:39:35.385 DEBUG:boto:Data: 2023-07-29T00:39:35.385 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.386 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:35.386 DEBUG:boto:Port: 8000 2023-07-29T00:39:35.386 DEBUG:boto:Params: {} 2023-07-29T00:39:35.387 DEBUG:boto:Token: None 2023-07-29T00:39:35.387 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/ 2023-07-29T00:39:35.388 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Svl+Yg6zeuZObtmqp4jIascQLG4= 2023-07-29T00:39:35.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Svl+Yg6zeuZObtmqp4jIascQLG4=', 'Content-Length': '0'} 2023-07-29T00:39:35.392 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000966521d8398ff7e4-0064c45fc7-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.393 DEBUG:boto:b'hiiqos-59obj1551000falseobj1552023-07-29T00:35:31.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:35.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-07-29T00:39:35.394 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:39:35.395 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:39:35.395 DEBUG:boto:path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:39:35.395 DEBUG:boto:auth_path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:39:35.396 DEBUG:boto:Method: GET 2023-07-29T00:39:35.396 DEBUG:boto:Path: /hiiqos-59/obj17?versionId=null 2023-07-29T00:39:35.397 DEBUG:boto:Data: 2023-07-29T00:39:35.397 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.397 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:35.398 DEBUG:boto:Port: 8001 2023-07-29T00:39:35.398 DEBUG:boto:Params: {} 2023-07-29T00:39:35.398 DEBUG:boto:Token: None 2023-07-29T00:39:35.399 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj17?versionId=null 2023-07-29T00:39:35.399 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bc9H3LosyD9nTzKloUBllWo8TCA= 2023-07-29T00:39:35.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bc9H3LosyD9nTzKloUBllWo8TCA=', 'Content-Length': '0'} 2023-07-29T00:39:35.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027ac7bd33b396ffb-0064c45fc7-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.444 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:39:35.445 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:39:35.445 DEBUG:boto:path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:39:35.446 DEBUG:boto:auth_path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:39:35.446 DEBUG:boto:Method: GET 2023-07-29T00:39:35.446 DEBUG:boto:Path: /hiiqos-59/obj17?versionId=null 2023-07-29T00:39:35.447 DEBUG:boto:Data: 2023-07-29T00:39:35.447 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.447 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:35.448 DEBUG:boto:Port: 8000 2023-07-29T00:39:35.448 DEBUG:boto:Params: {} 2023-07-29T00:39:35.449 DEBUG:boto:Token: None 2023-07-29T00:39:35.449 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj17?versionId=null 2023-07-29T00:39:35.450 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bc9H3LosyD9nTzKloUBllWo8TCA= 2023-07-29T00:39:35.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bc9H3LosyD9nTzKloUBllWo8TCA=', 'Content-Length': '0'} 2023-07-29T00:39:35.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000215c3f2e58ff72cf-0064c45fc7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.452 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:39:35.453 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:39:35.453 DEBUG:boto:path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:39:35.454 DEBUG:boto:auth_path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:39:35.454 DEBUG:boto:Method: HEAD 2023-07-29T00:39:35.454 DEBUG:boto:Path: /hiiqos-59/obj17?versionId=null 2023-07-29T00:39:35.455 DEBUG:boto:Data: 2023-07-29T00:39:35.455 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.455 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:35.456 DEBUG:boto:Port: 8001 2023-07-29T00:39:35.456 DEBUG:boto:Params: {} 2023-07-29T00:39:35.456 DEBUG:boto:Token: None 2023-07-29T00:39:35.457 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj17?versionId=null 2023-07-29T00:39:35.457 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6kSaNOcozMK+y6wOQulkA45UtFs= 2023-07-29T00:39:35.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6kSaNOcozMK+y6wOQulkA45UtFs=', 'Content-Length': '0'} 2023-07-29T00:39:35.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071dd8458c4c389c4-0064c45fc7-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.460 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:39:35.461 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:39:35.461 DEBUG:boto:path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:39:35.462 DEBUG:boto:auth_path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:39:35.462 DEBUG:boto:Method: HEAD 2023-07-29T00:39:35.462 DEBUG:boto:Path: /hiiqos-59/obj17?versionId=null 2023-07-29T00:39:35.463 DEBUG:boto:Data: 2023-07-29T00:39:35.463 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.463 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:35.464 DEBUG:boto:Port: 8000 2023-07-29T00:39:35.464 DEBUG:boto:Params: {} 2023-07-29T00:39:35.464 DEBUG:boto:Token: None 2023-07-29T00:39:35.465 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj17?versionId=null 2023-07-29T00:39:35.465 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6kSaNOcozMK+y6wOQulkA45UtFs= 2023-07-29T00:39:35.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6kSaNOcozMK+y6wOQulkA45UtFs=', 'Content-Length': '0'} 2023-07-29T00:39:35.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007dc66278dfc07a1-0064c45fc7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-07-29T00:39:35.470 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:39:35.471 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:39:35.471 DEBUG:boto:path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:39:35.471 DEBUG:boto:auth_path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:39:35.472 DEBUG:boto:Method: GET 2023-07-29T00:39:35.472 DEBUG:boto:Path: /hiiqos-59/obj17?versionId=null 2023-07-29T00:39:35.473 DEBUG:boto:Data: 2023-07-29T00:39:35.473 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.474 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:35.474 DEBUG:boto:Port: 8001 2023-07-29T00:39:35.475 DEBUG:boto:Params: {} 2023-07-29T00:39:35.475 DEBUG:boto:Token: None 2023-07-29T00:39:35.476 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj17?versionId=null 2023-07-29T00:39:35.476 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bc9H3LosyD9nTzKloUBllWo8TCA= 2023-07-29T00:39:35.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bc9H3LosyD9nTzKloUBllWo8TCA=', 'Content-Length': '0'} 2023-07-29T00:39:35.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e13cbc7b3004e41-0064c45fc7-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.479 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:39:35.480 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:39:35.480 DEBUG:boto:path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:39:35.481 DEBUG:boto:auth_path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:39:35.481 DEBUG:boto:Method: GET 2023-07-29T00:39:35.481 DEBUG:boto:Path: /hiiqos-59/obj17?versionId=null 2023-07-29T00:39:35.482 DEBUG:boto:Data: 2023-07-29T00:39:35.482 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.483 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:35.483 DEBUG:boto:Port: 8000 2023-07-29T00:39:35.483 DEBUG:boto:Params: {} 2023-07-29T00:39:35.484 DEBUG:boto:Token: None 2023-07-29T00:39:35.485 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj17?versionId=null 2023-07-29T00:39:35.486 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bc9H3LosyD9nTzKloUBllWo8TCA= 2023-07-29T00:39:35.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bc9H3LosyD9nTzKloUBllWo8TCA=', 'Content-Length': '0'} 2023-07-29T00:39:35.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092d86d3e3278b016-0064c45fc7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.491 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:35.492 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:35.493 DEBUG:boto:path=/hiiqos-59/?prefix=obj17 2023-07-29T00:39:35.493 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj17 2023-07-29T00:39:35.494 DEBUG:boto:Method: GET 2023-07-29T00:39:35.494 DEBUG:boto:Path: /hiiqos-59/?prefix=obj17 2023-07-29T00:39:35.494 DEBUG:boto:Data: 2023-07-29T00:39:35.495 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.495 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:35.496 DEBUG:boto:Port: 8001 2023-07-29T00:39:35.496 DEBUG:boto:Params: {} 2023-07-29T00:39:35.497 DEBUG:boto:Token: None 2023-07-29T00:39:35.497 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/ 2023-07-29T00:39:35.498 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Svl+Yg6zeuZObtmqp4jIascQLG4= 2023-07-29T00:39:35.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Svl+Yg6zeuZObtmqp4jIascQLG4=', 'Content-Length': '0'} 2023-07-29T00:39:35.500 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094fb2b6f97d70116-0064c45fc7-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.501 DEBUG:boto:b'hiiqos-59obj171000falseobj172023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:35.502 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:35.502 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:35.503 DEBUG:boto:path=/hiiqos-59/?prefix=obj17 2023-07-29T00:39:35.503 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj17 2023-07-29T00:39:35.504 DEBUG:boto:Method: GET 2023-07-29T00:39:35.504 DEBUG:boto:Path: /hiiqos-59/?prefix=obj17 2023-07-29T00:39:35.505 DEBUG:boto:Data: 2023-07-29T00:39:35.505 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.505 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:35.506 DEBUG:boto:Port: 8000 2023-07-29T00:39:35.506 DEBUG:boto:Params: {} 2023-07-29T00:39:35.507 DEBUG:boto:Token: None 2023-07-29T00:39:35.507 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/ 2023-07-29T00:39:35.508 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Svl+Yg6zeuZObtmqp4jIascQLG4= 2023-07-29T00:39:35.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Svl+Yg6zeuZObtmqp4jIascQLG4=', 'Content-Length': '0'} 2023-07-29T00:39:35.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4017ffcacefe9e7-0064c45fc7-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.514 DEBUG:boto:b'hiiqos-59obj171000falseobj172023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:35.515 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:39:35.515 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:39:35.516 DEBUG:boto:Method: HEAD 2023-07-29T00:39:35.517 DEBUG:boto:Path: /hiiqos-59/obj17 2023-07-29T00:39:35.517 DEBUG:boto:Data: 2023-07-29T00:39:35.518 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.519 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:35.519 DEBUG:boto:Port: 8001 2023-07-29T00:39:35.520 DEBUG:boto:Params: {} 2023-07-29T00:39:35.521 DEBUG:boto:Token: None 2023-07-29T00:39:35.521 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj17 2023-07-29T00:39:35.522 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:x0WKjlA9ZnpCx3PRLQdCXvQpV3M= 2023-07-29T00:39:35.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:x0WKjlA9ZnpCx3PRLQdCXvQpV3M=', 'Content-Length': '0'} 2023-07-29T00:39:35.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8f842190198b125-0064c45fc7-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.525 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:39:35.526 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:39:35.526 DEBUG:boto:Method: HEAD 2023-07-29T00:39:35.526 DEBUG:boto:Path: /hiiqos-59/obj17 2023-07-29T00:39:35.527 DEBUG:boto:Data: 2023-07-29T00:39:35.527 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.527 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:35.528 DEBUG:boto:Port: 8000 2023-07-29T00:39:35.528 DEBUG:boto:Params: {} 2023-07-29T00:39:35.529 DEBUG:boto:Token: None 2023-07-29T00:39:35.529 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj17 2023-07-29T00:39:35.529 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:x0WKjlA9ZnpCx3PRLQdCXvQpV3M= 2023-07-29T00:39:35.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:x0WKjlA9ZnpCx3PRLQdCXvQpV3M=', 'Content-Length': '0'} 2023-07-29T00:39:35.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047934f091207c39d-0064c45fc7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.532 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-07-29T00:39:35.533 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:39:35.533 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:39:35.534 DEBUG:boto:Method: GET 2023-07-29T00:39:35.534 DEBUG:boto:Path: /hiiqos-59/obj17 2023-07-29T00:39:35.534 DEBUG:boto:Data: 2023-07-29T00:39:35.535 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.535 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:35.535 DEBUG:boto:Port: 8001 2023-07-29T00:39:35.536 DEBUG:boto:Params: {} 2023-07-29T00:39:35.536 DEBUG:boto:Token: None 2023-07-29T00:39:35.537 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj17 2023-07-29T00:39:35.537 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:62qOSnPuFRpJu1Zou4pqERrnqwQ= 2023-07-29T00:39:35.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:62qOSnPuFRpJu1Zou4pqERrnqwQ=', 'Content-Length': '0'} 2023-07-29T00:39:35.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afb1236e6e588ad8-0064c45fc7-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.580 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:39:35.581 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:39:35.582 DEBUG:boto:Method: GET 2023-07-29T00:39:35.583 DEBUG:boto:Path: /hiiqos-59/obj17 2023-07-29T00:39:35.584 DEBUG:boto:Data: 2023-07-29T00:39:35.585 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.586 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:35.587 DEBUG:boto:Port: 8000 2023-07-29T00:39:35.588 DEBUG:boto:Params: {} 2023-07-29T00:39:35.589 DEBUG:boto:Token: None 2023-07-29T00:39:35.590 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj17 2023-07-29T00:39:35.590 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:62qOSnPuFRpJu1Zou4pqERrnqwQ= 2023-07-29T00:39:35.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:62qOSnPuFRpJu1Zou4pqERrnqwQ=', 'Content-Length': '0'} 2023-07-29T00:39:35.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000950271cb52f2ef0-0064c45fc7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.594 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:35.595 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:35.595 DEBUG:boto:path=/hiiqos-59/?prefix=obj17 2023-07-29T00:39:35.595 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj17 2023-07-29T00:39:35.596 DEBUG:boto:Method: GET 2023-07-29T00:39:35.596 DEBUG:boto:Path: /hiiqos-59/?prefix=obj17 2023-07-29T00:39:35.597 DEBUG:boto:Data: 2023-07-29T00:39:35.597 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.598 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:35.598 DEBUG:boto:Port: 8001 2023-07-29T00:39:35.598 DEBUG:boto:Params: {} 2023-07-29T00:39:35.599 DEBUG:boto:Token: None 2023-07-29T00:39:35.599 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/ 2023-07-29T00:39:35.600 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Svl+Yg6zeuZObtmqp4jIascQLG4= 2023-07-29T00:39:35.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Svl+Yg6zeuZObtmqp4jIascQLG4=', 'Content-Length': '0'} 2023-07-29T00:39:35.603 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c7a277a1d8325b6-0064c45fc7-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.603 DEBUG:boto:b'hiiqos-59obj171000falseobj172023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:35.604 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:35.604 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:35.605 DEBUG:boto:path=/hiiqos-59/?prefix=obj17 2023-07-29T00:39:35.605 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj17 2023-07-29T00:39:35.605 DEBUG:boto:Method: GET 2023-07-29T00:39:35.606 DEBUG:boto:Path: /hiiqos-59/?prefix=obj17 2023-07-29T00:39:35.606 DEBUG:boto:Data: 2023-07-29T00:39:35.607 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.607 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:35.608 DEBUG:boto:Port: 8000 2023-07-29T00:39:35.608 DEBUG:boto:Params: {} 2023-07-29T00:39:35.609 DEBUG:boto:Token: None 2023-07-29T00:39:35.609 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/ 2023-07-29T00:39:35.610 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Svl+Yg6zeuZObtmqp4jIascQLG4= 2023-07-29T00:39:35.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Svl+Yg6zeuZObtmqp4jIascQLG4=', 'Content-Length': '0'} 2023-07-29T00:39:35.614 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cb967088d537dd8-0064c45fc7-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.615 DEBUG:boto:b'hiiqos-59obj171000falseobj172023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:35.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-07-29T00:39:35.617 DEBUG:boto:path=/hiiqos-59/obj185 2023-07-29T00:39:35.617 DEBUG:boto:auth_path=/hiiqos-59/obj185 2023-07-29T00:39:35.618 DEBUG:boto:path=/hiiqos-59/obj185?versionId=null 2023-07-29T00:39:35.619 DEBUG:boto:auth_path=/hiiqos-59/obj185?versionId=null 2023-07-29T00:39:35.620 DEBUG:boto:Method: GET 2023-07-29T00:39:35.621 DEBUG:boto:Path: /hiiqos-59/obj185?versionId=null 2023-07-29T00:39:35.622 DEBUG:boto:Data: 2023-07-29T00:39:35.622 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.623 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:35.623 DEBUG:boto:Port: 8001 2023-07-29T00:39:35.624 DEBUG:boto:Params: {} 2023-07-29T00:39:35.624 DEBUG:boto:Token: None 2023-07-29T00:39:35.625 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj185?versionId=null 2023-07-29T00:39:35.626 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:m86Xt7Of+3Zpv2rUi8sDL33pWJc= 2023-07-29T00:39:35.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:m86Xt7Of+3Zpv2rUi8sDL33pWJc=', 'Content-Length': '0'} 2023-07-29T00:39:35.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ac42b73fef23ed1-0064c45fc7-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.632 DEBUG:boto:path=/hiiqos-59/obj185 2023-07-29T00:39:35.632 DEBUG:boto:auth_path=/hiiqos-59/obj185 2023-07-29T00:39:35.633 DEBUG:boto:path=/hiiqos-59/obj185?versionId=null 2023-07-29T00:39:35.634 DEBUG:boto:auth_path=/hiiqos-59/obj185?versionId=null 2023-07-29T00:39:35.634 DEBUG:boto:Method: GET 2023-07-29T00:39:35.635 DEBUG:boto:Path: /hiiqos-59/obj185?versionId=null 2023-07-29T00:39:35.635 DEBUG:boto:Data: 2023-07-29T00:39:35.636 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.636 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:35.636 DEBUG:boto:Port: 8000 2023-07-29T00:39:35.637 DEBUG:boto:Params: {} 2023-07-29T00:39:35.637 DEBUG:boto:Token: None 2023-07-29T00:39:35.638 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj185?versionId=null 2023-07-29T00:39:35.638 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:m86Xt7Of+3Zpv2rUi8sDL33pWJc= 2023-07-29T00:39:35.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:m86Xt7Of+3Zpv2rUi8sDL33pWJc=', 'Content-Length': '0'} 2023-07-29T00:39:35.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e22a70e4180d0807-0064c45fc7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.642 DEBUG:boto:path=/hiiqos-59/obj185 2023-07-29T00:39:35.642 DEBUG:boto:auth_path=/hiiqos-59/obj185 2023-07-29T00:39:35.642 DEBUG:boto:path=/hiiqos-59/obj185?versionId=null 2023-07-29T00:39:35.643 DEBUG:boto:auth_path=/hiiqos-59/obj185?versionId=null 2023-07-29T00:39:35.643 DEBUG:boto:Method: HEAD 2023-07-29T00:39:35.643 DEBUG:boto:Path: /hiiqos-59/obj185?versionId=null 2023-07-29T00:39:35.644 DEBUG:boto:Data: 2023-07-29T00:39:35.644 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.645 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:35.645 DEBUG:boto:Port: 8001 2023-07-29T00:39:35.645 DEBUG:boto:Params: {} 2023-07-29T00:39:35.646 DEBUG:boto:Token: None 2023-07-29T00:39:35.646 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj185?versionId=null 2023-07-29T00:39:35.646 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rjd8p6pa8x0ruNYdcbTwI5JHFY8= 2023-07-29T00:39:35.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rjd8p6pa8x0ruNYdcbTwI5JHFY8=', 'Content-Length': '0'} 2023-07-29T00:39:35.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005de1ace01eb4414a-0064c45fc7-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.649 DEBUG:boto:path=/hiiqos-59/obj185 2023-07-29T00:39:35.650 DEBUG:boto:auth_path=/hiiqos-59/obj185 2023-07-29T00:39:35.651 DEBUG:boto:path=/hiiqos-59/obj185?versionId=null 2023-07-29T00:39:35.651 DEBUG:boto:auth_path=/hiiqos-59/obj185?versionId=null 2023-07-29T00:39:35.651 DEBUG:boto:Method: HEAD 2023-07-29T00:39:35.652 DEBUG:boto:Path: /hiiqos-59/obj185?versionId=null 2023-07-29T00:39:35.652 DEBUG:boto:Data: 2023-07-29T00:39:35.652 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.653 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:35.653 DEBUG:boto:Port: 8000 2023-07-29T00:39:35.653 DEBUG:boto:Params: {} 2023-07-29T00:39:35.654 DEBUG:boto:Token: None 2023-07-29T00:39:35.654 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj185?versionId=null 2023-07-29T00:39:35.655 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rjd8p6pa8x0ruNYdcbTwI5JHFY8= 2023-07-29T00:39:35.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rjd8p6pa8x0ruNYdcbTwI5JHFY8=', 'Content-Length': '0'} 2023-07-29T00:39:35.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c83133558b80f50c-0064c45fc7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-07-29T00:39:35.661 DEBUG:boto:path=/hiiqos-59/obj185 2023-07-29T00:39:35.662 DEBUG:boto:auth_path=/hiiqos-59/obj185 2023-07-29T00:39:35.663 DEBUG:boto:path=/hiiqos-59/obj185?versionId=null 2023-07-29T00:39:35.664 DEBUG:boto:auth_path=/hiiqos-59/obj185?versionId=null 2023-07-29T00:39:35.665 DEBUG:boto:Method: GET 2023-07-29T00:39:35.666 DEBUG:boto:Path: /hiiqos-59/obj185?versionId=null 2023-07-29T00:39:35.666 DEBUG:boto:Data: 2023-07-29T00:39:35.666 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.667 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:35.667 DEBUG:boto:Port: 8001 2023-07-29T00:39:35.668 DEBUG:boto:Params: {} 2023-07-29T00:39:35.668 DEBUG:boto:Token: None 2023-07-29T00:39:35.668 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj185?versionId=null 2023-07-29T00:39:35.669 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:m86Xt7Of+3Zpv2rUi8sDL33pWJc= 2023-07-29T00:39:35.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:m86Xt7Of+3Zpv2rUi8sDL33pWJc=', 'Content-Length': '0'} 2023-07-29T00:39:35.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae740929ce3de446-0064c45fc7-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.672 DEBUG:boto:path=/hiiqos-59/obj185 2023-07-29T00:39:35.672 DEBUG:boto:auth_path=/hiiqos-59/obj185 2023-07-29T00:39:35.673 DEBUG:boto:path=/hiiqos-59/obj185?versionId=null 2023-07-29T00:39:35.673 DEBUG:boto:auth_path=/hiiqos-59/obj185?versionId=null 2023-07-29T00:39:35.674 DEBUG:boto:Method: GET 2023-07-29T00:39:35.674 DEBUG:boto:Path: /hiiqos-59/obj185?versionId=null 2023-07-29T00:39:35.675 DEBUG:boto:Data: 2023-07-29T00:39:35.675 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.675 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:35.676 DEBUG:boto:Port: 8000 2023-07-29T00:39:35.677 DEBUG:boto:Params: {} 2023-07-29T00:39:35.677 DEBUG:boto:Token: None 2023-07-29T00:39:35.677 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj185?versionId=null 2023-07-29T00:39:35.678 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:m86Xt7Of+3Zpv2rUi8sDL33pWJc= 2023-07-29T00:39:35.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:m86Xt7Of+3Zpv2rUi8sDL33pWJc=', 'Content-Length': '0'} 2023-07-29T00:39:35.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006238b9076b19137-0064c45fc7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.681 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:35.682 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:35.682 DEBUG:boto:path=/hiiqos-59/?prefix=obj185 2023-07-29T00:39:35.682 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj185 2023-07-29T00:39:35.683 DEBUG:boto:Method: GET 2023-07-29T00:39:35.683 DEBUG:boto:Path: /hiiqos-59/?prefix=obj185 2023-07-29T00:39:35.684 DEBUG:boto:Data: 2023-07-29T00:39:35.684 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.684 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:35.685 DEBUG:boto:Port: 8001 2023-07-29T00:39:35.686 DEBUG:boto:Params: {} 2023-07-29T00:39:35.687 DEBUG:boto:Token: None 2023-07-29T00:39:35.688 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/ 2023-07-29T00:39:35.689 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Svl+Yg6zeuZObtmqp4jIascQLG4= 2023-07-29T00:39:35.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Svl+Yg6zeuZObtmqp4jIascQLG4=', 'Content-Length': '0'} 2023-07-29T00:39:35.693 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ae196b0955333ec-0064c45fc7-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.694 DEBUG:boto:b'hiiqos-59obj1851000falseobj1852023-07-29T00:37:53.291Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:35.695 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:35.695 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:35.695 DEBUG:boto:path=/hiiqos-59/?prefix=obj185 2023-07-29T00:39:35.696 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj185 2023-07-29T00:39:35.696 DEBUG:boto:Method: GET 2023-07-29T00:39:35.697 DEBUG:boto:Path: /hiiqos-59/?prefix=obj185 2023-07-29T00:39:35.697 DEBUG:boto:Data: 2023-07-29T00:39:35.698 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.698 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:35.699 DEBUG:boto:Port: 8000 2023-07-29T00:39:35.699 DEBUG:boto:Params: {} 2023-07-29T00:39:35.700 DEBUG:boto:Token: None 2023-07-29T00:39:35.700 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/ 2023-07-29T00:39:35.701 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Svl+Yg6zeuZObtmqp4jIascQLG4= 2023-07-29T00:39:35.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Svl+Yg6zeuZObtmqp4jIascQLG4=', 'Content-Length': '0'} 2023-07-29T00:39:35.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3ff62f3b77942f2-0064c45fc7-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.748 DEBUG:boto:b'hiiqos-59obj1851000falseobj1852023-07-29T00:37:53.291Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:35.749 DEBUG:boto:path=/hiiqos-59/obj185 2023-07-29T00:39:35.749 DEBUG:boto:auth_path=/hiiqos-59/obj185 2023-07-29T00:39:35.750 DEBUG:boto:Method: HEAD 2023-07-29T00:39:35.750 DEBUG:boto:Path: /hiiqos-59/obj185 2023-07-29T00:39:35.751 DEBUG:boto:Data: 2023-07-29T00:39:35.751 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.751 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:35.752 DEBUG:boto:Port: 8001 2023-07-29T00:39:35.752 DEBUG:boto:Params: {} 2023-07-29T00:39:35.752 DEBUG:boto:Token: None 2023-07-29T00:39:35.753 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj185 2023-07-29T00:39:35.753 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:l+PUoptmpgyAcCawmX+VOmwYBRA= 2023-07-29T00:39:35.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:l+PUoptmpgyAcCawmX+VOmwYBRA=', 'Content-Length': '0'} 2023-07-29T00:39:35.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000371503682cd23181-0064c45fc7-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.757 DEBUG:boto:path=/hiiqos-59/obj185 2023-07-29T00:39:35.758 DEBUG:boto:auth_path=/hiiqos-59/obj185 2023-07-29T00:39:35.758 DEBUG:boto:Method: HEAD 2023-07-29T00:39:35.758 DEBUG:boto:Path: /hiiqos-59/obj185 2023-07-29T00:39:35.759 DEBUG:boto:Data: 2023-07-29T00:39:35.759 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.759 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:35.760 DEBUG:boto:Port: 8000 2023-07-29T00:39:35.760 DEBUG:boto:Params: {} 2023-07-29T00:39:35.760 DEBUG:boto:Token: None 2023-07-29T00:39:35.761 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj185 2023-07-29T00:39:35.761 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:l+PUoptmpgyAcCawmX+VOmwYBRA= 2023-07-29T00:39:35.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:l+PUoptmpgyAcCawmX+VOmwYBRA=', 'Content-Length': '0'} 2023-07-29T00:39:35.763 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bfc17beb70608fa-0064c45fc7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-07-29T00:39:35.764 DEBUG:boto:path=/hiiqos-59/obj185 2023-07-29T00:39:35.765 DEBUG:boto:auth_path=/hiiqos-59/obj185 2023-07-29T00:39:35.765 DEBUG:boto:Method: GET 2023-07-29T00:39:35.766 DEBUG:boto:Path: /hiiqos-59/obj185 2023-07-29T00:39:35.766 DEBUG:boto:Data: 2023-07-29T00:39:35.766 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.767 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:35.767 DEBUG:boto:Port: 8001 2023-07-29T00:39:35.768 DEBUG:boto:Params: {} 2023-07-29T00:39:35.768 DEBUG:boto:Token: None 2023-07-29T00:39:35.769 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj185 2023-07-29T00:39:35.769 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pCVlCE9oiaoxZtG+zkv687ALmsE= 2023-07-29T00:39:35.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pCVlCE9oiaoxZtG+zkv687ALmsE=', 'Content-Length': '0'} 2023-07-29T00:39:35.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c03cdfb638162248-0064c45fc7-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.816 DEBUG:boto:path=/hiiqos-59/obj185 2023-07-29T00:39:35.818 DEBUG:boto:auth_path=/hiiqos-59/obj185 2023-07-29T00:39:35.819 DEBUG:boto:Method: GET 2023-07-29T00:39:35.819 DEBUG:boto:Path: /hiiqos-59/obj185 2023-07-29T00:39:35.819 DEBUG:boto:Data: 2023-07-29T00:39:35.820 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.820 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:35.821 DEBUG:boto:Port: 8000 2023-07-29T00:39:35.821 DEBUG:boto:Params: {} 2023-07-29T00:39:35.821 DEBUG:boto:Token: None 2023-07-29T00:39:35.822 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj185 2023-07-29T00:39:35.822 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pCVlCE9oiaoxZtG+zkv687ALmsE= 2023-07-29T00:39:35.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pCVlCE9oiaoxZtG+zkv687ALmsE=', 'Content-Length': '0'} 2023-07-29T00:39:35.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9213bcde5386e38-0064c45fc7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.825 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:35.826 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:35.826 DEBUG:boto:path=/hiiqos-59/?prefix=obj185 2023-07-29T00:39:35.827 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj185 2023-07-29T00:39:35.827 DEBUG:boto:Method: GET 2023-07-29T00:39:35.827 DEBUG:boto:Path: /hiiqos-59/?prefix=obj185 2023-07-29T00:39:35.828 DEBUG:boto:Data: 2023-07-29T00:39:35.828 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.828 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:35.829 DEBUG:boto:Port: 8001 2023-07-29T00:39:35.829 DEBUG:boto:Params: {} 2023-07-29T00:39:35.829 DEBUG:boto:Token: None 2023-07-29T00:39:35.830 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/ 2023-07-29T00:39:35.830 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Svl+Yg6zeuZObtmqp4jIascQLG4= 2023-07-29T00:39:35.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Svl+Yg6zeuZObtmqp4jIascQLG4=', 'Content-Length': '0'} 2023-07-29T00:39:35.833 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c2ece9a194126b8-0064c45fc7-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.834 DEBUG:boto:b'hiiqos-59obj1851000falseobj1852023-07-29T00:37:53.291Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:35.834 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:35.835 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:35.835 DEBUG:boto:path=/hiiqos-59/?prefix=obj185 2023-07-29T00:39:35.836 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj185 2023-07-29T00:39:35.836 DEBUG:boto:Method: GET 2023-07-29T00:39:35.836 DEBUG:boto:Path: /hiiqos-59/?prefix=obj185 2023-07-29T00:39:35.837 DEBUG:boto:Data: 2023-07-29T00:39:35.837 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.837 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:35.838 DEBUG:boto:Port: 8000 2023-07-29T00:39:35.838 DEBUG:boto:Params: {} 2023-07-29T00:39:35.839 DEBUG:boto:Token: None 2023-07-29T00:39:35.839 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/ 2023-07-29T00:39:35.839 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Svl+Yg6zeuZObtmqp4jIascQLG4= 2023-07-29T00:39:35.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Svl+Yg6zeuZObtmqp4jIascQLG4=', 'Content-Length': '0'} 2023-07-29T00:39:35.844 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c476968a7953bd8b-0064c45fc7-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.845 DEBUG:boto:b'hiiqos-59obj1851000falseobj1852023-07-29T00:37:53.291Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:35.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-07-29T00:39:35.846 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:39:35.846 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:39:35.847 DEBUG:boto:path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:39:35.847 DEBUG:boto:auth_path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:39:35.847 DEBUG:boto:Method: GET 2023-07-29T00:39:35.848 DEBUG:boto:Path: /hiiqos-59/obj19?versionId=null 2023-07-29T00:39:35.848 DEBUG:boto:Data: 2023-07-29T00:39:35.848 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.849 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:35.849 DEBUG:boto:Port: 8001 2023-07-29T00:39:35.850 DEBUG:boto:Params: {} 2023-07-29T00:39:35.850 DEBUG:boto:Token: None 2023-07-29T00:39:35.850 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj19?versionId=null 2023-07-29T00:39:35.851 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BDsMOpNAIhvC7w3NMNwM+JhPozw= 2023-07-29T00:39:35.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BDsMOpNAIhvC7w3NMNwM+JhPozw=', 'Content-Length': '0'} 2023-07-29T00:39:35.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000785f0573eb43b33e-0064c45fc7-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.854 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:39:35.854 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:39:35.855 DEBUG:boto:path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:39:35.855 DEBUG:boto:auth_path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:39:35.855 DEBUG:boto:Method: GET 2023-07-29T00:39:35.856 DEBUG:boto:Path: /hiiqos-59/obj19?versionId=null 2023-07-29T00:39:35.856 DEBUG:boto:Data: 2023-07-29T00:39:35.856 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.857 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:35.857 DEBUG:boto:Port: 8000 2023-07-29T00:39:35.857 DEBUG:boto:Params: {} 2023-07-29T00:39:35.858 DEBUG:boto:Token: None 2023-07-29T00:39:35.858 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj19?versionId=null 2023-07-29T00:39:35.859 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BDsMOpNAIhvC7w3NMNwM+JhPozw= 2023-07-29T00:39:35.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BDsMOpNAIhvC7w3NMNwM+JhPozw=', 'Content-Length': '0'} 2023-07-29T00:39:35.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e085915340083a4-0064c45fc7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.904 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:39:35.905 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:39:35.905 DEBUG:boto:path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:39:35.906 DEBUG:boto:auth_path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:39:35.906 DEBUG:boto:Method: HEAD 2023-07-29T00:39:35.906 DEBUG:boto:Path: /hiiqos-59/obj19?versionId=null 2023-07-29T00:39:35.907 DEBUG:boto:Data: 2023-07-29T00:39:35.907 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.907 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:35.908 DEBUG:boto:Port: 8001 2023-07-29T00:39:35.908 DEBUG:boto:Params: {} 2023-07-29T00:39:35.908 DEBUG:boto:Token: None 2023-07-29T00:39:35.909 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj19?versionId=null 2023-07-29T00:39:35.909 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zb764Wl0cAUg0GXaYUlFfqjKVm8= 2023-07-29T00:39:35.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zb764Wl0cAUg0GXaYUlFfqjKVm8=', 'Content-Length': '0'} 2023-07-29T00:39:35.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2f14987d262f01d-0064c45fc7-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.912 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:39:35.913 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:39:35.913 DEBUG:boto:path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:39:35.913 DEBUG:boto:auth_path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:39:35.914 DEBUG:boto:Method: HEAD 2023-07-29T00:39:35.914 DEBUG:boto:Path: /hiiqos-59/obj19?versionId=null 2023-07-29T00:39:35.914 DEBUG:boto:Data: 2023-07-29T00:39:35.915 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.915 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:35.915 DEBUG:boto:Port: 8000 2023-07-29T00:39:35.916 DEBUG:boto:Params: {} 2023-07-29T00:39:35.916 DEBUG:boto:Token: None 2023-07-29T00:39:35.917 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj19?versionId=null 2023-07-29T00:39:35.917 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zb764Wl0cAUg0GXaYUlFfqjKVm8= 2023-07-29T00:39:35.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zb764Wl0cAUg0GXaYUlFfqjKVm8=', 'Content-Length': '0'} 2023-07-29T00:39:35.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e4c0ebdad87dda6-0064c45fc7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-07-29T00:39:35.921 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:39:35.921 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:39:35.921 DEBUG:boto:path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:39:35.922 DEBUG:boto:auth_path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:39:35.922 DEBUG:boto:Method: GET 2023-07-29T00:39:35.922 DEBUG:boto:Path: /hiiqos-59/obj19?versionId=null 2023-07-29T00:39:35.923 DEBUG:boto:Data: 2023-07-29T00:39:35.923 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.923 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:35.924 DEBUG:boto:Port: 8001 2023-07-29T00:39:35.924 DEBUG:boto:Params: {} 2023-07-29T00:39:35.925 DEBUG:boto:Token: None 2023-07-29T00:39:35.925 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj19?versionId=null 2023-07-29T00:39:35.925 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BDsMOpNAIhvC7w3NMNwM+JhPozw= 2023-07-29T00:39:35.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BDsMOpNAIhvC7w3NMNwM+JhPozw=', 'Content-Length': '0'} 2023-07-29T00:39:35.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000589815b06f10f7fd-0064c45fc7-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.929 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:39:35.929 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:39:35.929 DEBUG:boto:path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:39:35.930 DEBUG:boto:auth_path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:39:35.930 DEBUG:boto:Method: GET 2023-07-29T00:39:35.930 DEBUG:boto:Path: /hiiqos-59/obj19?versionId=null 2023-07-29T00:39:35.931 DEBUG:boto:Data: 2023-07-29T00:39:35.931 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.932 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:35.932 DEBUG:boto:Port: 8000 2023-07-29T00:39:35.932 DEBUG:boto:Params: {} 2023-07-29T00:39:35.933 DEBUG:boto:Token: None 2023-07-29T00:39:35.933 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj19?versionId=null 2023-07-29T00:39:35.934 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BDsMOpNAIhvC7w3NMNwM+JhPozw= 2023-07-29T00:39:35.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BDsMOpNAIhvC7w3NMNwM+JhPozw=', 'Content-Length': '0'} 2023-07-29T00:39:35.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d198b27eb97dbb8b-0064c45fc7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.937 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:35.937 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:35.937 DEBUG:boto:path=/hiiqos-59/?prefix=obj19 2023-07-29T00:39:35.938 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj19 2023-07-29T00:39:35.938 DEBUG:boto:Method: GET 2023-07-29T00:39:35.938 DEBUG:boto:Path: /hiiqos-59/?prefix=obj19 2023-07-29T00:39:35.939 DEBUG:boto:Data: 2023-07-29T00:39:35.939 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.940 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:35.940 DEBUG:boto:Port: 8001 2023-07-29T00:39:35.940 DEBUG:boto:Params: {} 2023-07-29T00:39:35.941 DEBUG:boto:Token: None 2023-07-29T00:39:35.941 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/ 2023-07-29T00:39:35.942 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Svl+Yg6zeuZObtmqp4jIascQLG4= 2023-07-29T00:39:35.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Svl+Yg6zeuZObtmqp4jIascQLG4=', 'Content-Length': '0'} 2023-07-29T00:39:35.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcd17e59c83962e2-0064c45fc7-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.945 DEBUG:boto:b'hiiqos-59obj191000falseobj192023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:35.946 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:35.946 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:35.947 DEBUG:boto:path=/hiiqos-59/?prefix=obj19 2023-07-29T00:39:35.947 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj19 2023-07-29T00:39:35.947 DEBUG:boto:Method: GET 2023-07-29T00:39:35.948 DEBUG:boto:Path: /hiiqos-59/?prefix=obj19 2023-07-29T00:39:35.948 DEBUG:boto:Data: 2023-07-29T00:39:35.948 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.949 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:35.949 DEBUG:boto:Port: 8000 2023-07-29T00:39:35.949 DEBUG:boto:Params: {} 2023-07-29T00:39:35.950 DEBUG:boto:Token: None 2023-07-29T00:39:35.950 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/ 2023-07-29T00:39:35.951 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Svl+Yg6zeuZObtmqp4jIascQLG4= 2023-07-29T00:39:35.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Svl+Yg6zeuZObtmqp4jIascQLG4=', 'Content-Length': '0'} 2023-07-29T00:39:35.955 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074c8baca799d5e11-0064c45fc7-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.956 DEBUG:boto:b'hiiqos-59obj191000falseobj192023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:35.956 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:39:35.957 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:39:35.957 DEBUG:boto:Method: HEAD 2023-07-29T00:39:35.957 DEBUG:boto:Path: /hiiqos-59/obj19 2023-07-29T00:39:35.958 DEBUG:boto:Data: 2023-07-29T00:39:35.958 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.958 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:35.959 DEBUG:boto:Port: 8001 2023-07-29T00:39:35.959 DEBUG:boto:Params: {} 2023-07-29T00:39:35.959 DEBUG:boto:Token: None 2023-07-29T00:39:35.960 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj19 2023-07-29T00:39:35.960 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:92Id4zMSyl6zw1UiB2MsXOYa/wE= 2023-07-29T00:39:35.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:92Id4zMSyl6zw1UiB2MsXOYa/wE=', 'Content-Length': '0'} 2023-07-29T00:39:35.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007975f74379a2c8d7-0064c45fc7-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.963 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:39:35.963 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:39:35.964 DEBUG:boto:Method: HEAD 2023-07-29T00:39:35.964 DEBUG:boto:Path: /hiiqos-59/obj19 2023-07-29T00:39:35.965 DEBUG:boto:Data: 2023-07-29T00:39:35.965 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.965 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:35.966 DEBUG:boto:Port: 8000 2023-07-29T00:39:35.966 DEBUG:boto:Params: {} 2023-07-29T00:39:35.966 DEBUG:boto:Token: None 2023-07-29T00:39:35.967 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj19 2023-07-29T00:39:35.967 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:92Id4zMSyl6zw1UiB2MsXOYa/wE= 2023-07-29T00:39:35.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:92Id4zMSyl6zw1UiB2MsXOYa/wE=', 'Content-Length': '0'} 2023-07-29T00:39:35.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c0626353f55d0f4-0064c45fc7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:35.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-07-29T00:39:35.970 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:39:35.971 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:39:35.971 DEBUG:boto:Method: GET 2023-07-29T00:39:35.972 DEBUG:boto:Path: /hiiqos-59/obj19 2023-07-29T00:39:35.972 DEBUG:boto:Data: 2023-07-29T00:39:35.972 DEBUG:boto:Headers: {} 2023-07-29T00:39:35.973 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:35.973 DEBUG:boto:Port: 8001 2023-07-29T00:39:35.973 DEBUG:boto:Params: {} 2023-07-29T00:39:35.974 DEBUG:boto:Token: None 2023-07-29T00:39:35.974 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:35 GMT /hiiqos-59/obj19 2023-07-29T00:39:35.975 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:q9mluETSFSqq0YbGbVWCR2QU6Yc= 2023-07-29T00:39:35.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:q9mluETSFSqq0YbGbVWCR2QU6Yc=', 'Content-Length': '0'} 2023-07-29T00:39:35.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da1dc178469e068d-0064c45fc7-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.020 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:39:36.021 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:39:36.021 DEBUG:boto:Method: GET 2023-07-29T00:39:36.022 DEBUG:boto:Path: /hiiqos-59/obj19 2023-07-29T00:39:36.022 DEBUG:boto:Data: 2023-07-29T00:39:36.022 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.023 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:36.023 DEBUG:boto:Port: 8000 2023-07-29T00:39:36.023 DEBUG:boto:Params: {} 2023-07-29T00:39:36.024 DEBUG:boto:Token: None 2023-07-29T00:39:36.024 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj19 2023-07-29T00:39:36.025 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PI1CzcSmT/oN4Mulc8XGEvXGtEI= 2023-07-29T00:39:36.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PI1CzcSmT/oN4Mulc8XGEvXGtEI=', 'Content-Length': '0'} 2023-07-29T00:39:36.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7b2bd7970a2aad3-0064c45fc7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.028 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:36.028 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:36.028 DEBUG:boto:path=/hiiqos-59/?prefix=obj19 2023-07-29T00:39:36.029 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj19 2023-07-29T00:39:36.029 DEBUG:boto:Method: GET 2023-07-29T00:39:36.030 DEBUG:boto:Path: /hiiqos-59/?prefix=obj19 2023-07-29T00:39:36.030 DEBUG:boto:Data: 2023-07-29T00:39:36.030 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.031 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:36.031 DEBUG:boto:Port: 8001 2023-07-29T00:39:36.031 DEBUG:boto:Params: {} 2023-07-29T00:39:36.032 DEBUG:boto:Token: None 2023-07-29T00:39:36.032 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/ 2023-07-29T00:39:36.033 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA= 2023-07-29T00:39:36.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA=', 'Content-Length': '0'} 2023-07-29T00:39:36.036 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2ca4bfb535ebfd8-0064c45fc7-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.036 DEBUG:boto:b'hiiqos-59obj191000falseobj192023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:36.037 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:36.037 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:36.038 DEBUG:boto:path=/hiiqos-59/?prefix=obj19 2023-07-29T00:39:36.038 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj19 2023-07-29T00:39:36.038 DEBUG:boto:Method: GET 2023-07-29T00:39:36.039 DEBUG:boto:Path: /hiiqos-59/?prefix=obj19 2023-07-29T00:39:36.039 DEBUG:boto:Data: 2023-07-29T00:39:36.039 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.040 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:36.040 DEBUG:boto:Port: 8000 2023-07-29T00:39:36.041 DEBUG:boto:Params: {} 2023-07-29T00:39:36.041 DEBUG:boto:Token: None 2023-07-29T00:39:36.041 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/ 2023-07-29T00:39:36.042 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA= 2023-07-29T00:39:36.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA=', 'Content-Length': '0'} 2023-07-29T00:39:36.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000df17d769b0237b4-0064c45fc7-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.048 DEBUG:boto:b'hiiqos-59obj191000falseobj192023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:36.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-07-29T00:39:36.049 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:39:36.049 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:39:36.049 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:39:36.050 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:39:36.050 DEBUG:boto:Method: GET 2023-07-29T00:39:36.050 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:39:36.051 DEBUG:boto:Data: 2023-07-29T00:39:36.051 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.051 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:36.052 DEBUG:boto:Port: 8001 2023-07-29T00:39:36.052 DEBUG:boto:Params: {} 2023-07-29T00:39:36.053 DEBUG:boto:Token: None 2023-07-29T00:39:36.053 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:39:36.053 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oMgDDtX1vwTZlcixMj0CXqjXRBI= 2023-07-29T00:39:36.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oMgDDtX1vwTZlcixMj0CXqjXRBI=', 'Content-Length': '0'} 2023-07-29T00:39:36.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001208f02a497b7fef-0064c45fc7-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.056 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:39:36.057 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:39:36.057 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:39:36.058 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:39:36.058 DEBUG:boto:Method: GET 2023-07-29T00:39:36.058 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:39:36.059 DEBUG:boto:Data: 2023-07-29T00:39:36.059 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.059 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:36.060 DEBUG:boto:Port: 8000 2023-07-29T00:39:36.060 DEBUG:boto:Params: {} 2023-07-29T00:39:36.061 DEBUG:boto:Token: None 2023-07-29T00:39:36.061 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:39:36.061 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oMgDDtX1vwTZlcixMj0CXqjXRBI= 2023-07-29T00:39:36.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oMgDDtX1vwTZlcixMj0CXqjXRBI=', 'Content-Length': '0'} 2023-07-29T00:39:36.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acd8e7a5753f64d3-0064c45fc7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.064 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:39:36.065 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:39:36.065 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:39:36.065 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:39:36.066 DEBUG:boto:Method: HEAD 2023-07-29T00:39:36.066 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:39:36.066 DEBUG:boto:Data: 2023-07-29T00:39:36.067 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.067 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:36.067 DEBUG:boto:Port: 8001 2023-07-29T00:39:36.068 DEBUG:boto:Params: {} 2023-07-29T00:39:36.068 DEBUG:boto:Token: None 2023-07-29T00:39:36.068 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:39:36.069 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:juVz2s+lYZUCZ5N38SKkute0eN0= 2023-07-29T00:39:36.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:juVz2s+lYZUCZ5N38SKkute0eN0=', 'Content-Length': '0'} 2023-07-29T00:39:36.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f4210101751da74-0064c45fc8-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.072 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:39:36.072 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:39:36.072 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:39:36.073 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:39:36.073 DEBUG:boto:Method: HEAD 2023-07-29T00:39:36.073 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:39:36.074 DEBUG:boto:Data: 2023-07-29T00:39:36.074 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.074 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:36.075 DEBUG:boto:Port: 8000 2023-07-29T00:39:36.075 DEBUG:boto:Params: {} 2023-07-29T00:39:36.075 DEBUG:boto:Token: None 2023-07-29T00:39:36.076 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:39:36.076 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:juVz2s+lYZUCZ5N38SKkute0eN0= 2023-07-29T00:39:36.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:juVz2s+lYZUCZ5N38SKkute0eN0=', 'Content-Length': '0'} 2023-07-29T00:39:36.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d57174384147ed42-0064c45fc8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-07-29T00:39:36.080 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:39:36.080 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:39:36.080 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:39:36.081 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:39:36.081 DEBUG:boto:Method: GET 2023-07-29T00:39:36.082 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:39:36.082 DEBUG:boto:Data: 2023-07-29T00:39:36.082 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.083 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:36.083 DEBUG:boto:Port: 8001 2023-07-29T00:39:36.083 DEBUG:boto:Params: {} 2023-07-29T00:39:36.084 DEBUG:boto:Token: None 2023-07-29T00:39:36.084 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:39:36.085 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oMgDDtX1vwTZlcixMj0CXqjXRBI= 2023-07-29T00:39:36.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oMgDDtX1vwTZlcixMj0CXqjXRBI=', 'Content-Length': '0'} 2023-07-29T00:39:36.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007976e123f194453d-0064c45fc8-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.088 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:39:36.088 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:39:36.088 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:39:36.089 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:39:36.089 DEBUG:boto:Method: GET 2023-07-29T00:39:36.089 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:39:36.090 DEBUG:boto:Data: 2023-07-29T00:39:36.090 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.091 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:36.091 DEBUG:boto:Port: 8000 2023-07-29T00:39:36.091 DEBUG:boto:Params: {} 2023-07-29T00:39:36.092 DEBUG:boto:Token: None 2023-07-29T00:39:36.092 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:39:36.093 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oMgDDtX1vwTZlcixMj0CXqjXRBI= 2023-07-29T00:39:36.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oMgDDtX1vwTZlcixMj0CXqjXRBI=', 'Content-Length': '0'} 2023-07-29T00:39:36.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d887d38b9886f76a-0064c45fc8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.095 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:36.096 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:36.096 DEBUG:boto:path=/hiiqos-59/?prefix=obj22 2023-07-29T00:39:36.097 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj22 2023-07-29T00:39:36.097 DEBUG:boto:Method: GET 2023-07-29T00:39:36.097 DEBUG:boto:Path: /hiiqos-59/?prefix=obj22 2023-07-29T00:39:36.098 DEBUG:boto:Data: 2023-07-29T00:39:36.098 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.099 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:36.099 DEBUG:boto:Port: 8001 2023-07-29T00:39:36.099 DEBUG:boto:Params: {} 2023-07-29T00:39:36.100 DEBUG:boto:Token: None 2023-07-29T00:39:36.100 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/ 2023-07-29T00:39:36.101 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA= 2023-07-29T00:39:36.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA=', 'Content-Length': '0'} 2023-07-29T00:39:36.103 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b0c853bd9cce6cc-0064c45fc8-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.104 DEBUG:boto:b'hiiqos-59obj221000falseobj222023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:36.105 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:36.105 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:36.106 DEBUG:boto:path=/hiiqos-59/?prefix=obj22 2023-07-29T00:39:36.106 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj22 2023-07-29T00:39:36.106 DEBUG:boto:Method: GET 2023-07-29T00:39:36.107 DEBUG:boto:Path: /hiiqos-59/?prefix=obj22 2023-07-29T00:39:36.107 DEBUG:boto:Data: 2023-07-29T00:39:36.107 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.108 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:36.108 DEBUG:boto:Port: 8000 2023-07-29T00:39:36.108 DEBUG:boto:Params: {} 2023-07-29T00:39:36.109 DEBUG:boto:Token: None 2023-07-29T00:39:36.109 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/ 2023-07-29T00:39:36.110 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA= 2023-07-29T00:39:36.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA=', 'Content-Length': '0'} 2023-07-29T00:39:36.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004256abea0707ba48-0064c45fc8-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.115 DEBUG:boto:b'hiiqos-59obj221000falseobj222023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:36.116 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:39:36.116 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:39:36.116 DEBUG:boto:Method: HEAD 2023-07-29T00:39:36.117 DEBUG:boto:Path: /hiiqos-59/obj22 2023-07-29T00:39:36.117 DEBUG:boto:Data: 2023-07-29T00:39:36.117 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.118 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:36.118 DEBUG:boto:Port: 8001 2023-07-29T00:39:36.118 DEBUG:boto:Params: {} 2023-07-29T00:39:36.119 DEBUG:boto:Token: None 2023-07-29T00:39:36.119 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj22 2023-07-29T00:39:36.120 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IEoBHVSQe1M+ZfqOqUxG1Nn0g4c= 2023-07-29T00:39:36.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IEoBHVSQe1M+ZfqOqUxG1Nn0g4c=', 'Content-Length': '0'} 2023-07-29T00:39:36.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be98a4683443d066-0064c45fc8-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.123 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:39:36.123 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:39:36.124 DEBUG:boto:Method: HEAD 2023-07-29T00:39:36.124 DEBUG:boto:Path: /hiiqos-59/obj22 2023-07-29T00:39:36.124 DEBUG:boto:Data: 2023-07-29T00:39:36.125 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.125 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:36.125 DEBUG:boto:Port: 8000 2023-07-29T00:39:36.126 DEBUG:boto:Params: {} 2023-07-29T00:39:36.126 DEBUG:boto:Token: None 2023-07-29T00:39:36.126 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj22 2023-07-29T00:39:36.127 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IEoBHVSQe1M+ZfqOqUxG1Nn0g4c= 2023-07-29T00:39:36.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IEoBHVSQe1M+ZfqOqUxG1Nn0g4c=', 'Content-Length': '0'} 2023-07-29T00:39:36.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e72221b94723eea-0064c45fc8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-07-29T00:39:36.130 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:39:36.130 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:39:36.131 DEBUG:boto:Method: GET 2023-07-29T00:39:36.131 DEBUG:boto:Path: /hiiqos-59/obj22 2023-07-29T00:39:36.132 DEBUG:boto:Data: 2023-07-29T00:39:36.132 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.132 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:36.133 DEBUG:boto:Port: 8001 2023-07-29T00:39:36.133 DEBUG:boto:Params: {} 2023-07-29T00:39:36.134 DEBUG:boto:Token: None 2023-07-29T00:39:36.134 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj22 2023-07-29T00:39:36.134 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6YnEndE8g1/s6hBmWBAE7Y4bmVg= 2023-07-29T00:39:36.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6YnEndE8g1/s6hBmWBAE7Y4bmVg=', 'Content-Length': '0'} 2023-07-29T00:39:36.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009759cf3e35054a5b-0064c45fc8-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.137 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:39:36.138 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:39:36.138 DEBUG:boto:Method: GET 2023-07-29T00:39:36.139 DEBUG:boto:Path: /hiiqos-59/obj22 2023-07-29T00:39:36.139 DEBUG:boto:Data: 2023-07-29T00:39:36.139 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.140 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:36.140 DEBUG:boto:Port: 8000 2023-07-29T00:39:36.141 DEBUG:boto:Params: {} 2023-07-29T00:39:36.141 DEBUG:boto:Token: None 2023-07-29T00:39:36.142 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj22 2023-07-29T00:39:36.142 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6YnEndE8g1/s6hBmWBAE7Y4bmVg= 2023-07-29T00:39:36.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6YnEndE8g1/s6hBmWBAE7Y4bmVg=', 'Content-Length': '0'} 2023-07-29T00:39:36.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003559b400530b03aa-0064c45fc8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.188 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:36.189 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:36.189 DEBUG:boto:path=/hiiqos-59/?prefix=obj22 2023-07-29T00:39:36.190 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj22 2023-07-29T00:39:36.190 DEBUG:boto:Method: GET 2023-07-29T00:39:36.190 DEBUG:boto:Path: /hiiqos-59/?prefix=obj22 2023-07-29T00:39:36.191 DEBUG:boto:Data: 2023-07-29T00:39:36.191 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.191 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:36.192 DEBUG:boto:Port: 8001 2023-07-29T00:39:36.192 DEBUG:boto:Params: {} 2023-07-29T00:39:36.193 DEBUG:boto:Token: None 2023-07-29T00:39:36.193 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/ 2023-07-29T00:39:36.194 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA= 2023-07-29T00:39:36.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA=', 'Content-Length': '0'} 2023-07-29T00:39:36.197 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e0035fc27b0e572-0064c45fc8-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.198 DEBUG:boto:b'hiiqos-59obj221000falseobj222023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:36.198 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:36.199 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:36.199 DEBUG:boto:path=/hiiqos-59/?prefix=obj22 2023-07-29T00:39:36.200 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj22 2023-07-29T00:39:36.200 DEBUG:boto:Method: GET 2023-07-29T00:39:36.201 DEBUG:boto:Path: /hiiqos-59/?prefix=obj22 2023-07-29T00:39:36.201 DEBUG:boto:Data: 2023-07-29T00:39:36.201 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.202 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:36.202 DEBUG:boto:Port: 8000 2023-07-29T00:39:36.203 DEBUG:boto:Params: {} 2023-07-29T00:39:36.203 DEBUG:boto:Token: None 2023-07-29T00:39:36.203 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/ 2023-07-29T00:39:36.204 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA= 2023-07-29T00:39:36.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA=', 'Content-Length': '0'} 2023-07-29T00:39:36.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c0ef2cfe293509d-0064c45fc8-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.209 DEBUG:boto:b'hiiqos-59obj221000falseobj222023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:36.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-07-29T00:39:36.211 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:39:36.212 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:39:36.212 DEBUG:boto:path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:39:36.213 DEBUG:boto:auth_path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:39:36.213 DEBUG:boto:Method: GET 2023-07-29T00:39:36.213 DEBUG:boto:Path: /hiiqos-59/obj23?versionId=null 2023-07-29T00:39:36.214 DEBUG:boto:Data: 2023-07-29T00:39:36.214 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.215 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:36.215 DEBUG:boto:Port: 8001 2023-07-29T00:39:36.215 DEBUG:boto:Params: {} 2023-07-29T00:39:36.216 DEBUG:boto:Token: None 2023-07-29T00:39:36.216 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj23?versionId=null 2023-07-29T00:39:36.217 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lLHQCVAX7dIzHwpz/O44QyoRTlc= 2023-07-29T00:39:36.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lLHQCVAX7dIzHwpz/O44QyoRTlc=', 'Content-Length': '0'} 2023-07-29T00:39:36.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fcce9ccece1db4d-0064c45fc8-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.220 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:39:36.220 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:39:36.221 DEBUG:boto:path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:39:36.221 DEBUG:boto:auth_path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:39:36.221 DEBUG:boto:Method: GET 2023-07-29T00:39:36.222 DEBUG:boto:Path: /hiiqos-59/obj23?versionId=null 2023-07-29T00:39:36.222 DEBUG:boto:Data: 2023-07-29T00:39:36.223 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.223 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:36.223 DEBUG:boto:Port: 8000 2023-07-29T00:39:36.224 DEBUG:boto:Params: {} 2023-07-29T00:39:36.224 DEBUG:boto:Token: None 2023-07-29T00:39:36.225 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj23?versionId=null 2023-07-29T00:39:36.225 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lLHQCVAX7dIzHwpz/O44QyoRTlc= 2023-07-29T00:39:36.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lLHQCVAX7dIzHwpz/O44QyoRTlc=', 'Content-Length': '0'} 2023-07-29T00:39:36.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a727ae161f904d01-0064c45fc8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.228 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:39:36.228 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:39:36.229 DEBUG:boto:path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:39:36.229 DEBUG:boto:auth_path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:39:36.230 DEBUG:boto:Method: HEAD 2023-07-29T00:39:36.230 DEBUG:boto:Path: /hiiqos-59/obj23?versionId=null 2023-07-29T00:39:36.230 DEBUG:boto:Data: 2023-07-29T00:39:36.231 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.231 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:36.231 DEBUG:boto:Port: 8001 2023-07-29T00:39:36.232 DEBUG:boto:Params: {} 2023-07-29T00:39:36.232 DEBUG:boto:Token: None 2023-07-29T00:39:36.233 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj23?versionId=null 2023-07-29T00:39:36.233 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nG4YNOp3V2Dn6S/b9F6r1651fR4= 2023-07-29T00:39:36.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nG4YNOp3V2Dn6S/b9F6r1651fR4=', 'Content-Length': '0'} 2023-07-29T00:39:36.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000135717a228451050-0064c45fc8-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.236 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:39:36.236 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:39:36.237 DEBUG:boto:path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:39:36.237 DEBUG:boto:auth_path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:39:36.238 DEBUG:boto:Method: HEAD 2023-07-29T00:39:36.238 DEBUG:boto:Path: /hiiqos-59/obj23?versionId=null 2023-07-29T00:39:36.238 DEBUG:boto:Data: 2023-07-29T00:39:36.239 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.239 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:36.239 DEBUG:boto:Port: 8000 2023-07-29T00:39:36.240 DEBUG:boto:Params: {} 2023-07-29T00:39:36.240 DEBUG:boto:Token: None 2023-07-29T00:39:36.241 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj23?versionId=null 2023-07-29T00:39:36.241 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nG4YNOp3V2Dn6S/b9F6r1651fR4= 2023-07-29T00:39:36.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nG4YNOp3V2Dn6S/b9F6r1651fR4=', 'Content-Length': '0'} 2023-07-29T00:39:36.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9eaad21b470eba9-0064c45fc8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-07-29T00:39:36.245 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:39:36.245 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:39:36.245 DEBUG:boto:path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:39:36.246 DEBUG:boto:auth_path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:39:36.246 DEBUG:boto:Method: GET 2023-07-29T00:39:36.247 DEBUG:boto:Path: /hiiqos-59/obj23?versionId=null 2023-07-29T00:39:36.247 DEBUG:boto:Data: 2023-07-29T00:39:36.247 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.248 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:36.248 DEBUG:boto:Port: 8001 2023-07-29T00:39:36.248 DEBUG:boto:Params: {} 2023-07-29T00:39:36.249 DEBUG:boto:Token: None 2023-07-29T00:39:36.249 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj23?versionId=null 2023-07-29T00:39:36.250 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lLHQCVAX7dIzHwpz/O44QyoRTlc= 2023-07-29T00:39:36.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lLHQCVAX7dIzHwpz/O44QyoRTlc=', 'Content-Length': '0'} 2023-07-29T00:39:36.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000045426a8501f436-0064c45fc8-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.253 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:39:36.254 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:39:36.254 DEBUG:boto:path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:39:36.254 DEBUG:boto:auth_path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:39:36.255 DEBUG:boto:Method: GET 2023-07-29T00:39:36.255 DEBUG:boto:Path: /hiiqos-59/obj23?versionId=null 2023-07-29T00:39:36.255 DEBUG:boto:Data: 2023-07-29T00:39:36.256 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.256 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:36.256 DEBUG:boto:Port: 8000 2023-07-29T00:39:36.257 DEBUG:boto:Params: {} 2023-07-29T00:39:36.257 DEBUG:boto:Token: None 2023-07-29T00:39:36.258 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj23?versionId=null 2023-07-29T00:39:36.258 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lLHQCVAX7dIzHwpz/O44QyoRTlc= 2023-07-29T00:39:36.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lLHQCVAX7dIzHwpz/O44QyoRTlc=', 'Content-Length': '0'} 2023-07-29T00:39:36.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fef3dc95fc625d02-0064c45fc8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.261 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:36.261 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:36.262 DEBUG:boto:path=/hiiqos-59/?prefix=obj23 2023-07-29T00:39:36.262 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj23 2023-07-29T00:39:36.263 DEBUG:boto:Method: GET 2023-07-29T00:39:36.263 DEBUG:boto:Path: /hiiqos-59/?prefix=obj23 2023-07-29T00:39:36.263 DEBUG:boto:Data: 2023-07-29T00:39:36.264 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.264 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:36.265 DEBUG:boto:Port: 8001 2023-07-29T00:39:36.265 DEBUG:boto:Params: {} 2023-07-29T00:39:36.265 DEBUG:boto:Token: None 2023-07-29T00:39:36.266 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/ 2023-07-29T00:39:36.266 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA= 2023-07-29T00:39:36.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA=', 'Content-Length': '0'} 2023-07-29T00:39:36.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b4c98f4f66ae9a7-0064c45fc8-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.316 DEBUG:boto:b'hiiqos-59obj231000falseobj232023-07-29T00:33:06.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:36.317 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:36.317 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:36.318 DEBUG:boto:path=/hiiqos-59/?prefix=obj23 2023-07-29T00:39:36.318 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj23 2023-07-29T00:39:36.319 DEBUG:boto:Method: GET 2023-07-29T00:39:36.319 DEBUG:boto:Path: /hiiqos-59/?prefix=obj23 2023-07-29T00:39:36.319 DEBUG:boto:Data: 2023-07-29T00:39:36.320 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.320 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:36.320 DEBUG:boto:Port: 8000 2023-07-29T00:39:36.321 DEBUG:boto:Params: {} 2023-07-29T00:39:36.321 DEBUG:boto:Token: None 2023-07-29T00:39:36.322 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/ 2023-07-29T00:39:36.322 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA= 2023-07-29T00:39:36.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA=', 'Content-Length': '0'} 2023-07-29T00:39:36.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab6f2db7f28b1c39-0064c45fc8-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.327 DEBUG:boto:b'hiiqos-59obj231000falseobj232023-07-29T00:33:06.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:36.328 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:39:36.328 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:39:36.329 DEBUG:boto:Method: HEAD 2023-07-29T00:39:36.329 DEBUG:boto:Path: /hiiqos-59/obj23 2023-07-29T00:39:36.330 DEBUG:boto:Data: 2023-07-29T00:39:36.330 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.330 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:36.331 DEBUG:boto:Port: 8001 2023-07-29T00:39:36.331 DEBUG:boto:Params: {} 2023-07-29T00:39:36.331 DEBUG:boto:Token: None 2023-07-29T00:39:36.332 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj23 2023-07-29T00:39:36.332 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:25QQl67tVk7bapfVbc/KASUgnqM= 2023-07-29T00:39:36.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:25QQl67tVk7bapfVbc/KASUgnqM=', 'Content-Length': '0'} 2023-07-29T00:39:36.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a868d641d3f42d5d-0064c45fc8-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.335 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:39:36.335 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:39:36.336 DEBUG:boto:Method: HEAD 2023-07-29T00:39:36.336 DEBUG:boto:Path: /hiiqos-59/obj23 2023-07-29T00:39:36.336 DEBUG:boto:Data: 2023-07-29T00:39:36.337 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.337 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:36.337 DEBUG:boto:Port: 8000 2023-07-29T00:39:36.338 DEBUG:boto:Params: {} 2023-07-29T00:39:36.338 DEBUG:boto:Token: None 2023-07-29T00:39:36.338 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj23 2023-07-29T00:39:36.339 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:25QQl67tVk7bapfVbc/KASUgnqM= 2023-07-29T00:39:36.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:25QQl67tVk7bapfVbc/KASUgnqM=', 'Content-Length': '0'} 2023-07-29T00:39:36.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053292802195ff279-0064c45fc8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-07-29T00:39:36.342 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:39:36.343 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:39:36.343 DEBUG:boto:Method: GET 2023-07-29T00:39:36.343 DEBUG:boto:Path: /hiiqos-59/obj23 2023-07-29T00:39:36.344 DEBUG:boto:Data: 2023-07-29T00:39:36.344 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.345 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:36.345 DEBUG:boto:Port: 8001 2023-07-29T00:39:36.345 DEBUG:boto:Params: {} 2023-07-29T00:39:36.346 DEBUG:boto:Token: None 2023-07-29T00:39:36.346 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj23 2023-07-29T00:39:36.346 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:T5MMt3RQj3UiAaqcAUHg4yotOMA= 2023-07-29T00:39:36.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:T5MMt3RQj3UiAaqcAUHg4yotOMA=', 'Content-Length': '0'} 2023-07-29T00:39:36.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f17a098940318c5-0064c45fc8-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.349 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:39:36.350 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:39:36.350 DEBUG:boto:Method: GET 2023-07-29T00:39:36.350 DEBUG:boto:Path: /hiiqos-59/obj23 2023-07-29T00:39:36.351 DEBUG:boto:Data: 2023-07-29T00:39:36.351 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.351 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:36.352 DEBUG:boto:Port: 8000 2023-07-29T00:39:36.352 DEBUG:boto:Params: {} 2023-07-29T00:39:36.353 DEBUG:boto:Token: None 2023-07-29T00:39:36.353 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj23 2023-07-29T00:39:36.354 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:T5MMt3RQj3UiAaqcAUHg4yotOMA= 2023-07-29T00:39:36.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:T5MMt3RQj3UiAaqcAUHg4yotOMA=', 'Content-Length': '0'} 2023-07-29T00:39:36.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001303a032da06bce5-0064c45fc8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.396 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:36.397 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:36.397 DEBUG:boto:path=/hiiqos-59/?prefix=obj23 2023-07-29T00:39:36.398 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj23 2023-07-29T00:39:36.398 DEBUG:boto:Method: GET 2023-07-29T00:39:36.398 DEBUG:boto:Path: /hiiqos-59/?prefix=obj23 2023-07-29T00:39:36.399 DEBUG:boto:Data: 2023-07-29T00:39:36.399 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.399 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:36.400 DEBUG:boto:Port: 8001 2023-07-29T00:39:36.400 DEBUG:boto:Params: {} 2023-07-29T00:39:36.401 DEBUG:boto:Token: None 2023-07-29T00:39:36.401 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/ 2023-07-29T00:39:36.402 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA= 2023-07-29T00:39:36.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA=', 'Content-Length': '0'} 2023-07-29T00:39:36.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007864550a2486b845-0064c45fc8-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.405 DEBUG:boto:b'hiiqos-59obj231000falseobj232023-07-29T00:33:06.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:36.406 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:36.406 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:36.406 DEBUG:boto:path=/hiiqos-59/?prefix=obj23 2023-07-29T00:39:36.407 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj23 2023-07-29T00:39:36.407 DEBUG:boto:Method: GET 2023-07-29T00:39:36.407 DEBUG:boto:Path: /hiiqos-59/?prefix=obj23 2023-07-29T00:39:36.408 DEBUG:boto:Data: 2023-07-29T00:39:36.408 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.409 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:36.409 DEBUG:boto:Port: 8000 2023-07-29T00:39:36.409 DEBUG:boto:Params: {} 2023-07-29T00:39:36.410 DEBUG:boto:Token: None 2023-07-29T00:39:36.410 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/ 2023-07-29T00:39:36.411 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA= 2023-07-29T00:39:36.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA=', 'Content-Length': '0'} 2023-07-29T00:39:36.415 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038bbb8317585504c-0064c45fc8-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.415 DEBUG:boto:b'hiiqos-59obj231000falseobj232023-07-29T00:33:06.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:36.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-07-29T00:39:36.416 DEBUG:boto:path=/hiiqos-59/obj29 2023-07-29T00:39:36.417 DEBUG:boto:auth_path=/hiiqos-59/obj29 2023-07-29T00:39:36.417 DEBUG:boto:path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:39:36.417 DEBUG:boto:auth_path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:39:36.418 DEBUG:boto:Method: GET 2023-07-29T00:39:36.418 DEBUG:boto:Path: /hiiqos-59/obj29?versionId=null 2023-07-29T00:39:36.418 DEBUG:boto:Data: 2023-07-29T00:39:36.419 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.419 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:36.419 DEBUG:boto:Port: 8001 2023-07-29T00:39:36.420 DEBUG:boto:Params: {} 2023-07-29T00:39:36.420 DEBUG:boto:Token: None 2023-07-29T00:39:36.421 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj29?versionId=null 2023-07-29T00:39:36.421 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dhdxedbcQ/L2btyim8eepgiZBdQ= 2023-07-29T00:39:36.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dhdxedbcQ/L2btyim8eepgiZBdQ=', 'Content-Length': '0'} 2023-07-29T00:39:36.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc0a74623497e0a3-0064c45fc8-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.424 DEBUG:boto:path=/hiiqos-59/obj29 2023-07-29T00:39:36.425 DEBUG:boto:auth_path=/hiiqos-59/obj29 2023-07-29T00:39:36.425 DEBUG:boto:path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:39:36.426 DEBUG:boto:auth_path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:39:36.426 DEBUG:boto:Method: GET 2023-07-29T00:39:36.426 DEBUG:boto:Path: /hiiqos-59/obj29?versionId=null 2023-07-29T00:39:36.427 DEBUG:boto:Data: 2023-07-29T00:39:36.427 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.427 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:36.428 DEBUG:boto:Port: 8000 2023-07-29T00:39:36.428 DEBUG:boto:Params: {} 2023-07-29T00:39:36.429 DEBUG:boto:Token: None 2023-07-29T00:39:36.429 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj29?versionId=null 2023-07-29T00:39:36.429 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dhdxedbcQ/L2btyim8eepgiZBdQ= 2023-07-29T00:39:36.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dhdxedbcQ/L2btyim8eepgiZBdQ=', 'Content-Length': '0'} 2023-07-29T00:39:36.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063f27c2579a0c21a-0064c45fc8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.432 DEBUG:boto:path=/hiiqos-59/obj29 2023-07-29T00:39:36.433 DEBUG:boto:auth_path=/hiiqos-59/obj29 2023-07-29T00:39:36.433 DEBUG:boto:path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:39:36.433 DEBUG:boto:auth_path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:39:36.434 DEBUG:boto:Method: HEAD 2023-07-29T00:39:36.434 DEBUG:boto:Path: /hiiqos-59/obj29?versionId=null 2023-07-29T00:39:36.434 DEBUG:boto:Data: 2023-07-29T00:39:36.435 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.435 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:36.435 DEBUG:boto:Port: 8001 2023-07-29T00:39:36.436 DEBUG:boto:Params: {} 2023-07-29T00:39:36.436 DEBUG:boto:Token: None 2023-07-29T00:39:36.437 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj29?versionId=null 2023-07-29T00:39:36.437 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cV3h4IV7ZQucv2pdQrpgstn5ESw= 2023-07-29T00:39:36.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cV3h4IV7ZQucv2pdQrpgstn5ESw=', 'Content-Length': '0'} 2023-07-29T00:39:36.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d1cff514109e82a-0064c45fc8-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.440 DEBUG:boto:path=/hiiqos-59/obj29 2023-07-29T00:39:36.440 DEBUG:boto:auth_path=/hiiqos-59/obj29 2023-07-29T00:39:36.440 DEBUG:boto:path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:39:36.441 DEBUG:boto:auth_path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:39:36.441 DEBUG:boto:Method: HEAD 2023-07-29T00:39:36.442 DEBUG:boto:Path: /hiiqos-59/obj29?versionId=null 2023-07-29T00:39:36.442 DEBUG:boto:Data: 2023-07-29T00:39:36.442 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.443 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:36.443 DEBUG:boto:Port: 8000 2023-07-29T00:39:36.444 DEBUG:boto:Params: {} 2023-07-29T00:39:36.444 DEBUG:boto:Token: None 2023-07-29T00:39:36.444 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj29?versionId=null 2023-07-29T00:39:36.445 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cV3h4IV7ZQucv2pdQrpgstn5ESw= 2023-07-29T00:39:36.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cV3h4IV7ZQucv2pdQrpgstn5ESw=', 'Content-Length': '0'} 2023-07-29T00:39:36.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fcf39f69c913d36-0064c45fc8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.448 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-07-29T00:39:36.448 DEBUG:boto:path=/hiiqos-59/obj29 2023-07-29T00:39:36.449 DEBUG:boto:auth_path=/hiiqos-59/obj29 2023-07-29T00:39:36.449 DEBUG:boto:path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:39:36.449 DEBUG:boto:auth_path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:39:36.450 DEBUG:boto:Method: GET 2023-07-29T00:39:36.450 DEBUG:boto:Path: /hiiqos-59/obj29?versionId=null 2023-07-29T00:39:36.450 DEBUG:boto:Data: 2023-07-29T00:39:36.451 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.451 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:36.452 DEBUG:boto:Port: 8001 2023-07-29T00:39:36.452 DEBUG:boto:Params: {} 2023-07-29T00:39:36.452 DEBUG:boto:Token: None 2023-07-29T00:39:36.453 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj29?versionId=null 2023-07-29T00:39:36.453 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dhdxedbcQ/L2btyim8eepgiZBdQ= 2023-07-29T00:39:36.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dhdxedbcQ/L2btyim8eepgiZBdQ=', 'Content-Length': '0'} 2023-07-29T00:39:36.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086242047c2f8404d-0064c45fc8-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.457 DEBUG:boto:path=/hiiqos-59/obj29 2023-07-29T00:39:36.458 DEBUG:boto:auth_path=/hiiqos-59/obj29 2023-07-29T00:39:36.458 DEBUG:boto:path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:39:36.459 DEBUG:boto:auth_path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:39:36.460 DEBUG:boto:Method: GET 2023-07-29T00:39:36.461 DEBUG:boto:Path: /hiiqos-59/obj29?versionId=null 2023-07-29T00:39:36.461 DEBUG:boto:Data: 2023-07-29T00:39:36.462 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.462 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:36.462 DEBUG:boto:Port: 8000 2023-07-29T00:39:36.463 DEBUG:boto:Params: {} 2023-07-29T00:39:36.463 DEBUG:boto:Token: None 2023-07-29T00:39:36.464 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj29?versionId=null 2023-07-29T00:39:36.464 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dhdxedbcQ/L2btyim8eepgiZBdQ= 2023-07-29T00:39:36.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dhdxedbcQ/L2btyim8eepgiZBdQ=', 'Content-Length': '0'} 2023-07-29T00:39:36.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad8fd1a7c3de1022-0064c45fc8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.467 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:36.468 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:36.468 DEBUG:boto:path=/hiiqos-59/?prefix=obj29 2023-07-29T00:39:36.469 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj29 2023-07-29T00:39:36.469 DEBUG:boto:Method: GET 2023-07-29T00:39:36.470 DEBUG:boto:Path: /hiiqos-59/?prefix=obj29 2023-07-29T00:39:36.470 DEBUG:boto:Data: 2023-07-29T00:39:36.470 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.471 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:36.471 DEBUG:boto:Port: 8001 2023-07-29T00:39:36.471 DEBUG:boto:Params: {} 2023-07-29T00:39:36.472 DEBUG:boto:Token: None 2023-07-29T00:39:36.472 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/ 2023-07-29T00:39:36.473 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA= 2023-07-29T00:39:36.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA=', 'Content-Length': '0'} 2023-07-29T00:39:36.475 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9496513ddf3e326-0064c45fc8-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.476 DEBUG:boto:b'hiiqos-59obj291000falseobj292023-07-29T00:35:01.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:36.477 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:36.477 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:36.477 DEBUG:boto:path=/hiiqos-59/?prefix=obj29 2023-07-29T00:39:36.478 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj29 2023-07-29T00:39:36.478 DEBUG:boto:Method: GET 2023-07-29T00:39:36.478 DEBUG:boto:Path: /hiiqos-59/?prefix=obj29 2023-07-29T00:39:36.479 DEBUG:boto:Data: 2023-07-29T00:39:36.479 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.479 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:36.480 DEBUG:boto:Port: 8000 2023-07-29T00:39:36.480 DEBUG:boto:Params: {} 2023-07-29T00:39:36.481 DEBUG:boto:Token: None 2023-07-29T00:39:36.481 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/ 2023-07-29T00:39:36.482 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA= 2023-07-29T00:39:36.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA=', 'Content-Length': '0'} 2023-07-29T00:39:36.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ba38ac74d4aa294-0064c45fc8-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.487 DEBUG:boto:b'hiiqos-59obj291000falseobj292023-07-29T00:35:01.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:36.487 DEBUG:boto:path=/hiiqos-59/obj29 2023-07-29T00:39:36.488 DEBUG:boto:auth_path=/hiiqos-59/obj29 2023-07-29T00:39:36.488 DEBUG:boto:Method: HEAD 2023-07-29T00:39:36.488 DEBUG:boto:Path: /hiiqos-59/obj29 2023-07-29T00:39:36.489 DEBUG:boto:Data: 2023-07-29T00:39:36.490 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.490 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:36.490 DEBUG:boto:Port: 8001 2023-07-29T00:39:36.491 DEBUG:boto:Params: {} 2023-07-29T00:39:36.491 DEBUG:boto:Token: None 2023-07-29T00:39:36.491 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj29 2023-07-29T00:39:36.492 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zl1HqyMvnfZoI3bz2u7PL3MlUB0= 2023-07-29T00:39:36.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zl1HqyMvnfZoI3bz2u7PL3MlUB0=', 'Content-Length': '0'} 2023-07-29T00:39:36.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000facaa402b084c540-0064c45fc8-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.495 DEBUG:boto:path=/hiiqos-59/obj29 2023-07-29T00:39:36.495 DEBUG:boto:auth_path=/hiiqos-59/obj29 2023-07-29T00:39:36.495 DEBUG:boto:Method: HEAD 2023-07-29T00:39:36.496 DEBUG:boto:Path: /hiiqos-59/obj29 2023-07-29T00:39:36.496 DEBUG:boto:Data: 2023-07-29T00:39:36.497 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.497 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:36.497 DEBUG:boto:Port: 8000 2023-07-29T00:39:36.498 DEBUG:boto:Params: {} 2023-07-29T00:39:36.498 DEBUG:boto:Token: None 2023-07-29T00:39:36.498 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj29 2023-07-29T00:39:36.499 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zl1HqyMvnfZoI3bz2u7PL3MlUB0= 2023-07-29T00:39:36.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zl1HqyMvnfZoI3bz2u7PL3MlUB0=', 'Content-Length': '0'} 2023-07-29T00:39:36.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c989a0da2f1b6c3a-0064c45fc8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-07-29T00:39:36.502 DEBUG:boto:path=/hiiqos-59/obj29 2023-07-29T00:39:36.502 DEBUG:boto:auth_path=/hiiqos-59/obj29 2023-07-29T00:39:36.503 DEBUG:boto:Method: GET 2023-07-29T00:39:36.503 DEBUG:boto:Path: /hiiqos-59/obj29 2023-07-29T00:39:36.504 DEBUG:boto:Data: 2023-07-29T00:39:36.504 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.504 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:36.505 DEBUG:boto:Port: 8001 2023-07-29T00:39:36.505 DEBUG:boto:Params: {} 2023-07-29T00:39:36.505 DEBUG:boto:Token: None 2023-07-29T00:39:36.506 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj29 2023-07-29T00:39:36.506 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pgu88qtyRuVn+jG1qBSCL+nwFZ8= 2023-07-29T00:39:36.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pgu88qtyRuVn+jG1qBSCL+nwFZ8=', 'Content-Length': '0'} 2023-07-29T00:39:36.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fa3fde510553d8a-0064c45fc8-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.509 DEBUG:boto:path=/hiiqos-59/obj29 2023-07-29T00:39:36.510 DEBUG:boto:auth_path=/hiiqos-59/obj29 2023-07-29T00:39:36.510 DEBUG:boto:Method: GET 2023-07-29T00:39:36.510 DEBUG:boto:Path: /hiiqos-59/obj29 2023-07-29T00:39:36.511 DEBUG:boto:Data: 2023-07-29T00:39:36.511 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.511 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:36.512 DEBUG:boto:Port: 8000 2023-07-29T00:39:36.512 DEBUG:boto:Params: {} 2023-07-29T00:39:36.513 DEBUG:boto:Token: None 2023-07-29T00:39:36.513 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj29 2023-07-29T00:39:36.513 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pgu88qtyRuVn+jG1qBSCL+nwFZ8= 2023-07-29T00:39:36.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pgu88qtyRuVn+jG1qBSCL+nwFZ8=', 'Content-Length': '0'} 2023-07-29T00:39:36.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077984b1e0077d396-0064c45fc8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.556 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:36.558 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:36.558 DEBUG:boto:path=/hiiqos-59/?prefix=obj29 2023-07-29T00:39:36.559 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj29 2023-07-29T00:39:36.559 DEBUG:boto:Method: GET 2023-07-29T00:39:36.559 DEBUG:boto:Path: /hiiqos-59/?prefix=obj29 2023-07-29T00:39:36.560 DEBUG:boto:Data: 2023-07-29T00:39:36.560 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.561 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:36.561 DEBUG:boto:Port: 8001 2023-07-29T00:39:36.561 DEBUG:boto:Params: {} 2023-07-29T00:39:36.562 DEBUG:boto:Token: None 2023-07-29T00:39:36.562 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/ 2023-07-29T00:39:36.563 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA= 2023-07-29T00:39:36.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA=', 'Content-Length': '0'} 2023-07-29T00:39:36.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000005ebd9c966a26c4-0064c45fc8-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.566 DEBUG:boto:b'hiiqos-59obj291000falseobj292023-07-29T00:35:01.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:36.567 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:36.568 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:36.568 DEBUG:boto:path=/hiiqos-59/?prefix=obj29 2023-07-29T00:39:36.568 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj29 2023-07-29T00:39:36.572 DEBUG:boto:Method: GET 2023-07-29T00:39:36.572 DEBUG:boto:Path: /hiiqos-59/?prefix=obj29 2023-07-29T00:39:36.573 DEBUG:boto:Data: 2023-07-29T00:39:36.573 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.573 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:36.574 DEBUG:boto:Port: 8000 2023-07-29T00:39:36.574 DEBUG:boto:Params: {} 2023-07-29T00:39:36.574 DEBUG:boto:Token: None 2023-07-29T00:39:36.575 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/ 2023-07-29T00:39:36.575 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA= 2023-07-29T00:39:36.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA=', 'Content-Length': '0'} 2023-07-29T00:39:36.580 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096d8c278062e04f5-0064c45fc8-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.580 DEBUG:boto:b'hiiqos-59obj291000falseobj292023-07-29T00:35:01.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:36.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-07-29T00:39:36.581 DEBUG:boto:path=/hiiqos-59/obj31 2023-07-29T00:39:36.582 DEBUG:boto:auth_path=/hiiqos-59/obj31 2023-07-29T00:39:36.582 DEBUG:boto:path=/hiiqos-59/obj31?versionId=null 2023-07-29T00:39:36.583 DEBUG:boto:auth_path=/hiiqos-59/obj31?versionId=null 2023-07-29T00:39:36.583 DEBUG:boto:Method: GET 2023-07-29T00:39:36.583 DEBUG:boto:Path: /hiiqos-59/obj31?versionId=null 2023-07-29T00:39:36.584 DEBUG:boto:Data: 2023-07-29T00:39:36.584 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.584 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:36.585 DEBUG:boto:Port: 8001 2023-07-29T00:39:36.585 DEBUG:boto:Params: {} 2023-07-29T00:39:36.586 DEBUG:boto:Token: None 2023-07-29T00:39:36.586 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj31?versionId=null 2023-07-29T00:39:36.586 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wyIiAKwCgVfntzqB6U3HmL98iWk= 2023-07-29T00:39:36.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wyIiAKwCgVfntzqB6U3HmL98iWk=', 'Content-Length': '0'} 2023-07-29T00:39:36.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eae0de1e76ad9360-0064c45fc8-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.636 DEBUG:boto:path=/hiiqos-59/obj31 2023-07-29T00:39:36.637 DEBUG:boto:auth_path=/hiiqos-59/obj31 2023-07-29T00:39:36.637 DEBUG:boto:path=/hiiqos-59/obj31?versionId=null 2023-07-29T00:39:36.638 DEBUG:boto:auth_path=/hiiqos-59/obj31?versionId=null 2023-07-29T00:39:36.638 DEBUG:boto:Method: GET 2023-07-29T00:39:36.638 DEBUG:boto:Path: /hiiqos-59/obj31?versionId=null 2023-07-29T00:39:36.639 DEBUG:boto:Data: 2023-07-29T00:39:36.639 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.640 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:36.640 DEBUG:boto:Port: 8000 2023-07-29T00:39:36.641 DEBUG:boto:Params: {} 2023-07-29T00:39:36.641 DEBUG:boto:Token: None 2023-07-29T00:39:36.641 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj31?versionId=null 2023-07-29T00:39:36.642 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wyIiAKwCgVfntzqB6U3HmL98iWk= 2023-07-29T00:39:36.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wyIiAKwCgVfntzqB6U3HmL98iWk=', 'Content-Length': '0'} 2023-07-29T00:39:36.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a1575f405915787-0064c45fc8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.645 DEBUG:boto:path=/hiiqos-59/obj31 2023-07-29T00:39:36.645 DEBUG:boto:auth_path=/hiiqos-59/obj31 2023-07-29T00:39:36.645 DEBUG:boto:path=/hiiqos-59/obj31?versionId=null 2023-07-29T00:39:36.646 DEBUG:boto:auth_path=/hiiqos-59/obj31?versionId=null 2023-07-29T00:39:36.646 DEBUG:boto:Method: HEAD 2023-07-29T00:39:36.647 DEBUG:boto:Path: /hiiqos-59/obj31?versionId=null 2023-07-29T00:39:36.647 DEBUG:boto:Data: 2023-07-29T00:39:36.647 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.647 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:36.648 DEBUG:boto:Port: 8001 2023-07-29T00:39:36.648 DEBUG:boto:Params: {} 2023-07-29T00:39:36.649 DEBUG:boto:Token: None 2023-07-29T00:39:36.649 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj31?versionId=null 2023-07-29T00:39:36.649 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Uiz6YwZidYhlucWyFrvgK97IG3Q= 2023-07-29T00:39:36.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Uiz6YwZidYhlucWyFrvgK97IG3Q=', 'Content-Length': '0'} 2023-07-29T00:39:36.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ded7f59aed23e5d-0064c45fc8-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.652 DEBUG:boto:path=/hiiqos-59/obj31 2023-07-29T00:39:36.652 DEBUG:boto:auth_path=/hiiqos-59/obj31 2023-07-29T00:39:36.653 DEBUG:boto:path=/hiiqos-59/obj31?versionId=null 2023-07-29T00:39:36.653 DEBUG:boto:auth_path=/hiiqos-59/obj31?versionId=null 2023-07-29T00:39:36.653 DEBUG:boto:Method: HEAD 2023-07-29T00:39:36.654 DEBUG:boto:Path: /hiiqos-59/obj31?versionId=null 2023-07-29T00:39:36.654 DEBUG:boto:Data: 2023-07-29T00:39:36.654 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.655 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:36.655 DEBUG:boto:Port: 8000 2023-07-29T00:39:36.655 DEBUG:boto:Params: {} 2023-07-29T00:39:36.656 DEBUG:boto:Token: None 2023-07-29T00:39:36.656 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj31?versionId=null 2023-07-29T00:39:36.657 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Uiz6YwZidYhlucWyFrvgK97IG3Q= 2023-07-29T00:39:36.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Uiz6YwZidYhlucWyFrvgK97IG3Q=', 'Content-Length': '0'} 2023-07-29T00:39:36.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ef69259802a6b5e-0064c45fc8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-07-29T00:39:36.660 DEBUG:boto:path=/hiiqos-59/obj31 2023-07-29T00:39:36.660 DEBUG:boto:auth_path=/hiiqos-59/obj31 2023-07-29T00:39:36.660 DEBUG:boto:path=/hiiqos-59/obj31?versionId=null 2023-07-29T00:39:36.661 DEBUG:boto:auth_path=/hiiqos-59/obj31?versionId=null 2023-07-29T00:39:36.661 DEBUG:boto:Method: GET 2023-07-29T00:39:36.662 DEBUG:boto:Path: /hiiqos-59/obj31?versionId=null 2023-07-29T00:39:36.662 DEBUG:boto:Data: 2023-07-29T00:39:36.662 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.663 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:36.663 DEBUG:boto:Port: 8001 2023-07-29T00:39:36.664 DEBUG:boto:Params: {} 2023-07-29T00:39:36.664 DEBUG:boto:Token: None 2023-07-29T00:39:36.664 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj31?versionId=null 2023-07-29T00:39:36.665 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wyIiAKwCgVfntzqB6U3HmL98iWk= 2023-07-29T00:39:36.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wyIiAKwCgVfntzqB6U3HmL98iWk=', 'Content-Length': '0'} 2023-07-29T00:39:36.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fccb4411169a76ff-0064c45fc8-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.668 DEBUG:boto:path=/hiiqos-59/obj31 2023-07-29T00:39:36.668 DEBUG:boto:auth_path=/hiiqos-59/obj31 2023-07-29T00:39:36.669 DEBUG:boto:path=/hiiqos-59/obj31?versionId=null 2023-07-29T00:39:36.669 DEBUG:boto:auth_path=/hiiqos-59/obj31?versionId=null 2023-07-29T00:39:36.669 DEBUG:boto:Method: GET 2023-07-29T00:39:36.670 DEBUG:boto:Path: /hiiqos-59/obj31?versionId=null 2023-07-29T00:39:36.670 DEBUG:boto:Data: 2023-07-29T00:39:36.670 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.671 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:36.671 DEBUG:boto:Port: 8000 2023-07-29T00:39:36.672 DEBUG:boto:Params: {} 2023-07-29T00:39:36.672 DEBUG:boto:Token: None 2023-07-29T00:39:36.672 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj31?versionId=null 2023-07-29T00:39:36.673 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wyIiAKwCgVfntzqB6U3HmL98iWk= 2023-07-29T00:39:36.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wyIiAKwCgVfntzqB6U3HmL98iWk=', 'Content-Length': '0'} 2023-07-29T00:39:36.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b720e0b0427e0d8d-0064c45fc8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.675 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:36.676 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:36.676 DEBUG:boto:path=/hiiqos-59/?prefix=obj31 2023-07-29T00:39:36.677 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj31 2023-07-29T00:39:36.677 DEBUG:boto:Method: GET 2023-07-29T00:39:36.678 DEBUG:boto:Path: /hiiqos-59/?prefix=obj31 2023-07-29T00:39:36.678 DEBUG:boto:Data: 2023-07-29T00:39:36.678 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.679 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:36.679 DEBUG:boto:Port: 8001 2023-07-29T00:39:36.680 DEBUG:boto:Params: {} 2023-07-29T00:39:36.680 DEBUG:boto:Token: None 2023-07-29T00:39:36.680 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/ 2023-07-29T00:39:36.681 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA= 2023-07-29T00:39:36.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA=', 'Content-Length': '0'} 2023-07-29T00:39:36.683 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1b9283e1f201f68-0064c45fc8-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.684 DEBUG:boto:b'hiiqos-59obj311000falseobj312023-07-29T00:35:31.771Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:36.685 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:36.685 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:36.685 DEBUG:boto:path=/hiiqos-59/?prefix=obj31 2023-07-29T00:39:36.686 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj31 2023-07-29T00:39:36.686 DEBUG:boto:Method: GET 2023-07-29T00:39:36.687 DEBUG:boto:Path: /hiiqos-59/?prefix=obj31 2023-07-29T00:39:36.687 DEBUG:boto:Data: 2023-07-29T00:39:36.687 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.688 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:36.688 DEBUG:boto:Port: 8000 2023-07-29T00:39:36.688 DEBUG:boto:Params: {} 2023-07-29T00:39:36.689 DEBUG:boto:Token: None 2023-07-29T00:39:36.689 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/ 2023-07-29T00:39:36.690 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA= 2023-07-29T00:39:36.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA=', 'Content-Length': '0'} 2023-07-29T00:39:36.693 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5bd2638133f1fea-0064c45fc8-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.694 DEBUG:boto:b'hiiqos-59obj311000falseobj312023-07-29T00:35:31.771Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:36.695 DEBUG:boto:path=/hiiqos-59/obj31 2023-07-29T00:39:36.695 DEBUG:boto:auth_path=/hiiqos-59/obj31 2023-07-29T00:39:36.695 DEBUG:boto:Method: HEAD 2023-07-29T00:39:36.696 DEBUG:boto:Path: /hiiqos-59/obj31 2023-07-29T00:39:36.696 DEBUG:boto:Data: 2023-07-29T00:39:36.697 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.697 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:36.697 DEBUG:boto:Port: 8001 2023-07-29T00:39:36.697 DEBUG:boto:Params: {} 2023-07-29T00:39:36.698 DEBUG:boto:Token: None 2023-07-29T00:39:36.698 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj31 2023-07-29T00:39:36.699 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:k6+J35WNQpilZdr+Msmf2VTv7PA= 2023-07-29T00:39:36.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:k6+J35WNQpilZdr+Msmf2VTv7PA=', 'Content-Length': '0'} 2023-07-29T00:39:36.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf4953a9d104fe8a-0064c45fc8-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.702 DEBUG:boto:path=/hiiqos-59/obj31 2023-07-29T00:39:36.702 DEBUG:boto:auth_path=/hiiqos-59/obj31 2023-07-29T00:39:36.703 DEBUG:boto:Method: HEAD 2023-07-29T00:39:36.703 DEBUG:boto:Path: /hiiqos-59/obj31 2023-07-29T00:39:36.703 DEBUG:boto:Data: 2023-07-29T00:39:36.704 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.704 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:36.705 DEBUG:boto:Port: 8000 2023-07-29T00:39:36.705 DEBUG:boto:Params: {} 2023-07-29T00:39:36.706 DEBUG:boto:Token: None 2023-07-29T00:39:36.706 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj31 2023-07-29T00:39:36.707 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:k6+J35WNQpilZdr+Msmf2VTv7PA= 2023-07-29T00:39:36.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:k6+J35WNQpilZdr+Msmf2VTv7PA=', 'Content-Length': '0'} 2023-07-29T00:39:36.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c998f41b8b2401b9-0064c45fc8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-07-29T00:39:36.710 DEBUG:boto:path=/hiiqos-59/obj31 2023-07-29T00:39:36.711 DEBUG:boto:auth_path=/hiiqos-59/obj31 2023-07-29T00:39:36.711 DEBUG:boto:Method: GET 2023-07-29T00:39:36.711 DEBUG:boto:Path: /hiiqos-59/obj31 2023-07-29T00:39:36.712 DEBUG:boto:Data: 2023-07-29T00:39:36.712 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.712 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:36.713 DEBUG:boto:Port: 8001 2023-07-29T00:39:36.713 DEBUG:boto:Params: {} 2023-07-29T00:39:36.714 DEBUG:boto:Token: None 2023-07-29T00:39:36.714 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj31 2023-07-29T00:39:36.715 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rZcN4wnU5njzqtKG5UfCpYJmA10= 2023-07-29T00:39:36.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rZcN4wnU5njzqtKG5UfCpYJmA10=', 'Content-Length': '0'} 2023-07-29T00:39:36.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058dc0076c3d66924-0064c45fc8-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.718 DEBUG:boto:path=/hiiqos-59/obj31 2023-07-29T00:39:36.718 DEBUG:boto:auth_path=/hiiqos-59/obj31 2023-07-29T00:39:36.718 DEBUG:boto:Method: GET 2023-07-29T00:39:36.719 DEBUG:boto:Path: /hiiqos-59/obj31 2023-07-29T00:39:36.719 DEBUG:boto:Data: 2023-07-29T00:39:36.720 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.720 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:36.720 DEBUG:boto:Port: 8000 2023-07-29T00:39:36.721 DEBUG:boto:Params: {} 2023-07-29T00:39:36.721 DEBUG:boto:Token: None 2023-07-29T00:39:36.722 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj31 2023-07-29T00:39:36.722 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rZcN4wnU5njzqtKG5UfCpYJmA10= 2023-07-29T00:39:36.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rZcN4wnU5njzqtKG5UfCpYJmA10=', 'Content-Length': '0'} 2023-07-29T00:39:36.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcca9c31c08291e3-0064c45fc8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.772 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:36.773 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:36.774 DEBUG:boto:path=/hiiqos-59/?prefix=obj31 2023-07-29T00:39:36.774 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj31 2023-07-29T00:39:36.774 DEBUG:boto:Method: GET 2023-07-29T00:39:36.775 DEBUG:boto:Path: /hiiqos-59/?prefix=obj31 2023-07-29T00:39:36.775 DEBUG:boto:Data: 2023-07-29T00:39:36.775 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.776 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:36.776 DEBUG:boto:Port: 8001 2023-07-29T00:39:36.777 DEBUG:boto:Params: {} 2023-07-29T00:39:36.777 DEBUG:boto:Token: None 2023-07-29T00:39:36.778 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/ 2023-07-29T00:39:36.778 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA= 2023-07-29T00:39:36.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA=', 'Content-Length': '0'} 2023-07-29T00:39:36.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1ca13984c737723-0064c45fc8-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.782 DEBUG:boto:b'hiiqos-59obj311000falseobj312023-07-29T00:35:31.771Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:36.783 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:36.783 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:36.783 DEBUG:boto:path=/hiiqos-59/?prefix=obj31 2023-07-29T00:39:36.784 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj31 2023-07-29T00:39:36.784 DEBUG:boto:Method: GET 2023-07-29T00:39:36.784 DEBUG:boto:Path: /hiiqos-59/?prefix=obj31 2023-07-29T00:39:36.785 DEBUG:boto:Data: 2023-07-29T00:39:36.785 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.786 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:36.786 DEBUG:boto:Port: 8000 2023-07-29T00:39:36.786 DEBUG:boto:Params: {} 2023-07-29T00:39:36.787 DEBUG:boto:Token: None 2023-07-29T00:39:36.787 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/ 2023-07-29T00:39:36.788 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA= 2023-07-29T00:39:36.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA=', 'Content-Length': '0'} 2023-07-29T00:39:36.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d85d39fb3fa1e39f-0064c45fc8-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.792 DEBUG:boto:b'hiiqos-59obj311000falseobj312023-07-29T00:35:31.771Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:36.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-07-29T00:39:36.795 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:39:36.795 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:39:36.796 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:39:36.797 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:39:36.798 DEBUG:boto:Method: GET 2023-07-29T00:39:36.799 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:39:36.799 DEBUG:boto:Data: 2023-07-29T00:39:36.801 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.801 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:36.801 DEBUG:boto:Port: 8001 2023-07-29T00:39:36.802 DEBUG:boto:Params: {} 2023-07-29T00:39:36.802 DEBUG:boto:Token: None 2023-07-29T00:39:36.803 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:39:36.803 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:t0T5pS7oTHO+z4KAHYiF433yZr8= 2023-07-29T00:39:36.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t0T5pS7oTHO+z4KAHYiF433yZr8=', 'Content-Length': '0'} 2023-07-29T00:39:36.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000caed417d911b876a-0064c45fc8-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.852 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:39:36.853 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:39:36.853 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:39:36.854 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:39:36.854 DEBUG:boto:Method: GET 2023-07-29T00:39:36.854 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:39:36.855 DEBUG:boto:Data: 2023-07-29T00:39:36.855 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.855 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:36.856 DEBUG:boto:Port: 8000 2023-07-29T00:39:36.856 DEBUG:boto:Params: {} 2023-07-29T00:39:36.857 DEBUG:boto:Token: None 2023-07-29T00:39:36.857 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:39:36.857 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:t0T5pS7oTHO+z4KAHYiF433yZr8= 2023-07-29T00:39:36.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t0T5pS7oTHO+z4KAHYiF433yZr8=', 'Content-Length': '0'} 2023-07-29T00:39:36.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbfd1e0f1d90111f-0064c45fc8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.860 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:39:36.861 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:39:36.861 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:39:36.862 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:39:36.862 DEBUG:boto:Method: HEAD 2023-07-29T00:39:36.862 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:39:36.863 DEBUG:boto:Data: 2023-07-29T00:39:36.863 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.863 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:36.864 DEBUG:boto:Port: 8001 2023-07-29T00:39:36.864 DEBUG:boto:Params: {} 2023-07-29T00:39:36.864 DEBUG:boto:Token: None 2023-07-29T00:39:36.865 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:39:36.865 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HWWlkJh+pftdFpaM/4Tv+UzduNI= 2023-07-29T00:39:36.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HWWlkJh+pftdFpaM/4Tv+UzduNI=', 'Content-Length': '0'} 2023-07-29T00:39:36.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031ae035a6804fdea-0064c45fc8-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.868 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:39:36.868 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:39:36.868 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:39:36.869 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:39:36.870 DEBUG:boto:Method: HEAD 2023-07-29T00:39:36.870 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:39:36.870 DEBUG:boto:Data: 2023-07-29T00:39:36.871 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.871 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:36.872 DEBUG:boto:Port: 8000 2023-07-29T00:39:36.872 DEBUG:boto:Params: {} 2023-07-29T00:39:36.872 DEBUG:boto:Token: None 2023-07-29T00:39:36.873 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:39:36.873 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HWWlkJh+pftdFpaM/4Tv+UzduNI= 2023-07-29T00:39:36.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HWWlkJh+pftdFpaM/4Tv+UzduNI=', 'Content-Length': '0'} 2023-07-29T00:39:36.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2eee8af5a2383e6-0064c45fc8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-07-29T00:39:36.877 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:39:36.877 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:39:36.878 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:39:36.878 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:39:36.878 DEBUG:boto:Method: GET 2023-07-29T00:39:36.879 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:39:36.879 DEBUG:boto:Data: 2023-07-29T00:39:36.879 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.880 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:36.880 DEBUG:boto:Port: 8001 2023-07-29T00:39:36.880 DEBUG:boto:Params: {} 2023-07-29T00:39:36.881 DEBUG:boto:Token: None 2023-07-29T00:39:36.881 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:39:36.882 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:t0T5pS7oTHO+z4KAHYiF433yZr8= 2023-07-29T00:39:36.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t0T5pS7oTHO+z4KAHYiF433yZr8=', 'Content-Length': '0'} 2023-07-29T00:39:36.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007983a4847d3f20af-0064c45fc8-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.885 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:39:36.885 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:39:36.885 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:39:36.886 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:39:36.886 DEBUG:boto:Method: GET 2023-07-29T00:39:36.886 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:39:36.887 DEBUG:boto:Data: 2023-07-29T00:39:36.887 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.888 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:36.888 DEBUG:boto:Port: 8000 2023-07-29T00:39:36.888 DEBUG:boto:Params: {} 2023-07-29T00:39:36.889 DEBUG:boto:Token: None 2023-07-29T00:39:36.889 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:39:36.889 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:t0T5pS7oTHO+z4KAHYiF433yZr8= 2023-07-29T00:39:36.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t0T5pS7oTHO+z4KAHYiF433yZr8=', 'Content-Length': '0'} 2023-07-29T00:39:36.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f68e4eb01773144-0064c45fc8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.892 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:36.893 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:36.893 DEBUG:boto:path=/hiiqos-59/?prefix=obj33 2023-07-29T00:39:36.893 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj33 2023-07-29T00:39:36.894 DEBUG:boto:Method: GET 2023-07-29T00:39:36.894 DEBUG:boto:Path: /hiiqos-59/?prefix=obj33 2023-07-29T00:39:36.894 DEBUG:boto:Data: 2023-07-29T00:39:36.895 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.895 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:36.895 DEBUG:boto:Port: 8001 2023-07-29T00:39:36.896 DEBUG:boto:Params: {} 2023-07-29T00:39:36.896 DEBUG:boto:Token: None 2023-07-29T00:39:36.897 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/ 2023-07-29T00:39:36.897 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA= 2023-07-29T00:39:36.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA=', 'Content-Length': '0'} 2023-07-29T00:39:36.900 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000145070db8901486f-0064c45fc8-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.900 DEBUG:boto:b'hiiqos-59obj331000falseobj332023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:36.901 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:36.901 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:36.902 DEBUG:boto:path=/hiiqos-59/?prefix=obj33 2023-07-29T00:39:36.902 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj33 2023-07-29T00:39:36.902 DEBUG:boto:Method: GET 2023-07-29T00:39:36.903 DEBUG:boto:Path: /hiiqos-59/?prefix=obj33 2023-07-29T00:39:36.903 DEBUG:boto:Data: 2023-07-29T00:39:36.903 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.904 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:36.904 DEBUG:boto:Port: 8000 2023-07-29T00:39:36.905 DEBUG:boto:Params: {} 2023-07-29T00:39:36.905 DEBUG:boto:Token: None 2023-07-29T00:39:36.905 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/ 2023-07-29T00:39:36.906 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA= 2023-07-29T00:39:36.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA=', 'Content-Length': '0'} 2023-07-29T00:39:36.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057d87800a9fc93fe-0064c45fc8-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.911 DEBUG:boto:b'hiiqos-59obj331000falseobj332023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:36.911 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:39:36.912 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:39:36.912 DEBUG:boto:Method: HEAD 2023-07-29T00:39:36.913 DEBUG:boto:Path: /hiiqos-59/obj33 2023-07-29T00:39:36.913 DEBUG:boto:Data: 2023-07-29T00:39:36.913 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.914 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:36.914 DEBUG:boto:Port: 8001 2023-07-29T00:39:36.914 DEBUG:boto:Params: {} 2023-07-29T00:39:36.915 DEBUG:boto:Token: None 2023-07-29T00:39:36.915 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj33 2023-07-29T00:39:36.915 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O4cEvxeHN7UNADq8cT2nVIwSk/A= 2023-07-29T00:39:36.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O4cEvxeHN7UNADq8cT2nVIwSk/A=', 'Content-Length': '0'} 2023-07-29T00:39:36.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005b98e601b7e8f45-0064c45fc8-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.918 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:39:36.919 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:39:36.919 DEBUG:boto:Method: HEAD 2023-07-29T00:39:36.919 DEBUG:boto:Path: /hiiqos-59/obj33 2023-07-29T00:39:36.920 DEBUG:boto:Data: 2023-07-29T00:39:36.920 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.920 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:36.921 DEBUG:boto:Port: 8000 2023-07-29T00:39:36.921 DEBUG:boto:Params: {} 2023-07-29T00:39:36.921 DEBUG:boto:Token: None 2023-07-29T00:39:36.922 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj33 2023-07-29T00:39:36.922 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O4cEvxeHN7UNADq8cT2nVIwSk/A= 2023-07-29T00:39:36.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O4cEvxeHN7UNADq8cT2nVIwSk/A=', 'Content-Length': '0'} 2023-07-29T00:39:36.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec0140dd5c91dcbe-0064c45fc8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-07-29T00:39:36.925 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:39:36.926 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:39:36.926 DEBUG:boto:Method: GET 2023-07-29T00:39:36.926 DEBUG:boto:Path: /hiiqos-59/obj33 2023-07-29T00:39:36.927 DEBUG:boto:Data: 2023-07-29T00:39:36.927 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.927 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:36.928 DEBUG:boto:Port: 8001 2023-07-29T00:39:36.928 DEBUG:boto:Params: {} 2023-07-29T00:39:36.929 DEBUG:boto:Token: None 2023-07-29T00:39:36.929 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj33 2023-07-29T00:39:36.929 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YsadzcQyX8gbBXvrf3vl5ULuyEY= 2023-07-29T00:39:36.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YsadzcQyX8gbBXvrf3vl5ULuyEY=', 'Content-Length': '0'} 2023-07-29T00:39:36.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050462e381df15d1f-0064c45fc8-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.932 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:39:36.933 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:39:36.933 DEBUG:boto:Method: GET 2023-07-29T00:39:36.934 DEBUG:boto:Path: /hiiqos-59/obj33 2023-07-29T00:39:36.934 DEBUG:boto:Data: 2023-07-29T00:39:36.934 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.935 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:36.935 DEBUG:boto:Port: 8000 2023-07-29T00:39:36.935 DEBUG:boto:Params: {} 2023-07-29T00:39:36.936 DEBUG:boto:Token: None 2023-07-29T00:39:36.936 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/obj33 2023-07-29T00:39:36.937 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YsadzcQyX8gbBXvrf3vl5ULuyEY= 2023-07-29T00:39:36.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YsadzcQyX8gbBXvrf3vl5ULuyEY=', 'Content-Length': '0'} 2023-07-29T00:39:36.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2cfcb2fc50db49a-0064c45fc8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.980 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:36.981 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:36.981 DEBUG:boto:path=/hiiqos-59/?prefix=obj33 2023-07-29T00:39:36.982 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj33 2023-07-29T00:39:36.982 DEBUG:boto:Method: GET 2023-07-29T00:39:36.982 DEBUG:boto:Path: /hiiqos-59/?prefix=obj33 2023-07-29T00:39:36.983 DEBUG:boto:Data: 2023-07-29T00:39:36.983 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.983 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:36.984 DEBUG:boto:Port: 8001 2023-07-29T00:39:36.984 DEBUG:boto:Params: {} 2023-07-29T00:39:36.985 DEBUG:boto:Token: None 2023-07-29T00:39:36.985 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/ 2023-07-29T00:39:36.985 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA= 2023-07-29T00:39:36.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA=', 'Content-Length': '0'} 2023-07-29T00:39:36.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be80ca6bf8d297d0-0064c45fc8-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:36.989 DEBUG:boto:b'hiiqos-59obj331000falseobj332023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:36.990 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:36.990 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:36.990 DEBUG:boto:path=/hiiqos-59/?prefix=obj33 2023-07-29T00:39:36.991 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj33 2023-07-29T00:39:36.991 DEBUG:boto:Method: GET 2023-07-29T00:39:36.991 DEBUG:boto:Path: /hiiqos-59/?prefix=obj33 2023-07-29T00:39:36.992 DEBUG:boto:Data: 2023-07-29T00:39:36.992 DEBUG:boto:Headers: {} 2023-07-29T00:39:36.993 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:36.993 DEBUG:boto:Port: 8000 2023-07-29T00:39:36.993 DEBUG:boto:Params: {} 2023-07-29T00:39:36.994 DEBUG:boto:Token: None 2023-07-29T00:39:36.994 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:36 GMT /hiiqos-59/ 2023-07-29T00:39:36.994 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA= 2023-07-29T00:39:36.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K6oMfXv105n6xCtwJam+FkoaNRA=', 'Content-Length': '0'} 2023-07-29T00:39:36.999 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e54a15c130b6830-0064c45fc8-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.000 DEBUG:boto:b'hiiqos-59obj331000falseobj332023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:37.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-07-29T00:39:37.001 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:39:37.001 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:39:37.002 DEBUG:boto:path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:39:37.002 DEBUG:boto:auth_path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:39:37.002 DEBUG:boto:Method: GET 2023-07-29T00:39:37.003 DEBUG:boto:Path: /hiiqos-59/obj34?versionId=null 2023-07-29T00:39:37.003 DEBUG:boto:Data: 2023-07-29T00:39:37.003 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.004 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:37.004 DEBUG:boto:Port: 8001 2023-07-29T00:39:37.005 DEBUG:boto:Params: {} 2023-07-29T00:39:37.005 DEBUG:boto:Token: None 2023-07-29T00:39:37.005 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj34?versionId=null 2023-07-29T00:39:37.006 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:d2UCmJGr7VlXjcsRyqFLh24dXUM= 2023-07-29T00:39:37.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:d2UCmJGr7VlXjcsRyqFLh24dXUM=', 'Content-Length': '0'} 2023-07-29T00:39:37.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007130d7c4b689a8a5-0064c45fc8-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.009 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:39:37.009 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:39:37.009 DEBUG:boto:path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:39:37.010 DEBUG:boto:auth_path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:39:37.010 DEBUG:boto:Method: GET 2023-07-29T00:39:37.010 DEBUG:boto:Path: /hiiqos-59/obj34?versionId=null 2023-07-29T00:39:37.011 DEBUG:boto:Data: 2023-07-29T00:39:37.011 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.012 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:37.012 DEBUG:boto:Port: 8000 2023-07-29T00:39:37.012 DEBUG:boto:Params: {} 2023-07-29T00:39:37.013 DEBUG:boto:Token: None 2023-07-29T00:39:37.013 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj34?versionId=null 2023-07-29T00:39:37.014 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:d2UCmJGr7VlXjcsRyqFLh24dXUM= 2023-07-29T00:39:37.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:d2UCmJGr7VlXjcsRyqFLh24dXUM=', 'Content-Length': '0'} 2023-07-29T00:39:37.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ad09f1341b5b5c3-0064c45fc8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.016 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:39:37.017 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:39:37.017 DEBUG:boto:path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:39:37.017 DEBUG:boto:auth_path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:39:37.018 DEBUG:boto:Method: HEAD 2023-07-29T00:39:37.018 DEBUG:boto:Path: /hiiqos-59/obj34?versionId=null 2023-07-29T00:39:37.018 DEBUG:boto:Data: 2023-07-29T00:39:37.019 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.019 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:37.019 DEBUG:boto:Port: 8001 2023-07-29T00:39:37.020 DEBUG:boto:Params: {} 2023-07-29T00:39:37.020 DEBUG:boto:Token: None 2023-07-29T00:39:37.020 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj34?versionId=null 2023-07-29T00:39:37.021 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CrLMbHAQaqccPmpSN0WBHlWG/pM= 2023-07-29T00:39:37.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CrLMbHAQaqccPmpSN0WBHlWG/pM=', 'Content-Length': '0'} 2023-07-29T00:39:37.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000630184af65579561-0064c45fc8-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.023 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:39:37.024 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:39:37.024 DEBUG:boto:path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:39:37.024 DEBUG:boto:auth_path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:39:37.025 DEBUG:boto:Method: HEAD 2023-07-29T00:39:37.025 DEBUG:boto:Path: /hiiqos-59/obj34?versionId=null 2023-07-29T00:39:37.025 DEBUG:boto:Data: 2023-07-29T00:39:37.026 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.026 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:37.026 DEBUG:boto:Port: 8000 2023-07-29T00:39:37.027 DEBUG:boto:Params: {} 2023-07-29T00:39:37.027 DEBUG:boto:Token: None 2023-07-29T00:39:37.027 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj34?versionId=null 2023-07-29T00:39:37.028 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CrLMbHAQaqccPmpSN0WBHlWG/pM= 2023-07-29T00:39:37.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CrLMbHAQaqccPmpSN0WBHlWG/pM=', 'Content-Length': '0'} 2023-07-29T00:39:37.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000023df0ddd8293c79-0064c45fc8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-07-29T00:39:37.031 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:39:37.031 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:39:37.032 DEBUG:boto:path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:39:37.032 DEBUG:boto:auth_path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:39:37.032 DEBUG:boto:Method: GET 2023-07-29T00:39:37.033 DEBUG:boto:Path: /hiiqos-59/obj34?versionId=null 2023-07-29T00:39:37.033 DEBUG:boto:Data: 2023-07-29T00:39:37.033 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.034 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:37.034 DEBUG:boto:Port: 8001 2023-07-29T00:39:37.034 DEBUG:boto:Params: {} 2023-07-29T00:39:37.035 DEBUG:boto:Token: None 2023-07-29T00:39:37.035 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj34?versionId=null 2023-07-29T00:39:37.036 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:d2UCmJGr7VlXjcsRyqFLh24dXUM= 2023-07-29T00:39:37.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:d2UCmJGr7VlXjcsRyqFLh24dXUM=', 'Content-Length': '0'} 2023-07-29T00:39:37.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092e0bdecf6298fb5-0064c45fc8-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.039 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:39:37.039 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:39:37.039 DEBUG:boto:path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:39:37.040 DEBUG:boto:auth_path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:39:37.040 DEBUG:boto:Method: GET 2023-07-29T00:39:37.040 DEBUG:boto:Path: /hiiqos-59/obj34?versionId=null 2023-07-29T00:39:37.041 DEBUG:boto:Data: 2023-07-29T00:39:37.041 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.042 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:37.042 DEBUG:boto:Port: 8000 2023-07-29T00:39:37.042 DEBUG:boto:Params: {} 2023-07-29T00:39:37.043 DEBUG:boto:Token: None 2023-07-29T00:39:37.043 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj34?versionId=null 2023-07-29T00:39:37.043 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:d2UCmJGr7VlXjcsRyqFLh24dXUM= 2023-07-29T00:39:37.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:d2UCmJGr7VlXjcsRyqFLh24dXUM=', 'Content-Length': '0'} 2023-07-29T00:39:37.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000691170dc4bcce79b-0064c45fc8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.046 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:37.047 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:37.047 DEBUG:boto:path=/hiiqos-59/?prefix=obj34 2023-07-29T00:39:37.047 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj34 2023-07-29T00:39:37.048 DEBUG:boto:Method: GET 2023-07-29T00:39:37.048 DEBUG:boto:Path: /hiiqos-59/?prefix=obj34 2023-07-29T00:39:37.048 DEBUG:boto:Data: 2023-07-29T00:39:37.049 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.049 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:37.049 DEBUG:boto:Port: 8001 2023-07-29T00:39:37.050 DEBUG:boto:Params: {} 2023-07-29T00:39:37.050 DEBUG:boto:Token: None 2023-07-29T00:39:37.051 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/ 2023-07-29T00:39:37.051 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Mle3aYo3xzdeHcrINCie96a4FQc= 2023-07-29T00:39:37.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Mle3aYo3xzdeHcrINCie96a4FQc=', 'Content-Length': '0'} 2023-07-29T00:39:37.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000256d7ac5d7793746-0064c45fc8-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.054 DEBUG:boto:b'hiiqos-59obj341000falseobj342023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:37.055 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:37.055 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:37.056 DEBUG:boto:path=/hiiqos-59/?prefix=obj34 2023-07-29T00:39:37.056 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj34 2023-07-29T00:39:37.056 DEBUG:boto:Method: GET 2023-07-29T00:39:37.057 DEBUG:boto:Path: /hiiqos-59/?prefix=obj34 2023-07-29T00:39:37.057 DEBUG:boto:Data: 2023-07-29T00:39:37.057 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.058 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:37.058 DEBUG:boto:Port: 8000 2023-07-29T00:39:37.058 DEBUG:boto:Params: {} 2023-07-29T00:39:37.059 DEBUG:boto:Token: None 2023-07-29T00:39:37.059 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/ 2023-07-29T00:39:37.060 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Mle3aYo3xzdeHcrINCie96a4FQc= 2023-07-29T00:39:37.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Mle3aYo3xzdeHcrINCie96a4FQc=', 'Content-Length': '0'} 2023-07-29T00:39:37.064 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a520e91292ca8d6a-0064c45fc8-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.065 DEBUG:boto:b'hiiqos-59obj341000falseobj342023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:37.065 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:39:37.066 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:39:37.066 DEBUG:boto:Method: HEAD 2023-07-29T00:39:37.066 DEBUG:boto:Path: /hiiqos-59/obj34 2023-07-29T00:39:37.067 DEBUG:boto:Data: 2023-07-29T00:39:37.067 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.067 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:37.068 DEBUG:boto:Port: 8001 2023-07-29T00:39:37.068 DEBUG:boto:Params: {} 2023-07-29T00:39:37.069 DEBUG:boto:Token: None 2023-07-29T00:39:37.069 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj34 2023-07-29T00:39:37.069 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mbBXW8JuqPpzogCxKD3CRB3C8/Y= 2023-07-29T00:39:37.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mbBXW8JuqPpzogCxKD3CRB3C8/Y=', 'Content-Length': '0'} 2023-07-29T00:39:37.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b336f938ef70846-0064c45fc9-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.072 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:39:37.073 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:39:37.073 DEBUG:boto:Method: HEAD 2023-07-29T00:39:37.073 DEBUG:boto:Path: /hiiqos-59/obj34 2023-07-29T00:39:37.074 DEBUG:boto:Data: 2023-07-29T00:39:37.074 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.074 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:37.075 DEBUG:boto:Port: 8000 2023-07-29T00:39:37.075 DEBUG:boto:Params: {} 2023-07-29T00:39:37.075 DEBUG:boto:Token: None 2023-07-29T00:39:37.076 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj34 2023-07-29T00:39:37.076 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mbBXW8JuqPpzogCxKD3CRB3C8/Y= 2023-07-29T00:39:37.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mbBXW8JuqPpzogCxKD3CRB3C8/Y=', 'Content-Length': '0'} 2023-07-29T00:39:37.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002365e76b60e14d88-0064c45fc9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-07-29T00:39:37.079 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:39:37.079 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:39:37.080 DEBUG:boto:Method: GET 2023-07-29T00:39:37.080 DEBUG:boto:Path: /hiiqos-59/obj34 2023-07-29T00:39:37.081 DEBUG:boto:Data: 2023-07-29T00:39:37.081 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.081 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:37.082 DEBUG:boto:Port: 8001 2023-07-29T00:39:37.082 DEBUG:boto:Params: {} 2023-07-29T00:39:37.082 DEBUG:boto:Token: None 2023-07-29T00:39:37.083 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj34 2023-07-29T00:39:37.083 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:paB+4IBtgVWnnE3Kn2fy4VxGXco= 2023-07-29T00:39:37.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:paB+4IBtgVWnnE3Kn2fy4VxGXco=', 'Content-Length': '0'} 2023-07-29T00:39:37.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce0f93a889134c0f-0064c45fc9-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.086 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:39:37.086 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:39:37.087 DEBUG:boto:Method: GET 2023-07-29T00:39:37.087 DEBUG:boto:Path: /hiiqos-59/obj34 2023-07-29T00:39:37.088 DEBUG:boto:Data: 2023-07-29T00:39:37.088 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.088 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:37.089 DEBUG:boto:Port: 8000 2023-07-29T00:39:37.089 DEBUG:boto:Params: {} 2023-07-29T00:39:37.089 DEBUG:boto:Token: None 2023-07-29T00:39:37.090 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj34 2023-07-29T00:39:37.090 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:paB+4IBtgVWnnE3Kn2fy4VxGXco= 2023-07-29T00:39:37.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:paB+4IBtgVWnnE3Kn2fy4VxGXco=', 'Content-Length': '0'} 2023-07-29T00:39:37.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddd9052eae6600a2-0064c45fc9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.136 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:37.137 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:37.137 DEBUG:boto:path=/hiiqos-59/?prefix=obj34 2023-07-29T00:39:37.138 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj34 2023-07-29T00:39:37.138 DEBUG:boto:Method: GET 2023-07-29T00:39:37.138 DEBUG:boto:Path: /hiiqos-59/?prefix=obj34 2023-07-29T00:39:37.139 DEBUG:boto:Data: 2023-07-29T00:39:37.139 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.140 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:37.140 DEBUG:boto:Port: 8001 2023-07-29T00:39:37.140 DEBUG:boto:Params: {} 2023-07-29T00:39:37.141 DEBUG:boto:Token: None 2023-07-29T00:39:37.141 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/ 2023-07-29T00:39:37.142 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Mle3aYo3xzdeHcrINCie96a4FQc= 2023-07-29T00:39:37.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Mle3aYo3xzdeHcrINCie96a4FQc=', 'Content-Length': '0'} 2023-07-29T00:39:37.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047cf13a4b8ea2b4d-0064c45fc9-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.146 DEBUG:boto:b'hiiqos-59obj341000falseobj342023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:37.147 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:37.148 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:37.148 DEBUG:boto:path=/hiiqos-59/?prefix=obj34 2023-07-29T00:39:37.149 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj34 2023-07-29T00:39:37.150 DEBUG:boto:Method: GET 2023-07-29T00:39:37.151 DEBUG:boto:Path: /hiiqos-59/?prefix=obj34 2023-07-29T00:39:37.152 DEBUG:boto:Data: 2023-07-29T00:39:37.152 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.152 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:37.153 DEBUG:boto:Port: 8000 2023-07-29T00:39:37.153 DEBUG:boto:Params: {} 2023-07-29T00:39:37.154 DEBUG:boto:Token: None 2023-07-29T00:39:37.154 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/ 2023-07-29T00:39:37.155 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Mle3aYo3xzdeHcrINCie96a4FQc= 2023-07-29T00:39:37.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Mle3aYo3xzdeHcrINCie96a4FQc=', 'Content-Length': '0'} 2023-07-29T00:39:37.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dffcaf080c9f6833-0064c45fc9-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.160 DEBUG:boto:b'hiiqos-59obj341000falseobj342023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:37.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-07-29T00:39:37.162 DEBUG:boto:path=/hiiqos-59/obj37 2023-07-29T00:39:37.163 DEBUG:boto:auth_path=/hiiqos-59/obj37 2023-07-29T00:39:37.164 DEBUG:boto:path=/hiiqos-59/obj37?versionId=null 2023-07-29T00:39:37.165 DEBUG:boto:auth_path=/hiiqos-59/obj37?versionId=null 2023-07-29T00:39:37.165 DEBUG:boto:Method: GET 2023-07-29T00:39:37.166 DEBUG:boto:Path: /hiiqos-59/obj37?versionId=null 2023-07-29T00:39:37.167 DEBUG:boto:Data: 2023-07-29T00:39:37.168 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.169 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:37.170 DEBUG:boto:Port: 8001 2023-07-29T00:39:37.170 DEBUG:boto:Params: {} 2023-07-29T00:39:37.171 DEBUG:boto:Token: None 2023-07-29T00:39:37.171 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj37?versionId=null 2023-07-29T00:39:37.172 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BRt9Vq9gljeQL83kY4xq1ctSeLM= 2023-07-29T00:39:37.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BRt9Vq9gljeQL83kY4xq1ctSeLM=', 'Content-Length': '0'} 2023-07-29T00:39:37.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcf6ae97daa90ef4-0064c45fc9-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.216 DEBUG:boto:path=/hiiqos-59/obj37 2023-07-29T00:39:37.217 DEBUG:boto:auth_path=/hiiqos-59/obj37 2023-07-29T00:39:37.217 DEBUG:boto:path=/hiiqos-59/obj37?versionId=null 2023-07-29T00:39:37.218 DEBUG:boto:auth_path=/hiiqos-59/obj37?versionId=null 2023-07-29T00:39:37.218 DEBUG:boto:Method: GET 2023-07-29T00:39:37.218 DEBUG:boto:Path: /hiiqos-59/obj37?versionId=null 2023-07-29T00:39:37.219 DEBUG:boto:Data: 2023-07-29T00:39:37.219 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.220 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:37.220 DEBUG:boto:Port: 8000 2023-07-29T00:39:37.220 DEBUG:boto:Params: {} 2023-07-29T00:39:37.221 DEBUG:boto:Token: None 2023-07-29T00:39:37.221 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj37?versionId=null 2023-07-29T00:39:37.222 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BRt9Vq9gljeQL83kY4xq1ctSeLM= 2023-07-29T00:39:37.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BRt9Vq9gljeQL83kY4xq1ctSeLM=', 'Content-Length': '0'} 2023-07-29T00:39:37.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5cc9d0f8187c0c5-0064c45fc9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.225 DEBUG:boto:path=/hiiqos-59/obj37 2023-07-29T00:39:37.226 DEBUG:boto:auth_path=/hiiqos-59/obj37 2023-07-29T00:39:37.226 DEBUG:boto:path=/hiiqos-59/obj37?versionId=null 2023-07-29T00:39:37.227 DEBUG:boto:auth_path=/hiiqos-59/obj37?versionId=null 2023-07-29T00:39:37.227 DEBUG:boto:Method: HEAD 2023-07-29T00:39:37.228 DEBUG:boto:Path: /hiiqos-59/obj37?versionId=null 2023-07-29T00:39:37.228 DEBUG:boto:Data: 2023-07-29T00:39:37.228 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.229 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:37.229 DEBUG:boto:Port: 8001 2023-07-29T00:39:37.229 DEBUG:boto:Params: {} 2023-07-29T00:39:37.230 DEBUG:boto:Token: None 2023-07-29T00:39:37.230 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj37?versionId=null 2023-07-29T00:39:37.231 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RmwSY4rMk6hurqYzpONeNV5KGbA= 2023-07-29T00:39:37.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RmwSY4rMk6hurqYzpONeNV5KGbA=', 'Content-Length': '0'} 2023-07-29T00:39:37.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c31beb662f8a961a-0064c45fc9-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.233 DEBUG:boto:path=/hiiqos-59/obj37 2023-07-29T00:39:37.234 DEBUG:boto:auth_path=/hiiqos-59/obj37 2023-07-29T00:39:37.234 DEBUG:boto:path=/hiiqos-59/obj37?versionId=null 2023-07-29T00:39:37.235 DEBUG:boto:auth_path=/hiiqos-59/obj37?versionId=null 2023-07-29T00:39:37.235 DEBUG:boto:Method: HEAD 2023-07-29T00:39:37.235 DEBUG:boto:Path: /hiiqos-59/obj37?versionId=null 2023-07-29T00:39:37.236 DEBUG:boto:Data: 2023-07-29T00:39:37.236 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.236 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:37.237 DEBUG:boto:Port: 8000 2023-07-29T00:39:37.237 DEBUG:boto:Params: {} 2023-07-29T00:39:37.238 DEBUG:boto:Token: None 2023-07-29T00:39:37.238 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj37?versionId=null 2023-07-29T00:39:37.239 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RmwSY4rMk6hurqYzpONeNV5KGbA= 2023-07-29T00:39:37.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RmwSY4rMk6hurqYzpONeNV5KGbA=', 'Content-Length': '0'} 2023-07-29T00:39:37.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8c51c7fa6933c40-0064c45fc9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-07-29T00:39:37.242 DEBUG:boto:path=/hiiqos-59/obj37 2023-07-29T00:39:37.242 DEBUG:boto:auth_path=/hiiqos-59/obj37 2023-07-29T00:39:37.243 DEBUG:boto:path=/hiiqos-59/obj37?versionId=null 2023-07-29T00:39:37.243 DEBUG:boto:auth_path=/hiiqos-59/obj37?versionId=null 2023-07-29T00:39:37.243 DEBUG:boto:Method: GET 2023-07-29T00:39:37.244 DEBUG:boto:Path: /hiiqos-59/obj37?versionId=null 2023-07-29T00:39:37.245 DEBUG:boto:Data: 2023-07-29T00:39:37.245 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.246 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:37.247 DEBUG:boto:Port: 8001 2023-07-29T00:39:37.247 DEBUG:boto:Params: {} 2023-07-29T00:39:37.248 DEBUG:boto:Token: None 2023-07-29T00:39:37.248 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj37?versionId=null 2023-07-29T00:39:37.249 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BRt9Vq9gljeQL83kY4xq1ctSeLM= 2023-07-29T00:39:37.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BRt9Vq9gljeQL83kY4xq1ctSeLM=', 'Content-Length': '0'} 2023-07-29T00:39:37.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8b38d51c53ba0f1-0064c45fc9-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.252 DEBUG:boto:path=/hiiqos-59/obj37 2023-07-29T00:39:37.252 DEBUG:boto:auth_path=/hiiqos-59/obj37 2023-07-29T00:39:37.253 DEBUG:boto:path=/hiiqos-59/obj37?versionId=null 2023-07-29T00:39:37.253 DEBUG:boto:auth_path=/hiiqos-59/obj37?versionId=null 2023-07-29T00:39:37.253 DEBUG:boto:Method: GET 2023-07-29T00:39:37.254 DEBUG:boto:Path: /hiiqos-59/obj37?versionId=null 2023-07-29T00:39:37.254 DEBUG:boto:Data: 2023-07-29T00:39:37.255 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.255 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:37.255 DEBUG:boto:Port: 8000 2023-07-29T00:39:37.256 DEBUG:boto:Params: {} 2023-07-29T00:39:37.256 DEBUG:boto:Token: None 2023-07-29T00:39:37.257 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj37?versionId=null 2023-07-29T00:39:37.257 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BRt9Vq9gljeQL83kY4xq1ctSeLM= 2023-07-29T00:39:37.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BRt9Vq9gljeQL83kY4xq1ctSeLM=', 'Content-Length': '0'} 2023-07-29T00:39:37.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000394918e276ddd707-0064c45fc9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.304 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:37.305 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:37.305 DEBUG:boto:path=/hiiqos-59/?prefix=obj37 2023-07-29T00:39:37.306 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj37 2023-07-29T00:39:37.306 DEBUG:boto:Method: GET 2023-07-29T00:39:37.306 DEBUG:boto:Path: /hiiqos-59/?prefix=obj37 2023-07-29T00:39:37.307 DEBUG:boto:Data: 2023-07-29T00:39:37.307 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.307 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:37.308 DEBUG:boto:Port: 8001 2023-07-29T00:39:37.308 DEBUG:boto:Params: {} 2023-07-29T00:39:37.309 DEBUG:boto:Token: None 2023-07-29T00:39:37.309 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/ 2023-07-29T00:39:37.310 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Mle3aYo3xzdeHcrINCie96a4FQc= 2023-07-29T00:39:37.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Mle3aYo3xzdeHcrINCie96a4FQc=', 'Content-Length': '0'} 2023-07-29T00:39:37.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005daa3474ac0c6f7e-0064c45fc9-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.313 DEBUG:boto:b'hiiqos-59obj371000falseobj372023-07-29T00:37:53.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:37.314 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:37.314 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:37.315 DEBUG:boto:path=/hiiqos-59/?prefix=obj37 2023-07-29T00:39:37.315 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj37 2023-07-29T00:39:37.316 DEBUG:boto:Method: GET 2023-07-29T00:39:37.316 DEBUG:boto:Path: /hiiqos-59/?prefix=obj37 2023-07-29T00:39:37.317 DEBUG:boto:Data: 2023-07-29T00:39:37.317 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.317 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:37.318 DEBUG:boto:Port: 8000 2023-07-29T00:39:37.318 DEBUG:boto:Params: {} 2023-07-29T00:39:37.318 DEBUG:boto:Token: None 2023-07-29T00:39:37.319 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/ 2023-07-29T00:39:37.319 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Mle3aYo3xzdeHcrINCie96a4FQc= 2023-07-29T00:39:37.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Mle3aYo3xzdeHcrINCie96a4FQc=', 'Content-Length': '0'} 2023-07-29T00:39:37.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a652bb0869a221c0-0064c45fc9-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.325 DEBUG:boto:b'hiiqos-59obj371000falseobj372023-07-29T00:37:53.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:37.326 DEBUG:boto:path=/hiiqos-59/obj37 2023-07-29T00:39:37.326 DEBUG:boto:auth_path=/hiiqos-59/obj37 2023-07-29T00:39:37.327 DEBUG:boto:Method: HEAD 2023-07-29T00:39:37.327 DEBUG:boto:Path: /hiiqos-59/obj37 2023-07-29T00:39:37.327 DEBUG:boto:Data: 2023-07-29T00:39:37.328 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.328 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:37.328 DEBUG:boto:Port: 8001 2023-07-29T00:39:37.329 DEBUG:boto:Params: {} 2023-07-29T00:39:37.329 DEBUG:boto:Token: None 2023-07-29T00:39:37.330 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj37 2023-07-29T00:39:37.330 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+nVA1Yo0BM9jjrfDRyoHaM1jim8= 2023-07-29T00:39:37.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+nVA1Yo0BM9jjrfDRyoHaM1jim8=', 'Content-Length': '0'} 2023-07-29T00:39:37.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000615a84bbf617fccc-0064c45fc9-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.334 DEBUG:boto:path=/hiiqos-59/obj37 2023-07-29T00:39:37.334 DEBUG:boto:auth_path=/hiiqos-59/obj37 2023-07-29T00:39:37.335 DEBUG:boto:Method: HEAD 2023-07-29T00:39:37.335 DEBUG:boto:Path: /hiiqos-59/obj37 2023-07-29T00:39:37.335 DEBUG:boto:Data: 2023-07-29T00:39:37.336 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.336 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:37.336 DEBUG:boto:Port: 8000 2023-07-29T00:39:37.337 DEBUG:boto:Params: {} 2023-07-29T00:39:37.337 DEBUG:boto:Token: None 2023-07-29T00:39:37.338 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj37 2023-07-29T00:39:37.338 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+nVA1Yo0BM9jjrfDRyoHaM1jim8= 2023-07-29T00:39:37.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+nVA1Yo0BM9jjrfDRyoHaM1jim8=', 'Content-Length': '0'} 2023-07-29T00:39:37.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045e024e565ad2776-0064c45fc9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-07-29T00:39:37.343 DEBUG:boto:path=/hiiqos-59/obj37 2023-07-29T00:39:37.343 DEBUG:boto:auth_path=/hiiqos-59/obj37 2023-07-29T00:39:37.344 DEBUG:boto:Method: GET 2023-07-29T00:39:37.344 DEBUG:boto:Path: /hiiqos-59/obj37 2023-07-29T00:39:37.344 DEBUG:boto:Data: 2023-07-29T00:39:37.345 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.345 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:37.346 DEBUG:boto:Port: 8001 2023-07-29T00:39:37.346 DEBUG:boto:Params: {} 2023-07-29T00:39:37.346 DEBUG:boto:Token: None 2023-07-29T00:39:37.347 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj37 2023-07-29T00:39:37.347 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hd/sKAuFS80DVU0IN84NuFoeb70= 2023-07-29T00:39:37.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hd/sKAuFS80DVU0IN84NuFoeb70=', 'Content-Length': '0'} 2023-07-29T00:39:37.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008d212d38c5bf655-0064c45fc9-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.392 DEBUG:boto:path=/hiiqos-59/obj37 2023-07-29T00:39:37.393 DEBUG:boto:auth_path=/hiiqos-59/obj37 2023-07-29T00:39:37.393 DEBUG:boto:Method: GET 2023-07-29T00:39:37.394 DEBUG:boto:Path: /hiiqos-59/obj37 2023-07-29T00:39:37.394 DEBUG:boto:Data: 2023-07-29T00:39:37.395 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.396 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:37.397 DEBUG:boto:Port: 8000 2023-07-29T00:39:37.397 DEBUG:boto:Params: {} 2023-07-29T00:39:37.398 DEBUG:boto:Token: None 2023-07-29T00:39:37.398 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj37 2023-07-29T00:39:37.399 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hd/sKAuFS80DVU0IN84NuFoeb70= 2023-07-29T00:39:37.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hd/sKAuFS80DVU0IN84NuFoeb70=', 'Content-Length': '0'} 2023-07-29T00:39:37.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa82651c2372a2d5-0064c45fc9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.402 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:37.402 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:37.403 DEBUG:boto:path=/hiiqos-59/?prefix=obj37 2023-07-29T00:39:37.403 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj37 2023-07-29T00:39:37.404 DEBUG:boto:Method: GET 2023-07-29T00:39:37.404 DEBUG:boto:Path: /hiiqos-59/?prefix=obj37 2023-07-29T00:39:37.404 DEBUG:boto:Data: 2023-07-29T00:39:37.405 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.405 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:37.405 DEBUG:boto:Port: 8001 2023-07-29T00:39:37.406 DEBUG:boto:Params: {} 2023-07-29T00:39:37.406 DEBUG:boto:Token: None 2023-07-29T00:39:37.407 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/ 2023-07-29T00:39:37.407 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Mle3aYo3xzdeHcrINCie96a4FQc= 2023-07-29T00:39:37.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Mle3aYo3xzdeHcrINCie96a4FQc=', 'Content-Length': '0'} 2023-07-29T00:39:37.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000676fc0e316039cf5-0064c45fc9-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.411 DEBUG:boto:b'hiiqos-59obj371000falseobj372023-07-29T00:37:53.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:37.412 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:37.412 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:37.412 DEBUG:boto:path=/hiiqos-59/?prefix=obj37 2023-07-29T00:39:37.413 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj37 2023-07-29T00:39:37.413 DEBUG:boto:Method: GET 2023-07-29T00:39:37.413 DEBUG:boto:Path: /hiiqos-59/?prefix=obj37 2023-07-29T00:39:37.414 DEBUG:boto:Data: 2023-07-29T00:39:37.414 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.415 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:37.415 DEBUG:boto:Port: 8000 2023-07-29T00:39:37.415 DEBUG:boto:Params: {} 2023-07-29T00:39:37.416 DEBUG:boto:Token: None 2023-07-29T00:39:37.416 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/ 2023-07-29T00:39:37.417 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Mle3aYo3xzdeHcrINCie96a4FQc= 2023-07-29T00:39:37.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Mle3aYo3xzdeHcrINCie96a4FQc=', 'Content-Length': '0'} 2023-07-29T00:39:37.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007445ef55e5b1f05e-0064c45fc9-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.464 DEBUG:boto:b'hiiqos-59obj371000falseobj372023-07-29T00:37:53.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:37.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-07-29T00:39:37.466 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:39:37.466 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:39:37.467 DEBUG:boto:path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:39:37.467 DEBUG:boto:auth_path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:39:37.468 DEBUG:boto:Method: GET 2023-07-29T00:39:37.468 DEBUG:boto:Path: /hiiqos-59/obj38?versionId=null 2023-07-29T00:39:37.468 DEBUG:boto:Data: 2023-07-29T00:39:37.469 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.469 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:37.469 DEBUG:boto:Port: 8001 2023-07-29T00:39:37.470 DEBUG:boto:Params: {} 2023-07-29T00:39:37.470 DEBUG:boto:Token: None 2023-07-29T00:39:37.471 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj38?versionId=null 2023-07-29T00:39:37.471 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BqrRuMDYt4JUfl1T12bAD5ww6uc= 2023-07-29T00:39:37.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BqrRuMDYt4JUfl1T12bAD5ww6uc=', 'Content-Length': '0'} 2023-07-29T00:39:37.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000176bd08270dd66e5-0064c45fc9-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.475 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:39:37.475 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:39:37.475 DEBUG:boto:path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:39:37.476 DEBUG:boto:auth_path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:39:37.476 DEBUG:boto:Method: GET 2023-07-29T00:39:37.477 DEBUG:boto:Path: /hiiqos-59/obj38?versionId=null 2023-07-29T00:39:37.477 DEBUG:boto:Data: 2023-07-29T00:39:37.479 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.480 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:37.480 DEBUG:boto:Port: 8000 2023-07-29T00:39:37.481 DEBUG:boto:Params: {} 2023-07-29T00:39:37.482 DEBUG:boto:Token: None 2023-07-29T00:39:37.483 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj38?versionId=null 2023-07-29T00:39:37.484 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BqrRuMDYt4JUfl1T12bAD5ww6uc= 2023-07-29T00:39:37.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BqrRuMDYt4JUfl1T12bAD5ww6uc=', 'Content-Length': '0'} 2023-07-29T00:39:37.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b1b0dce8de2f63b-0064c45fc9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.488 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:39:37.488 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:39:37.489 DEBUG:boto:path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:39:37.489 DEBUG:boto:auth_path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:39:37.489 DEBUG:boto:Method: HEAD 2023-07-29T00:39:37.490 DEBUG:boto:Path: /hiiqos-59/obj38?versionId=null 2023-07-29T00:39:37.490 DEBUG:boto:Data: 2023-07-29T00:39:37.491 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.491 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:37.492 DEBUG:boto:Port: 8001 2023-07-29T00:39:37.493 DEBUG:boto:Params: {} 2023-07-29T00:39:37.493 DEBUG:boto:Token: None 2023-07-29T00:39:37.494 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj38?versionId=null 2023-07-29T00:39:37.495 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CutcNOsCp1NSVY2Gj3Q7FS/GvNk= 2023-07-29T00:39:37.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CutcNOsCp1NSVY2Gj3Q7FS/GvNk=', 'Content-Length': '0'} 2023-07-29T00:39:37.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bb8c87df202bc18-0064c45fc9-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.500 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:39:37.501 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:39:37.501 DEBUG:boto:path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:39:37.501 DEBUG:boto:auth_path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:39:37.502 DEBUG:boto:Method: HEAD 2023-07-29T00:39:37.502 DEBUG:boto:Path: /hiiqos-59/obj38?versionId=null 2023-07-29T00:39:37.502 DEBUG:boto:Data: 2023-07-29T00:39:37.503 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.503 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:37.504 DEBUG:boto:Port: 8000 2023-07-29T00:39:37.504 DEBUG:boto:Params: {} 2023-07-29T00:39:37.505 DEBUG:boto:Token: None 2023-07-29T00:39:37.505 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj38?versionId=null 2023-07-29T00:39:37.506 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CutcNOsCp1NSVY2Gj3Q7FS/GvNk= 2023-07-29T00:39:37.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CutcNOsCp1NSVY2Gj3Q7FS/GvNk=', 'Content-Length': '0'} 2023-07-29T00:39:37.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f741d092a6c55340-0064c45fc9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-07-29T00:39:37.510 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:39:37.510 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:39:37.511 DEBUG:boto:path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:39:37.511 DEBUG:boto:auth_path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:39:37.511 DEBUG:boto:Method: GET 2023-07-29T00:39:37.512 DEBUG:boto:Path: /hiiqos-59/obj38?versionId=null 2023-07-29T00:39:37.512 DEBUG:boto:Data: 2023-07-29T00:39:37.512 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.513 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:37.513 DEBUG:boto:Port: 8001 2023-07-29T00:39:37.513 DEBUG:boto:Params: {} 2023-07-29T00:39:37.514 DEBUG:boto:Token: None 2023-07-29T00:39:37.514 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj38?versionId=null 2023-07-29T00:39:37.515 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BqrRuMDYt4JUfl1T12bAD5ww6uc= 2023-07-29T00:39:37.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BqrRuMDYt4JUfl1T12bAD5ww6uc=', 'Content-Length': '0'} 2023-07-29T00:39:37.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000177931784d59500f-0064c45fc9-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.518 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:39:37.518 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:39:37.518 DEBUG:boto:path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:39:37.519 DEBUG:boto:auth_path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:39:37.519 DEBUG:boto:Method: GET 2023-07-29T00:39:37.519 DEBUG:boto:Path: /hiiqos-59/obj38?versionId=null 2023-07-29T00:39:37.520 DEBUG:boto:Data: 2023-07-29T00:39:37.520 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.521 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:37.521 DEBUG:boto:Port: 8000 2023-07-29T00:39:37.521 DEBUG:boto:Params: {} 2023-07-29T00:39:37.522 DEBUG:boto:Token: None 2023-07-29T00:39:37.522 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj38?versionId=null 2023-07-29T00:39:37.523 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BqrRuMDYt4JUfl1T12bAD5ww6uc= 2023-07-29T00:39:37.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BqrRuMDYt4JUfl1T12bAD5ww6uc=', 'Content-Length': '0'} 2023-07-29T00:39:37.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ebac82fb66f3184-0064c45fc9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.526 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:37.526 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:37.526 DEBUG:boto:path=/hiiqos-59/?prefix=obj38 2023-07-29T00:39:37.527 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj38 2023-07-29T00:39:37.527 DEBUG:boto:Method: GET 2023-07-29T00:39:37.528 DEBUG:boto:Path: /hiiqos-59/?prefix=obj38 2023-07-29T00:39:37.528 DEBUG:boto:Data: 2023-07-29T00:39:37.528 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.529 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:37.529 DEBUG:boto:Port: 8001 2023-07-29T00:39:37.529 DEBUG:boto:Params: {} 2023-07-29T00:39:37.530 DEBUG:boto:Token: None 2023-07-29T00:39:37.530 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/ 2023-07-29T00:39:37.531 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Mle3aYo3xzdeHcrINCie96a4FQc= 2023-07-29T00:39:37.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Mle3aYo3xzdeHcrINCie96a4FQc=', 'Content-Length': '0'} 2023-07-29T00:39:37.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6132976afc54d79-0064c45fc9-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.534 DEBUG:boto:b'hiiqos-59obj381000falseobj382023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:37.535 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:37.535 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:37.536 DEBUG:boto:path=/hiiqos-59/?prefix=obj38 2023-07-29T00:39:37.536 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj38 2023-07-29T00:39:37.536 DEBUG:boto:Method: GET 2023-07-29T00:39:37.537 DEBUG:boto:Path: /hiiqos-59/?prefix=obj38 2023-07-29T00:39:37.537 DEBUG:boto:Data: 2023-07-29T00:39:37.537 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.538 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:37.538 DEBUG:boto:Port: 8000 2023-07-29T00:39:37.538 DEBUG:boto:Params: {} 2023-07-29T00:39:37.539 DEBUG:boto:Token: None 2023-07-29T00:39:37.539 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/ 2023-07-29T00:39:37.540 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Mle3aYo3xzdeHcrINCie96a4FQc= 2023-07-29T00:39:37.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Mle3aYo3xzdeHcrINCie96a4FQc=', 'Content-Length': '0'} 2023-07-29T00:39:37.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fcfa531b39021cc-0064c45fc9-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.592 DEBUG:boto:b'hiiqos-59obj381000falseobj382023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:37.593 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:39:37.594 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:39:37.594 DEBUG:boto:Method: HEAD 2023-07-29T00:39:37.594 DEBUG:boto:Path: /hiiqos-59/obj38 2023-07-29T00:39:37.595 DEBUG:boto:Data: 2023-07-29T00:39:37.595 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.595 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:37.596 DEBUG:boto:Port: 8001 2023-07-29T00:39:37.596 DEBUG:boto:Params: {} 2023-07-29T00:39:37.596 DEBUG:boto:Token: None 2023-07-29T00:39:37.597 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj38 2023-07-29T00:39:37.597 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:d84bMj/phixemMnjMPcNkONT7YQ= 2023-07-29T00:39:37.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:d84bMj/phixemMnjMPcNkONT7YQ=', 'Content-Length': '0'} 2023-07-29T00:39:37.600 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b8820361dd79e55-0064c45fc9-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.601 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:39:37.601 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:39:37.602 DEBUG:boto:Method: HEAD 2023-07-29T00:39:37.602 DEBUG:boto:Path: /hiiqos-59/obj38 2023-07-29T00:39:37.602 DEBUG:boto:Data: 2023-07-29T00:39:37.603 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.603 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:37.603 DEBUG:boto:Port: 8000 2023-07-29T00:39:37.604 DEBUG:boto:Params: {} 2023-07-29T00:39:37.604 DEBUG:boto:Token: None 2023-07-29T00:39:37.605 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj38 2023-07-29T00:39:37.605 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:d84bMj/phixemMnjMPcNkONT7YQ= 2023-07-29T00:39:37.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:d84bMj/phixemMnjMPcNkONT7YQ=', 'Content-Length': '0'} 2023-07-29T00:39:37.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b931cd2b04651a94-0064c45fc9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-07-29T00:39:37.609 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:39:37.609 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:39:37.610 DEBUG:boto:Method: GET 2023-07-29T00:39:37.610 DEBUG:boto:Path: /hiiqos-59/obj38 2023-07-29T00:39:37.611 DEBUG:boto:Data: 2023-07-29T00:39:37.611 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.612 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:37.612 DEBUG:boto:Port: 8001 2023-07-29T00:39:37.613 DEBUG:boto:Params: {} 2023-07-29T00:39:37.614 DEBUG:boto:Token: None 2023-07-29T00:39:37.614 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj38 2023-07-29T00:39:37.615 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lSIoIfFtXUC+DPG3IHoxRXXJ5Ao= 2023-07-29T00:39:37.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lSIoIfFtXUC+DPG3IHoxRXXJ5Ao=', 'Content-Length': '0'} 2023-07-29T00:39:37.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d7bec63cfcad6fb-0064c45fc9-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.618 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:39:37.619 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:39:37.619 DEBUG:boto:Method: GET 2023-07-29T00:39:37.620 DEBUG:boto:Path: /hiiqos-59/obj38 2023-07-29T00:39:37.620 DEBUG:boto:Data: 2023-07-29T00:39:37.620 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.621 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:37.621 DEBUG:boto:Port: 8000 2023-07-29T00:39:37.622 DEBUG:boto:Params: {} 2023-07-29T00:39:37.622 DEBUG:boto:Token: None 2023-07-29T00:39:37.622 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj38 2023-07-29T00:39:37.623 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lSIoIfFtXUC+DPG3IHoxRXXJ5Ao= 2023-07-29T00:39:37.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lSIoIfFtXUC+DPG3IHoxRXXJ5Ao=', 'Content-Length': '0'} 2023-07-29T00:39:37.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008086598c66aad5d7-0064c45fc9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.626 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:37.627 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:37.627 DEBUG:boto:path=/hiiqos-59/?prefix=obj38 2023-07-29T00:39:37.628 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj38 2023-07-29T00:39:37.629 DEBUG:boto:Method: GET 2023-07-29T00:39:37.629 DEBUG:boto:Path: /hiiqos-59/?prefix=obj38 2023-07-29T00:39:37.630 DEBUG:boto:Data: 2023-07-29T00:39:37.630 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.631 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:37.632 DEBUG:boto:Port: 8001 2023-07-29T00:39:37.632 DEBUG:boto:Params: {} 2023-07-29T00:39:37.633 DEBUG:boto:Token: None 2023-07-29T00:39:37.634 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/ 2023-07-29T00:39:37.635 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Mle3aYo3xzdeHcrINCie96a4FQc= 2023-07-29T00:39:37.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Mle3aYo3xzdeHcrINCie96a4FQc=', 'Content-Length': '0'} 2023-07-29T00:39:37.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006787e6f1f8784a64-0064c45fc9-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.639 DEBUG:boto:b'hiiqos-59obj381000falseobj382023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:37.640 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:37.640 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:37.641 DEBUG:boto:path=/hiiqos-59/?prefix=obj38 2023-07-29T00:39:37.641 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj38 2023-07-29T00:39:37.642 DEBUG:boto:Method: GET 2023-07-29T00:39:37.642 DEBUG:boto:Path: /hiiqos-59/?prefix=obj38 2023-07-29T00:39:37.643 DEBUG:boto:Data: 2023-07-29T00:39:37.643 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.644 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:37.644 DEBUG:boto:Port: 8000 2023-07-29T00:39:37.645 DEBUG:boto:Params: {} 2023-07-29T00:39:37.645 DEBUG:boto:Token: None 2023-07-29T00:39:37.645 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/ 2023-07-29T00:39:37.646 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Mle3aYo3xzdeHcrINCie96a4FQc= 2023-07-29T00:39:37.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Mle3aYo3xzdeHcrINCie96a4FQc=', 'Content-Length': '0'} 2023-07-29T00:39:37.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009355965e62b2357c-0064c45fc9-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.651 DEBUG:boto:b'hiiqos-59obj381000falseobj382023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:37.652 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-07-29T00:39:37.653 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:39:37.653 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:39:37.653 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:39:37.654 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:39:37.654 DEBUG:boto:Method: GET 2023-07-29T00:39:37.654 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:39:37.655 DEBUG:boto:Data: 2023-07-29T00:39:37.655 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.660 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:37.661 DEBUG:boto:Port: 8001 2023-07-29T00:39:37.662 DEBUG:boto:Params: {} 2023-07-29T00:39:37.662 DEBUG:boto:Token: None 2023-07-29T00:39:37.663 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:39:37.663 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O6V7IdYAmmg1xRxq4F7h0hftXh8= 2023-07-29T00:39:37.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O6V7IdYAmmg1xRxq4F7h0hftXh8=', 'Content-Length': '0'} 2023-07-29T00:39:37.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fbe77278407f9b7-0064c45fc9-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.666 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:39:37.667 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:39:37.667 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:39:37.668 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:39:37.669 DEBUG:boto:Method: GET 2023-07-29T00:39:37.669 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:39:37.670 DEBUG:boto:Data: 2023-07-29T00:39:37.670 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.670 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:37.671 DEBUG:boto:Port: 8000 2023-07-29T00:39:37.671 DEBUG:boto:Params: {} 2023-07-29T00:39:37.671 DEBUG:boto:Token: None 2023-07-29T00:39:37.672 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:39:37.673 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O6V7IdYAmmg1xRxq4F7h0hftXh8= 2023-07-29T00:39:37.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O6V7IdYAmmg1xRxq4F7h0hftXh8=', 'Content-Length': '0'} 2023-07-29T00:39:37.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094097eb826992c93-0064c45fc9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.720 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:39:37.721 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:39:37.721 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:39:37.722 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:39:37.722 DEBUG:boto:Method: HEAD 2023-07-29T00:39:37.722 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:39:37.723 DEBUG:boto:Data: 2023-07-29T00:39:37.723 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.723 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:37.724 DEBUG:boto:Port: 8001 2023-07-29T00:39:37.724 DEBUG:boto:Params: {} 2023-07-29T00:39:37.724 DEBUG:boto:Token: None 2023-07-29T00:39:37.725 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:39:37.725 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TZHAtUZzuhXIf/HxzsgLniCLPcM= 2023-07-29T00:39:37.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TZHAtUZzuhXIf/HxzsgLniCLPcM=', 'Content-Length': '0'} 2023-07-29T00:39:37.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6b60e96fa1a4540-0064c45fc9-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.728 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:39:37.728 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:39:37.729 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:39:37.729 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:39:37.729 DEBUG:boto:Method: HEAD 2023-07-29T00:39:37.730 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:39:37.730 DEBUG:boto:Data: 2023-07-29T00:39:37.730 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.731 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:37.731 DEBUG:boto:Port: 8000 2023-07-29T00:39:37.731 DEBUG:boto:Params: {} 2023-07-29T00:39:37.732 DEBUG:boto:Token: None 2023-07-29T00:39:37.732 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:39:37.733 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TZHAtUZzuhXIf/HxzsgLniCLPcM= 2023-07-29T00:39:37.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TZHAtUZzuhXIf/HxzsgLniCLPcM=', 'Content-Length': '0'} 2023-07-29T00:39:37.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094926c75dfd6200c-0064c45fc9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-07-29T00:39:37.736 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:39:37.736 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:39:37.737 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:39:37.737 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:39:37.737 DEBUG:boto:Method: GET 2023-07-29T00:39:37.738 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:39:37.738 DEBUG:boto:Data: 2023-07-29T00:39:37.739 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.739 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:37.739 DEBUG:boto:Port: 8001 2023-07-29T00:39:37.740 DEBUG:boto:Params: {} 2023-07-29T00:39:37.740 DEBUG:boto:Token: None 2023-07-29T00:39:37.741 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:39:37.741 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O6V7IdYAmmg1xRxq4F7h0hftXh8= 2023-07-29T00:39:37.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O6V7IdYAmmg1xRxq4F7h0hftXh8=', 'Content-Length': '0'} 2023-07-29T00:39:37.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f685748d8d4ee26-0064c45fc9-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.744 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:39:37.745 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:39:37.745 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:39:37.745 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:39:37.746 DEBUG:boto:Method: GET 2023-07-29T00:39:37.746 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:39:37.746 DEBUG:boto:Data: 2023-07-29T00:39:37.747 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.747 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:37.748 DEBUG:boto:Port: 8000 2023-07-29T00:39:37.748 DEBUG:boto:Params: {} 2023-07-29T00:39:37.748 DEBUG:boto:Token: None 2023-07-29T00:39:37.749 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:39:37.749 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O6V7IdYAmmg1xRxq4F7h0hftXh8= 2023-07-29T00:39:37.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O6V7IdYAmmg1xRxq4F7h0hftXh8=', 'Content-Length': '0'} 2023-07-29T00:39:37.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011961510e22a1d9a-0064c45fc9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.752 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:37.753 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:37.753 DEBUG:boto:path=/hiiqos-59/?prefix=obj44 2023-07-29T00:39:37.753 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj44 2023-07-29T00:39:37.754 DEBUG:boto:Method: GET 2023-07-29T00:39:37.754 DEBUG:boto:Path: /hiiqos-59/?prefix=obj44 2023-07-29T00:39:37.755 DEBUG:boto:Data: 2023-07-29T00:39:37.755 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.755 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:37.756 DEBUG:boto:Port: 8001 2023-07-29T00:39:37.756 DEBUG:boto:Params: {} 2023-07-29T00:39:37.757 DEBUG:boto:Token: None 2023-07-29T00:39:37.757 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/ 2023-07-29T00:39:37.758 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Mle3aYo3xzdeHcrINCie96a4FQc= 2023-07-29T00:39:37.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Mle3aYo3xzdeHcrINCie96a4FQc=', 'Content-Length': '0'} 2023-07-29T00:39:37.760 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac85cfd258098880-0064c45fc9-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.761 DEBUG:boto:b'hiiqos-59obj441000falseobj442023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:37.762 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:37.762 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:37.763 DEBUG:boto:path=/hiiqos-59/?prefix=obj44 2023-07-29T00:39:37.763 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj44 2023-07-29T00:39:37.763 DEBUG:boto:Method: GET 2023-07-29T00:39:37.764 DEBUG:boto:Path: /hiiqos-59/?prefix=obj44 2023-07-29T00:39:37.764 DEBUG:boto:Data: 2023-07-29T00:39:37.765 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.765 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:37.765 DEBUG:boto:Port: 8000 2023-07-29T00:39:37.766 DEBUG:boto:Params: {} 2023-07-29T00:39:37.766 DEBUG:boto:Token: None 2023-07-29T00:39:37.767 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/ 2023-07-29T00:39:37.767 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Mle3aYo3xzdeHcrINCie96a4FQc= 2023-07-29T00:39:37.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Mle3aYo3xzdeHcrINCie96a4FQc=', 'Content-Length': '0'} 2023-07-29T00:39:37.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081496c8f71c8de7d-0064c45fc9-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.772 DEBUG:boto:b'hiiqos-59obj441000falseobj442023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:37.773 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:39:37.773 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:39:37.774 DEBUG:boto:Method: HEAD 2023-07-29T00:39:37.774 DEBUG:boto:Path: /hiiqos-59/obj44 2023-07-29T00:39:37.774 DEBUG:boto:Data: 2023-07-29T00:39:37.775 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.775 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:37.776 DEBUG:boto:Port: 8001 2023-07-29T00:39:37.776 DEBUG:boto:Params: {} 2023-07-29T00:39:37.776 DEBUG:boto:Token: None 2023-07-29T00:39:37.777 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj44 2023-07-29T00:39:37.777 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0ysY9ky//XFOH4xU4kzrWTJ9oJs= 2023-07-29T00:39:37.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0ysY9ky//XFOH4xU4kzrWTJ9oJs=', 'Content-Length': '0'} 2023-07-29T00:39:37.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cd08bde6bdb989c-0064c45fc9-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.780 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:39:37.781 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:39:37.781 DEBUG:boto:Method: HEAD 2023-07-29T00:39:37.782 DEBUG:boto:Path: /hiiqos-59/obj44 2023-07-29T00:39:37.782 DEBUG:boto:Data: 2023-07-29T00:39:37.782 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.783 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:37.783 DEBUG:boto:Port: 8000 2023-07-29T00:39:37.783 DEBUG:boto:Params: {} 2023-07-29T00:39:37.784 DEBUG:boto:Token: None 2023-07-29T00:39:37.784 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj44 2023-07-29T00:39:37.785 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0ysY9ky//XFOH4xU4kzrWTJ9oJs= 2023-07-29T00:39:37.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0ysY9ky//XFOH4xU4kzrWTJ9oJs=', 'Content-Length': '0'} 2023-07-29T00:39:37.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e74716bc3b5293c-0064c45fc9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-07-29T00:39:37.788 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:39:37.789 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:39:37.789 DEBUG:boto:Method: GET 2023-07-29T00:39:37.789 DEBUG:boto:Path: /hiiqos-59/obj44 2023-07-29T00:39:37.790 DEBUG:boto:Data: 2023-07-29T00:39:37.790 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.791 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:37.791 DEBUG:boto:Port: 8001 2023-07-29T00:39:37.791 DEBUG:boto:Params: {} 2023-07-29T00:39:37.792 DEBUG:boto:Token: None 2023-07-29T00:39:37.792 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj44 2023-07-29T00:39:37.793 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Nq1rwRaSud5W0ks2tLQCfilwZSs= 2023-07-29T00:39:37.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Nq1rwRaSud5W0ks2tLQCfilwZSs=', 'Content-Length': '0'} 2023-07-29T00:39:37.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d8e32152ebd95cf-0064c45fc9-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.844 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:39:37.845 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:39:37.845 DEBUG:boto:Method: GET 2023-07-29T00:39:37.846 DEBUG:boto:Path: /hiiqos-59/obj44 2023-07-29T00:39:37.846 DEBUG:boto:Data: 2023-07-29T00:39:37.847 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.847 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:37.847 DEBUG:boto:Port: 8000 2023-07-29T00:39:37.848 DEBUG:boto:Params: {} 2023-07-29T00:39:37.848 DEBUG:boto:Token: None 2023-07-29T00:39:37.849 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj44 2023-07-29T00:39:37.849 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Nq1rwRaSud5W0ks2tLQCfilwZSs= 2023-07-29T00:39:37.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Nq1rwRaSud5W0ks2tLQCfilwZSs=', 'Content-Length': '0'} 2023-07-29T00:39:37.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ee85eed77bfac90-0064c45fc9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.853 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:37.853 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:37.853 DEBUG:boto:path=/hiiqos-59/?prefix=obj44 2023-07-29T00:39:37.854 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj44 2023-07-29T00:39:37.854 DEBUG:boto:Method: GET 2023-07-29T00:39:37.855 DEBUG:boto:Path: /hiiqos-59/?prefix=obj44 2023-07-29T00:39:37.855 DEBUG:boto:Data: 2023-07-29T00:39:37.855 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.856 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:37.856 DEBUG:boto:Port: 8001 2023-07-29T00:39:37.857 DEBUG:boto:Params: {} 2023-07-29T00:39:37.857 DEBUG:boto:Token: None 2023-07-29T00:39:37.858 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/ 2023-07-29T00:39:37.858 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Mle3aYo3xzdeHcrINCie96a4FQc= 2023-07-29T00:39:37.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Mle3aYo3xzdeHcrINCie96a4FQc=', 'Content-Length': '0'} 2023-07-29T00:39:37.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b436a3888d7ce4ff-0064c45fc9-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.862 DEBUG:boto:b'hiiqos-59obj441000falseobj442023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:37.863 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:37.863 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:37.863 DEBUG:boto:path=/hiiqos-59/?prefix=obj44 2023-07-29T00:39:37.864 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj44 2023-07-29T00:39:37.864 DEBUG:boto:Method: GET 2023-07-29T00:39:37.865 DEBUG:boto:Path: /hiiqos-59/?prefix=obj44 2023-07-29T00:39:37.865 DEBUG:boto:Data: 2023-07-29T00:39:37.865 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.866 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:37.866 DEBUG:boto:Port: 8000 2023-07-29T00:39:37.867 DEBUG:boto:Params: {} 2023-07-29T00:39:37.867 DEBUG:boto:Token: None 2023-07-29T00:39:37.868 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/ 2023-07-29T00:39:37.868 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Mle3aYo3xzdeHcrINCie96a4FQc= 2023-07-29T00:39:37.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Mle3aYo3xzdeHcrINCie96a4FQc=', 'Content-Length': '0'} 2023-07-29T00:39:37.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c8d04f74a50301b-0064c45fc9-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.874 DEBUG:boto:b'hiiqos-59obj441000falseobj442023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:37.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-07-29T00:39:37.875 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:39:37.876 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:39:37.876 DEBUG:boto:path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:39:37.877 DEBUG:boto:auth_path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:39:37.877 DEBUG:boto:Method: GET 2023-07-29T00:39:37.878 DEBUG:boto:Path: /hiiqos-59/obj46?versionId=null 2023-07-29T00:39:37.878 DEBUG:boto:Data: 2023-07-29T00:39:37.879 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.879 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:37.880 DEBUG:boto:Port: 8001 2023-07-29T00:39:37.880 DEBUG:boto:Params: {} 2023-07-29T00:39:37.880 DEBUG:boto:Token: None 2023-07-29T00:39:37.881 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj46?versionId=null 2023-07-29T00:39:37.881 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4OOF/axPEX3pIJmNbjppcxELEB8= 2023-07-29T00:39:37.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4OOF/axPEX3pIJmNbjppcxELEB8=', 'Content-Length': '0'} 2023-07-29T00:39:37.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a96d2c07d7322b8-0064c45fc9-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.885 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:39:37.885 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:39:37.886 DEBUG:boto:path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:39:37.886 DEBUG:boto:auth_path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:39:37.887 DEBUG:boto:Method: GET 2023-07-29T00:39:37.887 DEBUG:boto:Path: /hiiqos-59/obj46?versionId=null 2023-07-29T00:39:37.888 DEBUG:boto:Data: 2023-07-29T00:39:37.888 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.889 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:37.889 DEBUG:boto:Port: 8000 2023-07-29T00:39:37.890 DEBUG:boto:Params: {} 2023-07-29T00:39:37.890 DEBUG:boto:Token: None 2023-07-29T00:39:37.890 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj46?versionId=null 2023-07-29T00:39:37.891 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4OOF/axPEX3pIJmNbjppcxELEB8= 2023-07-29T00:39:37.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4OOF/axPEX3pIJmNbjppcxELEB8=', 'Content-Length': '0'} 2023-07-29T00:39:37.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd106a23ad123b97-0064c45fc9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.894 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:39:37.895 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:39:37.895 DEBUG:boto:path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:39:37.896 DEBUG:boto:auth_path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:39:37.897 DEBUG:boto:Method: HEAD 2023-07-29T00:39:37.897 DEBUG:boto:Path: /hiiqos-59/obj46?versionId=null 2023-07-29T00:39:37.898 DEBUG:boto:Data: 2023-07-29T00:39:37.898 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.898 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:37.899 DEBUG:boto:Port: 8001 2023-07-29T00:39:37.899 DEBUG:boto:Params: {} 2023-07-29T00:39:37.900 DEBUG:boto:Token: None 2023-07-29T00:39:37.900 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj46?versionId=null 2023-07-29T00:39:37.900 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/goHc5N72lfQmtmgZEJndlmRwX4= 2023-07-29T00:39:37.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/goHc5N72lfQmtmgZEJndlmRwX4=', 'Content-Length': '0'} 2023-07-29T00:39:37.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc546843c1c41016-0064c45fc9-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.904 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:39:37.904 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:39:37.905 DEBUG:boto:path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:39:37.905 DEBUG:boto:auth_path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:39:37.905 DEBUG:boto:Method: HEAD 2023-07-29T00:39:37.906 DEBUG:boto:Path: /hiiqos-59/obj46?versionId=null 2023-07-29T00:39:37.906 DEBUG:boto:Data: 2023-07-29T00:39:37.906 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.907 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:37.907 DEBUG:boto:Port: 8000 2023-07-29T00:39:37.907 DEBUG:boto:Params: {} 2023-07-29T00:39:37.908 DEBUG:boto:Token: None 2023-07-29T00:39:37.908 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj46?versionId=null 2023-07-29T00:39:37.909 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/goHc5N72lfQmtmgZEJndlmRwX4= 2023-07-29T00:39:37.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/goHc5N72lfQmtmgZEJndlmRwX4=', 'Content-Length': '0'} 2023-07-29T00:39:37.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e8deecf563b870f-0064c45fc9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-07-29T00:39:37.912 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:39:37.912 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:39:37.913 DEBUG:boto:path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:39:37.913 DEBUG:boto:auth_path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:39:37.913 DEBUG:boto:Method: GET 2023-07-29T00:39:37.914 DEBUG:boto:Path: /hiiqos-59/obj46?versionId=null 2023-07-29T00:39:37.914 DEBUG:boto:Data: 2023-07-29T00:39:37.915 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.915 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:37.915 DEBUG:boto:Port: 8001 2023-07-29T00:39:37.916 DEBUG:boto:Params: {} 2023-07-29T00:39:37.916 DEBUG:boto:Token: None 2023-07-29T00:39:37.916 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj46?versionId=null 2023-07-29T00:39:37.917 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4OOF/axPEX3pIJmNbjppcxELEB8= 2023-07-29T00:39:37.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4OOF/axPEX3pIJmNbjppcxELEB8=', 'Content-Length': '0'} 2023-07-29T00:39:37.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fd471e6c9b37a41-0064c45fc9-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.920 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:39:37.921 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:39:37.921 DEBUG:boto:path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:39:37.921 DEBUG:boto:auth_path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:39:37.922 DEBUG:boto:Method: GET 2023-07-29T00:39:37.922 DEBUG:boto:Path: /hiiqos-59/obj46?versionId=null 2023-07-29T00:39:37.922 DEBUG:boto:Data: 2023-07-29T00:39:37.923 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.923 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:37.923 DEBUG:boto:Port: 8000 2023-07-29T00:39:37.924 DEBUG:boto:Params: {} 2023-07-29T00:39:37.924 DEBUG:boto:Token: None 2023-07-29T00:39:37.925 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj46?versionId=null 2023-07-29T00:39:37.925 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4OOF/axPEX3pIJmNbjppcxELEB8= 2023-07-29T00:39:37.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4OOF/axPEX3pIJmNbjppcxELEB8=', 'Content-Length': '0'} 2023-07-29T00:39:37.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000526357b0a69ad7c-0064c45fc9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.928 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:37.928 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:37.929 DEBUG:boto:path=/hiiqos-59/?prefix=obj46 2023-07-29T00:39:37.929 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj46 2023-07-29T00:39:37.930 DEBUG:boto:Method: GET 2023-07-29T00:39:37.930 DEBUG:boto:Path: /hiiqos-59/?prefix=obj46 2023-07-29T00:39:37.930 DEBUG:boto:Data: 2023-07-29T00:39:37.931 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.931 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:37.931 DEBUG:boto:Port: 8001 2023-07-29T00:39:37.932 DEBUG:boto:Params: {} 2023-07-29T00:39:37.932 DEBUG:boto:Token: None 2023-07-29T00:39:37.932 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/ 2023-07-29T00:39:37.933 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Mle3aYo3xzdeHcrINCie96a4FQc= 2023-07-29T00:39:37.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Mle3aYo3xzdeHcrINCie96a4FQc=', 'Content-Length': '0'} 2023-07-29T00:39:37.936 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae42646c58974ccc-0064c45fc9-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.937 DEBUG:boto:b'hiiqos-59obj461000falseobj462023-07-29T00:33:06.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:37.938 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:37.938 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:37.938 DEBUG:boto:path=/hiiqos-59/?prefix=obj46 2023-07-29T00:39:37.939 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj46 2023-07-29T00:39:37.939 DEBUG:boto:Method: GET 2023-07-29T00:39:37.939 DEBUG:boto:Path: /hiiqos-59/?prefix=obj46 2023-07-29T00:39:37.940 DEBUG:boto:Data: 2023-07-29T00:39:37.940 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.941 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:37.941 DEBUG:boto:Port: 8000 2023-07-29T00:39:37.944 DEBUG:boto:Params: {} 2023-07-29T00:39:37.945 DEBUG:boto:Token: None 2023-07-29T00:39:37.946 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/ 2023-07-29T00:39:37.946 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Mle3aYo3xzdeHcrINCie96a4FQc= 2023-07-29T00:39:37.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Mle3aYo3xzdeHcrINCie96a4FQc=', 'Content-Length': '0'} 2023-07-29T00:39:37.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008380b150ddf3d26-0064c45fc9-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.952 DEBUG:boto:b'hiiqos-59obj461000falseobj462023-07-29T00:33:06.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:37.952 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:39:37.953 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:39:37.953 DEBUG:boto:Method: HEAD 2023-07-29T00:39:37.954 DEBUG:boto:Path: /hiiqos-59/obj46 2023-07-29T00:39:37.954 DEBUG:boto:Data: 2023-07-29T00:39:37.954 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.955 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:37.955 DEBUG:boto:Port: 8001 2023-07-29T00:39:37.956 DEBUG:boto:Params: {} 2023-07-29T00:39:37.957 DEBUG:boto:Token: None 2023-07-29T00:39:37.957 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj46 2023-07-29T00:39:37.958 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:t7whUFVZ3M3xK8qux6eSs6dL5ko= 2023-07-29T00:39:37.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t7whUFVZ3M3xK8qux6eSs6dL5ko=', 'Content-Length': '0'} 2023-07-29T00:39:37.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000280533913ae4c742-0064c45fc9-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.962 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:39:37.963 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:39:37.963 DEBUG:boto:Method: HEAD 2023-07-29T00:39:37.964 DEBUG:boto:Path: /hiiqos-59/obj46 2023-07-29T00:39:37.964 DEBUG:boto:Data: 2023-07-29T00:39:37.965 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.965 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:37.968 DEBUG:boto:Port: 8000 2023-07-29T00:39:37.969 DEBUG:boto:Params: {} 2023-07-29T00:39:37.969 DEBUG:boto:Token: None 2023-07-29T00:39:37.969 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj46 2023-07-29T00:39:37.970 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:t7whUFVZ3M3xK8qux6eSs6dL5ko= 2023-07-29T00:39:37.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t7whUFVZ3M3xK8qux6eSs6dL5ko=', 'Content-Length': '0'} 2023-07-29T00:39:37.972 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f94747ef086de9ea-0064c45fc9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:37.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-07-29T00:39:37.973 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:39:37.974 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:39:37.974 DEBUG:boto:Method: GET 2023-07-29T00:39:37.974 DEBUG:boto:Path: /hiiqos-59/obj46 2023-07-29T00:39:37.975 DEBUG:boto:Data: 2023-07-29T00:39:37.975 DEBUG:boto:Headers: {} 2023-07-29T00:39:37.976 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:37.976 DEBUG:boto:Port: 8001 2023-07-29T00:39:37.976 DEBUG:boto:Params: {} 2023-07-29T00:39:37.977 DEBUG:boto:Token: None 2023-07-29T00:39:37.977 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:37 GMT /hiiqos-59/obj46 2023-07-29T00:39:37.978 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YYnyKR0rHD1Bnec/+ADeIAM6P9E= 2023-07-29T00:39:37.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YYnyKR0rHD1Bnec/+ADeIAM6P9E=', 'Content-Length': '0'} 2023-07-29T00:39:37.980 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061d77581896ae889-0064c45fc9-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.020 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:39:38.021 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:39:38.022 DEBUG:boto:Method: GET 2023-07-29T00:39:38.022 DEBUG:boto:Path: /hiiqos-59/obj46 2023-07-29T00:39:38.023 DEBUG:boto:Data: 2023-07-29T00:39:38.024 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.025 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:38.025 DEBUG:boto:Port: 8000 2023-07-29T00:39:38.026 DEBUG:boto:Params: {} 2023-07-29T00:39:38.027 DEBUG:boto:Token: None 2023-07-29T00:39:38.027 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj46 2023-07-29T00:39:38.028 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7h6weiGHJGFQ3GcxoYCnppbLUT4= 2023-07-29T00:39:38.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7h6weiGHJGFQ3GcxoYCnppbLUT4=', 'Content-Length': '0'} 2023-07-29T00:39:38.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c99beabd103704b-0064c45fc9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.031 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:38.032 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:38.032 DEBUG:boto:path=/hiiqos-59/?prefix=obj46 2023-07-29T00:39:38.033 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj46 2023-07-29T00:39:38.033 DEBUG:boto:Method: GET 2023-07-29T00:39:38.033 DEBUG:boto:Path: /hiiqos-59/?prefix=obj46 2023-07-29T00:39:38.034 DEBUG:boto:Data: 2023-07-29T00:39:38.035 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.035 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:38.035 DEBUG:boto:Port: 8001 2023-07-29T00:39:38.036 DEBUG:boto:Params: {} 2023-07-29T00:39:38.037 DEBUG:boto:Token: None 2023-07-29T00:39:38.037 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/ 2023-07-29T00:39:38.038 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RvKVsO6e+NPVZ6UkJytUnnUWrBE= 2023-07-29T00:39:38.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RvKVsO6e+NPVZ6UkJytUnnUWrBE=', 'Content-Length': '0'} 2023-07-29T00:39:38.040 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b762487c4d8372c8-0064c45fc9-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.041 DEBUG:boto:b'hiiqos-59obj461000falseobj462023-07-29T00:33:06.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:38.042 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:38.042 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:38.042 DEBUG:boto:path=/hiiqos-59/?prefix=obj46 2023-07-29T00:39:38.043 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj46 2023-07-29T00:39:38.043 DEBUG:boto:Method: GET 2023-07-29T00:39:38.044 DEBUG:boto:Path: /hiiqos-59/?prefix=obj46 2023-07-29T00:39:38.044 DEBUG:boto:Data: 2023-07-29T00:39:38.044 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.045 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:38.045 DEBUG:boto:Port: 8000 2023-07-29T00:39:38.045 DEBUG:boto:Params: {} 2023-07-29T00:39:38.046 DEBUG:boto:Token: None 2023-07-29T00:39:38.047 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/ 2023-07-29T00:39:38.047 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RvKVsO6e+NPVZ6UkJytUnnUWrBE= 2023-07-29T00:39:38.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RvKVsO6e+NPVZ6UkJytUnnUWrBE=', 'Content-Length': '0'} 2023-07-29T00:39:38.052 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c44f25433ef56837-0064c45fc9-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.053 DEBUG:boto:b'hiiqos-59obj461000falseobj462023-07-29T00:33:06.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:38.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-07-29T00:39:38.054 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:39:38.054 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:39:38.055 DEBUG:boto:path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:39:38.055 DEBUG:boto:auth_path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:39:38.055 DEBUG:boto:Method: GET 2023-07-29T00:39:38.056 DEBUG:boto:Path: /hiiqos-59/obj51?versionId=null 2023-07-29T00:39:38.056 DEBUG:boto:Data: 2023-07-29T00:39:38.057 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.057 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:38.057 DEBUG:boto:Port: 8001 2023-07-29T00:39:38.058 DEBUG:boto:Params: {} 2023-07-29T00:39:38.058 DEBUG:boto:Token: None 2023-07-29T00:39:38.059 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj51?versionId=null 2023-07-29T00:39:38.059 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lL6nPV58Ymutykn0eelmDjN5Uzg= 2023-07-29T00:39:38.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lL6nPV58Ymutykn0eelmDjN5Uzg=', 'Content-Length': '0'} 2023-07-29T00:39:38.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fa019d991836fb3-0064c45fc9-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.062 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:39:38.063 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:39:38.063 DEBUG:boto:path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:39:38.063 DEBUG:boto:auth_path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:39:38.064 DEBUG:boto:Method: GET 2023-07-29T00:39:38.064 DEBUG:boto:Path: /hiiqos-59/obj51?versionId=null 2023-07-29T00:39:38.065 DEBUG:boto:Data: 2023-07-29T00:39:38.065 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.065 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:38.066 DEBUG:boto:Port: 8000 2023-07-29T00:39:38.066 DEBUG:boto:Params: {} 2023-07-29T00:39:38.067 DEBUG:boto:Token: None 2023-07-29T00:39:38.067 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj51?versionId=null 2023-07-29T00:39:38.068 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lL6nPV58Ymutykn0eelmDjN5Uzg= 2023-07-29T00:39:38.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lL6nPV58Ymutykn0eelmDjN5Uzg=', 'Content-Length': '0'} 2023-07-29T00:39:38.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000607d66200f839077-0064c45fca-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.071 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:39:38.071 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:39:38.072 DEBUG:boto:path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:39:38.072 DEBUG:boto:auth_path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:39:38.072 DEBUG:boto:Method: HEAD 2023-07-29T00:39:38.073 DEBUG:boto:Path: /hiiqos-59/obj51?versionId=null 2023-07-29T00:39:38.073 DEBUG:boto:Data: 2023-07-29T00:39:38.074 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.074 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:38.074 DEBUG:boto:Port: 8001 2023-07-29T00:39:38.075 DEBUG:boto:Params: {} 2023-07-29T00:39:38.075 DEBUG:boto:Token: None 2023-07-29T00:39:38.075 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj51?versionId=null 2023-07-29T00:39:38.076 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hyys/Z5hN9wtyBn5D814MysNCg0= 2023-07-29T00:39:38.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hyys/Z5hN9wtyBn5D814MysNCg0=', 'Content-Length': '0'} 2023-07-29T00:39:38.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b99cfb887c1a5a89-0064c45fca-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.081 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:39:38.082 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:39:38.082 DEBUG:boto:path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:39:38.083 DEBUG:boto:auth_path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:39:38.084 DEBUG:boto:Method: HEAD 2023-07-29T00:39:38.084 DEBUG:boto:Path: /hiiqos-59/obj51?versionId=null 2023-07-29T00:39:38.085 DEBUG:boto:Data: 2023-07-29T00:39:38.086 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.087 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:38.087 DEBUG:boto:Port: 8000 2023-07-29T00:39:38.087 DEBUG:boto:Params: {} 2023-07-29T00:39:38.088 DEBUG:boto:Token: None 2023-07-29T00:39:38.088 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj51?versionId=null 2023-07-29T00:39:38.089 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hyys/Z5hN9wtyBn5D814MysNCg0= 2023-07-29T00:39:38.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hyys/Z5hN9wtyBn5D814MysNCg0=', 'Content-Length': '0'} 2023-07-29T00:39:38.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001176a950c4f1b7fa-0064c45fca-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-07-29T00:39:38.093 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:39:38.094 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:39:38.094 DEBUG:boto:path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:39:38.094 DEBUG:boto:auth_path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:39:38.095 DEBUG:boto:Method: GET 2023-07-29T00:39:38.095 DEBUG:boto:Path: /hiiqos-59/obj51?versionId=null 2023-07-29T00:39:38.096 DEBUG:boto:Data: 2023-07-29T00:39:38.096 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.097 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:38.098 DEBUG:boto:Port: 8001 2023-07-29T00:39:38.099 DEBUG:boto:Params: {} 2023-07-29T00:39:38.100 DEBUG:boto:Token: None 2023-07-29T00:39:38.101 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj51?versionId=null 2023-07-29T00:39:38.102 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lL6nPV58Ymutykn0eelmDjN5Uzg= 2023-07-29T00:39:38.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lL6nPV58Ymutykn0eelmDjN5Uzg=', 'Content-Length': '0'} 2023-07-29T00:39:38.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d4f0a97ab459a1f-0064c45fca-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.107 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:39:38.107 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:39:38.108 DEBUG:boto:path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:39:38.108 DEBUG:boto:auth_path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:39:38.108 DEBUG:boto:Method: GET 2023-07-29T00:39:38.109 DEBUG:boto:Path: /hiiqos-59/obj51?versionId=null 2023-07-29T00:39:38.109 DEBUG:boto:Data: 2023-07-29T00:39:38.109 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.110 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:38.110 DEBUG:boto:Port: 8000 2023-07-29T00:39:38.110 DEBUG:boto:Params: {} 2023-07-29T00:39:38.111 DEBUG:boto:Token: None 2023-07-29T00:39:38.111 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj51?versionId=null 2023-07-29T00:39:38.112 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lL6nPV58Ymutykn0eelmDjN5Uzg= 2023-07-29T00:39:38.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lL6nPV58Ymutykn0eelmDjN5Uzg=', 'Content-Length': '0'} 2023-07-29T00:39:38.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e51a49bfad5a0cfd-0064c45fca-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.115 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:38.115 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:38.116 DEBUG:boto:path=/hiiqos-59/?prefix=obj51 2023-07-29T00:39:38.116 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj51 2023-07-29T00:39:38.116 DEBUG:boto:Method: GET 2023-07-29T00:39:38.117 DEBUG:boto:Path: /hiiqos-59/?prefix=obj51 2023-07-29T00:39:38.117 DEBUG:boto:Data: 2023-07-29T00:39:38.117 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.118 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:38.118 DEBUG:boto:Port: 8001 2023-07-29T00:39:38.118 DEBUG:boto:Params: {} 2023-07-29T00:39:38.119 DEBUG:boto:Token: None 2023-07-29T00:39:38.119 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/ 2023-07-29T00:39:38.120 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RvKVsO6e+NPVZ6UkJytUnnUWrBE= 2023-07-29T00:39:38.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RvKVsO6e+NPVZ6UkJytUnnUWrBE=', 'Content-Length': '0'} 2023-07-29T00:39:38.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cb3335d7f16109d-0064c45fca-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.123 DEBUG:boto:b'hiiqos-59obj511000falseobj512023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:38.124 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:38.124 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:38.125 DEBUG:boto:path=/hiiqos-59/?prefix=obj51 2023-07-29T00:39:38.125 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj51 2023-07-29T00:39:38.125 DEBUG:boto:Method: GET 2023-07-29T00:39:38.126 DEBUG:boto:Path: /hiiqos-59/?prefix=obj51 2023-07-29T00:39:38.126 DEBUG:boto:Data: 2023-07-29T00:39:38.126 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.127 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:38.127 DEBUG:boto:Port: 8000 2023-07-29T00:39:38.127 DEBUG:boto:Params: {} 2023-07-29T00:39:38.128 DEBUG:boto:Token: None 2023-07-29T00:39:38.128 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/ 2023-07-29T00:39:38.129 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RvKVsO6e+NPVZ6UkJytUnnUWrBE= 2023-07-29T00:39:38.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RvKVsO6e+NPVZ6UkJytUnnUWrBE=', 'Content-Length': '0'} 2023-07-29T00:39:38.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2a4777eccb452bc-0064c45fca-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.135 DEBUG:boto:b'hiiqos-59obj511000falseobj512023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:38.136 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:39:38.136 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:39:38.136 DEBUG:boto:Method: HEAD 2023-07-29T00:39:38.137 DEBUG:boto:Path: /hiiqos-59/obj51 2023-07-29T00:39:38.137 DEBUG:boto:Data: 2023-07-29T00:39:38.138 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.138 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:38.138 DEBUG:boto:Port: 8001 2023-07-29T00:39:38.139 DEBUG:boto:Params: {} 2023-07-29T00:39:38.139 DEBUG:boto:Token: None 2023-07-29T00:39:38.139 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj51 2023-07-29T00:39:38.140 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Dh3eck/XFVShLkgvDXK6dq78zSs= 2023-07-29T00:39:38.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Dh3eck/XFVShLkgvDXK6dq78zSs=', 'Content-Length': '0'} 2023-07-29T00:39:38.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d57ed77d1ef2ac9c-0064c45fca-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.143 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:39:38.143 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:39:38.144 DEBUG:boto:Method: HEAD 2023-07-29T00:39:38.144 DEBUG:boto:Path: /hiiqos-59/obj51 2023-07-29T00:39:38.144 DEBUG:boto:Data: 2023-07-29T00:39:38.145 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.145 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:38.145 DEBUG:boto:Port: 8000 2023-07-29T00:39:38.146 DEBUG:boto:Params: {} 2023-07-29T00:39:38.146 DEBUG:boto:Token: None 2023-07-29T00:39:38.147 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj51 2023-07-29T00:39:38.147 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Dh3eck/XFVShLkgvDXK6dq78zSs= 2023-07-29T00:39:38.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Dh3eck/XFVShLkgvDXK6dq78zSs=', 'Content-Length': '0'} 2023-07-29T00:39:38.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006088014efffd7918-0064c45fca-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-07-29T00:39:38.150 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:39:38.151 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:39:38.151 DEBUG:boto:Method: GET 2023-07-29T00:39:38.151 DEBUG:boto:Path: /hiiqos-59/obj51 2023-07-29T00:39:38.152 DEBUG:boto:Data: 2023-07-29T00:39:38.152 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.153 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:38.153 DEBUG:boto:Port: 8001 2023-07-29T00:39:38.153 DEBUG:boto:Params: {} 2023-07-29T00:39:38.154 DEBUG:boto:Token: None 2023-07-29T00:39:38.154 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj51 2023-07-29T00:39:38.155 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CF+xVOlvYmsUlb+c/xuVMKzYmmw= 2023-07-29T00:39:38.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CF+xVOlvYmsUlb+c/xuVMKzYmmw=', 'Content-Length': '0'} 2023-07-29T00:39:38.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6cb7093f16d0f78-0064c45fca-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.157 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:39:38.158 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:39:38.158 DEBUG:boto:Method: GET 2023-07-29T00:39:38.158 DEBUG:boto:Path: /hiiqos-59/obj51 2023-07-29T00:39:38.159 DEBUG:boto:Data: 2023-07-29T00:39:38.159 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.159 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:38.160 DEBUG:boto:Port: 8000 2023-07-29T00:39:38.160 DEBUG:boto:Params: {} 2023-07-29T00:39:38.161 DEBUG:boto:Token: None 2023-07-29T00:39:38.161 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj51 2023-07-29T00:39:38.161 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CF+xVOlvYmsUlb+c/xuVMKzYmmw= 2023-07-29T00:39:38.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CF+xVOlvYmsUlb+c/xuVMKzYmmw=', 'Content-Length': '0'} 2023-07-29T00:39:38.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006573312645745749-0064c45fca-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.204 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:38.205 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:38.205 DEBUG:boto:path=/hiiqos-59/?prefix=obj51 2023-07-29T00:39:38.206 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj51 2023-07-29T00:39:38.206 DEBUG:boto:Method: GET 2023-07-29T00:39:38.206 DEBUG:boto:Path: /hiiqos-59/?prefix=obj51 2023-07-29T00:39:38.207 DEBUG:boto:Data: 2023-07-29T00:39:38.207 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.207 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:38.208 DEBUG:boto:Port: 8001 2023-07-29T00:39:38.208 DEBUG:boto:Params: {} 2023-07-29T00:39:38.209 DEBUG:boto:Token: None 2023-07-29T00:39:38.209 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/ 2023-07-29T00:39:38.209 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RvKVsO6e+NPVZ6UkJytUnnUWrBE= 2023-07-29T00:39:38.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RvKVsO6e+NPVZ6UkJytUnnUWrBE=', 'Content-Length': '0'} 2023-07-29T00:39:38.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e227fecd56f6485e-0064c45fca-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.213 DEBUG:boto:b'hiiqos-59obj511000falseobj512023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:38.213 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:38.214 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:38.214 DEBUG:boto:path=/hiiqos-59/?prefix=obj51 2023-07-29T00:39:38.214 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj51 2023-07-29T00:39:38.215 DEBUG:boto:Method: GET 2023-07-29T00:39:38.215 DEBUG:boto:Path: /hiiqos-59/?prefix=obj51 2023-07-29T00:39:38.215 DEBUG:boto:Data: 2023-07-29T00:39:38.216 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.216 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:38.216 DEBUG:boto:Port: 8000 2023-07-29T00:39:38.217 DEBUG:boto:Params: {} 2023-07-29T00:39:38.217 DEBUG:boto:Token: None 2023-07-29T00:39:38.218 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/ 2023-07-29T00:39:38.218 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RvKVsO6e+NPVZ6UkJytUnnUWrBE= 2023-07-29T00:39:38.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RvKVsO6e+NPVZ6UkJytUnnUWrBE=', 'Content-Length': '0'} 2023-07-29T00:39:38.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045f06b293e56432f-0064c45fca-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.223 DEBUG:boto:b'hiiqos-59obj511000falseobj512023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:38.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-07-29T00:39:38.224 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:39:38.225 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:39:38.225 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:39:38.225 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:39:38.226 DEBUG:boto:Method: GET 2023-07-29T00:39:38.226 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:39:38.226 DEBUG:boto:Data: 2023-07-29T00:39:38.227 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.227 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:38.227 DEBUG:boto:Port: 8001 2023-07-29T00:39:38.228 DEBUG:boto:Params: {} 2023-07-29T00:39:38.228 DEBUG:boto:Token: None 2023-07-29T00:39:38.229 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:39:38.229 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pN85ABL48oARittEnKzF2cdTUyo= 2023-07-29T00:39:38.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pN85ABL48oARittEnKzF2cdTUyo=', 'Content-Length': '0'} 2023-07-29T00:39:38.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9041921140ef819-0064c45fca-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.276 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:39:38.277 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:39:38.278 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:39:38.279 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:39:38.280 DEBUG:boto:Method: GET 2023-07-29T00:39:38.281 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:39:38.281 DEBUG:boto:Data: 2023-07-29T00:39:38.282 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.283 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:38.283 DEBUG:boto:Port: 8000 2023-07-29T00:39:38.284 DEBUG:boto:Params: {} 2023-07-29T00:39:38.285 DEBUG:boto:Token: None 2023-07-29T00:39:38.286 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:39:38.287 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pN85ABL48oARittEnKzF2cdTUyo= 2023-07-29T00:39:38.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pN85ABL48oARittEnKzF2cdTUyo=', 'Content-Length': '0'} 2023-07-29T00:39:38.291 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd92ac31da8732b0-0064c45fca-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.292 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:39:38.293 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:39:38.293 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:39:38.293 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:39:38.294 DEBUG:boto:Method: HEAD 2023-07-29T00:39:38.294 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:39:38.294 DEBUG:boto:Data: 2023-07-29T00:39:38.295 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.295 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:38.295 DEBUG:boto:Port: 8001 2023-07-29T00:39:38.296 DEBUG:boto:Params: {} 2023-07-29T00:39:38.296 DEBUG:boto:Token: None 2023-07-29T00:39:38.297 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:39:38.297 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6YhHF+PeSTDHo6x7WhRdWlVdpO4= 2023-07-29T00:39:38.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6YhHF+PeSTDHo6x7WhRdWlVdpO4=', 'Content-Length': '0'} 2023-07-29T00:39:38.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3592118f3b62161-0064c45fca-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.300 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:39:38.301 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:39:38.301 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:39:38.301 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:39:38.302 DEBUG:boto:Method: HEAD 2023-07-29T00:39:38.302 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:39:38.302 DEBUG:boto:Data: 2023-07-29T00:39:38.303 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.303 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:38.304 DEBUG:boto:Port: 8000 2023-07-29T00:39:38.304 DEBUG:boto:Params: {} 2023-07-29T00:39:38.305 DEBUG:boto:Token: None 2023-07-29T00:39:38.305 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:39:38.306 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6YhHF+PeSTDHo6x7WhRdWlVdpO4= 2023-07-29T00:39:38.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6YhHF+PeSTDHo6x7WhRdWlVdpO4=', 'Content-Length': '0'} 2023-07-29T00:39:38.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000085f64cfbc299cb7-0064c45fca-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-07-29T00:39:38.309 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:39:38.310 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:39:38.310 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:39:38.311 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:39:38.311 DEBUG:boto:Method: GET 2023-07-29T00:39:38.311 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:39:38.312 DEBUG:boto:Data: 2023-07-29T00:39:38.312 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.313 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:38.313 DEBUG:boto:Port: 8001 2023-07-29T00:39:38.314 DEBUG:boto:Params: {} 2023-07-29T00:39:38.315 DEBUG:boto:Token: None 2023-07-29T00:39:38.316 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:39:38.317 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pN85ABL48oARittEnKzF2cdTUyo= 2023-07-29T00:39:38.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pN85ABL48oARittEnKzF2cdTUyo=', 'Content-Length': '0'} 2023-07-29T00:39:38.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8904fc5b28c2bbd-0064c45fca-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.321 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:39:38.322 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:39:38.322 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:39:38.322 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:39:38.323 DEBUG:boto:Method: GET 2023-07-29T00:39:38.323 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:39:38.323 DEBUG:boto:Data: 2023-07-29T00:39:38.324 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.324 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:38.324 DEBUG:boto:Port: 8000 2023-07-29T00:39:38.325 DEBUG:boto:Params: {} 2023-07-29T00:39:38.325 DEBUG:boto:Token: None 2023-07-29T00:39:38.326 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:39:38.326 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pN85ABL48oARittEnKzF2cdTUyo= 2023-07-29T00:39:38.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pN85ABL48oARittEnKzF2cdTUyo=', 'Content-Length': '0'} 2023-07-29T00:39:38.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca52d24cdd919147-0064c45fca-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.329 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:38.330 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:38.331 DEBUG:boto:path=/hiiqos-59/?prefix=obj55 2023-07-29T00:39:38.331 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj55 2023-07-29T00:39:38.332 DEBUG:boto:Method: GET 2023-07-29T00:39:38.333 DEBUG:boto:Path: /hiiqos-59/?prefix=obj55 2023-07-29T00:39:38.334 DEBUG:boto:Data: 2023-07-29T00:39:38.335 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.336 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:38.337 DEBUG:boto:Port: 8001 2023-07-29T00:39:38.338 DEBUG:boto:Params: {} 2023-07-29T00:39:38.338 DEBUG:boto:Token: None 2023-07-29T00:39:38.339 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/ 2023-07-29T00:39:38.340 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RvKVsO6e+NPVZ6UkJytUnnUWrBE= 2023-07-29T00:39:38.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RvKVsO6e+NPVZ6UkJytUnnUWrBE=', 'Content-Length': '0'} 2023-07-29T00:39:38.343 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000107c22e549bdd966-0064c45fca-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.344 DEBUG:boto:b'hiiqos-59obj551000falseobj552023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:38.345 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:38.345 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:38.346 DEBUG:boto:path=/hiiqos-59/?prefix=obj55 2023-07-29T00:39:38.346 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj55 2023-07-29T00:39:38.346 DEBUG:boto:Method: GET 2023-07-29T00:39:38.347 DEBUG:boto:Path: /hiiqos-59/?prefix=obj55 2023-07-29T00:39:38.347 DEBUG:boto:Data: 2023-07-29T00:39:38.348 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.348 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:38.348 DEBUG:boto:Port: 8000 2023-07-29T00:39:38.349 DEBUG:boto:Params: {} 2023-07-29T00:39:38.349 DEBUG:boto:Token: None 2023-07-29T00:39:38.350 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/ 2023-07-29T00:39:38.350 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RvKVsO6e+NPVZ6UkJytUnnUWrBE= 2023-07-29T00:39:38.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RvKVsO6e+NPVZ6UkJytUnnUWrBE=', 'Content-Length': '0'} 2023-07-29T00:39:38.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000609fc1eb2e6afa48-0064c45fca-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.396 DEBUG:boto:b'hiiqos-59obj551000falseobj552023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:38.397 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:39:38.398 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:39:38.398 DEBUG:boto:Method: HEAD 2023-07-29T00:39:38.399 DEBUG:boto:Path: /hiiqos-59/obj55 2023-07-29T00:39:38.399 DEBUG:boto:Data: 2023-07-29T00:39:38.399 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.400 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:38.400 DEBUG:boto:Port: 8001 2023-07-29T00:39:38.400 DEBUG:boto:Params: {} 2023-07-29T00:39:38.401 DEBUG:boto:Token: None 2023-07-29T00:39:38.401 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj55 2023-07-29T00:39:38.402 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ubn82oyBh7l8Ot9aUxK+lP//tQw= 2023-07-29T00:39:38.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ubn82oyBh7l8Ot9aUxK+lP//tQw=', 'Content-Length': '0'} 2023-07-29T00:39:38.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000155ac5669f40daae-0064c45fca-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.405 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:39:38.406 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:39:38.406 DEBUG:boto:Method: HEAD 2023-07-29T00:39:38.407 DEBUG:boto:Path: /hiiqos-59/obj55 2023-07-29T00:39:38.408 DEBUG:boto:Data: 2023-07-29T00:39:38.409 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.409 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:38.410 DEBUG:boto:Port: 8000 2023-07-29T00:39:38.411 DEBUG:boto:Params: {} 2023-07-29T00:39:38.412 DEBUG:boto:Token: None 2023-07-29T00:39:38.413 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj55 2023-07-29T00:39:38.414 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ubn82oyBh7l8Ot9aUxK+lP//tQw= 2023-07-29T00:39:38.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ubn82oyBh7l8Ot9aUxK+lP//tQw=', 'Content-Length': '0'} 2023-07-29T00:39:38.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053ed7fd54b1290e6-0064c45fca-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-07-29T00:39:38.418 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:39:38.418 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:39:38.418 DEBUG:boto:Method: GET 2023-07-29T00:39:38.419 DEBUG:boto:Path: /hiiqos-59/obj55 2023-07-29T00:39:38.419 DEBUG:boto:Data: 2023-07-29T00:39:38.419 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.420 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:38.420 DEBUG:boto:Port: 8001 2023-07-29T00:39:38.421 DEBUG:boto:Params: {} 2023-07-29T00:39:38.421 DEBUG:boto:Token: None 2023-07-29T00:39:38.421 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj55 2023-07-29T00:39:38.422 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:STEGd1SRbvh1yP6PextBL/i4RJM= 2023-07-29T00:39:38.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:STEGd1SRbvh1yP6PextBL/i4RJM=', 'Content-Length': '0'} 2023-07-29T00:39:38.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8a4ca6ef33cbfcb-0064c45fca-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.426 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:39:38.426 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:39:38.427 DEBUG:boto:Method: GET 2023-07-29T00:39:38.427 DEBUG:boto:Path: /hiiqos-59/obj55 2023-07-29T00:39:38.428 DEBUG:boto:Data: 2023-07-29T00:39:38.428 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.429 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:38.429 DEBUG:boto:Port: 8000 2023-07-29T00:39:38.430 DEBUG:boto:Params: {} 2023-07-29T00:39:38.430 DEBUG:boto:Token: None 2023-07-29T00:39:38.431 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj55 2023-07-29T00:39:38.431 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:STEGd1SRbvh1yP6PextBL/i4RJM= 2023-07-29T00:39:38.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:STEGd1SRbvh1yP6PextBL/i4RJM=', 'Content-Length': '0'} 2023-07-29T00:39:38.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6a24077b1e787bf-0064c45fca-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.434 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:38.435 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:38.435 DEBUG:boto:path=/hiiqos-59/?prefix=obj55 2023-07-29T00:39:38.435 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj55 2023-07-29T00:39:38.436 DEBUG:boto:Method: GET 2023-07-29T00:39:38.436 DEBUG:boto:Path: /hiiqos-59/?prefix=obj55 2023-07-29T00:39:38.437 DEBUG:boto:Data: 2023-07-29T00:39:38.438 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.438 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:38.439 DEBUG:boto:Port: 8001 2023-07-29T00:39:38.440 DEBUG:boto:Params: {} 2023-07-29T00:39:38.441 DEBUG:boto:Token: None 2023-07-29T00:39:38.442 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/ 2023-07-29T00:39:38.443 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RvKVsO6e+NPVZ6UkJytUnnUWrBE= 2023-07-29T00:39:38.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RvKVsO6e+NPVZ6UkJytUnnUWrBE=', 'Content-Length': '0'} 2023-07-29T00:39:38.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3dac1454e25e8cc-0064c45fca-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.448 DEBUG:boto:b'hiiqos-59obj551000falseobj552023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:38.449 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:38.449 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:38.450 DEBUG:boto:path=/hiiqos-59/?prefix=obj55 2023-07-29T00:39:38.450 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj55 2023-07-29T00:39:38.450 DEBUG:boto:Method: GET 2023-07-29T00:39:38.451 DEBUG:boto:Path: /hiiqos-59/?prefix=obj55 2023-07-29T00:39:38.451 DEBUG:boto:Data: 2023-07-29T00:39:38.451 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.452 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:38.452 DEBUG:boto:Port: 8000 2023-07-29T00:39:38.453 DEBUG:boto:Params: {} 2023-07-29T00:39:38.453 DEBUG:boto:Token: None 2023-07-29T00:39:38.453 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/ 2023-07-29T00:39:38.454 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RvKVsO6e+NPVZ6UkJytUnnUWrBE= 2023-07-29T00:39:38.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RvKVsO6e+NPVZ6UkJytUnnUWrBE=', 'Content-Length': '0'} 2023-07-29T00:39:38.458 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007acb7ed1a0b9e4e7-0064c45fca-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.459 DEBUG:boto:b'hiiqos-59obj551000falseobj552023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:38.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-07-29T00:39:38.460 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:39:38.460 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:39:38.461 DEBUG:boto:path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:39:38.461 DEBUG:boto:auth_path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:39:38.461 DEBUG:boto:Method: GET 2023-07-29T00:39:38.462 DEBUG:boto:Path: /hiiqos-59/obj57?versionId=null 2023-07-29T00:39:38.462 DEBUG:boto:Data: 2023-07-29T00:39:38.462 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.463 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:38.463 DEBUG:boto:Port: 8001 2023-07-29T00:39:38.464 DEBUG:boto:Params: {} 2023-07-29T00:39:38.464 DEBUG:boto:Token: None 2023-07-29T00:39:38.465 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj57?versionId=null 2023-07-29T00:39:38.465 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eAlTEWPDAZk/EMK0QMO2VzWGTT4= 2023-07-29T00:39:38.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eAlTEWPDAZk/EMK0QMO2VzWGTT4=', 'Content-Length': '0'} 2023-07-29T00:39:38.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd09039ec1793ea5-0064c45fca-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.469 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:39:38.469 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:39:38.470 DEBUG:boto:path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:39:38.470 DEBUG:boto:auth_path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:39:38.470 DEBUG:boto:Method: GET 2023-07-29T00:39:38.471 DEBUG:boto:Path: /hiiqos-59/obj57?versionId=null 2023-07-29T00:39:38.471 DEBUG:boto:Data: 2023-07-29T00:39:38.472 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.472 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:38.472 DEBUG:boto:Port: 8000 2023-07-29T00:39:38.473 DEBUG:boto:Params: {} 2023-07-29T00:39:38.473 DEBUG:boto:Token: None 2023-07-29T00:39:38.474 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj57?versionId=null 2023-07-29T00:39:38.474 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eAlTEWPDAZk/EMK0QMO2VzWGTT4= 2023-07-29T00:39:38.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eAlTEWPDAZk/EMK0QMO2VzWGTT4=', 'Content-Length': '0'} 2023-07-29T00:39:38.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006eb3529443195492-0064c45fca-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.520 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:39:38.521 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:39:38.521 DEBUG:boto:path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:39:38.522 DEBUG:boto:auth_path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:39:38.522 DEBUG:boto:Method: HEAD 2023-07-29T00:39:38.523 DEBUG:boto:Path: /hiiqos-59/obj57?versionId=null 2023-07-29T00:39:38.523 DEBUG:boto:Data: 2023-07-29T00:39:38.523 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.524 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:38.524 DEBUG:boto:Port: 8001 2023-07-29T00:39:38.525 DEBUG:boto:Params: {} 2023-07-29T00:39:38.525 DEBUG:boto:Token: None 2023-07-29T00:39:38.525 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj57?versionId=null 2023-07-29T00:39:38.526 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ocgX4/W1IfTKq9/rRci1mO7rq90= 2023-07-29T00:39:38.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ocgX4/W1IfTKq9/rRci1mO7rq90=', 'Content-Length': '0'} 2023-07-29T00:39:38.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b0909cdd0a14ade-0064c45fca-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.529 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:39:38.529 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:39:38.529 DEBUG:boto:path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:39:38.530 DEBUG:boto:auth_path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:39:38.530 DEBUG:boto:Method: HEAD 2023-07-29T00:39:38.530 DEBUG:boto:Path: /hiiqos-59/obj57?versionId=null 2023-07-29T00:39:38.531 DEBUG:boto:Data: 2023-07-29T00:39:38.531 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.531 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:38.532 DEBUG:boto:Port: 8000 2023-07-29T00:39:38.532 DEBUG:boto:Params: {} 2023-07-29T00:39:38.533 DEBUG:boto:Token: None 2023-07-29T00:39:38.533 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj57?versionId=null 2023-07-29T00:39:38.534 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ocgX4/W1IfTKq9/rRci1mO7rq90= 2023-07-29T00:39:38.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ocgX4/W1IfTKq9/rRci1mO7rq90=', 'Content-Length': '0'} 2023-07-29T00:39:38.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe841e5fadbfbef4-0064c45fca-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-07-29T00:39:38.537 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:39:38.538 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:39:38.538 DEBUG:boto:path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:39:38.538 DEBUG:boto:auth_path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:39:38.539 DEBUG:boto:Method: GET 2023-07-29T00:39:38.539 DEBUG:boto:Path: /hiiqos-59/obj57?versionId=null 2023-07-29T00:39:38.540 DEBUG:boto:Data: 2023-07-29T00:39:38.540 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.540 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:38.541 DEBUG:boto:Port: 8001 2023-07-29T00:39:38.541 DEBUG:boto:Params: {} 2023-07-29T00:39:38.542 DEBUG:boto:Token: None 2023-07-29T00:39:38.542 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj57?versionId=null 2023-07-29T00:39:38.542 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eAlTEWPDAZk/EMK0QMO2VzWGTT4= 2023-07-29T00:39:38.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eAlTEWPDAZk/EMK0QMO2VzWGTT4=', 'Content-Length': '0'} 2023-07-29T00:39:38.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d568627c1a99f4e-0064c45fca-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.546 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:39:38.546 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:39:38.547 DEBUG:boto:path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:39:38.547 DEBUG:boto:auth_path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:39:38.547 DEBUG:boto:Method: GET 2023-07-29T00:39:38.548 DEBUG:boto:Path: /hiiqos-59/obj57?versionId=null 2023-07-29T00:39:38.548 DEBUG:boto:Data: 2023-07-29T00:39:38.548 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.549 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:38.549 DEBUG:boto:Port: 8000 2023-07-29T00:39:38.550 DEBUG:boto:Params: {} 2023-07-29T00:39:38.550 DEBUG:boto:Token: None 2023-07-29T00:39:38.550 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj57?versionId=null 2023-07-29T00:39:38.551 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eAlTEWPDAZk/EMK0QMO2VzWGTT4= 2023-07-29T00:39:38.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eAlTEWPDAZk/EMK0QMO2VzWGTT4=', 'Content-Length': '0'} 2023-07-29T00:39:38.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee89819c5116538d-0064c45fca-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.555 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:38.555 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:38.555 DEBUG:boto:path=/hiiqos-59/?prefix=obj57 2023-07-29T00:39:38.556 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj57 2023-07-29T00:39:38.556 DEBUG:boto:Method: GET 2023-07-29T00:39:38.557 DEBUG:boto:Path: /hiiqos-59/?prefix=obj57 2023-07-29T00:39:38.557 DEBUG:boto:Data: 2023-07-29T00:39:38.557 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.558 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:38.558 DEBUG:boto:Port: 8001 2023-07-29T00:39:38.559 DEBUG:boto:Params: {} 2023-07-29T00:39:38.559 DEBUG:boto:Token: None 2023-07-29T00:39:38.559 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/ 2023-07-29T00:39:38.560 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RvKVsO6e+NPVZ6UkJytUnnUWrBE= 2023-07-29T00:39:38.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RvKVsO6e+NPVZ6UkJytUnnUWrBE=', 'Content-Length': '0'} 2023-07-29T00:39:38.563 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f84882c7dfe5f69f-0064c45fca-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.563 DEBUG:boto:b'hiiqos-59obj571000falseobj572023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:38.564 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:38.565 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:38.565 DEBUG:boto:path=/hiiqos-59/?prefix=obj57 2023-07-29T00:39:38.565 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj57 2023-07-29T00:39:38.566 DEBUG:boto:Method: GET 2023-07-29T00:39:38.566 DEBUG:boto:Path: /hiiqos-59/?prefix=obj57 2023-07-29T00:39:38.567 DEBUG:boto:Data: 2023-07-29T00:39:38.567 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.567 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:38.568 DEBUG:boto:Port: 8000 2023-07-29T00:39:38.568 DEBUG:boto:Params: {} 2023-07-29T00:39:38.569 DEBUG:boto:Token: None 2023-07-29T00:39:38.569 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/ 2023-07-29T00:39:38.569 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RvKVsO6e+NPVZ6UkJytUnnUWrBE= 2023-07-29T00:39:38.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RvKVsO6e+NPVZ6UkJytUnnUWrBE=', 'Content-Length': '0'} 2023-07-29T00:39:38.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002770930481c6788d-0064c45fca-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.620 DEBUG:boto:b'hiiqos-59obj571000falseobj572023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:38.621 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:39:38.622 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:39:38.622 DEBUG:boto:Method: HEAD 2023-07-29T00:39:38.622 DEBUG:boto:Path: /hiiqos-59/obj57 2023-07-29T00:39:38.623 DEBUG:boto:Data: 2023-07-29T00:39:38.623 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.623 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:38.624 DEBUG:boto:Port: 8001 2023-07-29T00:39:38.624 DEBUG:boto:Params: {} 2023-07-29T00:39:38.625 DEBUG:boto:Token: None 2023-07-29T00:39:38.625 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj57 2023-07-29T00:39:38.625 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4qlPtWpn6N/Kv8G27q8iFe+g930= 2023-07-29T00:39:38.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4qlPtWpn6N/Kv8G27q8iFe+g930=', 'Content-Length': '0'} 2023-07-29T00:39:38.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb26e85282242c81-0064c45fca-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.628 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:39:38.629 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:39:38.629 DEBUG:boto:Method: HEAD 2023-07-29T00:39:38.629 DEBUG:boto:Path: /hiiqos-59/obj57 2023-07-29T00:39:38.630 DEBUG:boto:Data: 2023-07-29T00:39:38.630 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.630 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:38.631 DEBUG:boto:Port: 8000 2023-07-29T00:39:38.631 DEBUG:boto:Params: {} 2023-07-29T00:39:38.632 DEBUG:boto:Token: None 2023-07-29T00:39:38.632 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj57 2023-07-29T00:39:38.633 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4qlPtWpn6N/Kv8G27q8iFe+g930= 2023-07-29T00:39:38.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4qlPtWpn6N/Kv8G27q8iFe+g930=', 'Content-Length': '0'} 2023-07-29T00:39:38.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006eea503176c63cd2-0064c45fca-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-07-29T00:39:38.636 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:39:38.636 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:39:38.637 DEBUG:boto:Method: GET 2023-07-29T00:39:38.637 DEBUG:boto:Path: /hiiqos-59/obj57 2023-07-29T00:39:38.638 DEBUG:boto:Data: 2023-07-29T00:39:38.638 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.638 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:38.639 DEBUG:boto:Port: 8001 2023-07-29T00:39:38.639 DEBUG:boto:Params: {} 2023-07-29T00:39:38.639 DEBUG:boto:Token: None 2023-07-29T00:39:38.640 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj57 2023-07-29T00:39:38.640 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tcs6C3t5YboFJiTKIMCUax0ImNQ= 2023-07-29T00:39:38.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tcs6C3t5YboFJiTKIMCUax0ImNQ=', 'Content-Length': '0'} 2023-07-29T00:39:38.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc819afa8fc8b03a-0064c45fca-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.644 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:39:38.644 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:39:38.644 DEBUG:boto:Method: GET 2023-07-29T00:39:38.645 DEBUG:boto:Path: /hiiqos-59/obj57 2023-07-29T00:39:38.645 DEBUG:boto:Data: 2023-07-29T00:39:38.646 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.646 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:38.646 DEBUG:boto:Port: 8000 2023-07-29T00:39:38.647 DEBUG:boto:Params: {} 2023-07-29T00:39:38.647 DEBUG:boto:Token: None 2023-07-29T00:39:38.647 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj57 2023-07-29T00:39:38.648 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tcs6C3t5YboFJiTKIMCUax0ImNQ= 2023-07-29T00:39:38.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tcs6C3t5YboFJiTKIMCUax0ImNQ=', 'Content-Length': '0'} 2023-07-29T00:39:38.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daafe23cb7a00d70-0064c45fca-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.651 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:38.651 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:38.652 DEBUG:boto:path=/hiiqos-59/?prefix=obj57 2023-07-29T00:39:38.653 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj57 2023-07-29T00:39:38.653 DEBUG:boto:Method: GET 2023-07-29T00:39:38.654 DEBUG:boto:Path: /hiiqos-59/?prefix=obj57 2023-07-29T00:39:38.654 DEBUG:boto:Data: 2023-07-29T00:39:38.655 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.655 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:38.656 DEBUG:boto:Port: 8001 2023-07-29T00:39:38.656 DEBUG:boto:Params: {} 2023-07-29T00:39:38.656 DEBUG:boto:Token: None 2023-07-29T00:39:38.657 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/ 2023-07-29T00:39:38.657 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RvKVsO6e+NPVZ6UkJytUnnUWrBE= 2023-07-29T00:39:38.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RvKVsO6e+NPVZ6UkJytUnnUWrBE=', 'Content-Length': '0'} 2023-07-29T00:39:38.660 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000044bc37d02efbf30-0064c45fca-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.661 DEBUG:boto:b'hiiqos-59obj571000falseobj572023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:38.661 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:38.662 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:38.662 DEBUG:boto:path=/hiiqos-59/?prefix=obj57 2023-07-29T00:39:38.662 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj57 2023-07-29T00:39:38.663 DEBUG:boto:Method: GET 2023-07-29T00:39:38.663 DEBUG:boto:Path: /hiiqos-59/?prefix=obj57 2023-07-29T00:39:38.663 DEBUG:boto:Data: 2023-07-29T00:39:38.664 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.665 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:38.665 DEBUG:boto:Port: 8000 2023-07-29T00:39:38.666 DEBUG:boto:Params: {} 2023-07-29T00:39:38.667 DEBUG:boto:Token: None 2023-07-29T00:39:38.668 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/ 2023-07-29T00:39:38.668 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RvKVsO6e+NPVZ6UkJytUnnUWrBE= 2023-07-29T00:39:38.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RvKVsO6e+NPVZ6UkJytUnnUWrBE=', 'Content-Length': '0'} 2023-07-29T00:39:38.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077b2028cb0c4a6c7-0064c45fca-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.675 DEBUG:boto:b'hiiqos-59obj571000falseobj572023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:38.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-07-29T00:39:38.677 DEBUG:boto:path=/hiiqos-59/obj58 2023-07-29T00:39:38.678 DEBUG:boto:auth_path=/hiiqos-59/obj58 2023-07-29T00:39:38.679 DEBUG:boto:path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:39:38.680 DEBUG:boto:auth_path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:39:38.680 DEBUG:boto:Method: GET 2023-07-29T00:39:38.681 DEBUG:boto:Path: /hiiqos-59/obj58?versionId=null 2023-07-29T00:39:38.681 DEBUG:boto:Data: 2023-07-29T00:39:38.682 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.682 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:38.682 DEBUG:boto:Port: 8001 2023-07-29T00:39:38.683 DEBUG:boto:Params: {} 2023-07-29T00:39:38.683 DEBUG:boto:Token: None 2023-07-29T00:39:38.684 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj58?versionId=null 2023-07-29T00:39:38.684 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:azMkBezRp/9Duw1kaQ57VU3x+lQ= 2023-07-29T00:39:38.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:azMkBezRp/9Duw1kaQ57VU3x+lQ=', 'Content-Length': '0'} 2023-07-29T00:39:38.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9890c4822e91fe6-0064c45fca-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.687 DEBUG:boto:path=/hiiqos-59/obj58 2023-07-29T00:39:38.687 DEBUG:boto:auth_path=/hiiqos-59/obj58 2023-07-29T00:39:38.688 DEBUG:boto:path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:39:38.688 DEBUG:boto:auth_path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:39:38.689 DEBUG:boto:Method: GET 2023-07-29T00:39:38.689 DEBUG:boto:Path: /hiiqos-59/obj58?versionId=null 2023-07-29T00:39:38.689 DEBUG:boto:Data: 2023-07-29T00:39:38.690 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.690 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:38.691 DEBUG:boto:Port: 8000 2023-07-29T00:39:38.692 DEBUG:boto:Params: {} 2023-07-29T00:39:38.692 DEBUG:boto:Token: None 2023-07-29T00:39:38.693 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj58?versionId=null 2023-07-29T00:39:38.693 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:azMkBezRp/9Duw1kaQ57VU3x+lQ= 2023-07-29T00:39:38.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:azMkBezRp/9Duw1kaQ57VU3x+lQ=', 'Content-Length': '0'} 2023-07-29T00:39:38.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0509fa018ffb5e1-0064c45fca-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.697 DEBUG:boto:path=/hiiqos-59/obj58 2023-07-29T00:39:38.697 DEBUG:boto:auth_path=/hiiqos-59/obj58 2023-07-29T00:39:38.698 DEBUG:boto:path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:39:38.698 DEBUG:boto:auth_path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:39:38.698 DEBUG:boto:Method: HEAD 2023-07-29T00:39:38.699 DEBUG:boto:Path: /hiiqos-59/obj58?versionId=null 2023-07-29T00:39:38.699 DEBUG:boto:Data: 2023-07-29T00:39:38.700 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.700 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:38.700 DEBUG:boto:Port: 8001 2023-07-29T00:39:38.701 DEBUG:boto:Params: {} 2023-07-29T00:39:38.701 DEBUG:boto:Token: None 2023-07-29T00:39:38.702 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj58?versionId=null 2023-07-29T00:39:38.702 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bobdOnCbmeHIdG/KWFebWw/Zm9E= 2023-07-29T00:39:38.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bobdOnCbmeHIdG/KWFebWw/Zm9E=', 'Content-Length': '0'} 2023-07-29T00:39:38.705 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034ad1954ce008ae6-0064c45fca-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.705 DEBUG:boto:path=/hiiqos-59/obj58 2023-07-29T00:39:38.706 DEBUG:boto:auth_path=/hiiqos-59/obj58 2023-07-29T00:39:38.706 DEBUG:boto:path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:39:38.707 DEBUG:boto:auth_path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:39:38.708 DEBUG:boto:Method: HEAD 2023-07-29T00:39:38.708 DEBUG:boto:Path: /hiiqos-59/obj58?versionId=null 2023-07-29T00:39:38.709 DEBUG:boto:Data: 2023-07-29T00:39:38.710 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.711 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:38.711 DEBUG:boto:Port: 8000 2023-07-29T00:39:38.712 DEBUG:boto:Params: {} 2023-07-29T00:39:38.712 DEBUG:boto:Token: None 2023-07-29T00:39:38.713 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj58?versionId=null 2023-07-29T00:39:38.713 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bobdOnCbmeHIdG/KWFebWw/Zm9E= 2023-07-29T00:39:38.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bobdOnCbmeHIdG/KWFebWw/Zm9E=', 'Content-Length': '0'} 2023-07-29T00:39:38.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0179740d93a6b8b-0064c45fca-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-07-29T00:39:38.716 DEBUG:boto:path=/hiiqos-59/obj58 2023-07-29T00:39:38.717 DEBUG:boto:auth_path=/hiiqos-59/obj58 2023-07-29T00:39:38.717 DEBUG:boto:path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:39:38.717 DEBUG:boto:auth_path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:39:38.718 DEBUG:boto:Method: GET 2023-07-29T00:39:38.718 DEBUG:boto:Path: /hiiqos-59/obj58?versionId=null 2023-07-29T00:39:38.718 DEBUG:boto:Data: 2023-07-29T00:39:38.719 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.719 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:38.719 DEBUG:boto:Port: 8001 2023-07-29T00:39:38.720 DEBUG:boto:Params: {} 2023-07-29T00:39:38.720 DEBUG:boto:Token: None 2023-07-29T00:39:38.721 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj58?versionId=null 2023-07-29T00:39:38.721 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:azMkBezRp/9Duw1kaQ57VU3x+lQ= 2023-07-29T00:39:38.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:azMkBezRp/9Duw1kaQ57VU3x+lQ=', 'Content-Length': '0'} 2023-07-29T00:39:38.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000855af39c16f5936c-0064c45fca-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.764 DEBUG:boto:path=/hiiqos-59/obj58 2023-07-29T00:39:38.765 DEBUG:boto:auth_path=/hiiqos-59/obj58 2023-07-29T00:39:38.765 DEBUG:boto:path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:39:38.766 DEBUG:boto:auth_path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:39:38.766 DEBUG:boto:Method: GET 2023-07-29T00:39:38.766 DEBUG:boto:Path: /hiiqos-59/obj58?versionId=null 2023-07-29T00:39:38.767 DEBUG:boto:Data: 2023-07-29T00:39:38.767 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.767 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:38.768 DEBUG:boto:Port: 8000 2023-07-29T00:39:38.768 DEBUG:boto:Params: {} 2023-07-29T00:39:38.769 DEBUG:boto:Token: None 2023-07-29T00:39:38.769 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj58?versionId=null 2023-07-29T00:39:38.769 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:azMkBezRp/9Duw1kaQ57VU3x+lQ= 2023-07-29T00:39:38.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:azMkBezRp/9Duw1kaQ57VU3x+lQ=', 'Content-Length': '0'} 2023-07-29T00:39:38.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006038dd5669b4a7cc-0064c45fca-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.773 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:38.773 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:38.774 DEBUG:boto:path=/hiiqos-59/?prefix=obj58 2023-07-29T00:39:38.774 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj58 2023-07-29T00:39:38.774 DEBUG:boto:Method: GET 2023-07-29T00:39:38.775 DEBUG:boto:Path: /hiiqos-59/?prefix=obj58 2023-07-29T00:39:38.775 DEBUG:boto:Data: 2023-07-29T00:39:38.775 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.776 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:38.776 DEBUG:boto:Port: 8001 2023-07-29T00:39:38.777 DEBUG:boto:Params: {} 2023-07-29T00:39:38.777 DEBUG:boto:Token: None 2023-07-29T00:39:38.777 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/ 2023-07-29T00:39:38.778 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RvKVsO6e+NPVZ6UkJytUnnUWrBE= 2023-07-29T00:39:38.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RvKVsO6e+NPVZ6UkJytUnnUWrBE=', 'Content-Length': '0'} 2023-07-29T00:39:38.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c7ac91570667c5a-0064c45fca-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.782 DEBUG:boto:b'hiiqos-59obj581000falseobj582023-07-29T00:35:01.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:38.782 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:38.783 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:38.783 DEBUG:boto:path=/hiiqos-59/?prefix=obj58 2023-07-29T00:39:38.783 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj58 2023-07-29T00:39:38.784 DEBUG:boto:Method: GET 2023-07-29T00:39:38.784 DEBUG:boto:Path: /hiiqos-59/?prefix=obj58 2023-07-29T00:39:38.784 DEBUG:boto:Data: 2023-07-29T00:39:38.785 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.785 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:38.785 DEBUG:boto:Port: 8000 2023-07-29T00:39:38.786 DEBUG:boto:Params: {} 2023-07-29T00:39:38.786 DEBUG:boto:Token: None 2023-07-29T00:39:38.787 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/ 2023-07-29T00:39:38.787 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RvKVsO6e+NPVZ6UkJytUnnUWrBE= 2023-07-29T00:39:38.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RvKVsO6e+NPVZ6UkJytUnnUWrBE=', 'Content-Length': '0'} 2023-07-29T00:39:38.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae36cd96e70e7289-0064c45fca-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.793 DEBUG:boto:b'hiiqos-59obj581000falseobj582023-07-29T00:35:01.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:38.793 DEBUG:boto:path=/hiiqos-59/obj58 2023-07-29T00:39:38.794 DEBUG:boto:auth_path=/hiiqos-59/obj58 2023-07-29T00:39:38.794 DEBUG:boto:Method: HEAD 2023-07-29T00:39:38.794 DEBUG:boto:Path: /hiiqos-59/obj58 2023-07-29T00:39:38.795 DEBUG:boto:Data: 2023-07-29T00:39:38.795 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.795 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:38.796 DEBUG:boto:Port: 8001 2023-07-29T00:39:38.796 DEBUG:boto:Params: {} 2023-07-29T00:39:38.796 DEBUG:boto:Token: None 2023-07-29T00:39:38.797 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj58 2023-07-29T00:39:38.797 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eye1JV/rljpTkse9fTHXkqKSWr0= 2023-07-29T00:39:38.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eye1JV/rljpTkse9fTHXkqKSWr0=', 'Content-Length': '0'} 2023-07-29T00:39:38.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000418ffee2a30eded1-0064c45fca-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.800 DEBUG:boto:path=/hiiqos-59/obj58 2023-07-29T00:39:38.801 DEBUG:boto:auth_path=/hiiqos-59/obj58 2023-07-29T00:39:38.801 DEBUG:boto:Method: HEAD 2023-07-29T00:39:38.801 DEBUG:boto:Path: /hiiqos-59/obj58 2023-07-29T00:39:38.802 DEBUG:boto:Data: 2023-07-29T00:39:38.802 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.802 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:38.803 DEBUG:boto:Port: 8000 2023-07-29T00:39:38.803 DEBUG:boto:Params: {} 2023-07-29T00:39:38.803 DEBUG:boto:Token: None 2023-07-29T00:39:38.804 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj58 2023-07-29T00:39:38.804 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eye1JV/rljpTkse9fTHXkqKSWr0= 2023-07-29T00:39:38.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eye1JV/rljpTkse9fTHXkqKSWr0=', 'Content-Length': '0'} 2023-07-29T00:39:38.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007133624104236625-0064c45fca-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-07-29T00:39:38.807 DEBUG:boto:path=/hiiqos-59/obj58 2023-07-29T00:39:38.808 DEBUG:boto:auth_path=/hiiqos-59/obj58 2023-07-29T00:39:38.808 DEBUG:boto:Method: GET 2023-07-29T00:39:38.808 DEBUG:boto:Path: /hiiqos-59/obj58 2023-07-29T00:39:38.809 DEBUG:boto:Data: 2023-07-29T00:39:38.809 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.809 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:38.810 DEBUG:boto:Port: 8001 2023-07-29T00:39:38.810 DEBUG:boto:Params: {} 2023-07-29T00:39:38.811 DEBUG:boto:Token: None 2023-07-29T00:39:38.811 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj58 2023-07-29T00:39:38.811 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:72C1nkccV4kEnpHW7v6rDF/wGBI= 2023-07-29T00:39:38.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:72C1nkccV4kEnpHW7v6rDF/wGBI=', 'Content-Length': '0'} 2023-07-29T00:39:38.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004dc6e5f043299dbf-0064c45fca-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.815 DEBUG:boto:path=/hiiqos-59/obj58 2023-07-29T00:39:38.816 DEBUG:boto:auth_path=/hiiqos-59/obj58 2023-07-29T00:39:38.817 DEBUG:boto:Method: GET 2023-07-29T00:39:38.818 DEBUG:boto:Path: /hiiqos-59/obj58 2023-07-29T00:39:38.818 DEBUG:boto:Data: 2023-07-29T00:39:38.819 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.820 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:38.820 DEBUG:boto:Port: 8000 2023-07-29T00:39:38.821 DEBUG:boto:Params: {} 2023-07-29T00:39:38.821 DEBUG:boto:Token: None 2023-07-29T00:39:38.822 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj58 2023-07-29T00:39:38.823 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:72C1nkccV4kEnpHW7v6rDF/wGBI= 2023-07-29T00:39:38.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:72C1nkccV4kEnpHW7v6rDF/wGBI=', 'Content-Length': '0'} 2023-07-29T00:39:38.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000993e1a8a938af443-0064c45fca-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.868 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:38.869 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:38.869 DEBUG:boto:path=/hiiqos-59/?prefix=obj58 2023-07-29T00:39:38.870 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj58 2023-07-29T00:39:38.870 DEBUG:boto:Method: GET 2023-07-29T00:39:38.870 DEBUG:boto:Path: /hiiqos-59/?prefix=obj58 2023-07-29T00:39:38.871 DEBUG:boto:Data: 2023-07-29T00:39:38.871 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.871 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:38.872 DEBUG:boto:Port: 8001 2023-07-29T00:39:38.872 DEBUG:boto:Params: {} 2023-07-29T00:39:38.873 DEBUG:boto:Token: None 2023-07-29T00:39:38.873 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/ 2023-07-29T00:39:38.874 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RvKVsO6e+NPVZ6UkJytUnnUWrBE= 2023-07-29T00:39:38.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RvKVsO6e+NPVZ6UkJytUnnUWrBE=', 'Content-Length': '0'} 2023-07-29T00:39:38.876 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7f027148f6d9349-0064c45fca-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.877 DEBUG:boto:b'hiiqos-59obj581000falseobj582023-07-29T00:35:01.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:38.878 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:38.878 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:38.878 DEBUG:boto:path=/hiiqos-59/?prefix=obj58 2023-07-29T00:39:38.879 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj58 2023-07-29T00:39:38.879 DEBUG:boto:Method: GET 2023-07-29T00:39:38.879 DEBUG:boto:Path: /hiiqos-59/?prefix=obj58 2023-07-29T00:39:38.880 DEBUG:boto:Data: 2023-07-29T00:39:38.880 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.881 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:38.881 DEBUG:boto:Port: 8000 2023-07-29T00:39:38.881 DEBUG:boto:Params: {} 2023-07-29T00:39:38.882 DEBUG:boto:Token: None 2023-07-29T00:39:38.882 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/ 2023-07-29T00:39:38.882 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RvKVsO6e+NPVZ6UkJytUnnUWrBE= 2023-07-29T00:39:38.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RvKVsO6e+NPVZ6UkJytUnnUWrBE=', 'Content-Length': '0'} 2023-07-29T00:39:38.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2a91e890f86a93b-0064c45fca-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.887 DEBUG:boto:b'hiiqos-59obj581000falseobj582023-07-29T00:35:01.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:38.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-07-29T00:39:38.888 DEBUG:boto:path=/hiiqos-59/obj62 2023-07-29T00:39:38.889 DEBUG:boto:auth_path=/hiiqos-59/obj62 2023-07-29T00:39:38.889 DEBUG:boto:path=/hiiqos-59/obj62?versionId=null 2023-07-29T00:39:38.889 DEBUG:boto:auth_path=/hiiqos-59/obj62?versionId=null 2023-07-29T00:39:38.890 DEBUG:boto:Method: GET 2023-07-29T00:39:38.890 DEBUG:boto:Path: /hiiqos-59/obj62?versionId=null 2023-07-29T00:39:38.891 DEBUG:boto:Data: 2023-07-29T00:39:38.891 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.891 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:38.892 DEBUG:boto:Port: 8001 2023-07-29T00:39:38.892 DEBUG:boto:Params: {} 2023-07-29T00:39:38.892 DEBUG:boto:Token: None 2023-07-29T00:39:38.893 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj62?versionId=null 2023-07-29T00:39:38.893 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mk0XpeoMVsqVtTcRmBScHvKXH+E= 2023-07-29T00:39:38.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mk0XpeoMVsqVtTcRmBScHvKXH+E=', 'Content-Length': '0'} 2023-07-29T00:39:38.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000124e751d709c4774-0064c45fca-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.936 DEBUG:boto:path=/hiiqos-59/obj62 2023-07-29T00:39:38.937 DEBUG:boto:auth_path=/hiiqos-59/obj62 2023-07-29T00:39:38.937 DEBUG:boto:path=/hiiqos-59/obj62?versionId=null 2023-07-29T00:39:38.938 DEBUG:boto:auth_path=/hiiqos-59/obj62?versionId=null 2023-07-29T00:39:38.938 DEBUG:boto:Method: GET 2023-07-29T00:39:38.938 DEBUG:boto:Path: /hiiqos-59/obj62?versionId=null 2023-07-29T00:39:38.939 DEBUG:boto:Data: 2023-07-29T00:39:38.939 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.939 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:38.940 DEBUG:boto:Port: 8000 2023-07-29T00:39:38.940 DEBUG:boto:Params: {} 2023-07-29T00:39:38.941 DEBUG:boto:Token: None 2023-07-29T00:39:38.941 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj62?versionId=null 2023-07-29T00:39:38.942 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mk0XpeoMVsqVtTcRmBScHvKXH+E= 2023-07-29T00:39:38.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mk0XpeoMVsqVtTcRmBScHvKXH+E=', 'Content-Length': '0'} 2023-07-29T00:39:38.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000306a565568eb6a2c-0064c45fca-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.944 DEBUG:boto:path=/hiiqos-59/obj62 2023-07-29T00:39:38.945 DEBUG:boto:auth_path=/hiiqos-59/obj62 2023-07-29T00:39:38.945 DEBUG:boto:path=/hiiqos-59/obj62?versionId=null 2023-07-29T00:39:38.945 DEBUG:boto:auth_path=/hiiqos-59/obj62?versionId=null 2023-07-29T00:39:38.946 DEBUG:boto:Method: HEAD 2023-07-29T00:39:38.946 DEBUG:boto:Path: /hiiqos-59/obj62?versionId=null 2023-07-29T00:39:38.946 DEBUG:boto:Data: 2023-07-29T00:39:38.947 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.947 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:38.947 DEBUG:boto:Port: 8001 2023-07-29T00:39:38.948 DEBUG:boto:Params: {} 2023-07-29T00:39:38.948 DEBUG:boto:Token: None 2023-07-29T00:39:38.949 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj62?versionId=null 2023-07-29T00:39:38.949 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:iuyOjW0MEZJpFLRATerpLzjxwm0= 2023-07-29T00:39:38.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:iuyOjW0MEZJpFLRATerpLzjxwm0=', 'Content-Length': '0'} 2023-07-29T00:39:38.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018ed5dce2b9e2469-0064c45fca-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.952 DEBUG:boto:path=/hiiqos-59/obj62 2023-07-29T00:39:38.952 DEBUG:boto:auth_path=/hiiqos-59/obj62 2023-07-29T00:39:38.953 DEBUG:boto:path=/hiiqos-59/obj62?versionId=null 2023-07-29T00:39:38.953 DEBUG:boto:auth_path=/hiiqos-59/obj62?versionId=null 2023-07-29T00:39:38.953 DEBUG:boto:Method: HEAD 2023-07-29T00:39:38.954 DEBUG:boto:Path: /hiiqos-59/obj62?versionId=null 2023-07-29T00:39:38.954 DEBUG:boto:Data: 2023-07-29T00:39:38.955 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.955 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:38.955 DEBUG:boto:Port: 8000 2023-07-29T00:39:38.956 DEBUG:boto:Params: {} 2023-07-29T00:39:38.956 DEBUG:boto:Token: None 2023-07-29T00:39:38.957 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj62?versionId=null 2023-07-29T00:39:38.957 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:iuyOjW0MEZJpFLRATerpLzjxwm0= 2023-07-29T00:39:38.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:iuyOjW0MEZJpFLRATerpLzjxwm0=', 'Content-Length': '0'} 2023-07-29T00:39:38.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000255ee8e75e9e3166-0064c45fca-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-07-29T00:39:38.960 DEBUG:boto:path=/hiiqos-59/obj62 2023-07-29T00:39:38.961 DEBUG:boto:auth_path=/hiiqos-59/obj62 2023-07-29T00:39:38.961 DEBUG:boto:path=/hiiqos-59/obj62?versionId=null 2023-07-29T00:39:38.962 DEBUG:boto:auth_path=/hiiqos-59/obj62?versionId=null 2023-07-29T00:39:38.962 DEBUG:boto:Method: GET 2023-07-29T00:39:38.962 DEBUG:boto:Path: /hiiqos-59/obj62?versionId=null 2023-07-29T00:39:38.963 DEBUG:boto:Data: 2023-07-29T00:39:38.963 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.964 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:38.964 DEBUG:boto:Port: 8001 2023-07-29T00:39:38.965 DEBUG:boto:Params: {} 2023-07-29T00:39:38.965 DEBUG:boto:Token: None 2023-07-29T00:39:38.966 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj62?versionId=null 2023-07-29T00:39:38.966 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mk0XpeoMVsqVtTcRmBScHvKXH+E= 2023-07-29T00:39:38.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mk0XpeoMVsqVtTcRmBScHvKXH+E=', 'Content-Length': '0'} 2023-07-29T00:39:38.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005e6e71c5d0269cb-0064c45fca-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.969 DEBUG:boto:path=/hiiqos-59/obj62 2023-07-29T00:39:38.970 DEBUG:boto:auth_path=/hiiqos-59/obj62 2023-07-29T00:39:38.970 DEBUG:boto:path=/hiiqos-59/obj62?versionId=null 2023-07-29T00:39:38.971 DEBUG:boto:auth_path=/hiiqos-59/obj62?versionId=null 2023-07-29T00:39:38.971 DEBUG:boto:Method: GET 2023-07-29T00:39:38.972 DEBUG:boto:Path: /hiiqos-59/obj62?versionId=null 2023-07-29T00:39:38.972 DEBUG:boto:Data: 2023-07-29T00:39:38.972 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.973 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:38.973 DEBUG:boto:Port: 8000 2023-07-29T00:39:38.974 DEBUG:boto:Params: {} 2023-07-29T00:39:38.974 DEBUG:boto:Token: None 2023-07-29T00:39:38.975 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/obj62?versionId=null 2023-07-29T00:39:38.975 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mk0XpeoMVsqVtTcRmBScHvKXH+E= 2023-07-29T00:39:38.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mk0XpeoMVsqVtTcRmBScHvKXH+E=', 'Content-Length': '0'} 2023-07-29T00:39:38.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079ec4c84a977f2c0-0064c45fca-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.980 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:38.981 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:38.981 DEBUG:boto:path=/hiiqos-59/?prefix=obj62 2023-07-29T00:39:38.981 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj62 2023-07-29T00:39:38.982 DEBUG:boto:Method: GET 2023-07-29T00:39:38.982 DEBUG:boto:Path: /hiiqos-59/?prefix=obj62 2023-07-29T00:39:38.983 DEBUG:boto:Data: 2023-07-29T00:39:38.983 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.983 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:38.984 DEBUG:boto:Port: 8001 2023-07-29T00:39:38.984 DEBUG:boto:Params: {} 2023-07-29T00:39:38.985 DEBUG:boto:Token: None 2023-07-29T00:39:38.985 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/ 2023-07-29T00:39:38.986 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RvKVsO6e+NPVZ6UkJytUnnUWrBE= 2023-07-29T00:39:38.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RvKVsO6e+NPVZ6UkJytUnnUWrBE=', 'Content-Length': '0'} 2023-07-29T00:39:38.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000eebd595fe77d050-0064c45fca-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:38.989 DEBUG:boto:b'hiiqos-59obj621000falseobj622023-07-29T00:35:31.826Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:38.990 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:38.990 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:38.990 DEBUG:boto:path=/hiiqos-59/?prefix=obj62 2023-07-29T00:39:38.991 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj62 2023-07-29T00:39:38.992 DEBUG:boto:Method: GET 2023-07-29T00:39:38.992 DEBUG:boto:Path: /hiiqos-59/?prefix=obj62 2023-07-29T00:39:38.992 DEBUG:boto:Data: 2023-07-29T00:39:38.993 DEBUG:boto:Headers: {} 2023-07-29T00:39:38.993 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:38.993 DEBUG:boto:Port: 8000 2023-07-29T00:39:38.994 DEBUG:boto:Params: {} 2023-07-29T00:39:38.994 DEBUG:boto:Token: None 2023-07-29T00:39:38.995 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:38 GMT /hiiqos-59/ 2023-07-29T00:39:38.995 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RvKVsO6e+NPVZ6UkJytUnnUWrBE= 2023-07-29T00:39:38.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RvKVsO6e+NPVZ6UkJytUnnUWrBE=', 'Content-Length': '0'} 2023-07-29T00:39:39.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000414c7c2a83a6ed5c-0064c45fca-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.001 DEBUG:boto:b'hiiqos-59obj621000falseobj622023-07-29T00:35:31.826Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:39.001 DEBUG:boto:path=/hiiqos-59/obj62 2023-07-29T00:39:39.002 DEBUG:boto:auth_path=/hiiqos-59/obj62 2023-07-29T00:39:39.002 DEBUG:boto:Method: HEAD 2023-07-29T00:39:39.003 DEBUG:boto:Path: /hiiqos-59/obj62 2023-07-29T00:39:39.003 DEBUG:boto:Data: 2023-07-29T00:39:39.003 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.004 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:39.004 DEBUG:boto:Port: 8001 2023-07-29T00:39:39.004 DEBUG:boto:Params: {} 2023-07-29T00:39:39.005 DEBUG:boto:Token: None 2023-07-29T00:39:39.005 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj62 2023-07-29T00:39:39.006 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Q7w9E0MWXuWNHmB6W39eUJHidA4= 2023-07-29T00:39:39.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q7w9E0MWXuWNHmB6W39eUJHidA4=', 'Content-Length': '0'} 2023-07-29T00:39:39.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c9e53c9aaa5a9ee-0064c45fca-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.008 DEBUG:boto:path=/hiiqos-59/obj62 2023-07-29T00:39:39.009 DEBUG:boto:auth_path=/hiiqos-59/obj62 2023-07-29T00:39:39.009 DEBUG:boto:Method: HEAD 2023-07-29T00:39:39.010 DEBUG:boto:Path: /hiiqos-59/obj62 2023-07-29T00:39:39.010 DEBUG:boto:Data: 2023-07-29T00:39:39.011 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.011 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:39.011 DEBUG:boto:Port: 8000 2023-07-29T00:39:39.012 DEBUG:boto:Params: {} 2023-07-29T00:39:39.012 DEBUG:boto:Token: None 2023-07-29T00:39:39.012 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj62 2023-07-29T00:39:39.013 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Q7w9E0MWXuWNHmB6W39eUJHidA4= 2023-07-29T00:39:39.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q7w9E0MWXuWNHmB6W39eUJHidA4=', 'Content-Length': '0'} 2023-07-29T00:39:39.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa74cb7029e1330f-0064c45fca-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-07-29T00:39:39.017 DEBUG:boto:path=/hiiqos-59/obj62 2023-07-29T00:39:39.018 DEBUG:boto:auth_path=/hiiqos-59/obj62 2023-07-29T00:39:39.019 DEBUG:boto:Method: GET 2023-07-29T00:39:39.020 DEBUG:boto:Path: /hiiqos-59/obj62 2023-07-29T00:39:39.021 DEBUG:boto:Data: 2023-07-29T00:39:39.021 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.022 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:39.022 DEBUG:boto:Port: 8001 2023-07-29T00:39:39.022 DEBUG:boto:Params: {} 2023-07-29T00:39:39.023 DEBUG:boto:Token: None 2023-07-29T00:39:39.024 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj62 2023-07-29T00:39:39.024 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SMsysUI9ASeQLaHl4/jy4xHKhHo= 2023-07-29T00:39:39.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SMsysUI9ASeQLaHl4/jy4xHKhHo=', 'Content-Length': '0'} 2023-07-29T00:39:39.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009901f6a9bbccb3a1-0064c45fca-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:38 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.068 DEBUG:boto:path=/hiiqos-59/obj62 2023-07-29T00:39:39.069 DEBUG:boto:auth_path=/hiiqos-59/obj62 2023-07-29T00:39:39.069 DEBUG:boto:Method: GET 2023-07-29T00:39:39.070 DEBUG:boto:Path: /hiiqos-59/obj62 2023-07-29T00:39:39.070 DEBUG:boto:Data: 2023-07-29T00:39:39.071 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.071 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:39.072 DEBUG:boto:Port: 8000 2023-07-29T00:39:39.072 DEBUG:boto:Params: {} 2023-07-29T00:39:39.073 DEBUG:boto:Token: None 2023-07-29T00:39:39.073 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj62 2023-07-29T00:39:39.074 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SMsysUI9ASeQLaHl4/jy4xHKhHo= 2023-07-29T00:39:39.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SMsysUI9ASeQLaHl4/jy4xHKhHo=', 'Content-Length': '0'} 2023-07-29T00:39:39.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d2f80bb3e9d9579-0064c45fcb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.076 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:39.077 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:39.077 DEBUG:boto:path=/hiiqos-59/?prefix=obj62 2023-07-29T00:39:39.078 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj62 2023-07-29T00:39:39.078 DEBUG:boto:Method: GET 2023-07-29T00:39:39.078 DEBUG:boto:Path: /hiiqos-59/?prefix=obj62 2023-07-29T00:39:39.079 DEBUG:boto:Data: 2023-07-29T00:39:39.079 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.080 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:39.080 DEBUG:boto:Port: 8001 2023-07-29T00:39:39.080 DEBUG:boto:Params: {} 2023-07-29T00:39:39.081 DEBUG:boto:Token: None 2023-07-29T00:39:39.082 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/ 2023-07-29T00:39:39.083 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qK1sfPK+p7ymNycEAB5Yr1COdqw= 2023-07-29T00:39:39.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qK1sfPK+p7ymNycEAB5Yr1COdqw=', 'Content-Length': '0'} 2023-07-29T00:39:39.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e8a13053ba61389-0064c45fcb-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.088 DEBUG:boto:b'hiiqos-59obj621000falseobj622023-07-29T00:35:31.826Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:39.089 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:39.090 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:39.091 DEBUG:boto:path=/hiiqos-59/?prefix=obj62 2023-07-29T00:39:39.091 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj62 2023-07-29T00:39:39.092 DEBUG:boto:Method: GET 2023-07-29T00:39:39.092 DEBUG:boto:Path: /hiiqos-59/?prefix=obj62 2023-07-29T00:39:39.093 DEBUG:boto:Data: 2023-07-29T00:39:39.093 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.093 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:39.094 DEBUG:boto:Port: 8000 2023-07-29T00:39:39.094 DEBUG:boto:Params: {} 2023-07-29T00:39:39.095 DEBUG:boto:Token: None 2023-07-29T00:39:39.095 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/ 2023-07-29T00:39:39.095 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qK1sfPK+p7ymNycEAB5Yr1COdqw= 2023-07-29T00:39:39.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qK1sfPK+p7ymNycEAB5Yr1COdqw=', 'Content-Length': '0'} 2023-07-29T00:39:39.100 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000552c6ee6db99a1bc-0064c45fcb-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.101 DEBUG:boto:b'hiiqos-59obj621000falseobj622023-07-29T00:35:31.826Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:39.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-07-29T00:39:39.102 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:39:39.102 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:39:39.102 DEBUG:boto:path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:39:39.103 DEBUG:boto:auth_path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:39:39.103 DEBUG:boto:Method: GET 2023-07-29T00:39:39.103 DEBUG:boto:Path: /hiiqos-59/obj68?versionId=null 2023-07-29T00:39:39.104 DEBUG:boto:Data: 2023-07-29T00:39:39.104 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.105 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:39.105 DEBUG:boto:Port: 8001 2023-07-29T00:39:39.106 DEBUG:boto:Params: {} 2023-07-29T00:39:39.106 DEBUG:boto:Token: None 2023-07-29T00:39:39.107 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj68?versionId=null 2023-07-29T00:39:39.108 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:e+NvXxNnm968AeXDdxrFplF/96I= 2023-07-29T00:39:39.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:e+NvXxNnm968AeXDdxrFplF/96I=', 'Content-Length': '0'} 2023-07-29T00:39:39.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a75daac34f94138-0064c45fcb-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.113 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:39:39.114 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:39:39.114 DEBUG:boto:path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:39:39.115 DEBUG:boto:auth_path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:39:39.116 DEBUG:boto:Method: GET 2023-07-29T00:39:39.116 DEBUG:boto:Path: /hiiqos-59/obj68?versionId=null 2023-07-29T00:39:39.117 DEBUG:boto:Data: 2023-07-29T00:39:39.118 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.118 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:39.118 DEBUG:boto:Port: 8000 2023-07-29T00:39:39.119 DEBUG:boto:Params: {} 2023-07-29T00:39:39.119 DEBUG:boto:Token: None 2023-07-29T00:39:39.120 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj68?versionId=null 2023-07-29T00:39:39.121 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:e+NvXxNnm968AeXDdxrFplF/96I= 2023-07-29T00:39:39.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:e+NvXxNnm968AeXDdxrFplF/96I=', 'Content-Length': '0'} 2023-07-29T00:39:39.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002867667d97568584-0064c45fcb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.168 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:39:39.169 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:39:39.169 DEBUG:boto:path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:39:39.170 DEBUG:boto:auth_path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:39:39.170 DEBUG:boto:Method: HEAD 2023-07-29T00:39:39.170 DEBUG:boto:Path: /hiiqos-59/obj68?versionId=null 2023-07-29T00:39:39.171 DEBUG:boto:Data: 2023-07-29T00:39:39.171 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.171 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:39.172 DEBUG:boto:Port: 8001 2023-07-29T00:39:39.172 DEBUG:boto:Params: {} 2023-07-29T00:39:39.172 DEBUG:boto:Token: None 2023-07-29T00:39:39.173 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj68?versionId=null 2023-07-29T00:39:39.173 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+/kTMib14CvbJYygGFCKIqrjkjY= 2023-07-29T00:39:39.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+/kTMib14CvbJYygGFCKIqrjkjY=', 'Content-Length': '0'} 2023-07-29T00:39:39.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d5195c179391738-0064c45fcb-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.176 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:39:39.177 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:39:39.177 DEBUG:boto:path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:39:39.178 DEBUG:boto:auth_path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:39:39.178 DEBUG:boto:Method: HEAD 2023-07-29T00:39:39.179 DEBUG:boto:Path: /hiiqos-59/obj68?versionId=null 2023-07-29T00:39:39.179 DEBUG:boto:Data: 2023-07-29T00:39:39.180 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.180 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:39.181 DEBUG:boto:Port: 8000 2023-07-29T00:39:39.181 DEBUG:boto:Params: {} 2023-07-29T00:39:39.182 DEBUG:boto:Token: None 2023-07-29T00:39:39.182 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj68?versionId=null 2023-07-29T00:39:39.183 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+/kTMib14CvbJYygGFCKIqrjkjY= 2023-07-29T00:39:39.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+/kTMib14CvbJYygGFCKIqrjkjY=', 'Content-Length': '0'} 2023-07-29T00:39:39.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f5de240fbed0863-0064c45fcb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-07-29T00:39:39.187 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:39:39.187 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:39:39.187 DEBUG:boto:path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:39:39.188 DEBUG:boto:auth_path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:39:39.188 DEBUG:boto:Method: GET 2023-07-29T00:39:39.189 DEBUG:boto:Path: /hiiqos-59/obj68?versionId=null 2023-07-29T00:39:39.189 DEBUG:boto:Data: 2023-07-29T00:39:39.189 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.190 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:39.190 DEBUG:boto:Port: 8001 2023-07-29T00:39:39.191 DEBUG:boto:Params: {} 2023-07-29T00:39:39.191 DEBUG:boto:Token: None 2023-07-29T00:39:39.191 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj68?versionId=null 2023-07-29T00:39:39.192 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:e+NvXxNnm968AeXDdxrFplF/96I= 2023-07-29T00:39:39.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:e+NvXxNnm968AeXDdxrFplF/96I=', 'Content-Length': '0'} 2023-07-29T00:39:39.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a0f97c6a32d2151-0064c45fcb-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.195 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:39:39.195 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:39:39.196 DEBUG:boto:path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:39:39.196 DEBUG:boto:auth_path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:39:39.196 DEBUG:boto:Method: GET 2023-07-29T00:39:39.197 DEBUG:boto:Path: /hiiqos-59/obj68?versionId=null 2023-07-29T00:39:39.197 DEBUG:boto:Data: 2023-07-29T00:39:39.197 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.198 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:39.198 DEBUG:boto:Port: 8000 2023-07-29T00:39:39.199 DEBUG:boto:Params: {} 2023-07-29T00:39:39.199 DEBUG:boto:Token: None 2023-07-29T00:39:39.199 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj68?versionId=null 2023-07-29T00:39:39.200 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:e+NvXxNnm968AeXDdxrFplF/96I= 2023-07-29T00:39:39.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:e+NvXxNnm968AeXDdxrFplF/96I=', 'Content-Length': '0'} 2023-07-29T00:39:39.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ddd50a5e23ea244-0064c45fcb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.203 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:39.203 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:39.204 DEBUG:boto:path=/hiiqos-59/?prefix=obj68 2023-07-29T00:39:39.204 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj68 2023-07-29T00:39:39.204 DEBUG:boto:Method: GET 2023-07-29T00:39:39.205 DEBUG:boto:Path: /hiiqos-59/?prefix=obj68 2023-07-29T00:39:39.205 DEBUG:boto:Data: 2023-07-29T00:39:39.205 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.206 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:39.206 DEBUG:boto:Port: 8001 2023-07-29T00:39:39.207 DEBUG:boto:Params: {} 2023-07-29T00:39:39.207 DEBUG:boto:Token: None 2023-07-29T00:39:39.208 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/ 2023-07-29T00:39:39.209 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qK1sfPK+p7ymNycEAB5Yr1COdqw= 2023-07-29T00:39:39.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qK1sfPK+p7ymNycEAB5Yr1COdqw=', 'Content-Length': '0'} 2023-07-29T00:39:39.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fd51e4302180ec9-0064c45fcb-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.212 DEBUG:boto:b'hiiqos-59obj681000falseobj682023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:39.213 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:39.213 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:39.213 DEBUG:boto:path=/hiiqos-59/?prefix=obj68 2023-07-29T00:39:39.214 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj68 2023-07-29T00:39:39.214 DEBUG:boto:Method: GET 2023-07-29T00:39:39.214 DEBUG:boto:Path: /hiiqos-59/?prefix=obj68 2023-07-29T00:39:39.215 DEBUG:boto:Data: 2023-07-29T00:39:39.215 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.215 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:39.216 DEBUG:boto:Port: 8000 2023-07-29T00:39:39.216 DEBUG:boto:Params: {} 2023-07-29T00:39:39.217 DEBUG:boto:Token: None 2023-07-29T00:39:39.217 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/ 2023-07-29T00:39:39.218 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qK1sfPK+p7ymNycEAB5Yr1COdqw= 2023-07-29T00:39:39.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qK1sfPK+p7ymNycEAB5Yr1COdqw=', 'Content-Length': '0'} 2023-07-29T00:39:39.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005541b76ac5dc23fa-0064c45fcb-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.223 DEBUG:boto:b'hiiqos-59obj681000falseobj682023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:39.223 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:39:39.224 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:39:39.224 DEBUG:boto:Method: HEAD 2023-07-29T00:39:39.225 DEBUG:boto:Path: /hiiqos-59/obj68 2023-07-29T00:39:39.225 DEBUG:boto:Data: 2023-07-29T00:39:39.225 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.226 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:39.226 DEBUG:boto:Port: 8001 2023-07-29T00:39:39.226 DEBUG:boto:Params: {} 2023-07-29T00:39:39.227 DEBUG:boto:Token: None 2023-07-29T00:39:39.227 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj68 2023-07-29T00:39:39.228 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Nwx2dvQz0g8nOhH/4Mkriv3F5qk= 2023-07-29T00:39:39.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Nwx2dvQz0g8nOhH/4Mkriv3F5qk=', 'Content-Length': '0'} 2023-07-29T00:39:39.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5ff87d2be61b866-0064c45fcb-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.231 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:39:39.231 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:39:39.232 DEBUG:boto:Method: HEAD 2023-07-29T00:39:39.232 DEBUG:boto:Path: /hiiqos-59/obj68 2023-07-29T00:39:39.232 DEBUG:boto:Data: 2023-07-29T00:39:39.233 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.233 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:39.233 DEBUG:boto:Port: 8000 2023-07-29T00:39:39.234 DEBUG:boto:Params: {} 2023-07-29T00:39:39.234 DEBUG:boto:Token: None 2023-07-29T00:39:39.234 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj68 2023-07-29T00:39:39.235 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Nwx2dvQz0g8nOhH/4Mkriv3F5qk= 2023-07-29T00:39:39.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Nwx2dvQz0g8nOhH/4Mkriv3F5qk=', 'Content-Length': '0'} 2023-07-29T00:39:39.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000058e504fee3c6e79-0064c45fcb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-07-29T00:39:39.238 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:39:39.238 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:39:39.239 DEBUG:boto:Method: GET 2023-07-29T00:39:39.239 DEBUG:boto:Path: /hiiqos-59/obj68 2023-07-29T00:39:39.239 DEBUG:boto:Data: 2023-07-29T00:39:39.240 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.240 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:39.241 DEBUG:boto:Port: 8001 2023-07-29T00:39:39.241 DEBUG:boto:Params: {} 2023-07-29T00:39:39.241 DEBUG:boto:Token: None 2023-07-29T00:39:39.242 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj68 2023-07-29T00:39:39.242 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:b5JhlBmMp8YTi4KoS3pQaZLdDtA= 2023-07-29T00:39:39.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:b5JhlBmMp8YTi4KoS3pQaZLdDtA=', 'Content-Length': '0'} 2023-07-29T00:39:39.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f54e854416e81f69-0064c45fcb-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.245 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:39:39.245 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:39:39.246 DEBUG:boto:Method: GET 2023-07-29T00:39:39.246 DEBUG:boto:Path: /hiiqos-59/obj68 2023-07-29T00:39:39.246 DEBUG:boto:Data: 2023-07-29T00:39:39.247 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.247 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:39.248 DEBUG:boto:Port: 8000 2023-07-29T00:39:39.248 DEBUG:boto:Params: {} 2023-07-29T00:39:39.248 DEBUG:boto:Token: None 2023-07-29T00:39:39.249 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj68 2023-07-29T00:39:39.249 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:b5JhlBmMp8YTi4KoS3pQaZLdDtA= 2023-07-29T00:39:39.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:b5JhlBmMp8YTi4KoS3pQaZLdDtA=', 'Content-Length': '0'} 2023-07-29T00:39:39.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea995e2dcb8a446c-0064c45fcb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.296 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:39.297 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:39.297 DEBUG:boto:path=/hiiqos-59/?prefix=obj68 2023-07-29T00:39:39.298 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj68 2023-07-29T00:39:39.298 DEBUG:boto:Method: GET 2023-07-29T00:39:39.298 DEBUG:boto:Path: /hiiqos-59/?prefix=obj68 2023-07-29T00:39:39.299 DEBUG:boto:Data: 2023-07-29T00:39:39.299 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.299 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:39.300 DEBUG:boto:Port: 8001 2023-07-29T00:39:39.300 DEBUG:boto:Params: {} 2023-07-29T00:39:39.301 DEBUG:boto:Token: None 2023-07-29T00:39:39.301 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/ 2023-07-29T00:39:39.301 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qK1sfPK+p7ymNycEAB5Yr1COdqw= 2023-07-29T00:39:39.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qK1sfPK+p7ymNycEAB5Yr1COdqw=', 'Content-Length': '0'} 2023-07-29T00:39:39.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee704c04e3604f28-0064c45fcb-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.305 DEBUG:boto:b'hiiqos-59obj681000falseobj682023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:39.306 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:39.306 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:39.306 DEBUG:boto:path=/hiiqos-59/?prefix=obj68 2023-07-29T00:39:39.307 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj68 2023-07-29T00:39:39.307 DEBUG:boto:Method: GET 2023-07-29T00:39:39.307 DEBUG:boto:Path: /hiiqos-59/?prefix=obj68 2023-07-29T00:39:39.308 DEBUG:boto:Data: 2023-07-29T00:39:39.308 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.308 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:39.309 DEBUG:boto:Port: 8000 2023-07-29T00:39:39.309 DEBUG:boto:Params: {} 2023-07-29T00:39:39.310 DEBUG:boto:Token: None 2023-07-29T00:39:39.310 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/ 2023-07-29T00:39:39.311 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qK1sfPK+p7ymNycEAB5Yr1COdqw= 2023-07-29T00:39:39.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qK1sfPK+p7ymNycEAB5Yr1COdqw=', 'Content-Length': '0'} 2023-07-29T00:39:39.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fec39a4b3c8313c8-0064c45fcb-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.316 DEBUG:boto:b'hiiqos-59obj681000falseobj682023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:39.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-07-29T00:39:39.317 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:39:39.317 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:39:39.318 DEBUG:boto:path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:39:39.318 DEBUG:boto:auth_path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:39:39.318 DEBUG:boto:Method: GET 2023-07-29T00:39:39.319 DEBUG:boto:Path: /hiiqos-59/obj69?versionId=null 2023-07-29T00:39:39.319 DEBUG:boto:Data: 2023-07-29T00:39:39.319 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.320 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:39.320 DEBUG:boto:Port: 8001 2023-07-29T00:39:39.321 DEBUG:boto:Params: {} 2023-07-29T00:39:39.321 DEBUG:boto:Token: None 2023-07-29T00:39:39.321 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj69?versionId=null 2023-07-29T00:39:39.322 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yJDMaB2kUjYjx3g6SAK5bTLgSAM= 2023-07-29T00:39:39.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yJDMaB2kUjYjx3g6SAK5bTLgSAM=', 'Content-Length': '0'} 2023-07-29T00:39:39.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072ec7c1a3bdf2d49-0064c45fcb-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.325 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:39:39.325 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:39:39.326 DEBUG:boto:path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:39:39.326 DEBUG:boto:auth_path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:39:39.327 DEBUG:boto:Method: GET 2023-07-29T00:39:39.327 DEBUG:boto:Path: /hiiqos-59/obj69?versionId=null 2023-07-29T00:39:39.327 DEBUG:boto:Data: 2023-07-29T00:39:39.328 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.328 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:39.328 DEBUG:boto:Port: 8000 2023-07-29T00:39:39.329 DEBUG:boto:Params: {} 2023-07-29T00:39:39.329 DEBUG:boto:Token: None 2023-07-29T00:39:39.330 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj69?versionId=null 2023-07-29T00:39:39.330 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yJDMaB2kUjYjx3g6SAK5bTLgSAM= 2023-07-29T00:39:39.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yJDMaB2kUjYjx3g6SAK5bTLgSAM=', 'Content-Length': '0'} 2023-07-29T00:39:39.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053a77eefe460ddc7-0064c45fcb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.333 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:39:39.334 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:39:39.334 DEBUG:boto:path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:39:39.334 DEBUG:boto:auth_path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:39:39.335 DEBUG:boto:Method: HEAD 2023-07-29T00:39:39.335 DEBUG:boto:Path: /hiiqos-59/obj69?versionId=null 2023-07-29T00:39:39.335 DEBUG:boto:Data: 2023-07-29T00:39:39.336 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.336 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:39.336 DEBUG:boto:Port: 8001 2023-07-29T00:39:39.337 DEBUG:boto:Params: {} 2023-07-29T00:39:39.337 DEBUG:boto:Token: None 2023-07-29T00:39:39.337 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj69?versionId=null 2023-07-29T00:39:39.338 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Wou5UzCq3w3Q6ZU8B9/No79mn1M= 2023-07-29T00:39:39.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Wou5UzCq3w3Q6ZU8B9/No79mn1M=', 'Content-Length': '0'} 2023-07-29T00:39:39.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008c613186507593b-0064c45fcb-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.341 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:39:39.341 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:39:39.342 DEBUG:boto:path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:39:39.342 DEBUG:boto:auth_path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:39:39.342 DEBUG:boto:Method: HEAD 2023-07-29T00:39:39.343 DEBUG:boto:Path: /hiiqos-59/obj69?versionId=null 2023-07-29T00:39:39.343 DEBUG:boto:Data: 2023-07-29T00:39:39.343 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.344 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:39.344 DEBUG:boto:Port: 8000 2023-07-29T00:39:39.344 DEBUG:boto:Params: {} 2023-07-29T00:39:39.345 DEBUG:boto:Token: None 2023-07-29T00:39:39.345 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj69?versionId=null 2023-07-29T00:39:39.346 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Wou5UzCq3w3Q6ZU8B9/No79mn1M= 2023-07-29T00:39:39.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Wou5UzCq3w3Q6ZU8B9/No79mn1M=', 'Content-Length': '0'} 2023-07-29T00:39:39.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a13139bec8d2d8c-0064c45fcb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-07-29T00:39:39.349 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:39:39.350 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:39:39.350 DEBUG:boto:path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:39:39.350 DEBUG:boto:auth_path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:39:39.351 DEBUG:boto:Method: GET 2023-07-29T00:39:39.351 DEBUG:boto:Path: /hiiqos-59/obj69?versionId=null 2023-07-29T00:39:39.351 DEBUG:boto:Data: 2023-07-29T00:39:39.352 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.352 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:39.353 DEBUG:boto:Port: 8001 2023-07-29T00:39:39.353 DEBUG:boto:Params: {} 2023-07-29T00:39:39.353 DEBUG:boto:Token: None 2023-07-29T00:39:39.354 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj69?versionId=null 2023-07-29T00:39:39.354 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yJDMaB2kUjYjx3g6SAK5bTLgSAM= 2023-07-29T00:39:39.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yJDMaB2kUjYjx3g6SAK5bTLgSAM=', 'Content-Length': '0'} 2023-07-29T00:39:39.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b46e5ef622821b5f-0064c45fcb-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.400 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:39:39.401 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:39:39.401 DEBUG:boto:path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:39:39.402 DEBUG:boto:auth_path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:39:39.402 DEBUG:boto:Method: GET 2023-07-29T00:39:39.402 DEBUG:boto:Path: /hiiqos-59/obj69?versionId=null 2023-07-29T00:39:39.403 DEBUG:boto:Data: 2023-07-29T00:39:39.403 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.404 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:39.404 DEBUG:boto:Port: 8000 2023-07-29T00:39:39.404 DEBUG:boto:Params: {} 2023-07-29T00:39:39.405 DEBUG:boto:Token: None 2023-07-29T00:39:39.405 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj69?versionId=null 2023-07-29T00:39:39.406 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yJDMaB2kUjYjx3g6SAK5bTLgSAM= 2023-07-29T00:39:39.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yJDMaB2kUjYjx3g6SAK5bTLgSAM=', 'Content-Length': '0'} 2023-07-29T00:39:39.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098534852a617c8c8-0064c45fcb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.409 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:39.410 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:39.410 DEBUG:boto:path=/hiiqos-59/?prefix=obj69 2023-07-29T00:39:39.411 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj69 2023-07-29T00:39:39.412 DEBUG:boto:Method: GET 2023-07-29T00:39:39.412 DEBUG:boto:Path: /hiiqos-59/?prefix=obj69 2023-07-29T00:39:39.413 DEBUG:boto:Data: 2023-07-29T00:39:39.413 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.413 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:39.414 DEBUG:boto:Port: 8001 2023-07-29T00:39:39.414 DEBUG:boto:Params: {} 2023-07-29T00:39:39.415 DEBUG:boto:Token: None 2023-07-29T00:39:39.415 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/ 2023-07-29T00:39:39.415 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qK1sfPK+p7ymNycEAB5Yr1COdqw= 2023-07-29T00:39:39.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qK1sfPK+p7ymNycEAB5Yr1COdqw=', 'Content-Length': '0'} 2023-07-29T00:39:39.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d94a6fb366b093c2-0064c45fcb-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.419 DEBUG:boto:b'hiiqos-59obj691000falseobj692023-07-29T00:33:06.857Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:39.420 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:39.420 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:39.421 DEBUG:boto:path=/hiiqos-59/?prefix=obj69 2023-07-29T00:39:39.421 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj69 2023-07-29T00:39:39.421 DEBUG:boto:Method: GET 2023-07-29T00:39:39.422 DEBUG:boto:Path: /hiiqos-59/?prefix=obj69 2023-07-29T00:39:39.422 DEBUG:boto:Data: 2023-07-29T00:39:39.423 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.423 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:39.423 DEBUG:boto:Port: 8000 2023-07-29T00:39:39.424 DEBUG:boto:Params: {} 2023-07-29T00:39:39.424 DEBUG:boto:Token: None 2023-07-29T00:39:39.425 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/ 2023-07-29T00:39:39.425 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qK1sfPK+p7ymNycEAB5Yr1COdqw= 2023-07-29T00:39:39.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qK1sfPK+p7ymNycEAB5Yr1COdqw=', 'Content-Length': '0'} 2023-07-29T00:39:39.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008de6eebc30ed9370-0064c45fcb-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.431 DEBUG:boto:b'hiiqos-59obj691000falseobj692023-07-29T00:33:06.857Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:39.431 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:39:39.432 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:39:39.432 DEBUG:boto:Method: HEAD 2023-07-29T00:39:39.433 DEBUG:boto:Path: /hiiqos-59/obj69 2023-07-29T00:39:39.433 DEBUG:boto:Data: 2023-07-29T00:39:39.433 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.434 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:39.434 DEBUG:boto:Port: 8001 2023-07-29T00:39:39.434 DEBUG:boto:Params: {} 2023-07-29T00:39:39.435 DEBUG:boto:Token: None 2023-07-29T00:39:39.435 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj69 2023-07-29T00:39:39.436 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3kz8u4fIp5YBPc2Xl684BR9D80w= 2023-07-29T00:39:39.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3kz8u4fIp5YBPc2Xl684BR9D80w=', 'Content-Length': '0'} 2023-07-29T00:39:39.438 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3504b506bc3c27b-0064c45fcb-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.438 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:39:39.439 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:39:39.439 DEBUG:boto:Method: HEAD 2023-07-29T00:39:39.440 DEBUG:boto:Path: /hiiqos-59/obj69 2023-07-29T00:39:39.440 DEBUG:boto:Data: 2023-07-29T00:39:39.440 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.441 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:39.441 DEBUG:boto:Port: 8000 2023-07-29T00:39:39.441 DEBUG:boto:Params: {} 2023-07-29T00:39:39.442 DEBUG:boto:Token: None 2023-07-29T00:39:39.442 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj69 2023-07-29T00:39:39.443 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3kz8u4fIp5YBPc2Xl684BR9D80w= 2023-07-29T00:39:39.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3kz8u4fIp5YBPc2Xl684BR9D80w=', 'Content-Length': '0'} 2023-07-29T00:39:39.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5ddda8578f6326a-0064c45fcb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-07-29T00:39:39.446 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:39:39.446 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:39:39.447 DEBUG:boto:Method: GET 2023-07-29T00:39:39.447 DEBUG:boto:Path: /hiiqos-59/obj69 2023-07-29T00:39:39.448 DEBUG:boto:Data: 2023-07-29T00:39:39.448 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.448 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:39.449 DEBUG:boto:Port: 8001 2023-07-29T00:39:39.449 DEBUG:boto:Params: {} 2023-07-29T00:39:39.450 DEBUG:boto:Token: None 2023-07-29T00:39:39.450 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj69 2023-07-29T00:39:39.451 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DllwM6t8DRlLoxCj4CSN8zLWPqo= 2023-07-29T00:39:39.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DllwM6t8DRlLoxCj4CSN8zLWPqo=', 'Content-Length': '0'} 2023-07-29T00:39:39.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037a3cb186646faf3-0064c45fcb-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.454 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:39:39.454 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:39:39.455 DEBUG:boto:Method: GET 2023-07-29T00:39:39.455 DEBUG:boto:Path: /hiiqos-59/obj69 2023-07-29T00:39:39.455 DEBUG:boto:Data: 2023-07-29T00:39:39.456 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.456 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:39.457 DEBUG:boto:Port: 8000 2023-07-29T00:39:39.457 DEBUG:boto:Params: {} 2023-07-29T00:39:39.457 DEBUG:boto:Token: None 2023-07-29T00:39:39.458 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj69 2023-07-29T00:39:39.458 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DllwM6t8DRlLoxCj4CSN8zLWPqo= 2023-07-29T00:39:39.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DllwM6t8DRlLoxCj4CSN8zLWPqo=', 'Content-Length': '0'} 2023-07-29T00:39:39.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009565448f015d631e-0064c45fcb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.504 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:39.505 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:39.505 DEBUG:boto:path=/hiiqos-59/?prefix=obj69 2023-07-29T00:39:39.506 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj69 2023-07-29T00:39:39.506 DEBUG:boto:Method: GET 2023-07-29T00:39:39.506 DEBUG:boto:Path: /hiiqos-59/?prefix=obj69 2023-07-29T00:39:39.507 DEBUG:boto:Data: 2023-07-29T00:39:39.507 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.508 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:39.508 DEBUG:boto:Port: 8001 2023-07-29T00:39:39.508 DEBUG:boto:Params: {} 2023-07-29T00:39:39.509 DEBUG:boto:Token: None 2023-07-29T00:39:39.509 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/ 2023-07-29T00:39:39.510 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qK1sfPK+p7ymNycEAB5Yr1COdqw= 2023-07-29T00:39:39.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qK1sfPK+p7ymNycEAB5Yr1COdqw=', 'Content-Length': '0'} 2023-07-29T00:39:39.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079d9cbec2c50bd18-0064c45fcb-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.513 DEBUG:boto:b'hiiqos-59obj691000falseobj692023-07-29T00:33:06.857Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:39.514 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:39.514 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:39.515 DEBUG:boto:path=/hiiqos-59/?prefix=obj69 2023-07-29T00:39:39.515 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj69 2023-07-29T00:39:39.515 DEBUG:boto:Method: GET 2023-07-29T00:39:39.516 DEBUG:boto:Path: /hiiqos-59/?prefix=obj69 2023-07-29T00:39:39.516 DEBUG:boto:Data: 2023-07-29T00:39:39.517 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.517 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:39.517 DEBUG:boto:Port: 8000 2023-07-29T00:39:39.518 DEBUG:boto:Params: {} 2023-07-29T00:39:39.518 DEBUG:boto:Token: None 2023-07-29T00:39:39.519 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/ 2023-07-29T00:39:39.519 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qK1sfPK+p7ymNycEAB5Yr1COdqw= 2023-07-29T00:39:39.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qK1sfPK+p7ymNycEAB5Yr1COdqw=', 'Content-Length': '0'} 2023-07-29T00:39:39.524 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044cfa042a307a7e6-0064c45fcb-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.525 DEBUG:boto:b'hiiqos-59obj691000falseobj692023-07-29T00:33:06.857Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:39.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-07-29T00:39:39.526 DEBUG:boto:path=/hiiqos-59/obj74 2023-07-29T00:39:39.527 DEBUG:boto:auth_path=/hiiqos-59/obj74 2023-07-29T00:39:39.527 DEBUG:boto:path=/hiiqos-59/obj74?versionId=null 2023-07-29T00:39:39.527 DEBUG:boto:auth_path=/hiiqos-59/obj74?versionId=null 2023-07-29T00:39:39.528 DEBUG:boto:Method: GET 2023-07-29T00:39:39.528 DEBUG:boto:Path: /hiiqos-59/obj74?versionId=null 2023-07-29T00:39:39.529 DEBUG:boto:Data: 2023-07-29T00:39:39.529 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.529 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:39.530 DEBUG:boto:Port: 8001 2023-07-29T00:39:39.530 DEBUG:boto:Params: {} 2023-07-29T00:39:39.530 DEBUG:boto:Token: None 2023-07-29T00:39:39.531 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj74?versionId=null 2023-07-29T00:39:39.531 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uyBvx0popDP7P5/m1TdLsFh84BA= 2023-07-29T00:39:39.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uyBvx0popDP7P5/m1TdLsFh84BA=', 'Content-Length': '0'} 2023-07-29T00:39:39.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004930ce2421995413-0064c45fcb-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.535 DEBUG:boto:path=/hiiqos-59/obj74 2023-07-29T00:39:39.535 DEBUG:boto:auth_path=/hiiqos-59/obj74 2023-07-29T00:39:39.536 DEBUG:boto:path=/hiiqos-59/obj74?versionId=null 2023-07-29T00:39:39.536 DEBUG:boto:auth_path=/hiiqos-59/obj74?versionId=null 2023-07-29T00:39:39.537 DEBUG:boto:Method: GET 2023-07-29T00:39:39.537 DEBUG:boto:Path: /hiiqos-59/obj74?versionId=null 2023-07-29T00:39:39.537 DEBUG:boto:Data: 2023-07-29T00:39:39.538 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.538 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:39.538 DEBUG:boto:Port: 8000 2023-07-29T00:39:39.539 DEBUG:boto:Params: {} 2023-07-29T00:39:39.539 DEBUG:boto:Token: None 2023-07-29T00:39:39.539 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj74?versionId=null 2023-07-29T00:39:39.540 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uyBvx0popDP7P5/m1TdLsFh84BA= 2023-07-29T00:39:39.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uyBvx0popDP7P5/m1TdLsFh84BA=', 'Content-Length': '0'} 2023-07-29T00:39:39.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000893290480874e515-0064c45fcb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.543 DEBUG:boto:path=/hiiqos-59/obj74 2023-07-29T00:39:39.544 DEBUG:boto:auth_path=/hiiqos-59/obj74 2023-07-29T00:39:39.544 DEBUG:boto:path=/hiiqos-59/obj74?versionId=null 2023-07-29T00:39:39.544 DEBUG:boto:auth_path=/hiiqos-59/obj74?versionId=null 2023-07-29T00:39:39.545 DEBUG:boto:Method: HEAD 2023-07-29T00:39:39.545 DEBUG:boto:Path: /hiiqos-59/obj74?versionId=null 2023-07-29T00:39:39.545 DEBUG:boto:Data: 2023-07-29T00:39:39.546 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.546 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:39.546 DEBUG:boto:Port: 8001 2023-07-29T00:39:39.547 DEBUG:boto:Params: {} 2023-07-29T00:39:39.547 DEBUG:boto:Token: None 2023-07-29T00:39:39.547 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj74?versionId=null 2023-07-29T00:39:39.548 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4FQIxdQGXsscnM7HJBqtgkdFh9g= 2023-07-29T00:39:39.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4FQIxdQGXsscnM7HJBqtgkdFh9g=', 'Content-Length': '0'} 2023-07-29T00:39:39.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e91eb44e193a822-0064c45fcb-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.551 DEBUG:boto:path=/hiiqos-59/obj74 2023-07-29T00:39:39.551 DEBUG:boto:auth_path=/hiiqos-59/obj74 2023-07-29T00:39:39.551 DEBUG:boto:path=/hiiqos-59/obj74?versionId=null 2023-07-29T00:39:39.552 DEBUG:boto:auth_path=/hiiqos-59/obj74?versionId=null 2023-07-29T00:39:39.552 DEBUG:boto:Method: HEAD 2023-07-29T00:39:39.552 DEBUG:boto:Path: /hiiqos-59/obj74?versionId=null 2023-07-29T00:39:39.553 DEBUG:boto:Data: 2023-07-29T00:39:39.553 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.553 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:39.554 DEBUG:boto:Port: 8000 2023-07-29T00:39:39.554 DEBUG:boto:Params: {} 2023-07-29T00:39:39.554 DEBUG:boto:Token: None 2023-07-29T00:39:39.555 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj74?versionId=null 2023-07-29T00:39:39.555 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4FQIxdQGXsscnM7HJBqtgkdFh9g= 2023-07-29T00:39:39.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4FQIxdQGXsscnM7HJBqtgkdFh9g=', 'Content-Length': '0'} 2023-07-29T00:39:39.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e3610cae6391c51-0064c45fcb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-07-29T00:39:39.558 DEBUG:boto:path=/hiiqos-59/obj74 2023-07-29T00:39:39.559 DEBUG:boto:auth_path=/hiiqos-59/obj74 2023-07-29T00:39:39.559 DEBUG:boto:path=/hiiqos-59/obj74?versionId=null 2023-07-29T00:39:39.560 DEBUG:boto:auth_path=/hiiqos-59/obj74?versionId=null 2023-07-29T00:39:39.560 DEBUG:boto:Method: GET 2023-07-29T00:39:39.560 DEBUG:boto:Path: /hiiqos-59/obj74?versionId=null 2023-07-29T00:39:39.561 DEBUG:boto:Data: 2023-07-29T00:39:39.561 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.561 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:39.562 DEBUG:boto:Port: 8001 2023-07-29T00:39:39.562 DEBUG:boto:Params: {} 2023-07-29T00:39:39.562 DEBUG:boto:Token: None 2023-07-29T00:39:39.563 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj74?versionId=null 2023-07-29T00:39:39.563 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uyBvx0popDP7P5/m1TdLsFh84BA= 2023-07-29T00:39:39.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uyBvx0popDP7P5/m1TdLsFh84BA=', 'Content-Length': '0'} 2023-07-29T00:39:39.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a08f8274593fc369-0064c45fcb-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.566 DEBUG:boto:path=/hiiqos-59/obj74 2023-07-29T00:39:39.567 DEBUG:boto:auth_path=/hiiqos-59/obj74 2023-07-29T00:39:39.567 DEBUG:boto:path=/hiiqos-59/obj74?versionId=null 2023-07-29T00:39:39.568 DEBUG:boto:auth_path=/hiiqos-59/obj74?versionId=null 2023-07-29T00:39:39.568 DEBUG:boto:Method: GET 2023-07-29T00:39:39.568 DEBUG:boto:Path: /hiiqos-59/obj74?versionId=null 2023-07-29T00:39:39.569 DEBUG:boto:Data: 2023-07-29T00:39:39.569 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.569 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:39.570 DEBUG:boto:Port: 8000 2023-07-29T00:39:39.570 DEBUG:boto:Params: {} 2023-07-29T00:39:39.571 DEBUG:boto:Token: None 2023-07-29T00:39:39.571 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj74?versionId=null 2023-07-29T00:39:39.571 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uyBvx0popDP7P5/m1TdLsFh84BA= 2023-07-29T00:39:39.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uyBvx0popDP7P5/m1TdLsFh84BA=', 'Content-Length': '0'} 2023-07-29T00:39:39.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009ecfa29228396ae-0064c45fcb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.574 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:39.575 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:39.575 DEBUG:boto:path=/hiiqos-59/?prefix=obj74 2023-07-29T00:39:39.576 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj74 2023-07-29T00:39:39.576 DEBUG:boto:Method: GET 2023-07-29T00:39:39.576 DEBUG:boto:Path: /hiiqos-59/?prefix=obj74 2023-07-29T00:39:39.577 DEBUG:boto:Data: 2023-07-29T00:39:39.577 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.577 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:39.578 DEBUG:boto:Port: 8001 2023-07-29T00:39:39.578 DEBUG:boto:Params: {} 2023-07-29T00:39:39.578 DEBUG:boto:Token: None 2023-07-29T00:39:39.579 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/ 2023-07-29T00:39:39.579 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qK1sfPK+p7ymNycEAB5Yr1COdqw= 2023-07-29T00:39:39.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qK1sfPK+p7ymNycEAB5Yr1COdqw=', 'Content-Length': '0'} 2023-07-29T00:39:39.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a6fd4c97e374d92-0064c45fcb-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.583 DEBUG:boto:b'hiiqos-59obj741000falseobj742023-07-29T00:37:53.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:39.583 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:39.584 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:39.584 DEBUG:boto:path=/hiiqos-59/?prefix=obj74 2023-07-29T00:39:39.585 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj74 2023-07-29T00:39:39.585 DEBUG:boto:Method: GET 2023-07-29T00:39:39.585 DEBUG:boto:Path: /hiiqos-59/?prefix=obj74 2023-07-29T00:39:39.586 DEBUG:boto:Data: 2023-07-29T00:39:39.586 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.586 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:39.587 DEBUG:boto:Port: 8000 2023-07-29T00:39:39.587 DEBUG:boto:Params: {} 2023-07-29T00:39:39.588 DEBUG:boto:Token: None 2023-07-29T00:39:39.588 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/ 2023-07-29T00:39:39.588 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qK1sfPK+p7ymNycEAB5Yr1COdqw= 2023-07-29T00:39:39.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qK1sfPK+p7ymNycEAB5Yr1COdqw=', 'Content-Length': '0'} 2023-07-29T00:39:39.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1af1f35f99a6b0e-0064c45fcb-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.594 DEBUG:boto:b'hiiqos-59obj741000falseobj742023-07-29T00:37:53.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:39.594 DEBUG:boto:path=/hiiqos-59/obj74 2023-07-29T00:39:39.595 DEBUG:boto:auth_path=/hiiqos-59/obj74 2023-07-29T00:39:39.595 DEBUG:boto:Method: HEAD 2023-07-29T00:39:39.595 DEBUG:boto:Path: /hiiqos-59/obj74 2023-07-29T00:39:39.596 DEBUG:boto:Data: 2023-07-29T00:39:39.596 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.597 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:39.597 DEBUG:boto:Port: 8001 2023-07-29T00:39:39.597 DEBUG:boto:Params: {} 2023-07-29T00:39:39.598 DEBUG:boto:Token: None 2023-07-29T00:39:39.598 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj74 2023-07-29T00:39:39.598 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:p9BCeUMyloMcWZ550rz8Ap6DTwo= 2023-07-29T00:39:39.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:p9BCeUMyloMcWZ550rz8Ap6DTwo=', 'Content-Length': '0'} 2023-07-29T00:39:39.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbe3f910785d44c7-0064c45fcb-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.601 DEBUG:boto:path=/hiiqos-59/obj74 2023-07-29T00:39:39.602 DEBUG:boto:auth_path=/hiiqos-59/obj74 2023-07-29T00:39:39.602 DEBUG:boto:Method: HEAD 2023-07-29T00:39:39.602 DEBUG:boto:Path: /hiiqos-59/obj74 2023-07-29T00:39:39.603 DEBUG:boto:Data: 2023-07-29T00:39:39.603 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.603 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:39.604 DEBUG:boto:Port: 8000 2023-07-29T00:39:39.604 DEBUG:boto:Params: {} 2023-07-29T00:39:39.604 DEBUG:boto:Token: None 2023-07-29T00:39:39.605 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj74 2023-07-29T00:39:39.605 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:p9BCeUMyloMcWZ550rz8Ap6DTwo= 2023-07-29T00:39:39.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:p9BCeUMyloMcWZ550rz8Ap6DTwo=', 'Content-Length': '0'} 2023-07-29T00:39:39.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092f42a3eec34d57d-0064c45fcb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-07-29T00:39:39.609 DEBUG:boto:path=/hiiqos-59/obj74 2023-07-29T00:39:39.609 DEBUG:boto:auth_path=/hiiqos-59/obj74 2023-07-29T00:39:39.609 DEBUG:boto:Method: GET 2023-07-29T00:39:39.610 DEBUG:boto:Path: /hiiqos-59/obj74 2023-07-29T00:39:39.610 DEBUG:boto:Data: 2023-07-29T00:39:39.610 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.611 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:39.611 DEBUG:boto:Port: 8001 2023-07-29T00:39:39.611 DEBUG:boto:Params: {} 2023-07-29T00:39:39.612 DEBUG:boto:Token: None 2023-07-29T00:39:39.612 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj74 2023-07-29T00:39:39.613 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QylmE3YxDMpY0PJPNbrkwcCj+hs= 2023-07-29T00:39:39.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QylmE3YxDMpY0PJPNbrkwcCj+hs=', 'Content-Length': '0'} 2023-07-29T00:39:39.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000725956ee4550e9fa-0064c45fcb-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.656 DEBUG:boto:path=/hiiqos-59/obj74 2023-07-29T00:39:39.657 DEBUG:boto:auth_path=/hiiqos-59/obj74 2023-07-29T00:39:39.658 DEBUG:boto:Method: GET 2023-07-29T00:39:39.658 DEBUG:boto:Path: /hiiqos-59/obj74 2023-07-29T00:39:39.659 DEBUG:boto:Data: 2023-07-29T00:39:39.659 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.660 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:39.660 DEBUG:boto:Port: 8000 2023-07-29T00:39:39.660 DEBUG:boto:Params: {} 2023-07-29T00:39:39.661 DEBUG:boto:Token: None 2023-07-29T00:39:39.661 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj74 2023-07-29T00:39:39.662 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QylmE3YxDMpY0PJPNbrkwcCj+hs= 2023-07-29T00:39:39.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QylmE3YxDMpY0PJPNbrkwcCj+hs=', 'Content-Length': '0'} 2023-07-29T00:39:39.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcb8068f7890c6de-0064c45fcb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.665 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:39.665 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:39.666 DEBUG:boto:path=/hiiqos-59/?prefix=obj74 2023-07-29T00:39:39.666 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj74 2023-07-29T00:39:39.666 DEBUG:boto:Method: GET 2023-07-29T00:39:39.667 DEBUG:boto:Path: /hiiqos-59/?prefix=obj74 2023-07-29T00:39:39.667 DEBUG:boto:Data: 2023-07-29T00:39:39.667 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.668 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:39.668 DEBUG:boto:Port: 8001 2023-07-29T00:39:39.669 DEBUG:boto:Params: {} 2023-07-29T00:39:39.669 DEBUG:boto:Token: None 2023-07-29T00:39:39.669 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/ 2023-07-29T00:39:39.670 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qK1sfPK+p7ymNycEAB5Yr1COdqw= 2023-07-29T00:39:39.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qK1sfPK+p7ymNycEAB5Yr1COdqw=', 'Content-Length': '0'} 2023-07-29T00:39:39.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000472247d01aa981af-0064c45fcb-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.673 DEBUG:boto:b'hiiqos-59obj741000falseobj742023-07-29T00:37:53.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:39.674 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:39.675 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:39.675 DEBUG:boto:path=/hiiqos-59/?prefix=obj74 2023-07-29T00:39:39.675 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj74 2023-07-29T00:39:39.676 DEBUG:boto:Method: GET 2023-07-29T00:39:39.676 DEBUG:boto:Path: /hiiqos-59/?prefix=obj74 2023-07-29T00:39:39.677 DEBUG:boto:Data: 2023-07-29T00:39:39.677 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.677 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:39.678 DEBUG:boto:Port: 8000 2023-07-29T00:39:39.678 DEBUG:boto:Params: {} 2023-07-29T00:39:39.679 DEBUG:boto:Token: None 2023-07-29T00:39:39.679 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/ 2023-07-29T00:39:39.680 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qK1sfPK+p7ymNycEAB5Yr1COdqw= 2023-07-29T00:39:39.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qK1sfPK+p7ymNycEAB5Yr1COdqw=', 'Content-Length': '0'} 2023-07-29T00:39:39.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006269299742aff7d5-0064c45fcb-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.686 DEBUG:boto:b'hiiqos-59obj741000falseobj742023-07-29T00:37:53.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:39.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-07-29T00:39:39.688 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:39:39.689 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:39:39.690 DEBUG:boto:path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:39:39.690 DEBUG:boto:auth_path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:39:39.691 DEBUG:boto:Method: GET 2023-07-29T00:39:39.691 DEBUG:boto:Path: /hiiqos-59/obj76?versionId=null 2023-07-29T00:39:39.692 DEBUG:boto:Data: 2023-07-29T00:39:39.692 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.693 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:39.693 DEBUG:boto:Port: 8001 2023-07-29T00:39:39.694 DEBUG:boto:Params: {} 2023-07-29T00:39:39.694 DEBUG:boto:Token: None 2023-07-29T00:39:39.695 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj76?versionId=null 2023-07-29T00:39:39.695 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BzxbcgJWvmgMPxCslCKZZWJei3I= 2023-07-29T00:39:39.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BzxbcgJWvmgMPxCslCKZZWJei3I=', 'Content-Length': '0'} 2023-07-29T00:39:39.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a81f168481850970-0064c45fcb-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.699 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:39:39.699 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:39:39.700 DEBUG:boto:path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:39:39.700 DEBUG:boto:auth_path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:39:39.701 DEBUG:boto:Method: GET 2023-07-29T00:39:39.701 DEBUG:boto:Path: /hiiqos-59/obj76?versionId=null 2023-07-29T00:39:39.701 DEBUG:boto:Data: 2023-07-29T00:39:39.702 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.702 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:39.703 DEBUG:boto:Port: 8000 2023-07-29T00:39:39.703 DEBUG:boto:Params: {} 2023-07-29T00:39:39.703 DEBUG:boto:Token: None 2023-07-29T00:39:39.704 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj76?versionId=null 2023-07-29T00:39:39.704 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BzxbcgJWvmgMPxCslCKZZWJei3I= 2023-07-29T00:39:39.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BzxbcgJWvmgMPxCslCKZZWJei3I=', 'Content-Length': '0'} 2023-07-29T00:39:39.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fd6e7dcacc57f31-0064c45fcb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.708 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:39:39.708 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:39:39.709 DEBUG:boto:path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:39:39.709 DEBUG:boto:auth_path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:39:39.709 DEBUG:boto:Method: HEAD 2023-07-29T00:39:39.710 DEBUG:boto:Path: /hiiqos-59/obj76?versionId=null 2023-07-29T00:39:39.710 DEBUG:boto:Data: 2023-07-29T00:39:39.711 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.711 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:39.711 DEBUG:boto:Port: 8001 2023-07-29T00:39:39.712 DEBUG:boto:Params: {} 2023-07-29T00:39:39.712 DEBUG:boto:Token: None 2023-07-29T00:39:39.713 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj76?versionId=null 2023-07-29T00:39:39.713 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ewef9nvDhQqQUwYjaUZDTeeFwWw= 2023-07-29T00:39:39.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ewef9nvDhQqQUwYjaUZDTeeFwWw=', 'Content-Length': '0'} 2023-07-29T00:39:39.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fb9a9404d48a8fd-0064c45fcb-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.716 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:39:39.716 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:39:39.717 DEBUG:boto:path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:39:39.717 DEBUG:boto:auth_path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:39:39.718 DEBUG:boto:Method: HEAD 2023-07-29T00:39:39.718 DEBUG:boto:Path: /hiiqos-59/obj76?versionId=null 2023-07-29T00:39:39.718 DEBUG:boto:Data: 2023-07-29T00:39:39.719 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.719 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:39.719 DEBUG:boto:Port: 8000 2023-07-29T00:39:39.720 DEBUG:boto:Params: {} 2023-07-29T00:39:39.720 DEBUG:boto:Token: None 2023-07-29T00:39:39.721 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj76?versionId=null 2023-07-29T00:39:39.721 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ewef9nvDhQqQUwYjaUZDTeeFwWw= 2023-07-29T00:39:39.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ewef9nvDhQqQUwYjaUZDTeeFwWw=', 'Content-Length': '0'} 2023-07-29T00:39:39.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e31fce13eda9473d-0064c45fcb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-07-29T00:39:39.724 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:39:39.725 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:39:39.725 DEBUG:boto:path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:39:39.726 DEBUG:boto:auth_path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:39:39.726 DEBUG:boto:Method: GET 2023-07-29T00:39:39.726 DEBUG:boto:Path: /hiiqos-59/obj76?versionId=null 2023-07-29T00:39:39.727 DEBUG:boto:Data: 2023-07-29T00:39:39.727 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.728 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:39.728 DEBUG:boto:Port: 8001 2023-07-29T00:39:39.729 DEBUG:boto:Params: {} 2023-07-29T00:39:39.729 DEBUG:boto:Token: None 2023-07-29T00:39:39.729 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj76?versionId=null 2023-07-29T00:39:39.730 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BzxbcgJWvmgMPxCslCKZZWJei3I= 2023-07-29T00:39:39.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BzxbcgJWvmgMPxCslCKZZWJei3I=', 'Content-Length': '0'} 2023-07-29T00:39:39.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002072b202923f60cd-0064c45fcb-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.733 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:39:39.734 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:39:39.734 DEBUG:boto:path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:39:39.734 DEBUG:boto:auth_path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:39:39.735 DEBUG:boto:Method: GET 2023-07-29T00:39:39.735 DEBUG:boto:Path: /hiiqos-59/obj76?versionId=null 2023-07-29T00:39:39.736 DEBUG:boto:Data: 2023-07-29T00:39:39.736 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.736 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:39.737 DEBUG:boto:Port: 8000 2023-07-29T00:39:39.737 DEBUG:boto:Params: {} 2023-07-29T00:39:39.738 DEBUG:boto:Token: None 2023-07-29T00:39:39.738 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj76?versionId=null 2023-07-29T00:39:39.739 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BzxbcgJWvmgMPxCslCKZZWJei3I= 2023-07-29T00:39:39.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BzxbcgJWvmgMPxCslCKZZWJei3I=', 'Content-Length': '0'} 2023-07-29T00:39:39.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b9a8b496ff32d4c-0064c45fcb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.742 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:39.742 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:39.743 DEBUG:boto:path=/hiiqos-59/?prefix=obj76 2023-07-29T00:39:39.743 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj76 2023-07-29T00:39:39.743 DEBUG:boto:Method: GET 2023-07-29T00:39:39.744 DEBUG:boto:Path: /hiiqos-59/?prefix=obj76 2023-07-29T00:39:39.744 DEBUG:boto:Data: 2023-07-29T00:39:39.745 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.745 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:39.745 DEBUG:boto:Port: 8001 2023-07-29T00:39:39.746 DEBUG:boto:Params: {} 2023-07-29T00:39:39.746 DEBUG:boto:Token: None 2023-07-29T00:39:39.747 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/ 2023-07-29T00:39:39.747 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qK1sfPK+p7ymNycEAB5Yr1COdqw= 2023-07-29T00:39:39.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qK1sfPK+p7ymNycEAB5Yr1COdqw=', 'Content-Length': '0'} 2023-07-29T00:39:39.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009dde3e885c405a6e-0064c45fcb-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.792 DEBUG:boto:b'hiiqos-59obj761000falseobj762023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:39.793 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:39.794 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:39.794 DEBUG:boto:path=/hiiqos-59/?prefix=obj76 2023-07-29T00:39:39.794 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj76 2023-07-29T00:39:39.795 DEBUG:boto:Method: GET 2023-07-29T00:39:39.795 DEBUG:boto:Path: /hiiqos-59/?prefix=obj76 2023-07-29T00:39:39.795 DEBUG:boto:Data: 2023-07-29T00:39:39.796 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.796 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:39.796 DEBUG:boto:Port: 8000 2023-07-29T00:39:39.797 DEBUG:boto:Params: {} 2023-07-29T00:39:39.797 DEBUG:boto:Token: None 2023-07-29T00:39:39.798 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/ 2023-07-29T00:39:39.798 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qK1sfPK+p7ymNycEAB5Yr1COdqw= 2023-07-29T00:39:39.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qK1sfPK+p7ymNycEAB5Yr1COdqw=', 'Content-Length': '0'} 2023-07-29T00:39:39.803 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d152721432389cfa-0064c45fcb-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.804 DEBUG:boto:b'hiiqos-59obj761000falseobj762023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:39.804 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:39:39.805 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:39:39.805 DEBUG:boto:Method: HEAD 2023-07-29T00:39:39.805 DEBUG:boto:Path: /hiiqos-59/obj76 2023-07-29T00:39:39.806 DEBUG:boto:Data: 2023-07-29T00:39:39.806 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.806 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:39.807 DEBUG:boto:Port: 8001 2023-07-29T00:39:39.807 DEBUG:boto:Params: {} 2023-07-29T00:39:39.807 DEBUG:boto:Token: None 2023-07-29T00:39:39.808 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj76 2023-07-29T00:39:39.809 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JXpGIihA8LAJih0tY6Yu6lEiiMo= 2023-07-29T00:39:39.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JXpGIihA8LAJih0tY6Yu6lEiiMo=', 'Content-Length': '0'} 2023-07-29T00:39:39.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b6fb9b45a53cbf0-0064c45fcb-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.811 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:39:39.812 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:39:39.812 DEBUG:boto:Method: HEAD 2023-07-29T00:39:39.813 DEBUG:boto:Path: /hiiqos-59/obj76 2023-07-29T00:39:39.813 DEBUG:boto:Data: 2023-07-29T00:39:39.813 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.814 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:39.814 DEBUG:boto:Port: 8000 2023-07-29T00:39:39.814 DEBUG:boto:Params: {} 2023-07-29T00:39:39.815 DEBUG:boto:Token: None 2023-07-29T00:39:39.815 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj76 2023-07-29T00:39:39.815 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JXpGIihA8LAJih0tY6Yu6lEiiMo= 2023-07-29T00:39:39.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JXpGIihA8LAJih0tY6Yu6lEiiMo=', 'Content-Length': '0'} 2023-07-29T00:39:39.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e57e2952e5a0e6e1-0064c45fcb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-07-29T00:39:39.819 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:39:39.819 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:39:39.819 DEBUG:boto:Method: GET 2023-07-29T00:39:39.820 DEBUG:boto:Path: /hiiqos-59/obj76 2023-07-29T00:39:39.820 DEBUG:boto:Data: 2023-07-29T00:39:39.821 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.821 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:39.821 DEBUG:boto:Port: 8001 2023-07-29T00:39:39.822 DEBUG:boto:Params: {} 2023-07-29T00:39:39.822 DEBUG:boto:Token: None 2023-07-29T00:39:39.822 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj76 2023-07-29T00:39:39.823 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MCr/dAKn2/YNV7SdylYl+ly01C4= 2023-07-29T00:39:39.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MCr/dAKn2/YNV7SdylYl+ly01C4=', 'Content-Length': '0'} 2023-07-29T00:39:39.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000685d7c08017fa59b-0064c45fcb-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.826 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:39:39.826 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:39:39.826 DEBUG:boto:Method: GET 2023-07-29T00:39:39.827 DEBUG:boto:Path: /hiiqos-59/obj76 2023-07-29T00:39:39.827 DEBUG:boto:Data: 2023-07-29T00:39:39.828 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.828 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:39.829 DEBUG:boto:Port: 8000 2023-07-29T00:39:39.830 DEBUG:boto:Params: {} 2023-07-29T00:39:39.831 DEBUG:boto:Token: None 2023-07-29T00:39:39.832 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj76 2023-07-29T00:39:39.833 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MCr/dAKn2/YNV7SdylYl+ly01C4= 2023-07-29T00:39:39.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MCr/dAKn2/YNV7SdylYl+ly01C4=', 'Content-Length': '0'} 2023-07-29T00:39:39.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8ddd538743f5d3d-0064c45fcb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.880 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:39.881 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:39.881 DEBUG:boto:path=/hiiqos-59/?prefix=obj76 2023-07-29T00:39:39.882 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj76 2023-07-29T00:39:39.882 DEBUG:boto:Method: GET 2023-07-29T00:39:39.882 DEBUG:boto:Path: /hiiqos-59/?prefix=obj76 2023-07-29T00:39:39.883 DEBUG:boto:Data: 2023-07-29T00:39:39.883 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.883 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:39.884 DEBUG:boto:Port: 8001 2023-07-29T00:39:39.884 DEBUG:boto:Params: {} 2023-07-29T00:39:39.885 DEBUG:boto:Token: None 2023-07-29T00:39:39.886 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/ 2023-07-29T00:39:39.887 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qK1sfPK+p7ymNycEAB5Yr1COdqw= 2023-07-29T00:39:39.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qK1sfPK+p7ymNycEAB5Yr1COdqw=', 'Content-Length': '0'} 2023-07-29T00:39:39.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d849ee4c5a52b10-0064c45fcb-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.892 DEBUG:boto:b'hiiqos-59obj761000falseobj762023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:39.893 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:39.893 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:39.894 DEBUG:boto:path=/hiiqos-59/?prefix=obj76 2023-07-29T00:39:39.895 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj76 2023-07-29T00:39:39.895 DEBUG:boto:Method: GET 2023-07-29T00:39:39.896 DEBUG:boto:Path: /hiiqos-59/?prefix=obj76 2023-07-29T00:39:39.897 DEBUG:boto:Data: 2023-07-29T00:39:39.897 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.898 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:39.899 DEBUG:boto:Port: 8000 2023-07-29T00:39:39.899 DEBUG:boto:Params: {} 2023-07-29T00:39:39.900 DEBUG:boto:Token: None 2023-07-29T00:39:39.901 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/ 2023-07-29T00:39:39.902 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qK1sfPK+p7ymNycEAB5Yr1COdqw= 2023-07-29T00:39:39.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qK1sfPK+p7ymNycEAB5Yr1COdqw=', 'Content-Length': '0'} 2023-07-29T00:39:39.907 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073f5063267016148-0064c45fcb-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.909 DEBUG:boto:b'hiiqos-59obj761000falseobj762023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:39.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-07-29T00:39:39.911 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:39:39.912 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:39:39.912 DEBUG:boto:path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:39:39.913 DEBUG:boto:auth_path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:39:39.913 DEBUG:boto:Method: GET 2023-07-29T00:39:39.914 DEBUG:boto:Path: /hiiqos-59/obj85?versionId=null 2023-07-29T00:39:39.914 DEBUG:boto:Data: 2023-07-29T00:39:39.914 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.915 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:39.915 DEBUG:boto:Port: 8001 2023-07-29T00:39:39.916 DEBUG:boto:Params: {} 2023-07-29T00:39:39.916 DEBUG:boto:Token: None 2023-07-29T00:39:39.917 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj85?versionId=null 2023-07-29T00:39:39.917 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZnE2LkZYHVcw8eykQ5OtxkJHnLc= 2023-07-29T00:39:39.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZnE2LkZYHVcw8eykQ5OtxkJHnLc=', 'Content-Length': '0'} 2023-07-29T00:39:39.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb9878011f7ef715-0064c45fcb-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.920 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:39:39.920 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:39:39.921 DEBUG:boto:path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:39:39.921 DEBUG:boto:auth_path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:39:39.922 DEBUG:boto:Method: GET 2023-07-29T00:39:39.922 DEBUG:boto:Path: /hiiqos-59/obj85?versionId=null 2023-07-29T00:39:39.922 DEBUG:boto:Data: 2023-07-29T00:39:39.923 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.923 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:39.923 DEBUG:boto:Port: 8000 2023-07-29T00:39:39.924 DEBUG:boto:Params: {} 2023-07-29T00:39:39.924 DEBUG:boto:Token: None 2023-07-29T00:39:39.925 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj85?versionId=null 2023-07-29T00:39:39.925 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZnE2LkZYHVcw8eykQ5OtxkJHnLc= 2023-07-29T00:39:39.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZnE2LkZYHVcw8eykQ5OtxkJHnLc=', 'Content-Length': '0'} 2023-07-29T00:39:39.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046659bff5833e383-0064c45fcb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.928 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:39:39.928 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:39:39.929 DEBUG:boto:path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:39:39.929 DEBUG:boto:auth_path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:39:39.929 DEBUG:boto:Method: HEAD 2023-07-29T00:39:39.930 DEBUG:boto:Path: /hiiqos-59/obj85?versionId=null 2023-07-29T00:39:39.930 DEBUG:boto:Data: 2023-07-29T00:39:39.930 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.931 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:39.931 DEBUG:boto:Port: 8001 2023-07-29T00:39:39.931 DEBUG:boto:Params: {} 2023-07-29T00:39:39.932 DEBUG:boto:Token: None 2023-07-29T00:39:39.932 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj85?versionId=null 2023-07-29T00:39:39.933 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QL2Eja+gqkd4kdGcIEMY6j1/by0= 2023-07-29T00:39:39.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QL2Eja+gqkd4kdGcIEMY6j1/by0=', 'Content-Length': '0'} 2023-07-29T00:39:39.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d98d3f3a2a32e7b-0064c45fcb-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.936 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:39:39.936 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:39:39.936 DEBUG:boto:path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:39:39.937 DEBUG:boto:auth_path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:39:39.937 DEBUG:boto:Method: HEAD 2023-07-29T00:39:39.937 DEBUG:boto:Path: /hiiqos-59/obj85?versionId=null 2023-07-29T00:39:39.938 DEBUG:boto:Data: 2023-07-29T00:39:39.938 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.938 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:39.939 DEBUG:boto:Port: 8000 2023-07-29T00:39:39.939 DEBUG:boto:Params: {} 2023-07-29T00:39:39.939 DEBUG:boto:Token: None 2023-07-29T00:39:39.940 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj85?versionId=null 2023-07-29T00:39:39.940 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QL2Eja+gqkd4kdGcIEMY6j1/by0= 2023-07-29T00:39:39.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QL2Eja+gqkd4kdGcIEMY6j1/by0=', 'Content-Length': '0'} 2023-07-29T00:39:39.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a150eb3609c35244-0064c45fcb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-07-29T00:39:39.943 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:39:39.944 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:39:39.944 DEBUG:boto:path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:39:39.945 DEBUG:boto:auth_path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:39:39.945 DEBUG:boto:Method: GET 2023-07-29T00:39:39.945 DEBUG:boto:Path: /hiiqos-59/obj85?versionId=null 2023-07-29T00:39:39.946 DEBUG:boto:Data: 2023-07-29T00:39:39.946 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.946 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:39.947 DEBUG:boto:Port: 8001 2023-07-29T00:39:39.947 DEBUG:boto:Params: {} 2023-07-29T00:39:39.947 DEBUG:boto:Token: None 2023-07-29T00:39:39.948 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj85?versionId=null 2023-07-29T00:39:39.948 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZnE2LkZYHVcw8eykQ5OtxkJHnLc= 2023-07-29T00:39:39.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZnE2LkZYHVcw8eykQ5OtxkJHnLc=', 'Content-Length': '0'} 2023-07-29T00:39:39.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015f6826dfe0055f9-0064c45fcb-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.951 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:39:39.952 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:39:39.952 DEBUG:boto:path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:39:39.953 DEBUG:boto:auth_path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:39:39.953 DEBUG:boto:Method: GET 2023-07-29T00:39:39.953 DEBUG:boto:Path: /hiiqos-59/obj85?versionId=null 2023-07-29T00:39:39.954 DEBUG:boto:Data: 2023-07-29T00:39:39.954 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.954 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:39.955 DEBUG:boto:Port: 8000 2023-07-29T00:39:39.955 DEBUG:boto:Params: {} 2023-07-29T00:39:39.956 DEBUG:boto:Token: None 2023-07-29T00:39:39.956 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj85?versionId=null 2023-07-29T00:39:39.956 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZnE2LkZYHVcw8eykQ5OtxkJHnLc= 2023-07-29T00:39:39.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZnE2LkZYHVcw8eykQ5OtxkJHnLc=', 'Content-Length': '0'} 2023-07-29T00:39:39.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074b3263de85c87de-0064c45fcb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.959 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:39.960 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:39.960 DEBUG:boto:path=/hiiqos-59/?prefix=obj85 2023-07-29T00:39:39.961 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj85 2023-07-29T00:39:39.961 DEBUG:boto:Method: GET 2023-07-29T00:39:39.961 DEBUG:boto:Path: /hiiqos-59/?prefix=obj85 2023-07-29T00:39:39.962 DEBUG:boto:Data: 2023-07-29T00:39:39.962 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.962 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:39.963 DEBUG:boto:Port: 8001 2023-07-29T00:39:39.963 DEBUG:boto:Params: {} 2023-07-29T00:39:39.964 DEBUG:boto:Token: None 2023-07-29T00:39:39.964 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/ 2023-07-29T00:39:39.964 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qK1sfPK+p7ymNycEAB5Yr1COdqw= 2023-07-29T00:39:39.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qK1sfPK+p7ymNycEAB5Yr1COdqw=', 'Content-Length': '0'} 2023-07-29T00:39:39.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009574ff1b125ef3a6-0064c45fcb-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.968 DEBUG:boto:b'hiiqos-59obj851000falseobj852023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:39.968 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:39.969 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:39.969 DEBUG:boto:path=/hiiqos-59/?prefix=obj85 2023-07-29T00:39:39.970 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj85 2023-07-29T00:39:39.970 DEBUG:boto:Method: GET 2023-07-29T00:39:39.970 DEBUG:boto:Path: /hiiqos-59/?prefix=obj85 2023-07-29T00:39:39.971 DEBUG:boto:Data: 2023-07-29T00:39:39.971 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.971 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:39.972 DEBUG:boto:Port: 8000 2023-07-29T00:39:39.972 DEBUG:boto:Params: {} 2023-07-29T00:39:39.972 DEBUG:boto:Token: None 2023-07-29T00:39:39.973 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/ 2023-07-29T00:39:39.973 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qK1sfPK+p7ymNycEAB5Yr1COdqw= 2023-07-29T00:39:39.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qK1sfPK+p7ymNycEAB5Yr1COdqw=', 'Content-Length': '0'} 2023-07-29T00:39:39.978 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092caf470c9c2ddc7-0064c45fcb-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.979 DEBUG:boto:b'hiiqos-59obj851000falseobj852023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:39.979 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:39:39.980 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:39:39.980 DEBUG:boto:Method: HEAD 2023-07-29T00:39:39.981 DEBUG:boto:Path: /hiiqos-59/obj85 2023-07-29T00:39:39.981 DEBUG:boto:Data: 2023-07-29T00:39:39.981 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.982 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:39.982 DEBUG:boto:Port: 8001 2023-07-29T00:39:39.982 DEBUG:boto:Params: {} 2023-07-29T00:39:39.983 DEBUG:boto:Token: None 2023-07-29T00:39:39.983 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj85 2023-07-29T00:39:39.984 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:y6RMbOi/QuLhBrPwdTAgBfty1MI= 2023-07-29T00:39:39.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:y6RMbOi/QuLhBrPwdTAgBfty1MI=', 'Content-Length': '0'} 2023-07-29T00:39:39.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000436853fbdf2344ef-0064c45fcb-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.986 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:39:39.987 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:39:39.987 DEBUG:boto:Method: HEAD 2023-07-29T00:39:39.988 DEBUG:boto:Path: /hiiqos-59/obj85 2023-07-29T00:39:39.988 DEBUG:boto:Data: 2023-07-29T00:39:39.988 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.989 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:39.989 DEBUG:boto:Port: 8000 2023-07-29T00:39:39.989 DEBUG:boto:Params: {} 2023-07-29T00:39:39.990 DEBUG:boto:Token: None 2023-07-29T00:39:39.990 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj85 2023-07-29T00:39:39.990 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:y6RMbOi/QuLhBrPwdTAgBfty1MI= 2023-07-29T00:39:39.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:y6RMbOi/QuLhBrPwdTAgBfty1MI=', 'Content-Length': '0'} 2023-07-29T00:39:39.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b28be406b22201b5-0064c45fcb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:39.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-07-29T00:39:39.993 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:39:39.994 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:39:39.994 DEBUG:boto:Method: GET 2023-07-29T00:39:39.995 DEBUG:boto:Path: /hiiqos-59/obj85 2023-07-29T00:39:39.995 DEBUG:boto:Data: 2023-07-29T00:39:39.995 DEBUG:boto:Headers: {} 2023-07-29T00:39:39.996 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:39.996 DEBUG:boto:Port: 8001 2023-07-29T00:39:39.996 DEBUG:boto:Params: {} 2023-07-29T00:39:39.997 DEBUG:boto:Token: None 2023-07-29T00:39:39.997 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:39 GMT /hiiqos-59/obj85 2023-07-29T00:39:39.998 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CCJW7KfPqkFt5WnFqG5YmkKpQds= 2023-07-29T00:39:39.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:39 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CCJW7KfPqkFt5WnFqG5YmkKpQds=', 'Content-Length': '0'} 2023-07-29T00:39:40.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9816a37b8c5d599-0064c45fcb-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.001 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:39:40.001 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:39:40.001 DEBUG:boto:Method: GET 2023-07-29T00:39:40.002 DEBUG:boto:Path: /hiiqos-59/obj85 2023-07-29T00:39:40.002 DEBUG:boto:Data: 2023-07-29T00:39:40.003 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.003 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:40.003 DEBUG:boto:Port: 8000 2023-07-29T00:39:40.004 DEBUG:boto:Params: {} 2023-07-29T00:39:40.004 DEBUG:boto:Token: None 2023-07-29T00:39:40.004 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/obj85 2023-07-29T00:39:40.005 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:83jynzQDZ8gFu/syH1myCunTdq4= 2023-07-29T00:39:40.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:83jynzQDZ8gFu/syH1myCunTdq4=', 'Content-Length': '0'} 2023-07-29T00:39:40.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0bf9d4124df80de-0064c45fcb-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.048 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:40.049 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:40.049 DEBUG:boto:path=/hiiqos-59/?prefix=obj85 2023-07-29T00:39:40.050 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj85 2023-07-29T00:39:40.050 DEBUG:boto:Method: GET 2023-07-29T00:39:40.050 DEBUG:boto:Path: /hiiqos-59/?prefix=obj85 2023-07-29T00:39:40.051 DEBUG:boto:Data: 2023-07-29T00:39:40.051 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.051 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:40.052 DEBUG:boto:Port: 8001 2023-07-29T00:39:40.052 DEBUG:boto:Params: {} 2023-07-29T00:39:40.053 DEBUG:boto:Token: None 2023-07-29T00:39:40.053 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/ 2023-07-29T00:39:40.053 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:giRBAYclmTvBX66mnmk5f6TQ5ow= 2023-07-29T00:39:40.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:giRBAYclmTvBX66mnmk5f6TQ5ow=', 'Content-Length': '0'} 2023-07-29T00:39:40.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098ff3e0bd68a7bf2-0064c45fcb-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:39 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.057 DEBUG:boto:b'hiiqos-59obj851000falseobj852023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:40.058 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:40.059 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:40.059 DEBUG:boto:path=/hiiqos-59/?prefix=obj85 2023-07-29T00:39:40.060 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj85 2023-07-29T00:39:40.060 DEBUG:boto:Method: GET 2023-07-29T00:39:40.060 DEBUG:boto:Path: /hiiqos-59/?prefix=obj85 2023-07-29T00:39:40.061 DEBUG:boto:Data: 2023-07-29T00:39:40.061 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.061 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:40.062 DEBUG:boto:Port: 8000 2023-07-29T00:39:40.062 DEBUG:boto:Params: {} 2023-07-29T00:39:40.063 DEBUG:boto:Token: None 2023-07-29T00:39:40.063 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/ 2023-07-29T00:39:40.063 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:giRBAYclmTvBX66mnmk5f6TQ5ow= 2023-07-29T00:39:40.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:giRBAYclmTvBX66mnmk5f6TQ5ow=', 'Content-Length': '0'} 2023-07-29T00:39:40.068 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bdbb3a07a6d6c1e-0064c45fcc-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.069 DEBUG:boto:b'hiiqos-59obj851000falseobj852023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:40.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-07-29T00:39:40.070 DEBUG:boto:path=/hiiqos-59/obj87 2023-07-29T00:39:40.070 DEBUG:boto:auth_path=/hiiqos-59/obj87 2023-07-29T00:39:40.071 DEBUG:boto:path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:39:40.071 DEBUG:boto:auth_path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:39:40.071 DEBUG:boto:Method: GET 2023-07-29T00:39:40.072 DEBUG:boto:Path: /hiiqos-59/obj87?versionId=null 2023-07-29T00:39:40.072 DEBUG:boto:Data: 2023-07-29T00:39:40.073 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.073 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:40.073 DEBUG:boto:Port: 8001 2023-07-29T00:39:40.074 DEBUG:boto:Params: {} 2023-07-29T00:39:40.074 DEBUG:boto:Token: None 2023-07-29T00:39:40.074 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/obj87?versionId=null 2023-07-29T00:39:40.075 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7Wl3yIKskuGPwJ7rJ2333RYqHzg= 2023-07-29T00:39:40.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7Wl3yIKskuGPwJ7rJ2333RYqHzg=', 'Content-Length': '0'} 2023-07-29T00:39:40.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028560b8fb0ca72a6-0064c45fcc-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.078 DEBUG:boto:path=/hiiqos-59/obj87 2023-07-29T00:39:40.078 DEBUG:boto:auth_path=/hiiqos-59/obj87 2023-07-29T00:39:40.079 DEBUG:boto:path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:39:40.079 DEBUG:boto:auth_path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:39:40.080 DEBUG:boto:Method: GET 2023-07-29T00:39:40.080 DEBUG:boto:Path: /hiiqos-59/obj87?versionId=null 2023-07-29T00:39:40.080 DEBUG:boto:Data: 2023-07-29T00:39:40.081 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.081 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:40.082 DEBUG:boto:Port: 8000 2023-07-29T00:39:40.082 DEBUG:boto:Params: {} 2023-07-29T00:39:40.082 DEBUG:boto:Token: None 2023-07-29T00:39:40.083 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/obj87?versionId=null 2023-07-29T00:39:40.083 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7Wl3yIKskuGPwJ7rJ2333RYqHzg= 2023-07-29T00:39:40.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7Wl3yIKskuGPwJ7rJ2333RYqHzg=', 'Content-Length': '0'} 2023-07-29T00:39:40.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5abce806e583a2e-0064c45fcc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.086 DEBUG:boto:path=/hiiqos-59/obj87 2023-07-29T00:39:40.087 DEBUG:boto:auth_path=/hiiqos-59/obj87 2023-07-29T00:39:40.087 DEBUG:boto:path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:39:40.088 DEBUG:boto:auth_path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:39:40.089 DEBUG:boto:Method: HEAD 2023-07-29T00:39:40.090 DEBUG:boto:Path: /hiiqos-59/obj87?versionId=null 2023-07-29T00:39:40.090 DEBUG:boto:Data: 2023-07-29T00:39:40.091 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.092 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:40.092 DEBUG:boto:Port: 8001 2023-07-29T00:39:40.093 DEBUG:boto:Params: {} 2023-07-29T00:39:40.094 DEBUG:boto:Token: None 2023-07-29T00:39:40.095 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/obj87?versionId=null 2023-07-29T00:39:40.096 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7W8KYSuJrkh3+HwKJSUABRhC1sY= 2023-07-29T00:39:40.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7W8KYSuJrkh3+HwKJSUABRhC1sY=', 'Content-Length': '0'} 2023-07-29T00:39:40.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d7725bbfcc28967-0064c45fcc-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.100 DEBUG:boto:path=/hiiqos-59/obj87 2023-07-29T00:39:40.101 DEBUG:boto:auth_path=/hiiqos-59/obj87 2023-07-29T00:39:40.101 DEBUG:boto:path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:39:40.102 DEBUG:boto:auth_path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:39:40.102 DEBUG:boto:Method: HEAD 2023-07-29T00:39:40.103 DEBUG:boto:Path: /hiiqos-59/obj87?versionId=null 2023-07-29T00:39:40.103 DEBUG:boto:Data: 2023-07-29T00:39:40.104 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.104 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:40.105 DEBUG:boto:Port: 8000 2023-07-29T00:39:40.105 DEBUG:boto:Params: {} 2023-07-29T00:39:40.106 DEBUG:boto:Token: None 2023-07-29T00:39:40.106 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/obj87?versionId=null 2023-07-29T00:39:40.107 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7W8KYSuJrkh3+HwKJSUABRhC1sY= 2023-07-29T00:39:40.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7W8KYSuJrkh3+HwKJSUABRhC1sY=', 'Content-Length': '0'} 2023-07-29T00:39:40.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010d32430e88ddc12-0064c45fcc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-07-29T00:39:40.110 DEBUG:boto:path=/hiiqos-59/obj87 2023-07-29T00:39:40.110 DEBUG:boto:auth_path=/hiiqos-59/obj87 2023-07-29T00:39:40.111 DEBUG:boto:path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:39:40.111 DEBUG:boto:auth_path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:39:40.111 DEBUG:boto:Method: GET 2023-07-29T00:39:40.112 DEBUG:boto:Path: /hiiqos-59/obj87?versionId=null 2023-07-29T00:39:40.112 DEBUG:boto:Data: 2023-07-29T00:39:40.113 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.113 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:40.114 DEBUG:boto:Port: 8001 2023-07-29T00:39:40.114 DEBUG:boto:Params: {} 2023-07-29T00:39:40.115 DEBUG:boto:Token: None 2023-07-29T00:39:40.115 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/obj87?versionId=null 2023-07-29T00:39:40.116 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7Wl3yIKskuGPwJ7rJ2333RYqHzg= 2023-07-29T00:39:40.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7Wl3yIKskuGPwJ7rJ2333RYqHzg=', 'Content-Length': '0'} 2023-07-29T00:39:40.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049ff51d48aaeb76c-0064c45fcc-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.160 DEBUG:boto:path=/hiiqos-59/obj87 2023-07-29T00:39:40.161 DEBUG:boto:auth_path=/hiiqos-59/obj87 2023-07-29T00:39:40.161 DEBUG:boto:path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:39:40.162 DEBUG:boto:auth_path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:39:40.162 DEBUG:boto:Method: GET 2023-07-29T00:39:40.163 DEBUG:boto:Path: /hiiqos-59/obj87?versionId=null 2023-07-29T00:39:40.163 DEBUG:boto:Data: 2023-07-29T00:39:40.163 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.164 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:40.164 DEBUG:boto:Port: 8000 2023-07-29T00:39:40.165 DEBUG:boto:Params: {} 2023-07-29T00:39:40.165 DEBUG:boto:Token: None 2023-07-29T00:39:40.166 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/obj87?versionId=null 2023-07-29T00:39:40.166 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7Wl3yIKskuGPwJ7rJ2333RYqHzg= 2023-07-29T00:39:40.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7Wl3yIKskuGPwJ7rJ2333RYqHzg=', 'Content-Length': '0'} 2023-07-29T00:39:40.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086fa4bac99157b5b-0064c45fcc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.170 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:40.170 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:40.171 DEBUG:boto:path=/hiiqos-59/?prefix=obj87 2023-07-29T00:39:40.171 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj87 2023-07-29T00:39:40.171 DEBUG:boto:Method: GET 2023-07-29T00:39:40.172 DEBUG:boto:Path: /hiiqos-59/?prefix=obj87 2023-07-29T00:39:40.172 DEBUG:boto:Data: 2023-07-29T00:39:40.172 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.173 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:40.173 DEBUG:boto:Port: 8001 2023-07-29T00:39:40.174 DEBUG:boto:Params: {} 2023-07-29T00:39:40.174 DEBUG:boto:Token: None 2023-07-29T00:39:40.174 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/ 2023-07-29T00:39:40.175 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:giRBAYclmTvBX66mnmk5f6TQ5ow= 2023-07-29T00:39:40.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:giRBAYclmTvBX66mnmk5f6TQ5ow=', 'Content-Length': '0'} 2023-07-29T00:39:40.178 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013404e61f8b323b9-0064c45fcc-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.178 DEBUG:boto:b'hiiqos-59obj871000falseobj872023-07-29T00:35:01.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:40.179 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:40.179 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:40.180 DEBUG:boto:path=/hiiqos-59/?prefix=obj87 2023-07-29T00:39:40.180 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj87 2023-07-29T00:39:40.181 DEBUG:boto:Method: GET 2023-07-29T00:39:40.181 DEBUG:boto:Path: /hiiqos-59/?prefix=obj87 2023-07-29T00:39:40.181 DEBUG:boto:Data: 2023-07-29T00:39:40.182 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.182 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:40.183 DEBUG:boto:Port: 8000 2023-07-29T00:39:40.183 DEBUG:boto:Params: {} 2023-07-29T00:39:40.184 DEBUG:boto:Token: None 2023-07-29T00:39:40.184 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/ 2023-07-29T00:39:40.185 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:giRBAYclmTvBX66mnmk5f6TQ5ow= 2023-07-29T00:39:40.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:giRBAYclmTvBX66mnmk5f6TQ5ow=', 'Content-Length': '0'} 2023-07-29T00:39:40.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000517ec915ef8dbc8c-0064c45fcc-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.232 DEBUG:boto:b'hiiqos-59obj871000falseobj872023-07-29T00:35:01.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:40.233 DEBUG:boto:path=/hiiqos-59/obj87 2023-07-29T00:39:40.234 DEBUG:boto:auth_path=/hiiqos-59/obj87 2023-07-29T00:39:40.234 DEBUG:boto:Method: HEAD 2023-07-29T00:39:40.235 DEBUG:boto:Path: /hiiqos-59/obj87 2023-07-29T00:39:40.235 DEBUG:boto:Data: 2023-07-29T00:39:40.235 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.236 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:40.236 DEBUG:boto:Port: 8001 2023-07-29T00:39:40.236 DEBUG:boto:Params: {} 2023-07-29T00:39:40.237 DEBUG:boto:Token: None 2023-07-29T00:39:40.237 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/obj87 2023-07-29T00:39:40.238 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:T6hEE+C2MpGWvGWJe0rfvGEKjxY= 2023-07-29T00:39:40.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:T6hEE+C2MpGWvGWJe0rfvGEKjxY=', 'Content-Length': '0'} 2023-07-29T00:39:40.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c42c64726c3eb8a9-0064c45fcc-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.241 DEBUG:boto:path=/hiiqos-59/obj87 2023-07-29T00:39:40.241 DEBUG:boto:auth_path=/hiiqos-59/obj87 2023-07-29T00:39:40.242 DEBUG:boto:Method: HEAD 2023-07-29T00:39:40.242 DEBUG:boto:Path: /hiiqos-59/obj87 2023-07-29T00:39:40.242 DEBUG:boto:Data: 2023-07-29T00:39:40.243 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.243 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:40.243 DEBUG:boto:Port: 8000 2023-07-29T00:39:40.244 DEBUG:boto:Params: {} 2023-07-29T00:39:40.244 DEBUG:boto:Token: None 2023-07-29T00:39:40.245 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/obj87 2023-07-29T00:39:40.245 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:T6hEE+C2MpGWvGWJe0rfvGEKjxY= 2023-07-29T00:39:40.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:T6hEE+C2MpGWvGWJe0rfvGEKjxY=', 'Content-Length': '0'} 2023-07-29T00:39:40.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044cae8ea37d85ecc-0064c45fcc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-07-29T00:39:40.248 DEBUG:boto:path=/hiiqos-59/obj87 2023-07-29T00:39:40.249 DEBUG:boto:auth_path=/hiiqos-59/obj87 2023-07-29T00:39:40.249 DEBUG:boto:Method: GET 2023-07-29T00:39:40.250 DEBUG:boto:Path: /hiiqos-59/obj87 2023-07-29T00:39:40.250 DEBUG:boto:Data: 2023-07-29T00:39:40.250 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.251 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:40.251 DEBUG:boto:Port: 8001 2023-07-29T00:39:40.251 DEBUG:boto:Params: {} 2023-07-29T00:39:40.252 DEBUG:boto:Token: None 2023-07-29T00:39:40.252 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/obj87 2023-07-29T00:39:40.253 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8YbSFCjftnDwgWG4OFRE0D+lfpA= 2023-07-29T00:39:40.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8YbSFCjftnDwgWG4OFRE0D+lfpA=', 'Content-Length': '0'} 2023-07-29T00:39:40.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000212fcd770df5eebc-0064c45fcc-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.255 DEBUG:boto:path=/hiiqos-59/obj87 2023-07-29T00:39:40.256 DEBUG:boto:auth_path=/hiiqos-59/obj87 2023-07-29T00:39:40.256 DEBUG:boto:Method: GET 2023-07-29T00:39:40.257 DEBUG:boto:Path: /hiiqos-59/obj87 2023-07-29T00:39:40.257 DEBUG:boto:Data: 2023-07-29T00:39:40.257 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.258 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:40.258 DEBUG:boto:Port: 8000 2023-07-29T00:39:40.259 DEBUG:boto:Params: {} 2023-07-29T00:39:40.259 DEBUG:boto:Token: None 2023-07-29T00:39:40.259 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/obj87 2023-07-29T00:39:40.260 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8YbSFCjftnDwgWG4OFRE0D+lfpA= 2023-07-29T00:39:40.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8YbSFCjftnDwgWG4OFRE0D+lfpA=', 'Content-Length': '0'} 2023-07-29T00:39:40.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000509d9214ef4e525b-0064c45fcc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.263 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:40.263 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:40.263 DEBUG:boto:path=/hiiqos-59/?prefix=obj87 2023-07-29T00:39:40.264 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj87 2023-07-29T00:39:40.265 DEBUG:boto:Method: GET 2023-07-29T00:39:40.265 DEBUG:boto:Path: /hiiqos-59/?prefix=obj87 2023-07-29T00:39:40.266 DEBUG:boto:Data: 2023-07-29T00:39:40.266 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.267 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:40.267 DEBUG:boto:Port: 8001 2023-07-29T00:39:40.267 DEBUG:boto:Params: {} 2023-07-29T00:39:40.268 DEBUG:boto:Token: None 2023-07-29T00:39:40.269 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/ 2023-07-29T00:39:40.269 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:giRBAYclmTvBX66mnmk5f6TQ5ow= 2023-07-29T00:39:40.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:giRBAYclmTvBX66mnmk5f6TQ5ow=', 'Content-Length': '0'} 2023-07-29T00:39:40.272 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b8e1c9b4fb845c6-0064c45fcc-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.273 DEBUG:boto:b'hiiqos-59obj871000falseobj872023-07-29T00:35:01.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:40.274 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:40.275 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:40.276 DEBUG:boto:path=/hiiqos-59/?prefix=obj87 2023-07-29T00:39:40.277 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj87 2023-07-29T00:39:40.278 DEBUG:boto:Method: GET 2023-07-29T00:39:40.279 DEBUG:boto:Path: /hiiqos-59/?prefix=obj87 2023-07-29T00:39:40.280 DEBUG:boto:Data: 2023-07-29T00:39:40.281 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.281 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:40.282 DEBUG:boto:Port: 8000 2023-07-29T00:39:40.283 DEBUG:boto:Params: {} 2023-07-29T00:39:40.283 DEBUG:boto:Token: None 2023-07-29T00:39:40.284 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/ 2023-07-29T00:39:40.285 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:giRBAYclmTvBX66mnmk5f6TQ5ow= 2023-07-29T00:39:40.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:giRBAYclmTvBX66mnmk5f6TQ5ow=', 'Content-Length': '0'} 2023-07-29T00:39:40.291 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049ae6e410ed873f6-0064c45fcc-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.291 DEBUG:boto:b'hiiqos-59obj871000falseobj872023-07-29T00:35:01.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:40.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-07-29T00:39:40.294 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:39:40.294 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:39:40.295 DEBUG:boto:path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:39:40.296 DEBUG:boto:auth_path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:39:40.297 DEBUG:boto:Method: GET 2023-07-29T00:39:40.297 DEBUG:boto:Path: /hiiqos-59/obj92?versionId=null 2023-07-29T00:39:40.297 DEBUG:boto:Data: 2023-07-29T00:39:40.298 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.298 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:40.298 DEBUG:boto:Port: 8001 2023-07-29T00:39:40.299 DEBUG:boto:Params: {} 2023-07-29T00:39:40.299 DEBUG:boto:Token: None 2023-07-29T00:39:40.300 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/obj92?versionId=null 2023-07-29T00:39:40.300 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dhNYxUOCH9B0CS5tbmIJeOKfVpE= 2023-07-29T00:39:40.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dhNYxUOCH9B0CS5tbmIJeOKfVpE=', 'Content-Length': '0'} 2023-07-29T00:39:40.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da9c026e865ab02a-0064c45fcc-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.303 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:39:40.304 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:39:40.304 DEBUG:boto:path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:39:40.305 DEBUG:boto:auth_path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:39:40.305 DEBUG:boto:Method: GET 2023-07-29T00:39:40.305 DEBUG:boto:Path: /hiiqos-59/obj92?versionId=null 2023-07-29T00:39:40.306 DEBUG:boto:Data: 2023-07-29T00:39:40.306 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.306 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:40.307 DEBUG:boto:Port: 8000 2023-07-29T00:39:40.307 DEBUG:boto:Params: {} 2023-07-29T00:39:40.307 DEBUG:boto:Token: None 2023-07-29T00:39:40.308 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/obj92?versionId=null 2023-07-29T00:39:40.309 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dhNYxUOCH9B0CS5tbmIJeOKfVpE= 2023-07-29T00:39:40.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dhNYxUOCH9B0CS5tbmIJeOKfVpE=', 'Content-Length': '0'} 2023-07-29T00:39:40.313 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000566dc961a9cabb94-0064c45fcc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.314 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:39:40.315 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:39:40.316 DEBUG:boto:path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:39:40.316 DEBUG:boto:auth_path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:39:40.317 DEBUG:boto:Method: HEAD 2023-07-29T00:39:40.317 DEBUG:boto:Path: /hiiqos-59/obj92?versionId=null 2023-07-29T00:39:40.318 DEBUG:boto:Data: 2023-07-29T00:39:40.319 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.320 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:40.320 DEBUG:boto:Port: 8001 2023-07-29T00:39:40.320 DEBUG:boto:Params: {} 2023-07-29T00:39:40.321 DEBUG:boto:Token: None 2023-07-29T00:39:40.321 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/obj92?versionId=null 2023-07-29T00:39:40.322 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mV5pZX+NcUJYa8JCc5iCD0L6Y8M= 2023-07-29T00:39:40.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mV5pZX+NcUJYa8JCc5iCD0L6Y8M=', 'Content-Length': '0'} 2023-07-29T00:39:40.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006351f6cf26fef2f4-0064c45fcc-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.325 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:39:40.325 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:39:40.325 DEBUG:boto:path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:39:40.326 DEBUG:boto:auth_path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:39:40.326 DEBUG:boto:Method: HEAD 2023-07-29T00:39:40.326 DEBUG:boto:Path: /hiiqos-59/obj92?versionId=null 2023-07-29T00:39:40.327 DEBUG:boto:Data: 2023-07-29T00:39:40.327 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.327 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:40.328 DEBUG:boto:Port: 8000 2023-07-29T00:39:40.328 DEBUG:boto:Params: {} 2023-07-29T00:39:40.328 DEBUG:boto:Token: None 2023-07-29T00:39:40.329 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/obj92?versionId=null 2023-07-29T00:39:40.329 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mV5pZX+NcUJYa8JCc5iCD0L6Y8M= 2023-07-29T00:39:40.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mV5pZX+NcUJYa8JCc5iCD0L6Y8M=', 'Content-Length': '0'} 2023-07-29T00:39:40.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077e34f3ff9446bb7-0064c45fcc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-07-29T00:39:40.333 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:39:40.333 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:39:40.333 DEBUG:boto:path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:39:40.334 DEBUG:boto:auth_path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:39:40.334 DEBUG:boto:Method: GET 2023-07-29T00:39:40.334 DEBUG:boto:Path: /hiiqos-59/obj92?versionId=null 2023-07-29T00:39:40.335 DEBUG:boto:Data: 2023-07-29T00:39:40.335 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.336 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:40.336 DEBUG:boto:Port: 8001 2023-07-29T00:39:40.336 DEBUG:boto:Params: {} 2023-07-29T00:39:40.337 DEBUG:boto:Token: None 2023-07-29T00:39:40.337 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/obj92?versionId=null 2023-07-29T00:39:40.338 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dhNYxUOCH9B0CS5tbmIJeOKfVpE= 2023-07-29T00:39:40.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dhNYxUOCH9B0CS5tbmIJeOKfVpE=', 'Content-Length': '0'} 2023-07-29T00:39:40.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cff33abd93615911-0064c45fcc-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.341 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:39:40.341 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:39:40.341 DEBUG:boto:path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:39:40.342 DEBUG:boto:auth_path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:39:40.342 DEBUG:boto:Method: GET 2023-07-29T00:39:40.343 DEBUG:boto:Path: /hiiqos-59/obj92?versionId=null 2023-07-29T00:39:40.343 DEBUG:boto:Data: 2023-07-29T00:39:40.343 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.344 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:40.344 DEBUG:boto:Port: 8000 2023-07-29T00:39:40.345 DEBUG:boto:Params: {} 2023-07-29T00:39:40.345 DEBUG:boto:Token: None 2023-07-29T00:39:40.346 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/obj92?versionId=null 2023-07-29T00:39:40.346 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dhNYxUOCH9B0CS5tbmIJeOKfVpE= 2023-07-29T00:39:40.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dhNYxUOCH9B0CS5tbmIJeOKfVpE=', 'Content-Length': '0'} 2023-07-29T00:39:40.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000126480ca3ef52e8d-0064c45fcc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.392 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:40.393 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:40.393 DEBUG:boto:path=/hiiqos-59/?prefix=obj92 2023-07-29T00:39:40.394 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj92 2023-07-29T00:39:40.394 DEBUG:boto:Method: GET 2023-07-29T00:39:40.394 DEBUG:boto:Path: /hiiqos-59/?prefix=obj92 2023-07-29T00:39:40.395 DEBUG:boto:Data: 2023-07-29T00:39:40.395 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.396 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:40.396 DEBUG:boto:Port: 8001 2023-07-29T00:39:40.396 DEBUG:boto:Params: {} 2023-07-29T00:39:40.397 DEBUG:boto:Token: None 2023-07-29T00:39:40.397 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/ 2023-07-29T00:39:40.398 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:giRBAYclmTvBX66mnmk5f6TQ5ow= 2023-07-29T00:39:40.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:giRBAYclmTvBX66mnmk5f6TQ5ow=', 'Content-Length': '0'} 2023-07-29T00:39:40.401 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c48ee19d6ec42a22-0064c45fcc-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.401 DEBUG:boto:b'hiiqos-59obj921000falseobj922023-07-29T00:33:06.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:40.402 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:40.402 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:40.403 DEBUG:boto:path=/hiiqos-59/?prefix=obj92 2023-07-29T00:39:40.403 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj92 2023-07-29T00:39:40.403 DEBUG:boto:Method: GET 2023-07-29T00:39:40.404 DEBUG:boto:Path: /hiiqos-59/?prefix=obj92 2023-07-29T00:39:40.404 DEBUG:boto:Data: 2023-07-29T00:39:40.405 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.405 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:40.405 DEBUG:boto:Port: 8000 2023-07-29T00:39:40.406 DEBUG:boto:Params: {} 2023-07-29T00:39:40.406 DEBUG:boto:Token: None 2023-07-29T00:39:40.406 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/ 2023-07-29T00:39:40.407 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:giRBAYclmTvBX66mnmk5f6TQ5ow= 2023-07-29T00:39:40.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:giRBAYclmTvBX66mnmk5f6TQ5ow=', 'Content-Length': '0'} 2023-07-29T00:39:40.411 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c088430761c6c84-0064c45fcc-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.412 DEBUG:boto:b'hiiqos-59obj921000falseobj922023-07-29T00:33:06.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:40.413 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:39:40.413 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:39:40.413 DEBUG:boto:Method: HEAD 2023-07-29T00:39:40.414 DEBUG:boto:Path: /hiiqos-59/obj92 2023-07-29T00:39:40.414 DEBUG:boto:Data: 2023-07-29T00:39:40.414 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.415 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:40.415 DEBUG:boto:Port: 8001 2023-07-29T00:39:40.416 DEBUG:boto:Params: {} 2023-07-29T00:39:40.416 DEBUG:boto:Token: None 2023-07-29T00:39:40.417 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/obj92 2023-07-29T00:39:40.417 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:L/I1ZL0DKCMeRrRkz4dTJ5v1ACM= 2023-07-29T00:39:40.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:L/I1ZL0DKCMeRrRkz4dTJ5v1ACM=', 'Content-Length': '0'} 2023-07-29T00:39:40.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee8ca6faee1e7c61-0064c45fcc-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.420 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:39:40.421 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:39:40.421 DEBUG:boto:Method: HEAD 2023-07-29T00:39:40.422 DEBUG:boto:Path: /hiiqos-59/obj92 2023-07-29T00:39:40.423 DEBUG:boto:Data: 2023-07-29T00:39:40.423 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.424 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:40.425 DEBUG:boto:Port: 8000 2023-07-29T00:39:40.425 DEBUG:boto:Params: {} 2023-07-29T00:39:40.426 DEBUG:boto:Token: None 2023-07-29T00:39:40.427 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/obj92 2023-07-29T00:39:40.428 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:L/I1ZL0DKCMeRrRkz4dTJ5v1ACM= 2023-07-29T00:39:40.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:L/I1ZL0DKCMeRrRkz4dTJ5v1ACM=', 'Content-Length': '0'} 2023-07-29T00:39:40.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f50ada876302009-0064c45fcc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-07-29T00:39:40.433 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:39:40.433 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:39:40.434 DEBUG:boto:Method: GET 2023-07-29T00:39:40.434 DEBUG:boto:Path: /hiiqos-59/obj92 2023-07-29T00:39:40.434 DEBUG:boto:Data: 2023-07-29T00:39:40.435 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.435 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:40.436 DEBUG:boto:Port: 8001 2023-07-29T00:39:40.436 DEBUG:boto:Params: {} 2023-07-29T00:39:40.437 DEBUG:boto:Token: None 2023-07-29T00:39:40.437 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/obj92 2023-07-29T00:39:40.438 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:52jvK6itz2+DQzz6G3TEKAvphuk= 2023-07-29T00:39:40.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:52jvK6itz2+DQzz6G3TEKAvphuk=', 'Content-Length': '0'} 2023-07-29T00:39:40.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a775cd53b048033-0064c45fcc-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.480 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:39:40.481 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:39:40.481 DEBUG:boto:Method: GET 2023-07-29T00:39:40.482 DEBUG:boto:Path: /hiiqos-59/obj92 2023-07-29T00:39:40.482 DEBUG:boto:Data: 2023-07-29T00:39:40.483 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.484 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:40.485 DEBUG:boto:Port: 8000 2023-07-29T00:39:40.486 DEBUG:boto:Params: {} 2023-07-29T00:39:40.487 DEBUG:boto:Token: None 2023-07-29T00:39:40.488 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/obj92 2023-07-29T00:39:40.489 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:52jvK6itz2+DQzz6G3TEKAvphuk= 2023-07-29T00:39:40.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:52jvK6itz2+DQzz6G3TEKAvphuk=', 'Content-Length': '0'} 2023-07-29T00:39:40.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a1aeb1a721733a5-0064c45fcc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.493 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:40.493 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:40.494 DEBUG:boto:path=/hiiqos-59/?prefix=obj92 2023-07-29T00:39:40.494 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj92 2023-07-29T00:39:40.494 DEBUG:boto:Method: GET 2023-07-29T00:39:40.495 DEBUG:boto:Path: /hiiqos-59/?prefix=obj92 2023-07-29T00:39:40.495 DEBUG:boto:Data: 2023-07-29T00:39:40.496 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.496 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:40.496 DEBUG:boto:Port: 8001 2023-07-29T00:39:40.497 DEBUG:boto:Params: {} 2023-07-29T00:39:40.497 DEBUG:boto:Token: None 2023-07-29T00:39:40.498 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/ 2023-07-29T00:39:40.498 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:giRBAYclmTvBX66mnmk5f6TQ5ow= 2023-07-29T00:39:40.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:giRBAYclmTvBX66mnmk5f6TQ5ow=', 'Content-Length': '0'} 2023-07-29T00:39:40.501 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f86397dc43838c28-0064c45fcc-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.502 DEBUG:boto:b'hiiqos-59obj921000falseobj922023-07-29T00:33:06.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:40.503 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:40.503 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:40.503 DEBUG:boto:path=/hiiqos-59/?prefix=obj92 2023-07-29T00:39:40.504 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj92 2023-07-29T00:39:40.504 DEBUG:boto:Method: GET 2023-07-29T00:39:40.505 DEBUG:boto:Path: /hiiqos-59/?prefix=obj92 2023-07-29T00:39:40.505 DEBUG:boto:Data: 2023-07-29T00:39:40.505 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.506 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:40.506 DEBUG:boto:Port: 8000 2023-07-29T00:39:40.506 DEBUG:boto:Params: {} 2023-07-29T00:39:40.507 DEBUG:boto:Token: None 2023-07-29T00:39:40.507 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/ 2023-07-29T00:39:40.508 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:giRBAYclmTvBX66mnmk5f6TQ5ow= 2023-07-29T00:39:40.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:giRBAYclmTvBX66mnmk5f6TQ5ow=', 'Content-Length': '0'} 2023-07-29T00:39:40.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005eb61e9e92519eb5-0064c45fcc-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.514 DEBUG:boto:b'hiiqos-59obj921000falseobj922023-07-29T00:33:06.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:40.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-07-29T00:39:40.515 DEBUG:boto:path=/hiiqos-59/obj93 2023-07-29T00:39:40.516 DEBUG:boto:auth_path=/hiiqos-59/obj93 2023-07-29T00:39:40.516 DEBUG:boto:path=/hiiqos-59/obj93?versionId=null 2023-07-29T00:39:40.517 DEBUG:boto:auth_path=/hiiqos-59/obj93?versionId=null 2023-07-29T00:39:40.517 DEBUG:boto:Method: GET 2023-07-29T00:39:40.517 DEBUG:boto:Path: /hiiqos-59/obj93?versionId=null 2023-07-29T00:39:40.518 DEBUG:boto:Data: 2023-07-29T00:39:40.518 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.518 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:40.519 DEBUG:boto:Port: 8001 2023-07-29T00:39:40.519 DEBUG:boto:Params: {} 2023-07-29T00:39:40.520 DEBUG:boto:Token: None 2023-07-29T00:39:40.520 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/obj93?versionId=null 2023-07-29T00:39:40.520 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:z3SBE3roErJ76HeLc9zBuqmq1cY= 2023-07-29T00:39:40.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:z3SBE3roErJ76HeLc9zBuqmq1cY=', 'Content-Length': '0'} 2023-07-29T00:39:40.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d64f9d093062ad90-0064c45fcc-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.524 DEBUG:boto:path=/hiiqos-59/obj93 2023-07-29T00:39:40.524 DEBUG:boto:auth_path=/hiiqos-59/obj93 2023-07-29T00:39:40.525 DEBUG:boto:path=/hiiqos-59/obj93?versionId=null 2023-07-29T00:39:40.525 DEBUG:boto:auth_path=/hiiqos-59/obj93?versionId=null 2023-07-29T00:39:40.525 DEBUG:boto:Method: GET 2023-07-29T00:39:40.526 DEBUG:boto:Path: /hiiqos-59/obj93?versionId=null 2023-07-29T00:39:40.526 DEBUG:boto:Data: 2023-07-29T00:39:40.526 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.527 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:40.527 DEBUG:boto:Port: 8000 2023-07-29T00:39:40.527 DEBUG:boto:Params: {} 2023-07-29T00:39:40.528 DEBUG:boto:Token: None 2023-07-29T00:39:40.528 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/obj93?versionId=null 2023-07-29T00:39:40.529 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:z3SBE3roErJ76HeLc9zBuqmq1cY= 2023-07-29T00:39:40.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:z3SBE3roErJ76HeLc9zBuqmq1cY=', 'Content-Length': '0'} 2023-07-29T00:39:40.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df19ca3e2ff5e6e6-0064c45fcc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.576 DEBUG:boto:path=/hiiqos-59/obj93 2023-07-29T00:39:40.577 DEBUG:boto:auth_path=/hiiqos-59/obj93 2023-07-29T00:39:40.577 DEBUG:boto:path=/hiiqos-59/obj93?versionId=null 2023-07-29T00:39:40.578 DEBUG:boto:auth_path=/hiiqos-59/obj93?versionId=null 2023-07-29T00:39:40.578 DEBUG:boto:Method: HEAD 2023-07-29T00:39:40.578 DEBUG:boto:Path: /hiiqos-59/obj93?versionId=null 2023-07-29T00:39:40.579 DEBUG:boto:Data: 2023-07-29T00:39:40.579 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.579 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:40.580 DEBUG:boto:Port: 8001 2023-07-29T00:39:40.580 DEBUG:boto:Params: {} 2023-07-29T00:39:40.580 DEBUG:boto:Token: None 2023-07-29T00:39:40.581 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/obj93?versionId=null 2023-07-29T00:39:40.581 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5ARLLR6M4YWOJrUSkD6psTIl0jQ= 2023-07-29T00:39:40.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5ARLLR6M4YWOJrUSkD6psTIl0jQ=', 'Content-Length': '0'} 2023-07-29T00:39:40.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f85a14e5e2e2af3-0064c45fcc-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.584 DEBUG:boto:path=/hiiqos-59/obj93 2023-07-29T00:39:40.584 DEBUG:boto:auth_path=/hiiqos-59/obj93 2023-07-29T00:39:40.585 DEBUG:boto:path=/hiiqos-59/obj93?versionId=null 2023-07-29T00:39:40.585 DEBUG:boto:auth_path=/hiiqos-59/obj93?versionId=null 2023-07-29T00:39:40.585 DEBUG:boto:Method: HEAD 2023-07-29T00:39:40.586 DEBUG:boto:Path: /hiiqos-59/obj93?versionId=null 2023-07-29T00:39:40.586 DEBUG:boto:Data: 2023-07-29T00:39:40.586 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.587 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:40.587 DEBUG:boto:Port: 8000 2023-07-29T00:39:40.587 DEBUG:boto:Params: {} 2023-07-29T00:39:40.588 DEBUG:boto:Token: None 2023-07-29T00:39:40.588 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/obj93?versionId=null 2023-07-29T00:39:40.589 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5ARLLR6M4YWOJrUSkD6psTIl0jQ= 2023-07-29T00:39:40.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5ARLLR6M4YWOJrUSkD6psTIl0jQ=', 'Content-Length': '0'} 2023-07-29T00:39:40.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000744cbccce2cd39cb-0064c45fcc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-07-29T00:39:40.592 DEBUG:boto:path=/hiiqos-59/obj93 2023-07-29T00:39:40.592 DEBUG:boto:auth_path=/hiiqos-59/obj93 2023-07-29T00:39:40.593 DEBUG:boto:path=/hiiqos-59/obj93?versionId=null 2023-07-29T00:39:40.593 DEBUG:boto:auth_path=/hiiqos-59/obj93?versionId=null 2023-07-29T00:39:40.593 DEBUG:boto:Method: GET 2023-07-29T00:39:40.594 DEBUG:boto:Path: /hiiqos-59/obj93?versionId=null 2023-07-29T00:39:40.594 DEBUG:boto:Data: 2023-07-29T00:39:40.595 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.595 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:40.595 DEBUG:boto:Port: 8001 2023-07-29T00:39:40.596 DEBUG:boto:Params: {} 2023-07-29T00:39:40.596 DEBUG:boto:Token: None 2023-07-29T00:39:40.596 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/obj93?versionId=null 2023-07-29T00:39:40.597 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:z3SBE3roErJ76HeLc9zBuqmq1cY= 2023-07-29T00:39:40.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:z3SBE3roErJ76HeLc9zBuqmq1cY=', 'Content-Length': '0'} 2023-07-29T00:39:40.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4cca8e67d998cc5-0064c45fcc-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.600 DEBUG:boto:path=/hiiqos-59/obj93 2023-07-29T00:39:40.600 DEBUG:boto:auth_path=/hiiqos-59/obj93 2023-07-29T00:39:40.601 DEBUG:boto:path=/hiiqos-59/obj93?versionId=null 2023-07-29T00:39:40.601 DEBUG:boto:auth_path=/hiiqos-59/obj93?versionId=null 2023-07-29T00:39:40.601 DEBUG:boto:Method: GET 2023-07-29T00:39:40.602 DEBUG:boto:Path: /hiiqos-59/obj93?versionId=null 2023-07-29T00:39:40.602 DEBUG:boto:Data: 2023-07-29T00:39:40.602 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.603 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:40.603 DEBUG:boto:Port: 8000 2023-07-29T00:39:40.603 DEBUG:boto:Params: {} 2023-07-29T00:39:40.604 DEBUG:boto:Token: None 2023-07-29T00:39:40.604 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/obj93?versionId=null 2023-07-29T00:39:40.605 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:z3SBE3roErJ76HeLc9zBuqmq1cY= 2023-07-29T00:39:40.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:z3SBE3roErJ76HeLc9zBuqmq1cY=', 'Content-Length': '0'} 2023-07-29T00:39:40.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000841ed3e5f75a458c-0064c45fcc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.607 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:40.608 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:40.608 DEBUG:boto:path=/hiiqos-59/?prefix=obj93 2023-07-29T00:39:40.609 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj93 2023-07-29T00:39:40.609 DEBUG:boto:Method: GET 2023-07-29T00:39:40.609 DEBUG:boto:Path: /hiiqos-59/?prefix=obj93 2023-07-29T00:39:40.610 DEBUG:boto:Data: 2023-07-29T00:39:40.610 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.610 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:40.611 DEBUG:boto:Port: 8001 2023-07-29T00:39:40.611 DEBUG:boto:Params: {} 2023-07-29T00:39:40.612 DEBUG:boto:Token: None 2023-07-29T00:39:40.612 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/ 2023-07-29T00:39:40.612 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:giRBAYclmTvBX66mnmk5f6TQ5ow= 2023-07-29T00:39:40.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:giRBAYclmTvBX66mnmk5f6TQ5ow=', 'Content-Length': '0'} 2023-07-29T00:39:40.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f96862eb82ceec89-0064c45fcc-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.616 DEBUG:boto:b'hiiqos-59obj931000falseobj932023-07-29T00:35:31.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:40.616 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:40.617 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:40.617 DEBUG:boto:path=/hiiqos-59/?prefix=obj93 2023-07-29T00:39:40.618 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj93 2023-07-29T00:39:40.618 DEBUG:boto:Method: GET 2023-07-29T00:39:40.618 DEBUG:boto:Path: /hiiqos-59/?prefix=obj93 2023-07-29T00:39:40.619 DEBUG:boto:Data: 2023-07-29T00:39:40.619 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.620 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:40.620 DEBUG:boto:Port: 8000 2023-07-29T00:39:40.620 DEBUG:boto:Params: {} 2023-07-29T00:39:40.621 DEBUG:boto:Token: None 2023-07-29T00:39:40.621 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/ 2023-07-29T00:39:40.621 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:giRBAYclmTvBX66mnmk5f6TQ5ow= 2023-07-29T00:39:40.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:giRBAYclmTvBX66mnmk5f6TQ5ow=', 'Content-Length': '0'} 2023-07-29T00:39:40.628 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b63efb00e8ab69e-0064c45fcc-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.629 DEBUG:boto:b'hiiqos-59obj931000falseobj932023-07-29T00:35:31.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:40.630 DEBUG:boto:path=/hiiqos-59/obj93 2023-07-29T00:39:40.631 DEBUG:boto:auth_path=/hiiqos-59/obj93 2023-07-29T00:39:40.632 DEBUG:boto:Method: HEAD 2023-07-29T00:39:40.633 DEBUG:boto:Path: /hiiqos-59/obj93 2023-07-29T00:39:40.634 DEBUG:boto:Data: 2023-07-29T00:39:40.634 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.635 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:40.635 DEBUG:boto:Port: 8001 2023-07-29T00:39:40.635 DEBUG:boto:Params: {} 2023-07-29T00:39:40.636 DEBUG:boto:Token: None 2023-07-29T00:39:40.636 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/obj93 2023-07-29T00:39:40.637 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vRfFTp707C27LotKYXVqHQ3GkJo= 2023-07-29T00:39:40.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vRfFTp707C27LotKYXVqHQ3GkJo=', 'Content-Length': '0'} 2023-07-29T00:39:40.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048bed7f9afb06bd5-0064c45fcc-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.640 DEBUG:boto:path=/hiiqos-59/obj93 2023-07-29T00:39:40.641 DEBUG:boto:auth_path=/hiiqos-59/obj93 2023-07-29T00:39:40.642 DEBUG:boto:Method: HEAD 2023-07-29T00:39:40.643 DEBUG:boto:Path: /hiiqos-59/obj93 2023-07-29T00:39:40.644 DEBUG:boto:Data: 2023-07-29T00:39:40.644 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.645 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:40.645 DEBUG:boto:Port: 8000 2023-07-29T00:39:40.646 DEBUG:boto:Params: {} 2023-07-29T00:39:40.646 DEBUG:boto:Token: None 2023-07-29T00:39:40.647 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/obj93 2023-07-29T00:39:40.647 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vRfFTp707C27LotKYXVqHQ3GkJo= 2023-07-29T00:39:40.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vRfFTp707C27LotKYXVqHQ3GkJo=', 'Content-Length': '0'} 2023-07-29T00:39:40.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000719ade6b96970a58-0064c45fcc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-07-29T00:39:40.650 DEBUG:boto:path=/hiiqos-59/obj93 2023-07-29T00:39:40.651 DEBUG:boto:auth_path=/hiiqos-59/obj93 2023-07-29T00:39:40.651 DEBUG:boto:Method: GET 2023-07-29T00:39:40.652 DEBUG:boto:Path: /hiiqos-59/obj93 2023-07-29T00:39:40.652 DEBUG:boto:Data: 2023-07-29T00:39:40.652 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.653 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:40.653 DEBUG:boto:Port: 8001 2023-07-29T00:39:40.653 DEBUG:boto:Params: {} 2023-07-29T00:39:40.654 DEBUG:boto:Token: None 2023-07-29T00:39:40.654 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/obj93 2023-07-29T00:39:40.655 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:01r0Qv5Ku+xck1hFo22p+bzrUlg= 2023-07-29T00:39:40.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:01r0Qv5Ku+xck1hFo22p+bzrUlg=', 'Content-Length': '0'} 2023-07-29T00:39:40.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b01803a1b0cf72ff-0064c45fcc-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.658 DEBUG:boto:path=/hiiqos-59/obj93 2023-07-29T00:39:40.658 DEBUG:boto:auth_path=/hiiqos-59/obj93 2023-07-29T00:39:40.659 DEBUG:boto:Method: GET 2023-07-29T00:39:40.660 DEBUG:boto:Path: /hiiqos-59/obj93 2023-07-29T00:39:40.661 DEBUG:boto:Data: 2023-07-29T00:39:40.662 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.662 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:40.662 DEBUG:boto:Port: 8000 2023-07-29T00:39:40.663 DEBUG:boto:Params: {} 2023-07-29T00:39:40.663 DEBUG:boto:Token: None 2023-07-29T00:39:40.664 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/obj93 2023-07-29T00:39:40.664 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:01r0Qv5Ku+xck1hFo22p+bzrUlg= 2023-07-29T00:39:40.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:01r0Qv5Ku+xck1hFo22p+bzrUlg=', 'Content-Length': '0'} 2023-07-29T00:39:40.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000431cf2dc36a61307-0064c45fcc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.669 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:40.669 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:40.670 DEBUG:boto:path=/hiiqos-59/?prefix=obj93 2023-07-29T00:39:40.670 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj93 2023-07-29T00:39:40.671 DEBUG:boto:Method: GET 2023-07-29T00:39:40.671 DEBUG:boto:Path: /hiiqos-59/?prefix=obj93 2023-07-29T00:39:40.671 DEBUG:boto:Data: 2023-07-29T00:39:40.672 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.672 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:40.673 DEBUG:boto:Port: 8001 2023-07-29T00:39:40.673 DEBUG:boto:Params: {} 2023-07-29T00:39:40.673 DEBUG:boto:Token: None 2023-07-29T00:39:40.674 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/ 2023-07-29T00:39:40.674 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:giRBAYclmTvBX66mnmk5f6TQ5ow= 2023-07-29T00:39:40.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:giRBAYclmTvBX66mnmk5f6TQ5ow=', 'Content-Length': '0'} 2023-07-29T00:39:40.679 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000488dbecf819ea77f-0064c45fcc-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.720 DEBUG:boto:b'hiiqos-59obj931000falseobj932023-07-29T00:35:31.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:40.721 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:40.722 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:40.722 DEBUG:boto:path=/hiiqos-59/?prefix=obj93 2023-07-29T00:39:40.722 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj93 2023-07-29T00:39:40.723 DEBUG:boto:Method: GET 2023-07-29T00:39:40.723 DEBUG:boto:Path: /hiiqos-59/?prefix=obj93 2023-07-29T00:39:40.724 DEBUG:boto:Data: 2023-07-29T00:39:40.724 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.725 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:40.725 DEBUG:boto:Port: 8000 2023-07-29T00:39:40.725 DEBUG:boto:Params: {} 2023-07-29T00:39:40.726 DEBUG:boto:Token: None 2023-07-29T00:39:40.726 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/ 2023-07-29T00:39:40.727 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:giRBAYclmTvBX66mnmk5f6TQ5ow= 2023-07-29T00:39:40.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:giRBAYclmTvBX66mnmk5f6TQ5ow=', 'Content-Length': '0'} 2023-07-29T00:39:40.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4dbd81b6725a337-0064c45fcc-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.732 DEBUG:boto:b'hiiqos-59obj931000falseobj932023-07-29T00:35:31.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:40.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-07-29T00:39:40.734 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:39:40.734 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:39:40.734 DEBUG:boto:path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:39:40.735 DEBUG:boto:auth_path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:39:40.735 DEBUG:boto:Method: GET 2023-07-29T00:39:40.735 DEBUG:boto:Path: /hiiqos-59/obj95?versionId=null 2023-07-29T00:39:40.736 DEBUG:boto:Data: 2023-07-29T00:39:40.736 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.737 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:40.737 DEBUG:boto:Port: 8001 2023-07-29T00:39:40.737 DEBUG:boto:Params: {} 2023-07-29T00:39:40.738 DEBUG:boto:Token: None 2023-07-29T00:39:40.738 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/obj95?versionId=null 2023-07-29T00:39:40.739 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Lhbi0JSOr8/39UgFVlArkHzBO6M= 2023-07-29T00:39:40.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Lhbi0JSOr8/39UgFVlArkHzBO6M=', 'Content-Length': '0'} 2023-07-29T00:39:40.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fec2799f287927f-0064c45fcc-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.741 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:39:40.742 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:39:40.742 DEBUG:boto:path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:39:40.743 DEBUG:boto:auth_path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:39:40.743 DEBUG:boto:Method: GET 2023-07-29T00:39:40.743 DEBUG:boto:Path: /hiiqos-59/obj95?versionId=null 2023-07-29T00:39:40.744 DEBUG:boto:Data: 2023-07-29T00:39:40.744 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.745 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:40.745 DEBUG:boto:Port: 8000 2023-07-29T00:39:40.745 DEBUG:boto:Params: {} 2023-07-29T00:39:40.746 DEBUG:boto:Token: None 2023-07-29T00:39:40.746 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/obj95?versionId=null 2023-07-29T00:39:40.747 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Lhbi0JSOr8/39UgFVlArkHzBO6M= 2023-07-29T00:39:40.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Lhbi0JSOr8/39UgFVlArkHzBO6M=', 'Content-Length': '0'} 2023-07-29T00:39:40.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dfed607b3316dbc-0064c45fcc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.792 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:39:40.793 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:39:40.794 DEBUG:boto:path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:39:40.795 DEBUG:boto:auth_path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:39:40.795 DEBUG:boto:Method: HEAD 2023-07-29T00:39:40.796 DEBUG:boto:Path: /hiiqos-59/obj95?versionId=null 2023-07-29T00:39:40.796 DEBUG:boto:Data: 2023-07-29T00:39:40.796 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.797 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:40.797 DEBUG:boto:Port: 8001 2023-07-29T00:39:40.797 DEBUG:boto:Params: {} 2023-07-29T00:39:40.798 DEBUG:boto:Token: None 2023-07-29T00:39:40.798 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/obj95?versionId=null 2023-07-29T00:39:40.799 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Crlh/EBkO47CX3eHECVYC4RzK8w= 2023-07-29T00:39:40.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Crlh/EBkO47CX3eHECVYC4RzK8w=', 'Content-Length': '0'} 2023-07-29T00:39:40.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006eb93eb2cca3728f-0064c45fcc-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.802 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:39:40.802 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:39:40.803 DEBUG:boto:path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:39:40.803 DEBUG:boto:auth_path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:39:40.803 DEBUG:boto:Method: HEAD 2023-07-29T00:39:40.804 DEBUG:boto:Path: /hiiqos-59/obj95?versionId=null 2023-07-29T00:39:40.804 DEBUG:boto:Data: 2023-07-29T00:39:40.805 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.805 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:40.805 DEBUG:boto:Port: 8000 2023-07-29T00:39:40.806 DEBUG:boto:Params: {} 2023-07-29T00:39:40.806 DEBUG:boto:Token: None 2023-07-29T00:39:40.806 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/obj95?versionId=null 2023-07-29T00:39:40.807 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Crlh/EBkO47CX3eHECVYC4RzK8w= 2023-07-29T00:39:40.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Crlh/EBkO47CX3eHECVYC4RzK8w=', 'Content-Length': '0'} 2023-07-29T00:39:40.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e4d88e1cf7587a6-0064c45fcc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-07-29T00:39:40.810 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:39:40.810 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:39:40.811 DEBUG:boto:path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:39:40.811 DEBUG:boto:auth_path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:39:40.811 DEBUG:boto:Method: GET 2023-07-29T00:39:40.812 DEBUG:boto:Path: /hiiqos-59/obj95?versionId=null 2023-07-29T00:39:40.812 DEBUG:boto:Data: 2023-07-29T00:39:40.813 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.813 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:40.813 DEBUG:boto:Port: 8001 2023-07-29T00:39:40.814 DEBUG:boto:Params: {} 2023-07-29T00:39:40.814 DEBUG:boto:Token: None 2023-07-29T00:39:40.814 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/obj95?versionId=null 2023-07-29T00:39:40.815 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Lhbi0JSOr8/39UgFVlArkHzBO6M= 2023-07-29T00:39:40.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Lhbi0JSOr8/39UgFVlArkHzBO6M=', 'Content-Length': '0'} 2023-07-29T00:39:40.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051f4a61d61ba77b0-0064c45fcc-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.818 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:39:40.818 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:39:40.819 DEBUG:boto:path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:39:40.819 DEBUG:boto:auth_path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:39:40.819 DEBUG:boto:Method: GET 2023-07-29T00:39:40.820 DEBUG:boto:Path: /hiiqos-59/obj95?versionId=null 2023-07-29T00:39:40.820 DEBUG:boto:Data: 2023-07-29T00:39:40.821 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.821 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:40.821 DEBUG:boto:Port: 8000 2023-07-29T00:39:40.822 DEBUG:boto:Params: {} 2023-07-29T00:39:40.823 DEBUG:boto:Token: None 2023-07-29T00:39:40.824 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/obj95?versionId=null 2023-07-29T00:39:40.824 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Lhbi0JSOr8/39UgFVlArkHzBO6M= 2023-07-29T00:39:40.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Lhbi0JSOr8/39UgFVlArkHzBO6M=', 'Content-Length': '0'} 2023-07-29T00:39:40.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000502cd1321c026749-0064c45fcc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.829 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:40.830 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:40.830 DEBUG:boto:path=/hiiqos-59/?prefix=obj95 2023-07-29T00:39:40.831 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj95 2023-07-29T00:39:40.831 DEBUG:boto:Method: GET 2023-07-29T00:39:40.832 DEBUG:boto:Path: /hiiqos-59/?prefix=obj95 2023-07-29T00:39:40.833 DEBUG:boto:Data: 2023-07-29T00:39:40.833 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.834 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:40.835 DEBUG:boto:Port: 8001 2023-07-29T00:39:40.835 DEBUG:boto:Params: {} 2023-07-29T00:39:40.836 DEBUG:boto:Token: None 2023-07-29T00:39:40.837 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/ 2023-07-29T00:39:40.838 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:giRBAYclmTvBX66mnmk5f6TQ5ow= 2023-07-29T00:39:40.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:giRBAYclmTvBX66mnmk5f6TQ5ow=', 'Content-Length': '0'} 2023-07-29T00:39:40.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000143bf4008734b72-0064c45fcc-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.888 DEBUG:boto:b'hiiqos-59obj951000falseobj952023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:40.889 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:40.890 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:40.890 DEBUG:boto:path=/hiiqos-59/?prefix=obj95 2023-07-29T00:39:40.890 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj95 2023-07-29T00:39:40.891 DEBUG:boto:Method: GET 2023-07-29T00:39:40.891 DEBUG:boto:Path: /hiiqos-59/?prefix=obj95 2023-07-29T00:39:40.891 DEBUG:boto:Data: 2023-07-29T00:39:40.892 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.892 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:40.893 DEBUG:boto:Port: 8000 2023-07-29T00:39:40.893 DEBUG:boto:Params: {} 2023-07-29T00:39:40.893 DEBUG:boto:Token: None 2023-07-29T00:39:40.894 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/ 2023-07-29T00:39:40.894 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:giRBAYclmTvBX66mnmk5f6TQ5ow= 2023-07-29T00:39:40.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:giRBAYclmTvBX66mnmk5f6TQ5ow=', 'Content-Length': '0'} 2023-07-29T00:39:40.899 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067d51a561015519b-0064c45fcc-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.900 DEBUG:boto:b'hiiqos-59obj951000falseobj952023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:40.901 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:39:40.901 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:39:40.902 DEBUG:boto:Method: HEAD 2023-07-29T00:39:40.902 DEBUG:boto:Path: /hiiqos-59/obj95 2023-07-29T00:39:40.902 DEBUG:boto:Data: 2023-07-29T00:39:40.903 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.903 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:40.903 DEBUG:boto:Port: 8001 2023-07-29T00:39:40.904 DEBUG:boto:Params: {} 2023-07-29T00:39:40.904 DEBUG:boto:Token: None 2023-07-29T00:39:40.904 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/obj95 2023-07-29T00:39:40.905 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0Fna1oraEnm1Y+wNatVUSNWR9tQ= 2023-07-29T00:39:40.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0Fna1oraEnm1Y+wNatVUSNWR9tQ=', 'Content-Length': '0'} 2023-07-29T00:39:40.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005877c164da95f789-0064c45fcc-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.908 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:39:40.909 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:39:40.910 DEBUG:boto:Method: HEAD 2023-07-29T00:39:40.910 DEBUG:boto:Path: /hiiqos-59/obj95 2023-07-29T00:39:40.911 DEBUG:boto:Data: 2023-07-29T00:39:40.912 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.912 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:40.913 DEBUG:boto:Port: 8000 2023-07-29T00:39:40.913 DEBUG:boto:Params: {} 2023-07-29T00:39:40.914 DEBUG:boto:Token: None 2023-07-29T00:39:40.915 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/obj95 2023-07-29T00:39:40.916 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0Fna1oraEnm1Y+wNatVUSNWR9tQ= 2023-07-29T00:39:40.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0Fna1oraEnm1Y+wNatVUSNWR9tQ=', 'Content-Length': '0'} 2023-07-29T00:39:40.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087dfc439481a3a63-0064c45fcc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-07-29T00:39:40.920 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:39:40.921 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:39:40.922 DEBUG:boto:Method: GET 2023-07-29T00:39:40.923 DEBUG:boto:Path: /hiiqos-59/obj95 2023-07-29T00:39:40.923 DEBUG:boto:Data: 2023-07-29T00:39:40.924 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.925 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:40.925 DEBUG:boto:Port: 8001 2023-07-29T00:39:40.926 DEBUG:boto:Params: {} 2023-07-29T00:39:40.927 DEBUG:boto:Token: None 2023-07-29T00:39:40.927 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/obj95 2023-07-29T00:39:40.928 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AXOOVuozBJTz/rXuXanens7MGgE= 2023-07-29T00:39:40.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AXOOVuozBJTz/rXuXanens7MGgE=', 'Content-Length': '0'} 2023-07-29T00:39:40.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dc57d06aa121fce-0064c45fcc-1c08-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.933 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:39:40.933 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:39:40.934 DEBUG:boto:Method: GET 2023-07-29T00:39:40.934 DEBUG:boto:Path: /hiiqos-59/obj95 2023-07-29T00:39:40.935 DEBUG:boto:Data: 2023-07-29T00:39:40.935 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.935 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:40.936 DEBUG:boto:Port: 8000 2023-07-29T00:39:40.936 DEBUG:boto:Params: {} 2023-07-29T00:39:40.937 DEBUG:boto:Token: None 2023-07-29T00:39:40.938 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/obj95 2023-07-29T00:39:40.938 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AXOOVuozBJTz/rXuXanens7MGgE= 2023-07-29T00:39:40.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AXOOVuozBJTz/rXuXanens7MGgE=', 'Content-Length': '0'} 2023-07-29T00:39:40.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000683a0340d070a54b-0064c45fcc-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.984 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:40.985 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:40.986 DEBUG:boto:path=/hiiqos-59/?prefix=obj95 2023-07-29T00:39:40.987 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj95 2023-07-29T00:39:40.987 DEBUG:boto:Method: GET 2023-07-29T00:39:40.988 DEBUG:boto:Path: /hiiqos-59/?prefix=obj95 2023-07-29T00:39:40.988 DEBUG:boto:Data: 2023-07-29T00:39:40.988 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.989 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:40.989 DEBUG:boto:Port: 8001 2023-07-29T00:39:40.989 DEBUG:boto:Params: {} 2023-07-29T00:39:40.990 DEBUG:boto:Token: None 2023-07-29T00:39:40.990 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/ 2023-07-29T00:39:40.991 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:giRBAYclmTvBX66mnmk5f6TQ5ow= 2023-07-29T00:39:40.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:giRBAYclmTvBX66mnmk5f6TQ5ow=', 'Content-Length': '0'} 2023-07-29T00:39:40.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093ec9af855b4589b-0064c45fcc-1c08-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:40.994 DEBUG:boto:b'hiiqos-59obj951000falseobj952023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:40.995 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:40.995 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:40.996 DEBUG:boto:path=/hiiqos-59/?prefix=obj95 2023-07-29T00:39:40.996 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj95 2023-07-29T00:39:40.996 DEBUG:boto:Method: GET 2023-07-29T00:39:40.997 DEBUG:boto:Path: /hiiqos-59/?prefix=obj95 2023-07-29T00:39:40.997 DEBUG:boto:Data: 2023-07-29T00:39:40.997 DEBUG:boto:Headers: {} 2023-07-29T00:39:40.998 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:39:40.998 DEBUG:boto:Port: 8000 2023-07-29T00:39:40.999 DEBUG:boto:Params: {} 2023-07-29T00:39:40.999 DEBUG:boto:Token: None 2023-07-29T00:39:40.999 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:39:40 GMT /hiiqos-59/ 2023-07-29T00:39:41.000 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:giRBAYclmTvBX66mnmk5f6TQ5ow= 2023-07-29T00:39:41.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:giRBAYclmTvBX66mnmk5f6TQ5ow=', 'Content-Length': '0'} 2023-07-29T00:39:41.005 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002443d0842f4e7aa6-0064c45fcc-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:39:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:39:41.005 DEBUG:boto:b'hiiqos-59obj951000falseobj952023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:39:41.006 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-59 zones={a2, a1} 2023-07-29T00:39:41.007 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-07-29T00:39:41.007 INFO:teuthology.orchestra.run:waiting for 300 2023-07-29T00:39:41.008 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:39:40.943+0000 7f1631871640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 128854) UID: 0 2023-07-29T00:39:41.008 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:39:40.943+0000 7f1637833880 -1 shutting down 2023-07-29T00:39:45.917 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-07-29T00:39:47.106 INFO:tasks.rgw.c1.client.1:Stopped 2023-07-29T00:39:47.107 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:39:47.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:39:47.108 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:39:47.165 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:47.165 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:47.199 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T00:39:47.199 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:47.200 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj11", 2023-07-29T00:39:47.200 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:47.200 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:47.200 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:47.200 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 3 2023-07-29T00:39:47.200 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.201 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:47.201 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:47.201 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:47.201 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:47.201 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:39:47.201 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:26:54.480753Z", 2023-07-29T00:39:47.201 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:47.202 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:47.202 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:47.202 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:47.202 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:47.202 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:47.202 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:47.203 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:47.203 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.203 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_uAgHKnwIu8R6wz4ZLs5Z2eqzuULKiKx", 2023-07-29T00:39:47.203 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:47.203 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:47.203 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:47.203 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.204 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:47.204 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj111", 2023-07-29T00:39:47.204 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:47.204 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:47.204 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:47.204 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 2 2023-07-29T00:39:47.205 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.205 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:47.205 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:47.205 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:47.205 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:47.205 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:39:47.205 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:37:53.213506Z", 2023-07-29T00:39:47.206 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:47.206 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:47.206 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:47.206 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:47.206 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:47.206 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:47.206 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:47.207 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:47.207 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.207 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_C9elAuHcZ5FdfJDeh1xnL_Q5OVBLVEN", 2023-07-29T00:39:47.207 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:47.207 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:47.207 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:47.207 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.208 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:47.208 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj115", 2023-07-29T00:39:47.208 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:47.208 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:47.208 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:47.208 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 2 2023-07-29T00:39:47.209 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.209 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:47.209 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:47.209 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:47.209 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:47.209 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:39:47.209 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:33:06.941054Z", 2023-07-29T00:39:47.210 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:47.210 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:47.210 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:47.210 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:47.210 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:47.210 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:47.210 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:47.211 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:47.211 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.211 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_m2foGXfpz6EGyyx6gGK1eACauQGSAXX", 2023-07-29T00:39:47.211 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:47.211 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:47.211 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:47.211 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.212 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:47.212 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj116", 2023-07-29T00:39:47.212 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:47.212 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:47.212 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:47.212 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 4 2023-07-29T00:39:47.213 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.213 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:47.213 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:47.213 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:47.213 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:47.213 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 6, 2023-07-29T00:39:47.213 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:35:01.404138Z", 2023-07-29T00:39:47.214 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:47.214 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:47.214 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:47.214 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:47.214 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:47.214 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:47.214 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:47.215 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:47.215 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.215 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_D1bDDt30NUSH-RsqKOlpUwdBi9AmyOA", 2023-07-29T00:39:47.215 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:47.215 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:47.215 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:47.215 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.216 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:47.216 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj124", 2023-07-29T00:39:47.216 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:47.216 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:47.216 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:47.216 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 2 2023-07-29T00:39:47.217 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.217 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:47.217 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:47.217 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:47.217 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:47.217 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 6, 2023-07-29T00:39:47.217 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:35:31.901495Z", 2023-07-29T00:39:47.218 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:47.218 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:47.218 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:47.218 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:47.218 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:47.218 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:47.218 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:47.219 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:47.219 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.219 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_h59R8T0O6EYl21-iGa9jA-4HbXH28Rd", 2023-07-29T00:39:47.219 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:47.219 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:47.219 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:47.220 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.220 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:47.220 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj145", 2023-07-29T00:39:47.220 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:47.220 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:47.220 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:47.220 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 6 2023-07-29T00:39:47.221 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.221 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:47.221 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:47.221 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:47.221 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:47.221 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:47.222 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:35:01.459486Z", 2023-07-29T00:39:47.222 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:47.222 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:47.222 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:47.222 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:47.222 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:47.222 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:47.223 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:47.223 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:47.223 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.223 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_mogC4qZBAqABvnqWiEa6ZIt9-HIafYi", 2023-07-29T00:39:47.223 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:47.223 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:47.223 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:47.224 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.224 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:47.224 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj148", 2023-07-29T00:39:47.224 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:47.224 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:47.224 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:47.224 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 1 2023-07-29T00:39:47.225 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.225 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:47.225 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:47.225 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:47.225 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:47.225 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:47.225 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:37:53.270196Z", 2023-07-29T00:39:47.226 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:47.226 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:47.226 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:47.226 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:47.226 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:47.226 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:47.227 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:47.227 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:47.227 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.227 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_ZksU_dOTwuXzZ0npWlkfTApc1Iiskak", 2023-07-29T00:39:47.227 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:47.227 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:47.227 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:47.228 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.228 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:47.228 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj155", 2023-07-29T00:39:47.228 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:47.228 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:47.228 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:47.228 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 5 2023-07-29T00:39:47.229 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.229 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:47.229 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:47.229 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:47.229 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:47.229 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:39:47.230 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:35:31.919584Z", 2023-07-29T00:39:47.230 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:47.230 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:47.230 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:47.230 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:47.230 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:47.230 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:47.231 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:47.231 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:47.231 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.231 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_fN88wsuQR_i8jDARZCbm0yNnedij565", 2023-07-29T00:39:47.231 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:47.231 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:47.231 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:47.232 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.232 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:47.232 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj17", 2023-07-29T00:39:47.232 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:47.232 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:47.232 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:47.232 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 8 2023-07-29T00:39:47.233 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.233 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:47.233 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:47.233 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:47.233 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:47.233 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:47.233 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:27:51.133766Z", 2023-07-29T00:39:47.234 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:47.234 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:47.234 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:47.234 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:47.234 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:47.234 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:47.234 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:47.235 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:47.235 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.235 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_J6Lhzp0LPzCrP_HGgvvYXFQah44IC4x", 2023-07-29T00:39:47.235 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:47.235 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:47.235 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:47.236 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.236 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:47.236 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj185", 2023-07-29T00:39:47.236 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:47.236 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:47.236 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:47.236 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 11 2023-07-29T00:39:47.237 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.237 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:47.237 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:47.237 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:47.237 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:47.237 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:47.237 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:37:53.291240Z", 2023-07-29T00:39:47.238 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:47.238 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:47.238 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:47.238 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:47.238 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:47.238 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:47.239 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:47.239 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:47.239 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.239 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_Y-MWJTmwvXjnGmb5i9oO8mV-8cE8Mml", 2023-07-29T00:39:47.239 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:47.239 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:47.239 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:47.240 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.240 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:47.240 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj19", 2023-07-29T00:39:47.240 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:47.240 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:47.240 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:47.240 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 5 2023-07-29T00:39:47.241 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.241 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:47.241 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:47.241 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:47.241 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:47.241 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:39:47.241 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:29:58.641540Z", 2023-07-29T00:39:47.242 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:47.242 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:47.242 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:47.242 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:47.242 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:47.242 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:47.242 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:47.243 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:47.243 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.243 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_gma6_GYeZ3zbTc8vJemo5Gn8SoxK2Hw", 2023-07-29T00:39:47.243 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:47.243 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:47.243 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:47.244 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.244 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:47.244 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj22", 2023-07-29T00:39:47.244 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:47.244 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:47.244 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:47.244 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 10 2023-07-29T00:39:47.245 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.245 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:47.245 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:47.245 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:47.245 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:47.245 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 6, 2023-07-29T00:39:47.245 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:26:54.504376Z", 2023-07-29T00:39:47.246 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:47.246 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:47.246 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:47.246 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:47.246 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:47.247 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:47.247 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:47.247 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:47.247 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.247 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_QKogUExsa8Dg3Qymk-sozOdIN8tT7KA", 2023-07-29T00:39:47.247 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:47.247 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:47.248 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:47.248 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.248 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:47.248 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj23", 2023-07-29T00:39:47.248 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:47.248 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:47.249 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:47.249 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 2 2023-07-29T00:39:47.249 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.249 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:47.249 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:47.249 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:47.249 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:47.250 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:47.250 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:33:06.784058Z", 2023-07-29T00:39:47.250 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:47.250 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:47.250 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:47.250 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:47.250 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:47.251 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:47.251 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:47.251 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:47.251 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.251 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_6sm5UEGKRNUcH_h2Xg9dSvg2Zbc1HFl", 2023-07-29T00:39:47.251 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:47.251 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:47.252 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:47.252 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.252 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:47.252 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj29", 2023-07-29T00:39:47.252 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:47.252 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:47.252 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:47.253 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 6 2023-07-29T00:39:47.253 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.253 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:47.253 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:47.253 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:47.253 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:47.254 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:47.254 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:35:01.307381Z", 2023-07-29T00:39:47.254 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:47.254 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:47.254 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:47.254 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:47.254 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:47.255 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:47.255 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:47.255 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:47.255 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.255 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_lHhAe9mN54BlKr1g5020HxmP08-iIRg", 2023-07-29T00:39:47.255 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:47.255 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:47.256 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:47.256 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.256 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:47.256 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj31", 2023-07-29T00:39:47.256 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:47.256 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:47.256 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:47.257 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 4 2023-07-29T00:39:47.257 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.257 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:47.257 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:47.257 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:47.257 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:47.257 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 6, 2023-07-29T00:39:47.258 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:35:31.771442Z", 2023-07-29T00:39:47.258 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:47.258 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:47.258 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:47.258 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:47.258 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:47.258 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:47.259 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:47.259 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:47.259 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.259 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_xnwnXpwO-iAxMJz73-1XKzBELzztFiH", 2023-07-29T00:39:47.259 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:47.259 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:47.259 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:47.260 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.260 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:47.260 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj33", 2023-07-29T00:39:47.260 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:47.260 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:47.260 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:47.261 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 7 2023-07-29T00:39:47.261 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.261 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:47.261 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:47.261 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:47.261 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:47.261 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 6, 2023-07-29T00:39:47.262 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:26:54.567027Z", 2023-07-29T00:39:47.262 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:47.262 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:47.262 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:47.262 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:47.262 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:47.262 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:47.263 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:47.263 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:47.263 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.263 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_xieuBcqJdaHfWKvaAXDBBZV1JHqQLcB", 2023-07-29T00:39:47.263 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:47.263 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:47.263 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:47.264 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.264 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:47.264 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj34", 2023-07-29T00:39:47.264 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:47.264 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:47.264 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:47.265 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 3 2023-07-29T00:39:47.265 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.265 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:47.265 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:47.265 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:47.265 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:47.265 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 6, 2023-07-29T00:39:47.266 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:27:51.159033Z", 2023-07-29T00:39:47.266 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:47.266 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:47.266 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:47.266 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:47.266 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:47.266 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:47.267 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:47.267 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:47.267 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.267 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_muqQ1zSDNOgWh1j6Rofo0M9XCnYGVVk", 2023-07-29T00:39:47.267 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:47.267 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:47.267 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:47.268 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.268 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:47.268 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj37", 2023-07-29T00:39:47.268 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:47.268 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:47.268 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:47.268 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 3 2023-07-29T00:39:47.269 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.269 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:47.269 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:47.269 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:47.269 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:47.269 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:47.270 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:37:53.137274Z", 2023-07-29T00:39:47.270 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:47.270 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:47.270 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:47.270 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:47.270 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:47.270 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:47.271 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:47.271 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:47.271 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.271 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_T9EvCD3QahiyO4KZJFMoXwwUn05FL3M", 2023-07-29T00:39:47.271 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:47.271 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:47.271 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:47.272 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.272 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:47.272 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj38", 2023-07-29T00:39:47.272 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:47.272 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:47.272 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:47.272 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 5 2023-07-29T00:39:47.273 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.273 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:47.273 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:47.273 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:47.273 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:47.273 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:47.273 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:29:58.663185Z", 2023-07-29T00:39:47.274 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:47.274 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:47.274 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:47.274 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:47.274 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:47.274 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:47.274 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:47.275 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:47.275 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.275 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_xqaSElj9UcYCnLwwiLDxo0GV75qY3_O", 2023-07-29T00:39:47.275 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:47.275 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:47.275 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:47.275 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.276 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:47.276 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj44", 2023-07-29T00:39:47.276 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:47.276 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:47.276 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:47.276 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 5 2023-07-29T00:39:47.276 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.277 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:47.277 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:47.277 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:47.277 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:47.277 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:47.277 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:26:54.593602Z", 2023-07-29T00:39:47.278 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:47.278 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:47.278 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:47.278 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:47.278 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:47.278 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:47.279 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:47.279 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:47.279 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.279 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_WstBMRtLfBEKxPOuuxMR4QB1EXaM-ps", 2023-07-29T00:39:47.279 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:47.279 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:47.279 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:47.280 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.280 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:47.280 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj46", 2023-07-29T00:39:47.280 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:47.280 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:47.280 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:47.280 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 2 2023-07-29T00:39:47.281 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.281 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:47.281 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:47.281 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:47.281 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:47.281 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:47.281 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:33:06.801664Z", 2023-07-29T00:39:47.282 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:47.282 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:47.282 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:47.282 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:47.282 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:47.282 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:47.282 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:47.283 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:47.283 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.283 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_BgKmUOg0Gk_tGKSFryCQNKb9c2NnDYG", 2023-07-29T00:39:47.283 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:47.283 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:47.283 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:47.284 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.284 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:47.284 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj51", 2023-07-29T00:39:47.284 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:47.284 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:47.284 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:47.284 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 8 2023-07-29T00:39:47.285 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.285 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:47.285 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:47.285 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:47.285 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:47.285 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:47.285 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:27:51.180793Z", 2023-07-29T00:39:47.286 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:47.286 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:47.286 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:47.286 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:47.286 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:47.286 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:47.286 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:47.287 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:47.287 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.287 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_YWSGBxq_3Ftrjibq7SF7UIv00_5f66g", 2023-07-29T00:39:47.287 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:47.287 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:47.287 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:47.288 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.288 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:47.288 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj55", 2023-07-29T00:39:47.288 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:47.288 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:47.288 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:47.288 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 4 2023-07-29T00:39:47.289 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.289 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:47.289 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:47.289 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:47.289 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:47.289 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:39:47.289 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:26:54.656875Z", 2023-07-29T00:39:47.290 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:47.290 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:47.290 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:47.290 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:47.290 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:47.290 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:47.290 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:47.291 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:47.291 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.291 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "__SFSMlO-CUc1BVT9J1TPWpWxZ1QugZW", 2023-07-29T00:39:47.291 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:47.291 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:47.291 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:47.291 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.292 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:47.292 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj57", 2023-07-29T00:39:47.292 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:47.292 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:47.292 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:47.292 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 2 2023-07-29T00:39:47.292 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.293 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:47.293 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:47.293 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:47.293 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:47.293 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:39:47.293 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:29:58.720061Z", 2023-07-29T00:39:47.293 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:47.294 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:47.294 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:47.294 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:47.294 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:47.294 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:47.294 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:47.294 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:47.295 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.295 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_owsFXaeohE3rkyaSI6TtXfMSKTiwrdN", 2023-07-29T00:39:47.295 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:47.295 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:47.295 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:47.295 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.296 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:47.296 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj58", 2023-07-29T00:39:47.296 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:47.296 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:47.296 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:47.296 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 4 2023-07-29T00:39:47.296 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.297 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:47.297 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:47.297 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:47.297 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:47.297 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:39:47.297 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:35:01.328082Z", 2023-07-29T00:39:47.297 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:47.298 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:47.298 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:47.298 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:47.298 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:47.298 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:47.298 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:47.298 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:47.299 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.299 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_bVDoKe54-1qLKJCtT3mlM9975bzMDfx", 2023-07-29T00:39:47.299 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:47.299 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:47.299 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:47.299 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.300 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:47.300 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj62", 2023-07-29T00:39:47.300 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:47.300 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:47.300 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:47.300 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 3 2023-07-29T00:39:47.300 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.301 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:47.301 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:47.301 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:47.301 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:47.301 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:47.301 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:35:31.826974Z", 2023-07-29T00:39:47.301 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:47.302 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:47.302 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:47.302 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:47.302 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:47.302 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:47.302 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:47.303 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:47.303 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.303 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_kFJ7aXZlj-fT1wM7_FmJrZ1unEMlif7", 2023-07-29T00:39:47.303 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:47.303 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:47.303 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:47.303 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.304 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:47.304 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj68", 2023-07-29T00:39:47.304 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:47.304 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:47.304 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:47.304 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 9 2023-07-29T00:39:47.304 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.305 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:47.305 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:47.305 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:47.305 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:47.305 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:47.305 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:27:51.249779Z", 2023-07-29T00:39:47.305 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:47.306 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:47.306 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:47.306 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:47.306 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:47.306 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:47.306 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:47.306 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:47.307 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.307 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_woORDM3QRu5EEFovrIcd9vXbuswCT25", 2023-07-29T00:39:47.307 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:47.307 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:47.307 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:47.307 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.307 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:47.308 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj69", 2023-07-29T00:39:47.308 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:47.308 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:47.308 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:47.308 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 13 2023-07-29T00:39:47.308 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.309 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:47.309 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:47.309 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:47.309 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:47.309 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:39:47.309 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:33:06.857762Z", 2023-07-29T00:39:47.310 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:47.310 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:47.310 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:47.310 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:47.310 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:47.311 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:47.311 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:47.311 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:47.311 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.311 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_vW_NApm_rEPcqONI0p4m073dINuNbQF", 2023-07-29T00:39:47.311 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:47.312 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:47.312 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:47.312 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.312 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:47.312 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj74", 2023-07-29T00:39:47.313 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:47.313 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:47.313 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:47.313 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 9 2023-07-29T00:39:47.313 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.313 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:47.314 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:47.314 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:47.314 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:47.314 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 6, 2023-07-29T00:39:47.314 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:37:53.195304Z", 2023-07-29T00:39:47.314 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:47.314 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:47.315 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:47.315 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:47.315 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:47.315 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:47.315 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:47.315 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:47.315 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.316 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_aZbNIDudbMV244XWRof2ujeiqMXFiWJ", 2023-07-29T00:39:47.316 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:47.316 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:47.316 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:47.316 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.316 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:47.316 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj76", 2023-07-29T00:39:47.317 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:47.317 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:47.317 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:47.317 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 22 2023-07-29T00:39:47.317 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.317 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:47.317 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:47.318 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:47.318 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:47.318 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:39:47.318 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:29:58.736037Z", 2023-07-29T00:39:47.318 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:47.318 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:47.318 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:47.319 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:47.319 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:47.319 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:47.319 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:47.319 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:47.319 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.320 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_TCcuByAat14mq3_qKWtfBXsb-Nv3lTE", 2023-07-29T00:39:47.320 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:47.320 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:47.320 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:47.320 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.320 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:47.320 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj85", 2023-07-29T00:39:47.321 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:47.321 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:47.321 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:47.321 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 39 2023-07-29T00:39:47.321 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.321 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:47.321 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:47.322 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:47.322 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:47.322 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:47.322 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:27:51.275401Z", 2023-07-29T00:39:47.322 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:47.322 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:47.322 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:47.323 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:47.323 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:47.323 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:47.323 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:47.323 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:47.323 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.323 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_ydzDP5r-YGZ0vK3t9XNfGcO3hpL2KG5", 2023-07-29T00:39:47.324 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:47.324 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:47.324 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:47.324 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.324 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:47.324 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj87", 2023-07-29T00:39:47.325 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:47.325 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:47.325 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:47.325 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 14 2023-07-29T00:39:47.325 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.325 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:47.325 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:47.326 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:47.327 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:47.327 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:47.327 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:35:01.384870Z", 2023-07-29T00:39:47.327 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:47.327 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:47.327 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:47.328 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:47.328 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:47.328 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:47.328 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:47.328 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:47.328 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.329 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_kvXy1F96qjFRwqyKwvRnEJn6Znvfgb8", 2023-07-29T00:39:47.329 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:47.329 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:47.329 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:47.329 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.330 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:47.330 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj92", 2023-07-29T00:39:47.330 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:47.330 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:47.330 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:47.330 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 3 2023-07-29T00:39:47.330 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.331 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:47.331 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:47.331 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:47.331 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:47.331 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:39:47.331 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:33:06.875188Z", 2023-07-29T00:39:47.332 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:47.332 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:47.332 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:47.332 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:47.332 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:47.332 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:47.332 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:47.333 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:47.333 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.333 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_J7M7Wz_JkkdZEQR73xq2lb71G_5oTbq", 2023-07-29T00:39:47.333 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:47.333 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:47.333 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:47.333 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.334 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:47.334 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj93", 2023-07-29T00:39:47.334 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:47.334 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:47.334 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:47.334 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 3 2023-07-29T00:39:47.334 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.335 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:47.335 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:47.335 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:47.335 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:47.335 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:47.335 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:35:31.844362Z", 2023-07-29T00:39:47.336 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:47.336 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:47.336 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:47.336 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:47.336 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:47.336 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:47.336 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:47.337 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:47.337 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.337 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_sp0yk65yvPv3jI-yOj86Fsu4s3mjiBD", 2023-07-29T00:39:47.337 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:47.337 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:47.337 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:47.337 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.338 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:47.338 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj95", 2023-07-29T00:39:47.338 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:47.338 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:47.338 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:47.338 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 2 2023-07-29T00:39:47.338 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.339 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:47.339 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:47.339 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:47.339 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:47.340 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 6, 2023-07-29T00:39:47.340 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:29:58.792344Z", 2023-07-29T00:39:47.340 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:47.340 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:47.340 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:47.341 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:47.341 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:47.341 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:47.341 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:47.341 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:47.341 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:47.341 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_7cTHpDrZQOirlimEVhy87mUNzY4YTJI", 2023-07-29T00:39:47.342 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:47.342 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:47.342 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:47.342 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:39:47.342 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T00:39:47.343 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:47.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:47.344 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --object obj11 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:39:47.396 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:47.397 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:47.481 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj111', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:47.482 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj111', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:47.482 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --object obj111 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:39:47.530 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:47.531 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:47.614 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj115', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:47.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj115', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:47.616 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --object obj115 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:39:47.664 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:47.664 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:47.748 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj116', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:47.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj116', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:47.749 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --object obj116 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:39:47.796 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:47.797 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:47.882 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj124', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:47.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj124', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:47.883 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --object obj124 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:39:47.932 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:47.932 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:48.017 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj145', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:48.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj145', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:48.018 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --object obj145 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:39:48.067 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:48.067 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:48.151 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj148', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:48.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj148', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:48.152 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --object obj148 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:39:48.200 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:48.200 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:48.281 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj155', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:48.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj155', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:48.282 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --object obj155 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:39:48.330 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:48.330 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:48.416 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:48.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:48.417 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --object obj17 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:39:48.465 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:48.465 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:48.548 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj185', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:48.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj185', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:48.550 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --object obj185 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:39:48.597 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:48.598 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:48.680 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:48.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:48.681 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --object obj19 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:39:48.729 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:48.729 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:48.814 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:48.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:48.815 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --object obj22 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:39:48.863 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:48.863 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:48.946 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:48.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:48.948 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --object obj23 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:39:48.996 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:48.997 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:49.082 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:49.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:49.083 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --object obj29 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:39:49.131 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:49.132 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:49.216 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:49.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:49.217 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --object obj31 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:39:49.264 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:49.265 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:49.352 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj33', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:49.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj33', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:49.353 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --object obj33 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:39:49.402 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:49.402 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:49.487 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj34', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:49.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj34', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:49.489 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --object obj34 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:39:49.536 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:49.536 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:49.619 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:49.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:49.620 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --object obj37 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:39:49.668 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:49.669 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:49.753 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj38', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:49.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj38', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:49.754 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --object obj38 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:39:49.802 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:49.802 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:49.884 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj44', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:49.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj44', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:49.886 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --object obj44 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:39:49.934 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:49.935 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:50.018 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj46', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:50.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj46', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:50.019 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --object obj46 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:39:50.066 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:50.066 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:50.149 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj51', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:50.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj51', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:50.150 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --object obj51 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:39:50.198 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:50.198 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:50.281 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:50.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:50.283 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --object obj55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:39:50.331 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:50.331 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:50.418 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:50.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:50.419 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --object obj57 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:39:50.467 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:50.468 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:50.553 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:50.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:50.555 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --object obj58 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:39:50.604 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:50.605 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:50.692 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:50.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:50.693 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --object obj62 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:39:50.740 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:50.741 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:50.824 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:50.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:50.825 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --object obj68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:39:50.872 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:50.873 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:50.955 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:50.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:50.957 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --object obj69 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:39:51.005 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:51.006 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:51.090 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:51.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:51.091 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --object obj74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:39:51.139 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:51.139 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:51.223 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:51.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:51.224 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --object obj76 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:39:51.272 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:51.272 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:51.360 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:51.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:51.361 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --object obj85 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:39:51.410 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:51.410 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:51.494 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj87', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:51.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj87', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:51.495 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --object obj87 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:39:51.521 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-07-29T00:39:51.543 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:51.543 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:51.626 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:51.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:51.627 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --object obj92 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:39:51.674 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:51.675 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:51.760 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:51.761 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:51.761 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --object obj93 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:39:51.809 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:51.810 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:51.893 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj95', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:51.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--object', 'obj95', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:51.894 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --object obj95 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:39:51.941 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:51.942 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:52.024 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'trim', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:52.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'trim', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:52.025 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog trim --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:39:52.070 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:52.071 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:52.119 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:52.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:52.120 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:39:52.168 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:52.169 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:52.212 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:52.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:39:52.213 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync run --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:39:52.261 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:52.262 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:52.328 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.263+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: sync status for source bucket: init. lease is: not taken. stop indications is: 0 2023-07-29T00:39:52.328 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.263+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: waiting on bucket lease 2023-07-29T00:39:52.332 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.267+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: status after acquiring the lock is: init 2023-07-29T00:39:52.344 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.279+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: listing bucket for full sync 2023-07-29T00:39:52.390 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj11 2023-07-29T00:39:52.390 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj11]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj11 op=0 op_state=1 2023-07-29T00:39:52.390 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj11]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj11[0] 2023-07-29T00:39:52.391 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj111 2023-07-29T00:39:52.391 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj111]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj111 op=0 op_state=1 2023-07-29T00:39:52.391 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj111]: : injecting data sync error on key=obj111 2023-07-29T00:39:52.391 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj111]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:39:52.391 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj111]: ERROR: failed to sync object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj111 2023-07-29T00:39:52.392 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj115 2023-07-29T00:39:52.392 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj115]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj115 op=0 op_state=1 2023-07-29T00:39:52.392 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj115]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj115[0] 2023-07-29T00:39:52.392 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj116 2023-07-29T00:39:52.392 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj116]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj116 op=0 op_state=1 2023-07-29T00:39:52.392 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj116]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj116[0] 2023-07-29T00:39:52.393 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj124 2023-07-29T00:39:52.393 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj124]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj124 op=0 op_state=1 2023-07-29T00:39:52.393 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj124]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj124[0] 2023-07-29T00:39:52.393 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj145 2023-07-29T00:39:52.393 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj145]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj145 op=0 op_state=1 2023-07-29T00:39:52.393 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj145]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj145[0] 2023-07-29T00:39:52.394 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj148 2023-07-29T00:39:52.394 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj148]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj148 op=0 op_state=1 2023-07-29T00:39:52.394 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj148]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj148[0] 2023-07-29T00:39:52.394 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj155 2023-07-29T00:39:52.394 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj155]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj155 op=0 op_state=1 2023-07-29T00:39:52.394 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj155]: : injecting data sync error on key=obj155 2023-07-29T00:39:52.395 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj155]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:39:52.395 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj155]: ERROR: failed to sync object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj155 2023-07-29T00:39:52.395 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj17 2023-07-29T00:39:52.395 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj17]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj17 op=0 op_state=1 2023-07-29T00:39:52.395 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj17]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj17[0] 2023-07-29T00:39:52.395 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj185 2023-07-29T00:39:52.396 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj185]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj185 op=0 op_state=1 2023-07-29T00:39:52.396 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj185]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj185[0] 2023-07-29T00:39:52.396 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj19 2023-07-29T00:39:52.396 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj19]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj19 op=0 op_state=1 2023-07-29T00:39:52.396 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj19]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj19[0] 2023-07-29T00:39:52.396 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj22 2023-07-29T00:39:52.397 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj22]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj22 op=0 op_state=1 2023-07-29T00:39:52.397 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj22]: : injecting data sync error on key=obj22 2023-07-29T00:39:52.397 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj22]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:39:52.397 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj22]: ERROR: failed to sync object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj22 2023-07-29T00:39:52.397 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj23 2023-07-29T00:39:52.397 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj23]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj23 op=0 op_state=1 2023-07-29T00:39:52.398 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj23]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj23[0] 2023-07-29T00:39:52.398 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj29 2023-07-29T00:39:52.398 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj29]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj29 op=0 op_state=1 2023-07-29T00:39:52.398 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj29]: : injecting data sync error on key=obj29 2023-07-29T00:39:52.398 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj29]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:39:52.398 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj29]: ERROR: failed to sync object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj29 2023-07-29T00:39:52.399 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj31 2023-07-29T00:39:52.399 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj31]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj31 op=0 op_state=1 2023-07-29T00:39:52.399 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.324+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj31]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj31[0] 2023-07-29T00:39:52.399 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.325+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj33 2023-07-29T00:39:52.400 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.325+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj33]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj33 op=0 op_state=1 2023-07-29T00:39:52.400 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.325+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj33]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj33[0] 2023-07-29T00:39:52.400 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.325+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj34 2023-07-29T00:39:52.400 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.325+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj34]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj34 op=0 op_state=1 2023-07-29T00:39:52.401 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.325+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj34]: : injecting data sync error on key=obj34 2023-07-29T00:39:52.401 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.325+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj34]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:39:52.401 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.325+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj34]: ERROR: failed to sync object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj34 2023-07-29T00:39:52.402 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.325+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj37 2023-07-29T00:39:52.402 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.325+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj37]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj37 op=0 op_state=1 2023-07-29T00:39:52.402 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.325+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj37]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj37[0] 2023-07-29T00:39:52.403 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.325+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj38 2023-07-29T00:39:52.403 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.325+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj38]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj38 op=0 op_state=1 2023-07-29T00:39:52.403 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.325+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj38]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj38[0] 2023-07-29T00:39:52.404 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.325+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj44 2023-07-29T00:39:52.404 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.325+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj44]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj44 op=0 op_state=1 2023-07-29T00:39:52.405 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.325+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj44]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj44[0] 2023-07-29T00:39:52.405 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.325+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj46 2023-07-29T00:39:52.405 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.325+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj46]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj46 op=0 op_state=1 2023-07-29T00:39:52.406 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.325+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj46]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj46[0] 2023-07-29T00:39:52.407 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.326+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: a sync operation returned error 2023-07-29T00:39:52.407 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.326+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj51 2023-07-29T00:39:52.408 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.326+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj51]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj51 op=0 op_state=1 2023-07-29T00:39:52.408 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.326+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj51]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj51[0] 2023-07-29T00:39:52.408 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.327+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: a sync operation returned error 2023-07-29T00:39:52.408 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.327+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj55 2023-07-29T00:39:52.409 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.327+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj55]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj55 op=0 op_state=1 2023-07-29T00:39:52.409 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.327+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj55]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj55[0] 2023-07-29T00:39:52.409 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.327+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: a sync operation returned error 2023-07-29T00:39:52.410 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.327+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj57 2023-07-29T00:39:52.410 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.327+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj57]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj57 op=0 op_state=1 2023-07-29T00:39:52.410 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.327+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj57]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj57[0] 2023-07-29T00:39:52.411 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.328+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: a sync operation returned error 2023-07-29T00:39:52.411 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.328+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj58 2023-07-29T00:39:52.411 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.328+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj58]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj58 op=0 op_state=1 2023-07-29T00:39:52.412 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.328+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj58]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj58[0] 2023-07-29T00:39:52.412 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.329+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: a sync operation returned error 2023-07-29T00:39:52.412 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.329+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj62 2023-07-29T00:39:52.413 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.329+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj62]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj62 op=0 op_state=1 2023-07-29T00:39:52.413 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.329+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj62]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj62[0] 2023-07-29T00:39:52.414 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.349+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj148]: success 2023-07-29T00:39:52.414 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.349+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj68 2023-07-29T00:39:52.414 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.349+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj68]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj68 op=0 op_state=1 2023-07-29T00:39:52.415 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.349+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj68]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj68[0] 2023-07-29T00:39:52.416 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.349+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj116]: success 2023-07-29T00:39:52.416 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.349+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj17]: success 2023-07-29T00:39:52.416 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.349+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj69 2023-07-29T00:39:52.416 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.349+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj69]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj69 op=0 op_state=1 2023-07-29T00:39:52.416 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.349+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj69]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj69[0] 2023-07-29T00:39:52.417 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.349+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj74 2023-07-29T00:39:52.417 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.349+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj74]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj74 op=0 op_state=1 2023-07-29T00:39:52.417 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.349+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj74]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj74[0] 2023-07-29T00:39:52.417 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.350+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj44]: success 2023-07-29T00:39:52.417 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.350+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj76 2023-07-29T00:39:52.417 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.350+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj76]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj76 op=0 op_state=1 2023-07-29T00:39:52.418 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.350+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj76]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj76[0] 2023-07-29T00:39:52.418 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.350+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj33]: success 2023-07-29T00:39:52.418 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.350+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj85 2023-07-29T00:39:52.418 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.350+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj85]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj85 op=0 op_state=1 2023-07-29T00:39:52.418 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.350+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj85]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj85[0] 2023-07-29T00:39:52.418 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.350+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj37]: success 2023-07-29T00:39:52.419 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.350+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj19]: success 2023-07-29T00:39:52.419 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.350+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj115]: success 2023-07-29T00:39:52.419 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.350+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj87 2023-07-29T00:39:52.419 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.350+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj87]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj87 op=0 op_state=1 2023-07-29T00:39:52.419 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.350+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj87]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj87[0] 2023-07-29T00:39:52.419 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.350+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj92 2023-07-29T00:39:52.420 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.350+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj92]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj92 op=0 op_state=1 2023-07-29T00:39:52.420 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.350+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj92]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj92[0] 2023-07-29T00:39:52.420 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.350+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj93 2023-07-29T00:39:52.420 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.350+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj93]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj93 op=0 op_state=1 2023-07-29T00:39:52.420 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.350+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj93]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj93[0] 2023-07-29T00:39:52.420 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.351+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj124]: success 2023-07-29T00:39:52.421 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.351+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj95 2023-07-29T00:39:52.421 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.351+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj95]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj95 op=0 op_state=1 2023-07-29T00:39:52.421 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.351+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj95]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj95[0] 2023-07-29T00:39:52.421 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.351+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj23]: success 2023-07-29T00:39:52.422 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.352+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj38]: success 2023-07-29T00:39:52.422 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.353+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj11]: success 2023-07-29T00:39:52.422 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.353+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: updating marker oid=bucket.full-sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24 marker=obj11 2023-07-29T00:39:52.422 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.353+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj46]: success 2023-07-29T00:39:52.422 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.353+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj31]: success 2023-07-29T00:39:52.423 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.353+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj145]: success 2023-07-29T00:39:52.423 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.353+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj185]: success 2023-07-29T00:39:52.446 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.381+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj62]: success 2023-07-29T00:39:52.447 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.381+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj57]: success 2023-07-29T00:39:52.448 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.382+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj58]: success 2023-07-29T00:39:52.448 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.382+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj51]: success 2023-07-29T00:39:52.448 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.383+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj55]: success 2023-07-29T00:39:52.461 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.396+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj74]: success 2023-07-29T00:39:52.461 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.396+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj93]: success 2023-07-29T00:39:52.464 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.399+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj68]: success 2023-07-29T00:39:52.466 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.401+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj85]: success 2023-07-29T00:39:52.467 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.402+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj87]: success 2023-07-29T00:39:52.467 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.402+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj95]: success 2023-07-29T00:39:52.467 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.402+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj76]: success 2023-07-29T00:39:52.467 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.403+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj92]: success 2023-07-29T00:39:52.468 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.403+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj69]: success 2023-07-29T00:39:52.468 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.403+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: backing out with sync_status=-5 2023-07-29T00:39:52.468 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.403+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: ERROR: full sync failed. error: -5 2023-07-29T00:39:52.469 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.405+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2023-07-29T00:39:52.470 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.405+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: waiting on bucket lease 2023-07-29T00:39:52.473 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.409+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: status after acquiring the lock is: full 2023-07-29T00:39:52.484 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.420+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: listing bucket for full sync 2023-07-29T00:39:52.531 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj11 2023-07-29T00:39:52.531 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj11]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj11 op=0 op_state=1 2023-07-29T00:39:52.531 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj11]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj11[0] 2023-07-29T00:39:52.531 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj111 2023-07-29T00:39:52.532 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj111]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj111 op=0 op_state=1 2023-07-29T00:39:52.532 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj111]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj111[0] 2023-07-29T00:39:52.532 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj115 2023-07-29T00:39:52.532 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj115]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj115 op=0 op_state=1 2023-07-29T00:39:52.532 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj115]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj115[0] 2023-07-29T00:39:52.533 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj116 2023-07-29T00:39:52.533 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj116]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj116 op=0 op_state=1 2023-07-29T00:39:52.533 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj116]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj116[0] 2023-07-29T00:39:52.533 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj124 2023-07-29T00:39:52.533 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj124]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj124 op=0 op_state=1 2023-07-29T00:39:52.534 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj124]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj124[0] 2023-07-29T00:39:52.534 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj145 2023-07-29T00:39:52.534 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj145]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj145 op=0 op_state=1 2023-07-29T00:39:52.534 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj145]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj145[0] 2023-07-29T00:39:52.534 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj148 2023-07-29T00:39:52.534 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj148]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj148 op=0 op_state=1 2023-07-29T00:39:52.535 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj148]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj148[0] 2023-07-29T00:39:52.535 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj155 2023-07-29T00:39:52.535 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj155]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj155 op=0 op_state=1 2023-07-29T00:39:52.535 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj155]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj155[0] 2023-07-29T00:39:52.535 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj17 2023-07-29T00:39:52.535 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj17]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj17 op=0 op_state=1 2023-07-29T00:39:52.536 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj17]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj17[0] 2023-07-29T00:39:52.536 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj185 2023-07-29T00:39:52.536 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj185]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj185 op=0 op_state=1 2023-07-29T00:39:52.536 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj185]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj185[0] 2023-07-29T00:39:52.536 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj19 2023-07-29T00:39:52.536 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj19]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj19 op=0 op_state=1 2023-07-29T00:39:52.537 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj19]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj19[0] 2023-07-29T00:39:52.537 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj22 2023-07-29T00:39:52.537 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj22]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj22 op=0 op_state=1 2023-07-29T00:39:52.537 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj22]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj22[0] 2023-07-29T00:39:52.537 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj23 2023-07-29T00:39:52.537 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj23]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj23 op=0 op_state=1 2023-07-29T00:39:52.538 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj23]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj23[0] 2023-07-29T00:39:52.538 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj29 2023-07-29T00:39:52.538 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj29]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj29 op=0 op_state=1 2023-07-29T00:39:52.538 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj29]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj29[0] 2023-07-29T00:39:52.538 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj31 2023-07-29T00:39:52.539 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj31]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj31 op=0 op_state=1 2023-07-29T00:39:52.539 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj31]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj31[0] 2023-07-29T00:39:52.539 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj33 2023-07-29T00:39:52.539 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj33]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj33 op=0 op_state=1 2023-07-29T00:39:52.539 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj33]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj33[0] 2023-07-29T00:39:52.539 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj34 2023-07-29T00:39:52.540 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj34]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj34 op=0 op_state=1 2023-07-29T00:39:52.540 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj34]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj34[0] 2023-07-29T00:39:52.540 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj37 2023-07-29T00:39:52.540 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj37]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj37 op=0 op_state=1 2023-07-29T00:39:52.540 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj37]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj37[0] 2023-07-29T00:39:52.540 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj38 2023-07-29T00:39:52.541 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj38]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj38 op=0 op_state=1 2023-07-29T00:39:52.541 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.465+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj38]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj38[0] 2023-07-29T00:39:52.541 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.466+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj44 2023-07-29T00:39:52.541 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.466+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj44]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj44 op=0 op_state=1 2023-07-29T00:39:52.541 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.466+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj44]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj44[0] 2023-07-29T00:39:52.541 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.466+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj46 2023-07-29T00:39:52.542 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.466+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj46]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj46 op=0 op_state=1 2023-07-29T00:39:52.542 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.466+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj46]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj46[0] 2023-07-29T00:39:52.543 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.471+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj11]: success 2023-07-29T00:39:52.544 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.471+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj145]: success 2023-07-29T00:39:52.544 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.471+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj51 2023-07-29T00:39:52.544 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.471+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj51]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj51 op=0 op_state=1 2023-07-29T00:39:52.544 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.471+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj51]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj51[0] 2023-07-29T00:39:52.544 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.471+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj55 2023-07-29T00:39:52.545 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.471+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj55]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj55 op=0 op_state=1 2023-07-29T00:39:52.545 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.471+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj55]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj55[0] 2023-07-29T00:39:52.545 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.472+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj148]: success 2023-07-29T00:39:52.545 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.472+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj57 2023-07-29T00:39:52.545 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.472+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj57]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj57 op=0 op_state=1 2023-07-29T00:39:52.545 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.472+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj57]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj57[0] 2023-07-29T00:39:52.546 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.472+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj33]: success 2023-07-29T00:39:52.546 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.472+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj58 2023-07-29T00:39:52.546 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.472+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj58]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj58 op=0 op_state=1 2023-07-29T00:39:52.546 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.472+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj58]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj58[0] 2023-07-29T00:39:52.546 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.472+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj31]: success 2023-07-29T00:39:52.546 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.472+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj62 2023-07-29T00:39:52.546 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.472+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj62]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj62 op=0 op_state=1 2023-07-29T00:39:52.547 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.472+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj115]: success 2023-07-29T00:39:52.547 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.472+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj17]: success 2023-07-29T00:39:52.547 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.472+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj19]: success 2023-07-29T00:39:52.547 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.472+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj62]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj62[0] 2023-07-29T00:39:52.547 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.472+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj68 2023-07-29T00:39:52.547 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.472+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj68]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj68 op=0 op_state=1 2023-07-29T00:39:52.548 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.472+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj68]: : injecting data sync error on key=obj68 2023-07-29T00:39:52.548 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.472+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj23]: success 2023-07-29T00:39:52.548 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.472+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj124]: success 2023-07-29T00:39:52.548 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.472+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj68]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:39:52.548 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.472+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj68]: ERROR: failed to sync object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj68 2023-07-29T00:39:52.548 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.472+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj69 2023-07-29T00:39:52.549 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.472+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj69]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj69 op=0 op_state=1 2023-07-29T00:39:52.549 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.472+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj185]: success 2023-07-29T00:39:52.549 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.473+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj69]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj69[0] 2023-07-29T00:39:52.549 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.473+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj74 2023-07-29T00:39:52.549 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.473+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj74]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj74 op=0 op_state=1 2023-07-29T00:39:52.549 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.473+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj74]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj74[0] 2023-07-29T00:39:52.550 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.473+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj76 2023-07-29T00:39:52.550 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.473+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj76]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj76 op=0 op_state=1 2023-07-29T00:39:52.550 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.473+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj76]: : injecting data sync error on key=obj76 2023-07-29T00:39:52.550 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.473+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj76]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:39:52.550 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.473+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj76]: ERROR: failed to sync object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj76 2023-07-29T00:39:52.550 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.473+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj85 2023-07-29T00:39:52.550 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.473+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj85]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj85 op=0 op_state=1 2023-07-29T00:39:52.551 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.473+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj85]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj85[0] 2023-07-29T00:39:52.551 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.473+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj87 2023-07-29T00:39:52.551 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.473+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj87]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj87 op=0 op_state=1 2023-07-29T00:39:52.551 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.473+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj116]: success 2023-07-29T00:39:52.551 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.473+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj87]: : injecting data sync error on key=obj87 2023-07-29T00:39:52.551 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.473+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj87]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:39:52.551 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.473+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj87]: ERROR: failed to sync object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj87 2023-07-29T00:39:52.552 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.473+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj92 2023-07-29T00:39:52.552 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.473+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj92]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj92 op=0 op_state=1 2023-07-29T00:39:52.552 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.473+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj92]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj92[0] 2023-07-29T00:39:52.552 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.476+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: a sync operation returned error 2023-07-29T00:39:52.552 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.476+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj93 2023-07-29T00:39:52.552 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.476+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj93]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj93 op=0 op_state=1 2023-07-29T00:39:52.553 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.476+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj93]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj93[0] 2023-07-29T00:39:52.553 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.476+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: a sync operation returned error 2023-07-29T00:39:52.553 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.476+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj95 2023-07-29T00:39:52.553 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.476+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj95]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj95 op=0 op_state=1 2023-07-29T00:39:52.553 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.476+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj95]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj95[0] 2023-07-29T00:39:52.554 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.476+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: a sync operation returned error 2023-07-29T00:39:52.554 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.477+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj44]: success 2023-07-29T00:39:52.554 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.477+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj38]: success 2023-07-29T00:39:52.554 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.477+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj51]: success 2023-07-29T00:39:52.554 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.477+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj37]: success 2023-07-29T00:39:52.554 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.477+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj46]: success 2023-07-29T00:39:52.554 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.477+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj57]: success 2023-07-29T00:39:52.555 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.478+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj74]: success 2023-07-29T00:39:52.555 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.478+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj62]: success 2023-07-29T00:39:52.555 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.478+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj55]: success 2023-07-29T00:39:52.555 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.478+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj58]: success 2023-07-29T00:39:52.555 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.479+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj95]: success 2023-07-29T00:39:52.556 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.479+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj69]: success 2023-07-29T00:39:52.557 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.479+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj85]: success 2023-07-29T00:39:52.557 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.480+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj92]: success 2023-07-29T00:39:52.557 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.480+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj93]: success 2023-07-29T00:39:52.557 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.480+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj155]: success 2023-07-29T00:39:52.557 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.480+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj111]: success 2023-07-29T00:39:52.558 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.480+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: updating marker oid=bucket.full-sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24 marker=obj19 2023-07-29T00:39:52.558 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.480+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj34]: success 2023-07-29T00:39:52.558 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.482+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj22]: success 2023-07-29T00:39:52.558 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.482+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj29]: success 2023-07-29T00:39:52.558 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.482+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: updating marker oid=bucket.full-sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24 marker=obj62 2023-07-29T00:39:52.558 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.483+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: backing out with sync_status=-5 2023-07-29T00:39:52.558 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.483+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: ERROR: full sync failed. error: -5 2023-07-29T00:39:52.559 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.484+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2023-07-29T00:39:52.559 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.484+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: waiting on bucket lease 2023-07-29T00:39:52.594 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.529+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: status after acquiring the lock is: full 2023-07-29T00:39:52.604 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.539+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: listing bucket for full sync 2023-07-29T00:39:52.650 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.584+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj11 2023-07-29T00:39:52.650 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.584+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj11]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj11 op=0 op_state=1 2023-07-29T00:39:52.650 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.584+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj11]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj11[0] 2023-07-29T00:39:52.651 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.584+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj111 2023-07-29T00:39:52.651 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.584+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj111]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj111 op=0 op_state=1 2023-07-29T00:39:52.651 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.584+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj111]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj111[0] 2023-07-29T00:39:52.651 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.584+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj115 2023-07-29T00:39:52.651 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.584+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj115]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj115 op=0 op_state=1 2023-07-29T00:39:52.651 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.584+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj115]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj115[0] 2023-07-29T00:39:52.652 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.584+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj116 2023-07-29T00:39:52.652 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.584+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj116]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj116 op=0 op_state=1 2023-07-29T00:39:52.652 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.584+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj116]: : injecting data sync error on key=obj116 2023-07-29T00:39:52.652 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.584+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj116]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:39:52.652 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.584+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj116]: ERROR: failed to sync object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj116 2023-07-29T00:39:52.652 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.584+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj124 2023-07-29T00:39:52.653 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.584+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj124]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj124 op=0 op_state=1 2023-07-29T00:39:52.653 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.584+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj124]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj124[0] 2023-07-29T00:39:52.653 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.584+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj145 2023-07-29T00:39:52.654 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.584+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj145]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj145 op=0 op_state=1 2023-07-29T00:39:52.654 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.584+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj145]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj145[0] 2023-07-29T00:39:52.655 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.584+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj148 2023-07-29T00:39:52.655 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.584+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj148]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj148 op=0 op_state=1 2023-07-29T00:39:52.655 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.584+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj148]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj148[0] 2023-07-29T00:39:52.655 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.584+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj155 2023-07-29T00:39:52.655 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.584+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj155]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj155 op=0 op_state=1 2023-07-29T00:39:52.655 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.584+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj155]: : injecting data sync error on key=obj155 2023-07-29T00:39:52.656 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.584+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj155]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:39:52.656 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.584+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj155]: ERROR: failed to sync object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj155 2023-07-29T00:39:52.656 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.584+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj17 2023-07-29T00:39:52.656 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.584+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj17]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj17 op=0 op_state=1 2023-07-29T00:39:52.656 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.584+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj17]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj17[0] 2023-07-29T00:39:52.656 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.584+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj185 2023-07-29T00:39:52.657 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.584+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj185]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj185 op=0 op_state=1 2023-07-29T00:39:52.657 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.584+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj185]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj185[0] 2023-07-29T00:39:52.657 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.585+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj19 2023-07-29T00:39:52.657 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.585+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj19]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj19 op=0 op_state=1 2023-07-29T00:39:52.657 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.585+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj19]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj19[0] 2023-07-29T00:39:52.657 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.585+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj22 2023-07-29T00:39:52.657 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.585+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj22]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj22 op=0 op_state=1 2023-07-29T00:39:52.658 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.585+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj22]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj22[0] 2023-07-29T00:39:52.658 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.585+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj23 2023-07-29T00:39:52.658 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.585+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj23]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj23 op=0 op_state=1 2023-07-29T00:39:52.658 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.585+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj23]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj23[0] 2023-07-29T00:39:52.658 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.585+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj29 2023-07-29T00:39:52.658 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.585+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj29]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj29 op=0 op_state=1 2023-07-29T00:39:52.659 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.585+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj29]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj29[0] 2023-07-29T00:39:52.659 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.585+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj31 2023-07-29T00:39:52.659 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.585+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj31]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj31 op=0 op_state=1 2023-07-29T00:39:52.659 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.585+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj31]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj31[0] 2023-07-29T00:39:52.659 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.585+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj33 2023-07-29T00:39:52.659 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.585+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj33]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj33 op=0 op_state=1 2023-07-29T00:39:52.659 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.585+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj33]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj33[0] 2023-07-29T00:39:52.660 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.585+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj34 2023-07-29T00:39:52.660 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.585+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj34]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj34 op=0 op_state=1 2023-07-29T00:39:52.660 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.585+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj34]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj34[0] 2023-07-29T00:39:52.660 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.585+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj37 2023-07-29T00:39:52.660 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.585+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj37]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj37 op=0 op_state=1 2023-07-29T00:39:52.660 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.585+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj37]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj37[0] 2023-07-29T00:39:52.661 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.585+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj38 2023-07-29T00:39:52.661 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.585+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj38]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj38 op=0 op_state=1 2023-07-29T00:39:52.661 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.585+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj38]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj38[0] 2023-07-29T00:39:52.661 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.585+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj44 2023-07-29T00:39:52.661 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.585+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj44]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj44 op=0 op_state=1 2023-07-29T00:39:52.661 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.585+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj44]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj44[0] 2023-07-29T00:39:52.662 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.585+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj46 2023-07-29T00:39:52.662 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.585+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj46]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj46 op=0 op_state=1 2023-07-29T00:39:52.662 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.585+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj46]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj46[0] 2023-07-29T00:39:52.666 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.588+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj124]: success 2023-07-29T00:39:52.666 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.588+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: a sync operation returned error 2023-07-29T00:39:52.666 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.588+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj51 2023-07-29T00:39:52.667 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.588+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj51]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj51 op=0 op_state=1 2023-07-29T00:39:52.667 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.588+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj51]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj51[0] 2023-07-29T00:39:52.667 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.588+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj55 2023-07-29T00:39:52.667 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.588+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj55]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj55 op=0 op_state=1 2023-07-29T00:39:52.667 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.588+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj55]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj55[0] 2023-07-29T00:39:52.667 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.588+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: a sync operation returned error 2023-07-29T00:39:52.668 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.589+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj34]: success 2023-07-29T00:39:52.668 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.589+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj57 2023-07-29T00:39:52.668 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.589+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj57]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj57 op=0 op_state=1 2023-07-29T00:39:52.668 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.589+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj57]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj57[0] 2023-07-29T00:39:52.668 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.589+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj58 2023-07-29T00:39:52.668 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.589+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj58]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj58 op=0 op_state=1 2023-07-29T00:39:52.669 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.590+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj58]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj58[0] 2023-07-29T00:39:52.669 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.590+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj11]: success 2023-07-29T00:39:52.669 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.590+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj62 2023-07-29T00:39:52.669 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.590+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj62]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj62 op=0 op_state=1 2023-07-29T00:39:52.669 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.590+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj62]: : injecting data sync error on key=obj62 2023-07-29T00:39:52.669 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.590+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj62]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:39:52.669 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.590+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj62]: ERROR: failed to sync object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj62 2023-07-29T00:39:52.670 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.590+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj37]: success 2023-07-29T00:39:52.670 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.591+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj19]: success 2023-07-29T00:39:52.670 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.591+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj44]: success 2023-07-29T00:39:52.670 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.591+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj29]: success 2023-07-29T00:39:52.670 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.591+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj115]: success 2023-07-29T00:39:52.670 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.591+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj31]: success 2023-07-29T00:39:52.671 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.591+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj33]: success 2023-07-29T00:39:52.671 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.591+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj68 2023-07-29T00:39:52.671 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.591+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj68]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj68 op=0 op_state=1 2023-07-29T00:39:52.671 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.591+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj68]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj68[0] 2023-07-29T00:39:52.671 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.591+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj69 2023-07-29T00:39:52.671 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.591+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj69]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj69 op=0 op_state=1 2023-07-29T00:39:52.671 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.591+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj69]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj69[0] 2023-07-29T00:39:52.672 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.591+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj74 2023-07-29T00:39:52.672 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.591+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj74]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj74 op=0 op_state=1 2023-07-29T00:39:52.672 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.591+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj74]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj74[0] 2023-07-29T00:39:52.672 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.591+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj76 2023-07-29T00:39:52.672 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.591+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj76]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj76 op=0 op_state=1 2023-07-29T00:39:52.672 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.591+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj76]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj76[0] 2023-07-29T00:39:52.673 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.591+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj85 2023-07-29T00:39:52.673 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.591+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj85]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj85 op=0 op_state=1 2023-07-29T00:39:52.673 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.591+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj17]: success 2023-07-29T00:39:52.673 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.591+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj148]: success 2023-07-29T00:39:52.673 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.591+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj85]: : injecting data sync error on key=obj85 2023-07-29T00:39:52.673 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.591+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj38]: success 2023-07-29T00:39:52.673 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.591+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj111]: success 2023-07-29T00:39:52.674 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.591+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: updating marker oid=bucket.full-sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24 marker=obj115 2023-07-29T00:39:52.674 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.591+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj85]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:39:52.674 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.591+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj85]: ERROR: failed to sync object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj85 2023-07-29T00:39:52.674 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.591+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj185]: success 2023-07-29T00:39:52.674 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.592+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj46]: success 2023-07-29T00:39:52.674 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.592+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj87 2023-07-29T00:39:52.674 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.592+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj87]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj87 op=0 op_state=1 2023-07-29T00:39:52.675 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.592+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj87]: : injecting data sync error on key=obj87 2023-07-29T00:39:52.675 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.592+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj22]: success 2023-07-29T00:39:52.675 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.592+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj87]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:39:52.675 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.592+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj87]: ERROR: failed to sync object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj87 2023-07-29T00:39:52.675 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.592+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj92 2023-07-29T00:39:52.675 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.592+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj92]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj92 op=0 op_state=1 2023-07-29T00:39:52.675 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.592+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj92]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj92[0] 2023-07-29T00:39:52.676 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.592+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj145]: success 2023-07-29T00:39:52.676 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.592+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj93 2023-07-29T00:39:52.676 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.592+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj93]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj93 op=0 op_state=1 2023-07-29T00:39:52.676 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.592+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj93]: : injecting data sync error on key=obj93 2023-07-29T00:39:52.676 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.592+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj93]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:39:52.676 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.592+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj93]: ERROR: failed to sync object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj93 2023-07-29T00:39:52.677 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.592+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj95 2023-07-29T00:39:52.677 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.592+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj95]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj95 op=0 op_state=1 2023-07-29T00:39:52.677 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.592+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj23]: success 2023-07-29T00:39:52.677 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.592+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj55]: success 2023-07-29T00:39:52.677 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.592+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj95]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj95[0] 2023-07-29T00:39:52.677 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.594+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj51]: success 2023-07-29T00:39:52.677 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.594+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj57]: success 2023-07-29T00:39:52.678 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.594+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: a sync operation returned error 2023-07-29T00:39:52.678 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.595+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: a sync operation returned error 2023-07-29T00:39:52.678 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.595+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: a sync operation returned error 2023-07-29T00:39:52.678 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.595+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj58]: success 2023-07-29T00:39:52.678 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.595+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj69]: success 2023-07-29T00:39:52.678 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.595+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj95]: success 2023-07-29T00:39:52.679 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.595+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj74]: success 2023-07-29T00:39:52.679 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.595+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: a sync operation returned error 2023-07-29T00:39:52.679 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.595+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj68]: success 2023-07-29T00:39:52.679 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.595+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj76]: success 2023-07-29T00:39:52.679 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.595+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj92]: success 2023-07-29T00:39:52.679 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.595+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: backing out with sync_status=-5 2023-07-29T00:39:52.679 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.595+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: ERROR: full sync failed. error: -5 2023-07-29T00:39:52.680 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.597+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2023-07-29T00:39:52.680 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.597+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: waiting on bucket lease 2023-07-29T00:39:52.706 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.641+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: status after acquiring the lock is: full 2023-07-29T00:39:52.716 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.652+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: listing bucket for full sync 2023-07-29T00:39:52.765 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.698+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj11 2023-07-29T00:39:52.765 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.698+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj11]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj11 op=0 op_state=1 2023-07-29T00:39:52.766 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.698+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj11]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj11[0] 2023-07-29T00:39:52.766 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.698+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj111 2023-07-29T00:39:52.766 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.698+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj111]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj111 op=0 op_state=1 2023-07-29T00:39:52.766 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.698+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj111]: : injecting data sync error on key=obj111 2023-07-29T00:39:52.766 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.698+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj111]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:39:52.766 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.698+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj111]: ERROR: failed to sync object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj111 2023-07-29T00:39:52.767 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.698+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj115 2023-07-29T00:39:52.767 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.698+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj115]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj115 op=0 op_state=1 2023-07-29T00:39:52.767 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.698+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj115]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj115[0] 2023-07-29T00:39:52.767 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.698+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj116 2023-07-29T00:39:52.767 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.698+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj116]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj116 op=0 op_state=1 2023-07-29T00:39:52.767 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.698+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj116]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj116[0] 2023-07-29T00:39:52.768 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.698+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj124 2023-07-29T00:39:52.768 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.698+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj124]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj124 op=0 op_state=1 2023-07-29T00:39:52.768 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.698+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj124]: : injecting data sync error on key=obj124 2023-07-29T00:39:52.768 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.698+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj124]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:39:52.768 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.698+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj124]: ERROR: failed to sync object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj124 2023-07-29T00:39:52.768 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.698+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj145 2023-07-29T00:39:52.769 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.698+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj145]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj145 op=0 op_state=1 2023-07-29T00:39:52.769 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.698+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj145]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj145[0] 2023-07-29T00:39:52.769 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.698+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj148 2023-07-29T00:39:52.769 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.698+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj148]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj148 op=0 op_state=1 2023-07-29T00:39:52.769 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.698+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj148]: : injecting data sync error on key=obj148 2023-07-29T00:39:52.769 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.698+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj148]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:39:52.769 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.698+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj148]: ERROR: failed to sync object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj148 2023-07-29T00:39:52.770 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.698+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj155 2023-07-29T00:39:52.770 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.698+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj155]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj155 op=0 op_state=1 2023-07-29T00:39:52.770 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.698+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj155]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj155[0] 2023-07-29T00:39:52.770 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.699+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj17 2023-07-29T00:39:52.770 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.699+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj17]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj17 op=0 op_state=1 2023-07-29T00:39:52.770 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.699+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj17]: : injecting data sync error on key=obj17 2023-07-29T00:39:52.771 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.699+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj17]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:39:52.771 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.699+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj17]: ERROR: failed to sync object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj17 2023-07-29T00:39:52.771 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.699+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj185 2023-07-29T00:39:52.771 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.699+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj185]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj185 op=0 op_state=1 2023-07-29T00:39:52.771 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.699+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj185]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj185[0] 2023-07-29T00:39:52.771 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.699+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj19 2023-07-29T00:39:52.771 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.699+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj19]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj19 op=0 op_state=1 2023-07-29T00:39:52.772 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.699+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj19]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj19[0] 2023-07-29T00:39:52.772 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.699+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj22 2023-07-29T00:39:52.772 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.699+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj22]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj22 op=0 op_state=1 2023-07-29T00:39:52.772 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.699+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj22]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj22[0] 2023-07-29T00:39:52.772 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.699+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj23 2023-07-29T00:39:52.772 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.699+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj23]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj23 op=0 op_state=1 2023-07-29T00:39:52.773 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.699+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj23]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj23[0] 2023-07-29T00:39:52.773 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.699+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj29 2023-07-29T00:39:52.773 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.699+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj29]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj29 op=0 op_state=1 2023-07-29T00:39:52.773 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.699+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj29]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj29[0] 2023-07-29T00:39:52.773 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.699+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj31 2023-07-29T00:39:52.773 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.699+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj31]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj31 op=0 op_state=1 2023-07-29T00:39:52.773 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.699+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj31]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj31[0] 2023-07-29T00:39:52.774 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.700+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj33 2023-07-29T00:39:52.774 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.700+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj33]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj33 op=0 op_state=1 2023-07-29T00:39:52.774 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.700+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj33]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj33[0] 2023-07-29T00:39:52.774 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.700+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj34 2023-07-29T00:39:52.774 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.700+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj34]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj34 op=0 op_state=1 2023-07-29T00:39:52.774 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.700+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj34]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj34[0] 2023-07-29T00:39:52.775 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.700+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj37 2023-07-29T00:39:52.775 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.700+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj37]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj37 op=0 op_state=1 2023-07-29T00:39:52.775 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.700+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj37]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj37[0] 2023-07-29T00:39:52.775 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.700+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj38 2023-07-29T00:39:52.775 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.700+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj38]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj38 op=0 op_state=1 2023-07-29T00:39:52.775 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.700+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj38]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj38[0] 2023-07-29T00:39:52.775 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.700+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj44 2023-07-29T00:39:52.776 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.700+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj44]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj44 op=0 op_state=1 2023-07-29T00:39:52.776 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.700+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj44]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj44[0] 2023-07-29T00:39:52.776 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.700+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj46 2023-07-29T00:39:52.776 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.700+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj46]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj46 op=0 op_state=1 2023-07-29T00:39:52.776 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.700+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj46]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj46[0] 2023-07-29T00:39:52.779 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.701+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: a sync operation returned error 2023-07-29T00:39:52.780 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.701+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj51 2023-07-29T00:39:52.780 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.701+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj51]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj51 op=0 op_state=1 2023-07-29T00:39:52.780 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.701+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj51]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj51[0] 2023-07-29T00:39:52.780 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.701+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: a sync operation returned error 2023-07-29T00:39:52.780 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.701+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: a sync operation returned error 2023-07-29T00:39:52.781 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.701+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj55 2023-07-29T00:39:52.781 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.701+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj55]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj55 op=0 op_state=1 2023-07-29T00:39:52.781 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.701+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj55]: : injecting data sync error on key=obj55 2023-07-29T00:39:52.781 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.701+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj55]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:39:52.781 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.701+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj55]: ERROR: failed to sync object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj55 2023-07-29T00:39:52.781 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.701+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj57 2023-07-29T00:39:52.782 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.701+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj57]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj57 op=0 op_state=1 2023-07-29T00:39:52.782 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.701+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj57]: : injecting data sync error on key=obj57 2023-07-29T00:39:52.782 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.701+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj57]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:39:52.782 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.701+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj57]: ERROR: failed to sync object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj57 2023-07-29T00:39:52.782 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.703+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: a sync operation returned error 2023-07-29T00:39:52.782 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.703+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj58 2023-07-29T00:39:52.783 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.703+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj58]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj58 op=0 op_state=1 2023-07-29T00:39:52.783 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.703+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj58]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj58[0] 2023-07-29T00:39:52.783 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.704+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj11]: success 2023-07-29T00:39:52.783 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.704+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj115]: success 2023-07-29T00:39:52.783 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.704+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj145]: success 2023-07-29T00:39:52.784 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.704+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj19]: success 2023-07-29T00:39:52.784 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.704+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj155]: success 2023-07-29T00:39:52.784 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.704+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj185]: success 2023-07-29T00:39:52.784 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.704+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj31]: success 2023-07-29T00:39:52.784 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.704+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj29]: success 2023-07-29T00:39:52.785 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.704+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj62 2023-07-29T00:39:52.785 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.704+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj62]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj62 op=0 op_state=1 2023-07-29T00:39:52.785 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.705+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj62]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj62[0] 2023-07-29T00:39:52.785 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.705+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj68 2023-07-29T00:39:52.785 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.705+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj68]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj68 op=0 op_state=1 2023-07-29T00:39:52.785 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.705+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj116]: success 2023-07-29T00:39:52.786 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.705+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj68]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj68[0] 2023-07-29T00:39:52.786 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.705+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj69 2023-07-29T00:39:52.786 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.706+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj69]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj69 op=0 op_state=1 2023-07-29T00:39:52.786 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.706+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj69]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj69[0] 2023-07-29T00:39:52.786 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.706+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj74 2023-07-29T00:39:52.786 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.706+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj74]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj74 op=0 op_state=1 2023-07-29T00:39:52.786 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.706+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj37]: success 2023-07-29T00:39:52.787 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.706+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj33]: success 2023-07-29T00:39:52.787 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.706+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj38]: success 2023-07-29T00:39:52.787 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.706+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj74]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj74[0] 2023-07-29T00:39:52.787 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.706+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj76 2023-07-29T00:39:52.787 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.706+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj76]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj76 op=0 op_state=1 2023-07-29T00:39:52.787 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.706+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj22]: success 2023-07-29T00:39:52.787 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.706+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj76]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj76[0] 2023-07-29T00:39:52.788 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.706+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj85 2023-07-29T00:39:52.788 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.706+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj85]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj85 op=0 op_state=1 2023-07-29T00:39:52.788 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.706+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj85]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj85[0] 2023-07-29T00:39:52.789 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.706+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj87 2023-07-29T00:39:52.789 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.706+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj87]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj87 op=0 op_state=1 2023-07-29T00:39:52.789 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.706+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj87]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj87[0] 2023-07-29T00:39:52.789 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.706+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj23]: success 2023-07-29T00:39:52.789 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.706+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj92 2023-07-29T00:39:52.789 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.706+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj92]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj92 op=0 op_state=1 2023-07-29T00:39:52.789 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.706+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj34]: success 2023-07-29T00:39:52.790 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.706+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj44]: success 2023-07-29T00:39:52.790 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.706+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj92]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj92[0] 2023-07-29T00:39:52.790 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.706+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj46]: success 2023-07-29T00:39:52.790 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.707+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: a sync operation returned error 2023-07-29T00:39:52.790 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.707+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: a sync operation returned error 2023-07-29T00:39:52.790 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.707+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj93 2023-07-29T00:39:52.790 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.707+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj93]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj93 op=0 op_state=1 2023-07-29T00:39:52.791 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.707+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj93]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj93[0] 2023-07-29T00:39:52.791 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.707+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj95 2023-07-29T00:39:52.791 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.707+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj95]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj95 op=0 op_state=1 2023-07-29T00:39:52.791 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.707+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj95]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj95[0] 2023-07-29T00:39:52.791 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.707+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj58]: success 2023-07-29T00:39:52.791 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.707+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj51]: success 2023-07-29T00:39:52.792 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.708+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj62]: success 2023-07-29T00:39:52.792 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.709+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj69]: success 2023-07-29T00:39:52.792 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.709+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj76]: success 2023-07-29T00:39:52.792 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.709+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj68]: success 2023-07-29T00:39:52.792 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.709+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj85]: success 2023-07-29T00:39:52.792 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.709+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj74]: success 2023-07-29T00:39:52.792 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.709+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj92]: success 2023-07-29T00:39:52.793 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.709+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj87]: success 2023-07-29T00:39:52.793 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.709+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj95]: success 2023-07-29T00:39:52.793 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.709+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj93]: success 2023-07-29T00:39:52.793 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.709+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: updating marker oid=bucket.full-sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24 marker=obj11 2023-07-29T00:39:52.793 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.711+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: backing out with sync_status=-5 2023-07-29T00:39:52.793 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.711+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: ERROR: full sync failed. error: -5 2023-07-29T00:39:52.794 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.712+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2023-07-29T00:39:52.794 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.712+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: waiting on bucket lease 2023-07-29T00:39:52.822 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.757+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: status after acquiring the lock is: full 2023-07-29T00:39:52.833 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.768+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: listing bucket for full sync 2023-07-29T00:39:52.879 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.813+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj11 2023-07-29T00:39:52.879 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.813+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj11]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj11 op=0 op_state=1 2023-07-29T00:39:52.879 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.813+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj11]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj11[0] 2023-07-29T00:39:52.880 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.813+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj111 2023-07-29T00:39:52.880 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.813+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj111]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj111 op=0 op_state=1 2023-07-29T00:39:52.880 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.813+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj111]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj111[0] 2023-07-29T00:39:52.880 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.813+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj115 2023-07-29T00:39:52.880 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.813+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj115]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj115 op=0 op_state=1 2023-07-29T00:39:52.880 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.813+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj115]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj115[0] 2023-07-29T00:39:52.881 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.813+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj116 2023-07-29T00:39:52.881 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.813+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj116]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj116 op=0 op_state=1 2023-07-29T00:39:52.881 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.813+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj116]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj116[0] 2023-07-29T00:39:52.881 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.813+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj124 2023-07-29T00:39:52.881 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.813+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj124]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj124 op=0 op_state=1 2023-07-29T00:39:52.881 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.813+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj124]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj124[0] 2023-07-29T00:39:52.881 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.813+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj145 2023-07-29T00:39:52.882 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.813+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj145]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj145 op=0 op_state=1 2023-07-29T00:39:52.882 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.813+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj145]: : injecting data sync error on key=obj145 2023-07-29T00:39:52.882 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.813+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj145]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:39:52.882 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.813+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj145]: ERROR: failed to sync object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj145 2023-07-29T00:39:52.882 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.813+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj148 2023-07-29T00:39:52.882 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.813+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj148]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj148 op=0 op_state=1 2023-07-29T00:39:52.883 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.813+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj148]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj148[0] 2023-07-29T00:39:52.883 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.813+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj155 2023-07-29T00:39:52.883 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.813+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj155]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj155 op=0 op_state=1 2023-07-29T00:39:52.883 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.813+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj155]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj155[0] 2023-07-29T00:39:52.883 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.813+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj17 2023-07-29T00:39:52.883 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.813+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj17]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj17 op=0 op_state=1 2023-07-29T00:39:52.884 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.813+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj17]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj17[0] 2023-07-29T00:39:52.884 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.813+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj185 2023-07-29T00:39:52.884 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.813+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj185]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj185 op=0 op_state=1 2023-07-29T00:39:52.884 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.813+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj185]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj185[0] 2023-07-29T00:39:52.884 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.813+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj19 2023-07-29T00:39:52.884 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.813+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj19]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj19 op=0 op_state=1 2023-07-29T00:39:52.885 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.814+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj19]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj19[0] 2023-07-29T00:39:52.885 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.814+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj22 2023-07-29T00:39:52.885 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.814+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj22]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj22 op=0 op_state=1 2023-07-29T00:39:52.885 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.814+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj22]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj22[0] 2023-07-29T00:39:52.890 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.814+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj23 2023-07-29T00:39:52.890 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.814+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj23]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj23 op=0 op_state=1 2023-07-29T00:39:52.890 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.814+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj23]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj23[0] 2023-07-29T00:39:52.890 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.814+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj29 2023-07-29T00:39:52.890 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.814+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj29]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj29 op=0 op_state=1 2023-07-29T00:39:52.891 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.814+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj29]: : injecting data sync error on key=obj29 2023-07-29T00:39:52.891 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.814+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj29]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:39:52.891 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.814+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj29]: ERROR: failed to sync object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj29 2023-07-29T00:39:52.891 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.814+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj31 2023-07-29T00:39:52.891 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.814+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj31]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj31 op=0 op_state=1 2023-07-29T00:39:52.891 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.814+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj31]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj31[0] 2023-07-29T00:39:52.892 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.814+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj33 2023-07-29T00:39:52.892 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.814+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj33]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj33 op=0 op_state=1 2023-07-29T00:39:52.892 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.814+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj33]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj33[0] 2023-07-29T00:39:52.892 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.814+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj34 2023-07-29T00:39:52.892 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.814+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj34]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj34 op=0 op_state=1 2023-07-29T00:39:52.892 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.814+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj34]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj34[0] 2023-07-29T00:39:52.892 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.814+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj37 2023-07-29T00:39:52.893 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.814+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj37]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj37 op=0 op_state=1 2023-07-29T00:39:52.893 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.814+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj37]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj37[0] 2023-07-29T00:39:52.893 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.814+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj38 2023-07-29T00:39:52.893 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.814+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj38]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj38 op=0 op_state=1 2023-07-29T00:39:52.893 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.814+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj38]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj38[0] 2023-07-29T00:39:52.893 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.814+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj44 2023-07-29T00:39:52.894 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.814+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj44]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj44 op=0 op_state=1 2023-07-29T00:39:52.894 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.814+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj44]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj44[0] 2023-07-29T00:39:52.894 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.814+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj46 2023-07-29T00:39:52.894 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.814+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj46]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj46 op=0 op_state=1 2023-07-29T00:39:52.894 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.814+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj46]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj46[0] 2023-07-29T00:39:52.894 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.817+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: a sync operation returned error 2023-07-29T00:39:52.894 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.817+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj51 2023-07-29T00:39:52.895 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.817+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj51]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj51 op=0 op_state=1 2023-07-29T00:39:52.895 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.817+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj51]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj51[0] 2023-07-29T00:39:52.895 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.817+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: a sync operation returned error 2023-07-29T00:39:52.895 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.817+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj55 2023-07-29T00:39:52.895 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.818+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj55]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj55 op=0 op_state=1 2023-07-29T00:39:52.896 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.818+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj55]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj55[0] 2023-07-29T00:39:52.896 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.818+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj116]: success 2023-07-29T00:39:52.896 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.818+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj124]: success 2023-07-29T00:39:52.896 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.818+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj57 2023-07-29T00:39:52.896 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.819+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj57]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj57 op=0 op_state=1 2023-07-29T00:39:52.896 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.819+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj57]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj57[0] 2023-07-29T00:39:52.897 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.819+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj115]: success 2023-07-29T00:39:52.897 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.819+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj148]: success 2023-07-29T00:39:52.897 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.819+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj58 2023-07-29T00:39:52.897 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.819+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj58]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj58 op=0 op_state=1 2023-07-29T00:39:52.897 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.819+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj17]: success 2023-07-29T00:39:52.897 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.819+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj33]: success 2023-07-29T00:39:52.898 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.819+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj37]: success 2023-07-29T00:39:52.898 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.819+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj11]: success 2023-07-29T00:39:52.898 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.819+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj58]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj58[0] 2023-07-29T00:39:52.898 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.819+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj62 2023-07-29T00:39:52.898 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.819+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj62]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj62 op=0 op_state=1 2023-07-29T00:39:52.898 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.819+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj62]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj62[0] 2023-07-29T00:39:52.898 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.820+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj68 2023-07-29T00:39:52.899 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.820+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj68]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj68 op=0 op_state=1 2023-07-29T00:39:52.899 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.820+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj34]: success 2023-07-29T00:39:52.899 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.820+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj68]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj68[0] 2023-07-29T00:39:52.899 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.820+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj31]: success 2023-07-29T00:39:52.899 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.820+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj111]: success 2023-07-29T00:39:52.899 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.820+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: updating marker oid=bucket.full-sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24 marker=obj124 2023-07-29T00:39:52.899 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.820+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj69 2023-07-29T00:39:52.900 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.820+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj69]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj69 op=0 op_state=1 2023-07-29T00:39:52.900 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.820+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj69]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj69[0] 2023-07-29T00:39:52.900 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.820+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj74 2023-07-29T00:39:52.900 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.820+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj74]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj74 op=0 op_state=1 2023-07-29T00:39:52.900 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.821+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj44]: success 2023-07-29T00:39:52.900 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.821+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj19]: success 2023-07-29T00:39:52.901 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.821+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj74]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj74[0] 2023-07-29T00:39:52.901 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.821+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj76 2023-07-29T00:39:52.901 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.821+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj76]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj76 op=0 op_state=1 2023-07-29T00:39:52.901 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.821+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj76]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj76[0] 2023-07-29T00:39:52.901 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.821+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj85 2023-07-29T00:39:52.901 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.821+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj85]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj85 op=0 op_state=1 2023-07-29T00:39:52.902 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.821+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj85]: : injecting data sync error on key=obj85 2023-07-29T00:39:52.902 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.821+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj51]: success 2023-07-29T00:39:52.902 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.821+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj85]: failed, retcode=-5 ((5) Input/output error) 2023-07-29T00:39:52.902 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.821+0000 7f7342926880 0 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj85]: ERROR: failed to sync object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj85 2023-07-29T00:39:52.902 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.821+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj46]: success 2023-07-29T00:39:52.902 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.821+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj185]: success 2023-07-29T00:39:52.902 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.821+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj155]: success 2023-07-29T00:39:52.903 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.821+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj87 2023-07-29T00:39:52.903 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.821+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj87]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj87 op=0 op_state=1 2023-07-29T00:39:52.903 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.822+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj87]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj87[0] 2023-07-29T00:39:52.903 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.822+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj92 2023-07-29T00:39:52.903 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.822+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj92]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj92 op=0 op_state=1 2023-07-29T00:39:52.903 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.822+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj92]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj92[0] 2023-07-29T00:39:52.903 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.822+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj93 2023-07-29T00:39:52.904 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.822+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj93]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj93 op=0 op_state=1 2023-07-29T00:39:52.904 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.822+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj93]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj93[0] 2023-07-29T00:39:52.904 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.822+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj22]: success 2023-07-29T00:39:52.904 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.822+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj95 2023-07-29T00:39:52.904 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.822+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj95]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj95 op=0 op_state=1 2023-07-29T00:39:52.904 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.822+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj38]: success 2023-07-29T00:39:52.905 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.822+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj23]: success 2023-07-29T00:39:52.905 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.822+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj95]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj95[0] 2023-07-29T00:39:52.905 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.822+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj55]: success 2023-07-29T00:39:52.905 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.823+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj57]: success 2023-07-29T00:39:52.905 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.823+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: a sync operation returned error 2023-07-29T00:39:52.905 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.824+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj62]: success 2023-07-29T00:39:52.906 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.824+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj76]: success 2023-07-29T00:39:52.906 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.824+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj58]: success 2023-07-29T00:39:52.906 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.824+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj74]: success 2023-07-29T00:39:52.906 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.824+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj68]: success 2023-07-29T00:39:52.906 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.825+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj87]: success 2023-07-29T00:39:52.906 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.825+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj69]: success 2023-07-29T00:39:52.907 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.825+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj95]: success 2023-07-29T00:39:52.907 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.825+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj92]: success 2023-07-29T00:39:52.907 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.825+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj93]: success 2023-07-29T00:39:52.907 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.825+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: backing out with sync_status=-5 2023-07-29T00:39:52.907 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.825+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: ERROR: full sync failed. error: -5 2023-07-29T00:39:52.908 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.826+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2023-07-29T00:39:52.908 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.826+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: waiting on bucket lease 2023-07-29T00:39:52.936 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.871+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: status after acquiring the lock is: full 2023-07-29T00:39:52.947 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.882+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: listing bucket for full sync 2023-07-29T00:39:52.993 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.927+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj11 2023-07-29T00:39:52.994 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.927+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj11]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj11 op=0 op_state=1 2023-07-29T00:39:52.994 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.927+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj11]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj11[0] 2023-07-29T00:39:52.994 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.927+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj111 2023-07-29T00:39:52.994 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.927+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj111]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj111 op=0 op_state=1 2023-07-29T00:39:52.994 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.927+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj111]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj111[0] 2023-07-29T00:39:52.995 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.927+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj115 2023-07-29T00:39:52.995 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.927+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj115]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj115 op=0 op_state=1 2023-07-29T00:39:52.995 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.927+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj115]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj115[0] 2023-07-29T00:39:52.995 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.927+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj116 2023-07-29T00:39:52.995 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.927+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj116]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj116 op=0 op_state=1 2023-07-29T00:39:52.995 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.927+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj116]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj116[0] 2023-07-29T00:39:52.996 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.927+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj124 2023-07-29T00:39:52.996 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.927+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj124]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj124 op=0 op_state=1 2023-07-29T00:39:52.996 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.927+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj124]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj124[0] 2023-07-29T00:39:52.996 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.927+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj145 2023-07-29T00:39:52.996 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.927+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj145]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj145 op=0 op_state=1 2023-07-29T00:39:52.996 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.927+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj145]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj145[0] 2023-07-29T00:39:52.996 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.927+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj148 2023-07-29T00:39:52.997 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.927+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj148]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj148 op=0 op_state=1 2023-07-29T00:39:52.997 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.927+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj148]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj148[0] 2023-07-29T00:39:52.997 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.927+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj155 2023-07-29T00:39:52.997 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.927+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj155]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj155 op=0 op_state=1 2023-07-29T00:39:52.997 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.927+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj155]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj155[0] 2023-07-29T00:39:52.997 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.927+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj17 2023-07-29T00:39:52.998 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.927+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj17]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj17 op=0 op_state=1 2023-07-29T00:39:52.998 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.927+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj17]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj17[0] 2023-07-29T00:39:52.998 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.927+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj185 2023-07-29T00:39:52.998 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.927+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj185]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj185 op=0 op_state=1 2023-07-29T00:39:52.998 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.927+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj185]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj185[0] 2023-07-29T00:39:52.998 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.928+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj19 2023-07-29T00:39:52.999 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.928+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj19]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj19 op=0 op_state=1 2023-07-29T00:39:52.999 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.928+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj19]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj19[0] 2023-07-29T00:39:52.999 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.928+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj22 2023-07-29T00:39:52.999 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.928+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj22]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj22 op=0 op_state=1 2023-07-29T00:39:52.999 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.928+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj22]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj22[0] 2023-07-29T00:39:52.999 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.928+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj23 2023-07-29T00:39:53.000 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.928+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj23]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj23 op=0 op_state=1 2023-07-29T00:39:53.000 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.928+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj23]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj23[0] 2023-07-29T00:39:53.000 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.928+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj29 2023-07-29T00:39:53.000 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.928+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj29]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj29 op=0 op_state=1 2023-07-29T00:39:53.000 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.928+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj29]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj29[0] 2023-07-29T00:39:53.000 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.928+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj31 2023-07-29T00:39:53.000 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.928+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj31]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj31 op=0 op_state=1 2023-07-29T00:39:53.001 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.928+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj31]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj31[0] 2023-07-29T00:39:53.001 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.928+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj33 2023-07-29T00:39:53.001 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.928+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj33]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj33 op=0 op_state=1 2023-07-29T00:39:53.001 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.928+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj33]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj33[0] 2023-07-29T00:39:53.001 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.928+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj34 2023-07-29T00:39:53.001 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.928+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj34]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj34 op=0 op_state=1 2023-07-29T00:39:53.002 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.928+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj34]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj34[0] 2023-07-29T00:39:53.002 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.928+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj37 2023-07-29T00:39:53.002 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.928+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj37]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj37 op=0 op_state=1 2023-07-29T00:39:53.002 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.928+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj37]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj37[0] 2023-07-29T00:39:53.002 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.928+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj38 2023-07-29T00:39:53.002 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.928+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj38]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj38 op=0 op_state=1 2023-07-29T00:39:53.002 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.928+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj38]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj38[0] 2023-07-29T00:39:53.003 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.928+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj44 2023-07-29T00:39:53.003 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.928+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj44]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj44 op=0 op_state=1 2023-07-29T00:39:53.003 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.928+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj44]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj44[0] 2023-07-29T00:39:53.003 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.928+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj46 2023-07-29T00:39:53.003 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.928+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj46]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj46 op=0 op_state=1 2023-07-29T00:39:53.003 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.928+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj46]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj46[0] 2023-07-29T00:39:53.006 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.932+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj111]: success 2023-07-29T00:39:53.006 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.932+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj51 2023-07-29T00:39:53.006 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.932+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj51]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj51 op=0 op_state=1 2023-07-29T00:39:53.006 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.932+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj51]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj51[0] 2023-07-29T00:39:53.006 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.933+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj145]: success 2023-07-29T00:39:53.006 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.933+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj116]: success 2023-07-29T00:39:53.007 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.933+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj11]: success 2023-07-29T00:39:53.007 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.933+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj155]: success 2023-07-29T00:39:53.007 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.933+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj124]: success 2023-07-29T00:39:53.007 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.933+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj148]: success 2023-07-29T00:39:53.007 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.933+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj55 2023-07-29T00:39:53.007 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.933+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj55]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj55 op=0 op_state=1 2023-07-29T00:39:53.008 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.933+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj55]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj55[0] 2023-07-29T00:39:53.008 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.933+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj31]: success 2023-07-29T00:39:53.008 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.933+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj57 2023-07-29T00:39:53.008 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.933+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj57]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj57 op=0 op_state=1 2023-07-29T00:39:53.008 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.933+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj17]: success 2023-07-29T00:39:53.008 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.933+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj33]: success 2023-07-29T00:39:53.009 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.933+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj37]: success 2023-07-29T00:39:53.009 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.933+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj23]: success 2023-07-29T00:39:53.009 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.933+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj19]: success 2023-07-29T00:39:53.009 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.933+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj57]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj57[0] 2023-07-29T00:39:53.009 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.933+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj58 2023-07-29T00:39:53.009 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.933+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj58]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj58 op=0 op_state=1 2023-07-29T00:39:53.009 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.933+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj58]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj58[0] 2023-07-29T00:39:53.010 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.933+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj22]: success 2023-07-29T00:39:53.010 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.933+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj62 2023-07-29T00:39:53.010 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.933+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj62]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj62 op=0 op_state=1 2023-07-29T00:39:53.010 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.933+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj185]: success 2023-07-29T00:39:53.010 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.933+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj38]: success 2023-07-29T00:39:53.010 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.933+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj44]: success 2023-07-29T00:39:53.010 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.934+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj62]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj62[0] 2023-07-29T00:39:53.011 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.934+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj68 2023-07-29T00:39:53.011 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.934+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj68]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj68 op=0 op_state=1 2023-07-29T00:39:53.011 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.934+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj29]: success 2023-07-29T00:39:53.011 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.934+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj68]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj68[0] 2023-07-29T00:39:53.011 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.934+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj115]: success 2023-07-29T00:39:53.011 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.934+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: updating marker oid=bucket.full-sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24 marker=obj33 2023-07-29T00:39:53.012 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.934+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj69 2023-07-29T00:39:53.012 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.934+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj69]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj69 op=0 op_state=1 2023-07-29T00:39:53.012 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.934+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj69]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj69[0] 2023-07-29T00:39:53.012 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.934+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj74 2023-07-29T00:39:53.012 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.934+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj74]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj74 op=0 op_state=1 2023-07-29T00:39:53.012 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.934+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj74]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj74[0] 2023-07-29T00:39:53.013 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.934+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj76 2023-07-29T00:39:53.013 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.934+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj76]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj76 op=0 op_state=1 2023-07-29T00:39:53.013 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.934+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj76]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj76[0] 2023-07-29T00:39:53.013 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.934+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj85 2023-07-29T00:39:53.013 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.934+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj85]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj85 op=0 op_state=1 2023-07-29T00:39:53.013 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.934+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj85]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj85[0] 2023-07-29T00:39:53.013 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.934+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj46]: success 2023-07-29T00:39:53.014 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.934+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj87 2023-07-29T00:39:53.014 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.934+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj87]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj87 op=0 op_state=1 2023-07-29T00:39:53.014 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.934+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj87]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj87[0] 2023-07-29T00:39:53.014 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.934+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj92 2023-07-29T00:39:53.014 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.934+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj92]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj92 op=0 op_state=1 2023-07-29T00:39:53.014 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.934+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj92]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj92[0] 2023-07-29T00:39:53.015 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.934+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj93 2023-07-29T00:39:53.015 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.934+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj93]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj93 op=0 op_state=1 2023-07-29T00:39:53.015 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.934+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj93]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj93[0] 2023-07-29T00:39:53.015 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.934+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: [full sync] syncing object: hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0/obj95 2023-07-29T00:39:53.015 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.934+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj95]: bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b=bucket sync single entry (source_zone=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433) b= log_entry=obj95 op=0 op_state=1 2023-07-29T00:39:53.015 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.935+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj95]: bucket sync: sync obj: 3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433/:hiiqos-59[3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24])/obj95[0] 2023-07-29T00:39:53.015 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.936+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj34]: success 2023-07-29T00:39:53.016 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.936+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj51]: success 2023-07-29T00:39:53.016 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.936+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj58]: success 2023-07-29T00:39:53.016 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.937+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj55]: success 2023-07-29T00:39:53.016 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.938+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj62]: success 2023-07-29T00:39:53.017 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.938+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj92]: success 2023-07-29T00:39:53.017 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.938+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj76]: success 2023-07-29T00:39:53.017 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.938+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj69]: success 2023-07-29T00:39:53.017 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.938+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj95]: success 2023-07-29T00:39:53.017 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.938+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj57]: success 2023-07-29T00:39:53.017 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.938+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: updating marker oid=bucket.full-sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24 marker=obj62 2023-07-29T00:39:53.017 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.939+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj74]: success 2023-07-29T00:39:53.018 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.939+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj85]: success 2023-07-29T00:39:53.018 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.939+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj68]: success 2023-07-29T00:39:53.018 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.939+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj93]: success 2023-07-29T00:39:53.018 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.939+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:entry[obj87]: success 2023-07-29T00:39:53.018 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.939+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: updating marker oid=bucket.full-sync-status.3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24 marker=obj95 2023-07-29T00:39:53.018 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.941+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: set bucket state=incremental 2023-07-29T00:39:53.023 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.942+0000 7f7342926880 5 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:full_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: bucket status objv={r=_6P-hrNbwWao55B6ozsiWY3u:154,w=:0} 2023-07-29T00:39:53.023 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.943+0000 7f7342926880 10 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: WARNING: requested sync of past generation 0 < 6, returning success 2023-07-29T00:39:53.023 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.945+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:39:53.023 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.945+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:39:53.023 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.945+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:39:53.024 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.945+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:39:53.024 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.945+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:39:53.024 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.945+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: sync status for source bucket shard: 2 2023-07-29T00:39:53.024 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.945+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]:inc_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:0]: listing bilog for incremental sync; position= 2023-07-29T00:39:53.024 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.945+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:1]: sync status for source bucket shard: 2 2023-07-29T00:39:53.024 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.946+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:1]:inc_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:1]: listing bilog for incremental sync; position= 2023-07-29T00:39:53.025 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.946+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:39:53.025 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.946+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:4]: sync status for source bucket shard: 2 2023-07-29T00:39:53.025 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.946+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:4]:inc_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:4]: listing bilog for incremental sync; position= 2023-07-29T00:39:53.025 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.946+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:39:53.025 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.946+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:5]: sync status for source bucket shard: 2 2023-07-29T00:39:53.025 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.946+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:5]:inc_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:5]: listing bilog for incremental sync; position= 2023-07-29T00:39:53.026 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.946+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:6]: sync status for source bucket shard: 2 2023-07-29T00:39:53.026 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.946+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:6]:inc_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:6]: listing bilog for incremental sync; position=00000000002.101.6 2023-07-29T00:39:53.026 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.947+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:39:53.026 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.947+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:39:53.026 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.947+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:39:53.026 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.947+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:3]: sync status for source bucket shard: 2 2023-07-29T00:39:53.026 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.947+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:3]:inc_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:3]: listing bilog for incremental sync; position= 2023-07-29T00:39:53.027 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.948+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:39:53.027 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.948+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:39:53.027 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.948+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:39:53.027 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.948+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:39:53.027 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.948+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:8]: sync status for source bucket shard: 2 2023-07-29T00:39:53.027 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.948+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:7]: sync status for source bucket shard: 2 2023-07-29T00:39:53.027 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.948+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:10]: sync status for source bucket shard: 2 2023-07-29T00:39:53.028 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.948+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:8]:inc_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:8]: listing bilog for incremental sync; position= 2023-07-29T00:39:53.028 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.948+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:7]:inc_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:7]: listing bilog for incremental sync; position= 2023-07-29T00:39:53.028 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.948+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:10]:inc_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:10]: listing bilog for incremental sync; position= 2023-07-29T00:39:53.028 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.949+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:39:53.028 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.949+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:39:53.028 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.949+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:39:53.029 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.949+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:39:53.029 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.949+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:9]: sync status for source bucket shard: 2 2023-07-29T00:39:53.029 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.949+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:12]: sync status for source bucket shard: 2 2023-07-29T00:39:53.029 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.949+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:11]: sync status for source bucket shard: 2 2023-07-29T00:39:53.029 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.949+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:39:53.029 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.949+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:9]:inc_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:9]: listing bilog for incremental sync; position= 2023-07-29T00:39:53.029 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.949+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:12]:inc_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:12]: listing bilog for incremental sync; position=00000000002.70.6 2023-07-29T00:39:53.030 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.949+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:11]:inc_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:11]: listing bilog for incremental sync; position= 2023-07-29T00:39:53.030 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.950+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:39:53.030 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.950+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:39:53.030 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.950+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:39:53.030 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.950+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:17]: sync status for source bucket shard: 2 2023-07-29T00:39:53.030 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.950+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:16]: sync status for source bucket shard: 2 2023-07-29T00:39:53.031 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.950+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:14]: sync status for source bucket shard: 2 2023-07-29T00:39:53.031 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.950+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:15]: sync status for source bucket shard: 2 2023-07-29T00:39:53.031 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.950+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:17]:inc_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:17]: listing bilog for incremental sync; position=00000000002.62.6 2023-07-29T00:39:53.031 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.950+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:16]:inc_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:16]: listing bilog for incremental sync; position= 2023-07-29T00:39:53.031 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.950+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:14]:inc_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:14]: listing bilog for incremental sync; position= 2023-07-29T00:39:53.031 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.950+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:15]:inc_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:15]: listing bilog for incremental sync; position= 2023-07-29T00:39:53.031 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.951+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:39:53.032 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.951+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:39:53.032 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.951+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:39:53.032 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.952+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:39:53.032 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.952+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:39:53.032 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.952+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:20]: sync status for source bucket shard: 2 2023-07-29T00:39:53.032 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.952+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:19]: sync status for source bucket shard: 2 2023-07-29T00:39:53.032 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.952+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:18]: sync status for source bucket shard: 2 2023-07-29T00:39:53.033 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.952+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:21]: sync status for source bucket shard: 2 2023-07-29T00:39:53.033 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.952+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:20]:inc_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:20]: listing bilog for incremental sync; position= 2023-07-29T00:39:53.033 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.952+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:19]:inc_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:19]: listing bilog for incremental sync; position= 2023-07-29T00:39:53.033 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.952+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:18]:inc_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:18]: listing bilog for incremental sync; position= 2023-07-29T00:39:53.033 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.952+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:21]:inc_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:21]: listing bilog for incremental sync; position= 2023-07-29T00:39:53.033 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.953+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:39:53.034 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.953+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:22]: sync status for source bucket shard: 2 2023-07-29T00:39:53.034 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.953+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:22]:inc_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:22]: listing bilog for incremental sync; position= 2023-07-29T00:39:53.034 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.953+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:39:53.034 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.953+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:39:53.034 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.953+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:39:53.034 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.953+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:31]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:39:53.034 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.953+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:24]: sync status for source bucket shard: 2 2023-07-29T00:39:53.035 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.953+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:25]: sync status for source bucket shard: 2 2023-07-29T00:39:53.035 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.953+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:24]:inc_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:24]: listing bilog for incremental sync; position= 2023-07-29T00:39:53.035 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.953+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:25]:inc_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:25]: listing bilog for incremental sync; position= 2023-07-29T00:39:53.035 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.953+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:33]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:39:53.035 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.954+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:32]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:39:53.035 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.954+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:27]: sync status for source bucket shard: 2 2023-07-29T00:39:53.035 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.954+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:27]:inc_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:27]: listing bilog for incremental sync; position= 2023-07-29T00:39:53.036 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.954+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:31]: sync status for source bucket shard: 2 2023-07-29T00:39:53.036 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.954+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:31]:inc_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:31]: listing bilog for incremental sync; position= 2023-07-29T00:39:53.036 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.954+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:34]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:39:53.036 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.954+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:28]: sync status for source bucket shard: 2 2023-07-29T00:39:53.036 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.954+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:29]: sync status for source bucket shard: 2 2023-07-29T00:39:53.036 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.954+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:30]: sync status for source bucket shard: 2 2023-07-29T00:39:53.037 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.954+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:28]:inc_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:28]: listing bilog for incremental sync; position= 2023-07-29T00:39:53.037 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.954+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:29]:inc_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:29]: listing bilog for incremental sync; position= 2023-07-29T00:39:53.037 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.954+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:30]:inc_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:30]: listing bilog for incremental sync; position= 2023-07-29T00:39:53.037 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.955+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:35]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:39:53.037 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.955+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:32]: sync status for source bucket shard: 2 2023-07-29T00:39:53.037 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.955+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:2]: sync status for source bucket shard: 2 2023-07-29T00:39:53.037 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.955+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:32]:inc_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:32]: listing bilog for incremental sync; position=00000000001.88.6 2023-07-29T00:39:53.038 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.955+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:2]:inc_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:2]: listing bilog for incremental sync; position= 2023-07-29T00:39:53.038 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.956+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:36]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-07-29T00:39:53.038 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.956+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:13]: sync status for source bucket shard: 2 2023-07-29T00:39:53.038 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.956+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:26]: sync status for source bucket shard: 2 2023-07-29T00:39:53.038 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.956+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:23]: sync status for source bucket shard: 2 2023-07-29T00:39:53.038 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.956+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:34]: sync status for source bucket shard: 2 2023-07-29T00:39:53.038 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.956+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:35]: sync status for source bucket shard: 2 2023-07-29T00:39:53.039 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.956+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:33]: sync status for source bucket shard: 2 2023-07-29T00:39:53.039 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.956+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:13]:inc_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:13]: listing bilog for incremental sync; position= 2023-07-29T00:39:53.039 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.956+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:26]:inc_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:26]: listing bilog for incremental sync; position= 2023-07-29T00:39:53.039 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.957+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:23]:inc_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:23]: listing bilog for incremental sync; position= 2023-07-29T00:39:53.039 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.957+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:34]:inc_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:34]: listing bilog for incremental sync; position= 2023-07-29T00:39:53.039 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.957+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:35]:inc_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:35]: listing bilog for incremental sync; position= 2023-07-29T00:39:53.040 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.957+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:33]:inc_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:33]: listing bilog for incremental sync; position= 2023-07-29T00:39:53.040 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.958+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:36]: sync status for source bucket shard: 2 2023-07-29T00:39:53.040 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:52.958+0000 7f7342926880 20 RGW-SYNC:bucket[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24<-hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:36]:inc_sync[hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24:36]: listing bilog for incremental sync; position=00000000001.74.6 2023-07-29T00:39:53.085 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:39:53.086 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:39:53.086 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:39:53.174 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:53.175 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:53.206 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T00:39:53.206 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.206 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj11", 2023-07-29T00:39:53.207 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.207 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.207 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:39:53.208 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 3 2023-07-29T00:39:53.208 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.208 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.209 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.209 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.209 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.209 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:53.210 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:26:54.480753Z", 2023-07-29T00:39:53.210 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.210 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.211 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.211 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.211 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.212 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.212 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.212 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.213 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.213 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.14832257007510506436", 2023-07-29T00:39:53.213 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.214 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.214 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.214 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.214 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.215 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj111", 2023-07-29T00:39:53.215 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.215 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.215 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:39:53.215 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 2 2023-07-29T00:39:53.215 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.216 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.216 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.216 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.216 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.216 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:53.216 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:37:53.213506Z", 2023-07-29T00:39:53.217 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.217 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.217 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.217 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.217 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.218 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.218 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.218 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.218 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.218 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.7298560051294070076", 2023-07-29T00:39:53.218 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.219 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.219 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.219 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.219 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.219 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj115", 2023-07-29T00:39:53.219 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.220 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.220 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:39:53.220 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 2 2023-07-29T00:39:53.220 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.221 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.221 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.221 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.221 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.221 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 6, 2023-07-29T00:39:53.221 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:33:06.941054Z", 2023-07-29T00:39:53.221 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.222 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.222 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.222 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.222 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.222 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.222 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.223 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.223 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.223 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.12875750498704464913", 2023-07-29T00:39:53.223 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.223 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.223 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.223 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.224 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.224 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj116", 2023-07-29T00:39:53.224 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.224 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.224 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:39:53.224 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 10 2023-07-29T00:39:53.225 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.225 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.225 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.225 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.225 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.226 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:53.226 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:35:01.404138Z", 2023-07-29T00:39:53.226 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.226 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.226 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.226 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.227 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.227 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.227 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.227 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.227 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.227 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.8272829750339019172", 2023-07-29T00:39:53.228 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.228 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.228 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.228 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.228 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.228 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj124", 2023-07-29T00:39:53.228 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.229 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.229 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:39:53.229 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 2 2023-07-29T00:39:53.229 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.229 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.229 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.229 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.230 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.230 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:39:53.230 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:35:31.901495Z", 2023-07-29T00:39:53.230 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.230 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.230 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.230 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.231 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.231 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.231 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.231 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.231 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.232 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.2551816591193919500", 2023-07-29T00:39:53.232 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.232 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.232 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.232 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.232 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.232 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj145", 2023-07-29T00:39:53.233 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.233 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.233 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:39:53.233 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 6 2023-07-29T00:39:53.233 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.233 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.233 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.234 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.234 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.234 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:53.234 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:35:01.459486Z", 2023-07-29T00:39:53.234 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.234 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.234 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.235 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.235 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.235 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.235 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.235 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.235 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.235 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.5711428050720100775", 2023-07-29T00:39:53.236 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.236 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.236 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.236 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.236 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.237 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj148", 2023-07-29T00:39:53.237 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.237 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.237 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:39:53.237 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 1 2023-07-29T00:39:53.237 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.237 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.238 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.238 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.238 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.238 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:53.238 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:37:53.270196Z", 2023-07-29T00:39:53.238 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.238 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.239 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.239 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.239 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.239 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.239 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.239 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.239 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.240 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.4182403843262121459", 2023-07-29T00:39:53.240 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.240 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.240 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.240 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.240 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.240 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj155", 2023-07-29T00:39:53.241 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.241 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.241 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:39:53.241 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 11 2023-07-29T00:39:53.241 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.241 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.241 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.242 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.242 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.242 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:53.243 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:35:31.919584Z", 2023-07-29T00:39:53.243 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.243 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.243 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.244 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.244 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.244 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.244 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.244 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.244 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.245 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.6568146459053762088", 2023-07-29T00:39:53.245 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.245 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.245 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.245 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.245 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.246 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj17", 2023-07-29T00:39:53.246 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.246 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.246 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:39:53.246 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 20 2023-07-29T00:39:53.246 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.246 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.247 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.247 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.247 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.247 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 6, 2023-07-29T00:39:53.247 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:27:51.133766Z", 2023-07-29T00:39:53.247 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.247 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.248 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.248 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.248 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.248 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.248 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.248 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.249 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.249 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.522398537848267640", 2023-07-29T00:39:53.249 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.249 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.249 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.250 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.250 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.250 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj185", 2023-07-29T00:39:53.250 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.250 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.250 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:39:53.250 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 27 2023-07-29T00:39:53.251 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.251 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.251 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.251 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.251 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.251 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:53.252 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:37:53.291240Z", 2023-07-29T00:39:53.252 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.252 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.252 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.253 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.253 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.253 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.254 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.254 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.254 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.254 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.6256432204843257847", 2023-07-29T00:39:53.254 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.255 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.255 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.255 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.255 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.256 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj19", 2023-07-29T00:39:53.256 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.256 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.256 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:39:53.256 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 11 2023-07-29T00:39:53.256 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.257 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.257 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.257 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.257 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.257 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:39:53.257 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:29:58.641540Z", 2023-07-29T00:39:53.257 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.258 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.258 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.258 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.258 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.258 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.258 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.258 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.259 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.259 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.12188890004521546428", 2023-07-29T00:39:53.259 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.259 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.259 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.260 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.260 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.260 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj22", 2023-07-29T00:39:53.260 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.260 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.260 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:39:53.260 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 26 2023-07-29T00:39:53.261 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.261 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.261 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.261 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.261 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.261 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:39:53.261 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:26:54.504376Z", 2023-07-29T00:39:53.262 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.262 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.262 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.262 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.262 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.262 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.263 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.263 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.263 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.263 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.13758179519055660563", 2023-07-29T00:39:53.263 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.263 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.263 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.263 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.264 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.264 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj23", 2023-07-29T00:39:53.264 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.264 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.264 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:39:53.264 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 2 2023-07-29T00:39:53.265 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.265 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.265 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.265 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.265 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.265 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:53.265 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:33:06.784058Z", 2023-07-29T00:39:53.266 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.266 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.266 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.266 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.266 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.266 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.266 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.267 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.267 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.267 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.9269913819882926537", 2023-07-29T00:39:53.267 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.267 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.268 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.268 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.268 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.268 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj29", 2023-07-29T00:39:53.268 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.268 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.268 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:39:53.269 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 4 2023-07-29T00:39:53.269 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.269 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.269 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.269 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.269 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.269 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 6, 2023-07-29T00:39:53.270 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:35:01.307381Z", 2023-07-29T00:39:53.270 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.270 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.270 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.270 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.270 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.270 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.271 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.271 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.271 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.271 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.11332671532145893480", 2023-07-29T00:39:53.271 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.271 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.271 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.272 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.272 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.272 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj31", 2023-07-29T00:39:53.272 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.272 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.272 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:39:53.272 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 4 2023-07-29T00:39:53.273 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.273 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.273 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.273 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.273 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.273 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:53.273 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:35:31.771442Z", 2023-07-29T00:39:53.274 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.274 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.274 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.274 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.274 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.275 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.275 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.275 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.276 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.276 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.17507146512515813903", 2023-07-29T00:39:53.276 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.277 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.277 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.277 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.277 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.278 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj33", 2023-07-29T00:39:53.278 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.278 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.279 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:39:53.279 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 28 2023-07-29T00:39:53.279 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.280 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.280 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.280 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.281 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.281 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 6, 2023-07-29T00:39:53.281 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:26:54.567027Z", 2023-07-29T00:39:53.282 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.282 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.282 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.282 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.283 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.283 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.283 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.283 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.284 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.284 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.6837111927141591146", 2023-07-29T00:39:53.284 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.285 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.285 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.285 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.285 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.285 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj34", 2023-07-29T00:39:53.286 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.286 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.286 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:39:53.286 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 3 2023-07-29T00:39:53.286 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.286 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.286 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.287 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.287 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.287 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:39:53.287 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:27:51.159033Z", 2023-07-29T00:39:53.287 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.287 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.287 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.288 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.288 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.288 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.288 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.289 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.289 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.289 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.7685927806540265810", 2023-07-29T00:39:53.289 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.289 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.289 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.289 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.290 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.290 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj37", 2023-07-29T00:39:53.290 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.290 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.290 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:39:53.290 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 3 2023-07-29T00:39:53.290 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.291 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.291 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.291 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.291 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.291 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:39:53.291 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:37:53.137274Z", 2023-07-29T00:39:53.291 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.292 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.292 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.292 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.292 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.292 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.292 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.292 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.293 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.293 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.15737676585740292221", 2023-07-29T00:39:53.293 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.293 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.293 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.293 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.293 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.294 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj38", 2023-07-29T00:39:53.294 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.294 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.294 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:39:53.294 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 5 2023-07-29T00:39:53.294 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.295 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.295 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.295 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.295 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.295 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:39:53.295 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:29:58.663185Z", 2023-07-29T00:39:53.296 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.296 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.296 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.296 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.296 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.296 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.297 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.297 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.297 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.297 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.3215769567045764454", 2023-07-29T00:39:53.297 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.297 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.297 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.298 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.298 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.298 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj44", 2023-07-29T00:39:53.298 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.298 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.298 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:39:53.298 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 5 2023-07-29T00:39:53.299 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.299 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.299 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.299 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.299 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.299 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:39:53.299 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:26:54.593602Z", 2023-07-29T00:39:53.300 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.300 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.300 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.300 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.300 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.300 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.301 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.301 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.301 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.301 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.7711008481512402586", 2023-07-29T00:39:53.301 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.301 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.301 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.302 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.302 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.302 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj46", 2023-07-29T00:39:53.302 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.302 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.302 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:39:53.302 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 2 2023-07-29T00:39:53.303 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.303 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.303 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.303 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.303 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.303 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:39:53.303 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:33:06.801664Z", 2023-07-29T00:39:53.304 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.304 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.304 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.304 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.304 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.304 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.305 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.305 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.305 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.305 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.743245964017995746", 2023-07-29T00:39:53.305 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.305 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.305 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.306 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.306 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.306 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj51", 2023-07-29T00:39:53.306 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.306 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.306 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:39:53.306 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 8 2023-07-29T00:39:53.307 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.307 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.307 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.307 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.307 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.307 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:39:53.307 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:27:51.180793Z", 2023-07-29T00:39:53.308 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.308 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.308 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.308 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.308 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.308 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.309 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.309 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.309 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.309 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.7422264557873721673", 2023-07-29T00:39:53.309 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.309 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.309 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.310 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.310 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.310 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj55", 2023-07-29T00:39:53.310 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.310 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.310 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:39:53.310 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 4 2023-07-29T00:39:53.311 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.311 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.311 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.311 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.311 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.311 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:39:53.312 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:26:54.656875Z", 2023-07-29T00:39:53.312 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.312 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.312 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.313 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.313 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.313 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.313 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.313 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.314 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.314 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.8171173214296575173", 2023-07-29T00:39:53.314 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.314 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.314 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.315 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.315 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.315 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj57", 2023-07-29T00:39:53.315 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.315 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.315 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:39:53.315 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 2 2023-07-29T00:39:53.316 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.316 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.316 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.316 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.316 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.316 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:39:53.316 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:29:58.720061Z", 2023-07-29T00:39:53.317 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.317 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.317 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.317 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.317 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.317 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.318 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.318 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.318 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.318 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.15966729634763942490", 2023-07-29T00:39:53.318 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.318 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.318 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.319 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.319 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.319 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj58", 2023-07-29T00:39:53.319 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.319 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.319 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:39:53.319 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 10 2023-07-29T00:39:53.320 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.320 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.320 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.320 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.320 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.321 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:53.321 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:35:01.328082Z", 2023-07-29T00:39:53.321 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.321 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.321 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.321 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.321 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.322 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.322 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.322 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.322 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.322 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.16632985783376759862", 2023-07-29T00:39:53.322 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.322 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.323 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.323 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.323 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.323 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj62", 2023-07-29T00:39:53.323 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.323 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.323 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:39:53.324 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 9 2023-07-29T00:39:53.324 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.324 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.324 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.325 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.325 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.325 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:53.325 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:35:31.826974Z", 2023-07-29T00:39:53.325 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.326 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.326 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.326 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.326 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.326 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.327 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.327 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.327 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.327 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.2520529618437238462", 2023-07-29T00:39:53.327 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.327 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.327 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.328 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.328 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.328 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj68", 2023-07-29T00:39:53.328 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.328 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.328 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:39:53.329 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 8 2023-07-29T00:39:53.329 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.329 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.329 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.329 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.329 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.329 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:39:53.330 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:27:51.249779Z", 2023-07-29T00:39:53.330 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.330 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.330 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.330 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.330 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.330 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.331 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.331 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.331 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.331 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.14623704719372606825", 2023-07-29T00:39:53.331 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.331 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.331 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.332 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.332 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.332 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj69", 2023-07-29T00:39:53.332 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.332 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.332 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:39:53.333 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 3 2023-07-29T00:39:53.333 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.333 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.333 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.333 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.333 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.333 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 6, 2023-07-29T00:39:53.334 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:33:06.857762Z", 2023-07-29T00:39:53.334 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.334 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.334 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.334 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.334 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.334 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.335 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.335 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.335 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.335 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.15913985204702152047", 2023-07-29T00:39:53.335 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.335 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.335 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.336 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.336 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.336 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj74", 2023-07-29T00:39:53.336 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.336 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.336 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:39:53.337 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 15 2023-07-29T00:39:53.337 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.337 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.337 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.337 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.337 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.337 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:39:53.338 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:37:53.195304Z", 2023-07-29T00:39:53.338 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.338 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.338 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.338 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.338 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.338 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.339 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.339 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.339 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.339 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.12267894682161863352", 2023-07-29T00:39:53.339 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.339 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.339 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.340 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.340 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.340 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj76", 2023-07-29T00:39:53.340 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.340 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.340 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:39:53.340 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 27 2023-07-29T00:39:53.341 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.341 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.341 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.342 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.342 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.342 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:53.342 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:29:58.736037Z", 2023-07-29T00:39:53.343 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.343 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.343 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.343 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.343 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.343 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.344 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.344 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.344 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.344 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.7092795770651948946", 2023-07-29T00:39:53.344 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.344 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.344 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.345 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.345 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.345 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj85", 2023-07-29T00:39:53.345 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.345 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.345 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:39:53.346 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 55 2023-07-29T00:39:53.346 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.347 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.347 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.347 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.347 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.347 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:39:53.347 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:27:51.275401Z", 2023-07-29T00:39:53.348 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.348 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.348 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.348 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.348 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.348 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.348 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.349 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.349 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.349 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.15456683764337138762", 2023-07-29T00:39:53.349 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.349 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.349 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.349 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.350 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.350 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj87", 2023-07-29T00:39:53.350 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.350 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.350 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:39:53.350 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 27 2023-07-29T00:39:53.350 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.351 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.351 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.351 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.351 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.351 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 6, 2023-07-29T00:39:53.351 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:35:01.384870Z", 2023-07-29T00:39:53.351 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.352 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.352 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.352 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.352 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.352 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.352 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.353 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.353 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.353 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.8451912843248802673", 2023-07-29T00:39:53.353 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.353 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.353 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.353 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.354 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.354 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj92", 2023-07-29T00:39:53.354 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.354 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.354 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:39:53.354 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 3 2023-07-29T00:39:53.354 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.355 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.355 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.355 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.355 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.355 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:39:53.355 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:33:06.875188Z", 2023-07-29T00:39:53.356 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.356 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.356 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.356 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.356 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.356 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.357 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.357 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.357 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.357 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.11130366415168660080", 2023-07-29T00:39:53.357 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.357 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.357 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.358 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.358 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.358 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj93", 2023-07-29T00:39:53.358 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.358 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.358 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:39:53.358 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 3 2023-07-29T00:39:53.359 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.359 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.359 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.359 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.359 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.359 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:53.359 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:35:31.844362Z", 2023-07-29T00:39:53.359 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.360 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.360 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.360 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.360 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.360 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.360 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.361 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.361 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.361 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.11128177405679269595", 2023-07-29T00:39:53.361 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.361 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.361 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.361 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.362 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.362 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj95", 2023-07-29T00:39:53.362 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.362 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.362 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 9, 2023-07-29T00:39:53.362 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 2 2023-07-29T00:39:53.362 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.363 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.363 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.363 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.363 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.363 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:53.363 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:29:58.792344Z", 2023-07-29T00:39:53.363 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.364 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.364 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.364 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.364 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.364 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.364 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.364 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.365 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.365 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.2029434651376669903", 2023-07-29T00:39:53.365 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.365 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.365 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.365 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:39:53.365 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T00:39:53.366 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:39:53.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:39:53.367 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-59 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:39:53.415 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:53.416 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:53.447 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T00:39:53.447 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.447 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj11", 2023-07-29T00:39:53.447 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.448 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.448 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:53.448 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 5 2023-07-29T00:39:53.448 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.448 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.448 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.449 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.449 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.449 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:39:53.449 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:26:54.480753Z", 2023-07-29T00:39:53.449 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.449 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.450 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.450 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.450 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.450 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.450 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.450 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.450 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.451 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_GKIvl1U9UHmY9N1-i9upeOYzPb1s108", 2023-07-29T00:39:53.451 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.451 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.451 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.451 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.451 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.451 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj111", 2023-07-29T00:39:53.452 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.452 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.452 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:53.452 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 4 2023-07-29T00:39:53.452 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.453 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.453 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.453 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.453 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.453 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:39:53.453 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:37:53.213506Z", 2023-07-29T00:39:53.454 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.454 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.454 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.454 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.454 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.454 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.454 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.455 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.455 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.455 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_KtO6tNpVxyYB8EWCmDxuRh0aMI3hiUe", 2023-07-29T00:39:53.455 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.455 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.455 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.455 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.456 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.456 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj115", 2023-07-29T00:39:53.456 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.456 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.456 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:53.456 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 4 2023-07-29T00:39:53.457 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.457 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.457 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.457 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.457 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.457 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 6, 2023-07-29T00:39:53.457 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:33:06.941054Z", 2023-07-29T00:39:53.458 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.458 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.458 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.458 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.458 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.458 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.458 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.459 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.459 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.459 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_dgukq_dQScCHzbAjoaJB2YNkXCPUG5j", 2023-07-29T00:39:53.459 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.459 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.460 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.460 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.460 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.460 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj116", 2023-07-29T00:39:53.461 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.461 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.461 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:53.462 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 6 2023-07-29T00:39:53.462 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.462 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.462 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.463 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.463 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.463 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 6, 2023-07-29T00:39:53.464 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:35:01.404138Z", 2023-07-29T00:39:53.464 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.464 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.465 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.465 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.465 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.466 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.466 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.466 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.467 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.467 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_O-vmj4-fHpvSF4LUsQ4ImYs04TadG6N", 2023-07-29T00:39:53.467 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.467 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.468 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.468 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.468 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.469 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj124", 2023-07-29T00:39:53.469 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.469 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.469 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:53.469 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 4 2023-07-29T00:39:53.470 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.470 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.470 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.470 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.470 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.470 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:39:53.470 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:35:31.901495Z", 2023-07-29T00:39:53.471 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.471 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.471 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.471 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.471 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.471 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.471 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.472 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.472 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.472 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_qcbuB313IokL--O0BIzszELFQUELLKJ", 2023-07-29T00:39:53.472 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.472 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.472 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.472 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.473 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.473 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj145", 2023-07-29T00:39:53.473 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.473 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.473 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:53.473 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 10 2023-07-29T00:39:53.473 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.474 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.474 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.474 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.474 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.474 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:53.474 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:35:01.459486Z", 2023-07-29T00:39:53.475 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.475 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.475 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.475 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.475 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.475 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.476 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.476 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.476 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.476 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_osfWUihZQTLeud3a4LEUm9FdyEnqwLf", 2023-07-29T00:39:53.477 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.477 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.477 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.477 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.477 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.477 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj148", 2023-07-29T00:39:53.478 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.478 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.478 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:53.478 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 3 2023-07-29T00:39:53.478 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.478 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.478 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.479 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.479 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.479 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:53.479 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:37:53.270196Z", 2023-07-29T00:39:53.479 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.479 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.480 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.480 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.480 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.480 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.480 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.480 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.480 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.481 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_wwBkbRWe3D8q9WqCFpBdC_rCazcfSN4", 2023-07-29T00:39:53.481 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.481 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.481 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.481 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.481 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.481 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj155", 2023-07-29T00:39:53.482 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.482 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.482 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:53.482 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 9 2023-07-29T00:39:53.482 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.482 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.483 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.483 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.483 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.483 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 6, 2023-07-29T00:39:53.483 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:35:31.919584Z", 2023-07-29T00:39:53.483 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.484 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.484 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.484 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.484 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.484 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.485 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.485 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.485 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.485 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_08rLHjp-UH27WnCoxFzQp-VBC5OFVWD", 2023-07-29T00:39:53.485 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.485 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.485 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.486 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.486 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.486 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj17", 2023-07-29T00:39:53.486 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.486 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.486 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:53.486 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 10 2023-07-29T00:39:53.487 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.487 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.487 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.487 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.487 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.487 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 6, 2023-07-29T00:39:53.487 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:27:51.133766Z", 2023-07-29T00:39:53.488 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.488 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.488 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.488 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.488 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.488 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.489 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.489 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.489 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.489 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_9tWrV53BLP6eHuwZs3Bo2TTG-4wk9kM", 2023-07-29T00:39:53.489 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.489 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.489 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.490 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.490 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.490 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj185", 2023-07-29T00:39:53.490 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.490 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.490 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:53.490 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 14 2023-07-29T00:39:53.491 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.491 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.491 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.491 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.491 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.491 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:53.491 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:37:53.291240Z", 2023-07-29T00:39:53.492 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.492 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.492 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.492 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.492 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.492 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.493 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.493 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.493 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.493 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_Wng18qKdsMLzdLOkR7VtU1Iqxa1X3Ew", 2023-07-29T00:39:53.493 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.493 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.494 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.494 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.494 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.494 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj19", 2023-07-29T00:39:53.494 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.495 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.495 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:53.495 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 7 2023-07-29T00:39:53.495 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.496 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.496 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.496 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.496 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.497 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:53.497 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:29:58.641540Z", 2023-07-29T00:39:53.497 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.497 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.498 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.498 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.498 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.498 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.498 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.499 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.499 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.499 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_XkOl12_OulKOjRqHTnI80pgKReyJ23J", 2023-07-29T00:39:53.499 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.499 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.500 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.500 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.500 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.500 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj22", 2023-07-29T00:39:53.500 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.500 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.500 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:53.501 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 15 2023-07-29T00:39:53.501 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.501 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.501 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.501 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.501 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.502 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:53.502 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:26:54.504376Z", 2023-07-29T00:39:53.502 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.502 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.502 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.502 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.503 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.503 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.503 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.503 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.503 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.503 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_TSTdRu8zh_u_codotXRXHpOYn-fKEUz", 2023-07-29T00:39:53.503 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.504 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.504 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.504 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.504 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.504 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj23", 2023-07-29T00:39:53.504 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.504 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.505 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:53.505 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 4 2023-07-29T00:39:53.505 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.505 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.505 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.505 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.505 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.506 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:39:53.506 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:33:06.784058Z", 2023-07-29T00:39:53.506 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.506 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.506 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.507 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.507 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.507 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.508 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.508 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.508 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.508 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_2JI6xTMHJyRRE6rZUuvQrevNcUy4CZL", 2023-07-29T00:39:53.509 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.509 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.509 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.509 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.510 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.510 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj29", 2023-07-29T00:39:53.510 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.510 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.510 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:53.511 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 8 2023-07-29T00:39:53.511 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.511 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.512 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.512 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.512 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.513 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:39:53.513 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:35:01.307381Z", 2023-07-29T00:39:53.513 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.513 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.514 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.514 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.514 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.515 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.515 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.515 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.515 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.516 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_o0wOGRuemgX7rbRosgnzyUrls72zYS1", 2023-07-29T00:39:53.516 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.516 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.517 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.517 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.517 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.517 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj31", 2023-07-29T00:39:53.517 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.518 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.518 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:53.518 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 6 2023-07-29T00:39:53.518 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.518 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.518 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.519 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.519 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.519 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:39:53.519 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:35:31.771442Z", 2023-07-29T00:39:53.519 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.520 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.520 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.520 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.520 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.520 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.521 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.521 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.521 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.521 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_uo_U216-H5fe_3oXcIthLM5Js28wqlk", 2023-07-29T00:39:53.521 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.522 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.522 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.522 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.522 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.522 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj33", 2023-07-29T00:39:53.522 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.523 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.523 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:53.523 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 9 2023-07-29T00:39:53.523 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.523 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.523 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.524 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.524 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.524 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:39:53.524 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:26:54.567027Z", 2023-07-29T00:39:53.524 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.524 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.525 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.525 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.525 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.525 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.525 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.526 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.526 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.526 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_J9pBGg_W2M2kV1w8-mHjAEabBe787zO", 2023-07-29T00:39:53.526 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.526 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.526 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.527 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.527 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.527 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj34", 2023-07-29T00:39:53.527 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.527 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.527 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:53.528 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 5 2023-07-29T00:39:53.528 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.528 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.528 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.528 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.528 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.528 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:39:53.529 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:27:51.159033Z", 2023-07-29T00:39:53.529 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.529 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.529 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.529 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.529 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.530 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.530 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.530 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.530 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.530 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_yQE-_Jhs_2UmsFsyI3gclepszAkmEmF", 2023-07-29T00:39:53.530 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.531 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.531 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.531 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.531 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.531 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj37", 2023-07-29T00:39:53.531 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.531 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.532 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:53.532 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 5 2023-07-29T00:39:53.532 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.532 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.532 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.532 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.533 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.533 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:53.533 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:37:53.137274Z", 2023-07-29T00:39:53.533 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.533 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.533 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.534 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.534 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.534 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.534 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.534 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.534 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.534 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_9012MjmmAQpL9gd4PRLtxJLIkrDfrdF", 2023-07-29T00:39:53.535 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.535 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.535 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.535 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.535 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.535 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj38", 2023-07-29T00:39:53.535 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.536 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.536 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:53.536 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 7 2023-07-29T00:39:53.536 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.536 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.536 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.537 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.537 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.537 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 6, 2023-07-29T00:39:53.537 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:29:58.663185Z", 2023-07-29T00:39:53.537 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.537 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.537 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.538 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.538 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.538 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.538 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.538 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.538 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.538 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_TWNGo0cQp26aYrv-Af2c2H5AI49OICR", 2023-07-29T00:39:53.539 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.539 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.539 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.539 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.539 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.539 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj44", 2023-07-29T00:39:53.539 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.540 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.540 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:53.540 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 9 2023-07-29T00:39:53.540 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.540 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.541 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.541 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.541 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.541 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:53.541 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:26:54.593602Z", 2023-07-29T00:39:53.541 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.541 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.542 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.542 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.542 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.542 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.542 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.542 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.543 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.543 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_PmEiO5BUOgj4F7-pKGsfWQdqbW8QvFY", 2023-07-29T00:39:53.543 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.543 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.544 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.544 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.544 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.544 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj46", 2023-07-29T00:39:53.544 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.544 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.545 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:53.545 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 4 2023-07-29T00:39:53.545 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.545 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.545 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.545 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.546 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.546 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:53.546 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:33:06.801664Z", 2023-07-29T00:39:53.546 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.547 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.547 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.547 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.547 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.548 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.548 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.548 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.549 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.549 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_zBMiWvwTy1-eowmmxfWviLaUPl9T9bC", 2023-07-29T00:39:53.549 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.550 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.550 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.550 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.551 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.551 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj51", 2023-07-29T00:39:53.551 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.552 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.552 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:53.552 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 10 2023-07-29T00:39:53.553 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.553 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.553 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.554 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.554 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.554 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:53.554 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:27:51.180793Z", 2023-07-29T00:39:53.555 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.555 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.555 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.555 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.556 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.556 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.556 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.557 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.557 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.557 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_w6jXDzhk6ysq5SBNp_oZTLYX1NjYyUx", 2023-07-29T00:39:53.557 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.557 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.558 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.558 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.558 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.558 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj55", 2023-07-29T00:39:53.558 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.559 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.559 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:53.559 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 6 2023-07-29T00:39:53.559 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.559 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.559 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.560 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.560 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.560 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:53.560 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:26:54.656875Z", 2023-07-29T00:39:53.560 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.560 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.561 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.561 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.561 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.561 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.561 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.561 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.561 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.562 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_1bo_zqpj14FwVyPkHPOIs-g6B6jspCj", 2023-07-29T00:39:53.562 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.562 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.562 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.562 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.562 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.562 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj57", 2023-07-29T00:39:53.563 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.563 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.563 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:53.563 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 4 2023-07-29T00:39:53.564 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.564 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.564 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.564 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.565 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.565 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 6, 2023-07-29T00:39:53.565 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:29:58.720061Z", 2023-07-29T00:39:53.566 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.566 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.566 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.566 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.567 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.567 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.567 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.567 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.568 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.568 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_Xsgh3gciSQj1f1ATBb-vlO_oJdH5PiB", 2023-07-29T00:39:53.568 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.568 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.568 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.569 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.569 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.569 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj58", 2023-07-29T00:39:53.569 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.569 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.569 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:53.570 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 8 2023-07-29T00:39:53.570 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.570 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.570 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.570 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.570 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.570 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:39:53.571 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:35:01.328082Z", 2023-07-29T00:39:53.571 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.571 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.571 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.571 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.571 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.571 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.572 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.572 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.572 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.572 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_21EfC00vX-DZChRwT2lhvYVCh0N-VoO", 2023-07-29T00:39:53.572 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.572 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.573 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.573 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.573 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.573 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj62", 2023-07-29T00:39:53.573 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.573 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.573 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:53.574 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 5 2023-07-29T00:39:53.574 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.574 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.574 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.574 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.574 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.575 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:39:53.575 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:35:31.826974Z", 2023-07-29T00:39:53.575 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.575 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.575 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.575 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.575 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.576 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.576 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.576 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.576 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.576 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_zrRQGue8AJhYfU-QeQTKD9BttbSk6FD", 2023-07-29T00:39:53.576 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.577 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.577 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.577 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.577 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.577 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj68", 2023-07-29T00:39:53.577 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.577 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.578 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:53.578 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 11 2023-07-29T00:39:53.578 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.578 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.578 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.578 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.578 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.579 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:53.579 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:27:51.249779Z", 2023-07-29T00:39:53.579 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.579 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.579 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.579 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.579 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.580 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.580 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.580 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.580 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.580 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_lLooJthol2DfZp8ue3QrvPRq2yXVun3", 2023-07-29T00:39:53.580 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.581 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.581 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.581 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.581 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.581 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj69", 2023-07-29T00:39:53.581 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.581 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.582 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:53.582 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 15 2023-07-29T00:39:53.582 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.582 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.582 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.582 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.582 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.583 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:53.583 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:33:06.857762Z", 2023-07-29T00:39:53.583 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.583 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.583 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.583 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.583 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.584 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.584 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.584 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.584 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.584 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_0poPsl5ViuTBnUxqpPUsjae5TEXcptZ", 2023-07-29T00:39:53.584 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.585 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.585 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.585 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.585 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.585 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj74", 2023-07-29T00:39:53.585 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.585 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.586 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:53.586 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 11 2023-07-29T00:39:53.586 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.586 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.586 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.586 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.586 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.586 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:39:53.587 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:37:53.195304Z", 2023-07-29T00:39:53.587 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.587 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.587 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.587 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.587 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.588 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.588 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.588 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.588 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.588 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_USnUyksa_8iAX8XJtRR9ohSwgx4wHmX", 2023-07-29T00:39:53.588 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.589 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.589 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.589 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.589 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.589 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj76", 2023-07-29T00:39:53.590 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.590 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.590 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:53.591 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 24 2023-07-29T00:39:53.591 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.591 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.591 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.591 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.591 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.592 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:39:53.592 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:29:58.736037Z", 2023-07-29T00:39:53.592 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.592 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.592 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.592 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.593 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.593 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.593 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.593 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.593 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.593 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_nqVpsX6xleIidxPvMS8xfA8v7JgG2xC", 2023-07-29T00:39:53.593 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.594 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.594 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.594 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.594 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.594 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj85", 2023-07-29T00:39:53.594 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.594 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.595 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:53.595 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 41 2023-07-29T00:39:53.595 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.595 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.595 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.595 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.595 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.596 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 16, 2023-07-29T00:39:53.596 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:27:51.275401Z", 2023-07-29T00:39:53.596 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.596 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.596 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.596 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.597 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.597 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.597 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.597 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.597 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.597 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_KO0opj_SoRTBXOFWf6hZcMq4As_FNgR", 2023-07-29T00:39:53.597 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.598 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.598 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.598 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.598 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.598 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj87", 2023-07-29T00:39:53.598 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.598 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.599 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:53.599 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 16 2023-07-29T00:39:53.599 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.599 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.600 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.600 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.601 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.601 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:39:53.601 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:35:01.384870Z", 2023-07-29T00:39:53.601 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.601 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.601 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.601 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.602 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.602 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.602 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.603 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.603 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.603 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_YL0K1rQ1YsPFbdNeuJYeJ8XM4iX6KRU", 2023-07-29T00:39:53.603 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.603 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.603 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.603 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.604 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.604 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj92", 2023-07-29T00:39:53.604 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.604 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.604 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:53.604 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 5 2023-07-29T00:39:53.605 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.605 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.605 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.605 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.605 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.605 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 3, 2023-07-29T00:39:53.605 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:33:06.875188Z", 2023-07-29T00:39:53.606 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.606 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.606 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.606 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.606 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.606 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.606 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.607 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.607 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.607 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_r7fQVxyQA40bWde_IIrU7iyg-zeRTaI", 2023-07-29T00:39:53.607 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.607 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.607 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.607 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.608 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.608 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj93", 2023-07-29T00:39:53.608 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.608 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.608 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:53.609 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 5 2023-07-29T00:39:53.609 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.609 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.609 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.609 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.609 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.609 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 5, 2023-07-29T00:39:53.610 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:35:31.844362Z", 2023-07-29T00:39:53.610 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.610 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.610 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.610 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.610 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.610 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.611 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.611 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.611 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.611 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_Te5q9QNgoqRVFC0MzeCF-8-b9knHUo8", 2023-07-29T00:39:53.611 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.611 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.611 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.612 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.612 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:39:53.612 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "obj95", 2023-07-29T00:39:53.612 INFO:teuthology.orchestra.run.smithi101.stdout: "instance": "", 2023-07-29T00:39:53.612 INFO:teuthology.orchestra.run.smithi101.stdout: "ver": { 2023-07-29T00:39:53.612 INFO:teuthology.orchestra.run.smithi101.stdout: "pool": 13, 2023-07-29T00:39:53.612 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 4 2023-07-29T00:39:53.613 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.613 INFO:teuthology.orchestra.run.smithi101.stdout: "locator": "", 2023-07-29T00:39:53.613 INFO:teuthology.orchestra.run.smithi101.stdout: "exists": true, 2023-07-29T00:39:53.613 INFO:teuthology.orchestra.run.smithi101.stdout: "meta": { 2023-07-29T00:39:53.613 INFO:teuthology.orchestra.run.smithi101.stdout: "category": 1, 2023-07-29T00:39:53.613 INFO:teuthology.orchestra.run.smithi101.stdout: "size": 6, 2023-07-29T00:39:53.613 INFO:teuthology.orchestra.run.smithi101.stdout: "mtime": "2023-07-29T00:29:58.792344Z", 2023-07-29T00:39:53.614 INFO:teuthology.orchestra.run.smithi101.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-07-29T00:39:53.614 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_class": "", 2023-07-29T00:39:53.614 INFO:teuthology.orchestra.run.smithi101.stdout: "owner": "rgw-multisite-test-user", 2023-07-29T00:39:53.614 INFO:teuthology.orchestra.run.smithi101.stdout: "owner_display_name": "Multisite Test User", 2023-07-29T00:39:53.614 INFO:teuthology.orchestra.run.smithi101.stdout: "content_type": "application/octet-stream", 2023-07-29T00:39:53.614 INFO:teuthology.orchestra.run.smithi101.stdout: "accounted_size": 3, 2023-07-29T00:39:53.614 INFO:teuthology.orchestra.run.smithi101.stdout: "user_data": "", 2023-07-29T00:39:53.615 INFO:teuthology.orchestra.run.smithi101.stdout: "appendable": false 2023-07-29T00:39:53.615 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:39:53.615 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "_9VT62E7RI-AfopRfbQxq42ZyUnC7YHi", 2023-07-29T00:39:53.615 INFO:teuthology.orchestra.run.smithi101.stdout: "flags": 0, 2023-07-29T00:39:53.615 INFO:teuthology.orchestra.run.smithi101.stdout: "pending_map": [], 2023-07-29T00:39:53.615 INFO:teuthology.orchestra.run.smithi101.stdout: "versioned_epoch": 0 2023-07-29T00:39:53.615 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:39:53.616 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T00:39:53.616 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-07-29T00:39:53.617 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-07-29T00:39:53.620 INFO:tasks.rgw.c1.client.1:Started 2023-07-29T00:39:53.620 DEBUG:teuthology.orchestra.run.smithi101:> curl http://smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:53.638 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-07-29T00:39:53.638 INFO:teuthology.orchestra.run.smithi101.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-07-29T00:39:53.639 INFO:teuthology.orchestra.run.smithi101.stderr: Dload Upload Total Spent Left Speed 2023-07-29T00:39:53.639 INFO:teuthology.orchestra.run.smithi101.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-07-29T00:39:53.639 INFO:teuthology.orchestra.run.smithi101.stderr:curl: (7) Failed to connect to smithi101.front.sepia.ceph.com port 8001: Connection refused 2023-07-29T00:39:54.640 DEBUG:teuthology.orchestra.run.smithi101:> curl http://smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:54.659 INFO:teuthology.orchestra.run.smithi101.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-07-29T00:39:54.659 INFO:teuthology.orchestra.run.smithi101.stderr: Dload Upload Total Spent Left Speed 2023-07-29T00:39:54.659 INFO:teuthology.orchestra.run.smithi101.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-07-29T00:39:54.660 INFO:teuthology.orchestra.run.smithi101.stdout:anonymous 2023-07-29T00:39:54.660 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-59 2023-07-29T00:39:54.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:39:54.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:39:54.661 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:39:54.755 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:54.755 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:54.794 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:54.729+0000 7f33ddac0880 1 bucket sync caught up with source: 2023-07-29T00:39:54.795 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.101.6, , , , , , 00000000002.70.6, , , , , 00000000002.62.6, , , , , , , , , , , , , , , 00000000001.88.6, , , , 00000000001.74.6] 2023-07-29T00:39:54.795 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.101.6, , , , , , 00000000002.70.6, , , , , 00000000002.62.6, , , , , , , , , , , , , , , 00000000001.88.6, , , , 00000000001.74.6] 2023-07-29T00:39:54.795 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:54.729+0000 7f33ddac0880 0 bucket checkpoint complete 2023-07-29T00:39:54.808 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-59 2023-07-29T00:39:54.808 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:39:54.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:39:54.810 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:39:54.864 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:39:54.864 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:39:54.897 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:54.833+0000 7f1380134880 1 bucket sync caught up with source: 2023-07-29T00:39:54.898 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000009.878.6, 00000000006.718.6, 00000000006.1226.6, 00000000015.986.6, 00000000012.769.6, 00000000003.1028.6, 00000000003.444.6, 00000000012.331.6, 00000000021.837.6, 00000000018.1411.6] 2023-07-29T00:39:54.898 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000009.878.6, 00000000006.718.6, 00000000006.1226.6, 00000000015.986.6, 00000000012.769.6, 00000000003.1028.6, 00000000003.444.6, 00000000012.331.6, 00000000021.837.6, 00000000018.1411.6] 2023-07-29T00:39:54.898 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:39:54.833+0000 7f1380134880 0 bucket checkpoint complete 2023-07-29T00:39:54.911 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-59 zones={a2, a1} 2023-07-29T00:39:54.912 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:39:54.912 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:39:54.913 DEBUG:boto:Method: HEAD 2023-07-29T00:39:54.914 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:39:54.915 DEBUG:boto:Data: 2023-07-29T00:39:54.915 DEBUG:boto:Headers: {} 2023-07-29T00:39:54.916 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:39:54.917 DEBUG:boto:Port: 8001 2023-07-29T00:39:54.918 DEBUG:boto:Params: {} 2023-07-29T00:39:54.919 DEBUG:boto:Token: None 2023-07-29T00:39:54.920 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:54 GMT /hiiqos-59/ 2023-07-29T00:39:54.920 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9ywRk69yEROJDfV/TgEhwSyi2KI= 2023-07-29T00:39:54.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9ywRk69yEROJDfV/TgEhwSyi2KI=', 'Content-Length': '0'} 2023-07-29T00:39:54.921 DEBUG:boto:encountered RemoteDisconnected exception, reconnecting 2023-07-29T00:39:54.922 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-29T00:39:55.591 DEBUG:boto:Token: None 2023-07-29T00:39:55.592 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:39:54 GMT /hiiqos-59/ 2023-07-29T00:39:55.593 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9ywRk69yEROJDfV/TgEhwSyi2KI= 2023-07-29T00:39:55.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:39:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9ywRk69yEROJDfV/TgEhwSyi2KI=', 'Content-Length': '0'} 2023-07-29T00:40:00.599 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4ed7d1553abf615-0064c45fe0-1eb0-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:00.600 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:00.600 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:00.601 DEBUG:boto:Method: HEAD 2023-07-29T00:40:00.601 DEBUG:boto:Path: /hiiqos-59/ 2023-07-29T00:40:00.601 DEBUG:boto:Data: 2023-07-29T00:40:00.602 DEBUG:boto:Headers: {} 2023-07-29T00:40:00.602 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:00.602 DEBUG:boto:Port: 8000 2023-07-29T00:40:00.603 DEBUG:boto:Params: {} 2023-07-29T00:40:00.603 DEBUG:boto:Token: None 2023-07-29T00:40:00.603 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:00 GMT /hiiqos-59/ 2023-07-29T00:40:00.604 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7a/kPNXDIssDblrvFWXJC4+VQGc= 2023-07-29T00:40:00.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7a/kPNXDIssDblrvFWXJC4+VQGc=', 'Content-Length': '0'} 2023-07-29T00:40:00.609 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001f24bfe4ced208ca-0064c45fe0-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:00.609 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-29T00:40:00.609 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:00.610 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:00.610 DEBUG:boto:path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:40:00.610 DEBUG:boto:auth_path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:40:00.611 DEBUG:boto:Method: GET 2023-07-29T00:40:00.611 DEBUG:boto:Path: /hiiqos-59/?versions&max-keys=999 2023-07-29T00:40:00.611 DEBUG:boto:Data: 2023-07-29T00:40:00.612 DEBUG:boto:Headers: {} 2023-07-29T00:40:00.612 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:00.613 DEBUG:boto:Port: 8001 2023-07-29T00:40:00.613 DEBUG:boto:Params: {} 2023-07-29T00:40:00.613 DEBUG:boto:Token: None 2023-07-29T00:40:00.614 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:00 GMT /hiiqos-59/?versions 2023-07-29T00:40:00.614 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Ha8EwFjvfeQcrMKvlam/i/yg4Cw= 2023-07-29T00:40:00.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Ha8EwFjvfeQcrMKvlam/i/yg4Cw=', 'Content-Length': '0'} 2023-07-29T00:40:00.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf52a3f3b7cc6408-0064c45fe0-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:00.619 DEBUG:boto:b'hiiqos-59999falseobj11nulltrue2023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-07-29T00:37:53.213Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-07-29T00:35:31.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-07-29T00:35:01.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-07-29T00:37:53.270Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-07-29T00:35:31.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-07-29T00:37:53.291Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-07-29T00:33:06.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-07-29T00:35:01.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-07-29T00:35:31.771Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-07-29T00:37:53.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-07-29T00:33:06.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-07-29T00:35:01.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-07-29T00:35:31.826Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-07-29T00:33:06.857Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-07-29T00:37:53.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-07-29T00:35:01.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-07-29T00:33:06.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-07-29T00:35:31.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:00.622 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-07-29T00:40:00.622 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-07-29T00:40:00.622 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-07-29T00:40:00.623 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-07-29T00:40:00.623 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-07-29T00:40:00.623 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-07-29T00:40:00.624 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-07-29T00:40:00.624 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-07-29T00:40:00.624 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-07-29T00:40:00.625 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-07-29T00:40:00.625 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-07-29T00:40:00.625 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-07-29T00:40:00.626 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-07-29T00:40:00.626 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-07-29T00:40:00.626 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-07-29T00:40:00.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-07-29T00:40:00.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-07-29T00:40:00.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-07-29T00:40:00.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-07-29T00:40:00.628 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-07-29T00:40:00.628 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-07-29T00:40:00.628 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-07-29T00:40:00.629 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-07-29T00:40:00.629 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-07-29T00:40:00.629 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-07-29T00:40:00.629 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-07-29T00:40:00.630 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-07-29T00:40:00.630 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-07-29T00:40:00.630 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-07-29T00:40:00.631 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-07-29T00:40:00.631 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-07-29T00:40:00.631 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-07-29T00:40:00.632 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-07-29T00:40:00.632 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-07-29T00:40:00.632 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-07-29T00:40:00.633 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-29T00:40:00.633 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:00.633 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:00.634 DEBUG:boto:path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:40:00.634 DEBUG:boto:auth_path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:40:00.635 DEBUG:boto:Method: GET 2023-07-29T00:40:00.635 DEBUG:boto:Path: /hiiqos-59/?versions&max-keys=999 2023-07-29T00:40:00.635 DEBUG:boto:Data: 2023-07-29T00:40:00.636 DEBUG:boto:Headers: {} 2023-07-29T00:40:00.637 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:00.638 DEBUG:boto:Port: 8000 2023-07-29T00:40:00.639 DEBUG:boto:Params: {} 2023-07-29T00:40:00.640 DEBUG:boto:Token: None 2023-07-29T00:40:00.641 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:00 GMT /hiiqos-59/?versions 2023-07-29T00:40:00.642 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Ha8EwFjvfeQcrMKvlam/i/yg4Cw= 2023-07-29T00:40:00.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Ha8EwFjvfeQcrMKvlam/i/yg4Cw=', 'Content-Length': '0'} 2023-07-29T00:40:00.648 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8ce6300685499fc-0064c45fe0-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:00.649 DEBUG:boto:b'hiiqos-59999falseobj11nulltrue2023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-07-29T00:37:53.213Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-07-29T00:35:31.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-07-29T00:35:01.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-07-29T00:37:53.270Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-07-29T00:35:31.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-07-29T00:37:53.291Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-07-29T00:33:06.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-07-29T00:35:01.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-07-29T00:35:31.771Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-07-29T00:37:53.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-07-29T00:33:06.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-07-29T00:35:01.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-07-29T00:35:31.826Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-07-29T00:33:06.857Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-07-29T00:37:53.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-07-29T00:35:01.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-07-29T00:33:06.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-07-29T00:35:31.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:00.651 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-07-29T00:40:00.651 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-07-29T00:40:00.651 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-07-29T00:40:00.652 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-07-29T00:40:00.652 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-07-29T00:40:00.652 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-07-29T00:40:00.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-07-29T00:40:00.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-07-29T00:40:00.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-07-29T00:40:00.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-07-29T00:40:00.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-07-29T00:40:00.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-07-29T00:40:00.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-07-29T00:40:00.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-07-29T00:40:00.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-07-29T00:40:00.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-07-29T00:40:00.656 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-07-29T00:40:00.656 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-07-29T00:40:00.656 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-07-29T00:40:00.657 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-07-29T00:40:00.657 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-07-29T00:40:00.657 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-07-29T00:40:00.657 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-07-29T00:40:00.658 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-07-29T00:40:00.658 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-07-29T00:40:00.659 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-07-29T00:40:00.659 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-07-29T00:40:00.659 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-07-29T00:40:00.659 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-07-29T00:40:00.660 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-07-29T00:40:00.660 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-07-29T00:40:00.660 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-07-29T00:40:00.661 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-07-29T00:40:00.661 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-07-29T00:40:00.661 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-07-29T00:40:00.662 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:00.662 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:00.662 DEBUG:boto:path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:40:00.663 DEBUG:boto:auth_path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:40:00.663 DEBUG:boto:Method: GET 2023-07-29T00:40:00.663 DEBUG:boto:Path: /hiiqos-59/?versions&max-keys=999 2023-07-29T00:40:00.664 DEBUG:boto:Data: 2023-07-29T00:40:00.664 DEBUG:boto:Headers: {} 2023-07-29T00:40:00.665 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:00.665 DEBUG:boto:Port: 8001 2023-07-29T00:40:00.665 DEBUG:boto:Params: {} 2023-07-29T00:40:00.666 DEBUG:boto:Token: None 2023-07-29T00:40:00.666 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:00 GMT /hiiqos-59/?versions 2023-07-29T00:40:00.667 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Ha8EwFjvfeQcrMKvlam/i/yg4Cw= 2023-07-29T00:40:00.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Ha8EwFjvfeQcrMKvlam/i/yg4Cw=', 'Content-Length': '0'} 2023-07-29T00:40:00.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb97b12cf9869f54-0064c45fe0-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:00.671 DEBUG:boto:b'hiiqos-59999falseobj11nulltrue2023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-07-29T00:37:53.213Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-07-29T00:35:31.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-07-29T00:35:01.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-07-29T00:37:53.270Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-07-29T00:35:31.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-07-29T00:37:53.291Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-07-29T00:33:06.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-07-29T00:35:01.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-07-29T00:35:31.771Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-07-29T00:37:53.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-07-29T00:33:06.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-07-29T00:35:01.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-07-29T00:35:31.826Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-07-29T00:33:06.857Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-07-29T00:37:53.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-07-29T00:35:01.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-07-29T00:33:06.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-07-29T00:35:31.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:00.673 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:00.673 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:00.674 DEBUG:boto:path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:40:00.674 DEBUG:boto:auth_path=/hiiqos-59/?versions&max-keys=999 2023-07-29T00:40:00.674 DEBUG:boto:Method: GET 2023-07-29T00:40:00.675 DEBUG:boto:Path: /hiiqos-59/?versions&max-keys=999 2023-07-29T00:40:00.675 DEBUG:boto:Data: 2023-07-29T00:40:00.675 DEBUG:boto:Headers: {} 2023-07-29T00:40:00.676 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:00.676 DEBUG:boto:Port: 8000 2023-07-29T00:40:00.676 DEBUG:boto:Params: {} 2023-07-29T00:40:00.677 DEBUG:boto:Token: None 2023-07-29T00:40:00.677 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:00 GMT /hiiqos-59/?versions 2023-07-29T00:40:00.678 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Ha8EwFjvfeQcrMKvlam/i/yg4Cw= 2023-07-29T00:40:00.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Ha8EwFjvfeQcrMKvlam/i/yg4Cw=', 'Content-Length': '0'} 2023-07-29T00:40:00.683 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099cc03285a990c81-0064c45fe0-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:00.683 DEBUG:boto:b'hiiqos-59999falseobj11nulltrue2023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-07-29T00:37:53.213Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-07-29T00:35:31.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-07-29T00:35:01.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-07-29T00:37:53.270Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-07-29T00:35:31.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-07-29T00:37:53.291Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-07-29T00:33:06.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-07-29T00:35:01.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-07-29T00:35:31.771Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-07-29T00:37:53.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-07-29T00:33:06.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-07-29T00:35:01.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-07-29T00:35:31.826Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-07-29T00:33:06.857Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-07-29T00:37:53.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-07-29T00:35:01.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-07-29T00:33:06.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-07-29T00:35:31.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:00.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-07-29T00:40:00.686 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:40:00.686 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:40:00.687 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:40:00.687 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:40:00.687 DEBUG:boto:Method: GET 2023-07-29T00:40:00.688 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:40:00.688 DEBUG:boto:Data: 2023-07-29T00:40:00.689 DEBUG:boto:Headers: {} 2023-07-29T00:40:00.689 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:00.690 DEBUG:boto:Port: 8001 2023-07-29T00:40:00.690 DEBUG:boto:Params: {} 2023-07-29T00:40:00.690 DEBUG:boto:Token: None 2023-07-29T00:40:00.691 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:00 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:40:00.691 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aaCkq1KSsV3sEUq4cY4a6zLlPFo= 2023-07-29T00:40:00.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aaCkq1KSsV3sEUq4cY4a6zLlPFo=', 'Content-Length': '0'} 2023-07-29T00:40:00.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb90300f8048a08f-0064c45fe0-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:00.695 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:40:00.695 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:40:00.696 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:40:00.696 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:40:00.696 DEBUG:boto:Method: GET 2023-07-29T00:40:00.697 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:40:00.697 DEBUG:boto:Data: 2023-07-29T00:40:00.697 DEBUG:boto:Headers: {} 2023-07-29T00:40:00.698 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:00.698 DEBUG:boto:Port: 8000 2023-07-29T00:40:00.699 DEBUG:boto:Params: {} 2023-07-29T00:40:00.699 DEBUG:boto:Token: None 2023-07-29T00:40:00.699 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:00 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:40:00.700 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aaCkq1KSsV3sEUq4cY4a6zLlPFo= 2023-07-29T00:40:00.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aaCkq1KSsV3sEUq4cY4a6zLlPFo=', 'Content-Length': '0'} 2023-07-29T00:40:00.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071a9591b838e9147-0064c45fe0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:00.744 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:40:00.745 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:40:00.745 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:40:00.746 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:40:00.746 DEBUG:boto:Method: HEAD 2023-07-29T00:40:00.746 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:40:00.747 DEBUG:boto:Data: 2023-07-29T00:40:00.747 DEBUG:boto:Headers: {} 2023-07-29T00:40:00.747 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:00.748 DEBUG:boto:Port: 8001 2023-07-29T00:40:00.748 DEBUG:boto:Params: {} 2023-07-29T00:40:00.748 DEBUG:boto:Token: None 2023-07-29T00:40:00.749 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:00 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:40:00.749 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RTlmRXbfvl6mJhAETpcpztecSxY= 2023-07-29T00:40:00.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RTlmRXbfvl6mJhAETpcpztecSxY=', 'Content-Length': '0'} 2023-07-29T00:40:00.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007393da40916dd6f0-0064c45fe0-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:00.752 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:40:00.752 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:40:00.753 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:40:00.753 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:40:00.753 DEBUG:boto:Method: HEAD 2023-07-29T00:40:00.754 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:40:00.754 DEBUG:boto:Data: 2023-07-29T00:40:00.754 DEBUG:boto:Headers: {} 2023-07-29T00:40:00.755 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:00.755 DEBUG:boto:Port: 8000 2023-07-29T00:40:00.755 DEBUG:boto:Params: {} 2023-07-29T00:40:00.756 DEBUG:boto:Token: None 2023-07-29T00:40:00.756 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:00 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:40:00.757 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RTlmRXbfvl6mJhAETpcpztecSxY= 2023-07-29T00:40:00.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RTlmRXbfvl6mJhAETpcpztecSxY=', 'Content-Length': '0'} 2023-07-29T00:40:00.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026a0a7105fee258a-0064c45fe0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:00.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-07-29T00:40:00.759 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:40:00.760 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:40:00.760 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:40:00.761 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:40:00.761 DEBUG:boto:Method: GET 2023-07-29T00:40:00.761 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:40:00.762 DEBUG:boto:Data: 2023-07-29T00:40:00.762 DEBUG:boto:Headers: {} 2023-07-29T00:40:00.762 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:00.763 DEBUG:boto:Port: 8001 2023-07-29T00:40:00.763 DEBUG:boto:Params: {} 2023-07-29T00:40:00.763 DEBUG:boto:Token: None 2023-07-29T00:40:00.764 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:00 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:40:00.764 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aaCkq1KSsV3sEUq4cY4a6zLlPFo= 2023-07-29T00:40:00.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aaCkq1KSsV3sEUq4cY4a6zLlPFo=', 'Content-Length': '0'} 2023-07-29T00:40:00.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090c6a3a4ff2b1cbd-0064c45fe0-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:00.767 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:40:00.768 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:40:00.768 DEBUG:boto:path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:40:00.769 DEBUG:boto:auth_path=/hiiqos-59/obj11?versionId=null 2023-07-29T00:40:00.769 DEBUG:boto:Method: GET 2023-07-29T00:40:00.770 DEBUG:boto:Path: /hiiqos-59/obj11?versionId=null 2023-07-29T00:40:00.770 DEBUG:boto:Data: 2023-07-29T00:40:00.770 DEBUG:boto:Headers: {} 2023-07-29T00:40:00.771 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:00.771 DEBUG:boto:Port: 8000 2023-07-29T00:40:00.771 DEBUG:boto:Params: {} 2023-07-29T00:40:00.772 DEBUG:boto:Token: None 2023-07-29T00:40:00.772 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:00 GMT /hiiqos-59/obj11?versionId=null 2023-07-29T00:40:00.773 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aaCkq1KSsV3sEUq4cY4a6zLlPFo= 2023-07-29T00:40:00.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aaCkq1KSsV3sEUq4cY4a6zLlPFo=', 'Content-Length': '0'} 2023-07-29T00:40:00.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9c11a4242450e92-0064c45fe0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:00.775 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:00.776 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:00.776 DEBUG:boto:path=/hiiqos-59/?prefix=obj11 2023-07-29T00:40:00.777 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj11 2023-07-29T00:40:00.777 DEBUG:boto:Method: GET 2023-07-29T00:40:00.777 DEBUG:boto:Path: /hiiqos-59/?prefix=obj11 2023-07-29T00:40:00.778 DEBUG:boto:Data: 2023-07-29T00:40:00.778 DEBUG:boto:Headers: {} 2023-07-29T00:40:00.778 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:00.779 DEBUG:boto:Port: 8001 2023-07-29T00:40:00.779 DEBUG:boto:Params: {} 2023-07-29T00:40:00.779 DEBUG:boto:Token: None 2023-07-29T00:40:00.780 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:00 GMT /hiiqos-59/ 2023-07-29T00:40:00.780 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5odQuPUC98x7OMD+RoZdXnSdoaU= 2023-07-29T00:40:00.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5odQuPUC98x7OMD+RoZdXnSdoaU=', 'Content-Length': '0'} 2023-07-29T00:40:00.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009873b4f7c70f9f9b-0064c45fe0-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:00.783 DEBUG:boto:b'hiiqos-59obj111000falseobj112023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-07-29T00:37:53.213Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:00.784 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:00.784 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:00.785 DEBUG:boto:path=/hiiqos-59/?prefix=obj11 2023-07-29T00:40:00.785 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj11 2023-07-29T00:40:00.786 DEBUG:boto:Method: GET 2023-07-29T00:40:00.786 DEBUG:boto:Path: /hiiqos-59/?prefix=obj11 2023-07-29T00:40:00.786 DEBUG:boto:Data: 2023-07-29T00:40:00.787 DEBUG:boto:Headers: {} 2023-07-29T00:40:00.787 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:00.787 DEBUG:boto:Port: 8000 2023-07-29T00:40:00.788 DEBUG:boto:Params: {} 2023-07-29T00:40:00.788 DEBUG:boto:Token: None 2023-07-29T00:40:00.789 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:00 GMT /hiiqos-59/ 2023-07-29T00:40:00.791 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5odQuPUC98x7OMD+RoZdXnSdoaU= 2023-07-29T00:40:00.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5odQuPUC98x7OMD+RoZdXnSdoaU=', 'Content-Length': '0'} 2023-07-29T00:40:00.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fe38212035427be-0064c45fe0-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:00.796 DEBUG:boto:b'hiiqos-59obj111000falseobj112023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-07-29T00:37:53.213Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:00.797 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:40:00.797 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:40:00.798 DEBUG:boto:Method: HEAD 2023-07-29T00:40:00.798 DEBUG:boto:Path: /hiiqos-59/obj11 2023-07-29T00:40:00.799 DEBUG:boto:Data: 2023-07-29T00:40:00.799 DEBUG:boto:Headers: {} 2023-07-29T00:40:00.799 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:00.800 DEBUG:boto:Port: 8001 2023-07-29T00:40:00.800 DEBUG:boto:Params: {} 2023-07-29T00:40:00.800 DEBUG:boto:Token: None 2023-07-29T00:40:00.801 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:00 GMT /hiiqos-59/obj11 2023-07-29T00:40:00.801 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VcVwHNY59/Y5pWt3sQyFy9S8/+0= 2023-07-29T00:40:00.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VcVwHNY59/Y5pWt3sQyFy9S8/+0=', 'Content-Length': '0'} 2023-07-29T00:40:00.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e59d751bf289b14-0064c45fe0-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:00.804 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:40:00.804 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:40:00.805 DEBUG:boto:Method: HEAD 2023-07-29T00:40:00.805 DEBUG:boto:Path: /hiiqos-59/obj11 2023-07-29T00:40:00.805 DEBUG:boto:Data: 2023-07-29T00:40:00.806 DEBUG:boto:Headers: {} 2023-07-29T00:40:00.806 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:00.806 DEBUG:boto:Port: 8000 2023-07-29T00:40:00.807 DEBUG:boto:Params: {} 2023-07-29T00:40:00.807 DEBUG:boto:Token: None 2023-07-29T00:40:00.807 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:00 GMT /hiiqos-59/obj11 2023-07-29T00:40:00.808 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VcVwHNY59/Y5pWt3sQyFy9S8/+0= 2023-07-29T00:40:00.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VcVwHNY59/Y5pWt3sQyFy9S8/+0=', 'Content-Length': '0'} 2023-07-29T00:40:00.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017465d2b3c048a99-0064c45fe0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:00.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-07-29T00:40:00.811 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:40:00.811 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:40:00.811 DEBUG:boto:Method: GET 2023-07-29T00:40:00.812 DEBUG:boto:Path: /hiiqos-59/obj11 2023-07-29T00:40:00.812 DEBUG:boto:Data: 2023-07-29T00:40:00.813 DEBUG:boto:Headers: {} 2023-07-29T00:40:00.813 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:00.813 DEBUG:boto:Port: 8001 2023-07-29T00:40:00.814 DEBUG:boto:Params: {} 2023-07-29T00:40:00.814 DEBUG:boto:Token: None 2023-07-29T00:40:00.814 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:00 GMT /hiiqos-59/obj11 2023-07-29T00:40:00.815 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:v0XOufXwUcu2uxuZrQuL7LdiRcM= 2023-07-29T00:40:00.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:v0XOufXwUcu2uxuZrQuL7LdiRcM=', 'Content-Length': '0'} 2023-07-29T00:40:00.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de7f318f672436b1-0064c45fe0-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:00.818 DEBUG:boto:path=/hiiqos-59/obj11 2023-07-29T00:40:00.818 DEBUG:boto:auth_path=/hiiqos-59/obj11 2023-07-29T00:40:00.818 DEBUG:boto:Method: GET 2023-07-29T00:40:00.819 DEBUG:boto:Path: /hiiqos-59/obj11 2023-07-29T00:40:00.819 DEBUG:boto:Data: 2023-07-29T00:40:00.820 DEBUG:boto:Headers: {} 2023-07-29T00:40:00.820 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:00.820 DEBUG:boto:Port: 8000 2023-07-29T00:40:00.821 DEBUG:boto:Params: {} 2023-07-29T00:40:00.821 DEBUG:boto:Token: None 2023-07-29T00:40:00.821 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:00 GMT /hiiqos-59/obj11 2023-07-29T00:40:00.822 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:v0XOufXwUcu2uxuZrQuL7LdiRcM= 2023-07-29T00:40:00.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:v0XOufXwUcu2uxuZrQuL7LdiRcM=', 'Content-Length': '0'} 2023-07-29T00:40:00.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003514988962032dfd-0064c45fe0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:00.825 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:00.825 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:00.825 DEBUG:boto:path=/hiiqos-59/?prefix=obj11 2023-07-29T00:40:00.826 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj11 2023-07-29T00:40:00.826 DEBUG:boto:Method: GET 2023-07-29T00:40:00.827 DEBUG:boto:Path: /hiiqos-59/?prefix=obj11 2023-07-29T00:40:00.827 DEBUG:boto:Data: 2023-07-29T00:40:00.827 DEBUG:boto:Headers: {} 2023-07-29T00:40:00.828 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:00.828 DEBUG:boto:Port: 8001 2023-07-29T00:40:00.828 DEBUG:boto:Params: {} 2023-07-29T00:40:00.829 DEBUG:boto:Token: None 2023-07-29T00:40:00.829 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:00 GMT /hiiqos-59/ 2023-07-29T00:40:00.830 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5odQuPUC98x7OMD+RoZdXnSdoaU= 2023-07-29T00:40:00.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5odQuPUC98x7OMD+RoZdXnSdoaU=', 'Content-Length': '0'} 2023-07-29T00:40:00.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006638c4bf19184d4e-0064c45fe0-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:00.833 DEBUG:boto:b'hiiqos-59obj111000falseobj112023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-07-29T00:37:53.213Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:00.834 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:00.834 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:00.834 DEBUG:boto:path=/hiiqos-59/?prefix=obj11 2023-07-29T00:40:00.835 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj11 2023-07-29T00:40:00.835 DEBUG:boto:Method: GET 2023-07-29T00:40:00.836 DEBUG:boto:Path: /hiiqos-59/?prefix=obj11 2023-07-29T00:40:00.836 DEBUG:boto:Data: 2023-07-29T00:40:00.836 DEBUG:boto:Headers: {} 2023-07-29T00:40:00.837 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:00.837 DEBUG:boto:Port: 8000 2023-07-29T00:40:00.837 DEBUG:boto:Params: {} 2023-07-29T00:40:00.838 DEBUG:boto:Token: None 2023-07-29T00:40:00.838 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:00 GMT /hiiqos-59/ 2023-07-29T00:40:00.839 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5odQuPUC98x7OMD+RoZdXnSdoaU= 2023-07-29T00:40:00.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5odQuPUC98x7OMD+RoZdXnSdoaU=', 'Content-Length': '0'} 2023-07-29T00:40:00.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005519e8efdbb18343-0064c45fe0-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:00.844 DEBUG:boto:b'hiiqos-59obj111000falseobj112023-07-29T00:26:54.480Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-07-29T00:37:53.213Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:00.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-07-29T00:40:00.845 DEBUG:boto:path=/hiiqos-59/obj111 2023-07-29T00:40:00.845 DEBUG:boto:auth_path=/hiiqos-59/obj111 2023-07-29T00:40:00.845 DEBUG:boto:path=/hiiqos-59/obj111?versionId=null 2023-07-29T00:40:00.846 DEBUG:boto:auth_path=/hiiqos-59/obj111?versionId=null 2023-07-29T00:40:00.846 DEBUG:boto:Method: GET 2023-07-29T00:40:00.847 DEBUG:boto:Path: /hiiqos-59/obj111?versionId=null 2023-07-29T00:40:00.847 DEBUG:boto:Data: 2023-07-29T00:40:00.847 DEBUG:boto:Headers: {} 2023-07-29T00:40:00.848 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:00.848 DEBUG:boto:Port: 8001 2023-07-29T00:40:00.848 DEBUG:boto:Params: {} 2023-07-29T00:40:00.849 DEBUG:boto:Token: None 2023-07-29T00:40:00.849 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:00 GMT /hiiqos-59/obj111?versionId=null 2023-07-29T00:40:00.850 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:X86mEegP9SYdEmJSHNnaN1YcDgA= 2023-07-29T00:40:00.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:X86mEegP9SYdEmJSHNnaN1YcDgA=', 'Content-Length': '0'} 2023-07-29T00:40:00.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000774d2a9a791d22c6-0064c45fe0-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:00.896 DEBUG:boto:path=/hiiqos-59/obj111 2023-07-29T00:40:00.897 DEBUG:boto:auth_path=/hiiqos-59/obj111 2023-07-29T00:40:00.897 DEBUG:boto:path=/hiiqos-59/obj111?versionId=null 2023-07-29T00:40:00.898 DEBUG:boto:auth_path=/hiiqos-59/obj111?versionId=null 2023-07-29T00:40:00.898 DEBUG:boto:Method: GET 2023-07-29T00:40:00.898 DEBUG:boto:Path: /hiiqos-59/obj111?versionId=null 2023-07-29T00:40:00.899 DEBUG:boto:Data: 2023-07-29T00:40:00.899 DEBUG:boto:Headers: {} 2023-07-29T00:40:00.899 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:00.900 DEBUG:boto:Port: 8000 2023-07-29T00:40:00.900 DEBUG:boto:Params: {} 2023-07-29T00:40:00.900 DEBUG:boto:Token: None 2023-07-29T00:40:00.901 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:00 GMT /hiiqos-59/obj111?versionId=null 2023-07-29T00:40:00.901 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:X86mEegP9SYdEmJSHNnaN1YcDgA= 2023-07-29T00:40:00.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:X86mEegP9SYdEmJSHNnaN1YcDgA=', 'Content-Length': '0'} 2023-07-29T00:40:00.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025486e3b5758b797-0064c45fe0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:00.904 DEBUG:boto:path=/hiiqos-59/obj111 2023-07-29T00:40:00.905 DEBUG:boto:auth_path=/hiiqos-59/obj111 2023-07-29T00:40:00.905 DEBUG:boto:path=/hiiqos-59/obj111?versionId=null 2023-07-29T00:40:00.905 DEBUG:boto:auth_path=/hiiqos-59/obj111?versionId=null 2023-07-29T00:40:00.906 DEBUG:boto:Method: HEAD 2023-07-29T00:40:00.906 DEBUG:boto:Path: /hiiqos-59/obj111?versionId=null 2023-07-29T00:40:00.906 DEBUG:boto:Data: 2023-07-29T00:40:00.907 DEBUG:boto:Headers: {} 2023-07-29T00:40:00.907 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:00.907 DEBUG:boto:Port: 8001 2023-07-29T00:40:00.908 DEBUG:boto:Params: {} 2023-07-29T00:40:00.908 DEBUG:boto:Token: None 2023-07-29T00:40:00.908 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:00 GMT /hiiqos-59/obj111?versionId=null 2023-07-29T00:40:00.909 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ISdqayHn8p+dzaF+zRdO7Qr9dYQ= 2023-07-29T00:40:00.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ISdqayHn8p+dzaF+zRdO7Qr9dYQ=', 'Content-Length': '0'} 2023-07-29T00:40:00.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b534b5dbda4eeca7-0064c45fe0-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:00.912 DEBUG:boto:path=/hiiqos-59/obj111 2023-07-29T00:40:00.912 DEBUG:boto:auth_path=/hiiqos-59/obj111 2023-07-29T00:40:00.912 DEBUG:boto:path=/hiiqos-59/obj111?versionId=null 2023-07-29T00:40:00.913 DEBUG:boto:auth_path=/hiiqos-59/obj111?versionId=null 2023-07-29T00:40:00.913 DEBUG:boto:Method: HEAD 2023-07-29T00:40:00.913 DEBUG:boto:Path: /hiiqos-59/obj111?versionId=null 2023-07-29T00:40:00.914 DEBUG:boto:Data: 2023-07-29T00:40:00.914 DEBUG:boto:Headers: {} 2023-07-29T00:40:00.914 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:00.915 DEBUG:boto:Port: 8000 2023-07-29T00:40:00.915 DEBUG:boto:Params: {} 2023-07-29T00:40:00.915 DEBUG:boto:Token: None 2023-07-29T00:40:00.916 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:00 GMT /hiiqos-59/obj111?versionId=null 2023-07-29T00:40:00.916 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ISdqayHn8p+dzaF+zRdO7Qr9dYQ= 2023-07-29T00:40:00.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ISdqayHn8p+dzaF+zRdO7Qr9dYQ=', 'Content-Length': '0'} 2023-07-29T00:40:00.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000376ee25ec3e4c89e-0064c45fe0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:00.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-07-29T00:40:00.919 DEBUG:boto:path=/hiiqos-59/obj111 2023-07-29T00:40:00.920 DEBUG:boto:auth_path=/hiiqos-59/obj111 2023-07-29T00:40:00.920 DEBUG:boto:path=/hiiqos-59/obj111?versionId=null 2023-07-29T00:40:00.920 DEBUG:boto:auth_path=/hiiqos-59/obj111?versionId=null 2023-07-29T00:40:00.921 DEBUG:boto:Method: GET 2023-07-29T00:40:00.921 DEBUG:boto:Path: /hiiqos-59/obj111?versionId=null 2023-07-29T00:40:00.921 DEBUG:boto:Data: 2023-07-29T00:40:00.922 DEBUG:boto:Headers: {} 2023-07-29T00:40:00.922 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:00.923 DEBUG:boto:Port: 8001 2023-07-29T00:40:00.923 DEBUG:boto:Params: {} 2023-07-29T00:40:00.923 DEBUG:boto:Token: None 2023-07-29T00:40:00.924 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:00 GMT /hiiqos-59/obj111?versionId=null 2023-07-29T00:40:00.924 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:X86mEegP9SYdEmJSHNnaN1YcDgA= 2023-07-29T00:40:00.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:X86mEegP9SYdEmJSHNnaN1YcDgA=', 'Content-Length': '0'} 2023-07-29T00:40:00.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcdae4c11a58f9ef-0064c45fe0-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:00.927 DEBUG:boto:path=/hiiqos-59/obj111 2023-07-29T00:40:00.927 DEBUG:boto:auth_path=/hiiqos-59/obj111 2023-07-29T00:40:00.928 DEBUG:boto:path=/hiiqos-59/obj111?versionId=null 2023-07-29T00:40:00.928 DEBUG:boto:auth_path=/hiiqos-59/obj111?versionId=null 2023-07-29T00:40:00.928 DEBUG:boto:Method: GET 2023-07-29T00:40:00.929 DEBUG:boto:Path: /hiiqos-59/obj111?versionId=null 2023-07-29T00:40:00.929 DEBUG:boto:Data: 2023-07-29T00:40:00.929 DEBUG:boto:Headers: {} 2023-07-29T00:40:00.930 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:00.930 DEBUG:boto:Port: 8000 2023-07-29T00:40:00.931 DEBUG:boto:Params: {} 2023-07-29T00:40:00.931 DEBUG:boto:Token: None 2023-07-29T00:40:00.932 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:00 GMT /hiiqos-59/obj111?versionId=null 2023-07-29T00:40:00.932 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:X86mEegP9SYdEmJSHNnaN1YcDgA= 2023-07-29T00:40:00.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:X86mEegP9SYdEmJSHNnaN1YcDgA=', 'Content-Length': '0'} 2023-07-29T00:40:00.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002691f83ed83a686c-0064c45fe0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:00.935 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:00.935 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:00.936 DEBUG:boto:path=/hiiqos-59/?prefix=obj111 2023-07-29T00:40:00.936 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj111 2023-07-29T00:40:00.936 DEBUG:boto:Method: GET 2023-07-29T00:40:00.937 DEBUG:boto:Path: /hiiqos-59/?prefix=obj111 2023-07-29T00:40:00.937 DEBUG:boto:Data: 2023-07-29T00:40:00.938 DEBUG:boto:Headers: {} 2023-07-29T00:40:00.938 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:00.938 DEBUG:boto:Port: 8001 2023-07-29T00:40:00.939 DEBUG:boto:Params: {} 2023-07-29T00:40:00.939 DEBUG:boto:Token: None 2023-07-29T00:40:00.940 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:00 GMT /hiiqos-59/ 2023-07-29T00:40:00.940 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5odQuPUC98x7OMD+RoZdXnSdoaU= 2023-07-29T00:40:00.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5odQuPUC98x7OMD+RoZdXnSdoaU=', 'Content-Length': '0'} 2023-07-29T00:40:00.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebe7599393ba233f-0064c45fe0-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:00.943 DEBUG:boto:b'hiiqos-59obj1111000falseobj1112023-07-29T00:37:53.213Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:00.944 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:00.945 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:00.945 DEBUG:boto:path=/hiiqos-59/?prefix=obj111 2023-07-29T00:40:00.945 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj111 2023-07-29T00:40:00.946 DEBUG:boto:Method: GET 2023-07-29T00:40:00.946 DEBUG:boto:Path: /hiiqos-59/?prefix=obj111 2023-07-29T00:40:00.946 DEBUG:boto:Data: 2023-07-29T00:40:00.947 DEBUG:boto:Headers: {} 2023-07-29T00:40:00.947 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:00.948 DEBUG:boto:Port: 8000 2023-07-29T00:40:00.948 DEBUG:boto:Params: {} 2023-07-29T00:40:00.949 DEBUG:boto:Token: None 2023-07-29T00:40:00.949 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:00 GMT /hiiqos-59/ 2023-07-29T00:40:00.950 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5odQuPUC98x7OMD+RoZdXnSdoaU= 2023-07-29T00:40:00.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5odQuPUC98x7OMD+RoZdXnSdoaU=', 'Content-Length': '0'} 2023-07-29T00:40:00.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000486dd6bf24a52119-0064c45fe0-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:00.955 DEBUG:boto:b'hiiqos-59obj1111000falseobj1112023-07-29T00:37:53.213Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:00.955 DEBUG:boto:path=/hiiqos-59/obj111 2023-07-29T00:40:00.956 DEBUG:boto:auth_path=/hiiqos-59/obj111 2023-07-29T00:40:00.956 DEBUG:boto:Method: HEAD 2023-07-29T00:40:00.956 DEBUG:boto:Path: /hiiqos-59/obj111 2023-07-29T00:40:00.957 DEBUG:boto:Data: 2023-07-29T00:40:00.957 DEBUG:boto:Headers: {} 2023-07-29T00:40:00.958 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:00.958 DEBUG:boto:Port: 8001 2023-07-29T00:40:00.958 DEBUG:boto:Params: {} 2023-07-29T00:40:00.959 DEBUG:boto:Token: None 2023-07-29T00:40:00.959 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:00 GMT /hiiqos-59/obj111 2023-07-29T00:40:00.960 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PDPG/2qL3n69w42/MEu0QKgwn3k= 2023-07-29T00:40:00.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PDPG/2qL3n69w42/MEu0QKgwn3k=', 'Content-Length': '0'} 2023-07-29T00:40:00.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002310ef8ccd7acb60-0064c45fe0-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:00.963 DEBUG:boto:path=/hiiqos-59/obj111 2023-07-29T00:40:00.963 DEBUG:boto:auth_path=/hiiqos-59/obj111 2023-07-29T00:40:00.963 DEBUG:boto:Method: HEAD 2023-07-29T00:40:00.964 DEBUG:boto:Path: /hiiqos-59/obj111 2023-07-29T00:40:00.964 DEBUG:boto:Data: 2023-07-29T00:40:00.965 DEBUG:boto:Headers: {} 2023-07-29T00:40:00.965 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:00.966 DEBUG:boto:Port: 8000 2023-07-29T00:40:00.966 DEBUG:boto:Params: {} 2023-07-29T00:40:00.967 DEBUG:boto:Token: None 2023-07-29T00:40:00.967 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:00 GMT /hiiqos-59/obj111 2023-07-29T00:40:00.968 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PDPG/2qL3n69w42/MEu0QKgwn3k= 2023-07-29T00:40:00.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PDPG/2qL3n69w42/MEu0QKgwn3k=', 'Content-Length': '0'} 2023-07-29T00:40:00.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000871abad7bb589eea-0064c45fe0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:00.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-07-29T00:40:00.971 DEBUG:boto:path=/hiiqos-59/obj111 2023-07-29T00:40:00.972 DEBUG:boto:auth_path=/hiiqos-59/obj111 2023-07-29T00:40:00.972 DEBUG:boto:Method: GET 2023-07-29T00:40:00.973 DEBUG:boto:Path: /hiiqos-59/obj111 2023-07-29T00:40:00.973 DEBUG:boto:Data: 2023-07-29T00:40:00.974 DEBUG:boto:Headers: {} 2023-07-29T00:40:00.974 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:00.975 DEBUG:boto:Port: 8001 2023-07-29T00:40:00.975 DEBUG:boto:Params: {} 2023-07-29T00:40:00.976 DEBUG:boto:Token: None 2023-07-29T00:40:00.977 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:00 GMT /hiiqos-59/obj111 2023-07-29T00:40:00.977 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YRiYmRme//mUV6qc/xggHni1KVI= 2023-07-29T00:40:00.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YRiYmRme//mUV6qc/xggHni1KVI=', 'Content-Length': '0'} 2023-07-29T00:40:00.980 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ddfcea30971812f-0064c45fe0-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:00.981 DEBUG:boto:path=/hiiqos-59/obj111 2023-07-29T00:40:00.981 DEBUG:boto:auth_path=/hiiqos-59/obj111 2023-07-29T00:40:00.982 DEBUG:boto:Method: GET 2023-07-29T00:40:00.982 DEBUG:boto:Path: /hiiqos-59/obj111 2023-07-29T00:40:00.983 DEBUG:boto:Data: 2023-07-29T00:40:00.983 DEBUG:boto:Headers: {} 2023-07-29T00:40:00.984 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:00.984 DEBUG:boto:Port: 8000 2023-07-29T00:40:00.985 DEBUG:boto:Params: {} 2023-07-29T00:40:00.985 DEBUG:boto:Token: None 2023-07-29T00:40:00.986 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:00 GMT /hiiqos-59/obj111 2023-07-29T00:40:00.987 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YRiYmRme//mUV6qc/xggHni1KVI= 2023-07-29T00:40:00.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YRiYmRme//mUV6qc/xggHni1KVI=', 'Content-Length': '0'} 2023-07-29T00:40:00.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000109c7b5ce05c9292-0064c45fe0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.036 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:01.037 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:01.038 DEBUG:boto:path=/hiiqos-59/?prefix=obj111 2023-07-29T00:40:01.039 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj111 2023-07-29T00:40:01.039 DEBUG:boto:Method: GET 2023-07-29T00:40:01.040 DEBUG:boto:Path: /hiiqos-59/?prefix=obj111 2023-07-29T00:40:01.041 DEBUG:boto:Data: 2023-07-29T00:40:01.042 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.042 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:01.043 DEBUG:boto:Port: 8001 2023-07-29T00:40:01.044 DEBUG:boto:Params: {} 2023-07-29T00:40:01.045 DEBUG:boto:Token: None 2023-07-29T00:40:01.045 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/ 2023-07-29T00:40:01.046 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:F6z8zHPL1D3/2sdDjIeuqnNvTns= 2023-07-29T00:40:01.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:F6z8zHPL1D3/2sdDjIeuqnNvTns=', 'Content-Length': '0'} 2023-07-29T00:40:01.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011bf0cf9e4de021e-0064c45fe0-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.051 DEBUG:boto:b'hiiqos-59obj1111000falseobj1112023-07-29T00:37:53.213Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:01.052 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:01.053 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:01.053 DEBUG:boto:path=/hiiqos-59/?prefix=obj111 2023-07-29T00:40:01.054 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj111 2023-07-29T00:40:01.055 DEBUG:boto:Method: GET 2023-07-29T00:40:01.056 DEBUG:boto:Path: /hiiqos-59/?prefix=obj111 2023-07-29T00:40:01.056 DEBUG:boto:Data: 2023-07-29T00:40:01.057 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.058 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:01.058 DEBUG:boto:Port: 8000 2023-07-29T00:40:01.059 DEBUG:boto:Params: {} 2023-07-29T00:40:01.060 DEBUG:boto:Token: None 2023-07-29T00:40:01.061 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/ 2023-07-29T00:40:01.061 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:F6z8zHPL1D3/2sdDjIeuqnNvTns= 2023-07-29T00:40:01.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:F6z8zHPL1D3/2sdDjIeuqnNvTns=', 'Content-Length': '0'} 2023-07-29T00:40:01.067 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000949fdc94c4186610-0064c45fe0-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.068 DEBUG:boto:b'hiiqos-59obj1111000falseobj1112023-07-29T00:37:53.213Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:01.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-07-29T00:40:01.069 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:40:01.070 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:40:01.071 DEBUG:boto:path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:40:01.072 DEBUG:boto:auth_path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:40:01.072 DEBUG:boto:Method: GET 2023-07-29T00:40:01.073 DEBUG:boto:Path: /hiiqos-59/obj115?versionId=null 2023-07-29T00:40:01.074 DEBUG:boto:Data: 2023-07-29T00:40:01.074 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.075 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:01.076 DEBUG:boto:Port: 8001 2023-07-29T00:40:01.077 DEBUG:boto:Params: {} 2023-07-29T00:40:01.077 DEBUG:boto:Token: None 2023-07-29T00:40:01.078 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj115?versionId=null 2023-07-29T00:40:01.079 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Fw0QnNDnmgqFceH9S0DvWQRaU1w= 2023-07-29T00:40:01.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Fw0QnNDnmgqFceH9S0DvWQRaU1w=', 'Content-Length': '0'} 2023-07-29T00:40:01.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a290a1a580891e47-0064c45fe1-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.084 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:40:01.085 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:40:01.085 DEBUG:boto:path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:40:01.086 DEBUG:boto:auth_path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:40:01.087 DEBUG:boto:Method: GET 2023-07-29T00:40:01.088 DEBUG:boto:Path: /hiiqos-59/obj115?versionId=null 2023-07-29T00:40:01.088 DEBUG:boto:Data: 2023-07-29T00:40:01.089 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.090 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:01.091 DEBUG:boto:Port: 8000 2023-07-29T00:40:01.092 DEBUG:boto:Params: {} 2023-07-29T00:40:01.092 DEBUG:boto:Token: None 2023-07-29T00:40:01.093 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj115?versionId=null 2023-07-29T00:40:01.094 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Fw0QnNDnmgqFceH9S0DvWQRaU1w= 2023-07-29T00:40:01.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Fw0QnNDnmgqFceH9S0DvWQRaU1w=', 'Content-Length': '0'} 2023-07-29T00:40:01.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007109ff5d4a9265b8-0064c45fe1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.099 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:40:01.099 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:40:01.100 DEBUG:boto:path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:40:01.101 DEBUG:boto:auth_path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:40:01.102 DEBUG:boto:Method: HEAD 2023-07-29T00:40:01.102 DEBUG:boto:Path: /hiiqos-59/obj115?versionId=null 2023-07-29T00:40:01.103 DEBUG:boto:Data: 2023-07-29T00:40:01.104 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.105 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:01.106 DEBUG:boto:Port: 8001 2023-07-29T00:40:01.106 DEBUG:boto:Params: {} 2023-07-29T00:40:01.107 DEBUG:boto:Token: None 2023-07-29T00:40:01.108 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj115?versionId=null 2023-07-29T00:40:01.109 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lYlGvXMQWPt6yIoZM38s/keopzw= 2023-07-29T00:40:01.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lYlGvXMQWPt6yIoZM38s/keopzw=', 'Content-Length': '0'} 2023-07-29T00:40:01.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000174c2ef1b80bc180-0064c45fe1-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.112 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:40:01.112 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:40:01.112 DEBUG:boto:path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:40:01.113 DEBUG:boto:auth_path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:40:01.113 DEBUG:boto:Method: HEAD 2023-07-29T00:40:01.113 DEBUG:boto:Path: /hiiqos-59/obj115?versionId=null 2023-07-29T00:40:01.114 DEBUG:boto:Data: 2023-07-29T00:40:01.114 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.114 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:01.115 DEBUG:boto:Port: 8000 2023-07-29T00:40:01.115 DEBUG:boto:Params: {} 2023-07-29T00:40:01.115 DEBUG:boto:Token: None 2023-07-29T00:40:01.116 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj115?versionId=null 2023-07-29T00:40:01.116 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lYlGvXMQWPt6yIoZM38s/keopzw= 2023-07-29T00:40:01.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lYlGvXMQWPt6yIoZM38s/keopzw=', 'Content-Length': '0'} 2023-07-29T00:40:01.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013e999264ae54eb4-0064c45fe1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-07-29T00:40:01.119 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:40:01.120 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:40:01.120 DEBUG:boto:path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:40:01.120 DEBUG:boto:auth_path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:40:01.121 DEBUG:boto:Method: GET 2023-07-29T00:40:01.121 DEBUG:boto:Path: /hiiqos-59/obj115?versionId=null 2023-07-29T00:40:01.121 DEBUG:boto:Data: 2023-07-29T00:40:01.122 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.122 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:01.122 DEBUG:boto:Port: 8001 2023-07-29T00:40:01.123 DEBUG:boto:Params: {} 2023-07-29T00:40:01.123 DEBUG:boto:Token: None 2023-07-29T00:40:01.124 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj115?versionId=null 2023-07-29T00:40:01.124 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Fw0QnNDnmgqFceH9S0DvWQRaU1w= 2023-07-29T00:40:01.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Fw0QnNDnmgqFceH9S0DvWQRaU1w=', 'Content-Length': '0'} 2023-07-29T00:40:01.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ec8669b7270a344-0064c45fe1-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.127 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:40:01.127 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:40:01.128 DEBUG:boto:path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:40:01.128 DEBUG:boto:auth_path=/hiiqos-59/obj115?versionId=null 2023-07-29T00:40:01.128 DEBUG:boto:Method: GET 2023-07-29T00:40:01.129 DEBUG:boto:Path: /hiiqos-59/obj115?versionId=null 2023-07-29T00:40:01.129 DEBUG:boto:Data: 2023-07-29T00:40:01.129 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.130 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:01.130 DEBUG:boto:Port: 8000 2023-07-29T00:40:01.131 DEBUG:boto:Params: {} 2023-07-29T00:40:01.131 DEBUG:boto:Token: None 2023-07-29T00:40:01.131 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj115?versionId=null 2023-07-29T00:40:01.132 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Fw0QnNDnmgqFceH9S0DvWQRaU1w= 2023-07-29T00:40:01.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Fw0QnNDnmgqFceH9S0DvWQRaU1w=', 'Content-Length': '0'} 2023-07-29T00:40:01.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013817bf865de54fd-0064c45fe1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.135 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:01.135 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:01.135 DEBUG:boto:path=/hiiqos-59/?prefix=obj115 2023-07-29T00:40:01.136 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj115 2023-07-29T00:40:01.136 DEBUG:boto:Method: GET 2023-07-29T00:40:01.136 DEBUG:boto:Path: /hiiqos-59/?prefix=obj115 2023-07-29T00:40:01.137 DEBUG:boto:Data: 2023-07-29T00:40:01.137 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.138 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:01.138 DEBUG:boto:Port: 8001 2023-07-29T00:40:01.138 DEBUG:boto:Params: {} 2023-07-29T00:40:01.139 DEBUG:boto:Token: None 2023-07-29T00:40:01.139 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/ 2023-07-29T00:40:01.139 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:F6z8zHPL1D3/2sdDjIeuqnNvTns= 2023-07-29T00:40:01.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:F6z8zHPL1D3/2sdDjIeuqnNvTns=', 'Content-Length': '0'} 2023-07-29T00:40:01.142 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064ca54efabf4cb5b-0064c45fe1-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.143 DEBUG:boto:b'hiiqos-59obj1151000falseobj1152023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:01.143 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:01.144 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:01.144 DEBUG:boto:path=/hiiqos-59/?prefix=obj115 2023-07-29T00:40:01.144 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj115 2023-07-29T00:40:01.145 DEBUG:boto:Method: GET 2023-07-29T00:40:01.145 DEBUG:boto:Path: /hiiqos-59/?prefix=obj115 2023-07-29T00:40:01.145 DEBUG:boto:Data: 2023-07-29T00:40:01.146 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.146 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:01.146 DEBUG:boto:Port: 8000 2023-07-29T00:40:01.147 DEBUG:boto:Params: {} 2023-07-29T00:40:01.147 DEBUG:boto:Token: None 2023-07-29T00:40:01.148 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/ 2023-07-29T00:40:01.148 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:F6z8zHPL1D3/2sdDjIeuqnNvTns= 2023-07-29T00:40:01.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:F6z8zHPL1D3/2sdDjIeuqnNvTns=', 'Content-Length': '0'} 2023-07-29T00:40:01.152 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049f0a2c202d9a484-0064c45fe1-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.153 DEBUG:boto:b'hiiqos-59obj1151000falseobj1152023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:01.153 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:40:01.154 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:40:01.154 DEBUG:boto:Method: HEAD 2023-07-29T00:40:01.154 DEBUG:boto:Path: /hiiqos-59/obj115 2023-07-29T00:40:01.155 DEBUG:boto:Data: 2023-07-29T00:40:01.155 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.155 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:01.156 DEBUG:boto:Port: 8001 2023-07-29T00:40:01.156 DEBUG:boto:Params: {} 2023-07-29T00:40:01.157 DEBUG:boto:Token: None 2023-07-29T00:40:01.157 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj115 2023-07-29T00:40:01.157 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RKDxE90vdRacevlxcJHWTqAm9Zw= 2023-07-29T00:40:01.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RKDxE90vdRacevlxcJHWTqAm9Zw=', 'Content-Length': '0'} 2023-07-29T00:40:01.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4452682beb59517-0064c45fe1-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.160 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:40:01.161 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:40:01.161 DEBUG:boto:Method: HEAD 2023-07-29T00:40:01.161 DEBUG:boto:Path: /hiiqos-59/obj115 2023-07-29T00:40:01.162 DEBUG:boto:Data: 2023-07-29T00:40:01.162 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.162 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:01.163 DEBUG:boto:Port: 8000 2023-07-29T00:40:01.163 DEBUG:boto:Params: {} 2023-07-29T00:40:01.163 DEBUG:boto:Token: None 2023-07-29T00:40:01.164 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj115 2023-07-29T00:40:01.164 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RKDxE90vdRacevlxcJHWTqAm9Zw= 2023-07-29T00:40:01.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RKDxE90vdRacevlxcJHWTqAm9Zw=', 'Content-Length': '0'} 2023-07-29T00:40:01.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff2af9c8c041e6f9-0064c45fe1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-07-29T00:40:01.167 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:40:01.168 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:40:01.168 DEBUG:boto:Method: GET 2023-07-29T00:40:01.168 DEBUG:boto:Path: /hiiqos-59/obj115 2023-07-29T00:40:01.169 DEBUG:boto:Data: 2023-07-29T00:40:01.169 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.170 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:01.170 DEBUG:boto:Port: 8001 2023-07-29T00:40:01.170 DEBUG:boto:Params: {} 2023-07-29T00:40:01.171 DEBUG:boto:Token: None 2023-07-29T00:40:01.171 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj115 2023-07-29T00:40:01.172 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KJ6wxM3znrFkOHbbzRr5m7tMkUQ= 2023-07-29T00:40:01.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KJ6wxM3znrFkOHbbzRr5m7tMkUQ=', 'Content-Length': '0'} 2023-07-29T00:40:01.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000db1f81d876920b9-0064c45fe1-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.174 DEBUG:boto:path=/hiiqos-59/obj115 2023-07-29T00:40:01.175 DEBUG:boto:auth_path=/hiiqos-59/obj115 2023-07-29T00:40:01.175 DEBUG:boto:Method: GET 2023-07-29T00:40:01.176 DEBUG:boto:Path: /hiiqos-59/obj115 2023-07-29T00:40:01.176 DEBUG:boto:Data: 2023-07-29T00:40:01.176 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.177 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:01.177 DEBUG:boto:Port: 8000 2023-07-29T00:40:01.177 DEBUG:boto:Params: {} 2023-07-29T00:40:01.178 DEBUG:boto:Token: None 2023-07-29T00:40:01.178 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj115 2023-07-29T00:40:01.179 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KJ6wxM3znrFkOHbbzRr5m7tMkUQ= 2023-07-29T00:40:01.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KJ6wxM3znrFkOHbbzRr5m7tMkUQ=', 'Content-Length': '0'} 2023-07-29T00:40:01.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5fd209f02df8269-0064c45fe1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.224 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:01.225 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:01.225 DEBUG:boto:path=/hiiqos-59/?prefix=obj115 2023-07-29T00:40:01.226 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj115 2023-07-29T00:40:01.226 DEBUG:boto:Method: GET 2023-07-29T00:40:01.226 DEBUG:boto:Path: /hiiqos-59/?prefix=obj115 2023-07-29T00:40:01.227 DEBUG:boto:Data: 2023-07-29T00:40:01.227 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.227 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:01.228 DEBUG:boto:Port: 8001 2023-07-29T00:40:01.228 DEBUG:boto:Params: {} 2023-07-29T00:40:01.229 DEBUG:boto:Token: None 2023-07-29T00:40:01.229 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/ 2023-07-29T00:40:01.229 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:F6z8zHPL1D3/2sdDjIeuqnNvTns= 2023-07-29T00:40:01.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:F6z8zHPL1D3/2sdDjIeuqnNvTns=', 'Content-Length': '0'} 2023-07-29T00:40:01.232 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f32c894d95262478-0064c45fe1-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.233 DEBUG:boto:b'hiiqos-59obj1151000falseobj1152023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:01.233 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:01.234 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:01.234 DEBUG:boto:path=/hiiqos-59/?prefix=obj115 2023-07-29T00:40:01.234 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj115 2023-07-29T00:40:01.235 DEBUG:boto:Method: GET 2023-07-29T00:40:01.235 DEBUG:boto:Path: /hiiqos-59/?prefix=obj115 2023-07-29T00:40:01.235 DEBUG:boto:Data: 2023-07-29T00:40:01.236 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.236 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:01.237 DEBUG:boto:Port: 8000 2023-07-29T00:40:01.237 DEBUG:boto:Params: {} 2023-07-29T00:40:01.237 DEBUG:boto:Token: None 2023-07-29T00:40:01.238 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/ 2023-07-29T00:40:01.238 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:F6z8zHPL1D3/2sdDjIeuqnNvTns= 2023-07-29T00:40:01.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:F6z8zHPL1D3/2sdDjIeuqnNvTns=', 'Content-Length': '0'} 2023-07-29T00:40:01.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5f2327007d4ac31-0064c45fe1-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.243 DEBUG:boto:b'hiiqos-59obj1151000falseobj1152023-07-29T00:33:06.941Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:01.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-07-29T00:40:01.244 DEBUG:boto:path=/hiiqos-59/obj116 2023-07-29T00:40:01.244 DEBUG:boto:auth_path=/hiiqos-59/obj116 2023-07-29T00:40:01.244 DEBUG:boto:path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:40:01.245 DEBUG:boto:auth_path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:40:01.245 DEBUG:boto:Method: GET 2023-07-29T00:40:01.245 DEBUG:boto:Path: /hiiqos-59/obj116?versionId=null 2023-07-29T00:40:01.246 DEBUG:boto:Data: 2023-07-29T00:40:01.246 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.246 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:01.247 DEBUG:boto:Port: 8001 2023-07-29T00:40:01.247 DEBUG:boto:Params: {} 2023-07-29T00:40:01.248 DEBUG:boto:Token: None 2023-07-29T00:40:01.248 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj116?versionId=null 2023-07-29T00:40:01.248 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hK/AwJueDrxsIF5b9C10QSxrb64= 2023-07-29T00:40:01.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hK/AwJueDrxsIF5b9C10QSxrb64=', 'Content-Length': '0'} 2023-07-29T00:40:01.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d794091291728476-0064c45fe1-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.292 DEBUG:boto:path=/hiiqos-59/obj116 2023-07-29T00:40:01.293 DEBUG:boto:auth_path=/hiiqos-59/obj116 2023-07-29T00:40:01.293 DEBUG:boto:path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:40:01.293 DEBUG:boto:auth_path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:40:01.294 DEBUG:boto:Method: GET 2023-07-29T00:40:01.294 DEBUG:boto:Path: /hiiqos-59/obj116?versionId=null 2023-07-29T00:40:01.295 DEBUG:boto:Data: 2023-07-29T00:40:01.295 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.295 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:01.296 DEBUG:boto:Port: 8000 2023-07-29T00:40:01.296 DEBUG:boto:Params: {} 2023-07-29T00:40:01.296 DEBUG:boto:Token: None 2023-07-29T00:40:01.297 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj116?versionId=null 2023-07-29T00:40:01.297 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hK/AwJueDrxsIF5b9C10QSxrb64= 2023-07-29T00:40:01.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hK/AwJueDrxsIF5b9C10QSxrb64=', 'Content-Length': '0'} 2023-07-29T00:40:01.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049835bde77fcbbd1-0064c45fe1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.300 DEBUG:boto:path=/hiiqos-59/obj116 2023-07-29T00:40:01.301 DEBUG:boto:auth_path=/hiiqos-59/obj116 2023-07-29T00:40:01.301 DEBUG:boto:path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:40:01.301 DEBUG:boto:auth_path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:40:01.302 DEBUG:boto:Method: HEAD 2023-07-29T00:40:01.302 DEBUG:boto:Path: /hiiqos-59/obj116?versionId=null 2023-07-29T00:40:01.302 DEBUG:boto:Data: 2023-07-29T00:40:01.303 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.303 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:01.303 DEBUG:boto:Port: 8001 2023-07-29T00:40:01.304 DEBUG:boto:Params: {} 2023-07-29T00:40:01.304 DEBUG:boto:Token: None 2023-07-29T00:40:01.304 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj116?versionId=null 2023-07-29T00:40:01.305 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WAsB7bpQ0PoYELyqNLhO5wxBU08= 2023-07-29T00:40:01.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WAsB7bpQ0PoYELyqNLhO5wxBU08=', 'Content-Length': '0'} 2023-07-29T00:40:01.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032b3d7b5af5ef244-0064c45fe1-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.308 DEBUG:boto:path=/hiiqos-59/obj116 2023-07-29T00:40:01.308 DEBUG:boto:auth_path=/hiiqos-59/obj116 2023-07-29T00:40:01.308 DEBUG:boto:path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:40:01.309 DEBUG:boto:auth_path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:40:01.309 DEBUG:boto:Method: HEAD 2023-07-29T00:40:01.309 DEBUG:boto:Path: /hiiqos-59/obj116?versionId=null 2023-07-29T00:40:01.310 DEBUG:boto:Data: 2023-07-29T00:40:01.310 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.310 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:01.311 DEBUG:boto:Port: 8000 2023-07-29T00:40:01.311 DEBUG:boto:Params: {} 2023-07-29T00:40:01.311 DEBUG:boto:Token: None 2023-07-29T00:40:01.312 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj116?versionId=null 2023-07-29T00:40:01.312 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WAsB7bpQ0PoYELyqNLhO5wxBU08= 2023-07-29T00:40:01.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WAsB7bpQ0PoYELyqNLhO5wxBU08=', 'Content-Length': '0'} 2023-07-29T00:40:01.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6e57717ca487dbb-0064c45fe1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-07-29T00:40:01.315 DEBUG:boto:path=/hiiqos-59/obj116 2023-07-29T00:40:01.315 DEBUG:boto:auth_path=/hiiqos-59/obj116 2023-07-29T00:40:01.316 DEBUG:boto:path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:40:01.316 DEBUG:boto:auth_path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:40:01.316 DEBUG:boto:Method: GET 2023-07-29T00:40:01.317 DEBUG:boto:Path: /hiiqos-59/obj116?versionId=null 2023-07-29T00:40:01.317 DEBUG:boto:Data: 2023-07-29T00:40:01.318 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.318 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:01.318 DEBUG:boto:Port: 8001 2023-07-29T00:40:01.319 DEBUG:boto:Params: {} 2023-07-29T00:40:01.319 DEBUG:boto:Token: None 2023-07-29T00:40:01.319 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj116?versionId=null 2023-07-29T00:40:01.320 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hK/AwJueDrxsIF5b9C10QSxrb64= 2023-07-29T00:40:01.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hK/AwJueDrxsIF5b9C10QSxrb64=', 'Content-Length': '0'} 2023-07-29T00:40:01.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023a4a6200e9b0d1b-0064c45fe1-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.323 DEBUG:boto:path=/hiiqos-59/obj116 2023-07-29T00:40:01.323 DEBUG:boto:auth_path=/hiiqos-59/obj116 2023-07-29T00:40:01.323 DEBUG:boto:path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:40:01.324 DEBUG:boto:auth_path=/hiiqos-59/obj116?versionId=null 2023-07-29T00:40:01.324 DEBUG:boto:Method: GET 2023-07-29T00:40:01.324 DEBUG:boto:Path: /hiiqos-59/obj116?versionId=null 2023-07-29T00:40:01.325 DEBUG:boto:Data: 2023-07-29T00:40:01.325 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.325 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:01.326 DEBUG:boto:Port: 8000 2023-07-29T00:40:01.326 DEBUG:boto:Params: {} 2023-07-29T00:40:01.327 DEBUG:boto:Token: None 2023-07-29T00:40:01.327 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj116?versionId=null 2023-07-29T00:40:01.327 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hK/AwJueDrxsIF5b9C10QSxrb64= 2023-07-29T00:40:01.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hK/AwJueDrxsIF5b9C10QSxrb64=', 'Content-Length': '0'} 2023-07-29T00:40:01.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1486fbb702053cc-0064c45fe1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.330 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:01.331 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:01.331 DEBUG:boto:path=/hiiqos-59/?prefix=obj116 2023-07-29T00:40:01.331 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj116 2023-07-29T00:40:01.332 DEBUG:boto:Method: GET 2023-07-29T00:40:01.332 DEBUG:boto:Path: /hiiqos-59/?prefix=obj116 2023-07-29T00:40:01.333 DEBUG:boto:Data: 2023-07-29T00:40:01.333 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.333 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:01.334 DEBUG:boto:Port: 8001 2023-07-29T00:40:01.334 DEBUG:boto:Params: {} 2023-07-29T00:40:01.334 DEBUG:boto:Token: None 2023-07-29T00:40:01.335 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/ 2023-07-29T00:40:01.335 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:F6z8zHPL1D3/2sdDjIeuqnNvTns= 2023-07-29T00:40:01.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:F6z8zHPL1D3/2sdDjIeuqnNvTns=', 'Content-Length': '0'} 2023-07-29T00:40:01.338 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1c9c5a8a47b1fc2-0064c45fe1-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.338 DEBUG:boto:b'hiiqos-59obj1161000falseobj1162023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:01.339 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:01.339 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:01.340 DEBUG:boto:path=/hiiqos-59/?prefix=obj116 2023-07-29T00:40:01.340 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj116 2023-07-29T00:40:01.340 DEBUG:boto:Method: GET 2023-07-29T00:40:01.341 DEBUG:boto:Path: /hiiqos-59/?prefix=obj116 2023-07-29T00:40:01.341 DEBUG:boto:Data: 2023-07-29T00:40:01.341 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.342 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:01.342 DEBUG:boto:Port: 8000 2023-07-29T00:40:01.343 DEBUG:boto:Params: {} 2023-07-29T00:40:01.343 DEBUG:boto:Token: None 2023-07-29T00:40:01.343 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/ 2023-07-29T00:40:01.344 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:F6z8zHPL1D3/2sdDjIeuqnNvTns= 2023-07-29T00:40:01.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:F6z8zHPL1D3/2sdDjIeuqnNvTns=', 'Content-Length': '0'} 2023-07-29T00:40:01.348 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e39d089aa6cf47c4-0064c45fe1-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.348 DEBUG:boto:b'hiiqos-59obj1161000falseobj1162023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:01.349 DEBUG:boto:path=/hiiqos-59/obj116 2023-07-29T00:40:01.349 DEBUG:boto:auth_path=/hiiqos-59/obj116 2023-07-29T00:40:01.350 DEBUG:boto:Method: HEAD 2023-07-29T00:40:01.350 DEBUG:boto:Path: /hiiqos-59/obj116 2023-07-29T00:40:01.350 DEBUG:boto:Data: 2023-07-29T00:40:01.351 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.351 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:01.351 DEBUG:boto:Port: 8001 2023-07-29T00:40:01.352 DEBUG:boto:Params: {} 2023-07-29T00:40:01.352 DEBUG:boto:Token: None 2023-07-29T00:40:01.352 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj116 2023-07-29T00:40:01.353 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:a7QFRFH9jZ3EGmyw1OSEaFFO4vk= 2023-07-29T00:40:01.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:a7QFRFH9jZ3EGmyw1OSEaFFO4vk=', 'Content-Length': '0'} 2023-07-29T00:40:01.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbcb8041eb77a6e5-0064c45fe1-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.356 DEBUG:boto:path=/hiiqos-59/obj116 2023-07-29T00:40:01.356 DEBUG:boto:auth_path=/hiiqos-59/obj116 2023-07-29T00:40:01.356 DEBUG:boto:Method: HEAD 2023-07-29T00:40:01.357 DEBUG:boto:Path: /hiiqos-59/obj116 2023-07-29T00:40:01.357 DEBUG:boto:Data: 2023-07-29T00:40:01.357 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.358 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:01.358 DEBUG:boto:Port: 8000 2023-07-29T00:40:01.358 DEBUG:boto:Params: {} 2023-07-29T00:40:01.359 DEBUG:boto:Token: None 2023-07-29T00:40:01.359 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj116 2023-07-29T00:40:01.360 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:a7QFRFH9jZ3EGmyw1OSEaFFO4vk= 2023-07-29T00:40:01.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:a7QFRFH9jZ3EGmyw1OSEaFFO4vk=', 'Content-Length': '0'} 2023-07-29T00:40:01.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003508cf7756b6d8bd-0064c45fe1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-07-29T00:40:01.362 DEBUG:boto:path=/hiiqos-59/obj116 2023-07-29T00:40:01.363 DEBUG:boto:auth_path=/hiiqos-59/obj116 2023-07-29T00:40:01.363 DEBUG:boto:Method: GET 2023-07-29T00:40:01.364 DEBUG:boto:Path: /hiiqos-59/obj116 2023-07-29T00:40:01.364 DEBUG:boto:Data: 2023-07-29T00:40:01.364 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.365 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:01.365 DEBUG:boto:Port: 8001 2023-07-29T00:40:01.365 DEBUG:boto:Params: {} 2023-07-29T00:40:01.366 DEBUG:boto:Token: None 2023-07-29T00:40:01.366 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj116 2023-07-29T00:40:01.367 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aoAaSCZAVxpfFjc8NwqfwWUKsy8= 2023-07-29T00:40:01.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aoAaSCZAVxpfFjc8NwqfwWUKsy8=', 'Content-Length': '0'} 2023-07-29T00:40:01.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000996a35c03237740d-0064c45fe1-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.370 DEBUG:boto:path=/hiiqos-59/obj116 2023-07-29T00:40:01.370 DEBUG:boto:auth_path=/hiiqos-59/obj116 2023-07-29T00:40:01.370 DEBUG:boto:Method: GET 2023-07-29T00:40:01.371 DEBUG:boto:Path: /hiiqos-59/obj116 2023-07-29T00:40:01.371 DEBUG:boto:Data: 2023-07-29T00:40:01.371 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.372 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:01.372 DEBUG:boto:Port: 8000 2023-07-29T00:40:01.372 DEBUG:boto:Params: {} 2023-07-29T00:40:01.373 DEBUG:boto:Token: None 2023-07-29T00:40:01.373 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj116 2023-07-29T00:40:01.374 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aoAaSCZAVxpfFjc8NwqfwWUKsy8= 2023-07-29T00:40:01.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aoAaSCZAVxpfFjc8NwqfwWUKsy8=', 'Content-Length': '0'} 2023-07-29T00:40:01.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd1fbbb7283e9cd9-0064c45fe1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.377 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:01.377 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:01.377 DEBUG:boto:path=/hiiqos-59/?prefix=obj116 2023-07-29T00:40:01.378 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj116 2023-07-29T00:40:01.378 DEBUG:boto:Method: GET 2023-07-29T00:40:01.379 DEBUG:boto:Path: /hiiqos-59/?prefix=obj116 2023-07-29T00:40:01.379 DEBUG:boto:Data: 2023-07-29T00:40:01.379 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.380 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:01.380 DEBUG:boto:Port: 8001 2023-07-29T00:40:01.380 DEBUG:boto:Params: {} 2023-07-29T00:40:01.381 DEBUG:boto:Token: None 2023-07-29T00:40:01.381 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/ 2023-07-29T00:40:01.382 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:F6z8zHPL1D3/2sdDjIeuqnNvTns= 2023-07-29T00:40:01.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:F6z8zHPL1D3/2sdDjIeuqnNvTns=', 'Content-Length': '0'} 2023-07-29T00:40:01.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9ac881bd5892c45-0064c45fe1-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.385 DEBUG:boto:b'hiiqos-59obj1161000falseobj1162023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:01.386 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:01.386 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:01.386 DEBUG:boto:path=/hiiqos-59/?prefix=obj116 2023-07-29T00:40:01.387 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj116 2023-07-29T00:40:01.387 DEBUG:boto:Method: GET 2023-07-29T00:40:01.388 DEBUG:boto:Path: /hiiqos-59/?prefix=obj116 2023-07-29T00:40:01.388 DEBUG:boto:Data: 2023-07-29T00:40:01.388 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.389 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:01.389 DEBUG:boto:Port: 8000 2023-07-29T00:40:01.389 DEBUG:boto:Params: {} 2023-07-29T00:40:01.390 DEBUG:boto:Token: None 2023-07-29T00:40:01.390 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/ 2023-07-29T00:40:01.391 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:F6z8zHPL1D3/2sdDjIeuqnNvTns= 2023-07-29T00:40:01.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:F6z8zHPL1D3/2sdDjIeuqnNvTns=', 'Content-Length': '0'} 2023-07-29T00:40:01.395 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000815a8a74b907f366-0064c45fe1-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.395 DEBUG:boto:b'hiiqos-59obj1161000falseobj1162023-07-29T00:35:01.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:01.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-07-29T00:40:01.396 DEBUG:boto:path=/hiiqos-59/obj124 2023-07-29T00:40:01.397 DEBUG:boto:auth_path=/hiiqos-59/obj124 2023-07-29T00:40:01.397 DEBUG:boto:path=/hiiqos-59/obj124?versionId=null 2023-07-29T00:40:01.397 DEBUG:boto:auth_path=/hiiqos-59/obj124?versionId=null 2023-07-29T00:40:01.398 DEBUG:boto:Method: GET 2023-07-29T00:40:01.398 DEBUG:boto:Path: /hiiqos-59/obj124?versionId=null 2023-07-29T00:40:01.398 DEBUG:boto:Data: 2023-07-29T00:40:01.399 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.399 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:01.399 DEBUG:boto:Port: 8001 2023-07-29T00:40:01.400 DEBUG:boto:Params: {} 2023-07-29T00:40:01.400 DEBUG:boto:Token: None 2023-07-29T00:40:01.401 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj124?versionId=null 2023-07-29T00:40:01.401 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1SGsnFPtFxvOjKBG21yu+N8GFlw= 2023-07-29T00:40:01.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1SGsnFPtFxvOjKBG21yu+N8GFlw=', 'Content-Length': '0'} 2023-07-29T00:40:01.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010884db8872c3370-0064c45fe1-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.444 DEBUG:boto:path=/hiiqos-59/obj124 2023-07-29T00:40:01.445 DEBUG:boto:auth_path=/hiiqos-59/obj124 2023-07-29T00:40:01.445 DEBUG:boto:path=/hiiqos-59/obj124?versionId=null 2023-07-29T00:40:01.446 DEBUG:boto:auth_path=/hiiqos-59/obj124?versionId=null 2023-07-29T00:40:01.446 DEBUG:boto:Method: GET 2023-07-29T00:40:01.446 DEBUG:boto:Path: /hiiqos-59/obj124?versionId=null 2023-07-29T00:40:01.447 DEBUG:boto:Data: 2023-07-29T00:40:01.447 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.447 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:01.448 DEBUG:boto:Port: 8000 2023-07-29T00:40:01.448 DEBUG:boto:Params: {} 2023-07-29T00:40:01.449 DEBUG:boto:Token: None 2023-07-29T00:40:01.449 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj124?versionId=null 2023-07-29T00:40:01.449 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1SGsnFPtFxvOjKBG21yu+N8GFlw= 2023-07-29T00:40:01.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1SGsnFPtFxvOjKBG21yu+N8GFlw=', 'Content-Length': '0'} 2023-07-29T00:40:01.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000864d5fbb6a5cf400-0064c45fe1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.452 DEBUG:boto:path=/hiiqos-59/obj124 2023-07-29T00:40:01.453 DEBUG:boto:auth_path=/hiiqos-59/obj124 2023-07-29T00:40:01.453 DEBUG:boto:path=/hiiqos-59/obj124?versionId=null 2023-07-29T00:40:01.453 DEBUG:boto:auth_path=/hiiqos-59/obj124?versionId=null 2023-07-29T00:40:01.454 DEBUG:boto:Method: HEAD 2023-07-29T00:40:01.454 DEBUG:boto:Path: /hiiqos-59/obj124?versionId=null 2023-07-29T00:40:01.454 DEBUG:boto:Data: 2023-07-29T00:40:01.455 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.455 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:01.455 DEBUG:boto:Port: 8001 2023-07-29T00:40:01.456 DEBUG:boto:Params: {} 2023-07-29T00:40:01.456 DEBUG:boto:Token: None 2023-07-29T00:40:01.456 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj124?versionId=null 2023-07-29T00:40:01.457 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jFzA81Ny3t66Pb+LEfgLKOG5dWU= 2023-07-29T00:40:01.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jFzA81Ny3t66Pb+LEfgLKOG5dWU=', 'Content-Length': '0'} 2023-07-29T00:40:01.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daa790ce738cf7f6-0064c45fe1-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.460 DEBUG:boto:path=/hiiqos-59/obj124 2023-07-29T00:40:01.460 DEBUG:boto:auth_path=/hiiqos-59/obj124 2023-07-29T00:40:01.460 DEBUG:boto:path=/hiiqos-59/obj124?versionId=null 2023-07-29T00:40:01.461 DEBUG:boto:auth_path=/hiiqos-59/obj124?versionId=null 2023-07-29T00:40:01.461 DEBUG:boto:Method: HEAD 2023-07-29T00:40:01.461 DEBUG:boto:Path: /hiiqos-59/obj124?versionId=null 2023-07-29T00:40:01.462 DEBUG:boto:Data: 2023-07-29T00:40:01.462 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.462 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:01.463 DEBUG:boto:Port: 8000 2023-07-29T00:40:01.463 DEBUG:boto:Params: {} 2023-07-29T00:40:01.463 DEBUG:boto:Token: None 2023-07-29T00:40:01.464 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj124?versionId=null 2023-07-29T00:40:01.464 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jFzA81Ny3t66Pb+LEfgLKOG5dWU= 2023-07-29T00:40:01.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jFzA81Ny3t66Pb+LEfgLKOG5dWU=', 'Content-Length': '0'} 2023-07-29T00:40:01.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034a30d67a6fe8fb4-0064c45fe1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-07-29T00:40:01.468 DEBUG:boto:path=/hiiqos-59/obj124 2023-07-29T00:40:01.468 DEBUG:boto:auth_path=/hiiqos-59/obj124 2023-07-29T00:40:01.469 DEBUG:boto:path=/hiiqos-59/obj124?versionId=null 2023-07-29T00:40:01.469 DEBUG:boto:auth_path=/hiiqos-59/obj124?versionId=null 2023-07-29T00:40:01.469 DEBUG:boto:Method: GET 2023-07-29T00:40:01.470 DEBUG:boto:Path: /hiiqos-59/obj124?versionId=null 2023-07-29T00:40:01.470 DEBUG:boto:Data: 2023-07-29T00:40:01.471 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.471 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:01.471 DEBUG:boto:Port: 8001 2023-07-29T00:40:01.472 DEBUG:boto:Params: {} 2023-07-29T00:40:01.472 DEBUG:boto:Token: None 2023-07-29T00:40:01.473 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj124?versionId=null 2023-07-29T00:40:01.473 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1SGsnFPtFxvOjKBG21yu+N8GFlw= 2023-07-29T00:40:01.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1SGsnFPtFxvOjKBG21yu+N8GFlw=', 'Content-Length': '0'} 2023-07-29T00:40:01.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094bd8a567c96a49d-0064c45fe1-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.476 DEBUG:boto:path=/hiiqos-59/obj124 2023-07-29T00:40:01.477 DEBUG:boto:auth_path=/hiiqos-59/obj124 2023-07-29T00:40:01.478 DEBUG:boto:path=/hiiqos-59/obj124?versionId=null 2023-07-29T00:40:01.478 DEBUG:boto:auth_path=/hiiqos-59/obj124?versionId=null 2023-07-29T00:40:01.479 DEBUG:boto:Method: GET 2023-07-29T00:40:01.479 DEBUG:boto:Path: /hiiqos-59/obj124?versionId=null 2023-07-29T00:40:01.479 DEBUG:boto:Data: 2023-07-29T00:40:01.480 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.480 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:01.481 DEBUG:boto:Port: 8000 2023-07-29T00:40:01.481 DEBUG:boto:Params: {} 2023-07-29T00:40:01.482 DEBUG:boto:Token: None 2023-07-29T00:40:01.482 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj124?versionId=null 2023-07-29T00:40:01.483 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1SGsnFPtFxvOjKBG21yu+N8GFlw= 2023-07-29T00:40:01.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1SGsnFPtFxvOjKBG21yu+N8GFlw=', 'Content-Length': '0'} 2023-07-29T00:40:01.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093d84de86c66e1e6-0064c45fe1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.486 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:01.486 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:01.487 DEBUG:boto:path=/hiiqos-59/?prefix=obj124 2023-07-29T00:40:01.487 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj124 2023-07-29T00:40:01.487 DEBUG:boto:Method: GET 2023-07-29T00:40:01.488 DEBUG:boto:Path: /hiiqos-59/?prefix=obj124 2023-07-29T00:40:01.488 DEBUG:boto:Data: 2023-07-29T00:40:01.489 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.489 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:01.490 DEBUG:boto:Port: 8001 2023-07-29T00:40:01.490 DEBUG:boto:Params: {} 2023-07-29T00:40:01.490 DEBUG:boto:Token: None 2023-07-29T00:40:01.491 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/ 2023-07-29T00:40:01.491 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:F6z8zHPL1D3/2sdDjIeuqnNvTns= 2023-07-29T00:40:01.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:F6z8zHPL1D3/2sdDjIeuqnNvTns=', 'Content-Length': '0'} 2023-07-29T00:40:01.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e74c44c5e340f29f-0064c45fe1-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.495 DEBUG:boto:b'hiiqos-59obj1241000falseobj1242023-07-29T00:35:31.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:01.495 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:01.496 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:01.496 DEBUG:boto:path=/hiiqos-59/?prefix=obj124 2023-07-29T00:40:01.496 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj124 2023-07-29T00:40:01.497 DEBUG:boto:Method: GET 2023-07-29T00:40:01.497 DEBUG:boto:Path: /hiiqos-59/?prefix=obj124 2023-07-29T00:40:01.498 DEBUG:boto:Data: 2023-07-29T00:40:01.498 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.498 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:01.499 DEBUG:boto:Port: 8000 2023-07-29T00:40:01.499 DEBUG:boto:Params: {} 2023-07-29T00:40:01.500 DEBUG:boto:Token: None 2023-07-29T00:40:01.500 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/ 2023-07-29T00:40:01.501 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:F6z8zHPL1D3/2sdDjIeuqnNvTns= 2023-07-29T00:40:01.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:F6z8zHPL1D3/2sdDjIeuqnNvTns=', 'Content-Length': '0'} 2023-07-29T00:40:01.505 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e70e54539f4ed9c6-0064c45fe1-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.552 DEBUG:boto:b'hiiqos-59obj1241000falseobj1242023-07-29T00:35:31.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:01.554 DEBUG:boto:path=/hiiqos-59/obj124 2023-07-29T00:40:01.555 DEBUG:boto:auth_path=/hiiqos-59/obj124 2023-07-29T00:40:01.555 DEBUG:boto:Method: HEAD 2023-07-29T00:40:01.556 DEBUG:boto:Path: /hiiqos-59/obj124 2023-07-29T00:40:01.556 DEBUG:boto:Data: 2023-07-29T00:40:01.557 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.557 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:01.557 DEBUG:boto:Port: 8001 2023-07-29T00:40:01.558 DEBUG:boto:Params: {} 2023-07-29T00:40:01.558 DEBUG:boto:Token: None 2023-07-29T00:40:01.558 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj124 2023-07-29T00:40:01.559 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FAQkqKtUR2vy1x0hLDgbvNSl5Mw= 2023-07-29T00:40:01.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FAQkqKtUR2vy1x0hLDgbvNSl5Mw=', 'Content-Length': '0'} 2023-07-29T00:40:01.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000931de67bd398764a-0064c45fe1-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.562 DEBUG:boto:path=/hiiqos-59/obj124 2023-07-29T00:40:01.562 DEBUG:boto:auth_path=/hiiqos-59/obj124 2023-07-29T00:40:01.563 DEBUG:boto:Method: HEAD 2023-07-29T00:40:01.563 DEBUG:boto:Path: /hiiqos-59/obj124 2023-07-29T00:40:01.563 DEBUG:boto:Data: 2023-07-29T00:40:01.564 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.564 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:01.565 DEBUG:boto:Port: 8000 2023-07-29T00:40:01.565 DEBUG:boto:Params: {} 2023-07-29T00:40:01.565 DEBUG:boto:Token: None 2023-07-29T00:40:01.566 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj124 2023-07-29T00:40:01.566 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FAQkqKtUR2vy1x0hLDgbvNSl5Mw= 2023-07-29T00:40:01.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FAQkqKtUR2vy1x0hLDgbvNSl5Mw=', 'Content-Length': '0'} 2023-07-29T00:40:01.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d75fb6d03c3af72d-0064c45fe1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-07-29T00:40:01.569 DEBUG:boto:path=/hiiqos-59/obj124 2023-07-29T00:40:01.570 DEBUG:boto:auth_path=/hiiqos-59/obj124 2023-07-29T00:40:01.570 DEBUG:boto:Method: GET 2023-07-29T00:40:01.571 DEBUG:boto:Path: /hiiqos-59/obj124 2023-07-29T00:40:01.571 DEBUG:boto:Data: 2023-07-29T00:40:01.571 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.572 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:01.573 DEBUG:boto:Port: 8001 2023-07-29T00:40:01.573 DEBUG:boto:Params: {} 2023-07-29T00:40:01.574 DEBUG:boto:Token: None 2023-07-29T00:40:01.574 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj124 2023-07-29T00:40:01.575 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rRQdYi6s0zAfRaAayeGXo+RqjFk= 2023-07-29T00:40:01.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rRQdYi6s0zAfRaAayeGXo+RqjFk=', 'Content-Length': '0'} 2023-07-29T00:40:01.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005271da0e177f8d35-0064c45fe1-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.577 DEBUG:boto:path=/hiiqos-59/obj124 2023-07-29T00:40:01.578 DEBUG:boto:auth_path=/hiiqos-59/obj124 2023-07-29T00:40:01.578 DEBUG:boto:Method: GET 2023-07-29T00:40:01.579 DEBUG:boto:Path: /hiiqos-59/obj124 2023-07-29T00:40:01.579 DEBUG:boto:Data: 2023-07-29T00:40:01.579 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.580 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:01.580 DEBUG:boto:Port: 8000 2023-07-29T00:40:01.581 DEBUG:boto:Params: {} 2023-07-29T00:40:01.581 DEBUG:boto:Token: None 2023-07-29T00:40:01.582 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj124 2023-07-29T00:40:01.582 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rRQdYi6s0zAfRaAayeGXo+RqjFk= 2023-07-29T00:40:01.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rRQdYi6s0zAfRaAayeGXo+RqjFk=', 'Content-Length': '0'} 2023-07-29T00:40:01.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000da58c0aa8842a2e-0064c45fe1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.585 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:01.586 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:01.586 DEBUG:boto:path=/hiiqos-59/?prefix=obj124 2023-07-29T00:40:01.586 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj124 2023-07-29T00:40:01.587 DEBUG:boto:Method: GET 2023-07-29T00:40:01.587 DEBUG:boto:Path: /hiiqos-59/?prefix=obj124 2023-07-29T00:40:01.588 DEBUG:boto:Data: 2023-07-29T00:40:01.588 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.589 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:01.589 DEBUG:boto:Port: 8001 2023-07-29T00:40:01.589 DEBUG:boto:Params: {} 2023-07-29T00:40:01.590 DEBUG:boto:Token: None 2023-07-29T00:40:01.591 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/ 2023-07-29T00:40:01.591 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:F6z8zHPL1D3/2sdDjIeuqnNvTns= 2023-07-29T00:40:01.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:F6z8zHPL1D3/2sdDjIeuqnNvTns=', 'Content-Length': '0'} 2023-07-29T00:40:01.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c04c9d94cacbe029-0064c45fe1-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.636 DEBUG:boto:b'hiiqos-59obj1241000falseobj1242023-07-29T00:35:31.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:01.638 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:01.639 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:01.639 DEBUG:boto:path=/hiiqos-59/?prefix=obj124 2023-07-29T00:40:01.640 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj124 2023-07-29T00:40:01.641 DEBUG:boto:Method: GET 2023-07-29T00:40:01.642 DEBUG:boto:Path: /hiiqos-59/?prefix=obj124 2023-07-29T00:40:01.643 DEBUG:boto:Data: 2023-07-29T00:40:01.644 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.645 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:01.645 DEBUG:boto:Port: 8000 2023-07-29T00:40:01.646 DEBUG:boto:Params: {} 2023-07-29T00:40:01.647 DEBUG:boto:Token: None 2023-07-29T00:40:01.648 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/ 2023-07-29T00:40:01.649 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:F6z8zHPL1D3/2sdDjIeuqnNvTns= 2023-07-29T00:40:01.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:F6z8zHPL1D3/2sdDjIeuqnNvTns=', 'Content-Length': '0'} 2023-07-29T00:40:01.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ea27d86a134cf82-0064c45fe1-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.655 DEBUG:boto:b'hiiqos-59obj1241000falseobj1242023-07-29T00:35:31.901Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:01.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-07-29T00:40:01.657 DEBUG:boto:path=/hiiqos-59/obj145 2023-07-29T00:40:01.658 DEBUG:boto:auth_path=/hiiqos-59/obj145 2023-07-29T00:40:01.658 DEBUG:boto:path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:40:01.659 DEBUG:boto:auth_path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:40:01.660 DEBUG:boto:Method: GET 2023-07-29T00:40:01.661 DEBUG:boto:Path: /hiiqos-59/obj145?versionId=null 2023-07-29T00:40:01.661 DEBUG:boto:Data: 2023-07-29T00:40:01.662 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.663 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:01.664 DEBUG:boto:Port: 8001 2023-07-29T00:40:01.664 DEBUG:boto:Params: {} 2023-07-29T00:40:01.665 DEBUG:boto:Token: None 2023-07-29T00:40:01.666 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj145?versionId=null 2023-07-29T00:40:01.667 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QJfsklTuwLRmpb2uVyL/y1LzneY= 2023-07-29T00:40:01.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QJfsklTuwLRmpb2uVyL/y1LzneY=', 'Content-Length': '0'} 2023-07-29T00:40:01.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f993711c1783d83a-0064c45fe1-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.671 DEBUG:boto:path=/hiiqos-59/obj145 2023-07-29T00:40:01.672 DEBUG:boto:auth_path=/hiiqos-59/obj145 2023-07-29T00:40:01.673 DEBUG:boto:path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:40:01.674 DEBUG:boto:auth_path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:40:01.675 DEBUG:boto:Method: GET 2023-07-29T00:40:01.676 DEBUG:boto:Path: /hiiqos-59/obj145?versionId=null 2023-07-29T00:40:01.677 DEBUG:boto:Data: 2023-07-29T00:40:01.679 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.679 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:01.680 DEBUG:boto:Port: 8000 2023-07-29T00:40:01.680 DEBUG:boto:Params: {} 2023-07-29T00:40:01.681 DEBUG:boto:Token: None 2023-07-29T00:40:01.681 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj145?versionId=null 2023-07-29T00:40:01.682 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QJfsklTuwLRmpb2uVyL/y1LzneY= 2023-07-29T00:40:01.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QJfsklTuwLRmpb2uVyL/y1LzneY=', 'Content-Length': '0'} 2023-07-29T00:40:01.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f879a0311025322b-0064c45fe1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.724 DEBUG:boto:path=/hiiqos-59/obj145 2023-07-29T00:40:01.725 DEBUG:boto:auth_path=/hiiqos-59/obj145 2023-07-29T00:40:01.725 DEBUG:boto:path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:40:01.726 DEBUG:boto:auth_path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:40:01.726 DEBUG:boto:Method: HEAD 2023-07-29T00:40:01.726 DEBUG:boto:Path: /hiiqos-59/obj145?versionId=null 2023-07-29T00:40:01.727 DEBUG:boto:Data: 2023-07-29T00:40:01.727 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.727 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:01.728 DEBUG:boto:Port: 8001 2023-07-29T00:40:01.728 DEBUG:boto:Params: {} 2023-07-29T00:40:01.728 DEBUG:boto:Token: None 2023-07-29T00:40:01.729 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj145?versionId=null 2023-07-29T00:40:01.729 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rX/T974iR3BTXZP5QAYP9Z3zF68= 2023-07-29T00:40:01.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rX/T974iR3BTXZP5QAYP9Z3zF68=', 'Content-Length': '0'} 2023-07-29T00:40:01.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b263dac0a9d8746-0064c45fe1-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.732 DEBUG:boto:path=/hiiqos-59/obj145 2023-07-29T00:40:01.733 DEBUG:boto:auth_path=/hiiqos-59/obj145 2023-07-29T00:40:01.733 DEBUG:boto:path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:40:01.733 DEBUG:boto:auth_path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:40:01.734 DEBUG:boto:Method: HEAD 2023-07-29T00:40:01.734 DEBUG:boto:Path: /hiiqos-59/obj145?versionId=null 2023-07-29T00:40:01.734 DEBUG:boto:Data: 2023-07-29T00:40:01.735 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.735 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:01.735 DEBUG:boto:Port: 8000 2023-07-29T00:40:01.736 DEBUG:boto:Params: {} 2023-07-29T00:40:01.736 DEBUG:boto:Token: None 2023-07-29T00:40:01.736 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj145?versionId=null 2023-07-29T00:40:01.737 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rX/T974iR3BTXZP5QAYP9Z3zF68= 2023-07-29T00:40:01.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rX/T974iR3BTXZP5QAYP9Z3zF68=', 'Content-Length': '0'} 2023-07-29T00:40:01.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000baea2949dad39a4e-0064c45fe1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-07-29T00:40:01.740 DEBUG:boto:path=/hiiqos-59/obj145 2023-07-29T00:40:01.740 DEBUG:boto:auth_path=/hiiqos-59/obj145 2023-07-29T00:40:01.741 DEBUG:boto:path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:40:01.741 DEBUG:boto:auth_path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:40:01.741 DEBUG:boto:Method: GET 2023-07-29T00:40:01.742 DEBUG:boto:Path: /hiiqos-59/obj145?versionId=null 2023-07-29T00:40:01.742 DEBUG:boto:Data: 2023-07-29T00:40:01.742 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.743 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:01.743 DEBUG:boto:Port: 8001 2023-07-29T00:40:01.743 DEBUG:boto:Params: {} 2023-07-29T00:40:01.744 DEBUG:boto:Token: None 2023-07-29T00:40:01.744 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj145?versionId=null 2023-07-29T00:40:01.745 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QJfsklTuwLRmpb2uVyL/y1LzneY= 2023-07-29T00:40:01.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QJfsklTuwLRmpb2uVyL/y1LzneY=', 'Content-Length': '0'} 2023-07-29T00:40:01.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c239b8bbae3ab069-0064c45fe1-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.747 DEBUG:boto:path=/hiiqos-59/obj145 2023-07-29T00:40:01.748 DEBUG:boto:auth_path=/hiiqos-59/obj145 2023-07-29T00:40:01.748 DEBUG:boto:path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:40:01.749 DEBUG:boto:auth_path=/hiiqos-59/obj145?versionId=null 2023-07-29T00:40:01.749 DEBUG:boto:Method: GET 2023-07-29T00:40:01.749 DEBUG:boto:Path: /hiiqos-59/obj145?versionId=null 2023-07-29T00:40:01.750 DEBUG:boto:Data: 2023-07-29T00:40:01.750 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.750 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:01.751 DEBUG:boto:Port: 8000 2023-07-29T00:40:01.751 DEBUG:boto:Params: {} 2023-07-29T00:40:01.752 DEBUG:boto:Token: None 2023-07-29T00:40:01.752 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj145?versionId=null 2023-07-29T00:40:01.753 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QJfsklTuwLRmpb2uVyL/y1LzneY= 2023-07-29T00:40:01.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QJfsklTuwLRmpb2uVyL/y1LzneY=', 'Content-Length': '0'} 2023-07-29T00:40:01.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d54b69d53f1cef3-0064c45fe1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.756 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:01.756 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:01.756 DEBUG:boto:path=/hiiqos-59/?prefix=obj145 2023-07-29T00:40:01.757 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj145 2023-07-29T00:40:01.757 DEBUG:boto:Method: GET 2023-07-29T00:40:01.757 DEBUG:boto:Path: /hiiqos-59/?prefix=obj145 2023-07-29T00:40:01.758 DEBUG:boto:Data: 2023-07-29T00:40:01.758 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.759 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:01.759 DEBUG:boto:Port: 8001 2023-07-29T00:40:01.759 DEBUG:boto:Params: {} 2023-07-29T00:40:01.760 DEBUG:boto:Token: None 2023-07-29T00:40:01.760 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/ 2023-07-29T00:40:01.760 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:F6z8zHPL1D3/2sdDjIeuqnNvTns= 2023-07-29T00:40:01.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:F6z8zHPL1D3/2sdDjIeuqnNvTns=', 'Content-Length': '0'} 2023-07-29T00:40:01.763 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a09b20b7ce9b62e-0064c45fe1-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.763 DEBUG:boto:b'hiiqos-59obj1451000falseobj1452023-07-29T00:35:01.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:01.764 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:01.765 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:01.765 DEBUG:boto:path=/hiiqos-59/?prefix=obj145 2023-07-29T00:40:01.765 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj145 2023-07-29T00:40:01.766 DEBUG:boto:Method: GET 2023-07-29T00:40:01.766 DEBUG:boto:Path: /hiiqos-59/?prefix=obj145 2023-07-29T00:40:01.766 DEBUG:boto:Data: 2023-07-29T00:40:01.767 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.767 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:01.767 DEBUG:boto:Port: 8000 2023-07-29T00:40:01.768 DEBUG:boto:Params: {} 2023-07-29T00:40:01.768 DEBUG:boto:Token: None 2023-07-29T00:40:01.769 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/ 2023-07-29T00:40:01.769 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:F6z8zHPL1D3/2sdDjIeuqnNvTns= 2023-07-29T00:40:01.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:F6z8zHPL1D3/2sdDjIeuqnNvTns=', 'Content-Length': '0'} 2023-07-29T00:40:01.773 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093bcd2ab54650b28-0064c45fe1-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.774 DEBUG:boto:b'hiiqos-59obj1451000falseobj1452023-07-29T00:35:01.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:01.775 DEBUG:boto:path=/hiiqos-59/obj145 2023-07-29T00:40:01.775 DEBUG:boto:auth_path=/hiiqos-59/obj145 2023-07-29T00:40:01.775 DEBUG:boto:Method: HEAD 2023-07-29T00:40:01.776 DEBUG:boto:Path: /hiiqos-59/obj145 2023-07-29T00:40:01.776 DEBUG:boto:Data: 2023-07-29T00:40:01.776 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.777 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:01.777 DEBUG:boto:Port: 8001 2023-07-29T00:40:01.777 DEBUG:boto:Params: {} 2023-07-29T00:40:01.778 DEBUG:boto:Token: None 2023-07-29T00:40:01.778 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj145 2023-07-29T00:40:01.779 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fEA9Q4FpBEqhbU6Syj69NsbgtV8= 2023-07-29T00:40:01.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fEA9Q4FpBEqhbU6Syj69NsbgtV8=', 'Content-Length': '0'} 2023-07-29T00:40:01.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a6f56f81db2b749-0064c45fe1-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.781 DEBUG:boto:path=/hiiqos-59/obj145 2023-07-29T00:40:01.782 DEBUG:boto:auth_path=/hiiqos-59/obj145 2023-07-29T00:40:01.782 DEBUG:boto:Method: HEAD 2023-07-29T00:40:01.782 DEBUG:boto:Path: /hiiqos-59/obj145 2023-07-29T00:40:01.783 DEBUG:boto:Data: 2023-07-29T00:40:01.783 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.783 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:01.784 DEBUG:boto:Port: 8000 2023-07-29T00:40:01.784 DEBUG:boto:Params: {} 2023-07-29T00:40:01.785 DEBUG:boto:Token: None 2023-07-29T00:40:01.785 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj145 2023-07-29T00:40:01.785 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fEA9Q4FpBEqhbU6Syj69NsbgtV8= 2023-07-29T00:40:01.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fEA9Q4FpBEqhbU6Syj69NsbgtV8=', 'Content-Length': '0'} 2023-07-29T00:40:01.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000050a62a7c57c81bc-0064c45fe1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-07-29T00:40:01.788 DEBUG:boto:path=/hiiqos-59/obj145 2023-07-29T00:40:01.789 DEBUG:boto:auth_path=/hiiqos-59/obj145 2023-07-29T00:40:01.789 DEBUG:boto:Method: GET 2023-07-29T00:40:01.789 DEBUG:boto:Path: /hiiqos-59/obj145 2023-07-29T00:40:01.790 DEBUG:boto:Data: 2023-07-29T00:40:01.790 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.791 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:01.791 DEBUG:boto:Port: 8001 2023-07-29T00:40:01.791 DEBUG:boto:Params: {} 2023-07-29T00:40:01.792 DEBUG:boto:Token: None 2023-07-29T00:40:01.792 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj145 2023-07-29T00:40:01.792 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:H5Kpz4RMb2pPWg7Ayt76EAnALKo= 2023-07-29T00:40:01.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:H5Kpz4RMb2pPWg7Ayt76EAnALKo=', 'Content-Length': '0'} 2023-07-29T00:40:01.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000853b9288ff36cace-0064c45fe1-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.795 DEBUG:boto:path=/hiiqos-59/obj145 2023-07-29T00:40:01.796 DEBUG:boto:auth_path=/hiiqos-59/obj145 2023-07-29T00:40:01.796 DEBUG:boto:Method: GET 2023-07-29T00:40:01.797 DEBUG:boto:Path: /hiiqos-59/obj145 2023-07-29T00:40:01.797 DEBUG:boto:Data: 2023-07-29T00:40:01.797 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.798 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:01.798 DEBUG:boto:Port: 8000 2023-07-29T00:40:01.798 DEBUG:boto:Params: {} 2023-07-29T00:40:01.799 DEBUG:boto:Token: None 2023-07-29T00:40:01.799 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj145 2023-07-29T00:40:01.800 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:H5Kpz4RMb2pPWg7Ayt76EAnALKo= 2023-07-29T00:40:01.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:H5Kpz4RMb2pPWg7Ayt76EAnALKo=', 'Content-Length': '0'} 2023-07-29T00:40:01.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6fde8cc13891bfb-0064c45fe1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.802 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:01.803 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:01.803 DEBUG:boto:path=/hiiqos-59/?prefix=obj145 2023-07-29T00:40:01.804 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj145 2023-07-29T00:40:01.804 DEBUG:boto:Method: GET 2023-07-29T00:40:01.804 DEBUG:boto:Path: /hiiqos-59/?prefix=obj145 2023-07-29T00:40:01.805 DEBUG:boto:Data: 2023-07-29T00:40:01.805 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.806 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:01.806 DEBUG:boto:Port: 8001 2023-07-29T00:40:01.806 DEBUG:boto:Params: {} 2023-07-29T00:40:01.807 DEBUG:boto:Token: None 2023-07-29T00:40:01.807 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/ 2023-07-29T00:40:01.808 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:F6z8zHPL1D3/2sdDjIeuqnNvTns= 2023-07-29T00:40:01.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:F6z8zHPL1D3/2sdDjIeuqnNvTns=', 'Content-Length': '0'} 2023-07-29T00:40:01.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a341e8aceb203c02-0064c45fe1-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.811 DEBUG:boto:b'hiiqos-59obj1451000falseobj1452023-07-29T00:35:01.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:01.811 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:01.812 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:01.812 DEBUG:boto:path=/hiiqos-59/?prefix=obj145 2023-07-29T00:40:01.812 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj145 2023-07-29T00:40:01.813 DEBUG:boto:Method: GET 2023-07-29T00:40:01.813 DEBUG:boto:Path: /hiiqos-59/?prefix=obj145 2023-07-29T00:40:01.814 DEBUG:boto:Data: 2023-07-29T00:40:01.814 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.815 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:01.815 DEBUG:boto:Port: 8000 2023-07-29T00:40:01.815 DEBUG:boto:Params: {} 2023-07-29T00:40:01.816 DEBUG:boto:Token: None 2023-07-29T00:40:01.816 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/ 2023-07-29T00:40:01.817 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:F6z8zHPL1D3/2sdDjIeuqnNvTns= 2023-07-29T00:40:01.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:F6z8zHPL1D3/2sdDjIeuqnNvTns=', 'Content-Length': '0'} 2023-07-29T00:40:01.821 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cc31014a19dc439-0064c45fe1-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.821 DEBUG:boto:b'hiiqos-59obj1451000falseobj1452023-07-29T00:35:01.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:01.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-07-29T00:40:01.822 DEBUG:boto:path=/hiiqos-59/obj148 2023-07-29T00:40:01.823 DEBUG:boto:auth_path=/hiiqos-59/obj148 2023-07-29T00:40:01.823 DEBUG:boto:path=/hiiqos-59/obj148?versionId=null 2023-07-29T00:40:01.823 DEBUG:boto:auth_path=/hiiqos-59/obj148?versionId=null 2023-07-29T00:40:01.824 DEBUG:boto:Method: GET 2023-07-29T00:40:01.824 DEBUG:boto:Path: /hiiqos-59/obj148?versionId=null 2023-07-29T00:40:01.825 DEBUG:boto:Data: 2023-07-29T00:40:01.825 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.825 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:01.826 DEBUG:boto:Port: 8001 2023-07-29T00:40:01.826 DEBUG:boto:Params: {} 2023-07-29T00:40:01.826 DEBUG:boto:Token: None 2023-07-29T00:40:01.827 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj148?versionId=null 2023-07-29T00:40:01.827 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/L8BSPTUoRfgajG6x/YF6hS8KSQ= 2023-07-29T00:40:01.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/L8BSPTUoRfgajG6x/YF6hS8KSQ=', 'Content-Length': '0'} 2023-07-29T00:40:01.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b78463e54b661a45-0064c45fe1-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.831 DEBUG:boto:path=/hiiqos-59/obj148 2023-07-29T00:40:01.831 DEBUG:boto:auth_path=/hiiqos-59/obj148 2023-07-29T00:40:01.832 DEBUG:boto:path=/hiiqos-59/obj148?versionId=null 2023-07-29T00:40:01.832 DEBUG:boto:auth_path=/hiiqos-59/obj148?versionId=null 2023-07-29T00:40:01.832 DEBUG:boto:Method: GET 2023-07-29T00:40:01.833 DEBUG:boto:Path: /hiiqos-59/obj148?versionId=null 2023-07-29T00:40:01.833 DEBUG:boto:Data: 2023-07-29T00:40:01.834 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.834 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:01.835 DEBUG:boto:Port: 8000 2023-07-29T00:40:01.835 DEBUG:boto:Params: {} 2023-07-29T00:40:01.835 DEBUG:boto:Token: None 2023-07-29T00:40:01.836 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj148?versionId=null 2023-07-29T00:40:01.836 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/L8BSPTUoRfgajG6x/YF6hS8KSQ= 2023-07-29T00:40:01.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/L8BSPTUoRfgajG6x/YF6hS8KSQ=', 'Content-Length': '0'} 2023-07-29T00:40:01.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a88ff59c6453ae2-0064c45fe1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.884 DEBUG:boto:path=/hiiqos-59/obj148 2023-07-29T00:40:01.885 DEBUG:boto:auth_path=/hiiqos-59/obj148 2023-07-29T00:40:01.885 DEBUG:boto:path=/hiiqos-59/obj148?versionId=null 2023-07-29T00:40:01.886 DEBUG:boto:auth_path=/hiiqos-59/obj148?versionId=null 2023-07-29T00:40:01.886 DEBUG:boto:Method: HEAD 2023-07-29T00:40:01.886 DEBUG:boto:Path: /hiiqos-59/obj148?versionId=null 2023-07-29T00:40:01.887 DEBUG:boto:Data: 2023-07-29T00:40:01.887 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.887 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:01.888 DEBUG:boto:Port: 8001 2023-07-29T00:40:01.888 DEBUG:boto:Params: {} 2023-07-29T00:40:01.888 DEBUG:boto:Token: None 2023-07-29T00:40:01.889 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj148?versionId=null 2023-07-29T00:40:01.889 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2YrH7U55iBo9XExPFJ1G1btBIb0= 2023-07-29T00:40:01.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2YrH7U55iBo9XExPFJ1G1btBIb0=', 'Content-Length': '0'} 2023-07-29T00:40:01.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016a9c9a16f424614-0064c45fe1-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.892 DEBUG:boto:path=/hiiqos-59/obj148 2023-07-29T00:40:01.893 DEBUG:boto:auth_path=/hiiqos-59/obj148 2023-07-29T00:40:01.893 DEBUG:boto:path=/hiiqos-59/obj148?versionId=null 2023-07-29T00:40:01.893 DEBUG:boto:auth_path=/hiiqos-59/obj148?versionId=null 2023-07-29T00:40:01.894 DEBUG:boto:Method: HEAD 2023-07-29T00:40:01.894 DEBUG:boto:Path: /hiiqos-59/obj148?versionId=null 2023-07-29T00:40:01.895 DEBUG:boto:Data: 2023-07-29T00:40:01.895 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.895 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:01.896 DEBUG:boto:Port: 8000 2023-07-29T00:40:01.896 DEBUG:boto:Params: {} 2023-07-29T00:40:01.897 DEBUG:boto:Token: None 2023-07-29T00:40:01.897 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj148?versionId=null 2023-07-29T00:40:01.897 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2YrH7U55iBo9XExPFJ1G1btBIb0= 2023-07-29T00:40:01.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2YrH7U55iBo9XExPFJ1G1btBIb0=', 'Content-Length': '0'} 2023-07-29T00:40:01.900 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002472d60cccbaedcb-0064c45fe1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-07-29T00:40:01.901 DEBUG:boto:path=/hiiqos-59/obj148 2023-07-29T00:40:01.901 DEBUG:boto:auth_path=/hiiqos-59/obj148 2023-07-29T00:40:01.902 DEBUG:boto:path=/hiiqos-59/obj148?versionId=null 2023-07-29T00:40:01.902 DEBUG:boto:auth_path=/hiiqos-59/obj148?versionId=null 2023-07-29T00:40:01.903 DEBUG:boto:Method: GET 2023-07-29T00:40:01.903 DEBUG:boto:Path: /hiiqos-59/obj148?versionId=null 2023-07-29T00:40:01.904 DEBUG:boto:Data: 2023-07-29T00:40:01.904 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.905 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:01.905 DEBUG:boto:Port: 8001 2023-07-29T00:40:01.905 DEBUG:boto:Params: {} 2023-07-29T00:40:01.906 DEBUG:boto:Token: None 2023-07-29T00:40:01.906 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj148?versionId=null 2023-07-29T00:40:01.907 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/L8BSPTUoRfgajG6x/YF6hS8KSQ= 2023-07-29T00:40:01.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/L8BSPTUoRfgajG6x/YF6hS8KSQ=', 'Content-Length': '0'} 2023-07-29T00:40:01.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb5145563bfd226d-0064c45fe1-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.910 DEBUG:boto:path=/hiiqos-59/obj148 2023-07-29T00:40:01.910 DEBUG:boto:auth_path=/hiiqos-59/obj148 2023-07-29T00:40:01.911 DEBUG:boto:path=/hiiqos-59/obj148?versionId=null 2023-07-29T00:40:01.911 DEBUG:boto:auth_path=/hiiqos-59/obj148?versionId=null 2023-07-29T00:40:01.911 DEBUG:boto:Method: GET 2023-07-29T00:40:01.912 DEBUG:boto:Path: /hiiqos-59/obj148?versionId=null 2023-07-29T00:40:01.912 DEBUG:boto:Data: 2023-07-29T00:40:01.913 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.913 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:01.913 DEBUG:boto:Port: 8000 2023-07-29T00:40:01.914 DEBUG:boto:Params: {} 2023-07-29T00:40:01.914 DEBUG:boto:Token: None 2023-07-29T00:40:01.915 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj148?versionId=null 2023-07-29T00:40:01.915 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/L8BSPTUoRfgajG6x/YF6hS8KSQ= 2023-07-29T00:40:01.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/L8BSPTUoRfgajG6x/YF6hS8KSQ=', 'Content-Length': '0'} 2023-07-29T00:40:01.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025436afbc53c48b7-0064c45fe1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.918 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:01.919 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:01.919 DEBUG:boto:path=/hiiqos-59/?prefix=obj148 2023-07-29T00:40:01.920 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj148 2023-07-29T00:40:01.920 DEBUG:boto:Method: GET 2023-07-29T00:40:01.920 DEBUG:boto:Path: /hiiqos-59/?prefix=obj148 2023-07-29T00:40:01.921 DEBUG:boto:Data: 2023-07-29T00:40:01.921 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.921 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:01.922 DEBUG:boto:Port: 8001 2023-07-29T00:40:01.922 DEBUG:boto:Params: {} 2023-07-29T00:40:01.923 DEBUG:boto:Token: None 2023-07-29T00:40:01.923 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/ 2023-07-29T00:40:01.924 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:F6z8zHPL1D3/2sdDjIeuqnNvTns= 2023-07-29T00:40:01.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:F6z8zHPL1D3/2sdDjIeuqnNvTns=', 'Content-Length': '0'} 2023-07-29T00:40:01.928 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc026e431723380d-0064c45fe1-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.972 DEBUG:boto:b'hiiqos-59obj1481000falseobj1482023-07-29T00:37:53.270Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:01.973 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:01.974 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:01.974 DEBUG:boto:path=/hiiqos-59/?prefix=obj148 2023-07-29T00:40:01.974 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj148 2023-07-29T00:40:01.975 DEBUG:boto:Method: GET 2023-07-29T00:40:01.975 DEBUG:boto:Path: /hiiqos-59/?prefix=obj148 2023-07-29T00:40:01.975 DEBUG:boto:Data: 2023-07-29T00:40:01.976 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.976 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:01.976 DEBUG:boto:Port: 8000 2023-07-29T00:40:01.977 DEBUG:boto:Params: {} 2023-07-29T00:40:01.977 DEBUG:boto:Token: None 2023-07-29T00:40:01.978 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/ 2023-07-29T00:40:01.978 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:F6z8zHPL1D3/2sdDjIeuqnNvTns= 2023-07-29T00:40:01.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:F6z8zHPL1D3/2sdDjIeuqnNvTns=', 'Content-Length': '0'} 2023-07-29T00:40:01.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a97f338f962a1107-0064c45fe1-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.983 DEBUG:boto:b'hiiqos-59obj1481000falseobj1482023-07-29T00:37:53.270Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:01.984 DEBUG:boto:path=/hiiqos-59/obj148 2023-07-29T00:40:01.984 DEBUG:boto:auth_path=/hiiqos-59/obj148 2023-07-29T00:40:01.985 DEBUG:boto:Method: HEAD 2023-07-29T00:40:01.985 DEBUG:boto:Path: /hiiqos-59/obj148 2023-07-29T00:40:01.985 DEBUG:boto:Data: 2023-07-29T00:40:01.986 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.986 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:01.987 DEBUG:boto:Port: 8001 2023-07-29T00:40:01.987 DEBUG:boto:Params: {} 2023-07-29T00:40:01.988 DEBUG:boto:Token: None 2023-07-29T00:40:01.988 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj148 2023-07-29T00:40:01.989 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Xk1lZ8jVzofutq8pEJM2kvbxWq4= 2023-07-29T00:40:01.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Xk1lZ8jVzofutq8pEJM2kvbxWq4=', 'Content-Length': '0'} 2023-07-29T00:40:01.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d02850b4cd0094a-0064c45fe1-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.991 DEBUG:boto:path=/hiiqos-59/obj148 2023-07-29T00:40:01.992 DEBUG:boto:auth_path=/hiiqos-59/obj148 2023-07-29T00:40:01.992 DEBUG:boto:Method: HEAD 2023-07-29T00:40:01.992 DEBUG:boto:Path: /hiiqos-59/obj148 2023-07-29T00:40:01.993 DEBUG:boto:Data: 2023-07-29T00:40:01.993 DEBUG:boto:Headers: {} 2023-07-29T00:40:01.993 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:01.994 DEBUG:boto:Port: 8000 2023-07-29T00:40:01.994 DEBUG:boto:Params: {} 2023-07-29T00:40:01.995 DEBUG:boto:Token: None 2023-07-29T00:40:01.995 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:01 GMT /hiiqos-59/obj148 2023-07-29T00:40:01.996 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Xk1lZ8jVzofutq8pEJM2kvbxWq4= 2023-07-29T00:40:01.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Xk1lZ8jVzofutq8pEJM2kvbxWq4=', 'Content-Length': '0'} 2023-07-29T00:40:01.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2c0f32428351d12-0064c45fe1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:01.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-07-29T00:40:01.999 DEBUG:boto:path=/hiiqos-59/obj148 2023-07-29T00:40:01.999 DEBUG:boto:auth_path=/hiiqos-59/obj148 2023-07-29T00:40:02.000 DEBUG:boto:Method: GET 2023-07-29T00:40:02.000 DEBUG:boto:Path: /hiiqos-59/obj148 2023-07-29T00:40:02.001 DEBUG:boto:Data: 2023-07-29T00:40:02.001 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.001 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:02.002 DEBUG:boto:Port: 8001 2023-07-29T00:40:02.002 DEBUG:boto:Params: {} 2023-07-29T00:40:02.002 DEBUG:boto:Token: None 2023-07-29T00:40:02.003 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj148 2023-07-29T00:40:02.004 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Dh/7m9f5ej/oo9wEwWAVqHwy0HE= 2023-07-29T00:40:02.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Dh/7m9f5ej/oo9wEwWAVqHwy0HE=', 'Content-Length': '0'} 2023-07-29T00:40:02.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0908383fd16590c-0064c45fe1-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.007 DEBUG:boto:path=/hiiqos-59/obj148 2023-07-29T00:40:02.007 DEBUG:boto:auth_path=/hiiqos-59/obj148 2023-07-29T00:40:02.008 DEBUG:boto:Method: GET 2023-07-29T00:40:02.008 DEBUG:boto:Path: /hiiqos-59/obj148 2023-07-29T00:40:02.009 DEBUG:boto:Data: 2023-07-29T00:40:02.009 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.009 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:02.010 DEBUG:boto:Port: 8000 2023-07-29T00:40:02.010 DEBUG:boto:Params: {} 2023-07-29T00:40:02.011 DEBUG:boto:Token: None 2023-07-29T00:40:02.011 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj148 2023-07-29T00:40:02.012 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Dh/7m9f5ej/oo9wEwWAVqHwy0HE= 2023-07-29T00:40:02.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Dh/7m9f5ej/oo9wEwWAVqHwy0HE=', 'Content-Length': '0'} 2023-07-29T00:40:02.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000caa49a4bce8c9cab-0064c45fe1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.016 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:02.017 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:02.018 DEBUG:boto:path=/hiiqos-59/?prefix=obj148 2023-07-29T00:40:02.019 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj148 2023-07-29T00:40:02.020 DEBUG:boto:Method: GET 2023-07-29T00:40:02.020 DEBUG:boto:Path: /hiiqos-59/?prefix=obj148 2023-07-29T00:40:02.021 DEBUG:boto:Data: 2023-07-29T00:40:02.021 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.021 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:02.022 DEBUG:boto:Port: 8001 2023-07-29T00:40:02.022 DEBUG:boto:Params: {} 2023-07-29T00:40:02.023 DEBUG:boto:Token: None 2023-07-29T00:40:02.023 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/ 2023-07-29T00:40:02.023 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DAd2RdF9oane3LajXOMXR/uMzXM= 2023-07-29T00:40:02.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DAd2RdF9oane3LajXOMXR/uMzXM=', 'Content-Length': '0'} 2023-07-29T00:40:02.026 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ff2ee430c27d99f-0064c45fe1-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.027 DEBUG:boto:b'hiiqos-59obj1481000falseobj1482023-07-29T00:37:53.270Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:02.027 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:02.028 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:02.029 DEBUG:boto:path=/hiiqos-59/?prefix=obj148 2023-07-29T00:40:02.029 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj148 2023-07-29T00:40:02.029 DEBUG:boto:Method: GET 2023-07-29T00:40:02.030 DEBUG:boto:Path: /hiiqos-59/?prefix=obj148 2023-07-29T00:40:02.030 DEBUG:boto:Data: 2023-07-29T00:40:02.031 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.031 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:02.031 DEBUG:boto:Port: 8000 2023-07-29T00:40:02.032 DEBUG:boto:Params: {} 2023-07-29T00:40:02.032 DEBUG:boto:Token: None 2023-07-29T00:40:02.032 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/ 2023-07-29T00:40:02.033 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DAd2RdF9oane3LajXOMXR/uMzXM= 2023-07-29T00:40:02.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DAd2RdF9oane3LajXOMXR/uMzXM=', 'Content-Length': '0'} 2023-07-29T00:40:02.037 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027bcac7d08ce7f30-0064c45fe1-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.037 DEBUG:boto:b'hiiqos-59obj1481000falseobj1482023-07-29T00:37:53.270Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:02.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-07-29T00:40:02.039 DEBUG:boto:path=/hiiqos-59/obj155 2023-07-29T00:40:02.039 DEBUG:boto:auth_path=/hiiqos-59/obj155 2023-07-29T00:40:02.040 DEBUG:boto:path=/hiiqos-59/obj155?versionId=null 2023-07-29T00:40:02.040 DEBUG:boto:auth_path=/hiiqos-59/obj155?versionId=null 2023-07-29T00:40:02.041 DEBUG:boto:Method: GET 2023-07-29T00:40:02.041 DEBUG:boto:Path: /hiiqos-59/obj155?versionId=null 2023-07-29T00:40:02.042 DEBUG:boto:Data: 2023-07-29T00:40:02.042 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.042 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:02.043 DEBUG:boto:Port: 8001 2023-07-29T00:40:02.043 DEBUG:boto:Params: {} 2023-07-29T00:40:02.043 DEBUG:boto:Token: None 2023-07-29T00:40:02.044 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj155?versionId=null 2023-07-29T00:40:02.044 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rCKC6yCgqYAW6C191KS2hThpuho= 2023-07-29T00:40:02.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rCKC6yCgqYAW6C191KS2hThpuho=', 'Content-Length': '0'} 2023-07-29T00:40:02.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cf901b54bdbc4ed-0064c45fe1-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.047 DEBUG:boto:path=/hiiqos-59/obj155 2023-07-29T00:40:02.048 DEBUG:boto:auth_path=/hiiqos-59/obj155 2023-07-29T00:40:02.048 DEBUG:boto:path=/hiiqos-59/obj155?versionId=null 2023-07-29T00:40:02.049 DEBUG:boto:auth_path=/hiiqos-59/obj155?versionId=null 2023-07-29T00:40:02.049 DEBUG:boto:Method: GET 2023-07-29T00:40:02.050 DEBUG:boto:Path: /hiiqos-59/obj155?versionId=null 2023-07-29T00:40:02.050 DEBUG:boto:Data: 2023-07-29T00:40:02.051 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.051 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:02.051 DEBUG:boto:Port: 8000 2023-07-29T00:40:02.052 DEBUG:boto:Params: {} 2023-07-29T00:40:02.052 DEBUG:boto:Token: None 2023-07-29T00:40:02.053 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj155?versionId=null 2023-07-29T00:40:02.053 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rCKC6yCgqYAW6C191KS2hThpuho= 2023-07-29T00:40:02.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rCKC6yCgqYAW6C191KS2hThpuho=', 'Content-Length': '0'} 2023-07-29T00:40:02.055 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040f4b1c4fdb47b2b-0064c45fe1-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.096 DEBUG:boto:path=/hiiqos-59/obj155 2023-07-29T00:40:02.097 DEBUG:boto:auth_path=/hiiqos-59/obj155 2023-07-29T00:40:02.097 DEBUG:boto:path=/hiiqos-59/obj155?versionId=null 2023-07-29T00:40:02.097 DEBUG:boto:auth_path=/hiiqos-59/obj155?versionId=null 2023-07-29T00:40:02.098 DEBUG:boto:Method: HEAD 2023-07-29T00:40:02.098 DEBUG:boto:Path: /hiiqos-59/obj155?versionId=null 2023-07-29T00:40:02.099 DEBUG:boto:Data: 2023-07-29T00:40:02.099 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.099 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:02.100 DEBUG:boto:Port: 8001 2023-07-29T00:40:02.100 DEBUG:boto:Params: {} 2023-07-29T00:40:02.100 DEBUG:boto:Token: None 2023-07-29T00:40:02.101 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj155?versionId=null 2023-07-29T00:40:02.102 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hbx1ER/zNDM5fSJW09fY+nkAkEg= 2023-07-29T00:40:02.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hbx1ER/zNDM5fSJW09fY+nkAkEg=', 'Content-Length': '0'} 2023-07-29T00:40:02.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f65b92721271dc6-0064c45fe2-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.105 DEBUG:boto:path=/hiiqos-59/obj155 2023-07-29T00:40:02.105 DEBUG:boto:auth_path=/hiiqos-59/obj155 2023-07-29T00:40:02.105 DEBUG:boto:path=/hiiqos-59/obj155?versionId=null 2023-07-29T00:40:02.106 DEBUG:boto:auth_path=/hiiqos-59/obj155?versionId=null 2023-07-29T00:40:02.106 DEBUG:boto:Method: HEAD 2023-07-29T00:40:02.106 DEBUG:boto:Path: /hiiqos-59/obj155?versionId=null 2023-07-29T00:40:02.107 DEBUG:boto:Data: 2023-07-29T00:40:02.107 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.107 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:02.108 DEBUG:boto:Port: 8000 2023-07-29T00:40:02.108 DEBUG:boto:Params: {} 2023-07-29T00:40:02.109 DEBUG:boto:Token: None 2023-07-29T00:40:02.109 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj155?versionId=null 2023-07-29T00:40:02.110 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hbx1ER/zNDM5fSJW09fY+nkAkEg= 2023-07-29T00:40:02.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hbx1ER/zNDM5fSJW09fY+nkAkEg=', 'Content-Length': '0'} 2023-07-29T00:40:02.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000976f94e7df7629f0-0064c45fe2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-07-29T00:40:02.114 DEBUG:boto:path=/hiiqos-59/obj155 2023-07-29T00:40:02.114 DEBUG:boto:auth_path=/hiiqos-59/obj155 2023-07-29T00:40:02.114 DEBUG:boto:path=/hiiqos-59/obj155?versionId=null 2023-07-29T00:40:02.115 DEBUG:boto:auth_path=/hiiqos-59/obj155?versionId=null 2023-07-29T00:40:02.115 DEBUG:boto:Method: GET 2023-07-29T00:40:02.115 DEBUG:boto:Path: /hiiqos-59/obj155?versionId=null 2023-07-29T00:40:02.116 DEBUG:boto:Data: 2023-07-29T00:40:02.116 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.117 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:02.117 DEBUG:boto:Port: 8001 2023-07-29T00:40:02.117 DEBUG:boto:Params: {} 2023-07-29T00:40:02.118 DEBUG:boto:Token: None 2023-07-29T00:40:02.118 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj155?versionId=null 2023-07-29T00:40:02.119 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rCKC6yCgqYAW6C191KS2hThpuho= 2023-07-29T00:40:02.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rCKC6yCgqYAW6C191KS2hThpuho=', 'Content-Length': '0'} 2023-07-29T00:40:02.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd24c29ebc3b1249-0064c45fe2-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.122 DEBUG:boto:path=/hiiqos-59/obj155 2023-07-29T00:40:02.122 DEBUG:boto:auth_path=/hiiqos-59/obj155 2023-07-29T00:40:02.122 DEBUG:boto:path=/hiiqos-59/obj155?versionId=null 2023-07-29T00:40:02.123 DEBUG:boto:auth_path=/hiiqos-59/obj155?versionId=null 2023-07-29T00:40:02.123 DEBUG:boto:Method: GET 2023-07-29T00:40:02.123 DEBUG:boto:Path: /hiiqos-59/obj155?versionId=null 2023-07-29T00:40:02.124 DEBUG:boto:Data: 2023-07-29T00:40:02.124 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.125 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:02.125 DEBUG:boto:Port: 8000 2023-07-29T00:40:02.125 DEBUG:boto:Params: {} 2023-07-29T00:40:02.126 DEBUG:boto:Token: None 2023-07-29T00:40:02.126 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj155?versionId=null 2023-07-29T00:40:02.127 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rCKC6yCgqYAW6C191KS2hThpuho= 2023-07-29T00:40:02.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rCKC6yCgqYAW6C191KS2hThpuho=', 'Content-Length': '0'} 2023-07-29T00:40:02.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081436a085cc10e71-0064c45fe2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.129 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:02.130 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:02.130 DEBUG:boto:path=/hiiqos-59/?prefix=obj155 2023-07-29T00:40:02.130 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj155 2023-07-29T00:40:02.131 DEBUG:boto:Method: GET 2023-07-29T00:40:02.131 DEBUG:boto:Path: /hiiqos-59/?prefix=obj155 2023-07-29T00:40:02.131 DEBUG:boto:Data: 2023-07-29T00:40:02.132 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.132 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:02.133 DEBUG:boto:Port: 8001 2023-07-29T00:40:02.133 DEBUG:boto:Params: {} 2023-07-29T00:40:02.133 DEBUG:boto:Token: None 2023-07-29T00:40:02.134 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/ 2023-07-29T00:40:02.134 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DAd2RdF9oane3LajXOMXR/uMzXM= 2023-07-29T00:40:02.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DAd2RdF9oane3LajXOMXR/uMzXM=', 'Content-Length': '0'} 2023-07-29T00:40:02.137 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d126cdf550c43c3-0064c45fe2-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.137 DEBUG:boto:b'hiiqos-59obj1551000falseobj1552023-07-29T00:35:31.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:02.138 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:02.138 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:02.139 DEBUG:boto:path=/hiiqos-59/?prefix=obj155 2023-07-29T00:40:02.139 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj155 2023-07-29T00:40:02.139 DEBUG:boto:Method: GET 2023-07-29T00:40:02.140 DEBUG:boto:Path: /hiiqos-59/?prefix=obj155 2023-07-29T00:40:02.140 DEBUG:boto:Data: 2023-07-29T00:40:02.141 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.141 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:02.141 DEBUG:boto:Port: 8000 2023-07-29T00:40:02.142 DEBUG:boto:Params: {} 2023-07-29T00:40:02.142 DEBUG:boto:Token: None 2023-07-29T00:40:02.143 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/ 2023-07-29T00:40:02.143 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DAd2RdF9oane3LajXOMXR/uMzXM= 2023-07-29T00:40:02.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DAd2RdF9oane3LajXOMXR/uMzXM=', 'Content-Length': '0'} 2023-07-29T00:40:02.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018c3f48d6ec97474-0064c45fe2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.188 DEBUG:boto:b'hiiqos-59obj1551000falseobj1552023-07-29T00:35:31.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:02.189 DEBUG:boto:path=/hiiqos-59/obj155 2023-07-29T00:40:02.190 DEBUG:boto:auth_path=/hiiqos-59/obj155 2023-07-29T00:40:02.190 DEBUG:boto:Method: HEAD 2023-07-29T00:40:02.191 DEBUG:boto:Path: /hiiqos-59/obj155 2023-07-29T00:40:02.191 DEBUG:boto:Data: 2023-07-29T00:40:02.191 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.192 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:02.192 DEBUG:boto:Port: 8001 2023-07-29T00:40:02.193 DEBUG:boto:Params: {} 2023-07-29T00:40:02.193 DEBUG:boto:Token: None 2023-07-29T00:40:02.193 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj155 2023-07-29T00:40:02.194 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oSk/nQEuMnMqDb0pjdOE8GNx+9w= 2023-07-29T00:40:02.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oSk/nQEuMnMqDb0pjdOE8GNx+9w=', 'Content-Length': '0'} 2023-07-29T00:40:02.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a223d14b327d3f0-0064c45fe2-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.197 DEBUG:boto:path=/hiiqos-59/obj155 2023-07-29T00:40:02.197 DEBUG:boto:auth_path=/hiiqos-59/obj155 2023-07-29T00:40:02.197 DEBUG:boto:Method: HEAD 2023-07-29T00:40:02.198 DEBUG:boto:Path: /hiiqos-59/obj155 2023-07-29T00:40:02.198 DEBUG:boto:Data: 2023-07-29T00:40:02.198 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.199 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:02.199 DEBUG:boto:Port: 8000 2023-07-29T00:40:02.199 DEBUG:boto:Params: {} 2023-07-29T00:40:02.200 DEBUG:boto:Token: None 2023-07-29T00:40:02.200 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj155 2023-07-29T00:40:02.201 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oSk/nQEuMnMqDb0pjdOE8GNx+9w= 2023-07-29T00:40:02.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oSk/nQEuMnMqDb0pjdOE8GNx+9w=', 'Content-Length': '0'} 2023-07-29T00:40:02.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000862cbd0aadad59b4-0064c45fe2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-07-29T00:40:02.204 DEBUG:boto:path=/hiiqos-59/obj155 2023-07-29T00:40:02.204 DEBUG:boto:auth_path=/hiiqos-59/obj155 2023-07-29T00:40:02.205 DEBUG:boto:Method: GET 2023-07-29T00:40:02.206 DEBUG:boto:Path: /hiiqos-59/obj155 2023-07-29T00:40:02.206 DEBUG:boto:Data: 2023-07-29T00:40:02.206 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.207 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:02.207 DEBUG:boto:Port: 8001 2023-07-29T00:40:02.207 DEBUG:boto:Params: {} 2023-07-29T00:40:02.208 DEBUG:boto:Token: None 2023-07-29T00:40:02.208 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj155 2023-07-29T00:40:02.209 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vVJB371R2RpqLKY563n7etl0NUk= 2023-07-29T00:40:02.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vVJB371R2RpqLKY563n7etl0NUk=', 'Content-Length': '0'} 2023-07-29T00:40:02.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a1f16c735a671ad-0064c45fe2-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.212 DEBUG:boto:path=/hiiqos-59/obj155 2023-07-29T00:40:02.212 DEBUG:boto:auth_path=/hiiqos-59/obj155 2023-07-29T00:40:02.212 DEBUG:boto:Method: GET 2023-07-29T00:40:02.213 DEBUG:boto:Path: /hiiqos-59/obj155 2023-07-29T00:40:02.213 DEBUG:boto:Data: 2023-07-29T00:40:02.214 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.214 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:02.214 DEBUG:boto:Port: 8000 2023-07-29T00:40:02.215 DEBUG:boto:Params: {} 2023-07-29T00:40:02.215 DEBUG:boto:Token: None 2023-07-29T00:40:02.215 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj155 2023-07-29T00:40:02.216 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vVJB371R2RpqLKY563n7etl0NUk= 2023-07-29T00:40:02.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vVJB371R2RpqLKY563n7etl0NUk=', 'Content-Length': '0'} 2023-07-29T00:40:02.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000155781aff2a8e46a-0064c45fe2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.219 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:02.219 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:02.219 DEBUG:boto:path=/hiiqos-59/?prefix=obj155 2023-07-29T00:40:02.220 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj155 2023-07-29T00:40:02.220 DEBUG:boto:Method: GET 2023-07-29T00:40:02.221 DEBUG:boto:Path: /hiiqos-59/?prefix=obj155 2023-07-29T00:40:02.222 DEBUG:boto:Data: 2023-07-29T00:40:02.223 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.224 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:02.225 DEBUG:boto:Port: 8001 2023-07-29T00:40:02.226 DEBUG:boto:Params: {} 2023-07-29T00:40:02.227 DEBUG:boto:Token: None 2023-07-29T00:40:02.228 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/ 2023-07-29T00:40:02.228 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DAd2RdF9oane3LajXOMXR/uMzXM= 2023-07-29T00:40:02.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DAd2RdF9oane3LajXOMXR/uMzXM=', 'Content-Length': '0'} 2023-07-29T00:40:02.232 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a3178b1c31003e9-0064c45fe2-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.233 DEBUG:boto:b'hiiqos-59obj1551000falseobj1552023-07-29T00:35:31.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:02.233 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:02.234 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:02.234 DEBUG:boto:path=/hiiqos-59/?prefix=obj155 2023-07-29T00:40:02.234 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj155 2023-07-29T00:40:02.235 DEBUG:boto:Method: GET 2023-07-29T00:40:02.235 DEBUG:boto:Path: /hiiqos-59/?prefix=obj155 2023-07-29T00:40:02.235 DEBUG:boto:Data: 2023-07-29T00:40:02.236 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.236 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:02.237 DEBUG:boto:Port: 8000 2023-07-29T00:40:02.237 DEBUG:boto:Params: {} 2023-07-29T00:40:02.237 DEBUG:boto:Token: None 2023-07-29T00:40:02.238 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/ 2023-07-29T00:40:02.238 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DAd2RdF9oane3LajXOMXR/uMzXM= 2023-07-29T00:40:02.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DAd2RdF9oane3LajXOMXR/uMzXM=', 'Content-Length': '0'} 2023-07-29T00:40:02.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000780f78eb1f5af852-0064c45fe2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.243 DEBUG:boto:b'hiiqos-59obj1551000falseobj1552023-07-29T00:35:31.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:02.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-07-29T00:40:02.245 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:40:02.245 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:40:02.245 DEBUG:boto:path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:40:02.246 DEBUG:boto:auth_path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:40:02.246 DEBUG:boto:Method: GET 2023-07-29T00:40:02.246 DEBUG:boto:Path: /hiiqos-59/obj17?versionId=null 2023-07-29T00:40:02.247 DEBUG:boto:Data: 2023-07-29T00:40:02.247 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.247 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:02.248 DEBUG:boto:Port: 8001 2023-07-29T00:40:02.248 DEBUG:boto:Params: {} 2023-07-29T00:40:02.249 DEBUG:boto:Token: None 2023-07-29T00:40:02.249 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj17?versionId=null 2023-07-29T00:40:02.249 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UjQrBzgmvoTEgLr1WQ6QuFzO8ts= 2023-07-29T00:40:02.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UjQrBzgmvoTEgLr1WQ6QuFzO8ts=', 'Content-Length': '0'} 2023-07-29T00:40:02.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac1f84c4e20fedb0-0064c45fe2-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.253 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:40:02.253 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:40:02.253 DEBUG:boto:path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:40:02.254 DEBUG:boto:auth_path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:40:02.254 DEBUG:boto:Method: GET 2023-07-29T00:40:02.254 DEBUG:boto:Path: /hiiqos-59/obj17?versionId=null 2023-07-29T00:40:02.255 DEBUG:boto:Data: 2023-07-29T00:40:02.255 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.255 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:02.256 DEBUG:boto:Port: 8000 2023-07-29T00:40:02.256 DEBUG:boto:Params: {} 2023-07-29T00:40:02.257 DEBUG:boto:Token: None 2023-07-29T00:40:02.257 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj17?versionId=null 2023-07-29T00:40:02.257 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UjQrBzgmvoTEgLr1WQ6QuFzO8ts= 2023-07-29T00:40:02.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UjQrBzgmvoTEgLr1WQ6QuFzO8ts=', 'Content-Length': '0'} 2023-07-29T00:40:02.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd7f8e22cc9e72ce-0064c45fe2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.260 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:40:02.261 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:40:02.261 DEBUG:boto:path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:40:02.262 DEBUG:boto:auth_path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:40:02.262 DEBUG:boto:Method: HEAD 2023-07-29T00:40:02.262 DEBUG:boto:Path: /hiiqos-59/obj17?versionId=null 2023-07-29T00:40:02.263 DEBUG:boto:Data: 2023-07-29T00:40:02.263 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.263 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:02.264 DEBUG:boto:Port: 8001 2023-07-29T00:40:02.264 DEBUG:boto:Params: {} 2023-07-29T00:40:02.264 DEBUG:boto:Token: None 2023-07-29T00:40:02.265 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj17?versionId=null 2023-07-29T00:40:02.265 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:U0hZqBgXrLq5EyxonEagILyH2GQ= 2023-07-29T00:40:02.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:U0hZqBgXrLq5EyxonEagILyH2GQ=', 'Content-Length': '0'} 2023-07-29T00:40:02.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000487c2dbcff4e4094-0064c45fe2-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.268 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:40:02.268 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:40:02.269 DEBUG:boto:path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:40:02.269 DEBUG:boto:auth_path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:40:02.269 DEBUG:boto:Method: HEAD 2023-07-29T00:40:02.270 DEBUG:boto:Path: /hiiqos-59/obj17?versionId=null 2023-07-29T00:40:02.270 DEBUG:boto:Data: 2023-07-29T00:40:02.270 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.271 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:02.271 DEBUG:boto:Port: 8000 2023-07-29T00:40:02.271 DEBUG:boto:Params: {} 2023-07-29T00:40:02.272 DEBUG:boto:Token: None 2023-07-29T00:40:02.272 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj17?versionId=null 2023-07-29T00:40:02.273 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:U0hZqBgXrLq5EyxonEagILyH2GQ= 2023-07-29T00:40:02.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:U0hZqBgXrLq5EyxonEagILyH2GQ=', 'Content-Length': '0'} 2023-07-29T00:40:02.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3da928c362fbc52-0064c45fe2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-07-29T00:40:02.277 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:40:02.277 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:40:02.278 DEBUG:boto:path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:40:02.278 DEBUG:boto:auth_path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:40:02.278 DEBUG:boto:Method: GET 2023-07-29T00:40:02.279 DEBUG:boto:Path: /hiiqos-59/obj17?versionId=null 2023-07-29T00:40:02.279 DEBUG:boto:Data: 2023-07-29T00:40:02.280 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.280 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:02.280 DEBUG:boto:Port: 8001 2023-07-29T00:40:02.281 DEBUG:boto:Params: {} 2023-07-29T00:40:02.281 DEBUG:boto:Token: None 2023-07-29T00:40:02.282 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj17?versionId=null 2023-07-29T00:40:02.282 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UjQrBzgmvoTEgLr1WQ6QuFzO8ts= 2023-07-29T00:40:02.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UjQrBzgmvoTEgLr1WQ6QuFzO8ts=', 'Content-Length': '0'} 2023-07-29T00:40:02.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c9bea5eae00cbde-0064c45fe2-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.285 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:40:02.285 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:40:02.286 DEBUG:boto:path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:40:02.286 DEBUG:boto:auth_path=/hiiqos-59/obj17?versionId=null 2023-07-29T00:40:02.287 DEBUG:boto:Method: GET 2023-07-29T00:40:02.288 DEBUG:boto:Path: /hiiqos-59/obj17?versionId=null 2023-07-29T00:40:02.289 DEBUG:boto:Data: 2023-07-29T00:40:02.290 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.291 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:02.292 DEBUG:boto:Port: 8000 2023-07-29T00:40:02.292 DEBUG:boto:Params: {} 2023-07-29T00:40:02.293 DEBUG:boto:Token: None 2023-07-29T00:40:02.294 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj17?versionId=null 2023-07-29T00:40:02.295 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UjQrBzgmvoTEgLr1WQ6QuFzO8ts= 2023-07-29T00:40:02.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UjQrBzgmvoTEgLr1WQ6QuFzO8ts=', 'Content-Length': '0'} 2023-07-29T00:40:02.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a0bf72446ed8ad4-0064c45fe2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.300 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:02.300 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:02.301 DEBUG:boto:path=/hiiqos-59/?prefix=obj17 2023-07-29T00:40:02.302 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj17 2023-07-29T00:40:02.303 DEBUG:boto:Method: GET 2023-07-29T00:40:02.303 DEBUG:boto:Path: /hiiqos-59/?prefix=obj17 2023-07-29T00:40:02.304 DEBUG:boto:Data: 2023-07-29T00:40:02.304 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.305 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:02.305 DEBUG:boto:Port: 8001 2023-07-29T00:40:02.306 DEBUG:boto:Params: {} 2023-07-29T00:40:02.307 DEBUG:boto:Token: None 2023-07-29T00:40:02.307 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/ 2023-07-29T00:40:02.308 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DAd2RdF9oane3LajXOMXR/uMzXM= 2023-07-29T00:40:02.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DAd2RdF9oane3LajXOMXR/uMzXM=', 'Content-Length': '0'} 2023-07-29T00:40:02.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074ae011cc4dfe94f-0064c45fe2-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.356 DEBUG:boto:b'hiiqos-59obj171000falseobj172023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:02.357 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:02.358 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:02.358 DEBUG:boto:path=/hiiqos-59/?prefix=obj17 2023-07-29T00:40:02.358 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj17 2023-07-29T00:40:02.359 DEBUG:boto:Method: GET 2023-07-29T00:40:02.359 DEBUG:boto:Path: /hiiqos-59/?prefix=obj17 2023-07-29T00:40:02.359 DEBUG:boto:Data: 2023-07-29T00:40:02.360 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.360 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:02.360 DEBUG:boto:Port: 8000 2023-07-29T00:40:02.361 DEBUG:boto:Params: {} 2023-07-29T00:40:02.361 DEBUG:boto:Token: None 2023-07-29T00:40:02.362 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/ 2023-07-29T00:40:02.362 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DAd2RdF9oane3LajXOMXR/uMzXM= 2023-07-29T00:40:02.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DAd2RdF9oane3LajXOMXR/uMzXM=', 'Content-Length': '0'} 2023-07-29T00:40:02.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020be80b872f00211-0064c45fe2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.367 DEBUG:boto:b'hiiqos-59obj171000falseobj172023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:02.368 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:40:02.368 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:40:02.368 DEBUG:boto:Method: HEAD 2023-07-29T00:40:02.369 DEBUG:boto:Path: /hiiqos-59/obj17 2023-07-29T00:40:02.369 DEBUG:boto:Data: 2023-07-29T00:40:02.369 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.370 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:02.370 DEBUG:boto:Port: 8001 2023-07-29T00:40:02.370 DEBUG:boto:Params: {} 2023-07-29T00:40:02.371 DEBUG:boto:Token: None 2023-07-29T00:40:02.371 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj17 2023-07-29T00:40:02.372 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Qme/tyBZlJgYuQAcufX0ht1EMJU= 2023-07-29T00:40:02.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Qme/tyBZlJgYuQAcufX0ht1EMJU=', 'Content-Length': '0'} 2023-07-29T00:40:02.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007be92acc224f4f4-0064c45fe2-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.374 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:40:02.375 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:40:02.375 DEBUG:boto:Method: HEAD 2023-07-29T00:40:02.375 DEBUG:boto:Path: /hiiqos-59/obj17 2023-07-29T00:40:02.376 DEBUG:boto:Data: 2023-07-29T00:40:02.376 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.376 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:02.377 DEBUG:boto:Port: 8000 2023-07-29T00:40:02.377 DEBUG:boto:Params: {} 2023-07-29T00:40:02.377 DEBUG:boto:Token: None 2023-07-29T00:40:02.378 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj17 2023-07-29T00:40:02.378 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Qme/tyBZlJgYuQAcufX0ht1EMJU= 2023-07-29T00:40:02.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Qme/tyBZlJgYuQAcufX0ht1EMJU=', 'Content-Length': '0'} 2023-07-29T00:40:02.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b60da26d9a11c414-0064c45fe2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-07-29T00:40:02.381 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:40:02.382 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:40:02.382 DEBUG:boto:Method: GET 2023-07-29T00:40:02.382 DEBUG:boto:Path: /hiiqos-59/obj17 2023-07-29T00:40:02.383 DEBUG:boto:Data: 2023-07-29T00:40:02.383 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.384 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:02.384 DEBUG:boto:Port: 8001 2023-07-29T00:40:02.384 DEBUG:boto:Params: {} 2023-07-29T00:40:02.385 DEBUG:boto:Token: None 2023-07-29T00:40:02.385 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj17 2023-07-29T00:40:02.386 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GUJY5XroPW2Cy7m6JGVUujq6xyI= 2023-07-29T00:40:02.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GUJY5XroPW2Cy7m6JGVUujq6xyI=', 'Content-Length': '0'} 2023-07-29T00:40:02.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0e7255a6af1e4e6-0064c45fe2-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.388 DEBUG:boto:path=/hiiqos-59/obj17 2023-07-29T00:40:02.389 DEBUG:boto:auth_path=/hiiqos-59/obj17 2023-07-29T00:40:02.389 DEBUG:boto:Method: GET 2023-07-29T00:40:02.390 DEBUG:boto:Path: /hiiqos-59/obj17 2023-07-29T00:40:02.390 DEBUG:boto:Data: 2023-07-29T00:40:02.390 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.391 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:02.391 DEBUG:boto:Port: 8000 2023-07-29T00:40:02.391 DEBUG:boto:Params: {} 2023-07-29T00:40:02.392 DEBUG:boto:Token: None 2023-07-29T00:40:02.392 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj17 2023-07-29T00:40:02.393 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GUJY5XroPW2Cy7m6JGVUujq6xyI= 2023-07-29T00:40:02.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GUJY5XroPW2Cy7m6JGVUujq6xyI=', 'Content-Length': '0'} 2023-07-29T00:40:02.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a32e812377893355-0064c45fe2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.436 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:02.437 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:02.437 DEBUG:boto:path=/hiiqos-59/?prefix=obj17 2023-07-29T00:40:02.438 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj17 2023-07-29T00:40:02.438 DEBUG:boto:Method: GET 2023-07-29T00:40:02.438 DEBUG:boto:Path: /hiiqos-59/?prefix=obj17 2023-07-29T00:40:02.439 DEBUG:boto:Data: 2023-07-29T00:40:02.439 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.439 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:02.440 DEBUG:boto:Port: 8001 2023-07-29T00:40:02.440 DEBUG:boto:Params: {} 2023-07-29T00:40:02.441 DEBUG:boto:Token: None 2023-07-29T00:40:02.441 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/ 2023-07-29T00:40:02.441 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DAd2RdF9oane3LajXOMXR/uMzXM= 2023-07-29T00:40:02.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DAd2RdF9oane3LajXOMXR/uMzXM=', 'Content-Length': '0'} 2023-07-29T00:40:02.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c27a0a01aa6900e7-0064c45fe2-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.445 DEBUG:boto:b'hiiqos-59obj171000falseobj172023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:02.445 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:02.446 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:02.446 DEBUG:boto:path=/hiiqos-59/?prefix=obj17 2023-07-29T00:40:02.446 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj17 2023-07-29T00:40:02.447 DEBUG:boto:Method: GET 2023-07-29T00:40:02.447 DEBUG:boto:Path: /hiiqos-59/?prefix=obj17 2023-07-29T00:40:02.448 DEBUG:boto:Data: 2023-07-29T00:40:02.448 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.448 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:02.449 DEBUG:boto:Port: 8000 2023-07-29T00:40:02.449 DEBUG:boto:Params: {} 2023-07-29T00:40:02.449 DEBUG:boto:Token: None 2023-07-29T00:40:02.450 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/ 2023-07-29T00:40:02.450 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DAd2RdF9oane3LajXOMXR/uMzXM= 2023-07-29T00:40:02.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DAd2RdF9oane3LajXOMXR/uMzXM=', 'Content-Length': '0'} 2023-07-29T00:40:02.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bf535196f0eba61-0064c45fe2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.455 DEBUG:boto:b'hiiqos-59obj171000falseobj172023-07-29T00:27:51.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:02.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-07-29T00:40:02.456 DEBUG:boto:path=/hiiqos-59/obj185 2023-07-29T00:40:02.456 DEBUG:boto:auth_path=/hiiqos-59/obj185 2023-07-29T00:40:02.457 DEBUG:boto:path=/hiiqos-59/obj185?versionId=null 2023-07-29T00:40:02.457 DEBUG:boto:auth_path=/hiiqos-59/obj185?versionId=null 2023-07-29T00:40:02.458 DEBUG:boto:Method: GET 2023-07-29T00:40:02.458 DEBUG:boto:Path: /hiiqos-59/obj185?versionId=null 2023-07-29T00:40:02.458 DEBUG:boto:Data: 2023-07-29T00:40:02.459 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.459 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:02.459 DEBUG:boto:Port: 8001 2023-07-29T00:40:02.460 DEBUG:boto:Params: {} 2023-07-29T00:40:02.460 DEBUG:boto:Token: None 2023-07-29T00:40:02.461 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj185?versionId=null 2023-07-29T00:40:02.461 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vR9siQ2tJjSzWF5DHgGPtJ8l1iw= 2023-07-29T00:40:02.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vR9siQ2tJjSzWF5DHgGPtJ8l1iw=', 'Content-Length': '0'} 2023-07-29T00:40:02.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073e5092e33e93210-0064c45fe2-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.464 DEBUG:boto:path=/hiiqos-59/obj185 2023-07-29T00:40:02.465 DEBUG:boto:auth_path=/hiiqos-59/obj185 2023-07-29T00:40:02.465 DEBUG:boto:path=/hiiqos-59/obj185?versionId=null 2023-07-29T00:40:02.466 DEBUG:boto:auth_path=/hiiqos-59/obj185?versionId=null 2023-07-29T00:40:02.466 DEBUG:boto:Method: GET 2023-07-29T00:40:02.466 DEBUG:boto:Path: /hiiqos-59/obj185?versionId=null 2023-07-29T00:40:02.467 DEBUG:boto:Data: 2023-07-29T00:40:02.467 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.467 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:02.468 DEBUG:boto:Port: 8000 2023-07-29T00:40:02.468 DEBUG:boto:Params: {} 2023-07-29T00:40:02.469 DEBUG:boto:Token: None 2023-07-29T00:40:02.469 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj185?versionId=null 2023-07-29T00:40:02.469 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vR9siQ2tJjSzWF5DHgGPtJ8l1iw= 2023-07-29T00:40:02.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vR9siQ2tJjSzWF5DHgGPtJ8l1iw=', 'Content-Length': '0'} 2023-07-29T00:40:02.472 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a7f79784402d7b2-0064c45fe2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.473 DEBUG:boto:path=/hiiqos-59/obj185 2023-07-29T00:40:02.473 DEBUG:boto:auth_path=/hiiqos-59/obj185 2023-07-29T00:40:02.474 DEBUG:boto:path=/hiiqos-59/obj185?versionId=null 2023-07-29T00:40:02.474 DEBUG:boto:auth_path=/hiiqos-59/obj185?versionId=null 2023-07-29T00:40:02.474 DEBUG:boto:Method: HEAD 2023-07-29T00:40:02.475 DEBUG:boto:Path: /hiiqos-59/obj185?versionId=null 2023-07-29T00:40:02.475 DEBUG:boto:Data: 2023-07-29T00:40:02.475 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.476 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:02.476 DEBUG:boto:Port: 8001 2023-07-29T00:40:02.476 DEBUG:boto:Params: {} 2023-07-29T00:40:02.477 DEBUG:boto:Token: None 2023-07-29T00:40:02.477 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj185?versionId=null 2023-07-29T00:40:02.478 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GF2n83l6miVwVVCf9NllIh3AyWo= 2023-07-29T00:40:02.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GF2n83l6miVwVVCf9NllIh3AyWo=', 'Content-Length': '0'} 2023-07-29T00:40:02.480 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f27d434431c5169-0064c45fe2-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.480 DEBUG:boto:path=/hiiqos-59/obj185 2023-07-29T00:40:02.481 DEBUG:boto:auth_path=/hiiqos-59/obj185 2023-07-29T00:40:02.481 DEBUG:boto:path=/hiiqos-59/obj185?versionId=null 2023-07-29T00:40:02.481 DEBUG:boto:auth_path=/hiiqos-59/obj185?versionId=null 2023-07-29T00:40:02.482 DEBUG:boto:Method: HEAD 2023-07-29T00:40:02.482 DEBUG:boto:Path: /hiiqos-59/obj185?versionId=null 2023-07-29T00:40:02.482 DEBUG:boto:Data: 2023-07-29T00:40:02.483 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.483 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:02.483 DEBUG:boto:Port: 8000 2023-07-29T00:40:02.484 DEBUG:boto:Params: {} 2023-07-29T00:40:02.484 DEBUG:boto:Token: None 2023-07-29T00:40:02.484 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj185?versionId=null 2023-07-29T00:40:02.485 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GF2n83l6miVwVVCf9NllIh3AyWo= 2023-07-29T00:40:02.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GF2n83l6miVwVVCf9NllIh3AyWo=', 'Content-Length': '0'} 2023-07-29T00:40:02.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000602bd239393f9a5d-0064c45fe2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-07-29T00:40:02.488 DEBUG:boto:path=/hiiqos-59/obj185 2023-07-29T00:40:02.488 DEBUG:boto:auth_path=/hiiqos-59/obj185 2023-07-29T00:40:02.488 DEBUG:boto:path=/hiiqos-59/obj185?versionId=null 2023-07-29T00:40:02.490 DEBUG:boto:auth_path=/hiiqos-59/obj185?versionId=null 2023-07-29T00:40:02.491 DEBUG:boto:Method: GET 2023-07-29T00:40:02.492 DEBUG:boto:Path: /hiiqos-59/obj185?versionId=null 2023-07-29T00:40:02.493 DEBUG:boto:Data: 2023-07-29T00:40:02.493 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.494 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:02.494 DEBUG:boto:Port: 8001 2023-07-29T00:40:02.495 DEBUG:boto:Params: {} 2023-07-29T00:40:02.495 DEBUG:boto:Token: None 2023-07-29T00:40:02.495 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj185?versionId=null 2023-07-29T00:40:02.496 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vR9siQ2tJjSzWF5DHgGPtJ8l1iw= 2023-07-29T00:40:02.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vR9siQ2tJjSzWF5DHgGPtJ8l1iw=', 'Content-Length': '0'} 2023-07-29T00:40:02.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008457aca19eb0de79-0064c45fe2-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.499 DEBUG:boto:path=/hiiqos-59/obj185 2023-07-29T00:40:02.499 DEBUG:boto:auth_path=/hiiqos-59/obj185 2023-07-29T00:40:02.500 DEBUG:boto:path=/hiiqos-59/obj185?versionId=null 2023-07-29T00:40:02.500 DEBUG:boto:auth_path=/hiiqos-59/obj185?versionId=null 2023-07-29T00:40:02.500 DEBUG:boto:Method: GET 2023-07-29T00:40:02.501 DEBUG:boto:Path: /hiiqos-59/obj185?versionId=null 2023-07-29T00:40:02.501 DEBUG:boto:Data: 2023-07-29T00:40:02.502 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.502 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:02.502 DEBUG:boto:Port: 8000 2023-07-29T00:40:02.503 DEBUG:boto:Params: {} 2023-07-29T00:40:02.503 DEBUG:boto:Token: None 2023-07-29T00:40:02.503 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj185?versionId=null 2023-07-29T00:40:02.504 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vR9siQ2tJjSzWF5DHgGPtJ8l1iw= 2023-07-29T00:40:02.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vR9siQ2tJjSzWF5DHgGPtJ8l1iw=', 'Content-Length': '0'} 2023-07-29T00:40:02.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098458e209bd81ea0-0064c45fe2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.507 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:02.507 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:02.508 DEBUG:boto:path=/hiiqos-59/?prefix=obj185 2023-07-29T00:40:02.508 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj185 2023-07-29T00:40:02.509 DEBUG:boto:Method: GET 2023-07-29T00:40:02.509 DEBUG:boto:Path: /hiiqos-59/?prefix=obj185 2023-07-29T00:40:02.509 DEBUG:boto:Data: 2023-07-29T00:40:02.510 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.510 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:02.511 DEBUG:boto:Port: 8001 2023-07-29T00:40:02.511 DEBUG:boto:Params: {} 2023-07-29T00:40:02.511 DEBUG:boto:Token: None 2023-07-29T00:40:02.512 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/ 2023-07-29T00:40:02.512 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DAd2RdF9oane3LajXOMXR/uMzXM= 2023-07-29T00:40:02.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DAd2RdF9oane3LajXOMXR/uMzXM=', 'Content-Length': '0'} 2023-07-29T00:40:02.515 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab48d66648c6e85d-0064c45fe2-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.515 DEBUG:boto:b'hiiqos-59obj1851000falseobj1852023-07-29T00:37:53.291Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:02.516 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:02.517 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:02.517 DEBUG:boto:path=/hiiqos-59/?prefix=obj185 2023-07-29T00:40:02.518 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj185 2023-07-29T00:40:02.518 DEBUG:boto:Method: GET 2023-07-29T00:40:02.518 DEBUG:boto:Path: /hiiqos-59/?prefix=obj185 2023-07-29T00:40:02.519 DEBUG:boto:Data: 2023-07-29T00:40:02.520 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.520 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:02.521 DEBUG:boto:Port: 8000 2023-07-29T00:40:02.522 DEBUG:boto:Params: {} 2023-07-29T00:40:02.523 DEBUG:boto:Token: None 2023-07-29T00:40:02.524 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/ 2023-07-29T00:40:02.525 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DAd2RdF9oane3LajXOMXR/uMzXM= 2023-07-29T00:40:02.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DAd2RdF9oane3LajXOMXR/uMzXM=', 'Content-Length': '0'} 2023-07-29T00:40:02.529 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005edde39ef79f90dd-0064c45fe2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.530 DEBUG:boto:b'hiiqos-59obj1851000falseobj1852023-07-29T00:37:53.291Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:02.531 DEBUG:boto:path=/hiiqos-59/obj185 2023-07-29T00:40:02.531 DEBUG:boto:auth_path=/hiiqos-59/obj185 2023-07-29T00:40:02.531 DEBUG:boto:Method: HEAD 2023-07-29T00:40:02.532 DEBUG:boto:Path: /hiiqos-59/obj185 2023-07-29T00:40:02.532 DEBUG:boto:Data: 2023-07-29T00:40:02.533 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.533 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:02.533 DEBUG:boto:Port: 8001 2023-07-29T00:40:02.534 DEBUG:boto:Params: {} 2023-07-29T00:40:02.534 DEBUG:boto:Token: None 2023-07-29T00:40:02.535 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj185 2023-07-29T00:40:02.535 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eAMV+78w7pFhgUrrwcsfYtVOGWE= 2023-07-29T00:40:02.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eAMV+78w7pFhgUrrwcsfYtVOGWE=', 'Content-Length': '0'} 2023-07-29T00:40:02.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024bafdba4de42ebc-0064c45fe2-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.538 DEBUG:boto:path=/hiiqos-59/obj185 2023-07-29T00:40:02.538 DEBUG:boto:auth_path=/hiiqos-59/obj185 2023-07-29T00:40:02.539 DEBUG:boto:Method: HEAD 2023-07-29T00:40:02.539 DEBUG:boto:Path: /hiiqos-59/obj185 2023-07-29T00:40:02.539 DEBUG:boto:Data: 2023-07-29T00:40:02.540 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.540 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:02.540 DEBUG:boto:Port: 8000 2023-07-29T00:40:02.541 DEBUG:boto:Params: {} 2023-07-29T00:40:02.541 DEBUG:boto:Token: None 2023-07-29T00:40:02.542 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj185 2023-07-29T00:40:02.542 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eAMV+78w7pFhgUrrwcsfYtVOGWE= 2023-07-29T00:40:02.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eAMV+78w7pFhgUrrwcsfYtVOGWE=', 'Content-Length': '0'} 2023-07-29T00:40:02.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af12caba392e8888-0064c45fe2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-07-29T00:40:02.545 DEBUG:boto:path=/hiiqos-59/obj185 2023-07-29T00:40:02.546 DEBUG:boto:auth_path=/hiiqos-59/obj185 2023-07-29T00:40:02.546 DEBUG:boto:Method: GET 2023-07-29T00:40:02.546 DEBUG:boto:Path: /hiiqos-59/obj185 2023-07-29T00:40:02.547 DEBUG:boto:Data: 2023-07-29T00:40:02.547 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.547 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:02.548 DEBUG:boto:Port: 8001 2023-07-29T00:40:02.548 DEBUG:boto:Params: {} 2023-07-29T00:40:02.549 DEBUG:boto:Token: None 2023-07-29T00:40:02.549 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj185 2023-07-29T00:40:02.549 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:c7bxEdFaEpEnI1QsmMQqjvmbti8= 2023-07-29T00:40:02.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c7bxEdFaEpEnI1QsmMQqjvmbti8=', 'Content-Length': '0'} 2023-07-29T00:40:02.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4ea1b4517a2f7b4-0064c45fe2-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.553 DEBUG:boto:path=/hiiqos-59/obj185 2023-07-29T00:40:02.554 DEBUG:boto:auth_path=/hiiqos-59/obj185 2023-07-29T00:40:02.555 DEBUG:boto:Method: GET 2023-07-29T00:40:02.555 DEBUG:boto:Path: /hiiqos-59/obj185 2023-07-29T00:40:02.556 DEBUG:boto:Data: 2023-07-29T00:40:02.557 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.558 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:02.558 DEBUG:boto:Port: 8000 2023-07-29T00:40:02.559 DEBUG:boto:Params: {} 2023-07-29T00:40:02.560 DEBUG:boto:Token: None 2023-07-29T00:40:02.560 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj185 2023-07-29T00:40:02.561 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:c7bxEdFaEpEnI1QsmMQqjvmbti8= 2023-07-29T00:40:02.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c7bxEdFaEpEnI1QsmMQqjvmbti8=', 'Content-Length': '0'} 2023-07-29T00:40:02.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059877df5b6a0ad8a-0064c45fe2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.604 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:02.605 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:02.605 DEBUG:boto:path=/hiiqos-59/?prefix=obj185 2023-07-29T00:40:02.606 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj185 2023-07-29T00:40:02.606 DEBUG:boto:Method: GET 2023-07-29T00:40:02.606 DEBUG:boto:Path: /hiiqos-59/?prefix=obj185 2023-07-29T00:40:02.607 DEBUG:boto:Data: 2023-07-29T00:40:02.607 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.607 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:02.608 DEBUG:boto:Port: 8001 2023-07-29T00:40:02.608 DEBUG:boto:Params: {} 2023-07-29T00:40:02.609 DEBUG:boto:Token: None 2023-07-29T00:40:02.609 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/ 2023-07-29T00:40:02.609 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DAd2RdF9oane3LajXOMXR/uMzXM= 2023-07-29T00:40:02.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DAd2RdF9oane3LajXOMXR/uMzXM=', 'Content-Length': '0'} 2023-07-29T00:40:02.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039b74788f8968719-0064c45fe2-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.613 DEBUG:boto:b'hiiqos-59obj1851000falseobj1852023-07-29T00:37:53.291Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:02.614 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:02.614 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:02.615 DEBUG:boto:path=/hiiqos-59/?prefix=obj185 2023-07-29T00:40:02.616 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj185 2023-07-29T00:40:02.617 DEBUG:boto:Method: GET 2023-07-29T00:40:02.618 DEBUG:boto:Path: /hiiqos-59/?prefix=obj185 2023-07-29T00:40:02.619 DEBUG:boto:Data: 2023-07-29T00:40:02.619 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.619 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:02.620 DEBUG:boto:Port: 8000 2023-07-29T00:40:02.620 DEBUG:boto:Params: {} 2023-07-29T00:40:02.620 DEBUG:boto:Token: None 2023-07-29T00:40:02.621 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/ 2023-07-29T00:40:02.621 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DAd2RdF9oane3LajXOMXR/uMzXM= 2023-07-29T00:40:02.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DAd2RdF9oane3LajXOMXR/uMzXM=', 'Content-Length': '0'} 2023-07-29T00:40:02.626 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a2486b81f7f653d-0064c45fe2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.626 DEBUG:boto:b'hiiqos-59obj1851000falseobj1852023-07-29T00:37:53.291Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:02.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-07-29T00:40:02.628 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:40:02.628 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:40:02.629 DEBUG:boto:path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:40:02.629 DEBUG:boto:auth_path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:40:02.629 DEBUG:boto:Method: GET 2023-07-29T00:40:02.630 DEBUG:boto:Path: /hiiqos-59/obj19?versionId=null 2023-07-29T00:40:02.630 DEBUG:boto:Data: 2023-07-29T00:40:02.630 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.631 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:02.631 DEBUG:boto:Port: 8001 2023-07-29T00:40:02.631 DEBUG:boto:Params: {} 2023-07-29T00:40:02.632 DEBUG:boto:Token: None 2023-07-29T00:40:02.632 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj19?versionId=null 2023-07-29T00:40:02.633 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SSqKvs66iqmsK6SImUBYvNzJjkA= 2023-07-29T00:40:02.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SSqKvs66iqmsK6SImUBYvNzJjkA=', 'Content-Length': '0'} 2023-07-29T00:40:02.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052b4a0165623256f-0064c45fe2-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.676 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:40:02.677 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:40:02.678 DEBUG:boto:path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:40:02.679 DEBUG:boto:auth_path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:40:02.679 DEBUG:boto:Method: GET 2023-07-29T00:40:02.680 DEBUG:boto:Path: /hiiqos-59/obj19?versionId=null 2023-07-29T00:40:02.681 DEBUG:boto:Data: 2023-07-29T00:40:02.681 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.682 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:02.683 DEBUG:boto:Port: 8000 2023-07-29T00:40:02.684 DEBUG:boto:Params: {} 2023-07-29T00:40:02.685 DEBUG:boto:Token: None 2023-07-29T00:40:02.685 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj19?versionId=null 2023-07-29T00:40:02.686 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SSqKvs66iqmsK6SImUBYvNzJjkA= 2023-07-29T00:40:02.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SSqKvs66iqmsK6SImUBYvNzJjkA=', 'Content-Length': '0'} 2023-07-29T00:40:02.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc86d9568035ebaa-0064c45fe2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.691 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:40:02.691 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:40:02.692 DEBUG:boto:path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:40:02.693 DEBUG:boto:auth_path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:40:02.693 DEBUG:boto:Method: HEAD 2023-07-29T00:40:02.694 DEBUG:boto:Path: /hiiqos-59/obj19?versionId=null 2023-07-29T00:40:02.694 DEBUG:boto:Data: 2023-07-29T00:40:02.694 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.695 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:02.695 DEBUG:boto:Port: 8001 2023-07-29T00:40:02.695 DEBUG:boto:Params: {} 2023-07-29T00:40:02.696 DEBUG:boto:Token: None 2023-07-29T00:40:02.696 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj19?versionId=null 2023-07-29T00:40:02.697 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XkVebBdFhJd9PalmhbHQeKaQMgo= 2023-07-29T00:40:02.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XkVebBdFhJd9PalmhbHQeKaQMgo=', 'Content-Length': '0'} 2023-07-29T00:40:02.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c445a4df5f831f45-0064c45fe2-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.700 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:40:02.700 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:40:02.700 DEBUG:boto:path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:40:02.701 DEBUG:boto:auth_path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:40:02.701 DEBUG:boto:Method: HEAD 2023-07-29T00:40:02.701 DEBUG:boto:Path: /hiiqos-59/obj19?versionId=null 2023-07-29T00:40:02.702 DEBUG:boto:Data: 2023-07-29T00:40:02.702 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.702 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:02.703 DEBUG:boto:Port: 8000 2023-07-29T00:40:02.703 DEBUG:boto:Params: {} 2023-07-29T00:40:02.704 DEBUG:boto:Token: None 2023-07-29T00:40:02.704 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj19?versionId=null 2023-07-29T00:40:02.705 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XkVebBdFhJd9PalmhbHQeKaQMgo= 2023-07-29T00:40:02.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XkVebBdFhJd9PalmhbHQeKaQMgo=', 'Content-Length': '0'} 2023-07-29T00:40:02.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007eb14df45a26705f-0064c45fe2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-07-29T00:40:02.708 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:40:02.709 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:40:02.709 DEBUG:boto:path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:40:02.709 DEBUG:boto:auth_path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:40:02.710 DEBUG:boto:Method: GET 2023-07-29T00:40:02.710 DEBUG:boto:Path: /hiiqos-59/obj19?versionId=null 2023-07-29T00:40:02.710 DEBUG:boto:Data: 2023-07-29T00:40:02.711 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.711 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:02.711 DEBUG:boto:Port: 8001 2023-07-29T00:40:02.712 DEBUG:boto:Params: {} 2023-07-29T00:40:02.712 DEBUG:boto:Token: None 2023-07-29T00:40:02.713 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj19?versionId=null 2023-07-29T00:40:02.713 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SSqKvs66iqmsK6SImUBYvNzJjkA= 2023-07-29T00:40:02.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SSqKvs66iqmsK6SImUBYvNzJjkA=', 'Content-Length': '0'} 2023-07-29T00:40:02.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054d8ff0e113f41ac-0064c45fe2-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.716 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:40:02.716 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:40:02.717 DEBUG:boto:path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:40:02.718 DEBUG:boto:auth_path=/hiiqos-59/obj19?versionId=null 2023-07-29T00:40:02.719 DEBUG:boto:Method: GET 2023-07-29T00:40:02.719 DEBUG:boto:Path: /hiiqos-59/obj19?versionId=null 2023-07-29T00:40:02.720 DEBUG:boto:Data: 2023-07-29T00:40:02.721 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.722 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:02.723 DEBUG:boto:Port: 8000 2023-07-29T00:40:02.724 DEBUG:boto:Params: {} 2023-07-29T00:40:02.725 DEBUG:boto:Token: None 2023-07-29T00:40:02.725 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj19?versionId=null 2023-07-29T00:40:02.726 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SSqKvs66iqmsK6SImUBYvNzJjkA= 2023-07-29T00:40:02.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SSqKvs66iqmsK6SImUBYvNzJjkA=', 'Content-Length': '0'} 2023-07-29T00:40:02.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f60f840b9111dc5a-0064c45fe2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.730 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:02.730 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:02.731 DEBUG:boto:path=/hiiqos-59/?prefix=obj19 2023-07-29T00:40:02.731 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj19 2023-07-29T00:40:02.732 DEBUG:boto:Method: GET 2023-07-29T00:40:02.732 DEBUG:boto:Path: /hiiqos-59/?prefix=obj19 2023-07-29T00:40:02.732 DEBUG:boto:Data: 2023-07-29T00:40:02.733 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.733 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:02.734 DEBUG:boto:Port: 8001 2023-07-29T00:40:02.734 DEBUG:boto:Params: {} 2023-07-29T00:40:02.734 DEBUG:boto:Token: None 2023-07-29T00:40:02.735 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/ 2023-07-29T00:40:02.735 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DAd2RdF9oane3LajXOMXR/uMzXM= 2023-07-29T00:40:02.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DAd2RdF9oane3LajXOMXR/uMzXM=', 'Content-Length': '0'} 2023-07-29T00:40:02.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b61616d870162c0b-0064c45fe2-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.738 DEBUG:boto:b'hiiqos-59obj191000falseobj192023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:02.739 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:02.740 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:02.740 DEBUG:boto:path=/hiiqos-59/?prefix=obj19 2023-07-29T00:40:02.740 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj19 2023-07-29T00:40:02.741 DEBUG:boto:Method: GET 2023-07-29T00:40:02.741 DEBUG:boto:Path: /hiiqos-59/?prefix=obj19 2023-07-29T00:40:02.741 DEBUG:boto:Data: 2023-07-29T00:40:02.742 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.742 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:02.742 DEBUG:boto:Port: 8000 2023-07-29T00:40:02.743 DEBUG:boto:Params: {} 2023-07-29T00:40:02.743 DEBUG:boto:Token: None 2023-07-29T00:40:02.744 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/ 2023-07-29T00:40:02.744 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DAd2RdF9oane3LajXOMXR/uMzXM= 2023-07-29T00:40:02.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DAd2RdF9oane3LajXOMXR/uMzXM=', 'Content-Length': '0'} 2023-07-29T00:40:02.748 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5c7681c95ed7562-0064c45fe2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.792 DEBUG:boto:b'hiiqos-59obj191000falseobj192023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:02.793 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:40:02.794 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:40:02.794 DEBUG:boto:Method: HEAD 2023-07-29T00:40:02.794 DEBUG:boto:Path: /hiiqos-59/obj19 2023-07-29T00:40:02.795 DEBUG:boto:Data: 2023-07-29T00:40:02.795 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.795 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:02.796 DEBUG:boto:Port: 8001 2023-07-29T00:40:02.796 DEBUG:boto:Params: {} 2023-07-29T00:40:02.796 DEBUG:boto:Token: None 2023-07-29T00:40:02.797 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj19 2023-07-29T00:40:02.797 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FIiSFNW5IW2poc8FOqYr3BppoH4= 2023-07-29T00:40:02.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FIiSFNW5IW2poc8FOqYr3BppoH4=', 'Content-Length': '0'} 2023-07-29T00:40:02.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4d1b829f5e076fc-0064c45fe2-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.801 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:40:02.803 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:40:02.804 DEBUG:boto:Method: HEAD 2023-07-29T00:40:02.804 DEBUG:boto:Path: /hiiqos-59/obj19 2023-07-29T00:40:02.805 DEBUG:boto:Data: 2023-07-29T00:40:02.805 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.805 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:02.806 DEBUG:boto:Port: 8000 2023-07-29T00:40:02.806 DEBUG:boto:Params: {} 2023-07-29T00:40:02.806 DEBUG:boto:Token: None 2023-07-29T00:40:02.807 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj19 2023-07-29T00:40:02.807 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FIiSFNW5IW2poc8FOqYr3BppoH4= 2023-07-29T00:40:02.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FIiSFNW5IW2poc8FOqYr3BppoH4=', 'Content-Length': '0'} 2023-07-29T00:40:02.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e03aed4c787a341a-0064c45fe2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-07-29T00:40:02.811 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:40:02.811 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:40:02.811 DEBUG:boto:Method: GET 2023-07-29T00:40:02.812 DEBUG:boto:Path: /hiiqos-59/obj19 2023-07-29T00:40:02.812 DEBUG:boto:Data: 2023-07-29T00:40:02.813 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.813 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:02.813 DEBUG:boto:Port: 8001 2023-07-29T00:40:02.814 DEBUG:boto:Params: {} 2023-07-29T00:40:02.814 DEBUG:boto:Token: None 2023-07-29T00:40:02.814 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj19 2023-07-29T00:40:02.815 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7+jU+ztRFenxfHY40pVr0njC81k= 2023-07-29T00:40:02.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7+jU+ztRFenxfHY40pVr0njC81k=', 'Content-Length': '0'} 2023-07-29T00:40:02.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e57e839e93efe4da-0064c45fe2-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.818 DEBUG:boto:path=/hiiqos-59/obj19 2023-07-29T00:40:02.818 DEBUG:boto:auth_path=/hiiqos-59/obj19 2023-07-29T00:40:02.818 DEBUG:boto:Method: GET 2023-07-29T00:40:02.819 DEBUG:boto:Path: /hiiqos-59/obj19 2023-07-29T00:40:02.819 DEBUG:boto:Data: 2023-07-29T00:40:02.819 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.820 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:02.820 DEBUG:boto:Port: 8000 2023-07-29T00:40:02.821 DEBUG:boto:Params: {} 2023-07-29T00:40:02.821 DEBUG:boto:Token: None 2023-07-29T00:40:02.822 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj19 2023-07-29T00:40:02.822 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7+jU+ztRFenxfHY40pVr0njC81k= 2023-07-29T00:40:02.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7+jU+ztRFenxfHY40pVr0njC81k=', 'Content-Length': '0'} 2023-07-29T00:40:02.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c84433f24a4f37dc-0064c45fe2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.825 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:02.825 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:02.826 DEBUG:boto:path=/hiiqos-59/?prefix=obj19 2023-07-29T00:40:02.826 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj19 2023-07-29T00:40:02.826 DEBUG:boto:Method: GET 2023-07-29T00:40:02.827 DEBUG:boto:Path: /hiiqos-59/?prefix=obj19 2023-07-29T00:40:02.827 DEBUG:boto:Data: 2023-07-29T00:40:02.828 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.828 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:02.828 DEBUG:boto:Port: 8001 2023-07-29T00:40:02.829 DEBUG:boto:Params: {} 2023-07-29T00:40:02.829 DEBUG:boto:Token: None 2023-07-29T00:40:02.829 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/ 2023-07-29T00:40:02.830 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DAd2RdF9oane3LajXOMXR/uMzXM= 2023-07-29T00:40:02.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DAd2RdF9oane3LajXOMXR/uMzXM=', 'Content-Length': '0'} 2023-07-29T00:40:02.833 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035c4fef6596a9134-0064c45fe2-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.833 DEBUG:boto:b'hiiqos-59obj191000falseobj192023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:02.834 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:02.834 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:02.835 DEBUG:boto:path=/hiiqos-59/?prefix=obj19 2023-07-29T00:40:02.835 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj19 2023-07-29T00:40:02.835 DEBUG:boto:Method: GET 2023-07-29T00:40:02.836 DEBUG:boto:Path: /hiiqos-59/?prefix=obj19 2023-07-29T00:40:02.836 DEBUG:boto:Data: 2023-07-29T00:40:02.836 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.837 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:02.837 DEBUG:boto:Port: 8000 2023-07-29T00:40:02.837 DEBUG:boto:Params: {} 2023-07-29T00:40:02.838 DEBUG:boto:Token: None 2023-07-29T00:40:02.838 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/ 2023-07-29T00:40:02.839 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DAd2RdF9oane3LajXOMXR/uMzXM= 2023-07-29T00:40:02.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DAd2RdF9oane3LajXOMXR/uMzXM=', 'Content-Length': '0'} 2023-07-29T00:40:02.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8d7c0abb5f9350b-0064c45fe2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.844 DEBUG:boto:b'hiiqos-59obj191000falseobj192023-07-29T00:29:58.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:02.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-07-29T00:40:02.845 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:40:02.845 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:40:02.846 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:40:02.846 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:40:02.846 DEBUG:boto:Method: GET 2023-07-29T00:40:02.847 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:40:02.847 DEBUG:boto:Data: 2023-07-29T00:40:02.847 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.848 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:02.851 DEBUG:boto:Port: 8001 2023-07-29T00:40:02.852 DEBUG:boto:Params: {} 2023-07-29T00:40:02.852 DEBUG:boto:Token: None 2023-07-29T00:40:02.853 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:40:02.853 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2EehuD2UwvN61EbRTEfu3F0bCI0= 2023-07-29T00:40:02.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2EehuD2UwvN61EbRTEfu3F0bCI0=', 'Content-Length': '0'} 2023-07-29T00:40:02.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002961720953863f57-0064c45fe2-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.896 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:40:02.897 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:40:02.897 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:40:02.898 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:40:02.898 DEBUG:boto:Method: GET 2023-07-29T00:40:02.898 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:40:02.899 DEBUG:boto:Data: 2023-07-29T00:40:02.899 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.899 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:02.900 DEBUG:boto:Port: 8000 2023-07-29T00:40:02.900 DEBUG:boto:Params: {} 2023-07-29T00:40:02.901 DEBUG:boto:Token: None 2023-07-29T00:40:02.901 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:40:02.901 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2EehuD2UwvN61EbRTEfu3F0bCI0= 2023-07-29T00:40:02.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2EehuD2UwvN61EbRTEfu3F0bCI0=', 'Content-Length': '0'} 2023-07-29T00:40:02.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051a6f56c89dad136-0064c45fe2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.905 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:40:02.905 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:40:02.906 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:40:02.906 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:40:02.906 DEBUG:boto:Method: HEAD 2023-07-29T00:40:02.907 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:40:02.907 DEBUG:boto:Data: 2023-07-29T00:40:02.907 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.908 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:02.908 DEBUG:boto:Port: 8001 2023-07-29T00:40:02.909 DEBUG:boto:Params: {} 2023-07-29T00:40:02.909 DEBUG:boto:Token: None 2023-07-29T00:40:02.909 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:40:02.910 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:i0kefBeOpBVX+Xvmwk2tFRPm+M0= 2023-07-29T00:40:02.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:i0kefBeOpBVX+Xvmwk2tFRPm+M0=', 'Content-Length': '0'} 2023-07-29T00:40:02.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e18d66450e389870-0064c45fe2-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.913 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:40:02.913 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:40:02.913 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:40:02.914 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:40:02.914 DEBUG:boto:Method: HEAD 2023-07-29T00:40:02.914 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:40:02.915 DEBUG:boto:Data: 2023-07-29T00:40:02.915 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.915 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:02.916 DEBUG:boto:Port: 8000 2023-07-29T00:40:02.916 DEBUG:boto:Params: {} 2023-07-29T00:40:02.916 DEBUG:boto:Token: None 2023-07-29T00:40:02.917 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:40:02.917 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:i0kefBeOpBVX+Xvmwk2tFRPm+M0= 2023-07-29T00:40:02.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:i0kefBeOpBVX+Xvmwk2tFRPm+M0=', 'Content-Length': '0'} 2023-07-29T00:40:02.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059a647150071ad9f-0064c45fe2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-07-29T00:40:02.921 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:40:02.921 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:40:02.921 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:40:02.922 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:40:02.922 DEBUG:boto:Method: GET 2023-07-29T00:40:02.923 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:40:02.923 DEBUG:boto:Data: 2023-07-29T00:40:02.923 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.924 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:02.924 DEBUG:boto:Port: 8001 2023-07-29T00:40:02.924 DEBUG:boto:Params: {} 2023-07-29T00:40:02.925 DEBUG:boto:Token: None 2023-07-29T00:40:02.925 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:40:02.926 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2EehuD2UwvN61EbRTEfu3F0bCI0= 2023-07-29T00:40:02.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2EehuD2UwvN61EbRTEfu3F0bCI0=', 'Content-Length': '0'} 2023-07-29T00:40:02.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000206210960ad631df-0064c45fe2-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.930 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:40:02.931 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:40:02.931 DEBUG:boto:path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:40:02.932 DEBUG:boto:auth_path=/hiiqos-59/obj22?versionId=null 2023-07-29T00:40:02.933 DEBUG:boto:Method: GET 2023-07-29T00:40:02.933 DEBUG:boto:Path: /hiiqos-59/obj22?versionId=null 2023-07-29T00:40:02.934 DEBUG:boto:Data: 2023-07-29T00:40:02.935 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.935 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:02.936 DEBUG:boto:Port: 8000 2023-07-29T00:40:02.936 DEBUG:boto:Params: {} 2023-07-29T00:40:02.936 DEBUG:boto:Token: None 2023-07-29T00:40:02.937 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/obj22?versionId=null 2023-07-29T00:40:02.937 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2EehuD2UwvN61EbRTEfu3F0bCI0= 2023-07-29T00:40:02.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2EehuD2UwvN61EbRTEfu3F0bCI0=', 'Content-Length': '0'} 2023-07-29T00:40:02.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018b1e5d3ac084643-0064c45fe2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.985 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:02.987 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:02.988 DEBUG:boto:path=/hiiqos-59/?prefix=obj22 2023-07-29T00:40:02.989 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj22 2023-07-29T00:40:02.989 DEBUG:boto:Method: GET 2023-07-29T00:40:02.990 DEBUG:boto:Path: /hiiqos-59/?prefix=obj22 2023-07-29T00:40:02.991 DEBUG:boto:Data: 2023-07-29T00:40:02.991 DEBUG:boto:Headers: {} 2023-07-29T00:40:02.992 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:02.992 DEBUG:boto:Port: 8001 2023-07-29T00:40:02.993 DEBUG:boto:Params: {} 2023-07-29T00:40:02.994 DEBUG:boto:Token: None 2023-07-29T00:40:02.995 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:02 GMT /hiiqos-59/ 2023-07-29T00:40:02.995 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DAd2RdF9oane3LajXOMXR/uMzXM= 2023-07-29T00:40:02.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DAd2RdF9oane3LajXOMXR/uMzXM=', 'Content-Length': '0'} 2023-07-29T00:40:02.999 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000330cb7c5c3b828e5-0064c45fe2-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:02.999 DEBUG:boto:b'hiiqos-59obj221000falseobj222023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:03.000 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:03.001 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:03.002 DEBUG:boto:path=/hiiqos-59/?prefix=obj22 2023-07-29T00:40:03.002 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj22 2023-07-29T00:40:03.003 DEBUG:boto:Method: GET 2023-07-29T00:40:03.003 DEBUG:boto:Path: /hiiqos-59/?prefix=obj22 2023-07-29T00:40:03.004 DEBUG:boto:Data: 2023-07-29T00:40:03.004 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.005 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:03.005 DEBUG:boto:Port: 8000 2023-07-29T00:40:03.006 DEBUG:boto:Params: {} 2023-07-29T00:40:03.007 DEBUG:boto:Token: None 2023-07-29T00:40:03.008 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/ 2023-07-29T00:40:03.008 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JN8e9nTO+13r1+V6jH1TEK+0T9Q= 2023-07-29T00:40:03.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JN8e9nTO+13r1+V6jH1TEK+0T9Q=', 'Content-Length': '0'} 2023-07-29T00:40:03.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017b4cfbee093242d-0064c45fe2-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.015 DEBUG:boto:b'hiiqos-59obj221000falseobj222023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:03.016 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:40:03.016 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:40:03.017 DEBUG:boto:Method: HEAD 2023-07-29T00:40:03.017 DEBUG:boto:Path: /hiiqos-59/obj22 2023-07-29T00:40:03.018 DEBUG:boto:Data: 2023-07-29T00:40:03.018 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.019 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:03.019 DEBUG:boto:Port: 8001 2023-07-29T00:40:03.020 DEBUG:boto:Params: {} 2023-07-29T00:40:03.020 DEBUG:boto:Token: None 2023-07-29T00:40:03.021 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj22 2023-07-29T00:40:03.022 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OmdKL/sKreXA3SxMSySKujVKL+M= 2023-07-29T00:40:03.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OmdKL/sKreXA3SxMSySKujVKL+M=', 'Content-Length': '0'} 2023-07-29T00:40:03.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002088320d98fc0511-0064c45fe2-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.026 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:40:03.026 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:40:03.027 DEBUG:boto:Method: HEAD 2023-07-29T00:40:03.027 DEBUG:boto:Path: /hiiqos-59/obj22 2023-07-29T00:40:03.028 DEBUG:boto:Data: 2023-07-29T00:40:03.028 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.029 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:03.030 DEBUG:boto:Port: 8000 2023-07-29T00:40:03.030 DEBUG:boto:Params: {} 2023-07-29T00:40:03.031 DEBUG:boto:Token: None 2023-07-29T00:40:03.032 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj22 2023-07-29T00:40:03.032 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OmdKL/sKreXA3SxMSySKujVKL+M= 2023-07-29T00:40:03.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OmdKL/sKreXA3SxMSySKujVKL+M=', 'Content-Length': '0'} 2023-07-29T00:40:03.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8c386adf673c999-0064c45fe2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-07-29T00:40:03.037 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:40:03.037 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:40:03.038 DEBUG:boto:Method: GET 2023-07-29T00:40:03.038 DEBUG:boto:Path: /hiiqos-59/obj22 2023-07-29T00:40:03.039 DEBUG:boto:Data: 2023-07-29T00:40:03.040 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.040 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:03.041 DEBUG:boto:Port: 8001 2023-07-29T00:40:03.041 DEBUG:boto:Params: {} 2023-07-29T00:40:03.042 DEBUG:boto:Token: None 2023-07-29T00:40:03.042 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj22 2023-07-29T00:40:03.043 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QyCqalQEYZA59GsahwwPvGMr+Ew= 2023-07-29T00:40:03.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QyCqalQEYZA59GsahwwPvGMr+Ew=', 'Content-Length': '0'} 2023-07-29T00:40:03.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d65fbee8f661d6bb-0064c45fe2-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.047 DEBUG:boto:path=/hiiqos-59/obj22 2023-07-29T00:40:03.048 DEBUG:boto:auth_path=/hiiqos-59/obj22 2023-07-29T00:40:03.048 DEBUG:boto:Method: GET 2023-07-29T00:40:03.048 DEBUG:boto:Path: /hiiqos-59/obj22 2023-07-29T00:40:03.049 DEBUG:boto:Data: 2023-07-29T00:40:03.049 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.050 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:03.050 DEBUG:boto:Port: 8000 2023-07-29T00:40:03.050 DEBUG:boto:Params: {} 2023-07-29T00:40:03.051 DEBUG:boto:Token: None 2023-07-29T00:40:03.051 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj22 2023-07-29T00:40:03.052 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QyCqalQEYZA59GsahwwPvGMr+Ew= 2023-07-29T00:40:03.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QyCqalQEYZA59GsahwwPvGMr+Ew=', 'Content-Length': '0'} 2023-07-29T00:40:03.055 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071e97773659e5ef9-0064c45fe2-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.055 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:03.056 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:03.056 DEBUG:boto:path=/hiiqos-59/?prefix=obj22 2023-07-29T00:40:03.057 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj22 2023-07-29T00:40:03.057 DEBUG:boto:Method: GET 2023-07-29T00:40:03.057 DEBUG:boto:Path: /hiiqos-59/?prefix=obj22 2023-07-29T00:40:03.058 DEBUG:boto:Data: 2023-07-29T00:40:03.058 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.058 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:03.059 DEBUG:boto:Port: 8001 2023-07-29T00:40:03.059 DEBUG:boto:Params: {} 2023-07-29T00:40:03.059 DEBUG:boto:Token: None 2023-07-29T00:40:03.060 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/ 2023-07-29T00:40:03.060 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JN8e9nTO+13r1+V6jH1TEK+0T9Q= 2023-07-29T00:40:03.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JN8e9nTO+13r1+V6jH1TEK+0T9Q=', 'Content-Length': '0'} 2023-07-29T00:40:03.064 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016743980f705faf8-0064c45fe2-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.065 DEBUG:boto:b'hiiqos-59obj221000falseobj222023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:03.066 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:03.066 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:03.066 DEBUG:boto:path=/hiiqos-59/?prefix=obj22 2023-07-29T00:40:03.067 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj22 2023-07-29T00:40:03.067 DEBUG:boto:Method: GET 2023-07-29T00:40:03.068 DEBUG:boto:Path: /hiiqos-59/?prefix=obj22 2023-07-29T00:40:03.069 DEBUG:boto:Data: 2023-07-29T00:40:03.069 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.070 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:03.070 DEBUG:boto:Port: 8000 2023-07-29T00:40:03.071 DEBUG:boto:Params: {} 2023-07-29T00:40:03.071 DEBUG:boto:Token: None 2023-07-29T00:40:03.072 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/ 2023-07-29T00:40:03.072 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JN8e9nTO+13r1+V6jH1TEK+0T9Q= 2023-07-29T00:40:03.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JN8e9nTO+13r1+V6jH1TEK+0T9Q=', 'Content-Length': '0'} 2023-07-29T00:40:03.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099fa7e7db12573e8-0064c45fe3-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.079 DEBUG:boto:b'hiiqos-59obj221000falseobj222023-07-29T00:26:54.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:03.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-07-29T00:40:03.081 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:40:03.081 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:40:03.082 DEBUG:boto:path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:40:03.082 DEBUG:boto:auth_path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:40:03.083 DEBUG:boto:Method: GET 2023-07-29T00:40:03.083 DEBUG:boto:Path: /hiiqos-59/obj23?versionId=null 2023-07-29T00:40:03.084 DEBUG:boto:Data: 2023-07-29T00:40:03.085 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.085 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:03.086 DEBUG:boto:Port: 8001 2023-07-29T00:40:03.086 DEBUG:boto:Params: {} 2023-07-29T00:40:03.086 DEBUG:boto:Token: None 2023-07-29T00:40:03.087 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj23?versionId=null 2023-07-29T00:40:03.087 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+ooLeri4E56Fe1wRrOAo6xp99+g= 2023-07-29T00:40:03.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+ooLeri4E56Fe1wRrOAo6xp99+g=', 'Content-Length': '0'} 2023-07-29T00:40:03.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b3de7ea4d92c959-0064c45fe3-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.132 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:40:03.133 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:40:03.134 DEBUG:boto:path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:40:03.135 DEBUG:boto:auth_path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:40:03.135 DEBUG:boto:Method: GET 2023-07-29T00:40:03.136 DEBUG:boto:Path: /hiiqos-59/obj23?versionId=null 2023-07-29T00:40:03.137 DEBUG:boto:Data: 2023-07-29T00:40:03.138 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.138 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:03.139 DEBUG:boto:Port: 8000 2023-07-29T00:40:03.140 DEBUG:boto:Params: {} 2023-07-29T00:40:03.141 DEBUG:boto:Token: None 2023-07-29T00:40:03.142 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj23?versionId=null 2023-07-29T00:40:03.142 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+ooLeri4E56Fe1wRrOAo6xp99+g= 2023-07-29T00:40:03.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+ooLeri4E56Fe1wRrOAo6xp99+g=', 'Content-Length': '0'} 2023-07-29T00:40:03.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000915600ad053719a0-0064c45fe3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.146 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:40:03.147 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:40:03.147 DEBUG:boto:path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:40:03.148 DEBUG:boto:auth_path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:40:03.148 DEBUG:boto:Method: HEAD 2023-07-29T00:40:03.149 DEBUG:boto:Path: /hiiqos-59/obj23?versionId=null 2023-07-29T00:40:03.149 DEBUG:boto:Data: 2023-07-29T00:40:03.150 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.150 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:03.150 DEBUG:boto:Port: 8001 2023-07-29T00:40:03.150 DEBUG:boto:Params: {} 2023-07-29T00:40:03.151 DEBUG:boto:Token: None 2023-07-29T00:40:03.151 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj23?versionId=null 2023-07-29T00:40:03.152 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Udl1NGwS8R+2izk+rhI5+/HAQZo= 2023-07-29T00:40:03.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Udl1NGwS8R+2izk+rhI5+/HAQZo=', 'Content-Length': '0'} 2023-07-29T00:40:03.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a503f1b8b7ea345-0064c45fe3-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.155 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:40:03.155 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:40:03.156 DEBUG:boto:path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:40:03.156 DEBUG:boto:auth_path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:40:03.157 DEBUG:boto:Method: HEAD 2023-07-29T00:40:03.157 DEBUG:boto:Path: /hiiqos-59/obj23?versionId=null 2023-07-29T00:40:03.157 DEBUG:boto:Data: 2023-07-29T00:40:03.158 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.158 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:03.159 DEBUG:boto:Port: 8000 2023-07-29T00:40:03.159 DEBUG:boto:Params: {} 2023-07-29T00:40:03.160 DEBUG:boto:Token: None 2023-07-29T00:40:03.160 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj23?versionId=null 2023-07-29T00:40:03.161 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Udl1NGwS8R+2izk+rhI5+/HAQZo= 2023-07-29T00:40:03.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Udl1NGwS8R+2izk+rhI5+/HAQZo=', 'Content-Length': '0'} 2023-07-29T00:40:03.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004da3a07c2c6d2e13-0064c45fe3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-07-29T00:40:03.164 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:40:03.165 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:40:03.165 DEBUG:boto:path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:40:03.165 DEBUG:boto:auth_path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:40:03.166 DEBUG:boto:Method: GET 2023-07-29T00:40:03.166 DEBUG:boto:Path: /hiiqos-59/obj23?versionId=null 2023-07-29T00:40:03.166 DEBUG:boto:Data: 2023-07-29T00:40:03.167 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.167 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:03.168 DEBUG:boto:Port: 8001 2023-07-29T00:40:03.169 DEBUG:boto:Params: {} 2023-07-29T00:40:03.170 DEBUG:boto:Token: None 2023-07-29T00:40:03.170 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj23?versionId=null 2023-07-29T00:40:03.171 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+ooLeri4E56Fe1wRrOAo6xp99+g= 2023-07-29T00:40:03.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+ooLeri4E56Fe1wRrOAo6xp99+g=', 'Content-Length': '0'} 2023-07-29T00:40:03.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091ea507294e839cf-0064c45fe3-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.174 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:40:03.174 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:40:03.175 DEBUG:boto:path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:40:03.175 DEBUG:boto:auth_path=/hiiqos-59/obj23?versionId=null 2023-07-29T00:40:03.175 DEBUG:boto:Method: GET 2023-07-29T00:40:03.176 DEBUG:boto:Path: /hiiqos-59/obj23?versionId=null 2023-07-29T00:40:03.176 DEBUG:boto:Data: 2023-07-29T00:40:03.177 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.177 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:03.178 DEBUG:boto:Port: 8000 2023-07-29T00:40:03.178 DEBUG:boto:Params: {} 2023-07-29T00:40:03.179 DEBUG:boto:Token: None 2023-07-29T00:40:03.179 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj23?versionId=null 2023-07-29T00:40:03.180 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+ooLeri4E56Fe1wRrOAo6xp99+g= 2023-07-29T00:40:03.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+ooLeri4E56Fe1wRrOAo6xp99+g=', 'Content-Length': '0'} 2023-07-29T00:40:03.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005be6ad5b1b6553d0-0064c45fe3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.225 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:03.226 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:03.226 DEBUG:boto:path=/hiiqos-59/?prefix=obj23 2023-07-29T00:40:03.227 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj23 2023-07-29T00:40:03.228 DEBUG:boto:Method: GET 2023-07-29T00:40:03.228 DEBUG:boto:Path: /hiiqos-59/?prefix=obj23 2023-07-29T00:40:03.229 DEBUG:boto:Data: 2023-07-29T00:40:03.229 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.229 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:03.230 DEBUG:boto:Port: 8001 2023-07-29T00:40:03.230 DEBUG:boto:Params: {} 2023-07-29T00:40:03.231 DEBUG:boto:Token: None 2023-07-29T00:40:03.231 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/ 2023-07-29T00:40:03.232 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JN8e9nTO+13r1+V6jH1TEK+0T9Q= 2023-07-29T00:40:03.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JN8e9nTO+13r1+V6jH1TEK+0T9Q=', 'Content-Length': '0'} 2023-07-29T00:40:03.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008518e25935074cca-0064c45fe3-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.236 DEBUG:boto:b'hiiqos-59obj231000falseobj232023-07-29T00:33:06.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:03.237 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:03.238 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:03.238 DEBUG:boto:path=/hiiqos-59/?prefix=obj23 2023-07-29T00:40:03.238 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj23 2023-07-29T00:40:03.239 DEBUG:boto:Method: GET 2023-07-29T00:40:03.239 DEBUG:boto:Path: /hiiqos-59/?prefix=obj23 2023-07-29T00:40:03.240 DEBUG:boto:Data: 2023-07-29T00:40:03.240 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.240 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:03.241 DEBUG:boto:Port: 8000 2023-07-29T00:40:03.241 DEBUG:boto:Params: {} 2023-07-29T00:40:03.242 DEBUG:boto:Token: None 2023-07-29T00:40:03.242 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/ 2023-07-29T00:40:03.243 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JN8e9nTO+13r1+V6jH1TEK+0T9Q= 2023-07-29T00:40:03.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JN8e9nTO+13r1+V6jH1TEK+0T9Q=', 'Content-Length': '0'} 2023-07-29T00:40:03.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9776fd53ac4663d-0064c45fe3-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.249 DEBUG:boto:b'hiiqos-59obj231000falseobj232023-07-29T00:33:06.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:03.250 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:40:03.250 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:40:03.251 DEBUG:boto:Method: HEAD 2023-07-29T00:40:03.252 DEBUG:boto:Path: /hiiqos-59/obj23 2023-07-29T00:40:03.253 DEBUG:boto:Data: 2023-07-29T00:40:03.253 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.254 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:03.254 DEBUG:boto:Port: 8001 2023-07-29T00:40:03.254 DEBUG:boto:Params: {} 2023-07-29T00:40:03.255 DEBUG:boto:Token: None 2023-07-29T00:40:03.256 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj23 2023-07-29T00:40:03.256 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3Rfc8QU80hBCj68Uh4+o6iksygM= 2023-07-29T00:40:03.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3Rfc8QU80hBCj68Uh4+o6iksygM=', 'Content-Length': '0'} 2023-07-29T00:40:03.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090db264003ee2ea0-0064c45fe3-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.260 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:40:03.260 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:40:03.260 DEBUG:boto:Method: HEAD 2023-07-29T00:40:03.261 DEBUG:boto:Path: /hiiqos-59/obj23 2023-07-29T00:40:03.261 DEBUG:boto:Data: 2023-07-29T00:40:03.262 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.262 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:03.262 DEBUG:boto:Port: 8000 2023-07-29T00:40:03.263 DEBUG:boto:Params: {} 2023-07-29T00:40:03.263 DEBUG:boto:Token: None 2023-07-29T00:40:03.264 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj23 2023-07-29T00:40:03.264 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3Rfc8QU80hBCj68Uh4+o6iksygM= 2023-07-29T00:40:03.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3Rfc8QU80hBCj68Uh4+o6iksygM=', 'Content-Length': '0'} 2023-07-29T00:40:03.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000662f46a83ad4159e-0064c45fe3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-07-29T00:40:03.267 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:40:03.268 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:40:03.269 DEBUG:boto:Method: GET 2023-07-29T00:40:03.269 DEBUG:boto:Path: /hiiqos-59/obj23 2023-07-29T00:40:03.270 DEBUG:boto:Data: 2023-07-29T00:40:03.270 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.271 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:03.271 DEBUG:boto:Port: 8001 2023-07-29T00:40:03.272 DEBUG:boto:Params: {} 2023-07-29T00:40:03.272 DEBUG:boto:Token: None 2023-07-29T00:40:03.273 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj23 2023-07-29T00:40:03.273 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:iRSNKbds6Jle1c94+uzw1nVJVWA= 2023-07-29T00:40:03.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:iRSNKbds6Jle1c94+uzw1nVJVWA=', 'Content-Length': '0'} 2023-07-29T00:40:03.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b952baa0e76d20d2-0064c45fe3-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.276 DEBUG:boto:path=/hiiqos-59/obj23 2023-07-29T00:40:03.276 DEBUG:boto:auth_path=/hiiqos-59/obj23 2023-07-29T00:40:03.277 DEBUG:boto:Method: GET 2023-07-29T00:40:03.277 DEBUG:boto:Path: /hiiqos-59/obj23 2023-07-29T00:40:03.277 DEBUG:boto:Data: 2023-07-29T00:40:03.278 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.278 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:03.279 DEBUG:boto:Port: 8000 2023-07-29T00:40:03.279 DEBUG:boto:Params: {} 2023-07-29T00:40:03.279 DEBUG:boto:Token: None 2023-07-29T00:40:03.280 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj23 2023-07-29T00:40:03.280 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:iRSNKbds6Jle1c94+uzw1nVJVWA= 2023-07-29T00:40:03.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:iRSNKbds6Jle1c94+uzw1nVJVWA=', 'Content-Length': '0'} 2023-07-29T00:40:03.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c46516917fc8547f-0064c45fe3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.283 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:03.283 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:03.284 DEBUG:boto:path=/hiiqos-59/?prefix=obj23 2023-07-29T00:40:03.284 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj23 2023-07-29T00:40:03.284 DEBUG:boto:Method: GET 2023-07-29T00:40:03.285 DEBUG:boto:Path: /hiiqos-59/?prefix=obj23 2023-07-29T00:40:03.285 DEBUG:boto:Data: 2023-07-29T00:40:03.286 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.286 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:03.286 DEBUG:boto:Port: 8001 2023-07-29T00:40:03.287 DEBUG:boto:Params: {} 2023-07-29T00:40:03.287 DEBUG:boto:Token: None 2023-07-29T00:40:03.287 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/ 2023-07-29T00:40:03.288 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JN8e9nTO+13r1+V6jH1TEK+0T9Q= 2023-07-29T00:40:03.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JN8e9nTO+13r1+V6jH1TEK+0T9Q=', 'Content-Length': '0'} 2023-07-29T00:40:03.291 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039c1463298cae0a2-0064c45fe3-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.291 DEBUG:boto:b'hiiqos-59obj231000falseobj232023-07-29T00:33:06.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:03.292 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:03.292 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:03.293 DEBUG:boto:path=/hiiqos-59/?prefix=obj23 2023-07-29T00:40:03.293 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj23 2023-07-29T00:40:03.294 DEBUG:boto:Method: GET 2023-07-29T00:40:03.294 DEBUG:boto:Path: /hiiqos-59/?prefix=obj23 2023-07-29T00:40:03.294 DEBUG:boto:Data: 2023-07-29T00:40:03.295 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.295 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:03.295 DEBUG:boto:Port: 8000 2023-07-29T00:40:03.296 DEBUG:boto:Params: {} 2023-07-29T00:40:03.296 DEBUG:boto:Token: None 2023-07-29T00:40:03.297 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/ 2023-07-29T00:40:03.297 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JN8e9nTO+13r1+V6jH1TEK+0T9Q= 2023-07-29T00:40:03.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JN8e9nTO+13r1+V6jH1TEK+0T9Q=', 'Content-Length': '0'} 2023-07-29T00:40:03.301 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e5b75ee33c993a4-0064c45fe3-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.302 DEBUG:boto:b'hiiqos-59obj231000falseobj232023-07-29T00:33:06.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:03.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-07-29T00:40:03.303 DEBUG:boto:path=/hiiqos-59/obj29 2023-07-29T00:40:03.304 DEBUG:boto:auth_path=/hiiqos-59/obj29 2023-07-29T00:40:03.304 DEBUG:boto:path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:40:03.304 DEBUG:boto:auth_path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:40:03.305 DEBUG:boto:Method: GET 2023-07-29T00:40:03.305 DEBUG:boto:Path: /hiiqos-59/obj29?versionId=null 2023-07-29T00:40:03.305 DEBUG:boto:Data: 2023-07-29T00:40:03.306 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.306 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:03.306 DEBUG:boto:Port: 8001 2023-07-29T00:40:03.307 DEBUG:boto:Params: {} 2023-07-29T00:40:03.307 DEBUG:boto:Token: None 2023-07-29T00:40:03.308 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj29?versionId=null 2023-07-29T00:40:03.308 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:A/fgR8uhGbALV+2SccbOIr9aCW0= 2023-07-29T00:40:03.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:A/fgR8uhGbALV+2SccbOIr9aCW0=', 'Content-Length': '0'} 2023-07-29T00:40:03.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ec8ec7e171b346c-0064c45fe3-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.311 DEBUG:boto:path=/hiiqos-59/obj29 2023-07-29T00:40:03.312 DEBUG:boto:auth_path=/hiiqos-59/obj29 2023-07-29T00:40:03.312 DEBUG:boto:path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:40:03.312 DEBUG:boto:auth_path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:40:03.313 DEBUG:boto:Method: GET 2023-07-29T00:40:03.313 DEBUG:boto:Path: /hiiqos-59/obj29?versionId=null 2023-07-29T00:40:03.313 DEBUG:boto:Data: 2023-07-29T00:40:03.314 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.314 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:03.315 DEBUG:boto:Port: 8000 2023-07-29T00:40:03.315 DEBUG:boto:Params: {} 2023-07-29T00:40:03.315 DEBUG:boto:Token: None 2023-07-29T00:40:03.316 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj29?versionId=null 2023-07-29T00:40:03.316 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:A/fgR8uhGbALV+2SccbOIr9aCW0= 2023-07-29T00:40:03.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:A/fgR8uhGbALV+2SccbOIr9aCW0=', 'Content-Length': '0'} 2023-07-29T00:40:03.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c86f2a0352a50bf2-0064c45fe3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.364 DEBUG:boto:path=/hiiqos-59/obj29 2023-07-29T00:40:03.365 DEBUG:boto:auth_path=/hiiqos-59/obj29 2023-07-29T00:40:03.365 DEBUG:boto:path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:40:03.366 DEBUG:boto:auth_path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:40:03.366 DEBUG:boto:Method: HEAD 2023-07-29T00:40:03.366 DEBUG:boto:Path: /hiiqos-59/obj29?versionId=null 2023-07-29T00:40:03.367 DEBUG:boto:Data: 2023-07-29T00:40:03.367 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.367 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:03.368 DEBUG:boto:Port: 8001 2023-07-29T00:40:03.368 DEBUG:boto:Params: {} 2023-07-29T00:40:03.369 DEBUG:boto:Token: None 2023-07-29T00:40:03.369 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj29?versionId=null 2023-07-29T00:40:03.370 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zTdkALbhkrPNtArJPsjXNgSogX8= 2023-07-29T00:40:03.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zTdkALbhkrPNtArJPsjXNgSogX8=', 'Content-Length': '0'} 2023-07-29T00:40:03.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcfd6d749047f5fd-0064c45fe3-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.372 DEBUG:boto:path=/hiiqos-59/obj29 2023-07-29T00:40:03.373 DEBUG:boto:auth_path=/hiiqos-59/obj29 2023-07-29T00:40:03.373 DEBUG:boto:path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:40:03.374 DEBUG:boto:auth_path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:40:03.374 DEBUG:boto:Method: HEAD 2023-07-29T00:40:03.374 DEBUG:boto:Path: /hiiqos-59/obj29?versionId=null 2023-07-29T00:40:03.375 DEBUG:boto:Data: 2023-07-29T00:40:03.375 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.375 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:03.376 DEBUG:boto:Port: 8000 2023-07-29T00:40:03.376 DEBUG:boto:Params: {} 2023-07-29T00:40:03.376 DEBUG:boto:Token: None 2023-07-29T00:40:03.377 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj29?versionId=null 2023-07-29T00:40:03.377 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zTdkALbhkrPNtArJPsjXNgSogX8= 2023-07-29T00:40:03.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zTdkALbhkrPNtArJPsjXNgSogX8=', 'Content-Length': '0'} 2023-07-29T00:40:03.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008df44c2c8b9b2783-0064c45fe3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-07-29T00:40:03.380 DEBUG:boto:path=/hiiqos-59/obj29 2023-07-29T00:40:03.381 DEBUG:boto:auth_path=/hiiqos-59/obj29 2023-07-29T00:40:03.381 DEBUG:boto:path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:40:03.382 DEBUG:boto:auth_path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:40:03.382 DEBUG:boto:Method: GET 2023-07-29T00:40:03.382 DEBUG:boto:Path: /hiiqos-59/obj29?versionId=null 2023-07-29T00:40:03.383 DEBUG:boto:Data: 2023-07-29T00:40:03.383 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.383 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:03.384 DEBUG:boto:Port: 8001 2023-07-29T00:40:03.384 DEBUG:boto:Params: {} 2023-07-29T00:40:03.385 DEBUG:boto:Token: None 2023-07-29T00:40:03.385 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj29?versionId=null 2023-07-29T00:40:03.385 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:A/fgR8uhGbALV+2SccbOIr9aCW0= 2023-07-29T00:40:03.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:A/fgR8uhGbALV+2SccbOIr9aCW0=', 'Content-Length': '0'} 2023-07-29T00:40:03.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000738433b20aacc62f-0064c45fe3-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.389 DEBUG:boto:path=/hiiqos-59/obj29 2023-07-29T00:40:03.389 DEBUG:boto:auth_path=/hiiqos-59/obj29 2023-07-29T00:40:03.389 DEBUG:boto:path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:40:03.390 DEBUG:boto:auth_path=/hiiqos-59/obj29?versionId=null 2023-07-29T00:40:03.390 DEBUG:boto:Method: GET 2023-07-29T00:40:03.390 DEBUG:boto:Path: /hiiqos-59/obj29?versionId=null 2023-07-29T00:40:03.391 DEBUG:boto:Data: 2023-07-29T00:40:03.391 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.391 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:03.392 DEBUG:boto:Port: 8000 2023-07-29T00:40:03.392 DEBUG:boto:Params: {} 2023-07-29T00:40:03.393 DEBUG:boto:Token: None 2023-07-29T00:40:03.393 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj29?versionId=null 2023-07-29T00:40:03.394 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:A/fgR8uhGbALV+2SccbOIr9aCW0= 2023-07-29T00:40:03.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:A/fgR8uhGbALV+2SccbOIr9aCW0=', 'Content-Length': '0'} 2023-07-29T00:40:03.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a65bd99916ea72f-0064c45fe3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.397 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:03.397 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:03.397 DEBUG:boto:path=/hiiqos-59/?prefix=obj29 2023-07-29T00:40:03.398 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj29 2023-07-29T00:40:03.398 DEBUG:boto:Method: GET 2023-07-29T00:40:03.398 DEBUG:boto:Path: /hiiqos-59/?prefix=obj29 2023-07-29T00:40:03.399 DEBUG:boto:Data: 2023-07-29T00:40:03.399 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.400 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:03.400 DEBUG:boto:Port: 8001 2023-07-29T00:40:03.400 DEBUG:boto:Params: {} 2023-07-29T00:40:03.401 DEBUG:boto:Token: None 2023-07-29T00:40:03.401 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/ 2023-07-29T00:40:03.402 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JN8e9nTO+13r1+V6jH1TEK+0T9Q= 2023-07-29T00:40:03.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JN8e9nTO+13r1+V6jH1TEK+0T9Q=', 'Content-Length': '0'} 2023-07-29T00:40:03.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b83b259f857528a-0064c45fe3-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.405 DEBUG:boto:b'hiiqos-59obj291000falseobj292023-07-29T00:35:01.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:03.406 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:03.406 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:03.406 DEBUG:boto:path=/hiiqos-59/?prefix=obj29 2023-07-29T00:40:03.407 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj29 2023-07-29T00:40:03.407 DEBUG:boto:Method: GET 2023-07-29T00:40:03.407 DEBUG:boto:Path: /hiiqos-59/?prefix=obj29 2023-07-29T00:40:03.408 DEBUG:boto:Data: 2023-07-29T00:40:03.408 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.409 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:03.409 DEBUG:boto:Port: 8000 2023-07-29T00:40:03.409 DEBUG:boto:Params: {} 2023-07-29T00:40:03.410 DEBUG:boto:Token: None 2023-07-29T00:40:03.410 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/ 2023-07-29T00:40:03.410 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JN8e9nTO+13r1+V6jH1TEK+0T9Q= 2023-07-29T00:40:03.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JN8e9nTO+13r1+V6jH1TEK+0T9Q=', 'Content-Length': '0'} 2023-07-29T00:40:03.415 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000183b41475388cfac-0064c45fe3-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.415 DEBUG:boto:b'hiiqos-59obj291000falseobj292023-07-29T00:35:01.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:03.416 DEBUG:boto:path=/hiiqos-59/obj29 2023-07-29T00:40:03.416 DEBUG:boto:auth_path=/hiiqos-59/obj29 2023-07-29T00:40:03.417 DEBUG:boto:Method: HEAD 2023-07-29T00:40:03.417 DEBUG:boto:Path: /hiiqos-59/obj29 2023-07-29T00:40:03.417 DEBUG:boto:Data: 2023-07-29T00:40:03.418 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.418 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:03.418 DEBUG:boto:Port: 8001 2023-07-29T00:40:03.419 DEBUG:boto:Params: {} 2023-07-29T00:40:03.419 DEBUG:boto:Token: None 2023-07-29T00:40:03.419 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj29 2023-07-29T00:40:03.420 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:j5Cqg3QgerxFm2F8MG2aGl2Sw8k= 2023-07-29T00:40:03.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:j5Cqg3QgerxFm2F8MG2aGl2Sw8k=', 'Content-Length': '0'} 2023-07-29T00:40:03.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ade0f1d05edeb3e-0064c45fe3-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.422 DEBUG:boto:path=/hiiqos-59/obj29 2023-07-29T00:40:03.423 DEBUG:boto:auth_path=/hiiqos-59/obj29 2023-07-29T00:40:03.423 DEBUG:boto:Method: HEAD 2023-07-29T00:40:03.424 DEBUG:boto:Path: /hiiqos-59/obj29 2023-07-29T00:40:03.424 DEBUG:boto:Data: 2023-07-29T00:40:03.424 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.425 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:03.425 DEBUG:boto:Port: 8000 2023-07-29T00:40:03.425 DEBUG:boto:Params: {} 2023-07-29T00:40:03.426 DEBUG:boto:Token: None 2023-07-29T00:40:03.426 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj29 2023-07-29T00:40:03.426 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:j5Cqg3QgerxFm2F8MG2aGl2Sw8k= 2023-07-29T00:40:03.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:j5Cqg3QgerxFm2F8MG2aGl2Sw8k=', 'Content-Length': '0'} 2023-07-29T00:40:03.429 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000406ef2eae6617ff8-0064c45fe3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-07-29T00:40:03.430 DEBUG:boto:path=/hiiqos-59/obj29 2023-07-29T00:40:03.430 DEBUG:boto:auth_path=/hiiqos-59/obj29 2023-07-29T00:40:03.431 DEBUG:boto:Method: GET 2023-07-29T00:40:03.431 DEBUG:boto:Path: /hiiqos-59/obj29 2023-07-29T00:40:03.431 DEBUG:boto:Data: 2023-07-29T00:40:03.432 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.432 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:03.433 DEBUG:boto:Port: 8001 2023-07-29T00:40:03.433 DEBUG:boto:Params: {} 2023-07-29T00:40:03.433 DEBUG:boto:Token: None 2023-07-29T00:40:03.434 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj29 2023-07-29T00:40:03.434 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:t6l9RH80PSXE4ZJbS64YaJO1/xg= 2023-07-29T00:40:03.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t6l9RH80PSXE4ZJbS64YaJO1/xg=', 'Content-Length': '0'} 2023-07-29T00:40:03.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000686477f816704902-0064c45fe3-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.437 DEBUG:boto:path=/hiiqos-59/obj29 2023-07-29T00:40:03.437 DEBUG:boto:auth_path=/hiiqos-59/obj29 2023-07-29T00:40:03.438 DEBUG:boto:Method: GET 2023-07-29T00:40:03.438 DEBUG:boto:Path: /hiiqos-59/obj29 2023-07-29T00:40:03.438 DEBUG:boto:Data: 2023-07-29T00:40:03.439 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.439 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:03.440 DEBUG:boto:Port: 8000 2023-07-29T00:40:03.440 DEBUG:boto:Params: {} 2023-07-29T00:40:03.440 DEBUG:boto:Token: None 2023-07-29T00:40:03.441 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj29 2023-07-29T00:40:03.441 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:t6l9RH80PSXE4ZJbS64YaJO1/xg= 2023-07-29T00:40:03.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t6l9RH80PSXE4ZJbS64YaJO1/xg=', 'Content-Length': '0'} 2023-07-29T00:40:03.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024e77d1e1ff12dd2-0064c45fe3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.484 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:03.485 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:03.485 DEBUG:boto:path=/hiiqos-59/?prefix=obj29 2023-07-29T00:40:03.486 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj29 2023-07-29T00:40:03.486 DEBUG:boto:Method: GET 2023-07-29T00:40:03.487 DEBUG:boto:Path: /hiiqos-59/?prefix=obj29 2023-07-29T00:40:03.487 DEBUG:boto:Data: 2023-07-29T00:40:03.487 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.488 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:03.488 DEBUG:boto:Port: 8001 2023-07-29T00:40:03.489 DEBUG:boto:Params: {} 2023-07-29T00:40:03.489 DEBUG:boto:Token: None 2023-07-29T00:40:03.490 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/ 2023-07-29T00:40:03.490 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JN8e9nTO+13r1+V6jH1TEK+0T9Q= 2023-07-29T00:40:03.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JN8e9nTO+13r1+V6jH1TEK+0T9Q=', 'Content-Length': '0'} 2023-07-29T00:40:03.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c7f7176d3e80676-0064c45fe3-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.494 DEBUG:boto:b'hiiqos-59obj291000falseobj292023-07-29T00:35:01.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:03.494 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:03.495 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:03.495 DEBUG:boto:path=/hiiqos-59/?prefix=obj29 2023-07-29T00:40:03.496 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj29 2023-07-29T00:40:03.496 DEBUG:boto:Method: GET 2023-07-29T00:40:03.496 DEBUG:boto:Path: /hiiqos-59/?prefix=obj29 2023-07-29T00:40:03.497 DEBUG:boto:Data: 2023-07-29T00:40:03.497 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.497 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:03.498 DEBUG:boto:Port: 8000 2023-07-29T00:40:03.498 DEBUG:boto:Params: {} 2023-07-29T00:40:03.499 DEBUG:boto:Token: None 2023-07-29T00:40:03.499 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/ 2023-07-29T00:40:03.499 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JN8e9nTO+13r1+V6jH1TEK+0T9Q= 2023-07-29T00:40:03.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JN8e9nTO+13r1+V6jH1TEK+0T9Q=', 'Content-Length': '0'} 2023-07-29T00:40:03.504 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fc8420849af2adc-0064c45fe3-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.504 DEBUG:boto:b'hiiqos-59obj291000falseobj292023-07-29T00:35:01.307Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:03.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-07-29T00:40:03.506 DEBUG:boto:path=/hiiqos-59/obj31 2023-07-29T00:40:03.506 DEBUG:boto:auth_path=/hiiqos-59/obj31 2023-07-29T00:40:03.506 DEBUG:boto:path=/hiiqos-59/obj31?versionId=null 2023-07-29T00:40:03.507 DEBUG:boto:auth_path=/hiiqos-59/obj31?versionId=null 2023-07-29T00:40:03.507 DEBUG:boto:Method: GET 2023-07-29T00:40:03.507 DEBUG:boto:Path: /hiiqos-59/obj31?versionId=null 2023-07-29T00:40:03.508 DEBUG:boto:Data: 2023-07-29T00:40:03.508 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.509 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:03.509 DEBUG:boto:Port: 8001 2023-07-29T00:40:03.509 DEBUG:boto:Params: {} 2023-07-29T00:40:03.510 DEBUG:boto:Token: None 2023-07-29T00:40:03.510 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj31?versionId=null 2023-07-29T00:40:03.511 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6xQb/Q9WBltYb7Nc9gy5Yr+Xe/M= 2023-07-29T00:40:03.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6xQb/Q9WBltYb7Nc9gy5Yr+Xe/M=', 'Content-Length': '0'} 2023-07-29T00:40:03.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6b261ff97b8daf1-0064c45fe3-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.556 DEBUG:boto:path=/hiiqos-59/obj31 2023-07-29T00:40:03.557 DEBUG:boto:auth_path=/hiiqos-59/obj31 2023-07-29T00:40:03.558 DEBUG:boto:path=/hiiqos-59/obj31?versionId=null 2023-07-29T00:40:03.558 DEBUG:boto:auth_path=/hiiqos-59/obj31?versionId=null 2023-07-29T00:40:03.558 DEBUG:boto:Method: GET 2023-07-29T00:40:03.559 DEBUG:boto:Path: /hiiqos-59/obj31?versionId=null 2023-07-29T00:40:03.559 DEBUG:boto:Data: 2023-07-29T00:40:03.559 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.560 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:03.560 DEBUG:boto:Port: 8000 2023-07-29T00:40:03.560 DEBUG:boto:Params: {} 2023-07-29T00:40:03.561 DEBUG:boto:Token: None 2023-07-29T00:40:03.561 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj31?versionId=null 2023-07-29T00:40:03.562 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6xQb/Q9WBltYb7Nc9gy5Yr+Xe/M= 2023-07-29T00:40:03.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6xQb/Q9WBltYb7Nc9gy5Yr+Xe/M=', 'Content-Length': '0'} 2023-07-29T00:40:03.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eea45b821659c07f-0064c45fe3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.565 DEBUG:boto:path=/hiiqos-59/obj31 2023-07-29T00:40:03.565 DEBUG:boto:auth_path=/hiiqos-59/obj31 2023-07-29T00:40:03.566 DEBUG:boto:path=/hiiqos-59/obj31?versionId=null 2023-07-29T00:40:03.566 DEBUG:boto:auth_path=/hiiqos-59/obj31?versionId=null 2023-07-29T00:40:03.566 DEBUG:boto:Method: HEAD 2023-07-29T00:40:03.567 DEBUG:boto:Path: /hiiqos-59/obj31?versionId=null 2023-07-29T00:40:03.567 DEBUG:boto:Data: 2023-07-29T00:40:03.567 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.568 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:03.568 DEBUG:boto:Port: 8001 2023-07-29T00:40:03.568 DEBUG:boto:Params: {} 2023-07-29T00:40:03.569 DEBUG:boto:Token: None 2023-07-29T00:40:03.569 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj31?versionId=null 2023-07-29T00:40:03.569 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CTdjJwXAr4L05iRwuamMgyvGPSg= 2023-07-29T00:40:03.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CTdjJwXAr4L05iRwuamMgyvGPSg=', 'Content-Length': '0'} 2023-07-29T00:40:03.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056718d29981069c2-0064c45fe3-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.572 DEBUG:boto:path=/hiiqos-59/obj31 2023-07-29T00:40:03.572 DEBUG:boto:auth_path=/hiiqos-59/obj31 2023-07-29T00:40:03.573 DEBUG:boto:path=/hiiqos-59/obj31?versionId=null 2023-07-29T00:40:03.573 DEBUG:boto:auth_path=/hiiqos-59/obj31?versionId=null 2023-07-29T00:40:03.574 DEBUG:boto:Method: HEAD 2023-07-29T00:40:03.574 DEBUG:boto:Path: /hiiqos-59/obj31?versionId=null 2023-07-29T00:40:03.574 DEBUG:boto:Data: 2023-07-29T00:40:03.575 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.575 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:03.575 DEBUG:boto:Port: 8000 2023-07-29T00:40:03.576 DEBUG:boto:Params: {} 2023-07-29T00:40:03.576 DEBUG:boto:Token: None 2023-07-29T00:40:03.576 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj31?versionId=null 2023-07-29T00:40:03.577 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CTdjJwXAr4L05iRwuamMgyvGPSg= 2023-07-29T00:40:03.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CTdjJwXAr4L05iRwuamMgyvGPSg=', 'Content-Length': '0'} 2023-07-29T00:40:03.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039eaeb1df4390f60-0064c45fe3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-07-29T00:40:03.580 DEBUG:boto:path=/hiiqos-59/obj31 2023-07-29T00:40:03.580 DEBUG:boto:auth_path=/hiiqos-59/obj31 2023-07-29T00:40:03.581 DEBUG:boto:path=/hiiqos-59/obj31?versionId=null 2023-07-29T00:40:03.581 DEBUG:boto:auth_path=/hiiqos-59/obj31?versionId=null 2023-07-29T00:40:03.582 DEBUG:boto:Method: GET 2023-07-29T00:40:03.582 DEBUG:boto:Path: /hiiqos-59/obj31?versionId=null 2023-07-29T00:40:03.582 DEBUG:boto:Data: 2023-07-29T00:40:03.583 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.583 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:03.583 DEBUG:boto:Port: 8001 2023-07-29T00:40:03.584 DEBUG:boto:Params: {} 2023-07-29T00:40:03.584 DEBUG:boto:Token: None 2023-07-29T00:40:03.585 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj31?versionId=null 2023-07-29T00:40:03.585 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6xQb/Q9WBltYb7Nc9gy5Yr+Xe/M= 2023-07-29T00:40:03.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6xQb/Q9WBltYb7Nc9gy5Yr+Xe/M=', 'Content-Length': '0'} 2023-07-29T00:40:03.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5b3b4943b5d9c1e-0064c45fe3-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.588 DEBUG:boto:path=/hiiqos-59/obj31 2023-07-29T00:40:03.589 DEBUG:boto:auth_path=/hiiqos-59/obj31 2023-07-29T00:40:03.589 DEBUG:boto:path=/hiiqos-59/obj31?versionId=null 2023-07-29T00:40:03.589 DEBUG:boto:auth_path=/hiiqos-59/obj31?versionId=null 2023-07-29T00:40:03.590 DEBUG:boto:Method: GET 2023-07-29T00:40:03.590 DEBUG:boto:Path: /hiiqos-59/obj31?versionId=null 2023-07-29T00:40:03.591 DEBUG:boto:Data: 2023-07-29T00:40:03.591 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.591 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:03.592 DEBUG:boto:Port: 8000 2023-07-29T00:40:03.592 DEBUG:boto:Params: {} 2023-07-29T00:40:03.593 DEBUG:boto:Token: None 2023-07-29T00:40:03.593 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj31?versionId=null 2023-07-29T00:40:03.593 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6xQb/Q9WBltYb7Nc9gy5Yr+Xe/M= 2023-07-29T00:40:03.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6xQb/Q9WBltYb7Nc9gy5Yr+Xe/M=', 'Content-Length': '0'} 2023-07-29T00:40:03.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001564dc534be6c23b-0064c45fe3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.596 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:03.597 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:03.597 DEBUG:boto:path=/hiiqos-59/?prefix=obj31 2023-07-29T00:40:03.598 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj31 2023-07-29T00:40:03.598 DEBUG:boto:Method: GET 2023-07-29T00:40:03.598 DEBUG:boto:Path: /hiiqos-59/?prefix=obj31 2023-07-29T00:40:03.599 DEBUG:boto:Data: 2023-07-29T00:40:03.599 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.599 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:03.600 DEBUG:boto:Port: 8001 2023-07-29T00:40:03.600 DEBUG:boto:Params: {} 2023-07-29T00:40:03.601 DEBUG:boto:Token: None 2023-07-29T00:40:03.601 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/ 2023-07-29T00:40:03.601 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JN8e9nTO+13r1+V6jH1TEK+0T9Q= 2023-07-29T00:40:03.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JN8e9nTO+13r1+V6jH1TEK+0T9Q=', 'Content-Length': '0'} 2023-07-29T00:40:03.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e3e0e473f4080e7-0064c45fe3-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.605 DEBUG:boto:b'hiiqos-59obj311000falseobj312023-07-29T00:35:31.771Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:03.606 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:03.606 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:03.606 DEBUG:boto:path=/hiiqos-59/?prefix=obj31 2023-07-29T00:40:03.607 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj31 2023-07-29T00:40:03.607 DEBUG:boto:Method: GET 2023-07-29T00:40:03.608 DEBUG:boto:Path: /hiiqos-59/?prefix=obj31 2023-07-29T00:40:03.608 DEBUG:boto:Data: 2023-07-29T00:40:03.608 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.609 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:03.609 DEBUG:boto:Port: 8000 2023-07-29T00:40:03.609 DEBUG:boto:Params: {} 2023-07-29T00:40:03.610 DEBUG:boto:Token: None 2023-07-29T00:40:03.610 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/ 2023-07-29T00:40:03.611 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JN8e9nTO+13r1+V6jH1TEK+0T9Q= 2023-07-29T00:40:03.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JN8e9nTO+13r1+V6jH1TEK+0T9Q=', 'Content-Length': '0'} 2023-07-29T00:40:03.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e91fc25ea3bac2c-0064c45fe3-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.616 DEBUG:boto:b'hiiqos-59obj311000falseobj312023-07-29T00:35:31.771Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:03.616 DEBUG:boto:path=/hiiqos-59/obj31 2023-07-29T00:40:03.617 DEBUG:boto:auth_path=/hiiqos-59/obj31 2023-07-29T00:40:03.617 DEBUG:boto:Method: HEAD 2023-07-29T00:40:03.617 DEBUG:boto:Path: /hiiqos-59/obj31 2023-07-29T00:40:03.618 DEBUG:boto:Data: 2023-07-29T00:40:03.618 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.618 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:03.619 DEBUG:boto:Port: 8001 2023-07-29T00:40:03.619 DEBUG:boto:Params: {} 2023-07-29T00:40:03.620 DEBUG:boto:Token: None 2023-07-29T00:40:03.620 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj31 2023-07-29T00:40:03.620 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:khYsfzJtiqBV9w0cRoSkXHlvMjg= 2023-07-29T00:40:03.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:khYsfzJtiqBV9w0cRoSkXHlvMjg=', 'Content-Length': '0'} 2023-07-29T00:40:03.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ab38721760c8767-0064c45fe3-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.623 DEBUG:boto:path=/hiiqos-59/obj31 2023-07-29T00:40:03.624 DEBUG:boto:auth_path=/hiiqos-59/obj31 2023-07-29T00:40:03.624 DEBUG:boto:Method: HEAD 2023-07-29T00:40:03.625 DEBUG:boto:Path: /hiiqos-59/obj31 2023-07-29T00:40:03.625 DEBUG:boto:Data: 2023-07-29T00:40:03.625 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.626 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:03.626 DEBUG:boto:Port: 8000 2023-07-29T00:40:03.626 DEBUG:boto:Params: {} 2023-07-29T00:40:03.627 DEBUG:boto:Token: None 2023-07-29T00:40:03.627 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj31 2023-07-29T00:40:03.627 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:khYsfzJtiqBV9w0cRoSkXHlvMjg= 2023-07-29T00:40:03.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:khYsfzJtiqBV9w0cRoSkXHlvMjg=', 'Content-Length': '0'} 2023-07-29T00:40:03.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000047d766e7fdf0db2-0064c45fe3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-07-29T00:40:03.631 DEBUG:boto:path=/hiiqos-59/obj31 2023-07-29T00:40:03.631 DEBUG:boto:auth_path=/hiiqos-59/obj31 2023-07-29T00:40:03.631 DEBUG:boto:Method: GET 2023-07-29T00:40:03.632 DEBUG:boto:Path: /hiiqos-59/obj31 2023-07-29T00:40:03.632 DEBUG:boto:Data: 2023-07-29T00:40:03.633 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.633 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:03.633 DEBUG:boto:Port: 8001 2023-07-29T00:40:03.634 DEBUG:boto:Params: {} 2023-07-29T00:40:03.634 DEBUG:boto:Token: None 2023-07-29T00:40:03.634 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj31 2023-07-29T00:40:03.635 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oHE5zjgSmHkvc/2FuOrQRW7xr/Q= 2023-07-29T00:40:03.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oHE5zjgSmHkvc/2FuOrQRW7xr/Q=', 'Content-Length': '0'} 2023-07-29T00:40:03.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6db731ba75962d9-0064c45fe3-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.638 DEBUG:boto:path=/hiiqos-59/obj31 2023-07-29T00:40:03.638 DEBUG:boto:auth_path=/hiiqos-59/obj31 2023-07-29T00:40:03.639 DEBUG:boto:Method: GET 2023-07-29T00:40:03.639 DEBUG:boto:Path: /hiiqos-59/obj31 2023-07-29T00:40:03.639 DEBUG:boto:Data: 2023-07-29T00:40:03.640 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.640 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:03.640 DEBUG:boto:Port: 8000 2023-07-29T00:40:03.641 DEBUG:boto:Params: {} 2023-07-29T00:40:03.641 DEBUG:boto:Token: None 2023-07-29T00:40:03.642 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj31 2023-07-29T00:40:03.642 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oHE5zjgSmHkvc/2FuOrQRW7xr/Q= 2023-07-29T00:40:03.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oHE5zjgSmHkvc/2FuOrQRW7xr/Q=', 'Content-Length': '0'} 2023-07-29T00:40:03.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f5ba4e6f5e24ec6-0064c45fe3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.688 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:03.689 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:03.690 DEBUG:boto:path=/hiiqos-59/?prefix=obj31 2023-07-29T00:40:03.690 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj31 2023-07-29T00:40:03.690 DEBUG:boto:Method: GET 2023-07-29T00:40:03.691 DEBUG:boto:Path: /hiiqos-59/?prefix=obj31 2023-07-29T00:40:03.691 DEBUG:boto:Data: 2023-07-29T00:40:03.691 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.692 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:03.692 DEBUG:boto:Port: 8001 2023-07-29T00:40:03.693 DEBUG:boto:Params: {} 2023-07-29T00:40:03.693 DEBUG:boto:Token: None 2023-07-29T00:40:03.694 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/ 2023-07-29T00:40:03.694 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JN8e9nTO+13r1+V6jH1TEK+0T9Q= 2023-07-29T00:40:03.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JN8e9nTO+13r1+V6jH1TEK+0T9Q=', 'Content-Length': '0'} 2023-07-29T00:40:03.697 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c2b945e51858cf3-0064c45fe3-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.698 DEBUG:boto:b'hiiqos-59obj311000falseobj312023-07-29T00:35:31.771Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:03.699 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:03.699 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:03.699 DEBUG:boto:path=/hiiqos-59/?prefix=obj31 2023-07-29T00:40:03.700 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj31 2023-07-29T00:40:03.700 DEBUG:boto:Method: GET 2023-07-29T00:40:03.701 DEBUG:boto:Path: /hiiqos-59/?prefix=obj31 2023-07-29T00:40:03.701 DEBUG:boto:Data: 2023-07-29T00:40:03.701 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.702 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:03.702 DEBUG:boto:Port: 8000 2023-07-29T00:40:03.702 DEBUG:boto:Params: {} 2023-07-29T00:40:03.703 DEBUG:boto:Token: None 2023-07-29T00:40:03.703 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/ 2023-07-29T00:40:03.704 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JN8e9nTO+13r1+V6jH1TEK+0T9Q= 2023-07-29T00:40:03.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JN8e9nTO+13r1+V6jH1TEK+0T9Q=', 'Content-Length': '0'} 2023-07-29T00:40:03.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e834fb6f3bfe3b67-0064c45fe3-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.710 DEBUG:boto:b'hiiqos-59obj311000falseobj312023-07-29T00:35:31.771Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:03.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-07-29T00:40:03.711 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:40:03.712 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:40:03.712 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:40:03.713 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:40:03.713 DEBUG:boto:Method: GET 2023-07-29T00:40:03.713 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:40:03.714 DEBUG:boto:Data: 2023-07-29T00:40:03.714 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.715 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:03.715 DEBUG:boto:Port: 8001 2023-07-29T00:40:03.715 DEBUG:boto:Params: {} 2023-07-29T00:40:03.716 DEBUG:boto:Token: None 2023-07-29T00:40:03.716 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:40:03.717 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qqr0UhxxRRt9wmTEnSuZn3jAWws= 2023-07-29T00:40:03.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qqr0UhxxRRt9wmTEnSuZn3jAWws=', 'Content-Length': '0'} 2023-07-29T00:40:03.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005caaa8651d82037d-0064c45fe3-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.764 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:40:03.765 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:40:03.765 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:40:03.766 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:40:03.766 DEBUG:boto:Method: GET 2023-07-29T00:40:03.767 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:40:03.767 DEBUG:boto:Data: 2023-07-29T00:40:03.767 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.768 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:03.768 DEBUG:boto:Port: 8000 2023-07-29T00:40:03.768 DEBUG:boto:Params: {} 2023-07-29T00:40:03.769 DEBUG:boto:Token: None 2023-07-29T00:40:03.769 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:40:03.770 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qqr0UhxxRRt9wmTEnSuZn3jAWws= 2023-07-29T00:40:03.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qqr0UhxxRRt9wmTEnSuZn3jAWws=', 'Content-Length': '0'} 2023-07-29T00:40:03.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3b14ba4eaf8b653-0064c45fe3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.773 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:40:03.773 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:40:03.774 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:40:03.774 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:40:03.774 DEBUG:boto:Method: HEAD 2023-07-29T00:40:03.775 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:40:03.775 DEBUG:boto:Data: 2023-07-29T00:40:03.775 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.776 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:03.776 DEBUG:boto:Port: 8001 2023-07-29T00:40:03.777 DEBUG:boto:Params: {} 2023-07-29T00:40:03.777 DEBUG:boto:Token: None 2023-07-29T00:40:03.778 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:40:03.778 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nOsIOjqL3hkt67GZnazHqyzZ8go= 2023-07-29T00:40:03.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nOsIOjqL3hkt67GZnazHqyzZ8go=', 'Content-Length': '0'} 2023-07-29T00:40:03.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000510b2569a15f26b8-0064c45fe3-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.782 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:40:03.782 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:40:03.782 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:40:03.783 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:40:03.783 DEBUG:boto:Method: HEAD 2023-07-29T00:40:03.783 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:40:03.784 DEBUG:boto:Data: 2023-07-29T00:40:03.784 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.785 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:03.785 DEBUG:boto:Port: 8000 2023-07-29T00:40:03.785 DEBUG:boto:Params: {} 2023-07-29T00:40:03.786 DEBUG:boto:Token: None 2023-07-29T00:40:03.786 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:40:03.786 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nOsIOjqL3hkt67GZnazHqyzZ8go= 2023-07-29T00:40:03.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nOsIOjqL3hkt67GZnazHqyzZ8go=', 'Content-Length': '0'} 2023-07-29T00:40:03.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085aaa6faf6c90a34-0064c45fe3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-07-29T00:40:03.790 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:40:03.790 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:40:03.790 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:40:03.791 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:40:03.791 DEBUG:boto:Method: GET 2023-07-29T00:40:03.792 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:40:03.792 DEBUG:boto:Data: 2023-07-29T00:40:03.792 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.793 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:03.793 DEBUG:boto:Port: 8001 2023-07-29T00:40:03.793 DEBUG:boto:Params: {} 2023-07-29T00:40:03.794 DEBUG:boto:Token: None 2023-07-29T00:40:03.794 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:40:03.795 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qqr0UhxxRRt9wmTEnSuZn3jAWws= 2023-07-29T00:40:03.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qqr0UhxxRRt9wmTEnSuZn3jAWws=', 'Content-Length': '0'} 2023-07-29T00:40:03.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075428d5373905528-0064c45fe3-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.798 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:40:03.798 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:40:03.799 DEBUG:boto:path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:40:03.799 DEBUG:boto:auth_path=/hiiqos-59/obj33?versionId=null 2023-07-29T00:40:03.799 DEBUG:boto:Method: GET 2023-07-29T00:40:03.800 DEBUG:boto:Path: /hiiqos-59/obj33?versionId=null 2023-07-29T00:40:03.800 DEBUG:boto:Data: 2023-07-29T00:40:03.800 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.801 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:03.801 DEBUG:boto:Port: 8000 2023-07-29T00:40:03.802 DEBUG:boto:Params: {} 2023-07-29T00:40:03.802 DEBUG:boto:Token: None 2023-07-29T00:40:03.802 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj33?versionId=null 2023-07-29T00:40:03.803 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qqr0UhxxRRt9wmTEnSuZn3jAWws= 2023-07-29T00:40:03.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qqr0UhxxRRt9wmTEnSuZn3jAWws=', 'Content-Length': '0'} 2023-07-29T00:40:03.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c862d00e7d1eaca-0064c45fe3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.848 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:03.849 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:03.850 DEBUG:boto:path=/hiiqos-59/?prefix=obj33 2023-07-29T00:40:03.850 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj33 2023-07-29T00:40:03.850 DEBUG:boto:Method: GET 2023-07-29T00:40:03.851 DEBUG:boto:Path: /hiiqos-59/?prefix=obj33 2023-07-29T00:40:03.851 DEBUG:boto:Data: 2023-07-29T00:40:03.851 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.852 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:03.852 DEBUG:boto:Port: 8001 2023-07-29T00:40:03.852 DEBUG:boto:Params: {} 2023-07-29T00:40:03.853 DEBUG:boto:Token: None 2023-07-29T00:40:03.853 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/ 2023-07-29T00:40:03.854 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JN8e9nTO+13r1+V6jH1TEK+0T9Q= 2023-07-29T00:40:03.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JN8e9nTO+13r1+V6jH1TEK+0T9Q=', 'Content-Length': '0'} 2023-07-29T00:40:03.857 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8cb05fd8ec3e2a3-0064c45fe3-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.857 DEBUG:boto:b'hiiqos-59obj331000falseobj332023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:03.858 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:03.859 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:03.859 DEBUG:boto:path=/hiiqos-59/?prefix=obj33 2023-07-29T00:40:03.859 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj33 2023-07-29T00:40:03.860 DEBUG:boto:Method: GET 2023-07-29T00:40:03.860 DEBUG:boto:Path: /hiiqos-59/?prefix=obj33 2023-07-29T00:40:03.860 DEBUG:boto:Data: 2023-07-29T00:40:03.861 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.861 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:03.861 DEBUG:boto:Port: 8000 2023-07-29T00:40:03.862 DEBUG:boto:Params: {} 2023-07-29T00:40:03.862 DEBUG:boto:Token: None 2023-07-29T00:40:03.863 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/ 2023-07-29T00:40:03.863 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JN8e9nTO+13r1+V6jH1TEK+0T9Q= 2023-07-29T00:40:03.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JN8e9nTO+13r1+V6jH1TEK+0T9Q=', 'Content-Length': '0'} 2023-07-29T00:40:03.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000335ad677d4021ea9-0064c45fe3-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.869 DEBUG:boto:b'hiiqos-59obj331000falseobj332023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:03.870 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:40:03.870 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:40:03.870 DEBUG:boto:Method: HEAD 2023-07-29T00:40:03.871 DEBUG:boto:Path: /hiiqos-59/obj33 2023-07-29T00:40:03.871 DEBUG:boto:Data: 2023-07-29T00:40:03.872 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.872 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:03.872 DEBUG:boto:Port: 8001 2023-07-29T00:40:03.873 DEBUG:boto:Params: {} 2023-07-29T00:40:03.874 DEBUG:boto:Token: None 2023-07-29T00:40:03.874 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj33 2023-07-29T00:40:03.874 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+pa++tIR+atmYiax8vs2O75UTEg= 2023-07-29T00:40:03.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+pa++tIR+atmYiax8vs2O75UTEg=', 'Content-Length': '0'} 2023-07-29T00:40:03.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000996be0e68d475f0a-0064c45fe3-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.878 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:40:03.878 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:40:03.878 DEBUG:boto:Method: HEAD 2023-07-29T00:40:03.879 DEBUG:boto:Path: /hiiqos-59/obj33 2023-07-29T00:40:03.879 DEBUG:boto:Data: 2023-07-29T00:40:03.879 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.880 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:03.880 DEBUG:boto:Port: 8000 2023-07-29T00:40:03.881 DEBUG:boto:Params: {} 2023-07-29T00:40:03.881 DEBUG:boto:Token: None 2023-07-29T00:40:03.882 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj33 2023-07-29T00:40:03.882 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+pa++tIR+atmYiax8vs2O75UTEg= 2023-07-29T00:40:03.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+pa++tIR+atmYiax8vs2O75UTEg=', 'Content-Length': '0'} 2023-07-29T00:40:03.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce7206db067b708d-0064c45fe3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-07-29T00:40:03.885 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:40:03.886 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:40:03.886 DEBUG:boto:Method: GET 2023-07-29T00:40:03.886 DEBUG:boto:Path: /hiiqos-59/obj33 2023-07-29T00:40:03.887 DEBUG:boto:Data: 2023-07-29T00:40:03.887 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.887 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:03.888 DEBUG:boto:Port: 8001 2023-07-29T00:40:03.888 DEBUG:boto:Params: {} 2023-07-29T00:40:03.889 DEBUG:boto:Token: None 2023-07-29T00:40:03.889 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj33 2023-07-29T00:40:03.889 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tvzxVDdRVcgi4kqwhjVAFoZlfZw= 2023-07-29T00:40:03.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tvzxVDdRVcgi4kqwhjVAFoZlfZw=', 'Content-Length': '0'} 2023-07-29T00:40:03.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049c23d27b772362b-0064c45fe3-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.936 DEBUG:boto:path=/hiiqos-59/obj33 2023-07-29T00:40:03.937 DEBUG:boto:auth_path=/hiiqos-59/obj33 2023-07-29T00:40:03.937 DEBUG:boto:Method: GET 2023-07-29T00:40:03.938 DEBUG:boto:Path: /hiiqos-59/obj33 2023-07-29T00:40:03.938 DEBUG:boto:Data: 2023-07-29T00:40:03.938 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.939 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:03.939 DEBUG:boto:Port: 8000 2023-07-29T00:40:03.940 DEBUG:boto:Params: {} 2023-07-29T00:40:03.940 DEBUG:boto:Token: None 2023-07-29T00:40:03.941 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj33 2023-07-29T00:40:03.941 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tvzxVDdRVcgi4kqwhjVAFoZlfZw= 2023-07-29T00:40:03.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tvzxVDdRVcgi4kqwhjVAFoZlfZw=', 'Content-Length': '0'} 2023-07-29T00:40:03.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e03aa3eb52d85b10-0064c45fe3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.944 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:03.945 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:03.945 DEBUG:boto:path=/hiiqos-59/?prefix=obj33 2023-07-29T00:40:03.945 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj33 2023-07-29T00:40:03.946 DEBUG:boto:Method: GET 2023-07-29T00:40:03.946 DEBUG:boto:Path: /hiiqos-59/?prefix=obj33 2023-07-29T00:40:03.947 DEBUG:boto:Data: 2023-07-29T00:40:03.947 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.947 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:03.948 DEBUG:boto:Port: 8001 2023-07-29T00:40:03.948 DEBUG:boto:Params: {} 2023-07-29T00:40:03.949 DEBUG:boto:Token: None 2023-07-29T00:40:03.949 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/ 2023-07-29T00:40:03.950 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JN8e9nTO+13r1+V6jH1TEK+0T9Q= 2023-07-29T00:40:03.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JN8e9nTO+13r1+V6jH1TEK+0T9Q=', 'Content-Length': '0'} 2023-07-29T00:40:03.953 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a8c0e94b7f48e14-0064c45fe3-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.953 DEBUG:boto:b'hiiqos-59obj331000falseobj332023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:03.954 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:03.954 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:03.955 DEBUG:boto:path=/hiiqos-59/?prefix=obj33 2023-07-29T00:40:03.955 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj33 2023-07-29T00:40:03.955 DEBUG:boto:Method: GET 2023-07-29T00:40:03.956 DEBUG:boto:Path: /hiiqos-59/?prefix=obj33 2023-07-29T00:40:03.956 DEBUG:boto:Data: 2023-07-29T00:40:03.956 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.957 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:03.957 DEBUG:boto:Port: 8000 2023-07-29T00:40:03.958 DEBUG:boto:Params: {} 2023-07-29T00:40:03.958 DEBUG:boto:Token: None 2023-07-29T00:40:03.959 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/ 2023-07-29T00:40:03.959 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JN8e9nTO+13r1+V6jH1TEK+0T9Q= 2023-07-29T00:40:03.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JN8e9nTO+13r1+V6jH1TEK+0T9Q=', 'Content-Length': '0'} 2023-07-29T00:40:03.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048bb7728f42bbe0d-0064c45fe3-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.964 DEBUG:boto:b'hiiqos-59obj331000falseobj332023-07-29T00:26:54.567Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:03.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-07-29T00:40:03.965 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:40:03.965 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:40:03.966 DEBUG:boto:path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:40:03.966 DEBUG:boto:auth_path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:40:03.966 DEBUG:boto:Method: GET 2023-07-29T00:40:03.967 DEBUG:boto:Path: /hiiqos-59/obj34?versionId=null 2023-07-29T00:40:03.967 DEBUG:boto:Data: 2023-07-29T00:40:03.968 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.968 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:03.968 DEBUG:boto:Port: 8001 2023-07-29T00:40:03.969 DEBUG:boto:Params: {} 2023-07-29T00:40:03.969 DEBUG:boto:Token: None 2023-07-29T00:40:03.970 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj34?versionId=null 2023-07-29T00:40:03.970 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:L0xGL7KmbyA1KOsq5BZ/Eg34A5U= 2023-07-29T00:40:03.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:L0xGL7KmbyA1KOsq5BZ/Eg34A5U=', 'Content-Length': '0'} 2023-07-29T00:40:03.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c082d2c097b5397-0064c45fe3-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.973 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:40:03.974 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:40:03.974 DEBUG:boto:path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:40:03.974 DEBUG:boto:auth_path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:40:03.975 DEBUG:boto:Method: GET 2023-07-29T00:40:03.975 DEBUG:boto:Path: /hiiqos-59/obj34?versionId=null 2023-07-29T00:40:03.976 DEBUG:boto:Data: 2023-07-29T00:40:03.976 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.976 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:03.977 DEBUG:boto:Port: 8000 2023-07-29T00:40:03.977 DEBUG:boto:Params: {} 2023-07-29T00:40:03.977 DEBUG:boto:Token: None 2023-07-29T00:40:03.978 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj34?versionId=null 2023-07-29T00:40:03.979 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:L0xGL7KmbyA1KOsq5BZ/Eg34A5U= 2023-07-29T00:40:03.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:L0xGL7KmbyA1KOsq5BZ/Eg34A5U=', 'Content-Length': '0'} 2023-07-29T00:40:03.983 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000903d7bed28327900-0064c45fe3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.985 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:40:03.985 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:40:03.986 DEBUG:boto:path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:40:03.986 DEBUG:boto:auth_path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:40:03.987 DEBUG:boto:Method: HEAD 2023-07-29T00:40:03.988 DEBUG:boto:Path: /hiiqos-59/obj34?versionId=null 2023-07-29T00:40:03.988 DEBUG:boto:Data: 2023-07-29T00:40:03.988 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.989 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:03.989 DEBUG:boto:Port: 8001 2023-07-29T00:40:03.990 DEBUG:boto:Params: {} 2023-07-29T00:40:03.990 DEBUG:boto:Token: None 2023-07-29T00:40:03.991 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj34?versionId=null 2023-07-29T00:40:03.991 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Vec7JvCqXYNB/58y9Exc2jS88eo= 2023-07-29T00:40:03.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Vec7JvCqXYNB/58y9Exc2jS88eo=', 'Content-Length': '0'} 2023-07-29T00:40:03.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3304496bf1081c3-0064c45fe3-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:03.994 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:40:03.994 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:40:03.994 DEBUG:boto:path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:40:03.995 DEBUG:boto:auth_path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:40:03.995 DEBUG:boto:Method: HEAD 2023-07-29T00:40:03.995 DEBUG:boto:Path: /hiiqos-59/obj34?versionId=null 2023-07-29T00:40:03.996 DEBUG:boto:Data: 2023-07-29T00:40:03.996 DEBUG:boto:Headers: {} 2023-07-29T00:40:03.996 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:03.997 DEBUG:boto:Port: 8000 2023-07-29T00:40:03.997 DEBUG:boto:Params: {} 2023-07-29T00:40:03.997 DEBUG:boto:Token: None 2023-07-29T00:40:03.998 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:03 GMT /hiiqos-59/obj34?versionId=null 2023-07-29T00:40:03.998 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Vec7JvCqXYNB/58y9Exc2jS88eo= 2023-07-29T00:40:03.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Vec7JvCqXYNB/58y9Exc2jS88eo=', 'Content-Length': '0'} 2023-07-29T00:40:04.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d45e72c4b80b6645-0064c45fe3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-07-29T00:40:04.002 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:40:04.002 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:40:04.002 DEBUG:boto:path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:40:04.003 DEBUG:boto:auth_path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:40:04.003 DEBUG:boto:Method: GET 2023-07-29T00:40:04.003 DEBUG:boto:Path: /hiiqos-59/obj34?versionId=null 2023-07-29T00:40:04.004 DEBUG:boto:Data: 2023-07-29T00:40:04.004 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.005 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:04.005 DEBUG:boto:Port: 8001 2023-07-29T00:40:04.006 DEBUG:boto:Params: {} 2023-07-29T00:40:04.006 DEBUG:boto:Token: None 2023-07-29T00:40:04.007 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/obj34?versionId=null 2023-07-29T00:40:04.007 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ixbXeuCw4E0bdc8byIoJsH4othM= 2023-07-29T00:40:04.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ixbXeuCw4E0bdc8byIoJsH4othM=', 'Content-Length': '0'} 2023-07-29T00:40:04.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005faea39706333909-0064c45fe3-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.056 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:40:04.057 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:40:04.057 DEBUG:boto:path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:40:04.058 DEBUG:boto:auth_path=/hiiqos-59/obj34?versionId=null 2023-07-29T00:40:04.058 DEBUG:boto:Method: GET 2023-07-29T00:40:04.058 DEBUG:boto:Path: /hiiqos-59/obj34?versionId=null 2023-07-29T00:40:04.059 DEBUG:boto:Data: 2023-07-29T00:40:04.059 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.059 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:04.060 DEBUG:boto:Port: 8000 2023-07-29T00:40:04.060 DEBUG:boto:Params: {} 2023-07-29T00:40:04.061 DEBUG:boto:Token: None 2023-07-29T00:40:04.061 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/obj34?versionId=null 2023-07-29T00:40:04.061 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ixbXeuCw4E0bdc8byIoJsH4othM= 2023-07-29T00:40:04.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ixbXeuCw4E0bdc8byIoJsH4othM=', 'Content-Length': '0'} 2023-07-29T00:40:04.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8c77eaa5f7e9719-0064c45fe3-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.065 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:04.065 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:04.065 DEBUG:boto:path=/hiiqos-59/?prefix=obj34 2023-07-29T00:40:04.066 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj34 2023-07-29T00:40:04.066 DEBUG:boto:Method: GET 2023-07-29T00:40:04.067 DEBUG:boto:Path: /hiiqos-59/?prefix=obj34 2023-07-29T00:40:04.067 DEBUG:boto:Data: 2023-07-29T00:40:04.068 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.068 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:04.069 DEBUG:boto:Port: 8001 2023-07-29T00:40:04.069 DEBUG:boto:Params: {} 2023-07-29T00:40:04.070 DEBUG:boto:Token: None 2023-07-29T00:40:04.070 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/ 2023-07-29T00:40:04.070 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fJfzv399nmY7cDGVwqo3xvwwWjg= 2023-07-29T00:40:04.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fJfzv399nmY7cDGVwqo3xvwwWjg=', 'Content-Length': '0'} 2023-07-29T00:40:04.074 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa4c94ab8a3a8ef3-0064c45fe4-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.074 DEBUG:boto:b'hiiqos-59obj341000falseobj342023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:04.076 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:04.076 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:04.077 DEBUG:boto:path=/hiiqos-59/?prefix=obj34 2023-07-29T00:40:04.078 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj34 2023-07-29T00:40:04.078 DEBUG:boto:Method: GET 2023-07-29T00:40:04.079 DEBUG:boto:Path: /hiiqos-59/?prefix=obj34 2023-07-29T00:40:04.079 DEBUG:boto:Data: 2023-07-29T00:40:04.079 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.080 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:04.081 DEBUG:boto:Port: 8000 2023-07-29T00:40:04.081 DEBUG:boto:Params: {} 2023-07-29T00:40:04.082 DEBUG:boto:Token: None 2023-07-29T00:40:04.083 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/ 2023-07-29T00:40:04.084 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fJfzv399nmY7cDGVwqo3xvwwWjg= 2023-07-29T00:40:04.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fJfzv399nmY7cDGVwqo3xvwwWjg=', 'Content-Length': '0'} 2023-07-29T00:40:04.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000584e27bd84eed673-0064c45fe4-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.090 DEBUG:boto:b'hiiqos-59obj341000falseobj342023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:04.091 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:40:04.092 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:40:04.092 DEBUG:boto:Method: HEAD 2023-07-29T00:40:04.092 DEBUG:boto:Path: /hiiqos-59/obj34 2023-07-29T00:40:04.093 DEBUG:boto:Data: 2023-07-29T00:40:04.094 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.095 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:04.095 DEBUG:boto:Port: 8001 2023-07-29T00:40:04.096 DEBUG:boto:Params: {} 2023-07-29T00:40:04.097 DEBUG:boto:Token: None 2023-07-29T00:40:04.098 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/obj34 2023-07-29T00:40:04.099 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I69ULSN0hQUc0HwUOeLhCATndMk= 2023-07-29T00:40:04.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I69ULSN0hQUc0HwUOeLhCATndMk=', 'Content-Length': '0'} 2023-07-29T00:40:04.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea763ed5c8592cd8-0064c45fe4-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.101 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:40:04.102 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:40:04.102 DEBUG:boto:Method: HEAD 2023-07-29T00:40:04.103 DEBUG:boto:Path: /hiiqos-59/obj34 2023-07-29T00:40:04.103 DEBUG:boto:Data: 2023-07-29T00:40:04.103 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.104 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:04.104 DEBUG:boto:Port: 8000 2023-07-29T00:40:04.104 DEBUG:boto:Params: {} 2023-07-29T00:40:04.105 DEBUG:boto:Token: None 2023-07-29T00:40:04.105 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/obj34 2023-07-29T00:40:04.105 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I69ULSN0hQUc0HwUOeLhCATndMk= 2023-07-29T00:40:04.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I69ULSN0hQUc0HwUOeLhCATndMk=', 'Content-Length': '0'} 2023-07-29T00:40:04.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000187fb3fdee055c33-0064c45fe4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-07-29T00:40:04.109 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:40:04.110 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:40:04.111 DEBUG:boto:Method: GET 2023-07-29T00:40:04.112 DEBUG:boto:Path: /hiiqos-59/obj34 2023-07-29T00:40:04.113 DEBUG:boto:Data: 2023-07-29T00:40:04.114 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.114 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:04.115 DEBUG:boto:Port: 8001 2023-07-29T00:40:04.116 DEBUG:boto:Params: {} 2023-07-29T00:40:04.117 DEBUG:boto:Token: None 2023-07-29T00:40:04.117 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/obj34 2023-07-29T00:40:04.118 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RaN62vJRkqbmHQTY0iYnkSQhfaI= 2023-07-29T00:40:04.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RaN62vJRkqbmHQTY0iYnkSQhfaI=', 'Content-Length': '0'} 2023-07-29T00:40:04.120 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003db87934bd59eaed-0064c45fe4-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.164 DEBUG:boto:path=/hiiqos-59/obj34 2023-07-29T00:40:04.165 DEBUG:boto:auth_path=/hiiqos-59/obj34 2023-07-29T00:40:04.165 DEBUG:boto:Method: GET 2023-07-29T00:40:04.166 DEBUG:boto:Path: /hiiqos-59/obj34 2023-07-29T00:40:04.166 DEBUG:boto:Data: 2023-07-29T00:40:04.166 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.167 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:04.167 DEBUG:boto:Port: 8000 2023-07-29T00:40:04.167 DEBUG:boto:Params: {} 2023-07-29T00:40:04.168 DEBUG:boto:Token: None 2023-07-29T00:40:04.168 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/obj34 2023-07-29T00:40:04.169 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RaN62vJRkqbmHQTY0iYnkSQhfaI= 2023-07-29T00:40:04.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RaN62vJRkqbmHQTY0iYnkSQhfaI=', 'Content-Length': '0'} 2023-07-29T00:40:04.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000162481fa1510d8a0-0064c45fe4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.172 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:04.173 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:04.174 DEBUG:boto:path=/hiiqos-59/?prefix=obj34 2023-07-29T00:40:04.175 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj34 2023-07-29T00:40:04.176 DEBUG:boto:Method: GET 2023-07-29T00:40:04.176 DEBUG:boto:Path: /hiiqos-59/?prefix=obj34 2023-07-29T00:40:04.177 DEBUG:boto:Data: 2023-07-29T00:40:04.178 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.179 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:04.180 DEBUG:boto:Port: 8001 2023-07-29T00:40:04.180 DEBUG:boto:Params: {} 2023-07-29T00:40:04.181 DEBUG:boto:Token: None 2023-07-29T00:40:04.182 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/ 2023-07-29T00:40:04.183 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fJfzv399nmY7cDGVwqo3xvwwWjg= 2023-07-29T00:40:04.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fJfzv399nmY7cDGVwqo3xvwwWjg=', 'Content-Length': '0'} 2023-07-29T00:40:04.187 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d3f86a6f645c784-0064c45fe4-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.188 DEBUG:boto:b'hiiqos-59obj341000falseobj342023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:04.189 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:04.190 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:04.190 DEBUG:boto:path=/hiiqos-59/?prefix=obj34 2023-07-29T00:40:04.191 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj34 2023-07-29T00:40:04.191 DEBUG:boto:Method: GET 2023-07-29T00:40:04.192 DEBUG:boto:Path: /hiiqos-59/?prefix=obj34 2023-07-29T00:40:04.192 DEBUG:boto:Data: 2023-07-29T00:40:04.193 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.193 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:04.194 DEBUG:boto:Port: 8000 2023-07-29T00:40:04.194 DEBUG:boto:Params: {} 2023-07-29T00:40:04.195 DEBUG:boto:Token: None 2023-07-29T00:40:04.195 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/ 2023-07-29T00:40:04.196 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fJfzv399nmY7cDGVwqo3xvwwWjg= 2023-07-29T00:40:04.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fJfzv399nmY7cDGVwqo3xvwwWjg=', 'Content-Length': '0'} 2023-07-29T00:40:04.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c120ab120afeffca-0064c45fe4-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.202 DEBUG:boto:b'hiiqos-59obj341000falseobj342023-07-29T00:27:51.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:04.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-07-29T00:40:04.204 DEBUG:boto:path=/hiiqos-59/obj37 2023-07-29T00:40:04.205 DEBUG:boto:auth_path=/hiiqos-59/obj37 2023-07-29T00:40:04.206 DEBUG:boto:path=/hiiqos-59/obj37?versionId=null 2023-07-29T00:40:04.207 DEBUG:boto:auth_path=/hiiqos-59/obj37?versionId=null 2023-07-29T00:40:04.207 DEBUG:boto:Method: GET 2023-07-29T00:40:04.208 DEBUG:boto:Path: /hiiqos-59/obj37?versionId=null 2023-07-29T00:40:04.209 DEBUG:boto:Data: 2023-07-29T00:40:04.210 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.210 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:04.211 DEBUG:boto:Port: 8001 2023-07-29T00:40:04.212 DEBUG:boto:Params: {} 2023-07-29T00:40:04.213 DEBUG:boto:Token: None 2023-07-29T00:40:04.213 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/obj37?versionId=null 2023-07-29T00:40:04.214 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0dv0QZszY2SstGNa3uyqxgdtTeE= 2023-07-29T00:40:04.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0dv0QZszY2SstGNa3uyqxgdtTeE=', 'Content-Length': '0'} 2023-07-29T00:40:04.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c14ae17fb10cc41c-0064c45fe4-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.218 DEBUG:boto:path=/hiiqos-59/obj37 2023-07-29T00:40:04.219 DEBUG:boto:auth_path=/hiiqos-59/obj37 2023-07-29T00:40:04.220 DEBUG:boto:path=/hiiqos-59/obj37?versionId=null 2023-07-29T00:40:04.221 DEBUG:boto:auth_path=/hiiqos-59/obj37?versionId=null 2023-07-29T00:40:04.221 DEBUG:boto:Method: GET 2023-07-29T00:40:04.222 DEBUG:boto:Path: /hiiqos-59/obj37?versionId=null 2023-07-29T00:40:04.223 DEBUG:boto:Data: 2023-07-29T00:40:04.223 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.224 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:04.225 DEBUG:boto:Port: 8000 2023-07-29T00:40:04.226 DEBUG:boto:Params: {} 2023-07-29T00:40:04.227 DEBUG:boto:Token: None 2023-07-29T00:40:04.227 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/obj37?versionId=null 2023-07-29T00:40:04.228 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0dv0QZszY2SstGNa3uyqxgdtTeE= 2023-07-29T00:40:04.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0dv0QZszY2SstGNa3uyqxgdtTeE=', 'Content-Length': '0'} 2023-07-29T00:40:04.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abec33d690d07d0a-0064c45fe4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.276 DEBUG:boto:path=/hiiqos-59/obj37 2023-07-29T00:40:04.277 DEBUG:boto:auth_path=/hiiqos-59/obj37 2023-07-29T00:40:04.277 DEBUG:boto:path=/hiiqos-59/obj37?versionId=null 2023-07-29T00:40:04.278 DEBUG:boto:auth_path=/hiiqos-59/obj37?versionId=null 2023-07-29T00:40:04.278 DEBUG:boto:Method: HEAD 2023-07-29T00:40:04.278 DEBUG:boto:Path: /hiiqos-59/obj37?versionId=null 2023-07-29T00:40:04.279 DEBUG:boto:Data: 2023-07-29T00:40:04.279 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.279 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:04.280 DEBUG:boto:Port: 8001 2023-07-29T00:40:04.280 DEBUG:boto:Params: {} 2023-07-29T00:40:04.280 DEBUG:boto:Token: None 2023-07-29T00:40:04.281 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/obj37?versionId=null 2023-07-29T00:40:04.281 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AdYqR2+xgwxScujUHvn0oyBYo0A= 2023-07-29T00:40:04.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AdYqR2+xgwxScujUHvn0oyBYo0A=', 'Content-Length': '0'} 2023-07-29T00:40:04.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f6806c92f8c2925-0064c45fe4-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.284 DEBUG:boto:path=/hiiqos-59/obj37 2023-07-29T00:40:04.284 DEBUG:boto:auth_path=/hiiqos-59/obj37 2023-07-29T00:40:04.285 DEBUG:boto:path=/hiiqos-59/obj37?versionId=null 2023-07-29T00:40:04.285 DEBUG:boto:auth_path=/hiiqos-59/obj37?versionId=null 2023-07-29T00:40:04.285 DEBUG:boto:Method: HEAD 2023-07-29T00:40:04.286 DEBUG:boto:Path: /hiiqos-59/obj37?versionId=null 2023-07-29T00:40:04.286 DEBUG:boto:Data: 2023-07-29T00:40:04.286 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.287 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:04.287 DEBUG:boto:Port: 8000 2023-07-29T00:40:04.287 DEBUG:boto:Params: {} 2023-07-29T00:40:04.288 DEBUG:boto:Token: None 2023-07-29T00:40:04.288 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/obj37?versionId=null 2023-07-29T00:40:04.289 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AdYqR2+xgwxScujUHvn0oyBYo0A= 2023-07-29T00:40:04.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AdYqR2+xgwxScujUHvn0oyBYo0A=', 'Content-Length': '0'} 2023-07-29T00:40:04.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d01efa46032dfa4-0064c45fe4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-07-29T00:40:04.295 DEBUG:boto:path=/hiiqos-59/obj37 2023-07-29T00:40:04.295 DEBUG:boto:auth_path=/hiiqos-59/obj37 2023-07-29T00:40:04.296 DEBUG:boto:path=/hiiqos-59/obj37?versionId=null 2023-07-29T00:40:04.297 DEBUG:boto:auth_path=/hiiqos-59/obj37?versionId=null 2023-07-29T00:40:04.297 DEBUG:boto:Method: GET 2023-07-29T00:40:04.298 DEBUG:boto:Path: /hiiqos-59/obj37?versionId=null 2023-07-29T00:40:04.299 DEBUG:boto:Data: 2023-07-29T00:40:04.299 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.300 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:04.300 DEBUG:boto:Port: 8001 2023-07-29T00:40:04.301 DEBUG:boto:Params: {} 2023-07-29T00:40:04.301 DEBUG:boto:Token: None 2023-07-29T00:40:04.302 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/obj37?versionId=null 2023-07-29T00:40:04.302 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0dv0QZszY2SstGNa3uyqxgdtTeE= 2023-07-29T00:40:04.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0dv0QZszY2SstGNa3uyqxgdtTeE=', 'Content-Length': '0'} 2023-07-29T00:40:04.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000379a42c602c35b90-0064c45fe4-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.305 DEBUG:boto:path=/hiiqos-59/obj37 2023-07-29T00:40:04.306 DEBUG:boto:auth_path=/hiiqos-59/obj37 2023-07-29T00:40:04.306 DEBUG:boto:path=/hiiqos-59/obj37?versionId=null 2023-07-29T00:40:04.306 DEBUG:boto:auth_path=/hiiqos-59/obj37?versionId=null 2023-07-29T00:40:04.307 DEBUG:boto:Method: GET 2023-07-29T00:40:04.307 DEBUG:boto:Path: /hiiqos-59/obj37?versionId=null 2023-07-29T00:40:04.307 DEBUG:boto:Data: 2023-07-29T00:40:04.308 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.308 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:04.309 DEBUG:boto:Port: 8000 2023-07-29T00:40:04.309 DEBUG:boto:Params: {} 2023-07-29T00:40:04.309 DEBUG:boto:Token: None 2023-07-29T00:40:04.310 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/obj37?versionId=null 2023-07-29T00:40:04.310 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0dv0QZszY2SstGNa3uyqxgdtTeE= 2023-07-29T00:40:04.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0dv0QZszY2SstGNa3uyqxgdtTeE=', 'Content-Length': '0'} 2023-07-29T00:40:04.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f28da7f499fbf26-0064c45fe4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.313 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:04.313 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:04.314 DEBUG:boto:path=/hiiqos-59/?prefix=obj37 2023-07-29T00:40:04.314 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj37 2023-07-29T00:40:04.315 DEBUG:boto:Method: GET 2023-07-29T00:40:04.315 DEBUG:boto:Path: /hiiqos-59/?prefix=obj37 2023-07-29T00:40:04.315 DEBUG:boto:Data: 2023-07-29T00:40:04.316 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.316 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:04.316 DEBUG:boto:Port: 8001 2023-07-29T00:40:04.317 DEBUG:boto:Params: {} 2023-07-29T00:40:04.317 DEBUG:boto:Token: None 2023-07-29T00:40:04.318 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/ 2023-07-29T00:40:04.318 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fJfzv399nmY7cDGVwqo3xvwwWjg= 2023-07-29T00:40:04.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fJfzv399nmY7cDGVwqo3xvwwWjg=', 'Content-Length': '0'} 2023-07-29T00:40:04.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfbcb808bebd4396-0064c45fe4-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.321 DEBUG:boto:b'hiiqos-59obj371000falseobj372023-07-29T00:37:53.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:04.322 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:04.323 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:04.323 DEBUG:boto:path=/hiiqos-59/?prefix=obj37 2023-07-29T00:40:04.323 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj37 2023-07-29T00:40:04.324 DEBUG:boto:Method: GET 2023-07-29T00:40:04.324 DEBUG:boto:Path: /hiiqos-59/?prefix=obj37 2023-07-29T00:40:04.324 DEBUG:boto:Data: 2023-07-29T00:40:04.325 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.325 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:04.325 DEBUG:boto:Port: 8000 2023-07-29T00:40:04.326 DEBUG:boto:Params: {} 2023-07-29T00:40:04.326 DEBUG:boto:Token: None 2023-07-29T00:40:04.327 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/ 2023-07-29T00:40:04.327 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fJfzv399nmY7cDGVwqo3xvwwWjg= 2023-07-29T00:40:04.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fJfzv399nmY7cDGVwqo3xvwwWjg=', 'Content-Length': '0'} 2023-07-29T00:40:04.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000723c8e98827aff00-0064c45fe4-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.376 DEBUG:boto:b'hiiqos-59obj371000falseobj372023-07-29T00:37:53.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:04.377 DEBUG:boto:path=/hiiqos-59/obj37 2023-07-29T00:40:04.378 DEBUG:boto:auth_path=/hiiqos-59/obj37 2023-07-29T00:40:04.378 DEBUG:boto:Method: HEAD 2023-07-29T00:40:04.379 DEBUG:boto:Path: /hiiqos-59/obj37 2023-07-29T00:40:04.379 DEBUG:boto:Data: 2023-07-29T00:40:04.379 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.380 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:04.380 DEBUG:boto:Port: 8001 2023-07-29T00:40:04.380 DEBUG:boto:Params: {} 2023-07-29T00:40:04.381 DEBUG:boto:Token: None 2023-07-29T00:40:04.381 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/obj37 2023-07-29T00:40:04.382 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J/e+Yvkf8nBrtV3IdDyQIxJRhlQ= 2023-07-29T00:40:04.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J/e+Yvkf8nBrtV3IdDyQIxJRhlQ=', 'Content-Length': '0'} 2023-07-29T00:40:04.384 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030a2d93264385548-0064c45fe4-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.385 DEBUG:boto:path=/hiiqos-59/obj37 2023-07-29T00:40:04.385 DEBUG:boto:auth_path=/hiiqos-59/obj37 2023-07-29T00:40:04.386 DEBUG:boto:Method: HEAD 2023-07-29T00:40:04.386 DEBUG:boto:Path: /hiiqos-59/obj37 2023-07-29T00:40:04.386 DEBUG:boto:Data: 2023-07-29T00:40:04.387 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.387 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:04.387 DEBUG:boto:Port: 8000 2023-07-29T00:40:04.388 DEBUG:boto:Params: {} 2023-07-29T00:40:04.388 DEBUG:boto:Token: None 2023-07-29T00:40:04.389 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/obj37 2023-07-29T00:40:04.389 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J/e+Yvkf8nBrtV3IdDyQIxJRhlQ= 2023-07-29T00:40:04.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J/e+Yvkf8nBrtV3IdDyQIxJRhlQ=', 'Content-Length': '0'} 2023-07-29T00:40:04.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4c2e39bfad165b2-0064c45fe4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-07-29T00:40:04.393 DEBUG:boto:path=/hiiqos-59/obj37 2023-07-29T00:40:04.393 DEBUG:boto:auth_path=/hiiqos-59/obj37 2023-07-29T00:40:04.393 DEBUG:boto:Method: GET 2023-07-29T00:40:04.394 DEBUG:boto:Path: /hiiqos-59/obj37 2023-07-29T00:40:04.394 DEBUG:boto:Data: 2023-07-29T00:40:04.394 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.395 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:04.395 DEBUG:boto:Port: 8001 2023-07-29T00:40:04.396 DEBUG:boto:Params: {} 2023-07-29T00:40:04.396 DEBUG:boto:Token: None 2023-07-29T00:40:04.396 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/obj37 2023-07-29T00:40:04.397 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8BUGp+C/8DGQDT63yujrlGIqurk= 2023-07-29T00:40:04.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8BUGp+C/8DGQDT63yujrlGIqurk=', 'Content-Length': '0'} 2023-07-29T00:40:04.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044dca085cdec0451-0064c45fe4-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.440 DEBUG:boto:path=/hiiqos-59/obj37 2023-07-29T00:40:04.441 DEBUG:boto:auth_path=/hiiqos-59/obj37 2023-07-29T00:40:04.442 DEBUG:boto:Method: GET 2023-07-29T00:40:04.442 DEBUG:boto:Path: /hiiqos-59/obj37 2023-07-29T00:40:04.442 DEBUG:boto:Data: 2023-07-29T00:40:04.443 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.443 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:04.443 DEBUG:boto:Port: 8000 2023-07-29T00:40:04.444 DEBUG:boto:Params: {} 2023-07-29T00:40:04.444 DEBUG:boto:Token: None 2023-07-29T00:40:04.445 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/obj37 2023-07-29T00:40:04.445 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8BUGp+C/8DGQDT63yujrlGIqurk= 2023-07-29T00:40:04.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8BUGp+C/8DGQDT63yujrlGIqurk=', 'Content-Length': '0'} 2023-07-29T00:40:04.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fb03760f5475940-0064c45fe4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.448 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:04.449 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:04.449 DEBUG:boto:path=/hiiqos-59/?prefix=obj37 2023-07-29T00:40:04.449 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj37 2023-07-29T00:40:04.450 DEBUG:boto:Method: GET 2023-07-29T00:40:04.450 DEBUG:boto:Path: /hiiqos-59/?prefix=obj37 2023-07-29T00:40:04.450 DEBUG:boto:Data: 2023-07-29T00:40:04.451 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.451 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:04.452 DEBUG:boto:Port: 8001 2023-07-29T00:40:04.452 DEBUG:boto:Params: {} 2023-07-29T00:40:04.452 DEBUG:boto:Token: None 2023-07-29T00:40:04.453 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/ 2023-07-29T00:40:04.453 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fJfzv399nmY7cDGVwqo3xvwwWjg= 2023-07-29T00:40:04.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fJfzv399nmY7cDGVwqo3xvwwWjg=', 'Content-Length': '0'} 2023-07-29T00:40:04.456 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089849e0c9d386db6-0064c45fe4-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.456 DEBUG:boto:b'hiiqos-59obj371000falseobj372023-07-29T00:37:53.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:04.457 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:04.458 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:04.458 DEBUG:boto:path=/hiiqos-59/?prefix=obj37 2023-07-29T00:40:04.458 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj37 2023-07-29T00:40:04.459 DEBUG:boto:Method: GET 2023-07-29T00:40:04.459 DEBUG:boto:Path: /hiiqos-59/?prefix=obj37 2023-07-29T00:40:04.459 DEBUG:boto:Data: 2023-07-29T00:40:04.460 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.460 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:04.461 DEBUG:boto:Port: 8000 2023-07-29T00:40:04.461 DEBUG:boto:Params: {} 2023-07-29T00:40:04.461 DEBUG:boto:Token: None 2023-07-29T00:40:04.462 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/ 2023-07-29T00:40:04.462 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fJfzv399nmY7cDGVwqo3xvwwWjg= 2023-07-29T00:40:04.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fJfzv399nmY7cDGVwqo3xvwwWjg=', 'Content-Length': '0'} 2023-07-29T00:40:04.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eca30d9ed71e11d3-0064c45fe4-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.468 DEBUG:boto:b'hiiqos-59obj371000falseobj372023-07-29T00:37:53.137Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:04.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-07-29T00:40:04.469 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:40:04.469 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:40:04.470 DEBUG:boto:path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:40:04.470 DEBUG:boto:auth_path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:40:04.470 DEBUG:boto:Method: GET 2023-07-29T00:40:04.471 DEBUG:boto:Path: /hiiqos-59/obj38?versionId=null 2023-07-29T00:40:04.471 DEBUG:boto:Data: 2023-07-29T00:40:04.471 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.472 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:04.472 DEBUG:boto:Port: 8001 2023-07-29T00:40:04.472 DEBUG:boto:Params: {} 2023-07-29T00:40:04.473 DEBUG:boto:Token: None 2023-07-29T00:40:04.473 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/obj38?versionId=null 2023-07-29T00:40:04.474 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zm0lLWHV+jDJPoXGyMxrNqUnvZw= 2023-07-29T00:40:04.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zm0lLWHV+jDJPoXGyMxrNqUnvZw=', 'Content-Length': '0'} 2023-07-29T00:40:04.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad9270189bb3afcf-0064c45fe4-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.477 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:40:04.477 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:40:04.478 DEBUG:boto:path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:40:04.478 DEBUG:boto:auth_path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:40:04.479 DEBUG:boto:Method: GET 2023-07-29T00:40:04.479 DEBUG:boto:Path: /hiiqos-59/obj38?versionId=null 2023-07-29T00:40:04.479 DEBUG:boto:Data: 2023-07-29T00:40:04.480 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.480 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:04.480 DEBUG:boto:Port: 8000 2023-07-29T00:40:04.481 DEBUG:boto:Params: {} 2023-07-29T00:40:04.481 DEBUG:boto:Token: None 2023-07-29T00:40:04.482 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/obj38?versionId=null 2023-07-29T00:40:04.482 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zm0lLWHV+jDJPoXGyMxrNqUnvZw= 2023-07-29T00:40:04.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zm0lLWHV+jDJPoXGyMxrNqUnvZw=', 'Content-Length': '0'} 2023-07-29T00:40:04.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000681ea313464f8f88-0064c45fe4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.485 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:40:04.485 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:40:04.486 DEBUG:boto:path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:40:04.486 DEBUG:boto:auth_path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:40:04.486 DEBUG:boto:Method: HEAD 2023-07-29T00:40:04.487 DEBUG:boto:Path: /hiiqos-59/obj38?versionId=null 2023-07-29T00:40:04.487 DEBUG:boto:Data: 2023-07-29T00:40:04.487 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.488 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:04.488 DEBUG:boto:Port: 8001 2023-07-29T00:40:04.488 DEBUG:boto:Params: {} 2023-07-29T00:40:04.489 DEBUG:boto:Token: None 2023-07-29T00:40:04.489 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/obj38?versionId=null 2023-07-29T00:40:04.490 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7q9pDq/UKQj8geIjLrICMV3qIKc= 2023-07-29T00:40:04.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7q9pDq/UKQj8geIjLrICMV3qIKc=', 'Content-Length': '0'} 2023-07-29T00:40:04.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f84410e4986a243-0064c45fe4-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.492 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:40:04.493 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:40:04.493 DEBUG:boto:path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:40:04.493 DEBUG:boto:auth_path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:40:04.494 DEBUG:boto:Method: HEAD 2023-07-29T00:40:04.494 DEBUG:boto:Path: /hiiqos-59/obj38?versionId=null 2023-07-29T00:40:04.495 DEBUG:boto:Data: 2023-07-29T00:40:04.495 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.495 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:04.496 DEBUG:boto:Port: 8000 2023-07-29T00:40:04.496 DEBUG:boto:Params: {} 2023-07-29T00:40:04.496 DEBUG:boto:Token: None 2023-07-29T00:40:04.497 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/obj38?versionId=null 2023-07-29T00:40:04.497 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7q9pDq/UKQj8geIjLrICMV3qIKc= 2023-07-29T00:40:04.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7q9pDq/UKQj8geIjLrICMV3qIKc=', 'Content-Length': '0'} 2023-07-29T00:40:04.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e111d9ab319a2c4-0064c45fe4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-07-29T00:40:04.500 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:40:04.501 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:40:04.501 DEBUG:boto:path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:40:04.501 DEBUG:boto:auth_path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:40:04.502 DEBUG:boto:Method: GET 2023-07-29T00:40:04.502 DEBUG:boto:Path: /hiiqos-59/obj38?versionId=null 2023-07-29T00:40:04.503 DEBUG:boto:Data: 2023-07-29T00:40:04.503 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.503 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:04.504 DEBUG:boto:Port: 8001 2023-07-29T00:40:04.504 DEBUG:boto:Params: {} 2023-07-29T00:40:04.505 DEBUG:boto:Token: None 2023-07-29T00:40:04.505 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/obj38?versionId=null 2023-07-29T00:40:04.505 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zm0lLWHV+jDJPoXGyMxrNqUnvZw= 2023-07-29T00:40:04.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zm0lLWHV+jDJPoXGyMxrNqUnvZw=', 'Content-Length': '0'} 2023-07-29T00:40:04.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6786fd9c00f5950-0064c45fe4-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.508 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:40:04.509 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:40:04.509 DEBUG:boto:path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:40:04.509 DEBUG:boto:auth_path=/hiiqos-59/obj38?versionId=null 2023-07-29T00:40:04.510 DEBUG:boto:Method: GET 2023-07-29T00:40:04.510 DEBUG:boto:Path: /hiiqos-59/obj38?versionId=null 2023-07-29T00:40:04.510 DEBUG:boto:Data: 2023-07-29T00:40:04.511 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.511 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:04.511 DEBUG:boto:Port: 8000 2023-07-29T00:40:04.512 DEBUG:boto:Params: {} 2023-07-29T00:40:04.512 DEBUG:boto:Token: None 2023-07-29T00:40:04.513 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/obj38?versionId=null 2023-07-29T00:40:04.513 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zm0lLWHV+jDJPoXGyMxrNqUnvZw= 2023-07-29T00:40:04.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zm0lLWHV+jDJPoXGyMxrNqUnvZw=', 'Content-Length': '0'} 2023-07-29T00:40:04.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003198795ddc2ee557-0064c45fe4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.560 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:04.561 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:04.561 DEBUG:boto:path=/hiiqos-59/?prefix=obj38 2023-07-29T00:40:04.562 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj38 2023-07-29T00:40:04.562 DEBUG:boto:Method: GET 2023-07-29T00:40:04.562 DEBUG:boto:Path: /hiiqos-59/?prefix=obj38 2023-07-29T00:40:04.563 DEBUG:boto:Data: 2023-07-29T00:40:04.563 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.563 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:04.564 DEBUG:boto:Port: 8001 2023-07-29T00:40:04.564 DEBUG:boto:Params: {} 2023-07-29T00:40:04.565 DEBUG:boto:Token: None 2023-07-29T00:40:04.565 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/ 2023-07-29T00:40:04.566 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fJfzv399nmY7cDGVwqo3xvwwWjg= 2023-07-29T00:40:04.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fJfzv399nmY7cDGVwqo3xvwwWjg=', 'Content-Length': '0'} 2023-07-29T00:40:04.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e900eaa052577362-0064c45fe4-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.569 DEBUG:boto:b'hiiqos-59obj381000falseobj382023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:04.570 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:04.570 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:04.570 DEBUG:boto:path=/hiiqos-59/?prefix=obj38 2023-07-29T00:40:04.571 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj38 2023-07-29T00:40:04.571 DEBUG:boto:Method: GET 2023-07-29T00:40:04.572 DEBUG:boto:Path: /hiiqos-59/?prefix=obj38 2023-07-29T00:40:04.572 DEBUG:boto:Data: 2023-07-29T00:40:04.572 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.573 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:04.573 DEBUG:boto:Port: 8000 2023-07-29T00:40:04.573 DEBUG:boto:Params: {} 2023-07-29T00:40:04.574 DEBUG:boto:Token: None 2023-07-29T00:40:04.574 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/ 2023-07-29T00:40:04.575 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fJfzv399nmY7cDGVwqo3xvwwWjg= 2023-07-29T00:40:04.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fJfzv399nmY7cDGVwqo3xvwwWjg=', 'Content-Length': '0'} 2023-07-29T00:40:04.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075251d865ad81cc8-0064c45fe4-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.580 DEBUG:boto:b'hiiqos-59obj381000falseobj382023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:04.580 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:40:04.581 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:40:04.581 DEBUG:boto:Method: HEAD 2023-07-29T00:40:04.581 DEBUG:boto:Path: /hiiqos-59/obj38 2023-07-29T00:40:04.582 DEBUG:boto:Data: 2023-07-29T00:40:04.582 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.582 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:04.583 DEBUG:boto:Port: 8001 2023-07-29T00:40:04.583 DEBUG:boto:Params: {} 2023-07-29T00:40:04.583 DEBUG:boto:Token: None 2023-07-29T00:40:04.584 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/obj38 2023-07-29T00:40:04.584 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ApgMhggviEKNAmCF+kDjJLIZi/U= 2023-07-29T00:40:04.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ApgMhggviEKNAmCF+kDjJLIZi/U=', 'Content-Length': '0'} 2023-07-29T00:40:04.586 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffdfcf457dc6532f-0064c45fe4-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.588 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:40:04.588 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:40:04.589 DEBUG:boto:Method: HEAD 2023-07-29T00:40:04.589 DEBUG:boto:Path: /hiiqos-59/obj38 2023-07-29T00:40:04.589 DEBUG:boto:Data: 2023-07-29T00:40:04.590 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.590 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:04.590 DEBUG:boto:Port: 8000 2023-07-29T00:40:04.591 DEBUG:boto:Params: {} 2023-07-29T00:40:04.591 DEBUG:boto:Token: None 2023-07-29T00:40:04.592 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/obj38 2023-07-29T00:40:04.592 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ApgMhggviEKNAmCF+kDjJLIZi/U= 2023-07-29T00:40:04.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ApgMhggviEKNAmCF+kDjJLIZi/U=', 'Content-Length': '0'} 2023-07-29T00:40:04.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000477f09a32ca8820c-0064c45fe4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-07-29T00:40:04.595 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:40:04.595 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:40:04.596 DEBUG:boto:Method: GET 2023-07-29T00:40:04.596 DEBUG:boto:Path: /hiiqos-59/obj38 2023-07-29T00:40:04.597 DEBUG:boto:Data: 2023-07-29T00:40:04.597 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.597 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:04.598 DEBUG:boto:Port: 8001 2023-07-29T00:40:04.598 DEBUG:boto:Params: {} 2023-07-29T00:40:04.598 DEBUG:boto:Token: None 2023-07-29T00:40:04.599 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/obj38 2023-07-29T00:40:04.599 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1141VZiqjbXKSuAPsifERWQGWIc= 2023-07-29T00:40:04.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1141VZiqjbXKSuAPsifERWQGWIc=', 'Content-Length': '0'} 2023-07-29T00:40:04.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004959c11d21c4fb37-0064c45fe4-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.648 DEBUG:boto:path=/hiiqos-59/obj38 2023-07-29T00:40:04.649 DEBUG:boto:auth_path=/hiiqos-59/obj38 2023-07-29T00:40:04.649 DEBUG:boto:Method: GET 2023-07-29T00:40:04.650 DEBUG:boto:Path: /hiiqos-59/obj38 2023-07-29T00:40:04.650 DEBUG:boto:Data: 2023-07-29T00:40:04.650 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.651 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:04.651 DEBUG:boto:Port: 8000 2023-07-29T00:40:04.652 DEBUG:boto:Params: {} 2023-07-29T00:40:04.652 DEBUG:boto:Token: None 2023-07-29T00:40:04.652 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/obj38 2023-07-29T00:40:04.653 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1141VZiqjbXKSuAPsifERWQGWIc= 2023-07-29T00:40:04.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1141VZiqjbXKSuAPsifERWQGWIc=', 'Content-Length': '0'} 2023-07-29T00:40:04.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008364cd02900ddc8a-0064c45fe4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.656 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:04.656 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:04.657 DEBUG:boto:path=/hiiqos-59/?prefix=obj38 2023-07-29T00:40:04.657 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj38 2023-07-29T00:40:04.657 DEBUG:boto:Method: GET 2023-07-29T00:40:04.658 DEBUG:boto:Path: /hiiqos-59/?prefix=obj38 2023-07-29T00:40:04.658 DEBUG:boto:Data: 2023-07-29T00:40:04.659 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.659 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:04.659 DEBUG:boto:Port: 8001 2023-07-29T00:40:04.660 DEBUG:boto:Params: {} 2023-07-29T00:40:04.660 DEBUG:boto:Token: None 2023-07-29T00:40:04.660 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/ 2023-07-29T00:40:04.661 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fJfzv399nmY7cDGVwqo3xvwwWjg= 2023-07-29T00:40:04.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fJfzv399nmY7cDGVwqo3xvwwWjg=', 'Content-Length': '0'} 2023-07-29T00:40:04.663 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002face6d5c21af00f-0064c45fe4-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.664 DEBUG:boto:b'hiiqos-59obj381000falseobj382023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:04.665 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:04.665 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:04.666 DEBUG:boto:path=/hiiqos-59/?prefix=obj38 2023-07-29T00:40:04.666 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj38 2023-07-29T00:40:04.666 DEBUG:boto:Method: GET 2023-07-29T00:40:04.667 DEBUG:boto:Path: /hiiqos-59/?prefix=obj38 2023-07-29T00:40:04.667 DEBUG:boto:Data: 2023-07-29T00:40:04.668 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.668 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:04.668 DEBUG:boto:Port: 8000 2023-07-29T00:40:04.669 DEBUG:boto:Params: {} 2023-07-29T00:40:04.669 DEBUG:boto:Token: None 2023-07-29T00:40:04.669 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/ 2023-07-29T00:40:04.670 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fJfzv399nmY7cDGVwqo3xvwwWjg= 2023-07-29T00:40:04.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fJfzv399nmY7cDGVwqo3xvwwWjg=', 'Content-Length': '0'} 2023-07-29T00:40:04.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5fbeb826459579d-0064c45fe4-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.675 DEBUG:boto:b'hiiqos-59obj381000falseobj382023-07-29T00:29:58.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:04.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-07-29T00:40:04.676 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:40:04.677 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:40:04.677 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:40:04.677 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:40:04.678 DEBUG:boto:Method: GET 2023-07-29T00:40:04.678 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:40:04.678 DEBUG:boto:Data: 2023-07-29T00:40:04.679 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.679 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:04.679 DEBUG:boto:Port: 8001 2023-07-29T00:40:04.680 DEBUG:boto:Params: {} 2023-07-29T00:40:04.680 DEBUG:boto:Token: None 2023-07-29T00:40:04.681 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:40:04.681 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ELzQhwH3jmrLk0fNE0/6AqqxLDE= 2023-07-29T00:40:04.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ELzQhwH3jmrLk0fNE0/6AqqxLDE=', 'Content-Length': '0'} 2023-07-29T00:40:04.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000234207dcfb1e0af-0064c45fe4-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.684 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:40:04.685 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:40:04.685 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:40:04.686 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:40:04.686 DEBUG:boto:Method: GET 2023-07-29T00:40:04.686 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:40:04.687 DEBUG:boto:Data: 2023-07-29T00:40:04.687 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.687 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:04.688 DEBUG:boto:Port: 8000 2023-07-29T00:40:04.688 DEBUG:boto:Params: {} 2023-07-29T00:40:04.689 DEBUG:boto:Token: None 2023-07-29T00:40:04.689 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:40:04.689 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ELzQhwH3jmrLk0fNE0/6AqqxLDE= 2023-07-29T00:40:04.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ELzQhwH3jmrLk0fNE0/6AqqxLDE=', 'Content-Length': '0'} 2023-07-29T00:40:04.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0c280803ffdfb35-0064c45fe4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.736 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:40:04.737 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:40:04.737 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:40:04.738 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:40:04.738 DEBUG:boto:Method: HEAD 2023-07-29T00:40:04.739 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:40:04.739 DEBUG:boto:Data: 2023-07-29T00:40:04.739 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.740 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:04.740 DEBUG:boto:Port: 8001 2023-07-29T00:40:04.740 DEBUG:boto:Params: {} 2023-07-29T00:40:04.741 DEBUG:boto:Token: None 2023-07-29T00:40:04.741 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:40:04.742 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HUEx+P5Ealcb1RUr0BPPb3+rp6I= 2023-07-29T00:40:04.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HUEx+P5Ealcb1RUr0BPPb3+rp6I=', 'Content-Length': '0'} 2023-07-29T00:40:04.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee2cfe7ddf3fd0dd-0064c45fe4-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.745 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:40:04.745 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:40:04.746 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:40:04.746 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:40:04.746 DEBUG:boto:Method: HEAD 2023-07-29T00:40:04.747 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:40:04.747 DEBUG:boto:Data: 2023-07-29T00:40:04.747 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.748 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:04.748 DEBUG:boto:Port: 8000 2023-07-29T00:40:04.748 DEBUG:boto:Params: {} 2023-07-29T00:40:04.749 DEBUG:boto:Token: None 2023-07-29T00:40:04.749 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:40:04.749 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HUEx+P5Ealcb1RUr0BPPb3+rp6I= 2023-07-29T00:40:04.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HUEx+P5Ealcb1RUr0BPPb3+rp6I=', 'Content-Length': '0'} 2023-07-29T00:40:04.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a7bcc9281797d41-0064c45fe4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-07-29T00:40:04.753 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:40:04.753 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:40:04.753 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:40:04.754 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:40:04.754 DEBUG:boto:Method: GET 2023-07-29T00:40:04.755 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:40:04.755 DEBUG:boto:Data: 2023-07-29T00:40:04.755 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.756 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:04.756 DEBUG:boto:Port: 8001 2023-07-29T00:40:04.756 DEBUG:boto:Params: {} 2023-07-29T00:40:04.757 DEBUG:boto:Token: None 2023-07-29T00:40:04.757 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:40:04.758 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ELzQhwH3jmrLk0fNE0/6AqqxLDE= 2023-07-29T00:40:04.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ELzQhwH3jmrLk0fNE0/6AqqxLDE=', 'Content-Length': '0'} 2023-07-29T00:40:04.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa5b67afa47edbd0-0064c45fe4-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.761 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:40:04.761 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:40:04.762 DEBUG:boto:path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:40:04.762 DEBUG:boto:auth_path=/hiiqos-59/obj44?versionId=null 2023-07-29T00:40:04.763 DEBUG:boto:Method: GET 2023-07-29T00:40:04.763 DEBUG:boto:Path: /hiiqos-59/obj44?versionId=null 2023-07-29T00:40:04.764 DEBUG:boto:Data: 2023-07-29T00:40:04.764 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.764 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:04.765 DEBUG:boto:Port: 8000 2023-07-29T00:40:04.765 DEBUG:boto:Params: {} 2023-07-29T00:40:04.766 DEBUG:boto:Token: None 2023-07-29T00:40:04.766 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/obj44?versionId=null 2023-07-29T00:40:04.766 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ELzQhwH3jmrLk0fNE0/6AqqxLDE= 2023-07-29T00:40:04.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ELzQhwH3jmrLk0fNE0/6AqqxLDE=', 'Content-Length': '0'} 2023-07-29T00:40:04.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079988ecfb72f7d18-0064c45fe4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.769 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:04.770 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:04.770 DEBUG:boto:path=/hiiqos-59/?prefix=obj44 2023-07-29T00:40:04.770 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj44 2023-07-29T00:40:04.771 DEBUG:boto:Method: GET 2023-07-29T00:40:04.771 DEBUG:boto:Path: /hiiqos-59/?prefix=obj44 2023-07-29T00:40:04.771 DEBUG:boto:Data: 2023-07-29T00:40:04.772 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.772 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:04.772 DEBUG:boto:Port: 8001 2023-07-29T00:40:04.773 DEBUG:boto:Params: {} 2023-07-29T00:40:04.773 DEBUG:boto:Token: None 2023-07-29T00:40:04.774 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/ 2023-07-29T00:40:04.774 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fJfzv399nmY7cDGVwqo3xvwwWjg= 2023-07-29T00:40:04.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fJfzv399nmY7cDGVwqo3xvwwWjg=', 'Content-Length': '0'} 2023-07-29T00:40:04.777 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c32c3f200f1c81d-0064c45fe4-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.777 DEBUG:boto:b'hiiqos-59obj441000falseobj442023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:04.778 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:04.779 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:04.779 DEBUG:boto:path=/hiiqos-59/?prefix=obj44 2023-07-29T00:40:04.779 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj44 2023-07-29T00:40:04.780 DEBUG:boto:Method: GET 2023-07-29T00:40:04.780 DEBUG:boto:Path: /hiiqos-59/?prefix=obj44 2023-07-29T00:40:04.780 DEBUG:boto:Data: 2023-07-29T00:40:04.781 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.781 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:04.782 DEBUG:boto:Port: 8000 2023-07-29T00:40:04.782 DEBUG:boto:Params: {} 2023-07-29T00:40:04.783 DEBUG:boto:Token: None 2023-07-29T00:40:04.784 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/ 2023-07-29T00:40:04.784 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fJfzv399nmY7cDGVwqo3xvwwWjg= 2023-07-29T00:40:04.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fJfzv399nmY7cDGVwqo3xvwwWjg=', 'Content-Length': '0'} 2023-07-29T00:40:04.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085e285ff1e0e8d95-0064c45fe4-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.790 DEBUG:boto:b'hiiqos-59obj441000falseobj442023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:04.791 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:40:04.792 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:40:04.793 DEBUG:boto:Method: HEAD 2023-07-29T00:40:04.793 DEBUG:boto:Path: /hiiqos-59/obj44 2023-07-29T00:40:04.793 DEBUG:boto:Data: 2023-07-29T00:40:04.794 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.794 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:04.795 DEBUG:boto:Port: 8001 2023-07-29T00:40:04.795 DEBUG:boto:Params: {} 2023-07-29T00:40:04.796 DEBUG:boto:Token: None 2023-07-29T00:40:04.796 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/obj44 2023-07-29T00:40:04.797 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4buhs8Q0vh1OJ+MwPsyebzlRJqw= 2023-07-29T00:40:04.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4buhs8Q0vh1OJ+MwPsyebzlRJqw=', 'Content-Length': '0'} 2023-07-29T00:40:04.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a725c41aa446f7cc-0064c45fe4-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.800 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:40:04.800 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:40:04.800 DEBUG:boto:Method: HEAD 2023-07-29T00:40:04.801 DEBUG:boto:Path: /hiiqos-59/obj44 2023-07-29T00:40:04.801 DEBUG:boto:Data: 2023-07-29T00:40:04.801 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.802 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:04.802 DEBUG:boto:Port: 8000 2023-07-29T00:40:04.802 DEBUG:boto:Params: {} 2023-07-29T00:40:04.803 DEBUG:boto:Token: None 2023-07-29T00:40:04.804 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/obj44 2023-07-29T00:40:04.805 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4buhs8Q0vh1OJ+MwPsyebzlRJqw= 2023-07-29T00:40:04.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4buhs8Q0vh1OJ+MwPsyebzlRJqw=', 'Content-Length': '0'} 2023-07-29T00:40:04.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3040de8324c7176-0064c45fe4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-07-29T00:40:04.811 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:40:04.811 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:40:04.812 DEBUG:boto:Method: GET 2023-07-29T00:40:04.812 DEBUG:boto:Path: /hiiqos-59/obj44 2023-07-29T00:40:04.812 DEBUG:boto:Data: 2023-07-29T00:40:04.813 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.813 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:04.814 DEBUG:boto:Port: 8001 2023-07-29T00:40:04.814 DEBUG:boto:Params: {} 2023-07-29T00:40:04.814 DEBUG:boto:Token: None 2023-07-29T00:40:04.815 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/obj44 2023-07-29T00:40:04.815 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4sGKjaVTbkUSqegZSjD62K/l1lg= 2023-07-29T00:40:04.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4sGKjaVTbkUSqegZSjD62K/l1lg=', 'Content-Length': '0'} 2023-07-29T00:40:04.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089307a0c767a0560-0064c45fe4-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.818 DEBUG:boto:path=/hiiqos-59/obj44 2023-07-29T00:40:04.819 DEBUG:boto:auth_path=/hiiqos-59/obj44 2023-07-29T00:40:04.819 DEBUG:boto:Method: GET 2023-07-29T00:40:04.819 DEBUG:boto:Path: /hiiqos-59/obj44 2023-07-29T00:40:04.820 DEBUG:boto:Data: 2023-07-29T00:40:04.822 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.823 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:04.823 DEBUG:boto:Port: 8000 2023-07-29T00:40:04.823 DEBUG:boto:Params: {} 2023-07-29T00:40:04.825 DEBUG:boto:Token: None 2023-07-29T00:40:04.825 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/obj44 2023-07-29T00:40:04.826 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4sGKjaVTbkUSqegZSjD62K/l1lg= 2023-07-29T00:40:04.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4sGKjaVTbkUSqegZSjD62K/l1lg=', 'Content-Length': '0'} 2023-07-29T00:40:04.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3238b8b78b17dc3-0064c45fe4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.872 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:04.873 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:04.873 DEBUG:boto:path=/hiiqos-59/?prefix=obj44 2023-07-29T00:40:04.874 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj44 2023-07-29T00:40:04.874 DEBUG:boto:Method: GET 2023-07-29T00:40:04.875 DEBUG:boto:Path: /hiiqos-59/?prefix=obj44 2023-07-29T00:40:04.875 DEBUG:boto:Data: 2023-07-29T00:40:04.875 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.876 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:04.876 DEBUG:boto:Port: 8001 2023-07-29T00:40:04.876 DEBUG:boto:Params: {} 2023-07-29T00:40:04.877 DEBUG:boto:Token: None 2023-07-29T00:40:04.877 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/ 2023-07-29T00:40:04.878 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fJfzv399nmY7cDGVwqo3xvwwWjg= 2023-07-29T00:40:04.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fJfzv399nmY7cDGVwqo3xvwwWjg=', 'Content-Length': '0'} 2023-07-29T00:40:04.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a74dd3112f9b89a-0064c45fe4-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.881 DEBUG:boto:b'hiiqos-59obj441000falseobj442023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:04.882 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:04.883 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:04.883 DEBUG:boto:path=/hiiqos-59/?prefix=obj44 2023-07-29T00:40:04.883 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj44 2023-07-29T00:40:04.884 DEBUG:boto:Method: GET 2023-07-29T00:40:04.884 DEBUG:boto:Path: /hiiqos-59/?prefix=obj44 2023-07-29T00:40:04.885 DEBUG:boto:Data: 2023-07-29T00:40:04.885 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.885 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:04.886 DEBUG:boto:Port: 8000 2023-07-29T00:40:04.886 DEBUG:boto:Params: {} 2023-07-29T00:40:04.886 DEBUG:boto:Token: None 2023-07-29T00:40:04.887 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/ 2023-07-29T00:40:04.887 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fJfzv399nmY7cDGVwqo3xvwwWjg= 2023-07-29T00:40:04.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fJfzv399nmY7cDGVwqo3xvwwWjg=', 'Content-Length': '0'} 2023-07-29T00:40:04.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0bc4b3a30efd35c-0064c45fe4-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.893 DEBUG:boto:b'hiiqos-59obj441000falseobj442023-07-29T00:26:54.593Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:04.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-07-29T00:40:04.894 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:40:04.894 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:40:04.895 DEBUG:boto:path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:40:04.895 DEBUG:boto:auth_path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:40:04.895 DEBUG:boto:Method: GET 2023-07-29T00:40:04.896 DEBUG:boto:Path: /hiiqos-59/obj46?versionId=null 2023-07-29T00:40:04.896 DEBUG:boto:Data: 2023-07-29T00:40:04.897 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.897 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:04.897 DEBUG:boto:Port: 8001 2023-07-29T00:40:04.898 DEBUG:boto:Params: {} 2023-07-29T00:40:04.898 DEBUG:boto:Token: None 2023-07-29T00:40:04.898 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/obj46?versionId=null 2023-07-29T00:40:04.899 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4WVTewZNTKebAO49nJg4G3ktMuE= 2023-07-29T00:40:04.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4WVTewZNTKebAO49nJg4G3ktMuE=', 'Content-Length': '0'} 2023-07-29T00:40:04.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000407d195528211a50-0064c45fe4-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.948 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:40:04.949 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:40:04.949 DEBUG:boto:path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:40:04.950 DEBUG:boto:auth_path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:40:04.950 DEBUG:boto:Method: GET 2023-07-29T00:40:04.950 DEBUG:boto:Path: /hiiqos-59/obj46?versionId=null 2023-07-29T00:40:04.951 DEBUG:boto:Data: 2023-07-29T00:40:04.951 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.952 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:04.952 DEBUG:boto:Port: 8000 2023-07-29T00:40:04.952 DEBUG:boto:Params: {} 2023-07-29T00:40:04.953 DEBUG:boto:Token: None 2023-07-29T00:40:04.953 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/obj46?versionId=null 2023-07-29T00:40:04.954 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4WVTewZNTKebAO49nJg4G3ktMuE= 2023-07-29T00:40:04.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4WVTewZNTKebAO49nJg4G3ktMuE=', 'Content-Length': '0'} 2023-07-29T00:40:04.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000624b0a77d00dda1b-0064c45fe4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.957 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:40:04.957 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:40:04.958 DEBUG:boto:path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:40:04.958 DEBUG:boto:auth_path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:40:04.958 DEBUG:boto:Method: HEAD 2023-07-29T00:40:04.959 DEBUG:boto:Path: /hiiqos-59/obj46?versionId=null 2023-07-29T00:40:04.959 DEBUG:boto:Data: 2023-07-29T00:40:04.959 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.960 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:04.960 DEBUG:boto:Port: 8001 2023-07-29T00:40:04.960 DEBUG:boto:Params: {} 2023-07-29T00:40:04.961 DEBUG:boto:Token: None 2023-07-29T00:40:04.961 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/obj46?versionId=null 2023-07-29T00:40:04.962 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:h5SlNueo+be81D1TBai6/AF8k6Y= 2023-07-29T00:40:04.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:h5SlNueo+be81D1TBai6/AF8k6Y=', 'Content-Length': '0'} 2023-07-29T00:40:04.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f1b730d09d0928e-0064c45fe4-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.965 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:40:04.965 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:40:04.965 DEBUG:boto:path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:40:04.966 DEBUG:boto:auth_path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:40:04.967 DEBUG:boto:Method: HEAD 2023-07-29T00:40:04.967 DEBUG:boto:Path: /hiiqos-59/obj46?versionId=null 2023-07-29T00:40:04.967 DEBUG:boto:Data: 2023-07-29T00:40:04.968 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.968 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:04.968 DEBUG:boto:Port: 8000 2023-07-29T00:40:04.969 DEBUG:boto:Params: {} 2023-07-29T00:40:04.969 DEBUG:boto:Token: None 2023-07-29T00:40:04.970 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/obj46?versionId=null 2023-07-29T00:40:04.970 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:h5SlNueo+be81D1TBai6/AF8k6Y= 2023-07-29T00:40:04.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:h5SlNueo+be81D1TBai6/AF8k6Y=', 'Content-Length': '0'} 2023-07-29T00:40:04.972 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000789e843afb7e2e05-0064c45fe4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-07-29T00:40:04.973 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:40:04.974 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:40:04.974 DEBUG:boto:path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:40:04.974 DEBUG:boto:auth_path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:40:04.975 DEBUG:boto:Method: GET 2023-07-29T00:40:04.975 DEBUG:boto:Path: /hiiqos-59/obj46?versionId=null 2023-07-29T00:40:04.976 DEBUG:boto:Data: 2023-07-29T00:40:04.976 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.976 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:04.977 DEBUG:boto:Port: 8001 2023-07-29T00:40:04.978 DEBUG:boto:Params: {} 2023-07-29T00:40:04.978 DEBUG:boto:Token: None 2023-07-29T00:40:04.978 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/obj46?versionId=null 2023-07-29T00:40:04.979 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4WVTewZNTKebAO49nJg4G3ktMuE= 2023-07-29T00:40:04.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4WVTewZNTKebAO49nJg4G3ktMuE=', 'Content-Length': '0'} 2023-07-29T00:40:04.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e29702f9c630432d-0064c45fe4-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:04.982 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:40:04.982 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:40:04.983 DEBUG:boto:path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:40:04.983 DEBUG:boto:auth_path=/hiiqos-59/obj46?versionId=null 2023-07-29T00:40:04.983 DEBUG:boto:Method: GET 2023-07-29T00:40:04.984 DEBUG:boto:Path: /hiiqos-59/obj46?versionId=null 2023-07-29T00:40:04.984 DEBUG:boto:Data: 2023-07-29T00:40:04.985 DEBUG:boto:Headers: {} 2023-07-29T00:40:04.985 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:04.986 DEBUG:boto:Port: 8000 2023-07-29T00:40:04.986 DEBUG:boto:Params: {} 2023-07-29T00:40:04.986 DEBUG:boto:Token: None 2023-07-29T00:40:04.987 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:04 GMT /hiiqos-59/obj46?versionId=null 2023-07-29T00:40:04.987 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4WVTewZNTKebAO49nJg4G3ktMuE= 2023-07-29T00:40:04.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4WVTewZNTKebAO49nJg4G3ktMuE=', 'Content-Length': '0'} 2023-07-29T00:40:04.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000676ad1f28e5179eb-0064c45fe4-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.032 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:05.033 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:05.033 DEBUG:boto:path=/hiiqos-59/?prefix=obj46 2023-07-29T00:40:05.034 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj46 2023-07-29T00:40:05.034 DEBUG:boto:Method: GET 2023-07-29T00:40:05.034 DEBUG:boto:Path: /hiiqos-59/?prefix=obj46 2023-07-29T00:40:05.035 DEBUG:boto:Data: 2023-07-29T00:40:05.035 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.035 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:05.036 DEBUG:boto:Port: 8001 2023-07-29T00:40:05.036 DEBUG:boto:Params: {} 2023-07-29T00:40:05.037 DEBUG:boto:Token: None 2023-07-29T00:40:05.037 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/ 2023-07-29T00:40:05.037 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI= 2023-07-29T00:40:05.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI=', 'Content-Length': '0'} 2023-07-29T00:40:05.040 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2e5728afff2f175-0064c45fe4-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.041 DEBUG:boto:b'hiiqos-59obj461000falseobj462023-07-29T00:33:06.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:05.042 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:05.042 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:05.042 DEBUG:boto:path=/hiiqos-59/?prefix=obj46 2023-07-29T00:40:05.043 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj46 2023-07-29T00:40:05.043 DEBUG:boto:Method: GET 2023-07-29T00:40:05.043 DEBUG:boto:Path: /hiiqos-59/?prefix=obj46 2023-07-29T00:40:05.044 DEBUG:boto:Data: 2023-07-29T00:40:05.044 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.045 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:05.045 DEBUG:boto:Port: 8000 2023-07-29T00:40:05.045 DEBUG:boto:Params: {} 2023-07-29T00:40:05.046 DEBUG:boto:Token: None 2023-07-29T00:40:05.046 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/ 2023-07-29T00:40:05.047 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI= 2023-07-29T00:40:05.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI=', 'Content-Length': '0'} 2023-07-29T00:40:05.051 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe4e0a1d71bfdc03-0064c45fe4-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.052 DEBUG:boto:b'hiiqos-59obj461000falseobj462023-07-29T00:33:06.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:05.052 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:40:05.053 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:40:05.053 DEBUG:boto:Method: HEAD 2023-07-29T00:40:05.054 DEBUG:boto:Path: /hiiqos-59/obj46 2023-07-29T00:40:05.054 DEBUG:boto:Data: 2023-07-29T00:40:05.054 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.055 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:05.055 DEBUG:boto:Port: 8001 2023-07-29T00:40:05.055 DEBUG:boto:Params: {} 2023-07-29T00:40:05.056 DEBUG:boto:Token: None 2023-07-29T00:40:05.056 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj46 2023-07-29T00:40:05.056 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:e1DnRuB46i2FotnFqleC64iFZX8= 2023-07-29T00:40:05.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:e1DnRuB46i2FotnFqleC64iFZX8=', 'Content-Length': '0'} 2023-07-29T00:40:05.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003840c358e6afbfb9-0064c45fe4-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.059 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:40:05.060 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:40:05.060 DEBUG:boto:Method: HEAD 2023-07-29T00:40:05.060 DEBUG:boto:Path: /hiiqos-59/obj46 2023-07-29T00:40:05.061 DEBUG:boto:Data: 2023-07-29T00:40:05.061 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.061 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:05.062 DEBUG:boto:Port: 8000 2023-07-29T00:40:05.062 DEBUG:boto:Params: {} 2023-07-29T00:40:05.063 DEBUG:boto:Token: None 2023-07-29T00:40:05.063 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj46 2023-07-29T00:40:05.063 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:e1DnRuB46i2FotnFqleC64iFZX8= 2023-07-29T00:40:05.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:e1DnRuB46i2FotnFqleC64iFZX8=', 'Content-Length': '0'} 2023-07-29T00:40:05.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f42ba812eba8fa6-0064c45fe5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-07-29T00:40:05.067 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:40:05.067 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:40:05.067 DEBUG:boto:Method: GET 2023-07-29T00:40:05.068 DEBUG:boto:Path: /hiiqos-59/obj46 2023-07-29T00:40:05.068 DEBUG:boto:Data: 2023-07-29T00:40:05.068 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.069 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:05.069 DEBUG:boto:Port: 8001 2023-07-29T00:40:05.070 DEBUG:boto:Params: {} 2023-07-29T00:40:05.070 DEBUG:boto:Token: None 2023-07-29T00:40:05.070 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj46 2023-07-29T00:40:05.071 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:am0wlRHgjSNNCeXKc6UwjzqXVr0= 2023-07-29T00:40:05.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:am0wlRHgjSNNCeXKc6UwjzqXVr0=', 'Content-Length': '0'} 2023-07-29T00:40:05.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000797cb9e97131feda-0064c45fe5-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.074 DEBUG:boto:path=/hiiqos-59/obj46 2023-07-29T00:40:05.074 DEBUG:boto:auth_path=/hiiqos-59/obj46 2023-07-29T00:40:05.075 DEBUG:boto:Method: GET 2023-07-29T00:40:05.075 DEBUG:boto:Path: /hiiqos-59/obj46 2023-07-29T00:40:05.075 DEBUG:boto:Data: 2023-07-29T00:40:05.076 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.076 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:05.076 DEBUG:boto:Port: 8000 2023-07-29T00:40:05.077 DEBUG:boto:Params: {} 2023-07-29T00:40:05.077 DEBUG:boto:Token: None 2023-07-29T00:40:05.078 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj46 2023-07-29T00:40:05.078 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:am0wlRHgjSNNCeXKc6UwjzqXVr0= 2023-07-29T00:40:05.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:am0wlRHgjSNNCeXKc6UwjzqXVr0=', 'Content-Length': '0'} 2023-07-29T00:40:05.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3bfd7d260e88409-0064c45fe5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.081 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:05.081 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:05.082 DEBUG:boto:path=/hiiqos-59/?prefix=obj46 2023-07-29T00:40:05.082 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj46 2023-07-29T00:40:05.082 DEBUG:boto:Method: GET 2023-07-29T00:40:05.083 DEBUG:boto:Path: /hiiqos-59/?prefix=obj46 2023-07-29T00:40:05.083 DEBUG:boto:Data: 2023-07-29T00:40:05.084 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.084 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:05.084 DEBUG:boto:Port: 8001 2023-07-29T00:40:05.085 DEBUG:boto:Params: {} 2023-07-29T00:40:05.085 DEBUG:boto:Token: None 2023-07-29T00:40:05.085 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/ 2023-07-29T00:40:05.086 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI= 2023-07-29T00:40:05.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI=', 'Content-Length': '0'} 2023-07-29T00:40:05.089 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005effd2d35ed693ad-0064c45fe5-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.136 DEBUG:boto:b'hiiqos-59obj461000falseobj462023-07-29T00:33:06.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:05.137 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:05.138 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:05.138 DEBUG:boto:path=/hiiqos-59/?prefix=obj46 2023-07-29T00:40:05.139 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj46 2023-07-29T00:40:05.139 DEBUG:boto:Method: GET 2023-07-29T00:40:05.139 DEBUG:boto:Path: /hiiqos-59/?prefix=obj46 2023-07-29T00:40:05.140 DEBUG:boto:Data: 2023-07-29T00:40:05.140 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.140 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:05.141 DEBUG:boto:Port: 8000 2023-07-29T00:40:05.141 DEBUG:boto:Params: {} 2023-07-29T00:40:05.142 DEBUG:boto:Token: None 2023-07-29T00:40:05.142 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/ 2023-07-29T00:40:05.143 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI= 2023-07-29T00:40:05.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI=', 'Content-Length': '0'} 2023-07-29T00:40:05.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056ddec6b2c22b946-0064c45fe5-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.148 DEBUG:boto:b'hiiqos-59obj461000falseobj462023-07-29T00:33:06.801Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:05.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-07-29T00:40:05.149 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:40:05.149 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:40:05.150 DEBUG:boto:path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:40:05.150 DEBUG:boto:auth_path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:40:05.151 DEBUG:boto:Method: GET 2023-07-29T00:40:05.151 DEBUG:boto:Path: /hiiqos-59/obj51?versionId=null 2023-07-29T00:40:05.151 DEBUG:boto:Data: 2023-07-29T00:40:05.152 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.152 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:05.152 DEBUG:boto:Port: 8001 2023-07-29T00:40:05.153 DEBUG:boto:Params: {} 2023-07-29T00:40:05.153 DEBUG:boto:Token: None 2023-07-29T00:40:05.154 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj51?versionId=null 2023-07-29T00:40:05.154 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DWwTrMLElRmta+X4Mpph05S+7DI= 2023-07-29T00:40:05.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DWwTrMLElRmta+X4Mpph05S+7DI=', 'Content-Length': '0'} 2023-07-29T00:40:05.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c248f2c77d5d8f0-0064c45fe5-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.157 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:40:05.158 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:40:05.158 DEBUG:boto:path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:40:05.159 DEBUG:boto:auth_path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:40:05.159 DEBUG:boto:Method: GET 2023-07-29T00:40:05.159 DEBUG:boto:Path: /hiiqos-59/obj51?versionId=null 2023-07-29T00:40:05.160 DEBUG:boto:Data: 2023-07-29T00:40:05.160 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.160 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:05.161 DEBUG:boto:Port: 8000 2023-07-29T00:40:05.161 DEBUG:boto:Params: {} 2023-07-29T00:40:05.162 DEBUG:boto:Token: None 2023-07-29T00:40:05.162 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj51?versionId=null 2023-07-29T00:40:05.162 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DWwTrMLElRmta+X4Mpph05S+7DI= 2023-07-29T00:40:05.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DWwTrMLElRmta+X4Mpph05S+7DI=', 'Content-Length': '0'} 2023-07-29T00:40:05.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e4ca72dbbb6a717-0064c45fe5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.208 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:40:05.209 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:40:05.209 DEBUG:boto:path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:40:05.210 DEBUG:boto:auth_path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:40:05.210 DEBUG:boto:Method: HEAD 2023-07-29T00:40:05.210 DEBUG:boto:Path: /hiiqos-59/obj51?versionId=null 2023-07-29T00:40:05.211 DEBUG:boto:Data: 2023-07-29T00:40:05.211 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.211 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:05.212 DEBUG:boto:Port: 8001 2023-07-29T00:40:05.212 DEBUG:boto:Params: {} 2023-07-29T00:40:05.212 DEBUG:boto:Token: None 2023-07-29T00:40:05.213 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj51?versionId=null 2023-07-29T00:40:05.213 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CkRKHZKEUserfVw/+37ciSYhNtI= 2023-07-29T00:40:05.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CkRKHZKEUserfVw/+37ciSYhNtI=', 'Content-Length': '0'} 2023-07-29T00:40:05.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000585a54d3438620dc-0064c45fe5-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.216 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:40:05.217 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:40:05.217 DEBUG:boto:path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:40:05.217 DEBUG:boto:auth_path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:40:05.218 DEBUG:boto:Method: HEAD 2023-07-29T00:40:05.218 DEBUG:boto:Path: /hiiqos-59/obj51?versionId=null 2023-07-29T00:40:05.218 DEBUG:boto:Data: 2023-07-29T00:40:05.219 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.219 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:05.219 DEBUG:boto:Port: 8000 2023-07-29T00:40:05.220 DEBUG:boto:Params: {} 2023-07-29T00:40:05.220 DEBUG:boto:Token: None 2023-07-29T00:40:05.220 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj51?versionId=null 2023-07-29T00:40:05.221 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CkRKHZKEUserfVw/+37ciSYhNtI= 2023-07-29T00:40:05.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CkRKHZKEUserfVw/+37ciSYhNtI=', 'Content-Length': '0'} 2023-07-29T00:40:05.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cde2ecb3a2bbaa0f-0064c45fe5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-07-29T00:40:05.224 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:40:05.225 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:40:05.225 DEBUG:boto:path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:40:05.225 DEBUG:boto:auth_path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:40:05.226 DEBUG:boto:Method: GET 2023-07-29T00:40:05.226 DEBUG:boto:Path: /hiiqos-59/obj51?versionId=null 2023-07-29T00:40:05.227 DEBUG:boto:Data: 2023-07-29T00:40:05.227 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.227 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:05.228 DEBUG:boto:Port: 8001 2023-07-29T00:40:05.228 DEBUG:boto:Params: {} 2023-07-29T00:40:05.229 DEBUG:boto:Token: None 2023-07-29T00:40:05.229 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj51?versionId=null 2023-07-29T00:40:05.229 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DWwTrMLElRmta+X4Mpph05S+7DI= 2023-07-29T00:40:05.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DWwTrMLElRmta+X4Mpph05S+7DI=', 'Content-Length': '0'} 2023-07-29T00:40:05.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008012cb212362e973-0064c45fe5-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.234 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:40:05.234 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:40:05.234 DEBUG:boto:path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:40:05.235 DEBUG:boto:auth_path=/hiiqos-59/obj51?versionId=null 2023-07-29T00:40:05.235 DEBUG:boto:Method: GET 2023-07-29T00:40:05.236 DEBUG:boto:Path: /hiiqos-59/obj51?versionId=null 2023-07-29T00:40:05.236 DEBUG:boto:Data: 2023-07-29T00:40:05.236 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.237 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:05.237 DEBUG:boto:Port: 8000 2023-07-29T00:40:05.237 DEBUG:boto:Params: {} 2023-07-29T00:40:05.238 DEBUG:boto:Token: None 2023-07-29T00:40:05.238 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj51?versionId=null 2023-07-29T00:40:05.239 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DWwTrMLElRmta+X4Mpph05S+7DI= 2023-07-29T00:40:05.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DWwTrMLElRmta+X4Mpph05S+7DI=', 'Content-Length': '0'} 2023-07-29T00:40:05.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043ec4ccd8d9ca82b-0064c45fe5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.242 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:05.242 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:05.243 DEBUG:boto:path=/hiiqos-59/?prefix=obj51 2023-07-29T00:40:05.243 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj51 2023-07-29T00:40:05.243 DEBUG:boto:Method: GET 2023-07-29T00:40:05.244 DEBUG:boto:Path: /hiiqos-59/?prefix=obj51 2023-07-29T00:40:05.244 DEBUG:boto:Data: 2023-07-29T00:40:05.244 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.245 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:05.245 DEBUG:boto:Port: 8001 2023-07-29T00:40:05.246 DEBUG:boto:Params: {} 2023-07-29T00:40:05.246 DEBUG:boto:Token: None 2023-07-29T00:40:05.246 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/ 2023-07-29T00:40:05.247 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI= 2023-07-29T00:40:05.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI=', 'Content-Length': '0'} 2023-07-29T00:40:05.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee21c20c1c1201dd-0064c45fe5-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.250 DEBUG:boto:b'hiiqos-59obj511000falseobj512023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:05.251 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:05.251 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:05.252 DEBUG:boto:path=/hiiqos-59/?prefix=obj51 2023-07-29T00:40:05.252 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj51 2023-07-29T00:40:05.252 DEBUG:boto:Method: GET 2023-07-29T00:40:05.253 DEBUG:boto:Path: /hiiqos-59/?prefix=obj51 2023-07-29T00:40:05.253 DEBUG:boto:Data: 2023-07-29T00:40:05.254 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.254 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:05.254 DEBUG:boto:Port: 8000 2023-07-29T00:40:05.255 DEBUG:boto:Params: {} 2023-07-29T00:40:05.255 DEBUG:boto:Token: None 2023-07-29T00:40:05.256 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/ 2023-07-29T00:40:05.256 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI= 2023-07-29T00:40:05.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI=', 'Content-Length': '0'} 2023-07-29T00:40:05.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005845e86d4a26994b-0064c45fe5-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.262 DEBUG:boto:b'hiiqos-59obj511000falseobj512023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:05.263 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:40:05.263 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:40:05.263 DEBUG:boto:Method: HEAD 2023-07-29T00:40:05.264 DEBUG:boto:Path: /hiiqos-59/obj51 2023-07-29T00:40:05.264 DEBUG:boto:Data: 2023-07-29T00:40:05.265 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.265 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:05.265 DEBUG:boto:Port: 8001 2023-07-29T00:40:05.266 DEBUG:boto:Params: {} 2023-07-29T00:40:05.266 DEBUG:boto:Token: None 2023-07-29T00:40:05.266 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj51 2023-07-29T00:40:05.267 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:avEVS/csWGpL0cbfRt0qXIq+K6E= 2023-07-29T00:40:05.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:avEVS/csWGpL0cbfRt0qXIq+K6E=', 'Content-Length': '0'} 2023-07-29T00:40:05.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000173eec63eb27da66-0064c45fe5-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.270 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:40:05.270 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:40:05.270 DEBUG:boto:Method: HEAD 2023-07-29T00:40:05.271 DEBUG:boto:Path: /hiiqos-59/obj51 2023-07-29T00:40:05.271 DEBUG:boto:Data: 2023-07-29T00:40:05.272 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.272 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:05.272 DEBUG:boto:Port: 8000 2023-07-29T00:40:05.273 DEBUG:boto:Params: {} 2023-07-29T00:40:05.273 DEBUG:boto:Token: None 2023-07-29T00:40:05.273 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj51 2023-07-29T00:40:05.274 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:avEVS/csWGpL0cbfRt0qXIq+K6E= 2023-07-29T00:40:05.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:avEVS/csWGpL0cbfRt0qXIq+K6E=', 'Content-Length': '0'} 2023-07-29T00:40:05.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062364384283dbd23-0064c45fe5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-07-29T00:40:05.277 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:40:05.278 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:40:05.278 DEBUG:boto:Method: GET 2023-07-29T00:40:05.279 DEBUG:boto:Path: /hiiqos-59/obj51 2023-07-29T00:40:05.279 DEBUG:boto:Data: 2023-07-29T00:40:05.279 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.280 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:05.280 DEBUG:boto:Port: 8001 2023-07-29T00:40:05.280 DEBUG:boto:Params: {} 2023-07-29T00:40:05.281 DEBUG:boto:Token: None 2023-07-29T00:40:05.281 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj51 2023-07-29T00:40:05.282 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:U3rFYn4ITc3VaVXXMd6kq5fxeFc= 2023-07-29T00:40:05.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:U3rFYn4ITc3VaVXXMd6kq5fxeFc=', 'Content-Length': '0'} 2023-07-29T00:40:05.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bd759cfff372d65-0064c45fe5-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.332 DEBUG:boto:path=/hiiqos-59/obj51 2023-07-29T00:40:05.333 DEBUG:boto:auth_path=/hiiqos-59/obj51 2023-07-29T00:40:05.333 DEBUG:boto:Method: GET 2023-07-29T00:40:05.334 DEBUG:boto:Path: /hiiqos-59/obj51 2023-07-29T00:40:05.334 DEBUG:boto:Data: 2023-07-29T00:40:05.334 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.335 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:05.335 DEBUG:boto:Port: 8000 2023-07-29T00:40:05.335 DEBUG:boto:Params: {} 2023-07-29T00:40:05.336 DEBUG:boto:Token: None 2023-07-29T00:40:05.336 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj51 2023-07-29T00:40:05.337 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:U3rFYn4ITc3VaVXXMd6kq5fxeFc= 2023-07-29T00:40:05.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:U3rFYn4ITc3VaVXXMd6kq5fxeFc=', 'Content-Length': '0'} 2023-07-29T00:40:05.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d492a3367c0fa43-0064c45fe5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.340 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:05.340 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:05.341 DEBUG:boto:path=/hiiqos-59/?prefix=obj51 2023-07-29T00:40:05.341 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj51 2023-07-29T00:40:05.341 DEBUG:boto:Method: GET 2023-07-29T00:40:05.342 DEBUG:boto:Path: /hiiqos-59/?prefix=obj51 2023-07-29T00:40:05.342 DEBUG:boto:Data: 2023-07-29T00:40:05.342 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.343 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:05.343 DEBUG:boto:Port: 8001 2023-07-29T00:40:05.344 DEBUG:boto:Params: {} 2023-07-29T00:40:05.344 DEBUG:boto:Token: None 2023-07-29T00:40:05.345 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/ 2023-07-29T00:40:05.345 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI= 2023-07-29T00:40:05.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI=', 'Content-Length': '0'} 2023-07-29T00:40:05.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068eacb5e79f1d8ad-0064c45fe5-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.350 DEBUG:boto:b'hiiqos-59obj511000falseobj512023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:05.350 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:05.351 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:05.351 DEBUG:boto:path=/hiiqos-59/?prefix=obj51 2023-07-29T00:40:05.352 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj51 2023-07-29T00:40:05.352 DEBUG:boto:Method: GET 2023-07-29T00:40:05.352 DEBUG:boto:Path: /hiiqos-59/?prefix=obj51 2023-07-29T00:40:05.353 DEBUG:boto:Data: 2023-07-29T00:40:05.353 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.354 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:05.354 DEBUG:boto:Port: 8000 2023-07-29T00:40:05.354 DEBUG:boto:Params: {} 2023-07-29T00:40:05.355 DEBUG:boto:Token: None 2023-07-29T00:40:05.355 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/ 2023-07-29T00:40:05.356 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI= 2023-07-29T00:40:05.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI=', 'Content-Length': '0'} 2023-07-29T00:40:05.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ffd1c7224b8e100-0064c45fe5-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.362 DEBUG:boto:b'hiiqos-59obj511000falseobj512023-07-29T00:27:51.180Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:05.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-07-29T00:40:05.363 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:40:05.363 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:40:05.364 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:40:05.364 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:40:05.365 DEBUG:boto:Method: GET 2023-07-29T00:40:05.365 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:40:05.366 DEBUG:boto:Data: 2023-07-29T00:40:05.366 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.367 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:05.367 DEBUG:boto:Port: 8001 2023-07-29T00:40:05.368 DEBUG:boto:Params: {} 2023-07-29T00:40:05.368 DEBUG:boto:Token: None 2023-07-29T00:40:05.368 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:40:05.369 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yIwoEwJg8wagmIOOKeGYU3UK65M= 2023-07-29T00:40:05.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yIwoEwJg8wagmIOOKeGYU3UK65M=', 'Content-Length': '0'} 2023-07-29T00:40:05.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1944dbe38aa7cda-0064c45fe5-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.372 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:40:05.373 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:40:05.373 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:40:05.373 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:40:05.374 DEBUG:boto:Method: GET 2023-07-29T00:40:05.374 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:40:05.374 DEBUG:boto:Data: 2023-07-29T00:40:05.375 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.375 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:05.376 DEBUG:boto:Port: 8000 2023-07-29T00:40:05.376 DEBUG:boto:Params: {} 2023-07-29T00:40:05.376 DEBUG:boto:Token: None 2023-07-29T00:40:05.377 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:40:05.377 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yIwoEwJg8wagmIOOKeGYU3UK65M= 2023-07-29T00:40:05.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yIwoEwJg8wagmIOOKeGYU3UK65M=', 'Content-Length': '0'} 2023-07-29T00:40:05.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcd6714ed1fc0351-0064c45fe5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.380 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:40:05.381 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:40:05.381 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:40:05.381 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:40:05.382 DEBUG:boto:Method: HEAD 2023-07-29T00:40:05.382 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:40:05.382 DEBUG:boto:Data: 2023-07-29T00:40:05.383 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.383 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:05.383 DEBUG:boto:Port: 8001 2023-07-29T00:40:05.384 DEBUG:boto:Params: {} 2023-07-29T00:40:05.384 DEBUG:boto:Token: None 2023-07-29T00:40:05.385 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:40:05.385 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9vgW8O11lSPoKRXNb7cy1rwetJw= 2023-07-29T00:40:05.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9vgW8O11lSPoKRXNb7cy1rwetJw=', 'Content-Length': '0'} 2023-07-29T00:40:05.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b8e29657647a839-0064c45fe5-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.388 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:40:05.388 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:40:05.389 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:40:05.389 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:40:05.389 DEBUG:boto:Method: HEAD 2023-07-29T00:40:05.390 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:40:05.390 DEBUG:boto:Data: 2023-07-29T00:40:05.390 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.391 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:05.391 DEBUG:boto:Port: 8000 2023-07-29T00:40:05.391 DEBUG:boto:Params: {} 2023-07-29T00:40:05.392 DEBUG:boto:Token: None 2023-07-29T00:40:05.392 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:40:05.393 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9vgW8O11lSPoKRXNb7cy1rwetJw= 2023-07-29T00:40:05.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9vgW8O11lSPoKRXNb7cy1rwetJw=', 'Content-Length': '0'} 2023-07-29T00:40:05.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d5b5d3b1d1f79ff-0064c45fe5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-07-29T00:40:05.396 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:40:05.396 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:40:05.397 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:40:05.397 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:40:05.398 DEBUG:boto:Method: GET 2023-07-29T00:40:05.398 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:40:05.398 DEBUG:boto:Data: 2023-07-29T00:40:05.399 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.399 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:05.400 DEBUG:boto:Port: 8001 2023-07-29T00:40:05.400 DEBUG:boto:Params: {} 2023-07-29T00:40:05.400 DEBUG:boto:Token: None 2023-07-29T00:40:05.401 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:40:05.401 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yIwoEwJg8wagmIOOKeGYU3UK65M= 2023-07-29T00:40:05.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yIwoEwJg8wagmIOOKeGYU3UK65M=', 'Content-Length': '0'} 2023-07-29T00:40:05.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a38fb785d290f88-0064c45fe5-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.404 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:40:05.405 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:40:05.405 DEBUG:boto:path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:40:05.406 DEBUG:boto:auth_path=/hiiqos-59/obj55?versionId=null 2023-07-29T00:40:05.406 DEBUG:boto:Method: GET 2023-07-29T00:40:05.406 DEBUG:boto:Path: /hiiqos-59/obj55?versionId=null 2023-07-29T00:40:05.407 DEBUG:boto:Data: 2023-07-29T00:40:05.407 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.408 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:05.408 DEBUG:boto:Port: 8000 2023-07-29T00:40:05.408 DEBUG:boto:Params: {} 2023-07-29T00:40:05.409 DEBUG:boto:Token: None 2023-07-29T00:40:05.409 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj55?versionId=null 2023-07-29T00:40:05.410 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yIwoEwJg8wagmIOOKeGYU3UK65M= 2023-07-29T00:40:05.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yIwoEwJg8wagmIOOKeGYU3UK65M=', 'Content-Length': '0'} 2023-07-29T00:40:05.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000551ebf1e56b13839-0064c45fe5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.413 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:05.413 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:05.414 DEBUG:boto:path=/hiiqos-59/?prefix=obj55 2023-07-29T00:40:05.414 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj55 2023-07-29T00:40:05.414 DEBUG:boto:Method: GET 2023-07-29T00:40:05.415 DEBUG:boto:Path: /hiiqos-59/?prefix=obj55 2023-07-29T00:40:05.415 DEBUG:boto:Data: 2023-07-29T00:40:05.416 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.416 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:05.417 DEBUG:boto:Port: 8001 2023-07-29T00:40:05.417 DEBUG:boto:Params: {} 2023-07-29T00:40:05.417 DEBUG:boto:Token: None 2023-07-29T00:40:05.418 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/ 2023-07-29T00:40:05.418 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI= 2023-07-29T00:40:05.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI=', 'Content-Length': '0'} 2023-07-29T00:40:05.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a0f6d202d4ef257-0064c45fe5-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.422 DEBUG:boto:b'hiiqos-59obj551000falseobj552023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:05.423 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:05.423 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:05.423 DEBUG:boto:path=/hiiqos-59/?prefix=obj55 2023-07-29T00:40:05.424 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj55 2023-07-29T00:40:05.424 DEBUG:boto:Method: GET 2023-07-29T00:40:05.425 DEBUG:boto:Path: /hiiqos-59/?prefix=obj55 2023-07-29T00:40:05.425 DEBUG:boto:Data: 2023-07-29T00:40:05.425 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.426 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:05.426 DEBUG:boto:Port: 8000 2023-07-29T00:40:05.427 DEBUG:boto:Params: {} 2023-07-29T00:40:05.427 DEBUG:boto:Token: None 2023-07-29T00:40:05.428 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/ 2023-07-29T00:40:05.428 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI= 2023-07-29T00:40:05.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI=', 'Content-Length': '0'} 2023-07-29T00:40:05.433 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083ad798955588f35-0064c45fe5-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.434 DEBUG:boto:b'hiiqos-59obj551000falseobj552023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:05.434 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:40:05.435 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:40:05.435 DEBUG:boto:Method: HEAD 2023-07-29T00:40:05.435 DEBUG:boto:Path: /hiiqos-59/obj55 2023-07-29T00:40:05.436 DEBUG:boto:Data: 2023-07-29T00:40:05.436 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.437 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:05.437 DEBUG:boto:Port: 8001 2023-07-29T00:40:05.437 DEBUG:boto:Params: {} 2023-07-29T00:40:05.438 DEBUG:boto:Token: None 2023-07-29T00:40:05.438 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj55 2023-07-29T00:40:05.439 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HRkAKEYK05tzOMDScB263QWZXyQ= 2023-07-29T00:40:05.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HRkAKEYK05tzOMDScB263QWZXyQ=', 'Content-Length': '0'} 2023-07-29T00:40:05.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5e69d4d172bad9f-0064c45fe5-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.442 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:40:05.443 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:40:05.443 DEBUG:boto:Method: HEAD 2023-07-29T00:40:05.443 DEBUG:boto:Path: /hiiqos-59/obj55 2023-07-29T00:40:05.444 DEBUG:boto:Data: 2023-07-29T00:40:05.444 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.445 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:05.445 DEBUG:boto:Port: 8000 2023-07-29T00:40:05.445 DEBUG:boto:Params: {} 2023-07-29T00:40:05.446 DEBUG:boto:Token: None 2023-07-29T00:40:05.446 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj55 2023-07-29T00:40:05.447 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HRkAKEYK05tzOMDScB263QWZXyQ= 2023-07-29T00:40:05.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HRkAKEYK05tzOMDScB263QWZXyQ=', 'Content-Length': '0'} 2023-07-29T00:40:05.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000299b5a26860ab24-0064c45fe5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-07-29T00:40:05.450 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:40:05.451 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:40:05.451 DEBUG:boto:Method: GET 2023-07-29T00:40:05.452 DEBUG:boto:Path: /hiiqos-59/obj55 2023-07-29T00:40:05.452 DEBUG:boto:Data: 2023-07-29T00:40:05.452 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.453 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:05.453 DEBUG:boto:Port: 8001 2023-07-29T00:40:05.453 DEBUG:boto:Params: {} 2023-07-29T00:40:05.454 DEBUG:boto:Token: None 2023-07-29T00:40:05.454 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj55 2023-07-29T00:40:05.455 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9X+Ri5kSuzobJEAbguxGa9HnEEg= 2023-07-29T00:40:05.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9X+Ri5kSuzobJEAbguxGa9HnEEg=', 'Content-Length': '0'} 2023-07-29T00:40:05.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049b5b6f7cb28dc3e-0064c45fe5-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.500 DEBUG:boto:path=/hiiqos-59/obj55 2023-07-29T00:40:05.501 DEBUG:boto:auth_path=/hiiqos-59/obj55 2023-07-29T00:40:05.502 DEBUG:boto:Method: GET 2023-07-29T00:40:05.502 DEBUG:boto:Path: /hiiqos-59/obj55 2023-07-29T00:40:05.502 DEBUG:boto:Data: 2023-07-29T00:40:05.503 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.503 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:05.504 DEBUG:boto:Port: 8000 2023-07-29T00:40:05.504 DEBUG:boto:Params: {} 2023-07-29T00:40:05.505 DEBUG:boto:Token: None 2023-07-29T00:40:05.505 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj55 2023-07-29T00:40:05.506 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9X+Ri5kSuzobJEAbguxGa9HnEEg= 2023-07-29T00:40:05.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9X+Ri5kSuzobJEAbguxGa9HnEEg=', 'Content-Length': '0'} 2023-07-29T00:40:05.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:26:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a708920e2b3a1a91-0064c45fe5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.509 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:05.509 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:05.509 DEBUG:boto:path=/hiiqos-59/?prefix=obj55 2023-07-29T00:40:05.510 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj55 2023-07-29T00:40:05.510 DEBUG:boto:Method: GET 2023-07-29T00:40:05.510 DEBUG:boto:Path: /hiiqos-59/?prefix=obj55 2023-07-29T00:40:05.511 DEBUG:boto:Data: 2023-07-29T00:40:05.511 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.511 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:05.512 DEBUG:boto:Port: 8001 2023-07-29T00:40:05.512 DEBUG:boto:Params: {} 2023-07-29T00:40:05.513 DEBUG:boto:Token: None 2023-07-29T00:40:05.513 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/ 2023-07-29T00:40:05.514 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI= 2023-07-29T00:40:05.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI=', 'Content-Length': '0'} 2023-07-29T00:40:05.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000707f008e02008eae-0064c45fe5-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.517 DEBUG:boto:b'hiiqos-59obj551000falseobj552023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:05.518 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:05.519 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:05.519 DEBUG:boto:path=/hiiqos-59/?prefix=obj55 2023-07-29T00:40:05.519 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj55 2023-07-29T00:40:05.520 DEBUG:boto:Method: GET 2023-07-29T00:40:05.520 DEBUG:boto:Path: /hiiqos-59/?prefix=obj55 2023-07-29T00:40:05.520 DEBUG:boto:Data: 2023-07-29T00:40:05.521 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.521 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:05.522 DEBUG:boto:Port: 8000 2023-07-29T00:40:05.522 DEBUG:boto:Params: {} 2023-07-29T00:40:05.522 DEBUG:boto:Token: None 2023-07-29T00:40:05.523 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/ 2023-07-29T00:40:05.523 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI= 2023-07-29T00:40:05.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI=', 'Content-Length': '0'} 2023-07-29T00:40:05.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c4005b9ed713afc-0064c45fe5-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.528 DEBUG:boto:b'hiiqos-59obj551000falseobj552023-07-29T00:26:54.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:05.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-07-29T00:40:05.530 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:40:05.530 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:40:05.530 DEBUG:boto:path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:40:05.531 DEBUG:boto:auth_path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:40:05.531 DEBUG:boto:Method: GET 2023-07-29T00:40:05.531 DEBUG:boto:Path: /hiiqos-59/obj57?versionId=null 2023-07-29T00:40:05.532 DEBUG:boto:Data: 2023-07-29T00:40:05.532 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.533 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:05.533 DEBUG:boto:Port: 8001 2023-07-29T00:40:05.533 DEBUG:boto:Params: {} 2023-07-29T00:40:05.534 DEBUG:boto:Token: None 2023-07-29T00:40:05.534 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj57?versionId=null 2023-07-29T00:40:05.535 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:twZmXp+CmxuLD2XCF7VuH8aAZ8Q= 2023-07-29T00:40:05.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:twZmXp+CmxuLD2XCF7VuH8aAZ8Q=', 'Content-Length': '0'} 2023-07-29T00:40:05.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c71f36a4881a42d-0064c45fe5-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.538 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:40:05.538 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:40:05.539 DEBUG:boto:path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:40:05.539 DEBUG:boto:auth_path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:40:05.540 DEBUG:boto:Method: GET 2023-07-29T00:40:05.540 DEBUG:boto:Path: /hiiqos-59/obj57?versionId=null 2023-07-29T00:40:05.540 DEBUG:boto:Data: 2023-07-29T00:40:05.541 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.541 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:05.541 DEBUG:boto:Port: 8000 2023-07-29T00:40:05.542 DEBUG:boto:Params: {} 2023-07-29T00:40:05.542 DEBUG:boto:Token: None 2023-07-29T00:40:05.543 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj57?versionId=null 2023-07-29T00:40:05.543 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:twZmXp+CmxuLD2XCF7VuH8aAZ8Q= 2023-07-29T00:40:05.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:twZmXp+CmxuLD2XCF7VuH8aAZ8Q=', 'Content-Length': '0'} 2023-07-29T00:40:05.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ee449a9ffb1138a-0064c45fe5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.546 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:40:05.547 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:40:05.547 DEBUG:boto:path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:40:05.547 DEBUG:boto:auth_path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:40:05.548 DEBUG:boto:Method: HEAD 2023-07-29T00:40:05.548 DEBUG:boto:Path: /hiiqos-59/obj57?versionId=null 2023-07-29T00:40:05.549 DEBUG:boto:Data: 2023-07-29T00:40:05.549 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.549 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:05.550 DEBUG:boto:Port: 8001 2023-07-29T00:40:05.550 DEBUG:boto:Params: {} 2023-07-29T00:40:05.550 DEBUG:boto:Token: None 2023-07-29T00:40:05.551 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj57?versionId=null 2023-07-29T00:40:05.551 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mKAJjbO3REqE1c9CKBOzNpr+S60= 2023-07-29T00:40:05.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mKAJjbO3REqE1c9CKBOzNpr+S60=', 'Content-Length': '0'} 2023-07-29T00:40:05.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000970bf2bfc6aed1fa-0064c45fe5-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.554 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:40:05.554 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:40:05.555 DEBUG:boto:path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:40:05.555 DEBUG:boto:auth_path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:40:05.555 DEBUG:boto:Method: HEAD 2023-07-29T00:40:05.556 DEBUG:boto:Path: /hiiqos-59/obj57?versionId=null 2023-07-29T00:40:05.556 DEBUG:boto:Data: 2023-07-29T00:40:05.557 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.557 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:05.557 DEBUG:boto:Port: 8000 2023-07-29T00:40:05.558 DEBUG:boto:Params: {} 2023-07-29T00:40:05.558 DEBUG:boto:Token: None 2023-07-29T00:40:05.558 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj57?versionId=null 2023-07-29T00:40:05.559 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mKAJjbO3REqE1c9CKBOzNpr+S60= 2023-07-29T00:40:05.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mKAJjbO3REqE1c9CKBOzNpr+S60=', 'Content-Length': '0'} 2023-07-29T00:40:05.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bc7aa6bf3ed47d5-0064c45fe5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-07-29T00:40:05.562 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:40:05.562 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:40:05.563 DEBUG:boto:path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:40:05.563 DEBUG:boto:auth_path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:40:05.564 DEBUG:boto:Method: GET 2023-07-29T00:40:05.564 DEBUG:boto:Path: /hiiqos-59/obj57?versionId=null 2023-07-29T00:40:05.564 DEBUG:boto:Data: 2023-07-29T00:40:05.565 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.565 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:05.565 DEBUG:boto:Port: 8001 2023-07-29T00:40:05.566 DEBUG:boto:Params: {} 2023-07-29T00:40:05.566 DEBUG:boto:Token: None 2023-07-29T00:40:05.567 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj57?versionId=null 2023-07-29T00:40:05.567 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:twZmXp+CmxuLD2XCF7VuH8aAZ8Q= 2023-07-29T00:40:05.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:twZmXp+CmxuLD2XCF7VuH8aAZ8Q=', 'Content-Length': '0'} 2023-07-29T00:40:05.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000417cb305bda8d9db-0064c45fe5-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.612 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:40:05.613 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:40:05.613 DEBUG:boto:path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:40:05.614 DEBUG:boto:auth_path=/hiiqos-59/obj57?versionId=null 2023-07-29T00:40:05.614 DEBUG:boto:Method: GET 2023-07-29T00:40:05.614 DEBUG:boto:Path: /hiiqos-59/obj57?versionId=null 2023-07-29T00:40:05.615 DEBUG:boto:Data: 2023-07-29T00:40:05.615 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.616 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:05.616 DEBUG:boto:Port: 8000 2023-07-29T00:40:05.616 DEBUG:boto:Params: {} 2023-07-29T00:40:05.617 DEBUG:boto:Token: None 2023-07-29T00:40:05.617 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj57?versionId=null 2023-07-29T00:40:05.618 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:twZmXp+CmxuLD2XCF7VuH8aAZ8Q= 2023-07-29T00:40:05.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:twZmXp+CmxuLD2XCF7VuH8aAZ8Q=', 'Content-Length': '0'} 2023-07-29T00:40:05.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043eb16ec6eb0b07a-0064c45fe5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.621 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:05.621 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:05.622 DEBUG:boto:path=/hiiqos-59/?prefix=obj57 2023-07-29T00:40:05.622 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj57 2023-07-29T00:40:05.622 DEBUG:boto:Method: GET 2023-07-29T00:40:05.623 DEBUG:boto:Path: /hiiqos-59/?prefix=obj57 2023-07-29T00:40:05.623 DEBUG:boto:Data: 2023-07-29T00:40:05.624 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.624 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:05.624 DEBUG:boto:Port: 8001 2023-07-29T00:40:05.625 DEBUG:boto:Params: {} 2023-07-29T00:40:05.625 DEBUG:boto:Token: None 2023-07-29T00:40:05.625 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/ 2023-07-29T00:40:05.626 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI= 2023-07-29T00:40:05.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI=', 'Content-Length': '0'} 2023-07-29T00:40:05.629 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdf5415d095adfa1-0064c45fe5-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.629 DEBUG:boto:b'hiiqos-59obj571000falseobj572023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:05.630 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:05.631 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:05.631 DEBUG:boto:path=/hiiqos-59/?prefix=obj57 2023-07-29T00:40:05.631 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj57 2023-07-29T00:40:05.632 DEBUG:boto:Method: GET 2023-07-29T00:40:05.632 DEBUG:boto:Path: /hiiqos-59/?prefix=obj57 2023-07-29T00:40:05.632 DEBUG:boto:Data: 2023-07-29T00:40:05.633 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.633 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:05.634 DEBUG:boto:Port: 8000 2023-07-29T00:40:05.634 DEBUG:boto:Params: {} 2023-07-29T00:40:05.634 DEBUG:boto:Token: None 2023-07-29T00:40:05.635 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/ 2023-07-29T00:40:05.635 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI= 2023-07-29T00:40:05.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI=', 'Content-Length': '0'} 2023-07-29T00:40:05.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5de4f6c94eae6b9-0064c45fe5-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.640 DEBUG:boto:b'hiiqos-59obj571000falseobj572023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:05.641 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:40:05.641 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:40:05.642 DEBUG:boto:Method: HEAD 2023-07-29T00:40:05.642 DEBUG:boto:Path: /hiiqos-59/obj57 2023-07-29T00:40:05.642 DEBUG:boto:Data: 2023-07-29T00:40:05.643 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.643 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:05.643 DEBUG:boto:Port: 8001 2023-07-29T00:40:05.644 DEBUG:boto:Params: {} 2023-07-29T00:40:05.644 DEBUG:boto:Token: None 2023-07-29T00:40:05.645 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj57 2023-07-29T00:40:05.645 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JWEoHn082D+byv/hLFVCHGCLL7M= 2023-07-29T00:40:05.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JWEoHn082D+byv/hLFVCHGCLL7M=', 'Content-Length': '0'} 2023-07-29T00:40:05.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000065cd633d8728d8f-0064c45fe5-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.648 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:40:05.648 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:40:05.649 DEBUG:boto:Method: HEAD 2023-07-29T00:40:05.649 DEBUG:boto:Path: /hiiqos-59/obj57 2023-07-29T00:40:05.649 DEBUG:boto:Data: 2023-07-29T00:40:05.650 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.650 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:05.650 DEBUG:boto:Port: 8000 2023-07-29T00:40:05.651 DEBUG:boto:Params: {} 2023-07-29T00:40:05.651 DEBUG:boto:Token: None 2023-07-29T00:40:05.652 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj57 2023-07-29T00:40:05.652 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JWEoHn082D+byv/hLFVCHGCLL7M= 2023-07-29T00:40:05.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JWEoHn082D+byv/hLFVCHGCLL7M=', 'Content-Length': '0'} 2023-07-29T00:40:05.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c474059f286d56f5-0064c45fe5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-07-29T00:40:05.655 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:40:05.656 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:40:05.656 DEBUG:boto:Method: GET 2023-07-29T00:40:05.656 DEBUG:boto:Path: /hiiqos-59/obj57 2023-07-29T00:40:05.657 DEBUG:boto:Data: 2023-07-29T00:40:05.657 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.658 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:05.658 DEBUG:boto:Port: 8001 2023-07-29T00:40:05.658 DEBUG:boto:Params: {} 2023-07-29T00:40:05.659 DEBUG:boto:Token: None 2023-07-29T00:40:05.659 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj57 2023-07-29T00:40:05.660 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QcnI2xyEdXwwn/BPi5pZojN4Gc4= 2023-07-29T00:40:05.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QcnI2xyEdXwwn/BPi5pZojN4Gc4=', 'Content-Length': '0'} 2023-07-29T00:40:05.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ce4ca62cdb47f16-0064c45fe5-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.663 DEBUG:boto:path=/hiiqos-59/obj57 2023-07-29T00:40:05.663 DEBUG:boto:auth_path=/hiiqos-59/obj57 2023-07-29T00:40:05.663 DEBUG:boto:Method: GET 2023-07-29T00:40:05.664 DEBUG:boto:Path: /hiiqos-59/obj57 2023-07-29T00:40:05.664 DEBUG:boto:Data: 2023-07-29T00:40:05.665 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.665 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:05.665 DEBUG:boto:Port: 8000 2023-07-29T00:40:05.666 DEBUG:boto:Params: {} 2023-07-29T00:40:05.666 DEBUG:boto:Token: None 2023-07-29T00:40:05.667 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj57 2023-07-29T00:40:05.667 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QcnI2xyEdXwwn/BPi5pZojN4Gc4= 2023-07-29T00:40:05.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QcnI2xyEdXwwn/BPi5pZojN4Gc4=', 'Content-Length': '0'} 2023-07-29T00:40:05.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e52d315c5f5b786-0064c45fe5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.670 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:05.670 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:05.671 DEBUG:boto:path=/hiiqos-59/?prefix=obj57 2023-07-29T00:40:05.671 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj57 2023-07-29T00:40:05.672 DEBUG:boto:Method: GET 2023-07-29T00:40:05.672 DEBUG:boto:Path: /hiiqos-59/?prefix=obj57 2023-07-29T00:40:05.672 DEBUG:boto:Data: 2023-07-29T00:40:05.673 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.673 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:05.673 DEBUG:boto:Port: 8001 2023-07-29T00:40:05.674 DEBUG:boto:Params: {} 2023-07-29T00:40:05.674 DEBUG:boto:Token: None 2023-07-29T00:40:05.675 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/ 2023-07-29T00:40:05.675 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI= 2023-07-29T00:40:05.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI=', 'Content-Length': '0'} 2023-07-29T00:40:05.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdfbb04835f8de85-0064c45fe5-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.678 DEBUG:boto:b'hiiqos-59obj571000falseobj572023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:05.679 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:05.680 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:05.680 DEBUG:boto:path=/hiiqos-59/?prefix=obj57 2023-07-29T00:40:05.680 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj57 2023-07-29T00:40:05.681 DEBUG:boto:Method: GET 2023-07-29T00:40:05.681 DEBUG:boto:Path: /hiiqos-59/?prefix=obj57 2023-07-29T00:40:05.681 DEBUG:boto:Data: 2023-07-29T00:40:05.682 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.682 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:05.682 DEBUG:boto:Port: 8000 2023-07-29T00:40:05.683 DEBUG:boto:Params: {} 2023-07-29T00:40:05.683 DEBUG:boto:Token: None 2023-07-29T00:40:05.684 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/ 2023-07-29T00:40:05.684 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI= 2023-07-29T00:40:05.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI=', 'Content-Length': '0'} 2023-07-29T00:40:05.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098951a6c67a190ea-0064c45fe5-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.690 DEBUG:boto:b'hiiqos-59obj571000falseobj572023-07-29T00:29:58.720Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:05.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-07-29T00:40:05.691 DEBUG:boto:path=/hiiqos-59/obj58 2023-07-29T00:40:05.691 DEBUG:boto:auth_path=/hiiqos-59/obj58 2023-07-29T00:40:05.692 DEBUG:boto:path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:40:05.692 DEBUG:boto:auth_path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:40:05.692 DEBUG:boto:Method: GET 2023-07-29T00:40:05.693 DEBUG:boto:Path: /hiiqos-59/obj58?versionId=null 2023-07-29T00:40:05.693 DEBUG:boto:Data: 2023-07-29T00:40:05.693 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.694 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:05.694 DEBUG:boto:Port: 8001 2023-07-29T00:40:05.695 DEBUG:boto:Params: {} 2023-07-29T00:40:05.695 DEBUG:boto:Token: None 2023-07-29T00:40:05.695 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj58?versionId=null 2023-07-29T00:40:05.696 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yClI6x2iCI9LniFGLEzbVSgIg7Y= 2023-07-29T00:40:05.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yClI6x2iCI9LniFGLEzbVSgIg7Y=', 'Content-Length': '0'} 2023-07-29T00:40:05.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f48a1e455d76d59-0064c45fe5-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.700 DEBUG:boto:path=/hiiqos-59/obj58 2023-07-29T00:40:05.700 DEBUG:boto:auth_path=/hiiqos-59/obj58 2023-07-29T00:40:05.700 DEBUG:boto:path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:40:05.701 DEBUG:boto:auth_path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:40:05.701 DEBUG:boto:Method: GET 2023-07-29T00:40:05.702 DEBUG:boto:Path: /hiiqos-59/obj58?versionId=null 2023-07-29T00:40:05.702 DEBUG:boto:Data: 2023-07-29T00:40:05.703 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.703 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:05.703 DEBUG:boto:Port: 8000 2023-07-29T00:40:05.704 DEBUG:boto:Params: {} 2023-07-29T00:40:05.704 DEBUG:boto:Token: None 2023-07-29T00:40:05.705 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj58?versionId=null 2023-07-29T00:40:05.705 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yClI6x2iCI9LniFGLEzbVSgIg7Y= 2023-07-29T00:40:05.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yClI6x2iCI9LniFGLEzbVSgIg7Y=', 'Content-Length': '0'} 2023-07-29T00:40:05.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c334addc62557e0-0064c45fe5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.708 DEBUG:boto:path=/hiiqos-59/obj58 2023-07-29T00:40:05.708 DEBUG:boto:auth_path=/hiiqos-59/obj58 2023-07-29T00:40:05.709 DEBUG:boto:path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:40:05.709 DEBUG:boto:auth_path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:40:05.709 DEBUG:boto:Method: HEAD 2023-07-29T00:40:05.710 DEBUG:boto:Path: /hiiqos-59/obj58?versionId=null 2023-07-29T00:40:05.710 DEBUG:boto:Data: 2023-07-29T00:40:05.710 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.711 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:05.711 DEBUG:boto:Port: 8001 2023-07-29T00:40:05.711 DEBUG:boto:Params: {} 2023-07-29T00:40:05.712 DEBUG:boto:Token: None 2023-07-29T00:40:05.712 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj58?versionId=null 2023-07-29T00:40:05.713 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dNKtvYks+B8kd8aJxO7setKrh9g= 2023-07-29T00:40:05.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dNKtvYks+B8kd8aJxO7setKrh9g=', 'Content-Length': '0'} 2023-07-29T00:40:05.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000413159f9194fb671-0064c45fe5-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.715 DEBUG:boto:path=/hiiqos-59/obj58 2023-07-29T00:40:05.716 DEBUG:boto:auth_path=/hiiqos-59/obj58 2023-07-29T00:40:05.716 DEBUG:boto:path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:40:05.717 DEBUG:boto:auth_path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:40:05.717 DEBUG:boto:Method: HEAD 2023-07-29T00:40:05.717 DEBUG:boto:Path: /hiiqos-59/obj58?versionId=null 2023-07-29T00:40:05.718 DEBUG:boto:Data: 2023-07-29T00:40:05.718 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.718 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:05.719 DEBUG:boto:Port: 8000 2023-07-29T00:40:05.719 DEBUG:boto:Params: {} 2023-07-29T00:40:05.719 DEBUG:boto:Token: None 2023-07-29T00:40:05.720 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj58?versionId=null 2023-07-29T00:40:05.720 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dNKtvYks+B8kd8aJxO7setKrh9g= 2023-07-29T00:40:05.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dNKtvYks+B8kd8aJxO7setKrh9g=', 'Content-Length': '0'} 2023-07-29T00:40:05.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052c1689fe47379bb-0064c45fe5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-07-29T00:40:05.724 DEBUG:boto:path=/hiiqos-59/obj58 2023-07-29T00:40:05.724 DEBUG:boto:auth_path=/hiiqos-59/obj58 2023-07-29T00:40:05.724 DEBUG:boto:path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:40:05.725 DEBUG:boto:auth_path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:40:05.725 DEBUG:boto:Method: GET 2023-07-29T00:40:05.725 DEBUG:boto:Path: /hiiqos-59/obj58?versionId=null 2023-07-29T00:40:05.726 DEBUG:boto:Data: 2023-07-29T00:40:05.726 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.727 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:05.727 DEBUG:boto:Port: 8001 2023-07-29T00:40:05.727 DEBUG:boto:Params: {} 2023-07-29T00:40:05.728 DEBUG:boto:Token: None 2023-07-29T00:40:05.728 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj58?versionId=null 2023-07-29T00:40:05.729 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yClI6x2iCI9LniFGLEzbVSgIg7Y= 2023-07-29T00:40:05.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yClI6x2iCI9LniFGLEzbVSgIg7Y=', 'Content-Length': '0'} 2023-07-29T00:40:05.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ca0f1ce0243a4be-0064c45fe5-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.731 DEBUG:boto:path=/hiiqos-59/obj58 2023-07-29T00:40:05.732 DEBUG:boto:auth_path=/hiiqos-59/obj58 2023-07-29T00:40:05.732 DEBUG:boto:path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:40:05.733 DEBUG:boto:auth_path=/hiiqos-59/obj58?versionId=null 2023-07-29T00:40:05.733 DEBUG:boto:Method: GET 2023-07-29T00:40:05.733 DEBUG:boto:Path: /hiiqos-59/obj58?versionId=null 2023-07-29T00:40:05.734 DEBUG:boto:Data: 2023-07-29T00:40:05.734 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.734 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:05.735 DEBUG:boto:Port: 8000 2023-07-29T00:40:05.735 DEBUG:boto:Params: {} 2023-07-29T00:40:05.736 DEBUG:boto:Token: None 2023-07-29T00:40:05.736 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj58?versionId=null 2023-07-29T00:40:05.737 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yClI6x2iCI9LniFGLEzbVSgIg7Y= 2023-07-29T00:40:05.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yClI6x2iCI9LniFGLEzbVSgIg7Y=', 'Content-Length': '0'} 2023-07-29T00:40:05.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ffc292ec7f85da3-0064c45fe5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.739 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:05.740 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:05.740 DEBUG:boto:path=/hiiqos-59/?prefix=obj58 2023-07-29T00:40:05.741 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj58 2023-07-29T00:40:05.741 DEBUG:boto:Method: GET 2023-07-29T00:40:05.741 DEBUG:boto:Path: /hiiqos-59/?prefix=obj58 2023-07-29T00:40:05.742 DEBUG:boto:Data: 2023-07-29T00:40:05.742 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.742 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:05.743 DEBUG:boto:Port: 8001 2023-07-29T00:40:05.743 DEBUG:boto:Params: {} 2023-07-29T00:40:05.744 DEBUG:boto:Token: None 2023-07-29T00:40:05.744 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/ 2023-07-29T00:40:05.744 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI= 2023-07-29T00:40:05.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI=', 'Content-Length': '0'} 2023-07-29T00:40:05.747 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdd7569491911c95-0064c45fe5-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.748 DEBUG:boto:b'hiiqos-59obj581000falseobj582023-07-29T00:35:01.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:05.749 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:05.749 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:05.749 DEBUG:boto:path=/hiiqos-59/?prefix=obj58 2023-07-29T00:40:05.750 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj58 2023-07-29T00:40:05.750 DEBUG:boto:Method: GET 2023-07-29T00:40:05.750 DEBUG:boto:Path: /hiiqos-59/?prefix=obj58 2023-07-29T00:40:05.751 DEBUG:boto:Data: 2023-07-29T00:40:05.751 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.752 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:05.752 DEBUG:boto:Port: 8000 2023-07-29T00:40:05.752 DEBUG:boto:Params: {} 2023-07-29T00:40:05.753 DEBUG:boto:Token: None 2023-07-29T00:40:05.753 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/ 2023-07-29T00:40:05.754 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI= 2023-07-29T00:40:05.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI=', 'Content-Length': '0'} 2023-07-29T00:40:05.758 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012d73eda17620dfd-0064c45fe5-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.759 DEBUG:boto:b'hiiqos-59obj581000falseobj582023-07-29T00:35:01.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:05.759 DEBUG:boto:path=/hiiqos-59/obj58 2023-07-29T00:40:05.760 DEBUG:boto:auth_path=/hiiqos-59/obj58 2023-07-29T00:40:05.760 DEBUG:boto:Method: HEAD 2023-07-29T00:40:05.761 DEBUG:boto:Path: /hiiqos-59/obj58 2023-07-29T00:40:05.761 DEBUG:boto:Data: 2023-07-29T00:40:05.761 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.762 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:05.762 DEBUG:boto:Port: 8001 2023-07-29T00:40:05.762 DEBUG:boto:Params: {} 2023-07-29T00:40:05.763 DEBUG:boto:Token: None 2023-07-29T00:40:05.763 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj58 2023-07-29T00:40:05.763 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J3zYgDdWe+AUPxZsTquQ8d9XFwE= 2023-07-29T00:40:05.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J3zYgDdWe+AUPxZsTquQ8d9XFwE=', 'Content-Length': '0'} 2023-07-29T00:40:05.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae0080e70e1e3771-0064c45fe5-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.766 DEBUG:boto:path=/hiiqos-59/obj58 2023-07-29T00:40:05.767 DEBUG:boto:auth_path=/hiiqos-59/obj58 2023-07-29T00:40:05.767 DEBUG:boto:Method: HEAD 2023-07-29T00:40:05.767 DEBUG:boto:Path: /hiiqos-59/obj58 2023-07-29T00:40:05.768 DEBUG:boto:Data: 2023-07-29T00:40:05.768 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.768 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:05.769 DEBUG:boto:Port: 8000 2023-07-29T00:40:05.769 DEBUG:boto:Params: {} 2023-07-29T00:40:05.770 DEBUG:boto:Token: None 2023-07-29T00:40:05.770 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj58 2023-07-29T00:40:05.770 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J3zYgDdWe+AUPxZsTquQ8d9XFwE= 2023-07-29T00:40:05.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J3zYgDdWe+AUPxZsTquQ8d9XFwE=', 'Content-Length': '0'} 2023-07-29T00:40:05.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001234512e08aa559e-0064c45fe5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-07-29T00:40:05.773 DEBUG:boto:path=/hiiqos-59/obj58 2023-07-29T00:40:05.774 DEBUG:boto:auth_path=/hiiqos-59/obj58 2023-07-29T00:40:05.774 DEBUG:boto:Method: GET 2023-07-29T00:40:05.774 DEBUG:boto:Path: /hiiqos-59/obj58 2023-07-29T00:40:05.775 DEBUG:boto:Data: 2023-07-29T00:40:05.775 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.776 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:05.776 DEBUG:boto:Port: 8001 2023-07-29T00:40:05.776 DEBUG:boto:Params: {} 2023-07-29T00:40:05.777 DEBUG:boto:Token: None 2023-07-29T00:40:05.777 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj58 2023-07-29T00:40:05.778 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MnnimZTxD06SbuDfqs0GP0rfNes= 2023-07-29T00:40:05.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MnnimZTxD06SbuDfqs0GP0rfNes=', 'Content-Length': '0'} 2023-07-29T00:40:05.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e637bfcd9347582e-0064c45fe5-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.780 DEBUG:boto:path=/hiiqos-59/obj58 2023-07-29T00:40:05.781 DEBUG:boto:auth_path=/hiiqos-59/obj58 2023-07-29T00:40:05.781 DEBUG:boto:Method: GET 2023-07-29T00:40:05.782 DEBUG:boto:Path: /hiiqos-59/obj58 2023-07-29T00:40:05.782 DEBUG:boto:Data: 2023-07-29T00:40:05.782 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.783 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:05.783 DEBUG:boto:Port: 8000 2023-07-29T00:40:05.783 DEBUG:boto:Params: {} 2023-07-29T00:40:05.784 DEBUG:boto:Token: None 2023-07-29T00:40:05.784 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj58 2023-07-29T00:40:05.785 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MnnimZTxD06SbuDfqs0GP0rfNes= 2023-07-29T00:40:05.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MnnimZTxD06SbuDfqs0GP0rfNes=', 'Content-Length': '0'} 2023-07-29T00:40:05.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003071fb7454a0022a-0064c45fe5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.832 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:05.833 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:05.833 DEBUG:boto:path=/hiiqos-59/?prefix=obj58 2023-07-29T00:40:05.834 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj58 2023-07-29T00:40:05.834 DEBUG:boto:Method: GET 2023-07-29T00:40:05.834 DEBUG:boto:Path: /hiiqos-59/?prefix=obj58 2023-07-29T00:40:05.835 DEBUG:boto:Data: 2023-07-29T00:40:05.835 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.835 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:05.836 DEBUG:boto:Port: 8001 2023-07-29T00:40:05.836 DEBUG:boto:Params: {} 2023-07-29T00:40:05.837 DEBUG:boto:Token: None 2023-07-29T00:40:05.837 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/ 2023-07-29T00:40:05.837 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI= 2023-07-29T00:40:05.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI=', 'Content-Length': '0'} 2023-07-29T00:40:05.840 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004439d705d1c144aa-0064c45fe5-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.841 DEBUG:boto:b'hiiqos-59obj581000falseobj582023-07-29T00:35:01.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:05.842 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:05.842 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:05.842 DEBUG:boto:path=/hiiqos-59/?prefix=obj58 2023-07-29T00:40:05.843 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj58 2023-07-29T00:40:05.843 DEBUG:boto:Method: GET 2023-07-29T00:40:05.844 DEBUG:boto:Path: /hiiqos-59/?prefix=obj58 2023-07-29T00:40:05.844 DEBUG:boto:Data: 2023-07-29T00:40:05.844 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.845 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:05.845 DEBUG:boto:Port: 8000 2023-07-29T00:40:05.846 DEBUG:boto:Params: {} 2023-07-29T00:40:05.846 DEBUG:boto:Token: None 2023-07-29T00:40:05.846 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/ 2023-07-29T00:40:05.847 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI= 2023-07-29T00:40:05.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI=', 'Content-Length': '0'} 2023-07-29T00:40:05.852 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d81879defd7bfa5-0064c45fe5-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.852 DEBUG:boto:b'hiiqos-59obj581000falseobj582023-07-29T00:35:01.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:05.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-07-29T00:40:05.854 DEBUG:boto:path=/hiiqos-59/obj62 2023-07-29T00:40:05.855 DEBUG:boto:auth_path=/hiiqos-59/obj62 2023-07-29T00:40:05.856 DEBUG:boto:path=/hiiqos-59/obj62?versionId=null 2023-07-29T00:40:05.857 DEBUG:boto:auth_path=/hiiqos-59/obj62?versionId=null 2023-07-29T00:40:05.857 DEBUG:boto:Method: GET 2023-07-29T00:40:05.858 DEBUG:boto:Path: /hiiqos-59/obj62?versionId=null 2023-07-29T00:40:05.859 DEBUG:boto:Data: 2023-07-29T00:40:05.859 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.860 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:05.861 DEBUG:boto:Port: 8001 2023-07-29T00:40:05.862 DEBUG:boto:Params: {} 2023-07-29T00:40:05.862 DEBUG:boto:Token: None 2023-07-29T00:40:05.863 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj62?versionId=null 2023-07-29T00:40:05.864 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:W3sRZAU72CFNYs30nUNUWuYWQ7k= 2023-07-29T00:40:05.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:W3sRZAU72CFNYs30nUNUWuYWQ7k=', 'Content-Length': '0'} 2023-07-29T00:40:05.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc1dc3e8dbde4e9f-0064c45fe5-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.912 DEBUG:boto:path=/hiiqos-59/obj62 2023-07-29T00:40:05.913 DEBUG:boto:auth_path=/hiiqos-59/obj62 2023-07-29T00:40:05.913 DEBUG:boto:path=/hiiqos-59/obj62?versionId=null 2023-07-29T00:40:05.914 DEBUG:boto:auth_path=/hiiqos-59/obj62?versionId=null 2023-07-29T00:40:05.914 DEBUG:boto:Method: GET 2023-07-29T00:40:05.914 DEBUG:boto:Path: /hiiqos-59/obj62?versionId=null 2023-07-29T00:40:05.915 DEBUG:boto:Data: 2023-07-29T00:40:05.915 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.915 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:05.916 DEBUG:boto:Port: 8000 2023-07-29T00:40:05.916 DEBUG:boto:Params: {} 2023-07-29T00:40:05.917 DEBUG:boto:Token: None 2023-07-29T00:40:05.917 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj62?versionId=null 2023-07-29T00:40:05.918 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:W3sRZAU72CFNYs30nUNUWuYWQ7k= 2023-07-29T00:40:05.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:W3sRZAU72CFNYs30nUNUWuYWQ7k=', 'Content-Length': '0'} 2023-07-29T00:40:05.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002962d9fa8c1edfa8-0064c45fe5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.921 DEBUG:boto:path=/hiiqos-59/obj62 2023-07-29T00:40:05.921 DEBUG:boto:auth_path=/hiiqos-59/obj62 2023-07-29T00:40:05.922 DEBUG:boto:path=/hiiqos-59/obj62?versionId=null 2023-07-29T00:40:05.922 DEBUG:boto:auth_path=/hiiqos-59/obj62?versionId=null 2023-07-29T00:40:05.922 DEBUG:boto:Method: HEAD 2023-07-29T00:40:05.923 DEBUG:boto:Path: /hiiqos-59/obj62?versionId=null 2023-07-29T00:40:05.923 DEBUG:boto:Data: 2023-07-29T00:40:05.924 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.924 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:05.925 DEBUG:boto:Port: 8001 2023-07-29T00:40:05.925 DEBUG:boto:Params: {} 2023-07-29T00:40:05.926 DEBUG:boto:Token: None 2023-07-29T00:40:05.927 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj62?versionId=null 2023-07-29T00:40:05.927 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:saAWUpRdFl3kY4ToxTdXXKrzcfE= 2023-07-29T00:40:05.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:saAWUpRdFl3kY4ToxTdXXKrzcfE=', 'Content-Length': '0'} 2023-07-29T00:40:05.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9363adfc9d17683-0064c45fe5-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.931 DEBUG:boto:path=/hiiqos-59/obj62 2023-07-29T00:40:05.931 DEBUG:boto:auth_path=/hiiqos-59/obj62 2023-07-29T00:40:05.931 DEBUG:boto:path=/hiiqos-59/obj62?versionId=null 2023-07-29T00:40:05.932 DEBUG:boto:auth_path=/hiiqos-59/obj62?versionId=null 2023-07-29T00:40:05.932 DEBUG:boto:Method: HEAD 2023-07-29T00:40:05.932 DEBUG:boto:Path: /hiiqos-59/obj62?versionId=null 2023-07-29T00:40:05.933 DEBUG:boto:Data: 2023-07-29T00:40:05.933 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.934 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:05.934 DEBUG:boto:Port: 8000 2023-07-29T00:40:05.934 DEBUG:boto:Params: {} 2023-07-29T00:40:05.935 DEBUG:boto:Token: None 2023-07-29T00:40:05.935 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj62?versionId=null 2023-07-29T00:40:05.935 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:saAWUpRdFl3kY4ToxTdXXKrzcfE= 2023-07-29T00:40:05.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:saAWUpRdFl3kY4ToxTdXXKrzcfE=', 'Content-Length': '0'} 2023-07-29T00:40:05.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087eb8fca54ae4559-0064c45fe5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-07-29T00:40:05.939 DEBUG:boto:path=/hiiqos-59/obj62 2023-07-29T00:40:05.939 DEBUG:boto:auth_path=/hiiqos-59/obj62 2023-07-29T00:40:05.940 DEBUG:boto:path=/hiiqos-59/obj62?versionId=null 2023-07-29T00:40:05.940 DEBUG:boto:auth_path=/hiiqos-59/obj62?versionId=null 2023-07-29T00:40:05.940 DEBUG:boto:Method: GET 2023-07-29T00:40:05.941 DEBUG:boto:Path: /hiiqos-59/obj62?versionId=null 2023-07-29T00:40:05.941 DEBUG:boto:Data: 2023-07-29T00:40:05.941 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.942 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:05.942 DEBUG:boto:Port: 8001 2023-07-29T00:40:05.943 DEBUG:boto:Params: {} 2023-07-29T00:40:05.943 DEBUG:boto:Token: None 2023-07-29T00:40:05.943 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj62?versionId=null 2023-07-29T00:40:05.944 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:W3sRZAU72CFNYs30nUNUWuYWQ7k= 2023-07-29T00:40:05.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:W3sRZAU72CFNYs30nUNUWuYWQ7k=', 'Content-Length': '0'} 2023-07-29T00:40:05.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5bd7dab00179c09-0064c45fe5-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.947 DEBUG:boto:path=/hiiqos-59/obj62 2023-07-29T00:40:05.947 DEBUG:boto:auth_path=/hiiqos-59/obj62 2023-07-29T00:40:05.948 DEBUG:boto:path=/hiiqos-59/obj62?versionId=null 2023-07-29T00:40:05.948 DEBUG:boto:auth_path=/hiiqos-59/obj62?versionId=null 2023-07-29T00:40:05.949 DEBUG:boto:Method: GET 2023-07-29T00:40:05.949 DEBUG:boto:Path: /hiiqos-59/obj62?versionId=null 2023-07-29T00:40:05.949 DEBUG:boto:Data: 2023-07-29T00:40:05.950 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.950 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:05.950 DEBUG:boto:Port: 8000 2023-07-29T00:40:05.951 DEBUG:boto:Params: {} 2023-07-29T00:40:05.951 DEBUG:boto:Token: None 2023-07-29T00:40:05.951 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj62?versionId=null 2023-07-29T00:40:05.952 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:W3sRZAU72CFNYs30nUNUWuYWQ7k= 2023-07-29T00:40:05.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:W3sRZAU72CFNYs30nUNUWuYWQ7k=', 'Content-Length': '0'} 2023-07-29T00:40:05.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3035b0ef693c425-0064c45fe5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.955 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:05.955 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:05.956 DEBUG:boto:path=/hiiqos-59/?prefix=obj62 2023-07-29T00:40:05.956 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj62 2023-07-29T00:40:05.957 DEBUG:boto:Method: GET 2023-07-29T00:40:05.957 DEBUG:boto:Path: /hiiqos-59/?prefix=obj62 2023-07-29T00:40:05.957 DEBUG:boto:Data: 2023-07-29T00:40:05.958 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.958 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:05.958 DEBUG:boto:Port: 8001 2023-07-29T00:40:05.959 DEBUG:boto:Params: {} 2023-07-29T00:40:05.959 DEBUG:boto:Token: None 2023-07-29T00:40:05.960 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/ 2023-07-29T00:40:05.960 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI= 2023-07-29T00:40:05.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI=', 'Content-Length': '0'} 2023-07-29T00:40:05.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084820c5dd9edbefc-0064c45fe5-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.964 DEBUG:boto:b'hiiqos-59obj621000falseobj622023-07-29T00:35:31.826Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:05.965 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:05.965 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:05.966 DEBUG:boto:path=/hiiqos-59/?prefix=obj62 2023-07-29T00:40:05.966 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj62 2023-07-29T00:40:05.966 DEBUG:boto:Method: GET 2023-07-29T00:40:05.967 DEBUG:boto:Path: /hiiqos-59/?prefix=obj62 2023-07-29T00:40:05.967 DEBUG:boto:Data: 2023-07-29T00:40:05.967 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.968 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:05.968 DEBUG:boto:Port: 8000 2023-07-29T00:40:05.968 DEBUG:boto:Params: {} 2023-07-29T00:40:05.969 DEBUG:boto:Token: None 2023-07-29T00:40:05.969 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/ 2023-07-29T00:40:05.970 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI= 2023-07-29T00:40:05.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kPbHUGK8htVkEYWXgtZkBWfuMVI=', 'Content-Length': '0'} 2023-07-29T00:40:05.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fecca87c616440fb-0064c45fe5-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.975 DEBUG:boto:b'hiiqos-59obj621000falseobj622023-07-29T00:35:31.826Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:05.976 DEBUG:boto:path=/hiiqos-59/obj62 2023-07-29T00:40:05.976 DEBUG:boto:auth_path=/hiiqos-59/obj62 2023-07-29T00:40:05.977 DEBUG:boto:Method: HEAD 2023-07-29T00:40:05.977 DEBUG:boto:Path: /hiiqos-59/obj62 2023-07-29T00:40:05.977 DEBUG:boto:Data: 2023-07-29T00:40:05.978 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.978 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:05.978 DEBUG:boto:Port: 8001 2023-07-29T00:40:05.979 DEBUG:boto:Params: {} 2023-07-29T00:40:05.979 DEBUG:boto:Token: None 2023-07-29T00:40:05.979 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj62 2023-07-29T00:40:05.980 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:51Fd/qZIUj5fTGKcxmWlL8e6BeY= 2023-07-29T00:40:05.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:51Fd/qZIUj5fTGKcxmWlL8e6BeY=', 'Content-Length': '0'} 2023-07-29T00:40:05.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ffc1a50c05770df-0064c45fe5-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.983 DEBUG:boto:path=/hiiqos-59/obj62 2023-07-29T00:40:05.983 DEBUG:boto:auth_path=/hiiqos-59/obj62 2023-07-29T00:40:05.983 DEBUG:boto:Method: HEAD 2023-07-29T00:40:05.984 DEBUG:boto:Path: /hiiqos-59/obj62 2023-07-29T00:40:05.984 DEBUG:boto:Data: 2023-07-29T00:40:05.984 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.985 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:05.985 DEBUG:boto:Port: 8000 2023-07-29T00:40:05.985 DEBUG:boto:Params: {} 2023-07-29T00:40:05.986 DEBUG:boto:Token: None 2023-07-29T00:40:05.986 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj62 2023-07-29T00:40:05.987 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:51Fd/qZIUj5fTGKcxmWlL8e6BeY= 2023-07-29T00:40:05.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:51Fd/qZIUj5fTGKcxmWlL8e6BeY=', 'Content-Length': '0'} 2023-07-29T00:40:05.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006db7d0c0c6f691ed-0064c45fe5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-07-29T00:40:05.990 DEBUG:boto:path=/hiiqos-59/obj62 2023-07-29T00:40:05.990 DEBUG:boto:auth_path=/hiiqos-59/obj62 2023-07-29T00:40:05.991 DEBUG:boto:Method: GET 2023-07-29T00:40:05.991 DEBUG:boto:Path: /hiiqos-59/obj62 2023-07-29T00:40:05.992 DEBUG:boto:Data: 2023-07-29T00:40:05.992 DEBUG:boto:Headers: {} 2023-07-29T00:40:05.992 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:05.993 DEBUG:boto:Port: 8001 2023-07-29T00:40:05.993 DEBUG:boto:Params: {} 2023-07-29T00:40:05.993 DEBUG:boto:Token: None 2023-07-29T00:40:05.994 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:05 GMT /hiiqos-59/obj62 2023-07-29T00:40:05.994 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uNSmhwmxny1MG2vXGI16gJ+A+tk= 2023-07-29T00:40:05.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uNSmhwmxny1MG2vXGI16gJ+A+tk=', 'Content-Length': '0'} 2023-07-29T00:40:05.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b98069cf55468ff2-0064c45fe5-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:05.997 DEBUG:boto:path=/hiiqos-59/obj62 2023-07-29T00:40:05.998 DEBUG:boto:auth_path=/hiiqos-59/obj62 2023-07-29T00:40:05.998 DEBUG:boto:Method: GET 2023-07-29T00:40:05.999 DEBUG:boto:Path: /hiiqos-59/obj62 2023-07-29T00:40:05.999 DEBUG:boto:Data: 2023-07-29T00:40:05.999 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.000 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:06.000 DEBUG:boto:Port: 8000 2023-07-29T00:40:06.000 DEBUG:boto:Params: {} 2023-07-29T00:40:06.001 DEBUG:boto:Token: None 2023-07-29T00:40:06.001 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj62 2023-07-29T00:40:06.002 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zuy6v3+7LhuQap3sTGD+IYp1Mvs= 2023-07-29T00:40:06.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zuy6v3+7LhuQap3sTGD+IYp1Mvs=', 'Content-Length': '0'} 2023-07-29T00:40:06.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f8c02e663f57e32-0064c45fe5-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.044 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:06.045 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:06.045 DEBUG:boto:path=/hiiqos-59/?prefix=obj62 2023-07-29T00:40:06.046 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj62 2023-07-29T00:40:06.046 DEBUG:boto:Method: GET 2023-07-29T00:40:06.046 DEBUG:boto:Path: /hiiqos-59/?prefix=obj62 2023-07-29T00:40:06.047 DEBUG:boto:Data: 2023-07-29T00:40:06.047 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.047 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:06.048 DEBUG:boto:Port: 8001 2023-07-29T00:40:06.048 DEBUG:boto:Params: {} 2023-07-29T00:40:06.049 DEBUG:boto:Token: None 2023-07-29T00:40:06.049 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/ 2023-07-29T00:40:06.049 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sN3umzbYPsTy+jGzbN8Ukc5/qnE= 2023-07-29T00:40:06.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sN3umzbYPsTy+jGzbN8Ukc5/qnE=', 'Content-Length': '0'} 2023-07-29T00:40:06.052 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088932d86cad95402-0064c45fe5-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.053 DEBUG:boto:b'hiiqos-59obj621000falseobj622023-07-29T00:35:31.826Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:06.053 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:06.054 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:06.054 DEBUG:boto:path=/hiiqos-59/?prefix=obj62 2023-07-29T00:40:06.055 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj62 2023-07-29T00:40:06.055 DEBUG:boto:Method: GET 2023-07-29T00:40:06.055 DEBUG:boto:Path: /hiiqos-59/?prefix=obj62 2023-07-29T00:40:06.056 DEBUG:boto:Data: 2023-07-29T00:40:06.056 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.056 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:06.057 DEBUG:boto:Port: 8000 2023-07-29T00:40:06.057 DEBUG:boto:Params: {} 2023-07-29T00:40:06.057 DEBUG:boto:Token: None 2023-07-29T00:40:06.058 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/ 2023-07-29T00:40:06.058 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sN3umzbYPsTy+jGzbN8Ukc5/qnE= 2023-07-29T00:40:06.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sN3umzbYPsTy+jGzbN8Ukc5/qnE=', 'Content-Length': '0'} 2023-07-29T00:40:06.063 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000361dac88d3deb568-0064c45fe5-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.064 DEBUG:boto:b'hiiqos-59obj621000falseobj622023-07-29T00:35:31.826Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:06.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-07-29T00:40:06.065 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:40:06.065 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:40:06.065 DEBUG:boto:path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:40:06.066 DEBUG:boto:auth_path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:40:06.066 DEBUG:boto:Method: GET 2023-07-29T00:40:06.066 DEBUG:boto:Path: /hiiqos-59/obj68?versionId=null 2023-07-29T00:40:06.067 DEBUG:boto:Data: 2023-07-29T00:40:06.067 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.067 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:06.068 DEBUG:boto:Port: 8001 2023-07-29T00:40:06.068 DEBUG:boto:Params: {} 2023-07-29T00:40:06.069 DEBUG:boto:Token: None 2023-07-29T00:40:06.069 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj68?versionId=null 2023-07-29T00:40:06.069 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O8wTLZ8UBhqhl7/KQIjlKpwC46E= 2023-07-29T00:40:06.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O8wTLZ8UBhqhl7/KQIjlKpwC46E=', 'Content-Length': '0'} 2023-07-29T00:40:06.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050455ad8318a235e-0064c45fe6-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.073 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:40:06.073 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:40:06.073 DEBUG:boto:path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:40:06.074 DEBUG:boto:auth_path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:40:06.074 DEBUG:boto:Method: GET 2023-07-29T00:40:06.075 DEBUG:boto:Path: /hiiqos-59/obj68?versionId=null 2023-07-29T00:40:06.075 DEBUG:boto:Data: 2023-07-29T00:40:06.075 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.076 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:06.076 DEBUG:boto:Port: 8000 2023-07-29T00:40:06.076 DEBUG:boto:Params: {} 2023-07-29T00:40:06.077 DEBUG:boto:Token: None 2023-07-29T00:40:06.077 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj68?versionId=null 2023-07-29T00:40:06.078 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O8wTLZ8UBhqhl7/KQIjlKpwC46E= 2023-07-29T00:40:06.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O8wTLZ8UBhqhl7/KQIjlKpwC46E=', 'Content-Length': '0'} 2023-07-29T00:40:06.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9a2fde1110a51a4-0064c45fe6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.080 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:40:06.081 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:40:06.081 DEBUG:boto:path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:40:06.081 DEBUG:boto:auth_path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:40:06.082 DEBUG:boto:Method: HEAD 2023-07-29T00:40:06.082 DEBUG:boto:Path: /hiiqos-59/obj68?versionId=null 2023-07-29T00:40:06.083 DEBUG:boto:Data: 2023-07-29T00:40:06.083 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.083 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:06.084 DEBUG:boto:Port: 8001 2023-07-29T00:40:06.084 DEBUG:boto:Params: {} 2023-07-29T00:40:06.084 DEBUG:boto:Token: None 2023-07-29T00:40:06.085 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj68?versionId=null 2023-07-29T00:40:06.085 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xl83duBGUJgyzEJWVHQopACPf4k= 2023-07-29T00:40:06.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xl83duBGUJgyzEJWVHQopACPf4k=', 'Content-Length': '0'} 2023-07-29T00:40:06.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052c752ccecfe31c3-0064c45fe6-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.088 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:40:06.088 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:40:06.089 DEBUG:boto:path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:40:06.089 DEBUG:boto:auth_path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:40:06.089 DEBUG:boto:Method: HEAD 2023-07-29T00:40:06.090 DEBUG:boto:Path: /hiiqos-59/obj68?versionId=null 2023-07-29T00:40:06.090 DEBUG:boto:Data: 2023-07-29T00:40:06.090 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.091 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:06.091 DEBUG:boto:Port: 8000 2023-07-29T00:40:06.091 DEBUG:boto:Params: {} 2023-07-29T00:40:06.092 DEBUG:boto:Token: None 2023-07-29T00:40:06.092 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj68?versionId=null 2023-07-29T00:40:06.092 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xl83duBGUJgyzEJWVHQopACPf4k= 2023-07-29T00:40:06.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xl83duBGUJgyzEJWVHQopACPf4k=', 'Content-Length': '0'} 2023-07-29T00:40:06.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075d0cfb668a82f58-0064c45fe6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-07-29T00:40:06.096 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:40:06.096 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:40:06.097 DEBUG:boto:path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:40:06.097 DEBUG:boto:auth_path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:40:06.097 DEBUG:boto:Method: GET 2023-07-29T00:40:06.098 DEBUG:boto:Path: /hiiqos-59/obj68?versionId=null 2023-07-29T00:40:06.098 DEBUG:boto:Data: 2023-07-29T00:40:06.098 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.099 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:06.099 DEBUG:boto:Port: 8001 2023-07-29T00:40:06.099 DEBUG:boto:Params: {} 2023-07-29T00:40:06.100 DEBUG:boto:Token: None 2023-07-29T00:40:06.100 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj68?versionId=null 2023-07-29T00:40:06.101 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O8wTLZ8UBhqhl7/KQIjlKpwC46E= 2023-07-29T00:40:06.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O8wTLZ8UBhqhl7/KQIjlKpwC46E=', 'Content-Length': '0'} 2023-07-29T00:40:06.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c7f85cbbe55d8ea-0064c45fe6-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.104 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:40:06.105 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:40:06.105 DEBUG:boto:path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:40:06.106 DEBUG:boto:auth_path=/hiiqos-59/obj68?versionId=null 2023-07-29T00:40:06.107 DEBUG:boto:Method: GET 2023-07-29T00:40:06.107 DEBUG:boto:Path: /hiiqos-59/obj68?versionId=null 2023-07-29T00:40:06.108 DEBUG:boto:Data: 2023-07-29T00:40:06.108 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.109 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:06.109 DEBUG:boto:Port: 8000 2023-07-29T00:40:06.110 DEBUG:boto:Params: {} 2023-07-29T00:40:06.110 DEBUG:boto:Token: None 2023-07-29T00:40:06.111 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj68?versionId=null 2023-07-29T00:40:06.112 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O8wTLZ8UBhqhl7/KQIjlKpwC46E= 2023-07-29T00:40:06.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O8wTLZ8UBhqhl7/KQIjlKpwC46E=', 'Content-Length': '0'} 2023-07-29T00:40:06.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a6bb29c1f89ce9e-0064c45fe6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.115 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:06.116 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:06.117 DEBUG:boto:path=/hiiqos-59/?prefix=obj68 2023-07-29T00:40:06.117 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj68 2023-07-29T00:40:06.118 DEBUG:boto:Method: GET 2023-07-29T00:40:06.118 DEBUG:boto:Path: /hiiqos-59/?prefix=obj68 2023-07-29T00:40:06.119 DEBUG:boto:Data: 2023-07-29T00:40:06.119 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.120 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:06.120 DEBUG:boto:Port: 8001 2023-07-29T00:40:06.121 DEBUG:boto:Params: {} 2023-07-29T00:40:06.121 DEBUG:boto:Token: None 2023-07-29T00:40:06.122 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/ 2023-07-29T00:40:06.123 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sN3umzbYPsTy+jGzbN8Ukc5/qnE= 2023-07-29T00:40:06.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sN3umzbYPsTy+jGzbN8Ukc5/qnE=', 'Content-Length': '0'} 2023-07-29T00:40:06.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008745c2634ca52fe9-0064c45fe6-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.127 DEBUG:boto:b'hiiqos-59obj681000falseobj682023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:06.128 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:06.128 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:06.129 DEBUG:boto:path=/hiiqos-59/?prefix=obj68 2023-07-29T00:40:06.129 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj68 2023-07-29T00:40:06.130 DEBUG:boto:Method: GET 2023-07-29T00:40:06.130 DEBUG:boto:Path: /hiiqos-59/?prefix=obj68 2023-07-29T00:40:06.131 DEBUG:boto:Data: 2023-07-29T00:40:06.131 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.132 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:06.132 DEBUG:boto:Port: 8000 2023-07-29T00:40:06.133 DEBUG:boto:Params: {} 2023-07-29T00:40:06.133 DEBUG:boto:Token: None 2023-07-29T00:40:06.134 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/ 2023-07-29T00:40:06.135 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sN3umzbYPsTy+jGzbN8Ukc5/qnE= 2023-07-29T00:40:06.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sN3umzbYPsTy+jGzbN8Ukc5/qnE=', 'Content-Length': '0'} 2023-07-29T00:40:06.140 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a19022b0f56d0b6d-0064c45fe6-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.140 DEBUG:boto:b'hiiqos-59obj681000falseobj682023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:06.141 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:40:06.142 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:40:06.142 DEBUG:boto:Method: HEAD 2023-07-29T00:40:06.143 DEBUG:boto:Path: /hiiqos-59/obj68 2023-07-29T00:40:06.144 DEBUG:boto:Data: 2023-07-29T00:40:06.144 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.145 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:06.145 DEBUG:boto:Port: 8001 2023-07-29T00:40:06.146 DEBUG:boto:Params: {} 2023-07-29T00:40:06.146 DEBUG:boto:Token: None 2023-07-29T00:40:06.147 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj68 2023-07-29T00:40:06.148 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xGMPBeprlPPgCjc+Q7p7T1atMYg= 2023-07-29T00:40:06.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xGMPBeprlPPgCjc+Q7p7T1atMYg=', 'Content-Length': '0'} 2023-07-29T00:40:06.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfc2ee316e3fa4d1-0064c45fe6-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.151 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:40:06.152 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:40:06.152 DEBUG:boto:Method: HEAD 2023-07-29T00:40:06.153 DEBUG:boto:Path: /hiiqos-59/obj68 2023-07-29T00:40:06.153 DEBUG:boto:Data: 2023-07-29T00:40:06.154 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.154 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:06.155 DEBUG:boto:Port: 8000 2023-07-29T00:40:06.156 DEBUG:boto:Params: {} 2023-07-29T00:40:06.156 DEBUG:boto:Token: None 2023-07-29T00:40:06.157 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj68 2023-07-29T00:40:06.157 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xGMPBeprlPPgCjc+Q7p7T1atMYg= 2023-07-29T00:40:06.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xGMPBeprlPPgCjc+Q7p7T1atMYg=', 'Content-Length': '0'} 2023-07-29T00:40:06.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095e73e53a9bf6596-0064c45fe6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-07-29T00:40:06.161 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:40:06.162 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:40:06.163 DEBUG:boto:Method: GET 2023-07-29T00:40:06.163 DEBUG:boto:Path: /hiiqos-59/obj68 2023-07-29T00:40:06.164 DEBUG:boto:Data: 2023-07-29T00:40:06.165 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.165 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:06.166 DEBUG:boto:Port: 8001 2023-07-29T00:40:06.166 DEBUG:boto:Params: {} 2023-07-29T00:40:06.167 DEBUG:boto:Token: None 2023-07-29T00:40:06.167 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj68 2023-07-29T00:40:06.168 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:NHVyEwE1xOO+YgymHddpSBFvUaw= 2023-07-29T00:40:06.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:NHVyEwE1xOO+YgymHddpSBFvUaw=', 'Content-Length': '0'} 2023-07-29T00:40:06.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000087ab072bbc1e2f1-0064c45fe6-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.171 DEBUG:boto:path=/hiiqos-59/obj68 2023-07-29T00:40:06.171 DEBUG:boto:auth_path=/hiiqos-59/obj68 2023-07-29T00:40:06.171 DEBUG:boto:Method: GET 2023-07-29T00:40:06.172 DEBUG:boto:Path: /hiiqos-59/obj68 2023-07-29T00:40:06.172 DEBUG:boto:Data: 2023-07-29T00:40:06.173 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.173 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:06.173 DEBUG:boto:Port: 8000 2023-07-29T00:40:06.174 DEBUG:boto:Params: {} 2023-07-29T00:40:06.174 DEBUG:boto:Token: None 2023-07-29T00:40:06.174 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj68 2023-07-29T00:40:06.175 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:NHVyEwE1xOO+YgymHddpSBFvUaw= 2023-07-29T00:40:06.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:NHVyEwE1xOO+YgymHddpSBFvUaw=', 'Content-Length': '0'} 2023-07-29T00:40:06.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7a6e39ffcbc7c69-0064c45fe6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.178 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:06.178 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:06.178 DEBUG:boto:path=/hiiqos-59/?prefix=obj68 2023-07-29T00:40:06.179 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj68 2023-07-29T00:40:06.179 DEBUG:boto:Method: GET 2023-07-29T00:40:06.179 DEBUG:boto:Path: /hiiqos-59/?prefix=obj68 2023-07-29T00:40:06.180 DEBUG:boto:Data: 2023-07-29T00:40:06.180 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.181 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:06.181 DEBUG:boto:Port: 8001 2023-07-29T00:40:06.181 DEBUG:boto:Params: {} 2023-07-29T00:40:06.182 DEBUG:boto:Token: None 2023-07-29T00:40:06.182 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/ 2023-07-29T00:40:06.182 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sN3umzbYPsTy+jGzbN8Ukc5/qnE= 2023-07-29T00:40:06.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sN3umzbYPsTy+jGzbN8Ukc5/qnE=', 'Content-Length': '0'} 2023-07-29T00:40:06.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045ecc533df5ba9b4-0064c45fe6-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.186 DEBUG:boto:b'hiiqos-59obj681000falseobj682023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:06.187 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:06.187 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:06.187 DEBUG:boto:path=/hiiqos-59/?prefix=obj68 2023-07-29T00:40:06.188 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj68 2023-07-29T00:40:06.188 DEBUG:boto:Method: GET 2023-07-29T00:40:06.188 DEBUG:boto:Path: /hiiqos-59/?prefix=obj68 2023-07-29T00:40:06.189 DEBUG:boto:Data: 2023-07-29T00:40:06.189 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.189 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:06.190 DEBUG:boto:Port: 8000 2023-07-29T00:40:06.190 DEBUG:boto:Params: {} 2023-07-29T00:40:06.191 DEBUG:boto:Token: None 2023-07-29T00:40:06.191 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/ 2023-07-29T00:40:06.191 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sN3umzbYPsTy+jGzbN8Ukc5/qnE= 2023-07-29T00:40:06.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sN3umzbYPsTy+jGzbN8Ukc5/qnE=', 'Content-Length': '0'} 2023-07-29T00:40:06.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f30e609f1d9d1668-0064c45fe6-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.240 DEBUG:boto:b'hiiqos-59obj681000falseobj682023-07-29T00:27:51.249Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:06.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-07-29T00:40:06.241 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:40:06.242 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:40:06.242 DEBUG:boto:path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:40:06.242 DEBUG:boto:auth_path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:40:06.243 DEBUG:boto:Method: GET 2023-07-29T00:40:06.243 DEBUG:boto:Path: /hiiqos-59/obj69?versionId=null 2023-07-29T00:40:06.244 DEBUG:boto:Data: 2023-07-29T00:40:06.244 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.244 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:06.245 DEBUG:boto:Port: 8001 2023-07-29T00:40:06.245 DEBUG:boto:Params: {} 2023-07-29T00:40:06.245 DEBUG:boto:Token: None 2023-07-29T00:40:06.246 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj69?versionId=null 2023-07-29T00:40:06.246 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Pzc6dBKSFrrVzlHiN5YtPCldHts= 2023-07-29T00:40:06.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Pzc6dBKSFrrVzlHiN5YtPCldHts=', 'Content-Length': '0'} 2023-07-29T00:40:06.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f90d629dd7720d2-0064c45fe6-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.249 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:40:06.250 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:40:06.250 DEBUG:boto:path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:40:06.251 DEBUG:boto:auth_path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:40:06.251 DEBUG:boto:Method: GET 2023-07-29T00:40:06.251 DEBUG:boto:Path: /hiiqos-59/obj69?versionId=null 2023-07-29T00:40:06.252 DEBUG:boto:Data: 2023-07-29T00:40:06.252 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.252 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:06.253 DEBUG:boto:Port: 8000 2023-07-29T00:40:06.253 DEBUG:boto:Params: {} 2023-07-29T00:40:06.253 DEBUG:boto:Token: None 2023-07-29T00:40:06.254 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj69?versionId=null 2023-07-29T00:40:06.254 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Pzc6dBKSFrrVzlHiN5YtPCldHts= 2023-07-29T00:40:06.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Pzc6dBKSFrrVzlHiN5YtPCldHts=', 'Content-Length': '0'} 2023-07-29T00:40:06.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e228e3c64c880bc-0064c45fe6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.258 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:40:06.258 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:40:06.258 DEBUG:boto:path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:40:06.259 DEBUG:boto:auth_path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:40:06.259 DEBUG:boto:Method: HEAD 2023-07-29T00:40:06.259 DEBUG:boto:Path: /hiiqos-59/obj69?versionId=null 2023-07-29T00:40:06.260 DEBUG:boto:Data: 2023-07-29T00:40:06.260 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.260 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:06.261 DEBUG:boto:Port: 8001 2023-07-29T00:40:06.261 DEBUG:boto:Params: {} 2023-07-29T00:40:06.261 DEBUG:boto:Token: None 2023-07-29T00:40:06.262 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj69?versionId=null 2023-07-29T00:40:06.262 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gDWtEgoDvQjyW6/V1nBbmC0jEbA= 2023-07-29T00:40:06.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gDWtEgoDvQjyW6/V1nBbmC0jEbA=', 'Content-Length': '0'} 2023-07-29T00:40:06.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5e43199b95ba0d0-0064c45fe6-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.265 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:40:06.265 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:40:06.266 DEBUG:boto:path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:40:06.266 DEBUG:boto:auth_path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:40:06.266 DEBUG:boto:Method: HEAD 2023-07-29T00:40:06.267 DEBUG:boto:Path: /hiiqos-59/obj69?versionId=null 2023-07-29T00:40:06.267 DEBUG:boto:Data: 2023-07-29T00:40:06.267 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.268 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:06.268 DEBUG:boto:Port: 8000 2023-07-29T00:40:06.268 DEBUG:boto:Params: {} 2023-07-29T00:40:06.269 DEBUG:boto:Token: None 2023-07-29T00:40:06.269 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj69?versionId=null 2023-07-29T00:40:06.269 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gDWtEgoDvQjyW6/V1nBbmC0jEbA= 2023-07-29T00:40:06.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gDWtEgoDvQjyW6/V1nBbmC0jEbA=', 'Content-Length': '0'} 2023-07-29T00:40:06.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a004d0fab86d98a-0064c45fe6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-07-29T00:40:06.273 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:40:06.273 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:40:06.274 DEBUG:boto:path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:40:06.274 DEBUG:boto:auth_path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:40:06.274 DEBUG:boto:Method: GET 2023-07-29T00:40:06.275 DEBUG:boto:Path: /hiiqos-59/obj69?versionId=null 2023-07-29T00:40:06.275 DEBUG:boto:Data: 2023-07-29T00:40:06.275 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.276 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:06.276 DEBUG:boto:Port: 8001 2023-07-29T00:40:06.276 DEBUG:boto:Params: {} 2023-07-29T00:40:06.277 DEBUG:boto:Token: None 2023-07-29T00:40:06.277 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj69?versionId=null 2023-07-29T00:40:06.278 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Pzc6dBKSFrrVzlHiN5YtPCldHts= 2023-07-29T00:40:06.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Pzc6dBKSFrrVzlHiN5YtPCldHts=', 'Content-Length': '0'} 2023-07-29T00:40:06.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b07fe0a5f4eea437-0064c45fe6-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.320 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:40:06.321 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:40:06.321 DEBUG:boto:path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:40:06.321 DEBUG:boto:auth_path=/hiiqos-59/obj69?versionId=null 2023-07-29T00:40:06.322 DEBUG:boto:Method: GET 2023-07-29T00:40:06.322 DEBUG:boto:Path: /hiiqos-59/obj69?versionId=null 2023-07-29T00:40:06.323 DEBUG:boto:Data: 2023-07-29T00:40:06.323 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.323 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:06.324 DEBUG:boto:Port: 8000 2023-07-29T00:40:06.324 DEBUG:boto:Params: {} 2023-07-29T00:40:06.324 DEBUG:boto:Token: None 2023-07-29T00:40:06.325 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj69?versionId=null 2023-07-29T00:40:06.325 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Pzc6dBKSFrrVzlHiN5YtPCldHts= 2023-07-29T00:40:06.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Pzc6dBKSFrrVzlHiN5YtPCldHts=', 'Content-Length': '0'} 2023-07-29T00:40:06.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f05005caea1c5650-0064c45fe6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.328 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:06.329 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:06.329 DEBUG:boto:path=/hiiqos-59/?prefix=obj69 2023-07-29T00:40:06.329 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj69 2023-07-29T00:40:06.330 DEBUG:boto:Method: GET 2023-07-29T00:40:06.330 DEBUG:boto:Path: /hiiqos-59/?prefix=obj69 2023-07-29T00:40:06.330 DEBUG:boto:Data: 2023-07-29T00:40:06.331 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.331 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:06.331 DEBUG:boto:Port: 8001 2023-07-29T00:40:06.332 DEBUG:boto:Params: {} 2023-07-29T00:40:06.332 DEBUG:boto:Token: None 2023-07-29T00:40:06.333 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/ 2023-07-29T00:40:06.333 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sN3umzbYPsTy+jGzbN8Ukc5/qnE= 2023-07-29T00:40:06.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sN3umzbYPsTy+jGzbN8Ukc5/qnE=', 'Content-Length': '0'} 2023-07-29T00:40:06.336 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000861132d00dfb752f-0064c45fe6-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.336 DEBUG:boto:b'hiiqos-59obj691000falseobj692023-07-29T00:33:06.857Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:06.337 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:06.337 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:06.338 DEBUG:boto:path=/hiiqos-59/?prefix=obj69 2023-07-29T00:40:06.338 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj69 2023-07-29T00:40:06.338 DEBUG:boto:Method: GET 2023-07-29T00:40:06.339 DEBUG:boto:Path: /hiiqos-59/?prefix=obj69 2023-07-29T00:40:06.339 DEBUG:boto:Data: 2023-07-29T00:40:06.339 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.340 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:06.340 DEBUG:boto:Port: 8000 2023-07-29T00:40:06.340 DEBUG:boto:Params: {} 2023-07-29T00:40:06.341 DEBUG:boto:Token: None 2023-07-29T00:40:06.341 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/ 2023-07-29T00:40:06.342 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sN3umzbYPsTy+jGzbN8Ukc5/qnE= 2023-07-29T00:40:06.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sN3umzbYPsTy+jGzbN8Ukc5/qnE=', 'Content-Length': '0'} 2023-07-29T00:40:06.346 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049fd860b1ee1b2ab-0064c45fe6-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.347 DEBUG:boto:b'hiiqos-59obj691000falseobj692023-07-29T00:33:06.857Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:06.347 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:40:06.348 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:40:06.348 DEBUG:boto:Method: HEAD 2023-07-29T00:40:06.348 DEBUG:boto:Path: /hiiqos-59/obj69 2023-07-29T00:40:06.349 DEBUG:boto:Data: 2023-07-29T00:40:06.349 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.349 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:06.350 DEBUG:boto:Port: 8001 2023-07-29T00:40:06.350 DEBUG:boto:Params: {} 2023-07-29T00:40:06.350 DEBUG:boto:Token: None 2023-07-29T00:40:06.351 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj69 2023-07-29T00:40:06.351 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gPW+iBHxJeSNw5CkB/LSFAu9XU4= 2023-07-29T00:40:06.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gPW+iBHxJeSNw5CkB/LSFAu9XU4=', 'Content-Length': '0'} 2023-07-29T00:40:06.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f65b58f474f0a302-0064c45fe6-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.354 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:40:06.354 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:40:06.355 DEBUG:boto:Method: HEAD 2023-07-29T00:40:06.355 DEBUG:boto:Path: /hiiqos-59/obj69 2023-07-29T00:40:06.355 DEBUG:boto:Data: 2023-07-29T00:40:06.356 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.356 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:06.356 DEBUG:boto:Port: 8000 2023-07-29T00:40:06.357 DEBUG:boto:Params: {} 2023-07-29T00:40:06.357 DEBUG:boto:Token: None 2023-07-29T00:40:06.357 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj69 2023-07-29T00:40:06.358 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gPW+iBHxJeSNw5CkB/LSFAu9XU4= 2023-07-29T00:40:06.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gPW+iBHxJeSNw5CkB/LSFAu9XU4=', 'Content-Length': '0'} 2023-07-29T00:40:06.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000388c1a0a09d36230-0064c45fe6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-07-29T00:40:06.361 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:40:06.361 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:40:06.362 DEBUG:boto:Method: GET 2023-07-29T00:40:06.362 DEBUG:boto:Path: /hiiqos-59/obj69 2023-07-29T00:40:06.362 DEBUG:boto:Data: 2023-07-29T00:40:06.363 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.363 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:06.363 DEBUG:boto:Port: 8001 2023-07-29T00:40:06.364 DEBUG:boto:Params: {} 2023-07-29T00:40:06.364 DEBUG:boto:Token: None 2023-07-29T00:40:06.364 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj69 2023-07-29T00:40:06.365 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Oq2+UWBwM7agORryDW18rn4XD20= 2023-07-29T00:40:06.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Oq2+UWBwM7agORryDW18rn4XD20=', 'Content-Length': '0'} 2023-07-29T00:40:06.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078366125136fc73e-0064c45fe6-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.368 DEBUG:boto:path=/hiiqos-59/obj69 2023-07-29T00:40:06.368 DEBUG:boto:auth_path=/hiiqos-59/obj69 2023-07-29T00:40:06.369 DEBUG:boto:Method: GET 2023-07-29T00:40:06.370 DEBUG:boto:Path: /hiiqos-59/obj69 2023-07-29T00:40:06.370 DEBUG:boto:Data: 2023-07-29T00:40:06.371 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.371 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:06.372 DEBUG:boto:Port: 8000 2023-07-29T00:40:06.372 DEBUG:boto:Params: {} 2023-07-29T00:40:06.372 DEBUG:boto:Token: None 2023-07-29T00:40:06.373 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj69 2023-07-29T00:40:06.373 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Oq2+UWBwM7agORryDW18rn4XD20= 2023-07-29T00:40:06.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Oq2+UWBwM7agORryDW18rn4XD20=', 'Content-Length': '0'} 2023-07-29T00:40:06.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8f340c7707aa1a9-0064c45fe6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.416 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:06.417 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:06.417 DEBUG:boto:path=/hiiqos-59/?prefix=obj69 2023-07-29T00:40:06.418 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj69 2023-07-29T00:40:06.418 DEBUG:boto:Method: GET 2023-07-29T00:40:06.419 DEBUG:boto:Path: /hiiqos-59/?prefix=obj69 2023-07-29T00:40:06.419 DEBUG:boto:Data: 2023-07-29T00:40:06.419 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.420 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:06.420 DEBUG:boto:Port: 8001 2023-07-29T00:40:06.421 DEBUG:boto:Params: {} 2023-07-29T00:40:06.421 DEBUG:boto:Token: None 2023-07-29T00:40:06.422 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/ 2023-07-29T00:40:06.422 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sN3umzbYPsTy+jGzbN8Ukc5/qnE= 2023-07-29T00:40:06.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sN3umzbYPsTy+jGzbN8Ukc5/qnE=', 'Content-Length': '0'} 2023-07-29T00:40:06.425 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085c6f61bfb75ccdb-0064c45fe6-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.426 DEBUG:boto:b'hiiqos-59obj691000falseobj692023-07-29T00:33:06.857Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:06.427 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:06.427 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:06.428 DEBUG:boto:path=/hiiqos-59/?prefix=obj69 2023-07-29T00:40:06.428 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj69 2023-07-29T00:40:06.428 DEBUG:boto:Method: GET 2023-07-29T00:40:06.429 DEBUG:boto:Path: /hiiqos-59/?prefix=obj69 2023-07-29T00:40:06.429 DEBUG:boto:Data: 2023-07-29T00:40:06.430 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.430 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:06.430 DEBUG:boto:Port: 8000 2023-07-29T00:40:06.431 DEBUG:boto:Params: {} 2023-07-29T00:40:06.431 DEBUG:boto:Token: None 2023-07-29T00:40:06.433 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/ 2023-07-29T00:40:06.433 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sN3umzbYPsTy+jGzbN8Ukc5/qnE= 2023-07-29T00:40:06.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sN3umzbYPsTy+jGzbN8Ukc5/qnE=', 'Content-Length': '0'} 2023-07-29T00:40:06.438 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a26887a0efa109ab-0064c45fe6-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.439 DEBUG:boto:b'hiiqos-59obj691000falseobj692023-07-29T00:33:06.857Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:06.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-07-29T00:40:06.441 DEBUG:boto:path=/hiiqos-59/obj74 2023-07-29T00:40:06.442 DEBUG:boto:auth_path=/hiiqos-59/obj74 2023-07-29T00:40:06.442 DEBUG:boto:path=/hiiqos-59/obj74?versionId=null 2023-07-29T00:40:06.442 DEBUG:boto:auth_path=/hiiqos-59/obj74?versionId=null 2023-07-29T00:40:06.443 DEBUG:boto:Method: GET 2023-07-29T00:40:06.443 DEBUG:boto:Path: /hiiqos-59/obj74?versionId=null 2023-07-29T00:40:06.444 DEBUG:boto:Data: 2023-07-29T00:40:06.444 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.445 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:06.445 DEBUG:boto:Port: 8001 2023-07-29T00:40:06.445 DEBUG:boto:Params: {} 2023-07-29T00:40:06.446 DEBUG:boto:Token: None 2023-07-29T00:40:06.446 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj74?versionId=null 2023-07-29T00:40:06.447 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xm8CzK7lhRGMT8KxZVYqdoLoowE= 2023-07-29T00:40:06.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xm8CzK7lhRGMT8KxZVYqdoLoowE=', 'Content-Length': '0'} 2023-07-29T00:40:06.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b336f03d6ad3af4b-0064c45fe6-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.450 DEBUG:boto:path=/hiiqos-59/obj74 2023-07-29T00:40:06.451 DEBUG:boto:auth_path=/hiiqos-59/obj74 2023-07-29T00:40:06.451 DEBUG:boto:path=/hiiqos-59/obj74?versionId=null 2023-07-29T00:40:06.451 DEBUG:boto:auth_path=/hiiqos-59/obj74?versionId=null 2023-07-29T00:40:06.452 DEBUG:boto:Method: GET 2023-07-29T00:40:06.452 DEBUG:boto:Path: /hiiqos-59/obj74?versionId=null 2023-07-29T00:40:06.453 DEBUG:boto:Data: 2023-07-29T00:40:06.453 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.454 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:06.454 DEBUG:boto:Port: 8000 2023-07-29T00:40:06.454 DEBUG:boto:Params: {} 2023-07-29T00:40:06.455 DEBUG:boto:Token: None 2023-07-29T00:40:06.455 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj74?versionId=null 2023-07-29T00:40:06.456 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xm8CzK7lhRGMT8KxZVYqdoLoowE= 2023-07-29T00:40:06.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xm8CzK7lhRGMT8KxZVYqdoLoowE=', 'Content-Length': '0'} 2023-07-29T00:40:06.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028481452073c8251-0064c45fe6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.459 DEBUG:boto:path=/hiiqos-59/obj74 2023-07-29T00:40:06.460 DEBUG:boto:auth_path=/hiiqos-59/obj74 2023-07-29T00:40:06.460 DEBUG:boto:path=/hiiqos-59/obj74?versionId=null 2023-07-29T00:40:06.461 DEBUG:boto:auth_path=/hiiqos-59/obj74?versionId=null 2023-07-29T00:40:06.461 DEBUG:boto:Method: HEAD 2023-07-29T00:40:06.461 DEBUG:boto:Path: /hiiqos-59/obj74?versionId=null 2023-07-29T00:40:06.462 DEBUG:boto:Data: 2023-07-29T00:40:06.462 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.463 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:06.463 DEBUG:boto:Port: 8001 2023-07-29T00:40:06.463 DEBUG:boto:Params: {} 2023-07-29T00:40:06.464 DEBUG:boto:Token: None 2023-07-29T00:40:06.464 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj74?versionId=null 2023-07-29T00:40:06.465 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pesbjX2OzMCYq1OezC4O1V1nad0= 2023-07-29T00:40:06.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pesbjX2OzMCYq1OezC4O1V1nad0=', 'Content-Length': '0'} 2023-07-29T00:40:06.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c442dd61f2a07e21-0064c45fe6-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.468 DEBUG:boto:path=/hiiqos-59/obj74 2023-07-29T00:40:06.468 DEBUG:boto:auth_path=/hiiqos-59/obj74 2023-07-29T00:40:06.469 DEBUG:boto:path=/hiiqos-59/obj74?versionId=null 2023-07-29T00:40:06.469 DEBUG:boto:auth_path=/hiiqos-59/obj74?versionId=null 2023-07-29T00:40:06.469 DEBUG:boto:Method: HEAD 2023-07-29T00:40:06.470 DEBUG:boto:Path: /hiiqos-59/obj74?versionId=null 2023-07-29T00:40:06.470 DEBUG:boto:Data: 2023-07-29T00:40:06.470 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.471 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:06.471 DEBUG:boto:Port: 8000 2023-07-29T00:40:06.472 DEBUG:boto:Params: {} 2023-07-29T00:40:06.472 DEBUG:boto:Token: None 2023-07-29T00:40:06.473 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj74?versionId=null 2023-07-29T00:40:06.473 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pesbjX2OzMCYq1OezC4O1V1nad0= 2023-07-29T00:40:06.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pesbjX2OzMCYq1OezC4O1V1nad0=', 'Content-Length': '0'} 2023-07-29T00:40:06.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074855f9aee7bb391-0064c45fe6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-07-29T00:40:06.476 DEBUG:boto:path=/hiiqos-59/obj74 2023-07-29T00:40:06.477 DEBUG:boto:auth_path=/hiiqos-59/obj74 2023-07-29T00:40:06.477 DEBUG:boto:path=/hiiqos-59/obj74?versionId=null 2023-07-29T00:40:06.477 DEBUG:boto:auth_path=/hiiqos-59/obj74?versionId=null 2023-07-29T00:40:06.478 DEBUG:boto:Method: GET 2023-07-29T00:40:06.478 DEBUG:boto:Path: /hiiqos-59/obj74?versionId=null 2023-07-29T00:40:06.479 DEBUG:boto:Data: 2023-07-29T00:40:06.479 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.480 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:06.480 DEBUG:boto:Port: 8001 2023-07-29T00:40:06.480 DEBUG:boto:Params: {} 2023-07-29T00:40:06.481 DEBUG:boto:Token: None 2023-07-29T00:40:06.481 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj74?versionId=null 2023-07-29T00:40:06.482 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xm8CzK7lhRGMT8KxZVYqdoLoowE= 2023-07-29T00:40:06.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xm8CzK7lhRGMT8KxZVYqdoLoowE=', 'Content-Length': '0'} 2023-07-29T00:40:06.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042c1ee708913c917-0064c45fe6-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.485 DEBUG:boto:path=/hiiqos-59/obj74 2023-07-29T00:40:06.485 DEBUG:boto:auth_path=/hiiqos-59/obj74 2023-07-29T00:40:06.485 DEBUG:boto:path=/hiiqos-59/obj74?versionId=null 2023-07-29T00:40:06.486 DEBUG:boto:auth_path=/hiiqos-59/obj74?versionId=null 2023-07-29T00:40:06.486 DEBUG:boto:Method: GET 2023-07-29T00:40:06.487 DEBUG:boto:Path: /hiiqos-59/obj74?versionId=null 2023-07-29T00:40:06.487 DEBUG:boto:Data: 2023-07-29T00:40:06.487 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.488 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:06.488 DEBUG:boto:Port: 8000 2023-07-29T00:40:06.488 DEBUG:boto:Params: {} 2023-07-29T00:40:06.489 DEBUG:boto:Token: None 2023-07-29T00:40:06.489 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj74?versionId=null 2023-07-29T00:40:06.490 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xm8CzK7lhRGMT8KxZVYqdoLoowE= 2023-07-29T00:40:06.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xm8CzK7lhRGMT8KxZVYqdoLoowE=', 'Content-Length': '0'} 2023-07-29T00:40:06.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b0f8693445abe8b-0064c45fe6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.493 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:06.493 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:06.494 DEBUG:boto:path=/hiiqos-59/?prefix=obj74 2023-07-29T00:40:06.494 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj74 2023-07-29T00:40:06.495 DEBUG:boto:Method: GET 2023-07-29T00:40:06.495 DEBUG:boto:Path: /hiiqos-59/?prefix=obj74 2023-07-29T00:40:06.495 DEBUG:boto:Data: 2023-07-29T00:40:06.496 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.496 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:06.497 DEBUG:boto:Port: 8001 2023-07-29T00:40:06.497 DEBUG:boto:Params: {} 2023-07-29T00:40:06.498 DEBUG:boto:Token: None 2023-07-29T00:40:06.498 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/ 2023-07-29T00:40:06.499 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sN3umzbYPsTy+jGzbN8Ukc5/qnE= 2023-07-29T00:40:06.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sN3umzbYPsTy+jGzbN8Ukc5/qnE=', 'Content-Length': '0'} 2023-07-29T00:40:06.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3f29bc569da817b-0064c45fe6-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.502 DEBUG:boto:b'hiiqos-59obj741000falseobj742023-07-29T00:37:53.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:06.503 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:06.503 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:06.504 DEBUG:boto:path=/hiiqos-59/?prefix=obj74 2023-07-29T00:40:06.504 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj74 2023-07-29T00:40:06.505 DEBUG:boto:Method: GET 2023-07-29T00:40:06.505 DEBUG:boto:Path: /hiiqos-59/?prefix=obj74 2023-07-29T00:40:06.506 DEBUG:boto:Data: 2023-07-29T00:40:06.506 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.506 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:06.507 DEBUG:boto:Port: 8000 2023-07-29T00:40:06.507 DEBUG:boto:Params: {} 2023-07-29T00:40:06.507 DEBUG:boto:Token: None 2023-07-29T00:40:06.508 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/ 2023-07-29T00:40:06.508 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sN3umzbYPsTy+jGzbN8Ukc5/qnE= 2023-07-29T00:40:06.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sN3umzbYPsTy+jGzbN8Ukc5/qnE=', 'Content-Length': '0'} 2023-07-29T00:40:06.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb1a0f658b6cc4a8-0064c45fe6-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.556 DEBUG:boto:b'hiiqos-59obj741000falseobj742023-07-29T00:37:53.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:06.557 DEBUG:boto:path=/hiiqos-59/obj74 2023-07-29T00:40:06.558 DEBUG:boto:auth_path=/hiiqos-59/obj74 2023-07-29T00:40:06.558 DEBUG:boto:Method: HEAD 2023-07-29T00:40:06.558 DEBUG:boto:Path: /hiiqos-59/obj74 2023-07-29T00:40:06.559 DEBUG:boto:Data: 2023-07-29T00:40:06.559 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.559 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:06.560 DEBUG:boto:Port: 8001 2023-07-29T00:40:06.560 DEBUG:boto:Params: {} 2023-07-29T00:40:06.560 DEBUG:boto:Token: None 2023-07-29T00:40:06.561 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj74 2023-07-29T00:40:06.561 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DXzMeQ7j/EY/YodZum/8Upu+Ncw= 2023-07-29T00:40:06.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DXzMeQ7j/EY/YodZum/8Upu+Ncw=', 'Content-Length': '0'} 2023-07-29T00:40:06.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049585eb16c0185a3-0064c45fe6-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.564 DEBUG:boto:path=/hiiqos-59/obj74 2023-07-29T00:40:06.565 DEBUG:boto:auth_path=/hiiqos-59/obj74 2023-07-29T00:40:06.565 DEBUG:boto:Method: HEAD 2023-07-29T00:40:06.565 DEBUG:boto:Path: /hiiqos-59/obj74 2023-07-29T00:40:06.566 DEBUG:boto:Data: 2023-07-29T00:40:06.567 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.567 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:06.568 DEBUG:boto:Port: 8000 2023-07-29T00:40:06.568 DEBUG:boto:Params: {} 2023-07-29T00:40:06.569 DEBUG:boto:Token: None 2023-07-29T00:40:06.570 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj74 2023-07-29T00:40:06.570 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DXzMeQ7j/EY/YodZum/8Upu+Ncw= 2023-07-29T00:40:06.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DXzMeQ7j/EY/YodZum/8Upu+Ncw=', 'Content-Length': '0'} 2023-07-29T00:40:06.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097c10910a572b91e-0064c45fe6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-07-29T00:40:06.574 DEBUG:boto:path=/hiiqos-59/obj74 2023-07-29T00:40:06.574 DEBUG:boto:auth_path=/hiiqos-59/obj74 2023-07-29T00:40:06.574 DEBUG:boto:Method: GET 2023-07-29T00:40:06.575 DEBUG:boto:Path: /hiiqos-59/obj74 2023-07-29T00:40:06.575 DEBUG:boto:Data: 2023-07-29T00:40:06.575 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.576 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:06.576 DEBUG:boto:Port: 8001 2023-07-29T00:40:06.577 DEBUG:boto:Params: {} 2023-07-29T00:40:06.577 DEBUG:boto:Token: None 2023-07-29T00:40:06.578 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj74 2023-07-29T00:40:06.578 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ze8q9prthoVDiS5NHlLNMZI8LIA= 2023-07-29T00:40:06.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ze8q9prthoVDiS5NHlLNMZI8LIA=', 'Content-Length': '0'} 2023-07-29T00:40:06.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cea892597fa17ed6-0064c45fe6-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.581 DEBUG:boto:path=/hiiqos-59/obj74 2023-07-29T00:40:06.582 DEBUG:boto:auth_path=/hiiqos-59/obj74 2023-07-29T00:40:06.582 DEBUG:boto:Method: GET 2023-07-29T00:40:06.582 DEBUG:boto:Path: /hiiqos-59/obj74 2023-07-29T00:40:06.583 DEBUG:boto:Data: 2023-07-29T00:40:06.583 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.583 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:06.584 DEBUG:boto:Port: 8000 2023-07-29T00:40:06.584 DEBUG:boto:Params: {} 2023-07-29T00:40:06.585 DEBUG:boto:Token: None 2023-07-29T00:40:06.585 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj74 2023-07-29T00:40:06.585 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ze8q9prthoVDiS5NHlLNMZI8LIA= 2023-07-29T00:40:06.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ze8q9prthoVDiS5NHlLNMZI8LIA=', 'Content-Length': '0'} 2023-07-29T00:40:06.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:37:53 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d36d0fc5ec9edb1-0064c45fe6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.589 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:06.589 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:06.589 DEBUG:boto:path=/hiiqos-59/?prefix=obj74 2023-07-29T00:40:06.590 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj74 2023-07-29T00:40:06.590 DEBUG:boto:Method: GET 2023-07-29T00:40:06.590 DEBUG:boto:Path: /hiiqos-59/?prefix=obj74 2023-07-29T00:40:06.591 DEBUG:boto:Data: 2023-07-29T00:40:06.591 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.592 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:06.592 DEBUG:boto:Port: 8001 2023-07-29T00:40:06.592 DEBUG:boto:Params: {} 2023-07-29T00:40:06.593 DEBUG:boto:Token: None 2023-07-29T00:40:06.593 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/ 2023-07-29T00:40:06.593 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sN3umzbYPsTy+jGzbN8Ukc5/qnE= 2023-07-29T00:40:06.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sN3umzbYPsTy+jGzbN8Ukc5/qnE=', 'Content-Length': '0'} 2023-07-29T00:40:06.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018d8641169c23cb1-0064c45fe6-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.640 DEBUG:boto:b'hiiqos-59obj741000falseobj742023-07-29T00:37:53.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:06.641 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:06.642 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:06.642 DEBUG:boto:path=/hiiqos-59/?prefix=obj74 2023-07-29T00:40:06.642 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj74 2023-07-29T00:40:06.643 DEBUG:boto:Method: GET 2023-07-29T00:40:06.643 DEBUG:boto:Path: /hiiqos-59/?prefix=obj74 2023-07-29T00:40:06.643 DEBUG:boto:Data: 2023-07-29T00:40:06.644 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.644 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:06.645 DEBUG:boto:Port: 8000 2023-07-29T00:40:06.645 DEBUG:boto:Params: {} 2023-07-29T00:40:06.646 DEBUG:boto:Token: None 2023-07-29T00:40:06.646 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/ 2023-07-29T00:40:06.647 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sN3umzbYPsTy+jGzbN8Ukc5/qnE= 2023-07-29T00:40:06.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sN3umzbYPsTy+jGzbN8Ukc5/qnE=', 'Content-Length': '0'} 2023-07-29T00:40:06.651 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066a982566ab3a20d-0064c45fe6-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.653 DEBUG:boto:b'hiiqos-59obj741000falseobj742023-07-29T00:37:53.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:06.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-07-29T00:40:06.654 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:40:06.654 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:40:06.655 DEBUG:boto:path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:40:06.655 DEBUG:boto:auth_path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:40:06.656 DEBUG:boto:Method: GET 2023-07-29T00:40:06.656 DEBUG:boto:Path: /hiiqos-59/obj76?versionId=null 2023-07-29T00:40:06.656 DEBUG:boto:Data: 2023-07-29T00:40:06.657 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.657 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:06.657 DEBUG:boto:Port: 8001 2023-07-29T00:40:06.658 DEBUG:boto:Params: {} 2023-07-29T00:40:06.658 DEBUG:boto:Token: None 2023-07-29T00:40:06.659 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj76?versionId=null 2023-07-29T00:40:06.659 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hZKWB1BuEEZnoDu7x/yMj3Snwlc= 2023-07-29T00:40:06.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hZKWB1BuEEZnoDu7x/yMj3Snwlc=', 'Content-Length': '0'} 2023-07-29T00:40:06.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001748aadf6a721ca3-0064c45fe6-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.663 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:40:06.663 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:40:06.664 DEBUG:boto:path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:40:06.665 DEBUG:boto:auth_path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:40:06.666 DEBUG:boto:Method: GET 2023-07-29T00:40:06.667 DEBUG:boto:Path: /hiiqos-59/obj76?versionId=null 2023-07-29T00:40:06.667 DEBUG:boto:Data: 2023-07-29T00:40:06.668 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.669 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:06.670 DEBUG:boto:Port: 8000 2023-07-29T00:40:06.671 DEBUG:boto:Params: {} 2023-07-29T00:40:06.672 DEBUG:boto:Token: None 2023-07-29T00:40:06.672 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj76?versionId=null 2023-07-29T00:40:06.673 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hZKWB1BuEEZnoDu7x/yMj3Snwlc= 2023-07-29T00:40:06.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hZKWB1BuEEZnoDu7x/yMj3Snwlc=', 'Content-Length': '0'} 2023-07-29T00:40:06.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000811c89916ca746e3-0064c45fe6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.716 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:40:06.717 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:40:06.717 DEBUG:boto:path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:40:06.717 DEBUG:boto:auth_path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:40:06.718 DEBUG:boto:Method: HEAD 2023-07-29T00:40:06.718 DEBUG:boto:Path: /hiiqos-59/obj76?versionId=null 2023-07-29T00:40:06.719 DEBUG:boto:Data: 2023-07-29T00:40:06.719 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.719 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:06.720 DEBUG:boto:Port: 8001 2023-07-29T00:40:06.720 DEBUG:boto:Params: {} 2023-07-29T00:40:06.720 DEBUG:boto:Token: None 2023-07-29T00:40:06.721 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj76?versionId=null 2023-07-29T00:40:06.721 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gTfs9FBM0JA7jct2YazdE+2CTAc= 2023-07-29T00:40:06.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gTfs9FBM0JA7jct2YazdE+2CTAc=', 'Content-Length': '0'} 2023-07-29T00:40:06.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e15bfb41b62cdeba-0064c45fe6-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.724 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:40:06.724 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:40:06.725 DEBUG:boto:path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:40:06.725 DEBUG:boto:auth_path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:40:06.725 DEBUG:boto:Method: HEAD 2023-07-29T00:40:06.726 DEBUG:boto:Path: /hiiqos-59/obj76?versionId=null 2023-07-29T00:40:06.726 DEBUG:boto:Data: 2023-07-29T00:40:06.726 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.727 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:06.727 DEBUG:boto:Port: 8000 2023-07-29T00:40:06.727 DEBUG:boto:Params: {} 2023-07-29T00:40:06.728 DEBUG:boto:Token: None 2023-07-29T00:40:06.728 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj76?versionId=null 2023-07-29T00:40:06.729 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gTfs9FBM0JA7jct2YazdE+2CTAc= 2023-07-29T00:40:06.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gTfs9FBM0JA7jct2YazdE+2CTAc=', 'Content-Length': '0'} 2023-07-29T00:40:06.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c1fc027008230d4-0064c45fe6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-07-29T00:40:06.732 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:40:06.732 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:40:06.733 DEBUG:boto:path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:40:06.733 DEBUG:boto:auth_path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:40:06.733 DEBUG:boto:Method: GET 2023-07-29T00:40:06.734 DEBUG:boto:Path: /hiiqos-59/obj76?versionId=null 2023-07-29T00:40:06.734 DEBUG:boto:Data: 2023-07-29T00:40:06.734 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.735 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:06.735 DEBUG:boto:Port: 8001 2023-07-29T00:40:06.735 DEBUG:boto:Params: {} 2023-07-29T00:40:06.736 DEBUG:boto:Token: None 2023-07-29T00:40:06.736 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj76?versionId=null 2023-07-29T00:40:06.737 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hZKWB1BuEEZnoDu7x/yMj3Snwlc= 2023-07-29T00:40:06.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hZKWB1BuEEZnoDu7x/yMj3Snwlc=', 'Content-Length': '0'} 2023-07-29T00:40:06.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b3cdd75e4edc079-0064c45fe6-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.740 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:40:06.740 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:40:06.741 DEBUG:boto:path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:40:06.741 DEBUG:boto:auth_path=/hiiqos-59/obj76?versionId=null 2023-07-29T00:40:06.742 DEBUG:boto:Method: GET 2023-07-29T00:40:06.742 DEBUG:boto:Path: /hiiqos-59/obj76?versionId=null 2023-07-29T00:40:06.742 DEBUG:boto:Data: 2023-07-29T00:40:06.743 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.743 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:06.743 DEBUG:boto:Port: 8000 2023-07-29T00:40:06.744 DEBUG:boto:Params: {} 2023-07-29T00:40:06.744 DEBUG:boto:Token: None 2023-07-29T00:40:06.745 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj76?versionId=null 2023-07-29T00:40:06.745 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hZKWB1BuEEZnoDu7x/yMj3Snwlc= 2023-07-29T00:40:06.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hZKWB1BuEEZnoDu7x/yMj3Snwlc=', 'Content-Length': '0'} 2023-07-29T00:40:06.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ac7d771a1051114-0064c45fe6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.752 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:06.752 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:06.753 DEBUG:boto:path=/hiiqos-59/?prefix=obj76 2023-07-29T00:40:06.753 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj76 2023-07-29T00:40:06.754 DEBUG:boto:Method: GET 2023-07-29T00:40:06.754 DEBUG:boto:Path: /hiiqos-59/?prefix=obj76 2023-07-29T00:40:06.754 DEBUG:boto:Data: 2023-07-29T00:40:06.755 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.755 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:06.756 DEBUG:boto:Port: 8001 2023-07-29T00:40:06.756 DEBUG:boto:Params: {} 2023-07-29T00:40:06.757 DEBUG:boto:Token: None 2023-07-29T00:40:06.758 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/ 2023-07-29T00:40:06.758 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sN3umzbYPsTy+jGzbN8Ukc5/qnE= 2023-07-29T00:40:06.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sN3umzbYPsTy+jGzbN8Ukc5/qnE=', 'Content-Length': '0'} 2023-07-29T00:40:06.761 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bd17c421d260c09-0064c45fe6-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.762 DEBUG:boto:b'hiiqos-59obj761000falseobj762023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:06.762 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:06.763 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:06.763 DEBUG:boto:path=/hiiqos-59/?prefix=obj76 2023-07-29T00:40:06.763 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj76 2023-07-29T00:40:06.764 DEBUG:boto:Method: GET 2023-07-29T00:40:06.764 DEBUG:boto:Path: /hiiqos-59/?prefix=obj76 2023-07-29T00:40:06.765 DEBUG:boto:Data: 2023-07-29T00:40:06.765 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.766 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:06.767 DEBUG:boto:Port: 8000 2023-07-29T00:40:06.768 DEBUG:boto:Params: {} 2023-07-29T00:40:06.769 DEBUG:boto:Token: None 2023-07-29T00:40:06.770 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/ 2023-07-29T00:40:06.771 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sN3umzbYPsTy+jGzbN8Ukc5/qnE= 2023-07-29T00:40:06.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sN3umzbYPsTy+jGzbN8Ukc5/qnE=', 'Content-Length': '0'} 2023-07-29T00:40:06.776 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a3b02194722046d-0064c45fe6-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.777 DEBUG:boto:b'hiiqos-59obj761000falseobj762023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:06.778 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:40:06.778 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:40:06.778 DEBUG:boto:Method: HEAD 2023-07-29T00:40:06.779 DEBUG:boto:Path: /hiiqos-59/obj76 2023-07-29T00:40:06.779 DEBUG:boto:Data: 2023-07-29T00:40:06.779 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.780 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:06.780 DEBUG:boto:Port: 8001 2023-07-29T00:40:06.781 DEBUG:boto:Params: {} 2023-07-29T00:40:06.781 DEBUG:boto:Token: None 2023-07-29T00:40:06.781 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj76 2023-07-29T00:40:06.782 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Jnf2puXf+VITa8TEBsFfnEtD3ss= 2023-07-29T00:40:06.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Jnf2puXf+VITa8TEBsFfnEtD3ss=', 'Content-Length': '0'} 2023-07-29T00:40:06.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026958b04ad5e27ac-0064c45fe6-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.785 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:40:06.788 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:40:06.788 DEBUG:boto:Method: HEAD 2023-07-29T00:40:06.789 DEBUG:boto:Path: /hiiqos-59/obj76 2023-07-29T00:40:06.789 DEBUG:boto:Data: 2023-07-29T00:40:06.789 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.790 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:06.790 DEBUG:boto:Port: 8000 2023-07-29T00:40:06.790 DEBUG:boto:Params: {} 2023-07-29T00:40:06.791 DEBUG:boto:Token: None 2023-07-29T00:40:06.791 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj76 2023-07-29T00:40:06.792 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Jnf2puXf+VITa8TEBsFfnEtD3ss= 2023-07-29T00:40:06.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Jnf2puXf+VITa8TEBsFfnEtD3ss=', 'Content-Length': '0'} 2023-07-29T00:40:06.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b5c9fc62e88ce89-0064c45fe6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-07-29T00:40:06.795 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:40:06.796 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:40:06.796 DEBUG:boto:Method: GET 2023-07-29T00:40:06.797 DEBUG:boto:Path: /hiiqos-59/obj76 2023-07-29T00:40:06.797 DEBUG:boto:Data: 2023-07-29T00:40:06.797 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.798 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:06.798 DEBUG:boto:Port: 8001 2023-07-29T00:40:06.798 DEBUG:boto:Params: {} 2023-07-29T00:40:06.799 DEBUG:boto:Token: None 2023-07-29T00:40:06.799 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj76 2023-07-29T00:40:06.800 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nfO8fC9t1839cibBuHAW0NGJB5k= 2023-07-29T00:40:06.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nfO8fC9t1839cibBuHAW0NGJB5k=', 'Content-Length': '0'} 2023-07-29T00:40:06.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001017f965eb8d9ba4-0064c45fe6-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.844 DEBUG:boto:path=/hiiqos-59/obj76 2023-07-29T00:40:06.845 DEBUG:boto:auth_path=/hiiqos-59/obj76 2023-07-29T00:40:06.845 DEBUG:boto:Method: GET 2023-07-29T00:40:06.846 DEBUG:boto:Path: /hiiqos-59/obj76 2023-07-29T00:40:06.846 DEBUG:boto:Data: 2023-07-29T00:40:06.847 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.847 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:06.847 DEBUG:boto:Port: 8000 2023-07-29T00:40:06.848 DEBUG:boto:Params: {} 2023-07-29T00:40:06.848 DEBUG:boto:Token: None 2023-07-29T00:40:06.849 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj76 2023-07-29T00:40:06.849 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nfO8fC9t1839cibBuHAW0NGJB5k= 2023-07-29T00:40:06.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nfO8fC9t1839cibBuHAW0NGJB5k=', 'Content-Length': '0'} 2023-07-29T00:40:06.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008be0c50d73d1396c-0064c45fe6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.852 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:06.853 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:06.853 DEBUG:boto:path=/hiiqos-59/?prefix=obj76 2023-07-29T00:40:06.853 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj76 2023-07-29T00:40:06.854 DEBUG:boto:Method: GET 2023-07-29T00:40:06.854 DEBUG:boto:Path: /hiiqos-59/?prefix=obj76 2023-07-29T00:40:06.855 DEBUG:boto:Data: 2023-07-29T00:40:06.855 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.855 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:06.856 DEBUG:boto:Port: 8001 2023-07-29T00:40:06.856 DEBUG:boto:Params: {} 2023-07-29T00:40:06.856 DEBUG:boto:Token: None 2023-07-29T00:40:06.857 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/ 2023-07-29T00:40:06.857 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sN3umzbYPsTy+jGzbN8Ukc5/qnE= 2023-07-29T00:40:06.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sN3umzbYPsTy+jGzbN8Ukc5/qnE=', 'Content-Length': '0'} 2023-07-29T00:40:06.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092f10a48d1f2b65e-0064c45fe6-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.861 DEBUG:boto:b'hiiqos-59obj761000falseobj762023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:06.861 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:06.862 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:06.862 DEBUG:boto:path=/hiiqos-59/?prefix=obj76 2023-07-29T00:40:06.862 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj76 2023-07-29T00:40:06.863 DEBUG:boto:Method: GET 2023-07-29T00:40:06.863 DEBUG:boto:Path: /hiiqos-59/?prefix=obj76 2023-07-29T00:40:06.864 DEBUG:boto:Data: 2023-07-29T00:40:06.864 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.864 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:06.865 DEBUG:boto:Port: 8000 2023-07-29T00:40:06.865 DEBUG:boto:Params: {} 2023-07-29T00:40:06.865 DEBUG:boto:Token: None 2023-07-29T00:40:06.866 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/ 2023-07-29T00:40:06.866 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sN3umzbYPsTy+jGzbN8Ukc5/qnE= 2023-07-29T00:40:06.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sN3umzbYPsTy+jGzbN8Ukc5/qnE=', 'Content-Length': '0'} 2023-07-29T00:40:06.871 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005677bff3c5d8b7d2-0064c45fe6-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.871 DEBUG:boto:b'hiiqos-59obj761000falseobj762023-07-29T00:29:58.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:06.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-07-29T00:40:06.872 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:40:06.873 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:40:06.873 DEBUG:boto:path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:40:06.873 DEBUG:boto:auth_path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:40:06.874 DEBUG:boto:Method: GET 2023-07-29T00:40:06.874 DEBUG:boto:Path: /hiiqos-59/obj85?versionId=null 2023-07-29T00:40:06.874 DEBUG:boto:Data: 2023-07-29T00:40:06.875 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.875 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:06.876 DEBUG:boto:Port: 8001 2023-07-29T00:40:06.876 DEBUG:boto:Params: {} 2023-07-29T00:40:06.876 DEBUG:boto:Token: None 2023-07-29T00:40:06.877 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj85?versionId=null 2023-07-29T00:40:06.877 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fnkvYkivVYf+8fZeWskSgaXjD9o= 2023-07-29T00:40:06.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fnkvYkivVYf+8fZeWskSgaXjD9o=', 'Content-Length': '0'} 2023-07-29T00:40:06.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e727a7fe9c718a4e-0064c45fe6-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.880 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:40:06.881 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:40:06.881 DEBUG:boto:path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:40:06.881 DEBUG:boto:auth_path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:40:06.882 DEBUG:boto:Method: GET 2023-07-29T00:40:06.882 DEBUG:boto:Path: /hiiqos-59/obj85?versionId=null 2023-07-29T00:40:06.882 DEBUG:boto:Data: 2023-07-29T00:40:06.883 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.883 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:06.883 DEBUG:boto:Port: 8000 2023-07-29T00:40:06.884 DEBUG:boto:Params: {} 2023-07-29T00:40:06.884 DEBUG:boto:Token: None 2023-07-29T00:40:06.885 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj85?versionId=null 2023-07-29T00:40:06.885 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fnkvYkivVYf+8fZeWskSgaXjD9o= 2023-07-29T00:40:06.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fnkvYkivVYf+8fZeWskSgaXjD9o=', 'Content-Length': '0'} 2023-07-29T00:40:06.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021aadf0e2b8949e0-0064c45fe6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.928 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:40:06.929 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:40:06.929 DEBUG:boto:path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:40:06.929 DEBUG:boto:auth_path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:40:06.930 DEBUG:boto:Method: HEAD 2023-07-29T00:40:06.930 DEBUG:boto:Path: /hiiqos-59/obj85?versionId=null 2023-07-29T00:40:06.930 DEBUG:boto:Data: 2023-07-29T00:40:06.931 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.931 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:06.931 DEBUG:boto:Port: 8001 2023-07-29T00:40:06.932 DEBUG:boto:Params: {} 2023-07-29T00:40:06.932 DEBUG:boto:Token: None 2023-07-29T00:40:06.933 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj85?versionId=null 2023-07-29T00:40:06.933 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QrX0hll3GaV+FmiOPgcT8IB9kgg= 2023-07-29T00:40:06.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QrX0hll3GaV+FmiOPgcT8IB9kgg=', 'Content-Length': '0'} 2023-07-29T00:40:06.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000057818d3096b6d40-0064c45fe6-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.936 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:40:06.936 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:40:06.936 DEBUG:boto:path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:40:06.937 DEBUG:boto:auth_path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:40:06.937 DEBUG:boto:Method: HEAD 2023-07-29T00:40:06.937 DEBUG:boto:Path: /hiiqos-59/obj85?versionId=null 2023-07-29T00:40:06.938 DEBUG:boto:Data: 2023-07-29T00:40:06.938 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.938 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:06.939 DEBUG:boto:Port: 8000 2023-07-29T00:40:06.939 DEBUG:boto:Params: {} 2023-07-29T00:40:06.940 DEBUG:boto:Token: None 2023-07-29T00:40:06.940 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj85?versionId=null 2023-07-29T00:40:06.940 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QrX0hll3GaV+FmiOPgcT8IB9kgg= 2023-07-29T00:40:06.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QrX0hll3GaV+FmiOPgcT8IB9kgg=', 'Content-Length': '0'} 2023-07-29T00:40:06.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003994168c6c592878-0064c45fe6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-07-29T00:40:06.944 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:40:06.944 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:40:06.944 DEBUG:boto:path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:40:06.945 DEBUG:boto:auth_path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:40:06.945 DEBUG:boto:Method: GET 2023-07-29T00:40:06.945 DEBUG:boto:Path: /hiiqos-59/obj85?versionId=null 2023-07-29T00:40:06.946 DEBUG:boto:Data: 2023-07-29T00:40:06.946 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.946 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:06.947 DEBUG:boto:Port: 8001 2023-07-29T00:40:06.947 DEBUG:boto:Params: {} 2023-07-29T00:40:06.948 DEBUG:boto:Token: None 2023-07-29T00:40:06.948 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj85?versionId=null 2023-07-29T00:40:06.949 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fnkvYkivVYf+8fZeWskSgaXjD9o= 2023-07-29T00:40:06.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fnkvYkivVYf+8fZeWskSgaXjD9o=', 'Content-Length': '0'} 2023-07-29T00:40:06.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f44db4d962056019-0064c45fe6-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.952 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:40:06.952 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:40:06.952 DEBUG:boto:path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:40:06.953 DEBUG:boto:auth_path=/hiiqos-59/obj85?versionId=null 2023-07-29T00:40:06.953 DEBUG:boto:Method: GET 2023-07-29T00:40:06.953 DEBUG:boto:Path: /hiiqos-59/obj85?versionId=null 2023-07-29T00:40:06.954 DEBUG:boto:Data: 2023-07-29T00:40:06.954 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.954 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:06.955 DEBUG:boto:Port: 8000 2023-07-29T00:40:06.955 DEBUG:boto:Params: {} 2023-07-29T00:40:06.956 DEBUG:boto:Token: None 2023-07-29T00:40:06.956 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj85?versionId=null 2023-07-29T00:40:06.956 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fnkvYkivVYf+8fZeWskSgaXjD9o= 2023-07-29T00:40:06.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fnkvYkivVYf+8fZeWskSgaXjD9o=', 'Content-Length': '0'} 2023-07-29T00:40:06.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a3a864d27a4e7de-0064c45fe6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.960 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:06.960 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:06.960 DEBUG:boto:path=/hiiqos-59/?prefix=obj85 2023-07-29T00:40:06.961 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj85 2023-07-29T00:40:06.961 DEBUG:boto:Method: GET 2023-07-29T00:40:06.961 DEBUG:boto:Path: /hiiqos-59/?prefix=obj85 2023-07-29T00:40:06.962 DEBUG:boto:Data: 2023-07-29T00:40:06.962 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.963 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:06.963 DEBUG:boto:Port: 8001 2023-07-29T00:40:06.963 DEBUG:boto:Params: {} 2023-07-29T00:40:06.964 DEBUG:boto:Token: None 2023-07-29T00:40:06.964 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/ 2023-07-29T00:40:06.964 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sN3umzbYPsTy+jGzbN8Ukc5/qnE= 2023-07-29T00:40:06.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sN3umzbYPsTy+jGzbN8Ukc5/qnE=', 'Content-Length': '0'} 2023-07-29T00:40:06.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7778fc29722ceb0-0064c45fe6-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.968 DEBUG:boto:b'hiiqos-59obj851000falseobj852023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:06.968 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:06.969 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:06.969 DEBUG:boto:path=/hiiqos-59/?prefix=obj85 2023-07-29T00:40:06.970 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj85 2023-07-29T00:40:06.970 DEBUG:boto:Method: GET 2023-07-29T00:40:06.970 DEBUG:boto:Path: /hiiqos-59/?prefix=obj85 2023-07-29T00:40:06.971 DEBUG:boto:Data: 2023-07-29T00:40:06.971 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.971 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:06.972 DEBUG:boto:Port: 8000 2023-07-29T00:40:06.972 DEBUG:boto:Params: {} 2023-07-29T00:40:06.972 DEBUG:boto:Token: None 2023-07-29T00:40:06.973 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/ 2023-07-29T00:40:06.973 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:sN3umzbYPsTy+jGzbN8Ukc5/qnE= 2023-07-29T00:40:06.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:sN3umzbYPsTy+jGzbN8Ukc5/qnE=', 'Content-Length': '0'} 2023-07-29T00:40:06.978 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdd5cbc34714c0ec-0064c45fe6-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.979 DEBUG:boto:b'hiiqos-59obj851000falseobj852023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:06.979 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:40:06.980 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:40:06.980 DEBUG:boto:Method: HEAD 2023-07-29T00:40:06.981 DEBUG:boto:Path: /hiiqos-59/obj85 2023-07-29T00:40:06.981 DEBUG:boto:Data: 2023-07-29T00:40:06.981 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.982 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:06.982 DEBUG:boto:Port: 8001 2023-07-29T00:40:06.982 DEBUG:boto:Params: {} 2023-07-29T00:40:06.983 DEBUG:boto:Token: None 2023-07-29T00:40:06.983 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj85 2023-07-29T00:40:06.983 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MLOAVMcuxtUI4QaecsZi/hGQHiA= 2023-07-29T00:40:06.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MLOAVMcuxtUI4QaecsZi/hGQHiA=', 'Content-Length': '0'} 2023-07-29T00:40:06.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d17e932dd3ea7cac-0064c45fe6-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.986 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:40:06.987 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:40:06.987 DEBUG:boto:Method: HEAD 2023-07-29T00:40:06.987 DEBUG:boto:Path: /hiiqos-59/obj85 2023-07-29T00:40:06.988 DEBUG:boto:Data: 2023-07-29T00:40:06.988 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.988 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:06.989 DEBUG:boto:Port: 8000 2023-07-29T00:40:06.989 DEBUG:boto:Params: {} 2023-07-29T00:40:06.989 DEBUG:boto:Token: None 2023-07-29T00:40:06.990 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj85 2023-07-29T00:40:06.990 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MLOAVMcuxtUI4QaecsZi/hGQHiA= 2023-07-29T00:40:06.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MLOAVMcuxtUI4QaecsZi/hGQHiA=', 'Content-Length': '0'} 2023-07-29T00:40:06.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040442031f02beb52-0064c45fe6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:06.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-07-29T00:40:06.993 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:40:06.994 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:40:06.994 DEBUG:boto:Method: GET 2023-07-29T00:40:06.995 DEBUG:boto:Path: /hiiqos-59/obj85 2023-07-29T00:40:06.995 DEBUG:boto:Data: 2023-07-29T00:40:06.995 DEBUG:boto:Headers: {} 2023-07-29T00:40:06.996 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:06.996 DEBUG:boto:Port: 8001 2023-07-29T00:40:06.996 DEBUG:boto:Params: {} 2023-07-29T00:40:06.997 DEBUG:boto:Token: None 2023-07-29T00:40:06.997 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:06 GMT /hiiqos-59/obj85 2023-07-29T00:40:06.998 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ptucYDRB8aRNHemkV1sWW0Gkry8= 2023-07-29T00:40:06.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ptucYDRB8aRNHemkV1sWW0Gkry8=', 'Content-Length': '0'} 2023-07-29T00:40:07.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1fb2998d8e5c2eb-0064c45fe6-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.000 DEBUG:boto:path=/hiiqos-59/obj85 2023-07-29T00:40:07.001 DEBUG:boto:auth_path=/hiiqos-59/obj85 2023-07-29T00:40:07.001 DEBUG:boto:Method: GET 2023-07-29T00:40:07.002 DEBUG:boto:Path: /hiiqos-59/obj85 2023-07-29T00:40:07.002 DEBUG:boto:Data: 2023-07-29T00:40:07.002 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.003 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:07.003 DEBUG:boto:Port: 8000 2023-07-29T00:40:07.003 DEBUG:boto:Params: {} 2023-07-29T00:40:07.004 DEBUG:boto:Token: None 2023-07-29T00:40:07.004 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/obj85 2023-07-29T00:40:07.005 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7T9muGrrsKQHOGD2EoN86CwXBBc= 2023-07-29T00:40:07.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7T9muGrrsKQHOGD2EoN86CwXBBc=', 'Content-Length': '0'} 2023-07-29T00:40:07.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:27:51 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071683edba04e3a84-0064c45fe6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.048 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:07.049 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:07.049 DEBUG:boto:path=/hiiqos-59/?prefix=obj85 2023-07-29T00:40:07.050 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj85 2023-07-29T00:40:07.050 DEBUG:boto:Method: GET 2023-07-29T00:40:07.050 DEBUG:boto:Path: /hiiqos-59/?prefix=obj85 2023-07-29T00:40:07.051 DEBUG:boto:Data: 2023-07-29T00:40:07.051 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.051 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:07.052 DEBUG:boto:Port: 8001 2023-07-29T00:40:07.053 DEBUG:boto:Params: {} 2023-07-29T00:40:07.054 DEBUG:boto:Token: None 2023-07-29T00:40:07.055 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/ 2023-07-29T00:40:07.056 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:a29jdrQmja9yUKofwSodYQGXo5w= 2023-07-29T00:40:07.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:a29jdrQmja9yUKofwSodYQGXo5w=', 'Content-Length': '0'} 2023-07-29T00:40:07.060 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002c0f335083ab0cb-0064c45fe6-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.061 DEBUG:boto:b'hiiqos-59obj851000falseobj852023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:07.062 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:07.063 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:07.063 DEBUG:boto:path=/hiiqos-59/?prefix=obj85 2023-07-29T00:40:07.064 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj85 2023-07-29T00:40:07.065 DEBUG:boto:Method: GET 2023-07-29T00:40:07.065 DEBUG:boto:Path: /hiiqos-59/?prefix=obj85 2023-07-29T00:40:07.066 DEBUG:boto:Data: 2023-07-29T00:40:07.067 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.068 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:07.068 DEBUG:boto:Port: 8000 2023-07-29T00:40:07.069 DEBUG:boto:Params: {} 2023-07-29T00:40:07.070 DEBUG:boto:Token: None 2023-07-29T00:40:07.071 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/ 2023-07-29T00:40:07.071 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:a29jdrQmja9yUKofwSodYQGXo5w= 2023-07-29T00:40:07.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:a29jdrQmja9yUKofwSodYQGXo5w=', 'Content-Length': '0'} 2023-07-29T00:40:07.076 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bc09542520ced4f-0064c45fe7-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.076 DEBUG:boto:b'hiiqos-59obj851000falseobj852023-07-29T00:27:51.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:07.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-07-29T00:40:07.077 DEBUG:boto:path=/hiiqos-59/obj87 2023-07-29T00:40:07.078 DEBUG:boto:auth_path=/hiiqos-59/obj87 2023-07-29T00:40:07.078 DEBUG:boto:path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:40:07.079 DEBUG:boto:auth_path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:40:07.079 DEBUG:boto:Method: GET 2023-07-29T00:40:07.079 DEBUG:boto:Path: /hiiqos-59/obj87?versionId=null 2023-07-29T00:40:07.080 DEBUG:boto:Data: 2023-07-29T00:40:07.080 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.080 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:07.081 DEBUG:boto:Port: 8001 2023-07-29T00:40:07.081 DEBUG:boto:Params: {} 2023-07-29T00:40:07.082 DEBUG:boto:Token: None 2023-07-29T00:40:07.082 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/obj87?versionId=null 2023-07-29T00:40:07.082 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MuGl78ztyHvRKLLMt9z9jKSywgQ= 2023-07-29T00:40:07.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MuGl78ztyHvRKLLMt9z9jKSywgQ=', 'Content-Length': '0'} 2023-07-29T00:40:07.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000795c6a59df2490be-0064c45fe7-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.086 DEBUG:boto:path=/hiiqos-59/obj87 2023-07-29T00:40:07.086 DEBUG:boto:auth_path=/hiiqos-59/obj87 2023-07-29T00:40:07.086 DEBUG:boto:path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:40:07.087 DEBUG:boto:auth_path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:40:07.087 DEBUG:boto:Method: GET 2023-07-29T00:40:07.088 DEBUG:boto:Path: /hiiqos-59/obj87?versionId=null 2023-07-29T00:40:07.088 DEBUG:boto:Data: 2023-07-29T00:40:07.088 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.089 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:07.089 DEBUG:boto:Port: 8000 2023-07-29T00:40:07.090 DEBUG:boto:Params: {} 2023-07-29T00:40:07.090 DEBUG:boto:Token: None 2023-07-29T00:40:07.090 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/obj87?versionId=null 2023-07-29T00:40:07.091 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MuGl78ztyHvRKLLMt9z9jKSywgQ= 2023-07-29T00:40:07.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MuGl78ztyHvRKLLMt9z9jKSywgQ=', 'Content-Length': '0'} 2023-07-29T00:40:07.093 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026d2fd1f3084b314-0064c45fe7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.094 DEBUG:boto:path=/hiiqos-59/obj87 2023-07-29T00:40:07.095 DEBUG:boto:auth_path=/hiiqos-59/obj87 2023-07-29T00:40:07.095 DEBUG:boto:path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:40:07.095 DEBUG:boto:auth_path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:40:07.096 DEBUG:boto:Method: HEAD 2023-07-29T00:40:07.096 DEBUG:boto:Path: /hiiqos-59/obj87?versionId=null 2023-07-29T00:40:07.096 DEBUG:boto:Data: 2023-07-29T00:40:07.097 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.097 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:07.097 DEBUG:boto:Port: 8001 2023-07-29T00:40:07.098 DEBUG:boto:Params: {} 2023-07-29T00:40:07.098 DEBUG:boto:Token: None 2023-07-29T00:40:07.099 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/obj87?versionId=null 2023-07-29T00:40:07.099 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5kRURY3eajEXDia/6ZHfBAq+kuo= 2023-07-29T00:40:07.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5kRURY3eajEXDia/6ZHfBAq+kuo=', 'Content-Length': '0'} 2023-07-29T00:40:07.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007ec884fd4205f41-0064c45fe7-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.102 DEBUG:boto:path=/hiiqos-59/obj87 2023-07-29T00:40:07.102 DEBUG:boto:auth_path=/hiiqos-59/obj87 2023-07-29T00:40:07.102 DEBUG:boto:path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:40:07.103 DEBUG:boto:auth_path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:40:07.103 DEBUG:boto:Method: HEAD 2023-07-29T00:40:07.104 DEBUG:boto:Path: /hiiqos-59/obj87?versionId=null 2023-07-29T00:40:07.104 DEBUG:boto:Data: 2023-07-29T00:40:07.104 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.105 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:07.105 DEBUG:boto:Port: 8000 2023-07-29T00:40:07.105 DEBUG:boto:Params: {} 2023-07-29T00:40:07.106 DEBUG:boto:Token: None 2023-07-29T00:40:07.106 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/obj87?versionId=null 2023-07-29T00:40:07.107 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5kRURY3eajEXDia/6ZHfBAq+kuo= 2023-07-29T00:40:07.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5kRURY3eajEXDia/6ZHfBAq+kuo=', 'Content-Length': '0'} 2023-07-29T00:40:07.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006db764348f4c10c-0064c45fe7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-07-29T00:40:07.110 DEBUG:boto:path=/hiiqos-59/obj87 2023-07-29T00:40:07.110 DEBUG:boto:auth_path=/hiiqos-59/obj87 2023-07-29T00:40:07.111 DEBUG:boto:path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:40:07.111 DEBUG:boto:auth_path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:40:07.111 DEBUG:boto:Method: GET 2023-07-29T00:40:07.112 DEBUG:boto:Path: /hiiqos-59/obj87?versionId=null 2023-07-29T00:40:07.112 DEBUG:boto:Data: 2023-07-29T00:40:07.112 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.113 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:07.113 DEBUG:boto:Port: 8001 2023-07-29T00:40:07.113 DEBUG:boto:Params: {} 2023-07-29T00:40:07.114 DEBUG:boto:Token: None 2023-07-29T00:40:07.114 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/obj87?versionId=null 2023-07-29T00:40:07.115 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MuGl78ztyHvRKLLMt9z9jKSywgQ= 2023-07-29T00:40:07.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MuGl78ztyHvRKLLMt9z9jKSywgQ=', 'Content-Length': '0'} 2023-07-29T00:40:07.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076108d6d9929ab7f-0064c45fe7-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.164 DEBUG:boto:path=/hiiqos-59/obj87 2023-07-29T00:40:07.165 DEBUG:boto:auth_path=/hiiqos-59/obj87 2023-07-29T00:40:07.165 DEBUG:boto:path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:40:07.166 DEBUG:boto:auth_path=/hiiqos-59/obj87?versionId=null 2023-07-29T00:40:07.166 DEBUG:boto:Method: GET 2023-07-29T00:40:07.166 DEBUG:boto:Path: /hiiqos-59/obj87?versionId=null 2023-07-29T00:40:07.167 DEBUG:boto:Data: 2023-07-29T00:40:07.167 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.167 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:07.168 DEBUG:boto:Port: 8000 2023-07-29T00:40:07.168 DEBUG:boto:Params: {} 2023-07-29T00:40:07.169 DEBUG:boto:Token: None 2023-07-29T00:40:07.169 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/obj87?versionId=null 2023-07-29T00:40:07.169 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MuGl78ztyHvRKLLMt9z9jKSywgQ= 2023-07-29T00:40:07.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MuGl78ztyHvRKLLMt9z9jKSywgQ=', 'Content-Length': '0'} 2023-07-29T00:40:07.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073d73a631f1948ae-0064c45fe7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.172 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:07.173 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:07.173 DEBUG:boto:path=/hiiqos-59/?prefix=obj87 2023-07-29T00:40:07.174 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj87 2023-07-29T00:40:07.174 DEBUG:boto:Method: GET 2023-07-29T00:40:07.174 DEBUG:boto:Path: /hiiqos-59/?prefix=obj87 2023-07-29T00:40:07.175 DEBUG:boto:Data: 2023-07-29T00:40:07.175 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.175 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:07.176 DEBUG:boto:Port: 8001 2023-07-29T00:40:07.176 DEBUG:boto:Params: {} 2023-07-29T00:40:07.177 DEBUG:boto:Token: None 2023-07-29T00:40:07.177 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/ 2023-07-29T00:40:07.177 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:a29jdrQmja9yUKofwSodYQGXo5w= 2023-07-29T00:40:07.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:a29jdrQmja9yUKofwSodYQGXo5w=', 'Content-Length': '0'} 2023-07-29T00:40:07.180 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f35e495318ca76c2-0064c45fe7-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.181 DEBUG:boto:b'hiiqos-59obj871000falseobj872023-07-29T00:35:01.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:07.181 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:07.182 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:07.182 DEBUG:boto:path=/hiiqos-59/?prefix=obj87 2023-07-29T00:40:07.182 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj87 2023-07-29T00:40:07.183 DEBUG:boto:Method: GET 2023-07-29T00:40:07.183 DEBUG:boto:Path: /hiiqos-59/?prefix=obj87 2023-07-29T00:40:07.184 DEBUG:boto:Data: 2023-07-29T00:40:07.184 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.184 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:07.185 DEBUG:boto:Port: 8000 2023-07-29T00:40:07.185 DEBUG:boto:Params: {} 2023-07-29T00:40:07.185 DEBUG:boto:Token: None 2023-07-29T00:40:07.186 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/ 2023-07-29T00:40:07.186 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:a29jdrQmja9yUKofwSodYQGXo5w= 2023-07-29T00:40:07.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:a29jdrQmja9yUKofwSodYQGXo5w=', 'Content-Length': '0'} 2023-07-29T00:40:07.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ceed05d29a81cb16-0064c45fe7-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.191 DEBUG:boto:b'hiiqos-59obj871000falseobj872023-07-29T00:35:01.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:07.191 DEBUG:boto:path=/hiiqos-59/obj87 2023-07-29T00:40:07.192 DEBUG:boto:auth_path=/hiiqos-59/obj87 2023-07-29T00:40:07.192 DEBUG:boto:Method: HEAD 2023-07-29T00:40:07.193 DEBUG:boto:Path: /hiiqos-59/obj87 2023-07-29T00:40:07.193 DEBUG:boto:Data: 2023-07-29T00:40:07.193 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.194 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:07.194 DEBUG:boto:Port: 8001 2023-07-29T00:40:07.194 DEBUG:boto:Params: {} 2023-07-29T00:40:07.195 DEBUG:boto:Token: None 2023-07-29T00:40:07.195 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/obj87 2023-07-29T00:40:07.195 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HGVNMwWUlS0pFE4kCpGKuWYhvWw= 2023-07-29T00:40:07.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HGVNMwWUlS0pFE4kCpGKuWYhvWw=', 'Content-Length': '0'} 2023-07-29T00:40:07.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd520be4d5d1024a-0064c45fe7-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.198 DEBUG:boto:path=/hiiqos-59/obj87 2023-07-29T00:40:07.199 DEBUG:boto:auth_path=/hiiqos-59/obj87 2023-07-29T00:40:07.199 DEBUG:boto:Method: HEAD 2023-07-29T00:40:07.199 DEBUG:boto:Path: /hiiqos-59/obj87 2023-07-29T00:40:07.200 DEBUG:boto:Data: 2023-07-29T00:40:07.200 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.200 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:07.201 DEBUG:boto:Port: 8000 2023-07-29T00:40:07.201 DEBUG:boto:Params: {} 2023-07-29T00:40:07.201 DEBUG:boto:Token: None 2023-07-29T00:40:07.202 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/obj87 2023-07-29T00:40:07.202 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HGVNMwWUlS0pFE4kCpGKuWYhvWw= 2023-07-29T00:40:07.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HGVNMwWUlS0pFE4kCpGKuWYhvWw=', 'Content-Length': '0'} 2023-07-29T00:40:07.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d40582fe1ba6d87-0064c45fe7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-07-29T00:40:07.205 DEBUG:boto:path=/hiiqos-59/obj87 2023-07-29T00:40:07.206 DEBUG:boto:auth_path=/hiiqos-59/obj87 2023-07-29T00:40:07.206 DEBUG:boto:Method: GET 2023-07-29T00:40:07.206 DEBUG:boto:Path: /hiiqos-59/obj87 2023-07-29T00:40:07.207 DEBUG:boto:Data: 2023-07-29T00:40:07.207 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.207 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:07.208 DEBUG:boto:Port: 8001 2023-07-29T00:40:07.208 DEBUG:boto:Params: {} 2023-07-29T00:40:07.209 DEBUG:boto:Token: None 2023-07-29T00:40:07.209 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/obj87 2023-07-29T00:40:07.209 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TWPObh29CKMCW3vv4iwm+peKs1A= 2023-07-29T00:40:07.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TWPObh29CKMCW3vv4iwm+peKs1A=', 'Content-Length': '0'} 2023-07-29T00:40:07.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db86e30f7b3202a5-0064c45fe7-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.212 DEBUG:boto:path=/hiiqos-59/obj87 2023-07-29T00:40:07.213 DEBUG:boto:auth_path=/hiiqos-59/obj87 2023-07-29T00:40:07.213 DEBUG:boto:Method: GET 2023-07-29T00:40:07.213 DEBUG:boto:Path: /hiiqos-59/obj87 2023-07-29T00:40:07.214 DEBUG:boto:Data: 2023-07-29T00:40:07.214 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.214 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:07.215 DEBUG:boto:Port: 8000 2023-07-29T00:40:07.215 DEBUG:boto:Params: {} 2023-07-29T00:40:07.216 DEBUG:boto:Token: None 2023-07-29T00:40:07.216 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/obj87 2023-07-29T00:40:07.216 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TWPObh29CKMCW3vv4iwm+peKs1A= 2023-07-29T00:40:07.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TWPObh29CKMCW3vv4iwm+peKs1A=', 'Content-Length': '0'} 2023-07-29T00:40:07.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013a526a8b085bc01-0064c45fe7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.260 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:07.261 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:07.261 DEBUG:boto:path=/hiiqos-59/?prefix=obj87 2023-07-29T00:40:07.262 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj87 2023-07-29T00:40:07.262 DEBUG:boto:Method: GET 2023-07-29T00:40:07.262 DEBUG:boto:Path: /hiiqos-59/?prefix=obj87 2023-07-29T00:40:07.263 DEBUG:boto:Data: 2023-07-29T00:40:07.263 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.263 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:07.264 DEBUG:boto:Port: 8001 2023-07-29T00:40:07.264 DEBUG:boto:Params: {} 2023-07-29T00:40:07.265 DEBUG:boto:Token: None 2023-07-29T00:40:07.265 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/ 2023-07-29T00:40:07.265 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:a29jdrQmja9yUKofwSodYQGXo5w= 2023-07-29T00:40:07.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:a29jdrQmja9yUKofwSodYQGXo5w=', 'Content-Length': '0'} 2023-07-29T00:40:07.268 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004649c41a10ad4c35-0064c45fe7-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.269 DEBUG:boto:b'hiiqos-59obj871000falseobj872023-07-29T00:35:01.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:07.270 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:07.270 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:07.270 DEBUG:boto:path=/hiiqos-59/?prefix=obj87 2023-07-29T00:40:07.271 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj87 2023-07-29T00:40:07.271 DEBUG:boto:Method: GET 2023-07-29T00:40:07.272 DEBUG:boto:Path: /hiiqos-59/?prefix=obj87 2023-07-29T00:40:07.272 DEBUG:boto:Data: 2023-07-29T00:40:07.272 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.273 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:07.273 DEBUG:boto:Port: 8000 2023-07-29T00:40:07.274 DEBUG:boto:Params: {} 2023-07-29T00:40:07.274 DEBUG:boto:Token: None 2023-07-29T00:40:07.274 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/ 2023-07-29T00:40:07.275 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:a29jdrQmja9yUKofwSodYQGXo5w= 2023-07-29T00:40:07.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:a29jdrQmja9yUKofwSodYQGXo5w=', 'Content-Length': '0'} 2023-07-29T00:40:07.280 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbec149b60ccf985-0064c45fe7-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.281 DEBUG:boto:b'hiiqos-59obj871000falseobj872023-07-29T00:35:01.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:07.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-07-29T00:40:07.282 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:40:07.282 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:40:07.283 DEBUG:boto:path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:40:07.283 DEBUG:boto:auth_path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:40:07.284 DEBUG:boto:Method: GET 2023-07-29T00:40:07.284 DEBUG:boto:Path: /hiiqos-59/obj92?versionId=null 2023-07-29T00:40:07.284 DEBUG:boto:Data: 2023-07-29T00:40:07.285 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.285 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:07.285 DEBUG:boto:Port: 8001 2023-07-29T00:40:07.286 DEBUG:boto:Params: {} 2023-07-29T00:40:07.286 DEBUG:boto:Token: None 2023-07-29T00:40:07.287 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/obj92?versionId=null 2023-07-29T00:40:07.287 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BXaS5OVEPN5bTUEXHGncLNPbrAM= 2023-07-29T00:40:07.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BXaS5OVEPN5bTUEXHGncLNPbrAM=', 'Content-Length': '0'} 2023-07-29T00:40:07.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0045858da7a7a96-0064c45fe7-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.336 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:40:07.337 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:40:07.337 DEBUG:boto:path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:40:07.338 DEBUG:boto:auth_path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:40:07.338 DEBUG:boto:Method: GET 2023-07-29T00:40:07.338 DEBUG:boto:Path: /hiiqos-59/obj92?versionId=null 2023-07-29T00:40:07.339 DEBUG:boto:Data: 2023-07-29T00:40:07.339 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.339 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:07.340 DEBUG:boto:Port: 8000 2023-07-29T00:40:07.340 DEBUG:boto:Params: {} 2023-07-29T00:40:07.341 DEBUG:boto:Token: None 2023-07-29T00:40:07.341 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/obj92?versionId=null 2023-07-29T00:40:07.342 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BXaS5OVEPN5bTUEXHGncLNPbrAM= 2023-07-29T00:40:07.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BXaS5OVEPN5bTUEXHGncLNPbrAM=', 'Content-Length': '0'} 2023-07-29T00:40:07.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2cfe90c345dd9c8-0064c45fe7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.345 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:40:07.345 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:40:07.345 DEBUG:boto:path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:40:07.346 DEBUG:boto:auth_path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:40:07.346 DEBUG:boto:Method: HEAD 2023-07-29T00:40:07.346 DEBUG:boto:Path: /hiiqos-59/obj92?versionId=null 2023-07-29T00:40:07.347 DEBUG:boto:Data: 2023-07-29T00:40:07.347 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.347 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:07.348 DEBUG:boto:Port: 8001 2023-07-29T00:40:07.348 DEBUG:boto:Params: {} 2023-07-29T00:40:07.348 DEBUG:boto:Token: None 2023-07-29T00:40:07.349 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/obj92?versionId=null 2023-07-29T00:40:07.349 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BY9WRn9Sg365hkbZzGtdyDbNz80= 2023-07-29T00:40:07.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BY9WRn9Sg365hkbZzGtdyDbNz80=', 'Content-Length': '0'} 2023-07-29T00:40:07.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005aa76e34e1cc5de7-0064c45fe7-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.352 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:40:07.353 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:40:07.353 DEBUG:boto:path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:40:07.353 DEBUG:boto:auth_path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:40:07.354 DEBUG:boto:Method: HEAD 2023-07-29T00:40:07.354 DEBUG:boto:Path: /hiiqos-59/obj92?versionId=null 2023-07-29T00:40:07.354 DEBUG:boto:Data: 2023-07-29T00:40:07.355 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.355 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:07.355 DEBUG:boto:Port: 8000 2023-07-29T00:40:07.356 DEBUG:boto:Params: {} 2023-07-29T00:40:07.356 DEBUG:boto:Token: None 2023-07-29T00:40:07.356 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/obj92?versionId=null 2023-07-29T00:40:07.357 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BY9WRn9Sg365hkbZzGtdyDbNz80= 2023-07-29T00:40:07.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BY9WRn9Sg365hkbZzGtdyDbNz80=', 'Content-Length': '0'} 2023-07-29T00:40:07.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000760f1e4bdce22c4-0064c45fe7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-07-29T00:40:07.360 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:40:07.360 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:40:07.360 DEBUG:boto:path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:40:07.361 DEBUG:boto:auth_path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:40:07.361 DEBUG:boto:Method: GET 2023-07-29T00:40:07.362 DEBUG:boto:Path: /hiiqos-59/obj92?versionId=null 2023-07-29T00:40:07.362 DEBUG:boto:Data: 2023-07-29T00:40:07.362 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.363 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:07.363 DEBUG:boto:Port: 8001 2023-07-29T00:40:07.363 DEBUG:boto:Params: {} 2023-07-29T00:40:07.364 DEBUG:boto:Token: None 2023-07-29T00:40:07.364 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/obj92?versionId=null 2023-07-29T00:40:07.365 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BXaS5OVEPN5bTUEXHGncLNPbrAM= 2023-07-29T00:40:07.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BXaS5OVEPN5bTUEXHGncLNPbrAM=', 'Content-Length': '0'} 2023-07-29T00:40:07.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006efce38afd8ba144-0064c45fe7-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.367 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:40:07.368 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:40:07.368 DEBUG:boto:path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:40:07.369 DEBUG:boto:auth_path=/hiiqos-59/obj92?versionId=null 2023-07-29T00:40:07.369 DEBUG:boto:Method: GET 2023-07-29T00:40:07.369 DEBUG:boto:Path: /hiiqos-59/obj92?versionId=null 2023-07-29T00:40:07.370 DEBUG:boto:Data: 2023-07-29T00:40:07.370 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.371 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:07.371 DEBUG:boto:Port: 8000 2023-07-29T00:40:07.371 DEBUG:boto:Params: {} 2023-07-29T00:40:07.372 DEBUG:boto:Token: None 2023-07-29T00:40:07.372 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/obj92?versionId=null 2023-07-29T00:40:07.373 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BXaS5OVEPN5bTUEXHGncLNPbrAM= 2023-07-29T00:40:07.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BXaS5OVEPN5bTUEXHGncLNPbrAM=', 'Content-Length': '0'} 2023-07-29T00:40:07.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057253fddc0747028-0064c45fe7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.376 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:07.376 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:07.377 DEBUG:boto:path=/hiiqos-59/?prefix=obj92 2023-07-29T00:40:07.377 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj92 2023-07-29T00:40:07.377 DEBUG:boto:Method: GET 2023-07-29T00:40:07.378 DEBUG:boto:Path: /hiiqos-59/?prefix=obj92 2023-07-29T00:40:07.378 DEBUG:boto:Data: 2023-07-29T00:40:07.378 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.379 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:07.379 DEBUG:boto:Port: 8001 2023-07-29T00:40:07.380 DEBUG:boto:Params: {} 2023-07-29T00:40:07.380 DEBUG:boto:Token: None 2023-07-29T00:40:07.381 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/ 2023-07-29T00:40:07.381 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:a29jdrQmja9yUKofwSodYQGXo5w= 2023-07-29T00:40:07.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:a29jdrQmja9yUKofwSodYQGXo5w=', 'Content-Length': '0'} 2023-07-29T00:40:07.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009866f8593edbfe3d-0064c45fe7-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.384 DEBUG:boto:b'hiiqos-59obj921000falseobj922023-07-29T00:33:06.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:07.385 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:07.385 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:07.386 DEBUG:boto:path=/hiiqos-59/?prefix=obj92 2023-07-29T00:40:07.386 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj92 2023-07-29T00:40:07.386 DEBUG:boto:Method: GET 2023-07-29T00:40:07.387 DEBUG:boto:Path: /hiiqos-59/?prefix=obj92 2023-07-29T00:40:07.387 DEBUG:boto:Data: 2023-07-29T00:40:07.387 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.388 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:07.388 DEBUG:boto:Port: 8000 2023-07-29T00:40:07.389 DEBUG:boto:Params: {} 2023-07-29T00:40:07.389 DEBUG:boto:Token: None 2023-07-29T00:40:07.389 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/ 2023-07-29T00:40:07.390 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:a29jdrQmja9yUKofwSodYQGXo5w= 2023-07-29T00:40:07.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:a29jdrQmja9yUKofwSodYQGXo5w=', 'Content-Length': '0'} 2023-07-29T00:40:07.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000603e3529273103d0-0064c45fe7-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.395 DEBUG:boto:b'hiiqos-59obj921000falseobj922023-07-29T00:33:06.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:07.396 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:40:07.396 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:40:07.396 DEBUG:boto:Method: HEAD 2023-07-29T00:40:07.397 DEBUG:boto:Path: /hiiqos-59/obj92 2023-07-29T00:40:07.397 DEBUG:boto:Data: 2023-07-29T00:40:07.398 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.398 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:07.398 DEBUG:boto:Port: 8001 2023-07-29T00:40:07.399 DEBUG:boto:Params: {} 2023-07-29T00:40:07.399 DEBUG:boto:Token: None 2023-07-29T00:40:07.399 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/obj92 2023-07-29T00:40:07.400 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:50gsbY9Ofnu1Go8RSG+BuDWHCUo= 2023-07-29T00:40:07.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:50gsbY9Ofnu1Go8RSG+BuDWHCUo=', 'Content-Length': '0'} 2023-07-29T00:40:07.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000873e3e9bf20a850-0064c45fe7-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.403 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:40:07.403 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:40:07.404 DEBUG:boto:Method: HEAD 2023-07-29T00:40:07.404 DEBUG:boto:Path: /hiiqos-59/obj92 2023-07-29T00:40:07.405 DEBUG:boto:Data: 2023-07-29T00:40:07.405 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.405 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:07.406 DEBUG:boto:Port: 8000 2023-07-29T00:40:07.406 DEBUG:boto:Params: {} 2023-07-29T00:40:07.406 DEBUG:boto:Token: None 2023-07-29T00:40:07.407 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/obj92 2023-07-29T00:40:07.408 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:50gsbY9Ofnu1Go8RSG+BuDWHCUo= 2023-07-29T00:40:07.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:50gsbY9Ofnu1Go8RSG+BuDWHCUo=', 'Content-Length': '0'} 2023-07-29T00:40:07.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff077b540c49bba8-0064c45fe7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-07-29T00:40:07.411 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:40:07.412 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:40:07.412 DEBUG:boto:Method: GET 2023-07-29T00:40:07.413 DEBUG:boto:Path: /hiiqos-59/obj92 2023-07-29T00:40:07.413 DEBUG:boto:Data: 2023-07-29T00:40:07.413 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.414 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:07.414 DEBUG:boto:Port: 8001 2023-07-29T00:40:07.414 DEBUG:boto:Params: {} 2023-07-29T00:40:07.415 DEBUG:boto:Token: None 2023-07-29T00:40:07.415 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/obj92 2023-07-29T00:40:07.416 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OBoF86x5ikI2uO8+6PLRbgUsguE= 2023-07-29T00:40:07.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OBoF86x5ikI2uO8+6PLRbgUsguE=', 'Content-Length': '0'} 2023-07-29T00:40:07.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e09dcc6fad3ddf43-0064c45fe7-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.419 DEBUG:boto:path=/hiiqos-59/obj92 2023-07-29T00:40:07.419 DEBUG:boto:auth_path=/hiiqos-59/obj92 2023-07-29T00:40:07.420 DEBUG:boto:Method: GET 2023-07-29T00:40:07.420 DEBUG:boto:Path: /hiiqos-59/obj92 2023-07-29T00:40:07.420 DEBUG:boto:Data: 2023-07-29T00:40:07.421 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.421 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:07.422 DEBUG:boto:Port: 8000 2023-07-29T00:40:07.422 DEBUG:boto:Params: {} 2023-07-29T00:40:07.422 DEBUG:boto:Token: None 2023-07-29T00:40:07.423 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/obj92 2023-07-29T00:40:07.423 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OBoF86x5ikI2uO8+6PLRbgUsguE= 2023-07-29T00:40:07.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OBoF86x5ikI2uO8+6PLRbgUsguE=', 'Content-Length': '0'} 2023-07-29T00:40:07.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:33:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cab61efdca86e05-0064c45fe7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.426 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:07.427 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:07.427 DEBUG:boto:path=/hiiqos-59/?prefix=obj92 2023-07-29T00:40:07.428 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj92 2023-07-29T00:40:07.428 DEBUG:boto:Method: GET 2023-07-29T00:40:07.429 DEBUG:boto:Path: /hiiqos-59/?prefix=obj92 2023-07-29T00:40:07.429 DEBUG:boto:Data: 2023-07-29T00:40:07.429 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.430 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:07.430 DEBUG:boto:Port: 8001 2023-07-29T00:40:07.430 DEBUG:boto:Params: {} 2023-07-29T00:40:07.431 DEBUG:boto:Token: None 2023-07-29T00:40:07.431 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/ 2023-07-29T00:40:07.432 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:a29jdrQmja9yUKofwSodYQGXo5w= 2023-07-29T00:40:07.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:a29jdrQmja9yUKofwSodYQGXo5w=', 'Content-Length': '0'} 2023-07-29T00:40:07.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000091e44f7200a9f66-0064c45fe7-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.435 DEBUG:boto:b'hiiqos-59obj921000falseobj922023-07-29T00:33:06.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:07.436 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:07.437 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:07.437 DEBUG:boto:path=/hiiqos-59/?prefix=obj92 2023-07-29T00:40:07.437 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj92 2023-07-29T00:40:07.438 DEBUG:boto:Method: GET 2023-07-29T00:40:07.438 DEBUG:boto:Path: /hiiqos-59/?prefix=obj92 2023-07-29T00:40:07.438 DEBUG:boto:Data: 2023-07-29T00:40:07.439 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.439 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:07.439 DEBUG:boto:Port: 8000 2023-07-29T00:40:07.440 DEBUG:boto:Params: {} 2023-07-29T00:40:07.440 DEBUG:boto:Token: None 2023-07-29T00:40:07.441 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/ 2023-07-29T00:40:07.441 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:a29jdrQmja9yUKofwSodYQGXo5w= 2023-07-29T00:40:07.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:a29jdrQmja9yUKofwSodYQGXo5w=', 'Content-Length': '0'} 2023-07-29T00:40:07.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034e6180288ccf1c7-0064c45fe7-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.447 DEBUG:boto:b'hiiqos-59obj921000falseobj922023-07-29T00:33:06.875Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:07.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-07-29T00:40:07.448 DEBUG:boto:path=/hiiqos-59/obj93 2023-07-29T00:40:07.448 DEBUG:boto:auth_path=/hiiqos-59/obj93 2023-07-29T00:40:07.448 DEBUG:boto:path=/hiiqos-59/obj93?versionId=null 2023-07-29T00:40:07.449 DEBUG:boto:auth_path=/hiiqos-59/obj93?versionId=null 2023-07-29T00:40:07.449 DEBUG:boto:Method: GET 2023-07-29T00:40:07.449 DEBUG:boto:Path: /hiiqos-59/obj93?versionId=null 2023-07-29T00:40:07.450 DEBUG:boto:Data: 2023-07-29T00:40:07.450 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.450 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:07.451 DEBUG:boto:Port: 8001 2023-07-29T00:40:07.451 DEBUG:boto:Params: {} 2023-07-29T00:40:07.452 DEBUG:boto:Token: None 2023-07-29T00:40:07.452 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/obj93?versionId=null 2023-07-29T00:40:07.453 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SkyEH7TOzlt3MQs6cmGASsLNqWI= 2023-07-29T00:40:07.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SkyEH7TOzlt3MQs6cmGASsLNqWI=', 'Content-Length': '0'} 2023-07-29T00:40:07.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b21cece849ac6da-0064c45fe7-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.457 DEBUG:boto:path=/hiiqos-59/obj93 2023-07-29T00:40:07.457 DEBUG:boto:auth_path=/hiiqos-59/obj93 2023-07-29T00:40:07.458 DEBUG:boto:path=/hiiqos-59/obj93?versionId=null 2023-07-29T00:40:07.458 DEBUG:boto:auth_path=/hiiqos-59/obj93?versionId=null 2023-07-29T00:40:07.459 DEBUG:boto:Method: GET 2023-07-29T00:40:07.459 DEBUG:boto:Path: /hiiqos-59/obj93?versionId=null 2023-07-29T00:40:07.460 DEBUG:boto:Data: 2023-07-29T00:40:07.460 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.460 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:07.461 DEBUG:boto:Port: 8000 2023-07-29T00:40:07.461 DEBUG:boto:Params: {} 2023-07-29T00:40:07.462 DEBUG:boto:Token: None 2023-07-29T00:40:07.462 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/obj93?versionId=null 2023-07-29T00:40:07.463 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SkyEH7TOzlt3MQs6cmGASsLNqWI= 2023-07-29T00:40:07.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SkyEH7TOzlt3MQs6cmGASsLNqWI=', 'Content-Length': '0'} 2023-07-29T00:40:07.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000104ecdc30a1295be-0064c45fe7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.508 DEBUG:boto:path=/hiiqos-59/obj93 2023-07-29T00:40:07.509 DEBUG:boto:auth_path=/hiiqos-59/obj93 2023-07-29T00:40:07.510 DEBUG:boto:path=/hiiqos-59/obj93?versionId=null 2023-07-29T00:40:07.510 DEBUG:boto:auth_path=/hiiqos-59/obj93?versionId=null 2023-07-29T00:40:07.510 DEBUG:boto:Method: HEAD 2023-07-29T00:40:07.511 DEBUG:boto:Path: /hiiqos-59/obj93?versionId=null 2023-07-29T00:40:07.511 DEBUG:boto:Data: 2023-07-29T00:40:07.511 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.512 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:07.512 DEBUG:boto:Port: 8001 2023-07-29T00:40:07.512 DEBUG:boto:Params: {} 2023-07-29T00:40:07.513 DEBUG:boto:Token: None 2023-07-29T00:40:07.513 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/obj93?versionId=null 2023-07-29T00:40:07.514 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:al4j3XmcU4Ys2JL/KTNML/WeTxc= 2023-07-29T00:40:07.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:al4j3XmcU4Ys2JL/KTNML/WeTxc=', 'Content-Length': '0'} 2023-07-29T00:40:07.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f9b52b39e38661a-0064c45fe7-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.517 DEBUG:boto:path=/hiiqos-59/obj93 2023-07-29T00:40:07.517 DEBUG:boto:auth_path=/hiiqos-59/obj93 2023-07-29T00:40:07.518 DEBUG:boto:path=/hiiqos-59/obj93?versionId=null 2023-07-29T00:40:07.518 DEBUG:boto:auth_path=/hiiqos-59/obj93?versionId=null 2023-07-29T00:40:07.518 DEBUG:boto:Method: HEAD 2023-07-29T00:40:07.519 DEBUG:boto:Path: /hiiqos-59/obj93?versionId=null 2023-07-29T00:40:07.519 DEBUG:boto:Data: 2023-07-29T00:40:07.519 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.520 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:07.520 DEBUG:boto:Port: 8000 2023-07-29T00:40:07.520 DEBUG:boto:Params: {} 2023-07-29T00:40:07.521 DEBUG:boto:Token: None 2023-07-29T00:40:07.521 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/obj93?versionId=null 2023-07-29T00:40:07.521 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:al4j3XmcU4Ys2JL/KTNML/WeTxc= 2023-07-29T00:40:07.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:al4j3XmcU4Ys2JL/KTNML/WeTxc=', 'Content-Length': '0'} 2023-07-29T00:40:07.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c310ce5dfc7919bf-0064c45fe7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-07-29T00:40:07.525 DEBUG:boto:path=/hiiqos-59/obj93 2023-07-29T00:40:07.525 DEBUG:boto:auth_path=/hiiqos-59/obj93 2023-07-29T00:40:07.526 DEBUG:boto:path=/hiiqos-59/obj93?versionId=null 2023-07-29T00:40:07.526 DEBUG:boto:auth_path=/hiiqos-59/obj93?versionId=null 2023-07-29T00:40:07.526 DEBUG:boto:Method: GET 2023-07-29T00:40:07.527 DEBUG:boto:Path: /hiiqos-59/obj93?versionId=null 2023-07-29T00:40:07.527 DEBUG:boto:Data: 2023-07-29T00:40:07.528 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.528 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:07.528 DEBUG:boto:Port: 8001 2023-07-29T00:40:07.529 DEBUG:boto:Params: {} 2023-07-29T00:40:07.529 DEBUG:boto:Token: None 2023-07-29T00:40:07.529 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/obj93?versionId=null 2023-07-29T00:40:07.530 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SkyEH7TOzlt3MQs6cmGASsLNqWI= 2023-07-29T00:40:07.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SkyEH7TOzlt3MQs6cmGASsLNqWI=', 'Content-Length': '0'} 2023-07-29T00:40:07.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000337fb4a93c5dad9e-0064c45fe7-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.533 DEBUG:boto:path=/hiiqos-59/obj93 2023-07-29T00:40:07.533 DEBUG:boto:auth_path=/hiiqos-59/obj93 2023-07-29T00:40:07.534 DEBUG:boto:path=/hiiqos-59/obj93?versionId=null 2023-07-29T00:40:07.534 DEBUG:boto:auth_path=/hiiqos-59/obj93?versionId=null 2023-07-29T00:40:07.534 DEBUG:boto:Method: GET 2023-07-29T00:40:07.535 DEBUG:boto:Path: /hiiqos-59/obj93?versionId=null 2023-07-29T00:40:07.535 DEBUG:boto:Data: 2023-07-29T00:40:07.536 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.536 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:07.536 DEBUG:boto:Port: 8000 2023-07-29T00:40:07.537 DEBUG:boto:Params: {} 2023-07-29T00:40:07.537 DEBUG:boto:Token: None 2023-07-29T00:40:07.538 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/obj93?versionId=null 2023-07-29T00:40:07.538 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SkyEH7TOzlt3MQs6cmGASsLNqWI= 2023-07-29T00:40:07.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SkyEH7TOzlt3MQs6cmGASsLNqWI=', 'Content-Length': '0'} 2023-07-29T00:40:07.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000041dbb706782c43e-0064c45fe7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.541 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:07.541 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:07.542 DEBUG:boto:path=/hiiqos-59/?prefix=obj93 2023-07-29T00:40:07.542 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj93 2023-07-29T00:40:07.542 DEBUG:boto:Method: GET 2023-07-29T00:40:07.543 DEBUG:boto:Path: /hiiqos-59/?prefix=obj93 2023-07-29T00:40:07.543 DEBUG:boto:Data: 2023-07-29T00:40:07.544 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.544 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:07.544 DEBUG:boto:Port: 8001 2023-07-29T00:40:07.545 DEBUG:boto:Params: {} 2023-07-29T00:40:07.545 DEBUG:boto:Token: None 2023-07-29T00:40:07.546 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/ 2023-07-29T00:40:07.546 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:a29jdrQmja9yUKofwSodYQGXo5w= 2023-07-29T00:40:07.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:a29jdrQmja9yUKofwSodYQGXo5w=', 'Content-Length': '0'} 2023-07-29T00:40:07.549 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2a4f2f38c039951-0064c45fe7-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.592 DEBUG:boto:b'hiiqos-59obj931000falseobj932023-07-29T00:35:31.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:07.593 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:07.594 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:07.594 DEBUG:boto:path=/hiiqos-59/?prefix=obj93 2023-07-29T00:40:07.594 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj93 2023-07-29T00:40:07.595 DEBUG:boto:Method: GET 2023-07-29T00:40:07.595 DEBUG:boto:Path: /hiiqos-59/?prefix=obj93 2023-07-29T00:40:07.595 DEBUG:boto:Data: 2023-07-29T00:40:07.596 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.596 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:07.597 DEBUG:boto:Port: 8000 2023-07-29T00:40:07.597 DEBUG:boto:Params: {} 2023-07-29T00:40:07.597 DEBUG:boto:Token: None 2023-07-29T00:40:07.598 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/ 2023-07-29T00:40:07.598 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:a29jdrQmja9yUKofwSodYQGXo5w= 2023-07-29T00:40:07.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:a29jdrQmja9yUKofwSodYQGXo5w=', 'Content-Length': '0'} 2023-07-29T00:40:07.603 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c42d3b7eda8847c7-0064c45fe7-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.603 DEBUG:boto:b'hiiqos-59obj931000falseobj932023-07-29T00:35:31.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:07.604 DEBUG:boto:path=/hiiqos-59/obj93 2023-07-29T00:40:07.604 DEBUG:boto:auth_path=/hiiqos-59/obj93 2023-07-29T00:40:07.605 DEBUG:boto:Method: HEAD 2023-07-29T00:40:07.605 DEBUG:boto:Path: /hiiqos-59/obj93 2023-07-29T00:40:07.605 DEBUG:boto:Data: 2023-07-29T00:40:07.606 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.606 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:07.606 DEBUG:boto:Port: 8001 2023-07-29T00:40:07.607 DEBUG:boto:Params: {} 2023-07-29T00:40:07.607 DEBUG:boto:Token: None 2023-07-29T00:40:07.607 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/obj93 2023-07-29T00:40:07.608 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AtZSH2LZwE6qTMpGheE5f+yAqcw= 2023-07-29T00:40:07.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AtZSH2LZwE6qTMpGheE5f+yAqcw=', 'Content-Length': '0'} 2023-07-29T00:40:07.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067df023f249e95ff-0064c45fe7-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.611 DEBUG:boto:path=/hiiqos-59/obj93 2023-07-29T00:40:07.611 DEBUG:boto:auth_path=/hiiqos-59/obj93 2023-07-29T00:40:07.612 DEBUG:boto:Method: HEAD 2023-07-29T00:40:07.612 DEBUG:boto:Path: /hiiqos-59/obj93 2023-07-29T00:40:07.612 DEBUG:boto:Data: 2023-07-29T00:40:07.613 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.613 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:07.613 DEBUG:boto:Port: 8000 2023-07-29T00:40:07.614 DEBUG:boto:Params: {} 2023-07-29T00:40:07.614 DEBUG:boto:Token: None 2023-07-29T00:40:07.614 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/obj93 2023-07-29T00:40:07.615 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AtZSH2LZwE6qTMpGheE5f+yAqcw= 2023-07-29T00:40:07.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AtZSH2LZwE6qTMpGheE5f+yAqcw=', 'Content-Length': '0'} 2023-07-29T00:40:07.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fda2e603acee6d8-0064c45fe7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-07-29T00:40:07.618 DEBUG:boto:path=/hiiqos-59/obj93 2023-07-29T00:40:07.619 DEBUG:boto:auth_path=/hiiqos-59/obj93 2023-07-29T00:40:07.619 DEBUG:boto:Method: GET 2023-07-29T00:40:07.619 DEBUG:boto:Path: /hiiqos-59/obj93 2023-07-29T00:40:07.620 DEBUG:boto:Data: 2023-07-29T00:40:07.620 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.620 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:07.621 DEBUG:boto:Port: 8001 2023-07-29T00:40:07.621 DEBUG:boto:Params: {} 2023-07-29T00:40:07.621 DEBUG:boto:Token: None 2023-07-29T00:40:07.622 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/obj93 2023-07-29T00:40:07.622 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PlKhQW6oVnZ4E36o/nAWrRFN/to= 2023-07-29T00:40:07.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PlKhQW6oVnZ4E36o/nAWrRFN/to=', 'Content-Length': '0'} 2023-07-29T00:40:07.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f059cf576f2b996-0064c45fe7-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.625 DEBUG:boto:path=/hiiqos-59/obj93 2023-07-29T00:40:07.626 DEBUG:boto:auth_path=/hiiqos-59/obj93 2023-07-29T00:40:07.626 DEBUG:boto:Method: GET 2023-07-29T00:40:07.626 DEBUG:boto:Path: /hiiqos-59/obj93 2023-07-29T00:40:07.627 DEBUG:boto:Data: 2023-07-29T00:40:07.627 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.627 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:07.628 DEBUG:boto:Port: 8000 2023-07-29T00:40:07.628 DEBUG:boto:Params: {} 2023-07-29T00:40:07.629 DEBUG:boto:Token: None 2023-07-29T00:40:07.629 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/obj93 2023-07-29T00:40:07.629 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PlKhQW6oVnZ4E36o/nAWrRFN/to= 2023-07-29T00:40:07.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PlKhQW6oVnZ4E36o/nAWrRFN/to=', 'Content-Length': '0'} 2023-07-29T00:40:07.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:35:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009147bdd2b4e1f6e8-0064c45fe7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.676 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:07.677 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:07.677 DEBUG:boto:path=/hiiqos-59/?prefix=obj93 2023-07-29T00:40:07.678 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj93 2023-07-29T00:40:07.678 DEBUG:boto:Method: GET 2023-07-29T00:40:07.679 DEBUG:boto:Path: /hiiqos-59/?prefix=obj93 2023-07-29T00:40:07.679 DEBUG:boto:Data: 2023-07-29T00:40:07.679 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.680 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:07.680 DEBUG:boto:Port: 8001 2023-07-29T00:40:07.680 DEBUG:boto:Params: {} 2023-07-29T00:40:07.681 DEBUG:boto:Token: None 2023-07-29T00:40:07.681 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/ 2023-07-29T00:40:07.682 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:a29jdrQmja9yUKofwSodYQGXo5w= 2023-07-29T00:40:07.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:a29jdrQmja9yUKofwSodYQGXo5w=', 'Content-Length': '0'} 2023-07-29T00:40:07.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9d7a0748f72cafd-0064c45fe7-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.686 DEBUG:boto:b'hiiqos-59obj931000falseobj932023-07-29T00:35:31.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:07.686 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:07.687 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:07.687 DEBUG:boto:path=/hiiqos-59/?prefix=obj93 2023-07-29T00:40:07.687 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj93 2023-07-29T00:40:07.688 DEBUG:boto:Method: GET 2023-07-29T00:40:07.688 DEBUG:boto:Path: /hiiqos-59/?prefix=obj93 2023-07-29T00:40:07.689 DEBUG:boto:Data: 2023-07-29T00:40:07.689 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.689 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:07.690 DEBUG:boto:Port: 8000 2023-07-29T00:40:07.690 DEBUG:boto:Params: {} 2023-07-29T00:40:07.691 DEBUG:boto:Token: None 2023-07-29T00:40:07.691 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/ 2023-07-29T00:40:07.691 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:a29jdrQmja9yUKofwSodYQGXo5w= 2023-07-29T00:40:07.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:a29jdrQmja9yUKofwSodYQGXo5w=', 'Content-Length': '0'} 2023-07-29T00:40:07.697 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ae569dacb7c17f4-0064c45fe7-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.697 DEBUG:boto:b'hiiqos-59obj931000falseobj932023-07-29T00:35:31.844Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:07.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-07-29T00:40:07.699 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:40:07.699 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:40:07.699 DEBUG:boto:path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:40:07.700 DEBUG:boto:auth_path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:40:07.700 DEBUG:boto:Method: GET 2023-07-29T00:40:07.700 DEBUG:boto:Path: /hiiqos-59/obj95?versionId=null 2023-07-29T00:40:07.701 DEBUG:boto:Data: 2023-07-29T00:40:07.701 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.702 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:07.702 DEBUG:boto:Port: 8001 2023-07-29T00:40:07.702 DEBUG:boto:Params: {} 2023-07-29T00:40:07.703 DEBUG:boto:Token: None 2023-07-29T00:40:07.703 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/obj95?versionId=null 2023-07-29T00:40:07.704 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:br7/9mL0fca2c7WZWIF6LrBN15E= 2023-07-29T00:40:07.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:br7/9mL0fca2c7WZWIF6LrBN15E=', 'Content-Length': '0'} 2023-07-29T00:40:07.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094fb77a7beedf2ec-0064c45fe7-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.707 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:40:07.707 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:40:07.708 DEBUG:boto:path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:40:07.708 DEBUG:boto:auth_path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:40:07.708 DEBUG:boto:Method: GET 2023-07-29T00:40:07.709 DEBUG:boto:Path: /hiiqos-59/obj95?versionId=null 2023-07-29T00:40:07.709 DEBUG:boto:Data: 2023-07-29T00:40:07.709 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.710 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:07.710 DEBUG:boto:Port: 8000 2023-07-29T00:40:07.710 DEBUG:boto:Params: {} 2023-07-29T00:40:07.711 DEBUG:boto:Token: None 2023-07-29T00:40:07.711 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/obj95?versionId=null 2023-07-29T00:40:07.712 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:br7/9mL0fca2c7WZWIF6LrBN15E= 2023-07-29T00:40:07.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:br7/9mL0fca2c7WZWIF6LrBN15E=', 'Content-Length': '0'} 2023-07-29T00:40:07.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ebf7f2a616ee991-0064c45fe7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.715 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:40:07.715 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:40:07.716 DEBUG:boto:path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:40:07.716 DEBUG:boto:auth_path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:40:07.716 DEBUG:boto:Method: HEAD 2023-07-29T00:40:07.717 DEBUG:boto:Path: /hiiqos-59/obj95?versionId=null 2023-07-29T00:40:07.717 DEBUG:boto:Data: 2023-07-29T00:40:07.717 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.718 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:07.718 DEBUG:boto:Port: 8001 2023-07-29T00:40:07.718 DEBUG:boto:Params: {} 2023-07-29T00:40:07.719 DEBUG:boto:Token: None 2023-07-29T00:40:07.719 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/obj95?versionId=null 2023-07-29T00:40:07.719 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0y3C1Y7Y7F1C0jhGdB9/kJBeJJA= 2023-07-29T00:40:07.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0y3C1Y7Y7F1C0jhGdB9/kJBeJJA=', 'Content-Length': '0'} 2023-07-29T00:40:07.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2b0041518faa122-0064c45fe7-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.722 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:40:07.723 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:40:07.723 DEBUG:boto:path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:40:07.724 DEBUG:boto:auth_path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:40:07.724 DEBUG:boto:Method: HEAD 2023-07-29T00:40:07.724 DEBUG:boto:Path: /hiiqos-59/obj95?versionId=null 2023-07-29T00:40:07.725 DEBUG:boto:Data: 2023-07-29T00:40:07.725 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.725 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:07.726 DEBUG:boto:Port: 8000 2023-07-29T00:40:07.726 DEBUG:boto:Params: {} 2023-07-29T00:40:07.726 DEBUG:boto:Token: None 2023-07-29T00:40:07.727 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/obj95?versionId=null 2023-07-29T00:40:07.727 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0y3C1Y7Y7F1C0jhGdB9/kJBeJJA= 2023-07-29T00:40:07.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0y3C1Y7Y7F1C0jhGdB9/kJBeJJA=', 'Content-Length': '0'} 2023-07-29T00:40:07.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8380f9e7036700f-0064c45fe7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-07-29T00:40:07.730 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:40:07.731 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:40:07.731 DEBUG:boto:path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:40:07.732 DEBUG:boto:auth_path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:40:07.732 DEBUG:boto:Method: GET 2023-07-29T00:40:07.732 DEBUG:boto:Path: /hiiqos-59/obj95?versionId=null 2023-07-29T00:40:07.733 DEBUG:boto:Data: 2023-07-29T00:40:07.733 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.733 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:07.734 DEBUG:boto:Port: 8001 2023-07-29T00:40:07.734 DEBUG:boto:Params: {} 2023-07-29T00:40:07.735 DEBUG:boto:Token: None 2023-07-29T00:40:07.735 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/obj95?versionId=null 2023-07-29T00:40:07.735 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:br7/9mL0fca2c7WZWIF6LrBN15E= 2023-07-29T00:40:07.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:br7/9mL0fca2c7WZWIF6LrBN15E=', 'Content-Length': '0'} 2023-07-29T00:40:07.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fd7b56f3633fa0c-0064c45fe7-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.739 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:40:07.739 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:40:07.739 DEBUG:boto:path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:40:07.740 DEBUG:boto:auth_path=/hiiqos-59/obj95?versionId=null 2023-07-29T00:40:07.740 DEBUG:boto:Method: GET 2023-07-29T00:40:07.741 DEBUG:boto:Path: /hiiqos-59/obj95?versionId=null 2023-07-29T00:40:07.741 DEBUG:boto:Data: 2023-07-29T00:40:07.742 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.742 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:07.742 DEBUG:boto:Port: 8000 2023-07-29T00:40:07.743 DEBUG:boto:Params: {} 2023-07-29T00:40:07.743 DEBUG:boto:Token: None 2023-07-29T00:40:07.744 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/obj95?versionId=null 2023-07-29T00:40:07.744 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:br7/9mL0fca2c7WZWIF6LrBN15E= 2023-07-29T00:40:07.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:br7/9mL0fca2c7WZWIF6LrBN15E=', 'Content-Length': '0'} 2023-07-29T00:40:07.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6253dc05560753b-0064c45fe7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.747 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:07.747 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:07.748 DEBUG:boto:path=/hiiqos-59/?prefix=obj95 2023-07-29T00:40:07.748 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj95 2023-07-29T00:40:07.749 DEBUG:boto:Method: GET 2023-07-29T00:40:07.749 DEBUG:boto:Path: /hiiqos-59/?prefix=obj95 2023-07-29T00:40:07.749 DEBUG:boto:Data: 2023-07-29T00:40:07.750 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.750 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:07.750 DEBUG:boto:Port: 8001 2023-07-29T00:40:07.751 DEBUG:boto:Params: {} 2023-07-29T00:40:07.751 DEBUG:boto:Token: None 2023-07-29T00:40:07.751 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/ 2023-07-29T00:40:07.752 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:a29jdrQmja9yUKofwSodYQGXo5w= 2023-07-29T00:40:07.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:a29jdrQmja9yUKofwSodYQGXo5w=', 'Content-Length': '0'} 2023-07-29T00:40:07.755 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000389d7a6473278beb-0064c45fe7-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.755 DEBUG:boto:b'hiiqos-59obj951000falseobj952023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:07.756 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:07.757 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:07.757 DEBUG:boto:path=/hiiqos-59/?prefix=obj95 2023-07-29T00:40:07.757 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj95 2023-07-29T00:40:07.758 DEBUG:boto:Method: GET 2023-07-29T00:40:07.758 DEBUG:boto:Path: /hiiqos-59/?prefix=obj95 2023-07-29T00:40:07.758 DEBUG:boto:Data: 2023-07-29T00:40:07.759 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.759 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:07.759 DEBUG:boto:Port: 8000 2023-07-29T00:40:07.760 DEBUG:boto:Params: {} 2023-07-29T00:40:07.760 DEBUG:boto:Token: None 2023-07-29T00:40:07.761 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/ 2023-07-29T00:40:07.761 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:a29jdrQmja9yUKofwSodYQGXo5w= 2023-07-29T00:40:07.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:a29jdrQmja9yUKofwSodYQGXo5w=', 'Content-Length': '0'} 2023-07-29T00:40:07.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007baef98dca499428-0064c45fe7-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.766 DEBUG:boto:b'hiiqos-59obj951000falseobj952023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:07.767 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:40:07.767 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:40:07.768 DEBUG:boto:Method: HEAD 2023-07-29T00:40:07.768 DEBUG:boto:Path: /hiiqos-59/obj95 2023-07-29T00:40:07.768 DEBUG:boto:Data: 2023-07-29T00:40:07.769 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.769 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:07.769 DEBUG:boto:Port: 8001 2023-07-29T00:40:07.770 DEBUG:boto:Params: {} 2023-07-29T00:40:07.770 DEBUG:boto:Token: None 2023-07-29T00:40:07.771 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/obj95 2023-07-29T00:40:07.771 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3oQcekSSpPEk19e3TOvLVwzRTlM= 2023-07-29T00:40:07.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3oQcekSSpPEk19e3TOvLVwzRTlM=', 'Content-Length': '0'} 2023-07-29T00:40:07.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b33f8980d3bd8757-0064c45fe7-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.774 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:40:07.775 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:40:07.775 DEBUG:boto:Method: HEAD 2023-07-29T00:40:07.775 DEBUG:boto:Path: /hiiqos-59/obj95 2023-07-29T00:40:07.776 DEBUG:boto:Data: 2023-07-29T00:40:07.776 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.776 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:07.777 DEBUG:boto:Port: 8000 2023-07-29T00:40:07.777 DEBUG:boto:Params: {} 2023-07-29T00:40:07.777 DEBUG:boto:Token: None 2023-07-29T00:40:07.778 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/obj95 2023-07-29T00:40:07.778 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3oQcekSSpPEk19e3TOvLVwzRTlM= 2023-07-29T00:40:07.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3oQcekSSpPEk19e3TOvLVwzRTlM=', 'Content-Length': '0'} 2023-07-29T00:40:07.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013ecbc9d66f3741d-0064c45fe7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-07-29T00:40:07.782 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:40:07.782 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:40:07.782 DEBUG:boto:Method: GET 2023-07-29T00:40:07.783 DEBUG:boto:Path: /hiiqos-59/obj95 2023-07-29T00:40:07.783 DEBUG:boto:Data: 2023-07-29T00:40:07.783 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.784 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:07.784 DEBUG:boto:Port: 8001 2023-07-29T00:40:07.785 DEBUG:boto:Params: {} 2023-07-29T00:40:07.785 DEBUG:boto:Token: None 2023-07-29T00:40:07.785 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/obj95 2023-07-29T00:40:07.786 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FCbLWQmoY4zaMS0FQO9xrc4yjgc= 2023-07-29T00:40:07.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FCbLWQmoY4zaMS0FQO9xrc4yjgc=', 'Content-Length': '0'} 2023-07-29T00:40:07.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-59:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f192d0787521b642-0064c45fe7-1eb0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.789 DEBUG:boto:path=/hiiqos-59/obj95 2023-07-29T00:40:07.789 DEBUG:boto:auth_path=/hiiqos-59/obj95 2023-07-29T00:40:07.789 DEBUG:boto:Method: GET 2023-07-29T00:40:07.790 DEBUG:boto:Path: /hiiqos-59/obj95 2023-07-29T00:40:07.790 DEBUG:boto:Data: 2023-07-29T00:40:07.791 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.791 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:07.791 DEBUG:boto:Port: 8000 2023-07-29T00:40:07.792 DEBUG:boto:Params: {} 2023-07-29T00:40:07.792 DEBUG:boto:Token: None 2023-07-29T00:40:07.792 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/obj95 2023-07-29T00:40:07.793 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FCbLWQmoY4zaMS0FQO9xrc4yjgc= 2023-07-29T00:40:07.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FCbLWQmoY4zaMS0FQO9xrc4yjgc=', 'Content-Length': '0'} 2023-07-29T00:40:07.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:29:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c69275b2589a227-0064c45fe7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.836 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:07.837 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:07.837 DEBUG:boto:path=/hiiqos-59/?prefix=obj95 2023-07-29T00:40:07.838 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj95 2023-07-29T00:40:07.838 DEBUG:boto:Method: GET 2023-07-29T00:40:07.839 DEBUG:boto:Path: /hiiqos-59/?prefix=obj95 2023-07-29T00:40:07.839 DEBUG:boto:Data: 2023-07-29T00:40:07.839 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.840 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:40:07.840 DEBUG:boto:Port: 8001 2023-07-29T00:40:07.840 DEBUG:boto:Params: {} 2023-07-29T00:40:07.841 DEBUG:boto:Token: None 2023-07-29T00:40:07.841 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/ 2023-07-29T00:40:07.842 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:a29jdrQmja9yUKofwSodYQGXo5w= 2023-07-29T00:40:07.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:a29jdrQmja9yUKofwSodYQGXo5w=', 'Content-Length': '0'} 2023-07-29T00:40:07.845 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b3e93d265bcd689-0064c45fe7-1eb0-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.845 DEBUG:boto:b'hiiqos-59obj951000falseobj952023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:07.846 DEBUG:boto:path=/hiiqos-59/ 2023-07-29T00:40:07.846 DEBUG:boto:auth_path=/hiiqos-59/ 2023-07-29T00:40:07.847 DEBUG:boto:path=/hiiqos-59/?prefix=obj95 2023-07-29T00:40:07.847 DEBUG:boto:auth_path=/hiiqos-59/?prefix=obj95 2023-07-29T00:40:07.847 DEBUG:boto:Method: GET 2023-07-29T00:40:07.848 DEBUG:boto:Path: /hiiqos-59/?prefix=obj95 2023-07-29T00:40:07.848 DEBUG:boto:Data: 2023-07-29T00:40:07.849 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.849 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:40:07.849 DEBUG:boto:Port: 8000 2023-07-29T00:40:07.850 DEBUG:boto:Params: {} 2023-07-29T00:40:07.850 DEBUG:boto:Token: None 2023-07-29T00:40:07.850 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:40:07 GMT /hiiqos-59/ 2023-07-29T00:40:07.851 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:a29jdrQmja9yUKofwSodYQGXo5w= 2023-07-29T00:40:07.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:40:07 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:a29jdrQmja9yUKofwSodYQGXo5w=', 'Content-Length': '0'} 2023-07-29T00:40:07.856 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5954649802148e7-0064c45fe7-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:40:07 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:07.856 DEBUG:boto:b'hiiqos-59obj951000falseobj952023-07-29T00:29:58.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:40:07.857 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-59 zones={a2, a1} 2023-07-29T00:40:07.858 INFO:tasks.rgw_multisite_tests:Create several generations of objects, trash them, then run bucket sync init ... ok 2023-07-29T00:40:07.862 INFO:rgw_multi.tests:create role zone=a1 name=roles-hiiqos-1 2023-07-29T00:40:07.863 DEBUG:boto:Method: POST 2023-07-29T00:40:07.863 DEBUG:boto:Path: / 2023-07-29T00:40:07.863 DEBUG:boto:Data: 2023-07-29T00:40:07.864 DEBUG:boto:Headers: {} 2023-07-29T00:40:07.864 DEBUG:boto:Host: smithi101.front.sepia.ceph.com 2023-07-29T00:40:07.864 DEBUG:boto:Port: 8000 2023-07-29T00:40:07.865 DEBUG:boto:Params: {'RoleName': 'roles-hiiqos-1', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2023-07-29T00:40:07.865 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-29T00:40:07.865 DEBUG:boto:Token: None 2023-07-29T00:40:07.866 DEBUG:boto:CanonicalRequest: POST / host:smithi101.front.sepia.ceph.com:8000 x-amz-date:20230729T004007Z host;x-amz-date 569c330df3dde167eb892f21bb3ab466d7a71eb933071bf019ce8f586ff54b14 2023-07-29T00:40:07.866 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230729T004007Z 20230729/front/smithi101/aws4_request f1387310dfd633cfcbb421993a297c72137a868f4ee0f04f1c23669602e4534d 2023-07-29T00:40:07.867 DEBUG:boto:Signature: 5ab158905da90a9fb6e9cb841364d07fb4499e8859f0d81ca3d1da43a949a586 2023-07-29T00:40:07.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'X-Amz-Date': '20230729T004007Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=S7W2J6K8BFBCG65TLASH/20230729/front/smithi101/aws4_request,SignedHeaders=host;x-amz-date,Signature=5ab158905da90a9fb6e9cb841364d07fb4499e8859f0d81ca3d1da43a949a586', 'Host': 'smithi101.front.sepia.ceph.com:8000'} 2023-07-29T00:40:12.879 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007d29be848a90a86d-0064c45fec-165f-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '723'), ('Date', 'Sat, 29 Jul 2023 00:40:12 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:12.880 DEBUG:boto:b'7d63324a-e189-4107-be76-d86c6ac405b8roles-hiiqos-1/arn:aws:iam:::role/roles-hiiqos-12023-07-29T00:40:12.807Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx000007d29be848a90a86d-0064c45fec-165f-a1' 2023-07-29T00:40:12.884 INFO:rgw_multi.tests:create role zone=a2 name=roles-hiiqos-2 2023-07-29T00:40:12.884 DEBUG:boto:Method: POST 2023-07-29T00:40:12.885 DEBUG:boto:Path: / 2023-07-29T00:40:12.885 DEBUG:boto:Data: 2023-07-29T00:40:12.886 DEBUG:boto:Headers: {} 2023-07-29T00:40:12.886 DEBUG:boto:Host: smithi101.front.sepia.ceph.com 2023-07-29T00:40:12.886 DEBUG:boto:Port: 8001 2023-07-29T00:40:12.887 DEBUG:boto:Params: {'RoleName': 'roles-hiiqos-2', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2023-07-29T00:40:12.887 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-29T00:40:12.888 DEBUG:boto:Token: None 2023-07-29T00:40:12.888 DEBUG:boto:CanonicalRequest: POST / host:smithi101.front.sepia.ceph.com:8001 x-amz-date:20230729T004012Z host;x-amz-date 68c06e6f376ea9028405bcc430985356d4b54d36d784f6efecf57414987becbf 2023-07-29T00:40:12.889 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230729T004012Z 20230729/front/smithi101/aws4_request fd716f8c3a192785f127192e5d237f67dabe91fda8635b1a2b0d018bdf2840a1 2023-07-29T00:40:12.890 DEBUG:boto:Signature: 13eb63161d6793740dc47fbe07cebafcfbe52f77de2e70b83c71c68821c7d047 2023-07-29T00:40:12.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'X-Amz-Date': '20230729T004012Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=S7W2J6K8BFBCG65TLASH/20230729/front/smithi101/aws4_request,SignedHeaders=host;x-amz-date,Signature=13eb63161d6793740dc47fbe07cebafcfbe52f77de2e70b83c71c68821c7d047', 'Host': 'smithi101.front.sepia.ceph.com:8001'} 2023-07-29T00:40:17.911 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000058e20566da1aa68a-0064c45ff1-1eb0-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '723'), ('Date', 'Sat, 29 Jul 2023 00:40:17 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:40:17.912 DEBUG:boto:b'a75d7c6d-3c80-4c31-b763-8905e38b45cdroles-hiiqos-2/arn:aws:iam:::role/roles-hiiqos-22023-07-29T00:40:17.841Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx0000058e20566da1aa68a-0064c45ff1-1eb0-a2' 2023-07-29T00:40:17.913 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:40:17.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:40:17.914 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:40:17.964 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:40:17.965 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:40:17.997 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T00:40:18.011 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T00:40:18.011 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:40:18.012 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T00:40:18.012 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:17.836402Z" 2023-07-29T00:40:18.012 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:18.012 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:40:18.013 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591073.070100_1058.1", 2023-07-29T00:40:18.013 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:37:53.070100Z" 2023-07-29T00:40:18.013 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:18.013 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:40:18.014 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:40:18.014 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T00:40:18.014 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:18.014 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:40:18.014 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T00:40:18.015 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:12.811406Z" 2023-07-29T00:40:18.015 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:18.015 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T00:40:18.016 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690591217.836402_294.1', 1: '1_1690591073.070100_1058.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T00:40:18.017 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T00:40:18.018 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:40:18.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:40:18.019 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:40:18.067 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:40:18.068 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:40:18.096 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:40:18.031+0000 7f58453af880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:40:18.097 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:40:18.032+0000 7f58453af880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:40:18.110 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:40:18.110 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:40:18.110 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:40:18.110 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:40:18.111 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:40:18.111 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:40:18.111 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:18.111 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:18.111 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:40:18.111 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:40:18.112 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:40:18.112 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:40:18.112 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:40:18.112 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:40:18.113 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:40:18.113 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:40:18.113 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:40:18.114 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:40:18.114 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:18.114 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:18.114 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:18.115 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:40:18.115 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:40:18.115 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:40:18.115 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:40:18.115 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591073.070100_1058.1", 2023-07-29T00:40:18.115 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:40:18.116 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:40:18.116 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:40:18.116 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:37:53.070100Z", 2023-07-29T00:40:18.116 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:18.116 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:18.116 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:18.117 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:40:18.117 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:40:18.117 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:40:18.117 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:40:18.117 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:40:18.117 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:40:18.118 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:40:18.118 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:40:18.118 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:40:18.118 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:18.118 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:18.119 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:18.119 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:40:18.119 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:40:18.119 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:40:18.119 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:40:18.119 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:40:18.119 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:40:18.120 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:40:18.120 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:40:18.120 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:40:18.120 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:18.120 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:18.121 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:18.121 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:40:18.121 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:18.121 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:40:18.121 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:40:18.121 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:40:18.121 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:18.122 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:40:18Z" 2023-07-29T00:40:18.122 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:40:18.123 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690591073.070100_1058.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:37:53.070100Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:40:18Z" } 2023-07-29T00:40:18.124 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690591073.070100_1058.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:37:53.070100Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:40:18.124 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690591073.070100_1058.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T00:40:18.125 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690591073.070100_1058.1', 2: '', 3: ''} 2023-07-29T00:40:18.125 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1690591217.836402_294.1 target=1_1690587969.233742_99.1, shard=3 master=1_1690591212.811406_223.1 target= 2023-07-29T00:40:23.126 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:40:23.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:40:23.127 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:40:23.185 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:40:23.186 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:40:23.218 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:40:23.153+0000 7ff63b128880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:40:23.219 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:40:23.154+0000 7ff63b128880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:40:23.233 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:40:23.233 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:40:23.233 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:40:23.234 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:40:23.234 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:40:23.234 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:40:23.234 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:23.234 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:23.234 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:40:23.234 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:40:23.235 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:40:23.235 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:40:23.235 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:40:23.235 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:40:23.235 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:40:23.236 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:40:23.236 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:40:23.236 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:40:23.236 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:23.236 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:23.236 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:23.237 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:40:23.237 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:40:23.237 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:40:23.237 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:40:23.237 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591073.070100_1058.1", 2023-07-29T00:40:23.237 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:40:23.238 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:40:23.238 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:40:23.238 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:37:53.070100Z", 2023-07-29T00:40:23.238 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:23.238 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:23.238 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:23.239 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:40:23.239 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:40:23.239 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:40:23.239 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:40:23.239 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:40:23.240 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:40:23.240 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:40:23.240 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:40:23.240 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:40:23.240 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:23.240 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:23.240 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:23.241 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:40:23.241 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:40:23.241 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:40:23.241 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:40:23.241 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:40:23.241 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:40:23.241 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:40:23.242 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:40:23.242 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:40:23.242 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:23.242 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:23.242 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:23.242 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:40:23.242 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:23.243 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:40:23.243 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:40:23.243 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:40:23.243 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:23.243 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:40:23Z" 2023-07-29T00:40:23.243 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:40:23.245 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690591073.070100_1058.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:37:53.070100Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:40:23Z" } 2023-07-29T00:40:23.245 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690591073.070100_1058.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:37:53.070100Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:40:23.246 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690591073.070100_1058.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T00:40:23.246 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690591073.070100_1058.1', 2: '', 3: ''} 2023-07-29T00:40:23.247 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1690591217.836402_294.1 target=1_1690587969.233742_99.1, shard=3 master=1_1690591212.811406_223.1 target= 2023-07-29T00:40:28.247 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:40:28.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:40:28.248 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:40:28.302 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:40:28.302 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:40:28.336 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:40:28.270+0000 7fb13eda2880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:40:28.336 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:40:28.271+0000 7fb13eda2880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:40:28.352 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:40:28.352 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:40:28.352 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:40:28.352 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:40:28.353 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:40:28.353 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:40:28.353 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:28.353 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:28.353 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:40:28.353 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:40:28.354 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:40:28.354 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:40:28.354 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:40:28.354 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:40:28.354 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:40:28.354 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:40:28.354 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:40:28.355 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:40:28.355 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:28.355 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:28.355 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:28.355 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:40:28.355 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:40:28.355 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:40:28.356 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:40:28.356 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591073.070100_1058.1", 2023-07-29T00:40:28.356 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:40:28.356 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:40:28.356 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:40:28.356 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:37:53.070100Z", 2023-07-29T00:40:28.356 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:28.357 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:28.357 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:28.357 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:40:28.357 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:40:28.357 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:40:28.357 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:40:28.357 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:40:28.358 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:40:28.358 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:40:28.358 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:40:28.358 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:40:28.358 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:28.358 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:28.358 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:28.359 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:40:28.359 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:40:28.359 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:40:28.359 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:40:28.359 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:40:28.359 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:40:28.360 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:40:28.360 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:40:28.360 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:40:28.360 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:28.360 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:28.360 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:28.360 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:40:28.361 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:28.361 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:40:28.361 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:40:28.361 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:40:28.362 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:28.362 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:40:28Z" 2023-07-29T00:40:28.362 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:40:28.364 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690591073.070100_1058.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:37:53.070100Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:40:28Z" } 2023-07-29T00:40:28.364 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690591073.070100_1058.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:37:53.070100Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:40:28.365 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690591073.070100_1058.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T00:40:28.366 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690591073.070100_1058.1', 2: '', 3: ''} 2023-07-29T00:40:28.367 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1690591217.836402_294.1 target=1_1690587969.233742_99.1, shard=3 master=1_1690591212.811406_223.1 target= 2023-07-29T00:40:33.368 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:40:33.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:40:33.370 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:40:33.422 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:40:33.423 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:40:33.453 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:40:33.388+0000 7fa708a8b880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:40:33.454 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:40:33.389+0000 7fa708a8b880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:40:33.468 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:40:33.468 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:40:33.468 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:40:33.469 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:40:33.469 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:40:33.469 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:40:33.469 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:33.469 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:33.470 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:40:33.470 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:40:33.470 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:40:33.470 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:40:33.470 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:40:33.471 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:40:33.471 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:40:33.471 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:40:33.471 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:40:33.472 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:40:33.472 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:33.472 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:33.472 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:33.472 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:40:33.473 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:40:33.473 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:40:33.473 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:40:33.473 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591073.070100_1058.1", 2023-07-29T00:40:33.473 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:40:33.473 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:40:33.474 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:40:33.474 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:37:53.070100Z", 2023-07-29T00:40:33.474 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:33.474 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:33.475 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:33.475 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:40:33.475 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:40:33.475 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:40:33.476 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:40:33.476 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:40:33.476 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:40:33.476 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:40:33.477 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:40:33.477 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:40:33.477 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:33.478 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:33.478 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:33.478 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:40:33.479 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:40:33.479 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:40:33.479 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:40:33.479 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:40:33.480 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:40:33.480 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:40:33.480 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:40:33.481 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:40:33.481 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:33.481 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:33.482 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:33.482 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:40:33.482 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:33.482 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:40:33.483 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:40:33.483 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:40:33.483 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:33.483 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:40:33Z" 2023-07-29T00:40:33.483 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:40:33.484 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690591073.070100_1058.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:37:53.070100Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:40:33Z" } 2023-07-29T00:40:33.485 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690591073.070100_1058.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:37:53.070100Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:40:33.485 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690591073.070100_1058.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T00:40:33.486 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690591073.070100_1058.1', 2: '', 3: ''} 2023-07-29T00:40:33.486 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1690591217.836402_294.1 target=1_1690587969.233742_99.1, shard=3 master=1_1690591212.811406_223.1 target= 2023-07-29T00:40:38.487 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:40:38.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:40:38.490 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:40:38.545 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:40:38.546 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:40:38.578 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:40:38.512+0000 7f139154b880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:40:38.578 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:40:38.513+0000 7f139154b880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:40:38.592 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:40:38.593 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:40:38.593 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:40:38.593 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:40:38.593 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:40:38.594 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:40:38.594 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:38.594 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:38.594 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:40:38.594 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:40:38.595 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:40:38.595 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:40:38.595 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:40:38.595 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:40:38.595 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:40:38.595 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:40:38.596 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:40:38.596 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:40:38.596 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:38.596 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:38.596 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:38.597 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:40:38.597 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:40:38.597 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:40:38.597 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:40:38.597 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591073.070100_1058.1", 2023-07-29T00:40:38.598 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:40:38.598 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:40:38.598 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:40:38.598 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:37:53.070100Z", 2023-07-29T00:40:38.598 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:38.598 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:38.599 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:38.599 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:40:38.599 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:40:38.599 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:40:38.599 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:40:38.599 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:40:38.600 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:40:38.600 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:40:38.600 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:40:38.600 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:40:38.600 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:38.601 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:38.601 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:38.601 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:40:38.601 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:40:38.601 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:40:38.602 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:40:38.602 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:40:38.602 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:40:38.602 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:40:38.602 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:40:38.602 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:40:38.603 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:38.603 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:38.603 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:38.603 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:40:38.603 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:38.604 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:40:38.604 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:40:38.604 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:40:38.604 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:38.604 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:40:38Z" 2023-07-29T00:40:38.604 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:40:38.605 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690591073.070100_1058.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:37:53.070100Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:40:38Z" } 2023-07-29T00:40:38.606 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690591073.070100_1058.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:37:53.070100Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:40:38.606 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690591073.070100_1058.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T00:40:38.607 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690591073.070100_1058.1', 2: '', 3: ''} 2023-07-29T00:40:38.607 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1690591217.836402_294.1 target=1_1690587969.233742_99.1, shard=3 master=1_1690591212.811406_223.1 target= 2023-07-29T00:40:43.608 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:40:43.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:40:43.609 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:40:43.662 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:40:43.663 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:40:43.695 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:40:43.629+0000 7f89f7b32880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:40:43.696 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:40:43.630+0000 7f89f7b32880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:40:43.709 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:40:43.710 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:40:43.710 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:40:43.710 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:40:43.711 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:40:43.711 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:40:43.711 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:43.711 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:43.711 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:40:43.712 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:40:43.712 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:40:43.712 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:40:43.712 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:40:43.712 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:40:43.712 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:40:43.713 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:40:43.713 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:40:43.713 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:40:43.714 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:43.714 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:43.714 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:43.714 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:40:43.715 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:40:43.715 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:40:43.715 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:40:43.715 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591073.070100_1058.1", 2023-07-29T00:40:43.715 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:40:43.715 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:40:43.715 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:40:43.716 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:37:53.070100Z", 2023-07-29T00:40:43.716 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:43.716 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:43.716 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:43.716 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:40:43.716 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:40:43.717 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:40:43.717 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:40:43.717 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:40:43.718 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:40:43.718 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:40:43.718 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:40:43.718 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:40:43.719 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:43.719 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:43.719 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:43.720 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:40:43.720 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:40:43.720 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:40:43.721 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:40:43.721 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:40:43.721 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:40:43.722 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:40:43.722 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:40:43.722 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:40:43.723 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:43.723 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:43.723 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:43.724 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:40:43.724 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:43.724 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:40:43.725 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:40:43.725 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:40:43.725 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:43.725 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:40:43Z" 2023-07-29T00:40:43.725 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:40:43.727 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690591073.070100_1058.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:37:53.070100Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:40:43Z" } 2023-07-29T00:40:43.728 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690591073.070100_1058.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:37:53.070100Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:40:43.728 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690591073.070100_1058.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T00:40:43.729 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690591073.070100_1058.1', 2: '', 3: ''} 2023-07-29T00:40:43.729 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1690591217.836402_294.1 target=1_1690587969.233742_99.1, shard=3 master=1_1690591212.811406_223.1 target= 2023-07-29T00:40:48.731 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:40:48.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:40:48.732 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:40:48.783 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:40:48.784 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:40:48.815 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:40:48.749+0000 7efd7432a880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:40:48.815 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:40:48.750+0000 7efd7432a880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:40:48.829 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:40:48.830 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:40:48.830 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:40:48.830 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:40:48.831 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:40:48.831 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:40:48.831 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:48.831 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:48.831 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:40:48.831 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:40:48.831 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:40:48.832 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:40:48.832 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:40:48.832 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:40:48.832 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:40:48.832 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:40:48.832 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:40:48.833 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:40:48.833 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:48.833 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:48.833 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:48.833 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:40:48.833 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:40:48.833 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:40:48.834 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:40:48.834 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591073.070100_1058.1", 2023-07-29T00:40:48.834 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:40:48.834 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:40:48.834 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:40:48.834 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:37:53.070100Z", 2023-07-29T00:40:48.834 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:48.835 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:48.835 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:48.835 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:40:48.835 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:40:48.835 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:40:48.835 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:40:48.835 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:40:48.836 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:40:48.836 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:40:48.836 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:40:48.836 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:40:48.836 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:48.836 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:48.836 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:48.837 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:40:48.837 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:40:48.837 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:40:48.837 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:40:48.837 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:40:48.837 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:40:48.837 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:40:48.838 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:40:48.838 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:40:48.838 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:48.838 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:48.838 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:48.838 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:40:48.838 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:48.839 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:40:48.839 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:40:48.839 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:40:48.839 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:48.839 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:40:48Z" 2023-07-29T00:40:48.839 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:40:48.840 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690591073.070100_1058.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:37:53.070100Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:40:48Z" } 2023-07-29T00:40:48.841 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690591073.070100_1058.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:37:53.070100Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:40:48.841 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690591073.070100_1058.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T00:40:48.842 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690591073.070100_1058.1', 2: '', 3: ''} 2023-07-29T00:40:48.842 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1690591217.836402_294.1 target=1_1690587969.233742_99.1, shard=3 master=1_1690591212.811406_223.1 target= 2023-07-29T00:40:53.843 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:40:53.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:40:53.844 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:40:53.897 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:40:53.898 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:40:53.931 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:40:53.864+0000 7f8e80410880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:40:53.931 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:40:53.865+0000 7f8e80410880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:40:53.944 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:40:53.944 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:40:53.945 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:40:53.945 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:40:53.945 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:40:53.945 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:40:53.945 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:53.945 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:53.945 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:40:53.946 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:40:53.946 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:40:53.946 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:40:53.946 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:40:53.946 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:40:53.946 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:40:53.947 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:40:53.947 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:40:53.947 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:40:53.947 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:53.947 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:53.948 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:53.948 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:40:53.948 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:40:53.948 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:40:53.948 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:40:53.948 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591073.070100_1058.1", 2023-07-29T00:40:53.949 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:40:53.949 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:40:53.949 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:40:53.949 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:37:53.070100Z", 2023-07-29T00:40:53.949 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:53.949 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:53.949 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:53.950 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:40:53.950 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:40:53.950 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:40:53.950 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:40:53.950 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:40:53.950 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:40:53.951 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:40:53.951 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:40:53.951 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:40:53.951 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:53.951 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:53.951 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:53.951 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:40:53.952 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:40:53.952 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:40:53.952 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:40:53.952 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:40:53.952 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:40:53.952 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:40:53.953 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:40:53.953 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:40:53.953 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:53.953 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:53.953 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:53.953 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:40:53.953 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:53.953 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:40:53.954 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:40:53.954 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:40:53.954 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:53.954 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:40:53Z" 2023-07-29T00:40:53.954 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:40:53.955 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690591073.070100_1058.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:37:53.070100Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:40:53Z" } 2023-07-29T00:40:53.956 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690591073.070100_1058.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:37:53.070100Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:40:53.956 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690591073.070100_1058.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T00:40:53.957 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690591073.070100_1058.1', 2: '', 3: ''} 2023-07-29T00:40:53.957 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1690591217.836402_294.1 target=1_1690587969.233742_99.1, shard=3 master=1_1690591212.811406_223.1 target= 2023-07-29T00:40:58.958 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:40:58.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:40:58.959 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:40:59.012 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:40:59.012 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:40:59.044 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:40:58.978+0000 7fdf3f42d880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:40:59.045 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:40:58.979+0000 7fdf3f42d880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:40:59.058 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:40:59.059 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:40:59.059 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:40:59.059 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:40:59.059 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:40:59.059 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:40:59.060 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:59.060 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:59.060 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:40:59.060 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:40:59.060 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:40:59.061 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:40:59.061 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:40:59.061 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:40:59.061 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:40:59.061 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:40:59.061 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:40:59.061 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:40:59.062 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:59.062 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:59.062 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:59.062 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:40:59.062 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:40:59.062 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:40:59.062 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:40:59.063 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591073.070100_1058.1", 2023-07-29T00:40:59.063 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:40:59.063 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:40:59.063 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:40:59.063 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:37:53.070100Z", 2023-07-29T00:40:59.063 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:59.064 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:59.064 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:59.064 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:40:59.064 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:40:59.064 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:40:59.065 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:40:59.065 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:40:59.065 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:40:59.065 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:40:59.065 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:40:59.065 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:40:59.065 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:59.066 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:59.066 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:59.066 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:40:59.066 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:40:59.066 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:40:59.066 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:40:59.066 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:40:59.067 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:40:59.067 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:40:59.067 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:40:59.067 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:40:59.067 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:40:59.067 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:59.068 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:40:59.068 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:40:59.068 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:59.068 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:40:59.068 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:40:59.068 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:40:59.068 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:40:59.069 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:40:58Z" 2023-07-29T00:40:59.069 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:40:59.070 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690591073.070100_1058.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:37:53.070100Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:40:58Z" } 2023-07-29T00:40:59.070 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690591073.070100_1058.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:37:53.070100Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:40:59.070 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690591073.070100_1058.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T00:40:59.071 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690591073.070100_1058.1', 2: '', 3: ''} 2023-07-29T00:40:59.071 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1690591217.836402_294.1 target=1_1690587969.233742_99.1, shard=3 master=1_1690591212.811406_223.1 target= 2023-07-29T00:41:04.072 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:41:04.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:41:04.073 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:41:04.127 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:41:04.127 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:41:04.159 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:41:04.093+0000 7f3479934880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:41:04.160 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:41:04.094+0000 7f3479934880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:41:04.175 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:41:04.175 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:41:04.176 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:41:04.176 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:41:04.176 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:41:04.177 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:41:04.177 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:04.177 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:04.177 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:41:04.178 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:41:04.178 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:41:04.178 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:41:04.178 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:41:04.179 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:41:04.179 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:41:04.179 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:41:04.179 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:41:04.179 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:41:04.180 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:04.180 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:04.180 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:04.180 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:41:04.181 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:41:04.181 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:41:04.181 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:41:04.181 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591073.070100_1058.1", 2023-07-29T00:41:04.182 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:41:04.182 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:41:04.182 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:41:04.182 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:37:53.070100Z", 2023-07-29T00:41:04.183 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:04.183 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:04.183 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:04.183 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:41:04.183 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:41:04.184 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:41:04.184 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:41:04.184 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:41:04.184 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:41:04.185 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:41:04.185 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:41:04.185 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:41:04.185 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:04.185 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:04.186 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:04.186 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:41:04.186 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:41:04.187 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:41:04.187 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:41:04.187 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:41:04.188 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:41:04.188 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:41:04.188 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:41:04.189 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:41:04.189 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:04.189 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:04.190 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:04.190 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:41:04.190 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:04.191 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:41:04.191 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:41:04.191 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:41:04.192 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:04.192 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:41:04Z" 2023-07-29T00:41:04.192 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:41:04.194 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690591073.070100_1058.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:37:53.070100Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:41:04Z" } 2023-07-29T00:41:04.195 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690591073.070100_1058.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:37:53.070100Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:41:04.195 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690591073.070100_1058.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T00:41:04.196 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690591073.070100_1058.1', 2: '', 3: ''} 2023-07-29T00:41:04.197 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1690591217.836402_294.1 target=1_1690587969.233742_99.1, shard=3 master=1_1690591212.811406_223.1 target= 2023-07-29T00:41:09.199 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:41:09.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:41:09.200 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:41:09.253 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:41:09.254 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:41:09.286 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:41:09.220+0000 7f89d7f9c880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:41:09.286 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:41:09.221+0000 7f89d7f9c880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:41:09.300 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:41:09.301 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:41:09.301 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:41:09.301 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:41:09.301 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:41:09.301 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:41:09.302 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:09.302 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:09.302 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:41:09.302 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:41:09.302 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:41:09.302 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:41:09.303 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:41:09.303 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:41:09.303 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:41:09.303 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:41:09.303 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:41:09.303 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:41:09.303 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:09.304 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:09.304 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:09.304 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:41:09.304 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:41:09.304 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:41:09.304 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:41:09.305 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591073.070100_1058.1", 2023-07-29T00:41:09.305 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:41:09.305 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:41:09.305 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:41:09.305 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:37:53.070100Z", 2023-07-29T00:41:09.305 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:09.305 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:09.306 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:09.306 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:41:09.306 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:41:09.306 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:41:09.306 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:41:09.306 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:41:09.306 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:41:09.307 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:41:09.307 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:41:09.307 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:41:09.307 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:09.307 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:09.307 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:09.307 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:41:09.308 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:41:09.308 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:41:09.308 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:41:09.308 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:41:09.308 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:41:09.308 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:41:09.309 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:41:09.309 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:41:09.309 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:09.309 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:09.309 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:09.309 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:41:09.309 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:09.310 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:41:09.310 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:41:09.310 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:41:09.310 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:09.310 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:41:09Z" 2023-07-29T00:41:09.310 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:41:09.311 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690591073.070100_1058.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:37:53.070100Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:41:09Z" } 2023-07-29T00:41:09.312 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690591073.070100_1058.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:37:53.070100Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:41:09.312 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690591073.070100_1058.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T00:41:09.312 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690591073.070100_1058.1', 2: '', 3: ''} 2023-07-29T00:41:09.313 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1690591217.836402_294.1 target=1_1690587969.233742_99.1, shard=3 master=1_1690591212.811406_223.1 target= 2023-07-29T00:41:14.314 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:41:14.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:41:14.315 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:41:14.368 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:41:14.369 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:41:14.401 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:41:14.335+0000 7fbd58611880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:41:14.401 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:41:14.336+0000 7fbd58611880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:41:14.415 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:41:14.416 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:41:14.416 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:41:14.416 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:41:14.416 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:41:14.416 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:41:14.417 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:14.417 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:14.417 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:41:14.417 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:41:14.417 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:41:14.418 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:41:14.418 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:41:14.418 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:41:14.419 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:41:14.419 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:41:14.419 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:41:14.420 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:41:14.420 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:14.420 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:14.420 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:14.421 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:41:14.421 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:41:14.421 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:41:14.422 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:41:14.422 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591073.070100_1058.1", 2023-07-29T00:41:14.423 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:41:14.423 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:41:14.423 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:41:14.423 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:37:53.070100Z", 2023-07-29T00:41:14.424 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:14.424 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:14.424 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:14.424 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:41:14.424 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:41:14.424 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:41:14.424 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:41:14.425 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:41:14.425 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:41:14.425 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:41:14.425 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:41:14.425 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:41:14.425 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:14.425 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:14.426 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:14.426 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:41:14.426 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:41:14.426 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:41:14.426 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:41:14.426 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:41:14.426 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:41:14.427 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:41:14.427 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:41:14.427 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:41:14.427 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:14.427 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:14.427 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:14.427 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:41:14.428 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:14.428 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:41:14.428 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:41:14.428 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:41:14.428 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:14.428 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:41:14Z" 2023-07-29T00:41:14.429 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:41:14.430 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690591073.070100_1058.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:37:53.070100Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:41:14Z" } 2023-07-29T00:41:14.430 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690591073.070100_1058.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:37:53.070100Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:41:14.430 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690591073.070100_1058.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T00:41:14.431 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690591073.070100_1058.1', 2: '', 3: ''} 2023-07-29T00:41:14.431 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1690591217.836402_294.1 target=1_1690587969.233742_99.1, shard=3 master=1_1690591212.811406_223.1 target= 2023-07-29T00:41:19.432 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:41:19.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:41:19.434 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:41:19.487 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:41:19.488 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:41:19.520 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:41:19.454+0000 7fd1fc812880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:41:19.521 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:41:19.455+0000 7fd1fc812880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:41:19.535 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:41:19.535 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:41:19.535 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:41:19.535 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:41:19.536 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:41:19.536 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:41:19.536 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:19.536 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:19.536 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:41:19.536 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:41:19.537 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:41:19.537 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:41:19.537 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:41:19.537 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:41:19.537 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:41:19.537 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:41:19.537 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:41:19.538 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:41:19.538 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:19.538 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:19.538 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:19.538 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:41:19.538 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:41:19.538 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:41:19.539 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:41:19.539 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591073.070100_1058.1", 2023-07-29T00:41:19.539 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:41:19.539 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:41:19.539 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:41:19.539 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:37:53.070100Z", 2023-07-29T00:41:19.539 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:19.540 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:19.540 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:19.540 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:41:19.540 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:41:19.540 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:41:19.540 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:41:19.540 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:41:19.541 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:41:19.541 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:41:19.541 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:41:19.541 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:41:19.541 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:19.541 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:19.541 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:19.542 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:41:19.542 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:41:19.542 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:41:19.542 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:41:19.542 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:41:19.542 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:41:19.542 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:41:19.543 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:41:19.543 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:41:19.543 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:19.543 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:19.543 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:19.543 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:41:19.543 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:19.544 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:41:19.544 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:41:19.544 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:41:19.544 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:19.544 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:41:19Z" 2023-07-29T00:41:19.544 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:41:19.545 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690591073.070100_1058.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:37:53.070100Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:41:19Z" } 2023-07-29T00:41:19.546 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690591073.070100_1058.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:37:53.070100Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:41:19.546 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690591073.070100_1058.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T00:41:19.546 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690591073.070100_1058.1', 2: '', 3: ''} 2023-07-29T00:41:19.547 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1690591217.836402_294.1 target=1_1690587969.233742_99.1, shard=3 master=1_1690591212.811406_223.1 target= 2023-07-29T00:41:24.548 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:41:24.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:41:24.549 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:41:24.602 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:41:24.603 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:41:24.635 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:41:24.569+0000 7f2c2d213880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:41:24.635 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:41:24.570+0000 7f2c2d213880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:41:24.650 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:41:24.650 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:41:24.650 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:41:24.650 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:41:24.650 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:41:24.650 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:41:24.651 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:24.651 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:24.651 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:41:24.651 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:41:24.651 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:41:24.651 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:41:24.652 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:41:24.652 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:41:24.652 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:41:24.652 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:41:24.652 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:41:24.652 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:41:24.652 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:24.653 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:24.653 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:24.653 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:41:24.653 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:41:24.653 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:41:24.653 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:41:24.654 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591073.070100_1058.1", 2023-07-29T00:41:24.654 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:41:24.654 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:41:24.654 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:41:24.654 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:37:53.070100Z", 2023-07-29T00:41:24.654 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:24.655 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:24.655 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:24.655 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:41:24.655 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:41:24.655 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:41:24.655 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:41:24.655 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:41:24.655 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:41:24.656 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:41:24.656 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:41:24.656 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:41:24.656 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:24.656 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:24.656 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:24.657 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:41:24.657 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:41:24.657 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:41:24.657 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:41:24.657 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:41:24.657 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:41:24.657 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:41:24.658 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:41:24.658 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:41:24.658 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:24.658 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:24.658 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:24.658 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:41:24.658 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:24.659 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:41:24.659 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:41:24.659 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:41:24.659 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:24.659 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:41:24Z" 2023-07-29T00:41:24.659 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:41:24.660 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690591073.070100_1058.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:37:53.070100Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:41:24Z" } 2023-07-29T00:41:24.661 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690591073.070100_1058.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:37:53.070100Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:41:24.661 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690591073.070100_1058.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T00:41:24.661 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690591073.070100_1058.1', 2: '', 3: ''} 2023-07-29T00:41:24.662 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1690591217.836402_294.1 target=1_1690587969.233742_99.1, shard=3 master=1_1690591212.811406_223.1 target= 2023-07-29T00:41:29.662 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:41:29.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:41:29.663 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:41:29.717 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:41:29.717 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:41:29.750 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:41:29.684+0000 7f1397126880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:41:29.751 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:41:29.685+0000 7f1397126880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:41:29.764 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:41:29.765 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:41:29.765 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:41:29.765 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:41:29.765 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:41:29.765 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:41:29.765 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:29.766 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:29.766 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:41:29.766 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:41:29.766 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:41:29.766 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:41:29.767 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:41:29.767 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:41:29.767 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:41:29.767 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:41:29.767 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:41:29.767 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:41:29.767 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:29.768 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:29.768 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:29.768 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:41:29.768 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:41:29.768 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:41:29.768 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:41:29.768 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591073.070100_1058.1", 2023-07-29T00:41:29.769 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:41:29.769 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:41:29.769 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:41:29.769 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:37:53.070100Z", 2023-07-29T00:41:29.769 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:29.769 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:29.769 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:29.770 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:41:29.770 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:41:29.770 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:41:29.770 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:41:29.770 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:41:29.770 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:41:29.771 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:41:29.771 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:41:29.771 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:41:29.771 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:29.771 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:29.771 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:29.771 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:41:29.771 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:41:29.772 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:41:29.772 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:41:29.772 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:41:29.772 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:41:29.772 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:41:29.772 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:41:29.773 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:41:29.773 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:29.773 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:29.773 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:29.773 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:41:29.773 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:29.773 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:41:29.773 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:41:29.774 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:41:29.774 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:29.774 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:41:29Z" 2023-07-29T00:41:29.774 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:41:29.775 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690591073.070100_1058.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:37:53.070100Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:41:29Z" } 2023-07-29T00:41:29.775 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690591073.070100_1058.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:37:53.070100Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:41:29.776 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690591073.070100_1058.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T00:41:29.776 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690591073.070100_1058.1', 2: '', 3: ''} 2023-07-29T00:41:29.776 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1690591217.836402_294.1 target=1_1690587969.233742_99.1, shard=3 master=1_1690591212.811406_223.1 target= 2023-07-29T00:41:34.777 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:41:34.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:41:34.778 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:41:34.832 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:41:34.832 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:41:34.865 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:41:34.799+0000 7f6901fdc880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:41:34.865 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:41:34.800+0000 7f6901fdc880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:41:34.880 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:41:34.880 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:41:34.880 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:41:34.880 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:41:34.880 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:41:34.881 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:41:34.881 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:34.881 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:34.881 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:41:34.881 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:41:34.881 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:41:34.881 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:41:34.882 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:41:34.882 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:41:34.882 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:41:34.882 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:41:34.882 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:41:34.882 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:41:34.882 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:34.883 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:34.883 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:34.883 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:41:34.883 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:41:34.883 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:41:34.883 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:41:34.883 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591073.070100_1058.1", 2023-07-29T00:41:34.884 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:41:34.884 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:41:34.884 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:41:34.884 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:37:53.070100Z", 2023-07-29T00:41:34.884 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:34.884 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:34.885 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:34.885 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:41:34.885 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:41:34.885 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:41:34.885 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:41:34.885 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:41:34.885 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:41:34.886 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:41:34.886 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:41:34.886 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:41:34.886 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:34.886 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:34.886 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:34.886 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:41:34.887 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:41:34.887 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:41:34.887 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:41:34.887 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:41:34.887 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:41:34.887 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:41:34.887 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:41:34.888 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:41:34.888 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:34.888 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:34.888 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:34.888 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:41:34.888 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:34.888 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:41:34.889 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:41:34.889 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:41:34.889 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:34.889 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:41:34Z" 2023-07-29T00:41:34.889 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:41:34.890 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690591073.070100_1058.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:37:53.070100Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:41:34Z" } 2023-07-29T00:41:34.891 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690591073.070100_1058.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:37:53.070100Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:41:34.891 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690591073.070100_1058.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T00:41:34.892 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690591073.070100_1058.1', 2: '', 3: ''} 2023-07-29T00:41:34.892 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1690591217.836402_294.1 target=1_1690587969.233742_99.1, shard=3 master=1_1690591212.811406_223.1 target= 2023-07-29T00:41:39.893 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:41:39.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:41:39.894 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:41:39.946 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:41:39.947 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:41:39.979 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:41:39.913+0000 7fed58412880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:41:39.979 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:41:39.914+0000 7fed58412880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:41:39.997 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:41:39.997 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:41:39.997 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:41:39.998 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:41:39.998 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:41:39.998 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:41:39.998 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:39.998 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:39.998 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:41:39.998 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:41:39.999 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:41:39.999 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:41:39.999 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:41:39.999 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:41:39.999 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:41:39.999 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:41:40.000 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:41:40.000 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:41:40.000 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:40.000 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:40.000 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:40.000 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:41:40.000 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:41:40.001 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:41:40.001 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:41:40.001 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591073.070100_1058.1", 2023-07-29T00:41:40.001 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:41:40.001 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:41:40.001 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:41:40.001 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:37:53.070100Z", 2023-07-29T00:41:40.002 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:40.002 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:40.002 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:40.002 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:41:40.002 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:41:40.002 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:41:40.002 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:41:40.003 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:41:40.003 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:41:40.003 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:41:40.003 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:41:40.003 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:41:40.003 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:40.003 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:40.004 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:40.004 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:41:40.004 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:41:40.004 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:41:40.004 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:41:40.004 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:41:40.004 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:41:40.005 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:41:40.005 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:41:40.005 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:41:40.005 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:40.005 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:40.005 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:40.006 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:41:40.006 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:40.006 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:41:40.006 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:41:40.006 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:41:40.006 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:40.006 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:41:39Z" 2023-07-29T00:41:40.007 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:41:40.007 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690591073.070100_1058.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:37:53.070100Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:41:39Z" } 2023-07-29T00:41:40.008 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690591073.070100_1058.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:37:53.070100Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T00:41:40.008 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690591073.070100_1058.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T00:41:40.009 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690591073.070100_1058.1', 2: '', 3: ''} 2023-07-29T00:41:40.009 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1690591217.836402_294.1 target=1_1690587969.233742_99.1, shard=3 master=1_1690591212.811406_223.1 target= 2023-07-29T00:41:45.011 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:41:45.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:41:45.012 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:41:45.065 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:41:45.066 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:41:45.097 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:41:45.031+0000 7f09d9b2c880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:41:45.098 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:41:45.032+0000 7f09d9b2c880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:41:45.112 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:41:45.112 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:41:45.112 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:41:45.112 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:41:45.113 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:41:45.113 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:41:45.113 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:45.113 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:45.113 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:41:45.113 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:41:45.114 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:41:45.114 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:41:45.114 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:41:45.114 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690587969.233742_99.1", 2023-07-29T00:41:45.114 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:41:45.114 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:41:45.115 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:41:45.115 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T00:41:45.115 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:45.115 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:45.115 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:45.115 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:41:45.115 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:41:45.116 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:41:45.116 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:41:45.116 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591073.070100_1058.1", 2023-07-29T00:41:45.116 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:41:45.116 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:41:45.116 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:41:45.116 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:37:53.070100Z", 2023-07-29T00:41:45.117 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:45.117 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:45.117 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:45.117 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:41:45.117 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:41:45.117 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:41:45.117 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:41:45.118 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:41:45.118 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:41:45.118 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:41:45.118 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:41:45.118 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:41:45.118 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:45.118 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:45.119 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:45.119 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:41:45.119 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:41:45.119 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:41:45.119 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:41:45.119 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T00:41:45.119 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:41:45.120 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:41:45.120 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:41:45.120 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T00:41:45.120 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:45.120 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:45.120 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:45.120 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:41:45.121 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:45.121 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:41:45.121 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:41:45.121 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:41:45.121 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:45.121 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:41:45Z" 2023-07-29T00:41:45.121 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:41:45.122 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690587969.233742_99.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-28T23:46:09.233742Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690591073.070100_1058.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:37:53.070100Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:41:45Z" } 2023-07-29T00:41:45.123 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690591073.070100_1058.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:37:53.070100Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T00:41:45.123 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690591073.070100_1058.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T00:41:45.123 DEBUG:rgw_multi.tests:sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690591073.070100_1058.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T00:41:45.124 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1690591217.836402_294.1 target=1_1690587969.233742_99.1 2023-07-29T00:41:50.125 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:41:50.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:41:50.126 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:41:50.180 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:41:50.180 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:41:50.212 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:41:50.146+0000 7fdb7f126880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:41:50.213 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:41:50.147+0000 7fdb7f126880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:41:50.227 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:41:50.227 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:41:50.227 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:41:50.227 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:41:50.228 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:41:50.228 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:41:50.228 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:50.228 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:50.228 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:41:50.228 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:41:50.229 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:41:50.229 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:41:50.229 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:41:50.229 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T00:41:50.229 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:41:50.230 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:41:50.230 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:41:50.230 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T00:41:50.230 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:50.230 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:50.230 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:50.231 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:41:50.231 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:41:50.231 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:41:50.231 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:41:50.231 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591073.070100_1058.1", 2023-07-29T00:41:50.231 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:41:50.231 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:41:50.232 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:41:50.232 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:37:53.070100Z", 2023-07-29T00:41:50.232 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:50.232 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:50.232 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:50.232 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:41:50.232 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:41:50.233 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:41:50.233 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:41:50.233 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:41:50.233 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:41:50.233 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:41:50.233 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:41:50.233 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:41:50.234 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:50.234 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:50.234 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:50.234 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:41:50.234 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:41:50.234 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:41:50.234 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:41:50.235 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T00:41:50.235 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:41:50.235 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:41:50.235 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:41:50.235 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T00:41:50.235 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:41:50.235 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:50.236 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:41:50.236 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:41:50.236 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:50.236 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:41:50.236 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:41:50.236 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:41:50.236 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:41:50.237 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:41:50Z" 2023-07-29T00:41:50.237 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:41:50.238 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690591073.070100_1058.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:37:53.070100Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:41:50Z" } 2023-07-29T00:41:50.238 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690591073.070100_1058.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:37:53.070100Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T00:41:50.239 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690591073.070100_1058.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T00:41:50.239 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690591073.070100_1058.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T00:41:50.239 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T00:41:50.240 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-hiiqos-1 zones={a2, a1} 2023-07-29T00:41:50.240 DEBUG:boto:Method: POST 2023-07-29T00:41:50.241 DEBUG:boto:Path: / 2023-07-29T00:41:50.241 DEBUG:boto:Data: 2023-07-29T00:41:50.241 DEBUG:boto:Headers: {} 2023-07-29T00:41:50.242 DEBUG:boto:Host: smithi101.front.sepia.ceph.com 2023-07-29T00:41:50.242 DEBUG:boto:Port: 8001 2023-07-29T00:41:50.242 DEBUG:boto:Params: {'RoleName': 'roles-hiiqos-1', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-07-29T00:41:50.243 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-29T00:41:50.243 DEBUG:boto:Token: None 2023-07-29T00:41:50.244 DEBUG:boto:CanonicalRequest: POST / host:smithi101.front.sepia.ceph.com:8001 x-amz-date:20230729T004150Z host;x-amz-date a7f49b7e012f84ef9b1058a70c2f7c7d8d492dc59101b55c23241e8faa14e99d 2023-07-29T00:41:50.244 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230729T004150Z 20230729/front/smithi101/aws4_request 9cc003e1ce42abf4309e0c673982eab07e39d2ab9e3a52bca3ec457ab49cf7d8 2023-07-29T00:41:50.245 DEBUG:boto:Signature: 14374708853fd29984bf92a71394b57178d98f13ac1c9f1874e8302e264d4b5f 2023-07-29T00:41:50.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'X-Amz-Date': '20230729T004150Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=S7W2J6K8BFBCG65TLASH/20230729/front/smithi101/aws4_request,SignedHeaders=host;x-amz-date,Signature=14374708853fd29984bf92a71394b57178d98f13ac1c9f1874e8302e264d4b5f', 'Host': 'smithi101.front.sepia.ceph.com:8001'} 2023-07-29T00:41:55.249 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a8b5dab5d37ed334-0064c46053-1eb0-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '710'), ('Date', 'Sat, 29 Jul 2023 00:41:55 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:41:55.251 DEBUG:boto:b'tx00000a8b5dab5d37ed334-0064c46053-1eb0-a27d63324a-e189-4107-be76-d86c6ac405b8roles-hiiqos-1/arn:aws:iam:::role/roles-hiiqos-12023-07-29T00:41:44.29Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-07-29T00:41:55.253 DEBUG:boto:Method: POST 2023-07-29T00:41:55.254 DEBUG:boto:Path: / 2023-07-29T00:41:55.254 DEBUG:boto:Data: 2023-07-29T00:41:55.255 DEBUG:boto:Headers: {} 2023-07-29T00:41:55.256 DEBUG:boto:Host: smithi101.front.sepia.ceph.com 2023-07-29T00:41:55.256 DEBUG:boto:Port: 8000 2023-07-29T00:41:55.257 DEBUG:boto:Params: {'RoleName': 'roles-hiiqos-1', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-07-29T00:41:55.258 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-29T00:41:55.258 DEBUG:boto:Token: None 2023-07-29T00:41:55.259 DEBUG:boto:CanonicalRequest: POST / host:smithi101.front.sepia.ceph.com:8000 x-amz-date:20230729T004155Z host;x-amz-date a7f49b7e012f84ef9b1058a70c2f7c7d8d492dc59101b55c23241e8faa14e99d 2023-07-29T00:41:55.259 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230729T004155Z 20230729/front/smithi101/aws4_request 1095502e0d3b84b997255e1ace4843e3b5022b9839fc7c077048aa9395e454b1 2023-07-29T00:41:55.260 DEBUG:boto:Signature: 4f1f3f08e40f21495fa1e7c229ce89849e32ffef46d07f182ba09f91562ce6e6 2023-07-29T00:41:55.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'X-Amz-Date': '20230729T004155Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=S7W2J6K8BFBCG65TLASH/20230729/front/smithi101/aws4_request,SignedHeaders=host;x-amz-date,Signature=4f1f3f08e40f21495fa1e7c229ce89849e32ffef46d07f182ba09f91562ce6e6', 'Host': 'smithi101.front.sepia.ceph.com:8000'} 2023-07-29T00:42:00.263 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000965eb9bb706f087b-0064c46058-165f-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Sat, 29 Jul 2023 00:42:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:42:00.264 DEBUG:boto:b'tx00000965eb9bb706f087b-0064c46058-165f-a17d63324a-e189-4107-be76-d86c6ac405b8roles-hiiqos-1/arn:aws:iam:::role/roles-hiiqos-12023-07-29T00:40:12.807Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-07-29T00:42:00.265 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-hiiqos-1 2023-07-29T00:42:00.266 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-hiiqos-1 zones={a2, a1} 2023-07-29T00:42:00.266 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-hiiqos-2 zones={a1, a2} 2023-07-29T00:42:00.266 DEBUG:boto:Method: POST 2023-07-29T00:42:00.267 DEBUG:boto:Path: / 2023-07-29T00:42:00.267 DEBUG:boto:Data: 2023-07-29T00:42:00.267 DEBUG:boto:Headers: {} 2023-07-29T00:42:00.268 DEBUG:boto:Host: smithi101.front.sepia.ceph.com 2023-07-29T00:42:00.268 DEBUG:boto:Port: 8000 2023-07-29T00:42:00.268 DEBUG:boto:Params: {'RoleName': 'roles-hiiqos-2', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-07-29T00:42:00.269 DEBUG:boto:Token: None 2023-07-29T00:42:00.269 DEBUG:boto:CanonicalRequest: POST / host:smithi101.front.sepia.ceph.com:8000 x-amz-date:20230729T004200Z host;x-amz-date 353cdc050394aa416fd5c03754ce30e8219edeab1b49b1c3bedd0cf48ab226e1 2023-07-29T00:42:00.270 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230729T004200Z 20230729/front/smithi101/aws4_request 060e117157494653615789a34ba819d65ab4a8b65de929c7ceb0a280147331ef 2023-07-29T00:42:00.270 DEBUG:boto:Signature: c0eb41de4212e817fcbf4c15fb5cc40b24d5794df3669166d69c170fa4a6a4cf 2023-07-29T00:42:00.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'X-Amz-Date': '20230729T004200Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=S7W2J6K8BFBCG65TLASH/20230729/front/smithi101/aws4_request,SignedHeaders=host;x-amz-date,Signature=c0eb41de4212e817fcbf4c15fb5cc40b24d5794df3669166d69c170fa4a6a4cf', 'Host': 'smithi101.front.sepia.ceph.com:8000'} 2023-07-29T00:42:00.272 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005b099878b9cd61f4-0064c46058-165f-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Sat, 29 Jul 2023 00:42:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:42:00.272 DEBUG:boto:b'tx000005b099878b9cd61f4-0064c46058-165f-a1a75d7c6d-3c80-4c31-b763-8905e38b45cdroles-hiiqos-2/arn:aws:iam:::role/roles-hiiqos-22023-07-29T00:40:17.832Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-07-29T00:42:00.273 DEBUG:boto:Method: POST 2023-07-29T00:42:00.274 DEBUG:boto:Path: / 2023-07-29T00:42:00.274 DEBUG:boto:Data: 2023-07-29T00:42:00.274 DEBUG:boto:Headers: {} 2023-07-29T00:42:00.275 DEBUG:boto:Host: smithi101.front.sepia.ceph.com 2023-07-29T00:42:00.275 DEBUG:boto:Port: 8001 2023-07-29T00:42:00.276 DEBUG:boto:Params: {'RoleName': 'roles-hiiqos-2', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-07-29T00:42:00.276 DEBUG:boto:Token: None 2023-07-29T00:42:00.277 DEBUG:boto:CanonicalRequest: POST / host:smithi101.front.sepia.ceph.com:8001 x-amz-date:20230729T004200Z host;x-amz-date 353cdc050394aa416fd5c03754ce30e8219edeab1b49b1c3bedd0cf48ab226e1 2023-07-29T00:42:00.277 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230729T004200Z 20230729/front/smithi101/aws4_request 50e0e703fe8c43d9fb38e17aa71b06e20b780ca0b0644ef12ef13d5ea9fc1427 2023-07-29T00:42:00.277 DEBUG:boto:Signature: 46833f0c96253706609df8c592863b66899ebf14d3963cfa09d9079e1e9cff4d 2023-07-29T00:42:00.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'X-Amz-Date': '20230729T004200Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=S7W2J6K8BFBCG65TLASH/20230729/front/smithi101/aws4_request,SignedHeaders=host;x-amz-date,Signature=46833f0c96253706609df8c592863b66899ebf14d3963cfa09d9079e1e9cff4d', 'Host': 'smithi101.front.sepia.ceph.com:8001'} 2023-07-29T00:42:00.280 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000e4ad74e6dcfb880-0064c46058-1eb0-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Sat, 29 Jul 2023 00:42:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:42:00.281 DEBUG:boto:b'tx000000e4ad74e6dcfb880-0064c46058-1eb0-a2a75d7c6d-3c80-4c31-b763-8905e38b45cdroles-hiiqos-2/arn:aws:iam:::role/roles-hiiqos-22023-07-29T00:40:17.832Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-07-29T00:42:00.282 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-hiiqos-2 2023-07-29T00:42:00.282 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-hiiqos-2 zones={a1, a2} 2023-07-29T00:42:00.282 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_role_sync ... ok 2023-07-29T00:42:00.284 DEBUG:boto:path=/hiiqos-60/ 2023-07-29T00:42:00.284 DEBUG:boto:auth_path=/hiiqos-60/ 2023-07-29T00:42:00.285 DEBUG:boto:Method: PUT 2023-07-29T00:42:00.285 DEBUG:boto:Path: /hiiqos-60/ 2023-07-29T00:42:00.285 DEBUG:boto:Data: 2023-07-29T00:42:00.286 DEBUG:boto:Headers: {} 2023-07-29T00:42:00.286 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:42:00.286 DEBUG:boto:Port: 8000 2023-07-29T00:42:00.287 DEBUG:boto:Params: {} 2023-07-29T00:42:00.287 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-29T00:42:00.288 DEBUG:boto:Token: None 2023-07-29T00:42:00.288 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 00:42:00 GMT /hiiqos-60/ 2023-07-29T00:42:00.289 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IRHw49ap3hInNY9uMTceFYm+5qc= 2023-07-29T00:42:00.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:42:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IRHw49ap3hInNY9uMTceFYm+5qc=', 'Content-Length': '0'} 2023-07-29T00:42:02.020 INFO:tasks.rgw.c2.client.0.smithi139.stdout:2023-07-29T00:42:01.952+0000 7f5f2671b640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T00:42:05.308 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c7fd89200c7860e0-0064c4605d-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:42:05.309 DEBUG:rgw_multi.tests:created bucket=hiiqos-60 2023-07-29T00:42:05.310 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:42:05.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:42:05.311 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:42:05.365 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:42:05.365 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:42:05.400 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T00:42:05.413 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T00:42:05.414 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:42:05.414 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T00:42:05.414 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:17.836402Z" 2023-07-29T00:42:05.414 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:42:05.414 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:42:05.415 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591325.241299_1083.1", 2023-07-29T00:42:05.415 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:42:05.241299Z" 2023-07-29T00:42:05.415 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:42:05.415 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:42:05.415 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:42:05.415 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T00:42:05.416 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:42:05.416 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:42:05.416 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T00:42:05.416 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:12.811406Z" 2023-07-29T00:42:05.416 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:42:05.416 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T00:42:05.417 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690591217.836402_294.1', 1: '1_1690591325.241299_1083.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T00:42:05.418 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T00:42:05.418 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:42:05.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:42:05.419 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:42:05.466 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:42:05.467 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:42:05.496 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:42:05.429+0000 7fb10f126880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:42:05.496 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:42:05.430+0000 7fb10f126880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:42:05.509 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:42:05.509 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:42:05.510 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:42:05.510 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:42:05.510 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:42:05.510 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:42:05.510 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:42:05.510 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:42:05.510 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:42:05.511 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:42:05.511 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:42:05.511 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:42:05.511 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:42:05.511 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T00:42:05.511 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:42:05.512 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:42:05.512 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:42:05.512 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T00:42:05.512 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:42:05.512 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:42:05.512 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:42:05.513 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:42:05.513 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:42:05.513 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:42:05.514 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:42:05.514 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591325.233838_1080.1", 2023-07-29T00:42:05.514 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:42:05.514 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:42:05.515 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:42:05.515 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:42:05.233838Z", 2023-07-29T00:42:05.515 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:42:05.516 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:42:05.516 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:42:05.516 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:42:05.517 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:42:05.517 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:42:05.518 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:42:05.518 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:42:05.518 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:42:05.519 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:42:05.519 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:42:05.519 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:42:05.520 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:42:05.520 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:42:05.520 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:42:05.521 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:42:05.521 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:42:05.521 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:42:05.521 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:42:05.522 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T00:42:05.522 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:42:05.522 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:42:05.523 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:42:05.523 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T00:42:05.523 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:42:05.524 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:42:05.524 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:42:05.524 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:42:05.524 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:42:05.524 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:42:05.525 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:42:05.525 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:42:05.525 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:42:05.525 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:42:05Z" 2023-07-29T00:42:05.525 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:42:05.527 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690591325.233838_1080.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:42:05.233838Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:42:05Z" } 2023-07-29T00:42:05.527 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690591325.233838_1080.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:42:05.233838Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T00:42:05.528 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690591325.241299_1083.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T00:42:05.528 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690591325.233838_1080.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T00:42:05.529 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690591325.241299_1083.1 target=1_1690591325.233838_1080.1 2023-07-29T00:42:10.531 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:42:10.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:42:10.532 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:42:10.585 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:42:10.586 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:42:10.619 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:42:10.553+0000 7f1139a26880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:42:10.620 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:42:10.554+0000 7f1139a26880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:42:10.634 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:42:10.634 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:42:10.635 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:42:10.635 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:42:10.635 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:42:10.635 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:42:10.635 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:42:10.635 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:42:10.636 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:42:10.636 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:42:10.636 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:42:10.636 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:42:10.636 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:42:10.636 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T00:42:10.636 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:42:10.637 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:42:10.637 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:42:10.637 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T00:42:10.637 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:42:10.637 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:42:10.637 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:42:10.638 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:42:10.638 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:42:10.638 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:42:10.638 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:42:10.638 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591325.241299_1083.1", 2023-07-29T00:42:10.638 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:42:10.638 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:42:10.639 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:42:10.639 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:42:05.241299Z", 2023-07-29T00:42:10.639 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:42:10.639 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:42:10.639 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:42:10.639 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:42:10.640 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:42:10.640 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:42:10.640 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:42:10.640 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:42:10.640 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:42:10.640 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:42:10.641 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:42:10.641 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:42:10.641 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:42:10.641 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:42:10.641 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:42:10.641 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:42:10.641 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:42:10.642 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:42:10.642 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:42:10.642 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T00:42:10.642 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:42:10.642 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:42:10.642 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:42:10.643 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T00:42:10.643 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:42:10.643 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:42:10.643 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:42:10.643 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:42:10.643 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:42:10.644 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:42:10.644 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:42:10.644 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:42:10.644 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:42:10.644 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:42:10Z" 2023-07-29T00:42:10.645 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:42:10.646 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690591325.241299_1083.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:42:05.241299Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:42:10Z" } 2023-07-29T00:42:10.646 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690591325.241299_1083.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:42:05.241299Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T00:42:10.647 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690591325.241299_1083.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T00:42:10.647 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690591325.241299_1083.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T00:42:10.647 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T00:42:10.648 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-07-29T00:42:10.648 INFO:teuthology.orchestra.run:waiting for 300 2023-07-29T00:42:10.649 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:42:10.582+0000 7fbf0ed5f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 138046) UID: 0 2023-07-29T00:42:10.649 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:42:10.582+0000 7fbf14e34880 -1 shutting down 2023-07-29T00:42:14.075 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-07-29T00:42:16.750 INFO:tasks.rgw.c1.client.1:Stopped 2023-07-29T00:42:16.751 DEBUG:boto:path=/hiiqos-60/ 2023-07-29T00:42:16.751 DEBUG:boto:auth_path=/hiiqos-60/ 2023-07-29T00:42:16.751 DEBUG:boto:Method: HEAD 2023-07-29T00:42:16.752 DEBUG:boto:Path: /hiiqos-60/ 2023-07-29T00:42:16.752 DEBUG:boto:Data: 2023-07-29T00:42:16.753 DEBUG:boto:Headers: {} 2023-07-29T00:42:16.753 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:42:16.753 DEBUG:boto:Port: 8000 2023-07-29T00:42:16.754 DEBUG:boto:Params: {} 2023-07-29T00:42:16.754 DEBUG:boto:Token: None 2023-07-29T00:42:16.754 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:42:16 GMT /hiiqos-60/ 2023-07-29T00:42:16.755 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XOgLWy00W2fSRaoAa6De2QjjBdk= 2023-07-29T00:42:16.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:42:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XOgLWy00W2fSRaoAa6De2QjjBdk=', 'Content-Length': '0'} 2023-07-29T00:42:16.758 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e79a80a682822354-0064c46068-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:42:16.758 DEBUG:boto:path=/hiiqos-60/obj11 2023-07-29T00:42:16.759 DEBUG:boto:auth_path=/hiiqos-60/obj11 2023-07-29T00:42:16.759 DEBUG:boto:Method: PUT 2023-07-29T00:42:16.759 DEBUG:boto:Path: /hiiqos-60/obj11 2023-07-29T00:42:16.760 DEBUG:boto:Data: 2023-07-29T00:42:16.760 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:42:16.760 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:42:16.761 DEBUG:boto:Port: 8000 2023-07-29T00:42:16.761 DEBUG:boto:Params: {} 2023-07-29T00:42:16.761 DEBUG:boto:Token: None 2023-07-29T00:42:16.762 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:42:16 GMT /hiiqos-60/obj11 2023-07-29T00:42:16.762 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:iv0cLUudygr+ikOPEYbF/tNz5A0= 2023-07-29T00:42:16.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:42:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:iv0cLUudygr+ikOPEYbF/tNz5A0='} 2023-07-29T00:42:16.770 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001cd681a69a5a4e81-0064c46068-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:42:16.770 DEBUG:boto:path=/hiiqos-60/ 2023-07-29T00:42:16.771 DEBUG:boto:auth_path=/hiiqos-60/ 2023-07-29T00:42:16.771 DEBUG:boto:Method: HEAD 2023-07-29T00:42:16.772 DEBUG:boto:Path: /hiiqos-60/ 2023-07-29T00:42:16.772 DEBUG:boto:Data: 2023-07-29T00:42:16.772 DEBUG:boto:Headers: {} 2023-07-29T00:42:16.773 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:42:16.773 DEBUG:boto:Port: 8000 2023-07-29T00:42:16.773 DEBUG:boto:Params: {} 2023-07-29T00:42:16.774 DEBUG:boto:Token: None 2023-07-29T00:42:16.774 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:42:16 GMT /hiiqos-60/ 2023-07-29T00:42:16.774 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XOgLWy00W2fSRaoAa6De2QjjBdk= 2023-07-29T00:42:16.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:42:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XOgLWy00W2fSRaoAa6De2QjjBdk=', 'Content-Length': '0'} 2023-07-29T00:42:16.777 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e87005840337e4b6-0064c46068-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:42:16.777 DEBUG:boto:path=/hiiqos-60/obj22 2023-07-29T00:42:16.778 DEBUG:boto:auth_path=/hiiqos-60/obj22 2023-07-29T00:42:16.778 DEBUG:boto:Method: PUT 2023-07-29T00:42:16.779 DEBUG:boto:Path: /hiiqos-60/obj22 2023-07-29T00:42:16.779 DEBUG:boto:Data: 2023-07-29T00:42:16.779 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:42:16.780 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:42:16.780 DEBUG:boto:Port: 8000 2023-07-29T00:42:16.781 DEBUG:boto:Params: {} 2023-07-29T00:42:16.781 DEBUG:boto:Token: None 2023-07-29T00:42:16.781 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:42:16 GMT /hiiqos-60/obj22 2023-07-29T00:42:16.782 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OCgb1mNTM8fI0aOzpoklnegaLfM= 2023-07-29T00:42:16.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:42:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OCgb1mNTM8fI0aOzpoklnegaLfM='} 2023-07-29T00:42:16.824 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a826ced2f7107ba9-0064c46068-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:42:16.825 DEBUG:boto:path=/hiiqos-60/ 2023-07-29T00:42:16.825 DEBUG:boto:auth_path=/hiiqos-60/ 2023-07-29T00:42:16.826 DEBUG:boto:Method: HEAD 2023-07-29T00:42:16.826 DEBUG:boto:Path: /hiiqos-60/ 2023-07-29T00:42:16.826 DEBUG:boto:Data: 2023-07-29T00:42:16.827 DEBUG:boto:Headers: {} 2023-07-29T00:42:16.827 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:42:16.827 DEBUG:boto:Port: 8000 2023-07-29T00:42:16.828 DEBUG:boto:Params: {} 2023-07-29T00:42:16.828 DEBUG:boto:Token: None 2023-07-29T00:42:16.829 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:42:16 GMT /hiiqos-60/ 2023-07-29T00:42:16.829 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XOgLWy00W2fSRaoAa6De2QjjBdk= 2023-07-29T00:42:16.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:42:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XOgLWy00W2fSRaoAa6De2QjjBdk=', 'Content-Length': '0'} 2023-07-29T00:42:16.832 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f9b9d236466517c-0064c46068-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:42:16.833 DEBUG:boto:path=/hiiqos-60/obj33 2023-07-29T00:42:16.833 DEBUG:boto:auth_path=/hiiqos-60/obj33 2023-07-29T00:42:16.833 DEBUG:boto:Method: PUT 2023-07-29T00:42:16.834 DEBUG:boto:Path: /hiiqos-60/obj33 2023-07-29T00:42:16.834 DEBUG:boto:Data: 2023-07-29T00:42:16.834 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:42:16.835 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:42:16.835 DEBUG:boto:Port: 8000 2023-07-29T00:42:16.835 DEBUG:boto:Params: {} 2023-07-29T00:42:16.836 DEBUG:boto:Token: None 2023-07-29T00:42:16.836 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:42:16 GMT /hiiqos-60/obj33 2023-07-29T00:42:16.837 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LwV2QknVRhjOBeZbU+3u1UKxskM= 2023-07-29T00:42:16.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:42:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LwV2QknVRhjOBeZbU+3u1UKxskM='} 2023-07-29T00:42:16.843 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000046462f173a63b053-0064c46068-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:42:16.844 DEBUG:boto:path=/hiiqos-60/ 2023-07-29T00:42:16.844 DEBUG:boto:auth_path=/hiiqos-60/ 2023-07-29T00:42:16.844 DEBUG:boto:Method: HEAD 2023-07-29T00:42:16.845 DEBUG:boto:Path: /hiiqos-60/ 2023-07-29T00:42:16.845 DEBUG:boto:Data: 2023-07-29T00:42:16.845 DEBUG:boto:Headers: {} 2023-07-29T00:42:16.846 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:42:16.847 DEBUG:boto:Port: 8000 2023-07-29T00:42:16.847 DEBUG:boto:Params: {} 2023-07-29T00:42:16.847 DEBUG:boto:Token: None 2023-07-29T00:42:16.848 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:42:16 GMT /hiiqos-60/ 2023-07-29T00:42:16.848 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XOgLWy00W2fSRaoAa6De2QjjBdk= 2023-07-29T00:42:16.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:42:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XOgLWy00W2fSRaoAa6De2QjjBdk=', 'Content-Length': '0'} 2023-07-29T00:42:16.851 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a4f4bd9d6bc8dcc5-0064c46068-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:42:16.852 DEBUG:boto:path=/hiiqos-60/obj44 2023-07-29T00:42:16.852 DEBUG:boto:auth_path=/hiiqos-60/obj44 2023-07-29T00:42:16.852 DEBUG:boto:Method: PUT 2023-07-29T00:42:16.853 DEBUG:boto:Path: /hiiqos-60/obj44 2023-07-29T00:42:16.853 DEBUG:boto:Data: 2023-07-29T00:42:16.853 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:42:16.854 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:42:16.854 DEBUG:boto:Port: 8000 2023-07-29T00:42:16.854 DEBUG:boto:Params: {} 2023-07-29T00:42:16.855 DEBUG:boto:Token: None 2023-07-29T00:42:16.855 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:42:16 GMT /hiiqos-60/obj44 2023-07-29T00:42:16.856 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:C6pI2T7ruojGBK9WZA4yDBZTA3w= 2023-07-29T00:42:16.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:42:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:C6pI2T7ruojGBK9WZA4yDBZTA3w='} 2023-07-29T00:42:16.862 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005056cdf5a47d8277-0064c46068-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:42:16.862 DEBUG:boto:path=/hiiqos-60/ 2023-07-29T00:42:16.863 DEBUG:boto:auth_path=/hiiqos-60/ 2023-07-29T00:42:16.863 DEBUG:boto:Method: HEAD 2023-07-29T00:42:16.863 DEBUG:boto:Path: /hiiqos-60/ 2023-07-29T00:42:16.864 DEBUG:boto:Data: 2023-07-29T00:42:16.864 DEBUG:boto:Headers: {} 2023-07-29T00:42:16.864 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:42:16.865 DEBUG:boto:Port: 8000 2023-07-29T00:42:16.865 DEBUG:boto:Params: {} 2023-07-29T00:42:16.865 DEBUG:boto:Token: None 2023-07-29T00:42:16.866 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:42:16 GMT /hiiqos-60/ 2023-07-29T00:42:16.866 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XOgLWy00W2fSRaoAa6De2QjjBdk= 2023-07-29T00:42:16.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:42:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XOgLWy00W2fSRaoAa6De2QjjBdk=', 'Content-Length': '0'} 2023-07-29T00:42:16.869 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000777d869dc62d867b-0064c46068-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:42:16.870 DEBUG:boto:path=/hiiqos-60/obj55 2023-07-29T00:42:16.870 DEBUG:boto:auth_path=/hiiqos-60/obj55 2023-07-29T00:42:16.871 DEBUG:boto:Method: PUT 2023-07-29T00:42:16.871 DEBUG:boto:Path: /hiiqos-60/obj55 2023-07-29T00:42:16.871 DEBUG:boto:Data: 2023-07-29T00:42:16.872 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:42:16.872 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:42:16.872 DEBUG:boto:Port: 8000 2023-07-29T00:42:16.873 DEBUG:boto:Params: {} 2023-07-29T00:42:16.873 DEBUG:boto:Token: None 2023-07-29T00:42:16.873 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:42:16 GMT /hiiqos-60/obj55 2023-07-29T00:42:16.874 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2m9ciDPNproURa3Is+IHk17KT0M= 2023-07-29T00:42:16.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:42:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2m9ciDPNproURa3Is+IHk17KT0M='} 2023-07-29T00:42:16.916 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b0d284c956d0414f-0064c46068-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:42:16.917 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:42:16.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:42:16.918 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:42:16.972 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:42:16.972 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:42:17.005 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:42:16.938+0000 7f5f8ce12880 5 RGW-SYNC:data: waiting on data sync status lease 2023-07-29T00:42:17.006 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:42:16.940+0000 7f5f8ce12880 5 RGW-SYNC:data: acquired data sync status lease 2023-07-29T00:42:17.028 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-07-29T00:42:17.028 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-07-29T00:42:17.031 INFO:tasks.rgw.c1.client.1:Started 2023-07-29T00:42:17.031 DEBUG:teuthology.orchestra.run.smithi101:> curl http://smithi101.front.sepia.ceph.com:8001 2023-07-29T00:42:17.050 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-07-29T00:42:17.051 INFO:teuthology.orchestra.run.smithi101.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-07-29T00:42:17.051 INFO:teuthology.orchestra.run.smithi101.stderr: Dload Upload Total Spent Left Speed 2023-07-29T00:42:17.051 INFO:teuthology.orchestra.run.smithi101.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-07-29T00:42:17.052 INFO:teuthology.orchestra.run.smithi101.stderr:curl: (7) Failed to connect to smithi101.front.sepia.ceph.com port 8001: Connection refused 2023-07-29T00:42:18.053 DEBUG:teuthology.orchestra.run.smithi101:> curl http://smithi101.front.sepia.ceph.com:8001 2023-07-29T00:42:18.072 INFO:teuthology.orchestra.run.smithi101.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-07-29T00:42:18.073 INFO:teuthology.orchestra.run.smithi101.stderr: Dload Upload Total Spent Left Speed 2023-07-29T00:42:18.073 INFO:teuthology.orchestra.run.smithi101.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-07-29T00:42:18.074 INFO:teuthology.orchestra.run.smithi101.stdout:anonymous 2023-07-29T00:42:18.074 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-60 2023-07-29T00:42:18.074 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-60', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:42:18.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-60', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:42:18.075 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-60 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:42:18.167 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:42:18.168 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:42:18.202 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:42:18.135+0000 7f7e88149880 1 waiting to reach incremental sync.. 2023-07-29T00:42:19.781 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-07-29T00:42:23.203 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:42:23.136+0000 7f7e88149880 1 waiting to reach incremental sync.. 2023-07-29T00:42:28.203 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:42:28.137+0000 7f7e88149880 1 waiting to reach incremental sync.. 2023-07-29T00:42:33.204 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:42:33.137+0000 7f7e88149880 1 waiting to reach incremental sync.. 2023-07-29T00:42:38.205 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:42:38.138+0000 7f7e88149880 1 waiting to reach incremental sync.. 2023-07-29T00:42:43.205 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:42:43.138+0000 7f7e88149880 1 waiting to reach incremental sync.. 2023-07-29T00:42:48.206 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:42:48.139+0000 7f7e88149880 1 waiting to reach incremental sync.. 2023-07-29T00:42:53.207 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:42:53.139+0000 7f7e88149880 1 waiting to reach incremental sync.. 2023-07-29T00:42:57.773 INFO:tasks.rgw.c2.client.1.smithi139.stdout:2023-07-29T00:42:57.706+0000 7f5731a6a640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T00:42:58.207 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:42:58.140+0000 7f7e88149880 1 waiting to reach incremental sync.. 2023-07-29T00:43:03.208 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:43:03.141+0000 7f7e88149880 1 waiting to reach incremental sync.. 2023-07-29T00:43:08.209 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:43:08.141+0000 7f7e88149880 1 waiting to reach incremental sync.. 2023-07-29T00:43:13.208 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:43:13.141+0000 7f7e88149880 1 waiting to reach incremental sync.. 2023-07-29T00:43:18.209 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:43:18.142+0000 7f7e88149880 1 waiting to reach incremental sync.. 2023-07-29T00:43:23.209 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:43:23.142+0000 7f7e88149880 1 waiting to reach incremental sync.. 2023-07-29T00:43:28.210 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:43:28.143+0000 7f7e88149880 1 waiting to reach incremental sync.. 2023-07-29T00:43:33.211 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:43:33.143+0000 7f7e88149880 1 waiting to reach incremental sync.. 2023-07-29T00:43:38.212 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:43:38.144+0000 7f7e88149880 1 waiting to reach incremental sync.. 2023-07-29T00:43:43.212 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:43:43.144+0000 7f7e88149880 1 waiting to reach incremental sync.. 2023-07-29T00:43:48.213 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:43:48.145+0000 7f7e88149880 1 waiting to reach incremental sync.. 2023-07-29T00:43:53.213 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:43:53.146+0000 7f7e88149880 1 waiting to reach incremental sync.. 2023-07-29T00:43:58.214 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:43:58.147+0000 7f7e88149880 1 waiting to reach incremental sync.. 2023-07-29T00:44:03.215 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:44:03.147+0000 7f7e88149880 1 waiting to reach incremental sync.. 2023-07-29T00:44:08.216 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:44:08.148+0000 7f7e88149880 1 waiting to reach incremental sync.. 2023-07-29T00:44:13.216 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:44:13.148+0000 7f7e88149880 1 waiting to reach incremental sync.. 2023-07-29T00:44:18.217 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:44:18.149+0000 7f7e88149880 1 waiting to reach incremental sync.. 2023-07-29T00:44:23.218 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:44:23.150+0000 7f7e88149880 1 waiting to reach incremental sync.. 2023-07-29T00:44:28.219 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:44:28.150+0000 7f7e88149880 1 waiting to reach incremental sync.. 2023-07-29T00:44:33.219 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:44:33.151+0000 7f7e88149880 1 waiting to reach incremental sync.. 2023-07-29T00:44:38.220 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:44:38.152+0000 7f7e88149880 1 waiting to reach incremental sync.. 2023-07-29T00:44:43.221 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:44:43.152+0000 7f7e88149880 1 waiting to reach incremental sync.. 2023-07-29T00:44:48.221 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:44:48.153+0000 7f7e88149880 1 waiting to reach incremental sync.. 2023-07-29T00:44:53.223 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:44:53.155+0000 7f7e88149880 1 bucket sync caught up with source: 2023-07-29T00:44:53.224 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , 00000000001.89.6, 00000000002.98.6, 00000000002.42.6] 2023-07-29T00:44:53.224 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000001.89.6, 00000000002.98.6, 00000000002.42.6] 2023-07-29T00:44:53.224 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:44:53.155+0000 7f7e88149880 0 bucket checkpoint complete 2023-07-29T00:44:53.242 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-60 2023-07-29T00:44:53.242 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-60', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:44:53.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-60', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:44:53.243 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-60 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:44:53.301 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:44:53.302 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:44:53.338 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:44:53.269+0000 7f06dee16880 1 bucket sync caught up with source: 2023-07-29T00:44:53.338 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , 00000000001.945.6, 00000000002.1417.6, 00000000002.724.6] 2023-07-29T00:44:53.338 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000001.945.6, 00000000002.1417.6, 00000000002.724.6] 2023-07-29T00:44:53.338 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:44:53.269+0000 7f06dee16880 0 bucket checkpoint complete 2023-07-29T00:44:53.352 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-60 zones={a2, a1} 2023-07-29T00:44:53.353 DEBUG:boto:path=/hiiqos-60/ 2023-07-29T00:44:53.353 DEBUG:boto:auth_path=/hiiqos-60/ 2023-07-29T00:44:53.354 DEBUG:boto:Method: HEAD 2023-07-29T00:44:53.354 DEBUG:boto:Path: /hiiqos-60/ 2023-07-29T00:44:53.354 DEBUG:boto:Data: 2023-07-29T00:44:53.355 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.355 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:44:53.356 DEBUG:boto:Port: 8001 2023-07-29T00:44:53.356 DEBUG:boto:Params: {} 2023-07-29T00:44:53.357 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-29T00:44:53.357 DEBUG:boto:Token: None 2023-07-29T00:44:53.358 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/ 2023-07-29T00:44:53.358 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1vxrrOwQhcezBTDmyde1rxG++mY= 2023-07-29T00:44:53.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1vxrrOwQhcezBTDmyde1rxG++mY=', 'Content-Length': '0'} 2023-07-29T00:44:53.367 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f4688f5427ab033-0064c46105-1f88-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.367 DEBUG:boto:path=/hiiqos-60/ 2023-07-29T00:44:53.368 DEBUG:boto:auth_path=/hiiqos-60/ 2023-07-29T00:44:53.368 DEBUG:boto:Method: HEAD 2023-07-29T00:44:53.369 DEBUG:boto:Path: /hiiqos-60/ 2023-07-29T00:44:53.369 DEBUG:boto:Data: 2023-07-29T00:44:53.369 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.370 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:44:53.370 DEBUG:boto:Port: 8000 2023-07-29T00:44:53.371 DEBUG:boto:Params: {} 2023-07-29T00:44:53.371 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-29T00:44:53.371 DEBUG:boto:Token: None 2023-07-29T00:44:53.372 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/ 2023-07-29T00:44:53.372 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1vxrrOwQhcezBTDmyde1rxG++mY= 2023-07-29T00:44:53.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1vxrrOwQhcezBTDmyde1rxG++mY=', 'Content-Length': '0'} 2023-07-29T00:44:53.375 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008754a4706e9f7060-0064c46105-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.376 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-29T00:44:53.377 DEBUG:boto:path=/hiiqos-60/ 2023-07-29T00:44:53.377 DEBUG:boto:auth_path=/hiiqos-60/ 2023-07-29T00:44:53.377 DEBUG:boto:path=/hiiqos-60/?versions&max-keys=999 2023-07-29T00:44:53.378 DEBUG:boto:auth_path=/hiiqos-60/?versions&max-keys=999 2023-07-29T00:44:53.378 DEBUG:boto:Method: GET 2023-07-29T00:44:53.378 DEBUG:boto:Path: /hiiqos-60/?versions&max-keys=999 2023-07-29T00:44:53.379 DEBUG:boto:Data: 2023-07-29T00:44:53.379 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.379 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:44:53.380 DEBUG:boto:Port: 8001 2023-07-29T00:44:53.380 DEBUG:boto:Params: {} 2023-07-29T00:44:53.381 DEBUG:boto:Token: None 2023-07-29T00:44:53.381 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/?versions 2023-07-29T00:44:53.381 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Er3eSaAsUjYomH3fkc7kRNckVdM= 2023-07-29T00:44:53.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Er3eSaAsUjYomH3fkc7kRNckVdM=', 'Content-Length': '0'} 2023-07-29T00:44:53.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000801288b901547f63-0064c46105-1f88-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.384 DEBUG:boto:b'hiiqos-60999falseobj11nulltrue2023-07-29T00:42:16.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:42:16.718Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:42:16.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:42:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:42:16.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:44:53.385 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-07-29T00:44:53.386 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-07-29T00:44:53.386 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-07-29T00:44:53.386 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-07-29T00:44:53.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-07-29T00:44:53.387 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-29T00:44:53.388 DEBUG:boto:path=/hiiqos-60/ 2023-07-29T00:44:53.388 DEBUG:boto:auth_path=/hiiqos-60/ 2023-07-29T00:44:53.388 DEBUG:boto:path=/hiiqos-60/?versions&max-keys=999 2023-07-29T00:44:53.389 DEBUG:boto:auth_path=/hiiqos-60/?versions&max-keys=999 2023-07-29T00:44:53.389 DEBUG:boto:Method: GET 2023-07-29T00:44:53.389 DEBUG:boto:Path: /hiiqos-60/?versions&max-keys=999 2023-07-29T00:44:53.390 DEBUG:boto:Data: 2023-07-29T00:44:53.390 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.390 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:44:53.391 DEBUG:boto:Port: 8000 2023-07-29T00:44:53.391 DEBUG:boto:Params: {} 2023-07-29T00:44:53.391 DEBUG:boto:Token: None 2023-07-29T00:44:53.392 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/?versions 2023-07-29T00:44:53.392 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Er3eSaAsUjYomH3fkc7kRNckVdM= 2023-07-29T00:44:53.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Er3eSaAsUjYomH3fkc7kRNckVdM=', 'Content-Length': '0'} 2023-07-29T00:44:53.395 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfa04fbe8d28ee06-0064c46105-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.396 DEBUG:boto:b'hiiqos-60999falseobj11nulltrue2023-07-29T00:42:16.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:42:16.718Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:42:16.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:42:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:42:16.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:44:53.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-07-29T00:44:53.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-07-29T00:44:53.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-07-29T00:44:53.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-07-29T00:44:53.398 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-07-29T00:44:53.398 DEBUG:boto:path=/hiiqos-60/ 2023-07-29T00:44:53.399 DEBUG:boto:auth_path=/hiiqos-60/ 2023-07-29T00:44:53.399 DEBUG:boto:path=/hiiqos-60/?versions&max-keys=999 2023-07-29T00:44:53.399 DEBUG:boto:auth_path=/hiiqos-60/?versions&max-keys=999 2023-07-29T00:44:53.400 DEBUG:boto:Method: GET 2023-07-29T00:44:53.400 DEBUG:boto:Path: /hiiqos-60/?versions&max-keys=999 2023-07-29T00:44:53.400 DEBUG:boto:Data: 2023-07-29T00:44:53.401 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.401 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:44:53.401 DEBUG:boto:Port: 8001 2023-07-29T00:44:53.402 DEBUG:boto:Params: {} 2023-07-29T00:44:53.402 DEBUG:boto:Token: None 2023-07-29T00:44:53.402 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/?versions 2023-07-29T00:44:53.403 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Er3eSaAsUjYomH3fkc7kRNckVdM= 2023-07-29T00:44:53.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Er3eSaAsUjYomH3fkc7kRNckVdM=', 'Content-Length': '0'} 2023-07-29T00:44:53.406 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002cab10c8730d7a0-0064c46105-1f88-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.406 DEBUG:boto:b'hiiqos-60999falseobj11nulltrue2023-07-29T00:42:16.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:42:16.718Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:42:16.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:42:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:42:16.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:44:53.407 DEBUG:boto:path=/hiiqos-60/ 2023-07-29T00:44:53.408 DEBUG:boto:auth_path=/hiiqos-60/ 2023-07-29T00:44:53.408 DEBUG:boto:path=/hiiqos-60/?versions&max-keys=999 2023-07-29T00:44:53.408 DEBUG:boto:auth_path=/hiiqos-60/?versions&max-keys=999 2023-07-29T00:44:53.409 DEBUG:boto:Method: GET 2023-07-29T00:44:53.409 DEBUG:boto:Path: /hiiqos-60/?versions&max-keys=999 2023-07-29T00:44:53.409 DEBUG:boto:Data: 2023-07-29T00:44:53.410 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.410 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:44:53.410 DEBUG:boto:Port: 8000 2023-07-29T00:44:53.411 DEBUG:boto:Params: {} 2023-07-29T00:44:53.411 DEBUG:boto:Token: None 2023-07-29T00:44:53.411 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/?versions 2023-07-29T00:44:53.412 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Er3eSaAsUjYomH3fkc7kRNckVdM= 2023-07-29T00:44:53.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Er3eSaAsUjYomH3fkc7kRNckVdM=', 'Content-Length': '0'} 2023-07-29T00:44:53.415 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c7318fb1f105738-0064c46105-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.415 DEBUG:boto:b'hiiqos-60999falseobj11nulltrue2023-07-29T00:42:16.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:42:16.718Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:42:16.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:42:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:42:16.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:44:53.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-07-29T00:44:53.416 DEBUG:boto:path=/hiiqos-60/obj11 2023-07-29T00:44:53.417 DEBUG:boto:auth_path=/hiiqos-60/obj11 2023-07-29T00:44:53.417 DEBUG:boto:path=/hiiqos-60/obj11?versionId=null 2023-07-29T00:44:53.417 DEBUG:boto:auth_path=/hiiqos-60/obj11?versionId=null 2023-07-29T00:44:53.418 DEBUG:boto:Method: GET 2023-07-29T00:44:53.418 DEBUG:boto:Path: /hiiqos-60/obj11?versionId=null 2023-07-29T00:44:53.418 DEBUG:boto:Data: 2023-07-29T00:44:53.419 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.419 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:44:53.419 DEBUG:boto:Port: 8001 2023-07-29T00:44:53.420 DEBUG:boto:Params: {} 2023-07-29T00:44:53.420 DEBUG:boto:Token: None 2023-07-29T00:44:53.421 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/obj11?versionId=null 2023-07-29T00:44:53.421 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:y07T9voLvon9Dv5LVRGT0itojdg= 2023-07-29T00:44:53.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:y07T9voLvon9Dv5LVRGT0itojdg=', 'Content-Length': '0'} 2023-07-29T00:44:53.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-60:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b5c0f4ee4743453-0064c46105-1f88-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.424 DEBUG:boto:path=/hiiqos-60/obj11 2023-07-29T00:44:53.425 DEBUG:boto:auth_path=/hiiqos-60/obj11 2023-07-29T00:44:53.425 DEBUG:boto:path=/hiiqos-60/obj11?versionId=null 2023-07-29T00:44:53.426 DEBUG:boto:auth_path=/hiiqos-60/obj11?versionId=null 2023-07-29T00:44:53.426 DEBUG:boto:Method: GET 2023-07-29T00:44:53.426 DEBUG:boto:Path: /hiiqos-60/obj11?versionId=null 2023-07-29T00:44:53.427 DEBUG:boto:Data: 2023-07-29T00:44:53.427 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.428 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:44:53.428 DEBUG:boto:Port: 8000 2023-07-29T00:44:53.428 DEBUG:boto:Params: {} 2023-07-29T00:44:53.429 DEBUG:boto:Token: None 2023-07-29T00:44:53.429 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/obj11?versionId=null 2023-07-29T00:44:53.430 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:y07T9voLvon9Dv5LVRGT0itojdg= 2023-07-29T00:44:53.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:y07T9voLvon9Dv5LVRGT0itojdg=', 'Content-Length': '0'} 2023-07-29T00:44:53.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c85d7da14253118c-0064c46105-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.432 DEBUG:boto:path=/hiiqos-60/obj11 2023-07-29T00:44:53.433 DEBUG:boto:auth_path=/hiiqos-60/obj11 2023-07-29T00:44:53.433 DEBUG:boto:path=/hiiqos-60/obj11?versionId=null 2023-07-29T00:44:53.433 DEBUG:boto:auth_path=/hiiqos-60/obj11?versionId=null 2023-07-29T00:44:53.434 DEBUG:boto:Method: HEAD 2023-07-29T00:44:53.434 DEBUG:boto:Path: /hiiqos-60/obj11?versionId=null 2023-07-29T00:44:53.435 DEBUG:boto:Data: 2023-07-29T00:44:53.435 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.436 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:44:53.436 DEBUG:boto:Port: 8001 2023-07-29T00:44:53.437 DEBUG:boto:Params: {} 2023-07-29T00:44:53.437 DEBUG:boto:Token: None 2023-07-29T00:44:53.437 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/obj11?versionId=null 2023-07-29T00:44:53.438 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HEFMJNTuy3xyU078Se5l5QHmmLs= 2023-07-29T00:44:53.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HEFMJNTuy3xyU078Se5l5QHmmLs=', 'Content-Length': '0'} 2023-07-29T00:44:53.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-60:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035b5a474f9aa2335-0064c46105-1f88-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.441 DEBUG:boto:path=/hiiqos-60/obj11 2023-07-29T00:44:53.442 DEBUG:boto:auth_path=/hiiqos-60/obj11 2023-07-29T00:44:53.442 DEBUG:boto:path=/hiiqos-60/obj11?versionId=null 2023-07-29T00:44:53.442 DEBUG:boto:auth_path=/hiiqos-60/obj11?versionId=null 2023-07-29T00:44:53.443 DEBUG:boto:Method: HEAD 2023-07-29T00:44:53.443 DEBUG:boto:Path: /hiiqos-60/obj11?versionId=null 2023-07-29T00:44:53.443 DEBUG:boto:Data: 2023-07-29T00:44:53.444 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.444 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:44:53.444 DEBUG:boto:Port: 8000 2023-07-29T00:44:53.445 DEBUG:boto:Params: {} 2023-07-29T00:44:53.445 DEBUG:boto:Token: None 2023-07-29T00:44:53.445 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/obj11?versionId=null 2023-07-29T00:44:53.446 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HEFMJNTuy3xyU078Se5l5QHmmLs= 2023-07-29T00:44:53.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HEFMJNTuy3xyU078Se5l5QHmmLs=', 'Content-Length': '0'} 2023-07-29T00:44:53.448 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013fa3dd9d3fcc528-0064c46105-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-07-29T00:44:53.449 DEBUG:boto:path=/hiiqos-60/obj11 2023-07-29T00:44:53.450 DEBUG:boto:auth_path=/hiiqos-60/obj11 2023-07-29T00:44:53.450 DEBUG:boto:path=/hiiqos-60/obj11?versionId=null 2023-07-29T00:44:53.451 DEBUG:boto:auth_path=/hiiqos-60/obj11?versionId=null 2023-07-29T00:44:53.451 DEBUG:boto:Method: GET 2023-07-29T00:44:53.451 DEBUG:boto:Path: /hiiqos-60/obj11?versionId=null 2023-07-29T00:44:53.452 DEBUG:boto:Data: 2023-07-29T00:44:53.452 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.453 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:44:53.453 DEBUG:boto:Port: 8001 2023-07-29T00:44:53.453 DEBUG:boto:Params: {} 2023-07-29T00:44:53.454 DEBUG:boto:Token: None 2023-07-29T00:44:53.454 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/obj11?versionId=null 2023-07-29T00:44:53.455 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:y07T9voLvon9Dv5LVRGT0itojdg= 2023-07-29T00:44:53.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:y07T9voLvon9Dv5LVRGT0itojdg=', 'Content-Length': '0'} 2023-07-29T00:44:53.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-60:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005667fcdb3074c92d-0064c46105-1f88-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.457 DEBUG:boto:path=/hiiqos-60/obj11 2023-07-29T00:44:53.458 DEBUG:boto:auth_path=/hiiqos-60/obj11 2023-07-29T00:44:53.458 DEBUG:boto:path=/hiiqos-60/obj11?versionId=null 2023-07-29T00:44:53.458 DEBUG:boto:auth_path=/hiiqos-60/obj11?versionId=null 2023-07-29T00:44:53.459 DEBUG:boto:Method: GET 2023-07-29T00:44:53.459 DEBUG:boto:Path: /hiiqos-60/obj11?versionId=null 2023-07-29T00:44:53.459 DEBUG:boto:Data: 2023-07-29T00:44:53.460 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.460 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:44:53.461 DEBUG:boto:Port: 8000 2023-07-29T00:44:53.461 DEBUG:boto:Params: {} 2023-07-29T00:44:53.461 DEBUG:boto:Token: None 2023-07-29T00:44:53.462 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/obj11?versionId=null 2023-07-29T00:44:53.462 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:y07T9voLvon9Dv5LVRGT0itojdg= 2023-07-29T00:44:53.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:y07T9voLvon9Dv5LVRGT0itojdg=', 'Content-Length': '0'} 2023-07-29T00:44:53.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec0436d7e14c26aa-0064c46105-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.465 DEBUG:boto:path=/hiiqos-60/ 2023-07-29T00:44:53.465 DEBUG:boto:auth_path=/hiiqos-60/ 2023-07-29T00:44:53.466 DEBUG:boto:path=/hiiqos-60/?prefix=obj11 2023-07-29T00:44:53.466 DEBUG:boto:auth_path=/hiiqos-60/?prefix=obj11 2023-07-29T00:44:53.467 DEBUG:boto:Method: GET 2023-07-29T00:44:53.467 DEBUG:boto:Path: /hiiqos-60/?prefix=obj11 2023-07-29T00:44:53.468 DEBUG:boto:Data: 2023-07-29T00:44:53.468 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.468 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:44:53.469 DEBUG:boto:Port: 8001 2023-07-29T00:44:53.469 DEBUG:boto:Params: {} 2023-07-29T00:44:53.469 DEBUG:boto:Token: None 2023-07-29T00:44:53.470 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/ 2023-07-29T00:44:53.470 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pkyqY2jYdDPQ0N54FHpFRz+oyyU= 2023-07-29T00:44:53.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pkyqY2jYdDPQ0N54FHpFRz+oyyU=', 'Content-Length': '0'} 2023-07-29T00:44:53.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002bad8529eb4dac3d-0064c46105-1f88-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.516 DEBUG:boto:b'hiiqos-60obj111000falseobj112023-07-29T00:42:16.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:44:53.517 DEBUG:boto:path=/hiiqos-60/ 2023-07-29T00:44:53.518 DEBUG:boto:auth_path=/hiiqos-60/ 2023-07-29T00:44:53.518 DEBUG:boto:path=/hiiqos-60/?prefix=obj11 2023-07-29T00:44:53.518 DEBUG:boto:auth_path=/hiiqos-60/?prefix=obj11 2023-07-29T00:44:53.519 DEBUG:boto:Method: GET 2023-07-29T00:44:53.519 DEBUG:boto:Path: /hiiqos-60/?prefix=obj11 2023-07-29T00:44:53.519 DEBUG:boto:Data: 2023-07-29T00:44:53.520 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.520 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:44:53.521 DEBUG:boto:Port: 8000 2023-07-29T00:44:53.521 DEBUG:boto:Params: {} 2023-07-29T00:44:53.522 DEBUG:boto:Token: None 2023-07-29T00:44:53.522 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/ 2023-07-29T00:44:53.523 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pkyqY2jYdDPQ0N54FHpFRz+oyyU= 2023-07-29T00:44:53.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pkyqY2jYdDPQ0N54FHpFRz+oyyU=', 'Content-Length': '0'} 2023-07-29T00:44:53.525 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001088308ff8380c3b-0064c46105-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.526 DEBUG:boto:b'hiiqos-60obj111000falseobj112023-07-29T00:42:16.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:44:53.526 DEBUG:boto:path=/hiiqos-60/obj11 2023-07-29T00:44:53.527 DEBUG:boto:auth_path=/hiiqos-60/obj11 2023-07-29T00:44:53.527 DEBUG:boto:Method: HEAD 2023-07-29T00:44:53.528 DEBUG:boto:Path: /hiiqos-60/obj11 2023-07-29T00:44:53.528 DEBUG:boto:Data: 2023-07-29T00:44:53.528 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.529 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:44:53.529 DEBUG:boto:Port: 8001 2023-07-29T00:44:53.530 DEBUG:boto:Params: {} 2023-07-29T00:44:53.531 DEBUG:boto:Token: None 2023-07-29T00:44:53.531 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/obj11 2023-07-29T00:44:53.532 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:71aKWX74QzJygpm61rJMnhSLcaU= 2023-07-29T00:44:53.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:71aKWX74QzJygpm61rJMnhSLcaU=', 'Content-Length': '0'} 2023-07-29T00:44:53.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-60:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c844eb29284c80e-0064c46105-1f88-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.535 DEBUG:boto:path=/hiiqos-60/obj11 2023-07-29T00:44:53.535 DEBUG:boto:auth_path=/hiiqos-60/obj11 2023-07-29T00:44:53.535 DEBUG:boto:Method: HEAD 2023-07-29T00:44:53.536 DEBUG:boto:Path: /hiiqos-60/obj11 2023-07-29T00:44:53.536 DEBUG:boto:Data: 2023-07-29T00:44:53.536 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.537 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:44:53.537 DEBUG:boto:Port: 8000 2023-07-29T00:44:53.537 DEBUG:boto:Params: {} 2023-07-29T00:44:53.538 DEBUG:boto:Token: None 2023-07-29T00:44:53.538 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/obj11 2023-07-29T00:44:53.539 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:71aKWX74QzJygpm61rJMnhSLcaU= 2023-07-29T00:44:53.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:71aKWX74QzJygpm61rJMnhSLcaU=', 'Content-Length': '0'} 2023-07-29T00:44:53.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cac44dd21094552d-0064c46105-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-07-29T00:44:53.542 DEBUG:boto:path=/hiiqos-60/obj11 2023-07-29T00:44:53.542 DEBUG:boto:auth_path=/hiiqos-60/obj11 2023-07-29T00:44:53.543 DEBUG:boto:Method: GET 2023-07-29T00:44:53.543 DEBUG:boto:Path: /hiiqos-60/obj11 2023-07-29T00:44:53.543 DEBUG:boto:Data: 2023-07-29T00:44:53.544 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.544 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:44:53.545 DEBUG:boto:Port: 8001 2023-07-29T00:44:53.545 DEBUG:boto:Params: {} 2023-07-29T00:44:53.545 DEBUG:boto:Token: None 2023-07-29T00:44:53.546 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/obj11 2023-07-29T00:44:53.546 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:U/83s/g3eYqX0QlpkrgQ7g1nMb0= 2023-07-29T00:44:53.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:U/83s/g3eYqX0QlpkrgQ7g1nMb0=', 'Content-Length': '0'} 2023-07-29T00:44:53.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-60:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbe0a792c4b47dfd-0064c46105-1f88-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.549 DEBUG:boto:path=/hiiqos-60/obj11 2023-07-29T00:44:53.549 DEBUG:boto:auth_path=/hiiqos-60/obj11 2023-07-29T00:44:53.550 DEBUG:boto:Method: GET 2023-07-29T00:44:53.550 DEBUG:boto:Path: /hiiqos-60/obj11 2023-07-29T00:44:53.551 DEBUG:boto:Data: 2023-07-29T00:44:53.551 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.552 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:44:53.552 DEBUG:boto:Port: 8000 2023-07-29T00:44:53.552 DEBUG:boto:Params: {} 2023-07-29T00:44:53.553 DEBUG:boto:Token: None 2023-07-29T00:44:53.553 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/obj11 2023-07-29T00:44:53.554 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:U/83s/g3eYqX0QlpkrgQ7g1nMb0= 2023-07-29T00:44:53.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:U/83s/g3eYqX0QlpkrgQ7g1nMb0=', 'Content-Length': '0'} 2023-07-29T00:44:53.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081daa2656a4ef0ee-0064c46105-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.596 DEBUG:boto:path=/hiiqos-60/ 2023-07-29T00:44:53.597 DEBUG:boto:auth_path=/hiiqos-60/ 2023-07-29T00:44:53.597 DEBUG:boto:path=/hiiqos-60/?prefix=obj11 2023-07-29T00:44:53.598 DEBUG:boto:auth_path=/hiiqos-60/?prefix=obj11 2023-07-29T00:44:53.598 DEBUG:boto:Method: GET 2023-07-29T00:44:53.598 DEBUG:boto:Path: /hiiqos-60/?prefix=obj11 2023-07-29T00:44:53.599 DEBUG:boto:Data: 2023-07-29T00:44:53.599 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.600 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:44:53.600 DEBUG:boto:Port: 8001 2023-07-29T00:44:53.600 DEBUG:boto:Params: {} 2023-07-29T00:44:53.601 DEBUG:boto:Token: None 2023-07-29T00:44:53.601 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/ 2023-07-29T00:44:53.602 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pkyqY2jYdDPQ0N54FHpFRz+oyyU= 2023-07-29T00:44:53.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pkyqY2jYdDPQ0N54FHpFRz+oyyU=', 'Content-Length': '0'} 2023-07-29T00:44:53.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032d76c88eed0320c-0064c46105-1f88-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.605 DEBUG:boto:b'hiiqos-60obj111000falseobj112023-07-29T00:42:16.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:44:53.606 DEBUG:boto:path=/hiiqos-60/ 2023-07-29T00:44:53.606 DEBUG:boto:auth_path=/hiiqos-60/ 2023-07-29T00:44:53.606 DEBUG:boto:path=/hiiqos-60/?prefix=obj11 2023-07-29T00:44:53.607 DEBUG:boto:auth_path=/hiiqos-60/?prefix=obj11 2023-07-29T00:44:53.607 DEBUG:boto:Method: GET 2023-07-29T00:44:53.608 DEBUG:boto:Path: /hiiqos-60/?prefix=obj11 2023-07-29T00:44:53.608 DEBUG:boto:Data: 2023-07-29T00:44:53.608 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.609 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:44:53.609 DEBUG:boto:Port: 8000 2023-07-29T00:44:53.609 DEBUG:boto:Params: {} 2023-07-29T00:44:53.610 DEBUG:boto:Token: None 2023-07-29T00:44:53.610 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/ 2023-07-29T00:44:53.611 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pkyqY2jYdDPQ0N54FHpFRz+oyyU= 2023-07-29T00:44:53.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pkyqY2jYdDPQ0N54FHpFRz+oyyU=', 'Content-Length': '0'} 2023-07-29T00:44:53.613 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052a9e7aed6ace0f9-0064c46105-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.614 DEBUG:boto:b'hiiqos-60obj111000falseobj112023-07-29T00:42:16.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:44:53.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-07-29T00:44:53.615 DEBUG:boto:path=/hiiqos-60/obj22 2023-07-29T00:44:53.615 DEBUG:boto:auth_path=/hiiqos-60/obj22 2023-07-29T00:44:53.616 DEBUG:boto:path=/hiiqos-60/obj22?versionId=null 2023-07-29T00:44:53.616 DEBUG:boto:auth_path=/hiiqos-60/obj22?versionId=null 2023-07-29T00:44:53.617 DEBUG:boto:Method: GET 2023-07-29T00:44:53.617 DEBUG:boto:Path: /hiiqos-60/obj22?versionId=null 2023-07-29T00:44:53.617 DEBUG:boto:Data: 2023-07-29T00:44:53.618 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.618 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:44:53.618 DEBUG:boto:Port: 8001 2023-07-29T00:44:53.619 DEBUG:boto:Params: {} 2023-07-29T00:44:53.619 DEBUG:boto:Token: None 2023-07-29T00:44:53.620 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/obj22?versionId=null 2023-07-29T00:44:53.620 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oSwZnB7dVeAWwF9VbsxWzAAGt8c= 2023-07-29T00:44:53.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oSwZnB7dVeAWwF9VbsxWzAAGt8c=', 'Content-Length': '0'} 2023-07-29T00:44:53.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-60:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006257e91c69b46335-0064c46105-1f88-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.623 DEBUG:boto:path=/hiiqos-60/obj22 2023-07-29T00:44:53.623 DEBUG:boto:auth_path=/hiiqos-60/obj22 2023-07-29T00:44:53.624 DEBUG:boto:path=/hiiqos-60/obj22?versionId=null 2023-07-29T00:44:53.624 DEBUG:boto:auth_path=/hiiqos-60/obj22?versionId=null 2023-07-29T00:44:53.625 DEBUG:boto:Method: GET 2023-07-29T00:44:53.625 DEBUG:boto:Path: /hiiqos-60/obj22?versionId=null 2023-07-29T00:44:53.625 DEBUG:boto:Data: 2023-07-29T00:44:53.626 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.626 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:44:53.626 DEBUG:boto:Port: 8000 2023-07-29T00:44:53.627 DEBUG:boto:Params: {} 2023-07-29T00:44:53.627 DEBUG:boto:Token: None 2023-07-29T00:44:53.627 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/obj22?versionId=null 2023-07-29T00:44:53.628 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oSwZnB7dVeAWwF9VbsxWzAAGt8c= 2023-07-29T00:44:53.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oSwZnB7dVeAWwF9VbsxWzAAGt8c=', 'Content-Length': '0'} 2023-07-29T00:44:53.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094e839dbe0beb1e0-0064c46105-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.631 DEBUG:boto:path=/hiiqos-60/obj22 2023-07-29T00:44:53.632 DEBUG:boto:auth_path=/hiiqos-60/obj22 2023-07-29T00:44:53.632 DEBUG:boto:path=/hiiqos-60/obj22?versionId=null 2023-07-29T00:44:53.632 DEBUG:boto:auth_path=/hiiqos-60/obj22?versionId=null 2023-07-29T00:44:53.633 DEBUG:boto:Method: HEAD 2023-07-29T00:44:53.633 DEBUG:boto:Path: /hiiqos-60/obj22?versionId=null 2023-07-29T00:44:53.634 DEBUG:boto:Data: 2023-07-29T00:44:53.634 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.634 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:44:53.635 DEBUG:boto:Port: 8001 2023-07-29T00:44:53.635 DEBUG:boto:Params: {} 2023-07-29T00:44:53.635 DEBUG:boto:Token: None 2023-07-29T00:44:53.636 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/obj22?versionId=null 2023-07-29T00:44:53.636 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZmHAW7VJQ+ZNzHSlYIkawrq3o2E= 2023-07-29T00:44:53.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZmHAW7VJQ+ZNzHSlYIkawrq3o2E=', 'Content-Length': '0'} 2023-07-29T00:44:53.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-60:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c02da9d4eb1e2a89-0064c46105-1f88-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.639 DEBUG:boto:path=/hiiqos-60/obj22 2023-07-29T00:44:53.639 DEBUG:boto:auth_path=/hiiqos-60/obj22 2023-07-29T00:44:53.640 DEBUG:boto:path=/hiiqos-60/obj22?versionId=null 2023-07-29T00:44:53.640 DEBUG:boto:auth_path=/hiiqos-60/obj22?versionId=null 2023-07-29T00:44:53.640 DEBUG:boto:Method: HEAD 2023-07-29T00:44:53.641 DEBUG:boto:Path: /hiiqos-60/obj22?versionId=null 2023-07-29T00:44:53.641 DEBUG:boto:Data: 2023-07-29T00:44:53.641 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.642 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:44:53.642 DEBUG:boto:Port: 8000 2023-07-29T00:44:53.642 DEBUG:boto:Params: {} 2023-07-29T00:44:53.643 DEBUG:boto:Token: None 2023-07-29T00:44:53.643 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/obj22?versionId=null 2023-07-29T00:44:53.644 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZmHAW7VJQ+ZNzHSlYIkawrq3o2E= 2023-07-29T00:44:53.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZmHAW7VJQ+ZNzHSlYIkawrq3o2E=', 'Content-Length': '0'} 2023-07-29T00:44:53.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000376bdb76367bcfe6-0064c46105-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-07-29T00:44:53.647 DEBUG:boto:path=/hiiqos-60/obj22 2023-07-29T00:44:53.647 DEBUG:boto:auth_path=/hiiqos-60/obj22 2023-07-29T00:44:53.647 DEBUG:boto:path=/hiiqos-60/obj22?versionId=null 2023-07-29T00:44:53.648 DEBUG:boto:auth_path=/hiiqos-60/obj22?versionId=null 2023-07-29T00:44:53.648 DEBUG:boto:Method: GET 2023-07-29T00:44:53.649 DEBUG:boto:Path: /hiiqos-60/obj22?versionId=null 2023-07-29T00:44:53.649 DEBUG:boto:Data: 2023-07-29T00:44:53.649 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.650 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:44:53.650 DEBUG:boto:Port: 8001 2023-07-29T00:44:53.650 DEBUG:boto:Params: {} 2023-07-29T00:44:53.651 DEBUG:boto:Token: None 2023-07-29T00:44:53.651 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/obj22?versionId=null 2023-07-29T00:44:53.652 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oSwZnB7dVeAWwF9VbsxWzAAGt8c= 2023-07-29T00:44:53.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oSwZnB7dVeAWwF9VbsxWzAAGt8c=', 'Content-Length': '0'} 2023-07-29T00:44:53.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-60:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fb766b7c495a46f-0064c46105-1f88-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.654 DEBUG:boto:path=/hiiqos-60/obj22 2023-07-29T00:44:53.655 DEBUG:boto:auth_path=/hiiqos-60/obj22 2023-07-29T00:44:53.655 DEBUG:boto:path=/hiiqos-60/obj22?versionId=null 2023-07-29T00:44:53.656 DEBUG:boto:auth_path=/hiiqos-60/obj22?versionId=null 2023-07-29T00:44:53.656 DEBUG:boto:Method: GET 2023-07-29T00:44:53.656 DEBUG:boto:Path: /hiiqos-60/obj22?versionId=null 2023-07-29T00:44:53.657 DEBUG:boto:Data: 2023-07-29T00:44:53.657 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.657 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:44:53.658 DEBUG:boto:Port: 8000 2023-07-29T00:44:53.658 DEBUG:boto:Params: {} 2023-07-29T00:44:53.659 DEBUG:boto:Token: None 2023-07-29T00:44:53.659 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/obj22?versionId=null 2023-07-29T00:44:53.660 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oSwZnB7dVeAWwF9VbsxWzAAGt8c= 2023-07-29T00:44:53.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oSwZnB7dVeAWwF9VbsxWzAAGt8c=', 'Content-Length': '0'} 2023-07-29T00:44:53.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e65cf81fa9db9608-0064c46105-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.662 DEBUG:boto:path=/hiiqos-60/ 2023-07-29T00:44:53.663 DEBUG:boto:auth_path=/hiiqos-60/ 2023-07-29T00:44:53.663 DEBUG:boto:path=/hiiqos-60/?prefix=obj22 2023-07-29T00:44:53.663 DEBUG:boto:auth_path=/hiiqos-60/?prefix=obj22 2023-07-29T00:44:53.664 DEBUG:boto:Method: GET 2023-07-29T00:44:53.664 DEBUG:boto:Path: /hiiqos-60/?prefix=obj22 2023-07-29T00:44:53.665 DEBUG:boto:Data: 2023-07-29T00:44:53.665 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.665 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:44:53.666 DEBUG:boto:Port: 8001 2023-07-29T00:44:53.666 DEBUG:boto:Params: {} 2023-07-29T00:44:53.666 DEBUG:boto:Token: None 2023-07-29T00:44:53.667 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/ 2023-07-29T00:44:53.667 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pkyqY2jYdDPQ0N54FHpFRz+oyyU= 2023-07-29T00:44:53.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pkyqY2jYdDPQ0N54FHpFRz+oyyU=', 'Content-Length': '0'} 2023-07-29T00:44:53.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f35d35d10b85331-0064c46105-1f88-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.670 DEBUG:boto:b'hiiqos-60obj221000falseobj222023-07-29T00:42:16.718Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:44:53.671 DEBUG:boto:path=/hiiqos-60/ 2023-07-29T00:44:53.672 DEBUG:boto:auth_path=/hiiqos-60/ 2023-07-29T00:44:53.672 DEBUG:boto:path=/hiiqos-60/?prefix=obj22 2023-07-29T00:44:53.672 DEBUG:boto:auth_path=/hiiqos-60/?prefix=obj22 2023-07-29T00:44:53.673 DEBUG:boto:Method: GET 2023-07-29T00:44:53.673 DEBUG:boto:Path: /hiiqos-60/?prefix=obj22 2023-07-29T00:44:53.674 DEBUG:boto:Data: 2023-07-29T00:44:53.674 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.674 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:44:53.675 DEBUG:boto:Port: 8000 2023-07-29T00:44:53.675 DEBUG:boto:Params: {} 2023-07-29T00:44:53.676 DEBUG:boto:Token: None 2023-07-29T00:44:53.676 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/ 2023-07-29T00:44:53.677 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pkyqY2jYdDPQ0N54FHpFRz+oyyU= 2023-07-29T00:44:53.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pkyqY2jYdDPQ0N54FHpFRz+oyyU=', 'Content-Length': '0'} 2023-07-29T00:44:53.679 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008198490d6369153a-0064c46105-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.680 DEBUG:boto:b'hiiqos-60obj221000falseobj222023-07-29T00:42:16.718Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:44:53.680 DEBUG:boto:path=/hiiqos-60/obj22 2023-07-29T00:44:53.681 DEBUG:boto:auth_path=/hiiqos-60/obj22 2023-07-29T00:44:53.681 DEBUG:boto:Method: HEAD 2023-07-29T00:44:53.682 DEBUG:boto:Path: /hiiqos-60/obj22 2023-07-29T00:44:53.682 DEBUG:boto:Data: 2023-07-29T00:44:53.682 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.683 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:44:53.683 DEBUG:boto:Port: 8001 2023-07-29T00:44:53.683 DEBUG:boto:Params: {} 2023-07-29T00:44:53.684 DEBUG:boto:Token: None 2023-07-29T00:44:53.684 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/obj22 2023-07-29T00:44:53.684 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZpVWe3KG4bZUOjvvqLrWUwYEsjw= 2023-07-29T00:44:53.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZpVWe3KG4bZUOjvvqLrWUwYEsjw=', 'Content-Length': '0'} 2023-07-29T00:44:53.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-60:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a4c0d50c4cd0b82-0064c46105-1f88-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.687 DEBUG:boto:path=/hiiqos-60/obj22 2023-07-29T00:44:53.687 DEBUG:boto:auth_path=/hiiqos-60/obj22 2023-07-29T00:44:53.688 DEBUG:boto:Method: HEAD 2023-07-29T00:44:53.688 DEBUG:boto:Path: /hiiqos-60/obj22 2023-07-29T00:44:53.689 DEBUG:boto:Data: 2023-07-29T00:44:53.689 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.689 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:44:53.690 DEBUG:boto:Port: 8000 2023-07-29T00:44:53.690 DEBUG:boto:Params: {} 2023-07-29T00:44:53.690 DEBUG:boto:Token: None 2023-07-29T00:44:53.691 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/obj22 2023-07-29T00:44:53.691 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZpVWe3KG4bZUOjvvqLrWUwYEsjw= 2023-07-29T00:44:53.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZpVWe3KG4bZUOjvvqLrWUwYEsjw=', 'Content-Length': '0'} 2023-07-29T00:44:53.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac9ffa5a435d2a45-0064c46105-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-07-29T00:44:53.694 DEBUG:boto:path=/hiiqos-60/obj22 2023-07-29T00:44:53.694 DEBUG:boto:auth_path=/hiiqos-60/obj22 2023-07-29T00:44:53.695 DEBUG:boto:Method: GET 2023-07-29T00:44:53.696 DEBUG:boto:Path: /hiiqos-60/obj22 2023-07-29T00:44:53.696 DEBUG:boto:Data: 2023-07-29T00:44:53.697 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.697 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:44:53.698 DEBUG:boto:Port: 8001 2023-07-29T00:44:53.698 DEBUG:boto:Params: {} 2023-07-29T00:44:53.698 DEBUG:boto:Token: None 2023-07-29T00:44:53.699 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/obj22 2023-07-29T00:44:53.699 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:c5GynpyHp9s8HpQW0oY3DWk4t7E= 2023-07-29T00:44:53.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c5GynpyHp9s8HpQW0oY3DWk4t7E=', 'Content-Length': '0'} 2023-07-29T00:44:53.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-60:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000883c10421db0d05c-0064c46105-1f88-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.702 DEBUG:boto:path=/hiiqos-60/obj22 2023-07-29T00:44:53.702 DEBUG:boto:auth_path=/hiiqos-60/obj22 2023-07-29T00:44:53.703 DEBUG:boto:Method: GET 2023-07-29T00:44:53.703 DEBUG:boto:Path: /hiiqos-60/obj22 2023-07-29T00:44:53.703 DEBUG:boto:Data: 2023-07-29T00:44:53.704 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.704 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:44:53.705 DEBUG:boto:Port: 8000 2023-07-29T00:44:53.705 DEBUG:boto:Params: {} 2023-07-29T00:44:53.705 DEBUG:boto:Token: None 2023-07-29T00:44:53.706 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/obj22 2023-07-29T00:44:53.707 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:c5GynpyHp9s8HpQW0oY3DWk4t7E= 2023-07-29T00:44:53.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c5GynpyHp9s8HpQW0oY3DWk4t7E=', 'Content-Length': '0'} 2023-07-29T00:44:53.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d37f8103e1e3c096-0064c46105-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.710 DEBUG:boto:path=/hiiqos-60/ 2023-07-29T00:44:53.710 DEBUG:boto:auth_path=/hiiqos-60/ 2023-07-29T00:44:53.710 DEBUG:boto:path=/hiiqos-60/?prefix=obj22 2023-07-29T00:44:53.711 DEBUG:boto:auth_path=/hiiqos-60/?prefix=obj22 2023-07-29T00:44:53.711 DEBUG:boto:Method: GET 2023-07-29T00:44:53.712 DEBUG:boto:Path: /hiiqos-60/?prefix=obj22 2023-07-29T00:44:53.712 DEBUG:boto:Data: 2023-07-29T00:44:53.712 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.713 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:44:53.713 DEBUG:boto:Port: 8001 2023-07-29T00:44:53.713 DEBUG:boto:Params: {} 2023-07-29T00:44:53.714 DEBUG:boto:Token: None 2023-07-29T00:44:53.714 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/ 2023-07-29T00:44:53.715 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pkyqY2jYdDPQ0N54FHpFRz+oyyU= 2023-07-29T00:44:53.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pkyqY2jYdDPQ0N54FHpFRz+oyyU=', 'Content-Length': '0'} 2023-07-29T00:44:53.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cfccbb15d4f6740-0064c46105-1f88-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.718 DEBUG:boto:b'hiiqos-60obj221000falseobj222023-07-29T00:42:16.718Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:44:53.719 DEBUG:boto:path=/hiiqos-60/ 2023-07-29T00:44:53.719 DEBUG:boto:auth_path=/hiiqos-60/ 2023-07-29T00:44:53.720 DEBUG:boto:path=/hiiqos-60/?prefix=obj22 2023-07-29T00:44:53.720 DEBUG:boto:auth_path=/hiiqos-60/?prefix=obj22 2023-07-29T00:44:53.720 DEBUG:boto:Method: GET 2023-07-29T00:44:53.721 DEBUG:boto:Path: /hiiqos-60/?prefix=obj22 2023-07-29T00:44:53.721 DEBUG:boto:Data: 2023-07-29T00:44:53.721 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.722 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:44:53.722 DEBUG:boto:Port: 8000 2023-07-29T00:44:53.723 DEBUG:boto:Params: {} 2023-07-29T00:44:53.723 DEBUG:boto:Token: None 2023-07-29T00:44:53.723 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/ 2023-07-29T00:44:53.724 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pkyqY2jYdDPQ0N54FHpFRz+oyyU= 2023-07-29T00:44:53.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pkyqY2jYdDPQ0N54FHpFRz+oyyU=', 'Content-Length': '0'} 2023-07-29T00:44:53.726 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080b34654f0d1e17a-0064c46105-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.727 DEBUG:boto:b'hiiqos-60obj221000falseobj222023-07-29T00:42:16.718Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:44:53.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-07-29T00:44:53.728 DEBUG:boto:path=/hiiqos-60/obj33 2023-07-29T00:44:53.729 DEBUG:boto:auth_path=/hiiqos-60/obj33 2023-07-29T00:44:53.729 DEBUG:boto:path=/hiiqos-60/obj33?versionId=null 2023-07-29T00:44:53.729 DEBUG:boto:auth_path=/hiiqos-60/obj33?versionId=null 2023-07-29T00:44:53.730 DEBUG:boto:Method: GET 2023-07-29T00:44:53.730 DEBUG:boto:Path: /hiiqos-60/obj33?versionId=null 2023-07-29T00:44:53.731 DEBUG:boto:Data: 2023-07-29T00:44:53.731 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.731 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:44:53.732 DEBUG:boto:Port: 8001 2023-07-29T00:44:53.732 DEBUG:boto:Params: {} 2023-07-29T00:44:53.733 DEBUG:boto:Token: None 2023-07-29T00:44:53.733 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/obj33?versionId=null 2023-07-29T00:44:53.734 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mjNuKhpPH/Yshzqovloa5jkhMyA= 2023-07-29T00:44:53.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mjNuKhpPH/Yshzqovloa5jkhMyA=', 'Content-Length': '0'} 2023-07-29T00:44:53.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-60:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054db28c42d37b3f0-0064c46105-1f88-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.776 DEBUG:boto:path=/hiiqos-60/obj33 2023-07-29T00:44:53.777 DEBUG:boto:auth_path=/hiiqos-60/obj33 2023-07-29T00:44:53.777 DEBUG:boto:path=/hiiqos-60/obj33?versionId=null 2023-07-29T00:44:53.778 DEBUG:boto:auth_path=/hiiqos-60/obj33?versionId=null 2023-07-29T00:44:53.778 DEBUG:boto:Method: GET 2023-07-29T00:44:53.778 DEBUG:boto:Path: /hiiqos-60/obj33?versionId=null 2023-07-29T00:44:53.779 DEBUG:boto:Data: 2023-07-29T00:44:53.779 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.780 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:44:53.780 DEBUG:boto:Port: 8000 2023-07-29T00:44:53.781 DEBUG:boto:Params: {} 2023-07-29T00:44:53.782 DEBUG:boto:Token: None 2023-07-29T00:44:53.782 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/obj33?versionId=null 2023-07-29T00:44:53.782 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mjNuKhpPH/Yshzqovloa5jkhMyA= 2023-07-29T00:44:53.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mjNuKhpPH/Yshzqovloa5jkhMyA=', 'Content-Length': '0'} 2023-07-29T00:44:53.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009e4be0fe9616c6c-0064c46105-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.785 DEBUG:boto:path=/hiiqos-60/obj33 2023-07-29T00:44:53.786 DEBUG:boto:auth_path=/hiiqos-60/obj33 2023-07-29T00:44:53.786 DEBUG:boto:path=/hiiqos-60/obj33?versionId=null 2023-07-29T00:44:53.786 DEBUG:boto:auth_path=/hiiqos-60/obj33?versionId=null 2023-07-29T00:44:53.787 DEBUG:boto:Method: HEAD 2023-07-29T00:44:53.787 DEBUG:boto:Path: /hiiqos-60/obj33?versionId=null 2023-07-29T00:44:53.787 DEBUG:boto:Data: 2023-07-29T00:44:53.788 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.788 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:44:53.788 DEBUG:boto:Port: 8001 2023-07-29T00:44:53.789 DEBUG:boto:Params: {} 2023-07-29T00:44:53.789 DEBUG:boto:Token: None 2023-07-29T00:44:53.790 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/obj33?versionId=null 2023-07-29T00:44:53.790 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:59jAvyKFWKm+LmIw5K4Hh3o1rRs= 2023-07-29T00:44:53.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:59jAvyKFWKm+LmIw5K4Hh3o1rRs=', 'Content-Length': '0'} 2023-07-29T00:44:53.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-60:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005676c3bc447f6946-0064c46105-1f88-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.793 DEBUG:boto:path=/hiiqos-60/obj33 2023-07-29T00:44:53.793 DEBUG:boto:auth_path=/hiiqos-60/obj33 2023-07-29T00:44:53.793 DEBUG:boto:path=/hiiqos-60/obj33?versionId=null 2023-07-29T00:44:53.794 DEBUG:boto:auth_path=/hiiqos-60/obj33?versionId=null 2023-07-29T00:44:53.794 DEBUG:boto:Method: HEAD 2023-07-29T00:44:53.794 DEBUG:boto:Path: /hiiqos-60/obj33?versionId=null 2023-07-29T00:44:53.795 DEBUG:boto:Data: 2023-07-29T00:44:53.795 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.796 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:44:53.796 DEBUG:boto:Port: 8000 2023-07-29T00:44:53.796 DEBUG:boto:Params: {} 2023-07-29T00:44:53.797 DEBUG:boto:Token: None 2023-07-29T00:44:53.798 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/obj33?versionId=null 2023-07-29T00:44:53.798 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:59jAvyKFWKm+LmIw5K4Hh3o1rRs= 2023-07-29T00:44:53.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:59jAvyKFWKm+LmIw5K4Hh3o1rRs=', 'Content-Length': '0'} 2023-07-29T00:44:53.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d0079652044746d-0064c46105-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-07-29T00:44:53.804 DEBUG:boto:path=/hiiqos-60/obj33 2023-07-29T00:44:53.804 DEBUG:boto:auth_path=/hiiqos-60/obj33 2023-07-29T00:44:53.805 DEBUG:boto:path=/hiiqos-60/obj33?versionId=null 2023-07-29T00:44:53.806 DEBUG:boto:auth_path=/hiiqos-60/obj33?versionId=null 2023-07-29T00:44:53.806 DEBUG:boto:Method: GET 2023-07-29T00:44:53.807 DEBUG:boto:Path: /hiiqos-60/obj33?versionId=null 2023-07-29T00:44:53.808 DEBUG:boto:Data: 2023-07-29T00:44:53.808 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.809 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:44:53.809 DEBUG:boto:Port: 8001 2023-07-29T00:44:53.809 DEBUG:boto:Params: {} 2023-07-29T00:44:53.810 DEBUG:boto:Token: None 2023-07-29T00:44:53.810 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/obj33?versionId=null 2023-07-29T00:44:53.811 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mjNuKhpPH/Yshzqovloa5jkhMyA= 2023-07-29T00:44:53.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mjNuKhpPH/Yshzqovloa5jkhMyA=', 'Content-Length': '0'} 2023-07-29T00:44:53.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-60:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074abd740a8d3fa0f-0064c46105-1f88-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.813 DEBUG:boto:path=/hiiqos-60/obj33 2023-07-29T00:44:53.814 DEBUG:boto:auth_path=/hiiqos-60/obj33 2023-07-29T00:44:53.814 DEBUG:boto:path=/hiiqos-60/obj33?versionId=null 2023-07-29T00:44:53.814 DEBUG:boto:auth_path=/hiiqos-60/obj33?versionId=null 2023-07-29T00:44:53.815 DEBUG:boto:Method: GET 2023-07-29T00:44:53.815 DEBUG:boto:Path: /hiiqos-60/obj33?versionId=null 2023-07-29T00:44:53.816 DEBUG:boto:Data: 2023-07-29T00:44:53.816 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.816 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:44:53.817 DEBUG:boto:Port: 8000 2023-07-29T00:44:53.817 DEBUG:boto:Params: {} 2023-07-29T00:44:53.817 DEBUG:boto:Token: None 2023-07-29T00:44:53.818 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/obj33?versionId=null 2023-07-29T00:44:53.818 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mjNuKhpPH/Yshzqovloa5jkhMyA= 2023-07-29T00:44:53.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mjNuKhpPH/Yshzqovloa5jkhMyA=', 'Content-Length': '0'} 2023-07-29T00:44:53.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5199119cea7bb00-0064c46105-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.864 DEBUG:boto:path=/hiiqos-60/ 2023-07-29T00:44:53.865 DEBUG:boto:auth_path=/hiiqos-60/ 2023-07-29T00:44:53.865 DEBUG:boto:path=/hiiqos-60/?prefix=obj33 2023-07-29T00:44:53.866 DEBUG:boto:auth_path=/hiiqos-60/?prefix=obj33 2023-07-29T00:44:53.866 DEBUG:boto:Method: GET 2023-07-29T00:44:53.866 DEBUG:boto:Path: /hiiqos-60/?prefix=obj33 2023-07-29T00:44:53.867 DEBUG:boto:Data: 2023-07-29T00:44:53.867 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.867 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:44:53.868 DEBUG:boto:Port: 8001 2023-07-29T00:44:53.868 DEBUG:boto:Params: {} 2023-07-29T00:44:53.869 DEBUG:boto:Token: None 2023-07-29T00:44:53.869 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/ 2023-07-29T00:44:53.870 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pkyqY2jYdDPQ0N54FHpFRz+oyyU= 2023-07-29T00:44:53.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pkyqY2jYdDPQ0N54FHpFRz+oyyU=', 'Content-Length': '0'} 2023-07-29T00:44:53.872 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006adfdab771a5e93-0064c46105-1f88-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.873 DEBUG:boto:b'hiiqos-60obj331000falseobj332023-07-29T00:42:16.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:44:53.874 DEBUG:boto:path=/hiiqos-60/ 2023-07-29T00:44:53.874 DEBUG:boto:auth_path=/hiiqos-60/ 2023-07-29T00:44:53.874 DEBUG:boto:path=/hiiqos-60/?prefix=obj33 2023-07-29T00:44:53.875 DEBUG:boto:auth_path=/hiiqos-60/?prefix=obj33 2023-07-29T00:44:53.875 DEBUG:boto:Method: GET 2023-07-29T00:44:53.876 DEBUG:boto:Path: /hiiqos-60/?prefix=obj33 2023-07-29T00:44:53.876 DEBUG:boto:Data: 2023-07-29T00:44:53.876 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.877 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:44:53.877 DEBUG:boto:Port: 8000 2023-07-29T00:44:53.877 DEBUG:boto:Params: {} 2023-07-29T00:44:53.878 DEBUG:boto:Token: None 2023-07-29T00:44:53.878 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/ 2023-07-29T00:44:53.879 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pkyqY2jYdDPQ0N54FHpFRz+oyyU= 2023-07-29T00:44:53.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pkyqY2jYdDPQ0N54FHpFRz+oyyU=', 'Content-Length': '0'} 2023-07-29T00:44:53.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009bf3507f96893082-0064c46105-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.882 DEBUG:boto:b'hiiqos-60obj331000falseobj332023-07-29T00:42:16.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:44:53.883 DEBUG:boto:path=/hiiqos-60/obj33 2023-07-29T00:44:53.883 DEBUG:boto:auth_path=/hiiqos-60/obj33 2023-07-29T00:44:53.884 DEBUG:boto:Method: HEAD 2023-07-29T00:44:53.884 DEBUG:boto:Path: /hiiqos-60/obj33 2023-07-29T00:44:53.884 DEBUG:boto:Data: 2023-07-29T00:44:53.885 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.885 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:44:53.886 DEBUG:boto:Port: 8001 2023-07-29T00:44:53.886 DEBUG:boto:Params: {} 2023-07-29T00:44:53.886 DEBUG:boto:Token: None 2023-07-29T00:44:53.887 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/obj33 2023-07-29T00:44:53.887 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0x4EEpwH8oCgANEJuczhs2Go2tI= 2023-07-29T00:44:53.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0x4EEpwH8oCgANEJuczhs2Go2tI=', 'Content-Length': '0'} 2023-07-29T00:44:53.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-60:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000911f1bb65e2ec6a7-0064c46105-1f88-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.890 DEBUG:boto:path=/hiiqos-60/obj33 2023-07-29T00:44:53.890 DEBUG:boto:auth_path=/hiiqos-60/obj33 2023-07-29T00:44:53.891 DEBUG:boto:Method: HEAD 2023-07-29T00:44:53.891 DEBUG:boto:Path: /hiiqos-60/obj33 2023-07-29T00:44:53.891 DEBUG:boto:Data: 2023-07-29T00:44:53.892 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.892 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:44:53.892 DEBUG:boto:Port: 8000 2023-07-29T00:44:53.893 DEBUG:boto:Params: {} 2023-07-29T00:44:53.893 DEBUG:boto:Token: None 2023-07-29T00:44:53.893 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/obj33 2023-07-29T00:44:53.894 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0x4EEpwH8oCgANEJuczhs2Go2tI= 2023-07-29T00:44:53.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0x4EEpwH8oCgANEJuczhs2Go2tI=', 'Content-Length': '0'} 2023-07-29T00:44:53.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3cc3fb27da586b6-0064c46105-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-07-29T00:44:53.897 DEBUG:boto:path=/hiiqos-60/obj33 2023-07-29T00:44:53.898 DEBUG:boto:auth_path=/hiiqos-60/obj33 2023-07-29T00:44:53.898 DEBUG:boto:Method: GET 2023-07-29T00:44:53.899 DEBUG:boto:Path: /hiiqos-60/obj33 2023-07-29T00:44:53.900 DEBUG:boto:Data: 2023-07-29T00:44:53.901 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.902 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:44:53.903 DEBUG:boto:Port: 8001 2023-07-29T00:44:53.903 DEBUG:boto:Params: {} 2023-07-29T00:44:53.904 DEBUG:boto:Token: None 2023-07-29T00:44:53.905 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/obj33 2023-07-29T00:44:53.905 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:g16rwklet8gnJrUWudiLjpl40qI= 2023-07-29T00:44:53.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:g16rwklet8gnJrUWudiLjpl40qI=', 'Content-Length': '0'} 2023-07-29T00:44:53.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-60:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079c779ae46ef3d19-0064c46105-1f88-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.908 DEBUG:boto:path=/hiiqos-60/obj33 2023-07-29T00:44:53.909 DEBUG:boto:auth_path=/hiiqos-60/obj33 2023-07-29T00:44:53.909 DEBUG:boto:Method: GET 2023-07-29T00:44:53.909 DEBUG:boto:Path: /hiiqos-60/obj33 2023-07-29T00:44:53.910 DEBUG:boto:Data: 2023-07-29T00:44:53.910 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.911 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:44:53.911 DEBUG:boto:Port: 8000 2023-07-29T00:44:53.911 DEBUG:boto:Params: {} 2023-07-29T00:44:53.912 DEBUG:boto:Token: None 2023-07-29T00:44:53.912 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/obj33 2023-07-29T00:44:53.913 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:g16rwklet8gnJrUWudiLjpl40qI= 2023-07-29T00:44:53.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:g16rwklet8gnJrUWudiLjpl40qI=', 'Content-Length': '0'} 2023-07-29T00:44:53.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000851f83549b528804-0064c46105-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.916 DEBUG:boto:path=/hiiqos-60/ 2023-07-29T00:44:53.916 DEBUG:boto:auth_path=/hiiqos-60/ 2023-07-29T00:44:53.917 DEBUG:boto:path=/hiiqos-60/?prefix=obj33 2023-07-29T00:44:53.917 DEBUG:boto:auth_path=/hiiqos-60/?prefix=obj33 2023-07-29T00:44:53.917 DEBUG:boto:Method: GET 2023-07-29T00:44:53.918 DEBUG:boto:Path: /hiiqos-60/?prefix=obj33 2023-07-29T00:44:53.918 DEBUG:boto:Data: 2023-07-29T00:44:53.918 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.919 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:44:53.919 DEBUG:boto:Port: 8001 2023-07-29T00:44:53.920 DEBUG:boto:Params: {} 2023-07-29T00:44:53.920 DEBUG:boto:Token: None 2023-07-29T00:44:53.921 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/ 2023-07-29T00:44:53.921 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pkyqY2jYdDPQ0N54FHpFRz+oyyU= 2023-07-29T00:44:53.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pkyqY2jYdDPQ0N54FHpFRz+oyyU=', 'Content-Length': '0'} 2023-07-29T00:44:53.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c72331ba92a78ec-0064c46105-1f88-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.924 DEBUG:boto:b'hiiqos-60obj331000falseobj332023-07-29T00:42:16.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:44:53.925 DEBUG:boto:path=/hiiqos-60/ 2023-07-29T00:44:53.925 DEBUG:boto:auth_path=/hiiqos-60/ 2023-07-29T00:44:53.926 DEBUG:boto:path=/hiiqos-60/?prefix=obj33 2023-07-29T00:44:53.926 DEBUG:boto:auth_path=/hiiqos-60/?prefix=obj33 2023-07-29T00:44:53.927 DEBUG:boto:Method: GET 2023-07-29T00:44:53.927 DEBUG:boto:Path: /hiiqos-60/?prefix=obj33 2023-07-29T00:44:53.928 DEBUG:boto:Data: 2023-07-29T00:44:53.928 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.929 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:44:53.929 DEBUG:boto:Port: 8000 2023-07-29T00:44:53.930 DEBUG:boto:Params: {} 2023-07-29T00:44:53.930 DEBUG:boto:Token: None 2023-07-29T00:44:53.930 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/ 2023-07-29T00:44:53.931 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pkyqY2jYdDPQ0N54FHpFRz+oyyU= 2023-07-29T00:44:53.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pkyqY2jYdDPQ0N54FHpFRz+oyyU=', 'Content-Length': '0'} 2023-07-29T00:44:53.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089bfb40dea8f8ee2-0064c46105-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.934 DEBUG:boto:b'hiiqos-60obj331000falseobj332023-07-29T00:42:16.773Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:44:53.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-07-29T00:44:53.935 DEBUG:boto:path=/hiiqos-60/obj44 2023-07-29T00:44:53.936 DEBUG:boto:auth_path=/hiiqos-60/obj44 2023-07-29T00:44:53.936 DEBUG:boto:path=/hiiqos-60/obj44?versionId=null 2023-07-29T00:44:53.937 DEBUG:boto:auth_path=/hiiqos-60/obj44?versionId=null 2023-07-29T00:44:53.937 DEBUG:boto:Method: GET 2023-07-29T00:44:53.938 DEBUG:boto:Path: /hiiqos-60/obj44?versionId=null 2023-07-29T00:44:53.938 DEBUG:boto:Data: 2023-07-29T00:44:53.938 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.939 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:44:53.939 DEBUG:boto:Port: 8001 2023-07-29T00:44:53.939 DEBUG:boto:Params: {} 2023-07-29T00:44:53.940 DEBUG:boto:Token: None 2023-07-29T00:44:53.940 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/obj44?versionId=null 2023-07-29T00:44:53.941 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cQaREjas3bFZPAShwdawNpyemFo= 2023-07-29T00:44:53.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cQaREjas3bFZPAShwdawNpyemFo=', 'Content-Length': '0'} 2023-07-29T00:44:53.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-60:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008d6471b3453a984-0064c46105-1f88-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.944 DEBUG:boto:path=/hiiqos-60/obj44 2023-07-29T00:44:53.944 DEBUG:boto:auth_path=/hiiqos-60/obj44 2023-07-29T00:44:53.945 DEBUG:boto:path=/hiiqos-60/obj44?versionId=null 2023-07-29T00:44:53.945 DEBUG:boto:auth_path=/hiiqos-60/obj44?versionId=null 2023-07-29T00:44:53.945 DEBUG:boto:Method: GET 2023-07-29T00:44:53.946 DEBUG:boto:Path: /hiiqos-60/obj44?versionId=null 2023-07-29T00:44:53.946 DEBUG:boto:Data: 2023-07-29T00:44:53.946 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.947 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:44:53.947 DEBUG:boto:Port: 8000 2023-07-29T00:44:53.948 DEBUG:boto:Params: {} 2023-07-29T00:44:53.948 DEBUG:boto:Token: None 2023-07-29T00:44:53.949 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/obj44?versionId=null 2023-07-29T00:44:53.949 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cQaREjas3bFZPAShwdawNpyemFo= 2023-07-29T00:44:53.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cQaREjas3bFZPAShwdawNpyemFo=', 'Content-Length': '0'} 2023-07-29T00:44:53.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d867cbe52b5acf4e-0064c46105-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.952 DEBUG:boto:path=/hiiqos-60/obj44 2023-07-29T00:44:53.953 DEBUG:boto:auth_path=/hiiqos-60/obj44 2023-07-29T00:44:53.953 DEBUG:boto:path=/hiiqos-60/obj44?versionId=null 2023-07-29T00:44:53.954 DEBUG:boto:auth_path=/hiiqos-60/obj44?versionId=null 2023-07-29T00:44:53.954 DEBUG:boto:Method: HEAD 2023-07-29T00:44:53.954 DEBUG:boto:Path: /hiiqos-60/obj44?versionId=null 2023-07-29T00:44:53.955 DEBUG:boto:Data: 2023-07-29T00:44:53.955 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.955 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:44:53.956 DEBUG:boto:Port: 8001 2023-07-29T00:44:53.956 DEBUG:boto:Params: {} 2023-07-29T00:44:53.956 DEBUG:boto:Token: None 2023-07-29T00:44:53.957 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/obj44?versionId=null 2023-07-29T00:44:53.957 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SsfcDbNYv/NhVKcibz8a3dZ2DO4= 2023-07-29T00:44:53.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SsfcDbNYv/NhVKcibz8a3dZ2DO4=', 'Content-Length': '0'} 2023-07-29T00:44:53.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-60:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9e89457e1274908-0064c46105-1f88-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.960 DEBUG:boto:path=/hiiqos-60/obj44 2023-07-29T00:44:53.961 DEBUG:boto:auth_path=/hiiqos-60/obj44 2023-07-29T00:44:53.961 DEBUG:boto:path=/hiiqos-60/obj44?versionId=null 2023-07-29T00:44:53.961 DEBUG:boto:auth_path=/hiiqos-60/obj44?versionId=null 2023-07-29T00:44:53.962 DEBUG:boto:Method: HEAD 2023-07-29T00:44:53.962 DEBUG:boto:Path: /hiiqos-60/obj44?versionId=null 2023-07-29T00:44:53.962 DEBUG:boto:Data: 2023-07-29T00:44:53.963 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.963 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:44:53.963 DEBUG:boto:Port: 8000 2023-07-29T00:44:53.964 DEBUG:boto:Params: {} 2023-07-29T00:44:53.964 DEBUG:boto:Token: None 2023-07-29T00:44:53.964 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/obj44?versionId=null 2023-07-29T00:44:53.965 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SsfcDbNYv/NhVKcibz8a3dZ2DO4= 2023-07-29T00:44:53.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SsfcDbNYv/NhVKcibz8a3dZ2DO4=', 'Content-Length': '0'} 2023-07-29T00:44:53.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c24b79cdc928b5af-0064c46105-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-07-29T00:44:53.968 DEBUG:boto:path=/hiiqos-60/obj44 2023-07-29T00:44:53.968 DEBUG:boto:auth_path=/hiiqos-60/obj44 2023-07-29T00:44:53.968 DEBUG:boto:path=/hiiqos-60/obj44?versionId=null 2023-07-29T00:44:53.969 DEBUG:boto:auth_path=/hiiqos-60/obj44?versionId=null 2023-07-29T00:44:53.969 DEBUG:boto:Method: GET 2023-07-29T00:44:53.970 DEBUG:boto:Path: /hiiqos-60/obj44?versionId=null 2023-07-29T00:44:53.970 DEBUG:boto:Data: 2023-07-29T00:44:53.971 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.971 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:44:53.971 DEBUG:boto:Port: 8001 2023-07-29T00:44:53.972 DEBUG:boto:Params: {} 2023-07-29T00:44:53.972 DEBUG:boto:Token: None 2023-07-29T00:44:53.972 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/obj44?versionId=null 2023-07-29T00:44:53.973 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cQaREjas3bFZPAShwdawNpyemFo= 2023-07-29T00:44:53.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cQaREjas3bFZPAShwdawNpyemFo=', 'Content-Length': '0'} 2023-07-29T00:44:53.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-60:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045e0668c075896bd-0064c46105-1f88-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.976 DEBUG:boto:path=/hiiqos-60/obj44 2023-07-29T00:44:53.977 DEBUG:boto:auth_path=/hiiqos-60/obj44 2023-07-29T00:44:53.977 DEBUG:boto:path=/hiiqos-60/obj44?versionId=null 2023-07-29T00:44:53.977 DEBUG:boto:auth_path=/hiiqos-60/obj44?versionId=null 2023-07-29T00:44:53.978 DEBUG:boto:Method: GET 2023-07-29T00:44:53.978 DEBUG:boto:Path: /hiiqos-60/obj44?versionId=null 2023-07-29T00:44:53.979 DEBUG:boto:Data: 2023-07-29T00:44:53.979 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.980 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:44:53.980 DEBUG:boto:Port: 8000 2023-07-29T00:44:53.981 DEBUG:boto:Params: {} 2023-07-29T00:44:53.981 DEBUG:boto:Token: None 2023-07-29T00:44:53.982 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/obj44?versionId=null 2023-07-29T00:44:53.982 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cQaREjas3bFZPAShwdawNpyemFo= 2023-07-29T00:44:53.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cQaREjas3bFZPAShwdawNpyemFo=', 'Content-Length': '0'} 2023-07-29T00:44:53.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b8ab6ab7193f491-0064c46105-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.985 DEBUG:boto:path=/hiiqos-60/ 2023-07-29T00:44:53.985 DEBUG:boto:auth_path=/hiiqos-60/ 2023-07-29T00:44:53.986 DEBUG:boto:path=/hiiqos-60/?prefix=obj44 2023-07-29T00:44:53.986 DEBUG:boto:auth_path=/hiiqos-60/?prefix=obj44 2023-07-29T00:44:53.986 DEBUG:boto:Method: GET 2023-07-29T00:44:53.987 DEBUG:boto:Path: /hiiqos-60/?prefix=obj44 2023-07-29T00:44:53.987 DEBUG:boto:Data: 2023-07-29T00:44:53.987 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.988 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:44:53.988 DEBUG:boto:Port: 8001 2023-07-29T00:44:53.989 DEBUG:boto:Params: {} 2023-07-29T00:44:53.989 DEBUG:boto:Token: None 2023-07-29T00:44:53.989 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/ 2023-07-29T00:44:53.990 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pkyqY2jYdDPQ0N54FHpFRz+oyyU= 2023-07-29T00:44:53.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pkyqY2jYdDPQ0N54FHpFRz+oyyU=', 'Content-Length': '0'} 2023-07-29T00:44:53.993 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000786153e49a12749-0064c46105-1f88-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:53.993 DEBUG:boto:b'hiiqos-60obj441000falseobj442023-07-29T00:42:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:44:53.994 DEBUG:boto:path=/hiiqos-60/ 2023-07-29T00:44:53.994 DEBUG:boto:auth_path=/hiiqos-60/ 2023-07-29T00:44:53.995 DEBUG:boto:path=/hiiqos-60/?prefix=obj44 2023-07-29T00:44:53.995 DEBUG:boto:auth_path=/hiiqos-60/?prefix=obj44 2023-07-29T00:44:53.995 DEBUG:boto:Method: GET 2023-07-29T00:44:53.996 DEBUG:boto:Path: /hiiqos-60/?prefix=obj44 2023-07-29T00:44:53.996 DEBUG:boto:Data: 2023-07-29T00:44:53.997 DEBUG:boto:Headers: {} 2023-07-29T00:44:53.997 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:44:53.997 DEBUG:boto:Port: 8000 2023-07-29T00:44:53.998 DEBUG:boto:Params: {} 2023-07-29T00:44:53.998 DEBUG:boto:Token: None 2023-07-29T00:44:53.998 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:53 GMT /hiiqos-60/ 2023-07-29T00:44:53.999 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pkyqY2jYdDPQ0N54FHpFRz+oyyU= 2023-07-29T00:44:54.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pkyqY2jYdDPQ0N54FHpFRz+oyyU=', 'Content-Length': '0'} 2023-07-29T00:44:54.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afbd7cefc3b7aed4-0064c46105-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:54.044 DEBUG:boto:b'hiiqos-60obj441000falseobj442023-07-29T00:42:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:44:54.045 DEBUG:boto:path=/hiiqos-60/obj44 2023-07-29T00:44:54.046 DEBUG:boto:auth_path=/hiiqos-60/obj44 2023-07-29T00:44:54.046 DEBUG:boto:Method: HEAD 2023-07-29T00:44:54.046 DEBUG:boto:Path: /hiiqos-60/obj44 2023-07-29T00:44:54.047 DEBUG:boto:Data: 2023-07-29T00:44:54.047 DEBUG:boto:Headers: {} 2023-07-29T00:44:54.047 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:44:54.048 DEBUG:boto:Port: 8001 2023-07-29T00:44:54.048 DEBUG:boto:Params: {} 2023-07-29T00:44:54.049 DEBUG:boto:Token: None 2023-07-29T00:44:54.050 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:44:54 GMT /hiiqos-60/obj44 2023-07-29T00:44:54.051 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tVD8vA0ZZwRIBSVhzE29liQefCw= 2023-07-29T00:44:54.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tVD8vA0ZZwRIBSVhzE29liQefCw=', 'Content-Length': '0'} 2023-07-29T00:44:54.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-60:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a712d940f0b4c25-0064c46105-1f88-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:54.055 DEBUG:boto:path=/hiiqos-60/obj44 2023-07-29T00:44:54.056 DEBUG:boto:auth_path=/hiiqos-60/obj44 2023-07-29T00:44:54.057 DEBUG:boto:Method: HEAD 2023-07-29T00:44:54.058 DEBUG:boto:Path: /hiiqos-60/obj44 2023-07-29T00:44:54.059 DEBUG:boto:Data: 2023-07-29T00:44:54.059 DEBUG:boto:Headers: {} 2023-07-29T00:44:54.060 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:44:54.060 DEBUG:boto:Port: 8000 2023-07-29T00:44:54.061 DEBUG:boto:Params: {} 2023-07-29T00:44:54.062 DEBUG:boto:Token: None 2023-07-29T00:44:54.062 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:44:54 GMT /hiiqos-60/obj44 2023-07-29T00:44:54.063 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tVD8vA0ZZwRIBSVhzE29liQefCw= 2023-07-29T00:44:54.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tVD8vA0ZZwRIBSVhzE29liQefCw=', 'Content-Length': '0'} 2023-07-29T00:44:54.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3acd3bf20535bd2-0064c46105-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:54.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-07-29T00:44:54.066 DEBUG:boto:path=/hiiqos-60/obj44 2023-07-29T00:44:54.066 DEBUG:boto:auth_path=/hiiqos-60/obj44 2023-07-29T00:44:54.067 DEBUG:boto:Method: GET 2023-07-29T00:44:54.067 DEBUG:boto:Path: /hiiqos-60/obj44 2023-07-29T00:44:54.068 DEBUG:boto:Data: 2023-07-29T00:44:54.068 DEBUG:boto:Headers: {} 2023-07-29T00:44:54.068 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:44:54.069 DEBUG:boto:Port: 8001 2023-07-29T00:44:54.069 DEBUG:boto:Params: {} 2023-07-29T00:44:54.069 DEBUG:boto:Token: None 2023-07-29T00:44:54.070 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:54 GMT /hiiqos-60/obj44 2023-07-29T00:44:54.070 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4y15bLqaZfrjIfkqDIY96w6B3Hw= 2023-07-29T00:44:54.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4y15bLqaZfrjIfkqDIY96w6B3Hw=', 'Content-Length': '0'} 2023-07-29T00:44:54.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-60:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c54f0f9dfddbb88c-0064c46106-1f88-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:54.073 DEBUG:boto:path=/hiiqos-60/obj44 2023-07-29T00:44:54.074 DEBUG:boto:auth_path=/hiiqos-60/obj44 2023-07-29T00:44:54.074 DEBUG:boto:Method: GET 2023-07-29T00:44:54.074 DEBUG:boto:Path: /hiiqos-60/obj44 2023-07-29T00:44:54.075 DEBUG:boto:Data: 2023-07-29T00:44:54.075 DEBUG:boto:Headers: {} 2023-07-29T00:44:54.075 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:44:54.076 DEBUG:boto:Port: 8000 2023-07-29T00:44:54.076 DEBUG:boto:Params: {} 2023-07-29T00:44:54.077 DEBUG:boto:Token: None 2023-07-29T00:44:54.077 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:54 GMT /hiiqos-60/obj44 2023-07-29T00:44:54.077 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4y15bLqaZfrjIfkqDIY96w6B3Hw= 2023-07-29T00:44:54.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4y15bLqaZfrjIfkqDIY96w6B3Hw=', 'Content-Length': '0'} 2023-07-29T00:44:54.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a8d2d94eb22c8f4-0064c46106-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:54.080 DEBUG:boto:path=/hiiqos-60/ 2023-07-29T00:44:54.081 DEBUG:boto:auth_path=/hiiqos-60/ 2023-07-29T00:44:54.082 DEBUG:boto:path=/hiiqos-60/?prefix=obj44 2023-07-29T00:44:54.082 DEBUG:boto:auth_path=/hiiqos-60/?prefix=obj44 2023-07-29T00:44:54.082 DEBUG:boto:Method: GET 2023-07-29T00:44:54.083 DEBUG:boto:Path: /hiiqos-60/?prefix=obj44 2023-07-29T00:44:54.083 DEBUG:boto:Data: 2023-07-29T00:44:54.084 DEBUG:boto:Headers: {} 2023-07-29T00:44:54.084 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:44:54.085 DEBUG:boto:Port: 8001 2023-07-29T00:44:54.085 DEBUG:boto:Params: {} 2023-07-29T00:44:54.085 DEBUG:boto:Token: None 2023-07-29T00:44:54.086 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:54 GMT /hiiqos-60/ 2023-07-29T00:44:54.086 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ka4R+DOpm2+zi/bqi5gJmE5hovE= 2023-07-29T00:44:54.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ka4R+DOpm2+zi/bqi5gJmE5hovE=', 'Content-Length': '0'} 2023-07-29T00:44:54.089 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034c7b6b7f43bb29f-0064c46106-1f88-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:44:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:54.136 DEBUG:boto:b'hiiqos-60obj441000falseobj442023-07-29T00:42:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:44:54.137 DEBUG:boto:path=/hiiqos-60/ 2023-07-29T00:44:54.138 DEBUG:boto:auth_path=/hiiqos-60/ 2023-07-29T00:44:54.138 DEBUG:boto:path=/hiiqos-60/?prefix=obj44 2023-07-29T00:44:54.138 DEBUG:boto:auth_path=/hiiqos-60/?prefix=obj44 2023-07-29T00:44:54.139 DEBUG:boto:Method: GET 2023-07-29T00:44:54.139 DEBUG:boto:Path: /hiiqos-60/?prefix=obj44 2023-07-29T00:44:54.139 DEBUG:boto:Data: 2023-07-29T00:44:54.140 DEBUG:boto:Headers: {} 2023-07-29T00:44:54.140 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:44:54.140 DEBUG:boto:Port: 8000 2023-07-29T00:44:54.141 DEBUG:boto:Params: {} 2023-07-29T00:44:54.142 DEBUG:boto:Token: None 2023-07-29T00:44:54.143 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:54 GMT /hiiqos-60/ 2023-07-29T00:44:54.144 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ka4R+DOpm2+zi/bqi5gJmE5hovE= 2023-07-29T00:44:54.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ka4R+DOpm2+zi/bqi5gJmE5hovE=', 'Content-Length': '0'} 2023-07-29T00:44:54.148 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087fa806afe2236f3-0064c46106-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:44:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:54.148 DEBUG:boto:b'hiiqos-60obj441000falseobj442023-07-29T00:42:16.792Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:44:54.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-07-29T00:44:54.149 DEBUG:boto:path=/hiiqos-60/obj55 2023-07-29T00:44:54.150 DEBUG:boto:auth_path=/hiiqos-60/obj55 2023-07-29T00:44:54.150 DEBUG:boto:path=/hiiqos-60/obj55?versionId=null 2023-07-29T00:44:54.150 DEBUG:boto:auth_path=/hiiqos-60/obj55?versionId=null 2023-07-29T00:44:54.151 DEBUG:boto:Method: GET 2023-07-29T00:44:54.151 DEBUG:boto:Path: /hiiqos-60/obj55?versionId=null 2023-07-29T00:44:54.151 DEBUG:boto:Data: 2023-07-29T00:44:54.152 DEBUG:boto:Headers: {} 2023-07-29T00:44:54.152 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:44:54.153 DEBUG:boto:Port: 8001 2023-07-29T00:44:54.153 DEBUG:boto:Params: {} 2023-07-29T00:44:54.153 DEBUG:boto:Token: None 2023-07-29T00:44:54.154 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:54 GMT /hiiqos-60/obj55?versionId=null 2023-07-29T00:44:54.154 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Yzj3bGIxbAROKmP6xqKpMnD58fI= 2023-07-29T00:44:54.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Yzj3bGIxbAROKmP6xqKpMnD58fI=', 'Content-Length': '0'} 2023-07-29T00:44:54.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-60:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000273adc91f5e323bd-0064c46106-1f88-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:54.157 DEBUG:boto:path=/hiiqos-60/obj55 2023-07-29T00:44:54.158 DEBUG:boto:auth_path=/hiiqos-60/obj55 2023-07-29T00:44:54.159 DEBUG:boto:path=/hiiqos-60/obj55?versionId=null 2023-07-29T00:44:54.160 DEBUG:boto:auth_path=/hiiqos-60/obj55?versionId=null 2023-07-29T00:44:54.160 DEBUG:boto:Method: GET 2023-07-29T00:44:54.161 DEBUG:boto:Path: /hiiqos-60/obj55?versionId=null 2023-07-29T00:44:54.162 DEBUG:boto:Data: 2023-07-29T00:44:54.163 DEBUG:boto:Headers: {} 2023-07-29T00:44:54.164 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:44:54.165 DEBUG:boto:Port: 8000 2023-07-29T00:44:54.165 DEBUG:boto:Params: {} 2023-07-29T00:44:54.166 DEBUG:boto:Token: None 2023-07-29T00:44:54.167 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:54 GMT /hiiqos-60/obj55?versionId=null 2023-07-29T00:44:54.168 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Yzj3bGIxbAROKmP6xqKpMnD58fI= 2023-07-29T00:44:54.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Yzj3bGIxbAROKmP6xqKpMnD58fI=', 'Content-Length': '0'} 2023-07-29T00:44:54.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0a3ffde7b5a9ac9-0064c46106-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:54.171 DEBUG:boto:path=/hiiqos-60/obj55 2023-07-29T00:44:54.171 DEBUG:boto:auth_path=/hiiqos-60/obj55 2023-07-29T00:44:54.172 DEBUG:boto:path=/hiiqos-60/obj55?versionId=null 2023-07-29T00:44:54.172 DEBUG:boto:auth_path=/hiiqos-60/obj55?versionId=null 2023-07-29T00:44:54.172 DEBUG:boto:Method: HEAD 2023-07-29T00:44:54.173 DEBUG:boto:Path: /hiiqos-60/obj55?versionId=null 2023-07-29T00:44:54.173 DEBUG:boto:Data: 2023-07-29T00:44:54.173 DEBUG:boto:Headers: {} 2023-07-29T00:44:54.174 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:44:54.174 DEBUG:boto:Port: 8001 2023-07-29T00:44:54.174 DEBUG:boto:Params: {} 2023-07-29T00:44:54.175 DEBUG:boto:Token: None 2023-07-29T00:44:54.175 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:44:54 GMT /hiiqos-60/obj55?versionId=null 2023-07-29T00:44:54.175 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TxN0gg+YKLVKexKtc52DZySEHYM= 2023-07-29T00:44:54.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TxN0gg+YKLVKexKtc52DZySEHYM=', 'Content-Length': '0'} 2023-07-29T00:44:54.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-60:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068418b69ad4afc2d-0064c46106-1f88-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:54.178 DEBUG:boto:path=/hiiqos-60/obj55 2023-07-29T00:44:54.178 DEBUG:boto:auth_path=/hiiqos-60/obj55 2023-07-29T00:44:54.179 DEBUG:boto:path=/hiiqos-60/obj55?versionId=null 2023-07-29T00:44:54.179 DEBUG:boto:auth_path=/hiiqos-60/obj55?versionId=null 2023-07-29T00:44:54.179 DEBUG:boto:Method: HEAD 2023-07-29T00:44:54.180 DEBUG:boto:Path: /hiiqos-60/obj55?versionId=null 2023-07-29T00:44:54.180 DEBUG:boto:Data: 2023-07-29T00:44:54.180 DEBUG:boto:Headers: {} 2023-07-29T00:44:54.181 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:44:54.181 DEBUG:boto:Port: 8000 2023-07-29T00:44:54.181 DEBUG:boto:Params: {} 2023-07-29T00:44:54.182 DEBUG:boto:Token: None 2023-07-29T00:44:54.182 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:44:54 GMT /hiiqos-60/obj55?versionId=null 2023-07-29T00:44:54.183 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TxN0gg+YKLVKexKtc52DZySEHYM= 2023-07-29T00:44:54.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TxN0gg+YKLVKexKtc52DZySEHYM=', 'Content-Length': '0'} 2023-07-29T00:44:54.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abec0d5bfd14ae21-0064c46106-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:54.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-07-29T00:44:54.186 DEBUG:boto:path=/hiiqos-60/obj55 2023-07-29T00:44:54.186 DEBUG:boto:auth_path=/hiiqos-60/obj55 2023-07-29T00:44:54.186 DEBUG:boto:path=/hiiqos-60/obj55?versionId=null 2023-07-29T00:44:54.187 DEBUG:boto:auth_path=/hiiqos-60/obj55?versionId=null 2023-07-29T00:44:54.187 DEBUG:boto:Method: GET 2023-07-29T00:44:54.188 DEBUG:boto:Path: /hiiqos-60/obj55?versionId=null 2023-07-29T00:44:54.188 DEBUG:boto:Data: 2023-07-29T00:44:54.189 DEBUG:boto:Headers: {} 2023-07-29T00:44:54.189 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:44:54.189 DEBUG:boto:Port: 8001 2023-07-29T00:44:54.190 DEBUG:boto:Params: {} 2023-07-29T00:44:54.190 DEBUG:boto:Token: None 2023-07-29T00:44:54.190 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:54 GMT /hiiqos-60/obj55?versionId=null 2023-07-29T00:44:54.191 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Yzj3bGIxbAROKmP6xqKpMnD58fI= 2023-07-29T00:44:54.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Yzj3bGIxbAROKmP6xqKpMnD58fI=', 'Content-Length': '0'} 2023-07-29T00:44:54.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-60:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d2ac1330f39f56d-0064c46106-1f88-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:54.194 DEBUG:boto:path=/hiiqos-60/obj55 2023-07-29T00:44:54.195 DEBUG:boto:auth_path=/hiiqos-60/obj55 2023-07-29T00:44:54.195 DEBUG:boto:path=/hiiqos-60/obj55?versionId=null 2023-07-29T00:44:54.196 DEBUG:boto:auth_path=/hiiqos-60/obj55?versionId=null 2023-07-29T00:44:54.197 DEBUG:boto:Method: GET 2023-07-29T00:44:54.198 DEBUG:boto:Path: /hiiqos-60/obj55?versionId=null 2023-07-29T00:44:54.198 DEBUG:boto:Data: 2023-07-29T00:44:54.198 DEBUG:boto:Headers: {} 2023-07-29T00:44:54.199 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:44:54.199 DEBUG:boto:Port: 8000 2023-07-29T00:44:54.200 DEBUG:boto:Params: {} 2023-07-29T00:44:54.200 DEBUG:boto:Token: None 2023-07-29T00:44:54.200 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:54 GMT /hiiqos-60/obj55?versionId=null 2023-07-29T00:44:54.201 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Yzj3bGIxbAROKmP6xqKpMnD58fI= 2023-07-29T00:44:54.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Yzj3bGIxbAROKmP6xqKpMnD58fI=', 'Content-Length': '0'} 2023-07-29T00:44:54.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084a25a83eb41c819-0064c46106-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:54.244 DEBUG:boto:path=/hiiqos-60/ 2023-07-29T00:44:54.245 DEBUG:boto:auth_path=/hiiqos-60/ 2023-07-29T00:44:54.245 DEBUG:boto:path=/hiiqos-60/?prefix=obj55 2023-07-29T00:44:54.245 DEBUG:boto:auth_path=/hiiqos-60/?prefix=obj55 2023-07-29T00:44:54.246 DEBUG:boto:Method: GET 2023-07-29T00:44:54.246 DEBUG:boto:Path: /hiiqos-60/?prefix=obj55 2023-07-29T00:44:54.247 DEBUG:boto:Data: 2023-07-29T00:44:54.247 DEBUG:boto:Headers: {} 2023-07-29T00:44:54.247 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:44:54.248 DEBUG:boto:Port: 8001 2023-07-29T00:44:54.248 DEBUG:boto:Params: {} 2023-07-29T00:44:54.248 DEBUG:boto:Token: None 2023-07-29T00:44:54.249 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:54 GMT /hiiqos-60/ 2023-07-29T00:44:54.249 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ka4R+DOpm2+zi/bqi5gJmE5hovE= 2023-07-29T00:44:54.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ka4R+DOpm2+zi/bqi5gJmE5hovE=', 'Content-Length': '0'} 2023-07-29T00:44:54.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c36688edf3dc1b3-0064c46106-1f88-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:44:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:54.252 DEBUG:boto:b'hiiqos-60obj551000falseobj552023-07-29T00:42:16.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:44:54.253 DEBUG:boto:path=/hiiqos-60/ 2023-07-29T00:44:54.253 DEBUG:boto:auth_path=/hiiqos-60/ 2023-07-29T00:44:54.254 DEBUG:boto:path=/hiiqos-60/?prefix=obj55 2023-07-29T00:44:54.254 DEBUG:boto:auth_path=/hiiqos-60/?prefix=obj55 2023-07-29T00:44:54.255 DEBUG:boto:Method: GET 2023-07-29T00:44:54.255 DEBUG:boto:Path: /hiiqos-60/?prefix=obj55 2023-07-29T00:44:54.255 DEBUG:boto:Data: 2023-07-29T00:44:54.256 DEBUG:boto:Headers: {} 2023-07-29T00:44:54.256 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:44:54.256 DEBUG:boto:Port: 8000 2023-07-29T00:44:54.257 DEBUG:boto:Params: {} 2023-07-29T00:44:54.257 DEBUG:boto:Token: None 2023-07-29T00:44:54.258 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:54 GMT /hiiqos-60/ 2023-07-29T00:44:54.258 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ka4R+DOpm2+zi/bqi5gJmE5hovE= 2023-07-29T00:44:54.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ka4R+DOpm2+zi/bqi5gJmE5hovE=', 'Content-Length': '0'} 2023-07-29T00:44:54.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cf6a26df46a9356-0064c46106-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:44:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:54.261 DEBUG:boto:b'hiiqos-60obj551000falseobj552023-07-29T00:42:16.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:44:54.262 DEBUG:boto:path=/hiiqos-60/obj55 2023-07-29T00:44:54.262 DEBUG:boto:auth_path=/hiiqos-60/obj55 2023-07-29T00:44:54.262 DEBUG:boto:Method: HEAD 2023-07-29T00:44:54.263 DEBUG:boto:Path: /hiiqos-60/obj55 2023-07-29T00:44:54.263 DEBUG:boto:Data: 2023-07-29T00:44:54.263 DEBUG:boto:Headers: {} 2023-07-29T00:44:54.264 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:44:54.264 DEBUG:boto:Port: 8001 2023-07-29T00:44:54.264 DEBUG:boto:Params: {} 2023-07-29T00:44:54.265 DEBUG:boto:Token: None 2023-07-29T00:44:54.265 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:44:54 GMT /hiiqos-60/obj55 2023-07-29T00:44:54.266 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ypagwiJ9SEBFUGThY4I0xUdmn/Y= 2023-07-29T00:44:54.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ypagwiJ9SEBFUGThY4I0xUdmn/Y=', 'Content-Length': '0'} 2023-07-29T00:44:54.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-60:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057e9e388990bffba-0064c46106-1f88-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:54.268 DEBUG:boto:path=/hiiqos-60/obj55 2023-07-29T00:44:54.269 DEBUG:boto:auth_path=/hiiqos-60/obj55 2023-07-29T00:44:54.269 DEBUG:boto:Method: HEAD 2023-07-29T00:44:54.269 DEBUG:boto:Path: /hiiqos-60/obj55 2023-07-29T00:44:54.270 DEBUG:boto:Data: 2023-07-29T00:44:54.270 DEBUG:boto:Headers: {} 2023-07-29T00:44:54.270 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:44:54.271 DEBUG:boto:Port: 8000 2023-07-29T00:44:54.271 DEBUG:boto:Params: {} 2023-07-29T00:44:54.271 DEBUG:boto:Token: None 2023-07-29T00:44:54.272 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:44:54 GMT /hiiqos-60/obj55 2023-07-29T00:44:54.272 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ypagwiJ9SEBFUGThY4I0xUdmn/Y= 2023-07-29T00:44:54.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ypagwiJ9SEBFUGThY4I0xUdmn/Y=', 'Content-Length': '0'} 2023-07-29T00:44:54.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e48908672eecdf53-0064c46106-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:54.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-07-29T00:44:54.275 DEBUG:boto:path=/hiiqos-60/obj55 2023-07-29T00:44:54.276 DEBUG:boto:auth_path=/hiiqos-60/obj55 2023-07-29T00:44:54.276 DEBUG:boto:Method: GET 2023-07-29T00:44:54.276 DEBUG:boto:Path: /hiiqos-60/obj55 2023-07-29T00:44:54.277 DEBUG:boto:Data: 2023-07-29T00:44:54.277 DEBUG:boto:Headers: {} 2023-07-29T00:44:54.277 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:44:54.278 DEBUG:boto:Port: 8001 2023-07-29T00:44:54.278 DEBUG:boto:Params: {} 2023-07-29T00:44:54.279 DEBUG:boto:Token: None 2023-07-29T00:44:54.279 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:54 GMT /hiiqos-60/obj55 2023-07-29T00:44:54.280 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eKVcx9uheXQbvGRK8UGgZffEhe8= 2023-07-29T00:44:54.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eKVcx9uheXQbvGRK8UGgZffEhe8=', 'Content-Length': '0'} 2023-07-29T00:44:54.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-60:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d34ede36709840d-0064c46106-1f88-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:54.282 DEBUG:boto:path=/hiiqos-60/obj55 2023-07-29T00:44:54.283 DEBUG:boto:auth_path=/hiiqos-60/obj55 2023-07-29T00:44:54.283 DEBUG:boto:Method: GET 2023-07-29T00:44:54.283 DEBUG:boto:Path: /hiiqos-60/obj55 2023-07-29T00:44:54.284 DEBUG:boto:Data: 2023-07-29T00:44:54.284 DEBUG:boto:Headers: {} 2023-07-29T00:44:54.285 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:44:54.285 DEBUG:boto:Port: 8000 2023-07-29T00:44:54.285 DEBUG:boto:Params: {} 2023-07-29T00:44:54.286 DEBUG:boto:Token: None 2023-07-29T00:44:54.286 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:54 GMT /hiiqos-60/obj55 2023-07-29T00:44:54.286 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eKVcx9uheXQbvGRK8UGgZffEhe8= 2023-07-29T00:44:54.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eKVcx9uheXQbvGRK8UGgZffEhe8=', 'Content-Length': '0'} 2023-07-29T00:44:54.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:42:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd996dd251904c9c-0064c46106-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:44:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:54.289 DEBUG:boto:path=/hiiqos-60/ 2023-07-29T00:44:54.290 DEBUG:boto:auth_path=/hiiqos-60/ 2023-07-29T00:44:54.290 DEBUG:boto:path=/hiiqos-60/?prefix=obj55 2023-07-29T00:44:54.290 DEBUG:boto:auth_path=/hiiqos-60/?prefix=obj55 2023-07-29T00:44:54.291 DEBUG:boto:Method: GET 2023-07-29T00:44:54.291 DEBUG:boto:Path: /hiiqos-60/?prefix=obj55 2023-07-29T00:44:54.292 DEBUG:boto:Data: 2023-07-29T00:44:54.292 DEBUG:boto:Headers: {} 2023-07-29T00:44:54.292 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:44:54.293 DEBUG:boto:Port: 8001 2023-07-29T00:44:54.293 DEBUG:boto:Params: {} 2023-07-29T00:44:54.293 DEBUG:boto:Token: None 2023-07-29T00:44:54.294 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:54 GMT /hiiqos-60/ 2023-07-29T00:44:54.294 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ka4R+DOpm2+zi/bqi5gJmE5hovE= 2023-07-29T00:44:54.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ka4R+DOpm2+zi/bqi5gJmE5hovE=', 'Content-Length': '0'} 2023-07-29T00:44:54.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b586ecee845bb99-0064c46106-1f88-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:44:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:54.297 DEBUG:boto:b'hiiqos-60obj551000falseobj552023-07-29T00:42:16.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:44:54.298 DEBUG:boto:path=/hiiqos-60/ 2023-07-29T00:44:54.298 DEBUG:boto:auth_path=/hiiqos-60/ 2023-07-29T00:44:54.298 DEBUG:boto:path=/hiiqos-60/?prefix=obj55 2023-07-29T00:44:54.299 DEBUG:boto:auth_path=/hiiqos-60/?prefix=obj55 2023-07-29T00:44:54.299 DEBUG:boto:Method: GET 2023-07-29T00:44:54.300 DEBUG:boto:Path: /hiiqos-60/?prefix=obj55 2023-07-29T00:44:54.300 DEBUG:boto:Data: 2023-07-29T00:44:54.300 DEBUG:boto:Headers: {} 2023-07-29T00:44:54.301 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:44:54.301 DEBUG:boto:Port: 8000 2023-07-29T00:44:54.301 DEBUG:boto:Params: {} 2023-07-29T00:44:54.302 DEBUG:boto:Token: None 2023-07-29T00:44:54.302 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:44:54 GMT /hiiqos-60/ 2023-07-29T00:44:54.303 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ka4R+DOpm2+zi/bqi5gJmE5hovE= 2023-07-29T00:44:54.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ka4R+DOpm2+zi/bqi5gJmE5hovE=', 'Content-Length': '0'} 2023-07-29T00:44:54.305 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022bcdb6daee11d14-0064c46106-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:44:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:54.348 DEBUG:boto:b'hiiqos-60obj551000falseobj552023-07-29T00:42:16.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:44:54.349 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-60 zones={a2, a1} 2023-07-29T00:44:54.349 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-07-29T00:44:54.350 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:44:54.350 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:44:54.351 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:44:54.395 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:44:54.395 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:44:54.442 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T00:44:54.443 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:44:54.443 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031161", 2023-07-29T00:44:54.444 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:42:16.795785Z" 2023-07-29T00:44:54.444 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:44:54.444 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:44:54.444 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000027343", 2023-07-29T00:44:54.445 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:42:17.162169Z" 2023-07-29T00:44:54.445 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:44:54.445 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:44:54.445 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000025978", 2023-07-29T00:44:54.446 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:42:17.164875Z" 2023-07-29T00:44:54.446 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:44:54.446 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:44:54.447 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000018328", 2023-07-29T00:44:54.447 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:37:53.102396Z" 2023-07-29T00:44:54.447 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:44:54.447 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T00:44:54.448 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000031161', 1: '00000000000000000000:00000000000000027343', 2: '00000000000000000000:00000000000000025978', 3: '00000000000000000000:00000000000000018328'} 2023-07-29T00:44:54.449 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-07-29T00:44:54.449 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:44:54.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:44:54.450 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:44:54.497 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:44:54.498 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:44:54.543 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:44:54.543 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:44:54.543 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:44:54.544 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:44:54.544 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:44:54.544 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 15041999191289473323 2023-07-29T00:44:54.544 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:44:54.544 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:44:54.545 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:44:54.545 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:44:54.545 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:44:54.545 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:44:54.545 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031161", 2023-07-29T00:44:54.546 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:44:54.546 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 51, 2023-07-29T00:44:54.546 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:44:54.546 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:42:16.795785Z" 2023-07-29T00:44:54.546 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:44:54.546 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:44:54.547 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:44:54.547 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:44:54.547 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:44:54.547 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:44:54.547 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000027343", 2023-07-29T00:44:54.547 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:44:54.547 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T00:44:54.548 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:44:54.548 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:42:17.162169Z" 2023-07-29T00:44:54.548 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:44:54.548 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:44:54.548 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:44:54.548 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:44:54.549 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:44:54.549 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:44:54.549 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000025978", 2023-07-29T00:44:54.549 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:44:54.549 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:44:54.549 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:44:54.549 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:42:17.164875Z" 2023-07-29T00:44:54.550 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:44:54.550 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:44:54.550 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:44:54.550 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:44:54.550 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:44:54.550 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:44:54.550 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000018328", 2023-07-29T00:44:54.551 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:44:54.551 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:44:54.551 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:44:54.551 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:37:53.102396Z" 2023-07-29T00:44:54.551 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:44:54.551 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:44:54.551 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:44:54.552 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:44:54.552 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:44:54.552 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 60, 2023-07-29T00:44:54.552 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 60 2023-07-29T00:44:54.552 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:44:54.552 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:44:54Z" 2023-07-29T00:44:54.553 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:44:54.553 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 15041999191289473323 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031161", "next_step_marker": "", "total_entries": 51, "pos": 0, "timestamp": "2023-07-29T00:42:16.795785Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027343", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T00:42:17.162169Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000025978", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:42:17.164875Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018328", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:37:53.102396Z" } } ] }, "full_sync": { "total": 60, "complete": 60 }, "current_time": "2023-07-29T00:44:54Z" } 2023-07-29T00:44:54.554 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031161', 'next_step_marker': '', 'total_entries': 51, 'pos': 0, 'timestamp': '2023-07-29T00:42:16.795785Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027343', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T00:42:17.162169Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000025978', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:42:17.164875Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018328', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:37:53.102396Z'}}] 2023-07-29T00:44:54.554 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000031161', 1: '00000000000000000000:00000000000000027343', 2: '00000000000000000000:00000000000000025978', 3: '00000000000000000000:00000000000000018328'} 2023-07-29T00:44:54.555 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000031161', 1: '00000000000000000000:00000000000000027343', 2: '00000000000000000000:00000000000000025978', 3: '00000000000000000000:00000000000000018328'} 2023-07-29T00:44:54.555 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-07-29T00:44:54.555 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-07-29T00:44:54.556 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:44:54.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:44:54.557 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:44:54.601 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:44:54.602 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:44:54.649 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T00:44:54.650 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:44:54.650 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000014460", 2023-07-29T00:44:54.650 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:43:47.791779Z" 2023-07-29T00:44:54.650 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:44:54.650 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:44:54.651 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000013194", 2023-07-29T00:44:54.651 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:44:07.069014Z" 2023-07-29T00:44:54.651 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:44:54.651 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:44:54.651 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000010405", 2023-07-29T00:44:54.651 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:44:07.070709Z" 2023-07-29T00:44:54.652 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:44:54.652 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:44:54.652 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000005022", 2023-07-29T00:44:54.652 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:39:53.006554Z" 2023-07-29T00:44:54.652 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:44:54.652 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T00:44:54.653 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000014460', 1: '00000000000000000000:00000000000000013194', 2: '00000000000000000000:00000000000000010405', 3: '00000000000000000000:00000000000000005022'} 2023-07-29T00:44:54.654 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-07-29T00:44:54.654 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:44:54.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:44:54.655 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:44:54.703 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:44:54.703 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:44:54.745 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:44:54.745 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:44:54.745 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:44:54.746 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:44:54.746 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:44:54.746 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 17882949198279251152 2023-07-29T00:44:54.746 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:44:54.747 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:44:54.747 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:44:54.747 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:44:54.747 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:44:54.747 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:44:54.748 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000014460", 2023-07-29T00:44:54.748 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:44:54.748 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:44:54.748 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:44:54.748 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:43:47.791779Z" 2023-07-29T00:44:54.748 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:44:54.748 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:44:54.749 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:44:54.749 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:44:54.749 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:44:54.749 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:44:54.749 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000013194", 2023-07-29T00:44:54.749 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:44:54.749 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:44:54.750 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:44:54.750 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:44:07.069014Z" 2023-07-29T00:44:54.750 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:44:54.750 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:44:54.750 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:44:54.750 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:44:54.750 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:44:54.751 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:44:54.751 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000010405", 2023-07-29T00:44:54.751 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:44:54.751 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:44:54.751 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:44:54.751 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:44:07.070709Z" 2023-07-29T00:44:54.752 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:44:54.752 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:44:54.752 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:44:54.752 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:44:54.753 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:44:54.753 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:44:54.753 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000005022", 2023-07-29T00:44:54.753 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:44:54.754 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:44:54.754 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:44:54.754 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:39:53.006554Z" 2023-07-29T00:44:54.754 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:44:54.754 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:44:54.755 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:44:54.755 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:44:54.755 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:44:54.755 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 0, 2023-07-29T00:44:54.755 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 0 2023-07-29T00:44:54.755 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:44:54.756 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:44:54Z" 2023-07-29T00:44:54.756 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:44:54.757 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17882949198279251152 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000014460", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:43:47.791779Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013194", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:44:07.069014Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000010405", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:44:07.070709Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005022", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:39:53.006554Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-07-29T00:44:54Z" } 2023-07-29T00:44:54.757 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014460', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:43:47.791779Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013194', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:44:07.069014Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000010405', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:44:07.070709Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005022', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:39:53.006554Z'}}] 2023-07-29T00:44:54.758 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000014460', 1: '00000000000000000000:00000000000000013194', 2: '00000000000000000000:00000000000000010405', 3: '00000000000000000000:00000000000000005022'} 2023-07-29T00:44:54.758 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000014460', 1: '00000000000000000000:00000000000000013194', 2: '00000000000000000000:00000000000000010405', 3: '00000000000000000000:00000000000000005022'} 2023-07-29T00:44:54.759 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-07-29T00:44:54.759 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_full_sync_after_data_sync_init ... ok 2023-07-29T00:44:54.760 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:44:54.761 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:44:54.761 DEBUG:boto:Method: PUT 2023-07-29T00:44:54.761 DEBUG:boto:Path: /hiiqos-61/ 2023-07-29T00:44:54.762 DEBUG:boto:Data: 2023-07-29T00:44:54.762 DEBUG:boto:Headers: {} 2023-07-29T00:44:54.763 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:44:54.763 DEBUG:boto:Port: 8000 2023-07-29T00:44:54.763 DEBUG:boto:Params: {} 2023-07-29T00:44:54.764 DEBUG:boto:Token: None 2023-07-29T00:44:54.764 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 00:44:54 GMT /hiiqos-61/ 2023-07-29T00:44:54.765 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:E2xauxpbOQIskHtKE0TGz8lrlLE= 2023-07-29T00:44:54.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:44:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:E2xauxpbOQIskHtKE0TGz8lrlLE=', 'Content-Length': '0'} 2023-07-29T00:44:54.779 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000aa745e0a51c508eb-0064c46106-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:44:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:44:54.779 DEBUG:rgw_multi.tests:created bucket=hiiqos-61 2023-07-29T00:44:54.780 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:44:54.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:44:54.781 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:44:54.827 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:44:54.827 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:44:54.865 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T00:44:54.878 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T00:44:54.878 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:44:54.879 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T00:44:54.879 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:17.836402Z" 2023-07-29T00:44:54.879 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:44:54.879 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:44:54.879 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591494.709843_1096.1", 2023-07-29T00:44:54.879 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:44:54.709843Z" 2023-07-29T00:44:54.879 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:44:54.880 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:44:54.880 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:44:54.880 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T00:44:54.880 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:44:54.880 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:44:54.880 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T00:44:54.881 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:12.811406Z" 2023-07-29T00:44:54.881 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:44:54.881 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T00:44:54.882 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690591217.836402_294.1', 1: '1_1690591494.709843_1096.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T00:44:54.882 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T00:44:54.883 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:44:54.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:44:54.884 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:44:54.931 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:44:54.931 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:44:54.960 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:44:54.892+0000 7ffaf0012880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:44:54.961 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:44:54.893+0000 7ffaf0012880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:44:54.974 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:44:54.975 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:44:54.975 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:44:54.975 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:44:54.975 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:44:54.975 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:44:54.976 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:44:54.976 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:44:54.976 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:44:54.976 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:44:54.976 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:44:54.976 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:44:54.977 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:44:54.977 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T00:44:54.977 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:44:54.977 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:44:54.977 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:44:54.977 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T00:44:54.978 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:44:54.978 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:44:54.978 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:44:54.978 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:44:54.978 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:44:54.978 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:44:54.978 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:44:54.979 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591494.709843_1096.1", 2023-07-29T00:44:54.979 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:44:54.979 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:44:54.979 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:44:54.979 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:44:54.709843Z", 2023-07-29T00:44:54.979 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:44:54.979 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:44:54.980 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:44:54.980 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:44:54.980 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:44:54.980 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:44:54.980 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:44:54.981 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:44:54.981 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:44:54.982 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:44:54.982 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:44:54.982 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:44:54.982 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:44:54.982 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:44:54.982 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:44:54.982 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:44:54.983 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:44:54.983 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:44:54.983 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:44:54.983 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T00:44:54.983 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:44:54.983 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:44:54.984 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:44:54.984 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T00:44:54.984 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:44:54.984 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:44:54.984 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:44:54.984 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:44:54.985 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:44:54.985 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:44:54.985 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:44:54.985 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:44:54.985 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:44:54.985 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:44:54Z" 2023-07-29T00:44:54.985 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:44:54.986 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690591494.709843_1096.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:44:54.709843Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:44:54Z" } 2023-07-29T00:44:54.987 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690591494.709843_1096.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:44:54.709843Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T00:44:54.987 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690591494.709843_1096.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T00:44:54.988 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690591494.709843_1096.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T00:44:54.988 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T00:44:54.989 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-07-29T00:44:54.989 INFO:teuthology.orchestra.run:waiting for 300 2023-07-29T00:44:54.990 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:44:54.921+0000 7f31e5d5f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 141384) UID: 0 2023-07-29T00:44:54.990 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:44:54.921+0000 7f31ebe34880 -1 shutting down 2023-07-29T00:44:59.451 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-07-29T00:45:01.092 INFO:tasks.rgw.c1.client.1:Stopped 2023-07-29T00:45:01.092 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:45:01.093 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:45:01.093 DEBUG:boto:Method: HEAD 2023-07-29T00:45:01.093 DEBUG:boto:Path: /hiiqos-61/ 2023-07-29T00:45:01.094 DEBUG:boto:Data: 2023-07-29T00:45:01.094 DEBUG:boto:Headers: {} 2023-07-29T00:45:01.095 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:01.095 DEBUG:boto:Port: 8000 2023-07-29T00:45:01.095 DEBUG:boto:Params: {} 2023-07-29T00:45:01.096 DEBUG:boto:Token: None 2023-07-29T00:45:01.096 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:45:01 GMT /hiiqos-61/ 2023-07-29T00:45:01.096 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gT8Me8QNKQIxMkTUOJIA3HgQsvs= 2023-07-29T00:45:01.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:45:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gT8Me8QNKQIxMkTUOJIA3HgQsvs=', 'Content-Length': '0'} 2023-07-29T00:45:01.100 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000034d54660e8aab487-0064c4610d-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:01.100 DEBUG:boto:path=/hiiqos-61/obj11 2023-07-29T00:45:01.101 DEBUG:boto:auth_path=/hiiqos-61/obj11 2023-07-29T00:45:01.101 DEBUG:boto:Method: PUT 2023-07-29T00:45:01.101 DEBUG:boto:Path: /hiiqos-61/obj11 2023-07-29T00:45:01.102 DEBUG:boto:Data: 2023-07-29T00:45:01.102 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:45:01.102 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:01.103 DEBUG:boto:Port: 8000 2023-07-29T00:45:01.103 DEBUG:boto:Params: {} 2023-07-29T00:45:01.104 DEBUG:boto:Token: None 2023-07-29T00:45:01.104 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:45:01 GMT /hiiqos-61/obj11 2023-07-29T00:45:01.104 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2MYk4WG4CFR3+r+87WBukdoxoGM= 2023-07-29T00:45:01.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:45:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2MYk4WG4CFR3+r+87WBukdoxoGM='} 2023-07-29T00:45:01.113 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004046785723f496ac-0064c4610d-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:01.113 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:45:01.114 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:45:01.114 DEBUG:boto:Method: HEAD 2023-07-29T00:45:01.115 DEBUG:boto:Path: /hiiqos-61/ 2023-07-29T00:45:01.115 DEBUG:boto:Data: 2023-07-29T00:45:01.116 DEBUG:boto:Headers: {} 2023-07-29T00:45:01.116 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:01.116 DEBUG:boto:Port: 8000 2023-07-29T00:45:01.117 DEBUG:boto:Params: {} 2023-07-29T00:45:01.117 DEBUG:boto:Token: None 2023-07-29T00:45:01.118 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:45:01 GMT /hiiqos-61/ 2023-07-29T00:45:01.118 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gT8Me8QNKQIxMkTUOJIA3HgQsvs= 2023-07-29T00:45:01.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:45:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gT8Me8QNKQIxMkTUOJIA3HgQsvs=', 'Content-Length': '0'} 2023-07-29T00:45:01.121 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000037e30709543d740-0064c4610d-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:01.122 DEBUG:boto:path=/hiiqos-61/obj22 2023-07-29T00:45:01.122 DEBUG:boto:auth_path=/hiiqos-61/obj22 2023-07-29T00:45:01.122 DEBUG:boto:Method: PUT 2023-07-29T00:45:01.123 DEBUG:boto:Path: /hiiqos-61/obj22 2023-07-29T00:45:01.123 DEBUG:boto:Data: 2023-07-29T00:45:01.123 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:45:01.124 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:01.124 DEBUG:boto:Port: 8000 2023-07-29T00:45:01.124 DEBUG:boto:Params: {} 2023-07-29T00:45:01.125 DEBUG:boto:Token: None 2023-07-29T00:45:01.126 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:45:01 GMT /hiiqos-61/obj22 2023-07-29T00:45:01.126 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:27iDnh8v1d4Bn9iyLSlcE6QFDkU= 2023-07-29T00:45:01.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:45:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:27iDnh8v1d4Bn9iyLSlcE6QFDkU='} 2023-07-29T00:45:01.131 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b735edca0e47e381-0064c4610d-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:01.132 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:45:01.133 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:45:01.133 DEBUG:boto:Method: HEAD 2023-07-29T00:45:01.134 DEBUG:boto:Path: /hiiqos-61/ 2023-07-29T00:45:01.135 DEBUG:boto:Data: 2023-07-29T00:45:01.135 DEBUG:boto:Headers: {} 2023-07-29T00:45:01.136 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:01.136 DEBUG:boto:Port: 8000 2023-07-29T00:45:01.137 DEBUG:boto:Params: {} 2023-07-29T00:45:01.137 DEBUG:boto:Token: None 2023-07-29T00:45:01.138 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:45:01 GMT /hiiqos-61/ 2023-07-29T00:45:01.138 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gT8Me8QNKQIxMkTUOJIA3HgQsvs= 2023-07-29T00:45:01.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:45:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gT8Me8QNKQIxMkTUOJIA3HgQsvs=', 'Content-Length': '0'} 2023-07-29T00:45:01.141 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d85ef15321093a7-0064c4610d-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:01.142 DEBUG:boto:path=/hiiqos-61/obj33 2023-07-29T00:45:01.142 DEBUG:boto:auth_path=/hiiqos-61/obj33 2023-07-29T00:45:01.142 DEBUG:boto:Method: PUT 2023-07-29T00:45:01.143 DEBUG:boto:Path: /hiiqos-61/obj33 2023-07-29T00:45:01.143 DEBUG:boto:Data: 2023-07-29T00:45:01.143 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:45:01.144 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:01.144 DEBUG:boto:Port: 8000 2023-07-29T00:45:01.145 DEBUG:boto:Params: {} 2023-07-29T00:45:01.146 DEBUG:boto:Token: None 2023-07-29T00:45:01.146 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:45:01 GMT /hiiqos-61/obj33 2023-07-29T00:45:01.147 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2lYU1bzAH0GlOtGztjrCUmx4VCk= 2023-07-29T00:45:01.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:45:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2lYU1bzAH0GlOtGztjrCUmx4VCk='} 2023-07-29T00:45:01.192 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000af4564a1ac907c27-0064c4610d-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:01.193 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:45:01.194 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:45:01.194 DEBUG:boto:Method: HEAD 2023-07-29T00:45:01.195 DEBUG:boto:Path: /hiiqos-61/ 2023-07-29T00:45:01.195 DEBUG:boto:Data: 2023-07-29T00:45:01.195 DEBUG:boto:Headers: {} 2023-07-29T00:45:01.195 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:01.196 DEBUG:boto:Port: 8000 2023-07-29T00:45:01.196 DEBUG:boto:Params: {} 2023-07-29T00:45:01.197 DEBUG:boto:Token: None 2023-07-29T00:45:01.197 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:45:01 GMT /hiiqos-61/ 2023-07-29T00:45:01.198 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gT8Me8QNKQIxMkTUOJIA3HgQsvs= 2023-07-29T00:45:01.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:45:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gT8Me8QNKQIxMkTUOJIA3HgQsvs=', 'Content-Length': '0'} 2023-07-29T00:45:01.200 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ca9fb73a4cf10517-0064c4610d-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:01.201 DEBUG:boto:path=/hiiqos-61/obj44 2023-07-29T00:45:01.201 DEBUG:boto:auth_path=/hiiqos-61/obj44 2023-07-29T00:45:01.202 DEBUG:boto:Method: PUT 2023-07-29T00:45:01.202 DEBUG:boto:Path: /hiiqos-61/obj44 2023-07-29T00:45:01.202 DEBUG:boto:Data: 2023-07-29T00:45:01.203 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:45:01.203 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:01.203 DEBUG:boto:Port: 8000 2023-07-29T00:45:01.204 DEBUG:boto:Params: {} 2023-07-29T00:45:01.204 DEBUG:boto:Token: None 2023-07-29T00:45:01.205 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:45:01 GMT /hiiqos-61/obj44 2023-07-29T00:45:01.205 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VWh1/9JVXbdW2yCT5bg2JWjAENo= 2023-07-29T00:45:01.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:45:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VWh1/9JVXbdW2yCT5bg2JWjAENo='} 2023-07-29T00:45:01.212 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000318d7e38ba7e5d9a-0064c4610d-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:01.212 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:45:01.212 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:45:01.213 DEBUG:boto:Method: HEAD 2023-07-29T00:45:01.213 DEBUG:boto:Path: /hiiqos-61/ 2023-07-29T00:45:01.213 DEBUG:boto:Data: 2023-07-29T00:45:01.214 DEBUG:boto:Headers: {} 2023-07-29T00:45:01.214 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:01.214 DEBUG:boto:Port: 8000 2023-07-29T00:45:01.215 DEBUG:boto:Params: {} 2023-07-29T00:45:01.215 DEBUG:boto:Token: None 2023-07-29T00:45:01.216 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:45:01 GMT /hiiqos-61/ 2023-07-29T00:45:01.216 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gT8Me8QNKQIxMkTUOJIA3HgQsvs= 2023-07-29T00:45:01.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:45:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gT8Me8QNKQIxMkTUOJIA3HgQsvs=', 'Content-Length': '0'} 2023-07-29T00:45:01.219 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d8edfda07db9fc2-0064c4610d-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:01.220 DEBUG:boto:path=/hiiqos-61/obj55 2023-07-29T00:45:01.221 DEBUG:boto:auth_path=/hiiqos-61/obj55 2023-07-29T00:45:01.221 DEBUG:boto:Method: PUT 2023-07-29T00:45:01.221 DEBUG:boto:Path: /hiiqos-61/obj55 2023-07-29T00:45:01.222 DEBUG:boto:Data: 2023-07-29T00:45:01.222 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:45:01.223 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:01.223 DEBUG:boto:Port: 8000 2023-07-29T00:45:01.223 DEBUG:boto:Params: {} 2023-07-29T00:45:01.224 DEBUG:boto:Token: None 2023-07-29T00:45:01.224 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:45:01 GMT /hiiqos-61/obj55 2023-07-29T00:45:01.225 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YYiVWpqai9T1eE3hg8X62xr+VcA= 2023-07-29T00:45:01.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:45:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YYiVWpqai9T1eE3hg8X62xr+VcA='} 2023-07-29T00:45:01.268 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005ae432b48bea6980-0064c4610d-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:01.269 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-61', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:45:01.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-61', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:45:01.270 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-61 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:45:01.327 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:45:01.328 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:45:01.378 INFO:teuthology.orchestra.run.smithi101.stdout:tenant: 2023-07-29T00:45:01.378 INFO:teuthology.orchestra.run.smithi101.stdout:bucket name: hiiqos-61 2023-07-29T00:45:01.382 INFO:teuthology.orchestra.run.smithi101.stdout:total entries: 5 2023-07-29T00:45:01.421 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:45:01.421 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:45:01.422 DEBUG:boto:Method: HEAD 2023-07-29T00:45:01.422 DEBUG:boto:Path: /hiiqos-61/ 2023-07-29T00:45:01.422 DEBUG:boto:Data: 2023-07-29T00:45:01.423 DEBUG:boto:Headers: {} 2023-07-29T00:45:01.423 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:01.423 DEBUG:boto:Port: 8000 2023-07-29T00:45:01.424 DEBUG:boto:Params: {} 2023-07-29T00:45:01.424 DEBUG:boto:Token: None 2023-07-29T00:45:01.424 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:45:01 GMT /hiiqos-61/ 2023-07-29T00:45:01.425 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gT8Me8QNKQIxMkTUOJIA3HgQsvs= 2023-07-29T00:45:01.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:45:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gT8Me8QNKQIxMkTUOJIA3HgQsvs=', 'Content-Length': '0'} 2023-07-29T00:45:01.428 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000198a8ccbc81972d1-0064c4610d-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:01.428 DEBUG:boto:path=/hiiqos-61/obj17 2023-07-29T00:45:01.429 DEBUG:boto:auth_path=/hiiqos-61/obj17 2023-07-29T00:45:01.429 DEBUG:boto:Method: PUT 2023-07-29T00:45:01.429 DEBUG:boto:Path: /hiiqos-61/obj17 2023-07-29T00:45:01.430 DEBUG:boto:Data: 2023-07-29T00:45:01.430 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:45:01.431 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:01.431 DEBUG:boto:Port: 8000 2023-07-29T00:45:01.431 DEBUG:boto:Params: {} 2023-07-29T00:45:01.432 DEBUG:boto:Token: None 2023-07-29T00:45:01.432 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:45:01 GMT /hiiqos-61/obj17 2023-07-29T00:45:01.432 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lnStMeikbwml/kWvQqAyNboD9pA= 2023-07-29T00:45:01.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:45:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lnStMeikbwml/kWvQqAyNboD9pA='} 2023-07-29T00:45:01.438 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000057d2c345bcdaa4ea-0064c4610d-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:01.438 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:45:01.439 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:45:01.439 DEBUG:boto:Method: HEAD 2023-07-29T00:45:01.439 DEBUG:boto:Path: /hiiqos-61/ 2023-07-29T00:45:01.440 DEBUG:boto:Data: 2023-07-29T00:45:01.440 DEBUG:boto:Headers: {} 2023-07-29T00:45:01.440 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:01.441 DEBUG:boto:Port: 8000 2023-07-29T00:45:01.441 DEBUG:boto:Params: {} 2023-07-29T00:45:01.441 DEBUG:boto:Token: None 2023-07-29T00:45:01.442 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:45:01 GMT /hiiqos-61/ 2023-07-29T00:45:01.442 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gT8Me8QNKQIxMkTUOJIA3HgQsvs= 2023-07-29T00:45:01.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:45:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gT8Me8QNKQIxMkTUOJIA3HgQsvs=', 'Content-Length': '0'} 2023-07-29T00:45:01.445 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006e0bbb86505bc180-0064c4610d-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:01.445 DEBUG:boto:path=/hiiqos-61/obj34 2023-07-29T00:45:01.446 DEBUG:boto:auth_path=/hiiqos-61/obj34 2023-07-29T00:45:01.446 DEBUG:boto:Method: PUT 2023-07-29T00:45:01.446 DEBUG:boto:Path: /hiiqos-61/obj34 2023-07-29T00:45:01.447 DEBUG:boto:Data: 2023-07-29T00:45:01.447 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:45:01.447 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:01.448 DEBUG:boto:Port: 8000 2023-07-29T00:45:01.448 DEBUG:boto:Params: {} 2023-07-29T00:45:01.448 DEBUG:boto:Token: None 2023-07-29T00:45:01.449 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:45:01 GMT /hiiqos-61/obj34 2023-07-29T00:45:01.449 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qWFDCPJP/1MNV4W05fMD2Vsrs/o= 2023-07-29T00:45:01.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:45:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qWFDCPJP/1MNV4W05fMD2Vsrs/o='} 2023-07-29T00:45:01.454 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fc67b2fa06937f4d-0064c4610d-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:01.455 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:45:01.455 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:45:01.456 DEBUG:boto:Method: HEAD 2023-07-29T00:45:01.456 DEBUG:boto:Path: /hiiqos-61/ 2023-07-29T00:45:01.456 DEBUG:boto:Data: 2023-07-29T00:45:01.457 DEBUG:boto:Headers: {} 2023-07-29T00:45:01.457 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:01.457 DEBUG:boto:Port: 8000 2023-07-29T00:45:01.458 DEBUG:boto:Params: {} 2023-07-29T00:45:01.458 DEBUG:boto:Token: None 2023-07-29T00:45:01.458 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:45:01 GMT /hiiqos-61/ 2023-07-29T00:45:01.459 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gT8Me8QNKQIxMkTUOJIA3HgQsvs= 2023-07-29T00:45:01.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:45:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gT8Me8QNKQIxMkTUOJIA3HgQsvs=', 'Content-Length': '0'} 2023-07-29T00:45:01.461 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a425754b9ddc3fe2-0064c4610d-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:01.462 DEBUG:boto:path=/hiiqos-61/obj51 2023-07-29T00:45:01.462 DEBUG:boto:auth_path=/hiiqos-61/obj51 2023-07-29T00:45:01.463 DEBUG:boto:Method: PUT 2023-07-29T00:45:01.463 DEBUG:boto:Path: /hiiqos-61/obj51 2023-07-29T00:45:01.463 DEBUG:boto:Data: 2023-07-29T00:45:01.464 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:45:01.464 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:01.464 DEBUG:boto:Port: 8000 2023-07-29T00:45:01.465 DEBUG:boto:Params: {} 2023-07-29T00:45:01.465 DEBUG:boto:Token: None 2023-07-29T00:45:01.465 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:45:01 GMT /hiiqos-61/obj51 2023-07-29T00:45:01.466 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xm6I9vRdtWzuPvSUMLxh+1XFCjY= 2023-07-29T00:45:01.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:45:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xm6I9vRdtWzuPvSUMLxh+1XFCjY='} 2023-07-29T00:45:01.512 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000061991033098a7418-0064c4610d-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:01.513 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:45:01.513 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:45:01.514 DEBUG:boto:Method: HEAD 2023-07-29T00:45:01.514 DEBUG:boto:Path: /hiiqos-61/ 2023-07-29T00:45:01.514 DEBUG:boto:Data: 2023-07-29T00:45:01.515 DEBUG:boto:Headers: {} 2023-07-29T00:45:01.515 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:01.515 DEBUG:boto:Port: 8000 2023-07-29T00:45:01.516 DEBUG:boto:Params: {} 2023-07-29T00:45:01.516 DEBUG:boto:Token: None 2023-07-29T00:45:01.516 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:45:01 GMT /hiiqos-61/ 2023-07-29T00:45:01.517 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gT8Me8QNKQIxMkTUOJIA3HgQsvs= 2023-07-29T00:45:01.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:45:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gT8Me8QNKQIxMkTUOJIA3HgQsvs=', 'Content-Length': '0'} 2023-07-29T00:45:01.520 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000028113d806775350a-0064c4610d-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:01.520 DEBUG:boto:path=/hiiqos-61/obj68 2023-07-29T00:45:01.521 DEBUG:boto:auth_path=/hiiqos-61/obj68 2023-07-29T00:45:01.521 DEBUG:boto:Method: PUT 2023-07-29T00:45:01.521 DEBUG:boto:Path: /hiiqos-61/obj68 2023-07-29T00:45:01.522 DEBUG:boto:Data: 2023-07-29T00:45:01.522 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:45:01.522 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:01.523 DEBUG:boto:Port: 8000 2023-07-29T00:45:01.523 DEBUG:boto:Params: {} 2023-07-29T00:45:01.523 DEBUG:boto:Token: None 2023-07-29T00:45:01.524 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:45:01 GMT /hiiqos-61/obj68 2023-07-29T00:45:01.524 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TKHy15FDV5nuSE1MJuesQoceYx4= 2023-07-29T00:45:01.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:45:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TKHy15FDV5nuSE1MJuesQoceYx4='} 2023-07-29T00:45:01.529 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000049ddcd71da42188-0064c4610d-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:01.530 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:45:01.530 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:45:01.531 DEBUG:boto:Method: HEAD 2023-07-29T00:45:01.531 DEBUG:boto:Path: /hiiqos-61/ 2023-07-29T00:45:01.531 DEBUG:boto:Data: 2023-07-29T00:45:01.532 DEBUG:boto:Headers: {} 2023-07-29T00:45:01.532 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:01.532 DEBUG:boto:Port: 8000 2023-07-29T00:45:01.533 DEBUG:boto:Params: {} 2023-07-29T00:45:01.533 DEBUG:boto:Token: None 2023-07-29T00:45:01.533 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:45:01 GMT /hiiqos-61/ 2023-07-29T00:45:01.534 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gT8Me8QNKQIxMkTUOJIA3HgQsvs= 2023-07-29T00:45:01.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:45:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gT8Me8QNKQIxMkTUOJIA3HgQsvs=', 'Content-Length': '0'} 2023-07-29T00:45:01.536 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7a94a0bb7d75ba4-0064c4610d-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:01.537 DEBUG:boto:path=/hiiqos-61/obj85 2023-07-29T00:45:01.537 DEBUG:boto:auth_path=/hiiqos-61/obj85 2023-07-29T00:45:01.538 DEBUG:boto:Method: PUT 2023-07-29T00:45:01.538 DEBUG:boto:Path: /hiiqos-61/obj85 2023-07-29T00:45:01.538 DEBUG:boto:Data: 2023-07-29T00:45:01.539 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:45:01.539 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:01.539 DEBUG:boto:Port: 8000 2023-07-29T00:45:01.540 DEBUG:boto:Params: {} 2023-07-29T00:45:01.540 DEBUG:boto:Token: None 2023-07-29T00:45:01.540 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:45:01 GMT /hiiqos-61/obj85 2023-07-29T00:45:01.541 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yGRCaUP08b+bPux/uyCh7SbCCOw= 2023-07-29T00:45:01.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:45:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yGRCaUP08b+bPux/uyCh7SbCCOw='} 2023-07-29T00:45:01.584 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009188c50e8e61c283-0064c4610d-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:01.585 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-61', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:45:01.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-61', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:45:01.586 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-61 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:45:01.635 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:45:01.636 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:45:01.677 INFO:teuthology.orchestra.run.smithi101.stdout:tenant: 2023-07-29T00:45:01.677 INFO:teuthology.orchestra.run.smithi101.stdout:bucket name: hiiqos-61 2023-07-29T00:45:01.683 INFO:teuthology.orchestra.run.smithi101.stdout:total entries: 10 2023-07-29T00:45:01.724 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:45:01.724 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:45:01.725 DEBUG:boto:Method: HEAD 2023-07-29T00:45:01.725 DEBUG:boto:Path: /hiiqos-61/ 2023-07-29T00:45:01.725 DEBUG:boto:Data: 2023-07-29T00:45:01.726 DEBUG:boto:Headers: {} 2023-07-29T00:45:01.726 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:01.726 DEBUG:boto:Port: 8000 2023-07-29T00:45:01.726 DEBUG:boto:Params: {} 2023-07-29T00:45:01.727 DEBUG:boto:Token: None 2023-07-29T00:45:01.727 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:45:01 GMT /hiiqos-61/ 2023-07-29T00:45:01.728 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gT8Me8QNKQIxMkTUOJIA3HgQsvs= 2023-07-29T00:45:01.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:45:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gT8Me8QNKQIxMkTUOJIA3HgQsvs=', 'Content-Length': '0'} 2023-07-29T00:45:01.731 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f134701e8e8bcddd-0064c4610d-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:01.731 DEBUG:boto:path=/hiiqos-61/obj19 2023-07-29T00:45:01.732 DEBUG:boto:auth_path=/hiiqos-61/obj19 2023-07-29T00:45:01.732 DEBUG:boto:Method: PUT 2023-07-29T00:45:01.733 DEBUG:boto:Path: /hiiqos-61/obj19 2023-07-29T00:45:01.733 DEBUG:boto:Data: 2023-07-29T00:45:01.733 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:45:01.734 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:01.735 DEBUG:boto:Port: 8000 2023-07-29T00:45:01.736 DEBUG:boto:Params: {} 2023-07-29T00:45:01.737 DEBUG:boto:Token: None 2023-07-29T00:45:01.738 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:45:01 GMT /hiiqos-61/obj19 2023-07-29T00:45:01.739 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FJDT8W96kyJLYjDFKlAqJHbA9go= 2023-07-29T00:45:01.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:45:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FJDT8W96kyJLYjDFKlAqJHbA9go='} 2023-07-29T00:45:01.745 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009d73c1593f691d6b-0064c4610d-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:01.746 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:45:01.746 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:45:01.746 DEBUG:boto:Method: HEAD 2023-07-29T00:45:01.747 DEBUG:boto:Path: /hiiqos-61/ 2023-07-29T00:45:01.747 DEBUG:boto:Data: 2023-07-29T00:45:01.747 DEBUG:boto:Headers: {} 2023-07-29T00:45:01.748 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:01.748 DEBUG:boto:Port: 8000 2023-07-29T00:45:01.748 DEBUG:boto:Params: {} 2023-07-29T00:45:01.749 DEBUG:boto:Token: None 2023-07-29T00:45:01.749 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:45:01 GMT /hiiqos-61/ 2023-07-29T00:45:01.749 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gT8Me8QNKQIxMkTUOJIA3HgQsvs= 2023-07-29T00:45:01.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:45:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gT8Me8QNKQIxMkTUOJIA3HgQsvs=', 'Content-Length': '0'} 2023-07-29T00:45:01.752 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d4e0e5a51e893b8a-0064c4610d-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:01.753 DEBUG:boto:path=/hiiqos-61/obj38 2023-07-29T00:45:01.753 DEBUG:boto:auth_path=/hiiqos-61/obj38 2023-07-29T00:45:01.754 DEBUG:boto:Method: PUT 2023-07-29T00:45:01.754 DEBUG:boto:Path: /hiiqos-61/obj38 2023-07-29T00:45:01.754 DEBUG:boto:Data: 2023-07-29T00:45:01.755 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:45:01.755 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:01.755 DEBUG:boto:Port: 8000 2023-07-29T00:45:01.756 DEBUG:boto:Params: {} 2023-07-29T00:45:01.756 DEBUG:boto:Token: None 2023-07-29T00:45:01.756 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:45:01 GMT /hiiqos-61/obj38 2023-07-29T00:45:01.757 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ALEC3NNbM+qDKEDQis+ib/iJ63U= 2023-07-29T00:45:01.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:45:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ALEC3NNbM+qDKEDQis+ib/iJ63U='} 2023-07-29T00:45:01.761 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007c232950604fea58-0064c4610d-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:01.761 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:45:01.762 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:45:01.762 DEBUG:boto:Method: HEAD 2023-07-29T00:45:01.762 DEBUG:boto:Path: /hiiqos-61/ 2023-07-29T00:45:01.763 DEBUG:boto:Data: 2023-07-29T00:45:01.763 DEBUG:boto:Headers: {} 2023-07-29T00:45:01.763 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:01.764 DEBUG:boto:Port: 8000 2023-07-29T00:45:01.764 DEBUG:boto:Params: {} 2023-07-29T00:45:01.764 DEBUG:boto:Token: None 2023-07-29T00:45:01.765 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:45:01 GMT /hiiqos-61/ 2023-07-29T00:45:01.765 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gT8Me8QNKQIxMkTUOJIA3HgQsvs= 2023-07-29T00:45:01.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:45:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gT8Me8QNKQIxMkTUOJIA3HgQsvs=', 'Content-Length': '0'} 2023-07-29T00:45:01.768 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b1a30448c4535b80-0064c4610d-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:01.769 DEBUG:boto:path=/hiiqos-61/obj57 2023-07-29T00:45:01.769 DEBUG:boto:auth_path=/hiiqos-61/obj57 2023-07-29T00:45:01.770 DEBUG:boto:Method: PUT 2023-07-29T00:45:01.770 DEBUG:boto:Path: /hiiqos-61/obj57 2023-07-29T00:45:01.770 DEBUG:boto:Data: 2023-07-29T00:45:01.771 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:45:01.771 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:01.771 DEBUG:boto:Port: 8000 2023-07-29T00:45:01.772 DEBUG:boto:Params: {} 2023-07-29T00:45:01.772 DEBUG:boto:Token: None 2023-07-29T00:45:01.773 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:45:01 GMT /hiiqos-61/obj57 2023-07-29T00:45:01.773 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7rBwTHZnXD2gX6U9ZfGOFlTq4/M= 2023-07-29T00:45:01.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:45:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7rBwTHZnXD2gX6U9ZfGOFlTq4/M='} 2023-07-29T00:45:01.816 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000204ceff4b5081e9a-0064c4610d-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:01.817 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:45:01.817 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:45:01.818 DEBUG:boto:Method: HEAD 2023-07-29T00:45:01.818 DEBUG:boto:Path: /hiiqos-61/ 2023-07-29T00:45:01.818 DEBUG:boto:Data: 2023-07-29T00:45:01.819 DEBUG:boto:Headers: {} 2023-07-29T00:45:01.819 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:01.820 DEBUG:boto:Port: 8000 2023-07-29T00:45:01.820 DEBUG:boto:Params: {} 2023-07-29T00:45:01.820 DEBUG:boto:Token: None 2023-07-29T00:45:01.821 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:45:01 GMT /hiiqos-61/ 2023-07-29T00:45:01.821 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gT8Me8QNKQIxMkTUOJIA3HgQsvs= 2023-07-29T00:45:01.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:45:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gT8Me8QNKQIxMkTUOJIA3HgQsvs=', 'Content-Length': '0'} 2023-07-29T00:45:01.824 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005bc6c1a6bc8dd402-0064c4610d-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:01.825 DEBUG:boto:path=/hiiqos-61/obj76 2023-07-29T00:45:01.825 DEBUG:boto:auth_path=/hiiqos-61/obj76 2023-07-29T00:45:01.826 DEBUG:boto:Method: PUT 2023-07-29T00:45:01.826 DEBUG:boto:Path: /hiiqos-61/obj76 2023-07-29T00:45:01.826 DEBUG:boto:Data: 2023-07-29T00:45:01.827 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:45:01.827 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:01.827 DEBUG:boto:Port: 8000 2023-07-29T00:45:01.828 DEBUG:boto:Params: {} 2023-07-29T00:45:01.828 DEBUG:boto:Token: None 2023-07-29T00:45:01.828 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:45:01 GMT /hiiqos-61/obj76 2023-07-29T00:45:01.829 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zeFWJtLXM9s0N2LpYoxnNM9GWYk= 2023-07-29T00:45:01.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:45:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zeFWJtLXM9s0N2LpYoxnNM9GWYk='} 2023-07-29T00:45:01.834 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000009f34e3de15ea3c7-0064c4610d-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:01.835 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:45:01.835 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:45:01.835 DEBUG:boto:Method: HEAD 2023-07-29T00:45:01.836 DEBUG:boto:Path: /hiiqos-61/ 2023-07-29T00:45:01.836 DEBUG:boto:Data: 2023-07-29T00:45:01.836 DEBUG:boto:Headers: {} 2023-07-29T00:45:01.837 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:01.837 DEBUG:boto:Port: 8000 2023-07-29T00:45:01.837 DEBUG:boto:Params: {} 2023-07-29T00:45:01.838 DEBUG:boto:Token: None 2023-07-29T00:45:01.838 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:45:01 GMT /hiiqos-61/ 2023-07-29T00:45:01.838 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gT8Me8QNKQIxMkTUOJIA3HgQsvs= 2023-07-29T00:45:01.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:45:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gT8Me8QNKQIxMkTUOJIA3HgQsvs=', 'Content-Length': '0'} 2023-07-29T00:45:01.841 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d3aec4711f0e0548-0064c4610d-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:01.842 DEBUG:boto:path=/hiiqos-61/obj95 2023-07-29T00:45:01.842 DEBUG:boto:auth_path=/hiiqos-61/obj95 2023-07-29T00:45:01.842 DEBUG:boto:Method: PUT 2023-07-29T00:45:01.843 DEBUG:boto:Path: /hiiqos-61/obj95 2023-07-29T00:45:01.843 DEBUG:boto:Data: 2023-07-29T00:45:01.843 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:45:01.844 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:01.844 DEBUG:boto:Port: 8000 2023-07-29T00:45:01.844 DEBUG:boto:Params: {} 2023-07-29T00:45:01.845 DEBUG:boto:Token: None 2023-07-29T00:45:01.845 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:45:01 GMT /hiiqos-61/obj95 2023-07-29T00:45:01.845 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Gvno7rwPgtOcp/A391oTjI6Evi8= 2023-07-29T00:45:01.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:45:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Gvno7rwPgtOcp/A391oTjI6Evi8='} 2023-07-29T00:45:01.888 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000200a171cecbee7e5-0064c4610d-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:01.889 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-61', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:45:01.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-61', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:45:01.890 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-61 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:45:01.940 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:45:01.940 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:45:01.983 INFO:teuthology.orchestra.run.smithi101.stdout:tenant: 2023-07-29T00:45:01.983 INFO:teuthology.orchestra.run.smithi101.stdout:bucket name: hiiqos-61 2023-07-29T00:45:01.990 INFO:teuthology.orchestra.run.smithi101.stdout:total entries: 15 2023-07-29T00:45:02.034 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:45:02.035 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:45:02.036 DEBUG:boto:Method: HEAD 2023-07-29T00:45:02.037 DEBUG:boto:Path: /hiiqos-61/ 2023-07-29T00:45:02.037 DEBUG:boto:Data: 2023-07-29T00:45:02.038 DEBUG:boto:Headers: {} 2023-07-29T00:45:02.040 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:02.040 DEBUG:boto:Port: 8000 2023-07-29T00:45:02.040 DEBUG:boto:Params: {} 2023-07-29T00:45:02.041 DEBUG:boto:Token: None 2023-07-29T00:45:02.042 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:45:02 GMT /hiiqos-61/ 2023-07-29T00:45:02.042 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HziVamrvAAHMGALuksdKTAY03zs= 2023-07-29T00:45:02.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:45:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HziVamrvAAHMGALuksdKTAY03zs=', 'Content-Length': '0'} 2023-07-29T00:45:02.046 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003fca4f6cf720813c-0064c4610d-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:02.046 DEBUG:boto:path=/hiiqos-61/obj23 2023-07-29T00:45:02.047 DEBUG:boto:auth_path=/hiiqos-61/obj23 2023-07-29T00:45:02.047 DEBUG:boto:Method: PUT 2023-07-29T00:45:02.047 DEBUG:boto:Path: /hiiqos-61/obj23 2023-07-29T00:45:02.048 DEBUG:boto:Data: 2023-07-29T00:45:02.048 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:45:02.048 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:02.049 DEBUG:boto:Port: 8000 2023-07-29T00:45:02.049 DEBUG:boto:Params: {} 2023-07-29T00:45:02.050 DEBUG:boto:Token: None 2023-07-29T00:45:02.050 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:45:02 GMT /hiiqos-61/obj23 2023-07-29T00:45:02.050 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XYX+jKQ1FxEeHEbmxXOi+w48xqY= 2023-07-29T00:45:02.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:45:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XYX+jKQ1FxEeHEbmxXOi+w48xqY='} 2023-07-29T00:45:02.056 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000fb07941d71024d2-0064c4610d-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:02.057 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:45:02.057 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:45:02.057 DEBUG:boto:Method: HEAD 2023-07-29T00:45:02.058 DEBUG:boto:Path: /hiiqos-61/ 2023-07-29T00:45:02.058 DEBUG:boto:Data: 2023-07-29T00:45:02.058 DEBUG:boto:Headers: {} 2023-07-29T00:45:02.059 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:02.059 DEBUG:boto:Port: 8000 2023-07-29T00:45:02.059 DEBUG:boto:Params: {} 2023-07-29T00:45:02.060 DEBUG:boto:Token: None 2023-07-29T00:45:02.060 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:45:02 GMT /hiiqos-61/ 2023-07-29T00:45:02.061 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HziVamrvAAHMGALuksdKTAY03zs= 2023-07-29T00:45:02.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:45:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HziVamrvAAHMGALuksdKTAY03zs=', 'Content-Length': '0'} 2023-07-29T00:45:02.064 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a30514973903ed0d-0064c4610d-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:02.065 DEBUG:boto:path=/hiiqos-61/obj46 2023-07-29T00:45:02.065 DEBUG:boto:auth_path=/hiiqos-61/obj46 2023-07-29T00:45:02.065 DEBUG:boto:Method: PUT 2023-07-29T00:45:02.066 DEBUG:boto:Path: /hiiqos-61/obj46 2023-07-29T00:45:02.066 DEBUG:boto:Data: 2023-07-29T00:45:02.066 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:45:02.067 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:02.067 DEBUG:boto:Port: 8000 2023-07-29T00:45:02.067 DEBUG:boto:Params: {} 2023-07-29T00:45:02.068 DEBUG:boto:Token: None 2023-07-29T00:45:02.068 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:45:02 GMT /hiiqos-61/obj46 2023-07-29T00:45:02.069 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dJ54kpzgXr2Z9t74F25ewk+nT10= 2023-07-29T00:45:02.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:45:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dJ54kpzgXr2Z9t74F25ewk+nT10='} 2023-07-29T00:45:02.074 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f794fe9acf5eb9e5-0064c4610e-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:02.074 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:45:02.075 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:45:02.075 DEBUG:boto:Method: HEAD 2023-07-29T00:45:02.075 DEBUG:boto:Path: /hiiqos-61/ 2023-07-29T00:45:02.076 DEBUG:boto:Data: 2023-07-29T00:45:02.076 DEBUG:boto:Headers: {} 2023-07-29T00:45:02.076 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:02.077 DEBUG:boto:Port: 8000 2023-07-29T00:45:02.077 DEBUG:boto:Params: {} 2023-07-29T00:45:02.077 DEBUG:boto:Token: None 2023-07-29T00:45:02.078 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:45:02 GMT /hiiqos-61/ 2023-07-29T00:45:02.078 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HziVamrvAAHMGALuksdKTAY03zs= 2023-07-29T00:45:02.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:45:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HziVamrvAAHMGALuksdKTAY03zs=', 'Content-Length': '0'} 2023-07-29T00:45:02.081 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008da6acb3bc902339-0064c4610e-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:02.082 DEBUG:boto:path=/hiiqos-61/obj69 2023-07-29T00:45:02.082 DEBUG:boto:auth_path=/hiiqos-61/obj69 2023-07-29T00:45:02.082 DEBUG:boto:Method: PUT 2023-07-29T00:45:02.083 DEBUG:boto:Path: /hiiqos-61/obj69 2023-07-29T00:45:02.083 DEBUG:boto:Data: 2023-07-29T00:45:02.083 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:45:02.084 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:02.084 DEBUG:boto:Port: 8000 2023-07-29T00:45:02.085 DEBUG:boto:Params: {} 2023-07-29T00:45:02.085 DEBUG:boto:Token: None 2023-07-29T00:45:02.085 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:45:02 GMT /hiiqos-61/obj69 2023-07-29T00:45:02.086 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4zNJE4scR+5SXtzc/nNBmEBDcPU= 2023-07-29T00:45:02.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:45:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4zNJE4scR+5SXtzc/nNBmEBDcPU='} 2023-07-29T00:45:02.133 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009ad41c6c4bde2738-0064c4610e-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:02.134 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:45:02.134 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:45:02.135 DEBUG:boto:Method: HEAD 2023-07-29T00:45:02.136 DEBUG:boto:Path: /hiiqos-61/ 2023-07-29T00:45:02.137 DEBUG:boto:Data: 2023-07-29T00:45:02.137 DEBUG:boto:Headers: {} 2023-07-29T00:45:02.138 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:02.139 DEBUG:boto:Port: 8000 2023-07-29T00:45:02.140 DEBUG:boto:Params: {} 2023-07-29T00:45:02.141 DEBUG:boto:Token: None 2023-07-29T00:45:02.141 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:45:02 GMT /hiiqos-61/ 2023-07-29T00:45:02.142 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HziVamrvAAHMGALuksdKTAY03zs= 2023-07-29T00:45:02.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:45:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HziVamrvAAHMGALuksdKTAY03zs=', 'Content-Length': '0'} 2023-07-29T00:45:02.146 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000058406d10fa709b08-0064c4610e-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:02.147 DEBUG:boto:path=/hiiqos-61/obj92 2023-07-29T00:45:02.148 DEBUG:boto:auth_path=/hiiqos-61/obj92 2023-07-29T00:45:02.149 DEBUG:boto:Method: PUT 2023-07-29T00:45:02.150 DEBUG:boto:Path: /hiiqos-61/obj92 2023-07-29T00:45:02.150 DEBUG:boto:Data: 2023-07-29T00:45:02.151 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:45:02.152 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:02.152 DEBUG:boto:Port: 8000 2023-07-29T00:45:02.153 DEBUG:boto:Params: {} 2023-07-29T00:45:02.153 DEBUG:boto:Token: None 2023-07-29T00:45:02.154 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:45:02 GMT /hiiqos-61/obj92 2023-07-29T00:45:02.155 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FTh4ovpWRibZwgQwnDRNHdjT5qs= 2023-07-29T00:45:02.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:45:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FTh4ovpWRibZwgQwnDRNHdjT5qs='} 2023-07-29T00:45:02.160 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000483c195ba25bbfe-0064c4610e-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:02.160 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:45:02.161 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:45:02.161 DEBUG:boto:Method: HEAD 2023-07-29T00:45:02.162 DEBUG:boto:Path: /hiiqos-61/ 2023-07-29T00:45:02.162 DEBUG:boto:Data: 2023-07-29T00:45:02.162 DEBUG:boto:Headers: {} 2023-07-29T00:45:02.163 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:02.163 DEBUG:boto:Port: 8000 2023-07-29T00:45:02.163 DEBUG:boto:Params: {} 2023-07-29T00:45:02.164 DEBUG:boto:Token: None 2023-07-29T00:45:02.164 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:45:02 GMT /hiiqos-61/ 2023-07-29T00:45:02.164 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HziVamrvAAHMGALuksdKTAY03zs= 2023-07-29T00:45:02.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:45:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HziVamrvAAHMGALuksdKTAY03zs=', 'Content-Length': '0'} 2023-07-29T00:45:02.167 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007bc13c2427a3d73c-0064c4610e-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:02.168 DEBUG:boto:path=/hiiqos-61/obj115 2023-07-29T00:45:02.168 DEBUG:boto:auth_path=/hiiqos-61/obj115 2023-07-29T00:45:02.169 DEBUG:boto:Method: PUT 2023-07-29T00:45:02.169 DEBUG:boto:Path: /hiiqos-61/obj115 2023-07-29T00:45:02.169 DEBUG:boto:Data: 2023-07-29T00:45:02.170 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:45:02.170 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:02.170 DEBUG:boto:Port: 8000 2023-07-29T00:45:02.171 DEBUG:boto:Params: {} 2023-07-29T00:45:02.171 DEBUG:boto:Token: None 2023-07-29T00:45:02.171 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:45:02 GMT /hiiqos-61/obj115 2023-07-29T00:45:02.172 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Gh35+L9ojdhHcDXLkTTp1t7K12E= 2023-07-29T00:45:02.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:45:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Gh35+L9ojdhHcDXLkTTp1t7K12E='} 2023-07-29T00:45:02.177 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007c35a53e5b128737-0064c4610e-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:02.178 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-61', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:45:02.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-61', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:45:02.180 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-61 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:45:02.230 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:45:02.230 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:45:02.275 INFO:teuthology.orchestra.run.smithi101.stdout:tenant: 2023-07-29T00:45:02.275 INFO:teuthology.orchestra.run.smithi101.stdout:bucket name: hiiqos-61 2023-07-29T00:45:02.282 INFO:teuthology.orchestra.run.smithi101.stdout:total entries: 20 2023-07-29T00:45:02.331 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:45:02.332 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:45:02.332 DEBUG:boto:Method: HEAD 2023-07-29T00:45:02.333 DEBUG:boto:Path: /hiiqos-61/ 2023-07-29T00:45:02.333 DEBUG:boto:Data: 2023-07-29T00:45:02.333 DEBUG:boto:Headers: {} 2023-07-29T00:45:02.334 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:02.334 DEBUG:boto:Port: 8000 2023-07-29T00:45:02.334 DEBUG:boto:Params: {} 2023-07-29T00:45:02.335 DEBUG:boto:Token: None 2023-07-29T00:45:02.335 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:45:02 GMT /hiiqos-61/ 2023-07-29T00:45:02.335 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HziVamrvAAHMGALuksdKTAY03zs= 2023-07-29T00:45:02.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:45:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HziVamrvAAHMGALuksdKTAY03zs=', 'Content-Length': '0'} 2023-07-29T00:45:02.339 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d696dd13bac09062-0064c4610e-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:02.339 DEBUG:boto:path=/hiiqos-61/obj29 2023-07-29T00:45:02.340 DEBUG:boto:auth_path=/hiiqos-61/obj29 2023-07-29T00:45:02.340 DEBUG:boto:Method: PUT 2023-07-29T00:45:02.341 DEBUG:boto:Path: /hiiqos-61/obj29 2023-07-29T00:45:02.341 DEBUG:boto:Data: 2023-07-29T00:45:02.341 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:45:02.342 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:02.342 DEBUG:boto:Port: 8000 2023-07-29T00:45:02.342 DEBUG:boto:Params: {} 2023-07-29T00:45:02.343 DEBUG:boto:Token: None 2023-07-29T00:45:02.343 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:45:02 GMT /hiiqos-61/obj29 2023-07-29T00:45:02.343 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0DFuHEC95YcHzjzpeHoL5x+wQ4Q= 2023-07-29T00:45:02.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:45:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0DFuHEC95YcHzjzpeHoL5x+wQ4Q='} 2023-07-29T00:45:02.349 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000011808cb6572a63db-0064c4610e-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:02.349 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:45:02.350 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:45:02.350 DEBUG:boto:Method: HEAD 2023-07-29T00:45:02.350 DEBUG:boto:Path: /hiiqos-61/ 2023-07-29T00:45:02.351 DEBUG:boto:Data: 2023-07-29T00:45:02.351 DEBUG:boto:Headers: {} 2023-07-29T00:45:02.351 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:02.352 DEBUG:boto:Port: 8000 2023-07-29T00:45:02.352 DEBUG:boto:Params: {} 2023-07-29T00:45:02.352 DEBUG:boto:Token: None 2023-07-29T00:45:02.353 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:45:02 GMT /hiiqos-61/ 2023-07-29T00:45:02.353 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HziVamrvAAHMGALuksdKTAY03zs= 2023-07-29T00:45:02.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:45:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HziVamrvAAHMGALuksdKTAY03zs=', 'Content-Length': '0'} 2023-07-29T00:45:02.356 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d1c88da8447993e1-0064c4610e-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:02.357 DEBUG:boto:path=/hiiqos-61/obj58 2023-07-29T00:45:02.357 DEBUG:boto:auth_path=/hiiqos-61/obj58 2023-07-29T00:45:02.358 DEBUG:boto:Method: PUT 2023-07-29T00:45:02.358 DEBUG:boto:Path: /hiiqos-61/obj58 2023-07-29T00:45:02.358 DEBUG:boto:Data: 2023-07-29T00:45:02.359 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:45:02.360 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:02.361 DEBUG:boto:Port: 8000 2023-07-29T00:45:02.361 DEBUG:boto:Params: {} 2023-07-29T00:45:02.362 DEBUG:boto:Token: None 2023-07-29T00:45:02.363 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:45:02 GMT /hiiqos-61/obj58 2023-07-29T00:45:02.364 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:X4ctYQZnffBjJ/c6hitcLvR3yHo= 2023-07-29T00:45:02.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:45:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:X4ctYQZnffBjJ/c6hitcLvR3yHo='} 2023-07-29T00:45:02.408 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f4e44f291e84aa2e-0064c4610e-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:02.409 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:45:02.409 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:45:02.410 DEBUG:boto:Method: HEAD 2023-07-29T00:45:02.410 DEBUG:boto:Path: /hiiqos-61/ 2023-07-29T00:45:02.411 DEBUG:boto:Data: 2023-07-29T00:45:02.412 DEBUG:boto:Headers: {} 2023-07-29T00:45:02.413 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:02.413 DEBUG:boto:Port: 8000 2023-07-29T00:45:02.414 DEBUG:boto:Params: {} 2023-07-29T00:45:02.415 DEBUG:boto:Token: None 2023-07-29T00:45:02.416 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:45:02 GMT /hiiqos-61/ 2023-07-29T00:45:02.417 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HziVamrvAAHMGALuksdKTAY03zs= 2023-07-29T00:45:02.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:45:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HziVamrvAAHMGALuksdKTAY03zs=', 'Content-Length': '0'} 2023-07-29T00:45:02.421 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000078a40e30d2b0f127-0064c4610e-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:02.422 DEBUG:boto:path=/hiiqos-61/obj87 2023-07-29T00:45:02.423 DEBUG:boto:auth_path=/hiiqos-61/obj87 2023-07-29T00:45:02.424 DEBUG:boto:Method: PUT 2023-07-29T00:45:02.424 DEBUG:boto:Path: /hiiqos-61/obj87 2023-07-29T00:45:02.425 DEBUG:boto:Data: 2023-07-29T00:45:02.426 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:45:02.427 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:02.427 DEBUG:boto:Port: 8000 2023-07-29T00:45:02.428 DEBUG:boto:Params: {} 2023-07-29T00:45:02.429 DEBUG:boto:Token: None 2023-07-29T00:45:02.430 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:45:02 GMT /hiiqos-61/obj87 2023-07-29T00:45:02.431 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GPc2oT3qNE+DurpVkmmGH8Qzpk0= 2023-07-29T00:45:02.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:45:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GPc2oT3qNE+DurpVkmmGH8Qzpk0='} 2023-07-29T00:45:02.436 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000947fecb4631d2d8a-0064c4610e-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:02.437 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:45:02.438 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:45:02.438 DEBUG:boto:Method: HEAD 2023-07-29T00:45:02.439 DEBUG:boto:Path: /hiiqos-61/ 2023-07-29T00:45:02.439 DEBUG:boto:Data: 2023-07-29T00:45:02.440 DEBUG:boto:Headers: {} 2023-07-29T00:45:02.441 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:02.441 DEBUG:boto:Port: 8000 2023-07-29T00:45:02.442 DEBUG:boto:Params: {} 2023-07-29T00:45:02.443 DEBUG:boto:Token: None 2023-07-29T00:45:02.444 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:45:02 GMT /hiiqos-61/ 2023-07-29T00:45:02.444 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HziVamrvAAHMGALuksdKTAY03zs= 2023-07-29T00:45:02.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:45:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HziVamrvAAHMGALuksdKTAY03zs=', 'Content-Length': '0'} 2023-07-29T00:45:02.449 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e48654b34c2fd148-0064c4610e-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:02.450 DEBUG:boto:path=/hiiqos-61/obj116 2023-07-29T00:45:02.450 DEBUG:boto:auth_path=/hiiqos-61/obj116 2023-07-29T00:45:02.451 DEBUG:boto:Method: PUT 2023-07-29T00:45:02.451 DEBUG:boto:Path: /hiiqos-61/obj116 2023-07-29T00:45:02.452 DEBUG:boto:Data: 2023-07-29T00:45:02.453 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:45:02.454 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:02.454 DEBUG:boto:Port: 8000 2023-07-29T00:45:02.455 DEBUG:boto:Params: {} 2023-07-29T00:45:02.456 DEBUG:boto:Token: None 2023-07-29T00:45:02.457 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:45:02 GMT /hiiqos-61/obj116 2023-07-29T00:45:02.458 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZHdLhay7lF1tgUMQQhYtojn2CQI= 2023-07-29T00:45:02.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:45:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZHdLhay7lF1tgUMQQhYtojn2CQI='} 2023-07-29T00:45:02.504 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000048a2406985ddec18-0064c4610e-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:02.505 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:45:02.505 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:45:02.506 DEBUG:boto:Method: HEAD 2023-07-29T00:45:02.506 DEBUG:boto:Path: /hiiqos-61/ 2023-07-29T00:45:02.506 DEBUG:boto:Data: 2023-07-29T00:45:02.507 DEBUG:boto:Headers: {} 2023-07-29T00:45:02.507 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:02.508 DEBUG:boto:Port: 8000 2023-07-29T00:45:02.508 DEBUG:boto:Params: {} 2023-07-29T00:45:02.508 DEBUG:boto:Token: None 2023-07-29T00:45:02.509 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:45:02 GMT /hiiqos-61/ 2023-07-29T00:45:02.509 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HziVamrvAAHMGALuksdKTAY03zs= 2023-07-29T00:45:02.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:45:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HziVamrvAAHMGALuksdKTAY03zs=', 'Content-Length': '0'} 2023-07-29T00:45:02.513 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000407532df1aea1a2c-0064c4610e-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:02.513 DEBUG:boto:path=/hiiqos-61/obj145 2023-07-29T00:45:02.514 DEBUG:boto:auth_path=/hiiqos-61/obj145 2023-07-29T00:45:02.514 DEBUG:boto:Method: PUT 2023-07-29T00:45:02.514 DEBUG:boto:Path: /hiiqos-61/obj145 2023-07-29T00:45:02.515 DEBUG:boto:Data: 2023-07-29T00:45:02.515 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:45:02.515 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:02.516 DEBUG:boto:Port: 8000 2023-07-29T00:45:02.516 DEBUG:boto:Params: {} 2023-07-29T00:45:02.517 DEBUG:boto:Token: None 2023-07-29T00:45:02.517 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:45:02 GMT /hiiqos-61/obj145 2023-07-29T00:45:02.517 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pH150HIDFavqqElTHWGqiZA9Keg= 2023-07-29T00:45:02.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:45:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pH150HIDFavqqElTHWGqiZA9Keg='} 2023-07-29T00:45:02.523 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000ee18e7e391dfb55-0064c4610e-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:02.524 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-61', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:45:02.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-61', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:45:02.524 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-61 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:45:02.573 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:45:02.574 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:45:02.619 INFO:teuthology.orchestra.run.smithi101.stdout:tenant: 2023-07-29T00:45:02.620 INFO:teuthology.orchestra.run.smithi101.stdout:bucket name: hiiqos-61 2023-07-29T00:45:02.629 INFO:teuthology.orchestra.run.smithi101.stdout:total entries: 25 2023-07-29T00:45:02.685 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:45:02.686 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:45:02.686 DEBUG:boto:Method: HEAD 2023-07-29T00:45:02.686 DEBUG:boto:Path: /hiiqos-61/ 2023-07-29T00:45:02.687 DEBUG:boto:Data: 2023-07-29T00:45:02.687 DEBUG:boto:Headers: {} 2023-07-29T00:45:02.688 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:02.688 DEBUG:boto:Port: 8000 2023-07-29T00:45:02.689 DEBUG:boto:Params: {} 2023-07-29T00:45:02.690 DEBUG:boto:Token: None 2023-07-29T00:45:02.691 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:45:02 GMT /hiiqos-61/ 2023-07-29T00:45:02.692 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HziVamrvAAHMGALuksdKTAY03zs= 2023-07-29T00:45:02.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:45:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HziVamrvAAHMGALuksdKTAY03zs=', 'Content-Length': '0'} 2023-07-29T00:45:02.696 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000059e7be43935c351-0064c4610e-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:02.697 DEBUG:boto:path=/hiiqos-61/obj31 2023-07-29T00:45:02.698 DEBUG:boto:auth_path=/hiiqos-61/obj31 2023-07-29T00:45:02.699 DEBUG:boto:Method: PUT 2023-07-29T00:45:02.700 DEBUG:boto:Path: /hiiqos-61/obj31 2023-07-29T00:45:02.701 DEBUG:boto:Data: 2023-07-29T00:45:02.701 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:45:02.702 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:02.703 DEBUG:boto:Port: 8000 2023-07-29T00:45:02.703 DEBUG:boto:Params: {} 2023-07-29T00:45:02.704 DEBUG:boto:Token: None 2023-07-29T00:45:02.705 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:45:02 GMT /hiiqos-61/obj31 2023-07-29T00:45:02.706 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cokd/I0GOcLj/Tv0NF61M9iL4Ko= 2023-07-29T00:45:02.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:45:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cokd/I0GOcLj/Tv0NF61M9iL4Ko='} 2023-07-29T00:45:02.756 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000015f4900cfe68a2ed-0064c4610e-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:02.757 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:45:02.757 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:45:02.758 DEBUG:boto:Method: HEAD 2023-07-29T00:45:02.758 DEBUG:boto:Path: /hiiqos-61/ 2023-07-29T00:45:02.758 DEBUG:boto:Data: 2023-07-29T00:45:02.759 DEBUG:boto:Headers: {} 2023-07-29T00:45:02.759 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:02.759 DEBUG:boto:Port: 8000 2023-07-29T00:45:02.760 DEBUG:boto:Params: {} 2023-07-29T00:45:02.760 DEBUG:boto:Token: None 2023-07-29T00:45:02.761 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:45:02 GMT /hiiqos-61/ 2023-07-29T00:45:02.761 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HziVamrvAAHMGALuksdKTAY03zs= 2023-07-29T00:45:02.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:45:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HziVamrvAAHMGALuksdKTAY03zs=', 'Content-Length': '0'} 2023-07-29T00:45:02.764 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000458341214246046f-0064c4610e-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:02.765 DEBUG:boto:path=/hiiqos-61/obj62 2023-07-29T00:45:02.765 DEBUG:boto:auth_path=/hiiqos-61/obj62 2023-07-29T00:45:02.765 DEBUG:boto:Method: PUT 2023-07-29T00:45:02.766 DEBUG:boto:Path: /hiiqos-61/obj62 2023-07-29T00:45:02.766 DEBUG:boto:Data: 2023-07-29T00:45:02.766 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:45:02.767 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:02.767 DEBUG:boto:Port: 8000 2023-07-29T00:45:02.767 DEBUG:boto:Params: {} 2023-07-29T00:45:02.768 DEBUG:boto:Token: None 2023-07-29T00:45:02.768 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:45:02 GMT /hiiqos-61/obj62 2023-07-29T00:45:02.769 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:W3JCS9A9Fy9Q+EQAiLrWaoZl7WE= 2023-07-29T00:45:02.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:45:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:W3JCS9A9Fy9Q+EQAiLrWaoZl7WE='} 2023-07-29T00:45:02.774 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c6793100631e8611-0064c4610e-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:02.775 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:45:02.775 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:45:02.775 DEBUG:boto:Method: HEAD 2023-07-29T00:45:02.776 DEBUG:boto:Path: /hiiqos-61/ 2023-07-29T00:45:02.776 DEBUG:boto:Data: 2023-07-29T00:45:02.776 DEBUG:boto:Headers: {} 2023-07-29T00:45:02.777 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:02.777 DEBUG:boto:Port: 8000 2023-07-29T00:45:02.777 DEBUG:boto:Params: {} 2023-07-29T00:45:02.778 DEBUG:boto:Token: None 2023-07-29T00:45:02.778 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:45:02 GMT /hiiqos-61/ 2023-07-29T00:45:02.778 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HziVamrvAAHMGALuksdKTAY03zs= 2023-07-29T00:45:02.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:45:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HziVamrvAAHMGALuksdKTAY03zs=', 'Content-Length': '0'} 2023-07-29T00:45:02.782 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005ff4006218bbecfe-0064c4610e-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:02.782 DEBUG:boto:path=/hiiqos-61/obj93 2023-07-29T00:45:02.783 DEBUG:boto:auth_path=/hiiqos-61/obj93 2023-07-29T00:45:02.783 DEBUG:boto:Method: PUT 2023-07-29T00:45:02.784 DEBUG:boto:Path: /hiiqos-61/obj93 2023-07-29T00:45:02.784 DEBUG:boto:Data: 2023-07-29T00:45:02.785 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:45:02.785 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:02.785 DEBUG:boto:Port: 8000 2023-07-29T00:45:02.786 DEBUG:boto:Params: {} 2023-07-29T00:45:02.786 DEBUG:boto:Token: None 2023-07-29T00:45:02.786 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:45:02 GMT /hiiqos-61/obj93 2023-07-29T00:45:02.787 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hOLlzrH3lT0beu8LTmyNy18Gm/8= 2023-07-29T00:45:02.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:45:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hOLlzrH3lT0beu8LTmyNy18Gm/8='} 2023-07-29T00:45:02.836 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000377f99a6568359da-0064c4610e-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:02.837 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:45:02.837 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:45:02.838 DEBUG:boto:Method: HEAD 2023-07-29T00:45:02.838 DEBUG:boto:Path: /hiiqos-61/ 2023-07-29T00:45:02.838 DEBUG:boto:Data: 2023-07-29T00:45:02.839 DEBUG:boto:Headers: {} 2023-07-29T00:45:02.839 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:02.839 DEBUG:boto:Port: 8000 2023-07-29T00:45:02.840 DEBUG:boto:Params: {} 2023-07-29T00:45:02.840 DEBUG:boto:Token: None 2023-07-29T00:45:02.840 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:45:02 GMT /hiiqos-61/ 2023-07-29T00:45:02.841 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HziVamrvAAHMGALuksdKTAY03zs= 2023-07-29T00:45:02.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:45:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HziVamrvAAHMGALuksdKTAY03zs=', 'Content-Length': '0'} 2023-07-29T00:45:02.844 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f898c4ae88d1f712-0064c4610e-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:02.845 DEBUG:boto:path=/hiiqos-61/obj124 2023-07-29T00:45:02.845 DEBUG:boto:auth_path=/hiiqos-61/obj124 2023-07-29T00:45:02.845 DEBUG:boto:Method: PUT 2023-07-29T00:45:02.846 DEBUG:boto:Path: /hiiqos-61/obj124 2023-07-29T00:45:02.846 DEBUG:boto:Data: 2023-07-29T00:45:02.846 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:45:02.847 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:02.847 DEBUG:boto:Port: 8000 2023-07-29T00:45:02.847 DEBUG:boto:Params: {} 2023-07-29T00:45:02.848 DEBUG:boto:Token: None 2023-07-29T00:45:02.848 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:45:02 GMT /hiiqos-61/obj124 2023-07-29T00:45:02.849 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AfOV0HwqaNo5FLF0c06NYHrgnic= 2023-07-29T00:45:02.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:45:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AfOV0HwqaNo5FLF0c06NYHrgnic='} 2023-07-29T00:45:02.854 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001bce40eb94d7e664-0064c4610e-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:02.854 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:45:02.855 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:45:02.855 DEBUG:boto:Method: HEAD 2023-07-29T00:45:02.856 DEBUG:boto:Path: /hiiqos-61/ 2023-07-29T00:45:02.856 DEBUG:boto:Data: 2023-07-29T00:45:02.856 DEBUG:boto:Headers: {} 2023-07-29T00:45:02.857 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:02.857 DEBUG:boto:Port: 8000 2023-07-29T00:45:02.857 DEBUG:boto:Params: {} 2023-07-29T00:45:02.858 DEBUG:boto:Token: None 2023-07-29T00:45:02.858 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:45:02 GMT /hiiqos-61/ 2023-07-29T00:45:02.858 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:HziVamrvAAHMGALuksdKTAY03zs= 2023-07-29T00:45:02.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:45:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HziVamrvAAHMGALuksdKTAY03zs=', 'Content-Length': '0'} 2023-07-29T00:45:02.861 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b35279d28a30b379-0064c4610e-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:02.862 DEBUG:boto:path=/hiiqos-61/obj155 2023-07-29T00:45:02.862 DEBUG:boto:auth_path=/hiiqos-61/obj155 2023-07-29T00:45:02.863 DEBUG:boto:Method: PUT 2023-07-29T00:45:02.863 DEBUG:boto:Path: /hiiqos-61/obj155 2023-07-29T00:45:02.863 DEBUG:boto:Data: 2023-07-29T00:45:02.864 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:45:02.864 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:02.864 DEBUG:boto:Port: 8000 2023-07-29T00:45:02.865 DEBUG:boto:Params: {} 2023-07-29T00:45:02.865 DEBUG:boto:Token: None 2023-07-29T00:45:02.866 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:45:02 GMT /hiiqos-61/obj155 2023-07-29T00:45:02.866 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:d7oCmFGC4O7T+v80fCdgOWRzLqc= 2023-07-29T00:45:02.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:45:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:d7oCmFGC4O7T+v80fCdgOWRzLqc='} 2023-07-29T00:45:02.912 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000059703b7ac8e284e4-0064c4610e-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:02.913 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-61', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:45:02.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-61', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:45:02.914 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-61 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:45:02.963 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:45:02.963 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:45:03.009 INFO:teuthology.orchestra.run.smithi101.stdout:tenant: 2023-07-29T00:45:03.010 INFO:teuthology.orchestra.run.smithi101.stdout:bucket name: hiiqos-61 2023-07-29T00:45:03.020 INFO:teuthology.orchestra.run.smithi101.stdout:total entries: 30 2023-07-29T00:45:03.078 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:45:03.079 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:45:03.079 DEBUG:boto:Method: HEAD 2023-07-29T00:45:03.081 DEBUG:boto:Path: /hiiqos-61/ 2023-07-29T00:45:03.082 DEBUG:boto:Data: 2023-07-29T00:45:03.083 DEBUG:boto:Headers: {} 2023-07-29T00:45:03.084 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:03.084 DEBUG:boto:Port: 8000 2023-07-29T00:45:03.084 DEBUG:boto:Params: {} 2023-07-29T00:45:03.085 DEBUG:boto:Token: None 2023-07-29T00:45:03.085 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:45:03 GMT /hiiqos-61/ 2023-07-29T00:45:03.086 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zdOgQJYshKIV6jciaN+JjnE7g9A= 2023-07-29T00:45:03.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:45:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zdOgQJYshKIV6jciaN+JjnE7g9A=', 'Content-Length': '0'} 2023-07-29T00:45:03.089 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dca873da7bfdbf7c-0064c4610f-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:03.090 DEBUG:boto:path=/hiiqos-61/obj37 2023-07-29T00:45:03.090 DEBUG:boto:auth_path=/hiiqos-61/obj37 2023-07-29T00:45:03.091 DEBUG:boto:Method: PUT 2023-07-29T00:45:03.091 DEBUG:boto:Path: /hiiqos-61/obj37 2023-07-29T00:45:03.091 DEBUG:boto:Data: 2023-07-29T00:45:03.092 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:45:03.092 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:03.092 DEBUG:boto:Port: 8000 2023-07-29T00:45:03.093 DEBUG:boto:Params: {} 2023-07-29T00:45:03.093 DEBUG:boto:Token: None 2023-07-29T00:45:03.093 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:45:03 GMT /hiiqos-61/obj37 2023-07-29T00:45:03.094 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:siISgN5qpprBjh1Yb8bKVaQwJpg= 2023-07-29T00:45:03.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:45:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:siISgN5qpprBjh1Yb8bKVaQwJpg='} 2023-07-29T00:45:03.099 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005bf5d0775ebccdf7-0064c4610f-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:03.100 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:45:03.100 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:45:03.101 DEBUG:boto:Method: HEAD 2023-07-29T00:45:03.101 DEBUG:boto:Path: /hiiqos-61/ 2023-07-29T00:45:03.101 DEBUG:boto:Data: 2023-07-29T00:45:03.102 DEBUG:boto:Headers: {} 2023-07-29T00:45:03.102 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:03.102 DEBUG:boto:Port: 8000 2023-07-29T00:45:03.103 DEBUG:boto:Params: {} 2023-07-29T00:45:03.103 DEBUG:boto:Token: None 2023-07-29T00:45:03.103 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:45:03 GMT /hiiqos-61/ 2023-07-29T00:45:03.104 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zdOgQJYshKIV6jciaN+JjnE7g9A= 2023-07-29T00:45:03.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:45:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zdOgQJYshKIV6jciaN+JjnE7g9A=', 'Content-Length': '0'} 2023-07-29T00:45:03.107 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006409ab92dd993eb3-0064c4610f-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:03.108 DEBUG:boto:path=/hiiqos-61/obj74 2023-07-29T00:45:03.109 DEBUG:boto:auth_path=/hiiqos-61/obj74 2023-07-29T00:45:03.109 DEBUG:boto:Method: PUT 2023-07-29T00:45:03.109 DEBUG:boto:Path: /hiiqos-61/obj74 2023-07-29T00:45:03.110 DEBUG:boto:Data: 2023-07-29T00:45:03.110 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:45:03.110 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:03.111 DEBUG:boto:Port: 8000 2023-07-29T00:45:03.111 DEBUG:boto:Params: {} 2023-07-29T00:45:03.111 DEBUG:boto:Token: None 2023-07-29T00:45:03.112 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:45:03 GMT /hiiqos-61/obj74 2023-07-29T00:45:03.112 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XiDkSSxtjWSrRrJiiGW1XQ5WnYE= 2023-07-29T00:45:03.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:45:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XiDkSSxtjWSrRrJiiGW1XQ5WnYE='} 2023-07-29T00:45:03.118 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cfa24d6692893898-0064c4610f-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:03.118 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:45:03.119 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:45:03.119 DEBUG:boto:Method: HEAD 2023-07-29T00:45:03.120 DEBUG:boto:Path: /hiiqos-61/ 2023-07-29T00:45:03.120 DEBUG:boto:Data: 2023-07-29T00:45:03.120 DEBUG:boto:Headers: {} 2023-07-29T00:45:03.121 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:03.121 DEBUG:boto:Port: 8000 2023-07-29T00:45:03.121 DEBUG:boto:Params: {} 2023-07-29T00:45:03.122 DEBUG:boto:Token: None 2023-07-29T00:45:03.122 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:45:03 GMT /hiiqos-61/ 2023-07-29T00:45:03.122 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zdOgQJYshKIV6jciaN+JjnE7g9A= 2023-07-29T00:45:03.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:45:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zdOgQJYshKIV6jciaN+JjnE7g9A=', 'Content-Length': '0'} 2023-07-29T00:45:03.126 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e57c03414b86f150-0064c4610f-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:03.127 DEBUG:boto:path=/hiiqos-61/obj111 2023-07-29T00:45:03.127 DEBUG:boto:auth_path=/hiiqos-61/obj111 2023-07-29T00:45:03.128 DEBUG:boto:Method: PUT 2023-07-29T00:45:03.128 DEBUG:boto:Path: /hiiqos-61/obj111 2023-07-29T00:45:03.128 DEBUG:boto:Data: 2023-07-29T00:45:03.129 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:45:03.129 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:03.130 DEBUG:boto:Port: 8000 2023-07-29T00:45:03.130 DEBUG:boto:Params: {} 2023-07-29T00:45:03.130 DEBUG:boto:Token: None 2023-07-29T00:45:03.131 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:45:03 GMT /hiiqos-61/obj111 2023-07-29T00:45:03.131 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hABDEG+HUIK+8yVE0sRazdG6uyA= 2023-07-29T00:45:03.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:45:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hABDEG+HUIK+8yVE0sRazdG6uyA='} 2023-07-29T00:45:03.176 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009fb6889592aaacba-0064c4610f-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:03.177 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:45:03.177 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:45:03.178 DEBUG:boto:Method: HEAD 2023-07-29T00:45:03.178 DEBUG:boto:Path: /hiiqos-61/ 2023-07-29T00:45:03.178 DEBUG:boto:Data: 2023-07-29T00:45:03.179 DEBUG:boto:Headers: {} 2023-07-29T00:45:03.179 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:03.179 DEBUG:boto:Port: 8000 2023-07-29T00:45:03.180 DEBUG:boto:Params: {} 2023-07-29T00:45:03.180 DEBUG:boto:Token: None 2023-07-29T00:45:03.180 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:45:03 GMT /hiiqos-61/ 2023-07-29T00:45:03.181 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zdOgQJYshKIV6jciaN+JjnE7g9A= 2023-07-29T00:45:03.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:45:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zdOgQJYshKIV6jciaN+JjnE7g9A=', 'Content-Length': '0'} 2023-07-29T00:45:03.185 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ba7584248899ac7f-0064c4610f-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:03.185 DEBUG:boto:path=/hiiqos-61/obj148 2023-07-29T00:45:03.186 DEBUG:boto:auth_path=/hiiqos-61/obj148 2023-07-29T00:45:03.187 DEBUG:boto:Method: PUT 2023-07-29T00:45:03.187 DEBUG:boto:Path: /hiiqos-61/obj148 2023-07-29T00:45:03.188 DEBUG:boto:Data: 2023-07-29T00:45:03.189 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:45:03.191 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:03.192 DEBUG:boto:Port: 8000 2023-07-29T00:45:03.192 DEBUG:boto:Params: {} 2023-07-29T00:45:03.193 DEBUG:boto:Token: None 2023-07-29T00:45:03.193 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:45:03 GMT /hiiqos-61/obj148 2023-07-29T00:45:03.194 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EG9PfhfTXp36AoiCz2Ibqdwg3Ig= 2023-07-29T00:45:03.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:45:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EG9PfhfTXp36AoiCz2Ibqdwg3Ig='} 2023-07-29T00:45:03.199 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000098b2f93839465c01-0064c4610f-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:03.200 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:45:03.200 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:45:03.200 DEBUG:boto:Method: HEAD 2023-07-29T00:45:03.201 DEBUG:boto:Path: /hiiqos-61/ 2023-07-29T00:45:03.201 DEBUG:boto:Data: 2023-07-29T00:45:03.201 DEBUG:boto:Headers: {} 2023-07-29T00:45:03.202 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:03.202 DEBUG:boto:Port: 8000 2023-07-29T00:45:03.202 DEBUG:boto:Params: {} 2023-07-29T00:45:03.203 DEBUG:boto:Token: None 2023-07-29T00:45:03.203 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:45:03 GMT /hiiqos-61/ 2023-07-29T00:45:03.204 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zdOgQJYshKIV6jciaN+JjnE7g9A= 2023-07-29T00:45:03.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:45:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zdOgQJYshKIV6jciaN+JjnE7g9A=', 'Content-Length': '0'} 2023-07-29T00:45:03.207 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000059095937855c9277-0064c4610f-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:03.208 DEBUG:boto:path=/hiiqos-61/obj185 2023-07-29T00:45:03.208 DEBUG:boto:auth_path=/hiiqos-61/obj185 2023-07-29T00:45:03.209 DEBUG:boto:Method: PUT 2023-07-29T00:45:03.209 DEBUG:boto:Path: /hiiqos-61/obj185 2023-07-29T00:45:03.209 DEBUG:boto:Data: 2023-07-29T00:45:03.210 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:45:03.210 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:45:03.210 DEBUG:boto:Port: 8000 2023-07-29T00:45:03.211 DEBUG:boto:Params: {} 2023-07-29T00:45:03.211 DEBUG:boto:Token: None 2023-07-29T00:45:03.211 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:45:03 GMT /hiiqos-61/obj185 2023-07-29T00:45:03.212 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kMvNS/KW8GwsI9h6rYsSPWfL2No= 2023-07-29T00:45:03.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:45:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kMvNS/KW8GwsI9h6rYsSPWfL2No='} 2023-07-29T00:45:03.256 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005fbe66c41ba1ab1b-0064c4610f-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:45:03.257 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:45:03.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:45:03.258 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:45:03.306 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:45:03.306 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:45:03.336 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:45:03.267+0000 7f8583214880 5 RGW-SYNC:data: waiting on data sync status lease 2023-07-29T00:45:03.337 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:45:03.268+0000 7f8583214880 5 RGW-SYNC:data: acquired data sync status lease 2023-07-29T00:45:03.357 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-07-29T00:45:03.358 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-07-29T00:45:03.360 INFO:tasks.rgw.c1.client.1:Started 2023-07-29T00:45:03.361 DEBUG:teuthology.orchestra.run.smithi101:> curl http://smithi101.front.sepia.ceph.com:8001 2023-07-29T00:45:03.378 INFO:teuthology.orchestra.run.smithi101.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-07-29T00:45:03.378 INFO:teuthology.orchestra.run.smithi101.stderr: Dload Upload Total Spent Left Speed 2023-07-29T00:45:03.379 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-07-29T00:45:03.380 INFO:teuthology.orchestra.run.smithi101.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-07-29T00:45:03.380 INFO:teuthology.orchestra.run.smithi101.stderr:curl: (7) Failed to connect to smithi101.front.sepia.ceph.com port 8001: Connection refused 2023-07-29T00:45:04.381 DEBUG:teuthology.orchestra.run.smithi101:> curl http://smithi101.front.sepia.ceph.com:8001 2023-07-29T00:45:04.399 INFO:teuthology.orchestra.run.smithi101.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-07-29T00:45:04.400 INFO:teuthology.orchestra.run.smithi101.stderr: Dload Upload Total Spent Left Speed 2023-07-29T00:45:04.401 INFO:teuthology.orchestra.run.smithi101.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-07-29T00:45:04.402 INFO:teuthology.orchestra.run.smithi101.stdout:anonymous 2023-07-29T00:45:04.402 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-61 2023-07-29T00:45:04.402 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:45:04.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:45:04.403 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-61 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:45:04.494 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:45:04.494 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:45:04.530 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:45:04.461+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:45:05.157 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-07-29T00:45:09.531 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:45:09.462+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:45:14.531 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:45:14.462+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:45:19.532 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:45:19.463+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:45:24.532 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:45:24.464+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:45:29.533 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:45:29.464+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:45:34.533 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:45:34.464+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:45:39.534 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:45:39.465+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:45:44.535 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:45:44.466+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:45:49.535 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:45:49.466+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:45:54.536 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:45:54.467+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:45:59.537 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:45:59.468+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:46:04.537 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:46:04.468+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:46:09.538 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:46:09.468+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:46:14.539 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:46:14.469+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:46:19.539 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:46:19.470+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:46:24.540 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:46:24.470+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:46:29.540 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:46:29.471+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:46:34.541 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:46:34.471+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:46:39.541 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:46:39.472+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:46:44.542 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:46:44.472+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:46:49.543 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:46:49.473+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:46:54.543 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:46:54.473+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:46:59.544 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:46:59.474+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:47:02.951 INFO:tasks.rgw.c2.client.0.smithi139.stdout:2023-07-29T00:47:02.881+0000 7f5f2671b640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T00:47:04.098 INFO:tasks.rgw.c1.client.0.smithi101.stdout:2023-07-29T00:47:04.027+0000 7f0081a5e640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T00:47:04.545 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:47:04.474+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:47:09.545 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:47:09.475+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:47:14.546 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:47:14.475+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:47:19.546 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:47:19.476+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:47:24.547 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:47:24.477+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:47:29.548 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:47:29.478+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:47:34.549 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:47:34.478+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:47:39.550 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:47:39.479+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:47:44.550 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:47:44.479+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:47:49.551 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:47:49.480+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:47:54.552 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:47:54.480+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:47:59.552 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:47:59.481+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:48:04.553 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:48:04.482+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:48:09.554 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:48:09.483+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:48:14.554 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:48:14.483+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:48:18.049 INFO:tasks.rgw.c2.client.1.smithi139.stdout:2023-07-29T00:48:17.977+0000 7f5731a6a640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T00:48:19.555 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:48:19.484+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:48:24.556 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:48:24.484+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:48:29.556 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:48:29.485+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:48:34.557 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:48:34.486+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:48:39.558 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:48:39.486+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:48:44.559 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:48:44.487+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:48:49.559 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:48:49.488+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:48:54.560 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:48:54.488+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:48:59.560 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:48:59.489+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:49:04.561 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:49:04.489+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:49:09.562 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:49:09.490+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:49:14.562 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:49:14.491+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:49:19.563 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:49:19.491+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:49:24.564 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:49:24.492+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:49:29.564 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:49:29.493+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:49:34.565 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:49:34.493+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:49:39.566 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:49:39.494+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:49:44.567 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:49:44.494+0000 7f8afa414880 1 waiting to reach incremental sync.. 2023-07-29T00:49:49.570 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:49:49.498+0000 7f8afa414880 1 bucket sync caught up with source: 2023-07-29T00:49:49.571 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , 00000000002.99.6, , , , , , 00000000002.96.6, , , , , 00000000002.64.6, , , , , , , , , , , , , , , 00000000001.54.6, , , , 00000000001.87.6] 2023-07-29T00:49:49.571 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , 00000000002.99.6, , , , , , 00000000002.96.6, , , , , 00000000002.64.6, , , , , , , , , , , , , , , 00000000001.54.6, , , , 00000000001.87.6] 2023-07-29T00:49:49.571 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:49:49.498+0000 7f8afa414880 0 bucket checkpoint complete 2023-07-29T00:49:49.588 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-61 2023-07-29T00:49:49.589 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:49:49.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:49:49.590 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-61 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:49:49.648 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:49:49.648 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:49:49.685 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:49:49.613+0000 7ff259213880 1 bucket sync caught up with source: 2023-07-29T00:49:49.685 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, 00000000003.972.6, 00000000002.974.6, 00000000002.584.6, 00000000005.803.6, 00000000004.176.6, 00000000001.1054.6, 00000000001.904.6, 00000000004.1487.6, 00000000007.482.6, 00000000006.1485.6] 2023-07-29T00:49:49.686 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, 00000000003.972.6, 00000000002.974.6, 00000000002.584.6, 00000000005.803.6, 00000000004.176.6, 00000000001.1054.6, 00000000001.904.6, 00000000004.1487.6, 00000000007.482.6, 00000000006.1485.6] 2023-07-29T00:49:49.686 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:49:49.613+0000 7ff259213880 0 bucket checkpoint complete 2023-07-29T00:49:49.699 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-61 zones={a2, a1} 2023-07-29T00:49:49.700 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:49:49.700 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:49:49.701 DEBUG:boto:Method: HEAD 2023-07-29T00:49:49.701 DEBUG:boto:Path: /hiiqos-61/ 2023-07-29T00:49:49.701 DEBUG:boto:Data: 2023-07-29T00:49:49.702 DEBUG:boto:Headers: {} 2023-07-29T00:49:49.702 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:49:49.702 DEBUG:boto:Port: 8001 2023-07-29T00:49:49.703 DEBUG:boto:Params: {} 2023-07-29T00:49:49.703 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-29T00:49:49.704 DEBUG:boto:Token: None 2023-07-29T00:49:49.704 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:49:49 GMT /hiiqos-61/ 2023-07-29T00:49:49.705 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YJGp/JjiPf0ONCQMktcBdQuUE7s= 2023-07-29T00:49:49.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:49:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YJGp/JjiPf0ONCQMktcBdQuUE7s=', 'Content-Length': '0'} 2023-07-29T00:49:54.712 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000040b9d735ced9def3-0064c46232-2008-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:49:54 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:49:54.712 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:49:54.713 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:49:54.713 DEBUG:boto:Method: HEAD 2023-07-29T00:49:54.714 DEBUG:boto:Path: /hiiqos-61/ 2023-07-29T00:49:54.714 DEBUG:boto:Data: 2023-07-29T00:49:54.714 DEBUG:boto:Headers: {} 2023-07-29T00:49:54.715 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:49:54.715 DEBUG:boto:Port: 8000 2023-07-29T00:49:54.715 DEBUG:boto:Params: {} 2023-07-29T00:49:54.716 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-29T00:49:54.716 DEBUG:boto:Token: None 2023-07-29T00:49:54.717 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:49:54 GMT /hiiqos-61/ 2023-07-29T00:49:54.717 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:psQUZIiCRCf6mRgQYrmLT7934xg= 2023-07-29T00:49:54.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:49:54 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:psQUZIiCRCf6mRgQYrmLT7934xg=', 'Content-Length': '0'} 2023-07-29T00:49:59.724 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d58af069d333282b-0064c46237-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:49:59.725 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-29T00:49:59.725 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:49:59.726 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:49:59.726 DEBUG:boto:path=/hiiqos-61/?versions&max-keys=999 2023-07-29T00:49:59.726 DEBUG:boto:auth_path=/hiiqos-61/?versions&max-keys=999 2023-07-29T00:49:59.727 DEBUG:boto:Method: GET 2023-07-29T00:49:59.727 DEBUG:boto:Path: /hiiqos-61/?versions&max-keys=999 2023-07-29T00:49:59.727 DEBUG:boto:Data: 2023-07-29T00:49:59.728 DEBUG:boto:Headers: {} 2023-07-29T00:49:59.728 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:49:59.729 DEBUG:boto:Port: 8001 2023-07-29T00:49:59.729 DEBUG:boto:Params: {} 2023-07-29T00:49:59.729 DEBUG:boto:Token: None 2023-07-29T00:49:59.730 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:49:59 GMT /hiiqos-61/?versions 2023-07-29T00:49:59.730 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4mEdPNr0gfnrfglomjuVSo0BL7M= 2023-07-29T00:49:59.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:49:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4mEdPNr0gfnrfglomjuVSo0BL7M=', 'Content-Length': '0'} 2023-07-29T00:49:59.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2c6aaba56ce9492-0064c46237-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:49:59.734 DEBUG:boto:b'hiiqos-61999falseobj11nulltrue2023-07-29T00:45:01.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-07-29T00:45:03.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-07-29T00:45:02.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-07-29T00:45:02.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-07-29T00:45:02.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-07-29T00:45:02.452Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-07-29T00:45:03.128Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-07-29T00:45:02.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-07-29T00:45:01.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-07-29T00:45:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-07-29T00:45:01.674Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:45:01.061Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-07-29T00:45:01.985Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-07-29T00:45:02.278Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-07-29T00:45:02.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:45:01.083Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-07-29T00:45:01.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-07-29T00:45:03.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-07-29T00:45:01.691Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:45:01.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-07-29T00:45:02.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-07-29T00:45:01.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:45:01.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-07-29T00:45:01.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-07-29T00:45:02.299Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-07-29T00:45:02.703Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-07-29T00:45:01.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-07-29T00:45:02.020Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-07-29T00:45:03.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-07-29T00:45:01.763Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-07-29T00:45:01.475Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-07-29T00:45:02.365Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-07-29T00:45:02.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-07-29T00:45:02.721Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-07-29T00:45:01.780Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:49:59.736 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-07-29T00:49:59.737 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-07-29T00:49:59.737 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-07-29T00:49:59.737 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-07-29T00:49:59.738 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-07-29T00:49:59.738 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-07-29T00:49:59.738 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-07-29T00:49:59.739 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-07-29T00:49:59.739 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-07-29T00:49:59.739 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-07-29T00:49:59.739 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-07-29T00:49:59.740 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-07-29T00:49:59.740 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-07-29T00:49:59.740 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-07-29T00:49:59.741 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-07-29T00:49:59.741 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-07-29T00:49:59.741 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-07-29T00:49:59.742 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-07-29T00:49:59.742 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-07-29T00:49:59.742 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-07-29T00:49:59.742 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-07-29T00:49:59.743 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-07-29T00:49:59.743 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-07-29T00:49:59.743 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-07-29T00:49:59.744 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-07-29T00:49:59.744 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-07-29T00:49:59.744 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-07-29T00:49:59.744 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-07-29T00:49:59.745 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-07-29T00:49:59.745 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-07-29T00:49:59.745 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-07-29T00:49:59.746 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-07-29T00:49:59.746 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-07-29T00:49:59.746 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-07-29T00:49:59.746 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-07-29T00:49:59.747 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-29T00:49:59.747 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:49:59.748 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:49:59.748 DEBUG:boto:path=/hiiqos-61/?versions&max-keys=999 2023-07-29T00:49:59.748 DEBUG:boto:auth_path=/hiiqos-61/?versions&max-keys=999 2023-07-29T00:49:59.749 DEBUG:boto:Method: GET 2023-07-29T00:49:59.749 DEBUG:boto:Path: /hiiqos-61/?versions&max-keys=999 2023-07-29T00:49:59.749 DEBUG:boto:Data: 2023-07-29T00:49:59.750 DEBUG:boto:Headers: {} 2023-07-29T00:49:59.750 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:49:59.750 DEBUG:boto:Port: 8000 2023-07-29T00:49:59.751 DEBUG:boto:Params: {} 2023-07-29T00:49:59.751 DEBUG:boto:Token: None 2023-07-29T00:49:59.751 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:49:59 GMT /hiiqos-61/?versions 2023-07-29T00:49:59.752 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4mEdPNr0gfnrfglomjuVSo0BL7M= 2023-07-29T00:49:59.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:49:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4mEdPNr0gfnrfglomjuVSo0BL7M=', 'Content-Length': '0'} 2023-07-29T00:49:59.757 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5c3d8c1028ad68b-0064c46237-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:49:59.758 DEBUG:boto:b'hiiqos-61999falseobj11nulltrue2023-07-29T00:45:01.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-07-29T00:45:03.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-07-29T00:45:02.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-07-29T00:45:02.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-07-29T00:45:02.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-07-29T00:45:02.452Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-07-29T00:45:03.128Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-07-29T00:45:02.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-07-29T00:45:01.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-07-29T00:45:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-07-29T00:45:01.674Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:45:01.061Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-07-29T00:45:01.985Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-07-29T00:45:02.278Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-07-29T00:45:02.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:45:01.083Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-07-29T00:45:01.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-07-29T00:45:03.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-07-29T00:45:01.691Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:45:01.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-07-29T00:45:02.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-07-29T00:45:01.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:45:01.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-07-29T00:45:01.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-07-29T00:45:02.299Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-07-29T00:45:02.703Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-07-29T00:45:01.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-07-29T00:45:02.020Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-07-29T00:45:03.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-07-29T00:45:01.763Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-07-29T00:45:01.475Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-07-29T00:45:02.365Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-07-29T00:45:02.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-07-29T00:45:02.721Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-07-29T00:45:01.780Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:49:59.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-07-29T00:49:59.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-07-29T00:49:59.761 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-07-29T00:49:59.761 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-07-29T00:49:59.761 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-07-29T00:49:59.762 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-07-29T00:49:59.762 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-07-29T00:49:59.762 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-07-29T00:49:59.762 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-07-29T00:49:59.763 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-07-29T00:49:59.763 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-07-29T00:49:59.763 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-07-29T00:49:59.764 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-07-29T00:49:59.764 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-07-29T00:49:59.764 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-07-29T00:49:59.765 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-07-29T00:49:59.765 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-07-29T00:49:59.765 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-07-29T00:49:59.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-07-29T00:49:59.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-07-29T00:49:59.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-07-29T00:49:59.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-07-29T00:49:59.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-07-29T00:49:59.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-07-29T00:49:59.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-07-29T00:49:59.768 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-07-29T00:49:59.768 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-07-29T00:49:59.768 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-07-29T00:49:59.769 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-07-29T00:49:59.769 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-07-29T00:49:59.769 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-07-29T00:49:59.770 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-07-29T00:49:59.770 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-07-29T00:49:59.770 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-07-29T00:49:59.770 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-07-29T00:49:59.771 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:49:59.771 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:49:59.772 DEBUG:boto:path=/hiiqos-61/?versions&max-keys=999 2023-07-29T00:49:59.772 DEBUG:boto:auth_path=/hiiqos-61/?versions&max-keys=999 2023-07-29T00:49:59.772 DEBUG:boto:Method: GET 2023-07-29T00:49:59.773 DEBUG:boto:Path: /hiiqos-61/?versions&max-keys=999 2023-07-29T00:49:59.773 DEBUG:boto:Data: 2023-07-29T00:49:59.774 DEBUG:boto:Headers: {} 2023-07-29T00:49:59.774 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:49:59.774 DEBUG:boto:Port: 8001 2023-07-29T00:49:59.775 DEBUG:boto:Params: {} 2023-07-29T00:49:59.775 DEBUG:boto:Token: None 2023-07-29T00:49:59.776 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:49:59 GMT /hiiqos-61/?versions 2023-07-29T00:49:59.776 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4mEdPNr0gfnrfglomjuVSo0BL7M= 2023-07-29T00:49:59.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:49:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4mEdPNr0gfnrfglomjuVSo0BL7M=', 'Content-Length': '0'} 2023-07-29T00:49:59.780 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000296679eb680539e-0064c46237-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:49:59.781 DEBUG:boto:b'hiiqos-61999falseobj11nulltrue2023-07-29T00:45:01.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-07-29T00:45:03.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-07-29T00:45:02.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-07-29T00:45:02.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-07-29T00:45:02.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-07-29T00:45:02.452Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-07-29T00:45:03.128Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-07-29T00:45:02.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-07-29T00:45:01.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-07-29T00:45:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-07-29T00:45:01.674Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:45:01.061Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-07-29T00:45:01.985Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-07-29T00:45:02.278Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-07-29T00:45:02.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:45:01.083Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-07-29T00:45:01.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-07-29T00:45:03.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-07-29T00:45:01.691Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:45:01.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-07-29T00:45:02.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-07-29T00:45:01.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:45:01.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-07-29T00:45:01.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-07-29T00:45:02.299Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-07-29T00:45:02.703Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-07-29T00:45:01.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-07-29T00:45:02.020Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-07-29T00:45:03.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-07-29T00:45:01.763Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-07-29T00:45:01.475Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-07-29T00:45:02.365Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-07-29T00:45:02.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-07-29T00:45:02.721Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-07-29T00:45:01.780Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:49:59.784 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:49:59.784 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:49:59.785 DEBUG:boto:path=/hiiqos-61/?versions&max-keys=999 2023-07-29T00:49:59.785 DEBUG:boto:auth_path=/hiiqos-61/?versions&max-keys=999 2023-07-29T00:49:59.785 DEBUG:boto:Method: GET 2023-07-29T00:49:59.786 DEBUG:boto:Path: /hiiqos-61/?versions&max-keys=999 2023-07-29T00:49:59.786 DEBUG:boto:Data: 2023-07-29T00:49:59.787 DEBUG:boto:Headers: {} 2023-07-29T00:49:59.787 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:49:59.787 DEBUG:boto:Port: 8000 2023-07-29T00:49:59.788 DEBUG:boto:Params: {} 2023-07-29T00:49:59.788 DEBUG:boto:Token: None 2023-07-29T00:49:59.789 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:49:59 GMT /hiiqos-61/?versions 2023-07-29T00:49:59.789 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4mEdPNr0gfnrfglomjuVSo0BL7M= 2023-07-29T00:49:59.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:49:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4mEdPNr0gfnrfglomjuVSo0BL7M=', 'Content-Length': '0'} 2023-07-29T00:49:59.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af44591ca35e18b4-0064c46237-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:49:59.795 DEBUG:boto:b'hiiqos-61999falseobj11nulltrue2023-07-29T00:45:01.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-07-29T00:45:03.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-07-29T00:45:02.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-07-29T00:45:02.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-07-29T00:45:02.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-07-29T00:45:02.452Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-07-29T00:45:03.128Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-07-29T00:45:02.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-07-29T00:45:01.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-07-29T00:45:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-07-29T00:45:01.674Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:45:01.061Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-07-29T00:45:01.985Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-07-29T00:45:02.278Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-07-29T00:45:02.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:45:01.083Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-07-29T00:45:01.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-07-29T00:45:03.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-07-29T00:45:01.691Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:45:01.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-07-29T00:45:02.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-07-29T00:45:01.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:45:01.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-07-29T00:45:01.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-07-29T00:45:02.299Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-07-29T00:45:02.703Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-07-29T00:45:01.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-07-29T00:45:02.020Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-07-29T00:45:03.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-07-29T00:45:01.763Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-07-29T00:45:01.475Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-07-29T00:45:02.365Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-07-29T00:45:02.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-07-29T00:45:02.721Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-07-29T00:45:01.780Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:49:59.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-07-29T00:49:59.798 DEBUG:boto:path=/hiiqos-61/obj11 2023-07-29T00:49:59.799 DEBUG:boto:auth_path=/hiiqos-61/obj11 2023-07-29T00:49:59.799 DEBUG:boto:path=/hiiqos-61/obj11?versionId=null 2023-07-29T00:49:59.800 DEBUG:boto:auth_path=/hiiqos-61/obj11?versionId=null 2023-07-29T00:49:59.800 DEBUG:boto:Method: GET 2023-07-29T00:49:59.800 DEBUG:boto:Path: /hiiqos-61/obj11?versionId=null 2023-07-29T00:49:59.801 DEBUG:boto:Data: 2023-07-29T00:49:59.801 DEBUG:boto:Headers: {} 2023-07-29T00:49:59.802 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:49:59.802 DEBUG:boto:Port: 8001 2023-07-29T00:49:59.802 DEBUG:boto:Params: {} 2023-07-29T00:49:59.803 DEBUG:boto:Token: None 2023-07-29T00:49:59.803 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:49:59 GMT /hiiqos-61/obj11?versionId=null 2023-07-29T00:49:59.804 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RM9eVIvOC3A6JBz5kCcUXWTdeUM= 2023-07-29T00:49:59.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:49:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RM9eVIvOC3A6JBz5kCcUXWTdeUM=', 'Content-Length': '0'} 2023-07-29T00:49:59.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029115b1ac0a39294-0064c46237-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:49:59.808 DEBUG:boto:path=/hiiqos-61/obj11 2023-07-29T00:49:59.808 DEBUG:boto:auth_path=/hiiqos-61/obj11 2023-07-29T00:49:59.809 DEBUG:boto:path=/hiiqos-61/obj11?versionId=null 2023-07-29T00:49:59.809 DEBUG:boto:auth_path=/hiiqos-61/obj11?versionId=null 2023-07-29T00:49:59.809 DEBUG:boto:Method: GET 2023-07-29T00:49:59.810 DEBUG:boto:Path: /hiiqos-61/obj11?versionId=null 2023-07-29T00:49:59.810 DEBUG:boto:Data: 2023-07-29T00:49:59.811 DEBUG:boto:Headers: {} 2023-07-29T00:49:59.811 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:49:59.811 DEBUG:boto:Port: 8000 2023-07-29T00:49:59.812 DEBUG:boto:Params: {} 2023-07-29T00:49:59.812 DEBUG:boto:Token: None 2023-07-29T00:49:59.813 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:49:59 GMT /hiiqos-61/obj11?versionId=null 2023-07-29T00:49:59.813 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RM9eVIvOC3A6JBz5kCcUXWTdeUM= 2023-07-29T00:49:59.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:49:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RM9eVIvOC3A6JBz5kCcUXWTdeUM=', 'Content-Length': '0'} 2023-07-29T00:49:59.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3a4cc2cc221d80f-0064c46237-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:49:59.860 DEBUG:boto:path=/hiiqos-61/obj11 2023-07-29T00:49:59.861 DEBUG:boto:auth_path=/hiiqos-61/obj11 2023-07-29T00:49:59.861 DEBUG:boto:path=/hiiqos-61/obj11?versionId=null 2023-07-29T00:49:59.862 DEBUG:boto:auth_path=/hiiqos-61/obj11?versionId=null 2023-07-29T00:49:59.862 DEBUG:boto:Method: HEAD 2023-07-29T00:49:59.863 DEBUG:boto:Path: /hiiqos-61/obj11?versionId=null 2023-07-29T00:49:59.863 DEBUG:boto:Data: 2023-07-29T00:49:59.864 DEBUG:boto:Headers: {} 2023-07-29T00:49:59.864 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:49:59.865 DEBUG:boto:Port: 8001 2023-07-29T00:49:59.865 DEBUG:boto:Params: {} 2023-07-29T00:49:59.865 DEBUG:boto:Token: None 2023-07-29T00:49:59.866 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:49:59 GMT /hiiqos-61/obj11?versionId=null 2023-07-29T00:49:59.866 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jqIVifDv0N/tlocr4DeEfBJaBVw= 2023-07-29T00:49:59.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:49:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jqIVifDv0N/tlocr4DeEfBJaBVw=', 'Content-Length': '0'} 2023-07-29T00:49:59.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bdd4dc06cc3b207-0064c46237-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:49:59.869 DEBUG:boto:path=/hiiqos-61/obj11 2023-07-29T00:49:59.870 DEBUG:boto:auth_path=/hiiqos-61/obj11 2023-07-29T00:49:59.870 DEBUG:boto:path=/hiiqos-61/obj11?versionId=null 2023-07-29T00:49:59.870 DEBUG:boto:auth_path=/hiiqos-61/obj11?versionId=null 2023-07-29T00:49:59.871 DEBUG:boto:Method: HEAD 2023-07-29T00:49:59.871 DEBUG:boto:Path: /hiiqos-61/obj11?versionId=null 2023-07-29T00:49:59.871 DEBUG:boto:Data: 2023-07-29T00:49:59.872 DEBUG:boto:Headers: {} 2023-07-29T00:49:59.872 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:49:59.872 DEBUG:boto:Port: 8000 2023-07-29T00:49:59.873 DEBUG:boto:Params: {} 2023-07-29T00:49:59.873 DEBUG:boto:Token: None 2023-07-29T00:49:59.874 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:49:59 GMT /hiiqos-61/obj11?versionId=null 2023-07-29T00:49:59.874 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jqIVifDv0N/tlocr4DeEfBJaBVw= 2023-07-29T00:49:59.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:49:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jqIVifDv0N/tlocr4DeEfBJaBVw=', 'Content-Length': '0'} 2023-07-29T00:49:59.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000262d45ad33be5576-0064c46237-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:49:59.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-07-29T00:49:59.878 DEBUG:boto:path=/hiiqos-61/obj11 2023-07-29T00:49:59.878 DEBUG:boto:auth_path=/hiiqos-61/obj11 2023-07-29T00:49:59.879 DEBUG:boto:path=/hiiqos-61/obj11?versionId=null 2023-07-29T00:49:59.879 DEBUG:boto:auth_path=/hiiqos-61/obj11?versionId=null 2023-07-29T00:49:59.879 DEBUG:boto:Method: GET 2023-07-29T00:49:59.880 DEBUG:boto:Path: /hiiqos-61/obj11?versionId=null 2023-07-29T00:49:59.880 DEBUG:boto:Data: 2023-07-29T00:49:59.881 DEBUG:boto:Headers: {} 2023-07-29T00:49:59.882 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:49:59.882 DEBUG:boto:Port: 8001 2023-07-29T00:49:59.882 DEBUG:boto:Params: {} 2023-07-29T00:49:59.883 DEBUG:boto:Token: None 2023-07-29T00:49:59.884 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:49:59 GMT /hiiqos-61/obj11?versionId=null 2023-07-29T00:49:59.884 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RM9eVIvOC3A6JBz5kCcUXWTdeUM= 2023-07-29T00:49:59.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:49:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RM9eVIvOC3A6JBz5kCcUXWTdeUM=', 'Content-Length': '0'} 2023-07-29T00:49:59.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c62ead3dac27e80c-0064c46237-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:49:59.887 DEBUG:boto:path=/hiiqos-61/obj11 2023-07-29T00:49:59.887 DEBUG:boto:auth_path=/hiiqos-61/obj11 2023-07-29T00:49:59.888 DEBUG:boto:path=/hiiqos-61/obj11?versionId=null 2023-07-29T00:49:59.888 DEBUG:boto:auth_path=/hiiqos-61/obj11?versionId=null 2023-07-29T00:49:59.889 DEBUG:boto:Method: GET 2023-07-29T00:49:59.889 DEBUG:boto:Path: /hiiqos-61/obj11?versionId=null 2023-07-29T00:49:59.890 DEBUG:boto:Data: 2023-07-29T00:49:59.890 DEBUG:boto:Headers: {} 2023-07-29T00:49:59.890 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:49:59.891 DEBUG:boto:Port: 8000 2023-07-29T00:49:59.891 DEBUG:boto:Params: {} 2023-07-29T00:49:59.892 DEBUG:boto:Token: None 2023-07-29T00:49:59.892 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:49:59 GMT /hiiqos-61/obj11?versionId=null 2023-07-29T00:49:59.893 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RM9eVIvOC3A6JBz5kCcUXWTdeUM= 2023-07-29T00:49:59.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:49:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RM9eVIvOC3A6JBz5kCcUXWTdeUM=', 'Content-Length': '0'} 2023-07-29T00:49:59.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dea3369e60245a2c-0064c46237-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:49:59.896 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:49:59.896 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:49:59.896 DEBUG:boto:path=/hiiqos-61/?prefix=obj11 2023-07-29T00:49:59.897 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj11 2023-07-29T00:49:59.897 DEBUG:boto:Method: GET 2023-07-29T00:49:59.898 DEBUG:boto:Path: /hiiqos-61/?prefix=obj11 2023-07-29T00:49:59.898 DEBUG:boto:Data: 2023-07-29T00:49:59.898 DEBUG:boto:Headers: {} 2023-07-29T00:49:59.899 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:49:59.899 DEBUG:boto:Port: 8001 2023-07-29T00:49:59.899 DEBUG:boto:Params: {} 2023-07-29T00:49:59.900 DEBUG:boto:Token: None 2023-07-29T00:49:59.900 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:49:59 GMT /hiiqos-61/ 2023-07-29T00:49:59.901 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yQCpzrAmKVU/PnzOzVNQ33Q9RIY= 2023-07-29T00:49:59.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:49:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yQCpzrAmKVU/PnzOzVNQ33Q9RIY=', 'Content-Length': '0'} 2023-07-29T00:49:59.904 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a264cd920657d5f7-0064c46237-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:49:59.904 DEBUG:boto:b'hiiqos-61obj111000falseobj112023-07-29T00:45:01.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-07-29T00:45:03.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-07-29T00:45:02.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-07-29T00:45:02.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:49:59.905 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:49:59.906 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:49:59.906 DEBUG:boto:path=/hiiqos-61/?prefix=obj11 2023-07-29T00:49:59.907 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj11 2023-07-29T00:49:59.907 DEBUG:boto:Method: GET 2023-07-29T00:49:59.907 DEBUG:boto:Path: /hiiqos-61/?prefix=obj11 2023-07-29T00:49:59.908 DEBUG:boto:Data: 2023-07-29T00:49:59.908 DEBUG:boto:Headers: {} 2023-07-29T00:49:59.908 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:49:59.909 DEBUG:boto:Port: 8000 2023-07-29T00:49:59.909 DEBUG:boto:Params: {} 2023-07-29T00:49:59.910 DEBUG:boto:Token: None 2023-07-29T00:49:59.910 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:49:59 GMT /hiiqos-61/ 2023-07-29T00:49:59.911 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yQCpzrAmKVU/PnzOzVNQ33Q9RIY= 2023-07-29T00:49:59.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:49:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yQCpzrAmKVU/PnzOzVNQ33Q9RIY=', 'Content-Length': '0'} 2023-07-29T00:49:59.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000796b58d3877dac6c-0064c46237-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:49:59.915 DEBUG:boto:b'hiiqos-61obj111000falseobj112023-07-29T00:45:01.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-07-29T00:45:03.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-07-29T00:45:02.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-07-29T00:45:02.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:49:59.916 DEBUG:boto:path=/hiiqos-61/obj11 2023-07-29T00:49:59.916 DEBUG:boto:auth_path=/hiiqos-61/obj11 2023-07-29T00:49:59.917 DEBUG:boto:Method: HEAD 2023-07-29T00:49:59.917 DEBUG:boto:Path: /hiiqos-61/obj11 2023-07-29T00:49:59.917 DEBUG:boto:Data: 2023-07-29T00:49:59.918 DEBUG:boto:Headers: {} 2023-07-29T00:49:59.918 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:49:59.918 DEBUG:boto:Port: 8001 2023-07-29T00:49:59.919 DEBUG:boto:Params: {} 2023-07-29T00:49:59.919 DEBUG:boto:Token: None 2023-07-29T00:49:59.920 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:49:59 GMT /hiiqos-61/obj11 2023-07-29T00:49:59.920 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SSy+2vBzgYnzeUh4rbP0C80qgSM= 2023-07-29T00:49:59.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:49:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SSy+2vBzgYnzeUh4rbP0C80qgSM=', 'Content-Length': '0'} 2023-07-29T00:49:59.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000218ad7bc3e88360b-0064c46237-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:49:59.923 DEBUG:boto:path=/hiiqos-61/obj11 2023-07-29T00:49:59.923 DEBUG:boto:auth_path=/hiiqos-61/obj11 2023-07-29T00:49:59.923 DEBUG:boto:Method: HEAD 2023-07-29T00:49:59.924 DEBUG:boto:Path: /hiiqos-61/obj11 2023-07-29T00:49:59.924 DEBUG:boto:Data: 2023-07-29T00:49:59.925 DEBUG:boto:Headers: {} 2023-07-29T00:49:59.925 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:49:59.925 DEBUG:boto:Port: 8000 2023-07-29T00:49:59.926 DEBUG:boto:Params: {} 2023-07-29T00:49:59.926 DEBUG:boto:Token: None 2023-07-29T00:49:59.926 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:49:59 GMT /hiiqos-61/obj11 2023-07-29T00:49:59.927 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SSy+2vBzgYnzeUh4rbP0C80qgSM= 2023-07-29T00:49:59.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:49:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SSy+2vBzgYnzeUh4rbP0C80qgSM=', 'Content-Length': '0'} 2023-07-29T00:49:59.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047b354007dd3d9c9-0064c46237-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:49:59.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-07-29T00:49:59.930 DEBUG:boto:path=/hiiqos-61/obj11 2023-07-29T00:49:59.930 DEBUG:boto:auth_path=/hiiqos-61/obj11 2023-07-29T00:49:59.931 DEBUG:boto:Method: GET 2023-07-29T00:49:59.931 DEBUG:boto:Path: /hiiqos-61/obj11 2023-07-29T00:49:59.931 DEBUG:boto:Data: 2023-07-29T00:49:59.932 DEBUG:boto:Headers: {} 2023-07-29T00:49:59.932 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:49:59.932 DEBUG:boto:Port: 8001 2023-07-29T00:49:59.933 DEBUG:boto:Params: {} 2023-07-29T00:49:59.933 DEBUG:boto:Token: None 2023-07-29T00:49:59.934 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:49:59 GMT /hiiqos-61/obj11 2023-07-29T00:49:59.934 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bAG2zNOea9TmxZVhZezEHQ/1OiQ= 2023-07-29T00:49:59.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:49:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bAG2zNOea9TmxZVhZezEHQ/1OiQ=', 'Content-Length': '0'} 2023-07-29T00:49:59.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000640abcc069b27e5b-0064c46237-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:49:59.937 DEBUG:boto:path=/hiiqos-61/obj11 2023-07-29T00:49:59.937 DEBUG:boto:auth_path=/hiiqos-61/obj11 2023-07-29T00:49:59.938 DEBUG:boto:Method: GET 2023-07-29T00:49:59.938 DEBUG:boto:Path: /hiiqos-61/obj11 2023-07-29T00:49:59.938 DEBUG:boto:Data: 2023-07-29T00:49:59.939 DEBUG:boto:Headers: {} 2023-07-29T00:49:59.939 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:49:59.939 DEBUG:boto:Port: 8000 2023-07-29T00:49:59.940 DEBUG:boto:Params: {} 2023-07-29T00:49:59.940 DEBUG:boto:Token: None 2023-07-29T00:49:59.941 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:49:59 GMT /hiiqos-61/obj11 2023-07-29T00:49:59.941 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bAG2zNOea9TmxZVhZezEHQ/1OiQ= 2023-07-29T00:49:59.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:49:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bAG2zNOea9TmxZVhZezEHQ/1OiQ=', 'Content-Length': '0'} 2023-07-29T00:49:59.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b4462710024f757-0064c46237-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:49:59.984 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:49:59.985 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:49:59.985 DEBUG:boto:path=/hiiqos-61/?prefix=obj11 2023-07-29T00:49:59.986 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj11 2023-07-29T00:49:59.986 DEBUG:boto:Method: GET 2023-07-29T00:49:59.986 DEBUG:boto:Path: /hiiqos-61/?prefix=obj11 2023-07-29T00:49:59.987 DEBUG:boto:Data: 2023-07-29T00:49:59.987 DEBUG:boto:Headers: {} 2023-07-29T00:49:59.987 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:49:59.988 DEBUG:boto:Port: 8001 2023-07-29T00:49:59.988 DEBUG:boto:Params: {} 2023-07-29T00:49:59.989 DEBUG:boto:Token: None 2023-07-29T00:49:59.989 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:49:59 GMT /hiiqos-61/ 2023-07-29T00:49:59.989 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yQCpzrAmKVU/PnzOzVNQ33Q9RIY= 2023-07-29T00:49:59.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:49:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yQCpzrAmKVU/PnzOzVNQ33Q9RIY=', 'Content-Length': '0'} 2023-07-29T00:49:59.992 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006aa778d78740ea6f-0064c46237-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:49:59.993 DEBUG:boto:b'hiiqos-61obj111000falseobj112023-07-29T00:45:01.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-07-29T00:45:03.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-07-29T00:45:02.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-07-29T00:45:02.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:49:59.994 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:49:59.994 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:49:59.995 DEBUG:boto:path=/hiiqos-61/?prefix=obj11 2023-07-29T00:49:59.995 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj11 2023-07-29T00:49:59.995 DEBUG:boto:Method: GET 2023-07-29T00:49:59.996 DEBUG:boto:Path: /hiiqos-61/?prefix=obj11 2023-07-29T00:49:59.996 DEBUG:boto:Data: 2023-07-29T00:49:59.996 DEBUG:boto:Headers: {} 2023-07-29T00:49:59.997 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:49:59.997 DEBUG:boto:Port: 8000 2023-07-29T00:49:59.997 DEBUG:boto:Params: {} 2023-07-29T00:49:59.998 DEBUG:boto:Token: None 2023-07-29T00:49:59.998 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:49:59 GMT /hiiqos-61/ 2023-07-29T00:49:59.999 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yQCpzrAmKVU/PnzOzVNQ33Q9RIY= 2023-07-29T00:49:59.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:49:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yQCpzrAmKVU/PnzOzVNQ33Q9RIY=', 'Content-Length': '0'} 2023-07-29T00:50:00.003 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d81350a5d7ddb01c-0064c46237-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.004 DEBUG:boto:b'hiiqos-61obj111000falseobj112023-07-29T00:45:01.039Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-07-29T00:45:03.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-07-29T00:45:02.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-07-29T00:45:02.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:00.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-07-29T00:50:00.005 DEBUG:boto:path=/hiiqos-61/obj111 2023-07-29T00:50:00.005 DEBUG:boto:auth_path=/hiiqos-61/obj111 2023-07-29T00:50:00.006 DEBUG:boto:path=/hiiqos-61/obj111?versionId=null 2023-07-29T00:50:00.006 DEBUG:boto:auth_path=/hiiqos-61/obj111?versionId=null 2023-07-29T00:50:00.006 DEBUG:boto:Method: GET 2023-07-29T00:50:00.007 DEBUG:boto:Path: /hiiqos-61/obj111?versionId=null 2023-07-29T00:50:00.007 DEBUG:boto:Data: 2023-07-29T00:50:00.007 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.008 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:00.008 DEBUG:boto:Port: 8001 2023-07-29T00:50:00.008 DEBUG:boto:Params: {} 2023-07-29T00:50:00.009 DEBUG:boto:Token: None 2023-07-29T00:50:00.009 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj111?versionId=null 2023-07-29T00:50:00.010 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JLkkIQJB1fwXyjAU8/9OGhWjjHE= 2023-07-29T00:50:00.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JLkkIQJB1fwXyjAU8/9OGhWjjHE=', 'Content-Length': '0'} 2023-07-29T00:50:00.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000561480d8dfdda89e-0064c46237-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.052 DEBUG:boto:path=/hiiqos-61/obj111 2023-07-29T00:50:00.053 DEBUG:boto:auth_path=/hiiqos-61/obj111 2023-07-29T00:50:00.053 DEBUG:boto:path=/hiiqos-61/obj111?versionId=null 2023-07-29T00:50:00.053 DEBUG:boto:auth_path=/hiiqos-61/obj111?versionId=null 2023-07-29T00:50:00.054 DEBUG:boto:Method: GET 2023-07-29T00:50:00.054 DEBUG:boto:Path: /hiiqos-61/obj111?versionId=null 2023-07-29T00:50:00.055 DEBUG:boto:Data: 2023-07-29T00:50:00.055 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.055 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:00.056 DEBUG:boto:Port: 8000 2023-07-29T00:50:00.056 DEBUG:boto:Params: {} 2023-07-29T00:50:00.056 DEBUG:boto:Token: None 2023-07-29T00:50:00.057 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj111?versionId=null 2023-07-29T00:50:00.057 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JLkkIQJB1fwXyjAU8/9OGhWjjHE= 2023-07-29T00:50:00.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JLkkIQJB1fwXyjAU8/9OGhWjjHE=', 'Content-Length': '0'} 2023-07-29T00:50:00.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6a0cff474e8f248-0064c46237-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.060 DEBUG:boto:path=/hiiqos-61/obj111 2023-07-29T00:50:00.061 DEBUG:boto:auth_path=/hiiqos-61/obj111 2023-07-29T00:50:00.061 DEBUG:boto:path=/hiiqos-61/obj111?versionId=null 2023-07-29T00:50:00.061 DEBUG:boto:auth_path=/hiiqos-61/obj111?versionId=null 2023-07-29T00:50:00.062 DEBUG:boto:Method: HEAD 2023-07-29T00:50:00.062 DEBUG:boto:Path: /hiiqos-61/obj111?versionId=null 2023-07-29T00:50:00.062 DEBUG:boto:Data: 2023-07-29T00:50:00.063 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.063 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:00.063 DEBUG:boto:Port: 8001 2023-07-29T00:50:00.064 DEBUG:boto:Params: {} 2023-07-29T00:50:00.064 DEBUG:boto:Token: None 2023-07-29T00:50:00.065 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj111?versionId=null 2023-07-29T00:50:00.065 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EE+KuHhT5XeTxLMs7HU5L7RN2rk= 2023-07-29T00:50:00.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EE+KuHhT5XeTxLMs7HU5L7RN2rk=', 'Content-Length': '0'} 2023-07-29T00:50:00.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e6e2fbdae4a74f9-0064c46237-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:49:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.068 DEBUG:boto:path=/hiiqos-61/obj111 2023-07-29T00:50:00.068 DEBUG:boto:auth_path=/hiiqos-61/obj111 2023-07-29T00:50:00.068 DEBUG:boto:path=/hiiqos-61/obj111?versionId=null 2023-07-29T00:50:00.069 DEBUG:boto:auth_path=/hiiqos-61/obj111?versionId=null 2023-07-29T00:50:00.069 DEBUG:boto:Method: HEAD 2023-07-29T00:50:00.069 DEBUG:boto:Path: /hiiqos-61/obj111?versionId=null 2023-07-29T00:50:00.070 DEBUG:boto:Data: 2023-07-29T00:50:00.070 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.070 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:00.071 DEBUG:boto:Port: 8000 2023-07-29T00:50:00.071 DEBUG:boto:Params: {} 2023-07-29T00:50:00.072 DEBUG:boto:Token: None 2023-07-29T00:50:00.072 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj111?versionId=null 2023-07-29T00:50:00.072 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EE+KuHhT5XeTxLMs7HU5L7RN2rk= 2023-07-29T00:50:00.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EE+KuHhT5XeTxLMs7HU5L7RN2rk=', 'Content-Length': '0'} 2023-07-29T00:50:00.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f0e7559803b6bef-0064c46238-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-07-29T00:50:00.076 DEBUG:boto:path=/hiiqos-61/obj111 2023-07-29T00:50:00.076 DEBUG:boto:auth_path=/hiiqos-61/obj111 2023-07-29T00:50:00.076 DEBUG:boto:path=/hiiqos-61/obj111?versionId=null 2023-07-29T00:50:00.077 DEBUG:boto:auth_path=/hiiqos-61/obj111?versionId=null 2023-07-29T00:50:00.077 DEBUG:boto:Method: GET 2023-07-29T00:50:00.077 DEBUG:boto:Path: /hiiqos-61/obj111?versionId=null 2023-07-29T00:50:00.078 DEBUG:boto:Data: 2023-07-29T00:50:00.078 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.078 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:00.079 DEBUG:boto:Port: 8001 2023-07-29T00:50:00.079 DEBUG:boto:Params: {} 2023-07-29T00:50:00.080 DEBUG:boto:Token: None 2023-07-29T00:50:00.080 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj111?versionId=null 2023-07-29T00:50:00.081 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JLkkIQJB1fwXyjAU8/9OGhWjjHE= 2023-07-29T00:50:00.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JLkkIQJB1fwXyjAU8/9OGhWjjHE=', 'Content-Length': '0'} 2023-07-29T00:50:00.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edf05a96c5d8f933-0064c46238-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.084 DEBUG:boto:path=/hiiqos-61/obj111 2023-07-29T00:50:00.084 DEBUG:boto:auth_path=/hiiqos-61/obj111 2023-07-29T00:50:00.084 DEBUG:boto:path=/hiiqos-61/obj111?versionId=null 2023-07-29T00:50:00.085 DEBUG:boto:auth_path=/hiiqos-61/obj111?versionId=null 2023-07-29T00:50:00.085 DEBUG:boto:Method: GET 2023-07-29T00:50:00.085 DEBUG:boto:Path: /hiiqos-61/obj111?versionId=null 2023-07-29T00:50:00.086 DEBUG:boto:Data: 2023-07-29T00:50:00.086 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.086 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:00.087 DEBUG:boto:Port: 8000 2023-07-29T00:50:00.087 DEBUG:boto:Params: {} 2023-07-29T00:50:00.088 DEBUG:boto:Token: None 2023-07-29T00:50:00.088 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj111?versionId=null 2023-07-29T00:50:00.088 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JLkkIQJB1fwXyjAU8/9OGhWjjHE= 2023-07-29T00:50:00.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JLkkIQJB1fwXyjAU8/9OGhWjjHE=', 'Content-Length': '0'} 2023-07-29T00:50:00.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d74dca1d0b01349-0064c46238-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.091 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:00.092 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:00.092 DEBUG:boto:path=/hiiqos-61/?prefix=obj111 2023-07-29T00:50:00.092 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj111 2023-07-29T00:50:00.093 DEBUG:boto:Method: GET 2023-07-29T00:50:00.093 DEBUG:boto:Path: /hiiqos-61/?prefix=obj111 2023-07-29T00:50:00.093 DEBUG:boto:Data: 2023-07-29T00:50:00.094 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.094 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:00.095 DEBUG:boto:Port: 8001 2023-07-29T00:50:00.095 DEBUG:boto:Params: {} 2023-07-29T00:50:00.095 DEBUG:boto:Token: None 2023-07-29T00:50:00.096 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/ 2023-07-29T00:50:00.096 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:A5qU4OtGaFhwcdcXg0UzfjcCpTo= 2023-07-29T00:50:00.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:A5qU4OtGaFhwcdcXg0UzfjcCpTo=', 'Content-Length': '0'} 2023-07-29T00:50:00.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039bb76e89e0a58de-0064c46238-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.099 DEBUG:boto:b'hiiqos-61obj1111000falseobj1112023-07-29T00:45:03.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:00.100 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:00.101 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:00.101 DEBUG:boto:path=/hiiqos-61/?prefix=obj111 2023-07-29T00:50:00.101 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj111 2023-07-29T00:50:00.102 DEBUG:boto:Method: GET 2023-07-29T00:50:00.102 DEBUG:boto:Path: /hiiqos-61/?prefix=obj111 2023-07-29T00:50:00.102 DEBUG:boto:Data: 2023-07-29T00:50:00.103 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.103 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:00.104 DEBUG:boto:Port: 8000 2023-07-29T00:50:00.104 DEBUG:boto:Params: {} 2023-07-29T00:50:00.104 DEBUG:boto:Token: None 2023-07-29T00:50:00.105 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/ 2023-07-29T00:50:00.105 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:A5qU4OtGaFhwcdcXg0UzfjcCpTo= 2023-07-29T00:50:00.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:A5qU4OtGaFhwcdcXg0UzfjcCpTo=', 'Content-Length': '0'} 2023-07-29T00:50:00.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f23bcc2a9ec68e6-0064c46238-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.110 DEBUG:boto:b'hiiqos-61obj1111000falseobj1112023-07-29T00:45:03.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:00.111 DEBUG:boto:path=/hiiqos-61/obj111 2023-07-29T00:50:00.111 DEBUG:boto:auth_path=/hiiqos-61/obj111 2023-07-29T00:50:00.111 DEBUG:boto:Method: HEAD 2023-07-29T00:50:00.112 DEBUG:boto:Path: /hiiqos-61/obj111 2023-07-29T00:50:00.112 DEBUG:boto:Data: 2023-07-29T00:50:00.112 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.113 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:00.113 DEBUG:boto:Port: 8001 2023-07-29T00:50:00.113 DEBUG:boto:Params: {} 2023-07-29T00:50:00.114 DEBUG:boto:Token: None 2023-07-29T00:50:00.114 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj111 2023-07-29T00:50:00.115 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fQjLQQo0F034XsnpzpZtNxceHGs= 2023-07-29T00:50:00.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fQjLQQo0F034XsnpzpZtNxceHGs=', 'Content-Length': '0'} 2023-07-29T00:50:00.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd458b38cc5b38cf-0064c46238-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.117 DEBUG:boto:path=/hiiqos-61/obj111 2023-07-29T00:50:00.118 DEBUG:boto:auth_path=/hiiqos-61/obj111 2023-07-29T00:50:00.118 DEBUG:boto:Method: HEAD 2023-07-29T00:50:00.118 DEBUG:boto:Path: /hiiqos-61/obj111 2023-07-29T00:50:00.119 DEBUG:boto:Data: 2023-07-29T00:50:00.119 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.119 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:00.120 DEBUG:boto:Port: 8000 2023-07-29T00:50:00.120 DEBUG:boto:Params: {} 2023-07-29T00:50:00.121 DEBUG:boto:Token: None 2023-07-29T00:50:00.121 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj111 2023-07-29T00:50:00.121 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fQjLQQo0F034XsnpzpZtNxceHGs= 2023-07-29T00:50:00.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fQjLQQo0F034XsnpzpZtNxceHGs=', 'Content-Length': '0'} 2023-07-29T00:50:00.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c5d794f68190dd0-0064c46238-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-07-29T00:50:00.125 DEBUG:boto:path=/hiiqos-61/obj111 2023-07-29T00:50:00.125 DEBUG:boto:auth_path=/hiiqos-61/obj111 2023-07-29T00:50:00.126 DEBUG:boto:Method: GET 2023-07-29T00:50:00.126 DEBUG:boto:Path: /hiiqos-61/obj111 2023-07-29T00:50:00.126 DEBUG:boto:Data: 2023-07-29T00:50:00.127 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.127 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:00.127 DEBUG:boto:Port: 8001 2023-07-29T00:50:00.128 DEBUG:boto:Params: {} 2023-07-29T00:50:00.128 DEBUG:boto:Token: None 2023-07-29T00:50:00.129 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj111 2023-07-29T00:50:00.129 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2nscVmkbDY4E1O1TNMfc5K/eob8= 2023-07-29T00:50:00.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2nscVmkbDY4E1O1TNMfc5K/eob8=', 'Content-Length': '0'} 2023-07-29T00:50:00.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e73cdd017cb36642-0064c46238-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.132 DEBUG:boto:path=/hiiqos-61/obj111 2023-07-29T00:50:00.132 DEBUG:boto:auth_path=/hiiqos-61/obj111 2023-07-29T00:50:00.133 DEBUG:boto:Method: GET 2023-07-29T00:50:00.133 DEBUG:boto:Path: /hiiqos-61/obj111 2023-07-29T00:50:00.134 DEBUG:boto:Data: 2023-07-29T00:50:00.134 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.134 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:00.135 DEBUG:boto:Port: 8000 2023-07-29T00:50:00.135 DEBUG:boto:Params: {} 2023-07-29T00:50:00.135 DEBUG:boto:Token: None 2023-07-29T00:50:00.136 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj111 2023-07-29T00:50:00.136 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2nscVmkbDY4E1O1TNMfc5K/eob8= 2023-07-29T00:50:00.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2nscVmkbDY4E1O1TNMfc5K/eob8=', 'Content-Length': '0'} 2023-07-29T00:50:00.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000678b76714bedc21d-0064c46238-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.139 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:00.140 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:00.140 DEBUG:boto:path=/hiiqos-61/?prefix=obj111 2023-07-29T00:50:00.140 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj111 2023-07-29T00:50:00.141 DEBUG:boto:Method: GET 2023-07-29T00:50:00.141 DEBUG:boto:Path: /hiiqos-61/?prefix=obj111 2023-07-29T00:50:00.141 DEBUG:boto:Data: 2023-07-29T00:50:00.142 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.142 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:00.143 DEBUG:boto:Port: 8001 2023-07-29T00:50:00.143 DEBUG:boto:Params: {} 2023-07-29T00:50:00.144 DEBUG:boto:Token: None 2023-07-29T00:50:00.144 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/ 2023-07-29T00:50:00.145 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:A5qU4OtGaFhwcdcXg0UzfjcCpTo= 2023-07-29T00:50:00.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:A5qU4OtGaFhwcdcXg0UzfjcCpTo=', 'Content-Length': '0'} 2023-07-29T00:50:00.148 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044ad2977f647b1ba-0064c46238-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.149 DEBUG:boto:b'hiiqos-61obj1111000falseobj1112023-07-29T00:45:03.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:00.150 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:00.150 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:00.150 DEBUG:boto:path=/hiiqos-61/?prefix=obj111 2023-07-29T00:50:00.151 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj111 2023-07-29T00:50:00.151 DEBUG:boto:Method: GET 2023-07-29T00:50:00.151 DEBUG:boto:Path: /hiiqos-61/?prefix=obj111 2023-07-29T00:50:00.152 DEBUG:boto:Data: 2023-07-29T00:50:00.152 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.153 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:00.153 DEBUG:boto:Port: 8000 2023-07-29T00:50:00.153 DEBUG:boto:Params: {} 2023-07-29T00:50:00.154 DEBUG:boto:Token: None 2023-07-29T00:50:00.154 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/ 2023-07-29T00:50:00.155 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:A5qU4OtGaFhwcdcXg0UzfjcCpTo= 2023-07-29T00:50:00.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:A5qU4OtGaFhwcdcXg0UzfjcCpTo=', 'Content-Length': '0'} 2023-07-29T00:50:00.159 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fabe2a29ca94e41-0064c46238-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.160 DEBUG:boto:b'hiiqos-61obj1111000falseobj1112023-07-29T00:45:03.066Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:00.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-07-29T00:50:00.161 DEBUG:boto:path=/hiiqos-61/obj115 2023-07-29T00:50:00.161 DEBUG:boto:auth_path=/hiiqos-61/obj115 2023-07-29T00:50:00.161 DEBUG:boto:path=/hiiqos-61/obj115?versionId=null 2023-07-29T00:50:00.162 DEBUG:boto:auth_path=/hiiqos-61/obj115?versionId=null 2023-07-29T00:50:00.162 DEBUG:boto:Method: GET 2023-07-29T00:50:00.163 DEBUG:boto:Path: /hiiqos-61/obj115?versionId=null 2023-07-29T00:50:00.163 DEBUG:boto:Data: 2023-07-29T00:50:00.163 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.164 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:00.164 DEBUG:boto:Port: 8001 2023-07-29T00:50:00.164 DEBUG:boto:Params: {} 2023-07-29T00:50:00.165 DEBUG:boto:Token: None 2023-07-29T00:50:00.165 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj115?versionId=null 2023-07-29T00:50:00.166 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SjKw3TeR8Fp3kUPNgAMx3e7Vx6o= 2023-07-29T00:50:00.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SjKw3TeR8Fp3kUPNgAMx3e7Vx6o=', 'Content-Length': '0'} 2023-07-29T00:50:00.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfa9e1f715c06df4-0064c46238-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.212 DEBUG:boto:path=/hiiqos-61/obj115 2023-07-29T00:50:00.213 DEBUG:boto:auth_path=/hiiqos-61/obj115 2023-07-29T00:50:00.213 DEBUG:boto:path=/hiiqos-61/obj115?versionId=null 2023-07-29T00:50:00.214 DEBUG:boto:auth_path=/hiiqos-61/obj115?versionId=null 2023-07-29T00:50:00.214 DEBUG:boto:Method: GET 2023-07-29T00:50:00.214 DEBUG:boto:Path: /hiiqos-61/obj115?versionId=null 2023-07-29T00:50:00.215 DEBUG:boto:Data: 2023-07-29T00:50:00.215 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.215 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:00.216 DEBUG:boto:Port: 8000 2023-07-29T00:50:00.216 DEBUG:boto:Params: {} 2023-07-29T00:50:00.217 DEBUG:boto:Token: None 2023-07-29T00:50:00.217 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj115?versionId=null 2023-07-29T00:50:00.217 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SjKw3TeR8Fp3kUPNgAMx3e7Vx6o= 2023-07-29T00:50:00.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SjKw3TeR8Fp3kUPNgAMx3e7Vx6o=', 'Content-Length': '0'} 2023-07-29T00:50:00.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063eb1e06b255b01c-0064c46238-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.220 DEBUG:boto:path=/hiiqos-61/obj115 2023-07-29T00:50:00.221 DEBUG:boto:auth_path=/hiiqos-61/obj115 2023-07-29T00:50:00.221 DEBUG:boto:path=/hiiqos-61/obj115?versionId=null 2023-07-29T00:50:00.222 DEBUG:boto:auth_path=/hiiqos-61/obj115?versionId=null 2023-07-29T00:50:00.222 DEBUG:boto:Method: HEAD 2023-07-29T00:50:00.222 DEBUG:boto:Path: /hiiqos-61/obj115?versionId=null 2023-07-29T00:50:00.223 DEBUG:boto:Data: 2023-07-29T00:50:00.223 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.223 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:00.224 DEBUG:boto:Port: 8001 2023-07-29T00:50:00.224 DEBUG:boto:Params: {} 2023-07-29T00:50:00.224 DEBUG:boto:Token: None 2023-07-29T00:50:00.225 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj115?versionId=null 2023-07-29T00:50:00.225 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zDgVdfmQAO6rBeRO+73CTvAkguU= 2023-07-29T00:50:00.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zDgVdfmQAO6rBeRO+73CTvAkguU=', 'Content-Length': '0'} 2023-07-29T00:50:00.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085d0103226709940-0064c46238-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.228 DEBUG:boto:path=/hiiqos-61/obj115 2023-07-29T00:50:00.228 DEBUG:boto:auth_path=/hiiqos-61/obj115 2023-07-29T00:50:00.229 DEBUG:boto:path=/hiiqos-61/obj115?versionId=null 2023-07-29T00:50:00.229 DEBUG:boto:auth_path=/hiiqos-61/obj115?versionId=null 2023-07-29T00:50:00.229 DEBUG:boto:Method: HEAD 2023-07-29T00:50:00.230 DEBUG:boto:Path: /hiiqos-61/obj115?versionId=null 2023-07-29T00:50:00.230 DEBUG:boto:Data: 2023-07-29T00:50:00.230 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.231 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:00.231 DEBUG:boto:Port: 8000 2023-07-29T00:50:00.231 DEBUG:boto:Params: {} 2023-07-29T00:50:00.232 DEBUG:boto:Token: None 2023-07-29T00:50:00.232 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj115?versionId=null 2023-07-29T00:50:00.232 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zDgVdfmQAO6rBeRO+73CTvAkguU= 2023-07-29T00:50:00.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zDgVdfmQAO6rBeRO+73CTvAkguU=', 'Content-Length': '0'} 2023-07-29T00:50:00.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eafe443b0a6dad3b-0064c46238-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-07-29T00:50:00.235 DEBUG:boto:path=/hiiqos-61/obj115 2023-07-29T00:50:00.236 DEBUG:boto:auth_path=/hiiqos-61/obj115 2023-07-29T00:50:00.236 DEBUG:boto:path=/hiiqos-61/obj115?versionId=null 2023-07-29T00:50:00.237 DEBUG:boto:auth_path=/hiiqos-61/obj115?versionId=null 2023-07-29T00:50:00.237 DEBUG:boto:Method: GET 2023-07-29T00:50:00.237 DEBUG:boto:Path: /hiiqos-61/obj115?versionId=null 2023-07-29T00:50:00.238 DEBUG:boto:Data: 2023-07-29T00:50:00.238 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.238 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:00.239 DEBUG:boto:Port: 8001 2023-07-29T00:50:00.239 DEBUG:boto:Params: {} 2023-07-29T00:50:00.239 DEBUG:boto:Token: None 2023-07-29T00:50:00.240 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj115?versionId=null 2023-07-29T00:50:00.240 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SjKw3TeR8Fp3kUPNgAMx3e7Vx6o= 2023-07-29T00:50:00.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SjKw3TeR8Fp3kUPNgAMx3e7Vx6o=', 'Content-Length': '0'} 2023-07-29T00:50:00.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd89b31c8078cf77-0064c46238-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.243 DEBUG:boto:path=/hiiqos-61/obj115 2023-07-29T00:50:00.243 DEBUG:boto:auth_path=/hiiqos-61/obj115 2023-07-29T00:50:00.244 DEBUG:boto:path=/hiiqos-61/obj115?versionId=null 2023-07-29T00:50:00.244 DEBUG:boto:auth_path=/hiiqos-61/obj115?versionId=null 2023-07-29T00:50:00.244 DEBUG:boto:Method: GET 2023-07-29T00:50:00.245 DEBUG:boto:Path: /hiiqos-61/obj115?versionId=null 2023-07-29T00:50:00.245 DEBUG:boto:Data: 2023-07-29T00:50:00.245 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.246 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:00.246 DEBUG:boto:Port: 8000 2023-07-29T00:50:00.246 DEBUG:boto:Params: {} 2023-07-29T00:50:00.247 DEBUG:boto:Token: None 2023-07-29T00:50:00.247 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj115?versionId=null 2023-07-29T00:50:00.248 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SjKw3TeR8Fp3kUPNgAMx3e7Vx6o= 2023-07-29T00:50:00.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SjKw3TeR8Fp3kUPNgAMx3e7Vx6o=', 'Content-Length': '0'} 2023-07-29T00:50:00.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebb92d2a65a791b6-0064c46238-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.250 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:00.251 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:00.251 DEBUG:boto:path=/hiiqos-61/?prefix=obj115 2023-07-29T00:50:00.252 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj115 2023-07-29T00:50:00.252 DEBUG:boto:Method: GET 2023-07-29T00:50:00.252 DEBUG:boto:Path: /hiiqos-61/?prefix=obj115 2023-07-29T00:50:00.253 DEBUG:boto:Data: 2023-07-29T00:50:00.253 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.253 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:00.254 DEBUG:boto:Port: 8001 2023-07-29T00:50:00.254 DEBUG:boto:Params: {} 2023-07-29T00:50:00.254 DEBUG:boto:Token: None 2023-07-29T00:50:00.255 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/ 2023-07-29T00:50:00.255 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:A5qU4OtGaFhwcdcXg0UzfjcCpTo= 2023-07-29T00:50:00.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:A5qU4OtGaFhwcdcXg0UzfjcCpTo=', 'Content-Length': '0'} 2023-07-29T00:50:00.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6eee1c7fe99543b-0064c46238-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.258 DEBUG:boto:b'hiiqos-61obj1151000falseobj1152023-07-29T00:45:02.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:00.259 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:00.259 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:00.260 DEBUG:boto:path=/hiiqos-61/?prefix=obj115 2023-07-29T00:50:00.260 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj115 2023-07-29T00:50:00.261 DEBUG:boto:Method: GET 2023-07-29T00:50:00.261 DEBUG:boto:Path: /hiiqos-61/?prefix=obj115 2023-07-29T00:50:00.261 DEBUG:boto:Data: 2023-07-29T00:50:00.262 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.262 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:00.262 DEBUG:boto:Port: 8000 2023-07-29T00:50:00.263 DEBUG:boto:Params: {} 2023-07-29T00:50:00.263 DEBUG:boto:Token: None 2023-07-29T00:50:00.263 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/ 2023-07-29T00:50:00.264 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:A5qU4OtGaFhwcdcXg0UzfjcCpTo= 2023-07-29T00:50:00.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:A5qU4OtGaFhwcdcXg0UzfjcCpTo=', 'Content-Length': '0'} 2023-07-29T00:50:00.268 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5554e4ef1becd35-0064c46238-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.269 DEBUG:boto:b'hiiqos-61obj1151000falseobj1152023-07-29T00:45:02.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:00.269 DEBUG:boto:path=/hiiqos-61/obj115 2023-07-29T00:50:00.270 DEBUG:boto:auth_path=/hiiqos-61/obj115 2023-07-29T00:50:00.270 DEBUG:boto:Method: HEAD 2023-07-29T00:50:00.270 DEBUG:boto:Path: /hiiqos-61/obj115 2023-07-29T00:50:00.271 DEBUG:boto:Data: 2023-07-29T00:50:00.271 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.271 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:00.272 DEBUG:boto:Port: 8001 2023-07-29T00:50:00.272 DEBUG:boto:Params: {} 2023-07-29T00:50:00.272 DEBUG:boto:Token: None 2023-07-29T00:50:00.273 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj115 2023-07-29T00:50:00.273 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lApnd5cGyi4GFlhaT/8TongljfA= 2023-07-29T00:50:00.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lApnd5cGyi4GFlhaT/8TongljfA=', 'Content-Length': '0'} 2023-07-29T00:50:00.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077909d829069121b-0064c46238-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.276 DEBUG:boto:path=/hiiqos-61/obj115 2023-07-29T00:50:00.276 DEBUG:boto:auth_path=/hiiqos-61/obj115 2023-07-29T00:50:00.276 DEBUG:boto:Method: HEAD 2023-07-29T00:50:00.277 DEBUG:boto:Path: /hiiqos-61/obj115 2023-07-29T00:50:00.277 DEBUG:boto:Data: 2023-07-29T00:50:00.277 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.278 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:00.278 DEBUG:boto:Port: 8000 2023-07-29T00:50:00.278 DEBUG:boto:Params: {} 2023-07-29T00:50:00.279 DEBUG:boto:Token: None 2023-07-29T00:50:00.279 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj115 2023-07-29T00:50:00.280 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lApnd5cGyi4GFlhaT/8TongljfA= 2023-07-29T00:50:00.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lApnd5cGyi4GFlhaT/8TongljfA=', 'Content-Length': '0'} 2023-07-29T00:50:00.281 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005847757895c5ff7a-0064c46238-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-07-29T00:50:00.282 DEBUG:boto:path=/hiiqos-61/obj115 2023-07-29T00:50:00.283 DEBUG:boto:auth_path=/hiiqos-61/obj115 2023-07-29T00:50:00.283 DEBUG:boto:Method: GET 2023-07-29T00:50:00.284 DEBUG:boto:Path: /hiiqos-61/obj115 2023-07-29T00:50:00.284 DEBUG:boto:Data: 2023-07-29T00:50:00.284 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.285 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:00.285 DEBUG:boto:Port: 8001 2023-07-29T00:50:00.285 DEBUG:boto:Params: {} 2023-07-29T00:50:00.286 DEBUG:boto:Token: None 2023-07-29T00:50:00.286 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj115 2023-07-29T00:50:00.287 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gc0AnDFfPy5JYdkiv3FzxdYWXQc= 2023-07-29T00:50:00.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gc0AnDFfPy5JYdkiv3FzxdYWXQc=', 'Content-Length': '0'} 2023-07-29T00:50:00.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030d61441aa4014d2-0064c46238-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.289 DEBUG:boto:path=/hiiqos-61/obj115 2023-07-29T00:50:00.290 DEBUG:boto:auth_path=/hiiqos-61/obj115 2023-07-29T00:50:00.290 DEBUG:boto:Method: GET 2023-07-29T00:50:00.290 DEBUG:boto:Path: /hiiqos-61/obj115 2023-07-29T00:50:00.291 DEBUG:boto:Data: 2023-07-29T00:50:00.291 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.292 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:00.292 DEBUG:boto:Port: 8000 2023-07-29T00:50:00.292 DEBUG:boto:Params: {} 2023-07-29T00:50:00.293 DEBUG:boto:Token: None 2023-07-29T00:50:00.293 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj115 2023-07-29T00:50:00.293 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gc0AnDFfPy5JYdkiv3FzxdYWXQc= 2023-07-29T00:50:00.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gc0AnDFfPy5JYdkiv3FzxdYWXQc=', 'Content-Length': '0'} 2023-07-29T00:50:00.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fcf02207fef9df2-0064c46238-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.336 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:00.337 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:00.337 DEBUG:boto:path=/hiiqos-61/?prefix=obj115 2023-07-29T00:50:00.337 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj115 2023-07-29T00:50:00.338 DEBUG:boto:Method: GET 2023-07-29T00:50:00.338 DEBUG:boto:Path: /hiiqos-61/?prefix=obj115 2023-07-29T00:50:00.339 DEBUG:boto:Data: 2023-07-29T00:50:00.339 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.339 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:00.340 DEBUG:boto:Port: 8001 2023-07-29T00:50:00.340 DEBUG:boto:Params: {} 2023-07-29T00:50:00.340 DEBUG:boto:Token: None 2023-07-29T00:50:00.341 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/ 2023-07-29T00:50:00.341 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:A5qU4OtGaFhwcdcXg0UzfjcCpTo= 2023-07-29T00:50:00.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:A5qU4OtGaFhwcdcXg0UzfjcCpTo=', 'Content-Length': '0'} 2023-07-29T00:50:00.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094f488f8b0c9674f-0064c46238-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.344 DEBUG:boto:b'hiiqos-61obj1151000falseobj1152023-07-29T00:45:02.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:00.345 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:00.345 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:00.346 DEBUG:boto:path=/hiiqos-61/?prefix=obj115 2023-07-29T00:50:00.346 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj115 2023-07-29T00:50:00.346 DEBUG:boto:Method: GET 2023-07-29T00:50:00.347 DEBUG:boto:Path: /hiiqos-61/?prefix=obj115 2023-07-29T00:50:00.347 DEBUG:boto:Data: 2023-07-29T00:50:00.347 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.348 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:00.348 DEBUG:boto:Port: 8000 2023-07-29T00:50:00.349 DEBUG:boto:Params: {} 2023-07-29T00:50:00.349 DEBUG:boto:Token: None 2023-07-29T00:50:00.349 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/ 2023-07-29T00:50:00.350 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:A5qU4OtGaFhwcdcXg0UzfjcCpTo= 2023-07-29T00:50:00.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:A5qU4OtGaFhwcdcXg0UzfjcCpTo=', 'Content-Length': '0'} 2023-07-29T00:50:00.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027cc11a2ee042e3c-0064c46238-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.354 DEBUG:boto:b'hiiqos-61obj1151000falseobj1152023-07-29T00:45:02.106Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:00.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-07-29T00:50:00.355 DEBUG:boto:path=/hiiqos-61/obj116 2023-07-29T00:50:00.356 DEBUG:boto:auth_path=/hiiqos-61/obj116 2023-07-29T00:50:00.356 DEBUG:boto:path=/hiiqos-61/obj116?versionId=null 2023-07-29T00:50:00.356 DEBUG:boto:auth_path=/hiiqos-61/obj116?versionId=null 2023-07-29T00:50:00.357 DEBUG:boto:Method: GET 2023-07-29T00:50:00.357 DEBUG:boto:Path: /hiiqos-61/obj116?versionId=null 2023-07-29T00:50:00.358 DEBUG:boto:Data: 2023-07-29T00:50:00.358 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.358 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:00.359 DEBUG:boto:Port: 8001 2023-07-29T00:50:00.359 DEBUG:boto:Params: {} 2023-07-29T00:50:00.360 DEBUG:boto:Token: None 2023-07-29T00:50:00.360 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj116?versionId=null 2023-07-29T00:50:00.360 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VACoKltT4abAcRfeG0iiwTQWkws= 2023-07-29T00:50:00.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VACoKltT4abAcRfeG0iiwTQWkws=', 'Content-Length': '0'} 2023-07-29T00:50:00.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c78c28426727711-0064c46238-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.363 DEBUG:boto:path=/hiiqos-61/obj116 2023-07-29T00:50:00.363 DEBUG:boto:auth_path=/hiiqos-61/obj116 2023-07-29T00:50:00.364 DEBUG:boto:path=/hiiqos-61/obj116?versionId=null 2023-07-29T00:50:00.364 DEBUG:boto:auth_path=/hiiqos-61/obj116?versionId=null 2023-07-29T00:50:00.365 DEBUG:boto:Method: GET 2023-07-29T00:50:00.365 DEBUG:boto:Path: /hiiqos-61/obj116?versionId=null 2023-07-29T00:50:00.365 DEBUG:boto:Data: 2023-07-29T00:50:00.366 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.366 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:00.366 DEBUG:boto:Port: 8000 2023-07-29T00:50:00.367 DEBUG:boto:Params: {} 2023-07-29T00:50:00.367 DEBUG:boto:Token: None 2023-07-29T00:50:00.368 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj116?versionId=null 2023-07-29T00:50:00.368 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VACoKltT4abAcRfeG0iiwTQWkws= 2023-07-29T00:50:00.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VACoKltT4abAcRfeG0iiwTQWkws=', 'Content-Length': '0'} 2023-07-29T00:50:00.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080f938384fd13805-0064c46238-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.371 DEBUG:boto:path=/hiiqos-61/obj116 2023-07-29T00:50:00.371 DEBUG:boto:auth_path=/hiiqos-61/obj116 2023-07-29T00:50:00.371 DEBUG:boto:path=/hiiqos-61/obj116?versionId=null 2023-07-29T00:50:00.372 DEBUG:boto:auth_path=/hiiqos-61/obj116?versionId=null 2023-07-29T00:50:00.372 DEBUG:boto:Method: HEAD 2023-07-29T00:50:00.373 DEBUG:boto:Path: /hiiqos-61/obj116?versionId=null 2023-07-29T00:50:00.373 DEBUG:boto:Data: 2023-07-29T00:50:00.373 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.374 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:00.374 DEBUG:boto:Port: 8001 2023-07-29T00:50:00.374 DEBUG:boto:Params: {} 2023-07-29T00:50:00.375 DEBUG:boto:Token: None 2023-07-29T00:50:00.375 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj116?versionId=null 2023-07-29T00:50:00.375 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:apzYm7U4RD3dJFVZUHDB8QLTnE4= 2023-07-29T00:50:00.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:apzYm7U4RD3dJFVZUHDB8QLTnE4=', 'Content-Length': '0'} 2023-07-29T00:50:00.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063319dcc1018aef5-0064c46238-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.378 DEBUG:boto:path=/hiiqos-61/obj116 2023-07-29T00:50:00.379 DEBUG:boto:auth_path=/hiiqos-61/obj116 2023-07-29T00:50:00.379 DEBUG:boto:path=/hiiqos-61/obj116?versionId=null 2023-07-29T00:50:00.379 DEBUG:boto:auth_path=/hiiqos-61/obj116?versionId=null 2023-07-29T00:50:00.380 DEBUG:boto:Method: HEAD 2023-07-29T00:50:00.380 DEBUG:boto:Path: /hiiqos-61/obj116?versionId=null 2023-07-29T00:50:00.380 DEBUG:boto:Data: 2023-07-29T00:50:00.381 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.381 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:00.381 DEBUG:boto:Port: 8000 2023-07-29T00:50:00.382 DEBUG:boto:Params: {} 2023-07-29T00:50:00.382 DEBUG:boto:Token: None 2023-07-29T00:50:00.382 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj116?versionId=null 2023-07-29T00:50:00.383 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:apzYm7U4RD3dJFVZUHDB8QLTnE4= 2023-07-29T00:50:00.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:apzYm7U4RD3dJFVZUHDB8QLTnE4=', 'Content-Length': '0'} 2023-07-29T00:50:00.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb13d356b8e3693b-0064c46238-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-07-29T00:50:00.386 DEBUG:boto:path=/hiiqos-61/obj116 2023-07-29T00:50:00.386 DEBUG:boto:auth_path=/hiiqos-61/obj116 2023-07-29T00:50:00.386 DEBUG:boto:path=/hiiqos-61/obj116?versionId=null 2023-07-29T00:50:00.387 DEBUG:boto:auth_path=/hiiqos-61/obj116?versionId=null 2023-07-29T00:50:00.387 DEBUG:boto:Method: GET 2023-07-29T00:50:00.387 DEBUG:boto:Path: /hiiqos-61/obj116?versionId=null 2023-07-29T00:50:00.388 DEBUG:boto:Data: 2023-07-29T00:50:00.388 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.389 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:00.389 DEBUG:boto:Port: 8001 2023-07-29T00:50:00.389 DEBUG:boto:Params: {} 2023-07-29T00:50:00.390 DEBUG:boto:Token: None 2023-07-29T00:50:00.390 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj116?versionId=null 2023-07-29T00:50:00.391 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VACoKltT4abAcRfeG0iiwTQWkws= 2023-07-29T00:50:00.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VACoKltT4abAcRfeG0iiwTQWkws=', 'Content-Length': '0'} 2023-07-29T00:50:00.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c51bf508489eaa7-0064c46238-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.393 DEBUG:boto:path=/hiiqos-61/obj116 2023-07-29T00:50:00.394 DEBUG:boto:auth_path=/hiiqos-61/obj116 2023-07-29T00:50:00.394 DEBUG:boto:path=/hiiqos-61/obj116?versionId=null 2023-07-29T00:50:00.395 DEBUG:boto:auth_path=/hiiqos-61/obj116?versionId=null 2023-07-29T00:50:00.395 DEBUG:boto:Method: GET 2023-07-29T00:50:00.395 DEBUG:boto:Path: /hiiqos-61/obj116?versionId=null 2023-07-29T00:50:00.396 DEBUG:boto:Data: 2023-07-29T00:50:00.396 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.396 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:00.397 DEBUG:boto:Port: 8000 2023-07-29T00:50:00.397 DEBUG:boto:Params: {} 2023-07-29T00:50:00.398 DEBUG:boto:Token: None 2023-07-29T00:50:00.398 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj116?versionId=null 2023-07-29T00:50:00.398 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VACoKltT4abAcRfeG0iiwTQWkws= 2023-07-29T00:50:00.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VACoKltT4abAcRfeG0iiwTQWkws=', 'Content-Length': '0'} 2023-07-29T00:50:00.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050d73246fcf0370a-0064c46238-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.444 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:00.445 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:00.445 DEBUG:boto:path=/hiiqos-61/?prefix=obj116 2023-07-29T00:50:00.445 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj116 2023-07-29T00:50:00.446 DEBUG:boto:Method: GET 2023-07-29T00:50:00.446 DEBUG:boto:Path: /hiiqos-61/?prefix=obj116 2023-07-29T00:50:00.447 DEBUG:boto:Data: 2023-07-29T00:50:00.447 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.447 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:00.448 DEBUG:boto:Port: 8001 2023-07-29T00:50:00.448 DEBUG:boto:Params: {} 2023-07-29T00:50:00.448 DEBUG:boto:Token: None 2023-07-29T00:50:00.449 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/ 2023-07-29T00:50:00.449 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:A5qU4OtGaFhwcdcXg0UzfjcCpTo= 2023-07-29T00:50:00.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:A5qU4OtGaFhwcdcXg0UzfjcCpTo=', 'Content-Length': '0'} 2023-07-29T00:50:00.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a641f709d07dfcf8-0064c46238-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.453 DEBUG:boto:b'hiiqos-61obj1161000falseobj1162023-07-29T00:45:02.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:00.453 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:00.454 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:00.454 DEBUG:boto:path=/hiiqos-61/?prefix=obj116 2023-07-29T00:50:00.454 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj116 2023-07-29T00:50:00.455 DEBUG:boto:Method: GET 2023-07-29T00:50:00.455 DEBUG:boto:Path: /hiiqos-61/?prefix=obj116 2023-07-29T00:50:00.456 DEBUG:boto:Data: 2023-07-29T00:50:00.456 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.456 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:00.457 DEBUG:boto:Port: 8000 2023-07-29T00:50:00.457 DEBUG:boto:Params: {} 2023-07-29T00:50:00.457 DEBUG:boto:Token: None 2023-07-29T00:50:00.458 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/ 2023-07-29T00:50:00.458 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:A5qU4OtGaFhwcdcXg0UzfjcCpTo= 2023-07-29T00:50:00.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:A5qU4OtGaFhwcdcXg0UzfjcCpTo=', 'Content-Length': '0'} 2023-07-29T00:50:00.462 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000478bf2c34baf4559-0064c46238-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.463 DEBUG:boto:b'hiiqos-61obj1161000falseobj1162023-07-29T00:45:02.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:00.463 DEBUG:boto:path=/hiiqos-61/obj116 2023-07-29T00:50:00.464 DEBUG:boto:auth_path=/hiiqos-61/obj116 2023-07-29T00:50:00.464 DEBUG:boto:Method: HEAD 2023-07-29T00:50:00.465 DEBUG:boto:Path: /hiiqos-61/obj116 2023-07-29T00:50:00.465 DEBUG:boto:Data: 2023-07-29T00:50:00.465 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.466 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:00.466 DEBUG:boto:Port: 8001 2023-07-29T00:50:00.466 DEBUG:boto:Params: {} 2023-07-29T00:50:00.467 DEBUG:boto:Token: None 2023-07-29T00:50:00.467 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj116 2023-07-29T00:50:00.467 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eoSohFfUmIksb6Rf8pxPDxtIgG8= 2023-07-29T00:50:00.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eoSohFfUmIksb6Rf8pxPDxtIgG8=', 'Content-Length': '0'} 2023-07-29T00:50:00.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c0a1a25a64f1a86-0064c46238-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.470 DEBUG:boto:path=/hiiqos-61/obj116 2023-07-29T00:50:00.470 DEBUG:boto:auth_path=/hiiqos-61/obj116 2023-07-29T00:50:00.471 DEBUG:boto:Method: HEAD 2023-07-29T00:50:00.471 DEBUG:boto:Path: /hiiqos-61/obj116 2023-07-29T00:50:00.471 DEBUG:boto:Data: 2023-07-29T00:50:00.472 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.472 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:00.472 DEBUG:boto:Port: 8000 2023-07-29T00:50:00.473 DEBUG:boto:Params: {} 2023-07-29T00:50:00.473 DEBUG:boto:Token: None 2023-07-29T00:50:00.474 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj116 2023-07-29T00:50:00.474 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eoSohFfUmIksb6Rf8pxPDxtIgG8= 2023-07-29T00:50:00.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eoSohFfUmIksb6Rf8pxPDxtIgG8=', 'Content-Length': '0'} 2023-07-29T00:50:00.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000363f79c1a9a83305-0064c46238-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-07-29T00:50:00.477 DEBUG:boto:path=/hiiqos-61/obj116 2023-07-29T00:50:00.478 DEBUG:boto:auth_path=/hiiqos-61/obj116 2023-07-29T00:50:00.478 DEBUG:boto:Method: GET 2023-07-29T00:50:00.478 DEBUG:boto:Path: /hiiqos-61/obj116 2023-07-29T00:50:00.479 DEBUG:boto:Data: 2023-07-29T00:50:00.479 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.480 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:00.480 DEBUG:boto:Port: 8001 2023-07-29T00:50:00.480 DEBUG:boto:Params: {} 2023-07-29T00:50:00.481 DEBUG:boto:Token: None 2023-07-29T00:50:00.481 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj116 2023-07-29T00:50:00.481 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xTBtiGzNSFTv8C178voWOEAcxxs= 2023-07-29T00:50:00.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xTBtiGzNSFTv8C178voWOEAcxxs=', 'Content-Length': '0'} 2023-07-29T00:50:00.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbac5ba092b8ff4c-0064c46238-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.484 DEBUG:boto:path=/hiiqos-61/obj116 2023-07-29T00:50:00.485 DEBUG:boto:auth_path=/hiiqos-61/obj116 2023-07-29T00:50:00.485 DEBUG:boto:Method: GET 2023-07-29T00:50:00.485 DEBUG:boto:Path: /hiiqos-61/obj116 2023-07-29T00:50:00.486 DEBUG:boto:Data: 2023-07-29T00:50:00.486 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.486 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:00.487 DEBUG:boto:Port: 8000 2023-07-29T00:50:00.487 DEBUG:boto:Params: {} 2023-07-29T00:50:00.487 DEBUG:boto:Token: None 2023-07-29T00:50:00.488 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj116 2023-07-29T00:50:00.488 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xTBtiGzNSFTv8C178voWOEAcxxs= 2023-07-29T00:50:00.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xTBtiGzNSFTv8C178voWOEAcxxs=', 'Content-Length': '0'} 2023-07-29T00:50:00.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f16356ee1c36b488-0064c46238-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.532 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:00.533 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:00.533 DEBUG:boto:path=/hiiqos-61/?prefix=obj116 2023-07-29T00:50:00.533 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj116 2023-07-29T00:50:00.534 DEBUG:boto:Method: GET 2023-07-29T00:50:00.534 DEBUG:boto:Path: /hiiqos-61/?prefix=obj116 2023-07-29T00:50:00.534 DEBUG:boto:Data: 2023-07-29T00:50:00.535 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.535 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:00.536 DEBUG:boto:Port: 8001 2023-07-29T00:50:00.536 DEBUG:boto:Params: {} 2023-07-29T00:50:00.536 DEBUG:boto:Token: None 2023-07-29T00:50:00.537 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/ 2023-07-29T00:50:00.537 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:A5qU4OtGaFhwcdcXg0UzfjcCpTo= 2023-07-29T00:50:00.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:A5qU4OtGaFhwcdcXg0UzfjcCpTo=', 'Content-Length': '0'} 2023-07-29T00:50:00.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd4976dfcf13cc19-0064c46238-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.541 DEBUG:boto:b'hiiqos-61obj1161000falseobj1162023-07-29T00:45:02.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:00.541 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:00.542 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:00.542 DEBUG:boto:path=/hiiqos-61/?prefix=obj116 2023-07-29T00:50:00.543 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj116 2023-07-29T00:50:00.543 DEBUG:boto:Method: GET 2023-07-29T00:50:00.543 DEBUG:boto:Path: /hiiqos-61/?prefix=obj116 2023-07-29T00:50:00.544 DEBUG:boto:Data: 2023-07-29T00:50:00.544 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.544 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:00.545 DEBUG:boto:Port: 8000 2023-07-29T00:50:00.545 DEBUG:boto:Params: {} 2023-07-29T00:50:00.546 DEBUG:boto:Token: None 2023-07-29T00:50:00.546 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/ 2023-07-29T00:50:00.547 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:A5qU4OtGaFhwcdcXg0UzfjcCpTo= 2023-07-29T00:50:00.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:A5qU4OtGaFhwcdcXg0UzfjcCpTo=', 'Content-Length': '0'} 2023-07-29T00:50:00.551 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093b4f90b4dc342e0-0064c46238-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.551 DEBUG:boto:b'hiiqos-61obj1161000falseobj1162023-07-29T00:45:02.393Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:00.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-07-29T00:50:00.552 DEBUG:boto:path=/hiiqos-61/obj124 2023-07-29T00:50:00.553 DEBUG:boto:auth_path=/hiiqos-61/obj124 2023-07-29T00:50:00.553 DEBUG:boto:path=/hiiqos-61/obj124?versionId=null 2023-07-29T00:50:00.554 DEBUG:boto:auth_path=/hiiqos-61/obj124?versionId=null 2023-07-29T00:50:00.554 DEBUG:boto:Method: GET 2023-07-29T00:50:00.554 DEBUG:boto:Path: /hiiqos-61/obj124?versionId=null 2023-07-29T00:50:00.555 DEBUG:boto:Data: 2023-07-29T00:50:00.555 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.555 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:00.556 DEBUG:boto:Port: 8001 2023-07-29T00:50:00.556 DEBUG:boto:Params: {} 2023-07-29T00:50:00.556 DEBUG:boto:Token: None 2023-07-29T00:50:00.557 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj124?versionId=null 2023-07-29T00:50:00.557 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fWJpz3nHKqSnRomiUPZkdfAvJ1M= 2023-07-29T00:50:00.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fWJpz3nHKqSnRomiUPZkdfAvJ1M=', 'Content-Length': '0'} 2023-07-29T00:50:00.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca01ae0c408fd329-0064c46238-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.600 DEBUG:boto:path=/hiiqos-61/obj124 2023-07-29T00:50:00.601 DEBUG:boto:auth_path=/hiiqos-61/obj124 2023-07-29T00:50:00.601 DEBUG:boto:path=/hiiqos-61/obj124?versionId=null 2023-07-29T00:50:00.601 DEBUG:boto:auth_path=/hiiqos-61/obj124?versionId=null 2023-07-29T00:50:00.602 DEBUG:boto:Method: GET 2023-07-29T00:50:00.602 DEBUG:boto:Path: /hiiqos-61/obj124?versionId=null 2023-07-29T00:50:00.602 DEBUG:boto:Data: 2023-07-29T00:50:00.603 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.603 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:00.604 DEBUG:boto:Port: 8000 2023-07-29T00:50:00.604 DEBUG:boto:Params: {} 2023-07-29T00:50:00.604 DEBUG:boto:Token: None 2023-07-29T00:50:00.605 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj124?versionId=null 2023-07-29T00:50:00.605 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fWJpz3nHKqSnRomiUPZkdfAvJ1M= 2023-07-29T00:50:00.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fWJpz3nHKqSnRomiUPZkdfAvJ1M=', 'Content-Length': '0'} 2023-07-29T00:50:00.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e961bc9879e828b-0064c46238-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.608 DEBUG:boto:path=/hiiqos-61/obj124 2023-07-29T00:50:00.609 DEBUG:boto:auth_path=/hiiqos-61/obj124 2023-07-29T00:50:00.609 DEBUG:boto:path=/hiiqos-61/obj124?versionId=null 2023-07-29T00:50:00.609 DEBUG:boto:auth_path=/hiiqos-61/obj124?versionId=null 2023-07-29T00:50:00.610 DEBUG:boto:Method: HEAD 2023-07-29T00:50:00.610 DEBUG:boto:Path: /hiiqos-61/obj124?versionId=null 2023-07-29T00:50:00.610 DEBUG:boto:Data: 2023-07-29T00:50:00.611 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.611 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:00.611 DEBUG:boto:Port: 8001 2023-07-29T00:50:00.612 DEBUG:boto:Params: {} 2023-07-29T00:50:00.612 DEBUG:boto:Token: None 2023-07-29T00:50:00.613 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj124?versionId=null 2023-07-29T00:50:00.613 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ENHuvsXyfVx+TeNGmtQzSvRx9uI= 2023-07-29T00:50:00.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ENHuvsXyfVx+TeNGmtQzSvRx9uI=', 'Content-Length': '0'} 2023-07-29T00:50:00.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb25c83c8e6637da-0064c46238-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.616 DEBUG:boto:path=/hiiqos-61/obj124 2023-07-29T00:50:00.616 DEBUG:boto:auth_path=/hiiqos-61/obj124 2023-07-29T00:50:00.616 DEBUG:boto:path=/hiiqos-61/obj124?versionId=null 2023-07-29T00:50:00.617 DEBUG:boto:auth_path=/hiiqos-61/obj124?versionId=null 2023-07-29T00:50:00.617 DEBUG:boto:Method: HEAD 2023-07-29T00:50:00.617 DEBUG:boto:Path: /hiiqos-61/obj124?versionId=null 2023-07-29T00:50:00.618 DEBUG:boto:Data: 2023-07-29T00:50:00.618 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.618 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:00.619 DEBUG:boto:Port: 8000 2023-07-29T00:50:00.619 DEBUG:boto:Params: {} 2023-07-29T00:50:00.619 DEBUG:boto:Token: None 2023-07-29T00:50:00.620 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj124?versionId=null 2023-07-29T00:50:00.620 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ENHuvsXyfVx+TeNGmtQzSvRx9uI= 2023-07-29T00:50:00.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ENHuvsXyfVx+TeNGmtQzSvRx9uI=', 'Content-Length': '0'} 2023-07-29T00:50:00.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043fcfc669c4ab576-0064c46238-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-07-29T00:50:00.623 DEBUG:boto:path=/hiiqos-61/obj124 2023-07-29T00:50:00.624 DEBUG:boto:auth_path=/hiiqos-61/obj124 2023-07-29T00:50:00.624 DEBUG:boto:path=/hiiqos-61/obj124?versionId=null 2023-07-29T00:50:00.624 DEBUG:boto:auth_path=/hiiqos-61/obj124?versionId=null 2023-07-29T00:50:00.625 DEBUG:boto:Method: GET 2023-07-29T00:50:00.625 DEBUG:boto:Path: /hiiqos-61/obj124?versionId=null 2023-07-29T00:50:00.625 DEBUG:boto:Data: 2023-07-29T00:50:00.626 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.626 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:00.626 DEBUG:boto:Port: 8001 2023-07-29T00:50:00.627 DEBUG:boto:Params: {} 2023-07-29T00:50:00.627 DEBUG:boto:Token: None 2023-07-29T00:50:00.628 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj124?versionId=null 2023-07-29T00:50:00.628 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fWJpz3nHKqSnRomiUPZkdfAvJ1M= 2023-07-29T00:50:00.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fWJpz3nHKqSnRomiUPZkdfAvJ1M=', 'Content-Length': '0'} 2023-07-29T00:50:00.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efca068331beb2f6-0064c46238-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.631 DEBUG:boto:path=/hiiqos-61/obj124 2023-07-29T00:50:00.631 DEBUG:boto:auth_path=/hiiqos-61/obj124 2023-07-29T00:50:00.632 DEBUG:boto:path=/hiiqos-61/obj124?versionId=null 2023-07-29T00:50:00.632 DEBUG:boto:auth_path=/hiiqos-61/obj124?versionId=null 2023-07-29T00:50:00.632 DEBUG:boto:Method: GET 2023-07-29T00:50:00.633 DEBUG:boto:Path: /hiiqos-61/obj124?versionId=null 2023-07-29T00:50:00.633 DEBUG:boto:Data: 2023-07-29T00:50:00.633 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.634 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:00.634 DEBUG:boto:Port: 8000 2023-07-29T00:50:00.635 DEBUG:boto:Params: {} 2023-07-29T00:50:00.635 DEBUG:boto:Token: None 2023-07-29T00:50:00.635 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj124?versionId=null 2023-07-29T00:50:00.636 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fWJpz3nHKqSnRomiUPZkdfAvJ1M= 2023-07-29T00:50:00.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fWJpz3nHKqSnRomiUPZkdfAvJ1M=', 'Content-Length': '0'} 2023-07-29T00:50:00.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d923f3b72a1ec8c3-0064c46238-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.639 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:00.639 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:00.639 DEBUG:boto:path=/hiiqos-61/?prefix=obj124 2023-07-29T00:50:00.640 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj124 2023-07-29T00:50:00.640 DEBUG:boto:Method: GET 2023-07-29T00:50:00.640 DEBUG:boto:Path: /hiiqos-61/?prefix=obj124 2023-07-29T00:50:00.641 DEBUG:boto:Data: 2023-07-29T00:50:00.641 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.642 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:00.642 DEBUG:boto:Port: 8001 2023-07-29T00:50:00.642 DEBUG:boto:Params: {} 2023-07-29T00:50:00.643 DEBUG:boto:Token: None 2023-07-29T00:50:00.643 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/ 2023-07-29T00:50:00.643 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:A5qU4OtGaFhwcdcXg0UzfjcCpTo= 2023-07-29T00:50:00.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:A5qU4OtGaFhwcdcXg0UzfjcCpTo=', 'Content-Length': '0'} 2023-07-29T00:50:00.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6a42b894dbebac7-0064c46238-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.647 DEBUG:boto:b'hiiqos-61obj1241000falseobj1242023-07-29T00:45:02.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:00.647 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:00.648 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:00.648 DEBUG:boto:path=/hiiqos-61/?prefix=obj124 2023-07-29T00:50:00.648 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj124 2023-07-29T00:50:00.649 DEBUG:boto:Method: GET 2023-07-29T00:50:00.649 DEBUG:boto:Path: /hiiqos-61/?prefix=obj124 2023-07-29T00:50:00.650 DEBUG:boto:Data: 2023-07-29T00:50:00.650 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.650 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:00.651 DEBUG:boto:Port: 8000 2023-07-29T00:50:00.651 DEBUG:boto:Params: {} 2023-07-29T00:50:00.651 DEBUG:boto:Token: None 2023-07-29T00:50:00.652 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/ 2023-07-29T00:50:00.652 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:A5qU4OtGaFhwcdcXg0UzfjcCpTo= 2023-07-29T00:50:00.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:A5qU4OtGaFhwcdcXg0UzfjcCpTo=', 'Content-Length': '0'} 2023-07-29T00:50:00.656 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d2040e7ec8be7c6-0064c46238-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.657 DEBUG:boto:b'hiiqos-61obj1241000falseobj1242023-07-29T00:45:02.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:00.658 DEBUG:boto:path=/hiiqos-61/obj124 2023-07-29T00:50:00.658 DEBUG:boto:auth_path=/hiiqos-61/obj124 2023-07-29T00:50:00.658 DEBUG:boto:Method: HEAD 2023-07-29T00:50:00.659 DEBUG:boto:Path: /hiiqos-61/obj124 2023-07-29T00:50:00.659 DEBUG:boto:Data: 2023-07-29T00:50:00.659 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.660 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:00.660 DEBUG:boto:Port: 8001 2023-07-29T00:50:00.660 DEBUG:boto:Params: {} 2023-07-29T00:50:00.661 DEBUG:boto:Token: None 2023-07-29T00:50:00.661 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj124 2023-07-29T00:50:00.661 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pjhKY/Howqd7IeTq9UhoRsIcKBw= 2023-07-29T00:50:00.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pjhKY/Howqd7IeTq9UhoRsIcKBw=', 'Content-Length': '0'} 2023-07-29T00:50:00.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9e25203eb667c9a-0064c46238-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.664 DEBUG:boto:path=/hiiqos-61/obj124 2023-07-29T00:50:00.665 DEBUG:boto:auth_path=/hiiqos-61/obj124 2023-07-29T00:50:00.665 DEBUG:boto:Method: HEAD 2023-07-29T00:50:00.665 DEBUG:boto:Path: /hiiqos-61/obj124 2023-07-29T00:50:00.666 DEBUG:boto:Data: 2023-07-29T00:50:00.666 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.666 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:00.667 DEBUG:boto:Port: 8000 2023-07-29T00:50:00.667 DEBUG:boto:Params: {} 2023-07-29T00:50:00.667 DEBUG:boto:Token: None 2023-07-29T00:50:00.668 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj124 2023-07-29T00:50:00.668 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pjhKY/Howqd7IeTq9UhoRsIcKBw= 2023-07-29T00:50:00.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pjhKY/Howqd7IeTq9UhoRsIcKBw=', 'Content-Length': '0'} 2023-07-29T00:50:00.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c653f6904b97b50-0064c46238-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-07-29T00:50:00.671 DEBUG:boto:path=/hiiqos-61/obj124 2023-07-29T00:50:00.671 DEBUG:boto:auth_path=/hiiqos-61/obj124 2023-07-29T00:50:00.672 DEBUG:boto:Method: GET 2023-07-29T00:50:00.672 DEBUG:boto:Path: /hiiqos-61/obj124 2023-07-29T00:50:00.672 DEBUG:boto:Data: 2023-07-29T00:50:00.673 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.673 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:00.673 DEBUG:boto:Port: 8001 2023-07-29T00:50:00.674 DEBUG:boto:Params: {} 2023-07-29T00:50:00.674 DEBUG:boto:Token: None 2023-07-29T00:50:00.675 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj124 2023-07-29T00:50:00.675 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BwBaZ/GiN5WF+kDFdG/JlRJJPV8= 2023-07-29T00:50:00.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BwBaZ/GiN5WF+kDFdG/JlRJJPV8=', 'Content-Length': '0'} 2023-07-29T00:50:00.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce3bf0dc4997d945-0064c46238-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.678 DEBUG:boto:path=/hiiqos-61/obj124 2023-07-29T00:50:00.678 DEBUG:boto:auth_path=/hiiqos-61/obj124 2023-07-29T00:50:00.678 DEBUG:boto:Method: GET 2023-07-29T00:50:00.679 DEBUG:boto:Path: /hiiqos-61/obj124 2023-07-29T00:50:00.679 DEBUG:boto:Data: 2023-07-29T00:50:00.680 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.680 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:00.680 DEBUG:boto:Port: 8000 2023-07-29T00:50:00.681 DEBUG:boto:Params: {} 2023-07-29T00:50:00.681 DEBUG:boto:Token: None 2023-07-29T00:50:00.681 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj124 2023-07-29T00:50:00.682 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BwBaZ/GiN5WF+kDFdG/JlRJJPV8= 2023-07-29T00:50:00.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BwBaZ/GiN5WF+kDFdG/JlRJJPV8=', 'Content-Length': '0'} 2023-07-29T00:50:00.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f0a334b3132d7d0-0064c46238-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.685 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:00.685 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:00.685 DEBUG:boto:path=/hiiqos-61/?prefix=obj124 2023-07-29T00:50:00.686 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj124 2023-07-29T00:50:00.686 DEBUG:boto:Method: GET 2023-07-29T00:50:00.686 DEBUG:boto:Path: /hiiqos-61/?prefix=obj124 2023-07-29T00:50:00.687 DEBUG:boto:Data: 2023-07-29T00:50:00.687 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.688 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:00.688 DEBUG:boto:Port: 8001 2023-07-29T00:50:00.688 DEBUG:boto:Params: {} 2023-07-29T00:50:00.689 DEBUG:boto:Token: None 2023-07-29T00:50:00.689 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/ 2023-07-29T00:50:00.690 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:A5qU4OtGaFhwcdcXg0UzfjcCpTo= 2023-07-29T00:50:00.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:A5qU4OtGaFhwcdcXg0UzfjcCpTo=', 'Content-Length': '0'} 2023-07-29T00:50:00.692 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000831039bc303e3652-0064c46238-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.692 DEBUG:boto:b'hiiqos-61obj1241000falseobj1242023-07-29T00:45:02.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:00.693 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:00.693 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:00.694 DEBUG:boto:path=/hiiqos-61/?prefix=obj124 2023-07-29T00:50:00.694 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj124 2023-07-29T00:50:00.695 DEBUG:boto:Method: GET 2023-07-29T00:50:00.695 DEBUG:boto:Path: /hiiqos-61/?prefix=obj124 2023-07-29T00:50:00.695 DEBUG:boto:Data: 2023-07-29T00:50:00.696 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.696 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:00.696 DEBUG:boto:Port: 8000 2023-07-29T00:50:00.697 DEBUG:boto:Params: {} 2023-07-29T00:50:00.697 DEBUG:boto:Token: None 2023-07-29T00:50:00.697 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/ 2023-07-29T00:50:00.698 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:A5qU4OtGaFhwcdcXg0UzfjcCpTo= 2023-07-29T00:50:00.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:A5qU4OtGaFhwcdcXg0UzfjcCpTo=', 'Content-Length': '0'} 2023-07-29T00:50:00.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000252f09e0d1a765d4-0064c46238-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.744 DEBUG:boto:b'hiiqos-61obj1241000falseobj1242023-07-29T00:45:02.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:00.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-07-29T00:50:00.745 DEBUG:boto:path=/hiiqos-61/obj145 2023-07-29T00:50:00.746 DEBUG:boto:auth_path=/hiiqos-61/obj145 2023-07-29T00:50:00.746 DEBUG:boto:path=/hiiqos-61/obj145?versionId=null 2023-07-29T00:50:00.746 DEBUG:boto:auth_path=/hiiqos-61/obj145?versionId=null 2023-07-29T00:50:00.747 DEBUG:boto:Method: GET 2023-07-29T00:50:00.747 DEBUG:boto:Path: /hiiqos-61/obj145?versionId=null 2023-07-29T00:50:00.747 DEBUG:boto:Data: 2023-07-29T00:50:00.748 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.748 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:00.748 DEBUG:boto:Port: 8001 2023-07-29T00:50:00.749 DEBUG:boto:Params: {} 2023-07-29T00:50:00.749 DEBUG:boto:Token: None 2023-07-29T00:50:00.750 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj145?versionId=null 2023-07-29T00:50:00.750 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wPmwSB89vG6jH23JHbMGya8i/Cw= 2023-07-29T00:50:00.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wPmwSB89vG6jH23JHbMGya8i/Cw=', 'Content-Length': '0'} 2023-07-29T00:50:00.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1471c6a021c2dda-0064c46238-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.753 DEBUG:boto:path=/hiiqos-61/obj145 2023-07-29T00:50:00.754 DEBUG:boto:auth_path=/hiiqos-61/obj145 2023-07-29T00:50:00.754 DEBUG:boto:path=/hiiqos-61/obj145?versionId=null 2023-07-29T00:50:00.754 DEBUG:boto:auth_path=/hiiqos-61/obj145?versionId=null 2023-07-29T00:50:00.755 DEBUG:boto:Method: GET 2023-07-29T00:50:00.755 DEBUG:boto:Path: /hiiqos-61/obj145?versionId=null 2023-07-29T00:50:00.755 DEBUG:boto:Data: 2023-07-29T00:50:00.756 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.756 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:00.756 DEBUG:boto:Port: 8000 2023-07-29T00:50:00.757 DEBUG:boto:Params: {} 2023-07-29T00:50:00.757 DEBUG:boto:Token: None 2023-07-29T00:50:00.758 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj145?versionId=null 2023-07-29T00:50:00.758 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wPmwSB89vG6jH23JHbMGya8i/Cw= 2023-07-29T00:50:00.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wPmwSB89vG6jH23JHbMGya8i/Cw=', 'Content-Length': '0'} 2023-07-29T00:50:00.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073c3c40ca3ead019-0064c46238-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.761 DEBUG:boto:path=/hiiqos-61/obj145 2023-07-29T00:50:00.761 DEBUG:boto:auth_path=/hiiqos-61/obj145 2023-07-29T00:50:00.762 DEBUG:boto:path=/hiiqos-61/obj145?versionId=null 2023-07-29T00:50:00.762 DEBUG:boto:auth_path=/hiiqos-61/obj145?versionId=null 2023-07-29T00:50:00.762 DEBUG:boto:Method: HEAD 2023-07-29T00:50:00.763 DEBUG:boto:Path: /hiiqos-61/obj145?versionId=null 2023-07-29T00:50:00.763 DEBUG:boto:Data: 2023-07-29T00:50:00.763 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.764 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:00.764 DEBUG:boto:Port: 8001 2023-07-29T00:50:00.764 DEBUG:boto:Params: {} 2023-07-29T00:50:00.765 DEBUG:boto:Token: None 2023-07-29T00:50:00.765 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj145?versionId=null 2023-07-29T00:50:00.766 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gUZrlsst1lDNeJUSiiSD59q+2E4= 2023-07-29T00:50:00.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gUZrlsst1lDNeJUSiiSD59q+2E4=', 'Content-Length': '0'} 2023-07-29T00:50:00.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e37229ffaa819dca-0064c46238-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.768 DEBUG:boto:path=/hiiqos-61/obj145 2023-07-29T00:50:00.769 DEBUG:boto:auth_path=/hiiqos-61/obj145 2023-07-29T00:50:00.769 DEBUG:boto:path=/hiiqos-61/obj145?versionId=null 2023-07-29T00:50:00.769 DEBUG:boto:auth_path=/hiiqos-61/obj145?versionId=null 2023-07-29T00:50:00.770 DEBUG:boto:Method: HEAD 2023-07-29T00:50:00.770 DEBUG:boto:Path: /hiiqos-61/obj145?versionId=null 2023-07-29T00:50:00.770 DEBUG:boto:Data: 2023-07-29T00:50:00.771 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.771 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:00.771 DEBUG:boto:Port: 8000 2023-07-29T00:50:00.772 DEBUG:boto:Params: {} 2023-07-29T00:50:00.772 DEBUG:boto:Token: None 2023-07-29T00:50:00.772 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj145?versionId=null 2023-07-29T00:50:00.773 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gUZrlsst1lDNeJUSiiSD59q+2E4= 2023-07-29T00:50:00.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gUZrlsst1lDNeJUSiiSD59q+2E4=', 'Content-Length': '0'} 2023-07-29T00:50:00.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008359ce1b6c6fb713-0064c46238-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-07-29T00:50:00.776 DEBUG:boto:path=/hiiqos-61/obj145 2023-07-29T00:50:00.776 DEBUG:boto:auth_path=/hiiqos-61/obj145 2023-07-29T00:50:00.777 DEBUG:boto:path=/hiiqos-61/obj145?versionId=null 2023-07-29T00:50:00.777 DEBUG:boto:auth_path=/hiiqos-61/obj145?versionId=null 2023-07-29T00:50:00.777 DEBUG:boto:Method: GET 2023-07-29T00:50:00.778 DEBUG:boto:Path: /hiiqos-61/obj145?versionId=null 2023-07-29T00:50:00.778 DEBUG:boto:Data: 2023-07-29T00:50:00.778 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.779 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:00.779 DEBUG:boto:Port: 8001 2023-07-29T00:50:00.779 DEBUG:boto:Params: {} 2023-07-29T00:50:00.780 DEBUG:boto:Token: None 2023-07-29T00:50:00.780 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj145?versionId=null 2023-07-29T00:50:00.781 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wPmwSB89vG6jH23JHbMGya8i/Cw= 2023-07-29T00:50:00.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wPmwSB89vG6jH23JHbMGya8i/Cw=', 'Content-Length': '0'} 2023-07-29T00:50:00.783 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bedf2f1ae1a13fc7-0064c46238-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.824 DEBUG:boto:path=/hiiqos-61/obj145 2023-07-29T00:50:00.825 DEBUG:boto:auth_path=/hiiqos-61/obj145 2023-07-29T00:50:00.825 DEBUG:boto:path=/hiiqos-61/obj145?versionId=null 2023-07-29T00:50:00.825 DEBUG:boto:auth_path=/hiiqos-61/obj145?versionId=null 2023-07-29T00:50:00.826 DEBUG:boto:Method: GET 2023-07-29T00:50:00.826 DEBUG:boto:Path: /hiiqos-61/obj145?versionId=null 2023-07-29T00:50:00.826 DEBUG:boto:Data: 2023-07-29T00:50:00.827 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.827 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:00.828 DEBUG:boto:Port: 8000 2023-07-29T00:50:00.828 DEBUG:boto:Params: {} 2023-07-29T00:50:00.828 DEBUG:boto:Token: None 2023-07-29T00:50:00.829 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj145?versionId=null 2023-07-29T00:50:00.829 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:wPmwSB89vG6jH23JHbMGya8i/Cw= 2023-07-29T00:50:00.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wPmwSB89vG6jH23JHbMGya8i/Cw=', 'Content-Length': '0'} 2023-07-29T00:50:00.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c19747a35f66752d-0064c46238-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.832 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:00.832 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:00.833 DEBUG:boto:path=/hiiqos-61/?prefix=obj145 2023-07-29T00:50:00.833 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj145 2023-07-29T00:50:00.833 DEBUG:boto:Method: GET 2023-07-29T00:50:00.834 DEBUG:boto:Path: /hiiqos-61/?prefix=obj145 2023-07-29T00:50:00.834 DEBUG:boto:Data: 2023-07-29T00:50:00.834 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.835 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:00.835 DEBUG:boto:Port: 8001 2023-07-29T00:50:00.835 DEBUG:boto:Params: {} 2023-07-29T00:50:00.836 DEBUG:boto:Token: None 2023-07-29T00:50:00.836 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/ 2023-07-29T00:50:00.837 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:A5qU4OtGaFhwcdcXg0UzfjcCpTo= 2023-07-29T00:50:00.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:A5qU4OtGaFhwcdcXg0UzfjcCpTo=', 'Content-Length': '0'} 2023-07-29T00:50:00.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006835e1a4b20c1a2f-0064c46238-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.840 DEBUG:boto:b'hiiqos-61obj1451000falseobj1452023-07-29T00:45:02.452Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:00.841 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:00.841 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:00.841 DEBUG:boto:path=/hiiqos-61/?prefix=obj145 2023-07-29T00:50:00.842 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj145 2023-07-29T00:50:00.842 DEBUG:boto:Method: GET 2023-07-29T00:50:00.842 DEBUG:boto:Path: /hiiqos-61/?prefix=obj145 2023-07-29T00:50:00.843 DEBUG:boto:Data: 2023-07-29T00:50:00.843 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.844 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:00.844 DEBUG:boto:Port: 8000 2023-07-29T00:50:00.844 DEBUG:boto:Params: {} 2023-07-29T00:50:00.845 DEBUG:boto:Token: None 2023-07-29T00:50:00.845 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/ 2023-07-29T00:50:00.845 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:A5qU4OtGaFhwcdcXg0UzfjcCpTo= 2023-07-29T00:50:00.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:A5qU4OtGaFhwcdcXg0UzfjcCpTo=', 'Content-Length': '0'} 2023-07-29T00:50:00.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae00035e4dbd147c-0064c46238-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.850 DEBUG:boto:b'hiiqos-61obj1451000falseobj1452023-07-29T00:45:02.452Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:00.851 DEBUG:boto:path=/hiiqos-61/obj145 2023-07-29T00:50:00.851 DEBUG:boto:auth_path=/hiiqos-61/obj145 2023-07-29T00:50:00.851 DEBUG:boto:Method: HEAD 2023-07-29T00:50:00.852 DEBUG:boto:Path: /hiiqos-61/obj145 2023-07-29T00:50:00.852 DEBUG:boto:Data: 2023-07-29T00:50:00.852 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.853 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:00.853 DEBUG:boto:Port: 8001 2023-07-29T00:50:00.853 DEBUG:boto:Params: {} 2023-07-29T00:50:00.854 DEBUG:boto:Token: None 2023-07-29T00:50:00.854 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj145 2023-07-29T00:50:00.855 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GwlmbZwVruhe66063Jv78QmqG30= 2023-07-29T00:50:00.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GwlmbZwVruhe66063Jv78QmqG30=', 'Content-Length': '0'} 2023-07-29T00:50:00.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008db3e994868adf51-0064c46238-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.857 DEBUG:boto:path=/hiiqos-61/obj145 2023-07-29T00:50:00.858 DEBUG:boto:auth_path=/hiiqos-61/obj145 2023-07-29T00:50:00.858 DEBUG:boto:Method: HEAD 2023-07-29T00:50:00.858 DEBUG:boto:Path: /hiiqos-61/obj145 2023-07-29T00:50:00.859 DEBUG:boto:Data: 2023-07-29T00:50:00.859 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.859 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:00.860 DEBUG:boto:Port: 8000 2023-07-29T00:50:00.860 DEBUG:boto:Params: {} 2023-07-29T00:50:00.860 DEBUG:boto:Token: None 2023-07-29T00:50:00.861 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj145 2023-07-29T00:50:00.861 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GwlmbZwVruhe66063Jv78QmqG30= 2023-07-29T00:50:00.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GwlmbZwVruhe66063Jv78QmqG30=', 'Content-Length': '0'} 2023-07-29T00:50:00.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000698ab8e87ebcb1b-0064c46238-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-07-29T00:50:00.865 DEBUG:boto:path=/hiiqos-61/obj145 2023-07-29T00:50:00.865 DEBUG:boto:auth_path=/hiiqos-61/obj145 2023-07-29T00:50:00.865 DEBUG:boto:Method: GET 2023-07-29T00:50:00.866 DEBUG:boto:Path: /hiiqos-61/obj145 2023-07-29T00:50:00.866 DEBUG:boto:Data: 2023-07-29T00:50:00.866 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.867 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:00.867 DEBUG:boto:Port: 8001 2023-07-29T00:50:00.867 DEBUG:boto:Params: {} 2023-07-29T00:50:00.868 DEBUG:boto:Token: None 2023-07-29T00:50:00.868 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj145 2023-07-29T00:50:00.869 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:abjyNlDIOLbl6J4qj53kvdeMx70= 2023-07-29T00:50:00.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:abjyNlDIOLbl6J4qj53kvdeMx70=', 'Content-Length': '0'} 2023-07-29T00:50:00.871 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000988932c9cca87465-0064c46238-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.871 DEBUG:boto:path=/hiiqos-61/obj145 2023-07-29T00:50:00.872 DEBUG:boto:auth_path=/hiiqos-61/obj145 2023-07-29T00:50:00.872 DEBUG:boto:Method: GET 2023-07-29T00:50:00.873 DEBUG:boto:Path: /hiiqos-61/obj145 2023-07-29T00:50:00.873 DEBUG:boto:Data: 2023-07-29T00:50:00.873 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.874 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:00.874 DEBUG:boto:Port: 8000 2023-07-29T00:50:00.874 DEBUG:boto:Params: {} 2023-07-29T00:50:00.875 DEBUG:boto:Token: None 2023-07-29T00:50:00.875 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj145 2023-07-29T00:50:00.876 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:abjyNlDIOLbl6J4qj53kvdeMx70= 2023-07-29T00:50:00.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:abjyNlDIOLbl6J4qj53kvdeMx70=', 'Content-Length': '0'} 2023-07-29T00:50:00.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1a38ff9cbf7f3ca-0064c46238-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.920 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:00.921 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:00.921 DEBUG:boto:path=/hiiqos-61/?prefix=obj145 2023-07-29T00:50:00.921 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj145 2023-07-29T00:50:00.922 DEBUG:boto:Method: GET 2023-07-29T00:50:00.922 DEBUG:boto:Path: /hiiqos-61/?prefix=obj145 2023-07-29T00:50:00.922 DEBUG:boto:Data: 2023-07-29T00:50:00.923 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.923 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:00.923 DEBUG:boto:Port: 8001 2023-07-29T00:50:00.924 DEBUG:boto:Params: {} 2023-07-29T00:50:00.924 DEBUG:boto:Token: None 2023-07-29T00:50:00.925 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/ 2023-07-29T00:50:00.925 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:A5qU4OtGaFhwcdcXg0UzfjcCpTo= 2023-07-29T00:50:00.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:A5qU4OtGaFhwcdcXg0UzfjcCpTo=', 'Content-Length': '0'} 2023-07-29T00:50:00.928 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060b7ac25f2309ec9-0064c46238-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.929 DEBUG:boto:b'hiiqos-61obj1451000falseobj1452023-07-29T00:45:02.452Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:00.929 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:00.930 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:00.930 DEBUG:boto:path=/hiiqos-61/?prefix=obj145 2023-07-29T00:50:00.930 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj145 2023-07-29T00:50:00.931 DEBUG:boto:Method: GET 2023-07-29T00:50:00.931 DEBUG:boto:Path: /hiiqos-61/?prefix=obj145 2023-07-29T00:50:00.931 DEBUG:boto:Data: 2023-07-29T00:50:00.932 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.932 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:00.932 DEBUG:boto:Port: 8000 2023-07-29T00:50:00.933 DEBUG:boto:Params: {} 2023-07-29T00:50:00.933 DEBUG:boto:Token: None 2023-07-29T00:50:00.934 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/ 2023-07-29T00:50:00.934 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:A5qU4OtGaFhwcdcXg0UzfjcCpTo= 2023-07-29T00:50:00.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:A5qU4OtGaFhwcdcXg0UzfjcCpTo=', 'Content-Length': '0'} 2023-07-29T00:50:00.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9a0376264aad9b0-0064c46238-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.939 DEBUG:boto:b'hiiqos-61obj1451000falseobj1452023-07-29T00:45:02.452Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:00.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-07-29T00:50:00.940 DEBUG:boto:path=/hiiqos-61/obj148 2023-07-29T00:50:00.940 DEBUG:boto:auth_path=/hiiqos-61/obj148 2023-07-29T00:50:00.940 DEBUG:boto:path=/hiiqos-61/obj148?versionId=null 2023-07-29T00:50:00.941 DEBUG:boto:auth_path=/hiiqos-61/obj148?versionId=null 2023-07-29T00:50:00.941 DEBUG:boto:Method: GET 2023-07-29T00:50:00.941 DEBUG:boto:Path: /hiiqos-61/obj148?versionId=null 2023-07-29T00:50:00.942 DEBUG:boto:Data: 2023-07-29T00:50:00.942 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.942 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:00.943 DEBUG:boto:Port: 8001 2023-07-29T00:50:00.943 DEBUG:boto:Params: {} 2023-07-29T00:50:00.944 DEBUG:boto:Token: None 2023-07-29T00:50:00.944 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj148?versionId=null 2023-07-29T00:50:00.944 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lVcrhQvK0KDLL1N3V/XyFFThTMQ= 2023-07-29T00:50:00.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lVcrhQvK0KDLL1N3V/XyFFThTMQ=', 'Content-Length': '0'} 2023-07-29T00:50:00.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f72dfa78fa7a21a7-0064c46238-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.988 DEBUG:boto:path=/hiiqos-61/obj148 2023-07-29T00:50:00.989 DEBUG:boto:auth_path=/hiiqos-61/obj148 2023-07-29T00:50:00.989 DEBUG:boto:path=/hiiqos-61/obj148?versionId=null 2023-07-29T00:50:00.990 DEBUG:boto:auth_path=/hiiqos-61/obj148?versionId=null 2023-07-29T00:50:00.990 DEBUG:boto:Method: GET 2023-07-29T00:50:00.990 DEBUG:boto:Path: /hiiqos-61/obj148?versionId=null 2023-07-29T00:50:00.991 DEBUG:boto:Data: 2023-07-29T00:50:00.991 DEBUG:boto:Headers: {} 2023-07-29T00:50:00.992 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:00.992 DEBUG:boto:Port: 8000 2023-07-29T00:50:00.993 DEBUG:boto:Params: {} 2023-07-29T00:50:00.993 DEBUG:boto:Token: None 2023-07-29T00:50:00.993 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:00 GMT /hiiqos-61/obj148?versionId=null 2023-07-29T00:50:00.994 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lVcrhQvK0KDLL1N3V/XyFFThTMQ= 2023-07-29T00:50:00.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lVcrhQvK0KDLL1N3V/XyFFThTMQ=', 'Content-Length': '0'} 2023-07-29T00:50:00.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031c19872f2f9a093-0064c46238-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:00.997 DEBUG:boto:path=/hiiqos-61/obj148 2023-07-29T00:50:00.998 DEBUG:boto:auth_path=/hiiqos-61/obj148 2023-07-29T00:50:00.998 DEBUG:boto:path=/hiiqos-61/obj148?versionId=null 2023-07-29T00:50:00.999 DEBUG:boto:auth_path=/hiiqos-61/obj148?versionId=null 2023-07-29T00:50:00.999 DEBUG:boto:Method: HEAD 2023-07-29T00:50:01.000 DEBUG:boto:Path: /hiiqos-61/obj148?versionId=null 2023-07-29T00:50:01.001 DEBUG:boto:Data: 2023-07-29T00:50:01.002 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.003 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:01.003 DEBUG:boto:Port: 8001 2023-07-29T00:50:01.004 DEBUG:boto:Params: {} 2023-07-29T00:50:01.004 DEBUG:boto:Token: None 2023-07-29T00:50:01.005 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj148?versionId=null 2023-07-29T00:50:01.005 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Xz7E/la0wWJ5Izql/S6Um5io/98= 2023-07-29T00:50:01.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Xz7E/la0wWJ5Izql/S6Um5io/98=', 'Content-Length': '0'} 2023-07-29T00:50:01.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082cf625b6f495105-0064c46238-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.008 DEBUG:boto:path=/hiiqos-61/obj148 2023-07-29T00:50:01.009 DEBUG:boto:auth_path=/hiiqos-61/obj148 2023-07-29T00:50:01.009 DEBUG:boto:path=/hiiqos-61/obj148?versionId=null 2023-07-29T00:50:01.009 DEBUG:boto:auth_path=/hiiqos-61/obj148?versionId=null 2023-07-29T00:50:01.010 DEBUG:boto:Method: HEAD 2023-07-29T00:50:01.010 DEBUG:boto:Path: /hiiqos-61/obj148?versionId=null 2023-07-29T00:50:01.010 DEBUG:boto:Data: 2023-07-29T00:50:01.011 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.011 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:01.011 DEBUG:boto:Port: 8000 2023-07-29T00:50:01.012 DEBUG:boto:Params: {} 2023-07-29T00:50:01.012 DEBUG:boto:Token: None 2023-07-29T00:50:01.013 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj148?versionId=null 2023-07-29T00:50:01.013 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Xz7E/la0wWJ5Izql/S6Um5io/98= 2023-07-29T00:50:01.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Xz7E/la0wWJ5Izql/S6Um5io/98=', 'Content-Length': '0'} 2023-07-29T00:50:01.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024cedf978037d158-0064c46238-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-07-29T00:50:01.017 DEBUG:boto:path=/hiiqos-61/obj148 2023-07-29T00:50:01.017 DEBUG:boto:auth_path=/hiiqos-61/obj148 2023-07-29T00:50:01.018 DEBUG:boto:path=/hiiqos-61/obj148?versionId=null 2023-07-29T00:50:01.018 DEBUG:boto:auth_path=/hiiqos-61/obj148?versionId=null 2023-07-29T00:50:01.018 DEBUG:boto:Method: GET 2023-07-29T00:50:01.019 DEBUG:boto:Path: /hiiqos-61/obj148?versionId=null 2023-07-29T00:50:01.019 DEBUG:boto:Data: 2023-07-29T00:50:01.019 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.020 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:01.020 DEBUG:boto:Port: 8001 2023-07-29T00:50:01.021 DEBUG:boto:Params: {} 2023-07-29T00:50:01.021 DEBUG:boto:Token: None 2023-07-29T00:50:01.022 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj148?versionId=null 2023-07-29T00:50:01.023 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FjdmTmxt7Ur6PShaFhjajV7MxMM= 2023-07-29T00:50:01.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FjdmTmxt7Ur6PShaFhjajV7MxMM=', 'Content-Length': '0'} 2023-07-29T00:50:01.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea320008487bdcab-0064c46238-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.027 DEBUG:boto:path=/hiiqos-61/obj148 2023-07-29T00:50:01.028 DEBUG:boto:auth_path=/hiiqos-61/obj148 2023-07-29T00:50:01.029 DEBUG:boto:path=/hiiqos-61/obj148?versionId=null 2023-07-29T00:50:01.029 DEBUG:boto:auth_path=/hiiqos-61/obj148?versionId=null 2023-07-29T00:50:01.030 DEBUG:boto:Method: GET 2023-07-29T00:50:01.030 DEBUG:boto:Path: /hiiqos-61/obj148?versionId=null 2023-07-29T00:50:01.030 DEBUG:boto:Data: 2023-07-29T00:50:01.031 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.031 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:01.031 DEBUG:boto:Port: 8000 2023-07-29T00:50:01.032 DEBUG:boto:Params: {} 2023-07-29T00:50:01.032 DEBUG:boto:Token: None 2023-07-29T00:50:01.033 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj148?versionId=null 2023-07-29T00:50:01.033 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FjdmTmxt7Ur6PShaFhjajV7MxMM= 2023-07-29T00:50:01.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FjdmTmxt7Ur6PShaFhjajV7MxMM=', 'Content-Length': '0'} 2023-07-29T00:50:01.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4d439a01ed8f30b-0064c46238-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.037 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:01.037 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:01.037 DEBUG:boto:path=/hiiqos-61/?prefix=obj148 2023-07-29T00:50:01.038 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj148 2023-07-29T00:50:01.038 DEBUG:boto:Method: GET 2023-07-29T00:50:01.038 DEBUG:boto:Path: /hiiqos-61/?prefix=obj148 2023-07-29T00:50:01.039 DEBUG:boto:Data: 2023-07-29T00:50:01.039 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.040 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:01.040 DEBUG:boto:Port: 8001 2023-07-29T00:50:01.040 DEBUG:boto:Params: {} 2023-07-29T00:50:01.041 DEBUG:boto:Token: None 2023-07-29T00:50:01.041 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/ 2023-07-29T00:50:01.042 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+J1dOPW7rrFcNkZdJdAc5Ld3sfw= 2023-07-29T00:50:01.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+J1dOPW7rrFcNkZdJdAc5Ld3sfw=', 'Content-Length': '0'} 2023-07-29T00:50:01.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fdcd2ea15c40e27-0064c46238-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.045 DEBUG:boto:b'hiiqos-61obj1481000falseobj1482023-07-29T00:45:03.128Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:01.046 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:01.046 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:01.047 DEBUG:boto:path=/hiiqos-61/?prefix=obj148 2023-07-29T00:50:01.047 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj148 2023-07-29T00:50:01.048 DEBUG:boto:Method: GET 2023-07-29T00:50:01.048 DEBUG:boto:Path: /hiiqos-61/?prefix=obj148 2023-07-29T00:50:01.048 DEBUG:boto:Data: 2023-07-29T00:50:01.049 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.049 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:01.050 DEBUG:boto:Port: 8000 2023-07-29T00:50:01.050 DEBUG:boto:Params: {} 2023-07-29T00:50:01.050 DEBUG:boto:Token: None 2023-07-29T00:50:01.051 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/ 2023-07-29T00:50:01.051 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+J1dOPW7rrFcNkZdJdAc5Ld3sfw= 2023-07-29T00:50:01.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+J1dOPW7rrFcNkZdJdAc5Ld3sfw=', 'Content-Length': '0'} 2023-07-29T00:50:01.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000010b1cd8fcf9a472-0064c46238-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.057 DEBUG:boto:b'hiiqos-61obj1481000falseobj1482023-07-29T00:45:03.128Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:01.058 DEBUG:boto:path=/hiiqos-61/obj148 2023-07-29T00:50:01.058 DEBUG:boto:auth_path=/hiiqos-61/obj148 2023-07-29T00:50:01.059 DEBUG:boto:Method: HEAD 2023-07-29T00:50:01.059 DEBUG:boto:Path: /hiiqos-61/obj148 2023-07-29T00:50:01.059 DEBUG:boto:Data: 2023-07-29T00:50:01.060 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.060 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:01.060 DEBUG:boto:Port: 8001 2023-07-29T00:50:01.061 DEBUG:boto:Params: {} 2023-07-29T00:50:01.061 DEBUG:boto:Token: None 2023-07-29T00:50:01.062 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj148 2023-07-29T00:50:01.062 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dpRC1fXi682QjwosUWwR9ArLAQ4= 2023-07-29T00:50:01.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dpRC1fXi682QjwosUWwR9ArLAQ4=', 'Content-Length': '0'} 2023-07-29T00:50:01.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000127a71ba7f97a403-0064c46238-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.065 DEBUG:boto:path=/hiiqos-61/obj148 2023-07-29T00:50:01.065 DEBUG:boto:auth_path=/hiiqos-61/obj148 2023-07-29T00:50:01.066 DEBUG:boto:Method: HEAD 2023-07-29T00:50:01.067 DEBUG:boto:Path: /hiiqos-61/obj148 2023-07-29T00:50:01.067 DEBUG:boto:Data: 2023-07-29T00:50:01.068 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.069 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:01.069 DEBUG:boto:Port: 8000 2023-07-29T00:50:01.069 DEBUG:boto:Params: {} 2023-07-29T00:50:01.070 DEBUG:boto:Token: None 2023-07-29T00:50:01.070 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj148 2023-07-29T00:50:01.071 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dpRC1fXi682QjwosUWwR9ArLAQ4= 2023-07-29T00:50:01.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dpRC1fXi682QjwosUWwR9ArLAQ4=', 'Content-Length': '0'} 2023-07-29T00:50:01.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000199d48c063b1e511-0064c46239-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-07-29T00:50:01.074 DEBUG:boto:path=/hiiqos-61/obj148 2023-07-29T00:50:01.074 DEBUG:boto:auth_path=/hiiqos-61/obj148 2023-07-29T00:50:01.075 DEBUG:boto:Method: GET 2023-07-29T00:50:01.075 DEBUG:boto:Path: /hiiqos-61/obj148 2023-07-29T00:50:01.076 DEBUG:boto:Data: 2023-07-29T00:50:01.076 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.076 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:01.077 DEBUG:boto:Port: 8001 2023-07-29T00:50:01.077 DEBUG:boto:Params: {} 2023-07-29T00:50:01.078 DEBUG:boto:Token: None 2023-07-29T00:50:01.078 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj148 2023-07-29T00:50:01.078 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RhVo1krQJisi+eFxU6ksHp978VI= 2023-07-29T00:50:01.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RhVo1krQJisi+eFxU6ksHp978VI=', 'Content-Length': '0'} 2023-07-29T00:50:01.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db41f78da2c2aa70-0064c46239-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.128 DEBUG:boto:path=/hiiqos-61/obj148 2023-07-29T00:50:01.129 DEBUG:boto:auth_path=/hiiqos-61/obj148 2023-07-29T00:50:01.129 DEBUG:boto:Method: GET 2023-07-29T00:50:01.130 DEBUG:boto:Path: /hiiqos-61/obj148 2023-07-29T00:50:01.130 DEBUG:boto:Data: 2023-07-29T00:50:01.130 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.131 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:01.131 DEBUG:boto:Port: 8000 2023-07-29T00:50:01.131 DEBUG:boto:Params: {} 2023-07-29T00:50:01.132 DEBUG:boto:Token: None 2023-07-29T00:50:01.132 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj148 2023-07-29T00:50:01.133 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RhVo1krQJisi+eFxU6ksHp978VI= 2023-07-29T00:50:01.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RhVo1krQJisi+eFxU6ksHp978VI=', 'Content-Length': '0'} 2023-07-29T00:50:01.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b17f53cd76b32528-0064c46239-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.136 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:01.136 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:01.137 DEBUG:boto:path=/hiiqos-61/?prefix=obj148 2023-07-29T00:50:01.137 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj148 2023-07-29T00:50:01.137 DEBUG:boto:Method: GET 2023-07-29T00:50:01.138 DEBUG:boto:Path: /hiiqos-61/?prefix=obj148 2023-07-29T00:50:01.138 DEBUG:boto:Data: 2023-07-29T00:50:01.138 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.139 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:01.139 DEBUG:boto:Port: 8001 2023-07-29T00:50:01.139 DEBUG:boto:Params: {} 2023-07-29T00:50:01.140 DEBUG:boto:Token: None 2023-07-29T00:50:01.140 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/ 2023-07-29T00:50:01.141 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+J1dOPW7rrFcNkZdJdAc5Ld3sfw= 2023-07-29T00:50:01.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+J1dOPW7rrFcNkZdJdAc5Ld3sfw=', 'Content-Length': '0'} 2023-07-29T00:50:01.143 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002053f8f0d0a9ab79-0064c46239-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.144 DEBUG:boto:b'hiiqos-61obj1481000falseobj1482023-07-29T00:45:03.128Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:01.145 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:01.145 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:01.145 DEBUG:boto:path=/hiiqos-61/?prefix=obj148 2023-07-29T00:50:01.146 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj148 2023-07-29T00:50:01.146 DEBUG:boto:Method: GET 2023-07-29T00:50:01.147 DEBUG:boto:Path: /hiiqos-61/?prefix=obj148 2023-07-29T00:50:01.147 DEBUG:boto:Data: 2023-07-29T00:50:01.147 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.148 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:01.148 DEBUG:boto:Port: 8000 2023-07-29T00:50:01.148 DEBUG:boto:Params: {} 2023-07-29T00:50:01.149 DEBUG:boto:Token: None 2023-07-29T00:50:01.149 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/ 2023-07-29T00:50:01.150 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+J1dOPW7rrFcNkZdJdAc5Ld3sfw= 2023-07-29T00:50:01.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+J1dOPW7rrFcNkZdJdAc5Ld3sfw=', 'Content-Length': '0'} 2023-07-29T00:50:01.154 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea0a02dd465fad14-0064c46239-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.154 DEBUG:boto:b'hiiqos-61obj1481000falseobj1482023-07-29T00:45:03.128Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:01.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-07-29T00:50:01.155 DEBUG:boto:path=/hiiqos-61/obj155 2023-07-29T00:50:01.156 DEBUG:boto:auth_path=/hiiqos-61/obj155 2023-07-29T00:50:01.156 DEBUG:boto:path=/hiiqos-61/obj155?versionId=null 2023-07-29T00:50:01.157 DEBUG:boto:auth_path=/hiiqos-61/obj155?versionId=null 2023-07-29T00:50:01.157 DEBUG:boto:Method: GET 2023-07-29T00:50:01.157 DEBUG:boto:Path: /hiiqos-61/obj155?versionId=null 2023-07-29T00:50:01.158 DEBUG:boto:Data: 2023-07-29T00:50:01.158 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.158 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:01.159 DEBUG:boto:Port: 8001 2023-07-29T00:50:01.159 DEBUG:boto:Params: {} 2023-07-29T00:50:01.159 DEBUG:boto:Token: None 2023-07-29T00:50:01.160 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj155?versionId=null 2023-07-29T00:50:01.160 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Qq/hz3t/4TXrLK2jaVfDWgYR5IU= 2023-07-29T00:50:01.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Qq/hz3t/4TXrLK2jaVfDWgYR5IU=', 'Content-Length': '0'} 2023-07-29T00:50:01.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a66e176c71d202c4-0064c46239-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.164 DEBUG:boto:path=/hiiqos-61/obj155 2023-07-29T00:50:01.164 DEBUG:boto:auth_path=/hiiqos-61/obj155 2023-07-29T00:50:01.164 DEBUG:boto:path=/hiiqos-61/obj155?versionId=null 2023-07-29T00:50:01.165 DEBUG:boto:auth_path=/hiiqos-61/obj155?versionId=null 2023-07-29T00:50:01.165 DEBUG:boto:Method: GET 2023-07-29T00:50:01.165 DEBUG:boto:Path: /hiiqos-61/obj155?versionId=null 2023-07-29T00:50:01.166 DEBUG:boto:Data: 2023-07-29T00:50:01.166 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.166 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:01.167 DEBUG:boto:Port: 8000 2023-07-29T00:50:01.167 DEBUG:boto:Params: {} 2023-07-29T00:50:01.168 DEBUG:boto:Token: None 2023-07-29T00:50:01.168 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj155?versionId=null 2023-07-29T00:50:01.168 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Qq/hz3t/4TXrLK2jaVfDWgYR5IU= 2023-07-29T00:50:01.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Qq/hz3t/4TXrLK2jaVfDWgYR5IU=', 'Content-Length': '0'} 2023-07-29T00:50:01.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027520d08da63c995-0064c46239-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.216 DEBUG:boto:path=/hiiqos-61/obj155 2023-07-29T00:50:01.217 DEBUG:boto:auth_path=/hiiqos-61/obj155 2023-07-29T00:50:01.217 DEBUG:boto:path=/hiiqos-61/obj155?versionId=null 2023-07-29T00:50:01.217 DEBUG:boto:auth_path=/hiiqos-61/obj155?versionId=null 2023-07-29T00:50:01.218 DEBUG:boto:Method: HEAD 2023-07-29T00:50:01.218 DEBUG:boto:Path: /hiiqos-61/obj155?versionId=null 2023-07-29T00:50:01.218 DEBUG:boto:Data: 2023-07-29T00:50:01.219 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.219 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:01.219 DEBUG:boto:Port: 8001 2023-07-29T00:50:01.220 DEBUG:boto:Params: {} 2023-07-29T00:50:01.220 DEBUG:boto:Token: None 2023-07-29T00:50:01.221 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj155?versionId=null 2023-07-29T00:50:01.221 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CjFYLT9UDaNht6MKwgXY28AQpjQ= 2023-07-29T00:50:01.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CjFYLT9UDaNht6MKwgXY28AQpjQ=', 'Content-Length': '0'} 2023-07-29T00:50:01.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df94a2bcb0413775-0064c46239-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.224 DEBUG:boto:path=/hiiqos-61/obj155 2023-07-29T00:50:01.224 DEBUG:boto:auth_path=/hiiqos-61/obj155 2023-07-29T00:50:01.225 DEBUG:boto:path=/hiiqos-61/obj155?versionId=null 2023-07-29T00:50:01.225 DEBUG:boto:auth_path=/hiiqos-61/obj155?versionId=null 2023-07-29T00:50:01.225 DEBUG:boto:Method: HEAD 2023-07-29T00:50:01.226 DEBUG:boto:Path: /hiiqos-61/obj155?versionId=null 2023-07-29T00:50:01.226 DEBUG:boto:Data: 2023-07-29T00:50:01.226 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.227 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:01.227 DEBUG:boto:Port: 8000 2023-07-29T00:50:01.227 DEBUG:boto:Params: {} 2023-07-29T00:50:01.228 DEBUG:boto:Token: None 2023-07-29T00:50:01.228 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj155?versionId=null 2023-07-29T00:50:01.228 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CjFYLT9UDaNht6MKwgXY28AQpjQ= 2023-07-29T00:50:01.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CjFYLT9UDaNht6MKwgXY28AQpjQ=', 'Content-Length': '0'} 2023-07-29T00:50:01.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf2562fc6278a3be-0064c46239-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-07-29T00:50:01.231 DEBUG:boto:path=/hiiqos-61/obj155 2023-07-29T00:50:01.232 DEBUG:boto:auth_path=/hiiqos-61/obj155 2023-07-29T00:50:01.232 DEBUG:boto:path=/hiiqos-61/obj155?versionId=null 2023-07-29T00:50:01.232 DEBUG:boto:auth_path=/hiiqos-61/obj155?versionId=null 2023-07-29T00:50:01.233 DEBUG:boto:Method: GET 2023-07-29T00:50:01.233 DEBUG:boto:Path: /hiiqos-61/obj155?versionId=null 2023-07-29T00:50:01.233 DEBUG:boto:Data: 2023-07-29T00:50:01.234 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.234 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:01.235 DEBUG:boto:Port: 8001 2023-07-29T00:50:01.235 DEBUG:boto:Params: {} 2023-07-29T00:50:01.235 DEBUG:boto:Token: None 2023-07-29T00:50:01.236 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj155?versionId=null 2023-07-29T00:50:01.236 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Qq/hz3t/4TXrLK2jaVfDWgYR5IU= 2023-07-29T00:50:01.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Qq/hz3t/4TXrLK2jaVfDWgYR5IU=', 'Content-Length': '0'} 2023-07-29T00:50:01.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044361e1f690fde85-0064c46239-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.239 DEBUG:boto:path=/hiiqos-61/obj155 2023-07-29T00:50:01.239 DEBUG:boto:auth_path=/hiiqos-61/obj155 2023-07-29T00:50:01.239 DEBUG:boto:path=/hiiqos-61/obj155?versionId=null 2023-07-29T00:50:01.240 DEBUG:boto:auth_path=/hiiqos-61/obj155?versionId=null 2023-07-29T00:50:01.240 DEBUG:boto:Method: GET 2023-07-29T00:50:01.241 DEBUG:boto:Path: /hiiqos-61/obj155?versionId=null 2023-07-29T00:50:01.241 DEBUG:boto:Data: 2023-07-29T00:50:01.241 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.242 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:01.242 DEBUG:boto:Port: 8000 2023-07-29T00:50:01.242 DEBUG:boto:Params: {} 2023-07-29T00:50:01.243 DEBUG:boto:Token: None 2023-07-29T00:50:01.243 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj155?versionId=null 2023-07-29T00:50:01.244 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Qq/hz3t/4TXrLK2jaVfDWgYR5IU= 2023-07-29T00:50:01.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Qq/hz3t/4TXrLK2jaVfDWgYR5IU=', 'Content-Length': '0'} 2023-07-29T00:50:01.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c701f29805b7448e-0064c46239-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.246 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:01.247 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:01.247 DEBUG:boto:path=/hiiqos-61/?prefix=obj155 2023-07-29T00:50:01.247 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj155 2023-07-29T00:50:01.248 DEBUG:boto:Method: GET 2023-07-29T00:50:01.248 DEBUG:boto:Path: /hiiqos-61/?prefix=obj155 2023-07-29T00:50:01.249 DEBUG:boto:Data: 2023-07-29T00:50:01.249 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.249 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:01.250 DEBUG:boto:Port: 8001 2023-07-29T00:50:01.250 DEBUG:boto:Params: {} 2023-07-29T00:50:01.250 DEBUG:boto:Token: None 2023-07-29T00:50:01.251 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/ 2023-07-29T00:50:01.251 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+J1dOPW7rrFcNkZdJdAc5Ld3sfw= 2023-07-29T00:50:01.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+J1dOPW7rrFcNkZdJdAc5Ld3sfw=', 'Content-Length': '0'} 2023-07-29T00:50:01.254 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047165b48fd2e0d9d-0064c46239-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.255 DEBUG:boto:b'hiiqos-61obj1551000falseobj1552023-07-29T00:45:02.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:01.255 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:01.256 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:01.256 DEBUG:boto:path=/hiiqos-61/?prefix=obj155 2023-07-29T00:50:01.256 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj155 2023-07-29T00:50:01.257 DEBUG:boto:Method: GET 2023-07-29T00:50:01.257 DEBUG:boto:Path: /hiiqos-61/?prefix=obj155 2023-07-29T00:50:01.257 DEBUG:boto:Data: 2023-07-29T00:50:01.258 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.258 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:01.259 DEBUG:boto:Port: 8000 2023-07-29T00:50:01.259 DEBUG:boto:Params: {} 2023-07-29T00:50:01.259 DEBUG:boto:Token: None 2023-07-29T00:50:01.260 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/ 2023-07-29T00:50:01.260 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+J1dOPW7rrFcNkZdJdAc5Ld3sfw= 2023-07-29T00:50:01.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+J1dOPW7rrFcNkZdJdAc5Ld3sfw=', 'Content-Length': '0'} 2023-07-29T00:50:01.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0e92c1774fa8e8a-0064c46239-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.265 DEBUG:boto:b'hiiqos-61obj1551000falseobj1552023-07-29T00:45:02.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:01.265 DEBUG:boto:path=/hiiqos-61/obj155 2023-07-29T00:50:01.266 DEBUG:boto:auth_path=/hiiqos-61/obj155 2023-07-29T00:50:01.266 DEBUG:boto:Method: HEAD 2023-07-29T00:50:01.266 DEBUG:boto:Path: /hiiqos-61/obj155 2023-07-29T00:50:01.267 DEBUG:boto:Data: 2023-07-29T00:50:01.267 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.267 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:01.268 DEBUG:boto:Port: 8001 2023-07-29T00:50:01.268 DEBUG:boto:Params: {} 2023-07-29T00:50:01.268 DEBUG:boto:Token: None 2023-07-29T00:50:01.269 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj155 2023-07-29T00:50:01.269 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:So8RJXPUD8ujyhwUXjJZWWzG2ao= 2023-07-29T00:50:01.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:So8RJXPUD8ujyhwUXjJZWWzG2ao=', 'Content-Length': '0'} 2023-07-29T00:50:01.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029706a0180be16bc-0064c46239-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.272 DEBUG:boto:path=/hiiqos-61/obj155 2023-07-29T00:50:01.272 DEBUG:boto:auth_path=/hiiqos-61/obj155 2023-07-29T00:50:01.273 DEBUG:boto:Method: HEAD 2023-07-29T00:50:01.273 DEBUG:boto:Path: /hiiqos-61/obj155 2023-07-29T00:50:01.273 DEBUG:boto:Data: 2023-07-29T00:50:01.274 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.274 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:01.274 DEBUG:boto:Port: 8000 2023-07-29T00:50:01.275 DEBUG:boto:Params: {} 2023-07-29T00:50:01.275 DEBUG:boto:Token: None 2023-07-29T00:50:01.275 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj155 2023-07-29T00:50:01.276 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:So8RJXPUD8ujyhwUXjJZWWzG2ao= 2023-07-29T00:50:01.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:So8RJXPUD8ujyhwUXjJZWWzG2ao=', 'Content-Length': '0'} 2023-07-29T00:50:01.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0abbdc4f3c472f2-0064c46239-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-07-29T00:50:01.279 DEBUG:boto:path=/hiiqos-61/obj155 2023-07-29T00:50:01.279 DEBUG:boto:auth_path=/hiiqos-61/obj155 2023-07-29T00:50:01.280 DEBUG:boto:Method: GET 2023-07-29T00:50:01.280 DEBUG:boto:Path: /hiiqos-61/obj155 2023-07-29T00:50:01.280 DEBUG:boto:Data: 2023-07-29T00:50:01.281 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.281 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:01.281 DEBUG:boto:Port: 8001 2023-07-29T00:50:01.282 DEBUG:boto:Params: {} 2023-07-29T00:50:01.282 DEBUG:boto:Token: None 2023-07-29T00:50:01.283 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj155 2023-07-29T00:50:01.283 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zl9PEiUKCFUZ9CxuSaLSq5U2NjQ= 2023-07-29T00:50:01.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zl9PEiUKCFUZ9CxuSaLSq5U2NjQ=', 'Content-Length': '0'} 2023-07-29T00:50:01.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090b2757f25b853ac-0064c46239-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.286 DEBUG:boto:path=/hiiqos-61/obj155 2023-07-29T00:50:01.286 DEBUG:boto:auth_path=/hiiqos-61/obj155 2023-07-29T00:50:01.287 DEBUG:boto:Method: GET 2023-07-29T00:50:01.287 DEBUG:boto:Path: /hiiqos-61/obj155 2023-07-29T00:50:01.287 DEBUG:boto:Data: 2023-07-29T00:50:01.288 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.288 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:01.289 DEBUG:boto:Port: 8000 2023-07-29T00:50:01.289 DEBUG:boto:Params: {} 2023-07-29T00:50:01.290 DEBUG:boto:Token: None 2023-07-29T00:50:01.290 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj155 2023-07-29T00:50:01.291 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zl9PEiUKCFUZ9CxuSaLSq5U2NjQ= 2023-07-29T00:50:01.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zl9PEiUKCFUZ9CxuSaLSq5U2NjQ=', 'Content-Length': '0'} 2023-07-29T00:50:01.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035ca6c7822057d28-0064c46239-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.293 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:01.294 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:01.294 DEBUG:boto:path=/hiiqos-61/?prefix=obj155 2023-07-29T00:50:01.295 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj155 2023-07-29T00:50:01.295 DEBUG:boto:Method: GET 2023-07-29T00:50:01.295 DEBUG:boto:Path: /hiiqos-61/?prefix=obj155 2023-07-29T00:50:01.296 DEBUG:boto:Data: 2023-07-29T00:50:01.297 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.297 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:01.298 DEBUG:boto:Port: 8001 2023-07-29T00:50:01.299 DEBUG:boto:Params: {} 2023-07-29T00:50:01.300 DEBUG:boto:Token: None 2023-07-29T00:50:01.301 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/ 2023-07-29T00:50:01.302 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+J1dOPW7rrFcNkZdJdAc5Ld3sfw= 2023-07-29T00:50:01.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+J1dOPW7rrFcNkZdJdAc5Ld3sfw=', 'Content-Length': '0'} 2023-07-29T00:50:01.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002640de35407604da-0064c46239-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.305 DEBUG:boto:b'hiiqos-61obj1551000falseobj1552023-07-29T00:45:02.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:01.305 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:01.306 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:01.306 DEBUG:boto:path=/hiiqos-61/?prefix=obj155 2023-07-29T00:50:01.307 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj155 2023-07-29T00:50:01.307 DEBUG:boto:Method: GET 2023-07-29T00:50:01.308 DEBUG:boto:Path: /hiiqos-61/?prefix=obj155 2023-07-29T00:50:01.308 DEBUG:boto:Data: 2023-07-29T00:50:01.308 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.309 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:01.309 DEBUG:boto:Port: 8000 2023-07-29T00:50:01.310 DEBUG:boto:Params: {} 2023-07-29T00:50:01.310 DEBUG:boto:Token: None 2023-07-29T00:50:01.310 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/ 2023-07-29T00:50:01.311 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+J1dOPW7rrFcNkZdJdAc5Ld3sfw= 2023-07-29T00:50:01.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+J1dOPW7rrFcNkZdJdAc5Ld3sfw=', 'Content-Length': '0'} 2023-07-29T00:50:01.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034634ae8f4099b62-0064c46239-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.356 DEBUG:boto:b'hiiqos-61obj1551000falseobj1552023-07-29T00:45:02.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:01.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-07-29T00:50:01.358 DEBUG:boto:path=/hiiqos-61/obj17 2023-07-29T00:50:01.358 DEBUG:boto:auth_path=/hiiqos-61/obj17 2023-07-29T00:50:01.359 DEBUG:boto:path=/hiiqos-61/obj17?versionId=null 2023-07-29T00:50:01.359 DEBUG:boto:auth_path=/hiiqos-61/obj17?versionId=null 2023-07-29T00:50:01.359 DEBUG:boto:Method: GET 2023-07-29T00:50:01.360 DEBUG:boto:Path: /hiiqos-61/obj17?versionId=null 2023-07-29T00:50:01.360 DEBUG:boto:Data: 2023-07-29T00:50:01.361 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.361 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:01.361 DEBUG:boto:Port: 8001 2023-07-29T00:50:01.362 DEBUG:boto:Params: {} 2023-07-29T00:50:01.363 DEBUG:boto:Token: None 2023-07-29T00:50:01.364 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj17?versionId=null 2023-07-29T00:50:01.365 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bzR+MAjeKOU97FJcZvXGT/cppeU= 2023-07-29T00:50:01.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bzR+MAjeKOU97FJcZvXGT/cppeU=', 'Content-Length': '0'} 2023-07-29T00:50:01.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005f8c1c70511204c-0064c46239-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.368 DEBUG:boto:path=/hiiqos-61/obj17 2023-07-29T00:50:01.368 DEBUG:boto:auth_path=/hiiqos-61/obj17 2023-07-29T00:50:01.369 DEBUG:boto:path=/hiiqos-61/obj17?versionId=null 2023-07-29T00:50:01.369 DEBUG:boto:auth_path=/hiiqos-61/obj17?versionId=null 2023-07-29T00:50:01.370 DEBUG:boto:Method: GET 2023-07-29T00:50:01.370 DEBUG:boto:Path: /hiiqos-61/obj17?versionId=null 2023-07-29T00:50:01.370 DEBUG:boto:Data: 2023-07-29T00:50:01.371 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.371 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:01.371 DEBUG:boto:Port: 8000 2023-07-29T00:50:01.372 DEBUG:boto:Params: {} 2023-07-29T00:50:01.372 DEBUG:boto:Token: None 2023-07-29T00:50:01.373 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj17?versionId=null 2023-07-29T00:50:01.373 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bzR+MAjeKOU97FJcZvXGT/cppeU= 2023-07-29T00:50:01.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bzR+MAjeKOU97FJcZvXGT/cppeU=', 'Content-Length': '0'} 2023-07-29T00:50:01.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4cfe53726662d8d-0064c46239-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.376 DEBUG:boto:path=/hiiqos-61/obj17 2023-07-29T00:50:01.377 DEBUG:boto:auth_path=/hiiqos-61/obj17 2023-07-29T00:50:01.377 DEBUG:boto:path=/hiiqos-61/obj17?versionId=null 2023-07-29T00:50:01.377 DEBUG:boto:auth_path=/hiiqos-61/obj17?versionId=null 2023-07-29T00:50:01.378 DEBUG:boto:Method: HEAD 2023-07-29T00:50:01.378 DEBUG:boto:Path: /hiiqos-61/obj17?versionId=null 2023-07-29T00:50:01.379 DEBUG:boto:Data: 2023-07-29T00:50:01.379 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.379 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:01.380 DEBUG:boto:Port: 8001 2023-07-29T00:50:01.380 DEBUG:boto:Params: {} 2023-07-29T00:50:01.381 DEBUG:boto:Token: None 2023-07-29T00:50:01.381 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj17?versionId=null 2023-07-29T00:50:01.382 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GuKcskUuVj9qGVEtUvk+HCGFnCQ= 2023-07-29T00:50:01.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GuKcskUuVj9qGVEtUvk+HCGFnCQ=', 'Content-Length': '0'} 2023-07-29T00:50:01.384 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000686f0f7385044540-0064c46239-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.384 DEBUG:boto:path=/hiiqos-61/obj17 2023-07-29T00:50:01.385 DEBUG:boto:auth_path=/hiiqos-61/obj17 2023-07-29T00:50:01.385 DEBUG:boto:path=/hiiqos-61/obj17?versionId=null 2023-07-29T00:50:01.385 DEBUG:boto:auth_path=/hiiqos-61/obj17?versionId=null 2023-07-29T00:50:01.386 DEBUG:boto:Method: HEAD 2023-07-29T00:50:01.386 DEBUG:boto:Path: /hiiqos-61/obj17?versionId=null 2023-07-29T00:50:01.386 DEBUG:boto:Data: 2023-07-29T00:50:01.387 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.387 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:01.387 DEBUG:boto:Port: 8000 2023-07-29T00:50:01.388 DEBUG:boto:Params: {} 2023-07-29T00:50:01.388 DEBUG:boto:Token: None 2023-07-29T00:50:01.389 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj17?versionId=null 2023-07-29T00:50:01.389 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GuKcskUuVj9qGVEtUvk+HCGFnCQ= 2023-07-29T00:50:01.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GuKcskUuVj9qGVEtUvk+HCGFnCQ=', 'Content-Length': '0'} 2023-07-29T00:50:01.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b879b43e1a6c9e55-0064c46239-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-07-29T00:50:01.392 DEBUG:boto:path=/hiiqos-61/obj17 2023-07-29T00:50:01.392 DEBUG:boto:auth_path=/hiiqos-61/obj17 2023-07-29T00:50:01.393 DEBUG:boto:path=/hiiqos-61/obj17?versionId=null 2023-07-29T00:50:01.393 DEBUG:boto:auth_path=/hiiqos-61/obj17?versionId=null 2023-07-29T00:50:01.393 DEBUG:boto:Method: GET 2023-07-29T00:50:01.394 DEBUG:boto:Path: /hiiqos-61/obj17?versionId=null 2023-07-29T00:50:01.394 DEBUG:boto:Data: 2023-07-29T00:50:01.395 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.395 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:01.395 DEBUG:boto:Port: 8001 2023-07-29T00:50:01.396 DEBUG:boto:Params: {} 2023-07-29T00:50:01.396 DEBUG:boto:Token: None 2023-07-29T00:50:01.396 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj17?versionId=null 2023-07-29T00:50:01.397 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bzR+MAjeKOU97FJcZvXGT/cppeU= 2023-07-29T00:50:01.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bzR+MAjeKOU97FJcZvXGT/cppeU=', 'Content-Length': '0'} 2023-07-29T00:50:01.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ce76301228c702c-0064c46239-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.401 DEBUG:boto:path=/hiiqos-61/obj17 2023-07-29T00:50:01.401 DEBUG:boto:auth_path=/hiiqos-61/obj17 2023-07-29T00:50:01.401 DEBUG:boto:path=/hiiqos-61/obj17?versionId=null 2023-07-29T00:50:01.402 DEBUG:boto:auth_path=/hiiqos-61/obj17?versionId=null 2023-07-29T00:50:01.402 DEBUG:boto:Method: GET 2023-07-29T00:50:01.403 DEBUG:boto:Path: /hiiqos-61/obj17?versionId=null 2023-07-29T00:50:01.403 DEBUG:boto:Data: 2023-07-29T00:50:01.403 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.404 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:01.404 DEBUG:boto:Port: 8000 2023-07-29T00:50:01.404 DEBUG:boto:Params: {} 2023-07-29T00:50:01.405 DEBUG:boto:Token: None 2023-07-29T00:50:01.405 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj17?versionId=null 2023-07-29T00:50:01.406 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bzR+MAjeKOU97FJcZvXGT/cppeU= 2023-07-29T00:50:01.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bzR+MAjeKOU97FJcZvXGT/cppeU=', 'Content-Length': '0'} 2023-07-29T00:50:01.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9649d7773ad6c94-0064c46239-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.408 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:01.409 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:01.409 DEBUG:boto:path=/hiiqos-61/?prefix=obj17 2023-07-29T00:50:01.409 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj17 2023-07-29T00:50:01.410 DEBUG:boto:Method: GET 2023-07-29T00:50:01.410 DEBUG:boto:Path: /hiiqos-61/?prefix=obj17 2023-07-29T00:50:01.410 DEBUG:boto:Data: 2023-07-29T00:50:01.411 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.411 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:01.412 DEBUG:boto:Port: 8001 2023-07-29T00:50:01.412 DEBUG:boto:Params: {} 2023-07-29T00:50:01.412 DEBUG:boto:Token: None 2023-07-29T00:50:01.413 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/ 2023-07-29T00:50:01.413 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+J1dOPW7rrFcNkZdJdAc5Ld3sfw= 2023-07-29T00:50:01.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+J1dOPW7rrFcNkZdJdAc5Ld3sfw=', 'Content-Length': '0'} 2023-07-29T00:50:01.416 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f56116a99cd38e9-0064c46239-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.416 DEBUG:boto:b'hiiqos-61obj171000falseobj172023-07-29T00:45:01.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:01.417 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:01.417 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:01.418 DEBUG:boto:path=/hiiqos-61/?prefix=obj17 2023-07-29T00:50:01.418 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj17 2023-07-29T00:50:01.419 DEBUG:boto:Method: GET 2023-07-29T00:50:01.419 DEBUG:boto:Path: /hiiqos-61/?prefix=obj17 2023-07-29T00:50:01.419 DEBUG:boto:Data: 2023-07-29T00:50:01.420 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.420 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:01.420 DEBUG:boto:Port: 8000 2023-07-29T00:50:01.421 DEBUG:boto:Params: {} 2023-07-29T00:50:01.421 DEBUG:boto:Token: None 2023-07-29T00:50:01.422 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/ 2023-07-29T00:50:01.422 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+J1dOPW7rrFcNkZdJdAc5Ld3sfw= 2023-07-29T00:50:01.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+J1dOPW7rrFcNkZdJdAc5Ld3sfw=', 'Content-Length': '0'} 2023-07-29T00:50:01.426 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a1c717d1a4e7066-0064c46239-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.427 DEBUG:boto:b'hiiqos-61obj171000falseobj172023-07-29T00:45:01.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:01.427 DEBUG:boto:path=/hiiqos-61/obj17 2023-07-29T00:50:01.428 DEBUG:boto:auth_path=/hiiqos-61/obj17 2023-07-29T00:50:01.428 DEBUG:boto:Method: HEAD 2023-07-29T00:50:01.429 DEBUG:boto:Path: /hiiqos-61/obj17 2023-07-29T00:50:01.429 DEBUG:boto:Data: 2023-07-29T00:50:01.429 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.430 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:01.430 DEBUG:boto:Port: 8001 2023-07-29T00:50:01.430 DEBUG:boto:Params: {} 2023-07-29T00:50:01.431 DEBUG:boto:Token: None 2023-07-29T00:50:01.431 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj17 2023-07-29T00:50:01.431 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+3fZP4eOK0CRwenlv7v0zG4Hf7M= 2023-07-29T00:50:01.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+3fZP4eOK0CRwenlv7v0zG4Hf7M=', 'Content-Length': '0'} 2023-07-29T00:50:01.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e394996fd37c244d-0064c46239-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.434 DEBUG:boto:path=/hiiqos-61/obj17 2023-07-29T00:50:01.435 DEBUG:boto:auth_path=/hiiqos-61/obj17 2023-07-29T00:50:01.435 DEBUG:boto:Method: HEAD 2023-07-29T00:50:01.435 DEBUG:boto:Path: /hiiqos-61/obj17 2023-07-29T00:50:01.436 DEBUG:boto:Data: 2023-07-29T00:50:01.436 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.436 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:01.437 DEBUG:boto:Port: 8000 2023-07-29T00:50:01.437 DEBUG:boto:Params: {} 2023-07-29T00:50:01.437 DEBUG:boto:Token: None 2023-07-29T00:50:01.438 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj17 2023-07-29T00:50:01.438 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+3fZP4eOK0CRwenlv7v0zG4Hf7M= 2023-07-29T00:50:01.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+3fZP4eOK0CRwenlv7v0zG4Hf7M=', 'Content-Length': '0'} 2023-07-29T00:50:01.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ac1f3e2bbde682f-0064c46239-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-07-29T00:50:01.441 DEBUG:boto:path=/hiiqos-61/obj17 2023-07-29T00:50:01.442 DEBUG:boto:auth_path=/hiiqos-61/obj17 2023-07-29T00:50:01.442 DEBUG:boto:Method: GET 2023-07-29T00:50:01.442 DEBUG:boto:Path: /hiiqos-61/obj17 2023-07-29T00:50:01.443 DEBUG:boto:Data: 2023-07-29T00:50:01.443 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.444 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:01.444 DEBUG:boto:Port: 8001 2023-07-29T00:50:01.444 DEBUG:boto:Params: {} 2023-07-29T00:50:01.445 DEBUG:boto:Token: None 2023-07-29T00:50:01.445 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj17 2023-07-29T00:50:01.445 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:seH7rlvI1o77Yd90QBxKeQmqpbg= 2023-07-29T00:50:01.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:seH7rlvI1o77Yd90QBxKeQmqpbg=', 'Content-Length': '0'} 2023-07-29T00:50:01.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025c10ac4325012ad-0064c46239-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.448 DEBUG:boto:path=/hiiqos-61/obj17 2023-07-29T00:50:01.449 DEBUG:boto:auth_path=/hiiqos-61/obj17 2023-07-29T00:50:01.449 DEBUG:boto:Method: GET 2023-07-29T00:50:01.449 DEBUG:boto:Path: /hiiqos-61/obj17 2023-07-29T00:50:01.450 DEBUG:boto:Data: 2023-07-29T00:50:01.450 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.450 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:01.451 DEBUG:boto:Port: 8000 2023-07-29T00:50:01.451 DEBUG:boto:Params: {} 2023-07-29T00:50:01.452 DEBUG:boto:Token: None 2023-07-29T00:50:01.452 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj17 2023-07-29T00:50:01.452 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:seH7rlvI1o77Yd90QBxKeQmqpbg= 2023-07-29T00:50:01.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:seH7rlvI1o77Yd90QBxKeQmqpbg=', 'Content-Length': '0'} 2023-07-29T00:50:01.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000441b3a199e9b2453-0064c46239-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.496 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:01.497 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:01.497 DEBUG:boto:path=/hiiqos-61/?prefix=obj17 2023-07-29T00:50:01.498 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj17 2023-07-29T00:50:01.498 DEBUG:boto:Method: GET 2023-07-29T00:50:01.498 DEBUG:boto:Path: /hiiqos-61/?prefix=obj17 2023-07-29T00:50:01.499 DEBUG:boto:Data: 2023-07-29T00:50:01.499 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.499 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:01.500 DEBUG:boto:Port: 8001 2023-07-29T00:50:01.500 DEBUG:boto:Params: {} 2023-07-29T00:50:01.501 DEBUG:boto:Token: None 2023-07-29T00:50:01.501 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/ 2023-07-29T00:50:01.502 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+J1dOPW7rrFcNkZdJdAc5Ld3sfw= 2023-07-29T00:50:01.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+J1dOPW7rrFcNkZdJdAc5Ld3sfw=', 'Content-Length': '0'} 2023-07-29T00:50:01.504 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000214d10130f8103f2-0064c46239-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.505 DEBUG:boto:b'hiiqos-61obj171000falseobj172023-07-29T00:45:01.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:01.505 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:01.506 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:01.506 DEBUG:boto:path=/hiiqos-61/?prefix=obj17 2023-07-29T00:50:01.507 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj17 2023-07-29T00:50:01.507 DEBUG:boto:Method: GET 2023-07-29T00:50:01.508 DEBUG:boto:Path: /hiiqos-61/?prefix=obj17 2023-07-29T00:50:01.508 DEBUG:boto:Data: 2023-07-29T00:50:01.509 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.509 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:01.509 DEBUG:boto:Port: 8000 2023-07-29T00:50:01.510 DEBUG:boto:Params: {} 2023-07-29T00:50:01.510 DEBUG:boto:Token: None 2023-07-29T00:50:01.511 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/ 2023-07-29T00:50:01.511 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+J1dOPW7rrFcNkZdJdAc5Ld3sfw= 2023-07-29T00:50:01.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+J1dOPW7rrFcNkZdJdAc5Ld3sfw=', 'Content-Length': '0'} 2023-07-29T00:50:01.515 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000839aaff0b2c13f8d-0064c46239-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.516 DEBUG:boto:b'hiiqos-61obj171000falseobj172023-07-29T00:45:01.367Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:01.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-07-29T00:50:01.517 DEBUG:boto:path=/hiiqos-61/obj185 2023-07-29T00:50:01.517 DEBUG:boto:auth_path=/hiiqos-61/obj185 2023-07-29T00:50:01.518 DEBUG:boto:path=/hiiqos-61/obj185?versionId=null 2023-07-29T00:50:01.518 DEBUG:boto:auth_path=/hiiqos-61/obj185?versionId=null 2023-07-29T00:50:01.519 DEBUG:boto:Method: GET 2023-07-29T00:50:01.519 DEBUG:boto:Path: /hiiqos-61/obj185?versionId=null 2023-07-29T00:50:01.519 DEBUG:boto:Data: 2023-07-29T00:50:01.520 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.520 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:01.520 DEBUG:boto:Port: 8001 2023-07-29T00:50:01.521 DEBUG:boto:Params: {} 2023-07-29T00:50:01.521 DEBUG:boto:Token: None 2023-07-29T00:50:01.522 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj185?versionId=null 2023-07-29T00:50:01.522 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kz+vMdrT0gne6iwlce7zb9LdTCk= 2023-07-29T00:50:01.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kz+vMdrT0gne6iwlce7zb9LdTCk=', 'Content-Length': '0'} 2023-07-29T00:50:01.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7292a33e2814609-0064c46239-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.525 DEBUG:boto:path=/hiiqos-61/obj185 2023-07-29T00:50:01.526 DEBUG:boto:auth_path=/hiiqos-61/obj185 2023-07-29T00:50:01.526 DEBUG:boto:path=/hiiqos-61/obj185?versionId=null 2023-07-29T00:50:01.526 DEBUG:boto:auth_path=/hiiqos-61/obj185?versionId=null 2023-07-29T00:50:01.527 DEBUG:boto:Method: GET 2023-07-29T00:50:01.527 DEBUG:boto:Path: /hiiqos-61/obj185?versionId=null 2023-07-29T00:50:01.527 DEBUG:boto:Data: 2023-07-29T00:50:01.528 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.528 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:01.529 DEBUG:boto:Port: 8000 2023-07-29T00:50:01.529 DEBUG:boto:Params: {} 2023-07-29T00:50:01.529 DEBUG:boto:Token: None 2023-07-29T00:50:01.530 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj185?versionId=null 2023-07-29T00:50:01.530 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kz+vMdrT0gne6iwlce7zb9LdTCk= 2023-07-29T00:50:01.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kz+vMdrT0gne6iwlce7zb9LdTCk=', 'Content-Length': '0'} 2023-07-29T00:50:01.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f351de553a7c88f-0064c46239-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.533 DEBUG:boto:path=/hiiqos-61/obj185 2023-07-29T00:50:01.534 DEBUG:boto:auth_path=/hiiqos-61/obj185 2023-07-29T00:50:01.534 DEBUG:boto:path=/hiiqos-61/obj185?versionId=null 2023-07-29T00:50:01.534 DEBUG:boto:auth_path=/hiiqos-61/obj185?versionId=null 2023-07-29T00:50:01.535 DEBUG:boto:Method: HEAD 2023-07-29T00:50:01.535 DEBUG:boto:Path: /hiiqos-61/obj185?versionId=null 2023-07-29T00:50:01.535 DEBUG:boto:Data: 2023-07-29T00:50:01.536 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.536 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:01.536 DEBUG:boto:Port: 8001 2023-07-29T00:50:01.537 DEBUG:boto:Params: {} 2023-07-29T00:50:01.537 DEBUG:boto:Token: None 2023-07-29T00:50:01.537 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj185?versionId=null 2023-07-29T00:50:01.538 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/KojdoJyKps2PTqw5GQb/YGXg3o= 2023-07-29T00:50:01.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/KojdoJyKps2PTqw5GQb/YGXg3o=', 'Content-Length': '0'} 2023-07-29T00:50:01.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa35a87128b73f71-0064c46239-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.541 DEBUG:boto:path=/hiiqos-61/obj185 2023-07-29T00:50:01.541 DEBUG:boto:auth_path=/hiiqos-61/obj185 2023-07-29T00:50:01.541 DEBUG:boto:path=/hiiqos-61/obj185?versionId=null 2023-07-29T00:50:01.542 DEBUG:boto:auth_path=/hiiqos-61/obj185?versionId=null 2023-07-29T00:50:01.542 DEBUG:boto:Method: HEAD 2023-07-29T00:50:01.543 DEBUG:boto:Path: /hiiqos-61/obj185?versionId=null 2023-07-29T00:50:01.543 DEBUG:boto:Data: 2023-07-29T00:50:01.544 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.544 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:01.544 DEBUG:boto:Port: 8000 2023-07-29T00:50:01.545 DEBUG:boto:Params: {} 2023-07-29T00:50:01.545 DEBUG:boto:Token: None 2023-07-29T00:50:01.545 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj185?versionId=null 2023-07-29T00:50:01.546 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/KojdoJyKps2PTqw5GQb/YGXg3o= 2023-07-29T00:50:01.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/KojdoJyKps2PTqw5GQb/YGXg3o=', 'Content-Length': '0'} 2023-07-29T00:50:01.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082ef5221d0373407-0064c46239-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-07-29T00:50:01.549 DEBUG:boto:path=/hiiqos-61/obj185 2023-07-29T00:50:01.549 DEBUG:boto:auth_path=/hiiqos-61/obj185 2023-07-29T00:50:01.550 DEBUG:boto:path=/hiiqos-61/obj185?versionId=null 2023-07-29T00:50:01.550 DEBUG:boto:auth_path=/hiiqos-61/obj185?versionId=null 2023-07-29T00:50:01.551 DEBUG:boto:Method: GET 2023-07-29T00:50:01.551 DEBUG:boto:Path: /hiiqos-61/obj185?versionId=null 2023-07-29T00:50:01.552 DEBUG:boto:Data: 2023-07-29T00:50:01.553 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.553 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:01.553 DEBUG:boto:Port: 8001 2023-07-29T00:50:01.554 DEBUG:boto:Params: {} 2023-07-29T00:50:01.554 DEBUG:boto:Token: None 2023-07-29T00:50:01.554 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj185?versionId=null 2023-07-29T00:50:01.555 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kz+vMdrT0gne6iwlce7zb9LdTCk= 2023-07-29T00:50:01.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kz+vMdrT0gne6iwlce7zb9LdTCk=', 'Content-Length': '0'} 2023-07-29T00:50:01.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0c3228b731927eb-0064c46239-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.604 DEBUG:boto:path=/hiiqos-61/obj185 2023-07-29T00:50:01.605 DEBUG:boto:auth_path=/hiiqos-61/obj185 2023-07-29T00:50:01.605 DEBUG:boto:path=/hiiqos-61/obj185?versionId=null 2023-07-29T00:50:01.606 DEBUG:boto:auth_path=/hiiqos-61/obj185?versionId=null 2023-07-29T00:50:01.606 DEBUG:boto:Method: GET 2023-07-29T00:50:01.606 DEBUG:boto:Path: /hiiqos-61/obj185?versionId=null 2023-07-29T00:50:01.607 DEBUG:boto:Data: 2023-07-29T00:50:01.607 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.608 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:01.608 DEBUG:boto:Port: 8000 2023-07-29T00:50:01.609 DEBUG:boto:Params: {} 2023-07-29T00:50:01.609 DEBUG:boto:Token: None 2023-07-29T00:50:01.610 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj185?versionId=null 2023-07-29T00:50:01.610 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kz+vMdrT0gne6iwlce7zb9LdTCk= 2023-07-29T00:50:01.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kz+vMdrT0gne6iwlce7zb9LdTCk=', 'Content-Length': '0'} 2023-07-29T00:50:01.612 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d459eed56739b485-0064c46239-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.613 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:01.613 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:01.614 DEBUG:boto:path=/hiiqos-61/?prefix=obj185 2023-07-29T00:50:01.614 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj185 2023-07-29T00:50:01.615 DEBUG:boto:Method: GET 2023-07-29T00:50:01.615 DEBUG:boto:Path: /hiiqos-61/?prefix=obj185 2023-07-29T00:50:01.615 DEBUG:boto:Data: 2023-07-29T00:50:01.616 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.616 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:01.616 DEBUG:boto:Port: 8001 2023-07-29T00:50:01.617 DEBUG:boto:Params: {} 2023-07-29T00:50:01.617 DEBUG:boto:Token: None 2023-07-29T00:50:01.618 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/ 2023-07-29T00:50:01.618 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+J1dOPW7rrFcNkZdJdAc5Ld3sfw= 2023-07-29T00:50:01.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+J1dOPW7rrFcNkZdJdAc5Ld3sfw=', 'Content-Length': '0'} 2023-07-29T00:50:01.621 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000871b25b2f225d4de-0064c46239-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.621 DEBUG:boto:b'hiiqos-61obj1851000falseobj1852023-07-29T00:45:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:01.622 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:01.622 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:01.623 DEBUG:boto:path=/hiiqos-61/?prefix=obj185 2023-07-29T00:50:01.623 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj185 2023-07-29T00:50:01.623 DEBUG:boto:Method: GET 2023-07-29T00:50:01.624 DEBUG:boto:Path: /hiiqos-61/?prefix=obj185 2023-07-29T00:50:01.624 DEBUG:boto:Data: 2023-07-29T00:50:01.625 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.625 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:01.626 DEBUG:boto:Port: 8000 2023-07-29T00:50:01.626 DEBUG:boto:Params: {} 2023-07-29T00:50:01.626 DEBUG:boto:Token: None 2023-07-29T00:50:01.627 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/ 2023-07-29T00:50:01.627 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+J1dOPW7rrFcNkZdJdAc5Ld3sfw= 2023-07-29T00:50:01.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+J1dOPW7rrFcNkZdJdAc5Ld3sfw=', 'Content-Length': '0'} 2023-07-29T00:50:01.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa19347c11275b82-0064c46239-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.672 DEBUG:boto:b'hiiqos-61obj1851000falseobj1852023-07-29T00:45:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:01.674 DEBUG:boto:path=/hiiqos-61/obj185 2023-07-29T00:50:01.674 DEBUG:boto:auth_path=/hiiqos-61/obj185 2023-07-29T00:50:01.674 DEBUG:boto:Method: HEAD 2023-07-29T00:50:01.675 DEBUG:boto:Path: /hiiqos-61/obj185 2023-07-29T00:50:01.675 DEBUG:boto:Data: 2023-07-29T00:50:01.676 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.676 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:01.676 DEBUG:boto:Port: 8001 2023-07-29T00:50:01.677 DEBUG:boto:Params: {} 2023-07-29T00:50:01.678 DEBUG:boto:Token: None 2023-07-29T00:50:01.678 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj185 2023-07-29T00:50:01.679 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:NJrZ8alSD/4fpAiiZNIoZztjJIs= 2023-07-29T00:50:01.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:NJrZ8alSD/4fpAiiZNIoZztjJIs=', 'Content-Length': '0'} 2023-07-29T00:50:01.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fde8d8898c210c79-0064c46239-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.681 DEBUG:boto:path=/hiiqos-61/obj185 2023-07-29T00:50:01.682 DEBUG:boto:auth_path=/hiiqos-61/obj185 2023-07-29T00:50:01.682 DEBUG:boto:Method: HEAD 2023-07-29T00:50:01.682 DEBUG:boto:Path: /hiiqos-61/obj185 2023-07-29T00:50:01.683 DEBUG:boto:Data: 2023-07-29T00:50:01.683 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.683 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:01.684 DEBUG:boto:Port: 8000 2023-07-29T00:50:01.684 DEBUG:boto:Params: {} 2023-07-29T00:50:01.685 DEBUG:boto:Token: None 2023-07-29T00:50:01.685 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj185 2023-07-29T00:50:01.685 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:NJrZ8alSD/4fpAiiZNIoZztjJIs= 2023-07-29T00:50:01.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:NJrZ8alSD/4fpAiiZNIoZztjJIs=', 'Content-Length': '0'} 2023-07-29T00:50:01.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eda057c13d44e6aa-0064c46239-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-07-29T00:50:01.689 DEBUG:boto:path=/hiiqos-61/obj185 2023-07-29T00:50:01.689 DEBUG:boto:auth_path=/hiiqos-61/obj185 2023-07-29T00:50:01.690 DEBUG:boto:Method: GET 2023-07-29T00:50:01.690 DEBUG:boto:Path: /hiiqos-61/obj185 2023-07-29T00:50:01.690 DEBUG:boto:Data: 2023-07-29T00:50:01.691 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.691 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:01.691 DEBUG:boto:Port: 8001 2023-07-29T00:50:01.692 DEBUG:boto:Params: {} 2023-07-29T00:50:01.692 DEBUG:boto:Token: None 2023-07-29T00:50:01.693 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj185 2023-07-29T00:50:01.693 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WmJ/QQUijQhp53/pg4rb0njnlKo= 2023-07-29T00:50:01.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WmJ/QQUijQhp53/pg4rb0njnlKo=', 'Content-Length': '0'} 2023-07-29T00:50:01.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046d50494faece152-0064c46239-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.696 DEBUG:boto:path=/hiiqos-61/obj185 2023-07-29T00:50:01.696 DEBUG:boto:auth_path=/hiiqos-61/obj185 2023-07-29T00:50:01.697 DEBUG:boto:Method: GET 2023-07-29T00:50:01.697 DEBUG:boto:Path: /hiiqos-61/obj185 2023-07-29T00:50:01.698 DEBUG:boto:Data: 2023-07-29T00:50:01.698 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.698 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:01.699 DEBUG:boto:Port: 8000 2023-07-29T00:50:01.699 DEBUG:boto:Params: {} 2023-07-29T00:50:01.700 DEBUG:boto:Token: None 2023-07-29T00:50:01.700 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj185 2023-07-29T00:50:01.701 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WmJ/QQUijQhp53/pg4rb0njnlKo= 2023-07-29T00:50:01.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WmJ/QQUijQhp53/pg4rb0njnlKo=', 'Content-Length': '0'} 2023-07-29T00:50:01.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8cf99a267a3ff5a-0064c46239-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.703 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:01.704 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:01.704 DEBUG:boto:path=/hiiqos-61/?prefix=obj185 2023-07-29T00:50:01.704 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj185 2023-07-29T00:50:01.705 DEBUG:boto:Method: GET 2023-07-29T00:50:01.705 DEBUG:boto:Path: /hiiqos-61/?prefix=obj185 2023-07-29T00:50:01.706 DEBUG:boto:Data: 2023-07-29T00:50:01.706 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.706 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:01.707 DEBUG:boto:Port: 8001 2023-07-29T00:50:01.707 DEBUG:boto:Params: {} 2023-07-29T00:50:01.707 DEBUG:boto:Token: None 2023-07-29T00:50:01.708 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/ 2023-07-29T00:50:01.708 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+J1dOPW7rrFcNkZdJdAc5Ld3sfw= 2023-07-29T00:50:01.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+J1dOPW7rrFcNkZdJdAc5Ld3sfw=', 'Content-Length': '0'} 2023-07-29T00:50:01.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb0cdfbe883aebcd-0064c46239-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.711 DEBUG:boto:b'hiiqos-61obj1851000falseobj1852023-07-29T00:45:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:01.712 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:01.712 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:01.713 DEBUG:boto:path=/hiiqos-61/?prefix=obj185 2023-07-29T00:50:01.713 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj185 2023-07-29T00:50:01.713 DEBUG:boto:Method: GET 2023-07-29T00:50:01.714 DEBUG:boto:Path: /hiiqos-61/?prefix=obj185 2023-07-29T00:50:01.714 DEBUG:boto:Data: 2023-07-29T00:50:01.714 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.715 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:01.715 DEBUG:boto:Port: 8000 2023-07-29T00:50:01.715 DEBUG:boto:Params: {} 2023-07-29T00:50:01.716 DEBUG:boto:Token: None 2023-07-29T00:50:01.716 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/ 2023-07-29T00:50:01.717 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+J1dOPW7rrFcNkZdJdAc5Ld3sfw= 2023-07-29T00:50:01.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+J1dOPW7rrFcNkZdJdAc5Ld3sfw=', 'Content-Length': '0'} 2023-07-29T00:50:01.721 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007a7b0cb85f186d3-0064c46239-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.721 DEBUG:boto:b'hiiqos-61obj1851000falseobj1852023-07-29T00:45:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:01.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-07-29T00:50:01.722 DEBUG:boto:path=/hiiqos-61/obj19 2023-07-29T00:50:01.723 DEBUG:boto:auth_path=/hiiqos-61/obj19 2023-07-29T00:50:01.723 DEBUG:boto:path=/hiiqos-61/obj19?versionId=null 2023-07-29T00:50:01.723 DEBUG:boto:auth_path=/hiiqos-61/obj19?versionId=null 2023-07-29T00:50:01.724 DEBUG:boto:Method: GET 2023-07-29T00:50:01.724 DEBUG:boto:Path: /hiiqos-61/obj19?versionId=null 2023-07-29T00:50:01.725 DEBUG:boto:Data: 2023-07-29T00:50:01.725 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.725 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:01.726 DEBUG:boto:Port: 8001 2023-07-29T00:50:01.726 DEBUG:boto:Params: {} 2023-07-29T00:50:01.726 DEBUG:boto:Token: None 2023-07-29T00:50:01.727 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj19?versionId=null 2023-07-29T00:50:01.727 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:f7DNVRY7Zhl6Fbg7N4PokLu49P8= 2023-07-29T00:50:01.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:f7DNVRY7Zhl6Fbg7N4PokLu49P8=', 'Content-Length': '0'} 2023-07-29T00:50:01.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065274a61586cd309-0064c46239-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.730 DEBUG:boto:path=/hiiqos-61/obj19 2023-07-29T00:50:01.731 DEBUG:boto:auth_path=/hiiqos-61/obj19 2023-07-29T00:50:01.731 DEBUG:boto:path=/hiiqos-61/obj19?versionId=null 2023-07-29T00:50:01.732 DEBUG:boto:auth_path=/hiiqos-61/obj19?versionId=null 2023-07-29T00:50:01.732 DEBUG:boto:Method: GET 2023-07-29T00:50:01.732 DEBUG:boto:Path: /hiiqos-61/obj19?versionId=null 2023-07-29T00:50:01.733 DEBUG:boto:Data: 2023-07-29T00:50:01.733 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.733 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:01.734 DEBUG:boto:Port: 8000 2023-07-29T00:50:01.734 DEBUG:boto:Params: {} 2023-07-29T00:50:01.734 DEBUG:boto:Token: None 2023-07-29T00:50:01.735 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj19?versionId=null 2023-07-29T00:50:01.735 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:f7DNVRY7Zhl6Fbg7N4PokLu49P8= 2023-07-29T00:50:01.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:f7DNVRY7Zhl6Fbg7N4PokLu49P8=', 'Content-Length': '0'} 2023-07-29T00:50:01.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017f55d6d7bbc6460-0064c46239-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.738 DEBUG:boto:path=/hiiqos-61/obj19 2023-07-29T00:50:01.738 DEBUG:boto:auth_path=/hiiqos-61/obj19 2023-07-29T00:50:01.739 DEBUG:boto:path=/hiiqos-61/obj19?versionId=null 2023-07-29T00:50:01.739 DEBUG:boto:auth_path=/hiiqos-61/obj19?versionId=null 2023-07-29T00:50:01.739 DEBUG:boto:Method: HEAD 2023-07-29T00:50:01.740 DEBUG:boto:Path: /hiiqos-61/obj19?versionId=null 2023-07-29T00:50:01.740 DEBUG:boto:Data: 2023-07-29T00:50:01.741 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.741 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:01.741 DEBUG:boto:Port: 8001 2023-07-29T00:50:01.742 DEBUG:boto:Params: {} 2023-07-29T00:50:01.742 DEBUG:boto:Token: None 2023-07-29T00:50:01.742 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj19?versionId=null 2023-07-29T00:50:01.743 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qtZyWfrZEaFAlqTOuQMBQ4BzlKA= 2023-07-29T00:50:01.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qtZyWfrZEaFAlqTOuQMBQ4BzlKA=', 'Content-Length': '0'} 2023-07-29T00:50:01.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000badc8b85df28b398-0064c46239-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.745 DEBUG:boto:path=/hiiqos-61/obj19 2023-07-29T00:50:01.746 DEBUG:boto:auth_path=/hiiqos-61/obj19 2023-07-29T00:50:01.746 DEBUG:boto:path=/hiiqos-61/obj19?versionId=null 2023-07-29T00:50:01.746 DEBUG:boto:auth_path=/hiiqos-61/obj19?versionId=null 2023-07-29T00:50:01.747 DEBUG:boto:Method: HEAD 2023-07-29T00:50:01.747 DEBUG:boto:Path: /hiiqos-61/obj19?versionId=null 2023-07-29T00:50:01.747 DEBUG:boto:Data: 2023-07-29T00:50:01.748 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.748 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:01.748 DEBUG:boto:Port: 8000 2023-07-29T00:50:01.749 DEBUG:boto:Params: {} 2023-07-29T00:50:01.749 DEBUG:boto:Token: None 2023-07-29T00:50:01.749 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj19?versionId=null 2023-07-29T00:50:01.750 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qtZyWfrZEaFAlqTOuQMBQ4BzlKA= 2023-07-29T00:50:01.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qtZyWfrZEaFAlqTOuQMBQ4BzlKA=', 'Content-Length': '0'} 2023-07-29T00:50:01.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bd50fd312b02bd6-0064c46239-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-07-29T00:50:01.753 DEBUG:boto:path=/hiiqos-61/obj19 2023-07-29T00:50:01.753 DEBUG:boto:auth_path=/hiiqos-61/obj19 2023-07-29T00:50:01.754 DEBUG:boto:path=/hiiqos-61/obj19?versionId=null 2023-07-29T00:50:01.754 DEBUG:boto:auth_path=/hiiqos-61/obj19?versionId=null 2023-07-29T00:50:01.754 DEBUG:boto:Method: GET 2023-07-29T00:50:01.755 DEBUG:boto:Path: /hiiqos-61/obj19?versionId=null 2023-07-29T00:50:01.755 DEBUG:boto:Data: 2023-07-29T00:50:01.755 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.756 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:01.756 DEBUG:boto:Port: 8001 2023-07-29T00:50:01.756 DEBUG:boto:Params: {} 2023-07-29T00:50:01.757 DEBUG:boto:Token: None 2023-07-29T00:50:01.757 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj19?versionId=null 2023-07-29T00:50:01.758 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:f7DNVRY7Zhl6Fbg7N4PokLu49P8= 2023-07-29T00:50:01.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:f7DNVRY7Zhl6Fbg7N4PokLu49P8=', 'Content-Length': '0'} 2023-07-29T00:50:01.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db409118eaed2d9c-0064c46239-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.804 DEBUG:boto:path=/hiiqos-61/obj19 2023-07-29T00:50:01.805 DEBUG:boto:auth_path=/hiiqos-61/obj19 2023-07-29T00:50:01.805 DEBUG:boto:path=/hiiqos-61/obj19?versionId=null 2023-07-29T00:50:01.806 DEBUG:boto:auth_path=/hiiqos-61/obj19?versionId=null 2023-07-29T00:50:01.806 DEBUG:boto:Method: GET 2023-07-29T00:50:01.806 DEBUG:boto:Path: /hiiqos-61/obj19?versionId=null 2023-07-29T00:50:01.807 DEBUG:boto:Data: 2023-07-29T00:50:01.807 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.807 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:01.808 DEBUG:boto:Port: 8000 2023-07-29T00:50:01.808 DEBUG:boto:Params: {} 2023-07-29T00:50:01.809 DEBUG:boto:Token: None 2023-07-29T00:50:01.809 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj19?versionId=null 2023-07-29T00:50:01.810 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:f7DNVRY7Zhl6Fbg7N4PokLu49P8= 2023-07-29T00:50:01.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:f7DNVRY7Zhl6Fbg7N4PokLu49P8=', 'Content-Length': '0'} 2023-07-29T00:50:01.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000628cc89f075d084a-0064c46239-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.813 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:01.813 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:01.814 DEBUG:boto:path=/hiiqos-61/?prefix=obj19 2023-07-29T00:50:01.814 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj19 2023-07-29T00:50:01.814 DEBUG:boto:Method: GET 2023-07-29T00:50:01.815 DEBUG:boto:Path: /hiiqos-61/?prefix=obj19 2023-07-29T00:50:01.815 DEBUG:boto:Data: 2023-07-29T00:50:01.816 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.816 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:01.816 DEBUG:boto:Port: 8001 2023-07-29T00:50:01.817 DEBUG:boto:Params: {} 2023-07-29T00:50:01.817 DEBUG:boto:Token: None 2023-07-29T00:50:01.817 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/ 2023-07-29T00:50:01.818 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+J1dOPW7rrFcNkZdJdAc5Ld3sfw= 2023-07-29T00:50:01.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+J1dOPW7rrFcNkZdJdAc5Ld3sfw=', 'Content-Length': '0'} 2023-07-29T00:50:01.820 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1b5070520eeeda0-0064c46239-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.821 DEBUG:boto:b'hiiqos-61obj191000falseobj192023-07-29T00:45:01.674Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:01.822 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:01.822 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:01.822 DEBUG:boto:path=/hiiqos-61/?prefix=obj19 2023-07-29T00:50:01.823 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj19 2023-07-29T00:50:01.823 DEBUG:boto:Method: GET 2023-07-29T00:50:01.824 DEBUG:boto:Path: /hiiqos-61/?prefix=obj19 2023-07-29T00:50:01.824 DEBUG:boto:Data: 2023-07-29T00:50:01.824 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.825 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:01.825 DEBUG:boto:Port: 8000 2023-07-29T00:50:01.825 DEBUG:boto:Params: {} 2023-07-29T00:50:01.826 DEBUG:boto:Token: None 2023-07-29T00:50:01.826 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/ 2023-07-29T00:50:01.827 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+J1dOPW7rrFcNkZdJdAc5Ld3sfw= 2023-07-29T00:50:01.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+J1dOPW7rrFcNkZdJdAc5Ld3sfw=', 'Content-Length': '0'} 2023-07-29T00:50:01.831 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d733248ce08024fc-0064c46239-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.831 DEBUG:boto:b'hiiqos-61obj191000falseobj192023-07-29T00:45:01.674Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:01.832 DEBUG:boto:path=/hiiqos-61/obj19 2023-07-29T00:50:01.832 DEBUG:boto:auth_path=/hiiqos-61/obj19 2023-07-29T00:50:01.832 DEBUG:boto:Method: HEAD 2023-07-29T00:50:01.833 DEBUG:boto:Path: /hiiqos-61/obj19 2023-07-29T00:50:01.833 DEBUG:boto:Data: 2023-07-29T00:50:01.834 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.834 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:01.834 DEBUG:boto:Port: 8001 2023-07-29T00:50:01.834 DEBUG:boto:Params: {} 2023-07-29T00:50:01.835 DEBUG:boto:Token: None 2023-07-29T00:50:01.835 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj19 2023-07-29T00:50:01.836 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/awLThyzkRaLINZ1/XrDo7WPIXk= 2023-07-29T00:50:01.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/awLThyzkRaLINZ1/XrDo7WPIXk=', 'Content-Length': '0'} 2023-07-29T00:50:01.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001dbdb8b72e5dabb3-0064c46239-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.839 DEBUG:boto:path=/hiiqos-61/obj19 2023-07-29T00:50:01.839 DEBUG:boto:auth_path=/hiiqos-61/obj19 2023-07-29T00:50:01.839 DEBUG:boto:Method: HEAD 2023-07-29T00:50:01.840 DEBUG:boto:Path: /hiiqos-61/obj19 2023-07-29T00:50:01.840 DEBUG:boto:Data: 2023-07-29T00:50:01.840 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.841 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:01.841 DEBUG:boto:Port: 8000 2023-07-29T00:50:01.842 DEBUG:boto:Params: {} 2023-07-29T00:50:01.842 DEBUG:boto:Token: None 2023-07-29T00:50:01.842 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj19 2023-07-29T00:50:01.843 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/awLThyzkRaLINZ1/XrDo7WPIXk= 2023-07-29T00:50:01.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/awLThyzkRaLINZ1/XrDo7WPIXk=', 'Content-Length': '0'} 2023-07-29T00:50:01.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ac69dcaa02a43dc-0064c46239-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-07-29T00:50:01.846 DEBUG:boto:path=/hiiqos-61/obj19 2023-07-29T00:50:01.846 DEBUG:boto:auth_path=/hiiqos-61/obj19 2023-07-29T00:50:01.847 DEBUG:boto:Method: GET 2023-07-29T00:50:01.847 DEBUG:boto:Path: /hiiqos-61/obj19 2023-07-29T00:50:01.847 DEBUG:boto:Data: 2023-07-29T00:50:01.848 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.848 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:01.848 DEBUG:boto:Port: 8001 2023-07-29T00:50:01.849 DEBUG:boto:Params: {} 2023-07-29T00:50:01.849 DEBUG:boto:Token: None 2023-07-29T00:50:01.850 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj19 2023-07-29T00:50:01.850 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rR+uuKfxxgheOzH9wflcd1LYWIY= 2023-07-29T00:50:01.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rR+uuKfxxgheOzH9wflcd1LYWIY=', 'Content-Length': '0'} 2023-07-29T00:50:01.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059061c0d5c206650-0064c46239-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.853 DEBUG:boto:path=/hiiqos-61/obj19 2023-07-29T00:50:01.853 DEBUG:boto:auth_path=/hiiqos-61/obj19 2023-07-29T00:50:01.853 DEBUG:boto:Method: GET 2023-07-29T00:50:01.854 DEBUG:boto:Path: /hiiqos-61/obj19 2023-07-29T00:50:01.854 DEBUG:boto:Data: 2023-07-29T00:50:01.854 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.855 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:01.855 DEBUG:boto:Port: 8000 2023-07-29T00:50:01.855 DEBUG:boto:Params: {} 2023-07-29T00:50:01.856 DEBUG:boto:Token: None 2023-07-29T00:50:01.856 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj19 2023-07-29T00:50:01.857 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rR+uuKfxxgheOzH9wflcd1LYWIY= 2023-07-29T00:50:01.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rR+uuKfxxgheOzH9wflcd1LYWIY=', 'Content-Length': '0'} 2023-07-29T00:50:01.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ea6269e81caf876-0064c46239-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.900 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:01.901 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:01.901 DEBUG:boto:path=/hiiqos-61/?prefix=obj19 2023-07-29T00:50:01.902 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj19 2023-07-29T00:50:01.902 DEBUG:boto:Method: GET 2023-07-29T00:50:01.903 DEBUG:boto:Path: /hiiqos-61/?prefix=obj19 2023-07-29T00:50:01.903 DEBUG:boto:Data: 2023-07-29T00:50:01.903 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.904 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:01.904 DEBUG:boto:Port: 8001 2023-07-29T00:50:01.904 DEBUG:boto:Params: {} 2023-07-29T00:50:01.905 DEBUG:boto:Token: None 2023-07-29T00:50:01.905 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/ 2023-07-29T00:50:01.906 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+J1dOPW7rrFcNkZdJdAc5Ld3sfw= 2023-07-29T00:50:01.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+J1dOPW7rrFcNkZdJdAc5Ld3sfw=', 'Content-Length': '0'} 2023-07-29T00:50:01.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000402938fc2414b540-0064c46239-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.909 DEBUG:boto:b'hiiqos-61obj191000falseobj192023-07-29T00:45:01.674Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:01.910 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:01.910 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:01.911 DEBUG:boto:path=/hiiqos-61/?prefix=obj19 2023-07-29T00:50:01.911 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj19 2023-07-29T00:50:01.911 DEBUG:boto:Method: GET 2023-07-29T00:50:01.912 DEBUG:boto:Path: /hiiqos-61/?prefix=obj19 2023-07-29T00:50:01.912 DEBUG:boto:Data: 2023-07-29T00:50:01.912 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.913 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:01.913 DEBUG:boto:Port: 8000 2023-07-29T00:50:01.914 DEBUG:boto:Params: {} 2023-07-29T00:50:01.914 DEBUG:boto:Token: None 2023-07-29T00:50:01.915 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/ 2023-07-29T00:50:01.915 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+J1dOPW7rrFcNkZdJdAc5Ld3sfw= 2023-07-29T00:50:01.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+J1dOPW7rrFcNkZdJdAc5Ld3sfw=', 'Content-Length': '0'} 2023-07-29T00:50:01.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be0d91f2619e17bb-0064c46239-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.920 DEBUG:boto:b'hiiqos-61obj191000falseobj192023-07-29T00:45:01.674Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:01.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-07-29T00:50:01.921 DEBUG:boto:path=/hiiqos-61/obj22 2023-07-29T00:50:01.921 DEBUG:boto:auth_path=/hiiqos-61/obj22 2023-07-29T00:50:01.922 DEBUG:boto:path=/hiiqos-61/obj22?versionId=null 2023-07-29T00:50:01.922 DEBUG:boto:auth_path=/hiiqos-61/obj22?versionId=null 2023-07-29T00:50:01.922 DEBUG:boto:Method: GET 2023-07-29T00:50:01.923 DEBUG:boto:Path: /hiiqos-61/obj22?versionId=null 2023-07-29T00:50:01.923 DEBUG:boto:Data: 2023-07-29T00:50:01.923 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.924 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:01.924 DEBUG:boto:Port: 8001 2023-07-29T00:50:01.924 DEBUG:boto:Params: {} 2023-07-29T00:50:01.925 DEBUG:boto:Token: None 2023-07-29T00:50:01.925 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj22?versionId=null 2023-07-29T00:50:01.926 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jsKg0/C8WO00c8cym0F35EBsEdk= 2023-07-29T00:50:01.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jsKg0/C8WO00c8cym0F35EBsEdk=', 'Content-Length': '0'} 2023-07-29T00:50:01.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c418b22bd2531cb4-0064c46239-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.968 DEBUG:boto:path=/hiiqos-61/obj22 2023-07-29T00:50:01.969 DEBUG:boto:auth_path=/hiiqos-61/obj22 2023-07-29T00:50:01.969 DEBUG:boto:path=/hiiqos-61/obj22?versionId=null 2023-07-29T00:50:01.970 DEBUG:boto:auth_path=/hiiqos-61/obj22?versionId=null 2023-07-29T00:50:01.970 DEBUG:boto:Method: GET 2023-07-29T00:50:01.971 DEBUG:boto:Path: /hiiqos-61/obj22?versionId=null 2023-07-29T00:50:01.971 DEBUG:boto:Data: 2023-07-29T00:50:01.972 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.972 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:01.972 DEBUG:boto:Port: 8000 2023-07-29T00:50:01.974 DEBUG:boto:Params: {} 2023-07-29T00:50:01.975 DEBUG:boto:Token: None 2023-07-29T00:50:01.975 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj22?versionId=null 2023-07-29T00:50:01.976 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jsKg0/C8WO00c8cym0F35EBsEdk= 2023-07-29T00:50:01.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jsKg0/C8WO00c8cym0F35EBsEdk=', 'Content-Length': '0'} 2023-07-29T00:50:01.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a371515c3b6b6e6-0064c46239-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.979 DEBUG:boto:path=/hiiqos-61/obj22 2023-07-29T00:50:01.979 DEBUG:boto:auth_path=/hiiqos-61/obj22 2023-07-29T00:50:01.979 DEBUG:boto:path=/hiiqos-61/obj22?versionId=null 2023-07-29T00:50:01.980 DEBUG:boto:auth_path=/hiiqos-61/obj22?versionId=null 2023-07-29T00:50:01.980 DEBUG:boto:Method: HEAD 2023-07-29T00:50:01.981 DEBUG:boto:Path: /hiiqos-61/obj22?versionId=null 2023-07-29T00:50:01.981 DEBUG:boto:Data: 2023-07-29T00:50:01.981 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.982 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:01.982 DEBUG:boto:Port: 8001 2023-07-29T00:50:01.983 DEBUG:boto:Params: {} 2023-07-29T00:50:01.983 DEBUG:boto:Token: None 2023-07-29T00:50:01.984 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj22?versionId=null 2023-07-29T00:50:01.984 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7mVJcWXGZrY02vY4xxgzRHBnjA4= 2023-07-29T00:50:01.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7mVJcWXGZrY02vY4xxgzRHBnjA4=', 'Content-Length': '0'} 2023-07-29T00:50:01.987 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ef59fbcc3be5573-0064c46239-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.988 DEBUG:boto:path=/hiiqos-61/obj22 2023-07-29T00:50:01.988 DEBUG:boto:auth_path=/hiiqos-61/obj22 2023-07-29T00:50:01.988 DEBUG:boto:path=/hiiqos-61/obj22?versionId=null 2023-07-29T00:50:01.989 DEBUG:boto:auth_path=/hiiqos-61/obj22?versionId=null 2023-07-29T00:50:01.989 DEBUG:boto:Method: HEAD 2023-07-29T00:50:01.989 DEBUG:boto:Path: /hiiqos-61/obj22?versionId=null 2023-07-29T00:50:01.990 DEBUG:boto:Data: 2023-07-29T00:50:01.990 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.990 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:01.991 DEBUG:boto:Port: 8000 2023-07-29T00:50:01.991 DEBUG:boto:Params: {} 2023-07-29T00:50:01.992 DEBUG:boto:Token: None 2023-07-29T00:50:01.992 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:01 GMT /hiiqos-61/obj22?versionId=null 2023-07-29T00:50:01.992 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7mVJcWXGZrY02vY4xxgzRHBnjA4= 2023-07-29T00:50:01.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7mVJcWXGZrY02vY4xxgzRHBnjA4=', 'Content-Length': '0'} 2023-07-29T00:50:01.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000789ff4dcb88be48-0064c46239-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:01.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-07-29T00:50:01.996 DEBUG:boto:path=/hiiqos-61/obj22 2023-07-29T00:50:01.996 DEBUG:boto:auth_path=/hiiqos-61/obj22 2023-07-29T00:50:01.997 DEBUG:boto:path=/hiiqos-61/obj22?versionId=null 2023-07-29T00:50:01.997 DEBUG:boto:auth_path=/hiiqos-61/obj22?versionId=null 2023-07-29T00:50:01.997 DEBUG:boto:Method: GET 2023-07-29T00:50:01.998 DEBUG:boto:Path: /hiiqos-61/obj22?versionId=null 2023-07-29T00:50:01.999 DEBUG:boto:Data: 2023-07-29T00:50:01.999 DEBUG:boto:Headers: {} 2023-07-29T00:50:01.999 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:02.000 DEBUG:boto:Port: 8001 2023-07-29T00:50:02.000 DEBUG:boto:Params: {} 2023-07-29T00:50:02.001 DEBUG:boto:Token: None 2023-07-29T00:50:02.001 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj22?versionId=null 2023-07-29T00:50:02.001 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gEbB2X79AZUD2muy1tudXKBfH94= 2023-07-29T00:50:02.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gEbB2X79AZUD2muy1tudXKBfH94=', 'Content-Length': '0'} 2023-07-29T00:50:02.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000509874f3cfec03cc-0064c46239-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.005 DEBUG:boto:path=/hiiqos-61/obj22 2023-07-29T00:50:02.005 DEBUG:boto:auth_path=/hiiqos-61/obj22 2023-07-29T00:50:02.005 DEBUG:boto:path=/hiiqos-61/obj22?versionId=null 2023-07-29T00:50:02.006 DEBUG:boto:auth_path=/hiiqos-61/obj22?versionId=null 2023-07-29T00:50:02.006 DEBUG:boto:Method: GET 2023-07-29T00:50:02.007 DEBUG:boto:Path: /hiiqos-61/obj22?versionId=null 2023-07-29T00:50:02.007 DEBUG:boto:Data: 2023-07-29T00:50:02.007 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.008 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:02.008 DEBUG:boto:Port: 8000 2023-07-29T00:50:02.009 DEBUG:boto:Params: {} 2023-07-29T00:50:02.009 DEBUG:boto:Token: None 2023-07-29T00:50:02.010 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj22?versionId=null 2023-07-29T00:50:02.010 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gEbB2X79AZUD2muy1tudXKBfH94= 2023-07-29T00:50:02.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gEbB2X79AZUD2muy1tudXKBfH94=', 'Content-Length': '0'} 2023-07-29T00:50:02.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f385e68be9c1b8f-0064c46239-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.013 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:02.014 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:02.014 DEBUG:boto:path=/hiiqos-61/?prefix=obj22 2023-07-29T00:50:02.014 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj22 2023-07-29T00:50:02.015 DEBUG:boto:Method: GET 2023-07-29T00:50:02.015 DEBUG:boto:Path: /hiiqos-61/?prefix=obj22 2023-07-29T00:50:02.015 DEBUG:boto:Data: 2023-07-29T00:50:02.016 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.016 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:02.017 DEBUG:boto:Port: 8001 2023-07-29T00:50:02.017 DEBUG:boto:Params: {} 2023-07-29T00:50:02.018 DEBUG:boto:Token: None 2023-07-29T00:50:02.018 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/ 2023-07-29T00:50:02.018 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6jMSgLwXCW6tFHtDjt7WrDtXJ2g= 2023-07-29T00:50:02.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6jMSgLwXCW6tFHtDjt7WrDtXJ2g=', 'Content-Length': '0'} 2023-07-29T00:50:02.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000702bd2d1598de05c-0064c46239-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.022 DEBUG:boto:b'hiiqos-61obj221000falseobj222023-07-29T00:45:01.061Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:02.024 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:02.025 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:02.025 DEBUG:boto:path=/hiiqos-61/?prefix=obj22 2023-07-29T00:50:02.026 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj22 2023-07-29T00:50:02.027 DEBUG:boto:Method: GET 2023-07-29T00:50:02.028 DEBUG:boto:Path: /hiiqos-61/?prefix=obj22 2023-07-29T00:50:02.029 DEBUG:boto:Data: 2023-07-29T00:50:02.030 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.030 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:02.031 DEBUG:boto:Port: 8000 2023-07-29T00:50:02.031 DEBUG:boto:Params: {} 2023-07-29T00:50:02.032 DEBUG:boto:Token: None 2023-07-29T00:50:02.032 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/ 2023-07-29T00:50:02.033 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6jMSgLwXCW6tFHtDjt7WrDtXJ2g= 2023-07-29T00:50:02.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6jMSgLwXCW6tFHtDjt7WrDtXJ2g=', 'Content-Length': '0'} 2023-07-29T00:50:02.038 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cd12a5e9d259e24-0064c46239-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.039 DEBUG:boto:b'hiiqos-61obj221000falseobj222023-07-29T00:45:01.061Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:02.039 DEBUG:boto:path=/hiiqos-61/obj22 2023-07-29T00:50:02.040 DEBUG:boto:auth_path=/hiiqos-61/obj22 2023-07-29T00:50:02.040 DEBUG:boto:Method: HEAD 2023-07-29T00:50:02.041 DEBUG:boto:Path: /hiiqos-61/obj22 2023-07-29T00:50:02.041 DEBUG:boto:Data: 2023-07-29T00:50:02.041 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.042 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:02.042 DEBUG:boto:Port: 8001 2023-07-29T00:50:02.042 DEBUG:boto:Params: {} 2023-07-29T00:50:02.043 DEBUG:boto:Token: None 2023-07-29T00:50:02.043 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj22 2023-07-29T00:50:02.044 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FNwa1gHm5T8KMXmtVQ4udx6bF68= 2023-07-29T00:50:02.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FNwa1gHm5T8KMXmtVQ4udx6bF68=', 'Content-Length': '0'} 2023-07-29T00:50:02.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006efb8498c990594f-0064c46239-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.046 DEBUG:boto:path=/hiiqos-61/obj22 2023-07-29T00:50:02.047 DEBUG:boto:auth_path=/hiiqos-61/obj22 2023-07-29T00:50:02.047 DEBUG:boto:Method: HEAD 2023-07-29T00:50:02.048 DEBUG:boto:Path: /hiiqos-61/obj22 2023-07-29T00:50:02.048 DEBUG:boto:Data: 2023-07-29T00:50:02.048 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.049 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:02.049 DEBUG:boto:Port: 8000 2023-07-29T00:50:02.049 DEBUG:boto:Params: {} 2023-07-29T00:50:02.050 DEBUG:boto:Token: None 2023-07-29T00:50:02.050 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj22 2023-07-29T00:50:02.050 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FNwa1gHm5T8KMXmtVQ4udx6bF68= 2023-07-29T00:50:02.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FNwa1gHm5T8KMXmtVQ4udx6bF68=', 'Content-Length': '0'} 2023-07-29T00:50:02.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000434ce2b64c3df63c-0064c46239-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-07-29T00:50:02.054 DEBUG:boto:path=/hiiqos-61/obj22 2023-07-29T00:50:02.054 DEBUG:boto:auth_path=/hiiqos-61/obj22 2023-07-29T00:50:02.055 DEBUG:boto:Method: GET 2023-07-29T00:50:02.055 DEBUG:boto:Path: /hiiqos-61/obj22 2023-07-29T00:50:02.056 DEBUG:boto:Data: 2023-07-29T00:50:02.056 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.057 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:02.057 DEBUG:boto:Port: 8001 2023-07-29T00:50:02.057 DEBUG:boto:Params: {} 2023-07-29T00:50:02.058 DEBUG:boto:Token: None 2023-07-29T00:50:02.059 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj22 2023-07-29T00:50:02.059 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hhZ5h+1MpNDiPPzLJPUwijsVn9Y= 2023-07-29T00:50:02.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hhZ5h+1MpNDiPPzLJPUwijsVn9Y=', 'Content-Length': '0'} 2023-07-29T00:50:02.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a4c9b382d93d8fe-0064c46239-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.063 DEBUG:boto:path=/hiiqos-61/obj22 2023-07-29T00:50:02.063 DEBUG:boto:auth_path=/hiiqos-61/obj22 2023-07-29T00:50:02.064 DEBUG:boto:Method: GET 2023-07-29T00:50:02.065 DEBUG:boto:Path: /hiiqos-61/obj22 2023-07-29T00:50:02.065 DEBUG:boto:Data: 2023-07-29T00:50:02.066 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.067 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:02.067 DEBUG:boto:Port: 8000 2023-07-29T00:50:02.068 DEBUG:boto:Params: {} 2023-07-29T00:50:02.069 DEBUG:boto:Token: None 2023-07-29T00:50:02.069 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj22 2023-07-29T00:50:02.070 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hhZ5h+1MpNDiPPzLJPUwijsVn9Y= 2023-07-29T00:50:02.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hhZ5h+1MpNDiPPzLJPUwijsVn9Y=', 'Content-Length': '0'} 2023-07-29T00:50:02.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce090b7f2f82b389-0064c4623a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.116 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:02.117 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:02.117 DEBUG:boto:path=/hiiqos-61/?prefix=obj22 2023-07-29T00:50:02.118 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj22 2023-07-29T00:50:02.118 DEBUG:boto:Method: GET 2023-07-29T00:50:02.118 DEBUG:boto:Path: /hiiqos-61/?prefix=obj22 2023-07-29T00:50:02.119 DEBUG:boto:Data: 2023-07-29T00:50:02.119 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.119 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:02.120 DEBUG:boto:Port: 8001 2023-07-29T00:50:02.120 DEBUG:boto:Params: {} 2023-07-29T00:50:02.121 DEBUG:boto:Token: None 2023-07-29T00:50:02.121 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/ 2023-07-29T00:50:02.122 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6jMSgLwXCW6tFHtDjt7WrDtXJ2g= 2023-07-29T00:50:02.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6jMSgLwXCW6tFHtDjt7WrDtXJ2g=', 'Content-Length': '0'} 2023-07-29T00:50:02.125 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000811e98047aa44d23-0064c4623a-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.125 DEBUG:boto:b'hiiqos-61obj221000falseobj222023-07-29T00:45:01.061Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:02.126 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:02.126 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:02.127 DEBUG:boto:path=/hiiqos-61/?prefix=obj22 2023-07-29T00:50:02.127 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj22 2023-07-29T00:50:02.127 DEBUG:boto:Method: GET 2023-07-29T00:50:02.128 DEBUG:boto:Path: /hiiqos-61/?prefix=obj22 2023-07-29T00:50:02.128 DEBUG:boto:Data: 2023-07-29T00:50:02.129 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.129 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:02.129 DEBUG:boto:Port: 8000 2023-07-29T00:50:02.130 DEBUG:boto:Params: {} 2023-07-29T00:50:02.130 DEBUG:boto:Token: None 2023-07-29T00:50:02.130 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/ 2023-07-29T00:50:02.131 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6jMSgLwXCW6tFHtDjt7WrDtXJ2g= 2023-07-29T00:50:02.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6jMSgLwXCW6tFHtDjt7WrDtXJ2g=', 'Content-Length': '0'} 2023-07-29T00:50:02.135 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002777bd7c2e93e59f-0064c4623a-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.136 DEBUG:boto:b'hiiqos-61obj221000falseobj222023-07-29T00:45:01.061Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:02.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-07-29T00:50:02.137 DEBUG:boto:path=/hiiqos-61/obj23 2023-07-29T00:50:02.137 DEBUG:boto:auth_path=/hiiqos-61/obj23 2023-07-29T00:50:02.137 DEBUG:boto:path=/hiiqos-61/obj23?versionId=null 2023-07-29T00:50:02.138 DEBUG:boto:auth_path=/hiiqos-61/obj23?versionId=null 2023-07-29T00:50:02.138 DEBUG:boto:Method: GET 2023-07-29T00:50:02.139 DEBUG:boto:Path: /hiiqos-61/obj23?versionId=null 2023-07-29T00:50:02.139 DEBUG:boto:Data: 2023-07-29T00:50:02.139 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.140 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:02.140 DEBUG:boto:Port: 8001 2023-07-29T00:50:02.140 DEBUG:boto:Params: {} 2023-07-29T00:50:02.141 DEBUG:boto:Token: None 2023-07-29T00:50:02.141 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj23?versionId=null 2023-07-29T00:50:02.142 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I00fuXx8YNUAyso1s6fTaP2BrQo= 2023-07-29T00:50:02.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I00fuXx8YNUAyso1s6fTaP2BrQo=', 'Content-Length': '0'} 2023-07-29T00:50:02.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002264b65b7a6ffb1-0064c4623a-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.189 DEBUG:boto:path=/hiiqos-61/obj23 2023-07-29T00:50:02.189 DEBUG:boto:auth_path=/hiiqos-61/obj23 2023-07-29T00:50:02.190 DEBUG:boto:path=/hiiqos-61/obj23?versionId=null 2023-07-29T00:50:02.190 DEBUG:boto:auth_path=/hiiqos-61/obj23?versionId=null 2023-07-29T00:50:02.190 DEBUG:boto:Method: GET 2023-07-29T00:50:02.191 DEBUG:boto:Path: /hiiqos-61/obj23?versionId=null 2023-07-29T00:50:02.191 DEBUG:boto:Data: 2023-07-29T00:50:02.191 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.192 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:02.192 DEBUG:boto:Port: 8000 2023-07-29T00:50:02.193 DEBUG:boto:Params: {} 2023-07-29T00:50:02.193 DEBUG:boto:Token: None 2023-07-29T00:50:02.194 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj23?versionId=null 2023-07-29T00:50:02.194 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I00fuXx8YNUAyso1s6fTaP2BrQo= 2023-07-29T00:50:02.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I00fuXx8YNUAyso1s6fTaP2BrQo=', 'Content-Length': '0'} 2023-07-29T00:50:02.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba559ef3ad199ada-0064c4623a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.197 DEBUG:boto:path=/hiiqos-61/obj23 2023-07-29T00:50:02.197 DEBUG:boto:auth_path=/hiiqos-61/obj23 2023-07-29T00:50:02.198 DEBUG:boto:path=/hiiqos-61/obj23?versionId=null 2023-07-29T00:50:02.198 DEBUG:boto:auth_path=/hiiqos-61/obj23?versionId=null 2023-07-29T00:50:02.198 DEBUG:boto:Method: HEAD 2023-07-29T00:50:02.199 DEBUG:boto:Path: /hiiqos-61/obj23?versionId=null 2023-07-29T00:50:02.199 DEBUG:boto:Data: 2023-07-29T00:50:02.199 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.200 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:02.200 DEBUG:boto:Port: 8001 2023-07-29T00:50:02.201 DEBUG:boto:Params: {} 2023-07-29T00:50:02.201 DEBUG:boto:Token: None 2023-07-29T00:50:02.202 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj23?versionId=null 2023-07-29T00:50:02.202 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6lFjgpCAWWa+YO8/0YDR/hU/SJY= 2023-07-29T00:50:02.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6lFjgpCAWWa+YO8/0YDR/hU/SJY=', 'Content-Length': '0'} 2023-07-29T00:50:02.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c46e95a75ea28e8-0064c4623a-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.205 DEBUG:boto:path=/hiiqos-61/obj23 2023-07-29T00:50:02.205 DEBUG:boto:auth_path=/hiiqos-61/obj23 2023-07-29T00:50:02.206 DEBUG:boto:path=/hiiqos-61/obj23?versionId=null 2023-07-29T00:50:02.206 DEBUG:boto:auth_path=/hiiqos-61/obj23?versionId=null 2023-07-29T00:50:02.206 DEBUG:boto:Method: HEAD 2023-07-29T00:50:02.207 DEBUG:boto:Path: /hiiqos-61/obj23?versionId=null 2023-07-29T00:50:02.207 DEBUG:boto:Data: 2023-07-29T00:50:02.207 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.208 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:02.208 DEBUG:boto:Port: 8000 2023-07-29T00:50:02.208 DEBUG:boto:Params: {} 2023-07-29T00:50:02.209 DEBUG:boto:Token: None 2023-07-29T00:50:02.209 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj23?versionId=null 2023-07-29T00:50:02.210 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6lFjgpCAWWa+YO8/0YDR/hU/SJY= 2023-07-29T00:50:02.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6lFjgpCAWWa+YO8/0YDR/hU/SJY=', 'Content-Length': '0'} 2023-07-29T00:50:02.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068298d8cdf73c4d8-0064c4623a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-07-29T00:50:02.213 DEBUG:boto:path=/hiiqos-61/obj23 2023-07-29T00:50:02.213 DEBUG:boto:auth_path=/hiiqos-61/obj23 2023-07-29T00:50:02.213 DEBUG:boto:path=/hiiqos-61/obj23?versionId=null 2023-07-29T00:50:02.214 DEBUG:boto:auth_path=/hiiqos-61/obj23?versionId=null 2023-07-29T00:50:02.214 DEBUG:boto:Method: GET 2023-07-29T00:50:02.215 DEBUG:boto:Path: /hiiqos-61/obj23?versionId=null 2023-07-29T00:50:02.215 DEBUG:boto:Data: 2023-07-29T00:50:02.215 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.216 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:02.216 DEBUG:boto:Port: 8001 2023-07-29T00:50:02.217 DEBUG:boto:Params: {} 2023-07-29T00:50:02.217 DEBUG:boto:Token: None 2023-07-29T00:50:02.217 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj23?versionId=null 2023-07-29T00:50:02.218 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I00fuXx8YNUAyso1s6fTaP2BrQo= 2023-07-29T00:50:02.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I00fuXx8YNUAyso1s6fTaP2BrQo=', 'Content-Length': '0'} 2023-07-29T00:50:02.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6e7a77749ac0c61-0064c4623a-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.221 DEBUG:boto:path=/hiiqos-61/obj23 2023-07-29T00:50:02.221 DEBUG:boto:auth_path=/hiiqos-61/obj23 2023-07-29T00:50:02.222 DEBUG:boto:path=/hiiqos-61/obj23?versionId=null 2023-07-29T00:50:02.222 DEBUG:boto:auth_path=/hiiqos-61/obj23?versionId=null 2023-07-29T00:50:02.223 DEBUG:boto:Method: GET 2023-07-29T00:50:02.223 DEBUG:boto:Path: /hiiqos-61/obj23?versionId=null 2023-07-29T00:50:02.224 DEBUG:boto:Data: 2023-07-29T00:50:02.224 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.225 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:02.225 DEBUG:boto:Port: 8000 2023-07-29T00:50:02.225 DEBUG:boto:Params: {} 2023-07-29T00:50:02.226 DEBUG:boto:Token: None 2023-07-29T00:50:02.226 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj23?versionId=null 2023-07-29T00:50:02.227 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I00fuXx8YNUAyso1s6fTaP2BrQo= 2023-07-29T00:50:02.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I00fuXx8YNUAyso1s6fTaP2BrQo=', 'Content-Length': '0'} 2023-07-29T00:50:02.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a09f239b5d3b4dbf-0064c4623a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.230 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:02.230 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:02.231 DEBUG:boto:path=/hiiqos-61/?prefix=obj23 2023-07-29T00:50:02.231 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj23 2023-07-29T00:50:02.232 DEBUG:boto:Method: GET 2023-07-29T00:50:02.232 DEBUG:boto:Path: /hiiqos-61/?prefix=obj23 2023-07-29T00:50:02.233 DEBUG:boto:Data: 2023-07-29T00:50:02.234 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.235 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:02.236 DEBUG:boto:Port: 8001 2023-07-29T00:50:02.237 DEBUG:boto:Params: {} 2023-07-29T00:50:02.237 DEBUG:boto:Token: None 2023-07-29T00:50:02.238 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/ 2023-07-29T00:50:02.238 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6jMSgLwXCW6tFHtDjt7WrDtXJ2g= 2023-07-29T00:50:02.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6jMSgLwXCW6tFHtDjt7WrDtXJ2g=', 'Content-Length': '0'} 2023-07-29T00:50:02.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026c0246c062e7643-0064c4623a-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.288 DEBUG:boto:b'hiiqos-61obj231000falseobj232023-07-29T00:45:01.985Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:02.290 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:02.290 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:02.290 DEBUG:boto:path=/hiiqos-61/?prefix=obj23 2023-07-29T00:50:02.291 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj23 2023-07-29T00:50:02.291 DEBUG:boto:Method: GET 2023-07-29T00:50:02.292 DEBUG:boto:Path: /hiiqos-61/?prefix=obj23 2023-07-29T00:50:02.292 DEBUG:boto:Data: 2023-07-29T00:50:02.293 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.293 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:02.293 DEBUG:boto:Port: 8000 2023-07-29T00:50:02.294 DEBUG:boto:Params: {} 2023-07-29T00:50:02.294 DEBUG:boto:Token: None 2023-07-29T00:50:02.295 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/ 2023-07-29T00:50:02.295 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6jMSgLwXCW6tFHtDjt7WrDtXJ2g= 2023-07-29T00:50:02.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6jMSgLwXCW6tFHtDjt7WrDtXJ2g=', 'Content-Length': '0'} 2023-07-29T00:50:02.299 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000600f6f21aa55eef5-0064c4623a-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.300 DEBUG:boto:b'hiiqos-61obj231000falseobj232023-07-29T00:45:01.985Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:02.300 DEBUG:boto:path=/hiiqos-61/obj23 2023-07-29T00:50:02.301 DEBUG:boto:auth_path=/hiiqos-61/obj23 2023-07-29T00:50:02.301 DEBUG:boto:Method: HEAD 2023-07-29T00:50:02.302 DEBUG:boto:Path: /hiiqos-61/obj23 2023-07-29T00:50:02.302 DEBUG:boto:Data: 2023-07-29T00:50:02.302 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.303 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:02.303 DEBUG:boto:Port: 8001 2023-07-29T00:50:02.303 DEBUG:boto:Params: {} 2023-07-29T00:50:02.304 DEBUG:boto:Token: None 2023-07-29T00:50:02.304 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj23 2023-07-29T00:50:02.305 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+mKMZkf29mtlOuI1toLpDaW6amE= 2023-07-29T00:50:02.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+mKMZkf29mtlOuI1toLpDaW6amE=', 'Content-Length': '0'} 2023-07-29T00:50:02.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017bac5a9ef237b21-0064c4623a-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.307 DEBUG:boto:path=/hiiqos-61/obj23 2023-07-29T00:50:02.308 DEBUG:boto:auth_path=/hiiqos-61/obj23 2023-07-29T00:50:02.308 DEBUG:boto:Method: HEAD 2023-07-29T00:50:02.308 DEBUG:boto:Path: /hiiqos-61/obj23 2023-07-29T00:50:02.309 DEBUG:boto:Data: 2023-07-29T00:50:02.309 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.309 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:02.310 DEBUG:boto:Port: 8000 2023-07-29T00:50:02.310 DEBUG:boto:Params: {} 2023-07-29T00:50:02.310 DEBUG:boto:Token: None 2023-07-29T00:50:02.311 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj23 2023-07-29T00:50:02.311 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+mKMZkf29mtlOuI1toLpDaW6amE= 2023-07-29T00:50:02.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+mKMZkf29mtlOuI1toLpDaW6amE=', 'Content-Length': '0'} 2023-07-29T00:50:02.313 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006b75f4b46c53537-0064c4623a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-07-29T00:50:02.314 DEBUG:boto:path=/hiiqos-61/obj23 2023-07-29T00:50:02.315 DEBUG:boto:auth_path=/hiiqos-61/obj23 2023-07-29T00:50:02.315 DEBUG:boto:Method: GET 2023-07-29T00:50:02.316 DEBUG:boto:Path: /hiiqos-61/obj23 2023-07-29T00:50:02.316 DEBUG:boto:Data: 2023-07-29T00:50:02.316 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.317 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:02.317 DEBUG:boto:Port: 8001 2023-07-29T00:50:02.317 DEBUG:boto:Params: {} 2023-07-29T00:50:02.318 DEBUG:boto:Token: None 2023-07-29T00:50:02.318 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj23 2023-07-29T00:50:02.319 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FVUSS7mKbn9MxOHay3xJdu4OijI= 2023-07-29T00:50:02.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FVUSS7mKbn9MxOHay3xJdu4OijI=', 'Content-Length': '0'} 2023-07-29T00:50:02.321 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6b36d87e0edae07-0064c4623a-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.321 DEBUG:boto:path=/hiiqos-61/obj23 2023-07-29T00:50:02.322 DEBUG:boto:auth_path=/hiiqos-61/obj23 2023-07-29T00:50:02.322 DEBUG:boto:Method: GET 2023-07-29T00:50:02.323 DEBUG:boto:Path: /hiiqos-61/obj23 2023-07-29T00:50:02.323 DEBUG:boto:Data: 2023-07-29T00:50:02.323 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.324 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:02.324 DEBUG:boto:Port: 8000 2023-07-29T00:50:02.324 DEBUG:boto:Params: {} 2023-07-29T00:50:02.325 DEBUG:boto:Token: None 2023-07-29T00:50:02.325 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj23 2023-07-29T00:50:02.326 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FVUSS7mKbn9MxOHay3xJdu4OijI= 2023-07-29T00:50:02.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FVUSS7mKbn9MxOHay3xJdu4OijI=', 'Content-Length': '0'} 2023-07-29T00:50:02.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7e752bf483957ce-0064c4623a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.329 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:02.329 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:02.329 DEBUG:boto:path=/hiiqos-61/?prefix=obj23 2023-07-29T00:50:02.330 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj23 2023-07-29T00:50:02.330 DEBUG:boto:Method: GET 2023-07-29T00:50:02.330 DEBUG:boto:Path: /hiiqos-61/?prefix=obj23 2023-07-29T00:50:02.331 DEBUG:boto:Data: 2023-07-29T00:50:02.331 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.332 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:02.332 DEBUG:boto:Port: 8001 2023-07-29T00:50:02.332 DEBUG:boto:Params: {} 2023-07-29T00:50:02.333 DEBUG:boto:Token: None 2023-07-29T00:50:02.333 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/ 2023-07-29T00:50:02.334 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6jMSgLwXCW6tFHtDjt7WrDtXJ2g= 2023-07-29T00:50:02.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6jMSgLwXCW6tFHtDjt7WrDtXJ2g=', 'Content-Length': '0'} 2023-07-29T00:50:02.336 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3ae892940a7ae96-0064c4623a-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.337 DEBUG:boto:b'hiiqos-61obj231000falseobj232023-07-29T00:45:01.985Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:02.338 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:02.338 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:02.338 DEBUG:boto:path=/hiiqos-61/?prefix=obj23 2023-07-29T00:50:02.339 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj23 2023-07-29T00:50:02.339 DEBUG:boto:Method: GET 2023-07-29T00:50:02.339 DEBUG:boto:Path: /hiiqos-61/?prefix=obj23 2023-07-29T00:50:02.340 DEBUG:boto:Data: 2023-07-29T00:50:02.340 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.341 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:02.341 DEBUG:boto:Port: 8000 2023-07-29T00:50:02.341 DEBUG:boto:Params: {} 2023-07-29T00:50:02.342 DEBUG:boto:Token: None 2023-07-29T00:50:02.342 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/ 2023-07-29T00:50:02.342 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6jMSgLwXCW6tFHtDjt7WrDtXJ2g= 2023-07-29T00:50:02.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6jMSgLwXCW6tFHtDjt7WrDtXJ2g=', 'Content-Length': '0'} 2023-07-29T00:50:02.346 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004614e213f1bae443-0064c4623a-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.347 DEBUG:boto:b'hiiqos-61obj231000falseobj232023-07-29T00:45:01.985Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:02.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-07-29T00:50:02.348 DEBUG:boto:path=/hiiqos-61/obj29 2023-07-29T00:50:02.348 DEBUG:boto:auth_path=/hiiqos-61/obj29 2023-07-29T00:50:02.349 DEBUG:boto:path=/hiiqos-61/obj29?versionId=null 2023-07-29T00:50:02.349 DEBUG:boto:auth_path=/hiiqos-61/obj29?versionId=null 2023-07-29T00:50:02.349 DEBUG:boto:Method: GET 2023-07-29T00:50:02.350 DEBUG:boto:Path: /hiiqos-61/obj29?versionId=null 2023-07-29T00:50:02.350 DEBUG:boto:Data: 2023-07-29T00:50:02.351 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.351 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:02.351 DEBUG:boto:Port: 8001 2023-07-29T00:50:02.352 DEBUG:boto:Params: {} 2023-07-29T00:50:02.353 DEBUG:boto:Token: None 2023-07-29T00:50:02.353 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj29?versionId=null 2023-07-29T00:50:02.354 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:idJb5fY4K9YX16ziCQtqjabOjM4= 2023-07-29T00:50:02.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:idJb5fY4K9YX16ziCQtqjabOjM4=', 'Content-Length': '0'} 2023-07-29T00:50:02.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a48c3fdf55556af-0064c4623a-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.396 DEBUG:boto:path=/hiiqos-61/obj29 2023-07-29T00:50:02.397 DEBUG:boto:auth_path=/hiiqos-61/obj29 2023-07-29T00:50:02.397 DEBUG:boto:path=/hiiqos-61/obj29?versionId=null 2023-07-29T00:50:02.397 DEBUG:boto:auth_path=/hiiqos-61/obj29?versionId=null 2023-07-29T00:50:02.398 DEBUG:boto:Method: GET 2023-07-29T00:50:02.398 DEBUG:boto:Path: /hiiqos-61/obj29?versionId=null 2023-07-29T00:50:02.399 DEBUG:boto:Data: 2023-07-29T00:50:02.399 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.399 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:02.400 DEBUG:boto:Port: 8000 2023-07-29T00:50:02.400 DEBUG:boto:Params: {} 2023-07-29T00:50:02.400 DEBUG:boto:Token: None 2023-07-29T00:50:02.401 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj29?versionId=null 2023-07-29T00:50:02.401 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:idJb5fY4K9YX16ziCQtqjabOjM4= 2023-07-29T00:50:02.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:idJb5fY4K9YX16ziCQtqjabOjM4=', 'Content-Length': '0'} 2023-07-29T00:50:02.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dae21013f39652c9-0064c4623a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.404 DEBUG:boto:path=/hiiqos-61/obj29 2023-07-29T00:50:02.405 DEBUG:boto:auth_path=/hiiqos-61/obj29 2023-07-29T00:50:02.405 DEBUG:boto:path=/hiiqos-61/obj29?versionId=null 2023-07-29T00:50:02.405 DEBUG:boto:auth_path=/hiiqos-61/obj29?versionId=null 2023-07-29T00:50:02.406 DEBUG:boto:Method: HEAD 2023-07-29T00:50:02.406 DEBUG:boto:Path: /hiiqos-61/obj29?versionId=null 2023-07-29T00:50:02.406 DEBUG:boto:Data: 2023-07-29T00:50:02.407 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.407 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:02.407 DEBUG:boto:Port: 8001 2023-07-29T00:50:02.408 DEBUG:boto:Params: {} 2023-07-29T00:50:02.408 DEBUG:boto:Token: None 2023-07-29T00:50:02.408 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj29?versionId=null 2023-07-29T00:50:02.409 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:D7JgNipFj1rssFnAIT8muoNE+jM= 2023-07-29T00:50:02.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:D7JgNipFj1rssFnAIT8muoNE+jM=', 'Content-Length': '0'} 2023-07-29T00:50:02.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c2c63ea9bf6e3f3-0064c4623a-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.412 DEBUG:boto:path=/hiiqos-61/obj29 2023-07-29T00:50:02.412 DEBUG:boto:auth_path=/hiiqos-61/obj29 2023-07-29T00:50:02.412 DEBUG:boto:path=/hiiqos-61/obj29?versionId=null 2023-07-29T00:50:02.413 DEBUG:boto:auth_path=/hiiqos-61/obj29?versionId=null 2023-07-29T00:50:02.413 DEBUG:boto:Method: HEAD 2023-07-29T00:50:02.413 DEBUG:boto:Path: /hiiqos-61/obj29?versionId=null 2023-07-29T00:50:02.414 DEBUG:boto:Data: 2023-07-29T00:50:02.414 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.414 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:02.415 DEBUG:boto:Port: 8000 2023-07-29T00:50:02.415 DEBUG:boto:Params: {} 2023-07-29T00:50:02.415 DEBUG:boto:Token: None 2023-07-29T00:50:02.416 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj29?versionId=null 2023-07-29T00:50:02.416 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:D7JgNipFj1rssFnAIT8muoNE+jM= 2023-07-29T00:50:02.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:D7JgNipFj1rssFnAIT8muoNE+jM=', 'Content-Length': '0'} 2023-07-29T00:50:02.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a67991b254b4eb66-0064c4623a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-07-29T00:50:02.419 DEBUG:boto:path=/hiiqos-61/obj29 2023-07-29T00:50:02.420 DEBUG:boto:auth_path=/hiiqos-61/obj29 2023-07-29T00:50:02.420 DEBUG:boto:path=/hiiqos-61/obj29?versionId=null 2023-07-29T00:50:02.421 DEBUG:boto:auth_path=/hiiqos-61/obj29?versionId=null 2023-07-29T00:50:02.421 DEBUG:boto:Method: GET 2023-07-29T00:50:02.421 DEBUG:boto:Path: /hiiqos-61/obj29?versionId=null 2023-07-29T00:50:02.422 DEBUG:boto:Data: 2023-07-29T00:50:02.422 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.422 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:02.423 DEBUG:boto:Port: 8001 2023-07-29T00:50:02.423 DEBUG:boto:Params: {} 2023-07-29T00:50:02.423 DEBUG:boto:Token: None 2023-07-29T00:50:02.424 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj29?versionId=null 2023-07-29T00:50:02.424 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:idJb5fY4K9YX16ziCQtqjabOjM4= 2023-07-29T00:50:02.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:idJb5fY4K9YX16ziCQtqjabOjM4=', 'Content-Length': '0'} 2023-07-29T00:50:02.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ce989bf3cfedd1a-0064c4623a-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.427 DEBUG:boto:path=/hiiqos-61/obj29 2023-07-29T00:50:02.428 DEBUG:boto:auth_path=/hiiqos-61/obj29 2023-07-29T00:50:02.428 DEBUG:boto:path=/hiiqos-61/obj29?versionId=null 2023-07-29T00:50:02.428 DEBUG:boto:auth_path=/hiiqos-61/obj29?versionId=null 2023-07-29T00:50:02.429 DEBUG:boto:Method: GET 2023-07-29T00:50:02.429 DEBUG:boto:Path: /hiiqos-61/obj29?versionId=null 2023-07-29T00:50:02.430 DEBUG:boto:Data: 2023-07-29T00:50:02.430 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.430 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:02.431 DEBUG:boto:Port: 8000 2023-07-29T00:50:02.431 DEBUG:boto:Params: {} 2023-07-29T00:50:02.431 DEBUG:boto:Token: None 2023-07-29T00:50:02.432 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj29?versionId=null 2023-07-29T00:50:02.432 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:idJb5fY4K9YX16ziCQtqjabOjM4= 2023-07-29T00:50:02.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:idJb5fY4K9YX16ziCQtqjabOjM4=', 'Content-Length': '0'} 2023-07-29T00:50:02.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4b617624a9da50f-0064c4623a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.435 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:02.435 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:02.436 DEBUG:boto:path=/hiiqos-61/?prefix=obj29 2023-07-29T00:50:02.436 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj29 2023-07-29T00:50:02.436 DEBUG:boto:Method: GET 2023-07-29T00:50:02.437 DEBUG:boto:Path: /hiiqos-61/?prefix=obj29 2023-07-29T00:50:02.437 DEBUG:boto:Data: 2023-07-29T00:50:02.437 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.438 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:02.438 DEBUG:boto:Port: 8001 2023-07-29T00:50:02.438 DEBUG:boto:Params: {} 2023-07-29T00:50:02.439 DEBUG:boto:Token: None 2023-07-29T00:50:02.439 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/ 2023-07-29T00:50:02.440 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6jMSgLwXCW6tFHtDjt7WrDtXJ2g= 2023-07-29T00:50:02.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6jMSgLwXCW6tFHtDjt7WrDtXJ2g=', 'Content-Length': '0'} 2023-07-29T00:50:02.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ed6b36f47e45b33-0064c4623a-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.443 DEBUG:boto:b'hiiqos-61obj291000falseobj292023-07-29T00:45:02.278Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:02.444 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:02.444 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:02.444 DEBUG:boto:path=/hiiqos-61/?prefix=obj29 2023-07-29T00:50:02.445 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj29 2023-07-29T00:50:02.445 DEBUG:boto:Method: GET 2023-07-29T00:50:02.445 DEBUG:boto:Path: /hiiqos-61/?prefix=obj29 2023-07-29T00:50:02.446 DEBUG:boto:Data: 2023-07-29T00:50:02.446 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.447 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:02.447 DEBUG:boto:Port: 8000 2023-07-29T00:50:02.447 DEBUG:boto:Params: {} 2023-07-29T00:50:02.448 DEBUG:boto:Token: None 2023-07-29T00:50:02.448 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/ 2023-07-29T00:50:02.449 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6jMSgLwXCW6tFHtDjt7WrDtXJ2g= 2023-07-29T00:50:02.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6jMSgLwXCW6tFHtDjt7WrDtXJ2g=', 'Content-Length': '0'} 2023-07-29T00:50:02.453 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045ee4fd4a9be1f40-0064c4623a-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.453 DEBUG:boto:b'hiiqos-61obj291000falseobj292023-07-29T00:45:02.278Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:02.454 DEBUG:boto:path=/hiiqos-61/obj29 2023-07-29T00:50:02.454 DEBUG:boto:auth_path=/hiiqos-61/obj29 2023-07-29T00:50:02.455 DEBUG:boto:Method: HEAD 2023-07-29T00:50:02.455 DEBUG:boto:Path: /hiiqos-61/obj29 2023-07-29T00:50:02.455 DEBUG:boto:Data: 2023-07-29T00:50:02.456 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.456 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:02.456 DEBUG:boto:Port: 8001 2023-07-29T00:50:02.457 DEBUG:boto:Params: {} 2023-07-29T00:50:02.457 DEBUG:boto:Token: None 2023-07-29T00:50:02.457 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj29 2023-07-29T00:50:02.458 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CyLAb3vpBN+8cBK2SD+8xL2Wi2k= 2023-07-29T00:50:02.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CyLAb3vpBN+8cBK2SD+8xL2Wi2k=', 'Content-Length': '0'} 2023-07-29T00:50:02.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e89e5c384712ea3-0064c4623a-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.461 DEBUG:boto:path=/hiiqos-61/obj29 2023-07-29T00:50:02.461 DEBUG:boto:auth_path=/hiiqos-61/obj29 2023-07-29T00:50:02.461 DEBUG:boto:Method: HEAD 2023-07-29T00:50:02.462 DEBUG:boto:Path: /hiiqos-61/obj29 2023-07-29T00:50:02.462 DEBUG:boto:Data: 2023-07-29T00:50:02.462 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.463 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:02.463 DEBUG:boto:Port: 8000 2023-07-29T00:50:02.463 DEBUG:boto:Params: {} 2023-07-29T00:50:02.464 DEBUG:boto:Token: None 2023-07-29T00:50:02.464 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj29 2023-07-29T00:50:02.465 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CyLAb3vpBN+8cBK2SD+8xL2Wi2k= 2023-07-29T00:50:02.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CyLAb3vpBN+8cBK2SD+8xL2Wi2k=', 'Content-Length': '0'} 2023-07-29T00:50:02.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000704a8d549be0d359-0064c4623a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-07-29T00:50:02.467 DEBUG:boto:path=/hiiqos-61/obj29 2023-07-29T00:50:02.468 DEBUG:boto:auth_path=/hiiqos-61/obj29 2023-07-29T00:50:02.468 DEBUG:boto:Method: GET 2023-07-29T00:50:02.469 DEBUG:boto:Path: /hiiqos-61/obj29 2023-07-29T00:50:02.469 DEBUG:boto:Data: 2023-07-29T00:50:02.469 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.470 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:02.470 DEBUG:boto:Port: 8001 2023-07-29T00:50:02.470 DEBUG:boto:Params: {} 2023-07-29T00:50:02.471 DEBUG:boto:Token: None 2023-07-29T00:50:02.471 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj29 2023-07-29T00:50:02.472 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Ruob+3hYFpZ49qxKVzVRR35qPDo= 2023-07-29T00:50:02.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Ruob+3hYFpZ49qxKVzVRR35qPDo=', 'Content-Length': '0'} 2023-07-29T00:50:02.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef802518ce0a4c07-0064c4623a-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.474 DEBUG:boto:path=/hiiqos-61/obj29 2023-07-29T00:50:02.475 DEBUG:boto:auth_path=/hiiqos-61/obj29 2023-07-29T00:50:02.475 DEBUG:boto:Method: GET 2023-07-29T00:50:02.476 DEBUG:boto:Path: /hiiqos-61/obj29 2023-07-29T00:50:02.476 DEBUG:boto:Data: 2023-07-29T00:50:02.476 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.477 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:02.477 DEBUG:boto:Port: 8000 2023-07-29T00:50:02.477 DEBUG:boto:Params: {} 2023-07-29T00:50:02.478 DEBUG:boto:Token: None 2023-07-29T00:50:02.478 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj29 2023-07-29T00:50:02.479 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Ruob+3hYFpZ49qxKVzVRR35qPDo= 2023-07-29T00:50:02.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Ruob+3hYFpZ49qxKVzVRR35qPDo=', 'Content-Length': '0'} 2023-07-29T00:50:02.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019c14ae3db32adf9-0064c4623a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.524 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:02.532 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:02.533 DEBUG:boto:path=/hiiqos-61/?prefix=obj29 2023-07-29T00:50:02.533 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj29 2023-07-29T00:50:02.533 DEBUG:boto:Method: GET 2023-07-29T00:50:02.534 DEBUG:boto:Path: /hiiqos-61/?prefix=obj29 2023-07-29T00:50:02.534 DEBUG:boto:Data: 2023-07-29T00:50:02.534 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.535 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:02.535 DEBUG:boto:Port: 8001 2023-07-29T00:50:02.535 DEBUG:boto:Params: {} 2023-07-29T00:50:02.536 DEBUG:boto:Token: None 2023-07-29T00:50:02.536 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/ 2023-07-29T00:50:02.537 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6jMSgLwXCW6tFHtDjt7WrDtXJ2g= 2023-07-29T00:50:02.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6jMSgLwXCW6tFHtDjt7WrDtXJ2g=', 'Content-Length': '0'} 2023-07-29T00:50:02.539 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066eb511f2a8a01c5-0064c4623a-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.540 DEBUG:boto:b'hiiqos-61obj291000falseobj292023-07-29T00:45:02.278Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:02.540 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:02.541 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:02.541 DEBUG:boto:path=/hiiqos-61/?prefix=obj29 2023-07-29T00:50:02.542 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj29 2023-07-29T00:50:02.542 DEBUG:boto:Method: GET 2023-07-29T00:50:02.542 DEBUG:boto:Path: /hiiqos-61/?prefix=obj29 2023-07-29T00:50:02.543 DEBUG:boto:Data: 2023-07-29T00:50:02.543 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.543 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:02.544 DEBUG:boto:Port: 8000 2023-07-29T00:50:02.544 DEBUG:boto:Params: {} 2023-07-29T00:50:02.544 DEBUG:boto:Token: None 2023-07-29T00:50:02.545 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/ 2023-07-29T00:50:02.545 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6jMSgLwXCW6tFHtDjt7WrDtXJ2g= 2023-07-29T00:50:02.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6jMSgLwXCW6tFHtDjt7WrDtXJ2g=', 'Content-Length': '0'} 2023-07-29T00:50:02.549 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c151bcc6353d3a04-0064c4623a-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.550 DEBUG:boto:b'hiiqos-61obj291000falseobj292023-07-29T00:45:02.278Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:02.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-07-29T00:50:02.551 DEBUG:boto:path=/hiiqos-61/obj31 2023-07-29T00:50:02.551 DEBUG:boto:auth_path=/hiiqos-61/obj31 2023-07-29T00:50:02.551 DEBUG:boto:path=/hiiqos-61/obj31?versionId=null 2023-07-29T00:50:02.552 DEBUG:boto:auth_path=/hiiqos-61/obj31?versionId=null 2023-07-29T00:50:02.552 DEBUG:boto:Method: GET 2023-07-29T00:50:02.552 DEBUG:boto:Path: /hiiqos-61/obj31?versionId=null 2023-07-29T00:50:02.553 DEBUG:boto:Data: 2023-07-29T00:50:02.553 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.554 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:02.554 DEBUG:boto:Port: 8001 2023-07-29T00:50:02.554 DEBUG:boto:Params: {} 2023-07-29T00:50:02.555 DEBUG:boto:Token: None 2023-07-29T00:50:02.555 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj31?versionId=null 2023-07-29T00:50:02.556 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Zv1yV2LADiscRCcs7/l93yEL/Jo= 2023-07-29T00:50:02.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Zv1yV2LADiscRCcs7/l93yEL/Jo=', 'Content-Length': '0'} 2023-07-29T00:50:02.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c606d30ebbff182-0064c4623a-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.559 DEBUG:boto:path=/hiiqos-61/obj31 2023-07-29T00:50:02.559 DEBUG:boto:auth_path=/hiiqos-61/obj31 2023-07-29T00:50:02.560 DEBUG:boto:path=/hiiqos-61/obj31?versionId=null 2023-07-29T00:50:02.560 DEBUG:boto:auth_path=/hiiqos-61/obj31?versionId=null 2023-07-29T00:50:02.560 DEBUG:boto:Method: GET 2023-07-29T00:50:02.561 DEBUG:boto:Path: /hiiqos-61/obj31?versionId=null 2023-07-29T00:50:02.561 DEBUG:boto:Data: 2023-07-29T00:50:02.562 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.562 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:02.562 DEBUG:boto:Port: 8000 2023-07-29T00:50:02.563 DEBUG:boto:Params: {} 2023-07-29T00:50:02.563 DEBUG:boto:Token: None 2023-07-29T00:50:02.563 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj31?versionId=null 2023-07-29T00:50:02.564 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Zv1yV2LADiscRCcs7/l93yEL/Jo= 2023-07-29T00:50:02.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Zv1yV2LADiscRCcs7/l93yEL/Jo=', 'Content-Length': '0'} 2023-07-29T00:50:02.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e5990507abe3fbb-0064c4623a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.567 DEBUG:boto:path=/hiiqos-61/obj31 2023-07-29T00:50:02.567 DEBUG:boto:auth_path=/hiiqos-61/obj31 2023-07-29T00:50:02.567 DEBUG:boto:path=/hiiqos-61/obj31?versionId=null 2023-07-29T00:50:02.568 DEBUG:boto:auth_path=/hiiqos-61/obj31?versionId=null 2023-07-29T00:50:02.568 DEBUG:boto:Method: HEAD 2023-07-29T00:50:02.568 DEBUG:boto:Path: /hiiqos-61/obj31?versionId=null 2023-07-29T00:50:02.569 DEBUG:boto:Data: 2023-07-29T00:50:02.569 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.569 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:02.570 DEBUG:boto:Port: 8001 2023-07-29T00:50:02.570 DEBUG:boto:Params: {} 2023-07-29T00:50:02.571 DEBUG:boto:Token: None 2023-07-29T00:50:02.571 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj31?versionId=null 2023-07-29T00:50:02.571 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1sa5pxVI0g0uAOF++uMJR2YHSWY= 2023-07-29T00:50:02.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1sa5pxVI0g0uAOF++uMJR2YHSWY=', 'Content-Length': '0'} 2023-07-29T00:50:02.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6e4f240062895e2-0064c4623a-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.574 DEBUG:boto:path=/hiiqos-61/obj31 2023-07-29T00:50:02.574 DEBUG:boto:auth_path=/hiiqos-61/obj31 2023-07-29T00:50:02.575 DEBUG:boto:path=/hiiqos-61/obj31?versionId=null 2023-07-29T00:50:02.575 DEBUG:boto:auth_path=/hiiqos-61/obj31?versionId=null 2023-07-29T00:50:02.575 DEBUG:boto:Method: HEAD 2023-07-29T00:50:02.576 DEBUG:boto:Path: /hiiqos-61/obj31?versionId=null 2023-07-29T00:50:02.576 DEBUG:boto:Data: 2023-07-29T00:50:02.576 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.577 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:02.577 DEBUG:boto:Port: 8000 2023-07-29T00:50:02.577 DEBUG:boto:Params: {} 2023-07-29T00:50:02.578 DEBUG:boto:Token: None 2023-07-29T00:50:02.578 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj31?versionId=null 2023-07-29T00:50:02.579 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1sa5pxVI0g0uAOF++uMJR2YHSWY= 2023-07-29T00:50:02.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1sa5pxVI0g0uAOF++uMJR2YHSWY=', 'Content-Length': '0'} 2023-07-29T00:50:02.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000eff3163ec0ac3f0-0064c4623a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-07-29T00:50:02.581 DEBUG:boto:path=/hiiqos-61/obj31 2023-07-29T00:50:02.582 DEBUG:boto:auth_path=/hiiqos-61/obj31 2023-07-29T00:50:02.582 DEBUG:boto:path=/hiiqos-61/obj31?versionId=null 2023-07-29T00:50:02.582 DEBUG:boto:auth_path=/hiiqos-61/obj31?versionId=null 2023-07-29T00:50:02.583 DEBUG:boto:Method: GET 2023-07-29T00:50:02.583 DEBUG:boto:Path: /hiiqos-61/obj31?versionId=null 2023-07-29T00:50:02.583 DEBUG:boto:Data: 2023-07-29T00:50:02.584 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.584 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:02.585 DEBUG:boto:Port: 8001 2023-07-29T00:50:02.585 DEBUG:boto:Params: {} 2023-07-29T00:50:02.585 DEBUG:boto:Token: None 2023-07-29T00:50:02.586 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj31?versionId=null 2023-07-29T00:50:02.586 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Zv1yV2LADiscRCcs7/l93yEL/Jo= 2023-07-29T00:50:02.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Zv1yV2LADiscRCcs7/l93yEL/Jo=', 'Content-Length': '0'} 2023-07-29T00:50:02.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034deb052bc9a2a85-0064c4623a-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.589 DEBUG:boto:path=/hiiqos-61/obj31 2023-07-29T00:50:02.589 DEBUG:boto:auth_path=/hiiqos-61/obj31 2023-07-29T00:50:02.590 DEBUG:boto:path=/hiiqos-61/obj31?versionId=null 2023-07-29T00:50:02.590 DEBUG:boto:auth_path=/hiiqos-61/obj31?versionId=null 2023-07-29T00:50:02.590 DEBUG:boto:Method: GET 2023-07-29T00:50:02.591 DEBUG:boto:Path: /hiiqos-61/obj31?versionId=null 2023-07-29T00:50:02.591 DEBUG:boto:Data: 2023-07-29T00:50:02.591 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.592 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:02.592 DEBUG:boto:Port: 8000 2023-07-29T00:50:02.592 DEBUG:boto:Params: {} 2023-07-29T00:50:02.593 DEBUG:boto:Token: None 2023-07-29T00:50:02.593 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj31?versionId=null 2023-07-29T00:50:02.594 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Zv1yV2LADiscRCcs7/l93yEL/Jo= 2023-07-29T00:50:02.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Zv1yV2LADiscRCcs7/l93yEL/Jo=', 'Content-Length': '0'} 2023-07-29T00:50:02.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d443db5414ecb2b-0064c4623a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.636 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:02.637 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:02.638 DEBUG:boto:path=/hiiqos-61/?prefix=obj31 2023-07-29T00:50:02.639 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj31 2023-07-29T00:50:02.640 DEBUG:boto:Method: GET 2023-07-29T00:50:02.640 DEBUG:boto:Path: /hiiqos-61/?prefix=obj31 2023-07-29T00:50:02.641 DEBUG:boto:Data: 2023-07-29T00:50:02.642 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.642 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:02.643 DEBUG:boto:Port: 8001 2023-07-29T00:50:02.644 DEBUG:boto:Params: {} 2023-07-29T00:50:02.644 DEBUG:boto:Token: None 2023-07-29T00:50:02.645 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/ 2023-07-29T00:50:02.646 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6jMSgLwXCW6tFHtDjt7WrDtXJ2g= 2023-07-29T00:50:02.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6jMSgLwXCW6tFHtDjt7WrDtXJ2g=', 'Content-Length': '0'} 2023-07-29T00:50:02.649 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d76c0de8105d2a84-0064c4623a-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.650 DEBUG:boto:b'hiiqos-61obj311000falseobj312023-07-29T00:45:02.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:02.651 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:02.651 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:02.652 DEBUG:boto:path=/hiiqos-61/?prefix=obj31 2023-07-29T00:50:02.653 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj31 2023-07-29T00:50:02.653 DEBUG:boto:Method: GET 2023-07-29T00:50:02.654 DEBUG:boto:Path: /hiiqos-61/?prefix=obj31 2023-07-29T00:50:02.655 DEBUG:boto:Data: 2023-07-29T00:50:02.655 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.656 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:02.656 DEBUG:boto:Port: 8000 2023-07-29T00:50:02.657 DEBUG:boto:Params: {} 2023-07-29T00:50:02.658 DEBUG:boto:Token: None 2023-07-29T00:50:02.659 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/ 2023-07-29T00:50:02.659 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6jMSgLwXCW6tFHtDjt7WrDtXJ2g= 2023-07-29T00:50:02.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6jMSgLwXCW6tFHtDjt7WrDtXJ2g=', 'Content-Length': '0'} 2023-07-29T00:50:02.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1274f22b2de39f6-0064c4623a-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.665 DEBUG:boto:b'hiiqos-61obj311000falseobj312023-07-29T00:45:02.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:02.666 DEBUG:boto:path=/hiiqos-61/obj31 2023-07-29T00:50:02.667 DEBUG:boto:auth_path=/hiiqos-61/obj31 2023-07-29T00:50:02.667 DEBUG:boto:Method: HEAD 2023-07-29T00:50:02.668 DEBUG:boto:Path: /hiiqos-61/obj31 2023-07-29T00:50:02.669 DEBUG:boto:Data: 2023-07-29T00:50:02.669 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.670 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:02.671 DEBUG:boto:Port: 8001 2023-07-29T00:50:02.671 DEBUG:boto:Params: {} 2023-07-29T00:50:02.672 DEBUG:boto:Token: None 2023-07-29T00:50:02.673 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj31 2023-07-29T00:50:02.674 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0TfqdZy+vCWF/7LSJaVpU9+fnrs= 2023-07-29T00:50:02.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0TfqdZy+vCWF/7LSJaVpU9+fnrs=', 'Content-Length': '0'} 2023-07-29T00:50:02.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051c0a7c2b78a602c-0064c4623a-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.678 DEBUG:boto:path=/hiiqos-61/obj31 2023-07-29T00:50:02.678 DEBUG:boto:auth_path=/hiiqos-61/obj31 2023-07-29T00:50:02.679 DEBUG:boto:Method: HEAD 2023-07-29T00:50:02.680 DEBUG:boto:Path: /hiiqos-61/obj31 2023-07-29T00:50:02.680 DEBUG:boto:Data: 2023-07-29T00:50:02.681 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.682 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:02.682 DEBUG:boto:Port: 8000 2023-07-29T00:50:02.683 DEBUG:boto:Params: {} 2023-07-29T00:50:02.684 DEBUG:boto:Token: None 2023-07-29T00:50:02.685 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj31 2023-07-29T00:50:02.685 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0TfqdZy+vCWF/7LSJaVpU9+fnrs= 2023-07-29T00:50:02.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0TfqdZy+vCWF/7LSJaVpU9+fnrs=', 'Content-Length': '0'} 2023-07-29T00:50:02.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bd25fd7b1db9aa0-0064c4623a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-07-29T00:50:02.690 DEBUG:boto:path=/hiiqos-61/obj31 2023-07-29T00:50:02.691 DEBUG:boto:auth_path=/hiiqos-61/obj31 2023-07-29T00:50:02.691 DEBUG:boto:Method: GET 2023-07-29T00:50:02.692 DEBUG:boto:Path: /hiiqos-61/obj31 2023-07-29T00:50:02.693 DEBUG:boto:Data: 2023-07-29T00:50:02.693 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.694 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:02.695 DEBUG:boto:Port: 8001 2023-07-29T00:50:02.696 DEBUG:boto:Params: {} 2023-07-29T00:50:02.696 DEBUG:boto:Token: None 2023-07-29T00:50:02.697 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj31 2023-07-29T00:50:02.698 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DSwbgDdyss65oJpqoTY0mS5soPg= 2023-07-29T00:50:02.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DSwbgDdyss65oJpqoTY0mS5soPg=', 'Content-Length': '0'} 2023-07-29T00:50:02.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb42c2766ad23abc-0064c4623a-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.744 DEBUG:boto:path=/hiiqos-61/obj31 2023-07-29T00:50:02.745 DEBUG:boto:auth_path=/hiiqos-61/obj31 2023-07-29T00:50:02.745 DEBUG:boto:Method: GET 2023-07-29T00:50:02.746 DEBUG:boto:Path: /hiiqos-61/obj31 2023-07-29T00:50:02.746 DEBUG:boto:Data: 2023-07-29T00:50:02.746 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.747 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:02.747 DEBUG:boto:Port: 8000 2023-07-29T00:50:02.747 DEBUG:boto:Params: {} 2023-07-29T00:50:02.748 DEBUG:boto:Token: None 2023-07-29T00:50:02.748 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj31 2023-07-29T00:50:02.749 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DSwbgDdyss65oJpqoTY0mS5soPg= 2023-07-29T00:50:02.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DSwbgDdyss65oJpqoTY0mS5soPg=', 'Content-Length': '0'} 2023-07-29T00:50:02.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b74f83812d4a1127-0064c4623a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.752 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:02.752 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:02.752 DEBUG:boto:path=/hiiqos-61/?prefix=obj31 2023-07-29T00:50:02.753 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj31 2023-07-29T00:50:02.753 DEBUG:boto:Method: GET 2023-07-29T00:50:02.753 DEBUG:boto:Path: /hiiqos-61/?prefix=obj31 2023-07-29T00:50:02.754 DEBUG:boto:Data: 2023-07-29T00:50:02.754 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.754 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:02.755 DEBUG:boto:Port: 8001 2023-07-29T00:50:02.755 DEBUG:boto:Params: {} 2023-07-29T00:50:02.756 DEBUG:boto:Token: None 2023-07-29T00:50:02.756 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/ 2023-07-29T00:50:02.756 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6jMSgLwXCW6tFHtDjt7WrDtXJ2g= 2023-07-29T00:50:02.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6jMSgLwXCW6tFHtDjt7WrDtXJ2g=', 'Content-Length': '0'} 2023-07-29T00:50:02.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042ff2f5572fd78e2-0064c4623a-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.759 DEBUG:boto:b'hiiqos-61obj311000falseobj312023-07-29T00:45:02.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:02.760 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:02.761 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:02.761 DEBUG:boto:path=/hiiqos-61/?prefix=obj31 2023-07-29T00:50:02.761 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj31 2023-07-29T00:50:02.762 DEBUG:boto:Method: GET 2023-07-29T00:50:02.762 DEBUG:boto:Path: /hiiqos-61/?prefix=obj31 2023-07-29T00:50:02.762 DEBUG:boto:Data: 2023-07-29T00:50:02.763 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.763 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:02.763 DEBUG:boto:Port: 8000 2023-07-29T00:50:02.764 DEBUG:boto:Params: {} 2023-07-29T00:50:02.764 DEBUG:boto:Token: None 2023-07-29T00:50:02.765 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/ 2023-07-29T00:50:02.765 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6jMSgLwXCW6tFHtDjt7WrDtXJ2g= 2023-07-29T00:50:02.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6jMSgLwXCW6tFHtDjt7WrDtXJ2g=', 'Content-Length': '0'} 2023-07-29T00:50:02.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089efa316ffbc42ad-0064c4623a-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.770 DEBUG:boto:b'hiiqos-61obj311000falseobj312023-07-29T00:45:02.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:02.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-07-29T00:50:02.771 DEBUG:boto:path=/hiiqos-61/obj33 2023-07-29T00:50:02.771 DEBUG:boto:auth_path=/hiiqos-61/obj33 2023-07-29T00:50:02.771 DEBUG:boto:path=/hiiqos-61/obj33?versionId=null 2023-07-29T00:50:02.772 DEBUG:boto:auth_path=/hiiqos-61/obj33?versionId=null 2023-07-29T00:50:02.772 DEBUG:boto:Method: GET 2023-07-29T00:50:02.772 DEBUG:boto:Path: /hiiqos-61/obj33?versionId=null 2023-07-29T00:50:02.773 DEBUG:boto:Data: 2023-07-29T00:50:02.773 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.774 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:02.774 DEBUG:boto:Port: 8001 2023-07-29T00:50:02.774 DEBUG:boto:Params: {} 2023-07-29T00:50:02.775 DEBUG:boto:Token: None 2023-07-29T00:50:02.775 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj33?versionId=null 2023-07-29T00:50:02.776 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Hl9wlxqkzPJaXFyjOGZDKTZVAZg= 2023-07-29T00:50:02.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Hl9wlxqkzPJaXFyjOGZDKTZVAZg=', 'Content-Length': '0'} 2023-07-29T00:50:02.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cff3cf58df97e085-0064c4623a-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.779 DEBUG:boto:path=/hiiqos-61/obj33 2023-07-29T00:50:02.779 DEBUG:boto:auth_path=/hiiqos-61/obj33 2023-07-29T00:50:02.779 DEBUG:boto:path=/hiiqos-61/obj33?versionId=null 2023-07-29T00:50:02.780 DEBUG:boto:auth_path=/hiiqos-61/obj33?versionId=null 2023-07-29T00:50:02.780 DEBUG:boto:Method: GET 2023-07-29T00:50:02.780 DEBUG:boto:Path: /hiiqos-61/obj33?versionId=null 2023-07-29T00:50:02.781 DEBUG:boto:Data: 2023-07-29T00:50:02.781 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.782 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:02.782 DEBUG:boto:Port: 8000 2023-07-29T00:50:02.782 DEBUG:boto:Params: {} 2023-07-29T00:50:02.783 DEBUG:boto:Token: None 2023-07-29T00:50:02.783 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj33?versionId=null 2023-07-29T00:50:02.784 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Hl9wlxqkzPJaXFyjOGZDKTZVAZg= 2023-07-29T00:50:02.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Hl9wlxqkzPJaXFyjOGZDKTZVAZg=', 'Content-Length': '0'} 2023-07-29T00:50:02.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004dc0778276cba992-0064c4623a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.786 DEBUG:boto:path=/hiiqos-61/obj33 2023-07-29T00:50:02.787 DEBUG:boto:auth_path=/hiiqos-61/obj33 2023-07-29T00:50:02.787 DEBUG:boto:path=/hiiqos-61/obj33?versionId=null 2023-07-29T00:50:02.787 DEBUG:boto:auth_path=/hiiqos-61/obj33?versionId=null 2023-07-29T00:50:02.788 DEBUG:boto:Method: HEAD 2023-07-29T00:50:02.788 DEBUG:boto:Path: /hiiqos-61/obj33?versionId=null 2023-07-29T00:50:02.788 DEBUG:boto:Data: 2023-07-29T00:50:02.789 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.789 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:02.789 DEBUG:boto:Port: 8001 2023-07-29T00:50:02.790 DEBUG:boto:Params: {} 2023-07-29T00:50:02.790 DEBUG:boto:Token: None 2023-07-29T00:50:02.791 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj33?versionId=null 2023-07-29T00:50:02.791 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:b7OyGjLFQOdzMlgKD72UrDLmIOI= 2023-07-29T00:50:02.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:b7OyGjLFQOdzMlgKD72UrDLmIOI=', 'Content-Length': '0'} 2023-07-29T00:50:02.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0b08fb84c976533-0064c4623a-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.793 DEBUG:boto:path=/hiiqos-61/obj33 2023-07-29T00:50:02.794 DEBUG:boto:auth_path=/hiiqos-61/obj33 2023-07-29T00:50:02.794 DEBUG:boto:path=/hiiqos-61/obj33?versionId=null 2023-07-29T00:50:02.795 DEBUG:boto:auth_path=/hiiqos-61/obj33?versionId=null 2023-07-29T00:50:02.795 DEBUG:boto:Method: HEAD 2023-07-29T00:50:02.795 DEBUG:boto:Path: /hiiqos-61/obj33?versionId=null 2023-07-29T00:50:02.796 DEBUG:boto:Data: 2023-07-29T00:50:02.796 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.796 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:02.797 DEBUG:boto:Port: 8000 2023-07-29T00:50:02.797 DEBUG:boto:Params: {} 2023-07-29T00:50:02.797 DEBUG:boto:Token: None 2023-07-29T00:50:02.798 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj33?versionId=null 2023-07-29T00:50:02.798 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:b7OyGjLFQOdzMlgKD72UrDLmIOI= 2023-07-29T00:50:02.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:b7OyGjLFQOdzMlgKD72UrDLmIOI=', 'Content-Length': '0'} 2023-07-29T00:50:02.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026cd837a61b01a5c-0064c4623a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-07-29T00:50:02.801 DEBUG:boto:path=/hiiqos-61/obj33 2023-07-29T00:50:02.801 DEBUG:boto:auth_path=/hiiqos-61/obj33 2023-07-29T00:50:02.802 DEBUG:boto:path=/hiiqos-61/obj33?versionId=null 2023-07-29T00:50:02.802 DEBUG:boto:auth_path=/hiiqos-61/obj33?versionId=null 2023-07-29T00:50:02.802 DEBUG:boto:Method: GET 2023-07-29T00:50:02.803 DEBUG:boto:Path: /hiiqos-61/obj33?versionId=null 2023-07-29T00:50:02.803 DEBUG:boto:Data: 2023-07-29T00:50:02.804 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.804 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:02.804 DEBUG:boto:Port: 8001 2023-07-29T00:50:02.805 DEBUG:boto:Params: {} 2023-07-29T00:50:02.805 DEBUG:boto:Token: None 2023-07-29T00:50:02.805 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj33?versionId=null 2023-07-29T00:50:02.806 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Hl9wlxqkzPJaXFyjOGZDKTZVAZg= 2023-07-29T00:50:02.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Hl9wlxqkzPJaXFyjOGZDKTZVAZg=', 'Content-Length': '0'} 2023-07-29T00:50:02.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001753e78e2dbf3445-0064c4623a-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.809 DEBUG:boto:path=/hiiqos-61/obj33 2023-07-29T00:50:02.809 DEBUG:boto:auth_path=/hiiqos-61/obj33 2023-07-29T00:50:02.809 DEBUG:boto:path=/hiiqos-61/obj33?versionId=null 2023-07-29T00:50:02.810 DEBUG:boto:auth_path=/hiiqos-61/obj33?versionId=null 2023-07-29T00:50:02.810 DEBUG:boto:Method: GET 2023-07-29T00:50:02.811 DEBUG:boto:Path: /hiiqos-61/obj33?versionId=null 2023-07-29T00:50:02.811 DEBUG:boto:Data: 2023-07-29T00:50:02.811 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.812 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:02.812 DEBUG:boto:Port: 8000 2023-07-29T00:50:02.812 DEBUG:boto:Params: {} 2023-07-29T00:50:02.813 DEBUG:boto:Token: None 2023-07-29T00:50:02.813 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj33?versionId=null 2023-07-29T00:50:02.814 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Hl9wlxqkzPJaXFyjOGZDKTZVAZg= 2023-07-29T00:50:02.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Hl9wlxqkzPJaXFyjOGZDKTZVAZg=', 'Content-Length': '0'} 2023-07-29T00:50:02.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097b1e412e1240da2-0064c4623a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.816 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:02.817 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:02.817 DEBUG:boto:path=/hiiqos-61/?prefix=obj33 2023-07-29T00:50:02.817 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj33 2023-07-29T00:50:02.818 DEBUG:boto:Method: GET 2023-07-29T00:50:02.818 DEBUG:boto:Path: /hiiqos-61/?prefix=obj33 2023-07-29T00:50:02.819 DEBUG:boto:Data: 2023-07-29T00:50:02.819 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.819 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:02.820 DEBUG:boto:Port: 8001 2023-07-29T00:50:02.820 DEBUG:boto:Params: {} 2023-07-29T00:50:02.820 DEBUG:boto:Token: None 2023-07-29T00:50:02.821 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/ 2023-07-29T00:50:02.821 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6jMSgLwXCW6tFHtDjt7WrDtXJ2g= 2023-07-29T00:50:02.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6jMSgLwXCW6tFHtDjt7WrDtXJ2g=', 'Content-Length': '0'} 2023-07-29T00:50:02.824 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000938d1e58dd54b39e-0064c4623a-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.824 DEBUG:boto:b'hiiqos-61obj331000falseobj332023-07-29T00:45:01.083Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:02.825 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:02.825 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:02.826 DEBUG:boto:path=/hiiqos-61/?prefix=obj33 2023-07-29T00:50:02.826 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj33 2023-07-29T00:50:02.826 DEBUG:boto:Method: GET 2023-07-29T00:50:02.827 DEBUG:boto:Path: /hiiqos-61/?prefix=obj33 2023-07-29T00:50:02.827 DEBUG:boto:Data: 2023-07-29T00:50:02.827 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.828 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:02.828 DEBUG:boto:Port: 8000 2023-07-29T00:50:02.828 DEBUG:boto:Params: {} 2023-07-29T00:50:02.829 DEBUG:boto:Token: None 2023-07-29T00:50:02.829 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/ 2023-07-29T00:50:02.830 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6jMSgLwXCW6tFHtDjt7WrDtXJ2g= 2023-07-29T00:50:02.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6jMSgLwXCW6tFHtDjt7WrDtXJ2g=', 'Content-Length': '0'} 2023-07-29T00:50:02.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b09a15bbe5d232a-0064c4623a-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.834 DEBUG:boto:b'hiiqos-61obj331000falseobj332023-07-29T00:45:01.083Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:02.835 DEBUG:boto:path=/hiiqos-61/obj33 2023-07-29T00:50:02.835 DEBUG:boto:auth_path=/hiiqos-61/obj33 2023-07-29T00:50:02.836 DEBUG:boto:Method: HEAD 2023-07-29T00:50:02.836 DEBUG:boto:Path: /hiiqos-61/obj33 2023-07-29T00:50:02.836 DEBUG:boto:Data: 2023-07-29T00:50:02.837 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.837 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:02.837 DEBUG:boto:Port: 8001 2023-07-29T00:50:02.838 DEBUG:boto:Params: {} 2023-07-29T00:50:02.838 DEBUG:boto:Token: None 2023-07-29T00:50:02.839 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj33 2023-07-29T00:50:02.839 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OUNW4OIA127Npe1JpISa64DM0nQ= 2023-07-29T00:50:02.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OUNW4OIA127Npe1JpISa64DM0nQ=', 'Content-Length': '0'} 2023-07-29T00:50:02.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd4c3f443c6e2d41-0064c4623a-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.842 DEBUG:boto:path=/hiiqos-61/obj33 2023-07-29T00:50:02.842 DEBUG:boto:auth_path=/hiiqos-61/obj33 2023-07-29T00:50:02.842 DEBUG:boto:Method: HEAD 2023-07-29T00:50:02.843 DEBUG:boto:Path: /hiiqos-61/obj33 2023-07-29T00:50:02.843 DEBUG:boto:Data: 2023-07-29T00:50:02.843 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.844 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:02.844 DEBUG:boto:Port: 8000 2023-07-29T00:50:02.844 DEBUG:boto:Params: {} 2023-07-29T00:50:02.845 DEBUG:boto:Token: None 2023-07-29T00:50:02.845 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj33 2023-07-29T00:50:02.846 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OUNW4OIA127Npe1JpISa64DM0nQ= 2023-07-29T00:50:02.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OUNW4OIA127Npe1JpISa64DM0nQ=', 'Content-Length': '0'} 2023-07-29T00:50:02.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000283cf3e6526ddf0d-0064c4623a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-07-29T00:50:02.849 DEBUG:boto:path=/hiiqos-61/obj33 2023-07-29T00:50:02.849 DEBUG:boto:auth_path=/hiiqos-61/obj33 2023-07-29T00:50:02.849 DEBUG:boto:Method: GET 2023-07-29T00:50:02.850 DEBUG:boto:Path: /hiiqos-61/obj33 2023-07-29T00:50:02.850 DEBUG:boto:Data: 2023-07-29T00:50:02.850 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.851 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:02.851 DEBUG:boto:Port: 8001 2023-07-29T00:50:02.851 DEBUG:boto:Params: {} 2023-07-29T00:50:02.852 DEBUG:boto:Token: None 2023-07-29T00:50:02.852 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj33 2023-07-29T00:50:02.853 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH://7fkfb6EtJBfcdSQE5SiFZyb20= 2023-07-29T00:50:02.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH://7fkfb6EtJBfcdSQE5SiFZyb20=', 'Content-Length': '0'} 2023-07-29T00:50:02.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000663bd1e9a9b6db1a-0064c4623a-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.855 DEBUG:boto:path=/hiiqos-61/obj33 2023-07-29T00:50:02.856 DEBUG:boto:auth_path=/hiiqos-61/obj33 2023-07-29T00:50:02.856 DEBUG:boto:Method: GET 2023-07-29T00:50:02.857 DEBUG:boto:Path: /hiiqos-61/obj33 2023-07-29T00:50:02.857 DEBUG:boto:Data: 2023-07-29T00:50:02.857 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.858 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:02.858 DEBUG:boto:Port: 8000 2023-07-29T00:50:02.858 DEBUG:boto:Params: {} 2023-07-29T00:50:02.859 DEBUG:boto:Token: None 2023-07-29T00:50:02.859 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj33 2023-07-29T00:50:02.860 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH://7fkfb6EtJBfcdSQE5SiFZyb20= 2023-07-29T00:50:02.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH://7fkfb6EtJBfcdSQE5SiFZyb20=', 'Content-Length': '0'} 2023-07-29T00:50:02.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d8f583fa59904f1-0064c4623a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.908 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:02.909 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:02.909 DEBUG:boto:path=/hiiqos-61/?prefix=obj33 2023-07-29T00:50:02.910 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj33 2023-07-29T00:50:02.910 DEBUG:boto:Method: GET 2023-07-29T00:50:02.910 DEBUG:boto:Path: /hiiqos-61/?prefix=obj33 2023-07-29T00:50:02.911 DEBUG:boto:Data: 2023-07-29T00:50:02.911 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.911 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:02.912 DEBUG:boto:Port: 8001 2023-07-29T00:50:02.912 DEBUG:boto:Params: {} 2023-07-29T00:50:02.913 DEBUG:boto:Token: None 2023-07-29T00:50:02.913 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/ 2023-07-29T00:50:02.914 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6jMSgLwXCW6tFHtDjt7WrDtXJ2g= 2023-07-29T00:50:02.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6jMSgLwXCW6tFHtDjt7WrDtXJ2g=', 'Content-Length': '0'} 2023-07-29T00:50:02.916 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e60ac656829f569a-0064c4623a-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.917 DEBUG:boto:b'hiiqos-61obj331000falseobj332023-07-29T00:45:01.083Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:02.918 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:02.918 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:02.918 DEBUG:boto:path=/hiiqos-61/?prefix=obj33 2023-07-29T00:50:02.919 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj33 2023-07-29T00:50:02.919 DEBUG:boto:Method: GET 2023-07-29T00:50:02.919 DEBUG:boto:Path: /hiiqos-61/?prefix=obj33 2023-07-29T00:50:02.920 DEBUG:boto:Data: 2023-07-29T00:50:02.920 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.921 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:02.921 DEBUG:boto:Port: 8000 2023-07-29T00:50:02.921 DEBUG:boto:Params: {} 2023-07-29T00:50:02.922 DEBUG:boto:Token: None 2023-07-29T00:50:02.922 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/ 2023-07-29T00:50:02.923 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6jMSgLwXCW6tFHtDjt7WrDtXJ2g= 2023-07-29T00:50:02.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6jMSgLwXCW6tFHtDjt7WrDtXJ2g=', 'Content-Length': '0'} 2023-07-29T00:50:02.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041f8455d9d442fca-0064c4623a-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.927 DEBUG:boto:b'hiiqos-61obj331000falseobj332023-07-29T00:45:01.083Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:02.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-07-29T00:50:02.928 DEBUG:boto:path=/hiiqos-61/obj34 2023-07-29T00:50:02.929 DEBUG:boto:auth_path=/hiiqos-61/obj34 2023-07-29T00:50:02.929 DEBUG:boto:path=/hiiqos-61/obj34?versionId=null 2023-07-29T00:50:02.929 DEBUG:boto:auth_path=/hiiqos-61/obj34?versionId=null 2023-07-29T00:50:02.930 DEBUG:boto:Method: GET 2023-07-29T00:50:02.930 DEBUG:boto:Path: /hiiqos-61/obj34?versionId=null 2023-07-29T00:50:02.931 DEBUG:boto:Data: 2023-07-29T00:50:02.931 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.931 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:02.932 DEBUG:boto:Port: 8001 2023-07-29T00:50:02.932 DEBUG:boto:Params: {} 2023-07-29T00:50:02.932 DEBUG:boto:Token: None 2023-07-29T00:50:02.933 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj34?versionId=null 2023-07-29T00:50:02.933 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:u87wkMpf+fJZORelw04vchwaTPM= 2023-07-29T00:50:02.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:u87wkMpf+fJZORelw04vchwaTPM=', 'Content-Length': '0'} 2023-07-29T00:50:02.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0e27cd3675ac034-0064c4623a-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.976 DEBUG:boto:path=/hiiqos-61/obj34 2023-07-29T00:50:02.977 DEBUG:boto:auth_path=/hiiqos-61/obj34 2023-07-29T00:50:02.977 DEBUG:boto:path=/hiiqos-61/obj34?versionId=null 2023-07-29T00:50:02.978 DEBUG:boto:auth_path=/hiiqos-61/obj34?versionId=null 2023-07-29T00:50:02.978 DEBUG:boto:Method: GET 2023-07-29T00:50:02.978 DEBUG:boto:Path: /hiiqos-61/obj34?versionId=null 2023-07-29T00:50:02.979 DEBUG:boto:Data: 2023-07-29T00:50:02.979 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.979 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:02.980 DEBUG:boto:Port: 8000 2023-07-29T00:50:02.980 DEBUG:boto:Params: {} 2023-07-29T00:50:02.981 DEBUG:boto:Token: None 2023-07-29T00:50:02.981 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj34?versionId=null 2023-07-29T00:50:02.982 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:u87wkMpf+fJZORelw04vchwaTPM= 2023-07-29T00:50:02.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:u87wkMpf+fJZORelw04vchwaTPM=', 'Content-Length': '0'} 2023-07-29T00:50:02.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aeacc10d64bc76c3-0064c4623a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.985 DEBUG:boto:path=/hiiqos-61/obj34 2023-07-29T00:50:02.985 DEBUG:boto:auth_path=/hiiqos-61/obj34 2023-07-29T00:50:02.985 DEBUG:boto:path=/hiiqos-61/obj34?versionId=null 2023-07-29T00:50:02.986 DEBUG:boto:auth_path=/hiiqos-61/obj34?versionId=null 2023-07-29T00:50:02.986 DEBUG:boto:Method: HEAD 2023-07-29T00:50:02.986 DEBUG:boto:Path: /hiiqos-61/obj34?versionId=null 2023-07-29T00:50:02.987 DEBUG:boto:Data: 2023-07-29T00:50:02.987 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.987 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:02.988 DEBUG:boto:Port: 8001 2023-07-29T00:50:02.988 DEBUG:boto:Params: {} 2023-07-29T00:50:02.989 DEBUG:boto:Token: None 2023-07-29T00:50:02.989 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:02 GMT /hiiqos-61/obj34?versionId=null 2023-07-29T00:50:02.989 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LX5gVV7leHbjGhiBFISX+XBBQdY= 2023-07-29T00:50:02.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LX5gVV7leHbjGhiBFISX+XBBQdY=', 'Content-Length': '0'} 2023-07-29T00:50:02.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aea9028ff92d2df8-0064c4623a-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:02.992 DEBUG:boto:path=/hiiqos-61/obj34 2023-07-29T00:50:02.993 DEBUG:boto:auth_path=/hiiqos-61/obj34 2023-07-29T00:50:02.994 DEBUG:boto:path=/hiiqos-61/obj34?versionId=null 2023-07-29T00:50:02.994 DEBUG:boto:auth_path=/hiiqos-61/obj34?versionId=null 2023-07-29T00:50:02.995 DEBUG:boto:Method: HEAD 2023-07-29T00:50:02.996 DEBUG:boto:Path: /hiiqos-61/obj34?versionId=null 2023-07-29T00:50:02.997 DEBUG:boto:Data: 2023-07-29T00:50:02.998 DEBUG:boto:Headers: {} 2023-07-29T00:50:02.998 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:02.998 DEBUG:boto:Port: 8000 2023-07-29T00:50:02.999 DEBUG:boto:Params: {} 2023-07-29T00:50:02.999 DEBUG:boto:Token: None 2023-07-29T00:50:03.000 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj34?versionId=null 2023-07-29T00:50:03.000 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0Y9aAL9WZE4gJ3W1qv3mdsBvViA= 2023-07-29T00:50:03.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0Y9aAL9WZE4gJ3W1qv3mdsBvViA=', 'Content-Length': '0'} 2023-07-29T00:50:03.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000febacbe6b859fd32-0064c4623a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-07-29T00:50:03.003 DEBUG:boto:path=/hiiqos-61/obj34 2023-07-29T00:50:03.004 DEBUG:boto:auth_path=/hiiqos-61/obj34 2023-07-29T00:50:03.004 DEBUG:boto:path=/hiiqos-61/obj34?versionId=null 2023-07-29T00:50:03.005 DEBUG:boto:auth_path=/hiiqos-61/obj34?versionId=null 2023-07-29T00:50:03.005 DEBUG:boto:Method: GET 2023-07-29T00:50:03.005 DEBUG:boto:Path: /hiiqos-61/obj34?versionId=null 2023-07-29T00:50:03.006 DEBUG:boto:Data: 2023-07-29T00:50:03.006 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.006 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:03.007 DEBUG:boto:Port: 8001 2023-07-29T00:50:03.007 DEBUG:boto:Params: {} 2023-07-29T00:50:03.007 DEBUG:boto:Token: None 2023-07-29T00:50:03.008 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj34?versionId=null 2023-07-29T00:50:03.008 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/D+71xjzGSxUCnB4vgegBL+gdh0= 2023-07-29T00:50:03.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/D+71xjzGSxUCnB4vgegBL+gdh0=', 'Content-Length': '0'} 2023-07-29T00:50:03.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008af84ec3bc172919-0064c4623a-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.011 DEBUG:boto:path=/hiiqos-61/obj34 2023-07-29T00:50:03.011 DEBUG:boto:auth_path=/hiiqos-61/obj34 2023-07-29T00:50:03.012 DEBUG:boto:path=/hiiqos-61/obj34?versionId=null 2023-07-29T00:50:03.012 DEBUG:boto:auth_path=/hiiqos-61/obj34?versionId=null 2023-07-29T00:50:03.012 DEBUG:boto:Method: GET 2023-07-29T00:50:03.013 DEBUG:boto:Path: /hiiqos-61/obj34?versionId=null 2023-07-29T00:50:03.013 DEBUG:boto:Data: 2023-07-29T00:50:03.014 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.014 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:03.014 DEBUG:boto:Port: 8000 2023-07-29T00:50:03.015 DEBUG:boto:Params: {} 2023-07-29T00:50:03.015 DEBUG:boto:Token: None 2023-07-29T00:50:03.015 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj34?versionId=null 2023-07-29T00:50:03.016 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/D+71xjzGSxUCnB4vgegBL+gdh0= 2023-07-29T00:50:03.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/D+71xjzGSxUCnB4vgegBL+gdh0=', 'Content-Length': '0'} 2023-07-29T00:50:03.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e76b33ccc09d098-0064c4623a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.019 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:03.019 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:03.019 DEBUG:boto:path=/hiiqos-61/?prefix=obj34 2023-07-29T00:50:03.020 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj34 2023-07-29T00:50:03.020 DEBUG:boto:Method: GET 2023-07-29T00:50:03.021 DEBUG:boto:Path: /hiiqos-61/?prefix=obj34 2023-07-29T00:50:03.021 DEBUG:boto:Data: 2023-07-29T00:50:03.021 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.022 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:03.022 DEBUG:boto:Port: 8001 2023-07-29T00:50:03.022 DEBUG:boto:Params: {} 2023-07-29T00:50:03.023 DEBUG:boto:Token: None 2023-07-29T00:50:03.023 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/ 2023-07-29T00:50:03.024 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts= 2023-07-29T00:50:03.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts=', 'Content-Length': '0'} 2023-07-29T00:50:03.026 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056d07cc4c81e3f97-0064c4623a-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.027 DEBUG:boto:b'hiiqos-61obj341000falseobj342023-07-29T00:45:01.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:03.027 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:03.028 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:03.028 DEBUG:boto:path=/hiiqos-61/?prefix=obj34 2023-07-29T00:50:03.028 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj34 2023-07-29T00:50:03.029 DEBUG:boto:Method: GET 2023-07-29T00:50:03.029 DEBUG:boto:Path: /hiiqos-61/?prefix=obj34 2023-07-29T00:50:03.029 DEBUG:boto:Data: 2023-07-29T00:50:03.030 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.030 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:03.030 DEBUG:boto:Port: 8000 2023-07-29T00:50:03.031 DEBUG:boto:Params: {} 2023-07-29T00:50:03.031 DEBUG:boto:Token: None 2023-07-29T00:50:03.032 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/ 2023-07-29T00:50:03.032 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts= 2023-07-29T00:50:03.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts=', 'Content-Length': '0'} 2023-07-29T00:50:03.036 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a326a3e16f0a3693-0064c4623a-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.036 DEBUG:boto:b'hiiqos-61obj341000falseobj342023-07-29T00:45:01.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:03.037 DEBUG:boto:path=/hiiqos-61/obj34 2023-07-29T00:50:03.037 DEBUG:boto:auth_path=/hiiqos-61/obj34 2023-07-29T00:50:03.038 DEBUG:boto:Method: HEAD 2023-07-29T00:50:03.038 DEBUG:boto:Path: /hiiqos-61/obj34 2023-07-29T00:50:03.038 DEBUG:boto:Data: 2023-07-29T00:50:03.039 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.039 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:03.039 DEBUG:boto:Port: 8001 2023-07-29T00:50:03.040 DEBUG:boto:Params: {} 2023-07-29T00:50:03.040 DEBUG:boto:Token: None 2023-07-29T00:50:03.041 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj34 2023-07-29T00:50:03.041 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CrvWScaWW3MjDb+mm0hIyyY5Hw4= 2023-07-29T00:50:03.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CrvWScaWW3MjDb+mm0hIyyY5Hw4=', 'Content-Length': '0'} 2023-07-29T00:50:03.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f61cc06b5aca7028-0064c4623a-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.044 DEBUG:boto:path=/hiiqos-61/obj34 2023-07-29T00:50:03.044 DEBUG:boto:auth_path=/hiiqos-61/obj34 2023-07-29T00:50:03.044 DEBUG:boto:Method: HEAD 2023-07-29T00:50:03.045 DEBUG:boto:Path: /hiiqos-61/obj34 2023-07-29T00:50:03.045 DEBUG:boto:Data: 2023-07-29T00:50:03.045 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.046 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:03.046 DEBUG:boto:Port: 8000 2023-07-29T00:50:03.046 DEBUG:boto:Params: {} 2023-07-29T00:50:03.047 DEBUG:boto:Token: None 2023-07-29T00:50:03.047 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj34 2023-07-29T00:50:03.048 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CrvWScaWW3MjDb+mm0hIyyY5Hw4= 2023-07-29T00:50:03.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CrvWScaWW3MjDb+mm0hIyyY5Hw4=', 'Content-Length': '0'} 2023-07-29T00:50:03.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca5683324227e27c-0064c4623a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-07-29T00:50:03.051 DEBUG:boto:path=/hiiqos-61/obj34 2023-07-29T00:50:03.051 DEBUG:boto:auth_path=/hiiqos-61/obj34 2023-07-29T00:50:03.051 DEBUG:boto:Method: GET 2023-07-29T00:50:03.052 DEBUG:boto:Path: /hiiqos-61/obj34 2023-07-29T00:50:03.052 DEBUG:boto:Data: 2023-07-29T00:50:03.053 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.053 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:03.053 DEBUG:boto:Port: 8001 2023-07-29T00:50:03.054 DEBUG:boto:Params: {} 2023-07-29T00:50:03.054 DEBUG:boto:Token: None 2023-07-29T00:50:03.054 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj34 2023-07-29T00:50:03.055 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0h9AbMRNyntYpo1rbuY1Bhbw8pI= 2023-07-29T00:50:03.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0h9AbMRNyntYpo1rbuY1Bhbw8pI=', 'Content-Length': '0'} 2023-07-29T00:50:03.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003f58620489b6af5-0064c4623a-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.058 DEBUG:boto:path=/hiiqos-61/obj34 2023-07-29T00:50:03.058 DEBUG:boto:auth_path=/hiiqos-61/obj34 2023-07-29T00:50:03.058 DEBUG:boto:Method: GET 2023-07-29T00:50:03.059 DEBUG:boto:Path: /hiiqos-61/obj34 2023-07-29T00:50:03.059 DEBUG:boto:Data: 2023-07-29T00:50:03.059 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.060 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:03.060 DEBUG:boto:Port: 8000 2023-07-29T00:50:03.060 DEBUG:boto:Params: {} 2023-07-29T00:50:03.061 DEBUG:boto:Token: None 2023-07-29T00:50:03.061 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj34 2023-07-29T00:50:03.062 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0h9AbMRNyntYpo1rbuY1Bhbw8pI= 2023-07-29T00:50:03.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0h9AbMRNyntYpo1rbuY1Bhbw8pI=', 'Content-Length': '0'} 2023-07-29T00:50:03.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f9f86ada6090eeb-0064c4623a-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.064 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:03.065 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:03.065 DEBUG:boto:path=/hiiqos-61/?prefix=obj34 2023-07-29T00:50:03.066 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj34 2023-07-29T00:50:03.066 DEBUG:boto:Method: GET 2023-07-29T00:50:03.066 DEBUG:boto:Path: /hiiqos-61/?prefix=obj34 2023-07-29T00:50:03.067 DEBUG:boto:Data: 2023-07-29T00:50:03.067 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.067 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:03.068 DEBUG:boto:Port: 8001 2023-07-29T00:50:03.068 DEBUG:boto:Params: {} 2023-07-29T00:50:03.068 DEBUG:boto:Token: None 2023-07-29T00:50:03.069 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/ 2023-07-29T00:50:03.069 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts= 2023-07-29T00:50:03.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts=', 'Content-Length': '0'} 2023-07-29T00:50:03.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007dd14054f269e0de-0064c4623a-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.073 DEBUG:boto:b'hiiqos-61obj341000falseobj342023-07-29T00:45:01.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:03.073 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:03.074 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:03.074 DEBUG:boto:path=/hiiqos-61/?prefix=obj34 2023-07-29T00:50:03.075 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj34 2023-07-29T00:50:03.075 DEBUG:boto:Method: GET 2023-07-29T00:50:03.075 DEBUG:boto:Path: /hiiqos-61/?prefix=obj34 2023-07-29T00:50:03.076 DEBUG:boto:Data: 2023-07-29T00:50:03.076 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.076 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:03.077 DEBUG:boto:Port: 8000 2023-07-29T00:50:03.077 DEBUG:boto:Params: {} 2023-07-29T00:50:03.078 DEBUG:boto:Token: None 2023-07-29T00:50:03.078 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/ 2023-07-29T00:50:03.078 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts= 2023-07-29T00:50:03.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts=', 'Content-Length': '0'} 2023-07-29T00:50:03.083 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073ba6afdd8a4a65f-0064c4623b-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.083 DEBUG:boto:b'hiiqos-61obj341000falseobj342023-07-29T00:45:01.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:03.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-07-29T00:50:03.084 DEBUG:boto:path=/hiiqos-61/obj37 2023-07-29T00:50:03.085 DEBUG:boto:auth_path=/hiiqos-61/obj37 2023-07-29T00:50:03.085 DEBUG:boto:path=/hiiqos-61/obj37?versionId=null 2023-07-29T00:50:03.085 DEBUG:boto:auth_path=/hiiqos-61/obj37?versionId=null 2023-07-29T00:50:03.086 DEBUG:boto:Method: GET 2023-07-29T00:50:03.086 DEBUG:boto:Path: /hiiqos-61/obj37?versionId=null 2023-07-29T00:50:03.086 DEBUG:boto:Data: 2023-07-29T00:50:03.087 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.087 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:03.088 DEBUG:boto:Port: 8001 2023-07-29T00:50:03.088 DEBUG:boto:Params: {} 2023-07-29T00:50:03.088 DEBUG:boto:Token: None 2023-07-29T00:50:03.089 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj37?versionId=null 2023-07-29T00:50:03.089 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xayUUNq+DVaneg74MHLfVHEireU= 2023-07-29T00:50:03.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xayUUNq+DVaneg74MHLfVHEireU=', 'Content-Length': '0'} 2023-07-29T00:50:03.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007139c4a7ca568cf2-0064c4623b-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.132 DEBUG:boto:path=/hiiqos-61/obj37 2023-07-29T00:50:03.133 DEBUG:boto:auth_path=/hiiqos-61/obj37 2023-07-29T00:50:03.133 DEBUG:boto:path=/hiiqos-61/obj37?versionId=null 2023-07-29T00:50:03.134 DEBUG:boto:auth_path=/hiiqos-61/obj37?versionId=null 2023-07-29T00:50:03.134 DEBUG:boto:Method: GET 2023-07-29T00:50:03.134 DEBUG:boto:Path: /hiiqos-61/obj37?versionId=null 2023-07-29T00:50:03.135 DEBUG:boto:Data: 2023-07-29T00:50:03.136 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.137 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:03.138 DEBUG:boto:Port: 8000 2023-07-29T00:50:03.139 DEBUG:boto:Params: {} 2023-07-29T00:50:03.140 DEBUG:boto:Token: None 2023-07-29T00:50:03.141 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj37?versionId=null 2023-07-29T00:50:03.142 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xayUUNq+DVaneg74MHLfVHEireU= 2023-07-29T00:50:03.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xayUUNq+DVaneg74MHLfVHEireU=', 'Content-Length': '0'} 2023-07-29T00:50:03.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6065419dae0656a-0064c4623b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.146 DEBUG:boto:path=/hiiqos-61/obj37 2023-07-29T00:50:03.147 DEBUG:boto:auth_path=/hiiqos-61/obj37 2023-07-29T00:50:03.147 DEBUG:boto:path=/hiiqos-61/obj37?versionId=null 2023-07-29T00:50:03.147 DEBUG:boto:auth_path=/hiiqos-61/obj37?versionId=null 2023-07-29T00:50:03.148 DEBUG:boto:Method: HEAD 2023-07-29T00:50:03.148 DEBUG:boto:Path: /hiiqos-61/obj37?versionId=null 2023-07-29T00:50:03.148 DEBUG:boto:Data: 2023-07-29T00:50:03.149 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.149 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:03.149 DEBUG:boto:Port: 8001 2023-07-29T00:50:03.150 DEBUG:boto:Params: {} 2023-07-29T00:50:03.150 DEBUG:boto:Token: None 2023-07-29T00:50:03.151 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj37?versionId=null 2023-07-29T00:50:03.152 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+2UHzYxX8SkYcOzx0OzPb4e/FbU= 2023-07-29T00:50:03.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+2UHzYxX8SkYcOzx0OzPb4e/FbU=', 'Content-Length': '0'} 2023-07-29T00:50:03.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000009e1a0295030e81-0064c4623b-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.155 DEBUG:boto:path=/hiiqos-61/obj37 2023-07-29T00:50:03.156 DEBUG:boto:auth_path=/hiiqos-61/obj37 2023-07-29T00:50:03.156 DEBUG:boto:path=/hiiqos-61/obj37?versionId=null 2023-07-29T00:50:03.156 DEBUG:boto:auth_path=/hiiqos-61/obj37?versionId=null 2023-07-29T00:50:03.157 DEBUG:boto:Method: HEAD 2023-07-29T00:50:03.157 DEBUG:boto:Path: /hiiqos-61/obj37?versionId=null 2023-07-29T00:50:03.157 DEBUG:boto:Data: 2023-07-29T00:50:03.158 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.158 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:03.158 DEBUG:boto:Port: 8000 2023-07-29T00:50:03.159 DEBUG:boto:Params: {} 2023-07-29T00:50:03.159 DEBUG:boto:Token: None 2023-07-29T00:50:03.160 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj37?versionId=null 2023-07-29T00:50:03.160 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+2UHzYxX8SkYcOzx0OzPb4e/FbU= 2023-07-29T00:50:03.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+2UHzYxX8SkYcOzx0OzPb4e/FbU=', 'Content-Length': '0'} 2023-07-29T00:50:03.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a930db2768474aa7-0064c4623b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-07-29T00:50:03.163 DEBUG:boto:path=/hiiqos-61/obj37 2023-07-29T00:50:03.164 DEBUG:boto:auth_path=/hiiqos-61/obj37 2023-07-29T00:50:03.164 DEBUG:boto:path=/hiiqos-61/obj37?versionId=null 2023-07-29T00:50:03.165 DEBUG:boto:auth_path=/hiiqos-61/obj37?versionId=null 2023-07-29T00:50:03.165 DEBUG:boto:Method: GET 2023-07-29T00:50:03.165 DEBUG:boto:Path: /hiiqos-61/obj37?versionId=null 2023-07-29T00:50:03.166 DEBUG:boto:Data: 2023-07-29T00:50:03.166 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.166 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:03.167 DEBUG:boto:Port: 8001 2023-07-29T00:50:03.167 DEBUG:boto:Params: {} 2023-07-29T00:50:03.167 DEBUG:boto:Token: None 2023-07-29T00:50:03.168 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj37?versionId=null 2023-07-29T00:50:03.168 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xayUUNq+DVaneg74MHLfVHEireU= 2023-07-29T00:50:03.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xayUUNq+DVaneg74MHLfVHEireU=', 'Content-Length': '0'} 2023-07-29T00:50:03.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023d76cf49907443c-0064c4623b-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.171 DEBUG:boto:path=/hiiqos-61/obj37 2023-07-29T00:50:03.171 DEBUG:boto:auth_path=/hiiqos-61/obj37 2023-07-29T00:50:03.172 DEBUG:boto:path=/hiiqos-61/obj37?versionId=null 2023-07-29T00:50:03.172 DEBUG:boto:auth_path=/hiiqos-61/obj37?versionId=null 2023-07-29T00:50:03.173 DEBUG:boto:Method: GET 2023-07-29T00:50:03.173 DEBUG:boto:Path: /hiiqos-61/obj37?versionId=null 2023-07-29T00:50:03.173 DEBUG:boto:Data: 2023-07-29T00:50:03.174 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.174 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:03.175 DEBUG:boto:Port: 8000 2023-07-29T00:50:03.175 DEBUG:boto:Params: {} 2023-07-29T00:50:03.176 DEBUG:boto:Token: None 2023-07-29T00:50:03.176 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj37?versionId=null 2023-07-29T00:50:03.176 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xayUUNq+DVaneg74MHLfVHEireU= 2023-07-29T00:50:03.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xayUUNq+DVaneg74MHLfVHEireU=', 'Content-Length': '0'} 2023-07-29T00:50:03.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b619637a2140f73d-0064c4623b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.179 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:03.180 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:03.180 DEBUG:boto:path=/hiiqos-61/?prefix=obj37 2023-07-29T00:50:03.181 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj37 2023-07-29T00:50:03.181 DEBUG:boto:Method: GET 2023-07-29T00:50:03.181 DEBUG:boto:Path: /hiiqos-61/?prefix=obj37 2023-07-29T00:50:03.182 DEBUG:boto:Data: 2023-07-29T00:50:03.182 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.182 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:03.183 DEBUG:boto:Port: 8001 2023-07-29T00:50:03.183 DEBUG:boto:Params: {} 2023-07-29T00:50:03.183 DEBUG:boto:Token: None 2023-07-29T00:50:03.184 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/ 2023-07-29T00:50:03.184 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts= 2023-07-29T00:50:03.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts=', 'Content-Length': '0'} 2023-07-29T00:50:03.187 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033ae15ff05949104-0064c4623b-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.187 DEBUG:boto:b'hiiqos-61obj371000falseobj372023-07-29T00:45:03.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:03.188 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:03.189 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:03.189 DEBUG:boto:path=/hiiqos-61/?prefix=obj37 2023-07-29T00:50:03.189 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj37 2023-07-29T00:50:03.190 DEBUG:boto:Method: GET 2023-07-29T00:50:03.190 DEBUG:boto:Path: /hiiqos-61/?prefix=obj37 2023-07-29T00:50:03.190 DEBUG:boto:Data: 2023-07-29T00:50:03.191 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.191 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:03.191 DEBUG:boto:Port: 8000 2023-07-29T00:50:03.192 DEBUG:boto:Params: {} 2023-07-29T00:50:03.192 DEBUG:boto:Token: None 2023-07-29T00:50:03.193 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/ 2023-07-29T00:50:03.193 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts= 2023-07-29T00:50:03.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts=', 'Content-Length': '0'} 2023-07-29T00:50:03.197 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076e31dca15e6bc94-0064c4623b-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.198 DEBUG:boto:b'hiiqos-61obj371000falseobj372023-07-29T00:45:03.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:03.199 DEBUG:boto:path=/hiiqos-61/obj37 2023-07-29T00:50:03.199 DEBUG:boto:auth_path=/hiiqos-61/obj37 2023-07-29T00:50:03.199 DEBUG:boto:Method: HEAD 2023-07-29T00:50:03.200 DEBUG:boto:Path: /hiiqos-61/obj37 2023-07-29T00:50:03.201 DEBUG:boto:Data: 2023-07-29T00:50:03.202 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.202 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:03.203 DEBUG:boto:Port: 8001 2023-07-29T00:50:03.203 DEBUG:boto:Params: {} 2023-07-29T00:50:03.203 DEBUG:boto:Token: None 2023-07-29T00:50:03.204 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj37 2023-07-29T00:50:03.204 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LbUMTGP989/7aOMDzkZZCWYturA= 2023-07-29T00:50:03.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LbUMTGP989/7aOMDzkZZCWYturA=', 'Content-Length': '0'} 2023-07-29T00:50:03.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098906b070085ed0e-0064c4623b-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.207 DEBUG:boto:path=/hiiqos-61/obj37 2023-07-29T00:50:03.207 DEBUG:boto:auth_path=/hiiqos-61/obj37 2023-07-29T00:50:03.208 DEBUG:boto:Method: HEAD 2023-07-29T00:50:03.208 DEBUG:boto:Path: /hiiqos-61/obj37 2023-07-29T00:50:03.208 DEBUG:boto:Data: 2023-07-29T00:50:03.209 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.209 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:03.210 DEBUG:boto:Port: 8000 2023-07-29T00:50:03.210 DEBUG:boto:Params: {} 2023-07-29T00:50:03.211 DEBUG:boto:Token: None 2023-07-29T00:50:03.211 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj37 2023-07-29T00:50:03.212 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LbUMTGP989/7aOMDzkZZCWYturA= 2023-07-29T00:50:03.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LbUMTGP989/7aOMDzkZZCWYturA=', 'Content-Length': '0'} 2023-07-29T00:50:03.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004677ee203d868420-0064c4623b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-07-29T00:50:03.215 DEBUG:boto:path=/hiiqos-61/obj37 2023-07-29T00:50:03.215 DEBUG:boto:auth_path=/hiiqos-61/obj37 2023-07-29T00:50:03.216 DEBUG:boto:Method: GET 2023-07-29T00:50:03.216 DEBUG:boto:Path: /hiiqos-61/obj37 2023-07-29T00:50:03.217 DEBUG:boto:Data: 2023-07-29T00:50:03.217 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.217 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:03.218 DEBUG:boto:Port: 8001 2023-07-29T00:50:03.218 DEBUG:boto:Params: {} 2023-07-29T00:50:03.218 DEBUG:boto:Token: None 2023-07-29T00:50:03.219 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj37 2023-07-29T00:50:03.219 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:u9cYrckeedQFQ+2yIY0uHRuevyo= 2023-07-29T00:50:03.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:u9cYrckeedQFQ+2yIY0uHRuevyo=', 'Content-Length': '0'} 2023-07-29T00:50:03.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000996dfb4fa166fe0c-0064c4623b-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.222 DEBUG:boto:path=/hiiqos-61/obj37 2023-07-29T00:50:03.222 DEBUG:boto:auth_path=/hiiqos-61/obj37 2023-07-29T00:50:03.223 DEBUG:boto:Method: GET 2023-07-29T00:50:03.223 DEBUG:boto:Path: /hiiqos-61/obj37 2023-07-29T00:50:03.224 DEBUG:boto:Data: 2023-07-29T00:50:03.224 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.224 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:03.225 DEBUG:boto:Port: 8000 2023-07-29T00:50:03.225 DEBUG:boto:Params: {} 2023-07-29T00:50:03.226 DEBUG:boto:Token: None 2023-07-29T00:50:03.226 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj37 2023-07-29T00:50:03.227 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:u9cYrckeedQFQ+2yIY0uHRuevyo= 2023-07-29T00:50:03.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:u9cYrckeedQFQ+2yIY0uHRuevyo=', 'Content-Length': '0'} 2023-07-29T00:50:03.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5a8ff90fda4e108-0064c4623b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.272 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:03.273 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:03.273 DEBUG:boto:path=/hiiqos-61/?prefix=obj37 2023-07-29T00:50:03.274 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj37 2023-07-29T00:50:03.274 DEBUG:boto:Method: GET 2023-07-29T00:50:03.274 DEBUG:boto:Path: /hiiqos-61/?prefix=obj37 2023-07-29T00:50:03.275 DEBUG:boto:Data: 2023-07-29T00:50:03.275 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.275 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:03.276 DEBUG:boto:Port: 8001 2023-07-29T00:50:03.276 DEBUG:boto:Params: {} 2023-07-29T00:50:03.277 DEBUG:boto:Token: None 2023-07-29T00:50:03.277 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/ 2023-07-29T00:50:03.277 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts= 2023-07-29T00:50:03.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts=', 'Content-Length': '0'} 2023-07-29T00:50:03.280 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de3a0d6d71aa5e96-0064c4623b-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.281 DEBUG:boto:b'hiiqos-61obj371000falseobj372023-07-29T00:45:03.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:03.281 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:03.282 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:03.282 DEBUG:boto:path=/hiiqos-61/?prefix=obj37 2023-07-29T00:50:03.282 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj37 2023-07-29T00:50:03.283 DEBUG:boto:Method: GET 2023-07-29T00:50:03.283 DEBUG:boto:Path: /hiiqos-61/?prefix=obj37 2023-07-29T00:50:03.283 DEBUG:boto:Data: 2023-07-29T00:50:03.284 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.284 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:03.285 DEBUG:boto:Port: 8000 2023-07-29T00:50:03.285 DEBUG:boto:Params: {} 2023-07-29T00:50:03.285 DEBUG:boto:Token: None 2023-07-29T00:50:03.286 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/ 2023-07-29T00:50:03.286 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts= 2023-07-29T00:50:03.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts=', 'Content-Length': '0'} 2023-07-29T00:50:03.290 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db40c83887323b37-0064c4623b-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.291 DEBUG:boto:b'hiiqos-61obj371000falseobj372023-07-29T00:45:03.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:03.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-07-29T00:50:03.292 DEBUG:boto:path=/hiiqos-61/obj38 2023-07-29T00:50:03.292 DEBUG:boto:auth_path=/hiiqos-61/obj38 2023-07-29T00:50:03.293 DEBUG:boto:path=/hiiqos-61/obj38?versionId=null 2023-07-29T00:50:03.293 DEBUG:boto:auth_path=/hiiqos-61/obj38?versionId=null 2023-07-29T00:50:03.293 DEBUG:boto:Method: GET 2023-07-29T00:50:03.294 DEBUG:boto:Path: /hiiqos-61/obj38?versionId=null 2023-07-29T00:50:03.294 DEBUG:boto:Data: 2023-07-29T00:50:03.294 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.295 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:03.295 DEBUG:boto:Port: 8001 2023-07-29T00:50:03.295 DEBUG:boto:Params: {} 2023-07-29T00:50:03.296 DEBUG:boto:Token: None 2023-07-29T00:50:03.296 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj38?versionId=null 2023-07-29T00:50:03.297 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:H//iSwovnxjKc6RVvviP7vGilXE= 2023-07-29T00:50:03.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:H//iSwovnxjKc6RVvviP7vGilXE=', 'Content-Length': '0'} 2023-07-29T00:50:03.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067c170867a810d0c-0064c4623b-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.299 DEBUG:boto:path=/hiiqos-61/obj38 2023-07-29T00:50:03.300 DEBUG:boto:auth_path=/hiiqos-61/obj38 2023-07-29T00:50:03.300 DEBUG:boto:path=/hiiqos-61/obj38?versionId=null 2023-07-29T00:50:03.301 DEBUG:boto:auth_path=/hiiqos-61/obj38?versionId=null 2023-07-29T00:50:03.301 DEBUG:boto:Method: GET 2023-07-29T00:50:03.301 DEBUG:boto:Path: /hiiqos-61/obj38?versionId=null 2023-07-29T00:50:03.302 DEBUG:boto:Data: 2023-07-29T00:50:03.302 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.302 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:03.303 DEBUG:boto:Port: 8000 2023-07-29T00:50:03.303 DEBUG:boto:Params: {} 2023-07-29T00:50:03.303 DEBUG:boto:Token: None 2023-07-29T00:50:03.304 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj38?versionId=null 2023-07-29T00:50:03.304 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:H//iSwovnxjKc6RVvviP7vGilXE= 2023-07-29T00:50:03.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:H//iSwovnxjKc6RVvviP7vGilXE=', 'Content-Length': '0'} 2023-07-29T00:50:03.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008dbfb8af14ef4c86-0064c4623b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.307 DEBUG:boto:path=/hiiqos-61/obj38 2023-07-29T00:50:03.307 DEBUG:boto:auth_path=/hiiqos-61/obj38 2023-07-29T00:50:03.308 DEBUG:boto:path=/hiiqos-61/obj38?versionId=null 2023-07-29T00:50:03.308 DEBUG:boto:auth_path=/hiiqos-61/obj38?versionId=null 2023-07-29T00:50:03.308 DEBUG:boto:Method: HEAD 2023-07-29T00:50:03.309 DEBUG:boto:Path: /hiiqos-61/obj38?versionId=null 2023-07-29T00:50:03.309 DEBUG:boto:Data: 2023-07-29T00:50:03.309 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.310 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:03.310 DEBUG:boto:Port: 8001 2023-07-29T00:50:03.310 DEBUG:boto:Params: {} 2023-07-29T00:50:03.311 DEBUG:boto:Token: None 2023-07-29T00:50:03.311 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj38?versionId=null 2023-07-29T00:50:03.312 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:acPZHVuTGtekzMDiNrsOaIksLis= 2023-07-29T00:50:03.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:acPZHVuTGtekzMDiNrsOaIksLis=', 'Content-Length': '0'} 2023-07-29T00:50:03.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffe618b7372ce42a-0064c4623b-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.314 DEBUG:boto:path=/hiiqos-61/obj38 2023-07-29T00:50:03.315 DEBUG:boto:auth_path=/hiiqos-61/obj38 2023-07-29T00:50:03.315 DEBUG:boto:path=/hiiqos-61/obj38?versionId=null 2023-07-29T00:50:03.315 DEBUG:boto:auth_path=/hiiqos-61/obj38?versionId=null 2023-07-29T00:50:03.316 DEBUG:boto:Method: HEAD 2023-07-29T00:50:03.316 DEBUG:boto:Path: /hiiqos-61/obj38?versionId=null 2023-07-29T00:50:03.316 DEBUG:boto:Data: 2023-07-29T00:50:03.317 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.317 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:03.317 DEBUG:boto:Port: 8000 2023-07-29T00:50:03.318 DEBUG:boto:Params: {} 2023-07-29T00:50:03.318 DEBUG:boto:Token: None 2023-07-29T00:50:03.318 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj38?versionId=null 2023-07-29T00:50:03.319 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:acPZHVuTGtekzMDiNrsOaIksLis= 2023-07-29T00:50:03.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:acPZHVuTGtekzMDiNrsOaIksLis=', 'Content-Length': '0'} 2023-07-29T00:50:03.321 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5409f6ad6444afa-0064c4623b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-07-29T00:50:03.321 DEBUG:boto:path=/hiiqos-61/obj38 2023-07-29T00:50:03.322 DEBUG:boto:auth_path=/hiiqos-61/obj38 2023-07-29T00:50:03.322 DEBUG:boto:path=/hiiqos-61/obj38?versionId=null 2023-07-29T00:50:03.323 DEBUG:boto:auth_path=/hiiqos-61/obj38?versionId=null 2023-07-29T00:50:03.323 DEBUG:boto:Method: GET 2023-07-29T00:50:03.323 DEBUG:boto:Path: /hiiqos-61/obj38?versionId=null 2023-07-29T00:50:03.324 DEBUG:boto:Data: 2023-07-29T00:50:03.324 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.324 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:03.325 DEBUG:boto:Port: 8001 2023-07-29T00:50:03.325 DEBUG:boto:Params: {} 2023-07-29T00:50:03.325 DEBUG:boto:Token: None 2023-07-29T00:50:03.326 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj38?versionId=null 2023-07-29T00:50:03.326 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:H//iSwovnxjKc6RVvviP7vGilXE= 2023-07-29T00:50:03.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:H//iSwovnxjKc6RVvviP7vGilXE=', 'Content-Length': '0'} 2023-07-29T00:50:03.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9eb944a0afe2b68-0064c4623b-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.329 DEBUG:boto:path=/hiiqos-61/obj38 2023-07-29T00:50:03.330 DEBUG:boto:auth_path=/hiiqos-61/obj38 2023-07-29T00:50:03.330 DEBUG:boto:path=/hiiqos-61/obj38?versionId=null 2023-07-29T00:50:03.330 DEBUG:boto:auth_path=/hiiqos-61/obj38?versionId=null 2023-07-29T00:50:03.331 DEBUG:boto:Method: GET 2023-07-29T00:50:03.331 DEBUG:boto:Path: /hiiqos-61/obj38?versionId=null 2023-07-29T00:50:03.331 DEBUG:boto:Data: 2023-07-29T00:50:03.332 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.332 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:03.332 DEBUG:boto:Port: 8000 2023-07-29T00:50:03.333 DEBUG:boto:Params: {} 2023-07-29T00:50:03.333 DEBUG:boto:Token: None 2023-07-29T00:50:03.334 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj38?versionId=null 2023-07-29T00:50:03.334 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:H//iSwovnxjKc6RVvviP7vGilXE= 2023-07-29T00:50:03.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:H//iSwovnxjKc6RVvviP7vGilXE=', 'Content-Length': '0'} 2023-07-29T00:50:03.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033cdca2c47f1eecc-0064c4623b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.384 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:03.385 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:03.385 DEBUG:boto:path=/hiiqos-61/?prefix=obj38 2023-07-29T00:50:03.386 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj38 2023-07-29T00:50:03.386 DEBUG:boto:Method: GET 2023-07-29T00:50:03.386 DEBUG:boto:Path: /hiiqos-61/?prefix=obj38 2023-07-29T00:50:03.387 DEBUG:boto:Data: 2023-07-29T00:50:03.387 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.387 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:03.388 DEBUG:boto:Port: 8001 2023-07-29T00:50:03.388 DEBUG:boto:Params: {} 2023-07-29T00:50:03.389 DEBUG:boto:Token: None 2023-07-29T00:50:03.389 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/ 2023-07-29T00:50:03.390 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts= 2023-07-29T00:50:03.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts=', 'Content-Length': '0'} 2023-07-29T00:50:03.392 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbd11ca96c20670c-0064c4623b-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.393 DEBUG:boto:b'hiiqos-61obj381000falseobj382023-07-29T00:45:01.691Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:03.393 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:03.394 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:03.394 DEBUG:boto:path=/hiiqos-61/?prefix=obj38 2023-07-29T00:50:03.395 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj38 2023-07-29T00:50:03.395 DEBUG:boto:Method: GET 2023-07-29T00:50:03.395 DEBUG:boto:Path: /hiiqos-61/?prefix=obj38 2023-07-29T00:50:03.396 DEBUG:boto:Data: 2023-07-29T00:50:03.396 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.396 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:03.397 DEBUG:boto:Port: 8000 2023-07-29T00:50:03.397 DEBUG:boto:Params: {} 2023-07-29T00:50:03.397 DEBUG:boto:Token: None 2023-07-29T00:50:03.398 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/ 2023-07-29T00:50:03.398 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts= 2023-07-29T00:50:03.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts=', 'Content-Length': '0'} 2023-07-29T00:50:03.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9886742ee47eddd-0064c4623b-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.403 DEBUG:boto:b'hiiqos-61obj381000falseobj382023-07-29T00:45:01.691Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:03.403 DEBUG:boto:path=/hiiqos-61/obj38 2023-07-29T00:50:03.404 DEBUG:boto:auth_path=/hiiqos-61/obj38 2023-07-29T00:50:03.404 DEBUG:boto:Method: HEAD 2023-07-29T00:50:03.405 DEBUG:boto:Path: /hiiqos-61/obj38 2023-07-29T00:50:03.405 DEBUG:boto:Data: 2023-07-29T00:50:03.405 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.406 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:03.406 DEBUG:boto:Port: 8001 2023-07-29T00:50:03.406 DEBUG:boto:Params: {} 2023-07-29T00:50:03.407 DEBUG:boto:Token: None 2023-07-29T00:50:03.407 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj38 2023-07-29T00:50:03.407 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:B8EGzNt5xJdIo74uAoBpfpao3is= 2023-07-29T00:50:03.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:B8EGzNt5xJdIo74uAoBpfpao3is=', 'Content-Length': '0'} 2023-07-29T00:50:03.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d150799daf822737-0064c4623b-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.410 DEBUG:boto:path=/hiiqos-61/obj38 2023-07-29T00:50:03.410 DEBUG:boto:auth_path=/hiiqos-61/obj38 2023-07-29T00:50:03.411 DEBUG:boto:Method: HEAD 2023-07-29T00:50:03.411 DEBUG:boto:Path: /hiiqos-61/obj38 2023-07-29T00:50:03.411 DEBUG:boto:Data: 2023-07-29T00:50:03.412 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.412 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:03.413 DEBUG:boto:Port: 8000 2023-07-29T00:50:03.413 DEBUG:boto:Params: {} 2023-07-29T00:50:03.413 DEBUG:boto:Token: None 2023-07-29T00:50:03.414 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj38 2023-07-29T00:50:03.414 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:B8EGzNt5xJdIo74uAoBpfpao3is= 2023-07-29T00:50:03.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:B8EGzNt5xJdIo74uAoBpfpao3is=', 'Content-Length': '0'} 2023-07-29T00:50:03.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a10ff71b3a12f133-0064c4623b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-07-29T00:50:03.417 DEBUG:boto:path=/hiiqos-61/obj38 2023-07-29T00:50:03.418 DEBUG:boto:auth_path=/hiiqos-61/obj38 2023-07-29T00:50:03.418 DEBUG:boto:Method: GET 2023-07-29T00:50:03.418 DEBUG:boto:Path: /hiiqos-61/obj38 2023-07-29T00:50:03.419 DEBUG:boto:Data: 2023-07-29T00:50:03.419 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.420 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:03.420 DEBUG:boto:Port: 8001 2023-07-29T00:50:03.420 DEBUG:boto:Params: {} 2023-07-29T00:50:03.421 DEBUG:boto:Token: None 2023-07-29T00:50:03.421 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj38 2023-07-29T00:50:03.421 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lQp/w5PHyHaYTEdLT/NBdrHMr28= 2023-07-29T00:50:03.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lQp/w5PHyHaYTEdLT/NBdrHMr28=', 'Content-Length': '0'} 2023-07-29T00:50:03.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026514627aa652884-0064c4623b-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.424 DEBUG:boto:path=/hiiqos-61/obj38 2023-07-29T00:50:03.425 DEBUG:boto:auth_path=/hiiqos-61/obj38 2023-07-29T00:50:03.425 DEBUG:boto:Method: GET 2023-07-29T00:50:03.425 DEBUG:boto:Path: /hiiqos-61/obj38 2023-07-29T00:50:03.426 DEBUG:boto:Data: 2023-07-29T00:50:03.426 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.426 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:03.427 DEBUG:boto:Port: 8000 2023-07-29T00:50:03.427 DEBUG:boto:Params: {} 2023-07-29T00:50:03.428 DEBUG:boto:Token: None 2023-07-29T00:50:03.428 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj38 2023-07-29T00:50:03.428 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lQp/w5PHyHaYTEdLT/NBdrHMr28= 2023-07-29T00:50:03.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lQp/w5PHyHaYTEdLT/NBdrHMr28=', 'Content-Length': '0'} 2023-07-29T00:50:03.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ece4bc5d868610a7-0064c4623b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.472 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:03.473 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:03.473 DEBUG:boto:path=/hiiqos-61/?prefix=obj38 2023-07-29T00:50:03.474 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj38 2023-07-29T00:50:03.474 DEBUG:boto:Method: GET 2023-07-29T00:50:03.474 DEBUG:boto:Path: /hiiqos-61/?prefix=obj38 2023-07-29T00:50:03.475 DEBUG:boto:Data: 2023-07-29T00:50:03.475 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.475 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:03.476 DEBUG:boto:Port: 8001 2023-07-29T00:50:03.476 DEBUG:boto:Params: {} 2023-07-29T00:50:03.477 DEBUG:boto:Token: None 2023-07-29T00:50:03.477 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/ 2023-07-29T00:50:03.477 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts= 2023-07-29T00:50:03.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts=', 'Content-Length': '0'} 2023-07-29T00:50:03.480 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c469f024fc9a378-0064c4623b-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.481 DEBUG:boto:b'hiiqos-61obj381000falseobj382023-07-29T00:45:01.691Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:03.481 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:03.482 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:03.482 DEBUG:boto:path=/hiiqos-61/?prefix=obj38 2023-07-29T00:50:03.482 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj38 2023-07-29T00:50:03.483 DEBUG:boto:Method: GET 2023-07-29T00:50:03.483 DEBUG:boto:Path: /hiiqos-61/?prefix=obj38 2023-07-29T00:50:03.483 DEBUG:boto:Data: 2023-07-29T00:50:03.484 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.484 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:03.484 DEBUG:boto:Port: 8000 2023-07-29T00:50:03.485 DEBUG:boto:Params: {} 2023-07-29T00:50:03.485 DEBUG:boto:Token: None 2023-07-29T00:50:03.486 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/ 2023-07-29T00:50:03.486 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts= 2023-07-29T00:50:03.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts=', 'Content-Length': '0'} 2023-07-29T00:50:03.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026a159d6bb9d666c-0064c4623b-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.491 DEBUG:boto:b'hiiqos-61obj381000falseobj382023-07-29T00:45:01.691Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:03.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-07-29T00:50:03.492 DEBUG:boto:path=/hiiqos-61/obj44 2023-07-29T00:50:03.492 DEBUG:boto:auth_path=/hiiqos-61/obj44 2023-07-29T00:50:03.492 DEBUG:boto:path=/hiiqos-61/obj44?versionId=null 2023-07-29T00:50:03.493 DEBUG:boto:auth_path=/hiiqos-61/obj44?versionId=null 2023-07-29T00:50:03.493 DEBUG:boto:Method: GET 2023-07-29T00:50:03.494 DEBUG:boto:Path: /hiiqos-61/obj44?versionId=null 2023-07-29T00:50:03.494 DEBUG:boto:Data: 2023-07-29T00:50:03.494 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.495 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:03.495 DEBUG:boto:Port: 8001 2023-07-29T00:50:03.495 DEBUG:boto:Params: {} 2023-07-29T00:50:03.496 DEBUG:boto:Token: None 2023-07-29T00:50:03.496 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj44?versionId=null 2023-07-29T00:50:03.497 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MVp1+fTd3W55cF1P+4U7QQvW1n0= 2023-07-29T00:50:03.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MVp1+fTd3W55cF1P+4U7QQvW1n0=', 'Content-Length': '0'} 2023-07-29T00:50:03.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd3c4c922ea03b35-0064c4623b-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.500 DEBUG:boto:path=/hiiqos-61/obj44 2023-07-29T00:50:03.500 DEBUG:boto:auth_path=/hiiqos-61/obj44 2023-07-29T00:50:03.500 DEBUG:boto:path=/hiiqos-61/obj44?versionId=null 2023-07-29T00:50:03.501 DEBUG:boto:auth_path=/hiiqos-61/obj44?versionId=null 2023-07-29T00:50:03.501 DEBUG:boto:Method: GET 2023-07-29T00:50:03.501 DEBUG:boto:Path: /hiiqos-61/obj44?versionId=null 2023-07-29T00:50:03.502 DEBUG:boto:Data: 2023-07-29T00:50:03.502 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.503 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:03.503 DEBUG:boto:Port: 8000 2023-07-29T00:50:03.503 DEBUG:boto:Params: {} 2023-07-29T00:50:03.504 DEBUG:boto:Token: None 2023-07-29T00:50:03.504 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj44?versionId=null 2023-07-29T00:50:03.505 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MVp1+fTd3W55cF1P+4U7QQvW1n0= 2023-07-29T00:50:03.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MVp1+fTd3W55cF1P+4U7QQvW1n0=', 'Content-Length': '0'} 2023-07-29T00:50:03.507 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa77bc3d0bf25cad-0064c4623b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.508 DEBUG:boto:path=/hiiqos-61/obj44 2023-07-29T00:50:03.508 DEBUG:boto:auth_path=/hiiqos-61/obj44 2023-07-29T00:50:03.509 DEBUG:boto:path=/hiiqos-61/obj44?versionId=null 2023-07-29T00:50:03.509 DEBUG:boto:auth_path=/hiiqos-61/obj44?versionId=null 2023-07-29T00:50:03.509 DEBUG:boto:Method: HEAD 2023-07-29T00:50:03.510 DEBUG:boto:Path: /hiiqos-61/obj44?versionId=null 2023-07-29T00:50:03.510 DEBUG:boto:Data: 2023-07-29T00:50:03.511 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.511 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:03.511 DEBUG:boto:Port: 8001 2023-07-29T00:50:03.512 DEBUG:boto:Params: {} 2023-07-29T00:50:03.512 DEBUG:boto:Token: None 2023-07-29T00:50:03.512 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj44?versionId=null 2023-07-29T00:50:03.513 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pfnrkpEA0jFgq6IXOZjQFpb/XZE= 2023-07-29T00:50:03.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pfnrkpEA0jFgq6IXOZjQFpb/XZE=', 'Content-Length': '0'} 2023-07-29T00:50:03.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000464fbf33a1e5ee1a-0064c4623b-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.516 DEBUG:boto:path=/hiiqos-61/obj44 2023-07-29T00:50:03.516 DEBUG:boto:auth_path=/hiiqos-61/obj44 2023-07-29T00:50:03.516 DEBUG:boto:path=/hiiqos-61/obj44?versionId=null 2023-07-29T00:50:03.517 DEBUG:boto:auth_path=/hiiqos-61/obj44?versionId=null 2023-07-29T00:50:03.517 DEBUG:boto:Method: HEAD 2023-07-29T00:50:03.517 DEBUG:boto:Path: /hiiqos-61/obj44?versionId=null 2023-07-29T00:50:03.518 DEBUG:boto:Data: 2023-07-29T00:50:03.518 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.518 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:03.519 DEBUG:boto:Port: 8000 2023-07-29T00:50:03.519 DEBUG:boto:Params: {} 2023-07-29T00:50:03.519 DEBUG:boto:Token: None 2023-07-29T00:50:03.520 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj44?versionId=null 2023-07-29T00:50:03.520 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pfnrkpEA0jFgq6IXOZjQFpb/XZE= 2023-07-29T00:50:03.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pfnrkpEA0jFgq6IXOZjQFpb/XZE=', 'Content-Length': '0'} 2023-07-29T00:50:03.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000469e8ade0ae50d34-0064c4623b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-07-29T00:50:03.523 DEBUG:boto:path=/hiiqos-61/obj44 2023-07-29T00:50:03.523 DEBUG:boto:auth_path=/hiiqos-61/obj44 2023-07-29T00:50:03.524 DEBUG:boto:path=/hiiqos-61/obj44?versionId=null 2023-07-29T00:50:03.524 DEBUG:boto:auth_path=/hiiqos-61/obj44?versionId=null 2023-07-29T00:50:03.525 DEBUG:boto:Method: GET 2023-07-29T00:50:03.525 DEBUG:boto:Path: /hiiqos-61/obj44?versionId=null 2023-07-29T00:50:03.525 DEBUG:boto:Data: 2023-07-29T00:50:03.526 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.526 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:03.526 DEBUG:boto:Port: 8001 2023-07-29T00:50:03.527 DEBUG:boto:Params: {} 2023-07-29T00:50:03.527 DEBUG:boto:Token: None 2023-07-29T00:50:03.528 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj44?versionId=null 2023-07-29T00:50:03.528 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MVp1+fTd3W55cF1P+4U7QQvW1n0= 2023-07-29T00:50:03.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MVp1+fTd3W55cF1P+4U7QQvW1n0=', 'Content-Length': '0'} 2023-07-29T00:50:03.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001656d166e2409556-0064c4623b-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.531 DEBUG:boto:path=/hiiqos-61/obj44 2023-07-29T00:50:03.531 DEBUG:boto:auth_path=/hiiqos-61/obj44 2023-07-29T00:50:03.532 DEBUG:boto:path=/hiiqos-61/obj44?versionId=null 2023-07-29T00:50:03.532 DEBUG:boto:auth_path=/hiiqos-61/obj44?versionId=null 2023-07-29T00:50:03.532 DEBUG:boto:Method: GET 2023-07-29T00:50:03.533 DEBUG:boto:Path: /hiiqos-61/obj44?versionId=null 2023-07-29T00:50:03.533 DEBUG:boto:Data: 2023-07-29T00:50:03.533 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.534 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:03.534 DEBUG:boto:Port: 8000 2023-07-29T00:50:03.534 DEBUG:boto:Params: {} 2023-07-29T00:50:03.535 DEBUG:boto:Token: None 2023-07-29T00:50:03.535 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj44?versionId=null 2023-07-29T00:50:03.536 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MVp1+fTd3W55cF1P+4U7QQvW1n0= 2023-07-29T00:50:03.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MVp1+fTd3W55cF1P+4U7QQvW1n0=', 'Content-Length': '0'} 2023-07-29T00:50:03.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062dadce7124fbd6a-0064c4623b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.539 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:03.539 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:03.539 DEBUG:boto:path=/hiiqos-61/?prefix=obj44 2023-07-29T00:50:03.540 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj44 2023-07-29T00:50:03.540 DEBUG:boto:Method: GET 2023-07-29T00:50:03.540 DEBUG:boto:Path: /hiiqos-61/?prefix=obj44 2023-07-29T00:50:03.541 DEBUG:boto:Data: 2023-07-29T00:50:03.541 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.542 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:03.542 DEBUG:boto:Port: 8001 2023-07-29T00:50:03.542 DEBUG:boto:Params: {} 2023-07-29T00:50:03.543 DEBUG:boto:Token: None 2023-07-29T00:50:03.543 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/ 2023-07-29T00:50:03.544 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts= 2023-07-29T00:50:03.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts=', 'Content-Length': '0'} 2023-07-29T00:50:03.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b407f4019c2c12c9-0064c4623b-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.547 DEBUG:boto:b'hiiqos-61obj441000falseobj442023-07-29T00:45:01.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:03.548 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:03.548 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:03.548 DEBUG:boto:path=/hiiqos-61/?prefix=obj44 2023-07-29T00:50:03.549 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj44 2023-07-29T00:50:03.549 DEBUG:boto:Method: GET 2023-07-29T00:50:03.550 DEBUG:boto:Path: /hiiqos-61/?prefix=obj44 2023-07-29T00:50:03.550 DEBUG:boto:Data: 2023-07-29T00:50:03.550 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.551 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:03.551 DEBUG:boto:Port: 8000 2023-07-29T00:50:03.551 DEBUG:boto:Params: {} 2023-07-29T00:50:03.552 DEBUG:boto:Token: None 2023-07-29T00:50:03.552 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/ 2023-07-29T00:50:03.553 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts= 2023-07-29T00:50:03.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts=', 'Content-Length': '0'} 2023-07-29T00:50:03.557 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005094015707794165-0064c4623b-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.557 DEBUG:boto:b'hiiqos-61obj441000falseobj442023-07-29T00:45:01.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:03.558 DEBUG:boto:path=/hiiqos-61/obj44 2023-07-29T00:50:03.558 DEBUG:boto:auth_path=/hiiqos-61/obj44 2023-07-29T00:50:03.559 DEBUG:boto:Method: HEAD 2023-07-29T00:50:03.559 DEBUG:boto:Path: /hiiqos-61/obj44 2023-07-29T00:50:03.559 DEBUG:boto:Data: 2023-07-29T00:50:03.560 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.560 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:03.560 DEBUG:boto:Port: 8001 2023-07-29T00:50:03.561 DEBUG:boto:Params: {} 2023-07-29T00:50:03.561 DEBUG:boto:Token: None 2023-07-29T00:50:03.561 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj44 2023-07-29T00:50:03.562 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FPcBUyIeDhykhXjpp89rtpWHSvI= 2023-07-29T00:50:03.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FPcBUyIeDhykhXjpp89rtpWHSvI=', 'Content-Length': '0'} 2023-07-29T00:50:03.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8993d1ca6950075-0064c4623b-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.564 DEBUG:boto:path=/hiiqos-61/obj44 2023-07-29T00:50:03.565 DEBUG:boto:auth_path=/hiiqos-61/obj44 2023-07-29T00:50:03.565 DEBUG:boto:Method: HEAD 2023-07-29T00:50:03.565 DEBUG:boto:Path: /hiiqos-61/obj44 2023-07-29T00:50:03.566 DEBUG:boto:Data: 2023-07-29T00:50:03.566 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.566 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:03.567 DEBUG:boto:Port: 8000 2023-07-29T00:50:03.567 DEBUG:boto:Params: {} 2023-07-29T00:50:03.567 DEBUG:boto:Token: None 2023-07-29T00:50:03.568 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj44 2023-07-29T00:50:03.568 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FPcBUyIeDhykhXjpp89rtpWHSvI= 2023-07-29T00:50:03.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FPcBUyIeDhykhXjpp89rtpWHSvI=', 'Content-Length': '0'} 2023-07-29T00:50:03.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d01a372991e2c2e-0064c4623b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-07-29T00:50:03.572 DEBUG:boto:path=/hiiqos-61/obj44 2023-07-29T00:50:03.573 DEBUG:boto:auth_path=/hiiqos-61/obj44 2023-07-29T00:50:03.573 DEBUG:boto:Method: GET 2023-07-29T00:50:03.574 DEBUG:boto:Path: /hiiqos-61/obj44 2023-07-29T00:50:03.574 DEBUG:boto:Data: 2023-07-29T00:50:03.574 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.575 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:03.575 DEBUG:boto:Port: 8001 2023-07-29T00:50:03.575 DEBUG:boto:Params: {} 2023-07-29T00:50:03.576 DEBUG:boto:Token: None 2023-07-29T00:50:03.576 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj44 2023-07-29T00:50:03.577 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DHajN5i3NcApU/1uqeOJBdhLF7w= 2023-07-29T00:50:03.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DHajN5i3NcApU/1uqeOJBdhLF7w=', 'Content-Length': '0'} 2023-07-29T00:50:03.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075e05052dbf37225-0064c4623b-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.579 DEBUG:boto:path=/hiiqos-61/obj44 2023-07-29T00:50:03.579 DEBUG:boto:auth_path=/hiiqos-61/obj44 2023-07-29T00:50:03.580 DEBUG:boto:Method: GET 2023-07-29T00:50:03.580 DEBUG:boto:Path: /hiiqos-61/obj44 2023-07-29T00:50:03.581 DEBUG:boto:Data: 2023-07-29T00:50:03.581 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.581 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:03.582 DEBUG:boto:Port: 8000 2023-07-29T00:50:03.582 DEBUG:boto:Params: {} 2023-07-29T00:50:03.582 DEBUG:boto:Token: None 2023-07-29T00:50:03.583 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj44 2023-07-29T00:50:03.583 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DHajN5i3NcApU/1uqeOJBdhLF7w= 2023-07-29T00:50:03.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DHajN5i3NcApU/1uqeOJBdhLF7w=', 'Content-Length': '0'} 2023-07-29T00:50:03.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006056cd45c42391a6-0064c4623b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.628 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:03.629 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:03.629 DEBUG:boto:path=/hiiqos-61/?prefix=obj44 2023-07-29T00:50:03.629 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj44 2023-07-29T00:50:03.630 DEBUG:boto:Method: GET 2023-07-29T00:50:03.630 DEBUG:boto:Path: /hiiqos-61/?prefix=obj44 2023-07-29T00:50:03.631 DEBUG:boto:Data: 2023-07-29T00:50:03.631 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.631 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:03.632 DEBUG:boto:Port: 8001 2023-07-29T00:50:03.632 DEBUG:boto:Params: {} 2023-07-29T00:50:03.632 DEBUG:boto:Token: None 2023-07-29T00:50:03.633 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/ 2023-07-29T00:50:03.633 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts= 2023-07-29T00:50:03.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts=', 'Content-Length': '0'} 2023-07-29T00:50:03.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000700e94cfda62017-0064c4623b-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.637 DEBUG:boto:b'hiiqos-61obj441000falseobj442023-07-29T00:45:01.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:03.637 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:03.638 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:03.638 DEBUG:boto:path=/hiiqos-61/?prefix=obj44 2023-07-29T00:50:03.638 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj44 2023-07-29T00:50:03.639 DEBUG:boto:Method: GET 2023-07-29T00:50:03.639 DEBUG:boto:Path: /hiiqos-61/?prefix=obj44 2023-07-29T00:50:03.639 DEBUG:boto:Data: 2023-07-29T00:50:03.640 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.640 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:03.641 DEBUG:boto:Port: 8000 2023-07-29T00:50:03.641 DEBUG:boto:Params: {} 2023-07-29T00:50:03.641 DEBUG:boto:Token: None 2023-07-29T00:50:03.642 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/ 2023-07-29T00:50:03.642 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts= 2023-07-29T00:50:03.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts=', 'Content-Length': '0'} 2023-07-29T00:50:03.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db0af2d46b706018-0064c4623b-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.647 DEBUG:boto:b'hiiqos-61obj441000falseobj442023-07-29T00:45:01.140Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:03.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-07-29T00:50:03.648 DEBUG:boto:path=/hiiqos-61/obj46 2023-07-29T00:50:03.648 DEBUG:boto:auth_path=/hiiqos-61/obj46 2023-07-29T00:50:03.649 DEBUG:boto:path=/hiiqos-61/obj46?versionId=null 2023-07-29T00:50:03.649 DEBUG:boto:auth_path=/hiiqos-61/obj46?versionId=null 2023-07-29T00:50:03.649 DEBUG:boto:Method: GET 2023-07-29T00:50:03.650 DEBUG:boto:Path: /hiiqos-61/obj46?versionId=null 2023-07-29T00:50:03.650 DEBUG:boto:Data: 2023-07-29T00:50:03.650 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.651 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:03.651 DEBUG:boto:Port: 8001 2023-07-29T00:50:03.651 DEBUG:boto:Params: {} 2023-07-29T00:50:03.652 DEBUG:boto:Token: None 2023-07-29T00:50:03.652 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj46?versionId=null 2023-07-29T00:50:03.653 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PdTGJZ0vouELyt/XKvhzzb0437Q= 2023-07-29T00:50:03.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PdTGJZ0vouELyt/XKvhzzb0437Q=', 'Content-Length': '0'} 2023-07-29T00:50:03.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000154b127d9229e844-0064c4623b-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.696 DEBUG:boto:path=/hiiqos-61/obj46 2023-07-29T00:50:03.697 DEBUG:boto:auth_path=/hiiqos-61/obj46 2023-07-29T00:50:03.697 DEBUG:boto:path=/hiiqos-61/obj46?versionId=null 2023-07-29T00:50:03.697 DEBUG:boto:auth_path=/hiiqos-61/obj46?versionId=null 2023-07-29T00:50:03.698 DEBUG:boto:Method: GET 2023-07-29T00:50:03.698 DEBUG:boto:Path: /hiiqos-61/obj46?versionId=null 2023-07-29T00:50:03.698 DEBUG:boto:Data: 2023-07-29T00:50:03.699 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.699 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:03.700 DEBUG:boto:Port: 8000 2023-07-29T00:50:03.700 DEBUG:boto:Params: {} 2023-07-29T00:50:03.700 DEBUG:boto:Token: None 2023-07-29T00:50:03.701 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj46?versionId=null 2023-07-29T00:50:03.701 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PdTGJZ0vouELyt/XKvhzzb0437Q= 2023-07-29T00:50:03.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PdTGJZ0vouELyt/XKvhzzb0437Q=', 'Content-Length': '0'} 2023-07-29T00:50:03.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e00cc29ccb4a0eb0-0064c4623b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.704 DEBUG:boto:path=/hiiqos-61/obj46 2023-07-29T00:50:03.704 DEBUG:boto:auth_path=/hiiqos-61/obj46 2023-07-29T00:50:03.705 DEBUG:boto:path=/hiiqos-61/obj46?versionId=null 2023-07-29T00:50:03.705 DEBUG:boto:auth_path=/hiiqos-61/obj46?versionId=null 2023-07-29T00:50:03.705 DEBUG:boto:Method: HEAD 2023-07-29T00:50:03.706 DEBUG:boto:Path: /hiiqos-61/obj46?versionId=null 2023-07-29T00:50:03.706 DEBUG:boto:Data: 2023-07-29T00:50:03.706 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.707 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:03.707 DEBUG:boto:Port: 8001 2023-07-29T00:50:03.707 DEBUG:boto:Params: {} 2023-07-29T00:50:03.708 DEBUG:boto:Token: None 2023-07-29T00:50:03.708 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj46?versionId=null 2023-07-29T00:50:03.709 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OV/NnL5vWlEYpfaWS2wj72GMQQ4= 2023-07-29T00:50:03.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OV/NnL5vWlEYpfaWS2wj72GMQQ4=', 'Content-Length': '0'} 2023-07-29T00:50:03.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024c727990ad5663f-0064c4623b-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.711 DEBUG:boto:path=/hiiqos-61/obj46 2023-07-29T00:50:03.711 DEBUG:boto:auth_path=/hiiqos-61/obj46 2023-07-29T00:50:03.712 DEBUG:boto:path=/hiiqos-61/obj46?versionId=null 2023-07-29T00:50:03.712 DEBUG:boto:auth_path=/hiiqos-61/obj46?versionId=null 2023-07-29T00:50:03.712 DEBUG:boto:Method: HEAD 2023-07-29T00:50:03.713 DEBUG:boto:Path: /hiiqos-61/obj46?versionId=null 2023-07-29T00:50:03.713 DEBUG:boto:Data: 2023-07-29T00:50:03.713 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.714 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:03.714 DEBUG:boto:Port: 8000 2023-07-29T00:50:03.714 DEBUG:boto:Params: {} 2023-07-29T00:50:03.715 DEBUG:boto:Token: None 2023-07-29T00:50:03.715 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj46?versionId=null 2023-07-29T00:50:03.716 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OV/NnL5vWlEYpfaWS2wj72GMQQ4= 2023-07-29T00:50:03.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OV/NnL5vWlEYpfaWS2wj72GMQQ4=', 'Content-Length': '0'} 2023-07-29T00:50:03.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078b86387c03d2162-0064c4623b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-07-29T00:50:03.719 DEBUG:boto:path=/hiiqos-61/obj46 2023-07-29T00:50:03.719 DEBUG:boto:auth_path=/hiiqos-61/obj46 2023-07-29T00:50:03.719 DEBUG:boto:path=/hiiqos-61/obj46?versionId=null 2023-07-29T00:50:03.720 DEBUG:boto:auth_path=/hiiqos-61/obj46?versionId=null 2023-07-29T00:50:03.720 DEBUG:boto:Method: GET 2023-07-29T00:50:03.721 DEBUG:boto:Path: /hiiqos-61/obj46?versionId=null 2023-07-29T00:50:03.721 DEBUG:boto:Data: 2023-07-29T00:50:03.722 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.722 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:03.723 DEBUG:boto:Port: 8001 2023-07-29T00:50:03.723 DEBUG:boto:Params: {} 2023-07-29T00:50:03.723 DEBUG:boto:Token: None 2023-07-29T00:50:03.724 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj46?versionId=null 2023-07-29T00:50:03.724 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PdTGJZ0vouELyt/XKvhzzb0437Q= 2023-07-29T00:50:03.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PdTGJZ0vouELyt/XKvhzzb0437Q=', 'Content-Length': '0'} 2023-07-29T00:50:03.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000057754352495bd93-0064c4623b-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.727 DEBUG:boto:path=/hiiqos-61/obj46 2023-07-29T00:50:03.727 DEBUG:boto:auth_path=/hiiqos-61/obj46 2023-07-29T00:50:03.728 DEBUG:boto:path=/hiiqos-61/obj46?versionId=null 2023-07-29T00:50:03.728 DEBUG:boto:auth_path=/hiiqos-61/obj46?versionId=null 2023-07-29T00:50:03.729 DEBUG:boto:Method: GET 2023-07-29T00:50:03.729 DEBUG:boto:Path: /hiiqos-61/obj46?versionId=null 2023-07-29T00:50:03.729 DEBUG:boto:Data: 2023-07-29T00:50:03.730 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.730 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:03.731 DEBUG:boto:Port: 8000 2023-07-29T00:50:03.731 DEBUG:boto:Params: {} 2023-07-29T00:50:03.731 DEBUG:boto:Token: None 2023-07-29T00:50:03.732 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj46?versionId=null 2023-07-29T00:50:03.732 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PdTGJZ0vouELyt/XKvhzzb0437Q= 2023-07-29T00:50:03.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PdTGJZ0vouELyt/XKvhzzb0437Q=', 'Content-Length': '0'} 2023-07-29T00:50:03.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000512e4f21a9e25e0c-0064c4623b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.735 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:03.736 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:03.736 DEBUG:boto:path=/hiiqos-61/?prefix=obj46 2023-07-29T00:50:03.736 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj46 2023-07-29T00:50:03.737 DEBUG:boto:Method: GET 2023-07-29T00:50:03.737 DEBUG:boto:Path: /hiiqos-61/?prefix=obj46 2023-07-29T00:50:03.738 DEBUG:boto:Data: 2023-07-29T00:50:03.738 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.738 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:03.739 DEBUG:boto:Port: 8001 2023-07-29T00:50:03.739 DEBUG:boto:Params: {} 2023-07-29T00:50:03.739 DEBUG:boto:Token: None 2023-07-29T00:50:03.740 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/ 2023-07-29T00:50:03.740 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts= 2023-07-29T00:50:03.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts=', 'Content-Length': '0'} 2023-07-29T00:50:03.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b94f18cc0a905a3c-0064c4623b-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.744 DEBUG:boto:b'hiiqos-61obj461000falseobj462023-07-29T00:45:02.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:03.744 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:03.745 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:03.745 DEBUG:boto:path=/hiiqos-61/?prefix=obj46 2023-07-29T00:50:03.745 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj46 2023-07-29T00:50:03.746 DEBUG:boto:Method: GET 2023-07-29T00:50:03.746 DEBUG:boto:Path: /hiiqos-61/?prefix=obj46 2023-07-29T00:50:03.747 DEBUG:boto:Data: 2023-07-29T00:50:03.747 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.747 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:03.748 DEBUG:boto:Port: 8000 2023-07-29T00:50:03.748 DEBUG:boto:Params: {} 2023-07-29T00:50:03.748 DEBUG:boto:Token: None 2023-07-29T00:50:03.749 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/ 2023-07-29T00:50:03.749 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts= 2023-07-29T00:50:03.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts=', 'Content-Length': '0'} 2023-07-29T00:50:03.753 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008573f0026756bb01-0064c4623b-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.753 DEBUG:boto:b'hiiqos-61obj461000falseobj462023-07-29T00:45:02.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:03.754 DEBUG:boto:path=/hiiqos-61/obj46 2023-07-29T00:50:03.754 DEBUG:boto:auth_path=/hiiqos-61/obj46 2023-07-29T00:50:03.755 DEBUG:boto:Method: HEAD 2023-07-29T00:50:03.755 DEBUG:boto:Path: /hiiqos-61/obj46 2023-07-29T00:50:03.756 DEBUG:boto:Data: 2023-07-29T00:50:03.756 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.756 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:03.757 DEBUG:boto:Port: 8001 2023-07-29T00:50:03.757 DEBUG:boto:Params: {} 2023-07-29T00:50:03.757 DEBUG:boto:Token: None 2023-07-29T00:50:03.758 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj46 2023-07-29T00:50:03.758 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J1Aaaw+5bZX+x0UuhqdwfgV7wiY= 2023-07-29T00:50:03.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J1Aaaw+5bZX+x0UuhqdwfgV7wiY=', 'Content-Length': '0'} 2023-07-29T00:50:03.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2e2a1db4e5932b1-0064c4623b-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.761 DEBUG:boto:path=/hiiqos-61/obj46 2023-07-29T00:50:03.761 DEBUG:boto:auth_path=/hiiqos-61/obj46 2023-07-29T00:50:03.762 DEBUG:boto:Method: HEAD 2023-07-29T00:50:03.762 DEBUG:boto:Path: /hiiqos-61/obj46 2023-07-29T00:50:03.763 DEBUG:boto:Data: 2023-07-29T00:50:03.763 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.763 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:03.764 DEBUG:boto:Port: 8000 2023-07-29T00:50:03.764 DEBUG:boto:Params: {} 2023-07-29T00:50:03.764 DEBUG:boto:Token: None 2023-07-29T00:50:03.765 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj46 2023-07-29T00:50:03.765 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J1Aaaw+5bZX+x0UuhqdwfgV7wiY= 2023-07-29T00:50:03.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J1Aaaw+5bZX+x0UuhqdwfgV7wiY=', 'Content-Length': '0'} 2023-07-29T00:50:03.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d04b15eb21eefd12-0064c4623b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-07-29T00:50:03.768 DEBUG:boto:path=/hiiqos-61/obj46 2023-07-29T00:50:03.769 DEBUG:boto:auth_path=/hiiqos-61/obj46 2023-07-29T00:50:03.769 DEBUG:boto:Method: GET 2023-07-29T00:50:03.769 DEBUG:boto:Path: /hiiqos-61/obj46 2023-07-29T00:50:03.770 DEBUG:boto:Data: 2023-07-29T00:50:03.770 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.770 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:03.771 DEBUG:boto:Port: 8001 2023-07-29T00:50:03.771 DEBUG:boto:Params: {} 2023-07-29T00:50:03.772 DEBUG:boto:Token: None 2023-07-29T00:50:03.772 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj46 2023-07-29T00:50:03.773 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7fJpxIktH4v3BYapVtF+925TvbM= 2023-07-29T00:50:03.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7fJpxIktH4v3BYapVtF+925TvbM=', 'Content-Length': '0'} 2023-07-29T00:50:03.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ef06482b4ede151-0064c4623b-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.775 DEBUG:boto:path=/hiiqos-61/obj46 2023-07-29T00:50:03.776 DEBUG:boto:auth_path=/hiiqos-61/obj46 2023-07-29T00:50:03.776 DEBUG:boto:Method: GET 2023-07-29T00:50:03.777 DEBUG:boto:Path: /hiiqos-61/obj46 2023-07-29T00:50:03.777 DEBUG:boto:Data: 2023-07-29T00:50:03.777 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.778 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:03.778 DEBUG:boto:Port: 8000 2023-07-29T00:50:03.778 DEBUG:boto:Params: {} 2023-07-29T00:50:03.779 DEBUG:boto:Token: None 2023-07-29T00:50:03.779 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj46 2023-07-29T00:50:03.780 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7fJpxIktH4v3BYapVtF+925TvbM= 2023-07-29T00:50:03.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7fJpxIktH4v3BYapVtF+925TvbM=', 'Content-Length': '0'} 2023-07-29T00:50:03.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd9c1f62d3341f02-0064c4623b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.782 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:03.783 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:03.783 DEBUG:boto:path=/hiiqos-61/?prefix=obj46 2023-07-29T00:50:03.784 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj46 2023-07-29T00:50:03.785 DEBUG:boto:Method: GET 2023-07-29T00:50:03.785 DEBUG:boto:Path: /hiiqos-61/?prefix=obj46 2023-07-29T00:50:03.785 DEBUG:boto:Data: 2023-07-29T00:50:03.786 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.786 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:03.787 DEBUG:boto:Port: 8001 2023-07-29T00:50:03.787 DEBUG:boto:Params: {} 2023-07-29T00:50:03.788 DEBUG:boto:Token: None 2023-07-29T00:50:03.788 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/ 2023-07-29T00:50:03.789 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts= 2023-07-29T00:50:03.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts=', 'Content-Length': '0'} 2023-07-29T00:50:03.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000749daf65665763b3-0064c4623b-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.792 DEBUG:boto:b'hiiqos-61obj461000falseobj462023-07-29T00:45:02.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:03.793 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:03.793 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:03.794 DEBUG:boto:path=/hiiqos-61/?prefix=obj46 2023-07-29T00:50:03.794 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj46 2023-07-29T00:50:03.794 DEBUG:boto:Method: GET 2023-07-29T00:50:03.795 DEBUG:boto:Path: /hiiqos-61/?prefix=obj46 2023-07-29T00:50:03.795 DEBUG:boto:Data: 2023-07-29T00:50:03.796 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.796 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:03.796 DEBUG:boto:Port: 8000 2023-07-29T00:50:03.797 DEBUG:boto:Params: {} 2023-07-29T00:50:03.797 DEBUG:boto:Token: None 2023-07-29T00:50:03.798 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/ 2023-07-29T00:50:03.798 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts= 2023-07-29T00:50:03.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts=', 'Content-Length': '0'} 2023-07-29T00:50:03.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006eaf58a5390dc878-0064c4623b-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.803 DEBUG:boto:b'hiiqos-61obj461000falseobj462023-07-29T00:45:02.003Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:03.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-07-29T00:50:03.804 DEBUG:boto:path=/hiiqos-61/obj51 2023-07-29T00:50:03.804 DEBUG:boto:auth_path=/hiiqos-61/obj51 2023-07-29T00:50:03.804 DEBUG:boto:path=/hiiqos-61/obj51?versionId=null 2023-07-29T00:50:03.805 DEBUG:boto:auth_path=/hiiqos-61/obj51?versionId=null 2023-07-29T00:50:03.805 DEBUG:boto:Method: GET 2023-07-29T00:50:03.806 DEBUG:boto:Path: /hiiqos-61/obj51?versionId=null 2023-07-29T00:50:03.806 DEBUG:boto:Data: 2023-07-29T00:50:03.806 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.807 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:03.807 DEBUG:boto:Port: 8001 2023-07-29T00:50:03.807 DEBUG:boto:Params: {} 2023-07-29T00:50:03.808 DEBUG:boto:Token: None 2023-07-29T00:50:03.808 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj51?versionId=null 2023-07-29T00:50:03.809 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OTqWQBazHWEni30GSraAg+8BbZE= 2023-07-29T00:50:03.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OTqWQBazHWEni30GSraAg+8BbZE=', 'Content-Length': '0'} 2023-07-29T00:50:03.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bea9b50574b2f4d9-0064c4623b-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.812 DEBUG:boto:path=/hiiqos-61/obj51 2023-07-29T00:50:03.812 DEBUG:boto:auth_path=/hiiqos-61/obj51 2023-07-29T00:50:03.812 DEBUG:boto:path=/hiiqos-61/obj51?versionId=null 2023-07-29T00:50:03.813 DEBUG:boto:auth_path=/hiiqos-61/obj51?versionId=null 2023-07-29T00:50:03.813 DEBUG:boto:Method: GET 2023-07-29T00:50:03.813 DEBUG:boto:Path: /hiiqos-61/obj51?versionId=null 2023-07-29T00:50:03.814 DEBUG:boto:Data: 2023-07-29T00:50:03.814 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.815 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:03.815 DEBUG:boto:Port: 8000 2023-07-29T00:50:03.815 DEBUG:boto:Params: {} 2023-07-29T00:50:03.816 DEBUG:boto:Token: None 2023-07-29T00:50:03.816 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj51?versionId=null 2023-07-29T00:50:03.817 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OTqWQBazHWEni30GSraAg+8BbZE= 2023-07-29T00:50:03.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OTqWQBazHWEni30GSraAg+8BbZE=', 'Content-Length': '0'} 2023-07-29T00:50:03.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033901c5f262d88a5-0064c4623b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.860 DEBUG:boto:path=/hiiqos-61/obj51 2023-07-29T00:50:03.861 DEBUG:boto:auth_path=/hiiqos-61/obj51 2023-07-29T00:50:03.861 DEBUG:boto:path=/hiiqos-61/obj51?versionId=null 2023-07-29T00:50:03.862 DEBUG:boto:auth_path=/hiiqos-61/obj51?versionId=null 2023-07-29T00:50:03.862 DEBUG:boto:Method: HEAD 2023-07-29T00:50:03.862 DEBUG:boto:Path: /hiiqos-61/obj51?versionId=null 2023-07-29T00:50:03.863 DEBUG:boto:Data: 2023-07-29T00:50:03.863 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.863 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:03.864 DEBUG:boto:Port: 8001 2023-07-29T00:50:03.864 DEBUG:boto:Params: {} 2023-07-29T00:50:03.865 DEBUG:boto:Token: None 2023-07-29T00:50:03.865 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj51?versionId=null 2023-07-29T00:50:03.866 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ecUQygVLjosQJGrEvxF/LswDbj4= 2023-07-29T00:50:03.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ecUQygVLjosQJGrEvxF/LswDbj4=', 'Content-Length': '0'} 2023-07-29T00:50:03.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d16427c3474ff048-0064c4623b-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.868 DEBUG:boto:path=/hiiqos-61/obj51 2023-07-29T00:50:03.869 DEBUG:boto:auth_path=/hiiqos-61/obj51 2023-07-29T00:50:03.869 DEBUG:boto:path=/hiiqos-61/obj51?versionId=null 2023-07-29T00:50:03.869 DEBUG:boto:auth_path=/hiiqos-61/obj51?versionId=null 2023-07-29T00:50:03.870 DEBUG:boto:Method: HEAD 2023-07-29T00:50:03.870 DEBUG:boto:Path: /hiiqos-61/obj51?versionId=null 2023-07-29T00:50:03.871 DEBUG:boto:Data: 2023-07-29T00:50:03.871 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.871 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:03.871 DEBUG:boto:Port: 8000 2023-07-29T00:50:03.872 DEBUG:boto:Params: {} 2023-07-29T00:50:03.872 DEBUG:boto:Token: None 2023-07-29T00:50:03.873 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj51?versionId=null 2023-07-29T00:50:03.873 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ecUQygVLjosQJGrEvxF/LswDbj4= 2023-07-29T00:50:03.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ecUQygVLjosQJGrEvxF/LswDbj4=', 'Content-Length': '0'} 2023-07-29T00:50:03.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9a621782d2a288b-0064c4623b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-07-29T00:50:03.876 DEBUG:boto:path=/hiiqos-61/obj51 2023-07-29T00:50:03.876 DEBUG:boto:auth_path=/hiiqos-61/obj51 2023-07-29T00:50:03.876 DEBUG:boto:path=/hiiqos-61/obj51?versionId=null 2023-07-29T00:50:03.877 DEBUG:boto:auth_path=/hiiqos-61/obj51?versionId=null 2023-07-29T00:50:03.877 DEBUG:boto:Method: GET 2023-07-29T00:50:03.878 DEBUG:boto:Path: /hiiqos-61/obj51?versionId=null 2023-07-29T00:50:03.878 DEBUG:boto:Data: 2023-07-29T00:50:03.878 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.879 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:03.879 DEBUG:boto:Port: 8001 2023-07-29T00:50:03.879 DEBUG:boto:Params: {} 2023-07-29T00:50:03.880 DEBUG:boto:Token: None 2023-07-29T00:50:03.880 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj51?versionId=null 2023-07-29T00:50:03.881 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OTqWQBazHWEni30GSraAg+8BbZE= 2023-07-29T00:50:03.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OTqWQBazHWEni30GSraAg+8BbZE=', 'Content-Length': '0'} 2023-07-29T00:50:03.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a338720490abf769-0064c4623b-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.883 DEBUG:boto:path=/hiiqos-61/obj51 2023-07-29T00:50:03.884 DEBUG:boto:auth_path=/hiiqos-61/obj51 2023-07-29T00:50:03.884 DEBUG:boto:path=/hiiqos-61/obj51?versionId=null 2023-07-29T00:50:03.884 DEBUG:boto:auth_path=/hiiqos-61/obj51?versionId=null 2023-07-29T00:50:03.885 DEBUG:boto:Method: GET 2023-07-29T00:50:03.885 DEBUG:boto:Path: /hiiqos-61/obj51?versionId=null 2023-07-29T00:50:03.885 DEBUG:boto:Data: 2023-07-29T00:50:03.886 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.886 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:03.886 DEBUG:boto:Port: 8000 2023-07-29T00:50:03.887 DEBUG:boto:Params: {} 2023-07-29T00:50:03.887 DEBUG:boto:Token: None 2023-07-29T00:50:03.888 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj51?versionId=null 2023-07-29T00:50:03.888 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OTqWQBazHWEni30GSraAg+8BbZE= 2023-07-29T00:50:03.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OTqWQBazHWEni30GSraAg+8BbZE=', 'Content-Length': '0'} 2023-07-29T00:50:03.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d89d7f9335d2c9d1-0064c4623b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.891 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:03.891 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:03.891 DEBUG:boto:path=/hiiqos-61/?prefix=obj51 2023-07-29T00:50:03.892 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj51 2023-07-29T00:50:03.892 DEBUG:boto:Method: GET 2023-07-29T00:50:03.893 DEBUG:boto:Path: /hiiqos-61/?prefix=obj51 2023-07-29T00:50:03.893 DEBUG:boto:Data: 2023-07-29T00:50:03.893 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.894 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:03.894 DEBUG:boto:Port: 8001 2023-07-29T00:50:03.894 DEBUG:boto:Params: {} 2023-07-29T00:50:03.895 DEBUG:boto:Token: None 2023-07-29T00:50:03.895 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/ 2023-07-29T00:50:03.896 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts= 2023-07-29T00:50:03.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts=', 'Content-Length': '0'} 2023-07-29T00:50:03.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1096be99ef5047d-0064c4623b-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.899 DEBUG:boto:b'hiiqos-61obj511000falseobj512023-07-29T00:45:01.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:03.900 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:03.900 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:03.900 DEBUG:boto:path=/hiiqos-61/?prefix=obj51 2023-07-29T00:50:03.901 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj51 2023-07-29T00:50:03.901 DEBUG:boto:Method: GET 2023-07-29T00:50:03.901 DEBUG:boto:Path: /hiiqos-61/?prefix=obj51 2023-07-29T00:50:03.902 DEBUG:boto:Data: 2023-07-29T00:50:03.902 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.902 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:03.903 DEBUG:boto:Port: 8000 2023-07-29T00:50:03.903 DEBUG:boto:Params: {} 2023-07-29T00:50:03.904 DEBUG:boto:Token: None 2023-07-29T00:50:03.904 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/ 2023-07-29T00:50:03.904 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts= 2023-07-29T00:50:03.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts=', 'Content-Length': '0'} 2023-07-29T00:50:03.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e62bd772f7ef9cd-0064c4623b-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.909 DEBUG:boto:b'hiiqos-61obj511000falseobj512023-07-29T00:45:01.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:03.909 DEBUG:boto:path=/hiiqos-61/obj51 2023-07-29T00:50:03.910 DEBUG:boto:auth_path=/hiiqos-61/obj51 2023-07-29T00:50:03.910 DEBUG:boto:Method: HEAD 2023-07-29T00:50:03.910 DEBUG:boto:Path: /hiiqos-61/obj51 2023-07-29T00:50:03.911 DEBUG:boto:Data: 2023-07-29T00:50:03.911 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.911 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:03.912 DEBUG:boto:Port: 8001 2023-07-29T00:50:03.912 DEBUG:boto:Params: {} 2023-07-29T00:50:03.913 DEBUG:boto:Token: None 2023-07-29T00:50:03.913 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj51 2023-07-29T00:50:03.913 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cyGQnp9/sxH6XxYD6r64JS0eACU= 2023-07-29T00:50:03.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cyGQnp9/sxH6XxYD6r64JS0eACU=', 'Content-Length': '0'} 2023-07-29T00:50:03.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e008356b75b8e9dc-0064c4623b-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.916 DEBUG:boto:path=/hiiqos-61/obj51 2023-07-29T00:50:03.917 DEBUG:boto:auth_path=/hiiqos-61/obj51 2023-07-29T00:50:03.917 DEBUG:boto:Method: HEAD 2023-07-29T00:50:03.917 DEBUG:boto:Path: /hiiqos-61/obj51 2023-07-29T00:50:03.918 DEBUG:boto:Data: 2023-07-29T00:50:03.918 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.918 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:03.919 DEBUG:boto:Port: 8000 2023-07-29T00:50:03.919 DEBUG:boto:Params: {} 2023-07-29T00:50:03.919 DEBUG:boto:Token: None 2023-07-29T00:50:03.920 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj51 2023-07-29T00:50:03.920 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cyGQnp9/sxH6XxYD6r64JS0eACU= 2023-07-29T00:50:03.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cyGQnp9/sxH6XxYD6r64JS0eACU=', 'Content-Length': '0'} 2023-07-29T00:50:03.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000908875409b7445f0-0064c4623b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-07-29T00:50:03.923 DEBUG:boto:path=/hiiqos-61/obj51 2023-07-29T00:50:03.923 DEBUG:boto:auth_path=/hiiqos-61/obj51 2023-07-29T00:50:03.924 DEBUG:boto:Method: GET 2023-07-29T00:50:03.924 DEBUG:boto:Path: /hiiqos-61/obj51 2023-07-29T00:50:03.924 DEBUG:boto:Data: 2023-07-29T00:50:03.925 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.925 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:03.925 DEBUG:boto:Port: 8001 2023-07-29T00:50:03.926 DEBUG:boto:Params: {} 2023-07-29T00:50:03.926 DEBUG:boto:Token: None 2023-07-29T00:50:03.927 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj51 2023-07-29T00:50:03.927 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zG5iG9yEL2VE94w3RMZeErOLypM= 2023-07-29T00:50:03.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zG5iG9yEL2VE94w3RMZeErOLypM=', 'Content-Length': '0'} 2023-07-29T00:50:03.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012a1024946b69a81-0064c4623b-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.930 DEBUG:boto:path=/hiiqos-61/obj51 2023-07-29T00:50:03.930 DEBUG:boto:auth_path=/hiiqos-61/obj51 2023-07-29T00:50:03.931 DEBUG:boto:Method: GET 2023-07-29T00:50:03.931 DEBUG:boto:Path: /hiiqos-61/obj51 2023-07-29T00:50:03.931 DEBUG:boto:Data: 2023-07-29T00:50:03.932 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.932 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:03.932 DEBUG:boto:Port: 8000 2023-07-29T00:50:03.933 DEBUG:boto:Params: {} 2023-07-29T00:50:03.933 DEBUG:boto:Token: None 2023-07-29T00:50:03.934 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/obj51 2023-07-29T00:50:03.934 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zG5iG9yEL2VE94w3RMZeErOLypM= 2023-07-29T00:50:03.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zG5iG9yEL2VE94w3RMZeErOLypM=', 'Content-Length': '0'} 2023-07-29T00:50:03.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004356e4726e05aa9e-0064c4623b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.937 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:03.937 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:03.937 DEBUG:boto:path=/hiiqos-61/?prefix=obj51 2023-07-29T00:50:03.938 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj51 2023-07-29T00:50:03.938 DEBUG:boto:Method: GET 2023-07-29T00:50:03.939 DEBUG:boto:Path: /hiiqos-61/?prefix=obj51 2023-07-29T00:50:03.939 DEBUG:boto:Data: 2023-07-29T00:50:03.939 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.940 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:03.940 DEBUG:boto:Port: 8001 2023-07-29T00:50:03.940 DEBUG:boto:Params: {} 2023-07-29T00:50:03.941 DEBUG:boto:Token: None 2023-07-29T00:50:03.941 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/ 2023-07-29T00:50:03.942 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts= 2023-07-29T00:50:03.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts=', 'Content-Length': '0'} 2023-07-29T00:50:03.944 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6a1f42a2ac51c4f-0064c4623b-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.945 DEBUG:boto:b'hiiqos-61obj511000falseobj512023-07-29T00:45:01.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:03.945 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:03.946 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:03.946 DEBUG:boto:path=/hiiqos-61/?prefix=obj51 2023-07-29T00:50:03.946 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj51 2023-07-29T00:50:03.947 DEBUG:boto:Method: GET 2023-07-29T00:50:03.947 DEBUG:boto:Path: /hiiqos-61/?prefix=obj51 2023-07-29T00:50:03.947 DEBUG:boto:Data: 2023-07-29T00:50:03.948 DEBUG:boto:Headers: {} 2023-07-29T00:50:03.948 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:03.948 DEBUG:boto:Port: 8000 2023-07-29T00:50:03.949 DEBUG:boto:Params: {} 2023-07-29T00:50:03.949 DEBUG:boto:Token: None 2023-07-29T00:50:03.950 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:03 GMT /hiiqos-61/ 2023-07-29T00:50:03.950 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts= 2023-07-29T00:50:03.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:03 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5Oc34W2l3+TKSF+2DigvX+edqts=', 'Content-Length': '0'} 2023-07-29T00:50:03.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4cd742465d60d4a-0064c4623b-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:03.996 DEBUG:boto:b'hiiqos-61obj511000falseobj512023-07-29T00:45:01.400Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:03.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-07-29T00:50:03.997 DEBUG:boto:path=/hiiqos-61/obj55 2023-07-29T00:50:03.998 DEBUG:boto:auth_path=/hiiqos-61/obj55 2023-07-29T00:50:03.998 DEBUG:boto:path=/hiiqos-61/obj55?versionId=null 2023-07-29T00:50:03.999 DEBUG:boto:auth_path=/hiiqos-61/obj55?versionId=null 2023-07-29T00:50:03.999 DEBUG:boto:Method: GET 2023-07-29T00:50:03.999 DEBUG:boto:Path: /hiiqos-61/obj55?versionId=null 2023-07-29T00:50:04.000 DEBUG:boto:Data: 2023-07-29T00:50:04.000 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.000 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:04.001 DEBUG:boto:Port: 8001 2023-07-29T00:50:04.001 DEBUG:boto:Params: {} 2023-07-29T00:50:04.002 DEBUG:boto:Token: None 2023-07-29T00:50:04.002 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj55?versionId=null 2023-07-29T00:50:04.003 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KB+sQ3GhmAnHYnLwbZvnk9nHl3g= 2023-07-29T00:50:04.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KB+sQ3GhmAnHYnLwbZvnk9nHl3g=', 'Content-Length': '0'} 2023-07-29T00:50:04.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2a8cf3546d7ae89-0064c4623b-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.005 DEBUG:boto:path=/hiiqos-61/obj55 2023-07-29T00:50:04.006 DEBUG:boto:auth_path=/hiiqos-61/obj55 2023-07-29T00:50:04.006 DEBUG:boto:path=/hiiqos-61/obj55?versionId=null 2023-07-29T00:50:04.007 DEBUG:boto:auth_path=/hiiqos-61/obj55?versionId=null 2023-07-29T00:50:04.007 DEBUG:boto:Method: GET 2023-07-29T00:50:04.007 DEBUG:boto:Path: /hiiqos-61/obj55?versionId=null 2023-07-29T00:50:04.008 DEBUG:boto:Data: 2023-07-29T00:50:04.008 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.008 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:04.009 DEBUG:boto:Port: 8000 2023-07-29T00:50:04.009 DEBUG:boto:Params: {} 2023-07-29T00:50:04.010 DEBUG:boto:Token: None 2023-07-29T00:50:04.010 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj55?versionId=null 2023-07-29T00:50:04.010 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KB+sQ3GhmAnHYnLwbZvnk9nHl3g= 2023-07-29T00:50:04.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KB+sQ3GhmAnHYnLwbZvnk9nHl3g=', 'Content-Length': '0'} 2023-07-29T00:50:04.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003681e8bc3ba67732-0064c4623b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.013 DEBUG:boto:path=/hiiqos-61/obj55 2023-07-29T00:50:04.013 DEBUG:boto:auth_path=/hiiqos-61/obj55 2023-07-29T00:50:04.014 DEBUG:boto:path=/hiiqos-61/obj55?versionId=null 2023-07-29T00:50:04.014 DEBUG:boto:auth_path=/hiiqos-61/obj55?versionId=null 2023-07-29T00:50:04.014 DEBUG:boto:Method: HEAD 2023-07-29T00:50:04.015 DEBUG:boto:Path: /hiiqos-61/obj55?versionId=null 2023-07-29T00:50:04.015 DEBUG:boto:Data: 2023-07-29T00:50:04.015 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.016 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:04.016 DEBUG:boto:Port: 8001 2023-07-29T00:50:04.017 DEBUG:boto:Params: {} 2023-07-29T00:50:04.017 DEBUG:boto:Token: None 2023-07-29T00:50:04.017 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj55?versionId=null 2023-07-29T00:50:04.018 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:N/dS9XUhUOyiZkdnBF0gtrna3SE= 2023-07-29T00:50:04.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:N/dS9XUhUOyiZkdnBF0gtrna3SE=', 'Content-Length': '0'} 2023-07-29T00:50:04.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f92ce61500bb12cb-0064c4623b-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.020 DEBUG:boto:path=/hiiqos-61/obj55 2023-07-29T00:50:04.021 DEBUG:boto:auth_path=/hiiqos-61/obj55 2023-07-29T00:50:04.021 DEBUG:boto:path=/hiiqos-61/obj55?versionId=null 2023-07-29T00:50:04.021 DEBUG:boto:auth_path=/hiiqos-61/obj55?versionId=null 2023-07-29T00:50:04.022 DEBUG:boto:Method: HEAD 2023-07-29T00:50:04.022 DEBUG:boto:Path: /hiiqos-61/obj55?versionId=null 2023-07-29T00:50:04.022 DEBUG:boto:Data: 2023-07-29T00:50:04.023 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.023 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:04.023 DEBUG:boto:Port: 8000 2023-07-29T00:50:04.024 DEBUG:boto:Params: {} 2023-07-29T00:50:04.024 DEBUG:boto:Token: None 2023-07-29T00:50:04.025 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj55?versionId=null 2023-07-29T00:50:04.025 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:N/dS9XUhUOyiZkdnBF0gtrna3SE= 2023-07-29T00:50:04.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:N/dS9XUhUOyiZkdnBF0gtrna3SE=', 'Content-Length': '0'} 2023-07-29T00:50:04.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a73bd4dcf53b579c-0064c4623b-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-07-29T00:50:04.028 DEBUG:boto:path=/hiiqos-61/obj55 2023-07-29T00:50:04.028 DEBUG:boto:auth_path=/hiiqos-61/obj55 2023-07-29T00:50:04.029 DEBUG:boto:path=/hiiqos-61/obj55?versionId=null 2023-07-29T00:50:04.029 DEBUG:boto:auth_path=/hiiqos-61/obj55?versionId=null 2023-07-29T00:50:04.030 DEBUG:boto:Method: GET 2023-07-29T00:50:04.030 DEBUG:boto:Path: /hiiqos-61/obj55?versionId=null 2023-07-29T00:50:04.030 DEBUG:boto:Data: 2023-07-29T00:50:04.031 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.031 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:04.031 DEBUG:boto:Port: 8001 2023-07-29T00:50:04.032 DEBUG:boto:Params: {} 2023-07-29T00:50:04.032 DEBUG:boto:Token: None 2023-07-29T00:50:04.033 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj55?versionId=null 2023-07-29T00:50:04.033 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KB+sQ3GhmAnHYnLwbZvnk9nHl3g= 2023-07-29T00:50:04.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KB+sQ3GhmAnHYnLwbZvnk9nHl3g=', 'Content-Length': '0'} 2023-07-29T00:50:04.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000078613cdc8e42b44-0064c4623b-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:03 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.076 DEBUG:boto:path=/hiiqos-61/obj55 2023-07-29T00:50:04.077 DEBUG:boto:auth_path=/hiiqos-61/obj55 2023-07-29T00:50:04.077 DEBUG:boto:path=/hiiqos-61/obj55?versionId=null 2023-07-29T00:50:04.077 DEBUG:boto:auth_path=/hiiqos-61/obj55?versionId=null 2023-07-29T00:50:04.078 DEBUG:boto:Method: GET 2023-07-29T00:50:04.078 DEBUG:boto:Path: /hiiqos-61/obj55?versionId=null 2023-07-29T00:50:04.079 DEBUG:boto:Data: 2023-07-29T00:50:04.079 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.079 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:04.080 DEBUG:boto:Port: 8000 2023-07-29T00:50:04.080 DEBUG:boto:Params: {} 2023-07-29T00:50:04.080 DEBUG:boto:Token: None 2023-07-29T00:50:04.081 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj55?versionId=null 2023-07-29T00:50:04.081 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KB+sQ3GhmAnHYnLwbZvnk9nHl3g= 2023-07-29T00:50:04.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KB+sQ3GhmAnHYnLwbZvnk9nHl3g=', 'Content-Length': '0'} 2023-07-29T00:50:04.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ece4a2b97966661-0064c4623c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.084 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:04.085 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:04.085 DEBUG:boto:path=/hiiqos-61/?prefix=obj55 2023-07-29T00:50:04.085 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj55 2023-07-29T00:50:04.086 DEBUG:boto:Method: GET 2023-07-29T00:50:04.086 DEBUG:boto:Path: /hiiqos-61/?prefix=obj55 2023-07-29T00:50:04.086 DEBUG:boto:Data: 2023-07-29T00:50:04.087 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.087 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:04.088 DEBUG:boto:Port: 8001 2023-07-29T00:50:04.088 DEBUG:boto:Params: {} 2023-07-29T00:50:04.088 DEBUG:boto:Token: None 2023-07-29T00:50:04.089 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/ 2023-07-29T00:50:04.089 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g= 2023-07-29T00:50:04.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g=', 'Content-Length': '0'} 2023-07-29T00:50:04.092 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9c6e224d9390908-0064c4623c-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.093 DEBUG:boto:b'hiiqos-61obj551000falseobj552023-07-29T00:45:01.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:04.093 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:04.094 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:04.094 DEBUG:boto:path=/hiiqos-61/?prefix=obj55 2023-07-29T00:50:04.094 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj55 2023-07-29T00:50:04.095 DEBUG:boto:Method: GET 2023-07-29T00:50:04.095 DEBUG:boto:Path: /hiiqos-61/?prefix=obj55 2023-07-29T00:50:04.095 DEBUG:boto:Data: 2023-07-29T00:50:04.096 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.096 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:04.097 DEBUG:boto:Port: 8000 2023-07-29T00:50:04.097 DEBUG:boto:Params: {} 2023-07-29T00:50:04.097 DEBUG:boto:Token: None 2023-07-29T00:50:04.098 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/ 2023-07-29T00:50:04.098 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g= 2023-07-29T00:50:04.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g=', 'Content-Length': '0'} 2023-07-29T00:50:04.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a83a59527c95c318-0064c4623c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.103 DEBUG:boto:b'hiiqos-61obj551000falseobj552023-07-29T00:45:01.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:04.103 DEBUG:boto:path=/hiiqos-61/obj55 2023-07-29T00:50:04.104 DEBUG:boto:auth_path=/hiiqos-61/obj55 2023-07-29T00:50:04.104 DEBUG:boto:Method: HEAD 2023-07-29T00:50:04.105 DEBUG:boto:Path: /hiiqos-61/obj55 2023-07-29T00:50:04.105 DEBUG:boto:Data: 2023-07-29T00:50:04.105 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.106 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:04.106 DEBUG:boto:Port: 8001 2023-07-29T00:50:04.106 DEBUG:boto:Params: {} 2023-07-29T00:50:04.107 DEBUG:boto:Token: None 2023-07-29T00:50:04.107 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj55 2023-07-29T00:50:04.107 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kgjxCQ2iffjFa45Gj518YekR9/o= 2023-07-29T00:50:04.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kgjxCQ2iffjFa45Gj518YekR9/o=', 'Content-Length': '0'} 2023-07-29T00:50:04.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a31e00eb81277b14-0064c4623c-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.110 DEBUG:boto:path=/hiiqos-61/obj55 2023-07-29T00:50:04.110 DEBUG:boto:auth_path=/hiiqos-61/obj55 2023-07-29T00:50:04.111 DEBUG:boto:Method: HEAD 2023-07-29T00:50:04.111 DEBUG:boto:Path: /hiiqos-61/obj55 2023-07-29T00:50:04.111 DEBUG:boto:Data: 2023-07-29T00:50:04.112 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.112 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:04.112 DEBUG:boto:Port: 8000 2023-07-29T00:50:04.113 DEBUG:boto:Params: {} 2023-07-29T00:50:04.113 DEBUG:boto:Token: None 2023-07-29T00:50:04.114 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj55 2023-07-29T00:50:04.114 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kgjxCQ2iffjFa45Gj518YekR9/o= 2023-07-29T00:50:04.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kgjxCQ2iffjFa45Gj518YekR9/o=', 'Content-Length': '0'} 2023-07-29T00:50:04.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007231f650937e2225-0064c4623c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-07-29T00:50:04.117 DEBUG:boto:path=/hiiqos-61/obj55 2023-07-29T00:50:04.117 DEBUG:boto:auth_path=/hiiqos-61/obj55 2023-07-29T00:50:04.118 DEBUG:boto:Method: GET 2023-07-29T00:50:04.118 DEBUG:boto:Path: /hiiqos-61/obj55 2023-07-29T00:50:04.118 DEBUG:boto:Data: 2023-07-29T00:50:04.119 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.119 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:04.120 DEBUG:boto:Port: 8001 2023-07-29T00:50:04.120 DEBUG:boto:Params: {} 2023-07-29T00:50:04.120 DEBUG:boto:Token: None 2023-07-29T00:50:04.121 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj55 2023-07-29T00:50:04.121 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RFLiVhKA+DEbgCB5/QEyjLGlo+8= 2023-07-29T00:50:04.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RFLiVhKA+DEbgCB5/QEyjLGlo+8=', 'Content-Length': '0'} 2023-07-29T00:50:04.123 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c52e53a9b437463d-0064c4623c-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.124 DEBUG:boto:path=/hiiqos-61/obj55 2023-07-29T00:50:04.124 DEBUG:boto:auth_path=/hiiqos-61/obj55 2023-07-29T00:50:04.125 DEBUG:boto:Method: GET 2023-07-29T00:50:04.125 DEBUG:boto:Path: /hiiqos-61/obj55 2023-07-29T00:50:04.125 DEBUG:boto:Data: 2023-07-29T00:50:04.126 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.126 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:04.126 DEBUG:boto:Port: 8000 2023-07-29T00:50:04.127 DEBUG:boto:Params: {} 2023-07-29T00:50:04.127 DEBUG:boto:Token: None 2023-07-29T00:50:04.127 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj55 2023-07-29T00:50:04.128 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RFLiVhKA+DEbgCB5/QEyjLGlo+8= 2023-07-29T00:50:04.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RFLiVhKA+DEbgCB5/QEyjLGlo+8=', 'Content-Length': '0'} 2023-07-29T00:50:04.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b10798eb0f7537b4-0064c4623c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.172 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:04.173 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:04.173 DEBUG:boto:path=/hiiqos-61/?prefix=obj55 2023-07-29T00:50:04.173 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj55 2023-07-29T00:50:04.174 DEBUG:boto:Method: GET 2023-07-29T00:50:04.174 DEBUG:boto:Path: /hiiqos-61/?prefix=obj55 2023-07-29T00:50:04.175 DEBUG:boto:Data: 2023-07-29T00:50:04.175 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.175 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:04.176 DEBUG:boto:Port: 8001 2023-07-29T00:50:04.176 DEBUG:boto:Params: {} 2023-07-29T00:50:04.176 DEBUG:boto:Token: None 2023-07-29T00:50:04.177 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/ 2023-07-29T00:50:04.177 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g= 2023-07-29T00:50:04.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g=', 'Content-Length': '0'} 2023-07-29T00:50:04.180 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef270c359c8caf68-0064c4623c-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.181 DEBUG:boto:b'hiiqos-61obj551000falseobj552023-07-29T00:45:01.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:04.181 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:04.182 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:04.182 DEBUG:boto:path=/hiiqos-61/?prefix=obj55 2023-07-29T00:50:04.183 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj55 2023-07-29T00:50:04.183 DEBUG:boto:Method: GET 2023-07-29T00:50:04.183 DEBUG:boto:Path: /hiiqos-61/?prefix=obj55 2023-07-29T00:50:04.184 DEBUG:boto:Data: 2023-07-29T00:50:04.184 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.184 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:04.185 DEBUG:boto:Port: 8000 2023-07-29T00:50:04.185 DEBUG:boto:Params: {} 2023-07-29T00:50:04.186 DEBUG:boto:Token: None 2023-07-29T00:50:04.186 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/ 2023-07-29T00:50:04.186 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g= 2023-07-29T00:50:04.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g=', 'Content-Length': '0'} 2023-07-29T00:50:04.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000428d409c04e5cbf4-0064c4623c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.191 DEBUG:boto:b'hiiqos-61obj551000falseobj552023-07-29T00:45:01.159Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:04.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-07-29T00:50:04.192 DEBUG:boto:path=/hiiqos-61/obj57 2023-07-29T00:50:04.192 DEBUG:boto:auth_path=/hiiqos-61/obj57 2023-07-29T00:50:04.193 DEBUG:boto:path=/hiiqos-61/obj57?versionId=null 2023-07-29T00:50:04.193 DEBUG:boto:auth_path=/hiiqos-61/obj57?versionId=null 2023-07-29T00:50:04.194 DEBUG:boto:Method: GET 2023-07-29T00:50:04.194 DEBUG:boto:Path: /hiiqos-61/obj57?versionId=null 2023-07-29T00:50:04.194 DEBUG:boto:Data: 2023-07-29T00:50:04.195 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.195 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:04.195 DEBUG:boto:Port: 8001 2023-07-29T00:50:04.196 DEBUG:boto:Params: {} 2023-07-29T00:50:04.196 DEBUG:boto:Token: None 2023-07-29T00:50:04.197 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj57?versionId=null 2023-07-29T00:50:04.197 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eRUU1mnFQPGfpAVE/iF4HQuSbco= 2023-07-29T00:50:04.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eRUU1mnFQPGfpAVE/iF4HQuSbco=', 'Content-Length': '0'} 2023-07-29T00:50:04.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5de2b59226ad0a5-0064c4623c-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.200 DEBUG:boto:path=/hiiqos-61/obj57 2023-07-29T00:50:04.201 DEBUG:boto:auth_path=/hiiqos-61/obj57 2023-07-29T00:50:04.201 DEBUG:boto:path=/hiiqos-61/obj57?versionId=null 2023-07-29T00:50:04.201 DEBUG:boto:auth_path=/hiiqos-61/obj57?versionId=null 2023-07-29T00:50:04.202 DEBUG:boto:Method: GET 2023-07-29T00:50:04.202 DEBUG:boto:Path: /hiiqos-61/obj57?versionId=null 2023-07-29T00:50:04.202 DEBUG:boto:Data: 2023-07-29T00:50:04.203 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.203 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:04.204 DEBUG:boto:Port: 8000 2023-07-29T00:50:04.204 DEBUG:boto:Params: {} 2023-07-29T00:50:04.204 DEBUG:boto:Token: None 2023-07-29T00:50:04.205 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj57?versionId=null 2023-07-29T00:50:04.205 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eRUU1mnFQPGfpAVE/iF4HQuSbco= 2023-07-29T00:50:04.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eRUU1mnFQPGfpAVE/iF4HQuSbco=', 'Content-Length': '0'} 2023-07-29T00:50:04.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9bdc120aab136c9-0064c4623c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.208 DEBUG:boto:path=/hiiqos-61/obj57 2023-07-29T00:50:04.209 DEBUG:boto:auth_path=/hiiqos-61/obj57 2023-07-29T00:50:04.209 DEBUG:boto:path=/hiiqos-61/obj57?versionId=null 2023-07-29T00:50:04.209 DEBUG:boto:auth_path=/hiiqos-61/obj57?versionId=null 2023-07-29T00:50:04.210 DEBUG:boto:Method: HEAD 2023-07-29T00:50:04.210 DEBUG:boto:Path: /hiiqos-61/obj57?versionId=null 2023-07-29T00:50:04.210 DEBUG:boto:Data: 2023-07-29T00:50:04.210 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.211 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:04.211 DEBUG:boto:Port: 8001 2023-07-29T00:50:04.211 DEBUG:boto:Params: {} 2023-07-29T00:50:04.212 DEBUG:boto:Token: None 2023-07-29T00:50:04.212 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj57?versionId=null 2023-07-29T00:50:04.213 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:E9i6RRaM0ZdMqofl469rmqn2hD4= 2023-07-29T00:50:04.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:E9i6RRaM0ZdMqofl469rmqn2hD4=', 'Content-Length': '0'} 2023-07-29T00:50:04.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c397dca53fec3b3-0064c4623c-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.215 DEBUG:boto:path=/hiiqos-61/obj57 2023-07-29T00:50:04.216 DEBUG:boto:auth_path=/hiiqos-61/obj57 2023-07-29T00:50:04.216 DEBUG:boto:path=/hiiqos-61/obj57?versionId=null 2023-07-29T00:50:04.216 DEBUG:boto:auth_path=/hiiqos-61/obj57?versionId=null 2023-07-29T00:50:04.217 DEBUG:boto:Method: HEAD 2023-07-29T00:50:04.217 DEBUG:boto:Path: /hiiqos-61/obj57?versionId=null 2023-07-29T00:50:04.217 DEBUG:boto:Data: 2023-07-29T00:50:04.218 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.218 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:04.218 DEBUG:boto:Port: 8000 2023-07-29T00:50:04.219 DEBUG:boto:Params: {} 2023-07-29T00:50:04.219 DEBUG:boto:Token: None 2023-07-29T00:50:04.220 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj57?versionId=null 2023-07-29T00:50:04.220 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:E9i6RRaM0ZdMqofl469rmqn2hD4= 2023-07-29T00:50:04.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:E9i6RRaM0ZdMqofl469rmqn2hD4=', 'Content-Length': '0'} 2023-07-29T00:50:04.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000301a9c1905929edd-0064c4623c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-07-29T00:50:04.223 DEBUG:boto:path=/hiiqos-61/obj57 2023-07-29T00:50:04.224 DEBUG:boto:auth_path=/hiiqos-61/obj57 2023-07-29T00:50:04.224 DEBUG:boto:path=/hiiqos-61/obj57?versionId=null 2023-07-29T00:50:04.225 DEBUG:boto:auth_path=/hiiqos-61/obj57?versionId=null 2023-07-29T00:50:04.225 DEBUG:boto:Method: GET 2023-07-29T00:50:04.225 DEBUG:boto:Path: /hiiqos-61/obj57?versionId=null 2023-07-29T00:50:04.226 DEBUG:boto:Data: 2023-07-29T00:50:04.226 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.226 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:04.227 DEBUG:boto:Port: 8001 2023-07-29T00:50:04.227 DEBUG:boto:Params: {} 2023-07-29T00:50:04.228 DEBUG:boto:Token: None 2023-07-29T00:50:04.228 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj57?versionId=null 2023-07-29T00:50:04.228 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eRUU1mnFQPGfpAVE/iF4HQuSbco= 2023-07-29T00:50:04.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eRUU1mnFQPGfpAVE/iF4HQuSbco=', 'Content-Length': '0'} 2023-07-29T00:50:04.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067e3ec2ca9bebd2b-0064c4623c-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.231 DEBUG:boto:path=/hiiqos-61/obj57 2023-07-29T00:50:04.232 DEBUG:boto:auth_path=/hiiqos-61/obj57 2023-07-29T00:50:04.232 DEBUG:boto:path=/hiiqos-61/obj57?versionId=null 2023-07-29T00:50:04.232 DEBUG:boto:auth_path=/hiiqos-61/obj57?versionId=null 2023-07-29T00:50:04.233 DEBUG:boto:Method: GET 2023-07-29T00:50:04.233 DEBUG:boto:Path: /hiiqos-61/obj57?versionId=null 2023-07-29T00:50:04.234 DEBUG:boto:Data: 2023-07-29T00:50:04.234 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.234 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:04.235 DEBUG:boto:Port: 8000 2023-07-29T00:50:04.235 DEBUG:boto:Params: {} 2023-07-29T00:50:04.235 DEBUG:boto:Token: None 2023-07-29T00:50:04.236 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj57?versionId=null 2023-07-29T00:50:04.236 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eRUU1mnFQPGfpAVE/iF4HQuSbco= 2023-07-29T00:50:04.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eRUU1mnFQPGfpAVE/iF4HQuSbco=', 'Content-Length': '0'} 2023-07-29T00:50:04.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4c953e8c8264993-0064c4623c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.239 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:04.239 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:04.240 DEBUG:boto:path=/hiiqos-61/?prefix=obj57 2023-07-29T00:50:04.240 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj57 2023-07-29T00:50:04.241 DEBUG:boto:Method: GET 2023-07-29T00:50:04.241 DEBUG:boto:Path: /hiiqos-61/?prefix=obj57 2023-07-29T00:50:04.241 DEBUG:boto:Data: 2023-07-29T00:50:04.242 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.242 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:04.242 DEBUG:boto:Port: 8001 2023-07-29T00:50:04.243 DEBUG:boto:Params: {} 2023-07-29T00:50:04.243 DEBUG:boto:Token: None 2023-07-29T00:50:04.244 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/ 2023-07-29T00:50:04.244 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g= 2023-07-29T00:50:04.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g=', 'Content-Length': '0'} 2023-07-29T00:50:04.247 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022ef0b20ebcb8005-0064c4623c-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.247 DEBUG:boto:b'hiiqos-61obj571000falseobj572023-07-29T00:45:01.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:04.248 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:04.248 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:04.249 DEBUG:boto:path=/hiiqos-61/?prefix=obj57 2023-07-29T00:50:04.249 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj57 2023-07-29T00:50:04.249 DEBUG:boto:Method: GET 2023-07-29T00:50:04.250 DEBUG:boto:Path: /hiiqos-61/?prefix=obj57 2023-07-29T00:50:04.250 DEBUG:boto:Data: 2023-07-29T00:50:04.250 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.251 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:04.251 DEBUG:boto:Port: 8000 2023-07-29T00:50:04.251 DEBUG:boto:Params: {} 2023-07-29T00:50:04.252 DEBUG:boto:Token: None 2023-07-29T00:50:04.252 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/ 2023-07-29T00:50:04.253 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g= 2023-07-29T00:50:04.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g=', 'Content-Length': '0'} 2023-07-29T00:50:04.257 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac750bdf3eeb498b-0064c4623c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.257 DEBUG:boto:b'hiiqos-61obj571000falseobj572023-07-29T00:45:01.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:04.258 DEBUG:boto:path=/hiiqos-61/obj57 2023-07-29T00:50:04.258 DEBUG:boto:auth_path=/hiiqos-61/obj57 2023-07-29T00:50:04.259 DEBUG:boto:Method: HEAD 2023-07-29T00:50:04.259 DEBUG:boto:Path: /hiiqos-61/obj57 2023-07-29T00:50:04.259 DEBUG:boto:Data: 2023-07-29T00:50:04.260 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.260 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:04.260 DEBUG:boto:Port: 8001 2023-07-29T00:50:04.261 DEBUG:boto:Params: {} 2023-07-29T00:50:04.261 DEBUG:boto:Token: None 2023-07-29T00:50:04.262 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj57 2023-07-29T00:50:04.262 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GwQcTh1aMFNKcJflRWBWMUV9d9s= 2023-07-29T00:50:04.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GwQcTh1aMFNKcJflRWBWMUV9d9s=', 'Content-Length': '0'} 2023-07-29T00:50:04.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000348f3a047c19144-0064c4623c-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.264 DEBUG:boto:path=/hiiqos-61/obj57 2023-07-29T00:50:04.265 DEBUG:boto:auth_path=/hiiqos-61/obj57 2023-07-29T00:50:04.265 DEBUG:boto:Method: HEAD 2023-07-29T00:50:04.265 DEBUG:boto:Path: /hiiqos-61/obj57 2023-07-29T00:50:04.266 DEBUG:boto:Data: 2023-07-29T00:50:04.266 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.266 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:04.267 DEBUG:boto:Port: 8000 2023-07-29T00:50:04.267 DEBUG:boto:Params: {} 2023-07-29T00:50:04.268 DEBUG:boto:Token: None 2023-07-29T00:50:04.268 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj57 2023-07-29T00:50:04.268 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GwQcTh1aMFNKcJflRWBWMUV9d9s= 2023-07-29T00:50:04.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GwQcTh1aMFNKcJflRWBWMUV9d9s=', 'Content-Length': '0'} 2023-07-29T00:50:04.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084dc9cd2cc07602b-0064c4623c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-07-29T00:50:04.271 DEBUG:boto:path=/hiiqos-61/obj57 2023-07-29T00:50:04.272 DEBUG:boto:auth_path=/hiiqos-61/obj57 2023-07-29T00:50:04.272 DEBUG:boto:Method: GET 2023-07-29T00:50:04.273 DEBUG:boto:Path: /hiiqos-61/obj57 2023-07-29T00:50:04.273 DEBUG:boto:Data: 2023-07-29T00:50:04.273 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.274 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:04.274 DEBUG:boto:Port: 8001 2023-07-29T00:50:04.274 DEBUG:boto:Params: {} 2023-07-29T00:50:04.275 DEBUG:boto:Token: None 2023-07-29T00:50:04.275 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj57 2023-07-29T00:50:04.276 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:x8Ep9ReNhALsT6mP7posSGIxqVw= 2023-07-29T00:50:04.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:x8Ep9ReNhALsT6mP7posSGIxqVw=', 'Content-Length': '0'} 2023-07-29T00:50:04.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1105a032ccce463-0064c4623c-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.278 DEBUG:boto:path=/hiiqos-61/obj57 2023-07-29T00:50:04.279 DEBUG:boto:auth_path=/hiiqos-61/obj57 2023-07-29T00:50:04.279 DEBUG:boto:Method: GET 2023-07-29T00:50:04.279 DEBUG:boto:Path: /hiiqos-61/obj57 2023-07-29T00:50:04.280 DEBUG:boto:Data: 2023-07-29T00:50:04.280 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.280 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:04.281 DEBUG:boto:Port: 8000 2023-07-29T00:50:04.281 DEBUG:boto:Params: {} 2023-07-29T00:50:04.282 DEBUG:boto:Token: None 2023-07-29T00:50:04.282 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj57 2023-07-29T00:50:04.282 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:x8Ep9ReNhALsT6mP7posSGIxqVw= 2023-07-29T00:50:04.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:x8Ep9ReNhALsT6mP7posSGIxqVw=', 'Content-Length': '0'} 2023-07-29T00:50:04.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c83ebe427d82649c-0064c4623c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.328 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:04.329 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:04.329 DEBUG:boto:path=/hiiqos-61/?prefix=obj57 2023-07-29T00:50:04.330 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj57 2023-07-29T00:50:04.330 DEBUG:boto:Method: GET 2023-07-29T00:50:04.330 DEBUG:boto:Path: /hiiqos-61/?prefix=obj57 2023-07-29T00:50:04.331 DEBUG:boto:Data: 2023-07-29T00:50:04.331 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.331 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:04.332 DEBUG:boto:Port: 8001 2023-07-29T00:50:04.332 DEBUG:boto:Params: {} 2023-07-29T00:50:04.333 DEBUG:boto:Token: None 2023-07-29T00:50:04.333 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/ 2023-07-29T00:50:04.333 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g= 2023-07-29T00:50:04.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g=', 'Content-Length': '0'} 2023-07-29T00:50:04.336 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b91a42caf1c7fabc-0064c4623c-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.336 DEBUG:boto:b'hiiqos-61obj571000falseobj572023-07-29T00:45:01.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:04.337 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:04.338 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:04.338 DEBUG:boto:path=/hiiqos-61/?prefix=obj57 2023-07-29T00:50:04.338 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj57 2023-07-29T00:50:04.339 DEBUG:boto:Method: GET 2023-07-29T00:50:04.339 DEBUG:boto:Path: /hiiqos-61/?prefix=obj57 2023-07-29T00:50:04.339 DEBUG:boto:Data: 2023-07-29T00:50:04.340 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.340 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:04.340 DEBUG:boto:Port: 8000 2023-07-29T00:50:04.341 DEBUG:boto:Params: {} 2023-07-29T00:50:04.341 DEBUG:boto:Token: None 2023-07-29T00:50:04.342 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/ 2023-07-29T00:50:04.342 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g= 2023-07-29T00:50:04.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g=', 'Content-Length': '0'} 2023-07-29T00:50:04.346 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d6dfe81eb08c979-0064c4623c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.346 DEBUG:boto:b'hiiqos-61obj571000falseobj572023-07-29T00:45:01.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:04.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-07-29T00:50:04.347 DEBUG:boto:path=/hiiqos-61/obj58 2023-07-29T00:50:04.348 DEBUG:boto:auth_path=/hiiqos-61/obj58 2023-07-29T00:50:04.348 DEBUG:boto:path=/hiiqos-61/obj58?versionId=null 2023-07-29T00:50:04.349 DEBUG:boto:auth_path=/hiiqos-61/obj58?versionId=null 2023-07-29T00:50:04.349 DEBUG:boto:Method: GET 2023-07-29T00:50:04.349 DEBUG:boto:Path: /hiiqos-61/obj58?versionId=null 2023-07-29T00:50:04.350 DEBUG:boto:Data: 2023-07-29T00:50:04.350 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.350 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:04.351 DEBUG:boto:Port: 8001 2023-07-29T00:50:04.351 DEBUG:boto:Params: {} 2023-07-29T00:50:04.351 DEBUG:boto:Token: None 2023-07-29T00:50:04.352 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj58?versionId=null 2023-07-29T00:50:04.352 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uBHMiGaQ49uWtYXBbJ+QD4MSVBo= 2023-07-29T00:50:04.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uBHMiGaQ49uWtYXBbJ+QD4MSVBo=', 'Content-Length': '0'} 2023-07-29T00:50:04.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8886125412561e0-0064c4623c-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.355 DEBUG:boto:path=/hiiqos-61/obj58 2023-07-29T00:50:04.356 DEBUG:boto:auth_path=/hiiqos-61/obj58 2023-07-29T00:50:04.356 DEBUG:boto:path=/hiiqos-61/obj58?versionId=null 2023-07-29T00:50:04.357 DEBUG:boto:auth_path=/hiiqos-61/obj58?versionId=null 2023-07-29T00:50:04.357 DEBUG:boto:Method: GET 2023-07-29T00:50:04.357 DEBUG:boto:Path: /hiiqos-61/obj58?versionId=null 2023-07-29T00:50:04.358 DEBUG:boto:Data: 2023-07-29T00:50:04.358 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.358 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:04.359 DEBUG:boto:Port: 8000 2023-07-29T00:50:04.359 DEBUG:boto:Params: {} 2023-07-29T00:50:04.359 DEBUG:boto:Token: None 2023-07-29T00:50:04.360 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj58?versionId=null 2023-07-29T00:50:04.360 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uBHMiGaQ49uWtYXBbJ+QD4MSVBo= 2023-07-29T00:50:04.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uBHMiGaQ49uWtYXBbJ+QD4MSVBo=', 'Content-Length': '0'} 2023-07-29T00:50:04.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0557b5956d7887f-0064c4623c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.363 DEBUG:boto:path=/hiiqos-61/obj58 2023-07-29T00:50:04.364 DEBUG:boto:auth_path=/hiiqos-61/obj58 2023-07-29T00:50:04.364 DEBUG:boto:path=/hiiqos-61/obj58?versionId=null 2023-07-29T00:50:04.364 DEBUG:boto:auth_path=/hiiqos-61/obj58?versionId=null 2023-07-29T00:50:04.365 DEBUG:boto:Method: HEAD 2023-07-29T00:50:04.365 DEBUG:boto:Path: /hiiqos-61/obj58?versionId=null 2023-07-29T00:50:04.365 DEBUG:boto:Data: 2023-07-29T00:50:04.366 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.366 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:04.366 DEBUG:boto:Port: 8001 2023-07-29T00:50:04.367 DEBUG:boto:Params: {} 2023-07-29T00:50:04.367 DEBUG:boto:Token: None 2023-07-29T00:50:04.367 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj58?versionId=null 2023-07-29T00:50:04.368 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SYyJ0HbMmCrAPLnQD4LQSTuuAWM= 2023-07-29T00:50:04.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SYyJ0HbMmCrAPLnQD4LQSTuuAWM=', 'Content-Length': '0'} 2023-07-29T00:50:04.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a833d16f2342325-0064c4623c-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.371 DEBUG:boto:path=/hiiqos-61/obj58 2023-07-29T00:50:04.371 DEBUG:boto:auth_path=/hiiqos-61/obj58 2023-07-29T00:50:04.371 DEBUG:boto:path=/hiiqos-61/obj58?versionId=null 2023-07-29T00:50:04.372 DEBUG:boto:auth_path=/hiiqos-61/obj58?versionId=null 2023-07-29T00:50:04.372 DEBUG:boto:Method: HEAD 2023-07-29T00:50:04.372 DEBUG:boto:Path: /hiiqos-61/obj58?versionId=null 2023-07-29T00:50:04.373 DEBUG:boto:Data: 2023-07-29T00:50:04.373 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.373 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:04.374 DEBUG:boto:Port: 8000 2023-07-29T00:50:04.374 DEBUG:boto:Params: {} 2023-07-29T00:50:04.374 DEBUG:boto:Token: None 2023-07-29T00:50:04.375 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj58?versionId=null 2023-07-29T00:50:04.375 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SYyJ0HbMmCrAPLnQD4LQSTuuAWM= 2023-07-29T00:50:04.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SYyJ0HbMmCrAPLnQD4LQSTuuAWM=', 'Content-Length': '0'} 2023-07-29T00:50:04.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1e82dd99c97473a-0064c4623c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-07-29T00:50:04.379 DEBUG:boto:path=/hiiqos-61/obj58 2023-07-29T00:50:04.379 DEBUG:boto:auth_path=/hiiqos-61/obj58 2023-07-29T00:50:04.379 DEBUG:boto:path=/hiiqos-61/obj58?versionId=null 2023-07-29T00:50:04.380 DEBUG:boto:auth_path=/hiiqos-61/obj58?versionId=null 2023-07-29T00:50:04.380 DEBUG:boto:Method: GET 2023-07-29T00:50:04.380 DEBUG:boto:Path: /hiiqos-61/obj58?versionId=null 2023-07-29T00:50:04.381 DEBUG:boto:Data: 2023-07-29T00:50:04.381 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.381 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:04.382 DEBUG:boto:Port: 8001 2023-07-29T00:50:04.382 DEBUG:boto:Params: {} 2023-07-29T00:50:04.383 DEBUG:boto:Token: None 2023-07-29T00:50:04.383 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj58?versionId=null 2023-07-29T00:50:04.383 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uBHMiGaQ49uWtYXBbJ+QD4MSVBo= 2023-07-29T00:50:04.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uBHMiGaQ49uWtYXBbJ+QD4MSVBo=', 'Content-Length': '0'} 2023-07-29T00:50:04.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a036bb4c51c243f-0064c4623c-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.386 DEBUG:boto:path=/hiiqos-61/obj58 2023-07-29T00:50:04.386 DEBUG:boto:auth_path=/hiiqos-61/obj58 2023-07-29T00:50:04.387 DEBUG:boto:path=/hiiqos-61/obj58?versionId=null 2023-07-29T00:50:04.387 DEBUG:boto:auth_path=/hiiqos-61/obj58?versionId=null 2023-07-29T00:50:04.388 DEBUG:boto:Method: GET 2023-07-29T00:50:04.388 DEBUG:boto:Path: /hiiqos-61/obj58?versionId=null 2023-07-29T00:50:04.388 DEBUG:boto:Data: 2023-07-29T00:50:04.389 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.389 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:04.389 DEBUG:boto:Port: 8000 2023-07-29T00:50:04.390 DEBUG:boto:Params: {} 2023-07-29T00:50:04.390 DEBUG:boto:Token: None 2023-07-29T00:50:04.391 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj58?versionId=null 2023-07-29T00:50:04.391 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uBHMiGaQ49uWtYXBbJ+QD4MSVBo= 2023-07-29T00:50:04.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uBHMiGaQ49uWtYXBbJ+QD4MSVBo=', 'Content-Length': '0'} 2023-07-29T00:50:04.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000097fc6d921b0d11e-0064c4623c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.394 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:04.394 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:04.395 DEBUG:boto:path=/hiiqos-61/?prefix=obj58 2023-07-29T00:50:04.395 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj58 2023-07-29T00:50:04.396 DEBUG:boto:Method: GET 2023-07-29T00:50:04.396 DEBUG:boto:Path: /hiiqos-61/?prefix=obj58 2023-07-29T00:50:04.396 DEBUG:boto:Data: 2023-07-29T00:50:04.397 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.397 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:04.397 DEBUG:boto:Port: 8001 2023-07-29T00:50:04.398 DEBUG:boto:Params: {} 2023-07-29T00:50:04.398 DEBUG:boto:Token: None 2023-07-29T00:50:04.399 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/ 2023-07-29T00:50:04.399 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g= 2023-07-29T00:50:04.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g=', 'Content-Length': '0'} 2023-07-29T00:50:04.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fdb317b5faee286-0064c4623c-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.402 DEBUG:boto:b'hiiqos-61obj581000falseobj582023-07-29T00:45:02.299Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:04.403 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:04.403 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:04.404 DEBUG:boto:path=/hiiqos-61/?prefix=obj58 2023-07-29T00:50:04.404 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj58 2023-07-29T00:50:04.404 DEBUG:boto:Method: GET 2023-07-29T00:50:04.405 DEBUG:boto:Path: /hiiqos-61/?prefix=obj58 2023-07-29T00:50:04.405 DEBUG:boto:Data: 2023-07-29T00:50:04.405 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.406 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:04.406 DEBUG:boto:Port: 8000 2023-07-29T00:50:04.406 DEBUG:boto:Params: {} 2023-07-29T00:50:04.407 DEBUG:boto:Token: None 2023-07-29T00:50:04.407 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/ 2023-07-29T00:50:04.408 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g= 2023-07-29T00:50:04.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g=', 'Content-Length': '0'} 2023-07-29T00:50:04.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c3c3c08dc2450fd-0064c4623c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.412 DEBUG:boto:b'hiiqos-61obj581000falseobj582023-07-29T00:45:02.299Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:04.413 DEBUG:boto:path=/hiiqos-61/obj58 2023-07-29T00:50:04.413 DEBUG:boto:auth_path=/hiiqos-61/obj58 2023-07-29T00:50:04.414 DEBUG:boto:Method: HEAD 2023-07-29T00:50:04.414 DEBUG:boto:Path: /hiiqos-61/obj58 2023-07-29T00:50:04.415 DEBUG:boto:Data: 2023-07-29T00:50:04.415 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.415 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:04.416 DEBUG:boto:Port: 8001 2023-07-29T00:50:04.416 DEBUG:boto:Params: {} 2023-07-29T00:50:04.416 DEBUG:boto:Token: None 2023-07-29T00:50:04.417 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj58 2023-07-29T00:50:04.417 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PY4ch6wJcC+d4e+gnYazwCYwfuk= 2023-07-29T00:50:04.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PY4ch6wJcC+d4e+gnYazwCYwfuk=', 'Content-Length': '0'} 2023-07-29T00:50:04.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1274e2cf260b4f9-0064c4623c-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.420 DEBUG:boto:path=/hiiqos-61/obj58 2023-07-29T00:50:04.420 DEBUG:boto:auth_path=/hiiqos-61/obj58 2023-07-29T00:50:04.420 DEBUG:boto:Method: HEAD 2023-07-29T00:50:04.421 DEBUG:boto:Path: /hiiqos-61/obj58 2023-07-29T00:50:04.421 DEBUG:boto:Data: 2023-07-29T00:50:04.421 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.422 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:04.422 DEBUG:boto:Port: 8000 2023-07-29T00:50:04.423 DEBUG:boto:Params: {} 2023-07-29T00:50:04.423 DEBUG:boto:Token: None 2023-07-29T00:50:04.423 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj58 2023-07-29T00:50:04.424 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PY4ch6wJcC+d4e+gnYazwCYwfuk= 2023-07-29T00:50:04.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PY4ch6wJcC+d4e+gnYazwCYwfuk=', 'Content-Length': '0'} 2023-07-29T00:50:04.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005020b1aa645ef392-0064c4623c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-07-29T00:50:04.427 DEBUG:boto:path=/hiiqos-61/obj58 2023-07-29T00:50:04.427 DEBUG:boto:auth_path=/hiiqos-61/obj58 2023-07-29T00:50:04.428 DEBUG:boto:Method: GET 2023-07-29T00:50:04.428 DEBUG:boto:Path: /hiiqos-61/obj58 2023-07-29T00:50:04.428 DEBUG:boto:Data: 2023-07-29T00:50:04.429 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.429 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:04.429 DEBUG:boto:Port: 8001 2023-07-29T00:50:04.430 DEBUG:boto:Params: {} 2023-07-29T00:50:04.430 DEBUG:boto:Token: None 2023-07-29T00:50:04.431 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj58 2023-07-29T00:50:04.431 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+HHZMaUzJbZWZZFWPIY3pza/In4= 2023-07-29T00:50:04.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+HHZMaUzJbZWZZFWPIY3pza/In4=', 'Content-Length': '0'} 2023-07-29T00:50:04.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a4d42f8639b446e-0064c4623c-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.434 DEBUG:boto:path=/hiiqos-61/obj58 2023-07-29T00:50:04.434 DEBUG:boto:auth_path=/hiiqos-61/obj58 2023-07-29T00:50:04.435 DEBUG:boto:Method: GET 2023-07-29T00:50:04.435 DEBUG:boto:Path: /hiiqos-61/obj58 2023-07-29T00:50:04.435 DEBUG:boto:Data: 2023-07-29T00:50:04.436 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.436 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:04.436 DEBUG:boto:Port: 8000 2023-07-29T00:50:04.437 DEBUG:boto:Params: {} 2023-07-29T00:50:04.437 DEBUG:boto:Token: None 2023-07-29T00:50:04.438 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj58 2023-07-29T00:50:04.438 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+HHZMaUzJbZWZZFWPIY3pza/In4= 2023-07-29T00:50:04.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+HHZMaUzJbZWZZFWPIY3pza/In4=', 'Content-Length': '0'} 2023-07-29T00:50:04.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b5d4fc428cc009d-0064c4623c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.480 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:04.481 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:04.481 DEBUG:boto:path=/hiiqos-61/?prefix=obj58 2023-07-29T00:50:04.481 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj58 2023-07-29T00:50:04.482 DEBUG:boto:Method: GET 2023-07-29T00:50:04.482 DEBUG:boto:Path: /hiiqos-61/?prefix=obj58 2023-07-29T00:50:04.483 DEBUG:boto:Data: 2023-07-29T00:50:04.483 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.483 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:04.484 DEBUG:boto:Port: 8001 2023-07-29T00:50:04.484 DEBUG:boto:Params: {} 2023-07-29T00:50:04.484 DEBUG:boto:Token: None 2023-07-29T00:50:04.485 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/ 2023-07-29T00:50:04.485 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g= 2023-07-29T00:50:04.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g=', 'Content-Length': '0'} 2023-07-29T00:50:04.488 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b07015d05b9ab50-0064c4623c-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.488 DEBUG:boto:b'hiiqos-61obj581000falseobj582023-07-29T00:45:02.299Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:04.489 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:04.489 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:04.490 DEBUG:boto:path=/hiiqos-61/?prefix=obj58 2023-07-29T00:50:04.490 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj58 2023-07-29T00:50:04.491 DEBUG:boto:Method: GET 2023-07-29T00:50:04.491 DEBUG:boto:Path: /hiiqos-61/?prefix=obj58 2023-07-29T00:50:04.491 DEBUG:boto:Data: 2023-07-29T00:50:04.492 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.492 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:04.492 DEBUG:boto:Port: 8000 2023-07-29T00:50:04.493 DEBUG:boto:Params: {} 2023-07-29T00:50:04.493 DEBUG:boto:Token: None 2023-07-29T00:50:04.493 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/ 2023-07-29T00:50:04.494 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g= 2023-07-29T00:50:04.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g=', 'Content-Length': '0'} 2023-07-29T00:50:04.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c60c0ab7690354d-0064c4623c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.498 DEBUG:boto:b'hiiqos-61obj581000falseobj582023-07-29T00:45:02.299Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:04.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-07-29T00:50:04.499 DEBUG:boto:path=/hiiqos-61/obj62 2023-07-29T00:50:04.499 DEBUG:boto:auth_path=/hiiqos-61/obj62 2023-07-29T00:50:04.500 DEBUG:boto:path=/hiiqos-61/obj62?versionId=null 2023-07-29T00:50:04.500 DEBUG:boto:auth_path=/hiiqos-61/obj62?versionId=null 2023-07-29T00:50:04.501 DEBUG:boto:Method: GET 2023-07-29T00:50:04.501 DEBUG:boto:Path: /hiiqos-61/obj62?versionId=null 2023-07-29T00:50:04.501 DEBUG:boto:Data: 2023-07-29T00:50:04.502 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.502 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:04.502 DEBUG:boto:Port: 8001 2023-07-29T00:50:04.503 DEBUG:boto:Params: {} 2023-07-29T00:50:04.503 DEBUG:boto:Token: None 2023-07-29T00:50:04.503 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj62?versionId=null 2023-07-29T00:50:04.504 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Q5tJSt7JwiILzuHUNneCr3DH/3g= 2023-07-29T00:50:04.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q5tJSt7JwiILzuHUNneCr3DH/3g=', 'Content-Length': '0'} 2023-07-29T00:50:04.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b43aaf33a0a0c571-0064c4623c-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.507 DEBUG:boto:path=/hiiqos-61/obj62 2023-07-29T00:50:04.507 DEBUG:boto:auth_path=/hiiqos-61/obj62 2023-07-29T00:50:04.507 DEBUG:boto:path=/hiiqos-61/obj62?versionId=null 2023-07-29T00:50:04.508 DEBUG:boto:auth_path=/hiiqos-61/obj62?versionId=null 2023-07-29T00:50:04.508 DEBUG:boto:Method: GET 2023-07-29T00:50:04.509 DEBUG:boto:Path: /hiiqos-61/obj62?versionId=null 2023-07-29T00:50:04.509 DEBUG:boto:Data: 2023-07-29T00:50:04.509 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.510 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:04.510 DEBUG:boto:Port: 8000 2023-07-29T00:50:04.510 DEBUG:boto:Params: {} 2023-07-29T00:50:04.511 DEBUG:boto:Token: None 2023-07-29T00:50:04.511 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj62?versionId=null 2023-07-29T00:50:04.512 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Q5tJSt7JwiILzuHUNneCr3DH/3g= 2023-07-29T00:50:04.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q5tJSt7JwiILzuHUNneCr3DH/3g=', 'Content-Length': '0'} 2023-07-29T00:50:04.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a61c81fc1ac43e6b-0064c4623c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.514 DEBUG:boto:path=/hiiqos-61/obj62 2023-07-29T00:50:04.515 DEBUG:boto:auth_path=/hiiqos-61/obj62 2023-07-29T00:50:04.515 DEBUG:boto:path=/hiiqos-61/obj62?versionId=null 2023-07-29T00:50:04.515 DEBUG:boto:auth_path=/hiiqos-61/obj62?versionId=null 2023-07-29T00:50:04.516 DEBUG:boto:Method: HEAD 2023-07-29T00:50:04.516 DEBUG:boto:Path: /hiiqos-61/obj62?versionId=null 2023-07-29T00:50:04.516 DEBUG:boto:Data: 2023-07-29T00:50:04.517 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.517 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:04.517 DEBUG:boto:Port: 8001 2023-07-29T00:50:04.518 DEBUG:boto:Params: {} 2023-07-29T00:50:04.518 DEBUG:boto:Token: None 2023-07-29T00:50:04.518 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj62?versionId=null 2023-07-29T00:50:04.519 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JfsscHklLiPeNQv/G4+T2Wexjqk= 2023-07-29T00:50:04.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JfsscHklLiPeNQv/G4+T2Wexjqk=', 'Content-Length': '0'} 2023-07-29T00:50:04.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c875d41e3da0994-0064c4623c-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.521 DEBUG:boto:path=/hiiqos-61/obj62 2023-07-29T00:50:04.522 DEBUG:boto:auth_path=/hiiqos-61/obj62 2023-07-29T00:50:04.522 DEBUG:boto:path=/hiiqos-61/obj62?versionId=null 2023-07-29T00:50:04.522 DEBUG:boto:auth_path=/hiiqos-61/obj62?versionId=null 2023-07-29T00:50:04.523 DEBUG:boto:Method: HEAD 2023-07-29T00:50:04.523 DEBUG:boto:Path: /hiiqos-61/obj62?versionId=null 2023-07-29T00:50:04.523 DEBUG:boto:Data: 2023-07-29T00:50:04.524 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.524 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:04.524 DEBUG:boto:Port: 8000 2023-07-29T00:50:04.525 DEBUG:boto:Params: {} 2023-07-29T00:50:04.525 DEBUG:boto:Token: None 2023-07-29T00:50:04.526 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj62?versionId=null 2023-07-29T00:50:04.526 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JfsscHklLiPeNQv/G4+T2Wexjqk= 2023-07-29T00:50:04.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JfsscHklLiPeNQv/G4+T2Wexjqk=', 'Content-Length': '0'} 2023-07-29T00:50:04.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce36f179b9765da1-0064c4623c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-07-29T00:50:04.529 DEBUG:boto:path=/hiiqos-61/obj62 2023-07-29T00:50:04.529 DEBUG:boto:auth_path=/hiiqos-61/obj62 2023-07-29T00:50:04.530 DEBUG:boto:path=/hiiqos-61/obj62?versionId=null 2023-07-29T00:50:04.530 DEBUG:boto:auth_path=/hiiqos-61/obj62?versionId=null 2023-07-29T00:50:04.530 DEBUG:boto:Method: GET 2023-07-29T00:50:04.531 DEBUG:boto:Path: /hiiqos-61/obj62?versionId=null 2023-07-29T00:50:04.531 DEBUG:boto:Data: 2023-07-29T00:50:04.531 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.532 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:04.532 DEBUG:boto:Port: 8001 2023-07-29T00:50:04.533 DEBUG:boto:Params: {} 2023-07-29T00:50:04.533 DEBUG:boto:Token: None 2023-07-29T00:50:04.533 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj62?versionId=null 2023-07-29T00:50:04.534 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Q5tJSt7JwiILzuHUNneCr3DH/3g= 2023-07-29T00:50:04.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q5tJSt7JwiILzuHUNneCr3DH/3g=', 'Content-Length': '0'} 2023-07-29T00:50:04.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bbe040380fcbaae-0064c4623c-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.576 DEBUG:boto:path=/hiiqos-61/obj62 2023-07-29T00:50:04.577 DEBUG:boto:auth_path=/hiiqos-61/obj62 2023-07-29T00:50:04.577 DEBUG:boto:path=/hiiqos-61/obj62?versionId=null 2023-07-29T00:50:04.577 DEBUG:boto:auth_path=/hiiqos-61/obj62?versionId=null 2023-07-29T00:50:04.578 DEBUG:boto:Method: GET 2023-07-29T00:50:04.578 DEBUG:boto:Path: /hiiqos-61/obj62?versionId=null 2023-07-29T00:50:04.579 DEBUG:boto:Data: 2023-07-29T00:50:04.579 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.579 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:04.580 DEBUG:boto:Port: 8000 2023-07-29T00:50:04.580 DEBUG:boto:Params: {} 2023-07-29T00:50:04.580 DEBUG:boto:Token: None 2023-07-29T00:50:04.581 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj62?versionId=null 2023-07-29T00:50:04.581 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Q5tJSt7JwiILzuHUNneCr3DH/3g= 2023-07-29T00:50:04.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q5tJSt7JwiILzuHUNneCr3DH/3g=', 'Content-Length': '0'} 2023-07-29T00:50:04.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000195ce4b65de3dbeb-0064c4623c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.584 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:04.585 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:04.585 DEBUG:boto:path=/hiiqos-61/?prefix=obj62 2023-07-29T00:50:04.585 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj62 2023-07-29T00:50:04.586 DEBUG:boto:Method: GET 2023-07-29T00:50:04.586 DEBUG:boto:Path: /hiiqos-61/?prefix=obj62 2023-07-29T00:50:04.586 DEBUG:boto:Data: 2023-07-29T00:50:04.587 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.587 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:04.587 DEBUG:boto:Port: 8001 2023-07-29T00:50:04.588 DEBUG:boto:Params: {} 2023-07-29T00:50:04.588 DEBUG:boto:Token: None 2023-07-29T00:50:04.589 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/ 2023-07-29T00:50:04.589 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g= 2023-07-29T00:50:04.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g=', 'Content-Length': '0'} 2023-07-29T00:50:04.592 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8d764ca86da8301-0064c4623c-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.592 DEBUG:boto:b'hiiqos-61obj621000falseobj622023-07-29T00:45:02.703Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:04.593 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:04.593 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:04.594 DEBUG:boto:path=/hiiqos-61/?prefix=obj62 2023-07-29T00:50:04.594 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj62 2023-07-29T00:50:04.594 DEBUG:boto:Method: GET 2023-07-29T00:50:04.595 DEBUG:boto:Path: /hiiqos-61/?prefix=obj62 2023-07-29T00:50:04.595 DEBUG:boto:Data: 2023-07-29T00:50:04.595 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.596 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:04.596 DEBUG:boto:Port: 8000 2023-07-29T00:50:04.596 DEBUG:boto:Params: {} 2023-07-29T00:50:04.597 DEBUG:boto:Token: None 2023-07-29T00:50:04.597 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/ 2023-07-29T00:50:04.598 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g= 2023-07-29T00:50:04.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g=', 'Content-Length': '0'} 2023-07-29T00:50:04.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000736bb5047ac55f6f-0064c4623c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.602 DEBUG:boto:b'hiiqos-61obj621000falseobj622023-07-29T00:45:02.703Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:04.603 DEBUG:boto:path=/hiiqos-61/obj62 2023-07-29T00:50:04.603 DEBUG:boto:auth_path=/hiiqos-61/obj62 2023-07-29T00:50:04.604 DEBUG:boto:Method: HEAD 2023-07-29T00:50:04.604 DEBUG:boto:Path: /hiiqos-61/obj62 2023-07-29T00:50:04.604 DEBUG:boto:Data: 2023-07-29T00:50:04.605 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.605 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:04.605 DEBUG:boto:Port: 8001 2023-07-29T00:50:04.606 DEBUG:boto:Params: {} 2023-07-29T00:50:04.606 DEBUG:boto:Token: None 2023-07-29T00:50:04.606 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj62 2023-07-29T00:50:04.607 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/XIQ6C6vFA5lBTFXT9l+TtWaZ2s= 2023-07-29T00:50:04.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/XIQ6C6vFA5lBTFXT9l+TtWaZ2s=', 'Content-Length': '0'} 2023-07-29T00:50:04.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dfc8b67e657eb10-0064c4623c-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.610 DEBUG:boto:path=/hiiqos-61/obj62 2023-07-29T00:50:04.610 DEBUG:boto:auth_path=/hiiqos-61/obj62 2023-07-29T00:50:04.610 DEBUG:boto:Method: HEAD 2023-07-29T00:50:04.611 DEBUG:boto:Path: /hiiqos-61/obj62 2023-07-29T00:50:04.611 DEBUG:boto:Data: 2023-07-29T00:50:04.611 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.612 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:04.612 DEBUG:boto:Port: 8000 2023-07-29T00:50:04.612 DEBUG:boto:Params: {} 2023-07-29T00:50:04.613 DEBUG:boto:Token: None 2023-07-29T00:50:04.613 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj62 2023-07-29T00:50:04.613 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/XIQ6C6vFA5lBTFXT9l+TtWaZ2s= 2023-07-29T00:50:04.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/XIQ6C6vFA5lBTFXT9l+TtWaZ2s=', 'Content-Length': '0'} 2023-07-29T00:50:04.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cc6a475b795b457-0064c4623c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-07-29T00:50:04.616 DEBUG:boto:path=/hiiqos-61/obj62 2023-07-29T00:50:04.617 DEBUG:boto:auth_path=/hiiqos-61/obj62 2023-07-29T00:50:04.617 DEBUG:boto:Method: GET 2023-07-29T00:50:04.618 DEBUG:boto:Path: /hiiqos-61/obj62 2023-07-29T00:50:04.618 DEBUG:boto:Data: 2023-07-29T00:50:04.618 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.619 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:04.619 DEBUG:boto:Port: 8001 2023-07-29T00:50:04.619 DEBUG:boto:Params: {} 2023-07-29T00:50:04.620 DEBUG:boto:Token: None 2023-07-29T00:50:04.620 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj62 2023-07-29T00:50:04.621 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Kj7PPLc9/7Pq+l2zCjlPxf7zX/g= 2023-07-29T00:50:04.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Kj7PPLc9/7Pq+l2zCjlPxf7zX/g=', 'Content-Length': '0'} 2023-07-29T00:50:04.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000302767e6958fb0d1-0064c4623c-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.623 DEBUG:boto:path=/hiiqos-61/obj62 2023-07-29T00:50:04.624 DEBUG:boto:auth_path=/hiiqos-61/obj62 2023-07-29T00:50:04.624 DEBUG:boto:Method: GET 2023-07-29T00:50:04.624 DEBUG:boto:Path: /hiiqos-61/obj62 2023-07-29T00:50:04.625 DEBUG:boto:Data: 2023-07-29T00:50:04.625 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.625 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:04.626 DEBUG:boto:Port: 8000 2023-07-29T00:50:04.626 DEBUG:boto:Params: {} 2023-07-29T00:50:04.627 DEBUG:boto:Token: None 2023-07-29T00:50:04.627 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj62 2023-07-29T00:50:04.627 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Kj7PPLc9/7Pq+l2zCjlPxf7zX/g= 2023-07-29T00:50:04.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Kj7PPLc9/7Pq+l2zCjlPxf7zX/g=', 'Content-Length': '0'} 2023-07-29T00:50:04.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f45b51bf5d3e6ca3-0064c4623c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.672 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:04.673 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:04.673 DEBUG:boto:path=/hiiqos-61/?prefix=obj62 2023-07-29T00:50:04.674 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj62 2023-07-29T00:50:04.674 DEBUG:boto:Method: GET 2023-07-29T00:50:04.674 DEBUG:boto:Path: /hiiqos-61/?prefix=obj62 2023-07-29T00:50:04.675 DEBUG:boto:Data: 2023-07-29T00:50:04.675 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.675 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:04.676 DEBUG:boto:Port: 8001 2023-07-29T00:50:04.676 DEBUG:boto:Params: {} 2023-07-29T00:50:04.677 DEBUG:boto:Token: None 2023-07-29T00:50:04.677 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/ 2023-07-29T00:50:04.677 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g= 2023-07-29T00:50:04.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g=', 'Content-Length': '0'} 2023-07-29T00:50:04.680 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cd616a52379fe0a-0064c4623c-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.680 DEBUG:boto:b'hiiqos-61obj621000falseobj622023-07-29T00:45:02.703Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:04.681 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:04.682 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:04.682 DEBUG:boto:path=/hiiqos-61/?prefix=obj62 2023-07-29T00:50:04.682 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj62 2023-07-29T00:50:04.683 DEBUG:boto:Method: GET 2023-07-29T00:50:04.683 DEBUG:boto:Path: /hiiqos-61/?prefix=obj62 2023-07-29T00:50:04.683 DEBUG:boto:Data: 2023-07-29T00:50:04.684 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.684 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:04.684 DEBUG:boto:Port: 8000 2023-07-29T00:50:04.685 DEBUG:boto:Params: {} 2023-07-29T00:50:04.685 DEBUG:boto:Token: None 2023-07-29T00:50:04.686 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/ 2023-07-29T00:50:04.686 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g= 2023-07-29T00:50:04.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g=', 'Content-Length': '0'} 2023-07-29T00:50:04.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b29408fd5f7ad51-0064c4623c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.691 DEBUG:boto:b'hiiqos-61obj621000falseobj622023-07-29T00:45:02.703Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:04.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-07-29T00:50:04.692 DEBUG:boto:path=/hiiqos-61/obj68 2023-07-29T00:50:04.692 DEBUG:boto:auth_path=/hiiqos-61/obj68 2023-07-29T00:50:04.692 DEBUG:boto:path=/hiiqos-61/obj68?versionId=null 2023-07-29T00:50:04.693 DEBUG:boto:auth_path=/hiiqos-61/obj68?versionId=null 2023-07-29T00:50:04.693 DEBUG:boto:Method: GET 2023-07-29T00:50:04.693 DEBUG:boto:Path: /hiiqos-61/obj68?versionId=null 2023-07-29T00:50:04.694 DEBUG:boto:Data: 2023-07-29T00:50:04.694 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.695 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:04.695 DEBUG:boto:Port: 8001 2023-07-29T00:50:04.695 DEBUG:boto:Params: {} 2023-07-29T00:50:04.696 DEBUG:boto:Token: None 2023-07-29T00:50:04.696 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj68?versionId=null 2023-07-29T00:50:04.697 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Nq+stFLrnaFijZIBMUCydlc5HJY= 2023-07-29T00:50:04.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Nq+stFLrnaFijZIBMUCydlc5HJY=', 'Content-Length': '0'} 2023-07-29T00:50:04.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006cdfe92e0d0d679-0064c4623c-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.699 DEBUG:boto:path=/hiiqos-61/obj68 2023-07-29T00:50:04.700 DEBUG:boto:auth_path=/hiiqos-61/obj68 2023-07-29T00:50:04.700 DEBUG:boto:path=/hiiqos-61/obj68?versionId=null 2023-07-29T00:50:04.701 DEBUG:boto:auth_path=/hiiqos-61/obj68?versionId=null 2023-07-29T00:50:04.701 DEBUG:boto:Method: GET 2023-07-29T00:50:04.701 DEBUG:boto:Path: /hiiqos-61/obj68?versionId=null 2023-07-29T00:50:04.702 DEBUG:boto:Data: 2023-07-29T00:50:04.702 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.702 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:04.703 DEBUG:boto:Port: 8000 2023-07-29T00:50:04.703 DEBUG:boto:Params: {} 2023-07-29T00:50:04.703 DEBUG:boto:Token: None 2023-07-29T00:50:04.704 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj68?versionId=null 2023-07-29T00:50:04.704 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Nq+stFLrnaFijZIBMUCydlc5HJY= 2023-07-29T00:50:04.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Nq+stFLrnaFijZIBMUCydlc5HJY=', 'Content-Length': '0'} 2023-07-29T00:50:04.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fe9957b4104d12d-0064c4623c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.707 DEBUG:boto:path=/hiiqos-61/obj68 2023-07-29T00:50:04.708 DEBUG:boto:auth_path=/hiiqos-61/obj68 2023-07-29T00:50:04.708 DEBUG:boto:path=/hiiqos-61/obj68?versionId=null 2023-07-29T00:50:04.708 DEBUG:boto:auth_path=/hiiqos-61/obj68?versionId=null 2023-07-29T00:50:04.709 DEBUG:boto:Method: HEAD 2023-07-29T00:50:04.709 DEBUG:boto:Path: /hiiqos-61/obj68?versionId=null 2023-07-29T00:50:04.709 DEBUG:boto:Data: 2023-07-29T00:50:04.710 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.710 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:04.710 DEBUG:boto:Port: 8001 2023-07-29T00:50:04.711 DEBUG:boto:Params: {} 2023-07-29T00:50:04.711 DEBUG:boto:Token: None 2023-07-29T00:50:04.711 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj68?versionId=null 2023-07-29T00:50:04.712 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zyvLrfT6PnU5bJPA80fF3uYvWfE= 2023-07-29T00:50:04.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zyvLrfT6PnU5bJPA80fF3uYvWfE=', 'Content-Length': '0'} 2023-07-29T00:50:04.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000595da65ac1340991-0064c4623c-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.714 DEBUG:boto:path=/hiiqos-61/obj68 2023-07-29T00:50:04.715 DEBUG:boto:auth_path=/hiiqos-61/obj68 2023-07-29T00:50:04.715 DEBUG:boto:path=/hiiqos-61/obj68?versionId=null 2023-07-29T00:50:04.715 DEBUG:boto:auth_path=/hiiqos-61/obj68?versionId=null 2023-07-29T00:50:04.716 DEBUG:boto:Method: HEAD 2023-07-29T00:50:04.716 DEBUG:boto:Path: /hiiqos-61/obj68?versionId=null 2023-07-29T00:50:04.716 DEBUG:boto:Data: 2023-07-29T00:50:04.717 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.717 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:04.717 DEBUG:boto:Port: 8000 2023-07-29T00:50:04.718 DEBUG:boto:Params: {} 2023-07-29T00:50:04.718 DEBUG:boto:Token: None 2023-07-29T00:50:04.718 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj68?versionId=null 2023-07-29T00:50:04.719 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zyvLrfT6PnU5bJPA80fF3uYvWfE= 2023-07-29T00:50:04.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zyvLrfT6PnU5bJPA80fF3uYvWfE=', 'Content-Length': '0'} 2023-07-29T00:50:04.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014d9dd78834f055e-0064c4623c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-07-29T00:50:04.722 DEBUG:boto:path=/hiiqos-61/obj68 2023-07-29T00:50:04.722 DEBUG:boto:auth_path=/hiiqos-61/obj68 2023-07-29T00:50:04.723 DEBUG:boto:path=/hiiqos-61/obj68?versionId=null 2023-07-29T00:50:04.723 DEBUG:boto:auth_path=/hiiqos-61/obj68?versionId=null 2023-07-29T00:50:04.723 DEBUG:boto:Method: GET 2023-07-29T00:50:04.724 DEBUG:boto:Path: /hiiqos-61/obj68?versionId=null 2023-07-29T00:50:04.724 DEBUG:boto:Data: 2023-07-29T00:50:04.724 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.725 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:04.725 DEBUG:boto:Port: 8001 2023-07-29T00:50:04.725 DEBUG:boto:Params: {} 2023-07-29T00:50:04.726 DEBUG:boto:Token: None 2023-07-29T00:50:04.726 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj68?versionId=null 2023-07-29T00:50:04.727 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Nq+stFLrnaFijZIBMUCydlc5HJY= 2023-07-29T00:50:04.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Nq+stFLrnaFijZIBMUCydlc5HJY=', 'Content-Length': '0'} 2023-07-29T00:50:04.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f9d8cffc2b22414-0064c4623c-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.729 DEBUG:boto:path=/hiiqos-61/obj68 2023-07-29T00:50:04.730 DEBUG:boto:auth_path=/hiiqos-61/obj68 2023-07-29T00:50:04.730 DEBUG:boto:path=/hiiqos-61/obj68?versionId=null 2023-07-29T00:50:04.731 DEBUG:boto:auth_path=/hiiqos-61/obj68?versionId=null 2023-07-29T00:50:04.731 DEBUG:boto:Method: GET 2023-07-29T00:50:04.731 DEBUG:boto:Path: /hiiqos-61/obj68?versionId=null 2023-07-29T00:50:04.732 DEBUG:boto:Data: 2023-07-29T00:50:04.732 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.732 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:04.733 DEBUG:boto:Port: 8000 2023-07-29T00:50:04.733 DEBUG:boto:Params: {} 2023-07-29T00:50:04.733 DEBUG:boto:Token: None 2023-07-29T00:50:04.734 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj68?versionId=null 2023-07-29T00:50:04.734 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Nq+stFLrnaFijZIBMUCydlc5HJY= 2023-07-29T00:50:04.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Nq+stFLrnaFijZIBMUCydlc5HJY=', 'Content-Length': '0'} 2023-07-29T00:50:04.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021279813584453cb-0064c4623c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.780 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:04.781 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:04.781 DEBUG:boto:path=/hiiqos-61/?prefix=obj68 2023-07-29T00:50:04.781 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj68 2023-07-29T00:50:04.782 DEBUG:boto:Method: GET 2023-07-29T00:50:04.782 DEBUG:boto:Path: /hiiqos-61/?prefix=obj68 2023-07-29T00:50:04.783 DEBUG:boto:Data: 2023-07-29T00:50:04.783 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.783 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:04.784 DEBUG:boto:Port: 8001 2023-07-29T00:50:04.784 DEBUG:boto:Params: {} 2023-07-29T00:50:04.784 DEBUG:boto:Token: None 2023-07-29T00:50:04.785 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/ 2023-07-29T00:50:04.785 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g= 2023-07-29T00:50:04.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g=', 'Content-Length': '0'} 2023-07-29T00:50:04.788 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008713282a5919703b-0064c4623c-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.788 DEBUG:boto:b'hiiqos-61obj681000falseobj682023-07-29T00:45:01.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:04.789 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:04.789 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:04.790 DEBUG:boto:path=/hiiqos-61/?prefix=obj68 2023-07-29T00:50:04.790 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj68 2023-07-29T00:50:04.791 DEBUG:boto:Method: GET 2023-07-29T00:50:04.791 DEBUG:boto:Path: /hiiqos-61/?prefix=obj68 2023-07-29T00:50:04.791 DEBUG:boto:Data: 2023-07-29T00:50:04.792 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.792 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:04.792 DEBUG:boto:Port: 8000 2023-07-29T00:50:04.793 DEBUG:boto:Params: {} 2023-07-29T00:50:04.793 DEBUG:boto:Token: None 2023-07-29T00:50:04.794 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/ 2023-07-29T00:50:04.794 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g= 2023-07-29T00:50:04.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g=', 'Content-Length': '0'} 2023-07-29T00:50:04.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000903f6da32772799e-0064c4623c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.799 DEBUG:boto:b'hiiqos-61obj681000falseobj682023-07-29T00:45:01.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:04.799 DEBUG:boto:path=/hiiqos-61/obj68 2023-07-29T00:50:04.800 DEBUG:boto:auth_path=/hiiqos-61/obj68 2023-07-29T00:50:04.800 DEBUG:boto:Method: HEAD 2023-07-29T00:50:04.800 DEBUG:boto:Path: /hiiqos-61/obj68 2023-07-29T00:50:04.801 DEBUG:boto:Data: 2023-07-29T00:50:04.801 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.801 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:04.802 DEBUG:boto:Port: 8001 2023-07-29T00:50:04.802 DEBUG:boto:Params: {} 2023-07-29T00:50:04.802 DEBUG:boto:Token: None 2023-07-29T00:50:04.803 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj68 2023-07-29T00:50:04.803 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:NOvDf0BlBjCbLcpWqTiJhZcA/Uo= 2023-07-29T00:50:04.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:NOvDf0BlBjCbLcpWqTiJhZcA/Uo=', 'Content-Length': '0'} 2023-07-29T00:50:04.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000687e77d46be2145a-0064c4623c-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.806 DEBUG:boto:path=/hiiqos-61/obj68 2023-07-29T00:50:04.806 DEBUG:boto:auth_path=/hiiqos-61/obj68 2023-07-29T00:50:04.807 DEBUG:boto:Method: HEAD 2023-07-29T00:50:04.807 DEBUG:boto:Path: /hiiqos-61/obj68 2023-07-29T00:50:04.807 DEBUG:boto:Data: 2023-07-29T00:50:04.808 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.808 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:04.808 DEBUG:boto:Port: 8000 2023-07-29T00:50:04.809 DEBUG:boto:Params: {} 2023-07-29T00:50:04.809 DEBUG:boto:Token: None 2023-07-29T00:50:04.809 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj68 2023-07-29T00:50:04.810 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:NOvDf0BlBjCbLcpWqTiJhZcA/Uo= 2023-07-29T00:50:04.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:NOvDf0BlBjCbLcpWqTiJhZcA/Uo=', 'Content-Length': '0'} 2023-07-29T00:50:04.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000892867d652d2de16-0064c4623c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-07-29T00:50:04.813 DEBUG:boto:path=/hiiqos-61/obj68 2023-07-29T00:50:04.813 DEBUG:boto:auth_path=/hiiqos-61/obj68 2023-07-29T00:50:04.813 DEBUG:boto:Method: GET 2023-07-29T00:50:04.814 DEBUG:boto:Path: /hiiqos-61/obj68 2023-07-29T00:50:04.814 DEBUG:boto:Data: 2023-07-29T00:50:04.814 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.815 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:04.815 DEBUG:boto:Port: 8001 2023-07-29T00:50:04.815 DEBUG:boto:Params: {} 2023-07-29T00:50:04.816 DEBUG:boto:Token: None 2023-07-29T00:50:04.816 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj68 2023-07-29T00:50:04.817 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I88Ft24yUw3tcxyi494WW/MHGIE= 2023-07-29T00:50:04.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I88Ft24yUw3tcxyi494WW/MHGIE=', 'Content-Length': '0'} 2023-07-29T00:50:04.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b76a65aa76e0bb8-0064c4623c-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.819 DEBUG:boto:path=/hiiqos-61/obj68 2023-07-29T00:50:04.820 DEBUG:boto:auth_path=/hiiqos-61/obj68 2023-07-29T00:50:04.820 DEBUG:boto:Method: GET 2023-07-29T00:50:04.820 DEBUG:boto:Path: /hiiqos-61/obj68 2023-07-29T00:50:04.821 DEBUG:boto:Data: 2023-07-29T00:50:04.821 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.821 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:04.822 DEBUG:boto:Port: 8000 2023-07-29T00:50:04.822 DEBUG:boto:Params: {} 2023-07-29T00:50:04.823 DEBUG:boto:Token: None 2023-07-29T00:50:04.823 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj68 2023-07-29T00:50:04.823 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I88Ft24yUw3tcxyi494WW/MHGIE= 2023-07-29T00:50:04.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I88Ft24yUw3tcxyi494WW/MHGIE=', 'Content-Length': '0'} 2023-07-29T00:50:04.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a294cfac254bbfac-0064c4623c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.826 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:04.827 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:04.827 DEBUG:boto:path=/hiiqos-61/?prefix=obj68 2023-07-29T00:50:04.827 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj68 2023-07-29T00:50:04.828 DEBUG:boto:Method: GET 2023-07-29T00:50:04.828 DEBUG:boto:Path: /hiiqos-61/?prefix=obj68 2023-07-29T00:50:04.828 DEBUG:boto:Data: 2023-07-29T00:50:04.829 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.829 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:04.829 DEBUG:boto:Port: 8001 2023-07-29T00:50:04.830 DEBUG:boto:Params: {} 2023-07-29T00:50:04.830 DEBUG:boto:Token: None 2023-07-29T00:50:04.831 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/ 2023-07-29T00:50:04.831 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g= 2023-07-29T00:50:04.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g=', 'Content-Length': '0'} 2023-07-29T00:50:04.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fabf51fa6249d63-0064c4623c-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.834 DEBUG:boto:b'hiiqos-61obj681000falseobj682023-07-29T00:45:01.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:04.835 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:04.835 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:04.835 DEBUG:boto:path=/hiiqos-61/?prefix=obj68 2023-07-29T00:50:04.836 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj68 2023-07-29T00:50:04.836 DEBUG:boto:Method: GET 2023-07-29T00:50:04.837 DEBUG:boto:Path: /hiiqos-61/?prefix=obj68 2023-07-29T00:50:04.837 DEBUG:boto:Data: 2023-07-29T00:50:04.837 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.838 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:04.838 DEBUG:boto:Port: 8000 2023-07-29T00:50:04.838 DEBUG:boto:Params: {} 2023-07-29T00:50:04.839 DEBUG:boto:Token: None 2023-07-29T00:50:04.839 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/ 2023-07-29T00:50:04.840 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g= 2023-07-29T00:50:04.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g=', 'Content-Length': '0'} 2023-07-29T00:50:04.844 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004677d73731c6a9c6-0064c4623c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.845 DEBUG:boto:b'hiiqos-61obj681000falseobj682023-07-29T00:45:01.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:04.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-07-29T00:50:04.846 DEBUG:boto:path=/hiiqos-61/obj69 2023-07-29T00:50:04.846 DEBUG:boto:auth_path=/hiiqos-61/obj69 2023-07-29T00:50:04.846 DEBUG:boto:path=/hiiqos-61/obj69?versionId=null 2023-07-29T00:50:04.847 DEBUG:boto:auth_path=/hiiqos-61/obj69?versionId=null 2023-07-29T00:50:04.847 DEBUG:boto:Method: GET 2023-07-29T00:50:04.847 DEBUG:boto:Path: /hiiqos-61/obj69?versionId=null 2023-07-29T00:50:04.848 DEBUG:boto:Data: 2023-07-29T00:50:04.848 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.849 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:04.849 DEBUG:boto:Port: 8001 2023-07-29T00:50:04.849 DEBUG:boto:Params: {} 2023-07-29T00:50:04.850 DEBUG:boto:Token: None 2023-07-29T00:50:04.850 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj69?versionId=null 2023-07-29T00:50:04.850 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:93mWELJzgX/hH0kcG5kCEKCeIL4= 2023-07-29T00:50:04.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:93mWELJzgX/hH0kcG5kCEKCeIL4=', 'Content-Length': '0'} 2023-07-29T00:50:04.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040ec0c510e8a4b9a-0064c4623c-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.853 DEBUG:boto:path=/hiiqos-61/obj69 2023-07-29T00:50:04.854 DEBUG:boto:auth_path=/hiiqos-61/obj69 2023-07-29T00:50:04.854 DEBUG:boto:path=/hiiqos-61/obj69?versionId=null 2023-07-29T00:50:04.855 DEBUG:boto:auth_path=/hiiqos-61/obj69?versionId=null 2023-07-29T00:50:04.855 DEBUG:boto:Method: GET 2023-07-29T00:50:04.855 DEBUG:boto:Path: /hiiqos-61/obj69?versionId=null 2023-07-29T00:50:04.856 DEBUG:boto:Data: 2023-07-29T00:50:04.856 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.856 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:04.857 DEBUG:boto:Port: 8000 2023-07-29T00:50:04.857 DEBUG:boto:Params: {} 2023-07-29T00:50:04.858 DEBUG:boto:Token: None 2023-07-29T00:50:04.858 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj69?versionId=null 2023-07-29T00:50:04.858 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:93mWELJzgX/hH0kcG5kCEKCeIL4= 2023-07-29T00:50:04.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:93mWELJzgX/hH0kcG5kCEKCeIL4=', 'Content-Length': '0'} 2023-07-29T00:50:04.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000872597324d3ea09e-0064c4623c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.904 DEBUG:boto:path=/hiiqos-61/obj69 2023-07-29T00:50:04.905 DEBUG:boto:auth_path=/hiiqos-61/obj69 2023-07-29T00:50:04.905 DEBUG:boto:path=/hiiqos-61/obj69?versionId=null 2023-07-29T00:50:04.906 DEBUG:boto:auth_path=/hiiqos-61/obj69?versionId=null 2023-07-29T00:50:04.906 DEBUG:boto:Method: HEAD 2023-07-29T00:50:04.906 DEBUG:boto:Path: /hiiqos-61/obj69?versionId=null 2023-07-29T00:50:04.907 DEBUG:boto:Data: 2023-07-29T00:50:04.907 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.907 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:04.908 DEBUG:boto:Port: 8001 2023-07-29T00:50:04.908 DEBUG:boto:Params: {} 2023-07-29T00:50:04.908 DEBUG:boto:Token: None 2023-07-29T00:50:04.909 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj69?versionId=null 2023-07-29T00:50:04.909 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yVF6OS5lw839KwB7FMG45Oupp4I= 2023-07-29T00:50:04.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yVF6OS5lw839KwB7FMG45Oupp4I=', 'Content-Length': '0'} 2023-07-29T00:50:04.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6ca271da87725e4-0064c4623c-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.913 DEBUG:boto:path=/hiiqos-61/obj69 2023-07-29T00:50:04.913 DEBUG:boto:auth_path=/hiiqos-61/obj69 2023-07-29T00:50:04.913 DEBUG:boto:path=/hiiqos-61/obj69?versionId=null 2023-07-29T00:50:04.914 DEBUG:boto:auth_path=/hiiqos-61/obj69?versionId=null 2023-07-29T00:50:04.914 DEBUG:boto:Method: HEAD 2023-07-29T00:50:04.914 DEBUG:boto:Path: /hiiqos-61/obj69?versionId=null 2023-07-29T00:50:04.915 DEBUG:boto:Data: 2023-07-29T00:50:04.915 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.915 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:04.916 DEBUG:boto:Port: 8000 2023-07-29T00:50:04.916 DEBUG:boto:Params: {} 2023-07-29T00:50:04.916 DEBUG:boto:Token: None 2023-07-29T00:50:04.917 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj69?versionId=null 2023-07-29T00:50:04.917 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yVF6OS5lw839KwB7FMG45Oupp4I= 2023-07-29T00:50:04.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yVF6OS5lw839KwB7FMG45Oupp4I=', 'Content-Length': '0'} 2023-07-29T00:50:04.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa6673acaed17156-0064c4623c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-07-29T00:50:04.920 DEBUG:boto:path=/hiiqos-61/obj69 2023-07-29T00:50:04.921 DEBUG:boto:auth_path=/hiiqos-61/obj69 2023-07-29T00:50:04.921 DEBUG:boto:path=/hiiqos-61/obj69?versionId=null 2023-07-29T00:50:04.921 DEBUG:boto:auth_path=/hiiqos-61/obj69?versionId=null 2023-07-29T00:50:04.922 DEBUG:boto:Method: GET 2023-07-29T00:50:04.922 DEBUG:boto:Path: /hiiqos-61/obj69?versionId=null 2023-07-29T00:50:04.923 DEBUG:boto:Data: 2023-07-29T00:50:04.923 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.923 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:04.924 DEBUG:boto:Port: 8001 2023-07-29T00:50:04.924 DEBUG:boto:Params: {} 2023-07-29T00:50:04.925 DEBUG:boto:Token: None 2023-07-29T00:50:04.925 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj69?versionId=null 2023-07-29T00:50:04.926 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:93mWELJzgX/hH0kcG5kCEKCeIL4= 2023-07-29T00:50:04.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:93mWELJzgX/hH0kcG5kCEKCeIL4=', 'Content-Length': '0'} 2023-07-29T00:50:04.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097fc0cd969716acf-0064c4623c-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.928 DEBUG:boto:path=/hiiqos-61/obj69 2023-07-29T00:50:04.929 DEBUG:boto:auth_path=/hiiqos-61/obj69 2023-07-29T00:50:04.929 DEBUG:boto:path=/hiiqos-61/obj69?versionId=null 2023-07-29T00:50:04.929 DEBUG:boto:auth_path=/hiiqos-61/obj69?versionId=null 2023-07-29T00:50:04.930 DEBUG:boto:Method: GET 2023-07-29T00:50:04.930 DEBUG:boto:Path: /hiiqos-61/obj69?versionId=null 2023-07-29T00:50:04.931 DEBUG:boto:Data: 2023-07-29T00:50:04.931 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.931 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:04.932 DEBUG:boto:Port: 8000 2023-07-29T00:50:04.932 DEBUG:boto:Params: {} 2023-07-29T00:50:04.932 DEBUG:boto:Token: None 2023-07-29T00:50:04.933 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj69?versionId=null 2023-07-29T00:50:04.933 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:93mWELJzgX/hH0kcG5kCEKCeIL4= 2023-07-29T00:50:04.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:93mWELJzgX/hH0kcG5kCEKCeIL4=', 'Content-Length': '0'} 2023-07-29T00:50:04.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d24a8450eb548ab-0064c4623c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.938 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:04.939 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:04.940 DEBUG:boto:path=/hiiqos-61/?prefix=obj69 2023-07-29T00:50:04.941 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj69 2023-07-29T00:50:04.941 DEBUG:boto:Method: GET 2023-07-29T00:50:04.942 DEBUG:boto:Path: /hiiqos-61/?prefix=obj69 2023-07-29T00:50:04.943 DEBUG:boto:Data: 2023-07-29T00:50:04.943 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.944 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:04.945 DEBUG:boto:Port: 8001 2023-07-29T00:50:04.946 DEBUG:boto:Params: {} 2023-07-29T00:50:04.946 DEBUG:boto:Token: None 2023-07-29T00:50:04.947 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/ 2023-07-29T00:50:04.947 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g= 2023-07-29T00:50:04.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g=', 'Content-Length': '0'} 2023-07-29T00:50:04.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe571197932fb4f6-0064c4623c-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.950 DEBUG:boto:b'hiiqos-61obj691000falseobj692023-07-29T00:45:02.020Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:04.951 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:04.952 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:04.952 DEBUG:boto:path=/hiiqos-61/?prefix=obj69 2023-07-29T00:50:04.953 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj69 2023-07-29T00:50:04.953 DEBUG:boto:Method: GET 2023-07-29T00:50:04.953 DEBUG:boto:Path: /hiiqos-61/?prefix=obj69 2023-07-29T00:50:04.954 DEBUG:boto:Data: 2023-07-29T00:50:04.954 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.954 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:04.955 DEBUG:boto:Port: 8000 2023-07-29T00:50:04.955 DEBUG:boto:Params: {} 2023-07-29T00:50:04.956 DEBUG:boto:Token: None 2023-07-29T00:50:04.956 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/ 2023-07-29T00:50:04.957 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g= 2023-07-29T00:50:04.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I5Qhtr/bBVwsniqtGjITW51Tx6g=', 'Content-Length': '0'} 2023-07-29T00:50:04.961 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000944cd9aa319b8e72-0064c4623c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.961 DEBUG:boto:b'hiiqos-61obj691000falseobj692023-07-29T00:45:02.020Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:04.962 DEBUG:boto:path=/hiiqos-61/obj69 2023-07-29T00:50:04.962 DEBUG:boto:auth_path=/hiiqos-61/obj69 2023-07-29T00:50:04.963 DEBUG:boto:Method: HEAD 2023-07-29T00:50:04.963 DEBUG:boto:Path: /hiiqos-61/obj69 2023-07-29T00:50:04.964 DEBUG:boto:Data: 2023-07-29T00:50:04.964 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.964 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:04.965 DEBUG:boto:Port: 8001 2023-07-29T00:50:04.965 DEBUG:boto:Params: {} 2023-07-29T00:50:04.965 DEBUG:boto:Token: None 2023-07-29T00:50:04.966 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj69 2023-07-29T00:50:04.966 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:N87mgCqedXDf8Z8Y6VFpyTw8KvM= 2023-07-29T00:50:04.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:N87mgCqedXDf8Z8Y6VFpyTw8KvM=', 'Content-Length': '0'} 2023-07-29T00:50:04.968 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069a7bca2e005a4bc-0064c4623c-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.969 DEBUG:boto:path=/hiiqos-61/obj69 2023-07-29T00:50:04.969 DEBUG:boto:auth_path=/hiiqos-61/obj69 2023-07-29T00:50:04.970 DEBUG:boto:Method: HEAD 2023-07-29T00:50:04.970 DEBUG:boto:Path: /hiiqos-61/obj69 2023-07-29T00:50:04.970 DEBUG:boto:Data: 2023-07-29T00:50:04.971 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.971 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:04.971 DEBUG:boto:Port: 8000 2023-07-29T00:50:04.972 DEBUG:boto:Params: {} 2023-07-29T00:50:04.972 DEBUG:boto:Token: None 2023-07-29T00:50:04.972 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj69 2023-07-29T00:50:04.973 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:N87mgCqedXDf8Z8Y6VFpyTw8KvM= 2023-07-29T00:50:04.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:N87mgCqedXDf8Z8Y6VFpyTw8KvM=', 'Content-Length': '0'} 2023-07-29T00:50:04.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004441465921e1bdd-0064c4623c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:04.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-07-29T00:50:04.976 DEBUG:boto:path=/hiiqos-61/obj69 2023-07-29T00:50:04.976 DEBUG:boto:auth_path=/hiiqos-61/obj69 2023-07-29T00:50:04.977 DEBUG:boto:Method: GET 2023-07-29T00:50:04.977 DEBUG:boto:Path: /hiiqos-61/obj69 2023-07-29T00:50:04.977 DEBUG:boto:Data: 2023-07-29T00:50:04.978 DEBUG:boto:Headers: {} 2023-07-29T00:50:04.978 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:04.979 DEBUG:boto:Port: 8001 2023-07-29T00:50:04.979 DEBUG:boto:Params: {} 2023-07-29T00:50:04.979 DEBUG:boto:Token: None 2023-07-29T00:50:04.980 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:04 GMT /hiiqos-61/obj69 2023-07-29T00:50:04.980 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0WvuNqsFvZuc6EfTLKWdY1GW0+o= 2023-07-29T00:50:04.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:04 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0WvuNqsFvZuc6EfTLKWdY1GW0+o=', 'Content-Length': '0'} 2023-07-29T00:50:04.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000775175f92cec8941-0064c4623c-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.024 DEBUG:boto:path=/hiiqos-61/obj69 2023-07-29T00:50:05.025 DEBUG:boto:auth_path=/hiiqos-61/obj69 2023-07-29T00:50:05.025 DEBUG:boto:Method: GET 2023-07-29T00:50:05.026 DEBUG:boto:Path: /hiiqos-61/obj69 2023-07-29T00:50:05.026 DEBUG:boto:Data: 2023-07-29T00:50:05.027 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.027 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:05.027 DEBUG:boto:Port: 8000 2023-07-29T00:50:05.028 DEBUG:boto:Params: {} 2023-07-29T00:50:05.028 DEBUG:boto:Token: None 2023-07-29T00:50:05.028 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj69 2023-07-29T00:50:05.029 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:C2/e1EzMF4sJpTClkxx3T9ljhVw= 2023-07-29T00:50:05.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:C2/e1EzMF4sJpTClkxx3T9ljhVw=', 'Content-Length': '0'} 2023-07-29T00:50:05.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7fff5498a843d42-0064c4623c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.032 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:05.032 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:05.033 DEBUG:boto:path=/hiiqos-61/?prefix=obj69 2023-07-29T00:50:05.033 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj69 2023-07-29T00:50:05.033 DEBUG:boto:Method: GET 2023-07-29T00:50:05.034 DEBUG:boto:Path: /hiiqos-61/?prefix=obj69 2023-07-29T00:50:05.034 DEBUG:boto:Data: 2023-07-29T00:50:05.034 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.035 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:05.035 DEBUG:boto:Port: 8001 2023-07-29T00:50:05.035 DEBUG:boto:Params: {} 2023-07-29T00:50:05.036 DEBUG:boto:Token: None 2023-07-29T00:50:05.036 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/ 2023-07-29T00:50:05.037 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU= 2023-07-29T00:50:05.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU=', 'Content-Length': '0'} 2023-07-29T00:50:05.039 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000571609daf36b4a3a-0064c4623c-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.040 DEBUG:boto:b'hiiqos-61obj691000falseobj692023-07-29T00:45:02.020Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:05.041 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:05.041 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:05.041 DEBUG:boto:path=/hiiqos-61/?prefix=obj69 2023-07-29T00:50:05.042 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj69 2023-07-29T00:50:05.042 DEBUG:boto:Method: GET 2023-07-29T00:50:05.042 DEBUG:boto:Path: /hiiqos-61/?prefix=obj69 2023-07-29T00:50:05.043 DEBUG:boto:Data: 2023-07-29T00:50:05.043 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.043 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:05.044 DEBUG:boto:Port: 8000 2023-07-29T00:50:05.044 DEBUG:boto:Params: {} 2023-07-29T00:50:05.045 DEBUG:boto:Token: None 2023-07-29T00:50:05.045 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/ 2023-07-29T00:50:05.045 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU= 2023-07-29T00:50:05.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU=', 'Content-Length': '0'} 2023-07-29T00:50:05.049 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000025cfc3543f04a64-0064c4623c-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.050 DEBUG:boto:b'hiiqos-61obj691000falseobj692023-07-29T00:45:02.020Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:05.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-07-29T00:50:05.051 DEBUG:boto:path=/hiiqos-61/obj74 2023-07-29T00:50:05.051 DEBUG:boto:auth_path=/hiiqos-61/obj74 2023-07-29T00:50:05.052 DEBUG:boto:path=/hiiqos-61/obj74?versionId=null 2023-07-29T00:50:05.052 DEBUG:boto:auth_path=/hiiqos-61/obj74?versionId=null 2023-07-29T00:50:05.052 DEBUG:boto:Method: GET 2023-07-29T00:50:05.053 DEBUG:boto:Path: /hiiqos-61/obj74?versionId=null 2023-07-29T00:50:05.053 DEBUG:boto:Data: 2023-07-29T00:50:05.053 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.054 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:05.054 DEBUG:boto:Port: 8001 2023-07-29T00:50:05.055 DEBUG:boto:Params: {} 2023-07-29T00:50:05.055 DEBUG:boto:Token: None 2023-07-29T00:50:05.055 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj74?versionId=null 2023-07-29T00:50:05.056 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QCvQa/pAKy/rXrlVsG6aztGJpm4= 2023-07-29T00:50:05.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QCvQa/pAKy/rXrlVsG6aztGJpm4=', 'Content-Length': '0'} 2023-07-29T00:50:05.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c057a58e733544e-0064c4623c-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.059 DEBUG:boto:path=/hiiqos-61/obj74 2023-07-29T00:50:05.059 DEBUG:boto:auth_path=/hiiqos-61/obj74 2023-07-29T00:50:05.059 DEBUG:boto:path=/hiiqos-61/obj74?versionId=null 2023-07-29T00:50:05.060 DEBUG:boto:auth_path=/hiiqos-61/obj74?versionId=null 2023-07-29T00:50:05.060 DEBUG:boto:Method: GET 2023-07-29T00:50:05.061 DEBUG:boto:Path: /hiiqos-61/obj74?versionId=null 2023-07-29T00:50:05.061 DEBUG:boto:Data: 2023-07-29T00:50:05.061 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.062 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:05.062 DEBUG:boto:Port: 8000 2023-07-29T00:50:05.062 DEBUG:boto:Params: {} 2023-07-29T00:50:05.063 DEBUG:boto:Token: None 2023-07-29T00:50:05.063 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj74?versionId=null 2023-07-29T00:50:05.064 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QCvQa/pAKy/rXrlVsG6aztGJpm4= 2023-07-29T00:50:05.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QCvQa/pAKy/rXrlVsG6aztGJpm4=', 'Content-Length': '0'} 2023-07-29T00:50:05.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da456ea41fb3996e-0064c4623c-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:04 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.066 DEBUG:boto:path=/hiiqos-61/obj74 2023-07-29T00:50:05.067 DEBUG:boto:auth_path=/hiiqos-61/obj74 2023-07-29T00:50:05.067 DEBUG:boto:path=/hiiqos-61/obj74?versionId=null 2023-07-29T00:50:05.067 DEBUG:boto:auth_path=/hiiqos-61/obj74?versionId=null 2023-07-29T00:50:05.068 DEBUG:boto:Method: HEAD 2023-07-29T00:50:05.068 DEBUG:boto:Path: /hiiqos-61/obj74?versionId=null 2023-07-29T00:50:05.068 DEBUG:boto:Data: 2023-07-29T00:50:05.069 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.069 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:05.069 DEBUG:boto:Port: 8001 2023-07-29T00:50:05.070 DEBUG:boto:Params: {} 2023-07-29T00:50:05.070 DEBUG:boto:Token: None 2023-07-29T00:50:05.070 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj74?versionId=null 2023-07-29T00:50:05.071 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vaTbtXCadIAcsFv2MK/kBKpvk1E= 2023-07-29T00:50:05.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vaTbtXCadIAcsFv2MK/kBKpvk1E=', 'Content-Length': '0'} 2023-07-29T00:50:05.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d936c3124faa4a24-0064c4623d-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.073 DEBUG:boto:path=/hiiqos-61/obj74 2023-07-29T00:50:05.074 DEBUG:boto:auth_path=/hiiqos-61/obj74 2023-07-29T00:50:05.074 DEBUG:boto:path=/hiiqos-61/obj74?versionId=null 2023-07-29T00:50:05.074 DEBUG:boto:auth_path=/hiiqos-61/obj74?versionId=null 2023-07-29T00:50:05.075 DEBUG:boto:Method: HEAD 2023-07-29T00:50:05.075 DEBUG:boto:Path: /hiiqos-61/obj74?versionId=null 2023-07-29T00:50:05.075 DEBUG:boto:Data: 2023-07-29T00:50:05.076 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.076 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:05.076 DEBUG:boto:Port: 8000 2023-07-29T00:50:05.077 DEBUG:boto:Params: {} 2023-07-29T00:50:05.077 DEBUG:boto:Token: None 2023-07-29T00:50:05.078 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj74?versionId=null 2023-07-29T00:50:05.078 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vaTbtXCadIAcsFv2MK/kBKpvk1E= 2023-07-29T00:50:05.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vaTbtXCadIAcsFv2MK/kBKpvk1E=', 'Content-Length': '0'} 2023-07-29T00:50:05.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e63b54e1e999ed9f-0064c4623d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-07-29T00:50:05.081 DEBUG:boto:path=/hiiqos-61/obj74 2023-07-29T00:50:05.082 DEBUG:boto:auth_path=/hiiqos-61/obj74 2023-07-29T00:50:05.082 DEBUG:boto:path=/hiiqos-61/obj74?versionId=null 2023-07-29T00:50:05.083 DEBUG:boto:auth_path=/hiiqos-61/obj74?versionId=null 2023-07-29T00:50:05.083 DEBUG:boto:Method: GET 2023-07-29T00:50:05.083 DEBUG:boto:Path: /hiiqos-61/obj74?versionId=null 2023-07-29T00:50:05.084 DEBUG:boto:Data: 2023-07-29T00:50:05.084 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.085 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:05.085 DEBUG:boto:Port: 8001 2023-07-29T00:50:05.085 DEBUG:boto:Params: {} 2023-07-29T00:50:05.086 DEBUG:boto:Token: None 2023-07-29T00:50:05.086 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj74?versionId=null 2023-07-29T00:50:05.087 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QCvQa/pAKy/rXrlVsG6aztGJpm4= 2023-07-29T00:50:05.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QCvQa/pAKy/rXrlVsG6aztGJpm4=', 'Content-Length': '0'} 2023-07-29T00:50:05.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fa79d364be6a6f6-0064c4623d-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.090 DEBUG:boto:path=/hiiqos-61/obj74 2023-07-29T00:50:05.090 DEBUG:boto:auth_path=/hiiqos-61/obj74 2023-07-29T00:50:05.090 DEBUG:boto:path=/hiiqos-61/obj74?versionId=null 2023-07-29T00:50:05.091 DEBUG:boto:auth_path=/hiiqos-61/obj74?versionId=null 2023-07-29T00:50:05.091 DEBUG:boto:Method: GET 2023-07-29T00:50:05.091 DEBUG:boto:Path: /hiiqos-61/obj74?versionId=null 2023-07-29T00:50:05.092 DEBUG:boto:Data: 2023-07-29T00:50:05.092 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.092 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:05.093 DEBUG:boto:Port: 8000 2023-07-29T00:50:05.093 DEBUG:boto:Params: {} 2023-07-29T00:50:05.094 DEBUG:boto:Token: None 2023-07-29T00:50:05.094 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj74?versionId=null 2023-07-29T00:50:05.094 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QCvQa/pAKy/rXrlVsG6aztGJpm4= 2023-07-29T00:50:05.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QCvQa/pAKy/rXrlVsG6aztGJpm4=', 'Content-Length': '0'} 2023-07-29T00:50:05.097 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004400261a3b8b9ddc-0064c4623d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.097 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:05.098 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:05.098 DEBUG:boto:path=/hiiqos-61/?prefix=obj74 2023-07-29T00:50:05.098 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj74 2023-07-29T00:50:05.099 DEBUG:boto:Method: GET 2023-07-29T00:50:05.099 DEBUG:boto:Path: /hiiqos-61/?prefix=obj74 2023-07-29T00:50:05.099 DEBUG:boto:Data: 2023-07-29T00:50:05.100 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.100 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:05.101 DEBUG:boto:Port: 8001 2023-07-29T00:50:05.101 DEBUG:boto:Params: {} 2023-07-29T00:50:05.101 DEBUG:boto:Token: None 2023-07-29T00:50:05.102 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/ 2023-07-29T00:50:05.102 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU= 2023-07-29T00:50:05.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU=', 'Content-Length': '0'} 2023-07-29T00:50:05.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003dc358bf50e26881-0064c4623d-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.148 DEBUG:boto:b'hiiqos-61obj741000falseobj742023-07-29T00:45:03.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:05.150 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:05.151 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:05.152 DEBUG:boto:path=/hiiqos-61/?prefix=obj74 2023-07-29T00:50:05.152 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj74 2023-07-29T00:50:05.153 DEBUG:boto:Method: GET 2023-07-29T00:50:05.154 DEBUG:boto:Path: /hiiqos-61/?prefix=obj74 2023-07-29T00:50:05.154 DEBUG:boto:Data: 2023-07-29T00:50:05.155 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.155 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:05.156 DEBUG:boto:Port: 8000 2023-07-29T00:50:05.156 DEBUG:boto:Params: {} 2023-07-29T00:50:05.157 DEBUG:boto:Token: None 2023-07-29T00:50:05.157 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/ 2023-07-29T00:50:05.158 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU= 2023-07-29T00:50:05.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU=', 'Content-Length': '0'} 2023-07-29T00:50:05.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a77ba5c52f9472ed-0064c4623d-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.162 DEBUG:boto:b'hiiqos-61obj741000falseobj742023-07-29T00:45:03.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:05.163 DEBUG:boto:path=/hiiqos-61/obj74 2023-07-29T00:50:05.163 DEBUG:boto:auth_path=/hiiqos-61/obj74 2023-07-29T00:50:05.164 DEBUG:boto:Method: HEAD 2023-07-29T00:50:05.164 DEBUG:boto:Path: /hiiqos-61/obj74 2023-07-29T00:50:05.165 DEBUG:boto:Data: 2023-07-29T00:50:05.165 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.165 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:05.166 DEBUG:boto:Port: 8001 2023-07-29T00:50:05.166 DEBUG:boto:Params: {} 2023-07-29T00:50:05.166 DEBUG:boto:Token: None 2023-07-29T00:50:05.167 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj74 2023-07-29T00:50:05.167 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:r+WTQ1vEJoQ4GENk1if+dcRtCsg= 2023-07-29T00:50:05.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:r+WTQ1vEJoQ4GENk1if+dcRtCsg=', 'Content-Length': '0'} 2023-07-29T00:50:05.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb433fe78b26bcb0-0064c4623d-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.170 DEBUG:boto:path=/hiiqos-61/obj74 2023-07-29T00:50:05.170 DEBUG:boto:auth_path=/hiiqos-61/obj74 2023-07-29T00:50:05.171 DEBUG:boto:Method: HEAD 2023-07-29T00:50:05.171 DEBUG:boto:Path: /hiiqos-61/obj74 2023-07-29T00:50:05.171 DEBUG:boto:Data: 2023-07-29T00:50:05.172 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.173 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:05.173 DEBUG:boto:Port: 8000 2023-07-29T00:50:05.174 DEBUG:boto:Params: {} 2023-07-29T00:50:05.175 DEBUG:boto:Token: None 2023-07-29T00:50:05.176 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj74 2023-07-29T00:50:05.177 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:r+WTQ1vEJoQ4GENk1if+dcRtCsg= 2023-07-29T00:50:05.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:r+WTQ1vEJoQ4GENk1if+dcRtCsg=', 'Content-Length': '0'} 2023-07-29T00:50:05.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085b36c55f64fa78b-0064c4623d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-07-29T00:50:05.181 DEBUG:boto:path=/hiiqos-61/obj74 2023-07-29T00:50:05.181 DEBUG:boto:auth_path=/hiiqos-61/obj74 2023-07-29T00:50:05.181 DEBUG:boto:Method: GET 2023-07-29T00:50:05.182 DEBUG:boto:Path: /hiiqos-61/obj74 2023-07-29T00:50:05.182 DEBUG:boto:Data: 2023-07-29T00:50:05.182 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.183 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:05.183 DEBUG:boto:Port: 8001 2023-07-29T00:50:05.184 DEBUG:boto:Params: {} 2023-07-29T00:50:05.184 DEBUG:boto:Token: None 2023-07-29T00:50:05.184 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj74 2023-07-29T00:50:05.185 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:i+kJG/rHwk3muZ3NkJtJS5M1oKE= 2023-07-29T00:50:05.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:i+kJG/rHwk3muZ3NkJtJS5M1oKE=', 'Content-Length': '0'} 2023-07-29T00:50:05.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dd806e3cbfa62d2-0064c4623d-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.187 DEBUG:boto:path=/hiiqos-61/obj74 2023-07-29T00:50:05.188 DEBUG:boto:auth_path=/hiiqos-61/obj74 2023-07-29T00:50:05.188 DEBUG:boto:Method: GET 2023-07-29T00:50:05.189 DEBUG:boto:Path: /hiiqos-61/obj74 2023-07-29T00:50:05.189 DEBUG:boto:Data: 2023-07-29T00:50:05.189 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.190 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:05.190 DEBUG:boto:Port: 8000 2023-07-29T00:50:05.190 DEBUG:boto:Params: {} 2023-07-29T00:50:05.191 DEBUG:boto:Token: None 2023-07-29T00:50:05.191 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj74 2023-07-29T00:50:05.192 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:i+kJG/rHwk3muZ3NkJtJS5M1oKE= 2023-07-29T00:50:05.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:i+kJG/rHwk3muZ3NkJtJS5M1oKE=', 'Content-Length': '0'} 2023-07-29T00:50:05.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e865ae1d359b55a-0064c4623d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.236 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:05.237 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:05.237 DEBUG:boto:path=/hiiqos-61/?prefix=obj74 2023-07-29T00:50:05.238 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj74 2023-07-29T00:50:05.238 DEBUG:boto:Method: GET 2023-07-29T00:50:05.238 DEBUG:boto:Path: /hiiqos-61/?prefix=obj74 2023-07-29T00:50:05.239 DEBUG:boto:Data: 2023-07-29T00:50:05.239 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.239 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:05.240 DEBUG:boto:Port: 8001 2023-07-29T00:50:05.240 DEBUG:boto:Params: {} 2023-07-29T00:50:05.240 DEBUG:boto:Token: None 2023-07-29T00:50:05.241 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/ 2023-07-29T00:50:05.241 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU= 2023-07-29T00:50:05.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU=', 'Content-Length': '0'} 2023-07-29T00:50:05.244 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008764a17a21ba71af-0064c4623d-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.245 DEBUG:boto:b'hiiqos-61obj741000falseobj742023-07-29T00:45:03.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:05.245 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:05.246 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:05.246 DEBUG:boto:path=/hiiqos-61/?prefix=obj74 2023-07-29T00:50:05.246 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj74 2023-07-29T00:50:05.247 DEBUG:boto:Method: GET 2023-07-29T00:50:05.247 DEBUG:boto:Path: /hiiqos-61/?prefix=obj74 2023-07-29T00:50:05.247 DEBUG:boto:Data: 2023-07-29T00:50:05.248 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.248 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:05.249 DEBUG:boto:Port: 8000 2023-07-29T00:50:05.249 DEBUG:boto:Params: {} 2023-07-29T00:50:05.249 DEBUG:boto:Token: None 2023-07-29T00:50:05.250 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/ 2023-07-29T00:50:05.250 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU= 2023-07-29T00:50:05.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU=', 'Content-Length': '0'} 2023-07-29T00:50:05.254 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f6dd331c8d20457-0064c4623d-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.255 DEBUG:boto:b'hiiqos-61obj741000falseobj742023-07-29T00:45:03.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:05.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-07-29T00:50:05.256 DEBUG:boto:path=/hiiqos-61/obj76 2023-07-29T00:50:05.256 DEBUG:boto:auth_path=/hiiqos-61/obj76 2023-07-29T00:50:05.257 DEBUG:boto:path=/hiiqos-61/obj76?versionId=null 2023-07-29T00:50:05.257 DEBUG:boto:auth_path=/hiiqos-61/obj76?versionId=null 2023-07-29T00:50:05.257 DEBUG:boto:Method: GET 2023-07-29T00:50:05.258 DEBUG:boto:Path: /hiiqos-61/obj76?versionId=null 2023-07-29T00:50:05.258 DEBUG:boto:Data: 2023-07-29T00:50:05.258 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.259 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:05.259 DEBUG:boto:Port: 8001 2023-07-29T00:50:05.259 DEBUG:boto:Params: {} 2023-07-29T00:50:05.260 DEBUG:boto:Token: None 2023-07-29T00:50:05.260 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj76?versionId=null 2023-07-29T00:50:05.261 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CSJoXyWzV/by0OfINPu8RJIvfxg= 2023-07-29T00:50:05.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CSJoXyWzV/by0OfINPu8RJIvfxg=', 'Content-Length': '0'} 2023-07-29T00:50:05.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032a5aa75ea1e5f5b-0064c4623d-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.263 DEBUG:boto:path=/hiiqos-61/obj76 2023-07-29T00:50:05.264 DEBUG:boto:auth_path=/hiiqos-61/obj76 2023-07-29T00:50:05.264 DEBUG:boto:path=/hiiqos-61/obj76?versionId=null 2023-07-29T00:50:05.264 DEBUG:boto:auth_path=/hiiqos-61/obj76?versionId=null 2023-07-29T00:50:05.265 DEBUG:boto:Method: GET 2023-07-29T00:50:05.265 DEBUG:boto:Path: /hiiqos-61/obj76?versionId=null 2023-07-29T00:50:05.266 DEBUG:boto:Data: 2023-07-29T00:50:05.266 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.266 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:05.267 DEBUG:boto:Port: 8000 2023-07-29T00:50:05.267 DEBUG:boto:Params: {} 2023-07-29T00:50:05.267 DEBUG:boto:Token: None 2023-07-29T00:50:05.268 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj76?versionId=null 2023-07-29T00:50:05.268 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CSJoXyWzV/by0OfINPu8RJIvfxg= 2023-07-29T00:50:05.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CSJoXyWzV/by0OfINPu8RJIvfxg=', 'Content-Length': '0'} 2023-07-29T00:50:05.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008bfdedab6808548-0064c4623d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.271 DEBUG:boto:path=/hiiqos-61/obj76 2023-07-29T00:50:05.271 DEBUG:boto:auth_path=/hiiqos-61/obj76 2023-07-29T00:50:05.272 DEBUG:boto:path=/hiiqos-61/obj76?versionId=null 2023-07-29T00:50:05.272 DEBUG:boto:auth_path=/hiiqos-61/obj76?versionId=null 2023-07-29T00:50:05.272 DEBUG:boto:Method: HEAD 2023-07-29T00:50:05.273 DEBUG:boto:Path: /hiiqos-61/obj76?versionId=null 2023-07-29T00:50:05.273 DEBUG:boto:Data: 2023-07-29T00:50:05.273 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.274 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:05.274 DEBUG:boto:Port: 8001 2023-07-29T00:50:05.274 DEBUG:boto:Params: {} 2023-07-29T00:50:05.275 DEBUG:boto:Token: None 2023-07-29T00:50:05.275 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj76?versionId=null 2023-07-29T00:50:05.276 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QkbqIz2l7cIEEXczwl8FYm0DvP8= 2023-07-29T00:50:05.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QkbqIz2l7cIEEXczwl8FYm0DvP8=', 'Content-Length': '0'} 2023-07-29T00:50:05.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae1321cf1bd77294-0064c4623d-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.278 DEBUG:boto:path=/hiiqos-61/obj76 2023-07-29T00:50:05.278 DEBUG:boto:auth_path=/hiiqos-61/obj76 2023-07-29T00:50:05.279 DEBUG:boto:path=/hiiqos-61/obj76?versionId=null 2023-07-29T00:50:05.279 DEBUG:boto:auth_path=/hiiqos-61/obj76?versionId=null 2023-07-29T00:50:05.279 DEBUG:boto:Method: HEAD 2023-07-29T00:50:05.280 DEBUG:boto:Path: /hiiqos-61/obj76?versionId=null 2023-07-29T00:50:05.280 DEBUG:boto:Data: 2023-07-29T00:50:05.280 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.281 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:05.281 DEBUG:boto:Port: 8000 2023-07-29T00:50:05.281 DEBUG:boto:Params: {} 2023-07-29T00:50:05.282 DEBUG:boto:Token: None 2023-07-29T00:50:05.282 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj76?versionId=null 2023-07-29T00:50:05.282 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QkbqIz2l7cIEEXczwl8FYm0DvP8= 2023-07-29T00:50:05.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QkbqIz2l7cIEEXczwl8FYm0DvP8=', 'Content-Length': '0'} 2023-07-29T00:50:05.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000747571acc32022b5-0064c4623d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-07-29T00:50:05.285 DEBUG:boto:path=/hiiqos-61/obj76 2023-07-29T00:50:05.285 DEBUG:boto:auth_path=/hiiqos-61/obj76 2023-07-29T00:50:05.286 DEBUG:boto:path=/hiiqos-61/obj76?versionId=null 2023-07-29T00:50:05.286 DEBUG:boto:auth_path=/hiiqos-61/obj76?versionId=null 2023-07-29T00:50:05.287 DEBUG:boto:Method: GET 2023-07-29T00:50:05.287 DEBUG:boto:Path: /hiiqos-61/obj76?versionId=null 2023-07-29T00:50:05.287 DEBUG:boto:Data: 2023-07-29T00:50:05.288 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.288 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:05.288 DEBUG:boto:Port: 8001 2023-07-29T00:50:05.289 DEBUG:boto:Params: {} 2023-07-29T00:50:05.289 DEBUG:boto:Token: None 2023-07-29T00:50:05.289 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj76?versionId=null 2023-07-29T00:50:05.290 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CSJoXyWzV/by0OfINPu8RJIvfxg= 2023-07-29T00:50:05.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CSJoXyWzV/by0OfINPu8RJIvfxg=', 'Content-Length': '0'} 2023-07-29T00:50:05.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003930c7647e22ce82-0064c4623d-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.293 DEBUG:boto:path=/hiiqos-61/obj76 2023-07-29T00:50:05.293 DEBUG:boto:auth_path=/hiiqos-61/obj76 2023-07-29T00:50:05.293 DEBUG:boto:path=/hiiqos-61/obj76?versionId=null 2023-07-29T00:50:05.294 DEBUG:boto:auth_path=/hiiqos-61/obj76?versionId=null 2023-07-29T00:50:05.294 DEBUG:boto:Method: GET 2023-07-29T00:50:05.294 DEBUG:boto:Path: /hiiqos-61/obj76?versionId=null 2023-07-29T00:50:05.295 DEBUG:boto:Data: 2023-07-29T00:50:05.295 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.296 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:05.296 DEBUG:boto:Port: 8000 2023-07-29T00:50:05.296 DEBUG:boto:Params: {} 2023-07-29T00:50:05.297 DEBUG:boto:Token: None 2023-07-29T00:50:05.297 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj76?versionId=null 2023-07-29T00:50:05.298 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CSJoXyWzV/by0OfINPu8RJIvfxg= 2023-07-29T00:50:05.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CSJoXyWzV/by0OfINPu8RJIvfxg=', 'Content-Length': '0'} 2023-07-29T00:50:05.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000426d1f740aa4be43-0064c4623d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.340 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:05.341 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:05.341 DEBUG:boto:path=/hiiqos-61/?prefix=obj76 2023-07-29T00:50:05.342 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj76 2023-07-29T00:50:05.342 DEBUG:boto:Method: GET 2023-07-29T00:50:05.342 DEBUG:boto:Path: /hiiqos-61/?prefix=obj76 2023-07-29T00:50:05.343 DEBUG:boto:Data: 2023-07-29T00:50:05.343 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.343 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:05.344 DEBUG:boto:Port: 8001 2023-07-29T00:50:05.344 DEBUG:boto:Params: {} 2023-07-29T00:50:05.345 DEBUG:boto:Token: None 2023-07-29T00:50:05.345 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/ 2023-07-29T00:50:05.345 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU= 2023-07-29T00:50:05.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU=', 'Content-Length': '0'} 2023-07-29T00:50:05.348 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007358daa3cf92ddf7-0064c4623d-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.349 DEBUG:boto:b'hiiqos-61obj761000falseobj762023-07-29T00:45:01.763Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:05.349 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:05.350 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:05.350 DEBUG:boto:path=/hiiqos-61/?prefix=obj76 2023-07-29T00:50:05.350 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj76 2023-07-29T00:50:05.351 DEBUG:boto:Method: GET 2023-07-29T00:50:05.351 DEBUG:boto:Path: /hiiqos-61/?prefix=obj76 2023-07-29T00:50:05.351 DEBUG:boto:Data: 2023-07-29T00:50:05.352 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.352 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:05.352 DEBUG:boto:Port: 8000 2023-07-29T00:50:05.353 DEBUG:boto:Params: {} 2023-07-29T00:50:05.353 DEBUG:boto:Token: None 2023-07-29T00:50:05.354 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/ 2023-07-29T00:50:05.354 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU= 2023-07-29T00:50:05.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU=', 'Content-Length': '0'} 2023-07-29T00:50:05.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9b3496a6c6783f4-0064c4623d-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.358 DEBUG:boto:b'hiiqos-61obj761000falseobj762023-07-29T00:45:01.763Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:05.359 DEBUG:boto:path=/hiiqos-61/obj76 2023-07-29T00:50:05.359 DEBUG:boto:auth_path=/hiiqos-61/obj76 2023-07-29T00:50:05.360 DEBUG:boto:Method: HEAD 2023-07-29T00:50:05.360 DEBUG:boto:Path: /hiiqos-61/obj76 2023-07-29T00:50:05.360 DEBUG:boto:Data: 2023-07-29T00:50:05.361 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.361 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:05.361 DEBUG:boto:Port: 8001 2023-07-29T00:50:05.362 DEBUG:boto:Params: {} 2023-07-29T00:50:05.362 DEBUG:boto:Token: None 2023-07-29T00:50:05.362 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj76 2023-07-29T00:50:05.363 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/MQThqcCa0rJQAfvncsVpubVDPQ= 2023-07-29T00:50:05.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/MQThqcCa0rJQAfvncsVpubVDPQ=', 'Content-Length': '0'} 2023-07-29T00:50:05.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000542a32196bd6e4cc-0064c4623d-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.365 DEBUG:boto:path=/hiiqos-61/obj76 2023-07-29T00:50:05.366 DEBUG:boto:auth_path=/hiiqos-61/obj76 2023-07-29T00:50:05.366 DEBUG:boto:Method: HEAD 2023-07-29T00:50:05.367 DEBUG:boto:Path: /hiiqos-61/obj76 2023-07-29T00:50:05.367 DEBUG:boto:Data: 2023-07-29T00:50:05.368 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.369 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:05.369 DEBUG:boto:Port: 8000 2023-07-29T00:50:05.370 DEBUG:boto:Params: {} 2023-07-29T00:50:05.371 DEBUG:boto:Token: None 2023-07-29T00:50:05.372 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj76 2023-07-29T00:50:05.373 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/MQThqcCa0rJQAfvncsVpubVDPQ= 2023-07-29T00:50:05.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/MQThqcCa0rJQAfvncsVpubVDPQ=', 'Content-Length': '0'} 2023-07-29T00:50:05.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002424f405db8e0275-0064c4623d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-07-29T00:50:05.376 DEBUG:boto:path=/hiiqos-61/obj76 2023-07-29T00:50:05.376 DEBUG:boto:auth_path=/hiiqos-61/obj76 2023-07-29T00:50:05.377 DEBUG:boto:Method: GET 2023-07-29T00:50:05.377 DEBUG:boto:Path: /hiiqos-61/obj76 2023-07-29T00:50:05.377 DEBUG:boto:Data: 2023-07-29T00:50:05.378 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.378 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:05.378 DEBUG:boto:Port: 8001 2023-07-29T00:50:05.379 DEBUG:boto:Params: {} 2023-07-29T00:50:05.379 DEBUG:boto:Token: None 2023-07-29T00:50:05.380 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj76 2023-07-29T00:50:05.380 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZnuUdW07ip/+SGP3fxDJWx2JKNk= 2023-07-29T00:50:05.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZnuUdW07ip/+SGP3fxDJWx2JKNk=', 'Content-Length': '0'} 2023-07-29T00:50:05.383 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000441a1e350d0326c5-0064c4623d-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.424 DEBUG:boto:path=/hiiqos-61/obj76 2023-07-29T00:50:05.425 DEBUG:boto:auth_path=/hiiqos-61/obj76 2023-07-29T00:50:05.425 DEBUG:boto:Method: GET 2023-07-29T00:50:05.426 DEBUG:boto:Path: /hiiqos-61/obj76 2023-07-29T00:50:05.426 DEBUG:boto:Data: 2023-07-29T00:50:05.426 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.427 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:05.427 DEBUG:boto:Port: 8000 2023-07-29T00:50:05.427 DEBUG:boto:Params: {} 2023-07-29T00:50:05.428 DEBUG:boto:Token: None 2023-07-29T00:50:05.428 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj76 2023-07-29T00:50:05.429 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZnuUdW07ip/+SGP3fxDJWx2JKNk= 2023-07-29T00:50:05.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZnuUdW07ip/+SGP3fxDJWx2JKNk=', 'Content-Length': '0'} 2023-07-29T00:50:05.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073cb75b7160e6aea-0064c4623d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.432 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:05.432 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:05.433 DEBUG:boto:path=/hiiqos-61/?prefix=obj76 2023-07-29T00:50:05.433 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj76 2023-07-29T00:50:05.433 DEBUG:boto:Method: GET 2023-07-29T00:50:05.434 DEBUG:boto:Path: /hiiqos-61/?prefix=obj76 2023-07-29T00:50:05.434 DEBUG:boto:Data: 2023-07-29T00:50:05.435 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.435 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:05.435 DEBUG:boto:Port: 8001 2023-07-29T00:50:05.436 DEBUG:boto:Params: {} 2023-07-29T00:50:05.436 DEBUG:boto:Token: None 2023-07-29T00:50:05.436 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/ 2023-07-29T00:50:05.437 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU= 2023-07-29T00:50:05.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU=', 'Content-Length': '0'} 2023-07-29T00:50:05.440 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016873812b4c84740-0064c4623d-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.440 DEBUG:boto:b'hiiqos-61obj761000falseobj762023-07-29T00:45:01.763Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:05.441 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:05.441 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:05.442 DEBUG:boto:path=/hiiqos-61/?prefix=obj76 2023-07-29T00:50:05.442 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj76 2023-07-29T00:50:05.442 DEBUG:boto:Method: GET 2023-07-29T00:50:05.443 DEBUG:boto:Path: /hiiqos-61/?prefix=obj76 2023-07-29T00:50:05.443 DEBUG:boto:Data: 2023-07-29T00:50:05.443 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.444 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:05.444 DEBUG:boto:Port: 8000 2023-07-29T00:50:05.444 DEBUG:boto:Params: {} 2023-07-29T00:50:05.445 DEBUG:boto:Token: None 2023-07-29T00:50:05.445 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/ 2023-07-29T00:50:05.446 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU= 2023-07-29T00:50:05.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU=', 'Content-Length': '0'} 2023-07-29T00:50:05.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000388ed8c47237f39c-0064c4623d-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.450 DEBUG:boto:b'hiiqos-61obj761000falseobj762023-07-29T00:45:01.763Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:05.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-07-29T00:50:05.451 DEBUG:boto:path=/hiiqos-61/obj85 2023-07-29T00:50:05.452 DEBUG:boto:auth_path=/hiiqos-61/obj85 2023-07-29T00:50:05.452 DEBUG:boto:path=/hiiqos-61/obj85?versionId=null 2023-07-29T00:50:05.452 DEBUG:boto:auth_path=/hiiqos-61/obj85?versionId=null 2023-07-29T00:50:05.453 DEBUG:boto:Method: GET 2023-07-29T00:50:05.453 DEBUG:boto:Path: /hiiqos-61/obj85?versionId=null 2023-07-29T00:50:05.453 DEBUG:boto:Data: 2023-07-29T00:50:05.454 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.454 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:05.455 DEBUG:boto:Port: 8001 2023-07-29T00:50:05.455 DEBUG:boto:Params: {} 2023-07-29T00:50:05.455 DEBUG:boto:Token: None 2023-07-29T00:50:05.456 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj85?versionId=null 2023-07-29T00:50:05.456 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9zgqcPc5qbaNxlzikRGRRMuSXlM= 2023-07-29T00:50:05.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9zgqcPc5qbaNxlzikRGRRMuSXlM=', 'Content-Length': '0'} 2023-07-29T00:50:05.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003979d40274b54e79-0064c4623d-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.459 DEBUG:boto:path=/hiiqos-61/obj85 2023-07-29T00:50:05.459 DEBUG:boto:auth_path=/hiiqos-61/obj85 2023-07-29T00:50:05.460 DEBUG:boto:path=/hiiqos-61/obj85?versionId=null 2023-07-29T00:50:05.460 DEBUG:boto:auth_path=/hiiqos-61/obj85?versionId=null 2023-07-29T00:50:05.460 DEBUG:boto:Method: GET 2023-07-29T00:50:05.461 DEBUG:boto:Path: /hiiqos-61/obj85?versionId=null 2023-07-29T00:50:05.461 DEBUG:boto:Data: 2023-07-29T00:50:05.462 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.462 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:05.462 DEBUG:boto:Port: 8000 2023-07-29T00:50:05.463 DEBUG:boto:Params: {} 2023-07-29T00:50:05.463 DEBUG:boto:Token: None 2023-07-29T00:50:05.463 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj85?versionId=null 2023-07-29T00:50:05.464 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9zgqcPc5qbaNxlzikRGRRMuSXlM= 2023-07-29T00:50:05.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9zgqcPc5qbaNxlzikRGRRMuSXlM=', 'Content-Length': '0'} 2023-07-29T00:50:05.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000260fbca70e16a79c-0064c4623d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.508 DEBUG:boto:path=/hiiqos-61/obj85 2023-07-29T00:50:05.509 DEBUG:boto:auth_path=/hiiqos-61/obj85 2023-07-29T00:50:05.509 DEBUG:boto:path=/hiiqos-61/obj85?versionId=null 2023-07-29T00:50:05.509 DEBUG:boto:auth_path=/hiiqos-61/obj85?versionId=null 2023-07-29T00:50:05.510 DEBUG:boto:Method: HEAD 2023-07-29T00:50:05.510 DEBUG:boto:Path: /hiiqos-61/obj85?versionId=null 2023-07-29T00:50:05.510 DEBUG:boto:Data: 2023-07-29T00:50:05.511 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.511 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:05.511 DEBUG:boto:Port: 8001 2023-07-29T00:50:05.512 DEBUG:boto:Params: {} 2023-07-29T00:50:05.512 DEBUG:boto:Token: None 2023-07-29T00:50:05.513 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj85?versionId=null 2023-07-29T00:50:05.513 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AcN72BW77DktihetlqJ8vLGFXPg= 2023-07-29T00:50:05.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AcN72BW77DktihetlqJ8vLGFXPg=', 'Content-Length': '0'} 2023-07-29T00:50:05.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079fc05103972ada3-0064c4623d-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.516 DEBUG:boto:path=/hiiqos-61/obj85 2023-07-29T00:50:05.516 DEBUG:boto:auth_path=/hiiqos-61/obj85 2023-07-29T00:50:05.516 DEBUG:boto:path=/hiiqos-61/obj85?versionId=null 2023-07-29T00:50:05.517 DEBUG:boto:auth_path=/hiiqos-61/obj85?versionId=null 2023-07-29T00:50:05.517 DEBUG:boto:Method: HEAD 2023-07-29T00:50:05.517 DEBUG:boto:Path: /hiiqos-61/obj85?versionId=null 2023-07-29T00:50:05.518 DEBUG:boto:Data: 2023-07-29T00:50:05.518 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.518 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:05.519 DEBUG:boto:Port: 8000 2023-07-29T00:50:05.519 DEBUG:boto:Params: {} 2023-07-29T00:50:05.519 DEBUG:boto:Token: None 2023-07-29T00:50:05.520 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj85?versionId=null 2023-07-29T00:50:05.520 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AcN72BW77DktihetlqJ8vLGFXPg= 2023-07-29T00:50:05.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AcN72BW77DktihetlqJ8vLGFXPg=', 'Content-Length': '0'} 2023-07-29T00:50:05.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3e737e152715168-0064c4623d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-07-29T00:50:05.523 DEBUG:boto:path=/hiiqos-61/obj85 2023-07-29T00:50:05.523 DEBUG:boto:auth_path=/hiiqos-61/obj85 2023-07-29T00:50:05.524 DEBUG:boto:path=/hiiqos-61/obj85?versionId=null 2023-07-29T00:50:05.524 DEBUG:boto:auth_path=/hiiqos-61/obj85?versionId=null 2023-07-29T00:50:05.525 DEBUG:boto:Method: GET 2023-07-29T00:50:05.525 DEBUG:boto:Path: /hiiqos-61/obj85?versionId=null 2023-07-29T00:50:05.525 DEBUG:boto:Data: 2023-07-29T00:50:05.526 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.526 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:05.526 DEBUG:boto:Port: 8001 2023-07-29T00:50:05.527 DEBUG:boto:Params: {} 2023-07-29T00:50:05.527 DEBUG:boto:Token: None 2023-07-29T00:50:05.527 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj85?versionId=null 2023-07-29T00:50:05.528 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9zgqcPc5qbaNxlzikRGRRMuSXlM= 2023-07-29T00:50:05.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9zgqcPc5qbaNxlzikRGRRMuSXlM=', 'Content-Length': '0'} 2023-07-29T00:50:05.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055e7774c46f60053-0064c4623d-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.531 DEBUG:boto:path=/hiiqos-61/obj85 2023-07-29T00:50:05.531 DEBUG:boto:auth_path=/hiiqos-61/obj85 2023-07-29T00:50:05.531 DEBUG:boto:path=/hiiqos-61/obj85?versionId=null 2023-07-29T00:50:05.532 DEBUG:boto:auth_path=/hiiqos-61/obj85?versionId=null 2023-07-29T00:50:05.532 DEBUG:boto:Method: GET 2023-07-29T00:50:05.533 DEBUG:boto:Path: /hiiqos-61/obj85?versionId=null 2023-07-29T00:50:05.533 DEBUG:boto:Data: 2023-07-29T00:50:05.533 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.534 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:05.534 DEBUG:boto:Port: 8000 2023-07-29T00:50:05.534 DEBUG:boto:Params: {} 2023-07-29T00:50:05.535 DEBUG:boto:Token: None 2023-07-29T00:50:05.535 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj85?versionId=null 2023-07-29T00:50:05.536 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9zgqcPc5qbaNxlzikRGRRMuSXlM= 2023-07-29T00:50:05.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9zgqcPc5qbaNxlzikRGRRMuSXlM=', 'Content-Length': '0'} 2023-07-29T00:50:05.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a713f48f820ffa6-0064c4623d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.538 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:05.539 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:05.539 DEBUG:boto:path=/hiiqos-61/?prefix=obj85 2023-07-29T00:50:05.539 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj85 2023-07-29T00:50:05.540 DEBUG:boto:Method: GET 2023-07-29T00:50:05.540 DEBUG:boto:Path: /hiiqos-61/?prefix=obj85 2023-07-29T00:50:05.540 DEBUG:boto:Data: 2023-07-29T00:50:05.541 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.541 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:05.541 DEBUG:boto:Port: 8001 2023-07-29T00:50:05.542 DEBUG:boto:Params: {} 2023-07-29T00:50:05.542 DEBUG:boto:Token: None 2023-07-29T00:50:05.543 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/ 2023-07-29T00:50:05.543 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU= 2023-07-29T00:50:05.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU=', 'Content-Length': '0'} 2023-07-29T00:50:05.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004298e7857ead87ff-0064c4623d-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.546 DEBUG:boto:b'hiiqos-61obj851000falseobj852023-07-29T00:45:01.475Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:05.547 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:05.547 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:05.547 DEBUG:boto:path=/hiiqos-61/?prefix=obj85 2023-07-29T00:50:05.548 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj85 2023-07-29T00:50:05.548 DEBUG:boto:Method: GET 2023-07-29T00:50:05.549 DEBUG:boto:Path: /hiiqos-61/?prefix=obj85 2023-07-29T00:50:05.549 DEBUG:boto:Data: 2023-07-29T00:50:05.549 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.550 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:05.550 DEBUG:boto:Port: 8000 2023-07-29T00:50:05.550 DEBUG:boto:Params: {} 2023-07-29T00:50:05.551 DEBUG:boto:Token: None 2023-07-29T00:50:05.551 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/ 2023-07-29T00:50:05.552 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU= 2023-07-29T00:50:05.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU=', 'Content-Length': '0'} 2023-07-29T00:50:05.555 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e905210d7b69378e-0064c4623d-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.556 DEBUG:boto:b'hiiqos-61obj851000falseobj852023-07-29T00:45:01.475Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:05.557 DEBUG:boto:path=/hiiqos-61/obj85 2023-07-29T00:50:05.557 DEBUG:boto:auth_path=/hiiqos-61/obj85 2023-07-29T00:50:05.557 DEBUG:boto:Method: HEAD 2023-07-29T00:50:05.558 DEBUG:boto:Path: /hiiqos-61/obj85 2023-07-29T00:50:05.558 DEBUG:boto:Data: 2023-07-29T00:50:05.558 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.559 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:05.559 DEBUG:boto:Port: 8001 2023-07-29T00:50:05.559 DEBUG:boto:Params: {} 2023-07-29T00:50:05.560 DEBUG:boto:Token: None 2023-07-29T00:50:05.560 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj85 2023-07-29T00:50:05.561 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:s1UDgE0F6zkqd41RQtIuyT1a+yw= 2023-07-29T00:50:05.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:s1UDgE0F6zkqd41RQtIuyT1a+yw=', 'Content-Length': '0'} 2023-07-29T00:50:05.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000252d679b7b252ee1-0064c4623d-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.563 DEBUG:boto:path=/hiiqos-61/obj85 2023-07-29T00:50:05.564 DEBUG:boto:auth_path=/hiiqos-61/obj85 2023-07-29T00:50:05.564 DEBUG:boto:Method: HEAD 2023-07-29T00:50:05.564 DEBUG:boto:Path: /hiiqos-61/obj85 2023-07-29T00:50:05.565 DEBUG:boto:Data: 2023-07-29T00:50:05.565 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.565 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:05.566 DEBUG:boto:Port: 8000 2023-07-29T00:50:05.566 DEBUG:boto:Params: {} 2023-07-29T00:50:05.566 DEBUG:boto:Token: None 2023-07-29T00:50:05.567 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj85 2023-07-29T00:50:05.567 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:s1UDgE0F6zkqd41RQtIuyT1a+yw= 2023-07-29T00:50:05.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:s1UDgE0F6zkqd41RQtIuyT1a+yw=', 'Content-Length': '0'} 2023-07-29T00:50:05.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042421bf1fb6a3483-0064c4623d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-07-29T00:50:05.570 DEBUG:boto:path=/hiiqos-61/obj85 2023-07-29T00:50:05.570 DEBUG:boto:auth_path=/hiiqos-61/obj85 2023-07-29T00:50:05.571 DEBUG:boto:Method: GET 2023-07-29T00:50:05.571 DEBUG:boto:Path: /hiiqos-61/obj85 2023-07-29T00:50:05.572 DEBUG:boto:Data: 2023-07-29T00:50:05.572 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.572 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:05.573 DEBUG:boto:Port: 8001 2023-07-29T00:50:05.573 DEBUG:boto:Params: {} 2023-07-29T00:50:05.573 DEBUG:boto:Token: None 2023-07-29T00:50:05.574 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj85 2023-07-29T00:50:05.574 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:g+ak39qAmO/S7+Uef3ngkaeq1yM= 2023-07-29T00:50:05.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:g+ak39qAmO/S7+Uef3ngkaeq1yM=', 'Content-Length': '0'} 2023-07-29T00:50:05.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e923dabfbd50fb21-0064c4623d-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.577 DEBUG:boto:path=/hiiqos-61/obj85 2023-07-29T00:50:05.577 DEBUG:boto:auth_path=/hiiqos-61/obj85 2023-07-29T00:50:05.578 DEBUG:boto:Method: GET 2023-07-29T00:50:05.578 DEBUG:boto:Path: /hiiqos-61/obj85 2023-07-29T00:50:05.578 DEBUG:boto:Data: 2023-07-29T00:50:05.579 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.579 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:05.579 DEBUG:boto:Port: 8000 2023-07-29T00:50:05.580 DEBUG:boto:Params: {} 2023-07-29T00:50:05.580 DEBUG:boto:Token: None 2023-07-29T00:50:05.581 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj85 2023-07-29T00:50:05.581 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:g+ak39qAmO/S7+Uef3ngkaeq1yM= 2023-07-29T00:50:05.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:g+ak39qAmO/S7+Uef3ngkaeq1yM=', 'Content-Length': '0'} 2023-07-29T00:50:05.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce7d1c4eb5a9be7b-0064c4623d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.584 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:05.584 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:05.585 DEBUG:boto:path=/hiiqos-61/?prefix=obj85 2023-07-29T00:50:05.585 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj85 2023-07-29T00:50:05.585 DEBUG:boto:Method: GET 2023-07-29T00:50:05.586 DEBUG:boto:Path: /hiiqos-61/?prefix=obj85 2023-07-29T00:50:05.586 DEBUG:boto:Data: 2023-07-29T00:50:05.586 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.587 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:05.587 DEBUG:boto:Port: 8001 2023-07-29T00:50:05.587 DEBUG:boto:Params: {} 2023-07-29T00:50:05.588 DEBUG:boto:Token: None 2023-07-29T00:50:05.588 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/ 2023-07-29T00:50:05.589 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU= 2023-07-29T00:50:05.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU=', 'Content-Length': '0'} 2023-07-29T00:50:05.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a153a74b70cc6f58-0064c4623d-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.591 DEBUG:boto:b'hiiqos-61obj851000falseobj852023-07-29T00:45:01.475Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:05.592 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:05.593 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:05.593 DEBUG:boto:path=/hiiqos-61/?prefix=obj85 2023-07-29T00:50:05.593 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj85 2023-07-29T00:50:05.594 DEBUG:boto:Method: GET 2023-07-29T00:50:05.594 DEBUG:boto:Path: /hiiqos-61/?prefix=obj85 2023-07-29T00:50:05.594 DEBUG:boto:Data: 2023-07-29T00:50:05.595 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.595 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:05.595 DEBUG:boto:Port: 8000 2023-07-29T00:50:05.596 DEBUG:boto:Params: {} 2023-07-29T00:50:05.596 DEBUG:boto:Token: None 2023-07-29T00:50:05.597 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/ 2023-07-29T00:50:05.597 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU= 2023-07-29T00:50:05.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU=', 'Content-Length': '0'} 2023-07-29T00:50:05.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1a92320a7e13070-0064c4623d-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.601 DEBUG:boto:b'hiiqos-61obj851000falseobj852023-07-29T00:45:01.475Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:05.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-07-29T00:50:05.602 DEBUG:boto:path=/hiiqos-61/obj87 2023-07-29T00:50:05.603 DEBUG:boto:auth_path=/hiiqos-61/obj87 2023-07-29T00:50:05.603 DEBUG:boto:path=/hiiqos-61/obj87?versionId=null 2023-07-29T00:50:05.603 DEBUG:boto:auth_path=/hiiqos-61/obj87?versionId=null 2023-07-29T00:50:05.604 DEBUG:boto:Method: GET 2023-07-29T00:50:05.604 DEBUG:boto:Path: /hiiqos-61/obj87?versionId=null 2023-07-29T00:50:05.605 DEBUG:boto:Data: 2023-07-29T00:50:05.605 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.605 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:05.606 DEBUG:boto:Port: 8001 2023-07-29T00:50:05.606 DEBUG:boto:Params: {} 2023-07-29T00:50:05.606 DEBUG:boto:Token: None 2023-07-29T00:50:05.607 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj87?versionId=null 2023-07-29T00:50:05.607 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Ll3CLn8l4tZGrdQWRNgknluG5ls= 2023-07-29T00:50:05.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Ll3CLn8l4tZGrdQWRNgknluG5ls=', 'Content-Length': '0'} 2023-07-29T00:50:05.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef6f7979c166115a-0064c4623d-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.610 DEBUG:boto:path=/hiiqos-61/obj87 2023-07-29T00:50:05.611 DEBUG:boto:auth_path=/hiiqos-61/obj87 2023-07-29T00:50:05.611 DEBUG:boto:path=/hiiqos-61/obj87?versionId=null 2023-07-29T00:50:05.611 DEBUG:boto:auth_path=/hiiqos-61/obj87?versionId=null 2023-07-29T00:50:05.612 DEBUG:boto:Method: GET 2023-07-29T00:50:05.612 DEBUG:boto:Path: /hiiqos-61/obj87?versionId=null 2023-07-29T00:50:05.612 DEBUG:boto:Data: 2023-07-29T00:50:05.613 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.613 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:05.613 DEBUG:boto:Port: 8000 2023-07-29T00:50:05.614 DEBUG:boto:Params: {} 2023-07-29T00:50:05.614 DEBUG:boto:Token: None 2023-07-29T00:50:05.615 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj87?versionId=null 2023-07-29T00:50:05.615 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Ll3CLn8l4tZGrdQWRNgknluG5ls= 2023-07-29T00:50:05.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Ll3CLn8l4tZGrdQWRNgknluG5ls=', 'Content-Length': '0'} 2023-07-29T00:50:05.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008513eb6129370115-0064c4623d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.617 DEBUG:boto:path=/hiiqos-61/obj87 2023-07-29T00:50:05.618 DEBUG:boto:auth_path=/hiiqos-61/obj87 2023-07-29T00:50:05.618 DEBUG:boto:path=/hiiqos-61/obj87?versionId=null 2023-07-29T00:50:05.619 DEBUG:boto:auth_path=/hiiqos-61/obj87?versionId=null 2023-07-29T00:50:05.619 DEBUG:boto:Method: HEAD 2023-07-29T00:50:05.619 DEBUG:boto:Path: /hiiqos-61/obj87?versionId=null 2023-07-29T00:50:05.620 DEBUG:boto:Data: 2023-07-29T00:50:05.620 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.620 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:05.621 DEBUG:boto:Port: 8001 2023-07-29T00:50:05.621 DEBUG:boto:Params: {} 2023-07-29T00:50:05.621 DEBUG:boto:Token: None 2023-07-29T00:50:05.622 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj87?versionId=null 2023-07-29T00:50:05.622 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FRFnPja/clHzJ71SMIAJ9x9/N5k= 2023-07-29T00:50:05.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FRFnPja/clHzJ71SMIAJ9x9/N5k=', 'Content-Length': '0'} 2023-07-29T00:50:05.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000736f7b84ea708218-0064c4623d-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.624 DEBUG:boto:path=/hiiqos-61/obj87 2023-07-29T00:50:05.625 DEBUG:boto:auth_path=/hiiqos-61/obj87 2023-07-29T00:50:05.625 DEBUG:boto:path=/hiiqos-61/obj87?versionId=null 2023-07-29T00:50:05.626 DEBUG:boto:auth_path=/hiiqos-61/obj87?versionId=null 2023-07-29T00:50:05.626 DEBUG:boto:Method: HEAD 2023-07-29T00:50:05.626 DEBUG:boto:Path: /hiiqos-61/obj87?versionId=null 2023-07-29T00:50:05.627 DEBUG:boto:Data: 2023-07-29T00:50:05.627 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.627 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:05.628 DEBUG:boto:Port: 8000 2023-07-29T00:50:05.628 DEBUG:boto:Params: {} 2023-07-29T00:50:05.628 DEBUG:boto:Token: None 2023-07-29T00:50:05.629 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj87?versionId=null 2023-07-29T00:50:05.629 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FRFnPja/clHzJ71SMIAJ9x9/N5k= 2023-07-29T00:50:05.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FRFnPja/clHzJ71SMIAJ9x9/N5k=', 'Content-Length': '0'} 2023-07-29T00:50:05.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf88fd86975881be-0064c4623d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-07-29T00:50:05.632 DEBUG:boto:path=/hiiqos-61/obj87 2023-07-29T00:50:05.632 DEBUG:boto:auth_path=/hiiqos-61/obj87 2023-07-29T00:50:05.633 DEBUG:boto:path=/hiiqos-61/obj87?versionId=null 2023-07-29T00:50:05.633 DEBUG:boto:auth_path=/hiiqos-61/obj87?versionId=null 2023-07-29T00:50:05.634 DEBUG:boto:Method: GET 2023-07-29T00:50:05.634 DEBUG:boto:Path: /hiiqos-61/obj87?versionId=null 2023-07-29T00:50:05.634 DEBUG:boto:Data: 2023-07-29T00:50:05.635 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.635 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:05.635 DEBUG:boto:Port: 8001 2023-07-29T00:50:05.636 DEBUG:boto:Params: {} 2023-07-29T00:50:05.636 DEBUG:boto:Token: None 2023-07-29T00:50:05.636 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj87?versionId=null 2023-07-29T00:50:05.637 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Ll3CLn8l4tZGrdQWRNgknluG5ls= 2023-07-29T00:50:05.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Ll3CLn8l4tZGrdQWRNgknluG5ls=', 'Content-Length': '0'} 2023-07-29T00:50:05.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068efbcd502ef4873-0064c4623d-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.639 DEBUG:boto:path=/hiiqos-61/obj87 2023-07-29T00:50:05.640 DEBUG:boto:auth_path=/hiiqos-61/obj87 2023-07-29T00:50:05.640 DEBUG:boto:path=/hiiqos-61/obj87?versionId=null 2023-07-29T00:50:05.641 DEBUG:boto:auth_path=/hiiqos-61/obj87?versionId=null 2023-07-29T00:50:05.641 DEBUG:boto:Method: GET 2023-07-29T00:50:05.641 DEBUG:boto:Path: /hiiqos-61/obj87?versionId=null 2023-07-29T00:50:05.642 DEBUG:boto:Data: 2023-07-29T00:50:05.642 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.642 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:05.643 DEBUG:boto:Port: 8000 2023-07-29T00:50:05.643 DEBUG:boto:Params: {} 2023-07-29T00:50:05.643 DEBUG:boto:Token: None 2023-07-29T00:50:05.644 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj87?versionId=null 2023-07-29T00:50:05.644 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Ll3CLn8l4tZGrdQWRNgknluG5ls= 2023-07-29T00:50:05.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Ll3CLn8l4tZGrdQWRNgknluG5ls=', 'Content-Length': '0'} 2023-07-29T00:50:05.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000548bb720ef1a2fdb-0064c4623d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.647 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:05.647 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:05.648 DEBUG:boto:path=/hiiqos-61/?prefix=obj87 2023-07-29T00:50:05.648 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj87 2023-07-29T00:50:05.648 DEBUG:boto:Method: GET 2023-07-29T00:50:05.649 DEBUG:boto:Path: /hiiqos-61/?prefix=obj87 2023-07-29T00:50:05.649 DEBUG:boto:Data: 2023-07-29T00:50:05.649 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.650 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:05.650 DEBUG:boto:Port: 8001 2023-07-29T00:50:05.650 DEBUG:boto:Params: {} 2023-07-29T00:50:05.651 DEBUG:boto:Token: None 2023-07-29T00:50:05.651 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/ 2023-07-29T00:50:05.652 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU= 2023-07-29T00:50:05.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU=', 'Content-Length': '0'} 2023-07-29T00:50:05.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7b062699b5bdcb9-0064c4623d-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.655 DEBUG:boto:b'hiiqos-61obj871000falseobj872023-07-29T00:45:02.365Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:05.655 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:05.656 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:05.656 DEBUG:boto:path=/hiiqos-61/?prefix=obj87 2023-07-29T00:50:05.657 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj87 2023-07-29T00:50:05.657 DEBUG:boto:Method: GET 2023-07-29T00:50:05.657 DEBUG:boto:Path: /hiiqos-61/?prefix=obj87 2023-07-29T00:50:05.658 DEBUG:boto:Data: 2023-07-29T00:50:05.658 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.658 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:05.659 DEBUG:boto:Port: 8000 2023-07-29T00:50:05.659 DEBUG:boto:Params: {} 2023-07-29T00:50:05.659 DEBUG:boto:Token: None 2023-07-29T00:50:05.660 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/ 2023-07-29T00:50:05.660 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU= 2023-07-29T00:50:05.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU=', 'Content-Length': '0'} 2023-07-29T00:50:05.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b8ac91bfb09e12e-0064c4623d-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.664 DEBUG:boto:b'hiiqos-61obj871000falseobj872023-07-29T00:45:02.365Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:05.665 DEBUG:boto:path=/hiiqos-61/obj87 2023-07-29T00:50:05.665 DEBUG:boto:auth_path=/hiiqos-61/obj87 2023-07-29T00:50:05.666 DEBUG:boto:Method: HEAD 2023-07-29T00:50:05.666 DEBUG:boto:Path: /hiiqos-61/obj87 2023-07-29T00:50:05.667 DEBUG:boto:Data: 2023-07-29T00:50:05.667 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.667 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:05.668 DEBUG:boto:Port: 8001 2023-07-29T00:50:05.668 DEBUG:boto:Params: {} 2023-07-29T00:50:05.668 DEBUG:boto:Token: None 2023-07-29T00:50:05.669 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj87 2023-07-29T00:50:05.669 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nfkYibeVvO4Zv1GqUyrYY7n0Ds8= 2023-07-29T00:50:05.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nfkYibeVvO4Zv1GqUyrYY7n0Ds8=', 'Content-Length': '0'} 2023-07-29T00:50:05.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041d195e5c64b0ef2-0064c4623d-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.672 DEBUG:boto:path=/hiiqos-61/obj87 2023-07-29T00:50:05.672 DEBUG:boto:auth_path=/hiiqos-61/obj87 2023-07-29T00:50:05.672 DEBUG:boto:Method: HEAD 2023-07-29T00:50:05.673 DEBUG:boto:Path: /hiiqos-61/obj87 2023-07-29T00:50:05.673 DEBUG:boto:Data: 2023-07-29T00:50:05.673 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.674 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:05.674 DEBUG:boto:Port: 8000 2023-07-29T00:50:05.674 DEBUG:boto:Params: {} 2023-07-29T00:50:05.675 DEBUG:boto:Token: None 2023-07-29T00:50:05.675 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj87 2023-07-29T00:50:05.676 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nfkYibeVvO4Zv1GqUyrYY7n0Ds8= 2023-07-29T00:50:05.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nfkYibeVvO4Zv1GqUyrYY7n0Ds8=', 'Content-Length': '0'} 2023-07-29T00:50:05.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad1af75dd66359ec-0064c4623d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-07-29T00:50:05.679 DEBUG:boto:path=/hiiqos-61/obj87 2023-07-29T00:50:05.679 DEBUG:boto:auth_path=/hiiqos-61/obj87 2023-07-29T00:50:05.679 DEBUG:boto:Method: GET 2023-07-29T00:50:05.680 DEBUG:boto:Path: /hiiqos-61/obj87 2023-07-29T00:50:05.680 DEBUG:boto:Data: 2023-07-29T00:50:05.680 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.681 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:05.681 DEBUG:boto:Port: 8001 2023-07-29T00:50:05.681 DEBUG:boto:Params: {} 2023-07-29T00:50:05.682 DEBUG:boto:Token: None 2023-07-29T00:50:05.682 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj87 2023-07-29T00:50:05.683 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fAMqqJihajb6LlVgPheXf6HSBYg= 2023-07-29T00:50:05.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fAMqqJihajb6LlVgPheXf6HSBYg=', 'Content-Length': '0'} 2023-07-29T00:50:05.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0818ab90f722ad6-0064c4623d-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.685 DEBUG:boto:path=/hiiqos-61/obj87 2023-07-29T00:50:05.686 DEBUG:boto:auth_path=/hiiqos-61/obj87 2023-07-29T00:50:05.686 DEBUG:boto:Method: GET 2023-07-29T00:50:05.686 DEBUG:boto:Path: /hiiqos-61/obj87 2023-07-29T00:50:05.687 DEBUG:boto:Data: 2023-07-29T00:50:05.687 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.687 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:05.688 DEBUG:boto:Port: 8000 2023-07-29T00:50:05.688 DEBUG:boto:Params: {} 2023-07-29T00:50:05.689 DEBUG:boto:Token: None 2023-07-29T00:50:05.689 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj87 2023-07-29T00:50:05.689 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fAMqqJihajb6LlVgPheXf6HSBYg= 2023-07-29T00:50:05.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fAMqqJihajb6LlVgPheXf6HSBYg=', 'Content-Length': '0'} 2023-07-29T00:50:05.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f61b58cf228005f-0064c4623d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.736 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:05.737 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:05.737 DEBUG:boto:path=/hiiqos-61/?prefix=obj87 2023-07-29T00:50:05.737 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj87 2023-07-29T00:50:05.738 DEBUG:boto:Method: GET 2023-07-29T00:50:05.738 DEBUG:boto:Path: /hiiqos-61/?prefix=obj87 2023-07-29T00:50:05.739 DEBUG:boto:Data: 2023-07-29T00:50:05.739 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.739 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:05.740 DEBUG:boto:Port: 8001 2023-07-29T00:50:05.740 DEBUG:boto:Params: {} 2023-07-29T00:50:05.740 DEBUG:boto:Token: None 2023-07-29T00:50:05.741 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/ 2023-07-29T00:50:05.741 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU= 2023-07-29T00:50:05.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU=', 'Content-Length': '0'} 2023-07-29T00:50:05.744 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002feee6b49b1dc223-0064c4623d-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.744 DEBUG:boto:b'hiiqos-61obj871000falseobj872023-07-29T00:45:02.365Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:05.745 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:05.745 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:05.746 DEBUG:boto:path=/hiiqos-61/?prefix=obj87 2023-07-29T00:50:05.746 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj87 2023-07-29T00:50:05.746 DEBUG:boto:Method: GET 2023-07-29T00:50:05.747 DEBUG:boto:Path: /hiiqos-61/?prefix=obj87 2023-07-29T00:50:05.747 DEBUG:boto:Data: 2023-07-29T00:50:05.747 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.748 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:05.748 DEBUG:boto:Port: 8000 2023-07-29T00:50:05.749 DEBUG:boto:Params: {} 2023-07-29T00:50:05.749 DEBUG:boto:Token: None 2023-07-29T00:50:05.749 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/ 2023-07-29T00:50:05.750 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU= 2023-07-29T00:50:05.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU=', 'Content-Length': '0'} 2023-07-29T00:50:05.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022414479cce89e0d-0064c4623d-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.754 DEBUG:boto:b'hiiqos-61obj871000falseobj872023-07-29T00:45:02.365Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:05.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-07-29T00:50:05.755 DEBUG:boto:path=/hiiqos-61/obj92 2023-07-29T00:50:05.756 DEBUG:boto:auth_path=/hiiqos-61/obj92 2023-07-29T00:50:05.756 DEBUG:boto:path=/hiiqos-61/obj92?versionId=null 2023-07-29T00:50:05.756 DEBUG:boto:auth_path=/hiiqos-61/obj92?versionId=null 2023-07-29T00:50:05.757 DEBUG:boto:Method: GET 2023-07-29T00:50:05.757 DEBUG:boto:Path: /hiiqos-61/obj92?versionId=null 2023-07-29T00:50:05.757 DEBUG:boto:Data: 2023-07-29T00:50:05.758 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.758 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:05.759 DEBUG:boto:Port: 8001 2023-07-29T00:50:05.759 DEBUG:boto:Params: {} 2023-07-29T00:50:05.759 DEBUG:boto:Token: None 2023-07-29T00:50:05.760 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj92?versionId=null 2023-07-29T00:50:05.760 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9dtYcfWxZHcYjk+Zr/FmYj85nUo= 2023-07-29T00:50:05.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9dtYcfWxZHcYjk+Zr/FmYj85nUo=', 'Content-Length': '0'} 2023-07-29T00:50:05.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061948edfba0f5770-0064c4623d-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.763 DEBUG:boto:path=/hiiqos-61/obj92 2023-07-29T00:50:05.763 DEBUG:boto:auth_path=/hiiqos-61/obj92 2023-07-29T00:50:05.764 DEBUG:boto:path=/hiiqos-61/obj92?versionId=null 2023-07-29T00:50:05.764 DEBUG:boto:auth_path=/hiiqos-61/obj92?versionId=null 2023-07-29T00:50:05.764 DEBUG:boto:Method: GET 2023-07-29T00:50:05.765 DEBUG:boto:Path: /hiiqos-61/obj92?versionId=null 2023-07-29T00:50:05.765 DEBUG:boto:Data: 2023-07-29T00:50:05.765 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.766 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:05.766 DEBUG:boto:Port: 8000 2023-07-29T00:50:05.766 DEBUG:boto:Params: {} 2023-07-29T00:50:05.767 DEBUG:boto:Token: None 2023-07-29T00:50:05.767 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj92?versionId=null 2023-07-29T00:50:05.768 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9dtYcfWxZHcYjk+Zr/FmYj85nUo= 2023-07-29T00:50:05.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9dtYcfWxZHcYjk+Zr/FmYj85nUo=', 'Content-Length': '0'} 2023-07-29T00:50:05.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a707536707ce894a-0064c4623d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.770 DEBUG:boto:path=/hiiqos-61/obj92 2023-07-29T00:50:05.771 DEBUG:boto:auth_path=/hiiqos-61/obj92 2023-07-29T00:50:05.771 DEBUG:boto:path=/hiiqos-61/obj92?versionId=null 2023-07-29T00:50:05.771 DEBUG:boto:auth_path=/hiiqos-61/obj92?versionId=null 2023-07-29T00:50:05.772 DEBUG:boto:Method: HEAD 2023-07-29T00:50:05.772 DEBUG:boto:Path: /hiiqos-61/obj92?versionId=null 2023-07-29T00:50:05.773 DEBUG:boto:Data: 2023-07-29T00:50:05.773 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.773 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:05.774 DEBUG:boto:Port: 8001 2023-07-29T00:50:05.774 DEBUG:boto:Params: {} 2023-07-29T00:50:05.774 DEBUG:boto:Token: None 2023-07-29T00:50:05.775 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj92?versionId=null 2023-07-29T00:50:05.775 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MuXwflsOC74QY04F1UQDyqFGtyY= 2023-07-29T00:50:05.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MuXwflsOC74QY04F1UQDyqFGtyY=', 'Content-Length': '0'} 2023-07-29T00:50:05.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f86acd0675f0488-0064c4623d-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.778 DEBUG:boto:path=/hiiqos-61/obj92 2023-07-29T00:50:05.778 DEBUG:boto:auth_path=/hiiqos-61/obj92 2023-07-29T00:50:05.778 DEBUG:boto:path=/hiiqos-61/obj92?versionId=null 2023-07-29T00:50:05.779 DEBUG:boto:auth_path=/hiiqos-61/obj92?versionId=null 2023-07-29T00:50:05.779 DEBUG:boto:Method: HEAD 2023-07-29T00:50:05.779 DEBUG:boto:Path: /hiiqos-61/obj92?versionId=null 2023-07-29T00:50:05.780 DEBUG:boto:Data: 2023-07-29T00:50:05.780 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.780 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:05.781 DEBUG:boto:Port: 8000 2023-07-29T00:50:05.781 DEBUG:boto:Params: {} 2023-07-29T00:50:05.781 DEBUG:boto:Token: None 2023-07-29T00:50:05.782 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj92?versionId=null 2023-07-29T00:50:05.782 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MuXwflsOC74QY04F1UQDyqFGtyY= 2023-07-29T00:50:05.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MuXwflsOC74QY04F1UQDyqFGtyY=', 'Content-Length': '0'} 2023-07-29T00:50:05.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078e20acc1cdf88da-0064c4623d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-07-29T00:50:05.785 DEBUG:boto:path=/hiiqos-61/obj92 2023-07-29T00:50:05.785 DEBUG:boto:auth_path=/hiiqos-61/obj92 2023-07-29T00:50:05.786 DEBUG:boto:path=/hiiqos-61/obj92?versionId=null 2023-07-29T00:50:05.786 DEBUG:boto:auth_path=/hiiqos-61/obj92?versionId=null 2023-07-29T00:50:05.787 DEBUG:boto:Method: GET 2023-07-29T00:50:05.787 DEBUG:boto:Path: /hiiqos-61/obj92?versionId=null 2023-07-29T00:50:05.787 DEBUG:boto:Data: 2023-07-29T00:50:05.788 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.788 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:05.788 DEBUG:boto:Port: 8001 2023-07-29T00:50:05.789 DEBUG:boto:Params: {} 2023-07-29T00:50:05.789 DEBUG:boto:Token: None 2023-07-29T00:50:05.790 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj92?versionId=null 2023-07-29T00:50:05.790 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9dtYcfWxZHcYjk+Zr/FmYj85nUo= 2023-07-29T00:50:05.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9dtYcfWxZHcYjk+Zr/FmYj85nUo=', 'Content-Length': '0'} 2023-07-29T00:50:05.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e031347c7c1857f0-0064c4623d-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.793 DEBUG:boto:path=/hiiqos-61/obj92 2023-07-29T00:50:05.793 DEBUG:boto:auth_path=/hiiqos-61/obj92 2023-07-29T00:50:05.794 DEBUG:boto:path=/hiiqos-61/obj92?versionId=null 2023-07-29T00:50:05.794 DEBUG:boto:auth_path=/hiiqos-61/obj92?versionId=null 2023-07-29T00:50:05.794 DEBUG:boto:Method: GET 2023-07-29T00:50:05.795 DEBUG:boto:Path: /hiiqos-61/obj92?versionId=null 2023-07-29T00:50:05.795 DEBUG:boto:Data: 2023-07-29T00:50:05.795 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.796 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:05.796 DEBUG:boto:Port: 8000 2023-07-29T00:50:05.797 DEBUG:boto:Params: {} 2023-07-29T00:50:05.797 DEBUG:boto:Token: None 2023-07-29T00:50:05.797 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj92?versionId=null 2023-07-29T00:50:05.798 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9dtYcfWxZHcYjk+Zr/FmYj85nUo= 2023-07-29T00:50:05.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9dtYcfWxZHcYjk+Zr/FmYj85nUo=', 'Content-Length': '0'} 2023-07-29T00:50:05.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f2802dafb084c81-0064c4623d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.840 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:05.841 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:05.841 DEBUG:boto:path=/hiiqos-61/?prefix=obj92 2023-07-29T00:50:05.841 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj92 2023-07-29T00:50:05.842 DEBUG:boto:Method: GET 2023-07-29T00:50:05.842 DEBUG:boto:Path: /hiiqos-61/?prefix=obj92 2023-07-29T00:50:05.842 DEBUG:boto:Data: 2023-07-29T00:50:05.843 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.843 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:05.844 DEBUG:boto:Port: 8001 2023-07-29T00:50:05.844 DEBUG:boto:Params: {} 2023-07-29T00:50:05.844 DEBUG:boto:Token: None 2023-07-29T00:50:05.845 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/ 2023-07-29T00:50:05.845 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU= 2023-07-29T00:50:05.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU=', 'Content-Length': '0'} 2023-07-29T00:50:05.848 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a553d4f9b04ba52-0064c4623d-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.848 DEBUG:boto:b'hiiqos-61obj921000falseobj922023-07-29T00:45:02.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:05.849 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:05.849 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:05.850 DEBUG:boto:path=/hiiqos-61/?prefix=obj92 2023-07-29T00:50:05.850 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj92 2023-07-29T00:50:05.850 DEBUG:boto:Method: GET 2023-07-29T00:50:05.851 DEBUG:boto:Path: /hiiqos-61/?prefix=obj92 2023-07-29T00:50:05.851 DEBUG:boto:Data: 2023-07-29T00:50:05.851 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.852 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:05.852 DEBUG:boto:Port: 8000 2023-07-29T00:50:05.852 DEBUG:boto:Params: {} 2023-07-29T00:50:05.853 DEBUG:boto:Token: None 2023-07-29T00:50:05.853 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/ 2023-07-29T00:50:05.854 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU= 2023-07-29T00:50:05.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU=', 'Content-Length': '0'} 2023-07-29T00:50:05.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef2910fb25932101-0064c4623d-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.858 DEBUG:boto:b'hiiqos-61obj921000falseobj922023-07-29T00:45:02.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:05.859 DEBUG:boto:path=/hiiqos-61/obj92 2023-07-29T00:50:05.859 DEBUG:boto:auth_path=/hiiqos-61/obj92 2023-07-29T00:50:05.859 DEBUG:boto:Method: HEAD 2023-07-29T00:50:05.860 DEBUG:boto:Path: /hiiqos-61/obj92 2023-07-29T00:50:05.860 DEBUG:boto:Data: 2023-07-29T00:50:05.860 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.861 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:05.861 DEBUG:boto:Port: 8001 2023-07-29T00:50:05.861 DEBUG:boto:Params: {} 2023-07-29T00:50:05.862 DEBUG:boto:Token: None 2023-07-29T00:50:05.862 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj92 2023-07-29T00:50:05.863 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lptVohbYLgYep0oWPkelbKOx1jA= 2023-07-29T00:50:05.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lptVohbYLgYep0oWPkelbKOx1jA=', 'Content-Length': '0'} 2023-07-29T00:50:05.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e17bd3b08b223bf1-0064c4623d-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.865 DEBUG:boto:path=/hiiqos-61/obj92 2023-07-29T00:50:05.865 DEBUG:boto:auth_path=/hiiqos-61/obj92 2023-07-29T00:50:05.866 DEBUG:boto:Method: HEAD 2023-07-29T00:50:05.866 DEBUG:boto:Path: /hiiqos-61/obj92 2023-07-29T00:50:05.866 DEBUG:boto:Data: 2023-07-29T00:50:05.867 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.867 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:05.867 DEBUG:boto:Port: 8000 2023-07-29T00:50:05.868 DEBUG:boto:Params: {} 2023-07-29T00:50:05.868 DEBUG:boto:Token: None 2023-07-29T00:50:05.869 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj92 2023-07-29T00:50:05.869 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lptVohbYLgYep0oWPkelbKOx1jA= 2023-07-29T00:50:05.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lptVohbYLgYep0oWPkelbKOx1jA=', 'Content-Length': '0'} 2023-07-29T00:50:05.871 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc7edf91e91724bd-0064c4623d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-07-29T00:50:05.872 DEBUG:boto:path=/hiiqos-61/obj92 2023-07-29T00:50:05.872 DEBUG:boto:auth_path=/hiiqos-61/obj92 2023-07-29T00:50:05.873 DEBUG:boto:Method: GET 2023-07-29T00:50:05.873 DEBUG:boto:Path: /hiiqos-61/obj92 2023-07-29T00:50:05.873 DEBUG:boto:Data: 2023-07-29T00:50:05.874 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.874 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:05.874 DEBUG:boto:Port: 8001 2023-07-29T00:50:05.875 DEBUG:boto:Params: {} 2023-07-29T00:50:05.875 DEBUG:boto:Token: None 2023-07-29T00:50:05.875 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj92 2023-07-29T00:50:05.876 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZSUdk5UorTz5d00vAwBU/3k7nO0= 2023-07-29T00:50:05.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZSUdk5UorTz5d00vAwBU/3k7nO0=', 'Content-Length': '0'} 2023-07-29T00:50:05.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e9cd04bd7ebdd7d-0064c4623d-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.920 DEBUG:boto:path=/hiiqos-61/obj92 2023-07-29T00:50:05.921 DEBUG:boto:auth_path=/hiiqos-61/obj92 2023-07-29T00:50:05.921 DEBUG:boto:Method: GET 2023-07-29T00:50:05.922 DEBUG:boto:Path: /hiiqos-61/obj92 2023-07-29T00:50:05.922 DEBUG:boto:Data: 2023-07-29T00:50:05.922 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.923 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:05.923 DEBUG:boto:Port: 8000 2023-07-29T00:50:05.923 DEBUG:boto:Params: {} 2023-07-29T00:50:05.924 DEBUG:boto:Token: None 2023-07-29T00:50:05.924 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj92 2023-07-29T00:50:05.925 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZSUdk5UorTz5d00vAwBU/3k7nO0= 2023-07-29T00:50:05.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZSUdk5UorTz5d00vAwBU/3k7nO0=', 'Content-Length': '0'} 2023-07-29T00:50:05.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd3ebf55900fc44a-0064c4623d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.927 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:05.928 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:05.928 DEBUG:boto:path=/hiiqos-61/?prefix=obj92 2023-07-29T00:50:05.929 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj92 2023-07-29T00:50:05.929 DEBUG:boto:Method: GET 2023-07-29T00:50:05.930 DEBUG:boto:Path: /hiiqos-61/?prefix=obj92 2023-07-29T00:50:05.930 DEBUG:boto:Data: 2023-07-29T00:50:05.930 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.931 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:05.931 DEBUG:boto:Port: 8001 2023-07-29T00:50:05.931 DEBUG:boto:Params: {} 2023-07-29T00:50:05.932 DEBUG:boto:Token: None 2023-07-29T00:50:05.932 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/ 2023-07-29T00:50:05.933 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU= 2023-07-29T00:50:05.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU=', 'Content-Length': '0'} 2023-07-29T00:50:05.936 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e06806ca198f0cb5-0064c4623d-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.936 DEBUG:boto:b'hiiqos-61obj921000falseobj922023-07-29T00:45:02.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:05.937 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:05.938 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:05.938 DEBUG:boto:path=/hiiqos-61/?prefix=obj92 2023-07-29T00:50:05.938 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj92 2023-07-29T00:50:05.939 DEBUG:boto:Method: GET 2023-07-29T00:50:05.939 DEBUG:boto:Path: /hiiqos-61/?prefix=obj92 2023-07-29T00:50:05.939 DEBUG:boto:Data: 2023-07-29T00:50:05.940 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.940 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:05.941 DEBUG:boto:Port: 8000 2023-07-29T00:50:05.941 DEBUG:boto:Params: {} 2023-07-29T00:50:05.942 DEBUG:boto:Token: None 2023-07-29T00:50:05.942 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/ 2023-07-29T00:50:05.942 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU= 2023-07-29T00:50:05.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:/7KcNNs9tIIwZN5mIP+rD45UkMU=', 'Content-Length': '0'} 2023-07-29T00:50:05.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d2e760eb5d82ceb-0064c4623d-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.947 DEBUG:boto:b'hiiqos-61obj921000falseobj922023-07-29T00:45:02.089Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:05.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-07-29T00:50:05.948 DEBUG:boto:path=/hiiqos-61/obj93 2023-07-29T00:50:05.948 DEBUG:boto:auth_path=/hiiqos-61/obj93 2023-07-29T00:50:05.949 DEBUG:boto:path=/hiiqos-61/obj93?versionId=null 2023-07-29T00:50:05.949 DEBUG:boto:auth_path=/hiiqos-61/obj93?versionId=null 2023-07-29T00:50:05.950 DEBUG:boto:Method: GET 2023-07-29T00:50:05.950 DEBUG:boto:Path: /hiiqos-61/obj93?versionId=null 2023-07-29T00:50:05.950 DEBUG:boto:Data: 2023-07-29T00:50:05.951 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.951 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:05.951 DEBUG:boto:Port: 8001 2023-07-29T00:50:05.952 DEBUG:boto:Params: {} 2023-07-29T00:50:05.952 DEBUG:boto:Token: None 2023-07-29T00:50:05.953 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj93?versionId=null 2023-07-29T00:50:05.953 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:W9BTnBhxGd8jOGnHyUCJ4qb5eWo= 2023-07-29T00:50:05.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:W9BTnBhxGd8jOGnHyUCJ4qb5eWo=', 'Content-Length': '0'} 2023-07-29T00:50:05.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c40c690b055a57c3-0064c4623d-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:05.956 DEBUG:boto:path=/hiiqos-61/obj93 2023-07-29T00:50:05.957 DEBUG:boto:auth_path=/hiiqos-61/obj93 2023-07-29T00:50:05.957 DEBUG:boto:path=/hiiqos-61/obj93?versionId=null 2023-07-29T00:50:05.958 DEBUG:boto:auth_path=/hiiqos-61/obj93?versionId=null 2023-07-29T00:50:05.958 DEBUG:boto:Method: GET 2023-07-29T00:50:05.958 DEBUG:boto:Path: /hiiqos-61/obj93?versionId=null 2023-07-29T00:50:05.959 DEBUG:boto:Data: 2023-07-29T00:50:05.959 DEBUG:boto:Headers: {} 2023-07-29T00:50:05.959 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:05.960 DEBUG:boto:Port: 8000 2023-07-29T00:50:05.960 DEBUG:boto:Params: {} 2023-07-29T00:50:05.961 DEBUG:boto:Token: None 2023-07-29T00:50:05.961 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:05 GMT /hiiqos-61/obj93?versionId=null 2023-07-29T00:50:05.961 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:W9BTnBhxGd8jOGnHyUCJ4qb5eWo= 2023-07-29T00:50:05.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:05 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:W9BTnBhxGd8jOGnHyUCJ4qb5eWo=', 'Content-Length': '0'} 2023-07-29T00:50:05.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce02e30a87076456-0064c4623d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:06.004 DEBUG:boto:path=/hiiqos-61/obj93 2023-07-29T00:50:06.005 DEBUG:boto:auth_path=/hiiqos-61/obj93 2023-07-29T00:50:06.005 DEBUG:boto:path=/hiiqos-61/obj93?versionId=null 2023-07-29T00:50:06.006 DEBUG:boto:auth_path=/hiiqos-61/obj93?versionId=null 2023-07-29T00:50:06.006 DEBUG:boto:Method: HEAD 2023-07-29T00:50:06.007 DEBUG:boto:Path: /hiiqos-61/obj93?versionId=null 2023-07-29T00:50:06.007 DEBUG:boto:Data: 2023-07-29T00:50:06.007 DEBUG:boto:Headers: {} 2023-07-29T00:50:06.008 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:06.008 DEBUG:boto:Port: 8001 2023-07-29T00:50:06.008 DEBUG:boto:Params: {} 2023-07-29T00:50:06.009 DEBUG:boto:Token: None 2023-07-29T00:50:06.009 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:06 GMT /hiiqos-61/obj93?versionId=null 2023-07-29T00:50:06.010 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yhrOzE0XLfgNPPzr1aaZGJMN4wQ= 2023-07-29T00:50:06.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yhrOzE0XLfgNPPzr1aaZGJMN4wQ=', 'Content-Length': '0'} 2023-07-29T00:50:06.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5dff21a0ba30771-0064c4623d-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:06.013 DEBUG:boto:path=/hiiqos-61/obj93 2023-07-29T00:50:06.013 DEBUG:boto:auth_path=/hiiqos-61/obj93 2023-07-29T00:50:06.013 DEBUG:boto:path=/hiiqos-61/obj93?versionId=null 2023-07-29T00:50:06.014 DEBUG:boto:auth_path=/hiiqos-61/obj93?versionId=null 2023-07-29T00:50:06.014 DEBUG:boto:Method: HEAD 2023-07-29T00:50:06.014 DEBUG:boto:Path: /hiiqos-61/obj93?versionId=null 2023-07-29T00:50:06.015 DEBUG:boto:Data: 2023-07-29T00:50:06.015 DEBUG:boto:Headers: {} 2023-07-29T00:50:06.015 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:06.016 DEBUG:boto:Port: 8000 2023-07-29T00:50:06.016 DEBUG:boto:Params: {} 2023-07-29T00:50:06.017 DEBUG:boto:Token: None 2023-07-29T00:50:06.017 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:06 GMT /hiiqos-61/obj93?versionId=null 2023-07-29T00:50:06.017 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yhrOzE0XLfgNPPzr1aaZGJMN4wQ= 2023-07-29T00:50:06.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yhrOzE0XLfgNPPzr1aaZGJMN4wQ=', 'Content-Length': '0'} 2023-07-29T00:50:06.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038c237c9cab13be8-0064c4623d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:06.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-07-29T00:50:06.021 DEBUG:boto:path=/hiiqos-61/obj93 2023-07-29T00:50:06.021 DEBUG:boto:auth_path=/hiiqos-61/obj93 2023-07-29T00:50:06.021 DEBUG:boto:path=/hiiqos-61/obj93?versionId=null 2023-07-29T00:50:06.022 DEBUG:boto:auth_path=/hiiqos-61/obj93?versionId=null 2023-07-29T00:50:06.022 DEBUG:boto:Method: GET 2023-07-29T00:50:06.022 DEBUG:boto:Path: /hiiqos-61/obj93?versionId=null 2023-07-29T00:50:06.023 DEBUG:boto:Data: 2023-07-29T00:50:06.023 DEBUG:boto:Headers: {} 2023-07-29T00:50:06.024 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:06.024 DEBUG:boto:Port: 8001 2023-07-29T00:50:06.024 DEBUG:boto:Params: {} 2023-07-29T00:50:06.025 DEBUG:boto:Token: None 2023-07-29T00:50:06.025 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:06 GMT /hiiqos-61/obj93?versionId=null 2023-07-29T00:50:06.026 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6eRxho3vYFyYT9t+/nzZ6+zv0+4= 2023-07-29T00:50:06.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6eRxho3vYFyYT9t+/nzZ6+zv0+4=', 'Content-Length': '0'} 2023-07-29T00:50:06.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095f19e86fbf94d5e-0064c4623d-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:06.028 DEBUG:boto:path=/hiiqos-61/obj93 2023-07-29T00:50:06.029 DEBUG:boto:auth_path=/hiiqos-61/obj93 2023-07-29T00:50:06.029 DEBUG:boto:path=/hiiqos-61/obj93?versionId=null 2023-07-29T00:50:06.029 DEBUG:boto:auth_path=/hiiqos-61/obj93?versionId=null 2023-07-29T00:50:06.030 DEBUG:boto:Method: GET 2023-07-29T00:50:06.030 DEBUG:boto:Path: /hiiqos-61/obj93?versionId=null 2023-07-29T00:50:06.031 DEBUG:boto:Data: 2023-07-29T00:50:06.031 DEBUG:boto:Headers: {} 2023-07-29T00:50:06.031 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:06.032 DEBUG:boto:Port: 8000 2023-07-29T00:50:06.032 DEBUG:boto:Params: {} 2023-07-29T00:50:06.032 DEBUG:boto:Token: None 2023-07-29T00:50:06.033 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:06 GMT /hiiqos-61/obj93?versionId=null 2023-07-29T00:50:06.033 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6eRxho3vYFyYT9t+/nzZ6+zv0+4= 2023-07-29T00:50:06.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6eRxho3vYFyYT9t+/nzZ6+zv0+4=', 'Content-Length': '0'} 2023-07-29T00:50:06.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002c3ba2ff4da64b6-0064c4623d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:06.036 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:06.037 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:06.037 DEBUG:boto:path=/hiiqos-61/?prefix=obj93 2023-07-29T00:50:06.037 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj93 2023-07-29T00:50:06.038 DEBUG:boto:Method: GET 2023-07-29T00:50:06.038 DEBUG:boto:Path: /hiiqos-61/?prefix=obj93 2023-07-29T00:50:06.039 DEBUG:boto:Data: 2023-07-29T00:50:06.039 DEBUG:boto:Headers: {} 2023-07-29T00:50:06.039 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:06.040 DEBUG:boto:Port: 8001 2023-07-29T00:50:06.040 DEBUG:boto:Params: {} 2023-07-29T00:50:06.040 DEBUG:boto:Token: None 2023-07-29T00:50:06.041 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:06 GMT /hiiqos-61/ 2023-07-29T00:50:06.041 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mRayTUBL6cFOJzQE0QF/Ijhs5PI= 2023-07-29T00:50:06.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mRayTUBL6cFOJzQE0QF/Ijhs5PI=', 'Content-Length': '0'} 2023-07-29T00:50:06.044 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6f9299a845372be-0064c4623d-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:06.044 DEBUG:boto:b'hiiqos-61obj931000falseobj932023-07-29T00:45:02.721Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:06.045 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:06.045 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:06.046 DEBUG:boto:path=/hiiqos-61/?prefix=obj93 2023-07-29T00:50:06.046 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj93 2023-07-29T00:50:06.046 DEBUG:boto:Method: GET 2023-07-29T00:50:06.047 DEBUG:boto:Path: /hiiqos-61/?prefix=obj93 2023-07-29T00:50:06.047 DEBUG:boto:Data: 2023-07-29T00:50:06.048 DEBUG:boto:Headers: {} 2023-07-29T00:50:06.048 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:06.048 DEBUG:boto:Port: 8000 2023-07-29T00:50:06.049 DEBUG:boto:Params: {} 2023-07-29T00:50:06.049 DEBUG:boto:Token: None 2023-07-29T00:50:06.049 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:06 GMT /hiiqos-61/ 2023-07-29T00:50:06.050 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mRayTUBL6cFOJzQE0QF/Ijhs5PI= 2023-07-29T00:50:06.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mRayTUBL6cFOJzQE0QF/Ijhs5PI=', 'Content-Length': '0'} 2023-07-29T00:50:06.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003920f1aedf33277b-0064c4623d-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:06.054 DEBUG:boto:b'hiiqos-61obj931000falseobj932023-07-29T00:45:02.721Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:06.055 DEBUG:boto:path=/hiiqos-61/obj93 2023-07-29T00:50:06.055 DEBUG:boto:auth_path=/hiiqos-61/obj93 2023-07-29T00:50:06.056 DEBUG:boto:Method: HEAD 2023-07-29T00:50:06.056 DEBUG:boto:Path: /hiiqos-61/obj93 2023-07-29T00:50:06.056 DEBUG:boto:Data: 2023-07-29T00:50:06.057 DEBUG:boto:Headers: {} 2023-07-29T00:50:06.057 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:06.057 DEBUG:boto:Port: 8001 2023-07-29T00:50:06.058 DEBUG:boto:Params: {} 2023-07-29T00:50:06.058 DEBUG:boto:Token: None 2023-07-29T00:50:06.058 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:06 GMT /hiiqos-61/obj93 2023-07-29T00:50:06.059 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oo4ey5lFP2FCo21oGlk3wNv48eI= 2023-07-29T00:50:06.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oo4ey5lFP2FCo21oGlk3wNv48eI=', 'Content-Length': '0'} 2023-07-29T00:50:06.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000443164f8fcf678-0064c4623d-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:06.062 DEBUG:boto:path=/hiiqos-61/obj93 2023-07-29T00:50:06.062 DEBUG:boto:auth_path=/hiiqos-61/obj93 2023-07-29T00:50:06.062 DEBUG:boto:Method: HEAD 2023-07-29T00:50:06.063 DEBUG:boto:Path: /hiiqos-61/obj93 2023-07-29T00:50:06.063 DEBUG:boto:Data: 2023-07-29T00:50:06.063 DEBUG:boto:Headers: {} 2023-07-29T00:50:06.064 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:06.064 DEBUG:boto:Port: 8000 2023-07-29T00:50:06.065 DEBUG:boto:Params: {} 2023-07-29T00:50:06.065 DEBUG:boto:Token: None 2023-07-29T00:50:06.065 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:06 GMT /hiiqos-61/obj93 2023-07-29T00:50:06.066 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oo4ey5lFP2FCo21oGlk3wNv48eI= 2023-07-29T00:50:06.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oo4ey5lFP2FCo21oGlk3wNv48eI=', 'Content-Length': '0'} 2023-07-29T00:50:06.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1942d44a802ed3b-0064c4623d-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:05 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:06.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-07-29T00:50:06.069 DEBUG:boto:path=/hiiqos-61/obj93 2023-07-29T00:50:06.069 DEBUG:boto:auth_path=/hiiqos-61/obj93 2023-07-29T00:50:06.070 DEBUG:boto:Method: GET 2023-07-29T00:50:06.070 DEBUG:boto:Path: /hiiqos-61/obj93 2023-07-29T00:50:06.070 DEBUG:boto:Data: 2023-07-29T00:50:06.071 DEBUG:boto:Headers: {} 2023-07-29T00:50:06.071 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:06.071 DEBUG:boto:Port: 8001 2023-07-29T00:50:06.072 DEBUG:boto:Params: {} 2023-07-29T00:50:06.072 DEBUG:boto:Token: None 2023-07-29T00:50:06.073 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:06 GMT /hiiqos-61/obj93 2023-07-29T00:50:06.073 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yrIANf78BBLPiHQfh+u5E95k1/I= 2023-07-29T00:50:06.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yrIANf78BBLPiHQfh+u5E95k1/I=', 'Content-Length': '0'} 2023-07-29T00:50:06.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000880c60d42922d05f-0064c4623e-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:06.076 DEBUG:boto:path=/hiiqos-61/obj93 2023-07-29T00:50:06.076 DEBUG:boto:auth_path=/hiiqos-61/obj93 2023-07-29T00:50:06.077 DEBUG:boto:Method: GET 2023-07-29T00:50:06.077 DEBUG:boto:Path: /hiiqos-61/obj93 2023-07-29T00:50:06.077 DEBUG:boto:Data: 2023-07-29T00:50:06.078 DEBUG:boto:Headers: {} 2023-07-29T00:50:06.078 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:06.078 DEBUG:boto:Port: 8000 2023-07-29T00:50:06.079 DEBUG:boto:Params: {} 2023-07-29T00:50:06.079 DEBUG:boto:Token: None 2023-07-29T00:50:06.079 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:06 GMT /hiiqos-61/obj93 2023-07-29T00:50:06.080 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yrIANf78BBLPiHQfh+u5E95k1/I= 2023-07-29T00:50:06.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yrIANf78BBLPiHQfh+u5E95k1/I=', 'Content-Length': '0'} 2023-07-29T00:50:06.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067671fc2a9116807-0064c4623e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:06.083 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:06.083 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:06.083 DEBUG:boto:path=/hiiqos-61/?prefix=obj93 2023-07-29T00:50:06.084 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj93 2023-07-29T00:50:06.084 DEBUG:boto:Method: GET 2023-07-29T00:50:06.085 DEBUG:boto:Path: /hiiqos-61/?prefix=obj93 2023-07-29T00:50:06.085 DEBUG:boto:Data: 2023-07-29T00:50:06.085 DEBUG:boto:Headers: {} 2023-07-29T00:50:06.086 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:06.086 DEBUG:boto:Port: 8001 2023-07-29T00:50:06.086 DEBUG:boto:Params: {} 2023-07-29T00:50:06.087 DEBUG:boto:Token: None 2023-07-29T00:50:06.087 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:06 GMT /hiiqos-61/ 2023-07-29T00:50:06.088 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mRayTUBL6cFOJzQE0QF/Ijhs5PI= 2023-07-29T00:50:06.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mRayTUBL6cFOJzQE0QF/Ijhs5PI=', 'Content-Length': '0'} 2023-07-29T00:50:06.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2282089e84277c3-0064c4623e-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:06.091 DEBUG:boto:b'hiiqos-61obj931000falseobj932023-07-29T00:45:02.721Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:06.091 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:06.092 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:06.092 DEBUG:boto:path=/hiiqos-61/?prefix=obj93 2023-07-29T00:50:06.093 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj93 2023-07-29T00:50:06.093 DEBUG:boto:Method: GET 2023-07-29T00:50:06.093 DEBUG:boto:Path: /hiiqos-61/?prefix=obj93 2023-07-29T00:50:06.094 DEBUG:boto:Data: 2023-07-29T00:50:06.094 DEBUG:boto:Headers: {} 2023-07-29T00:50:06.095 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:06.095 DEBUG:boto:Port: 8000 2023-07-29T00:50:06.096 DEBUG:boto:Params: {} 2023-07-29T00:50:06.097 DEBUG:boto:Token: None 2023-07-29T00:50:06.097 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:06 GMT /hiiqos-61/ 2023-07-29T00:50:06.097 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mRayTUBL6cFOJzQE0QF/Ijhs5PI= 2023-07-29T00:50:06.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mRayTUBL6cFOJzQE0QF/Ijhs5PI=', 'Content-Length': '0'} 2023-07-29T00:50:06.101 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8f6f7d2f837fa9e-0064c4623e-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:06.102 DEBUG:boto:b'hiiqos-61obj931000falseobj932023-07-29T00:45:02.721Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:06.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-07-29T00:50:06.103 DEBUG:boto:path=/hiiqos-61/obj95 2023-07-29T00:50:06.104 DEBUG:boto:auth_path=/hiiqos-61/obj95 2023-07-29T00:50:06.104 DEBUG:boto:path=/hiiqos-61/obj95?versionId=null 2023-07-29T00:50:06.105 DEBUG:boto:auth_path=/hiiqos-61/obj95?versionId=null 2023-07-29T00:50:06.105 DEBUG:boto:Method: GET 2023-07-29T00:50:06.106 DEBUG:boto:Path: /hiiqos-61/obj95?versionId=null 2023-07-29T00:50:06.106 DEBUG:boto:Data: 2023-07-29T00:50:06.106 DEBUG:boto:Headers: {} 2023-07-29T00:50:06.107 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:06.107 DEBUG:boto:Port: 8001 2023-07-29T00:50:06.107 DEBUG:boto:Params: {} 2023-07-29T00:50:06.108 DEBUG:boto:Token: None 2023-07-29T00:50:06.108 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:06 GMT /hiiqos-61/obj95?versionId=null 2023-07-29T00:50:06.109 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3ZTNUZxBn4qvnhs75JR1ntO0y+s= 2023-07-29T00:50:06.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3ZTNUZxBn4qvnhs75JR1ntO0y+s=', 'Content-Length': '0'} 2023-07-29T00:50:06.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3bba14dd7cb04a1-0064c4623e-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:06.111 DEBUG:boto:path=/hiiqos-61/obj95 2023-07-29T00:50:06.112 DEBUG:boto:auth_path=/hiiqos-61/obj95 2023-07-29T00:50:06.112 DEBUG:boto:path=/hiiqos-61/obj95?versionId=null 2023-07-29T00:50:06.112 DEBUG:boto:auth_path=/hiiqos-61/obj95?versionId=null 2023-07-29T00:50:06.113 DEBUG:boto:Method: GET 2023-07-29T00:50:06.113 DEBUG:boto:Path: /hiiqos-61/obj95?versionId=null 2023-07-29T00:50:06.114 DEBUG:boto:Data: 2023-07-29T00:50:06.114 DEBUG:boto:Headers: {} 2023-07-29T00:50:06.114 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:06.115 DEBUG:boto:Port: 8000 2023-07-29T00:50:06.115 DEBUG:boto:Params: {} 2023-07-29T00:50:06.115 DEBUG:boto:Token: None 2023-07-29T00:50:06.116 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:06 GMT /hiiqos-61/obj95?versionId=null 2023-07-29T00:50:06.116 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3ZTNUZxBn4qvnhs75JR1ntO0y+s= 2023-07-29T00:50:06.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3ZTNUZxBn4qvnhs75JR1ntO0y+s=', 'Content-Length': '0'} 2023-07-29T00:50:06.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002e98b5259cd8056-0064c4623e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:06.160 DEBUG:boto:path=/hiiqos-61/obj95 2023-07-29T00:50:06.161 DEBUG:boto:auth_path=/hiiqos-61/obj95 2023-07-29T00:50:06.161 DEBUG:boto:path=/hiiqos-61/obj95?versionId=null 2023-07-29T00:50:06.162 DEBUG:boto:auth_path=/hiiqos-61/obj95?versionId=null 2023-07-29T00:50:06.162 DEBUG:boto:Method: HEAD 2023-07-29T00:50:06.163 DEBUG:boto:Path: /hiiqos-61/obj95?versionId=null 2023-07-29T00:50:06.163 DEBUG:boto:Data: 2023-07-29T00:50:06.163 DEBUG:boto:Headers: {} 2023-07-29T00:50:06.164 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:06.164 DEBUG:boto:Port: 8001 2023-07-29T00:50:06.164 DEBUG:boto:Params: {} 2023-07-29T00:50:06.165 DEBUG:boto:Token: None 2023-07-29T00:50:06.165 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:06 GMT /hiiqos-61/obj95?versionId=null 2023-07-29T00:50:06.166 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OM7TGfIv4AfRydNMWcOnjRe9xIk= 2023-07-29T00:50:06.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OM7TGfIv4AfRydNMWcOnjRe9xIk=', 'Content-Length': '0'} 2023-07-29T00:50:06.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed3e859312709afa-0064c4623e-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:06.169 DEBUG:boto:path=/hiiqos-61/obj95 2023-07-29T00:50:06.169 DEBUG:boto:auth_path=/hiiqos-61/obj95 2023-07-29T00:50:06.169 DEBUG:boto:path=/hiiqos-61/obj95?versionId=null 2023-07-29T00:50:06.170 DEBUG:boto:auth_path=/hiiqos-61/obj95?versionId=null 2023-07-29T00:50:06.170 DEBUG:boto:Method: HEAD 2023-07-29T00:50:06.170 DEBUG:boto:Path: /hiiqos-61/obj95?versionId=null 2023-07-29T00:50:06.171 DEBUG:boto:Data: 2023-07-29T00:50:06.171 DEBUG:boto:Headers: {} 2023-07-29T00:50:06.171 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:06.172 DEBUG:boto:Port: 8000 2023-07-29T00:50:06.172 DEBUG:boto:Params: {} 2023-07-29T00:50:06.173 DEBUG:boto:Token: None 2023-07-29T00:50:06.173 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:06 GMT /hiiqos-61/obj95?versionId=null 2023-07-29T00:50:06.173 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:OM7TGfIv4AfRydNMWcOnjRe9xIk= 2023-07-29T00:50:06.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:OM7TGfIv4AfRydNMWcOnjRe9xIk=', 'Content-Length': '0'} 2023-07-29T00:50:06.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001744bf46e7c13ab-0064c4623e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:06.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-07-29T00:50:06.177 DEBUG:boto:path=/hiiqos-61/obj95 2023-07-29T00:50:06.177 DEBUG:boto:auth_path=/hiiqos-61/obj95 2023-07-29T00:50:06.177 DEBUG:boto:path=/hiiqos-61/obj95?versionId=null 2023-07-29T00:50:06.178 DEBUG:boto:auth_path=/hiiqos-61/obj95?versionId=null 2023-07-29T00:50:06.178 DEBUG:boto:Method: GET 2023-07-29T00:50:06.178 DEBUG:boto:Path: /hiiqos-61/obj95?versionId=null 2023-07-29T00:50:06.179 DEBUG:boto:Data: 2023-07-29T00:50:06.179 DEBUG:boto:Headers: {} 2023-07-29T00:50:06.179 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:06.180 DEBUG:boto:Port: 8001 2023-07-29T00:50:06.180 DEBUG:boto:Params: {} 2023-07-29T00:50:06.181 DEBUG:boto:Token: None 2023-07-29T00:50:06.181 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:06 GMT /hiiqos-61/obj95?versionId=null 2023-07-29T00:50:06.181 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3ZTNUZxBn4qvnhs75JR1ntO0y+s= 2023-07-29T00:50:06.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3ZTNUZxBn4qvnhs75JR1ntO0y+s=', 'Content-Length': '0'} 2023-07-29T00:50:06.184 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7701c98850daaa4-0064c4623e-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:06.184 DEBUG:boto:path=/hiiqos-61/obj95 2023-07-29T00:50:06.185 DEBUG:boto:auth_path=/hiiqos-61/obj95 2023-07-29T00:50:06.185 DEBUG:boto:path=/hiiqos-61/obj95?versionId=null 2023-07-29T00:50:06.185 DEBUG:boto:auth_path=/hiiqos-61/obj95?versionId=null 2023-07-29T00:50:06.186 DEBUG:boto:Method: GET 2023-07-29T00:50:06.186 DEBUG:boto:Path: /hiiqos-61/obj95?versionId=null 2023-07-29T00:50:06.186 DEBUG:boto:Data: 2023-07-29T00:50:06.187 DEBUG:boto:Headers: {} 2023-07-29T00:50:06.187 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:06.188 DEBUG:boto:Port: 8000 2023-07-29T00:50:06.188 DEBUG:boto:Params: {} 2023-07-29T00:50:06.188 DEBUG:boto:Token: None 2023-07-29T00:50:06.189 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:06 GMT /hiiqos-61/obj95?versionId=null 2023-07-29T00:50:06.189 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3ZTNUZxBn4qvnhs75JR1ntO0y+s= 2023-07-29T00:50:06.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3ZTNUZxBn4qvnhs75JR1ntO0y+s=', 'Content-Length': '0'} 2023-07-29T00:50:06.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d7b6a31ad04a858-0064c4623e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:06.192 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:06.192 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:06.193 DEBUG:boto:path=/hiiqos-61/?prefix=obj95 2023-07-29T00:50:06.193 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj95 2023-07-29T00:50:06.194 DEBUG:boto:Method: GET 2023-07-29T00:50:06.194 DEBUG:boto:Path: /hiiqos-61/?prefix=obj95 2023-07-29T00:50:06.194 DEBUG:boto:Data: 2023-07-29T00:50:06.195 DEBUG:boto:Headers: {} 2023-07-29T00:50:06.195 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:06.195 DEBUG:boto:Port: 8001 2023-07-29T00:50:06.196 DEBUG:boto:Params: {} 2023-07-29T00:50:06.196 DEBUG:boto:Token: None 2023-07-29T00:50:06.197 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:06 GMT /hiiqos-61/ 2023-07-29T00:50:06.197 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mRayTUBL6cFOJzQE0QF/Ijhs5PI= 2023-07-29T00:50:06.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mRayTUBL6cFOJzQE0QF/Ijhs5PI=', 'Content-Length': '0'} 2023-07-29T00:50:06.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016e7a2d627492ea6-0064c4623e-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:06.200 DEBUG:boto:b'hiiqos-61obj951000falseobj952023-07-29T00:45:01.780Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:06.201 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:06.201 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:06.201 DEBUG:boto:path=/hiiqos-61/?prefix=obj95 2023-07-29T00:50:06.202 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj95 2023-07-29T00:50:06.202 DEBUG:boto:Method: GET 2023-07-29T00:50:06.202 DEBUG:boto:Path: /hiiqos-61/?prefix=obj95 2023-07-29T00:50:06.203 DEBUG:boto:Data: 2023-07-29T00:50:06.203 DEBUG:boto:Headers: {} 2023-07-29T00:50:06.203 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:06.204 DEBUG:boto:Port: 8000 2023-07-29T00:50:06.204 DEBUG:boto:Params: {} 2023-07-29T00:50:06.205 DEBUG:boto:Token: None 2023-07-29T00:50:06.205 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:06 GMT /hiiqos-61/ 2023-07-29T00:50:06.205 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mRayTUBL6cFOJzQE0QF/Ijhs5PI= 2023-07-29T00:50:06.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mRayTUBL6cFOJzQE0QF/Ijhs5PI=', 'Content-Length': '0'} 2023-07-29T00:50:06.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000deb6bcc7e3705ebf-0064c4623e-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:06.210 DEBUG:boto:b'hiiqos-61obj951000falseobj952023-07-29T00:45:01.780Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:06.210 DEBUG:boto:path=/hiiqos-61/obj95 2023-07-29T00:50:06.211 DEBUG:boto:auth_path=/hiiqos-61/obj95 2023-07-29T00:50:06.211 DEBUG:boto:Method: HEAD 2023-07-29T00:50:06.211 DEBUG:boto:Path: /hiiqos-61/obj95 2023-07-29T00:50:06.212 DEBUG:boto:Data: 2023-07-29T00:50:06.212 DEBUG:boto:Headers: {} 2023-07-29T00:50:06.212 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:06.213 DEBUG:boto:Port: 8001 2023-07-29T00:50:06.213 DEBUG:boto:Params: {} 2023-07-29T00:50:06.213 DEBUG:boto:Token: None 2023-07-29T00:50:06.214 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:06 GMT /hiiqos-61/obj95 2023-07-29T00:50:06.214 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:N1Vui1Md39BQVuS3lR2t0QFUv6g= 2023-07-29T00:50:06.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:N1Vui1Md39BQVuS3lR2t0QFUv6g=', 'Content-Length': '0'} 2023-07-29T00:50:06.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b23c68246ec971f-0064c4623e-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:06.217 DEBUG:boto:path=/hiiqos-61/obj95 2023-07-29T00:50:06.217 DEBUG:boto:auth_path=/hiiqos-61/obj95 2023-07-29T00:50:06.217 DEBUG:boto:Method: HEAD 2023-07-29T00:50:06.218 DEBUG:boto:Path: /hiiqos-61/obj95 2023-07-29T00:50:06.218 DEBUG:boto:Data: 2023-07-29T00:50:06.218 DEBUG:boto:Headers: {} 2023-07-29T00:50:06.219 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:06.219 DEBUG:boto:Port: 8000 2023-07-29T00:50:06.219 DEBUG:boto:Params: {} 2023-07-29T00:50:06.220 DEBUG:boto:Token: None 2023-07-29T00:50:06.220 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:06 GMT /hiiqos-61/obj95 2023-07-29T00:50:06.221 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:N1Vui1Md39BQVuS3lR2t0QFUv6g= 2023-07-29T00:50:06.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:N1Vui1Md39BQVuS3lR2t0QFUv6g=', 'Content-Length': '0'} 2023-07-29T00:50:06.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d04036a30e93e2b-0064c4623e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:06.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-07-29T00:50:06.224 DEBUG:boto:path=/hiiqos-61/obj95 2023-07-29T00:50:06.224 DEBUG:boto:auth_path=/hiiqos-61/obj95 2023-07-29T00:50:06.224 DEBUG:boto:Method: GET 2023-07-29T00:50:06.225 DEBUG:boto:Path: /hiiqos-61/obj95 2023-07-29T00:50:06.225 DEBUG:boto:Data: 2023-07-29T00:50:06.225 DEBUG:boto:Headers: {} 2023-07-29T00:50:06.226 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:06.226 DEBUG:boto:Port: 8001 2023-07-29T00:50:06.226 DEBUG:boto:Params: {} 2023-07-29T00:50:06.227 DEBUG:boto:Token: None 2023-07-29T00:50:06.227 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:06 GMT /hiiqos-61/obj95 2023-07-29T00:50:06.228 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YGzjSlpFu5KreI+FSaRqw+uDdi4= 2023-07-29T00:50:06.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YGzjSlpFu5KreI+FSaRqw+uDdi4=', 'Content-Length': '0'} 2023-07-29T00:50:06.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-61:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ae195b314d238bd-0064c4623e-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:06.272 DEBUG:boto:path=/hiiqos-61/obj95 2023-07-29T00:50:06.273 DEBUG:boto:auth_path=/hiiqos-61/obj95 2023-07-29T00:50:06.273 DEBUG:boto:Method: GET 2023-07-29T00:50:06.274 DEBUG:boto:Path: /hiiqos-61/obj95 2023-07-29T00:50:06.274 DEBUG:boto:Data: 2023-07-29T00:50:06.274 DEBUG:boto:Headers: {} 2023-07-29T00:50:06.275 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:06.275 DEBUG:boto:Port: 8000 2023-07-29T00:50:06.276 DEBUG:boto:Params: {} 2023-07-29T00:50:06.276 DEBUG:boto:Token: None 2023-07-29T00:50:06.276 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:06 GMT /hiiqos-61/obj95 2023-07-29T00:50:06.277 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YGzjSlpFu5KreI+FSaRqw+uDdi4= 2023-07-29T00:50:06.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YGzjSlpFu5KreI+FSaRqw+uDdi4=', 'Content-Length': '0'} 2023-07-29T00:50:06.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:45:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008890f198a5893734-0064c4623e-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:06.280 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:06.280 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:06.281 DEBUG:boto:path=/hiiqos-61/?prefix=obj95 2023-07-29T00:50:06.281 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj95 2023-07-29T00:50:06.281 DEBUG:boto:Method: GET 2023-07-29T00:50:06.282 DEBUG:boto:Path: /hiiqos-61/?prefix=obj95 2023-07-29T00:50:06.282 DEBUG:boto:Data: 2023-07-29T00:50:06.282 DEBUG:boto:Headers: {} 2023-07-29T00:50:06.283 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:06.283 DEBUG:boto:Port: 8001 2023-07-29T00:50:06.284 DEBUG:boto:Params: {} 2023-07-29T00:50:06.284 DEBUG:boto:Token: None 2023-07-29T00:50:06.284 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:06 GMT /hiiqos-61/ 2023-07-29T00:50:06.285 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mRayTUBL6cFOJzQE0QF/Ijhs5PI= 2023-07-29T00:50:06.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mRayTUBL6cFOJzQE0QF/Ijhs5PI=', 'Content-Length': '0'} 2023-07-29T00:50:06.287 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000faab2067871027ac-0064c4623e-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:06.288 DEBUG:boto:b'hiiqos-61obj951000falseobj952023-07-29T00:45:01.780Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:06.289 DEBUG:boto:path=/hiiqos-61/ 2023-07-29T00:50:06.290 DEBUG:boto:auth_path=/hiiqos-61/ 2023-07-29T00:50:06.290 DEBUG:boto:path=/hiiqos-61/?prefix=obj95 2023-07-29T00:50:06.291 DEBUG:boto:auth_path=/hiiqos-61/?prefix=obj95 2023-07-29T00:50:06.291 DEBUG:boto:Method: GET 2023-07-29T00:50:06.291 DEBUG:boto:Path: /hiiqos-61/?prefix=obj95 2023-07-29T00:50:06.292 DEBUG:boto:Data: 2023-07-29T00:50:06.292 DEBUG:boto:Headers: {} 2023-07-29T00:50:06.293 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:06.293 DEBUG:boto:Port: 8000 2023-07-29T00:50:06.294 DEBUG:boto:Params: {} 2023-07-29T00:50:06.294 DEBUG:boto:Token: None 2023-07-29T00:50:06.295 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:06 GMT /hiiqos-61/ 2023-07-29T00:50:06.295 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mRayTUBL6cFOJzQE0QF/Ijhs5PI= 2023-07-29T00:50:06.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mRayTUBL6cFOJzQE0QF/Ijhs5PI=', 'Content-Length': '0'} 2023-07-29T00:50:06.299 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071d1beed3b567625-0064c4623e-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:06.300 DEBUG:boto:b'hiiqos-61obj951000falseobj952023-07-29T00:45:01.780Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:06.301 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-61 zones={a2, a1} 2023-07-29T00:50:06.301 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-07-29T00:50:06.301 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:50:06.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:50:06.302 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:50:06.347 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:50:06.348 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:50:06.395 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T00:50:06.395 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:50:06.396 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000037810", 2023-07-29T00:50:06.396 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:45:03.148930Z" 2023-07-29T00:50:06.396 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:50:06.396 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:50:06.396 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000033001", 2023-07-29T00:50:06.396 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:45:13.168461Z" 2023-07-29T00:50:06.397 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:50:06.397 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:50:06.397 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000032350", 2023-07-29T00:50:06.397 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:45:13.170151Z" 2023-07-29T00:50:06.397 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:50:06.397 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:50:06.397 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000022717", 2023-07-29T00:50:06.398 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:45:13.171537Z" 2023-07-29T00:50:06.398 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:50:06.398 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T00:50:06.399 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000037810', 1: '00000000000000000000:00000000000000033001', 2: '00000000000000000000:00000000000000032350', 3: '00000000000000000000:00000000000000022717'} 2023-07-29T00:50:06.399 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-07-29T00:50:06.400 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:50:06.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:50:06.401 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:50:06.447 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:50:06.448 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:50:06.491 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:50:06.491 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:50:06.491 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:50:06.491 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:50:06.491 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:50:06.492 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 914016003216195306 2023-07-29T00:50:06.492 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:50:06.492 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:50:06.492 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:50:06.492 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:50:06.492 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:50:06.493 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:50:06.493 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000037810", 2023-07-29T00:50:06.493 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:50:06.493 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 52, 2023-07-29T00:50:06.493 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:50:06.493 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:45:03.148930Z" 2023-07-29T00:50:06.493 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:50:06.494 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:50:06.494 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:50:06.494 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:50:06.494 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:50:06.494 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:50:06.495 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000033001", 2023-07-29T00:50:06.495 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:50:06.495 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T00:50:06.495 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:50:06.495 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:45:13.168461Z" 2023-07-29T00:50:06.495 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:50:06.496 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:50:06.496 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:50:06.496 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:50:06.496 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:50:06.496 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:50:06.496 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000032350", 2023-07-29T00:50:06.496 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:50:06.497 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:50:06.497 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:50:06.497 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:45:13.170151Z" 2023-07-29T00:50:06.497 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:50:06.497 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:50:06.497 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:50:06.497 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:50:06.498 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:50:06.498 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:50:06.498 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000022717", 2023-07-29T00:50:06.498 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:50:06.498 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:50:06.498 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:50:06.498 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:45:13.171537Z" 2023-07-29T00:50:06.499 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:50:06.499 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:50:06.499 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:50:06.499 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:50:06.499 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:50:06.499 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 61, 2023-07-29T00:50:06.499 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 61 2023-07-29T00:50:06.500 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:50:06.500 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:50:06Z" 2023-07-29T00:50:06.500 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:50:06.501 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 914016003216195306 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000037810", "next_step_marker": "", "total_entries": 52, "pos": 0, "timestamp": "2023-07-29T00:45:03.148930Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033001", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T00:45:13.168461Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032350", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:45:13.170151Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000022717", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:45:13.171537Z" } } ] }, "full_sync": { "total": 61, "complete": 61 }, "current_time": "2023-07-29T00:50:06Z" } 2023-07-29T00:50:06.501 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000037810', 'next_step_marker': '', 'total_entries': 52, 'pos': 0, 'timestamp': '2023-07-29T00:45:03.148930Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033001', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T00:45:13.168461Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032350', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:45:13.170151Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022717', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:45:13.171537Z'}}] 2023-07-29T00:50:06.502 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000037810', 1: '00000000000000000000:00000000000000033001', 2: '00000000000000000000:00000000000000032350', 3: '00000000000000000000:00000000000000022717'} 2023-07-29T00:50:06.502 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000037810', 1: '00000000000000000000:00000000000000033001', 2: '00000000000000000000:00000000000000032350', 3: '00000000000000000000:00000000000000022717'} 2023-07-29T00:50:06.502 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-07-29T00:50:06.503 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-07-29T00:50:06.503 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:50:06.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:50:06.504 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:50:06.549 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:50:06.550 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:50:06.600 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T00:50:06.601 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:50:06.601 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000015024", 2023-07-29T00:50:06.601 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:47:15.398911Z" 2023-07-29T00:50:06.602 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:50:06.602 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:50:06.602 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000014040", 2023-07-29T00:50:06.603 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:47:15.400570Z" 2023-07-29T00:50:06.603 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:50:06.603 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:50:06.604 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000011113", 2023-07-29T00:50:06.604 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:48:07.206799Z" 2023-07-29T00:50:06.604 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:50:06.604 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:50:06.604 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000005445", 2023-07-29T00:50:06.605 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:47:15.403544Z" 2023-07-29T00:50:06.605 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:50:06.605 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T00:50:06.607 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000015024', 1: '00000000000000000000:00000000000000014040', 2: '00000000000000000000:00000000000000011113', 3: '00000000000000000000:00000000000000005445'} 2023-07-29T00:50:06.607 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-07-29T00:50:06.608 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:50:06.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:50:06.609 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:50:06.658 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:50:06.658 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:50:06.702 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:50:06.703 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:50:06.703 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:50:06.703 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:50:06.703 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:50:06.703 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 17882949198279251152 2023-07-29T00:50:06.703 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:50:06.704 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:50:06.704 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:50:06.704 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:50:06.704 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:50:06.704 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:50:06.704 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000015024", 2023-07-29T00:50:06.705 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:50:06.705 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:50:06.705 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:50:06.705 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:47:15.398911Z" 2023-07-29T00:50:06.705 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:50:06.705 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:50:06.705 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:50:06.706 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:50:06.706 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:50:06.706 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:50:06.706 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000014040", 2023-07-29T00:50:06.706 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:50:06.706 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:50:06.707 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:50:06.707 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:47:15.400570Z" 2023-07-29T00:50:06.707 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:50:06.707 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:50:06.707 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:50:06.707 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:50:06.707 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:50:06.708 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:50:06.708 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000011113", 2023-07-29T00:50:06.708 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:50:06.708 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:50:06.708 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:50:06.708 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:48:07.206799Z" 2023-07-29T00:50:06.708 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:50:06.709 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:50:06.709 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:50:06.709 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:50:06.709 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:50:06.709 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:50:06.709 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000005445", 2023-07-29T00:50:06.709 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:50:06.710 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:50:06.710 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:50:06.710 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:47:15.403544Z" 2023-07-29T00:50:06.710 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:50:06.710 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:50:06.710 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:50:06.710 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:50:06.711 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:50:06.711 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 0, 2023-07-29T00:50:06.711 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 0 2023-07-29T00:50:06.711 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:50:06.711 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:50:06Z" 2023-07-29T00:50:06.711 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:50:06.712 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17882949198279251152 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015024", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:47:15.398911Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000014040", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:47:15.400570Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011113", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:48:07.206799Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005445", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:47:15.403544Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-07-29T00:50:06Z" } 2023-07-29T00:50:06.713 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015024', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:47:15.398911Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014040', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:47:15.400570Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011113', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:48:07.206799Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005445', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:47:15.403544Z'}}] 2023-07-29T00:50:06.713 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015024', 1: '00000000000000000000:00000000000000014040', 2: '00000000000000000000:00000000000000011113', 3: '00000000000000000000:00000000000000005445'} 2023-07-29T00:50:06.714 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015024', 1: '00000000000000000000:00000000000000014040', 2: '00000000000000000000:00000000000000011113', 3: '00000000000000000000:00000000000000005445'} 2023-07-29T00:50:06.714 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-07-29T00:50:06.715 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_full_sync_after_data_sync_init ... ok 2023-07-29T00:50:06.716 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:50:06.717 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:50:06.717 DEBUG:boto:Method: PUT 2023-07-29T00:50:06.718 DEBUG:boto:Path: /hiiqos-62/ 2023-07-29T00:50:06.718 DEBUG:boto:Data: 2023-07-29T00:50:06.718 DEBUG:boto:Headers: {} 2023-07-29T00:50:06.719 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:06.719 DEBUG:boto:Port: 8000 2023-07-29T00:50:06.719 DEBUG:boto:Params: {} 2023-07-29T00:50:06.720 DEBUG:boto:Token: None 2023-07-29T00:50:06.720 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 00:50:06 GMT /hiiqos-62/ 2023-07-29T00:50:06.721 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:herSqRhZjGifp/dN+WSyJ9BcDCE= 2023-07-29T00:50:06.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:herSqRhZjGifp/dN+WSyJ9BcDCE=', 'Content-Length': '0'} 2023-07-29T00:50:06.734 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fdfe50660eab87cc-0064c4623e-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:50:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:06.735 DEBUG:rgw_multi.tests:created bucket=hiiqos-62 2023-07-29T00:50:06.735 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:50:06.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:50:06.736 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:50:06.782 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:50:06.782 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:50:06.816 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T00:50:06.829 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T00:50:06.830 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:50:06.830 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T00:50:06.831 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:17.836402Z" 2023-07-29T00:50:06.831 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:50:06.831 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:50:06.831 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591806.661758_1144.1", 2023-07-29T00:50:06.832 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:50:06.661758Z" 2023-07-29T00:50:06.832 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:50:06.832 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:50:06.832 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:50:06.832 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T00:50:06.833 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:50:06.833 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:50:06.833 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T00:50:06.833 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:12.811406Z" 2023-07-29T00:50:06.833 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:50:06.834 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T00:50:06.835 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690591217.836402_294.1', 1: '1_1690591806.661758_1144.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T00:50:06.835 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T00:50:06.835 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:50:06.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:50:06.837 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:50:06.883 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:50:06.883 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:50:06.913 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:50:06.841+0000 7f2850d2c880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:50:06.914 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:50:06.842+0000 7f2850d2c880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:50:06.927 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:50:06.927 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:50:06.927 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:50:06.927 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:50:06.928 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:50:06.928 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:50:06.928 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:50:06.928 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:50:06.928 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:50:06.928 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:50:06.929 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:50:06.929 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:50:06.929 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:50:06.929 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T00:50:06.929 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:50:06.929 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:50:06.929 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:50:06.930 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T00:50:06.930 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:50:06.930 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:50:06.930 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:50:06.931 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:50:06.931 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:50:06.932 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:50:06.932 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:50:06.932 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591806.661758_1144.1", 2023-07-29T00:50:06.933 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:50:06.933 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:50:06.933 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:50:06.933 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:50:06.661758Z", 2023-07-29T00:50:06.934 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:50:06.934 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:50:06.934 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:50:06.935 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:50:06.935 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:50:06.936 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:50:06.936 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:50:06.936 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:50:06.936 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:50:06.936 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:50:06.937 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:50:06.937 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:50:06.937 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:50:06.937 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:50:06.937 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:50:06.937 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:50:06.938 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:50:06.938 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:50:06.938 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:50:06.938 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T00:50:06.938 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:50:06.938 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:50:06.938 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:50:06.939 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T00:50:06.939 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:50:06.939 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:50:06.939 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:50:06.939 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:50:06.939 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:50:06.939 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:50:06.940 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:50:06.940 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:50:06.940 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:50:06.940 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:50:06Z" 2023-07-29T00:50:06.940 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:50:06.941 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690591806.661758_1144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:50:06.661758Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:50:06Z" } 2023-07-29T00:50:06.942 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690591806.661758_1144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:50:06.661758Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T00:50:06.942 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690591806.661758_1144.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T00:50:06.943 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690591806.661758_1144.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T00:50:06.943 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T00:50:06.943 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:50:06.944 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:50:06.944 DEBUG:boto:Method: HEAD 2023-07-29T00:50:06.944 DEBUG:boto:Path: /hiiqos-62/ 2023-07-29T00:50:06.945 DEBUG:boto:Data: 2023-07-29T00:50:06.945 DEBUG:boto:Headers: {} 2023-07-29T00:50:06.945 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:06.946 DEBUG:boto:Port: 8000 2023-07-29T00:50:06.946 DEBUG:boto:Params: {} 2023-07-29T00:50:06.946 DEBUG:boto:Token: None 2023-07-29T00:50:06.947 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:06 GMT /hiiqos-62/ 2023-07-29T00:50:06.947 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:teYMkqliKpLF2Lv2iv9cq8IJf2c= 2023-07-29T00:50:06.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:teYMkqliKpLF2Lv2iv9cq8IJf2c=', 'Content-Length': '0'} 2023-07-29T00:50:06.950 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004fa83b3a9cf166b6-0064c4623e-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:50:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:06.950 DEBUG:boto:path=/hiiqos-62/dummy 2023-07-29T00:50:06.951 DEBUG:boto:auth_path=/hiiqos-62/dummy 2023-07-29T00:50:06.951 DEBUG:boto:Method: PUT 2023-07-29T00:50:06.952 DEBUG:boto:Path: /hiiqos-62/dummy 2023-07-29T00:50:06.952 DEBUG:boto:Data: 2023-07-29T00:50:06.952 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:50:06.953 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:06.953 DEBUG:boto:Port: 8000 2023-07-29T00:50:06.953 DEBUG:boto:Params: {} 2023-07-29T00:50:06.954 DEBUG:boto:Token: None 2023-07-29T00:50:06.954 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:50:06 GMT /hiiqos-62/dummy 2023-07-29T00:50:06.954 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:B8KeyOvOXfegJzSj1a4VP6RPdZs= 2023-07-29T00:50:06.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:50:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:B8KeyOvOXfegJzSj1a4VP6RPdZs='} 2023-07-29T00:50:06.961 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000945095511bb60016-0064c4623e-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:50:06 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:06.961 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-62 2023-07-29T00:50:06.962 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:50:06.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:50:06.962 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-62 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:50:07.015 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:50:07.015 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:50:07.048 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:50:06.975+0000 7f9ec4528880 1 waiting to reach incremental sync.. 2023-07-29T00:50:12.050 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:50:11.977+0000 7f9ec4528880 1 bucket sync caught up with source: 2023-07-29T00:50:12.050 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000001.157.6, , , , , ] 2023-07-29T00:50:12.050 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000001.157.6, , , , , ] 2023-07-29T00:50:12.050 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:50:11.977+0000 7f9ec4528880 0 bucket checkpoint complete 2023-07-29T00:50:12.067 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-62 2023-07-29T00:50:12.068 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:50:12.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:50:12.069 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-62 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:50:12.125 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:50:12.126 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:50:12.161 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:50:12.088+0000 7fe6b880f880 1 waiting to reach incremental sync.. 2023-07-29T00:50:17.161 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:50:17.089+0000 7fe6b880f880 1 waiting to reach incremental sync.. 2023-07-29T00:50:22.162 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:50:22.089+0000 7fe6b880f880 1 waiting to reach incremental sync.. 2023-07-29T00:50:27.164 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:50:27.091+0000 7fe6b880f880 1 bucket sync caught up with source: 2023-07-29T00:50:27.164 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000001.840.6, , , , , ] 2023-07-29T00:50:27.164 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000001.840.6, , , , , ] 2023-07-29T00:50:27.164 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:50:27.091+0000 7fe6b880f880 0 bucket checkpoint complete 2023-07-29T00:50:27.181 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-62 zones={a2, a1} 2023-07-29T00:50:27.182 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:50:27.182 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:50:27.183 DEBUG:boto:Method: HEAD 2023-07-29T00:50:27.183 DEBUG:boto:Path: /hiiqos-62/ 2023-07-29T00:50:27.183 DEBUG:boto:Data: 2023-07-29T00:50:27.184 DEBUG:boto:Headers: {} 2023-07-29T00:50:27.184 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:27.184 DEBUG:boto:Port: 8001 2023-07-29T00:50:27.185 DEBUG:boto:Params: {} 2023-07-29T00:50:27.185 DEBUG:boto:Token: None 2023-07-29T00:50:27.186 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:27 GMT /hiiqos-62/ 2023-07-29T00:50:27.186 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Ku4HLeFAO9jijBKb3ymMxaXfBDE= 2023-07-29T00:50:27.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Ku4HLeFAO9jijBKb3ymMxaXfBDE=', 'Content-Length': '0'} 2023-07-29T00:50:27.189 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000051a04c629f6dd64b-0064c46253-2008-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:27.190 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:50:27.190 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:50:27.190 DEBUG:boto:Method: HEAD 2023-07-29T00:50:27.191 DEBUG:boto:Path: /hiiqos-62/ 2023-07-29T00:50:27.191 DEBUG:boto:Data: 2023-07-29T00:50:27.191 DEBUG:boto:Headers: {} 2023-07-29T00:50:27.192 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:27.192 DEBUG:boto:Port: 8000 2023-07-29T00:50:27.192 DEBUG:boto:Params: {} 2023-07-29T00:50:27.193 DEBUG:boto:Token: None 2023-07-29T00:50:27.193 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:27 GMT /hiiqos-62/ 2023-07-29T00:50:27.194 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Ku4HLeFAO9jijBKb3ymMxaXfBDE= 2023-07-29T00:50:27.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Ku4HLeFAO9jijBKb3ymMxaXfBDE=', 'Content-Length': '0'} 2023-07-29T00:50:27.197 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e0d2b96fd1d16179-0064c46253-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:27.197 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-29T00:50:27.197 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:50:27.198 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:50:27.198 DEBUG:boto:path=/hiiqos-62/?versions&max-keys=999 2023-07-29T00:50:27.199 DEBUG:boto:auth_path=/hiiqos-62/?versions&max-keys=999 2023-07-29T00:50:27.199 DEBUG:boto:Method: GET 2023-07-29T00:50:27.200 DEBUG:boto:Path: /hiiqos-62/?versions&max-keys=999 2023-07-29T00:50:27.200 DEBUG:boto:Data: 2023-07-29T00:50:27.200 DEBUG:boto:Headers: {} 2023-07-29T00:50:27.201 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:27.201 DEBUG:boto:Port: 8001 2023-07-29T00:50:27.201 DEBUG:boto:Params: {} 2023-07-29T00:50:27.202 DEBUG:boto:Token: None 2023-07-29T00:50:27.202 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:27 GMT /hiiqos-62/?versions 2023-07-29T00:50:27.203 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oRIJCbdelCrBcxloiaiqsFcWp4o= 2023-07-29T00:50:27.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oRIJCbdelCrBcxloiaiqsFcWp4o=', 'Content-Length': '0'} 2023-07-29T00:50:27.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087c1fbb801e441fb-0064c46253-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:27.206 DEBUG:boto:b'hiiqos-62999falsedummynulltrue2023-07-29T00:50:06.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:27.206 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-07-29T00:50:27.207 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-29T00:50:27.207 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:50:27.208 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:50:27.208 DEBUG:boto:path=/hiiqos-62/?versions&max-keys=999 2023-07-29T00:50:27.209 DEBUG:boto:auth_path=/hiiqos-62/?versions&max-keys=999 2023-07-29T00:50:27.209 DEBUG:boto:Method: GET 2023-07-29T00:50:27.209 DEBUG:boto:Path: /hiiqos-62/?versions&max-keys=999 2023-07-29T00:50:27.210 DEBUG:boto:Data: 2023-07-29T00:50:27.210 DEBUG:boto:Headers: {} 2023-07-29T00:50:27.210 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:27.211 DEBUG:boto:Port: 8000 2023-07-29T00:50:27.211 DEBUG:boto:Params: {} 2023-07-29T00:50:27.211 DEBUG:boto:Token: None 2023-07-29T00:50:27.212 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:27 GMT /hiiqos-62/?versions 2023-07-29T00:50:27.212 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oRIJCbdelCrBcxloiaiqsFcWp4o= 2023-07-29T00:50:27.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oRIJCbdelCrBcxloiaiqsFcWp4o=', 'Content-Length': '0'} 2023-07-29T00:50:27.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d019591f200baab-0064c46253-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:27.260 DEBUG:boto:b'hiiqos-62999falsedummynulltrue2023-07-29T00:50:06.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:27.261 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-07-29T00:50:27.262 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:50:27.262 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:50:27.262 DEBUG:boto:path=/hiiqos-62/?versions&max-keys=999 2023-07-29T00:50:27.263 DEBUG:boto:auth_path=/hiiqos-62/?versions&max-keys=999 2023-07-29T00:50:27.263 DEBUG:boto:Method: GET 2023-07-29T00:50:27.263 DEBUG:boto:Path: /hiiqos-62/?versions&max-keys=999 2023-07-29T00:50:27.264 DEBUG:boto:Data: 2023-07-29T00:50:27.264 DEBUG:boto:Headers: {} 2023-07-29T00:50:27.265 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:27.265 DEBUG:boto:Port: 8001 2023-07-29T00:50:27.265 DEBUG:boto:Params: {} 2023-07-29T00:50:27.266 DEBUG:boto:Token: None 2023-07-29T00:50:27.266 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:27 GMT /hiiqos-62/?versions 2023-07-29T00:50:27.267 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oRIJCbdelCrBcxloiaiqsFcWp4o= 2023-07-29T00:50:27.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oRIJCbdelCrBcxloiaiqsFcWp4o=', 'Content-Length': '0'} 2023-07-29T00:50:27.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003dc25d9cd0172907-0064c46253-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:27.270 DEBUG:boto:b'hiiqos-62999falsedummynulltrue2023-07-29T00:50:06.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:27.271 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:50:27.271 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:50:27.272 DEBUG:boto:path=/hiiqos-62/?versions&max-keys=999 2023-07-29T00:50:27.272 DEBUG:boto:auth_path=/hiiqos-62/?versions&max-keys=999 2023-07-29T00:50:27.272 DEBUG:boto:Method: GET 2023-07-29T00:50:27.273 DEBUG:boto:Path: /hiiqos-62/?versions&max-keys=999 2023-07-29T00:50:27.273 DEBUG:boto:Data: 2023-07-29T00:50:27.273 DEBUG:boto:Headers: {} 2023-07-29T00:50:27.274 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:27.274 DEBUG:boto:Port: 8000 2023-07-29T00:50:27.274 DEBUG:boto:Params: {} 2023-07-29T00:50:27.275 DEBUG:boto:Token: None 2023-07-29T00:50:27.275 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:27 GMT /hiiqos-62/?versions 2023-07-29T00:50:27.276 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:oRIJCbdelCrBcxloiaiqsFcWp4o= 2023-07-29T00:50:27.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oRIJCbdelCrBcxloiaiqsFcWp4o=', 'Content-Length': '0'} 2023-07-29T00:50:27.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023f40f5fc86af847-0064c46253-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:27.279 DEBUG:boto:b'hiiqos-62999falsedummynulltrue2023-07-29T00:50:06.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:27.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-07-29T00:50:27.280 DEBUG:boto:path=/hiiqos-62/dummy 2023-07-29T00:50:27.281 DEBUG:boto:auth_path=/hiiqos-62/dummy 2023-07-29T00:50:27.281 DEBUG:boto:path=/hiiqos-62/dummy?versionId=null 2023-07-29T00:50:27.282 DEBUG:boto:auth_path=/hiiqos-62/dummy?versionId=null 2023-07-29T00:50:27.282 DEBUG:boto:Method: GET 2023-07-29T00:50:27.282 DEBUG:boto:Path: /hiiqos-62/dummy?versionId=null 2023-07-29T00:50:27.283 DEBUG:boto:Data: 2023-07-29T00:50:27.283 DEBUG:boto:Headers: {} 2023-07-29T00:50:27.283 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:27.284 DEBUG:boto:Port: 8001 2023-07-29T00:50:27.284 DEBUG:boto:Params: {} 2023-07-29T00:50:27.285 DEBUG:boto:Token: None 2023-07-29T00:50:27.285 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:27 GMT /hiiqos-62/dummy?versionId=null 2023-07-29T00:50:27.286 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ckE6H2xle/Rw/urUxYTMPzCbY0I= 2023-07-29T00:50:27.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ckE6H2xle/Rw/urUxYTMPzCbY0I=', 'Content-Length': '0'} 2023-07-29T00:50:27.288 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-62:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009126894bad67b2c0-0064c46253-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:27.328 DEBUG:boto:path=/hiiqos-62/dummy 2023-07-29T00:50:27.329 DEBUG:boto:auth_path=/hiiqos-62/dummy 2023-07-29T00:50:27.329 DEBUG:boto:path=/hiiqos-62/dummy?versionId=null 2023-07-29T00:50:27.330 DEBUG:boto:auth_path=/hiiqos-62/dummy?versionId=null 2023-07-29T00:50:27.330 DEBUG:boto:Method: GET 2023-07-29T00:50:27.330 DEBUG:boto:Path: /hiiqos-62/dummy?versionId=null 2023-07-29T00:50:27.331 DEBUG:boto:Data: 2023-07-29T00:50:27.331 DEBUG:boto:Headers: {} 2023-07-29T00:50:27.331 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:27.332 DEBUG:boto:Port: 8000 2023-07-29T00:50:27.332 DEBUG:boto:Params: {} 2023-07-29T00:50:27.333 DEBUG:boto:Token: None 2023-07-29T00:50:27.333 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:27 GMT /hiiqos-62/dummy?versionId=null 2023-07-29T00:50:27.334 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ckE6H2xle/Rw/urUxYTMPzCbY0I= 2023-07-29T00:50:27.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ckE6H2xle/Rw/urUxYTMPzCbY0I=', 'Content-Length': '0'} 2023-07-29T00:50:27.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b2d83a925892f9e-0064c46253-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:27.337 DEBUG:boto:path=/hiiqos-62/dummy 2023-07-29T00:50:27.338 DEBUG:boto:auth_path=/hiiqos-62/dummy 2023-07-29T00:50:27.338 DEBUG:boto:path=/hiiqos-62/dummy?versionId=null 2023-07-29T00:50:27.338 DEBUG:boto:auth_path=/hiiqos-62/dummy?versionId=null 2023-07-29T00:50:27.339 DEBUG:boto:Method: HEAD 2023-07-29T00:50:27.339 DEBUG:boto:Path: /hiiqos-62/dummy?versionId=null 2023-07-29T00:50:27.339 DEBUG:boto:Data: 2023-07-29T00:50:27.340 DEBUG:boto:Headers: {} 2023-07-29T00:50:27.340 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:27.341 DEBUG:boto:Port: 8001 2023-07-29T00:50:27.341 DEBUG:boto:Params: {} 2023-07-29T00:50:27.341 DEBUG:boto:Token: None 2023-07-29T00:50:27.342 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:27 GMT /hiiqos-62/dummy?versionId=null 2023-07-29T00:50:27.342 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rfXg4P/L4pL127fTgKP4V3YiNno= 2023-07-29T00:50:27.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rfXg4P/L4pL127fTgKP4V3YiNno=', 'Content-Length': '0'} 2023-07-29T00:50:27.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-62:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc99c7e3b5c0428d-0064c46253-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:27.346 DEBUG:boto:path=/hiiqos-62/dummy 2023-07-29T00:50:27.346 DEBUG:boto:auth_path=/hiiqos-62/dummy 2023-07-29T00:50:27.347 DEBUG:boto:path=/hiiqos-62/dummy?versionId=null 2023-07-29T00:50:27.347 DEBUG:boto:auth_path=/hiiqos-62/dummy?versionId=null 2023-07-29T00:50:27.347 DEBUG:boto:Method: HEAD 2023-07-29T00:50:27.348 DEBUG:boto:Path: /hiiqos-62/dummy?versionId=null 2023-07-29T00:50:27.348 DEBUG:boto:Data: 2023-07-29T00:50:27.348 DEBUG:boto:Headers: {} 2023-07-29T00:50:27.349 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:27.349 DEBUG:boto:Port: 8000 2023-07-29T00:50:27.349 DEBUG:boto:Params: {} 2023-07-29T00:50:27.350 DEBUG:boto:Token: None 2023-07-29T00:50:27.350 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:27 GMT /hiiqos-62/dummy?versionId=null 2023-07-29T00:50:27.351 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rfXg4P/L4pL127fTgKP4V3YiNno= 2023-07-29T00:50:27.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rfXg4P/L4pL127fTgKP4V3YiNno=', 'Content-Length': '0'} 2023-07-29T00:50:27.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013174d89716a1b7a-0064c46253-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:27.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-07-29T00:50:27.354 DEBUG:boto:path=/hiiqos-62/dummy 2023-07-29T00:50:27.354 DEBUG:boto:auth_path=/hiiqos-62/dummy 2023-07-29T00:50:27.355 DEBUG:boto:path=/hiiqos-62/dummy?versionId=null 2023-07-29T00:50:27.355 DEBUG:boto:auth_path=/hiiqos-62/dummy?versionId=null 2023-07-29T00:50:27.355 DEBUG:boto:Method: GET 2023-07-29T00:50:27.356 DEBUG:boto:Path: /hiiqos-62/dummy?versionId=null 2023-07-29T00:50:27.356 DEBUG:boto:Data: 2023-07-29T00:50:27.357 DEBUG:boto:Headers: {} 2023-07-29T00:50:27.357 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:27.357 DEBUG:boto:Port: 8001 2023-07-29T00:50:27.358 DEBUG:boto:Params: {} 2023-07-29T00:50:27.358 DEBUG:boto:Token: None 2023-07-29T00:50:27.358 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:27 GMT /hiiqos-62/dummy?versionId=null 2023-07-29T00:50:27.359 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ckE6H2xle/Rw/urUxYTMPzCbY0I= 2023-07-29T00:50:27.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ckE6H2xle/Rw/urUxYTMPzCbY0I=', 'Content-Length': '0'} 2023-07-29T00:50:27.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-62:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9683ecf9049b58f-0064c46253-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:27.362 DEBUG:boto:path=/hiiqos-62/dummy 2023-07-29T00:50:27.362 DEBUG:boto:auth_path=/hiiqos-62/dummy 2023-07-29T00:50:27.363 DEBUG:boto:path=/hiiqos-62/dummy?versionId=null 2023-07-29T00:50:27.363 DEBUG:boto:auth_path=/hiiqos-62/dummy?versionId=null 2023-07-29T00:50:27.363 DEBUG:boto:Method: GET 2023-07-29T00:50:27.364 DEBUG:boto:Path: /hiiqos-62/dummy?versionId=null 2023-07-29T00:50:27.364 DEBUG:boto:Data: 2023-07-29T00:50:27.364 DEBUG:boto:Headers: {} 2023-07-29T00:50:27.365 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:27.365 DEBUG:boto:Port: 8000 2023-07-29T00:50:27.366 DEBUG:boto:Params: {} 2023-07-29T00:50:27.366 DEBUG:boto:Token: None 2023-07-29T00:50:27.366 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:27 GMT /hiiqos-62/dummy?versionId=null 2023-07-29T00:50:27.367 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ckE6H2xle/Rw/urUxYTMPzCbY0I= 2023-07-29T00:50:27.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ckE6H2xle/Rw/urUxYTMPzCbY0I=', 'Content-Length': '0'} 2023-07-29T00:50:27.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5fe202d85ef4d1b-0064c46253-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:27.412 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:50:27.413 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:50:27.413 DEBUG:boto:path=/hiiqos-62/?prefix=dummy 2023-07-29T00:50:27.413 DEBUG:boto:auth_path=/hiiqos-62/?prefix=dummy 2023-07-29T00:50:27.414 DEBUG:boto:Method: GET 2023-07-29T00:50:27.414 DEBUG:boto:Path: /hiiqos-62/?prefix=dummy 2023-07-29T00:50:27.414 DEBUG:boto:Data: 2023-07-29T00:50:27.415 DEBUG:boto:Headers: {} 2023-07-29T00:50:27.415 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:27.415 DEBUG:boto:Port: 8001 2023-07-29T00:50:27.416 DEBUG:boto:Params: {} 2023-07-29T00:50:27.416 DEBUG:boto:Token: None 2023-07-29T00:50:27.417 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:27 GMT /hiiqos-62/ 2023-07-29T00:50:27.417 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:usPv3rKOiGP9d0gqJGz9FJkz3fs= 2023-07-29T00:50:27.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:usPv3rKOiGP9d0gqJGz9FJkz3fs=', 'Content-Length': '0'} 2023-07-29T00:50:27.420 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001db84122e0981d40-0064c46253-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:27.420 DEBUG:boto:b'hiiqos-62dummy1000falsedummy2023-07-29T00:50:06.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:27.421 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:50:27.422 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:50:27.422 DEBUG:boto:path=/hiiqos-62/?prefix=dummy 2023-07-29T00:50:27.422 DEBUG:boto:auth_path=/hiiqos-62/?prefix=dummy 2023-07-29T00:50:27.423 DEBUG:boto:Method: GET 2023-07-29T00:50:27.423 DEBUG:boto:Path: /hiiqos-62/?prefix=dummy 2023-07-29T00:50:27.423 DEBUG:boto:Data: 2023-07-29T00:50:27.424 DEBUG:boto:Headers: {} 2023-07-29T00:50:27.424 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:27.424 DEBUG:boto:Port: 8000 2023-07-29T00:50:27.425 DEBUG:boto:Params: {} 2023-07-29T00:50:27.425 DEBUG:boto:Token: None 2023-07-29T00:50:27.426 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:27 GMT /hiiqos-62/ 2023-07-29T00:50:27.426 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:usPv3rKOiGP9d0gqJGz9FJkz3fs= 2023-07-29T00:50:27.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:usPv3rKOiGP9d0gqJGz9FJkz3fs=', 'Content-Length': '0'} 2023-07-29T00:50:27.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ac8e5756b7fce5d-0064c46253-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:27.429 DEBUG:boto:b'hiiqos-62dummy1000falsedummy2023-07-29T00:50:06.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:27.430 DEBUG:boto:path=/hiiqos-62/dummy 2023-07-29T00:50:27.430 DEBUG:boto:auth_path=/hiiqos-62/dummy 2023-07-29T00:50:27.431 DEBUG:boto:Method: HEAD 2023-07-29T00:50:27.431 DEBUG:boto:Path: /hiiqos-62/dummy 2023-07-29T00:50:27.431 DEBUG:boto:Data: 2023-07-29T00:50:27.432 DEBUG:boto:Headers: {} 2023-07-29T00:50:27.432 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:27.433 DEBUG:boto:Port: 8001 2023-07-29T00:50:27.433 DEBUG:boto:Params: {} 2023-07-29T00:50:27.433 DEBUG:boto:Token: None 2023-07-29T00:50:27.434 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:27 GMT /hiiqos-62/dummy 2023-07-29T00:50:27.434 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9qDC4+VxwlD4OxwPEq9ZYn8Tduo= 2023-07-29T00:50:27.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9qDC4+VxwlD4OxwPEq9ZYn8Tduo=', 'Content-Length': '0'} 2023-07-29T00:50:27.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-62:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000885e6691b9c871fa-0064c46253-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:27.437 DEBUG:boto:path=/hiiqos-62/dummy 2023-07-29T00:50:27.437 DEBUG:boto:auth_path=/hiiqos-62/dummy 2023-07-29T00:50:27.437 DEBUG:boto:Method: HEAD 2023-07-29T00:50:27.438 DEBUG:boto:Path: /hiiqos-62/dummy 2023-07-29T00:50:27.438 DEBUG:boto:Data: 2023-07-29T00:50:27.438 DEBUG:boto:Headers: {} 2023-07-29T00:50:27.439 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:27.439 DEBUG:boto:Port: 8000 2023-07-29T00:50:27.439 DEBUG:boto:Params: {} 2023-07-29T00:50:27.440 DEBUG:boto:Token: None 2023-07-29T00:50:27.440 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:27 GMT /hiiqos-62/dummy 2023-07-29T00:50:27.441 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9qDC4+VxwlD4OxwPEq9ZYn8Tduo= 2023-07-29T00:50:27.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9qDC4+VxwlD4OxwPEq9ZYn8Tduo=', 'Content-Length': '0'} 2023-07-29T00:50:27.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c28569ee7fd178d-0064c46253-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:27.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-07-29T00:50:27.444 DEBUG:boto:path=/hiiqos-62/dummy 2023-07-29T00:50:27.444 DEBUG:boto:auth_path=/hiiqos-62/dummy 2023-07-29T00:50:27.445 DEBUG:boto:Method: GET 2023-07-29T00:50:27.445 DEBUG:boto:Path: /hiiqos-62/dummy 2023-07-29T00:50:27.445 DEBUG:boto:Data: 2023-07-29T00:50:27.446 DEBUG:boto:Headers: {} 2023-07-29T00:50:27.446 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:27.446 DEBUG:boto:Port: 8001 2023-07-29T00:50:27.447 DEBUG:boto:Params: {} 2023-07-29T00:50:27.447 DEBUG:boto:Token: None 2023-07-29T00:50:27.448 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:27 GMT /hiiqos-62/dummy 2023-07-29T00:50:27.448 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vAiqUxMg9VwPX+tHVanlPVYQ+28= 2023-07-29T00:50:27.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vAiqUxMg9VwPX+tHVanlPVYQ+28=', 'Content-Length': '0'} 2023-07-29T00:50:27.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-62:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000912d4d40a5a2b699-0064c46253-2008-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:27.492 DEBUG:boto:path=/hiiqos-62/dummy 2023-07-29T00:50:27.493 DEBUG:boto:auth_path=/hiiqos-62/dummy 2023-07-29T00:50:27.493 DEBUG:boto:Method: GET 2023-07-29T00:50:27.493 DEBUG:boto:Path: /hiiqos-62/dummy 2023-07-29T00:50:27.494 DEBUG:boto:Data: 2023-07-29T00:50:27.494 DEBUG:boto:Headers: {} 2023-07-29T00:50:27.495 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:27.495 DEBUG:boto:Port: 8000 2023-07-29T00:50:27.495 DEBUG:boto:Params: {} 2023-07-29T00:50:27.496 DEBUG:boto:Token: None 2023-07-29T00:50:27.496 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:27 GMT /hiiqos-62/dummy 2023-07-29T00:50:27.497 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vAiqUxMg9VwPX+tHVanlPVYQ+28= 2023-07-29T00:50:27.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vAiqUxMg9VwPX+tHVanlPVYQ+28=', 'Content-Length': '0'} 2023-07-29T00:50:27.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c1e1b12688f9f85-0064c46253-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:27.500 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:50:27.500 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:50:27.500 DEBUG:boto:path=/hiiqos-62/?prefix=dummy 2023-07-29T00:50:27.501 DEBUG:boto:auth_path=/hiiqos-62/?prefix=dummy 2023-07-29T00:50:27.501 DEBUG:boto:Method: GET 2023-07-29T00:50:27.502 DEBUG:boto:Path: /hiiqos-62/?prefix=dummy 2023-07-29T00:50:27.502 DEBUG:boto:Data: 2023-07-29T00:50:27.502 DEBUG:boto:Headers: {} 2023-07-29T00:50:27.503 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:27.503 DEBUG:boto:Port: 8001 2023-07-29T00:50:27.503 DEBUG:boto:Params: {} 2023-07-29T00:50:27.504 DEBUG:boto:Token: None 2023-07-29T00:50:27.504 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:27 GMT /hiiqos-62/ 2023-07-29T00:50:27.505 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:usPv3rKOiGP9d0gqJGz9FJkz3fs= 2023-07-29T00:50:27.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:usPv3rKOiGP9d0gqJGz9FJkz3fs=', 'Content-Length': '0'} 2023-07-29T00:50:27.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096efde46e016fbe3-0064c46253-2008-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:27.508 DEBUG:boto:b'hiiqos-62dummy1000falsedummy2023-07-29T00:50:06.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:27.509 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:50:27.509 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:50:27.509 DEBUG:boto:path=/hiiqos-62/?prefix=dummy 2023-07-29T00:50:27.510 DEBUG:boto:auth_path=/hiiqos-62/?prefix=dummy 2023-07-29T00:50:27.510 DEBUG:boto:Method: GET 2023-07-29T00:50:27.510 DEBUG:boto:Path: /hiiqos-62/?prefix=dummy 2023-07-29T00:50:27.511 DEBUG:boto:Data: 2023-07-29T00:50:27.511 DEBUG:boto:Headers: {} 2023-07-29T00:50:27.512 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:27.512 DEBUG:boto:Port: 8000 2023-07-29T00:50:27.512 DEBUG:boto:Params: {} 2023-07-29T00:50:27.513 DEBUG:boto:Token: None 2023-07-29T00:50:27.513 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:50:27 GMT /hiiqos-62/ 2023-07-29T00:50:27.513 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:usPv3rKOiGP9d0gqJGz9FJkz3fs= 2023-07-29T00:50:27.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:usPv3rKOiGP9d0gqJGz9FJkz3fs=', 'Content-Length': '0'} 2023-07-29T00:50:27.516 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cb3af681eb33abd-0064c46253-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:50:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:27.517 DEBUG:boto:b'hiiqos-62dummy1000falsedummy2023-07-29T00:50:06.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:50:27.517 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-62 zones={a2, a1} 2023-07-29T00:50:27.518 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-07-29T00:50:27.519 INFO:teuthology.orchestra.run:waiting for 300 2023-07-29T00:50:27.520 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:50:27.446+0000 7f947e4c2640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 143583) UID: 0 2023-07-29T00:50:27.520 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:50:27.446+0000 7f9484634880 -1 shutting down 2023-07-29T00:50:30.265 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-07-29T00:50:33.617 INFO:tasks.rgw.c1.client.1:Stopped 2023-07-29T00:50:33.618 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:50:33.618 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:50:33.619 DEBUG:boto:Method: HEAD 2023-07-29T00:50:33.619 DEBUG:boto:Path: /hiiqos-62/ 2023-07-29T00:50:33.620 DEBUG:boto:Data: 2023-07-29T00:50:33.620 DEBUG:boto:Headers: {} 2023-07-29T00:50:33.620 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:33.621 DEBUG:boto:Port: 8000 2023-07-29T00:50:33.621 DEBUG:boto:Params: {} 2023-07-29T00:50:33.621 DEBUG:boto:Token: None 2023-07-29T00:50:33.622 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:33 GMT /hiiqos-62/ 2023-07-29T00:50:33.622 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:l8gCEWKQmCIe4Wrruh3GQjrkOL4= 2023-07-29T00:50:33.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:l8gCEWKQmCIe4Wrruh3GQjrkOL4=', 'Content-Length': '0'} 2023-07-29T00:50:33.625 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004143b84484df9ad7-0064c46259-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:33.626 DEBUG:boto:path=/hiiqos-62/obj11 2023-07-29T00:50:33.627 DEBUG:boto:auth_path=/hiiqos-62/obj11 2023-07-29T00:50:33.627 DEBUG:boto:Method: PUT 2023-07-29T00:50:33.628 DEBUG:boto:Path: /hiiqos-62/obj11 2023-07-29T00:50:33.629 DEBUG:boto:Data: 2023-07-29T00:50:33.629 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:50:33.630 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:33.631 DEBUG:boto:Port: 8000 2023-07-29T00:50:33.632 DEBUG:boto:Params: {} 2023-07-29T00:50:33.632 DEBUG:boto:Token: None 2023-07-29T00:50:33.633 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:50:33 GMT /hiiqos-62/obj11 2023-07-29T00:50:33.634 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xED54wDtsCGr2dvElgsSGfIjLBQ= 2023-07-29T00:50:33.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:50:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xED54wDtsCGr2dvElgsSGfIjLBQ='} 2023-07-29T00:50:33.680 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000045226e6b9a763b15-0064c46259-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:33.681 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:50:33.682 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:50:33.682 DEBUG:boto:Method: HEAD 2023-07-29T00:50:33.682 DEBUG:boto:Path: /hiiqos-62/ 2023-07-29T00:50:33.683 DEBUG:boto:Data: 2023-07-29T00:50:33.683 DEBUG:boto:Headers: {} 2023-07-29T00:50:33.683 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:33.684 DEBUG:boto:Port: 8000 2023-07-29T00:50:33.684 DEBUG:boto:Params: {} 2023-07-29T00:50:33.684 DEBUG:boto:Token: None 2023-07-29T00:50:33.685 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:33 GMT /hiiqos-62/ 2023-07-29T00:50:33.685 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:l8gCEWKQmCIe4Wrruh3GQjrkOL4= 2023-07-29T00:50:33.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:l8gCEWKQmCIe4Wrruh3GQjrkOL4=', 'Content-Length': '0'} 2023-07-29T00:50:33.688 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009dd6111e2aa5fbc2-0064c46259-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:33.689 DEBUG:boto:path=/hiiqos-62/obj22 2023-07-29T00:50:33.689 DEBUG:boto:auth_path=/hiiqos-62/obj22 2023-07-29T00:50:33.689 DEBUG:boto:Method: PUT 2023-07-29T00:50:33.690 DEBUG:boto:Path: /hiiqos-62/obj22 2023-07-29T00:50:33.690 DEBUG:boto:Data: 2023-07-29T00:50:33.690 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:50:33.691 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:33.691 DEBUG:boto:Port: 8000 2023-07-29T00:50:33.691 DEBUG:boto:Params: {} 2023-07-29T00:50:33.692 DEBUG:boto:Token: None 2023-07-29T00:50:33.692 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:50:33 GMT /hiiqos-62/obj22 2023-07-29T00:50:33.693 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8ZkyygpE9Z4ZetahA2P57xJMdRM= 2023-07-29T00:50:33.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:50:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8ZkyygpE9Z4ZetahA2P57xJMdRM='} 2023-07-29T00:50:33.698 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004a6e251c60d5a2ae-0064c46259-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:33.698 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:50:33.699 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:50:33.699 DEBUG:boto:Method: HEAD 2023-07-29T00:50:33.700 DEBUG:boto:Path: /hiiqos-62/ 2023-07-29T00:50:33.700 DEBUG:boto:Data: 2023-07-29T00:50:33.700 DEBUG:boto:Headers: {} 2023-07-29T00:50:33.701 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:33.701 DEBUG:boto:Port: 8000 2023-07-29T00:50:33.701 DEBUG:boto:Params: {} 2023-07-29T00:50:33.702 DEBUG:boto:Token: None 2023-07-29T00:50:33.702 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:33 GMT /hiiqos-62/ 2023-07-29T00:50:33.702 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:l8gCEWKQmCIe4Wrruh3GQjrkOL4= 2023-07-29T00:50:33.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:l8gCEWKQmCIe4Wrruh3GQjrkOL4=', 'Content-Length': '0'} 2023-07-29T00:50:33.705 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000609197876aa18d51-0064c46259-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:33.706 DEBUG:boto:path=/hiiqos-62/obj33 2023-07-29T00:50:33.706 DEBUG:boto:auth_path=/hiiqos-62/obj33 2023-07-29T00:50:33.707 DEBUG:boto:Method: PUT 2023-07-29T00:50:33.707 DEBUG:boto:Path: /hiiqos-62/obj33 2023-07-29T00:50:33.707 DEBUG:boto:Data: 2023-07-29T00:50:33.708 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:50:33.708 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:33.709 DEBUG:boto:Port: 8000 2023-07-29T00:50:33.709 DEBUG:boto:Params: {} 2023-07-29T00:50:33.709 DEBUG:boto:Token: None 2023-07-29T00:50:33.710 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:50:33 GMT /hiiqos-62/obj33 2023-07-29T00:50:33.710 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:d+UQ1DtzwGY47wHOVJuOX6ZrV+o= 2023-07-29T00:50:33.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:50:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:d+UQ1DtzwGY47wHOVJuOX6ZrV+o='} 2023-07-29T00:50:33.752 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e267bb3c06cf8db6-0064c46259-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:33.753 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:50:33.753 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:50:33.754 DEBUG:boto:Method: HEAD 2023-07-29T00:50:33.754 DEBUG:boto:Path: /hiiqos-62/ 2023-07-29T00:50:33.755 DEBUG:boto:Data: 2023-07-29T00:50:33.755 DEBUG:boto:Headers: {} 2023-07-29T00:50:33.755 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:33.756 DEBUG:boto:Port: 8000 2023-07-29T00:50:33.756 DEBUG:boto:Params: {} 2023-07-29T00:50:33.756 DEBUG:boto:Token: None 2023-07-29T00:50:33.757 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:33 GMT /hiiqos-62/ 2023-07-29T00:50:33.757 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:l8gCEWKQmCIe4Wrruh3GQjrkOL4= 2023-07-29T00:50:33.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:l8gCEWKQmCIe4Wrruh3GQjrkOL4=', 'Content-Length': '0'} 2023-07-29T00:50:33.760 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008222bdbe652563f2-0064c46259-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:33.761 DEBUG:boto:path=/hiiqos-62/obj44 2023-07-29T00:50:33.761 DEBUG:boto:auth_path=/hiiqos-62/obj44 2023-07-29T00:50:33.761 DEBUG:boto:Method: PUT 2023-07-29T00:50:33.762 DEBUG:boto:Path: /hiiqos-62/obj44 2023-07-29T00:50:33.762 DEBUG:boto:Data: 2023-07-29T00:50:33.762 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:50:33.763 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:33.763 DEBUG:boto:Port: 8000 2023-07-29T00:50:33.763 DEBUG:boto:Params: {} 2023-07-29T00:50:33.764 DEBUG:boto:Token: None 2023-07-29T00:50:33.764 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:50:33 GMT /hiiqos-62/obj44 2023-07-29T00:50:33.765 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CamHZ6CHMSnCOFWmHexPXP/1XAA= 2023-07-29T00:50:33.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:50:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CamHZ6CHMSnCOFWmHexPXP/1XAA='} 2023-07-29T00:50:33.771 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cf1a0922b036b064-0064c46259-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:33.771 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:50:33.772 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:50:33.772 DEBUG:boto:Method: HEAD 2023-07-29T00:50:33.772 DEBUG:boto:Path: /hiiqos-62/ 2023-07-29T00:50:33.773 DEBUG:boto:Data: 2023-07-29T00:50:33.773 DEBUG:boto:Headers: {} 2023-07-29T00:50:33.773 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:33.774 DEBUG:boto:Port: 8000 2023-07-29T00:50:33.774 DEBUG:boto:Params: {} 2023-07-29T00:50:33.774 DEBUG:boto:Token: None 2023-07-29T00:50:33.775 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:50:33 GMT /hiiqos-62/ 2023-07-29T00:50:33.775 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:l8gCEWKQmCIe4Wrruh3GQjrkOL4= 2023-07-29T00:50:33.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:50:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:l8gCEWKQmCIe4Wrruh3GQjrkOL4=', 'Content-Length': '0'} 2023-07-29T00:50:33.778 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d377254818880835-0064c46259-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:33.779 DEBUG:boto:path=/hiiqos-62/obj55 2023-07-29T00:50:33.779 DEBUG:boto:auth_path=/hiiqos-62/obj55 2023-07-29T00:50:33.780 DEBUG:boto:Method: PUT 2023-07-29T00:50:33.780 DEBUG:boto:Path: /hiiqos-62/obj55 2023-07-29T00:50:33.781 DEBUG:boto:Data: 2023-07-29T00:50:33.781 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:50:33.781 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:50:33.782 DEBUG:boto:Port: 8000 2023-07-29T00:50:33.782 DEBUG:boto:Params: {} 2023-07-29T00:50:33.782 DEBUG:boto:Token: None 2023-07-29T00:50:33.783 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:50:33 GMT /hiiqos-62/obj55 2023-07-29T00:50:33.783 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Y4fD7bxWypgZ7ERrJ5PQACBHT7k= 2023-07-29T00:50:33.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:50:33 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Y4fD7bxWypgZ7ERrJ5PQACBHT7k='} 2023-07-29T00:50:33.788 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000097daa61e4838079f-0064c46259-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:50:33.789 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:50:33.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:50:33.790 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:50:33.843 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:50:33.844 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:50:33.876 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:50:33.803+0000 7f336b735880 5 RGW-SYNC:data: waiting on data sync status lease 2023-07-29T00:50:33.877 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:50:33.804+0000 7f336b735880 5 RGW-SYNC:data: acquired data sync status lease 2023-07-29T00:50:33.899 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-07-29T00:50:33.900 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-07-29T00:50:33.902 INFO:tasks.rgw.c1.client.1:Started 2023-07-29T00:50:33.903 DEBUG:teuthology.orchestra.run.smithi101:> curl http://smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:33.921 INFO:teuthology.orchestra.run.smithi101.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-07-29T00:50:33.921 INFO:teuthology.orchestra.run.smithi101.stderr: Dload Upload Total Spent Left Speed 2023-07-29T00:50:33.922 INFO:teuthology.orchestra.run.smithi101.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-07-29T00:50:33.923 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-07-29T00:50:33.923 INFO:teuthology.orchestra.run.smithi101.stderr:curl: (7) Failed to connect to smithi101.front.sepia.ceph.com port 8001: Connection refused 2023-07-29T00:50:34.924 DEBUG:teuthology.orchestra.run.smithi101:> curl http://smithi101.front.sepia.ceph.com:8001 2023-07-29T00:50:34.942 INFO:teuthology.orchestra.run.smithi101.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-07-29T00:50:34.943 INFO:teuthology.orchestra.run.smithi101.stderr: Dload Upload Total Spent Left Speed 2023-07-29T00:50:34.944 INFO:teuthology.orchestra.run.smithi101.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-07-29T00:50:34.944 INFO:teuthology.orchestra.run.smithi101.stdout:anonymous 2023-07-29T00:50:34.945 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-62 2023-07-29T00:50:34.945 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:50:34.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:50:34.946 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-62 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:50:35.037 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:50:35.037 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:50:35.072 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:50:34.999+0000 7fa107a20880 1 waiting for incremental sync to catch up: 2023-07-29T00:50:35.072 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000001.157.6, , , , , ] 2023-07-29T00:50:35.072 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000001.157.6, , , 00000000001.59.6, 00000000002.109.6, 00000000002.89.6] 2023-07-29T00:50:35.970 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-07-29T00:50:40.073 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:50:40.000+0000 7fa107a20880 1 waiting for incremental sync to catch up: 2023-07-29T00:50:40.074 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000001.157.6, , , , , ] 2023-07-29T00:50:40.074 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000001.157.6, , , 00000000001.59.6, 00000000002.109.6, 00000000002.89.6] 2023-07-29T00:50:45.074 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:50:45.001+0000 7fa107a20880 1 waiting for incremental sync to catch up: 2023-07-29T00:50:45.075 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000001.157.6, , , , , ] 2023-07-29T00:50:45.075 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000001.157.6, , , 00000000001.59.6, 00000000002.109.6, 00000000002.89.6] 2023-07-29T00:50:50.076 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:50:50.003+0000 7fa107a20880 1 waiting for incremental sync to catch up: 2023-07-29T00:50:50.076 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000001.157.6, , , , , ] 2023-07-29T00:50:50.076 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000001.157.6, , , 00000000001.59.6, 00000000002.109.6, 00000000002.89.6] 2023-07-29T00:50:55.077 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:50:55.004+0000 7fa107a20880 1 waiting for incremental sync to catch up: 2023-07-29T00:50:55.077 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000001.157.6, , , , , ] 2023-07-29T00:50:55.078 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000001.157.6, , , 00000000001.59.6, 00000000002.109.6, 00000000002.89.6] 2023-07-29T00:51:00.079 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:51:00.005+0000 7fa107a20880 1 waiting for incremental sync to catch up: 2023-07-29T00:51:00.079 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000001.157.6, , , , , ] 2023-07-29T00:51:00.080 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000001.157.6, , , 00000000001.59.6, 00000000002.109.6, 00000000002.89.6] 2023-07-29T00:51:05.080 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:51:05.007+0000 7fa107a20880 1 waiting for incremental sync to catch up: 2023-07-29T00:51:05.080 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000001.157.6, , , , , ] 2023-07-29T00:51:05.081 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000001.157.6, , , 00000000001.59.6, 00000000002.109.6, 00000000002.89.6] 2023-07-29T00:51:10.081 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:51:10.008+0000 7fa107a20880 1 waiting for incremental sync to catch up: 2023-07-29T00:51:10.082 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000001.157.6, , , , , ] 2023-07-29T00:51:10.082 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000001.157.6, , , 00000000001.59.6, 00000000002.109.6, 00000000002.89.6] 2023-07-29T00:51:15.083 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:51:15.009+0000 7fa107a20880 1 waiting for incremental sync to catch up: 2023-07-29T00:51:15.083 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000001.157.6, , , , , ] 2023-07-29T00:51:15.083 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000001.157.6, , , 00000000001.59.6, 00000000002.109.6, 00000000002.89.6] 2023-07-29T00:51:20.084 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:51:20.010+0000 7fa107a20880 1 waiting for incremental sync to catch up: 2023-07-29T00:51:20.084 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000001.157.6, , , , , ] 2023-07-29T00:51:20.084 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000001.157.6, , , 00000000001.59.6, 00000000002.109.6, 00000000002.89.6] 2023-07-29T00:51:25.085 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:51:25.012+0000 7fa107a20880 1 waiting for incremental sync to catch up: 2023-07-29T00:51:25.086 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000001.157.6, , , , , ] 2023-07-29T00:51:25.086 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000001.157.6, , , 00000000001.59.6, 00000000002.109.6, 00000000002.89.6] 2023-07-29T00:51:30.087 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:51:30.013+0000 7fa107a20880 1 waiting for incremental sync to catch up: 2023-07-29T00:51:30.087 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000001.157.6, , , , , ] 2023-07-29T00:51:30.088 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000001.157.6, , , 00000000001.59.6, 00000000002.109.6, 00000000002.89.6] 2023-07-29T00:51:35.088 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:51:35.015+0000 7fa107a20880 1 waiting for incremental sync to catch up: 2023-07-29T00:51:35.089 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000001.157.6, , , , , ] 2023-07-29T00:51:35.089 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000001.157.6, , , 00000000001.59.6, 00000000002.109.6, 00000000002.89.6] 2023-07-29T00:51:40.090 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:51:40.016+0000 7fa107a20880 1 waiting for incremental sync to catch up: 2023-07-29T00:51:40.090 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000001.157.6, , , , , ] 2023-07-29T00:51:40.090 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000001.157.6, , , 00000000001.59.6, 00000000002.109.6, 00000000002.89.6] 2023-07-29T00:51:45.091 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:51:45.017+0000 7fa107a20880 1 waiting for incremental sync to catch up: 2023-07-29T00:51:45.092 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000001.157.6, , , , , ] 2023-07-29T00:51:45.092 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000001.157.6, , , 00000000001.59.6, 00000000002.109.6, 00000000002.89.6] 2023-07-29T00:51:50.093 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:51:50.019+0000 7fa107a20880 1 waiting for incremental sync to catch up: 2023-07-29T00:51:50.093 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000001.157.6, , , , , ] 2023-07-29T00:51:50.093 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000001.157.6, , , 00000000001.59.6, 00000000002.109.6, 00000000002.89.6] 2023-07-29T00:51:55.094 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:51:55.020+0000 7fa107a20880 1 waiting for incremental sync to catch up: 2023-07-29T00:51:55.095 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000001.157.6, , , , , ] 2023-07-29T00:51:55.095 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000001.157.6, , , 00000000001.59.6, 00000000002.109.6, 00000000002.89.6] 2023-07-29T00:52:00.096 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:52:00.021+0000 7fa107a20880 1 waiting for incremental sync to catch up: 2023-07-29T00:52:00.096 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000001.157.6, , , , , ] 2023-07-29T00:52:00.096 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000001.157.6, , , 00000000001.59.6, 00000000002.109.6, 00000000002.89.6] 2023-07-29T00:52:05.096 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:52:05.022+0000 7fa107a20880 1 waiting for incremental sync to catch up: 2023-07-29T00:52:05.097 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000001.157.6, , , 00000000001.59.6, 00000000002.109.6, 00000000001.87.6] 2023-07-29T00:52:05.097 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000001.157.6, , , 00000000001.59.6, 00000000002.109.6, 00000000002.89.6] 2023-07-29T00:52:05.631 INFO:tasks.rgw.c1.client.0.smithi101.stdout:2023-07-29T00:52:05.557+0000 7f0081a5e640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T00:52:10.098 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:52:10.024+0000 7fa107a20880 1 bucket sync caught up with source: 2023-07-29T00:52:10.098 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000001.157.6, , , 00000000001.59.6, 00000000002.109.6, 00000000002.89.6] 2023-07-29T00:52:10.098 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000001.157.6, , , 00000000001.59.6, 00000000002.109.6, 00000000002.89.6] 2023-07-29T00:52:10.099 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:52:10.024+0000 7fa107a20880 0 bucket checkpoint complete 2023-07-29T00:52:10.116 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-62 2023-07-29T00:52:10.117 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:52:10.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:52:10.119 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-62 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:52:10.179 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:52:10.179 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:52:10.214 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:52:10.140+0000 7f0f4672e880 1 waiting for incremental sync to catch up: 2023-07-29T00:52:10.214 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000001.840.6, , , , 00000000002.1684.6, 00000000001.1004.6] 2023-07-29T00:52:10.214 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000001.840.6, , , 00000000001.727.6, 00000000002.1684.6, 00000000002.1006.6] 2023-07-29T00:52:15.215 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:52:15.141+0000 7f0f4672e880 1 waiting for incremental sync to catch up: 2023-07-29T00:52:15.215 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000001.840.6, , , , 00000000002.1684.6, 00000000001.1004.6] 2023-07-29T00:52:15.216 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000001.840.6, , , 00000000001.727.6, 00000000002.1684.6, 00000000002.1006.6] 2023-07-29T00:52:20.216 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:52:20.142+0000 7f0f4672e880 1 waiting for incremental sync to catch up: 2023-07-29T00:52:20.217 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000001.840.6, , , , 00000000002.1684.6, 00000000001.1004.6] 2023-07-29T00:52:20.217 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000001.840.6, , , 00000000001.727.6, 00000000002.1684.6, 00000000002.1006.6] 2023-07-29T00:52:25.218 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:52:25.143+0000 7f0f4672e880 1 waiting for incremental sync to catch up: 2023-07-29T00:52:25.218 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000001.840.6, , , 00000000001.727.6, 00000000002.1684.6, 00000000001.1004.6] 2023-07-29T00:52:25.218 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000001.840.6, , , 00000000001.727.6, 00000000002.1684.6, 00000000002.1006.6] 2023-07-29T00:52:30.219 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:52:30.145+0000 7f0f4672e880 1 bucket sync caught up with source: 2023-07-29T00:52:30.219 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , 00000000001.840.6, , , 00000000001.727.6, 00000000002.1684.6, 00000000002.1006.6] 2023-07-29T00:52:30.219 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , 00000000001.840.6, , , 00000000001.727.6, 00000000002.1684.6, 00000000002.1006.6] 2023-07-29T00:52:30.220 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:52:30.145+0000 7f0f4672e880 0 bucket checkpoint complete 2023-07-29T00:52:30.237 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-62 zones={a2, a1} 2023-07-29T00:52:30.238 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:52:30.238 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:52:30.239 DEBUG:boto:Method: HEAD 2023-07-29T00:52:30.239 DEBUG:boto:Path: /hiiqos-62/ 2023-07-29T00:52:30.239 DEBUG:boto:Data: 2023-07-29T00:52:30.240 DEBUG:boto:Headers: {} 2023-07-29T00:52:30.240 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:30.240 DEBUG:boto:Port: 8001 2023-07-29T00:52:30.241 DEBUG:boto:Params: {} 2023-07-29T00:52:30.241 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-29T00:52:30.242 DEBUG:boto:Token: None 2023-07-29T00:52:30.242 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:30 GMT /hiiqos-62/ 2023-07-29T00:52:30.243 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CQILnwQ2UzbAoZCsG+GWpR48BOA= 2023-07-29T00:52:30.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:30 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CQILnwQ2UzbAoZCsG+GWpR48BOA=', 'Content-Length': '0'} 2023-07-29T00:52:35.250 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ca40af661077456e-0064c462d3-2068-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:35 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:35.250 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:52:35.251 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:52:35.251 DEBUG:boto:Method: HEAD 2023-07-29T00:52:35.252 DEBUG:boto:Path: /hiiqos-62/ 2023-07-29T00:52:35.252 DEBUG:boto:Data: 2023-07-29T00:52:35.253 DEBUG:boto:Headers: {} 2023-07-29T00:52:35.253 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:35.254 DEBUG:boto:Port: 8000 2023-07-29T00:52:35.254 DEBUG:boto:Params: {} 2023-07-29T00:52:35.255 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-29T00:52:35.255 DEBUG:boto:Token: None 2023-07-29T00:52:35.256 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:35 GMT /hiiqos-62/ 2023-07-29T00:52:35.257 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7Gzuaf6CgnxL7nj3YxOLiYsBlKo= 2023-07-29T00:52:35.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:35 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7Gzuaf6CgnxL7nj3YxOLiYsBlKo=', 'Content-Length': '0'} 2023-07-29T00:52:40.249 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000de574cf2719fca26-0064c462d8-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.250 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-29T00:52:40.251 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:52:40.251 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:52:40.252 DEBUG:boto:path=/hiiqos-62/?versions&max-keys=999 2023-07-29T00:52:40.252 DEBUG:boto:auth_path=/hiiqos-62/?versions&max-keys=999 2023-07-29T00:52:40.252 DEBUG:boto:Method: GET 2023-07-29T00:52:40.253 DEBUG:boto:Path: /hiiqos-62/?versions&max-keys=999 2023-07-29T00:52:40.253 DEBUG:boto:Data: 2023-07-29T00:52:40.254 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.254 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:40.255 DEBUG:boto:Port: 8001 2023-07-29T00:52:40.255 DEBUG:boto:Params: {} 2023-07-29T00:52:40.255 DEBUG:boto:Token: None 2023-07-29T00:52:40.256 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/?versions 2023-07-29T00:52:40.256 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mSeU9KaghraNI62I+HM82hmtq/Y= 2023-07-29T00:52:40.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mSeU9KaghraNI62I+HM82hmtq/Y=', 'Content-Length': '0'} 2023-07-29T00:52:40.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da9237ce00d5fc3b-0064c462d8-2068-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.260 DEBUG:boto:b'hiiqos-62999falsedummynulltrue2023-07-29T00:50:06.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2023-07-29T00:50:33.566Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:50:33.623Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:50:33.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:50:33.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:50:33.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:52:40.261 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-07-29T00:52:40.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-07-29T00:52:40.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-07-29T00:52:40.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-07-29T00:52:40.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-07-29T00:52:40.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-07-29T00:52:40.264 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-29T00:52:40.264 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:52:40.265 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:52:40.265 DEBUG:boto:path=/hiiqos-62/?versions&max-keys=999 2023-07-29T00:52:40.265 DEBUG:boto:auth_path=/hiiqos-62/?versions&max-keys=999 2023-07-29T00:52:40.266 DEBUG:boto:Method: GET 2023-07-29T00:52:40.266 DEBUG:boto:Path: /hiiqos-62/?versions&max-keys=999 2023-07-29T00:52:40.266 DEBUG:boto:Data: 2023-07-29T00:52:40.267 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.267 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:40.268 DEBUG:boto:Port: 8000 2023-07-29T00:52:40.268 DEBUG:boto:Params: {} 2023-07-29T00:52:40.268 DEBUG:boto:Token: None 2023-07-29T00:52:40.269 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/?versions 2023-07-29T00:52:40.269 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mSeU9KaghraNI62I+HM82hmtq/Y= 2023-07-29T00:52:40.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mSeU9KaghraNI62I+HM82hmtq/Y=', 'Content-Length': '0'} 2023-07-29T00:52:40.272 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000915acc35fac10a3f-0064c462d8-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.273 DEBUG:boto:b'hiiqos-62999falsedummynulltrue2023-07-29T00:50:06.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2023-07-29T00:50:33.566Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:50:33.623Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:50:33.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:50:33.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:50:33.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:52:40.274 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-07-29T00:52:40.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-07-29T00:52:40.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-07-29T00:52:40.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-07-29T00:52:40.276 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-07-29T00:52:40.276 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-07-29T00:52:40.276 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:52:40.277 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:52:40.277 DEBUG:boto:path=/hiiqos-62/?versions&max-keys=999 2023-07-29T00:52:40.277 DEBUG:boto:auth_path=/hiiqos-62/?versions&max-keys=999 2023-07-29T00:52:40.278 DEBUG:boto:Method: GET 2023-07-29T00:52:40.278 DEBUG:boto:Path: /hiiqos-62/?versions&max-keys=999 2023-07-29T00:52:40.278 DEBUG:boto:Data: 2023-07-29T00:52:40.279 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.279 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:40.279 DEBUG:boto:Port: 8001 2023-07-29T00:52:40.280 DEBUG:boto:Params: {} 2023-07-29T00:52:40.280 DEBUG:boto:Token: None 2023-07-29T00:52:40.281 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/?versions 2023-07-29T00:52:40.281 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mSeU9KaghraNI62I+HM82hmtq/Y= 2023-07-29T00:52:40.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mSeU9KaghraNI62I+HM82hmtq/Y=', 'Content-Length': '0'} 2023-07-29T00:52:40.284 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003afecdb0f0e9c7f9-0064c462d8-2068-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.285 DEBUG:boto:b'hiiqos-62999falsedummynulltrue2023-07-29T00:50:06.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2023-07-29T00:50:33.566Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:50:33.623Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:50:33.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:50:33.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:50:33.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:52:40.286 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:52:40.286 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:52:40.286 DEBUG:boto:path=/hiiqos-62/?versions&max-keys=999 2023-07-29T00:52:40.287 DEBUG:boto:auth_path=/hiiqos-62/?versions&max-keys=999 2023-07-29T00:52:40.287 DEBUG:boto:Method: GET 2023-07-29T00:52:40.288 DEBUG:boto:Path: /hiiqos-62/?versions&max-keys=999 2023-07-29T00:52:40.288 DEBUG:boto:Data: 2023-07-29T00:52:40.288 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.289 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:40.289 DEBUG:boto:Port: 8000 2023-07-29T00:52:40.289 DEBUG:boto:Params: {} 2023-07-29T00:52:40.290 DEBUG:boto:Token: None 2023-07-29T00:52:40.290 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/?versions 2023-07-29T00:52:40.291 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mSeU9KaghraNI62I+HM82hmtq/Y= 2023-07-29T00:52:40.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mSeU9KaghraNI62I+HM82hmtq/Y=', 'Content-Length': '0'} 2023-07-29T00:52:40.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012a47c2c0f16bd7d-0064c462d8-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.296 DEBUG:boto:b'hiiqos-62999falsedummynulltrue2023-07-29T00:50:06.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2023-07-29T00:50:33.566Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:50:33.623Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:50:33.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:50:33.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:50:33.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:52:40.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-07-29T00:52:40.299 DEBUG:boto:path=/hiiqos-62/dummy 2023-07-29T00:52:40.300 DEBUG:boto:auth_path=/hiiqos-62/dummy 2023-07-29T00:52:40.301 DEBUG:boto:path=/hiiqos-62/dummy?versionId=null 2023-07-29T00:52:40.301 DEBUG:boto:auth_path=/hiiqos-62/dummy?versionId=null 2023-07-29T00:52:40.301 DEBUG:boto:Method: GET 2023-07-29T00:52:40.302 DEBUG:boto:Path: /hiiqos-62/dummy?versionId=null 2023-07-29T00:52:40.302 DEBUG:boto:Data: 2023-07-29T00:52:40.303 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.303 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:40.304 DEBUG:boto:Port: 8001 2023-07-29T00:52:40.304 DEBUG:boto:Params: {} 2023-07-29T00:52:40.305 DEBUG:boto:Token: None 2023-07-29T00:52:40.305 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/dummy?versionId=null 2023-07-29T00:52:40.306 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ta0I5FvMB5OVAiDkUgmBdYZUm9k= 2023-07-29T00:52:40.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ta0I5FvMB5OVAiDkUgmBdYZUm9k=', 'Content-Length': '0'} 2023-07-29T00:52:40.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-62:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eda0c7cdf6bbf16d-0064c462d8-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.310 DEBUG:boto:path=/hiiqos-62/dummy 2023-07-29T00:52:40.310 DEBUG:boto:auth_path=/hiiqos-62/dummy 2023-07-29T00:52:40.311 DEBUG:boto:path=/hiiqos-62/dummy?versionId=null 2023-07-29T00:52:40.311 DEBUG:boto:auth_path=/hiiqos-62/dummy?versionId=null 2023-07-29T00:52:40.311 DEBUG:boto:Method: GET 2023-07-29T00:52:40.312 DEBUG:boto:Path: /hiiqos-62/dummy?versionId=null 2023-07-29T00:52:40.312 DEBUG:boto:Data: 2023-07-29T00:52:40.312 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.313 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:40.313 DEBUG:boto:Port: 8000 2023-07-29T00:52:40.314 DEBUG:boto:Params: {} 2023-07-29T00:52:40.314 DEBUG:boto:Token: None 2023-07-29T00:52:40.315 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/dummy?versionId=null 2023-07-29T00:52:40.315 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ta0I5FvMB5OVAiDkUgmBdYZUm9k= 2023-07-29T00:52:40.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ta0I5FvMB5OVAiDkUgmBdYZUm9k=', 'Content-Length': '0'} 2023-07-29T00:52:40.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032fe655aeb6794e8-0064c462d8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.319 DEBUG:boto:path=/hiiqos-62/dummy 2023-07-29T00:52:40.319 DEBUG:boto:auth_path=/hiiqos-62/dummy 2023-07-29T00:52:40.319 DEBUG:boto:path=/hiiqos-62/dummy?versionId=null 2023-07-29T00:52:40.320 DEBUG:boto:auth_path=/hiiqos-62/dummy?versionId=null 2023-07-29T00:52:40.320 DEBUG:boto:Method: HEAD 2023-07-29T00:52:40.320 DEBUG:boto:Path: /hiiqos-62/dummy?versionId=null 2023-07-29T00:52:40.321 DEBUG:boto:Data: 2023-07-29T00:52:40.321 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.322 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:40.322 DEBUG:boto:Port: 8001 2023-07-29T00:52:40.322 DEBUG:boto:Params: {} 2023-07-29T00:52:40.323 DEBUG:boto:Token: None 2023-07-29T00:52:40.323 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/dummy?versionId=null 2023-07-29T00:52:40.323 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Q7e7bKkSKtqTp/xzinLqsZ/DlEM= 2023-07-29T00:52:40.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q7e7bKkSKtqTp/xzinLqsZ/DlEM=', 'Content-Length': '0'} 2023-07-29T00:52:40.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-62:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095e7ffe80d06165e-0064c462d8-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.326 DEBUG:boto:path=/hiiqos-62/dummy 2023-07-29T00:52:40.327 DEBUG:boto:auth_path=/hiiqos-62/dummy 2023-07-29T00:52:40.327 DEBUG:boto:path=/hiiqos-62/dummy?versionId=null 2023-07-29T00:52:40.328 DEBUG:boto:auth_path=/hiiqos-62/dummy?versionId=null 2023-07-29T00:52:40.328 DEBUG:boto:Method: HEAD 2023-07-29T00:52:40.329 DEBUG:boto:Path: /hiiqos-62/dummy?versionId=null 2023-07-29T00:52:40.329 DEBUG:boto:Data: 2023-07-29T00:52:40.329 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.330 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:40.330 DEBUG:boto:Port: 8000 2023-07-29T00:52:40.330 DEBUG:boto:Params: {} 2023-07-29T00:52:40.331 DEBUG:boto:Token: None 2023-07-29T00:52:40.331 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/dummy?versionId=null 2023-07-29T00:52:40.331 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Q7e7bKkSKtqTp/xzinLqsZ/DlEM= 2023-07-29T00:52:40.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q7e7bKkSKtqTp/xzinLqsZ/DlEM=', 'Content-Length': '0'} 2023-07-29T00:52:40.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f3915a61c94853f-0064c462d8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-07-29T00:52:40.335 DEBUG:boto:path=/hiiqos-62/dummy 2023-07-29T00:52:40.336 DEBUG:boto:auth_path=/hiiqos-62/dummy 2023-07-29T00:52:40.336 DEBUG:boto:path=/hiiqos-62/dummy?versionId=null 2023-07-29T00:52:40.337 DEBUG:boto:auth_path=/hiiqos-62/dummy?versionId=null 2023-07-29T00:52:40.337 DEBUG:boto:Method: GET 2023-07-29T00:52:40.338 DEBUG:boto:Path: /hiiqos-62/dummy?versionId=null 2023-07-29T00:52:40.338 DEBUG:boto:Data: 2023-07-29T00:52:40.338 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.339 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:40.339 DEBUG:boto:Port: 8001 2023-07-29T00:52:40.340 DEBUG:boto:Params: {} 2023-07-29T00:52:40.340 DEBUG:boto:Token: None 2023-07-29T00:52:40.340 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/dummy?versionId=null 2023-07-29T00:52:40.341 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ta0I5FvMB5OVAiDkUgmBdYZUm9k= 2023-07-29T00:52:40.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ta0I5FvMB5OVAiDkUgmBdYZUm9k=', 'Content-Length': '0'} 2023-07-29T00:52:40.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-62:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e98928a7b1773899-0064c462d8-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.344 DEBUG:boto:path=/hiiqos-62/dummy 2023-07-29T00:52:40.345 DEBUG:boto:auth_path=/hiiqos-62/dummy 2023-07-29T00:52:40.345 DEBUG:boto:path=/hiiqos-62/dummy?versionId=null 2023-07-29T00:52:40.346 DEBUG:boto:auth_path=/hiiqos-62/dummy?versionId=null 2023-07-29T00:52:40.346 DEBUG:boto:Method: GET 2023-07-29T00:52:40.347 DEBUG:boto:Path: /hiiqos-62/dummy?versionId=null 2023-07-29T00:52:40.347 DEBUG:boto:Data: 2023-07-29T00:52:40.348 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.349 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:40.349 DEBUG:boto:Port: 8000 2023-07-29T00:52:40.349 DEBUG:boto:Params: {} 2023-07-29T00:52:40.350 DEBUG:boto:Token: None 2023-07-29T00:52:40.350 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/dummy?versionId=null 2023-07-29T00:52:40.351 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ta0I5FvMB5OVAiDkUgmBdYZUm9k= 2023-07-29T00:52:40.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ta0I5FvMB5OVAiDkUgmBdYZUm9k=', 'Content-Length': '0'} 2023-07-29T00:52:40.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6872b04ce197b65-0064c462d8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.354 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:52:40.354 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:52:40.354 DEBUG:boto:path=/hiiqos-62/?prefix=dummy 2023-07-29T00:52:40.355 DEBUG:boto:auth_path=/hiiqos-62/?prefix=dummy 2023-07-29T00:52:40.355 DEBUG:boto:Method: GET 2023-07-29T00:52:40.356 DEBUG:boto:Path: /hiiqos-62/?prefix=dummy 2023-07-29T00:52:40.356 DEBUG:boto:Data: 2023-07-29T00:52:40.357 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.357 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:40.358 DEBUG:boto:Port: 8001 2023-07-29T00:52:40.358 DEBUG:boto:Params: {} 2023-07-29T00:52:40.359 DEBUG:boto:Token: None 2023-07-29T00:52:40.359 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/ 2023-07-29T00:52:40.360 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FR5UEZyoU0rbmbeXP6UnHKKuDgc= 2023-07-29T00:52:40.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FR5UEZyoU0rbmbeXP6UnHKKuDgc=', 'Content-Length': '0'} 2023-07-29T00:52:40.362 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cda5b0e0bb6b65d2-0064c462d8-2068-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.363 DEBUG:boto:b'hiiqos-62dummy1000falsedummy2023-07-29T00:50:06.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:52:40.363 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:52:40.364 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:52:40.364 DEBUG:boto:path=/hiiqos-62/?prefix=dummy 2023-07-29T00:52:40.365 DEBUG:boto:auth_path=/hiiqos-62/?prefix=dummy 2023-07-29T00:52:40.365 DEBUG:boto:Method: GET 2023-07-29T00:52:40.365 DEBUG:boto:Path: /hiiqos-62/?prefix=dummy 2023-07-29T00:52:40.366 DEBUG:boto:Data: 2023-07-29T00:52:40.366 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.366 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:40.367 DEBUG:boto:Port: 8000 2023-07-29T00:52:40.367 DEBUG:boto:Params: {} 2023-07-29T00:52:40.368 DEBUG:boto:Token: None 2023-07-29T00:52:40.369 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/ 2023-07-29T00:52:40.369 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FR5UEZyoU0rbmbeXP6UnHKKuDgc= 2023-07-29T00:52:40.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FR5UEZyoU0rbmbeXP6UnHKKuDgc=', 'Content-Length': '0'} 2023-07-29T00:52:40.373 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005cceffbc40b4c7c8-0064c462d8-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.420 DEBUG:boto:b'hiiqos-62dummy1000falsedummy2023-07-29T00:50:06.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:52:40.421 DEBUG:boto:path=/hiiqos-62/dummy 2023-07-29T00:52:40.422 DEBUG:boto:auth_path=/hiiqos-62/dummy 2023-07-29T00:52:40.422 DEBUG:boto:Method: HEAD 2023-07-29T00:52:40.423 DEBUG:boto:Path: /hiiqos-62/dummy 2023-07-29T00:52:40.423 DEBUG:boto:Data: 2023-07-29T00:52:40.423 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.424 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:40.424 DEBUG:boto:Port: 8001 2023-07-29T00:52:40.425 DEBUG:boto:Params: {} 2023-07-29T00:52:40.426 DEBUG:boto:Token: None 2023-07-29T00:52:40.427 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/dummy 2023-07-29T00:52:40.428 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Cl4thB0fiSNdorixKPXpBy56+7U= 2023-07-29T00:52:40.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Cl4thB0fiSNdorixKPXpBy56+7U=', 'Content-Length': '0'} 2023-07-29T00:52:40.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-62:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d984135af61898e8-0064c462d8-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.433 DEBUG:boto:path=/hiiqos-62/dummy 2023-07-29T00:52:40.433 DEBUG:boto:auth_path=/hiiqos-62/dummy 2023-07-29T00:52:40.434 DEBUG:boto:Method: HEAD 2023-07-29T00:52:40.435 DEBUG:boto:Path: /hiiqos-62/dummy 2023-07-29T00:52:40.435 DEBUG:boto:Data: 2023-07-29T00:52:40.436 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.436 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:40.437 DEBUG:boto:Port: 8000 2023-07-29T00:52:40.437 DEBUG:boto:Params: {} 2023-07-29T00:52:40.437 DEBUG:boto:Token: None 2023-07-29T00:52:40.438 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/dummy 2023-07-29T00:52:40.438 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Cl4thB0fiSNdorixKPXpBy56+7U= 2023-07-29T00:52:40.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Cl4thB0fiSNdorixKPXpBy56+7U=', 'Content-Length': '0'} 2023-07-29T00:52:40.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016d09e7759654e18-0064c462d8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-07-29T00:52:40.442 DEBUG:boto:path=/hiiqos-62/dummy 2023-07-29T00:52:40.442 DEBUG:boto:auth_path=/hiiqos-62/dummy 2023-07-29T00:52:40.442 DEBUG:boto:Method: GET 2023-07-29T00:52:40.443 DEBUG:boto:Path: /hiiqos-62/dummy 2023-07-29T00:52:40.443 DEBUG:boto:Data: 2023-07-29T00:52:40.444 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.444 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:40.445 DEBUG:boto:Port: 8001 2023-07-29T00:52:40.445 DEBUG:boto:Params: {} 2023-07-29T00:52:40.445 DEBUG:boto:Token: None 2023-07-29T00:52:40.446 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/dummy 2023-07-29T00:52:40.446 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BWAtgwv7zBG8YRERlL7LeLn8nKg= 2023-07-29T00:52:40.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BWAtgwv7zBG8YRERlL7LeLn8nKg=', 'Content-Length': '0'} 2023-07-29T00:52:40.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-62:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de9810ae288192c1-0064c462d8-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.449 DEBUG:boto:path=/hiiqos-62/dummy 2023-07-29T00:52:40.450 DEBUG:boto:auth_path=/hiiqos-62/dummy 2023-07-29T00:52:40.450 DEBUG:boto:Method: GET 2023-07-29T00:52:40.451 DEBUG:boto:Path: /hiiqos-62/dummy 2023-07-29T00:52:40.451 DEBUG:boto:Data: 2023-07-29T00:52:40.452 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.452 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:40.453 DEBUG:boto:Port: 8000 2023-07-29T00:52:40.453 DEBUG:boto:Params: {} 2023-07-29T00:52:40.454 DEBUG:boto:Token: None 2023-07-29T00:52:40.454 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/dummy 2023-07-29T00:52:40.454 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BWAtgwv7zBG8YRERlL7LeLn8nKg= 2023-07-29T00:52:40.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BWAtgwv7zBG8YRERlL7LeLn8nKg=', 'Content-Length': '0'} 2023-07-29T00:52:40.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061620703cfee7da4-0064c462d8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.459 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:52:40.460 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:52:40.461 DEBUG:boto:path=/hiiqos-62/?prefix=dummy 2023-07-29T00:52:40.462 DEBUG:boto:auth_path=/hiiqos-62/?prefix=dummy 2023-07-29T00:52:40.463 DEBUG:boto:Method: GET 2023-07-29T00:52:40.464 DEBUG:boto:Path: /hiiqos-62/?prefix=dummy 2023-07-29T00:52:40.464 DEBUG:boto:Data: 2023-07-29T00:52:40.465 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.466 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:40.466 DEBUG:boto:Port: 8001 2023-07-29T00:52:40.467 DEBUG:boto:Params: {} 2023-07-29T00:52:40.467 DEBUG:boto:Token: None 2023-07-29T00:52:40.468 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/ 2023-07-29T00:52:40.468 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FR5UEZyoU0rbmbeXP6UnHKKuDgc= 2023-07-29T00:52:40.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FR5UEZyoU0rbmbeXP6UnHKKuDgc=', 'Content-Length': '0'} 2023-07-29T00:52:40.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008659fb47619a4cdf-0064c462d8-2068-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.512 DEBUG:boto:b'hiiqos-62dummy1000falsedummy2023-07-29T00:50:06.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:52:40.513 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:52:40.515 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:52:40.516 DEBUG:boto:path=/hiiqos-62/?prefix=dummy 2023-07-29T00:52:40.516 DEBUG:boto:auth_path=/hiiqos-62/?prefix=dummy 2023-07-29T00:52:40.517 DEBUG:boto:Method: GET 2023-07-29T00:52:40.517 DEBUG:boto:Path: /hiiqos-62/?prefix=dummy 2023-07-29T00:52:40.517 DEBUG:boto:Data: 2023-07-29T00:52:40.518 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.518 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:40.519 DEBUG:boto:Port: 8000 2023-07-29T00:52:40.519 DEBUG:boto:Params: {} 2023-07-29T00:52:40.519 DEBUG:boto:Token: None 2023-07-29T00:52:40.520 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/ 2023-07-29T00:52:40.520 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FR5UEZyoU0rbmbeXP6UnHKKuDgc= 2023-07-29T00:52:40.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FR5UEZyoU0rbmbeXP6UnHKKuDgc=', 'Content-Length': '0'} 2023-07-29T00:52:40.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee01bc5c2fe74cdc-0064c462d8-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.524 DEBUG:boto:b'hiiqos-62dummy1000falsedummy2023-07-29T00:50:06.886Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:52:40.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-07-29T00:52:40.526 DEBUG:boto:path=/hiiqos-62/obj11 2023-07-29T00:52:40.526 DEBUG:boto:auth_path=/hiiqos-62/obj11 2023-07-29T00:52:40.527 DEBUG:boto:path=/hiiqos-62/obj11?versionId=null 2023-07-29T00:52:40.527 DEBUG:boto:auth_path=/hiiqos-62/obj11?versionId=null 2023-07-29T00:52:40.528 DEBUG:boto:Method: GET 2023-07-29T00:52:40.528 DEBUG:boto:Path: /hiiqos-62/obj11?versionId=null 2023-07-29T00:52:40.529 DEBUG:boto:Data: 2023-07-29T00:52:40.529 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.529 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:40.530 DEBUG:boto:Port: 8001 2023-07-29T00:52:40.530 DEBUG:boto:Params: {} 2023-07-29T00:52:40.531 DEBUG:boto:Token: None 2023-07-29T00:52:40.531 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/obj11?versionId=null 2023-07-29T00:52:40.531 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:10o6JmeeznIoBkwwfMCt0R3MiTQ= 2023-07-29T00:52:40.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:10o6JmeeznIoBkwwfMCt0R3MiTQ=', 'Content-Length': '0'} 2023-07-29T00:52:40.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-62:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a79c4b08ed9604b7-0064c462d8-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.535 DEBUG:boto:path=/hiiqos-62/obj11 2023-07-29T00:52:40.536 DEBUG:boto:auth_path=/hiiqos-62/obj11 2023-07-29T00:52:40.536 DEBUG:boto:path=/hiiqos-62/obj11?versionId=null 2023-07-29T00:52:40.536 DEBUG:boto:auth_path=/hiiqos-62/obj11?versionId=null 2023-07-29T00:52:40.537 DEBUG:boto:Method: GET 2023-07-29T00:52:40.537 DEBUG:boto:Path: /hiiqos-62/obj11?versionId=null 2023-07-29T00:52:40.538 DEBUG:boto:Data: 2023-07-29T00:52:40.538 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.538 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:40.539 DEBUG:boto:Port: 8000 2023-07-29T00:52:40.539 DEBUG:boto:Params: {} 2023-07-29T00:52:40.539 DEBUG:boto:Token: None 2023-07-29T00:52:40.540 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/obj11?versionId=null 2023-07-29T00:52:40.540 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:10o6JmeeznIoBkwwfMCt0R3MiTQ= 2023-07-29T00:52:40.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:10o6JmeeznIoBkwwfMCt0R3MiTQ=', 'Content-Length': '0'} 2023-07-29T00:52:40.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ceb2a2f32908d7bf-0064c462d8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.543 DEBUG:boto:path=/hiiqos-62/obj11 2023-07-29T00:52:40.544 DEBUG:boto:auth_path=/hiiqos-62/obj11 2023-07-29T00:52:40.544 DEBUG:boto:path=/hiiqos-62/obj11?versionId=null 2023-07-29T00:52:40.544 DEBUG:boto:auth_path=/hiiqos-62/obj11?versionId=null 2023-07-29T00:52:40.545 DEBUG:boto:Method: HEAD 2023-07-29T00:52:40.545 DEBUG:boto:Path: /hiiqos-62/obj11?versionId=null 2023-07-29T00:52:40.545 DEBUG:boto:Data: 2023-07-29T00:52:40.546 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.546 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:40.547 DEBUG:boto:Port: 8001 2023-07-29T00:52:40.547 DEBUG:boto:Params: {} 2023-07-29T00:52:40.547 DEBUG:boto:Token: None 2023-07-29T00:52:40.548 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/obj11?versionId=null 2023-07-29T00:52:40.548 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7q10acWR8A8QXs0yME/3jzxiKk0= 2023-07-29T00:52:40.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7q10acWR8A8QXs0yME/3jzxiKk0=', 'Content-Length': '0'} 2023-07-29T00:52:40.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-62:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090a3d50a256212cb-0064c462d8-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.551 DEBUG:boto:path=/hiiqos-62/obj11 2023-07-29T00:52:40.552 DEBUG:boto:auth_path=/hiiqos-62/obj11 2023-07-29T00:52:40.552 DEBUG:boto:path=/hiiqos-62/obj11?versionId=null 2023-07-29T00:52:40.552 DEBUG:boto:auth_path=/hiiqos-62/obj11?versionId=null 2023-07-29T00:52:40.553 DEBUG:boto:Method: HEAD 2023-07-29T00:52:40.553 DEBUG:boto:Path: /hiiqos-62/obj11?versionId=null 2023-07-29T00:52:40.553 DEBUG:boto:Data: 2023-07-29T00:52:40.554 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.554 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:40.554 DEBUG:boto:Port: 8000 2023-07-29T00:52:40.555 DEBUG:boto:Params: {} 2023-07-29T00:52:40.555 DEBUG:boto:Token: None 2023-07-29T00:52:40.556 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/obj11?versionId=null 2023-07-29T00:52:40.556 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7q10acWR8A8QXs0yME/3jzxiKk0= 2023-07-29T00:52:40.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7q10acWR8A8QXs0yME/3jzxiKk0=', 'Content-Length': '0'} 2023-07-29T00:52:40.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a03a95624b6f54f-0064c462d8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-07-29T00:52:40.559 DEBUG:boto:path=/hiiqos-62/obj11 2023-07-29T00:52:40.560 DEBUG:boto:auth_path=/hiiqos-62/obj11 2023-07-29T00:52:40.560 DEBUG:boto:path=/hiiqos-62/obj11?versionId=null 2023-07-29T00:52:40.561 DEBUG:boto:auth_path=/hiiqos-62/obj11?versionId=null 2023-07-29T00:52:40.561 DEBUG:boto:Method: GET 2023-07-29T00:52:40.561 DEBUG:boto:Path: /hiiqos-62/obj11?versionId=null 2023-07-29T00:52:40.562 DEBUG:boto:Data: 2023-07-29T00:52:40.562 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.563 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:40.563 DEBUG:boto:Port: 8001 2023-07-29T00:52:40.563 DEBUG:boto:Params: {} 2023-07-29T00:52:40.564 DEBUG:boto:Token: None 2023-07-29T00:52:40.564 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/obj11?versionId=null 2023-07-29T00:52:40.565 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:10o6JmeeznIoBkwwfMCt0R3MiTQ= 2023-07-29T00:52:40.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:10o6JmeeznIoBkwwfMCt0R3MiTQ=', 'Content-Length': '0'} 2023-07-29T00:52:40.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-62:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8716fc0bcfc891c-0064c462d8-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.568 DEBUG:boto:path=/hiiqos-62/obj11 2023-07-29T00:52:40.568 DEBUG:boto:auth_path=/hiiqos-62/obj11 2023-07-29T00:52:40.569 DEBUG:boto:path=/hiiqos-62/obj11?versionId=null 2023-07-29T00:52:40.569 DEBUG:boto:auth_path=/hiiqos-62/obj11?versionId=null 2023-07-29T00:52:40.570 DEBUG:boto:Method: GET 2023-07-29T00:52:40.570 DEBUG:boto:Path: /hiiqos-62/obj11?versionId=null 2023-07-29T00:52:40.570 DEBUG:boto:Data: 2023-07-29T00:52:40.571 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.571 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:40.571 DEBUG:boto:Port: 8000 2023-07-29T00:52:40.572 DEBUG:boto:Params: {} 2023-07-29T00:52:40.572 DEBUG:boto:Token: None 2023-07-29T00:52:40.573 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/obj11?versionId=null 2023-07-29T00:52:40.573 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:10o6JmeeznIoBkwwfMCt0R3MiTQ= 2023-07-29T00:52:40.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:10o6JmeeznIoBkwwfMCt0R3MiTQ=', 'Content-Length': '0'} 2023-07-29T00:52:40.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025a332ad641bd6e8-0064c462d8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.577 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:52:40.577 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:52:40.578 DEBUG:boto:path=/hiiqos-62/?prefix=obj11 2023-07-29T00:52:40.578 DEBUG:boto:auth_path=/hiiqos-62/?prefix=obj11 2023-07-29T00:52:40.578 DEBUG:boto:Method: GET 2023-07-29T00:52:40.579 DEBUG:boto:Path: /hiiqos-62/?prefix=obj11 2023-07-29T00:52:40.579 DEBUG:boto:Data: 2023-07-29T00:52:40.579 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.580 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:40.580 DEBUG:boto:Port: 8001 2023-07-29T00:52:40.580 DEBUG:boto:Params: {} 2023-07-29T00:52:40.581 DEBUG:boto:Token: None 2023-07-29T00:52:40.581 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/ 2023-07-29T00:52:40.582 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FR5UEZyoU0rbmbeXP6UnHKKuDgc= 2023-07-29T00:52:40.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FR5UEZyoU0rbmbeXP6UnHKKuDgc=', 'Content-Length': '0'} 2023-07-29T00:52:40.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085905ef57c556802-0064c462d8-2068-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.585 DEBUG:boto:b'hiiqos-62obj111000falseobj112023-07-29T00:50:33.566Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:52:40.586 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:52:40.586 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:52:40.587 DEBUG:boto:path=/hiiqos-62/?prefix=obj11 2023-07-29T00:52:40.587 DEBUG:boto:auth_path=/hiiqos-62/?prefix=obj11 2023-07-29T00:52:40.587 DEBUG:boto:Method: GET 2023-07-29T00:52:40.588 DEBUG:boto:Path: /hiiqos-62/?prefix=obj11 2023-07-29T00:52:40.588 DEBUG:boto:Data: 2023-07-29T00:52:40.588 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.589 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:40.589 DEBUG:boto:Port: 8000 2023-07-29T00:52:40.590 DEBUG:boto:Params: {} 2023-07-29T00:52:40.590 DEBUG:boto:Token: None 2023-07-29T00:52:40.590 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/ 2023-07-29T00:52:40.591 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FR5UEZyoU0rbmbeXP6UnHKKuDgc= 2023-07-29T00:52:40.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FR5UEZyoU0rbmbeXP6UnHKKuDgc=', 'Content-Length': '0'} 2023-07-29T00:52:40.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000854d4897147e94ec-0064c462d8-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.594 DEBUG:boto:b'hiiqos-62obj111000falseobj112023-07-29T00:50:33.566Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:52:40.595 DEBUG:boto:path=/hiiqos-62/obj11 2023-07-29T00:52:40.595 DEBUG:boto:auth_path=/hiiqos-62/obj11 2023-07-29T00:52:40.596 DEBUG:boto:Method: HEAD 2023-07-29T00:52:40.596 DEBUG:boto:Path: /hiiqos-62/obj11 2023-07-29T00:52:40.596 DEBUG:boto:Data: 2023-07-29T00:52:40.597 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.597 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:40.597 DEBUG:boto:Port: 8001 2023-07-29T00:52:40.598 DEBUG:boto:Params: {} 2023-07-29T00:52:40.598 DEBUG:boto:Token: None 2023-07-29T00:52:40.598 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/obj11 2023-07-29T00:52:40.599 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:iDp4gq36rYH6RsqoxOoTboGKE14= 2023-07-29T00:52:40.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:iDp4gq36rYH6RsqoxOoTboGKE14=', 'Content-Length': '0'} 2023-07-29T00:52:40.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-62:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000240818c70f05a6c7-0064c462d8-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.602 DEBUG:boto:path=/hiiqos-62/obj11 2023-07-29T00:52:40.602 DEBUG:boto:auth_path=/hiiqos-62/obj11 2023-07-29T00:52:40.602 DEBUG:boto:Method: HEAD 2023-07-29T00:52:40.603 DEBUG:boto:Path: /hiiqos-62/obj11 2023-07-29T00:52:40.603 DEBUG:boto:Data: 2023-07-29T00:52:40.603 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.604 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:40.604 DEBUG:boto:Port: 8000 2023-07-29T00:52:40.604 DEBUG:boto:Params: {} 2023-07-29T00:52:40.605 DEBUG:boto:Token: None 2023-07-29T00:52:40.605 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/obj11 2023-07-29T00:52:40.606 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:iDp4gq36rYH6RsqoxOoTboGKE14= 2023-07-29T00:52:40.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:iDp4gq36rYH6RsqoxOoTboGKE14=', 'Content-Length': '0'} 2023-07-29T00:52:40.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086cbbb72b7d21c67-0064c462d8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-07-29T00:52:40.609 DEBUG:boto:path=/hiiqos-62/obj11 2023-07-29T00:52:40.609 DEBUG:boto:auth_path=/hiiqos-62/obj11 2023-07-29T00:52:40.610 DEBUG:boto:Method: GET 2023-07-29T00:52:40.610 DEBUG:boto:Path: /hiiqos-62/obj11 2023-07-29T00:52:40.610 DEBUG:boto:Data: 2023-07-29T00:52:40.611 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.611 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:40.611 DEBUG:boto:Port: 8001 2023-07-29T00:52:40.612 DEBUG:boto:Params: {} 2023-07-29T00:52:40.612 DEBUG:boto:Token: None 2023-07-29T00:52:40.613 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/obj11 2023-07-29T00:52:40.613 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4HS30QcuWPNh1tCSgsZ7Nmf6VQY= 2023-07-29T00:52:40.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4HS30QcuWPNh1tCSgsZ7Nmf6VQY=', 'Content-Length': '0'} 2023-07-29T00:52:40.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-62:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f77f949c53f0dabe-0064c462d8-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.656 DEBUG:boto:path=/hiiqos-62/obj11 2023-07-29T00:52:40.657 DEBUG:boto:auth_path=/hiiqos-62/obj11 2023-07-29T00:52:40.657 DEBUG:boto:Method: GET 2023-07-29T00:52:40.658 DEBUG:boto:Path: /hiiqos-62/obj11 2023-07-29T00:52:40.658 DEBUG:boto:Data: 2023-07-29T00:52:40.658 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.659 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:40.659 DEBUG:boto:Port: 8000 2023-07-29T00:52:40.660 DEBUG:boto:Params: {} 2023-07-29T00:52:40.660 DEBUG:boto:Token: None 2023-07-29T00:52:40.660 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/obj11 2023-07-29T00:52:40.661 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4HS30QcuWPNh1tCSgsZ7Nmf6VQY= 2023-07-29T00:52:40.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4HS30QcuWPNh1tCSgsZ7Nmf6VQY=', 'Content-Length': '0'} 2023-07-29T00:52:40.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfcd7c8bd8251a85-0064c462d8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.664 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:52:40.665 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:52:40.665 DEBUG:boto:path=/hiiqos-62/?prefix=obj11 2023-07-29T00:52:40.665 DEBUG:boto:auth_path=/hiiqos-62/?prefix=obj11 2023-07-29T00:52:40.666 DEBUG:boto:Method: GET 2023-07-29T00:52:40.666 DEBUG:boto:Path: /hiiqos-62/?prefix=obj11 2023-07-29T00:52:40.667 DEBUG:boto:Data: 2023-07-29T00:52:40.667 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.667 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:40.668 DEBUG:boto:Port: 8001 2023-07-29T00:52:40.668 DEBUG:boto:Params: {} 2023-07-29T00:52:40.668 DEBUG:boto:Token: None 2023-07-29T00:52:40.669 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/ 2023-07-29T00:52:40.669 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FR5UEZyoU0rbmbeXP6UnHKKuDgc= 2023-07-29T00:52:40.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FR5UEZyoU0rbmbeXP6UnHKKuDgc=', 'Content-Length': '0'} 2023-07-29T00:52:40.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000da24d047e53e06f-0064c462d8-2068-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.673 DEBUG:boto:b'hiiqos-62obj111000falseobj112023-07-29T00:50:33.566Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:52:40.673 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:52:40.674 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:52:40.674 DEBUG:boto:path=/hiiqos-62/?prefix=obj11 2023-07-29T00:52:40.674 DEBUG:boto:auth_path=/hiiqos-62/?prefix=obj11 2023-07-29T00:52:40.675 DEBUG:boto:Method: GET 2023-07-29T00:52:40.675 DEBUG:boto:Path: /hiiqos-62/?prefix=obj11 2023-07-29T00:52:40.676 DEBUG:boto:Data: 2023-07-29T00:52:40.676 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.676 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:40.677 DEBUG:boto:Port: 8000 2023-07-29T00:52:40.677 DEBUG:boto:Params: {} 2023-07-29T00:52:40.677 DEBUG:boto:Token: None 2023-07-29T00:52:40.678 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/ 2023-07-29T00:52:40.678 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FR5UEZyoU0rbmbeXP6UnHKKuDgc= 2023-07-29T00:52:40.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FR5UEZyoU0rbmbeXP6UnHKKuDgc=', 'Content-Length': '0'} 2023-07-29T00:52:40.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000249734f073cd07d9-0064c462d8-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.682 DEBUG:boto:b'hiiqos-62obj111000falseobj112023-07-29T00:50:33.566Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:52:40.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-07-29T00:52:40.683 DEBUG:boto:path=/hiiqos-62/obj22 2023-07-29T00:52:40.683 DEBUG:boto:auth_path=/hiiqos-62/obj22 2023-07-29T00:52:40.683 DEBUG:boto:path=/hiiqos-62/obj22?versionId=null 2023-07-29T00:52:40.684 DEBUG:boto:auth_path=/hiiqos-62/obj22?versionId=null 2023-07-29T00:52:40.684 DEBUG:boto:Method: GET 2023-07-29T00:52:40.685 DEBUG:boto:Path: /hiiqos-62/obj22?versionId=null 2023-07-29T00:52:40.685 DEBUG:boto:Data: 2023-07-29T00:52:40.685 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.686 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:40.686 DEBUG:boto:Port: 8001 2023-07-29T00:52:40.686 DEBUG:boto:Params: {} 2023-07-29T00:52:40.687 DEBUG:boto:Token: None 2023-07-29T00:52:40.687 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/obj22?versionId=null 2023-07-29T00:52:40.688 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Ztmlm8VUAOGh7zh+yiXwB0DLrU0= 2023-07-29T00:52:40.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Ztmlm8VUAOGh7zh+yiXwB0DLrU0=', 'Content-Length': '0'} 2023-07-29T00:52:40.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-62:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4d451910b20214c-0064c462d8-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.691 DEBUG:boto:path=/hiiqos-62/obj22 2023-07-29T00:52:40.691 DEBUG:boto:auth_path=/hiiqos-62/obj22 2023-07-29T00:52:40.692 DEBUG:boto:path=/hiiqos-62/obj22?versionId=null 2023-07-29T00:52:40.692 DEBUG:boto:auth_path=/hiiqos-62/obj22?versionId=null 2023-07-29T00:52:40.693 DEBUG:boto:Method: GET 2023-07-29T00:52:40.693 DEBUG:boto:Path: /hiiqos-62/obj22?versionId=null 2023-07-29T00:52:40.694 DEBUG:boto:Data: 2023-07-29T00:52:40.694 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.695 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:40.695 DEBUG:boto:Port: 8000 2023-07-29T00:52:40.696 DEBUG:boto:Params: {} 2023-07-29T00:52:40.696 DEBUG:boto:Token: None 2023-07-29T00:52:40.697 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/obj22?versionId=null 2023-07-29T00:52:40.697 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Ztmlm8VUAOGh7zh+yiXwB0DLrU0= 2023-07-29T00:52:40.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Ztmlm8VUAOGh7zh+yiXwB0DLrU0=', 'Content-Length': '0'} 2023-07-29T00:52:40.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a9fab7166e50d74-0064c462d8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.700 DEBUG:boto:path=/hiiqos-62/obj22 2023-07-29T00:52:40.701 DEBUG:boto:auth_path=/hiiqos-62/obj22 2023-07-29T00:52:40.702 DEBUG:boto:path=/hiiqos-62/obj22?versionId=null 2023-07-29T00:52:40.702 DEBUG:boto:auth_path=/hiiqos-62/obj22?versionId=null 2023-07-29T00:52:40.703 DEBUG:boto:Method: HEAD 2023-07-29T00:52:40.704 DEBUG:boto:Path: /hiiqos-62/obj22?versionId=null 2023-07-29T00:52:40.704 DEBUG:boto:Data: 2023-07-29T00:52:40.705 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.706 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:40.706 DEBUG:boto:Port: 8001 2023-07-29T00:52:40.707 DEBUG:boto:Params: {} 2023-07-29T00:52:40.708 DEBUG:boto:Token: None 2023-07-29T00:52:40.708 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/obj22?versionId=null 2023-07-29T00:52:40.709 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+fUydE0kXCxZgpCSdWTyMx3Q7FA= 2023-07-29T00:52:40.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+fUydE0kXCxZgpCSdWTyMx3Q7FA=', 'Content-Length': '0'} 2023-07-29T00:52:40.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-62:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088cc1824fb5c418e-0064c462d8-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.713 DEBUG:boto:path=/hiiqos-62/obj22 2023-07-29T00:52:40.714 DEBUG:boto:auth_path=/hiiqos-62/obj22 2023-07-29T00:52:40.715 DEBUG:boto:path=/hiiqos-62/obj22?versionId=null 2023-07-29T00:52:40.715 DEBUG:boto:auth_path=/hiiqos-62/obj22?versionId=null 2023-07-29T00:52:40.716 DEBUG:boto:Method: HEAD 2023-07-29T00:52:40.716 DEBUG:boto:Path: /hiiqos-62/obj22?versionId=null 2023-07-29T00:52:40.717 DEBUG:boto:Data: 2023-07-29T00:52:40.717 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.717 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:40.718 DEBUG:boto:Port: 8000 2023-07-29T00:52:40.718 DEBUG:boto:Params: {} 2023-07-29T00:52:40.718 DEBUG:boto:Token: None 2023-07-29T00:52:40.719 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/obj22?versionId=null 2023-07-29T00:52:40.719 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+fUydE0kXCxZgpCSdWTyMx3Q7FA= 2023-07-29T00:52:40.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+fUydE0kXCxZgpCSdWTyMx3Q7FA=', 'Content-Length': '0'} 2023-07-29T00:52:40.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab1ab5ae20b34afe-0064c462d8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-07-29T00:52:40.723 DEBUG:boto:path=/hiiqos-62/obj22 2023-07-29T00:52:40.723 DEBUG:boto:auth_path=/hiiqos-62/obj22 2023-07-29T00:52:40.724 DEBUG:boto:path=/hiiqos-62/obj22?versionId=null 2023-07-29T00:52:40.724 DEBUG:boto:auth_path=/hiiqos-62/obj22?versionId=null 2023-07-29T00:52:40.724 DEBUG:boto:Method: GET 2023-07-29T00:52:40.725 DEBUG:boto:Path: /hiiqos-62/obj22?versionId=null 2023-07-29T00:52:40.725 DEBUG:boto:Data: 2023-07-29T00:52:40.726 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.726 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:40.727 DEBUG:boto:Port: 8001 2023-07-29T00:52:40.727 DEBUG:boto:Params: {} 2023-07-29T00:52:40.727 DEBUG:boto:Token: None 2023-07-29T00:52:40.728 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/obj22?versionId=null 2023-07-29T00:52:40.729 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Ztmlm8VUAOGh7zh+yiXwB0DLrU0= 2023-07-29T00:52:40.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Ztmlm8VUAOGh7zh+yiXwB0DLrU0=', 'Content-Length': '0'} 2023-07-29T00:52:40.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-62:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063e36d9fc0d7fd4e-0064c462d8-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.731 DEBUG:boto:path=/hiiqos-62/obj22 2023-07-29T00:52:40.732 DEBUG:boto:auth_path=/hiiqos-62/obj22 2023-07-29T00:52:40.732 DEBUG:boto:path=/hiiqos-62/obj22?versionId=null 2023-07-29T00:52:40.733 DEBUG:boto:auth_path=/hiiqos-62/obj22?versionId=null 2023-07-29T00:52:40.733 DEBUG:boto:Method: GET 2023-07-29T00:52:40.734 DEBUG:boto:Path: /hiiqos-62/obj22?versionId=null 2023-07-29T00:52:40.734 DEBUG:boto:Data: 2023-07-29T00:52:40.734 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.735 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:40.735 DEBUG:boto:Port: 8000 2023-07-29T00:52:40.736 DEBUG:boto:Params: {} 2023-07-29T00:52:40.736 DEBUG:boto:Token: None 2023-07-29T00:52:40.737 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/obj22?versionId=null 2023-07-29T00:52:40.737 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Ztmlm8VUAOGh7zh+yiXwB0DLrU0= 2023-07-29T00:52:40.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Ztmlm8VUAOGh7zh+yiXwB0DLrU0=', 'Content-Length': '0'} 2023-07-29T00:52:40.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002343f9317a2b0447-0064c462d8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.740 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:52:40.741 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:52:40.741 DEBUG:boto:path=/hiiqos-62/?prefix=obj22 2023-07-29T00:52:40.741 DEBUG:boto:auth_path=/hiiqos-62/?prefix=obj22 2023-07-29T00:52:40.742 DEBUG:boto:Method: GET 2023-07-29T00:52:40.742 DEBUG:boto:Path: /hiiqos-62/?prefix=obj22 2023-07-29T00:52:40.743 DEBUG:boto:Data: 2023-07-29T00:52:40.743 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.743 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:40.744 DEBUG:boto:Port: 8001 2023-07-29T00:52:40.744 DEBUG:boto:Params: {} 2023-07-29T00:52:40.745 DEBUG:boto:Token: None 2023-07-29T00:52:40.745 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/ 2023-07-29T00:52:40.746 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FR5UEZyoU0rbmbeXP6UnHKKuDgc= 2023-07-29T00:52:40.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FR5UEZyoU0rbmbeXP6UnHKKuDgc=', 'Content-Length': '0'} 2023-07-29T00:52:40.748 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be8aea512a1240b2-0064c462d8-2068-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.749 DEBUG:boto:b'hiiqos-62obj221000falseobj222023-07-29T00:50:33.623Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:52:40.750 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:52:40.750 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:52:40.751 DEBUG:boto:path=/hiiqos-62/?prefix=obj22 2023-07-29T00:52:40.751 DEBUG:boto:auth_path=/hiiqos-62/?prefix=obj22 2023-07-29T00:52:40.751 DEBUG:boto:Method: GET 2023-07-29T00:52:40.752 DEBUG:boto:Path: /hiiqos-62/?prefix=obj22 2023-07-29T00:52:40.752 DEBUG:boto:Data: 2023-07-29T00:52:40.753 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.753 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:40.753 DEBUG:boto:Port: 8000 2023-07-29T00:52:40.754 DEBUG:boto:Params: {} 2023-07-29T00:52:40.754 DEBUG:boto:Token: None 2023-07-29T00:52:40.755 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/ 2023-07-29T00:52:40.756 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FR5UEZyoU0rbmbeXP6UnHKKuDgc= 2023-07-29T00:52:40.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FR5UEZyoU0rbmbeXP6UnHKKuDgc=', 'Content-Length': '0'} 2023-07-29T00:52:40.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d03f7daeafd22e7-0064c462d8-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.759 DEBUG:boto:b'hiiqos-62obj221000falseobj222023-07-29T00:50:33.623Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:52:40.760 DEBUG:boto:path=/hiiqos-62/obj22 2023-07-29T00:52:40.760 DEBUG:boto:auth_path=/hiiqos-62/obj22 2023-07-29T00:52:40.761 DEBUG:boto:Method: HEAD 2023-07-29T00:52:40.761 DEBUG:boto:Path: /hiiqos-62/obj22 2023-07-29T00:52:40.762 DEBUG:boto:Data: 2023-07-29T00:52:40.762 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.762 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:40.763 DEBUG:boto:Port: 8001 2023-07-29T00:52:40.763 DEBUG:boto:Params: {} 2023-07-29T00:52:40.763 DEBUG:boto:Token: None 2023-07-29T00:52:40.764 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/obj22 2023-07-29T00:52:40.764 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pwGBssmtqD0hd75KgTE5qjYnW6A= 2023-07-29T00:52:40.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pwGBssmtqD0hd75KgTE5qjYnW6A=', 'Content-Length': '0'} 2023-07-29T00:52:40.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-62:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005116efc9ad56c684-0064c462d8-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.767 DEBUG:boto:path=/hiiqos-62/obj22 2023-07-29T00:52:40.768 DEBUG:boto:auth_path=/hiiqos-62/obj22 2023-07-29T00:52:40.768 DEBUG:boto:Method: HEAD 2023-07-29T00:52:40.769 DEBUG:boto:Path: /hiiqos-62/obj22 2023-07-29T00:52:40.769 DEBUG:boto:Data: 2023-07-29T00:52:40.769 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.770 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:40.771 DEBUG:boto:Port: 8000 2023-07-29T00:52:40.771 DEBUG:boto:Params: {} 2023-07-29T00:52:40.771 DEBUG:boto:Token: None 2023-07-29T00:52:40.772 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/obj22 2023-07-29T00:52:40.772 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pwGBssmtqD0hd75KgTE5qjYnW6A= 2023-07-29T00:52:40.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pwGBssmtqD0hd75KgTE5qjYnW6A=', 'Content-Length': '0'} 2023-07-29T00:52:40.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d49bae8f1f570ce7-0064c462d8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-07-29T00:52:40.776 DEBUG:boto:path=/hiiqos-62/obj22 2023-07-29T00:52:40.776 DEBUG:boto:auth_path=/hiiqos-62/obj22 2023-07-29T00:52:40.777 DEBUG:boto:Method: GET 2023-07-29T00:52:40.777 DEBUG:boto:Path: /hiiqos-62/obj22 2023-07-29T00:52:40.778 DEBUG:boto:Data: 2023-07-29T00:52:40.778 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.778 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:40.779 DEBUG:boto:Port: 8001 2023-07-29T00:52:40.780 DEBUG:boto:Params: {} 2023-07-29T00:52:40.780 DEBUG:boto:Token: None 2023-07-29T00:52:40.781 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/obj22 2023-07-29T00:52:40.781 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JWH/WppGEebvUpdjKEMZkNCWkt0= 2023-07-29T00:52:40.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JWH/WppGEebvUpdjKEMZkNCWkt0=', 'Content-Length': '0'} 2023-07-29T00:52:40.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-62:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003483f8853652e041-0064c462d8-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.829 DEBUG:boto:path=/hiiqos-62/obj22 2023-07-29T00:52:40.830 DEBUG:boto:auth_path=/hiiqos-62/obj22 2023-07-29T00:52:40.830 DEBUG:boto:Method: GET 2023-07-29T00:52:40.831 DEBUG:boto:Path: /hiiqos-62/obj22 2023-07-29T00:52:40.831 DEBUG:boto:Data: 2023-07-29T00:52:40.831 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.832 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:40.832 DEBUG:boto:Port: 8000 2023-07-29T00:52:40.833 DEBUG:boto:Params: {} 2023-07-29T00:52:40.833 DEBUG:boto:Token: None 2023-07-29T00:52:40.833 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/obj22 2023-07-29T00:52:40.834 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:JWH/WppGEebvUpdjKEMZkNCWkt0= 2023-07-29T00:52:40.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:JWH/WppGEebvUpdjKEMZkNCWkt0=', 'Content-Length': '0'} 2023-07-29T00:52:40.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d075d98690641ab1-0064c462d8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.837 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:52:40.837 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:52:40.838 DEBUG:boto:path=/hiiqos-62/?prefix=obj22 2023-07-29T00:52:40.838 DEBUG:boto:auth_path=/hiiqos-62/?prefix=obj22 2023-07-29T00:52:40.839 DEBUG:boto:Method: GET 2023-07-29T00:52:40.839 DEBUG:boto:Path: /hiiqos-62/?prefix=obj22 2023-07-29T00:52:40.839 DEBUG:boto:Data: 2023-07-29T00:52:40.840 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.840 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:40.840 DEBUG:boto:Port: 8001 2023-07-29T00:52:40.841 DEBUG:boto:Params: {} 2023-07-29T00:52:40.841 DEBUG:boto:Token: None 2023-07-29T00:52:40.842 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/ 2023-07-29T00:52:40.842 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FR5UEZyoU0rbmbeXP6UnHKKuDgc= 2023-07-29T00:52:40.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FR5UEZyoU0rbmbeXP6UnHKKuDgc=', 'Content-Length': '0'} 2023-07-29T00:52:40.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff1b210a9fc405fd-0064c462d8-2068-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.849 DEBUG:boto:b'hiiqos-62obj221000falseobj222023-07-29T00:50:33.623Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:52:40.850 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:52:40.851 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:52:40.851 DEBUG:boto:path=/hiiqos-62/?prefix=obj22 2023-07-29T00:52:40.852 DEBUG:boto:auth_path=/hiiqos-62/?prefix=obj22 2023-07-29T00:52:40.853 DEBUG:boto:Method: GET 2023-07-29T00:52:40.853 DEBUG:boto:Path: /hiiqos-62/?prefix=obj22 2023-07-29T00:52:40.854 DEBUG:boto:Data: 2023-07-29T00:52:40.855 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.856 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:40.856 DEBUG:boto:Port: 8000 2023-07-29T00:52:40.857 DEBUG:boto:Params: {} 2023-07-29T00:52:40.857 DEBUG:boto:Token: None 2023-07-29T00:52:40.858 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/ 2023-07-29T00:52:40.858 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FR5UEZyoU0rbmbeXP6UnHKKuDgc= 2023-07-29T00:52:40.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FR5UEZyoU0rbmbeXP6UnHKKuDgc=', 'Content-Length': '0'} 2023-07-29T00:52:40.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000112f9c1e54a70f98-0064c462d8-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.862 DEBUG:boto:b'hiiqos-62obj221000falseobj222023-07-29T00:50:33.623Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:52:40.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-07-29T00:52:40.864 DEBUG:boto:path=/hiiqos-62/obj33 2023-07-29T00:52:40.865 DEBUG:boto:auth_path=/hiiqos-62/obj33 2023-07-29T00:52:40.865 DEBUG:boto:path=/hiiqos-62/obj33?versionId=null 2023-07-29T00:52:40.866 DEBUG:boto:auth_path=/hiiqos-62/obj33?versionId=null 2023-07-29T00:52:40.866 DEBUG:boto:Method: GET 2023-07-29T00:52:40.867 DEBUG:boto:Path: /hiiqos-62/obj33?versionId=null 2023-07-29T00:52:40.867 DEBUG:boto:Data: 2023-07-29T00:52:40.867 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.868 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:40.868 DEBUG:boto:Port: 8001 2023-07-29T00:52:40.869 DEBUG:boto:Params: {} 2023-07-29T00:52:40.869 DEBUG:boto:Token: None 2023-07-29T00:52:40.869 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/obj33?versionId=null 2023-07-29T00:52:40.870 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O0q1nn32TtM93OUdDgnb1kGMpCY= 2023-07-29T00:52:40.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O0q1nn32TtM93OUdDgnb1kGMpCY=', 'Content-Length': '0'} 2023-07-29T00:52:40.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-62:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e161c397ad5cc99c-0064c462d8-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.873 DEBUG:boto:path=/hiiqos-62/obj33 2023-07-29T00:52:40.874 DEBUG:boto:auth_path=/hiiqos-62/obj33 2023-07-29T00:52:40.874 DEBUG:boto:path=/hiiqos-62/obj33?versionId=null 2023-07-29T00:52:40.874 DEBUG:boto:auth_path=/hiiqos-62/obj33?versionId=null 2023-07-29T00:52:40.875 DEBUG:boto:Method: GET 2023-07-29T00:52:40.875 DEBUG:boto:Path: /hiiqos-62/obj33?versionId=null 2023-07-29T00:52:40.875 DEBUG:boto:Data: 2023-07-29T00:52:40.876 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.876 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:40.877 DEBUG:boto:Port: 8000 2023-07-29T00:52:40.877 DEBUG:boto:Params: {} 2023-07-29T00:52:40.877 DEBUG:boto:Token: None 2023-07-29T00:52:40.878 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/obj33?versionId=null 2023-07-29T00:52:40.878 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O0q1nn32TtM93OUdDgnb1kGMpCY= 2023-07-29T00:52:40.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O0q1nn32TtM93OUdDgnb1kGMpCY=', 'Content-Length': '0'} 2023-07-29T00:52:40.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adebdc3ef7e4a6cc-0064c462d8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.924 DEBUG:boto:path=/hiiqos-62/obj33 2023-07-29T00:52:40.925 DEBUG:boto:auth_path=/hiiqos-62/obj33 2023-07-29T00:52:40.925 DEBUG:boto:path=/hiiqos-62/obj33?versionId=null 2023-07-29T00:52:40.926 DEBUG:boto:auth_path=/hiiqos-62/obj33?versionId=null 2023-07-29T00:52:40.926 DEBUG:boto:Method: HEAD 2023-07-29T00:52:40.926 DEBUG:boto:Path: /hiiqos-62/obj33?versionId=null 2023-07-29T00:52:40.927 DEBUG:boto:Data: 2023-07-29T00:52:40.927 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.927 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:40.928 DEBUG:boto:Port: 8001 2023-07-29T00:52:40.928 DEBUG:boto:Params: {} 2023-07-29T00:52:40.928 DEBUG:boto:Token: None 2023-07-29T00:52:40.929 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/obj33?versionId=null 2023-07-29T00:52:40.929 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jAWsYj5vFd6A8VJfBYkChfBQ4WA= 2023-07-29T00:52:40.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jAWsYj5vFd6A8VJfBYkChfBQ4WA=', 'Content-Length': '0'} 2023-07-29T00:52:40.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-62:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ff3f978f5306095-0064c462d8-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.932 DEBUG:boto:path=/hiiqos-62/obj33 2023-07-29T00:52:40.933 DEBUG:boto:auth_path=/hiiqos-62/obj33 2023-07-29T00:52:40.933 DEBUG:boto:path=/hiiqos-62/obj33?versionId=null 2023-07-29T00:52:40.933 DEBUG:boto:auth_path=/hiiqos-62/obj33?versionId=null 2023-07-29T00:52:40.934 DEBUG:boto:Method: HEAD 2023-07-29T00:52:40.934 DEBUG:boto:Path: /hiiqos-62/obj33?versionId=null 2023-07-29T00:52:40.934 DEBUG:boto:Data: 2023-07-29T00:52:40.935 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.935 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:40.935 DEBUG:boto:Port: 8000 2023-07-29T00:52:40.936 DEBUG:boto:Params: {} 2023-07-29T00:52:40.936 DEBUG:boto:Token: None 2023-07-29T00:52:40.937 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/obj33?versionId=null 2023-07-29T00:52:40.937 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jAWsYj5vFd6A8VJfBYkChfBQ4WA= 2023-07-29T00:52:40.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jAWsYj5vFd6A8VJfBYkChfBQ4WA=', 'Content-Length': '0'} 2023-07-29T00:52:40.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000952c90d76d028959-0064c462d8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-07-29T00:52:40.940 DEBUG:boto:path=/hiiqos-62/obj33 2023-07-29T00:52:40.941 DEBUG:boto:auth_path=/hiiqos-62/obj33 2023-07-29T00:52:40.941 DEBUG:boto:path=/hiiqos-62/obj33?versionId=null 2023-07-29T00:52:40.941 DEBUG:boto:auth_path=/hiiqos-62/obj33?versionId=null 2023-07-29T00:52:40.942 DEBUG:boto:Method: GET 2023-07-29T00:52:40.942 DEBUG:boto:Path: /hiiqos-62/obj33?versionId=null 2023-07-29T00:52:40.942 DEBUG:boto:Data: 2023-07-29T00:52:40.943 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.943 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:40.943 DEBUG:boto:Port: 8001 2023-07-29T00:52:40.944 DEBUG:boto:Params: {} 2023-07-29T00:52:40.944 DEBUG:boto:Token: None 2023-07-29T00:52:40.945 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/obj33?versionId=null 2023-07-29T00:52:40.945 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O0q1nn32TtM93OUdDgnb1kGMpCY= 2023-07-29T00:52:40.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O0q1nn32TtM93OUdDgnb1kGMpCY=', 'Content-Length': '0'} 2023-07-29T00:52:40.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-62:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b291129bda98cc7-0064c462d8-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.948 DEBUG:boto:path=/hiiqos-62/obj33 2023-07-29T00:52:40.949 DEBUG:boto:auth_path=/hiiqos-62/obj33 2023-07-29T00:52:40.949 DEBUG:boto:path=/hiiqos-62/obj33?versionId=null 2023-07-29T00:52:40.949 DEBUG:boto:auth_path=/hiiqos-62/obj33?versionId=null 2023-07-29T00:52:40.950 DEBUG:boto:Method: GET 2023-07-29T00:52:40.950 DEBUG:boto:Path: /hiiqos-62/obj33?versionId=null 2023-07-29T00:52:40.950 DEBUG:boto:Data: 2023-07-29T00:52:40.951 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.951 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:40.951 DEBUG:boto:Port: 8000 2023-07-29T00:52:40.952 DEBUG:boto:Params: {} 2023-07-29T00:52:40.952 DEBUG:boto:Token: None 2023-07-29T00:52:40.953 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/obj33?versionId=null 2023-07-29T00:52:40.953 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O0q1nn32TtM93OUdDgnb1kGMpCY= 2023-07-29T00:52:40.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O0q1nn32TtM93OUdDgnb1kGMpCY=', 'Content-Length': '0'} 2023-07-29T00:52:40.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d355f926157501c-0064c462d8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.956 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:52:40.956 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:52:40.957 DEBUG:boto:path=/hiiqos-62/?prefix=obj33 2023-07-29T00:52:40.957 DEBUG:boto:auth_path=/hiiqos-62/?prefix=obj33 2023-07-29T00:52:40.958 DEBUG:boto:Method: GET 2023-07-29T00:52:40.958 DEBUG:boto:Path: /hiiqos-62/?prefix=obj33 2023-07-29T00:52:40.958 DEBUG:boto:Data: 2023-07-29T00:52:40.959 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.959 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:40.959 DEBUG:boto:Port: 8001 2023-07-29T00:52:40.960 DEBUG:boto:Params: {} 2023-07-29T00:52:40.960 DEBUG:boto:Token: None 2023-07-29T00:52:40.961 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/ 2023-07-29T00:52:40.961 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FR5UEZyoU0rbmbeXP6UnHKKuDgc= 2023-07-29T00:52:40.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FR5UEZyoU0rbmbeXP6UnHKKuDgc=', 'Content-Length': '0'} 2023-07-29T00:52:40.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000177fa30e9b55e771-0064c462d8-2068-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.964 DEBUG:boto:b'hiiqos-62obj331000falseobj332023-07-29T00:50:33.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:52:40.965 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:52:40.965 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:52:40.966 DEBUG:boto:path=/hiiqos-62/?prefix=obj33 2023-07-29T00:52:40.966 DEBUG:boto:auth_path=/hiiqos-62/?prefix=obj33 2023-07-29T00:52:40.966 DEBUG:boto:Method: GET 2023-07-29T00:52:40.967 DEBUG:boto:Path: /hiiqos-62/?prefix=obj33 2023-07-29T00:52:40.967 DEBUG:boto:Data: 2023-07-29T00:52:40.967 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.968 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:40.968 DEBUG:boto:Port: 8000 2023-07-29T00:52:40.969 DEBUG:boto:Params: {} 2023-07-29T00:52:40.969 DEBUG:boto:Token: None 2023-07-29T00:52:40.969 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/ 2023-07-29T00:52:40.970 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FR5UEZyoU0rbmbeXP6UnHKKuDgc= 2023-07-29T00:52:40.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FR5UEZyoU0rbmbeXP6UnHKKuDgc=', 'Content-Length': '0'} 2023-07-29T00:52:40.973 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058b1986b84567325-0064c462d8-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.973 DEBUG:boto:b'hiiqos-62obj331000falseobj332023-07-29T00:50:33.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:52:40.974 DEBUG:boto:path=/hiiqos-62/obj33 2023-07-29T00:52:40.974 DEBUG:boto:auth_path=/hiiqos-62/obj33 2023-07-29T00:52:40.975 DEBUG:boto:Method: HEAD 2023-07-29T00:52:40.975 DEBUG:boto:Path: /hiiqos-62/obj33 2023-07-29T00:52:40.975 DEBUG:boto:Data: 2023-07-29T00:52:40.976 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.976 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:40.976 DEBUG:boto:Port: 8001 2023-07-29T00:52:40.977 DEBUG:boto:Params: {} 2023-07-29T00:52:40.977 DEBUG:boto:Token: None 2023-07-29T00:52:40.978 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/obj33 2023-07-29T00:52:40.978 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FvcS3APNSFTM6hLSYVS+Bsm1sLk= 2023-07-29T00:52:40.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FvcS3APNSFTM6hLSYVS+Bsm1sLk=', 'Content-Length': '0'} 2023-07-29T00:52:40.980 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-62:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f753c8a8ddea4bae-0064c462d8-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.981 DEBUG:boto:path=/hiiqos-62/obj33 2023-07-29T00:52:40.981 DEBUG:boto:auth_path=/hiiqos-62/obj33 2023-07-29T00:52:40.982 DEBUG:boto:Method: HEAD 2023-07-29T00:52:40.982 DEBUG:boto:Path: /hiiqos-62/obj33 2023-07-29T00:52:40.982 DEBUG:boto:Data: 2023-07-29T00:52:40.983 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.983 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:40.983 DEBUG:boto:Port: 8000 2023-07-29T00:52:40.984 DEBUG:boto:Params: {} 2023-07-29T00:52:40.984 DEBUG:boto:Token: None 2023-07-29T00:52:40.984 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/obj33 2023-07-29T00:52:40.985 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FvcS3APNSFTM6hLSYVS+Bsm1sLk= 2023-07-29T00:52:40.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FvcS3APNSFTM6hLSYVS+Bsm1sLk=', 'Content-Length': '0'} 2023-07-29T00:52:40.987 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000738b78e1c52e5617-0064c462d8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:40.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-07-29T00:52:40.988 DEBUG:boto:path=/hiiqos-62/obj33 2023-07-29T00:52:40.989 DEBUG:boto:auth_path=/hiiqos-62/obj33 2023-07-29T00:52:40.989 DEBUG:boto:Method: GET 2023-07-29T00:52:40.989 DEBUG:boto:Path: /hiiqos-62/obj33 2023-07-29T00:52:40.990 DEBUG:boto:Data: 2023-07-29T00:52:40.990 DEBUG:boto:Headers: {} 2023-07-29T00:52:40.990 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:40.991 DEBUG:boto:Port: 8001 2023-07-29T00:52:40.991 DEBUG:boto:Params: {} 2023-07-29T00:52:40.991 DEBUG:boto:Token: None 2023-07-29T00:52:40.992 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:40 GMT /hiiqos-62/obj33 2023-07-29T00:52:40.992 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xiXA2btY47c8adD+qACa9s6CPfg= 2023-07-29T00:52:40.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:40 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xiXA2btY47c8adD+qACa9s6CPfg=', 'Content-Length': '0'} 2023-07-29T00:52:40.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-62:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a6f4aef3784a4e8-0064c462d8-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:41.037 DEBUG:boto:path=/hiiqos-62/obj33 2023-07-29T00:52:41.037 DEBUG:boto:auth_path=/hiiqos-62/obj33 2023-07-29T00:52:41.038 DEBUG:boto:Method: GET 2023-07-29T00:52:41.038 DEBUG:boto:Path: /hiiqos-62/obj33 2023-07-29T00:52:41.038 DEBUG:boto:Data: 2023-07-29T00:52:41.039 DEBUG:boto:Headers: {} 2023-07-29T00:52:41.039 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:41.039 DEBUG:boto:Port: 8000 2023-07-29T00:52:41.040 DEBUG:boto:Params: {} 2023-07-29T00:52:41.040 DEBUG:boto:Token: None 2023-07-29T00:52:41.041 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:41 GMT /hiiqos-62/obj33 2023-07-29T00:52:41.041 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vljNFaiCuypNEEX9BzbKaQ9vra0= 2023-07-29T00:52:41.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vljNFaiCuypNEEX9BzbKaQ9vra0=', 'Content-Length': '0'} 2023-07-29T00:52:41.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b93f56eaf5f2a36-0064c462d8-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:41.044 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:52:41.044 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:52:41.045 DEBUG:boto:path=/hiiqos-62/?prefix=obj33 2023-07-29T00:52:41.045 DEBUG:boto:auth_path=/hiiqos-62/?prefix=obj33 2023-07-29T00:52:41.046 DEBUG:boto:Method: GET 2023-07-29T00:52:41.046 DEBUG:boto:Path: /hiiqos-62/?prefix=obj33 2023-07-29T00:52:41.046 DEBUG:boto:Data: 2023-07-29T00:52:41.047 DEBUG:boto:Headers: {} 2023-07-29T00:52:41.047 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:41.047 DEBUG:boto:Port: 8001 2023-07-29T00:52:41.048 DEBUG:boto:Params: {} 2023-07-29T00:52:41.048 DEBUG:boto:Token: None 2023-07-29T00:52:41.049 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:41 GMT /hiiqos-62/ 2023-07-29T00:52:41.049 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+F31U6MSxIhqn1PM+3SZALJsOk4= 2023-07-29T00:52:41.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+F31U6MSxIhqn1PM+3SZALJsOk4=', 'Content-Length': '0'} 2023-07-29T00:52:41.052 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1043ba355c1b11e-0064c462d8-2068-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:41.052 DEBUG:boto:b'hiiqos-62obj331000falseobj332023-07-29T00:50:33.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:52:41.053 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:52:41.053 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:52:41.054 DEBUG:boto:path=/hiiqos-62/?prefix=obj33 2023-07-29T00:52:41.054 DEBUG:boto:auth_path=/hiiqos-62/?prefix=obj33 2023-07-29T00:52:41.054 DEBUG:boto:Method: GET 2023-07-29T00:52:41.055 DEBUG:boto:Path: /hiiqos-62/?prefix=obj33 2023-07-29T00:52:41.055 DEBUG:boto:Data: 2023-07-29T00:52:41.055 DEBUG:boto:Headers: {} 2023-07-29T00:52:41.056 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:41.056 DEBUG:boto:Port: 8000 2023-07-29T00:52:41.057 DEBUG:boto:Params: {} 2023-07-29T00:52:41.057 DEBUG:boto:Token: None 2023-07-29T00:52:41.057 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:41 GMT /hiiqos-62/ 2023-07-29T00:52:41.058 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+F31U6MSxIhqn1PM+3SZALJsOk4= 2023-07-29T00:52:41.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+F31U6MSxIhqn1PM+3SZALJsOk4=', 'Content-Length': '0'} 2023-07-29T00:52:41.061 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004436e794d0613afd-0064c462d8-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:41.061 DEBUG:boto:b'hiiqos-62obj331000falseobj332023-07-29T00:50:33.641Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:52:41.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-07-29T00:52:41.062 DEBUG:boto:path=/hiiqos-62/obj44 2023-07-29T00:52:41.063 DEBUG:boto:auth_path=/hiiqos-62/obj44 2023-07-29T00:52:41.063 DEBUG:boto:path=/hiiqos-62/obj44?versionId=null 2023-07-29T00:52:41.063 DEBUG:boto:auth_path=/hiiqos-62/obj44?versionId=null 2023-07-29T00:52:41.064 DEBUG:boto:Method: GET 2023-07-29T00:52:41.064 DEBUG:boto:Path: /hiiqos-62/obj44?versionId=null 2023-07-29T00:52:41.064 DEBUG:boto:Data: 2023-07-29T00:52:41.065 DEBUG:boto:Headers: {} 2023-07-29T00:52:41.065 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:41.066 DEBUG:boto:Port: 8001 2023-07-29T00:52:41.066 DEBUG:boto:Params: {} 2023-07-29T00:52:41.066 DEBUG:boto:Token: None 2023-07-29T00:52:41.067 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:41 GMT /hiiqos-62/obj44?versionId=null 2023-07-29T00:52:41.067 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:s+LfeTj5zvBJLGqCaexdQ5maWh8= 2023-07-29T00:52:41.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:s+LfeTj5zvBJLGqCaexdQ5maWh8=', 'Content-Length': '0'} 2023-07-29T00:52:41.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-62:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ddf2bfc76059ed6-0064c462d8-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:40 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:41.070 DEBUG:boto:path=/hiiqos-62/obj44 2023-07-29T00:52:41.071 DEBUG:boto:auth_path=/hiiqos-62/obj44 2023-07-29T00:52:41.071 DEBUG:boto:path=/hiiqos-62/obj44?versionId=null 2023-07-29T00:52:41.071 DEBUG:boto:auth_path=/hiiqos-62/obj44?versionId=null 2023-07-29T00:52:41.072 DEBUG:boto:Method: GET 2023-07-29T00:52:41.072 DEBUG:boto:Path: /hiiqos-62/obj44?versionId=null 2023-07-29T00:52:41.073 DEBUG:boto:Data: 2023-07-29T00:52:41.073 DEBUG:boto:Headers: {} 2023-07-29T00:52:41.073 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:41.074 DEBUG:boto:Port: 8000 2023-07-29T00:52:41.074 DEBUG:boto:Params: {} 2023-07-29T00:52:41.074 DEBUG:boto:Token: None 2023-07-29T00:52:41.075 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:41 GMT /hiiqos-62/obj44?versionId=null 2023-07-29T00:52:41.075 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:s+LfeTj5zvBJLGqCaexdQ5maWh8= 2023-07-29T00:52:41.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:s+LfeTj5zvBJLGqCaexdQ5maWh8=', 'Content-Length': '0'} 2023-07-29T00:52:41.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000454ae8a95cb99a0-0064c462d9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:41.120 DEBUG:boto:path=/hiiqos-62/obj44 2023-07-29T00:52:41.121 DEBUG:boto:auth_path=/hiiqos-62/obj44 2023-07-29T00:52:41.121 DEBUG:boto:path=/hiiqos-62/obj44?versionId=null 2023-07-29T00:52:41.121 DEBUG:boto:auth_path=/hiiqos-62/obj44?versionId=null 2023-07-29T00:52:41.122 DEBUG:boto:Method: HEAD 2023-07-29T00:52:41.122 DEBUG:boto:Path: /hiiqos-62/obj44?versionId=null 2023-07-29T00:52:41.122 DEBUG:boto:Data: 2023-07-29T00:52:41.123 DEBUG:boto:Headers: {} 2023-07-29T00:52:41.123 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:41.124 DEBUG:boto:Port: 8001 2023-07-29T00:52:41.124 DEBUG:boto:Params: {} 2023-07-29T00:52:41.124 DEBUG:boto:Token: None 2023-07-29T00:52:41.125 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:41 GMT /hiiqos-62/obj44?versionId=null 2023-07-29T00:52:41.125 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:94TRPMgELTkxz6H8L5gfB0gjsv8= 2023-07-29T00:52:41.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:94TRPMgELTkxz6H8L5gfB0gjsv8=', 'Content-Length': '0'} 2023-07-29T00:52:41.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-62:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003622588039381178-0064c462d9-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:41.128 DEBUG:boto:path=/hiiqos-62/obj44 2023-07-29T00:52:41.128 DEBUG:boto:auth_path=/hiiqos-62/obj44 2023-07-29T00:52:41.129 DEBUG:boto:path=/hiiqos-62/obj44?versionId=null 2023-07-29T00:52:41.129 DEBUG:boto:auth_path=/hiiqos-62/obj44?versionId=null 2023-07-29T00:52:41.129 DEBUG:boto:Method: HEAD 2023-07-29T00:52:41.130 DEBUG:boto:Path: /hiiqos-62/obj44?versionId=null 2023-07-29T00:52:41.130 DEBUG:boto:Data: 2023-07-29T00:52:41.130 DEBUG:boto:Headers: {} 2023-07-29T00:52:41.131 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:41.131 DEBUG:boto:Port: 8000 2023-07-29T00:52:41.131 DEBUG:boto:Params: {} 2023-07-29T00:52:41.132 DEBUG:boto:Token: None 2023-07-29T00:52:41.132 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:41 GMT /hiiqos-62/obj44?versionId=null 2023-07-29T00:52:41.132 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:94TRPMgELTkxz6H8L5gfB0gjsv8= 2023-07-29T00:52:41.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:94TRPMgELTkxz6H8L5gfB0gjsv8=', 'Content-Length': '0'} 2023-07-29T00:52:41.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bda0fb9ab14bdb81-0064c462d9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:41.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-07-29T00:52:41.136 DEBUG:boto:path=/hiiqos-62/obj44 2023-07-29T00:52:41.136 DEBUG:boto:auth_path=/hiiqos-62/obj44 2023-07-29T00:52:41.136 DEBUG:boto:path=/hiiqos-62/obj44?versionId=null 2023-07-29T00:52:41.137 DEBUG:boto:auth_path=/hiiqos-62/obj44?versionId=null 2023-07-29T00:52:41.137 DEBUG:boto:Method: GET 2023-07-29T00:52:41.137 DEBUG:boto:Path: /hiiqos-62/obj44?versionId=null 2023-07-29T00:52:41.138 DEBUG:boto:Data: 2023-07-29T00:52:41.138 DEBUG:boto:Headers: {} 2023-07-29T00:52:41.139 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:41.139 DEBUG:boto:Port: 8001 2023-07-29T00:52:41.139 DEBUG:boto:Params: {} 2023-07-29T00:52:41.140 DEBUG:boto:Token: None 2023-07-29T00:52:41.140 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:41 GMT /hiiqos-62/obj44?versionId=null 2023-07-29T00:52:41.140 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:s+LfeTj5zvBJLGqCaexdQ5maWh8= 2023-07-29T00:52:41.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:s+LfeTj5zvBJLGqCaexdQ5maWh8=', 'Content-Length': '0'} 2023-07-29T00:52:41.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-62:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002aaf263f77343bef-0064c462d9-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:41.143 DEBUG:boto:path=/hiiqos-62/obj44 2023-07-29T00:52:41.144 DEBUG:boto:auth_path=/hiiqos-62/obj44 2023-07-29T00:52:41.144 DEBUG:boto:path=/hiiqos-62/obj44?versionId=null 2023-07-29T00:52:41.145 DEBUG:boto:auth_path=/hiiqos-62/obj44?versionId=null 2023-07-29T00:52:41.145 DEBUG:boto:Method: GET 2023-07-29T00:52:41.145 DEBUG:boto:Path: /hiiqos-62/obj44?versionId=null 2023-07-29T00:52:41.146 DEBUG:boto:Data: 2023-07-29T00:52:41.146 DEBUG:boto:Headers: {} 2023-07-29T00:52:41.146 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:41.147 DEBUG:boto:Port: 8000 2023-07-29T00:52:41.147 DEBUG:boto:Params: {} 2023-07-29T00:52:41.147 DEBUG:boto:Token: None 2023-07-29T00:52:41.148 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:41 GMT /hiiqos-62/obj44?versionId=null 2023-07-29T00:52:41.148 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:s+LfeTj5zvBJLGqCaexdQ5maWh8= 2023-07-29T00:52:41.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:s+LfeTj5zvBJLGqCaexdQ5maWh8=', 'Content-Length': '0'} 2023-07-29T00:52:41.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce0e23e9cc7a837d-0064c462d9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:41.151 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:52:41.152 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:52:41.152 DEBUG:boto:path=/hiiqos-62/?prefix=obj44 2023-07-29T00:52:41.153 DEBUG:boto:auth_path=/hiiqos-62/?prefix=obj44 2023-07-29T00:52:41.153 DEBUG:boto:Method: GET 2023-07-29T00:52:41.153 DEBUG:boto:Path: /hiiqos-62/?prefix=obj44 2023-07-29T00:52:41.154 DEBUG:boto:Data: 2023-07-29T00:52:41.154 DEBUG:boto:Headers: {} 2023-07-29T00:52:41.154 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:41.155 DEBUG:boto:Port: 8001 2023-07-29T00:52:41.155 DEBUG:boto:Params: {} 2023-07-29T00:52:41.156 DEBUG:boto:Token: None 2023-07-29T00:52:41.156 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:41 GMT /hiiqos-62/ 2023-07-29T00:52:41.156 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+F31U6MSxIhqn1PM+3SZALJsOk4= 2023-07-29T00:52:41.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+F31U6MSxIhqn1PM+3SZALJsOk4=', 'Content-Length': '0'} 2023-07-29T00:52:41.159 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092f1506bf5eae57c-0064c462d9-2068-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:41.159 DEBUG:boto:b'hiiqos-62obj441000falseobj442023-07-29T00:50:33.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:52:41.160 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:52:41.161 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:52:41.161 DEBUG:boto:path=/hiiqos-62/?prefix=obj44 2023-07-29T00:52:41.161 DEBUG:boto:auth_path=/hiiqos-62/?prefix=obj44 2023-07-29T00:52:41.162 DEBUG:boto:Method: GET 2023-07-29T00:52:41.162 DEBUG:boto:Path: /hiiqos-62/?prefix=obj44 2023-07-29T00:52:41.162 DEBUG:boto:Data: 2023-07-29T00:52:41.163 DEBUG:boto:Headers: {} 2023-07-29T00:52:41.163 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:41.163 DEBUG:boto:Port: 8000 2023-07-29T00:52:41.164 DEBUG:boto:Params: {} 2023-07-29T00:52:41.164 DEBUG:boto:Token: None 2023-07-29T00:52:41.165 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:41 GMT /hiiqos-62/ 2023-07-29T00:52:41.165 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+F31U6MSxIhqn1PM+3SZALJsOk4= 2023-07-29T00:52:41.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+F31U6MSxIhqn1PM+3SZALJsOk4=', 'Content-Length': '0'} 2023-07-29T00:52:41.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eef6195f0222b9b5-0064c462d9-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:41.168 DEBUG:boto:b'hiiqos-62obj441000falseobj442023-07-29T00:50:33.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:52:41.169 DEBUG:boto:path=/hiiqos-62/obj44 2023-07-29T00:52:41.169 DEBUG:boto:auth_path=/hiiqos-62/obj44 2023-07-29T00:52:41.170 DEBUG:boto:Method: HEAD 2023-07-29T00:52:41.170 DEBUG:boto:Path: /hiiqos-62/obj44 2023-07-29T00:52:41.171 DEBUG:boto:Data: 2023-07-29T00:52:41.171 DEBUG:boto:Headers: {} 2023-07-29T00:52:41.171 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:41.172 DEBUG:boto:Port: 8001 2023-07-29T00:52:41.172 DEBUG:boto:Params: {} 2023-07-29T00:52:41.172 DEBUG:boto:Token: None 2023-07-29T00:52:41.173 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:41 GMT /hiiqos-62/obj44 2023-07-29T00:52:41.173 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QrfIIoD+f6CgZMQIlw8wDCkOGiQ= 2023-07-29T00:52:41.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QrfIIoD+f6CgZMQIlw8wDCkOGiQ=', 'Content-Length': '0'} 2023-07-29T00:52:41.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-62:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003eb27af074815ea0-0064c462d9-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:41.176 DEBUG:boto:path=/hiiqos-62/obj44 2023-07-29T00:52:41.176 DEBUG:boto:auth_path=/hiiqos-62/obj44 2023-07-29T00:52:41.177 DEBUG:boto:Method: HEAD 2023-07-29T00:52:41.177 DEBUG:boto:Path: /hiiqos-62/obj44 2023-07-29T00:52:41.177 DEBUG:boto:Data: 2023-07-29T00:52:41.178 DEBUG:boto:Headers: {} 2023-07-29T00:52:41.178 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:41.178 DEBUG:boto:Port: 8000 2023-07-29T00:52:41.179 DEBUG:boto:Params: {} 2023-07-29T00:52:41.179 DEBUG:boto:Token: None 2023-07-29T00:52:41.179 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:41 GMT /hiiqos-62/obj44 2023-07-29T00:52:41.180 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QrfIIoD+f6CgZMQIlw8wDCkOGiQ= 2023-07-29T00:52:41.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QrfIIoD+f6CgZMQIlw8wDCkOGiQ=', 'Content-Length': '0'} 2023-07-29T00:52:41.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068258a81d0bc0100-0064c462d9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:41.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-07-29T00:52:41.183 DEBUG:boto:path=/hiiqos-62/obj44 2023-07-29T00:52:41.183 DEBUG:boto:auth_path=/hiiqos-62/obj44 2023-07-29T00:52:41.184 DEBUG:boto:Method: GET 2023-07-29T00:52:41.184 DEBUG:boto:Path: /hiiqos-62/obj44 2023-07-29T00:52:41.184 DEBUG:boto:Data: 2023-07-29T00:52:41.185 DEBUG:boto:Headers: {} 2023-07-29T00:52:41.185 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:41.185 DEBUG:boto:Port: 8001 2023-07-29T00:52:41.186 DEBUG:boto:Params: {} 2023-07-29T00:52:41.186 DEBUG:boto:Token: None 2023-07-29T00:52:41.187 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:41 GMT /hiiqos-62/obj44 2023-07-29T00:52:41.187 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XZvX91X7JK7DouwyLba2/KhPKsw= 2023-07-29T00:52:41.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XZvX91X7JK7DouwyLba2/KhPKsw=', 'Content-Length': '0'} 2023-07-29T00:52:41.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-62:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003805295855a6f9d7-0064c462d9-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:41.190 DEBUG:boto:path=/hiiqos-62/obj44 2023-07-29T00:52:41.190 DEBUG:boto:auth_path=/hiiqos-62/obj44 2023-07-29T00:52:41.191 DEBUG:boto:Method: GET 2023-07-29T00:52:41.191 DEBUG:boto:Path: /hiiqos-62/obj44 2023-07-29T00:52:41.192 DEBUG:boto:Data: 2023-07-29T00:52:41.192 DEBUG:boto:Headers: {} 2023-07-29T00:52:41.192 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:41.193 DEBUG:boto:Port: 8000 2023-07-29T00:52:41.193 DEBUG:boto:Params: {} 2023-07-29T00:52:41.193 DEBUG:boto:Token: None 2023-07-29T00:52:41.194 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:41 GMT /hiiqos-62/obj44 2023-07-29T00:52:41.194 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XZvX91X7JK7DouwyLba2/KhPKsw= 2023-07-29T00:52:41.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XZvX91X7JK7DouwyLba2/KhPKsw=', 'Content-Length': '0'} 2023-07-29T00:52:41.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008158e7946235fcff-0064c462d9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:41.198 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:52:41.198 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:52:41.198 DEBUG:boto:path=/hiiqos-62/?prefix=obj44 2023-07-29T00:52:41.199 DEBUG:boto:auth_path=/hiiqos-62/?prefix=obj44 2023-07-29T00:52:41.199 DEBUG:boto:Method: GET 2023-07-29T00:52:41.199 DEBUG:boto:Path: /hiiqos-62/?prefix=obj44 2023-07-29T00:52:41.200 DEBUG:boto:Data: 2023-07-29T00:52:41.200 DEBUG:boto:Headers: {} 2023-07-29T00:52:41.201 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:41.201 DEBUG:boto:Port: 8001 2023-07-29T00:52:41.201 DEBUG:boto:Params: {} 2023-07-29T00:52:41.202 DEBUG:boto:Token: None 2023-07-29T00:52:41.202 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:41 GMT /hiiqos-62/ 2023-07-29T00:52:41.203 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+F31U6MSxIhqn1PM+3SZALJsOk4= 2023-07-29T00:52:41.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+F31U6MSxIhqn1PM+3SZALJsOk4=', 'Content-Length': '0'} 2023-07-29T00:52:41.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfa81d9bedc234df-0064c462d9-2068-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:41.206 DEBUG:boto:b'hiiqos-62obj441000falseobj442023-07-29T00:50:33.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:52:41.207 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:52:41.207 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:52:41.207 DEBUG:boto:path=/hiiqos-62/?prefix=obj44 2023-07-29T00:52:41.208 DEBUG:boto:auth_path=/hiiqos-62/?prefix=obj44 2023-07-29T00:52:41.208 DEBUG:boto:Method: GET 2023-07-29T00:52:41.209 DEBUG:boto:Path: /hiiqos-62/?prefix=obj44 2023-07-29T00:52:41.209 DEBUG:boto:Data: 2023-07-29T00:52:41.209 DEBUG:boto:Headers: {} 2023-07-29T00:52:41.210 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:41.210 DEBUG:boto:Port: 8000 2023-07-29T00:52:41.210 DEBUG:boto:Params: {} 2023-07-29T00:52:41.211 DEBUG:boto:Token: None 2023-07-29T00:52:41.211 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:41 GMT /hiiqos-62/ 2023-07-29T00:52:41.212 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+F31U6MSxIhqn1PM+3SZALJsOk4= 2023-07-29T00:52:41.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+F31U6MSxIhqn1PM+3SZALJsOk4=', 'Content-Length': '0'} 2023-07-29T00:52:41.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a9a5b6e2d685661-0064c462d9-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:41.215 DEBUG:boto:b'hiiqos-62obj441000falseobj442023-07-29T00:50:33.695Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:52:41.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-07-29T00:52:41.216 DEBUG:boto:path=/hiiqos-62/obj55 2023-07-29T00:52:41.217 DEBUG:boto:auth_path=/hiiqos-62/obj55 2023-07-29T00:52:41.217 DEBUG:boto:path=/hiiqos-62/obj55?versionId=null 2023-07-29T00:52:41.218 DEBUG:boto:auth_path=/hiiqos-62/obj55?versionId=null 2023-07-29T00:52:41.218 DEBUG:boto:Method: GET 2023-07-29T00:52:41.218 DEBUG:boto:Path: /hiiqos-62/obj55?versionId=null 2023-07-29T00:52:41.219 DEBUG:boto:Data: 2023-07-29T00:52:41.219 DEBUG:boto:Headers: {} 2023-07-29T00:52:41.219 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:41.220 DEBUG:boto:Port: 8001 2023-07-29T00:52:41.220 DEBUG:boto:Params: {} 2023-07-29T00:52:41.220 DEBUG:boto:Token: None 2023-07-29T00:52:41.221 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:41 GMT /hiiqos-62/obj55?versionId=null 2023-07-29T00:52:41.221 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5Qg9y6KqFAWhxsSNYD20wPFdkhs= 2023-07-29T00:52:41.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5Qg9y6KqFAWhxsSNYD20wPFdkhs=', 'Content-Length': '0'} 2023-07-29T00:52:41.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-62:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5b76a434267b789-0064c462d9-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:41.225 DEBUG:boto:path=/hiiqos-62/obj55 2023-07-29T00:52:41.225 DEBUG:boto:auth_path=/hiiqos-62/obj55 2023-07-29T00:52:41.225 DEBUG:boto:path=/hiiqos-62/obj55?versionId=null 2023-07-29T00:52:41.226 DEBUG:boto:auth_path=/hiiqos-62/obj55?versionId=null 2023-07-29T00:52:41.226 DEBUG:boto:Method: GET 2023-07-29T00:52:41.226 DEBUG:boto:Path: /hiiqos-62/obj55?versionId=null 2023-07-29T00:52:41.227 DEBUG:boto:Data: 2023-07-29T00:52:41.227 DEBUG:boto:Headers: {} 2023-07-29T00:52:41.227 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:41.228 DEBUG:boto:Port: 8000 2023-07-29T00:52:41.228 DEBUG:boto:Params: {} 2023-07-29T00:52:41.229 DEBUG:boto:Token: None 2023-07-29T00:52:41.229 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:41 GMT /hiiqos-62/obj55?versionId=null 2023-07-29T00:52:41.229 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5Qg9y6KqFAWhxsSNYD20wPFdkhs= 2023-07-29T00:52:41.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5Qg9y6KqFAWhxsSNYD20wPFdkhs=', 'Content-Length': '0'} 2023-07-29T00:52:41.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e47402d566ff8d09-0064c462d9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:41.232 DEBUG:boto:path=/hiiqos-62/obj55 2023-07-29T00:52:41.233 DEBUG:boto:auth_path=/hiiqos-62/obj55 2023-07-29T00:52:41.233 DEBUG:boto:path=/hiiqos-62/obj55?versionId=null 2023-07-29T00:52:41.233 DEBUG:boto:auth_path=/hiiqos-62/obj55?versionId=null 2023-07-29T00:52:41.234 DEBUG:boto:Method: HEAD 2023-07-29T00:52:41.234 DEBUG:boto:Path: /hiiqos-62/obj55?versionId=null 2023-07-29T00:52:41.234 DEBUG:boto:Data: 2023-07-29T00:52:41.235 DEBUG:boto:Headers: {} 2023-07-29T00:52:41.235 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:41.235 DEBUG:boto:Port: 8001 2023-07-29T00:52:41.236 DEBUG:boto:Params: {} 2023-07-29T00:52:41.236 DEBUG:boto:Token: None 2023-07-29T00:52:41.237 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:41 GMT /hiiqos-62/obj55?versionId=null 2023-07-29T00:52:41.237 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:30FwiaXNCFQ50xC5MWoeWWb4JC4= 2023-07-29T00:52:41.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:30FwiaXNCFQ50xC5MWoeWWb4JC4=', 'Content-Length': '0'} 2023-07-29T00:52:41.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-62:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec06c3c1b0245eab-0064c462d9-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:41.240 DEBUG:boto:path=/hiiqos-62/obj55 2023-07-29T00:52:41.240 DEBUG:boto:auth_path=/hiiqos-62/obj55 2023-07-29T00:52:41.241 DEBUG:boto:path=/hiiqos-62/obj55?versionId=null 2023-07-29T00:52:41.241 DEBUG:boto:auth_path=/hiiqos-62/obj55?versionId=null 2023-07-29T00:52:41.241 DEBUG:boto:Method: HEAD 2023-07-29T00:52:41.242 DEBUG:boto:Path: /hiiqos-62/obj55?versionId=null 2023-07-29T00:52:41.242 DEBUG:boto:Data: 2023-07-29T00:52:41.242 DEBUG:boto:Headers: {} 2023-07-29T00:52:41.243 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:41.243 DEBUG:boto:Port: 8000 2023-07-29T00:52:41.243 DEBUG:boto:Params: {} 2023-07-29T00:52:41.244 DEBUG:boto:Token: None 2023-07-29T00:52:41.244 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:41 GMT /hiiqos-62/obj55?versionId=null 2023-07-29T00:52:41.245 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:30FwiaXNCFQ50xC5MWoeWWb4JC4= 2023-07-29T00:52:41.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:30FwiaXNCFQ50xC5MWoeWWb4JC4=', 'Content-Length': '0'} 2023-07-29T00:52:41.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd4b922463a3bb68-0064c462d9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:41.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-07-29T00:52:41.248 DEBUG:boto:path=/hiiqos-62/obj55 2023-07-29T00:52:41.248 DEBUG:boto:auth_path=/hiiqos-62/obj55 2023-07-29T00:52:41.248 DEBUG:boto:path=/hiiqos-62/obj55?versionId=null 2023-07-29T00:52:41.249 DEBUG:boto:auth_path=/hiiqos-62/obj55?versionId=null 2023-07-29T00:52:41.249 DEBUG:boto:Method: GET 2023-07-29T00:52:41.250 DEBUG:boto:Path: /hiiqos-62/obj55?versionId=null 2023-07-29T00:52:41.250 DEBUG:boto:Data: 2023-07-29T00:52:41.250 DEBUG:boto:Headers: {} 2023-07-29T00:52:41.251 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:41.251 DEBUG:boto:Port: 8001 2023-07-29T00:52:41.251 DEBUG:boto:Params: {} 2023-07-29T00:52:41.252 DEBUG:boto:Token: None 2023-07-29T00:52:41.252 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:41 GMT /hiiqos-62/obj55?versionId=null 2023-07-29T00:52:41.253 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5Qg9y6KqFAWhxsSNYD20wPFdkhs= 2023-07-29T00:52:41.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5Qg9y6KqFAWhxsSNYD20wPFdkhs=', 'Content-Length': '0'} 2023-07-29T00:52:41.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-62:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000135d6d2a67dea34c-0064c462d9-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:41.256 DEBUG:boto:path=/hiiqos-62/obj55 2023-07-29T00:52:41.256 DEBUG:boto:auth_path=/hiiqos-62/obj55 2023-07-29T00:52:41.257 DEBUG:boto:path=/hiiqos-62/obj55?versionId=null 2023-07-29T00:52:41.257 DEBUG:boto:auth_path=/hiiqos-62/obj55?versionId=null 2023-07-29T00:52:41.257 DEBUG:boto:Method: GET 2023-07-29T00:52:41.258 DEBUG:boto:Path: /hiiqos-62/obj55?versionId=null 2023-07-29T00:52:41.258 DEBUG:boto:Data: 2023-07-29T00:52:41.259 DEBUG:boto:Headers: {} 2023-07-29T00:52:41.259 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:41.259 DEBUG:boto:Port: 8000 2023-07-29T00:52:41.260 DEBUG:boto:Params: {} 2023-07-29T00:52:41.260 DEBUG:boto:Token: None 2023-07-29T00:52:41.261 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:41 GMT /hiiqos-62/obj55?versionId=null 2023-07-29T00:52:41.261 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5Qg9y6KqFAWhxsSNYD20wPFdkhs= 2023-07-29T00:52:41.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5Qg9y6KqFAWhxsSNYD20wPFdkhs=', 'Content-Length': '0'} 2023-07-29T00:52:41.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000969fb81f4565bda6-0064c462d9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:41.264 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:52:41.265 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:52:41.265 DEBUG:boto:path=/hiiqos-62/?prefix=obj55 2023-07-29T00:52:41.266 DEBUG:boto:auth_path=/hiiqos-62/?prefix=obj55 2023-07-29T00:52:41.266 DEBUG:boto:Method: GET 2023-07-29T00:52:41.266 DEBUG:boto:Path: /hiiqos-62/?prefix=obj55 2023-07-29T00:52:41.267 DEBUG:boto:Data: 2023-07-29T00:52:41.267 DEBUG:boto:Headers: {} 2023-07-29T00:52:41.267 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:41.268 DEBUG:boto:Port: 8001 2023-07-29T00:52:41.268 DEBUG:boto:Params: {} 2023-07-29T00:52:41.269 DEBUG:boto:Token: None 2023-07-29T00:52:41.269 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:41 GMT /hiiqos-62/ 2023-07-29T00:52:41.269 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+F31U6MSxIhqn1PM+3SZALJsOk4= 2023-07-29T00:52:41.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+F31U6MSxIhqn1PM+3SZALJsOk4=', 'Content-Length': '0'} 2023-07-29T00:52:41.273 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082a121f936f88e82-0064c462d9-2068-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:41.273 DEBUG:boto:b'hiiqos-62obj551000falseobj552023-07-29T00:50:33.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:52:41.274 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:52:41.274 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:52:41.275 DEBUG:boto:path=/hiiqos-62/?prefix=obj55 2023-07-29T00:52:41.275 DEBUG:boto:auth_path=/hiiqos-62/?prefix=obj55 2023-07-29T00:52:41.275 DEBUG:boto:Method: GET 2023-07-29T00:52:41.276 DEBUG:boto:Path: /hiiqos-62/?prefix=obj55 2023-07-29T00:52:41.276 DEBUG:boto:Data: 2023-07-29T00:52:41.276 DEBUG:boto:Headers: {} 2023-07-29T00:52:41.277 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:41.277 DEBUG:boto:Port: 8000 2023-07-29T00:52:41.277 DEBUG:boto:Params: {} 2023-07-29T00:52:41.278 DEBUG:boto:Token: None 2023-07-29T00:52:41.278 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:41 GMT /hiiqos-62/ 2023-07-29T00:52:41.279 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+F31U6MSxIhqn1PM+3SZALJsOk4= 2023-07-29T00:52:41.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+F31U6MSxIhqn1PM+3SZALJsOk4=', 'Content-Length': '0'} 2023-07-29T00:52:41.282 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ef0a20df47e6035-0064c462d9-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:41.282 DEBUG:boto:b'hiiqos-62obj551000falseobj552023-07-29T00:50:33.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:52:41.283 DEBUG:boto:path=/hiiqos-62/obj55 2023-07-29T00:52:41.283 DEBUG:boto:auth_path=/hiiqos-62/obj55 2023-07-29T00:52:41.284 DEBUG:boto:Method: HEAD 2023-07-29T00:52:41.284 DEBUG:boto:Path: /hiiqos-62/obj55 2023-07-29T00:52:41.284 DEBUG:boto:Data: 2023-07-29T00:52:41.285 DEBUG:boto:Headers: {} 2023-07-29T00:52:41.285 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:41.285 DEBUG:boto:Port: 8001 2023-07-29T00:52:41.286 DEBUG:boto:Params: {} 2023-07-29T00:52:41.286 DEBUG:boto:Token: None 2023-07-29T00:52:41.286 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:41 GMT /hiiqos-62/obj55 2023-07-29T00:52:41.287 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:14Ym4/DfFRnZTfiJTSkek3HHz4Y= 2023-07-29T00:52:41.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:14Ym4/DfFRnZTfiJTSkek3HHz4Y=', 'Content-Length': '0'} 2023-07-29T00:52:41.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-62:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbfe753fcd9116a5-0064c462d9-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:41.290 DEBUG:boto:path=/hiiqos-62/obj55 2023-07-29T00:52:41.290 DEBUG:boto:auth_path=/hiiqos-62/obj55 2023-07-29T00:52:41.290 DEBUG:boto:Method: HEAD 2023-07-29T00:52:41.291 DEBUG:boto:Path: /hiiqos-62/obj55 2023-07-29T00:52:41.291 DEBUG:boto:Data: 2023-07-29T00:52:41.291 DEBUG:boto:Headers: {} 2023-07-29T00:52:41.292 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:41.292 DEBUG:boto:Port: 8000 2023-07-29T00:52:41.292 DEBUG:boto:Params: {} 2023-07-29T00:52:41.293 DEBUG:boto:Token: None 2023-07-29T00:52:41.293 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:41 GMT /hiiqos-62/obj55 2023-07-29T00:52:41.294 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:14Ym4/DfFRnZTfiJTSkek3HHz4Y= 2023-07-29T00:52:41.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:14Ym4/DfFRnZTfiJTSkek3HHz4Y=', 'Content-Length': '0'} 2023-07-29T00:52:41.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000305862c1b4164551-0064c462d9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:41.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-07-29T00:52:41.297 DEBUG:boto:path=/hiiqos-62/obj55 2023-07-29T00:52:41.297 DEBUG:boto:auth_path=/hiiqos-62/obj55 2023-07-29T00:52:41.298 DEBUG:boto:Method: GET 2023-07-29T00:52:41.298 DEBUG:boto:Path: /hiiqos-62/obj55 2023-07-29T00:52:41.298 DEBUG:boto:Data: 2023-07-29T00:52:41.299 DEBUG:boto:Headers: {} 2023-07-29T00:52:41.299 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:41.299 DEBUG:boto:Port: 8001 2023-07-29T00:52:41.300 DEBUG:boto:Params: {} 2023-07-29T00:52:41.300 DEBUG:boto:Token: None 2023-07-29T00:52:41.301 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:41 GMT /hiiqos-62/obj55 2023-07-29T00:52:41.301 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Q38LZJvsVgdEaqOkdmKFQ5pJ7hY= 2023-07-29T00:52:41.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q38LZJvsVgdEaqOkdmKFQ5pJ7hY=', 'Content-Length': '0'} 2023-07-29T00:52:41.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-62:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bf28a0370bc704c-0064c462d9-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:41.344 DEBUG:boto:path=/hiiqos-62/obj55 2023-07-29T00:52:41.345 DEBUG:boto:auth_path=/hiiqos-62/obj55 2023-07-29T00:52:41.345 DEBUG:boto:Method: GET 2023-07-29T00:52:41.346 DEBUG:boto:Path: /hiiqos-62/obj55 2023-07-29T00:52:41.346 DEBUG:boto:Data: 2023-07-29T00:52:41.346 DEBUG:boto:Headers: {} 2023-07-29T00:52:41.347 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:41.347 DEBUG:boto:Port: 8000 2023-07-29T00:52:41.347 DEBUG:boto:Params: {} 2023-07-29T00:52:41.348 DEBUG:boto:Token: None 2023-07-29T00:52:41.348 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:41 GMT /hiiqos-62/obj55 2023-07-29T00:52:41.349 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Q38LZJvsVgdEaqOkdmKFQ5pJ7hY= 2023-07-29T00:52:41.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q38LZJvsVgdEaqOkdmKFQ5pJ7hY=', 'Content-Length': '0'} 2023-07-29T00:52:41.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:50:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e93a9916674d992-0064c462d9-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:41.353 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:52:41.353 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:52:41.354 DEBUG:boto:path=/hiiqos-62/?prefix=obj55 2023-07-29T00:52:41.354 DEBUG:boto:auth_path=/hiiqos-62/?prefix=obj55 2023-07-29T00:52:41.354 DEBUG:boto:Method: GET 2023-07-29T00:52:41.355 DEBUG:boto:Path: /hiiqos-62/?prefix=obj55 2023-07-29T00:52:41.355 DEBUG:boto:Data: 2023-07-29T00:52:41.355 DEBUG:boto:Headers: {} 2023-07-29T00:52:41.356 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:52:41.356 DEBUG:boto:Port: 8001 2023-07-29T00:52:41.357 DEBUG:boto:Params: {} 2023-07-29T00:52:41.357 DEBUG:boto:Token: None 2023-07-29T00:52:41.357 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:41 GMT /hiiqos-62/ 2023-07-29T00:52:41.358 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+F31U6MSxIhqn1PM+3SZALJsOk4= 2023-07-29T00:52:41.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+F31U6MSxIhqn1PM+3SZALJsOk4=', 'Content-Length': '0'} 2023-07-29T00:52:41.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aed3eccd7b725c6b-0064c462d9-2068-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:41.361 DEBUG:boto:b'hiiqos-62obj551000falseobj552023-07-29T00:50:33.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:52:41.362 DEBUG:boto:path=/hiiqos-62/ 2023-07-29T00:52:41.362 DEBUG:boto:auth_path=/hiiqos-62/ 2023-07-29T00:52:41.363 DEBUG:boto:path=/hiiqos-62/?prefix=obj55 2023-07-29T00:52:41.363 DEBUG:boto:auth_path=/hiiqos-62/?prefix=obj55 2023-07-29T00:52:41.363 DEBUG:boto:Method: GET 2023-07-29T00:52:41.364 DEBUG:boto:Path: /hiiqos-62/?prefix=obj55 2023-07-29T00:52:41.364 DEBUG:boto:Data: 2023-07-29T00:52:41.365 DEBUG:boto:Headers: {} 2023-07-29T00:52:41.365 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:41.366 DEBUG:boto:Port: 8000 2023-07-29T00:52:41.366 DEBUG:boto:Params: {} 2023-07-29T00:52:41.367 DEBUG:boto:Token: None 2023-07-29T00:52:41.367 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:52:41 GMT /hiiqos-62/ 2023-07-29T00:52:41.368 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+F31U6MSxIhqn1PM+3SZALJsOk4= 2023-07-29T00:52:41.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+F31U6MSxIhqn1PM+3SZALJsOk4=', 'Content-Length': '0'} 2023-07-29T00:52:41.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3eeca9db4b93805-0064c462d9-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:41.371 DEBUG:boto:b'hiiqos-62obj551000falseobj552023-07-29T00:50:33.714Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:52:41.372 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-62 zones={a2, a1} 2023-07-29T00:52:41.372 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-07-29T00:52:41.372 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:52:41.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:52:41.373 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:52:41.428 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:52:41.428 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:52:41.481 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T00:52:41.481 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:52:41.481 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000037952", 2023-07-29T00:52:41.481 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:50:33.698375Z" 2023-07-29T00:52:41.482 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:52:41.482 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:52:41.482 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000033424", 2023-07-29T00:52:41.482 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:50:43.174664Z" 2023-07-29T00:52:41.482 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:52:41.482 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:52:41.482 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000032634", 2023-07-29T00:52:41.483 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:50:43.176329Z" 2023-07-29T00:52:41.483 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:52:41.483 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:52:41.483 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000022717", 2023-07-29T00:52:41.483 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:45:13.171537Z" 2023-07-29T00:52:41.483 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:52:41.483 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T00:52:41.484 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000037952', 1: '00000000000000000000:00000000000000033424', 2: '00000000000000000000:00000000000000032634', 3: '00000000000000000000:00000000000000022717'} 2023-07-29T00:52:41.485 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-07-29T00:52:41.485 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:52:41.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:52:41.486 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:52:41.534 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:52:41.535 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:52:41.577 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:52:41.578 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:52:41.578 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:52:41.578 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:52:41.578 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:52:41.579 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 16864251767192347489 2023-07-29T00:52:41.579 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:52:41.579 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:52:41.579 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:52:41.579 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:52:41.580 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:52:41.580 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:52:41.580 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000037952", 2023-07-29T00:52:41.580 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:52:41.580 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 53, 2023-07-29T00:52:41.580 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:52:41.581 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:50:33.698375Z" 2023-07-29T00:52:41.581 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:52:41.581 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:52:41.581 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:52:41.581 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:52:41.581 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:52:41.582 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:52:41.582 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000033424", 2023-07-29T00:52:41.582 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:52:41.582 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T00:52:41.582 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:52:41.582 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:50:43.174664Z" 2023-07-29T00:52:41.582 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:52:41.583 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:52:41.583 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:52:41.583 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:52:41.583 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:52:41.583 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:52:41.583 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000032634", 2023-07-29T00:52:41.584 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:52:41.584 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:52:41.584 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:52:41.584 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:50:43.176329Z" 2023-07-29T00:52:41.584 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:52:41.584 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:52:41.585 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:52:41.585 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:52:41.585 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:52:41.585 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:52:41.585 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000022717", 2023-07-29T00:52:41.585 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:52:41.586 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:52:41.586 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:52:41.586 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:45:13.171537Z" 2023-07-29T00:52:41.586 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:52:41.586 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:52:41.586 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:52:41.586 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:52:41.587 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:52:41.587 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 62, 2023-07-29T00:52:41.587 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 62 2023-07-29T00:52:41.587 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:52:41.587 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:52:41Z" 2023-07-29T00:52:41.587 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:52:41.588 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16864251767192347489 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000037952", "next_step_marker": "", "total_entries": 53, "pos": 0, "timestamp": "2023-07-29T00:50:33.698375Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033424", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T00:50:43.174664Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032634", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:50:43.176329Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000022717", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:45:13.171537Z" } } ] }, "full_sync": { "total": 62, "complete": 62 }, "current_time": "2023-07-29T00:52:41Z" } 2023-07-29T00:52:41.589 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000037952', 'next_step_marker': '', 'total_entries': 53, 'pos': 0, 'timestamp': '2023-07-29T00:50:33.698375Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033424', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T00:50:43.174664Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032634', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:50:43.176329Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022717', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:45:13.171537Z'}}] 2023-07-29T00:52:41.589 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000037952', 1: '00000000000000000000:00000000000000033424', 2: '00000000000000000000:00000000000000032634', 3: '00000000000000000000:00000000000000022717'} 2023-07-29T00:52:41.589 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000037952', 1: '00000000000000000000:00000000000000033424', 2: '00000000000000000000:00000000000000032634', 3: '00000000000000000000:00000000000000022717'} 2023-07-29T00:52:41.590 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-07-29T00:52:41.590 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-07-29T00:52:41.591 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:52:41.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:52:41.592 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:52:41.637 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:52:41.637 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:52:41.685 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T00:52:41.685 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:52:41.685 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000015165", 2023-07-29T00:52:41.685 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:52:04.630196Z" 2023-07-29T00:52:41.685 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:52:41.686 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:52:41.686 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000014463", 2023-07-29T00:52:41.686 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:52:23.933540Z" 2023-07-29T00:52:41.686 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:52:41.686 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:52:41.686 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000011396", 2023-07-29T00:52:41.687 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:52:23.935203Z" 2023-07-29T00:52:41.687 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:52:41.687 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:52:41.687 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000005445", 2023-07-29T00:52:41.687 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:47:15.403544Z" 2023-07-29T00:52:41.687 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:52:41.688 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T00:52:41.689 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000015165', 1: '00000000000000000000:00000000000000014463', 2: '00000000000000000000:00000000000000011396', 3: '00000000000000000000:00000000000000005445'} 2023-07-29T00:52:41.689 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-07-29T00:52:41.690 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:52:41.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:52:41.691 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:52:41.777 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:52:41.777 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:52:41.821 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:52:41.821 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:52:41.822 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:52:41.822 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:52:41.822 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:52:41.822 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 17882949198279251152 2023-07-29T00:52:41.822 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:52:41.822 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:52:41.823 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:52:41.823 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:52:41.823 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:52:41.823 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:52:41.823 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000015165", 2023-07-29T00:52:41.823 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:52:41.824 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:52:41.824 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:52:41.824 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:52:04.630196Z" 2023-07-29T00:52:41.824 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:52:41.824 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:52:41.824 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:52:41.825 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:52:41.825 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:52:41.825 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:52:41.825 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000014463", 2023-07-29T00:52:41.825 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:52:41.825 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:52:41.826 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:52:41.826 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:52:23.933540Z" 2023-07-29T00:52:41.826 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:52:41.826 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:52:41.826 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:52:41.826 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:52:41.826 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:52:41.827 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:52:41.827 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000011396", 2023-07-29T00:52:41.827 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:52:41.827 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:52:41.827 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:52:41.827 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:52:23.935203Z" 2023-07-29T00:52:41.827 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:52:41.828 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:52:41.828 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:52:41.828 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:52:41.828 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:52:41.828 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T00:52:41.828 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000005445", 2023-07-29T00:52:41.829 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:52:41.829 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:52:41.829 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:52:41.829 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:47:15.403544Z" 2023-07-29T00:52:41.829 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:52:41.829 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:52:41.829 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:52:41.830 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:52:41.830 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:52:41.830 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 0, 2023-07-29T00:52:41.830 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 0 2023-07-29T00:52:41.830 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:52:41.830 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:52:41Z" 2023-07-29T00:52:41.831 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:52:41.831 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17882949198279251152 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015165", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:52:04.630196Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000014463", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:52:23.933540Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011396", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:52:23.935203Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005445", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:47:15.403544Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-07-29T00:52:41Z" } 2023-07-29T00:52:41.832 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015165', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:52:04.630196Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014463', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:52:23.933540Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011396', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:52:23.935203Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005445', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:47:15.403544Z'}}] 2023-07-29T00:52:41.832 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015165', 1: '00000000000000000000:00000000000000014463', 2: '00000000000000000000:00000000000000011396', 3: '00000000000000000000:00000000000000005445'} 2023-07-29T00:52:41.833 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015165', 1: '00000000000000000000:00000000000000014463', 2: '00000000000000000000:00000000000000011396', 3: '00000000000000000000:00000000000000005445'} 2023-07-29T00:52:41.833 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-07-29T00:52:41.834 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_incremental_sync_after_data_sync_init ... ok 2023-07-29T00:52:41.835 DEBUG:boto:path=/hiiqos-63/ 2023-07-29T00:52:41.835 DEBUG:boto:auth_path=/hiiqos-63/ 2023-07-29T00:52:41.835 DEBUG:boto:Method: PUT 2023-07-29T00:52:41.836 DEBUG:boto:Path: /hiiqos-63/ 2023-07-29T00:52:41.836 DEBUG:boto:Data: 2023-07-29T00:52:41.837 DEBUG:boto:Headers: {} 2023-07-29T00:52:41.837 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:41.837 DEBUG:boto:Port: 8000 2023-07-29T00:52:41.838 DEBUG:boto:Params: {} 2023-07-29T00:52:41.838 DEBUG:boto:Token: None 2023-07-29T00:52:41.838 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 00:52:41 GMT /hiiqos-63/ 2023-07-29T00:52:41.839 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2bvNZ1aBirhX3w6bCKRv8F8HJwY= 2023-07-29T00:52:41.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2bvNZ1aBirhX3w6bCKRv8F8HJwY=', 'Content-Length': '0'} 2023-07-29T00:52:41.853 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000078fe0d60fa4a4563-0064c462d9-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:41.853 DEBUG:rgw_multi.tests:created bucket=hiiqos-63 2023-07-29T00:52:41.854 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:52:41.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:52:41.855 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:52:41.901 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:52:41.901 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:52:41.930 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T00:52:41.943 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T00:52:41.944 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:52:41.944 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T00:52:41.944 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:17.836402Z" 2023-07-29T00:52:41.944 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:52:41.944 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:52:41.945 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591961.778137_1171.1", 2023-07-29T00:52:41.945 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:52:41.778137Z" 2023-07-29T00:52:41.945 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:52:41.945 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:52:41.945 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:52:41.945 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T00:52:41.946 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:52:41.946 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:52:41.946 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T00:52:41.946 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:12.811406Z" 2023-07-29T00:52:41.946 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:52:41.946 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T00:52:41.947 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690591217.836402_294.1', 1: '1_1690591961.778137_1171.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T00:52:41.948 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T00:52:41.948 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:52:41.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:52:41.949 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:52:41.996 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:52:41.997 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:52:42.023 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:52:41.948+0000 7fc769610880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:52:42.024 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:52:41.949+0000 7fc769610880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:52:42.038 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:52:42.038 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:52:42.038 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:52:42.039 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:52:42.039 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:52:42.039 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:52:42.040 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:52:42.040 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:52:42.040 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:52:42.040 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:52:42.041 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:52:42.041 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:52:42.041 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:52:42.041 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T00:52:42.042 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:52:42.042 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:52:42.042 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:52:42.043 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T00:52:42.043 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:52:42.043 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:52:42.043 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:52:42.044 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:52:42.044 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:52:42.044 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:52:42.044 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:52:42.045 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591806.661758_1144.1", 2023-07-29T00:52:42.045 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:52:42.045 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:52:42.045 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:52:42.046 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:50:06.661758Z", 2023-07-29T00:52:42.046 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:52:42.046 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:52:42.046 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:52:42.047 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:52:42.047 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:52:42.047 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:52:42.048 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:52:42.048 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:52:42.048 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:52:42.048 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:52:42.049 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:52:42.049 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:52:42.049 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:52:42.049 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:52:42.050 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:52:42.050 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:52:42.050 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:52:42.050 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:52:42.051 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:52:42.051 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T00:52:42.051 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:52:42.051 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:52:42.052 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:52:42.052 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T00:52:42.052 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:52:42.053 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:52:42.053 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:52:42.053 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:52:42.053 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:52:42.054 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:52:42.054 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:52:42.054 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:52:42.054 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:52:42.055 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:52:41Z" 2023-07-29T00:52:42.055 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:52:42.056 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690591806.661758_1144.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:50:06.661758Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:52:41Z" } 2023-07-29T00:52:42.057 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690591806.661758_1144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:50:06.661758Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T00:52:42.058 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690591961.778137_1171.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T00:52:42.058 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690591806.661758_1144.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T00:52:42.059 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690591961.778137_1171.1 target=1_1690591806.661758_1144.1 2023-07-29T00:52:42.964 INFO:tasks.rgw.c2.client.0.smithi139.stdout:2023-07-29T00:52:42.890+0000 7f5f2671b640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T00:52:47.060 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:52:47.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:52:47.062 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:52:47.118 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:52:47.118 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:52:47.150 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:52:47.076+0000 7f59b7a96880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:52:47.151 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:52:47.077+0000 7f59b7a96880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:52:47.166 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:52:47.167 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:52:47.167 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:52:47.167 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:52:47.167 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:52:47.167 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:52:47.168 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:52:47.168 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:52:47.168 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:52:47.168 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:52:47.168 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:52:47.169 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:52:47.169 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:52:47.169 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T00:52:47.169 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:52:47.169 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:52:47.169 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:52:47.169 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T00:52:47.170 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:52:47.170 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:52:47.170 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:52:47.170 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:52:47.170 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:52:47.170 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:52:47.170 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:52:47.171 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591961.778137_1171.1", 2023-07-29T00:52:47.171 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:52:47.171 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:52:47.171 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:52:47.171 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:52:41.778137Z", 2023-07-29T00:52:47.172 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:52:47.172 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:52:47.172 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:52:47.172 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:52:47.172 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:52:47.172 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:52:47.172 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:52:47.173 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:52:47.173 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:52:47.173 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:52:47.173 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:52:47.173 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:52:47.173 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:52:47.174 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:52:47.174 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:52:47.174 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:52:47.174 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:52:47.174 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:52:47.174 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:52:47.174 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T00:52:47.175 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:52:47.175 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:52:47.175 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:52:47.175 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T00:52:47.175 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:52:47.175 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:52:47.175 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:52:47.176 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:52:47.176 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:52:47.176 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:52:47.176 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:52:47.176 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:52:47.176 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:52:47.177 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:52:47Z" 2023-07-29T00:52:47.177 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:52:47.178 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690591961.778137_1171.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:52:41.778137Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:52:47Z" } 2023-07-29T00:52:47.178 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690591961.778137_1171.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:52:41.778137Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T00:52:47.179 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690591961.778137_1171.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T00:52:47.179 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690591961.778137_1171.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T00:52:47.180 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T00:52:47.180 DEBUG:boto:path=/hiiqos-63/ 2023-07-29T00:52:47.180 DEBUG:boto:auth_path=/hiiqos-63/ 2023-07-29T00:52:47.181 DEBUG:boto:Method: HEAD 2023-07-29T00:52:47.181 DEBUG:boto:Path: /hiiqos-63/ 2023-07-29T00:52:47.182 DEBUG:boto:Data: 2023-07-29T00:52:47.183 DEBUG:boto:Headers: {} 2023-07-29T00:52:47.184 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:47.185 DEBUG:boto:Port: 8000 2023-07-29T00:52:47.185 DEBUG:boto:Params: {} 2023-07-29T00:52:47.186 DEBUG:boto:Token: None 2023-07-29T00:52:47.187 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:47 GMT /hiiqos-63/ 2023-07-29T00:52:47.188 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY= 2023-07-29T00:52:47.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY=', 'Content-Length': '0'} 2023-07-29T00:52:47.192 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006aaefe0c62978ac9-0064c462df-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:47.193 DEBUG:boto:path=/hiiqos-63/obj11 2023-07-29T00:52:47.194 DEBUG:boto:auth_path=/hiiqos-63/obj11 2023-07-29T00:52:47.195 DEBUG:boto:Method: PUT 2023-07-29T00:52:47.195 DEBUG:boto:Path: /hiiqos-63/obj11 2023-07-29T00:52:47.196 DEBUG:boto:Data: 2023-07-29T00:52:47.196 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:52:47.197 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:47.197 DEBUG:boto:Port: 8000 2023-07-29T00:52:47.198 DEBUG:boto:Params: {} 2023-07-29T00:52:47.199 DEBUG:boto:Token: None 2023-07-29T00:52:47.200 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:52:47 GMT /hiiqos-63/obj11 2023-07-29T00:52:47.201 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FDEAqF4McvsPS8wFDkpORGSt3DU= 2023-07-29T00:52:47.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FDEAqF4McvsPS8wFDkpORGSt3DU='} 2023-07-29T00:52:47.208 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000091d131e7330402e2-0064c462df-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:47.209 DEBUG:boto:path=/hiiqos-63/ 2023-07-29T00:52:47.210 DEBUG:boto:auth_path=/hiiqos-63/ 2023-07-29T00:52:47.210 DEBUG:boto:Method: HEAD 2023-07-29T00:52:47.210 DEBUG:boto:Path: /hiiqos-63/ 2023-07-29T00:52:47.211 DEBUG:boto:Data: 2023-07-29T00:52:47.211 DEBUG:boto:Headers: {} 2023-07-29T00:52:47.211 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:47.212 DEBUG:boto:Port: 8000 2023-07-29T00:52:47.212 DEBUG:boto:Params: {} 2023-07-29T00:52:47.212 DEBUG:boto:Token: None 2023-07-29T00:52:47.213 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:47 GMT /hiiqos-63/ 2023-07-29T00:52:47.213 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY= 2023-07-29T00:52:47.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY=', 'Content-Length': '0'} 2023-07-29T00:52:47.216 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000056d781a5e1a4cba8-0064c462df-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:47.216 DEBUG:boto:path=/hiiqos-63/obj22 2023-07-29T00:52:47.217 DEBUG:boto:auth_path=/hiiqos-63/obj22 2023-07-29T00:52:47.217 DEBUG:boto:Method: PUT 2023-07-29T00:52:47.217 DEBUG:boto:Path: /hiiqos-63/obj22 2023-07-29T00:52:47.218 DEBUG:boto:Data: 2023-07-29T00:52:47.218 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:52:47.218 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:47.219 DEBUG:boto:Port: 8000 2023-07-29T00:52:47.219 DEBUG:boto:Params: {} 2023-07-29T00:52:47.219 DEBUG:boto:Token: None 2023-07-29T00:52:47.220 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:52:47 GMT /hiiqos-63/obj22 2023-07-29T00:52:47.220 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5RfvdwOAWQ1/fhrpxAmKgzv6ayY= 2023-07-29T00:52:47.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5RfvdwOAWQ1/fhrpxAmKgzv6ayY='} 2023-07-29T00:52:47.224 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000082848076080bd587-0064c462df-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:47.225 DEBUG:boto:path=/hiiqos-63/ 2023-07-29T00:52:47.226 DEBUG:boto:auth_path=/hiiqos-63/ 2023-07-29T00:52:47.226 DEBUG:boto:Method: HEAD 2023-07-29T00:52:47.227 DEBUG:boto:Path: /hiiqos-63/ 2023-07-29T00:52:47.227 DEBUG:boto:Data: 2023-07-29T00:52:47.227 DEBUG:boto:Headers: {} 2023-07-29T00:52:47.228 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:47.228 DEBUG:boto:Port: 8000 2023-07-29T00:52:47.228 DEBUG:boto:Params: {} 2023-07-29T00:52:47.229 DEBUG:boto:Token: None 2023-07-29T00:52:47.229 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:47 GMT /hiiqos-63/ 2023-07-29T00:52:47.230 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY= 2023-07-29T00:52:47.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY=', 'Content-Length': '0'} 2023-07-29T00:52:47.233 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000241fd97bbd7fb88c-0064c462df-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:47.233 DEBUG:boto:path=/hiiqos-63/obj33 2023-07-29T00:52:47.234 DEBUG:boto:auth_path=/hiiqos-63/obj33 2023-07-29T00:52:47.235 DEBUG:boto:Method: PUT 2023-07-29T00:52:47.235 DEBUG:boto:Path: /hiiqos-63/obj33 2023-07-29T00:52:47.235 DEBUG:boto:Data: 2023-07-29T00:52:47.236 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:52:47.236 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:47.237 DEBUG:boto:Port: 8000 2023-07-29T00:52:47.237 DEBUG:boto:Params: {} 2023-07-29T00:52:47.237 DEBUG:boto:Token: None 2023-07-29T00:52:47.238 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:52:47 GMT /hiiqos-63/obj33 2023-07-29T00:52:47.238 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rV2w/2Hzud3s1dV3aLkD89M7VrE= 2023-07-29T00:52:47.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rV2w/2Hzud3s1dV3aLkD89M7VrE='} 2023-07-29T00:52:47.280 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c2e6443fab7f9142-0064c462df-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:47.281 DEBUG:boto:path=/hiiqos-63/ 2023-07-29T00:52:47.282 DEBUG:boto:auth_path=/hiiqos-63/ 2023-07-29T00:52:47.283 DEBUG:boto:Method: HEAD 2023-07-29T00:52:47.284 DEBUG:boto:Path: /hiiqos-63/ 2023-07-29T00:52:47.284 DEBUG:boto:Data: 2023-07-29T00:52:47.285 DEBUG:boto:Headers: {} 2023-07-29T00:52:47.285 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:47.286 DEBUG:boto:Port: 8000 2023-07-29T00:52:47.287 DEBUG:boto:Params: {} 2023-07-29T00:52:47.287 DEBUG:boto:Token: None 2023-07-29T00:52:47.288 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:47 GMT /hiiqos-63/ 2023-07-29T00:52:47.289 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY= 2023-07-29T00:52:47.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY=', 'Content-Length': '0'} 2023-07-29T00:52:47.292 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c3fbc6e47d91c511-0064c462df-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:47.293 DEBUG:boto:path=/hiiqos-63/obj44 2023-07-29T00:52:47.294 DEBUG:boto:auth_path=/hiiqos-63/obj44 2023-07-29T00:52:47.294 DEBUG:boto:Method: PUT 2023-07-29T00:52:47.295 DEBUG:boto:Path: /hiiqos-63/obj44 2023-07-29T00:52:47.296 DEBUG:boto:Data: 2023-07-29T00:52:47.296 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:52:47.297 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:47.298 DEBUG:boto:Port: 8000 2023-07-29T00:52:47.298 DEBUG:boto:Params: {} 2023-07-29T00:52:47.299 DEBUG:boto:Token: None 2023-07-29T00:52:47.300 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:52:47 GMT /hiiqos-63/obj44 2023-07-29T00:52:47.300 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tXOJnsJsnCx3+rbpzL96dUT+gjk= 2023-07-29T00:52:47.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tXOJnsJsnCx3+rbpzL96dUT+gjk='} 2023-07-29T00:52:47.307 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ffa9cef4cc010a33-0064c462df-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:47.307 DEBUG:boto:path=/hiiqos-63/ 2023-07-29T00:52:47.308 DEBUG:boto:auth_path=/hiiqos-63/ 2023-07-29T00:52:47.309 DEBUG:boto:Method: HEAD 2023-07-29T00:52:47.310 DEBUG:boto:Path: /hiiqos-63/ 2023-07-29T00:52:47.310 DEBUG:boto:Data: 2023-07-29T00:52:47.311 DEBUG:boto:Headers: {} 2023-07-29T00:52:47.312 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:47.313 DEBUG:boto:Port: 8000 2023-07-29T00:52:47.314 DEBUG:boto:Params: {} 2023-07-29T00:52:47.314 DEBUG:boto:Token: None 2023-07-29T00:52:47.315 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:47 GMT /hiiqos-63/ 2023-07-29T00:52:47.315 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY= 2023-07-29T00:52:47.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY=', 'Content-Length': '0'} 2023-07-29T00:52:47.317 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003de7e3620e63e4c5-0064c462df-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:47.318 DEBUG:boto:path=/hiiqos-63/obj55 2023-07-29T00:52:47.319 DEBUG:boto:auth_path=/hiiqos-63/obj55 2023-07-29T00:52:47.319 DEBUG:boto:Method: PUT 2023-07-29T00:52:47.319 DEBUG:boto:Path: /hiiqos-63/obj55 2023-07-29T00:52:47.320 DEBUG:boto:Data: 2023-07-29T00:52:47.320 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:52:47.320 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:47.321 DEBUG:boto:Port: 8000 2023-07-29T00:52:47.321 DEBUG:boto:Params: {} 2023-07-29T00:52:47.322 DEBUG:boto:Token: None 2023-07-29T00:52:47.322 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:52:47 GMT /hiiqos-63/obj55 2023-07-29T00:52:47.323 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ixOggHqPmbOUN8Umfqmg1YBu1IA= 2023-07-29T00:52:47.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ixOggHqPmbOUN8Umfqmg1YBu1IA='} 2023-07-29T00:52:47.368 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000027b42e07ada3d0b0-0064c462df-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:47.369 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:52:47.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:52:47.371 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-63 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:52:47.422 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:52:47.422 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:52:47.466 INFO:teuthology.orchestra.run.smithi101.stdout:tenant: 2023-07-29T00:52:47.467 INFO:teuthology.orchestra.run.smithi101.stdout:bucket name: hiiqos-63 2023-07-29T00:52:47.469 INFO:teuthology.orchestra.run.smithi101.stdout:total entries: 5 2023-07-29T00:52:47.505 DEBUG:boto:path=/hiiqos-63/ 2023-07-29T00:52:47.506 DEBUG:boto:auth_path=/hiiqos-63/ 2023-07-29T00:52:47.507 DEBUG:boto:Method: HEAD 2023-07-29T00:52:47.507 DEBUG:boto:Path: /hiiqos-63/ 2023-07-29T00:52:47.508 DEBUG:boto:Data: 2023-07-29T00:52:47.508 DEBUG:boto:Headers: {} 2023-07-29T00:52:47.509 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:47.509 DEBUG:boto:Port: 8000 2023-07-29T00:52:47.510 DEBUG:boto:Params: {} 2023-07-29T00:52:47.511 DEBUG:boto:Token: None 2023-07-29T00:52:47.511 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:47 GMT /hiiqos-63/ 2023-07-29T00:52:47.512 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY= 2023-07-29T00:52:47.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY=', 'Content-Length': '0'} 2023-07-29T00:52:47.515 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000090fc102d652655e9-0064c462df-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:47.516 DEBUG:boto:path=/hiiqos-63/obj17 2023-07-29T00:52:47.516 DEBUG:boto:auth_path=/hiiqos-63/obj17 2023-07-29T00:52:47.517 DEBUG:boto:Method: PUT 2023-07-29T00:52:47.517 DEBUG:boto:Path: /hiiqos-63/obj17 2023-07-29T00:52:47.518 DEBUG:boto:Data: 2023-07-29T00:52:47.518 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:52:47.518 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:47.519 DEBUG:boto:Port: 8000 2023-07-29T00:52:47.519 DEBUG:boto:Params: {} 2023-07-29T00:52:47.519 DEBUG:boto:Token: None 2023-07-29T00:52:47.520 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:52:47 GMT /hiiqos-63/obj17 2023-07-29T00:52:47.520 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w9onP5BT0/8a6OKT7KxD0mPWkOE= 2023-07-29T00:52:47.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w9onP5BT0/8a6OKT7KxD0mPWkOE='} 2023-07-29T00:52:47.526 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004692f5d49a481be9-0064c462df-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:47.527 DEBUG:boto:path=/hiiqos-63/ 2023-07-29T00:52:47.528 DEBUG:boto:auth_path=/hiiqos-63/ 2023-07-29T00:52:47.528 DEBUG:boto:Method: HEAD 2023-07-29T00:52:47.528 DEBUG:boto:Path: /hiiqos-63/ 2023-07-29T00:52:47.529 DEBUG:boto:Data: 2023-07-29T00:52:47.529 DEBUG:boto:Headers: {} 2023-07-29T00:52:47.529 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:47.530 DEBUG:boto:Port: 8000 2023-07-29T00:52:47.530 DEBUG:boto:Params: {} 2023-07-29T00:52:47.531 DEBUG:boto:Token: None 2023-07-29T00:52:47.531 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:47 GMT /hiiqos-63/ 2023-07-29T00:52:47.531 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY= 2023-07-29T00:52:47.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY=', 'Content-Length': '0'} 2023-07-29T00:52:47.534 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001f49511e0fb4700f-0064c462df-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:47.535 DEBUG:boto:path=/hiiqos-63/obj34 2023-07-29T00:52:47.535 DEBUG:boto:auth_path=/hiiqos-63/obj34 2023-07-29T00:52:47.536 DEBUG:boto:Method: PUT 2023-07-29T00:52:47.536 DEBUG:boto:Path: /hiiqos-63/obj34 2023-07-29T00:52:47.537 DEBUG:boto:Data: 2023-07-29T00:52:47.537 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:52:47.537 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:47.538 DEBUG:boto:Port: 8000 2023-07-29T00:52:47.538 DEBUG:boto:Params: {} 2023-07-29T00:52:47.538 DEBUG:boto:Token: None 2023-07-29T00:52:47.539 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:52:47 GMT /hiiqos-63/obj34 2023-07-29T00:52:47.539 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Fn+vAlFC4/le7qoDTxucI2ljQTg= 2023-07-29T00:52:47.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Fn+vAlFC4/le7qoDTxucI2ljQTg='} 2023-07-29T00:52:47.545 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001bb2c888fe4c70a9-0064c462df-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:47.546 DEBUG:boto:path=/hiiqos-63/ 2023-07-29T00:52:47.546 DEBUG:boto:auth_path=/hiiqos-63/ 2023-07-29T00:52:47.546 DEBUG:boto:Method: HEAD 2023-07-29T00:52:47.547 DEBUG:boto:Path: /hiiqos-63/ 2023-07-29T00:52:47.547 DEBUG:boto:Data: 2023-07-29T00:52:47.547 DEBUG:boto:Headers: {} 2023-07-29T00:52:47.548 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:47.548 DEBUG:boto:Port: 8000 2023-07-29T00:52:47.548 DEBUG:boto:Params: {} 2023-07-29T00:52:47.549 DEBUG:boto:Token: None 2023-07-29T00:52:47.549 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:47 GMT /hiiqos-63/ 2023-07-29T00:52:47.549 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY= 2023-07-29T00:52:47.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY=', 'Content-Length': '0'} 2023-07-29T00:52:47.552 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008322bbe0ae308272-0064c462df-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:47.553 DEBUG:boto:path=/hiiqos-63/obj51 2023-07-29T00:52:47.553 DEBUG:boto:auth_path=/hiiqos-63/obj51 2023-07-29T00:52:47.553 DEBUG:boto:Method: PUT 2023-07-29T00:52:47.554 DEBUG:boto:Path: /hiiqos-63/obj51 2023-07-29T00:52:47.554 DEBUG:boto:Data: 2023-07-29T00:52:47.554 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:52:47.555 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:47.555 DEBUG:boto:Port: 8000 2023-07-29T00:52:47.555 DEBUG:boto:Params: {} 2023-07-29T00:52:47.556 DEBUG:boto:Token: None 2023-07-29T00:52:47.556 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:52:47 GMT /hiiqos-63/obj51 2023-07-29T00:52:47.557 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dG0Xb9tlpNUkuZyaj/LvCKOiCCo= 2023-07-29T00:52:47.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dG0Xb9tlpNUkuZyaj/LvCKOiCCo='} 2023-07-29T00:52:47.562 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000916ad571b41d2802-0064c462df-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:47.562 DEBUG:boto:path=/hiiqos-63/ 2023-07-29T00:52:47.563 DEBUG:boto:auth_path=/hiiqos-63/ 2023-07-29T00:52:47.563 DEBUG:boto:Method: HEAD 2023-07-29T00:52:47.563 DEBUG:boto:Path: /hiiqos-63/ 2023-07-29T00:52:47.564 DEBUG:boto:Data: 2023-07-29T00:52:47.564 DEBUG:boto:Headers: {} 2023-07-29T00:52:47.564 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:47.565 DEBUG:boto:Port: 8000 2023-07-29T00:52:47.565 DEBUG:boto:Params: {} 2023-07-29T00:52:47.565 DEBUG:boto:Token: None 2023-07-29T00:52:47.566 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:47 GMT /hiiqos-63/ 2023-07-29T00:52:47.566 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY= 2023-07-29T00:52:47.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY=', 'Content-Length': '0'} 2023-07-29T00:52:47.569 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f6a8f6d5c32aab01-0064c462df-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:47.569 DEBUG:boto:path=/hiiqos-63/obj68 2023-07-29T00:52:47.570 DEBUG:boto:auth_path=/hiiqos-63/obj68 2023-07-29T00:52:47.570 DEBUG:boto:Method: PUT 2023-07-29T00:52:47.571 DEBUG:boto:Path: /hiiqos-63/obj68 2023-07-29T00:52:47.571 DEBUG:boto:Data: 2023-07-29T00:52:47.571 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:52:47.572 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:47.572 DEBUG:boto:Port: 8000 2023-07-29T00:52:47.572 DEBUG:boto:Params: {} 2023-07-29T00:52:47.573 DEBUG:boto:Token: None 2023-07-29T00:52:47.573 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:52:47 GMT /hiiqos-63/obj68 2023-07-29T00:52:47.573 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qLFVOy6EgHyK1mThOaFQulzd7RI= 2023-07-29T00:52:47.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qLFVOy6EgHyK1mThOaFQulzd7RI='} 2023-07-29T00:52:47.616 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a3385110ffc5d686-0064c462df-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:47.617 DEBUG:boto:path=/hiiqos-63/ 2023-07-29T00:52:47.617 DEBUG:boto:auth_path=/hiiqos-63/ 2023-07-29T00:52:47.618 DEBUG:boto:Method: HEAD 2023-07-29T00:52:47.618 DEBUG:boto:Path: /hiiqos-63/ 2023-07-29T00:52:47.618 DEBUG:boto:Data: 2023-07-29T00:52:47.619 DEBUG:boto:Headers: {} 2023-07-29T00:52:47.619 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:47.620 DEBUG:boto:Port: 8000 2023-07-29T00:52:47.620 DEBUG:boto:Params: {} 2023-07-29T00:52:47.621 DEBUG:boto:Token: None 2023-07-29T00:52:47.621 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:47 GMT /hiiqos-63/ 2023-07-29T00:52:47.622 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY= 2023-07-29T00:52:47.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY=', 'Content-Length': '0'} 2023-07-29T00:52:47.625 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f5bc3e8d5c704b99-0064c462df-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:47.625 DEBUG:boto:path=/hiiqos-63/obj85 2023-07-29T00:52:47.626 DEBUG:boto:auth_path=/hiiqos-63/obj85 2023-07-29T00:52:47.626 DEBUG:boto:Method: PUT 2023-07-29T00:52:47.626 DEBUG:boto:Path: /hiiqos-63/obj85 2023-07-29T00:52:47.627 DEBUG:boto:Data: 2023-07-29T00:52:47.627 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:52:47.627 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:47.628 DEBUG:boto:Port: 8000 2023-07-29T00:52:47.628 DEBUG:boto:Params: {} 2023-07-29T00:52:47.628 DEBUG:boto:Token: None 2023-07-29T00:52:47.629 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:52:47 GMT /hiiqos-63/obj85 2023-07-29T00:52:47.629 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:3lDEpkPpuKph0n7R7e2qIytU7OU= 2023-07-29T00:52:47.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3lDEpkPpuKph0n7R7e2qIytU7OU='} 2023-07-29T00:52:47.634 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000e199bdaae282cc5-0064c462df-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:47.635 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:52:47.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:52:47.636 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-63 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:52:47.685 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:52:47.685 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:52:47.727 INFO:teuthology.orchestra.run.smithi101.stdout:tenant: 2023-07-29T00:52:47.728 INFO:teuthology.orchestra.run.smithi101.stdout:bucket name: hiiqos-63 2023-07-29T00:52:47.735 INFO:teuthology.orchestra.run.smithi101.stdout:total entries: 10 2023-07-29T00:52:47.784 DEBUG:boto:path=/hiiqos-63/ 2023-07-29T00:52:47.785 DEBUG:boto:auth_path=/hiiqos-63/ 2023-07-29T00:52:47.785 DEBUG:boto:Method: HEAD 2023-07-29T00:52:47.785 DEBUG:boto:Path: /hiiqos-63/ 2023-07-29T00:52:47.786 DEBUG:boto:Data: 2023-07-29T00:52:47.786 DEBUG:boto:Headers: {} 2023-07-29T00:52:47.786 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:47.787 DEBUG:boto:Port: 8000 2023-07-29T00:52:47.787 DEBUG:boto:Params: {} 2023-07-29T00:52:47.787 DEBUG:boto:Token: None 2023-07-29T00:52:47.788 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:47 GMT /hiiqos-63/ 2023-07-29T00:52:47.788 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY= 2023-07-29T00:52:47.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY=', 'Content-Length': '0'} 2023-07-29T00:52:47.795 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000abd4604958ac2f30-0064c462df-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:47.796 DEBUG:boto:path=/hiiqos-63/obj19 2023-07-29T00:52:47.796 DEBUG:boto:auth_path=/hiiqos-63/obj19 2023-07-29T00:52:47.796 DEBUG:boto:Method: PUT 2023-07-29T00:52:47.797 DEBUG:boto:Path: /hiiqos-63/obj19 2023-07-29T00:52:47.797 DEBUG:boto:Data: 2023-07-29T00:52:47.797 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:52:47.798 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:47.798 DEBUG:boto:Port: 8000 2023-07-29T00:52:47.798 DEBUG:boto:Params: {} 2023-07-29T00:52:47.799 DEBUG:boto:Token: None 2023-07-29T00:52:47.799 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:52:47 GMT /hiiqos-63/obj19 2023-07-29T00:52:47.800 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:DJ5M1Pei2AYkBC7+f1VLKgXPiKE= 2023-07-29T00:52:47.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DJ5M1Pei2AYkBC7+f1VLKgXPiKE='} 2023-07-29T00:52:47.844 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000038f60d6cdfff663b-0064c462df-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:47.845 DEBUG:boto:path=/hiiqos-63/ 2023-07-29T00:52:47.845 DEBUG:boto:auth_path=/hiiqos-63/ 2023-07-29T00:52:47.846 DEBUG:boto:Method: HEAD 2023-07-29T00:52:47.846 DEBUG:boto:Path: /hiiqos-63/ 2023-07-29T00:52:47.846 DEBUG:boto:Data: 2023-07-29T00:52:47.847 DEBUG:boto:Headers: {} 2023-07-29T00:52:47.847 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:47.847 DEBUG:boto:Port: 8000 2023-07-29T00:52:47.848 DEBUG:boto:Params: {} 2023-07-29T00:52:47.848 DEBUG:boto:Token: None 2023-07-29T00:52:47.849 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:47 GMT /hiiqos-63/ 2023-07-29T00:52:47.849 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY= 2023-07-29T00:52:47.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY=', 'Content-Length': '0'} 2023-07-29T00:52:47.854 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000285d5d49f1ba8217-0064c462df-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:47.855 DEBUG:boto:path=/hiiqos-63/obj38 2023-07-29T00:52:47.855 DEBUG:boto:auth_path=/hiiqos-63/obj38 2023-07-29T00:52:47.855 DEBUG:boto:Method: PUT 2023-07-29T00:52:47.856 DEBUG:boto:Path: /hiiqos-63/obj38 2023-07-29T00:52:47.856 DEBUG:boto:Data: 2023-07-29T00:52:47.856 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:52:47.857 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:47.857 DEBUG:boto:Port: 8000 2023-07-29T00:52:47.857 DEBUG:boto:Params: {} 2023-07-29T00:52:47.858 DEBUG:boto:Token: None 2023-07-29T00:52:47.858 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:52:47 GMT /hiiqos-63/obj38 2023-07-29T00:52:47.859 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lw1WXMcRdCaZFVWLYo9/80TRVAw= 2023-07-29T00:52:47.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lw1WXMcRdCaZFVWLYo9/80TRVAw='} 2023-07-29T00:52:47.863 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000a3ebed091c825ed-0064c462df-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:47.864 DEBUG:boto:path=/hiiqos-63/ 2023-07-29T00:52:47.864 DEBUG:boto:auth_path=/hiiqos-63/ 2023-07-29T00:52:47.865 DEBUG:boto:Method: HEAD 2023-07-29T00:52:47.865 DEBUG:boto:Path: /hiiqos-63/ 2023-07-29T00:52:47.865 DEBUG:boto:Data: 2023-07-29T00:52:47.866 DEBUG:boto:Headers: {} 2023-07-29T00:52:47.866 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:47.866 DEBUG:boto:Port: 8000 2023-07-29T00:52:47.867 DEBUG:boto:Params: {} 2023-07-29T00:52:47.867 DEBUG:boto:Token: None 2023-07-29T00:52:47.867 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:47 GMT /hiiqos-63/ 2023-07-29T00:52:47.868 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY= 2023-07-29T00:52:47.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY=', 'Content-Length': '0'} 2023-07-29T00:52:47.871 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001880060618172a83-0064c462df-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:47.872 DEBUG:boto:path=/hiiqos-63/obj57 2023-07-29T00:52:47.873 DEBUG:boto:auth_path=/hiiqos-63/obj57 2023-07-29T00:52:47.874 DEBUG:boto:Method: PUT 2023-07-29T00:52:47.875 DEBUG:boto:Path: /hiiqos-63/obj57 2023-07-29T00:52:47.876 DEBUG:boto:Data: 2023-07-29T00:52:47.876 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:52:47.877 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:47.877 DEBUG:boto:Port: 8000 2023-07-29T00:52:47.878 DEBUG:boto:Params: {} 2023-07-29T00:52:47.878 DEBUG:boto:Token: None 2023-07-29T00:52:47.879 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:52:47 GMT /hiiqos-63/obj57 2023-07-29T00:52:47.879 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gFreMUu8SggIj39pFnO+jwASbZY= 2023-07-29T00:52:47.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gFreMUu8SggIj39pFnO+jwASbZY='} 2023-07-29T00:52:47.924 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005c8f810930848ec9-0064c462df-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:47.925 DEBUG:boto:path=/hiiqos-63/ 2023-07-29T00:52:47.926 DEBUG:boto:auth_path=/hiiqos-63/ 2023-07-29T00:52:47.926 DEBUG:boto:Method: HEAD 2023-07-29T00:52:47.926 DEBUG:boto:Path: /hiiqos-63/ 2023-07-29T00:52:47.927 DEBUG:boto:Data: 2023-07-29T00:52:47.927 DEBUG:boto:Headers: {} 2023-07-29T00:52:47.927 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:47.927 DEBUG:boto:Port: 8000 2023-07-29T00:52:47.928 DEBUG:boto:Params: {} 2023-07-29T00:52:47.928 DEBUG:boto:Token: None 2023-07-29T00:52:47.929 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:47 GMT /hiiqos-63/ 2023-07-29T00:52:47.929 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY= 2023-07-29T00:52:47.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY=', 'Content-Length': '0'} 2023-07-29T00:52:47.933 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000b94a9266b87e628-0064c462df-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:47.934 DEBUG:boto:path=/hiiqos-63/obj76 2023-07-29T00:52:47.935 DEBUG:boto:auth_path=/hiiqos-63/obj76 2023-07-29T00:52:47.936 DEBUG:boto:Method: PUT 2023-07-29T00:52:47.936 DEBUG:boto:Path: /hiiqos-63/obj76 2023-07-29T00:52:47.937 DEBUG:boto:Data: 2023-07-29T00:52:47.938 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:52:47.938 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:47.939 DEBUG:boto:Port: 8000 2023-07-29T00:52:47.940 DEBUG:boto:Params: {} 2023-07-29T00:52:47.940 DEBUG:boto:Token: None 2023-07-29T00:52:47.941 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:52:47 GMT /hiiqos-63/obj76 2023-07-29T00:52:47.941 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UpOGZu55jT4r3YsXtLnCvzihgJI= 2023-07-29T00:52:47.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UpOGZu55jT4r3YsXtLnCvzihgJI='} 2023-07-29T00:52:47.947 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000093f0b411efbba67e-0064c462df-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:47.948 DEBUG:boto:path=/hiiqos-63/ 2023-07-29T00:52:47.949 DEBUG:boto:auth_path=/hiiqos-63/ 2023-07-29T00:52:47.950 DEBUG:boto:Method: HEAD 2023-07-29T00:52:47.950 DEBUG:boto:Path: /hiiqos-63/ 2023-07-29T00:52:47.951 DEBUG:boto:Data: 2023-07-29T00:52:47.951 DEBUG:boto:Headers: {} 2023-07-29T00:52:47.952 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:47.953 DEBUG:boto:Port: 8000 2023-07-29T00:52:47.953 DEBUG:boto:Params: {} 2023-07-29T00:52:47.954 DEBUG:boto:Token: None 2023-07-29T00:52:47.954 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:47 GMT /hiiqos-63/ 2023-07-29T00:52:47.955 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY= 2023-07-29T00:52:47.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY=', 'Content-Length': '0'} 2023-07-29T00:52:47.957 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003f118db536979af4-0064c462df-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:47.958 DEBUG:boto:path=/hiiqos-63/obj95 2023-07-29T00:52:47.958 DEBUG:boto:auth_path=/hiiqos-63/obj95 2023-07-29T00:52:47.959 DEBUG:boto:Method: PUT 2023-07-29T00:52:47.959 DEBUG:boto:Path: /hiiqos-63/obj95 2023-07-29T00:52:47.959 DEBUG:boto:Data: 2023-07-29T00:52:47.960 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:52:47.960 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:47.961 DEBUG:boto:Port: 8000 2023-07-29T00:52:47.961 DEBUG:boto:Params: {} 2023-07-29T00:52:47.961 DEBUG:boto:Token: None 2023-07-29T00:52:47.962 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:52:47 GMT /hiiqos-63/obj95 2023-07-29T00:52:47.962 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0izX6oNbpKgxXJc0CJ1Urj7Nol4= 2023-07-29T00:52:47.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0izX6oNbpKgxXJc0CJ1Urj7Nol4='} 2023-07-29T00:52:48.004 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000092b9064117b173d4-0064c462df-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:48.005 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:52:48.006 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:52:48.006 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-63 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:52:48.055 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:52:48.055 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:52:48.098 INFO:teuthology.orchestra.run.smithi101.stdout:tenant: 2023-07-29T00:52:48.098 INFO:teuthology.orchestra.run.smithi101.stdout:bucket name: hiiqos-63 2023-07-29T00:52:48.104 INFO:teuthology.orchestra.run.smithi101.stdout:total entries: 15 2023-07-29T00:52:48.149 DEBUG:boto:path=/hiiqos-63/ 2023-07-29T00:52:48.149 DEBUG:boto:auth_path=/hiiqos-63/ 2023-07-29T00:52:48.150 DEBUG:boto:Method: HEAD 2023-07-29T00:52:48.150 DEBUG:boto:Path: /hiiqos-63/ 2023-07-29T00:52:48.150 DEBUG:boto:Data: 2023-07-29T00:52:48.150 DEBUG:boto:Headers: {} 2023-07-29T00:52:48.151 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:48.151 DEBUG:boto:Port: 8000 2023-07-29T00:52:48.152 DEBUG:boto:Params: {} 2023-07-29T00:52:48.153 DEBUG:boto:Token: None 2023-07-29T00:52:48.153 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:48 GMT /hiiqos-63/ 2023-07-29T00:52:48.154 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw= 2023-07-29T00:52:48.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw=', 'Content-Length': '0'} 2023-07-29T00:52:48.157 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083ba17f2ce0e5641-0064c462e0-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:48.157 DEBUG:boto:path=/hiiqos-63/obj23 2023-07-29T00:52:48.158 DEBUG:boto:auth_path=/hiiqos-63/obj23 2023-07-29T00:52:48.158 DEBUG:boto:Method: PUT 2023-07-29T00:52:48.159 DEBUG:boto:Path: /hiiqos-63/obj23 2023-07-29T00:52:48.159 DEBUG:boto:Data: 2023-07-29T00:52:48.160 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:52:48.161 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:48.161 DEBUG:boto:Port: 8000 2023-07-29T00:52:48.162 DEBUG:boto:Params: {} 2023-07-29T00:52:48.163 DEBUG:boto:Token: None 2023-07-29T00:52:48.164 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:52:48 GMT /hiiqos-63/obj23 2023-07-29T00:52:48.164 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bMc5wXYR3mpXSs5wiZpy8ov9xrI= 2023-07-29T00:52:48.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bMc5wXYR3mpXSs5wiZpy8ov9xrI='} 2023-07-29T00:52:48.171 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004ee69e1fd1956cc7-0064c462e0-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:48.172 DEBUG:boto:path=/hiiqos-63/ 2023-07-29T00:52:48.172 DEBUG:boto:auth_path=/hiiqos-63/ 2023-07-29T00:52:48.172 DEBUG:boto:Method: HEAD 2023-07-29T00:52:48.173 DEBUG:boto:Path: /hiiqos-63/ 2023-07-29T00:52:48.173 DEBUG:boto:Data: 2023-07-29T00:52:48.174 DEBUG:boto:Headers: {} 2023-07-29T00:52:48.174 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:48.174 DEBUG:boto:Port: 8000 2023-07-29T00:52:48.175 DEBUG:boto:Params: {} 2023-07-29T00:52:48.175 DEBUG:boto:Token: None 2023-07-29T00:52:48.176 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:48 GMT /hiiqos-63/ 2023-07-29T00:52:48.176 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw= 2023-07-29T00:52:48.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw=', 'Content-Length': '0'} 2023-07-29T00:52:48.179 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a0a5995aad5a55a1-0064c462e0-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:48.180 DEBUG:boto:path=/hiiqos-63/obj46 2023-07-29T00:52:48.180 DEBUG:boto:auth_path=/hiiqos-63/obj46 2023-07-29T00:52:48.181 DEBUG:boto:Method: PUT 2023-07-29T00:52:48.181 DEBUG:boto:Path: /hiiqos-63/obj46 2023-07-29T00:52:48.182 DEBUG:boto:Data: 2023-07-29T00:52:48.182 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:52:48.182 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:48.183 DEBUG:boto:Port: 8000 2023-07-29T00:52:48.183 DEBUG:boto:Params: {} 2023-07-29T00:52:48.184 DEBUG:boto:Token: None 2023-07-29T00:52:48.184 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:52:48 GMT /hiiqos-63/obj46 2023-07-29T00:52:48.185 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XZWj4zmWj3v5ypaUtxK6ZhZzLco= 2023-07-29T00:52:48.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XZWj4zmWj3v5ypaUtxK6ZhZzLco='} 2023-07-29T00:52:48.190 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000083f365ae3165c12f-0064c462e0-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:48.190 DEBUG:boto:path=/hiiqos-63/ 2023-07-29T00:52:48.191 DEBUG:boto:auth_path=/hiiqos-63/ 2023-07-29T00:52:48.191 DEBUG:boto:Method: HEAD 2023-07-29T00:52:48.191 DEBUG:boto:Path: /hiiqos-63/ 2023-07-29T00:52:48.192 DEBUG:boto:Data: 2023-07-29T00:52:48.192 DEBUG:boto:Headers: {} 2023-07-29T00:52:48.193 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:48.193 DEBUG:boto:Port: 8000 2023-07-29T00:52:48.194 DEBUG:boto:Params: {} 2023-07-29T00:52:48.194 DEBUG:boto:Token: None 2023-07-29T00:52:48.195 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:48 GMT /hiiqos-63/ 2023-07-29T00:52:48.195 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw= 2023-07-29T00:52:48.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw=', 'Content-Length': '0'} 2023-07-29T00:52:48.198 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000005fccb74972fb4f7-0064c462e0-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:48.199 DEBUG:boto:path=/hiiqos-63/obj69 2023-07-29T00:52:48.199 DEBUG:boto:auth_path=/hiiqos-63/obj69 2023-07-29T00:52:48.200 DEBUG:boto:Method: PUT 2023-07-29T00:52:48.200 DEBUG:boto:Path: /hiiqos-63/obj69 2023-07-29T00:52:48.200 DEBUG:boto:Data: 2023-07-29T00:52:48.201 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:52:48.201 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:48.201 DEBUG:boto:Port: 8000 2023-07-29T00:52:48.202 DEBUG:boto:Params: {} 2023-07-29T00:52:48.202 DEBUG:boto:Token: None 2023-07-29T00:52:48.203 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:52:48 GMT /hiiqos-63/obj69 2023-07-29T00:52:48.204 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:x8MkGqGGcQcJmiEuzRYlMdBCW6Q= 2023-07-29T00:52:48.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:x8MkGqGGcQcJmiEuzRYlMdBCW6Q='} 2023-07-29T00:52:48.249 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007c73792c10649443-0064c462e0-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:48.249 DEBUG:boto:path=/hiiqos-63/ 2023-07-29T00:52:48.250 DEBUG:boto:auth_path=/hiiqos-63/ 2023-07-29T00:52:48.250 DEBUG:boto:Method: HEAD 2023-07-29T00:52:48.251 DEBUG:boto:Path: /hiiqos-63/ 2023-07-29T00:52:48.251 DEBUG:boto:Data: 2023-07-29T00:52:48.251 DEBUG:boto:Headers: {} 2023-07-29T00:52:48.251 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:48.252 DEBUG:boto:Port: 8000 2023-07-29T00:52:48.252 DEBUG:boto:Params: {} 2023-07-29T00:52:48.253 DEBUG:boto:Token: None 2023-07-29T00:52:48.253 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:48 GMT /hiiqos-63/ 2023-07-29T00:52:48.253 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw= 2023-07-29T00:52:48.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw=', 'Content-Length': '0'} 2023-07-29T00:52:48.256 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af128e4a1a2fc028-0064c462e0-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:48.257 DEBUG:boto:path=/hiiqos-63/obj92 2023-07-29T00:52:48.258 DEBUG:boto:auth_path=/hiiqos-63/obj92 2023-07-29T00:52:48.258 DEBUG:boto:Method: PUT 2023-07-29T00:52:48.258 DEBUG:boto:Path: /hiiqos-63/obj92 2023-07-29T00:52:48.259 DEBUG:boto:Data: 2023-07-29T00:52:48.259 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:52:48.259 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:48.260 DEBUG:boto:Port: 8000 2023-07-29T00:52:48.260 DEBUG:boto:Params: {} 2023-07-29T00:52:48.260 DEBUG:boto:Token: None 2023-07-29T00:52:48.261 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:52:48 GMT /hiiqos-63/obj92 2023-07-29T00:52:48.261 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KY2ph1h435YyAZT6MY9ZMCqsbI0= 2023-07-29T00:52:48.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KY2ph1h435YyAZT6MY9ZMCqsbI0='} 2023-07-29T00:52:48.266 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b874c8e1148ffe6e-0064c462e0-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:48.267 DEBUG:boto:path=/hiiqos-63/ 2023-07-29T00:52:48.267 DEBUG:boto:auth_path=/hiiqos-63/ 2023-07-29T00:52:48.268 DEBUG:boto:Method: HEAD 2023-07-29T00:52:48.268 DEBUG:boto:Path: /hiiqos-63/ 2023-07-29T00:52:48.268 DEBUG:boto:Data: 2023-07-29T00:52:48.269 DEBUG:boto:Headers: {} 2023-07-29T00:52:48.269 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:48.269 DEBUG:boto:Port: 8000 2023-07-29T00:52:48.270 DEBUG:boto:Params: {} 2023-07-29T00:52:48.270 DEBUG:boto:Token: None 2023-07-29T00:52:48.270 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:48 GMT /hiiqos-63/ 2023-07-29T00:52:48.271 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw= 2023-07-29T00:52:48.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw=', 'Content-Length': '0'} 2023-07-29T00:52:48.274 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009730670278b674d8-0064c462e0-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:48.275 DEBUG:boto:path=/hiiqos-63/obj115 2023-07-29T00:52:48.275 DEBUG:boto:auth_path=/hiiqos-63/obj115 2023-07-29T00:52:48.276 DEBUG:boto:Method: PUT 2023-07-29T00:52:48.276 DEBUG:boto:Path: /hiiqos-63/obj115 2023-07-29T00:52:48.276 DEBUG:boto:Data: 2023-07-29T00:52:48.277 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:52:48.277 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:48.277 DEBUG:boto:Port: 8000 2023-07-29T00:52:48.278 DEBUG:boto:Params: {} 2023-07-29T00:52:48.278 DEBUG:boto:Token: None 2023-07-29T00:52:48.278 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:52:48 GMT /hiiqos-63/obj115 2023-07-29T00:52:48.279 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:m4P6MSsNJ/ttbjbyi2E1eJq0Xx0= 2023-07-29T00:52:48.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:m4P6MSsNJ/ttbjbyi2E1eJq0Xx0='} 2023-07-29T00:52:48.324 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e83f9bf9af3782fb-0064c462e0-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:48.325 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:52:48.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:52:48.327 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-63 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:52:48.376 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:52:48.377 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:52:48.419 INFO:teuthology.orchestra.run.smithi101.stdout:tenant: 2023-07-29T00:52:48.420 INFO:teuthology.orchestra.run.smithi101.stdout:bucket name: hiiqos-63 2023-07-29T00:52:48.427 INFO:teuthology.orchestra.run.smithi101.stdout:total entries: 20 2023-07-29T00:52:48.476 DEBUG:boto:path=/hiiqos-63/ 2023-07-29T00:52:48.477 DEBUG:boto:auth_path=/hiiqos-63/ 2023-07-29T00:52:48.478 DEBUG:boto:Method: HEAD 2023-07-29T00:52:48.478 DEBUG:boto:Path: /hiiqos-63/ 2023-07-29T00:52:48.478 DEBUG:boto:Data: 2023-07-29T00:52:48.479 DEBUG:boto:Headers: {} 2023-07-29T00:52:48.479 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:48.479 DEBUG:boto:Port: 8000 2023-07-29T00:52:48.480 DEBUG:boto:Params: {} 2023-07-29T00:52:48.480 DEBUG:boto:Token: None 2023-07-29T00:52:48.480 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:48 GMT /hiiqos-63/ 2023-07-29T00:52:48.481 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw= 2023-07-29T00:52:48.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw=', 'Content-Length': '0'} 2023-07-29T00:52:48.485 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e5a2d696b4e4c8af-0064c462e0-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:48.486 DEBUG:boto:path=/hiiqos-63/obj29 2023-07-29T00:52:48.486 DEBUG:boto:auth_path=/hiiqos-63/obj29 2023-07-29T00:52:48.486 DEBUG:boto:Method: PUT 2023-07-29T00:52:48.487 DEBUG:boto:Path: /hiiqos-63/obj29 2023-07-29T00:52:48.487 DEBUG:boto:Data: 2023-07-29T00:52:48.488 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:52:48.488 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:48.489 DEBUG:boto:Port: 8000 2023-07-29T00:52:48.489 DEBUG:boto:Params: {} 2023-07-29T00:52:48.490 DEBUG:boto:Token: None 2023-07-29T00:52:48.491 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:52:48 GMT /hiiqos-63/obj29 2023-07-29T00:52:48.493 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7gBp5DQZrl0kOhYUmABrWX8yFDM= 2023-07-29T00:52:48.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7gBp5DQZrl0kOhYUmABrWX8yFDM='} 2023-07-29T00:52:48.499 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000054cd117ff33cec94-0064c462e0-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:48.500 DEBUG:boto:path=/hiiqos-63/ 2023-07-29T00:52:48.500 DEBUG:boto:auth_path=/hiiqos-63/ 2023-07-29T00:52:48.501 DEBUG:boto:Method: HEAD 2023-07-29T00:52:48.501 DEBUG:boto:Path: /hiiqos-63/ 2023-07-29T00:52:48.501 DEBUG:boto:Data: 2023-07-29T00:52:48.502 DEBUG:boto:Headers: {} 2023-07-29T00:52:48.502 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:48.503 DEBUG:boto:Port: 8000 2023-07-29T00:52:48.503 DEBUG:boto:Params: {} 2023-07-29T00:52:48.504 DEBUG:boto:Token: None 2023-07-29T00:52:48.504 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:48 GMT /hiiqos-63/ 2023-07-29T00:52:48.504 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw= 2023-07-29T00:52:48.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw=', 'Content-Length': '0'} 2023-07-29T00:52:48.508 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000247b801b3e418185-0064c462e0-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:48.509 DEBUG:boto:path=/hiiqos-63/obj58 2023-07-29T00:52:48.509 DEBUG:boto:auth_path=/hiiqos-63/obj58 2023-07-29T00:52:48.509 DEBUG:boto:Method: PUT 2023-07-29T00:52:48.510 DEBUG:boto:Path: /hiiqos-63/obj58 2023-07-29T00:52:48.510 DEBUG:boto:Data: 2023-07-29T00:52:48.510 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:52:48.511 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:48.511 DEBUG:boto:Port: 8000 2023-07-29T00:52:48.511 DEBUG:boto:Params: {} 2023-07-29T00:52:48.512 DEBUG:boto:Token: None 2023-07-29T00:52:48.512 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:52:48 GMT /hiiqos-63/obj58 2023-07-29T00:52:48.513 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+sjnACAhtQv2olv2zLGgLtnjcK4= 2023-07-29T00:52:48.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+sjnACAhtQv2olv2zLGgLtnjcK4='} 2023-07-29T00:52:48.519 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000042b2c974c4c6e92-0064c462e0-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:48.519 DEBUG:boto:path=/hiiqos-63/ 2023-07-29T00:52:48.520 DEBUG:boto:auth_path=/hiiqos-63/ 2023-07-29T00:52:48.520 DEBUG:boto:Method: HEAD 2023-07-29T00:52:48.520 DEBUG:boto:Path: /hiiqos-63/ 2023-07-29T00:52:48.521 DEBUG:boto:Data: 2023-07-29T00:52:48.521 DEBUG:boto:Headers: {} 2023-07-29T00:52:48.522 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:48.522 DEBUG:boto:Port: 8000 2023-07-29T00:52:48.522 DEBUG:boto:Params: {} 2023-07-29T00:52:48.523 DEBUG:boto:Token: None 2023-07-29T00:52:48.523 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:48 GMT /hiiqos-63/ 2023-07-29T00:52:48.524 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw= 2023-07-29T00:52:48.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw=', 'Content-Length': '0'} 2023-07-29T00:52:48.527 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000759e56b2862427fc-0064c462e0-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:48.528 DEBUG:boto:path=/hiiqos-63/obj87 2023-07-29T00:52:48.529 DEBUG:boto:auth_path=/hiiqos-63/obj87 2023-07-29T00:52:48.530 DEBUG:boto:Method: PUT 2023-07-29T00:52:48.530 DEBUG:boto:Path: /hiiqos-63/obj87 2023-07-29T00:52:48.530 DEBUG:boto:Data: 2023-07-29T00:52:48.531 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:52:48.531 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:48.531 DEBUG:boto:Port: 8000 2023-07-29T00:52:48.532 DEBUG:boto:Params: {} 2023-07-29T00:52:48.532 DEBUG:boto:Token: None 2023-07-29T00:52:48.532 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:52:48 GMT /hiiqos-63/obj87 2023-07-29T00:52:48.533 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ddeMguuD2QT+/urwvI7M/eVNTHE= 2023-07-29T00:52:48.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ddeMguuD2QT+/urwvI7M/eVNTHE='} 2023-07-29T00:52:48.576 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005fb6bfa22ee4f53f-0064c462e0-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:48.577 DEBUG:boto:path=/hiiqos-63/ 2023-07-29T00:52:48.578 DEBUG:boto:auth_path=/hiiqos-63/ 2023-07-29T00:52:48.578 DEBUG:boto:Method: HEAD 2023-07-29T00:52:48.579 DEBUG:boto:Path: /hiiqos-63/ 2023-07-29T00:52:48.580 DEBUG:boto:Data: 2023-07-29T00:52:48.581 DEBUG:boto:Headers: {} 2023-07-29T00:52:48.582 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:48.583 DEBUG:boto:Port: 8000 2023-07-29T00:52:48.583 DEBUG:boto:Params: {} 2023-07-29T00:52:48.584 DEBUG:boto:Token: None 2023-07-29T00:52:48.585 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:48 GMT /hiiqos-63/ 2023-07-29T00:52:48.586 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw= 2023-07-29T00:52:48.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw=', 'Content-Length': '0'} 2023-07-29T00:52:48.590 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003a30b46abe67a6ea-0064c462e0-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:48.591 DEBUG:boto:path=/hiiqos-63/obj116 2023-07-29T00:52:48.591 DEBUG:boto:auth_path=/hiiqos-63/obj116 2023-07-29T00:52:48.592 DEBUG:boto:Method: PUT 2023-07-29T00:52:48.592 DEBUG:boto:Path: /hiiqos-63/obj116 2023-07-29T00:52:48.592 DEBUG:boto:Data: 2023-07-29T00:52:48.593 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:52:48.593 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:48.593 DEBUG:boto:Port: 8000 2023-07-29T00:52:48.594 DEBUG:boto:Params: {} 2023-07-29T00:52:48.594 DEBUG:boto:Token: None 2023-07-29T00:52:48.595 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:52:48 GMT /hiiqos-63/obj116 2023-07-29T00:52:48.595 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LjMRX9lxTYaYC3sfBTG0qmY+v6A= 2023-07-29T00:52:48.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LjMRX9lxTYaYC3sfBTG0qmY+v6A='} 2023-07-29T00:52:48.600 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fb5ab7ca9449d5a4-0064c462e0-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:48.601 DEBUG:boto:path=/hiiqos-63/ 2023-07-29T00:52:48.602 DEBUG:boto:auth_path=/hiiqos-63/ 2023-07-29T00:52:48.602 DEBUG:boto:Method: HEAD 2023-07-29T00:52:48.603 DEBUG:boto:Path: /hiiqos-63/ 2023-07-29T00:52:48.603 DEBUG:boto:Data: 2023-07-29T00:52:48.603 DEBUG:boto:Headers: {} 2023-07-29T00:52:48.604 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:48.604 DEBUG:boto:Port: 8000 2023-07-29T00:52:48.604 DEBUG:boto:Params: {} 2023-07-29T00:52:48.605 DEBUG:boto:Token: None 2023-07-29T00:52:48.605 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:48 GMT /hiiqos-63/ 2023-07-29T00:52:48.606 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw= 2023-07-29T00:52:48.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw=', 'Content-Length': '0'} 2023-07-29T00:52:48.609 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e1a63bdd4abc182-0064c462e0-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:48.610 DEBUG:boto:path=/hiiqos-63/obj145 2023-07-29T00:52:48.610 DEBUG:boto:auth_path=/hiiqos-63/obj145 2023-07-29T00:52:48.611 DEBUG:boto:Method: PUT 2023-07-29T00:52:48.611 DEBUG:boto:Path: /hiiqos-63/obj145 2023-07-29T00:52:48.611 DEBUG:boto:Data: 2023-07-29T00:52:48.612 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:52:48.612 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:48.612 DEBUG:boto:Port: 8000 2023-07-29T00:52:48.613 DEBUG:boto:Params: {} 2023-07-29T00:52:48.613 DEBUG:boto:Token: None 2023-07-29T00:52:48.613 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:52:48 GMT /hiiqos-63/obj145 2023-07-29T00:52:48.614 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:uMLiTD6jD1lzKt+mSnbM7lEbSA8= 2023-07-29T00:52:48.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uMLiTD6jD1lzKt+mSnbM7lEbSA8='} 2023-07-29T00:52:48.660 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dc622bc3f0c96533-0064c462e0-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:48.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:52:48.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:52:48.662 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-63 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:52:48.714 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:52:48.714 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:52:48.760 INFO:teuthology.orchestra.run.smithi101.stdout:tenant: 2023-07-29T00:52:48.761 INFO:teuthology.orchestra.run.smithi101.stdout:bucket name: hiiqos-63 2023-07-29T00:52:48.770 INFO:teuthology.orchestra.run.smithi101.stdout:total entries: 25 2023-07-29T00:52:48.825 DEBUG:boto:path=/hiiqos-63/ 2023-07-29T00:52:48.826 DEBUG:boto:auth_path=/hiiqos-63/ 2023-07-29T00:52:48.826 DEBUG:boto:Method: HEAD 2023-07-29T00:52:48.827 DEBUG:boto:Path: /hiiqos-63/ 2023-07-29T00:52:48.827 DEBUG:boto:Data: 2023-07-29T00:52:48.827 DEBUG:boto:Headers: {} 2023-07-29T00:52:48.828 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:48.829 DEBUG:boto:Port: 8000 2023-07-29T00:52:48.829 DEBUG:boto:Params: {} 2023-07-29T00:52:48.829 DEBUG:boto:Token: None 2023-07-29T00:52:48.830 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:48 GMT /hiiqos-63/ 2023-07-29T00:52:48.830 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw= 2023-07-29T00:52:48.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw=', 'Content-Length': '0'} 2023-07-29T00:52:48.834 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000687fed2bab598481-0064c462e0-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:48.835 DEBUG:boto:path=/hiiqos-63/obj31 2023-07-29T00:52:48.835 DEBUG:boto:auth_path=/hiiqos-63/obj31 2023-07-29T00:52:48.835 DEBUG:boto:Method: PUT 2023-07-29T00:52:48.836 DEBUG:boto:Path: /hiiqos-63/obj31 2023-07-29T00:52:48.836 DEBUG:boto:Data: 2023-07-29T00:52:48.837 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:52:48.837 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:48.837 DEBUG:boto:Port: 8000 2023-07-29T00:52:48.838 DEBUG:boto:Params: {} 2023-07-29T00:52:48.838 DEBUG:boto:Token: None 2023-07-29T00:52:48.838 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:52:48 GMT /hiiqos-63/obj31 2023-07-29T00:52:48.839 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:NZ4DnIYN6uNC36fkB55JE33IXKI= 2023-07-29T00:52:48.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:NZ4DnIYN6uNC36fkB55JE33IXKI='} 2023-07-29T00:52:48.844 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000798730b0cf006b10-0064c462e0-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:48.845 DEBUG:boto:path=/hiiqos-63/ 2023-07-29T00:52:48.845 DEBUG:boto:auth_path=/hiiqos-63/ 2023-07-29T00:52:48.845 DEBUG:boto:Method: HEAD 2023-07-29T00:52:48.846 DEBUG:boto:Path: /hiiqos-63/ 2023-07-29T00:52:48.846 DEBUG:boto:Data: 2023-07-29T00:52:48.846 DEBUG:boto:Headers: {} 2023-07-29T00:52:48.847 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:48.847 DEBUG:boto:Port: 8000 2023-07-29T00:52:48.847 DEBUG:boto:Params: {} 2023-07-29T00:52:48.848 DEBUG:boto:Token: None 2023-07-29T00:52:48.848 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:48 GMT /hiiqos-63/ 2023-07-29T00:52:48.849 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw= 2023-07-29T00:52:48.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw=', 'Content-Length': '0'} 2023-07-29T00:52:48.852 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000efd34ce77372798d-0064c462e0-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:48.852 DEBUG:boto:path=/hiiqos-63/obj62 2023-07-29T00:52:48.853 DEBUG:boto:auth_path=/hiiqos-63/obj62 2023-07-29T00:52:48.853 DEBUG:boto:Method: PUT 2023-07-29T00:52:48.853 DEBUG:boto:Path: /hiiqos-63/obj62 2023-07-29T00:52:48.854 DEBUG:boto:Data: 2023-07-29T00:52:48.854 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:52:48.855 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:48.855 DEBUG:boto:Port: 8000 2023-07-29T00:52:48.855 DEBUG:boto:Params: {} 2023-07-29T00:52:48.856 DEBUG:boto:Token: None 2023-07-29T00:52:48.856 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:52:48 GMT /hiiqos-63/obj62 2023-07-29T00:52:48.857 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VVgKWJPhA7HwoabqUUFdudJIDuI= 2023-07-29T00:52:48.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VVgKWJPhA7HwoabqUUFdudJIDuI='} 2023-07-29T00:52:48.862 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bc6180bb5911f71e-0064c462e0-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:48.862 DEBUG:boto:path=/hiiqos-63/ 2023-07-29T00:52:48.863 DEBUG:boto:auth_path=/hiiqos-63/ 2023-07-29T00:52:48.863 DEBUG:boto:Method: HEAD 2023-07-29T00:52:48.863 DEBUG:boto:Path: /hiiqos-63/ 2023-07-29T00:52:48.864 DEBUG:boto:Data: 2023-07-29T00:52:48.864 DEBUG:boto:Headers: {} 2023-07-29T00:52:48.864 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:48.865 DEBUG:boto:Port: 8000 2023-07-29T00:52:48.865 DEBUG:boto:Params: {} 2023-07-29T00:52:48.866 DEBUG:boto:Token: None 2023-07-29T00:52:48.866 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:48 GMT /hiiqos-63/ 2023-07-29T00:52:48.866 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw= 2023-07-29T00:52:48.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw=', 'Content-Length': '0'} 2023-07-29T00:52:48.870 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001eddd0f46e0be57c-0064c462e0-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:48.871 DEBUG:boto:path=/hiiqos-63/obj93 2023-07-29T00:52:48.871 DEBUG:boto:auth_path=/hiiqos-63/obj93 2023-07-29T00:52:48.871 DEBUG:boto:Method: PUT 2023-07-29T00:52:48.872 DEBUG:boto:Path: /hiiqos-63/obj93 2023-07-29T00:52:48.872 DEBUG:boto:Data: 2023-07-29T00:52:48.872 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:52:48.873 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:48.873 DEBUG:boto:Port: 8000 2023-07-29T00:52:48.873 DEBUG:boto:Params: {} 2023-07-29T00:52:48.874 DEBUG:boto:Token: None 2023-07-29T00:52:48.874 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:52:48 GMT /hiiqos-63/obj93 2023-07-29T00:52:48.874 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VHkjJ0U5Gp+fR2fbQcEBCk5TOgE= 2023-07-29T00:52:48.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VHkjJ0U5Gp+fR2fbQcEBCk5TOgE='} 2023-07-29T00:52:48.916 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cb3826b67a0df531-0064c462e0-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:48.917 DEBUG:boto:path=/hiiqos-63/ 2023-07-29T00:52:48.918 DEBUG:boto:auth_path=/hiiqos-63/ 2023-07-29T00:52:48.918 DEBUG:boto:Method: HEAD 2023-07-29T00:52:48.918 DEBUG:boto:Path: /hiiqos-63/ 2023-07-29T00:52:48.919 DEBUG:boto:Data: 2023-07-29T00:52:48.919 DEBUG:boto:Headers: {} 2023-07-29T00:52:48.919 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:48.920 DEBUG:boto:Port: 8000 2023-07-29T00:52:48.920 DEBUG:boto:Params: {} 2023-07-29T00:52:48.920 DEBUG:boto:Token: None 2023-07-29T00:52:48.921 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:48 GMT /hiiqos-63/ 2023-07-29T00:52:48.921 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw= 2023-07-29T00:52:48.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw=', 'Content-Length': '0'} 2023-07-29T00:52:48.924 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000096cd2985138f75ab-0064c462e0-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:48.925 DEBUG:boto:path=/hiiqos-63/obj124 2023-07-29T00:52:48.925 DEBUG:boto:auth_path=/hiiqos-63/obj124 2023-07-29T00:52:48.926 DEBUG:boto:Method: PUT 2023-07-29T00:52:48.926 DEBUG:boto:Path: /hiiqos-63/obj124 2023-07-29T00:52:48.926 DEBUG:boto:Data: 2023-07-29T00:52:48.927 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:52:48.927 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:48.927 DEBUG:boto:Port: 8000 2023-07-29T00:52:48.928 DEBUG:boto:Params: {} 2023-07-29T00:52:48.928 DEBUG:boto:Token: None 2023-07-29T00:52:48.929 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:52:48 GMT /hiiqos-63/obj124 2023-07-29T00:52:48.929 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zvS9w9xgyMF6HpsXxFqZlEYpcZo= 2023-07-29T00:52:48.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zvS9w9xgyMF6HpsXxFqZlEYpcZo='} 2023-07-29T00:52:48.935 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008fe94d3cfbf8e9ca-0064c462e0-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:48.935 DEBUG:boto:path=/hiiqos-63/ 2023-07-29T00:52:48.936 DEBUG:boto:auth_path=/hiiqos-63/ 2023-07-29T00:52:48.936 DEBUG:boto:Method: HEAD 2023-07-29T00:52:48.936 DEBUG:boto:Path: /hiiqos-63/ 2023-07-29T00:52:48.937 DEBUG:boto:Data: 2023-07-29T00:52:48.937 DEBUG:boto:Headers: {} 2023-07-29T00:52:48.937 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:48.938 DEBUG:boto:Port: 8000 2023-07-29T00:52:48.938 DEBUG:boto:Params: {} 2023-07-29T00:52:48.938 DEBUG:boto:Token: None 2023-07-29T00:52:48.939 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:48 GMT /hiiqos-63/ 2023-07-29T00:52:48.939 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw= 2023-07-29T00:52:48.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw=', 'Content-Length': '0'} 2023-07-29T00:52:48.942 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e5f490e81edc5912-0064c462e0-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:48.943 DEBUG:boto:path=/hiiqos-63/obj155 2023-07-29T00:52:48.943 DEBUG:boto:auth_path=/hiiqos-63/obj155 2023-07-29T00:52:48.944 DEBUG:boto:Method: PUT 2023-07-29T00:52:48.944 DEBUG:boto:Path: /hiiqos-63/obj155 2023-07-29T00:52:48.944 DEBUG:boto:Data: 2023-07-29T00:52:48.945 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:52:48.945 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:48.945 DEBUG:boto:Port: 8000 2023-07-29T00:52:48.946 DEBUG:boto:Params: {} 2023-07-29T00:52:48.946 DEBUG:boto:Token: None 2023-07-29T00:52:48.947 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:52:48 GMT /hiiqos-63/obj155 2023-07-29T00:52:48.947 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XZxa61xLFumD8i1TnFtGPH7S3+4= 2023-07-29T00:52:48.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XZxa61xLFumD8i1TnFtGPH7S3+4='} 2023-07-29T00:52:48.996 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005dccba726097e368-0064c462e0-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:48.997 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:52:48.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:52:48.998 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-63 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:52:49.047 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:52:49.048 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:52:49.094 INFO:teuthology.orchestra.run.smithi101.stdout:tenant: 2023-07-29T00:52:49.094 INFO:teuthology.orchestra.run.smithi101.stdout:bucket name: hiiqos-63 2023-07-29T00:52:49.105 INFO:teuthology.orchestra.run.smithi101.stdout:total entries: 30 2023-07-29T00:52:49.162 DEBUG:boto:path=/hiiqos-63/ 2023-07-29T00:52:49.163 DEBUG:boto:auth_path=/hiiqos-63/ 2023-07-29T00:52:49.163 DEBUG:boto:Method: HEAD 2023-07-29T00:52:49.164 DEBUG:boto:Path: /hiiqos-63/ 2023-07-29T00:52:49.164 DEBUG:boto:Data: 2023-07-29T00:52:49.164 DEBUG:boto:Headers: {} 2023-07-29T00:52:49.165 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:49.165 DEBUG:boto:Port: 8000 2023-07-29T00:52:49.165 DEBUG:boto:Params: {} 2023-07-29T00:52:49.166 DEBUG:boto:Token: None 2023-07-29T00:52:49.166 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:49 GMT /hiiqos-63/ 2023-07-29T00:52:49.167 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eMFPRv3AuzfRuxq+oty/0aQF6gY= 2023-07-29T00:52:49.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eMFPRv3AuzfRuxq+oty/0aQF6gY=', 'Content-Length': '0'} 2023-07-29T00:52:49.170 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009d3a70f7dab5fc81-0064c462e1-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:49.171 DEBUG:boto:path=/hiiqos-63/obj37 2023-07-29T00:52:49.171 DEBUG:boto:auth_path=/hiiqos-63/obj37 2023-07-29T00:52:49.172 DEBUG:boto:Method: PUT 2023-07-29T00:52:49.172 DEBUG:boto:Path: /hiiqos-63/obj37 2023-07-29T00:52:49.172 DEBUG:boto:Data: 2023-07-29T00:52:49.173 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:52:49.173 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:49.173 DEBUG:boto:Port: 8000 2023-07-29T00:52:49.174 DEBUG:boto:Params: {} 2023-07-29T00:52:49.174 DEBUG:boto:Token: None 2023-07-29T00:52:49.174 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:52:49 GMT /hiiqos-63/obj37 2023-07-29T00:52:49.175 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cKEouCkG8diC3PJtoU7dOVWPKt0= 2023-07-29T00:52:49.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cKEouCkG8diC3PJtoU7dOVWPKt0='} 2023-07-29T00:52:49.180 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000069c7b146c372069f-0064c462e1-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:49.180 DEBUG:boto:path=/hiiqos-63/ 2023-07-29T00:52:49.181 DEBUG:boto:auth_path=/hiiqos-63/ 2023-07-29T00:52:49.181 DEBUG:boto:Method: HEAD 2023-07-29T00:52:49.181 DEBUG:boto:Path: /hiiqos-63/ 2023-07-29T00:52:49.182 DEBUG:boto:Data: 2023-07-29T00:52:49.182 DEBUG:boto:Headers: {} 2023-07-29T00:52:49.182 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:49.183 DEBUG:boto:Port: 8000 2023-07-29T00:52:49.183 DEBUG:boto:Params: {} 2023-07-29T00:52:49.183 DEBUG:boto:Token: None 2023-07-29T00:52:49.184 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:49 GMT /hiiqos-63/ 2023-07-29T00:52:49.184 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eMFPRv3AuzfRuxq+oty/0aQF6gY= 2023-07-29T00:52:49.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eMFPRv3AuzfRuxq+oty/0aQF6gY=', 'Content-Length': '0'} 2023-07-29T00:52:49.188 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d03baad2b9cd931-0064c462e1-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:49.188 DEBUG:boto:path=/hiiqos-63/obj74 2023-07-29T00:52:49.189 DEBUG:boto:auth_path=/hiiqos-63/obj74 2023-07-29T00:52:49.189 DEBUG:boto:Method: PUT 2023-07-29T00:52:49.189 DEBUG:boto:Path: /hiiqos-63/obj74 2023-07-29T00:52:49.190 DEBUG:boto:Data: 2023-07-29T00:52:49.190 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:52:49.190 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:49.191 DEBUG:boto:Port: 8000 2023-07-29T00:52:49.191 DEBUG:boto:Params: {} 2023-07-29T00:52:49.191 DEBUG:boto:Token: None 2023-07-29T00:52:49.192 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:52:49 GMT /hiiqos-63/obj74 2023-07-29T00:52:49.192 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:f0/VCNxcTcEocfUjNZJOjpSalAc= 2023-07-29T00:52:49.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:f0/VCNxcTcEocfUjNZJOjpSalAc='} 2023-07-29T00:52:49.198 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003b46accb4cdad56c-0064c462e1-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:49.198 DEBUG:boto:path=/hiiqos-63/ 2023-07-29T00:52:49.198 DEBUG:boto:auth_path=/hiiqos-63/ 2023-07-29T00:52:49.199 DEBUG:boto:Method: HEAD 2023-07-29T00:52:49.199 DEBUG:boto:Path: /hiiqos-63/ 2023-07-29T00:52:49.199 DEBUG:boto:Data: 2023-07-29T00:52:49.200 DEBUG:boto:Headers: {} 2023-07-29T00:52:49.200 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:49.200 DEBUG:boto:Port: 8000 2023-07-29T00:52:49.201 DEBUG:boto:Params: {} 2023-07-29T00:52:49.201 DEBUG:boto:Token: None 2023-07-29T00:52:49.201 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:49 GMT /hiiqos-63/ 2023-07-29T00:52:49.202 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eMFPRv3AuzfRuxq+oty/0aQF6gY= 2023-07-29T00:52:49.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eMFPRv3AuzfRuxq+oty/0aQF6gY=', 'Content-Length': '0'} 2023-07-29T00:52:49.205 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c04ebbe360eace5c-0064c462e1-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:49.206 DEBUG:boto:path=/hiiqos-63/obj111 2023-07-29T00:52:49.206 DEBUG:boto:auth_path=/hiiqos-63/obj111 2023-07-29T00:52:49.207 DEBUG:boto:Method: PUT 2023-07-29T00:52:49.207 DEBUG:boto:Path: /hiiqos-63/obj111 2023-07-29T00:52:49.207 DEBUG:boto:Data: 2023-07-29T00:52:49.208 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:52:49.208 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:49.208 DEBUG:boto:Port: 8000 2023-07-29T00:52:49.209 DEBUG:boto:Params: {} 2023-07-29T00:52:49.209 DEBUG:boto:Token: None 2023-07-29T00:52:49.209 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:52:49 GMT /hiiqos-63/obj111 2023-07-29T00:52:49.210 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:crnTV5NRePifXYga8PMz5xtJ7YM= 2023-07-29T00:52:49.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:crnTV5NRePifXYga8PMz5xtJ7YM='} 2023-07-29T00:52:49.252 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e5ad286b7fc06f4c-0064c462e1-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:49.253 DEBUG:boto:path=/hiiqos-63/ 2023-07-29T00:52:49.253 DEBUG:boto:auth_path=/hiiqos-63/ 2023-07-29T00:52:49.254 DEBUG:boto:Method: HEAD 2023-07-29T00:52:49.254 DEBUG:boto:Path: /hiiqos-63/ 2023-07-29T00:52:49.255 DEBUG:boto:Data: 2023-07-29T00:52:49.255 DEBUG:boto:Headers: {} 2023-07-29T00:52:49.256 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:49.256 DEBUG:boto:Port: 8000 2023-07-29T00:52:49.256 DEBUG:boto:Params: {} 2023-07-29T00:52:49.257 DEBUG:boto:Token: None 2023-07-29T00:52:49.257 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:49 GMT /hiiqos-63/ 2023-07-29T00:52:49.258 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eMFPRv3AuzfRuxq+oty/0aQF6gY= 2023-07-29T00:52:49.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eMFPRv3AuzfRuxq+oty/0aQF6gY=', 'Content-Length': '0'} 2023-07-29T00:52:49.261 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000213af5d046d52381-0064c462e1-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:49.262 DEBUG:boto:path=/hiiqos-63/obj148 2023-07-29T00:52:49.262 DEBUG:boto:auth_path=/hiiqos-63/obj148 2023-07-29T00:52:49.263 DEBUG:boto:Method: PUT 2023-07-29T00:52:49.263 DEBUG:boto:Path: /hiiqos-63/obj148 2023-07-29T00:52:49.263 DEBUG:boto:Data: 2023-07-29T00:52:49.264 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:52:49.264 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:49.264 DEBUG:boto:Port: 8000 2023-07-29T00:52:49.265 DEBUG:boto:Params: {} 2023-07-29T00:52:49.265 DEBUG:boto:Token: None 2023-07-29T00:52:49.266 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:52:49 GMT /hiiqos-63/obj148 2023-07-29T00:52:49.266 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:toELDn5wKsxk8j2y48UtOG7Jub8= 2023-07-29T00:52:49.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:toELDn5wKsxk8j2y48UtOG7Jub8='} 2023-07-29T00:52:49.272 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003efad09c8d416848-0064c462e1-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:49.272 DEBUG:boto:path=/hiiqos-63/ 2023-07-29T00:52:49.273 DEBUG:boto:auth_path=/hiiqos-63/ 2023-07-29T00:52:49.273 DEBUG:boto:Method: HEAD 2023-07-29T00:52:49.273 DEBUG:boto:Path: /hiiqos-63/ 2023-07-29T00:52:49.274 DEBUG:boto:Data: 2023-07-29T00:52:49.274 DEBUG:boto:Headers: {} 2023-07-29T00:52:49.274 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:49.275 DEBUG:boto:Port: 8000 2023-07-29T00:52:49.275 DEBUG:boto:Params: {} 2023-07-29T00:52:49.275 DEBUG:boto:Token: None 2023-07-29T00:52:49.276 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:52:49 GMT /hiiqos-63/ 2023-07-29T00:52:49.276 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eMFPRv3AuzfRuxq+oty/0aQF6gY= 2023-07-29T00:52:49.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eMFPRv3AuzfRuxq+oty/0aQF6gY=', 'Content-Length': '0'} 2023-07-29T00:52:49.280 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c25d69c0fc21aca5-0064c462e1-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:49.280 DEBUG:boto:path=/hiiqos-63/obj185 2023-07-29T00:52:49.281 DEBUG:boto:auth_path=/hiiqos-63/obj185 2023-07-29T00:52:49.281 DEBUG:boto:Method: PUT 2023-07-29T00:52:49.281 DEBUG:boto:Path: /hiiqos-63/obj185 2023-07-29T00:52:49.282 DEBUG:boto:Data: 2023-07-29T00:52:49.282 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:52:49.282 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:52:49.283 DEBUG:boto:Port: 8000 2023-07-29T00:52:49.283 DEBUG:boto:Params: {} 2023-07-29T00:52:49.283 DEBUG:boto:Token: None 2023-07-29T00:52:49.284 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:52:49 GMT /hiiqos-63/obj185 2023-07-29T00:52:49.284 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Oa45o+GYVxS7KdsPmI9YMsft/Fw= 2023-07-29T00:52:49.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Oa45o+GYVxS7KdsPmI9YMsft/Fw='} 2023-07-29T00:52:49.332 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bb419ee43549a8d1-0064c462e1-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:52:49.333 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-63 2023-07-29T00:52:49.334 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:52:49.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:52:49.336 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-63 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:52:49.391 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:52:49.391 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:52:49.425 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:52:49.350+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 1.. 2023-07-29T00:52:54.426 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:52:54.351+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 1.. 2023-07-29T00:52:59.426 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:52:59.352+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 1.. 2023-07-29T00:53:04.427 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:53:04.352+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 1.. 2023-07-29T00:53:09.428 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:53:09.353+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T00:53:14.429 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:53:14.353+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T00:53:18.159 INFO:tasks.rgw.c2.client.1.smithi139.stdout:2023-07-29T00:53:18.084+0000 7f5731a6a640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T00:53:19.430 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:53:19.355+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T00:53:24.431 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:53:24.355+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T00:53:29.431 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:53:29.356+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T00:53:34.432 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:53:34.357+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T00:53:39.433 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:53:39.358+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T00:53:44.433 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:53:44.358+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T00:53:49.434 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:53:49.358+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T00:53:54.434 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:53:54.359+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T00:53:59.436 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:53:59.361+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T00:54:04.437 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:54:04.361+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T00:54:09.437 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:54:09.362+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T00:54:14.438 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:54:14.363+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T00:54:19.439 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:54:19.363+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T00:54:24.440 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:54:24.364+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T00:54:29.440 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:54:29.364+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T00:54:34.441 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:54:34.365+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T00:54:39.442 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:54:39.366+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T00:54:44.442 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:54:44.366+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T00:54:49.443 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:54:49.367+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T00:54:54.444 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:54:54.368+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T00:54:59.444 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:54:59.368+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T00:55:04.445 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:55:04.369+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T00:55:09.446 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:55:09.370+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T00:55:14.447 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:55:14.370+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T00:55:19.447 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:55:19.371+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T00:55:24.448 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:55:24.372+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T00:55:29.449 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:55:29.372+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T00:55:34.450 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:55:34.373+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T00:55:39.451 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:55:39.374+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T00:55:44.451 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:55:44.375+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T00:55:49.452 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:55:49.375+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T00:55:54.453 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:55:54.376+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T00:55:59.453 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:55:59.376+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T00:56:04.454 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:56:04.377+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T00:56:09.455 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:56:09.378+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T00:56:14.455 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:56:14.378+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T00:56:19.456 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:56:19.379+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T00:56:24.457 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:56:24.380+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T00:56:29.457 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:56:29.380+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T00:56:34.458 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:56:34.381+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T00:56:39.459 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:56:39.381+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T00:56:44.459 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:56:44.382+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T00:56:49.460 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:56:49.382+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T00:56:54.461 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:56:54.383+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T00:56:59.462 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:56:59.384+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T00:57:04.462 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:57:04.385+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T00:57:07.858 INFO:tasks.rgw.c1.client.0.smithi101.stdout:2023-07-29T00:57:07.780+0000 7f0081a5e640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T00:57:09.462 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:57:09.384+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T00:57:14.463 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:57:14.385+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T00:57:19.463 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:57:19.385+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T00:57:24.464 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:57:24.386+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T00:57:29.465 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:57:29.387+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T00:57:34.466 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:57:34.387+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T00:57:39.466 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:57:39.388+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T00:57:44.467 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:57:44.388+0000 7f3d7a326880 -1 bucket checkpoint timed out waiting to reach latest generation 6 2023-07-29T00:57:44.467 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:57:44.388+0000 7f3d7a326880 0 bucket sync checkpoint failed: (110) Connection timed out 2023-07-29T00:57:44.467 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:57:44.388+0000 7f3d7a326880 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-07-29T00:57:44.485 DEBUG:teuthology.orchestra.run:got remote process result: 110 2023-07-29T00:57:44.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init ... FAIL 2023-07-29T00:57:44.489 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:57:44.489 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:57:44.490 DEBUG:boto:Method: PUT 2023-07-29T00:57:44.490 DEBUG:boto:Path: /hiiqos-64/ 2023-07-29T00:57:44.490 DEBUG:boto:Data: 2023-07-29T00:57:44.491 DEBUG:boto:Headers: {} 2023-07-29T00:57:44.491 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:57:44.491 DEBUG:boto:Port: 8000 2023-07-29T00:57:44.492 DEBUG:boto:Params: {} 2023-07-29T00:57:44.492 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-29T00:57:44.493 DEBUG:boto:Token: None 2023-07-29T00:57:44.493 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 00:57:44 GMT /hiiqos-64/ 2023-07-29T00:57:44.493 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aAONjYwPC56aXn0s+WXyn3/odc0= 2023-07-29T00:57:44.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:57:44 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aAONjYwPC56aXn0s+WXyn3/odc0=', 'Content-Length': '0'} 2023-07-29T00:57:44.514 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000029debdf09595a743-0064c46408-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:57:44.514 DEBUG:rgw_multi.tests:created bucket=hiiqos-64 2023-07-29T00:57:44.515 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:57:44.515 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:57:44.516 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:57:44.567 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:57:44.567 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:57:44.601 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T00:57:44.615 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T00:57:44.616 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:57:44.616 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T00:57:44.616 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:17.836402Z" 2023-07-29T00:57:44.617 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:57:44.617 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:57:44.617 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592264.435095_1219.1", 2023-07-29T00:57:44.618 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:57:44.435095Z" 2023-07-29T00:57:44.618 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:57:44.618 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:57:44.619 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:57:44.619 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T00:57:44.619 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:57:44.619 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:57:44.620 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T00:57:44.620 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:12.811406Z" 2023-07-29T00:57:44.620 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:57:44.621 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T00:57:44.622 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690591217.836402_294.1', 1: '1_1690592264.435095_1219.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T00:57:44.623 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T00:57:44.623 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:57:44.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T00:57:44.624 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T00:57:44.672 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:57:44.673 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:57:44.702 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:57:44.624+0000 7f2d19d22880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:57:44.702 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:57:44.624+0000 7f2d19d22880 20 RGW-SYNC:meta: read sync status 2023-07-29T00:57:44.715 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T00:57:44.716 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T00:57:44.716 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T00:57:44.717 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T00:57:44.717 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T00:57:44.717 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T00:57:44.717 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:57:44.718 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:57:44.718 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T00:57:44.718 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:57:44.718 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T00:57:44.718 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:57:44.718 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:57:44.719 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T00:57:44.719 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:57:44.719 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T00:57:44.719 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:57:44.719 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T00:57:44.720 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:57:44.720 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:57:44.720 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:57:44.720 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:57:44.720 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T00:57:44.720 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:57:44.723 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:57:44.723 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592264.435095_1219.1", 2023-07-29T00:57:44.724 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:57:44.724 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:57:44.724 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:57:44.724 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:57:44.435095Z", 2023-07-29T00:57:44.724 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:57:44.724 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:57:44.725 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:57:44.725 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:57:44.725 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T00:57:44.725 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:57:44.725 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:57:44.725 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T00:57:44.725 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:57:44.726 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:57:44.726 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:57:44.726 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T00:57:44.726 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:57:44.726 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:57:44.726 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:57:44.726 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T00:57:44.727 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T00:57:44.727 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T00:57:44.727 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T00:57:44.727 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T00:57:44.727 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T00:57:44.727 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T00:57:44.727 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T00:57:44.728 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T00:57:44.728 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T00:57:44.728 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:57:44.728 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T00:57:44.728 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T00:57:44.728 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:57:44.729 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T00:57:44.729 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T00:57:44.729 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T00:57:44.729 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T00:57:44.729 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T00:57:44Z" 2023-07-29T00:57:44.729 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T00:57:44.730 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592264.435095_1219.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:57:44.435095Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T00:57:44Z" } 2023-07-29T00:57:44.731 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592264.435095_1219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:57:44.435095Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T00:57:44.731 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592264.435095_1219.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T00:57:44.732 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592264.435095_1219.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T00:57:44.732 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T00:57:44.732 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:57:44.733 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:57:44.733 DEBUG:boto:Method: HEAD 2023-07-29T00:57:44.734 DEBUG:boto:Path: /hiiqos-64/ 2023-07-29T00:57:44.734 DEBUG:boto:Data: 2023-07-29T00:57:44.734 DEBUG:boto:Headers: {} 2023-07-29T00:57:44.734 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:57:44.735 DEBUG:boto:Port: 8000 2023-07-29T00:57:44.735 DEBUG:boto:Params: {} 2023-07-29T00:57:44.736 DEBUG:boto:Token: None 2023-07-29T00:57:44.736 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:57:44 GMT /hiiqos-64/ 2023-07-29T00:57:44.736 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:05l8NFP7NBoc+DCRvcZc/1nXjPo= 2023-07-29T00:57:44.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:57:44 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:05l8NFP7NBoc+DCRvcZc/1nXjPo=', 'Content-Length': '0'} 2023-07-29T00:57:44.739 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000019722d58c836027a-0064c46408-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:57:44.740 DEBUG:boto:path=/hiiqos-64/obj11 2023-07-29T00:57:44.741 DEBUG:boto:auth_path=/hiiqos-64/obj11 2023-07-29T00:57:44.741 DEBUG:boto:Method: PUT 2023-07-29T00:57:44.742 DEBUG:boto:Path: /hiiqos-64/obj11 2023-07-29T00:57:44.742 DEBUG:boto:Data: 2023-07-29T00:57:44.742 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:57:44.743 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:57:44.743 DEBUG:boto:Port: 8000 2023-07-29T00:57:44.743 DEBUG:boto:Params: {} 2023-07-29T00:57:44.744 DEBUG:boto:Token: None 2023-07-29T00:57:44.744 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:57:44 GMT /hiiqos-64/obj11 2023-07-29T00:57:44.745 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LkEVLCue1hw8tgAOFlQ9dO/8HiE= 2023-07-29T00:57:44.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:57:44 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LkEVLCue1hw8tgAOFlQ9dO/8HiE='} 2023-07-29T00:57:44.752 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000029f829289ecc4d8d-0064c46408-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:57:44.752 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:57:44.753 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:57:44.753 DEBUG:boto:Method: HEAD 2023-07-29T00:57:44.754 DEBUG:boto:Path: /hiiqos-64/ 2023-07-29T00:57:44.754 DEBUG:boto:Data: 2023-07-29T00:57:44.754 DEBUG:boto:Headers: {} 2023-07-29T00:57:44.755 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:57:44.755 DEBUG:boto:Port: 8000 2023-07-29T00:57:44.756 DEBUG:boto:Params: {} 2023-07-29T00:57:44.756 DEBUG:boto:Token: None 2023-07-29T00:57:44.757 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:57:44 GMT /hiiqos-64/ 2023-07-29T00:57:44.757 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:05l8NFP7NBoc+DCRvcZc/1nXjPo= 2023-07-29T00:57:44.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:57:44 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:05l8NFP7NBoc+DCRvcZc/1nXjPo=', 'Content-Length': '0'} 2023-07-29T00:57:44.760 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000916a4b9e6d6e5804-0064c46408-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:57:44.761 DEBUG:boto:path=/hiiqos-64/obj22 2023-07-29T00:57:44.761 DEBUG:boto:auth_path=/hiiqos-64/obj22 2023-07-29T00:57:44.761 DEBUG:boto:Method: PUT 2023-07-29T00:57:44.762 DEBUG:boto:Path: /hiiqos-64/obj22 2023-07-29T00:57:44.762 DEBUG:boto:Data: 2023-07-29T00:57:44.762 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:57:44.763 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:57:44.763 DEBUG:boto:Port: 8000 2023-07-29T00:57:44.763 DEBUG:boto:Params: {} 2023-07-29T00:57:44.764 DEBUG:boto:Token: None 2023-07-29T00:57:44.764 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:57:44 GMT /hiiqos-64/obj22 2023-07-29T00:57:44.765 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:P7R97tGUjlKWyrRjUWlSPgjWi4I= 2023-07-29T00:57:44.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:57:44 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P7R97tGUjlKWyrRjUWlSPgjWi4I='} 2023-07-29T00:57:44.769 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000db6936474333b8b7-0064c46408-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:57:44.769 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:57:44.770 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:57:44.770 DEBUG:boto:Method: HEAD 2023-07-29T00:57:44.771 DEBUG:boto:Path: /hiiqos-64/ 2023-07-29T00:57:44.771 DEBUG:boto:Data: 2023-07-29T00:57:44.771 DEBUG:boto:Headers: {} 2023-07-29T00:57:44.771 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:57:44.772 DEBUG:boto:Port: 8000 2023-07-29T00:57:44.772 DEBUG:boto:Params: {} 2023-07-29T00:57:44.773 DEBUG:boto:Token: None 2023-07-29T00:57:44.773 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:57:44 GMT /hiiqos-64/ 2023-07-29T00:57:44.774 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:05l8NFP7NBoc+DCRvcZc/1nXjPo= 2023-07-29T00:57:44.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:57:44 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:05l8NFP7NBoc+DCRvcZc/1nXjPo=', 'Content-Length': '0'} 2023-07-29T00:57:44.776 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005792cabcee2eb20e-0064c46408-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:57:44.777 DEBUG:boto:path=/hiiqos-64/obj33 2023-07-29T00:57:44.778 DEBUG:boto:auth_path=/hiiqos-64/obj33 2023-07-29T00:57:44.778 DEBUG:boto:Method: PUT 2023-07-29T00:57:44.779 DEBUG:boto:Path: /hiiqos-64/obj33 2023-07-29T00:57:44.779 DEBUG:boto:Data: 2023-07-29T00:57:44.779 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:57:44.780 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:57:44.780 DEBUG:boto:Port: 8000 2023-07-29T00:57:44.781 DEBUG:boto:Params: {} 2023-07-29T00:57:44.781 DEBUG:boto:Token: None 2023-07-29T00:57:44.782 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:57:44 GMT /hiiqos-64/obj33 2023-07-29T00:57:44.783 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RYGPY/pIfWse/kRpS2al0y+B5aU= 2023-07-29T00:57:44.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:57:44 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RYGPY/pIfWse/kRpS2al0y+B5aU='} 2023-07-29T00:57:44.828 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fa6e52bfd83ffe83-0064c46408-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:57:44.829 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:57:44.830 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:57:44.830 DEBUG:boto:Method: HEAD 2023-07-29T00:57:44.831 DEBUG:boto:Path: /hiiqos-64/ 2023-07-29T00:57:44.831 DEBUG:boto:Data: 2023-07-29T00:57:44.831 DEBUG:boto:Headers: {} 2023-07-29T00:57:44.832 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:57:44.832 DEBUG:boto:Port: 8000 2023-07-29T00:57:44.832 DEBUG:boto:Params: {} 2023-07-29T00:57:44.833 DEBUG:boto:Token: None 2023-07-29T00:57:44.833 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:57:44 GMT /hiiqos-64/ 2023-07-29T00:57:44.834 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:05l8NFP7NBoc+DCRvcZc/1nXjPo= 2023-07-29T00:57:44.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:57:44 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:05l8NFP7NBoc+DCRvcZc/1nXjPo=', 'Content-Length': '0'} 2023-07-29T00:57:44.836 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000a902a38d3476a50-0064c46408-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:57:44.837 DEBUG:boto:path=/hiiqos-64/obj44 2023-07-29T00:57:44.838 DEBUG:boto:auth_path=/hiiqos-64/obj44 2023-07-29T00:57:44.838 DEBUG:boto:Method: PUT 2023-07-29T00:57:44.839 DEBUG:boto:Path: /hiiqos-64/obj44 2023-07-29T00:57:44.839 DEBUG:boto:Data: 2023-07-29T00:57:44.840 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:57:44.840 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:57:44.841 DEBUG:boto:Port: 8000 2023-07-29T00:57:44.842 DEBUG:boto:Params: {} 2023-07-29T00:57:44.842 DEBUG:boto:Token: None 2023-07-29T00:57:44.843 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:57:44 GMT /hiiqos-64/obj44 2023-07-29T00:57:44.844 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4NRrMFSAGpJi2iXzWBgjDa0QXlQ= 2023-07-29T00:57:44.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:57:44 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4NRrMFSAGpJi2iXzWBgjDa0QXlQ='} 2023-07-29T00:57:44.851 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006aff2b7e8106040f-0064c46408-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:57:44.851 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:57:44.852 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:57:44.852 DEBUG:boto:Method: HEAD 2023-07-29T00:57:44.853 DEBUG:boto:Path: /hiiqos-64/ 2023-07-29T00:57:44.853 DEBUG:boto:Data: 2023-07-29T00:57:44.853 DEBUG:boto:Headers: {} 2023-07-29T00:57:44.854 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:57:44.854 DEBUG:boto:Port: 8000 2023-07-29T00:57:44.854 DEBUG:boto:Params: {} 2023-07-29T00:57:44.855 DEBUG:boto:Token: None 2023-07-29T00:57:44.855 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:57:44 GMT /hiiqos-64/ 2023-07-29T00:57:44.856 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:05l8NFP7NBoc+DCRvcZc/1nXjPo= 2023-07-29T00:57:44.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:57:44 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:05l8NFP7NBoc+DCRvcZc/1nXjPo=', 'Content-Length': '0'} 2023-07-29T00:57:44.858 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a73445e514e31be4-0064c46408-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:57:44.859 DEBUG:boto:path=/hiiqos-64/obj55 2023-07-29T00:57:44.859 DEBUG:boto:auth_path=/hiiqos-64/obj55 2023-07-29T00:57:44.860 DEBUG:boto:Method: PUT 2023-07-29T00:57:44.860 DEBUG:boto:Path: /hiiqos-64/obj55 2023-07-29T00:57:44.860 DEBUG:boto:Data: 2023-07-29T00:57:44.861 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:57:44.861 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:57:44.861 DEBUG:boto:Port: 8000 2023-07-29T00:57:44.862 DEBUG:boto:Params: {} 2023-07-29T00:57:44.862 DEBUG:boto:Token: None 2023-07-29T00:57:44.862 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:57:44 GMT /hiiqos-64/obj55 2023-07-29T00:57:44.863 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cJkwqsyowCL7EAURdLz82kAamV0= 2023-07-29T00:57:44.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:57:44 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cJkwqsyowCL7EAURdLz82kAamV0='} 2023-07-29T00:57:44.867 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000091524d199f40b679-0064c46408-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:57:44.867 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-64 2023-07-29T00:57:44.868 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:57:44.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:57:44.868 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-64 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:57:44.921 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:57:44.921 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:57:44.955 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:57:44.877+0000 7f78afb24880 1 waiting to reach incremental sync.. 2023-07-29T00:57:49.957 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:57:49.879+0000 7f78afb24880 1 bucket sync caught up with source: 2023-07-29T00:57:49.957 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , 00000000001.107.6, 00000000002.133.6, 00000000002.112.6] 2023-07-29T00:57:49.958 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000001.107.6, 00000000002.133.6, 00000000002.112.6] 2023-07-29T00:57:49.958 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:57:49.879+0000 7f78afb24880 0 bucket checkpoint complete 2023-07-29T00:57:49.975 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hiiqos-64 2023-07-29T00:57:49.976 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:57:49.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:57:49.977 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-64 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:57:50.034 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:57:50.034 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:57:50.070 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:57:49.992+0000 7f0f61b29880 1 waiting for incremental sync to catch up: 2023-07-29T00:57:50.070 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , 00000000001.228.6, 00000000002.977.6, 00000000001.1073.6] 2023-07-29T00:57:50.071 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000001.228.6, 00000000002.977.6, 00000000002.1075.6] 2023-07-29T00:57:55.071 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:57:54.993+0000 7f0f61b29880 1 waiting for incremental sync to catch up: 2023-07-29T00:57:55.072 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , 00000000001.228.6, 00000000002.977.6, 00000000001.1073.6] 2023-07-29T00:57:55.072 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000001.228.6, 00000000002.977.6, 00000000002.1075.6] 2023-07-29T00:58:00.073 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:57:59.994+0000 7f0f61b29880 1 waiting for incremental sync to catch up: 2023-07-29T00:58:00.073 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , 00000000001.228.6, 00000000002.977.6, 00000000001.1073.6] 2023-07-29T00:58:00.073 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000001.228.6, 00000000002.977.6, 00000000002.1075.6] 2023-07-29T00:58:05.074 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:58:04.995+0000 7f0f61b29880 1 waiting for incremental sync to catch up: 2023-07-29T00:58:05.074 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , 00000000001.228.6, 00000000002.977.6, 00000000001.1073.6] 2023-07-29T00:58:05.074 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000001.228.6, 00000000002.977.6, 00000000002.1075.6] 2023-07-29T00:58:10.075 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:58:09.996+0000 7f0f61b29880 1 bucket sync caught up with source: 2023-07-29T00:58:10.075 INFO:teuthology.orchestra.run.smithi101.stderr: local status: [, , , , , , , , 00000000001.228.6, 00000000002.977.6, 00000000002.1075.6] 2023-07-29T00:58:10.076 INFO:teuthology.orchestra.run.smithi101.stderr: remote markers: [, , , , , , , , 00000000001.228.6, 00000000002.977.6, 00000000002.1075.6] 2023-07-29T00:58:10.076 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:58:09.996+0000 7f0f61b29880 0 bucket checkpoint complete 2023-07-29T00:58:10.094 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hiiqos-64 zones={a2, a1} 2023-07-29T00:58:10.095 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:10.095 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:10.096 DEBUG:boto:Method: HEAD 2023-07-29T00:58:10.096 DEBUG:boto:Path: /hiiqos-64/ 2023-07-29T00:58:10.096 DEBUG:boto:Data: 2023-07-29T00:58:10.097 DEBUG:boto:Headers: {} 2023-07-29T00:58:10.097 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:58:10.097 DEBUG:boto:Port: 8001 2023-07-29T00:58:10.098 DEBUG:boto:Params: {} 2023-07-29T00:58:10.098 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-29T00:58:10.099 DEBUG:boto:Token: None 2023-07-29T00:58:10.099 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:10 GMT /hiiqos-64/ 2023-07-29T00:58:10.100 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WKXNvM9Ev91xGB4i6WmyYRo9S6g= 2023-07-29T00:58:10.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WKXNvM9Ev91xGB4i6WmyYRo9S6g=', 'Content-Length': '0'} 2023-07-29T00:58:15.104 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fef553293c7c1e27-0064c46427-2068-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.105 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:15.105 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:15.105 DEBUG:boto:Method: HEAD 2023-07-29T00:58:15.106 DEBUG:boto:Path: /hiiqos-64/ 2023-07-29T00:58:15.106 DEBUG:boto:Data: 2023-07-29T00:58:15.106 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.107 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:15.107 DEBUG:boto:Port: 8000 2023-07-29T00:58:15.107 DEBUG:boto:Params: {} 2023-07-29T00:58:15.108 DEBUG:boto:Token: None 2023-07-29T00:58:15.108 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/ 2023-07-29T00:58:15.109 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:CuuyMElb89TKlDMztYqnG3NGO5I= 2023-07-29T00:58:15.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CuuyMElb89TKlDMztYqnG3NGO5I=', 'Content-Length': '0'} 2023-07-29T00:58:15.112 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000062b1d15005841e9a-0064c46427-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.112 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-07-29T00:58:15.113 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:15.113 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:15.113 DEBUG:boto:path=/hiiqos-64/?versions&max-keys=999 2023-07-29T00:58:15.114 DEBUG:boto:auth_path=/hiiqos-64/?versions&max-keys=999 2023-07-29T00:58:15.114 DEBUG:boto:Method: GET 2023-07-29T00:58:15.115 DEBUG:boto:Path: /hiiqos-64/?versions&max-keys=999 2023-07-29T00:58:15.115 DEBUG:boto:Data: 2023-07-29T00:58:15.115 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.116 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:58:15.116 DEBUG:boto:Port: 8001 2023-07-29T00:58:15.116 DEBUG:boto:Params: {} 2023-07-29T00:58:15.117 DEBUG:boto:Token: None 2023-07-29T00:58:15.117 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/?versions 2023-07-29T00:58:15.117 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:koEO66FOp5U9ET2vBpIwkWbwDXM= 2023-07-29T00:58:15.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:koEO66FOp5U9ET2vBpIwkWbwDXM=', 'Content-Length': '0'} 2023-07-29T00:58:15.120 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7e9490669d028cd-0064c46427-2068-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.121 DEBUG:boto:b'hiiqos-64999falseobj11nulltrue2023-07-29T00:57:44.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:57:44.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:57:44.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:57:44.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:57:44.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:58:15.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-07-29T00:58:15.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-07-29T00:58:15.123 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-07-29T00:58:15.123 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-07-29T00:58:15.123 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-07-29T00:58:15.124 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-07-29T00:58:15.124 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:15.124 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:15.125 DEBUG:boto:path=/hiiqos-64/?versions&max-keys=999 2023-07-29T00:58:15.125 DEBUG:boto:auth_path=/hiiqos-64/?versions&max-keys=999 2023-07-29T00:58:15.125 DEBUG:boto:Method: GET 2023-07-29T00:58:15.126 DEBUG:boto:Path: /hiiqos-64/?versions&max-keys=999 2023-07-29T00:58:15.126 DEBUG:boto:Data: 2023-07-29T00:58:15.126 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.127 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:15.127 DEBUG:boto:Port: 8000 2023-07-29T00:58:15.127 DEBUG:boto:Params: {} 2023-07-29T00:58:15.128 DEBUG:boto:Token: None 2023-07-29T00:58:15.128 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/?versions 2023-07-29T00:58:15.129 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:koEO66FOp5U9ET2vBpIwkWbwDXM= 2023-07-29T00:58:15.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:koEO66FOp5U9ET2vBpIwkWbwDXM=', 'Content-Length': '0'} 2023-07-29T00:58:15.132 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000967738b3007c629f-0064c46427-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.132 DEBUG:boto:b'hiiqos-64999falseobj11nulltrue2023-07-29T00:57:44.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:57:44.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:57:44.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:57:44.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:57:44.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:58:15.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-07-29T00:58:15.134 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-07-29T00:58:15.134 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-07-29T00:58:15.134 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-07-29T00:58:15.135 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-07-29T00:58:15.135 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:15.135 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:15.136 DEBUG:boto:path=/hiiqos-64/?versions&max-keys=999 2023-07-29T00:58:15.136 DEBUG:boto:auth_path=/hiiqos-64/?versions&max-keys=999 2023-07-29T00:58:15.136 DEBUG:boto:Method: GET 2023-07-29T00:58:15.137 DEBUG:boto:Path: /hiiqos-64/?versions&max-keys=999 2023-07-29T00:58:15.137 DEBUG:boto:Data: 2023-07-29T00:58:15.137 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.138 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:58:15.138 DEBUG:boto:Port: 8001 2023-07-29T00:58:15.138 DEBUG:boto:Params: {} 2023-07-29T00:58:15.139 DEBUG:boto:Token: None 2023-07-29T00:58:15.139 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/?versions 2023-07-29T00:58:15.140 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:koEO66FOp5U9ET2vBpIwkWbwDXM= 2023-07-29T00:58:15.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:koEO66FOp5U9ET2vBpIwkWbwDXM=', 'Content-Length': '0'} 2023-07-29T00:58:15.142 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013a772ea5e358d18-0064c46427-2068-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.143 DEBUG:boto:b'hiiqos-64999falseobj11nulltrue2023-07-29T00:57:44.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:57:44.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:57:44.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:57:44.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:57:44.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:58:15.144 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:15.144 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:15.144 DEBUG:boto:path=/hiiqos-64/?versions&max-keys=999 2023-07-29T00:58:15.145 DEBUG:boto:auth_path=/hiiqos-64/?versions&max-keys=999 2023-07-29T00:58:15.145 DEBUG:boto:Method: GET 2023-07-29T00:58:15.145 DEBUG:boto:Path: /hiiqos-64/?versions&max-keys=999 2023-07-29T00:58:15.146 DEBUG:boto:Data: 2023-07-29T00:58:15.146 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.146 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:15.147 DEBUG:boto:Port: 8000 2023-07-29T00:58:15.147 DEBUG:boto:Params: {} 2023-07-29T00:58:15.148 DEBUG:boto:Token: None 2023-07-29T00:58:15.148 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/?versions 2023-07-29T00:58:15.148 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:koEO66FOp5U9ET2vBpIwkWbwDXM= 2023-07-29T00:58:15.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:koEO66FOp5U9ET2vBpIwkWbwDXM=', 'Content-Length': '0'} 2023-07-29T00:58:15.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b79c154ff222877-0064c46427-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.152 DEBUG:boto:b'hiiqos-64999falseobj11nulltrue2023-07-29T00:57:44.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:57:44.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:57:44.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:57:44.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:57:44.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:58:15.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-07-29T00:58:15.153 DEBUG:boto:path=/hiiqos-64/obj11 2023-07-29T00:58:15.153 DEBUG:boto:auth_path=/hiiqos-64/obj11 2023-07-29T00:58:15.154 DEBUG:boto:path=/hiiqos-64/obj11?versionId=null 2023-07-29T00:58:15.154 DEBUG:boto:auth_path=/hiiqos-64/obj11?versionId=null 2023-07-29T00:58:15.155 DEBUG:boto:Method: GET 2023-07-29T00:58:15.155 DEBUG:boto:Path: /hiiqos-64/obj11?versionId=null 2023-07-29T00:58:15.155 DEBUG:boto:Data: 2023-07-29T00:58:15.156 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.156 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:58:15.156 DEBUG:boto:Port: 8001 2023-07-29T00:58:15.157 DEBUG:boto:Params: {} 2023-07-29T00:58:15.157 DEBUG:boto:Token: None 2023-07-29T00:58:15.158 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj11?versionId=null 2023-07-29T00:58:15.158 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jzVbZ/D7p63u2+qeaR0iTVLlhWw= 2023-07-29T00:58:15.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jzVbZ/D7p63u2+qeaR0iTVLlhWw=', 'Content-Length': '0'} 2023-07-29T00:58:15.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000311774593151e260-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.161 DEBUG:boto:path=/hiiqos-64/obj11 2023-07-29T00:58:15.161 DEBUG:boto:auth_path=/hiiqos-64/obj11 2023-07-29T00:58:15.162 DEBUG:boto:path=/hiiqos-64/obj11?versionId=null 2023-07-29T00:58:15.162 DEBUG:boto:auth_path=/hiiqos-64/obj11?versionId=null 2023-07-29T00:58:15.162 DEBUG:boto:Method: GET 2023-07-29T00:58:15.163 DEBUG:boto:Path: /hiiqos-64/obj11?versionId=null 2023-07-29T00:58:15.163 DEBUG:boto:Data: 2023-07-29T00:58:15.164 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.164 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:15.164 DEBUG:boto:Port: 8000 2023-07-29T00:58:15.165 DEBUG:boto:Params: {} 2023-07-29T00:58:15.165 DEBUG:boto:Token: None 2023-07-29T00:58:15.166 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj11?versionId=null 2023-07-29T00:58:15.166 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jzVbZ/D7p63u2+qeaR0iTVLlhWw= 2023-07-29T00:58:15.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jzVbZ/D7p63u2+qeaR0iTVLlhWw=', 'Content-Length': '0'} 2023-07-29T00:58:15.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000119027d3d80d52f2-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.212 DEBUG:boto:path=/hiiqos-64/obj11 2023-07-29T00:58:15.213 DEBUG:boto:auth_path=/hiiqos-64/obj11 2023-07-29T00:58:15.213 DEBUG:boto:path=/hiiqos-64/obj11?versionId=null 2023-07-29T00:58:15.214 DEBUG:boto:auth_path=/hiiqos-64/obj11?versionId=null 2023-07-29T00:58:15.214 DEBUG:boto:Method: HEAD 2023-07-29T00:58:15.215 DEBUG:boto:Path: /hiiqos-64/obj11?versionId=null 2023-07-29T00:58:15.215 DEBUG:boto:Data: 2023-07-29T00:58:15.215 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.216 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:58:15.216 DEBUG:boto:Port: 8001 2023-07-29T00:58:15.216 DEBUG:boto:Params: {} 2023-07-29T00:58:15.217 DEBUG:boto:Token: None 2023-07-29T00:58:15.217 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj11?versionId=null 2023-07-29T00:58:15.218 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Hgv99SpyrzO5a5yNRc77KqKYlgQ= 2023-07-29T00:58:15.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Hgv99SpyrzO5a5yNRc77KqKYlgQ=', 'Content-Length': '0'} 2023-07-29T00:58:15.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e91cbfa698655ae0-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.222 DEBUG:boto:path=/hiiqos-64/obj11 2023-07-29T00:58:15.222 DEBUG:boto:auth_path=/hiiqos-64/obj11 2023-07-29T00:58:15.223 DEBUG:boto:path=/hiiqos-64/obj11?versionId=null 2023-07-29T00:58:15.223 DEBUG:boto:auth_path=/hiiqos-64/obj11?versionId=null 2023-07-29T00:58:15.224 DEBUG:boto:Method: HEAD 2023-07-29T00:58:15.224 DEBUG:boto:Path: /hiiqos-64/obj11?versionId=null 2023-07-29T00:58:15.225 DEBUG:boto:Data: 2023-07-29T00:58:15.225 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.226 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:15.226 DEBUG:boto:Port: 8000 2023-07-29T00:58:15.227 DEBUG:boto:Params: {} 2023-07-29T00:58:15.227 DEBUG:boto:Token: None 2023-07-29T00:58:15.227 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj11?versionId=null 2023-07-29T00:58:15.228 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Hgv99SpyrzO5a5yNRc77KqKYlgQ= 2023-07-29T00:58:15.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Hgv99SpyrzO5a5yNRc77KqKYlgQ=', 'Content-Length': '0'} 2023-07-29T00:58:15.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a7b767975522116-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-07-29T00:58:15.232 DEBUG:boto:path=/hiiqos-64/obj11 2023-07-29T00:58:15.233 DEBUG:boto:auth_path=/hiiqos-64/obj11 2023-07-29T00:58:15.234 DEBUG:boto:path=/hiiqos-64/obj11?versionId=null 2023-07-29T00:58:15.234 DEBUG:boto:auth_path=/hiiqos-64/obj11?versionId=null 2023-07-29T00:58:15.235 DEBUG:boto:Method: GET 2023-07-29T00:58:15.236 DEBUG:boto:Path: /hiiqos-64/obj11?versionId=null 2023-07-29T00:58:15.237 DEBUG:boto:Data: 2023-07-29T00:58:15.237 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.237 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:58:15.238 DEBUG:boto:Port: 8001 2023-07-29T00:58:15.239 DEBUG:boto:Params: {} 2023-07-29T00:58:15.240 DEBUG:boto:Token: None 2023-07-29T00:58:15.241 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj11?versionId=null 2023-07-29T00:58:15.242 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jzVbZ/D7p63u2+qeaR0iTVLlhWw= 2023-07-29T00:58:15.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jzVbZ/D7p63u2+qeaR0iTVLlhWw=', 'Content-Length': '0'} 2023-07-29T00:58:15.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c823e23fb72942a2-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.247 DEBUG:boto:path=/hiiqos-64/obj11 2023-07-29T00:58:15.248 DEBUG:boto:auth_path=/hiiqos-64/obj11 2023-07-29T00:58:15.248 DEBUG:boto:path=/hiiqos-64/obj11?versionId=null 2023-07-29T00:58:15.249 DEBUG:boto:auth_path=/hiiqos-64/obj11?versionId=null 2023-07-29T00:58:15.250 DEBUG:boto:Method: GET 2023-07-29T00:58:15.250 DEBUG:boto:Path: /hiiqos-64/obj11?versionId=null 2023-07-29T00:58:15.251 DEBUG:boto:Data: 2023-07-29T00:58:15.252 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.253 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:15.253 DEBUG:boto:Port: 8000 2023-07-29T00:58:15.254 DEBUG:boto:Params: {} 2023-07-29T00:58:15.255 DEBUG:boto:Token: None 2023-07-29T00:58:15.256 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj11?versionId=null 2023-07-29T00:58:15.257 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:jzVbZ/D7p63u2+qeaR0iTVLlhWw= 2023-07-29T00:58:15.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jzVbZ/D7p63u2+qeaR0iTVLlhWw=', 'Content-Length': '0'} 2023-07-29T00:58:15.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047375a9069596eea-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.261 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:15.261 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:15.262 DEBUG:boto:path=/hiiqos-64/?prefix=obj11 2023-07-29T00:58:15.262 DEBUG:boto:auth_path=/hiiqos-64/?prefix=obj11 2023-07-29T00:58:15.262 DEBUG:boto:Method: GET 2023-07-29T00:58:15.263 DEBUG:boto:Path: /hiiqos-64/?prefix=obj11 2023-07-29T00:58:15.263 DEBUG:boto:Data: 2023-07-29T00:58:15.263 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.264 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:58:15.264 DEBUG:boto:Port: 8001 2023-07-29T00:58:15.265 DEBUG:boto:Params: {} 2023-07-29T00:58:15.265 DEBUG:boto:Token: None 2023-07-29T00:58:15.265 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/ 2023-07-29T00:58:15.266 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU= 2023-07-29T00:58:15.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU=', 'Content-Length': '0'} 2023-07-29T00:58:15.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048871800185640fd-0064c46427-2068-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.269 DEBUG:boto:b'hiiqos-64obj111000falseobj112023-07-29T00:57:44.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:58:15.270 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:15.270 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:15.271 DEBUG:boto:path=/hiiqos-64/?prefix=obj11 2023-07-29T00:58:15.271 DEBUG:boto:auth_path=/hiiqos-64/?prefix=obj11 2023-07-29T00:58:15.271 DEBUG:boto:Method: GET 2023-07-29T00:58:15.272 DEBUG:boto:Path: /hiiqos-64/?prefix=obj11 2023-07-29T00:58:15.272 DEBUG:boto:Data: 2023-07-29T00:58:15.273 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.273 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:15.273 DEBUG:boto:Port: 8000 2023-07-29T00:58:15.274 DEBUG:boto:Params: {} 2023-07-29T00:58:15.274 DEBUG:boto:Token: None 2023-07-29T00:58:15.274 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/ 2023-07-29T00:58:15.275 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU= 2023-07-29T00:58:15.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU=', 'Content-Length': '0'} 2023-07-29T00:58:15.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000704f65ac17fc2329-0064c46427-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.278 DEBUG:boto:b'hiiqos-64obj111000falseobj112023-07-29T00:57:44.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:58:15.279 DEBUG:boto:path=/hiiqos-64/obj11 2023-07-29T00:58:15.279 DEBUG:boto:auth_path=/hiiqos-64/obj11 2023-07-29T00:58:15.280 DEBUG:boto:Method: HEAD 2023-07-29T00:58:15.280 DEBUG:boto:Path: /hiiqos-64/obj11 2023-07-29T00:58:15.280 DEBUG:boto:Data: 2023-07-29T00:58:15.281 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.281 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:58:15.281 DEBUG:boto:Port: 8001 2023-07-29T00:58:15.282 DEBUG:boto:Params: {} 2023-07-29T00:58:15.282 DEBUG:boto:Token: None 2023-07-29T00:58:15.283 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj11 2023-07-29T00:58:15.283 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xbcZQz3XaE81XFWG/f6GAO8hHLw= 2023-07-29T00:58:15.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xbcZQz3XaE81XFWG/f6GAO8hHLw=', 'Content-Length': '0'} 2023-07-29T00:58:15.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9d0ea91fe70462c-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.286 DEBUG:boto:path=/hiiqos-64/obj11 2023-07-29T00:58:15.286 DEBUG:boto:auth_path=/hiiqos-64/obj11 2023-07-29T00:58:15.287 DEBUG:boto:Method: HEAD 2023-07-29T00:58:15.287 DEBUG:boto:Path: /hiiqos-64/obj11 2023-07-29T00:58:15.287 DEBUG:boto:Data: 2023-07-29T00:58:15.288 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.288 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:15.288 DEBUG:boto:Port: 8000 2023-07-29T00:58:15.289 DEBUG:boto:Params: {} 2023-07-29T00:58:15.289 DEBUG:boto:Token: None 2023-07-29T00:58:15.290 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj11 2023-07-29T00:58:15.290 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xbcZQz3XaE81XFWG/f6GAO8hHLw= 2023-07-29T00:58:15.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xbcZQz3XaE81XFWG/f6GAO8hHLw=', 'Content-Length': '0'} 2023-07-29T00:58:15.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000351cd01c98d37783-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-07-29T00:58:15.294 DEBUG:boto:path=/hiiqos-64/obj11 2023-07-29T00:58:15.294 DEBUG:boto:auth_path=/hiiqos-64/obj11 2023-07-29T00:58:15.294 DEBUG:boto:Method: GET 2023-07-29T00:58:15.295 DEBUG:boto:Path: /hiiqos-64/obj11 2023-07-29T00:58:15.295 DEBUG:boto:Data: 2023-07-29T00:58:15.295 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.296 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:58:15.296 DEBUG:boto:Port: 8001 2023-07-29T00:58:15.296 DEBUG:boto:Params: {} 2023-07-29T00:58:15.297 DEBUG:boto:Token: None 2023-07-29T00:58:15.297 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj11 2023-07-29T00:58:15.298 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZcdF4BmKXTP0r+kDurGZmjcQlQ4= 2023-07-29T00:58:15.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZcdF4BmKXTP0r+kDurGZmjcQlQ4=', 'Content-Length': '0'} 2023-07-29T00:58:15.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047d262a2eb4b3c62-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.340 DEBUG:boto:path=/hiiqos-64/obj11 2023-07-29T00:58:15.341 DEBUG:boto:auth_path=/hiiqos-64/obj11 2023-07-29T00:58:15.341 DEBUG:boto:Method: GET 2023-07-29T00:58:15.342 DEBUG:boto:Path: /hiiqos-64/obj11 2023-07-29T00:58:15.342 DEBUG:boto:Data: 2023-07-29T00:58:15.342 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.343 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:15.343 DEBUG:boto:Port: 8000 2023-07-29T00:58:15.343 DEBUG:boto:Params: {} 2023-07-29T00:58:15.344 DEBUG:boto:Token: None 2023-07-29T00:58:15.344 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj11 2023-07-29T00:58:15.345 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ZcdF4BmKXTP0r+kDurGZmjcQlQ4= 2023-07-29T00:58:15.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZcdF4BmKXTP0r+kDurGZmjcQlQ4=', 'Content-Length': '0'} 2023-07-29T00:58:15.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0e2b1652251bf6a-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.348 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:15.349 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:15.349 DEBUG:boto:path=/hiiqos-64/?prefix=obj11 2023-07-29T00:58:15.349 DEBUG:boto:auth_path=/hiiqos-64/?prefix=obj11 2023-07-29T00:58:15.350 DEBUG:boto:Method: GET 2023-07-29T00:58:15.350 DEBUG:boto:Path: /hiiqos-64/?prefix=obj11 2023-07-29T00:58:15.351 DEBUG:boto:Data: 2023-07-29T00:58:15.351 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.351 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:58:15.352 DEBUG:boto:Port: 8001 2023-07-29T00:58:15.352 DEBUG:boto:Params: {} 2023-07-29T00:58:15.352 DEBUG:boto:Token: None 2023-07-29T00:58:15.353 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/ 2023-07-29T00:58:15.353 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU= 2023-07-29T00:58:15.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU=', 'Content-Length': '0'} 2023-07-29T00:58:15.356 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000097f987e89eb1b75-0064c46427-2068-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.357 DEBUG:boto:b'hiiqos-64obj111000falseobj112023-07-29T00:57:44.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:58:15.358 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:15.358 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:15.358 DEBUG:boto:path=/hiiqos-64/?prefix=obj11 2023-07-29T00:58:15.359 DEBUG:boto:auth_path=/hiiqos-64/?prefix=obj11 2023-07-29T00:58:15.359 DEBUG:boto:Method: GET 2023-07-29T00:58:15.359 DEBUG:boto:Path: /hiiqos-64/?prefix=obj11 2023-07-29T00:58:15.360 DEBUG:boto:Data: 2023-07-29T00:58:15.360 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.360 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:15.361 DEBUG:boto:Port: 8000 2023-07-29T00:58:15.361 DEBUG:boto:Params: {} 2023-07-29T00:58:15.362 DEBUG:boto:Token: None 2023-07-29T00:58:15.362 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/ 2023-07-29T00:58:15.362 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU= 2023-07-29T00:58:15.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU=', 'Content-Length': '0'} 2023-07-29T00:58:15.365 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5bc05c8bf53554c-0064c46427-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.366 DEBUG:boto:b'hiiqos-64obj111000falseobj112023-07-29T00:57:44.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:58:15.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-07-29T00:58:15.367 DEBUG:boto:path=/hiiqos-64/obj22 2023-07-29T00:58:15.368 DEBUG:boto:auth_path=/hiiqos-64/obj22 2023-07-29T00:58:15.368 DEBUG:boto:path=/hiiqos-64/obj22?versionId=null 2023-07-29T00:58:15.368 DEBUG:boto:auth_path=/hiiqos-64/obj22?versionId=null 2023-07-29T00:58:15.369 DEBUG:boto:Method: GET 2023-07-29T00:58:15.369 DEBUG:boto:Path: /hiiqos-64/obj22?versionId=null 2023-07-29T00:58:15.370 DEBUG:boto:Data: 2023-07-29T00:58:15.370 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.370 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:58:15.371 DEBUG:boto:Port: 8001 2023-07-29T00:58:15.371 DEBUG:boto:Params: {} 2023-07-29T00:58:15.371 DEBUG:boto:Token: None 2023-07-29T00:58:15.372 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj22?versionId=null 2023-07-29T00:58:15.373 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eKjgE+uYwh346RSIkLIvDkS6txo= 2023-07-29T00:58:15.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eKjgE+uYwh346RSIkLIvDkS6txo=', 'Content-Length': '0'} 2023-07-29T00:58:15.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5f8c48de68042e4-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.376 DEBUG:boto:path=/hiiqos-64/obj22 2023-07-29T00:58:15.377 DEBUG:boto:auth_path=/hiiqos-64/obj22 2023-07-29T00:58:15.377 DEBUG:boto:path=/hiiqos-64/obj22?versionId=null 2023-07-29T00:58:15.378 DEBUG:boto:auth_path=/hiiqos-64/obj22?versionId=null 2023-07-29T00:58:15.378 DEBUG:boto:Method: GET 2023-07-29T00:58:15.378 DEBUG:boto:Path: /hiiqos-64/obj22?versionId=null 2023-07-29T00:58:15.379 DEBUG:boto:Data: 2023-07-29T00:58:15.379 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.379 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:15.380 DEBUG:boto:Port: 8000 2023-07-29T00:58:15.380 DEBUG:boto:Params: {} 2023-07-29T00:58:15.381 DEBUG:boto:Token: None 2023-07-29T00:58:15.381 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj22?versionId=null 2023-07-29T00:58:15.381 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eKjgE+uYwh346RSIkLIvDkS6txo= 2023-07-29T00:58:15.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eKjgE+uYwh346RSIkLIvDkS6txo=', 'Content-Length': '0'} 2023-07-29T00:58:15.384 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006150b1776c24dd75-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.385 DEBUG:boto:path=/hiiqos-64/obj22 2023-07-29T00:58:15.385 DEBUG:boto:auth_path=/hiiqos-64/obj22 2023-07-29T00:58:15.386 DEBUG:boto:path=/hiiqos-64/obj22?versionId=null 2023-07-29T00:58:15.386 DEBUG:boto:auth_path=/hiiqos-64/obj22?versionId=null 2023-07-29T00:58:15.386 DEBUG:boto:Method: HEAD 2023-07-29T00:58:15.387 DEBUG:boto:Path: /hiiqos-64/obj22?versionId=null 2023-07-29T00:58:15.387 DEBUG:boto:Data: 2023-07-29T00:58:15.387 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.388 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:58:15.388 DEBUG:boto:Port: 8001 2023-07-29T00:58:15.388 DEBUG:boto:Params: {} 2023-07-29T00:58:15.389 DEBUG:boto:Token: None 2023-07-29T00:58:15.389 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj22?versionId=null 2023-07-29T00:58:15.390 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ym1jHBlbq73Lp2t+iw2fZs6r758= 2023-07-29T00:58:15.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ym1jHBlbq73Lp2t+iw2fZs6r758=', 'Content-Length': '0'} 2023-07-29T00:58:15.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a48c3948d1db5e6d-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.392 DEBUG:boto:path=/hiiqos-64/obj22 2023-07-29T00:58:15.393 DEBUG:boto:auth_path=/hiiqos-64/obj22 2023-07-29T00:58:15.393 DEBUG:boto:path=/hiiqos-64/obj22?versionId=null 2023-07-29T00:58:15.393 DEBUG:boto:auth_path=/hiiqos-64/obj22?versionId=null 2023-07-29T00:58:15.394 DEBUG:boto:Method: HEAD 2023-07-29T00:58:15.394 DEBUG:boto:Path: /hiiqos-64/obj22?versionId=null 2023-07-29T00:58:15.394 DEBUG:boto:Data: 2023-07-29T00:58:15.395 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.395 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:15.395 DEBUG:boto:Port: 8000 2023-07-29T00:58:15.396 DEBUG:boto:Params: {} 2023-07-29T00:58:15.396 DEBUG:boto:Token: None 2023-07-29T00:58:15.397 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj22?versionId=null 2023-07-29T00:58:15.397 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ym1jHBlbq73Lp2t+iw2fZs6r758= 2023-07-29T00:58:15.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ym1jHBlbq73Lp2t+iw2fZs6r758=', 'Content-Length': '0'} 2023-07-29T00:58:15.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4149a6cd8fdd489-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-07-29T00:58:15.400 DEBUG:boto:path=/hiiqos-64/obj22 2023-07-29T00:58:15.401 DEBUG:boto:auth_path=/hiiqos-64/obj22 2023-07-29T00:58:15.401 DEBUG:boto:path=/hiiqos-64/obj22?versionId=null 2023-07-29T00:58:15.402 DEBUG:boto:auth_path=/hiiqos-64/obj22?versionId=null 2023-07-29T00:58:15.402 DEBUG:boto:Method: GET 2023-07-29T00:58:15.402 DEBUG:boto:Path: /hiiqos-64/obj22?versionId=null 2023-07-29T00:58:15.403 DEBUG:boto:Data: 2023-07-29T00:58:15.403 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.403 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:58:15.404 DEBUG:boto:Port: 8001 2023-07-29T00:58:15.404 DEBUG:boto:Params: {} 2023-07-29T00:58:15.405 DEBUG:boto:Token: None 2023-07-29T00:58:15.405 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj22?versionId=null 2023-07-29T00:58:15.405 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eKjgE+uYwh346RSIkLIvDkS6txo= 2023-07-29T00:58:15.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eKjgE+uYwh346RSIkLIvDkS6txo=', 'Content-Length': '0'} 2023-07-29T00:58:15.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2e10562bad0b0d3-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.409 DEBUG:boto:path=/hiiqos-64/obj22 2023-07-29T00:58:15.409 DEBUG:boto:auth_path=/hiiqos-64/obj22 2023-07-29T00:58:15.410 DEBUG:boto:path=/hiiqos-64/obj22?versionId=null 2023-07-29T00:58:15.410 DEBUG:boto:auth_path=/hiiqos-64/obj22?versionId=null 2023-07-29T00:58:15.410 DEBUG:boto:Method: GET 2023-07-29T00:58:15.411 DEBUG:boto:Path: /hiiqos-64/obj22?versionId=null 2023-07-29T00:58:15.411 DEBUG:boto:Data: 2023-07-29T00:58:15.412 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.412 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:15.412 DEBUG:boto:Port: 8000 2023-07-29T00:58:15.413 DEBUG:boto:Params: {} 2023-07-29T00:58:15.413 DEBUG:boto:Token: None 2023-07-29T00:58:15.414 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj22?versionId=null 2023-07-29T00:58:15.414 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eKjgE+uYwh346RSIkLIvDkS6txo= 2023-07-29T00:58:15.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eKjgE+uYwh346RSIkLIvDkS6txo=', 'Content-Length': '0'} 2023-07-29T00:58:15.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056948ca90aec8341-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.460 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:15.461 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:15.461 DEBUG:boto:path=/hiiqos-64/?prefix=obj22 2023-07-29T00:58:15.462 DEBUG:boto:auth_path=/hiiqos-64/?prefix=obj22 2023-07-29T00:58:15.462 DEBUG:boto:Method: GET 2023-07-29T00:58:15.462 DEBUG:boto:Path: /hiiqos-64/?prefix=obj22 2023-07-29T00:58:15.463 DEBUG:boto:Data: 2023-07-29T00:58:15.463 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.463 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:58:15.464 DEBUG:boto:Port: 8001 2023-07-29T00:58:15.464 DEBUG:boto:Params: {} 2023-07-29T00:58:15.465 DEBUG:boto:Token: None 2023-07-29T00:58:15.465 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/ 2023-07-29T00:58:15.465 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU= 2023-07-29T00:58:15.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU=', 'Content-Length': '0'} 2023-07-29T00:58:15.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d4a8463394aa5f6-0064c46427-2068-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.469 DEBUG:boto:b'hiiqos-64obj221000falseobj222023-07-29T00:57:44.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:58:15.470 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:15.470 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:15.471 DEBUG:boto:path=/hiiqos-64/?prefix=obj22 2023-07-29T00:58:15.471 DEBUG:boto:auth_path=/hiiqos-64/?prefix=obj22 2023-07-29T00:58:15.471 DEBUG:boto:Method: GET 2023-07-29T00:58:15.472 DEBUG:boto:Path: /hiiqos-64/?prefix=obj22 2023-07-29T00:58:15.472 DEBUG:boto:Data: 2023-07-29T00:58:15.472 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.473 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:15.473 DEBUG:boto:Port: 8000 2023-07-29T00:58:15.474 DEBUG:boto:Params: {} 2023-07-29T00:58:15.474 DEBUG:boto:Token: None 2023-07-29T00:58:15.475 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/ 2023-07-29T00:58:15.476 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU= 2023-07-29T00:58:15.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU=', 'Content-Length': '0'} 2023-07-29T00:58:15.479 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029897b22dbc12f05-0064c46427-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.480 DEBUG:boto:b'hiiqos-64obj221000falseobj222023-07-29T00:57:44.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:58:15.480 DEBUG:boto:path=/hiiqos-64/obj22 2023-07-29T00:58:15.481 DEBUG:boto:auth_path=/hiiqos-64/obj22 2023-07-29T00:58:15.481 DEBUG:boto:Method: HEAD 2023-07-29T00:58:15.481 DEBUG:boto:Path: /hiiqos-64/obj22 2023-07-29T00:58:15.482 DEBUG:boto:Data: 2023-07-29T00:58:15.482 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.482 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:58:15.483 DEBUG:boto:Port: 8001 2023-07-29T00:58:15.483 DEBUG:boto:Params: {} 2023-07-29T00:58:15.483 DEBUG:boto:Token: None 2023-07-29T00:58:15.484 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj22 2023-07-29T00:58:15.484 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9oEB6tVZd9hKASyKVyRGrr8bEVg= 2023-07-29T00:58:15.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9oEB6tVZd9hKASyKVyRGrr8bEVg=', 'Content-Length': '0'} 2023-07-29T00:58:15.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054d229d9902bc2f8-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.487 DEBUG:boto:path=/hiiqos-64/obj22 2023-07-29T00:58:15.488 DEBUG:boto:auth_path=/hiiqos-64/obj22 2023-07-29T00:58:15.488 DEBUG:boto:Method: HEAD 2023-07-29T00:58:15.488 DEBUG:boto:Path: /hiiqos-64/obj22 2023-07-29T00:58:15.489 DEBUG:boto:Data: 2023-07-29T00:58:15.489 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.489 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:15.490 DEBUG:boto:Port: 8000 2023-07-29T00:58:15.490 DEBUG:boto:Params: {} 2023-07-29T00:58:15.490 DEBUG:boto:Token: None 2023-07-29T00:58:15.491 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj22 2023-07-29T00:58:15.491 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9oEB6tVZd9hKASyKVyRGrr8bEVg= 2023-07-29T00:58:15.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9oEB6tVZd9hKASyKVyRGrr8bEVg=', 'Content-Length': '0'} 2023-07-29T00:58:15.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b65c7d1b0ac3c30c-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-07-29T00:58:15.494 DEBUG:boto:path=/hiiqos-64/obj22 2023-07-29T00:58:15.495 DEBUG:boto:auth_path=/hiiqos-64/obj22 2023-07-29T00:58:15.495 DEBUG:boto:Method: GET 2023-07-29T00:58:15.496 DEBUG:boto:Path: /hiiqos-64/obj22 2023-07-29T00:58:15.496 DEBUG:boto:Data: 2023-07-29T00:58:15.496 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.497 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:58:15.497 DEBUG:boto:Port: 8001 2023-07-29T00:58:15.497 DEBUG:boto:Params: {} 2023-07-29T00:58:15.498 DEBUG:boto:Token: None 2023-07-29T00:58:15.498 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj22 2023-07-29T00:58:15.499 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:de4C/37PRLn6jRnBQQ8A/Kn77rE= 2023-07-29T00:58:15.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:de4C/37PRLn6jRnBQQ8A/Kn77rE=', 'Content-Length': '0'} 2023-07-29T00:58:15.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebfb7102e186f80f-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.547 DEBUG:boto:path=/hiiqos-64/obj22 2023-07-29T00:58:15.548 DEBUG:boto:auth_path=/hiiqos-64/obj22 2023-07-29T00:58:15.548 DEBUG:boto:Method: GET 2023-07-29T00:58:15.549 DEBUG:boto:Path: /hiiqos-64/obj22 2023-07-29T00:58:15.549 DEBUG:boto:Data: 2023-07-29T00:58:15.550 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.551 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:15.552 DEBUG:boto:Port: 8000 2023-07-29T00:58:15.552 DEBUG:boto:Params: {} 2023-07-29T00:58:15.553 DEBUG:boto:Token: None 2023-07-29T00:58:15.554 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj22 2023-07-29T00:58:15.555 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:de4C/37PRLn6jRnBQQ8A/Kn77rE= 2023-07-29T00:58:15.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:de4C/37PRLn6jRnBQQ8A/Kn77rE=', 'Content-Length': '0'} 2023-07-29T00:58:15.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b99c45709eda0e2-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.560 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:15.560 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:15.560 DEBUG:boto:path=/hiiqos-64/?prefix=obj22 2023-07-29T00:58:15.561 DEBUG:boto:auth_path=/hiiqos-64/?prefix=obj22 2023-07-29T00:58:15.561 DEBUG:boto:Method: GET 2023-07-29T00:58:15.562 DEBUG:boto:Path: /hiiqos-64/?prefix=obj22 2023-07-29T00:58:15.562 DEBUG:boto:Data: 2023-07-29T00:58:15.563 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.563 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:58:15.564 DEBUG:boto:Port: 8001 2023-07-29T00:58:15.565 DEBUG:boto:Params: {} 2023-07-29T00:58:15.566 DEBUG:boto:Token: None 2023-07-29T00:58:15.567 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/ 2023-07-29T00:58:15.568 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU= 2023-07-29T00:58:15.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU=', 'Content-Length': '0'} 2023-07-29T00:58:15.572 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053fe0d05c81b327c-0064c46427-2068-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.572 DEBUG:boto:b'hiiqos-64obj221000falseobj222023-07-29T00:57:44.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:58:15.573 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:15.574 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:15.574 DEBUG:boto:path=/hiiqos-64/?prefix=obj22 2023-07-29T00:58:15.574 DEBUG:boto:auth_path=/hiiqos-64/?prefix=obj22 2023-07-29T00:58:15.575 DEBUG:boto:Method: GET 2023-07-29T00:58:15.575 DEBUG:boto:Path: /hiiqos-64/?prefix=obj22 2023-07-29T00:58:15.576 DEBUG:boto:Data: 2023-07-29T00:58:15.576 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.577 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:15.577 DEBUG:boto:Port: 8000 2023-07-29T00:58:15.577 DEBUG:boto:Params: {} 2023-07-29T00:58:15.578 DEBUG:boto:Token: None 2023-07-29T00:58:15.578 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/ 2023-07-29T00:58:15.579 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU= 2023-07-29T00:58:15.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU=', 'Content-Length': '0'} 2023-07-29T00:58:15.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d28115dbe7834e01-0064c46427-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.582 DEBUG:boto:b'hiiqos-64obj221000falseobj222023-07-29T00:57:44.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:58:15.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-07-29T00:58:15.583 DEBUG:boto:path=/hiiqos-64/obj33 2023-07-29T00:58:15.584 DEBUG:boto:auth_path=/hiiqos-64/obj33 2023-07-29T00:58:15.584 DEBUG:boto:path=/hiiqos-64/obj33?versionId=null 2023-07-29T00:58:15.585 DEBUG:boto:auth_path=/hiiqos-64/obj33?versionId=null 2023-07-29T00:58:15.585 DEBUG:boto:Method: GET 2023-07-29T00:58:15.585 DEBUG:boto:Path: /hiiqos-64/obj33?versionId=null 2023-07-29T00:58:15.586 DEBUG:boto:Data: 2023-07-29T00:58:15.586 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.586 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:58:15.587 DEBUG:boto:Port: 8001 2023-07-29T00:58:15.587 DEBUG:boto:Params: {} 2023-07-29T00:58:15.587 DEBUG:boto:Token: None 2023-07-29T00:58:15.588 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj33?versionId=null 2023-07-29T00:58:15.588 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:98vId0VCVxPhdyj1T0JLJHIQvgc= 2023-07-29T00:58:15.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:98vId0VCVxPhdyj1T0JLJHIQvgc=', 'Content-Length': '0'} 2023-07-29T00:58:15.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1f2e34aba87d866-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.591 DEBUG:boto:path=/hiiqos-64/obj33 2023-07-29T00:58:15.592 DEBUG:boto:auth_path=/hiiqos-64/obj33 2023-07-29T00:58:15.592 DEBUG:boto:path=/hiiqos-64/obj33?versionId=null 2023-07-29T00:58:15.592 DEBUG:boto:auth_path=/hiiqos-64/obj33?versionId=null 2023-07-29T00:58:15.593 DEBUG:boto:Method: GET 2023-07-29T00:58:15.593 DEBUG:boto:Path: /hiiqos-64/obj33?versionId=null 2023-07-29T00:58:15.594 DEBUG:boto:Data: 2023-07-29T00:58:15.594 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.594 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:15.595 DEBUG:boto:Port: 8000 2023-07-29T00:58:15.595 DEBUG:boto:Params: {} 2023-07-29T00:58:15.595 DEBUG:boto:Token: None 2023-07-29T00:58:15.596 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj33?versionId=null 2023-07-29T00:58:15.596 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:98vId0VCVxPhdyj1T0JLJHIQvgc= 2023-07-29T00:58:15.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:98vId0VCVxPhdyj1T0JLJHIQvgc=', 'Content-Length': '0'} 2023-07-29T00:58:15.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081cc019de9ce60b4-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.600 DEBUG:boto:path=/hiiqos-64/obj33 2023-07-29T00:58:15.600 DEBUG:boto:auth_path=/hiiqos-64/obj33 2023-07-29T00:58:15.600 DEBUG:boto:path=/hiiqos-64/obj33?versionId=null 2023-07-29T00:58:15.601 DEBUG:boto:auth_path=/hiiqos-64/obj33?versionId=null 2023-07-29T00:58:15.601 DEBUG:boto:Method: HEAD 2023-07-29T00:58:15.601 DEBUG:boto:Path: /hiiqos-64/obj33?versionId=null 2023-07-29T00:58:15.602 DEBUG:boto:Data: 2023-07-29T00:58:15.602 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.602 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:58:15.603 DEBUG:boto:Port: 8001 2023-07-29T00:58:15.603 DEBUG:boto:Params: {} 2023-07-29T00:58:15.603 DEBUG:boto:Token: None 2023-07-29T00:58:15.604 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj33?versionId=null 2023-07-29T00:58:15.604 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rKXpeKBN4BlFWZ/78Y+K2c1eyT0= 2023-07-29T00:58:15.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rKXpeKBN4BlFWZ/78Y+K2c1eyT0=', 'Content-Length': '0'} 2023-07-29T00:58:15.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c26dcd1663b2e247-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.607 DEBUG:boto:path=/hiiqos-64/obj33 2023-07-29T00:58:15.608 DEBUG:boto:auth_path=/hiiqos-64/obj33 2023-07-29T00:58:15.608 DEBUG:boto:path=/hiiqos-64/obj33?versionId=null 2023-07-29T00:58:15.608 DEBUG:boto:auth_path=/hiiqos-64/obj33?versionId=null 2023-07-29T00:58:15.609 DEBUG:boto:Method: HEAD 2023-07-29T00:58:15.609 DEBUG:boto:Path: /hiiqos-64/obj33?versionId=null 2023-07-29T00:58:15.609 DEBUG:boto:Data: 2023-07-29T00:58:15.610 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.610 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:15.610 DEBUG:boto:Port: 8000 2023-07-29T00:58:15.611 DEBUG:boto:Params: {} 2023-07-29T00:58:15.611 DEBUG:boto:Token: None 2023-07-29T00:58:15.611 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj33?versionId=null 2023-07-29T00:58:15.612 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rKXpeKBN4BlFWZ/78Y+K2c1eyT0= 2023-07-29T00:58:15.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rKXpeKBN4BlFWZ/78Y+K2c1eyT0=', 'Content-Length': '0'} 2023-07-29T00:58:15.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec75d978a85d4b03-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-07-29T00:58:15.615 DEBUG:boto:path=/hiiqos-64/obj33 2023-07-29T00:58:15.615 DEBUG:boto:auth_path=/hiiqos-64/obj33 2023-07-29T00:58:15.616 DEBUG:boto:path=/hiiqos-64/obj33?versionId=null 2023-07-29T00:58:15.616 DEBUG:boto:auth_path=/hiiqos-64/obj33?versionId=null 2023-07-29T00:58:15.617 DEBUG:boto:Method: GET 2023-07-29T00:58:15.617 DEBUG:boto:Path: /hiiqos-64/obj33?versionId=null 2023-07-29T00:58:15.618 DEBUG:boto:Data: 2023-07-29T00:58:15.618 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.618 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:58:15.619 DEBUG:boto:Port: 8001 2023-07-29T00:58:15.619 DEBUG:boto:Params: {} 2023-07-29T00:58:15.619 DEBUG:boto:Token: None 2023-07-29T00:58:15.620 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj33?versionId=null 2023-07-29T00:58:15.620 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:98vId0VCVxPhdyj1T0JLJHIQvgc= 2023-07-29T00:58:15.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:98vId0VCVxPhdyj1T0JLJHIQvgc=', 'Content-Length': '0'} 2023-07-29T00:58:15.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006532b30c32489583-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.625 DEBUG:boto:path=/hiiqos-64/obj33 2023-07-29T00:58:15.625 DEBUG:boto:auth_path=/hiiqos-64/obj33 2023-07-29T00:58:15.626 DEBUG:boto:path=/hiiqos-64/obj33?versionId=null 2023-07-29T00:58:15.626 DEBUG:boto:auth_path=/hiiqos-64/obj33?versionId=null 2023-07-29T00:58:15.626 DEBUG:boto:Method: GET 2023-07-29T00:58:15.627 DEBUG:boto:Path: /hiiqos-64/obj33?versionId=null 2023-07-29T00:58:15.627 DEBUG:boto:Data: 2023-07-29T00:58:15.628 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.628 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:15.628 DEBUG:boto:Port: 8000 2023-07-29T00:58:15.629 DEBUG:boto:Params: {} 2023-07-29T00:58:15.629 DEBUG:boto:Token: None 2023-07-29T00:58:15.629 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj33?versionId=null 2023-07-29T00:58:15.630 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:98vId0VCVxPhdyj1T0JLJHIQvgc= 2023-07-29T00:58:15.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:98vId0VCVxPhdyj1T0JLJHIQvgc=', 'Content-Length': '0'} 2023-07-29T00:58:15.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000329de7456efee6c-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.633 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:15.634 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:15.634 DEBUG:boto:path=/hiiqos-64/?prefix=obj33 2023-07-29T00:58:15.635 DEBUG:boto:auth_path=/hiiqos-64/?prefix=obj33 2023-07-29T00:58:15.635 DEBUG:boto:Method: GET 2023-07-29T00:58:15.635 DEBUG:boto:Path: /hiiqos-64/?prefix=obj33 2023-07-29T00:58:15.636 DEBUG:boto:Data: 2023-07-29T00:58:15.636 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.636 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:58:15.637 DEBUG:boto:Port: 8001 2023-07-29T00:58:15.637 DEBUG:boto:Params: {} 2023-07-29T00:58:15.638 DEBUG:boto:Token: None 2023-07-29T00:58:15.638 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/ 2023-07-29T00:58:15.639 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU= 2023-07-29T00:58:15.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU=', 'Content-Length': '0'} 2023-07-29T00:58:15.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004dec7ea68b2d2627-0064c46427-2068-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.642 DEBUG:boto:b'hiiqos-64obj331000falseobj332023-07-29T00:57:44.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:58:15.643 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:15.643 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:15.644 DEBUG:boto:path=/hiiqos-64/?prefix=obj33 2023-07-29T00:58:15.644 DEBUG:boto:auth_path=/hiiqos-64/?prefix=obj33 2023-07-29T00:58:15.644 DEBUG:boto:Method: GET 2023-07-29T00:58:15.645 DEBUG:boto:Path: /hiiqos-64/?prefix=obj33 2023-07-29T00:58:15.645 DEBUG:boto:Data: 2023-07-29T00:58:15.646 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.646 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:15.646 DEBUG:boto:Port: 8000 2023-07-29T00:58:15.647 DEBUG:boto:Params: {} 2023-07-29T00:58:15.647 DEBUG:boto:Token: None 2023-07-29T00:58:15.647 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/ 2023-07-29T00:58:15.648 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU= 2023-07-29T00:58:15.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU=', 'Content-Length': '0'} 2023-07-29T00:58:15.651 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098b9fce2d22daad1-0064c46427-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.692 DEBUG:boto:b'hiiqos-64obj331000falseobj332023-07-29T00:57:44.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:58:15.693 DEBUG:boto:path=/hiiqos-64/obj33 2023-07-29T00:58:15.693 DEBUG:boto:auth_path=/hiiqos-64/obj33 2023-07-29T00:58:15.694 DEBUG:boto:Method: HEAD 2023-07-29T00:58:15.694 DEBUG:boto:Path: /hiiqos-64/obj33 2023-07-29T00:58:15.695 DEBUG:boto:Data: 2023-07-29T00:58:15.695 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.695 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:58:15.696 DEBUG:boto:Port: 8001 2023-07-29T00:58:15.696 DEBUG:boto:Params: {} 2023-07-29T00:58:15.696 DEBUG:boto:Token: None 2023-07-29T00:58:15.697 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj33 2023-07-29T00:58:15.697 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Giw8a+QOERvRUc02958X1eZ2PX4= 2023-07-29T00:58:15.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Giw8a+QOERvRUc02958X1eZ2PX4=', 'Content-Length': '0'} 2023-07-29T00:58:15.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd7ee6e536d8fe67-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.701 DEBUG:boto:path=/hiiqos-64/obj33 2023-07-29T00:58:15.701 DEBUG:boto:auth_path=/hiiqos-64/obj33 2023-07-29T00:58:15.701 DEBUG:boto:Method: HEAD 2023-07-29T00:58:15.702 DEBUG:boto:Path: /hiiqos-64/obj33 2023-07-29T00:58:15.702 DEBUG:boto:Data: 2023-07-29T00:58:15.703 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.703 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:15.703 DEBUG:boto:Port: 8000 2023-07-29T00:58:15.704 DEBUG:boto:Params: {} 2023-07-29T00:58:15.704 DEBUG:boto:Token: None 2023-07-29T00:58:15.704 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj33 2023-07-29T00:58:15.705 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Giw8a+QOERvRUc02958X1eZ2PX4= 2023-07-29T00:58:15.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Giw8a+QOERvRUc02958X1eZ2PX4=', 'Content-Length': '0'} 2023-07-29T00:58:15.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009a3748a6de2db12-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-07-29T00:58:15.708 DEBUG:boto:path=/hiiqos-64/obj33 2023-07-29T00:58:15.708 DEBUG:boto:auth_path=/hiiqos-64/obj33 2023-07-29T00:58:15.709 DEBUG:boto:Method: GET 2023-07-29T00:58:15.709 DEBUG:boto:Path: /hiiqos-64/obj33 2023-07-29T00:58:15.709 DEBUG:boto:Data: 2023-07-29T00:58:15.710 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.710 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:58:15.711 DEBUG:boto:Port: 8001 2023-07-29T00:58:15.711 DEBUG:boto:Params: {} 2023-07-29T00:58:15.712 DEBUG:boto:Token: None 2023-07-29T00:58:15.712 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj33 2023-07-29T00:58:15.713 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Q1P0LI9D6/vNWj/Q0w/jRhPBzTo= 2023-07-29T00:58:15.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q1P0LI9D6/vNWj/Q0w/jRhPBzTo=', 'Content-Length': '0'} 2023-07-29T00:58:15.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d871261a775e374-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.716 DEBUG:boto:path=/hiiqos-64/obj33 2023-07-29T00:58:15.716 DEBUG:boto:auth_path=/hiiqos-64/obj33 2023-07-29T00:58:15.716 DEBUG:boto:Method: GET 2023-07-29T00:58:15.717 DEBUG:boto:Path: /hiiqos-64/obj33 2023-07-29T00:58:15.717 DEBUG:boto:Data: 2023-07-29T00:58:15.718 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.718 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:15.719 DEBUG:boto:Port: 8000 2023-07-29T00:58:15.720 DEBUG:boto:Params: {} 2023-07-29T00:58:15.721 DEBUG:boto:Token: None 2023-07-29T00:58:15.722 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj33 2023-07-29T00:58:15.723 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Q1P0LI9D6/vNWj/Q0w/jRhPBzTo= 2023-07-29T00:58:15.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q1P0LI9D6/vNWj/Q0w/jRhPBzTo=', 'Content-Length': '0'} 2023-07-29T00:58:15.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008aa6b863648e9106-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.727 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:15.728 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:15.729 DEBUG:boto:path=/hiiqos-64/?prefix=obj33 2023-07-29T00:58:15.730 DEBUG:boto:auth_path=/hiiqos-64/?prefix=obj33 2023-07-29T00:58:15.730 DEBUG:boto:Method: GET 2023-07-29T00:58:15.730 DEBUG:boto:Path: /hiiqos-64/?prefix=obj33 2023-07-29T00:58:15.731 DEBUG:boto:Data: 2023-07-29T00:58:15.731 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.731 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:58:15.732 DEBUG:boto:Port: 8001 2023-07-29T00:58:15.732 DEBUG:boto:Params: {} 2023-07-29T00:58:15.733 DEBUG:boto:Token: None 2023-07-29T00:58:15.733 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/ 2023-07-29T00:58:15.734 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU= 2023-07-29T00:58:15.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU=', 'Content-Length': '0'} 2023-07-29T00:58:15.737 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c185c4a3644d971b-0064c46427-2068-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.737 DEBUG:boto:b'hiiqos-64obj331000falseobj332023-07-29T00:57:44.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:58:15.738 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:15.738 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:15.739 DEBUG:boto:path=/hiiqos-64/?prefix=obj33 2023-07-29T00:58:15.739 DEBUG:boto:auth_path=/hiiqos-64/?prefix=obj33 2023-07-29T00:58:15.739 DEBUG:boto:Method: GET 2023-07-29T00:58:15.740 DEBUG:boto:Path: /hiiqos-64/?prefix=obj33 2023-07-29T00:58:15.740 DEBUG:boto:Data: 2023-07-29T00:58:15.741 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.741 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:15.741 DEBUG:boto:Port: 8000 2023-07-29T00:58:15.742 DEBUG:boto:Params: {} 2023-07-29T00:58:15.742 DEBUG:boto:Token: None 2023-07-29T00:58:15.743 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/ 2023-07-29T00:58:15.743 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU= 2023-07-29T00:58:15.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU=', 'Content-Length': '0'} 2023-07-29T00:58:15.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6346cbb7ecf2f60-0064c46427-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.747 DEBUG:boto:b'hiiqos-64obj331000falseobj332023-07-29T00:57:44.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:58:15.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-07-29T00:58:15.748 DEBUG:boto:path=/hiiqos-64/obj44 2023-07-29T00:58:15.748 DEBUG:boto:auth_path=/hiiqos-64/obj44 2023-07-29T00:58:15.749 DEBUG:boto:path=/hiiqos-64/obj44?versionId=null 2023-07-29T00:58:15.749 DEBUG:boto:auth_path=/hiiqos-64/obj44?versionId=null 2023-07-29T00:58:15.749 DEBUG:boto:Method: GET 2023-07-29T00:58:15.750 DEBUG:boto:Path: /hiiqos-64/obj44?versionId=null 2023-07-29T00:58:15.750 DEBUG:boto:Data: 2023-07-29T00:58:15.751 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.751 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:58:15.751 DEBUG:boto:Port: 8001 2023-07-29T00:58:15.752 DEBUG:boto:Params: {} 2023-07-29T00:58:15.752 DEBUG:boto:Token: None 2023-07-29T00:58:15.753 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj44?versionId=null 2023-07-29T00:58:15.753 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eZlcpkHP4tYuMnSlNLK1W6c7aGw= 2023-07-29T00:58:15.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eZlcpkHP4tYuMnSlNLK1W6c7aGw=', 'Content-Length': '0'} 2023-07-29T00:58:15.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee342e170eaede1e-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.796 DEBUG:boto:path=/hiiqos-64/obj44 2023-07-29T00:58:15.797 DEBUG:boto:auth_path=/hiiqos-64/obj44 2023-07-29T00:58:15.797 DEBUG:boto:path=/hiiqos-64/obj44?versionId=null 2023-07-29T00:58:15.798 DEBUG:boto:auth_path=/hiiqos-64/obj44?versionId=null 2023-07-29T00:58:15.798 DEBUG:boto:Method: GET 2023-07-29T00:58:15.798 DEBUG:boto:Path: /hiiqos-64/obj44?versionId=null 2023-07-29T00:58:15.799 DEBUG:boto:Data: 2023-07-29T00:58:15.799 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.800 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:15.800 DEBUG:boto:Port: 8000 2023-07-29T00:58:15.800 DEBUG:boto:Params: {} 2023-07-29T00:58:15.801 DEBUG:boto:Token: None 2023-07-29T00:58:15.801 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj44?versionId=null 2023-07-29T00:58:15.802 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eZlcpkHP4tYuMnSlNLK1W6c7aGw= 2023-07-29T00:58:15.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eZlcpkHP4tYuMnSlNLK1W6c7aGw=', 'Content-Length': '0'} 2023-07-29T00:58:15.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d515e633cd249217-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.805 DEBUG:boto:path=/hiiqos-64/obj44 2023-07-29T00:58:15.806 DEBUG:boto:auth_path=/hiiqos-64/obj44 2023-07-29T00:58:15.807 DEBUG:boto:path=/hiiqos-64/obj44?versionId=null 2023-07-29T00:58:15.807 DEBUG:boto:auth_path=/hiiqos-64/obj44?versionId=null 2023-07-29T00:58:15.808 DEBUG:boto:Method: HEAD 2023-07-29T00:58:15.809 DEBUG:boto:Path: /hiiqos-64/obj44?versionId=null 2023-07-29T00:58:15.810 DEBUG:boto:Data: 2023-07-29T00:58:15.811 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.813 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:58:15.813 DEBUG:boto:Port: 8001 2023-07-29T00:58:15.814 DEBUG:boto:Params: {} 2023-07-29T00:58:15.815 DEBUG:boto:Token: None 2023-07-29T00:58:15.816 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj44?versionId=null 2023-07-29T00:58:15.817 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:joK7DBT9QGpmoeowI/S8mICmzd0= 2023-07-29T00:58:15.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:joK7DBT9QGpmoeowI/S8mICmzd0=', 'Content-Length': '0'} 2023-07-29T00:58:15.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b55f4e9232c47514-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.821 DEBUG:boto:path=/hiiqos-64/obj44 2023-07-29T00:58:15.822 DEBUG:boto:auth_path=/hiiqos-64/obj44 2023-07-29T00:58:15.823 DEBUG:boto:path=/hiiqos-64/obj44?versionId=null 2023-07-29T00:58:15.823 DEBUG:boto:auth_path=/hiiqos-64/obj44?versionId=null 2023-07-29T00:58:15.824 DEBUG:boto:Method: HEAD 2023-07-29T00:58:15.825 DEBUG:boto:Path: /hiiqos-64/obj44?versionId=null 2023-07-29T00:58:15.825 DEBUG:boto:Data: 2023-07-29T00:58:15.826 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.827 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:15.827 DEBUG:boto:Port: 8000 2023-07-29T00:58:15.828 DEBUG:boto:Params: {} 2023-07-29T00:58:15.829 DEBUG:boto:Token: None 2023-07-29T00:58:15.830 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj44?versionId=null 2023-07-29T00:58:15.830 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:joK7DBT9QGpmoeowI/S8mICmzd0= 2023-07-29T00:58:15.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:joK7DBT9QGpmoeowI/S8mICmzd0=', 'Content-Length': '0'} 2023-07-29T00:58:15.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052779aa98ab8ccae-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-07-29T00:58:15.836 DEBUG:boto:path=/hiiqos-64/obj44 2023-07-29T00:58:15.837 DEBUG:boto:auth_path=/hiiqos-64/obj44 2023-07-29T00:58:15.838 DEBUG:boto:path=/hiiqos-64/obj44?versionId=null 2023-07-29T00:58:15.839 DEBUG:boto:auth_path=/hiiqos-64/obj44?versionId=null 2023-07-29T00:58:15.839 DEBUG:boto:Method: GET 2023-07-29T00:58:15.840 DEBUG:boto:Path: /hiiqos-64/obj44?versionId=null 2023-07-29T00:58:15.840 DEBUG:boto:Data: 2023-07-29T00:58:15.841 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.841 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:58:15.841 DEBUG:boto:Port: 8001 2023-07-29T00:58:15.842 DEBUG:boto:Params: {} 2023-07-29T00:58:15.842 DEBUG:boto:Token: None 2023-07-29T00:58:15.843 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj44?versionId=null 2023-07-29T00:58:15.843 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eZlcpkHP4tYuMnSlNLK1W6c7aGw= 2023-07-29T00:58:15.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eZlcpkHP4tYuMnSlNLK1W6c7aGw=', 'Content-Length': '0'} 2023-07-29T00:58:15.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a63a8421fea0e09-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.846 DEBUG:boto:path=/hiiqos-64/obj44 2023-07-29T00:58:15.847 DEBUG:boto:auth_path=/hiiqos-64/obj44 2023-07-29T00:58:15.847 DEBUG:boto:path=/hiiqos-64/obj44?versionId=null 2023-07-29T00:58:15.848 DEBUG:boto:auth_path=/hiiqos-64/obj44?versionId=null 2023-07-29T00:58:15.848 DEBUG:boto:Method: GET 2023-07-29T00:58:15.849 DEBUG:boto:Path: /hiiqos-64/obj44?versionId=null 2023-07-29T00:58:15.850 DEBUG:boto:Data: 2023-07-29T00:58:15.851 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.851 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:15.852 DEBUG:boto:Port: 8000 2023-07-29T00:58:15.853 DEBUG:boto:Params: {} 2023-07-29T00:58:15.854 DEBUG:boto:Token: None 2023-07-29T00:58:15.855 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj44?versionId=null 2023-07-29T00:58:15.856 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:eZlcpkHP4tYuMnSlNLK1W6c7aGw= 2023-07-29T00:58:15.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eZlcpkHP4tYuMnSlNLK1W6c7aGw=', 'Content-Length': '0'} 2023-07-29T00:58:15.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f86644e622e92c5-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.860 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:15.860 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:15.861 DEBUG:boto:path=/hiiqos-64/?prefix=obj44 2023-07-29T00:58:15.861 DEBUG:boto:auth_path=/hiiqos-64/?prefix=obj44 2023-07-29T00:58:15.862 DEBUG:boto:Method: GET 2023-07-29T00:58:15.862 DEBUG:boto:Path: /hiiqos-64/?prefix=obj44 2023-07-29T00:58:15.862 DEBUG:boto:Data: 2023-07-29T00:58:15.863 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.863 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:58:15.863 DEBUG:boto:Port: 8001 2023-07-29T00:58:15.864 DEBUG:boto:Params: {} 2023-07-29T00:58:15.864 DEBUG:boto:Token: None 2023-07-29T00:58:15.865 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/ 2023-07-29T00:58:15.865 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU= 2023-07-29T00:58:15.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU=', 'Content-Length': '0'} 2023-07-29T00:58:15.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002eabddfdf047d897-0064c46427-2068-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.868 DEBUG:boto:b'hiiqos-64obj441000falseobj442023-07-29T00:57:44.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:58:15.869 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:15.869 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:15.870 DEBUG:boto:path=/hiiqos-64/?prefix=obj44 2023-07-29T00:58:15.870 DEBUG:boto:auth_path=/hiiqos-64/?prefix=obj44 2023-07-29T00:58:15.870 DEBUG:boto:Method: GET 2023-07-29T00:58:15.871 DEBUG:boto:Path: /hiiqos-64/?prefix=obj44 2023-07-29T00:58:15.871 DEBUG:boto:Data: 2023-07-29T00:58:15.871 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.872 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:15.872 DEBUG:boto:Port: 8000 2023-07-29T00:58:15.873 DEBUG:boto:Params: {} 2023-07-29T00:58:15.873 DEBUG:boto:Token: None 2023-07-29T00:58:15.873 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/ 2023-07-29T00:58:15.874 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU= 2023-07-29T00:58:15.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU=', 'Content-Length': '0'} 2023-07-29T00:58:15.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ccb7eb5f4aa21cb-0064c46427-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.877 DEBUG:boto:b'hiiqos-64obj441000falseobj442023-07-29T00:57:44.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:58:15.878 DEBUG:boto:path=/hiiqos-64/obj44 2023-07-29T00:58:15.879 DEBUG:boto:auth_path=/hiiqos-64/obj44 2023-07-29T00:58:15.879 DEBUG:boto:Method: HEAD 2023-07-29T00:58:15.879 DEBUG:boto:Path: /hiiqos-64/obj44 2023-07-29T00:58:15.880 DEBUG:boto:Data: 2023-07-29T00:58:15.880 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.880 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:58:15.881 DEBUG:boto:Port: 8001 2023-07-29T00:58:15.881 DEBUG:boto:Params: {} 2023-07-29T00:58:15.881 DEBUG:boto:Token: None 2023-07-29T00:58:15.882 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj44 2023-07-29T00:58:15.882 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bhtA/bVfmGvDybE6jG9HOdc8ckU= 2023-07-29T00:58:15.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bhtA/bVfmGvDybE6jG9HOdc8ckU=', 'Content-Length': '0'} 2023-07-29T00:58:15.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cc5eebc5a62399f-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.885 DEBUG:boto:path=/hiiqos-64/obj44 2023-07-29T00:58:15.886 DEBUG:boto:auth_path=/hiiqos-64/obj44 2023-07-29T00:58:15.886 DEBUG:boto:Method: HEAD 2023-07-29T00:58:15.886 DEBUG:boto:Path: /hiiqos-64/obj44 2023-07-29T00:58:15.887 DEBUG:boto:Data: 2023-07-29T00:58:15.887 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.887 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:15.888 DEBUG:boto:Port: 8000 2023-07-29T00:58:15.888 DEBUG:boto:Params: {} 2023-07-29T00:58:15.889 DEBUG:boto:Token: None 2023-07-29T00:58:15.889 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj44 2023-07-29T00:58:15.889 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bhtA/bVfmGvDybE6jG9HOdc8ckU= 2023-07-29T00:58:15.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bhtA/bVfmGvDybE6jG9HOdc8ckU=', 'Content-Length': '0'} 2023-07-29T00:58:15.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd0e1fdc6066e0f1-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-07-29T00:58:15.893 DEBUG:boto:path=/hiiqos-64/obj44 2023-07-29T00:58:15.893 DEBUG:boto:auth_path=/hiiqos-64/obj44 2023-07-29T00:58:15.894 DEBUG:boto:Method: GET 2023-07-29T00:58:15.894 DEBUG:boto:Path: /hiiqos-64/obj44 2023-07-29T00:58:15.894 DEBUG:boto:Data: 2023-07-29T00:58:15.895 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.895 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:58:15.896 DEBUG:boto:Port: 8001 2023-07-29T00:58:15.896 DEBUG:boto:Params: {} 2023-07-29T00:58:15.897 DEBUG:boto:Token: None 2023-07-29T00:58:15.897 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj44 2023-07-29T00:58:15.898 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4rm99mIbKH5BAR7UyBF8agtr4zQ= 2023-07-29T00:58:15.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4rm99mIbKH5BAR7UyBF8agtr4zQ=', 'Content-Length': '0'} 2023-07-29T00:58:15.900 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b5319676c06b2b0-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.944 DEBUG:boto:path=/hiiqos-64/obj44 2023-07-29T00:58:15.945 DEBUG:boto:auth_path=/hiiqos-64/obj44 2023-07-29T00:58:15.946 DEBUG:boto:Method: GET 2023-07-29T00:58:15.947 DEBUG:boto:Path: /hiiqos-64/obj44 2023-07-29T00:58:15.948 DEBUG:boto:Data: 2023-07-29T00:58:15.948 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.949 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:15.949 DEBUG:boto:Port: 8000 2023-07-29T00:58:15.949 DEBUG:boto:Params: {} 2023-07-29T00:58:15.950 DEBUG:boto:Token: None 2023-07-29T00:58:15.951 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj44 2023-07-29T00:58:15.951 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4rm99mIbKH5BAR7UyBF8agtr4zQ= 2023-07-29T00:58:15.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4rm99mIbKH5BAR7UyBF8agtr4zQ=', 'Content-Length': '0'} 2023-07-29T00:58:15.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001beddfddcc51ae78-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.954 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:15.955 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:15.955 DEBUG:boto:path=/hiiqos-64/?prefix=obj44 2023-07-29T00:58:15.955 DEBUG:boto:auth_path=/hiiqos-64/?prefix=obj44 2023-07-29T00:58:15.956 DEBUG:boto:Method: GET 2023-07-29T00:58:15.956 DEBUG:boto:Path: /hiiqos-64/?prefix=obj44 2023-07-29T00:58:15.957 DEBUG:boto:Data: 2023-07-29T00:58:15.957 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.957 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:58:15.958 DEBUG:boto:Port: 8001 2023-07-29T00:58:15.958 DEBUG:boto:Params: {} 2023-07-29T00:58:15.958 DEBUG:boto:Token: None 2023-07-29T00:58:15.959 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/ 2023-07-29T00:58:15.959 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU= 2023-07-29T00:58:15.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU=', 'Content-Length': '0'} 2023-07-29T00:58:15.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbfaeff3bf570d22-0064c46427-2068-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.963 DEBUG:boto:b'hiiqos-64obj441000falseobj442023-07-29T00:57:44.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:58:15.964 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:15.964 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:15.965 DEBUG:boto:path=/hiiqos-64/?prefix=obj44 2023-07-29T00:58:15.965 DEBUG:boto:auth_path=/hiiqos-64/?prefix=obj44 2023-07-29T00:58:15.965 DEBUG:boto:Method: GET 2023-07-29T00:58:15.966 DEBUG:boto:Path: /hiiqos-64/?prefix=obj44 2023-07-29T00:58:15.966 DEBUG:boto:Data: 2023-07-29T00:58:15.967 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.967 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:15.967 DEBUG:boto:Port: 8000 2023-07-29T00:58:15.968 DEBUG:boto:Params: {} 2023-07-29T00:58:15.968 DEBUG:boto:Token: None 2023-07-29T00:58:15.968 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/ 2023-07-29T00:58:15.969 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU= 2023-07-29T00:58:15.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU=', 'Content-Length': '0'} 2023-07-29T00:58:15.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f29e8a5742a51f4-0064c46427-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.973 DEBUG:boto:b'hiiqos-64obj441000falseobj442023-07-29T00:57:44.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:58:15.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-07-29T00:58:15.974 DEBUG:boto:path=/hiiqos-64/obj55 2023-07-29T00:58:15.974 DEBUG:boto:auth_path=/hiiqos-64/obj55 2023-07-29T00:58:15.974 DEBUG:boto:path=/hiiqos-64/obj55?versionId=null 2023-07-29T00:58:15.975 DEBUG:boto:auth_path=/hiiqos-64/obj55?versionId=null 2023-07-29T00:58:15.975 DEBUG:boto:Method: GET 2023-07-29T00:58:15.975 DEBUG:boto:Path: /hiiqos-64/obj55?versionId=null 2023-07-29T00:58:15.976 DEBUG:boto:Data: 2023-07-29T00:58:15.976 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.977 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:58:15.977 DEBUG:boto:Port: 8001 2023-07-29T00:58:15.977 DEBUG:boto:Params: {} 2023-07-29T00:58:15.978 DEBUG:boto:Token: None 2023-07-29T00:58:15.978 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj55?versionId=null 2023-07-29T00:58:15.978 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Rv51aK3Fw8oxOxbfoDeK2A00grA= 2023-07-29T00:58:15.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Rv51aK3Fw8oxOxbfoDeK2A00grA=', 'Content-Length': '0'} 2023-07-29T00:58:15.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e15294ade3f77be8-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.981 DEBUG:boto:path=/hiiqos-64/obj55 2023-07-29T00:58:15.982 DEBUG:boto:auth_path=/hiiqos-64/obj55 2023-07-29T00:58:15.982 DEBUG:boto:path=/hiiqos-64/obj55?versionId=null 2023-07-29T00:58:15.982 DEBUG:boto:auth_path=/hiiqos-64/obj55?versionId=null 2023-07-29T00:58:15.983 DEBUG:boto:Method: GET 2023-07-29T00:58:15.983 DEBUG:boto:Path: /hiiqos-64/obj55?versionId=null 2023-07-29T00:58:15.984 DEBUG:boto:Data: 2023-07-29T00:58:15.984 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.984 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:15.985 DEBUG:boto:Port: 8000 2023-07-29T00:58:15.985 DEBUG:boto:Params: {} 2023-07-29T00:58:15.986 DEBUG:boto:Token: None 2023-07-29T00:58:15.986 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj55?versionId=null 2023-07-29T00:58:15.986 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Rv51aK3Fw8oxOxbfoDeK2A00grA= 2023-07-29T00:58:15.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Rv51aK3Fw8oxOxbfoDeK2A00grA=', 'Content-Length': '0'} 2023-07-29T00:58:15.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a64cf36af2d0d5b3-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.989 DEBUG:boto:path=/hiiqos-64/obj55 2023-07-29T00:58:15.990 DEBUG:boto:auth_path=/hiiqos-64/obj55 2023-07-29T00:58:15.990 DEBUG:boto:path=/hiiqos-64/obj55?versionId=null 2023-07-29T00:58:15.991 DEBUG:boto:auth_path=/hiiqos-64/obj55?versionId=null 2023-07-29T00:58:15.991 DEBUG:boto:Method: HEAD 2023-07-29T00:58:15.991 DEBUG:boto:Path: /hiiqos-64/obj55?versionId=null 2023-07-29T00:58:15.992 DEBUG:boto:Data: 2023-07-29T00:58:15.992 DEBUG:boto:Headers: {} 2023-07-29T00:58:15.992 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:58:15.993 DEBUG:boto:Port: 8001 2023-07-29T00:58:15.993 DEBUG:boto:Params: {} 2023-07-29T00:58:15.994 DEBUG:boto:Token: None 2023-07-29T00:58:15.994 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:15 GMT /hiiqos-64/obj55?versionId=null 2023-07-29T00:58:15.994 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:labkhHXzNZFlaEsk2bPbC78lbk0= 2023-07-29T00:58:15.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:labkhHXzNZFlaEsk2bPbC78lbk0=', 'Content-Length': '0'} 2023-07-29T00:58:15.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e60fca35950d9215-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:15.997 DEBUG:boto:path=/hiiqos-64/obj55 2023-07-29T00:58:15.997 DEBUG:boto:auth_path=/hiiqos-64/obj55 2023-07-29T00:58:15.998 DEBUG:boto:path=/hiiqos-64/obj55?versionId=null 2023-07-29T00:58:15.998 DEBUG:boto:auth_path=/hiiqos-64/obj55?versionId=null 2023-07-29T00:58:15.998 DEBUG:boto:Method: HEAD 2023-07-29T00:58:15.999 DEBUG:boto:Path: /hiiqos-64/obj55?versionId=null 2023-07-29T00:58:15.999 DEBUG:boto:Data: 2023-07-29T00:58:16.000 DEBUG:boto:Headers: {} 2023-07-29T00:58:16.000 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:16.000 DEBUG:boto:Port: 8000 2023-07-29T00:58:16.001 DEBUG:boto:Params: {} 2023-07-29T00:58:16.001 DEBUG:boto:Token: None 2023-07-29T00:58:16.001 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:16 GMT /hiiqos-64/obj55?versionId=null 2023-07-29T00:58:16.002 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:b8iBa8YetOPg6zrV7fvouc7nXkA= 2023-07-29T00:58:16.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:b8iBa8YetOPg6zrV7fvouc7nXkA=', 'Content-Length': '0'} 2023-07-29T00:58:16.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000271eb0a08341db51-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:16.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-07-29T00:58:16.005 DEBUG:boto:path=/hiiqos-64/obj55 2023-07-29T00:58:16.005 DEBUG:boto:auth_path=/hiiqos-64/obj55 2023-07-29T00:58:16.006 DEBUG:boto:path=/hiiqos-64/obj55?versionId=null 2023-07-29T00:58:16.006 DEBUG:boto:auth_path=/hiiqos-64/obj55?versionId=null 2023-07-29T00:58:16.007 DEBUG:boto:Method: GET 2023-07-29T00:58:16.007 DEBUG:boto:Path: /hiiqos-64/obj55?versionId=null 2023-07-29T00:58:16.007 DEBUG:boto:Data: 2023-07-29T00:58:16.008 DEBUG:boto:Headers: {} 2023-07-29T00:58:16.008 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:58:16.008 DEBUG:boto:Port: 8001 2023-07-29T00:58:16.009 DEBUG:boto:Params: {} 2023-07-29T00:58:16.009 DEBUG:boto:Token: None 2023-07-29T00:58:16.010 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:16 GMT /hiiqos-64/obj55?versionId=null 2023-07-29T00:58:16.010 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w9m+PMoRaOk3/jG4Z85BxF6kmiA= 2023-07-29T00:58:16.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w9m+PMoRaOk3/jG4Z85BxF6kmiA=', 'Content-Length': '0'} 2023-07-29T00:58:16.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf78df1a331ddec0-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:16.056 DEBUG:boto:path=/hiiqos-64/obj55 2023-07-29T00:58:16.057 DEBUG:boto:auth_path=/hiiqos-64/obj55 2023-07-29T00:58:16.057 DEBUG:boto:path=/hiiqos-64/obj55?versionId=null 2023-07-29T00:58:16.057 DEBUG:boto:auth_path=/hiiqos-64/obj55?versionId=null 2023-07-29T00:58:16.058 DEBUG:boto:Method: GET 2023-07-29T00:58:16.058 DEBUG:boto:Path: /hiiqos-64/obj55?versionId=null 2023-07-29T00:58:16.058 DEBUG:boto:Data: 2023-07-29T00:58:16.059 DEBUG:boto:Headers: {} 2023-07-29T00:58:16.059 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:16.060 DEBUG:boto:Port: 8000 2023-07-29T00:58:16.060 DEBUG:boto:Params: {} 2023-07-29T00:58:16.060 DEBUG:boto:Token: None 2023-07-29T00:58:16.061 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:16 GMT /hiiqos-64/obj55?versionId=null 2023-07-29T00:58:16.061 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:w9m+PMoRaOk3/jG4Z85BxF6kmiA= 2023-07-29T00:58:16.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w9m+PMoRaOk3/jG4Z85BxF6kmiA=', 'Content-Length': '0'} 2023-07-29T00:58:16.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028145327fc6e2df8-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:16.065 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:16.065 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:16.065 DEBUG:boto:path=/hiiqos-64/?prefix=obj55 2023-07-29T00:58:16.066 DEBUG:boto:auth_path=/hiiqos-64/?prefix=obj55 2023-07-29T00:58:16.066 DEBUG:boto:Method: GET 2023-07-29T00:58:16.066 DEBUG:boto:Path: /hiiqos-64/?prefix=obj55 2023-07-29T00:58:16.067 DEBUG:boto:Data: 2023-07-29T00:58:16.067 DEBUG:boto:Headers: {} 2023-07-29T00:58:16.067 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:58:16.068 DEBUG:boto:Port: 8001 2023-07-29T00:58:16.068 DEBUG:boto:Params: {} 2023-07-29T00:58:16.069 DEBUG:boto:Token: None 2023-07-29T00:58:16.069 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:16 GMT /hiiqos-64/ 2023-07-29T00:58:16.069 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rHVKG8gNW6ZA3ljBfJt53iz2WJc= 2023-07-29T00:58:16.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rHVKG8gNW6ZA3ljBfJt53iz2WJc=', 'Content-Length': '0'} 2023-07-29T00:58:16.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058961bd4eba36c08-0064c46427-2068-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:16.072 DEBUG:boto:b'hiiqos-64obj551000falseobj552023-07-29T00:57:44.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:58:16.073 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:16.074 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:16.074 DEBUG:boto:path=/hiiqos-64/?prefix=obj55 2023-07-29T00:58:16.074 DEBUG:boto:auth_path=/hiiqos-64/?prefix=obj55 2023-07-29T00:58:16.075 DEBUG:boto:Method: GET 2023-07-29T00:58:16.075 DEBUG:boto:Path: /hiiqos-64/?prefix=obj55 2023-07-29T00:58:16.075 DEBUG:boto:Data: 2023-07-29T00:58:16.076 DEBUG:boto:Headers: {} 2023-07-29T00:58:16.076 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:16.077 DEBUG:boto:Port: 8000 2023-07-29T00:58:16.077 DEBUG:boto:Params: {} 2023-07-29T00:58:16.077 DEBUG:boto:Token: None 2023-07-29T00:58:16.078 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:16 GMT /hiiqos-64/ 2023-07-29T00:58:16.078 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rHVKG8gNW6ZA3ljBfJt53iz2WJc= 2023-07-29T00:58:16.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rHVKG8gNW6ZA3ljBfJt53iz2WJc=', 'Content-Length': '0'} 2023-07-29T00:58:16.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b361a1d2d021998-0064c46428-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:16.081 DEBUG:boto:b'hiiqos-64obj551000falseobj552023-07-29T00:57:44.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:58:16.082 DEBUG:boto:path=/hiiqos-64/obj55 2023-07-29T00:58:16.082 DEBUG:boto:auth_path=/hiiqos-64/obj55 2023-07-29T00:58:16.083 DEBUG:boto:Method: HEAD 2023-07-29T00:58:16.083 DEBUG:boto:Path: /hiiqos-64/obj55 2023-07-29T00:58:16.083 DEBUG:boto:Data: 2023-07-29T00:58:16.084 DEBUG:boto:Headers: {} 2023-07-29T00:58:16.085 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:58:16.085 DEBUG:boto:Port: 8001 2023-07-29T00:58:16.086 DEBUG:boto:Params: {} 2023-07-29T00:58:16.086 DEBUG:boto:Token: None 2023-07-29T00:58:16.086 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:16 GMT /hiiqos-64/obj55 2023-07-29T00:58:16.087 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fZ0+QhfGouvlodXzkysnYK9YlYI= 2023-07-29T00:58:16.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fZ0+QhfGouvlodXzkysnYK9YlYI=', 'Content-Length': '0'} 2023-07-29T00:58:16.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000449179b8c357e5c2-0064c46428-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:16.090 DEBUG:boto:path=/hiiqos-64/obj55 2023-07-29T00:58:16.090 DEBUG:boto:auth_path=/hiiqos-64/obj55 2023-07-29T00:58:16.090 DEBUG:boto:Method: HEAD 2023-07-29T00:58:16.091 DEBUG:boto:Path: /hiiqos-64/obj55 2023-07-29T00:58:16.091 DEBUG:boto:Data: 2023-07-29T00:58:16.092 DEBUG:boto:Headers: {} 2023-07-29T00:58:16.092 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:16.092 DEBUG:boto:Port: 8000 2023-07-29T00:58:16.093 DEBUG:boto:Params: {} 2023-07-29T00:58:16.093 DEBUG:boto:Token: None 2023-07-29T00:58:16.093 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:16 GMT /hiiqos-64/obj55 2023-07-29T00:58:16.094 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fZ0+QhfGouvlodXzkysnYK9YlYI= 2023-07-29T00:58:16.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fZ0+QhfGouvlodXzkysnYK9YlYI=', 'Content-Length': '0'} 2023-07-29T00:58:16.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b07df3e8f83b7e93-0064c46428-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:16.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-07-29T00:58:16.097 DEBUG:boto:path=/hiiqos-64/obj55 2023-07-29T00:58:16.098 DEBUG:boto:auth_path=/hiiqos-64/obj55 2023-07-29T00:58:16.098 DEBUG:boto:Method: GET 2023-07-29T00:58:16.098 DEBUG:boto:Path: /hiiqos-64/obj55 2023-07-29T00:58:16.099 DEBUG:boto:Data: 2023-07-29T00:58:16.099 DEBUG:boto:Headers: {} 2023-07-29T00:58:16.099 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:58:16.100 DEBUG:boto:Port: 8001 2023-07-29T00:58:16.100 DEBUG:boto:Params: {} 2023-07-29T00:58:16.101 DEBUG:boto:Token: None 2023-07-29T00:58:16.101 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:16 GMT /hiiqos-64/obj55 2023-07-29T00:58:16.101 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pbRFhGVXA+fSYmiWAXLRX7x3Te4= 2023-07-29T00:58:16.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pbRFhGVXA+fSYmiWAXLRX7x3Te4=', 'Content-Length': '0'} 2023-07-29T00:58:16.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000787a3d4960cec3-0064c46428-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:16.104 DEBUG:boto:path=/hiiqos-64/obj55 2023-07-29T00:58:16.105 DEBUG:boto:auth_path=/hiiqos-64/obj55 2023-07-29T00:58:16.105 DEBUG:boto:Method: GET 2023-07-29T00:58:16.106 DEBUG:boto:Path: /hiiqos-64/obj55 2023-07-29T00:58:16.106 DEBUG:boto:Data: 2023-07-29T00:58:16.106 DEBUG:boto:Headers: {} 2023-07-29T00:58:16.107 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:16.107 DEBUG:boto:Port: 8000 2023-07-29T00:58:16.107 DEBUG:boto:Params: {} 2023-07-29T00:58:16.108 DEBUG:boto:Token: None 2023-07-29T00:58:16.109 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:16 GMT /hiiqos-64/obj55 2023-07-29T00:58:16.109 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pbRFhGVXA+fSYmiWAXLRX7x3Te4= 2023-07-29T00:58:16.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pbRFhGVXA+fSYmiWAXLRX7x3Te4=', 'Content-Length': '0'} 2023-07-29T00:58:16.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e97a594b1912f24d-0064c46428-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:16.152 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:16.153 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:16.153 DEBUG:boto:path=/hiiqos-64/?prefix=obj55 2023-07-29T00:58:16.154 DEBUG:boto:auth_path=/hiiqos-64/?prefix=obj55 2023-07-29T00:58:16.154 DEBUG:boto:Method: GET 2023-07-29T00:58:16.154 DEBUG:boto:Path: /hiiqos-64/?prefix=obj55 2023-07-29T00:58:16.155 DEBUG:boto:Data: 2023-07-29T00:58:16.155 DEBUG:boto:Headers: {} 2023-07-29T00:58:16.155 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T00:58:16.156 DEBUG:boto:Port: 8001 2023-07-29T00:58:16.156 DEBUG:boto:Params: {} 2023-07-29T00:58:16.157 DEBUG:boto:Token: None 2023-07-29T00:58:16.157 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:16 GMT /hiiqos-64/ 2023-07-29T00:58:16.157 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rHVKG8gNW6ZA3ljBfJt53iz2WJc= 2023-07-29T00:58:16.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rHVKG8gNW6ZA3ljBfJt53iz2WJc=', 'Content-Length': '0'} 2023-07-29T00:58:16.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a014e2f8f633424-0064c46428-2068-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:16.162 DEBUG:boto:b'hiiqos-64obj551000falseobj552023-07-29T00:57:44.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:58:16.162 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:16.163 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:16.163 DEBUG:boto:path=/hiiqos-64/?prefix=obj55 2023-07-29T00:58:16.164 DEBUG:boto:auth_path=/hiiqos-64/?prefix=obj55 2023-07-29T00:58:16.164 DEBUG:boto:Method: GET 2023-07-29T00:58:16.165 DEBUG:boto:Path: /hiiqos-64/?prefix=obj55 2023-07-29T00:58:16.165 DEBUG:boto:Data: 2023-07-29T00:58:16.165 DEBUG:boto:Headers: {} 2023-07-29T00:58:16.166 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:16.167 DEBUG:boto:Port: 8000 2023-07-29T00:58:16.167 DEBUG:boto:Params: {} 2023-07-29T00:58:16.168 DEBUG:boto:Token: None 2023-07-29T00:58:16.168 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 00:58:16 GMT /hiiqos-64/ 2023-07-29T00:58:16.169 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rHVKG8gNW6ZA3ljBfJt53iz2WJc= 2023-07-29T00:58:16.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rHVKG8gNW6ZA3ljBfJt53iz2WJc=', 'Content-Length': '0'} 2023-07-29T00:58:16.172 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086afef46f64ed538-0064c46428-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:16.173 DEBUG:boto:b'hiiqos-64obj551000falseobj552023-07-29T00:57:44.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T00:58:16.174 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hiiqos-64 zones={a2, a1} 2023-07-29T00:58:16.174 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-07-29T00:58:16.175 INFO:teuthology.orchestra.run:waiting for 300 2023-07-29T00:58:16.175 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:58:16.097+0000 7f2e5bb77640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 145401) UID: 0 2023-07-29T00:58:16.176 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T00:58:16.097+0000 7f2e61524880 -1 shutting down 2023-07-29T00:58:18.021 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-07-29T00:58:22.049 INFO:tasks.rgw.c2.client.0.smithi139.stdout:2023-07-29T00:58:21.969+0000 7f5f2671b640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T00:58:22.273 INFO:tasks.rgw.c1.client.1:Stopped 2023-07-29T00:58:22.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-64', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:58:22.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-64', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:58:22.276 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-64 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:58:22.333 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:58:22.334 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:58:22.379 INFO:teuthology.orchestra.run.smithi101.stdout:tenant: 2023-07-29T00:58:22.380 INFO:teuthology.orchestra.run.smithi101.stdout:bucket name: hiiqos-64 2023-07-29T00:58:22.384 INFO:teuthology.orchestra.run.smithi101.stdout:total entries: 5 2023-07-29T00:58:22.425 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:22.426 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:22.427 DEBUG:boto:Method: HEAD 2023-07-29T00:58:22.428 DEBUG:boto:Path: /hiiqos-64/ 2023-07-29T00:58:22.428 DEBUG:boto:Data: 2023-07-29T00:58:22.429 DEBUG:boto:Headers: {} 2023-07-29T00:58:22.430 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:22.431 DEBUG:boto:Port: 8000 2023-07-29T00:58:22.432 DEBUG:boto:Params: {} 2023-07-29T00:58:22.433 DEBUG:boto:Token: None 2023-07-29T00:58:22.435 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:22 GMT /hiiqos-64/ 2023-07-29T00:58:22.435 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:viGS3AtqUOvuzqqrwcqs6CUp3eE= 2023-07-29T00:58:22.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:viGS3AtqUOvuzqqrwcqs6CUp3eE=', 'Content-Length': '0'} 2023-07-29T00:58:22.439 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000040bb43c709d1b584-0064c4642e-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:22.439 DEBUG:boto:path=/hiiqos-64/obj17 2023-07-29T00:58:22.440 DEBUG:boto:auth_path=/hiiqos-64/obj17 2023-07-29T00:58:22.440 DEBUG:boto:Method: PUT 2023-07-29T00:58:22.441 DEBUG:boto:Path: /hiiqos-64/obj17 2023-07-29T00:58:22.441 DEBUG:boto:Data: 2023-07-29T00:58:22.441 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:58:22.442 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:22.442 DEBUG:boto:Port: 8000 2023-07-29T00:58:22.442 DEBUG:boto:Params: {} 2023-07-29T00:58:22.443 DEBUG:boto:Token: None 2023-07-29T00:58:22.443 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:58:22 GMT /hiiqos-64/obj17 2023-07-29T00:58:22.444 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:W/xWD3XWVoAlkSY++0mf6/umbdw= 2023-07-29T00:58:22.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:W/xWD3XWVoAlkSY++0mf6/umbdw='} 2023-07-29T00:58:22.450 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009fa9b7f28032b2e9-0064c4642e-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:22.450 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:22.451 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:22.451 DEBUG:boto:Method: HEAD 2023-07-29T00:58:22.451 DEBUG:boto:Path: /hiiqos-64/ 2023-07-29T00:58:22.452 DEBUG:boto:Data: 2023-07-29T00:58:22.452 DEBUG:boto:Headers: {} 2023-07-29T00:58:22.452 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:22.453 DEBUG:boto:Port: 8000 2023-07-29T00:58:22.453 DEBUG:boto:Params: {} 2023-07-29T00:58:22.453 DEBUG:boto:Token: None 2023-07-29T00:58:22.454 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:22 GMT /hiiqos-64/ 2023-07-29T00:58:22.454 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:viGS3AtqUOvuzqqrwcqs6CUp3eE= 2023-07-29T00:58:22.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:viGS3AtqUOvuzqqrwcqs6CUp3eE=', 'Content-Length': '0'} 2023-07-29T00:58:22.457 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cf631aa8294d11c6-0064c4642e-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:22.458 DEBUG:boto:path=/hiiqos-64/obj34 2023-07-29T00:58:22.458 DEBUG:boto:auth_path=/hiiqos-64/obj34 2023-07-29T00:58:22.458 DEBUG:boto:Method: PUT 2023-07-29T00:58:22.459 DEBUG:boto:Path: /hiiqos-64/obj34 2023-07-29T00:58:22.459 DEBUG:boto:Data: 2023-07-29T00:58:22.459 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:58:22.460 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:22.460 DEBUG:boto:Port: 8000 2023-07-29T00:58:22.460 DEBUG:boto:Params: {} 2023-07-29T00:58:22.461 DEBUG:boto:Token: None 2023-07-29T00:58:22.461 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:58:22 GMT /hiiqos-64/obj34 2023-07-29T00:58:22.461 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:V4R47cW6lqgWVopUio+u0R34IMI= 2023-07-29T00:58:22.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:V4R47cW6lqgWVopUio+u0R34IMI='} 2023-07-29T00:58:22.504 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005db99e6913b92dbb-0064c4642e-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:22.505 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:22.505 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:22.506 DEBUG:boto:Method: HEAD 2023-07-29T00:58:22.506 DEBUG:boto:Path: /hiiqos-64/ 2023-07-29T00:58:22.506 DEBUG:boto:Data: 2023-07-29T00:58:22.507 DEBUG:boto:Headers: {} 2023-07-29T00:58:22.507 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:22.507 DEBUG:boto:Port: 8000 2023-07-29T00:58:22.508 DEBUG:boto:Params: {} 2023-07-29T00:58:22.508 DEBUG:boto:Token: None 2023-07-29T00:58:22.508 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:22 GMT /hiiqos-64/ 2023-07-29T00:58:22.509 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:viGS3AtqUOvuzqqrwcqs6CUp3eE= 2023-07-29T00:58:22.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:viGS3AtqUOvuzqqrwcqs6CUp3eE=', 'Content-Length': '0'} 2023-07-29T00:58:22.511 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ed024b9caf61c9c-0064c4642e-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:22.512 DEBUG:boto:path=/hiiqos-64/obj51 2023-07-29T00:58:22.512 DEBUG:boto:auth_path=/hiiqos-64/obj51 2023-07-29T00:58:22.513 DEBUG:boto:Method: PUT 2023-07-29T00:58:22.513 DEBUG:boto:Path: /hiiqos-64/obj51 2023-07-29T00:58:22.514 DEBUG:boto:Data: 2023-07-29T00:58:22.514 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:58:22.514 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:22.515 DEBUG:boto:Port: 8000 2023-07-29T00:58:22.515 DEBUG:boto:Params: {} 2023-07-29T00:58:22.515 DEBUG:boto:Token: None 2023-07-29T00:58:22.516 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:58:22 GMT /hiiqos-64/obj51 2023-07-29T00:58:22.516 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:n1DRIfknmEiJem+yZSBoyswxdk4= 2023-07-29T00:58:22.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:n1DRIfknmEiJem+yZSBoyswxdk4='} 2023-07-29T00:58:22.522 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e2bf925e934a6589-0064c4642e-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:22.522 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:22.523 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:22.523 DEBUG:boto:Method: HEAD 2023-07-29T00:58:22.523 DEBUG:boto:Path: /hiiqos-64/ 2023-07-29T00:58:22.524 DEBUG:boto:Data: 2023-07-29T00:58:22.524 DEBUG:boto:Headers: {} 2023-07-29T00:58:22.524 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:22.525 DEBUG:boto:Port: 8000 2023-07-29T00:58:22.525 DEBUG:boto:Params: {} 2023-07-29T00:58:22.525 DEBUG:boto:Token: None 2023-07-29T00:58:22.526 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:22 GMT /hiiqos-64/ 2023-07-29T00:58:22.526 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:viGS3AtqUOvuzqqrwcqs6CUp3eE= 2023-07-29T00:58:22.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:viGS3AtqUOvuzqqrwcqs6CUp3eE=', 'Content-Length': '0'} 2023-07-29T00:58:22.529 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008bc598a6043d6ce5-0064c4642e-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:22.529 DEBUG:boto:path=/hiiqos-64/obj68 2023-07-29T00:58:22.530 DEBUG:boto:auth_path=/hiiqos-64/obj68 2023-07-29T00:58:22.530 DEBUG:boto:Method: PUT 2023-07-29T00:58:22.530 DEBUG:boto:Path: /hiiqos-64/obj68 2023-07-29T00:58:22.531 DEBUG:boto:Data: 2023-07-29T00:58:22.531 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:58:22.531 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:22.532 DEBUG:boto:Port: 8000 2023-07-29T00:58:22.532 DEBUG:boto:Params: {} 2023-07-29T00:58:22.532 DEBUG:boto:Token: None 2023-07-29T00:58:22.533 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:58:22 GMT /hiiqos-64/obj68 2023-07-29T00:58:22.533 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TMJ7K8KV+po0mbn+pbo/DaSbDsA= 2023-07-29T00:58:22.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TMJ7K8KV+po0mbn+pbo/DaSbDsA='} 2023-07-29T00:58:22.576 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dfba0626dcef323f-0064c4642e-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:22.577 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:22.578 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:22.578 DEBUG:boto:Method: HEAD 2023-07-29T00:58:22.578 DEBUG:boto:Path: /hiiqos-64/ 2023-07-29T00:58:22.579 DEBUG:boto:Data: 2023-07-29T00:58:22.579 DEBUG:boto:Headers: {} 2023-07-29T00:58:22.579 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:22.580 DEBUG:boto:Port: 8000 2023-07-29T00:58:22.580 DEBUG:boto:Params: {} 2023-07-29T00:58:22.580 DEBUG:boto:Token: None 2023-07-29T00:58:22.581 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:22 GMT /hiiqos-64/ 2023-07-29T00:58:22.581 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:viGS3AtqUOvuzqqrwcqs6CUp3eE= 2023-07-29T00:58:22.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:viGS3AtqUOvuzqqrwcqs6CUp3eE=', 'Content-Length': '0'} 2023-07-29T00:58:22.584 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f8dcbd8be4d13fdf-0064c4642e-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:22.585 DEBUG:boto:path=/hiiqos-64/obj85 2023-07-29T00:58:22.585 DEBUG:boto:auth_path=/hiiqos-64/obj85 2023-07-29T00:58:22.585 DEBUG:boto:Method: PUT 2023-07-29T00:58:22.586 DEBUG:boto:Path: /hiiqos-64/obj85 2023-07-29T00:58:22.586 DEBUG:boto:Data: 2023-07-29T00:58:22.586 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:58:22.587 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:22.587 DEBUG:boto:Port: 8000 2023-07-29T00:58:22.587 DEBUG:boto:Params: {} 2023-07-29T00:58:22.588 DEBUG:boto:Token: None 2023-07-29T00:58:22.588 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:58:22 GMT /hiiqos-64/obj85 2023-07-29T00:58:22.589 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:t8Z1mEsEbUKr3SAhBcm+AoGnfBU= 2023-07-29T00:58:22.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t8Z1mEsEbUKr3SAhBcm+AoGnfBU='} 2023-07-29T00:58:22.594 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000005389d987b02de96-0064c4642e-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:22.595 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-64', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:58:22.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-64', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:58:22.595 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-64 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:58:22.645 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:58:22.645 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:58:22.687 INFO:teuthology.orchestra.run.smithi101.stdout:tenant: 2023-07-29T00:58:22.687 INFO:teuthology.orchestra.run.smithi101.stdout:bucket name: hiiqos-64 2023-07-29T00:58:22.693 INFO:teuthology.orchestra.run.smithi101.stdout:total entries: 10 2023-07-29T00:58:22.735 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:22.736 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:22.736 DEBUG:boto:Method: HEAD 2023-07-29T00:58:22.737 DEBUG:boto:Path: /hiiqos-64/ 2023-07-29T00:58:22.737 DEBUG:boto:Data: 2023-07-29T00:58:22.737 DEBUG:boto:Headers: {} 2023-07-29T00:58:22.737 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:22.738 DEBUG:boto:Port: 8000 2023-07-29T00:58:22.738 DEBUG:boto:Params: {} 2023-07-29T00:58:22.739 DEBUG:boto:Token: None 2023-07-29T00:58:22.739 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:22 GMT /hiiqos-64/ 2023-07-29T00:58:22.739 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:viGS3AtqUOvuzqqrwcqs6CUp3eE= 2023-07-29T00:58:22.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:viGS3AtqUOvuzqqrwcqs6CUp3eE=', 'Content-Length': '0'} 2023-07-29T00:58:22.743 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d57fe3862868c237-0064c4642e-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:22.743 DEBUG:boto:path=/hiiqos-64/obj19 2023-07-29T00:58:22.744 DEBUG:boto:auth_path=/hiiqos-64/obj19 2023-07-29T00:58:22.744 DEBUG:boto:Method: PUT 2023-07-29T00:58:22.744 DEBUG:boto:Path: /hiiqos-64/obj19 2023-07-29T00:58:22.745 DEBUG:boto:Data: 2023-07-29T00:58:22.745 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:58:22.745 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:22.746 DEBUG:boto:Port: 8000 2023-07-29T00:58:22.746 DEBUG:boto:Params: {} 2023-07-29T00:58:22.746 DEBUG:boto:Token: None 2023-07-29T00:58:22.747 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:58:22 GMT /hiiqos-64/obj19 2023-07-29T00:58:22.747 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rUt6UCIxlrKz4QZn2V3QPPv7loc= 2023-07-29T00:58:22.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rUt6UCIxlrKz4QZn2V3QPPv7loc='} 2023-07-29T00:58:22.753 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001b8946c3b4e51e8d-0064c4642e-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:22.753 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:22.754 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:22.754 DEBUG:boto:Method: HEAD 2023-07-29T00:58:22.754 DEBUG:boto:Path: /hiiqos-64/ 2023-07-29T00:58:22.755 DEBUG:boto:Data: 2023-07-29T00:58:22.755 DEBUG:boto:Headers: {} 2023-07-29T00:58:22.755 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:22.756 DEBUG:boto:Port: 8000 2023-07-29T00:58:22.756 DEBUG:boto:Params: {} 2023-07-29T00:58:22.756 DEBUG:boto:Token: None 2023-07-29T00:58:22.757 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:22 GMT /hiiqos-64/ 2023-07-29T00:58:22.757 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:viGS3AtqUOvuzqqrwcqs6CUp3eE= 2023-07-29T00:58:22.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:viGS3AtqUOvuzqqrwcqs6CUp3eE=', 'Content-Length': '0'} 2023-07-29T00:58:22.760 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b237c8948fc6bc6b-0064c4642e-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:22.761 DEBUG:boto:path=/hiiqos-64/obj38 2023-07-29T00:58:22.761 DEBUG:boto:auth_path=/hiiqos-64/obj38 2023-07-29T00:58:22.761 DEBUG:boto:Method: PUT 2023-07-29T00:58:22.762 DEBUG:boto:Path: /hiiqos-64/obj38 2023-07-29T00:58:22.762 DEBUG:boto:Data: 2023-07-29T00:58:22.762 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:58:22.763 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:22.763 DEBUG:boto:Port: 8000 2023-07-29T00:58:22.763 DEBUG:boto:Params: {} 2023-07-29T00:58:22.764 DEBUG:boto:Token: None 2023-07-29T00:58:22.764 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:58:22 GMT /hiiqos-64/obj38 2023-07-29T00:58:22.765 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gyhcN0+TBy/uimINIG1NNxLHxk8= 2023-07-29T00:58:22.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gyhcN0+TBy/uimINIG1NNxLHxk8='} 2023-07-29T00:58:22.769 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005fbad17e4535b2db-0064c4642e-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:22.769 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:22.770 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:22.770 DEBUG:boto:Method: HEAD 2023-07-29T00:58:22.770 DEBUG:boto:Path: /hiiqos-64/ 2023-07-29T00:58:22.771 DEBUG:boto:Data: 2023-07-29T00:58:22.771 DEBUG:boto:Headers: {} 2023-07-29T00:58:22.771 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:22.772 DEBUG:boto:Port: 8000 2023-07-29T00:58:22.772 DEBUG:boto:Params: {} 2023-07-29T00:58:22.772 DEBUG:boto:Token: None 2023-07-29T00:58:22.773 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:22 GMT /hiiqos-64/ 2023-07-29T00:58:22.773 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:viGS3AtqUOvuzqqrwcqs6CUp3eE= 2023-07-29T00:58:22.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:viGS3AtqUOvuzqqrwcqs6CUp3eE=', 'Content-Length': '0'} 2023-07-29T00:58:22.776 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f116d5b65ff9f3af-0064c4642e-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:22.776 DEBUG:boto:path=/hiiqos-64/obj57 2023-07-29T00:58:22.777 DEBUG:boto:auth_path=/hiiqos-64/obj57 2023-07-29T00:58:22.777 DEBUG:boto:Method: PUT 2023-07-29T00:58:22.777 DEBUG:boto:Path: /hiiqos-64/obj57 2023-07-29T00:58:22.778 DEBUG:boto:Data: 2023-07-29T00:58:22.778 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:58:22.778 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:22.779 DEBUG:boto:Port: 8000 2023-07-29T00:58:22.779 DEBUG:boto:Params: {} 2023-07-29T00:58:22.779 DEBUG:boto:Token: None 2023-07-29T00:58:22.780 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:58:22 GMT /hiiqos-64/obj57 2023-07-29T00:58:22.780 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8kUys6wYE/CA4JIZRs/80Qv97Po= 2023-07-29T00:58:22.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8kUys6wYE/CA4JIZRs/80Qv97Po='} 2023-07-29T00:58:22.828 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003015312dbe4b3444-0064c4642e-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:22.829 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:22.830 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:22.830 DEBUG:boto:Method: HEAD 2023-07-29T00:58:22.830 DEBUG:boto:Path: /hiiqos-64/ 2023-07-29T00:58:22.831 DEBUG:boto:Data: 2023-07-29T00:58:22.831 DEBUG:boto:Headers: {} 2023-07-29T00:58:22.831 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:22.832 DEBUG:boto:Port: 8000 2023-07-29T00:58:22.832 DEBUG:boto:Params: {} 2023-07-29T00:58:22.832 DEBUG:boto:Token: None 2023-07-29T00:58:22.833 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:22 GMT /hiiqos-64/ 2023-07-29T00:58:22.833 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:viGS3AtqUOvuzqqrwcqs6CUp3eE= 2023-07-29T00:58:22.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:viGS3AtqUOvuzqqrwcqs6CUp3eE=', 'Content-Length': '0'} 2023-07-29T00:58:22.836 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c2907760aadc3b74-0064c4642e-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:22.837 DEBUG:boto:path=/hiiqos-64/obj76 2023-07-29T00:58:22.838 DEBUG:boto:auth_path=/hiiqos-64/obj76 2023-07-29T00:58:22.838 DEBUG:boto:Method: PUT 2023-07-29T00:58:22.838 DEBUG:boto:Path: /hiiqos-64/obj76 2023-07-29T00:58:22.839 DEBUG:boto:Data: 2023-07-29T00:58:22.839 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:58:22.839 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:22.840 DEBUG:boto:Port: 8000 2023-07-29T00:58:22.840 DEBUG:boto:Params: {} 2023-07-29T00:58:22.841 DEBUG:boto:Token: None 2023-07-29T00:58:22.841 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:58:22 GMT /hiiqos-64/obj76 2023-07-29T00:58:22.841 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kmHC/RENItvlH+6U0bdM0rU2U8I= 2023-07-29T00:58:22.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kmHC/RENItvlH+6U0bdM0rU2U8I='} 2023-07-29T00:58:22.847 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000087ff0937d1726794-0064c4642e-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:22.848 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:22.848 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:22.849 DEBUG:boto:Method: HEAD 2023-07-29T00:58:22.849 DEBUG:boto:Path: /hiiqos-64/ 2023-07-29T00:58:22.849 DEBUG:boto:Data: 2023-07-29T00:58:22.850 DEBUG:boto:Headers: {} 2023-07-29T00:58:22.850 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:22.850 DEBUG:boto:Port: 8000 2023-07-29T00:58:22.851 DEBUG:boto:Params: {} 2023-07-29T00:58:22.851 DEBUG:boto:Token: None 2023-07-29T00:58:22.851 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:22 GMT /hiiqos-64/ 2023-07-29T00:58:22.852 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:viGS3AtqUOvuzqqrwcqs6CUp3eE= 2023-07-29T00:58:22.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:viGS3AtqUOvuzqqrwcqs6CUp3eE=', 'Content-Length': '0'} 2023-07-29T00:58:22.855 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000094cd01086acc0a21-0064c4642e-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:22.855 DEBUG:boto:path=/hiiqos-64/obj95 2023-07-29T00:58:22.856 DEBUG:boto:auth_path=/hiiqos-64/obj95 2023-07-29T00:58:22.856 DEBUG:boto:Method: PUT 2023-07-29T00:58:22.856 DEBUG:boto:Path: /hiiqos-64/obj95 2023-07-29T00:58:22.857 DEBUG:boto:Data: 2023-07-29T00:58:22.857 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:58:22.857 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:22.858 DEBUG:boto:Port: 8000 2023-07-29T00:58:22.858 DEBUG:boto:Params: {} 2023-07-29T00:58:22.858 DEBUG:boto:Token: None 2023-07-29T00:58:22.859 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:58:22 GMT /hiiqos-64/obj95 2023-07-29T00:58:22.859 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bjdmDoPjNNTRuwu4fyXCybAwOaQ= 2023-07-29T00:58:22.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bjdmDoPjNNTRuwu4fyXCybAwOaQ='} 2023-07-29T00:58:22.863 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d3c13157921b737c-0064c4642e-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:22.864 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-64', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:58:22.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-64', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:58:22.865 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-64 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:58:22.913 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:58:22.914 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:58:22.956 INFO:teuthology.orchestra.run.smithi101.stdout:tenant: 2023-07-29T00:58:22.956 INFO:teuthology.orchestra.run.smithi101.stdout:bucket name: hiiqos-64 2023-07-29T00:58:22.961 INFO:teuthology.orchestra.run.smithi101.stdout:total entries: 15 2023-07-29T00:58:23.007 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:23.008 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:23.008 DEBUG:boto:Method: HEAD 2023-07-29T00:58:23.008 DEBUG:boto:Path: /hiiqos-64/ 2023-07-29T00:58:23.009 DEBUG:boto:Data: 2023-07-29T00:58:23.009 DEBUG:boto:Headers: {} 2023-07-29T00:58:23.009 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:23.010 DEBUG:boto:Port: 8000 2023-07-29T00:58:23.010 DEBUG:boto:Params: {} 2023-07-29T00:58:23.010 DEBUG:boto:Token: None 2023-07-29T00:58:23.011 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:23 GMT /hiiqos-64/ 2023-07-29T00:58:23.011 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY= 2023-07-29T00:58:23.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY=', 'Content-Length': '0'} 2023-07-29T00:58:23.015 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fb951661839030e0-0064c4642e-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:23.015 DEBUG:boto:path=/hiiqos-64/obj23 2023-07-29T00:58:23.016 DEBUG:boto:auth_path=/hiiqos-64/obj23 2023-07-29T00:58:23.017 DEBUG:boto:Method: PUT 2023-07-29T00:58:23.017 DEBUG:boto:Path: /hiiqos-64/obj23 2023-07-29T00:58:23.017 DEBUG:boto:Data: 2023-07-29T00:58:23.018 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:58:23.018 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:23.018 DEBUG:boto:Port: 8000 2023-07-29T00:58:23.019 DEBUG:boto:Params: {} 2023-07-29T00:58:23.019 DEBUG:boto:Token: None 2023-07-29T00:58:23.020 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:58:23 GMT /hiiqos-64/obj23 2023-07-29T00:58:23.020 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EvCk7cAWZ8zDYz4uFP7DBNStlWM= 2023-07-29T00:58:23.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EvCk7cAWZ8zDYz4uFP7DBNStlWM='} 2023-07-29T00:58:23.026 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000075dd1f3338e42087-0064c4642e-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:23.026 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:23.026 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:23.027 DEBUG:boto:Method: HEAD 2023-07-29T00:58:23.027 DEBUG:boto:Path: /hiiqos-64/ 2023-07-29T00:58:23.027 DEBUG:boto:Data: 2023-07-29T00:58:23.028 DEBUG:boto:Headers: {} 2023-07-29T00:58:23.028 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:23.028 DEBUG:boto:Port: 8000 2023-07-29T00:58:23.029 DEBUG:boto:Params: {} 2023-07-29T00:58:23.029 DEBUG:boto:Token: None 2023-07-29T00:58:23.030 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:23 GMT /hiiqos-64/ 2023-07-29T00:58:23.030 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY= 2023-07-29T00:58:23.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY=', 'Content-Length': '0'} 2023-07-29T00:58:23.033 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000071b2878f38fe6ea-0064c4642e-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:23.033 DEBUG:boto:path=/hiiqos-64/obj46 2023-07-29T00:58:23.034 DEBUG:boto:auth_path=/hiiqos-64/obj46 2023-07-29T00:58:23.034 DEBUG:boto:Method: PUT 2023-07-29T00:58:23.035 DEBUG:boto:Path: /hiiqos-64/obj46 2023-07-29T00:58:23.035 DEBUG:boto:Data: 2023-07-29T00:58:23.035 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:58:23.036 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:23.036 DEBUG:boto:Port: 8000 2023-07-29T00:58:23.036 DEBUG:boto:Params: {} 2023-07-29T00:58:23.037 DEBUG:boto:Token: None 2023-07-29T00:58:23.037 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:58:23 GMT /hiiqos-64/obj46 2023-07-29T00:58:23.038 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fIHtw415vulG6mkYz3py2ENacQM= 2023-07-29T00:58:23.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fIHtw415vulG6mkYz3py2ENacQM='} 2023-07-29T00:58:23.084 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009b1cd9d619d22627-0064c4642e-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:23.085 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:23.086 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:23.087 DEBUG:boto:Method: HEAD 2023-07-29T00:58:23.087 DEBUG:boto:Path: /hiiqos-64/ 2023-07-29T00:58:23.087 DEBUG:boto:Data: 2023-07-29T00:58:23.088 DEBUG:boto:Headers: {} 2023-07-29T00:58:23.088 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:23.088 DEBUG:boto:Port: 8000 2023-07-29T00:58:23.089 DEBUG:boto:Params: {} 2023-07-29T00:58:23.089 DEBUG:boto:Token: None 2023-07-29T00:58:23.090 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:23 GMT /hiiqos-64/ 2023-07-29T00:58:23.090 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY= 2023-07-29T00:58:23.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY=', 'Content-Length': '0'} 2023-07-29T00:58:23.093 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ce2130b191864bbb-0064c4642f-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:23.094 DEBUG:boto:path=/hiiqos-64/obj69 2023-07-29T00:58:23.094 DEBUG:boto:auth_path=/hiiqos-64/obj69 2023-07-29T00:58:23.094 DEBUG:boto:Method: PUT 2023-07-29T00:58:23.095 DEBUG:boto:Path: /hiiqos-64/obj69 2023-07-29T00:58:23.095 DEBUG:boto:Data: 2023-07-29T00:58:23.096 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:58:23.096 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:23.096 DEBUG:boto:Port: 8000 2023-07-29T00:58:23.097 DEBUG:boto:Params: {} 2023-07-29T00:58:23.097 DEBUG:boto:Token: None 2023-07-29T00:58:23.097 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:58:23 GMT /hiiqos-64/obj69 2023-07-29T00:58:23.098 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2ZXUZSJXvoXKjA4Zy/7ehKQmbh8= 2023-07-29T00:58:23.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2ZXUZSJXvoXKjA4Zy/7ehKQmbh8='} 2023-07-29T00:58:23.103 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000071168e943b7caf05-0064c4642f-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:23.103 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:23.104 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:23.104 DEBUG:boto:Method: HEAD 2023-07-29T00:58:23.105 DEBUG:boto:Path: /hiiqos-64/ 2023-07-29T00:58:23.105 DEBUG:boto:Data: 2023-07-29T00:58:23.105 DEBUG:boto:Headers: {} 2023-07-29T00:58:23.106 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:23.106 DEBUG:boto:Port: 8000 2023-07-29T00:58:23.106 DEBUG:boto:Params: {} 2023-07-29T00:58:23.107 DEBUG:boto:Token: None 2023-07-29T00:58:23.107 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:23 GMT /hiiqos-64/ 2023-07-29T00:58:23.107 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY= 2023-07-29T00:58:23.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY=', 'Content-Length': '0'} 2023-07-29T00:58:23.110 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000285c4cdee6050228-0064c4642f-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:23.111 DEBUG:boto:path=/hiiqos-64/obj92 2023-07-29T00:58:23.111 DEBUG:boto:auth_path=/hiiqos-64/obj92 2023-07-29T00:58:23.112 DEBUG:boto:Method: PUT 2023-07-29T00:58:23.112 DEBUG:boto:Path: /hiiqos-64/obj92 2023-07-29T00:58:23.112 DEBUG:boto:Data: 2023-07-29T00:58:23.113 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:58:23.113 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:23.113 DEBUG:boto:Port: 8000 2023-07-29T00:58:23.114 DEBUG:boto:Params: {} 2023-07-29T00:58:23.114 DEBUG:boto:Token: None 2023-07-29T00:58:23.115 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:58:23 GMT /hiiqos-64/obj92 2023-07-29T00:58:23.115 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VZ7PhQmiNX9PGrRJ9Wz71ETfCeo= 2023-07-29T00:58:23.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VZ7PhQmiNX9PGrRJ9Wz71ETfCeo='} 2023-07-29T00:58:23.160 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006cd978a43f9a834c-0064c4642f-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:23.161 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:23.161 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:23.162 DEBUG:boto:Method: HEAD 2023-07-29T00:58:23.162 DEBUG:boto:Path: /hiiqos-64/ 2023-07-29T00:58:23.163 DEBUG:boto:Data: 2023-07-29T00:58:23.163 DEBUG:boto:Headers: {} 2023-07-29T00:58:23.163 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:23.164 DEBUG:boto:Port: 8000 2023-07-29T00:58:23.164 DEBUG:boto:Params: {} 2023-07-29T00:58:23.164 DEBUG:boto:Token: None 2023-07-29T00:58:23.165 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:23 GMT /hiiqos-64/ 2023-07-29T00:58:23.165 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY= 2023-07-29T00:58:23.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY=', 'Content-Length': '0'} 2023-07-29T00:58:23.169 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a991308a4fd16102-0064c4642f-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:23.169 DEBUG:boto:path=/hiiqos-64/obj115 2023-07-29T00:58:23.170 DEBUG:boto:auth_path=/hiiqos-64/obj115 2023-07-29T00:58:23.170 DEBUG:boto:Method: PUT 2023-07-29T00:58:23.170 DEBUG:boto:Path: /hiiqos-64/obj115 2023-07-29T00:58:23.171 DEBUG:boto:Data: 2023-07-29T00:58:23.171 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:58:23.171 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:23.172 DEBUG:boto:Port: 8000 2023-07-29T00:58:23.172 DEBUG:boto:Params: {} 2023-07-29T00:58:23.173 DEBUG:boto:Token: None 2023-07-29T00:58:23.173 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:58:23 GMT /hiiqos-64/obj115 2023-07-29T00:58:23.173 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yS0i+IBJBvz964tNQXTcwP14/U8= 2023-07-29T00:58:23.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yS0i+IBJBvz964tNQXTcwP14/U8='} 2023-07-29T00:58:23.179 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000077907eaf809a301d-0064c4642f-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:23.179 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-64', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:58:23.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-64', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:58:23.180 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-64 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:58:23.230 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:58:23.230 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:58:23.274 INFO:teuthology.orchestra.run.smithi101.stdout:tenant: 2023-07-29T00:58:23.274 INFO:teuthology.orchestra.run.smithi101.stdout:bucket name: hiiqos-64 2023-07-29T00:58:23.282 INFO:teuthology.orchestra.run.smithi101.stdout:total entries: 20 2023-07-29T00:58:23.331 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:23.333 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:23.333 DEBUG:boto:Method: HEAD 2023-07-29T00:58:23.333 DEBUG:boto:Path: /hiiqos-64/ 2023-07-29T00:58:23.334 DEBUG:boto:Data: 2023-07-29T00:58:23.334 DEBUG:boto:Headers: {} 2023-07-29T00:58:23.334 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:23.335 DEBUG:boto:Port: 8000 2023-07-29T00:58:23.335 DEBUG:boto:Params: {} 2023-07-29T00:58:23.336 DEBUG:boto:Token: None 2023-07-29T00:58:23.336 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:23 GMT /hiiqos-64/ 2023-07-29T00:58:23.337 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY= 2023-07-29T00:58:23.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY=', 'Content-Length': '0'} 2023-07-29T00:58:23.340 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000338f6a70caed4fd9-0064c4642f-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:23.342 DEBUG:boto:path=/hiiqos-64/obj29 2023-07-29T00:58:23.342 DEBUG:boto:auth_path=/hiiqos-64/obj29 2023-07-29T00:58:23.343 DEBUG:boto:Method: PUT 2023-07-29T00:58:23.344 DEBUG:boto:Path: /hiiqos-64/obj29 2023-07-29T00:58:23.345 DEBUG:boto:Data: 2023-07-29T00:58:23.346 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:58:23.347 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:23.347 DEBUG:boto:Port: 8000 2023-07-29T00:58:23.347 DEBUG:boto:Params: {} 2023-07-29T00:58:23.348 DEBUG:boto:Token: None 2023-07-29T00:58:23.348 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:58:23 GMT /hiiqos-64/obj29 2023-07-29T00:58:23.349 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:20LMFnIN5bzJ4sH3Ax3NRmptNFM= 2023-07-29T00:58:23.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:20LMFnIN5bzJ4sH3Ax3NRmptNFM='} 2023-07-29T00:58:23.354 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000594ad3a219ee7503-0064c4642f-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:23.355 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:23.355 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:23.356 DEBUG:boto:Method: HEAD 2023-07-29T00:58:23.356 DEBUG:boto:Path: /hiiqos-64/ 2023-07-29T00:58:23.356 DEBUG:boto:Data: 2023-07-29T00:58:23.357 DEBUG:boto:Headers: {} 2023-07-29T00:58:23.357 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:23.357 DEBUG:boto:Port: 8000 2023-07-29T00:58:23.358 DEBUG:boto:Params: {} 2023-07-29T00:58:23.358 DEBUG:boto:Token: None 2023-07-29T00:58:23.359 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:23 GMT /hiiqos-64/ 2023-07-29T00:58:23.359 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY= 2023-07-29T00:58:23.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY=', 'Content-Length': '0'} 2023-07-29T00:58:23.362 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000136f6ca122cc1f1f-0064c4642f-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:23.363 DEBUG:boto:path=/hiiqos-64/obj58 2023-07-29T00:58:23.363 DEBUG:boto:auth_path=/hiiqos-64/obj58 2023-07-29T00:58:23.364 DEBUG:boto:Method: PUT 2023-07-29T00:58:23.364 DEBUG:boto:Path: /hiiqos-64/obj58 2023-07-29T00:58:23.365 DEBUG:boto:Data: 2023-07-29T00:58:23.365 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:58:23.365 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:23.366 DEBUG:boto:Port: 8000 2023-07-29T00:58:23.366 DEBUG:boto:Params: {} 2023-07-29T00:58:23.366 DEBUG:boto:Token: None 2023-07-29T00:58:23.367 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:58:23 GMT /hiiqos-64/obj58 2023-07-29T00:58:23.367 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:L/LtnCmK4c4ltqCCZk1J9GhsxOM= 2023-07-29T00:58:23.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:L/LtnCmK4c4ltqCCZk1J9GhsxOM='} 2023-07-29T00:58:23.412 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b343bf92fca05e8f-0064c4642f-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:23.413 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:23.413 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:23.414 DEBUG:boto:Method: HEAD 2023-07-29T00:58:23.414 DEBUG:boto:Path: /hiiqos-64/ 2023-07-29T00:58:23.414 DEBUG:boto:Data: 2023-07-29T00:58:23.415 DEBUG:boto:Headers: {} 2023-07-29T00:58:23.415 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:23.415 DEBUG:boto:Port: 8000 2023-07-29T00:58:23.416 DEBUG:boto:Params: {} 2023-07-29T00:58:23.416 DEBUG:boto:Token: None 2023-07-29T00:58:23.416 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:23 GMT /hiiqos-64/ 2023-07-29T00:58:23.417 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY= 2023-07-29T00:58:23.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY=', 'Content-Length': '0'} 2023-07-29T00:58:23.420 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c60fa29b40de807-0064c4642f-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:23.421 DEBUG:boto:path=/hiiqos-64/obj87 2023-07-29T00:58:23.421 DEBUG:boto:auth_path=/hiiqos-64/obj87 2023-07-29T00:58:23.422 DEBUG:boto:Method: PUT 2023-07-29T00:58:23.422 DEBUG:boto:Path: /hiiqos-64/obj87 2023-07-29T00:58:23.422 DEBUG:boto:Data: 2023-07-29T00:58:23.423 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:58:23.423 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:23.423 DEBUG:boto:Port: 8000 2023-07-29T00:58:23.424 DEBUG:boto:Params: {} 2023-07-29T00:58:23.425 DEBUG:boto:Token: None 2023-07-29T00:58:23.425 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:58:23 GMT /hiiqos-64/obj87 2023-07-29T00:58:23.426 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH://xrAfWCQs1JKtug7LLY9W7MfXk= 2023-07-29T00:58:23.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH://xrAfWCQs1JKtug7LLY9W7MfXk='} 2023-07-29T00:58:23.431 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000062f5ae65d3440796-0064c4642f-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:23.432 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:23.432 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:23.433 DEBUG:boto:Method: HEAD 2023-07-29T00:58:23.433 DEBUG:boto:Path: /hiiqos-64/ 2023-07-29T00:58:23.434 DEBUG:boto:Data: 2023-07-29T00:58:23.434 DEBUG:boto:Headers: {} 2023-07-29T00:58:23.434 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:23.435 DEBUG:boto:Port: 8000 2023-07-29T00:58:23.435 DEBUG:boto:Params: {} 2023-07-29T00:58:23.435 DEBUG:boto:Token: None 2023-07-29T00:58:23.436 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:23 GMT /hiiqos-64/ 2023-07-29T00:58:23.436 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY= 2023-07-29T00:58:23.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY=', 'Content-Length': '0'} 2023-07-29T00:58:23.440 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001edfc5576eab331c-0064c4642f-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:23.440 DEBUG:boto:path=/hiiqos-64/obj116 2023-07-29T00:58:23.441 DEBUG:boto:auth_path=/hiiqos-64/obj116 2023-07-29T00:58:23.441 DEBUG:boto:Method: PUT 2023-07-29T00:58:23.441 DEBUG:boto:Path: /hiiqos-64/obj116 2023-07-29T00:58:23.442 DEBUG:boto:Data: 2023-07-29T00:58:23.442 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:58:23.442 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:23.443 DEBUG:boto:Port: 8000 2023-07-29T00:58:23.443 DEBUG:boto:Params: {} 2023-07-29T00:58:23.444 DEBUG:boto:Token: None 2023-07-29T00:58:23.444 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:58:23 GMT /hiiqos-64/obj116 2023-07-29T00:58:23.444 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BiIVMOn/8XnCRhiWxoDMPn9cpvs= 2023-07-29T00:58:23.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BiIVMOn/8XnCRhiWxoDMPn9cpvs='} 2023-07-29T00:58:23.492 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e8ce936601278e6b-0064c4642f-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:23.493 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:23.493 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:23.494 DEBUG:boto:Method: HEAD 2023-07-29T00:58:23.494 DEBUG:boto:Path: /hiiqos-64/ 2023-07-29T00:58:23.495 DEBUG:boto:Data: 2023-07-29T00:58:23.495 DEBUG:boto:Headers: {} 2023-07-29T00:58:23.496 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:23.497 DEBUG:boto:Port: 8000 2023-07-29T00:58:23.498 DEBUG:boto:Params: {} 2023-07-29T00:58:23.498 DEBUG:boto:Token: None 2023-07-29T00:58:23.499 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:23 GMT /hiiqos-64/ 2023-07-29T00:58:23.500 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY= 2023-07-29T00:58:23.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY=', 'Content-Length': '0'} 2023-07-29T00:58:23.504 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000071dc2dbb150da616-0064c4642f-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:23.509 DEBUG:boto:path=/hiiqos-64/obj145 2023-07-29T00:58:23.509 DEBUG:boto:auth_path=/hiiqos-64/obj145 2023-07-29T00:58:23.510 DEBUG:boto:Method: PUT 2023-07-29T00:58:23.510 DEBUG:boto:Path: /hiiqos-64/obj145 2023-07-29T00:58:23.511 DEBUG:boto:Data: 2023-07-29T00:58:23.511 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:58:23.511 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:23.512 DEBUG:boto:Port: 8000 2023-07-29T00:58:23.512 DEBUG:boto:Params: {} 2023-07-29T00:58:23.512 DEBUG:boto:Token: None 2023-07-29T00:58:23.513 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:58:23 GMT /hiiqos-64/obj145 2023-07-29T00:58:23.513 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:rJ4rCCqu75csmkLGWwu1Xj4pHuA= 2023-07-29T00:58:23.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rJ4rCCqu75csmkLGWwu1Xj4pHuA='} 2023-07-29T00:58:23.519 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f7ab446e3b8a239f-0064c4642f-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:23.520 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-64', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:58:23.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-64', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:58:23.521 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-64 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:58:23.569 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:58:23.569 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:58:23.615 INFO:teuthology.orchestra.run.smithi101.stdout:tenant: 2023-07-29T00:58:23.616 INFO:teuthology.orchestra.run.smithi101.stdout:bucket name: hiiqos-64 2023-07-29T00:58:23.625 INFO:teuthology.orchestra.run.smithi101.stdout:total entries: 25 2023-07-29T00:58:23.629 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-07-29T00:58:23.681 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:23.682 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:23.682 DEBUG:boto:Method: HEAD 2023-07-29T00:58:23.682 DEBUG:boto:Path: /hiiqos-64/ 2023-07-29T00:58:23.683 DEBUG:boto:Data: 2023-07-29T00:58:23.683 DEBUG:boto:Headers: {} 2023-07-29T00:58:23.684 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:23.684 DEBUG:boto:Port: 8000 2023-07-29T00:58:23.684 DEBUG:boto:Params: {} 2023-07-29T00:58:23.685 DEBUG:boto:Token: None 2023-07-29T00:58:23.685 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:23 GMT /hiiqos-64/ 2023-07-29T00:58:23.686 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY= 2023-07-29T00:58:23.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY=', 'Content-Length': '0'} 2023-07-29T00:58:23.691 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb96a4de6fece522-0064c4642f-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:23.692 DEBUG:boto:path=/hiiqos-64/obj31 2023-07-29T00:58:23.692 DEBUG:boto:auth_path=/hiiqos-64/obj31 2023-07-29T00:58:23.693 DEBUG:boto:Method: PUT 2023-07-29T00:58:23.693 DEBUG:boto:Path: /hiiqos-64/obj31 2023-07-29T00:58:23.693 DEBUG:boto:Data: 2023-07-29T00:58:23.694 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:58:23.694 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:23.694 DEBUG:boto:Port: 8000 2023-07-29T00:58:23.695 DEBUG:boto:Params: {} 2023-07-29T00:58:23.695 DEBUG:boto:Token: None 2023-07-29T00:58:23.696 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:58:23 GMT /hiiqos-64/obj31 2023-07-29T00:58:23.696 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ADX7jo0PAtVZeTQyf2WWGyliYw8= 2023-07-29T00:58:23.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ADX7jo0PAtVZeTQyf2WWGyliYw8='} 2023-07-29T00:58:23.701 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c3ee9d01bd548065-0064c4642f-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:23.702 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:23.702 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:23.702 DEBUG:boto:Method: HEAD 2023-07-29T00:58:23.703 DEBUG:boto:Path: /hiiqos-64/ 2023-07-29T00:58:23.703 DEBUG:boto:Data: 2023-07-29T00:58:23.703 DEBUG:boto:Headers: {} 2023-07-29T00:58:23.704 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:23.704 DEBUG:boto:Port: 8000 2023-07-29T00:58:23.704 DEBUG:boto:Params: {} 2023-07-29T00:58:23.705 DEBUG:boto:Token: None 2023-07-29T00:58:23.705 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:23 GMT /hiiqos-64/ 2023-07-29T00:58:23.706 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY= 2023-07-29T00:58:23.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY=', 'Content-Length': '0'} 2023-07-29T00:58:23.710 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004492d81d770addf6-0064c4642f-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:23.710 DEBUG:boto:path=/hiiqos-64/obj62 2023-07-29T00:58:23.711 DEBUG:boto:auth_path=/hiiqos-64/obj62 2023-07-29T00:58:23.711 DEBUG:boto:Method: PUT 2023-07-29T00:58:23.711 DEBUG:boto:Path: /hiiqos-64/obj62 2023-07-29T00:58:23.712 DEBUG:boto:Data: 2023-07-29T00:58:23.712 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:58:23.712 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:23.713 DEBUG:boto:Port: 8000 2023-07-29T00:58:23.713 DEBUG:boto:Params: {} 2023-07-29T00:58:23.713 DEBUG:boto:Token: None 2023-07-29T00:58:23.714 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:58:23 GMT /hiiqos-64/obj62 2023-07-29T00:58:23.714 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Ben/UU9dw8COBVcXtB6GkMwraxA= 2023-07-29T00:58:23.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Ben/UU9dw8COBVcXtB6GkMwraxA='} 2023-07-29T00:58:23.756 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005807066d8a53450a-0064c4642f-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:23.757 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:23.757 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:23.758 DEBUG:boto:Method: HEAD 2023-07-29T00:58:23.758 DEBUG:boto:Path: /hiiqos-64/ 2023-07-29T00:58:23.758 DEBUG:boto:Data: 2023-07-29T00:58:23.759 DEBUG:boto:Headers: {} 2023-07-29T00:58:23.759 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:23.759 DEBUG:boto:Port: 8000 2023-07-29T00:58:23.760 DEBUG:boto:Params: {} 2023-07-29T00:58:23.760 DEBUG:boto:Token: None 2023-07-29T00:58:23.760 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:23 GMT /hiiqos-64/ 2023-07-29T00:58:23.761 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY= 2023-07-29T00:58:23.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY=', 'Content-Length': '0'} 2023-07-29T00:58:23.764 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001e802c9167dbb77e-0064c4642f-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:23.765 DEBUG:boto:path=/hiiqos-64/obj93 2023-07-29T00:58:23.765 DEBUG:boto:auth_path=/hiiqos-64/obj93 2023-07-29T00:58:23.765 DEBUG:boto:Method: PUT 2023-07-29T00:58:23.766 DEBUG:boto:Path: /hiiqos-64/obj93 2023-07-29T00:58:23.766 DEBUG:boto:Data: 2023-07-29T00:58:23.766 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:58:23.767 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:23.767 DEBUG:boto:Port: 8000 2023-07-29T00:58:23.767 DEBUG:boto:Params: {} 2023-07-29T00:58:23.768 DEBUG:boto:Token: None 2023-07-29T00:58:23.768 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:58:23 GMT /hiiqos-64/obj93 2023-07-29T00:58:23.769 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tEyhZuyW0BSFmigNxxRTyl51LKo= 2023-07-29T00:58:23.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tEyhZuyW0BSFmigNxxRTyl51LKo='} 2023-07-29T00:58:23.774 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000de634436149d0fdb-0064c4642f-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:23.775 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:23.775 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:23.775 DEBUG:boto:Method: HEAD 2023-07-29T00:58:23.776 DEBUG:boto:Path: /hiiqos-64/ 2023-07-29T00:58:23.776 DEBUG:boto:Data: 2023-07-29T00:58:23.776 DEBUG:boto:Headers: {} 2023-07-29T00:58:23.777 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:23.777 DEBUG:boto:Port: 8000 2023-07-29T00:58:23.777 DEBUG:boto:Params: {} 2023-07-29T00:58:23.778 DEBUG:boto:Token: None 2023-07-29T00:58:23.778 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:23 GMT /hiiqos-64/ 2023-07-29T00:58:23.778 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY= 2023-07-29T00:58:23.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY=', 'Content-Length': '0'} 2023-07-29T00:58:23.782 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e17fa169fe81a014-0064c4642f-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:23.783 DEBUG:boto:path=/hiiqos-64/obj124 2023-07-29T00:58:23.783 DEBUG:boto:auth_path=/hiiqos-64/obj124 2023-07-29T00:58:23.783 DEBUG:boto:Method: PUT 2023-07-29T00:58:23.784 DEBUG:boto:Path: /hiiqos-64/obj124 2023-07-29T00:58:23.784 DEBUG:boto:Data: 2023-07-29T00:58:23.784 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:58:23.785 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:23.785 DEBUG:boto:Port: 8000 2023-07-29T00:58:23.786 DEBUG:boto:Params: {} 2023-07-29T00:58:23.787 DEBUG:boto:Token: None 2023-07-29T00:58:23.787 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:58:23 GMT /hiiqos-64/obj124 2023-07-29T00:58:23.788 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:x16d5Y7vIzwi1MoCpWG1YnD0qJY= 2023-07-29T00:58:23.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:x16d5Y7vIzwi1MoCpWG1YnD0qJY='} 2023-07-29T00:58:23.832 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008560f975adb190d1-0064c4642f-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:23.833 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:23.834 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:23.834 DEBUG:boto:Method: HEAD 2023-07-29T00:58:23.834 DEBUG:boto:Path: /hiiqos-64/ 2023-07-29T00:58:23.835 DEBUG:boto:Data: 2023-07-29T00:58:23.835 DEBUG:boto:Headers: {} 2023-07-29T00:58:23.835 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:23.836 DEBUG:boto:Port: 8000 2023-07-29T00:58:23.836 DEBUG:boto:Params: {} 2023-07-29T00:58:23.836 DEBUG:boto:Token: None 2023-07-29T00:58:23.837 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:23 GMT /hiiqos-64/ 2023-07-29T00:58:23.837 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY= 2023-07-29T00:58:23.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY=', 'Content-Length': '0'} 2023-07-29T00:58:23.841 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f05d22c2a3b95c68-0064c4642f-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:23.841 DEBUG:boto:path=/hiiqos-64/obj155 2023-07-29T00:58:23.842 DEBUG:boto:auth_path=/hiiqos-64/obj155 2023-07-29T00:58:23.842 DEBUG:boto:Method: PUT 2023-07-29T00:58:23.842 DEBUG:boto:Path: /hiiqos-64/obj155 2023-07-29T00:58:23.843 DEBUG:boto:Data: 2023-07-29T00:58:23.843 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:58:23.843 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:23.844 DEBUG:boto:Port: 8000 2023-07-29T00:58:23.844 DEBUG:boto:Params: {} 2023-07-29T00:58:23.844 DEBUG:boto:Token: None 2023-07-29T00:58:23.845 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:58:23 GMT /hiiqos-64/obj155 2023-07-29T00:58:23.845 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UPuNe24LCHpb6JxiDYiIERwX/xc= 2023-07-29T00:58:23.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UPuNe24LCHpb6JxiDYiIERwX/xc='} 2023-07-29T00:58:23.851 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000033bc912772c9e448-0064c4642f-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:23.851 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-64', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:58:23.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-64', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:58:23.852 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-64 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:58:23.903 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:58:23.904 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:58:23.951 INFO:teuthology.orchestra.run.smithi101.stdout:tenant: 2023-07-29T00:58:23.951 INFO:teuthology.orchestra.run.smithi101.stdout:bucket name: hiiqos-64 2023-07-29T00:58:23.962 INFO:teuthology.orchestra.run.smithi101.stdout:total entries: 30 2023-07-29T00:58:24.022 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:24.023 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:24.023 DEBUG:boto:Method: HEAD 2023-07-29T00:58:24.024 DEBUG:boto:Path: /hiiqos-64/ 2023-07-29T00:58:24.024 DEBUG:boto:Data: 2023-07-29T00:58:24.024 DEBUG:boto:Headers: {} 2023-07-29T00:58:24.025 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:24.025 DEBUG:boto:Port: 8000 2023-07-29T00:58:24.026 DEBUG:boto:Params: {} 2023-07-29T00:58:24.026 DEBUG:boto:Token: None 2023-07-29T00:58:24.026 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:24 GMT /hiiqos-64/ 2023-07-29T00:58:24.027 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QFflsEIpX1mG+hR5r/ygIAaBax4= 2023-07-29T00:58:24.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QFflsEIpX1mG+hR5r/ygIAaBax4=', 'Content-Length': '0'} 2023-07-29T00:58:24.031 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005ae0e84edbe4f7a6-0064c4642f-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:24.032 DEBUG:boto:path=/hiiqos-64/obj37 2023-07-29T00:58:24.032 DEBUG:boto:auth_path=/hiiqos-64/obj37 2023-07-29T00:58:24.033 DEBUG:boto:Method: PUT 2023-07-29T00:58:24.033 DEBUG:boto:Path: /hiiqos-64/obj37 2023-07-29T00:58:24.034 DEBUG:boto:Data: 2023-07-29T00:58:24.034 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:58:24.034 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:24.035 DEBUG:boto:Port: 8000 2023-07-29T00:58:24.035 DEBUG:boto:Params: {} 2023-07-29T00:58:24.036 DEBUG:boto:Token: None 2023-07-29T00:58:24.036 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:58:24 GMT /hiiqos-64/obj37 2023-07-29T00:58:24.037 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:VOH4YMi/PTl+sjx/EwL8mxVBnfE= 2023-07-29T00:58:24.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VOH4YMi/PTl+sjx/EwL8mxVBnfE='} 2023-07-29T00:58:24.084 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006f52a9297a76dced-0064c4642f-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:24.085 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:24.086 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:24.086 DEBUG:boto:Method: HEAD 2023-07-29T00:58:24.086 DEBUG:boto:Path: /hiiqos-64/ 2023-07-29T00:58:24.087 DEBUG:boto:Data: 2023-07-29T00:58:24.087 DEBUG:boto:Headers: {} 2023-07-29T00:58:24.087 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:24.088 DEBUG:boto:Port: 8000 2023-07-29T00:58:24.088 DEBUG:boto:Params: {} 2023-07-29T00:58:24.088 DEBUG:boto:Token: None 2023-07-29T00:58:24.089 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:24 GMT /hiiqos-64/ 2023-07-29T00:58:24.090 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QFflsEIpX1mG+hR5r/ygIAaBax4= 2023-07-29T00:58:24.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QFflsEIpX1mG+hR5r/ygIAaBax4=', 'Content-Length': '0'} 2023-07-29T00:58:24.094 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000661304cd3cf5fb33-0064c46430-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:24.094 DEBUG:boto:path=/hiiqos-64/obj74 2023-07-29T00:58:24.095 DEBUG:boto:auth_path=/hiiqos-64/obj74 2023-07-29T00:58:24.095 DEBUG:boto:Method: PUT 2023-07-29T00:58:24.095 DEBUG:boto:Path: /hiiqos-64/obj74 2023-07-29T00:58:24.096 DEBUG:boto:Data: 2023-07-29T00:58:24.096 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:58:24.096 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:24.097 DEBUG:boto:Port: 8000 2023-07-29T00:58:24.097 DEBUG:boto:Params: {} 2023-07-29T00:58:24.097 DEBUG:boto:Token: None 2023-07-29T00:58:24.098 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:58:24 GMT /hiiqos-64/obj74 2023-07-29T00:58:24.098 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:2WdMDkMbMXJUHTeS1G25M9pNwBE= 2023-07-29T00:58:24.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2WdMDkMbMXJUHTeS1G25M9pNwBE='} 2023-07-29T00:58:24.104 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006f0c4ed465e74113-0064c46430-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:24.105 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:24.106 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:24.106 DEBUG:boto:Method: HEAD 2023-07-29T00:58:24.107 DEBUG:boto:Path: /hiiqos-64/ 2023-07-29T00:58:24.108 DEBUG:boto:Data: 2023-07-29T00:58:24.108 DEBUG:boto:Headers: {} 2023-07-29T00:58:24.109 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:24.110 DEBUG:boto:Port: 8000 2023-07-29T00:58:24.110 DEBUG:boto:Params: {} 2023-07-29T00:58:24.111 DEBUG:boto:Token: None 2023-07-29T00:58:24.112 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:24 GMT /hiiqos-64/ 2023-07-29T00:58:24.112 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QFflsEIpX1mG+hR5r/ygIAaBax4= 2023-07-29T00:58:24.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QFflsEIpX1mG+hR5r/ygIAaBax4=', 'Content-Length': '0'} 2023-07-29T00:58:24.117 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007bb9058b7153c3f9-0064c46430-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:24.118 DEBUG:boto:path=/hiiqos-64/obj111 2023-07-29T00:58:24.119 DEBUG:boto:auth_path=/hiiqos-64/obj111 2023-07-29T00:58:24.119 DEBUG:boto:Method: PUT 2023-07-29T00:58:24.120 DEBUG:boto:Path: /hiiqos-64/obj111 2023-07-29T00:58:24.121 DEBUG:boto:Data: 2023-07-29T00:58:24.121 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:58:24.122 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:24.123 DEBUG:boto:Port: 8000 2023-07-29T00:58:24.124 DEBUG:boto:Params: {} 2023-07-29T00:58:24.124 DEBUG:boto:Token: None 2023-07-29T00:58:24.125 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:58:24 GMT /hiiqos-64/obj111 2023-07-29T00:58:24.126 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0DDFFPR8hY5rbxdfLIniRerLJ+o= 2023-07-29T00:58:24.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0DDFFPR8hY5rbxdfLIniRerLJ+o='} 2023-07-29T00:58:24.177 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000062f6c1f1e094520f-0064c46430-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:24.178 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:24.178 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:24.178 DEBUG:boto:Method: HEAD 2023-07-29T00:58:24.179 DEBUG:boto:Path: /hiiqos-64/ 2023-07-29T00:58:24.179 DEBUG:boto:Data: 2023-07-29T00:58:24.179 DEBUG:boto:Headers: {} 2023-07-29T00:58:24.180 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:24.180 DEBUG:boto:Port: 8000 2023-07-29T00:58:24.181 DEBUG:boto:Params: {} 2023-07-29T00:58:24.181 DEBUG:boto:Token: None 2023-07-29T00:58:24.182 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:24 GMT /hiiqos-64/ 2023-07-29T00:58:24.183 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QFflsEIpX1mG+hR5r/ygIAaBax4= 2023-07-29T00:58:24.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QFflsEIpX1mG+hR5r/ygIAaBax4=', 'Content-Length': '0'} 2023-07-29T00:58:24.187 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000003e958dd09ed0c8b-0064c46430-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:24.188 DEBUG:boto:path=/hiiqos-64/obj148 2023-07-29T00:58:24.189 DEBUG:boto:auth_path=/hiiqos-64/obj148 2023-07-29T00:58:24.189 DEBUG:boto:Method: PUT 2023-07-29T00:58:24.189 DEBUG:boto:Path: /hiiqos-64/obj148 2023-07-29T00:58:24.190 DEBUG:boto:Data: 2023-07-29T00:58:24.190 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:58:24.190 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:24.191 DEBUG:boto:Port: 8000 2023-07-29T00:58:24.191 DEBUG:boto:Params: {} 2023-07-29T00:58:24.192 DEBUG:boto:Token: None 2023-07-29T00:58:24.192 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:58:24 GMT /hiiqos-64/obj148 2023-07-29T00:58:24.193 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QKuNJNpD64X5Z61YBXus+gu34EQ= 2023-07-29T00:58:24.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QKuNJNpD64X5Z61YBXus+gu34EQ='} 2023-07-29T00:58:24.199 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000012383b1a8a7ff97f-0064c46430-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:24.200 DEBUG:boto:path=/hiiqos-64/ 2023-07-29T00:58:24.201 DEBUG:boto:auth_path=/hiiqos-64/ 2023-07-29T00:58:24.202 DEBUG:boto:Method: HEAD 2023-07-29T00:58:24.202 DEBUG:boto:Path: /hiiqos-64/ 2023-07-29T00:58:24.203 DEBUG:boto:Data: 2023-07-29T00:58:24.204 DEBUG:boto:Headers: {} 2023-07-29T00:58:24.205 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:24.205 DEBUG:boto:Port: 8000 2023-07-29T00:58:24.206 DEBUG:boto:Params: {} 2023-07-29T00:58:24.207 DEBUG:boto:Token: None 2023-07-29T00:58:24.207 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 00:58:24 GMT /hiiqos-64/ 2023-07-29T00:58:24.208 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:QFflsEIpX1mG+hR5r/ygIAaBax4= 2023-07-29T00:58:24.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QFflsEIpX1mG+hR5r/ygIAaBax4=', 'Content-Length': '0'} 2023-07-29T00:58:24.211 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000d8ab57464f360b7-0064c46430-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:24.212 DEBUG:boto:path=/hiiqos-64/obj185 2023-07-29T00:58:24.212 DEBUG:boto:auth_path=/hiiqos-64/obj185 2023-07-29T00:58:24.213 DEBUG:boto:Method: PUT 2023-07-29T00:58:24.213 DEBUG:boto:Path: /hiiqos-64/obj185 2023-07-29T00:58:24.214 DEBUG:boto:Data: 2023-07-29T00:58:24.214 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T00:58:24.214 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T00:58:24.215 DEBUG:boto:Port: 8000 2023-07-29T00:58:24.215 DEBUG:boto:Params: {} 2023-07-29T00:58:24.215 DEBUG:boto:Token: None 2023-07-29T00:58:24.216 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Sat, 29 Jul 2023 00:58:24 GMT /hiiqos-64/obj185 2023-07-29T00:58:24.216 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:1oP1h/rF3Y5Mwf0ltytXmhB6KMI= 2023-07-29T00:58:24.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1oP1h/rF3Y5Mwf0ltytXmhB6KMI='} 2023-07-29T00:58:24.223 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002c9df75fbb40112a-0064c46430-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T00:58:24.223 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:58:24.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T00:58:24.225 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T00:58:24.272 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:58:24.272 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:58:24.302 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:58:24.223+0000 7fe304010880 5 RGW-SYNC:data: waiting on data sync status lease 2023-07-29T00:58:24.303 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:58:24.224+0000 7fe304010880 5 RGW-SYNC:data: acquired data sync status lease 2023-07-29T00:58:24.323 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-07-29T00:58:24.324 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-07-29T00:58:24.326 INFO:tasks.rgw.c1.client.1:Started 2023-07-29T00:58:24.327 DEBUG:teuthology.orchestra.run.smithi101:> curl http://smithi101.front.sepia.ceph.com:8001 2023-07-29T00:58:24.344 INFO:teuthology.orchestra.run.smithi101.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-07-29T00:58:24.344 INFO:teuthology.orchestra.run.smithi101.stderr: Dload Upload Total Spent Left Speed 2023-07-29T00:58:24.344 INFO:teuthology.orchestra.run.smithi101.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-07-29T00:58:24.345 INFO:teuthology.orchestra.run.smithi101.stderr:curl: (7) Failed to connect to smithi101.front.sepia.ceph.com port 8001: Connection refused 2023-07-29T00:58:24.345 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-07-29T00:58:25.347 DEBUG:teuthology.orchestra.run.smithi101:> curl http://smithi101.front.sepia.ceph.com:8001 2023-07-29T00:58:25.365 INFO:teuthology.orchestra.run.smithi101.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-07-29T00:58:25.366 INFO:teuthology.orchestra.run.smithi101.stderr: Dload Upload Total Spent Left Speed 2023-07-29T00:58:25.367 INFO:teuthology.orchestra.run.smithi101.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-07-29T00:58:25.367 INFO:teuthology.orchestra.run.smithi101.stdout:anonymous 2023-07-29T00:58:25.367 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hiiqos-64 2023-07-29T00:58:25.368 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:58:25.368 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T00:58:25.368 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-64 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T00:58:25.459 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T00:58:25.459 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T00:58:25.495 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:58:25.416+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T00:58:30.495 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:58:30.416+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T00:58:35.496 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:58:35.417+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T00:58:40.497 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:58:40.418+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T00:58:45.498 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:58:45.419+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T00:58:50.498 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:58:50.419+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T00:58:55.499 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:58:55.420+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T00:59:00.499 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:59:00.420+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T00:59:05.500 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:59:05.421+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T00:59:10.501 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:59:10.422+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T00:59:15.502 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:59:15.422+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T00:59:17.966 INFO:tasks.rgw.c2.client.1.smithi139.stdout:2023-07-29T00:59:17.886+0000 7f5731a6a640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T00:59:20.502 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:59:20.423+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T00:59:25.503 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:59:25.423+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T00:59:30.503 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:59:30.424+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T00:59:35.504 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:59:35.424+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T00:59:40.505 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:59:40.425+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T00:59:45.506 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:59:45.426+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T00:59:50.506 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:59:50.427+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T00:59:55.507 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T00:59:55.427+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T01:00:00.508 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:00:00.428+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T01:00:05.509 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:00:05.428+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T01:00:10.509 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:00:10.429+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T01:00:15.510 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:00:15.430+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T01:00:20.511 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:00:20.430+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T01:00:25.511 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:00:25.431+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:00:30.512 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:00:30.431+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:00:35.512 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:00:35.432+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:00:40.513 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:00:40.433+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:00:45.514 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:00:45.433+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:00:50.514 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:00:50.434+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:00:55.515 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:00:55.434+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:01:00.516 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:01:00.435+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:01:05.516 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:01:05.435+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:01:10.516 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:01:10.435+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:01:15.517 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:01:15.436+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:01:20.518 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:01:20.437+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:01:25.518 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:01:25.437+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:01:30.519 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:01:30.438+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:01:35.519 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:01:35.438+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:01:40.520 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:01:40.439+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:01:45.521 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:01:45.440+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:01:50.521 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:01:50.440+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:01:55.522 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:01:55.441+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:02:00.523 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:02:00.442+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:02:05.524 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:02:05.442+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:02:10.524 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:02:10.442+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:02:15.525 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:02:15.443+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:02:20.525 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:02:20.444+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:02:24.782 INFO:tasks.rgw.c1.client.0.smithi101.stdout:2023-07-29T01:02:24.700+0000 7f0081a5e640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T01:02:25.526 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:02:25.444+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:02:30.527 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:02:30.445+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:02:35.527 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:02:35.445+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:02:40.528 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:02:40.446+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:02:45.529 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:02:45.448+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:02:50.530 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:02:50.448+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:02:55.530 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:02:55.448+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:03:00.531 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:03:00.449+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:03:05.532 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:03:05.449+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:03:10.532 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:03:10.450+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:03:15.533 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:03:15.451+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:03:20.534 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:03:20.452+0000 7fa62e326880 -1 bucket checkpoint timed out waiting to reach latest generation 6 2023-07-29T01:03:20.535 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:03:20.452+0000 7fa62e326880 0 bucket sync checkpoint failed: (110) Connection timed out 2023-07-29T01:03:20.535 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:03:20.452+0000 7fa62e326880 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-07-29T01:03:20.552 DEBUG:teuthology.orchestra.run:got remote process result: 110 2023-07-29T01:03:20.555 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init ... FAIL 2023-07-29T01:03:20.556 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:03:20.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:03:20.558 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:03:20.609 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:03:20.610 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:03:20.642 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T01:03:20.656 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T01:03:20.656 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:20.656 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:03:20.657 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:17.836402Z" 2023-07-29T01:03:20.657 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:20.657 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:20.657 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592303.891548_1258.1", 2023-07-29T01:03:20.657 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:58:23.891548Z" 2023-07-29T01:03:20.657 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:20.658 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:20.658 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:03:20.658 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T01:03:20.658 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:20.658 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:20.658 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:03:20.659 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:12.811406Z" 2023-07-29T01:03:20.659 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:20.659 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T01:03:20.660 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690591217.836402_294.1', 1: '1_1690592303.891548_1258.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:03:20.660 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T01:03:20.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:03:20.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:03:20.662 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:03:20.708 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:03:20.709 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:03:20.737 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:03:20.654+0000 7f4f61416880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:03:20.738 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:03:20.655+0000 7f4f61416880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:03:20.751 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:03:20.751 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:03:20.751 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:03:20.751 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:03:20.752 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:03:20.752 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:03:20.752 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:03:20.752 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:20.752 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:03:20.752 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:20.753 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:03:20.753 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:03:20.753 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:03:20.753 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:03:20.753 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:03:20.753 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:03:20.753 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:03:20.754 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:03:20.754 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:03:20.754 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:20.754 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:20.754 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:20.754 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:03:20.755 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:03:20.755 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:03:20.759 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592303.891548_1258.1", 2023-07-29T01:03:20.759 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:03:20.760 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:03:20.760 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:03:20.760 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:23.891548Z", 2023-07-29T01:03:20.760 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:03:20.761 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:20.761 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:20.761 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:20.761 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:03:20.761 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:03:20.761 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:03:20.762 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:03:20.762 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:03:20.762 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:03:20.762 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:03:20.762 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:03:20.763 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:03:20.763 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:20.763 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:20.763 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:20.764 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:03:20.764 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:03:20.764 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:03:20.764 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:03:20.765 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:03:20.765 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:03:20.765 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:03:20.765 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:03:20.765 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:03:20.765 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:20.765 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:20.766 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:03:20.766 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:20.766 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:03:20.766 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:03:20.766 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:03:20.766 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:20.766 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:03:20Z" 2023-07-29T01:03:20.767 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:03:20.767 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592303.891548_1258.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:23.891548Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:03:20Z" } 2023-07-29T01:03:20.768 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592303.891548_1258.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:23.891548Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:03:20.769 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592303.891548_1258.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:03:20.769 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592303.891548_1258.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:03:20.769 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T01:03:20.770 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:03:20.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:03:20.771 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:03:20.813 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:03:20.813 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:03:20.857 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[]} 2023-07-29T01:03:20.858 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:03:20.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:03:20.859 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:03:20.943 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:03:20.944 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:03:20.988 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-07-29T01:03:20.989 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:03:20.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:03:20.990 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:03:21.036 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:03:21.037 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:03:21.081 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-07-29T01:03:21.082 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:03:21.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:03:21.083 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:03:21.126 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:03:21.127 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:03:21.170 INFO:teuthology.orchestra.run.smithi101.stdout:{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"} 2023-07-29T01:03:21.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:03:21.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:03:21.172 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:03:21.214 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:03:21.215 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:03:21.258 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-07-29T01:03:21.259 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:03:21.260 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:03:21.260 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:03:21.348 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:03:21.348 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:03:21.436 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2023-07-29T01:03:21.437 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:03:21.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:03:21.437 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:03:21.487 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:03:21.488 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:03:21.532 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2023-07-29T01:03:21.533 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:03:21.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:03:21.534 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:03:21.582 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:03:21.582 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:03:21.626 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-07-29T01:03:21.627 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:03:21.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:03:21.628 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:03:21.672 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:03:21.672 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:03:21.717 INFO:teuthology.orchestra.run.smithi101.stdout:{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"} 2023-07-29T01:03:21.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:03:21.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:03:21.719 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:03:21.763 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:03:21.764 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:03:21.814 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:03:21.815 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:03:21.815 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 13, 2023-07-29T01:03:21.815 INFO:teuthology.orchestra.run.smithi101.stdout: "predecessor_uuid": "a0987484-ce37-4520-a564-54b7ea8c5251", 2023-07-29T01:03:21.815 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": [], 2023-07-29T01:03:21.815 INFO:teuthology.orchestra.run.smithi101.stdout: "period_map": { 2023-07-29T01:03:21.815 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:03:21.815 INFO:teuthology.orchestra.run.smithi101.stdout: "zonegroups": [ 2023-07-29T01:03:21.816 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:21.816 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "a", 2023-07-29T01:03:21.816 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a", 2023-07-29T01:03:21.816 INFO:teuthology.orchestra.run.smithi101.stdout: "api_name": "", 2023-07-29T01:03:21.816 INFO:teuthology.orchestra.run.smithi101.stdout: "is_master": true, 2023-07-29T01:03:21.816 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:03:21.817 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-29T01:03:21.817 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:03:21.817 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames": [], 2023-07-29T01:03:21.817 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames_s3website": [], 2023-07-29T01:03:21.817 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:03:21.817 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:03:21.818 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:21.818 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:03:21.818 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a1", 2023-07-29T01:03:21.818 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:03:21.818 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-29T01:03:21.818 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:03:21.818 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T01:03:21.819 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T01:03:21.819 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T01:03:21.819 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T01:03:21.819 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T01:03:21.819 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T01:03:21.819 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T01:03:21.819 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T01:03:21.820 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T01:03:21.820 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T01:03:21.820 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:03:21.820 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:03:21.820 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:21.820 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:21.820 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-29T01:03:21.821 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a2", 2023-07-29T01:03:21.821 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:03:21.821 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8001" 2023-07-29T01:03:21.821 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:03:21.821 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T01:03:21.821 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T01:03:21.821 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T01:03:21.822 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T01:03:21.822 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T01:03:21.822 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T01:03:21.822 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T01:03:21.822 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T01:03:21.822 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T01:03:21.822 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T01:03:21.823 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:03:21.823 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:03:21.823 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:21.823 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:03:21.823 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_targets": [ 2023-07-29T01:03:21.824 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:21.825 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default-placement", 2023-07-29T01:03:21.825 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [], 2023-07-29T01:03:21.825 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": [ 2023-07-29T01:03:21.825 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD" 2023-07-29T01:03:21.826 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:03:21.826 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:21.826 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:03:21.827 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "default-placement", 2023-07-29T01:03:21.827 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T01:03:21.827 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_policy": { 2023-07-29T01:03:21.827 INFO:teuthology.orchestra.run.smithi101.stdout: "groups": [ 2023-07-29T01:03:21.828 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:21.828 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "sync-group", 2023-07-29T01:03:21.828 INFO:teuthology.orchestra.run.smithi101.stdout: "data_flow": { 2023-07-29T01:03:21.829 INFO:teuthology.orchestra.run.smithi101.stdout: "symmetrical": [ 2023-07-29T01:03:21.829 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:21.829 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "sync-flow1", 2023-07-29T01:03:21.830 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:03:21.830 INFO:teuthology.orchestra.run.smithi101.stdout: "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:03:21.830 INFO:teuthology.orchestra.run.smithi101.stdout: "59d1bb30-3b1c-476f-9111-052c1ea0b711" 2023-07-29T01:03:21.831 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:03:21.831 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:21.831 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:03:21.831 INFO:teuthology.orchestra.run.smithi101.stdout: "directional": [ 2023-07-29T01:03:21.832 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:21.832 INFO:teuthology.orchestra.run.smithi101.stdout: "source_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:03:21.832 INFO:teuthology.orchestra.run.smithi101.stdout: "dest_zone": "59d1bb30-3b1c-476f-9111-052c1ea0b711" 2023-07-29T01:03:21.833 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:21.833 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:03:21.833 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:21.834 INFO:teuthology.orchestra.run.smithi101.stdout: "pipes": [ 2023-07-29T01:03:21.834 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:21.834 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "sync-pipe", 2023-07-29T01:03:21.834 INFO:teuthology.orchestra.run.smithi101.stdout: "source": { 2023-07-29T01:03:21.834 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket": "*", 2023-07-29T01:03:21.835 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:03:21.835 INFO:teuthology.orchestra.run.smithi101.stdout: "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:03:21.835 INFO:teuthology.orchestra.run.smithi101.stdout: "59d1bb30-3b1c-476f-9111-052c1ea0b711" 2023-07-29T01:03:21.836 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:03:21.836 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:21.836 INFO:teuthology.orchestra.run.smithi101.stdout: "dest": { 2023-07-29T01:03:21.836 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket": "*", 2023-07-29T01:03:21.837 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:03:21.837 INFO:teuthology.orchestra.run.smithi101.stdout: "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:03:21.837 INFO:teuthology.orchestra.run.smithi101.stdout: "59d1bb30-3b1c-476f-9111-052c1ea0b711" 2023-07-29T01:03:21.838 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:03:21.838 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:21.838 INFO:teuthology.orchestra.run.smithi101.stdout: "params": { 2023-07-29T01:03:21.839 INFO:teuthology.orchestra.run.smithi101.stdout: "source": { 2023-07-29T01:03:21.839 INFO:teuthology.orchestra.run.smithi101.stdout: "filter": { 2023-07-29T01:03:21.839 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [] 2023-07-29T01:03:21.839 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:21.840 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:21.840 INFO:teuthology.orchestra.run.smithi101.stdout: "dest": {}, 2023-07-29T01:03:21.840 INFO:teuthology.orchestra.run.smithi101.stdout: "priority": 0, 2023-07-29T01:03:21.841 INFO:teuthology.orchestra.run.smithi101.stdout: "mode": "system", 2023-07-29T01:03:21.841 INFO:teuthology.orchestra.run.smithi101.stdout: "user": "" 2023-07-29T01:03:21.842 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:21.842 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:21.843 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:03:21.843 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "enabled" 2023-07-29T01:03:21.843 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:21.844 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:03:21.844 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:21.844 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled_features": [ 2023-07-29T01:03:21.844 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:03:21.844 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:03:21.845 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:21.845 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:21.845 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "b", 2023-07-29T01:03:21.845 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b", 2023-07-29T01:03:21.845 INFO:teuthology.orchestra.run.smithi101.stdout: "api_name": "", 2023-07-29T01:03:21.845 INFO:teuthology.orchestra.run.smithi101.stdout: "is_master": false, 2023-07-29T01:03:21.845 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:03:21.846 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-29T01:03:21.846 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:03:21.846 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames": [], 2023-07-29T01:03:21.846 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames_s3website": [], 2023-07-29T01:03:21.846 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T01:03:21.846 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:03:21.846 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:21.847 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T01:03:21.847 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b1", 2023-07-29T01:03:21.847 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:03:21.847 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-29T01:03:21.847 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:03:21.847 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T01:03:21.847 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T01:03:21.848 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T01:03:21.848 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T01:03:21.848 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T01:03:21.848 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T01:03:21.848 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T01:03:21.849 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T01:03:21.849 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T01:03:21.849 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T01:03:21.849 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:03:21.849 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:03:21.849 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:21.849 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:21.850 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-29T01:03:21.850 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b2", 2023-07-29T01:03:21.850 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:03:21.850 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8001" 2023-07-29T01:03:21.850 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:03:21.850 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T01:03:21.850 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T01:03:21.851 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T01:03:21.851 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T01:03:21.851 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T01:03:21.851 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T01:03:21.851 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T01:03:21.851 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T01:03:21.852 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T01:03:21.852 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T01:03:21.852 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:03:21.852 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:03:21.852 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:21.852 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:03:21.852 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_targets": [ 2023-07-29T01:03:21.853 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:21.853 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default-placement", 2023-07-29T01:03:21.853 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [], 2023-07-29T01:03:21.853 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": [ 2023-07-29T01:03:21.853 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD" 2023-07-29T01:03:21.853 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:03:21.853 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:21.854 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:03:21.854 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "default-placement", 2023-07-29T01:03:21.854 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T01:03:21.854 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_policy": { 2023-07-29T01:03:21.854 INFO:teuthology.orchestra.run.smithi101.stdout: "groups": [] 2023-07-29T01:03:21.855 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:21.855 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled_features": [ 2023-07-29T01:03:21.856 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:03:21.856 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:03:21.856 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:21.856 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:03:21.857 INFO:teuthology.orchestra.run.smithi101.stdout: "short_zone_ids": [ 2023-07-29T01:03:21.857 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:21.857 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:03:21.857 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 2896287231 2023-07-29T01:03:21.857 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:21.857 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:21.858 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-29T01:03:21.858 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 815606810 2023-07-29T01:03:21.858 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:21.858 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:21.858 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T01:03:21.858 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 2009171363 2023-07-29T01:03:21.858 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:21.859 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:21.859 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-29T01:03:21.859 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 3105190295 2023-07-29T01:03:21.859 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:21.859 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:03:21.859 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:21.859 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zonegroup": "a", 2023-07-29T01:03:21.860 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:03:21.860 INFO:teuthology.orchestra.run.smithi101.stdout: "period_config": { 2023-07-29T01:03:21.860 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_quota": { 2023-07-29T01:03:21.860 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-29T01:03:21.860 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-29T01:03:21.860 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-29T01:03:21.861 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-29T01:03:21.861 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-29T01:03:21.861 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:21.861 INFO:teuthology.orchestra.run.smithi101.stdout: "user_quota": { 2023-07-29T01:03:21.861 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-29T01:03:21.861 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-29T01:03:21.862 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-29T01:03:21.862 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-29T01:03:21.862 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-29T01:03:21.862 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:21.862 INFO:teuthology.orchestra.run.smithi101.stdout: "user_ratelimit": { 2023-07-29T01:03:21.862 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-29T01:03:21.862 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-29T01:03:21.863 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-29T01:03:21.863 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-29T01:03:21.863 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-29T01:03:21.863 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:21.863 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_ratelimit": { 2023-07-29T01:03:21.863 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-29T01:03:21.863 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-29T01:03:21.864 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-29T01:03:21.864 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-29T01:03:21.864 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-29T01:03:21.864 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:21.864 INFO:teuthology.orchestra.run.smithi101.stdout: "anonymous_ratelimit": { 2023-07-29T01:03:21.864 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-29T01:03:21.865 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-29T01:03:21.865 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-29T01:03:21.865 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-29T01:03:21.865 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-29T01:03:21.865 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:21.866 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:21.866 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T01:03:21.866 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_name": "test-realm", 2023-07-29T01:03:21.866 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:03:21.866 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:03:21.867 INFO:rgw_multi.tests:create bucket zone=a1 name=hiiqos-65 2023-07-29T01:03:21.867 DEBUG:boto:path=/hiiqos-65/ 2023-07-29T01:03:21.867 DEBUG:boto:auth_path=/hiiqos-65/ 2023-07-29T01:03:21.868 DEBUG:boto:Method: PUT 2023-07-29T01:03:21.868 DEBUG:boto:Path: /hiiqos-65/ 2023-07-29T01:03:21.869 DEBUG:boto:Data: 2023-07-29T01:03:21.869 DEBUG:boto:Headers: {} 2023-07-29T01:03:21.869 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:03:21.870 DEBUG:boto:Port: 8000 2023-07-29T01:03:21.870 DEBUG:boto:Params: {} 2023-07-29T01:03:21.871 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-29T01:03:21.871 DEBUG:boto:Token: None 2023-07-29T01:03:21.872 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 01:03:21 GMT /hiiqos-65/ 2023-07-29T01:03:21.872 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lSdD2tIIzIJUvZvrnLbTAHcGIok= 2023-07-29T01:03:21.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:03:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lSdD2tIIzIJUvZvrnLbTAHcGIok=', 'Content-Length': '0'} 2023-07-29T01:03:26.895 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fed2426640c57ca5-0064c4655e-21ba-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:03:26 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:03:26.896 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'hiiqos-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:03:26.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'hiiqos-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:03:26.897 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket hiiqos-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:03:26.951 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:03:26.951 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:03:27.004 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-07-29T01:03:27.004 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'hiiqos-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:03:27.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'hiiqos-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:03:27.005 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket hiiqos-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:03:27.053 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:03:27.054 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:03:27.105 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-07-29T01:03:27.106 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'hiiqos-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:03:27.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'hiiqos-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:03:27.107 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-bucket --bucket hiiqos-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:03:27.153 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:03:27.153 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:03:27.197 INFO:teuthology.orchestra.run.smithi101.stdout:{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"} 2023-07-29T01:03:27.198 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'hiiqos-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:03:27.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'hiiqos-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:03:27.199 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket hiiqos-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:03:27.243 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:03:27.244 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:03:27.288 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-07-29T01:03:27.289 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'hiiqos-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:03:27.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'hiiqos-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:03:27.289 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --bucket hiiqos-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:03:27.341 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:03:27.341 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:03:27.389 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2023-07-29T01:03:27.389 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'hiiqos-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:03:27.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'hiiqos-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:03:27.390 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --bucket hiiqos-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:03:27.443 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:03:27.444 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:03:27.494 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2023-07-29T01:03:27.494 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=hiiqos-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:03:27.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=hiiqos-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:03:27.496 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=hiiqos-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:03:27.546 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:03:27.546 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:03:27.595 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-07-29T01:03:27.596 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'hiiqos-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:03:27.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'hiiqos-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:03:27.598 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-bucket --bucket hiiqos-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:03:27.643 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:03:27.643 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:03:27.688 INFO:teuthology.orchestra.run.smithi101.stdout:{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"} 2023-07-29T01:03:27.689 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:03:27.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:03:27.691 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:03:27.736 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:03:27.736 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:03:27.765 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T01:03:27.778 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T01:03:27.778 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:27.779 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:03:27.779 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:17.836402Z" 2023-07-29T01:03:27.779 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:27.779 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:27.779 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592607.499819_1357.1", 2023-07-29T01:03:27.779 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:03:27.499819Z" 2023-07-29T01:03:27.780 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:27.780 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:27.780 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:03:27.780 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T01:03:27.780 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:27.780 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:27.781 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:03:27.781 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:12.811406Z" 2023-07-29T01:03:27.781 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:27.781 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T01:03:27.782 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690591217.836402_294.1', 1: '1_1690592607.499819_1357.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:03:27.783 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T01:03:27.783 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:03:27.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:03:27.784 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:03:27.829 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:03:27.829 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:03:27.858 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:03:27.775+0000 7fab88267880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:03:27.859 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:03:27.776+0000 7fab88267880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:03:27.872 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:03:27.872 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:03:27.873 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:03:27.873 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:03:27.873 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:03:27.873 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:03:27.873 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:03:27.874 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:27.874 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:03:27.874 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:27.874 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:03:27.874 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:03:27.874 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:03:27.875 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:03:27.875 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:03:27.875 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:03:27.875 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:03:27.875 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:03:27.875 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:03:27.876 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:27.876 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:27.876 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:27.876 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:03:27.876 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:03:27.876 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:03:27.877 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592303.891548_1258.1", 2023-07-29T01:03:27.877 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:03:27.877 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:03:27.877 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:03:27.877 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:23.891548Z", 2023-07-29T01:03:27.877 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:03:27.878 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:27.878 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:27.878 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:27.879 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:03:27.879 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:03:27.879 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:03:27.879 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:03:27.879 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:03:27.880 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:03:27.880 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:03:27.880 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:03:27.880 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:03:27.880 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:27.881 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:27.881 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:27.881 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:03:27.881 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:03:27.881 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:03:27.881 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:03:27.882 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:03:27.882 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:03:27.882 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:03:27.882 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:03:27.882 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:03:27.882 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:27.883 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:27.883 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:03:27.883 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:27.883 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:03:27.883 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:03:27.883 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:03:27.884 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:27.884 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:03:27Z" 2023-07-29T01:03:27.884 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:03:27.885 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592303.891548_1258.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:23.891548Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:03:27Z" } 2023-07-29T01:03:27.886 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592303.891548_1258.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:23.891548Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:03:27.886 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592607.499819_1357.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:03:27.887 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592303.891548_1258.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:03:27.887 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592607.499819_1357.1 target=1_1690592303.891548_1258.1 2023-07-29T01:03:32.888 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:03:32.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:03:32.889 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:03:32.942 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:03:32.942 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:03:32.978 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:03:32.896+0000 7f0d9cf2f880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:03:32.979 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:03:32.897+0000 7f0d9cf2f880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:03:32.993 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:03:32.993 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:03:32.994 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:03:32.994 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:03:32.994 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:03:32.994 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:03:32.994 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:03:32.994 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:32.995 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:03:32.995 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:32.995 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:03:32.995 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:03:32.995 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:03:32.995 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:03:32.995 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:03:32.996 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:03:32.996 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:03:32.996 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:03:32.996 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:03:32.996 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:32.996 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:32.997 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:32.997 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:03:32.997 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:03:32.997 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:03:32.997 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592303.891548_1258.1", 2023-07-29T01:03:32.997 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:03:32.997 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:03:32.998 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:03:32.998 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:23.891548Z", 2023-07-29T01:03:32.998 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:03:32.998 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:32.998 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:32.998 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:32.998 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:03:32.999 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:03:32.999 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:03:32.999 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:03:32.999 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:03:32.999 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:03:32.999 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:03:32.999 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:03:33.000 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:03:33.000 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:33.000 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:33.000 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:33.001 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:03:33.001 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:03:33.001 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:03:33.001 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:03:33.001 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:03:33.001 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:03:33.001 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:03:33.002 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:03:33.002 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:03:33.002 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:33.002 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:33.002 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:03:33.002 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:33.003 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:03:33.003 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:03:33.003 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:03:33.004 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:33.004 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:03:32Z" 2023-07-29T01:03:33.004 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:03:33.005 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592303.891548_1258.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:23.891548Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:03:32Z" } 2023-07-29T01:03:33.005 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592303.891548_1258.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:23.891548Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:03:33.006 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592607.499819_1357.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:03:33.006 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592303.891548_1258.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:03:33.006 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592607.499819_1357.1 target=1_1690592303.891548_1258.1 2023-07-29T01:03:38.007 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:03:38.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:03:38.009 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:03:38.062 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:03:38.063 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:03:38.096 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:03:38.013+0000 7f4704535880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:03:38.096 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:03:38.014+0000 7f4704535880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:03:38.110 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:03:38.110 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:03:38.111 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:03:38.111 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:03:38.111 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:03:38.111 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:03:38.111 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:03:38.111 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:38.112 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:03:38.112 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:38.112 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:03:38.112 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:03:38.112 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:03:38.113 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:03:38.113 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:03:38.113 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:03:38.113 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:03:38.113 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:03:38.114 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:03:38.114 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:38.114 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:38.114 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:38.114 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:03:38.114 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:03:38.115 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:03:38.115 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592303.891548_1258.1", 2023-07-29T01:03:38.115 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:03:38.115 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:03:38.115 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:03:38.115 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:23.891548Z", 2023-07-29T01:03:38.115 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:03:38.116 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:38.116 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:38.116 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:38.116 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:03:38.116 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:03:38.117 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:03:38.117 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:03:38.117 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:03:38.117 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:03:38.117 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:03:38.117 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:03:38.117 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:03:38.118 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:38.118 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:38.118 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:38.118 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:03:38.118 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:03:38.119 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:03:38.119 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:03:38.119 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:03:38.119 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:03:38.119 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:03:38.120 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:03:38.120 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:03:38.120 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:38.120 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:38.120 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:03:38.121 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:38.121 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:03:38.121 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:03:38.121 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:03:38.121 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:38.121 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:03:38Z" 2023-07-29T01:03:38.122 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:03:38.123 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592303.891548_1258.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:23.891548Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:03:38Z" } 2023-07-29T01:03:38.123 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592303.891548_1258.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:23.891548Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:03:38.124 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592607.499819_1357.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:03:38.124 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592303.891548_1258.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:03:38.124 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592607.499819_1357.1 target=1_1690592303.891548_1258.1 2023-07-29T01:03:43.125 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:03:43.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:03:43.126 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:03:43.179 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:03:43.179 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:03:43.211 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:03:43.129+0000 7faab7726880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:03:43.212 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:03:43.130+0000 7faab7726880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:03:43.226 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:03:43.226 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:03:43.227 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:03:43.227 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:03:43.227 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:03:43.227 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:03:43.227 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:03:43.227 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:43.228 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:03:43.228 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:43.228 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:03:43.228 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:03:43.228 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:03:43.228 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:03:43.228 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:03:43.229 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:03:43.229 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:03:43.229 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:03:43.229 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:03:43.229 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:43.229 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:43.230 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:43.230 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:03:43.230 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:03:43.230 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:03:43.230 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592303.891548_1258.1", 2023-07-29T01:03:43.230 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:03:43.230 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:03:43.231 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:03:43.231 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:23.891548Z", 2023-07-29T01:03:43.231 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:03:43.231 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:43.231 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:43.231 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:43.231 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:03:43.232 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:03:43.232 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:03:43.232 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:03:43.232 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:03:43.232 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:03:43.232 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:03:43.232 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:03:43.233 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:03:43.233 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:43.233 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:43.233 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:43.233 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:03:43.233 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:03:43.233 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:03:43.234 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:03:43.234 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:03:43.234 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:03:43.234 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:03:43.234 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:03:43.234 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:03:43.234 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:43.235 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:43.235 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:03:43.235 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:43.235 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:03:43.235 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:03:43.235 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:03:43.235 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:43.236 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:03:43Z" 2023-07-29T01:03:43.236 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:03:43.237 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592303.891548_1258.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:23.891548Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:03:43Z" } 2023-07-29T01:03:43.237 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592303.891548_1258.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:23.891548Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:03:43.238 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592607.499819_1357.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:03:43.238 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592303.891548_1258.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:03:43.239 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592607.499819_1357.1 target=1_1690592303.891548_1258.1 2023-07-29T01:03:48.241 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:03:48.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:03:48.242 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:03:48.295 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:03:48.296 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:03:48.329 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:03:48.246+0000 7f0d88e14880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:03:48.330 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:03:48.247+0000 7f0d88e14880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:03:48.343 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:03:48.344 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:03:48.344 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:03:48.344 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:03:48.344 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:03:48.344 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:03:48.345 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:03:48.345 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:48.345 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:03:48.345 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:48.346 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:03:48.346 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:03:48.346 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:03:48.346 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:03:48.346 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:03:48.346 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:03:48.347 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:03:48.347 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:03:48.347 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:03:48.347 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:48.347 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:48.348 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:48.348 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:03:48.348 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:03:48.348 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:03:48.348 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592303.891548_1258.1", 2023-07-29T01:03:48.348 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:03:48.349 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:03:48.349 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:03:48.349 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:23.891548Z", 2023-07-29T01:03:48.349 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:03:48.349 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:48.349 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:48.349 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:48.350 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:03:48.350 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:03:48.350 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:03:48.350 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:03:48.350 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:03:48.350 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:03:48.350 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:03:48.351 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:03:48.351 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:03:48.351 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:48.351 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:48.351 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:48.352 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:03:48.352 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:03:48.352 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:03:48.352 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:03:48.352 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:03:48.352 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:03:48.353 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:03:48.353 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:03:48.353 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:03:48.353 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:48.353 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:48.353 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:03:48.353 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:48.354 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:03:48.354 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:03:48.354 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:03:48.354 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:48.354 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:03:48Z" 2023-07-29T01:03:48.354 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:03:48.356 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592303.891548_1258.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:23.891548Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:03:48Z" } 2023-07-29T01:03:48.357 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592303.891548_1258.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:23.891548Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:03:48.357 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592607.499819_1357.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:03:48.358 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592303.891548_1258.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:03:48.358 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592607.499819_1357.1 target=1_1690592303.891548_1258.1 2023-07-29T01:03:53.359 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:03:53.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:03:53.361 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:03:53.413 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:03:53.414 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:03:53.453 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:03:53.370+0000 7f4011e0f880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:03:53.453 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:03:53.371+0000 7f4011e0f880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:03:53.468 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:03:53.468 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:03:53.468 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:03:53.468 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:03:53.469 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:03:53.469 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:03:53.469 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:03:53.469 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:53.469 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:03:53.469 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:53.469 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:03:53.470 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:03:53.470 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:03:53.470 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:03:53.470 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:03:53.470 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:03:53.470 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:03:53.470 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:03:53.471 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:03:53.471 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:53.471 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:53.471 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:53.472 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:03:53.472 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:03:53.472 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:03:53.472 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592303.891548_1258.1", 2023-07-29T01:03:53.472 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:03:53.472 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:03:53.473 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:03:53.473 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:23.891548Z", 2023-07-29T01:03:53.473 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:03:53.473 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:53.473 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:53.473 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:53.474 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:03:53.474 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:03:53.474 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:03:53.474 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:03:53.474 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:03:53.474 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:03:53.474 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:03:53.475 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:03:53.475 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:03:53.475 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:53.475 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:53.475 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:53.475 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:03:53.475 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:03:53.476 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:03:53.476 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:03:53.476 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:03:53.476 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:03:53.476 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:03:53.476 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:03:53.477 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:03:53.477 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:53.477 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:53.477 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:03:53.477 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:53.478 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:03:53.478 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:03:53.478 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:03:53.478 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:53.479 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:03:53Z" 2023-07-29T01:03:53.479 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:03:53.480 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592303.891548_1258.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:23.891548Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:03:53Z" } 2023-07-29T01:03:53.480 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592303.891548_1258.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:23.891548Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:03:53.481 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592607.499819_1357.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:03:53.481 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592303.891548_1258.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:03:53.481 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592607.499819_1357.1 target=1_1690592303.891548_1258.1 2023-07-29T01:03:58.482 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:03:58.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:03:58.483 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:03:58.537 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:03:58.537 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:03:58.571 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:03:58.488+0000 7f97d0524880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:03:58.573 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:03:58.490+0000 7f97d0524880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:03:58.587 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:03:58.587 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:03:58.588 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:03:58.588 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:03:58.588 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:03:58.588 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:03:58.588 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:03:58.589 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:58.589 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:03:58.589 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:58.589 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:03:58.589 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:03:58.589 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:03:58.589 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:03:58.590 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:03:58.590 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:03:58.590 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:03:58.590 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:03:58.590 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:03:58.590 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:58.591 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:58.591 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:58.591 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:03:58.591 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:03:58.591 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:03:58.592 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592303.891548_1258.1", 2023-07-29T01:03:58.592 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:03:58.592 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:03:58.592 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:03:58.592 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:23.891548Z", 2023-07-29T01:03:58.592 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:03:58.592 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:58.593 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:58.593 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:58.593 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:03:58.593 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:03:58.594 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:03:58.594 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:03:58.594 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:03:58.594 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:03:58.594 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:03:58.595 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:03:58.595 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:03:58.595 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:58.595 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:58.595 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:03:58.595 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:03:58.595 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:03:58.596 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:03:58.596 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:03:58.596 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:03:58.596 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:03:58.597 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:03:58.597 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:03:58.597 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:03:58.597 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:58.597 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:03:58.597 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:03:58.598 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:58.598 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:03:58.598 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:03:58.598 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:03:58.598 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:03:58.598 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:03:58Z" 2023-07-29T01:03:58.599 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:03:58.600 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592303.891548_1258.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:23.891548Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:03:58Z" } 2023-07-29T01:03:58.600 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592303.891548_1258.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:23.891548Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:03:58.601 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592607.499819_1357.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:03:58.601 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592303.891548_1258.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:03:58.601 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592607.499819_1357.1 target=1_1690592303.891548_1258.1 2023-07-29T01:04:03.602 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:04:03.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:04:03.603 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:04:03.657 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:04:03.657 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:04:03.690 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:04:03.607+0000 7f674d711880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:04:03.691 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:04:03.608+0000 7f674d711880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:04:03.705 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:04:03.706 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:04:03.706 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:04:03.706 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:04:03.706 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:04:03.706 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:04:03.706 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:03.707 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:03.707 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:04:03.707 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:03.707 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:04:03.707 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:03.707 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:03.707 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:04:03.708 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:03.708 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:04:03.708 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:03.708 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:04:03.708 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:03.708 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:03.709 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:03.709 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:03.709 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:04:03.709 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:03.709 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:03.709 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592303.891548_1258.1", 2023-07-29T01:04:03.709 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:03.710 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:04:03.710 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:03.710 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:23.891548Z", 2023-07-29T01:04:03.710 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:03.710 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:03.710 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:03.710 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:03.711 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:04:03.711 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:03.711 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:03.711 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:04:03.711 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:03.711 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:04:03.712 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:03.712 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:04:03.712 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:03.712 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:03.712 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:03.712 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:03.712 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:04:03.713 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:03.713 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:03.713 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:04:03.713 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:03.713 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:04:03.713 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:03.713 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:04:03.714 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:03.714 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:03.714 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:03.714 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:04:03.714 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:03.714 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:04:03.715 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:04:03.715 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:04:03.715 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:03.715 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:04:03Z" 2023-07-29T01:04:03.715 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:04:03.716 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592303.891548_1258.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:23.891548Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:04:03Z" } 2023-07-29T01:04:03.717 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592303.891548_1258.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:23.891548Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:04:03.717 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592607.499819_1357.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:04:03.718 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592303.891548_1258.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:04:03.718 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592607.499819_1357.1 target=1_1690592303.891548_1258.1 2023-07-29T01:04:08.719 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:04:08.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:04:08.720 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:04:08.774 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:04:08.774 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:04:08.807 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:04:08.724+0000 7f492cb35880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:04:08.807 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:04:08.724+0000 7f492cb35880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:04:08.821 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:04:08.822 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:04:08.822 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:04:08.822 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:04:08.822 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:04:08.822 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:04:08.823 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:08.823 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:08.823 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:04:08.823 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:08.823 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:04:08.823 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:08.823 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:08.824 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:04:08.824 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:08.824 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:04:08.824 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:08.824 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:04:08.824 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:08.824 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:08.825 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:08.825 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:08.825 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:04:08.825 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:08.825 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:08.825 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592303.891548_1258.1", 2023-07-29T01:04:08.825 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:08.826 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:04:08.826 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:08.826 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:23.891548Z", 2023-07-29T01:04:08.826 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:08.826 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:08.826 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:08.826 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:08.827 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:04:08.827 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:08.827 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:08.827 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:04:08.827 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:08.827 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:04:08.827 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:08.828 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:04:08.828 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:08.828 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:08.828 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:08.828 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:08.828 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:04:08.829 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:08.829 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:08.829 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:04:08.829 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:08.829 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:04:08.829 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:08.829 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:04:08.829 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:08.830 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:08.830 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:08.830 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:04:08.830 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:08.830 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:04:08.830 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:04:08.830 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:04:08.831 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:08.831 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:04:08Z" 2023-07-29T01:04:08.831 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:04:08.832 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592303.891548_1258.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:23.891548Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:04:08Z" } 2023-07-29T01:04:08.832 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592303.891548_1258.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:23.891548Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:04:08.833 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592607.499819_1357.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:04:08.833 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592303.891548_1258.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:04:08.833 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592607.499819_1357.1 target=1_1690592303.891548_1258.1 2023-07-29T01:04:13.834 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:04:13.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:04:13.835 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:04:13.888 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:04:13.889 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:04:13.921 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:04:13.838+0000 7f14ad72a880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:04:13.922 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:04:13.839+0000 7f14ad72a880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:04:13.936 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:04:13.936 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:04:13.936 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:04:13.936 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:04:13.936 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:04:13.937 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:04:13.937 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:13.937 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:13.937 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:04:13.937 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:13.937 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:04:13.938 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:13.938 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:13.938 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:04:13.938 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:13.938 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:04:13.938 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:13.938 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:04:13.939 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:13.939 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:13.939 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:13.939 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:13.940 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:04:13.940 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:13.940 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:13.940 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592303.891548_1258.1", 2023-07-29T01:04:13.940 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:13.940 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:04:13.940 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:13.941 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:23.891548Z", 2023-07-29T01:04:13.941 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:13.941 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:13.941 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:13.941 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:13.941 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:04:13.942 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:13.942 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:13.942 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:04:13.942 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:13.942 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:04:13.942 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:13.942 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:04:13.943 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:13.943 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:13.943 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:13.943 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:13.943 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:04:13.943 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:13.943 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:13.944 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:04:13.944 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:13.944 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:04:13.944 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:13.944 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:04:13.944 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:13.944 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:13.945 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:13.945 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:04:13.945 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:13.945 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:04:13.945 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:04:13.945 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:04:13.946 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:13.946 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:04:13Z" 2023-07-29T01:04:13.946 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:04:13.947 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592303.891548_1258.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:23.891548Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:04:13Z" } 2023-07-29T01:04:13.947 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592303.891548_1258.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:23.891548Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:04:13.948 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592607.499819_1357.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:04:13.948 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592303.891548_1258.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:04:13.949 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592607.499819_1357.1 target=1_1690592303.891548_1258.1 2023-07-29T01:04:18.949 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:04:18.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:04:18.951 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:04:19.005 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:04:19.005 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:04:19.037 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:04:18.954+0000 7f4345722880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:04:19.038 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:04:18.955+0000 7f4345722880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:04:19.052 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:04:19.052 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:04:19.052 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:04:19.053 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:04:19.053 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:04:19.053 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:04:19.053 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:19.053 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:19.053 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:04:19.054 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:19.054 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:04:19.054 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:19.054 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:19.054 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:04:19.054 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:19.055 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:04:19.055 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:19.055 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:04:19.055 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:19.055 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:19.055 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:19.056 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:19.056 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:04:19.056 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:19.056 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:19.056 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592303.891548_1258.1", 2023-07-29T01:04:19.056 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:19.057 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:04:19.057 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:19.057 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:23.891548Z", 2023-07-29T01:04:19.057 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:19.057 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:19.057 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:19.057 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:19.058 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:04:19.058 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:19.058 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:19.058 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:04:19.058 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:19.058 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:04:19.059 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:19.059 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:04:19.059 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:19.059 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:19.059 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:19.059 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:19.060 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:04:19.060 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:19.060 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:19.060 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:04:19.060 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:19.060 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:04:19.061 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:19.061 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:04:19.061 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:19.061 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:19.061 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:19.061 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:04:19.061 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:19.062 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:04:19.062 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:04:19.062 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:04:19.062 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:19.062 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:04:18Z" 2023-07-29T01:04:19.062 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:04:19.063 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592303.891548_1258.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:23.891548Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:04:18Z" } 2023-07-29T01:04:19.064 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592303.891548_1258.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:23.891548Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:04:19.064 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592607.499819_1357.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:04:19.065 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592303.891548_1258.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:04:19.065 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592607.499819_1357.1 target=1_1690592303.891548_1258.1 2023-07-29T01:04:24.066 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:04:24.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:04:24.067 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:04:24.120 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:04:24.121 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:04:24.153 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:04:24.069+0000 7fc52e867880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:04:24.153 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:04:24.070+0000 7fc52e867880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:04:24.167 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:04:24.168 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:04:24.168 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:04:24.168 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:04:24.168 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:04:24.168 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:04:24.169 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:24.169 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:24.169 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:04:24.169 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:24.169 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:04:24.169 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:24.170 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:24.170 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:04:24.170 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:24.170 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:04:24.170 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:24.170 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:04:24.170 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:24.171 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:24.171 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:24.171 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:24.171 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:04:24.171 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:24.171 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:24.172 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592303.891548_1258.1", 2023-07-29T01:04:24.172 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:24.172 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:04:24.172 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:24.172 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:23.891548Z", 2023-07-29T01:04:24.173 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:24.173 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:24.173 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:24.173 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:24.174 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:04:24.174 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:24.174 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:24.174 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:04:24.175 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:24.175 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:04:24.175 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:24.175 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:04:24.175 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:24.175 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:24.175 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:24.176 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:24.176 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:04:24.176 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:24.176 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:24.176 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:04:24.177 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:24.177 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:04:24.177 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:24.177 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:04:24.177 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:24.178 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:24.178 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:24.178 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:04:24.178 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:24.178 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:04:24.178 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:04:24.178 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:04:24.179 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:24.179 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:04:24Z" 2023-07-29T01:04:24.179 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:04:24.180 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592303.891548_1258.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:23.891548Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:04:24Z" } 2023-07-29T01:04:24.180 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592303.891548_1258.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:23.891548Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:04:24.181 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592607.499819_1357.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:04:24.181 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592303.891548_1258.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:04:24.182 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592607.499819_1357.1 target=1_1690592303.891548_1258.1 2023-07-29T01:04:29.183 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:04:29.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:04:29.184 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:04:29.237 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:04:29.238 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:04:29.269 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:04:29.186+0000 7ff2ea520880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:04:29.270 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:04:29.187+0000 7ff2ea520880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:04:29.285 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:04:29.285 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:04:29.286 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:04:29.286 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:04:29.287 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:04:29.287 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:04:29.287 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:29.288 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:29.288 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:04:29.288 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:29.289 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:04:29.289 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:29.289 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:29.290 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:04:29.290 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:29.290 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:04:29.291 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:29.291 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:04:29.291 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:29.292 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:29.292 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:29.292 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:29.293 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:04:29.293 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:29.293 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:29.294 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592303.891548_1258.1", 2023-07-29T01:04:29.294 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:29.294 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:04:29.294 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:29.294 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:23.891548Z", 2023-07-29T01:04:29.295 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:29.295 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:29.295 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:29.296 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:29.296 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:04:29.296 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:29.296 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:29.296 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:04:29.297 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:29.297 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:04:29.297 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:29.297 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:04:29.298 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:29.298 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:29.298 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:29.298 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:29.299 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:04:29.299 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:29.299 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:29.300 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:04:29.300 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:29.300 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:04:29.301 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:29.301 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:04:29.302 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:29.302 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:29.302 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:29.303 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:04:29.303 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:29.303 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:04:29.304 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:04:29.304 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:04:29.304 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:29.305 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:04:29Z" 2023-07-29T01:04:29.305 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:04:29.307 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592303.891548_1258.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:23.891548Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:04:29Z" } 2023-07-29T01:04:29.307 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592303.891548_1258.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:23.891548Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:04:29.308 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592607.499819_1357.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:04:29.308 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592303.891548_1258.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:04:29.309 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592607.499819_1357.1 target=1_1690592303.891548_1258.1 2023-07-29T01:04:34.310 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:04:34.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:04:34.311 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:04:34.364 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:04:34.365 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:04:34.397 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:04:34.314+0000 7f9989d4b880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:04:34.398 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:04:34.315+0000 7f9989d4b880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:04:34.412 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:04:34.413 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:04:34.413 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:04:34.413 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:04:34.413 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:04:34.413 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:04:34.414 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:34.414 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:34.414 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:04:34.414 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:34.414 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:04:34.415 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:34.415 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:34.415 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:04:34.416 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:34.416 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:04:34.416 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:34.416 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:04:34.416 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:34.416 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:34.417 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:34.417 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:34.417 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:04:34.417 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:34.417 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:34.417 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592303.891548_1258.1", 2023-07-29T01:04:34.418 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:34.418 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:04:34.418 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:34.418 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:23.891548Z", 2023-07-29T01:04:34.418 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:34.419 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:34.419 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:34.419 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:34.419 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:04:34.419 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:34.419 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:34.419 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:04:34.420 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:34.420 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:04:34.420 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:34.420 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:04:34.420 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:34.420 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:34.421 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:34.421 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:34.421 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:04:34.421 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:34.421 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:34.421 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:04:34.421 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:34.422 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:04:34.422 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:34.422 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:04:34.422 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:34.422 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:34.422 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:34.422 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:04:34.423 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:34.423 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:04:34.423 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:04:34.423 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:04:34.423 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:34.423 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:04:34Z" 2023-07-29T01:04:34.423 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:04:34.425 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592303.891548_1258.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:23.891548Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:04:34Z" } 2023-07-29T01:04:34.425 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592303.891548_1258.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:23.891548Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:04:34.426 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592607.499819_1357.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:04:34.426 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592303.891548_1258.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:04:34.426 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592607.499819_1357.1 target=1_1690592303.891548_1258.1 2023-07-29T01:04:39.427 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:04:39.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:04:39.429 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:04:39.483 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:04:39.483 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:04:39.516 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:04:39.433+0000 7fe049926880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:04:39.517 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:04:39.434+0000 7fe049926880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:04:39.530 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:04:39.531 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:04:39.531 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:04:39.531 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:04:39.532 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:04:39.532 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:04:39.532 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:39.533 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:39.533 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:04:39.533 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:39.533 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:04:39.533 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:39.533 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:39.534 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:04:39.534 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:39.534 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:04:39.534 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:39.535 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:04:39.535 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:39.535 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:39.535 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:39.535 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:39.535 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:04:39.536 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:39.536 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:39.536 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592607.499819_1357.1", 2023-07-29T01:04:39.536 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:39.536 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:04:39.536 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:39.537 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:03:27.499819Z", 2023-07-29T01:04:39.537 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:39.537 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:39.537 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:39.537 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:39.537 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:04:39.537 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:39.538 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:39.538 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:04:39.538 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:39.538 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:04:39.538 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:39.538 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:04:39.539 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:39.539 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:39.539 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:39.539 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:39.539 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:04:39.539 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:39.539 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:39.540 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:04:39.540 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:39.540 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:04:39.540 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:39.540 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:04:39.541 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:39.541 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:39.541 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:39.541 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:04:39.541 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:39.541 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:04:39.541 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:04:39.542 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:04:39.542 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:39.542 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:04:39Z" 2023-07-29T01:04:39.542 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:04:39.543 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592607.499819_1357.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:03:27.499819Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:04:39Z" } 2023-07-29T01:04:39.544 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592607.499819_1357.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:03:27.499819Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:04:39.544 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592607.499819_1357.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:04:39.545 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592607.499819_1357.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:04:39.545 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T01:04:39.545 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--bucket=hiiqos-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:04:39.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--bucket=hiiqos-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:04:39.546 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-bucket --pipe-id sync-pipe --bucket=hiiqos-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:04:39.596 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:04:39.596 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:04:39.643 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2023-07-29T01:04:39.643 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'hiiqos-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:04:39.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'hiiqos-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:04:39.645 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-bucket --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --bucket hiiqos-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:04:39.697 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:04:39.697 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:04:39.745 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2023-07-29T01:04:39.745 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'hiiqos-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:04:39.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'hiiqos-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:04:39.746 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-bucket --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --bucket hiiqos-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:04:39.797 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:04:39.797 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:04:39.845 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-07-29T01:04:39.846 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'hiiqos-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:04:39.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'hiiqos-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:04:39.847 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket hiiqos-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:04:39.892 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:04:39.893 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:04:39.940 INFO:teuthology.orchestra.run.smithi101.stdout:{"sync_policy":{"groups":[]}} 2023-07-29T01:04:39.941 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'hiiqos-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:04:39.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'hiiqos-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:04:39.941 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket hiiqos-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:04:39.985 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:04:39.985 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:04:40.033 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[]} 2023-07-29T01:04:40.033 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:04:40.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:04:40.034 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:04:40.118 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:04:40.119 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:04:40.148 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T01:04:40.161 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T01:04:40.162 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:40.162 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:04:40.162 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:17.836402Z" 2023-07-29T01:04:40.162 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:40.162 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:40.163 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:04:40.163 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:04:39.844113Z" 2023-07-29T01:04:40.163 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:40.163 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:40.163 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:04:40.163 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T01:04:40.164 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:40.164 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:40.164 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:04:40.164 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:12.811406Z" 2023-07-29T01:04:40.164 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:40.164 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T01:04:40.165 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:04:40.166 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T01:04:40.166 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:04:40.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:04:40.167 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:04:40.213 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:04:40.213 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:04:40.241 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:04:40.158+0000 7f105af26880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:04:40.242 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:04:40.159+0000 7f105af26880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:04:40.255 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:04:40.256 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:04:40.256 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:04:40.256 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:04:40.256 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:04:40.257 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:04:40.257 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:40.257 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:40.257 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:04:40.257 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:40.257 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:04:40.257 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:40.258 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:40.258 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:04:40.258 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:40.258 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:04:40.258 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:40.258 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:04:40.259 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:40.259 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:40.259 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:40.259 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:40.259 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:04:40.260 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:40.260 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:40.260 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592607.499819_1357.1", 2023-07-29T01:04:40.260 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:40.260 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:04:40.260 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:40.261 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:03:27.499819Z", 2023-07-29T01:04:40.261 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:40.261 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:40.261 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:40.261 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:40.262 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:04:40.262 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:40.262 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:40.262 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:04:40.263 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:40.263 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:04:40.263 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:40.263 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:04:40.263 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:40.264 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:40.264 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:40.265 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:40.265 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:04:40.265 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:40.265 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:40.265 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:04:40.265 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:40.266 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:04:40.266 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:40.266 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:04:40.266 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:40.266 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:40.266 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:40.266 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:04:40.267 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:40.267 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:04:40.267 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:04:40.267 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:04:40.267 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:40.267 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:04:40Z" 2023-07-29T01:04:40.268 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:04:40.269 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592607.499819_1357.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:03:27.499819Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:04:40Z" } 2023-07-29T01:04:40.269 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592607.499819_1357.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:03:27.499819Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:04:40.270 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:04:40.270 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592607.499819_1357.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:04:40.270 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592679.844113_1377.1 target=1_1690592607.499819_1357.1 2023-07-29T01:04:45.271 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:04:45.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:04:45.273 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:04:45.325 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:04:45.326 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:04:45.357 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:04:45.274+0000 7ff430a2d880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:04:45.358 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:04:45.275+0000 7ff430a2d880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:04:45.372 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:04:45.372 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:04:45.372 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:04:45.372 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:04:45.373 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:04:45.373 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:04:45.373 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:45.373 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:45.373 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:04:45.374 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:45.374 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:04:45.374 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:45.374 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:45.374 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:04:45.374 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:45.374 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:04:45.375 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:45.375 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:04:45.375 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:45.375 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:45.375 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:45.375 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:45.376 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:04:45.376 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:45.376 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:45.376 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592607.499819_1357.1", 2023-07-29T01:04:45.376 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:45.377 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:04:45.377 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:45.377 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:03:27.499819Z", 2023-07-29T01:04:45.377 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:45.378 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:45.378 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:45.378 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:45.378 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:04:45.378 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:45.379 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:45.379 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:04:45.379 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:45.379 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:04:45.380 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:45.380 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:04:45.380 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:45.380 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:45.381 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:45.381 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:45.381 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:04:45.381 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:45.381 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:45.381 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:04:45.381 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:45.382 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:04:45.382 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:45.382 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:04:45.382 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:45.382 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:45.382 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:45.382 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:04:45.383 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:45.383 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:04:45.383 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:04:45.383 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:04:45.383 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:45.383 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:04:45Z" 2023-07-29T01:04:45.384 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:04:45.385 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592607.499819_1357.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:03:27.499819Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:04:45Z" } 2023-07-29T01:04:45.385 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592607.499819_1357.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:03:27.499819Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:04:45.386 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:04:45.386 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592607.499819_1357.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:04:45.386 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592679.844113_1377.1 target=1_1690592607.499819_1357.1 2023-07-29T01:04:50.388 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:04:50.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:04:50.389 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:04:50.442 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:04:50.442 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:04:50.475 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:04:50.391+0000 7f32d9d2a880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:04:50.476 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:04:50.392+0000 7f32d9d2a880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:04:50.489 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:04:50.490 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:04:50.490 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:04:50.491 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:04:50.491 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:04:50.491 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:04:50.491 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:50.492 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:50.492 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:04:50.492 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:50.493 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:04:50.493 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:50.493 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:50.494 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:04:50.494 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:50.494 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:04:50.494 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:50.495 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:04:50.495 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:50.495 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:50.496 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:50.496 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:50.496 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:04:50.496 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:50.497 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:50.497 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:04:50.497 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:50.497 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:04:50.497 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:50.498 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:04:50.498 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:50.498 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:50.499 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:50.499 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:50.499 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:04:50.500 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:50.500 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:50.500 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:04:50.501 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:50.501 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:04:50.501 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:50.501 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:04:50.502 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:50.502 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:50.502 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:50.503 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:50.503 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:04:50.503 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:50.504 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:50.504 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:04:50.504 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:50.504 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:04:50.504 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:50.505 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:04:50.505 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:50.505 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:50.505 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:50.505 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:04:50.506 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:50.506 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:04:50.506 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:04:50.506 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:04:50.506 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:50.506 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:04:50Z" 2023-07-29T01:04:50.506 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:04:50.507 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592679.844113_1377.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:04:39.844113Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:04:50Z" } 2023-07-29T01:04:50.508 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:04:50.509 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:04:50.509 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:04:50.509 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T01:04:50.509 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:04:50.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:04:50.510 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:04:50.559 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:04:50.559 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:04:50.604 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2023-07-29T01:04:50.605 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:04:50.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:04:50.606 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:04:50.657 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:04:50.657 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:04:50.702 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2023-07-29T01:04:50.703 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:04:50.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:04:50.704 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:04:50.752 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:04:50.752 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:04:50.797 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-07-29T01:04:50.797 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:04:50.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:04:50.798 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:04:50.843 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:04:50.843 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:04:50.888 INFO:teuthology.orchestra.run.smithi101.stdout:{"sync_policy":{"groups":[]}} 2023-07-29T01:04:50.888 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:04:50.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:04:50.889 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:04:50.931 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:04:50.931 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:04:50.975 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[]} 2023-07-29T01:04:50.976 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:04:50.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:04:50.977 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:04:51.060 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:04:51.061 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:04:51.112 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:04:51.112 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:04:51.112 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 14, 2023-07-29T01:04:51.112 INFO:teuthology.orchestra.run.smithi101.stdout: "predecessor_uuid": "a0987484-ce37-4520-a564-54b7ea8c5251", 2023-07-29T01:04:51.112 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": [], 2023-07-29T01:04:51.113 INFO:teuthology.orchestra.run.smithi101.stdout: "period_map": { 2023-07-29T01:04:51.113 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:04:51.113 INFO:teuthology.orchestra.run.smithi101.stdout: "zonegroups": [ 2023-07-29T01:04:51.113 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:51.113 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "a", 2023-07-29T01:04:51.113 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a", 2023-07-29T01:04:51.113 INFO:teuthology.orchestra.run.smithi101.stdout: "api_name": "", 2023-07-29T01:04:51.114 INFO:teuthology.orchestra.run.smithi101.stdout: "is_master": true, 2023-07-29T01:04:51.114 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:04:51.114 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-29T01:04:51.114 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:04:51.114 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames": [], 2023-07-29T01:04:51.114 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames_s3website": [], 2023-07-29T01:04:51.115 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:04:51.115 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:04:51.115 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:51.115 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:04:51.116 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a1", 2023-07-29T01:04:51.116 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:04:51.116 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-29T01:04:51.117 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:04:51.117 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T01:04:51.117 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T01:04:51.117 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T01:04:51.117 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T01:04:51.117 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T01:04:51.117 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T01:04:51.118 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T01:04:51.118 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T01:04:51.118 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T01:04:51.118 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T01:04:51.118 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:04:51.118 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:04:51.119 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:51.119 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:51.119 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-29T01:04:51.119 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a2", 2023-07-29T01:04:51.119 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:04:51.119 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8001" 2023-07-29T01:04:51.119 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:04:51.120 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T01:04:51.120 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T01:04:51.120 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T01:04:51.120 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T01:04:51.120 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T01:04:51.120 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T01:04:51.121 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T01:04:51.121 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T01:04:51.121 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T01:04:51.121 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T01:04:51.121 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:04:51.121 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:04:51.121 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:51.122 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:04:51.122 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_targets": [ 2023-07-29T01:04:51.122 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:51.123 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default-placement", 2023-07-29T01:04:51.123 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [], 2023-07-29T01:04:51.123 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": [ 2023-07-29T01:04:51.124 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD" 2023-07-29T01:04:51.124 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:04:51.124 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:51.125 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:04:51.125 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "default-placement", 2023-07-29T01:04:51.126 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T01:04:51.126 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_policy": { 2023-07-29T01:04:51.126 INFO:teuthology.orchestra.run.smithi101.stdout: "groups": [] 2023-07-29T01:04:51.126 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:51.127 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled_features": [ 2023-07-29T01:04:51.127 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:04:51.127 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:04:51.128 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:51.128 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:51.128 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "b", 2023-07-29T01:04:51.129 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b", 2023-07-29T01:04:51.129 INFO:teuthology.orchestra.run.smithi101.stdout: "api_name": "", 2023-07-29T01:04:51.129 INFO:teuthology.orchestra.run.smithi101.stdout: "is_master": false, 2023-07-29T01:04:51.130 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:04:51.130 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-29T01:04:51.130 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:04:51.130 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames": [], 2023-07-29T01:04:51.130 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames_s3website": [], 2023-07-29T01:04:51.130 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T01:04:51.131 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:04:51.131 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:51.132 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T01:04:51.132 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b1", 2023-07-29T01:04:51.132 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:04:51.133 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-29T01:04:51.133 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:04:51.133 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T01:04:51.133 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T01:04:51.133 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T01:04:51.133 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T01:04:51.133 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T01:04:51.134 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T01:04:51.134 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T01:04:51.134 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T01:04:51.134 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T01:04:51.134 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T01:04:51.135 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:04:51.135 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:04:51.135 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:51.136 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:51.136 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-29T01:04:51.136 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b2", 2023-07-29T01:04:51.136 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:04:51.136 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8001" 2023-07-29T01:04:51.136 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:04:51.137 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T01:04:51.137 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T01:04:51.137 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T01:04:51.137 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T01:04:51.137 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T01:04:51.137 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T01:04:51.137 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T01:04:51.138 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T01:04:51.138 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T01:04:51.138 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T01:04:51.138 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:04:51.138 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:04:51.138 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:51.139 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:04:51.139 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_targets": [ 2023-07-29T01:04:51.139 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:51.139 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default-placement", 2023-07-29T01:04:51.139 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [], 2023-07-29T01:04:51.139 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": [ 2023-07-29T01:04:51.139 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD" 2023-07-29T01:04:51.140 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:04:51.140 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:51.140 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:04:51.140 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "default-placement", 2023-07-29T01:04:51.140 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T01:04:51.140 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_policy": { 2023-07-29T01:04:51.140 INFO:teuthology.orchestra.run.smithi101.stdout: "groups": [] 2023-07-29T01:04:51.141 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:51.141 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled_features": [ 2023-07-29T01:04:51.141 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:04:51.141 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:04:51.141 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:51.141 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:04:51.141 INFO:teuthology.orchestra.run.smithi101.stdout: "short_zone_ids": [ 2023-07-29T01:04:51.142 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:51.142 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:04:51.142 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 2896287231 2023-07-29T01:04:51.142 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:51.142 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:51.142 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-29T01:04:51.142 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 815606810 2023-07-29T01:04:51.143 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:51.143 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:51.143 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T01:04:51.143 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 2009171363 2023-07-29T01:04:51.143 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:51.143 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:51.143 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-29T01:04:51.144 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 3105190295 2023-07-29T01:04:51.144 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:51.144 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:04:51.144 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:51.144 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zonegroup": "a", 2023-07-29T01:04:51.144 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:04:51.144 INFO:teuthology.orchestra.run.smithi101.stdout: "period_config": { 2023-07-29T01:04:51.145 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_quota": { 2023-07-29T01:04:51.145 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-29T01:04:51.145 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-29T01:04:51.145 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-29T01:04:51.145 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-29T01:04:51.145 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-29T01:04:51.146 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:51.146 INFO:teuthology.orchestra.run.smithi101.stdout: "user_quota": { 2023-07-29T01:04:51.146 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-29T01:04:51.146 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-29T01:04:51.146 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-29T01:04:51.146 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-29T01:04:51.146 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-29T01:04:51.147 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:51.147 INFO:teuthology.orchestra.run.smithi101.stdout: "user_ratelimit": { 2023-07-29T01:04:51.147 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-29T01:04:51.147 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-29T01:04:51.147 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-29T01:04:51.147 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-29T01:04:51.147 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-29T01:04:51.148 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:51.148 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_ratelimit": { 2023-07-29T01:04:51.148 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-29T01:04:51.148 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-29T01:04:51.148 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-29T01:04:51.148 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-29T01:04:51.148 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-29T01:04:51.149 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:51.149 INFO:teuthology.orchestra.run.smithi101.stdout: "anonymous_ratelimit": { 2023-07-29T01:04:51.149 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-29T01:04:51.149 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-29T01:04:51.149 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-29T01:04:51.149 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-29T01:04:51.149 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-29T01:04:51.150 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:51.150 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:51.150 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T01:04:51.150 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_name": "test-realm", 2023-07-29T01:04:51.150 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:51.150 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:04:51.151 INFO:tasks.rgw_multisite_tests:test_sync_policy_config_zonegroup: ... ok 2023-07-29T01:04:51.152 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:04:51.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:04:51.153 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:04:51.211 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:04:51.211 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:04:51.295 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T01:04:51.351 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T01:04:51.351 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:51.351 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:04:51.351 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:17.836402Z" 2023-07-29T01:04:51.351 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:51.352 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:51.352 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:04:51.352 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:04:39.844113Z" 2023-07-29T01:04:51.352 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:51.352 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:51.352 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:04:51.353 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T01:04:51.353 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:51.353 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:51.353 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:04:51.353 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:12.811406Z" 2023-07-29T01:04:51.353 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:51.354 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T01:04:51.354 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:04:51.355 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T01:04:51.355 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:04:51.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:04:51.356 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:04:51.468 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:04:51.469 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:04:51.548 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:04:51.464+0000 7feea0922880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:04:51.550 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:04:51.466+0000 7feea0922880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:04:51.586 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:04:51.586 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:04:51.587 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:04:51.587 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:04:51.587 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:04:51.588 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:04:51.588 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:51.588 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:51.588 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:04:51.589 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:51.589 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:04:51.589 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:51.589 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:51.589 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:04:51.589 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:51.590 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:04:51.590 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:51.590 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:04:51.590 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:51.590 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:51.590 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:51.590 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:51.591 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:04:51.591 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:51.591 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:51.591 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:04:51.591 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:51.591 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:04:51.591 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:51.592 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:04:51.592 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:51.592 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:51.592 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:51.592 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:51.592 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:04:51.593 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:51.593 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:51.593 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:04:51.593 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:51.593 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:04:51.593 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:51.594 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:04:51.594 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:51.594 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:51.594 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:51.594 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:51.594 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:04:51.594 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:51.595 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:51.595 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:04:51.595 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:51.595 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:04:51.595 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:51.595 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:04:51.596 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:51.596 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:51.596 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:51.596 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:04:51.596 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:51.596 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:04:51.597 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:04:51.597 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:04:51.597 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:51.597 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:04:51Z" 2023-07-29T01:04:51.597 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:04:51.598 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592679.844113_1377.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:04:39.844113Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:04:51Z" } 2023-07-29T01:04:51.599 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:04:51.599 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:04:51.599 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:04:51.600 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T01:04:51.600 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:04:51.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:04:51.601 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:04:51.682 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:04:51.683 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:04:51.749 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[]} 2023-07-29T01:04:51.750 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:04:51.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:04:51.750 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:04:51.809 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:04:51.810 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:04:51.868 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-07-29T01:04:51.869 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:04:51.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:04:51.869 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:04:51.918 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:04:51.919 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:04:51.963 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2023-07-29T01:04:51.964 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:04:51.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:04:51.964 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:04:52.013 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:04:52.013 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:04:52.060 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-07-29T01:04:52.060 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:04:52.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:04:52.061 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:04:52.107 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:04:52.107 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:04:52.152 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-07-29T01:04:52.153 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:04:52.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:04:52.153 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:04:52.198 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:04:52.198 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:04:52.249 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:04:52.249 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:04:52.249 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 15, 2023-07-29T01:04:52.250 INFO:teuthology.orchestra.run.smithi101.stdout: "predecessor_uuid": "a0987484-ce37-4520-a564-54b7ea8c5251", 2023-07-29T01:04:52.250 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": [], 2023-07-29T01:04:52.250 INFO:teuthology.orchestra.run.smithi101.stdout: "period_map": { 2023-07-29T01:04:52.250 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:04:52.250 INFO:teuthology.orchestra.run.smithi101.stdout: "zonegroups": [ 2023-07-29T01:04:52.251 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:52.251 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "a", 2023-07-29T01:04:52.251 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a", 2023-07-29T01:04:52.251 INFO:teuthology.orchestra.run.smithi101.stdout: "api_name": "", 2023-07-29T01:04:52.251 INFO:teuthology.orchestra.run.smithi101.stdout: "is_master": true, 2023-07-29T01:04:52.251 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:04:52.252 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-29T01:04:52.252 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:04:52.252 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames": [], 2023-07-29T01:04:52.252 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames_s3website": [], 2023-07-29T01:04:52.252 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:04:52.252 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:04:52.252 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:52.253 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:04:52.253 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a1", 2023-07-29T01:04:52.253 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:04:52.253 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-29T01:04:52.253 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:04:52.253 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T01:04:52.254 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T01:04:52.254 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T01:04:52.254 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T01:04:52.254 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T01:04:52.254 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T01:04:52.254 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T01:04:52.254 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T01:04:52.255 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T01:04:52.255 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T01:04:52.255 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:04:52.255 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:04:52.255 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:52.255 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:52.255 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-29T01:04:52.256 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a2", 2023-07-29T01:04:52.256 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:04:52.256 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8001" 2023-07-29T01:04:52.256 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:04:52.256 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T01:04:52.256 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T01:04:52.257 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T01:04:52.257 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T01:04:52.257 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T01:04:52.257 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T01:04:52.257 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T01:04:52.257 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T01:04:52.257 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T01:04:52.258 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T01:04:52.258 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:04:52.258 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:04:52.258 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:52.258 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:04:52.258 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_targets": [ 2023-07-29T01:04:52.258 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:52.259 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default-placement", 2023-07-29T01:04:52.259 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [], 2023-07-29T01:04:52.259 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": [ 2023-07-29T01:04:52.259 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD" 2023-07-29T01:04:52.259 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:04:52.259 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:52.259 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:04:52.260 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "default-placement", 2023-07-29T01:04:52.260 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T01:04:52.260 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_policy": { 2023-07-29T01:04:52.260 INFO:teuthology.orchestra.run.smithi101.stdout: "groups": [ 2023-07-29T01:04:52.260 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:52.260 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "sync-group", 2023-07-29T01:04:52.260 INFO:teuthology.orchestra.run.smithi101.stdout: "data_flow": { 2023-07-29T01:04:52.261 INFO:teuthology.orchestra.run.smithi101.stdout: "symmetrical": [ 2023-07-29T01:04:52.261 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:52.261 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "sync-flow1", 2023-07-29T01:04:52.261 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:04:52.261 INFO:teuthology.orchestra.run.smithi101.stdout: "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:04:52.261 INFO:teuthology.orchestra.run.smithi101.stdout: "59d1bb30-3b1c-476f-9111-052c1ea0b711" 2023-07-29T01:04:52.261 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:04:52.262 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:52.262 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:04:52.262 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:52.262 INFO:teuthology.orchestra.run.smithi101.stdout: "pipes": [ 2023-07-29T01:04:52.262 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:52.262 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "sync-pipe", 2023-07-29T01:04:52.262 INFO:teuthology.orchestra.run.smithi101.stdout: "source": { 2023-07-29T01:04:52.263 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket": "*", 2023-07-29T01:04:52.263 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:04:52.263 INFO:teuthology.orchestra.run.smithi101.stdout: "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:04:52.263 INFO:teuthology.orchestra.run.smithi101.stdout: "59d1bb30-3b1c-476f-9111-052c1ea0b711" 2023-07-29T01:04:52.263 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:04:52.263 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:52.264 INFO:teuthology.orchestra.run.smithi101.stdout: "dest": { 2023-07-29T01:04:52.264 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket": "*", 2023-07-29T01:04:52.264 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:04:52.264 INFO:teuthology.orchestra.run.smithi101.stdout: "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:04:52.264 INFO:teuthology.orchestra.run.smithi101.stdout: "59d1bb30-3b1c-476f-9111-052c1ea0b711" 2023-07-29T01:04:52.264 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:04:52.264 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:52.265 INFO:teuthology.orchestra.run.smithi101.stdout: "params": { 2023-07-29T01:04:52.265 INFO:teuthology.orchestra.run.smithi101.stdout: "source": { 2023-07-29T01:04:52.265 INFO:teuthology.orchestra.run.smithi101.stdout: "filter": { 2023-07-29T01:04:52.265 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [] 2023-07-29T01:04:52.265 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:52.265 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:52.265 INFO:teuthology.orchestra.run.smithi101.stdout: "dest": {}, 2023-07-29T01:04:52.266 INFO:teuthology.orchestra.run.smithi101.stdout: "priority": 0, 2023-07-29T01:04:52.266 INFO:teuthology.orchestra.run.smithi101.stdout: "mode": "system", 2023-07-29T01:04:52.266 INFO:teuthology.orchestra.run.smithi101.stdout: "user": "" 2023-07-29T01:04:52.266 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:52.266 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:52.266 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:04:52.267 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "enabled" 2023-07-29T01:04:52.267 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:52.267 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:04:52.267 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:52.267 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled_features": [ 2023-07-29T01:04:52.267 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:04:52.267 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:04:52.268 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:52.268 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:52.268 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "b", 2023-07-29T01:04:52.268 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b", 2023-07-29T01:04:52.268 INFO:teuthology.orchestra.run.smithi101.stdout: "api_name": "", 2023-07-29T01:04:52.268 INFO:teuthology.orchestra.run.smithi101.stdout: "is_master": false, 2023-07-29T01:04:52.268 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:04:52.269 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-29T01:04:52.269 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:04:52.269 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames": [], 2023-07-29T01:04:52.269 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames_s3website": [], 2023-07-29T01:04:52.269 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T01:04:52.269 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:04:52.269 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:52.270 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T01:04:52.270 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b1", 2023-07-29T01:04:52.270 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:04:52.270 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-29T01:04:52.270 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:04:52.270 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T01:04:52.270 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T01:04:52.271 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T01:04:52.271 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T01:04:52.271 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T01:04:52.271 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T01:04:52.271 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T01:04:52.271 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T01:04:52.271 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T01:04:52.272 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T01:04:52.272 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:04:52.272 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:04:52.272 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:52.272 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:52.272 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-29T01:04:52.273 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b2", 2023-07-29T01:04:52.273 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:04:52.273 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8001" 2023-07-29T01:04:52.273 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:04:52.273 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T01:04:52.273 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T01:04:52.273 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T01:04:52.274 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T01:04:52.274 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T01:04:52.274 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T01:04:52.274 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T01:04:52.274 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T01:04:52.274 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T01:04:52.274 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T01:04:52.275 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:04:52.275 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:04:52.275 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:52.275 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:04:52.275 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_targets": [ 2023-07-29T01:04:52.275 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:52.275 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default-placement", 2023-07-29T01:04:52.276 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [], 2023-07-29T01:04:52.276 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": [ 2023-07-29T01:04:52.276 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD" 2023-07-29T01:04:52.276 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:04:52.276 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:52.276 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:04:52.276 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "default-placement", 2023-07-29T01:04:52.277 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T01:04:52.277 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_policy": { 2023-07-29T01:04:52.277 INFO:teuthology.orchestra.run.smithi101.stdout: "groups": [] 2023-07-29T01:04:52.277 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:52.277 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled_features": [ 2023-07-29T01:04:52.277 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:04:52.277 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:04:52.278 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:52.278 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:04:52.278 INFO:teuthology.orchestra.run.smithi101.stdout: "short_zone_ids": [ 2023-07-29T01:04:52.278 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:52.278 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:04:52.278 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 2896287231 2023-07-29T01:04:52.278 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:52.279 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:52.279 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-29T01:04:52.279 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 815606810 2023-07-29T01:04:52.280 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:52.281 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:52.281 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T01:04:52.281 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 2009171363 2023-07-29T01:04:52.281 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:52.281 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:52.281 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-29T01:04:52.281 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 3105190295 2023-07-29T01:04:52.282 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:52.282 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:04:52.282 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:52.282 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zonegroup": "a", 2023-07-29T01:04:52.282 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:04:52.282 INFO:teuthology.orchestra.run.smithi101.stdout: "period_config": { 2023-07-29T01:04:52.283 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_quota": { 2023-07-29T01:04:52.283 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-29T01:04:52.283 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-29T01:04:52.283 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-29T01:04:52.283 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-29T01:04:52.283 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-29T01:04:52.283 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:52.284 INFO:teuthology.orchestra.run.smithi101.stdout: "user_quota": { 2023-07-29T01:04:52.284 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-29T01:04:52.284 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-29T01:04:52.284 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-29T01:04:52.284 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-29T01:04:52.284 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-29T01:04:52.285 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:52.285 INFO:teuthology.orchestra.run.smithi101.stdout: "user_ratelimit": { 2023-07-29T01:04:52.285 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-29T01:04:52.285 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-29T01:04:52.285 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-29T01:04:52.285 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-29T01:04:52.285 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-29T01:04:52.286 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:52.286 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_ratelimit": { 2023-07-29T01:04:52.286 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-29T01:04:52.286 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-29T01:04:52.286 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-29T01:04:52.286 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-29T01:04:52.286 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-29T01:04:52.287 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:52.287 INFO:teuthology.orchestra.run.smithi101.stdout: "anonymous_ratelimit": { 2023-07-29T01:04:52.287 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-29T01:04:52.287 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-29T01:04:52.287 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-29T01:04:52.287 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-29T01:04:52.287 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-29T01:04:52.288 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:52.288 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:52.288 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T01:04:52.288 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_name": "test-realm", 2023-07-29T01:04:52.288 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:52.288 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:04:52.289 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:04:52.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:04:52.290 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:04:52.338 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:04:52.338 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:04:52.410 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-07-29T01:04:52.410 INFO:rgw_multi.tests:create bucket zone=a1 name=hiiqos-66 2023-07-29T01:04:52.411 DEBUG:boto:path=/hiiqos-66/ 2023-07-29T01:04:52.411 DEBUG:boto:auth_path=/hiiqos-66/ 2023-07-29T01:04:52.412 DEBUG:boto:Method: PUT 2023-07-29T01:04:52.412 DEBUG:boto:Path: /hiiqos-66/ 2023-07-29T01:04:52.413 DEBUG:boto:Data: 2023-07-29T01:04:52.413 DEBUG:boto:Headers: {} 2023-07-29T01:04:52.413 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:04:52.414 DEBUG:boto:Port: 8000 2023-07-29T01:04:52.414 DEBUG:boto:Params: {} 2023-07-29T01:04:52.415 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-29T01:04:52.415 DEBUG:boto:Token: None 2023-07-29T01:04:52.415 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 01:04:52 GMT /hiiqos-66/ 2023-07-29T01:04:52.416 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yHATXwtMvF1zxWEuR4NG9pfhIOU= 2023-07-29T01:04:52.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:04:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yHATXwtMvF1zxWEuR4NG9pfhIOU=', 'Content-Length': '0'} 2023-07-29T01:04:52.585 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000aa8bf2aee9a9003-0064c465b4-2348-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:04:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:04:52.586 DEBUG:boto:path=/hiiqos-66/ 2023-07-29T01:04:52.586 DEBUG:boto:auth_path=/hiiqos-66/ 2023-07-29T01:04:52.587 DEBUG:boto:Method: HEAD 2023-07-29T01:04:52.587 DEBUG:boto:Path: /hiiqos-66/ 2023-07-29T01:04:52.588 DEBUG:boto:Data: 2023-07-29T01:04:52.588 DEBUG:boto:Headers: {} 2023-07-29T01:04:52.588 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:04:52.589 DEBUG:boto:Port: 8000 2023-07-29T01:04:52.589 DEBUG:boto:Params: {} 2023-07-29T01:04:52.589 DEBUG:boto:Token: None 2023-07-29T01:04:52.590 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:04:52 GMT /hiiqos-66/ 2023-07-29T01:04:52.590 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:d3d3momYAX1cK6JVhu/iBqD5YxI= 2023-07-29T01:04:52.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:04:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:d3d3momYAX1cK6JVhu/iBqD5YxI=', 'Content-Length': '0'} 2023-07-29T01:04:52.595 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000516e7d981436700f-0064c465b4-2348-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:04:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:04:52.595 DEBUG:boto:path=/hiiqos-66/obj1 2023-07-29T01:04:52.596 DEBUG:boto:auth_path=/hiiqos-66/obj1 2023-07-29T01:04:52.596 DEBUG:boto:Method: PUT 2023-07-29T01:04:52.597 DEBUG:boto:Path: /hiiqos-66/obj1 2023-07-29T01:04:52.597 DEBUG:boto:Data: 2023-07-29T01:04:52.597 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T01:04:52.598 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:04:52.598 DEBUG:boto:Port: 8000 2023-07-29T01:04:52.598 DEBUG:boto:Params: {} 2023-07-29T01:04:52.599 DEBUG:boto:Token: None 2023-07-29T01:04:52.599 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 01:04:52 GMT /hiiqos-66/obj1 2023-07-29T01:04:52.600 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BElmvig117lTNquBaWWmcijIFnw= 2023-07-29T01:04:52.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 01:04:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BElmvig117lTNquBaWWmcijIFnw='} 2023-07-29T01:04:52.614 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000033db44b3b18ab085-0064c465b4-2348-a1'), ('Date', 'Sat, 29 Jul 2023 01:04:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:04:52.615 INFO:rgw_multi.tests:create bucket zone=a2 name=hiiqos-67 2023-07-29T01:04:52.615 DEBUG:boto:path=/hiiqos-67/ 2023-07-29T01:04:52.616 DEBUG:boto:auth_path=/hiiqos-67/ 2023-07-29T01:04:52.616 DEBUG:boto:Method: PUT 2023-07-29T01:04:52.617 DEBUG:boto:Path: /hiiqos-67/ 2023-07-29T01:04:52.617 DEBUG:boto:Data: 2023-07-29T01:04:52.617 DEBUG:boto:Headers: {} 2023-07-29T01:04:52.618 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:04:52.618 DEBUG:boto:Port: 8001 2023-07-29T01:04:52.618 DEBUG:boto:Params: {} 2023-07-29T01:04:52.619 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-29T01:04:52.619 DEBUG:boto:Token: None 2023-07-29T01:04:52.620 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 01:04:52 GMT /hiiqos-67/ 2023-07-29T01:04:52.620 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:D4th+czaQ0xOyr1/BRs9I0S1jS0= 2023-07-29T01:04:52.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:04:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:D4th+czaQ0xOyr1/BRs9I0S1jS0=', 'Content-Length': '0'} 2023-07-29T01:04:52.667 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003fa8b7f2bce749d6-0064c465b4-233c-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:04:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:04:52.668 DEBUG:boto:path=/hiiqos-67/ 2023-07-29T01:04:52.669 DEBUG:boto:auth_path=/hiiqos-67/ 2023-07-29T01:04:52.669 DEBUG:boto:Method: HEAD 2023-07-29T01:04:52.669 DEBUG:boto:Path: /hiiqos-67/ 2023-07-29T01:04:52.670 DEBUG:boto:Data: 2023-07-29T01:04:52.670 DEBUG:boto:Headers: {} 2023-07-29T01:04:52.670 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:04:52.671 DEBUG:boto:Port: 8001 2023-07-29T01:04:52.671 DEBUG:boto:Params: {} 2023-07-29T01:04:52.671 DEBUG:boto:Token: None 2023-07-29T01:04:52.672 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:04:52 GMT /hiiqos-67/ 2023-07-29T01:04:52.672 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:KUbMrRZqlHA50kakeUand6rzPE4= 2023-07-29T01:04:52.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:04:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KUbMrRZqlHA50kakeUand6rzPE4=', 'Content-Length': '0'} 2023-07-29T01:04:52.676 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000059dbbb3c6a5577c1-0064c465b4-233c-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:04:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:04:52.676 DEBUG:boto:path=/hiiqos-67/obj2 2023-07-29T01:04:52.677 DEBUG:boto:auth_path=/hiiqos-67/obj2 2023-07-29T01:04:52.677 DEBUG:boto:Method: PUT 2023-07-29T01:04:52.677 DEBUG:boto:Path: /hiiqos-67/obj2 2023-07-29T01:04:52.678 DEBUG:boto:Data: 2023-07-29T01:04:52.678 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T01:04:52.678 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:04:52.679 DEBUG:boto:Port: 8001 2023-07-29T01:04:52.679 DEBUG:boto:Params: {} 2023-07-29T01:04:52.679 DEBUG:boto:Token: None 2023-07-29T01:04:52.680 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 01:04:52 GMT /hiiqos-67/obj2 2023-07-29T01:04:52.680 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:6VjbmdJZ4ckcf7JNRkgFcwNrViQ= 2023-07-29T01:04:52.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 01:04:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6VjbmdJZ4ckcf7JNRkgFcwNrViQ='} 2023-07-29T01:04:52.696 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007eb3a8db0f58ec28-0064c465b4-233c-a2'), ('Date', 'Sat, 29 Jul 2023 01:04:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:04:52.698 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:04:52.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:04:52.699 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:04:52.766 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:04:52.767 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:04:52.808 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T01:04:52.826 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T01:04:52.826 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:52.826 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:04:52.826 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:17.836402Z" 2023-07-29T01:04:52.826 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:52.827 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:52.827 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592692.567652_1462.1", 2023-07-29T01:04:52.827 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:04:52.567652Z" 2023-07-29T01:04:52.827 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:52.827 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:52.827 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:04:52.828 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T01:04:52.828 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:52.828 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:52.828 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:04:52.828 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:12.811406Z" 2023-07-29T01:04:52.828 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:52.828 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T01:04:52.829 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:04:52.830 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T01:04:52.830 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:04:52.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:04:52.831 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:04:52.885 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:04:52.885 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:04:52.920 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:04:52.836+0000 7fac1df35880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:04:52.921 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:04:52.837+0000 7fac1df35880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:04:52.936 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:04:52.937 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:04:52.937 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:04:52.937 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:04:52.937 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:04:52.937 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:04:52.937 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:52.938 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:52.938 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:04:52.938 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:52.938 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:04:52.938 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:52.938 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:52.939 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:04:52.939 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:52.939 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:04:52.939 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:52.939 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:04:52.939 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:52.939 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:52.940 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:52.940 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:52.940 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:04:52.940 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:52.940 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:52.940 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:04:52.941 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:52.941 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:04:52.941 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:52.941 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:04:52.941 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:52.941 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:52.941 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:52.942 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:52.942 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:04:52.942 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:52.942 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:52.942 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:04:52.942 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:52.942 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:04:52.943 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:52.943 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:04:52.943 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:52.943 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:52.943 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:52.943 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:52.943 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:04:52.944 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:52.944 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:52.944 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:04:52.944 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:52.944 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:04:52.944 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:52.944 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:04:52.945 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:52.945 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:52.945 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:52.945 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:04:52.945 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:52.945 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:04:52.945 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:04:52.946 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:04:52.946 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:52.946 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:04:52Z" 2023-07-29T01:04:52.946 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:04:52.947 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592679.844113_1377.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:04:39.844113Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:04:52Z" } 2023-07-29T01:04:52.948 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:04:52.948 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:04:52.948 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:04:52.949 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:04:57.950 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:04:57.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:04:57.951 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:04:58.005 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:04:58.005 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:04:58.037 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:04:57.954+0000 7f3f1b29d880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:04:58.038 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:04:57.955+0000 7f3f1b29d880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:04:58.052 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:04:58.053 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:04:58.053 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:04:58.053 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:04:58.053 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:04:58.053 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:04:58.053 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:58.053 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:58.054 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:04:58.054 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:58.054 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:04:58.054 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:58.054 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:58.054 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:04:58.055 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:58.055 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:04:58.055 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:58.055 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:04:58.055 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:58.055 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:58.055 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:58.056 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:58.056 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:04:58.056 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:58.056 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:58.056 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:04:58.056 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:58.057 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:04:58.057 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:58.057 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:04:58.057 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:58.057 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:58.057 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:58.057 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:58.058 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:04:58.058 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:58.058 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:58.058 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:04:58.059 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:58.059 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:04:58.059 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:58.059 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:04:58.060 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:58.060 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:58.060 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:58.060 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:04:58.060 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:04:58.061 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:04:58.061 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:04:58.061 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:04:58.061 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:04:58.061 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:04:58.061 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:04:58.062 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:04:58.062 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:04:58.062 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:58.062 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:04:58.062 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:04:58.063 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:58.063 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:04:58.063 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:04:58.064 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:04:58.064 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:04:58.064 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:04:57Z" 2023-07-29T01:04:58.065 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:04:58.066 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592679.844113_1377.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:04:39.844113Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:04:57Z" } 2023-07-29T01:04:58.066 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:04:58.067 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:04:58.067 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:04:58.067 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:05:03.069 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:05:03.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:05:03.070 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:05:03.124 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:05:03.124 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:05:03.157 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:05:03.073+0000 7ff8ebe9d880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:05:03.158 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:05:03.074+0000 7ff8ebe9d880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:05:03.172 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:05:03.172 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:05:03.173 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:05:03.173 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:05:03.173 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:05:03.173 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:05:03.173 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:03.173 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:03.174 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:05:03.174 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:03.174 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:05:03.174 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:03.174 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:03.174 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:05:03.175 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:03.175 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:05:03.175 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:03.175 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:05:03.175 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:03.175 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:03.175 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:03.176 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:03.176 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:05:03.176 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:03.176 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:03.176 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:05:03.177 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:03.177 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:05:03.177 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:03.177 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:05:03.177 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:03.177 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:03.177 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:03.178 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:03.178 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:05:03.178 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:03.178 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:03.178 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:05:03.178 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:03.178 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:05:03.179 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:03.179 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:05:03.179 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:03.179 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:03.179 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:03.179 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:03.179 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:05:03.180 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:03.180 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:03.180 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:05:03.180 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:03.180 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:05:03.180 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:03.180 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:05:03.181 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:03.181 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:03.181 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:03.181 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:05:03.181 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:03.181 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:05:03.181 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:05:03.182 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:05:03.182 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:03.182 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:05:03Z" 2023-07-29T01:05:03.182 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:05:03.183 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592679.844113_1377.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:04:39.844113Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:05:03Z" } 2023-07-29T01:05:03.183 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:05:03.184 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:05:03.184 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:05:03.185 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:05:08.185 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:05:08.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:05:08.187 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:05:08.240 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:05:08.240 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:05:08.273 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:05:08.189+0000 7f4188213880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:05:08.274 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:05:08.190+0000 7f4188213880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:05:08.288 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:05:08.288 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:05:08.288 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:05:08.288 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:05:08.288 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:05:08.289 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:05:08.289 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:08.289 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:08.289 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:05:08.289 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:08.289 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:05:08.289 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:08.290 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:08.290 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:05:08.290 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:08.290 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:05:08.290 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:08.290 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:05:08.290 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:08.291 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:08.291 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:08.291 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:08.291 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:05:08.291 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:08.291 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:08.292 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:05:08.292 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:08.292 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:05:08.292 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:08.292 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:05:08.292 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:08.292 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:08.293 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:08.293 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:08.293 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:05:08.293 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:08.293 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:08.293 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:05:08.294 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:08.294 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:05:08.294 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:08.294 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:05:08.294 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:08.294 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:08.294 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:08.295 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:08.295 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:05:08.295 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:08.295 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:08.295 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:05:08.295 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:08.296 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:05:08.296 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:08.296 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:05:08.296 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:08.296 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:08.297 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:08.297 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:05:08.297 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:08.297 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:05:08.297 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:05:08.297 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:05:08.298 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:08.298 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:05:08Z" 2023-07-29T01:05:08.298 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:05:08.299 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592679.844113_1377.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:04:39.844113Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:05:08Z" } 2023-07-29T01:05:08.300 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:05:08.300 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:05:08.301 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:05:08.301 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:05:13.302 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:05:13.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:05:13.303 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:05:13.356 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:05:13.356 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:05:13.388 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:05:13.305+0000 7efdc0216880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:05:13.389 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:05:13.306+0000 7efdc0216880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:05:13.403 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:05:13.404 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:05:13.404 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:05:13.404 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:05:13.404 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:05:13.404 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:05:13.404 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:13.404 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:13.405 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:05:13.405 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:13.405 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:05:13.405 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:13.405 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:13.405 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:05:13.406 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:13.406 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:05:13.406 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:13.406 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:05:13.406 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:13.406 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:13.406 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:13.407 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:13.407 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:05:13.407 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:13.407 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:13.407 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:05:13.407 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:13.407 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:05:13.408 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:13.408 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:05:13.408 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:13.408 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:13.408 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:13.408 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:13.409 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:05:13.409 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:13.409 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:13.409 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:05:13.409 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:13.409 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:05:13.409 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:13.410 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:05:13.410 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:13.410 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:13.410 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:13.410 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:13.410 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:05:13.410 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:13.411 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:13.411 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:05:13.411 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:13.411 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:05:13.411 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:13.411 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:05:13.411 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:13.412 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:13.412 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:13.412 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:05:13.412 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:13.412 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:05:13.412 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:05:13.412 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:05:13.413 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:13.413 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:05:13Z" 2023-07-29T01:05:13.413 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:05:13.414 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592679.844113_1377.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:04:39.844113Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:05:13Z" } 2023-07-29T01:05:13.414 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:05:13.415 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:05:13.415 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:05:13.415 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:05:18.417 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:05:18.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:05:18.418 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:05:18.471 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:05:18.471 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:05:18.503 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:05:18.419+0000 7fde5d3af880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:05:18.504 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:05:18.420+0000 7fde5d3af880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:05:18.518 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:05:18.518 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:05:18.518 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:05:18.518 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:05:18.519 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:05:18.519 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:05:18.519 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:18.519 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:18.519 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:05:18.519 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:18.520 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:05:18.520 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:18.520 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:18.520 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:05:18.520 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:18.520 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:05:18.521 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:18.521 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:05:18.521 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:18.521 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:18.521 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:18.521 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:18.521 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:05:18.522 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:18.522 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:18.522 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:05:18.522 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:18.522 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:05:18.522 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:18.522 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:05:18.523 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:18.523 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:18.523 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:18.523 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:18.523 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:05:18.523 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:18.524 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:18.524 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:05:18.524 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:18.524 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:05:18.524 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:18.525 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:05:18.525 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:18.525 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:18.525 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:18.525 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:18.526 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:05:18.526 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:18.526 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:18.527 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:05:18.527 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:18.527 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:05:18.528 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:18.528 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:05:18.528 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:18.529 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:18.529 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:18.529 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:05:18.530 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:18.530 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:05:18.530 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:05:18.531 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:05:18.531 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:18.531 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:05:18Z" 2023-07-29T01:05:18.532 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:05:18.533 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592679.844113_1377.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:04:39.844113Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:05:18Z" } 2023-07-29T01:05:18.534 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:05:18.535 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:05:18.535 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:05:18.536 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:05:23.537 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:05:23.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:05:23.539 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:05:23.570 INFO:tasks.rgw.c1.client.0.smithi101.stdout:2023-07-29T01:05:23.486+0000 7f0081a5e640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T01:05:23.601 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:05:23.601 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:05:23.639 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:05:23.555+0000 7f55e3013880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:05:23.640 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:05:23.556+0000 7f55e3013880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:05:23.654 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:05:23.654 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:05:23.655 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:05:23.655 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:05:23.655 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:05:23.655 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:05:23.655 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:23.655 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:23.656 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:05:23.656 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:23.656 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:05:23.656 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:23.657 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:23.657 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:05:23.657 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:23.657 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:05:23.657 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:23.657 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:05:23.658 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:23.658 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:23.658 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:23.658 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:23.658 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:05:23.658 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:23.658 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:23.659 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:05:23.659 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:23.659 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:05:23.659 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:23.659 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:05:23.659 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:23.660 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:23.660 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:23.660 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:23.660 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:05:23.660 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:23.660 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:23.661 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:05:23.661 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:23.661 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:05:23.661 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:23.661 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:05:23.661 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:23.662 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:23.662 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:23.662 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:23.662 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:05:23.662 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:23.662 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:23.662 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:05:23.663 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:23.663 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:05:23.663 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:23.663 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:05:23.663 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:23.663 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:23.664 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:23.664 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:05:23.664 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:23.664 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:05:23.664 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:05:23.665 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:05:23.665 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:23.665 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:05:23Z" 2023-07-29T01:05:23.665 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:05:23.666 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592679.844113_1377.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:04:39.844113Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:05:23Z" } 2023-07-29T01:05:23.666 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:05:23.667 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:05:23.667 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:05:23.668 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:05:28.669 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:05:28.670 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:05:28.670 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:05:28.724 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:05:28.724 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:05:28.757 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:05:28.673+0000 7f95e7826880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:05:28.757 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:05:28.674+0000 7f95e7826880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:05:28.771 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:05:28.772 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:05:28.772 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:05:28.772 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:05:28.772 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:05:28.772 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:05:28.773 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:28.773 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:28.773 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:05:28.773 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:28.773 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:05:28.774 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:28.774 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:28.774 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:05:28.774 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:28.774 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:05:28.774 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:28.774 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:05:28.775 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:28.775 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:28.775 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:28.775 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:28.775 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:05:28.775 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:28.776 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:28.776 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:05:28.776 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:28.776 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:05:28.777 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:28.777 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:05:28.777 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:28.778 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:28.778 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:28.778 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:28.778 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:05:28.779 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:28.779 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:28.779 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:05:28.780 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:28.780 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:05:28.780 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:28.780 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:05:28.781 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:28.781 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:28.781 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:28.782 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:28.782 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:05:28.782 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:28.782 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:28.783 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:05:28.783 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:28.783 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:05:28.783 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:28.784 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:05:28.784 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:28.784 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:28.785 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:28.785 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:05:28.785 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:28.785 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:05:28.786 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:05:28.786 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:05:28.786 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:28.787 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:05:28Z" 2023-07-29T01:05:28.787 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:05:28.788 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592679.844113_1377.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:04:39.844113Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:05:28Z" } 2023-07-29T01:05:28.789 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:05:28.790 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:05:28.790 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:05:28.790 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:05:33.792 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:05:33.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:05:33.793 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:05:33.846 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:05:33.847 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:05:33.879 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:05:33.795+0000 7f33b11a2880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:05:33.880 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:05:33.796+0000 7f33b11a2880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:05:33.894 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:05:33.895 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:05:33.895 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:05:33.895 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:05:33.895 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:05:33.895 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:05:33.895 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:33.896 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:33.896 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:05:33.896 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:33.896 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:05:33.896 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:33.896 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:33.897 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:05:33.897 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:33.897 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:05:33.897 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:33.897 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:05:33.897 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:33.897 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:33.898 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:33.898 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:33.898 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:05:33.898 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:33.898 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:33.898 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:05:33.898 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:33.899 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:05:33.899 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:33.899 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:05:33.899 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:33.899 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:33.899 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:33.899 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:33.900 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:05:33.900 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:33.900 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:33.900 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:05:33.900 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:33.900 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:05:33.900 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:33.901 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:05:33.901 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:33.901 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:33.901 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:33.901 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:33.901 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:05:33.901 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:33.902 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:33.902 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:05:33.902 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:33.902 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:05:33.902 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:33.902 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:05:33.902 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:33.903 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:33.903 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:33.903 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:05:33.903 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:33.903 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:05:33.903 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:05:33.903 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:05:33.904 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:33.904 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:05:33Z" 2023-07-29T01:05:33.904 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:05:33.905 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592679.844113_1377.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:04:39.844113Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:05:33Z" } 2023-07-29T01:05:33.905 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:05:33.906 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:05:33.906 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:05:33.906 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:05:38.907 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:05:38.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:05:38.908 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:05:38.961 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:05:38.962 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:05:38.996 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:05:38.910+0000 7fef91126880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:05:38.997 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:05:38.911+0000 7fef91126880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:05:39.009 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:05:39.009 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:05:39.010 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:05:39.010 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:05:39.010 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:05:39.010 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:05:39.010 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:39.010 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:39.010 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:05:39.011 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:39.011 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:05:39.011 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:39.011 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:39.011 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:05:39.011 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:39.012 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:05:39.012 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:39.012 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:05:39.012 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:39.012 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:39.012 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:39.012 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:39.013 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:05:39.013 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:39.013 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:39.013 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:05:39.013 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:39.013 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:05:39.013 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:39.014 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:05:39.014 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:39.014 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:39.014 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:39.014 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:39.014 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:05:39.014 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:39.015 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:39.015 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:05:39.015 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:39.015 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:05:39.015 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:39.015 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:05:39.015 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:39.016 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:39.016 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:39.016 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:39.016 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:05:39.016 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:39.016 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:39.016 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:05:39.017 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:39.017 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:05:39.017 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:39.017 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:05:39.017 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:39.017 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:39.018 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:39.018 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:05:39.018 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:39.018 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:05:39.018 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:05:39.018 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:05:39.019 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:39.019 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:05:38Z" 2023-07-29T01:05:39.019 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:05:39.020 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592679.844113_1377.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:04:39.844113Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:05:38Z" } 2023-07-29T01:05:39.020 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:05:39.021 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:05:39.021 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:05:39.021 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:05:44.022 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:05:44.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:05:44.024 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:05:44.075 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:05:44.076 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:05:44.108 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:05:44.024+0000 7f96da00f880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:05:44.109 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:05:44.025+0000 7f96da00f880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:05:44.123 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:05:44.124 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:05:44.124 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:05:44.124 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:05:44.124 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:05:44.125 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:05:44.125 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:44.125 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:44.125 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:05:44.126 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:44.126 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:05:44.126 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:44.127 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:44.127 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:05:44.127 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:44.127 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:05:44.128 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:44.128 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:05:44.128 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:44.128 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:44.128 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:44.128 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:44.129 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:05:44.129 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:44.129 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:44.129 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:05:44.129 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:44.129 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:05:44.129 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:44.130 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:05:44.130 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:44.130 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:44.130 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:44.130 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:44.130 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:05:44.130 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:44.131 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:44.131 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:05:44.131 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:44.131 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:05:44.131 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:44.131 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:05:44.131 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:44.132 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:44.132 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:44.132 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:44.132 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:05:44.132 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:44.132 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:44.133 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:05:44.133 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:44.133 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:05:44.133 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:44.133 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:05:44.133 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:44.133 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:44.134 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:44.134 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:05:44.134 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:44.134 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:05:44.134 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:05:44.134 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:05:44.134 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:44.135 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:05:44Z" 2023-07-29T01:05:44.135 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:05:44.136 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592679.844113_1377.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:04:39.844113Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:05:44Z" } 2023-07-29T01:05:44.136 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:05:44.137 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:05:44.137 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:05:44.137 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:05:49.138 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:05:49.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:05:49.140 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:05:49.194 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:05:49.194 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:05:49.226 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:05:49.142+0000 7ff9cb11f880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:05:49.227 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:05:49.143+0000 7ff9cb11f880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:05:49.241 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:05:49.242 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:05:49.242 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:05:49.242 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:05:49.242 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:05:49.242 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:05:49.242 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:49.243 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:49.243 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:05:49.243 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:49.243 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:05:49.243 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:49.243 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:49.243 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:05:49.244 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:49.244 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:05:49.244 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:49.244 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:05:49.244 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:49.244 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:49.245 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:49.245 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:49.245 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:05:49.245 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:49.245 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:49.245 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:05:49.245 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:49.246 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:05:49.246 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:49.246 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:05:49.246 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:49.246 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:49.246 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:49.247 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:49.247 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:05:49.247 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:49.247 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:49.247 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:05:49.247 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:49.247 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:05:49.248 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:49.248 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:05:49.248 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:49.248 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:49.248 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:49.248 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:49.248 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:05:49.249 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:49.249 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:49.249 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:05:49.249 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:49.249 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:05:49.249 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:49.249 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:05:49.250 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:49.250 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:49.250 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:49.250 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:05:49.250 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:49.250 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:05:49.250 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:05:49.251 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:05:49.251 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:49.251 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:05:49Z" 2023-07-29T01:05:49.251 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:05:49.252 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592679.844113_1377.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:04:39.844113Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:05:49Z" } 2023-07-29T01:05:49.253 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:05:49.254 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:05:49.255 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:05:49.255 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:05:54.256 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:05:54.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:05:54.258 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:05:54.311 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:05:54.312 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:05:54.344 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:05:54.260+0000 7fc354321880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:05:54.345 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:05:54.261+0000 7fc354321880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:05:54.359 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:05:54.359 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:05:54.360 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:05:54.360 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:05:54.360 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:05:54.360 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:05:54.361 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:54.361 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:54.361 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:05:54.361 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:54.361 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:05:54.361 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:54.361 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:54.362 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:05:54.362 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:54.362 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:05:54.362 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:54.362 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:05:54.362 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:54.362 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:54.363 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:54.363 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:54.363 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:05:54.363 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:54.363 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:54.363 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:05:54.363 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:54.364 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:05:54.364 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:54.364 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:05:54.364 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:54.364 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:54.365 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:54.365 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:54.365 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:05:54.365 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:54.365 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:54.366 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:05:54.366 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:54.366 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:05:54.366 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:54.366 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:05:54.367 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:54.367 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:54.367 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:54.367 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:54.367 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:05:54.367 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:54.367 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:54.368 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:05:54.368 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:54.368 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:05:54.368 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:54.368 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:05:54.368 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:54.368 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:54.369 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:54.369 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:05:54.369 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:54.369 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:05:54.369 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:05:54.369 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:05:54.370 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:54.370 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:05:54Z" 2023-07-29T01:05:54.370 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:05:54.371 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592679.844113_1377.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:04:39.844113Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:05:54Z" } 2023-07-29T01:05:54.372 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:05:54.372 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:05:54.372 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:05:54.373 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:05:59.374 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:05:59.374 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:05:59.375 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:05:59.431 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:05:59.431 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:05:59.487 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:05:59.403+0000 7f1e99c67880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:05:59.488 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:05:59.404+0000 7f1e99c67880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:05:59.503 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:05:59.504 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:05:59.504 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:05:59.505 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:05:59.505 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:05:59.505 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:05:59.506 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:59.506 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:59.506 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:05:59.507 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:59.507 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:05:59.507 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:59.508 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:59.508 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:05:59.508 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:59.509 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:05:59.509 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:59.509 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:05:59.510 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:59.510 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:59.510 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:59.511 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:59.511 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:05:59.511 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:59.512 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:59.512 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:05:59.512 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:59.513 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:05:59.513 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:59.513 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:05:59.513 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:59.514 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:59.514 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:59.514 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:59.514 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:05:59.515 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:59.515 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:59.515 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:05:59.516 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:59.516 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:05:59.516 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:59.516 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:05:59.517 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:59.517 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:59.517 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:59.518 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:05:59.518 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:05:59.518 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:05:59.518 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:05:59.518 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:05:59.519 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:05:59.519 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:05:59.519 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:05:59.520 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:05:59.520 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:05:59.520 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:59.521 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:05:59.521 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:05:59.521 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:59.522 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:05:59.522 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:05:59.522 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:05:59.522 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:05:59.523 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:05:59Z" 2023-07-29T01:05:59.523 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:05:59.525 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592679.844113_1377.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:04:39.844113Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:05:59Z" } 2023-07-29T01:05:59.525 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:05:59.525 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:05:59.526 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:05:59.526 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:06:04.527 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:06:04.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:06:04.528 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:06:04.582 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:06:04.582 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:06:04.614 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:06:04.530+0000 7f4072f35880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:06:04.615 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:06:04.531+0000 7f4072f35880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:06:04.629 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:06:04.629 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:06:04.629 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:06:04.630 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:06:04.630 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:06:04.630 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:06:04.630 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:04.630 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:04.630 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:06:04.631 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:04.631 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:06:04.631 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:04.631 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:04.631 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:06:04.631 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:04.632 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:06:04.632 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:04.632 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:06:04.632 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:04.632 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:04.632 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:04.633 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:04.633 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:06:04.633 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:04.633 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:04.633 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:06:04.633 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:04.633 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:06:04.634 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:04.634 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:06:04.634 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:04.634 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:04.634 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:04.634 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:04.634 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:06:04.635 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:04.635 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:04.635 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:06:04.635 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:04.635 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:06:04.635 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:04.635 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:06:04.636 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:04.636 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:04.636 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:04.636 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:04.636 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:06:04.636 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:04.637 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:04.637 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:06:04.637 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:04.637 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:06:04.637 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:04.637 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:06:04.637 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:04.638 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:04.638 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:04.638 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:06:04.638 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:04.638 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:06:04.638 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:06:04.638 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:06:04.639 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:04.639 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:06:04Z" 2023-07-29T01:06:04.639 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:06:04.640 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592679.844113_1377.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:04:39.844113Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:06:04Z" } 2023-07-29T01:06:04.640 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:06:04.641 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:06:04.641 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:06:04.642 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:06:09.643 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:06:09.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:06:09.644 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:06:09.698 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:06:09.698 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:06:09.730 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:06:09.646+0000 7ffba9216880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:06:09.731 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:06:09.647+0000 7ffba9216880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:06:09.746 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:06:09.746 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:06:09.746 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:06:09.747 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:06:09.747 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:06:09.747 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:06:09.747 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:09.747 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:09.747 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:06:09.748 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:09.748 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:06:09.748 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:09.748 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:09.748 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:06:09.748 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:09.749 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:06:09.749 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:09.749 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:06:09.749 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:09.749 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:09.749 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:09.750 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:09.750 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:06:09.750 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:09.750 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:09.750 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:06:09.750 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:09.750 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:06:09.751 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:09.751 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:06:09.751 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:09.751 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:09.751 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:09.751 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:09.752 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:06:09.752 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:09.752 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:09.752 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:06:09.752 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:09.752 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:06:09.753 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:09.753 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:06:09.753 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:09.753 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:09.753 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:09.753 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:09.753 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:06:09.754 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:09.754 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:09.754 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:06:09.754 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:09.754 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:06:09.754 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:09.755 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:06:09.755 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:09.755 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:09.755 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:09.755 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:06:09.755 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:09.755 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:06:09.756 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:06:09.756 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:06:09.756 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:09.756 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:06:09Z" 2023-07-29T01:06:09.756 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:06:09.757 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592679.844113_1377.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:04:39.844113Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:06:09Z" } 2023-07-29T01:06:09.758 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:06:09.758 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:06:09.758 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:06:09.759 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:06:14.760 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:06:14.761 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:06:14.761 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:06:14.814 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:06:14.814 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:06:14.846 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:06:14.761+0000 7fc70a410880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:06:14.847 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:06:14.762+0000 7fc70a410880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:06:14.862 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:06:14.863 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:06:14.863 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:06:14.864 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:06:14.864 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:06:14.864 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:06:14.864 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:14.865 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:14.865 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:06:14.865 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:14.865 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:06:14.865 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:14.865 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:14.866 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:06:14.866 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:14.866 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:06:14.866 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:14.866 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:06:14.866 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:14.866 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:14.867 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:14.867 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:14.867 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:06:14.867 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:14.867 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:14.867 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:06:14.868 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:14.868 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:06:14.868 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:14.868 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:06:14.868 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:14.868 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:14.868 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:14.869 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:14.869 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:06:14.869 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:14.869 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:14.869 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:06:14.869 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:14.869 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:06:14.870 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:14.870 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:06:14.870 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:14.870 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:14.870 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:14.870 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:14.870 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:06:14.871 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:14.871 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:14.871 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:06:14.871 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:14.871 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:06:14.871 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:14.871 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:06:14.872 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:14.872 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:14.872 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:14.872 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:06:14.872 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:14.873 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:06:14.873 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:06:14.873 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:06:14.873 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:14.873 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:06:14Z" 2023-07-29T01:06:14.873 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:06:14.874 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592679.844113_1377.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:04:39.844113Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:06:14Z" } 2023-07-29T01:06:14.875 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:06:14.875 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:06:14.875 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:06:14.876 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:06:19.877 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:06:19.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:06:19.878 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:06:19.931 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:06:19.932 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:06:19.964 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:06:19.879+0000 7f1b18611880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:06:19.964 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:06:19.880+0000 7f1b18611880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:06:19.978 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:06:19.979 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:06:19.979 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:06:19.979 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:06:19.979 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:06:19.979 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:06:19.979 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:19.980 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:19.980 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:06:19.980 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:19.980 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:06:19.980 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:19.980 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:19.980 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:06:19.981 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:19.981 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:06:19.981 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:19.981 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:06:19.981 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:19.981 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:19.982 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:19.982 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:19.982 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:06:19.982 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:19.982 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:19.982 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:06:19.982 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:19.983 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:06:19.983 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:19.983 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:06:19.983 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:19.984 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:19.984 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:19.984 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:19.984 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:06:19.984 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:19.984 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:19.984 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:06:19.985 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:19.985 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:06:19.985 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:19.985 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:06:19.985 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:19.985 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:19.985 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:19.986 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:19.986 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:06:19.986 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:19.986 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:19.986 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:06:19.986 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:19.987 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:06:19.987 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:19.987 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:06:19.987 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:19.987 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:19.988 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:19.988 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:06:19.988 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:19.988 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:06:19.988 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:06:19.988 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:06:19.988 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:19.989 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:06:19Z" 2023-07-29T01:06:19.989 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:06:19.990 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592679.844113_1377.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:04:39.844113Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:06:19Z" } 2023-07-29T01:06:19.990 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:06:19.990 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:06:19.991 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:06:19.991 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:06:24.992 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:06:24.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:06:24.993 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:06:25.045 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:06:25.045 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:06:25.077 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:06:24.992+0000 7f6a39934880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:06:25.078 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:06:24.993+0000 7f6a39934880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:06:25.091 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:06:25.092 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:06:25.092 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:06:25.092 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:06:25.092 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:06:25.093 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:06:25.093 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:25.093 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:25.094 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:06:25.094 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:25.094 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:06:25.094 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:25.095 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:25.095 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:06:25.095 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:25.095 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:06:25.096 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:25.096 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:06:25.096 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:25.097 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:25.097 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:25.097 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:25.097 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:06:25.098 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:25.098 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:25.098 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:06:25.099 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:25.099 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:06:25.099 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:25.100 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:06:25.100 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:25.100 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:25.101 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:25.101 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:25.101 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:06:25.101 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:25.102 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:25.102 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:06:25.102 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:25.103 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:06:25.103 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:25.103 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:06:25.104 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:25.104 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:25.104 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:25.105 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:25.105 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:06:25.105 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:25.106 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:25.106 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:06:25.106 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:25.107 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:06:25.107 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:25.107 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:06:25.108 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:25.108 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:25.108 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:25.108 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:06:25.109 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:25.109 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:06:25.109 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:06:25.109 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:06:25.109 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:25.109 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:06:24Z" 2023-07-29T01:06:25.110 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:06:25.111 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592679.844113_1377.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:04:39.844113Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:06:24Z" } 2023-07-29T01:06:25.111 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:06:25.112 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:06:25.112 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:06:25.112 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:06:30.113 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:06:30.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:06:30.114 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:06:30.167 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:06:30.168 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:06:30.199 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:06:30.115+0000 7fac2372a880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:06:30.200 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:06:30.116+0000 7fac2372a880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:06:30.214 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:06:30.214 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:06:30.214 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:06:30.215 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:06:30.215 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:06:30.215 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:06:30.215 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:30.215 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:30.215 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:06:30.216 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:30.216 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:06:30.216 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:30.216 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:30.216 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:06:30.216 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:30.216 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:06:30.217 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:30.217 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:06:30.217 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:30.217 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:30.217 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:30.217 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:30.217 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:06:30.218 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:30.218 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:30.218 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:06:30.218 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:30.218 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:06:30.218 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:30.218 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:06:30.219 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:30.219 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:30.219 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:30.219 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:30.219 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:06:30.219 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:30.220 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:30.220 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:06:30.220 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:30.220 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:06:30.220 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:30.220 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:06:30.220 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:30.221 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:30.221 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:30.221 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:30.221 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:06:30.221 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:30.221 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:30.221 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:06:30.222 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:30.222 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:06:30.222 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:30.222 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:06:30.222 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:30.222 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:30.222 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:30.223 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:06:30.223 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:30.223 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:06:30.223 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:06:30.223 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:06:30.223 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:30.223 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:06:30Z" 2023-07-29T01:06:30.224 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:06:30.225 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592679.844113_1377.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:04:39.844113Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:06:30Z" } 2023-07-29T01:06:30.225 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:06:30.226 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:06:30.226 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:06:30.226 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:06:35.227 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:06:35.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:06:35.228 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:06:35.281 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:06:35.282 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:06:35.314 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:06:35.229+0000 7f42857a2880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:06:35.315 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:06:35.230+0000 7f42857a2880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:06:35.329 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:06:35.329 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:06:35.330 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:06:35.330 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:06:35.330 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:06:35.330 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:06:35.330 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:35.331 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:35.331 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:06:35.331 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:35.331 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:06:35.331 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:35.331 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:35.331 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:06:35.332 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:35.332 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:06:35.332 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:35.332 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:06:35.332 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:35.332 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:35.333 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:35.333 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:35.333 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:06:35.333 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:35.333 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:35.333 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:06:35.334 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:35.334 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:06:35.334 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:35.334 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:06:35.334 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:35.334 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:35.335 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:35.335 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:35.335 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:06:35.335 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:35.335 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:35.335 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:06:35.335 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:35.336 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:06:35.336 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:35.336 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:06:35.336 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:35.336 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:35.336 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:35.337 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:35.337 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:06:35.337 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:35.337 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:35.337 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:06:35.337 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:35.337 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:06:35.338 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:35.338 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:06:35.338 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:35.338 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:35.338 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:35.338 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:06:35.338 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:35.339 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:06:35.339 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:06:35.339 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:06:35.339 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:35.339 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:06:35Z" 2023-07-29T01:06:35.339 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:06:35.340 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592679.844113_1377.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:04:39.844113Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:06:35Z" } 2023-07-29T01:06:35.341 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:06:35.341 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:06:35.341 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:06:35.342 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:06:40.343 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:06:40.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:06:40.344 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:06:40.400 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:06:40.400 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:06:40.434 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:06:40.349+0000 7f866a328880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:06:40.435 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:06:40.350+0000 7f866a328880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:06:40.449 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:06:40.449 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:06:40.450 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:06:40.450 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:06:40.450 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:06:40.450 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:06:40.450 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:40.450 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:40.451 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:06:40.451 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:40.451 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:06:40.451 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:40.451 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:40.451 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:06:40.451 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:40.452 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:06:40.452 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:40.452 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:06:40.452 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:40.452 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:40.452 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:40.453 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:40.453 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:06:40.453 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:40.453 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:40.453 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:06:40.453 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:40.454 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:06:40.454 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:40.454 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:06:40.454 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:40.454 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:40.454 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:40.454 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:40.455 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:06:40.455 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:40.455 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:40.455 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:06:40.456 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:40.456 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:06:40.456 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:40.456 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:06:40.456 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:40.457 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:40.457 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:40.457 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:40.457 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:06:40.458 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:40.458 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:40.458 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:06:40.458 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:40.459 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:06:40.459 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:40.459 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:06:40.459 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:40.459 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:40.459 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:40.459 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:06:40.460 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:40.460 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:06:40.460 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:06:40.460 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:06:40.460 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:40.460 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:06:40Z" 2023-07-29T01:06:40.460 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:06:40.461 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592679.844113_1377.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:04:39.844113Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:06:40Z" } 2023-07-29T01:06:40.462 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:06:40.462 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:06:40.463 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:06:40.463 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:06:45.464 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:06:45.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:06:45.466 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:06:45.519 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:06:45.519 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:06:45.551 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:06:45.466+0000 7fe45b210880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:06:45.551 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:06:45.467+0000 7fe45b210880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:06:45.565 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:06:45.565 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:06:45.566 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:06:45.566 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:06:45.566 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:06:45.566 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:06:45.566 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:45.566 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:45.567 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:06:45.567 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:45.567 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:06:45.567 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:45.567 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:45.567 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:06:45.567 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:45.568 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:06:45.568 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:45.568 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:06:45.568 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:45.568 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:45.568 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:45.569 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:45.569 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:06:45.569 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:45.569 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:45.569 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:06:45.569 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:45.569 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:06:45.570 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:45.570 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:06:45.570 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:45.570 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:45.570 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:45.570 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:45.571 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:06:45.571 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:45.571 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:45.571 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:06:45.571 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:45.571 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:06:45.571 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:45.572 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:06:45.572 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:45.572 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:45.572 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:45.572 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:45.572 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:06:45.573 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:45.573 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:45.573 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:06:45.573 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:45.573 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:06:45.573 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:45.573 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:06:45.574 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:45.574 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:45.574 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:45.574 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:06:45.574 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:45.574 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:06:45.575 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:06:45.575 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:06:45.575 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:45.575 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:06:45Z" 2023-07-29T01:06:45.575 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:06:45.576 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592679.844113_1377.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:04:39.844113Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:06:45Z" } 2023-07-29T01:06:45.577 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:06:45.577 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:06:45.577 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:06:45.578 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:06:50.579 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:06:50.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:06:50.580 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:06:50.633 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:06:50.633 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:06:50.667 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:06:50.582+0000 7f9043ac0880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:06:50.668 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:06:50.583+0000 7f9043ac0880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:06:50.682 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:06:50.683 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:06:50.683 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:06:50.683 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:06:50.683 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:06:50.683 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:06:50.684 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:50.684 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:50.684 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:06:50.684 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:50.684 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:06:50.684 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:50.684 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:50.685 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:06:50.685 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:50.685 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:06:50.685 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:50.685 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:06:50.686 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:50.686 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:50.686 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:50.686 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:50.686 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:06:50.687 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:50.687 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:50.687 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:06:50.687 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:50.687 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:06:50.687 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:50.687 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:06:50.688 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:50.688 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:50.688 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:50.688 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:50.688 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:06:50.688 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:50.689 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:50.689 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:06:50.689 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:50.689 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:06:50.689 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:50.689 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:06:50.689 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:50.690 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:50.690 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:50.690 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:50.690 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:06:50.690 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:50.690 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:50.690 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:06:50.691 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:50.691 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:06:50.691 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:50.691 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:06:50.691 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:50.691 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:50.691 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:50.692 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:06:50.692 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:50.692 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:06:50.692 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:06:50.692 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:06:50.692 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:50.692 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:06:50Z" 2023-07-29T01:06:50.693 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:06:50.693 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592679.844113_1377.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:04:39.844113Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:06:50Z" } 2023-07-29T01:06:50.694 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:06:50.694 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:06:50.695 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:06:50.695 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:06:53.580 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T01:06:53.495+0000 7f5a3155a640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-07-29T01:06:55.696 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:06:55.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:06:55.697 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:06:55.750 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:06:55.750 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:06:55.782 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:06:55.697+0000 7f0cdec67880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:06:55.783 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:06:55.698+0000 7f0cdec67880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:06:55.797 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:06:55.797 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:06:55.797 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:06:55.797 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:06:55.798 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:06:55.798 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:06:55.798 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:55.798 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:55.798 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:06:55.798 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:55.799 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:06:55.799 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:55.799 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:55.799 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:06:55.799 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:55.799 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:06:55.800 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:55.800 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:06:55.800 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:55.800 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:55.800 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:55.800 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:55.801 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:06:55.801 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:55.801 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:55.801 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:06:55.801 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:55.801 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:06:55.801 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:55.802 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:06:55.802 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:55.802 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:55.802 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:55.802 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:55.802 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:06:55.802 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:55.803 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:55.803 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:06:55.803 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:55.803 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:06:55.803 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:55.803 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:06:55.803 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:55.804 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:55.804 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:55.804 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:06:55.804 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:06:55.804 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:06:55.805 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:06:55.805 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:06:55.805 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:06:55.805 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:06:55.805 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:06:55.805 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:06:55.806 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:06:55.806 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:55.806 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:06:55.806 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:06:55.806 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:55.806 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:06:55.806 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:06:55.807 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:06:55.807 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:06:55.807 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:06:55Z" 2023-07-29T01:06:55.807 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:06:55.808 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592679.844113_1377.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:04:39.844113Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:06:55Z" } 2023-07-29T01:06:55.808 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:06:55.809 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:06:55.809 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:06:55.809 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:07:00.811 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:07:00.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:07:00.812 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:07:00.866 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:07:00.866 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:07:00.898 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:07:00.813+0000 7f7c29610880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:07:00.899 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:07:00.814+0000 7f7c29610880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:07:00.913 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:07:00.913 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:07:00.913 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:07:00.914 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:07:00.914 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:07:00.914 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:07:00.914 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:07:00.914 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:00.914 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:07:00.914 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:00.915 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:07:00.915 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:00.915 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:07:00.915 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:07:00.915 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:00.916 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:07:00.916 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:00.916 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:07:00.916 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:07:00.916 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:00.916 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:00.917 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:00.917 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:07:00.917 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:00.917 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:07:00.917 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:07:00.917 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:00.918 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:07:00.918 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:00.918 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:07:00.918 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:07:00.918 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:00.918 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:00.919 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:00.919 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:07:00.919 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:00.919 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:07:00.919 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:07:00.919 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:00.919 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:07:00.920 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:00.920 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:07:00.921 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:07:00.921 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:00.921 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:00.921 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:00.921 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:07:00.921 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:00.922 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:07:00.922 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:07:00.922 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:00.922 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:07:00.922 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:00.922 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:07:00.923 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:07:00.923 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:00.923 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:00.923 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:07:00.923 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:00.923 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:07:00.923 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:07:00.924 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:07:00.924 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:00.924 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:07:00Z" 2023-07-29T01:07:00.924 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:07:00.925 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592679.844113_1377.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:04:39.844113Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:07:00Z" } 2023-07-29T01:07:00.926 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:07:00.926 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:07:00.926 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:07:00.927 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:07:05.927 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:07:05.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:07:05.928 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:07:05.981 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:07:05.982 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:07:06.013 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:07:05.928+0000 7fb7e0124880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:07:06.014 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:07:05.929+0000 7fb7e0124880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:07:06.028 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:07:06.028 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:07:06.029 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:07:06.029 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:07:06.029 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:07:06.029 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:07:06.029 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:07:06.029 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:06.030 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:07:06.030 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:06.030 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:07:06.030 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:06.030 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:07:06.030 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:07:06.030 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:06.031 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:07:06.031 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:06.031 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:07:06.031 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:07:06.031 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:06.031 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:06.031 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:06.032 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:07:06.032 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:06.032 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:07:06.032 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592692.567652_1462.1", 2023-07-29T01:07:06.032 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:06.032 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:07:06.033 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:06.033 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:52.567652Z", 2023-07-29T01:07:06.033 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:07:06.033 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:06.033 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:06.033 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:06.033 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:07:06.034 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:06.034 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:07:06.034 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:07:06.035 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:06.035 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:07:06.035 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:06.035 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:07:06.036 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:07:06.036 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:06.036 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:06.036 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:06.036 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:07:06.037 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:06.037 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:07:06.037 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:07:06.037 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:06.037 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:07:06.037 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:06.037 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:07:06.038 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:07:06.038 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:06.038 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:06.038 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:07:06.038 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:06.038 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:07:06.038 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:07:06.039 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:07:06.039 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:06.039 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:07:05Z" 2023-07-29T01:07:06.039 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:07:06.040 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592692.567652_1462.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:04:52.567652Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:07:05Z" } 2023-07-29T01:07:06.041 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592692.567652_1462.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:52.567652Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:07:06.041 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:07:06.041 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:07:06.042 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T01:07:06.042 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:07:06.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:07:06.043 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:07:06.089 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:07:06.089 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:07:06.139 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T01:07:06.139 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:06.139 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051393", 2023-07-29T01:07:06.139 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:04:52.529981Z" 2023-07-29T01:07:06.140 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:06.140 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:06.140 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044739", 2023-07-29T01:07:06.140 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:58:23.964578Z" 2023-07-29T01:07:06.140 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:06.140 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:06.141 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045520", 2023-07-29T01:07:06.141 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:06:23.586689Z" 2023-07-29T01:07:06.141 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:06.141 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:06.141 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:07:06.141 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:07:06.141 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:06.142 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T01:07:06.143 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000051393', 1: '00000000000000000000:00000000000000044739', 2: '00000000000000000000:00000000000000045520', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:07:06.143 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-07-29T01:07:06.143 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:07:06.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:07:06.144 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:07:06.191 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:07:06.191 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:07:06.234 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:07:06.234 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:07:06.234 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:07:06.234 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:07:06.234 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:07:06.234 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:07:06.235 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:06.235 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:07:06.235 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:06.235 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:07:06.235 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:06.235 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:07:06.236 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051393", 2023-07-29T01:07:06.236 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:06.236 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:07:06.236 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:06.236 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:52.529981Z" 2023-07-29T01:07:06.236 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:06.237 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:06.237 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:06.237 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:07:06.237 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:06.237 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:07:06.237 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044739", 2023-07-29T01:07:06.237 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:06.238 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:07:06.238 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:06.238 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:23.964578Z" 2023-07-29T01:07:06.238 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:06.238 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:06.238 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:06.238 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:07:06.239 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:06.239 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:07:06.239 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045520", 2023-07-29T01:07:06.239 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:06.239 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:07:06.239 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:06.239 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:06:23.586689Z" 2023-07-29T01:07:06.240 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:06.240 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:06.240 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:06.240 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:07:06.240 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:06.240 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:07:06.240 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:07:06.241 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:06.241 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:07:06.241 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:06.241 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:07:06.241 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:06.241 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:06.241 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:07:06.242 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:06.242 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:07:06.242 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:07:06.242 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:07:06.242 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:06.242 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:07:06Z" 2023-07-29T01:07:06.242 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:07:06.243 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051393", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:04:52.529981Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044739", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T00:58:23.964578Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045520", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:06:23.586689Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:07:06Z" } 2023-07-29T01:07:06.244 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051393', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:04:52.529981Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044739', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T00:58:23.964578Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045520', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:06:23.586689Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:07:06.244 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051393', 1: '00000000000000000000:00000000000000044739', 2: '00000000000000000000:00000000000000045520', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:07:06.245 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051393', 1: '00000000000000000000:00000000000000044739', 2: '00000000000000000000:00000000000000045520', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:07:06.245 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-07-29T01:07:06.245 DEBUG:boto:path=/hiiqos-66/ 2023-07-29T01:07:06.246 DEBUG:boto:auth_path=/hiiqos-66/ 2023-07-29T01:07:06.246 DEBUG:boto:Method: HEAD 2023-07-29T01:07:06.246 DEBUG:boto:Path: /hiiqos-66/ 2023-07-29T01:07:06.247 DEBUG:boto:Data: 2023-07-29T01:07:06.247 DEBUG:boto:Headers: {} 2023-07-29T01:07:06.247 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:07:06.248 DEBUG:boto:Port: 8001 2023-07-29T01:07:06.248 DEBUG:boto:Params: {} 2023-07-29T01:07:06.248 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-29T01:07:06.249 DEBUG:boto:Token: None 2023-07-29T01:07:06.249 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:07:06 GMT /hiiqos-66/ 2023-07-29T01:07:06.250 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:K26aV3Uo69IWzbElfPvaDaz0Gpk= 2023-07-29T01:07:06.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:07:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K26aV3Uo69IWzbElfPvaDaz0Gpk=', 'Content-Length': '0'} 2023-07-29T01:07:11.254 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006389ac04b8d3b524-0064c4663f-233c-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:07:11.255 DEBUG:boto:path=/hiiqos-66/obj1 2023-07-29T01:07:11.256 DEBUG:boto:auth_path=/hiiqos-66/obj1 2023-07-29T01:07:11.256 DEBUG:boto:Method: HEAD 2023-07-29T01:07:11.256 DEBUG:boto:Path: /hiiqos-66/obj1 2023-07-29T01:07:11.257 DEBUG:boto:Data: 2023-07-29T01:07:11.257 DEBUG:boto:Headers: {} 2023-07-29T01:07:11.257 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:07:11.258 DEBUG:boto:Port: 8001 2023-07-29T01:07:11.258 DEBUG:boto:Params: {} 2023-07-29T01:07:11.258 DEBUG:boto:Token: None 2023-07-29T01:07:11.259 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:07:11 GMT /hiiqos-66/obj1 2023-07-29T01:07:11.259 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TMGyP0SD4cuVaZw+5fJv43Vtp4c= 2023-07-29T01:07:11.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:07:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TMGyP0SD4cuVaZw+5fJv43Vtp4c=', 'Content-Length': '0'} 2023-07-29T01:07:11.261 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000c8116c313f69b6de-0064c4663f-233c-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 01:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:07:11.264 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_all: ... FAIL 2023-07-29T01:07:11.265 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_select: ... SKIP: test_sync_flow_symmetrical_zonegroup_select skipped. Requires 3 or more zones in master zonegroup. 2023-07-29T01:07:11.265 INFO:tasks.rgw_multisite_tests:test_sync_flow_directional_zonegroup_select: ... SKIP: test_sync_flow_symmetrical_zonegroup_select skipped. Requires 3 or more zones in master zonegroup. 2023-07-29T01:07:11.266 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:07:11.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:07:11.267 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:07:11.319 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:07:11.319 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:07:11.353 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T01:07:11.368 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T01:07:11.368 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:11.368 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:07:11.368 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:17.836402Z" 2023-07-29T01:07:11.369 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:11.369 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:11.369 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592692.567652_1462.1", 2023-07-29T01:07:11.369 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:04:52.567652Z" 2023-07-29T01:07:11.369 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:11.369 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:11.369 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:07:11.370 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T01:07:11.370 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:11.370 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:11.370 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:07:11.370 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:12.811406Z" 2023-07-29T01:07:11.370 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:11.370 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T01:07:11.371 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:07:11.372 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T01:07:11.372 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:07:11.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:07:11.373 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:07:11.420 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:07:11.420 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:07:11.452 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:07:11.366+0000 7fb11bb35880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:07:11.452 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:07:11.367+0000 7fb11bb35880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:07:11.465 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:07:11.466 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:07:11.466 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:07:11.466 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:07:11.466 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:07:11.466 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:07:11.466 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:07:11.467 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:11.467 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:07:11.467 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:11.467 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:07:11.467 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:11.467 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:07:11.467 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:07:11.468 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:11.468 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:07:11.468 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:11.468 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:07:11.468 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:07:11.468 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:11.468 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:11.469 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:11.469 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:07:11.469 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:11.469 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:07:11.469 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592692.567652_1462.1", 2023-07-29T01:07:11.469 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:11.469 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:07:11.470 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:11.470 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:52.567652Z", 2023-07-29T01:07:11.470 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:07:11.470 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:11.470 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:11.470 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:11.470 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:07:11.471 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:11.471 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:07:11.471 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:07:11.471 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:11.471 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:07:11.471 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:11.471 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:07:11.472 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:07:11.472 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:11.472 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:11.472 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:11.472 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:07:11.472 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:11.472 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:07:11.473 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:07:11.473 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:11.473 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:07:11.473 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:11.473 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:07:11.473 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:07:11.473 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:11.474 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:11.474 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:07:11.474 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:11.474 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:07:11.474 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:07:11.474 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:07:11.474 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:11.475 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:07:11Z" 2023-07-29T01:07:11.475 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:07:11.476 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592692.567652_1462.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:04:52.567652Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:07:11Z" } 2023-07-29T01:07:11.476 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592692.567652_1462.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:52.567652Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:07:11.477 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:07:11.477 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:07:11.477 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T01:07:11.477 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:07:11.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:07:11.478 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:07:11.521 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:07:11.521 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:07:11.566 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-07-29T01:07:11.566 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:07:11.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:07:11.567 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:07:11.648 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:07:11.649 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:07:11.692 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-07-29T01:07:11.693 INFO:rgw_multi.tests:create bucket zone=a1 name=hiiqos-68 2023-07-29T01:07:11.693 DEBUG:boto:path=/hiiqos-68/ 2023-07-29T01:07:11.694 DEBUG:boto:auth_path=/hiiqos-68/ 2023-07-29T01:07:11.694 DEBUG:boto:Method: PUT 2023-07-29T01:07:11.695 DEBUG:boto:Path: /hiiqos-68/ 2023-07-29T01:07:11.695 DEBUG:boto:Data: 2023-07-29T01:07:11.695 DEBUG:boto:Headers: {} 2023-07-29T01:07:11.696 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:07:11.696 DEBUG:boto:Port: 8000 2023-07-29T01:07:11.696 DEBUG:boto:Params: {} 2023-07-29T01:07:11.697 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-29T01:07:11.697 DEBUG:boto:Token: None 2023-07-29T01:07:11.698 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 01:07:11 GMT /hiiqos-68/ 2023-07-29T01:07:11.698 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:MKlG2kEXrWgwSW8eaL69ViSfwik= 2023-07-29T01:07:11.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:07:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MKlG2kEXrWgwSW8eaL69ViSfwik=', 'Content-Length': '0'} 2023-07-29T01:07:16.718 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000123f7770c077ae08-0064c46644-2348-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:07:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:07:16.719 INFO:rgw_multi.tests:create bucket zone=a1 name=hiiqos-69 2023-07-29T01:07:16.719 DEBUG:boto:path=/hiiqos-69/ 2023-07-29T01:07:16.720 DEBUG:boto:auth_path=/hiiqos-69/ 2023-07-29T01:07:16.720 DEBUG:boto:Method: PUT 2023-07-29T01:07:16.720 DEBUG:boto:Path: /hiiqos-69/ 2023-07-29T01:07:16.721 DEBUG:boto:Data: 2023-07-29T01:07:16.721 DEBUG:boto:Headers: {} 2023-07-29T01:07:16.721 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:07:16.722 DEBUG:boto:Port: 8000 2023-07-29T01:07:16.722 DEBUG:boto:Params: {} 2023-07-29T01:07:16.722 DEBUG:boto:Token: None 2023-07-29T01:07:16.723 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 01:07:16 GMT /hiiqos-69/ 2023-07-29T01:07:16.723 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:k/VYyOM9EYUfs2ZaV4BbnTZPQcQ= 2023-07-29T01:07:16.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:07:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:k/VYyOM9EYUfs2ZaV4BbnTZPQcQ=', 'Content-Length': '0'} 2023-07-29T01:07:16.741 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a7458e2471da7cd1-0064c46644-2348-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:07:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:07:16.741 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:07:16.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:07:16.742 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:07:16.791 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:07:16.791 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:07:16.823 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T01:07:16.841 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T01:07:16.841 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:16.841 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:07:16.842 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:17.836402Z" 2023-07-29T01:07:16.842 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:16.842 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:16.842 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592836.654753_1494.1", 2023-07-29T01:07:16.842 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:07:16.654753Z" 2023-07-29T01:07:16.842 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:16.842 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:16.843 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:07:16.843 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T01:07:16.843 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:16.843 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:16.843 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:07:16.843 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:12.811406Z" 2023-07-29T01:07:16.843 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:16.844 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T01:07:16.845 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690591217.836402_294.1', 1: '1_1690592836.654753_1494.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:07:16.845 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T01:07:16.845 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:07:16.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:07:16.846 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:07:16.894 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:07:16.895 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:07:16.923 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:07:16.837+0000 7f9f1082d880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:07:16.923 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:07:16.838+0000 7f9f1082d880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:07:16.936 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:07:16.936 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:07:16.937 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:07:16.937 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:07:16.937 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:07:16.937 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:07:16.937 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:07:16.937 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:16.937 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:07:16.938 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:16.938 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:07:16.938 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:16.938 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:07:16.938 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:07:16.938 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:16.938 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:07:16.939 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:16.939 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:07:16.939 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:07:16.939 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:16.939 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:16.939 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:16.939 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:07:16.940 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:16.940 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:07:16.940 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592836.625550_1479.1", 2023-07-29T01:07:16.940 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:16.940 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:07:16.941 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:16.941 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:16.625550Z", 2023-07-29T01:07:16.941 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:07:16.941 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:16.941 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:16.941 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:16.942 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:07:16.942 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:16.942 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:07:16.942 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:07:16.942 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:16.943 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:07:16.943 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:16.943 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:07:16.943 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:07:16.943 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:16.943 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:16.943 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:16.944 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:07:16.944 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:16.944 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:07:16.944 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:07:16.944 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:16.944 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:07:16.945 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:16.945 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:07:16.945 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:07:16.945 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:16.945 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:16.945 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:07:16.946 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:16.946 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:07:16.946 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:07:16.946 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:07:16.946 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:16.946 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:07:16Z" 2023-07-29T01:07:16.946 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:07:16.947 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592836.625550_1479.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:16.625550Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:07:16Z" } 2023-07-29T01:07:16.948 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592836.625550_1479.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:16.625550Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:07:16.948 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592836.654753_1494.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:07:16.949 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592836.625550_1479.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:07:16.949 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592836.654753_1494.1 target=1_1690592836.625550_1479.1 2023-07-29T01:07:21.950 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:07:21.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:07:21.951 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:07:22.003 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:07:22.004 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:07:22.036 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:07:21.951+0000 7f7406b30880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:07:22.037 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:07:21.952+0000 7f7406b30880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:07:22.051 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:07:22.051 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:07:22.051 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:07:22.051 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:07:22.051 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:07:22.052 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:07:22.052 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:07:22.052 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:22.052 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:07:22.052 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:22.052 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:07:22.053 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:22.053 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:07:22.053 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:07:22.053 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:22.053 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:07:22.054 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:22.054 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:07:22.054 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:07:22.054 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:22.054 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:22.054 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:22.054 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:07:22.055 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:22.055 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:07:22.055 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592836.654753_1494.1", 2023-07-29T01:07:22.055 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:22.055 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:07:22.055 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:22.056 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:16.654753Z", 2023-07-29T01:07:22.056 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:07:22.056 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:22.056 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:22.056 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:22.056 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:07:22.056 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:22.057 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:07:22.057 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:07:22.057 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:22.057 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:07:22.057 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:22.057 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:07:22.057 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:07:22.058 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:22.058 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:22.058 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:22.058 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:07:22.058 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:22.058 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:07:22.059 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:07:22.059 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:22.059 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:07:22.059 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:22.059 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:07:22.059 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:07:22.059 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:22.060 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:22.060 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:07:22.060 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:22.060 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:07:22.060 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:07:22.060 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:07:22.060 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:22.061 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:07:21Z" 2023-07-29T01:07:22.061 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:07:22.062 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592836.654753_1494.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:16.654753Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:07:21Z" } 2023-07-29T01:07:22.063 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592836.654753_1494.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:16.654753Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:07:22.063 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592836.654753_1494.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:07:22.064 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592836.654753_1494.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:07:22.064 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T01:07:22.065 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:07:22.065 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:07:22.065 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:07:22.111 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:07:22.112 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:07:22.158 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-07-29T01:07:22.158 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:07:22.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:07:22.160 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:07:22.209 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:07:22.210 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:07:22.254 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-07-29T01:07:22.254 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--source-bucket=hiiqos-68', '--dest-bucket=hiiqos-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:07:22.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--source-bucket=hiiqos-68', '--dest-bucket=hiiqos-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:07:22.255 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --source-bucket=hiiqos-68 --dest-bucket=hiiqos-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:07:22.305 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:07:22.306 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:07:22.350 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hiiqos-68","zones":["a1","a2"]},"dest":{"bucket":"hiiqos-68","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-07-29T01:07:22.351 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:07:22.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:07:22.352 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:07:22.397 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:07:22.398 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:07:22.443 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hiiqos-68","zones":["a1","a2"]},"dest":{"bucket":"hiiqos-68","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-07-29T01:07:22.443 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:07:22.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:07:22.444 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:07:22.486 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:07:22.487 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:07:22.530 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hiiqos-68","zones":["a1","a2"]},"dest":{"bucket":"hiiqos-68","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-07-29T01:07:22.530 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:07:22.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:07:22.531 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:07:22.615 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:07:22.616 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:07:22.666 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:07:22.666 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:07:22.667 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 16, 2023-07-29T01:07:22.667 INFO:teuthology.orchestra.run.smithi101.stdout: "predecessor_uuid": "a0987484-ce37-4520-a564-54b7ea8c5251", 2023-07-29T01:07:22.667 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": [], 2023-07-29T01:07:22.667 INFO:teuthology.orchestra.run.smithi101.stdout: "period_map": { 2023-07-29T01:07:22.667 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:07:22.667 INFO:teuthology.orchestra.run.smithi101.stdout: "zonegroups": [ 2023-07-29T01:07:22.668 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:22.668 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "a", 2023-07-29T01:07:22.668 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a", 2023-07-29T01:07:22.668 INFO:teuthology.orchestra.run.smithi101.stdout: "api_name": "", 2023-07-29T01:07:22.668 INFO:teuthology.orchestra.run.smithi101.stdout: "is_master": true, 2023-07-29T01:07:22.668 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:07:22.669 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-29T01:07:22.669 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:07:22.669 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames": [], 2023-07-29T01:07:22.669 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames_s3website": [], 2023-07-29T01:07:22.669 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:07:22.669 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:07:22.669 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:22.670 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:07:22.670 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a1", 2023-07-29T01:07:22.670 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:07:22.670 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-29T01:07:22.670 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:07:22.670 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T01:07:22.670 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T01:07:22.674 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T01:07:22.674 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T01:07:22.674 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T01:07:22.674 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T01:07:22.675 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T01:07:22.675 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T01:07:22.675 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T01:07:22.675 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T01:07:22.675 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:07:22.675 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:07:22.675 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:22.676 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:22.676 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-29T01:07:22.676 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a2", 2023-07-29T01:07:22.676 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:07:22.676 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8001" 2023-07-29T01:07:22.676 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:07:22.677 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T01:07:22.677 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T01:07:22.677 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T01:07:22.677 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T01:07:22.677 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T01:07:22.677 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T01:07:22.677 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T01:07:22.678 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T01:07:22.678 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T01:07:22.678 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T01:07:22.678 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:07:22.678 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:07:22.678 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:22.679 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:07:22.679 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_targets": [ 2023-07-29T01:07:22.679 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:22.679 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default-placement", 2023-07-29T01:07:22.679 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [], 2023-07-29T01:07:22.679 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": [ 2023-07-29T01:07:22.679 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD" 2023-07-29T01:07:22.680 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:07:22.680 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:22.680 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:07:22.680 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "default-placement", 2023-07-29T01:07:22.680 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T01:07:22.681 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_policy": { 2023-07-29T01:07:22.681 INFO:teuthology.orchestra.run.smithi101.stdout: "groups": [ 2023-07-29T01:07:22.681 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:22.681 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "sync-group", 2023-07-29T01:07:22.681 INFO:teuthology.orchestra.run.smithi101.stdout: "data_flow": { 2023-07-29T01:07:22.681 INFO:teuthology.orchestra.run.smithi101.stdout: "symmetrical": [ 2023-07-29T01:07:22.681 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:22.682 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "sync-flow1", 2023-07-29T01:07:22.682 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:07:22.682 INFO:teuthology.orchestra.run.smithi101.stdout: "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:07:22.682 INFO:teuthology.orchestra.run.smithi101.stdout: "59d1bb30-3b1c-476f-9111-052c1ea0b711" 2023-07-29T01:07:22.683 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:07:22.683 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:22.683 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:07:22.684 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:22.684 INFO:teuthology.orchestra.run.smithi101.stdout: "pipes": [ 2023-07-29T01:07:22.684 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:22.685 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "sync-pipe", 2023-07-29T01:07:22.685 INFO:teuthology.orchestra.run.smithi101.stdout: "source": { 2023-07-29T01:07:22.685 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket": "hiiqos-68", 2023-07-29T01:07:22.685 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:07:22.686 INFO:teuthology.orchestra.run.smithi101.stdout: "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:07:22.686 INFO:teuthology.orchestra.run.smithi101.stdout: "59d1bb30-3b1c-476f-9111-052c1ea0b711" 2023-07-29T01:07:22.686 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:07:22.687 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:22.687 INFO:teuthology.orchestra.run.smithi101.stdout: "dest": { 2023-07-29T01:07:22.687 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket": "hiiqos-68", 2023-07-29T01:07:22.688 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:07:22.688 INFO:teuthology.orchestra.run.smithi101.stdout: "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:07:22.689 INFO:teuthology.orchestra.run.smithi101.stdout: "59d1bb30-3b1c-476f-9111-052c1ea0b711" 2023-07-29T01:07:22.689 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:07:22.689 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:22.690 INFO:teuthology.orchestra.run.smithi101.stdout: "params": { 2023-07-29T01:07:22.690 INFO:teuthology.orchestra.run.smithi101.stdout: "source": { 2023-07-29T01:07:22.690 INFO:teuthology.orchestra.run.smithi101.stdout: "filter": { 2023-07-29T01:07:22.691 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [] 2023-07-29T01:07:22.691 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:22.691 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:22.691 INFO:teuthology.orchestra.run.smithi101.stdout: "dest": {}, 2023-07-29T01:07:22.692 INFO:teuthology.orchestra.run.smithi101.stdout: "priority": 0, 2023-07-29T01:07:22.692 INFO:teuthology.orchestra.run.smithi101.stdout: "mode": "system", 2023-07-29T01:07:22.692 INFO:teuthology.orchestra.run.smithi101.stdout: "user": "" 2023-07-29T01:07:22.692 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:22.693 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:22.693 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:07:22.693 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "enabled" 2023-07-29T01:07:22.694 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:22.694 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:07:22.694 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:22.694 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled_features": [ 2023-07-29T01:07:22.695 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:07:22.695 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:07:22.695 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:22.695 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:22.695 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "b", 2023-07-29T01:07:22.696 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b", 2023-07-29T01:07:22.696 INFO:teuthology.orchestra.run.smithi101.stdout: "api_name": "", 2023-07-29T01:07:22.696 INFO:teuthology.orchestra.run.smithi101.stdout: "is_master": false, 2023-07-29T01:07:22.696 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:07:22.696 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-29T01:07:22.696 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:07:22.697 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames": [], 2023-07-29T01:07:22.697 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames_s3website": [], 2023-07-29T01:07:22.697 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T01:07:22.697 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:07:22.697 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:22.697 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T01:07:22.697 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b1", 2023-07-29T01:07:22.698 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:07:22.698 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-29T01:07:22.698 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:07:22.698 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T01:07:22.698 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T01:07:22.698 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T01:07:22.698 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T01:07:22.699 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T01:07:22.699 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T01:07:22.699 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T01:07:22.699 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T01:07:22.699 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T01:07:22.699 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T01:07:22.699 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:07:22.700 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:07:22.700 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:22.700 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:22.700 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-29T01:07:22.700 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b2", 2023-07-29T01:07:22.700 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:07:22.700 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8001" 2023-07-29T01:07:22.701 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:07:22.701 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T01:07:22.701 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T01:07:22.701 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T01:07:22.701 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T01:07:22.701 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T01:07:22.701 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T01:07:22.702 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T01:07:22.702 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T01:07:22.702 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T01:07:22.702 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T01:07:22.702 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:07:22.702 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:07:22.703 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:22.703 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:07:22.703 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_targets": [ 2023-07-29T01:07:22.703 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:22.703 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default-placement", 2023-07-29T01:07:22.703 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [], 2023-07-29T01:07:22.703 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": [ 2023-07-29T01:07:22.704 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD" 2023-07-29T01:07:22.704 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:07:22.704 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:22.704 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:07:22.704 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "default-placement", 2023-07-29T01:07:22.704 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T01:07:22.704 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_policy": { 2023-07-29T01:07:22.705 INFO:teuthology.orchestra.run.smithi101.stdout: "groups": [] 2023-07-29T01:07:22.705 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:22.705 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled_features": [ 2023-07-29T01:07:22.705 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:07:22.705 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:07:22.705 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:22.706 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:07:22.706 INFO:teuthology.orchestra.run.smithi101.stdout: "short_zone_ids": [ 2023-07-29T01:07:22.706 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:22.706 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:07:22.706 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 2896287231 2023-07-29T01:07:22.706 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:22.707 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:22.707 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-29T01:07:22.708 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 815606810 2023-07-29T01:07:22.708 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:22.708 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:22.709 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T01:07:22.709 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 2009171363 2023-07-29T01:07:22.709 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:22.709 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:22.709 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-29T01:07:22.709 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 3105190295 2023-07-29T01:07:22.709 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:22.710 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:07:22.710 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:22.710 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zonegroup": "a", 2023-07-29T01:07:22.710 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:07:22.710 INFO:teuthology.orchestra.run.smithi101.stdout: "period_config": { 2023-07-29T01:07:22.710 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_quota": { 2023-07-29T01:07:22.710 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-29T01:07:22.711 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-29T01:07:22.711 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-29T01:07:22.711 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-29T01:07:22.711 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-29T01:07:22.711 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:22.711 INFO:teuthology.orchestra.run.smithi101.stdout: "user_quota": { 2023-07-29T01:07:22.711 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-29T01:07:22.712 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-29T01:07:22.712 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-29T01:07:22.712 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-29T01:07:22.712 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-29T01:07:22.712 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:22.712 INFO:teuthology.orchestra.run.smithi101.stdout: "user_ratelimit": { 2023-07-29T01:07:22.713 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-29T01:07:22.713 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-29T01:07:22.713 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-29T01:07:22.713 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-29T01:07:22.714 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-29T01:07:22.714 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:22.714 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_ratelimit": { 2023-07-29T01:07:22.714 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-29T01:07:22.715 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-29T01:07:22.715 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-29T01:07:22.715 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-29T01:07:22.715 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-29T01:07:22.715 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:22.715 INFO:teuthology.orchestra.run.smithi101.stdout: "anonymous_ratelimit": { 2023-07-29T01:07:22.715 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-29T01:07:22.716 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-29T01:07:22.716 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-29T01:07:22.716 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-29T01:07:22.716 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-29T01:07:22.716 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:22.716 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:22.717 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T01:07:22.717 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_name": "test-realm", 2023-07-29T01:07:22.717 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:07:22.717 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:07:22.717 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'info', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:07:22.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'info', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:07:22.718 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync info --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:07:22.760 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:07:22.760 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:07:22.802 INFO:teuthology.orchestra.run.smithi101.stdout:{"sources":[{"id":"sync-pipe","source":{"zone":"a2","bucket":"hiiqos-68"},"dest":{"zone":"a1","bucket":"hiiqos-68"},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"dests":[{"id":"sync-pipe","source":{"zone":"a1","bucket":"hiiqos-68"},"dest":{"zone":"a2","bucket":"hiiqos-68"},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"hints":{"sources":[],"dests":[]},"resolved-hints-1":{"sources":[],"dests":[]},"resolved-hints":{"sources":[],"dests":[]}} 2023-07-29T01:07:22.803 DEBUG:boto:path=/hiiqos-68/ 2023-07-29T01:07:22.803 DEBUG:boto:auth_path=/hiiqos-68/ 2023-07-29T01:07:22.803 DEBUG:boto:Method: HEAD 2023-07-29T01:07:22.804 DEBUG:boto:Path: /hiiqos-68/ 2023-07-29T01:07:22.804 DEBUG:boto:Data: 2023-07-29T01:07:22.805 DEBUG:boto:Headers: {} 2023-07-29T01:07:22.805 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:07:22.805 DEBUG:boto:Port: 8000 2023-07-29T01:07:22.806 DEBUG:boto:Params: {} 2023-07-29T01:07:22.806 DEBUG:boto:Token: None 2023-07-29T01:07:22.806 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:07:22 GMT /hiiqos-68/ 2023-07-29T01:07:22.807 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O4WbOyOqrq17cMz3S1ZI+jtc3Ak= 2023-07-29T01:07:22.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:07:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O4WbOyOqrq17cMz3S1ZI+jtc3Ak=', 'Content-Length': '0'} 2023-07-29T01:07:22.925 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ce480d62ffb0ebb2-0064c4664a-24ac-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:07:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:07:22.926 DEBUG:boto:path=/hiiqos-68/obj1 2023-07-29T01:07:22.926 DEBUG:boto:auth_path=/hiiqos-68/obj1 2023-07-29T01:07:22.927 DEBUG:boto:Method: PUT 2023-07-29T01:07:22.927 DEBUG:boto:Path: /hiiqos-68/obj1 2023-07-29T01:07:22.927 DEBUG:boto:Data: 2023-07-29T01:07:22.928 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T01:07:22.928 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:07:22.928 DEBUG:boto:Port: 8000 2023-07-29T01:07:22.929 DEBUG:boto:Params: {} 2023-07-29T01:07:22.929 DEBUG:boto:Token: None 2023-07-29T01:07:22.930 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 01:07:22 GMT /hiiqos-68/obj1 2023-07-29T01:07:22.930 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ImJhkaV4fxsDAYhe0wkpz1LcOak= 2023-07-29T01:07:22.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 01:07:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ImJhkaV4fxsDAYhe0wkpz1LcOak='} 2023-07-29T01:07:22.954 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000087690c35e59f800c-0064c4664a-24ac-a1'), ('Date', 'Sat, 29 Jul 2023 01:07:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:07:22.955 DEBUG:boto:path=/hiiqos-68/ 2023-07-29T01:07:22.955 DEBUG:boto:auth_path=/hiiqos-68/ 2023-07-29T01:07:22.955 DEBUG:boto:Method: HEAD 2023-07-29T01:07:22.956 DEBUG:boto:Path: /hiiqos-68/ 2023-07-29T01:07:22.956 DEBUG:boto:Data: 2023-07-29T01:07:22.957 DEBUG:boto:Headers: {} 2023-07-29T01:07:22.957 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:07:22.957 DEBUG:boto:Port: 8000 2023-07-29T01:07:22.958 DEBUG:boto:Params: {} 2023-07-29T01:07:22.958 DEBUG:boto:Token: None 2023-07-29T01:07:22.958 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:07:22 GMT /hiiqos-68/ 2023-07-29T01:07:22.959 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:O4WbOyOqrq17cMz3S1ZI+jtc3Ak= 2023-07-29T01:07:22.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:07:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:O4WbOyOqrq17cMz3S1ZI+jtc3Ak=', 'Content-Length': '0'} 2023-07-29T01:07:22.962 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a8b2494bff0dce32-0064c4664a-24ac-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:07:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:07:22.963 DEBUG:boto:path=/hiiqos-68/obj2 2023-07-29T01:07:22.963 DEBUG:boto:auth_path=/hiiqos-68/obj2 2023-07-29T01:07:22.964 DEBUG:boto:Method: PUT 2023-07-29T01:07:22.964 DEBUG:boto:Path: /hiiqos-68/obj2 2023-07-29T01:07:22.965 DEBUG:boto:Data: 2023-07-29T01:07:22.965 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T01:07:22.966 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:07:22.966 DEBUG:boto:Port: 8000 2023-07-29T01:07:22.966 DEBUG:boto:Params: {} 2023-07-29T01:07:22.967 DEBUG:boto:Token: None 2023-07-29T01:07:22.967 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 01:07:22 GMT /hiiqos-68/obj2 2023-07-29T01:07:22.967 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:fHHzDPGEe+brJhuuS7U62iDWq40= 2023-07-29T01:07:22.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 01:07:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fHHzDPGEe+brJhuuS7U62iDWq40='} 2023-07-29T01:07:23.016 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eeebef4acbaa53e3-0064c4664a-24ac-a1'), ('Date', 'Sat, 29 Jul 2023 01:07:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:07:23.018 DEBUG:boto:path=/hiiqos-68/ 2023-07-29T01:07:23.019 DEBUG:boto:auth_path=/hiiqos-68/ 2023-07-29T01:07:23.019 DEBUG:boto:Method: HEAD 2023-07-29T01:07:23.020 DEBUG:boto:Path: /hiiqos-68/ 2023-07-29T01:07:23.021 DEBUG:boto:Data: 2023-07-29T01:07:23.021 DEBUG:boto:Headers: {} 2023-07-29T01:07:23.022 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:07:23.023 DEBUG:boto:Port: 8000 2023-07-29T01:07:23.023 DEBUG:boto:Params: {} 2023-07-29T01:07:23.024 DEBUG:boto:Token: None 2023-07-29T01:07:23.024 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:07:23 GMT /hiiqos-68/ 2023-07-29T01:07:23.025 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ST8nICL/D9IonM6Js0bwJhl0YXA= 2023-07-29T01:07:23.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:07:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ST8nICL/D9IonM6Js0bwJhl0YXA=', 'Content-Length': '0'} 2023-07-29T01:07:23.031 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b2078cf8d354eec7-0064c4664a-24ac-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:07:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:07:23.032 DEBUG:boto:path=/hiiqos-68/obj3 2023-07-29T01:07:23.032 DEBUG:boto:auth_path=/hiiqos-68/obj3 2023-07-29T01:07:23.032 DEBUG:boto:Method: PUT 2023-07-29T01:07:23.033 DEBUG:boto:Path: /hiiqos-68/obj3 2023-07-29T01:07:23.033 DEBUG:boto:Data: 2023-07-29T01:07:23.033 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T01:07:23.034 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:07:23.034 DEBUG:boto:Port: 8000 2023-07-29T01:07:23.035 DEBUG:boto:Params: {} 2023-07-29T01:07:23.035 DEBUG:boto:Token: None 2023-07-29T01:07:23.035 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 01:07:23 GMT /hiiqos-68/obj3 2023-07-29T01:07:23.036 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:al9GsEwVxL9GbGygkBSbimpzcXw= 2023-07-29T01:07:23.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 01:07:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:al9GsEwVxL9GbGygkBSbimpzcXw='} 2023-07-29T01:07:23.051 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e2f553e224daa043-0064c4664a-24ac-a1'), ('Date', 'Sat, 29 Jul 2023 01:07:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:07:23.052 DEBUG:boto:path=/hiiqos-69/ 2023-07-29T01:07:23.052 DEBUG:boto:auth_path=/hiiqos-69/ 2023-07-29T01:07:23.053 DEBUG:boto:Method: HEAD 2023-07-29T01:07:23.053 DEBUG:boto:Path: /hiiqos-69/ 2023-07-29T01:07:23.053 DEBUG:boto:Data: 2023-07-29T01:07:23.054 DEBUG:boto:Headers: {} 2023-07-29T01:07:23.054 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:07:23.054 DEBUG:boto:Port: 8000 2023-07-29T01:07:23.055 DEBUG:boto:Params: {} 2023-07-29T01:07:23.055 DEBUG:boto:Token: None 2023-07-29T01:07:23.056 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:07:23 GMT /hiiqos-69/ 2023-07-29T01:07:23.057 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:D6J6iRDKxTuaYivw962JOzgkZOg= 2023-07-29T01:07:23.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:07:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:D6J6iRDKxTuaYivw962JOzgkZOg=', 'Content-Length': '0'} 2023-07-29T01:07:23.063 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e867b5933fe87603-0064c4664a-24ac-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:07:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:07:23.064 DEBUG:boto:path=/hiiqos-69/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-07-29T01:07:23.064 DEBUG:boto:auth_path=/hiiqos-69/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-07-29T01:07:23.065 DEBUG:boto:Method: PUT 2023-07-29T01:07:23.065 DEBUG:boto:Path: /hiiqos-69/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-07-29T01:07:23.066 DEBUG:boto:Data: 2023-07-29T01:07:23.066 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T01:07:23.066 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:07:23.067 DEBUG:boto:Port: 8000 2023-07-29T01:07:23.067 DEBUG:boto:Params: {} 2023-07-29T01:07:23.067 DEBUG:boto:Token: None 2023-07-29T01:07:23.068 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 01:07:23 GMT /hiiqos-69/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-07-29T01:07:23.068 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:0SOWNae1p5i2BFdTw5YWQ9DO9sU= 2023-07-29T01:07:23.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 01:07:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0SOWNae1p5i2BFdTw5YWQ9DO9sU='} 2023-07-29T01:07:23.112 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009cad53747bd1296d-0064c4664a-24ac-a1'), ('Date', 'Sat, 29 Jul 2023 01:07:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:07:23.113 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:07:23.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:07:23.114 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:07:23.211 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:07:23.211 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:07:23.286 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T01:07:23.317 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T01:07:23.317 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:23.317 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:07:23.317 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:17.836402Z" 2023-07-29T01:07:23.318 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:23.318 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:23.318 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592836.654753_1494.1", 2023-07-29T01:07:23.318 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:07:16.654753Z" 2023-07-29T01:07:23.318 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:23.318 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:23.319 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:07:23.319 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T01:07:23.319 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:23.320 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:23.320 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:07:23.320 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:12.811406Z" 2023-07-29T01:07:23.320 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:23.321 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T01:07:23.322 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690591217.836402_294.1', 1: '1_1690592836.654753_1494.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:07:23.322 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T01:07:23.323 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:07:23.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:07:23.323 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:07:23.400 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:07:23.400 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:07:23.447 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:07:23.361+0000 7fb0aa620880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:07:23.448 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:07:23.363+0000 7fb0aa620880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:07:23.466 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:07:23.467 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:07:23.467 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:07:23.467 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:07:23.468 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:07:23.468 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:07:23.468 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:07:23.468 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:23.468 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:07:23.469 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:23.469 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:07:23.469 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:23.469 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:07:23.469 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:07:23.469 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:23.469 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:07:23.470 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:23.470 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:07:23.470 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:07:23.470 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:23.470 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:23.470 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:23.470 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:07:23.471 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:23.471 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:07:23.471 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592836.654753_1494.1", 2023-07-29T01:07:23.471 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:23.471 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:07:23.471 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:23.471 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:16.654753Z", 2023-07-29T01:07:23.472 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:07:23.472 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:23.472 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:23.472 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:23.472 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:07:23.472 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:23.473 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:07:23.473 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:07:23.473 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:23.473 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:07:23.473 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:23.473 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:07:23.473 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:07:23.474 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:23.474 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:23.474 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:23.474 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:07:23.474 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:23.474 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:07:23.474 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:07:23.475 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:23.475 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:07:23.475 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:23.475 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:07:23.475 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:07:23.475 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:23.475 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:23.476 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:07:23.476 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:23.476 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:07:23.476 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:07:23.476 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:07:23.476 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:23.477 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:07:23Z" 2023-07-29T01:07:23.477 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:07:23.478 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592836.654753_1494.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:16.654753Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:07:23Z" } 2023-07-29T01:07:23.478 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592836.654753_1494.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:16.654753Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:07:23.478 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592836.654753_1494.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:07:23.479 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592836.654753_1494.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:07:23.479 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T01:07:23.479 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:07:23.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:07:23.480 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:07:23.536 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:07:23.537 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:07:23.587 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T01:07:23.587 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:23.588 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051673", 2023-07-29T01:07:23.588 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:07:22.963361Z" 2023-07-29T01:07:23.588 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:23.589 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:23.589 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044881", 2023-07-29T01:07:23.589 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:07:23.001407Z" 2023-07-29T01:07:23.589 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:23.590 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:23.590 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045660", 2023-07-29T01:07:23.590 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:07:22.902962Z" 2023-07-29T01:07:23.591 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:23.591 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:23.591 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:07:23.592 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:07:23.592 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:23.592 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T01:07:23.594 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000051673', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:07:23.594 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-07-29T01:07:23.594 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:07:23.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:07:23.595 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:07:23.643 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:07:23.644 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:07:23.686 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:07:23.686 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:07:23.687 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:07:23.687 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:07:23.687 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:07:23.687 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:07:23.687 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:23.687 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:07:23.688 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:23.688 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:07:23.688 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:23.688 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:07:23.688 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051393", 2023-07-29T01:07:23.688 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:23.689 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:07:23.689 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:23.689 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:52.529981Z" 2023-07-29T01:07:23.689 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:23.689 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:23.689 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:23.689 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:07:23.690 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:23.690 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:07:23.690 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044739", 2023-07-29T01:07:23.690 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:23.690 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:07:23.690 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:23.691 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:23.964578Z" 2023-07-29T01:07:23.691 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:23.691 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:23.691 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:23.691 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:07:23.692 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:23.692 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:07:23.692 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045520", 2023-07-29T01:07:23.692 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:23.692 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:07:23.692 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:23.692 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:06:23.586689Z" 2023-07-29T01:07:23.693 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:23.693 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:23.693 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:23.693 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:07:23.693 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:23.693 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:07:23.693 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:07:23.694 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:23.694 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:07:23.694 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:23.694 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:07:23.694 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:23.694 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:23.694 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:07:23.695 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:23.695 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:07:23.695 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:07:23.695 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:07:23.695 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:23.695 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:07:23Z" 2023-07-29T01:07:23.695 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:07:23.696 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051393", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:04:52.529981Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044739", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T00:58:23.964578Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045520", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:06:23.586689Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:07:23Z" } 2023-07-29T01:07:23.697 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051393', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:04:52.529981Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044739', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T00:58:23.964578Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045520', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:06:23.586689Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:07:23.697 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051673', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:07:23.698 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051393', 1: '00000000000000000000:00000000000000044739', 2: '00000000000000000000:00000000000000045520', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:07:23.698 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051673 target=00000000000000000000:00000000000000051393, shard=1 master=00000000000000000000:00000000000000044881 target=00000000000000000000:00000000000000044739, shard=2 master=00000000000000000000:00000000000000045660 target=00000000000000000000:00000000000000045520 2023-07-29T01:07:28.699 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:07:28.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:07:28.701 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:07:28.755 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:07:28.755 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:07:28.801 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:07:28.801 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:07:28.801 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:07:28.801 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:07:28.802 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:07:28.802 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:07:28.802 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:28.802 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:07:28.802 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:28.802 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:07:28.803 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:28.803 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:07:28.803 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051393", 2023-07-29T01:07:28.803 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:28.803 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:07:28.803 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:28.803 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:52.529981Z" 2023-07-29T01:07:28.804 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:28.804 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:28.804 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:28.804 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:07:28.804 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:28.804 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:07:28.805 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044739", 2023-07-29T01:07:28.805 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:28.805 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:07:28.805 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:28.805 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:23.964578Z" 2023-07-29T01:07:28.805 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:28.805 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:28.806 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:28.806 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:07:28.806 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:28.806 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:07:28.806 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045520", 2023-07-29T01:07:28.806 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:28.806 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:07:28.807 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:28.807 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:06:23.586689Z" 2023-07-29T01:07:28.807 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:28.807 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:28.807 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:28.807 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:07:28.807 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:28.808 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:07:28.808 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:07:28.808 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:28.808 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:07:28.808 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:28.808 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:07:28.809 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:28.809 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:28.809 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:07:28.809 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:28.809 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:07:28.809 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:07:28.809 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:07:28.810 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:28.810 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:07:28Z" 2023-07-29T01:07:28.810 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:07:28.811 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051393", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:04:52.529981Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044739", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T00:58:23.964578Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045520", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:06:23.586689Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:07:28Z" } 2023-07-29T01:07:28.811 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051393', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:04:52.529981Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044739', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T00:58:23.964578Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045520', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:06:23.586689Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:07:28.812 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051673', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:07:28.812 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051393', 1: '00000000000000000000:00000000000000044739', 2: '00000000000000000000:00000000000000045520', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:07:28.813 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051673 target=00000000000000000000:00000000000000051393, shard=1 master=00000000000000000000:00000000000000044881 target=00000000000000000000:00000000000000044739, shard=2 master=00000000000000000000:00000000000000045660 target=00000000000000000000:00000000000000045520 2023-07-29T01:07:33.814 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:07:33.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:07:33.815 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:07:33.870 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:07:33.870 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:07:33.918 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:07:33.918 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:07:33.918 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:07:33.918 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:07:33.918 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:07:33.919 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:07:33.919 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:33.919 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:07:33.919 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:33.919 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:07:33.919 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:33.920 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:07:33.920 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051393", 2023-07-29T01:07:33.920 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:33.920 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:07:33.920 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:33.920 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:52.529981Z" 2023-07-29T01:07:33.920 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:33.921 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:33.921 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:33.921 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:07:33.921 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:33.921 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:07:33.921 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044739", 2023-07-29T01:07:33.921 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:33.922 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:07:33.922 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:33.922 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:23.964578Z" 2023-07-29T01:07:33.922 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:33.922 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:33.922 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:33.922 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:07:33.923 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:33.923 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:07:33.923 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045520", 2023-07-29T01:07:33.923 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:33.923 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:07:33.923 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:33.923 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:06:23.586689Z" 2023-07-29T01:07:33.924 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:33.924 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:33.924 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:33.924 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:07:33.924 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:33.924 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:07:33.924 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:07:33.925 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:33.925 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:07:33.925 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:33.925 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:07:33.925 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:33.925 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:33.926 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:07:33.926 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:33.926 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:07:33.926 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:07:33.926 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:07:33.926 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:33.926 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:07:33Z" 2023-07-29T01:07:33.927 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:07:33.927 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051393", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:04:52.529981Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044739", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T00:58:23.964578Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045520", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:06:23.586689Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:07:33Z" } 2023-07-29T01:07:33.928 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051393', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:04:52.529981Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044739', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T00:58:23.964578Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045520', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:06:23.586689Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:07:33.928 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051673', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:07:33.929 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051393', 1: '00000000000000000000:00000000000000044739', 2: '00000000000000000000:00000000000000045520', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:07:33.929 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051673 target=00000000000000000000:00000000000000051393, shard=1 master=00000000000000000000:00000000000000044881 target=00000000000000000000:00000000000000044739, shard=2 master=00000000000000000000:00000000000000045660 target=00000000000000000000:00000000000000045520 2023-07-29T01:07:38.930 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:07:38.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:07:38.931 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:07:38.986 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:07:38.986 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:07:39.035 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:07:39.035 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:07:39.035 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:07:39.036 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:07:39.036 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:07:39.036 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:07:39.036 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:39.036 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:07:39.036 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:39.037 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:07:39.037 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:39.037 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:07:39.037 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051393", 2023-07-29T01:07:39.037 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:39.037 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:07:39.038 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:39.038 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:52.529981Z" 2023-07-29T01:07:39.038 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:39.039 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:39.039 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:39.039 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:07:39.039 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:39.039 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:07:39.039 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044739", 2023-07-29T01:07:39.039 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:39.040 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:07:39.040 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:39.040 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:23.964578Z" 2023-07-29T01:07:39.040 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:39.040 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:39.040 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:39.041 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:07:39.041 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:39.041 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:07:39.041 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045520", 2023-07-29T01:07:39.042 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:39.042 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:07:39.042 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:39.043 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:06:23.586689Z" 2023-07-29T01:07:39.043 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:39.043 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:39.044 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:39.044 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:07:39.044 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:39.045 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:07:39.045 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:07:39.045 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:39.046 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:07:39.046 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:39.046 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:07:39.047 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:39.047 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:39.047 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:07:39.048 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:39.048 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:07:39.048 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:07:39.049 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:07:39.049 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:39.049 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:07:38Z" 2023-07-29T01:07:39.049 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:07:39.051 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051393", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:04:52.529981Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044739", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T00:58:23.964578Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045520", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:06:23.586689Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:07:38Z" } 2023-07-29T01:07:39.052 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051393', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:04:52.529981Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044739', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T00:58:23.964578Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045520', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:06:23.586689Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:07:39.052 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051673', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:07:39.053 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051393', 1: '00000000000000000000:00000000000000044739', 2: '00000000000000000000:00000000000000045520', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:07:39.053 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051673 target=00000000000000000000:00000000000000051393, shard=1 master=00000000000000000000:00000000000000044881 target=00000000000000000000:00000000000000044739, shard=2 master=00000000000000000000:00000000000000045660 target=00000000000000000000:00000000000000045520 2023-07-29T01:07:44.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:07:44.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:07:44.055 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:07:44.111 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:07:44.112 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:07:44.159 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:07:44.160 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:07:44.160 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:07:44.160 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:07:44.160 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:07:44.161 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:07:44.161 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:44.161 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:07:44.161 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:44.161 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:07:44.161 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:44.161 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:07:44.162 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051393", 2023-07-29T01:07:44.162 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:44.162 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:07:44.162 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:44.162 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:52.529981Z" 2023-07-29T01:07:44.162 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:44.162 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:44.163 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:44.163 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:07:44.163 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:44.163 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:07:44.163 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044739", 2023-07-29T01:07:44.163 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:44.163 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:07:44.164 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:44.164 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:23.964578Z" 2023-07-29T01:07:44.164 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:44.164 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:44.164 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:44.164 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:07:44.165 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:44.165 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:07:44.165 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045520", 2023-07-29T01:07:44.165 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:44.165 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:07:44.165 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:44.165 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:06:23.586689Z" 2023-07-29T01:07:44.166 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:44.166 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:44.166 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:44.166 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:07:44.166 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:44.166 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:07:44.166 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:07:44.167 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:44.167 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:07:44.167 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:44.167 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:07:44.167 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:44.167 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:44.167 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:07:44.168 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:44.168 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:07:44.168 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:07:44.168 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:07:44.168 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:44.168 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:07:44Z" 2023-07-29T01:07:44.169 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:07:44.170 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051393", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:04:52.529981Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044739", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T00:58:23.964578Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045520", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:06:23.586689Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:07:44Z" } 2023-07-29T01:07:44.170 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051393', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:04:52.529981Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044739', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T00:58:23.964578Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045520', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:06:23.586689Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:07:44.171 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051673', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:07:44.171 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051393', 1: '00000000000000000000:00000000000000044739', 2: '00000000000000000000:00000000000000045520', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:07:44.171 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051673 target=00000000000000000000:00000000000000051393, shard=1 master=00000000000000000000:00000000000000044881 target=00000000000000000000:00000000000000044739, shard=2 master=00000000000000000000:00000000000000045660 target=00000000000000000000:00000000000000045520 2023-07-29T01:07:49.173 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:07:49.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:07:49.175 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:07:49.232 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:07:49.233 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:07:49.281 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:07:49.281 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:07:49.281 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:07:49.281 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:07:49.281 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:07:49.282 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:07:49.282 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:49.282 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:07:49.282 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:49.282 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:07:49.282 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:49.283 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:07:49.283 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051393", 2023-07-29T01:07:49.283 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:49.283 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:07:49.283 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:49.283 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:52.529981Z" 2023-07-29T01:07:49.284 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:49.284 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:49.284 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:49.284 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:07:49.284 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:49.284 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:07:49.285 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044739", 2023-07-29T01:07:49.285 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:49.285 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:07:49.285 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:49.285 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:23.964578Z" 2023-07-29T01:07:49.285 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:49.286 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:49.286 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:49.286 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:07:49.286 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:49.286 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:07:49.286 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045520", 2023-07-29T01:07:49.287 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:49.287 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:07:49.287 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:49.287 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:06:23.586689Z" 2023-07-29T01:07:49.287 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:49.287 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:49.287 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:49.288 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:07:49.288 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:49.288 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:07:49.288 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:07:49.289 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:49.289 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:07:49.289 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:49.289 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:07:49.289 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:49.289 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:49.290 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:07:49.290 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:49.290 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:07:49.290 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:07:49.290 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:07:49.290 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:49.290 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:07:49Z" 2023-07-29T01:07:49.291 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:07:49.292 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051393", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:04:52.529981Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044739", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T00:58:23.964578Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045520", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:06:23.586689Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:07:49Z" } 2023-07-29T01:07:49.292 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051393', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:04:52.529981Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044739', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T00:58:23.964578Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045520', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:06:23.586689Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:07:49.293 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051673', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:07:49.293 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051393', 1: '00000000000000000000:00000000000000044739', 2: '00000000000000000000:00000000000000045520', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:07:49.293 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051673 target=00000000000000000000:00000000000000051393, shard=1 master=00000000000000000000:00000000000000044881 target=00000000000000000000:00000000000000044739, shard=2 master=00000000000000000000:00000000000000045660 target=00000000000000000000:00000000000000045520 2023-07-29T01:07:54.294 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:07:54.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:07:54.295 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:07:54.350 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:07:54.350 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:07:54.398 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:07:54.398 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:07:54.398 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:07:54.399 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:07:54.399 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:07:54.399 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:07:54.399 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:54.399 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:07:54.399 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:54.400 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:07:54.400 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:54.400 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:07:54.400 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051393", 2023-07-29T01:07:54.400 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:54.400 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:07:54.401 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:54.401 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:52.529981Z" 2023-07-29T01:07:54.401 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:54.401 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:54.401 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:54.401 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:07:54.402 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:54.402 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:07:54.402 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044739", 2023-07-29T01:07:54.402 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:54.402 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:07:54.402 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:54.402 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:23.964578Z" 2023-07-29T01:07:54.403 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:54.403 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:54.403 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:54.403 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:07:54.403 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:54.403 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:07:54.403 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045520", 2023-07-29T01:07:54.404 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:54.404 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:07:54.404 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:54.404 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:06:23.586689Z" 2023-07-29T01:07:54.404 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:54.404 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:54.405 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:54.405 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:07:54.405 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:54.405 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:07:54.405 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:07:54.405 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:54.405 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:07:54.406 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:54.406 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:07:54.406 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:54.406 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:54.406 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:07:54.406 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:54.406 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:07:54.407 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:07:54.407 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:07:54.407 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:54.407 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:07:54Z" 2023-07-29T01:07:54.407 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:07:54.408 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051393", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:04:52.529981Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044739", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T00:58:23.964578Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045520", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:06:23.586689Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:07:54Z" } 2023-07-29T01:07:54.409 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051393', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:04:52.529981Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044739', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T00:58:23.964578Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045520', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:06:23.586689Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:07:54.409 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051673', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:07:54.409 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051393', 1: '00000000000000000000:00000000000000044739', 2: '00000000000000000000:00000000000000045520', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:07:54.410 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051673 target=00000000000000000000:00000000000000051393, shard=1 master=00000000000000000000:00000000000000044881 target=00000000000000000000:00000000000000044739, shard=2 master=00000000000000000000:00000000000000045660 target=00000000000000000000:00000000000000045520 2023-07-29T01:07:59.411 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:07:59.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:07:59.413 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:07:59.472 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:07:59.472 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:07:59.521 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:07:59.522 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:07:59.522 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:07:59.522 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:07:59.523 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:07:59.523 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:07:59.523 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:59.523 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:07:59.524 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:59.524 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:07:59.524 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:59.525 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:07:59.525 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051393", 2023-07-29T01:07:59.525 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:59.525 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:07:59.525 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:59.525 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:52.529981Z" 2023-07-29T01:07:59.525 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:59.526 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:59.526 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:59.526 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:07:59.526 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:59.526 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:07:59.526 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044739", 2023-07-29T01:07:59.527 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:59.527 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:07:59.527 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:59.528 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:23.964578Z" 2023-07-29T01:07:59.528 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:59.528 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:59.528 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:59.529 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:07:59.529 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:59.529 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:07:59.530 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045520", 2023-07-29T01:07:59.530 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:59.530 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:07:59.531 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:59.531 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:06:23.586689Z" 2023-07-29T01:07:59.531 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:59.531 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:59.532 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:07:59.532 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:07:59.532 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:07:59.533 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:07:59.533 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:07:59.533 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:07:59.534 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:07:59.534 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:07:59.534 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:07:59.534 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:59.535 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:07:59.535 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:07:59.535 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:59.535 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:07:59.535 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:07:59.536 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:07:59.536 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:07:59.536 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:07:59Z" 2023-07-29T01:07:59.536 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:07:59.537 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051393", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:04:52.529981Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044739", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T00:58:23.964578Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045520", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:06:23.586689Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:07:59Z" } 2023-07-29T01:07:59.538 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051393', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:04:52.529981Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044739', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T00:58:23.964578Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045520', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:06:23.586689Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:07:59.538 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051673', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:07:59.538 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051393', 1: '00000000000000000000:00000000000000044739', 2: '00000000000000000000:00000000000000045520', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:07:59.539 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051673 target=00000000000000000000:00000000000000051393, shard=1 master=00000000000000000000:00000000000000044881 target=00000000000000000000:00000000000000044739, shard=2 master=00000000000000000000:00000000000000045660 target=00000000000000000000:00000000000000045520 2023-07-29T01:08:04.539 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:08:04.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:08:04.542 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:08:04.601 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:08:04.601 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:08:04.652 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:08:04.652 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:08:04.653 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:08:04.653 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:08:04.653 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:08:04.653 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:08:04.653 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:04.653 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:08:04.653 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:04.654 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:08:04.654 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:04.654 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:04.654 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051393", 2023-07-29T01:08:04.654 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:04.655 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:08:04.655 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:04.655 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:52.529981Z" 2023-07-29T01:08:04.655 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:04.655 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:04.656 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:04.656 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:08:04.656 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:04.656 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:04.656 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044739", 2023-07-29T01:08:04.656 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:04.656 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:08:04.657 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:04.657 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:23.964578Z" 2023-07-29T01:08:04.657 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:04.657 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:04.657 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:04.657 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:08:04.657 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:04.658 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:04.658 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045520", 2023-07-29T01:08:04.658 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:04.658 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:08:04.658 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:04.658 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:06:23.586689Z" 2023-07-29T01:08:04.658 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:04.659 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:04.659 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:04.659 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:08:04.659 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:04.659 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:04.659 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:08:04.659 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:04.660 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:08:04.660 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:04.660 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:08:04.660 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:04.660 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:04.660 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:08:04.660 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:04.661 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:08:04.661 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:08:04.661 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:08:04.661 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:04.661 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:08:04Z" 2023-07-29T01:08:04.661 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:08:04.662 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051393", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:04:52.529981Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044739", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T00:58:23.964578Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045520", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:06:23.586689Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:08:04Z" } 2023-07-29T01:08:04.662 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051393', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:04:52.529981Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044739', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T00:58:23.964578Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045520', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:06:23.586689Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:08:04.662 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051673', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:08:04.663 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051393', 1: '00000000000000000000:00000000000000044739', 2: '00000000000000000000:00000000000000045520', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:08:04.663 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051673 target=00000000000000000000:00000000000000051393, shard=1 master=00000000000000000000:00000000000000044881 target=00000000000000000000:00000000000000044739, shard=2 master=00000000000000000000:00000000000000045660 target=00000000000000000000:00000000000000045520 2023-07-29T01:08:09.664 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:08:09.665 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:08:09.666 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:08:09.718 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:08:09.719 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:08:09.765 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:08:09.766 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:08:09.766 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:08:09.766 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:08:09.766 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:08:09.767 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:08:09.767 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:09.767 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:08:09.767 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:09.767 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:08:09.768 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:09.768 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:09.768 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051393", 2023-07-29T01:08:09.768 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:09.769 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:08:09.769 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:09.769 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:52.529981Z" 2023-07-29T01:08:09.769 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:09.769 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:09.769 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:09.770 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:08:09.770 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:09.770 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:09.770 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044739", 2023-07-29T01:08:09.770 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:09.770 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:08:09.771 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:09.771 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:23.964578Z" 2023-07-29T01:08:09.771 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:09.771 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:09.771 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:09.772 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:08:09.772 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:09.772 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:09.773 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045520", 2023-07-29T01:08:09.773 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:09.773 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:08:09.773 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:09.773 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:06:23.586689Z" 2023-07-29T01:08:09.774 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:09.774 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:09.774 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:09.774 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:08:09.774 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:09.775 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:09.775 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:08:09.775 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:09.775 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:08:09.775 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:09.775 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:08:09.776 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:09.776 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:09.776 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:08:09.776 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:09.776 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:08:09.776 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:08:09.777 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:08:09.777 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:09.777 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:08:09Z" 2023-07-29T01:08:09.777 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:08:09.778 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051393", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:04:52.529981Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044739", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T00:58:23.964578Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045520", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:06:23.586689Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:08:09Z" } 2023-07-29T01:08:09.779 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051393', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:04:52.529981Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044739', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T00:58:23.964578Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045520', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:06:23.586689Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:08:09.780 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051673', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:08:09.780 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051393', 1: '00000000000000000000:00000000000000044739', 2: '00000000000000000000:00000000000000045520', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:08:09.781 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051673 target=00000000000000000000:00000000000000051393, shard=1 master=00000000000000000000:00000000000000044881 target=00000000000000000000:00000000000000044739, shard=2 master=00000000000000000000:00000000000000045660 target=00000000000000000000:00000000000000045520 2023-07-29T01:08:14.783 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:08:14.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:08:14.784 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:08:14.839 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:08:14.839 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:08:14.886 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:08:14.887 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:08:14.887 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:08:14.887 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:08:14.887 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:08:14.887 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:08:14.887 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:14.888 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:08:14.888 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:14.888 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:08:14.888 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:14.888 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:14.888 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051393", 2023-07-29T01:08:14.888 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:14.889 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:08:14.889 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:14.889 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:52.529981Z" 2023-07-29T01:08:14.889 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:14.889 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:14.889 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:14.889 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:08:14.890 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:14.890 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:14.890 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044739", 2023-07-29T01:08:14.890 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:14.890 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:08:14.890 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:14.891 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:23.964578Z" 2023-07-29T01:08:14.891 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:14.891 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:14.891 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:14.891 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:08:14.891 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:14.891 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:14.892 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045520", 2023-07-29T01:08:14.892 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:14.892 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:08:14.892 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:14.892 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:06:23.586689Z" 2023-07-29T01:08:14.893 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:14.893 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:14.893 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:14.893 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:08:14.893 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:14.893 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:14.893 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:08:14.894 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:14.894 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:08:14.894 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:14.894 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:08:14.894 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:14.894 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:14.895 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:08:14.895 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:14.895 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:08:14.895 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:08:14.895 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:08:14.895 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:14.895 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:08:14Z" 2023-07-29T01:08:14.896 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:08:14.897 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051393", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:04:52.529981Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044739", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T00:58:23.964578Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045520", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:06:23.586689Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:08:14Z" } 2023-07-29T01:08:14.897 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051393', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:04:52.529981Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044739', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T00:58:23.964578Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045520', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:06:23.586689Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:08:14.897 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051673', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:08:14.898 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051393', 1: '00000000000000000000:00000000000000044739', 2: '00000000000000000000:00000000000000045520', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:08:14.898 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051673 target=00000000000000000000:00000000000000051393, shard=1 master=00000000000000000000:00000000000000044881 target=00000000000000000000:00000000000000044739, shard=2 master=00000000000000000000:00000000000000045660 target=00000000000000000000:00000000000000045520 2023-07-29T01:08:19.899 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:08:19.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:08:19.901 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:08:19.956 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:08:19.957 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:08:20.004 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:08:20.005 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:08:20.005 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:08:20.005 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:08:20.005 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:08:20.006 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:08:20.006 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:20.006 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:08:20.006 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:20.007 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:08:20.007 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:20.007 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:20.007 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051393", 2023-07-29T01:08:20.007 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:20.007 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:08:20.008 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:20.008 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:52.529981Z" 2023-07-29T01:08:20.008 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:20.008 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:20.009 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:20.009 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:08:20.009 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:20.010 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:20.010 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044739", 2023-07-29T01:08:20.010 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:20.010 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:08:20.011 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:20.012 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:23.964578Z" 2023-07-29T01:08:20.012 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:20.012 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:20.012 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:20.012 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:08:20.013 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:20.013 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:20.013 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045520", 2023-07-29T01:08:20.013 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:20.013 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:08:20.013 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:20.013 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:06:23.586689Z" 2023-07-29T01:08:20.014 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:20.014 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:20.014 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:20.014 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:08:20.014 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:20.014 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:20.015 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:08:20.015 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:20.015 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:08:20.015 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:20.015 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:08:20.015 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:20.015 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:20.016 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:08:20.016 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:20.016 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:08:20.016 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:08:20.016 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:08:20.016 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:20.017 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:08:19Z" 2023-07-29T01:08:20.017 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:08:20.018 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051393", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:04:52.529981Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044739", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T00:58:23.964578Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045520", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:06:23.586689Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:08:19Z" } 2023-07-29T01:08:20.018 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051393', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:04:52.529981Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044739', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T00:58:23.964578Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045520', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:06:23.586689Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:08:20.019 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051673', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:08:20.019 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051393', 1: '00000000000000000000:00000000000000044739', 2: '00000000000000000000:00000000000000045520', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:08:20.019 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051673 target=00000000000000000000:00000000000000051393, shard=1 master=00000000000000000000:00000000000000044881 target=00000000000000000000:00000000000000044739, shard=2 master=00000000000000000000:00000000000000045660 target=00000000000000000000:00000000000000045520 2023-07-29T01:08:24.258 INFO:tasks.rgw.c2.client.0.smithi139.stdout:2023-07-29T01:08:24.171+0000 7f5f2671b640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T01:08:24.263 INFO:tasks.rgw.c2.client.1.smithi139.stdout:2023-07-29T01:08:24.176+0000 7f5731a6a640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T01:08:25.020 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:08:25.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:08:25.022 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:08:25.078 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:08:25.079 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:08:25.127 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:08:25.127 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:08:25.127 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:08:25.127 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:08:25.127 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:08:25.128 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:08:25.128 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:25.128 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:08:25.128 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:25.128 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:08:25.128 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:25.129 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:25.129 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051393", 2023-07-29T01:08:25.129 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:25.129 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:08:25.130 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:25.130 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:52.529981Z" 2023-07-29T01:08:25.130 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:25.130 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:25.131 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:25.131 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:08:25.131 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:25.132 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:25.132 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044739", 2023-07-29T01:08:25.132 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:25.133 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:08:25.133 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:25.133 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:23.964578Z" 2023-07-29T01:08:25.133 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:25.134 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:25.134 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:25.134 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:08:25.135 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:25.135 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:25.135 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045520", 2023-07-29T01:08:25.135 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:25.135 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:08:25.136 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:25.136 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:06:23.586689Z" 2023-07-29T01:08:25.136 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:25.136 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:25.136 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:25.136 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:08:25.137 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:25.137 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:25.137 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:08:25.137 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:25.137 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:08:25.137 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:25.137 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:08:25.138 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:25.138 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:25.138 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:08:25.138 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:25.138 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:08:25.138 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:08:25.138 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:08:25.139 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:25.139 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:08:25Z" 2023-07-29T01:08:25.139 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:08:25.140 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051393", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:04:52.529981Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044739", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T00:58:23.964578Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045520", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:06:23.586689Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:08:25Z" } 2023-07-29T01:08:25.140 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051393', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:04:52.529981Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044739', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T00:58:23.964578Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045520', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:06:23.586689Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:08:25.141 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051673', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:08:25.141 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051393', 1: '00000000000000000000:00000000000000044739', 2: '00000000000000000000:00000000000000045520', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:08:25.141 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051673 target=00000000000000000000:00000000000000051393, shard=1 master=00000000000000000000:00000000000000044881 target=00000000000000000000:00000000000000044739, shard=2 master=00000000000000000000:00000000000000045660 target=00000000000000000000:00000000000000045520 2023-07-29T01:08:30.142 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:08:30.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:08:30.143 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:08:30.198 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:08:30.199 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:08:30.249 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:08:30.249 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:08:30.249 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:08:30.250 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:08:30.250 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:08:30.250 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:08:30.250 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:30.250 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:08:30.250 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:30.250 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:08:30.251 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:30.251 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:30.251 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051393", 2023-07-29T01:08:30.251 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:30.251 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:08:30.251 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:30.251 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:52.529981Z" 2023-07-29T01:08:30.252 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:30.252 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:30.252 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:30.252 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:08:30.252 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:30.253 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:30.253 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044739", 2023-07-29T01:08:30.253 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:30.253 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:08:30.253 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:30.253 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:23.964578Z" 2023-07-29T01:08:30.253 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:30.254 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:30.254 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:30.254 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:08:30.254 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:30.254 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:30.254 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045520", 2023-07-29T01:08:30.254 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:30.255 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:08:30.255 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:30.255 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:06:23.586689Z" 2023-07-29T01:08:30.255 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:30.256 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:30.256 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:30.256 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:08:30.256 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:30.256 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:30.256 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:08:30.257 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:30.257 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:08:30.257 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:30.257 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:08:30.257 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:30.257 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:30.258 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:08:30.258 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:30.258 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:08:30.258 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:08:30.258 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:08:30.258 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:30.258 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:08:30Z" 2023-07-29T01:08:30.259 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:08:30.260 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051393", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:04:52.529981Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044739", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T00:58:23.964578Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045520", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:06:23.586689Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:08:30Z" } 2023-07-29T01:08:30.260 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051393', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:04:52.529981Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044739', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T00:58:23.964578Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045520', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:06:23.586689Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:08:30.261 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051673', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:08:30.261 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051393', 1: '00000000000000000000:00000000000000044739', 2: '00000000000000000000:00000000000000045520', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:08:30.261 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051673 target=00000000000000000000:00000000000000051393, shard=1 master=00000000000000000000:00000000000000044881 target=00000000000000000000:00000000000000044739, shard=2 master=00000000000000000000:00000000000000045660 target=00000000000000000000:00000000000000045520 2023-07-29T01:08:35.262 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:08:35.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:08:35.263 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:08:35.318 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:08:35.318 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:08:35.364 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:08:35.365 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:08:35.365 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:08:35.366 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:08:35.366 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:08:35.366 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:08:35.366 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:35.367 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:08:35.367 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:35.367 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:08:35.367 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:35.367 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:35.367 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051393", 2023-07-29T01:08:35.368 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:35.368 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:08:35.368 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:35.368 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:52.529981Z" 2023-07-29T01:08:35.368 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:35.368 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:35.369 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:35.369 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:08:35.369 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:35.369 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:35.369 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044739", 2023-07-29T01:08:35.369 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:35.370 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:08:35.370 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:35.370 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:23.964578Z" 2023-07-29T01:08:35.370 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:35.370 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:35.370 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:35.371 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:08:35.371 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:35.371 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:35.371 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045520", 2023-07-29T01:08:35.371 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:35.372 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:08:35.372 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:35.372 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:06:23.586689Z" 2023-07-29T01:08:35.372 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:35.373 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:35.373 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:35.373 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:08:35.373 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:35.373 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:35.374 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:08:35.374 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:35.374 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:08:35.374 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:35.374 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:08:35.374 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:35.374 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:35.375 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:08:35.375 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:35.375 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:08:35.375 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:08:35.375 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:08:35.375 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:35.375 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:08:35Z" 2023-07-29T01:08:35.376 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:08:35.377 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051393", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:04:52.529981Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044739", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T00:58:23.964578Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045520", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:06:23.586689Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:08:35Z" } 2023-07-29T01:08:35.378 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051393', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:04:52.529981Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044739', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T00:58:23.964578Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045520', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:06:23.586689Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:08:35.378 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051673', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:08:35.379 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051393', 1: '00000000000000000000:00000000000000044739', 2: '00000000000000000000:00000000000000045520', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:08:35.379 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051673 target=00000000000000000000:00000000000000051393, shard=1 master=00000000000000000000:00000000000000044881 target=00000000000000000000:00000000000000044739, shard=2 master=00000000000000000000:00000000000000045660 target=00000000000000000000:00000000000000045520 2023-07-29T01:08:40.380 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:08:40.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:08:40.381 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:08:40.437 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:08:40.438 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:08:40.485 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:08:40.485 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:08:40.485 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:08:40.485 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:08:40.486 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:08:40.486 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:08:40.486 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:40.486 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:08:40.486 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:40.486 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:08:40.486 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:40.487 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:40.487 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051393", 2023-07-29T01:08:40.487 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:40.487 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:08:40.487 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:40.487 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:52.529981Z" 2023-07-29T01:08:40.488 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:40.488 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:40.488 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:40.488 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:08:40.488 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:40.489 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:40.489 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044739", 2023-07-29T01:08:40.489 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:40.489 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:08:40.489 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:40.489 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:23.964578Z" 2023-07-29T01:08:40.489 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:40.490 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:40.490 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:40.490 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:08:40.490 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:40.490 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:40.490 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045520", 2023-07-29T01:08:40.490 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:40.491 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:08:40.491 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:40.491 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:06:23.586689Z" 2023-07-29T01:08:40.491 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:40.491 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:40.491 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:40.491 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:08:40.492 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:40.492 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:40.492 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:08:40.492 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:40.492 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:08:40.492 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:40.492 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:08:40.493 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:40.493 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:40.493 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:08:40.493 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:40.493 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:08:40.493 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:08:40.494 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:08:40.494 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:40.494 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:08:40Z" 2023-07-29T01:08:40.494 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:08:40.495 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051393", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:04:52.529981Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044739", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T00:58:23.964578Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045520", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:06:23.586689Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:08:40Z" } 2023-07-29T01:08:40.495 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051393', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:04:52.529981Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044739', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T00:58:23.964578Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045520', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:06:23.586689Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:08:40.496 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051673', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:08:40.496 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051393', 1: '00000000000000000000:00000000000000044739', 2: '00000000000000000000:00000000000000045520', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:08:40.496 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051673 target=00000000000000000000:00000000000000051393, shard=1 master=00000000000000000000:00000000000000044881 target=00000000000000000000:00000000000000044739, shard=2 master=00000000000000000000:00000000000000045660 target=00000000000000000000:00000000000000045520 2023-07-29T01:08:45.498 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:08:45.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:08:45.499 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:08:45.554 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:08:45.554 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:08:45.600 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:08:45.601 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:08:45.601 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:08:45.601 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:08:45.601 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:08:45.601 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:08:45.602 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:45.602 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:08:45.602 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:45.602 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:08:45.602 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:45.602 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:45.602 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051393", 2023-07-29T01:08:45.603 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:45.603 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:08:45.603 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:45.603 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:52.529981Z" 2023-07-29T01:08:45.603 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:45.603 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:45.603 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:45.604 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:08:45.604 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:45.604 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:45.604 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044739", 2023-07-29T01:08:45.604 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:45.604 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:08:45.605 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:45.605 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:23.964578Z" 2023-07-29T01:08:45.605 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:45.605 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:45.605 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:45.605 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:08:45.605 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:45.606 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:45.606 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045520", 2023-07-29T01:08:45.606 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:45.606 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:08:45.606 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:45.606 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:06:23.586689Z" 2023-07-29T01:08:45.607 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:45.607 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:45.607 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:45.607 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:08:45.607 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:45.607 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:45.607 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:08:45.608 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:45.608 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:08:45.608 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:45.608 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:08:45.608 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:45.608 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:45.609 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:08:45.609 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:45.609 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:08:45.609 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:08:45.609 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:08:45.609 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:45.609 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:08:45Z" 2023-07-29T01:08:45.610 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:08:45.610 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051393", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:04:52.529981Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044739", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T00:58:23.964578Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045520", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:06:23.586689Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:08:45Z" } 2023-07-29T01:08:45.611 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051393', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:04:52.529981Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044739', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T00:58:23.964578Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045520', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:06:23.586689Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:08:45.611 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051673', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:08:45.612 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051393', 1: '00000000000000000000:00000000000000044739', 2: '00000000000000000000:00000000000000045520', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:08:45.612 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051673 target=00000000000000000000:00000000000000051393, shard=1 master=00000000000000000000:00000000000000044881 target=00000000000000000000:00000000000000044739, shard=2 master=00000000000000000000:00000000000000045660 target=00000000000000000000:00000000000000045520 2023-07-29T01:08:50.613 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:08:50.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:08:50.614 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:08:50.670 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:08:50.670 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:08:50.719 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:08:50.719 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:08:50.719 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:08:50.719 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:08:50.720 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:08:50.720 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:08:50.720 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:50.720 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:08:50.720 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:50.720 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:08:50.721 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:50.721 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:50.721 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051393", 2023-07-29T01:08:50.721 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:50.721 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:08:50.721 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:50.721 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:04:52.529981Z" 2023-07-29T01:08:50.722 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:50.722 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:50.722 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:50.722 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:08:50.722 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:50.723 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:50.723 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044739", 2023-07-29T01:08:50.723 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:50.724 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:08:50.724 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:50.724 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:23.964578Z" 2023-07-29T01:08:50.724 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:50.724 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:50.724 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:50.724 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:08:50.725 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:50.725 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:50.725 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045520", 2023-07-29T01:08:50.725 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:50.725 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:08:50.725 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:50.726 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:06:23.586689Z" 2023-07-29T01:08:50.726 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:50.726 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:50.726 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:50.726 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:08:50.726 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:50.726 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:50.727 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:08:50.727 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:50.727 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:08:50.727 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:50.727 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:08:50.727 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:50.727 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:50.728 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:08:50.728 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:50.728 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:08:50.728 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:08:50.728 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:08:50.728 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:50.728 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:08:50Z" 2023-07-29T01:08:50.729 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:08:50.731 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051393", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:04:52.529981Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044739", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T00:58:23.964578Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045520", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:06:23.586689Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:08:50Z" } 2023-07-29T01:08:50.731 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051393', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:04:52.529981Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044739', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T00:58:23.964578Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045520', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:06:23.586689Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:08:50.732 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051673', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:08:50.732 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051393', 1: '00000000000000000000:00000000000000044739', 2: '00000000000000000000:00000000000000045520', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:08:50.733 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051673 target=00000000000000000000:00000000000000051393, shard=1 master=00000000000000000000:00000000000000044881 target=00000000000000000000:00000000000000044739, shard=2 master=00000000000000000000:00000000000000045660 target=00000000000000000000:00000000000000045520 2023-07-29T01:08:55.734 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:08:55.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:08:55.735 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:08:55.791 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:08:55.791 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:08:55.837 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:08:55.838 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:08:55.838 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:08:55.838 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:08:55.838 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:08:55.839 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:08:55.839 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:55.839 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:08:55.839 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:55.839 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:08:55.839 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:55.839 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:55.840 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051673", 2023-07-29T01:08:55.840 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:55.840 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:08:55.840 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:55.840 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:22.963361Z" 2023-07-29T01:08:55.840 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:55.841 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:55.841 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:55.841 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:08:55.841 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:55.841 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:55.841 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044739", 2023-07-29T01:08:55.841 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:55.842 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:08:55.842 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:55.842 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:23.964578Z" 2023-07-29T01:08:55.842 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:55.842 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:55.842 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:55.842 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:08:55.843 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:55.843 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:55.843 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045660", 2023-07-29T01:08:55.843 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:55.843 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:08:55.843 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:55.843 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:22.902962Z" 2023-07-29T01:08:55.844 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:55.844 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:55.844 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:08:55.844 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:08:55.844 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:08:55.844 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:08:55.845 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:08:55.845 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:08:55.845 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:08:55.845 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:08:55.845 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:08:55.845 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:55.845 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:08:55.846 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:08:55.846 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:55.846 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:08:55.846 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:08:55.846 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:08:55.846 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:08:55.846 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:08:55Z" 2023-07-29T01:08:55.847 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:08:55.847 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051673", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:07:22.963361Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044739", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T00:58:23.964578Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045660", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:22.902962Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:08:55Z" } 2023-07-29T01:08:55.848 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051673', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:07:22.963361Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044739', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T00:58:23.964578Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045660', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:22.902962Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:08:55.848 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051673', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:08:55.849 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051673', 1: '00000000000000000000:00000000000000044739', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:08:55.849 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000044881 target=00000000000000000000:00000000000000044739 2023-07-29T01:09:00.851 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:09:00.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:09:00.853 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:09:00.906 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:09:00.907 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:09:00.953 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:09:00.954 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:09:00.954 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:09:00.954 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:09:00.955 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:09:00.955 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:09:00.955 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:00.955 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:09:00.956 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:00.956 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:09:00.956 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:00.957 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:09:00.957 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051673", 2023-07-29T01:09:00.957 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:00.958 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:09:00.958 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:00.958 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:22.963361Z" 2023-07-29T01:09:00.959 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:00.959 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:00.959 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:00.960 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:09:00.960 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:00.960 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:09:00.960 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044739", 2023-07-29T01:09:00.961 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:00.961 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:09:00.961 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:00.962 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:23.964578Z" 2023-07-29T01:09:00.962 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:00.962 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:00.963 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:00.963 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:09:00.963 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:00.963 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:09:00.963 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045660", 2023-07-29T01:09:00.963 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:00.963 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:09:00.964 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:00.964 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:22.902962Z" 2023-07-29T01:09:00.964 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:00.964 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:00.964 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:00.965 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:09:00.965 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:00.965 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:09:00.965 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:09:00.965 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:00.965 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:09:00.965 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:00.966 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:09:00.966 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:00.966 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:00.966 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:09:00.966 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:00.966 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:09:00.966 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:09:00.967 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:09:00.967 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:00.967 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:09:00Z" 2023-07-29T01:09:00.967 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:09:00.968 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051673", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:07:22.963361Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044739", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T00:58:23.964578Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045660", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:22.902962Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:09:00Z" } 2023-07-29T01:09:00.969 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051673', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:07:22.963361Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044739', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T00:58:23.964578Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045660', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:22.902962Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:09:00.969 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051673', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:09:00.969 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051673', 1: '00000000000000000000:00000000000000044739', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:09:00.970 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000044881 target=00000000000000000000:00000000000000044739 2023-07-29T01:09:05.971 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:09:05.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:09:05.973 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:09:06.028 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:09:06.029 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:09:06.075 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:09:06.076 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:09:06.076 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:09:06.076 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:09:06.076 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:09:06.076 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:09:06.077 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:06.077 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:09:06.077 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:06.077 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:09:06.077 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:06.077 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:09:06.077 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051673", 2023-07-29T01:09:06.078 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:06.078 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:09:06.078 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:06.078 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:22.963361Z" 2023-07-29T01:09:06.078 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:06.078 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:06.078 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:06.079 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:09:06.079 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:06.079 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:09:06.079 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044739", 2023-07-29T01:09:06.079 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:06.079 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:09:06.079 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:06.080 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:23.964578Z" 2023-07-29T01:09:06.080 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:06.080 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:06.080 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:06.080 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:09:06.080 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:06.081 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:09:06.081 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045660", 2023-07-29T01:09:06.081 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:06.081 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:09:06.081 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:06.081 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:22.902962Z" 2023-07-29T01:09:06.081 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:06.082 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:06.082 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:06.082 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:09:06.082 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:06.082 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:09:06.082 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:09:06.082 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:06.083 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:09:06.083 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:06.083 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:09:06.083 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:06.083 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:06.083 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:09:06.083 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:06.084 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:09:06.084 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:09:06.084 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:09:06.084 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:06.084 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:09:05Z" 2023-07-29T01:09:06.084 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:09:06.086 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051673", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:07:22.963361Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044739", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T00:58:23.964578Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045660", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:22.902962Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:09:05Z" } 2023-07-29T01:09:06.086 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051673', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:07:22.963361Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044739', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T00:58:23.964578Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045660', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:22.902962Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:09:06.086 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051673', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:09:06.087 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051673', 1: '00000000000000000000:00000000000000044739', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:09:06.087 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000044881 target=00000000000000000000:00000000000000044739 2023-07-29T01:09:11.088 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:09:11.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:09:11.089 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:09:11.145 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:09:11.145 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:09:11.193 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:09:11.193 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:09:11.193 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:09:11.193 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:09:11.193 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:09:11.194 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:09:11.194 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:11.194 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:09:11.194 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:11.194 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:09:11.195 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:11.195 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:09:11.195 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051673", 2023-07-29T01:09:11.195 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:11.195 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:09:11.195 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:11.195 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:22.963361Z" 2023-07-29T01:09:11.196 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:11.196 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:11.196 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:11.196 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:09:11.196 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:11.196 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:09:11.197 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044739", 2023-07-29T01:09:11.197 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:11.197 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:09:11.197 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:11.197 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:23.964578Z" 2023-07-29T01:09:11.197 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:11.198 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:11.198 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:11.198 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:09:11.198 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:11.198 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:09:11.198 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045660", 2023-07-29T01:09:11.199 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:11.199 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:09:11.199 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:11.199 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:22.902962Z" 2023-07-29T01:09:11.199 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:11.199 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:11.200 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:11.200 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:09:11.200 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:11.200 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:09:11.200 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:09:11.200 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:11.200 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:09:11.201 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:11.201 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:09:11.201 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:11.201 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:11.201 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:09:11.201 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:11.202 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:09:11.202 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:09:11.202 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:09:11.202 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:11.202 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:09:11Z" 2023-07-29T01:09:11.202 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:09:11.203 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051673", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:07:22.963361Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044739", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T00:58:23.964578Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045660", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:22.902962Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:09:11Z" } 2023-07-29T01:09:11.204 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051673', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:07:22.963361Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044739', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T00:58:23.964578Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045660', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:22.902962Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:09:11.204 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051673', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:09:11.205 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051673', 1: '00000000000000000000:00000000000000044739', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:09:11.205 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000044881 target=00000000000000000000:00000000000000044739 2023-07-29T01:09:16.206 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:09:16.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:09:16.208 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:09:16.264 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:09:16.264 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:09:16.311 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:09:16.311 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:09:16.312 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:09:16.312 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:09:16.312 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:09:16.312 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:09:16.312 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:16.312 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:09:16.313 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:16.313 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:09:16.313 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:16.313 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:09:16.313 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051673", 2023-07-29T01:09:16.313 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:16.314 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:09:16.314 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:16.314 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:22.963361Z" 2023-07-29T01:09:16.314 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:16.314 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:16.314 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:16.315 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:09:16.315 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:16.315 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:09:16.315 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044739", 2023-07-29T01:09:16.315 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:16.315 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:09:16.316 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:16.316 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:23.964578Z" 2023-07-29T01:09:16.316 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:16.316 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:16.316 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:16.316 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:09:16.317 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:16.317 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:09:16.317 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045660", 2023-07-29T01:09:16.317 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:16.317 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:09:16.317 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:16.317 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:22.902962Z" 2023-07-29T01:09:16.318 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:16.318 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:16.318 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:16.318 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:09:16.318 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:16.318 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:09:16.319 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:09:16.319 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:16.319 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:09:16.319 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:16.319 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:09:16.319 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:16.319 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:16.320 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:09:16.320 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:16.320 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:09:16.320 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:09:16.320 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:09:16.320 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:16.321 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:09:16Z" 2023-07-29T01:09:16.321 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:09:16.322 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051673", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:07:22.963361Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044739", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T00:58:23.964578Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045660", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:22.902962Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:09:16Z" } 2023-07-29T01:09:16.322 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051673', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:07:22.963361Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044739', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T00:58:23.964578Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045660', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:22.902962Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:09:16.323 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051673', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:09:16.323 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051673', 1: '00000000000000000000:00000000000000044739', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:09:16.323 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000044881 target=00000000000000000000:00000000000000044739 2023-07-29T01:09:21.325 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:09:21.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:09:21.326 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:09:21.381 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:09:21.381 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:09:21.428 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:09:21.429 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:09:21.429 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:09:21.429 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:09:21.429 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:09:21.430 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:09:21.430 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:21.430 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:09:21.430 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:21.430 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:09:21.430 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:21.430 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:09:21.431 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051673", 2023-07-29T01:09:21.431 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:21.431 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:09:21.431 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:21.431 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:22.963361Z" 2023-07-29T01:09:21.431 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:21.431 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:21.432 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:21.432 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:09:21.432 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:21.432 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:09:21.432 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044739", 2023-07-29T01:09:21.432 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:21.433 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:09:21.433 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:21.433 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:23.964578Z" 2023-07-29T01:09:21.433 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:21.433 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:21.433 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:21.433 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:09:21.434 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:21.434 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:09:21.434 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045660", 2023-07-29T01:09:21.434 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:21.434 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:09:21.434 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:21.435 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:22.902962Z" 2023-07-29T01:09:21.435 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:21.435 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:21.435 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:21.435 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:09:21.435 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:21.435 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:09:21.436 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:09:21.436 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:21.436 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:09:21.436 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:21.436 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:09:21.436 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:21.437 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:21.437 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:09:21.437 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:21.437 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:09:21.437 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:09:21.437 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:09:21.437 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:21.438 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:09:21Z" 2023-07-29T01:09:21.438 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:09:21.439 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051673", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:07:22.963361Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044739", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T00:58:23.964578Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045660", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:22.902962Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:09:21Z" } 2023-07-29T01:09:21.439 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051673', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:07:22.963361Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044739', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T00:58:23.964578Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045660', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:22.902962Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:09:21.440 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051673', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:09:21.440 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051673', 1: '00000000000000000000:00000000000000044739', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:09:21.440 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000044881 target=00000000000000000000:00000000000000044739 2023-07-29T01:09:26.442 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:09:26.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:09:26.443 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:09:26.498 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:09:26.499 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:09:26.546 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:09:26.547 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:09:26.547 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:09:26.547 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:09:26.547 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:09:26.547 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:09:26.548 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:26.548 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:09:26.548 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:26.548 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:09:26.548 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:26.548 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:09:26.549 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051673", 2023-07-29T01:09:26.549 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:26.549 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:09:26.549 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:26.549 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:22.963361Z" 2023-07-29T01:09:26.549 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:26.549 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:26.550 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:26.550 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:09:26.550 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:26.550 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:09:26.550 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044881", 2023-07-29T01:09:26.550 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:26.551 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:09:26.551 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:26.551 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:23.001407Z" 2023-07-29T01:09:26.551 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:26.551 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:26.551 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:26.552 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:09:26.552 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:26.552 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:09:26.552 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045660", 2023-07-29T01:09:26.552 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:26.552 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:09:26.552 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:26.553 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:22.902962Z" 2023-07-29T01:09:26.553 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:26.553 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:26.553 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:26.553 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:09:26.553 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:26.554 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:09:26.554 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:09:26.554 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:26.554 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:09:26.554 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:26.554 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:09:26.555 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:26.555 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:26.555 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:09:26.555 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:26.555 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:09:26.555 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:09:26.555 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:09:26.556 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:26.556 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:09:26Z" 2023-07-29T01:09:26.556 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:09:26.557 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051673", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:07:22.963361Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044881", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:07:23.001407Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045660", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:22.902962Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:09:26Z" } 2023-07-29T01:09:26.557 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051673', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:07:22.963361Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044881', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:07:23.001407Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045660', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:22.902962Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:09:26.558 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051673', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:09:26.558 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051673', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:09:26.559 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-07-29T01:09:26.559 DEBUG:boto:path=/hiiqos-68/ 2023-07-29T01:09:26.559 DEBUG:boto:auth_path=/hiiqos-68/ 2023-07-29T01:09:26.560 DEBUG:boto:Method: HEAD 2023-07-29T01:09:26.560 DEBUG:boto:Path: /hiiqos-68/ 2023-07-29T01:09:26.560 DEBUG:boto:Data: 2023-07-29T01:09:26.561 DEBUG:boto:Headers: {} 2023-07-29T01:09:26.561 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:09:26.561 DEBUG:boto:Port: 8001 2023-07-29T01:09:26.562 DEBUG:boto:Params: {} 2023-07-29T01:09:26.562 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-29T01:09:26.563 DEBUG:boto:Token: None 2023-07-29T01:09:26.563 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:09:26 GMT /hiiqos-68/ 2023-07-29T01:09:26.564 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9gUxdF3D//b5XfGr3kg5dTjAyY8= 2023-07-29T01:09:26.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:09:26 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9gUxdF3D//b5XfGr3kg5dTjAyY8=', 'Content-Length': '0'} 2023-07-29T01:09:31.570 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e40346dc96390d41-0064c466cb-24b0-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:09:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:09:31.571 DEBUG:boto:path=/hiiqos-68/obj1 2023-07-29T01:09:31.571 DEBUG:boto:auth_path=/hiiqos-68/obj1 2023-07-29T01:09:31.572 DEBUG:boto:Method: HEAD 2023-07-29T01:09:31.572 DEBUG:boto:Path: /hiiqos-68/obj1 2023-07-29T01:09:31.573 DEBUG:boto:Data: 2023-07-29T01:09:31.573 DEBUG:boto:Headers: {} 2023-07-29T01:09:31.574 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:09:31.574 DEBUG:boto:Port: 8001 2023-07-29T01:09:31.574 DEBUG:boto:Params: {} 2023-07-29T01:09:31.575 DEBUG:boto:Token: None 2023-07-29T01:09:31.575 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:09:31 GMT /hiiqos-68/obj1 2023-07-29T01:09:31.576 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:c93I6a4qr6rgAIC1TgQIlGz9NO0= 2023-07-29T01:09:31.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:09:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c93I6a4qr6rgAIC1TgQIlGz9NO0=', 'Content-Length': '0'} 2023-07-29T01:09:31.579 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:07:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-68:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.9034.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a58151fb047e51c-0064c466cb-24b0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:09:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:09:31.580 DEBUG:boto:path=/hiiqos-68/obj1 2023-07-29T01:09:31.581 DEBUG:boto:auth_path=/hiiqos-68/obj1 2023-07-29T01:09:31.581 DEBUG:boto:Method: GET 2023-07-29T01:09:31.581 DEBUG:boto:Path: /hiiqos-68/obj1 2023-07-29T01:09:31.582 DEBUG:boto:Data: 2023-07-29T01:09:31.582 DEBUG:boto:Headers: {} 2023-07-29T01:09:31.583 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:09:31.584 DEBUG:boto:Port: 8001 2023-07-29T01:09:31.584 DEBUG:boto:Params: {} 2023-07-29T01:09:31.585 DEBUG:boto:Token: None 2023-07-29T01:09:31.586 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 01:09:31 GMT /hiiqos-68/obj1 2023-07-29T01:09:31.587 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FxEG8bdeMna5sL97aGuLLR39aWk= 2023-07-29T01:09:31.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:09:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FxEG8bdeMna5sL97aGuLLR39aWk=', 'Content-Length': '0'} 2023-07-29T01:09:31.591 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:07:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-68:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.9034.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000084b2cc33da2fb50a-0064c466cb-24b0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:09:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:09:31.592 DEBUG:boto:path=/hiiqos-68/obj2 2023-07-29T01:09:31.592 DEBUG:boto:auth_path=/hiiqos-68/obj2 2023-07-29T01:09:31.592 DEBUG:boto:Method: HEAD 2023-07-29T01:09:31.593 DEBUG:boto:Path: /hiiqos-68/obj2 2023-07-29T01:09:31.593 DEBUG:boto:Data: 2023-07-29T01:09:31.593 DEBUG:boto:Headers: {} 2023-07-29T01:09:31.594 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:09:31.594 DEBUG:boto:Port: 8001 2023-07-29T01:09:31.594 DEBUG:boto:Params: {} 2023-07-29T01:09:31.595 DEBUG:boto:Token: None 2023-07-29T01:09:31.595 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:09:31 GMT /hiiqos-68/obj2 2023-07-29T01:09:31.596 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qemWRBWunoZHuh0UYtVY1Y77M0E= 2023-07-29T01:09:31.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:09:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qemWRBWunoZHuh0UYtVY1Y77M0E=', 'Content-Length': '0'} 2023-07-29T01:09:31.598 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:07:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-68:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.9034.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000005a108ea881cdab5-0064c466cb-24b0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:09:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:09:31.599 DEBUG:boto:path=/hiiqos-68/obj2 2023-07-29T01:09:31.599 DEBUG:boto:auth_path=/hiiqos-68/obj2 2023-07-29T01:09:31.600 DEBUG:boto:Method: GET 2023-07-29T01:09:31.600 DEBUG:boto:Path: /hiiqos-68/obj2 2023-07-29T01:09:31.601 DEBUG:boto:Data: 2023-07-29T01:09:31.601 DEBUG:boto:Headers: {} 2023-07-29T01:09:31.602 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:09:31.602 DEBUG:boto:Port: 8001 2023-07-29T01:09:31.602 DEBUG:boto:Params: {} 2023-07-29T01:09:31.603 DEBUG:boto:Token: None 2023-07-29T01:09:31.603 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 01:09:31 GMT /hiiqos-68/obj2 2023-07-29T01:09:31.604 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Dz2b+rg3jo+ssTMCxjxU6v9j4bI= 2023-07-29T01:09:31.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:09:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Dz2b+rg3jo+ssTMCxjxU6v9j4bI=', 'Content-Length': '0'} 2023-07-29T01:09:31.606 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:07:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-68:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.9034.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c137d9646e665995-0064c466cb-24b0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:09:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:09:31.607 DEBUG:boto:path=/hiiqos-68/obj3 2023-07-29T01:09:31.607 DEBUG:boto:auth_path=/hiiqos-68/obj3 2023-07-29T01:09:31.608 DEBUG:boto:Method: HEAD 2023-07-29T01:09:31.608 DEBUG:boto:Path: /hiiqos-68/obj3 2023-07-29T01:09:31.608 DEBUG:boto:Data: 2023-07-29T01:09:31.609 DEBUG:boto:Headers: {} 2023-07-29T01:09:31.609 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:09:31.610 DEBUG:boto:Port: 8001 2023-07-29T01:09:31.610 DEBUG:boto:Params: {} 2023-07-29T01:09:31.610 DEBUG:boto:Token: None 2023-07-29T01:09:31.611 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:09:31 GMT /hiiqos-68/obj3 2023-07-29T01:09:31.611 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bdDiCsuauzkrM2+M6yVacQVWA3M= 2023-07-29T01:09:31.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:09:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bdDiCsuauzkrM2+M6yVacQVWA3M=', 'Content-Length': '0'} 2023-07-29T01:09:31.614 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:07:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-68:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.9034.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dbc72ddbf40e4784-0064c466cb-24b0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:09:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:09:31.614 DEBUG:boto:path=/hiiqos-68/obj3 2023-07-29T01:09:31.615 DEBUG:boto:auth_path=/hiiqos-68/obj3 2023-07-29T01:09:31.615 DEBUG:boto:Method: GET 2023-07-29T01:09:31.615 DEBUG:boto:Path: /hiiqos-68/obj3 2023-07-29T01:09:31.616 DEBUG:boto:Data: 2023-07-29T01:09:31.616 DEBUG:boto:Headers: {} 2023-07-29T01:09:31.617 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:09:31.617 DEBUG:boto:Port: 8001 2023-07-29T01:09:31.617 DEBUG:boto:Params: {} 2023-07-29T01:09:31.618 DEBUG:boto:Token: None 2023-07-29T01:09:31.618 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 01:09:31 GMT /hiiqos-68/obj3 2023-07-29T01:09:31.619 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7d5hUG7+amftXBc2r/c31pXyEkk= 2023-07-29T01:09:31.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:09:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7d5hUG7+amftXBc2r/c31pXyEkk=', 'Content-Length': '0'} 2023-07-29T01:09:31.622 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:07:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-68:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.9034.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000009a733b5d9cb6029-0064c466cb-24b0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:09:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:09:31.623 DEBUG:boto:path=/hiiqos-69/ 2023-07-29T01:09:31.624 DEBUG:boto:auth_path=/hiiqos-69/ 2023-07-29T01:09:31.625 DEBUG:boto:Method: HEAD 2023-07-29T01:09:31.626 DEBUG:boto:Path: /hiiqos-69/ 2023-07-29T01:09:31.627 DEBUG:boto:Data: 2023-07-29T01:09:31.627 DEBUG:boto:Headers: {} 2023-07-29T01:09:31.628 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:09:31.629 DEBUG:boto:Port: 8001 2023-07-29T01:09:31.629 DEBUG:boto:Params: {} 2023-07-29T01:09:31.630 DEBUG:boto:Token: None 2023-07-29T01:09:31.631 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:09:31 GMT /hiiqos-69/ 2023-07-29T01:09:31.631 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xWN+Gjy/iWNpz6TDXdSk5/c+Ou8= 2023-07-29T01:09:31.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:09:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xWN+Gjy/iWNpz6TDXdSk5/c+Ou8=', 'Content-Length': '0'} 2023-07-29T01:09:31.636 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000133ec3032def3d41-0064c466cb-24b0-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:09:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:09:31.637 DEBUG:boto:path=/hiiqos-69/obj1 2023-07-29T01:09:31.637 DEBUG:boto:auth_path=/hiiqos-69/obj1 2023-07-29T01:09:31.638 DEBUG:boto:Method: HEAD 2023-07-29T01:09:31.638 DEBUG:boto:Path: /hiiqos-69/obj1 2023-07-29T01:09:31.638 DEBUG:boto:Data: 2023-07-29T01:09:31.639 DEBUG:boto:Headers: {} 2023-07-29T01:09:31.639 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:09:31.640 DEBUG:boto:Port: 8001 2023-07-29T01:09:31.640 DEBUG:boto:Params: {} 2023-07-29T01:09:31.640 DEBUG:boto:Token: None 2023-07-29T01:09:31.641 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:09:31 GMT /hiiqos-69/obj1 2023-07-29T01:09:31.641 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:18psLUzcgX00W2IiiwSAlGbbHgM= 2023-07-29T01:09:31.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:09:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:18psLUzcgX00W2IiiwSAlGbbHgM=', 'Content-Length': '0'} 2023-07-29T01:09:31.643 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000285dc89128bed99d-0064c466cb-24b0-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 01:09:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:09:31.644 DEBUG:boto:path=/hiiqos-69/obj2 2023-07-29T01:09:31.645 DEBUG:boto:auth_path=/hiiqos-69/obj2 2023-07-29T01:09:31.646 DEBUG:boto:Method: HEAD 2023-07-29T01:09:31.646 DEBUG:boto:Path: /hiiqos-69/obj2 2023-07-29T01:09:31.647 DEBUG:boto:Data: 2023-07-29T01:09:31.647 DEBUG:boto:Headers: {} 2023-07-29T01:09:31.648 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:09:31.649 DEBUG:boto:Port: 8001 2023-07-29T01:09:31.649 DEBUG:boto:Params: {} 2023-07-29T01:09:31.650 DEBUG:boto:Token: None 2023-07-29T01:09:31.651 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:09:31 GMT /hiiqos-69/obj2 2023-07-29T01:09:31.651 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:310YTzqlcqBZuc9DDSjjjMTdRi8= 2023-07-29T01:09:31.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:09:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:310YTzqlcqBZuc9DDSjjjMTdRi8=', 'Content-Length': '0'} 2023-07-29T01:09:31.653 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000de3791d72bb4e338-0064c466cb-24b0-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 01:09:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:09:31.654 DEBUG:boto:path=/hiiqos-69/obj3 2023-07-29T01:09:31.654 DEBUG:boto:auth_path=/hiiqos-69/obj3 2023-07-29T01:09:31.655 DEBUG:boto:Method: HEAD 2023-07-29T01:09:31.655 DEBUG:boto:Path: /hiiqos-69/obj3 2023-07-29T01:09:31.656 DEBUG:boto:Data: 2023-07-29T01:09:31.656 DEBUG:boto:Headers: {} 2023-07-29T01:09:31.657 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:09:31.657 DEBUG:boto:Port: 8001 2023-07-29T01:09:31.657 DEBUG:boto:Params: {} 2023-07-29T01:09:31.658 DEBUG:boto:Token: None 2023-07-29T01:09:31.659 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:09:31 GMT /hiiqos-69/obj3 2023-07-29T01:09:31.659 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kf66aS7B+awGD32scJWdDxE1CF4= 2023-07-29T01:09:31.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:09:31 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kf66aS7B+awGD32scJWdDxE1CF4=', 'Content-Length': '0'} 2023-07-29T01:09:31.662 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000c8013156ec9b25ca-0064c466cb-24b0-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 01:09:31 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:09:31.663 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:09:31.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:09:31.665 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:09:31.720 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:09:31.720 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:09:31.770 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2023-07-29T01:09:31.770 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:09:31.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:09:31.771 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:09:31.821 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:09:31.822 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:09:31.866 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-07-29T01:09:31.867 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:09:31.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:09:31.867 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:09:31.913 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:09:31.914 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:09:31.958 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-07-29T01:09:31.959 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:09:31.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:09:31.960 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:09:32.004 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:09:32.005 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:09:32.054 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:09:32.055 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:09:32.055 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 17, 2023-07-29T01:09:32.055 INFO:teuthology.orchestra.run.smithi101.stdout: "predecessor_uuid": "a0987484-ce37-4520-a564-54b7ea8c5251", 2023-07-29T01:09:32.055 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": [], 2023-07-29T01:09:32.055 INFO:teuthology.orchestra.run.smithi101.stdout: "period_map": { 2023-07-29T01:09:32.055 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:09:32.056 INFO:teuthology.orchestra.run.smithi101.stdout: "zonegroups": [ 2023-07-29T01:09:32.056 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:32.056 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "a", 2023-07-29T01:09:32.056 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a", 2023-07-29T01:09:32.056 INFO:teuthology.orchestra.run.smithi101.stdout: "api_name": "", 2023-07-29T01:09:32.056 INFO:teuthology.orchestra.run.smithi101.stdout: "is_master": true, 2023-07-29T01:09:32.057 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:09:32.057 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-29T01:09:32.057 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:09:32.057 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames": [], 2023-07-29T01:09:32.057 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames_s3website": [], 2023-07-29T01:09:32.057 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:09:32.058 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:09:32.058 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:32.058 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:09:32.058 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a1", 2023-07-29T01:09:32.058 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:09:32.058 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-29T01:09:32.059 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:09:32.059 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T01:09:32.059 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T01:09:32.059 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T01:09:32.059 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T01:09:32.059 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T01:09:32.059 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T01:09:32.060 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T01:09:32.060 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T01:09:32.060 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T01:09:32.060 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T01:09:32.060 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:09:32.060 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:09:32.061 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:32.061 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:32.061 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-29T01:09:32.061 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a2", 2023-07-29T01:09:32.061 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:09:32.061 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8001" 2023-07-29T01:09:32.061 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:09:32.062 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T01:09:32.062 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T01:09:32.062 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T01:09:32.062 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T01:09:32.062 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T01:09:32.062 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T01:09:32.063 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T01:09:32.063 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T01:09:32.063 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T01:09:32.063 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T01:09:32.063 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:09:32.063 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:09:32.064 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:32.064 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:09:32.064 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_targets": [ 2023-07-29T01:09:32.064 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:32.064 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default-placement", 2023-07-29T01:09:32.064 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [], 2023-07-29T01:09:32.065 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": [ 2023-07-29T01:09:32.065 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD" 2023-07-29T01:09:32.065 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:09:32.065 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:32.065 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:09:32.066 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "default-placement", 2023-07-29T01:09:32.066 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T01:09:32.066 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_policy": { 2023-07-29T01:09:32.066 INFO:teuthology.orchestra.run.smithi101.stdout: "groups": [ 2023-07-29T01:09:32.066 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:32.066 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "sync-group", 2023-07-29T01:09:32.067 INFO:teuthology.orchestra.run.smithi101.stdout: "data_flow": { 2023-07-29T01:09:32.067 INFO:teuthology.orchestra.run.smithi101.stdout: "symmetrical": [ 2023-07-29T01:09:32.067 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:32.067 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "sync-flow1", 2023-07-29T01:09:32.067 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:09:32.067 INFO:teuthology.orchestra.run.smithi101.stdout: "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:09:32.068 INFO:teuthology.orchestra.run.smithi101.stdout: "59d1bb30-3b1c-476f-9111-052c1ea0b711" 2023-07-29T01:09:32.068 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:09:32.068 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:32.068 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:09:32.068 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:32.068 INFO:teuthology.orchestra.run.smithi101.stdout: "pipes": [ 2023-07-29T01:09:32.069 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:32.069 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "sync-pipe", 2023-07-29T01:09:32.069 INFO:teuthology.orchestra.run.smithi101.stdout: "source": { 2023-07-29T01:09:32.069 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket": "*", 2023-07-29T01:09:32.069 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:09:32.069 INFO:teuthology.orchestra.run.smithi101.stdout: "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:09:32.069 INFO:teuthology.orchestra.run.smithi101.stdout: "59d1bb30-3b1c-476f-9111-052c1ea0b711" 2023-07-29T01:09:32.070 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:09:32.070 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:32.070 INFO:teuthology.orchestra.run.smithi101.stdout: "dest": { 2023-07-29T01:09:32.070 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket": "*", 2023-07-29T01:09:32.070 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:09:32.070 INFO:teuthology.orchestra.run.smithi101.stdout: "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:09:32.071 INFO:teuthology.orchestra.run.smithi101.stdout: "59d1bb30-3b1c-476f-9111-052c1ea0b711" 2023-07-29T01:09:32.071 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:09:32.071 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:32.071 INFO:teuthology.orchestra.run.smithi101.stdout: "params": { 2023-07-29T01:09:32.071 INFO:teuthology.orchestra.run.smithi101.stdout: "source": { 2023-07-29T01:09:32.071 INFO:teuthology.orchestra.run.smithi101.stdout: "filter": { 2023-07-29T01:09:32.071 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [] 2023-07-29T01:09:32.072 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:32.072 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:32.072 INFO:teuthology.orchestra.run.smithi101.stdout: "dest": {}, 2023-07-29T01:09:32.072 INFO:teuthology.orchestra.run.smithi101.stdout: "priority": 0, 2023-07-29T01:09:32.072 INFO:teuthology.orchestra.run.smithi101.stdout: "mode": "system", 2023-07-29T01:09:32.072 INFO:teuthology.orchestra.run.smithi101.stdout: "user": "" 2023-07-29T01:09:32.073 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:32.073 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:32.073 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:09:32.073 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "allowed" 2023-07-29T01:09:32.073 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:32.073 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:09:32.074 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:32.074 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled_features": [ 2023-07-29T01:09:32.074 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:09:32.074 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:09:32.074 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:32.074 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:32.075 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "b", 2023-07-29T01:09:32.075 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b", 2023-07-29T01:09:32.075 INFO:teuthology.orchestra.run.smithi101.stdout: "api_name": "", 2023-07-29T01:09:32.075 INFO:teuthology.orchestra.run.smithi101.stdout: "is_master": false, 2023-07-29T01:09:32.075 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:09:32.075 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-29T01:09:32.076 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:09:32.076 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames": [], 2023-07-29T01:09:32.076 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames_s3website": [], 2023-07-29T01:09:32.076 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T01:09:32.076 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:09:32.076 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:32.076 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T01:09:32.077 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b1", 2023-07-29T01:09:32.077 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:09:32.077 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-29T01:09:32.077 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:09:32.077 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T01:09:32.077 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T01:09:32.078 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T01:09:32.078 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T01:09:32.078 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T01:09:32.078 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T01:09:32.078 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T01:09:32.078 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T01:09:32.079 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T01:09:32.079 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T01:09:32.079 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:09:32.079 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:09:32.079 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:32.079 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:32.079 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-29T01:09:32.080 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b2", 2023-07-29T01:09:32.080 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:09:32.080 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8001" 2023-07-29T01:09:32.080 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:09:32.080 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T01:09:32.080 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T01:09:32.081 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T01:09:32.081 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T01:09:32.081 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T01:09:32.081 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T01:09:32.081 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T01:09:32.081 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T01:09:32.082 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T01:09:32.082 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T01:09:32.082 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:09:32.082 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:09:32.082 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:32.082 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:09:32.082 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_targets": [ 2023-07-29T01:09:32.083 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:32.083 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default-placement", 2023-07-29T01:09:32.083 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [], 2023-07-29T01:09:32.083 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": [ 2023-07-29T01:09:32.083 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD" 2023-07-29T01:09:32.083 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:09:32.084 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:32.084 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:09:32.084 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "default-placement", 2023-07-29T01:09:32.084 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T01:09:32.084 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_policy": { 2023-07-29T01:09:32.084 INFO:teuthology.orchestra.run.smithi101.stdout: "groups": [] 2023-07-29T01:09:32.085 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:32.085 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled_features": [ 2023-07-29T01:09:32.085 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:09:32.085 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:09:32.085 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:32.085 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:09:32.085 INFO:teuthology.orchestra.run.smithi101.stdout: "short_zone_ids": [ 2023-07-29T01:09:32.086 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:32.086 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:09:32.086 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 2896287231 2023-07-29T01:09:32.086 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:32.086 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:32.086 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-29T01:09:32.087 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 815606810 2023-07-29T01:09:32.088 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:32.088 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:32.088 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T01:09:32.088 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 2009171363 2023-07-29T01:09:32.088 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:32.089 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:32.089 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-29T01:09:32.089 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 3105190295 2023-07-29T01:09:32.089 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:32.089 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:09:32.090 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:32.090 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zonegroup": "a", 2023-07-29T01:09:32.090 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:09:32.090 INFO:teuthology.orchestra.run.smithi101.stdout: "period_config": { 2023-07-29T01:09:32.090 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_quota": { 2023-07-29T01:09:32.090 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-29T01:09:32.091 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-29T01:09:32.091 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-29T01:09:32.091 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-29T01:09:32.091 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-29T01:09:32.091 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:32.091 INFO:teuthology.orchestra.run.smithi101.stdout: "user_quota": { 2023-07-29T01:09:32.092 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-29T01:09:32.092 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-29T01:09:32.092 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-29T01:09:32.092 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-29T01:09:32.092 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-29T01:09:32.092 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:32.093 INFO:teuthology.orchestra.run.smithi101.stdout: "user_ratelimit": { 2023-07-29T01:09:32.093 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-29T01:09:32.093 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-29T01:09:32.093 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-29T01:09:32.093 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-29T01:09:32.093 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-29T01:09:32.093 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:32.094 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_ratelimit": { 2023-07-29T01:09:32.094 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-29T01:09:32.094 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-29T01:09:32.094 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-29T01:09:32.094 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-29T01:09:32.094 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-29T01:09:32.095 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:32.095 INFO:teuthology.orchestra.run.smithi101.stdout: "anonymous_ratelimit": { 2023-07-29T01:09:32.095 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-29T01:09:32.095 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-29T01:09:32.096 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-29T01:09:32.096 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-29T01:09:32.096 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-29T01:09:32.096 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:32.096 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:32.096 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T01:09:32.097 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_name": "test-realm", 2023-07-29T01:09:32.097 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:09:32.097 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:09:32.097 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:09:32.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:09:32.098 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:09:32.141 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:09:32.141 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:09:32.186 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-07-29T01:09:32.187 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'hiiqos-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:09:32.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'hiiqos-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:09:32.188 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket hiiqos-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:09:32.300 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:09:32.301 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:09:32.411 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-07-29T01:09:32.412 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'hiiqos-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:09:32.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'hiiqos-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:09:32.413 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket hiiqos-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:09:32.520 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:09:32.520 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:09:32.627 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2023-07-29T01:09:32.628 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=hiiqos-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:09:32.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=hiiqos-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:09:32.629 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=hiiqos-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:09:32.695 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:09:32.695 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:09:32.762 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-07-29T01:09:32.762 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'hiiqos-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:09:32.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'hiiqos-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:09:32.764 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket hiiqos-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:09:32.818 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:09:32.818 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:09:32.871 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-07-29T01:09:32.871 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'hiiqos-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:09:32.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'hiiqos-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:09:32.872 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket hiiqos-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:09:32.917 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:09:32.918 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:09:32.962 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-07-29T01:09:32.963 DEBUG:boto:path=/hiiqos-68/ 2023-07-29T01:09:32.963 DEBUG:boto:auth_path=/hiiqos-68/ 2023-07-29T01:09:32.964 DEBUG:boto:Method: HEAD 2023-07-29T01:09:32.964 DEBUG:boto:Path: /hiiqos-68/ 2023-07-29T01:09:32.965 DEBUG:boto:Data: 2023-07-29T01:09:32.965 DEBUG:boto:Headers: {} 2023-07-29T01:09:32.965 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:09:32.966 DEBUG:boto:Port: 8000 2023-07-29T01:09:32.966 DEBUG:boto:Params: {} 2023-07-29T01:09:32.967 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-29T01:09:32.967 DEBUG:boto:Token: None 2023-07-29T01:09:32.967 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:09:32 GMT /hiiqos-68/ 2023-07-29T01:09:32.968 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IrO9FRMaMNgB7Dn5+KhUDYKEwZA= 2023-07-29T01:09:32.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:09:32 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IrO9FRMaMNgB7Dn5+KhUDYKEwZA=', 'Content-Length': '0'} 2023-07-29T01:09:37.974 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001ec3028caead44f6-0064c466d1-25bc-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:09:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:09:37.975 DEBUG:boto:path=/hiiqos-68/obj3 2023-07-29T01:09:37.976 DEBUG:boto:auth_path=/hiiqos-68/obj3 2023-07-29T01:09:37.976 DEBUG:boto:Method: PUT 2023-07-29T01:09:37.976 DEBUG:boto:Path: /hiiqos-68/obj3 2023-07-29T01:09:37.977 DEBUG:boto:Data: 2023-07-29T01:09:37.977 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T01:09:37.977 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:09:37.978 DEBUG:boto:Port: 8000 2023-07-29T01:09:37.978 DEBUG:boto:Params: {} 2023-07-29T01:09:37.978 DEBUG:boto:Token: None 2023-07-29T01:09:37.979 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 01:09:37 GMT /hiiqos-68/obj3 2023-07-29T01:09:37.979 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:aqNTtLNGVWlM/q0HS9Y2JV4qX48= 2023-07-29T01:09:37.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 01:09:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aqNTtLNGVWlM/q0HS9Y2JV4qX48='} 2023-07-29T01:09:37.991 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fd054353d4fd47f3-0064c466d1-25bc-a1'), ('Date', 'Sat, 29 Jul 2023 01:09:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:09:37.992 DEBUG:boto:path=/hiiqos-69/ 2023-07-29T01:09:37.992 DEBUG:boto:auth_path=/hiiqos-69/ 2023-07-29T01:09:37.992 DEBUG:boto:Method: HEAD 2023-07-29T01:09:37.993 DEBUG:boto:Path: /hiiqos-69/ 2023-07-29T01:09:37.993 DEBUG:boto:Data: 2023-07-29T01:09:37.993 DEBUG:boto:Headers: {} 2023-07-29T01:09:37.994 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:09:37.994 DEBUG:boto:Port: 8000 2023-07-29T01:09:37.994 DEBUG:boto:Params: {} 2023-07-29T01:09:37.995 DEBUG:boto:Token: None 2023-07-29T01:09:37.995 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:09:37 GMT /hiiqos-69/ 2023-07-29T01:09:37.996 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EhN6Kxqus8bx3avAMbcQEpCyVf0= 2023-07-29T01:09:37.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:09:37 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EhN6Kxqus8bx3avAMbcQEpCyVf0=', 'Content-Length': '0'} 2023-07-29T01:09:38.000 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000097291753b8d320e0-0064c466d1-25bc-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:09:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:09:38.000 DEBUG:boto:path=/hiiqos-69/obj3 2023-07-29T01:09:38.001 DEBUG:boto:auth_path=/hiiqos-69/obj3 2023-07-29T01:09:38.001 DEBUG:boto:Method: PUT 2023-07-29T01:09:38.001 DEBUG:boto:Path: /hiiqos-69/obj3 2023-07-29T01:09:38.002 DEBUG:boto:Data: 2023-07-29T01:09:38.002 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T01:09:38.002 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:09:38.003 DEBUG:boto:Port: 8000 2023-07-29T01:09:38.003 DEBUG:boto:Params: {} 2023-07-29T01:09:38.003 DEBUG:boto:Token: None 2023-07-29T01:09:38.004 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 01:09:38 GMT /hiiqos-69/obj3 2023-07-29T01:09:38.004 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hPWqHwwH01kEbX5kcsh58BY5F6A= 2023-07-29T01:09:38.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 01:09:38 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hPWqHwwH01kEbX5kcsh58BY5F6A='} 2023-07-29T01:09:38.012 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003ce785a914532144-0064c466d1-25bc-a1'), ('Date', 'Sat, 29 Jul 2023 01:09:37 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:09:38.012 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:09:38.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:09:38.013 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:09:38.064 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:09:38.064 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:09:38.097 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T01:09:38.111 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T01:09:38.111 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:38.111 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:09:38.112 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:17.836402Z" 2023-07-29T01:09:38.112 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:38.112 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:38.112 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592972.771303_1597.1", 2023-07-29T01:09:38.113 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:09:32.771303Z" 2023-07-29T01:09:38.113 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:38.113 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:38.113 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:09:38.114 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T01:09:38.114 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:38.114 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:38.114 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:09:38.114 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:12.811406Z" 2023-07-29T01:09:38.114 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:38.114 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T01:09:38.115 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690591217.836402_294.1', 1: '1_1690592972.771303_1597.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:09:38.116 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T01:09:38.116 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:09:38.117 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:09:38.117 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:09:38.163 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:09:38.163 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:09:38.192 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:09:38.104+0000 7fbb57f29880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:09:38.192 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:09:38.105+0000 7fbb57f29880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:09:38.205 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:09:38.205 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:09:38.206 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:09:38.206 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:09:38.206 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:09:38.207 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:09:38.207 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:09:38.207 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:38.208 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:09:38.208 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:38.208 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:09:38.208 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:38.209 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:09:38.209 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:09:38.209 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:38.210 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:09:38.210 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:38.210 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:09:38.211 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:09:38.211 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:38.211 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:38.212 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:38.212 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:09:38.212 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:38.213 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:09:38.213 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592836.654753_1494.1", 2023-07-29T01:09:38.213 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:38.213 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:09:38.213 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:38.214 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:16.654753Z", 2023-07-29T01:09:38.214 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:09:38.214 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:38.215 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:38.215 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:38.215 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:09:38.216 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:38.216 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:09:38.216 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:09:38.216 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:38.216 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:09:38.217 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:38.217 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:09:38.217 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:09:38.217 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:38.218 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:38.218 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:38.218 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:09:38.219 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:38.219 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:09:38.219 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:09:38.219 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:38.220 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:09:38.220 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:38.220 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:09:38.221 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:09:38.221 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:38.222 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:38.222 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:09:38.222 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:38.222 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:09:38.222 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:09:38.222 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:09:38.222 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:38.223 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:09:38Z" 2023-07-29T01:09:38.223 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:09:38.225 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592836.654753_1494.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:16.654753Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:09:38Z" } 2023-07-29T01:09:38.226 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592836.654753_1494.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:16.654753Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:09:38.227 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592972.771303_1597.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:09:38.227 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592836.654753_1494.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:09:38.228 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592972.771303_1597.1 target=1_1690592836.654753_1494.1 2023-07-29T01:09:43.230 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:09:43.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:09:43.231 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:09:43.284 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:09:43.285 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:09:43.317 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:09:43.230+0000 7f67ede24880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:09:43.318 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:09:43.231+0000 7f67ede24880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:09:43.332 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:09:43.332 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:09:43.332 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:09:43.333 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:09:43.333 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:09:43.333 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:09:43.333 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:09:43.333 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:43.333 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:09:43.334 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:43.334 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:09:43.334 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:43.334 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:09:43.334 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:09:43.334 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:43.334 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:09:43.335 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:43.335 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:09:43.335 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:09:43.335 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:43.335 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:43.335 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:43.336 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:09:43.336 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:43.336 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:09:43.336 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592836.654753_1494.1", 2023-07-29T01:09:43.336 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:43.336 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:09:43.337 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:43.337 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:16.654753Z", 2023-07-29T01:09:43.337 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:09:43.337 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:43.337 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:43.337 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:43.337 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:09:43.338 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:43.338 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:09:43.338 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:09:43.338 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:43.338 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:09:43.338 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:43.338 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:09:43.339 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:09:43.339 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:43.339 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:43.339 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:43.339 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:09:43.339 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:43.340 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:09:43.340 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:09:43.340 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:43.340 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:09:43.340 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:43.340 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:09:43.340 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:09:43.341 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:43.341 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:43.341 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:09:43.341 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:43.341 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:09:43.341 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:09:43.342 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:09:43.342 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:43.342 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:09:43Z" 2023-07-29T01:09:43.342 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:09:43.343 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592836.654753_1494.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:16.654753Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:09:43Z" } 2023-07-29T01:09:43.343 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592836.654753_1494.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:16.654753Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:09:43.344 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592972.771303_1597.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:09:43.344 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592836.654753_1494.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:09:43.344 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592972.771303_1597.1 target=1_1690592836.654753_1494.1 2023-07-29T01:09:48.346 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:09:48.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:09:48.347 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:09:48.400 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:09:48.400 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:09:48.435 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:09:48.347+0000 7f313372e880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:09:48.436 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:09:48.348+0000 7f313372e880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:09:48.449 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:09:48.449 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:09:48.449 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:09:48.450 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:09:48.450 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:09:48.450 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:09:48.450 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:09:48.450 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:48.450 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:09:48.451 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:48.451 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:09:48.451 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:48.451 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:09:48.451 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:09:48.451 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:48.451 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:09:48.452 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:48.452 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:09:48.452 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:09:48.452 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:48.452 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:48.452 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:48.453 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:09:48.453 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:48.453 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:09:48.453 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592836.654753_1494.1", 2023-07-29T01:09:48.453 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:48.454 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:09:48.454 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:48.454 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:16.654753Z", 2023-07-29T01:09:48.454 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:09:48.454 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:48.454 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:48.454 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:48.455 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:09:48.455 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:48.455 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:09:48.456 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:09:48.456 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:48.456 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:09:48.456 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:48.456 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:09:48.456 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:09:48.457 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:48.457 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:48.457 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:48.457 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:09:48.457 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:48.457 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:09:48.458 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:09:48.458 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:48.458 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:09:48.458 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:48.458 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:09:48.458 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:09:48.458 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:48.459 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:48.459 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:09:48.459 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:48.459 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:09:48.459 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:09:48.459 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:09:48.459 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:48.460 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:09:48Z" 2023-07-29T01:09:48.460 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:09:48.461 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592836.654753_1494.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:16.654753Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:09:48Z" } 2023-07-29T01:09:48.461 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592836.654753_1494.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:16.654753Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:09:48.462 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592972.771303_1597.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:09:48.462 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592836.654753_1494.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:09:48.462 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592972.771303_1597.1 target=1_1690592836.654753_1494.1 2023-07-29T01:09:53.463 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:09:53.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:09:53.465 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:09:53.518 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:09:53.519 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:09:53.551 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:09:53.463+0000 7f4b5fe20880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:09:53.551 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:09:53.464+0000 7f4b5fe20880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:09:53.565 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:09:53.566 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:09:53.566 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:09:53.566 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:09:53.566 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:09:53.566 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:09:53.566 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:09:53.567 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:53.567 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:09:53.567 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:53.567 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:09:53.567 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:53.568 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:09:53.568 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:09:53.568 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:53.568 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:09:53.568 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:53.568 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:09:53.569 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:09:53.569 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:53.569 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:53.569 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:53.569 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:09:53.570 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:53.570 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:09:53.570 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592836.654753_1494.1", 2023-07-29T01:09:53.570 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:53.570 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:09:53.570 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:53.570 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:16.654753Z", 2023-07-29T01:09:53.571 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:09:53.571 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:53.571 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:53.571 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:53.571 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:09:53.571 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:53.571 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:09:53.572 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:09:53.572 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:53.572 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:09:53.572 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:53.572 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:09:53.572 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:09:53.573 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:53.573 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:53.573 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:53.573 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:09:53.573 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:53.573 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:09:53.573 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:09:53.574 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:53.574 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:09:53.574 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:53.574 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:09:53.575 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:09:53.575 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:53.575 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:53.575 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:09:53.576 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:53.576 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:09:53.576 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:09:53.576 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:09:53.576 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:53.577 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:09:53Z" 2023-07-29T01:09:53.577 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:09:53.578 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592836.654753_1494.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:16.654753Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:09:53Z" } 2023-07-29T01:09:53.578 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592836.654753_1494.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:16.654753Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:09:53.579 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592972.771303_1597.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:09:53.579 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592836.654753_1494.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:09:53.580 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592972.771303_1597.1 target=1_1690592836.654753_1494.1 2023-07-29T01:09:58.581 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:09:58.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:09:58.582 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:09:58.639 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:09:58.640 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:09:58.669 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:09:58.581+0000 7f51da896880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:09:58.670 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:09:58.582+0000 7f51da896880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:09:58.684 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:09:58.684 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:09:58.684 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:09:58.684 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:09:58.685 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:09:58.685 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:09:58.685 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:09:58.685 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:58.685 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:09:58.685 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:58.686 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:09:58.686 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:58.686 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:09:58.686 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:09:58.686 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:58.686 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:09:58.687 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:58.687 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:09:58.687 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:09:58.687 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:58.687 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:58.687 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:58.687 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:09:58.688 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:58.688 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:09:58.688 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592836.654753_1494.1", 2023-07-29T01:09:58.688 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:58.688 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:09:58.688 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:58.689 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:16.654753Z", 2023-07-29T01:09:58.689 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:09:58.689 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:58.689 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:58.689 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:58.689 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:09:58.690 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:58.690 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:09:58.690 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:09:58.690 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:58.690 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:09:58.691 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:58.691 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:09:58.691 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:09:58.692 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:58.692 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:58.692 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:09:58.693 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:09:58.693 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:09:58.693 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:09:58.694 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:09:58.694 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:09:58.694 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:09:58.694 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:09:58.695 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:09:58.695 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:09:58.695 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:58.696 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:09:58.696 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:09:58.696 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:58.700 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:09:58.700 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:09:58.700 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:09:58.700 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:09:58.701 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:09:58Z" 2023-07-29T01:09:58.701 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:09:58.702 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592836.654753_1494.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:16.654753Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:09:58Z" } 2023-07-29T01:09:58.703 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592836.654753_1494.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:16.654753Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:09:58.703 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592972.771303_1597.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:09:58.704 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592836.654753_1494.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:09:58.704 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592972.771303_1597.1 target=1_1690592836.654753_1494.1 2023-07-29T01:10:03.705 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:10:03.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:10:03.706 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:10:03.760 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:10:03.761 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:10:03.793 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:10:03.706+0000 7f55dda0f880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:10:03.794 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:10:03.707+0000 7f55dda0f880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:10:03.808 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:10:03.809 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:10:03.809 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:10:03.809 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:10:03.810 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:10:03.810 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:10:03.810 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:03.811 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:03.812 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:10:03.812 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:03.812 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:10:03.812 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:03.812 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:03.813 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:10:03.813 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:03.814 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:10:03.814 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:03.814 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:10:03.814 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:03.814 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:03.814 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:03.815 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:03.815 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:10:03.815 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:03.815 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:03.815 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592836.654753_1494.1", 2023-07-29T01:10:03.815 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:03.815 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:10:03.816 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:03.816 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:16.654753Z", 2023-07-29T01:10:03.816 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:03.816 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:03.816 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:03.816 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:03.816 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:10:03.817 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:03.817 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:03.817 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:10:03.817 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:03.817 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:10:03.817 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:03.817 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:10:03.818 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:03.818 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:03.818 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:03.818 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:03.818 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:10:03.818 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:03.819 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:03.819 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:10:03.819 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:03.819 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:10:03.819 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:03.819 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:10:03.819 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:03.820 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:03.820 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:03.820 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:10:03.820 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:03.820 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:10:03.820 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:10:03.820 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:10:03.821 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:03.821 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:10:03Z" 2023-07-29T01:10:03.821 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:10:03.822 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592836.654753_1494.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:16.654753Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:10:03Z" } 2023-07-29T01:10:03.822 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592836.654753_1494.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:16.654753Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:10:03.823 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592972.771303_1597.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:10:03.823 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592836.654753_1494.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:10:03.823 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592972.771303_1597.1 target=1_1690592836.654753_1494.1 2023-07-29T01:10:08.824 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:10:08.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:10:08.825 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:10:08.879 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:10:08.880 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:10:08.911 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:10:08.824+0000 7f351f126880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:10:08.912 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:10:08.825+0000 7f351f126880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:10:08.926 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:10:08.927 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:10:08.927 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:10:08.927 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:10:08.927 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:10:08.927 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:10:08.928 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:08.928 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:08.928 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:10:08.928 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:08.928 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:10:08.928 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:08.928 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:08.929 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:10:08.929 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:08.929 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:10:08.929 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:08.929 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:10:08.929 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:08.929 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:08.930 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:08.930 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:08.930 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:10:08.930 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:08.930 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:08.930 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592836.654753_1494.1", 2023-07-29T01:10:08.930 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:08.931 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:10:08.931 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:08.931 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:16.654753Z", 2023-07-29T01:10:08.931 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:08.932 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:08.932 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:08.932 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:08.932 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:10:08.932 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:08.932 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:08.932 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:10:08.933 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:08.933 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:10:08.933 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:08.933 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:10:08.933 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:08.933 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:08.934 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:08.934 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:08.934 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:10:08.934 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:08.934 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:08.934 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:10:08.935 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:08.935 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:10:08.935 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:08.935 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:10:08.935 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:08.935 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:08.935 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:08.936 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:10:08.936 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:08.936 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:10:08.936 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:10:08.936 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:10:08.936 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:08.936 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:10:08Z" 2023-07-29T01:10:08.937 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:10:08.938 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592836.654753_1494.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:16.654753Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:10:08Z" } 2023-07-29T01:10:08.938 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592836.654753_1494.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:16.654753Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:10:08.939 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592972.771303_1597.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:10:08.939 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592836.654753_1494.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:10:08.939 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592972.771303_1597.1 target=1_1690592836.654753_1494.1 2023-07-29T01:10:13.940 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:10:13.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:10:13.942 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:10:13.996 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:10:13.997 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:10:14.029 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:10:13.941+0000 7fbd37c18880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:10:14.030 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:10:13.942+0000 7fbd37c18880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:10:14.044 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:10:14.044 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:10:14.044 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:10:14.045 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:10:14.045 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:10:14.045 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:10:14.045 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:14.045 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:14.045 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:10:14.046 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:14.046 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:10:14.046 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:14.046 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:14.046 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:10:14.046 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:14.046 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:10:14.047 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:14.047 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:10:14.047 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:14.047 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:14.047 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:14.047 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:14.047 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:10:14.048 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:14.048 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:14.048 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592836.654753_1494.1", 2023-07-29T01:10:14.048 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:14.048 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:10:14.048 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:14.048 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:16.654753Z", 2023-07-29T01:10:14.049 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:14.049 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:14.049 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:14.049 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:14.049 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:10:14.049 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:14.049 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:14.050 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:10:14.050 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:14.050 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:10:14.050 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:14.050 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:10:14.050 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:14.050 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:14.051 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:14.051 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:14.051 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:10:14.051 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:14.051 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:14.051 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:10:14.052 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:14.052 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:10:14.052 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:14.052 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:10:14.052 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:14.052 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:14.052 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:14.053 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:10:14.053 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:14.053 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:10:14.053 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:10:14.053 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:10:14.053 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:14.053 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:10:13Z" 2023-07-29T01:10:14.054 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:10:14.055 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592836.654753_1494.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:16.654753Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:10:13Z" } 2023-07-29T01:10:14.055 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592836.654753_1494.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:16.654753Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:10:14.056 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592972.771303_1597.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:10:14.056 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592836.654753_1494.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:10:14.056 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592972.771303_1597.1 target=1_1690592836.654753_1494.1 2023-07-29T01:10:19.059 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:10:19.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:10:19.060 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:10:19.113 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:10:19.114 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:10:19.146 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:10:19.058+0000 7f1737722880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:10:19.147 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:10:19.059+0000 7f1737722880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:10:19.161 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:10:19.161 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:10:19.161 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:10:19.162 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:10:19.162 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:10:19.162 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:10:19.162 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:19.162 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:19.162 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:10:19.163 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:19.163 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:10:19.163 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:19.163 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:19.163 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:10:19.163 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:19.163 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:10:19.164 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:19.164 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:10:19.164 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:19.164 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:19.164 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:19.164 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:19.165 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:10:19.165 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:19.165 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:19.166 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592836.654753_1494.1", 2023-07-29T01:10:19.166 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:19.166 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:10:19.166 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:19.166 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:16.654753Z", 2023-07-29T01:10:19.166 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:19.166 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:19.167 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:19.167 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:19.167 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:10:19.167 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:19.167 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:19.167 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:10:19.167 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:19.168 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:10:19.168 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:19.168 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:10:19.168 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:19.168 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:19.168 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:19.168 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:19.169 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:10:19.169 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:19.169 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:19.169 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:10:19.169 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:19.169 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:10:19.169 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:19.170 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:10:19.170 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:19.170 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:19.170 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:19.170 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:10:19.170 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:19.171 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:10:19.171 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:10:19.171 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:10:19.171 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:19.171 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:10:19Z" 2023-07-29T01:10:19.171 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:10:19.172 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592836.654753_1494.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:16.654753Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:10:19Z" } 2023-07-29T01:10:19.173 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592836.654753_1494.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:16.654753Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:10:19.173 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592972.771303_1597.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:10:19.174 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592836.654753_1494.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:10:19.174 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592972.771303_1597.1 target=1_1690592836.654753_1494.1 2023-07-29T01:10:24.175 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:10:24.176 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:10:24.176 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:10:24.229 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:10:24.230 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:10:24.262 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:10:24.174+0000 7f9db140f880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:10:24.263 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:10:24.175+0000 7f9db140f880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:10:24.277 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:10:24.277 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:10:24.277 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:10:24.278 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:10:24.278 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:10:24.278 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:10:24.278 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:24.278 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:24.278 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:10:24.278 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:24.279 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:10:24.279 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:24.279 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:24.279 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:10:24.279 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:24.279 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:10:24.280 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:24.280 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:10:24.280 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:24.280 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:24.280 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:24.280 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:24.280 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:10:24.281 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:24.281 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:24.281 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592836.654753_1494.1", 2023-07-29T01:10:24.281 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:24.281 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:10:24.281 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:24.281 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:16.654753Z", 2023-07-29T01:10:24.282 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:24.282 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:24.282 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:24.282 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:24.282 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:10:24.282 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:24.283 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:24.283 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:10:24.283 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:24.283 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:10:24.283 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:24.283 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:10:24.283 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:24.284 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:24.284 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:24.284 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:24.284 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:10:24.284 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:24.284 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:24.285 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:10:24.285 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:24.285 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:10:24.285 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:24.286 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:10:24.286 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:24.286 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:24.287 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:24.287 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:10:24.287 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:24.287 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:10:24.287 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:10:24.287 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:10:24.287 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:24.288 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:10:24Z" 2023-07-29T01:10:24.288 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:10:24.289 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592836.654753_1494.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:16.654753Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:10:24Z" } 2023-07-29T01:10:24.289 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592836.654753_1494.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:16.654753Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:10:24.290 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592972.771303_1597.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:10:24.290 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592836.654753_1494.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:10:24.290 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592972.771303_1597.1 target=1_1690592836.654753_1494.1 2023-07-29T01:10:29.291 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:10:29.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:10:29.293 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:10:29.347 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:10:29.348 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:10:29.380 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:10:29.292+0000 7f8da9616880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:10:29.381 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:10:29.293+0000 7f8da9616880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:10:29.395 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:10:29.395 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:10:29.395 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:10:29.396 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:10:29.396 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:10:29.396 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:10:29.397 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:29.397 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:29.397 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:10:29.397 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:29.397 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:10:29.397 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:29.398 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:29.398 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:10:29.398 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:29.398 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:10:29.398 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:29.398 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:10:29.398 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:29.399 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:29.399 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:29.399 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:29.399 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:10:29.399 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:29.399 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:29.399 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592836.654753_1494.1", 2023-07-29T01:10:29.400 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:29.400 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:10:29.400 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:29.400 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:16.654753Z", 2023-07-29T01:10:29.400 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:29.400 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:29.401 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:29.401 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:29.401 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:10:29.401 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:29.401 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:29.401 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:10:29.401 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:29.402 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:10:29.402 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:29.402 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:10:29.402 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:29.402 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:29.402 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:29.402 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:29.403 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:10:29.403 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:29.403 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:29.403 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:10:29.403 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:29.403 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:10:29.403 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:29.404 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:10:29.404 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:29.404 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:29.404 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:29.404 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:10:29.404 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:29.404 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:10:29.405 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:10:29.405 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:10:29.405 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:29.405 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:10:29Z" 2023-07-29T01:10:29.405 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:10:29.406 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592836.654753_1494.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:16.654753Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:10:29Z" } 2023-07-29T01:10:29.407 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592836.654753_1494.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:16.654753Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:10:29.407 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592972.771303_1597.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:10:29.408 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592836.654753_1494.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:10:29.408 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592972.771303_1597.1 target=1_1690592836.654753_1494.1 2023-07-29T01:10:34.409 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:10:34.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:10:34.410 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:10:34.464 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:10:34.464 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:10:34.496 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:10:34.408+0000 7ff458014880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:10:34.497 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:10:34.409+0000 7ff458014880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:10:34.511 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:10:34.511 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:10:34.512 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:10:34.512 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:10:34.512 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:10:34.512 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:10:34.512 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:34.512 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:34.513 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:10:34.513 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:34.513 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:10:34.513 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:34.513 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:34.513 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:10:34.513 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:34.514 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:10:34.514 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:34.514 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:10:34.514 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:34.514 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:34.514 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:34.514 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:34.515 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:10:34.515 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:34.515 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:34.515 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592836.654753_1494.1", 2023-07-29T01:10:34.515 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:34.515 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:10:34.515 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:34.516 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:16.654753Z", 2023-07-29T01:10:34.516 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:34.516 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:34.516 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:34.516 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:34.516 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:10:34.516 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:34.517 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:34.517 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:10:34.517 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:34.517 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:10:34.517 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:34.517 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:10:34.517 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:34.518 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:34.518 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:34.518 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:34.518 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:10:34.518 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:34.519 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:34.519 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:10:34.519 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:34.519 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:10:34.519 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:34.519 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:10:34.520 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:34.520 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:34.520 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:34.520 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:10:34.520 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:34.520 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:10:34.521 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:10:34.521 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:10:34.521 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:34.521 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:10:34Z" 2023-07-29T01:10:34.521 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:10:34.522 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592836.654753_1494.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:16.654753Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:10:34Z" } 2023-07-29T01:10:34.523 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592836.654753_1494.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:16.654753Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:10:34.523 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592972.771303_1597.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:10:34.523 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592836.654753_1494.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:10:34.524 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592972.771303_1597.1 target=1_1690592836.654753_1494.1 2023-07-29T01:10:39.524 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:10:39.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:10:39.526 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:10:39.579 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:10:39.580 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:10:39.612 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:10:39.524+0000 7f0e5a520880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:10:39.613 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:10:39.525+0000 7f0e5a520880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:10:39.627 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:10:39.627 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:10:39.627 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:10:39.627 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:10:39.628 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:10:39.628 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:10:39.628 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:39.628 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:39.628 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:10:39.628 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:39.629 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:10:39.629 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:39.629 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:39.629 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:10:39.629 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:39.629 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:10:39.629 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:39.630 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:10:39.630 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:39.630 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:39.630 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:39.630 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:39.630 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:10:39.631 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:39.631 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:39.631 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592836.654753_1494.1", 2023-07-29T01:10:39.631 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:39.631 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:10:39.631 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:39.631 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:16.654753Z", 2023-07-29T01:10:39.632 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:39.632 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:39.632 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:39.632 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:39.633 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:10:39.633 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:39.633 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:39.634 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:10:39.634 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:39.634 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:10:39.635 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:39.635 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:10:39.635 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:39.636 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:39.636 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:39.636 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:39.637 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:10:39.637 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:39.637 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:39.638 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:10:39.638 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:39.639 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:10:39.639 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:39.639 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:10:39.639 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:39.640 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:39.640 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:39.640 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:10:39.640 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:39.641 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:10:39.641 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:10:39.641 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:10:39.642 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:39.642 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:10:39Z" 2023-07-29T01:10:39.642 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:10:39.643 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592836.654753_1494.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:16.654753Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:10:39Z" } 2023-07-29T01:10:39.644 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592836.654753_1494.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:16.654753Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:10:39.644 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592972.771303_1597.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:10:39.644 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592836.654753_1494.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:10:39.645 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592972.771303_1597.1 target=1_1690592836.654753_1494.1 2023-07-29T01:10:44.646 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:10:44.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:10:44.647 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:10:44.699 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:10:44.700 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:10:44.732 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:10:44.644+0000 7f2daffa2880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:10:44.733 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:10:44.645+0000 7f2daffa2880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:10:44.747 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:10:44.747 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:10:44.747 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:10:44.747 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:10:44.748 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:10:44.748 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:10:44.748 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:44.748 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:44.748 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:10:44.748 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:44.749 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:10:44.749 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:44.749 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:44.749 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:10:44.749 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:44.750 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:10:44.750 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:44.750 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:10:44.750 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:44.750 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:44.750 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:44.750 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:44.751 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:10:44.751 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:44.751 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:44.751 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592836.654753_1494.1", 2023-07-29T01:10:44.751 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:44.751 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:10:44.751 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:44.752 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:16.654753Z", 2023-07-29T01:10:44.752 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:44.752 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:44.752 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:44.752 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:44.752 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:10:44.753 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:44.753 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:44.753 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:10:44.753 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:44.753 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:10:44.753 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:44.753 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:10:44.754 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:44.754 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:44.754 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:44.754 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:44.754 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:10:44.754 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:44.755 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:44.755 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:10:44.755 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:44.755 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:10:44.755 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:44.755 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:10:44.755 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:44.756 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:44.756 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:44.756 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:10:44.756 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:44.756 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:10:44.756 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:10:44.757 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:10:44.757 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:44.757 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:10:44Z" 2023-07-29T01:10:44.757 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:10:44.758 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592836.654753_1494.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:16.654753Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:10:44Z" } 2023-07-29T01:10:44.758 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592836.654753_1494.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:16.654753Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:10:44.759 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592972.771303_1597.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:10:44.759 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592836.654753_1494.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:10:44.759 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592972.771303_1597.1 target=1_1690592836.654753_1494.1 2023-07-29T01:10:49.761 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:10:49.762 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:10:49.763 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:10:49.817 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:10:49.817 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:10:49.851 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:10:49.763+0000 7f4c76c0f880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:10:49.852 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:10:49.764+0000 7f4c76c0f880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:10:49.868 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:10:49.868 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:10:49.868 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:10:49.869 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:10:49.869 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:10:49.869 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:10:49.869 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:49.869 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:49.869 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:10:49.869 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:49.870 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:10:49.870 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:49.870 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:49.870 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:10:49.870 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:49.870 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:10:49.870 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:49.871 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:10:49.871 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:49.871 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:49.871 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:49.871 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:49.871 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:10:49.872 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:49.872 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:49.872 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592836.654753_1494.1", 2023-07-29T01:10:49.872 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:49.872 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:10:49.872 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:49.872 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:16.654753Z", 2023-07-29T01:10:49.873 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:49.873 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:49.873 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:49.873 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:49.873 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:10:49.873 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:49.873 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:49.874 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:10:49.874 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:49.874 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:10:49.874 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:49.874 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:10:49.874 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:49.874 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:49.875 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:49.875 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:49.875 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:10:49.875 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:49.875 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:49.875 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:10:49.875 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:49.876 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:10:49.876 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:49.876 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:10:49.876 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:49.876 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:49.876 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:49.876 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:10:49.877 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:49.877 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:10:49.877 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:10:49.877 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:10:49.877 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:49.877 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:10:49Z" 2023-07-29T01:10:49.877 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:10:49.879 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592836.654753_1494.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:16.654753Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:10:49Z" } 2023-07-29T01:10:49.879 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592836.654753_1494.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:16.654753Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:10:49.880 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592972.771303_1597.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:10:49.880 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592836.654753_1494.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:10:49.880 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592972.771303_1597.1 target=1_1690592836.654753_1494.1 2023-07-29T01:10:54.881 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:10:54.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:10:54.882 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:10:54.933 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:10:54.933 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:10:54.965 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:10:54.877+0000 7fc12b1af880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:10:54.966 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:10:54.878+0000 7fc12b1af880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:10:54.979 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:10:54.979 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:10:54.979 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:10:54.980 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:10:54.980 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:10:54.980 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:10:54.980 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:54.980 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:54.980 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:10:54.981 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:54.981 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:10:54.981 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:54.981 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:54.981 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:10:54.981 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:54.981 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:10:54.982 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:54.982 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:10:54.982 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:54.982 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:54.982 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:54.982 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:54.982 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:10:54.983 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:54.983 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:54.983 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592836.654753_1494.1", 2023-07-29T01:10:54.983 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:54.983 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:10:54.983 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:54.983 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:16.654753Z", 2023-07-29T01:10:54.984 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:54.984 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:54.984 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:54.984 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:54.984 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:10:54.984 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:54.984 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:54.985 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:10:54.985 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:54.985 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:10:54.985 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:54.985 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:10:54.985 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:54.985 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:54.986 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:54.986 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:10:54.986 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:10:54.986 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:10:54.986 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:10:54.986 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:10:54.986 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:10:54.987 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:10:54.987 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:10:54.987 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:10:54.987 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:10:54.987 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:54.987 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:10:54.987 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:10:54.988 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:54.988 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:10:54.988 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:10:54.988 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:10:54.988 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:10:54.988 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:10:54Z" 2023-07-29T01:10:54.988 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:10:54.989 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592836.654753_1494.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:16.654753Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:10:54Z" } 2023-07-29T01:10:54.990 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592836.654753_1494.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:16.654753Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:10:54.990 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592972.771303_1597.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:10:54.991 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592836.654753_1494.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:10:54.991 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592972.771303_1597.1 target=1_1690592836.654753_1494.1 2023-07-29T01:10:59.992 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:10:59.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:10:59.994 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:11:00.047 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:11:00.048 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:11:00.080 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:10:59.992+0000 7fbe30220880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:11:00.081 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:10:59.993+0000 7fbe30220880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:11:00.095 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:11:00.096 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:11:00.096 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:11:00.096 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:11:00.096 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:11:00.096 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:11:00.096 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:00.097 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:00.097 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:11:00.097 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:00.097 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:11:00.097 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:00.098 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:00.098 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:11:00.098 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:00.098 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:11:00.098 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:00.098 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:11:00.098 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:00.099 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:00.099 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:00.099 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:00.099 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:11:00.099 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:00.099 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:00.100 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592836.654753_1494.1", 2023-07-29T01:11:00.100 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:00.100 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:11:00.100 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:00.100 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:16.654753Z", 2023-07-29T01:11:00.100 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:00.101 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:00.101 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:00.101 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:00.101 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:11:00.101 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:00.101 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:00.101 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:11:00.102 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:00.102 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:11:00.102 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:00.102 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:11:00.102 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:00.102 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:00.102 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:00.103 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:00.103 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:11:00.103 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:00.103 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:00.103 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:11:00.103 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:00.103 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:11:00.104 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:00.104 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:11:00.104 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:00.104 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:00.104 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:00.104 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:11:00.105 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:00.105 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:11:00.105 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:11:00.105 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:11:00.105 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:00.105 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:10:59Z" 2023-07-29T01:11:00.106 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:11:00.107 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592836.654753_1494.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:16.654753Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:10:59Z" } 2023-07-29T01:11:00.107 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592836.654753_1494.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:16.654753Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:11:00.108 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592972.771303_1597.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:11:00.108 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592836.654753_1494.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:11:00.108 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592972.771303_1597.1 target=1_1690592836.654753_1494.1 2023-07-29T01:11:05.110 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:11:05.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:11:05.111 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:11:05.164 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:11:05.165 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:11:05.197 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:11:05.109+0000 7fa2f3522880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:11:05.198 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:11:05.110+0000 7fa2f3522880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:11:05.214 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:11:05.214 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:11:05.214 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:11:05.214 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:11:05.214 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:11:05.215 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:11:05.215 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:05.215 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:05.215 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:11:05.215 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:05.216 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:11:05.216 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:05.216 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:05.216 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:11:05.216 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:05.216 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:11:05.217 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:05.217 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:11:05.217 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:05.217 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:05.217 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:05.217 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:05.217 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:11:05.218 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:05.218 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:05.218 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592836.654753_1494.1", 2023-07-29T01:11:05.218 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:05.218 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:11:05.218 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:05.219 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:16.654753Z", 2023-07-29T01:11:05.219 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:05.219 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:05.219 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:05.219 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:05.219 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:11:05.219 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:05.220 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:05.220 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:11:05.220 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:05.220 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:11:05.220 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:05.221 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:11:05.221 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:05.221 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:05.221 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:05.221 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:05.222 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:11:05.222 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:05.222 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:05.222 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:11:05.222 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:05.222 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:11:05.223 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:05.223 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:11:05.223 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:05.223 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:05.223 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:05.223 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:11:05.223 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:05.224 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:11:05.224 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:11:05.224 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:11:05.224 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:05.224 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:11:05Z" 2023-07-29T01:11:05.224 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:11:05.226 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592836.654753_1494.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:16.654753Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:11:05Z" } 2023-07-29T01:11:05.226 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592836.654753_1494.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:16.654753Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:11:05.227 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592972.771303_1597.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:11:05.227 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592836.654753_1494.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:11:05.228 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592972.771303_1597.1 target=1_1690592836.654753_1494.1 2023-07-29T01:11:10.229 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:11:10.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:11:10.230 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:11:10.284 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:11:10.284 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:11:10.316 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:11:10.228+0000 7f70e709d880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:11:10.317 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:11:10.229+0000 7f70e709d880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:11:10.331 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:11:10.331 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:11:10.331 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:11:10.332 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:11:10.332 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:11:10.332 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:11:10.332 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:10.332 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:10.332 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:11:10.333 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:10.333 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:11:10.333 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:10.333 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:10.333 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:11:10.333 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:10.333 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:11:10.334 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:10.334 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:11:10.334 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:10.334 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:10.334 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:10.334 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:10.334 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:11:10.335 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:10.335 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:10.335 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592836.654753_1494.1", 2023-07-29T01:11:10.335 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:10.335 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:11:10.335 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:10.335 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:16.654753Z", 2023-07-29T01:11:10.336 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:10.336 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:10.336 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:10.336 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:10.336 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:11:10.336 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:10.337 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:10.337 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:11:10.337 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:10.337 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:11:10.337 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:10.337 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:11:10.337 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:10.338 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:10.338 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:10.338 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:10.338 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:11:10.338 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:10.338 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:10.338 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:11:10.339 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:10.339 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:11:10.339 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:10.339 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:11:10.339 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:10.339 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:10.339 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:10.340 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:11:10.340 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:10.340 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:11:10.340 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:11:10.340 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:11:10.340 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:10.340 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:11:10Z" 2023-07-29T01:11:10.341 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:11:10.341 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592836.654753_1494.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:16.654753Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:11:10Z" } 2023-07-29T01:11:10.342 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592836.654753_1494.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:16.654753Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:11:10.342 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592972.771303_1597.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:11:10.343 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592836.654753_1494.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:11:10.343 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592972.771303_1597.1 target=1_1690592836.654753_1494.1 2023-07-29T01:11:15.344 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:11:15.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:11:15.345 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:11:15.398 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:11:15.399 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:11:15.431 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:11:15.342+0000 7fb24962e880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:11:15.432 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:11:15.343+0000 7fb24962e880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:11:15.447 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:11:15.448 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:11:15.448 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:11:15.448 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:11:15.448 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:11:15.448 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:11:15.448 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:15.449 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:15.449 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:11:15.449 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:15.449 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:11:15.449 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:15.449 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:15.450 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:11:15.450 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:15.450 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:11:15.450 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:15.450 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:11:15.450 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:15.451 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:15.451 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:15.451 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:15.451 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:11:15.451 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:15.451 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:15.451 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592836.654753_1494.1", 2023-07-29T01:11:15.452 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:15.452 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:11:15.452 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:15.452 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:16.654753Z", 2023-07-29T01:11:15.452 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:15.452 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:15.453 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:15.453 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:15.453 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:11:15.453 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:15.453 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:15.453 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:11:15.454 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:15.454 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:11:15.454 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:15.454 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:11:15.454 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:15.454 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:15.454 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:15.455 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:15.455 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:11:15.455 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:15.455 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:15.455 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:11:15.455 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:15.455 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:11:15.456 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:15.456 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:11:15.456 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:15.456 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:15.456 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:15.456 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:11:15.457 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:15.457 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:11:15.457 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:11:15.458 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:11:15.458 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:15.458 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:11:15Z" 2023-07-29T01:11:15.459 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:11:15.460 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592836.654753_1494.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:16.654753Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:11:15Z" } 2023-07-29T01:11:15.461 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592836.654753_1494.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:16.654753Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:11:15.462 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592972.771303_1597.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:11:15.463 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592836.654753_1494.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:11:15.463 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592972.771303_1597.1 target=1_1690592836.654753_1494.1 2023-07-29T01:11:20.464 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:11:20.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:11:20.466 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:11:20.518 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:11:20.519 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:11:20.551 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:11:20.462+0000 7f1636d26880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:11:20.551 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:11:20.463+0000 7f1636d26880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:11:20.566 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:11:20.566 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:11:20.566 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:11:20.567 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:11:20.567 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:11:20.567 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:11:20.567 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:20.567 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:20.568 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:11:20.568 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:20.568 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:11:20.568 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:20.568 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:20.568 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:11:20.568 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:20.569 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:11:20.569 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:20.569 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:11:20.569 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:20.569 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:20.569 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:20.569 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:20.570 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:11:20.570 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:20.570 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:20.570 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592836.654753_1494.1", 2023-07-29T01:11:20.570 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:20.570 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:11:20.570 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:20.571 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:16.654753Z", 2023-07-29T01:11:20.571 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:20.571 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:20.571 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:20.571 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:20.571 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:11:20.571 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:20.572 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:20.572 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:11:20.572 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:20.572 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:11:20.572 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:20.572 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:11:20.573 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:20.573 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:20.573 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:20.573 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:20.573 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:11:20.573 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:20.573 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:20.574 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:11:20.574 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:20.574 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:11:20.574 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:20.574 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:11:20.575 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:20.575 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:20.575 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:20.575 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:11:20.575 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:20.576 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:11:20.576 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:11:20.576 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:11:20.576 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:20.576 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:11:20Z" 2023-07-29T01:11:20.576 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:11:20.577 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592836.654753_1494.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:16.654753Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:11:20Z" } 2023-07-29T01:11:20.578 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592836.654753_1494.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:16.654753Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:11:20.578 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592972.771303_1597.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:11:20.579 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592836.654753_1494.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:11:20.579 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592972.771303_1597.1 target=1_1690592836.654753_1494.1 2023-07-29T01:11:25.580 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:11:25.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:11:25.582 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:11:25.637 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:11:25.637 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:11:25.673 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:11:25.584+0000 7fa18d126880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:11:25.674 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:11:25.585+0000 7fa18d126880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:11:25.687 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:11:25.688 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:11:25.688 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:11:25.689 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:11:25.689 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:11:25.689 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:11:25.689 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:25.689 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:25.690 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:11:25.690 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:25.690 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:11:25.690 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:25.690 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:25.690 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:11:25.690 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:25.691 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:11:25.691 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:25.691 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:11:25.691 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:25.691 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:25.691 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:25.692 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:25.692 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:11:25.692 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:25.692 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:25.692 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592836.654753_1494.1", 2023-07-29T01:11:25.692 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:25.692 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:11:25.693 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:25.693 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:16.654753Z", 2023-07-29T01:11:25.693 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:25.693 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:25.693 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:25.693 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:25.693 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:11:25.694 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:25.694 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:25.694 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:11:25.694 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:25.694 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:11:25.694 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:25.695 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:11:25.695 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:25.695 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:25.695 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:25.695 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:25.695 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:11:25.696 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:25.696 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:25.696 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:11:25.696 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:25.697 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:11:25.697 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:25.697 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:11:25.698 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:25.698 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:25.698 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:25.698 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:11:25.698 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:25.699 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:11:25.699 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:11:25.699 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:11:25.699 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:25.699 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:11:25Z" 2023-07-29T01:11:25.699 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:11:25.700 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592836.654753_1494.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:16.654753Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:11:25Z" } 2023-07-29T01:11:25.701 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592836.654753_1494.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:16.654753Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:11:25.701 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592972.771303_1597.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:11:25.701 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592836.654753_1494.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:11:25.702 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592972.771303_1597.1 target=1_1690592836.654753_1494.1 2023-07-29T01:11:30.703 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:11:30.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:11:30.704 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:11:30.759 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:11:30.759 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:11:30.791 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:11:30.702+0000 7fc50ae12880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:11:30.792 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:11:30.703+0000 7fc50ae12880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:11:30.805 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:11:30.806 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:11:30.806 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:11:30.806 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:11:30.806 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:11:30.806 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:11:30.807 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:30.807 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:30.807 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:11:30.807 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:30.807 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:11:30.807 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:30.807 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:30.808 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:11:30.808 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:30.808 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:11:30.808 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:30.808 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:11:30.808 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:30.809 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:30.809 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:30.809 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:30.809 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:11:30.809 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:30.809 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:30.809 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592836.654753_1494.1", 2023-07-29T01:11:30.810 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:30.810 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:11:30.810 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:30.810 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:16.654753Z", 2023-07-29T01:11:30.810 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:30.811 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:30.811 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:30.811 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:30.811 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:11:30.811 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:30.811 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:30.812 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:11:30.812 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:30.812 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:11:30.812 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:30.812 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:11:30.813 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:30.813 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:30.813 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:30.813 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:30.813 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:11:30.813 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:30.814 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:30.814 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:11:30.814 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:30.814 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:11:30.814 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:30.814 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:11:30.814 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:30.815 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:30.815 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:30.815 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:11:30.815 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:30.815 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:11:30.815 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:11:30.815 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:11:30.816 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:30.816 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:11:30Z" 2023-07-29T01:11:30.816 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:11:30.817 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592836.654753_1494.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:16.654753Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:11:30Z" } 2023-07-29T01:11:30.817 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592836.654753_1494.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:16.654753Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:11:30.818 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592972.771303_1597.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:11:30.818 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592836.654753_1494.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:11:30.818 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690592972.771303_1597.1 target=1_1690592836.654753_1494.1 2023-07-29T01:11:35.820 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:11:35.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:11:35.821 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:11:35.874 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:11:35.875 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:11:35.907 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:11:35.818+0000 7f28e5d26880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:11:35.908 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:11:35.819+0000 7f28e5d26880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:11:35.922 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:11:35.922 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:11:35.922 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:11:35.922 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:11:35.922 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:11:35.922 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:11:35.923 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:35.923 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:35.923 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:11:35.923 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:35.923 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:11:35.923 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:35.923 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:35.924 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:11:35.924 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:35.924 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:11:35.924 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:35.924 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:11:35.924 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:35.925 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:35.925 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:35.925 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:35.925 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:11:35.925 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:35.925 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:35.925 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592972.771303_1597.1", 2023-07-29T01:11:35.926 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:35.926 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:11:35.926 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:35.926 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:09:32.771303Z", 2023-07-29T01:11:35.926 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:35.926 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:35.926 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:35.927 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:35.927 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:11:35.927 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:35.927 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:35.927 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:11:35.927 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:35.927 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:11:35.928 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:35.928 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:11:35.928 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:35.928 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:35.928 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:35.928 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:35.928 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:11:35.929 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:35.929 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:35.929 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:11:35.929 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:35.929 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:11:35.929 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:35.929 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:11:35.930 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:35.930 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:35.930 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:35.930 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:11:35.930 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:35.930 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:11:35.930 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:11:35.931 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:11:35.931 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:35.931 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:11:35Z" 2023-07-29T01:11:35.931 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:11:35.932 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592972.771303_1597.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:09:32.771303Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:11:35Z" } 2023-07-29T01:11:35.932 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592972.771303_1597.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:09:32.771303Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:11:35.933 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592972.771303_1597.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:11:35.933 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592972.771303_1597.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:11:35.934 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T01:11:35.934 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:11:35.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:11:35.935 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:11:35.981 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:11:35.981 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:11:36.029 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T01:11:36.029 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:36.030 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051953", 2023-07-29T01:11:36.030 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:09:37.925040Z" 2023-07-29T01:11:36.030 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:36.030 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:36.030 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044881", 2023-07-29T01:11:36.030 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:07:23.001407Z" 2023-07-29T01:11:36.031 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:36.031 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:36.031 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045660", 2023-07-29T01:11:36.031 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:07:22.902962Z" 2023-07-29T01:11:36.031 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:36.031 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:36.031 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:11:36.032 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:11:36.032 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:36.032 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T01:11:36.033 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000051953', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:11:36.033 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-07-29T01:11:36.034 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:11:36.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:11:36.034 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:11:36.081 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:11:36.082 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:11:36.124 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:11:36.124 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:11:36.124 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:11:36.125 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:11:36.125 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:11:36.125 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:11:36.125 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:36.125 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:11:36.125 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:36.125 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:11:36.126 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:36.126 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:11:36.126 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051953", 2023-07-29T01:11:36.126 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:36.126 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:11:36.126 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:36.126 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:09:37.925040Z" 2023-07-29T01:11:36.127 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:36.127 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:36.127 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:36.127 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:11:36.127 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:36.127 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:11:36.127 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044881", 2023-07-29T01:11:36.128 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:36.128 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:11:36.128 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:36.128 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:23.001407Z" 2023-07-29T01:11:36.128 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:36.128 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:36.128 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:36.129 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:11:36.129 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:36.129 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:11:36.129 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045660", 2023-07-29T01:11:36.129 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:36.129 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:11:36.129 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:36.130 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:22.902962Z" 2023-07-29T01:11:36.130 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:36.130 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:36.130 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:36.130 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:11:36.130 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:36.130 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:11:36.131 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:11:36.131 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:36.131 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:11:36.131 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:36.131 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:11:36.131 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:36.132 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:36.132 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:11:36.132 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:36.132 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:11:36.132 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:11:36.132 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:11:36.132 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:36.133 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:11:36Z" 2023-07-29T01:11:36.133 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:11:36.134 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051953", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:09:37.925040Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044881", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:07:23.001407Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045660", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:22.902962Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:11:36Z" } 2023-07-29T01:11:36.134 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051953', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:09:37.925040Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044881', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:07:23.001407Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045660', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:22.902962Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:11:36.134 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051953', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:11:36.135 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051953', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:11:36.135 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-07-29T01:11:36.135 DEBUG:boto:path=/hiiqos-68/ 2023-07-29T01:11:36.136 DEBUG:boto:auth_path=/hiiqos-68/ 2023-07-29T01:11:36.136 DEBUG:boto:Method: HEAD 2023-07-29T01:11:36.137 DEBUG:boto:Path: /hiiqos-68/ 2023-07-29T01:11:36.137 DEBUG:boto:Data: 2023-07-29T01:11:36.137 DEBUG:boto:Headers: {} 2023-07-29T01:11:36.137 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:11:36.138 DEBUG:boto:Port: 8001 2023-07-29T01:11:36.138 DEBUG:boto:Params: {} 2023-07-29T01:11:36.139 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-29T01:11:36.139 DEBUG:boto:Token: None 2023-07-29T01:11:36.139 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:11:36 GMT /hiiqos-68/ 2023-07-29T01:11:36.140 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LW3SghwguM5ilOuYrJWqK7F5unI= 2023-07-29T01:11:36.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:11:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LW3SghwguM5ilOuYrJWqK7F5unI=', 'Content-Length': '0'} 2023-07-29T01:11:36.145 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000be1e80a090a7eba-0064c46748-25c0-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:11:36.145 DEBUG:boto:path=/hiiqos-68/obj3 2023-07-29T01:11:36.146 DEBUG:boto:auth_path=/hiiqos-68/obj3 2023-07-29T01:11:36.146 DEBUG:boto:Method: HEAD 2023-07-29T01:11:36.146 DEBUG:boto:Path: /hiiqos-68/obj3 2023-07-29T01:11:36.147 DEBUG:boto:Data: 2023-07-29T01:11:36.147 DEBUG:boto:Headers: {} 2023-07-29T01:11:36.147 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:11:36.148 DEBUG:boto:Port: 8001 2023-07-29T01:11:36.148 DEBUG:boto:Params: {} 2023-07-29T01:11:36.148 DEBUG:boto:Token: None 2023-07-29T01:11:36.149 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:11:36 GMT /hiiqos-68/obj3 2023-07-29T01:11:36.149 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pHxEv6jY/5HgXMIIqlrCRFBYmRo= 2023-07-29T01:11:36.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:11:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pHxEv6jY/5HgXMIIqlrCRFBYmRo=', 'Content-Length': '0'} 2023-07-29T01:11:36.151 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:07:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-68:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.9034.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009591568e9b5055a6-0064c46748-25c0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:11:36.152 DEBUG:boto:path=/hiiqos-68/obj3 2023-07-29T01:11:36.152 DEBUG:boto:auth_path=/hiiqos-68/obj3 2023-07-29T01:11:36.153 DEBUG:boto:Method: GET 2023-07-29T01:11:36.153 DEBUG:boto:Path: /hiiqos-68/obj3 2023-07-29T01:11:36.153 DEBUG:boto:Data: 2023-07-29T01:11:36.154 DEBUG:boto:Headers: {} 2023-07-29T01:11:36.154 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:11:36.154 DEBUG:boto:Port: 8001 2023-07-29T01:11:36.155 DEBUG:boto:Params: {} 2023-07-29T01:11:36.155 DEBUG:boto:Token: None 2023-07-29T01:11:36.155 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 01:11:36 GMT /hiiqos-68/obj3 2023-07-29T01:11:36.156 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:SUyESmkIQkEt6Ly4htER9SZ2xSM= 2023-07-29T01:11:36.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:11:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SUyESmkIQkEt6Ly4htER9SZ2xSM=', 'Content-Length': '0'} 2023-07-29T01:11:36.158 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:07:22 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-68:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.9034.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b45274be519efa6-0064c46748-25c0-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:11:36.158 DEBUG:boto:path=/hiiqos-69/ 2023-07-29T01:11:36.159 DEBUG:boto:auth_path=/hiiqos-69/ 2023-07-29T01:11:36.159 DEBUG:boto:Method: HEAD 2023-07-29T01:11:36.159 DEBUG:boto:Path: /hiiqos-69/ 2023-07-29T01:11:36.160 DEBUG:boto:Data: 2023-07-29T01:11:36.160 DEBUG:boto:Headers: {} 2023-07-29T01:11:36.160 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:11:36.161 DEBUG:boto:Port: 8001 2023-07-29T01:11:36.161 DEBUG:boto:Params: {} 2023-07-29T01:11:36.161 DEBUG:boto:Token: None 2023-07-29T01:11:36.162 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:11:36 GMT /hiiqos-69/ 2023-07-29T01:11:36.162 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:pGVfKh8DAQOaIHEgZDzPNjfq6uk= 2023-07-29T01:11:36.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:11:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pGVfKh8DAQOaIHEgZDzPNjfq6uk=', 'Content-Length': '0'} 2023-07-29T01:11:36.165 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001ad0d4089cde42c2-0064c46748-25c0-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:11:36.165 DEBUG:boto:path=/hiiqos-69/obj3 2023-07-29T01:11:36.165 DEBUG:boto:auth_path=/hiiqos-69/obj3 2023-07-29T01:11:36.166 DEBUG:boto:Method: HEAD 2023-07-29T01:11:36.166 DEBUG:boto:Path: /hiiqos-69/obj3 2023-07-29T01:11:36.166 DEBUG:boto:Data: 2023-07-29T01:11:36.167 DEBUG:boto:Headers: {} 2023-07-29T01:11:36.167 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:11:36.167 DEBUG:boto:Port: 8001 2023-07-29T01:11:36.168 DEBUG:boto:Params: {} 2023-07-29T01:11:36.168 DEBUG:boto:Token: None 2023-07-29T01:11:36.168 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:11:36 GMT /hiiqos-69/obj3 2023-07-29T01:11:36.169 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:83l8Sh+zoVYOxbtwrDh5rBC8aA4= 2023-07-29T01:11:36.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:11:36 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:83l8Sh+zoVYOxbtwrDh5rBC8aA4=', 'Content-Length': '0'} 2023-07-29T01:11:36.171 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000c5c01e40f9075d1e-0064c46748-25c0-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 01:11:36 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:11:36.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'hiiqos-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:11:36.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'hiiqos-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:11:36.172 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket hiiqos-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:11:36.218 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:11:36.218 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:11:36.267 INFO:teuthology.orchestra.run.smithi101.stdout:{"sync_policy":{"groups":[]}} 2023-07-29T01:11:36.267 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:11:36.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:11:36.268 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:11:36.312 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:11:36.312 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:11:36.358 INFO:teuthology.orchestra.run.smithi101.stdout:{"sync_policy":{"groups":[]}} 2023-07-29T01:11:36.358 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket: ... ok 2023-07-29T01:11:36.360 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:11:36.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:11:36.361 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:11:36.405 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:11:36.405 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:11:36.437 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T01:11:36.450 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T01:11:36.450 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:36.450 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:11:36.451 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:17.836402Z" 2023-07-29T01:11:36.451 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:36.451 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:36.451 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593096.165121_1607.1", 2023-07-29T01:11:36.451 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:11:36.165121Z" 2023-07-29T01:11:36.451 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:36.452 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:36.452 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:11:36.452 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T01:11:36.452 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:36.452 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:36.452 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:11:36.452 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:12.811406Z" 2023-07-29T01:11:36.453 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:36.453 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T01:11:36.454 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690591217.836402_294.1', 1: '1_1690593096.165121_1607.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:11:36.454 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T01:11:36.454 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:11:36.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:11:36.455 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:11:36.500 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:11:36.500 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:11:36.529 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:11:36.440+0000 7f236b012880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:11:36.530 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:11:36.441+0000 7f236b012880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:11:36.542 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:11:36.543 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:11:36.543 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:11:36.543 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:11:36.544 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:11:36.544 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:11:36.544 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:36.544 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:36.545 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:11:36.545 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:36.545 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:11:36.545 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:36.546 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:36.546 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:11:36.546 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:36.546 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:11:36.547 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:36.547 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:11:36.547 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:36.547 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:36.548 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:36.548 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:36.548 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:11:36.548 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:36.549 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:36.549 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592972.771303_1597.1", 2023-07-29T01:11:36.549 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:36.549 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:11:36.549 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:36.550 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:09:32.771303Z", 2023-07-29T01:11:36.550 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:36.550 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:36.550 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:36.551 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:36.551 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:11:36.551 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:36.551 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:36.552 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:11:36.552 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:36.552 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:11:36.552 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:36.553 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:11:36.553 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:36.553 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:36.553 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:36.554 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:36.554 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:11:36.554 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:36.554 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:36.554 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:11:36.555 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:36.555 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:11:36.555 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:36.556 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:11:36.556 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:36.556 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:36.557 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:36.557 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:11:36.557 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:36.557 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:11:36.558 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:11:36.558 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:11:36.558 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:36.558 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:11:36Z" 2023-07-29T01:11:36.558 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:11:36.560 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592972.771303_1597.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:09:32.771303Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:11:36Z" } 2023-07-29T01:11:36.560 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592972.771303_1597.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:09:32.771303Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:11:36.561 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593096.165121_1607.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:11:36.562 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592972.771303_1597.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:11:36.562 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593096.165121_1607.1 target=1_1690592972.771303_1597.1 2023-07-29T01:11:41.564 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:11:41.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:11:41.565 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:11:41.617 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:11:41.617 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:11:41.649 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:11:41.561+0000 7f9add735880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:11:41.650 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:11:41.562+0000 7f9add735880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:11:41.664 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:11:41.664 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:11:41.664 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:11:41.665 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:11:41.665 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:11:41.665 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:11:41.665 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:41.665 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:41.665 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:11:41.666 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:41.666 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:11:41.666 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:41.666 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:41.666 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:11:41.666 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:41.666 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:11:41.667 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:41.667 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:11:41.667 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:41.667 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:41.667 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:41.667 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:41.667 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:11:41.668 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:41.668 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:41.668 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592972.771303_1597.1", 2023-07-29T01:11:41.668 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:41.668 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:11:41.668 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:41.668 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:09:32.771303Z", 2023-07-29T01:11:41.669 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:41.669 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:41.669 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:41.669 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:41.669 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:11:41.669 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:41.669 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:41.670 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:11:41.670 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:41.670 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:11:41.670 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:41.670 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:11:41.670 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:41.670 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:41.671 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:41.671 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:41.671 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:11:41.671 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:41.671 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:41.671 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:11:41.671 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:41.672 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:11:41.672 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:41.672 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:11:41.672 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:41.672 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:41.672 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:41.673 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:11:41.673 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:41.673 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:11:41.673 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:11:41.673 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:11:41.673 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:41.673 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:11:41Z" 2023-07-29T01:11:41.674 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:11:41.674 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592972.771303_1597.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:09:32.771303Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:11:41Z" } 2023-07-29T01:11:41.675 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592972.771303_1597.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:09:32.771303Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:11:41.675 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593096.165121_1607.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:11:41.676 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592972.771303_1597.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:11:41.676 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593096.165121_1607.1 target=1_1690592972.771303_1597.1 2023-07-29T01:11:46.678 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:11:46.679 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:11:46.680 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:11:46.733 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:11:46.733 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:11:46.765 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:11:46.676+0000 7fd726538880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:11:46.766 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:11:46.677+0000 7fd726538880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:11:46.780 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:11:46.780 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:11:46.780 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:11:46.780 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:11:46.780 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:11:46.781 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:11:46.781 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:46.781 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:46.781 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:11:46.781 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:46.781 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:11:46.781 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:46.782 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:46.782 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:11:46.782 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:46.782 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:11:46.782 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:46.782 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:11:46.782 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:46.783 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:46.783 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:46.783 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:46.783 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:11:46.783 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:46.783 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:46.783 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592972.771303_1597.1", 2023-07-29T01:11:46.784 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:46.784 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:11:46.784 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:46.784 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:09:32.771303Z", 2023-07-29T01:11:46.784 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:46.784 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:46.785 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:46.785 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:46.785 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:11:46.785 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:46.785 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:46.785 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:11:46.785 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:46.786 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:11:46.786 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:46.786 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:11:46.786 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:46.786 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:46.786 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:46.786 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:46.787 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:11:46.787 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:46.787 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:46.787 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:11:46.787 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:46.787 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:11:46.787 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:46.788 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:11:46.788 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:46.788 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:46.788 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:46.788 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:11:46.788 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:46.789 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:11:46.789 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:11:46.789 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:11:46.789 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:46.789 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:11:46Z" 2023-07-29T01:11:46.789 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:11:46.790 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592972.771303_1597.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:09:32.771303Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:11:46Z" } 2023-07-29T01:11:46.790 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592972.771303_1597.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:09:32.771303Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:11:46.791 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593096.165121_1607.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:11:46.791 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592972.771303_1597.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:11:46.792 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593096.165121_1607.1 target=1_1690592972.771303_1597.1 2023-07-29T01:11:51.793 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:11:51.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:11:51.794 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:11:51.847 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:11:51.847 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:11:51.879 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:11:51.791+0000 7efe2a726880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:11:51.880 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:11:51.792+0000 7efe2a726880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:11:51.894 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:11:51.895 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:11:51.895 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:11:51.895 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:11:51.896 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:11:51.896 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:11:51.896 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:51.896 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:51.897 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:11:51.897 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:51.897 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:11:51.898 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:51.898 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:51.898 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:11:51.899 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:51.899 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:11:51.899 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:51.899 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:11:51.900 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:51.900 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:51.900 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:51.901 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:51.901 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:11:51.901 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:51.902 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:51.902 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690592972.771303_1597.1", 2023-07-29T01:11:51.902 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:51.902 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:11:51.903 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:51.903 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:09:32.771303Z", 2023-07-29T01:11:51.903 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:51.903 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:51.904 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:51.904 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:51.904 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:11:51.905 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:51.905 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:51.905 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:11:51.905 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:51.906 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:11:51.906 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:51.906 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:11:51.907 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:51.907 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:51.907 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:51.907 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:51.908 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:11:51.908 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:51.908 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:51.909 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:11:51.909 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:51.909 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:11:51.909 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:51.910 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:11:51.910 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:51.910 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:51.911 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:51.911 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:11:51.911 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:51.912 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:11:51.912 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:11:51.912 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:11:51.912 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:51.913 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:11:51Z" 2023-07-29T01:11:51.913 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:11:51.914 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690592972.771303_1597.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:09:32.771303Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:11:51Z" } 2023-07-29T01:11:51.915 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592972.771303_1597.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:09:32.771303Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:11:51.916 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593096.165121_1607.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:11:51.916 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592972.771303_1597.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:11:51.917 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593096.165121_1607.1 target=1_1690592972.771303_1597.1 2023-07-29T01:11:56.918 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:11:56.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:11:56.920 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:11:56.976 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:11:56.976 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:11:57.008 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:11:56.920+0000 7f4d92136880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:11:57.009 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:11:56.921+0000 7f4d92136880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:11:57.023 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:11:57.024 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:11:57.024 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:11:57.024 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:11:57.024 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:11:57.025 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:11:57.025 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:57.025 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:57.025 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:11:57.025 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:57.025 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:11:57.025 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:57.026 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:57.026 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:11:57.026 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:57.026 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:11:57.027 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:57.027 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:11:57.027 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:57.027 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:57.027 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:57.027 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:57.027 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:11:57.028 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:57.028 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:57.028 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593096.165121_1607.1", 2023-07-29T01:11:57.028 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:57.028 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:11:57.028 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:57.028 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:11:36.165121Z", 2023-07-29T01:11:57.029 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:57.029 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:57.029 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:57.029 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:57.029 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:11:57.029 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:57.029 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:57.030 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:11:57.030 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:57.030 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:11:57.030 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:57.030 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:11:57.030 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:57.030 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:57.031 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:57.031 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:11:57.031 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:11:57.031 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:11:57.031 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:11:57.031 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:11:57.031 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:11:57.032 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:11:57.032 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:11:57.032 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:11:57.032 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:11:57.032 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:57.032 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:11:57.032 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:11:57.033 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:57.033 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:11:57.033 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:11:57.033 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:11:57.033 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:11:57.033 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:11:56Z" 2023-07-29T01:11:57.033 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:11:57.035 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593096.165121_1607.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:11:36.165121Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:11:56Z" } 2023-07-29T01:11:57.036 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593096.165121_1607.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:11:36.165121Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:11:57.037 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593096.165121_1607.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:11:57.037 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593096.165121_1607.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:11:57.038 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T01:11:57.038 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:11:57.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:11:57.039 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:11:57.082 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:11:57.083 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:11:57.127 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[]} 2023-07-29T01:11:57.127 INFO:rgw_multi.tests:create bucket zone=a1 name=hiiqos-70 2023-07-29T01:11:57.127 DEBUG:boto:path=/hiiqos-70/ 2023-07-29T01:11:57.128 DEBUG:boto:auth_path=/hiiqos-70/ 2023-07-29T01:11:57.128 DEBUG:boto:Method: PUT 2023-07-29T01:11:57.129 DEBUG:boto:Path: /hiiqos-70/ 2023-07-29T01:11:57.129 DEBUG:boto:Data: 2023-07-29T01:11:57.129 DEBUG:boto:Headers: {} 2023-07-29T01:11:57.130 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:11:57.130 DEBUG:boto:Port: 8000 2023-07-29T01:11:57.131 DEBUG:boto:Params: {} 2023-07-29T01:11:57.131 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-29T01:11:57.131 DEBUG:boto:Token: None 2023-07-29T01:11:57.132 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 01:11:57 GMT /hiiqos-70/ 2023-07-29T01:11:57.132 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tpWct8z/eTcelD+FPvHwVY+CeL8= 2023-07-29T01:11:57.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:11:57 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tpWct8z/eTcelD+FPvHwVY+CeL8=', 'Content-Length': '0'} 2023-07-29T01:12:02.153 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000037d094945e53acf9-0064c46762-25bc-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:12:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:12:02.153 INFO:rgw_multi.tests:create bucket zone=a1 name=hiiqos-71 2023-07-29T01:12:02.154 DEBUG:boto:path=/hiiqos-71/ 2023-07-29T01:12:02.154 DEBUG:boto:auth_path=/hiiqos-71/ 2023-07-29T01:12:02.155 DEBUG:boto:Method: PUT 2023-07-29T01:12:02.155 DEBUG:boto:Path: /hiiqos-71/ 2023-07-29T01:12:02.155 DEBUG:boto:Data: 2023-07-29T01:12:02.155 DEBUG:boto:Headers: {} 2023-07-29T01:12:02.156 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:12:02.156 DEBUG:boto:Port: 8000 2023-07-29T01:12:02.157 DEBUG:boto:Params: {} 2023-07-29T01:12:02.157 DEBUG:boto:Token: None 2023-07-29T01:12:02.157 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 01:12:02 GMT /hiiqos-71/ 2023-07-29T01:12:02.158 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:PDxPqrVMncYYvCBWVeQ9E0MoS/Y= 2023-07-29T01:12:02.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:12:02 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:PDxPqrVMncYYvCBWVeQ9E0MoS/Y=', 'Content-Length': '0'} 2023-07-29T01:12:02.176 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000580837e726fd0050-0064c46762-25bc-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:12:02 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:12:02.176 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:12:02.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:12:02.177 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:12:02.226 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:12:02.226 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:12:02.259 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T01:12:02.273 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T01:12:02.273 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:02.274 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:12:02.274 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:17.836402Z" 2023-07-29T01:12:02.275 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:02.275 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:02.276 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593122.086179_1622.1", 2023-07-29T01:12:02.276 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:12:02.086179Z" 2023-07-29T01:12:02.276 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:02.277 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:02.277 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:12:02.277 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T01:12:02.278 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:02.278 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:02.278 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:12:02.279 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:12.811406Z" 2023-07-29T01:12:02.279 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:02.279 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T01:12:02.280 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690591217.836402_294.1', 1: '1_1690593122.086179_1622.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:12:02.281 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T01:12:02.282 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:12:02.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:12:02.283 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:12:02.331 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:12:02.332 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:12:02.364 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:12:02.275+0000 7f28ae416880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:12:02.365 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:12:02.276+0000 7f28ae416880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:12:02.381 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:12:02.382 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:12:02.382 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:12:02.382 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:12:02.383 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:12:02.383 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:12:02.383 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:12:02.384 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:02.384 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:12:02.384 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:02.385 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:12:02.385 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:02.385 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:12:02.385 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:12:02.386 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:02.386 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:12:02.386 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:02.387 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:12:02.387 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:12:02.387 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:02.388 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:02.388 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:02.388 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:12:02.389 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:02.389 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:12:02.389 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593122.055304_1608.1", 2023-07-29T01:12:02.390 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:02.390 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:12:02.390 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:02.390 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:12:02.055304Z", 2023-07-29T01:12:02.390 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:12:02.391 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:02.391 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:02.391 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:02.391 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:12:02.391 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:02.392 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:12:02.392 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:12:02.392 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:02.392 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:12:02.392 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:02.392 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:12:02.393 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:12:02.393 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:02.393 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:02.393 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:02.393 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:12:02.393 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:02.393 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:12:02.394 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:12:02.394 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:02.394 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:12:02.394 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:02.394 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:12:02.394 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:12:02.394 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:02.395 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:02.395 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:12:02.395 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:02.395 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:12:02.395 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:12:02.395 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:12:02.395 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:02.396 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:12:02Z" 2023-07-29T01:12:02.396 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:12:02.397 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593122.055304_1608.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:12:02.055304Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:12:02Z" } 2023-07-29T01:12:02.398 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593122.055304_1608.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:12:02.055304Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:12:02.398 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593122.086179_1622.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:12:02.398 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593122.055304_1608.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:12:02.399 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593122.086179_1622.1 target=1_1690593122.055304_1608.1 2023-07-29T01:12:07.400 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:12:07.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:12:07.401 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:12:07.457 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:12:07.458 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:12:07.491 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:12:07.402+0000 7fe954410880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:12:07.492 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:12:07.403+0000 7fe954410880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:12:07.506 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:12:07.507 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:12:07.507 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:12:07.507 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:12:07.507 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:12:07.507 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:12:07.507 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:12:07.508 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:07.508 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:12:07.508 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:07.508 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:12:07.508 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:07.508 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:12:07.509 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:12:07.509 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:07.509 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:12:07.509 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:07.509 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:12:07.509 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:12:07.509 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:07.510 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:07.510 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:07.510 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:12:07.510 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:07.510 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:12:07.510 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593122.086179_1622.1", 2023-07-29T01:12:07.510 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:07.511 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:12:07.511 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:07.511 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:12:02.086179Z", 2023-07-29T01:12:07.511 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:12:07.511 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:07.511 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:07.511 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:07.512 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:12:07.512 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:07.512 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:12:07.512 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:12:07.512 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:07.512 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:12:07.513 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:07.513 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:12:07.513 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:12:07.513 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:07.513 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:07.513 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:07.513 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:12:07.514 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:07.514 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:12:07.514 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:12:07.514 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:07.514 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:12:07.514 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:07.515 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:12:07.515 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:12:07.515 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:07.515 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:07.515 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:12:07.515 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:07.515 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:12:07.516 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:12:07.516 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:12:07.516 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:07.516 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:12:07Z" 2023-07-29T01:12:07.516 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:12:07.517 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593122.086179_1622.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:12:02.086179Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:12:07Z" } 2023-07-29T01:12:07.518 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593122.086179_1622.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:12:02.086179Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:12:07.518 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593122.086179_1622.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:12:07.518 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593122.086179_1622.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:12:07.519 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T01:12:07.519 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:12:07.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:12:07.520 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:12:07.567 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:12:07.567 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:12:07.612 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-07-29T01:12:07.613 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:12:07.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:12:07.614 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:12:07.663 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:12:07.664 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:12:07.710 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2023-07-29T01:12:07.710 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hiiqos-70', '--dest-bucket=hiiqos-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:12:07.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hiiqos-70', '--dest-bucket=hiiqos-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:12:07.711 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1 --dest-zones=a2 --source-bucket=hiiqos-70 --dest-bucket=hiiqos-71 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:12:07.762 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:12:07.763 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:12:07.811 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hiiqos-70","zones":["a1"]},"dest":{"bucket":"hiiqos-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-07-29T01:12:07.812 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:12:07.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:12:07.813 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:12:07.858 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:12:07.858 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:12:07.903 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hiiqos-70","zones":["a1"]},"dest":{"bucket":"hiiqos-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-07-29T01:12:07.903 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:12:07.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:12:07.904 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:12:07.948 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:12:07.949 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:12:07.999 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:12:07.999 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:12:07.999 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 18, 2023-07-29T01:12:08.000 INFO:teuthology.orchestra.run.smithi101.stdout: "predecessor_uuid": "a0987484-ce37-4520-a564-54b7ea8c5251", 2023-07-29T01:12:08.000 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": [], 2023-07-29T01:12:08.000 INFO:teuthology.orchestra.run.smithi101.stdout: "period_map": { 2023-07-29T01:12:08.000 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:12:08.001 INFO:teuthology.orchestra.run.smithi101.stdout: "zonegroups": [ 2023-07-29T01:12:08.001 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:08.001 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "a", 2023-07-29T01:12:08.002 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a", 2023-07-29T01:12:08.002 INFO:teuthology.orchestra.run.smithi101.stdout: "api_name": "", 2023-07-29T01:12:08.002 INFO:teuthology.orchestra.run.smithi101.stdout: "is_master": true, 2023-07-29T01:12:08.003 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:12:08.003 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-29T01:12:08.003 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:12:08.004 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames": [], 2023-07-29T01:12:08.004 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames_s3website": [], 2023-07-29T01:12:08.004 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:12:08.004 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:12:08.005 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:08.005 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:12:08.005 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a1", 2023-07-29T01:12:08.006 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:12:08.006 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-29T01:12:08.006 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:12:08.006 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T01:12:08.007 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T01:12:08.007 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T01:12:08.007 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T01:12:08.007 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T01:12:08.008 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T01:12:08.009 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T01:12:08.009 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T01:12:08.010 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T01:12:08.010 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T01:12:08.010 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:12:08.010 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:12:08.011 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:08.011 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:08.011 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-29T01:12:08.012 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a2", 2023-07-29T01:12:08.012 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:12:08.012 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8001" 2023-07-29T01:12:08.012 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:12:08.013 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T01:12:08.013 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T01:12:08.013 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T01:12:08.014 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T01:12:08.014 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T01:12:08.014 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T01:12:08.014 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T01:12:08.015 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T01:12:08.015 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T01:12:08.015 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T01:12:08.015 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:12:08.016 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:12:08.016 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:08.016 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:12:08.016 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_targets": [ 2023-07-29T01:12:08.017 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:08.017 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default-placement", 2023-07-29T01:12:08.017 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [], 2023-07-29T01:12:08.017 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": [ 2023-07-29T01:12:08.017 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD" 2023-07-29T01:12:08.017 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:12:08.017 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:08.018 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:12:08.018 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "default-placement", 2023-07-29T01:12:08.018 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T01:12:08.018 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_policy": { 2023-07-29T01:12:08.018 INFO:teuthology.orchestra.run.smithi101.stdout: "groups": [ 2023-07-29T01:12:08.018 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:08.018 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "sync-group", 2023-07-29T01:12:08.019 INFO:teuthology.orchestra.run.smithi101.stdout: "data_flow": { 2023-07-29T01:12:08.019 INFO:teuthology.orchestra.run.smithi101.stdout: "symmetrical": [ 2023-07-29T01:12:08.019 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:08.019 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "sync-flow1", 2023-07-29T01:12:08.019 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:12:08.019 INFO:teuthology.orchestra.run.smithi101.stdout: "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:12:08.020 INFO:teuthology.orchestra.run.smithi101.stdout: "59d1bb30-3b1c-476f-9111-052c1ea0b711" 2023-07-29T01:12:08.020 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:12:08.020 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:08.020 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:12:08.020 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:08.021 INFO:teuthology.orchestra.run.smithi101.stdout: "pipes": [ 2023-07-29T01:12:08.021 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:08.021 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "sync-pipe", 2023-07-29T01:12:08.022 INFO:teuthology.orchestra.run.smithi101.stdout: "source": { 2023-07-29T01:12:08.022 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket": "hiiqos-70", 2023-07-29T01:12:08.022 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:12:08.022 INFO:teuthology.orchestra.run.smithi101.stdout: "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433" 2023-07-29T01:12:08.023 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:12:08.023 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:08.023 INFO:teuthology.orchestra.run.smithi101.stdout: "dest": { 2023-07-29T01:12:08.023 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket": "hiiqos-71", 2023-07-29T01:12:08.023 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:12:08.024 INFO:teuthology.orchestra.run.smithi101.stdout: "59d1bb30-3b1c-476f-9111-052c1ea0b711" 2023-07-29T01:12:08.024 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:12:08.024 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:08.024 INFO:teuthology.orchestra.run.smithi101.stdout: "params": { 2023-07-29T01:12:08.024 INFO:teuthology.orchestra.run.smithi101.stdout: "source": { 2023-07-29T01:12:08.024 INFO:teuthology.orchestra.run.smithi101.stdout: "filter": { 2023-07-29T01:12:08.024 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [] 2023-07-29T01:12:08.025 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:08.025 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:08.025 INFO:teuthology.orchestra.run.smithi101.stdout: "dest": {}, 2023-07-29T01:12:08.025 INFO:teuthology.orchestra.run.smithi101.stdout: "priority": 0, 2023-07-29T01:12:08.025 INFO:teuthology.orchestra.run.smithi101.stdout: "mode": "system", 2023-07-29T01:12:08.025 INFO:teuthology.orchestra.run.smithi101.stdout: "user": "" 2023-07-29T01:12:08.025 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:08.026 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:08.026 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:12:08.026 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "enabled" 2023-07-29T01:12:08.026 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:08.026 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:12:08.026 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:08.026 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled_features": [ 2023-07-29T01:12:08.027 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:12:08.027 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:12:08.027 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:08.027 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:08.027 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "b", 2023-07-29T01:12:08.027 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b", 2023-07-29T01:12:08.027 INFO:teuthology.orchestra.run.smithi101.stdout: "api_name": "", 2023-07-29T01:12:08.028 INFO:teuthology.orchestra.run.smithi101.stdout: "is_master": false, 2023-07-29T01:12:08.028 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:12:08.028 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-29T01:12:08.028 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:12:08.028 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames": [], 2023-07-29T01:12:08.028 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames_s3website": [], 2023-07-29T01:12:08.028 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T01:12:08.029 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:12:08.029 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:08.029 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T01:12:08.029 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b1", 2023-07-29T01:12:08.029 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:12:08.029 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-29T01:12:08.029 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:12:08.030 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T01:12:08.030 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T01:12:08.030 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T01:12:08.030 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T01:12:08.030 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T01:12:08.030 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T01:12:08.030 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T01:12:08.031 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T01:12:08.031 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T01:12:08.031 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T01:12:08.031 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:12:08.031 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:12:08.031 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:08.031 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:08.032 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-29T01:12:08.032 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b2", 2023-07-29T01:12:08.032 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:12:08.032 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8001" 2023-07-29T01:12:08.032 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:12:08.032 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T01:12:08.033 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T01:12:08.033 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T01:12:08.033 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T01:12:08.033 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T01:12:08.033 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T01:12:08.033 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T01:12:08.033 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T01:12:08.034 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T01:12:08.034 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T01:12:08.034 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:12:08.034 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:12:08.034 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:08.034 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:12:08.034 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_targets": [ 2023-07-29T01:12:08.035 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:08.035 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default-placement", 2023-07-29T01:12:08.035 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [], 2023-07-29T01:12:08.035 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": [ 2023-07-29T01:12:08.035 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD" 2023-07-29T01:12:08.035 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:12:08.035 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:08.036 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:12:08.036 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "default-placement", 2023-07-29T01:12:08.036 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T01:12:08.036 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_policy": { 2023-07-29T01:12:08.036 INFO:teuthology.orchestra.run.smithi101.stdout: "groups": [] 2023-07-29T01:12:08.036 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:08.036 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled_features": [ 2023-07-29T01:12:08.037 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:12:08.037 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:12:08.037 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:08.037 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:12:08.037 INFO:teuthology.orchestra.run.smithi101.stdout: "short_zone_ids": [ 2023-07-29T01:12:08.037 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:08.037 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:12:08.038 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 2896287231 2023-07-29T01:12:08.038 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:08.038 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:08.038 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-29T01:12:08.038 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 815606810 2023-07-29T01:12:08.038 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:08.038 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:08.039 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T01:12:08.039 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 2009171363 2023-07-29T01:12:08.039 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:08.039 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:08.040 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-29T01:12:08.040 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 3105190295 2023-07-29T01:12:08.040 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:08.040 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:12:08.041 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:08.041 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zonegroup": "a", 2023-07-29T01:12:08.041 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:12:08.041 INFO:teuthology.orchestra.run.smithi101.stdout: "period_config": { 2023-07-29T01:12:08.041 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_quota": { 2023-07-29T01:12:08.041 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-29T01:12:08.041 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-29T01:12:08.042 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-29T01:12:08.042 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-29T01:12:08.042 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-29T01:12:08.042 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:08.042 INFO:teuthology.orchestra.run.smithi101.stdout: "user_quota": { 2023-07-29T01:12:08.042 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-29T01:12:08.042 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-29T01:12:08.043 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-29T01:12:08.043 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-29T01:12:08.043 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-29T01:12:08.043 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:08.043 INFO:teuthology.orchestra.run.smithi101.stdout: "user_ratelimit": { 2023-07-29T01:12:08.043 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-29T01:12:08.044 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-29T01:12:08.044 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-29T01:12:08.044 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-29T01:12:08.044 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-29T01:12:08.044 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:08.044 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_ratelimit": { 2023-07-29T01:12:08.044 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-29T01:12:08.045 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-29T01:12:08.045 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-29T01:12:08.045 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-29T01:12:08.045 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-29T01:12:08.045 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:08.045 INFO:teuthology.orchestra.run.smithi101.stdout: "anonymous_ratelimit": { 2023-07-29T01:12:08.045 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-29T01:12:08.046 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-29T01:12:08.046 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-29T01:12:08.046 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-29T01:12:08.046 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-29T01:12:08.046 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:08.046 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:08.046 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T01:12:08.047 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_name": "test-realm", 2023-07-29T01:12:08.047 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:12:08.047 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:12:08.047 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:12:08.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:12:08.048 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:12:08.127 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:12:08.128 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:12:08.199 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hiiqos-70","zones":["a1"]},"dest":{"bucket":"hiiqos-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-07-29T01:12:08.199 DEBUG:boto:path=/hiiqos-70/ 2023-07-29T01:12:08.200 DEBUG:boto:auth_path=/hiiqos-70/ 2023-07-29T01:12:08.201 DEBUG:boto:Method: HEAD 2023-07-29T01:12:08.202 DEBUG:boto:Path: /hiiqos-70/ 2023-07-29T01:12:08.203 DEBUG:boto:Data: 2023-07-29T01:12:08.204 DEBUG:boto:Headers: {} 2023-07-29T01:12:08.204 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:12:08.205 DEBUG:boto:Port: 8000 2023-07-29T01:12:08.206 DEBUG:boto:Params: {} 2023-07-29T01:12:08.207 DEBUG:boto:Token: None 2023-07-29T01:12:08.208 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:12:08 GMT /hiiqos-70/ 2023-07-29T01:12:08.209 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hiTTRErs3WaZbtkcySMIVeEVlkA= 2023-07-29T01:12:08.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:12:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hiTTRErs3WaZbtkcySMIVeEVlkA=', 'Content-Length': '0'} 2023-07-29T01:12:08.286 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f2c2d617dfe3f615-0064c46768-2755-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:12:08.287 DEBUG:boto:path=/hiiqos-70/obj1 2023-07-29T01:12:08.288 DEBUG:boto:auth_path=/hiiqos-70/obj1 2023-07-29T01:12:08.288 DEBUG:boto:Method: PUT 2023-07-29T01:12:08.289 DEBUG:boto:Path: /hiiqos-70/obj1 2023-07-29T01:12:08.289 DEBUG:boto:Data: 2023-07-29T01:12:08.289 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T01:12:08.290 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:12:08.290 DEBUG:boto:Port: 8000 2023-07-29T01:12:08.291 DEBUG:boto:Params: {} 2023-07-29T01:12:08.292 DEBUG:boto:Token: None 2023-07-29T01:12:08.292 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 01:12:08 GMT /hiiqos-70/obj1 2023-07-29T01:12:08.293 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nCMNgFg1Ez4+/hzw5HM5WapddVw= 2023-07-29T01:12:08.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 01:12:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nCMNgFg1Ez4+/hzw5HM5WapddVw='} 2023-07-29T01:12:08.321 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000084a3a31513c336aa-0064c46768-2755-a1'), ('Date', 'Sat, 29 Jul 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:12:08.322 DEBUG:boto:path=/hiiqos-70/ 2023-07-29T01:12:08.322 DEBUG:boto:auth_path=/hiiqos-70/ 2023-07-29T01:12:08.322 DEBUG:boto:Method: HEAD 2023-07-29T01:12:08.323 DEBUG:boto:Path: /hiiqos-70/ 2023-07-29T01:12:08.323 DEBUG:boto:Data: 2023-07-29T01:12:08.324 DEBUG:boto:Headers: {} 2023-07-29T01:12:08.324 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:12:08.325 DEBUG:boto:Port: 8000 2023-07-29T01:12:08.325 DEBUG:boto:Params: {} 2023-07-29T01:12:08.325 DEBUG:boto:Token: None 2023-07-29T01:12:08.326 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:12:08 GMT /hiiqos-70/ 2023-07-29T01:12:08.327 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hiTTRErs3WaZbtkcySMIVeEVlkA= 2023-07-29T01:12:08.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:12:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hiTTRErs3WaZbtkcySMIVeEVlkA=', 'Content-Length': '0'} 2023-07-29T01:12:08.332 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000163f6b40e23af1b-0064c46768-2755-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:12:08.333 DEBUG:boto:path=/hiiqos-70/obj2 2023-07-29T01:12:08.333 DEBUG:boto:auth_path=/hiiqos-70/obj2 2023-07-29T01:12:08.333 DEBUG:boto:Method: PUT 2023-07-29T01:12:08.334 DEBUG:boto:Path: /hiiqos-70/obj2 2023-07-29T01:12:08.334 DEBUG:boto:Data: 2023-07-29T01:12:08.334 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T01:12:08.335 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:12:08.335 DEBUG:boto:Port: 8000 2023-07-29T01:12:08.335 DEBUG:boto:Params: {} 2023-07-29T01:12:08.336 DEBUG:boto:Token: None 2023-07-29T01:12:08.336 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 01:12:08 GMT /hiiqos-70/obj2 2023-07-29T01:12:08.337 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:vGgdub2MXvGVVWJeJ7sUTCCzs+U= 2023-07-29T01:12:08.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 01:12:08 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vGgdub2MXvGVVWJeJ7sUTCCzs+U='} 2023-07-29T01:12:08.380 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000049e511ef1465ce7a-0064c46768-2755-a1'), ('Date', 'Sat, 29 Jul 2023 01:12:08 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:12:08.381 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:12:08.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:12:08.382 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:12:08.495 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:12:08.496 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:12:08.570 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T01:12:08.606 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T01:12:08.606 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:08.606 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:12:08.606 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:17.836402Z" 2023-07-29T01:12:08.607 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:08.607 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:08.607 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593122.086179_1622.1", 2023-07-29T01:12:08.607 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:12:02.086179Z" 2023-07-29T01:12:08.607 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:08.608 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:08.608 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:12:08.608 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T01:12:08.608 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:08.608 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:08.608 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:12:08.609 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:12.811406Z" 2023-07-29T01:12:08.609 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:08.609 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T01:12:08.610 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690591217.836402_294.1', 1: '1_1690593122.086179_1622.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:12:08.610 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T01:12:08.611 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:12:08.611 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:12:08.611 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:12:08.697 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:12:08.698 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:12:08.739 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:12:08.650+0000 7fc227c10880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:12:08.740 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:12:08.651+0000 7fc227c10880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:12:08.758 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:12:08.758 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:12:08.758 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:12:08.758 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:12:08.758 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:12:08.759 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:12:08.759 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:12:08.759 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:08.759 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:12:08.759 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:08.759 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:12:08.759 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:08.760 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:12:08.760 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:12:08.760 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:08.760 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:12:08.760 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:08.760 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:12:08.761 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:12:08.761 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:08.761 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:08.761 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:08.761 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:12:08.761 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:08.761 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:12:08.762 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593122.086179_1622.1", 2023-07-29T01:12:08.762 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:08.762 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:12:08.762 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:08.762 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:12:02.086179Z", 2023-07-29T01:12:08.762 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:12:08.762 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:08.763 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:08.763 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:08.763 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:12:08.763 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:08.763 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:12:08.763 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:12:08.763 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:08.764 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:12:08.764 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:08.764 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:12:08.764 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:12:08.764 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:08.764 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:08.764 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:08.765 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:12:08.765 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:08.765 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:12:08.765 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:12:08.765 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:08.765 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:12:08.765 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:08.766 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:12:08.766 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:12:08.766 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:08.766 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:08.766 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:12:08.766 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:08.766 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:12:08.767 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:12:08.767 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:12:08.767 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:08.767 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:12:08Z" 2023-07-29T01:12:08.767 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:12:08.768 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593122.086179_1622.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:12:02.086179Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:12:08Z" } 2023-07-29T01:12:08.769 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593122.086179_1622.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:12:02.086179Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:12:08.769 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593122.086179_1622.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:12:08.769 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593122.086179_1622.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:12:08.770 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T01:12:08.770 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:12:08.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:12:08.771 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:12:08.825 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:12:08.826 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:12:08.877 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T01:12:08.877 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:08.877 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052093", 2023-07-29T01:12:08.877 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:12:08.229055Z" 2023-07-29T01:12:08.878 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:08.878 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:08.878 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044881", 2023-07-29T01:12:08.878 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:07:23.001407Z" 2023-07-29T01:12:08.878 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:08.878 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:08.879 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045800", 2023-07-29T01:12:08.879 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:12:08.266555Z" 2023-07-29T01:12:08.879 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:08.879 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:08.879 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:12:08.879 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:12:08.879 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:08.880 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T01:12:08.881 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000052093', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045800', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:12:08.881 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-07-29T01:12:08.882 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:12:08.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:12:08.883 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:12:08.930 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:12:08.931 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:12:08.973 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:12:08.974 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:12:08.974 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:12:08.974 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:12:08.975 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:12:08.975 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:12:08.975 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:08.975 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:12:08.975 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:08.975 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:12:08.976 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:08.976 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:12:08.976 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051953", 2023-07-29T01:12:08.976 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:08.976 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:12:08.976 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:08.976 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:09:37.925040Z" 2023-07-29T01:12:08.977 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:08.977 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:08.977 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:08.977 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:12:08.977 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:08.977 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:12:08.978 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044881", 2023-07-29T01:12:08.978 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:08.978 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:12:08.978 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:08.979 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:23.001407Z" 2023-07-29T01:12:08.979 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:08.979 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:08.979 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:08.980 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:12:08.980 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:08.980 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:12:08.981 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045660", 2023-07-29T01:12:08.981 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:08.981 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:12:08.982 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:08.982 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:22.902962Z" 2023-07-29T01:12:08.983 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:08.983 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:08.983 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:08.984 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:12:08.984 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:08.984 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:12:08.985 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:12:08.985 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:08.985 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:12:08.986 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:08.986 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:12:08.986 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:08.986 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:08.986 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:12:08.987 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:08.987 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:12:08.987 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:12:08.988 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:12:08.988 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:08.988 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:12:08Z" 2023-07-29T01:12:08.989 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:12:08.990 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051953", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:09:37.925040Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044881", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:07:23.001407Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045660", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:22.902962Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:12:08Z" } 2023-07-29T01:12:08.991 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051953', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:09:37.925040Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044881', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:07:23.001407Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045660', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:22.902962Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:12:08.991 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052093', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045800', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:12:08.992 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051953', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:12:08.992 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052093 target=00000000000000000000:00000000000000051953, shard=2 master=00000000000000000000:00000000000000045800 target=00000000000000000000:00000000000000045660 2023-07-29T01:12:13.994 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:12:13.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:12:13.995 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:12:14.049 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:12:14.049 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:12:14.096 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:12:14.097 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:12:14.097 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:12:14.097 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:12:14.097 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:12:14.097 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:12:14.097 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:14.097 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:12:14.098 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:14.098 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:12:14.098 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:14.098 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:12:14.098 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051953", 2023-07-29T01:12:14.098 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:14.099 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:12:14.099 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:14.099 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:09:37.925040Z" 2023-07-29T01:12:14.099 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:14.099 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:14.099 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:14.099 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:12:14.100 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:14.100 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:12:14.100 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044881", 2023-07-29T01:12:14.100 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:14.100 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:12:14.100 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:14.101 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:23.001407Z" 2023-07-29T01:12:14.101 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:14.101 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:14.101 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:14.101 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:12:14.101 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:14.101 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:12:14.102 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045660", 2023-07-29T01:12:14.102 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:14.102 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:12:14.102 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:14.102 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:22.902962Z" 2023-07-29T01:12:14.102 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:14.103 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:14.103 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:14.103 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:12:14.103 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:14.103 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:12:14.103 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:12:14.103 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:14.104 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:12:14.104 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:14.104 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:12:14.104 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:14.104 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:14.104 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:12:14.105 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:14.105 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:12:14.105 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:12:14.105 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:12:14.105 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:14.105 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:12:13Z" 2023-07-29T01:12:14.105 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:12:14.106 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051953", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:09:37.925040Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044881", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:07:23.001407Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045660", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:22.902962Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:12:13Z" } 2023-07-29T01:12:14.107 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051953', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:09:37.925040Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044881', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:07:23.001407Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045660', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:22.902962Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:12:14.107 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052093', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045800', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:12:14.108 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051953', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:12:14.108 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052093 target=00000000000000000000:00000000000000051953, shard=2 master=00000000000000000000:00000000000000045800 target=00000000000000000000:00000000000000045660 2023-07-29T01:12:19.109 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:12:19.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:12:19.110 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:12:19.164 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:12:19.164 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:12:19.211 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:12:19.212 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:12:19.212 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:12:19.212 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:12:19.212 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:12:19.212 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:12:19.213 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:19.213 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:12:19.213 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:19.213 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:12:19.214 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:19.214 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:12:19.214 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051953", 2023-07-29T01:12:19.214 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:19.214 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:12:19.214 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:19.215 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:09:37.925040Z" 2023-07-29T01:12:19.215 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:19.215 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:19.215 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:19.215 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:12:19.215 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:19.215 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:12:19.216 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044881", 2023-07-29T01:12:19.216 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:19.216 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:12:19.216 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:19.216 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:23.001407Z" 2023-07-29T01:12:19.216 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:19.217 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:19.217 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:19.217 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:12:19.217 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:19.217 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:12:19.217 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045660", 2023-07-29T01:12:19.217 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:19.218 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:12:19.218 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:19.218 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:22.902962Z" 2023-07-29T01:12:19.218 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:19.218 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:19.218 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:19.218 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:12:19.219 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:19.219 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:12:19.219 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:12:19.219 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:19.219 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:12:19.220 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:19.220 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:12:19.220 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:19.220 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:19.220 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:12:19.221 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:19.221 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:12:19.221 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:12:19.221 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:12:19.221 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:19.221 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:12:19Z" 2023-07-29T01:12:19.222 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:12:19.223 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051953", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:09:37.925040Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044881", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:07:23.001407Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045660", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:22.902962Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:12:19Z" } 2023-07-29T01:12:19.223 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051953', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:09:37.925040Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044881', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:07:23.001407Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045660', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:22.902962Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:12:19.223 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052093', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045800', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:12:19.224 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051953', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:12:19.224 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052093 target=00000000000000000000:00000000000000051953, shard=2 master=00000000000000000000:00000000000000045800 target=00000000000000000000:00000000000000045660 2023-07-29T01:12:24.225 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:12:24.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:12:24.226 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:12:24.283 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:12:24.284 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:12:24.330 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:12:24.331 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:12:24.331 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:12:24.331 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:12:24.331 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:12:24.331 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:12:24.332 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:24.332 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:12:24.332 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:24.332 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:12:24.332 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:24.333 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:12:24.333 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051953", 2023-07-29T01:12:24.333 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:24.333 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:12:24.333 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:24.334 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:09:37.925040Z" 2023-07-29T01:12:24.334 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:24.334 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:24.334 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:24.334 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:12:24.334 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:24.334 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:12:24.335 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044881", 2023-07-29T01:12:24.335 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:24.335 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:12:24.335 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:24.335 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:23.001407Z" 2023-07-29T01:12:24.335 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:24.335 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:24.336 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:24.336 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:12:24.336 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:24.336 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:12:24.336 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045660", 2023-07-29T01:12:24.336 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:24.336 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:12:24.337 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:24.337 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:22.902962Z" 2023-07-29T01:12:24.337 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:24.337 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:24.337 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:24.337 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:12:24.337 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:24.338 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:12:24.338 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:12:24.338 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:24.338 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:12:24.338 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:24.338 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:12:24.338 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:24.339 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:24.339 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:12:24.339 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:24.339 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:12:24.339 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:12:24.339 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:12:24.339 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:24.340 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:12:24Z" 2023-07-29T01:12:24.340 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:12:24.341 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051953", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:09:37.925040Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044881", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:07:23.001407Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045660", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:22.902962Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:12:24Z" } 2023-07-29T01:12:24.341 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051953', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:09:37.925040Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044881', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:07:23.001407Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045660', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:22.902962Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:12:24.342 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052093', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045800', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:12:24.342 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051953', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:12:24.342 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052093 target=00000000000000000000:00000000000000051953, shard=2 master=00000000000000000000:00000000000000045800 target=00000000000000000000:00000000000000045660 2023-07-29T01:12:29.344 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:12:29.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:12:29.345 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:12:29.399 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:12:29.400 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:12:29.448 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:12:29.448 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:12:29.448 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:12:29.449 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:12:29.449 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:12:29.449 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:12:29.449 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:29.449 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:12:29.449 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:29.450 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:12:29.450 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:29.450 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:12:29.450 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051953", 2023-07-29T01:12:29.451 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:29.451 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:12:29.451 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:29.451 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:09:37.925040Z" 2023-07-29T01:12:29.451 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:29.452 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:29.452 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:29.452 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:12:29.452 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:29.452 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:12:29.452 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044881", 2023-07-29T01:12:29.453 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:29.453 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:12:29.453 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:29.453 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:23.001407Z" 2023-07-29T01:12:29.453 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:29.453 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:29.454 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:29.454 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:12:29.454 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:29.454 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:12:29.454 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045660", 2023-07-29T01:12:29.454 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:29.455 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:12:29.455 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:29.455 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:22.902962Z" 2023-07-29T01:12:29.455 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:29.455 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:29.455 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:29.456 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:12:29.456 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:29.456 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:12:29.456 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:12:29.457 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:29.457 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:12:29.457 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:29.457 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:12:29.457 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:29.457 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:29.458 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:12:29.458 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:29.458 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:12:29.458 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:12:29.458 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:12:29.458 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:29.458 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:12:29Z" 2023-07-29T01:12:29.459 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:12:29.460 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051953", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:09:37.925040Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044881", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:07:23.001407Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045660", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:22.902962Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:12:29Z" } 2023-07-29T01:12:29.461 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051953', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:09:37.925040Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044881', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:07:23.001407Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045660', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:22.902962Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:12:29.462 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052093', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045800', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:12:29.463 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051953', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:12:29.464 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052093 target=00000000000000000000:00000000000000051953, shard=2 master=00000000000000000000:00000000000000045800 target=00000000000000000000:00000000000000045660 2023-07-29T01:12:34.465 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:12:34.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:12:34.466 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:12:34.521 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:12:34.522 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:12:34.568 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:12:34.568 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:12:34.569 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:12:34.569 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:12:34.569 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:12:34.569 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:12:34.569 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:34.569 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:12:34.569 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:34.570 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:12:34.570 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:34.570 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:12:34.570 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051953", 2023-07-29T01:12:34.570 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:34.570 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:12:34.570 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:34.571 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:09:37.925040Z" 2023-07-29T01:12:34.571 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:34.571 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:34.571 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:34.571 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:12:34.571 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:34.572 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:12:34.572 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044881", 2023-07-29T01:12:34.572 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:34.572 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:12:34.572 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:34.572 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:23.001407Z" 2023-07-29T01:12:34.572 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:34.573 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:34.573 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:34.573 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:12:34.573 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:34.573 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:12:34.573 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045660", 2023-07-29T01:12:34.573 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:34.574 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:12:34.574 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:34.574 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:22.902962Z" 2023-07-29T01:12:34.574 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:34.574 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:34.574 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:34.574 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:12:34.575 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:34.575 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:12:34.575 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:12:34.575 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:34.575 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:12:34.575 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:34.575 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:12:34.576 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:34.576 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:34.576 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:12:34.576 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:34.576 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:12:34.576 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:12:34.576 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:12:34.577 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:34.577 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:12:34Z" 2023-07-29T01:12:34.577 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:12:34.578 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051953", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:09:37.925040Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044881", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:07:23.001407Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045660", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:22.902962Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:12:34Z" } 2023-07-29T01:12:34.578 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051953', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:09:37.925040Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044881', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:07:23.001407Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045660', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:22.902962Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:12:34.579 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052093', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045800', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:12:34.579 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051953', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:12:34.579 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052093 target=00000000000000000000:00000000000000051953, shard=2 master=00000000000000000000:00000000000000045800 target=00000000000000000000:00000000000000045660 2023-07-29T01:12:39.580 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:12:39.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:12:39.581 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:12:39.636 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:12:39.636 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:12:39.683 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:12:39.683 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:12:39.683 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:12:39.684 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:12:39.684 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:12:39.684 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:12:39.684 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:39.684 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:12:39.685 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:39.685 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:12:39.685 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:39.685 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:12:39.685 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051953", 2023-07-29T01:12:39.685 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:39.685 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:12:39.686 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:39.686 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:09:37.925040Z" 2023-07-29T01:12:39.686 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:39.686 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:39.686 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:39.686 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:12:39.686 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:39.687 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:12:39.687 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044881", 2023-07-29T01:12:39.687 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:39.687 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:12:39.687 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:39.687 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:23.001407Z" 2023-07-29T01:12:39.687 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:39.688 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:39.688 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:39.688 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:12:39.688 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:39.688 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:12:39.688 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045660", 2023-07-29T01:12:39.689 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:39.689 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:12:39.689 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:39.689 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:22.902962Z" 2023-07-29T01:12:39.689 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:39.689 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:39.689 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:39.690 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:12:39.690 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:39.690 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:12:39.690 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:12:39.690 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:39.690 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:12:39.690 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:39.691 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:12:39.691 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:39.692 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:39.692 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:12:39.692 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:39.693 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:12:39.693 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:12:39.693 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:12:39.693 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:39.693 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:12:39Z" 2023-07-29T01:12:39.693 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:12:39.694 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051953", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:09:37.925040Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044881", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:07:23.001407Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045660", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:22.902962Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:12:39Z" } 2023-07-29T01:12:39.695 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051953', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:09:37.925040Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044881', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:07:23.001407Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045660', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:22.902962Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:12:39.695 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052093', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045800', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:12:39.695 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051953', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:12:39.696 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052093 target=00000000000000000000:00000000000000051953, shard=2 master=00000000000000000000:00000000000000045800 target=00000000000000000000:00000000000000045660 2023-07-29T01:12:44.697 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:12:44.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:12:44.698 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:12:44.753 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:12:44.753 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:12:44.801 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:12:44.801 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:12:44.801 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:12:44.802 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:12:44.802 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:12:44.802 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:12:44.803 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:44.803 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:12:44.803 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:44.804 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:12:44.804 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:44.804 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:12:44.805 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051953", 2023-07-29T01:12:44.805 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:44.805 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:12:44.805 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:44.806 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:09:37.925040Z" 2023-07-29T01:12:44.806 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:44.806 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:44.806 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:44.807 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:12:44.807 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:44.807 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:12:44.807 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044881", 2023-07-29T01:12:44.808 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:44.808 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:12:44.808 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:44.809 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:23.001407Z" 2023-07-29T01:12:44.809 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:44.809 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:44.809 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:44.810 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:12:44.810 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:44.810 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:12:44.810 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045660", 2023-07-29T01:12:44.811 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:44.811 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:12:44.811 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:44.811 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:22.902962Z" 2023-07-29T01:12:44.812 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:44.812 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:44.812 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:44.813 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:12:44.813 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:44.813 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:12:44.814 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:12:44.814 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:44.814 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:12:44.814 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:44.815 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:12:44.815 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:44.815 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:44.816 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:12:44.816 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:44.816 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:12:44.817 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:12:44.817 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:12:44.817 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:44.818 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:12:44Z" 2023-07-29T01:12:44.818 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:12:44.819 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051953", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:09:37.925040Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044881", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:07:23.001407Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045660", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:22.902962Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:12:44Z" } 2023-07-29T01:12:44.820 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051953', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:09:37.925040Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044881', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:07:23.001407Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045660', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:22.902962Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:12:44.820 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052093', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045800', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:12:44.821 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051953', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:12:44.821 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052093 target=00000000000000000000:00000000000000051953, shard=2 master=00000000000000000000:00000000000000045800 target=00000000000000000000:00000000000000045660 2023-07-29T01:12:49.823 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:12:49.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:12:49.824 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:12:49.879 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:12:49.879 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:12:49.927 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:12:49.927 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:12:49.927 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:12:49.928 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:12:49.928 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:12:49.928 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:12:49.928 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:49.928 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:12:49.928 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:49.929 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:12:49.929 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:49.929 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:12:49.929 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051953", 2023-07-29T01:12:49.929 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:49.929 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:12:49.930 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:49.930 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:09:37.925040Z" 2023-07-29T01:12:49.930 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:49.930 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:49.930 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:49.930 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:12:49.931 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:49.931 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:12:49.931 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044881", 2023-07-29T01:12:49.931 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:49.931 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:12:49.931 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:49.931 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:23.001407Z" 2023-07-29T01:12:49.932 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:49.932 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:49.932 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:49.932 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:12:49.932 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:49.932 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:12:49.933 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045660", 2023-07-29T01:12:49.933 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:49.933 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:12:49.933 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:49.933 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:22.902962Z" 2023-07-29T01:12:49.933 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:49.934 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:49.934 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:49.934 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:12:49.934 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:49.934 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:12:49.934 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:12:49.935 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:49.935 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:12:49.935 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:49.935 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:12:49.935 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:49.935 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:49.935 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:12:49.936 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:49.936 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:12:49.936 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:12:49.936 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:12:49.936 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:49.936 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:12:49Z" 2023-07-29T01:12:49.937 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:12:49.937 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051953", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:09:37.925040Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044881", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:07:23.001407Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045660", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:22.902962Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:12:49Z" } 2023-07-29T01:12:49.938 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051953', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:09:37.925040Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044881', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:07:23.001407Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045660', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:22.902962Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:12:49.938 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052093', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045800', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:12:49.939 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051953', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:12:49.939 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052093 target=00000000000000000000:00000000000000051953, shard=2 master=00000000000000000000:00000000000000045800 target=00000000000000000000:00000000000000045660 2023-07-29T01:12:54.940 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:12:54.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:12:54.941 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:12:54.994 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:12:54.994 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:12:55.042 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:12:55.042 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:12:55.043 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:12:55.043 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:12:55.043 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:12:55.043 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:12:55.043 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:55.043 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:12:55.043 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:55.044 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:12:55.044 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:55.044 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:12:55.044 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051953", 2023-07-29T01:12:55.044 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:55.044 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:12:55.045 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:55.045 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:09:37.925040Z" 2023-07-29T01:12:55.045 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:55.045 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:55.045 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:55.045 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:12:55.046 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:55.046 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:12:55.046 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044881", 2023-07-29T01:12:55.046 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:55.046 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:12:55.046 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:55.046 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:23.001407Z" 2023-07-29T01:12:55.047 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:55.047 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:55.047 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:55.047 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:12:55.047 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:55.047 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:12:55.047 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045660", 2023-07-29T01:12:55.048 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:55.048 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:12:55.048 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:55.048 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:22.902962Z" 2023-07-29T01:12:55.048 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:55.048 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:55.048 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:12:55.049 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:12:55.049 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:12:55.049 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:12:55.049 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:12:55.049 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:12:55.049 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:12:55.049 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:12:55.050 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:12:55.050 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:55.050 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:12:55.050 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:12:55.050 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:55.050 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:12:55.050 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:12:55.051 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:12:55.051 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:12:55.051 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:12:54Z" 2023-07-29T01:12:55.051 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:12:55.052 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051953", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:09:37.925040Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044881", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:07:23.001407Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045660", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:22.902962Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:12:54Z" } 2023-07-29T01:12:55.052 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051953', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:09:37.925040Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044881', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:07:23.001407Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045660', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:22.902962Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:12:55.053 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052093', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045800', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:12:55.053 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051953', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:12:55.054 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052093 target=00000000000000000000:00000000000000051953, shard=2 master=00000000000000000000:00000000000000045800 target=00000000000000000000:00000000000000045660 2023-07-29T01:13:00.055 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:13:00.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:13:00.056 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:13:00.111 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:13:00.111 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:13:00.160 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:13:00.161 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:13:00.161 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:13:00.162 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:13:00.162 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:13:00.163 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:13:00.163 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:00.163 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:13:00.164 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:00.164 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:13:00.164 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:00.165 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:00.165 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051953", 2023-07-29T01:13:00.165 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:00.166 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:13:00.166 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:00.166 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:09:37.925040Z" 2023-07-29T01:13:00.166 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:00.167 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:00.167 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:00.167 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:13:00.167 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:00.168 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:00.168 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044881", 2023-07-29T01:13:00.169 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:00.169 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:13:00.169 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:00.170 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:23.001407Z" 2023-07-29T01:13:00.170 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:00.170 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:00.170 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:00.170 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:13:00.170 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:00.171 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:00.171 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045660", 2023-07-29T01:13:00.171 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:00.171 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:13:00.171 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:00.171 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:22.902962Z" 2023-07-29T01:13:00.171 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:00.172 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:00.172 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:00.172 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:13:00.172 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:00.172 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:00.172 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:13:00.172 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:00.173 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:13:00.173 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:00.173 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:13:00.173 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:00.173 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:00.173 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:13:00.173 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:00.174 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:13:00.174 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:13:00.174 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:13:00.174 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:00.174 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:13:00Z" 2023-07-29T01:13:00.174 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:13:00.175 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051953", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:09:37.925040Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044881", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:07:23.001407Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045660", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:22.902962Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:13:00Z" } 2023-07-29T01:13:00.176 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051953', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:09:37.925040Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044881', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:07:23.001407Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045660', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:22.902962Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:13:00.176 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052093', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045800', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:13:00.177 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051953', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:13:00.177 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052093 target=00000000000000000000:00000000000000051953, shard=2 master=00000000000000000000:00000000000000045800 target=00000000000000000000:00000000000000045660 2023-07-29T01:13:05.179 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:13:05.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:13:05.180 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:13:05.235 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:13:05.236 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:13:05.283 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:13:05.283 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:13:05.284 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:13:05.284 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:13:05.284 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:13:05.284 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:13:05.284 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:05.285 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:13:05.285 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:05.285 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:13:05.285 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:05.285 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:05.285 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051953", 2023-07-29T01:13:05.285 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:05.286 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:13:05.286 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:05.286 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:09:37.925040Z" 2023-07-29T01:13:05.286 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:05.286 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:05.286 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:05.286 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:13:05.287 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:05.287 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:05.287 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044881", 2023-07-29T01:13:05.287 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:05.287 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:13:05.287 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:05.287 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:23.001407Z" 2023-07-29T01:13:05.288 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:05.288 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:05.288 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:05.288 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:13:05.288 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:05.288 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:05.288 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045660", 2023-07-29T01:13:05.289 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:05.289 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:13:05.289 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:05.289 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:22.902962Z" 2023-07-29T01:13:05.289 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:05.289 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:05.290 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:05.290 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:13:05.290 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:05.290 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:05.290 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:13:05.290 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:05.290 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:13:05.291 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:05.291 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:13:05.291 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:05.291 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:05.291 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:13:05.291 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:05.291 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:13:05.292 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:13:05.292 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:13:05.292 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:05.292 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:13:05Z" 2023-07-29T01:13:05.292 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:13:05.293 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051953", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:09:37.925040Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044881", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:07:23.001407Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045660", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:22.902962Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:13:05Z" } 2023-07-29T01:13:05.294 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051953', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:09:37.925040Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044881', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:07:23.001407Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045660', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:22.902962Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:13:05.294 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052093', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045800', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:13:05.295 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051953', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:13:05.295 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052093 target=00000000000000000000:00000000000000051953, shard=2 master=00000000000000000000:00000000000000045800 target=00000000000000000000:00000000000000045660 2023-07-29T01:13:10.296 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:13:10.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:13:10.297 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:13:10.353 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:13:10.353 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:13:10.400 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:13:10.401 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:13:10.401 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:13:10.401 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:13:10.401 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:13:10.401 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:13:10.402 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:10.402 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:13:10.402 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:10.402 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:13:10.402 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:10.402 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:10.403 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051953", 2023-07-29T01:13:10.403 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:10.403 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:13:10.403 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:10.403 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:09:37.925040Z" 2023-07-29T01:13:10.403 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:10.403 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:10.404 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:10.404 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:13:10.404 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:10.404 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:10.404 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044881", 2023-07-29T01:13:10.404 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:10.405 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:13:10.405 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:10.405 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:23.001407Z" 2023-07-29T01:13:10.405 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:10.405 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:10.405 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:10.405 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:13:10.406 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:10.406 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:10.406 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045660", 2023-07-29T01:13:10.406 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:10.406 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:13:10.406 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:10.406 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:22.902962Z" 2023-07-29T01:13:10.407 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:10.407 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:10.407 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:10.407 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:13:10.407 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:10.407 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:10.407 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:13:10.408 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:10.408 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:13:10.408 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:10.408 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:13:10.408 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:10.408 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:10.408 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:13:10.409 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:10.409 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:13:10.409 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:13:10.409 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:13:10.409 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:10.409 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:13:10Z" 2023-07-29T01:13:10.409 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:13:10.410 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051953", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:09:37.925040Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044881", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:07:23.001407Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045660", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:22.902962Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:13:10Z" } 2023-07-29T01:13:10.411 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051953', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:09:37.925040Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044881', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:07:23.001407Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045660', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:22.902962Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:13:10.411 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052093', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045800', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:13:10.412 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051953', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:13:10.412 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052093 target=00000000000000000000:00000000000000051953, shard=2 master=00000000000000000000:00000000000000045800 target=00000000000000000000:00000000000000045660 2023-07-29T01:13:15.413 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:13:15.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:13:15.414 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:13:15.472 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:13:15.472 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:13:15.519 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:13:15.520 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:13:15.520 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:13:15.520 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:13:15.520 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:13:15.520 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:13:15.521 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:15.521 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:13:15.521 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:15.522 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:13:15.522 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:15.522 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:15.522 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051953", 2023-07-29T01:13:15.522 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:15.522 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:13:15.523 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:15.523 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:09:37.925040Z" 2023-07-29T01:13:15.523 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:15.523 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:15.523 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:15.523 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:13:15.524 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:15.524 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:15.524 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044881", 2023-07-29T01:13:15.524 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:15.524 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:13:15.524 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:15.524 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:23.001407Z" 2023-07-29T01:13:15.525 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:15.525 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:15.525 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:15.525 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:13:15.525 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:15.525 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:15.525 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045660", 2023-07-29T01:13:15.526 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:15.526 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:13:15.526 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:15.526 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:22.902962Z" 2023-07-29T01:13:15.526 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:15.526 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:15.526 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:15.527 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:13:15.527 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:15.527 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:15.527 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:13:15.527 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:15.527 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:13:15.527 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:15.528 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:13:15.528 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:15.528 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:15.528 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:13:15.529 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:15.529 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:13:15.529 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:13:15.529 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:13:15.529 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:15.529 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:13:15Z" 2023-07-29T01:13:15.529 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:13:15.530 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051953", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:09:37.925040Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044881", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:07:23.001407Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045660", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:22.902962Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:13:15Z" } 2023-07-29T01:13:15.531 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051953', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:09:37.925040Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044881', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:07:23.001407Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045660', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:22.902962Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:13:15.531 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052093', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045800', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:13:15.532 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051953', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:13:15.532 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052093 target=00000000000000000000:00000000000000051953, shard=2 master=00000000000000000000:00000000000000045800 target=00000000000000000000:00000000000000045660 2023-07-29T01:13:20.533 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:13:20.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:13:20.535 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:13:20.590 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:13:20.590 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:13:20.640 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:13:20.640 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:13:20.640 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:13:20.640 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:13:20.640 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:13:20.641 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:13:20.641 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:20.641 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:13:20.641 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:20.641 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:13:20.641 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:20.641 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:20.642 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051953", 2023-07-29T01:13:20.642 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:20.642 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:13:20.642 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:20.642 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:09:37.925040Z" 2023-07-29T01:13:20.642 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:20.643 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:20.643 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:20.643 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:13:20.643 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:20.643 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:20.643 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044881", 2023-07-29T01:13:20.643 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:20.644 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:13:20.644 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:20.644 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:23.001407Z" 2023-07-29T01:13:20.644 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:20.644 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:20.644 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:20.644 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:13:20.645 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:20.645 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:20.645 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045660", 2023-07-29T01:13:20.645 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:20.645 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:13:20.645 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:20.646 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:22.902962Z" 2023-07-29T01:13:20.646 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:20.646 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:20.646 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:20.646 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:13:20.646 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:20.646 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:20.647 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:13:20.647 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:20.647 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:13:20.647 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:20.647 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:13:20.647 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:20.647 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:20.648 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:13:20.648 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:20.648 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:13:20.648 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:13:20.648 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:13:20.648 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:20.649 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:13:20Z" 2023-07-29T01:13:20.649 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:13:20.650 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051953", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:09:37.925040Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044881", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:07:23.001407Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045660", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:22.902962Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:13:20Z" } 2023-07-29T01:13:20.650 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051953', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:09:37.925040Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044881', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:07:23.001407Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045660', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:22.902962Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:13:20.651 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052093', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045800', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:13:20.651 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051953', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:13:20.651 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052093 target=00000000000000000000:00000000000000051953, shard=2 master=00000000000000000000:00000000000000045800 target=00000000000000000000:00000000000000045660 2023-07-29T01:13:25.653 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:13:25.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:13:25.654 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:13:25.709 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:13:25.710 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:13:25.757 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:13:25.758 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:13:25.758 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:13:25.758 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:13:25.758 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:13:25.758 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:13:25.758 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:25.758 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:13:25.759 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:25.759 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:13:25.759 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:25.759 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:25.759 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051953", 2023-07-29T01:13:25.759 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:25.760 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:13:25.760 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:25.760 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:09:37.925040Z" 2023-07-29T01:13:25.760 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:25.760 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:25.760 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:25.760 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:13:25.761 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:25.761 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:25.761 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044881", 2023-07-29T01:13:25.761 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:25.761 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:13:25.761 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:25.761 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:23.001407Z" 2023-07-29T01:13:25.762 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:25.762 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:25.762 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:25.762 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:13:25.762 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:25.762 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:25.762 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045660", 2023-07-29T01:13:25.763 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:25.763 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:13:25.763 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:25.763 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:22.902962Z" 2023-07-29T01:13:25.763 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:25.763 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:25.764 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:25.764 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:13:25.764 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:25.764 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:25.764 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:13:25.764 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:25.764 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:13:25.765 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:25.765 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:13:25.765 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:25.765 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:25.765 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:13:25.765 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:25.765 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:13:25.766 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:13:25.766 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:13:25.766 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:25.766 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:13:25Z" 2023-07-29T01:13:25.766 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:13:25.767 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051953", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:09:37.925040Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044881", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:07:23.001407Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045660", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:22.902962Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:13:25Z" } 2023-07-29T01:13:25.768 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051953', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:09:37.925040Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044881', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:07:23.001407Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045660', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:22.902962Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:13:25.768 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052093', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045800', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:13:25.768 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051953', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:13:25.769 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052093 target=00000000000000000000:00000000000000051953, shard=2 master=00000000000000000000:00000000000000045800 target=00000000000000000000:00000000000000045660 2023-07-29T01:13:30.770 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:13:30.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:13:30.771 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:13:30.826 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:13:30.827 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:13:30.874 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:13:30.874 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:13:30.875 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:13:30.875 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:13:30.875 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:13:30.875 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:13:30.875 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:30.875 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:13:30.876 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:30.876 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:13:30.876 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:30.876 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:30.876 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051953", 2023-07-29T01:13:30.876 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:30.877 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:13:30.877 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:30.877 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:09:37.925040Z" 2023-07-29T01:13:30.877 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:30.877 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:30.877 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:30.878 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:13:30.878 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:30.878 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:30.878 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044881", 2023-07-29T01:13:30.878 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:30.878 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:13:30.878 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:30.879 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:23.001407Z" 2023-07-29T01:13:30.879 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:30.879 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:30.879 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:30.880 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:13:30.880 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:30.880 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:30.880 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045660", 2023-07-29T01:13:30.881 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:30.881 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:13:30.881 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:30.881 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:22.902962Z" 2023-07-29T01:13:30.881 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:30.882 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:30.882 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:30.882 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:13:30.883 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:30.883 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:30.883 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:13:30.883 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:30.883 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:13:30.884 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:30.884 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:13:30.884 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:30.884 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:30.884 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:13:30.884 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:30.885 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:13:30.885 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:13:30.885 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:13:30.886 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:30.886 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:13:30Z" 2023-07-29T01:13:30.886 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:13:30.888 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051953", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:09:37.925040Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044881", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:07:23.001407Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045660", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:22.902962Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:13:30Z" } 2023-07-29T01:13:30.889 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051953', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:09:37.925040Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044881', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:07:23.001407Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045660', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:22.902962Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:13:30.890 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052093', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045800', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:13:30.890 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051953', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:13:30.891 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052093 target=00000000000000000000:00000000000000051953, shard=2 master=00000000000000000000:00000000000000045800 target=00000000000000000000:00000000000000045660 2023-07-29T01:13:35.892 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:13:35.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:13:35.893 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:13:35.948 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:13:35.949 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:13:35.997 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:13:35.997 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:13:35.997 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:13:35.998 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:13:35.998 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:13:35.998 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:13:35.998 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:35.998 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:13:35.998 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:35.998 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:13:35.999 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:35.999 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:35.999 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051953", 2023-07-29T01:13:35.999 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:35.999 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:13:35.999 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:36.000 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:09:37.925040Z" 2023-07-29T01:13:36.000 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:36.000 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:36.000 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:36.000 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:13:36.000 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:36.000 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:36.001 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044881", 2023-07-29T01:13:36.001 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:36.001 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:13:36.001 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:36.001 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:23.001407Z" 2023-07-29T01:13:36.001 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:36.001 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:36.002 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:36.002 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:13:36.002 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:36.002 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:36.002 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045660", 2023-07-29T01:13:36.002 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:36.002 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:13:36.003 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:36.003 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:22.902962Z" 2023-07-29T01:13:36.003 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:36.003 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:36.003 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:36.003 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:13:36.003 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:36.004 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:36.004 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:13:36.004 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:36.004 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:13:36.004 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:36.004 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:13:36.005 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:36.005 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:36.005 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:13:36.005 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:36.005 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:13:36.005 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:13:36.006 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:13:36.006 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:36.006 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:13:35Z" 2023-07-29T01:13:36.006 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:13:36.007 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051953", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:09:37.925040Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044881", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:07:23.001407Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045660", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:22.902962Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:13:35Z" } 2023-07-29T01:13:36.007 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051953', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:09:37.925040Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044881', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:07:23.001407Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045660', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:22.902962Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:13:36.008 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052093', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045800', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:13:36.008 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051953', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:13:36.009 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052093 target=00000000000000000000:00000000000000051953, shard=2 master=00000000000000000000:00000000000000045800 target=00000000000000000000:00000000000000045660 2023-07-29T01:13:41.009 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:13:41.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:13:41.011 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:13:41.065 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:13:41.066 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:13:41.113 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:13:41.114 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:13:41.114 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:13:41.114 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:13:41.114 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:13:41.114 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:13:41.115 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:41.115 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:13:41.115 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:41.115 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:13:41.115 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:41.115 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:41.115 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051953", 2023-07-29T01:13:41.116 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:41.116 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:13:41.116 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:41.116 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:09:37.925040Z" 2023-07-29T01:13:41.116 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:41.116 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:41.117 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:41.117 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:13:41.117 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:41.117 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:41.117 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044881", 2023-07-29T01:13:41.117 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:41.117 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:13:41.118 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:41.118 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:23.001407Z" 2023-07-29T01:13:41.118 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:41.118 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:41.118 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:41.118 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:13:41.118 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:41.119 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:41.119 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045660", 2023-07-29T01:13:41.119 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:41.119 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:13:41.119 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:41.119 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:22.902962Z" 2023-07-29T01:13:41.119 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:41.120 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:41.120 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:41.120 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:13:41.120 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:41.120 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:41.120 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:13:41.121 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:41.121 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:13:41.121 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:41.121 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:13:41.121 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:41.121 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:41.121 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:13:41.122 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:41.122 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:13:41.122 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:13:41.122 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:13:41.122 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:41.122 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:13:41Z" 2023-07-29T01:13:41.122 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:13:41.123 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051953", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:09:37.925040Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044881", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:07:23.001407Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045660", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:22.902962Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:13:41Z" } 2023-07-29T01:13:41.124 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051953', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:09:37.925040Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044881', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:07:23.001407Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045660', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:22.902962Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:13:41.124 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052093', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045800', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:13:41.125 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051953', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:13:41.125 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052093 target=00000000000000000000:00000000000000051953, shard=2 master=00000000000000000000:00000000000000045800 target=00000000000000000000:00000000000000045660 2023-07-29T01:13:46.126 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:13:46.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:13:46.127 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:13:46.180 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:13:46.180 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:13:46.227 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:13:46.227 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:13:46.227 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:13:46.228 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:13:46.228 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:13:46.228 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:13:46.228 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:46.228 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:13:46.228 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:46.229 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:13:46.229 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:46.229 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:46.229 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051953", 2023-07-29T01:13:46.229 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:46.229 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:13:46.229 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:46.230 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:09:37.925040Z" 2023-07-29T01:13:46.230 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:46.230 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:46.230 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:46.230 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:13:46.230 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:46.230 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:46.231 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044881", 2023-07-29T01:13:46.231 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:46.231 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:13:46.231 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:46.231 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:23.001407Z" 2023-07-29T01:13:46.231 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:46.231 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:46.232 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:46.232 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:13:46.232 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:46.232 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:46.232 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045660", 2023-07-29T01:13:46.232 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:46.233 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:13:46.233 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:46.233 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:22.902962Z" 2023-07-29T01:13:46.233 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:46.233 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:46.233 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:46.233 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:13:46.234 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:46.234 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:46.234 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:13:46.234 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:46.234 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:13:46.234 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:46.234 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:13:46.235 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:46.235 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:46.235 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:13:46.235 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:46.235 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:13:46.235 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:13:46.235 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:13:46.236 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:46.236 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:13:46Z" 2023-07-29T01:13:46.236 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:13:46.237 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051953", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:09:37.925040Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044881", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:07:23.001407Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045660", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:22.902962Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:13:46Z" } 2023-07-29T01:13:46.237 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051953', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:09:37.925040Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044881', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:07:23.001407Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045660', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:22.902962Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:13:46.238 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052093', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045800', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:13:46.238 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051953', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:13:46.238 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052093 target=00000000000000000000:00000000000000051953, shard=2 master=00000000000000000000:00000000000000045800 target=00000000000000000000:00000000000000045660 2023-07-29T01:13:51.240 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:13:51.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:13:51.241 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:13:51.297 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:13:51.297 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:13:51.347 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:13:51.348 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:13:51.348 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:13:51.348 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:13:51.348 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:13:51.348 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:13:51.349 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:51.349 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:13:51.349 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:51.349 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:13:51.349 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:51.349 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:51.349 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051953", 2023-07-29T01:13:51.350 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:51.350 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:13:51.350 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:51.350 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:09:37.925040Z" 2023-07-29T01:13:51.350 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:51.350 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:51.350 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:51.351 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:13:51.351 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:51.351 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:51.351 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044881", 2023-07-29T01:13:51.351 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:51.351 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:13:51.352 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:51.352 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:23.001407Z" 2023-07-29T01:13:51.352 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:51.352 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:51.352 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:51.353 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:13:51.353 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:51.353 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:51.353 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045660", 2023-07-29T01:13:51.354 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:51.354 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:13:51.354 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:51.354 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:22.902962Z" 2023-07-29T01:13:51.355 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:51.355 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:51.355 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:51.356 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:13:51.356 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:51.357 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:51.357 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:13:51.357 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:51.358 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:13:51.358 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:51.358 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:13:51.358 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:51.359 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:51.359 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:13:51.359 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:51.360 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:13:51.360 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:13:51.360 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:13:51.360 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:51.361 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:13:51Z" 2023-07-29T01:13:51.361 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:13:51.362 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051953", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:09:37.925040Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044881", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:07:23.001407Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045660", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:22.902962Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:13:51Z" } 2023-07-29T01:13:51.362 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051953', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:09:37.925040Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044881', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:07:23.001407Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045660', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:22.902962Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:13:51.363 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052093', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045800', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:13:51.363 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051953', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:13:51.363 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052093 target=00000000000000000000:00000000000000051953, shard=2 master=00000000000000000000:00000000000000045800 target=00000000000000000000:00000000000000045660 2023-07-29T01:13:56.364 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:13:56.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:13:56.366 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:13:56.422 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:13:56.423 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:13:56.471 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:13:56.472 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:13:56.472 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:13:56.472 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:13:56.472 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:13:56.472 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:13:56.472 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:56.473 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:13:56.473 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:56.473 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:13:56.473 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:56.473 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:56.473 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051953", 2023-07-29T01:13:56.474 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:56.474 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:13:56.474 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:56.474 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:09:37.925040Z" 2023-07-29T01:13:56.474 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:56.474 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:56.474 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:56.475 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:13:56.475 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:56.475 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:56.475 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044881", 2023-07-29T01:13:56.475 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:56.475 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:13:56.475 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:56.476 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:23.001407Z" 2023-07-29T01:13:56.476 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:56.476 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:56.476 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:56.476 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:13:56.476 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:56.476 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:56.477 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045660", 2023-07-29T01:13:56.477 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:56.477 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:13:56.477 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:56.477 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:22.902962Z" 2023-07-29T01:13:56.477 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:56.477 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:56.478 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:13:56.478 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:13:56.478 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:13:56.478 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:13:56.478 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:13:56.478 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:13:56.478 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:13:56.479 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:13:56.479 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:13:56.479 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:56.479 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:13:56.479 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:13:56.479 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:56.479 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:13:56.480 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:13:56.480 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:13:56.480 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:13:56.480 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:13:56Z" 2023-07-29T01:13:56.480 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:13:56.481 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051953", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:09:37.925040Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044881", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:07:23.001407Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045660", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:22.902962Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:13:56Z" } 2023-07-29T01:13:56.482 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051953', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:09:37.925040Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044881', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:07:23.001407Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045660', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:22.902962Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:13:56.482 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052093', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045800', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:13:56.482 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051953', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:13:56.483 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052093 target=00000000000000000000:00000000000000051953, shard=2 master=00000000000000000000:00000000000000045800 target=00000000000000000000:00000000000000045660 2023-07-29T01:14:01.484 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:14:01.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:14:01.485 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:14:01.538 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:14:01.539 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:14:01.585 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:14:01.586 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:14:01.586 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:14:01.586 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:14:01.586 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:14:01.586 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:14:01.586 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:01.587 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:14:01.587 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:01.587 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:14:01.587 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:01.587 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:14:01.587 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051953", 2023-07-29T01:14:01.587 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:01.588 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:14:01.588 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:01.588 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:09:37.925040Z" 2023-07-29T01:14:01.588 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:01.588 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:01.588 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:01.588 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:14:01.589 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:01.589 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:14:01.589 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044881", 2023-07-29T01:14:01.589 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:01.589 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:14:01.589 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:01.590 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:23.001407Z" 2023-07-29T01:14:01.590 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:01.590 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:01.590 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:01.590 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:14:01.590 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:01.590 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:14:01.591 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045660", 2023-07-29T01:14:01.591 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:01.591 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:14:01.591 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:01.591 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:22.902962Z" 2023-07-29T01:14:01.591 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:01.591 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:01.592 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:01.592 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:14:01.592 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:01.592 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:14:01.592 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:14:01.592 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:01.592 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:14:01.593 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:01.593 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:14:01.593 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:01.593 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:01.593 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:14:01.593 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:01.593 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:14:01.594 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:14:01.594 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:14:01.594 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:01.594 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:14:01Z" 2023-07-29T01:14:01.594 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:14:01.595 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051953", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:09:37.925040Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044881", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:07:23.001407Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045660", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:22.902962Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:14:01Z" } 2023-07-29T01:14:01.595 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051953', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:09:37.925040Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044881', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:07:23.001407Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045660', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:22.902962Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:14:01.596 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052093', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045800', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:14:01.596 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051953', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:14:01.597 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052093 target=00000000000000000000:00000000000000051953, shard=2 master=00000000000000000000:00000000000000045800 target=00000000000000000000:00000000000000045660 2023-07-29T01:14:06.597 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:14:06.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:14:06.598 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:14:06.654 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:14:06.654 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:14:06.703 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:14:06.703 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:14:06.703 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:14:06.703 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:14:06.703 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:14:06.704 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:14:06.704 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:06.704 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:14:06.704 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:06.704 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:14:06.704 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:06.704 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:14:06.705 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000051953", 2023-07-29T01:14:06.705 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:06.705 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:14:06.705 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:06.705 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:09:37.925040Z" 2023-07-29T01:14:06.705 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:06.706 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:06.706 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:06.706 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:14:06.706 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:06.706 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:14:06.706 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044881", 2023-07-29T01:14:06.706 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:06.707 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:14:06.707 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:06.707 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:23.001407Z" 2023-07-29T01:14:06.707 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:06.707 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:06.707 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:06.707 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:14:06.708 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:06.708 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:14:06.708 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045660", 2023-07-29T01:14:06.708 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:06.708 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:14:06.708 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:06.708 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:22.902962Z" 2023-07-29T01:14:06.709 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:06.709 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:06.709 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:06.709 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:14:06.709 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:06.709 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:14:06.709 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:14:06.710 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:06.710 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:14:06.710 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:06.710 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:14:06.710 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:06.710 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:06.710 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:14:06.711 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:06.711 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:14:06.711 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:14:06.711 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:14:06.711 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:06.711 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:14:06Z" 2023-07-29T01:14:06.711 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:14:06.712 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051953", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:09:37.925040Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044881", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:07:23.001407Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045660", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:07:22.902962Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:14:06Z" } 2023-07-29T01:14:06.713 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051953', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:09:37.925040Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044881', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:07:23.001407Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045660', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:07:22.902962Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:14:06.713 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052093', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045800', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:14:06.713 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051953', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045660', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:14:06.714 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052093 target=00000000000000000000:00000000000000051953, shard=2 master=00000000000000000000:00000000000000045800 target=00000000000000000000:00000000000000045660 2023-07-29T01:14:11.714 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:14:11.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:14:11.716 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:14:11.770 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:14:11.771 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:14:11.818 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:14:11.819 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:14:11.819 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:14:11.819 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:14:11.819 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:14:11.819 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:14:11.819 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:11.820 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:14:11.820 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:11.820 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:14:11.820 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:11.820 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:14:11.820 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052093", 2023-07-29T01:14:11.820 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:11.821 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:14:11.821 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:11.821 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:12:08.229055Z" 2023-07-29T01:14:11.821 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:11.821 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:11.821 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:11.822 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:14:11.822 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:11.822 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:14:11.822 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044881", 2023-07-29T01:14:11.822 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:11.822 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:14:11.822 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:11.823 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:23.001407Z" 2023-07-29T01:14:11.823 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:11.823 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:11.823 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:11.823 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:14:11.823 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:11.823 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:14:11.824 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045800", 2023-07-29T01:14:11.824 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:11.824 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:14:11.824 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:11.824 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:12:08.266555Z" 2023-07-29T01:14:11.824 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:11.824 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:11.825 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:11.825 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:14:11.825 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:11.825 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:14:11.825 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:14:11.825 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:11.825 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:14:11.826 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:11.826 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:14:11.826 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:11.826 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:11.826 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:14:11.826 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:11.826 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:14:11.827 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:14:11.827 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:14:11.827 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:11.827 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:14:11Z" 2023-07-29T01:14:11.827 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:14:11.828 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052093", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:12:08.229055Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044881", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:07:23.001407Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045800", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:12:08.266555Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:14:11Z" } 2023-07-29T01:14:11.829 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052093', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:12:08.229055Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044881', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:07:23.001407Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045800', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:12:08.266555Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:14:11.829 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052093', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045800', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:14:11.830 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052093', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045800', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:14:11.830 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-07-29T01:14:11.830 DEBUG:boto:path=/hiiqos-70/ 2023-07-29T01:14:11.831 DEBUG:boto:auth_path=/hiiqos-70/ 2023-07-29T01:14:11.831 DEBUG:boto:Method: HEAD 2023-07-29T01:14:11.831 DEBUG:boto:Path: /hiiqos-70/ 2023-07-29T01:14:11.832 DEBUG:boto:Data: 2023-07-29T01:14:11.832 DEBUG:boto:Headers: {} 2023-07-29T01:14:11.832 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:14:11.833 DEBUG:boto:Port: 8001 2023-07-29T01:14:11.833 DEBUG:boto:Params: {} 2023-07-29T01:14:11.833 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-29T01:14:11.834 DEBUG:boto:Token: None 2023-07-29T01:14:11.834 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:14:11 GMT /hiiqos-70/ 2023-07-29T01:14:11.835 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:IXPYExVHjvbPaCNIUWsbnXvK3So= 2023-07-29T01:14:11.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:14:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IXPYExVHjvbPaCNIUWsbnXvK3So=', 'Content-Length': '0'} 2023-07-29T01:14:11.840 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000038ad145b73737261-0064c467e3-274b-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:14:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:14:11.841 DEBUG:boto:path=/hiiqos-70/obj1 2023-07-29T01:14:11.841 DEBUG:boto:auth_path=/hiiqos-70/obj1 2023-07-29T01:14:11.841 DEBUG:boto:Method: HEAD 2023-07-29T01:14:11.842 DEBUG:boto:Path: /hiiqos-70/obj1 2023-07-29T01:14:11.842 DEBUG:boto:Data: 2023-07-29T01:14:11.842 DEBUG:boto:Headers: {} 2023-07-29T01:14:11.843 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:14:11.843 DEBUG:boto:Port: 8001 2023-07-29T01:14:11.843 DEBUG:boto:Params: {} 2023-07-29T01:14:11.844 DEBUG:boto:Token: None 2023-07-29T01:14:11.844 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:14:11 GMT /hiiqos-70/obj1 2023-07-29T01:14:11.845 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:M1jgvV2fLjPcssSRFUY1goQhdJY= 2023-07-29T01:14:11.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:14:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:M1jgvV2fLjPcssSRFUY1goQhdJY=', 'Content-Length': '0'} 2023-07-29T01:14:11.847 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000fb071cd833d4f3b5-0064c467e3-274b-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 01:14:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:14:11.847 DEBUG:boto:path=/hiiqos-70/obj2 2023-07-29T01:14:11.847 DEBUG:boto:auth_path=/hiiqos-70/obj2 2023-07-29T01:14:11.848 DEBUG:boto:Method: HEAD 2023-07-29T01:14:11.848 DEBUG:boto:Path: /hiiqos-70/obj2 2023-07-29T01:14:11.848 DEBUG:boto:Data: 2023-07-29T01:14:11.849 DEBUG:boto:Headers: {} 2023-07-29T01:14:11.849 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:14:11.849 DEBUG:boto:Port: 8001 2023-07-29T01:14:11.850 DEBUG:boto:Params: {} 2023-07-29T01:14:11.850 DEBUG:boto:Token: None 2023-07-29T01:14:11.850 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:14:11 GMT /hiiqos-70/obj2 2023-07-29T01:14:11.851 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Y10GxrrEtq/AEt+69RESleqgkbo= 2023-07-29T01:14:11.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:14:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Y10GxrrEtq/AEt+69RESleqgkbo=', 'Content-Length': '0'} 2023-07-29T01:14:11.853 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000056cb3179910ed93e-0064c467e3-274b-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 01:14:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:14:11.853 DEBUG:boto:path=/hiiqos-71/ 2023-07-29T01:14:11.854 DEBUG:boto:auth_path=/hiiqos-71/ 2023-07-29T01:14:11.854 DEBUG:boto:Method: HEAD 2023-07-29T01:14:11.854 DEBUG:boto:Path: /hiiqos-71/ 2023-07-29T01:14:11.855 DEBUG:boto:Data: 2023-07-29T01:14:11.855 DEBUG:boto:Headers: {} 2023-07-29T01:14:11.855 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:14:11.856 DEBUG:boto:Port: 8001 2023-07-29T01:14:11.856 DEBUG:boto:Params: {} 2023-07-29T01:14:11.856 DEBUG:boto:Token: None 2023-07-29T01:14:11.857 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:14:11 GMT /hiiqos-71/ 2023-07-29T01:14:11.857 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:XLr4mKmSuzjw3qcADV20iJqnDsw= 2023-07-29T01:14:11.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:14:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XLr4mKmSuzjw3qcADV20iJqnDsw=', 'Content-Length': '0'} 2023-07-29T01:14:11.860 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002b24397901e23786-0064c467e3-274b-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:14:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:14:11.860 DEBUG:boto:path=/hiiqos-71/obj1 2023-07-29T01:14:11.860 DEBUG:boto:auth_path=/hiiqos-71/obj1 2023-07-29T01:14:11.861 DEBUG:boto:Method: HEAD 2023-07-29T01:14:11.861 DEBUG:boto:Path: /hiiqos-71/obj1 2023-07-29T01:14:11.861 DEBUG:boto:Data: 2023-07-29T01:14:11.862 DEBUG:boto:Headers: {} 2023-07-29T01:14:11.862 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:14:11.862 DEBUG:boto:Port: 8001 2023-07-29T01:14:11.863 DEBUG:boto:Params: {} 2023-07-29T01:14:11.863 DEBUG:boto:Token: None 2023-07-29T01:14:11.864 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:14:11 GMT /hiiqos-71/obj1 2023-07-29T01:14:11.864 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:b6dPOE27RMS2VoMA4UErjz7gP4U= 2023-07-29T01:14:11.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:14:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:b6dPOE27RMS2VoMA4UErjz7gP4U=', 'Content-Length': '0'} 2023-07-29T01:14:11.866 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:12:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-70:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.9662.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000057716d4b51b89a9d-0064c467e3-274b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:14:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:14:11.867 DEBUG:boto:path=/hiiqos-71/obj1 2023-07-29T01:14:11.867 DEBUG:boto:auth_path=/hiiqos-71/obj1 2023-07-29T01:14:11.867 DEBUG:boto:Method: GET 2023-07-29T01:14:11.868 DEBUG:boto:Path: /hiiqos-71/obj1 2023-07-29T01:14:11.868 DEBUG:boto:Data: 2023-07-29T01:14:11.868 DEBUG:boto:Headers: {} 2023-07-29T01:14:11.869 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:14:11.869 DEBUG:boto:Port: 8001 2023-07-29T01:14:11.870 DEBUG:boto:Params: {} 2023-07-29T01:14:11.870 DEBUG:boto:Token: None 2023-07-29T01:14:11.870 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 01:14:11 GMT /hiiqos-71/obj1 2023-07-29T01:14:11.871 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zysdOg2yHpeW+zN97dWJJ74dRQw= 2023-07-29T01:14:11.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:14:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zysdOg2yHpeW+zN97dWJJ74dRQw=', 'Content-Length': '0'} 2023-07-29T01:14:11.873 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:12:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-70:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.9662.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000acb6a11be13ea471-0064c467e3-274b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:14:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:14:11.874 DEBUG:boto:path=/hiiqos-71/obj2 2023-07-29T01:14:11.874 DEBUG:boto:auth_path=/hiiqos-71/obj2 2023-07-29T01:14:11.874 DEBUG:boto:Method: HEAD 2023-07-29T01:14:11.875 DEBUG:boto:Path: /hiiqos-71/obj2 2023-07-29T01:14:11.875 DEBUG:boto:Data: 2023-07-29T01:14:11.875 DEBUG:boto:Headers: {} 2023-07-29T01:14:11.876 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:14:11.876 DEBUG:boto:Port: 8001 2023-07-29T01:14:11.876 DEBUG:boto:Params: {} 2023-07-29T01:14:11.877 DEBUG:boto:Token: None 2023-07-29T01:14:11.877 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:14:11 GMT /hiiqos-71/obj2 2023-07-29T01:14:11.877 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5BtOwU+U+42+evCq+2IW1Q65nLs= 2023-07-29T01:14:11.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:14:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5BtOwU+U+42+evCq+2IW1Q65nLs=', 'Content-Length': '0'} 2023-07-29T01:14:11.879 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:12:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-70:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.9662.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d12c6b7831d8dfbb-0064c467e3-274b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:14:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:14:11.880 DEBUG:boto:path=/hiiqos-71/obj2 2023-07-29T01:14:11.880 DEBUG:boto:auth_path=/hiiqos-71/obj2 2023-07-29T01:14:11.881 DEBUG:boto:Method: GET 2023-07-29T01:14:11.881 DEBUG:boto:Path: /hiiqos-71/obj2 2023-07-29T01:14:11.881 DEBUG:boto:Data: 2023-07-29T01:14:11.882 DEBUG:boto:Headers: {} 2023-07-29T01:14:11.882 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:14:11.882 DEBUG:boto:Port: 8001 2023-07-29T01:14:11.883 DEBUG:boto:Params: {} 2023-07-29T01:14:11.883 DEBUG:boto:Token: None 2023-07-29T01:14:11.883 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 01:14:11 GMT /hiiqos-71/obj2 2023-07-29T01:14:11.884 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qjW/xzkQtmRNOuRxzkONNYZzRtA= 2023-07-29T01:14:11.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:14:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qjW/xzkQtmRNOuRxzkONNYZzRtA=', 'Content-Length': '0'} 2023-07-29T01:14:11.886 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:12:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-70:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.9662.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ff3cba33dada46f9-0064c467e3-274b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:14:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:14:11.887 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:14:11.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:14:11.887 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:14:11.936 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:14:11.936 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:14:11.981 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2023-07-29T01:14:11.982 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:14:11.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:14:11.983 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:14:12.031 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:14:12.032 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:14:12.076 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-07-29T01:14:12.076 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:14:12.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:14:12.077 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:14:12.123 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:14:12.123 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:14:12.168 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-07-29T01:14:12.169 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:14:12.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:14:12.170 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:14:12.215 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:14:12.216 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:14:12.265 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:14:12.265 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:14:12.265 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 19, 2023-07-29T01:14:12.265 INFO:teuthology.orchestra.run.smithi101.stdout: "predecessor_uuid": "a0987484-ce37-4520-a564-54b7ea8c5251", 2023-07-29T01:14:12.265 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": [], 2023-07-29T01:14:12.266 INFO:teuthology.orchestra.run.smithi101.stdout: "period_map": { 2023-07-29T01:14:12.266 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:14:12.266 INFO:teuthology.orchestra.run.smithi101.stdout: "zonegroups": [ 2023-07-29T01:14:12.266 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:12.266 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "a", 2023-07-29T01:14:12.266 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a", 2023-07-29T01:14:12.266 INFO:teuthology.orchestra.run.smithi101.stdout: "api_name": "", 2023-07-29T01:14:12.267 INFO:teuthology.orchestra.run.smithi101.stdout: "is_master": true, 2023-07-29T01:14:12.267 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:14:12.267 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-29T01:14:12.267 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:14:12.267 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames": [], 2023-07-29T01:14:12.267 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames_s3website": [], 2023-07-29T01:14:12.268 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:14:12.268 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:14:12.268 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:12.268 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:14:12.268 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a1", 2023-07-29T01:14:12.268 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:14:12.268 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-29T01:14:12.269 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:14:12.269 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T01:14:12.269 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T01:14:12.269 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T01:14:12.269 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T01:14:12.269 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T01:14:12.269 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T01:14:12.270 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T01:14:12.270 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T01:14:12.270 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T01:14:12.270 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T01:14:12.270 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:14:12.270 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:14:12.270 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:12.271 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:12.271 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-29T01:14:12.271 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a2", 2023-07-29T01:14:12.271 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:14:12.271 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8001" 2023-07-29T01:14:12.271 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:14:12.271 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T01:14:12.272 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T01:14:12.272 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T01:14:12.272 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T01:14:12.272 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T01:14:12.272 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T01:14:12.272 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T01:14:12.273 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T01:14:12.273 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T01:14:12.273 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T01:14:12.273 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:14:12.273 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:14:12.273 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:12.273 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:14:12.274 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_targets": [ 2023-07-29T01:14:12.274 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:12.274 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default-placement", 2023-07-29T01:14:12.274 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [], 2023-07-29T01:14:12.274 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": [ 2023-07-29T01:14:12.274 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD" 2023-07-29T01:14:12.274 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:14:12.275 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:12.275 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:14:12.275 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "default-placement", 2023-07-29T01:14:12.275 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T01:14:12.275 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_policy": { 2023-07-29T01:14:12.275 INFO:teuthology.orchestra.run.smithi101.stdout: "groups": [ 2023-07-29T01:14:12.275 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:12.276 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "sync-group", 2023-07-29T01:14:12.276 INFO:teuthology.orchestra.run.smithi101.stdout: "data_flow": { 2023-07-29T01:14:12.276 INFO:teuthology.orchestra.run.smithi101.stdout: "symmetrical": [ 2023-07-29T01:14:12.276 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:12.276 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "sync-flow1", 2023-07-29T01:14:12.276 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:14:12.276 INFO:teuthology.orchestra.run.smithi101.stdout: "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:14:12.277 INFO:teuthology.orchestra.run.smithi101.stdout: "59d1bb30-3b1c-476f-9111-052c1ea0b711" 2023-07-29T01:14:12.277 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:14:12.277 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:12.277 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:14:12.277 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:12.277 INFO:teuthology.orchestra.run.smithi101.stdout: "pipes": [ 2023-07-29T01:14:12.277 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:12.278 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "sync-pipe", 2023-07-29T01:14:12.278 INFO:teuthology.orchestra.run.smithi101.stdout: "source": { 2023-07-29T01:14:12.278 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket": "*", 2023-07-29T01:14:12.278 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:14:12.278 INFO:teuthology.orchestra.run.smithi101.stdout: "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:14:12.278 INFO:teuthology.orchestra.run.smithi101.stdout: "59d1bb30-3b1c-476f-9111-052c1ea0b711" 2023-07-29T01:14:12.278 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:14:12.279 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:12.279 INFO:teuthology.orchestra.run.smithi101.stdout: "dest": { 2023-07-29T01:14:12.279 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket": "*", 2023-07-29T01:14:12.279 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:14:12.279 INFO:teuthology.orchestra.run.smithi101.stdout: "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:14:12.279 INFO:teuthology.orchestra.run.smithi101.stdout: "59d1bb30-3b1c-476f-9111-052c1ea0b711" 2023-07-29T01:14:12.279 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:14:12.279 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:12.280 INFO:teuthology.orchestra.run.smithi101.stdout: "params": { 2023-07-29T01:14:12.280 INFO:teuthology.orchestra.run.smithi101.stdout: "source": { 2023-07-29T01:14:12.280 INFO:teuthology.orchestra.run.smithi101.stdout: "filter": { 2023-07-29T01:14:12.280 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [] 2023-07-29T01:14:12.280 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:12.280 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:12.281 INFO:teuthology.orchestra.run.smithi101.stdout: "dest": {}, 2023-07-29T01:14:12.281 INFO:teuthology.orchestra.run.smithi101.stdout: "priority": 0, 2023-07-29T01:14:12.281 INFO:teuthology.orchestra.run.smithi101.stdout: "mode": "system", 2023-07-29T01:14:12.281 INFO:teuthology.orchestra.run.smithi101.stdout: "user": "" 2023-07-29T01:14:12.281 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:12.281 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:12.281 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:14:12.282 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "allowed" 2023-07-29T01:14:12.282 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:12.282 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:14:12.282 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:12.282 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled_features": [ 2023-07-29T01:14:12.282 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:14:12.282 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:14:12.283 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:12.283 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:12.283 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "b", 2023-07-29T01:14:12.283 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b", 2023-07-29T01:14:12.283 INFO:teuthology.orchestra.run.smithi101.stdout: "api_name": "", 2023-07-29T01:14:12.283 INFO:teuthology.orchestra.run.smithi101.stdout: "is_master": false, 2023-07-29T01:14:12.284 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:14:12.284 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-29T01:14:12.284 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:14:12.284 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames": [], 2023-07-29T01:14:12.284 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames_s3website": [], 2023-07-29T01:14:12.284 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T01:14:12.284 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:14:12.285 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:12.285 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T01:14:12.285 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b1", 2023-07-29T01:14:12.285 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:14:12.285 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-29T01:14:12.285 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:14:12.285 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T01:14:12.286 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T01:14:12.286 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T01:14:12.286 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T01:14:12.286 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T01:14:12.286 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T01:14:12.286 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T01:14:12.286 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T01:14:12.287 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T01:14:12.287 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T01:14:12.287 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:14:12.287 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:14:12.287 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:12.287 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:12.287 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-29T01:14:12.288 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b2", 2023-07-29T01:14:12.288 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:14:12.288 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8001" 2023-07-29T01:14:12.288 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:14:12.288 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T01:14:12.288 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T01:14:12.288 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T01:14:12.289 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T01:14:12.289 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T01:14:12.289 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T01:14:12.289 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T01:14:12.289 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T01:14:12.289 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T01:14:12.289 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T01:14:12.290 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:14:12.290 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:14:12.290 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:12.290 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:14:12.290 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_targets": [ 2023-07-29T01:14:12.290 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:12.290 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default-placement", 2023-07-29T01:14:12.291 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [], 2023-07-29T01:14:12.291 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": [ 2023-07-29T01:14:12.291 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD" 2023-07-29T01:14:12.291 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:14:12.291 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:12.291 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:14:12.291 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "default-placement", 2023-07-29T01:14:12.292 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T01:14:12.292 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_policy": { 2023-07-29T01:14:12.292 INFO:teuthology.orchestra.run.smithi101.stdout: "groups": [] 2023-07-29T01:14:12.292 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:12.292 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled_features": [ 2023-07-29T01:14:12.292 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:14:12.293 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:14:12.293 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:12.293 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:14:12.293 INFO:teuthology.orchestra.run.smithi101.stdout: "short_zone_ids": [ 2023-07-29T01:14:12.293 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:12.293 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:14:12.293 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 2896287231 2023-07-29T01:14:12.294 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:12.294 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:12.294 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-29T01:14:12.294 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 815606810 2023-07-29T01:14:12.295 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:12.295 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:12.295 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T01:14:12.295 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 2009171363 2023-07-29T01:14:12.296 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:12.296 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:12.296 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-29T01:14:12.296 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 3105190295 2023-07-29T01:14:12.296 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:12.296 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:14:12.296 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:12.297 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zonegroup": "a", 2023-07-29T01:14:12.297 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:14:12.297 INFO:teuthology.orchestra.run.smithi101.stdout: "period_config": { 2023-07-29T01:14:12.297 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_quota": { 2023-07-29T01:14:12.297 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-29T01:14:12.297 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-29T01:14:12.297 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-29T01:14:12.298 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-29T01:14:12.298 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-29T01:14:12.298 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:12.298 INFO:teuthology.orchestra.run.smithi101.stdout: "user_quota": { 2023-07-29T01:14:12.298 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-29T01:14:12.298 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-29T01:14:12.298 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-29T01:14:12.299 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-29T01:14:12.299 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-29T01:14:12.299 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:12.299 INFO:teuthology.orchestra.run.smithi101.stdout: "user_ratelimit": { 2023-07-29T01:14:12.299 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-29T01:14:12.299 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-29T01:14:12.299 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-29T01:14:12.300 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-29T01:14:12.300 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-29T01:14:12.300 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:12.300 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_ratelimit": { 2023-07-29T01:14:12.300 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-29T01:14:12.300 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-29T01:14:12.300 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-29T01:14:12.301 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-29T01:14:12.301 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-29T01:14:12.301 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:12.301 INFO:teuthology.orchestra.run.smithi101.stdout: "anonymous_ratelimit": { 2023-07-29T01:14:12.301 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-29T01:14:12.301 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-29T01:14:12.301 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-29T01:14:12.302 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-29T01:14:12.302 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-29T01:14:12.302 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:12.302 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:12.302 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T01:14:12.302 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_name": "test-realm", 2023-07-29T01:14:12.302 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:12.303 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:14:12.303 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:14:12.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:14:12.304 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:14:12.350 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:14:12.350 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:14:12.402 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-07-29T01:14:12.402 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'hiiqos-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:14:12.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'hiiqos-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:14:12.403 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket hiiqos-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:14:12.484 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:14:12.484 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:14:12.581 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-07-29T01:14:12.582 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'hiiqos-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:14:12.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'hiiqos-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:14:12.583 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket hiiqos-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:14:12.676 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:14:12.676 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:14:12.783 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2023-07-29T01:14:12.783 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipeA', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=hiiqos-70', '--source-bucket=*', '--dest-bucket=hiiqos-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:14:12.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipeA', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=hiiqos-70', '--source-bucket=*', '--dest-bucket=hiiqos-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:14:12.785 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipeA --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=hiiqos-70 '--source-bucket=*' --dest-bucket=hiiqos-71 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:14:12.860 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:14:12.861 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:14:12.936 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipeA","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"hiiqos-71","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-07-29T01:14:12.936 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'hiiqos-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:14:12.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'hiiqos-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:14:12.937 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket hiiqos-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:14:12.990 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:14:12.991 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:14:13.046 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipeA","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"hiiqos-71","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-07-29T01:14:13.046 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'hiiqos-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:14:13.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'hiiqos-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:14:13.047 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket hiiqos-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:14:13.091 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:14:13.091 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:14:13.136 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipeA","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"hiiqos-71","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-07-29T01:14:13.136 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:14:13.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:14:13.137 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:14:13.182 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:14:13.182 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:14:13.212 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T01:14:13.226 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T01:14:13.226 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:13.226 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:14:13.226 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:17.836402Z" 2023-07-29T01:14:13.226 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:13.227 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:13.227 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593252.941877_1724.1", 2023-07-29T01:14:13.227 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:14:12.941877Z" 2023-07-29T01:14:13.227 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:13.227 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:13.227 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:14:13.227 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T01:14:13.228 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:13.228 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:13.228 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:14:13.228 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:12.811406Z" 2023-07-29T01:14:13.228 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:13.228 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T01:14:13.229 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690591217.836402_294.1', 1: '1_1690593252.941877_1724.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:14:13.230 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T01:14:13.230 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:14:13.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:14:13.231 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:14:13.276 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:14:13.284 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:14:13.304 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:14:13.213+0000 7f1f63f35880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:14:13.305 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:14:13.215+0000 7f1f63f35880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:14:13.318 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:14:13.318 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:14:13.318 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:14:13.318 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:14:13.319 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:14:13.319 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:14:13.319 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:13.319 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:13.319 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:14:13.319 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:13.319 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:14:13.320 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:13.320 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:14:13.320 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:14:13.320 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:13.320 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:14:13.320 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:13.320 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:14:13.321 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:13.321 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:13.321 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:13.321 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:13.321 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:14:13.321 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:13.321 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:14:13.322 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593122.086179_1622.1", 2023-07-29T01:14:13.322 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:13.322 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:14:13.322 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:13.322 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:12:02.086179Z", 2023-07-29T01:14:13.322 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:13.322 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:13.323 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:13.323 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:13.323 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:14:13.323 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:13.323 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:14:13.323 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:14:13.323 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:13.324 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:14:13.324 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:13.324 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:14:13.324 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:13.324 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:13.324 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:13.325 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:13.325 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:14:13.325 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:13.325 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:14:13.325 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:14:13.325 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:13.325 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:14:13.326 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:13.326 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:14:13.326 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:13.326 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:13.326 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:13.326 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:14:13.326 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:13.327 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:14:13.327 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:14:13.327 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:14:13.327 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:13.327 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:14:13Z" 2023-07-29T01:14:13.327 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:14:13.328 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593122.086179_1622.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:12:02.086179Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:14:13Z" } 2023-07-29T01:14:13.329 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593122.086179_1622.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:12:02.086179Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:14:13.329 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593252.941877_1724.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:14:13.329 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593122.086179_1622.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:14:13.330 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593252.941877_1724.1 target=1_1690593122.086179_1622.1 2023-07-29T01:14:18.331 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:14:18.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:14:18.332 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:14:18.386 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:14:18.386 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:14:18.418 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:14:18.327+0000 7f1e3142d880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:14:18.419 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:14:18.328+0000 7f1e3142d880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:14:18.432 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:14:18.432 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:14:18.433 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:14:18.433 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:14:18.433 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:14:18.433 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:14:18.433 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:18.433 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:18.434 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:14:18.434 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:18.434 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:14:18.434 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:18.434 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:14:18.434 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:14:18.434 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:18.435 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:14:18.435 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:18.435 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:14:18.435 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:18.435 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:18.435 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:18.435 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:18.436 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:14:18.436 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:18.436 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:14:18.436 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593122.086179_1622.1", 2023-07-29T01:14:18.436 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:18.436 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:14:18.436 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:18.437 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:12:02.086179Z", 2023-07-29T01:14:18.437 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:18.437 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:18.437 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:18.437 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:18.437 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:14:18.437 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:18.438 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:14:18.438 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:14:18.438 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:18.438 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:14:18.438 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:18.438 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:14:18.438 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:18.439 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:18.439 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:18.439 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:18.439 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:14:18.439 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:18.439 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:14:18.439 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:14:18.440 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:18.440 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:14:18.440 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:18.440 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:14:18.440 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:18.440 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:18.441 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:18.441 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:14:18.441 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:18.441 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:14:18.441 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:14:18.441 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:14:18.441 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:18.442 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:14:18Z" 2023-07-29T01:14:18.442 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:14:18.443 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593122.086179_1622.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:12:02.086179Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:14:18Z" } 2023-07-29T01:14:18.443 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593122.086179_1622.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:12:02.086179Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:14:18.444 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593252.941877_1724.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:14:18.444 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593122.086179_1622.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:14:18.444 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593252.941877_1724.1 target=1_1690593122.086179_1622.1 2023-07-29T01:14:23.446 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:14:23.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:14:23.447 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:14:23.502 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:14:23.503 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:14:23.536 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:14:23.445+0000 7f806c136880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:14:23.537 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:14:23.446+0000 7f806c136880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:14:23.550 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:14:23.550 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:14:23.551 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:14:23.551 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:14:23.551 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:14:23.551 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:14:23.551 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:23.551 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:23.552 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:14:23.552 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:23.552 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:14:23.552 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:23.552 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:14:23.552 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:14:23.552 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:23.553 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:14:23.553 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:23.553 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:14:23.553 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:23.553 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:23.553 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:23.553 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:23.554 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:14:23.554 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:23.554 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:14:23.554 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593122.086179_1622.1", 2023-07-29T01:14:23.554 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:23.554 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:14:23.554 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:23.555 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:12:02.086179Z", 2023-07-29T01:14:23.555 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:23.555 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:23.555 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:23.555 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:23.555 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:14:23.555 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:23.556 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:14:23.556 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:14:23.556 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:23.556 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:14:23.556 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:23.556 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:14:23.557 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:23.557 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:23.557 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:23.557 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:23.557 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:14:23.557 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:23.557 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:14:23.558 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:14:23.558 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:23.558 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:14:23.558 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:23.558 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:14:23.558 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:23.558 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:23.559 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:23.559 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:14:23.559 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:23.559 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:14:23.559 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:14:23.559 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:14:23.560 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:23.560 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:14:23Z" 2023-07-29T01:14:23.560 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:14:23.561 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593122.086179_1622.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:12:02.086179Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:14:23Z" } 2023-07-29T01:14:23.561 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593122.086179_1622.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:12:02.086179Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:14:23.562 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593252.941877_1724.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:14:23.562 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593122.086179_1622.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:14:23.562 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593252.941877_1724.1 target=1_1690593122.086179_1622.1 2023-07-29T01:14:28.563 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:14:28.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:14:28.565 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:14:28.621 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:14:28.629 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:14:28.653 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:14:28.562+0000 7f2cdf926880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:14:28.654 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:14:28.563+0000 7f2cdf926880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:14:28.667 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:14:28.668 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:14:28.668 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:14:28.668 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:14:28.668 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:14:28.668 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:14:28.669 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:28.669 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:28.669 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:14:28.669 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:28.669 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:14:28.669 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:28.670 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:14:28.670 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:14:28.670 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:28.670 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:14:28.670 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:28.670 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:14:28.671 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:28.671 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:28.671 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:28.671 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:28.671 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:14:28.671 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:28.671 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:14:28.672 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593122.086179_1622.1", 2023-07-29T01:14:28.672 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:28.672 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:14:28.672 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:28.672 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:12:02.086179Z", 2023-07-29T01:14:28.672 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:28.672 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:28.673 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:28.673 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:28.673 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:14:28.673 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:28.673 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:14:28.673 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:14:28.673 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:28.674 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:14:28.674 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:28.674 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:14:28.674 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:28.674 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:28.674 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:28.674 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:28.675 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:14:28.675 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:28.675 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:14:28.675 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:14:28.675 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:28.675 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:14:28.675 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:28.676 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:14:28.676 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:28.676 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:28.676 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:28.676 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:14:28.676 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:28.676 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:14:28.677 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:14:28.677 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:14:28.677 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:28.677 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:14:28Z" 2023-07-29T01:14:28.677 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:14:28.678 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593122.086179_1622.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:12:02.086179Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:14:28Z" } 2023-07-29T01:14:28.679 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593122.086179_1622.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:12:02.086179Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:14:28.679 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593252.941877_1724.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:14:28.679 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593122.086179_1622.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:14:28.680 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593252.941877_1724.1 target=1_1690593122.086179_1622.1 2023-07-29T01:14:33.681 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:14:33.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:14:33.683 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:14:33.736 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:14:33.736 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:14:33.769 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:14:33.678+0000 7f3731212880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:14:33.769 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:14:33.679+0000 7f3731212880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:14:33.783 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:14:33.783 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:14:33.784 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:14:33.784 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:14:33.784 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:14:33.784 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:14:33.784 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:33.784 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:33.785 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:14:33.785 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:33.785 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:14:33.785 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:33.785 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:14:33.785 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:14:33.785 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:33.786 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:14:33.786 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:33.786 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:14:33.786 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:33.786 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:33.786 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:33.786 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:33.787 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:14:33.787 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:33.787 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:14:33.787 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593122.086179_1622.1", 2023-07-29T01:14:33.787 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:33.787 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:14:33.787 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:33.788 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:12:02.086179Z", 2023-07-29T01:14:33.788 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:33.788 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:33.788 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:33.788 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:33.788 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:14:33.788 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:33.789 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:14:33.789 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:14:33.789 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:33.789 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:14:33.789 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:33.789 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:14:33.789 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:33.790 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:33.790 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:33.790 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:33.790 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:14:33.790 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:33.790 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:14:33.790 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:14:33.791 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:33.791 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:14:33.791 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:33.791 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:14:33.791 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:33.791 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:33.791 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:33.792 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:14:33.792 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:33.792 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:14:33.792 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:14:33.792 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:14:33.792 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:33.792 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:14:33Z" 2023-07-29T01:14:33.793 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:14:33.793 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593122.086179_1622.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:12:02.086179Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:14:33Z" } 2023-07-29T01:14:33.794 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593122.086179_1622.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:12:02.086179Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:14:33.794 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593252.941877_1724.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:14:33.795 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593122.086179_1622.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:14:33.795 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593252.941877_1724.1 target=1_1690593122.086179_1622.1 2023-07-29T01:14:38.796 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:14:38.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:14:38.798 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:14:38.850 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:14:38.851 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:14:38.883 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:14:38.792+0000 7f361602f880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:14:38.884 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:14:38.793+0000 7f361602f880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:14:38.897 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:14:38.897 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:14:38.898 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:14:38.898 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:14:38.898 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:14:38.898 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:14:38.898 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:38.898 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:38.899 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:14:38.899 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:38.899 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:14:38.899 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:38.900 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:14:38.900 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:14:38.900 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:38.900 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:14:38.900 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:38.900 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:14:38.900 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:38.901 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:38.901 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:38.901 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:38.901 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:14:38.901 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:38.901 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:14:38.902 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593122.086179_1622.1", 2023-07-29T01:14:38.902 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:38.902 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:14:38.902 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:38.902 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:12:02.086179Z", 2023-07-29T01:14:38.902 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:38.903 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:38.903 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:38.903 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:38.903 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:14:38.903 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:38.903 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:14:38.903 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:14:38.904 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:38.904 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:14:38.904 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:38.904 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:14:38.904 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:38.904 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:38.905 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:38.905 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:38.905 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:14:38.905 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:38.905 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:14:38.905 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:14:38.905 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:38.906 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:14:38.906 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:38.906 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:14:38.906 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:38.906 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:38.907 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:38.907 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:14:38.907 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:38.907 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:14:38.907 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:14:38.907 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:14:38.907 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:38.908 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:14:38Z" 2023-07-29T01:14:38.908 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:14:38.909 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593122.086179_1622.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:12:02.086179Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:14:38Z" } 2023-07-29T01:14:38.909 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593122.086179_1622.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:12:02.086179Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:14:38.910 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593252.941877_1724.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:14:38.910 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593122.086179_1622.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:14:38.910 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593252.941877_1724.1 target=1_1690593122.086179_1622.1 2023-07-29T01:14:43.911 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:14:43.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:14:43.913 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:14:43.965 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:14:43.966 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:14:43.998 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:14:43.907+0000 7f5da2e10880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:14:43.999 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:14:43.908+0000 7f5da2e10880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:14:44.013 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:14:44.013 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:14:44.013 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:14:44.014 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:14:44.014 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:14:44.014 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:14:44.014 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:44.014 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:44.014 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:14:44.015 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:44.015 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:14:44.015 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:44.015 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:14:44.015 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:14:44.015 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:44.015 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:14:44.016 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:44.016 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:14:44.016 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:44.016 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:44.016 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:44.016 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:44.016 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:14:44.017 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:44.017 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:14:44.017 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593122.086179_1622.1", 2023-07-29T01:14:44.017 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:44.017 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:14:44.017 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:44.018 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:12:02.086179Z", 2023-07-29T01:14:44.018 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:44.018 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:44.018 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:44.019 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:44.019 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:14:44.019 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:44.019 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:14:44.019 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:14:44.019 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:44.019 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:14:44.020 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:44.020 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:14:44.020 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:44.020 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:44.020 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:44.020 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:44.021 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:14:44.021 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:44.021 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:14:44.021 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:14:44.021 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:44.021 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:14:44.021 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:44.022 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:14:44.022 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:44.022 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:44.022 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:44.022 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:14:44.022 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:44.022 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:14:44.023 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:14:44.023 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:14:44.023 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:44.023 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:14:43Z" 2023-07-29T01:14:44.023 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:14:44.024 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593122.086179_1622.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:12:02.086179Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:14:43Z" } 2023-07-29T01:14:44.025 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593122.086179_1622.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:12:02.086179Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:14:44.025 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593252.941877_1724.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:14:44.025 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593122.086179_1622.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:14:44.026 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593252.941877_1724.1 target=1_1690593122.086179_1622.1 2023-07-29T01:14:49.027 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:14:49.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:14:49.028 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:14:49.081 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:14:49.081 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:14:49.114 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:14:49.022+0000 7ff3d3810880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:14:49.114 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:14:49.023+0000 7ff3d3810880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:14:49.128 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:14:49.128 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:14:49.129 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:14:49.129 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:14:49.129 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:14:49.129 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:14:49.129 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:49.129 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:49.129 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:14:49.130 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:49.130 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:14:49.130 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:49.130 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:14:49.130 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:14:49.130 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:49.130 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:14:49.131 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:49.131 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:14:49.131 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:49.131 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:49.131 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:49.131 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:49.132 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:14:49.132 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:49.132 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:14:49.132 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593122.086179_1622.1", 2023-07-29T01:14:49.132 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:49.132 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:14:49.132 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:49.133 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:12:02.086179Z", 2023-07-29T01:14:49.133 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:49.133 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:49.133 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:49.133 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:49.133 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:14:49.133 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:49.134 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:14:49.134 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:14:49.134 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:49.134 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:14:49.134 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:49.134 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:14:49.135 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:49.135 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:49.135 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:49.135 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:49.135 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:14:49.135 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:49.135 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:14:49.136 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:14:49.136 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:49.136 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:14:49.136 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:49.136 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:14:49.136 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:49.136 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:49.137 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:49.137 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:14:49.137 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:49.137 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:14:49.137 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:14:49.138 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:14:49.138 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:49.138 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:14:49Z" 2023-07-29T01:14:49.138 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:14:49.139 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593122.086179_1622.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:12:02.086179Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:14:49Z" } 2023-07-29T01:14:49.139 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593122.086179_1622.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:12:02.086179Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:14:49.140 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593252.941877_1724.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:14:49.140 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593122.086179_1622.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:14:49.140 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593252.941877_1724.1 target=1_1690593122.086179_1622.1 2023-07-29T01:14:54.141 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:14:54.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:14:54.143 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:14:54.196 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:14:54.196 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:14:54.227 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:14:54.135+0000 7fa58ce10880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:14:54.227 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:14:54.136+0000 7fa58ce10880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:14:54.241 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:14:54.241 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:14:54.241 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:14:54.241 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:14:54.241 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:14:54.242 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:14:54.242 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:54.242 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:54.242 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:14:54.242 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:54.242 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:14:54.243 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:54.243 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:14:54.243 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:14:54.243 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:54.243 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:14:54.243 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:54.244 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:14:54.244 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:54.244 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:54.244 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:54.244 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:54.244 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:14:54.244 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:54.245 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:14:54.245 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593122.086179_1622.1", 2023-07-29T01:14:54.245 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:54.245 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:14:54.245 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:54.245 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:12:02.086179Z", 2023-07-29T01:14:54.245 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:54.246 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:54.246 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:54.246 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:54.246 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:14:54.246 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:54.246 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:14:54.246 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:14:54.247 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:54.247 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:14:54.247 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:54.247 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:14:54.247 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:54.247 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:54.247 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:54.248 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:54.248 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:14:54.248 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:54.248 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:14:54.248 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:14:54.248 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:54.248 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:14:54.249 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:54.249 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:14:54.249 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:54.249 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:54.249 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:54.249 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:14:54.249 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:54.250 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:14:54.250 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:14:54.250 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:14:54.250 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:54.250 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:14:54Z" 2023-07-29T01:14:54.250 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:14:54.251 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593122.086179_1622.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:12:02.086179Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:14:54Z" } 2023-07-29T01:14:54.252 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593122.086179_1622.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:12:02.086179Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:14:54.252 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593252.941877_1724.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:14:54.253 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593122.086179_1622.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:14:54.253 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593252.941877_1724.1 target=1_1690593122.086179_1622.1 2023-07-29T01:14:59.254 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:14:59.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:14:59.255 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:14:59.311 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:14:59.311 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:14:59.344 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:14:59.252+0000 7f5b56b34880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:14:59.372 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:14:59.253+0000 7f5b56b34880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:14:59.373 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:14:59.373 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:14:59.373 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:14:59.373 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:14:59.373 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:14:59.373 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:14:59.374 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:59.374 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:59.374 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:14:59.374 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:59.374 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:14:59.374 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:59.375 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:14:59.375 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:14:59.375 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:59.375 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:14:59.376 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:59.376 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:14:59.376 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:59.376 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:59.377 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:59.377 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:59.377 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:14:59.377 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:59.377 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:14:59.377 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593122.086179_1622.1", 2023-07-29T01:14:59.377 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:59.378 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:14:59.378 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:59.378 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:12:02.086179Z", 2023-07-29T01:14:59.378 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:59.378 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:59.379 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:59.379 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:59.379 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:14:59.379 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:59.379 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:14:59.379 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:14:59.380 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:59.380 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:14:59.380 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:59.380 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:14:59.380 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:59.380 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:59.381 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:59.381 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:14:59.381 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:14:59.381 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:14:59.381 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:14:59.382 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:14:59.382 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:14:59.382 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:14:59.382 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:14:59.382 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:14:59.382 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:14:59.382 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:59.383 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:14:59.383 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:14:59.383 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:59.383 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:14:59.383 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:14:59.383 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:14:59.384 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:14:59.384 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:14:59Z" 2023-07-29T01:14:59.384 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:14:59.384 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593122.086179_1622.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:12:02.086179Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:14:59Z" } 2023-07-29T01:14:59.385 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593122.086179_1622.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:12:02.086179Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:14:59.385 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593252.941877_1724.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:14:59.385 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593122.086179_1622.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:14:59.386 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593252.941877_1724.1 target=1_1690593122.086179_1622.1 2023-07-29T01:15:04.387 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:15:04.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:15:04.388 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:15:04.443 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:15:04.443 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:15:04.475 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:15:04.383+0000 7fc787531880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:15:04.475 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:15:04.384+0000 7fc787531880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:15:04.489 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:15:04.489 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:15:04.490 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:15:04.490 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:15:04.490 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:15:04.490 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:15:04.490 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:04.490 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:04.491 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:15:04.491 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:04.491 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:15:04.491 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:04.491 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:15:04.491 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:15:04.491 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:04.492 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:15:04.492 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:04.492 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:15:04.492 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:04.492 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:04.492 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:04.492 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:04.493 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:15:04.493 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:04.493 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:15:04.493 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593122.086179_1622.1", 2023-07-29T01:15:04.493 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:04.493 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:15:04.493 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:04.494 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:12:02.086179Z", 2023-07-29T01:15:04.494 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:04.494 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:04.494 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:04.494 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:04.494 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:15:04.494 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:04.495 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:15:04.495 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:15:04.495 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:04.495 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:15:04.495 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:04.495 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:15:04.495 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:04.496 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:04.496 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:04.496 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:04.496 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:15:04.496 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:04.496 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:15:04.497 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:15:04.497 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:04.497 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:15:04.497 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:04.497 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:15:04.497 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:04.497 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:04.498 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:04.498 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:15:04.498 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:04.498 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:15:04.498 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:15:04.499 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:15:04.499 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:04.499 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:15:04Z" 2023-07-29T01:15:04.499 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:15:04.500 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593122.086179_1622.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:12:02.086179Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:15:04Z" } 2023-07-29T01:15:04.500 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593122.086179_1622.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:12:02.086179Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:15:04.501 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593252.941877_1724.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:15:04.501 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593122.086179_1622.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:15:04.501 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593252.941877_1724.1 target=1_1690593122.086179_1622.1 2023-07-29T01:15:09.502 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:15:09.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:15:09.503 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:15:09.556 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:15:09.556 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:15:09.588 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:15:09.496+0000 7f38c3a0f880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:15:09.588 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:15:09.497+0000 7f38c3a0f880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:15:09.602 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:15:09.603 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:15:09.603 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:15:09.603 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:15:09.603 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:15:09.603 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:15:09.603 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:09.604 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:09.604 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:15:09.604 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:09.604 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:15:09.604 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:09.604 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:15:09.605 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:15:09.605 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:09.605 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:15:09.605 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:09.605 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:15:09.605 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:09.605 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:09.606 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:09.606 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:09.606 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:15:09.606 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:09.606 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:15:09.606 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593122.086179_1622.1", 2023-07-29T01:15:09.606 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:09.607 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:15:09.607 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:09.607 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:12:02.086179Z", 2023-07-29T01:15:09.607 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:09.607 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:09.607 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:09.608 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:09.608 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:15:09.608 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:09.608 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:15:09.608 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:15:09.608 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:09.608 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:15:09.609 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:09.609 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:15:09.609 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:09.609 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:09.609 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:09.609 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:09.609 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:15:09.610 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:09.610 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:15:09.610 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:15:09.610 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:09.610 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:15:09.610 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:09.611 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:15:09.611 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:09.611 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:09.611 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:09.611 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:15:09.611 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:09.611 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:15:09.612 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:15:09.612 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:15:09.612 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:09.612 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:15:09Z" 2023-07-29T01:15:09.612 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:15:09.613 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593122.086179_1622.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:12:02.086179Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:15:09Z" } 2023-07-29T01:15:09.614 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593122.086179_1622.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:12:02.086179Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:15:09.614 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593252.941877_1724.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:15:09.614 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593122.086179_1622.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:15:09.615 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593252.941877_1724.1 target=1_1690593122.086179_1622.1 2023-07-29T01:15:14.616 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:15:14.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:15:14.617 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:15:14.671 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:15:14.671 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:15:14.703 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:15:14.611+0000 7fc8d4e24880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:15:14.704 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:15:14.612+0000 7fc8d4e24880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:15:14.718 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:15:14.718 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:15:14.718 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:15:14.718 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:15:14.718 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:15:14.719 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:15:14.719 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:14.719 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:14.719 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:15:14.719 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:14.720 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:15:14.720 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:14.720 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:15:14.720 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:15:14.720 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:14.720 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:15:14.721 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:14.721 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:15:14.721 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:14.721 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:14.721 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:14.721 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:14.722 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:15:14.722 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:14.722 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:15:14.722 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593122.086179_1622.1", 2023-07-29T01:15:14.722 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:14.722 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:15:14.723 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:14.723 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:12:02.086179Z", 2023-07-29T01:15:14.723 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:14.723 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:14.723 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:14.723 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:14.723 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:15:14.724 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:14.724 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:15:14.724 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:15:14.724 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:14.724 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:15:14.724 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:14.725 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:15:14.725 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:14.725 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:14.725 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:14.725 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:14.725 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:15:14.726 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:14.726 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:15:14.726 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:15:14.726 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:14.726 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:15:14.726 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:14.726 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:15:14.727 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:14.727 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:14.727 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:14.727 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:15:14.727 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:14.727 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:15:14.727 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:15:14.728 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:15:14.728 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:14.728 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:15:14Z" 2023-07-29T01:15:14.728 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:15:14.729 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593122.086179_1622.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:12:02.086179Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:15:14Z" } 2023-07-29T01:15:14.730 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593122.086179_1622.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:12:02.086179Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:15:14.730 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593252.941877_1724.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:15:14.730 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593122.086179_1622.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:15:14.731 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593252.941877_1724.1 target=1_1690593122.086179_1622.1 2023-07-29T01:15:19.732 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:15:19.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:15:19.734 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:15:19.787 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:15:19.788 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:15:19.820 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:15:19.728+0000 7f5b7bb24880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:15:19.821 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:15:19.729+0000 7f5b7bb24880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:15:19.834 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:15:19.835 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:15:19.835 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:15:19.835 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:15:19.835 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:15:19.835 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:15:19.836 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:19.836 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:19.836 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:15:19.836 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:19.836 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:15:19.836 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:19.837 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:15:19.837 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:15:19.837 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:19.837 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:15:19.837 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:19.837 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:15:19.838 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:19.838 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:19.838 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:19.838 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:19.838 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:15:19.838 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:19.838 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:15:19.839 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593122.086179_1622.1", 2023-07-29T01:15:19.839 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:19.839 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:15:19.839 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:19.839 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:12:02.086179Z", 2023-07-29T01:15:19.839 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:19.839 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:19.840 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:19.840 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:19.840 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:15:19.840 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:19.840 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:15:19.840 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:15:19.840 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:19.841 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:15:19.841 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:19.841 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:15:19.841 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:19.841 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:19.841 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:19.841 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:19.842 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:15:19.842 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:19.842 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:15:19.842 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:15:19.842 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:19.842 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:15:19.842 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:19.843 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:15:19.843 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:19.843 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:19.843 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:19.843 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:15:19.843 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:19.844 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:15:19.844 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:15:19.844 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:15:19.844 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:19.844 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:15:19Z" 2023-07-29T01:15:19.844 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:15:19.845 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593122.086179_1622.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:12:02.086179Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:15:19Z" } 2023-07-29T01:15:19.846 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593122.086179_1622.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:12:02.086179Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:15:19.846 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593252.941877_1724.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:15:19.846 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593122.086179_1622.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:15:19.847 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593252.941877_1724.1 target=1_1690593122.086179_1622.1 2023-07-29T01:15:24.848 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:15:24.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:15:24.849 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:15:24.904 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:15:24.904 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:15:24.937 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:15:24.845+0000 7f3cbf89d880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:15:24.938 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:15:24.846+0000 7f3cbf89d880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:15:24.951 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:15:24.952 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:15:24.952 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:15:24.952 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:15:24.952 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:15:24.952 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:15:24.952 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:24.953 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:24.953 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:15:24.953 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:24.953 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:15:24.953 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:24.953 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:15:24.954 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:15:24.954 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:24.954 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:15:24.954 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:24.954 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:15:24.954 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:24.954 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:24.955 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:24.955 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:24.955 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:15:24.955 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:24.955 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:15:24.955 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593122.086179_1622.1", 2023-07-29T01:15:24.955 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:24.956 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:15:24.956 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:24.956 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:12:02.086179Z", 2023-07-29T01:15:24.956 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:24.956 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:24.956 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:24.956 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:24.957 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:15:24.957 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:24.957 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:15:24.957 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:15:24.957 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:24.957 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:15:24.957 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:24.958 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:15:24.958 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:24.958 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:24.958 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:24.958 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:24.958 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:15:24.958 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:24.959 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:15:24.959 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:15:24.959 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:24.959 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:15:24.959 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:24.959 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:15:24.959 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:24.960 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:24.960 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:24.960 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:15:24.960 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:24.960 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:15:24.960 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:15:24.961 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:15:24.961 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:24.961 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:15:24Z" 2023-07-29T01:15:24.961 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:15:24.962 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593122.086179_1622.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:12:02.086179Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:15:24Z" } 2023-07-29T01:15:24.962 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593122.086179_1622.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:12:02.086179Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:15:24.963 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593252.941877_1724.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:15:24.963 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593122.086179_1622.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:15:24.963 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593252.941877_1724.1 target=1_1690593122.086179_1622.1 2023-07-29T01:15:29.965 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:15:29.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:15:29.966 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:15:30.019 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:15:30.020 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:15:30.052 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:15:29.960+0000 7f05cb267880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:15:30.060 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:15:29.961+0000 7f05cb267880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:15:30.067 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:15:30.067 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:15:30.068 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:15:30.068 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:15:30.069 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:15:30.069 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:15:30.069 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:30.069 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:30.070 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:15:30.070 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:30.070 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:15:30.071 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:30.071 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:15:30.071 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:15:30.072 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:30.072 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:15:30.072 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:30.073 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:15:30.073 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:30.073 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:30.074 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:30.074 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:30.074 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:15:30.075 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:30.075 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:15:30.075 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593122.086179_1622.1", 2023-07-29T01:15:30.075 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:30.076 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:15:30.076 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:30.076 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:12:02.086179Z", 2023-07-29T01:15:30.077 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:30.077 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:30.077 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:30.077 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:30.078 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:15:30.078 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:30.078 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:15:30.079 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:15:30.079 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:30.079 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:15:30.080 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:30.080 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:15:30.080 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:30.081 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:30.081 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:30.081 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:30.082 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:15:30.082 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:30.082 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:15:30.082 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:15:30.083 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:30.083 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:15:30.083 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:30.084 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:15:30.084 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:30.084 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:30.085 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:30.085 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:15:30.085 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:30.086 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:15:30.086 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:15:30.086 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:15:30.086 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:30.086 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:15:29Z" 2023-07-29T01:15:30.086 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:15:30.088 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593122.086179_1622.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:12:02.086179Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:15:29Z" } 2023-07-29T01:15:30.088 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593122.086179_1622.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:12:02.086179Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:15:30.089 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593252.941877_1724.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:15:30.089 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593122.086179_1622.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:15:30.089 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593252.941877_1724.1 target=1_1690593122.086179_1622.1 2023-07-29T01:15:35.091 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:15:35.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:15:35.092 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:15:35.146 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:15:35.146 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:15:35.179 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:15:35.087+0000 7f9bc4f26880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:15:35.179 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:15:35.088+0000 7f9bc4f26880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:15:35.193 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:15:35.194 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:15:35.194 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:15:35.194 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:15:35.194 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:15:35.194 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:15:35.194 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:35.195 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:35.195 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:15:35.195 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:35.195 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:15:35.195 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:35.195 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:15:35.196 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:15:35.196 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:35.196 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:15:35.196 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:35.196 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:15:35.196 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:35.196 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:35.197 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:35.197 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:35.197 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:15:35.197 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:35.197 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:15:35.197 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593122.086179_1622.1", 2023-07-29T01:15:35.197 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:35.198 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:15:35.198 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:35.198 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:12:02.086179Z", 2023-07-29T01:15:35.198 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:35.198 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:35.198 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:35.199 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:35.199 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:15:35.199 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:35.199 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:15:35.199 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:15:35.199 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:35.199 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:15:35.200 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:35.200 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:15:35.200 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:35.200 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:35.200 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:35.200 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:35.200 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:15:35.201 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:35.201 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:15:35.201 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:15:35.201 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:35.201 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:15:35.201 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:35.201 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:15:35.202 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:35.202 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:35.202 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:35.202 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:15:35.202 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:35.202 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:15:35.202 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:15:35.203 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:15:35.203 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:35.203 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:15:35Z" 2023-07-29T01:15:35.203 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:15:35.204 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593122.086179_1622.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:12:02.086179Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:15:35Z" } 2023-07-29T01:15:35.204 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593122.086179_1622.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:12:02.086179Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:15:35.205 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593252.941877_1724.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:15:35.205 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593122.086179_1622.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:15:35.205 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593252.941877_1724.1 target=1_1690593122.086179_1622.1 2023-07-29T01:15:40.207 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:15:40.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:15:40.208 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:15:40.261 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:15:40.262 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:15:40.294 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:15:40.202+0000 7efc5ff35880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:15:40.294 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:15:40.203+0000 7efc5ff35880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:15:40.308 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:15:40.309 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:15:40.309 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:15:40.309 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:15:40.309 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:15:40.309 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:15:40.310 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:40.310 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:40.310 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:15:40.310 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:40.310 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:15:40.310 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:40.310 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:15:40.311 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:15:40.311 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:40.311 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:15:40.311 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:40.311 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:15:40.311 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:40.311 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:40.312 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:40.312 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:40.312 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:15:40.312 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:40.312 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:15:40.312 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593122.086179_1622.1", 2023-07-29T01:15:40.312 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:40.313 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:15:40.313 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:40.313 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:12:02.086179Z", 2023-07-29T01:15:40.313 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:40.313 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:40.313 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:40.314 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:40.314 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:15:40.314 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:40.314 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:15:40.314 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:15:40.314 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:40.314 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:15:40.315 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:40.315 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:15:40.315 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:40.315 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:40.315 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:40.315 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:40.315 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:15:40.316 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:40.316 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:15:40.316 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:15:40.316 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:40.316 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:15:40.316 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:40.316 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:15:40.317 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:40.317 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:40.317 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:40.317 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:15:40.317 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:40.317 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:15:40.317 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:15:40.318 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:15:40.318 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:40.318 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:15:40Z" 2023-07-29T01:15:40.318 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:15:40.319 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593122.086179_1622.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:12:02.086179Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:15:40Z" } 2023-07-29T01:15:40.319 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593122.086179_1622.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:12:02.086179Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:15:40.320 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593252.941877_1724.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:15:40.320 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593122.086179_1622.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:15:40.320 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593252.941877_1724.1 target=1_1690593122.086179_1622.1 2023-07-29T01:15:45.322 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:15:45.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:15:45.323 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:15:45.376 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:15:45.377 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:15:45.409 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:15:45.317+0000 7f68ae40f880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:15:45.410 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:15:45.318+0000 7f68ae40f880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:15:45.423 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:15:45.424 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:15:45.424 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:15:45.424 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:15:45.424 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:15:45.424 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:15:45.425 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:45.425 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:45.425 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:15:45.425 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:45.425 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:15:45.425 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:45.426 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:15:45.426 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:15:45.426 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:45.426 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:15:45.426 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:45.426 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:15:45.426 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:45.427 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:45.427 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:45.427 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:45.427 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:15:45.427 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:45.427 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:15:45.427 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593252.941877_1724.1", 2023-07-29T01:15:45.428 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:45.428 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:15:45.428 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:45.428 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:14:12.941877Z", 2023-07-29T01:15:45.428 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:45.428 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:45.428 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:45.429 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:45.429 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:15:45.429 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:45.429 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:15:45.429 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:15:45.429 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:45.429 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:15:45.430 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:45.430 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:15:45.430 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:45.430 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:45.430 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:45.430 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:45.431 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:15:45.431 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:45.431 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:15:45.431 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:15:45.431 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:45.431 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:15:45.431 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:45.432 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:15:45.432 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:45.432 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:45.432 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:45.432 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:15:45.432 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:45.432 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:15:45.433 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:15:45.433 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:15:45.433 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:45.433 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:15:45Z" 2023-07-29T01:15:45.433 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:15:45.434 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593252.941877_1724.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:14:12.941877Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:15:45Z" } 2023-07-29T01:15:45.435 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593252.941877_1724.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:14:12.941877Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:15:45.435 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593252.941877_1724.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:15:45.435 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593252.941877_1724.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:15:45.436 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T01:15:45.436 DEBUG:boto:path=/hiiqos-70/ 2023-07-29T01:15:45.436 DEBUG:boto:auth_path=/hiiqos-70/ 2023-07-29T01:15:45.437 DEBUG:boto:Method: HEAD 2023-07-29T01:15:45.437 DEBUG:boto:Path: /hiiqos-70/ 2023-07-29T01:15:45.437 DEBUG:boto:Data: 2023-07-29T01:15:45.438 DEBUG:boto:Headers: {} 2023-07-29T01:15:45.438 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:15:45.438 DEBUG:boto:Port: 8000 2023-07-29T01:15:45.439 DEBUG:boto:Params: {} 2023-07-29T01:15:45.439 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-29T01:15:45.440 DEBUG:boto:Token: None 2023-07-29T01:15:45.440 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:15:45 GMT /hiiqos-70/ 2023-07-29T01:15:45.441 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LzIX1pJlJUci6bFRcYZWuQKJTyU= 2023-07-29T01:15:45.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:15:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LzIX1pJlJUci6bFRcYZWuQKJTyU=', 'Content-Length': '0'} 2023-07-29T01:15:45.446 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e849ca7667f2ea28-0064c46841-283f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:15:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:15:45.447 DEBUG:boto:path=/hiiqos-70/obj5 2023-07-29T01:15:45.448 DEBUG:boto:auth_path=/hiiqos-70/obj5 2023-07-29T01:15:45.448 DEBUG:boto:Method: PUT 2023-07-29T01:15:45.448 DEBUG:boto:Path: /hiiqos-70/obj5 2023-07-29T01:15:45.449 DEBUG:boto:Data: 2023-07-29T01:15:45.449 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T01:15:45.449 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:15:45.450 DEBUG:boto:Port: 8000 2023-07-29T01:15:45.450 DEBUG:boto:Params: {} 2023-07-29T01:15:45.450 DEBUG:boto:Token: None 2023-07-29T01:15:45.451 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 01:15:45 GMT /hiiqos-70/obj5 2023-07-29T01:15:45.451 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gK8iYOsoYkCwEuxcpWD6xceVkQ0= 2023-07-29T01:15:45.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 01:15:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gK8iYOsoYkCwEuxcpWD6xceVkQ0='} 2023-07-29T01:15:45.463 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000939920e6861bd911-0064c46841-283f-a1'), ('Date', 'Sat, 29 Jul 2023 01:15:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:15:45.463 DEBUG:boto:path=/hiiqos-70/ 2023-07-29T01:15:45.464 DEBUG:boto:auth_path=/hiiqos-70/ 2023-07-29T01:15:45.464 DEBUG:boto:Method: HEAD 2023-07-29T01:15:45.465 DEBUG:boto:Path: /hiiqos-70/ 2023-07-29T01:15:45.465 DEBUG:boto:Data: 2023-07-29T01:15:45.465 DEBUG:boto:Headers: {} 2023-07-29T01:15:45.466 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:15:45.466 DEBUG:boto:Port: 8000 2023-07-29T01:15:45.466 DEBUG:boto:Params: {} 2023-07-29T01:15:45.467 DEBUG:boto:Token: None 2023-07-29T01:15:45.467 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:15:45 GMT /hiiqos-70/ 2023-07-29T01:15:45.467 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LzIX1pJlJUci6bFRcYZWuQKJTyU= 2023-07-29T01:15:45.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:15:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LzIX1pJlJUci6bFRcYZWuQKJTyU=', 'Content-Length': '0'} 2023-07-29T01:15:45.470 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c21013b4d2be6738-0064c46841-283f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:15:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:15:45.471 DEBUG:boto:path=/hiiqos-70/obj6 2023-07-29T01:15:45.471 DEBUG:boto:auth_path=/hiiqos-70/obj6 2023-07-29T01:15:45.471 DEBUG:boto:Method: PUT 2023-07-29T01:15:45.472 DEBUG:boto:Path: /hiiqos-70/obj6 2023-07-29T01:15:45.472 DEBUG:boto:Data: 2023-07-29T01:15:45.472 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T01:15:45.473 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:15:45.473 DEBUG:boto:Port: 8000 2023-07-29T01:15:45.473 DEBUG:boto:Params: {} 2023-07-29T01:15:45.474 DEBUG:boto:Token: None 2023-07-29T01:15:45.474 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 01:15:45 GMT /hiiqos-70/obj6 2023-07-29T01:15:45.475 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:a5cCb4vXQ+u7Xge0sLnfwrsTcaE= 2023-07-29T01:15:45.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 01:15:45 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:a5cCb4vXQ+u7Xge0sLnfwrsTcaE='} 2023-07-29T01:15:45.482 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000024d31c202c6b2a51-0064c46841-283f-a1'), ('Date', 'Sat, 29 Jul 2023 01:15:45 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:15:45.482 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:15:45.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:15:45.483 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:15:45.529 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:15:45.529 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:15:45.559 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T01:15:45.571 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T01:15:45.572 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:45.572 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:15:45.572 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:17.836402Z" 2023-07-29T01:15:45.572 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:45.572 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:45.573 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593252.941877_1724.1", 2023-07-29T01:15:45.573 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:14:12.941877Z" 2023-07-29T01:15:45.573 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:45.573 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:45.573 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:15:45.573 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T01:15:45.573 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:45.574 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:45.574 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:15:45.574 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:12.811406Z" 2023-07-29T01:15:45.574 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:45.574 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T01:15:45.575 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690591217.836402_294.1', 1: '1_1690593252.941877_1724.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:15:45.576 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T01:15:45.576 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:15:45.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:15:45.577 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:15:45.622 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:15:45.623 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:15:45.651 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:15:45.559+0000 7fdeae012880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:15:45.652 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:15:45.560+0000 7fdeae012880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:15:45.665 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:15:45.665 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:15:45.665 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:15:45.665 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:15:45.666 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:15:45.666 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:15:45.666 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:45.666 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:45.666 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:15:45.666 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:45.666 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:15:45.667 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:45.667 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:15:45.667 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:15:45.667 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:45.667 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:15:45.667 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:45.667 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:15:45.668 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:45.668 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:45.668 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:45.668 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:45.668 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:15:45.668 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:45.669 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:15:45.669 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593252.941877_1724.1", 2023-07-29T01:15:45.669 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:45.669 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:15:45.669 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:45.669 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:14:12.941877Z", 2023-07-29T01:15:45.669 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:45.670 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:45.670 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:45.670 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:45.670 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:15:45.670 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:45.670 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:15:45.670 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:15:45.671 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:45.671 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:15:45.671 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:45.671 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:15:45.671 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:45.671 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:45.671 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:45.672 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:45.672 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:15:45.672 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:45.672 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:15:45.672 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:15:45.672 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:45.672 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:15:45.673 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:45.673 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:15:45.673 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:15:45.673 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:45.673 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:45.673 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:15:45.673 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:45.674 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:15:45.674 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:15:45.674 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:15:45.674 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:45.674 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:15:45Z" 2023-07-29T01:15:45.674 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:15:45.675 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593252.941877_1724.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:14:12.941877Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:15:45Z" } 2023-07-29T01:15:45.676 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593252.941877_1724.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:14:12.941877Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:15:45.676 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593252.941877_1724.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:15:45.676 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593252.941877_1724.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:15:45.677 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T01:15:45.677 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:15:45.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:15:45.678 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:15:45.722 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:15:45.723 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:15:45.771 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T01:15:45.771 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:45.771 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:15:45.771 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:15:45.772 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:45.772 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:45.772 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044881", 2023-07-29T01:15:45.772 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:07:23.001407Z" 2023-07-29T01:15:45.772 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:45.772 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:45.773 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:15:45.773 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:15:45.773 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:45.773 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:45.773 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:15:45.773 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:15:45.773 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:45.774 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T01:15:45.775 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:15:45.775 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-07-29T01:15:45.775 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:15:45.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:15:45.776 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:15:45.824 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:15:45.824 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:15:45.866 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:15:45.866 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:15:45.866 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:15:45.867 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:15:45.867 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:15:45.867 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:15:45.867 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:45.867 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:15:45.867 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:45.868 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:15:45.868 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:45.868 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:15:45.868 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052093", 2023-07-29T01:15:45.868 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:45.868 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:15:45.869 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:45.869 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:12:08.229055Z" 2023-07-29T01:15:45.869 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:45.869 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:45.869 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:45.870 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:15:45.870 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:45.870 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:15:45.870 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044881", 2023-07-29T01:15:45.870 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:45.870 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:15:45.871 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:45.871 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:23.001407Z" 2023-07-29T01:15:45.871 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:45.871 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:45.871 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:45.871 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:15:45.872 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:45.872 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:15:45.872 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045800", 2023-07-29T01:15:45.872 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:45.872 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:15:45.872 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:45.872 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:12:08.266555Z" 2023-07-29T01:15:45.873 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:45.873 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:45.873 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:45.873 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:15:45.873 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:45.873 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:15:45.874 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:15:45.874 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:45.874 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:15:45.874 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:45.874 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:15:45.874 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:45.874 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:45.875 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:15:45.875 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:45.875 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:15:45.875 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:15:45.876 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:15:45.876 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:45.876 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:15:45Z" 2023-07-29T01:15:45.876 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:15:45.877 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052093", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:12:08.229055Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044881", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:07:23.001407Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045800", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:12:08.266555Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:15:45Z" } 2023-07-29T01:15:45.878 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052093', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:12:08.229055Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044881', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:07:23.001407Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045800', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:12:08.266555Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:15:45.879 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:15:45.879 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052093', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045800', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:15:45.880 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052233 target=00000000000000000000:00000000000000052093, shard=2 master=00000000000000000000:00000000000000045940 target=00000000000000000000:00000000000000045800 2023-07-29T01:15:50.881 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:15:50.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:15:50.882 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:15:50.936 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:15:50.937 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:15:50.984 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:15:50.985 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:15:50.985 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:15:50.985 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:15:50.985 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:15:50.985 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:15:50.985 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:50.986 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:15:50.986 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:50.986 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:15:50.986 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:50.986 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:15:50.986 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052093", 2023-07-29T01:15:50.987 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:50.987 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:15:50.987 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:50.987 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:12:08.229055Z" 2023-07-29T01:15:50.987 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:50.987 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:50.987 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:50.988 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:15:50.988 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:50.988 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:15:50.988 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044881", 2023-07-29T01:15:50.988 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:50.988 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:15:50.989 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:50.989 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:23.001407Z" 2023-07-29T01:15:50.989 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:50.989 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:50.989 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:50.989 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:15:50.989 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:50.990 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:15:50.990 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045800", 2023-07-29T01:15:50.990 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:50.990 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:15:50.990 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:50.990 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:12:08.266555Z" 2023-07-29T01:15:50.990 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:50.991 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:50.991 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:50.991 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:15:50.991 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:50.991 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:15:50.991 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:15:50.991 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:50.992 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:15:50.992 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:50.992 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:15:50.992 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:50.992 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:50.992 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:15:50.992 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:50.993 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:15:50.993 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:15:50.993 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:15:50.993 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:50.993 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:15:50Z" 2023-07-29T01:15:50.993 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:15:50.994 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052093", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:12:08.229055Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044881", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:07:23.001407Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045800", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:12:08.266555Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:15:50Z" } 2023-07-29T01:15:50.995 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052093', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:12:08.229055Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044881', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:07:23.001407Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045800', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:12:08.266555Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:15:50.995 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:15:50.995 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052093', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045800', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:15:50.996 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052233 target=00000000000000000000:00000000000000052093, shard=2 master=00000000000000000000:00000000000000045940 target=00000000000000000000:00000000000000045800 2023-07-29T01:15:55.997 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:15:55.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:15:55.998 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:15:56.055 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:15:56.055 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:15:56.103 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:15:56.103 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:15:56.103 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:15:56.104 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:15:56.104 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:15:56.104 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:15:56.104 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:56.105 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:15:56.105 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:56.105 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:15:56.105 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:56.106 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:15:56.106 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052093", 2023-07-29T01:15:56.106 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:56.106 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:15:56.106 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:56.106 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:12:08.229055Z" 2023-07-29T01:15:56.106 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:56.107 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:56.107 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:56.107 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:15:56.107 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:56.107 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:15:56.107 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044881", 2023-07-29T01:15:56.107 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:56.108 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:15:56.108 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:56.108 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:23.001407Z" 2023-07-29T01:15:56.108 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:56.108 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:56.109 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:56.109 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:15:56.109 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:56.110 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:15:56.110 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045800", 2023-07-29T01:15:56.110 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:56.111 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:15:56.111 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:56.111 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:12:08.266555Z" 2023-07-29T01:15:56.111 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:56.112 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:56.112 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:15:56.112 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:15:56.113 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:15:56.113 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:15:56.113 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:15:56.114 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:15:56.114 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:15:56.114 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:15:56.115 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:15:56.115 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:56.115 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:15:56.115 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:15:56.116 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:56.116 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:15:56.116 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:15:56.117 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:15:56.117 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:15:56.117 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:15:55Z" 2023-07-29T01:15:56.117 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:15:56.119 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052093", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:12:08.229055Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044881", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:07:23.001407Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045800", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:12:08.266555Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:15:55Z" } 2023-07-29T01:15:56.119 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052093', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:12:08.229055Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044881', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:07:23.001407Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045800', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:12:08.266555Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:15:56.120 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:15:56.121 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052093', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045800', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:15:56.122 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052233 target=00000000000000000000:00000000000000052093, shard=2 master=00000000000000000000:00000000000000045940 target=00000000000000000000:00000000000000045800 2023-07-29T01:16:01.123 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:16:01.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:16:01.124 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:16:01.179 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:16:01.179 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:16:01.228 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:16:01.228 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:16:01.229 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:16:01.229 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:16:01.229 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:16:01.229 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:16:01.230 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:01.230 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:16:01.230 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:01.230 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:16:01.230 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:01.230 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:16:01.230 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052093", 2023-07-29T01:16:01.231 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:01.231 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:16:01.231 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:01.231 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:12:08.229055Z" 2023-07-29T01:16:01.231 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:01.231 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:01.231 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:01.232 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:16:01.232 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:01.232 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:16:01.232 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044881", 2023-07-29T01:16:01.232 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:01.232 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:16:01.232 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:01.233 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:23.001407Z" 2023-07-29T01:16:01.233 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:01.233 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:01.233 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:01.233 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:16:01.233 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:01.233 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:16:01.234 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045800", 2023-07-29T01:16:01.234 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:01.234 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:16:01.234 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:01.234 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:12:08.266555Z" 2023-07-29T01:16:01.234 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:01.235 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:01.235 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:01.235 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:16:01.235 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:01.235 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:16:01.235 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:16:01.235 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:01.236 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:16:01.236 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:01.236 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:16:01.236 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:01.236 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:01.236 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:16:01.236 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:01.237 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:16:01.237 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:16:01.237 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:16:01.237 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:01.237 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:16:01Z" 2023-07-29T01:16:01.237 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:16:01.238 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052093", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:12:08.229055Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044881", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:07:23.001407Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045800", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:12:08.266555Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:16:01Z" } 2023-07-29T01:16:01.239 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052093', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:12:08.229055Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044881', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:07:23.001407Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045800', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:12:08.266555Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:16:01.239 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:16:01.239 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052093', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045800', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:16:01.240 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052233 target=00000000000000000000:00000000000000052093, shard=2 master=00000000000000000000:00000000000000045940 target=00000000000000000000:00000000000000045800 2023-07-29T01:16:06.241 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:16:06.242 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:16:06.242 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:16:06.298 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:16:06.298 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:16:06.346 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:16:06.346 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:16:06.347 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:16:06.347 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:16:06.347 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:16:06.347 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:16:06.347 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:06.347 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:16:06.348 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:06.348 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:16:06.348 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:06.348 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:16:06.348 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052093", 2023-07-29T01:16:06.349 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:06.349 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:16:06.349 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:06.349 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:12:08.229055Z" 2023-07-29T01:16:06.349 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:06.349 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:06.350 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:06.350 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:16:06.350 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:06.350 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:16:06.350 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044881", 2023-07-29T01:16:06.350 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:06.351 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:16:06.351 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:06.351 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:23.001407Z" 2023-07-29T01:16:06.351 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:06.351 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:06.351 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:06.352 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:16:06.352 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:06.352 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:16:06.352 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045800", 2023-07-29T01:16:06.352 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:06.352 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:16:06.352 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:06.353 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:12:08.266555Z" 2023-07-29T01:16:06.353 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:06.353 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:06.353 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:06.354 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:16:06.354 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:06.354 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:16:06.354 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:16:06.354 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:06.354 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:16:06.355 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:06.355 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:16:06.355 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:06.355 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:06.355 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:16:06.355 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:06.355 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:16:06.356 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:16:06.356 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:16:06.356 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:06.356 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:16:06Z" 2023-07-29T01:16:06.356 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:16:06.357 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052093", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:12:08.229055Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044881", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:07:23.001407Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045800", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:12:08.266555Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:16:06Z" } 2023-07-29T01:16:06.358 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052093', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:12:08.229055Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044881', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:07:23.001407Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045800', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:12:08.266555Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:16:06.358 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:16:06.359 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052093', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045800', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:16:06.359 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052233 target=00000000000000000000:00000000000000052093, shard=2 master=00000000000000000000:00000000000000045940 target=00000000000000000000:00000000000000045800 2023-07-29T01:16:11.360 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:16:11.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:16:11.361 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:16:11.416 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:16:11.416 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:16:11.468 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:16:11.468 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:16:11.469 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:16:11.469 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:16:11.469 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:16:11.469 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:16:11.469 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:11.469 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:16:11.470 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:11.470 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:16:11.470 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:11.470 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:16:11.470 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052093", 2023-07-29T01:16:11.471 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:11.471 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:16:11.471 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:11.471 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:12:08.229055Z" 2023-07-29T01:16:11.471 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:11.471 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:11.472 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:11.472 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:16:11.472 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:11.472 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:16:11.472 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044881", 2023-07-29T01:16:11.473 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:11.473 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:16:11.473 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:11.473 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:23.001407Z" 2023-07-29T01:16:11.473 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:11.473 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:11.474 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:11.474 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:16:11.474 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:11.474 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:16:11.474 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045800", 2023-07-29T01:16:11.475 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:11.475 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:16:11.475 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:11.476 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:12:08.266555Z" 2023-07-29T01:16:11.476 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:11.476 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:11.476 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:11.476 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:16:11.477 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:11.477 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:16:11.477 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:16:11.477 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:11.477 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:16:11.478 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:11.478 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:16:11.478 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:11.478 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:11.478 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:16:11.478 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:11.478 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:16:11.479 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:16:11.479 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:16:11.479 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:11.479 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:16:11Z" 2023-07-29T01:16:11.479 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:16:11.480 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052093", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:12:08.229055Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044881", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:07:23.001407Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045800", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:12:08.266555Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:16:11Z" } 2023-07-29T01:16:11.481 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052093', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:12:08.229055Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044881', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:07:23.001407Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045800', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:12:08.266555Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:16:11.481 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:16:11.482 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052093', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045800', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:16:11.482 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052233 target=00000000000000000000:00000000000000052093, shard=2 master=00000000000000000000:00000000000000045940 target=00000000000000000000:00000000000000045800 2023-07-29T01:16:16.483 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:16:16.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:16:16.485 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:16:16.540 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:16:16.541 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:16:16.588 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:16:16.589 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:16:16.589 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:16:16.590 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:16:16.590 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:16:16.590 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:16:16.591 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:16.591 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:16:16.591 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:16.592 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:16:16.592 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:16.592 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:16:16.593 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:16:16.593 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:16.594 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:16:16.594 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:16.594 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:16:16.594 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:16.595 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:16.595 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:16.595 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:16:16.595 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:16.596 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:16:16.596 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000044881", 2023-07-29T01:16:16.596 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:16.597 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:16:16.597 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:16.597 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:07:23.001407Z" 2023-07-29T01:16:16.597 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:16.598 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:16.598 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:16.598 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:16:16.598 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:16.599 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:16:16.599 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:16:16.599 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:16.599 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:16:16.600 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:16.600 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:16:16.600 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:16.600 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:16.600 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:16.601 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:16:16.601 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:16.601 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:16:16.601 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:16:16.601 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:16.602 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:16:16.602 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:16.602 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:16:16.602 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:16.602 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:16.603 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:16:16.603 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:16.603 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:16:16.603 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:16:16.603 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:16:16.603 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:16.603 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:16:16Z" 2023-07-29T01:16:16.604 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:16:16.606 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052233", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:15:45.371644Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000044881", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:07:23.001407Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045940", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:15:45.390653Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:58:25.189790Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:16:16Z" } 2023-07-29T01:16:16.607 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052233', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.371644Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044881', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:07:23.001407Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045940', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.390653Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:16:16.607 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:16:16.608 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000044881', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:16:16.609 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-07-29T01:16:16.610 DEBUG:boto:path=/hiiqos-70/ 2023-07-29T01:16:16.611 DEBUG:boto:auth_path=/hiiqos-70/ 2023-07-29T01:16:16.611 DEBUG:boto:Method: HEAD 2023-07-29T01:16:16.612 DEBUG:boto:Path: /hiiqos-70/ 2023-07-29T01:16:16.613 DEBUG:boto:Data: 2023-07-29T01:16:16.613 DEBUG:boto:Headers: {} 2023-07-29T01:16:16.614 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:16:16.615 DEBUG:boto:Port: 8001 2023-07-29T01:16:16.615 DEBUG:boto:Params: {} 2023-07-29T01:16:16.616 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-29T01:16:16.616 DEBUG:boto:Token: None 2023-07-29T01:16:16.617 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:16:16 GMT /hiiqos-70/ 2023-07-29T01:16:16.617 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:420KTRoZhkFa/xVtLAUQ5AcYqdQ= 2023-07-29T01:16:16.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:16:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:420KTRoZhkFa/xVtLAUQ5AcYqdQ=', 'Content-Length': '0'} 2023-07-29T01:16:21.625 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003862ea0e6a5dd0d3-0064c46865-284a-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:16:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:16:21.626 DEBUG:boto:path=/hiiqos-70/obj5 2023-07-29T01:16:21.626 DEBUG:boto:auth_path=/hiiqos-70/obj5 2023-07-29T01:16:21.626 DEBUG:boto:Method: HEAD 2023-07-29T01:16:21.627 DEBUG:boto:Path: /hiiqos-70/obj5 2023-07-29T01:16:21.627 DEBUG:boto:Data: 2023-07-29T01:16:21.627 DEBUG:boto:Headers: {} 2023-07-29T01:16:21.628 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:16:21.628 DEBUG:boto:Port: 8001 2023-07-29T01:16:21.628 DEBUG:boto:Params: {} 2023-07-29T01:16:21.629 DEBUG:boto:Token: None 2023-07-29T01:16:21.629 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:16:21 GMT /hiiqos-70/obj5 2023-07-29T01:16:21.630 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:nAfd8mMZFBe2VIqwOXezXEo98GI= 2023-07-29T01:16:21.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:16:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nAfd8mMZFBe2VIqwOXezXEo98GI=', 'Content-Length': '0'} 2023-07-29T01:16:21.632 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000a43837b6bcc533e3-0064c46865-284a-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 01:16:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:16:21.632 DEBUG:boto:path=/hiiqos-70/obj6 2023-07-29T01:16:21.632 DEBUG:boto:auth_path=/hiiqos-70/obj6 2023-07-29T01:16:21.633 DEBUG:boto:Method: HEAD 2023-07-29T01:16:21.633 DEBUG:boto:Path: /hiiqos-70/obj6 2023-07-29T01:16:21.633 DEBUG:boto:Data: 2023-07-29T01:16:21.634 DEBUG:boto:Headers: {} 2023-07-29T01:16:21.634 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:16:21.634 DEBUG:boto:Port: 8001 2023-07-29T01:16:21.635 DEBUG:boto:Params: {} 2023-07-29T01:16:21.635 DEBUG:boto:Token: None 2023-07-29T01:16:21.636 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:16:21 GMT /hiiqos-70/obj6 2023-07-29T01:16:21.636 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:hHb0K+4FVXkVaPbv2hU/eNjhY5I= 2023-07-29T01:16:21.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:16:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:hHb0K+4FVXkVaPbv2hU/eNjhY5I=', 'Content-Length': '0'} 2023-07-29T01:16:21.638 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000c289357df20c06ce-0064c46865-284a-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 01:16:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:16:21.639 DEBUG:boto:path=/hiiqos-71/ 2023-07-29T01:16:21.639 DEBUG:boto:auth_path=/hiiqos-71/ 2023-07-29T01:16:21.639 DEBUG:boto:Method: HEAD 2023-07-29T01:16:21.640 DEBUG:boto:Path: /hiiqos-71/ 2023-07-29T01:16:21.640 DEBUG:boto:Data: 2023-07-29T01:16:21.640 DEBUG:boto:Headers: {} 2023-07-29T01:16:21.641 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:16:21.641 DEBUG:boto:Port: 8001 2023-07-29T01:16:21.641 DEBUG:boto:Params: {} 2023-07-29T01:16:21.642 DEBUG:boto:Token: None 2023-07-29T01:16:21.642 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:16:21 GMT /hiiqos-71/ 2023-07-29T01:16:21.642 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:k9ItgDNalw5Wgp4+HuT49cn+cZo= 2023-07-29T01:16:21.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:16:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:k9ItgDNalw5Wgp4+HuT49cn+cZo=', 'Content-Length': '0'} 2023-07-29T01:16:21.645 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d5defc5eed9cfc53-0064c46865-284a-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:16:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:16:21.645 DEBUG:boto:path=/hiiqos-71/obj5 2023-07-29T01:16:21.646 DEBUG:boto:auth_path=/hiiqos-71/obj5 2023-07-29T01:16:21.646 DEBUG:boto:Method: HEAD 2023-07-29T01:16:21.647 DEBUG:boto:Path: /hiiqos-71/obj5 2023-07-29T01:16:21.647 DEBUG:boto:Data: 2023-07-29T01:16:21.648 DEBUG:boto:Headers: {} 2023-07-29T01:16:21.648 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:16:21.649 DEBUG:boto:Port: 8001 2023-07-29T01:16:21.649 DEBUG:boto:Params: {} 2023-07-29T01:16:21.649 DEBUG:boto:Token: None 2023-07-29T01:16:21.650 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:16:21 GMT /hiiqos-71/obj5 2023-07-29T01:16:21.650 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tBraYkr5fW+ZJgKFy6ZxkTah0pQ= 2023-07-29T01:16:21.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:16:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tBraYkr5fW+ZJgKFy6ZxkTah0pQ=', 'Content-Length': '0'} 2023-07-29T01:16:21.653 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:15:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-70:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.9662.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005691c80a9f8291ed-0064c46865-284a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:16:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:16:21.653 DEBUG:boto:path=/hiiqos-71/obj5 2023-07-29T01:16:21.654 DEBUG:boto:auth_path=/hiiqos-71/obj5 2023-07-29T01:16:21.654 DEBUG:boto:Method: GET 2023-07-29T01:16:21.654 DEBUG:boto:Path: /hiiqos-71/obj5 2023-07-29T01:16:21.655 DEBUG:boto:Data: 2023-07-29T01:16:21.656 DEBUG:boto:Headers: {} 2023-07-29T01:16:21.656 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:16:21.657 DEBUG:boto:Port: 8001 2023-07-29T01:16:21.657 DEBUG:boto:Params: {} 2023-07-29T01:16:21.657 DEBUG:boto:Token: None 2023-07-29T01:16:21.658 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 01:16:21 GMT /hiiqos-71/obj5 2023-07-29T01:16:21.658 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kkGvfUBI1QfzUyE+WgYZL34I8h4= 2023-07-29T01:16:21.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:16:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kkGvfUBI1QfzUyE+WgYZL34I8h4=', 'Content-Length': '0'} 2023-07-29T01:16:21.661 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:15:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-70:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.9662.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd299370747b05d5-0064c46865-284a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:16:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:16:21.708 DEBUG:boto:path=/hiiqos-71/obj6 2023-07-29T01:16:21.709 DEBUG:boto:auth_path=/hiiqos-71/obj6 2023-07-29T01:16:21.709 DEBUG:boto:Method: HEAD 2023-07-29T01:16:21.710 DEBUG:boto:Path: /hiiqos-71/obj6 2023-07-29T01:16:21.710 DEBUG:boto:Data: 2023-07-29T01:16:21.710 DEBUG:boto:Headers: {} 2023-07-29T01:16:21.711 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:16:21.711 DEBUG:boto:Port: 8001 2023-07-29T01:16:21.712 DEBUG:boto:Params: {} 2023-07-29T01:16:21.713 DEBUG:boto:Token: None 2023-07-29T01:16:21.713 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:16:21 GMT /hiiqos-71/obj6 2023-07-29T01:16:21.714 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:irbUZl9kItKdRxQiwL9DlBunsfw= 2023-07-29T01:16:21.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:16:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:irbUZl9kItKdRxQiwL9DlBunsfw=', 'Content-Length': '0'} 2023-07-29T01:16:21.717 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:15:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-70:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.9662.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b0af79bd5e07ebb3-0064c46865-284a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:16:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:16:21.718 DEBUG:boto:path=/hiiqos-71/obj6 2023-07-29T01:16:21.719 DEBUG:boto:auth_path=/hiiqos-71/obj6 2023-07-29T01:16:21.719 DEBUG:boto:Method: GET 2023-07-29T01:16:21.720 DEBUG:boto:Path: /hiiqos-71/obj6 2023-07-29T01:16:21.721 DEBUG:boto:Data: 2023-07-29T01:16:21.721 DEBUG:boto:Headers: {} 2023-07-29T01:16:21.722 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:16:21.723 DEBUG:boto:Port: 8001 2023-07-29T01:16:21.723 DEBUG:boto:Params: {} 2023-07-29T01:16:21.724 DEBUG:boto:Token: None 2023-07-29T01:16:21.725 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 01:16:21 GMT /hiiqos-71/obj6 2023-07-29T01:16:21.726 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9KCbx9FUFjhJyiWNeAB1IRpzCmc= 2023-07-29T01:16:21.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:16:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9KCbx9FUFjhJyiWNeAB1IRpzCmc=', 'Content-Length': '0'} 2023-07-29T01:16:21.730 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:15:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-70:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.9662.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b7badba1ba6a2c7a-0064c46865-284a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:16:21 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:16:21.731 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'hiiqos-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:16:21.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'hiiqos-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:16:21.731 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket hiiqos-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:16:21.785 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:16:21.785 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:16:21.842 INFO:teuthology.orchestra.run.smithi101.stdout:{"sync_policy":{"groups":[]}} 2023-07-29T01:16:21.842 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:16:21.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:16:21.843 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:16:21.889 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:16:21.889 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:16:21.919 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T01:16:21.932 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T01:16:21.933 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:21.933 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:16:21.933 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:17.836402Z" 2023-07-29T01:16:21.933 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:21.933 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:21.933 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593381.731988_1736.1", 2023-07-29T01:16:21.934 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:16:21.731988Z" 2023-07-29T01:16:21.934 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:21.934 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:21.934 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:16:21.934 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T01:16:21.934 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:21.934 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:21.935 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:16:21.935 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:12.811406Z" 2023-07-29T01:16:21.935 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:21.935 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T01:16:21.936 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690591217.836402_294.1', 1: '1_1690593381.731988_1736.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:16:21.936 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T01:16:21.937 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:16:21.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:16:21.937 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:16:21.984 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:16:21.984 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:16:22.013 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:16:21.921+0000 7fe865375880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:16:22.014 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:16:21.922+0000 7fe865375880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:16:22.030 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:16:22.031 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:16:22.031 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:16:22.031 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:16:22.031 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:16:22.031 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:16:22.031 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:22.032 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:22.032 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:16:22.032 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:22.032 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:16:22.032 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:22.032 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:16:22.033 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:16:22.033 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:22.033 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:16:22.033 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:22.033 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:16:22.033 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:22.033 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:22.034 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:22.034 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:22.034 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:16:22.034 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:22.034 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:16:22.034 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593252.941877_1724.1", 2023-07-29T01:16:22.034 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:22.035 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:16:22.035 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:22.035 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:14:12.941877Z", 2023-07-29T01:16:22.035 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:22.035 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:22.035 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:22.035 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:22.036 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:16:22.036 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:22.036 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:16:22.036 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:16:22.036 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:22.036 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:16:22.036 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:22.037 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:16:22.037 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:22.037 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:22.037 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:22.037 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:22.037 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:16:22.037 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:22.038 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:16:22.038 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:16:22.038 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:22.038 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:16:22.038 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:22.038 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:16:22.038 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:22.039 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:22.039 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:22.039 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:16:22.039 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:22.039 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:16:22.039 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:16:22.039 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:16:22.040 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:22.040 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:16:21Z" 2023-07-29T01:16:22.040 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:16:22.041 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593252.941877_1724.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:14:12.941877Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:16:21Z" } 2023-07-29T01:16:22.041 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593252.941877_1724.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:14:12.941877Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:16:22.042 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593381.731988_1736.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:16:22.042 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593252.941877_1724.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:16:22.042 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593381.731988_1736.1 target=1_1690593252.941877_1724.1 2023-07-29T01:16:27.043 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:16:27.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:16:27.045 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:16:27.099 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:16:27.099 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:16:27.131 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:16:27.039+0000 7f4698322880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:16:27.132 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:16:27.040+0000 7f4698322880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:16:27.146 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:16:27.146 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:16:27.147 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:16:27.147 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:16:27.147 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:16:27.147 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:16:27.147 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:27.148 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:27.148 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:16:27.148 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:27.148 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:16:27.148 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:27.148 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:16:27.149 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:16:27.149 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:27.149 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:16:27.149 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:27.149 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:16:27.149 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:27.149 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:27.150 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:27.150 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:27.150 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:16:27.150 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:27.150 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:16:27.150 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593381.731988_1736.1", 2023-07-29T01:16:27.150 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:27.151 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:16:27.151 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:27.151 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:21.731988Z", 2023-07-29T01:16:27.151 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:27.151 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:27.151 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:27.151 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:27.152 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:16:27.152 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:27.152 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:16:27.152 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:16:27.152 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:27.152 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:16:27.152 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:27.153 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:16:27.153 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:27.153 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:27.153 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:27.153 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:27.153 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:16:27.153 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:27.154 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:16:27.154 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:16:27.154 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:27.154 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:16:27.154 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:27.154 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:16:27.154 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:27.155 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:27.155 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:27.155 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:16:27.155 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:27.155 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:16:27.155 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:16:27.156 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:16:27.156 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:27.156 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:16:27Z" 2023-07-29T01:16:27.156 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:16:27.157 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593381.731988_1736.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:16:21.731988Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:16:27Z" } 2023-07-29T01:16:27.157 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593381.731988_1736.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:21.731988Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:16:27.158 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593381.731988_1736.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:16:27.158 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593381.731988_1736.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:16:27.158 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T01:16:27.159 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'hiiqos-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:16:27.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'hiiqos-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:16:27.159 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket hiiqos-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:16:27.204 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:16:27.204 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:16:27.250 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[]} 2023-07-29T01:16:27.250 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'hiiqos-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:16:27.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'hiiqos-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:16:27.251 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket hiiqos-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:16:27.298 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:16:27.298 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:16:27.347 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-07-29T01:16:27.347 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'hiiqos-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:16:27.348 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'hiiqos-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:16:27.348 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket hiiqos-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:16:27.399 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:16:27.400 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:16:27.448 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2023-07-29T01:16:27.449 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=hiiqos-70', '--source-bucket=hiiqos-71', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:16:27.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=hiiqos-70', '--source-bucket=hiiqos-71', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:16:27.450 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=hiiqos-70 --source-bucket=hiiqos-71 '--dest-bucket=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:16:27.505 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:16:27.505 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:16:27.557 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hiiqos-71","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-07-29T01:16:27.558 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'hiiqos-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:16:27.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'hiiqos-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:16:27.559 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket hiiqos-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:16:27.607 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:16:27.608 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:16:27.656 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hiiqos-71","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-07-29T01:16:27.657 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'hiiqos-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:16:27.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'hiiqos-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:16:27.658 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket hiiqos-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:16:27.701 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:16:27.701 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:16:27.747 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hiiqos-71","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-07-29T01:16:27.747 DEBUG:boto:path=/hiiqos-70/ 2023-07-29T01:16:27.748 DEBUG:boto:auth_path=/hiiqos-70/ 2023-07-29T01:16:27.748 DEBUG:boto:Method: HEAD 2023-07-29T01:16:27.749 DEBUG:boto:Path: /hiiqos-70/ 2023-07-29T01:16:27.749 DEBUG:boto:Data: 2023-07-29T01:16:27.749 DEBUG:boto:Headers: {} 2023-07-29T01:16:27.750 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:16:27.750 DEBUG:boto:Port: 8001 2023-07-29T01:16:27.750 DEBUG:boto:Params: {} 2023-07-29T01:16:27.751 DEBUG:boto:Token: None 2023-07-29T01:16:27.751 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:16:27 GMT /hiiqos-70/ 2023-07-29T01:16:27.752 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5WNOnlZGs6Q0iN6Q0CfDwyM3o64= 2023-07-29T01:16:27.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:16:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5WNOnlZGs6Q0iN6Q0CfDwyM3o64=', 'Content-Length': '0'} 2023-07-29T01:16:27.754 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000447e9ee817ebf3ea-0064c4686b-284a-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:16:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:16:27.755 DEBUG:boto:path=/hiiqos-70/obj7 2023-07-29T01:16:27.756 DEBUG:boto:auth_path=/hiiqos-70/obj7 2023-07-29T01:16:27.756 DEBUG:boto:Method: PUT 2023-07-29T01:16:27.756 DEBUG:boto:Path: /hiiqos-70/obj7 2023-07-29T01:16:27.757 DEBUG:boto:Data: 2023-07-29T01:16:27.757 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T01:16:27.757 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:16:27.758 DEBUG:boto:Port: 8001 2023-07-29T01:16:27.758 DEBUG:boto:Params: {} 2023-07-29T01:16:27.758 DEBUG:boto:Token: None 2023-07-29T01:16:27.759 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 01:16:27 GMT /hiiqos-70/obj7 2023-07-29T01:16:27.759 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:F9mbdrMjceDGSyT/e8NhoNrqHKA= 2023-07-29T01:16:27.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 01:16:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:F9mbdrMjceDGSyT/e8NhoNrqHKA='} 2023-07-29T01:16:27.765 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003448d9506443bbb2-0064c4686b-284a-a2'), ('Date', 'Sat, 29 Jul 2023 01:16:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:16:27.766 DEBUG:boto:path=/hiiqos-70/ 2023-07-29T01:16:27.766 DEBUG:boto:auth_path=/hiiqos-70/ 2023-07-29T01:16:27.767 DEBUG:boto:Method: HEAD 2023-07-29T01:16:27.767 DEBUG:boto:Path: /hiiqos-70/ 2023-07-29T01:16:27.767 DEBUG:boto:Data: 2023-07-29T01:16:27.768 DEBUG:boto:Headers: {} 2023-07-29T01:16:27.768 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:16:27.768 DEBUG:boto:Port: 8001 2023-07-29T01:16:27.769 DEBUG:boto:Params: {} 2023-07-29T01:16:27.769 DEBUG:boto:Token: None 2023-07-29T01:16:27.770 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:16:27 GMT /hiiqos-70/ 2023-07-29T01:16:27.770 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5WNOnlZGs6Q0iN6Q0CfDwyM3o64= 2023-07-29T01:16:27.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:16:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5WNOnlZGs6Q0iN6Q0CfDwyM3o64=', 'Content-Length': '0'} 2023-07-29T01:16:27.772 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000044538a03ffc73dd6-0064c4686b-284a-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:16:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:16:27.773 DEBUG:boto:path=/hiiqos-70/obj8 2023-07-29T01:16:27.773 DEBUG:boto:auth_path=/hiiqos-70/obj8 2023-07-29T01:16:27.774 DEBUG:boto:Method: PUT 2023-07-29T01:16:27.774 DEBUG:boto:Path: /hiiqos-70/obj8 2023-07-29T01:16:27.775 DEBUG:boto:Data: 2023-07-29T01:16:27.775 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T01:16:27.775 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:16:27.776 DEBUG:boto:Port: 8001 2023-07-29T01:16:27.776 DEBUG:boto:Params: {} 2023-07-29T01:16:27.776 DEBUG:boto:Token: None 2023-07-29T01:16:27.777 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 01:16:27 GMT /hiiqos-70/obj8 2023-07-29T01:16:27.777 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LfZR3s+BbPxm9A98aZQGq86xhTM= 2023-07-29T01:16:27.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 01:16:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LfZR3s+BbPxm9A98aZQGq86xhTM='} 2023-07-29T01:16:27.820 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000026fcb09e136c92f5-0064c4686b-284a-a2'), ('Date', 'Sat, 29 Jul 2023 01:16:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:16:27.821 DEBUG:boto:path=/hiiqos-71/ 2023-07-29T01:16:27.822 DEBUG:boto:auth_path=/hiiqos-71/ 2023-07-29T01:16:27.822 DEBUG:boto:Method: HEAD 2023-07-29T01:16:27.822 DEBUG:boto:Path: /hiiqos-71/ 2023-07-29T01:16:27.823 DEBUG:boto:Data: 2023-07-29T01:16:27.823 DEBUG:boto:Headers: {} 2023-07-29T01:16:27.823 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:16:27.824 DEBUG:boto:Port: 8001 2023-07-29T01:16:27.824 DEBUG:boto:Params: {} 2023-07-29T01:16:27.825 DEBUG:boto:Token: None 2023-07-29T01:16:27.825 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:16:27 GMT /hiiqos-71/ 2023-07-29T01:16:27.825 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ssrYyUgYvvqgQiBETQWN+BfClIA= 2023-07-29T01:16:27.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:16:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ssrYyUgYvvqgQiBETQWN+BfClIA=', 'Content-Length': '0'} 2023-07-29T01:16:27.828 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000531b0f59deaed9db-0064c4686b-284a-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:16:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:16:27.828 DEBUG:boto:path=/hiiqos-71/obj9 2023-07-29T01:16:27.829 DEBUG:boto:auth_path=/hiiqos-71/obj9 2023-07-29T01:16:27.829 DEBUG:boto:Method: PUT 2023-07-29T01:16:27.830 DEBUG:boto:Path: /hiiqos-71/obj9 2023-07-29T01:16:27.830 DEBUG:boto:Data: 2023-07-29T01:16:27.830 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T01:16:27.831 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:16:27.831 DEBUG:boto:Port: 8001 2023-07-29T01:16:27.831 DEBUG:boto:Params: {} 2023-07-29T01:16:27.832 DEBUG:boto:Token: None 2023-07-29T01:16:27.832 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 01:16:27 GMT /hiiqos-71/obj9 2023-07-29T01:16:27.832 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7qNzUoMAI0EWlKKNI/M0f0LglZI= 2023-07-29T01:16:27.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 01:16:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7qNzUoMAI0EWlKKNI/M0f0LglZI='} 2023-07-29T01:16:27.838 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009aa2f3e215e9d122-0064c4686b-284a-a2'), ('Date', 'Sat, 29 Jul 2023 01:16:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:16:27.839 DEBUG:boto:path=/hiiqos-71/ 2023-07-29T01:16:27.839 DEBUG:boto:auth_path=/hiiqos-71/ 2023-07-29T01:16:27.840 DEBUG:boto:Method: HEAD 2023-07-29T01:16:27.840 DEBUG:boto:Path: /hiiqos-71/ 2023-07-29T01:16:27.840 DEBUG:boto:Data: 2023-07-29T01:16:27.841 DEBUG:boto:Headers: {} 2023-07-29T01:16:27.841 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:16:27.841 DEBUG:boto:Port: 8001 2023-07-29T01:16:27.842 DEBUG:boto:Params: {} 2023-07-29T01:16:27.842 DEBUG:boto:Token: None 2023-07-29T01:16:27.842 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:16:27 GMT /hiiqos-71/ 2023-07-29T01:16:27.843 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ssrYyUgYvvqgQiBETQWN+BfClIA= 2023-07-29T01:16:27.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:16:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ssrYyUgYvvqgQiBETQWN+BfClIA=', 'Content-Length': '0'} 2023-07-29T01:16:27.845 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000581051b94e8aed0c-0064c4686b-284a-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:16:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:16:27.846 DEBUG:boto:path=/hiiqos-71/obj10 2023-07-29T01:16:27.846 DEBUG:boto:auth_path=/hiiqos-71/obj10 2023-07-29T01:16:27.846 DEBUG:boto:Method: PUT 2023-07-29T01:16:27.847 DEBUG:boto:Path: /hiiqos-71/obj10 2023-07-29T01:16:27.847 DEBUG:boto:Data: 2023-07-29T01:16:27.847 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T01:16:27.848 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:16:27.848 DEBUG:boto:Port: 8001 2023-07-29T01:16:27.849 DEBUG:boto:Params: {} 2023-07-29T01:16:27.849 DEBUG:boto:Token: None 2023-07-29T01:16:27.849 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 01:16:27 GMT /hiiqos-71/obj10 2023-07-29T01:16:27.850 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ym3WVFJ6Jh3rQfJR9M2AMkY3wSk= 2023-07-29T01:16:27.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 01:16:27 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ym3WVFJ6Jh3rQfJR9M2AMkY3wSk='} 2023-07-29T01:16:27.892 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ccab682b5fddc180-0064c4686b-284a-a2'), ('Date', 'Sat, 29 Jul 2023 01:16:27 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:16:27.893 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:16:27.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:16:27.894 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:16:27.939 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:16:27.939 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:16:27.969 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T01:16:27.982 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T01:16:27.982 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:27.982 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:16:27.983 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:17.836402Z" 2023-07-29T01:16:27.983 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:27.983 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:27.983 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593387.549974_1756.1", 2023-07-29T01:16:27.983 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:16:27.549974Z" 2023-07-29T01:16:27.983 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:27.983 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:27.984 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:16:27.984 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T01:16:27.984 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:27.984 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:27.984 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:16:27.984 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:12.811406Z" 2023-07-29T01:16:27.984 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:27.985 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T01:16:27.985 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690591217.836402_294.1', 1: '1_1690593387.549974_1756.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:16:27.986 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T01:16:27.986 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:16:27.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:16:27.987 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:16:28.033 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:16:28.033 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:16:28.062 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:16:27.969+0000 7fa29f92a880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:16:28.062 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:16:27.970+0000 7fa29f92a880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:16:28.074 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:16:28.075 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:16:28.075 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:16:28.075 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:16:28.075 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:16:28.075 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:16:28.076 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:28.076 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:28.076 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:16:28.076 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:28.076 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:16:28.076 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:28.077 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:16:28.077 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:16:28.077 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:28.077 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:16:28.077 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:28.077 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:16:28.077 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:28.078 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:28.078 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:28.078 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:28.078 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:16:28.078 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:28.078 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:16:28.078 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593381.731988_1736.1", 2023-07-29T01:16:28.079 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:28.079 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:16:28.079 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:28.079 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:21.731988Z", 2023-07-29T01:16:28.079 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:28.079 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:28.079 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:28.080 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:28.080 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:16:28.080 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:28.080 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:16:28.080 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:16:28.080 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:28.081 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:16:28.081 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:28.081 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:16:28.081 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:28.081 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:28.081 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:28.081 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:28.082 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:16:28.082 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:28.082 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:16:28.082 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:16:28.082 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:28.082 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:16:28.082 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:28.083 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:16:28.083 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:28.083 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:28.083 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:28.083 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:16:28.083 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:28.083 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:16:28.084 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:16:28.084 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:16:28.084 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:28.084 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:16:27Z" 2023-07-29T01:16:28.084 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:16:28.085 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593381.731988_1736.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:16:21.731988Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:16:27Z" } 2023-07-29T01:16:28.086 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593381.731988_1736.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:21.731988Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:16:28.086 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593387.549974_1756.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:16:28.086 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593381.731988_1736.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:16:28.087 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593387.549974_1756.1 target=1_1690593381.731988_1736.1 2023-07-29T01:16:33.088 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:16:33.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:16:33.089 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:16:33.142 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:16:33.143 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:16:33.175 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:16:33.083+0000 7f35bb938880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:16:33.176 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:16:33.084+0000 7f35bb938880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:16:33.190 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:16:33.190 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:16:33.190 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:16:33.191 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:16:33.191 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:16:33.191 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:16:33.191 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:33.191 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:33.191 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:16:33.191 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:33.192 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:16:33.192 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:33.192 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:16:33.192 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:16:33.192 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:33.192 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:16:33.193 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:33.193 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:16:33.193 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:33.193 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:33.193 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:33.193 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:33.193 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:16:33.194 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:33.194 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:16:33.194 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593381.731988_1736.1", 2023-07-29T01:16:33.194 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:33.194 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:16:33.194 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:33.194 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:21.731988Z", 2023-07-29T01:16:33.195 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:33.195 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:33.195 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:33.195 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:33.195 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:16:33.195 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:33.195 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:16:33.196 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:16:33.196 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:33.196 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:16:33.196 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:33.196 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:16:33.196 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:33.197 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:33.197 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:33.197 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:33.197 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:16:33.197 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:33.197 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:16:33.197 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:16:33.198 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:33.198 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:16:33.198 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:33.198 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:16:33.198 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:33.198 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:33.198 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:33.199 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:16:33.199 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:33.199 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:16:33.199 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:16:33.199 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:16:33.199 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:33.199 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:16:33Z" 2023-07-29T01:16:33.200 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:16:33.200 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593381.731988_1736.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:16:21.731988Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:16:33Z" } 2023-07-29T01:16:33.201 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593381.731988_1736.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:21.731988Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:16:33.201 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593387.549974_1756.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:16:33.202 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593381.731988_1736.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:16:33.202 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593387.549974_1756.1 target=1_1690593381.731988_1736.1 2023-07-29T01:16:38.203 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:16:38.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:16:38.204 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:16:38.255 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:16:38.255 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:16:38.288 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:16:38.195+0000 7f63b5126880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:16:38.288 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:16:38.196+0000 7f63b5126880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:16:38.301 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:16:38.302 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:16:38.302 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:16:38.302 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:16:38.303 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:16:38.303 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:16:38.303 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:38.304 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:38.304 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:16:38.304 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:38.305 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:16:38.305 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:38.305 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:16:38.306 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:16:38.306 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:38.306 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:16:38.306 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:38.307 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:16:38.307 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:38.307 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:38.307 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:38.308 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:38.308 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:16:38.309 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:38.309 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:16:38.309 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593381.731988_1736.1", 2023-07-29T01:16:38.309 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:38.309 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:16:38.309 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:38.310 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:21.731988Z", 2023-07-29T01:16:38.310 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:38.310 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:38.310 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:38.310 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:38.310 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:16:38.311 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:38.311 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:16:38.311 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:16:38.311 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:38.311 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:16:38.311 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:38.311 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:16:38.312 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:38.312 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:38.312 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:38.312 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:38.312 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:16:38.312 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:38.313 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:16:38.313 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:16:38.313 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:38.313 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:16:38.313 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:38.313 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:16:38.313 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:38.314 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:38.314 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:38.314 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:16:38.314 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:38.314 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:16:38.314 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:16:38.314 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:16:38.315 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:38.315 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:16:38Z" 2023-07-29T01:16:38.315 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:16:38.316 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593381.731988_1736.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:16:21.731988Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:16:38Z" } 2023-07-29T01:16:38.317 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593381.731988_1736.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:21.731988Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:16:38.318 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593387.549974_1756.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:16:38.318 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593381.731988_1736.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:16:38.319 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593387.549974_1756.1 target=1_1690593381.731988_1736.1 2023-07-29T01:16:43.320 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:16:43.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:16:43.321 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:16:43.374 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:16:43.374 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:16:43.407 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:16:43.314+0000 7fa157210880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:16:43.408 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:16:43.315+0000 7fa157210880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:16:43.422 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:16:43.422 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:16:43.422 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:16:43.422 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:16:43.422 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:16:43.423 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:16:43.423 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:43.423 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:43.423 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:16:43.423 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:43.423 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:16:43.423 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:43.424 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:16:43.424 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:16:43.424 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:43.424 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:16:43.424 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:43.424 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:16:43.424 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:43.425 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:43.425 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:43.425 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:43.425 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:16:43.425 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:43.425 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:16:43.425 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593381.731988_1736.1", 2023-07-29T01:16:43.426 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:43.426 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:16:43.426 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:43.426 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:21.731988Z", 2023-07-29T01:16:43.426 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:43.426 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:43.426 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:43.427 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:43.427 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:16:43.427 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:43.427 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:16:43.427 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:16:43.427 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:43.427 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:16:43.428 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:43.428 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:16:43.428 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:43.428 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:43.428 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:43.428 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:43.429 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:16:43.429 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:43.429 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:16:43.429 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:16:43.429 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:43.429 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:16:43.429 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:43.430 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:16:43.430 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:43.430 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:43.430 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:43.430 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:16:43.430 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:43.430 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:16:43.430 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:16:43.431 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:16:43.431 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:43.431 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:16:43Z" 2023-07-29T01:16:43.431 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:16:43.432 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593381.731988_1736.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:16:21.731988Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:16:43Z" } 2023-07-29T01:16:43.432 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593381.731988_1736.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:21.731988Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:16:43.433 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593387.549974_1756.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:16:43.433 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593381.731988_1736.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:16:43.433 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593387.549974_1756.1 target=1_1690593381.731988_1736.1 2023-07-29T01:16:48.435 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:16:48.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:16:48.436 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:16:48.489 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:16:48.490 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:16:48.522 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:16:48.429+0000 7f2c68726880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:16:48.522 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:16:48.430+0000 7f2c68726880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:16:48.536 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:16:48.537 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:16:48.537 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:16:48.537 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:16:48.537 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:16:48.537 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:16:48.538 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:48.538 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:48.538 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:16:48.538 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:48.538 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:16:48.538 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:48.538 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:16:48.539 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:16:48.539 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:48.539 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:16:48.539 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:48.539 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:16:48.539 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:48.539 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:48.540 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:48.540 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:48.540 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:16:48.540 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:48.540 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:16:48.540 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593387.549974_1756.1", 2023-07-29T01:16:48.541 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:48.541 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:16:48.541 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:48.541 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:27.549974Z", 2023-07-29T01:16:48.541 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:48.541 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:48.541 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:48.542 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:48.542 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:16:48.542 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:48.542 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:16:48.542 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:16:48.542 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:48.542 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:16:48.543 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:48.543 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:16:48.543 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:48.543 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:48.543 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:48.543 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:48.543 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:16:48.544 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:48.544 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:16:48.544 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:16:48.544 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:48.544 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:16:48.544 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:48.544 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:16:48.545 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:48.545 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:48.545 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:48.545 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:16:48.545 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:48.545 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:16:48.545 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:16:48.546 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:16:48.546 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:48.546 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:16:48Z" 2023-07-29T01:16:48.546 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:16:48.547 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593387.549974_1756.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:16:27.549974Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:16:48Z" } 2023-07-29T01:16:48.547 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593387.549974_1756.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:27.549974Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:16:48.548 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593387.549974_1756.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:16:48.548 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593387.549974_1756.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:16:48.548 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T01:16:48.549 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:16:48.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:16:48.550 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:16:48.595 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:16:48.595 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:16:48.644 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T01:16:48.645 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:48.645 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000017417", 2023-07-29T01:16:48.645 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:16:27.673766Z" 2023-07-29T01:16:48.645 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:48.645 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:48.646 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000017424", 2023-07-29T01:16:48.646 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:16:27.746078Z" 2023-07-29T01:16:48.646 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:48.646 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:48.646 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000013658", 2023-07-29T01:16:48.646 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:16:13.631326Z" 2023-07-29T01:16:48.646 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:48.647 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:48.647 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000006290", 2023-07-29T01:16:48.647 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:16:27.763388Z" 2023-07-29T01:16:48.647 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:48.647 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T01:16:48.648 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000017417', 1: '00000000000000000000:00000000000000017424', 2: '00000000000000000000:00000000000000013658', 3: '00000000000000000000:00000000000000006290'} 2023-07-29T01:16:48.649 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-07-29T01:16:48.649 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:16:48.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:16:48.650 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:16:48.697 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:16:48.697 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:16:48.741 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:16:48.741 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:16:48.741 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:16:48.741 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:16:48.742 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:16:48.742 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 17882949198279251152 2023-07-29T01:16:48.742 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:48.742 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:16:48.742 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:48.742 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:16:48.743 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:48.743 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:16:48.743 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000017417", 2023-07-29T01:16:48.743 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:48.743 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:16:48.743 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:48.743 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:27.673766Z" 2023-07-29T01:16:48.744 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:48.744 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:48.744 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:48.744 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:16:48.744 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:48.744 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:16:48.745 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000017424", 2023-07-29T01:16:48.745 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:48.745 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:16:48.745 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:48.745 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:27.746078Z" 2023-07-29T01:16:48.745 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:48.745 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:48.746 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:48.746 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:16:48.746 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:48.746 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:16:48.746 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000013658", 2023-07-29T01:16:48.746 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:48.746 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:16:48.747 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:48.747 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:13.631326Z" 2023-07-29T01:16:48.747 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:48.747 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:48.747 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:48.747 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:16:48.747 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:48.748 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:16:48.748 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000006290", 2023-07-29T01:16:48.748 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:48.748 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:16:48.748 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:48.748 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:27.763388Z" 2023-07-29T01:16:48.749 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:48.749 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:48.749 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:16:48.749 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:48.749 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:16:48.749 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 0, 2023-07-29T01:16:48.750 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 0 2023-07-29T01:16:48.750 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:48.750 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:16:48Z" 2023-07-29T01:16:48.750 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:16:48.751 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17882949198279251152 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000017417", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:16:27.673766Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000017424", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:16:27.746078Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013658", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:16:13.631326Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006290", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:16:27.763388Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-07-29T01:16:48Z" } 2023-07-29T01:16:48.751 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000017417', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:27.673766Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000017424', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:27.746078Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013658', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:13.631326Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006290', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:27.763388Z'}}] 2023-07-29T01:16:48.752 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000017417', 1: '00000000000000000000:00000000000000017424', 2: '00000000000000000000:00000000000000013658', 3: '00000000000000000000:00000000000000006290'} 2023-07-29T01:16:48.752 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000017417', 1: '00000000000000000000:00000000000000017424', 2: '00000000000000000000:00000000000000013658', 3: '00000000000000000000:00000000000000006290'} 2023-07-29T01:16:48.752 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-07-29T01:16:48.753 DEBUG:boto:path=/hiiqos-70/ 2023-07-29T01:16:48.753 DEBUG:boto:auth_path=/hiiqos-70/ 2023-07-29T01:16:48.753 DEBUG:boto:Method: HEAD 2023-07-29T01:16:48.754 DEBUG:boto:Path: /hiiqos-70/ 2023-07-29T01:16:48.754 DEBUG:boto:Data: 2023-07-29T01:16:48.754 DEBUG:boto:Headers: {} 2023-07-29T01:16:48.755 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:16:48.755 DEBUG:boto:Port: 8000 2023-07-29T01:16:48.755 DEBUG:boto:Params: {} 2023-07-29T01:16:48.756 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-29T01:16:48.756 DEBUG:boto:Token: None 2023-07-29T01:16:48.757 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:16:48 GMT /hiiqos-70/ 2023-07-29T01:16:48.757 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BQGltkb8BS9+5t6ldaafQYLknSk= 2023-07-29T01:16:48.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:16:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BQGltkb8BS9+5t6ldaafQYLknSk=', 'Content-Length': '0'} 2023-07-29T01:16:53.763 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000363ea6b8bd6d976a-0064c46885-283f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:16:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:16:53.764 DEBUG:boto:path=/hiiqos-70/obj7 2023-07-29T01:16:53.764 DEBUG:boto:auth_path=/hiiqos-70/obj7 2023-07-29T01:16:53.764 DEBUG:boto:Method: HEAD 2023-07-29T01:16:53.765 DEBUG:boto:Path: /hiiqos-70/obj7 2023-07-29T01:16:53.765 DEBUG:boto:Data: 2023-07-29T01:16:53.765 DEBUG:boto:Headers: {} 2023-07-29T01:16:53.766 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:16:53.766 DEBUG:boto:Port: 8000 2023-07-29T01:16:53.766 DEBUG:boto:Params: {} 2023-07-29T01:16:53.767 DEBUG:boto:Token: None 2023-07-29T01:16:53.767 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:16:53 GMT /hiiqos-70/obj7 2023-07-29T01:16:53.768 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:GsDyH/1ljaKnlD/4yhfSYhyi3Eg= 2023-07-29T01:16:53.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:16:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:GsDyH/1ljaKnlD/4yhfSYhyi3Eg=', 'Content-Length': '0'} 2023-07-29T01:16:53.770 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000edc3b082dde7d4e3-0064c46885-283f-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 01:16:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:16:53.770 DEBUG:boto:path=/hiiqos-70/obj8 2023-07-29T01:16:53.770 DEBUG:boto:auth_path=/hiiqos-70/obj8 2023-07-29T01:16:53.771 DEBUG:boto:Method: HEAD 2023-07-29T01:16:53.771 DEBUG:boto:Path: /hiiqos-70/obj8 2023-07-29T01:16:53.771 DEBUG:boto:Data: 2023-07-29T01:16:53.772 DEBUG:boto:Headers: {} 2023-07-29T01:16:53.772 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:16:53.772 DEBUG:boto:Port: 8000 2023-07-29T01:16:53.773 DEBUG:boto:Params: {} 2023-07-29T01:16:53.773 DEBUG:boto:Token: None 2023-07-29T01:16:53.774 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:16:53 GMT /hiiqos-70/obj8 2023-07-29T01:16:53.774 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:C24lqaf8sfUNxU8NLJAktm5JyOw= 2023-07-29T01:16:53.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:16:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:C24lqaf8sfUNxU8NLJAktm5JyOw=', 'Content-Length': '0'} 2023-07-29T01:16:53.776 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000e3ec9d8d7708af54-0064c46885-283f-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 01:16:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:16:53.777 DEBUG:boto:path=/hiiqos-70/obj9 2023-07-29T01:16:53.777 DEBUG:boto:auth_path=/hiiqos-70/obj9 2023-07-29T01:16:53.777 DEBUG:boto:Method: HEAD 2023-07-29T01:16:53.778 DEBUG:boto:Path: /hiiqos-70/obj9 2023-07-29T01:16:53.778 DEBUG:boto:Data: 2023-07-29T01:16:53.778 DEBUG:boto:Headers: {} 2023-07-29T01:16:53.779 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:16:53.779 DEBUG:boto:Port: 8000 2023-07-29T01:16:53.779 DEBUG:boto:Params: {} 2023-07-29T01:16:53.780 DEBUG:boto:Token: None 2023-07-29T01:16:53.780 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:16:53 GMT /hiiqos-70/obj9 2023-07-29T01:16:53.780 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7X7+nA/XnHzn0Oz40/mAimUNLLA= 2023-07-29T01:16:53.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:16:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7X7+nA/XnHzn0Oz40/mAimUNLLA=', 'Content-Length': '0'} 2023-07-29T01:16:53.782 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:16:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-71:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.9662.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000058403525ddc3301a-0064c46885-283f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:16:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:16:53.783 DEBUG:boto:path=/hiiqos-70/obj9 2023-07-29T01:16:53.783 DEBUG:boto:auth_path=/hiiqos-70/obj9 2023-07-29T01:16:53.784 DEBUG:boto:Method: GET 2023-07-29T01:16:53.784 DEBUG:boto:Path: /hiiqos-70/obj9 2023-07-29T01:16:53.785 DEBUG:boto:Data: 2023-07-29T01:16:53.785 DEBUG:boto:Headers: {} 2023-07-29T01:16:53.785 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:16:53.786 DEBUG:boto:Port: 8000 2023-07-29T01:16:53.786 DEBUG:boto:Params: {} 2023-07-29T01:16:53.786 DEBUG:boto:Token: None 2023-07-29T01:16:53.787 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 01:16:53 GMT /hiiqos-70/obj9 2023-07-29T01:16:53.787 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RSclFvRvCd2whhNeSu/PW4jF7uw= 2023-07-29T01:16:53.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:16:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RSclFvRvCd2whhNeSu/PW4jF7uw=', 'Content-Length': '0'} 2023-07-29T01:16:53.790 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:16:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-71:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.9662.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006dc6b6c5eaf9e014-0064c46885-283f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:16:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:16:53.790 DEBUG:boto:path=/hiiqos-70/obj10 2023-07-29T01:16:53.791 DEBUG:boto:auth_path=/hiiqos-70/obj10 2023-07-29T01:16:53.791 DEBUG:boto:Method: HEAD 2023-07-29T01:16:53.791 DEBUG:boto:Path: /hiiqos-70/obj10 2023-07-29T01:16:53.792 DEBUG:boto:Data: 2023-07-29T01:16:53.792 DEBUG:boto:Headers: {} 2023-07-29T01:16:53.792 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:16:53.793 DEBUG:boto:Port: 8000 2023-07-29T01:16:53.793 DEBUG:boto:Params: {} 2023-07-29T01:16:53.793 DEBUG:boto:Token: None 2023-07-29T01:16:53.794 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:16:53 GMT /hiiqos-70/obj10 2023-07-29T01:16:53.794 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:c6degzr1RXfEeEXFE/kGSlke0tU= 2023-07-29T01:16:53.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:16:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c6degzr1RXfEeEXFE/kGSlke0tU=', 'Content-Length': '0'} 2023-07-29T01:16:53.797 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:16:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-71:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.9662.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e3928cdbc674c583-0064c46885-283f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:16:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:16:53.797 DEBUG:boto:path=/hiiqos-70/obj10 2023-07-29T01:16:53.798 DEBUG:boto:auth_path=/hiiqos-70/obj10 2023-07-29T01:16:53.798 DEBUG:boto:Method: GET 2023-07-29T01:16:53.798 DEBUG:boto:Path: /hiiqos-70/obj10 2023-07-29T01:16:53.799 DEBUG:boto:Data: 2023-07-29T01:16:53.799 DEBUG:boto:Headers: {} 2023-07-29T01:16:53.799 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:16:53.800 DEBUG:boto:Port: 8000 2023-07-29T01:16:53.800 DEBUG:boto:Params: {} 2023-07-29T01:16:53.800 DEBUG:boto:Token: None 2023-07-29T01:16:53.801 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 01:16:53 GMT /hiiqos-70/obj10 2023-07-29T01:16:53.801 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:YF9RIvGq2Fh2LnlWoejaOcTXnK8= 2023-07-29T01:16:53.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:16:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YF9RIvGq2Fh2LnlWoejaOcTXnK8=', 'Content-Length': '0'} 2023-07-29T01:16:53.804 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:16:27 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '59d1bb30-3b1c-476f-9111-052c1ea0b711:hiiqos-71:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.9662.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000052c41808c4fce918-0064c46885-283f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:16:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:16:53.805 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'hiiqos-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:16:53.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'hiiqos-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:16:53.805 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket hiiqos-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:16:53.859 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:16:53.860 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:16:53.919 INFO:teuthology.orchestra.run.smithi101.stdout:{"sync_policy":{"groups":[]}} 2023-07-29T01:16:53.919 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:16:53.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:16:53.920 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:16:53.964 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:16:53.965 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:16:54.010 INFO:teuthology.orchestra.run.smithi101.stdout:{"sync_policy":{"groups":[]}} 2023-07-29T01:16:54.011 INFO:tasks.rgw_multisite_tests:test_sync_different_buckets: ... ok 2023-07-29T01:16:54.012 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:16:54.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:16:54.013 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:16:54.057 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:16:54.058 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:16:54.088 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T01:16:54.101 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T01:16:54.102 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:54.102 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:16:54.102 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:17.836402Z" 2023-07-29T01:16:54.102 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:54.102 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:54.102 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593413.808092_1766.1", 2023-07-29T01:16:54.103 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:16:53.808092Z" 2023-07-29T01:16:54.103 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:54.103 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:54.103 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:16:54.103 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T01:16:54.103 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:54.103 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:54.104 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:16:54.104 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:12.811406Z" 2023-07-29T01:16:54.104 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:54.104 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T01:16:54.105 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690591217.836402_294.1', 1: '1_1690593413.808092_1766.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:16:54.105 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T01:16:54.106 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:16:54.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:16:54.107 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:16:54.152 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:16:54.152 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:16:54.182 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:16:54.089+0000 7fdd10122880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:16:54.183 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:16:54.090+0000 7fdd10122880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:16:54.196 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:16:54.196 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:16:54.196 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:16:54.196 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:16:54.196 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:16:54.197 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:16:54.197 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:54.197 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:54.197 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:16:54.197 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:54.197 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:16:54.197 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:54.198 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:16:54.198 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:16:54.198 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:54.198 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:16:54.198 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:54.198 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:16:54.198 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:54.199 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:54.199 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:54.199 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:54.199 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:16:54.199 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:54.199 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:16:54.199 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593387.549974_1756.1", 2023-07-29T01:16:54.200 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:54.200 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:16:54.200 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:54.200 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:27.549974Z", 2023-07-29T01:16:54.200 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:54.200 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:54.201 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:54.201 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:54.201 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:16:54.201 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:54.201 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:16:54.201 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:16:54.201 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:54.202 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:16:54.202 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:54.202 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:16:54.202 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:54.202 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:54.202 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:54.202 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:54.203 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:16:54.203 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:54.203 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:16:54.203 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:16:54.203 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:54.203 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:16:54.203 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:54.204 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:16:54.204 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:54.204 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:54.204 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:54.204 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:16:54.204 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:54.204 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:16:54.205 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:16:54.205 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:16:54.205 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:54.205 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:16:54Z" 2023-07-29T01:16:54.205 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:16:54.206 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593387.549974_1756.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:16:27.549974Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:16:54Z" } 2023-07-29T01:16:54.206 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593387.549974_1756.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:27.549974Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:16:54.207 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593413.808092_1766.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:16:54.207 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593387.549974_1756.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:16:54.208 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593413.808092_1766.1 target=1_1690593387.549974_1756.1 2023-07-29T01:16:59.209 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:16:59.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:16:59.211 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:16:59.264 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:16:59.264 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:16:59.297 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:16:59.204+0000 7f9377a13880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:16:59.298 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:16:59.205+0000 7f9377a13880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:16:59.312 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:16:59.312 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:16:59.312 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:16:59.313 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:16:59.313 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:16:59.313 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:16:59.313 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:59.313 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:59.313 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:16:59.314 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:59.314 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:16:59.314 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:59.314 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:16:59.314 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:16:59.314 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:59.315 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:16:59.315 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:59.315 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:16:59.315 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:59.315 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:59.315 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:59.316 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:59.316 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:16:59.316 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:59.316 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:16:59.316 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593387.549974_1756.1", 2023-07-29T01:16:59.316 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:59.317 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:16:59.317 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:59.317 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:27.549974Z", 2023-07-29T01:16:59.317 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:59.317 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:59.317 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:59.318 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:59.318 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:16:59.318 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:59.318 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:16:59.318 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:16:59.318 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:59.319 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:16:59.319 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:59.319 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:16:59.319 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:59.319 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:59.319 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:59.320 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:16:59.320 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:16:59.320 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:16:59.320 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:16:59.320 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:16:59.320 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:16:59.321 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:16:59.321 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:16:59.321 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:16:59.321 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:16:59.321 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:59.321 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:16:59.321 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:16:59.322 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:59.322 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:16:59.322 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:16:59.322 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:16:59.322 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:16:59.322 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:16:59Z" 2023-07-29T01:16:59.323 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:16:59.323 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593387.549974_1756.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:16:27.549974Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:16:59Z" } 2023-07-29T01:16:59.324 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593387.549974_1756.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:27.549974Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:16:59.325 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593413.808092_1766.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:16:59.325 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593387.549974_1756.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:16:59.325 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593413.808092_1766.1 target=1_1690593387.549974_1756.1 2023-07-29T01:17:04.326 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:17:04.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:17:04.327 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:17:04.378 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:17:04.378 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:17:04.409 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:17:04.316+0000 7f5776fa2880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:17:04.410 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:17:04.317+0000 7f5776fa2880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:17:04.424 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:17:04.424 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:17:04.424 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:17:04.424 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:17:04.425 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:17:04.425 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:17:04.425 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:17:04.425 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:04.425 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:17:04.425 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:04.426 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:17:04.426 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:04.426 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:17:04.426 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:17:04.426 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:04.426 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:17:04.427 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:04.427 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:17:04.427 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:17:04.427 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:04.427 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:04.427 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:04.428 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:17:04.428 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:04.428 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:17:04.428 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593387.549974_1756.1", 2023-07-29T01:17:04.428 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:04.428 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:17:04.428 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:04.429 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:27.549974Z", 2023-07-29T01:17:04.429 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:17:04.429 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:04.429 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:04.429 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:04.429 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:17:04.429 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:04.430 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:17:04.430 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:17:04.430 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:04.430 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:17:04.430 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:04.430 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:17:04.430 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:17:04.431 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:04.431 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:04.431 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:04.431 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:17:04.431 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:04.431 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:17:04.431 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:17:04.432 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:04.432 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:17:04.432 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:04.432 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:17:04.432 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:17:04.432 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:04.433 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:04.433 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:17:04.433 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:04.433 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:17:04.433 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:17:04.433 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:17:04.433 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:04.434 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:17:04Z" 2023-07-29T01:17:04.434 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:17:04.436 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593387.549974_1756.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:16:27.549974Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:17:04Z" } 2023-07-29T01:17:04.437 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593387.549974_1756.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:27.549974Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:17:04.437 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593413.808092_1766.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:17:04.438 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593387.549974_1756.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:17:04.438 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593413.808092_1766.1 target=1_1690593387.549974_1756.1 2023-07-29T01:17:09.439 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:17:09.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:17:09.441 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:17:09.498 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:17:09.499 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:17:09.531 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:17:09.438+0000 7f64397a2880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:17:09.532 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:17:09.439+0000 7f64397a2880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:17:09.546 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:17:09.546 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:17:09.547 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:17:09.547 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:17:09.547 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:17:09.547 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:17:09.547 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:17:09.547 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:09.548 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:17:09.548 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:09.548 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:17:09.548 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:09.548 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:17:09.548 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:17:09.548 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:09.549 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:17:09.549 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:09.549 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:17:09.549 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:17:09.549 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:09.549 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:09.549 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:09.550 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:17:09.550 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:09.550 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:17:09.550 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593413.808092_1766.1", 2023-07-29T01:17:09.550 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:09.551 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:17:09.551 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:09.551 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:53.808092Z", 2023-07-29T01:17:09.551 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:17:09.551 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:09.551 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:09.552 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:09.552 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:17:09.552 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:09.552 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:17:09.552 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:17:09.552 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:09.552 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:17:09.553 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:09.553 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:17:09.553 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:17:09.553 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:09.553 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:09.553 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:09.554 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:17:09.554 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:09.554 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:17:09.554 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:17:09.554 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:09.554 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:17:09.554 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:09.555 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:17:09.555 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:17:09.555 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:09.555 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:09.555 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:17:09.555 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:09.555 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:17:09.556 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:17:09.556 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:17:09.556 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:09.556 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:17:09Z" 2023-07-29T01:17:09.556 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:17:09.557 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593413.808092_1766.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:16:53.808092Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:17:09Z" } 2023-07-29T01:17:09.558 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593413.808092_1766.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:53.808092Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:17:09.558 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593413.808092_1766.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:17:09.558 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593413.808092_1766.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:17:09.559 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T01:17:09.559 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:17:09.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:17:09.560 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:17:09.605 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:17:09.605 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:17:09.653 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[]} 2023-07-29T01:17:09.653 INFO:rgw_multi.tests:create bucket zone=a1 name=hiiqos-72 2023-07-29T01:17:09.654 DEBUG:boto:path=/hiiqos-72/ 2023-07-29T01:17:09.655 DEBUG:boto:auth_path=/hiiqos-72/ 2023-07-29T01:17:09.655 DEBUG:boto:Method: PUT 2023-07-29T01:17:09.656 DEBUG:boto:Path: /hiiqos-72/ 2023-07-29T01:17:09.657 DEBUG:boto:Data: 2023-07-29T01:17:09.657 DEBUG:boto:Headers: {} 2023-07-29T01:17:09.658 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:17:09.659 DEBUG:boto:Port: 8000 2023-07-29T01:17:09.659 DEBUG:boto:Params: {} 2023-07-29T01:17:09.660 DEBUG:boto:Token: None 2023-07-29T01:17:09.661 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 01:17:09 GMT /hiiqos-72/ 2023-07-29T01:17:09.661 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kTbiUbMJLVHbqwDaRKD2l6tgZ2E= 2023-07-29T01:17:09.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:17:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kTbiUbMJLVHbqwDaRKD2l6tgZ2E=', 'Content-Length': '0'} 2023-07-29T01:17:09.676 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000017341a735e991956-0064c46895-283f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:17:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:17:09.677 INFO:rgw_multi.tests:create bucket zone=a1 name=hiiqos-73 2023-07-29T01:17:09.678 DEBUG:boto:path=/hiiqos-73/ 2023-07-29T01:17:09.678 DEBUG:boto:auth_path=/hiiqos-73/ 2023-07-29T01:17:09.679 DEBUG:boto:Method: PUT 2023-07-29T01:17:09.680 DEBUG:boto:Path: /hiiqos-73/ 2023-07-29T01:17:09.680 DEBUG:boto:Data: 2023-07-29T01:17:09.681 DEBUG:boto:Headers: {} 2023-07-29T01:17:09.681 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:17:09.682 DEBUG:boto:Port: 8000 2023-07-29T01:17:09.683 DEBUG:boto:Params: {} 2023-07-29T01:17:09.683 DEBUG:boto:Token: None 2023-07-29T01:17:09.684 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 01:17:09 GMT /hiiqos-73/ 2023-07-29T01:17:09.685 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:yW1vTzYWyxWX1+O1ylWlKWLA2e4= 2023-07-29T01:17:09.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:17:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yW1vTzYWyxWX1+O1ylWlKWLA2e4=', 'Content-Length': '0'} 2023-07-29T01:17:09.699 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000223736c1a4fb7ed3-0064c46895-283f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:17:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:17:09.700 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:17:09.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:17:09.701 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:17:09.752 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:17:09.753 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:17:09.785 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T01:17:09.798 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T01:17:09.798 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:09.799 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:17:09.799 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:17.836402Z" 2023-07-29T01:17:09.799 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:09.799 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:09.799 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593429.606201_1781.1", 2023-07-29T01:17:09.799 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:17:09.606201Z" 2023-07-29T01:17:09.799 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:09.800 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:09.800 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:17:09.800 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T01:17:09.800 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:09.800 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:09.800 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:17:09.801 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:12.811406Z" 2023-07-29T01:17:09.801 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:09.801 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T01:17:09.802 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690591217.836402_294.1', 1: '1_1690593429.606201_1781.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:17:09.803 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T01:17:09.803 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:17:09.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:17:09.804 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:17:09.849 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:17:09.850 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:17:09.878 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:17:09.785+0000 7f02fe926880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:17:09.879 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:17:09.786+0000 7f02fe926880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:17:09.892 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:17:09.892 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:17:09.892 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:17:09.893 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:17:09.893 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:17:09.893 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:17:09.893 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:17:09.893 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:09.893 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:17:09.893 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:09.894 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:17:09.894 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:09.894 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:17:09.894 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:17:09.895 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:09.895 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:17:09.895 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:09.895 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:17:09.895 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:17:09.895 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:09.896 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:09.896 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:09.896 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:17:09.896 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:09.896 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:17:09.896 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593429.606201_1781.1", 2023-07-29T01:17:09.896 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:09.897 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:17:09.897 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:09.897 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:09.606201Z", 2023-07-29T01:17:09.897 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:17:09.897 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:09.897 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:09.897 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:09.898 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:17:09.898 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:09.898 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:17:09.898 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:17:09.898 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:09.898 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:17:09.898 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:09.899 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:17:09.899 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:17:09.899 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:09.899 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:09.899 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:09.899 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:17:09.899 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:09.900 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:17:09.900 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:17:09.900 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:09.900 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:17:09.900 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:09.900 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:17:09.900 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:17:09.901 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:09.901 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:09.901 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:17:09.901 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:09.901 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:17:09.901 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:17:09.901 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:17:09.902 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:09.902 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:17:09Z" 2023-07-29T01:17:09.902 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:17:09.903 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593429.606201_1781.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:09.606201Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:17:09Z" } 2023-07-29T01:17:09.904 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593429.606201_1781.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:09.606201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:17:09.904 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593429.606201_1781.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:17:09.904 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593429.606201_1781.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:17:09.905 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T01:17:09.905 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:17:09.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:17:09.906 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:17:09.952 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:17:09.952 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:17:09.997 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-07-29T01:17:09.997 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:17:09.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:17:09.998 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:17:10.048 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:17:10.049 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:17:10.094 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"allowed"}]} 2023-07-29T01:17:10.094 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hiiqos-72', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hiiqos-72', '--dest-bucket=hiiqos-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:17:10.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hiiqos-72', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hiiqos-72', '--dest-bucket=hiiqos-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:17:10.095 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-hiiqos-72 --source-zones=a1 --dest-zones=a2 --source-bucket=hiiqos-72 --dest-bucket=hiiqos-73 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:17:10.145 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:17:10.146 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:17:10.191 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-hiiqos-72","source":{"bucket":"hiiqos-72","zones":["a1"]},"dest":{"bucket":"hiiqos-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-07-29T01:17:10.191 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hiiqos-73', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hiiqos-73', '--dest-bucket=hiiqos-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:17:10.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hiiqos-73', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hiiqos-73', '--dest-bucket=hiiqos-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:17:10.193 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-hiiqos-73 --source-zones=a1 --dest-zones=a2 --source-bucket=hiiqos-73 --dest-bucket=hiiqos-73 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:17:10.243 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:17:10.244 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:17:10.288 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-hiiqos-72","source":{"bucket":"hiiqos-72","zones":["a1"]},"dest":{"bucket":"hiiqos-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hiiqos-73","source":{"bucket":"hiiqos-73","zones":["a1"]},"dest":{"bucket":"hiiqos-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-07-29T01:17:10.288 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:17:10.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:17:10.289 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:17:10.335 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:17:10.335 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:17:10.380 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-hiiqos-72","source":{"bucket":"hiiqos-72","zones":["a1"]},"dest":{"bucket":"hiiqos-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hiiqos-73","source":{"bucket":"hiiqos-73","zones":["a1"]},"dest":{"bucket":"hiiqos-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-07-29T01:17:10.380 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:17:10.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:17:10.381 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:17:10.426 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:17:10.426 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:17:10.476 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:17:10.477 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:17:10.477 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 20, 2023-07-29T01:17:10.477 INFO:teuthology.orchestra.run.smithi101.stdout: "predecessor_uuid": "a0987484-ce37-4520-a564-54b7ea8c5251", 2023-07-29T01:17:10.477 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": [], 2023-07-29T01:17:10.477 INFO:teuthology.orchestra.run.smithi101.stdout: "period_map": { 2023-07-29T01:17:10.478 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:17:10.478 INFO:teuthology.orchestra.run.smithi101.stdout: "zonegroups": [ 2023-07-29T01:17:10.478 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:10.478 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "a", 2023-07-29T01:17:10.478 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a", 2023-07-29T01:17:10.478 INFO:teuthology.orchestra.run.smithi101.stdout: "api_name": "", 2023-07-29T01:17:10.479 INFO:teuthology.orchestra.run.smithi101.stdout: "is_master": true, 2023-07-29T01:17:10.479 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:17:10.479 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-29T01:17:10.479 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:17:10.479 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames": [], 2023-07-29T01:17:10.479 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames_s3website": [], 2023-07-29T01:17:10.479 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:17:10.480 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:17:10.480 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:10.480 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:17:10.480 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a1", 2023-07-29T01:17:10.480 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:17:10.480 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-29T01:17:10.484 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:17:10.484 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T01:17:10.484 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T01:17:10.485 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T01:17:10.485 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T01:17:10.485 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T01:17:10.485 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T01:17:10.485 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T01:17:10.485 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T01:17:10.485 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T01:17:10.486 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T01:17:10.486 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:17:10.486 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:17:10.486 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:10.486 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:10.486 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-29T01:17:10.487 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a2", 2023-07-29T01:17:10.487 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:17:10.487 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8001" 2023-07-29T01:17:10.487 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:17:10.487 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T01:17:10.487 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T01:17:10.487 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T01:17:10.488 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T01:17:10.488 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T01:17:10.488 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T01:17:10.488 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T01:17:10.488 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T01:17:10.488 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T01:17:10.488 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T01:17:10.489 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:17:10.489 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:17:10.489 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:10.489 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:17:10.489 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_targets": [ 2023-07-29T01:17:10.489 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:10.489 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default-placement", 2023-07-29T01:17:10.490 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [], 2023-07-29T01:17:10.490 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": [ 2023-07-29T01:17:10.490 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD" 2023-07-29T01:17:10.490 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:17:10.490 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:10.490 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:17:10.490 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "default-placement", 2023-07-29T01:17:10.491 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T01:17:10.491 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_policy": { 2023-07-29T01:17:10.491 INFO:teuthology.orchestra.run.smithi101.stdout: "groups": [ 2023-07-29T01:17:10.491 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:10.491 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "sync-group", 2023-07-29T01:17:10.491 INFO:teuthology.orchestra.run.smithi101.stdout: "data_flow": { 2023-07-29T01:17:10.491 INFO:teuthology.orchestra.run.smithi101.stdout: "directional": [ 2023-07-29T01:17:10.492 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:10.492 INFO:teuthology.orchestra.run.smithi101.stdout: "source_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:17:10.492 INFO:teuthology.orchestra.run.smithi101.stdout: "dest_zone": "59d1bb30-3b1c-476f-9111-052c1ea0b711" 2023-07-29T01:17:10.492 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:10.492 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:17:10.492 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:10.493 INFO:teuthology.orchestra.run.smithi101.stdout: "pipes": [ 2023-07-29T01:17:10.493 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:10.493 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "sync-pipe-hiiqos-72", 2023-07-29T01:17:10.493 INFO:teuthology.orchestra.run.smithi101.stdout: "source": { 2023-07-29T01:17:10.493 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket": "hiiqos-72", 2023-07-29T01:17:10.493 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:17:10.493 INFO:teuthology.orchestra.run.smithi101.stdout: "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433" 2023-07-29T01:17:10.494 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:17:10.494 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:10.494 INFO:teuthology.orchestra.run.smithi101.stdout: "dest": { 2023-07-29T01:17:10.494 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket": "hiiqos-73", 2023-07-29T01:17:10.494 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:17:10.494 INFO:teuthology.orchestra.run.smithi101.stdout: "59d1bb30-3b1c-476f-9111-052c1ea0b711" 2023-07-29T01:17:10.494 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:17:10.495 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:10.495 INFO:teuthology.orchestra.run.smithi101.stdout: "params": { 2023-07-29T01:17:10.495 INFO:teuthology.orchestra.run.smithi101.stdout: "source": { 2023-07-29T01:17:10.495 INFO:teuthology.orchestra.run.smithi101.stdout: "filter": { 2023-07-29T01:17:10.495 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [] 2023-07-29T01:17:10.495 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:10.495 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:10.496 INFO:teuthology.orchestra.run.smithi101.stdout: "dest": {}, 2023-07-29T01:17:10.496 INFO:teuthology.orchestra.run.smithi101.stdout: "priority": 0, 2023-07-29T01:17:10.496 INFO:teuthology.orchestra.run.smithi101.stdout: "mode": "system", 2023-07-29T01:17:10.496 INFO:teuthology.orchestra.run.smithi101.stdout: "user": "" 2023-07-29T01:17:10.496 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:10.496 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:10.497 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:10.497 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "sync-pipe-hiiqos-73", 2023-07-29T01:17:10.497 INFO:teuthology.orchestra.run.smithi101.stdout: "source": { 2023-07-29T01:17:10.497 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket": "hiiqos-73", 2023-07-29T01:17:10.497 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:17:10.497 INFO:teuthology.orchestra.run.smithi101.stdout: "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433" 2023-07-29T01:17:10.497 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:17:10.498 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:10.498 INFO:teuthology.orchestra.run.smithi101.stdout: "dest": { 2023-07-29T01:17:10.498 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket": "hiiqos-73", 2023-07-29T01:17:10.498 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:17:10.498 INFO:teuthology.orchestra.run.smithi101.stdout: "59d1bb30-3b1c-476f-9111-052c1ea0b711" 2023-07-29T01:17:10.498 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:17:10.498 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:10.499 INFO:teuthology.orchestra.run.smithi101.stdout: "params": { 2023-07-29T01:17:10.499 INFO:teuthology.orchestra.run.smithi101.stdout: "source": { 2023-07-29T01:17:10.499 INFO:teuthology.orchestra.run.smithi101.stdout: "filter": { 2023-07-29T01:17:10.499 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [] 2023-07-29T01:17:10.499 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:10.499 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:10.499 INFO:teuthology.orchestra.run.smithi101.stdout: "dest": {}, 2023-07-29T01:17:10.500 INFO:teuthology.orchestra.run.smithi101.stdout: "priority": 0, 2023-07-29T01:17:10.500 INFO:teuthology.orchestra.run.smithi101.stdout: "mode": "system", 2023-07-29T01:17:10.500 INFO:teuthology.orchestra.run.smithi101.stdout: "user": "" 2023-07-29T01:17:10.500 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:10.500 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:10.500 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:17:10.500 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "enabled" 2023-07-29T01:17:10.501 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:10.501 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:17:10.501 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:10.501 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled_features": [ 2023-07-29T01:17:10.501 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:17:10.501 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:17:10.502 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:10.502 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:10.502 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "b", 2023-07-29T01:17:10.502 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b", 2023-07-29T01:17:10.502 INFO:teuthology.orchestra.run.smithi101.stdout: "api_name": "", 2023-07-29T01:17:10.502 INFO:teuthology.orchestra.run.smithi101.stdout: "is_master": false, 2023-07-29T01:17:10.502 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:17:10.503 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-29T01:17:10.503 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:17:10.503 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames": [], 2023-07-29T01:17:10.503 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames_s3website": [], 2023-07-29T01:17:10.503 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T01:17:10.503 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:17:10.503 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:10.504 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T01:17:10.504 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b1", 2023-07-29T01:17:10.504 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:17:10.504 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-29T01:17:10.504 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:17:10.504 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T01:17:10.504 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T01:17:10.505 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T01:17:10.505 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T01:17:10.505 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T01:17:10.505 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T01:17:10.505 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T01:17:10.505 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T01:17:10.505 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T01:17:10.506 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T01:17:10.506 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:17:10.506 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:17:10.506 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:10.506 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:10.506 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-29T01:17:10.506 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b2", 2023-07-29T01:17:10.507 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:17:10.507 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8001" 2023-07-29T01:17:10.507 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:17:10.507 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T01:17:10.507 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T01:17:10.507 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T01:17:10.507 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T01:17:10.508 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T01:17:10.508 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T01:17:10.508 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T01:17:10.508 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T01:17:10.509 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T01:17:10.509 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T01:17:10.510 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:17:10.510 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:17:10.510 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:10.510 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:17:10.510 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_targets": [ 2023-07-29T01:17:10.510 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:10.511 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default-placement", 2023-07-29T01:17:10.511 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [], 2023-07-29T01:17:10.511 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": [ 2023-07-29T01:17:10.511 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD" 2023-07-29T01:17:10.511 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:17:10.511 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:10.511 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:17:10.512 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "default-placement", 2023-07-29T01:17:10.512 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T01:17:10.512 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_policy": { 2023-07-29T01:17:10.512 INFO:teuthology.orchestra.run.smithi101.stdout: "groups": [] 2023-07-29T01:17:10.512 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:10.512 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled_features": [ 2023-07-29T01:17:10.512 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:17:10.513 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:17:10.513 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:10.513 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:17:10.513 INFO:teuthology.orchestra.run.smithi101.stdout: "short_zone_ids": [ 2023-07-29T01:17:10.513 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:10.513 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:17:10.513 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 2896287231 2023-07-29T01:17:10.514 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:10.514 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:10.514 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-29T01:17:10.514 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 815606810 2023-07-29T01:17:10.514 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:10.514 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:10.514 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T01:17:10.515 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 2009171363 2023-07-29T01:17:10.515 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:10.515 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:10.515 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-29T01:17:10.515 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 3105190295 2023-07-29T01:17:10.515 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:10.515 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:17:10.516 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:10.516 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zonegroup": "a", 2023-07-29T01:17:10.516 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:17:10.516 INFO:teuthology.orchestra.run.smithi101.stdout: "period_config": { 2023-07-29T01:17:10.516 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_quota": { 2023-07-29T01:17:10.516 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-29T01:17:10.516 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-29T01:17:10.517 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-29T01:17:10.517 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-29T01:17:10.517 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-29T01:17:10.517 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:10.517 INFO:teuthology.orchestra.run.smithi101.stdout: "user_quota": { 2023-07-29T01:17:10.517 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-29T01:17:10.517 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-29T01:17:10.518 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-29T01:17:10.518 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-29T01:17:10.518 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-29T01:17:10.518 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:10.518 INFO:teuthology.orchestra.run.smithi101.stdout: "user_ratelimit": { 2023-07-29T01:17:10.518 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-29T01:17:10.518 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-29T01:17:10.519 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-29T01:17:10.519 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-29T01:17:10.519 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-29T01:17:10.519 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:10.519 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_ratelimit": { 2023-07-29T01:17:10.519 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-29T01:17:10.519 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-29T01:17:10.520 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-29T01:17:10.520 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-29T01:17:10.520 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-29T01:17:10.520 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:10.520 INFO:teuthology.orchestra.run.smithi101.stdout: "anonymous_ratelimit": { 2023-07-29T01:17:10.520 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-29T01:17:10.521 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-29T01:17:10.521 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-29T01:17:10.521 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-29T01:17:10.521 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-29T01:17:10.521 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:10.521 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:10.521 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T01:17:10.522 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_name": "test-realm", 2023-07-29T01:17:10.522 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:17:10.522 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:17:10.522 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:17:10.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:17:10.523 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:17:10.574 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:17:10.574 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:17:10.721 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-hiiqos-72","source":{"bucket":"hiiqos-72","zones":["a1"]},"dest":{"bucket":"hiiqos-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hiiqos-73","source":{"bucket":"hiiqos-73","zones":["a1"]},"dest":{"bucket":"hiiqos-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-07-29T01:17:10.721 DEBUG:boto:path=/hiiqos-72/ 2023-07-29T01:17:10.722 DEBUG:boto:auth_path=/hiiqos-72/ 2023-07-29T01:17:10.722 DEBUG:boto:Method: HEAD 2023-07-29T01:17:10.723 DEBUG:boto:Path: /hiiqos-72/ 2023-07-29T01:17:10.724 DEBUG:boto:Data: 2023-07-29T01:17:10.725 DEBUG:boto:Headers: {} 2023-07-29T01:17:10.725 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:17:10.725 DEBUG:boto:Port: 8000 2023-07-29T01:17:10.726 DEBUG:boto:Params: {} 2023-07-29T01:17:10.726 DEBUG:boto:Token: None 2023-07-29T01:17:10.727 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:17:10 GMT /hiiqos-72/ 2023-07-29T01:17:10.727 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+naYur/uScys9ZKHwDh3kM3J6sc= 2023-07-29T01:17:10.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:17:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+naYur/uScys9ZKHwDh3kM3J6sc=', 'Content-Length': '0'} 2023-07-29T01:17:10.742 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000066c24a9b20589373-0064c46896-2a77-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:17:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:17:10.743 DEBUG:boto:path=/hiiqos-72/obj1 2023-07-29T01:17:10.743 DEBUG:boto:auth_path=/hiiqos-72/obj1 2023-07-29T01:17:10.744 DEBUG:boto:Method: PUT 2023-07-29T01:17:10.744 DEBUG:boto:Path: /hiiqos-72/obj1 2023-07-29T01:17:10.745 DEBUG:boto:Data: 2023-07-29T01:17:10.745 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T01:17:10.745 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:17:10.746 DEBUG:boto:Port: 8000 2023-07-29T01:17:10.746 DEBUG:boto:Params: {} 2023-07-29T01:17:10.746 DEBUG:boto:Token: None 2023-07-29T01:17:10.747 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 01:17:10 GMT /hiiqos-72/obj1 2023-07-29T01:17:10.747 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:xjKkdr9R7OYDWtVRnszZoNMeQjg= 2023-07-29T01:17:10.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 01:17:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xjKkdr9R7OYDWtVRnszZoNMeQjg='} 2023-07-29T01:17:10.771 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a24a85ff0c1eec5e-0064c46896-2a77-a1'), ('Date', 'Sat, 29 Jul 2023 01:17:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:17:10.772 DEBUG:boto:path=/hiiqos-72/ 2023-07-29T01:17:10.772 DEBUG:boto:auth_path=/hiiqos-72/ 2023-07-29T01:17:10.773 DEBUG:boto:Method: HEAD 2023-07-29T01:17:10.773 DEBUG:boto:Path: /hiiqos-72/ 2023-07-29T01:17:10.773 DEBUG:boto:Data: 2023-07-29T01:17:10.774 DEBUG:boto:Headers: {} 2023-07-29T01:17:10.774 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:17:10.775 DEBUG:boto:Port: 8000 2023-07-29T01:17:10.775 DEBUG:boto:Params: {} 2023-07-29T01:17:10.776 DEBUG:boto:Token: None 2023-07-29T01:17:10.777 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:17:10 GMT /hiiqos-72/ 2023-07-29T01:17:10.778 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+naYur/uScys9ZKHwDh3kM3J6sc= 2023-07-29T01:17:10.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:17:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+naYur/uScys9ZKHwDh3kM3J6sc=', 'Content-Length': '0'} 2023-07-29T01:17:10.783 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002aac6f8792eb42bc-0064c46896-2a77-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:17:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:17:10.784 DEBUG:boto:path=/hiiqos-72/obj2 2023-07-29T01:17:10.784 DEBUG:boto:auth_path=/hiiqos-72/obj2 2023-07-29T01:17:10.785 DEBUG:boto:Method: PUT 2023-07-29T01:17:10.785 DEBUG:boto:Path: /hiiqos-72/obj2 2023-07-29T01:17:10.785 DEBUG:boto:Data: 2023-07-29T01:17:10.786 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T01:17:10.786 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:17:10.787 DEBUG:boto:Port: 8000 2023-07-29T01:17:10.787 DEBUG:boto:Params: {} 2023-07-29T01:17:10.788 DEBUG:boto:Token: None 2023-07-29T01:17:10.788 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 01:17:10 GMT /hiiqos-72/obj2 2023-07-29T01:17:10.789 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4kDvtFWQlxVLkvnaDakrDgDEq+E= 2023-07-29T01:17:10.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 01:17:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4kDvtFWQlxVLkvnaDakrDgDEq+E='} 2023-07-29T01:17:10.832 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004a67b85995526949-0064c46896-2a77-a1'), ('Date', 'Sat, 29 Jul 2023 01:17:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:17:10.833 DEBUG:boto:path=/hiiqos-73/ 2023-07-29T01:17:10.834 DEBUG:boto:auth_path=/hiiqos-73/ 2023-07-29T01:17:10.834 DEBUG:boto:Method: HEAD 2023-07-29T01:17:10.834 DEBUG:boto:Path: /hiiqos-73/ 2023-07-29T01:17:10.835 DEBUG:boto:Data: 2023-07-29T01:17:10.835 DEBUG:boto:Headers: {} 2023-07-29T01:17:10.835 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:17:10.836 DEBUG:boto:Port: 8000 2023-07-29T01:17:10.836 DEBUG:boto:Params: {} 2023-07-29T01:17:10.837 DEBUG:boto:Token: None 2023-07-29T01:17:10.837 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:17:10 GMT /hiiqos-73/ 2023-07-29T01:17:10.837 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9juOUUKQ1t7JuIifojmap4PQkPI= 2023-07-29T01:17:10.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:17:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9juOUUKQ1t7JuIifojmap4PQkPI=', 'Content-Length': '0'} 2023-07-29T01:17:10.846 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000065b0b6d2d2a440e5-0064c46896-2a77-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:17:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:17:10.847 DEBUG:boto:path=/hiiqos-73/obj3 2023-07-29T01:17:10.847 DEBUG:boto:auth_path=/hiiqos-73/obj3 2023-07-29T01:17:10.848 DEBUG:boto:Method: PUT 2023-07-29T01:17:10.848 DEBUG:boto:Path: /hiiqos-73/obj3 2023-07-29T01:17:10.849 DEBUG:boto:Data: 2023-07-29T01:17:10.849 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T01:17:10.849 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:17:10.850 DEBUG:boto:Port: 8000 2023-07-29T01:17:10.850 DEBUG:boto:Params: {} 2023-07-29T01:17:10.850 DEBUG:boto:Token: None 2023-07-29T01:17:10.851 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 01:17:10 GMT /hiiqos-73/obj3 2023-07-29T01:17:10.851 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ebaUGY7mWF+CIwMGkUkvcvcfpSo= 2023-07-29T01:17:10.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 01:17:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ebaUGY7mWF+CIwMGkUkvcvcfpSo='} 2023-07-29T01:17:10.868 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000e49c94f63582fc9-0064c46896-2a77-a1'), ('Date', 'Sat, 29 Jul 2023 01:17:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:17:10.869 DEBUG:boto:path=/hiiqos-73/ 2023-07-29T01:17:10.869 DEBUG:boto:auth_path=/hiiqos-73/ 2023-07-29T01:17:10.870 DEBUG:boto:Method: HEAD 2023-07-29T01:17:10.870 DEBUG:boto:Path: /hiiqos-73/ 2023-07-29T01:17:10.871 DEBUG:boto:Data: 2023-07-29T01:17:10.871 DEBUG:boto:Headers: {} 2023-07-29T01:17:10.871 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:17:10.872 DEBUG:boto:Port: 8000 2023-07-29T01:17:10.872 DEBUG:boto:Params: {} 2023-07-29T01:17:10.872 DEBUG:boto:Token: None 2023-07-29T01:17:10.873 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:17:10 GMT /hiiqos-73/ 2023-07-29T01:17:10.873 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:9juOUUKQ1t7JuIifojmap4PQkPI= 2023-07-29T01:17:10.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:17:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9juOUUKQ1t7JuIifojmap4PQkPI=', 'Content-Length': '0'} 2023-07-29T01:17:10.880 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000055819dfe0b5203b9-0064c46896-2a77-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:17:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:17:10.881 DEBUG:boto:path=/hiiqos-73/obj4 2023-07-29T01:17:10.881 DEBUG:boto:auth_path=/hiiqos-73/obj4 2023-07-29T01:17:10.882 DEBUG:boto:Method: PUT 2023-07-29T01:17:10.882 DEBUG:boto:Path: /hiiqos-73/obj4 2023-07-29T01:17:10.882 DEBUG:boto:Data: 2023-07-29T01:17:10.883 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T01:17:10.884 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:17:10.884 DEBUG:boto:Port: 8000 2023-07-29T01:17:10.885 DEBUG:boto:Params: {} 2023-07-29T01:17:10.886 DEBUG:boto:Token: None 2023-07-29T01:17:10.887 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 01:17:10 GMT /hiiqos-73/obj4 2023-07-29T01:17:10.887 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:FzT8DMZ8Vjxcn9vt2stunqRlq6I= 2023-07-29T01:17:10.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 01:17:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FzT8DMZ8Vjxcn9vt2stunqRlq6I='} 2023-07-29T01:17:10.932 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eec8dec4d012fb03-0064c46896-2a77-a1'), ('Date', 'Sat, 29 Jul 2023 01:17:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:17:10.934 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:17:10.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:17:10.935 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:17:11.024 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:17:11.024 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:17:11.084 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T01:17:11.103 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T01:17:11.103 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:11.103 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:17:11.103 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:17.836402Z" 2023-07-29T01:17:11.104 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:11.104 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:11.104 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593429.606201_1781.1", 2023-07-29T01:17:11.104 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:17:09.606201Z" 2023-07-29T01:17:11.104 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:11.104 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:11.105 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:17:11.105 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T01:17:11.105 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:11.105 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:11.105 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:17:11.105 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:12.811406Z" 2023-07-29T01:17:11.107 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:11.107 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T01:17:11.108 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690591217.836402_294.1', 1: '1_1690593429.606201_1781.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:17:11.108 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T01:17:11.109 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:17:11.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:17:11.109 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:17:11.168 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:17:11.169 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:17:11.210 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:17:11.117+0000 7fbe2b867880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:17:11.211 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:17:11.118+0000 7fbe2b867880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:17:11.227 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:17:11.228 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:17:11.228 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:17:11.228 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:17:11.228 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:17:11.228 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:17:11.228 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:17:11.229 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:11.229 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:17:11.229 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:11.229 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:17:11.229 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:11.229 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:17:11.229 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:17:11.230 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:11.230 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:17:11.230 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:11.230 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:17:11.230 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:17:11.230 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:11.231 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:11.231 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:11.231 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:17:11.231 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:11.232 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:17:11.232 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593429.606201_1781.1", 2023-07-29T01:17:11.232 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:11.232 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:17:11.233 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:11.233 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:09.606201Z", 2023-07-29T01:17:11.233 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:17:11.234 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:11.234 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:11.234 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:11.234 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:17:11.234 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:11.235 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:17:11.235 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:17:11.235 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:11.235 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:17:11.235 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:11.235 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:17:11.235 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:17:11.236 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:11.236 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:11.236 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:11.236 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:17:11.236 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:11.236 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:17:11.237 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:17:11.237 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:11.237 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:17:11.237 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:11.237 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:17:11.237 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:17:11.237 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:11.238 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:11.238 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:17:11.238 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:11.238 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:17:11.238 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:17:11.238 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:17:11.238 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:11.239 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:17:11Z" 2023-07-29T01:17:11.239 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:17:11.240 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593429.606201_1781.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:09.606201Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:17:11Z" } 2023-07-29T01:17:11.240 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593429.606201_1781.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:09.606201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:17:11.241 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593429.606201_1781.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:17:11.241 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593429.606201_1781.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:17:11.241 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T01:17:11.242 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:17:11.242 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:17:11.243 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:17:11.288 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:17:11.288 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:17:11.336 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T01:17:11.336 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:11.336 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052514", 2023-07-29T01:17:11.337 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:17:10.772704Z" 2023-07-29T01:17:11.337 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:11.337 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:11.337 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:17:11.337 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:17:11.337 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:11.338 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:11.338 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000046080", 2023-07-29T01:17:11.338 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:17:10.709533Z" 2023-07-29T01:17:11.338 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:11.338 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:11.338 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031774", 2023-07-29T01:17:11.339 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:17:10.812152Z" 2023-07-29T01:17:11.339 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:11.339 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T01:17:11.340 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:17:11.340 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-07-29T01:17:11.341 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:17:11.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:17:11.341 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:17:11.389 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:17:11.389 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:17:11.431 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:17:11.432 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:17:11.432 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:17:11.432 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:17:11.432 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:17:11.433 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:17:11.433 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:11.433 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:17:11.433 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:11.433 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:17:11.433 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:11.434 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:17:11.434 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:17:11.434 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:11.434 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:17:11.434 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:11.434 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:17:11.435 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:11.435 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:11.435 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:11.435 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:17:11.435 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:11.435 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:17:11.436 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:17:11.436 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:11.436 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:17:11.436 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:11.436 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:17:11.436 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:11.436 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:11.437 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:11.437 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:17:11.437 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:11.437 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:17:11.437 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:17:11.437 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:11.438 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:17:11.438 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:11.438 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:17:11.438 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:11.438 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:11.438 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:11.438 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:17:11.439 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:11.439 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:17:11.439 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:17:11.439 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:11.439 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:17:11.439 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:11.440 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:17:11.440 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:11.440 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:11.440 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:17:11.440 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:11.440 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:17:11.440 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:17:11.441 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:17:11.441 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:11.441 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:17:11Z" 2023-07-29T01:17:11.441 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:17:11.442 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052233", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:15:45.371644Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045940", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:15:45.390653Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031634", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:16:32.640507Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:17:11Z" } 2023-07-29T01:17:11.443 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052233', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.371644Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045940', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.390653Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031634', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640507Z'}}] 2023-07-29T01:17:11.443 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:17:11.443 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031634'} 2023-07-29T01:17:11.444 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052233, shard=2 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045940, shard=3 master=00000000000000000000:00000000000000031774 target=00000000000000000000:00000000000000031634 2023-07-29T01:17:16.445 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:17:16.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:17:16.447 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:17:16.502 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:17:16.503 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:17:16.550 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:17:16.551 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:17:16.551 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:17:16.551 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:17:16.551 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:17:16.551 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:17:16.551 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:16.552 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:17:16.552 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:16.552 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:17:16.552 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:16.552 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:17:16.552 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:17:16.553 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:16.553 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:17:16.553 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:16.553 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:17:16.553 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:16.553 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:16.553 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:16.554 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:17:16.554 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:16.554 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:17:16.554 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:17:16.554 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:16.554 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:17:16.555 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:16.555 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:17:16.555 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:16.555 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:16.555 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:16.555 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:17:16.555 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:16.556 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:17:16.556 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:17:16.556 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:16.556 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:17:16.556 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:16.556 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:17:16.556 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:16.557 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:16.557 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:16.557 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:17:16.557 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:16.557 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:17:16.557 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:17:16.557 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:16.558 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:17:16.558 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:16.558 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:17:16.558 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:16.558 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:16.559 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:17:16.559 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:16.559 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:17:16.560 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:17:16.560 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:17:16.560 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:16.560 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:17:16Z" 2023-07-29T01:17:16.560 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:17:16.561 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052233", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:15:45.371644Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045940", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:15:45.390653Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031634", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:16:32.640507Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:17:16Z" } 2023-07-29T01:17:16.562 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052233', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.371644Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045940', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.390653Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031634', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640507Z'}}] 2023-07-29T01:17:16.562 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:17:16.562 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031634'} 2023-07-29T01:17:16.563 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052233, shard=2 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045940, shard=3 master=00000000000000000000:00000000000000031774 target=00000000000000000000:00000000000000031634 2023-07-29T01:17:21.564 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:17:21.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:17:21.565 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:17:21.620 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:17:21.621 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:17:21.668 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:17:21.669 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:17:21.669 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:17:21.669 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:17:21.669 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:17:21.669 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:17:21.670 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:21.670 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:17:21.670 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:21.670 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:17:21.670 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:21.670 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:17:21.671 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:17:21.671 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:21.671 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:17:21.671 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:21.671 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:17:21.671 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:21.672 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:21.672 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:21.672 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:17:21.672 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:21.672 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:17:21.672 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:17:21.673 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:21.673 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:17:21.673 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:21.673 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:17:21.673 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:21.673 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:21.674 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:21.674 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:17:21.674 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:21.674 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:17:21.674 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:17:21.674 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:21.674 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:17:21.675 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:21.675 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:17:21.675 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:21.675 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:21.675 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:21.675 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:17:21.676 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:21.676 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:17:21.676 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:17:21.676 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:21.676 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:17:21.676 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:21.677 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:17:21.677 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:21.677 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:21.677 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:17:21.677 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:21.677 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:17:21.677 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:17:21.678 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:17:21.678 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:21.678 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:17:21Z" 2023-07-29T01:17:21.678 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:17:21.679 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052233", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:15:45.371644Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045940", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:15:45.390653Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031634", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:16:32.640507Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:17:21Z" } 2023-07-29T01:17:21.680 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052233', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.371644Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045940', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.390653Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031634', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640507Z'}}] 2023-07-29T01:17:21.680 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:17:21.680 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031634'} 2023-07-29T01:17:21.681 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052233, shard=2 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045940, shard=3 master=00000000000000000000:00000000000000031774 target=00000000000000000000:00000000000000031634 2023-07-29T01:17:26.681 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:17:26.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:17:26.683 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:17:26.738 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:17:26.738 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:17:26.786 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:17:26.786 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:17:26.786 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:17:26.786 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:17:26.786 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:17:26.787 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:17:26.787 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:26.787 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:17:26.787 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:26.787 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:17:26.787 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:26.788 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:17:26.788 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:17:26.788 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:26.788 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:17:26.788 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:26.788 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:17:26.789 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:26.789 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:26.789 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:26.789 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:17:26.789 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:26.789 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:17:26.789 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:17:26.790 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:26.790 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:17:26.790 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:26.790 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:17:26.790 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:26.790 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:26.790 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:26.791 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:17:26.791 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:26.791 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:17:26.791 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:17:26.791 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:26.791 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:17:26.791 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:26.792 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:17:26.792 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:26.792 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:26.792 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:26.792 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:17:26.793 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:26.793 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:17:26.793 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:17:26.793 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:26.793 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:17:26.793 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:26.793 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:17:26.794 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:26.794 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:26.794 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:17:26.794 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:26.794 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:17:26.794 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:17:26.794 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:17:26.795 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:26.795 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:17:26Z" 2023-07-29T01:17:26.795 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:17:26.796 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052233", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:15:45.371644Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045940", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:15:45.390653Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031634", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:16:32.640507Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:17:26Z" } 2023-07-29T01:17:26.796 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052233', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.371644Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045940', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.390653Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031634', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640507Z'}}] 2023-07-29T01:17:26.797 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:17:26.797 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031634'} 2023-07-29T01:17:26.797 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052233, shard=2 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045940, shard=3 master=00000000000000000000:00000000000000031774 target=00000000000000000000:00000000000000031634 2023-07-29T01:17:31.798 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:17:31.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:17:31.800 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:17:31.857 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:17:31.857 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:17:31.904 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:17:31.905 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:17:31.905 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:17:31.905 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:17:31.905 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:17:31.905 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:17:31.906 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:31.906 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:17:31.906 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:31.906 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:17:31.906 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:31.906 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:17:31.907 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:17:31.907 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:31.907 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:17:31.907 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:31.907 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:17:31.907 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:31.907 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:31.908 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:31.908 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:17:31.908 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:31.908 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:17:31.908 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:17:31.908 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:31.908 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:17:31.909 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:31.909 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:17:31.909 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:31.909 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:31.909 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:31.909 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:17:31.909 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:31.910 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:17:31.910 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:17:31.910 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:31.910 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:17:31.910 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:31.910 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:17:31.910 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:31.911 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:31.911 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:31.911 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:17:31.911 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:31.911 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:17:31.911 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:17:31.911 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:31.912 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:17:31.912 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:31.912 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:17:31.912 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:31.912 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:31.912 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:17:31.913 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:31.913 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:17:31.913 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:17:31.913 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:17:31.913 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:31.913 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:17:31Z" 2023-07-29T01:17:31.913 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:17:31.914 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052233", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:15:45.371644Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045940", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:15:45.390653Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031634", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:16:32.640507Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:17:31Z" } 2023-07-29T01:17:31.915 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052233', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.371644Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045940', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.390653Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031634', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640507Z'}}] 2023-07-29T01:17:31.915 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:17:31.916 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031634'} 2023-07-29T01:17:31.916 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052233, shard=2 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045940, shard=3 master=00000000000000000000:00000000000000031774 target=00000000000000000000:00000000000000031634 2023-07-29T01:17:36.917 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:17:36.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:17:36.918 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:17:36.973 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:17:36.973 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:17:37.020 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:17:37.021 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:17:37.021 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:17:37.021 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:17:37.021 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:17:37.021 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:17:37.022 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:37.022 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:17:37.022 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:37.022 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:17:37.022 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:37.022 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:17:37.022 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:17:37.023 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:37.023 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:17:37.023 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:37.023 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:17:37.023 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:37.023 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:37.023 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:37.024 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:17:37.024 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:37.024 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:17:37.024 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:17:37.024 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:37.024 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:17:37.025 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:37.025 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:17:37.025 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:37.025 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:37.025 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:37.026 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:17:37.026 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:37.026 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:17:37.026 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:17:37.027 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:37.027 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:17:37.027 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:37.027 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:17:37.027 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:37.027 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:37.028 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:37.028 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:17:37.028 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:37.028 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:17:37.028 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:17:37.028 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:37.029 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:17:37.029 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:37.029 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:17:37.029 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:37.029 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:37.029 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:17:37.029 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:37.030 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:17:37.030 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:17:37.030 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:17:37.030 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:37.030 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:17:36Z" 2023-07-29T01:17:37.030 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:17:37.031 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052233", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:15:45.371644Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045940", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:15:45.390653Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031634", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:16:32.640507Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:17:36Z" } 2023-07-29T01:17:37.032 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052233', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.371644Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045940', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.390653Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031634', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640507Z'}}] 2023-07-29T01:17:37.032 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:17:37.033 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031634'} 2023-07-29T01:17:37.033 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052233, shard=2 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045940, shard=3 master=00000000000000000000:00000000000000031774 target=00000000000000000000:00000000000000031634 2023-07-29T01:17:42.035 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:17:42.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:17:42.036 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:17:42.091 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:17:42.091 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:17:42.139 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:17:42.139 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:17:42.139 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:17:42.140 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:17:42.140 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:17:42.140 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:17:42.140 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:42.140 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:17:42.140 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:42.141 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:17:42.141 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:42.141 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:17:42.141 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:17:42.141 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:42.141 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:17:42.141 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:42.142 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:17:42.142 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:42.142 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:42.142 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:42.142 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:17:42.142 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:42.142 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:17:42.143 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:17:42.143 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:42.143 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:17:42.143 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:42.143 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:17:42.143 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:42.143 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:42.144 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:42.144 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:17:42.144 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:42.144 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:17:42.144 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:17:42.144 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:42.144 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:17:42.145 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:42.145 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:17:42.145 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:42.145 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:42.145 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:42.145 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:17:42.145 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:42.146 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:17:42.146 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:17:42.146 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:42.146 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:17:42.146 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:42.146 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:17:42.146 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:42.147 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:42.147 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:17:42.147 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:42.147 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:17:42.147 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:17:42.147 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:17:42.147 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:42.148 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:17:42Z" 2023-07-29T01:17:42.148 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:17:42.149 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052233", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:15:45.371644Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045940", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:15:45.390653Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031634", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:16:32.640507Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:17:42Z" } 2023-07-29T01:17:42.149 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052233', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.371644Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045940', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.390653Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031634', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640507Z'}}] 2023-07-29T01:17:42.150 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:17:42.150 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031634'} 2023-07-29T01:17:42.150 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052233, shard=2 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045940, shard=3 master=00000000000000000000:00000000000000031774 target=00000000000000000000:00000000000000031634 2023-07-29T01:17:47.151 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:17:47.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:17:47.152 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:17:47.207 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:17:47.207 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:17:47.254 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:17:47.255 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:17:47.255 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:17:47.255 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:17:47.255 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:17:47.256 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:17:47.256 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:47.256 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:17:47.256 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:47.256 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:17:47.256 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:47.256 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:17:47.257 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:17:47.257 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:47.257 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:17:47.257 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:47.257 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:17:47.257 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:47.258 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:47.258 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:47.258 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:17:47.258 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:47.258 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:17:47.258 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:17:47.259 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:47.259 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:17:47.259 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:47.259 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:17:47.259 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:47.259 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:47.259 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:47.260 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:17:47.260 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:47.260 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:17:47.260 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:17:47.260 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:47.260 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:17:47.261 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:47.261 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:17:47.261 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:47.261 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:47.261 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:47.261 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:17:47.262 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:47.262 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:17:47.262 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:17:47.262 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:47.262 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:17:47.262 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:47.262 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:17:47.263 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:47.263 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:47.263 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:17:47.263 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:47.263 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:17:47.263 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:17:47.264 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:17:47.264 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:47.264 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:17:47Z" 2023-07-29T01:17:47.264 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:17:47.265 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052233", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:15:45.371644Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045940", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:15:45.390653Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031634", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:16:32.640507Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:17:47Z" } 2023-07-29T01:17:47.265 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052233', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.371644Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045940', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.390653Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031634', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640507Z'}}] 2023-07-29T01:17:47.266 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:17:47.266 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031634'} 2023-07-29T01:17:47.267 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052233, shard=2 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045940, shard=3 master=00000000000000000000:00000000000000031774 target=00000000000000000000:00000000000000031634 2023-07-29T01:17:52.268 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:17:52.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:17:52.269 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:17:52.324 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:17:52.325 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:17:52.372 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:17:52.372 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:17:52.373 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:17:52.373 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:17:52.373 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:17:52.373 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:17:52.373 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:52.374 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:17:52.374 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:52.374 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:17:52.374 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:52.374 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:17:52.374 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:17:52.374 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:52.375 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:17:52.375 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:52.375 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:17:52.375 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:52.375 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:52.375 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:52.375 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:17:52.376 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:52.376 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:17:52.376 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:17:52.376 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:52.376 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:17:52.376 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:52.376 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:17:52.377 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:52.377 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:52.377 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:52.377 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:17:52.377 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:52.377 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:17:52.377 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:17:52.378 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:52.378 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:17:52.378 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:52.378 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:17:52.378 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:52.378 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:52.379 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:52.379 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:17:52.379 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:52.379 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:17:52.379 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:17:52.379 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:52.379 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:17:52.380 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:52.380 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:17:52.380 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:52.380 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:52.380 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:17:52.380 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:52.380 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:17:52.381 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:17:52.381 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:17:52.381 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:52.381 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:17:52Z" 2023-07-29T01:17:52.381 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:17:52.382 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052233", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:15:45.371644Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045940", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:15:45.390653Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031634", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:16:32.640507Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:17:52Z" } 2023-07-29T01:17:52.383 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052233', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.371644Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045940', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.390653Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031634', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640507Z'}}] 2023-07-29T01:17:52.383 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:17:52.383 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031634'} 2023-07-29T01:17:52.384 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052233, shard=2 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045940, shard=3 master=00000000000000000000:00000000000000031774 target=00000000000000000000:00000000000000031634 2023-07-29T01:17:57.384 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:17:57.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:17:57.385 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:17:57.440 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:17:57.440 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:17:57.495 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:17:57.496 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:17:57.496 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:17:57.496 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:17:57.496 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:17:57.496 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:17:57.497 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:57.497 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:17:57.497 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:57.497 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:17:57.497 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:57.498 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:17:57.498 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:17:57.498 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:57.498 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:17:57.498 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:57.498 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:17:57.498 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:57.499 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:57.499 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:57.499 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:17:57.499 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:57.499 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:17:57.499 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:17:57.499 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:57.500 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:17:57.500 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:57.500 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:17:57.500 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:57.500 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:57.500 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:57.500 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:17:57.501 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:57.501 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:17:57.501 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:17:57.501 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:57.501 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:17:57.501 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:57.501 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:17:57.502 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:57.502 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:57.502 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:17:57.502 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:17:57.502 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:17:57.502 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:17:57.502 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:17:57.503 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:17:57.503 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:17:57.503 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:17:57.503 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:17:57.503 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:57.503 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:17:57.503 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:17:57.504 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:57.504 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:17:57.504 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:17:57.504 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:17:57.504 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:17:57.504 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:17:57Z" 2023-07-29T01:17:57.505 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:17:57.505 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052233", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:15:45.371644Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045940", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:15:45.390653Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031634", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:16:32.640507Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:17:57Z" } 2023-07-29T01:17:57.506 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052233', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.371644Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045940', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.390653Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031634', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640507Z'}}] 2023-07-29T01:17:57.506 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:17:57.507 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031634'} 2023-07-29T01:17:57.507 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052233, shard=2 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045940, shard=3 master=00000000000000000000:00000000000000031774 target=00000000000000000000:00000000000000031634 2023-07-29T01:18:02.508 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:18:02.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:18:02.510 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:18:02.566 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:18:02.566 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:18:02.614 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:18:02.614 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:18:02.614 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:18:02.614 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:18:02.615 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:18:02.615 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:18:02.615 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:02.615 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:18:02.615 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:02.615 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:18:02.615 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:02.616 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:18:02.616 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:18:02.616 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:02.616 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:18:02.616 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:02.616 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:18:02.617 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:02.617 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:02.617 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:02.617 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:18:02.617 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:02.617 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:18:02.617 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:18:02.618 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:02.618 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:18:02.618 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:02.618 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:18:02.618 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:02.618 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:02.618 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:02.619 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:18:02.619 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:02.619 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:18:02.619 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:18:02.619 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:02.619 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:18:02.619 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:02.620 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:18:02.620 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:02.620 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:02.620 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:02.620 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:18:02.620 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:02.621 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:18:02.621 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:18:02.621 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:02.621 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:18:02.621 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:02.621 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:18:02.621 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:02.622 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:02.622 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:18:02.622 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:02.622 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:18:02.622 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:18:02.622 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:18:02.622 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:02.623 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:18:02Z" 2023-07-29T01:18:02.623 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:18:02.624 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052233", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:15:45.371644Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045940", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:15:45.390653Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031634", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:16:32.640507Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:18:02Z" } 2023-07-29T01:18:02.624 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052233', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.371644Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045940', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.390653Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031634', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640507Z'}}] 2023-07-29T01:18:02.625 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:18:02.625 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031634'} 2023-07-29T01:18:02.625 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052233, shard=2 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045940, shard=3 master=00000000000000000000:00000000000000031774 target=00000000000000000000:00000000000000031634 2023-07-29T01:18:07.627 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:18:07.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:18:07.628 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:18:07.682 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:18:07.683 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:18:07.730 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:18:07.731 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:18:07.731 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:18:07.732 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:18:07.732 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:18:07.732 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:18:07.733 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:07.733 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:18:07.733 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:07.734 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:18:07.734 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:07.734 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:18:07.735 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:18:07.735 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:07.735 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:18:07.736 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:07.736 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:18:07.736 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:07.737 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:07.737 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:07.737 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:18:07.738 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:07.738 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:18:07.738 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:18:07.739 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:07.739 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:18:07.739 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:07.740 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:18:07.740 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:07.740 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:07.741 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:07.741 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:18:07.741 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:07.742 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:18:07.742 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:18:07.742 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:07.743 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:18:07.743 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:07.744 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:18:07.744 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:07.744 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:07.745 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:07.745 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:18:07.745 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:07.746 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:18:07.746 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:18:07.746 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:07.747 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:18:07.747 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:07.747 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:18:07.748 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:07.748 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:07.748 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:18:07.749 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:07.749 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:18:07.749 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:18:07.750 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:18:07.750 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:07.750 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:18:07Z" 2023-07-29T01:18:07.751 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:18:07.752 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052233", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:15:45.371644Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045940", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:15:45.390653Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031634", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:16:32.640507Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:18:07Z" } 2023-07-29T01:18:07.753 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052233', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.371644Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045940', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.390653Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031634', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640507Z'}}] 2023-07-29T01:18:07.754 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:18:07.754 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031634'} 2023-07-29T01:18:07.755 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052233, shard=2 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045940, shard=3 master=00000000000000000000:00000000000000031774 target=00000000000000000000:00000000000000031634 2023-07-29T01:18:12.756 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:18:12.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:18:12.758 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:18:12.813 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:18:12.813 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:18:12.862 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:18:12.862 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:18:12.863 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:18:12.863 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:18:12.863 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:18:12.864 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:18:12.864 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:12.864 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:18:12.865 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:12.865 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:18:12.865 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:12.865 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:18:12.866 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:18:12.866 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:12.866 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:18:12.866 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:12.867 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:18:12.867 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:12.867 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:12.868 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:12.868 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:18:12.868 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:12.868 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:18:12.868 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:18:12.869 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:12.869 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:18:12.869 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:12.869 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:18:12.869 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:12.869 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:12.869 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:12.870 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:18:12.870 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:12.870 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:18:12.870 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:18:12.870 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:12.870 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:18:12.870 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:12.871 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:18:12.871 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:12.871 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:12.871 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:12.871 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:18:12.871 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:12.871 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:18:12.872 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:18:12.872 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:12.872 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:18:12.872 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:12.872 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:18:12.872 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:12.873 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:12.873 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:18:12.873 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:12.873 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:18:12.874 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:18:12.874 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:18:12.874 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:12.874 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:18:12Z" 2023-07-29T01:18:12.874 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:18:12.875 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052233", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:15:45.371644Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045940", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:15:45.390653Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031634", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:16:32.640507Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:18:12Z" } 2023-07-29T01:18:12.876 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052233', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.371644Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045940', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.390653Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031634', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640507Z'}}] 2023-07-29T01:18:12.876 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:18:12.877 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031634'} 2023-07-29T01:18:12.877 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052233, shard=2 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045940, shard=3 master=00000000000000000000:00000000000000031774 target=00000000000000000000:00000000000000031634 2023-07-29T01:18:17.877 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:18:17.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:18:17.879 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:18:17.933 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:18:17.934 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:18:17.981 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:18:17.982 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:18:17.982 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:18:17.982 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:18:17.982 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:18:17.982 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:18:17.983 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:17.983 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:18:17.983 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:17.983 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:18:17.983 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:17.983 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:18:17.983 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:18:17.984 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:17.984 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:18:17.984 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:17.984 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:18:17.984 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:17.985 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:17.985 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:17.985 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:18:17.985 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:17.985 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:18:17.985 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:18:17.985 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:17.986 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:18:17.986 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:17.986 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:18:17.986 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:17.986 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:17.986 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:17.987 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:18:17.987 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:17.987 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:18:17.987 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:18:17.987 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:17.987 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:18:17.987 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:17.988 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:18:17.988 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:17.988 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:17.988 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:17.988 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:18:17.988 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:17.989 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:18:17.989 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:18:17.989 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:17.989 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:18:17.989 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:17.989 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:18:17.990 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:17.990 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:17.990 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:18:17.991 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:17.991 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:18:17.991 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:18:17.992 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:18:17.992 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:17.992 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:18:17Z" 2023-07-29T01:18:17.992 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:18:17.993 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052233", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:15:45.371644Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045940", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:15:45.390653Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031634", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:16:32.640507Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:18:17Z" } 2023-07-29T01:18:17.994 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052233', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.371644Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045940', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.390653Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031634', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640507Z'}}] 2023-07-29T01:18:17.994 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:18:17.995 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031634'} 2023-07-29T01:18:17.995 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052233, shard=2 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045940, shard=3 master=00000000000000000000:00000000000000031774 target=00000000000000000000:00000000000000031634 2023-07-29T01:18:22.996 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:18:22.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:18:22.997 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:18:23.052 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:18:23.053 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:18:23.100 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:18:23.100 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:18:23.100 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:18:23.100 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:18:23.101 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:18:23.101 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:18:23.101 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:23.101 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:18:23.101 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:23.101 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:18:23.101 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:23.102 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:18:23.102 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:18:23.102 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:23.102 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:18:23.102 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:23.102 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:18:23.103 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:23.103 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:23.103 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:23.103 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:18:23.103 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:23.103 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:18:23.103 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:18:23.104 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:23.104 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:18:23.104 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:23.104 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:18:23.104 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:23.104 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:23.104 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:23.105 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:18:23.105 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:23.105 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:18:23.105 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:18:23.105 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:23.105 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:18:23.105 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:23.106 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:18:23.106 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:23.106 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:23.106 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:23.106 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:18:23.106 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:23.106 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:18:23.107 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:18:23.107 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:23.107 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:18:23.107 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:23.107 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:18:23.107 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:23.107 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:23.108 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:18:23.108 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:23.108 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:18:23.108 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:18:23.108 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:18:23.108 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:23.108 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:18:22Z" 2023-07-29T01:18:23.109 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:18:23.110 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052233", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:15:45.371644Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045940", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:15:45.390653Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031634", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:16:32.640507Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:18:22Z" } 2023-07-29T01:18:23.110 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052233', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.371644Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045940', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.390653Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031634', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640507Z'}}] 2023-07-29T01:18:23.111 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:18:23.111 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031634'} 2023-07-29T01:18:23.111 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052233, shard=2 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045940, shard=3 master=00000000000000000000:00000000000000031774 target=00000000000000000000:00000000000000031634 2023-07-29T01:18:28.112 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:18:28.113 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:18:28.113 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:18:28.168 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:18:28.169 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:18:28.219 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:18:28.219 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:18:28.219 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:18:28.219 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:18:28.220 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:18:28.220 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:18:28.220 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:28.220 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:18:28.220 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:28.220 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:18:28.221 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:28.221 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:18:28.221 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:18:28.221 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:28.221 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:18:28.221 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:28.221 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:18:28.222 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:28.222 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:28.222 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:28.222 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:18:28.222 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:28.222 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:18:28.222 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:18:28.223 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:28.223 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:18:28.223 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:28.223 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:18:28.223 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:28.223 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:28.223 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:28.224 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:18:28.224 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:28.224 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:18:28.224 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:18:28.224 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:28.224 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:18:28.225 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:28.225 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:18:28.225 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:28.225 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:28.225 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:28.225 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:18:28.225 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:28.226 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:18:28.226 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:18:28.226 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:28.226 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:18:28.226 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:28.226 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:18:28.226 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:28.227 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:28.227 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:18:28.227 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:28.227 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:18:28.227 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:18:28.228 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:18:28.228 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:28.229 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:18:28Z" 2023-07-29T01:18:28.229 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:18:28.230 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052233", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:15:45.371644Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045940", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:15:45.390653Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031634", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:16:32.640507Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:18:28Z" } 2023-07-29T01:18:28.230 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052233', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.371644Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045940', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.390653Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031634', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640507Z'}}] 2023-07-29T01:18:28.231 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:18:28.231 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031634'} 2023-07-29T01:18:28.231 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052233, shard=2 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045940, shard=3 master=00000000000000000000:00000000000000031774 target=00000000000000000000:00000000000000031634 2023-07-29T01:18:33.232 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:18:33.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:18:33.235 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:18:33.291 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:18:33.291 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:18:33.338 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:18:33.339 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:18:33.339 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:18:33.339 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:18:33.339 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:18:33.339 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:18:33.340 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:33.340 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:18:33.340 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:33.340 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:18:33.340 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:33.340 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:18:33.341 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:18:33.341 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:33.341 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:18:33.341 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:33.341 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:18:33.341 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:33.342 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:33.342 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:33.342 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:18:33.342 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:33.342 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:18:33.343 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:18:33.343 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:33.343 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:18:33.343 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:33.343 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:18:33.344 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:33.344 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:33.344 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:33.344 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:18:33.344 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:33.344 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:18:33.345 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:18:33.345 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:33.345 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:18:33.345 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:33.345 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:18:33.345 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:33.346 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:33.346 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:33.346 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:18:33.346 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:33.346 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:18:33.346 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:18:33.346 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:33.347 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:18:33.347 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:33.347 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:18:33.348 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:33.348 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:33.348 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:18:33.349 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:33.349 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:18:33.349 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:18:33.349 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:18:33.350 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:33.350 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:18:33Z" 2023-07-29T01:18:33.350 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:18:33.352 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052233", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:15:45.371644Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045940", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:15:45.390653Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031634", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:16:32.640507Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:18:33Z" } 2023-07-29T01:18:33.353 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052233', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.371644Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045940', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.390653Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031634', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640507Z'}}] 2023-07-29T01:18:33.354 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:18:33.354 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031634'} 2023-07-29T01:18:33.355 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052233, shard=2 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045940, shard=3 master=00000000000000000000:00000000000000031774 target=00000000000000000000:00000000000000031634 2023-07-29T01:18:38.356 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:18:38.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:18:38.357 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:18:38.413 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:18:38.414 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:18:38.461 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:18:38.461 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:18:38.461 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:18:38.461 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:18:38.462 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:18:38.462 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:18:38.462 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:38.462 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:18:38.462 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:38.462 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:18:38.463 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:38.463 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:18:38.463 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:18:38.463 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:38.463 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:18:38.463 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:38.463 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:18:38.464 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:38.464 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:38.464 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:38.464 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:18:38.464 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:38.464 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:18:38.465 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:18:38.465 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:38.465 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:18:38.465 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:38.465 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:18:38.465 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:38.466 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:38.466 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:38.466 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:18:38.466 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:38.466 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:18:38.466 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:18:38.467 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:38.467 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:18:38.467 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:38.468 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:18:38.468 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:38.468 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:38.468 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:38.468 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:18:38.468 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:38.469 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:18:38.469 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:18:38.469 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:38.469 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:18:38.469 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:38.469 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:18:38.470 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:38.470 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:38.470 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:18:38.470 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:38.470 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:18:38.470 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:18:38.470 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:18:38.471 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:38.471 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:18:38Z" 2023-07-29T01:18:38.471 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:18:38.472 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052233", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:15:45.371644Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045940", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:15:45.390653Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031634", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:16:32.640507Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:18:38Z" } 2023-07-29T01:18:38.472 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052233', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.371644Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045940', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.390653Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031634', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640507Z'}}] 2023-07-29T01:18:38.473 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:18:38.473 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031634'} 2023-07-29T01:18:38.474 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052233, shard=2 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045940, shard=3 master=00000000000000000000:00000000000000031774 target=00000000000000000000:00000000000000031634 2023-07-29T01:18:43.474 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:18:43.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:18:43.475 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:18:43.531 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:18:43.531 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:18:43.578 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:18:43.579 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:18:43.579 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:18:43.579 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:18:43.579 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:18:43.579 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:18:43.580 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:43.580 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:18:43.580 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:43.580 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:18:43.580 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:43.580 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:18:43.580 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052514", 2023-07-29T01:18:43.581 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:43.581 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:18:43.581 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:43.581 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.772704Z" 2023-07-29T01:18:43.581 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:43.581 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:43.582 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:43.582 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:18:43.582 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:43.582 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:18:43.582 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:18:43.582 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:43.583 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:18:43.583 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:43.583 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:18:43.583 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:43.583 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:43.583 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:43.584 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:18:43.584 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:43.584 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:18:43.584 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000046080", 2023-07-29T01:18:43.584 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:43.584 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:18:43.585 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:43.585 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.709533Z" 2023-07-29T01:18:43.585 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:43.585 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:43.585 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:43.585 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:18:43.586 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:43.586 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:18:43.586 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031774", 2023-07-29T01:18:43.586 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:43.586 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:18:43.586 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:43.586 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.812152Z" 2023-07-29T01:18:43.587 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:43.587 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:43.587 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:18:43.587 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:43.587 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:18:43.587 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:18:43.587 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:18:43.588 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:43.588 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:18:43Z" 2023-07-29T01:18:43.588 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:18:43.589 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052514", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:17:10.772704Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046080", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.709533Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.812152Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:18:43Z" } 2023-07-29T01:18:43.589 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052514', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.772704Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046080', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.709533Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.812152Z'}}] 2023-07-29T01:18:43.590 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:18:43.590 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:18:43.590 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-07-29T01:18:43.591 DEBUG:boto:path=/hiiqos-72/ 2023-07-29T01:18:43.591 DEBUG:boto:auth_path=/hiiqos-72/ 2023-07-29T01:18:43.591 DEBUG:boto:Method: HEAD 2023-07-29T01:18:43.592 DEBUG:boto:Path: /hiiqos-72/ 2023-07-29T01:18:43.592 DEBUG:boto:Data: 2023-07-29T01:18:43.592 DEBUG:boto:Headers: {} 2023-07-29T01:18:43.593 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:18:43.593 DEBUG:boto:Port: 8001 2023-07-29T01:18:43.593 DEBUG:boto:Params: {} 2023-07-29T01:18:43.594 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-29T01:18:43.594 DEBUG:boto:Token: None 2023-07-29T01:18:43.595 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:18:43 GMT /hiiqos-72/ 2023-07-29T01:18:43.595 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:cQh7ieXoBBfnWv31LydzJ+nRYyM= 2023-07-29T01:18:43.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:18:43 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cQh7ieXoBBfnWv31LydzJ+nRYyM=', 'Content-Length': '0'} 2023-07-29T01:18:43.601 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c7d1c1cb18d04321-0064c468f3-2a75-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:18:43 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:18:43.601 DEBUG:boto:path=/hiiqos-72/obj1 2023-07-29T01:18:43.602 DEBUG:boto:auth_path=/hiiqos-72/obj1 2023-07-29T01:18:43.602 DEBUG:boto:Method: HEAD 2023-07-29T01:18:43.602 DEBUG:boto:Path: /hiiqos-72/obj1 2023-07-29T01:18:43.603 DEBUG:boto:Data: 2023-07-29T01:18:43.603 DEBUG:boto:Headers: {} 2023-07-29T01:18:43.603 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:18:43.604 DEBUG:boto:Port: 8001 2023-07-29T01:18:43.604 DEBUG:boto:Params: {} 2023-07-29T01:18:43.604 DEBUG:boto:Token: None 2023-07-29T01:18:43.605 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:18:43 GMT /hiiqos-72/obj1 2023-07-29T01:18:43.605 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:erveqlHrB7vT3BJZyJzhXuFnctU= 2023-07-29T01:18:43.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:18:43 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:erveqlHrB7vT3BJZyJzhXuFnctU=', 'Content-Length': '0'} 2023-07-29T01:18:43.607 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000eaa6f090c15f3e18-0064c468f3-2a75-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 01:18:43 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:18:43.607 DEBUG:boto:path=/hiiqos-72/obj2 2023-07-29T01:18:43.608 DEBUG:boto:auth_path=/hiiqos-72/obj2 2023-07-29T01:18:43.608 DEBUG:boto:Method: HEAD 2023-07-29T01:18:43.608 DEBUG:boto:Path: /hiiqos-72/obj2 2023-07-29T01:18:43.609 DEBUG:boto:Data: 2023-07-29T01:18:43.609 DEBUG:boto:Headers: {} 2023-07-29T01:18:43.609 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:18:43.611 DEBUG:boto:Port: 8001 2023-07-29T01:18:43.611 DEBUG:boto:Params: {} 2023-07-29T01:18:43.611 DEBUG:boto:Token: None 2023-07-29T01:18:43.612 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:18:43 GMT /hiiqos-72/obj2 2023-07-29T01:18:43.612 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:kv8IPJblGjTzXBZV8xllcLYcQPY= 2023-07-29T01:18:43.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:18:43 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kv8IPJblGjTzXBZV8xllcLYcQPY=', 'Content-Length': '0'} 2023-07-29T01:18:43.614 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000453e87c0838c0cc4-0064c468f3-2a75-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 01:18:43 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:18:43.614 DEBUG:boto:path=/hiiqos-72/obj3 2023-07-29T01:18:43.615 DEBUG:boto:auth_path=/hiiqos-72/obj3 2023-07-29T01:18:43.615 DEBUG:boto:Method: HEAD 2023-07-29T01:18:43.615 DEBUG:boto:Path: /hiiqos-72/obj3 2023-07-29T01:18:43.616 DEBUG:boto:Data: 2023-07-29T01:18:43.616 DEBUG:boto:Headers: {} 2023-07-29T01:18:43.616 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:18:43.617 DEBUG:boto:Port: 8001 2023-07-29T01:18:43.617 DEBUG:boto:Params: {} 2023-07-29T01:18:43.617 DEBUG:boto:Token: None 2023-07-29T01:18:43.618 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:18:43 GMT /hiiqos-72/obj3 2023-07-29T01:18:43.618 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:LaqP0NQRyw18k5PVWxByol1NRf0= 2023-07-29T01:18:43.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:18:43 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LaqP0NQRyw18k5PVWxByol1NRf0=', 'Content-Length': '0'} 2023-07-29T01:18:43.620 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000772da370cdd5d427-0064c468f3-2a75-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 01:18:43 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:18:43.621 DEBUG:boto:path=/hiiqos-72/obj4 2023-07-29T01:18:43.621 DEBUG:boto:auth_path=/hiiqos-72/obj4 2023-07-29T01:18:43.621 DEBUG:boto:Method: HEAD 2023-07-29T01:18:43.622 DEBUG:boto:Path: /hiiqos-72/obj4 2023-07-29T01:18:43.622 DEBUG:boto:Data: 2023-07-29T01:18:43.622 DEBUG:boto:Headers: {} 2023-07-29T01:18:43.623 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:18:43.623 DEBUG:boto:Port: 8001 2023-07-29T01:18:43.623 DEBUG:boto:Params: {} 2023-07-29T01:18:43.624 DEBUG:boto:Token: None 2023-07-29T01:18:43.624 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:18:43 GMT /hiiqos-72/obj4 2023-07-29T01:18:43.625 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:4L5Xggp/26nGQ9ma25xX8cnLTBs= 2023-07-29T01:18:43.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:18:43 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4L5Xggp/26nGQ9ma25xX8cnLTBs=', 'Content-Length': '0'} 2023-07-29T01:18:43.626 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000a78b17d69b6a608c-0064c468f3-2a75-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 01:18:43 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:18:43.627 DEBUG:boto:path=/hiiqos-73/ 2023-07-29T01:18:43.627 DEBUG:boto:auth_path=/hiiqos-73/ 2023-07-29T01:18:43.628 DEBUG:boto:Method: HEAD 2023-07-29T01:18:43.628 DEBUG:boto:Path: /hiiqos-73/ 2023-07-29T01:18:43.628 DEBUG:boto:Data: 2023-07-29T01:18:43.629 DEBUG:boto:Headers: {} 2023-07-29T01:18:43.629 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:18:43.629 DEBUG:boto:Port: 8001 2023-07-29T01:18:43.630 DEBUG:boto:Params: {} 2023-07-29T01:18:43.630 DEBUG:boto:Token: None 2023-07-29T01:18:43.630 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:18:43 GMT /hiiqos-73/ 2023-07-29T01:18:43.631 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:dQh/WZ12xx8FsMcxhY9obKwGpeM= 2023-07-29T01:18:43.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:18:43 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dQh/WZ12xx8FsMcxhY9obKwGpeM=', 'Content-Length': '0'} 2023-07-29T01:18:43.633 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000075b1c92f7f593732-0064c468f3-2a75-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:18:43 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:18:43.634 DEBUG:boto:path=/hiiqos-73/obj1 2023-07-29T01:18:43.634 DEBUG:boto:auth_path=/hiiqos-73/obj1 2023-07-29T01:18:43.635 DEBUG:boto:Method: HEAD 2023-07-29T01:18:43.635 DEBUG:boto:Path: /hiiqos-73/obj1 2023-07-29T01:18:43.635 DEBUG:boto:Data: 2023-07-29T01:18:43.636 DEBUG:boto:Headers: {} 2023-07-29T01:18:43.636 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:18:43.636 DEBUG:boto:Port: 8001 2023-07-29T01:18:43.637 DEBUG:boto:Params: {} 2023-07-29T01:18:43.637 DEBUG:boto:Token: None 2023-07-29T01:18:43.637 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:18:43 GMT /hiiqos-73/obj1 2023-07-29T01:18:43.638 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+JfyKUYMFv5UnO+OkWO0JD3LdcY= 2023-07-29T01:18:43.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:18:43 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+JfyKUYMFv5UnO+OkWO0JD3LdcY=', 'Content-Length': '0'} 2023-07-29T01:18:43.640 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:17:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-72:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.10310.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d90eba63c7a67c0a-0064c468f3-2a75-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:18:43 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:18:43.641 DEBUG:boto:path=/hiiqos-73/obj1 2023-07-29T01:18:43.641 DEBUG:boto:auth_path=/hiiqos-73/obj1 2023-07-29T01:18:43.641 DEBUG:boto:Method: GET 2023-07-29T01:18:43.642 DEBUG:boto:Path: /hiiqos-73/obj1 2023-07-29T01:18:43.642 DEBUG:boto:Data: 2023-07-29T01:18:43.643 DEBUG:boto:Headers: {} 2023-07-29T01:18:43.643 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:18:43.643 DEBUG:boto:Port: 8001 2023-07-29T01:18:43.644 DEBUG:boto:Params: {} 2023-07-29T01:18:43.644 DEBUG:boto:Token: None 2023-07-29T01:18:43.644 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 01:18:43 GMT /hiiqos-73/obj1 2023-07-29T01:18:43.645 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:lD+wLDtxIAn/0EjwgSP4Jo/h+AU= 2023-07-29T01:18:43.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:18:43 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lD+wLDtxIAn/0EjwgSP4Jo/h+AU=', 'Content-Length': '0'} 2023-07-29T01:18:43.647 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:17:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-72:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.10310.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a83a7c3d8e7e1c06-0064c468f3-2a75-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:18:43 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:18:43.688 DEBUG:boto:path=/hiiqos-73/obj2 2023-07-29T01:18:43.689 DEBUG:boto:auth_path=/hiiqos-73/obj2 2023-07-29T01:18:43.689 DEBUG:boto:Method: HEAD 2023-07-29T01:18:43.689 DEBUG:boto:Path: /hiiqos-73/obj2 2023-07-29T01:18:43.690 DEBUG:boto:Data: 2023-07-29T01:18:43.690 DEBUG:boto:Headers: {} 2023-07-29T01:18:43.690 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:18:43.691 DEBUG:boto:Port: 8001 2023-07-29T01:18:43.691 DEBUG:boto:Params: {} 2023-07-29T01:18:43.691 DEBUG:boto:Token: None 2023-07-29T01:18:43.692 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:18:43 GMT /hiiqos-73/obj2 2023-07-29T01:18:43.692 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:g23/qhuyRA2ew1JJ3al9lPbvglQ= 2023-07-29T01:18:43.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:18:43 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:g23/qhuyRA2ew1JJ3al9lPbvglQ=', 'Content-Length': '0'} 2023-07-29T01:18:43.695 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:17:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-72:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.10310.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fbe3f7e3f7c3286a-0064c468f3-2a75-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:18:43 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:18:43.695 DEBUG:boto:path=/hiiqos-73/obj2 2023-07-29T01:18:43.696 DEBUG:boto:auth_path=/hiiqos-73/obj2 2023-07-29T01:18:43.696 DEBUG:boto:Method: GET 2023-07-29T01:18:43.696 DEBUG:boto:Path: /hiiqos-73/obj2 2023-07-29T01:18:43.697 DEBUG:boto:Data: 2023-07-29T01:18:43.697 DEBUG:boto:Headers: {} 2023-07-29T01:18:43.697 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:18:43.698 DEBUG:boto:Port: 8001 2023-07-29T01:18:43.698 DEBUG:boto:Params: {} 2023-07-29T01:18:43.699 DEBUG:boto:Token: None 2023-07-29T01:18:43.699 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 01:18:43 GMT /hiiqos-73/obj2 2023-07-29T01:18:43.699 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:tgDbr808w6oWIMnnH0BY+hkxe64= 2023-07-29T01:18:43.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:18:43 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tgDbr808w6oWIMnnH0BY+hkxe64=', 'Content-Length': '0'} 2023-07-29T01:18:43.702 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:17:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-72:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.10310.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d1f0a2ecd3064b3-0064c468f3-2a75-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:18:43 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:18:43.702 DEBUG:boto:path=/hiiqos-73/obj3 2023-07-29T01:18:43.703 DEBUG:boto:auth_path=/hiiqos-73/obj3 2023-07-29T01:18:43.703 DEBUG:boto:Method: HEAD 2023-07-29T01:18:43.703 DEBUG:boto:Path: /hiiqos-73/obj3 2023-07-29T01:18:43.704 DEBUG:boto:Data: 2023-07-29T01:18:43.704 DEBUG:boto:Headers: {} 2023-07-29T01:18:43.704 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:18:43.705 DEBUG:boto:Port: 8001 2023-07-29T01:18:43.705 DEBUG:boto:Params: {} 2023-07-29T01:18:43.705 DEBUG:boto:Token: None 2023-07-29T01:18:43.706 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:18:43 GMT /hiiqos-73/obj3 2023-07-29T01:18:43.706 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:feq6f+rFATV2ZQl1R4aqdmEp5vk= 2023-07-29T01:18:43.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:18:43 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:feq6f+rFATV2ZQl1R4aqdmEp5vk=', 'Content-Length': '0'} 2023-07-29T01:18:43.708 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:17:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-73:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.10310.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007ae6c0519b1f5403-0064c468f3-2a75-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:18:43 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:18:43.709 DEBUG:boto:path=/hiiqos-73/obj3 2023-07-29T01:18:43.709 DEBUG:boto:auth_path=/hiiqos-73/obj3 2023-07-29T01:18:43.709 DEBUG:boto:Method: GET 2023-07-29T01:18:43.710 DEBUG:boto:Path: /hiiqos-73/obj3 2023-07-29T01:18:43.710 DEBUG:boto:Data: 2023-07-29T01:18:43.710 DEBUG:boto:Headers: {} 2023-07-29T01:18:43.711 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:18:43.711 DEBUG:boto:Port: 8001 2023-07-29T01:18:43.711 DEBUG:boto:Params: {} 2023-07-29T01:18:43.712 DEBUG:boto:Token: None 2023-07-29T01:18:43.712 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 01:18:43 GMT /hiiqos-73/obj3 2023-07-29T01:18:43.713 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:zp93/oJOVsvol1FhZ1q8NR4RExA= 2023-07-29T01:18:43.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:18:43 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zp93/oJOVsvol1FhZ1q8NR4RExA=', 'Content-Length': '0'} 2023-07-29T01:18:43.715 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:17:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-73:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.10310.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000092c0dff41e5cfa9a-0064c468f3-2a75-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:18:43 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:18:43.715 DEBUG:boto:path=/hiiqos-73/obj4 2023-07-29T01:18:43.716 DEBUG:boto:auth_path=/hiiqos-73/obj4 2023-07-29T01:18:43.716 DEBUG:boto:Method: HEAD 2023-07-29T01:18:43.716 DEBUG:boto:Path: /hiiqos-73/obj4 2023-07-29T01:18:43.717 DEBUG:boto:Data: 2023-07-29T01:18:43.717 DEBUG:boto:Headers: {} 2023-07-29T01:18:43.717 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:18:43.718 DEBUG:boto:Port: 8001 2023-07-29T01:18:43.718 DEBUG:boto:Params: {} 2023-07-29T01:18:43.718 DEBUG:boto:Token: None 2023-07-29T01:18:43.719 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:18:43 GMT /hiiqos-73/obj4 2023-07-29T01:18:43.719 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:TcIGBx61tmdBKXVduhC0myt0g5k= 2023-07-29T01:18:43.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:18:43 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TcIGBx61tmdBKXVduhC0myt0g5k=', 'Content-Length': '0'} 2023-07-29T01:18:43.721 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000f9e0bb51bad4ee26-0064c468f3-2a75-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 01:18:43 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:18:43.723 INFO:tasks.rgw_multisite_tests:test_sync_multiple_buckets_to_single: ... FAIL 2023-07-29T01:18:43.724 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:18:43.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:18:43.725 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:18:43.770 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:18:43.770 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:18:43.801 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T01:18:43.814 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T01:18:43.814 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:43.814 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:18:43.815 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:17.836402Z" 2023-07-29T01:18:43.815 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:43.815 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:43.815 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593429.606201_1781.1", 2023-07-29T01:18:43.815 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:17:09.606201Z" 2023-07-29T01:18:43.815 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:43.815 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:43.816 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:18:43.816 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T01:18:43.816 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:43.816 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:43.816 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:18:43.816 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:12.811406Z" 2023-07-29T01:18:43.817 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:43.817 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T01:18:43.818 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690591217.836402_294.1', 1: '1_1690593429.606201_1781.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:18:43.818 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T01:18:43.818 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:18:43.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:18:43.819 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:18:43.864 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:18:43.865 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:18:43.896 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:18:43.802+0000 7f790ca96880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:18:43.897 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:18:43.803+0000 7f790ca96880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:18:43.910 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:18:43.910 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:18:43.910 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:18:43.910 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:18:43.911 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:18:43.911 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:18:43.911 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:18:43.911 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:43.911 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:18:43.911 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:43.912 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:18:43.912 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:43.912 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:18:43.912 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:18:43.912 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:43.912 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:18:43.912 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:43.913 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:18:43.913 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:18:43.913 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:43.913 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:43.913 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:43.913 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:18:43.913 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:43.914 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:18:43.914 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593429.606201_1781.1", 2023-07-29T01:18:43.914 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:43.914 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:18:43.914 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:43.914 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:09.606201Z", 2023-07-29T01:18:43.914 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:18:43.915 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:43.915 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:43.915 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:43.915 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:18:43.915 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:43.915 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:18:43.915 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:18:43.916 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:43.916 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:18:43.916 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:43.916 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:18:43.916 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:18:43.916 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:43.916 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:43.917 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:43.917 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:18:43.917 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:43.917 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:18:43.917 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:18:43.917 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:43.917 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:18:43.918 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:43.918 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:18:43.918 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:18:43.918 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:43.918 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:43.918 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:18:43.918 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:43.919 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:18:43.919 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:18:43.919 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:18:43.919 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:43.919 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:18:43Z" 2023-07-29T01:18:43.919 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:18:43.920 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593429.606201_1781.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:09.606201Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:18:43Z" } 2023-07-29T01:18:43.921 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593429.606201_1781.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:09.606201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:18:43.922 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593429.606201_1781.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:18:43.922 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593429.606201_1781.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:18:43.922 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T01:18:43.923 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:18:43.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:18:43.924 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:18:43.966 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:18:43.966 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:18:44.010 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-hiiqos-72","source":{"bucket":"hiiqos-72","zones":["a1"]},"dest":{"bucket":"hiiqos-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hiiqos-73","source":{"bucket":"hiiqos-73","zones":["a1"]},"dest":{"bucket":"hiiqos-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-07-29T01:18:44.011 INFO:rgw_multi.tests:create bucket zone=a1 name=hiiqos-74 2023-07-29T01:18:44.011 DEBUG:boto:path=/hiiqos-74/ 2023-07-29T01:18:44.011 DEBUG:boto:auth_path=/hiiqos-74/ 2023-07-29T01:18:44.012 DEBUG:boto:Method: PUT 2023-07-29T01:18:44.012 DEBUG:boto:Path: /hiiqos-74/ 2023-07-29T01:18:44.012 DEBUG:boto:Data: 2023-07-29T01:18:44.013 DEBUG:boto:Headers: {} 2023-07-29T01:18:44.013 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:18:44.013 DEBUG:boto:Port: 8000 2023-07-29T01:18:44.014 DEBUG:boto:Params: {} 2023-07-29T01:18:44.014 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-29T01:18:44.015 DEBUG:boto:Token: None 2023-07-29T01:18:44.015 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 01:18:44 GMT /hiiqos-74/ 2023-07-29T01:18:44.015 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:qT5wLpaGtFRd1PUoND4d4WeStSg= 2023-07-29T01:18:44.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:18:44 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qT5wLpaGtFRd1PUoND4d4WeStSg=', 'Content-Length': '0'} 2023-07-29T01:18:44.031 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001f9514ab6aac1e1e-0064c468f3-2a77-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:18:43 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:18:44.032 INFO:rgw_multi.tests:create bucket zone=a1 name=hiiqos-75 2023-07-29T01:18:44.032 DEBUG:boto:path=/hiiqos-75/ 2023-07-29T01:18:44.032 DEBUG:boto:auth_path=/hiiqos-75/ 2023-07-29T01:18:44.033 DEBUG:boto:Method: PUT 2023-07-29T01:18:44.033 DEBUG:boto:Path: /hiiqos-75/ 2023-07-29T01:18:44.033 DEBUG:boto:Data: 2023-07-29T01:18:44.034 DEBUG:boto:Headers: {} 2023-07-29T01:18:44.034 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:18:44.034 DEBUG:boto:Port: 8000 2023-07-29T01:18:44.035 DEBUG:boto:Params: {} 2023-07-29T01:18:44.035 DEBUG:boto:Token: None 2023-07-29T01:18:44.035 DEBUG:boto:StringToSign: PUT Sat, 29 Jul 2023 01:18:44 GMT /hiiqos-75/ 2023-07-29T01:18:44.036 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:V6rezyKyp0qv5DWx+JyoLHAiVps= 2023-07-29T01:18:44.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:18:44 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:V6rezyKyp0qv5DWx+JyoLHAiVps=', 'Content-Length': '0'} 2023-07-29T01:18:44.052 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000489e4b88fe6a41af-0064c468f3-2a77-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:18:43 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:18:44.053 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:18:44.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:18:44.054 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:18:44.099 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:18:44.099 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:18:44.129 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T01:18:44.142 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T01:18:44.143 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:44.143 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:18:44.143 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:17.836402Z" 2023-07-29T01:18:44.143 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:44.143 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:44.143 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593523.957474_1842.1", 2023-07-29T01:18:44.143 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:18:43.957474Z" 2023-07-29T01:18:44.144 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:44.144 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:44.144 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:18:44.144 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T01:18:44.144 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:44.144 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:44.145 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:18:44.145 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:12.811406Z" 2023-07-29T01:18:44.145 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:44.145 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T01:18:44.146 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690591217.836402_294.1', 1: '1_1690593523.957474_1842.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:18:44.146 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T01:18:44.147 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:18:44.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:18:44.147 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:18:44.193 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:18:44.193 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:18:44.221 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:18:44.127+0000 7fb16f01a880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:18:44.223 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:18:44.128+0000 7fb16f01a880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:18:44.235 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:18:44.235 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:18:44.236 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:18:44.236 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:18:44.236 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:18:44.236 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:18:44.236 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:18:44.236 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:44.237 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:18:44.237 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:44.237 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:18:44.237 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:44.237 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:18:44.237 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:18:44.237 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:44.238 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:18:44.238 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:44.238 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:18:44.238 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:18:44.238 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:44.238 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:44.238 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:44.239 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:18:44.239 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:44.239 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:18:44.239 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593429.606201_1781.1", 2023-07-29T01:18:44.239 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:44.239 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:18:44.239 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:44.240 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:09.606201Z", 2023-07-29T01:18:44.240 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:18:44.240 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:44.240 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:44.240 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:44.240 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:18:44.241 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:44.241 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:18:44.241 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:18:44.241 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:44.241 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:18:44.241 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:44.241 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:18:44.242 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:18:44.242 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:44.242 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:44.242 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:44.242 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:18:44.242 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:44.242 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:18:44.243 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:18:44.243 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:44.243 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:18:44.243 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:44.243 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:18:44.243 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:18:44.243 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:44.244 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:44.244 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:18:44.244 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:44.244 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:18:44.244 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:18:44.244 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:18:44.245 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:44.245 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:18:44Z" 2023-07-29T01:18:44.245 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:18:44.246 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593429.606201_1781.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:09.606201Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:18:44Z" } 2023-07-29T01:18:44.246 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593429.606201_1781.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:09.606201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:18:44.247 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593523.957474_1842.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:18:44.247 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593429.606201_1781.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:18:44.247 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593523.957474_1842.1 target=1_1690593429.606201_1781.1 2023-07-29T01:18:49.249 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:18:49.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:18:49.250 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:18:49.307 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:18:49.307 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:18:49.339 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:18:49.245+0000 7f2937811880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:18:49.340 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:18:49.246+0000 7f2937811880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:18:49.354 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:18:49.354 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:18:49.354 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:18:49.355 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:18:49.355 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:18:49.355 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:18:49.355 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:18:49.355 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:49.355 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:18:49.356 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:49.356 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:18:49.356 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:49.356 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:18:49.356 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:18:49.356 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:49.357 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:18:49.357 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:49.357 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:18:49.357 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:18:49.357 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:49.357 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:49.358 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:49.358 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:18:49.358 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:49.358 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:18:49.358 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593429.606201_1781.1", 2023-07-29T01:18:49.358 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:49.359 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:18:49.359 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:49.359 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:09.606201Z", 2023-07-29T01:18:49.359 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:18:49.359 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:49.359 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:49.359 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:49.360 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:18:49.360 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:49.360 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:18:49.360 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:18:49.360 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:49.361 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:18:49.361 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:49.361 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:18:49.361 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:18:49.361 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:49.361 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:49.361 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:49.362 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:18:49.362 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:49.362 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:18:49.362 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:18:49.362 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:49.362 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:18:49.363 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:49.363 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:18:49.363 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:18:49.363 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:49.363 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:49.363 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:18:49.364 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:49.364 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:18:49.364 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:18:49.364 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:18:49.364 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:49.364 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:18:49Z" 2023-07-29T01:18:49.365 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:18:49.365 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593429.606201_1781.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:09.606201Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:18:49Z" } 2023-07-29T01:18:49.366 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593429.606201_1781.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:09.606201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:18:49.366 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593523.957474_1842.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:18:49.367 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593429.606201_1781.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:18:49.367 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593523.957474_1842.1 target=1_1690593429.606201_1781.1 2023-07-29T01:18:54.368 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:18:54.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:18:54.369 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:18:54.422 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:18:54.423 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:18:54.456 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:18:54.361+0000 7fb59de96880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:18:54.456 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:18:54.362+0000 7fb59de96880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:18:54.470 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:18:54.471 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:18:54.471 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:18:54.471 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:18:54.471 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:18:54.471 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:18:54.471 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:18:54.472 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:54.472 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:18:54.472 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:54.472 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:18:54.472 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:54.472 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:18:54.473 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:18:54.473 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:54.473 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:18:54.473 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:54.473 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:18:54.473 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:18:54.473 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:54.474 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:54.474 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:54.474 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:18:54.474 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:54.474 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:18:54.474 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593429.606201_1781.1", 2023-07-29T01:18:54.474 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:54.475 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:18:54.475 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:54.475 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:09.606201Z", 2023-07-29T01:18:54.475 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:18:54.475 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:54.475 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:54.475 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:54.476 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:18:54.476 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:54.476 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:18:54.476 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:18:54.477 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:54.477 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:18:54.477 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:54.478 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:18:54.478 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:18:54.478 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:54.479 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:54.479 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:54.479 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:18:54.479 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:54.480 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:18:54.480 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:18:54.480 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:54.481 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:18:54.481 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:54.482 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:18:54.482 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:18:54.482 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:54.482 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:54.483 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:18:54.483 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:54.483 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:18:54.484 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:18:54.484 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:18:54.484 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:54.485 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:18:54Z" 2023-07-29T01:18:54.485 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:18:54.486 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593429.606201_1781.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:09.606201Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:18:54Z" } 2023-07-29T01:18:54.487 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593429.606201_1781.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:09.606201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:18:54.487 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593523.957474_1842.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:18:54.488 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593429.606201_1781.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:18:54.488 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593523.957474_1842.1 target=1_1690593429.606201_1781.1 2023-07-29T01:18:59.490 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:18:59.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:18:59.491 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:18:59.542 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:18:59.542 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:18:59.573 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:18:59.479+0000 7f79da467880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:18:59.574 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:18:59.480+0000 7f79da467880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:18:59.587 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:18:59.587 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:18:59.588 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:18:59.588 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:18:59.588 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:18:59.588 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:18:59.588 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:18:59.589 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:59.589 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:18:59.589 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:59.589 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:18:59.589 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:59.589 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:18:59.589 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:18:59.590 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:59.590 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:18:59.590 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:59.590 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:18:59.590 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:18:59.590 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:59.591 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:59.591 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:59.591 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:18:59.591 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:59.591 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:18:59.591 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593429.606201_1781.1", 2023-07-29T01:18:59.591 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:59.592 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:18:59.592 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:59.592 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:09.606201Z", 2023-07-29T01:18:59.592 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:18:59.592 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:59.592 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:59.593 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:59.593 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:18:59.593 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:59.593 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:18:59.593 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:18:59.593 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:59.593 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:18:59.594 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:59.594 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:18:59.594 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:18:59.594 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:59.594 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:59.594 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:18:59.594 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:18:59.595 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:18:59.595 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:18:59.595 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:18:59.595 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:18:59.595 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:18:59.595 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:18:59.596 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:18:59.596 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:18:59.596 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:59.596 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:18:59.596 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:18:59.596 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:59.596 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:18:59.597 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:18:59.597 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:18:59.597 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:18:59.597 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:18:59Z" 2023-07-29T01:18:59.597 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:18:59.598 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593429.606201_1781.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:09.606201Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:18:59Z" } 2023-07-29T01:18:59.599 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593429.606201_1781.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:09.606201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:18:59.599 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593523.957474_1842.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:18:59.599 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593429.606201_1781.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:18:59.600 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593523.957474_1842.1 target=1_1690593429.606201_1781.1 2023-07-29T01:19:04.601 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:19:04.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:19:04.602 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:19:04.656 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:19:04.656 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:19:04.689 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:19:04.594+0000 7fdf18335880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:19:04.689 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:19:04.595+0000 7fdf18335880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:19:04.703 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:19:04.704 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:19:04.704 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:19:04.704 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:19:04.704 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:19:04.704 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:19:04.704 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:19:04.704 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:04.705 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:19:04.705 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:04.705 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:19:04.705 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:04.705 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:19:04.705 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:19:04.706 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:04.706 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:19:04.706 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:04.706 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:19:04.706 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:19:04.706 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:04.706 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:04.707 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:04.707 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:19:04.707 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:04.707 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:19:04.707 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593429.606201_1781.1", 2023-07-29T01:19:04.707 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:04.707 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:19:04.708 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:04.708 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:09.606201Z", 2023-07-29T01:19:04.708 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:19:04.708 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:04.708 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:04.708 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:04.708 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:19:04.709 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:04.709 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:19:04.709 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:19:04.709 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:04.709 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:19:04.709 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:04.709 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:19:04.710 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:19:04.710 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:04.710 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:04.710 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:04.710 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:19:04.710 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:04.711 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:19:04.711 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:19:04.711 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:04.711 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:19:04.711 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:04.711 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:19:04.711 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:19:04.712 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:04.712 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:04.712 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:19:04.712 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:04.712 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:19:04.712 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:19:04.713 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:19:04.713 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:04.713 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:19:04Z" 2023-07-29T01:19:04.713 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:19:04.714 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593429.606201_1781.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:09.606201Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:19:04Z" } 2023-07-29T01:19:04.715 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593429.606201_1781.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:09.606201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:19:04.715 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593523.957474_1842.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:19:04.715 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593429.606201_1781.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:19:04.716 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593523.957474_1842.1 target=1_1690593429.606201_1781.1 2023-07-29T01:19:09.717 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:19:09.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:19:09.718 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:19:09.770 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:19:09.771 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:19:09.802 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:19:09.708+0000 7f3f28911880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:19:09.803 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:19:09.709+0000 7f3f28911880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:19:09.817 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:19:09.818 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:19:09.818 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:19:09.818 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:19:09.818 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:19:09.818 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:19:09.818 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:19:09.818 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:09.819 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:19:09.819 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:09.819 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:19:09.819 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:09.819 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:19:09.819 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:19:09.820 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:09.820 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:19:09.820 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:09.820 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:19:09.820 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:19:09.820 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:09.820 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:09.821 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:09.821 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:19:09.821 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:09.821 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:19:09.821 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593429.606201_1781.1", 2023-07-29T01:19:09.821 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:09.821 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:19:09.822 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:09.822 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:09.606201Z", 2023-07-29T01:19:09.822 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:19:09.822 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:09.822 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:09.822 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:09.823 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:19:09.823 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:09.823 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:19:09.823 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:19:09.823 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:09.823 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:19:09.823 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:09.824 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:19:09.824 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:19:09.824 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:09.824 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:09.824 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:09.824 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:19:09.825 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:09.825 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:19:09.825 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:19:09.825 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:09.825 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:19:09.825 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:09.825 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:19:09.826 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:19:09.826 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:09.826 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:09.826 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:19:09.826 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:09.826 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:19:09.826 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:19:09.827 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:19:09.827 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:09.827 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:19:09Z" 2023-07-29T01:19:09.827 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:19:09.828 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593429.606201_1781.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:09.606201Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:19:09Z" } 2023-07-29T01:19:09.828 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593429.606201_1781.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:09.606201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:19:09.829 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593523.957474_1842.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:19:09.829 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593429.606201_1781.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:19:09.830 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593523.957474_1842.1 target=1_1690593429.606201_1781.1 2023-07-29T01:19:14.830 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:19:14.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:19:14.832 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:19:14.884 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:19:14.885 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:19:14.917 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:19:14.823+0000 7fda58e1a880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:19:14.918 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:19:14.824+0000 7fda58e1a880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:19:14.932 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:19:14.932 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:19:14.933 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:19:14.933 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:19:14.933 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:19:14.933 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:19:14.933 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:19:14.933 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:14.934 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:19:14.934 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:14.934 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:19:14.934 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:14.934 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:19:14.934 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:19:14.934 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:14.935 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:19:14.935 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:14.935 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:19:14.935 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:19:14.935 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:14.935 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:14.935 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:14.936 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:19:14.936 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:14.936 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:19:14.936 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593523.957474_1842.1", 2023-07-29T01:19:14.936 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:14.936 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:19:14.936 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:14.937 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:18:43.957474Z", 2023-07-29T01:19:14.937 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:19:14.937 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:14.937 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:14.937 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:14.937 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:19:14.938 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:14.938 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:19:14.938 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:19:14.938 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:14.938 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:19:14.938 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:14.938 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:19:14.939 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:19:14.939 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:14.939 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:14.939 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:14.939 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:19:14.939 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:14.939 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:19:14.940 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:19:14.940 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:14.940 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:19:14.940 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:14.940 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:19:14.940 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:19:14.940 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:14.941 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:14.941 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:19:14.941 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:14.941 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:19:14.941 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:19:14.941 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:19:14.941 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:14.942 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:19:14Z" 2023-07-29T01:19:14.942 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:19:14.943 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593523.957474_1842.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:18:43.957474Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:19:14Z" } 2023-07-29T01:19:14.943 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593523.957474_1842.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:18:43.957474Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:19:14.944 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593523.957474_1842.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:19:14.944 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593523.957474_1842.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:19:14.944 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T01:19:14.945 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:19:14.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:19:14.945 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:19:14.992 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:19:14.993 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:19:15.038 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-hiiqos-72","source":{"bucket":"hiiqos-72","zones":["a1"]},"dest":{"bucket":"hiiqos-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hiiqos-73","source":{"bucket":"hiiqos-73","zones":["a1"]},"dest":{"bucket":"hiiqos-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-07-29T01:19:15.038 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:19:15.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:19:15.039 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:19:15.088 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:19:15.089 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:19:15.133 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-hiiqos-72","source":{"bucket":"hiiqos-72","zones":["a1"]},"dest":{"bucket":"hiiqos-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hiiqos-73","source":{"bucket":"hiiqos-73","zones":["a1"]},"dest":{"bucket":"hiiqos-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-07-29T01:19:15.134 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hiiqos-74', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hiiqos-74', '--dest-bucket=hiiqos-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:19:15.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hiiqos-74', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hiiqos-74', '--dest-bucket=hiiqos-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:19:15.135 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-hiiqos-74 --source-zones=a1 --dest-zones=a2 --source-bucket=hiiqos-74 --dest-bucket=hiiqos-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:19:15.185 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:19:15.186 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:19:15.233 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-hiiqos-72","source":{"bucket":"hiiqos-72","zones":["a1"]},"dest":{"bucket":"hiiqos-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hiiqos-73","source":{"bucket":"hiiqos-73","zones":["a1"]},"dest":{"bucket":"hiiqos-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hiiqos-74","source":{"bucket":"hiiqos-74","zones":["a1"]},"dest":{"bucket":"hiiqos-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-07-29T01:19:15.234 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hiiqos-75', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hiiqos-74', '--dest-bucket=hiiqos-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:19:15.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hiiqos-75', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hiiqos-74', '--dest-bucket=hiiqos-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:19:15.235 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-hiiqos-75 --source-zones=a1 --dest-zones=a2 --source-bucket=hiiqos-74 --dest-bucket=hiiqos-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:19:15.285 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:19:15.285 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:19:15.330 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-hiiqos-72","source":{"bucket":"hiiqos-72","zones":["a1"]},"dest":{"bucket":"hiiqos-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hiiqos-73","source":{"bucket":"hiiqos-73","zones":["a1"]},"dest":{"bucket":"hiiqos-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hiiqos-74","source":{"bucket":"hiiqos-74","zones":["a1"]},"dest":{"bucket":"hiiqos-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hiiqos-75","source":{"bucket":"hiiqos-74","zones":["a1"]},"dest":{"bucket":"hiiqos-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-07-29T01:19:15.330 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hiiqos-74', '--dest-bucket=hiiqos-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:19:15.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hiiqos-74', '--dest-bucket=hiiqos-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:19:15.331 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1 --dest-zones=a2 --source-bucket=hiiqos-74 --dest-bucket=hiiqos-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:19:15.382 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:19:15.383 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:19:15.428 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-hiiqos-72","source":{"bucket":"hiiqos-72","zones":["a1"]},"dest":{"bucket":"hiiqos-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hiiqos-73","source":{"bucket":"hiiqos-73","zones":["a1"]},"dest":{"bucket":"hiiqos-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hiiqos-74","source":{"bucket":"hiiqos-74","zones":["a1"]},"dest":{"bucket":"hiiqos-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hiiqos-75","source":{"bucket":"hiiqos-74","zones":["a1"]},"dest":{"bucket":"hiiqos-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"hiiqos-74","zones":["a1"]},"dest":{"bucket":"hiiqos-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-07-29T01:19:15.429 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:19:15.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:19:15.429 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:19:15.478 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:19:15.479 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:19:15.525 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-hiiqos-72","source":{"bucket":"hiiqos-72","zones":["a1"]},"dest":{"bucket":"hiiqos-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hiiqos-73","source":{"bucket":"hiiqos-73","zones":["a1"]},"dest":{"bucket":"hiiqos-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hiiqos-74","source":{"bucket":"hiiqos-74","zones":["a1"]},"dest":{"bucket":"hiiqos-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hiiqos-75","source":{"bucket":"hiiqos-74","zones":["a1"]},"dest":{"bucket":"hiiqos-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"hiiqos-74","zones":["a1"]},"dest":{"bucket":"hiiqos-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-07-29T01:19:15.526 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:19:15.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:19:15.527 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:19:15.572 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:19:15.572 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:19:15.621 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:19:15.622 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:19:15.622 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 21, 2023-07-29T01:19:15.622 INFO:teuthology.orchestra.run.smithi101.stdout: "predecessor_uuid": "a0987484-ce37-4520-a564-54b7ea8c5251", 2023-07-29T01:19:15.622 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": [], 2023-07-29T01:19:15.622 INFO:teuthology.orchestra.run.smithi101.stdout: "period_map": { 2023-07-29T01:19:15.622 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:19:15.622 INFO:teuthology.orchestra.run.smithi101.stdout: "zonegroups": [ 2023-07-29T01:19:15.623 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:15.623 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "a", 2023-07-29T01:19:15.623 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a", 2023-07-29T01:19:15.623 INFO:teuthology.orchestra.run.smithi101.stdout: "api_name": "", 2023-07-29T01:19:15.623 INFO:teuthology.orchestra.run.smithi101.stdout: "is_master": true, 2023-07-29T01:19:15.623 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:19:15.624 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-29T01:19:15.624 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:19:15.624 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames": [], 2023-07-29T01:19:15.624 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames_s3website": [], 2023-07-29T01:19:15.624 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:19:15.624 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:19:15.624 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:15.625 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:19:15.625 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a1", 2023-07-29T01:19:15.625 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:19:15.625 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-29T01:19:15.625 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:19:15.625 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T01:19:15.625 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T01:19:15.626 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T01:19:15.626 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T01:19:15.626 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T01:19:15.626 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T01:19:15.627 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T01:19:15.627 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T01:19:15.627 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T01:19:15.627 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T01:19:15.627 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:19:15.627 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:19:15.627 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:15.628 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:15.628 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-29T01:19:15.628 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a2", 2023-07-29T01:19:15.628 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:19:15.628 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8001" 2023-07-29T01:19:15.628 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:19:15.628 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T01:19:15.629 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T01:19:15.629 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T01:19:15.629 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T01:19:15.629 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T01:19:15.629 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T01:19:15.629 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T01:19:15.629 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T01:19:15.630 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T01:19:15.630 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T01:19:15.630 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:19:15.630 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:19:15.630 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:15.630 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:19:15.630 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_targets": [ 2023-07-29T01:19:15.631 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:15.631 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default-placement", 2023-07-29T01:19:15.631 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [], 2023-07-29T01:19:15.631 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": [ 2023-07-29T01:19:15.631 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD" 2023-07-29T01:19:15.631 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:19:15.631 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:15.632 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:19:15.632 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "default-placement", 2023-07-29T01:19:15.632 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T01:19:15.632 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_policy": { 2023-07-29T01:19:15.632 INFO:teuthology.orchestra.run.smithi101.stdout: "groups": [ 2023-07-29T01:19:15.632 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:15.633 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "sync-group", 2023-07-29T01:19:15.633 INFO:teuthology.orchestra.run.smithi101.stdout: "data_flow": { 2023-07-29T01:19:15.633 INFO:teuthology.orchestra.run.smithi101.stdout: "symmetrical": [ 2023-07-29T01:19:15.633 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:15.633 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "sync-flow1", 2023-07-29T01:19:15.633 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:19:15.633 INFO:teuthology.orchestra.run.smithi101.stdout: "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:19:15.634 INFO:teuthology.orchestra.run.smithi101.stdout: "59d1bb30-3b1c-476f-9111-052c1ea0b711" 2023-07-29T01:19:15.634 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:19:15.634 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:15.634 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:19:15.634 INFO:teuthology.orchestra.run.smithi101.stdout: "directional": [ 2023-07-29T01:19:15.634 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:15.634 INFO:teuthology.orchestra.run.smithi101.stdout: "source_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:19:15.635 INFO:teuthology.orchestra.run.smithi101.stdout: "dest_zone": "59d1bb30-3b1c-476f-9111-052c1ea0b711" 2023-07-29T01:19:15.635 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:15.635 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:19:15.635 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:15.635 INFO:teuthology.orchestra.run.smithi101.stdout: "pipes": [ 2023-07-29T01:19:15.635 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:15.635 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "sync-pipe-hiiqos-72", 2023-07-29T01:19:15.636 INFO:teuthology.orchestra.run.smithi101.stdout: "source": { 2023-07-29T01:19:15.636 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket": "hiiqos-72", 2023-07-29T01:19:15.636 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:19:15.636 INFO:teuthology.orchestra.run.smithi101.stdout: "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433" 2023-07-29T01:19:15.636 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:19:15.636 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:15.636 INFO:teuthology.orchestra.run.smithi101.stdout: "dest": { 2023-07-29T01:19:15.637 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket": "hiiqos-73", 2023-07-29T01:19:15.637 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:19:15.637 INFO:teuthology.orchestra.run.smithi101.stdout: "59d1bb30-3b1c-476f-9111-052c1ea0b711" 2023-07-29T01:19:15.637 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:19:15.637 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:15.637 INFO:teuthology.orchestra.run.smithi101.stdout: "params": { 2023-07-29T01:19:15.637 INFO:teuthology.orchestra.run.smithi101.stdout: "source": { 2023-07-29T01:19:15.638 INFO:teuthology.orchestra.run.smithi101.stdout: "filter": { 2023-07-29T01:19:15.638 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [] 2023-07-29T01:19:15.638 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:15.638 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:15.638 INFO:teuthology.orchestra.run.smithi101.stdout: "dest": {}, 2023-07-29T01:19:15.638 INFO:teuthology.orchestra.run.smithi101.stdout: "priority": 0, 2023-07-29T01:19:15.638 INFO:teuthology.orchestra.run.smithi101.stdout: "mode": "system", 2023-07-29T01:19:15.639 INFO:teuthology.orchestra.run.smithi101.stdout: "user": "" 2023-07-29T01:19:15.639 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:15.639 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:15.639 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:15.639 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "sync-pipe-hiiqos-73", 2023-07-29T01:19:15.639 INFO:teuthology.orchestra.run.smithi101.stdout: "source": { 2023-07-29T01:19:15.639 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket": "hiiqos-73", 2023-07-29T01:19:15.640 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:19:15.640 INFO:teuthology.orchestra.run.smithi101.stdout: "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433" 2023-07-29T01:19:15.640 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:19:15.640 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:15.640 INFO:teuthology.orchestra.run.smithi101.stdout: "dest": { 2023-07-29T01:19:15.640 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket": "hiiqos-73", 2023-07-29T01:19:15.640 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:19:15.641 INFO:teuthology.orchestra.run.smithi101.stdout: "59d1bb30-3b1c-476f-9111-052c1ea0b711" 2023-07-29T01:19:15.641 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:19:15.641 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:15.641 INFO:teuthology.orchestra.run.smithi101.stdout: "params": { 2023-07-29T01:19:15.641 INFO:teuthology.orchestra.run.smithi101.stdout: "source": { 2023-07-29T01:19:15.641 INFO:teuthology.orchestra.run.smithi101.stdout: "filter": { 2023-07-29T01:19:15.641 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [] 2023-07-29T01:19:15.642 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:15.642 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:15.642 INFO:teuthology.orchestra.run.smithi101.stdout: "dest": {}, 2023-07-29T01:19:15.642 INFO:teuthology.orchestra.run.smithi101.stdout: "priority": 0, 2023-07-29T01:19:15.642 INFO:teuthology.orchestra.run.smithi101.stdout: "mode": "system", 2023-07-29T01:19:15.642 INFO:teuthology.orchestra.run.smithi101.stdout: "user": "" 2023-07-29T01:19:15.642 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:15.643 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:15.643 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:15.643 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "sync-pipe-hiiqos-74", 2023-07-29T01:19:15.643 INFO:teuthology.orchestra.run.smithi101.stdout: "source": { 2023-07-29T01:19:15.643 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket": "hiiqos-74", 2023-07-29T01:19:15.643 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:19:15.643 INFO:teuthology.orchestra.run.smithi101.stdout: "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433" 2023-07-29T01:19:15.644 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:19:15.644 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:15.644 INFO:teuthology.orchestra.run.smithi101.stdout: "dest": { 2023-07-29T01:19:15.644 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket": "hiiqos-74", 2023-07-29T01:19:15.644 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:19:15.644 INFO:teuthology.orchestra.run.smithi101.stdout: "59d1bb30-3b1c-476f-9111-052c1ea0b711" 2023-07-29T01:19:15.644 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:19:15.645 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:15.645 INFO:teuthology.orchestra.run.smithi101.stdout: "params": { 2023-07-29T01:19:15.645 INFO:teuthology.orchestra.run.smithi101.stdout: "source": { 2023-07-29T01:19:15.645 INFO:teuthology.orchestra.run.smithi101.stdout: "filter": { 2023-07-29T01:19:15.645 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [] 2023-07-29T01:19:15.645 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:15.645 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:15.646 INFO:teuthology.orchestra.run.smithi101.stdout: "dest": {}, 2023-07-29T01:19:15.646 INFO:teuthology.orchestra.run.smithi101.stdout: "priority": 0, 2023-07-29T01:19:15.646 INFO:teuthology.orchestra.run.smithi101.stdout: "mode": "system", 2023-07-29T01:19:15.646 INFO:teuthology.orchestra.run.smithi101.stdout: "user": "" 2023-07-29T01:19:15.646 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:15.646 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:15.646 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:15.647 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "sync-pipe-hiiqos-75", 2023-07-29T01:19:15.647 INFO:teuthology.orchestra.run.smithi101.stdout: "source": { 2023-07-29T01:19:15.647 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket": "hiiqos-74", 2023-07-29T01:19:15.647 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:19:15.647 INFO:teuthology.orchestra.run.smithi101.stdout: "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433" 2023-07-29T01:19:15.647 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:19:15.647 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:15.648 INFO:teuthology.orchestra.run.smithi101.stdout: "dest": { 2023-07-29T01:19:15.648 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket": "hiiqos-75", 2023-07-29T01:19:15.648 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:19:15.648 INFO:teuthology.orchestra.run.smithi101.stdout: "59d1bb30-3b1c-476f-9111-052c1ea0b711" 2023-07-29T01:19:15.648 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:19:15.648 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:15.649 INFO:teuthology.orchestra.run.smithi101.stdout: "params": { 2023-07-29T01:19:15.649 INFO:teuthology.orchestra.run.smithi101.stdout: "source": { 2023-07-29T01:19:15.649 INFO:teuthology.orchestra.run.smithi101.stdout: "filter": { 2023-07-29T01:19:15.649 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [] 2023-07-29T01:19:15.649 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:15.649 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:15.649 INFO:teuthology.orchestra.run.smithi101.stdout: "dest": {}, 2023-07-29T01:19:15.650 INFO:teuthology.orchestra.run.smithi101.stdout: "priority": 0, 2023-07-29T01:19:15.650 INFO:teuthology.orchestra.run.smithi101.stdout: "mode": "system", 2023-07-29T01:19:15.650 INFO:teuthology.orchestra.run.smithi101.stdout: "user": "" 2023-07-29T01:19:15.650 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:15.650 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:15.650 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:15.650 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "sync-pipe", 2023-07-29T01:19:15.651 INFO:teuthology.orchestra.run.smithi101.stdout: "source": { 2023-07-29T01:19:15.651 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket": "hiiqos-74", 2023-07-29T01:19:15.651 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:19:15.651 INFO:teuthology.orchestra.run.smithi101.stdout: "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433" 2023-07-29T01:19:15.651 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:19:15.651 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:15.651 INFO:teuthology.orchestra.run.smithi101.stdout: "dest": { 2023-07-29T01:19:15.652 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket": "hiiqos-75", 2023-07-29T01:19:15.652 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:19:15.652 INFO:teuthology.orchestra.run.smithi101.stdout: "59d1bb30-3b1c-476f-9111-052c1ea0b711" 2023-07-29T01:19:15.652 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:19:15.652 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:15.652 INFO:teuthology.orchestra.run.smithi101.stdout: "params": { 2023-07-29T01:19:15.652 INFO:teuthology.orchestra.run.smithi101.stdout: "source": { 2023-07-29T01:19:15.653 INFO:teuthology.orchestra.run.smithi101.stdout: "filter": { 2023-07-29T01:19:15.653 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [] 2023-07-29T01:19:15.653 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:15.653 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:15.653 INFO:teuthology.orchestra.run.smithi101.stdout: "dest": {}, 2023-07-29T01:19:15.653 INFO:teuthology.orchestra.run.smithi101.stdout: "priority": 0, 2023-07-29T01:19:15.653 INFO:teuthology.orchestra.run.smithi101.stdout: "mode": "system", 2023-07-29T01:19:15.654 INFO:teuthology.orchestra.run.smithi101.stdout: "user": "" 2023-07-29T01:19:15.654 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:15.654 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:15.654 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:19:15.654 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "enabled" 2023-07-29T01:19:15.654 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:15.654 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:19:15.655 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:15.655 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled_features": [ 2023-07-29T01:19:15.655 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:19:15.655 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:19:15.655 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:15.655 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:15.655 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "b", 2023-07-29T01:19:15.656 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b", 2023-07-29T01:19:15.656 INFO:teuthology.orchestra.run.smithi101.stdout: "api_name": "", 2023-07-29T01:19:15.656 INFO:teuthology.orchestra.run.smithi101.stdout: "is_master": false, 2023-07-29T01:19:15.656 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:19:15.656 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-29T01:19:15.656 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:19:15.656 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames": [], 2023-07-29T01:19:15.657 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames_s3website": [], 2023-07-29T01:19:15.657 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T01:19:15.657 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:19:15.657 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:15.657 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T01:19:15.657 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b1", 2023-07-29T01:19:15.657 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:19:15.658 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-29T01:19:15.658 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:19:15.658 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T01:19:15.658 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T01:19:15.658 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T01:19:15.658 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T01:19:15.658 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T01:19:15.659 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T01:19:15.659 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T01:19:15.659 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T01:19:15.659 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T01:19:15.659 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T01:19:15.659 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:19:15.659 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:19:15.660 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:15.660 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:15.660 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-29T01:19:15.660 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b2", 2023-07-29T01:19:15.660 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:19:15.660 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8001" 2023-07-29T01:19:15.661 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:19:15.661 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T01:19:15.662 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T01:19:15.662 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T01:19:15.662 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T01:19:15.662 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T01:19:15.662 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T01:19:15.662 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T01:19:15.662 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T01:19:15.663 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T01:19:15.663 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T01:19:15.663 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:19:15.663 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:19:15.663 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:15.663 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:19:15.663 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_targets": [ 2023-07-29T01:19:15.664 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:15.664 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default-placement", 2023-07-29T01:19:15.664 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [], 2023-07-29T01:19:15.664 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": [ 2023-07-29T01:19:15.664 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD" 2023-07-29T01:19:15.664 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:19:15.665 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:15.665 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:19:15.665 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "default-placement", 2023-07-29T01:19:15.665 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T01:19:15.665 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_policy": { 2023-07-29T01:19:15.665 INFO:teuthology.orchestra.run.smithi101.stdout: "groups": [] 2023-07-29T01:19:15.665 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:15.666 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled_features": [ 2023-07-29T01:19:15.666 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:19:15.666 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:19:15.666 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:15.666 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:19:15.666 INFO:teuthology.orchestra.run.smithi101.stdout: "short_zone_ids": [ 2023-07-29T01:19:15.666 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:15.667 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:19:15.667 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 2896287231 2023-07-29T01:19:15.667 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:15.667 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:15.667 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-29T01:19:15.667 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 815606810 2023-07-29T01:19:15.667 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:15.667 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:15.668 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T01:19:15.668 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 2009171363 2023-07-29T01:19:15.668 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:15.668 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:15.668 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-29T01:19:15.668 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 3105190295 2023-07-29T01:19:15.669 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:15.669 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:19:15.669 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:15.669 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zonegroup": "a", 2023-07-29T01:19:15.669 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:19:15.669 INFO:teuthology.orchestra.run.smithi101.stdout: "period_config": { 2023-07-29T01:19:15.669 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_quota": { 2023-07-29T01:19:15.670 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-29T01:19:15.670 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-29T01:19:15.670 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-29T01:19:15.670 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-29T01:19:15.670 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-29T01:19:15.670 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:15.670 INFO:teuthology.orchestra.run.smithi101.stdout: "user_quota": { 2023-07-29T01:19:15.670 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-29T01:19:15.671 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-29T01:19:15.671 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-29T01:19:15.671 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-29T01:19:15.671 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-29T01:19:15.671 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:15.671 INFO:teuthology.orchestra.run.smithi101.stdout: "user_ratelimit": { 2023-07-29T01:19:15.671 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-29T01:19:15.672 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-29T01:19:15.672 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-29T01:19:15.672 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-29T01:19:15.672 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-29T01:19:15.672 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:15.672 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_ratelimit": { 2023-07-29T01:19:15.672 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-29T01:19:15.673 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-29T01:19:15.673 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-29T01:19:15.673 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-29T01:19:15.673 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-29T01:19:15.673 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:15.673 INFO:teuthology.orchestra.run.smithi101.stdout: "anonymous_ratelimit": { 2023-07-29T01:19:15.673 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-29T01:19:15.674 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-29T01:19:15.674 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-29T01:19:15.674 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-29T01:19:15.674 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-29T01:19:15.674 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:15.674 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:15.674 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T01:19:15.675 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_name": "test-realm", 2023-07-29T01:19:15.675 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:19:15.675 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:19:15.675 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:19:15.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:19:15.676 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:19:15.731 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:19:15.731 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:19:15.873 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-hiiqos-72","source":{"bucket":"hiiqos-72","zones":["a1"]},"dest":{"bucket":"hiiqos-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hiiqos-73","source":{"bucket":"hiiqos-73","zones":["a1"]},"dest":{"bucket":"hiiqos-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hiiqos-74","source":{"bucket":"hiiqos-74","zones":["a1"]},"dest":{"bucket":"hiiqos-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hiiqos-75","source":{"bucket":"hiiqos-74","zones":["a1"]},"dest":{"bucket":"hiiqos-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"hiiqos-74","zones":["a1"]},"dest":{"bucket":"hiiqos-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-07-29T01:19:15.874 DEBUG:boto:path=/hiiqos-74/ 2023-07-29T01:19:15.874 DEBUG:boto:auth_path=/hiiqos-74/ 2023-07-29T01:19:15.875 DEBUG:boto:Method: HEAD 2023-07-29T01:19:15.875 DEBUG:boto:Path: /hiiqos-74/ 2023-07-29T01:19:15.875 DEBUG:boto:Data: 2023-07-29T01:19:15.876 DEBUG:boto:Headers: {} 2023-07-29T01:19:15.876 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:19:15.876 DEBUG:boto:Port: 8000 2023-07-29T01:19:15.877 DEBUG:boto:Params: {} 2023-07-29T01:19:15.877 DEBUG:boto:Token: None 2023-07-29T01:19:15.877 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:19:15 GMT /hiiqos-74/ 2023-07-29T01:19:15.878 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Sx0WwIhGzoTAJuGasfBoQ9f0sHw= 2023-07-29T01:19:15.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:19:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Sx0WwIhGzoTAJuGasfBoQ9f0sHw=', 'Content-Length': '0'} 2023-07-29T01:19:15.891 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e6c7083a0ec1cd9e-0064c46913-2bc6-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:19:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:19:15.891 DEBUG:boto:path=/hiiqos-74/obj1 2023-07-29T01:19:15.892 DEBUG:boto:auth_path=/hiiqos-74/obj1 2023-07-29T01:19:15.892 DEBUG:boto:Method: PUT 2023-07-29T01:19:15.893 DEBUG:boto:Path: /hiiqos-74/obj1 2023-07-29T01:19:15.893 DEBUG:boto:Data: 2023-07-29T01:19:15.893 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T01:19:15.894 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:19:15.894 DEBUG:boto:Port: 8000 2023-07-29T01:19:15.894 DEBUG:boto:Params: {} 2023-07-29T01:19:15.895 DEBUG:boto:Token: None 2023-07-29T01:19:15.895 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 01:19:15 GMT /hiiqos-74/obj1 2023-07-29T01:19:15.895 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:UMKaiyAp9ABJkJDG+KuC9G4mRBM= 2023-07-29T01:19:15.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 01:19:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UMKaiyAp9ABJkJDG+KuC9G4mRBM='} 2023-07-29T01:19:15.921 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000060f549608b754317-0064c46913-2bc6-a1'), ('Date', 'Sat, 29 Jul 2023 01:19:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:19:15.921 DEBUG:boto:path=/hiiqos-74/ 2023-07-29T01:19:15.921 DEBUG:boto:auth_path=/hiiqos-74/ 2023-07-29T01:19:15.922 DEBUG:boto:Method: HEAD 2023-07-29T01:19:15.922 DEBUG:boto:Path: /hiiqos-74/ 2023-07-29T01:19:15.922 DEBUG:boto:Data: 2023-07-29T01:19:15.923 DEBUG:boto:Headers: {} 2023-07-29T01:19:15.923 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:19:15.923 DEBUG:boto:Port: 8000 2023-07-29T01:19:15.924 DEBUG:boto:Params: {} 2023-07-29T01:19:15.924 DEBUG:boto:Token: None 2023-07-29T01:19:15.925 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:19:15 GMT /hiiqos-74/ 2023-07-29T01:19:15.925 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Sx0WwIhGzoTAJuGasfBoQ9f0sHw= 2023-07-29T01:19:15.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:19:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Sx0WwIhGzoTAJuGasfBoQ9f0sHw=', 'Content-Length': '0'} 2023-07-29T01:19:15.930 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003a9859d16600ea75-0064c46913-2bc6-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:19:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:19:15.930 DEBUG:boto:path=/hiiqos-74/obj2 2023-07-29T01:19:15.931 DEBUG:boto:auth_path=/hiiqos-74/obj2 2023-07-29T01:19:15.931 DEBUG:boto:Method: PUT 2023-07-29T01:19:15.931 DEBUG:boto:Path: /hiiqos-74/obj2 2023-07-29T01:19:15.932 DEBUG:boto:Data: 2023-07-29T01:19:15.932 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T01:19:15.932 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:19:15.933 DEBUG:boto:Port: 8000 2023-07-29T01:19:15.933 DEBUG:boto:Params: {} 2023-07-29T01:19:15.934 DEBUG:boto:Token: None 2023-07-29T01:19:15.934 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 01:19:15 GMT /hiiqos-74/obj2 2023-07-29T01:19:15.934 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:l6saqn1Gr4tSNkrrkIWmRP1FbhA= 2023-07-29T01:19:15.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 01:19:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:l6saqn1Gr4tSNkrrkIWmRP1FbhA='} 2023-07-29T01:19:15.976 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002e73dedc3889b289-0064c46913-2bc6-a1'), ('Date', 'Sat, 29 Jul 2023 01:19:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:19:15.977 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:19:15.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:19:15.978 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:19:16.095 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:19:16.096 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:19:16.163 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T01:19:16.188 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T01:19:16.189 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:16.189 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:19:16.189 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:17.836402Z" 2023-07-29T01:19:16.189 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:16.189 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:16.189 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593523.957474_1842.1", 2023-07-29T01:19:16.190 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:18:43.957474Z" 2023-07-29T01:19:16.190 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:16.190 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:16.190 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:19:16.190 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T01:19:16.190 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:16.190 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:16.191 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:19:16.191 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:12.811406Z" 2023-07-29T01:19:16.191 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:16.191 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T01:19:16.192 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690591217.836402_294.1', 1: '1_1690593523.957474_1842.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:19:16.192 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T01:19:16.193 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:19:16.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:19:16.193 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:19:16.267 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:19:16.267 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:19:16.309 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:19:16.214+0000 7fdaf480f880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:19:16.310 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:19:16.216+0000 7fdaf480f880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:19:16.328 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:19:16.328 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:19:16.329 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:19:16.329 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:19:16.329 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:19:16.329 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:19:16.329 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:19:16.329 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:16.330 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:19:16.330 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:16.330 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:19:16.330 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:16.330 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:19:16.330 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:19:16.330 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:16.331 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:19:16.331 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:16.331 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:19:16.331 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:19:16.331 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:16.331 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:16.332 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:16.332 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:19:16.332 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:16.332 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:19:16.332 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593523.957474_1842.1", 2023-07-29T01:19:16.332 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:16.332 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:19:16.333 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:16.333 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:18:43.957474Z", 2023-07-29T01:19:16.333 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:19:16.333 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:16.333 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:16.333 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:16.333 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:19:16.334 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:16.334 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:19:16.334 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:19:16.334 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:16.334 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:19:16.334 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:16.334 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:19:16.335 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:19:16.335 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:16.335 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:16.335 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:16.335 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:19:16.335 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:16.335 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:19:16.336 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:19:16.336 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:16.336 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:19:16.336 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:16.336 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:19:16.336 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:19:16.337 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:16.337 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:16.337 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:19:16.337 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:16.337 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:19:16.337 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:19:16.337 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:19:16.338 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:16.338 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:19:16Z" 2023-07-29T01:19:16.338 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:19:16.339 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593523.957474_1842.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:18:43.957474Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:19:16Z" } 2023-07-29T01:19:16.339 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593523.957474_1842.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:18:43.957474Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:19:16.340 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593523.957474_1842.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:19:16.340 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593523.957474_1842.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:19:16.340 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T01:19:16.341 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:19:16.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:19:16.341 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:19:16.397 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:19:16.398 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:19:16.447 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T01:19:16.447 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:16.447 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052655", 2023-07-29T01:19:16.448 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:19:15.824401Z" 2023-07-29T01:19:16.448 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:16.448 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:16.448 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:19:16.448 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:19:16.449 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:16.449 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:16.449 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000046221", 2023-07-29T01:19:16.449 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:19:15.858505Z" 2023-07-29T01:19:16.449 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:16.449 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:16.450 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031774", 2023-07-29T01:19:16.450 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:17:10.812152Z" 2023-07-29T01:19:16.450 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:16.450 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T01:19:16.451 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000052655', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:19:16.451 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-07-29T01:19:16.452 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:19:16.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:19:16.452 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:19:16.500 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:19:16.501 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:19:16.543 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:19:16.543 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:19:16.543 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:19:16.544 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:19:16.544 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:19:16.544 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:19:16.544 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:16.544 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:19:16.545 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:16.545 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:19:16.545 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:16.545 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:19:16.545 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052514", 2023-07-29T01:19:16.545 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:16.546 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:19:16.546 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:16.546 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.772704Z" 2023-07-29T01:19:16.546 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:16.546 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:16.547 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:16.547 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:19:16.547 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:16.547 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:19:16.547 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:19:16.547 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:16.548 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:19:16.548 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:16.548 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:19:16.548 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:16.548 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:16.548 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:16.549 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:19:16.549 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:16.549 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:19:16.549 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000046080", 2023-07-29T01:19:16.549 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:16.549 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:19:16.549 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:16.550 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.709533Z" 2023-07-29T01:19:16.550 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:16.550 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:16.550 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:16.550 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:19:16.550 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:16.550 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:19:16.551 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031774", 2023-07-29T01:19:16.551 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:16.551 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:19:16.552 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:16.552 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.812152Z" 2023-07-29T01:19:16.552 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:16.552 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:16.553 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:19:16.553 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:16.553 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:19:16.553 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:19:16.554 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:19:16.554 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:16.554 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:19:16Z" 2023-07-29T01:19:16.554 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:19:16.556 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052514", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:17:10.772704Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046080", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.709533Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.812152Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:19:16Z" } 2023-07-29T01:19:16.556 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052514', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.772704Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046080', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.709533Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.812152Z'}}] 2023-07-29T01:19:16.557 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052655', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:19:16.558 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:19:16.558 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052655 target=00000000000000000000:00000000000000052514, shard=2 master=00000000000000000000:00000000000000046221 target=00000000000000000000:00000000000000046080 2023-07-29T01:19:21.559 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:19:21.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:19:21.560 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:19:21.615 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:19:21.615 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:19:21.663 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:19:21.664 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:19:21.664 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:19:21.664 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:19:21.664 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:19:21.664 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:19:21.664 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:21.665 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:19:21.665 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:21.665 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:19:21.665 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:21.665 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:19:21.665 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052514", 2023-07-29T01:19:21.665 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:21.666 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:19:21.666 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:21.666 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.772704Z" 2023-07-29T01:19:21.666 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:21.666 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:21.666 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:21.666 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:19:21.667 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:21.667 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:19:21.667 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:19:21.667 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:21.667 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:19:21.667 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:21.667 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:19:21.668 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:21.668 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:21.668 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:21.668 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:19:21.668 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:21.668 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:19:21.669 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000046080", 2023-07-29T01:19:21.669 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:21.669 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:19:21.669 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:21.669 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.709533Z" 2023-07-29T01:19:21.669 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:21.669 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:21.670 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:21.670 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:19:21.670 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:21.670 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:19:21.670 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031774", 2023-07-29T01:19:21.670 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:21.670 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:19:21.671 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:21.671 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.812152Z" 2023-07-29T01:19:21.671 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:21.671 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:21.671 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:19:21.671 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:21.671 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:19:21.672 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:19:21.672 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:19:21.672 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:21.672 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:19:21Z" 2023-07-29T01:19:21.672 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:19:21.673 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052514", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:17:10.772704Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046080", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.709533Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.812152Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:19:21Z" } 2023-07-29T01:19:21.673 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052514', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.772704Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046080', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.709533Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.812152Z'}}] 2023-07-29T01:19:21.674 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052655', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:19:21.674 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:19:21.674 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052655 target=00000000000000000000:00000000000000052514, shard=2 master=00000000000000000000:00000000000000046221 target=00000000000000000000:00000000000000046080 2023-07-29T01:19:26.675 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:19:26.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:19:26.677 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:19:26.732 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:19:26.732 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:19:26.780 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:19:26.780 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:19:26.780 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:19:26.780 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:19:26.780 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:19:26.781 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:19:26.781 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:26.781 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:19:26.781 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:26.781 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:19:26.781 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:26.781 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:19:26.782 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052514", 2023-07-29T01:19:26.782 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:26.782 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:19:26.782 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:26.782 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.772704Z" 2023-07-29T01:19:26.782 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:26.782 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:26.783 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:26.783 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:19:26.783 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:26.783 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:19:26.783 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:19:26.783 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:26.784 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:19:26.784 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:26.784 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:19:26.784 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:26.784 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:26.784 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:26.784 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:19:26.785 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:26.785 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:19:26.785 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000046080", 2023-07-29T01:19:26.785 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:26.785 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:19:26.785 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:26.785 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.709533Z" 2023-07-29T01:19:26.786 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:26.786 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:26.786 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:26.786 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:19:26.786 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:26.786 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:19:26.786 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031774", 2023-07-29T01:19:26.787 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:26.787 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:19:26.787 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:26.787 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.812152Z" 2023-07-29T01:19:26.787 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:26.787 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:26.787 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:19:26.788 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:26.788 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:19:26.788 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:19:26.788 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:19:26.788 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:26.788 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:19:26Z" 2023-07-29T01:19:26.788 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:19:26.789 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052514", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:17:10.772704Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046080", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.709533Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.812152Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:19:26Z" } 2023-07-29T01:19:26.790 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052514', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.772704Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046080', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.709533Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.812152Z'}}] 2023-07-29T01:19:26.790 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052655', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:19:26.790 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:19:26.791 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052655 target=00000000000000000000:00000000000000052514, shard=2 master=00000000000000000000:00000000000000046221 target=00000000000000000000:00000000000000046080 2023-07-29T01:19:31.792 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:19:31.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:19:31.793 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:19:31.848 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:19:31.849 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:19:31.896 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:19:31.897 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:19:31.897 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:19:31.897 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:19:31.897 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:19:31.898 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:19:31.898 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:31.898 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:19:31.898 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:31.899 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:19:31.899 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:31.899 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:19:31.899 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052514", 2023-07-29T01:19:31.900 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:31.900 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:19:31.900 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:31.900 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.772704Z" 2023-07-29T01:19:31.900 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:31.900 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:31.901 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:31.901 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:19:31.901 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:31.901 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:19:31.902 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:19:31.902 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:31.902 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:19:31.902 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:31.902 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:19:31.902 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:31.902 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:31.903 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:31.903 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:19:31.903 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:31.903 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:19:31.903 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000046080", 2023-07-29T01:19:31.903 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:31.903 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:19:31.904 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:31.904 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.709533Z" 2023-07-29T01:19:31.904 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:31.904 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:31.904 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:31.904 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:19:31.905 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:31.905 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:19:31.905 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031774", 2023-07-29T01:19:31.906 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:31.906 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:19:31.906 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:31.907 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.812152Z" 2023-07-29T01:19:31.907 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:31.907 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:31.907 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:19:31.907 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:31.907 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:19:31.908 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:19:31.908 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:19:31.908 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:31.908 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:19:31Z" 2023-07-29T01:19:31.908 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:19:31.909 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052514", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:17:10.772704Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046080", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.709533Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.812152Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:19:31Z" } 2023-07-29T01:19:31.910 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052514', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.772704Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046080', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.709533Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.812152Z'}}] 2023-07-29T01:19:31.910 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052655', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:19:31.911 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:19:31.911 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052655 target=00000000000000000000:00000000000000052514, shard=2 master=00000000000000000000:00000000000000046221 target=00000000000000000000:00000000000000046080 2023-07-29T01:19:36.912 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:19:36.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:19:36.913 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:19:36.968 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:19:36.968 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:19:37.016 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:19:37.016 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:19:37.016 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:19:37.016 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:19:37.017 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:19:37.017 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:19:37.017 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:37.017 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:19:37.018 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:37.018 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:19:37.018 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:37.018 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:19:37.018 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052514", 2023-07-29T01:19:37.018 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:37.019 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:19:37.019 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:37.019 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.772704Z" 2023-07-29T01:19:37.019 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:37.019 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:37.019 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:37.019 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:19:37.020 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:37.020 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:19:37.020 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:19:37.020 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:37.020 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:19:37.020 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:37.021 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:19:37.021 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:37.021 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:37.021 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:37.021 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:19:37.021 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:37.021 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:19:37.022 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000046080", 2023-07-29T01:19:37.022 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:37.022 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:19:37.022 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:37.022 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.709533Z" 2023-07-29T01:19:37.022 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:37.022 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:37.023 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:37.023 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:19:37.023 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:37.023 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:19:37.023 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031774", 2023-07-29T01:19:37.023 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:37.024 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:19:37.024 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:37.024 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.812152Z" 2023-07-29T01:19:37.024 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:37.024 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:37.024 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:19:37.024 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:37.025 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:19:37.025 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:19:37.025 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:19:37.025 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:37.025 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:19:36Z" 2023-07-29T01:19:37.025 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:19:37.026 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052514", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:17:10.772704Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046080", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.709533Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.812152Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:19:36Z" } 2023-07-29T01:19:37.027 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052514', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.772704Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046080', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.709533Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.812152Z'}}] 2023-07-29T01:19:37.027 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052655', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:19:37.027 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:19:37.028 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052655 target=00000000000000000000:00000000000000052514, shard=2 master=00000000000000000000:00000000000000046221 target=00000000000000000000:00000000000000046080 2023-07-29T01:19:42.029 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:19:42.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:19:42.030 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:19:42.086 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:19:42.086 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:19:42.134 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:19:42.134 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:19:42.135 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:19:42.135 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:19:42.135 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:19:42.135 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:19:42.135 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:42.135 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:19:42.136 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:42.136 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:19:42.136 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:42.136 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:19:42.136 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052514", 2023-07-29T01:19:42.137 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:42.137 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:19:42.137 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:42.137 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.772704Z" 2023-07-29T01:19:42.137 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:42.137 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:42.137 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:42.138 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:19:42.138 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:42.138 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:19:42.138 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:19:42.138 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:42.138 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:19:42.139 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:42.139 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:19:42.139 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:42.139 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:42.139 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:42.139 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:19:42.139 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:42.140 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:19:42.140 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000046080", 2023-07-29T01:19:42.140 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:42.140 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:19:42.140 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:42.140 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.709533Z" 2023-07-29T01:19:42.141 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:42.141 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:42.141 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:42.141 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:19:42.141 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:42.141 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:19:42.141 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031774", 2023-07-29T01:19:42.142 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:42.142 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:19:42.142 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:42.142 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.812152Z" 2023-07-29T01:19:42.142 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:42.143 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:42.143 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:19:42.143 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:42.143 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:19:42.143 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:19:42.143 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:19:42.144 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:42.144 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:19:42Z" 2023-07-29T01:19:42.144 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:19:42.145 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052514", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:17:10.772704Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046080", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.709533Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.812152Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:19:42Z" } 2023-07-29T01:19:42.145 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052514', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.772704Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046080', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.709533Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.812152Z'}}] 2023-07-29T01:19:42.146 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052655', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:19:42.146 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:19:42.147 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052655 target=00000000000000000000:00000000000000052514, shard=2 master=00000000000000000000:00000000000000046221 target=00000000000000000000:00000000000000046080 2023-07-29T01:19:47.147 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:19:47.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:19:47.149 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:19:47.203 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:19:47.204 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:19:47.253 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:19:47.253 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:19:47.254 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:19:47.254 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:19:47.254 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:19:47.254 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:19:47.254 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:47.254 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:19:47.255 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:47.255 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:19:47.255 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:47.255 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:19:47.256 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052514", 2023-07-29T01:19:47.256 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:47.256 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:19:47.257 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:47.257 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.772704Z" 2023-07-29T01:19:47.257 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:47.257 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:47.257 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:47.258 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:19:47.258 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:47.258 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:19:47.258 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:19:47.258 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:47.258 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:19:47.258 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:47.259 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:19:47.259 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:47.259 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:47.259 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:47.260 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:19:47.260 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:47.260 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:19:47.260 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000046080", 2023-07-29T01:19:47.260 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:47.260 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:19:47.261 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:47.261 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.709533Z" 2023-07-29T01:19:47.261 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:47.261 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:47.261 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:47.261 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:19:47.262 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:47.262 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:19:47.262 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031774", 2023-07-29T01:19:47.262 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:47.262 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:19:47.262 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:47.262 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.812152Z" 2023-07-29T01:19:47.263 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:47.263 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:47.263 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:19:47.263 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:47.263 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:19:47.263 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:19:47.263 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:19:47.264 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:47.264 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:19:47Z" 2023-07-29T01:19:47.264 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:19:47.265 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052514", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:17:10.772704Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046080", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.709533Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.812152Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:19:47Z" } 2023-07-29T01:19:47.265 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052514', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.772704Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046080', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.709533Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.812152Z'}}] 2023-07-29T01:19:47.266 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052655', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:19:47.266 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:19:47.266 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052655 target=00000000000000000000:00000000000000052514, shard=2 master=00000000000000000000:00000000000000046221 target=00000000000000000000:00000000000000046080 2023-07-29T01:19:52.267 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:19:52.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:19:52.268 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:19:52.323 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:19:52.323 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:19:52.370 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:19:52.371 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:19:52.371 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:19:52.371 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:19:52.371 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:19:52.371 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:19:52.372 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:52.372 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:19:52.372 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:52.372 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:19:52.373 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:52.373 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:19:52.373 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052514", 2023-07-29T01:19:52.373 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:52.373 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:19:52.373 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:52.374 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.772704Z" 2023-07-29T01:19:52.374 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:52.374 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:52.374 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:52.375 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:19:52.375 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:52.375 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:19:52.375 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:19:52.375 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:52.375 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:19:52.376 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:52.376 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:19:52.376 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:52.376 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:52.377 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:52.377 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:19:52.377 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:52.378 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:19:52.378 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000046080", 2023-07-29T01:19:52.378 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:52.379 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:19:52.379 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:52.379 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.709533Z" 2023-07-29T01:19:52.379 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:52.380 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:52.380 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:52.380 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:19:52.381 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:52.381 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:19:52.381 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031774", 2023-07-29T01:19:52.382 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:52.382 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:19:52.382 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:52.382 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.812152Z" 2023-07-29T01:19:52.383 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:52.383 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:52.383 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:19:52.383 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:52.383 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:19:52.383 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:19:52.384 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:19:52.384 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:52.384 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:19:52Z" 2023-07-29T01:19:52.384 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:19:52.385 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052514", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:17:10.772704Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046080", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.709533Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.812152Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:19:52Z" } 2023-07-29T01:19:52.385 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052514', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.772704Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046080', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.709533Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.812152Z'}}] 2023-07-29T01:19:52.386 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052655', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:19:52.386 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:19:52.386 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052655 target=00000000000000000000:00000000000000052514, shard=2 master=00000000000000000000:00000000000000046221 target=00000000000000000000:00000000000000046080 2023-07-29T01:19:57.388 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:19:57.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:19:57.389 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:19:57.444 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:19:57.444 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:19:57.499 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:19:57.499 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:19:57.499 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:19:57.499 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:19:57.500 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:19:57.500 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:19:57.500 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:57.500 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:19:57.500 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:57.500 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:19:57.501 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:57.501 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:19:57.501 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052514", 2023-07-29T01:19:57.501 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:57.501 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:19:57.501 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:57.501 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.772704Z" 2023-07-29T01:19:57.502 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:57.502 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:57.502 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:57.502 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:19:57.502 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:57.502 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:19:57.503 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:19:57.503 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:57.503 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:19:57.503 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:57.503 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:19:57.503 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:57.503 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:57.504 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:57.504 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:19:57.504 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:57.504 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:19:57.504 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000046080", 2023-07-29T01:19:57.504 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:57.504 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:19:57.505 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:57.505 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.709533Z" 2023-07-29T01:19:57.505 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:57.505 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:57.505 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:19:57.505 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:19:57.505 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:19:57.506 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:19:57.506 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031774", 2023-07-29T01:19:57.506 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:19:57.506 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:19:57.506 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:19:57.506 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.812152Z" 2023-07-29T01:19:57.506 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:57.507 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:19:57.507 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:19:57.507 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:57.507 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:19:57.507 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:19:57.507 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:19:57.507 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:19:57.508 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:19:57Z" 2023-07-29T01:19:57.508 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:19:57.509 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052514", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:17:10.772704Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046080", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.709533Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.812152Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:19:57Z" } 2023-07-29T01:19:57.509 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052514', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.772704Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046080', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.709533Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.812152Z'}}] 2023-07-29T01:19:57.510 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052655', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:19:57.510 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:19:57.510 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052655 target=00000000000000000000:00000000000000052514, shard=2 master=00000000000000000000:00000000000000046221 target=00000000000000000000:00000000000000046080 2023-07-29T01:20:02.512 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:20:02.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:20:02.513 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:20:02.568 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:20:02.569 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:20:02.616 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:20:02.616 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:20:02.617 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:20:02.617 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:20:02.617 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:20:02.617 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:20:02.617 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:02.617 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:20:02.617 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:02.618 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:20:02.618 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:02.618 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:20:02.618 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052514", 2023-07-29T01:20:02.618 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:02.618 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:20:02.618 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:02.619 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.772704Z" 2023-07-29T01:20:02.619 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:02.619 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:02.619 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:02.619 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:20:02.619 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:02.620 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:20:02.620 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:20:02.620 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:02.620 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:20:02.620 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:02.620 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:20:02.620 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:02.621 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:02.621 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:02.621 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:20:02.621 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:02.621 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:20:02.621 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000046080", 2023-07-29T01:20:02.621 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:02.622 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:20:02.622 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:02.622 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.709533Z" 2023-07-29T01:20:02.622 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:02.622 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:02.622 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:02.622 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:20:02.623 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:02.623 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:20:02.623 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031774", 2023-07-29T01:20:02.623 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:02.623 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:20:02.623 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:02.624 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.812152Z" 2023-07-29T01:20:02.624 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:02.624 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:02.624 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:20:02.624 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:02.624 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:20:02.624 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:20:02.625 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:20:02.625 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:02.625 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:20:02Z" 2023-07-29T01:20:02.625 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:20:02.626 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052514", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:17:10.772704Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046080", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.709533Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.812152Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:20:02Z" } 2023-07-29T01:20:02.627 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052514', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.772704Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046080', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.709533Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.812152Z'}}] 2023-07-29T01:20:02.627 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052655', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:20:02.627 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:20:02.628 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052655 target=00000000000000000000:00000000000000052514, shard=2 master=00000000000000000000:00000000000000046221 target=00000000000000000000:00000000000000046080 2023-07-29T01:20:07.629 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:20:07.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:20:07.631 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:20:07.686 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:20:07.687 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:20:07.735 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:20:07.735 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:20:07.735 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:20:07.735 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:20:07.735 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:20:07.736 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:20:07.736 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:07.736 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:20:07.736 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:07.736 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:20:07.736 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:07.737 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:20:07.737 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052514", 2023-07-29T01:20:07.737 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:07.737 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:20:07.737 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:07.737 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.772704Z" 2023-07-29T01:20:07.737 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:07.738 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:07.738 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:07.738 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:20:07.738 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:07.738 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:20:07.738 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:20:07.738 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:07.739 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:20:07.739 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:07.739 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:20:07.739 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:07.739 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:07.739 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:07.739 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:20:07.740 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:07.740 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:20:07.740 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000046080", 2023-07-29T01:20:07.740 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:07.740 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:20:07.740 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:07.740 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.709533Z" 2023-07-29T01:20:07.741 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:07.741 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:07.741 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:07.741 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:20:07.741 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:07.741 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:20:07.741 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031774", 2023-07-29T01:20:07.742 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:07.742 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:20:07.742 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:07.742 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.812152Z" 2023-07-29T01:20:07.742 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:07.742 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:07.743 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:20:07.743 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:07.743 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:20:07.743 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:20:07.743 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:20:07.743 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:07.744 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:20:07Z" 2023-07-29T01:20:07.744 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:20:07.745 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052514", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:17:10.772704Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046080", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.709533Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.812152Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:20:07Z" } 2023-07-29T01:20:07.745 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052514', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.772704Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046080', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.709533Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.812152Z'}}] 2023-07-29T01:20:07.745 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052655', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:20:07.746 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:20:07.746 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052655 target=00000000000000000000:00000000000000052514, shard=2 master=00000000000000000000:00000000000000046221 target=00000000000000000000:00000000000000046080 2023-07-29T01:20:12.747 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:20:12.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:20:12.748 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:20:12.803 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:20:12.803 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:20:12.851 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:20:12.851 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:20:12.851 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:20:12.852 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:20:12.852 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:20:12.852 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:20:12.852 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:12.852 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:20:12.852 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:12.853 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:20:12.853 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:12.853 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:20:12.853 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052514", 2023-07-29T01:20:12.853 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:12.853 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:20:12.853 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:12.854 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.772704Z" 2023-07-29T01:20:12.854 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:12.854 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:12.854 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:12.854 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:20:12.854 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:12.854 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:20:12.855 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:20:12.855 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:12.855 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:20:12.855 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:12.855 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:20:12.855 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:12.855 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:12.856 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:12.856 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:20:12.856 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:12.856 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:20:12.856 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000046080", 2023-07-29T01:20:12.856 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:12.856 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:20:12.857 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:12.857 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.709533Z" 2023-07-29T01:20:12.857 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:12.857 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:12.857 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:12.857 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:20:12.857 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:12.858 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:20:12.858 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031774", 2023-07-29T01:20:12.858 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:12.858 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:20:12.858 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:12.858 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.812152Z" 2023-07-29T01:20:12.858 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:12.859 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:12.859 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:20:12.859 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:12.859 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:20:12.859 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:20:12.859 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:20:12.859 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:12.860 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:20:12Z" 2023-07-29T01:20:12.860 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:20:12.861 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052514", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:17:10.772704Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046080", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.709533Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.812152Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:20:12Z" } 2023-07-29T01:20:12.861 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052514', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.772704Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046080', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.709533Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.812152Z'}}] 2023-07-29T01:20:12.862 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052655', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:20:12.862 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:20:12.862 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052655 target=00000000000000000000:00000000000000052514, shard=2 master=00000000000000000000:00000000000000046221 target=00000000000000000000:00000000000000046080 2023-07-29T01:20:17.863 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:20:17.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:20:17.864 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:20:17.919 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:20:17.919 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:20:17.968 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:20:17.968 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:20:17.968 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:20:17.968 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:20:17.968 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:20:17.969 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:20:17.969 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:17.969 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:20:17.969 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:17.969 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:20:17.969 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:17.971 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:20:17.972 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052514", 2023-07-29T01:20:17.972 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:17.972 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:20:17.972 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:17.972 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.772704Z" 2023-07-29T01:20:17.972 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:17.972 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:17.973 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:17.973 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:20:17.973 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:17.973 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:20:17.973 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:20:17.973 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:17.973 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:20:17.974 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:17.974 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:20:17.974 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:17.974 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:17.974 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:17.974 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:20:17.974 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:17.975 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:20:17.975 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000046080", 2023-07-29T01:20:17.975 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:17.975 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:20:17.975 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:17.975 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.709533Z" 2023-07-29T01:20:17.975 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:17.976 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:17.976 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:17.976 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:20:17.976 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:17.976 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:20:17.976 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031774", 2023-07-29T01:20:17.976 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:17.977 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:20:17.977 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:17.977 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.812152Z" 2023-07-29T01:20:17.977 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:17.977 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:17.977 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:20:17.977 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:17.978 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:20:17.978 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:20:17.978 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:20:17.978 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:17.978 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:20:17Z" 2023-07-29T01:20:17.978 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:20:17.979 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052514", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:17:10.772704Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046080", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.709533Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.812152Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:20:17Z" } 2023-07-29T01:20:17.980 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052514', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.772704Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046080', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.709533Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.812152Z'}}] 2023-07-29T01:20:17.980 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052655', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:20:17.980 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:20:17.981 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052655 target=00000000000000000000:00000000000000052514, shard=2 master=00000000000000000000:00000000000000046221 target=00000000000000000000:00000000000000046080 2023-07-29T01:20:22.982 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:20:22.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:20:22.983 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:20:23.037 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:20:23.038 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:20:23.086 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:20:23.086 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:20:23.086 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:20:23.087 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:20:23.087 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:20:23.087 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:20:23.087 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:23.087 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:20:23.087 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:23.087 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:20:23.088 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:23.088 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:20:23.088 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052514", 2023-07-29T01:20:23.088 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:23.088 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:20:23.088 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:23.088 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.772704Z" 2023-07-29T01:20:23.089 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:23.089 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:23.089 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:23.089 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:20:23.089 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:23.089 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:20:23.090 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:20:23.090 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:23.090 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:20:23.090 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:23.090 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:20:23.090 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:23.090 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:23.091 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:23.091 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:20:23.091 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:23.091 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:20:23.091 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000046080", 2023-07-29T01:20:23.091 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:23.091 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:20:23.092 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:23.092 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.709533Z" 2023-07-29T01:20:23.092 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:23.092 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:23.092 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:23.092 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:20:23.093 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:23.093 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:20:23.093 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031774", 2023-07-29T01:20:23.093 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:23.093 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:20:23.093 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:23.093 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.812152Z" 2023-07-29T01:20:23.094 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:23.094 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:23.094 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:20:23.094 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:23.094 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:20:23.094 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:20:23.094 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:20:23.095 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:23.095 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:20:22Z" 2023-07-29T01:20:23.095 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:20:23.096 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052514", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:17:10.772704Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046080", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.709533Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.812152Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:20:22Z" } 2023-07-29T01:20:23.096 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052514', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.772704Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046080', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.709533Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.812152Z'}}] 2023-07-29T01:20:23.097 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052655', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:20:23.097 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:20:23.097 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052655 target=00000000000000000000:00000000000000052514, shard=2 master=00000000000000000000:00000000000000046221 target=00000000000000000000:00000000000000046080 2023-07-29T01:20:28.098 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:20:28.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:20:28.100 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:20:28.154 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:20:28.155 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:20:28.203 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:20:28.203 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:20:28.204 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:20:28.204 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:20:28.204 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:20:28.204 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:20:28.204 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:28.204 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:20:28.205 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:28.205 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:20:28.205 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:28.205 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:20:28.205 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052514", 2023-07-29T01:20:28.205 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:28.205 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:20:28.206 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:28.206 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.772704Z" 2023-07-29T01:20:28.206 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:28.206 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:28.206 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:28.206 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:20:28.206 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:28.207 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:20:28.207 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:20:28.207 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:28.207 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:20:28.207 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:28.207 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:20:28.207 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:28.208 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:28.208 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:28.208 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:20:28.208 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:28.208 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:20:28.208 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000046080", 2023-07-29T01:20:28.209 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:28.209 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:20:28.209 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:28.209 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.709533Z" 2023-07-29T01:20:28.209 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:28.209 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:28.209 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:28.210 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:20:28.210 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:28.210 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:20:28.210 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031774", 2023-07-29T01:20:28.210 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:28.210 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:20:28.210 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:28.211 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.812152Z" 2023-07-29T01:20:28.211 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:28.211 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:28.211 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:20:28.211 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:28.211 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:20:28.211 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:20:28.212 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:20:28.212 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:28.212 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:20:28Z" 2023-07-29T01:20:28.212 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:20:28.213 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052514", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:17:10.772704Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046080", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.709533Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.812152Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:20:28Z" } 2023-07-29T01:20:28.214 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052514', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.772704Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046080', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.709533Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.812152Z'}}] 2023-07-29T01:20:28.214 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052655', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:20:28.214 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:20:28.215 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052655 target=00000000000000000000:00000000000000052514, shard=2 master=00000000000000000000:00000000000000046221 target=00000000000000000000:00000000000000046080 2023-07-29T01:20:33.215 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:20:33.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:20:33.216 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:20:33.271 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:20:33.271 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:20:33.319 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:20:33.319 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:20:33.319 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:20:33.320 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:20:33.320 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:20:33.320 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:20:33.320 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:33.320 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:20:33.321 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:33.321 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:20:33.321 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:33.321 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:20:33.322 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052514", 2023-07-29T01:20:33.322 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:33.322 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:20:33.322 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:33.322 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.772704Z" 2023-07-29T01:20:33.322 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:33.323 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:33.323 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:33.323 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:20:33.323 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:33.323 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:20:33.323 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:20:33.323 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:33.324 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:20:33.324 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:33.324 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:20:33.324 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:33.324 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:33.324 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:33.324 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:20:33.325 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:33.325 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:20:33.325 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000046080", 2023-07-29T01:20:33.325 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:33.325 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:20:33.325 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:33.325 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.709533Z" 2023-07-29T01:20:33.326 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:33.326 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:33.326 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:33.326 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:20:33.326 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:33.326 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:20:33.326 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031774", 2023-07-29T01:20:33.327 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:33.327 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:20:33.327 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:33.327 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.812152Z" 2023-07-29T01:20:33.327 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:33.327 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:33.327 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:20:33.328 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:33.328 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:20:33.328 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:20:33.328 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:20:33.328 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:33.328 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:20:33Z" 2023-07-29T01:20:33.328 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:20:33.329 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052514", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:17:10.772704Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046080", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.709533Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.812152Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:20:33Z" } 2023-07-29T01:20:33.330 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052514', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.772704Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046080', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.709533Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.812152Z'}}] 2023-07-29T01:20:33.330 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052655', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:20:33.331 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:20:33.331 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052655 target=00000000000000000000:00000000000000052514, shard=2 master=00000000000000000000:00000000000000046221 target=00000000000000000000:00000000000000046080 2023-07-29T01:20:38.332 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:20:38.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:20:38.334 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:20:38.389 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:20:38.390 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:20:38.437 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:20:38.437 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:20:38.438 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:20:38.438 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:20:38.438 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:20:38.438 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:20:38.438 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:38.438 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:20:38.439 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:38.439 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:20:38.439 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:38.439 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:20:38.439 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052514", 2023-07-29T01:20:38.439 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:38.439 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:20:38.440 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:38.440 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.772704Z" 2023-07-29T01:20:38.440 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:38.440 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:38.440 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:38.440 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:20:38.441 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:38.441 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:20:38.441 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:20:38.441 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:38.441 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:20:38.441 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:38.441 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:20:38.442 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:38.442 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:38.442 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:38.442 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:20:38.442 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:38.442 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:20:38.442 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000046080", 2023-07-29T01:20:38.443 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:38.443 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:20:38.443 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:38.443 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.709533Z" 2023-07-29T01:20:38.443 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:38.443 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:38.443 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:38.444 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:20:38.444 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:38.444 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:20:38.444 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031774", 2023-07-29T01:20:38.444 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:38.444 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:20:38.445 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:38.445 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.812152Z" 2023-07-29T01:20:38.445 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:38.445 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:38.445 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:20:38.445 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:38.445 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:20:38.446 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:20:38.446 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:20:38.446 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:38.446 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:20:38Z" 2023-07-29T01:20:38.446 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:20:38.447 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052514", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:17:10.772704Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046080", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.709533Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.812152Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:20:38Z" } 2023-07-29T01:20:38.447 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052514', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.772704Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046080', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.709533Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.812152Z'}}] 2023-07-29T01:20:38.448 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052655', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:20:38.448 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:20:38.448 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052655 target=00000000000000000000:00000000000000052514, shard=2 master=00000000000000000000:00000000000000046221 target=00000000000000000000:00000000000000046080 2023-07-29T01:20:43.450 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:20:43.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:20:43.451 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:20:43.509 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:20:43.510 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:20:43.558 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:20:43.559 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:20:43.559 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:20:43.559 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:20:43.559 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:20:43.559 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:20:43.559 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:43.560 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:20:43.560 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:43.560 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:20:43.560 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:43.561 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:20:43.561 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052514", 2023-07-29T01:20:43.561 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:43.561 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:20:43.561 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:43.562 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.772704Z" 2023-07-29T01:20:43.562 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:43.562 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:43.562 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:43.562 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:20:43.562 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:43.562 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:20:43.563 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:20:43.563 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:43.563 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:20:43.563 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:43.563 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:20:43.563 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:43.563 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:43.564 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:43.564 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:20:43.564 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:43.564 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:20:43.564 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000046080", 2023-07-29T01:20:43.564 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:43.564 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:20:43.565 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:43.565 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.709533Z" 2023-07-29T01:20:43.565 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:43.565 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:43.565 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:43.565 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:20:43.566 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:43.566 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:20:43.566 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031774", 2023-07-29T01:20:43.566 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:43.566 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:20:43.566 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:43.566 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.812152Z" 2023-07-29T01:20:43.567 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:43.567 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:43.567 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:20:43.567 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:43.567 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:20:43.567 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:20:43.567 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:20:43.568 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:43.568 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:20:43Z" 2023-07-29T01:20:43.568 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:20:43.569 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052514", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:17:10.772704Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046080", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.709533Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.812152Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:20:43Z" } 2023-07-29T01:20:43.569 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052514', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.772704Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046080', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.709533Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.812152Z'}}] 2023-07-29T01:20:43.570 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052655', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:20:43.570 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:20:43.571 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052655 target=00000000000000000000:00000000000000052514, shard=2 master=00000000000000000000:00000000000000046221 target=00000000000000000000:00000000000000046080 2023-07-29T01:20:48.571 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:20:48.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:20:48.572 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:20:48.628 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:20:48.629 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:20:48.677 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:20:48.677 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:20:48.678 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:20:48.678 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:20:48.678 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:20:48.678 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:20:48.678 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:48.678 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:20:48.679 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:48.679 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:20:48.679 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:48.679 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:20:48.679 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052655", 2023-07-29T01:20:48.679 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:48.679 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:20:48.680 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:48.680 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:19:15.824401Z" 2023-07-29T01:20:48.680 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:48.680 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:48.680 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:48.680 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:20:48.680 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:48.681 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:20:48.681 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:20:48.681 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:48.681 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:20:48.681 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:48.681 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:20:48.682 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:48.682 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:48.682 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:48.682 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:20:48.682 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:48.682 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:20:48.682 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000046221", 2023-07-29T01:20:48.683 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:48.683 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:20:48.683 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:48.683 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:19:15.858505Z" 2023-07-29T01:20:48.683 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:48.683 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:48.683 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:48.684 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:20:48.684 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:48.684 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:20:48.684 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031774", 2023-07-29T01:20:48.684 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:48.684 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:20:48.684 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:48.685 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.812152Z" 2023-07-29T01:20:48.685 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:48.685 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:48.685 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:20:48.685 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:48.685 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:20:48.685 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:20:48.686 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:20:48.686 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:48.686 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:20:48Z" 2023-07-29T01:20:48.686 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:20:48.687 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052655", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:19:15.824401Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046221", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:19:15.858505Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.812152Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:20:48Z" } 2023-07-29T01:20:48.687 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052655', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:19:15.824401Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046221', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:19:15.858505Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.812152Z'}}] 2023-07-29T01:20:48.688 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052655', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:20:48.688 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052655', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:20:48.688 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-07-29T01:20:48.689 DEBUG:boto:path=/hiiqos-74/ 2023-07-29T01:20:48.689 DEBUG:boto:auth_path=/hiiqos-74/ 2023-07-29T01:20:48.690 DEBUG:boto:Method: HEAD 2023-07-29T01:20:48.690 DEBUG:boto:Path: /hiiqos-74/ 2023-07-29T01:20:48.690 DEBUG:boto:Data: 2023-07-29T01:20:48.691 DEBUG:boto:Headers: {} 2023-07-29T01:20:48.691 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:20:48.691 DEBUG:boto:Port: 8001 2023-07-29T01:20:48.692 DEBUG:boto:Params: {} 2023-07-29T01:20:48.692 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-29T01:20:48.693 DEBUG:boto:Token: None 2023-07-29T01:20:48.693 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:20:48 GMT /hiiqos-74/ 2023-07-29T01:20:48.694 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bRWuC3DGmU78dMvDkjFmgd9yPjg= 2023-07-29T01:20:48.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:20:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bRWuC3DGmU78dMvDkjFmgd9yPjg=', 'Content-Length': '0'} 2023-07-29T01:20:48.699 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007ec2d8a1d92f4f18-0064c46970-2bc4-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:20:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:20:48.699 DEBUG:boto:path=/hiiqos-74/obj1 2023-07-29T01:20:48.700 DEBUG:boto:auth_path=/hiiqos-74/obj1 2023-07-29T01:20:48.700 DEBUG:boto:Method: HEAD 2023-07-29T01:20:48.700 DEBUG:boto:Path: /hiiqos-74/obj1 2023-07-29T01:20:48.701 DEBUG:boto:Data: 2023-07-29T01:20:48.701 DEBUG:boto:Headers: {} 2023-07-29T01:20:48.701 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:20:48.702 DEBUG:boto:Port: 8001 2023-07-29T01:20:48.702 DEBUG:boto:Params: {} 2023-07-29T01:20:48.702 DEBUG:boto:Token: None 2023-07-29T01:20:48.703 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:20:48 GMT /hiiqos-74/obj1 2023-07-29T01:20:48.703 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:iaZlYg+0lUpLfuqnlxM0R+VBi78= 2023-07-29T01:20:48.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:20:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:iaZlYg+0lUpLfuqnlxM0R+VBi78=', 'Content-Length': '0'} 2023-07-29T01:20:48.705 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:19:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-74:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.10875.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030abec068b7a4299-0064c46970-2bc4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:20:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:20:48.706 DEBUG:boto:path=/hiiqos-74/obj1 2023-07-29T01:20:48.706 DEBUG:boto:auth_path=/hiiqos-74/obj1 2023-07-29T01:20:48.707 DEBUG:boto:Method: GET 2023-07-29T01:20:48.707 DEBUG:boto:Path: /hiiqos-74/obj1 2023-07-29T01:20:48.707 DEBUG:boto:Data: 2023-07-29T01:20:48.708 DEBUG:boto:Headers: {} 2023-07-29T01:20:48.708 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:20:48.708 DEBUG:boto:Port: 8001 2023-07-29T01:20:48.709 DEBUG:boto:Params: {} 2023-07-29T01:20:48.709 DEBUG:boto:Token: None 2023-07-29T01:20:48.709 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 01:20:48 GMT /hiiqos-74/obj1 2023-07-29T01:20:48.710 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:RJCKr82p40xboN9qwoxMiZLjdM8= 2023-07-29T01:20:48.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:20:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RJCKr82p40xboN9qwoxMiZLjdM8=', 'Content-Length': '0'} 2023-07-29T01:20:48.712 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:19:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-74:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.10875.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001d882f8739b0739c-0064c46970-2bc4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:20:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:20:48.713 DEBUG:boto:path=/hiiqos-74/obj2 2023-07-29T01:20:48.713 DEBUG:boto:auth_path=/hiiqos-74/obj2 2023-07-29T01:20:48.713 DEBUG:boto:Method: HEAD 2023-07-29T01:20:48.714 DEBUG:boto:Path: /hiiqos-74/obj2 2023-07-29T01:20:48.714 DEBUG:boto:Data: 2023-07-29T01:20:48.714 DEBUG:boto:Headers: {} 2023-07-29T01:20:48.715 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:20:48.715 DEBUG:boto:Port: 8001 2023-07-29T01:20:48.715 DEBUG:boto:Params: {} 2023-07-29T01:20:48.716 DEBUG:boto:Token: None 2023-07-29T01:20:48.716 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:20:48 GMT /hiiqos-74/obj2 2023-07-29T01:20:48.716 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:ao/+EbojQsX/jqWjDxAUs+sM22k= 2023-07-29T01:20:48.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:20:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ao/+EbojQsX/jqWjDxAUs+sM22k=', 'Content-Length': '0'} 2023-07-29T01:20:48.718 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:19:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-74:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.10875.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004712b99d20664de1-0064c46970-2bc4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:20:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:20:48.719 DEBUG:boto:path=/hiiqos-74/obj2 2023-07-29T01:20:48.719 DEBUG:boto:auth_path=/hiiqos-74/obj2 2023-07-29T01:20:48.720 DEBUG:boto:Method: GET 2023-07-29T01:20:48.720 DEBUG:boto:Path: /hiiqos-74/obj2 2023-07-29T01:20:48.721 DEBUG:boto:Data: 2023-07-29T01:20:48.721 DEBUG:boto:Headers: {} 2023-07-29T01:20:48.721 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:20:48.722 DEBUG:boto:Port: 8001 2023-07-29T01:20:48.722 DEBUG:boto:Params: {} 2023-07-29T01:20:48.722 DEBUG:boto:Token: None 2023-07-29T01:20:48.723 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 01:20:48 GMT /hiiqos-74/obj2 2023-07-29T01:20:48.723 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:8Tt9ocVSoWNKuCvP9pgYfVrzUA8= 2023-07-29T01:20:48.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:20:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8Tt9ocVSoWNKuCvP9pgYfVrzUA8=', 'Content-Length': '0'} 2023-07-29T01:20:48.725 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:19:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-74:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.10875.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000df9d5cfc2a158c59-0064c46970-2bc4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:20:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:20:48.726 DEBUG:boto:path=/hiiqos-75/ 2023-07-29T01:20:48.726 DEBUG:boto:auth_path=/hiiqos-75/ 2023-07-29T01:20:48.727 DEBUG:boto:Method: HEAD 2023-07-29T01:20:48.727 DEBUG:boto:Path: /hiiqos-75/ 2023-07-29T01:20:48.727 DEBUG:boto:Data: 2023-07-29T01:20:48.728 DEBUG:boto:Headers: {} 2023-07-29T01:20:48.728 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:20:48.728 DEBUG:boto:Port: 8001 2023-07-29T01:20:48.729 DEBUG:boto:Params: {} 2023-07-29T01:20:48.729 DEBUG:boto:Token: None 2023-07-29T01:20:48.729 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:20:48 GMT /hiiqos-75/ 2023-07-29T01:20:48.730 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:q20UvsBnqaPALTyYoYiNmQAj51k= 2023-07-29T01:20:48.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:20:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:q20UvsBnqaPALTyYoYiNmQAj51k=', 'Content-Length': '0'} 2023-07-29T01:20:48.732 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c02736111c03a980-0064c46970-2bc4-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:20:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:20:48.733 DEBUG:boto:path=/hiiqos-75/obj1 2023-07-29T01:20:48.733 DEBUG:boto:auth_path=/hiiqos-75/obj1 2023-07-29T01:20:48.734 DEBUG:boto:Method: HEAD 2023-07-29T01:20:48.734 DEBUG:boto:Path: /hiiqos-75/obj1 2023-07-29T01:20:48.734 DEBUG:boto:Data: 2023-07-29T01:20:48.735 DEBUG:boto:Headers: {} 2023-07-29T01:20:48.735 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:20:48.735 DEBUG:boto:Port: 8001 2023-07-29T01:20:48.735 DEBUG:boto:Params: {} 2023-07-29T01:20:48.736 DEBUG:boto:Token: None 2023-07-29T01:20:48.736 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:20:48 GMT /hiiqos-75/obj1 2023-07-29T01:20:48.737 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:7h5PUljPTkPUlKvRecovPZorVaw= 2023-07-29T01:20:48.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:20:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7h5PUljPTkPUlKvRecovPZorVaw=', 'Content-Length': '0'} 2023-07-29T01:20:48.739 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:19:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-74:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.10875.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f839f334e8383d12-0064c46970-2bc4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:20:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:20:48.739 DEBUG:boto:path=/hiiqos-75/obj1 2023-07-29T01:20:48.740 DEBUG:boto:auth_path=/hiiqos-75/obj1 2023-07-29T01:20:48.740 DEBUG:boto:Method: GET 2023-07-29T01:20:48.740 DEBUG:boto:Path: /hiiqos-75/obj1 2023-07-29T01:20:48.741 DEBUG:boto:Data: 2023-07-29T01:20:48.741 DEBUG:boto:Headers: {} 2023-07-29T01:20:48.742 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:20:48.742 DEBUG:boto:Port: 8001 2023-07-29T01:20:48.742 DEBUG:boto:Params: {} 2023-07-29T01:20:48.743 DEBUG:boto:Token: None 2023-07-29T01:20:48.743 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 01:20:48 GMT /hiiqos-75/obj1 2023-07-29T01:20:48.743 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:gHrsxog79L2KR6sBuNBx4il+QLE= 2023-07-29T01:20:48.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:20:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gHrsxog79L2KR6sBuNBx4il+QLE=', 'Content-Length': '0'} 2023-07-29T01:20:48.746 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:19:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-74:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.10875.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc525d75d5a5d897-0064c46970-2bc4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:20:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:20:48.788 DEBUG:boto:path=/hiiqos-75/obj2 2023-07-29T01:20:48.789 DEBUG:boto:auth_path=/hiiqos-75/obj2 2023-07-29T01:20:48.789 DEBUG:boto:Method: HEAD 2023-07-29T01:20:48.789 DEBUG:boto:Path: /hiiqos-75/obj2 2023-07-29T01:20:48.790 DEBUG:boto:Data: 2023-07-29T01:20:48.790 DEBUG:boto:Headers: {} 2023-07-29T01:20:48.790 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:20:48.791 DEBUG:boto:Port: 8001 2023-07-29T01:20:48.791 DEBUG:boto:Params: {} 2023-07-29T01:20:48.791 DEBUG:boto:Token: None 2023-07-29T01:20:48.792 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:20:48 GMT /hiiqos-75/obj2 2023-07-29T01:20:48.792 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:f3qse+nohfS3ZTiuAR0euOkxAuw= 2023-07-29T01:20:48.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:20:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:f3qse+nohfS3ZTiuAR0euOkxAuw=', 'Content-Length': '0'} 2023-07-29T01:20:48.794 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:19:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-74:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.10875.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d3e2b7024332980d-0064c46970-2bc4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:20:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:20:48.795 DEBUG:boto:path=/hiiqos-75/obj2 2023-07-29T01:20:48.796 DEBUG:boto:auth_path=/hiiqos-75/obj2 2023-07-29T01:20:48.796 DEBUG:boto:Method: GET 2023-07-29T01:20:48.796 DEBUG:boto:Path: /hiiqos-75/obj2 2023-07-29T01:20:48.797 DEBUG:boto:Data: 2023-07-29T01:20:48.797 DEBUG:boto:Headers: {} 2023-07-29T01:20:48.797 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:20:48.798 DEBUG:boto:Port: 8001 2023-07-29T01:20:48.798 DEBUG:boto:Params: {} 2023-07-29T01:20:48.799 DEBUG:boto:Token: None 2023-07-29T01:20:48.799 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 01:20:48 GMT /hiiqos-75/obj2 2023-07-29T01:20:48.799 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Wsxz0XB7lChi5Nez4amxPIA1TKo= 2023-07-29T01:20:48.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:20:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Wsxz0XB7lChi5Nez4amxPIA1TKo=', 'Content-Length': '0'} 2023-07-29T01:20:48.802 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:19:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-74:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.10875.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf1d7c8b7544c7c2-0064c46970-2bc4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:20:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:20:48.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:20:48.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:20:48.803 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:20:48.851 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:20:48.851 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:20:48.897 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-hiiqos-72","source":{"bucket":"hiiqos-72","zones":["a1"]},"dest":{"bucket":"hiiqos-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hiiqos-73","source":{"bucket":"hiiqos-73","zones":["a1"]},"dest":{"bucket":"hiiqos-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hiiqos-74","source":{"bucket":"hiiqos-74","zones":["a1"]},"dest":{"bucket":"hiiqos-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hiiqos-75","source":{"bucket":"hiiqos-74","zones":["a1"]},"dest":{"bucket":"hiiqos-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-07-29T01:20:48.898 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=*', '--dest-zones=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:20:48.898 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=*', '--dest-zones=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:20:48.899 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe '--source-zones=*' '--dest-zones=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:20:48.950 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:20:48.951 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:20:48.980 INFO:teuthology.orchestra.run.smithi101.stderr:WARNING: cannot find source zone id for name=* 2023-07-29T01:20:48.981 INFO:teuthology.orchestra.run.smithi101.stderr:WARNING: cannot find source zone id for name=* 2023-07-29T01:20:48.997 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-hiiqos-72","source":{"bucket":"hiiqos-72","zones":["a1"]},"dest":{"bucket":"hiiqos-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hiiqos-73","source":{"bucket":"hiiqos-73","zones":["a1"]},"dest":{"bucket":"hiiqos-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hiiqos-74","source":{"bucket":"hiiqos-74","zones":["a1"]},"dest":{"bucket":"hiiqos-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hiiqos-75","source":{"bucket":"hiiqos-74","zones":["a1"]},"dest":{"bucket":"hiiqos-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-07-29T01:20:48.997 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:20:48.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:20:48.998 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:20:49.045 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:20:49.045 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:20:49.091 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-hiiqos-72","source":{"bucket":"hiiqos-72","zones":["a1"]},"dest":{"bucket":"hiiqos-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hiiqos-73","source":{"bucket":"hiiqos-73","zones":["a1"]},"dest":{"bucket":"hiiqos-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hiiqos-74","source":{"bucket":"hiiqos-74","zones":["a1"]},"dest":{"bucket":"hiiqos-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hiiqos-75","source":{"bucket":"hiiqos-74","zones":["a1"]},"dest":{"bucket":"hiiqos-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-07-29T01:20:49.091 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:20:49.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:20:49.093 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:20:49.138 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:20:49.139 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:20:49.189 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:20:49.190 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:20:49.190 INFO:teuthology.orchestra.run.smithi101.stdout: "epoch": 22, 2023-07-29T01:20:49.190 INFO:teuthology.orchestra.run.smithi101.stdout: "predecessor_uuid": "a0987484-ce37-4520-a564-54b7ea8c5251", 2023-07-29T01:20:49.190 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": [], 2023-07-29T01:20:49.191 INFO:teuthology.orchestra.run.smithi101.stdout: "period_map": { 2023-07-29T01:20:49.191 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:20:49.191 INFO:teuthology.orchestra.run.smithi101.stdout: "zonegroups": [ 2023-07-29T01:20:49.191 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:49.191 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "a", 2023-07-29T01:20:49.191 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a", 2023-07-29T01:20:49.192 INFO:teuthology.orchestra.run.smithi101.stdout: "api_name": "", 2023-07-29T01:20:49.192 INFO:teuthology.orchestra.run.smithi101.stdout: "is_master": true, 2023-07-29T01:20:49.192 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:20:49.192 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-29T01:20:49.193 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:20:49.193 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames": [], 2023-07-29T01:20:49.193 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames_s3website": [], 2023-07-29T01:20:49.193 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:20:49.193 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:20:49.194 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:49.194 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:20:49.194 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a1", 2023-07-29T01:20:49.194 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:20:49.195 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-29T01:20:49.195 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:20:49.195 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T01:20:49.196 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T01:20:49.196 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T01:20:49.197 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T01:20:49.197 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T01:20:49.197 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T01:20:49.198 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T01:20:49.198 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T01:20:49.198 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T01:20:49.198 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T01:20:49.199 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:20:49.199 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:20:49.199 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:49.200 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:49.200 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-29T01:20:49.200 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "a2", 2023-07-29T01:20:49.201 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:20:49.201 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi101.front.sepia.ceph.com:8001" 2023-07-29T01:20:49.201 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:20:49.201 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T01:20:49.202 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T01:20:49.202 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T01:20:49.202 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T01:20:49.202 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T01:20:49.203 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T01:20:49.203 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T01:20:49.203 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T01:20:49.203 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T01:20:49.203 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T01:20:49.203 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:20:49.204 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:20:49.204 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:49.204 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:20:49.205 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_targets": [ 2023-07-29T01:20:49.205 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:49.205 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default-placement", 2023-07-29T01:20:49.205 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [], 2023-07-29T01:20:49.205 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": [ 2023-07-29T01:20:49.205 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD" 2023-07-29T01:20:49.206 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:20:49.206 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:49.206 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:20:49.206 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "default-placement", 2023-07-29T01:20:49.206 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T01:20:49.206 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_policy": { 2023-07-29T01:20:49.207 INFO:teuthology.orchestra.run.smithi101.stdout: "groups": [ 2023-07-29T01:20:49.207 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:49.207 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "sync-group", 2023-07-29T01:20:49.207 INFO:teuthology.orchestra.run.smithi101.stdout: "data_flow": { 2023-07-29T01:20:49.207 INFO:teuthology.orchestra.run.smithi101.stdout: "symmetrical": [ 2023-07-29T01:20:49.207 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:49.208 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "sync-flow1", 2023-07-29T01:20:49.208 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:20:49.208 INFO:teuthology.orchestra.run.smithi101.stdout: "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:20:49.209 INFO:teuthology.orchestra.run.smithi101.stdout: "59d1bb30-3b1c-476f-9111-052c1ea0b711" 2023-07-29T01:20:49.209 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:20:49.209 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:49.209 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:20:49.210 INFO:teuthology.orchestra.run.smithi101.stdout: "directional": [ 2023-07-29T01:20:49.210 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:49.210 INFO:teuthology.orchestra.run.smithi101.stdout: "source_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:20:49.210 INFO:teuthology.orchestra.run.smithi101.stdout: "dest_zone": "59d1bb30-3b1c-476f-9111-052c1ea0b711" 2023-07-29T01:20:49.211 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:49.211 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:20:49.211 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:49.211 INFO:teuthology.orchestra.run.smithi101.stdout: "pipes": [ 2023-07-29T01:20:49.212 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:49.212 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "sync-pipe-hiiqos-72", 2023-07-29T01:20:49.212 INFO:teuthology.orchestra.run.smithi101.stdout: "source": { 2023-07-29T01:20:49.212 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket": "hiiqos-72", 2023-07-29T01:20:49.213 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:20:49.213 INFO:teuthology.orchestra.run.smithi101.stdout: "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433" 2023-07-29T01:20:49.213 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:20:49.213 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:49.214 INFO:teuthology.orchestra.run.smithi101.stdout: "dest": { 2023-07-29T01:20:49.214 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket": "hiiqos-73", 2023-07-29T01:20:49.214 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:20:49.214 INFO:teuthology.orchestra.run.smithi101.stdout: "59d1bb30-3b1c-476f-9111-052c1ea0b711" 2023-07-29T01:20:49.215 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:20:49.215 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:49.215 INFO:teuthology.orchestra.run.smithi101.stdout: "params": { 2023-07-29T01:20:49.215 INFO:teuthology.orchestra.run.smithi101.stdout: "source": { 2023-07-29T01:20:49.216 INFO:teuthology.orchestra.run.smithi101.stdout: "filter": { 2023-07-29T01:20:49.216 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [] 2023-07-29T01:20:49.216 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:49.216 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:49.217 INFO:teuthology.orchestra.run.smithi101.stdout: "dest": {}, 2023-07-29T01:20:49.217 INFO:teuthology.orchestra.run.smithi101.stdout: "priority": 0, 2023-07-29T01:20:49.217 INFO:teuthology.orchestra.run.smithi101.stdout: "mode": "system", 2023-07-29T01:20:49.217 INFO:teuthology.orchestra.run.smithi101.stdout: "user": "" 2023-07-29T01:20:49.218 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:49.218 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:49.218 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:49.218 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "sync-pipe-hiiqos-73", 2023-07-29T01:20:49.219 INFO:teuthology.orchestra.run.smithi101.stdout: "source": { 2023-07-29T01:20:49.219 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket": "hiiqos-73", 2023-07-29T01:20:49.219 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:20:49.219 INFO:teuthology.orchestra.run.smithi101.stdout: "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433" 2023-07-29T01:20:49.220 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:20:49.220 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:49.220 INFO:teuthology.orchestra.run.smithi101.stdout: "dest": { 2023-07-29T01:20:49.220 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket": "hiiqos-73", 2023-07-29T01:20:49.221 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:20:49.221 INFO:teuthology.orchestra.run.smithi101.stdout: "59d1bb30-3b1c-476f-9111-052c1ea0b711" 2023-07-29T01:20:49.221 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:20:49.221 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:49.222 INFO:teuthology.orchestra.run.smithi101.stdout: "params": { 2023-07-29T01:20:49.222 INFO:teuthology.orchestra.run.smithi101.stdout: "source": { 2023-07-29T01:20:49.222 INFO:teuthology.orchestra.run.smithi101.stdout: "filter": { 2023-07-29T01:20:49.222 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [] 2023-07-29T01:20:49.222 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:49.223 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:49.223 INFO:teuthology.orchestra.run.smithi101.stdout: "dest": {}, 2023-07-29T01:20:49.223 INFO:teuthology.orchestra.run.smithi101.stdout: "priority": 0, 2023-07-29T01:20:49.223 INFO:teuthology.orchestra.run.smithi101.stdout: "mode": "system", 2023-07-29T01:20:49.224 INFO:teuthology.orchestra.run.smithi101.stdout: "user": "" 2023-07-29T01:20:49.224 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:49.224 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:49.224 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:49.224 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "sync-pipe-hiiqos-74", 2023-07-29T01:20:49.224 INFO:teuthology.orchestra.run.smithi101.stdout: "source": { 2023-07-29T01:20:49.225 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket": "hiiqos-74", 2023-07-29T01:20:49.225 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:20:49.225 INFO:teuthology.orchestra.run.smithi101.stdout: "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433" 2023-07-29T01:20:49.225 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:20:49.225 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:49.225 INFO:teuthology.orchestra.run.smithi101.stdout: "dest": { 2023-07-29T01:20:49.225 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket": "hiiqos-74", 2023-07-29T01:20:49.226 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:20:49.226 INFO:teuthology.orchestra.run.smithi101.stdout: "59d1bb30-3b1c-476f-9111-052c1ea0b711" 2023-07-29T01:20:49.226 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:20:49.226 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:49.226 INFO:teuthology.orchestra.run.smithi101.stdout: "params": { 2023-07-29T01:20:49.226 INFO:teuthology.orchestra.run.smithi101.stdout: "source": { 2023-07-29T01:20:49.226 INFO:teuthology.orchestra.run.smithi101.stdout: "filter": { 2023-07-29T01:20:49.227 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [] 2023-07-29T01:20:49.227 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:49.227 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:49.227 INFO:teuthology.orchestra.run.smithi101.stdout: "dest": {}, 2023-07-29T01:20:49.227 INFO:teuthology.orchestra.run.smithi101.stdout: "priority": 0, 2023-07-29T01:20:49.227 INFO:teuthology.orchestra.run.smithi101.stdout: "mode": "system", 2023-07-29T01:20:49.228 INFO:teuthology.orchestra.run.smithi101.stdout: "user": "" 2023-07-29T01:20:49.228 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:49.228 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:49.228 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:49.228 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "sync-pipe-hiiqos-75", 2023-07-29T01:20:49.228 INFO:teuthology.orchestra.run.smithi101.stdout: "source": { 2023-07-29T01:20:49.228 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket": "hiiqos-74", 2023-07-29T01:20:49.229 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:20:49.229 INFO:teuthology.orchestra.run.smithi101.stdout: "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433" 2023-07-29T01:20:49.229 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:20:49.229 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:49.229 INFO:teuthology.orchestra.run.smithi101.stdout: "dest": { 2023-07-29T01:20:49.229 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket": "hiiqos-75", 2023-07-29T01:20:49.230 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:20:49.230 INFO:teuthology.orchestra.run.smithi101.stdout: "59d1bb30-3b1c-476f-9111-052c1ea0b711" 2023-07-29T01:20:49.230 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:20:49.230 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:49.230 INFO:teuthology.orchestra.run.smithi101.stdout: "params": { 2023-07-29T01:20:49.230 INFO:teuthology.orchestra.run.smithi101.stdout: "source": { 2023-07-29T01:20:49.231 INFO:teuthology.orchestra.run.smithi101.stdout: "filter": { 2023-07-29T01:20:49.231 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [] 2023-07-29T01:20:49.231 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:49.231 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:49.231 INFO:teuthology.orchestra.run.smithi101.stdout: "dest": {}, 2023-07-29T01:20:49.231 INFO:teuthology.orchestra.run.smithi101.stdout: "priority": 0, 2023-07-29T01:20:49.231 INFO:teuthology.orchestra.run.smithi101.stdout: "mode": "system", 2023-07-29T01:20:49.232 INFO:teuthology.orchestra.run.smithi101.stdout: "user": "" 2023-07-29T01:20:49.232 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:49.232 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:49.232 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:49.233 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "sync-pipe", 2023-07-29T01:20:49.233 INFO:teuthology.orchestra.run.smithi101.stdout: "source": { 2023-07-29T01:20:49.233 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket": "*", 2023-07-29T01:20:49.233 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:20:49.233 INFO:teuthology.orchestra.run.smithi101.stdout: "*" 2023-07-29T01:20:49.233 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:20:49.234 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:49.234 INFO:teuthology.orchestra.run.smithi101.stdout: "dest": { 2023-07-29T01:20:49.234 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket": "*", 2023-07-29T01:20:49.234 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:20:49.235 INFO:teuthology.orchestra.run.smithi101.stdout: "*" 2023-07-29T01:20:49.235 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:20:49.235 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:49.235 INFO:teuthology.orchestra.run.smithi101.stdout: "params": { 2023-07-29T01:20:49.235 INFO:teuthology.orchestra.run.smithi101.stdout: "source": { 2023-07-29T01:20:49.235 INFO:teuthology.orchestra.run.smithi101.stdout: "filter": { 2023-07-29T01:20:49.236 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [] 2023-07-29T01:20:49.236 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:49.236 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:49.236 INFO:teuthology.orchestra.run.smithi101.stdout: "dest": {}, 2023-07-29T01:20:49.236 INFO:teuthology.orchestra.run.smithi101.stdout: "priority": 0, 2023-07-29T01:20:49.236 INFO:teuthology.orchestra.run.smithi101.stdout: "mode": "system", 2023-07-29T01:20:49.236 INFO:teuthology.orchestra.run.smithi101.stdout: "user": "" 2023-07-29T01:20:49.237 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:49.237 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:49.237 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:20:49.237 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "allowed" 2023-07-29T01:20:49.238 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:49.238 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:20:49.238 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:49.238 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled_features": [ 2023-07-29T01:20:49.239 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:20:49.239 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:20:49.239 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:49.239 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:49.239 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "b", 2023-07-29T01:20:49.239 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b", 2023-07-29T01:20:49.240 INFO:teuthology.orchestra.run.smithi101.stdout: "api_name": "", 2023-07-29T01:20:49.240 INFO:teuthology.orchestra.run.smithi101.stdout: "is_master": false, 2023-07-29T01:20:49.240 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:20:49.240 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-29T01:20:49.240 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:20:49.240 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames": [], 2023-07-29T01:20:49.240 INFO:teuthology.orchestra.run.smithi101.stdout: "hostnames_s3website": [], 2023-07-29T01:20:49.241 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T01:20:49.241 INFO:teuthology.orchestra.run.smithi101.stdout: "zones": [ 2023-07-29T01:20:49.241 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:49.241 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T01:20:49.241 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b1", 2023-07-29T01:20:49.241 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:20:49.241 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-29T01:20:49.242 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:20:49.242 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T01:20:49.242 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T01:20:49.242 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T01:20:49.242 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T01:20:49.242 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T01:20:49.243 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T01:20:49.243 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T01:20:49.243 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T01:20:49.243 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T01:20:49.243 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T01:20:49.243 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:20:49.243 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:20:49.244 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:49.244 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:49.244 INFO:teuthology.orchestra.run.smithi101.stdout: "id": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-29T01:20:49.244 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "b2", 2023-07-29T01:20:49.244 INFO:teuthology.orchestra.run.smithi101.stdout: "endpoints": [ 2023-07-29T01:20:49.245 INFO:teuthology.orchestra.run.smithi101.stdout: "http://smithi139.front.sepia.ceph.com:8001" 2023-07-29T01:20:49.245 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:20:49.245 INFO:teuthology.orchestra.run.smithi101.stdout: "log_meta": false, 2023-07-29T01:20:49.246 INFO:teuthology.orchestra.run.smithi101.stdout: "log_data": true, 2023-07-29T01:20:49.246 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_index_max_shards": 11, 2023-07-29T01:20:49.246 INFO:teuthology.orchestra.run.smithi101.stdout: "read_only": false, 2023-07-29T01:20:49.247 INFO:teuthology.orchestra.run.smithi101.stdout: "tier_type": "", 2023-07-29T01:20:49.247 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from_all": true, 2023-07-29T01:20:49.247 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_from": [], 2023-07-29T01:20:49.247 INFO:teuthology.orchestra.run.smithi101.stdout: "redirect_zone": "", 2023-07-29T01:20:49.247 INFO:teuthology.orchestra.run.smithi101.stdout: "supported_features": [ 2023-07-29T01:20:49.247 INFO:teuthology.orchestra.run.smithi101.stdout: "compress-encrypted", 2023-07-29T01:20:49.248 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:20:49.248 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:20:49.248 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:49.248 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:20:49.248 INFO:teuthology.orchestra.run.smithi101.stdout: "placement_targets": [ 2023-07-29T01:20:49.248 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:49.249 INFO:teuthology.orchestra.run.smithi101.stdout: "name": "default-placement", 2023-07-29T01:20:49.249 INFO:teuthology.orchestra.run.smithi101.stdout: "tags": [], 2023-07-29T01:20:49.249 INFO:teuthology.orchestra.run.smithi101.stdout: "storage_classes": [ 2023-07-29T01:20:49.249 INFO:teuthology.orchestra.run.smithi101.stdout: "STANDARD" 2023-07-29T01:20:49.249 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:20:49.249 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:49.249 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:20:49.250 INFO:teuthology.orchestra.run.smithi101.stdout: "default_placement": "default-placement", 2023-07-29T01:20:49.250 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T01:20:49.250 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_policy": { 2023-07-29T01:20:49.250 INFO:teuthology.orchestra.run.smithi101.stdout: "groups": [] 2023-07-29T01:20:49.250 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:49.250 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled_features": [ 2023-07-29T01:20:49.250 INFO:teuthology.orchestra.run.smithi101.stdout: "resharding" 2023-07-29T01:20:49.251 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:20:49.251 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:49.251 INFO:teuthology.orchestra.run.smithi101.stdout: ], 2023-07-29T01:20:49.251 INFO:teuthology.orchestra.run.smithi101.stdout: "short_zone_ids": [ 2023-07-29T01:20:49.251 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:49.251 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:20:49.251 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 2896287231 2023-07-29T01:20:49.252 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:49.252 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:49.252 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-29T01:20:49.252 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 815606810 2023-07-29T01:20:49.252 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:49.252 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:49.253 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T01:20:49.253 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 2009171363 2023-07-29T01:20:49.253 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:49.253 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:49.253 INFO:teuthology.orchestra.run.smithi101.stdout: "key": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-29T01:20:49.253 INFO:teuthology.orchestra.run.smithi101.stdout: "val": 3105190295 2023-07-29T01:20:49.253 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:49.254 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:20:49.254 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:49.254 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zonegroup": "a", 2023-07-29T01:20:49.254 INFO:teuthology.orchestra.run.smithi101.stdout: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:20:49.254 INFO:teuthology.orchestra.run.smithi101.stdout: "period_config": { 2023-07-29T01:20:49.254 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_quota": { 2023-07-29T01:20:49.255 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-29T01:20:49.255 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-29T01:20:49.255 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-29T01:20:49.255 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-29T01:20:49.255 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-29T01:20:49.256 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:49.256 INFO:teuthology.orchestra.run.smithi101.stdout: "user_quota": { 2023-07-29T01:20:49.256 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false, 2023-07-29T01:20:49.256 INFO:teuthology.orchestra.run.smithi101.stdout: "check_on_raw": false, 2023-07-29T01:20:49.256 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size": -1, 2023-07-29T01:20:49.256 INFO:teuthology.orchestra.run.smithi101.stdout: "max_size_kb": 0, 2023-07-29T01:20:49.257 INFO:teuthology.orchestra.run.smithi101.stdout: "max_objects": -1 2023-07-29T01:20:49.257 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:49.257 INFO:teuthology.orchestra.run.smithi101.stdout: "user_ratelimit": { 2023-07-29T01:20:49.257 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-29T01:20:49.257 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-29T01:20:49.257 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-29T01:20:49.258 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-29T01:20:49.258 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-29T01:20:49.258 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:49.258 INFO:teuthology.orchestra.run.smithi101.stdout: "bucket_ratelimit": { 2023-07-29T01:20:49.258 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-29T01:20:49.258 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-29T01:20:49.258 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-29T01:20:49.259 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-29T01:20:49.259 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-29T01:20:49.259 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:49.259 INFO:teuthology.orchestra.run.smithi101.stdout: "anonymous_ratelimit": { 2023-07-29T01:20:49.259 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_ops": 0, 2023-07-29T01:20:49.259 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_ops": 0, 2023-07-29T01:20:49.259 INFO:teuthology.orchestra.run.smithi101.stdout: "max_read_bytes": 0, 2023-07-29T01:20:49.260 INFO:teuthology.orchestra.run.smithi101.stdout: "max_write_bytes": 0, 2023-07-29T01:20:49.260 INFO:teuthology.orchestra.run.smithi101.stdout: "enabled": false 2023-07-29T01:20:49.260 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:49.260 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:49.260 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T01:20:49.260 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_name": "test-realm", 2023-07-29T01:20:49.261 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:20:49.261 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:20:49.261 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:20:49.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:20:49.262 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:20:49.327 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:20:49.327 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:20:49.458 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-hiiqos-72","source":{"bucket":"hiiqos-72","zones":["a1"]},"dest":{"bucket":"hiiqos-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hiiqos-73","source":{"bucket":"hiiqos-73","zones":["a1"]},"dest":{"bucket":"hiiqos-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hiiqos-74","source":{"bucket":"hiiqos-74","zones":["a1"]},"dest":{"bucket":"hiiqos-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hiiqos-75","source":{"bucket":"hiiqos-74","zones":["a1"]},"dest":{"bucket":"hiiqos-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-07-29T01:20:49.459 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'hiiqos-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:20:49.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'hiiqos-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:20:49.460 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket hiiqos-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:20:49.556 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:20:49.556 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:20:49.646 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-07-29T01:20:49.647 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'hiiqos-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:20:49.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'hiiqos-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:20:49.648 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket hiiqos-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:20:49.715 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:20:49.716 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:20:49.787 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2023-07-29T01:20:49.788 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-hiiqos-74', '--source-zones=a1', '--dest-zones=a2', '--bucket=hiiqos-74', '--source-bucket=*', '--dest-bucket=hiiqos-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:20:49.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-hiiqos-74', '--source-zones=a1', '--dest-zones=a2', '--bucket=hiiqos-74', '--source-bucket=*', '--dest-bucket=hiiqos-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:20:49.789 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-hiiqos-74 --source-zones=a1 --dest-zones=a2 --bucket=hiiqos-74 '--source-bucket=*' --dest-bucket=hiiqos-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:20:49.853 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:20:49.853 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:20:49.906 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-hiiqos-74","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"hiiqos-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-07-29T01:20:49.906 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-hiiqos-75', '--source-zones=a1', '--dest-zones=a2', '--bucket=hiiqos-74', '--source-bucket=*', '--dest-bucket=hiiqos-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:20:49.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-hiiqos-75', '--source-zones=a1', '--dest-zones=a2', '--bucket=hiiqos-74', '--source-bucket=*', '--dest-bucket=hiiqos-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:20:49.907 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-hiiqos-75 --source-zones=a1 --dest-zones=a2 --bucket=hiiqos-74 '--source-bucket=*' --dest-bucket=hiiqos-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:20:49.959 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:20:49.960 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:20:50.014 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-hiiqos-74","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"hiiqos-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-hiiqos-75","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"hiiqos-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-07-29T01:20:50.015 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'hiiqos-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:20:50.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'hiiqos-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:20:50.016 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket hiiqos-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:20:50.063 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:20:50.064 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:20:50.113 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-hiiqos-74","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"hiiqos-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-hiiqos-75","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"hiiqos-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-07-29T01:20:50.114 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:20:50.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:20:50.115 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:20:50.158 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:20:50.158 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:20:50.202 INFO:teuthology.orchestra.run.smithi101.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-hiiqos-72","source":{"bucket":"hiiqos-72","zones":["a1"]},"dest":{"bucket":"hiiqos-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hiiqos-73","source":{"bucket":"hiiqos-73","zones":["a1"]},"dest":{"bucket":"hiiqos-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hiiqos-74","source":{"bucket":"hiiqos-74","zones":["a1"]},"dest":{"bucket":"hiiqos-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hiiqos-75","source":{"bucket":"hiiqos-74","zones":["a1"]},"dest":{"bucket":"hiiqos-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-07-29T01:20:50.203 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:20:50.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:20:50.204 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:20:50.287 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:20:50.288 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:20:50.317 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T01:20:50.330 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T01:20:50.331 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:50.331 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:20:50.331 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:17.836402Z" 2023-07-29T01:20:50.331 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:50.331 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:50.332 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593650.003775_1951.1", 2023-07-29T01:20:50.332 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:20:50.003775Z" 2023-07-29T01:20:50.332 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:50.332 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:50.332 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:20:50.332 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T01:20:50.333 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:50.333 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:50.333 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:20:50.333 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:12.811406Z" 2023-07-29T01:20:50.333 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:50.333 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T01:20:50.335 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690591217.836402_294.1', 1: '1_1690593650.003775_1951.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:20:50.335 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T01:20:50.336 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:20:50.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:20:50.337 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:20:50.383 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:20:50.384 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:20:50.412 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:20:50.317+0000 7f6593526880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:20:50.413 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:20:50.318+0000 7f6593526880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:20:50.426 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:20:50.426 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:20:50.426 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:20:50.427 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:20:50.427 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:20:50.427 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:20:50.427 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:20:50.427 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:50.427 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:20:50.427 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:50.428 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:20:50.428 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:50.428 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:20:50.428 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:20:50.428 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:50.428 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:20:50.429 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:50.429 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:20:50.429 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:20:50.429 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:50.429 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:50.429 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:50.429 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:20:50.430 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:50.430 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:20:50.430 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593523.957474_1842.1", 2023-07-29T01:20:50.430 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:50.430 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:20:50.430 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:50.430 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:18:43.957474Z", 2023-07-29T01:20:50.431 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:20:50.431 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:50.431 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:50.431 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:50.431 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:20:50.431 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:50.432 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:20:50.432 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:20:50.432 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:50.432 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:20:50.432 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:50.432 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:20:50.432 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:20:50.433 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:50.433 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:50.433 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:50.433 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:20:50.433 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:50.433 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:20:50.434 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:20:50.434 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:50.434 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:20:50.434 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:50.434 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:20:50.434 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:20:50.434 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:50.435 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:50.435 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:20:50.435 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:50.435 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:20:50.435 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:20:50.435 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:20:50.435 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:50.436 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:20:50Z" 2023-07-29T01:20:50.436 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:20:50.437 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593523.957474_1842.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:18:43.957474Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:20:50Z" } 2023-07-29T01:20:50.437 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593523.957474_1842.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:18:43.957474Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:20:50.438 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593650.003775_1951.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:20:50.438 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593523.957474_1842.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:20:50.438 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593650.003775_1951.1 target=1_1690593523.957474_1842.1 2023-07-29T01:20:55.440 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:20:55.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:20:55.441 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:20:55.496 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:20:55.497 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:20:55.531 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:20:55.435+0000 7fb990922880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:20:55.532 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:20:55.436+0000 7fb990922880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:20:55.547 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:20:55.547 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:20:55.547 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:20:55.547 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:20:55.547 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:20:55.548 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:20:55.548 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:20:55.548 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:55.548 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:20:55.548 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:55.548 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:20:55.549 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:55.549 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:20:55.549 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:20:55.549 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:55.549 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:20:55.549 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:55.549 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:20:55.550 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:20:55.550 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:55.550 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:55.550 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:55.550 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:20:55.550 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:55.550 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:20:55.551 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593523.957474_1842.1", 2023-07-29T01:20:55.551 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:55.551 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:20:55.551 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:55.551 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:18:43.957474Z", 2023-07-29T01:20:55.551 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:20:55.551 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:55.552 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:55.552 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:55.552 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:20:55.552 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:55.552 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:20:55.552 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:20:55.553 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:55.553 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:20:55.553 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:55.553 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:20:55.553 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:20:55.553 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:55.553 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:55.554 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:20:55.554 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:20:55.554 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:20:55.554 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:20:55.554 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:20:55.554 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:20:55.554 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:20:55.555 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:20:55.555 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:20:55.555 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:20:55.555 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:55.555 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:20:55.555 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:20:55.556 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:55.556 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:20:55.556 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:20:55.556 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:20:55.556 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:20:55.557 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:20:55Z" 2023-07-29T01:20:55.557 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:20:55.558 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593523.957474_1842.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:18:43.957474Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:20:55Z" } 2023-07-29T01:20:55.558 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593523.957474_1842.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:18:43.957474Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:20:55.558 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593650.003775_1951.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:20:55.559 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593523.957474_1842.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:20:55.559 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593650.003775_1951.1 target=1_1690593523.957474_1842.1 2023-07-29T01:21:00.560 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:21:00.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:21:00.561 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:21:00.614 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:21:00.614 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:21:00.648 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:21:00.552+0000 7f1361f35880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:21:00.649 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:21:00.553+0000 7f1361f35880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:21:00.663 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:21:00.663 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:21:00.663 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:21:00.663 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:21:00.663 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:21:00.663 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:21:00.664 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:21:00.664 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:00.664 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:21:00.664 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:00.664 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:21:00.664 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:00.665 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:21:00.665 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:21:00.665 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:00.665 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:21:00.665 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:00.665 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:21:00.665 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:21:00.666 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:00.666 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:00.666 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:00.666 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:21:00.666 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:00.666 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:21:00.666 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593523.957474_1842.1", 2023-07-29T01:21:00.667 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:00.667 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:21:00.667 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:00.667 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:18:43.957474Z", 2023-07-29T01:21:00.667 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:21:00.667 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:00.667 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:00.668 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:00.668 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:21:00.668 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:00.668 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:21:00.668 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:21:00.668 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:00.668 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:21:00.669 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:00.669 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:21:00.669 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:21:00.669 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:00.669 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:00.669 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:00.670 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:21:00.670 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:00.670 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:21:00.670 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:21:00.670 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:00.670 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:21:00.670 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:00.671 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:21:00.671 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:21:00.671 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:00.671 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:00.671 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:21:00.671 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:00.671 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:21:00.672 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:21:00.672 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:21:00.672 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:00.672 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:21:00Z" 2023-07-29T01:21:00.672 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:21:00.673 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593523.957474_1842.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:18:43.957474Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:21:00Z" } 2023-07-29T01:21:00.673 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593523.957474_1842.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:18:43.957474Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:21:00.674 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593650.003775_1951.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:21:00.674 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593523.957474_1842.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:21:00.675 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593650.003775_1951.1 target=1_1690593523.957474_1842.1 2023-07-29T01:21:05.676 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:21:05.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:21:05.677 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:21:05.729 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:21:05.730 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:21:05.764 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:21:05.668+0000 7f505659c880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:21:05.764 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:21:05.669+0000 7f505659c880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:21:05.778 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:21:05.779 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:21:05.779 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:21:05.779 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:21:05.779 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:21:05.779 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:21:05.780 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:21:05.780 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:05.780 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:21:05.780 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:05.780 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:21:05.780 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:05.781 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:21:05.781 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:21:05.781 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:05.781 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:21:05.781 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:05.781 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:21:05.781 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:21:05.782 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:05.782 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:05.782 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:05.782 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:21:05.782 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:05.782 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:21:05.783 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593523.957474_1842.1", 2023-07-29T01:21:05.783 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:05.783 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:21:05.783 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:05.783 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:18:43.957474Z", 2023-07-29T01:21:05.783 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:21:05.783 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:05.784 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:05.784 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:05.784 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:21:05.784 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:05.784 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:21:05.784 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:21:05.785 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:05.785 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:21:05.785 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:05.785 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:21:05.785 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:21:05.785 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:05.785 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:05.786 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:05.786 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:21:05.786 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:05.786 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:21:05.786 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:21:05.786 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:05.787 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:21:05.787 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:05.787 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:21:05.787 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:21:05.787 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:05.787 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:05.787 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:21:05.788 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:05.788 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:21:05.788 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:21:05.788 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:21:05.788 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:05.788 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:21:05Z" 2023-07-29T01:21:05.789 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:21:05.789 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593523.957474_1842.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:18:43.957474Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:21:05Z" } 2023-07-29T01:21:05.790 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593523.957474_1842.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:18:43.957474Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:21:05.790 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593650.003775_1951.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:21:05.791 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593523.957474_1842.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:21:05.791 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593650.003775_1951.1 target=1_1690593523.957474_1842.1 2023-07-29T01:21:10.792 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:21:10.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:21:10.793 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:21:10.846 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:21:10.846 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:21:10.879 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:21:10.783+0000 7f66a6a9d880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:21:10.879 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:21:10.784+0000 7f66a6a9d880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:21:10.893 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:21:10.894 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:21:10.894 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:21:10.894 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:21:10.894 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:21:10.895 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:21:10.895 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:21:10.895 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:10.895 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:21:10.895 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:10.895 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:21:10.895 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:10.896 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:21:10.896 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:21:10.896 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:10.896 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:21:10.896 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:10.896 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:21:10.897 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:21:10.897 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:10.897 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:10.897 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:10.897 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:21:10.897 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:10.897 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:21:10.898 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593523.957474_1842.1", 2023-07-29T01:21:10.898 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:10.898 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:21:10.898 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:10.898 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:18:43.957474Z", 2023-07-29T01:21:10.898 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:21:10.898 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:10.899 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:10.899 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:10.899 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:21:10.899 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:10.899 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:21:10.899 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:21:10.900 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:10.900 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:21:10.900 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:10.900 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:21:10.900 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:21:10.900 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:10.900 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:10.901 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:10.901 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:21:10.901 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:10.901 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:21:10.901 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:21:10.901 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:10.901 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:21:10.902 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:10.902 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:21:10.902 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:21:10.902 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:10.902 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:10.902 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:21:10.903 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:10.903 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:21:10.903 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:21:10.903 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:21:10.903 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:10.903 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:21:10Z" 2023-07-29T01:21:10.903 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:21:10.904 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593523.957474_1842.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:18:43.957474Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:21:10Z" } 2023-07-29T01:21:10.905 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593523.957474_1842.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:18:43.957474Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:21:10.905 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593650.003775_1951.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:21:10.906 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593523.957474_1842.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:21:10.906 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593650.003775_1951.1 target=1_1690593523.957474_1842.1 2023-07-29T01:21:15.907 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:21:15.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:21:15.908 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:21:15.961 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:21:15.961 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:21:15.993 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:21:15.897+0000 7fabba575880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:21:15.994 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:21:15.898+0000 7fabba575880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:21:16.008 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:21:16.008 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:21:16.008 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:21:16.008 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:21:16.009 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:21:16.009 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:21:16.009 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:21:16.009 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:16.009 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:21:16.009 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:16.009 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:21:16.010 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:16.010 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:21:16.010 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:21:16.010 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:16.010 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:21:16.010 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:16.010 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:21:16.011 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:21:16.011 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:16.011 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:16.011 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:16.011 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:21:16.011 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:16.012 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:21:16.012 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593523.957474_1842.1", 2023-07-29T01:21:16.012 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:16.012 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:21:16.012 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:16.012 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:18:43.957474Z", 2023-07-29T01:21:16.013 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:21:16.013 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:16.013 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:16.013 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:16.013 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:21:16.013 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:16.013 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:21:16.014 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:21:16.014 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:16.014 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:21:16.014 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:16.014 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:21:16.014 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:21:16.014 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:16.015 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:16.015 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:16.015 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:21:16.015 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:16.015 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:21:16.015 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:21:16.016 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:16.016 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:21:16.016 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:16.016 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:21:16.016 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:21:16.016 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:16.017 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:16.017 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:21:16.017 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:16.017 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:21:16.017 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:21:16.017 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:21:16.017 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:16.018 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:21:15Z" 2023-07-29T01:21:16.018 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:21:16.018 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593523.957474_1842.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:18:43.957474Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:21:15Z" } 2023-07-29T01:21:16.019 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593523.957474_1842.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:18:43.957474Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:21:16.019 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593650.003775_1951.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:21:16.020 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593523.957474_1842.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:21:16.020 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1690593650.003775_1951.1 target=1_1690593523.957474_1842.1 2023-07-29T01:21:21.021 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:21:21.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:21:21.022 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:21:21.076 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:21:21.076 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:21:21.108 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:21:21.012+0000 7f9c9b321880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:21:21.109 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:21:21.013+0000 7f9c9b321880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:21:21.124 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:21:21.124 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:21:21.124 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:21:21.125 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:21:21.125 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:21:21.125 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:21:21.125 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:21:21.125 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:21.125 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:21:21.126 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:21.126 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:21:21.126 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:21.126 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:21:21.126 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:21:21.126 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:21.126 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:21:21.127 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:21.127 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:21:21.127 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:21:21.127 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:21.127 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:21.127 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:21.127 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:21:21.128 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:21.128 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:21:21.128 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593650.003775_1951.1", 2023-07-29T01:21:21.128 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:21.128 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:21:21.128 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:21.128 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:20:50.003775Z", 2023-07-29T01:21:21.129 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:21:21.129 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:21.129 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:21.129 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:21.129 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:21:21.129 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:21.129 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:21:21.130 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:21:21.130 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:21.130 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:21:21.130 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:21.130 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:21:21.130 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:21:21.130 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:21.131 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:21.131 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:21.131 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:21:21.131 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:21.131 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:21:21.132 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:21:21.132 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:21.132 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:21:21.132 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:21.132 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:21:21.132 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:21:21.133 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:21.133 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:21.133 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:21:21.133 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:21.133 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:21:21.133 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:21:21.133 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:21:21.134 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:21.134 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:21:21Z" 2023-07-29T01:21:21.134 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:21:21.135 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593650.003775_1951.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:20:50.003775Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:21:21Z" } 2023-07-29T01:21:21.135 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593650.003775_1951.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:20:50.003775Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:21:21.136 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593650.003775_1951.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:21:21.136 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593650.003775_1951.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:21:21.136 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T01:21:21.137 DEBUG:boto:path=/hiiqos-74/ 2023-07-29T01:21:21.137 DEBUG:boto:auth_path=/hiiqos-74/ 2023-07-29T01:21:21.137 DEBUG:boto:Method: HEAD 2023-07-29T01:21:21.138 DEBUG:boto:Path: /hiiqos-74/ 2023-07-29T01:21:21.138 DEBUG:boto:Data: 2023-07-29T01:21:21.139 DEBUG:boto:Headers: {} 2023-07-29T01:21:21.139 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:21:21.139 DEBUG:boto:Port: 8000 2023-07-29T01:21:21.140 DEBUG:boto:Params: {} 2023-07-29T01:21:21.140 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-29T01:21:21.141 DEBUG:boto:Token: None 2023-07-29T01:21:21.141 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:21:21 GMT /hiiqos-74/ 2023-07-29T01:21:21.142 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:+7j8YcrIshslwj4VeuRLLykkDL4= 2023-07-29T01:21:21.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:21:21 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+7j8YcrIshslwj4VeuRLLykkDL4=', 'Content-Length': '0'} 2023-07-29T01:21:26.148 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000094b0a54fdfaa1d2f-0064c46996-2ca7-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:21:26 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:21:26.149 DEBUG:boto:path=/hiiqos-74/obj3 2023-07-29T01:21:26.150 DEBUG:boto:auth_path=/hiiqos-74/obj3 2023-07-29T01:21:26.150 DEBUG:boto:Method: PUT 2023-07-29T01:21:26.151 DEBUG:boto:Path: /hiiqos-74/obj3 2023-07-29T01:21:26.151 DEBUG:boto:Data: 2023-07-29T01:21:26.151 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T01:21:26.152 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:21:26.152 DEBUG:boto:Port: 8000 2023-07-29T01:21:26.152 DEBUG:boto:Params: {} 2023-07-29T01:21:26.153 DEBUG:boto:Token: None 2023-07-29T01:21:26.154 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 01:21:26 GMT /hiiqos-74/obj3 2023-07-29T01:21:26.154 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bqfEYyITxdp8qEEbHLNCubZm+RE= 2023-07-29T01:21:26.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 01:21:26 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bqfEYyITxdp8qEEbHLNCubZm+RE='} 2023-07-29T01:21:26.166 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009315084a4754e772-0064c46996-2ca7-a1'), ('Date', 'Sat, 29 Jul 2023 01:21:26 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:21:26.166 DEBUG:boto:path=/hiiqos-74/ 2023-07-29T01:21:26.167 DEBUG:boto:auth_path=/hiiqos-74/ 2023-07-29T01:21:26.167 DEBUG:boto:Method: HEAD 2023-07-29T01:21:26.167 DEBUG:boto:Path: /hiiqos-74/ 2023-07-29T01:21:26.168 DEBUG:boto:Data: 2023-07-29T01:21:26.168 DEBUG:boto:Headers: {} 2023-07-29T01:21:26.168 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:21:26.169 DEBUG:boto:Port: 8000 2023-07-29T01:21:26.169 DEBUG:boto:Params: {} 2023-07-29T01:21:26.169 DEBUG:boto:Token: None 2023-07-29T01:21:26.170 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:21:26 GMT /hiiqos-74/ 2023-07-29T01:21:26.170 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BetYiwdwLXeSGSymDUADjNGreAw= 2023-07-29T01:21:26.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:21:26 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BetYiwdwLXeSGSymDUADjNGreAw=', 'Content-Length': '0'} 2023-07-29T01:21:26.173 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006a26ab7d78f07fe7-0064c46996-2ca7-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:21:26 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:21:26.174 DEBUG:boto:path=/hiiqos-74/obj4 2023-07-29T01:21:26.174 DEBUG:boto:auth_path=/hiiqos-74/obj4 2023-07-29T01:21:26.175 DEBUG:boto:Method: PUT 2023-07-29T01:21:26.175 DEBUG:boto:Path: /hiiqos-74/obj4 2023-07-29T01:21:26.175 DEBUG:boto:Data: 2023-07-29T01:21:26.176 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T01:21:26.176 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:21:26.177 DEBUG:boto:Port: 8000 2023-07-29T01:21:26.177 DEBUG:boto:Params: {} 2023-07-29T01:21:26.177 DEBUG:boto:Token: None 2023-07-29T01:21:26.178 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Sat, 29 Jul 2023 01:21:26 GMT /hiiqos-74/obj4 2023-07-29T01:21:26.178 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:5wSL7X+yz9J4U0CrYWTY5HqmCx8= 2023-07-29T01:21:26.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 01:21:26 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5wSL7X+yz9J4U0CrYWTY5HqmCx8='} 2023-07-29T01:21:26.187 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fb954f3720b620bd-0064c46996-2ca7-a1'), ('Date', 'Sat, 29 Jul 2023 01:21:26 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:21:26.189 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:21:26.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:21:26.189 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:21:26.240 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:21:26.241 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:21:26.274 INFO:teuthology.orchestra.run.smithi101.stderr:No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T01:21:26.288 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T01:21:26.288 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:26.288 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:21:26.288 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:17.836402Z" 2023-07-29T01:21:26.289 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:26.289 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:26.289 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593650.003775_1951.1", 2023-07-29T01:21:26.289 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:20:50.003775Z" 2023-07-29T01:21:26.289 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:26.289 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:26.290 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:21:26.290 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "0.000000" 2023-07-29T01:21:26.290 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:26.290 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:26.290 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:21:26.290 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T00:40:12.811406Z" 2023-07-29T01:21:26.291 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:26.291 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T01:21:26.292 DEBUG:rgw_multi.tests:master meta markers={0: '1_1690591217.836402_294.1', 1: '1_1690593650.003775_1951.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:21:26.292 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-07-29T01:21:26.293 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:21:26.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:21:26.294 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:21:26.340 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:21:26.341 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:21:26.369 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:21:26.272+0000 7fb4e0ba2880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:21:26.369 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-29T01:21:26.273+0000 7fb4e0ba2880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:21:26.382 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:21:26.382 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:21:26.382 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:21:26.382 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:21:26.383 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:21:26.383 INFO:teuthology.orchestra.run.smithi101.stdout: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:21:26.383 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:21:26.383 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:26.384 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:21:26.384 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:26.384 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:21:26.384 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:26.384 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:21:26.384 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:21:26.385 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:26.385 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 1, 2023-07-29T01:21:26.385 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:26.385 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:21:26.385 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:21:26.385 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:26.385 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:26.386 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:26.386 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:21:26.386 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:26.386 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:21:26.386 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690593650.003775_1951.1", 2023-07-29T01:21:26.386 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:26.386 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:21:26.387 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:26.387 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:20:50.003775Z", 2023-07-29T01:21:26.387 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:21:26.387 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:26.387 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:26.387 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:26.387 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:21:26.388 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:26.388 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:21:26.388 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "", 2023-07-29T01:21:26.388 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:26.388 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:21:26.388 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:26.389 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "0.000000", 2023-07-29T01:21:26.389 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:21:26.389 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:26.389 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:26.389 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:26.389 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:21:26.390 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:26.390 INFO:teuthology.orchestra.run.smithi101.stdout: "state": 1, 2023-07-29T01:21:26.390 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:21:26.390 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:26.391 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:21:26.391 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:26.391 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:21:26.392 INFO:teuthology.orchestra.run.smithi101.stdout: "realm_epoch": 2 2023-07-29T01:21:26.392 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:26.392 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:26.392 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:21:26.393 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:26.393 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:21:26.393 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 1, 2023-07-29T01:21:26.394 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 1 2023-07-29T01:21:26.394 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:26.394 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:21:26Z" 2023-07-29T01:21:26.394 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:21:26.395 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1690591217.836402_294.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-07-29T00:40:17.836402Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1690593650.003775_1951.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:20:50.003775Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1690591212.811406_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T00:40:12.811406Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-07-29T01:21:26Z" } 2023-07-29T01:21:26.396 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593650.003775_1951.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:20:50.003775Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:21:26.397 DEBUG:rgw_multi.tests:log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593650.003775_1951.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:21:26.397 DEBUG:rgw_multi.tests:sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593650.003775_1951.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:21:26.397 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-07-29T01:21:26.398 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:21:26.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:21:26.399 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:21:26.442 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:21:26.443 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:21:26.491 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2023-07-29T01:21:26.492 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:26.492 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052796", 2023-07-29T01:21:26.492 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:21:26.070114Z" 2023-07-29T01:21:26.492 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:26.492 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:26.492 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:21:26.493 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:21:26.493 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:26.493 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:26.493 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000046221", 2023-07-29T01:21:26.493 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:19:15.858505Z" 2023-07-29T01:21:26.493 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:26.493 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:26.494 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031915", 2023-07-29T01:21:26.494 INFO:teuthology.orchestra.run.smithi101.stdout: "last_update": "2023-07-29T01:21:26.091582Z" 2023-07-29T01:21:26.494 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:26.494 INFO:teuthology.orchestra.run.smithi101.stdout:] 2023-07-29T01:21:26.495 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000052796', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031915'} 2023-07-29T01:21:26.495 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-07-29T01:21:26.496 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:21:26.496 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:21:26.496 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:21:26.544 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:21:26.544 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:21:26.586 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:21:26.586 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:21:26.587 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:21:26.587 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:21:26.587 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:21:26.587 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:21:26.587 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:26.588 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:21:26.588 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:26.588 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:21:26.588 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:26.589 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:21:26.589 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052655", 2023-07-29T01:21:26.589 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:26.589 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:21:26.589 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:26.590 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:19:15.824401Z" 2023-07-29T01:21:26.590 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:26.590 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:26.590 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:26.590 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:21:26.590 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:26.590 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:21:26.591 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:21:26.591 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:26.591 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:21:26.591 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:26.591 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:21:26.591 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:26.592 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:26.592 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:26.592 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:21:26.592 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:26.592 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:21:26.592 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000046221", 2023-07-29T01:21:26.593 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:26.593 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:21:26.593 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:26.593 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:19:15.858505Z" 2023-07-29T01:21:26.593 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:26.593 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:26.594 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:26.594 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:21:26.594 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:26.594 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:21:26.594 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031774", 2023-07-29T01:21:26.594 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:26.595 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:21:26.595 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:26.595 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.812152Z" 2023-07-29T01:21:26.595 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:26.595 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:26.596 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:21:26.596 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:26.596 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:21:26.596 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:21:26.596 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:21:26.596 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:26.597 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:21:26Z" 2023-07-29T01:21:26.597 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:21:26.598 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052655", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:19:15.824401Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046221", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:19:15.858505Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.812152Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:21:26Z" } 2023-07-29T01:21:26.598 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052655', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:19:15.824401Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046221', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:19:15.858505Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.812152Z'}}] 2023-07-29T01:21:26.598 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052796', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031915'} 2023-07-29T01:21:26.599 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052655', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:21:26.599 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052796 target=00000000000000000000:00000000000000052655, shard=3 master=00000000000000000000:00000000000000031915 target=00000000000000000000:00000000000000031774 2023-07-29T01:21:31.600 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:21:31.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:21:31.601 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:21:31.654 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:21:31.655 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:21:31.701 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:21:31.701 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:21:31.701 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:21:31.702 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:21:31.702 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:21:31.702 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:21:31.702 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:31.702 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:21:31.703 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:31.703 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:21:31.703 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:31.703 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:21:31.703 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052655", 2023-07-29T01:21:31.703 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:31.704 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:21:31.704 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:31.704 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:19:15.824401Z" 2023-07-29T01:21:31.704 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:31.704 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:31.705 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:31.705 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:21:31.705 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:31.705 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:21:31.705 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:21:31.706 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:31.706 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:21:31.706 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:31.706 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:21:31.706 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:31.706 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:31.707 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:31.707 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:21:31.707 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:31.707 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:21:31.707 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000046221", 2023-07-29T01:21:31.707 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:31.708 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:21:31.708 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:31.708 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:19:15.858505Z" 2023-07-29T01:21:31.708 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:31.708 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:31.708 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:31.709 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:21:31.709 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:31.709 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:21:31.709 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031774", 2023-07-29T01:21:31.709 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:31.710 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:21:31.710 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:31.710 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.812152Z" 2023-07-29T01:21:31.710 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:31.710 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:31.710 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:21:31.711 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:31.711 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:21:31.711 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:21:31.711 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:21:31.711 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:31.711 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:21:31Z" 2023-07-29T01:21:31.712 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:21:31.712 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052655", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:19:15.824401Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046221", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:19:15.858505Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.812152Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:21:31Z" } 2023-07-29T01:21:31.713 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052655', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:19:15.824401Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046221', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:19:15.858505Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.812152Z'}}] 2023-07-29T01:21:31.713 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052796', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031915'} 2023-07-29T01:21:31.714 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052655', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:21:31.714 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052796 target=00000000000000000000:00000000000000052655, shard=3 master=00000000000000000000:00000000000000031915 target=00000000000000000000:00000000000000031774 2023-07-29T01:21:36.716 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:21:36.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:21:36.717 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:21:36.772 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:21:36.773 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:21:36.820 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:21:36.820 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:21:36.820 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:21:36.820 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:21:36.821 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:21:36.821 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:21:36.821 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:36.821 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:21:36.821 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:36.821 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:21:36.822 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:36.822 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:21:36.822 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052655", 2023-07-29T01:21:36.822 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:36.822 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:21:36.822 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:36.823 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:19:15.824401Z" 2023-07-29T01:21:36.823 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:36.823 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:36.823 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:36.823 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:21:36.823 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:36.823 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:21:36.824 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:21:36.824 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:36.824 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:21:36.824 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:36.824 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:21:36.824 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:36.825 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:36.825 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:36.825 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:21:36.825 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:36.825 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:21:36.825 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000046221", 2023-07-29T01:21:36.825 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:36.826 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:21:36.826 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:36.826 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:19:15.858505Z" 2023-07-29T01:21:36.826 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:36.826 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:36.826 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:36.826 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:21:36.827 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:36.827 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:21:36.827 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031774", 2023-07-29T01:21:36.827 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:36.827 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:21:36.827 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:36.827 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.812152Z" 2023-07-29T01:21:36.828 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:36.828 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:36.828 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:21:36.828 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:36.828 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:21:36.828 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:21:36.829 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:21:36.829 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:36.829 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:21:36Z" 2023-07-29T01:21:36.829 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:21:36.830 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052655", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:19:15.824401Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046221", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:19:15.858505Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.812152Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:21:36Z" } 2023-07-29T01:21:36.830 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052655', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:19:15.824401Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046221', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:19:15.858505Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.812152Z'}}] 2023-07-29T01:21:36.831 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052796', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031915'} 2023-07-29T01:21:36.831 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052655', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:21:36.832 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052796 target=00000000000000000000:00000000000000052655, shard=3 master=00000000000000000000:00000000000000031915 target=00000000000000000000:00000000000000031774 2023-07-29T01:21:41.833 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:21:41.834 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:21:41.834 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:21:41.890 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:21:41.891 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:21:41.938 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:21:41.938 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:21:41.938 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:21:41.938 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:21:41.939 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:21:41.939 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:21:41.939 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:41.939 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:21:41.939 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:41.939 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:21:41.940 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:41.940 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:21:41.940 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052655", 2023-07-29T01:21:41.940 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:41.940 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:21:41.940 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:41.941 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:19:15.824401Z" 2023-07-29T01:21:41.941 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:41.941 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:41.941 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:41.941 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:21:41.941 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:41.941 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:21:41.942 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:21:41.942 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:41.942 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:21:41.942 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:41.942 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:21:41.942 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:41.943 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:41.943 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:41.943 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:21:41.943 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:41.943 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:21:41.943 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000046221", 2023-07-29T01:21:41.943 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:41.944 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:21:41.944 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:41.944 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:19:15.858505Z" 2023-07-29T01:21:41.944 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:41.944 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:41.944 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:41.945 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:21:41.945 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:41.945 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:21:41.945 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031774", 2023-07-29T01:21:41.945 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:41.945 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:21:41.945 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:41.946 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.812152Z" 2023-07-29T01:21:41.946 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:41.946 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:41.946 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:21:41.946 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:41.946 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:21:41.946 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:21:41.947 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:21:41.947 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:41.947 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:21:41Z" 2023-07-29T01:21:41.947 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:21:41.948 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052655", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:19:15.824401Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046221", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:19:15.858505Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.812152Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:21:41Z" } 2023-07-29T01:21:41.948 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052655', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:19:15.824401Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046221', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:19:15.858505Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.812152Z'}}] 2023-07-29T01:21:41.949 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052796', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031915'} 2023-07-29T01:21:41.949 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052655', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:21:41.949 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052796 target=00000000000000000000:00000000000000052655, shard=3 master=00000000000000000000:00000000000000031915 target=00000000000000000000:00000000000000031774 2023-07-29T01:21:46.950 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:21:46.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:21:46.951 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:21:47.006 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:21:47.006 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:21:47.053 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:21:47.054 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:21:47.054 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:21:47.054 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:21:47.054 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:21:47.054 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:21:47.055 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:47.055 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:21:47.055 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:47.055 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:21:47.055 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:47.055 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:21:47.055 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052655", 2023-07-29T01:21:47.056 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:47.056 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:21:47.056 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:47.056 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:19:15.824401Z" 2023-07-29T01:21:47.056 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:47.056 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:47.057 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:47.057 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:21:47.057 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:47.057 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:21:47.057 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:21:47.057 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:47.057 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:21:47.058 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:47.058 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:21:47.058 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:47.058 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:47.058 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:47.058 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:21:47.058 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:47.059 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:21:47.059 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000046221", 2023-07-29T01:21:47.059 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:47.059 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:21:47.059 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:47.060 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:19:15.858505Z" 2023-07-29T01:21:47.060 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:47.060 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:47.060 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:47.060 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:21:47.060 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:47.061 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:21:47.061 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031774", 2023-07-29T01:21:47.061 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:47.061 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:21:47.061 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:47.061 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.812152Z" 2023-07-29T01:21:47.061 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:47.062 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:47.062 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:21:47.062 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:47.062 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:21:47.062 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:21:47.062 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:21:47.062 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:47.063 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:21:46Z" 2023-07-29T01:21:47.063 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:21:47.064 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052655", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:19:15.824401Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046221", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:19:15.858505Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.812152Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:21:46Z" } 2023-07-29T01:21:47.064 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052655', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:19:15.824401Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046221', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:19:15.858505Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.812152Z'}}] 2023-07-29T01:21:47.064 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052796', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031915'} 2023-07-29T01:21:47.065 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052655', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:21:47.065 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052796 target=00000000000000000000:00000000000000052655, shard=3 master=00000000000000000000:00000000000000031915 target=00000000000000000000:00000000000000031774 2023-07-29T01:21:52.066 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:21:52.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:21:52.067 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:21:52.122 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:21:52.122 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:21:52.170 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:21:52.170 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:21:52.170 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:21:52.170 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:21:52.171 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:21:52.171 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:21:52.171 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:52.171 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:21:52.171 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:52.171 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:21:52.172 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:52.172 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:21:52.172 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052655", 2023-07-29T01:21:52.172 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:52.172 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:21:52.172 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:52.173 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:19:15.824401Z" 2023-07-29T01:21:52.173 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:52.173 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:52.173 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:52.173 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:21:52.173 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:52.174 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:21:52.174 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:21:52.174 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:52.174 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:21:52.174 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:52.174 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:21:52.174 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:52.175 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:52.175 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:52.175 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:21:52.175 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:52.175 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:21:52.175 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000046221", 2023-07-29T01:21:52.175 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:52.176 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:21:52.176 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:52.176 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:19:15.858505Z" 2023-07-29T01:21:52.176 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:52.176 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:52.176 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:52.177 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:21:52.177 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:52.177 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:21:52.177 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031774", 2023-07-29T01:21:52.177 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:52.177 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:21:52.177 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:52.178 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.812152Z" 2023-07-29T01:21:52.178 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:52.178 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:52.178 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:21:52.178 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:52.178 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:21:52.178 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:21:52.179 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:21:52.179 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:52.179 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:21:52Z" 2023-07-29T01:21:52.179 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:21:52.180 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052655", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:19:15.824401Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046221", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:19:15.858505Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.812152Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:21:52Z" } 2023-07-29T01:21:52.180 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052655', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:19:15.824401Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046221', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:19:15.858505Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.812152Z'}}] 2023-07-29T01:21:52.181 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052796', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031915'} 2023-07-29T01:21:52.181 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052655', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:21:52.181 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052796 target=00000000000000000000:00000000000000052655, shard=3 master=00000000000000000000:00000000000000031915 target=00000000000000000000:00000000000000031774 2023-07-29T01:21:57.182 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:21:57.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:21:57.183 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:21:57.238 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:21:57.238 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:21:57.287 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:21:57.287 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:21:57.287 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:21:57.287 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:21:57.287 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:21:57.288 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:21:57.288 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:57.288 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:21:57.288 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:57.288 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:21:57.288 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:57.289 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:21:57.289 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052655", 2023-07-29T01:21:57.289 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:57.289 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:21:57.289 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:57.289 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:19:15.824401Z" 2023-07-29T01:21:57.289 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:57.290 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:57.290 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:57.290 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:21:57.290 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:57.290 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:21:57.290 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:21:57.291 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:57.291 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:21:57.291 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:57.291 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:21:57.291 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:57.291 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:57.291 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:57.292 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:21:57.292 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:57.292 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:21:57.292 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000046221", 2023-07-29T01:21:57.292 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:57.292 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:21:57.293 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:57.293 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:19:15.858505Z" 2023-07-29T01:21:57.293 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:57.293 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:57.293 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:21:57.293 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:21:57.293 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:21:57.294 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:21:57.294 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031774", 2023-07-29T01:21:57.294 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:21:57.294 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:21:57.294 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:21:57.294 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.812152Z" 2023-07-29T01:21:57.294 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:57.295 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:21:57.295 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:21:57.295 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:57.295 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:21:57.295 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:21:57.295 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:21:57.295 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:21:57.296 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:21:57Z" 2023-07-29T01:21:57.296 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:21:57.297 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052655", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:19:15.824401Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046221", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:19:15.858505Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.812152Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:21:57Z" } 2023-07-29T01:21:57.297 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052655', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:19:15.824401Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046221', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:19:15.858505Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.812152Z'}}] 2023-07-29T01:21:57.298 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052796', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031915'} 2023-07-29T01:21:57.298 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052655', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:21:57.298 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052796 target=00000000000000000000:00000000000000052655, shard=3 master=00000000000000000000:00000000000000031915 target=00000000000000000000:00000000000000031774 2023-07-29T01:22:02.299 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:02.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:02.301 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:02.356 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:22:02.356 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:22:02.403 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:22:02.403 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:22:02.403 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:22:02.404 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:22:02.404 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:22:02.404 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:22:02.404 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:02.404 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:22:02.404 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:02.405 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:22:02.405 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:02.405 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:02.405 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052655", 2023-07-29T01:22:02.405 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:02.405 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:22:02.405 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:02.406 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:19:15.824401Z" 2023-07-29T01:22:02.406 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:02.406 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:02.406 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:02.406 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:22:02.406 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:02.407 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:02.407 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:02.407 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:02.407 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:22:02.407 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:02.407 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:02.407 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:02.408 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:02.408 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:02.408 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:22:02.408 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:02.408 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:02.408 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000046221", 2023-07-29T01:22:02.408 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:02.409 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:22:02.409 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:02.409 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:19:15.858505Z" 2023-07-29T01:22:02.409 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:02.409 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:02.409 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:02.409 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:22:02.410 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:02.410 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:02.410 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031774", 2023-07-29T01:22:02.410 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:02.410 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:22:02.410 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:02.410 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.812152Z" 2023-07-29T01:22:02.411 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:02.411 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:02.411 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:22:02.411 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:02.411 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:22:02.411 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:22:02.411 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:22:02.412 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:02.412 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:22:02Z" 2023-07-29T01:22:02.412 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:22:02.413 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052655", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:19:15.824401Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046221", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:19:15.858505Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.812152Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:22:02Z" } 2023-07-29T01:22:02.413 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052655', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:19:15.824401Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046221', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:19:15.858505Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.812152Z'}}] 2023-07-29T01:22:02.414 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052796', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031915'} 2023-07-29T01:22:02.414 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052655', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:22:02.414 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052796 target=00000000000000000000:00000000000000052655, shard=3 master=00000000000000000000:00000000000000031915 target=00000000000000000000:00000000000000031774 2023-07-29T01:22:07.415 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:07.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:07.417 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:07.474 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:22:07.475 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:22:07.525 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:22:07.525 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:22:07.525 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:22:07.526 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:22:07.526 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:22:07.526 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:22:07.526 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:07.526 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:22:07.526 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:07.527 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:22:07.527 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:07.527 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:07.527 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052655", 2023-07-29T01:22:07.527 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:07.527 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:22:07.527 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:07.528 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:19:15.824401Z" 2023-07-29T01:22:07.528 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:07.528 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:07.528 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:07.528 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:22:07.528 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:07.529 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:07.529 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:07.529 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:07.529 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:22:07.529 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:07.529 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:07.529 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:07.530 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:07.530 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:07.530 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:22:07.530 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:07.530 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:07.530 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000046221", 2023-07-29T01:22:07.530 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:07.531 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:22:07.531 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:07.531 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:19:15.858505Z" 2023-07-29T01:22:07.531 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:07.531 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:07.531 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:07.531 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:22:07.532 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:07.532 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:07.532 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031774", 2023-07-29T01:22:07.532 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:07.532 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:22:07.532 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:07.532 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.812152Z" 2023-07-29T01:22:07.533 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:07.533 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:07.533 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:22:07.533 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:07.533 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:22:07.533 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:22:07.533 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:22:07.534 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:07.534 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:22:07Z" 2023-07-29T01:22:07.534 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:22:07.535 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052655", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:19:15.824401Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046221", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:19:15.858505Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.812152Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:22:07Z" } 2023-07-29T01:22:07.535 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052655', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:19:15.824401Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046221', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:19:15.858505Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.812152Z'}}] 2023-07-29T01:22:07.536 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052796', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031915'} 2023-07-29T01:22:07.536 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052655', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:22:07.537 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052796 target=00000000000000000000:00000000000000052655, shard=3 master=00000000000000000000:00000000000000031915 target=00000000000000000000:00000000000000031774 2023-07-29T01:22:12.538 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:12.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:12.539 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:12.594 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:22:12.594 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:22:12.643 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:22:12.643 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:22:12.643 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:22:12.643 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:22:12.644 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:22:12.644 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:22:12.644 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:12.644 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:22:12.644 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:12.644 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:22:12.645 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:12.645 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:12.645 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052655", 2023-07-29T01:22:12.645 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:12.645 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:22:12.646 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:12.646 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:19:15.824401Z" 2023-07-29T01:22:12.646 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:12.646 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:12.647 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:12.647 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:22:12.647 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:12.647 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:12.647 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:12.647 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:12.648 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:22:12.648 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:12.648 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:12.648 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:12.648 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:12.649 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:12.649 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:22:12.649 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:12.649 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:12.649 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000046221", 2023-07-29T01:22:12.649 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:12.649 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:22:12.650 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:12.650 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:19:15.858505Z" 2023-07-29T01:22:12.650 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:12.650 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:12.650 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:12.650 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:22:12.651 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:12.651 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:12.651 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031774", 2023-07-29T01:22:12.652 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:12.652 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:22:12.653 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:12.653 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.812152Z" 2023-07-29T01:22:12.653 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:12.654 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:12.654 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:22:12.654 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:12.655 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:22:12.655 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:22:12.655 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:22:12.655 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:12.656 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:22:12Z" 2023-07-29T01:22:12.656 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:22:12.658 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052655", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:19:15.824401Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046221", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:19:15.858505Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.812152Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:22:12Z" } 2023-07-29T01:22:12.658 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052655', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:19:15.824401Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046221', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:19:15.858505Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.812152Z'}}] 2023-07-29T01:22:12.659 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052796', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031915'} 2023-07-29T01:22:12.659 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052655', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:22:12.660 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052796 target=00000000000000000000:00000000000000052655, shard=3 master=00000000000000000000:00000000000000031915 target=00000000000000000000:00000000000000031774 2023-07-29T01:22:17.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:17.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:17.662 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:17.717 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:22:17.717 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:22:17.764 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:22:17.764 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:22:17.765 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:22:17.765 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:22:17.765 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:22:17.765 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:22:17.765 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:17.766 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:22:17.766 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:17.766 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:22:17.766 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:17.766 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:17.766 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052655", 2023-07-29T01:22:17.767 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:17.767 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:22:17.767 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:17.767 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:19:15.824401Z" 2023-07-29T01:22:17.767 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:17.767 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:17.768 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:17.768 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:22:17.768 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:17.768 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:17.768 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:17.768 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:17.769 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:22:17.769 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:17.769 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:17.769 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:17.770 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:17.770 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:17.770 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:22:17.770 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:17.771 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:17.771 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000046221", 2023-07-29T01:22:17.771 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:17.771 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:22:17.771 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:17.772 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:19:15.858505Z" 2023-07-29T01:22:17.772 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:17.772 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:17.772 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:17.773 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:22:17.773 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:17.773 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:17.774 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031774", 2023-07-29T01:22:17.774 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:17.774 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:22:17.774 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:17.774 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.812152Z" 2023-07-29T01:22:17.774 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:17.774 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:17.775 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:22:17.775 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:17.775 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:22:17.775 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:22:17.776 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:22:17.776 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:17.776 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:22:17Z" 2023-07-29T01:22:17.776 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:22:17.777 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052655", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:19:15.824401Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046221", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:19:15.858505Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.812152Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:22:17Z" } 2023-07-29T01:22:17.778 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052655', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:19:15.824401Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046221', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:19:15.858505Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.812152Z'}}] 2023-07-29T01:22:17.778 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052796', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031915'} 2023-07-29T01:22:17.778 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052655', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:22:17.779 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052796 target=00000000000000000000:00000000000000052655, shard=3 master=00000000000000000000:00000000000000031915 target=00000000000000000000:00000000000000031774 2023-07-29T01:22:22.780 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:22.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:22.782 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:22.837 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:22:22.838 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:22:22.885 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:22:22.885 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:22:22.885 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:22:22.885 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:22:22.886 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:22:22.886 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:22:22.886 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:22.886 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:22:22.886 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:22.886 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:22:22.887 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:22.887 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:22.887 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052655", 2023-07-29T01:22:22.887 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:22.887 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:22:22.887 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:22.887 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:19:15.824401Z" 2023-07-29T01:22:22.888 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:22.888 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:22.888 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:22.888 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:22:22.889 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:22.889 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:22.889 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:22.889 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:22.889 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:22:22.890 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:22.890 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:22.890 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:22.890 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:22.891 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:22.891 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:22:22.891 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:22.891 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:22.892 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000046221", 2023-07-29T01:22:22.892 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:22.892 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:22:22.892 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:22.893 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:19:15.858505Z" 2023-07-29T01:22:22.893 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:22.893 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:22.893 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:22.894 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:22:22.894 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:22.894 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:22.894 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031774", 2023-07-29T01:22:22.895 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:22.895 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:22:22.895 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:22.895 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.812152Z" 2023-07-29T01:22:22.896 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:22.896 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:22.896 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:22:22.896 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:22.897 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:22:22.897 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:22:22.897 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:22:22.897 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:22.898 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:22:22Z" 2023-07-29T01:22:22.898 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:22:22.899 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052655", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:19:15.824401Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046221", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:19:15.858505Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.812152Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:22:22Z" } 2023-07-29T01:22:22.900 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052655', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:19:15.824401Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046221', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:19:15.858505Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.812152Z'}}] 2023-07-29T01:22:22.901 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052796', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031915'} 2023-07-29T01:22:22.901 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052655', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:22:22.902 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052796 target=00000000000000000000:00000000000000052655, shard=3 master=00000000000000000000:00000000000000031915 target=00000000000000000000:00000000000000031774 2023-07-29T01:22:27.903 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:27.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:27.905 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:27.959 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:22:27.960 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:22:28.008 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:22:28.008 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:22:28.008 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:22:28.009 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:22:28.009 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:22:28.009 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:22:28.009 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:28.009 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:22:28.009 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:28.009 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:22:28.010 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:28.010 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:28.010 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052655", 2023-07-29T01:22:28.010 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:28.010 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:22:28.010 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:28.011 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:19:15.824401Z" 2023-07-29T01:22:28.011 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:28.011 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:28.011 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:28.011 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:22:28.011 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:28.011 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:28.012 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:28.012 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:28.012 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:22:28.012 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:28.012 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:28.012 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:28.012 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:28.013 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:28.013 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:22:28.013 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:28.013 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:28.013 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000046221", 2023-07-29T01:22:28.013 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:28.013 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:22:28.014 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:28.014 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:19:15.858505Z" 2023-07-29T01:22:28.014 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:28.014 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:28.014 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:28.014 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:22:28.014 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:28.015 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:28.015 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031774", 2023-07-29T01:22:28.015 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:28.015 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:22:28.015 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:28.015 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.812152Z" 2023-07-29T01:22:28.016 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:28.016 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:28.016 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:22:28.016 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:28.016 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:22:28.016 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:22:28.016 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:22:28.017 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:28.017 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:22:27Z" 2023-07-29T01:22:28.017 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:22:28.018 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052655", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:19:15.824401Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046221", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:19:15.858505Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.812152Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:22:27Z" } 2023-07-29T01:22:28.018 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052655', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:19:15.824401Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046221', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:19:15.858505Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.812152Z'}}] 2023-07-29T01:22:28.019 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052796', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031915'} 2023-07-29T01:22:28.019 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052655', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:22:28.019 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052796 target=00000000000000000000:00000000000000052655, shard=3 master=00000000000000000000:00000000000000031915 target=00000000000000000000:00000000000000031774 2023-07-29T01:22:33.020 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:33.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:33.021 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:33.076 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:22:33.076 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:22:33.124 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:22:33.125 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:22:33.125 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:22:33.125 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:22:33.125 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:22:33.125 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:22:33.125 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:33.125 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:22:33.126 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:33.126 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:22:33.126 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:33.126 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:33.126 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052655", 2023-07-29T01:22:33.126 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:33.127 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:22:33.127 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:33.127 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:19:15.824401Z" 2023-07-29T01:22:33.127 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:33.127 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:33.128 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:33.128 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:22:33.128 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:33.128 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:33.128 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:33.128 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:33.128 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:22:33.129 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:33.129 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:33.129 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:33.129 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:33.129 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:33.129 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:22:33.129 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:33.130 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:33.130 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000046221", 2023-07-29T01:22:33.130 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:33.130 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:22:33.131 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:33.131 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:19:15.858505Z" 2023-07-29T01:22:33.131 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:33.131 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:33.131 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:33.131 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:22:33.131 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:33.132 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:33.132 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031774", 2023-07-29T01:22:33.132 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:33.132 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:22:33.132 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:33.133 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.812152Z" 2023-07-29T01:22:33.133 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:33.133 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:33.133 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:22:33.133 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:33.133 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:22:33.133 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:22:33.134 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:22:33.134 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:33.134 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:22:33Z" 2023-07-29T01:22:33.134 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:22:33.135 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052655", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:19:15.824401Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046221", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:19:15.858505Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.812152Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:22:33Z" } 2023-07-29T01:22:33.135 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052655', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:19:15.824401Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046221', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:19:15.858505Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.812152Z'}}] 2023-07-29T01:22:33.136 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052796', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031915'} 2023-07-29T01:22:33.136 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052655', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:22:33.136 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052796 target=00000000000000000000:00000000000000052655, shard=3 master=00000000000000000000:00000000000000031915 target=00000000000000000000:00000000000000031774 2023-07-29T01:22:38.138 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:38.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:38.139 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:38.194 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:22:38.194 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:22:38.242 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:22:38.243 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:22:38.243 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:22:38.243 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:22:38.243 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:22:38.244 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:22:38.244 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:38.244 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:22:38.244 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:38.244 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:22:38.244 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:38.244 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:38.245 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052655", 2023-07-29T01:22:38.245 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:38.245 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:22:38.245 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:38.245 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:19:15.824401Z" 2023-07-29T01:22:38.245 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:38.245 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:38.246 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:38.246 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:22:38.246 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:38.246 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:38.246 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:38.246 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:38.247 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:22:38.247 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:38.247 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:38.247 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:38.247 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:38.247 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:38.247 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:22:38.248 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:38.248 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:38.248 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000046221", 2023-07-29T01:22:38.248 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:38.248 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:22:38.248 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:38.249 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:19:15.858505Z" 2023-07-29T01:22:38.249 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:38.249 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:38.249 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:38.249 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:22:38.249 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:38.249 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:38.250 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031774", 2023-07-29T01:22:38.250 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:38.250 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:22:38.250 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:38.250 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.812152Z" 2023-07-29T01:22:38.250 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:38.250 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:38.251 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:22:38.251 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:38.251 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:22:38.251 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:22:38.251 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:22:38.251 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:38.251 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:22:38Z" 2023-07-29T01:22:38.252 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:22:38.252 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052655", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:19:15.824401Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046221", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:19:15.858505Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.812152Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:22:38Z" } 2023-07-29T01:22:38.253 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052655', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:19:15.824401Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046221', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:19:15.858505Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.812152Z'}}] 2023-07-29T01:22:38.253 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052796', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031915'} 2023-07-29T01:22:38.254 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052655', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:22:38.254 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052796 target=00000000000000000000:00000000000000052655, shard=3 master=00000000000000000000:00000000000000031915 target=00000000000000000000:00000000000000031774 2023-07-29T01:22:43.256 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:43.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:43.257 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:43.312 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:22:43.312 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:22:43.360 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:22:43.360 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:22:43.360 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:22:43.361 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:22:43.361 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:22:43.361 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:22:43.361 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:43.361 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:22:43.361 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:43.361 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:22:43.362 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:43.362 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:43.362 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052655", 2023-07-29T01:22:43.362 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:43.362 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:22:43.362 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:43.363 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:19:15.824401Z" 2023-07-29T01:22:43.363 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:43.363 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:43.363 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:43.363 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:22:43.363 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:43.363 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:43.364 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:43.364 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:43.364 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:22:43.364 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:43.364 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:43.364 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:43.364 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:43.365 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:43.365 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:22:43.365 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:43.365 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:43.365 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000046221", 2023-07-29T01:22:43.365 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:43.365 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:22:43.366 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:43.366 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:19:15.858505Z" 2023-07-29T01:22:43.366 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:43.366 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:43.366 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:43.366 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:22:43.366 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:43.367 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:43.367 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031774", 2023-07-29T01:22:43.367 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:43.367 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:22:43.367 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:43.367 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.812152Z" 2023-07-29T01:22:43.367 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:43.368 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:43.368 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:22:43.368 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:43.368 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:22:43.368 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:22:43.368 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:22:43.369 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:43.369 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:22:43Z" 2023-07-29T01:22:43.369 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:22:43.370 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052655", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:19:15.824401Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046221", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:19:15.858505Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.812152Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:22:43Z" } 2023-07-29T01:22:43.370 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052655', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:19:15.824401Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046221', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:19:15.858505Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.812152Z'}}] 2023-07-29T01:22:43.371 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052796', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031915'} 2023-07-29T01:22:43.371 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052655', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:22:43.371 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052796 target=00000000000000000000:00000000000000052655, shard=3 master=00000000000000000000:00000000000000031915 target=00000000000000000000:00000000000000031774 2023-07-29T01:22:48.372 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:48.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:48.373 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:48.427 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:22:48.428 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:22:48.476 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:22:48.476 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:22:48.476 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:22:48.476 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:22:48.476 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:22:48.477 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:22:48.477 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:48.477 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:22:48.477 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:48.477 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:22:48.477 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:48.478 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:48.478 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052655", 2023-07-29T01:22:48.478 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:48.478 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:22:48.478 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:48.478 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:19:15.824401Z" 2023-07-29T01:22:48.478 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:48.479 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:48.479 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:48.479 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:22:48.479 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:48.479 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:48.479 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:48.479 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:48.480 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:22:48.480 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:48.480 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:48.480 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:48.480 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:48.480 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:48.480 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:22:48.481 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:48.481 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:48.481 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000046221", 2023-07-29T01:22:48.481 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:48.481 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:22:48.481 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:48.481 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:19:15.858505Z" 2023-07-29T01:22:48.482 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:48.482 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:48.482 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:48.482 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:22:48.482 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:48.482 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:48.483 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031774", 2023-07-29T01:22:48.483 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:48.483 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:22:48.483 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:48.483 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:17:10.812152Z" 2023-07-29T01:22:48.483 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:48.483 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:48.484 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:22:48.484 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:48.484 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:22:48.484 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:22:48.484 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:22:48.484 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:48.484 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:22:48Z" 2023-07-29T01:22:48.485 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:22:48.485 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052655", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:19:15.824401Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046221", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:19:15.858505Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031774", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:17:10.812152Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:22:48Z" } 2023-07-29T01:22:48.486 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052655', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:19:15.824401Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046221', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:19:15.858505Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.812152Z'}}] 2023-07-29T01:22:48.486 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052796', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031915'} 2023-07-29T01:22:48.487 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052655', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:22:48.487 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052796 target=00000000000000000000:00000000000000052655, shard=3 master=00000000000000000000:00000000000000031915 target=00000000000000000000:00000000000000031774 2023-07-29T01:22:53.488 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:53.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:53.489 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:53.545 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:22:53.546 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:22:53.593 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-29T01:22:53.594 INFO:teuthology.orchestra.run.smithi101.stdout: "sync_status": { 2023-07-29T01:22:53.594 INFO:teuthology.orchestra.run.smithi101.stdout: "info": { 2023-07-29T01:22:53.594 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "sync", 2023-07-29T01:22:53.594 INFO:teuthology.orchestra.run.smithi101.stdout: "num_shards": 4, 2023-07-29T01:22:53.594 INFO:teuthology.orchestra.run.smithi101.stdout: "instance_id": 7555809319554727162 2023-07-29T01:22:53.595 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:53.595 INFO:teuthology.orchestra.run.smithi101.stdout: "markers": [ 2023-07-29T01:22:53.595 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:53.595 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 0, 2023-07-29T01:22:53.595 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:53.595 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:53.596 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000052796", 2023-07-29T01:22:53.596 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:53.596 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 55, 2023-07-29T01:22:53.596 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:53.596 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:21:26.070114Z" 2023-07-29T01:22:53.596 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:53.596 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:53.597 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:53.597 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 1, 2023-07-29T01:22:53.597 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:53.597 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:53.597 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:53.597 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:53.597 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 9, 2023-07-29T01:22:53.598 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:53.598 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:53.598 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:53.598 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:53.598 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:53.598 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 2, 2023-07-29T01:22:53.598 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:53.599 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:53.599 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000046221", 2023-07-29T01:22:53.599 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:53.599 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:22:53.599 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:53.599 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:19:15.858505Z" 2023-07-29T01:22:53.599 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:53.600 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:53.600 INFO:teuthology.orchestra.run.smithi101.stdout: { 2023-07-29T01:22:53.600 INFO:teuthology.orchestra.run.smithi101.stdout: "key": 3, 2023-07-29T01:22:53.600 INFO:teuthology.orchestra.run.smithi101.stdout: "val": { 2023-07-29T01:22:53.600 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "incremental-sync", 2023-07-29T01:22:53.600 INFO:teuthology.orchestra.run.smithi101.stdout: "marker": "00000000000000000000:00000000000000031915", 2023-07-29T01:22:53.601 INFO:teuthology.orchestra.run.smithi101.stdout: "next_step_marker": "", 2023-07-29T01:22:53.601 INFO:teuthology.orchestra.run.smithi101.stdout: "total_entries": 0, 2023-07-29T01:22:53.601 INFO:teuthology.orchestra.run.smithi101.stdout: "pos": 0, 2023-07-29T01:22:53.601 INFO:teuthology.orchestra.run.smithi101.stdout: "timestamp": "2023-07-29T01:21:26.091582Z" 2023-07-29T01:22:53.601 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:53.601 INFO:teuthology.orchestra.run.smithi101.stdout: } 2023-07-29T01:22:53.601 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-29T01:22:53.602 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:53.602 INFO:teuthology.orchestra.run.smithi101.stdout: "full_sync": { 2023-07-29T01:22:53.602 INFO:teuthology.orchestra.run.smithi101.stdout: "total": 64, 2023-07-29T01:22:53.602 INFO:teuthology.orchestra.run.smithi101.stdout: "complete": 64 2023-07-29T01:22:53.602 INFO:teuthology.orchestra.run.smithi101.stdout: }, 2023-07-29T01:22:53.602 INFO:teuthology.orchestra.run.smithi101.stdout: "current_time": "2023-07-29T01:22:53Z" 2023-07-29T01:22:53.602 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-29T01:22:53.603 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 7555809319554727162 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052796", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-07-29T01:21:26.070114Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045021", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-07-29T01:16:32.640326Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046221", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:19:15.858505Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031915", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-07-29T01:21:26.091582Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-07-29T01:22:53Z" } 2023-07-29T01:22:53.604 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052796', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:21:26.070114Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046221', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:19:15.858505Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031915', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:21:26.091582Z'}}] 2023-07-29T01:22:53.604 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052796', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031915'} 2023-07-29T01:22:53.604 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052796', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046221', 3: '00000000000000000000:00000000000000031915'} 2023-07-29T01:22:53.605 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-07-29T01:22:53.605 DEBUG:boto:path=/hiiqos-74/ 2023-07-29T01:22:53.605 DEBUG:boto:auth_path=/hiiqos-74/ 2023-07-29T01:22:53.606 DEBUG:boto:Method: HEAD 2023-07-29T01:22:53.606 DEBUG:boto:Path: /hiiqos-74/ 2023-07-29T01:22:53.607 DEBUG:boto:Data: 2023-07-29T01:22:53.607 DEBUG:boto:Headers: {} 2023-07-29T01:22:53.607 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:53.607 DEBUG:boto:Port: 8001 2023-07-29T01:22:53.608 DEBUG:boto:Params: {} 2023-07-29T01:22:53.608 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-29T01:22:53.609 DEBUG:boto:Token: None 2023-07-29T01:22:53.609 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:22:53 GMT /hiiqos-74/ 2023-07-29T01:22:53.610 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:mlAaNItjQNaDzgAt7835JC7eKSM= 2023-07-29T01:22:53.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:22:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mlAaNItjQNaDzgAt7835JC7eKSM=', 'Content-Length': '0'} 2023-07-29T01:22:53.615 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f96d27c674ebe4dc-0064c469ed-2ca4-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:53.615 DEBUG:boto:path=/hiiqos-74/obj3 2023-07-29T01:22:53.616 DEBUG:boto:auth_path=/hiiqos-74/obj3 2023-07-29T01:22:53.616 DEBUG:boto:Method: HEAD 2023-07-29T01:22:53.616 DEBUG:boto:Path: /hiiqos-74/obj3 2023-07-29T01:22:53.617 DEBUG:boto:Data: 2023-07-29T01:22:53.617 DEBUG:boto:Headers: {} 2023-07-29T01:22:53.617 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:53.618 DEBUG:boto:Port: 8001 2023-07-29T01:22:53.618 DEBUG:boto:Params: {} 2023-07-29T01:22:53.619 DEBUG:boto:Token: None 2023-07-29T01:22:53.619 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:22:53 GMT /hiiqos-74/obj3 2023-07-29T01:22:53.619 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:AO+rlJxMXOmxdAFMu1eTyFfW2+c= 2023-07-29T01:22:53.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:22:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:AO+rlJxMXOmxdAFMu1eTyFfW2+c=', 'Content-Length': '0'} 2023-07-29T01:22:53.622 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:21:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-74:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.10875.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000465407f2d45cb254-0064c469ed-2ca4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:53.623 DEBUG:boto:path=/hiiqos-74/obj3 2023-07-29T01:22:53.624 DEBUG:boto:auth_path=/hiiqos-74/obj3 2023-07-29T01:22:53.625 DEBUG:boto:Method: GET 2023-07-29T01:22:53.626 DEBUG:boto:Path: /hiiqos-74/obj3 2023-07-29T01:22:53.627 DEBUG:boto:Data: 2023-07-29T01:22:53.628 DEBUG:boto:Headers: {} 2023-07-29T01:22:53.629 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:53.629 DEBUG:boto:Port: 8001 2023-07-29T01:22:53.630 DEBUG:boto:Params: {} 2023-07-29T01:22:53.630 DEBUG:boto:Token: None 2023-07-29T01:22:53.631 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 01:22:53 GMT /hiiqos-74/obj3 2023-07-29T01:22:53.632 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:T1eSbNvJneA3pbrpeRVDGHRzXl8= 2023-07-29T01:22:53.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:22:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:T1eSbNvJneA3pbrpeRVDGHRzXl8=', 'Content-Length': '0'} 2023-07-29T01:22:53.635 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:21:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-74:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.10875.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001abd5ee5a24d7d90-0064c469ed-2ca4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:53.635 DEBUG:boto:path=/hiiqos-74/obj4 2023-07-29T01:22:53.636 DEBUG:boto:auth_path=/hiiqos-74/obj4 2023-07-29T01:22:53.636 DEBUG:boto:Method: HEAD 2023-07-29T01:22:53.637 DEBUG:boto:Path: /hiiqos-74/obj4 2023-07-29T01:22:53.637 DEBUG:boto:Data: 2023-07-29T01:22:53.637 DEBUG:boto:Headers: {} 2023-07-29T01:22:53.638 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:53.638 DEBUG:boto:Port: 8001 2023-07-29T01:22:53.638 DEBUG:boto:Params: {} 2023-07-29T01:22:53.639 DEBUG:boto:Token: None 2023-07-29T01:22:53.639 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:22:53 GMT /hiiqos-74/obj4 2023-07-29T01:22:53.640 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:f1egKlDvI00Zo++8Wk3iwsPeS1I= 2023-07-29T01:22:53.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:22:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:f1egKlDvI00Zo++8Wk3iwsPeS1I=', 'Content-Length': '0'} 2023-07-29T01:22:53.642 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:21:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-74:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.10875.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050e034d89bdb69a2-0064c469ed-2ca4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:53.643 DEBUG:boto:path=/hiiqos-74/obj4 2023-07-29T01:22:53.643 DEBUG:boto:auth_path=/hiiqos-74/obj4 2023-07-29T01:22:53.644 DEBUG:boto:Method: GET 2023-07-29T01:22:53.645 DEBUG:boto:Path: /hiiqos-74/obj4 2023-07-29T01:22:53.645 DEBUG:boto:Data: 2023-07-29T01:22:53.646 DEBUG:boto:Headers: {} 2023-07-29T01:22:53.647 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:53.647 DEBUG:boto:Port: 8001 2023-07-29T01:22:53.648 DEBUG:boto:Params: {} 2023-07-29T01:22:53.649 DEBUG:boto:Token: None 2023-07-29T01:22:53.649 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 01:22:53 GMT /hiiqos-74/obj4 2023-07-29T01:22:53.650 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:EAJQ1u6w2sJOcYPdZkPVsXyvirA= 2023-07-29T01:22:53.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:22:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EAJQ1u6w2sJOcYPdZkPVsXyvirA=', 'Content-Length': '0'} 2023-07-29T01:22:53.652 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:21:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-74:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.10875.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000011faea9918946f51-0064c469ed-2ca4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:53.653 DEBUG:boto:path=/hiiqos-75/ 2023-07-29T01:22:53.653 DEBUG:boto:auth_path=/hiiqos-75/ 2023-07-29T01:22:53.653 DEBUG:boto:Method: HEAD 2023-07-29T01:22:53.654 DEBUG:boto:Path: /hiiqos-75/ 2023-07-29T01:22:53.654 DEBUG:boto:Data: 2023-07-29T01:22:53.654 DEBUG:boto:Headers: {} 2023-07-29T01:22:53.655 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:53.655 DEBUG:boto:Port: 8001 2023-07-29T01:22:53.655 DEBUG:boto:Params: {} 2023-07-29T01:22:53.656 DEBUG:boto:Token: None 2023-07-29T01:22:53.656 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:22:53 GMT /hiiqos-75/ 2023-07-29T01:22:53.656 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:BMLv6N9ySpFc3c2d9OeMVgAveIQ= 2023-07-29T01:22:53.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:22:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BMLv6N9ySpFc3c2d9OeMVgAveIQ=', 'Content-Length': '0'} 2023-07-29T01:22:53.659 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef212d0819a5e3bb-0064c469ed-2ca4-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:53.660 DEBUG:boto:path=/hiiqos-75/obj3 2023-07-29T01:22:53.660 DEBUG:boto:auth_path=/hiiqos-75/obj3 2023-07-29T01:22:53.660 DEBUG:boto:Method: HEAD 2023-07-29T01:22:53.661 DEBUG:boto:Path: /hiiqos-75/obj3 2023-07-29T01:22:53.661 DEBUG:boto:Data: 2023-07-29T01:22:53.661 DEBUG:boto:Headers: {} 2023-07-29T01:22:53.662 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:53.662 DEBUG:boto:Port: 8001 2023-07-29T01:22:53.662 DEBUG:boto:Params: {} 2023-07-29T01:22:53.663 DEBUG:boto:Token: None 2023-07-29T01:22:53.663 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:22:53 GMT /hiiqos-75/obj3 2023-07-29T01:22:53.663 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:Gysy+XUJ99N5yG6ji+Zs7oH0jO0= 2023-07-29T01:22:53.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:22:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Gysy+XUJ99N5yG6ji+Zs7oH0jO0=', 'Content-Length': '0'} 2023-07-29T01:22:53.665 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:21:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-74:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.10875.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001f21c390c6d61c7f-0064c469ed-2ca4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:53.666 DEBUG:boto:path=/hiiqos-75/obj3 2023-07-29T01:22:53.666 DEBUG:boto:auth_path=/hiiqos-75/obj3 2023-07-29T01:22:53.667 DEBUG:boto:Method: GET 2023-07-29T01:22:53.667 DEBUG:boto:Path: /hiiqos-75/obj3 2023-07-29T01:22:53.667 DEBUG:boto:Data: 2023-07-29T01:22:53.668 DEBUG:boto:Headers: {} 2023-07-29T01:22:53.668 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:53.669 DEBUG:boto:Port: 8001 2023-07-29T01:22:53.669 DEBUG:boto:Params: {} 2023-07-29T01:22:53.669 DEBUG:boto:Token: None 2023-07-29T01:22:53.670 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 01:22:53 GMT /hiiqos-75/obj3 2023-07-29T01:22:53.670 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:G+oZAICY863Hgw70zboCWmwCKPk= 2023-07-29T01:22:53.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:22:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:G+oZAICY863Hgw70zboCWmwCKPk=', 'Content-Length': '0'} 2023-07-29T01:22:53.672 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:21:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-74:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.10875.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b156a3f398f3c1fa-0064c469ed-2ca4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:53.673 DEBUG:boto:path=/hiiqos-75/obj4 2023-07-29T01:22:53.673 DEBUG:boto:auth_path=/hiiqos-75/obj4 2023-07-29T01:22:53.674 DEBUG:boto:Method: HEAD 2023-07-29T01:22:53.674 DEBUG:boto:Path: /hiiqos-75/obj4 2023-07-29T01:22:53.674 DEBUG:boto:Data: 2023-07-29T01:22:53.675 DEBUG:boto:Headers: {} 2023-07-29T01:22:53.675 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:53.675 DEBUG:boto:Port: 8001 2023-07-29T01:22:53.676 DEBUG:boto:Params: {} 2023-07-29T01:22:53.676 DEBUG:boto:Token: None 2023-07-29T01:22:53.676 DEBUG:boto:StringToSign: HEAD Sat, 29 Jul 2023 01:22:53 GMT /hiiqos-75/obj4 2023-07-29T01:22:53.677 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:bbsAzEWN8tYFk1eObytB/rQFREU= 2023-07-29T01:22:53.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:22:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bbsAzEWN8tYFk1eObytB/rQFREU=', 'Content-Length': '0'} 2023-07-29T01:22:53.679 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:21:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-74:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.10875.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ccd7dd09ac8523d9-0064c469ed-2ca4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:53.680 DEBUG:boto:path=/hiiqos-75/obj4 2023-07-29T01:22:53.680 DEBUG:boto:auth_path=/hiiqos-75/obj4 2023-07-29T01:22:53.681 DEBUG:boto:Method: GET 2023-07-29T01:22:53.681 DEBUG:boto:Path: /hiiqos-75/obj4 2023-07-29T01:22:53.682 DEBUG:boto:Data: 2023-07-29T01:22:53.682 DEBUG:boto:Headers: {} 2023-07-29T01:22:53.683 DEBUG:boto:Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:53.683 DEBUG:boto:Port: 8001 2023-07-29T01:22:53.684 DEBUG:boto:Params: {} 2023-07-29T01:22:53.684 DEBUG:boto:Token: None 2023-07-29T01:22:53.685 DEBUG:boto:StringToSign: GET Sat, 29 Jul 2023 01:22:53 GMT /hiiqos-75/obj4 2023-07-29T01:22:53.686 DEBUG:boto:Signature: AWS S7W2J6K8BFBCG65TLASH:WyL/ksGyH3NskfL/ayobCPFXmqI= 2023-07-29T01:22:53.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:22:53 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WyL/ksGyH3NskfL/ayobCPFXmqI=', 'Content-Length': '0'} 2023-07-29T01:22:53.689 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:21:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-74:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.10875.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f57aefc65e3fb0d0-0064c469ed-2ca4-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:22:53 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:53.691 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'hiiqos-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:53.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'hiiqos-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:53.692 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket hiiqos-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:22:53.738 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:22:53.739 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:22:53.792 INFO:teuthology.orchestra.run.smithi101.stdout:{"sync_policy":{"groups":[]}} 2023-07-29T01:22:53.793 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:53.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:53.794 DEBUG:teuthology.orchestra.run.smithi101:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:22:53.838 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setuser ceph since I am not root 2023-07-29T01:22:53.838 INFO:teuthology.orchestra.run.smithi101.stderr:ignoring --setgroup ceph since I am not root 2023-07-29T01:22:53.883 INFO:teuthology.orchestra.run.smithi101.stdout:{"sync_policy":{"groups":[]}} 2023-07-29T01:22:53.883 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket_to_multiple: ... ok 2023-07-29T01:22:53.884 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:53.885 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-07-29T01:22:53.885 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_reshard_index_log_trim 2023-07-29T01:22:53.885 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-07-29T01:22:53.886 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-07-29T01:22:53.887 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_61140164d2579c30ffff3970615e0f82e40c6470/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-07-29T01:22:53.887 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-07-29T01:22:53.887 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_0484097b0572e96f42cf6402cbe0ac8dcb046577/qa/../src/test/rgw/rgw_multi/tests.py", line 1418, in test_bucket_reshard_index_log_trim 2023-07-29T01:22:53.888 INFO:tasks.rgw_multisite_tests: assert(len(json_obj_5['layout']['logs']) == 1) 2023-07-29T01:22:53.888 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-07-29T01:22:53.888 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-07-29T01:22:53.888 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=hiiqos-55 2023-07-29T01:22:53.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:53.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:53.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:53.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/ 2023-07-29T01:22:53.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:53.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:53.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:53.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:53.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:53.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:53.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:53.891 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:53.891 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:53.892 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:53.892 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:14:43 GMT 2023-07-29T01:22:53.892 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:53.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:53.893 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:3VTK0x/Na+wntAhkVteewKxndjk= 2023-07-29T01:22:53.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:43 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3VTK0x/Na+wntAhkVteewKxndjk=', 'Content-Length': '0'} 2023-07-29T01:22:53.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000f09f13a3750c20b6-0064c459f3-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:53.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:53.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:53.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:53.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/ 2023-07-29T01:22:53.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:53.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:53.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:53.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:53.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:53.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:53.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:53.897 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:53.897 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:53.897 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:53.898 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:14:43 GMT 2023-07-29T01:22:53.898 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:53.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:53.898 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:+kqoyPJALQOa5zDX6Bm/Q7wP5oo= 2023-07-29T01:22:53.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:43 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+kqoyPJALQOa5zDX6Bm/Q7wP5oo=', 'Content-Length': '0'} 2023-07-29T01:22:53.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000730d9cf93946d81-0064c459f3-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:53.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a 2023-07-29T01:22:53.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a 2023-07-29T01:22:53.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:53.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/a 2023-07-29T01:22:53.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:53.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:53.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:53.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:53.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:53.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:53.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:53.902 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:53.902 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:53.903 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:53.903 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:14:43 GMT 2023-07-29T01:22:53.903 INFO:tasks.rgw_multisite_tests:/hiiqos-55/a 2023-07-29T01:22:53.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:53.903 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:pewjfxGD8GkfXmgP7eBxaAKEbxM= 2023-07-29T01:22:53.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:14:43 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pewjfxGD8GkfXmgP7eBxaAKEbxM='} 2023-07-29T01:22:53.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006f8abd58bd00bfbc-0064c459f3-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:53.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:53.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:53.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:53.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/ 2023-07-29T01:22:53.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:53.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:53.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:53.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:53.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:53.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:53.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:53.907 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:53.908 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:53.908 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:53.908 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:14:44 GMT 2023-07-29T01:22:53.908 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:53.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:53.909 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:N4CsLT7ZXMxE5bLtKrnMycmgYoE= 2023-07-29T01:22:53.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:44 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:N4CsLT7ZXMxE5bLtKrnMycmgYoE=', 'Content-Length': '0'} 2023-07-29T01:22:53.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a20eef557ccca2a2-0064c459f3-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:53.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b 2023-07-29T01:22:53.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b 2023-07-29T01:22:53.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:53.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/b 2023-07-29T01:22:53.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:53.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:53.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:53.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:53.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:53.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:53.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:53.912 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:53.912 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:53.913 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:53.913 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:14:44 GMT 2023-07-29T01:22:53.913 INFO:tasks.rgw_multisite_tests:/hiiqos-55/b 2023-07-29T01:22:53.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:53.913 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:nYTRVtMAtJl6FmjI0Zq+riLs8Es= 2023-07-29T01:22:53.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:14:44 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nYTRVtMAtJl6FmjI0Zq+riLs8Es='} 2023-07-29T01:22:53.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ef207159b96557c9-0064c459f3-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:53.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:53.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:53.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:53.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/ 2023-07-29T01:22:53.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:53.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:53.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:53.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:53.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:53.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:53.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:53.917 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:53.917 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:53.917 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:53.918 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:14:44 GMT 2023-07-29T01:22:53.918 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:53.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:53.918 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:N4CsLT7ZXMxE5bLtKrnMycmgYoE= 2023-07-29T01:22:53.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:44 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:N4CsLT7ZXMxE5bLtKrnMycmgYoE=', 'Content-Length': '0'} 2023-07-29T01:22:53.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000872e0d52242325e7-0064c459f3-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:53.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c 2023-07-29T01:22:53.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c 2023-07-29T01:22:53.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:53.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/c 2023-07-29T01:22:53.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:53.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:53.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:53.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:53.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:53.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:53.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:53.922 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:53.922 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:53.922 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:53.923 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:14:44 GMT 2023-07-29T01:22:53.923 INFO:tasks.rgw_multisite_tests:/hiiqos-55/c 2023-07-29T01:22:53.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:53.923 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:TFZ3JkxiZwzLgUJxlrKnDcJ3akA= 2023-07-29T01:22:53.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:14:44 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TFZ3JkxiZwzLgUJxlrKnDcJ3akA='} 2023-07-29T01:22:53.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d5ebee5d112fe9cc-0064c459f4-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:53.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:53.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:53.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:53.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/ 2023-07-29T01:22:53.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:53.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:53.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:53.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:53.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:53.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:53.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:53.927 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:53.927 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:53.927 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:53.928 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:14:44 GMT 2023-07-29T01:22:53.928 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:53.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:53.928 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:N4CsLT7ZXMxE5bLtKrnMycmgYoE= 2023-07-29T01:22:53.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:44 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:N4CsLT7ZXMxE5bLtKrnMycmgYoE=', 'Content-Length': '0'} 2023-07-29T01:22:53.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000966df5e2409e60d1-0064c459f4-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:53.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d 2023-07-29T01:22:53.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d 2023-07-29T01:22:53.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:53.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/d 2023-07-29T01:22:53.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:53.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:53.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:53.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:53.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:53.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:53.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:53.933 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:53.933 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:53.933 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:53.933 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:14:44 GMT 2023-07-29T01:22:53.934 INFO:tasks.rgw_multisite_tests:/hiiqos-55/d 2023-07-29T01:22:53.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:53.934 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:4zJtmnnv8IpKzWYwPLr8krmyvGo= 2023-07-29T01:22:53.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:14:44 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4zJtmnnv8IpKzWYwPLr8krmyvGo='} 2023-07-29T01:22:53.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000652345e248f6894f-0064c459f4-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:53.935 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:53.935 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:53.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:53.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:53.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:53.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T01:22:53.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: [ 2023-07-29T01:22:53.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:53.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690587969.233742_99.1", 2023-07-29T01:22:53.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "last_update": "2023-07-28T23:46:09.233742Z" 2023-07-29T01:22:53.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:53.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:53.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690589683.919041_829.1", 2023-07-29T01:22:53.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "last_update": "2023-07-29T00:14:43.919041Z" 2023-07-29T01:22:53.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:53.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:53.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:53.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "last_update": "0.000000" 2023-07-29T01:22:53.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:53.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:53.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:53.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "last_update": "0.000000" 2023-07-29T01:22:53.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:53.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:53.941 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1690587969.233742_99.1', 1: '1_1690589683.919041_829.1', 2: '', 3: ''} 2023-07-29T01:22:53.941 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-07-29T01:22:53.941 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:53.941 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:53.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:53.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:53.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:53.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:14:44.238+0000 7f70cbe0f880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:53.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:14:44.239+0000 7f70cbe0f880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:53.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:53.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:53.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:53.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:53.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:53.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:53.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:53.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:53.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:53.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:53.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:53.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:53.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:53.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690587969.233742_99.1", 2023-07-29T01:22:53.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:53.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 1, 2023-07-29T01:22:53.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:53.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T01:22:53.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:53.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:53.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:53.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:53.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:53.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:53.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:53.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690589683.913457_812.1", 2023-07-29T01:22:53.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:53.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:53.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:53.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:14:43.913457Z", 2023-07-29T01:22:53.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:53.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:53.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:53.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:53.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:53.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:53.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:53.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:53.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:53.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:53.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:53.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "0.000000", 2023-07-29T01:22:53.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:53.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:53.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:53.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:53.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:53.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:53.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:53.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:53.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:53.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:53.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:53.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "0.000000", 2023-07-29T01:22:53.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:53.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:53.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:53.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:53.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:53.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:53.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 1, 2023-07-29T01:22:53.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 1 2023-07-29T01:22:53.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:53.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T00:14:44Z" 2023-07-29T01:22:53.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:53.966 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-07-29T01:22:53.966 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:53.967 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:53.967 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:53.967 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:53.967 INFO:tasks.rgw_multisite_tests: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:53.968 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:53.968 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:53.968 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:53.968 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:53.969 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:53.969 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:53.969 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:53.970 INFO:tasks.rgw_multisite_tests: "marker": "1_1690587969.233742_99.1", 2023-07-29T01:22:53.970 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:53.970 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-07-29T01:22:53.970 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:53.971 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T01:22:53.971 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:53.971 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:53.971 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:53.972 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:53.972 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:53.972 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:53.973 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:53.973 INFO:tasks.rgw_multisite_tests: "marker": "1_1690589683.913457_812.1", 2023-07-29T01:22:53.974 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:53.974 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:53.975 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:53.976 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:14:43.913457Z", 2023-07-29T01:22:53.976 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:53.977 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:53.977 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:53.978 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:53.978 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:53.979 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:53.979 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:53.979 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-07-29T01:22:53.980 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:53.980 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:53.981 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:53.981 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-07-29T01:22:53.981 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:53.982 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:53.982 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:53.982 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:53.982 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:53.983 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:53.983 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:53.983 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-07-29T01:22:53.983 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:53.984 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:53.984 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:53.984 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-07-29T01:22:53.984 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:53.985 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:53.985 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:53.985 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:53.985 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:53.985 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:53.986 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-07-29T01:22:53.986 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-07-29T01:22:53.986 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:53.986 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T00:14:44Z" 2023-07-29T01:22:53.987 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:53.987 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:53.987 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690589683.913457_812.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:14:43.913457Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T01:22:53.987 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1690587969.233742_99.1', 1: '1_1690589683.919041_829.1', 2: '', 3: ''} 2023-07-29T01:22:53.988 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690589683.913457_812.1', 2: '', 3: ''} 2023-07-29T01:22:53.988 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1690589683.919041_829.1 target=1_1690589683.913457_812.1 2023-07-29T01:22:53.988 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:53.988 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:53.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:53.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:53.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:53.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:14:49.352+0000 7f7a4f211880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:53.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:14:49.353+0000 7f7a4f211880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:53.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:53.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:53.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:53.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:53.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:53.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:53.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:53.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:53.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:53.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:53.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:53.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:53.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:53.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690587969.233742_99.1", 2023-07-29T01:22:53.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:53.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 1, 2023-07-29T01:22:53.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:53.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T01:22:53.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:53.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:53.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:53.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:53.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:53.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:53.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:53.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690589683.919041_829.1", 2023-07-29T01:22:53.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:53.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:53.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:53.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:14:43.919041Z", 2023-07-29T01:22:53.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:53.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:53.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:53.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:53.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:53.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:54.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:54.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:54.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:54.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:54.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:54.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "0.000000", 2023-07-29T01:22:54.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:54.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:54.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:54.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:54.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:54.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:54.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:54.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:54.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:54.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:54.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:54.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "0.000000", 2023-07-29T01:22:54.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:54.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:54.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:54.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:54.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:54.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:54.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 1, 2023-07-29T01:22:54.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 1 2023-07-29T01:22:54.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:54.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T00:14:49Z" 2023-07-29T01:22:54.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:54.007 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-07-29T01:22:54.007 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:54.007 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:54.007 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:54.008 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:54.008 INFO:tasks.rgw_multisite_tests: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:54.008 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:54.008 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:54.009 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:54.009 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:54.009 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:54.010 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:54.010 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:54.010 INFO:tasks.rgw_multisite_tests: "marker": "1_1690587969.233742_99.1", 2023-07-29T01:22:54.010 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:54.011 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-07-29T01:22:54.011 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:54.011 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-28T23:46:09.233742Z", 2023-07-29T01:22:54.011 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:54.011 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:54.012 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:54.012 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:54.012 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:54.013 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:54.013 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:54.013 INFO:tasks.rgw_multisite_tests: "marker": "1_1690589683.919041_829.1", 2023-07-29T01:22:54.013 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:54.014 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:54.014 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:54.014 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:14:43.919041Z", 2023-07-29T01:22:54.014 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:54.015 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:54.015 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:54.015 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:54.015 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:54.015 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:54.016 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:54.016 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-07-29T01:22:54.016 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:54.016 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:54.017 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:54.017 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-07-29T01:22:54.017 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:54.017 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:54.018 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:54.018 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:54.018 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:54.018 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:54.018 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:54.019 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-07-29T01:22:54.019 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:54.019 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:54.019 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:54.020 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-07-29T01:22:54.020 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:54.020 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:54.020 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:54.021 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:54.021 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:54.021 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:54.021 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-07-29T01:22:54.022 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-07-29T01:22:54.022 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:54.022 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T00:14:49Z" 2023-07-29T01:22:54.022 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:54.022 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.023 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690587969.233742_99.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-28T23:46:09.233742Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690589683.919041_829.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:14:43.919041Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-07-29T01:22:54.023 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1690587969.233742_99.1', 1: '1_1690589683.919041_829.1', 2: '', 3: ''} 2023-07-29T01:22:54.023 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1690587969.233742_99.1', 1: '1_1690589683.919041_829.1', 2: '', 3: ''} 2023-07-29T01:22:54.023 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-07-29T01:22:54.024 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=hiiqos-55 2023-07-29T01:22:54.024 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T01:22:54.024 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T01:22:54.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T01:22:54.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:54.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:54.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:14:49.468+0000 7fa471416880 1 waiting to reach incremental sync.. 2023-07-29T01:22:54.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:14:54.469+0000 7fa471416880 1 waiting to reach incremental sync.. 2023-07-29T01:22:54.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:14:59.471+0000 7fa471416880 1 bucket sync caught up with source: 2023-07-29T01:22:54.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: local status: [, 00000000001.56.6, , , , 00000000001.58.6, , , 00000000001.38.6, , 00000000001.23.6] 2023-07-29T01:22:54.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: remote markers: [, 00000000001.56.6, , , , 00000000001.58.6, , , 00000000001.38.6, , 00000000001.23.6] 2023-07-29T01:22:54.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:14:59.471+0000 7fa471416880 0 bucket checkpoint complete 2023-07-29T01:22:54.026 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=hiiqos-55 2023-07-29T01:22:54.027 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T01:22:54.027 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T01:22:54.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T01:22:54.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:54.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:54.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:14:59.584+0000 7f0a33610880 1 bucket sync caught up with source: 2023-07-29T01:22:54.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: local status: [, 00000000001.708.6, , , , 00000000001.707.6, , , 00000000001.382.6, , 00000000001.1165.6] 2023-07-29T01:22:54.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: remote markers: [, 00000000001.708.6, , , , 00000000001.707.6, , , 00000000001.382.6, , 00000000001.1165.6] 2023-07-29T01:22:54.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:14:59.584+0000 7f0a33610880 0 bucket checkpoint complete 2023-07-29T01:22:54.029 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=hiiqos-55 zones={a2, a1} 2023-07-29T01:22:54.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/ 2023-07-29T01:22:54.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.032 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.033 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.033 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.033 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:14:59 GMT 2023-07-29T01:22:54.033 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.034 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:1SfTQCuRQJkvt6bkf40LlGQrl50= 2023-07-29T01:22:54.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1SfTQCuRQJkvt6bkf40LlGQrl50=', 'Content-Length': '0'} 2023-07-29T01:22:54.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000084d72077fbcf1e7d-0064c45a03-1663-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/ 2023-07-29T01:22:54.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.037 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.037 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.038 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.038 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:14:59 GMT 2023-07-29T01:22:54.038 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.038 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:1SfTQCuRQJkvt6bkf40LlGQrl50= 2023-07-29T01:22:54.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1SfTQCuRQJkvt6bkf40LlGQrl50=', 'Content-Length': '0'} 2023-07-29T01:22:54.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001d804cbaae5a3ef7-0064c45a03-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.039 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2023-07-29T01:22:54.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?versions&max-keys=999 2023-07-29T01:22:54.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?versions&max-keys=999 2023-07-29T01:22:54.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?versions&max-keys=999 2023-07-29T01:22:54.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.043 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.043 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.043 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.043 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:14:59 GMT 2023-07-29T01:22:54.044 INFO:tasks.rgw_multisite_tests:/hiiqos-55/?versions 2023-07-29T01:22:54.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.045 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:bU+pz49rai6Gr+8IjSkxokZURt0= 2023-07-29T01:22:54.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bU+pz49rai6Gr+8IjSkxokZURt0=', 'Content-Length': '0'} 2023-07-29T01:22:54.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a385bce7345584b1-0064c45a03-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55999falseanulltrue2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.046 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2023-07-29T01:22:54.046 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2023-07-29T01:22:54.046 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2023-07-29T01:22:54.047 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2023-07-29T01:22:54.047 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2023-07-29T01:22:54.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?versions&max-keys=999 2023-07-29T01:22:54.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?versions&max-keys=999 2023-07-29T01:22:54.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?versions&max-keys=999 2023-07-29T01:22:54.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.051 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.051 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.052 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.052 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:14:59 GMT 2023-07-29T01:22:54.052 INFO:tasks.rgw_multisite_tests:/hiiqos-55/?versions 2023-07-29T01:22:54.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.053 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:bU+pz49rai6Gr+8IjSkxokZURt0= 2023-07-29T01:22:54.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bU+pz49rai6Gr+8IjSkxokZURt0=', 'Content-Length': '0'} 2023-07-29T01:22:54.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086a14adb73f04b96-0064c45a03-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55999falseanulltrue2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.054 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2023-07-29T01:22:54.055 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2023-07-29T01:22:54.055 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2023-07-29T01:22:54.055 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2023-07-29T01:22:54.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?versions&max-keys=999 2023-07-29T01:22:54.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?versions&max-keys=999 2023-07-29T01:22:54.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?versions&max-keys=999 2023-07-29T01:22:54.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.058 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.059 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.059 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.059 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:14:59 GMT 2023-07-29T01:22:54.059 INFO:tasks.rgw_multisite_tests:/hiiqos-55/?versions 2023-07-29T01:22:54.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.060 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:bU+pz49rai6Gr+8IjSkxokZURt0= 2023-07-29T01:22:54.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bU+pz49rai6Gr+8IjSkxokZURt0=', 'Content-Length': '0'} 2023-07-29T01:22:54.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae92abfe846bb352-0064c45a03-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55999falseanulltrue2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?versions&max-keys=999 2023-07-29T01:22:54.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?versions&max-keys=999 2023-07-29T01:22:54.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?versions&max-keys=999 2023-07-29T01:22:54.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.064 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.064 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.065 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.065 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:14:59 GMT 2023-07-29T01:22:54.065 INFO:tasks.rgw_multisite_tests:/hiiqos-55/?versions 2023-07-29T01:22:54.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.066 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:bU+pz49rai6Gr+8IjSkxokZURt0= 2023-07-29T01:22:54.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bU+pz49rai6Gr+8IjSkxokZURt0=', 'Content-Length': '0'} 2023-07-29T01:22:54.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8b9f0918bddcfd2-0064c45a03-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55999falseanulltrue2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.067 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-07-29T01:22:54.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a 2023-07-29T01:22:54.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a 2023-07-29T01:22:54.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a?versionId=null 2023-07-29T01:22:54.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a?versionId=null 2023-07-29T01:22:54.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/a?versionId=null 2023-07-29T01:22:54.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.070 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.071 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.071 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.071 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:14:59 GMT 2023-07-29T01:22:54.071 INFO:tasks.rgw_multisite_tests:/hiiqos-55/a?versionId=null 2023-07-29T01:22:54.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.072 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:EYPfq7fKj53U2CLFCjFVFRG/mn4= 2023-07-29T01:22:54.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EYPfq7fKj53U2CLFCjFVFRG/mn4=', 'Content-Length': '0'} 2023-07-29T01:22:54.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7cc8bde0cb67603-0064c45a03-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a 2023-07-29T01:22:54.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a 2023-07-29T01:22:54.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a?versionId=null 2023-07-29T01:22:54.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a?versionId=null 2023-07-29T01:22:54.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/a?versionId=null 2023-07-29T01:22:54.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.079 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.079 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.080 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.080 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:14:59 GMT 2023-07-29T01:22:54.081 INFO:tasks.rgw_multisite_tests:/hiiqos-55/a?versionId=null 2023-07-29T01:22:54.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.081 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:EYPfq7fKj53U2CLFCjFVFRG/mn4= 2023-07-29T01:22:54.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EYPfq7fKj53U2CLFCjFVFRG/mn4=', 'Content-Length': '0'} 2023-07-29T01:22:54.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000127c742d076cf09e-0064c45a03-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a 2023-07-29T01:22:54.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a 2023-07-29T01:22:54.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a?versionId=null 2023-07-29T01:22:54.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a?versionId=null 2023-07-29T01:22:54.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/a?versionId=null 2023-07-29T01:22:54.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.087 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.087 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.087 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.088 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:14:59 GMT 2023-07-29T01:22:54.088 INFO:tasks.rgw_multisite_tests:/hiiqos-55/a?versionId=null 2023-07-29T01:22:54.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.088 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:rFL5sGpu4O7Gul3f3mTkl+UIxCY= 2023-07-29T01:22:54.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rFL5sGpu4O7Gul3f3mTkl+UIxCY=', 'Content-Length': '0'} 2023-07-29T01:22:54.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073ce0190e45b65f1-0064c45a03-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a 2023-07-29T01:22:54.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a 2023-07-29T01:22:54.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a?versionId=null 2023-07-29T01:22:54.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a?versionId=null 2023-07-29T01:22:54.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/a?versionId=null 2023-07-29T01:22:54.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.093 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.094 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.094 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.095 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:14:59 GMT 2023-07-29T01:22:54.095 INFO:tasks.rgw_multisite_tests:/hiiqos-55/a?versionId=null 2023-07-29T01:22:54.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.096 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:rFL5sGpu4O7Gul3f3mTkl+UIxCY= 2023-07-29T01:22:54.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rFL5sGpu4O7Gul3f3mTkl+UIxCY=', 'Content-Length': '0'} 2023-07-29T01:22:54.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099d5dca667bd64f7-0064c45a03-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.098 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-07-29T01:22:54.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a 2023-07-29T01:22:54.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a 2023-07-29T01:22:54.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a?versionId=null 2023-07-29T01:22:54.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a?versionId=null 2023-07-29T01:22:54.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/a?versionId=null 2023-07-29T01:22:54.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.105 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.105 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.106 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.106 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:14:59 GMT 2023-07-29T01:22:54.106 INFO:tasks.rgw_multisite_tests:/hiiqos-55/a?versionId=null 2023-07-29T01:22:54.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.107 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:EYPfq7fKj53U2CLFCjFVFRG/mn4= 2023-07-29T01:22:54.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EYPfq7fKj53U2CLFCjFVFRG/mn4=', 'Content-Length': '0'} 2023-07-29T01:22:54.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f8f6c6d63f3bef3-0064c45a03-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a 2023-07-29T01:22:54.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a 2023-07-29T01:22:54.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a?versionId=null 2023-07-29T01:22:54.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a?versionId=null 2023-07-29T01:22:54.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/a?versionId=null 2023-07-29T01:22:54.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.111 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.112 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.112 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.112 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:14:59 GMT 2023-07-29T01:22:54.113 INFO:tasks.rgw_multisite_tests:/hiiqos-55/a?versionId=null 2023-07-29T01:22:54.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.113 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:EYPfq7fKj53U2CLFCjFVFRG/mn4= 2023-07-29T01:22:54.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EYPfq7fKj53U2CLFCjFVFRG/mn4=', 'Content-Length': '0'} 2023-07-29T01:22:54.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e27a4a880ba3ff9-0064c45a03-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=a 2023-07-29T01:22:54.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=a 2023-07-29T01:22:54.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=a 2023-07-29T01:22:54.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.117 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.117 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.117 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.118 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:14:59 GMT 2023-07-29T01:22:54.118 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.118 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:t/PJ+bAhtkXlQy9TiCoPZfTxBw0= 2023-07-29T01:22:54.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t/PJ+bAhtkXlQy9TiCoPZfTxBw0=', 'Content-Length': '0'} 2023-07-29T01:22:54.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2f7cd5689b869dc-0064c45a03-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55a1000falsea2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=a 2023-07-29T01:22:54.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=a 2023-07-29T01:22:54.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=a 2023-07-29T01:22:54.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.124 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.124 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.125 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.125 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:14:59 GMT 2023-07-29T01:22:54.126 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.126 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:t/PJ+bAhtkXlQy9TiCoPZfTxBw0= 2023-07-29T01:22:54.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t/PJ+bAhtkXlQy9TiCoPZfTxBw0=', 'Content-Length': '0'} 2023-07-29T01:22:54.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b93c7218f9de916b-0064c45a03-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55a1000falsea2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a 2023-07-29T01:22:54.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a 2023-07-29T01:22:54.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/a 2023-07-29T01:22:54.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.133 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.133 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.133 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.134 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:14:59 GMT 2023-07-29T01:22:54.134 INFO:tasks.rgw_multisite_tests:/hiiqos-55/a 2023-07-29T01:22:54.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.135 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:NjbT39xb6ixLxCtixw8Gub03bq8= 2023-07-29T01:22:54.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:NjbT39xb6ixLxCtixw8Gub03bq8=', 'Content-Length': '0'} 2023-07-29T01:22:54.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb09aca9042a377c-0064c45a03-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a 2023-07-29T01:22:54.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a 2023-07-29T01:22:54.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/a 2023-07-29T01:22:54.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.138 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.139 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.139 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.139 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:14:59 GMT 2023-07-29T01:22:54.139 INFO:tasks.rgw_multisite_tests:/hiiqos-55/a 2023-07-29T01:22:54.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.140 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:NjbT39xb6ixLxCtixw8Gub03bq8= 2023-07-29T01:22:54.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:NjbT39xb6ixLxCtixw8Gub03bq8=', 'Content-Length': '0'} 2023-07-29T01:22:54.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b64393f344675ca-0064c45a03-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.141 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-07-29T01:22:54.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a 2023-07-29T01:22:54.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a 2023-07-29T01:22:54.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/a 2023-07-29T01:22:54.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.144 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.144 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.145 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.145 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:14:59 GMT 2023-07-29T01:22:54.145 INFO:tasks.rgw_multisite_tests:/hiiqos-55/a 2023-07-29T01:22:54.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.146 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:yDoxt0Zmh2uge4Mq0f1WODESDrM= 2023-07-29T01:22:54.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yDoxt0Zmh2uge4Mq0f1WODESDrM=', 'Content-Length': '0'} 2023-07-29T01:22:54.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2d14ed4ebdadf3e-0064c45a03-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a 2023-07-29T01:22:54.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a 2023-07-29T01:22:54.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/a 2023-07-29T01:22:54.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.149 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.150 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.150 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.150 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:14:59 GMT 2023-07-29T01:22:54.150 INFO:tasks.rgw_multisite_tests:/hiiqos-55/a 2023-07-29T01:22:54.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.151 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:yDoxt0Zmh2uge4Mq0f1WODESDrM= 2023-07-29T01:22:54.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yDoxt0Zmh2uge4Mq0f1WODESDrM=', 'Content-Length': '0'} 2023-07-29T01:22:54.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ad3c4665eb7f8c4-0064c45a03-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=a 2023-07-29T01:22:54.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=a 2023-07-29T01:22:54.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=a 2023-07-29T01:22:54.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.155 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.155 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.155 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.155 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:14:59 GMT 2023-07-29T01:22:54.156 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.156 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:t/PJ+bAhtkXlQy9TiCoPZfTxBw0= 2023-07-29T01:22:54.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t/PJ+bAhtkXlQy9TiCoPZfTxBw0=', 'Content-Length': '0'} 2023-07-29T01:22:54.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000599a42145668f9ee-0064c45a03-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55a1000falsea2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=a 2023-07-29T01:22:54.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=a 2023-07-29T01:22:54.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=a 2023-07-29T01:22:54.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.161 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.161 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.161 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.161 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:14:59 GMT 2023-07-29T01:22:54.162 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.162 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:t/PJ+bAhtkXlQy9TiCoPZfTxBw0= 2023-07-29T01:22:54.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t/PJ+bAhtkXlQy9TiCoPZfTxBw0=', 'Content-Length': '0'} 2023-07-29T01:22:54.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000539fe7f7ffee520f-0064c45a03-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55a1000falsea2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.163 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-07-29T01:22:54.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b 2023-07-29T01:22:54.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b 2023-07-29T01:22:54.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b?versionId=null 2023-07-29T01:22:54.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b?versionId=null 2023-07-29T01:22:54.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/b?versionId=null 2023-07-29T01:22:54.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.166 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.166 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.167 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.167 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:14:59 GMT 2023-07-29T01:22:54.167 INFO:tasks.rgw_multisite_tests:/hiiqos-55/b?versionId=null 2023-07-29T01:22:54.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.168 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:K3jsoITgiOh1WpdTHXI210MlBVg= 2023-07-29T01:22:54.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K3jsoITgiOh1WpdTHXI210MlBVg=', 'Content-Length': '0'} 2023-07-29T01:22:54.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4e36fe16c02268a-0064c45a03-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b 2023-07-29T01:22:54.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b 2023-07-29T01:22:54.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b?versionId=null 2023-07-29T01:22:54.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b?versionId=null 2023-07-29T01:22:54.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/b?versionId=null 2023-07-29T01:22:54.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.171 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.171 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.172 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.172 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:14:59 GMT 2023-07-29T01:22:54.172 INFO:tasks.rgw_multisite_tests:/hiiqos-55/b?versionId=null 2023-07-29T01:22:54.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.173 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:K3jsoITgiOh1WpdTHXI210MlBVg= 2023-07-29T01:22:54.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K3jsoITgiOh1WpdTHXI210MlBVg=', 'Content-Length': '0'} 2023-07-29T01:22:54.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009695d293b21300c9-0064c45a03-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b 2023-07-29T01:22:54.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b 2023-07-29T01:22:54.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b?versionId=null 2023-07-29T01:22:54.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b?versionId=null 2023-07-29T01:22:54.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/b?versionId=null 2023-07-29T01:22:54.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.176 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.177 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.177 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.177 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:14:59 GMT 2023-07-29T01:22:54.177 INFO:tasks.rgw_multisite_tests:/hiiqos-55/b?versionId=null 2023-07-29T01:22:54.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.178 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:utXFtLpeF/logqu0Gi5UycTBPUs= 2023-07-29T01:22:54.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:utXFtLpeF/logqu0Gi5UycTBPUs=', 'Content-Length': '0'} 2023-07-29T01:22:54.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b85203c33fab547-0064c45a03-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b 2023-07-29T01:22:54.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b 2023-07-29T01:22:54.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b?versionId=null 2023-07-29T01:22:54.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b?versionId=null 2023-07-29T01:22:54.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/b?versionId=null 2023-07-29T01:22:54.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.181 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.182 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.182 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.182 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:14:59 GMT 2023-07-29T01:22:54.182 INFO:tasks.rgw_multisite_tests:/hiiqos-55/b?versionId=null 2023-07-29T01:22:54.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.183 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:utXFtLpeF/logqu0Gi5UycTBPUs= 2023-07-29T01:22:54.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:utXFtLpeF/logqu0Gi5UycTBPUs=', 'Content-Length': '0'} 2023-07-29T01:22:54.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8bfc302683b6081-0064c45a03-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.183 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-07-29T01:22:54.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b 2023-07-29T01:22:54.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b 2023-07-29T01:22:54.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b?versionId=null 2023-07-29T01:22:54.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b?versionId=null 2023-07-29T01:22:54.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/b?versionId=null 2023-07-29T01:22:54.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.187 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.187 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.187 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.187 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:14:59 GMT 2023-07-29T01:22:54.187 INFO:tasks.rgw_multisite_tests:/hiiqos-55/b?versionId=null 2023-07-29T01:22:54.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.188 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:K3jsoITgiOh1WpdTHXI210MlBVg= 2023-07-29T01:22:54.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K3jsoITgiOh1WpdTHXI210MlBVg=', 'Content-Length': '0'} 2023-07-29T01:22:54.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daaeeb853918d5e4-0064c45a03-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b 2023-07-29T01:22:54.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b 2023-07-29T01:22:54.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b?versionId=null 2023-07-29T01:22:54.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b?versionId=null 2023-07-29T01:22:54.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/b?versionId=null 2023-07-29T01:22:54.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.192 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.192 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.192 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.192 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:14:59 GMT 2023-07-29T01:22:54.193 INFO:tasks.rgw_multisite_tests:/hiiqos-55/b?versionId=null 2023-07-29T01:22:54.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.193 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:K3jsoITgiOh1WpdTHXI210MlBVg= 2023-07-29T01:22:54.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K3jsoITgiOh1WpdTHXI210MlBVg=', 'Content-Length': '0'} 2023-07-29T01:22:54.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000380171446d61880b-0064c45a03-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=b 2023-07-29T01:22:54.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=b 2023-07-29T01:22:54.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=b 2023-07-29T01:22:54.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.197 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.197 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.197 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.198 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:14:59 GMT 2023-07-29T01:22:54.198 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.199 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:t/PJ+bAhtkXlQy9TiCoPZfTxBw0= 2023-07-29T01:22:54.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t/PJ+bAhtkXlQy9TiCoPZfTxBw0=', 'Content-Length': '0'} 2023-07-29T01:22:54.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd7e9de391681f78-0064c45a03-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55b1000falseb2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=b 2023-07-29T01:22:54.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=b 2023-07-29T01:22:54.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=b 2023-07-29T01:22:54.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.203 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.203 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.203 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.203 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:14:59 GMT 2023-07-29T01:22:54.204 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.204 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:t/PJ+bAhtkXlQy9TiCoPZfTxBw0= 2023-07-29T01:22:54.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t/PJ+bAhtkXlQy9TiCoPZfTxBw0=', 'Content-Length': '0'} 2023-07-29T01:22:54.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c51e3c05f377c66a-0064c45a03-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55b1000falseb2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b 2023-07-29T01:22:54.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b 2023-07-29T01:22:54.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/b 2023-07-29T01:22:54.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.208 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.208 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.208 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.208 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:14:59 GMT 2023-07-29T01:22:54.209 INFO:tasks.rgw_multisite_tests:/hiiqos-55/b 2023-07-29T01:22:54.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.209 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:VWnzgs1UtCZ0poLsKOuJeiMZGmM= 2023-07-29T01:22:54.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VWnzgs1UtCZ0poLsKOuJeiMZGmM=', 'Content-Length': '0'} 2023-07-29T01:22:54.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001028bc788438768e-0064c45a03-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b 2023-07-29T01:22:54.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b 2023-07-29T01:22:54.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/b 2023-07-29T01:22:54.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.212 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.213 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.213 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.213 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:14:59 GMT 2023-07-29T01:22:54.213 INFO:tasks.rgw_multisite_tests:/hiiqos-55/b 2023-07-29T01:22:54.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.214 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:VWnzgs1UtCZ0poLsKOuJeiMZGmM= 2023-07-29T01:22:54.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VWnzgs1UtCZ0poLsKOuJeiMZGmM=', 'Content-Length': '0'} 2023-07-29T01:22:54.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a50159a9e3485def-0064c45a03-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.215 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-07-29T01:22:54.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b 2023-07-29T01:22:54.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b 2023-07-29T01:22:54.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/b 2023-07-29T01:22:54.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.218 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.218 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.218 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.218 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:14:59 GMT 2023-07-29T01:22:54.218 INFO:tasks.rgw_multisite_tests:/hiiqos-55/b 2023-07-29T01:22:54.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.219 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:4gJI8Oc2UBlMDSmw0aj3TixqZRw= 2023-07-29T01:22:54.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4gJI8Oc2UBlMDSmw0aj3TixqZRw=', 'Content-Length': '0'} 2023-07-29T01:22:54.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ce87ad2720ef04d-0064c45a03-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b 2023-07-29T01:22:54.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b 2023-07-29T01:22:54.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/b 2023-07-29T01:22:54.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.222 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.223 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.223 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.223 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:14:59 GMT 2023-07-29T01:22:54.223 INFO:tasks.rgw_multisite_tests:/hiiqos-55/b 2023-07-29T01:22:54.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.224 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:4gJI8Oc2UBlMDSmw0aj3TixqZRw= 2023-07-29T01:22:54.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:14:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4gJI8Oc2UBlMDSmw0aj3TixqZRw=', 'Content-Length': '0'} 2023-07-29T01:22:54.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d00be8623b5eb837-0064c45a03-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=b 2023-07-29T01:22:54.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=b 2023-07-29T01:22:54.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=b 2023-07-29T01:22:54.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.228 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.228 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.228 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.229 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:15:00 GMT 2023-07-29T01:22:54.229 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.229 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:8SN+2MFyD4AgqUBt82bgSDDOQ3k= 2023-07-29T01:22:54.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8SN+2MFyD4AgqUBt82bgSDDOQ3k=', 'Content-Length': '0'} 2023-07-29T01:22:54.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002362bed2c23187f0-0064c45a03-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:14:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55b1000falseb2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=b 2023-07-29T01:22:54.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=b 2023-07-29T01:22:54.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=b 2023-07-29T01:22:54.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.237 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.238 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.238 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.238 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:15:00 GMT 2023-07-29T01:22:54.239 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.239 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:8SN+2MFyD4AgqUBt82bgSDDOQ3k= 2023-07-29T01:22:54.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8SN+2MFyD4AgqUBt82bgSDDOQ3k=', 'Content-Length': '0'} 2023-07-29T01:22:54.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3c8405590d1b286-0064c45a04-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55b1000falseb2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.240 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-07-29T01:22:54.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c 2023-07-29T01:22:54.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c 2023-07-29T01:22:54.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c?versionId=null 2023-07-29T01:22:54.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c?versionId=null 2023-07-29T01:22:54.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/c?versionId=null 2023-07-29T01:22:54.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.244 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.244 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.244 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.244 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:15:00 GMT 2023-07-29T01:22:54.245 INFO:tasks.rgw_multisite_tests:/hiiqos-55/c?versionId=null 2023-07-29T01:22:54.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.245 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:YjAmfQCPaTtbcqbhvrR1pWDHv10= 2023-07-29T01:22:54.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YjAmfQCPaTtbcqbhvrR1pWDHv10=', 'Content-Length': '0'} 2023-07-29T01:22:54.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd827131c7a4c227-0064c45a04-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c 2023-07-29T01:22:54.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c 2023-07-29T01:22:54.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c?versionId=null 2023-07-29T01:22:54.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c?versionId=null 2023-07-29T01:22:54.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/c?versionId=null 2023-07-29T01:22:54.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.249 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.249 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.250 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.250 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:15:00 GMT 2023-07-29T01:22:54.250 INFO:tasks.rgw_multisite_tests:/hiiqos-55/c?versionId=null 2023-07-29T01:22:54.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.251 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:YjAmfQCPaTtbcqbhvrR1pWDHv10= 2023-07-29T01:22:54.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YjAmfQCPaTtbcqbhvrR1pWDHv10=', 'Content-Length': '0'} 2023-07-29T01:22:54.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ae6dabfc8ab86af-0064c45a04-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c 2023-07-29T01:22:54.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c 2023-07-29T01:22:54.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c?versionId=null 2023-07-29T01:22:54.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c?versionId=null 2023-07-29T01:22:54.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/c?versionId=null 2023-07-29T01:22:54.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.257 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.257 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.257 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.258 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:15:00 GMT 2023-07-29T01:22:54.258 INFO:tasks.rgw_multisite_tests:/hiiqos-55/c?versionId=null 2023-07-29T01:22:54.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.259 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:aSaZD/M6zbTj6g2ntftT/xs5Ink= 2023-07-29T01:22:54.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aSaZD/M6zbTj6g2ntftT/xs5Ink=', 'Content-Length': '0'} 2023-07-29T01:22:54.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000304cff565cebc758-0064c45a04-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c 2023-07-29T01:22:54.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c 2023-07-29T01:22:54.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c?versionId=null 2023-07-29T01:22:54.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c?versionId=null 2023-07-29T01:22:54.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/c?versionId=null 2023-07-29T01:22:54.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.265 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.266 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.266 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.267 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:15:00 GMT 2023-07-29T01:22:54.267 INFO:tasks.rgw_multisite_tests:/hiiqos-55/c?versionId=null 2023-07-29T01:22:54.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.268 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:aSaZD/M6zbTj6g2ntftT/xs5Ink= 2023-07-29T01:22:54.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aSaZD/M6zbTj6g2ntftT/xs5Ink=', 'Content-Length': '0'} 2023-07-29T01:22:54.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daaf1a8ad1363dfb-0064c45a04-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.269 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-07-29T01:22:54.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c 2023-07-29T01:22:54.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c 2023-07-29T01:22:54.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c?versionId=null 2023-07-29T01:22:54.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c?versionId=null 2023-07-29T01:22:54.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/c?versionId=null 2023-07-29T01:22:54.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.274 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.274 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.275 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.275 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:15:00 GMT 2023-07-29T01:22:54.275 INFO:tasks.rgw_multisite_tests:/hiiqos-55/c?versionId=null 2023-07-29T01:22:54.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.276 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:YjAmfQCPaTtbcqbhvrR1pWDHv10= 2023-07-29T01:22:54.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YjAmfQCPaTtbcqbhvrR1pWDHv10=', 'Content-Length': '0'} 2023-07-29T01:22:54.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf003ad0742d1696-0064c45a04-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c 2023-07-29T01:22:54.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c 2023-07-29T01:22:54.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c?versionId=null 2023-07-29T01:22:54.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c?versionId=null 2023-07-29T01:22:54.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/c?versionId=null 2023-07-29T01:22:54.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.281 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.282 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.283 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.283 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:15:00 GMT 2023-07-29T01:22:54.284 INFO:tasks.rgw_multisite_tests:/hiiqos-55/c?versionId=null 2023-07-29T01:22:54.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.285 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:YjAmfQCPaTtbcqbhvrR1pWDHv10= 2023-07-29T01:22:54.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YjAmfQCPaTtbcqbhvrR1pWDHv10=', 'Content-Length': '0'} 2023-07-29T01:22:54.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c7fb8983862db78-0064c45a04-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=c 2023-07-29T01:22:54.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=c 2023-07-29T01:22:54.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=c 2023-07-29T01:22:54.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.289 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.289 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.289 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.290 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:15:00 GMT 2023-07-29T01:22:54.290 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.290 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:8SN+2MFyD4AgqUBt82bgSDDOQ3k= 2023-07-29T01:22:54.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8SN+2MFyD4AgqUBt82bgSDDOQ3k=', 'Content-Length': '0'} 2023-07-29T01:22:54.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4f3aac13ed3cb5b-0064c45a04-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55c1000falsec2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=c 2023-07-29T01:22:54.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=c 2023-07-29T01:22:54.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=c 2023-07-29T01:22:54.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.295 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.295 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.295 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.295 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:15:00 GMT 2023-07-29T01:22:54.296 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.296 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:8SN+2MFyD4AgqUBt82bgSDDOQ3k= 2023-07-29T01:22:54.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8SN+2MFyD4AgqUBt82bgSDDOQ3k=', 'Content-Length': '0'} 2023-07-29T01:22:54.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076227edfcce70e2b-0064c45a04-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55c1000falsec2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c 2023-07-29T01:22:54.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c 2023-07-29T01:22:54.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/c 2023-07-29T01:22:54.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.300 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.300 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.300 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.300 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:15:00 GMT 2023-07-29T01:22:54.301 INFO:tasks.rgw_multisite_tests:/hiiqos-55/c 2023-07-29T01:22:54.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.301 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:ijOgwpG3/ky9gHXxa8KeS4yyK7s= 2023-07-29T01:22:54.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ijOgwpG3/ky9gHXxa8KeS4yyK7s=', 'Content-Length': '0'} 2023-07-29T01:22:54.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056dbe831a363d27f-0064c45a04-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c 2023-07-29T01:22:54.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c 2023-07-29T01:22:54.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/c 2023-07-29T01:22:54.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.305 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.305 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.306 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.306 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:15:00 GMT 2023-07-29T01:22:54.306 INFO:tasks.rgw_multisite_tests:/hiiqos-55/c 2023-07-29T01:22:54.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.307 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:ijOgwpG3/ky9gHXxa8KeS4yyK7s= 2023-07-29T01:22:54.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ijOgwpG3/ky9gHXxa8KeS4yyK7s=', 'Content-Length': '0'} 2023-07-29T01:22:54.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000861104ec7672c6ec-0064c45a04-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.307 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-07-29T01:22:54.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c 2023-07-29T01:22:54.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c 2023-07-29T01:22:54.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/c 2023-07-29T01:22:54.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.310 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.311 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.311 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.311 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:15:00 GMT 2023-07-29T01:22:54.311 INFO:tasks.rgw_multisite_tests:/hiiqos-55/c 2023-07-29T01:22:54.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.312 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:Bos7/aX/SnyTg1b0GI/CcwqdDJA= 2023-07-29T01:22:54.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Bos7/aX/SnyTg1b0GI/CcwqdDJA=', 'Content-Length': '0'} 2023-07-29T01:22:54.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073ee503875f1d8fd-0064c45a04-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c 2023-07-29T01:22:54.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c 2023-07-29T01:22:54.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/c 2023-07-29T01:22:54.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.316 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.316 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.316 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.316 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:15:00 GMT 2023-07-29T01:22:54.317 INFO:tasks.rgw_multisite_tests:/hiiqos-55/c 2023-07-29T01:22:54.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.318 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:Bos7/aX/SnyTg1b0GI/CcwqdDJA= 2023-07-29T01:22:54.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Bos7/aX/SnyTg1b0GI/CcwqdDJA=', 'Content-Length': '0'} 2023-07-29T01:22:54.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cbc49c4282e4605-0064c45a04-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=c 2023-07-29T01:22:54.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=c 2023-07-29T01:22:54.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=c 2023-07-29T01:22:54.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.322 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.322 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.322 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.322 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:15:00 GMT 2023-07-29T01:22:54.323 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.323 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:8SN+2MFyD4AgqUBt82bgSDDOQ3k= 2023-07-29T01:22:54.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8SN+2MFyD4AgqUBt82bgSDDOQ3k=', 'Content-Length': '0'} 2023-07-29T01:22:54.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c8921526bb23edc-0064c45a04-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55c1000falsec2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=c 2023-07-29T01:22:54.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=c 2023-07-29T01:22:54.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=c 2023-07-29T01:22:54.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.327 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.327 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.328 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.328 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:15:00 GMT 2023-07-29T01:22:54.328 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.329 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:8SN+2MFyD4AgqUBt82bgSDDOQ3k= 2023-07-29T01:22:54.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8SN+2MFyD4AgqUBt82bgSDDOQ3k=', 'Content-Length': '0'} 2023-07-29T01:22:54.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca50dcad5c4ff2aa-0064c45a04-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55c1000falsec2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.330 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-07-29T01:22:54.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d 2023-07-29T01:22:54.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d 2023-07-29T01:22:54.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d?versionId=null 2023-07-29T01:22:54.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d?versionId=null 2023-07-29T01:22:54.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/d?versionId=null 2023-07-29T01:22:54.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.333 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.333 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.333 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.334 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:15:00 GMT 2023-07-29T01:22:54.334 INFO:tasks.rgw_multisite_tests:/hiiqos-55/d?versionId=null 2023-07-29T01:22:54.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.334 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:EsQHzPYxlLmznzSu7D4nua6U214= 2023-07-29T01:22:54.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EsQHzPYxlLmznzSu7D4nua6U214=', 'Content-Length': '0'} 2023-07-29T01:22:54.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b50165b009733bba-0064c45a04-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d 2023-07-29T01:22:54.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d 2023-07-29T01:22:54.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d?versionId=null 2023-07-29T01:22:54.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d?versionId=null 2023-07-29T01:22:54.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/d?versionId=null 2023-07-29T01:22:54.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.338 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.339 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.339 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.339 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:15:00 GMT 2023-07-29T01:22:54.339 INFO:tasks.rgw_multisite_tests:/hiiqos-55/d?versionId=null 2023-07-29T01:22:54.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.340 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:EsQHzPYxlLmznzSu7D4nua6U214= 2023-07-29T01:22:54.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EsQHzPYxlLmznzSu7D4nua6U214=', 'Content-Length': '0'} 2023-07-29T01:22:54.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bebe8766a26861a-0064c45a04-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d 2023-07-29T01:22:54.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d 2023-07-29T01:22:54.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d?versionId=null 2023-07-29T01:22:54.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d?versionId=null 2023-07-29T01:22:54.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/d?versionId=null 2023-07-29T01:22:54.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.345 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.346 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.346 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.347 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:15:00 GMT 2023-07-29T01:22:54.347 INFO:tasks.rgw_multisite_tests:/hiiqos-55/d?versionId=null 2023-07-29T01:22:54.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.348 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:9lIoYyf5lgi/bGQwGetmKHx14Rw= 2023-07-29T01:22:54.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9lIoYyf5lgi/bGQwGetmKHx14Rw=', 'Content-Length': '0'} 2023-07-29T01:22:54.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032ed55bdab7b78cf-0064c45a04-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d 2023-07-29T01:22:54.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d 2023-07-29T01:22:54.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d?versionId=null 2023-07-29T01:22:54.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d?versionId=null 2023-07-29T01:22:54.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/d?versionId=null 2023-07-29T01:22:54.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.354 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.354 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.355 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.355 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:15:00 GMT 2023-07-29T01:22:54.356 INFO:tasks.rgw_multisite_tests:/hiiqos-55/d?versionId=null 2023-07-29T01:22:54.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.357 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:9lIoYyf5lgi/bGQwGetmKHx14Rw= 2023-07-29T01:22:54.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9lIoYyf5lgi/bGQwGetmKHx14Rw=', 'Content-Length': '0'} 2023-07-29T01:22:54.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa07b2e116c593a2-0064c45a04-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.359 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-07-29T01:22:54.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d 2023-07-29T01:22:54.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d 2023-07-29T01:22:54.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d?versionId=null 2023-07-29T01:22:54.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d?versionId=null 2023-07-29T01:22:54.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/d?versionId=null 2023-07-29T01:22:54.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.363 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.363 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.364 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.364 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:15:00 GMT 2023-07-29T01:22:54.364 INFO:tasks.rgw_multisite_tests:/hiiqos-55/d?versionId=null 2023-07-29T01:22:54.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.365 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:EsQHzPYxlLmznzSu7D4nua6U214= 2023-07-29T01:22:54.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EsQHzPYxlLmznzSu7D4nua6U214=', 'Content-Length': '0'} 2023-07-29T01:22:54.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e5e0a328668838c-0064c45a04-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d 2023-07-29T01:22:54.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d 2023-07-29T01:22:54.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d?versionId=null 2023-07-29T01:22:54.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d?versionId=null 2023-07-29T01:22:54.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/d?versionId=null 2023-07-29T01:22:54.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.369 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.369 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.369 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.370 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:15:00 GMT 2023-07-29T01:22:54.370 INFO:tasks.rgw_multisite_tests:/hiiqos-55/d?versionId=null 2023-07-29T01:22:54.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.370 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:EsQHzPYxlLmznzSu7D4nua6U214= 2023-07-29T01:22:54.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EsQHzPYxlLmznzSu7D4nua6U214=', 'Content-Length': '0'} 2023-07-29T01:22:54.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c22e6d89637c1c44-0064c45a04-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=d 2023-07-29T01:22:54.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=d 2023-07-29T01:22:54.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=d 2023-07-29T01:22:54.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.377 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.378 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.378 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.378 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:15:00 GMT 2023-07-29T01:22:54.378 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.379 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:8SN+2MFyD4AgqUBt82bgSDDOQ3k= 2023-07-29T01:22:54.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8SN+2MFyD4AgqUBt82bgSDDOQ3k=', 'Content-Length': '0'} 2023-07-29T01:22:54.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a77877f50e3c627-0064c45a04-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55d1000falsed2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=d 2023-07-29T01:22:54.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=d 2023-07-29T01:22:54.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=d 2023-07-29T01:22:54.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.383 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.383 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.383 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.383 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:15:00 GMT 2023-07-29T01:22:54.384 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.384 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:8SN+2MFyD4AgqUBt82bgSDDOQ3k= 2023-07-29T01:22:54.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8SN+2MFyD4AgqUBt82bgSDDOQ3k=', 'Content-Length': '0'} 2023-07-29T01:22:54.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a096c63a9307ee52-0064c45a04-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55d1000falsed2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d 2023-07-29T01:22:54.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d 2023-07-29T01:22:54.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/d 2023-07-29T01:22:54.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.388 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.388 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.388 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.388 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:15:00 GMT 2023-07-29T01:22:54.389 INFO:tasks.rgw_multisite_tests:/hiiqos-55/d 2023-07-29T01:22:54.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.389 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:vpIi4Sn3UkxFlRQyuw+9Syfas7U= 2023-07-29T01:22:54.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vpIi4Sn3UkxFlRQyuw+9Syfas7U=', 'Content-Length': '0'} 2023-07-29T01:22:54.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a871454aa75778f-0064c45a04-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d 2023-07-29T01:22:54.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d 2023-07-29T01:22:54.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/d 2023-07-29T01:22:54.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.392 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.393 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.393 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.393 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:15:00 GMT 2023-07-29T01:22:54.393 INFO:tasks.rgw_multisite_tests:/hiiqos-55/d 2023-07-29T01:22:54.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.394 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:vpIi4Sn3UkxFlRQyuw+9Syfas7U= 2023-07-29T01:22:54.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vpIi4Sn3UkxFlRQyuw+9Syfas7U=', 'Content-Length': '0'} 2023-07-29T01:22:54.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cad3aec9f3a544ed-0064c45a04-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.395 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-07-29T01:22:54.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d 2023-07-29T01:22:54.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d 2023-07-29T01:22:54.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/d 2023-07-29T01:22:54.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.397 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.398 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.398 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.398 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:15:00 GMT 2023-07-29T01:22:54.398 INFO:tasks.rgw_multisite_tests:/hiiqos-55/d 2023-07-29T01:22:54.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.399 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:MonvgxDid5FJgHcPN/I8YGfB6vY= 2023-07-29T01:22:54.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MonvgxDid5FJgHcPN/I8YGfB6vY=', 'Content-Length': '0'} 2023-07-29T01:22:54.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0179e24fcc46e1a-0064c45a04-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d 2023-07-29T01:22:54.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d 2023-07-29T01:22:54.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/d 2023-07-29T01:22:54.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.402 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.402 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.403 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.403 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:15:00 GMT 2023-07-29T01:22:54.403 INFO:tasks.rgw_multisite_tests:/hiiqos-55/d 2023-07-29T01:22:54.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.403 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:MonvgxDid5FJgHcPN/I8YGfB6vY= 2023-07-29T01:22:54.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:MonvgxDid5FJgHcPN/I8YGfB6vY=', 'Content-Length': '0'} 2023-07-29T01:22:54.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074fb0ab0c70b6a66-0064c45a04-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=d 2023-07-29T01:22:54.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=d 2023-07-29T01:22:54.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=d 2023-07-29T01:22:54.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.407 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.407 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.408 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.408 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:15:00 GMT 2023-07-29T01:22:54.408 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.409 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:8SN+2MFyD4AgqUBt82bgSDDOQ3k= 2023-07-29T01:22:54.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8SN+2MFyD4AgqUBt82bgSDDOQ3k=', 'Content-Length': '0'} 2023-07-29T01:22:54.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be17c7f39e3478b5-0064c45a04-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55d1000falsed2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=d 2023-07-29T01:22:54.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=d 2023-07-29T01:22:54.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=d 2023-07-29T01:22:54.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.412 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.413 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.413 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.413 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:15:00 GMT 2023-07-29T01:22:54.413 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.414 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:8SN+2MFyD4AgqUBt82bgSDDOQ3k= 2023-07-29T01:22:54.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:00 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8SN+2MFyD4AgqUBt82bgSDDOQ3k=', 'Content-Length': '0'} 2023-07-29T01:22:54.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000086cce5d9d0dc9d1-0064c45a04-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55d1000falsed2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.415 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=hiiqos-55 zones={a2, a1} 2023-07-29T01:22:54.415 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'hiiqos-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:54.415 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'hiiqos-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:54.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket hiiqos-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:22:54.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:54.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:54.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:54.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "layout": { 2023-07-29T01:22:54.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "resharding": "None", 2023-07-29T01:22:54.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_index": { 2023-07-29T01:22:54.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "gen": 0, 2023-07-29T01:22:54.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "layout": { 2023-07-29T01:22:54.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "type": "Normal", 2023-07-29T01:22:54.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "normal": { 2023-07-29T01:22:54.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 11, 2023-07-29T01:22:54.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "hash_type": "Mod" 2023-07-29T01:22:54.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:54.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:54.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:54.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "logs": [ 2023-07-29T01:22:54.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:54.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "gen": 0, 2023-07-29T01:22:54.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "layout": { 2023-07-29T01:22:54.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "type": "InIndex", 2023-07-29T01:22:54.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "in_index": { 2023-07-29T01:22:54.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "gen": 0, 2023-07-29T01:22:54.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "layout": { 2023-07-29T01:22:54.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 11, 2023-07-29T01:22:54.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "hash_type": "Mod" 2023-07-29T01:22:54.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:54.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:54.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:54.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:54.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:54.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:54.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:54.423 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'hiiqos-55', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:54.423 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'hiiqos-55', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:54.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket hiiqos-55 --gen 0 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:54.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:54.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:54.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: [ 2023-07-29T01:22:54.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:54.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "op_id": "1#00000000001.56.6", 2023-07-29T01:22:54.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "op_tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.8037444673821786044", 2023-07-29T01:22:54.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "op": "write", 2023-07-29T01:22:54.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "object": "a", 2023-07-29T01:22:54.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "instance": "", 2023-07-29T01:22:54.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": "complete", 2023-07-29T01:22:54.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "index_ver": 1, 2023-07-29T01:22:54.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:14:43.934557423Z", 2023-07-29T01:22:54.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "ver": { 2023-07-29T01:22:54.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pool": 9, 2023-07-29T01:22:54.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "epoch": 12 2023-07-29T01:22:54.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:54.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "bilog_flags": 0, 2023-07-29T01:22:54.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "versioned": false, 2023-07-29T01:22:54.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "owner": "", 2023-07-29T01:22:54.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "owner_display_name": "", 2023-07-29T01:22:54.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "zones_trace": [ 2023-07-29T01:22:54.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:54.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "entry": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20" 2023-07-29T01:22:54.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:54.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:54.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:54.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:54.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "op_id": "5#00000000001.58.6", 2023-07-29T01:22:54.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "op_tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.7287745460204636495", 2023-07-29T01:22:54.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "op": "write", 2023-07-29T01:22:54.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "object": "d", 2023-07-29T01:22:54.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "instance": "", 2023-07-29T01:22:54.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": "complete", 2023-07-29T01:22:54.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "index_ver": 1, 2023-07-29T01:22:54.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:14:44.026573109Z", 2023-07-29T01:22:54.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "ver": { 2023-07-29T01:22:54.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pool": 9, 2023-07-29T01:22:54.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "epoch": 2 2023-07-29T01:22:54.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:54.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "bilog_flags": 0, 2023-07-29T01:22:54.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "versioned": false, 2023-07-29T01:22:54.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "owner": "", 2023-07-29T01:22:54.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "owner_display_name": "", 2023-07-29T01:22:54.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "zones_trace": [ 2023-07-29T01:22:54.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:54.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "entry": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20" 2023-07-29T01:22:54.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:54.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:54.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:54.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:54.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "op_id": "8#00000000001.38.6", 2023-07-29T01:22:54.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "op_tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.15414676233093376460", 2023-07-29T01:22:54.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "op": "write", 2023-07-29T01:22:54.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "object": "c", 2023-07-29T01:22:54.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "instance": "", 2023-07-29T01:22:54.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": "complete", 2023-07-29T01:22:54.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "index_ver": 1, 2023-07-29T01:22:54.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:14:44.008991375Z", 2023-07-29T01:22:54.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "ver": { 2023-07-29T01:22:54.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pool": 9, 2023-07-29T01:22:54.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "epoch": 2 2023-07-29T01:22:54.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:54.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "bilog_flags": 0, 2023-07-29T01:22:54.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "versioned": false, 2023-07-29T01:22:54.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "owner": "", 2023-07-29T01:22:54.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "owner_display_name": "", 2023-07-29T01:22:54.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "zones_trace": [ 2023-07-29T01:22:54.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:54.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "entry": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20" 2023-07-29T01:22:54.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:54.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:54.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:54.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:54.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "op_id": "10#00000000001.23.6", 2023-07-29T01:22:54.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "op_tag": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5727.17230896804495972297", 2023-07-29T01:22:54.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "op": "write", 2023-07-29T01:22:54.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "object": "b", 2023-07-29T01:22:54.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "instance": "", 2023-07-29T01:22:54.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": "complete", 2023-07-29T01:22:54.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "index_ver": 1, 2023-07-29T01:22:54.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:14:43.991260949Z", 2023-07-29T01:22:54.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "ver": { 2023-07-29T01:22:54.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pool": 9, 2023-07-29T01:22:54.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "epoch": 2 2023-07-29T01:22:54.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:54.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "bilog_flags": 0, 2023-07-29T01:22:54.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "versioned": false, 2023-07-29T01:22:54.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "owner": "", 2023-07-29T01:22:54.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "owner_display_name": "", 2023-07-29T01:22:54.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "zones_trace": [ 2023-07-29T01:22:54.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:54.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "entry": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20" 2023-07-29T01:22:54.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:54.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:54.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:54.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:54.446 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'hiiqos-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:54.446 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'hiiqos-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:54.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket hiiqos-55 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:22:54.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:54.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:54.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: tenant: 2023-07-29T01:22:54.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: bucket name: hiiqos-55 2023-07-29T01:22:54.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: total entries: 4 2023-07-29T01:22:54.448 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'hiiqos-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:54.448 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'hiiqos-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:54.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket hiiqos-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:22:54.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:54.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:54.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:54.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "layout": { 2023-07-29T01:22:54.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "resharding": "None", 2023-07-29T01:22:54.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_index": { 2023-07-29T01:22:54.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "gen": 1, 2023-07-29T01:22:54.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "layout": { 2023-07-29T01:22:54.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "type": "Normal", 2023-07-29T01:22:54.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "normal": { 2023-07-29T01:22:54.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 3, 2023-07-29T01:22:54.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "hash_type": "Mod" 2023-07-29T01:22:54.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:54.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:54.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:54.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "logs": [ 2023-07-29T01:22:54.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:54.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "gen": 0, 2023-07-29T01:22:54.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "layout": { 2023-07-29T01:22:54.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "type": "InIndex", 2023-07-29T01:22:54.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "in_index": { 2023-07-29T01:22:54.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "gen": 0, 2023-07-29T01:22:54.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "layout": { 2023-07-29T01:22:54.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 11, 2023-07-29T01:22:54.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "hash_type": "Mod" 2023-07-29T01:22:54.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:54.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:54.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:54.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:54.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:54.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "gen": 1, 2023-07-29T01:22:54.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "layout": { 2023-07-29T01:22:54.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "type": "InIndex", 2023-07-29T01:22:54.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "in_index": { 2023-07-29T01:22:54.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "gen": 1, 2023-07-29T01:22:54.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "layout": { 2023-07-29T01:22:54.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 3, 2023-07-29T01:22:54.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "hash_type": "Mod" 2023-07-29T01:22:54.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:54.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:54.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:54.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:54.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:54.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:54.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:54.459 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'hiiqos-55', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:54.460 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'hiiqos-55', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:54.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket hiiqos-55 --gen 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:54.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:54.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:54.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: [] 2023-07-29T01:22:54.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/ 2023-07-29T01:22:54.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.463 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.464 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.464 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.464 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:15:01 GMT 2023-07-29T01:22:54.464 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.465 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:DeuQEaQFwkMkxTy5Q9DpGMIBJeI= 2023-07-29T01:22:54.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DeuQEaQFwkMkxTy5Q9DpGMIBJeI=', 'Content-Length': '0'} 2023-07-29T01:22:54.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a527f7f9b1751137-0064c45a04-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/e 2023-07-29T01:22:54.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/e 2023-07-29T01:22:54.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:54.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/e 2023-07-29T01:22:54.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:54.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.468 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:54.469 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:54.469 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:54.469 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:15:01 GMT 2023-07-29T01:22:54.469 INFO:tasks.rgw_multisite_tests:/hiiqos-55/e 2023-07-29T01:22:54.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.470 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:4egBKpz4HEG+RQEyrOl/R9L3Ph4= 2023-07-29T01:22:54.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:15:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4egBKpz4HEG+RQEyrOl/R9L3Ph4='} 2023-07-29T01:22:54.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005e2f734701b60eba-0064c45a04-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/ 2023-07-29T01:22:54.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.473 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.473 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.473 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.474 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:15:01 GMT 2023-07-29T01:22:54.474 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.474 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:DeuQEaQFwkMkxTy5Q9DpGMIBJeI= 2023-07-29T01:22:54.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DeuQEaQFwkMkxTy5Q9DpGMIBJeI=', 'Content-Length': '0'} 2023-07-29T01:22:54.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000023f33a93db49114e-0064c45a04-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/f 2023-07-29T01:22:54.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/f 2023-07-29T01:22:54.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:54.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/f 2023-07-29T01:22:54.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:54.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.478 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:54.478 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:54.478 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:54.478 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:15:01 GMT 2023-07-29T01:22:54.479 INFO:tasks.rgw_multisite_tests:/hiiqos-55/f 2023-07-29T01:22:54.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.479 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:Q9aDeK6k5VER/MLgN76iEUJj9lk= 2023-07-29T01:22:54.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:15:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q9aDeK6k5VER/MLgN76iEUJj9lk='} 2023-07-29T01:22:54.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006084addd4b540420-0064c45a05-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/ 2023-07-29T01:22:54.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.482 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.483 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.483 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.483 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:15:01 GMT 2023-07-29T01:22:54.483 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.484 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:DeuQEaQFwkMkxTy5Q9DpGMIBJeI= 2023-07-29T01:22:54.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DeuQEaQFwkMkxTy5Q9DpGMIBJeI=', 'Content-Length': '0'} 2023-07-29T01:22:54.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009c73f91c92024568-0064c45a05-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/g 2023-07-29T01:22:54.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/g 2023-07-29T01:22:54.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:54.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/g 2023-07-29T01:22:54.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:54.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.487 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:54.487 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:54.488 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:54.488 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:15:01 GMT 2023-07-29T01:22:54.488 INFO:tasks.rgw_multisite_tests:/hiiqos-55/g 2023-07-29T01:22:54.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.489 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:EN+oMQS9m6l/UexGwXkJzzAA86s= 2023-07-29T01:22:54.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:15:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EN+oMQS9m6l/UexGwXkJzzAA86s='} 2023-07-29T01:22:54.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000498c6a4f1cfe79df-0064c45a05-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/ 2023-07-29T01:22:54.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.492 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.492 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.492 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.493 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:15:01 GMT 2023-07-29T01:22:54.493 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.493 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:DeuQEaQFwkMkxTy5Q9DpGMIBJeI= 2023-07-29T01:22:54.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:15:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DeuQEaQFwkMkxTy5Q9DpGMIBJeI=', 'Content-Length': '0'} 2023-07-29T01:22:54.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006730b230bf44059b-0064c45a05-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/h 2023-07-29T01:22:54.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/h 2023-07-29T01:22:54.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:54.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/h 2023-07-29T01:22:54.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:54.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.497 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:54.497 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:54.497 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:54.497 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:15:01 GMT 2023-07-29T01:22:54.498 INFO:tasks.rgw_multisite_tests:/hiiqos-55/h 2023-07-29T01:22:54.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.498 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:YSotu97b1ZrT2gU7tvFJdNWA140= 2023-07-29T01:22:54.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:15:01 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YSotu97b1ZrT2gU7tvFJdNWA140='} 2023-07-29T01:22:54.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b0862846ad667a22-0064c45a05-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.499 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=hiiqos-55 2023-07-29T01:22:54.499 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T01:22:54.499 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T01:22:54.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T01:22:54.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:54.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:54.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:15:01.166+0000 7fb023329880 1 waiting to reach latest gen 1, on 0.. 2023-07-29T01:22:54.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:15:06.166+0000 7fb023329880 1 waiting to reach latest gen 1, on 0.. 2023-07-29T01:22:54.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:15:11.167+0000 7fb023329880 1 waiting to reach latest gen 1, on 0.. 2023-07-29T01:22:54.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:15:16.168+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T01:22:54.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: local status: [, , ] 2023-07-29T01:22:54.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T01:22:54.502 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.1.smithi139.stdout: INFO: 2023-07-29T00:15:17.831+0000 7f5731a6a640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T01:22:54.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:15:21.169+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T01:22:54.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: local status: [, , ] 2023-07-29T01:22:54.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T01:22:54.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:15:26.170+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T01:22:54.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: local status: [, , ] 2023-07-29T01:22:54.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T01:22:54.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:15:31.171+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T01:22:54.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: local status: [, , ] 2023-07-29T01:22:54.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T01:22:54.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:15:36.171+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T01:22:54.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: local status: [, , ] 2023-07-29T01:22:54.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T01:22:54.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:15:41.172+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T01:22:54.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: local status: [, , ] 2023-07-29T01:22:54.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T01:22:54.505 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi139.stdout: INFO: 2023-07-29T00:15:42.793+0000 7f5f2671b640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T01:22:54.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:15:46.173+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T01:22:54.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: local status: [, , ] 2023-07-29T01:22:54.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T01:22:54.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:15:51.173+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T01:22:54.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: local status: [, , ] 2023-07-29T01:22:54.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T01:22:54.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:15:56.174+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T01:22:54.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: local status: [, 00000000002.40.6, ] 2023-07-29T01:22:54.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T01:22:54.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:16:01.175+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T01:22:54.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: local status: [, 00000000002.40.6, ] 2023-07-29T01:22:54.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T01:22:54.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:16:06.176+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T01:22:54.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: local status: [, 00000000002.40.6, ] 2023-07-29T01:22:54.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T01:22:54.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:16:11.177+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T01:22:54.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: local status: [, 00000000002.40.6, ] 2023-07-29T01:22:54.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T01:22:54.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:16:16.177+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T01:22:54.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: local status: [, 00000000002.40.6, ] 2023-07-29T01:22:54.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T01:22:54.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:16:21.178+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T01:22:54.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: local status: [, 00000000002.40.6, ] 2023-07-29T01:22:54.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T01:22:54.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:16:26.179+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T01:22:54.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: local status: [, 00000000002.40.6, ] 2023-07-29T01:22:54.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T01:22:54.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:16:31.180+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T01:22:54.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: local status: [, 00000000002.40.6, ] 2023-07-29T01:22:54.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T01:22:54.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:16:36.181+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T01:22:54.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: local status: [, 00000000002.40.6, 00000000002.48.6] 2023-07-29T01:22:54.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T01:22:54.513 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi101.stdout: INFO: 2023-07-29T00:16:37.016+0000 7f0081a5e640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T01:22:54.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:16:41.181+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T01:22:54.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: local status: [, 00000000002.40.6, 00000000002.48.6] 2023-07-29T01:22:54.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T01:22:54.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:16:46.182+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T01:22:54.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: local status: [, 00000000002.40.6, 00000000002.48.6] 2023-07-29T01:22:54.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T01:22:54.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:16:51.183+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T01:22:54.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: local status: [, 00000000002.40.6, 00000000002.48.6] 2023-07-29T01:22:54.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T01:22:54.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:16:56.184+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T01:22:54.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: local status: [, 00000000002.40.6, 00000000002.48.6] 2023-07-29T01:22:54.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T01:22:54.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:17:01.184+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T01:22:54.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: local status: [, 00000000002.40.6, 00000000002.48.6] 2023-07-29T01:22:54.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T01:22:54.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:17:06.185+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T01:22:54.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: local status: [, 00000000002.40.6, 00000000002.48.6] 2023-07-29T01:22:54.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T01:22:54.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:17:11.186+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T01:22:54.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: local status: [, 00000000002.40.6, 00000000002.48.6] 2023-07-29T01:22:54.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T01:22:54.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:17:16.187+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T01:22:54.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: local status: [, 00000000002.40.6, 00000000002.48.6] 2023-07-29T01:22:54.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T01:22:54.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:17:21.188+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T01:22:54.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: local status: [, 00000000002.40.6, 00000000002.48.6] 2023-07-29T01:22:54.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T01:22:54.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:17:26.188+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T01:22:54.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: local status: [, 00000000002.40.6, 00000000002.48.6] 2023-07-29T01:22:54.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T01:22:54.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:17:31.189+0000 7fb023329880 1 waiting for incremental sync to catch up: 2023-07-29T01:22:54.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: local status: [, 00000000002.40.6, 00000000002.48.6] 2023-07-29T01:22:54.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T01:22:54.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:17:36.190+0000 7fb023329880 1 bucket sync caught up with source: 2023-07-29T01:22:54.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: local status: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T01:22:54.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: remote markers: [00000000003.51.6, 00000000002.40.6, 00000000002.48.6] 2023-07-29T01:22:54.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:17:36.190+0000 7fb023329880 0 bucket checkpoint complete 2023-07-29T01:22:54.522 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=hiiqos-55 2023-07-29T01:22:54.523 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T01:22:54.523 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T01:22:54.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T01:22:54.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:54.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:54.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:17:36.303+0000 7f4ba0126880 1 waiting for incremental sync to catch up: 2023-07-29T01:22:54.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: local status: [, 00000000002.714.6, , , 00000000001.1347.6, 00000000001.707.6, , , 00000000002.386.6, , 00000000001.1165.6] 2023-07-29T01:22:54.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: remote markers: [, 00000000002.714.6, , , 00000000001.1347.6, 00000000001.707.6, , , 00000000002.386.6, , 00000000002.1169.6] 2023-07-29T01:22:54.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:17:41.305+0000 7f4ba0126880 1 bucket sync caught up with source: 2023-07-29T01:22:54.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: local status: [, 00000000002.714.6, , , 00000000001.1347.6, 00000000001.707.6, , , 00000000002.386.6, , 00000000002.1169.6] 2023-07-29T01:22:54.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: remote markers: [, 00000000002.714.6, , , 00000000001.1347.6, 00000000001.707.6, , , 00000000002.386.6, , 00000000002.1169.6] 2023-07-29T01:22:54.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:17:41.305+0000 7f4ba0126880 0 bucket checkpoint complete 2023-07-29T01:22:54.525 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=hiiqos-55 zones={a2, a1} 2023-07-29T01:22:54.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/ 2023-07-29T01:22:54.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-29T01:22:54.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.528 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.529 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.529 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.529 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:41 GMT 2023-07-29T01:22:54.529 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.530 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:aKtJjMgBOwNro+aqU8Irrs67zAY= 2023-07-29T01:22:54.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aKtJjMgBOwNro+aqU8Irrs67zAY=', 'Content-Length': '0'} 2023-07-29T01:22:54.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c71712ce5b4e363f-0064c45aaa-1663-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:17:46 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/ 2023-07-29T01:22:54.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-29T01:22:54.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.533 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.534 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.534 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.534 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:46 GMT 2023-07-29T01:22:54.534 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.535 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:WOE4et53hzWWtjOpYL66yPz25AI= 2023-07-29T01:22:54.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:46 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WOE4et53hzWWtjOpYL66yPz25AI=', 'Content-Length': '0'} 2023-07-29T01:22:54.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000703a9bf9eec3bfa0-0064c45aaf-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.535 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2023-07-29T01:22:54.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?versions&max-keys=999 2023-07-29T01:22:54.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?versions&max-keys=999 2023-07-29T01:22:54.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?versions&max-keys=999 2023-07-29T01:22:54.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.539 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.539 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.539 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.539 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.540 INFO:tasks.rgw_multisite_tests:/hiiqos-55/?versions 2023-07-29T01:22:54.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.540 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:mJGH+AVFvJUrMB0TOMF+7f9NpsA= 2023-07-29T01:22:54.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mJGH+AVFvJUrMB0TOMF+7f9NpsA=', 'Content-Length': '0'} 2023-07-29T01:22:54.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c648f4c0a6589d05-0064c45aaf-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55999falseanulltrue2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-07-29T00:15:00.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-07-29T00:15:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-07-29T00:15:01.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-07-29T00:15:01.033Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.541 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2023-07-29T01:22:54.543 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2023-07-29T01:22:54.543 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2023-07-29T01:22:54.543 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2023-07-29T01:22:54.543 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2023-07-29T01:22:54.544 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2023-07-29T01:22:54.544 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2023-07-29T01:22:54.544 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2023-07-29T01:22:54.544 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2023-07-29T01:22:54.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?versions&max-keys=999 2023-07-29T01:22:54.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?versions&max-keys=999 2023-07-29T01:22:54.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?versions&max-keys=999 2023-07-29T01:22:54.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.548 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.548 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.548 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.548 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.549 INFO:tasks.rgw_multisite_tests:/hiiqos-55/?versions 2023-07-29T01:22:54.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.549 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:mJGH+AVFvJUrMB0TOMF+7f9NpsA= 2023-07-29T01:22:54.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mJGH+AVFvJUrMB0TOMF+7f9NpsA=', 'Content-Length': '0'} 2023-07-29T01:22:54.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000216b07236d99abe-0064c45aaf-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55999falseanulltrue2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-07-29T00:15:00.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-07-29T00:15:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-07-29T00:15:01.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-07-29T00:15:01.033Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.550 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2023-07-29T01:22:54.551 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2023-07-29T01:22:54.551 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2023-07-29T01:22:54.551 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2023-07-29T01:22:54.551 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2023-07-29T01:22:54.552 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2023-07-29T01:22:54.552 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2023-07-29T01:22:54.552 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2023-07-29T01:22:54.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?versions&max-keys=999 2023-07-29T01:22:54.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?versions&max-keys=999 2023-07-29T01:22:54.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?versions&max-keys=999 2023-07-29T01:22:54.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.555 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.556 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.556 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.556 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.556 INFO:tasks.rgw_multisite_tests:/hiiqos-55/?versions 2023-07-29T01:22:54.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.557 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:mJGH+AVFvJUrMB0TOMF+7f9NpsA= 2023-07-29T01:22:54.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mJGH+AVFvJUrMB0TOMF+7f9NpsA=', 'Content-Length': '0'} 2023-07-29T01:22:54.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003d929b53d1df2ea-0064c45aaf-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55999falseanulltrue2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-07-29T00:15:00.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-07-29T00:15:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-07-29T00:15:01.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-07-29T00:15:01.033Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?versions&max-keys=999 2023-07-29T01:22:54.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?versions&max-keys=999 2023-07-29T01:22:54.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?versions&max-keys=999 2023-07-29T01:22:54.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.561 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.561 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.561 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.562 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.562 INFO:tasks.rgw_multisite_tests:/hiiqos-55/?versions 2023-07-29T01:22:54.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.562 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:mJGH+AVFvJUrMB0TOMF+7f9NpsA= 2023-07-29T01:22:54.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mJGH+AVFvJUrMB0TOMF+7f9NpsA=', 'Content-Length': '0'} 2023-07-29T01:22:54.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fac22311d4146e3-0064c45aaf-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55999falseanulltrue2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-07-29T00:15:00.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-07-29T00:15:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-07-29T00:15:01.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-07-29T00:15:01.033Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.563 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-07-29T01:22:54.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a 2023-07-29T01:22:54.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a 2023-07-29T01:22:54.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a?versionId=null 2023-07-29T01:22:54.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a?versionId=null 2023-07-29T01:22:54.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/a?versionId=null 2023-07-29T01:22:54.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.567 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.567 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.567 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.567 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.568 INFO:tasks.rgw_multisite_tests:/hiiqos-55/a?versionId=null 2023-07-29T01:22:54.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.568 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:e6wNrspmnivzuBOKojSzwt7KgT4= 2023-07-29T01:22:54.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:e6wNrspmnivzuBOKojSzwt7KgT4=', 'Content-Length': '0'} 2023-07-29T01:22:54.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ace96c9eee118183-0064c45aaf-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a 2023-07-29T01:22:54.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a 2023-07-29T01:22:54.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a?versionId=null 2023-07-29T01:22:54.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a?versionId=null 2023-07-29T01:22:54.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/a?versionId=null 2023-07-29T01:22:54.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.572 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.572 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.572 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.572 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.573 INFO:tasks.rgw_multisite_tests:/hiiqos-55/a?versionId=null 2023-07-29T01:22:54.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.573 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:e6wNrspmnivzuBOKojSzwt7KgT4= 2023-07-29T01:22:54.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:e6wNrspmnivzuBOKojSzwt7KgT4=', 'Content-Length': '0'} 2023-07-29T01:22:54.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cb4ebeb01072aad-0064c45aaf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a 2023-07-29T01:22:54.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a 2023-07-29T01:22:54.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a?versionId=null 2023-07-29T01:22:54.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a?versionId=null 2023-07-29T01:22:54.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/a?versionId=null 2023-07-29T01:22:54.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.577 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.577 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.577 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.578 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.578 INFO:tasks.rgw_multisite_tests:/hiiqos-55/a?versionId=null 2023-07-29T01:22:54.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.578 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:2joamIz6RqsEMSyTGSOG0GVi6M8= 2023-07-29T01:22:54.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2joamIz6RqsEMSyTGSOG0GVi6M8=', 'Content-Length': '0'} 2023-07-29T01:22:54.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091f6c83479cb1df0-0064c45aaf-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a 2023-07-29T01:22:54.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a 2023-07-29T01:22:54.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a?versionId=null 2023-07-29T01:22:54.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a?versionId=null 2023-07-29T01:22:54.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/a?versionId=null 2023-07-29T01:22:54.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.582 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.582 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.582 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.583 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.583 INFO:tasks.rgw_multisite_tests:/hiiqos-55/a?versionId=null 2023-07-29T01:22:54.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.583 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:2joamIz6RqsEMSyTGSOG0GVi6M8= 2023-07-29T01:22:54.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2joamIz6RqsEMSyTGSOG0GVi6M8=', 'Content-Length': '0'} 2023-07-29T01:22:54.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d15bdb9685eee43d-0064c45aaf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.584 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-07-29T01:22:54.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a 2023-07-29T01:22:54.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a 2023-07-29T01:22:54.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a?versionId=null 2023-07-29T01:22:54.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a?versionId=null 2023-07-29T01:22:54.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/a?versionId=null 2023-07-29T01:22:54.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.587 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.588 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.588 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.588 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.588 INFO:tasks.rgw_multisite_tests:/hiiqos-55/a?versionId=null 2023-07-29T01:22:54.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.589 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:e6wNrspmnivzuBOKojSzwt7KgT4= 2023-07-29T01:22:54.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:e6wNrspmnivzuBOKojSzwt7KgT4=', 'Content-Length': '0'} 2023-07-29T01:22:54.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000344f4965b17b2270-0064c45aaf-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a 2023-07-29T01:22:54.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a 2023-07-29T01:22:54.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a?versionId=null 2023-07-29T01:22:54.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a?versionId=null 2023-07-29T01:22:54.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/a?versionId=null 2023-07-29T01:22:54.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.592 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.593 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.593 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.593 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.593 INFO:tasks.rgw_multisite_tests:/hiiqos-55/a?versionId=null 2023-07-29T01:22:54.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.594 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:e6wNrspmnivzuBOKojSzwt7KgT4= 2023-07-29T01:22:54.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:e6wNrspmnivzuBOKojSzwt7KgT4=', 'Content-Length': '0'} 2023-07-29T01:22:54.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7e67b63f9c69051-0064c45aaf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=a 2023-07-29T01:22:54.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=a 2023-07-29T01:22:54.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=a 2023-07-29T01:22:54.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.598 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.598 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.598 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.598 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.599 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.599 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:ViuMHmbO3wFHX4uFdsobzPFwMFk= 2023-07-29T01:22:54.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ViuMHmbO3wFHX4uFdsobzPFwMFk=', 'Content-Length': '0'} 2023-07-29T01:22:54.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e20a9224bfddb9d-0064c45aaf-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55a1000falsea2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=a 2023-07-29T01:22:54.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=a 2023-07-29T01:22:54.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=a 2023-07-29T01:22:54.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.603 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.603 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.603 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.604 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.604 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.604 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:ViuMHmbO3wFHX4uFdsobzPFwMFk= 2023-07-29T01:22:54.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ViuMHmbO3wFHX4uFdsobzPFwMFk=', 'Content-Length': '0'} 2023-07-29T01:22:54.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000647c6e0438458878-0064c45aaf-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55a1000falsea2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a 2023-07-29T01:22:54.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a 2023-07-29T01:22:54.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/a 2023-07-29T01:22:54.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.608 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.608 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.608 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.609 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.609 INFO:tasks.rgw_multisite_tests:/hiiqos-55/a 2023-07-29T01:22:54.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.609 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:Gazq58+W+/UQCVQssQp5BiMFKHc= 2023-07-29T01:22:54.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Gazq58+W+/UQCVQssQp5BiMFKHc=', 'Content-Length': '0'} 2023-07-29T01:22:54.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000881ff93db5cadb72-0064c45aaf-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a 2023-07-29T01:22:54.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a 2023-07-29T01:22:54.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/a 2023-07-29T01:22:54.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.613 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.613 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.613 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.613 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.613 INFO:tasks.rgw_multisite_tests:/hiiqos-55/a 2023-07-29T01:22:54.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.614 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:Gazq58+W+/UQCVQssQp5BiMFKHc= 2023-07-29T01:22:54.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Gazq58+W+/UQCVQssQp5BiMFKHc=', 'Content-Length': '0'} 2023-07-29T01:22:54.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e032f095dfd9c3d1-0064c45aaf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.615 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-07-29T01:22:54.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a 2023-07-29T01:22:54.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a 2023-07-29T01:22:54.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/a 2023-07-29T01:22:54.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.617 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.618 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.618 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.618 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.618 INFO:tasks.rgw_multisite_tests:/hiiqos-55/a 2023-07-29T01:22:54.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.619 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:EGhkezi8Jc3wXAJ3VxuFv3J+aZM= 2023-07-29T01:22:54.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EGhkezi8Jc3wXAJ3VxuFv3J+aZM=', 'Content-Length': '0'} 2023-07-29T01:22:54.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071d9e4dce91afbe8-0064c45aaf-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a 2023-07-29T01:22:54.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a 2023-07-29T01:22:54.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/a 2023-07-29T01:22:54.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.622 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.622 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.622 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.623 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.623 INFO:tasks.rgw_multisite_tests:/hiiqos-55/a 2023-07-29T01:22:54.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.623 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:EGhkezi8Jc3wXAJ3VxuFv3J+aZM= 2023-07-29T01:22:54.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EGhkezi8Jc3wXAJ3VxuFv3J+aZM=', 'Content-Length': '0'} 2023-07-29T01:22:54.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062f239f60daf50d8-0064c45aaf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=a 2023-07-29T01:22:54.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=a 2023-07-29T01:22:54.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=a 2023-07-29T01:22:54.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.627 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.627 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.628 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.628 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.628 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.628 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:ViuMHmbO3wFHX4uFdsobzPFwMFk= 2023-07-29T01:22:54.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ViuMHmbO3wFHX4uFdsobzPFwMFk=', 'Content-Length': '0'} 2023-07-29T01:22:54.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdbdbe757151fe47-0064c45aaf-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55a1000falsea2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=a 2023-07-29T01:22:54.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=a 2023-07-29T01:22:54.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=a 2023-07-29T01:22:54.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.632 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.633 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.633 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.633 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.633 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.634 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:ViuMHmbO3wFHX4uFdsobzPFwMFk= 2023-07-29T01:22:54.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ViuMHmbO3wFHX4uFdsobzPFwMFk=', 'Content-Length': '0'} 2023-07-29T01:22:54.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009aeaf1630171da10-0064c45aaf-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55a1000falsea2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.635 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-07-29T01:22:54.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b 2023-07-29T01:22:54.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b 2023-07-29T01:22:54.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b?versionId=null 2023-07-29T01:22:54.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b?versionId=null 2023-07-29T01:22:54.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/b?versionId=null 2023-07-29T01:22:54.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.638 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.638 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.638 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.639 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.639 INFO:tasks.rgw_multisite_tests:/hiiqos-55/b?versionId=null 2023-07-29T01:22:54.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.639 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:zJMcWDg9xOXxim63GHNQSwYIZHI= 2023-07-29T01:22:54.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zJMcWDg9xOXxim63GHNQSwYIZHI=', 'Content-Length': '0'} 2023-07-29T01:22:54.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004de841d55ed49eb9-0064c45aaf-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b 2023-07-29T01:22:54.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b 2023-07-29T01:22:54.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b?versionId=null 2023-07-29T01:22:54.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b?versionId=null 2023-07-29T01:22:54.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/b?versionId=null 2023-07-29T01:22:54.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.643 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.643 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.644 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.644 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.644 INFO:tasks.rgw_multisite_tests:/hiiqos-55/b?versionId=null 2023-07-29T01:22:54.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.645 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:zJMcWDg9xOXxim63GHNQSwYIZHI= 2023-07-29T01:22:54.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zJMcWDg9xOXxim63GHNQSwYIZHI=', 'Content-Length': '0'} 2023-07-29T01:22:54.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f767e89f3016ac84-0064c45aaf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b 2023-07-29T01:22:54.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b 2023-07-29T01:22:54.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b?versionId=null 2023-07-29T01:22:54.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b?versionId=null 2023-07-29T01:22:54.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/b?versionId=null 2023-07-29T01:22:54.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.648 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.648 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.649 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.649 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.649 INFO:tasks.rgw_multisite_tests:/hiiqos-55/b?versionId=null 2023-07-29T01:22:54.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.650 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:wsKKEj/6K/Xm0pDI7fjJ8Sxv2i8= 2023-07-29T01:22:54.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wsKKEj/6K/Xm0pDI7fjJ8Sxv2i8=', 'Content-Length': '0'} 2023-07-29T01:22:54.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fce70c835644a44-0064c45aaf-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b 2023-07-29T01:22:54.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b 2023-07-29T01:22:54.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b?versionId=null 2023-07-29T01:22:54.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b?versionId=null 2023-07-29T01:22:54.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/b?versionId=null 2023-07-29T01:22:54.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.653 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.654 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.654 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.654 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.654 INFO:tasks.rgw_multisite_tests:/hiiqos-55/b?versionId=null 2023-07-29T01:22:54.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.655 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:wsKKEj/6K/Xm0pDI7fjJ8Sxv2i8= 2023-07-29T01:22:54.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:wsKKEj/6K/Xm0pDI7fjJ8Sxv2i8=', 'Content-Length': '0'} 2023-07-29T01:22:54.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fff2452cf575b76-0064c45aaf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.655 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-07-29T01:22:54.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b 2023-07-29T01:22:54.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b 2023-07-29T01:22:54.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b?versionId=null 2023-07-29T01:22:54.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b?versionId=null 2023-07-29T01:22:54.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/b?versionId=null 2023-07-29T01:22:54.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.659 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.659 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.659 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.659 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.660 INFO:tasks.rgw_multisite_tests:/hiiqos-55/b?versionId=null 2023-07-29T01:22:54.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.660 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:zJMcWDg9xOXxim63GHNQSwYIZHI= 2023-07-29T01:22:54.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zJMcWDg9xOXxim63GHNQSwYIZHI=', 'Content-Length': '0'} 2023-07-29T01:22:54.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072303957af4a1729-0064c45aaf-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b 2023-07-29T01:22:54.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b 2023-07-29T01:22:54.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b?versionId=null 2023-07-29T01:22:54.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b?versionId=null 2023-07-29T01:22:54.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/b?versionId=null 2023-07-29T01:22:54.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.664 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.664 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.664 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.664 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.665 INFO:tasks.rgw_multisite_tests:/hiiqos-55/b?versionId=null 2023-07-29T01:22:54.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.665 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:zJMcWDg9xOXxim63GHNQSwYIZHI= 2023-07-29T01:22:54.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zJMcWDg9xOXxim63GHNQSwYIZHI=', 'Content-Length': '0'} 2023-07-29T01:22:54.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000011d9770f70c9acf-0064c45aaf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=b 2023-07-29T01:22:54.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=b 2023-07-29T01:22:54.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=b 2023-07-29T01:22:54.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.669 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.669 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.669 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.670 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.670 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.670 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:ViuMHmbO3wFHX4uFdsobzPFwMFk= 2023-07-29T01:22:54.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ViuMHmbO3wFHX4uFdsobzPFwMFk=', 'Content-Length': '0'} 2023-07-29T01:22:54.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000270f530221e8aff0-0064c45aaf-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55b1000falseb2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=b 2023-07-29T01:22:54.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=b 2023-07-29T01:22:54.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=b 2023-07-29T01:22:54.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.674 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.674 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.675 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.675 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.675 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.676 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:ViuMHmbO3wFHX4uFdsobzPFwMFk= 2023-07-29T01:22:54.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ViuMHmbO3wFHX4uFdsobzPFwMFk=', 'Content-Length': '0'} 2023-07-29T01:22:54.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077f03034941a8ff1-0064c45aaf-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55b1000falseb2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b 2023-07-29T01:22:54.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b 2023-07-29T01:22:54.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/b 2023-07-29T01:22:54.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.679 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.679 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.680 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.680 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.680 INFO:tasks.rgw_multisite_tests:/hiiqos-55/b 2023-07-29T01:22:54.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.680 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:RllwATRheJgeJUxadVoKtPSQuYg= 2023-07-29T01:22:54.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RllwATRheJgeJUxadVoKtPSQuYg=', 'Content-Length': '0'} 2023-07-29T01:22:54.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df660554483b3d90-0064c45aaf-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b 2023-07-29T01:22:54.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b 2023-07-29T01:22:54.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/b 2023-07-29T01:22:54.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.684 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.684 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.684 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.684 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.685 INFO:tasks.rgw_multisite_tests:/hiiqos-55/b 2023-07-29T01:22:54.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.685 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:RllwATRheJgeJUxadVoKtPSQuYg= 2023-07-29T01:22:54.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RllwATRheJgeJUxadVoKtPSQuYg=', 'Content-Length': '0'} 2023-07-29T01:22:54.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bfec04c540d16c1-0064c45aaf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.686 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-07-29T01:22:54.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b 2023-07-29T01:22:54.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b 2023-07-29T01:22:54.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/b 2023-07-29T01:22:54.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.689 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.689 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.689 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.689 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.690 INFO:tasks.rgw_multisite_tests:/hiiqos-55/b 2023-07-29T01:22:54.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.690 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:VMDvXjs1rzQLghcfDyLtGxUs7no= 2023-07-29T01:22:54.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VMDvXjs1rzQLghcfDyLtGxUs7no=', 'Content-Length': '0'} 2023-07-29T01:22:54.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000192749cbe01f146b-0064c45aaf-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b 2023-07-29T01:22:54.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b 2023-07-29T01:22:54.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/b 2023-07-29T01:22:54.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.693 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.693 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.694 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.694 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.694 INFO:tasks.rgw_multisite_tests:/hiiqos-55/b 2023-07-29T01:22:54.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.695 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:VMDvXjs1rzQLghcfDyLtGxUs7no= 2023-07-29T01:22:54.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VMDvXjs1rzQLghcfDyLtGxUs7no=', 'Content-Length': '0'} 2023-07-29T01:22:54.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efb1e43051348f85-0064c45aaf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=b 2023-07-29T01:22:54.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=b 2023-07-29T01:22:54.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=b 2023-07-29T01:22:54.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.698 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.699 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.699 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.699 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.699 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.700 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:ViuMHmbO3wFHX4uFdsobzPFwMFk= 2023-07-29T01:22:54.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ViuMHmbO3wFHX4uFdsobzPFwMFk=', 'Content-Length': '0'} 2023-07-29T01:22:54.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005736aa35aee835aa-0064c45aaf-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55b1000falseb2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=b 2023-07-29T01:22:54.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=b 2023-07-29T01:22:54.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=b 2023-07-29T01:22:54.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.704 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.704 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.704 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.704 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.705 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.705 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:ViuMHmbO3wFHX4uFdsobzPFwMFk= 2023-07-29T01:22:54.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ViuMHmbO3wFHX4uFdsobzPFwMFk=', 'Content-Length': '0'} 2023-07-29T01:22:54.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031dfcaba6f6ebfdd-0064c45aaf-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55b1000falseb2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.706 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-07-29T01:22:54.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c 2023-07-29T01:22:54.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c 2023-07-29T01:22:54.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c?versionId=null 2023-07-29T01:22:54.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c?versionId=null 2023-07-29T01:22:54.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/c?versionId=null 2023-07-29T01:22:54.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.709 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.709 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.710 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.710 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.710 INFO:tasks.rgw_multisite_tests:/hiiqos-55/c?versionId=null 2023-07-29T01:22:54.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.711 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:YAnGgL8OAhPLvsjllCt2QFxMvls= 2023-07-29T01:22:54.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YAnGgL8OAhPLvsjllCt2QFxMvls=', 'Content-Length': '0'} 2023-07-29T01:22:54.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0bdf7a2cca4c099-0064c45aaf-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c 2023-07-29T01:22:54.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c 2023-07-29T01:22:54.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c?versionId=null 2023-07-29T01:22:54.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c?versionId=null 2023-07-29T01:22:54.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/c?versionId=null 2023-07-29T01:22:54.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.714 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.715 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.715 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.715 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.715 INFO:tasks.rgw_multisite_tests:/hiiqos-55/c?versionId=null 2023-07-29T01:22:54.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.716 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:YAnGgL8OAhPLvsjllCt2QFxMvls= 2023-07-29T01:22:54.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YAnGgL8OAhPLvsjllCt2QFxMvls=', 'Content-Length': '0'} 2023-07-29T01:22:54.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ade5ad9acc43ea3-0064c45aaf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c 2023-07-29T01:22:54.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c 2023-07-29T01:22:54.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c?versionId=null 2023-07-29T01:22:54.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c?versionId=null 2023-07-29T01:22:54.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/c?versionId=null 2023-07-29T01:22:54.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.719 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.720 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.720 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.720 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.720 INFO:tasks.rgw_multisite_tests:/hiiqos-55/c?versionId=null 2023-07-29T01:22:54.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.721 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:ZthH3qNkOgcqhxzwANftn3UWglE= 2023-07-29T01:22:54.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZthH3qNkOgcqhxzwANftn3UWglE=', 'Content-Length': '0'} 2023-07-29T01:22:54.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fcaeb35c7056457-0064c45aaf-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c 2023-07-29T01:22:54.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c 2023-07-29T01:22:54.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c?versionId=null 2023-07-29T01:22:54.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c?versionId=null 2023-07-29T01:22:54.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/c?versionId=null 2023-07-29T01:22:54.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.724 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.725 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.725 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.725 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.725 INFO:tasks.rgw_multisite_tests:/hiiqos-55/c?versionId=null 2023-07-29T01:22:54.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.726 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:ZthH3qNkOgcqhxzwANftn3UWglE= 2023-07-29T01:22:54.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZthH3qNkOgcqhxzwANftn3UWglE=', 'Content-Length': '0'} 2023-07-29T01:22:54.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008219af0940008497-0064c45aaf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.727 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-07-29T01:22:54.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c 2023-07-29T01:22:54.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c 2023-07-29T01:22:54.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c?versionId=null 2023-07-29T01:22:54.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c?versionId=null 2023-07-29T01:22:54.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/c?versionId=null 2023-07-29T01:22:54.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.730 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.730 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.730 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.730 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.731 INFO:tasks.rgw_multisite_tests:/hiiqos-55/c?versionId=null 2023-07-29T01:22:54.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.731 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:YAnGgL8OAhPLvsjllCt2QFxMvls= 2023-07-29T01:22:54.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YAnGgL8OAhPLvsjllCt2QFxMvls=', 'Content-Length': '0'} 2023-07-29T01:22:54.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be75d7525b4245e6-0064c45aaf-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c 2023-07-29T01:22:54.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c 2023-07-29T01:22:54.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c?versionId=null 2023-07-29T01:22:54.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c?versionId=null 2023-07-29T01:22:54.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/c?versionId=null 2023-07-29T01:22:54.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.735 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.735 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.735 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.735 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.736 INFO:tasks.rgw_multisite_tests:/hiiqos-55/c?versionId=null 2023-07-29T01:22:54.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.736 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:YAnGgL8OAhPLvsjllCt2QFxMvls= 2023-07-29T01:22:54.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YAnGgL8OAhPLvsjllCt2QFxMvls=', 'Content-Length': '0'} 2023-07-29T01:22:54.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060dcdc8c04b51011-0064c45aaf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=c 2023-07-29T01:22:54.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=c 2023-07-29T01:22:54.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=c 2023-07-29T01:22:54.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.740 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.740 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.740 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.741 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.741 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.741 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:ViuMHmbO3wFHX4uFdsobzPFwMFk= 2023-07-29T01:22:54.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ViuMHmbO3wFHX4uFdsobzPFwMFk=', 'Content-Length': '0'} 2023-07-29T01:22:54.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afa3d1db99c0bd4c-0064c45aaf-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55c1000falsec2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=c 2023-07-29T01:22:54.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=c 2023-07-29T01:22:54.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=c 2023-07-29T01:22:54.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.745 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.745 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.746 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.746 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.746 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.747 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:ViuMHmbO3wFHX4uFdsobzPFwMFk= 2023-07-29T01:22:54.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ViuMHmbO3wFHX4uFdsobzPFwMFk=', 'Content-Length': '0'} 2023-07-29T01:22:54.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000177dd346ad7cc145-0064c45aaf-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55c1000falsec2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c 2023-07-29T01:22:54.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c 2023-07-29T01:22:54.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/c 2023-07-29T01:22:54.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.750 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.750 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.751 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.751 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.751 INFO:tasks.rgw_multisite_tests:/hiiqos-55/c 2023-07-29T01:22:54.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.751 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:SBwl7ICkhbIsIF4nZ0KaBwypOvY= 2023-07-29T01:22:54.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SBwl7ICkhbIsIF4nZ0KaBwypOvY=', 'Content-Length': '0'} 2023-07-29T01:22:54.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e101e0c38092bbf2-0064c45aaf-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c 2023-07-29T01:22:54.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c 2023-07-29T01:22:54.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/c 2023-07-29T01:22:54.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.755 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.755 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.755 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.755 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.756 INFO:tasks.rgw_multisite_tests:/hiiqos-55/c 2023-07-29T01:22:54.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.756 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:SBwl7ICkhbIsIF4nZ0KaBwypOvY= 2023-07-29T01:22:54.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SBwl7ICkhbIsIF4nZ0KaBwypOvY=', 'Content-Length': '0'} 2023-07-29T01:22:54.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b882c32e3fb00164-0064c45aaf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.757 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-07-29T01:22:54.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c 2023-07-29T01:22:54.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c 2023-07-29T01:22:54.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/c 2023-07-29T01:22:54.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.760 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.760 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.760 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.760 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.760 INFO:tasks.rgw_multisite_tests:/hiiqos-55/c 2023-07-29T01:22:54.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.761 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:I/IkE5WOh5yt9p/ptd3zOOL5dA8= 2023-07-29T01:22:54.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I/IkE5WOh5yt9p/ptd3zOOL5dA8=', 'Content-Length': '0'} 2023-07-29T01:22:54.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007564ab6c72d456d-0064c45aaf-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c 2023-07-29T01:22:54.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c 2023-07-29T01:22:54.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/c 2023-07-29T01:22:54.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.764 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.764 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.765 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.765 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:51 GMT 2023-07-29T01:22:54.765 INFO:tasks.rgw_multisite_tests:/hiiqos-55/c 2023-07-29T01:22:54.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.766 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:I/IkE5WOh5yt9p/ptd3zOOL5dA8= 2023-07-29T01:22:54.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:51 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:I/IkE5WOh5yt9p/ptd3zOOL5dA8=', 'Content-Length': '0'} 2023-07-29T01:22:54.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042120bc7553f86e3-0064c45aaf-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=c 2023-07-29T01:22:54.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=c 2023-07-29T01:22:54.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=c 2023-07-29T01:22:54.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.769 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.770 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.770 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.770 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.770 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.771 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T01:22:54.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T01:22:54.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d4f19e5531f42bc-0064c45aaf-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55c1000falsec2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=c 2023-07-29T01:22:54.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=c 2023-07-29T01:22:54.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=c 2023-07-29T01:22:54.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.775 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.775 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.775 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.775 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.776 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.776 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T01:22:54.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T01:22:54.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa290e70b24479bf-0064c45aaf-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55c1000falsec2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.777 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-07-29T01:22:54.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d 2023-07-29T01:22:54.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d 2023-07-29T01:22:54.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d?versionId=null 2023-07-29T01:22:54.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d?versionId=null 2023-07-29T01:22:54.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/d?versionId=null 2023-07-29T01:22:54.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.780 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.781 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.781 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.781 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.781 INFO:tasks.rgw_multisite_tests:/hiiqos-55/d?versionId=null 2023-07-29T01:22:54.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.782 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:TlZH8Pn8FdUzbwrkh/xPyg3TCgs= 2023-07-29T01:22:54.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TlZH8Pn8FdUzbwrkh/xPyg3TCgs=', 'Content-Length': '0'} 2023-07-29T01:22:54.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9268d239f60b174-0064c45aaf-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:51 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d 2023-07-29T01:22:54.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d 2023-07-29T01:22:54.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d?versionId=null 2023-07-29T01:22:54.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d?versionId=null 2023-07-29T01:22:54.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/d?versionId=null 2023-07-29T01:22:54.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.785 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.786 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.786 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.786 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.786 INFO:tasks.rgw_multisite_tests:/hiiqos-55/d?versionId=null 2023-07-29T01:22:54.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.787 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:TlZH8Pn8FdUzbwrkh/xPyg3TCgs= 2023-07-29T01:22:54.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TlZH8Pn8FdUzbwrkh/xPyg3TCgs=', 'Content-Length': '0'} 2023-07-29T01:22:54.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d972603c6251719-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d 2023-07-29T01:22:54.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d 2023-07-29T01:22:54.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d?versionId=null 2023-07-29T01:22:54.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d?versionId=null 2023-07-29T01:22:54.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/d?versionId=null 2023-07-29T01:22:54.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.791 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.791 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.791 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.791 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.791 INFO:tasks.rgw_multisite_tests:/hiiqos-55/d?versionId=null 2023-07-29T01:22:54.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.792 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:lN85Q9cM22ERpzAKcYPyYmsxToc= 2023-07-29T01:22:54.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lN85Q9cM22ERpzAKcYPyYmsxToc=', 'Content-Length': '0'} 2023-07-29T01:22:54.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da2e5c45caa1ee85-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d 2023-07-29T01:22:54.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d 2023-07-29T01:22:54.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d?versionId=null 2023-07-29T01:22:54.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d?versionId=null 2023-07-29T01:22:54.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/d?versionId=null 2023-07-29T01:22:54.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.796 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.796 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.796 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.796 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.797 INFO:tasks.rgw_multisite_tests:/hiiqos-55/d?versionId=null 2023-07-29T01:22:54.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.797 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:lN85Q9cM22ERpzAKcYPyYmsxToc= 2023-07-29T01:22:54.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lN85Q9cM22ERpzAKcYPyYmsxToc=', 'Content-Length': '0'} 2023-07-29T01:22:54.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070203ba4e7d867f3-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.798 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-07-29T01:22:54.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d 2023-07-29T01:22:54.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d 2023-07-29T01:22:54.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d?versionId=null 2023-07-29T01:22:54.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d?versionId=null 2023-07-29T01:22:54.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/d?versionId=null 2023-07-29T01:22:54.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.801 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.801 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.801 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.802 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.802 INFO:tasks.rgw_multisite_tests:/hiiqos-55/d?versionId=null 2023-07-29T01:22:54.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.802 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:TlZH8Pn8FdUzbwrkh/xPyg3TCgs= 2023-07-29T01:22:54.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TlZH8Pn8FdUzbwrkh/xPyg3TCgs=', 'Content-Length': '0'} 2023-07-29T01:22:54.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c846ee8e30ba607-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d 2023-07-29T01:22:54.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d 2023-07-29T01:22:54.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d?versionId=null 2023-07-29T01:22:54.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d?versionId=null 2023-07-29T01:22:54.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/d?versionId=null 2023-07-29T01:22:54.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.806 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.806 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.806 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.807 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.807 INFO:tasks.rgw_multisite_tests:/hiiqos-55/d?versionId=null 2023-07-29T01:22:54.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.807 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:TlZH8Pn8FdUzbwrkh/xPyg3TCgs= 2023-07-29T01:22:54.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TlZH8Pn8FdUzbwrkh/xPyg3TCgs=', 'Content-Length': '0'} 2023-07-29T01:22:54.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000432769ff14a36c30-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=d 2023-07-29T01:22:54.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=d 2023-07-29T01:22:54.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=d 2023-07-29T01:22:54.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.811 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.811 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.812 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.812 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.812 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.813 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T01:22:54.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T01:22:54.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000249a28b1f763824b-0064c45ab0-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55d1000falsed2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=d 2023-07-29T01:22:54.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=d 2023-07-29T01:22:54.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=d 2023-07-29T01:22:54.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.816 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.817 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.817 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.817 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.817 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.818 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T01:22:54.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T01:22:54.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c4c89f5d8b921c9-0064c45ab0-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55d1000falsed2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d 2023-07-29T01:22:54.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d 2023-07-29T01:22:54.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/d 2023-07-29T01:22:54.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.821 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.822 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.822 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.822 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.822 INFO:tasks.rgw_multisite_tests:/hiiqos-55/d 2023-07-29T01:22:54.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.823 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:ImAhZGRvKKXv+DmADCWJaUW6tgo= 2023-07-29T01:22:54.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ImAhZGRvKKXv+DmADCWJaUW6tgo=', 'Content-Length': '0'} 2023-07-29T01:22:54.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011f0caa0e74b4997-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d 2023-07-29T01:22:54.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d 2023-07-29T01:22:54.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/d 2023-07-29T01:22:54.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.826 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.826 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.826 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.827 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.827 INFO:tasks.rgw_multisite_tests:/hiiqos-55/d 2023-07-29T01:22:54.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.827 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:ImAhZGRvKKXv+DmADCWJaUW6tgo= 2023-07-29T01:22:54.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ImAhZGRvKKXv+DmADCWJaUW6tgo=', 'Content-Length': '0'} 2023-07-29T01:22:54.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c05c10a737259b12-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.828 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-07-29T01:22:54.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d 2023-07-29T01:22:54.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d 2023-07-29T01:22:54.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/d 2023-07-29T01:22:54.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.831 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.831 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.831 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.832 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.832 INFO:tasks.rgw_multisite_tests:/hiiqos-55/d 2023-07-29T01:22:54.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.832 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:HT+lbcN3VtKadl92wINcp5Z9skk= 2023-07-29T01:22:54.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HT+lbcN3VtKadl92wINcp5Z9skk=', 'Content-Length': '0'} 2023-07-29T01:22:54.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d59ee5858f6514b2-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d 2023-07-29T01:22:54.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d 2023-07-29T01:22:54.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/d 2023-07-29T01:22:54.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.835 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.836 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.836 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.836 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.836 INFO:tasks.rgw_multisite_tests:/hiiqos-55/d 2023-07-29T01:22:54.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.837 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:HT+lbcN3VtKadl92wINcp5Z9skk= 2023-07-29T01:22:54.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:HT+lbcN3VtKadl92wINcp5Z9skk=', 'Content-Length': '0'} 2023-07-29T01:22:54.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af0db3789ccf4b14-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=d 2023-07-29T01:22:54.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=d 2023-07-29T01:22:54.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=d 2023-07-29T01:22:54.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.841 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.841 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.841 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.841 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.841 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.842 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T01:22:54.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T01:22:54.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a2080980f354d29-0064c45ab0-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55d1000falsed2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=d 2023-07-29T01:22:54.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=d 2023-07-29T01:22:54.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=d 2023-07-29T01:22:54.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.846 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.846 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.846 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.847 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.847 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.847 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T01:22:54.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T01:22:54.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012ee2600564992b6-0064c45ab0-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55d1000falsed2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.848 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2023-07-29T01:22:54.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/e 2023-07-29T01:22:54.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/e 2023-07-29T01:22:54.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/e?versionId=null 2023-07-29T01:22:54.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/e?versionId=null 2023-07-29T01:22:54.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/e?versionId=null 2023-07-29T01:22:54.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.851 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.852 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.852 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.852 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.852 INFO:tasks.rgw_multisite_tests:/hiiqos-55/e?versionId=null 2023-07-29T01:22:54.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.853 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:N3uBRkJaALjdWYqGPneTe3xR1gw= 2023-07-29T01:22:54.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:N3uBRkJaALjdWYqGPneTe3xR1gw=', 'Content-Length': '0'} 2023-07-29T01:22:54.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d26d4a9040c2e79a-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/e 2023-07-29T01:22:54.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/e 2023-07-29T01:22:54.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/e?versionId=null 2023-07-29T01:22:54.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/e?versionId=null 2023-07-29T01:22:54.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/e?versionId=null 2023-07-29T01:22:54.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.857 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.857 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.857 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.857 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.857 INFO:tasks.rgw_multisite_tests:/hiiqos-55/e?versionId=null 2023-07-29T01:22:54.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.858 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:N3uBRkJaALjdWYqGPneTe3xR1gw= 2023-07-29T01:22:54.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:N3uBRkJaALjdWYqGPneTe3xR1gw=', 'Content-Length': '0'} 2023-07-29T01:22:54.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004373202868bd0f57-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/e 2023-07-29T01:22:54.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/e 2023-07-29T01:22:54.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/e?versionId=null 2023-07-29T01:22:54.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/e?versionId=null 2023-07-29T01:22:54.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/e?versionId=null 2023-07-29T01:22:54.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.862 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.862 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.862 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.862 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.863 INFO:tasks.rgw_multisite_tests:/hiiqos-55/e?versionId=null 2023-07-29T01:22:54.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.863 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:gEvnc1UMMCN8fiNCcgF3RYoBzrM= 2023-07-29T01:22:54.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gEvnc1UMMCN8fiNCcgF3RYoBzrM=', 'Content-Length': '0'} 2023-07-29T01:22:54.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b5435277d8bf831-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/e 2023-07-29T01:22:54.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/e 2023-07-29T01:22:54.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/e?versionId=null 2023-07-29T01:22:54.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/e?versionId=null 2023-07-29T01:22:54.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/e?versionId=null 2023-07-29T01:22:54.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.867 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.867 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.867 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.867 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.868 INFO:tasks.rgw_multisite_tests:/hiiqos-55/e?versionId=null 2023-07-29T01:22:54.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.868 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:gEvnc1UMMCN8fiNCcgF3RYoBzrM= 2023-07-29T01:22:54.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gEvnc1UMMCN8fiNCcgF3RYoBzrM=', 'Content-Length': '0'} 2023-07-29T01:22:54.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083475c4476da0b80-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.869 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2023-07-29T01:22:54.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/e 2023-07-29T01:22:54.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/e 2023-07-29T01:22:54.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/e?versionId=null 2023-07-29T01:22:54.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/e?versionId=null 2023-07-29T01:22:54.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/e?versionId=null 2023-07-29T01:22:54.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.872 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.872 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.873 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.873 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.873 INFO:tasks.rgw_multisite_tests:/hiiqos-55/e?versionId=null 2023-07-29T01:22:54.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.873 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:N3uBRkJaALjdWYqGPneTe3xR1gw= 2023-07-29T01:22:54.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:N3uBRkJaALjdWYqGPneTe3xR1gw=', 'Content-Length': '0'} 2023-07-29T01:22:54.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008972ee1a0b127a0b-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/e 2023-07-29T01:22:54.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/e 2023-07-29T01:22:54.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/e?versionId=null 2023-07-29T01:22:54.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/e?versionId=null 2023-07-29T01:22:54.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/e?versionId=null 2023-07-29T01:22:54.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.877 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.877 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.878 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.878 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.878 INFO:tasks.rgw_multisite_tests:/hiiqos-55/e?versionId=null 2023-07-29T01:22:54.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.879 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:N3uBRkJaALjdWYqGPneTe3xR1gw= 2023-07-29T01:22:54.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:N3uBRkJaALjdWYqGPneTe3xR1gw=', 'Content-Length': '0'} 2023-07-29T01:22:54.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000630dedb542ac032a-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=e 2023-07-29T01:22:54.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=e 2023-07-29T01:22:54.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=e 2023-07-29T01:22:54.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.882 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.883 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.883 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.883 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.883 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.884 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T01:22:54.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T01:22:54.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1e85a493360275d-0064c45ab0-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55e1000falsee2023-07-29T00:15:00.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=e 2023-07-29T01:22:54.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=e 2023-07-29T01:22:54.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=e 2023-07-29T01:22:54.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.888 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.888 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.888 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.888 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.889 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.889 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T01:22:54.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T01:22:54.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0b81074a5a417d0-0064c45ab0-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55e1000falsee2023-07-29T00:15:00.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/e 2023-07-29T01:22:54.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/e 2023-07-29T01:22:54.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/e 2023-07-29T01:22:54.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.892 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.893 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.893 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.893 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.893 INFO:tasks.rgw_multisite_tests:/hiiqos-55/e 2023-07-29T01:22:54.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.894 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:SgWvTdm5halu9UEj51FV4sX1P4E= 2023-07-29T01:22:54.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SgWvTdm5halu9UEj51FV4sX1P4E=', 'Content-Length': '0'} 2023-07-29T01:22:54.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000790df59022b3b1af-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/e 2023-07-29T01:22:54.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/e 2023-07-29T01:22:54.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/e 2023-07-29T01:22:54.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.897 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.897 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.898 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.898 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.898 INFO:tasks.rgw_multisite_tests:/hiiqos-55/e 2023-07-29T01:22:54.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.898 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:SgWvTdm5halu9UEj51FV4sX1P4E= 2023-07-29T01:22:54.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:SgWvTdm5halu9UEj51FV4sX1P4E=', 'Content-Length': '0'} 2023-07-29T01:22:54.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d58b51a2051ab62-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.899 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2023-07-29T01:22:54.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/e 2023-07-29T01:22:54.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/e 2023-07-29T01:22:54.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/e 2023-07-29T01:22:54.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.902 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.902 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.902 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.903 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.903 INFO:tasks.rgw_multisite_tests:/hiiqos-55/e 2023-07-29T01:22:54.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.903 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:YxNeDhSQ/lfu2QO3WerXuLe7XSo= 2023-07-29T01:22:54.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YxNeDhSQ/lfu2QO3WerXuLe7XSo=', 'Content-Length': '0'} 2023-07-29T01:22:54.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf6dc73622af9cdd-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/e 2023-07-29T01:22:54.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/e 2023-07-29T01:22:54.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/e 2023-07-29T01:22:54.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.907 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.907 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.907 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.907 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.907 INFO:tasks.rgw_multisite_tests:/hiiqos-55/e 2023-07-29T01:22:54.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.908 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:YxNeDhSQ/lfu2QO3WerXuLe7XSo= 2023-07-29T01:22:54.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YxNeDhSQ/lfu2QO3WerXuLe7XSo=', 'Content-Length': '0'} 2023-07-29T01:22:54.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005001bcc8041f564-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=e 2023-07-29T01:22:54.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=e 2023-07-29T01:22:54.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=e 2023-07-29T01:22:54.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.912 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.912 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.912 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.912 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.913 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.913 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T01:22:54.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T01:22:54.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b35f907dc0a680e-0064c45ab0-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55e1000falsee2023-07-29T00:15:00.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=e 2023-07-29T01:22:54.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=e 2023-07-29T01:22:54.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=e 2023-07-29T01:22:54.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.917 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.917 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.917 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.918 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.918 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.918 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T01:22:54.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T01:22:54.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031238a5a6413035c-0064c45ab0-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55e1000falsee2023-07-29T00:15:00.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.919 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2023-07-29T01:22:54.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/f 2023-07-29T01:22:54.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/f 2023-07-29T01:22:54.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/f?versionId=null 2023-07-29T01:22:54.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/f?versionId=null 2023-07-29T01:22:54.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/f?versionId=null 2023-07-29T01:22:54.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.923 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.923 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.923 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.923 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.923 INFO:tasks.rgw_multisite_tests:/hiiqos-55/f?versionId=null 2023-07-29T01:22:54.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.924 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:oPKjocaacwAeufKTae4WMimfxpk= 2023-07-29T01:22:54.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oPKjocaacwAeufKTae4WMimfxpk=', 'Content-Length': '0'} 2023-07-29T01:22:54.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003483810c790d6c46-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/f 2023-07-29T01:22:54.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/f 2023-07-29T01:22:54.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/f?versionId=null 2023-07-29T01:22:54.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/f?versionId=null 2023-07-29T01:22:54.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/f?versionId=null 2023-07-29T01:22:54.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.928 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.928 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.928 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.928 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.929 INFO:tasks.rgw_multisite_tests:/hiiqos-55/f?versionId=null 2023-07-29T01:22:54.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.929 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:oPKjocaacwAeufKTae4WMimfxpk= 2023-07-29T01:22:54.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oPKjocaacwAeufKTae4WMimfxpk=', 'Content-Length': '0'} 2023-07-29T01:22:54.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099503e7261290d66-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/f 2023-07-29T01:22:54.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/f 2023-07-29T01:22:54.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/f?versionId=null 2023-07-29T01:22:54.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/f?versionId=null 2023-07-29T01:22:54.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/f?versionId=null 2023-07-29T01:22:54.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.933 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.933 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.933 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.933 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.934 INFO:tasks.rgw_multisite_tests:/hiiqos-55/f?versionId=null 2023-07-29T01:22:54.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.934 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:VX//KDpAivFYiTnzXu27tVbaLhQ= 2023-07-29T01:22:54.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VX//KDpAivFYiTnzXu27tVbaLhQ=', 'Content-Length': '0'} 2023-07-29T01:22:54.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce015b253e0bab7d-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/f 2023-07-29T01:22:54.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/f 2023-07-29T01:22:54.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/f?versionId=null 2023-07-29T01:22:54.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/f?versionId=null 2023-07-29T01:22:54.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/f?versionId=null 2023-07-29T01:22:54.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.938 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.938 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.938 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.939 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.939 INFO:tasks.rgw_multisite_tests:/hiiqos-55/f?versionId=null 2023-07-29T01:22:54.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.939 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:VX//KDpAivFYiTnzXu27tVbaLhQ= 2023-07-29T01:22:54.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VX//KDpAivFYiTnzXu27tVbaLhQ=', 'Content-Length': '0'} 2023-07-29T01:22:54.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022a5abaccabd8c32-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.940 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2023-07-29T01:22:54.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/f 2023-07-29T01:22:54.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/f 2023-07-29T01:22:54.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/f?versionId=null 2023-07-29T01:22:54.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/f?versionId=null 2023-07-29T01:22:54.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/f?versionId=null 2023-07-29T01:22:54.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.943 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.943 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.944 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.944 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.944 INFO:tasks.rgw_multisite_tests:/hiiqos-55/f?versionId=null 2023-07-29T01:22:54.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.945 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:oPKjocaacwAeufKTae4WMimfxpk= 2023-07-29T01:22:54.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oPKjocaacwAeufKTae4WMimfxpk=', 'Content-Length': '0'} 2023-07-29T01:22:54.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb3693e28b767431-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/f 2023-07-29T01:22:54.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/f 2023-07-29T01:22:54.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/f?versionId=null 2023-07-29T01:22:54.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/f?versionId=null 2023-07-29T01:22:54.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/f?versionId=null 2023-07-29T01:22:54.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.948 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.949 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.949 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.949 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.949 INFO:tasks.rgw_multisite_tests:/hiiqos-55/f?versionId=null 2023-07-29T01:22:54.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.950 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:oPKjocaacwAeufKTae4WMimfxpk= 2023-07-29T01:22:54.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:oPKjocaacwAeufKTae4WMimfxpk=', 'Content-Length': '0'} 2023-07-29T01:22:54.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000587ba367585c93c6-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=f 2023-07-29T01:22:54.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=f 2023-07-29T01:22:54.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=f 2023-07-29T01:22:54.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.953 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.954 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.954 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.954 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.954 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.955 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T01:22:54.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T01:22:54.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8b1a70d2bd9b193-0064c45ab0-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55f1000falsef2023-07-29T00:15:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=f 2023-07-29T01:22:54.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=f 2023-07-29T01:22:54.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=f 2023-07-29T01:22:54.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.959 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.959 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.959 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.959 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.960 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.960 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T01:22:54.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T01:22:54.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006274d6c33fad8a99-0064c45ab0-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55f1000falsef2023-07-29T00:15:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/f 2023-07-29T01:22:54.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/f 2023-07-29T01:22:54.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/f 2023-07-29T01:22:54.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.964 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.964 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.964 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.964 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.965 INFO:tasks.rgw_multisite_tests:/hiiqos-55/f 2023-07-29T01:22:54.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.965 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:dPtyzLDV1b6Yqrf28S//81SkE/0= 2023-07-29T01:22:54.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dPtyzLDV1b6Yqrf28S//81SkE/0=', 'Content-Length': '0'} 2023-07-29T01:22:54.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ca3f1998c05b17c-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/f 2023-07-29T01:22:54.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/f 2023-07-29T01:22:54.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:54.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/f 2023-07-29T01:22:54.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.968 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:54.969 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.969 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.969 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.969 INFO:tasks.rgw_multisite_tests:/hiiqos-55/f 2023-07-29T01:22:54.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.970 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:dPtyzLDV1b6Yqrf28S//81SkE/0= 2023-07-29T01:22:54.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dPtyzLDV1b6Yqrf28S//81SkE/0=', 'Content-Length': '0'} 2023-07-29T01:22:54.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e03fa0ab61b4f36-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.970 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2023-07-29T01:22:54.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/f 2023-07-29T01:22:54.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/f 2023-07-29T01:22:54.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/f 2023-07-29T01:22:54.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.973 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.973 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.974 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.974 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.974 INFO:tasks.rgw_multisite_tests:/hiiqos-55/f 2023-07-29T01:22:54.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.974 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:mTgT+KfdB6wYOWIZ6BMsqVEEoNs= 2023-07-29T01:22:54.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mTgT+KfdB6wYOWIZ6BMsqVEEoNs=', 'Content-Length': '0'} 2023-07-29T01:22:54.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cbe500111f3f202-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/f 2023-07-29T01:22:54.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/f 2023-07-29T01:22:54.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/f 2023-07-29T01:22:54.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.978 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.978 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.978 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.978 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.979 INFO:tasks.rgw_multisite_tests:/hiiqos-55/f 2023-07-29T01:22:54.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.979 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:mTgT+KfdB6wYOWIZ6BMsqVEEoNs= 2023-07-29T01:22:54.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mTgT+KfdB6wYOWIZ6BMsqVEEoNs=', 'Content-Length': '0'} 2023-07-29T01:22:54.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ead464c090fd8bb-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=f 2023-07-29T01:22:54.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=f 2023-07-29T01:22:54.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=f 2023-07-29T01:22:54.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.983 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.983 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.983 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.983 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.984 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.984 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T01:22:54.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T01:22:54.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057da9643c0159c5e-0064c45ab0-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55f1000falsef2023-07-29T00:15:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:54.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:54.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=f 2023-07-29T01:22:54.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=f 2023-07-29T01:22:54.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=f 2023-07-29T01:22:54.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.988 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.988 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.989 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.989 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.989 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:54.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.990 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T01:22:54.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T01:22:54.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d66be81f32fe6f9-0064c45ab0-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55f1000falsef2023-07-29T00:15:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:54.990 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2023-07-29T01:22:54.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/g 2023-07-29T01:22:54.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/g 2023-07-29T01:22:54.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/g?versionId=null 2023-07-29T01:22:54.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/g?versionId=null 2023-07-29T01:22:54.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/g?versionId=null 2023-07-29T01:22:54.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:54.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:54.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.994 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.994 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.994 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.994 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:54.995 INFO:tasks.rgw_multisite_tests:/hiiqos-55/g?versionId=null 2023-07-29T01:22:54.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:54.995 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:IzU7oefaY1bviV3nIiOVEj01G8Y= 2023-07-29T01:22:54.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IzU7oefaY1bviV3nIiOVEj01G8Y=', 'Content-Length': '0'} 2023-07-29T01:22:54.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000193e7837c393c3bb-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:54.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/g 2023-07-29T01:22:54.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/g 2023-07-29T01:22:54.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/g?versionId=null 2023-07-29T01:22:54.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/g?versionId=null 2023-07-29T01:22:54.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:54.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/g?versionId=null 2023-07-29T01:22:54.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:54.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:54.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:54.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:54.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:54.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:54.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:54.999 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:54.999 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:54.999 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.000 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:55.000 INFO:tasks.rgw_multisite_tests:/hiiqos-55/g?versionId=null 2023-07-29T01:22:55.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.000 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:IzU7oefaY1bviV3nIiOVEj01G8Y= 2023-07-29T01:22:55.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IzU7oefaY1bviV3nIiOVEj01G8Y=', 'Content-Length': '0'} 2023-07-29T01:22:55.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f74116d3c78d689-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/g 2023-07-29T01:22:55.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/g 2023-07-29T01:22:55.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/g?versionId=null 2023-07-29T01:22:55.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/g?versionId=null 2023-07-29T01:22:55.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/g?versionId=null 2023-07-29T01:22:55.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.004 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.004 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.004 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.005 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:55.005 INFO:tasks.rgw_multisite_tests:/hiiqos-55/g?versionId=null 2023-07-29T01:22:55.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.005 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:Q/yqVXQpcZZ/bXbp1gBVOaL8fr4= 2023-07-29T01:22:55.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q/yqVXQpcZZ/bXbp1gBVOaL8fr4=', 'Content-Length': '0'} 2023-07-29T01:22:55.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbda2a0bf0971958-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/g 2023-07-29T01:22:55.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/g 2023-07-29T01:22:55.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/g?versionId=null 2023-07-29T01:22:55.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/g?versionId=null 2023-07-29T01:22:55.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/g?versionId=null 2023-07-29T01:22:55.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.009 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.009 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.009 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.010 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:55.010 INFO:tasks.rgw_multisite_tests:/hiiqos-55/g?versionId=null 2023-07-29T01:22:55.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.010 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:Q/yqVXQpcZZ/bXbp1gBVOaL8fr4= 2023-07-29T01:22:55.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q/yqVXQpcZZ/bXbp1gBVOaL8fr4=', 'Content-Length': '0'} 2023-07-29T01:22:55.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d99a7e52f565f72-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.011 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2023-07-29T01:22:55.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/g 2023-07-29T01:22:55.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/g 2023-07-29T01:22:55.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/g?versionId=null 2023-07-29T01:22:55.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/g?versionId=null 2023-07-29T01:22:55.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/g?versionId=null 2023-07-29T01:22:55.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.014 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.015 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.015 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.015 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:55.015 INFO:tasks.rgw_multisite_tests:/hiiqos-55/g?versionId=null 2023-07-29T01:22:55.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.016 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:IzU7oefaY1bviV3nIiOVEj01G8Y= 2023-07-29T01:22:55.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IzU7oefaY1bviV3nIiOVEj01G8Y=', 'Content-Length': '0'} 2023-07-29T01:22:55.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf425de4bf3e2491-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/g 2023-07-29T01:22:55.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/g 2023-07-29T01:22:55.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/g?versionId=null 2023-07-29T01:22:55.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/g?versionId=null 2023-07-29T01:22:55.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/g?versionId=null 2023-07-29T01:22:55.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.019 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.020 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.020 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.020 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:55.020 INFO:tasks.rgw_multisite_tests:/hiiqos-55/g?versionId=null 2023-07-29T01:22:55.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.021 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:IzU7oefaY1bviV3nIiOVEj01G8Y= 2023-07-29T01:22:55.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IzU7oefaY1bviV3nIiOVEj01G8Y=', 'Content-Length': '0'} 2023-07-29T01:22:55.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000048d9a5cc6ec3a8e-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=g 2023-07-29T01:22:55.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=g 2023-07-29T01:22:55.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=g 2023-07-29T01:22:55.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.024 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.025 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.025 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.025 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:55.025 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:55.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.026 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T01:22:55.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T01:22:55.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f90c91286bed80cd-0064c45ab0-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55g1000falseg2023-07-29T00:15:01.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=g 2023-07-29T01:22:55.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=g 2023-07-29T01:22:55.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=g 2023-07-29T01:22:55.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.030 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.030 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.030 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.030 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:55.031 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:55.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.031 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T01:22:55.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T01:22:55.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc60fa27340da666-0064c45ab0-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55g1000falseg2023-07-29T00:15:01.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/g 2023-07-29T01:22:55.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/g 2023-07-29T01:22:55.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/g 2023-07-29T01:22:55.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.035 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.035 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.035 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.035 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:55.036 INFO:tasks.rgw_multisite_tests:/hiiqos-55/g 2023-07-29T01:22:55.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.036 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:QxryRujaF8rFUevgWB9LcvBjN+E= 2023-07-29T01:22:55.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QxryRujaF8rFUevgWB9LcvBjN+E=', 'Content-Length': '0'} 2023-07-29T01:22:55.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bde0d67e70c38c05-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/g 2023-07-29T01:22:55.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/g 2023-07-29T01:22:55.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/g 2023-07-29T01:22:55.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.039 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.040 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.040 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.040 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:55.040 INFO:tasks.rgw_multisite_tests:/hiiqos-55/g 2023-07-29T01:22:55.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.041 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:QxryRujaF8rFUevgWB9LcvBjN+E= 2023-07-29T01:22:55.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QxryRujaF8rFUevgWB9LcvBjN+E=', 'Content-Length': '0'} 2023-07-29T01:22:55.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011e15deafe34baeb-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.041 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2023-07-29T01:22:55.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/g 2023-07-29T01:22:55.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/g 2023-07-29T01:22:55.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/g 2023-07-29T01:22:55.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.044 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.044 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.045 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.045 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:55.045 INFO:tasks.rgw_multisite_tests:/hiiqos-55/g 2023-07-29T01:22:55.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.046 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:Lflq/q/L06omp8cwv/ba+0g9MTs= 2023-07-29T01:22:55.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Lflq/q/L06omp8cwv/ba+0g9MTs=', 'Content-Length': '0'} 2023-07-29T01:22:55.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d5adbef7ad18ead-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/g 2023-07-29T01:22:55.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/g 2023-07-29T01:22:55.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/g 2023-07-29T01:22:55.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.049 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.049 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.049 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.049 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:55.050 INFO:tasks.rgw_multisite_tests:/hiiqos-55/g 2023-07-29T01:22:55.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.050 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:Lflq/q/L06omp8cwv/ba+0g9MTs= 2023-07-29T01:22:55.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Lflq/q/L06omp8cwv/ba+0g9MTs=', 'Content-Length': '0'} 2023-07-29T01:22:55.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e5a5990e1f57479-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=g 2023-07-29T01:22:55.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=g 2023-07-29T01:22:55.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=g 2023-07-29T01:22:55.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.054 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.054 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.054 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.055 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:55.055 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:55.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.055 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T01:22:55.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T01:22:55.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003003618b1be507b9-0064c45ab0-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55g1000falseg2023-07-29T00:15:01.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=g 2023-07-29T01:22:55.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=g 2023-07-29T01:22:55.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=g 2023-07-29T01:22:55.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.059 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.059 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.060 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.060 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:55.060 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:55.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.061 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T01:22:55.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T01:22:55.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d992e58a0fef678d-0064c45ab0-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55g1000falseg2023-07-29T00:15:01.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.062 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2023-07-29T01:22:55.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/h 2023-07-29T01:22:55.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/h 2023-07-29T01:22:55.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/h?versionId=null 2023-07-29T01:22:55.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/h?versionId=null 2023-07-29T01:22:55.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/h?versionId=null 2023-07-29T01:22:55.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.065 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.065 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.065 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.065 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:55.066 INFO:tasks.rgw_multisite_tests:/hiiqos-55/h?versionId=null 2023-07-29T01:22:55.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.066 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:ZX0bat4jFiw5mdqaom7C1S6xQCY= 2023-07-29T01:22:55.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZX0bat4jFiw5mdqaom7C1S6xQCY=', 'Content-Length': '0'} 2023-07-29T01:22:55.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d93beb4bf4bd5748-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/h 2023-07-29T01:22:55.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/h 2023-07-29T01:22:55.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/h?versionId=null 2023-07-29T01:22:55.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/h?versionId=null 2023-07-29T01:22:55.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/h?versionId=null 2023-07-29T01:22:55.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.070 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.070 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.070 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.071 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:55.071 INFO:tasks.rgw_multisite_tests:/hiiqos-55/h?versionId=null 2023-07-29T01:22:55.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.071 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:ZX0bat4jFiw5mdqaom7C1S6xQCY= 2023-07-29T01:22:55.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZX0bat4jFiw5mdqaom7C1S6xQCY=', 'Content-Length': '0'} 2023-07-29T01:22:55.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ed3f4083f27005b-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/h 2023-07-29T01:22:55.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/h 2023-07-29T01:22:55.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/h?versionId=null 2023-07-29T01:22:55.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/h?versionId=null 2023-07-29T01:22:55.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/h?versionId=null 2023-07-29T01:22:55.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.075 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.075 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.075 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.076 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:55.076 INFO:tasks.rgw_multisite_tests:/hiiqos-55/h?versionId=null 2023-07-29T01:22:55.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.076 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:7uSLB+kRtp7NRIh0+tHASwEKFJM= 2023-07-29T01:22:55.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7uSLB+kRtp7NRIh0+tHASwEKFJM=', 'Content-Length': '0'} 2023-07-29T01:22:55.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac6d46c542d001ff-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/h 2023-07-29T01:22:55.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/h 2023-07-29T01:22:55.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/h?versionId=null 2023-07-29T01:22:55.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/h?versionId=null 2023-07-29T01:22:55.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/h?versionId=null 2023-07-29T01:22:55.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.080 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.080 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.080 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.081 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:55.081 INFO:tasks.rgw_multisite_tests:/hiiqos-55/h?versionId=null 2023-07-29T01:22:55.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.081 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:7uSLB+kRtp7NRIh0+tHASwEKFJM= 2023-07-29T01:22:55.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7uSLB+kRtp7NRIh0+tHASwEKFJM=', 'Content-Length': '0'} 2023-07-29T01:22:55.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e898eb45fc9c94f4-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.082 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2023-07-29T01:22:55.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/h 2023-07-29T01:22:55.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/h 2023-07-29T01:22:55.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/h?versionId=null 2023-07-29T01:22:55.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/h?versionId=null 2023-07-29T01:22:55.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/h?versionId=null 2023-07-29T01:22:55.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.085 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.086 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.086 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.086 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:55.086 INFO:tasks.rgw_multisite_tests:/hiiqos-55/h?versionId=null 2023-07-29T01:22:55.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.087 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:ZX0bat4jFiw5mdqaom7C1S6xQCY= 2023-07-29T01:22:55.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZX0bat4jFiw5mdqaom7C1S6xQCY=', 'Content-Length': '0'} 2023-07-29T01:22:55.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a20eda04f1de0637-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/h 2023-07-29T01:22:55.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/h 2023-07-29T01:22:55.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/h?versionId=null 2023-07-29T01:22:55.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/h?versionId=null 2023-07-29T01:22:55.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/h?versionId=null 2023-07-29T01:22:55.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.090 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.091 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.091 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.091 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:55.091 INFO:tasks.rgw_multisite_tests:/hiiqos-55/h?versionId=null 2023-07-29T01:22:55.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.092 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:ZX0bat4jFiw5mdqaom7C1S6xQCY= 2023-07-29T01:22:55.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZX0bat4jFiw5mdqaom7C1S6xQCY=', 'Content-Length': '0'} 2023-07-29T01:22:55.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000314d547e5297ddef-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=h 2023-07-29T01:22:55.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=h 2023-07-29T01:22:55.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=h 2023-07-29T01:22:55.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.095 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.096 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.096 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.096 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:55.096 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:55.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.097 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T01:22:55.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T01:22:55.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008028961dd08ecc06-0064c45ab0-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55h1000falseh2023-07-29T00:15:01.033Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=h 2023-07-29T01:22:55.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=h 2023-07-29T01:22:55.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=h 2023-07-29T01:22:55.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.101 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.101 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.101 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.101 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:55.102 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:55.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.102 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T01:22:55.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T01:22:55.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a39f56dfd58fad1-0064c45ab0-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55h1000falseh2023-07-29T00:15:01.033Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/h 2023-07-29T01:22:55.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/h 2023-07-29T01:22:55.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/h 2023-07-29T01:22:55.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.106 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.106 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.106 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.106 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:55.107 INFO:tasks.rgw_multisite_tests:/hiiqos-55/h 2023-07-29T01:22:55.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.107 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:skFzbXXcGibuPHPSRPhi/hHw9r4= 2023-07-29T01:22:55.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:skFzbXXcGibuPHPSRPhi/hHw9r4=', 'Content-Length': '0'} 2023-07-29T01:22:55.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae27e2664064634f-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/h 2023-07-29T01:22:55.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/h 2023-07-29T01:22:55.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/h 2023-07-29T01:22:55.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.110 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.110 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.111 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.111 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:55.111 INFO:tasks.rgw_multisite_tests:/hiiqos-55/h 2023-07-29T01:22:55.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.112 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:skFzbXXcGibuPHPSRPhi/hHw9r4= 2023-07-29T01:22:55.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:skFzbXXcGibuPHPSRPhi/hHw9r4=', 'Content-Length': '0'} 2023-07-29T01:22:55.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009723aa127059076b-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.112 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2023-07-29T01:22:55.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/h 2023-07-29T01:22:55.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/h 2023-07-29T01:22:55.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/h 2023-07-29T01:22:55.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.115 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.115 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.116 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.116 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:55.116 INFO:tasks.rgw_multisite_tests:/hiiqos-55/h 2023-07-29T01:22:55.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.116 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:1AhkFk7LYGQONbZN3SfIj+1M53c= 2023-07-29T01:22:55.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1AhkFk7LYGQONbZN3SfIj+1M53c=', 'Content-Length': '0'} 2023-07-29T01:22:55.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d65cf23586dcd09-0064c45ab0-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/h 2023-07-29T01:22:55.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/h 2023-07-29T01:22:55.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/h 2023-07-29T01:22:55.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.120 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.120 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.120 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.120 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:55.121 INFO:tasks.rgw_multisite_tests:/hiiqos-55/h 2023-07-29T01:22:55.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.121 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:1AhkFk7LYGQONbZN3SfIj+1M53c= 2023-07-29T01:22:55.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1AhkFk7LYGQONbZN3SfIj+1M53c=', 'Content-Length': '0'} 2023-07-29T01:22:55.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085e1812d984bf485-0064c45ab0-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=h 2023-07-29T01:22:55.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=h 2023-07-29T01:22:55.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=h 2023-07-29T01:22:55.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.125 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.125 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.125 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.126 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:55.126 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:55.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.126 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T01:22:55.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T01:22:55.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c1403d11bbf1693-0064c45ab0-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55h1000falseh2023-07-29T00:15:01.033Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=h 2023-07-29T01:22:55.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=h 2023-07-29T01:22:55.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=h 2023-07-29T01:22:55.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.130 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.130 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.131 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.131 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:52 GMT 2023-07-29T01:22:55.131 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:55.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.132 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs= 2023-07-29T01:22:55.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:c99iRvtx/AMC9Zdn5cErvd+dBxs=', 'Content-Length': '0'} 2023-07-29T01:22:55.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035f5b21c532acc62-0064c45ab0-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55h1000falseh2023-07-29T00:15:01.033Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.133 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=hiiqos-55 zones={a2, a1} 2023-07-29T01:22:55.133 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'hiiqos-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:55.133 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'hiiqos-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:55.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket hiiqos-55 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:22:55.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:55.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:55.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: tenant: 2023-07-29T01:22:55.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: bucket name: hiiqos-55 2023-07-29T01:22:55.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: total entries: 8 2023-07-29T01:22:55.135 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'hiiqos-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:55.135 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'hiiqos-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:55.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket hiiqos-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:22:55.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:55.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:55.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:55.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "layout": { 2023-07-29T01:22:55.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "resharding": "None", 2023-07-29T01:22:55.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_index": { 2023-07-29T01:22:55.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "gen": 2, 2023-07-29T01:22:55.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "layout": { 2023-07-29T01:22:55.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "type": "Normal", 2023-07-29T01:22:55.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "normal": { 2023-07-29T01:22:55.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 3, 2023-07-29T01:22:55.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "hash_type": "Mod" 2023-07-29T01:22:55.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:55.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "logs": [ 2023-07-29T01:22:55.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:55.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "gen": 0, 2023-07-29T01:22:55.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "layout": { 2023-07-29T01:22:55.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "type": "InIndex", 2023-07-29T01:22:55.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "in_index": { 2023-07-29T01:22:55.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "gen": 0, 2023-07-29T01:22:55.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "layout": { 2023-07-29T01:22:55.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 11, 2023-07-29T01:22:55.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "hash_type": "Mod" 2023-07-29T01:22:55.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:55.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:55.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "gen": 1, 2023-07-29T01:22:55.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "layout": { 2023-07-29T01:22:55.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "type": "InIndex", 2023-07-29T01:22:55.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "in_index": { 2023-07-29T01:22:55.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "gen": 1, 2023-07-29T01:22:55.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "layout": { 2023-07-29T01:22:55.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 3, 2023-07-29T01:22:55.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "hash_type": "Mod" 2023-07-29T01:22:55.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:55.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:55.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "gen": 2, 2023-07-29T01:22:55.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "layout": { 2023-07-29T01:22:55.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "type": "InIndex", 2023-07-29T01:22:55.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "in_index": { 2023-07-29T01:22:55.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "gen": 2, 2023-07-29T01:22:55.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "layout": { 2023-07-29T01:22:55.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 3, 2023-07-29T01:22:55.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "hash_type": "Mod" 2023-07-29T01:22:55.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:55.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.149 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=hiiqos-55 2023-07-29T01:22:55.149 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T01:22:55.149 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T01:22:55.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T01:22:55.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:55.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:55.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:17:53.292+0000 7ff75a135880 1 waiting to reach latest gen 2, on 1.. 2023-07-29T01:22:55.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:17:58.293+0000 7ff75a135880 1 bucket sync caught up with empty source 2023-07-29T01:22:55.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:17:58.293+0000 7ff75a135880 0 bucket checkpoint complete 2023-07-29T01:22:55.151 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=hiiqos-55 2023-07-29T01:22:55.151 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T01:22:55.151 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T01:22:55.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T01:22:55.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:55.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:55.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:17:58.405+0000 7fd187f26880 1 bucket sync caught up with source: 2023-07-29T01:22:55.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: local status: [, 00000000002.714.6, , , 00000000001.1347.6, 00000000001.707.6, , , 00000000002.386.6, , 00000000002.1169.6] 2023-07-29T01:22:55.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: remote markers: [, 00000000002.714.6, , , 00000000001.1347.6, 00000000001.707.6, , , 00000000002.386.6, , 00000000002.1169.6] 2023-07-29T01:22:55.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:17:58.405+0000 7fd187f26880 0 bucket checkpoint complete 2023-07-29T01:22:55.153 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=hiiqos-55 zones={a2, a1} 2023-07-29T01:22:55.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/ 2023-07-29T01:22:55.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.156 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.156 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.156 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.157 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.157 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:55.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.157 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:t3BmvdFePG5jN7YR86Da8NTlLX8= 2023-07-29T01:22:55.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t3BmvdFePG5jN7YR86Da8NTlLX8=', 'Content-Length': '0'} 2023-07-29T01:22:55.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000595b2192f6874fbe-0064c45ab6-1663-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/ 2023-07-29T01:22:55.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.161 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.161 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.161 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.161 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.161 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:55.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.162 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:t3BmvdFePG5jN7YR86Da8NTlLX8= 2023-07-29T01:22:55.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t3BmvdFePG5jN7YR86Da8NTlLX8=', 'Content-Length': '0'} 2023-07-29T01:22:55.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000062d5f1618c5e584e-0064c45ab6-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.163 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2023-07-29T01:22:55.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?versions&max-keys=999 2023-07-29T01:22:55.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?versions&max-keys=999 2023-07-29T01:22:55.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?versions&max-keys=999 2023-07-29T01:22:55.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.166 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.166 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.166 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.166 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.167 INFO:tasks.rgw_multisite_tests:/hiiqos-55/?versions 2023-07-29T01:22:55.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.167 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:nkO23lZHuGy1GRu9+rLtGNhwFFA= 2023-07-29T01:22:55.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nkO23lZHuGy1GRu9+rLtGNhwFFA=', 'Content-Length': '0'} 2023-07-29T01:22:55.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080b4886b16007f7d-0064c45ab6-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55999falseanulltrue2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-07-29T00:15:00.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-07-29T00:15:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-07-29T00:15:01.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-07-29T00:15:01.033Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.168 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2023-07-29T01:22:55.168 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2023-07-29T01:22:55.169 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2023-07-29T01:22:55.169 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2023-07-29T01:22:55.169 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2023-07-29T01:22:55.169 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2023-07-29T01:22:55.170 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2023-07-29T01:22:55.170 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2023-07-29T01:22:55.170 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2023-07-29T01:22:55.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?versions&max-keys=999 2023-07-29T01:22:55.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?versions&max-keys=999 2023-07-29T01:22:55.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?versions&max-keys=999 2023-07-29T01:22:55.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.173 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.173 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.174 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.174 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.174 INFO:tasks.rgw_multisite_tests:/hiiqos-55/?versions 2023-07-29T01:22:55.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.175 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:nkO23lZHuGy1GRu9+rLtGNhwFFA= 2023-07-29T01:22:55.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nkO23lZHuGy1GRu9+rLtGNhwFFA=', 'Content-Length': '0'} 2023-07-29T01:22:55.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd795f2900cae01a-0064c45ab6-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55999falseanulltrue2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-07-29T00:15:00.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-07-29T00:15:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-07-29T00:15:01.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-07-29T00:15:01.033Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.176 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2023-07-29T01:22:55.176 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2023-07-29T01:22:55.176 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2023-07-29T01:22:55.176 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2023-07-29T01:22:55.176 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2023-07-29T01:22:55.177 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2023-07-29T01:22:55.177 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2023-07-29T01:22:55.177 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2023-07-29T01:22:55.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?versions&max-keys=999 2023-07-29T01:22:55.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?versions&max-keys=999 2023-07-29T01:22:55.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?versions&max-keys=999 2023-07-29T01:22:55.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.180 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.181 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.181 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.181 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.181 INFO:tasks.rgw_multisite_tests:/hiiqos-55/?versions 2023-07-29T01:22:55.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.182 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:nkO23lZHuGy1GRu9+rLtGNhwFFA= 2023-07-29T01:22:55.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nkO23lZHuGy1GRu9+rLtGNhwFFA=', 'Content-Length': '0'} 2023-07-29T01:22:55.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000397f81de5e512417-0064c45ab6-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55999falseanulltrue2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-07-29T00:15:00.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-07-29T00:15:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-07-29T00:15:01.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-07-29T00:15:01.033Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?versions&max-keys=999 2023-07-29T01:22:55.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?versions&max-keys=999 2023-07-29T01:22:55.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?versions&max-keys=999 2023-07-29T01:22:55.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.186 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.186 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.186 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.186 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.187 INFO:tasks.rgw_multisite_tests:/hiiqos-55/?versions 2023-07-29T01:22:55.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.187 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:nkO23lZHuGy1GRu9+rLtGNhwFFA= 2023-07-29T01:22:55.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:nkO23lZHuGy1GRu9+rLtGNhwFFA=', 'Content-Length': '0'} 2023-07-29T01:22:55.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000875153d43b6a0d9f-0064c45ab6-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55999falseanulltrue2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-07-29T00:15:00.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-07-29T00:15:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-07-29T00:15:01.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-07-29T00:15:01.033Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.188 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-07-29T01:22:55.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a 2023-07-29T01:22:55.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a 2023-07-29T01:22:55.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a?versionId=null 2023-07-29T01:22:55.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a?versionId=null 2023-07-29T01:22:55.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/a?versionId=null 2023-07-29T01:22:55.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.191 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.192 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.192 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.192 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.192 INFO:tasks.rgw_multisite_tests:/hiiqos-55/a?versionId=null 2023-07-29T01:22:55.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.193 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:FLpEUX7sV/aMDO7rT2S5vRKCWbQ= 2023-07-29T01:22:55.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FLpEUX7sV/aMDO7rT2S5vRKCWbQ=', 'Content-Length': '0'} 2023-07-29T01:22:55.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc5b4220253a8d4a-0064c45ab6-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a 2023-07-29T01:22:55.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a 2023-07-29T01:22:55.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a?versionId=null 2023-07-29T01:22:55.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a?versionId=null 2023-07-29T01:22:55.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/a?versionId=null 2023-07-29T01:22:55.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.196 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.197 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.197 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.197 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.197 INFO:tasks.rgw_multisite_tests:/hiiqos-55/a?versionId=null 2023-07-29T01:22:55.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.198 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:FLpEUX7sV/aMDO7rT2S5vRKCWbQ= 2023-07-29T01:22:55.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FLpEUX7sV/aMDO7rT2S5vRKCWbQ=', 'Content-Length': '0'} 2023-07-29T01:22:55.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000208daf13698bf92d-0064c45ab6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a 2023-07-29T01:22:55.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a 2023-07-29T01:22:55.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a?versionId=null 2023-07-29T01:22:55.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a?versionId=null 2023-07-29T01:22:55.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/a?versionId=null 2023-07-29T01:22:55.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.201 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.202 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.202 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.202 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.202 INFO:tasks.rgw_multisite_tests:/hiiqos-55/a?versionId=null 2023-07-29T01:22:55.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.203 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:x1PArLs/dXmbVDoWX7D8h1wmzks= 2023-07-29T01:22:55.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:x1PArLs/dXmbVDoWX7D8h1wmzks=', 'Content-Length': '0'} 2023-07-29T01:22:55.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080fb246b4c3d97c5-0064c45ab6-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a 2023-07-29T01:22:55.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a 2023-07-29T01:22:55.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a?versionId=null 2023-07-29T01:22:55.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a?versionId=null 2023-07-29T01:22:55.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/a?versionId=null 2023-07-29T01:22:55.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.207 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.207 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.207 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.207 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.208 INFO:tasks.rgw_multisite_tests:/hiiqos-55/a?versionId=null 2023-07-29T01:22:55.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.208 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:x1PArLs/dXmbVDoWX7D8h1wmzks= 2023-07-29T01:22:55.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:x1PArLs/dXmbVDoWX7D8h1wmzks=', 'Content-Length': '0'} 2023-07-29T01:22:55.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d516d7e44917c09d-0064c45ab6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.209 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-07-29T01:22:55.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a 2023-07-29T01:22:55.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a 2023-07-29T01:22:55.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a?versionId=null 2023-07-29T01:22:55.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a?versionId=null 2023-07-29T01:22:55.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/a?versionId=null 2023-07-29T01:22:55.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.212 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.212 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.212 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.213 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.213 INFO:tasks.rgw_multisite_tests:/hiiqos-55/a?versionId=null 2023-07-29T01:22:55.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.213 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:FLpEUX7sV/aMDO7rT2S5vRKCWbQ= 2023-07-29T01:22:55.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FLpEUX7sV/aMDO7rT2S5vRKCWbQ=', 'Content-Length': '0'} 2023-07-29T01:22:55.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d3b5c37d359afd2-0064c45ab6-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a 2023-07-29T01:22:55.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a 2023-07-29T01:22:55.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a?versionId=null 2023-07-29T01:22:55.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a?versionId=null 2023-07-29T01:22:55.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/a?versionId=null 2023-07-29T01:22:55.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.217 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.217 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.219 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.220 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.220 INFO:tasks.rgw_multisite_tests:/hiiqos-55/a?versionId=null 2023-07-29T01:22:55.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.220 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:FLpEUX7sV/aMDO7rT2S5vRKCWbQ= 2023-07-29T01:22:55.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FLpEUX7sV/aMDO7rT2S5vRKCWbQ=', 'Content-Length': '0'} 2023-07-29T01:22:55.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000154b1861952f482c-0064c45ab6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=a 2023-07-29T01:22:55.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=a 2023-07-29T01:22:55.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=a 2023-07-29T01:22:55.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.224 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.224 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.224 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.225 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.225 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:55.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.225 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0= 2023-07-29T01:22:55.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0=', 'Content-Length': '0'} 2023-07-29T01:22:55.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000195a36f67e346268-0064c45ab6-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55a1000falsea2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=a 2023-07-29T01:22:55.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=a 2023-07-29T01:22:55.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=a 2023-07-29T01:22:55.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.229 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.230 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.230 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.230 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.230 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:55.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.231 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0= 2023-07-29T01:22:55.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0=', 'Content-Length': '0'} 2023-07-29T01:22:55.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000451ff7ca570b4be3-0064c45ab6-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55a1000falsea2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a 2023-07-29T01:22:55.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a 2023-07-29T01:22:55.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/a 2023-07-29T01:22:55.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.234 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.234 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.235 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.235 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.235 INFO:tasks.rgw_multisite_tests:/hiiqos-55/a 2023-07-29T01:22:55.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.236 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:vaTRr3LNOYjSJ/0XsCl26xX2gD8= 2023-07-29T01:22:55.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vaTRr3LNOYjSJ/0XsCl26xX2gD8=', 'Content-Length': '0'} 2023-07-29T01:22:55.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008e03213ca6ebee3-0064c45ab6-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a 2023-07-29T01:22:55.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a 2023-07-29T01:22:55.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/a 2023-07-29T01:22:55.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.239 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.239 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.239 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.240 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.240 INFO:tasks.rgw_multisite_tests:/hiiqos-55/a 2023-07-29T01:22:55.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.240 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:vaTRr3LNOYjSJ/0XsCl26xX2gD8= 2023-07-29T01:22:55.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:vaTRr3LNOYjSJ/0XsCl26xX2gD8=', 'Content-Length': '0'} 2023-07-29T01:22:55.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000650f9da0342e2027-0064c45ab6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.241 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-07-29T01:22:55.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a 2023-07-29T01:22:55.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a 2023-07-29T01:22:55.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/a 2023-07-29T01:22:55.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.244 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.244 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.244 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.244 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.245 INFO:tasks.rgw_multisite_tests:/hiiqos-55/a 2023-07-29T01:22:55.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.245 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:jbWIkcwnNqnA+a2EoaUQKHtDObo= 2023-07-29T01:22:55.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jbWIkcwnNqnA+a2EoaUQKHtDObo=', 'Content-Length': '0'} 2023-07-29T01:22:55.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000129a48f453ee88e8-0064c45ab6-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/a 2023-07-29T01:22:55.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/a 2023-07-29T01:22:55.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/a 2023-07-29T01:22:55.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.248 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.249 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.249 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.249 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.249 INFO:tasks.rgw_multisite_tests:/hiiqos-55/a 2023-07-29T01:22:55.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.250 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:jbWIkcwnNqnA+a2EoaUQKHtDObo= 2023-07-29T01:22:55.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jbWIkcwnNqnA+a2EoaUQKHtDObo=', 'Content-Length': '0'} 2023-07-29T01:22:55.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043f6b9a6e65329ef-0064c45ab6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=a 2023-07-29T01:22:55.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=a 2023-07-29T01:22:55.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=a 2023-07-29T01:22:55.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.253 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.254 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.254 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.254 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.254 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:55.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.255 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0= 2023-07-29T01:22:55.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0=', 'Content-Length': '0'} 2023-07-29T01:22:55.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bcaccaaaf5ab687-0064c45ab6-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55a1000falsea2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=a 2023-07-29T01:22:55.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=a 2023-07-29T01:22:55.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=a 2023-07-29T01:22:55.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.259 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.259 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.259 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.259 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.260 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:55.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.260 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0= 2023-07-29T01:22:55.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0=', 'Content-Length': '0'} 2023-07-29T01:22:55.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6741626a4e7e98f-0064c45ab6-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55a1000falsea2023-07-29T00:14:43.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.261 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-07-29T01:22:55.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b 2023-07-29T01:22:55.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b 2023-07-29T01:22:55.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b?versionId=null 2023-07-29T01:22:55.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b?versionId=null 2023-07-29T01:22:55.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/b?versionId=null 2023-07-29T01:22:55.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.264 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.265 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.265 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.265 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.265 INFO:tasks.rgw_multisite_tests:/hiiqos-55/b?versionId=null 2023-07-29T01:22:55.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.266 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:F40X/fWMus0VglY4g75YfbW+5Gc= 2023-07-29T01:22:55.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:F40X/fWMus0VglY4g75YfbW+5Gc=', 'Content-Length': '0'} 2023-07-29T01:22:55.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089473e88239a03aa-0064c45ab6-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b 2023-07-29T01:22:55.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b 2023-07-29T01:22:55.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b?versionId=null 2023-07-29T01:22:55.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b?versionId=null 2023-07-29T01:22:55.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/b?versionId=null 2023-07-29T01:22:55.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.269 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.270 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.270 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.270 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.270 INFO:tasks.rgw_multisite_tests:/hiiqos-55/b?versionId=null 2023-07-29T01:22:55.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.271 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:F40X/fWMus0VglY4g75YfbW+5Gc= 2023-07-29T01:22:55.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:F40X/fWMus0VglY4g75YfbW+5Gc=', 'Content-Length': '0'} 2023-07-29T01:22:55.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000536bd1a74974ad15-0064c45ab6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b 2023-07-29T01:22:55.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b 2023-07-29T01:22:55.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b?versionId=null 2023-07-29T01:22:55.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b?versionId=null 2023-07-29T01:22:55.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/b?versionId=null 2023-07-29T01:22:55.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.275 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.275 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.275 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.275 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.275 INFO:tasks.rgw_multisite_tests:/hiiqos-55/b?versionId=null 2023-07-29T01:22:55.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.276 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:KM9msgrVxaszSpAPyPsyIBQoi4c= 2023-07-29T01:22:55.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KM9msgrVxaszSpAPyPsyIBQoi4c=', 'Content-Length': '0'} 2023-07-29T01:22:55.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c30d8b135018450d-0064c45ab6-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b 2023-07-29T01:22:55.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b 2023-07-29T01:22:55.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b?versionId=null 2023-07-29T01:22:55.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b?versionId=null 2023-07-29T01:22:55.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/b?versionId=null 2023-07-29T01:22:55.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.280 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.280 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.280 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.280 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.281 INFO:tasks.rgw_multisite_tests:/hiiqos-55/b?versionId=null 2023-07-29T01:22:55.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.281 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:KM9msgrVxaszSpAPyPsyIBQoi4c= 2023-07-29T01:22:55.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KM9msgrVxaszSpAPyPsyIBQoi4c=', 'Content-Length': '0'} 2023-07-29T01:22:55.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3a89c5a96c33d50-0064c45ab6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.282 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-07-29T01:22:55.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b 2023-07-29T01:22:55.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b 2023-07-29T01:22:55.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b?versionId=null 2023-07-29T01:22:55.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b?versionId=null 2023-07-29T01:22:55.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/b?versionId=null 2023-07-29T01:22:55.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.285 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.285 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.285 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.286 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.286 INFO:tasks.rgw_multisite_tests:/hiiqos-55/b?versionId=null 2023-07-29T01:22:55.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.286 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:F40X/fWMus0VglY4g75YfbW+5Gc= 2023-07-29T01:22:55.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:F40X/fWMus0VglY4g75YfbW+5Gc=', 'Content-Length': '0'} 2023-07-29T01:22:55.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093c5a7c05bfc6243-0064c45ab6-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b 2023-07-29T01:22:55.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b 2023-07-29T01:22:55.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b?versionId=null 2023-07-29T01:22:55.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b?versionId=null 2023-07-29T01:22:55.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/b?versionId=null 2023-07-29T01:22:55.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.290 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.290 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.290 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.291 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.291 INFO:tasks.rgw_multisite_tests:/hiiqos-55/b?versionId=null 2023-07-29T01:22:55.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.291 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:F40X/fWMus0VglY4g75YfbW+5Gc= 2023-07-29T01:22:55.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:F40X/fWMus0VglY4g75YfbW+5Gc=', 'Content-Length': '0'} 2023-07-29T01:22:55.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001654eaf368c6dd71-0064c45ab6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=b 2023-07-29T01:22:55.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=b 2023-07-29T01:22:55.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=b 2023-07-29T01:22:55.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.295 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.295 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.296 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.296 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.296 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:55.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.296 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0= 2023-07-29T01:22:55.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0=', 'Content-Length': '0'} 2023-07-29T01:22:55.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a959977f4c6e829b-0064c45ab6-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55b1000falseb2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=b 2023-07-29T01:22:55.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=b 2023-07-29T01:22:55.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=b 2023-07-29T01:22:55.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.300 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.301 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.301 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.301 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.301 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:55.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.302 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0= 2023-07-29T01:22:55.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0=', 'Content-Length': '0'} 2023-07-29T01:22:55.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac301243df40d177-0064c45ab6-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55b1000falseb2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b 2023-07-29T01:22:55.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b 2023-07-29T01:22:55.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/b 2023-07-29T01:22:55.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.305 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.306 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.306 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.306 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.306 INFO:tasks.rgw_multisite_tests:/hiiqos-55/b 2023-07-29T01:22:55.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.307 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:UZL2zVcfSioLjNM2EB5i90prEFg= 2023-07-29T01:22:55.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UZL2zVcfSioLjNM2EB5i90prEFg=', 'Content-Length': '0'} 2023-07-29T01:22:55.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000162a096e6afa9dd3-0064c45ab6-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b 2023-07-29T01:22:55.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b 2023-07-29T01:22:55.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/b 2023-07-29T01:22:55.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.310 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.310 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.310 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.311 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.311 INFO:tasks.rgw_multisite_tests:/hiiqos-55/b 2023-07-29T01:22:55.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.311 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:UZL2zVcfSioLjNM2EB5i90prEFg= 2023-07-29T01:22:55.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UZL2zVcfSioLjNM2EB5i90prEFg=', 'Content-Length': '0'} 2023-07-29T01:22:55.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1b2e660499c6acc-0064c45ab6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.312 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-07-29T01:22:55.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b 2023-07-29T01:22:55.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b 2023-07-29T01:22:55.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/b 2023-07-29T01:22:55.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.315 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.315 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.315 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.315 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.316 INFO:tasks.rgw_multisite_tests:/hiiqos-55/b 2023-07-29T01:22:55.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.316 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:YUgMFWbH8rRwqAFjfcnARdRhZfQ= 2023-07-29T01:22:55.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YUgMFWbH8rRwqAFjfcnARdRhZfQ=', 'Content-Length': '0'} 2023-07-29T01:22:55.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f0610a37245d16a-0064c45ab6-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/b 2023-07-29T01:22:55.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/b 2023-07-29T01:22:55.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/b 2023-07-29T01:22:55.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.319 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.320 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.320 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.320 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.320 INFO:tasks.rgw_multisite_tests:/hiiqos-55/b 2023-07-29T01:22:55.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.321 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:YUgMFWbH8rRwqAFjfcnARdRhZfQ= 2023-07-29T01:22:55.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:YUgMFWbH8rRwqAFjfcnARdRhZfQ=', 'Content-Length': '0'} 2023-07-29T01:22:55.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e222814f0641eb1-0064c45ab6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=b 2023-07-29T01:22:55.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=b 2023-07-29T01:22:55.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=b 2023-07-29T01:22:55.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.324 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.325 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.325 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.325 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.325 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:55.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.326 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0= 2023-07-29T01:22:55.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0=', 'Content-Length': '0'} 2023-07-29T01:22:55.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000950b59d064400c31-0064c45ab6-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55b1000falseb2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=b 2023-07-29T01:22:55.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=b 2023-07-29T01:22:55.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=b 2023-07-29T01:22:55.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.330 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.330 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.330 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.330 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.331 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:55.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.331 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0= 2023-07-29T01:22:55.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0=', 'Content-Length': '0'} 2023-07-29T01:22:55.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be1b991e1357ae82-0064c45ab6-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55b1000falseb2023-07-29T00:14:43.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.332 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-07-29T01:22:55.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c 2023-07-29T01:22:55.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c 2023-07-29T01:22:55.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c?versionId=null 2023-07-29T01:22:55.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c?versionId=null 2023-07-29T01:22:55.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/c?versionId=null 2023-07-29T01:22:55.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.335 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.335 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.336 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.336 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.336 INFO:tasks.rgw_multisite_tests:/hiiqos-55/c?versionId=null 2023-07-29T01:22:55.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.337 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:in0lt/VJy+SUY7l7ukxeojvT/Zw= 2023-07-29T01:22:55.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:in0lt/VJy+SUY7l7ukxeojvT/Zw=', 'Content-Length': '0'} 2023-07-29T01:22:55.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088b0439a358db043-0064c45ab6-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c 2023-07-29T01:22:55.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c 2023-07-29T01:22:55.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c?versionId=null 2023-07-29T01:22:55.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c?versionId=null 2023-07-29T01:22:55.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/c?versionId=null 2023-07-29T01:22:55.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.340 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.341 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.341 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.341 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.341 INFO:tasks.rgw_multisite_tests:/hiiqos-55/c?versionId=null 2023-07-29T01:22:55.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.342 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:in0lt/VJy+SUY7l7ukxeojvT/Zw= 2023-07-29T01:22:55.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:in0lt/VJy+SUY7l7ukxeojvT/Zw=', 'Content-Length': '0'} 2023-07-29T01:22:55.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000422d1f69b81f5b78-0064c45ab6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c 2023-07-29T01:22:55.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c 2023-07-29T01:22:55.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c?versionId=null 2023-07-29T01:22:55.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c?versionId=null 2023-07-29T01:22:55.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/c?versionId=null 2023-07-29T01:22:55.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.345 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.346 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.346 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.346 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.346 INFO:tasks.rgw_multisite_tests:/hiiqos-55/c?versionId=null 2023-07-29T01:22:55.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.347 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:M2kiVXMdN1Bb5T+of81Txn3s958= 2023-07-29T01:22:55.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:M2kiVXMdN1Bb5T+of81Txn3s958=', 'Content-Length': '0'} 2023-07-29T01:22:55.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000817d65fb19713beb-0064c45ab6-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c 2023-07-29T01:22:55.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c 2023-07-29T01:22:55.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c?versionId=null 2023-07-29T01:22:55.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c?versionId=null 2023-07-29T01:22:55.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/c?versionId=null 2023-07-29T01:22:55.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.350 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.351 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.351 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.351 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.351 INFO:tasks.rgw_multisite_tests:/hiiqos-55/c?versionId=null 2023-07-29T01:22:55.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.352 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:M2kiVXMdN1Bb5T+of81Txn3s958= 2023-07-29T01:22:55.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:M2kiVXMdN1Bb5T+of81Txn3s958=', 'Content-Length': '0'} 2023-07-29T01:22:55.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058cfc84e7f9173e2-0064c45ab6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.353 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-07-29T01:22:55.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c 2023-07-29T01:22:55.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c 2023-07-29T01:22:55.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c?versionId=null 2023-07-29T01:22:55.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c?versionId=null 2023-07-29T01:22:55.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/c?versionId=null 2023-07-29T01:22:55.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.356 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.356 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.356 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.356 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.357 INFO:tasks.rgw_multisite_tests:/hiiqos-55/c?versionId=null 2023-07-29T01:22:55.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.357 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:in0lt/VJy+SUY7l7ukxeojvT/Zw= 2023-07-29T01:22:55.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:in0lt/VJy+SUY7l7ukxeojvT/Zw=', 'Content-Length': '0'} 2023-07-29T01:22:55.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d7e98540e8c084d-0064c45ab6-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c 2023-07-29T01:22:55.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c 2023-07-29T01:22:55.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c?versionId=null 2023-07-29T01:22:55.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c?versionId=null 2023-07-29T01:22:55.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/c?versionId=null 2023-07-29T01:22:55.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.361 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.361 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.361 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.362 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.362 INFO:tasks.rgw_multisite_tests:/hiiqos-55/c?versionId=null 2023-07-29T01:22:55.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.362 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:in0lt/VJy+SUY7l7ukxeojvT/Zw= 2023-07-29T01:22:55.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:in0lt/VJy+SUY7l7ukxeojvT/Zw=', 'Content-Length': '0'} 2023-07-29T01:22:55.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e80422942ee77ae7-0064c45ab6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=c 2023-07-29T01:22:55.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=c 2023-07-29T01:22:55.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=c 2023-07-29T01:22:55.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.366 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.366 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.366 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.367 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.367 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:55.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.367 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0= 2023-07-29T01:22:55.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0=', 'Content-Length': '0'} 2023-07-29T01:22:55.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000396588c58c5757c2-0064c45ab6-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55c1000falsec2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=c 2023-07-29T01:22:55.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=c 2023-07-29T01:22:55.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=c 2023-07-29T01:22:55.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.371 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.371 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.372 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.372 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.372 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:55.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.373 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0= 2023-07-29T01:22:55.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0=', 'Content-Length': '0'} 2023-07-29T01:22:55.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b30d55281470da69-0064c45ab6-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55c1000falsec2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c 2023-07-29T01:22:55.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c 2023-07-29T01:22:55.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/c 2023-07-29T01:22:55.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.376 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.376 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.377 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.377 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.377 INFO:tasks.rgw_multisite_tests:/hiiqos-55/c 2023-07-29T01:22:55.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.377 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:Sn0Wzw5kIMGHTkEvKPnrZBf0rDA= 2023-07-29T01:22:55.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Sn0Wzw5kIMGHTkEvKPnrZBf0rDA=', 'Content-Length': '0'} 2023-07-29T01:22:55.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c55ced919a98346-0064c45ab6-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c 2023-07-29T01:22:55.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c 2023-07-29T01:22:55.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/c 2023-07-29T01:22:55.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.381 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.381 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.381 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.381 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.382 INFO:tasks.rgw_multisite_tests:/hiiqos-55/c 2023-07-29T01:22:55.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.382 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:Sn0Wzw5kIMGHTkEvKPnrZBf0rDA= 2023-07-29T01:22:55.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Sn0Wzw5kIMGHTkEvKPnrZBf0rDA=', 'Content-Length': '0'} 2023-07-29T01:22:55.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d44c50646b3b776-0064c45ab6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.383 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-07-29T01:22:55.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c 2023-07-29T01:22:55.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c 2023-07-29T01:22:55.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/c 2023-07-29T01:22:55.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.386 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.386 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.386 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.386 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.386 INFO:tasks.rgw_multisite_tests:/hiiqos-55/c 2023-07-29T01:22:55.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.387 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:43Jkn9X3FAOqavz5PVYX/nOwzbA= 2023-07-29T01:22:55.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:43Jkn9X3FAOqavz5PVYX/nOwzbA=', 'Content-Length': '0'} 2023-07-29T01:22:55.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c91126f1e5e676af-0064c45ab6-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/c 2023-07-29T01:22:55.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/c 2023-07-29T01:22:55.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/c 2023-07-29T01:22:55.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.390 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.390 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.391 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.391 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.391 INFO:tasks.rgw_multisite_tests:/hiiqos-55/c 2023-07-29T01:22:55.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.392 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:43Jkn9X3FAOqavz5PVYX/nOwzbA= 2023-07-29T01:22:55.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:43Jkn9X3FAOqavz5PVYX/nOwzbA=', 'Content-Length': '0'} 2023-07-29T01:22:55.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f3e402302d22d77-0064c45ab6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=c 2023-07-29T01:22:55.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=c 2023-07-29T01:22:55.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=c 2023-07-29T01:22:55.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.395 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.395 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.396 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.396 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:58 GMT 2023-07-29T01:22:55.396 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:55.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.397 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0= 2023-07-29T01:22:55.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:58 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TKTHo+2J9gEB67xuf9rrBFhMML0=', 'Content-Length': '0'} 2023-07-29T01:22:55.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004dfb3465966e6c6c-0064c45ab6-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55c1000falsec2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=c 2023-07-29T01:22:55.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=c 2023-07-29T01:22:55.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=c 2023-07-29T01:22:55.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.401 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.401 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.401 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.401 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.402 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:55.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.402 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4= 2023-07-29T01:22:55.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4=', 'Content-Length': '0'} 2023-07-29T01:22:55.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005579d3a28c84b2be-0064c45ab6-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55c1000falsec2023-07-29T00:14:44.008Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.403 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-07-29T01:22:55.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d 2023-07-29T01:22:55.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d 2023-07-29T01:22:55.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d?versionId=null 2023-07-29T01:22:55.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d?versionId=null 2023-07-29T01:22:55.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/d?versionId=null 2023-07-29T01:22:55.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.406 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.406 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.407 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.407 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.407 INFO:tasks.rgw_multisite_tests:/hiiqos-55/d?versionId=null 2023-07-29T01:22:55.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.408 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:5KQOUDw8e4yJatFUb6ylI+bnIL8= 2023-07-29T01:22:55.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5KQOUDw8e4yJatFUb6ylI+bnIL8=', 'Content-Length': '0'} 2023-07-29T01:22:55.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000108945ec86768e79-0064c45ab6-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d 2023-07-29T01:22:55.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d 2023-07-29T01:22:55.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d?versionId=null 2023-07-29T01:22:55.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d?versionId=null 2023-07-29T01:22:55.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/d?versionId=null 2023-07-29T01:22:55.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.411 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.412 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.412 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.412 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.412 INFO:tasks.rgw_multisite_tests:/hiiqos-55/d?versionId=null 2023-07-29T01:22:55.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.413 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:5KQOUDw8e4yJatFUb6ylI+bnIL8= 2023-07-29T01:22:55.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5KQOUDw8e4yJatFUb6ylI+bnIL8=', 'Content-Length': '0'} 2023-07-29T01:22:55.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000502a67ebdf71270b-0064c45ab6-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:58 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d 2023-07-29T01:22:55.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d 2023-07-29T01:22:55.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d?versionId=null 2023-07-29T01:22:55.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d?versionId=null 2023-07-29T01:22:55.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/d?versionId=null 2023-07-29T01:22:55.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.416 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.417 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.417 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.417 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.417 INFO:tasks.rgw_multisite_tests:/hiiqos-55/d?versionId=null 2023-07-29T01:22:55.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.418 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:mAAcVZoZTIkH9qt6YdN+KDWtwDE= 2023-07-29T01:22:55.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mAAcVZoZTIkH9qt6YdN+KDWtwDE=', 'Content-Length': '0'} 2023-07-29T01:22:55.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbdcbaade15861e5-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d 2023-07-29T01:22:55.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d 2023-07-29T01:22:55.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d?versionId=null 2023-07-29T01:22:55.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d?versionId=null 2023-07-29T01:22:55.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/d?versionId=null 2023-07-29T01:22:55.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.422 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.422 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.422 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.422 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.422 INFO:tasks.rgw_multisite_tests:/hiiqos-55/d?versionId=null 2023-07-29T01:22:55.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.423 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:mAAcVZoZTIkH9qt6YdN+KDWtwDE= 2023-07-29T01:22:55.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:mAAcVZoZTIkH9qt6YdN+KDWtwDE=', 'Content-Length': '0'} 2023-07-29T01:22:55.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049b3d11e51fe1edd-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.424 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-07-29T01:22:55.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d 2023-07-29T01:22:55.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d 2023-07-29T01:22:55.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d?versionId=null 2023-07-29T01:22:55.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d?versionId=null 2023-07-29T01:22:55.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/d?versionId=null 2023-07-29T01:22:55.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.427 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.427 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.427 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.427 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.428 INFO:tasks.rgw_multisite_tests:/hiiqos-55/d?versionId=null 2023-07-29T01:22:55.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.428 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:5KQOUDw8e4yJatFUb6ylI+bnIL8= 2023-07-29T01:22:55.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5KQOUDw8e4yJatFUb6ylI+bnIL8=', 'Content-Length': '0'} 2023-07-29T01:22:55.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bbc2ff9d0f430f5-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d 2023-07-29T01:22:55.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d 2023-07-29T01:22:55.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d?versionId=null 2023-07-29T01:22:55.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d?versionId=null 2023-07-29T01:22:55.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/d?versionId=null 2023-07-29T01:22:55.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.432 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.432 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.432 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.433 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.433 INFO:tasks.rgw_multisite_tests:/hiiqos-55/d?versionId=null 2023-07-29T01:22:55.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.433 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:5KQOUDw8e4yJatFUb6ylI+bnIL8= 2023-07-29T01:22:55.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5KQOUDw8e4yJatFUb6ylI+bnIL8=', 'Content-Length': '0'} 2023-07-29T01:22:55.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f2e6376004fecce-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=d 2023-07-29T01:22:55.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=d 2023-07-29T01:22:55.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=d 2023-07-29T01:22:55.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.437 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.437 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.437 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.438 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.438 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:55.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.438 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4= 2023-07-29T01:22:55.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4=', 'Content-Length': '0'} 2023-07-29T01:22:55.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b94ba521c560ed15-0064c45ab7-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55d1000falsed2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=d 2023-07-29T01:22:55.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=d 2023-07-29T01:22:55.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=d 2023-07-29T01:22:55.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.442 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.443 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.443 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.443 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.443 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:55.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.444 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4= 2023-07-29T01:22:55.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4=', 'Content-Length': '0'} 2023-07-29T01:22:55.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ec61d12a6273cd6-0064c45ab7-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55d1000falsed2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d 2023-07-29T01:22:55.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d 2023-07-29T01:22:55.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/d 2023-07-29T01:22:55.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.447 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.447 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.448 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.448 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.448 INFO:tasks.rgw_multisite_tests:/hiiqos-55/d 2023-07-29T01:22:55.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.449 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:cnK+njR1eCbNgdNpxRIaNnWloQE= 2023-07-29T01:22:55.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cnK+njR1eCbNgdNpxRIaNnWloQE=', 'Content-Length': '0'} 2023-07-29T01:22:55.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2f1ca5974730f1c-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d 2023-07-29T01:22:55.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d 2023-07-29T01:22:55.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/d 2023-07-29T01:22:55.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.452 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.452 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.452 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.453 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.453 INFO:tasks.rgw_multisite_tests:/hiiqos-55/d 2023-07-29T01:22:55.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.453 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:cnK+njR1eCbNgdNpxRIaNnWloQE= 2023-07-29T01:22:55.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cnK+njR1eCbNgdNpxRIaNnWloQE=', 'Content-Length': '0'} 2023-07-29T01:22:55.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007dc3f5c221695af8-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.454 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-07-29T01:22:55.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d 2023-07-29T01:22:55.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d 2023-07-29T01:22:55.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/d 2023-07-29T01:22:55.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.457 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.457 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.457 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.457 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.458 INFO:tasks.rgw_multisite_tests:/hiiqos-55/d 2023-07-29T01:22:55.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.458 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:0JFteaOb3BIH/EffAi7cmQ8mFVM= 2023-07-29T01:22:55.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0JFteaOb3BIH/EffAi7cmQ8mFVM=', 'Content-Length': '0'} 2023-07-29T01:22:55.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000700450029633ad4-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/d 2023-07-29T01:22:55.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/d 2023-07-29T01:22:55.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/d 2023-07-29T01:22:55.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.461 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.462 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.462 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.462 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.462 INFO:tasks.rgw_multisite_tests:/hiiqos-55/d 2023-07-29T01:22:55.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.463 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:0JFteaOb3BIH/EffAi7cmQ8mFVM= 2023-07-29T01:22:55.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0JFteaOb3BIH/EffAi7cmQ8mFVM=', 'Content-Length': '0'} 2023-07-29T01:22:55.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:14:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049cac657181d2c6f-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=d 2023-07-29T01:22:55.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=d 2023-07-29T01:22:55.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=d 2023-07-29T01:22:55.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.467 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.467 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.467 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.467 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.467 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:55.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.468 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4= 2023-07-29T01:22:55.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4=', 'Content-Length': '0'} 2023-07-29T01:22:55.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a781b5b1cb817c0-0064c45ab7-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55d1000falsed2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=d 2023-07-29T01:22:55.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=d 2023-07-29T01:22:55.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=d 2023-07-29T01:22:55.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.472 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.472 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.472 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.473 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.473 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:55.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.473 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4= 2023-07-29T01:22:55.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4=', 'Content-Length': '0'} 2023-07-29T01:22:55.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ade1402206f0af5f-0064c45ab7-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55d1000falsed2023-07-29T00:14:44.026Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.474 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2023-07-29T01:22:55.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/e 2023-07-29T01:22:55.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/e 2023-07-29T01:22:55.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/e?versionId=null 2023-07-29T01:22:55.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/e?versionId=null 2023-07-29T01:22:55.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/e?versionId=null 2023-07-29T01:22:55.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.477 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.478 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.478 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.478 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.478 INFO:tasks.rgw_multisite_tests:/hiiqos-55/e?versionId=null 2023-07-29T01:22:55.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.479 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:1E/UqzTuNI784IcAGiKLyGG931s= 2023-07-29T01:22:55.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1E/UqzTuNI784IcAGiKLyGG931s=', 'Content-Length': '0'} 2023-07-29T01:22:55.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b45d68b1efbd1a3-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/e 2023-07-29T01:22:55.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/e 2023-07-29T01:22:55.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/e?versionId=null 2023-07-29T01:22:55.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/e?versionId=null 2023-07-29T01:22:55.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/e?versionId=null 2023-07-29T01:22:55.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.483 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.483 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.483 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.483 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.483 INFO:tasks.rgw_multisite_tests:/hiiqos-55/e?versionId=null 2023-07-29T01:22:55.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.484 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:1E/UqzTuNI784IcAGiKLyGG931s= 2023-07-29T01:22:55.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1E/UqzTuNI784IcAGiKLyGG931s=', 'Content-Length': '0'} 2023-07-29T01:22:55.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051668bd88135902c-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/e 2023-07-29T01:22:55.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/e 2023-07-29T01:22:55.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/e?versionId=null 2023-07-29T01:22:55.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/e?versionId=null 2023-07-29T01:22:55.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/e?versionId=null 2023-07-29T01:22:55.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.488 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.488 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.488 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.488 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.489 INFO:tasks.rgw_multisite_tests:/hiiqos-55/e?versionId=null 2023-07-29T01:22:55.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.489 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:aaGWO4PYLoNIX0J0lONsLW3S1dA= 2023-07-29T01:22:55.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aaGWO4PYLoNIX0J0lONsLW3S1dA=', 'Content-Length': '0'} 2023-07-29T01:22:55.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003beb56383f269697-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/e 2023-07-29T01:22:55.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/e 2023-07-29T01:22:55.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/e?versionId=null 2023-07-29T01:22:55.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/e?versionId=null 2023-07-29T01:22:55.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/e?versionId=null 2023-07-29T01:22:55.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.493 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.493 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.493 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.493 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.494 INFO:tasks.rgw_multisite_tests:/hiiqos-55/e?versionId=null 2023-07-29T01:22:55.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.494 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:aaGWO4PYLoNIX0J0lONsLW3S1dA= 2023-07-29T01:22:55.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aaGWO4PYLoNIX0J0lONsLW3S1dA=', 'Content-Length': '0'} 2023-07-29T01:22:55.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004995a77e59f97824-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.495 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2023-07-29T01:22:55.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/e 2023-07-29T01:22:55.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/e 2023-07-29T01:22:55.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/e?versionId=null 2023-07-29T01:22:55.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/e?versionId=null 2023-07-29T01:22:55.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/e?versionId=null 2023-07-29T01:22:55.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.498 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.498 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.498 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.499 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.499 INFO:tasks.rgw_multisite_tests:/hiiqos-55/e?versionId=null 2023-07-29T01:22:55.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.499 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:1E/UqzTuNI784IcAGiKLyGG931s= 2023-07-29T01:22:55.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1E/UqzTuNI784IcAGiKLyGG931s=', 'Content-Length': '0'} 2023-07-29T01:22:55.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b678acb67f46189c-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/e 2023-07-29T01:22:55.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/e 2023-07-29T01:22:55.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/e?versionId=null 2023-07-29T01:22:55.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/e?versionId=null 2023-07-29T01:22:55.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/e?versionId=null 2023-07-29T01:22:55.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.503 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.503 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.503 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.504 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.504 INFO:tasks.rgw_multisite_tests:/hiiqos-55/e?versionId=null 2023-07-29T01:22:55.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.504 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:1E/UqzTuNI784IcAGiKLyGG931s= 2023-07-29T01:22:55.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1E/UqzTuNI784IcAGiKLyGG931s=', 'Content-Length': '0'} 2023-07-29T01:22:55.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6bf6bd06784381f-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=e 2023-07-29T01:22:55.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=e 2023-07-29T01:22:55.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=e 2023-07-29T01:22:55.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.508 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.508 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.509 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.509 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.509 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:55.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.509 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4= 2023-07-29T01:22:55.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4=', 'Content-Length': '0'} 2023-07-29T01:22:55.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003207164e967754cd-0064c45ab7-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55e1000falsee2023-07-29T00:15:00.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=e 2023-07-29T01:22:55.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=e 2023-07-29T01:22:55.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=e 2023-07-29T01:22:55.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.513 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.514 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.514 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.514 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.514 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:55.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.515 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4= 2023-07-29T01:22:55.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4=', 'Content-Length': '0'} 2023-07-29T01:22:55.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009022c770f61ed90d-0064c45ab7-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55e1000falsee2023-07-29T00:15:00.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/e 2023-07-29T01:22:55.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/e 2023-07-29T01:22:55.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/e 2023-07-29T01:22:55.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.518 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.518 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.519 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.519 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.519 INFO:tasks.rgw_multisite_tests:/hiiqos-55/e 2023-07-29T01:22:55.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.520 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:4uyE2n56zIVyLz8hhjN209tqjPs= 2023-07-29T01:22:55.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4uyE2n56zIVyLz8hhjN209tqjPs=', 'Content-Length': '0'} 2023-07-29T01:22:55.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f9bb6cb518e61fd-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/e 2023-07-29T01:22:55.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/e 2023-07-29T01:22:55.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/e 2023-07-29T01:22:55.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.523 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.523 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.523 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.524 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.524 INFO:tasks.rgw_multisite_tests:/hiiqos-55/e 2023-07-29T01:22:55.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.524 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:4uyE2n56zIVyLz8hhjN209tqjPs= 2023-07-29T01:22:55.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4uyE2n56zIVyLz8hhjN209tqjPs=', 'Content-Length': '0'} 2023-07-29T01:22:55.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000356242b0b68ba80-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.525 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2023-07-29T01:22:55.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/e 2023-07-29T01:22:55.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/e 2023-07-29T01:22:55.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/e 2023-07-29T01:22:55.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.528 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.528 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.528 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.528 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.529 INFO:tasks.rgw_multisite_tests:/hiiqos-55/e 2023-07-29T01:22:55.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.529 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:IoV2Ky/ucAZhdNFujM4/+mF7J/U= 2023-07-29T01:22:55.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IoV2Ky/ucAZhdNFujM4/+mF7J/U=', 'Content-Length': '0'} 2023-07-29T01:22:55.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c862ac8e6ad9802-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/e 2023-07-29T01:22:55.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/e 2023-07-29T01:22:55.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/e 2023-07-29T01:22:55.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.532 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.533 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.533 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.533 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.533 INFO:tasks.rgw_multisite_tests:/hiiqos-55/e 2023-07-29T01:22:55.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.534 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:IoV2Ky/ucAZhdNFujM4/+mF7J/U= 2023-07-29T01:22:55.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IoV2Ky/ucAZhdNFujM4/+mF7J/U=', 'Content-Length': '0'} 2023-07-29T01:22:55.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d65710e082e9080-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=e 2023-07-29T01:22:55.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=e 2023-07-29T01:22:55.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=e 2023-07-29T01:22:55.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.537 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.538 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.538 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.538 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.538 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:55.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.539 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4= 2023-07-29T01:22:55.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4=', 'Content-Length': '0'} 2023-07-29T01:22:55.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052193816a2ff14cf-0064c45ab7-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55e1000falsee2023-07-29T00:15:00.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=e 2023-07-29T01:22:55.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=e 2023-07-29T01:22:55.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=e 2023-07-29T01:22:55.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.543 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.543 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.543 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.543 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.544 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:55.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.544 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4= 2023-07-29T01:22:55.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4=', 'Content-Length': '0'} 2023-07-29T01:22:55.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb72bf4b47127052-0064c45ab7-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55e1000falsee2023-07-29T00:15:00.986Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.545 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2023-07-29T01:22:55.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/f 2023-07-29T01:22:55.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/f 2023-07-29T01:22:55.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/f?versionId=null 2023-07-29T01:22:55.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/f?versionId=null 2023-07-29T01:22:55.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/f?versionId=null 2023-07-29T01:22:55.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.548 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.549 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.549 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.549 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.549 INFO:tasks.rgw_multisite_tests:/hiiqos-55/f?versionId=null 2023-07-29T01:22:55.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.550 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:gLvuIDy4hi0Zr6zvW0YeFpRN28Y= 2023-07-29T01:22:55.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gLvuIDy4hi0Zr6zvW0YeFpRN28Y=', 'Content-Length': '0'} 2023-07-29T01:22:55.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4d651fed1ecabed-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/f 2023-07-29T01:22:55.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/f 2023-07-29T01:22:55.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/f?versionId=null 2023-07-29T01:22:55.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/f?versionId=null 2023-07-29T01:22:55.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/f?versionId=null 2023-07-29T01:22:55.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.553 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.554 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.554 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.554 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.554 INFO:tasks.rgw_multisite_tests:/hiiqos-55/f?versionId=null 2023-07-29T01:22:55.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.555 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:gLvuIDy4hi0Zr6zvW0YeFpRN28Y= 2023-07-29T01:22:55.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gLvuIDy4hi0Zr6zvW0YeFpRN28Y=', 'Content-Length': '0'} 2023-07-29T01:22:55.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005598885b4bdf9b58-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/f 2023-07-29T01:22:55.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/f 2023-07-29T01:22:55.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/f?versionId=null 2023-07-29T01:22:55.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/f?versionId=null 2023-07-29T01:22:55.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/f?versionId=null 2023-07-29T01:22:55.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.559 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.559 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.559 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.559 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.559 INFO:tasks.rgw_multisite_tests:/hiiqos-55/f?versionId=null 2023-07-29T01:22:55.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.560 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:pwJ581L8kNUghlSeFpOLE2E029o= 2023-07-29T01:22:55.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pwJ581L8kNUghlSeFpOLE2E029o=', 'Content-Length': '0'} 2023-07-29T01:22:55.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfdb624d78bb8902-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/f 2023-07-29T01:22:55.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/f 2023-07-29T01:22:55.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/f?versionId=null 2023-07-29T01:22:55.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/f?versionId=null 2023-07-29T01:22:55.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/f?versionId=null 2023-07-29T01:22:55.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.564 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.564 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.564 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.564 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.565 INFO:tasks.rgw_multisite_tests:/hiiqos-55/f?versionId=null 2023-07-29T01:22:55.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.565 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:pwJ581L8kNUghlSeFpOLE2E029o= 2023-07-29T01:22:55.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pwJ581L8kNUghlSeFpOLE2E029o=', 'Content-Length': '0'} 2023-07-29T01:22:55.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7cceadb5d6cad9c-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.566 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2023-07-29T01:22:55.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/f 2023-07-29T01:22:55.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/f 2023-07-29T01:22:55.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/f?versionId=null 2023-07-29T01:22:55.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/f?versionId=null 2023-07-29T01:22:55.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/f?versionId=null 2023-07-29T01:22:55.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.569 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.569 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.570 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.570 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.570 INFO:tasks.rgw_multisite_tests:/hiiqos-55/f?versionId=null 2023-07-29T01:22:55.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.570 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:gLvuIDy4hi0Zr6zvW0YeFpRN28Y= 2023-07-29T01:22:55.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gLvuIDy4hi0Zr6zvW0YeFpRN28Y=', 'Content-Length': '0'} 2023-07-29T01:22:55.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eda0b573ed529569-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/f 2023-07-29T01:22:55.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/f 2023-07-29T01:22:55.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/f?versionId=null 2023-07-29T01:22:55.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/f?versionId=null 2023-07-29T01:22:55.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/f?versionId=null 2023-07-29T01:22:55.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.574 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.574 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.575 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.575 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.575 INFO:tasks.rgw_multisite_tests:/hiiqos-55/f?versionId=null 2023-07-29T01:22:55.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.576 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:gLvuIDy4hi0Zr6zvW0YeFpRN28Y= 2023-07-29T01:22:55.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gLvuIDy4hi0Zr6zvW0YeFpRN28Y=', 'Content-Length': '0'} 2023-07-29T01:22:55.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005beadd3ac11a88ef-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=f 2023-07-29T01:22:55.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=f 2023-07-29T01:22:55.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=f 2023-07-29T01:22:55.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.579 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.580 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.580 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.580 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.580 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:55.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.581 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4= 2023-07-29T01:22:55.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4=', 'Content-Length': '0'} 2023-07-29T01:22:55.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005a9c67883a34fca-0064c45ab7-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55f1000falsef2023-07-29T00:15:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=f 2023-07-29T01:22:55.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=f 2023-07-29T01:22:55.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=f 2023-07-29T01:22:55.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.585 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.585 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.585 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.585 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.586 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:55.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.586 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4= 2023-07-29T01:22:55.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4=', 'Content-Length': '0'} 2023-07-29T01:22:55.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffa2f312c780d4d8-0064c45ab7-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55f1000falsef2023-07-29T00:15:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/f 2023-07-29T01:22:55.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/f 2023-07-29T01:22:55.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/f 2023-07-29T01:22:55.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.590 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.590 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.590 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.590 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.590 INFO:tasks.rgw_multisite_tests:/hiiqos-55/f 2023-07-29T01:22:55.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.591 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:Uy41MOvlnmjNiKyRMFohexsmmoo= 2023-07-29T01:22:55.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Uy41MOvlnmjNiKyRMFohexsmmoo=', 'Content-Length': '0'} 2023-07-29T01:22:55.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004798b34cd5e549a5-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/f 2023-07-29T01:22:55.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/f 2023-07-29T01:22:55.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/f 2023-07-29T01:22:55.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.594 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.594 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.595 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.595 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.595 INFO:tasks.rgw_multisite_tests:/hiiqos-55/f 2023-07-29T01:22:55.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.596 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:Uy41MOvlnmjNiKyRMFohexsmmoo= 2023-07-29T01:22:55.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Uy41MOvlnmjNiKyRMFohexsmmoo=', 'Content-Length': '0'} 2023-07-29T01:22:55.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a3ff7b1d2401937-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.596 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2023-07-29T01:22:55.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/f 2023-07-29T01:22:55.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/f 2023-07-29T01:22:55.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/f 2023-07-29T01:22:55.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.599 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.599 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.599 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.600 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.600 INFO:tasks.rgw_multisite_tests:/hiiqos-55/f 2023-07-29T01:22:55.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.600 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:9aaUUp3s170DLQ1Y8LBuXurTmws= 2023-07-29T01:22:55.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9aaUUp3s170DLQ1Y8LBuXurTmws=', 'Content-Length': '0'} 2023-07-29T01:22:55.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e0021329d06c01b-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/f 2023-07-29T01:22:55.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/f 2023-07-29T01:22:55.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/f 2023-07-29T01:22:55.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.604 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.604 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.604 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.604 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.605 INFO:tasks.rgw_multisite_tests:/hiiqos-55/f 2023-07-29T01:22:55.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.605 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:9aaUUp3s170DLQ1Y8LBuXurTmws= 2023-07-29T01:22:55.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9aaUUp3s170DLQ1Y8LBuXurTmws=', 'Content-Length': '0'} 2023-07-29T01:22:55.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000549c6a3c7f95cd5-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=f 2023-07-29T01:22:55.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=f 2023-07-29T01:22:55.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=f 2023-07-29T01:22:55.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.609 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.609 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.609 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.609 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.610 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:55.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.610 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4= 2023-07-29T01:22:55.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4=', 'Content-Length': '0'} 2023-07-29T01:22:55.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d480abe8c52d56d0-0064c45ab7-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55f1000falsef2023-07-29T00:15:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=f 2023-07-29T01:22:55.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=f 2023-07-29T01:22:55.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=f 2023-07-29T01:22:55.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.614 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.614 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.615 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.615 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.615 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:55.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.615 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4= 2023-07-29T01:22:55.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4=', 'Content-Length': '0'} 2023-07-29T01:22:55.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1e1ccd3b718d768-0064c45ab7-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55f1000falsef2023-07-29T00:15:01.001Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.616 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2023-07-29T01:22:55.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/g 2023-07-29T01:22:55.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/g 2023-07-29T01:22:55.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/g?versionId=null 2023-07-29T01:22:55.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/g?versionId=null 2023-07-29T01:22:55.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/g?versionId=null 2023-07-29T01:22:55.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.620 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.620 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.620 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.620 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.621 INFO:tasks.rgw_multisite_tests:/hiiqos-55/g?versionId=null 2023-07-29T01:22:55.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.621 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:ZNRr3CNlRcBpQFGsX4+KIN9uMY4= 2023-07-29T01:22:55.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZNRr3CNlRcBpQFGsX4+KIN9uMY4=', 'Content-Length': '0'} 2023-07-29T01:22:55.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9dd95e7bdc62af7-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/g 2023-07-29T01:22:55.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/g 2023-07-29T01:22:55.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/g?versionId=null 2023-07-29T01:22:55.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/g?versionId=null 2023-07-29T01:22:55.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/g?versionId=null 2023-07-29T01:22:55.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.625 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.625 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.625 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.625 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.626 INFO:tasks.rgw_multisite_tests:/hiiqos-55/g?versionId=null 2023-07-29T01:22:55.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.626 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:ZNRr3CNlRcBpQFGsX4+KIN9uMY4= 2023-07-29T01:22:55.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZNRr3CNlRcBpQFGsX4+KIN9uMY4=', 'Content-Length': '0'} 2023-07-29T01:22:55.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bad10613f68aca98-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/g 2023-07-29T01:22:55.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/g 2023-07-29T01:22:55.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/g?versionId=null 2023-07-29T01:22:55.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/g?versionId=null 2023-07-29T01:22:55.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/g?versionId=null 2023-07-29T01:22:55.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.630 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.630 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.630 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.631 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.631 INFO:tasks.rgw_multisite_tests:/hiiqos-55/g?versionId=null 2023-07-29T01:22:55.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.631 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:BnK/6Xcas2ogfhpOar48vO0D6mg= 2023-07-29T01:22:55.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BnK/6Xcas2ogfhpOar48vO0D6mg=', 'Content-Length': '0'} 2023-07-29T01:22:55.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddd7256a3859c453-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/g 2023-07-29T01:22:55.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/g 2023-07-29T01:22:55.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/g?versionId=null 2023-07-29T01:22:55.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/g?versionId=null 2023-07-29T01:22:55.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/g?versionId=null 2023-07-29T01:22:55.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.635 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.635 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.635 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.636 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.636 INFO:tasks.rgw_multisite_tests:/hiiqos-55/g?versionId=null 2023-07-29T01:22:55.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.636 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:BnK/6Xcas2ogfhpOar48vO0D6mg= 2023-07-29T01:22:55.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BnK/6Xcas2ogfhpOar48vO0D6mg=', 'Content-Length': '0'} 2023-07-29T01:22:55.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0814290aa6ff181-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.637 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2023-07-29T01:22:55.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/g 2023-07-29T01:22:55.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/g 2023-07-29T01:22:55.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/g?versionId=null 2023-07-29T01:22:55.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/g?versionId=null 2023-07-29T01:22:55.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/g?versionId=null 2023-07-29T01:22:55.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.640 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.641 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.641 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.641 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.641 INFO:tasks.rgw_multisite_tests:/hiiqos-55/g?versionId=null 2023-07-29T01:22:55.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.642 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:ZNRr3CNlRcBpQFGsX4+KIN9uMY4= 2023-07-29T01:22:55.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZNRr3CNlRcBpQFGsX4+KIN9uMY4=', 'Content-Length': '0'} 2023-07-29T01:22:55.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f9053e0baa52689-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/g 2023-07-29T01:22:55.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/g 2023-07-29T01:22:55.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/g?versionId=null 2023-07-29T01:22:55.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/g?versionId=null 2023-07-29T01:22:55.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/g?versionId=null 2023-07-29T01:22:55.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.645 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.646 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.646 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.646 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.646 INFO:tasks.rgw_multisite_tests:/hiiqos-55/g?versionId=null 2023-07-29T01:22:55.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.647 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:ZNRr3CNlRcBpQFGsX4+KIN9uMY4= 2023-07-29T01:22:55.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZNRr3CNlRcBpQFGsX4+KIN9uMY4=', 'Content-Length': '0'} 2023-07-29T01:22:55.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdcc814503478369-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=g 2023-07-29T01:22:55.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=g 2023-07-29T01:22:55.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=g 2023-07-29T01:22:55.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.650 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.651 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.651 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.651 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.651 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:55.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.652 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4= 2023-07-29T01:22:55.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4=', 'Content-Length': '0'} 2023-07-29T01:22:55.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002dcf10373327872e-0064c45ab7-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55g1000falseg2023-07-29T00:15:01.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=g 2023-07-29T01:22:55.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=g 2023-07-29T01:22:55.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=g 2023-07-29T01:22:55.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.656 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.656 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.656 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.656 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.657 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:55.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.657 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4= 2023-07-29T01:22:55.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4=', 'Content-Length': '0'} 2023-07-29T01:22:55.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083f2675daef90c69-0064c45ab7-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55g1000falseg2023-07-29T00:15:01.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/g 2023-07-29T01:22:55.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/g 2023-07-29T01:22:55.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/g 2023-07-29T01:22:55.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.661 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.661 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.661 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.661 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.662 INFO:tasks.rgw_multisite_tests:/hiiqos-55/g 2023-07-29T01:22:55.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.662 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:rtczgeAVntG5FH/w9a/I55RBkEk= 2023-07-29T01:22:55.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rtczgeAVntG5FH/w9a/I55RBkEk=', 'Content-Length': '0'} 2023-07-29T01:22:55.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013c57afdc505dd5a-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/g 2023-07-29T01:22:55.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/g 2023-07-29T01:22:55.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/g 2023-07-29T01:22:55.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.665 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.666 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.666 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.666 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.666 INFO:tasks.rgw_multisite_tests:/hiiqos-55/g 2023-07-29T01:22:55.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.667 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:rtczgeAVntG5FH/w9a/I55RBkEk= 2023-07-29T01:22:55.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rtczgeAVntG5FH/w9a/I55RBkEk=', 'Content-Length': '0'} 2023-07-29T01:22:55.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000071f706f259fd7b-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.667 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2023-07-29T01:22:55.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/g 2023-07-29T01:22:55.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/g 2023-07-29T01:22:55.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/g 2023-07-29T01:22:55.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.670 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.670 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.671 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.671 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.671 INFO:tasks.rgw_multisite_tests:/hiiqos-55/g 2023-07-29T01:22:55.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.672 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:q24ibV8HAnGltAk+0FvPg/aq+fk= 2023-07-29T01:22:55.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:q24ibV8HAnGltAk+0FvPg/aq+fk=', 'Content-Length': '0'} 2023-07-29T01:22:55.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099617417b4963fcf-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/g 2023-07-29T01:22:55.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/g 2023-07-29T01:22:55.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/g 2023-07-29T01:22:55.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.675 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.675 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.675 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.676 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.676 INFO:tasks.rgw_multisite_tests:/hiiqos-55/g 2023-07-29T01:22:55.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.676 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:q24ibV8HAnGltAk+0FvPg/aq+fk= 2023-07-29T01:22:55.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:q24ibV8HAnGltAk+0FvPg/aq+fk=', 'Content-Length': '0'} 2023-07-29T01:22:55.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b456d3548f3fd87-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=g 2023-07-29T01:22:55.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=g 2023-07-29T01:22:55.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=g 2023-07-29T01:22:55.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.680 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.680 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.680 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.681 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.681 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:55.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.681 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4= 2023-07-29T01:22:55.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4=', 'Content-Length': '0'} 2023-07-29T01:22:55.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004080cc390dbd7c3c-0064c45ab7-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55g1000falseg2023-07-29T00:15:01.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=g 2023-07-29T01:22:55.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=g 2023-07-29T01:22:55.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=g 2023-07-29T01:22:55.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.685 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.686 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.686 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.686 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.686 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:55.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.687 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4= 2023-07-29T01:22:55.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4=', 'Content-Length': '0'} 2023-07-29T01:22:55.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000237e45f4e04cfde0-0064c45ab7-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55g1000falseg2023-07-29T00:15:01.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.688 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2023-07-29T01:22:55.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/h 2023-07-29T01:22:55.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/h 2023-07-29T01:22:55.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/h?versionId=null 2023-07-29T01:22:55.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/h?versionId=null 2023-07-29T01:22:55.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/h?versionId=null 2023-07-29T01:22:55.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.691 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.691 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.691 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.692 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.692 INFO:tasks.rgw_multisite_tests:/hiiqos-55/h?versionId=null 2023-07-29T01:22:55.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.692 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:ZEYei45sbVojlw8zUXRDZ8zvJjs= 2023-07-29T01:22:55.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZEYei45sbVojlw8zUXRDZ8zvJjs=', 'Content-Length': '0'} 2023-07-29T01:22:55.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db3b743a4bacf9c6-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/h 2023-07-29T01:22:55.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/h 2023-07-29T01:22:55.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/h?versionId=null 2023-07-29T01:22:55.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/h?versionId=null 2023-07-29T01:22:55.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/h?versionId=null 2023-07-29T01:22:55.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.696 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.696 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.696 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.697 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.697 INFO:tasks.rgw_multisite_tests:/hiiqos-55/h?versionId=null 2023-07-29T01:22:55.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.697 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:ZEYei45sbVojlw8zUXRDZ8zvJjs= 2023-07-29T01:22:55.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZEYei45sbVojlw8zUXRDZ8zvJjs=', 'Content-Length': '0'} 2023-07-29T01:22:55.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcc9bcb14de3353d-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/h 2023-07-29T01:22:55.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/h 2023-07-29T01:22:55.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/h?versionId=null 2023-07-29T01:22:55.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/h?versionId=null 2023-07-29T01:22:55.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/h?versionId=null 2023-07-29T01:22:55.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.701 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.701 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.702 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.702 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.702 INFO:tasks.rgw_multisite_tests:/hiiqos-55/h?versionId=null 2023-07-29T01:22:55.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.702 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:6o39inkh5b4GqFq6WT4ng7z3+Fc= 2023-07-29T01:22:55.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6o39inkh5b4GqFq6WT4ng7z3+Fc=', 'Content-Length': '0'} 2023-07-29T01:22:55.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d96ab5e573f07457-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/h 2023-07-29T01:22:55.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/h 2023-07-29T01:22:55.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/h?versionId=null 2023-07-29T01:22:55.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/h?versionId=null 2023-07-29T01:22:55.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/h?versionId=null 2023-07-29T01:22:55.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.706 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.706 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.707 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.707 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.707 INFO:tasks.rgw_multisite_tests:/hiiqos-55/h?versionId=null 2023-07-29T01:22:55.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.708 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:6o39inkh5b4GqFq6WT4ng7z3+Fc= 2023-07-29T01:22:55.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6o39inkh5b4GqFq6WT4ng7z3+Fc=', 'Content-Length': '0'} 2023-07-29T01:22:55.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000167e0044fa8fbbaa-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.708 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2023-07-29T01:22:55.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/h 2023-07-29T01:22:55.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/h 2023-07-29T01:22:55.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/h?versionId=null 2023-07-29T01:22:55.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/h?versionId=null 2023-07-29T01:22:55.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/h?versionId=null 2023-07-29T01:22:55.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.712 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.712 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.712 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.712 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.712 INFO:tasks.rgw_multisite_tests:/hiiqos-55/h?versionId=null 2023-07-29T01:22:55.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.713 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:ZEYei45sbVojlw8zUXRDZ8zvJjs= 2023-07-29T01:22:55.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZEYei45sbVojlw8zUXRDZ8zvJjs=', 'Content-Length': '0'} 2023-07-29T01:22:55.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df69615601489a82-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/h 2023-07-29T01:22:55.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/h 2023-07-29T01:22:55.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/h?versionId=null 2023-07-29T01:22:55.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/h?versionId=null 2023-07-29T01:22:55.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/h?versionId=null 2023-07-29T01:22:55.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.717 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.717 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.717 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.717 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.718 INFO:tasks.rgw_multisite_tests:/hiiqos-55/h?versionId=null 2023-07-29T01:22:55.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.718 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:ZEYei45sbVojlw8zUXRDZ8zvJjs= 2023-07-29T01:22:55.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZEYei45sbVojlw8zUXRDZ8zvJjs=', 'Content-Length': '0'} 2023-07-29T01:22:55.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1f70643599f1dbf-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=h 2023-07-29T01:22:55.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=h 2023-07-29T01:22:55.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=h 2023-07-29T01:22:55.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.722 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.722 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.722 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.722 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.723 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:55.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.723 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4= 2023-07-29T01:22:55.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4=', 'Content-Length': '0'} 2023-07-29T01:22:55.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cce1a630b150b48-0064c45ab7-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55h1000falseh2023-07-29T00:15:01.033Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=h 2023-07-29T01:22:55.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=h 2023-07-29T01:22:55.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=h 2023-07-29T01:22:55.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.727 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.727 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.727 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.728 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.728 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:55.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.728 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4= 2023-07-29T01:22:55.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4=', 'Content-Length': '0'} 2023-07-29T01:22:55.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff79c5232917fef3-0064c45ab7-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55h1000falseh2023-07-29T00:15:01.033Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/h 2023-07-29T01:22:55.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/h 2023-07-29T01:22:55.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/h 2023-07-29T01:22:55.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.732 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.732 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.732 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.733 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.733 INFO:tasks.rgw_multisite_tests:/hiiqos-55/h 2023-07-29T01:22:55.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.733 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:4wOvzVQ/RNGzUG918pfrTmMumes= 2023-07-29T01:22:55.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4wOvzVQ/RNGzUG918pfrTmMumes=', 'Content-Length': '0'} 2023-07-29T01:22:55.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007179146942739fa2-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/h 2023-07-29T01:22:55.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/h 2023-07-29T01:22:55.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/h 2023-07-29T01:22:55.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.737 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.737 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.737 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.737 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.737 INFO:tasks.rgw_multisite_tests:/hiiqos-55/h 2023-07-29T01:22:55.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.738 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:4wOvzVQ/RNGzUG918pfrTmMumes= 2023-07-29T01:22:55.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4wOvzVQ/RNGzUG918pfrTmMumes=', 'Content-Length': '0'} 2023-07-29T01:22:55.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb22204b2e671787-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.739 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2023-07-29T01:22:55.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/h 2023-07-29T01:22:55.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/h 2023-07-29T01:22:55.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/h 2023-07-29T01:22:55.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.741 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.742 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.742 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.742 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.742 INFO:tasks.rgw_multisite_tests:/hiiqos-55/h 2023-07-29T01:22:55.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.743 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:3xolI2FkEQ6M3/+29sdFYA3Jits= 2023-07-29T01:22:55.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3xolI2FkEQ6M3/+29sdFYA3Jits=', 'Content-Length': '0'} 2023-07-29T01:22:55.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-55:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0d196ff1cdd8964-0064c45ab7-1663-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/h 2023-07-29T01:22:55.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/h 2023-07-29T01:22:55.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/h 2023-07-29T01:22:55.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.746 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.746 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.747 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.747 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.747 INFO:tasks.rgw_multisite_tests:/hiiqos-55/h 2023-07-29T01:22:55.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.747 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:3xolI2FkEQ6M3/+29sdFYA3Jits= 2023-07-29T01:22:55.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3xolI2FkEQ6M3/+29sdFYA3Jits=', 'Content-Length': '0'} 2023-07-29T01:22:55.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:15:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048a712a8fed930f8-0064c45ab7-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=h 2023-07-29T01:22:55.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=h 2023-07-29T01:22:55.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=h 2023-07-29T01:22:55.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:55.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:55.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.751 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.751 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.752 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.752 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.752 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:55.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.753 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4= 2023-07-29T01:22:55.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4=', 'Content-Length': '0'} 2023-07-29T01:22:55.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5611b30beff0877-0064c45ab7-1663-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55h1000falseh2023-07-29T00:15:01.033Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/ 2023-07-29T01:22:55.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/ 2023-07-29T01:22:55.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-55/?prefix=h 2023-07-29T01:22:55.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-55/?prefix=h 2023-07-29T01:22:55.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:55.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-55/?prefix=h 2023-07-29T01:22:55.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.757 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:55.757 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.757 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.757 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:17:59 GMT 2023-07-29T01:22:55.757 INFO:tasks.rgw_multisite_tests:/hiiqos-55/ 2023-07-29T01:22:55.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.758 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4= 2023-07-29T01:22:55.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:17:59 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kY4cdNR+925EkXoLjseRyMETcM4=', 'Content-Length': '0'} 2023-07-29T01:22:55.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3ac07bdb7b3e088-0064c45ab7-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:17:59 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-55h1000falseh2023-07-29T00:15:01.033Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:55.759 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=hiiqos-55 zones={a2, a1} 2023-07-29T01:22:55.759 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:55.759 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:55.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:22:55.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:55.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:55.760 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi101.stdout: INFO: 2023-07-29T00:17:59.912+0000 7f915205d640 -1 req 5459550028987789389 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-6:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.6,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-6:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.6,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T01:22:55.761 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi101.stdout: INFO: 2023-07-29T00:17:59.912+0000 7f9167888640 -1 req 13781160531294753919 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-36:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.1,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-36:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.1,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T01:22:55.761 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi101.stdout: INFO: 2023-07-29T00:17:59.912+0000 7f913d033640 -1 req 2754071433038047717 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-19:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.19,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-19:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.19,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T01:22:55.761 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi101.stdout: INFO: 2023-07-29T00:17:59.916+0000 7f9124001640 -1 req 3483033020764464808 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-23:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.23,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-23:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.23,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T01:22:55.761 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi101.stdout: INFO: 2023-07-29T00:17:59.957+0000 7f91197ec640 -1 req 264704333263720778 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-5:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.5,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-5:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.5,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T01:22:55.761 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi101.stdout: INFO: 2023-07-29T00:17:59.958+0000 7f911a7ee640 -1 req 4787960672522150306 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-51:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.16,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-51:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.16,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T01:22:55.762 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi101.stdout: INFO: 2023-07-29T00:18:00.000+0000 7f90fefb7640 -1 req 15118785672761743039 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-42:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.7,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-42:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.7,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T01:22:55.762 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi101.stdout: INFO: 2023-07-29T00:18:00.004+0000 7f91137e0640 -1 req 14210760628869684679 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-29:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.29,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-29:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.29,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T01:22:55.762 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'hiiqos-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:55.762 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'hiiqos-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:55.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket hiiqos-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:22:55.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:55.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:55.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:55.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "layout": { 2023-07-29T01:22:55.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "resharding": "None", 2023-07-29T01:22:55.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_index": { 2023-07-29T01:22:55.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "gen": 2, 2023-07-29T01:22:55.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "layout": { 2023-07-29T01:22:55.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "type": "Normal", 2023-07-29T01:22:55.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "normal": { 2023-07-29T01:22:55.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 3, 2023-07-29T01:22:55.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "hash_type": "Mod" 2023-07-29T01:22:55.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:55.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "logs": [ 2023-07-29T01:22:55.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:55.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "gen": 1, 2023-07-29T01:22:55.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "layout": { 2023-07-29T01:22:55.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "type": "InIndex", 2023-07-29T01:22:55.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "in_index": { 2023-07-29T01:22:55.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "gen": 1, 2023-07-29T01:22:55.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "layout": { 2023-07-29T01:22:55.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 3, 2023-07-29T01:22:55.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "hash_type": "Mod" 2023-07-29T01:22:55.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:55.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:55.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "gen": 2, 2023-07-29T01:22:55.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "layout": { 2023-07-29T01:22:55.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "type": "InIndex", 2023-07-29T01:22:55.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "in_index": { 2023-07-29T01:22:55.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "gen": 2, 2023-07-29T01:22:55.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "layout": { 2023-07-29T01:22:55.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 3, 2023-07-29T01:22:55.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "hash_type": "Mod" 2023-07-29T01:22:55.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:55.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.773 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:55.774 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:55.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:22:55.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:55.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:55.775 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi101.stdout: INFO: 2023-07-29T00:18:00.285+0000 7f90ff7b8640 -1 req 5304715750979617260 0.000999976s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-41:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.6,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-41:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.6,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T01:22:55.775 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi101.stdout: INFO: 2023-07-29T00:18:00.287+0000 7f9121ffd640 -1 req 357210395645382615 0.000999975s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-38:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.3,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-38:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.3,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T01:22:55.775 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi101.stdout: INFO: 2023-07-29T00:18:00.287+0000 7f90e5784640 -1 req 4449530075996876019 0.000999975s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-1:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.1,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-1:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.1,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T01:22:55.775 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi101.stdout: INFO: 2023-07-29T00:18:00.336+0000 7f90d2f5f640 -1 req 14306619141323723770 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-4:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.4,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-4:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.4,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T01:22:55.775 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi101.stdout: INFO: 2023-07-29T00:18:00.373+0000 7f91b3920640 -1 req 121437746836377053 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-25:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.25,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-25:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.25,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T01:22:55.776 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi101.stdout: INFO: 2023-07-29T00:18:00.378+0000 7f91c213d640 -1 req 11353311073100918810 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-3:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.3,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-3:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.4377.3,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T01:22:55.776 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi101.stdout: INFO: 2023-07-29T00:18:00.378+0000 7f91ad113640 -1 req 3289727646538734759 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hiiqos-43:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.8,z=3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433,az=0},d={b=hiiqos-43:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.8,z=59d1bb30-3b1c-476f-9111-052c1ea0b711,az=0}} returned ret=-2 2023-07-29T01:22:55.776 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'hiiqos-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:55.776 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'hiiqos-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:55.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket hiiqos-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:22:55.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:55.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:55.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:55.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "layout": { 2023-07-29T01:22:55.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "resharding": "None", 2023-07-29T01:22:55.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_index": { 2023-07-29T01:22:55.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "gen": 2, 2023-07-29T01:22:55.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "layout": { 2023-07-29T01:22:55.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "type": "Normal", 2023-07-29T01:22:55.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "normal": { 2023-07-29T01:22:55.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 3, 2023-07-29T01:22:55.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "hash_type": "Mod" 2023-07-29T01:22:55.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:55.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "logs": [ 2023-07-29T01:22:55.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:55.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "gen": 1, 2023-07-29T01:22:55.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "layout": { 2023-07-29T01:22:55.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "type": "InIndex", 2023-07-29T01:22:55.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "in_index": { 2023-07-29T01:22:55.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "gen": 1, 2023-07-29T01:22:55.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "layout": { 2023-07-29T01:22:55.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 3, 2023-07-29T01:22:55.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "hash_type": "Mod" 2023-07-29T01:22:55.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:55.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:55.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "gen": 2, 2023-07-29T01:22:55.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "layout": { 2023-07-29T01:22:55.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "type": "InIndex", 2023-07-29T01:22:55.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "in_index": { 2023-07-29T01:22:55.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "gen": 2, 2023-07-29T01:22:55.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "layout": { 2023-07-29T01:22:55.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 3, 2023-07-29T01:22:55.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "hash_type": "Mod" 2023-07-29T01:22:55.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:55.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.787 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-07-29T01:22:55.788 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.788 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-07-29T01:22:55.788 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2023-07-29T01:22:55.789 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-07-29T01:22:55.789 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-07-29T01:22:55.789 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_61140164d2579c30ffff3970615e0f82e40c6470/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-07-29T01:22:55.790 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-07-29T01:22:55.790 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_0484097b0572e96f42cf6402cbe0ac8dcb046577/qa/../src/test/rgw/rgw_multi/tests.py", line 1842, in test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2023-07-29T01:22:55.790 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2023-07-29T01:22:55.790 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_0484097b0572e96f42cf6402cbe0ac8dcb046577/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2023-07-29T01:22:55.791 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2023-07-29T01:22:55.791 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_0484097b0572e96f42cf6402cbe0ac8dcb046577/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2023-07-29T01:22:55.791 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2023-07-29T01:22:55.791 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_0484097b0572e96f42cf6402cbe0ac8dcb046577/qa/tasks/rgw_multisite.py", line 197, in admin 2023-07-29T01:22:55.792 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-07-29T01:22:55.792 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-07-29T01:22:55.792 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-07-29T01:22:55.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/ 2023-07-29T01:22:55.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/ 2023-07-29T01:22:55.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:55.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/ 2023-07-29T01:22:55.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.795 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:55.795 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.795 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.796 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:41 GMT 2023-07-29T01:22:55.796 INFO:tasks.rgw_multisite_tests:/hiiqos-63/ 2023-07-29T01:22:55.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.796 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:2bvNZ1aBirhX3w6bCKRv8F8HJwY= 2023-07-29T01:22:55.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:41 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2bvNZ1aBirhX3w6bCKRv8F8HJwY=', 'Content-Length': '0'} 2023-07-29T01:22:55.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000078fe0d60fa4a4563-0064c462d9-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:41 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.797 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=hiiqos-63 2023-07-29T01:22:55.797 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:55.797 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:55.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:55.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:55.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:55.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T01:22:55.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: [ 2023-07-29T01:22:55.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:55.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:55.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "last_update": "2023-07-29T00:40:17.836402Z" 2023-07-29T01:22:55.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:55.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:55.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591961.778137_1171.1", 2023-07-29T01:22:55.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "last_update": "2023-07-29T00:52:41.778137Z" 2023-07-29T01:22:55.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:55.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:55.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:55.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "last_update": "0.000000" 2023-07-29T01:22:55.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:55.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:55.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:55.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "last_update": "2023-07-29T00:40:12.811406Z" 2023-07-29T01:22:55.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:55.803 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1690591217.836402_294.1', 1: '1_1690591961.778137_1171.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:55.803 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-07-29T01:22:55.803 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:55.803 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:55.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:55.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:55.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:55.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:52:41.948+0000 7fc769610880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:55.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:52:41.949+0000 7fc769610880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:55.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:55.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:55.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:55.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:55.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:55.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:55.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:55.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:55.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:55.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:55.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:55.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:55.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:55.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:55.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:55.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 1, 2023-07-29T01:22:55.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:55.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:55.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:55.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:55.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:55.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:55.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:55.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:55.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591806.661758_1144.1", 2023-07-29T01:22:55.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:55.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:55.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:55.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:50:06.661758Z", 2023-07-29T01:22:55.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:55.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:55.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:55.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:55.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:55.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:55.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:55.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:55.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:55.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:55.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "0.000000", 2023-07-29T01:22:55.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:55.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:55.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:55.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:55.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:55.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:55.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:55.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:55.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:55.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:55.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:55.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:55.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:55.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:55.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:55.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 1, 2023-07-29T01:22:55.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 1 2023-07-29T01:22:55.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:55.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T00:52:41Z" 2023-07-29T01:22:55.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.820 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-07-29T01:22:55.820 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:55.820 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:55.820 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:55.821 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:55.821 INFO:tasks.rgw_multisite_tests: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:55.821 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:55.821 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:55.822 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:55.822 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:55.822 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:55.822 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:55.822 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:55.823 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:55.823 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:55.823 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-07-29T01:22:55.823 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:55.824 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:55.824 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:55.824 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:55.824 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:55.825 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:55.825 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:55.825 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:55.825 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:55.825 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591806.661758_1144.1", 2023-07-29T01:22:55.826 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:55.826 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:55.826 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:55.826 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:50:06.661758Z", 2023-07-29T01:22:55.827 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:55.827 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:55.827 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:55.827 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:55.827 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:55.828 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:55.828 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:55.828 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-07-29T01:22:55.828 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:55.829 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:55.829 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:55.829 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-07-29T01:22:55.829 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:55.830 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:55.830 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:55.830 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:55.830 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:55.830 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:55.831 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:55.831 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:55.831 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:55.831 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:55.832 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:55.832 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:55.832 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:55.832 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:55.832 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:55.833 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:55.833 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:55.833 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:55.833 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-07-29T01:22:55.834 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-07-29T01:22:55.834 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:55.834 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T00:52:41Z" 2023-07-29T01:22:55.834 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:55.835 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.835 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690591806.661758_1144.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:50:06.661758Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:22:55.835 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1690591217.836402_294.1', 1: '1_1690591961.778137_1171.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:55.835 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690591806.661758_1144.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:55.835 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1690591961.778137_1171.1 target=1_1690591806.661758_1144.1 2023-07-29T01:22:55.836 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi139.stdout: INFO: 2023-07-29T00:52:42.890+0000 7f5f2671b640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T01:22:55.836 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:55.836 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:55.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:55.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:55.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:55.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:52:47.076+0000 7f59b7a96880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:55.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:52:47.077+0000 7f59b7a96880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:55.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:55.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:55.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:55.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:55.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:55.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:55.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:55.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:55.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:55.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:55.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:55.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:55.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:55.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:55.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:55.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 1, 2023-07-29T01:22:55.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:55.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:55.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:55.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:55.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:55.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:55.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:55.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:55.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591961.778137_1171.1", 2023-07-29T01:22:55.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:55.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:55.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:55.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:52:41.778137Z", 2023-07-29T01:22:55.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:55.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:55.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:55.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:55.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:55.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:55.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:55.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:55.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:55.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:55.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "0.000000", 2023-07-29T01:22:55.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:55.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:55.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:55.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:55.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:55.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:55.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:55.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:55.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:55.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:55.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:55.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:55.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:55.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:55.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:55.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 1, 2023-07-29T01:22:55.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 1 2023-07-29T01:22:55.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:55.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T00:52:47Z" 2023-07-29T01:22:55.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:55.853 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-07-29T01:22:55.853 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:55.853 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:55.853 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:55.854 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:55.854 INFO:tasks.rgw_multisite_tests: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:55.854 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:55.854 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:55.855 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:55.855 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:55.855 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:55.855 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:55.855 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:55.856 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:55.856 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:55.856 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-07-29T01:22:55.856 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:55.857 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:55.857 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:55.857 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:55.857 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:55.858 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:55.858 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:55.858 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:55.858 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:55.858 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591961.778137_1171.1", 2023-07-29T01:22:55.859 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:55.859 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:55.859 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:55.859 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:52:41.778137Z", 2023-07-29T01:22:55.860 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:55.860 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:55.860 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:55.860 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:55.861 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:55.861 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:55.861 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:55.861 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-07-29T01:22:55.861 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:55.862 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:55.862 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:55.862 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-07-29T01:22:55.862 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:55.863 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:55.863 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:55.863 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:55.863 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:55.864 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:55.864 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:55.864 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:55.864 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:55.864 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:55.865 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:55.865 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:55.865 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:55.865 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:55.866 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:55.866 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:55.866 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:55.866 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:55.866 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-07-29T01:22:55.867 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-07-29T01:22:55.867 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:55.867 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T00:52:47Z" 2023-07-29T01:22:55.867 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:55.868 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.868 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690591961.778137_1171.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:52:41.778137Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:22:55.868 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1690591217.836402_294.1', 1: '1_1690591961.778137_1171.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:55.868 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690591961.778137_1171.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:55.869 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-07-29T01:22:55.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/ 2023-07-29T01:22:55.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/ 2023-07-29T01:22:55.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/ 2023-07-29T01:22:55.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.871 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.872 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.872 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.872 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:47 GMT 2023-07-29T01:22:55.872 INFO:tasks.rgw_multisite_tests:/hiiqos-63/ 2023-07-29T01:22:55.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.873 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY= 2023-07-29T01:22:55.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY=', 'Content-Length': '0'} 2023-07-29T01:22:55.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006aaefe0c62978ac9-0064c462df-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/obj11 2023-07-29T01:22:55.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/obj11 2023-07-29T01:22:55.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:55.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/obj11 2023-07-29T01:22:55.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:55.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.876 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:55.876 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:55.876 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:55.877 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:47 GMT 2023-07-29T01:22:55.877 INFO:tasks.rgw_multisite_tests:/hiiqos-63/obj11 2023-07-29T01:22:55.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.877 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:FDEAqF4McvsPS8wFDkpORGSt3DU= 2023-07-29T01:22:55.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FDEAqF4McvsPS8wFDkpORGSt3DU='} 2023-07-29T01:22:55.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000091d131e7330402e2-0064c462df-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/ 2023-07-29T01:22:55.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/ 2023-07-29T01:22:55.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/ 2023-07-29T01:22:55.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.881 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.881 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.881 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.881 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:47 GMT 2023-07-29T01:22:55.881 INFO:tasks.rgw_multisite_tests:/hiiqos-63/ 2023-07-29T01:22:55.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.882 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY= 2023-07-29T01:22:55.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY=', 'Content-Length': '0'} 2023-07-29T01:22:55.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000056d781a5e1a4cba8-0064c462df-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/obj22 2023-07-29T01:22:55.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/obj22 2023-07-29T01:22:55.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:55.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/obj22 2023-07-29T01:22:55.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:55.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.885 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:55.885 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:55.886 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:55.886 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:47 GMT 2023-07-29T01:22:55.886 INFO:tasks.rgw_multisite_tests:/hiiqos-63/obj22 2023-07-29T01:22:55.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.886 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:5RfvdwOAWQ1/fhrpxAmKgzv6ayY= 2023-07-29T01:22:55.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:5RfvdwOAWQ1/fhrpxAmKgzv6ayY='} 2023-07-29T01:22:55.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000082848076080bd587-0064c462df-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/ 2023-07-29T01:22:55.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/ 2023-07-29T01:22:55.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/ 2023-07-29T01:22:55.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.890 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.890 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.890 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.890 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:47 GMT 2023-07-29T01:22:55.891 INFO:tasks.rgw_multisite_tests:/hiiqos-63/ 2023-07-29T01:22:55.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.891 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY= 2023-07-29T01:22:55.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY=', 'Content-Length': '0'} 2023-07-29T01:22:55.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000241fd97bbd7fb88c-0064c462df-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/obj33 2023-07-29T01:22:55.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/obj33 2023-07-29T01:22:55.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:55.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/obj33 2023-07-29T01:22:55.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:55.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.894 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:55.895 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:55.895 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:55.895 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:47 GMT 2023-07-29T01:22:55.895 INFO:tasks.rgw_multisite_tests:/hiiqos-63/obj33 2023-07-29T01:22:55.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.896 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:rV2w/2Hzud3s1dV3aLkD89M7VrE= 2023-07-29T01:22:55.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rV2w/2Hzud3s1dV3aLkD89M7VrE='} 2023-07-29T01:22:55.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c2e6443fab7f9142-0064c462df-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/ 2023-07-29T01:22:55.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/ 2023-07-29T01:22:55.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/ 2023-07-29T01:22:55.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.899 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.899 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.899 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.900 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:47 GMT 2023-07-29T01:22:55.900 INFO:tasks.rgw_multisite_tests:/hiiqos-63/ 2023-07-29T01:22:55.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.900 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY= 2023-07-29T01:22:55.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY=', 'Content-Length': '0'} 2023-07-29T01:22:55.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c3fbc6e47d91c511-0064c462df-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/obj44 2023-07-29T01:22:55.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/obj44 2023-07-29T01:22:55.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:55.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/obj44 2023-07-29T01:22:55.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:55.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.903 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:55.904 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:55.904 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:55.904 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:47 GMT 2023-07-29T01:22:55.904 INFO:tasks.rgw_multisite_tests:/hiiqos-63/obj44 2023-07-29T01:22:55.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.905 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:tXOJnsJsnCx3+rbpzL96dUT+gjk= 2023-07-29T01:22:55.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tXOJnsJsnCx3+rbpzL96dUT+gjk='} 2023-07-29T01:22:55.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ffa9cef4cc010a33-0064c462df-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/ 2023-07-29T01:22:55.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/ 2023-07-29T01:22:55.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/ 2023-07-29T01:22:55.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.908 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.908 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.909 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.909 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:47 GMT 2023-07-29T01:22:55.909 INFO:tasks.rgw_multisite_tests:/hiiqos-63/ 2023-07-29T01:22:55.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.909 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY= 2023-07-29T01:22:55.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY=', 'Content-Length': '0'} 2023-07-29T01:22:55.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003de7e3620e63e4c5-0064c462df-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/obj55 2023-07-29T01:22:55.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/obj55 2023-07-29T01:22:55.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:55.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/obj55 2023-07-29T01:22:55.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:55.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.913 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:55.913 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:55.913 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:55.913 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:47 GMT 2023-07-29T01:22:55.914 INFO:tasks.rgw_multisite_tests:/hiiqos-63/obj55 2023-07-29T01:22:55.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.914 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:ixOggHqPmbOUN8Umfqmg1YBu1IA= 2023-07-29T01:22:55.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ixOggHqPmbOUN8Umfqmg1YBu1IA='} 2023-07-29T01:22:55.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000027b42e07ada3d0b0-0064c462df-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.915 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:55.915 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:55.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-63 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:22:55.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:55.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:55.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: tenant: 2023-07-29T01:22:55.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: bucket name: hiiqos-63 2023-07-29T01:22:55.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: total entries: 5 2023-07-29T01:22:55.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/ 2023-07-29T01:22:55.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/ 2023-07-29T01:22:55.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/ 2023-07-29T01:22:55.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.919 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.919 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.920 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.920 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:47 GMT 2023-07-29T01:22:55.920 INFO:tasks.rgw_multisite_tests:/hiiqos-63/ 2023-07-29T01:22:55.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.921 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY= 2023-07-29T01:22:55.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY=', 'Content-Length': '0'} 2023-07-29T01:22:55.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000090fc102d652655e9-0064c462df-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/obj17 2023-07-29T01:22:55.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/obj17 2023-07-29T01:22:55.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:55.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/obj17 2023-07-29T01:22:55.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:55.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.924 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:55.924 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:55.924 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:55.924 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:47 GMT 2023-07-29T01:22:55.925 INFO:tasks.rgw_multisite_tests:/hiiqos-63/obj17 2023-07-29T01:22:55.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.925 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:w9onP5BT0/8a6OKT7KxD0mPWkOE= 2023-07-29T01:22:55.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w9onP5BT0/8a6OKT7KxD0mPWkOE='} 2023-07-29T01:22:55.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004692f5d49a481be9-0064c462df-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/ 2023-07-29T01:22:55.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/ 2023-07-29T01:22:55.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/ 2023-07-29T01:22:55.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.928 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.929 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.929 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.929 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:47 GMT 2023-07-29T01:22:55.929 INFO:tasks.rgw_multisite_tests:/hiiqos-63/ 2023-07-29T01:22:55.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.930 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY= 2023-07-29T01:22:55.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY=', 'Content-Length': '0'} 2023-07-29T01:22:55.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001f49511e0fb4700f-0064c462df-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/obj34 2023-07-29T01:22:55.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/obj34 2023-07-29T01:22:55.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:55.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/obj34 2023-07-29T01:22:55.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:55.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.933 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:55.933 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:55.933 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:55.934 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:47 GMT 2023-07-29T01:22:55.934 INFO:tasks.rgw_multisite_tests:/hiiqos-63/obj34 2023-07-29T01:22:55.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.934 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:Fn+vAlFC4/le7qoDTxucI2ljQTg= 2023-07-29T01:22:55.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Fn+vAlFC4/le7qoDTxucI2ljQTg='} 2023-07-29T01:22:55.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001bb2c888fe4c70a9-0064c462df-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/ 2023-07-29T01:22:55.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/ 2023-07-29T01:22:55.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/ 2023-07-29T01:22:55.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.938 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.938 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.938 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.938 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:47 GMT 2023-07-29T01:22:55.939 INFO:tasks.rgw_multisite_tests:/hiiqos-63/ 2023-07-29T01:22:55.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.939 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY= 2023-07-29T01:22:55.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY=', 'Content-Length': '0'} 2023-07-29T01:22:55.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008322bbe0ae308272-0064c462df-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/obj51 2023-07-29T01:22:55.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/obj51 2023-07-29T01:22:55.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:55.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/obj51 2023-07-29T01:22:55.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:55.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.942 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:55.942 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:55.943 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:55.943 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:47 GMT 2023-07-29T01:22:55.943 INFO:tasks.rgw_multisite_tests:/hiiqos-63/obj51 2023-07-29T01:22:55.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.944 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:dG0Xb9tlpNUkuZyaj/LvCKOiCCo= 2023-07-29T01:22:55.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dG0Xb9tlpNUkuZyaj/LvCKOiCCo='} 2023-07-29T01:22:55.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000916ad571b41d2802-0064c462df-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/ 2023-07-29T01:22:55.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/ 2023-07-29T01:22:55.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/ 2023-07-29T01:22:55.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.947 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.947 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.947 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.947 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:47 GMT 2023-07-29T01:22:55.948 INFO:tasks.rgw_multisite_tests:/hiiqos-63/ 2023-07-29T01:22:55.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.948 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY= 2023-07-29T01:22:55.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY=', 'Content-Length': '0'} 2023-07-29T01:22:55.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f6a8f6d5c32aab01-0064c462df-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/obj68 2023-07-29T01:22:55.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/obj68 2023-07-29T01:22:55.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:55.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/obj68 2023-07-29T01:22:55.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:55.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.951 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:55.952 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:55.952 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:55.952 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:47 GMT 2023-07-29T01:22:55.952 INFO:tasks.rgw_multisite_tests:/hiiqos-63/obj68 2023-07-29T01:22:55.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.953 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:qLFVOy6EgHyK1mThOaFQulzd7RI= 2023-07-29T01:22:55.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:qLFVOy6EgHyK1mThOaFQulzd7RI='} 2023-07-29T01:22:55.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a3385110ffc5d686-0064c462df-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/ 2023-07-29T01:22:55.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/ 2023-07-29T01:22:55.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/ 2023-07-29T01:22:55.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.956 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.956 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.956 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.957 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:47 GMT 2023-07-29T01:22:55.957 INFO:tasks.rgw_multisite_tests:/hiiqos-63/ 2023-07-29T01:22:55.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.957 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY= 2023-07-29T01:22:55.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY=', 'Content-Length': '0'} 2023-07-29T01:22:55.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f5bc3e8d5c704b99-0064c462df-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/obj85 2023-07-29T01:22:55.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/obj85 2023-07-29T01:22:55.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:55.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/obj85 2023-07-29T01:22:55.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:55.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.960 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:55.961 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:55.961 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:55.961 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:47 GMT 2023-07-29T01:22:55.961 INFO:tasks.rgw_multisite_tests:/hiiqos-63/obj85 2023-07-29T01:22:55.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.962 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:3lDEpkPpuKph0n7R7e2qIytU7OU= 2023-07-29T01:22:55.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:3lDEpkPpuKph0n7R7e2qIytU7OU='} 2023-07-29T01:22:55.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000e199bdaae282cc5-0064c462df-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.963 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:55.963 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:55.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-63 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:22:55.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:55.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:55.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: tenant: 2023-07-29T01:22:55.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: bucket name: hiiqos-63 2023-07-29T01:22:55.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: total entries: 10 2023-07-29T01:22:55.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/ 2023-07-29T01:22:55.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/ 2023-07-29T01:22:55.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/ 2023-07-29T01:22:55.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.967 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.967 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.967 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.968 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:47 GMT 2023-07-29T01:22:55.968 INFO:tasks.rgw_multisite_tests:/hiiqos-63/ 2023-07-29T01:22:55.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.968 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY= 2023-07-29T01:22:55.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY=', 'Content-Length': '0'} 2023-07-29T01:22:55.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000abd4604958ac2f30-0064c462df-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/obj19 2023-07-29T01:22:55.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/obj19 2023-07-29T01:22:55.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:55.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/obj19 2023-07-29T01:22:55.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:55.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.971 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:55.972 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:55.972 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:55.972 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:47 GMT 2023-07-29T01:22:55.972 INFO:tasks.rgw_multisite_tests:/hiiqos-63/obj19 2023-07-29T01:22:55.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.973 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:DJ5M1Pei2AYkBC7+f1VLKgXPiKE= 2023-07-29T01:22:55.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:DJ5M1Pei2AYkBC7+f1VLKgXPiKE='} 2023-07-29T01:22:55.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000038f60d6cdfff663b-0064c462df-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/ 2023-07-29T01:22:55.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/ 2023-07-29T01:22:55.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/ 2023-07-29T01:22:55.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.976 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.976 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.977 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.977 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:47 GMT 2023-07-29T01:22:55.977 INFO:tasks.rgw_multisite_tests:/hiiqos-63/ 2023-07-29T01:22:55.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.977 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY= 2023-07-29T01:22:55.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY=', 'Content-Length': '0'} 2023-07-29T01:22:55.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000285d5d49f1ba8217-0064c462df-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/obj38 2023-07-29T01:22:55.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/obj38 2023-07-29T01:22:55.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:55.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/obj38 2023-07-29T01:22:55.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:55.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.981 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:55.981 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:55.981 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:55.981 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:47 GMT 2023-07-29T01:22:55.982 INFO:tasks.rgw_multisite_tests:/hiiqos-63/obj38 2023-07-29T01:22:55.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.982 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:lw1WXMcRdCaZFVWLYo9/80TRVAw= 2023-07-29T01:22:55.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lw1WXMcRdCaZFVWLYo9/80TRVAw='} 2023-07-29T01:22:55.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000a3ebed091c825ed-0064c462df-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/ 2023-07-29T01:22:55.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/ 2023-07-29T01:22:55.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/ 2023-07-29T01:22:55.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.985 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.986 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.986 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.986 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:47 GMT 2023-07-29T01:22:55.986 INFO:tasks.rgw_multisite_tests:/hiiqos-63/ 2023-07-29T01:22:55.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.987 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY= 2023-07-29T01:22:55.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY=', 'Content-Length': '0'} 2023-07-29T01:22:55.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001880060618172a83-0064c462df-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/obj57 2023-07-29T01:22:55.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/obj57 2023-07-29T01:22:55.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:55.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/obj57 2023-07-29T01:22:55.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:55.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.990 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:55.990 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:55.990 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:55.991 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:47 GMT 2023-07-29T01:22:55.991 INFO:tasks.rgw_multisite_tests:/hiiqos-63/obj57 2023-07-29T01:22:55.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.991 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:gFreMUu8SggIj39pFnO+jwASbZY= 2023-07-29T01:22:55.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gFreMUu8SggIj39pFnO+jwASbZY='} 2023-07-29T01:22:55.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005c8f810930848ec9-0064c462df-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/ 2023-07-29T01:22:55.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/ 2023-07-29T01:22:55.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:55.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/ 2023-07-29T01:22:55.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:55.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.994 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:55.995 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.995 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:55.995 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:47 GMT 2023-07-29T01:22:55.995 INFO:tasks.rgw_multisite_tests:/hiiqos-63/ 2023-07-29T01:22:55.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:55.996 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY= 2023-07-29T01:22:55.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY=', 'Content-Length': '0'} 2023-07-29T01:22:55.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000b94a9266b87e628-0064c462df-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:55.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/obj76 2023-07-29T01:22:55.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/obj76 2023-07-29T01:22:55.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:55.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/obj76 2023-07-29T01:22:55.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:55.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:55.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:55.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:55.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:55.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:55.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:55.999 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:55.999 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.000 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.000 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:47 GMT 2023-07-29T01:22:56.000 INFO:tasks.rgw_multisite_tests:/hiiqos-63/obj76 2023-07-29T01:22:56.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.001 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:UpOGZu55jT4r3YsXtLnCvzihgJI= 2023-07-29T01:22:56.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UpOGZu55jT4r3YsXtLnCvzihgJI='} 2023-07-29T01:22:56.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000093f0b411efbba67e-0064c462df-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/ 2023-07-29T01:22:56.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/ 2023-07-29T01:22:56.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/ 2023-07-29T01:22:56.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.004 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.004 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.004 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.004 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:47 GMT 2023-07-29T01:22:56.005 INFO:tasks.rgw_multisite_tests:/hiiqos-63/ 2023-07-29T01:22:56.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.005 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY= 2023-07-29T01:22:56.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CPZFqDTyqVAyN1Qo0pm/soT/nQY=', 'Content-Length': '0'} 2023-07-29T01:22:56.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003f118db536979af4-0064c462df-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/obj95 2023-07-29T01:22:56.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/obj95 2023-07-29T01:22:56.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/obj95 2023-07-29T01:22:56.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.008 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.009 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.009 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.009 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:47 GMT 2023-07-29T01:22:56.009 INFO:tasks.rgw_multisite_tests:/hiiqos-63/obj95 2023-07-29T01:22:56.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.010 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:0izX6oNbpKgxXJc0CJ1Urj7Nol4= 2023-07-29T01:22:56.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:47 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0izX6oNbpKgxXJc0CJ1Urj7Nol4='} 2023-07-29T01:22:56.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000092b9064117b173d4-0064c462df-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:47 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.010 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:56.011 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:56.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-63 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:22:56.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:56.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:56.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: tenant: 2023-07-29T01:22:56.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: bucket name: hiiqos-63 2023-07-29T01:22:56.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: total entries: 15 2023-07-29T01:22:56.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/ 2023-07-29T01:22:56.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/ 2023-07-29T01:22:56.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/ 2023-07-29T01:22:56.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.015 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.015 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.015 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.016 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:48 GMT 2023-07-29T01:22:56.016 INFO:tasks.rgw_multisite_tests:/hiiqos-63/ 2023-07-29T01:22:56.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.016 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw= 2023-07-29T01:22:56.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw=', 'Content-Length': '0'} 2023-07-29T01:22:56.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083ba17f2ce0e5641-0064c462e0-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/obj23 2023-07-29T01:22:56.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/obj23 2023-07-29T01:22:56.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/obj23 2023-07-29T01:22:56.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.019 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.020 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.020 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.020 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:48 GMT 2023-07-29T01:22:56.020 INFO:tasks.rgw_multisite_tests:/hiiqos-63/obj23 2023-07-29T01:22:56.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.021 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:bMc5wXYR3mpXSs5wiZpy8ov9xrI= 2023-07-29T01:22:56.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bMc5wXYR3mpXSs5wiZpy8ov9xrI='} 2023-07-29T01:22:56.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004ee69e1fd1956cc7-0064c462e0-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/ 2023-07-29T01:22:56.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/ 2023-07-29T01:22:56.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/ 2023-07-29T01:22:56.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.024 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.024 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.025 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.025 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:48 GMT 2023-07-29T01:22:56.025 INFO:tasks.rgw_multisite_tests:/hiiqos-63/ 2023-07-29T01:22:56.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.025 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw= 2023-07-29T01:22:56.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw=', 'Content-Length': '0'} 2023-07-29T01:22:56.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a0a5995aad5a55a1-0064c462e0-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/obj46 2023-07-29T01:22:56.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/obj46 2023-07-29T01:22:56.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/obj46 2023-07-29T01:22:56.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.029 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.029 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.029 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.029 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:48 GMT 2023-07-29T01:22:56.030 INFO:tasks.rgw_multisite_tests:/hiiqos-63/obj46 2023-07-29T01:22:56.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.030 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:XZWj4zmWj3v5ypaUtxK6ZhZzLco= 2023-07-29T01:22:56.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XZWj4zmWj3v5ypaUtxK6ZhZzLco='} 2023-07-29T01:22:56.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000083f365ae3165c12f-0064c462e0-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/ 2023-07-29T01:22:56.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/ 2023-07-29T01:22:56.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/ 2023-07-29T01:22:56.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.033 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.033 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.034 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.034 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:48 GMT 2023-07-29T01:22:56.034 INFO:tasks.rgw_multisite_tests:/hiiqos-63/ 2023-07-29T01:22:56.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.035 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw= 2023-07-29T01:22:56.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw=', 'Content-Length': '0'} 2023-07-29T01:22:56.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000005fccb74972fb4f7-0064c462e0-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/obj69 2023-07-29T01:22:56.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/obj69 2023-07-29T01:22:56.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/obj69 2023-07-29T01:22:56.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.038 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.038 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.038 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.038 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:48 GMT 2023-07-29T01:22:56.039 INFO:tasks.rgw_multisite_tests:/hiiqos-63/obj69 2023-07-29T01:22:56.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.039 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:x8MkGqGGcQcJmiEuzRYlMdBCW6Q= 2023-07-29T01:22:56.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:x8MkGqGGcQcJmiEuzRYlMdBCW6Q='} 2023-07-29T01:22:56.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007c73792c10649443-0064c462e0-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/ 2023-07-29T01:22:56.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/ 2023-07-29T01:22:56.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/ 2023-07-29T01:22:56.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.042 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.043 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.043 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.043 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:48 GMT 2023-07-29T01:22:56.043 INFO:tasks.rgw_multisite_tests:/hiiqos-63/ 2023-07-29T01:22:56.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.044 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw= 2023-07-29T01:22:56.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw=', 'Content-Length': '0'} 2023-07-29T01:22:56.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af128e4a1a2fc028-0064c462e0-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/obj92 2023-07-29T01:22:56.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/obj92 2023-07-29T01:22:56.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/obj92 2023-07-29T01:22:56.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.047 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.047 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.047 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.048 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:48 GMT 2023-07-29T01:22:56.048 INFO:tasks.rgw_multisite_tests:/hiiqos-63/obj92 2023-07-29T01:22:56.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.048 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:KY2ph1h435YyAZT6MY9ZMCqsbI0= 2023-07-29T01:22:56.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KY2ph1h435YyAZT6MY9ZMCqsbI0='} 2023-07-29T01:22:56.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b874c8e1148ffe6e-0064c462e0-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/ 2023-07-29T01:22:56.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/ 2023-07-29T01:22:56.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/ 2023-07-29T01:22:56.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.052 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.052 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.052 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.052 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:48 GMT 2023-07-29T01:22:56.053 INFO:tasks.rgw_multisite_tests:/hiiqos-63/ 2023-07-29T01:22:56.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.053 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw= 2023-07-29T01:22:56.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw=', 'Content-Length': '0'} 2023-07-29T01:22:56.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009730670278b674d8-0064c462e0-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/obj115 2023-07-29T01:22:56.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/obj115 2023-07-29T01:22:56.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/obj115 2023-07-29T01:22:56.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.056 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.056 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.057 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.057 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:48 GMT 2023-07-29T01:22:56.057 INFO:tasks.rgw_multisite_tests:/hiiqos-63/obj115 2023-07-29T01:22:56.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.058 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:m4P6MSsNJ/ttbjbyi2E1eJq0Xx0= 2023-07-29T01:22:56.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:m4P6MSsNJ/ttbjbyi2E1eJq0Xx0='} 2023-07-29T01:22:56.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e83f9bf9af3782fb-0064c462e0-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.058 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:56.059 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:56.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-63 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:22:56.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:56.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:56.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: tenant: 2023-07-29T01:22:56.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: bucket name: hiiqos-63 2023-07-29T01:22:56.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: total entries: 20 2023-07-29T01:22:56.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/ 2023-07-29T01:22:56.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/ 2023-07-29T01:22:56.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/ 2023-07-29T01:22:56.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.063 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.063 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.063 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.063 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:48 GMT 2023-07-29T01:22:56.064 INFO:tasks.rgw_multisite_tests:/hiiqos-63/ 2023-07-29T01:22:56.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.064 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw= 2023-07-29T01:22:56.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw=', 'Content-Length': '0'} 2023-07-29T01:22:56.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e5a2d696b4e4c8af-0064c462e0-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/obj29 2023-07-29T01:22:56.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/obj29 2023-07-29T01:22:56.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/obj29 2023-07-29T01:22:56.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.067 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.068 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.068 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.068 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:48 GMT 2023-07-29T01:22:56.068 INFO:tasks.rgw_multisite_tests:/hiiqos-63/obj29 2023-07-29T01:22:56.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.069 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:7gBp5DQZrl0kOhYUmABrWX8yFDM= 2023-07-29T01:22:56.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:7gBp5DQZrl0kOhYUmABrWX8yFDM='} 2023-07-29T01:22:56.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000054cd117ff33cec94-0064c462e0-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/ 2023-07-29T01:22:56.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/ 2023-07-29T01:22:56.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/ 2023-07-29T01:22:56.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.072 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.072 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.072 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.073 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:48 GMT 2023-07-29T01:22:56.073 INFO:tasks.rgw_multisite_tests:/hiiqos-63/ 2023-07-29T01:22:56.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.073 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw= 2023-07-29T01:22:56.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw=', 'Content-Length': '0'} 2023-07-29T01:22:56.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000247b801b3e418185-0064c462e0-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/obj58 2023-07-29T01:22:56.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/obj58 2023-07-29T01:22:56.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/obj58 2023-07-29T01:22:56.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.076 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.077 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.077 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.077 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:48 GMT 2023-07-29T01:22:56.077 INFO:tasks.rgw_multisite_tests:/hiiqos-63/obj58 2023-07-29T01:22:56.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.078 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:+sjnACAhtQv2olv2zLGgLtnjcK4= 2023-07-29T01:22:56.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+sjnACAhtQv2olv2zLGgLtnjcK4='} 2023-07-29T01:22:56.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000042b2c974c4c6e92-0064c462e0-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/ 2023-07-29T01:22:56.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/ 2023-07-29T01:22:56.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/ 2023-07-29T01:22:56.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.081 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.081 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.082 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.082 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:48 GMT 2023-07-29T01:22:56.082 INFO:tasks.rgw_multisite_tests:/hiiqos-63/ 2023-07-29T01:22:56.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.082 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw= 2023-07-29T01:22:56.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw=', 'Content-Length': '0'} 2023-07-29T01:22:56.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000759e56b2862427fc-0064c462e0-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/obj87 2023-07-29T01:22:56.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/obj87 2023-07-29T01:22:56.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/obj87 2023-07-29T01:22:56.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.086 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.086 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.086 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.086 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:48 GMT 2023-07-29T01:22:56.087 INFO:tasks.rgw_multisite_tests:/hiiqos-63/obj87 2023-07-29T01:22:56.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.087 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:ddeMguuD2QT+/urwvI7M/eVNTHE= 2023-07-29T01:22:56.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ddeMguuD2QT+/urwvI7M/eVNTHE='} 2023-07-29T01:22:56.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005fb6bfa22ee4f53f-0064c462e0-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/ 2023-07-29T01:22:56.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/ 2023-07-29T01:22:56.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/ 2023-07-29T01:22:56.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.090 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.091 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.091 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.091 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:48 GMT 2023-07-29T01:22:56.091 INFO:tasks.rgw_multisite_tests:/hiiqos-63/ 2023-07-29T01:22:56.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.092 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw= 2023-07-29T01:22:56.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw=', 'Content-Length': '0'} 2023-07-29T01:22:56.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003a30b46abe67a6ea-0064c462e0-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/obj116 2023-07-29T01:22:56.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/obj116 2023-07-29T01:22:56.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/obj116 2023-07-29T01:22:56.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.095 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.095 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.095 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.096 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:48 GMT 2023-07-29T01:22:56.096 INFO:tasks.rgw_multisite_tests:/hiiqos-63/obj116 2023-07-29T01:22:56.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.096 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:LjMRX9lxTYaYC3sfBTG0qmY+v6A= 2023-07-29T01:22:56.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LjMRX9lxTYaYC3sfBTG0qmY+v6A='} 2023-07-29T01:22:56.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fb5ab7ca9449d5a4-0064c462e0-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/ 2023-07-29T01:22:56.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/ 2023-07-29T01:22:56.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/ 2023-07-29T01:22:56.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.100 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.100 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.100 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.100 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:48 GMT 2023-07-29T01:22:56.100 INFO:tasks.rgw_multisite_tests:/hiiqos-63/ 2023-07-29T01:22:56.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.101 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw= 2023-07-29T01:22:56.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw=', 'Content-Length': '0'} 2023-07-29T01:22:56.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e1a63bdd4abc182-0064c462e0-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/obj145 2023-07-29T01:22:56.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/obj145 2023-07-29T01:22:56.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/obj145 2023-07-29T01:22:56.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.104 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.104 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.105 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.105 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:48 GMT 2023-07-29T01:22:56.105 INFO:tasks.rgw_multisite_tests:/hiiqos-63/obj145 2023-07-29T01:22:56.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.106 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:uMLiTD6jD1lzKt+mSnbM7lEbSA8= 2023-07-29T01:22:56.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:uMLiTD6jD1lzKt+mSnbM7lEbSA8='} 2023-07-29T01:22:56.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dc622bc3f0c96533-0064c462e0-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.106 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:56.106 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:56.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-63 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:22:56.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:56.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:56.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: tenant: 2023-07-29T01:22:56.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: bucket name: hiiqos-63 2023-07-29T01:22:56.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: total entries: 25 2023-07-29T01:22:56.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/ 2023-07-29T01:22:56.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/ 2023-07-29T01:22:56.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/ 2023-07-29T01:22:56.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.111 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.111 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.111 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.111 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:48 GMT 2023-07-29T01:22:56.112 INFO:tasks.rgw_multisite_tests:/hiiqos-63/ 2023-07-29T01:22:56.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.112 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw= 2023-07-29T01:22:56.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw=', 'Content-Length': '0'} 2023-07-29T01:22:56.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000687fed2bab598481-0064c462e0-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/obj31 2023-07-29T01:22:56.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/obj31 2023-07-29T01:22:56.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/obj31 2023-07-29T01:22:56.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.115 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.115 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.116 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.116 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:48 GMT 2023-07-29T01:22:56.116 INFO:tasks.rgw_multisite_tests:/hiiqos-63/obj31 2023-07-29T01:22:56.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.117 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:NZ4DnIYN6uNC36fkB55JE33IXKI= 2023-07-29T01:22:56.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:NZ4DnIYN6uNC36fkB55JE33IXKI='} 2023-07-29T01:22:56.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000798730b0cf006b10-0064c462e0-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/ 2023-07-29T01:22:56.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/ 2023-07-29T01:22:56.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/ 2023-07-29T01:22:56.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.120 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.120 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.120 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.120 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:48 GMT 2023-07-29T01:22:56.121 INFO:tasks.rgw_multisite_tests:/hiiqos-63/ 2023-07-29T01:22:56.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.121 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw= 2023-07-29T01:22:56.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw=', 'Content-Length': '0'} 2023-07-29T01:22:56.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000efd34ce77372798d-0064c462e0-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/obj62 2023-07-29T01:22:56.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/obj62 2023-07-29T01:22:56.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/obj62 2023-07-29T01:22:56.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.124 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.125 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.125 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.125 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:48 GMT 2023-07-29T01:22:56.125 INFO:tasks.rgw_multisite_tests:/hiiqos-63/obj62 2023-07-29T01:22:56.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.126 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:VVgKWJPhA7HwoabqUUFdudJIDuI= 2023-07-29T01:22:56.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VVgKWJPhA7HwoabqUUFdudJIDuI='} 2023-07-29T01:22:56.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bc6180bb5911f71e-0064c462e0-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/ 2023-07-29T01:22:56.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/ 2023-07-29T01:22:56.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/ 2023-07-29T01:22:56.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.129 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.129 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.129 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.130 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:48 GMT 2023-07-29T01:22:56.130 INFO:tasks.rgw_multisite_tests:/hiiqos-63/ 2023-07-29T01:22:56.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.130 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw= 2023-07-29T01:22:56.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw=', 'Content-Length': '0'} 2023-07-29T01:22:56.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001eddd0f46e0be57c-0064c462e0-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/obj93 2023-07-29T01:22:56.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/obj93 2023-07-29T01:22:56.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/obj93 2023-07-29T01:22:56.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.134 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.134 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.134 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.134 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:48 GMT 2023-07-29T01:22:56.134 INFO:tasks.rgw_multisite_tests:/hiiqos-63/obj93 2023-07-29T01:22:56.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.135 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:VHkjJ0U5Gp+fR2fbQcEBCk5TOgE= 2023-07-29T01:22:56.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VHkjJ0U5Gp+fR2fbQcEBCk5TOgE='} 2023-07-29T01:22:56.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cb3826b67a0df531-0064c462e0-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/ 2023-07-29T01:22:56.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/ 2023-07-29T01:22:56.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/ 2023-07-29T01:22:56.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.138 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.138 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.139 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.139 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:48 GMT 2023-07-29T01:22:56.139 INFO:tasks.rgw_multisite_tests:/hiiqos-63/ 2023-07-29T01:22:56.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.140 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw= 2023-07-29T01:22:56.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw=', 'Content-Length': '0'} 2023-07-29T01:22:56.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000096cd2985138f75ab-0064c462e0-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/obj124 2023-07-29T01:22:56.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/obj124 2023-07-29T01:22:56.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/obj124 2023-07-29T01:22:56.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.143 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.143 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.143 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.143 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:48 GMT 2023-07-29T01:22:56.144 INFO:tasks.rgw_multisite_tests:/hiiqos-63/obj124 2023-07-29T01:22:56.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.144 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:zvS9w9xgyMF6HpsXxFqZlEYpcZo= 2023-07-29T01:22:56.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zvS9w9xgyMF6HpsXxFqZlEYpcZo='} 2023-07-29T01:22:56.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008fe94d3cfbf8e9ca-0064c462e0-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/ 2023-07-29T01:22:56.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/ 2023-07-29T01:22:56.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/ 2023-07-29T01:22:56.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.147 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.148 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.148 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.148 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:48 GMT 2023-07-29T01:22:56.148 INFO:tasks.rgw_multisite_tests:/hiiqos-63/ 2023-07-29T01:22:56.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.149 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw= 2023-07-29T01:22:56.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rETDl0nxDyszxIzFYzDS+TOCQZw=', 'Content-Length': '0'} 2023-07-29T01:22:56.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e5f490e81edc5912-0064c462e0-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/obj155 2023-07-29T01:22:56.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/obj155 2023-07-29T01:22:56.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/obj155 2023-07-29T01:22:56.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.152 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.152 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.153 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.153 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:48 GMT 2023-07-29T01:22:56.153 INFO:tasks.rgw_multisite_tests:/hiiqos-63/obj155 2023-07-29T01:22:56.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.153 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:XZxa61xLFumD8i1TnFtGPH7S3+4= 2023-07-29T01:22:56.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:48 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:XZxa61xLFumD8i1TnFtGPH7S3+4='} 2023-07-29T01:22:56.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005dccba726097e368-0064c462e0-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:48 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.154 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:56.154 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:56.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-63 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:22:56.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:56.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:56.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: tenant: 2023-07-29T01:22:56.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: bucket name: hiiqos-63 2023-07-29T01:22:56.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: total entries: 30 2023-07-29T01:22:56.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/ 2023-07-29T01:22:56.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/ 2023-07-29T01:22:56.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/ 2023-07-29T01:22:56.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.159 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.159 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.159 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.159 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:49 GMT 2023-07-29T01:22:56.159 INFO:tasks.rgw_multisite_tests:/hiiqos-63/ 2023-07-29T01:22:56.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.160 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:eMFPRv3AuzfRuxq+oty/0aQF6gY= 2023-07-29T01:22:56.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eMFPRv3AuzfRuxq+oty/0aQF6gY=', 'Content-Length': '0'} 2023-07-29T01:22:56.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009d3a70f7dab5fc81-0064c462e1-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/obj37 2023-07-29T01:22:56.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/obj37 2023-07-29T01:22:56.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/obj37 2023-07-29T01:22:56.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.163 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.163 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.164 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.164 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:49 GMT 2023-07-29T01:22:56.164 INFO:tasks.rgw_multisite_tests:/hiiqos-63/obj37 2023-07-29T01:22:56.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.164 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:cKEouCkG8diC3PJtoU7dOVWPKt0= 2023-07-29T01:22:56.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cKEouCkG8diC3PJtoU7dOVWPKt0='} 2023-07-29T01:22:56.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000069c7b146c372069f-0064c462e1-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/ 2023-07-29T01:22:56.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/ 2023-07-29T01:22:56.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/ 2023-07-29T01:22:56.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.168 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.168 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.168 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.168 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:49 GMT 2023-07-29T01:22:56.169 INFO:tasks.rgw_multisite_tests:/hiiqos-63/ 2023-07-29T01:22:56.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.169 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:eMFPRv3AuzfRuxq+oty/0aQF6gY= 2023-07-29T01:22:56.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eMFPRv3AuzfRuxq+oty/0aQF6gY=', 'Content-Length': '0'} 2023-07-29T01:22:56.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d03baad2b9cd931-0064c462e1-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/obj74 2023-07-29T01:22:56.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/obj74 2023-07-29T01:22:56.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/obj74 2023-07-29T01:22:56.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.172 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.173 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.173 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.173 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:49 GMT 2023-07-29T01:22:56.173 INFO:tasks.rgw_multisite_tests:/hiiqos-63/obj74 2023-07-29T01:22:56.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.174 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:f0/VCNxcTcEocfUjNZJOjpSalAc= 2023-07-29T01:22:56.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:f0/VCNxcTcEocfUjNZJOjpSalAc='} 2023-07-29T01:22:56.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003b46accb4cdad56c-0064c462e1-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/ 2023-07-29T01:22:56.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/ 2023-07-29T01:22:56.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/ 2023-07-29T01:22:56.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.177 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.177 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.177 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.178 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:49 GMT 2023-07-29T01:22:56.178 INFO:tasks.rgw_multisite_tests:/hiiqos-63/ 2023-07-29T01:22:56.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.178 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:eMFPRv3AuzfRuxq+oty/0aQF6gY= 2023-07-29T01:22:56.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eMFPRv3AuzfRuxq+oty/0aQF6gY=', 'Content-Length': '0'} 2023-07-29T01:22:56.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c04ebbe360eace5c-0064c462e1-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/obj111 2023-07-29T01:22:56.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/obj111 2023-07-29T01:22:56.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/obj111 2023-07-29T01:22:56.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.182 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.182 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.182 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.182 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:49 GMT 2023-07-29T01:22:56.182 INFO:tasks.rgw_multisite_tests:/hiiqos-63/obj111 2023-07-29T01:22:56.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.183 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:crnTV5NRePifXYga8PMz5xtJ7YM= 2023-07-29T01:22:56.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:crnTV5NRePifXYga8PMz5xtJ7YM='} 2023-07-29T01:22:56.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e5ad286b7fc06f4c-0064c462e1-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/ 2023-07-29T01:22:56.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/ 2023-07-29T01:22:56.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/ 2023-07-29T01:22:56.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.186 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.186 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.187 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.187 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:49 GMT 2023-07-29T01:22:56.187 INFO:tasks.rgw_multisite_tests:/hiiqos-63/ 2023-07-29T01:22:56.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.188 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:eMFPRv3AuzfRuxq+oty/0aQF6gY= 2023-07-29T01:22:56.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eMFPRv3AuzfRuxq+oty/0aQF6gY=', 'Content-Length': '0'} 2023-07-29T01:22:56.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000213af5d046d52381-0064c462e1-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/obj148 2023-07-29T01:22:56.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/obj148 2023-07-29T01:22:56.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/obj148 2023-07-29T01:22:56.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.191 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.191 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.191 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.191 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:49 GMT 2023-07-29T01:22:56.192 INFO:tasks.rgw_multisite_tests:/hiiqos-63/obj148 2023-07-29T01:22:56.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.192 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:toELDn5wKsxk8j2y48UtOG7Jub8= 2023-07-29T01:22:56.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:toELDn5wKsxk8j2y48UtOG7Jub8='} 2023-07-29T01:22:56.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003efad09c8d416848-0064c462e1-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/ 2023-07-29T01:22:56.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/ 2023-07-29T01:22:56.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/ 2023-07-29T01:22:56.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.195 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.195 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.196 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.196 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:49 GMT 2023-07-29T01:22:56.196 INFO:tasks.rgw_multisite_tests:/hiiqos-63/ 2023-07-29T01:22:56.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.197 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:eMFPRv3AuzfRuxq+oty/0aQF6gY= 2023-07-29T01:22:56.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:52:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eMFPRv3AuzfRuxq+oty/0aQF6gY=', 'Content-Length': '0'} 2023-07-29T01:22:56.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c25d69c0fc21aca5-0064c462e1-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:52:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-63/obj185 2023-07-29T01:22:56.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-63/obj185 2023-07-29T01:22:56.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-63/obj185 2023-07-29T01:22:56.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.200 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.200 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.200 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.201 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:52:49 GMT 2023-07-29T01:22:56.201 INFO:tasks.rgw_multisite_tests:/hiiqos-63/obj185 2023-07-29T01:22:56.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.201 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:Oa45o+GYVxS7KdsPmI9YMsft/Fw= 2023-07-29T01:22:56.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:52:49 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Oa45o+GYVxS7KdsPmI9YMsft/Fw='} 2023-07-29T01:22:56.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bb419ee43549a8d1-0064c462e1-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:52:49 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.202 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=hiiqos-63 2023-07-29T01:22:56.202 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T01:22:56.202 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T01:22:56.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-63 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T01:22:56.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:56.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:56.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:52:49.350+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 1.. 2023-07-29T01:22:56.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:52:54.351+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 1.. 2023-07-29T01:22:56.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:52:59.352+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 1.. 2023-07-29T01:22:56.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:53:04.352+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 1.. 2023-07-29T01:22:56.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:53:09.353+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:56.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:53:14.353+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:56.205 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.1.smithi139.stdout: INFO: 2023-07-29T00:53:18.084+0000 7f5731a6a640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T01:22:56.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:53:19.355+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:56.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:53:24.355+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:56.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:53:29.356+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:56.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:53:34.357+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:56.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:53:39.358+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:56.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:53:44.358+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:56.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:53:49.358+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:56.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:53:54.359+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:56.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:53:59.361+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:56.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:54:04.361+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:56.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:54:09.362+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:56.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:54:14.363+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:56.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:54:19.363+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:56.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:54:24.364+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:56.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:54:29.364+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:56.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:54:34.365+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:56.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:54:39.366+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:56.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:54:44.366+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:56.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:54:49.367+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T01:22:56.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:54:54.368+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T01:22:56.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:54:59.368+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T01:22:56.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:55:04.369+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T01:22:56.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:55:09.370+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T01:22:56.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:55:14.370+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T01:22:56.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:55:19.371+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T01:22:56.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:55:24.372+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T01:22:56.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:55:29.372+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T01:22:56.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:55:34.373+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T01:22:56.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:55:39.374+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T01:22:56.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:55:44.375+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T01:22:56.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:55:49.375+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T01:22:56.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:55:54.376+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T01:22:56.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:55:59.376+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T01:22:56.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:56:04.377+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T01:22:56.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:56:09.378+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T01:22:56.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:56:14.378+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T01:22:56.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:56:19.379+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T01:22:56.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:56:24.380+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T01:22:56.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:56:29.380+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T01:22:56.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:56:34.381+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T01:22:56.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:56:39.381+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T01:22:56.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:56:44.382+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T01:22:56.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:56:49.382+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T01:22:56.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:56:54.383+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T01:22:56.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:56:59.384+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T01:22:56.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:57:04.385+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T01:22:56.216 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi101.stdout: INFO: 2023-07-29T00:57:07.780+0000 7f0081a5e640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T01:22:56.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:57:09.384+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T01:22:56.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:57:14.385+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T01:22:56.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:57:19.385+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T01:22:56.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:57:24.386+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T01:22:56.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:57:29.387+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T01:22:56.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:57:34.387+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T01:22:56.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:57:39.388+0000 7f3d7a326880 1 waiting to reach latest gen 6, on 3.. 2023-07-29T01:22:56.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:57:44.388+0000 7f3d7a326880 -1 bucket checkpoint timed out waiting to reach latest generation 6 2023-07-29T01:22:56.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:57:44.388+0000 7f3d7a326880 0 bucket sync checkpoint failed: (110) Connection timed out 2023-07-29T01:22:56.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:57:44.388+0000 7f3d7a326880 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-07-29T01:22:56.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2023-07-29T01:22:56.218 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-07-29T01:22:56.219 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.219 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-07-29T01:22:56.219 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2023-07-29T01:22:56.220 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-07-29T01:22:56.220 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-07-29T01:22:56.221 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_61140164d2579c30ffff3970615e0f82e40c6470/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-07-29T01:22:56.221 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-07-29T01:22:56.221 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_0484097b0572e96f42cf6402cbe0ac8dcb046577/qa/../src/test/rgw/rgw_multi/tests.py", line 1904, in test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2023-07-29T01:22:56.221 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2023-07-29T01:22:56.221 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_0484097b0572e96f42cf6402cbe0ac8dcb046577/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2023-07-29T01:22:56.222 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2023-07-29T01:22:56.222 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_0484097b0572e96f42cf6402cbe0ac8dcb046577/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2023-07-29T01:22:56.222 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2023-07-29T01:22:56.222 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_0484097b0572e96f42cf6402cbe0ac8dcb046577/qa/tasks/rgw_multisite.py", line 197, in admin 2023-07-29T01:22:56.223 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-07-29T01:22:56.223 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-07-29T01:22:56.223 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-07-29T01:22:56.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/ 2023-07-29T01:22:56.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-29T01:22:56.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.226 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.226 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.227 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.227 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:57:44 GMT 2023-07-29T01:22:56.227 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.228 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:aAONjYwPC56aXn0s+WXyn3/odc0= 2023-07-29T01:22:56.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:57:44 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:aAONjYwPC56aXn0s+WXyn3/odc0=', 'Content-Length': '0'} 2023-07-29T01:22:56.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000029debdf09595a743-0064c46408-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.228 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=hiiqos-64 2023-07-29T01:22:56.228 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:56.229 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:56.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:56.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:56.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:56.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T01:22:56.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: [ 2023-07-29T01:22:56.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:56.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:56.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "last_update": "2023-07-29T00:40:17.836402Z" 2023-07-29T01:22:56.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:56.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:56.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690592264.435095_1219.1", 2023-07-29T01:22:56.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "last_update": "2023-07-29T00:57:44.435095Z" 2023-07-29T01:22:56.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:56.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:56.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:56.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "last_update": "0.000000" 2023-07-29T01:22:56.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:56.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:56.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:56.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "last_update": "2023-07-29T00:40:12.811406Z" 2023-07-29T01:22:56.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:56.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:56.234 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1690591217.836402_294.1', 1: '1_1690592264.435095_1219.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:56.234 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-07-29T01:22:56.235 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:56.235 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:56.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:56.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:56.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:56.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:57:44.624+0000 7f2d19d22880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:56.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:57:44.624+0000 7f2d19d22880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:56.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:56.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:56.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:56.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:56.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:56.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:56.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:56.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:56.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:56.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:56.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:56.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:56.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:56.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:56.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:56.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 1, 2023-07-29T01:22:56.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:56.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:56.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:56.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:56.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:56.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:56.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:56.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:56.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:56.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690592264.435095_1219.1", 2023-07-29T01:22:56.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:56.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:56.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:56.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:57:44.435095Z", 2023-07-29T01:22:56.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:56.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:56.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:56.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:56.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:56.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:56.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:56.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:56.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:56.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:56.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:56.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "0.000000", 2023-07-29T01:22:56.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:56.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:56.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:56.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:56.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:56.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:56.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:56.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:56.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:56.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:56.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:56.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:56.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:56.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:56.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:56.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:56.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:56.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:56.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 1, 2023-07-29T01:22:56.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 1 2023-07-29T01:22:56.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:56.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T00:57:44Z" 2023-07-29T01:22:56.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:56.251 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-07-29T01:22:56.251 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:56.252 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:56.252 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:56.252 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:56.252 INFO:tasks.rgw_multisite_tests: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:56.253 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:56.253 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:56.253 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:56.253 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:56.253 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:56.254 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:56.254 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:56.254 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:56.254 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:56.255 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-07-29T01:22:56.255 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:56.255 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:56.255 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:56.256 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:56.256 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:56.256 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:56.256 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:56.256 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:56.257 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:56.257 INFO:tasks.rgw_multisite_tests: "marker": "1_1690592264.435095_1219.1", 2023-07-29T01:22:56.257 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:56.257 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:56.258 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:56.258 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:57:44.435095Z", 2023-07-29T01:22:56.258 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:56.258 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:56.259 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:56.259 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:56.259 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:56.259 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:56.259 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:56.260 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-07-29T01:22:56.260 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:56.260 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:56.260 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:56.261 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-07-29T01:22:56.261 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:56.261 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:56.261 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:56.262 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:56.262 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:56.262 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:56.262 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:56.262 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:56.263 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:56.263 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:56.263 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:56.263 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:56.264 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:56.264 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:56.264 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:56.264 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:56.265 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:56.265 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:56.265 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-07-29T01:22:56.265 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-07-29T01:22:56.265 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:56.266 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T00:57:44Z" 2023-07-29T01:22:56.266 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:56.266 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.266 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592264.435095_1219.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:57:44.435095Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:22:56.267 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592264.435095_1219.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:56.267 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592264.435095_1219.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:56.267 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-07-29T01:22:56.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/ 2023-07-29T01:22:56.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.270 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.270 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.270 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.271 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:57:44 GMT 2023-07-29T01:22:56.271 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.271 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:05l8NFP7NBoc+DCRvcZc/1nXjPo= 2023-07-29T01:22:56.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:57:44 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:05l8NFP7NBoc+DCRvcZc/1nXjPo=', 'Content-Length': '0'} 2023-07-29T01:22:56.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000019722d58c836027a-0064c46408-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj11 2023-07-29T01:22:56.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj11 2023-07-29T01:22:56.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj11 2023-07-29T01:22:56.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.275 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.275 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.275 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.275 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:57:44 GMT 2023-07-29T01:22:56.275 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj11 2023-07-29T01:22:56.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.276 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:LkEVLCue1hw8tgAOFlQ9dO/8HiE= 2023-07-29T01:22:56.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:57:44 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LkEVLCue1hw8tgAOFlQ9dO/8HiE='} 2023-07-29T01:22:56.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000029f829289ecc4d8d-0064c46408-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/ 2023-07-29T01:22:56.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.279 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.279 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.280 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.280 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:57:44 GMT 2023-07-29T01:22:56.280 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.281 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:05l8NFP7NBoc+DCRvcZc/1nXjPo= 2023-07-29T01:22:56.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:57:44 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:05l8NFP7NBoc+DCRvcZc/1nXjPo=', 'Content-Length': '0'} 2023-07-29T01:22:56.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000916a4b9e6d6e5804-0064c46408-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj22 2023-07-29T01:22:56.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj22 2023-07-29T01:22:56.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj22 2023-07-29T01:22:56.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.284 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.284 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.284 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.285 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:57:44 GMT 2023-07-29T01:22:56.285 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj22 2023-07-29T01:22:56.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.285 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:P7R97tGUjlKWyrRjUWlSPgjWi4I= 2023-07-29T01:22:56.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:57:44 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:P7R97tGUjlKWyrRjUWlSPgjWi4I='} 2023-07-29T01:22:56.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000db6936474333b8b7-0064c46408-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/ 2023-07-29T01:22:56.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.289 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.289 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.289 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.289 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:57:44 GMT 2023-07-29T01:22:56.289 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.290 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:05l8NFP7NBoc+DCRvcZc/1nXjPo= 2023-07-29T01:22:56.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:57:44 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:05l8NFP7NBoc+DCRvcZc/1nXjPo=', 'Content-Length': '0'} 2023-07-29T01:22:56.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005792cabcee2eb20e-0064c46408-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj33 2023-07-29T01:22:56.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj33 2023-07-29T01:22:56.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj33 2023-07-29T01:22:56.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.293 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.293 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.294 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.294 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:57:44 GMT 2023-07-29T01:22:56.294 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj33 2023-07-29T01:22:56.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.295 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:RYGPY/pIfWse/kRpS2al0y+B5aU= 2023-07-29T01:22:56.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:57:44 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:RYGPY/pIfWse/kRpS2al0y+B5aU='} 2023-07-29T01:22:56.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fa6e52bfd83ffe83-0064c46408-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/ 2023-07-29T01:22:56.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.298 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.298 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.298 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.298 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:57:44 GMT 2023-07-29T01:22:56.299 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.299 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:05l8NFP7NBoc+DCRvcZc/1nXjPo= 2023-07-29T01:22:56.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:57:44 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:05l8NFP7NBoc+DCRvcZc/1nXjPo=', 'Content-Length': '0'} 2023-07-29T01:22:56.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000a902a38d3476a50-0064c46408-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj44 2023-07-29T01:22:56.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj44 2023-07-29T01:22:56.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj44 2023-07-29T01:22:56.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.302 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.303 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.303 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.303 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:57:44 GMT 2023-07-29T01:22:56.303 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj44 2023-07-29T01:22:56.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.304 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:4NRrMFSAGpJi2iXzWBgjDa0QXlQ= 2023-07-29T01:22:56.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:57:44 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4NRrMFSAGpJi2iXzWBgjDa0QXlQ='} 2023-07-29T01:22:56.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006aff2b7e8106040f-0064c46408-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/ 2023-07-29T01:22:56.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.307 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.307 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.308 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.308 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:57:44 GMT 2023-07-29T01:22:56.308 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.308 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:05l8NFP7NBoc+DCRvcZc/1nXjPo= 2023-07-29T01:22:56.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:57:44 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:05l8NFP7NBoc+DCRvcZc/1nXjPo=', 'Content-Length': '0'} 2023-07-29T01:22:56.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a73445e514e31be4-0064c46408-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj55 2023-07-29T01:22:56.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj55 2023-07-29T01:22:56.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj55 2023-07-29T01:22:56.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.312 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.312 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.312 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.312 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:57:44 GMT 2023-07-29T01:22:56.313 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj55 2023-07-29T01:22:56.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.313 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:cJkwqsyowCL7EAURdLz82kAamV0= 2023-07-29T01:22:56.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:57:44 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cJkwqsyowCL7EAURdLz82kAamV0='} 2023-07-29T01:22:56.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000091524d199f40b679-0064c46408-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.314 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=hiiqos-64 2023-07-29T01:22:56.314 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T01:22:56.314 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T01:22:56.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-64 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T01:22:56.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:56.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:56.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:57:44.877+0000 7f78afb24880 1 waiting to reach incremental sync.. 2023-07-29T01:22:56.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:57:49.879+0000 7f78afb24880 1 bucket sync caught up with source: 2023-07-29T01:22:56.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: local status: [, , , , , , , , 00000000001.107.6, 00000000002.133.6, 00000000002.112.6] 2023-07-29T01:22:56.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: remote markers: [, , , , , , , , 00000000001.107.6, 00000000002.133.6, 00000000002.112.6] 2023-07-29T01:22:56.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:57:49.879+0000 7f78afb24880 0 bucket checkpoint complete 2023-07-29T01:22:56.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=hiiqos-64 2023-07-29T01:22:56.317 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T01:22:56.317 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T01:22:56.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-64 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T01:22:56.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:56.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:56.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:57:49.992+0000 7f0f61b29880 1 waiting for incremental sync to catch up: 2023-07-29T01:22:56.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: local status: [, , , , , , , , 00000000001.228.6, 00000000002.977.6, 00000000001.1073.6] 2023-07-29T01:22:56.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: remote markers: [, , , , , , , , 00000000001.228.6, 00000000002.977.6, 00000000002.1075.6] 2023-07-29T01:22:56.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:57:54.993+0000 7f0f61b29880 1 waiting for incremental sync to catch up: 2023-07-29T01:22:56.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: local status: [, , , , , , , , 00000000001.228.6, 00000000002.977.6, 00000000001.1073.6] 2023-07-29T01:22:56.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: remote markers: [, , , , , , , , 00000000001.228.6, 00000000002.977.6, 00000000002.1075.6] 2023-07-29T01:22:56.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:57:59.994+0000 7f0f61b29880 1 waiting for incremental sync to catch up: 2023-07-29T01:22:56.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: local status: [, , , , , , , , 00000000001.228.6, 00000000002.977.6, 00000000001.1073.6] 2023-07-29T01:22:56.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: remote markers: [, , , , , , , , 00000000001.228.6, 00000000002.977.6, 00000000002.1075.6] 2023-07-29T01:22:56.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:58:04.995+0000 7f0f61b29880 1 waiting for incremental sync to catch up: 2023-07-29T01:22:56.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: local status: [, , , , , , , , 00000000001.228.6, 00000000002.977.6, 00000000001.1073.6] 2023-07-29T01:22:56.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: remote markers: [, , , , , , , , 00000000001.228.6, 00000000002.977.6, 00000000002.1075.6] 2023-07-29T01:22:56.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:58:09.996+0000 7f0f61b29880 1 bucket sync caught up with source: 2023-07-29T01:22:56.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: local status: [, , , , , , , , 00000000001.228.6, 00000000002.977.6, 00000000002.1075.6] 2023-07-29T01:22:56.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: remote markers: [, , , , , , , , 00000000001.228.6, 00000000002.977.6, 00000000002.1075.6] 2023-07-29T01:22:56.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:58:09.996+0000 7f0f61b29880 0 bucket checkpoint complete 2023-07-29T01:22:56.321 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=hiiqos-64 zones={a2, a1} 2023-07-29T01:22:56.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/ 2023-07-29T01:22:56.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:56.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:56.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-29T01:22:56.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.324 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.325 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.325 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.325 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:10 GMT 2023-07-29T01:22:56.325 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.326 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:WKXNvM9Ev91xGB4i6WmyYRo9S6g= 2023-07-29T01:22:56.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:WKXNvM9Ev91xGB4i6WmyYRo9S6g=', 'Content-Length': '0'} 2023-07-29T01:22:56.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fef553293c7c1e27-0064c46427-2068-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/ 2023-07-29T01:22:56.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.329 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.329 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.329 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.330 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.330 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.330 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:CuuyMElb89TKlDMztYqnG3NGO5I= 2023-07-29T01:22:56.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:CuuyMElb89TKlDMztYqnG3NGO5I=', 'Content-Length': '0'} 2023-07-29T01:22:56.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000062b1d15005841e9a-0064c46427-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.331 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2023-07-29T01:22:56.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/?versions&max-keys=999 2023-07-29T01:22:56.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/?versions&max-keys=999 2023-07-29T01:22:56.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/?versions&max-keys=999 2023-07-29T01:22:56.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:56.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:56.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.334 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.335 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.335 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.335 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.335 INFO:tasks.rgw_multisite_tests:/hiiqos-64/?versions 2023-07-29T01:22:56.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.336 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:koEO66FOp5U9ET2vBpIwkWbwDXM= 2023-07-29T01:22:56.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:koEO66FOp5U9ET2vBpIwkWbwDXM=', 'Content-Length': '0'} 2023-07-29T01:22:56.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7e9490669d028cd-0064c46427-2068-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-64999falseobj11nulltrue2023-07-29T00:57:44.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:57:44.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:57:44.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:57:44.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:57:44.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:56.337 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj11 2023-07-29T01:22:56.337 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj22 2023-07-29T01:22:56.337 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj33 2023-07-29T01:22:56.337 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj44 2023-07-29T01:22:56.338 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj55 2023-07-29T01:22:56.338 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2023-07-29T01:22:56.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/?versions&max-keys=999 2023-07-29T01:22:56.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/?versions&max-keys=999 2023-07-29T01:22:56.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/?versions&max-keys=999 2023-07-29T01:22:56.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.341 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.341 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.341 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.342 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.342 INFO:tasks.rgw_multisite_tests:/hiiqos-64/?versions 2023-07-29T01:22:56.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.342 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:koEO66FOp5U9ET2vBpIwkWbwDXM= 2023-07-29T01:22:56.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:koEO66FOp5U9ET2vBpIwkWbwDXM=', 'Content-Length': '0'} 2023-07-29T01:22:56.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000967738b3007c629f-0064c46427-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-64999falseobj11nulltrue2023-07-29T00:57:44.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:57:44.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:57:44.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:57:44.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:57:44.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:56.343 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj11 2023-07-29T01:22:56.344 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj22 2023-07-29T01:22:56.344 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj33 2023-07-29T01:22:56.344 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj44 2023-07-29T01:22:56.344 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj55 2023-07-29T01:22:56.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/?versions&max-keys=999 2023-07-29T01:22:56.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/?versions&max-keys=999 2023-07-29T01:22:56.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/?versions&max-keys=999 2023-07-29T01:22:56.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:56.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:56.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.347 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.348 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.348 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.348 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.348 INFO:tasks.rgw_multisite_tests:/hiiqos-64/?versions 2023-07-29T01:22:56.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.349 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:koEO66FOp5U9ET2vBpIwkWbwDXM= 2023-07-29T01:22:56.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:koEO66FOp5U9ET2vBpIwkWbwDXM=', 'Content-Length': '0'} 2023-07-29T01:22:56.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013a772ea5e358d18-0064c46427-2068-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-64999falseobj11nulltrue2023-07-29T00:57:44.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:57:44.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:57:44.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:57:44.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:57:44.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:56.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/?versions&max-keys=999 2023-07-29T01:22:56.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/?versions&max-keys=999 2023-07-29T01:22:56.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/?versions&max-keys=999 2023-07-29T01:22:56.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.353 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.353 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.353 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.353 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.354 INFO:tasks.rgw_multisite_tests:/hiiqos-64/?versions 2023-07-29T01:22:56.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.354 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:koEO66FOp5U9ET2vBpIwkWbwDXM= 2023-07-29T01:22:56.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:koEO66FOp5U9ET2vBpIwkWbwDXM=', 'Content-Length': '0'} 2023-07-29T01:22:56.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b79c154ff222877-0064c46427-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-64999falseobj11nulltrue2023-07-29T00:57:44.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-07-29T00:57:44.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-07-29T00:57:44.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-07-29T00:57:44.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-07-29T00:57:44.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:56.355 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2023-07-29T01:22:56.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj11 2023-07-29T01:22:56.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj11 2023-07-29T01:22:56.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj11?versionId=null 2023-07-29T01:22:56.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj11?versionId=null 2023-07-29T01:22:56.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj11?versionId=null 2023-07-29T01:22:56.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:56.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:56.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.358 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.358 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.359 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.359 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.359 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj11?versionId=null 2023-07-29T01:22:56.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.360 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:jzVbZ/D7p63u2+qeaR0iTVLlhWw= 2023-07-29T01:22:56.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jzVbZ/D7p63u2+qeaR0iTVLlhWw=', 'Content-Length': '0'} 2023-07-29T01:22:56.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000311774593151e260-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj11 2023-07-29T01:22:56.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj11 2023-07-29T01:22:56.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj11?versionId=null 2023-07-29T01:22:56.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj11?versionId=null 2023-07-29T01:22:56.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj11?versionId=null 2023-07-29T01:22:56.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.363 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.363 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.364 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.364 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.364 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj11?versionId=null 2023-07-29T01:22:56.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.365 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:jzVbZ/D7p63u2+qeaR0iTVLlhWw= 2023-07-29T01:22:56.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jzVbZ/D7p63u2+qeaR0iTVLlhWw=', 'Content-Length': '0'} 2023-07-29T01:22:56.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000119027d3d80d52f2-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj11 2023-07-29T01:22:56.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj11 2023-07-29T01:22:56.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj11?versionId=null 2023-07-29T01:22:56.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj11?versionId=null 2023-07-29T01:22:56.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj11?versionId=null 2023-07-29T01:22:56.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:56.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:56.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.368 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.369 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.369 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.369 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.369 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj11?versionId=null 2023-07-29T01:22:56.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.370 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:Hgv99SpyrzO5a5yNRc77KqKYlgQ= 2023-07-29T01:22:56.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Hgv99SpyrzO5a5yNRc77KqKYlgQ=', 'Content-Length': '0'} 2023-07-29T01:22:56.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e91cbfa698655ae0-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj11 2023-07-29T01:22:56.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj11 2023-07-29T01:22:56.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj11?versionId=null 2023-07-29T01:22:56.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj11?versionId=null 2023-07-29T01:22:56.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj11?versionId=null 2023-07-29T01:22:56.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.373 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.374 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.374 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.374 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.374 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj11?versionId=null 2023-07-29T01:22:56.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.375 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:Hgv99SpyrzO5a5yNRc77KqKYlgQ= 2023-07-29T01:22:56.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Hgv99SpyrzO5a5yNRc77KqKYlgQ=', 'Content-Length': '0'} 2023-07-29T01:22:56.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a7b767975522116-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.376 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2023-07-29T01:22:56.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj11 2023-07-29T01:22:56.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj11 2023-07-29T01:22:56.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj11?versionId=null 2023-07-29T01:22:56.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj11?versionId=null 2023-07-29T01:22:56.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj11?versionId=null 2023-07-29T01:22:56.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:56.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:56.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.379 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.379 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.379 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.379 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.380 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj11?versionId=null 2023-07-29T01:22:56.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.380 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:jzVbZ/D7p63u2+qeaR0iTVLlhWw= 2023-07-29T01:22:56.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jzVbZ/D7p63u2+qeaR0iTVLlhWw=', 'Content-Length': '0'} 2023-07-29T01:22:56.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c823e23fb72942a2-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj11 2023-07-29T01:22:56.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj11 2023-07-29T01:22:56.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj11?versionId=null 2023-07-29T01:22:56.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj11?versionId=null 2023-07-29T01:22:56.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj11?versionId=null 2023-07-29T01:22:56.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.384 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.384 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.384 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.384 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.385 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj11?versionId=null 2023-07-29T01:22:56.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.385 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:jzVbZ/D7p63u2+qeaR0iTVLlhWw= 2023-07-29T01:22:56.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:jzVbZ/D7p63u2+qeaR0iTVLlhWw=', 'Content-Length': '0'} 2023-07-29T01:22:56.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047375a9069596eea-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/?prefix=obj11 2023-07-29T01:22:56.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/?prefix=obj11 2023-07-29T01:22:56.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/?prefix=obj11 2023-07-29T01:22:56.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:56.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:56.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.389 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.389 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.389 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.389 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.390 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.390 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU= 2023-07-29T01:22:56.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU=', 'Content-Length': '0'} 2023-07-29T01:22:56.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048871800185640fd-0064c46427-2068-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-64obj111000falseobj112023-07-29T00:57:44.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:56.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/?prefix=obj11 2023-07-29T01:22:56.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/?prefix=obj11 2023-07-29T01:22:56.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/?prefix=obj11 2023-07-29T01:22:56.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.394 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.394 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.395 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.395 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.395 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.395 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU= 2023-07-29T01:22:56.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU=', 'Content-Length': '0'} 2023-07-29T01:22:56.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000704f65ac17fc2329-0064c46427-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-64obj111000falseobj112023-07-29T00:57:44.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:56.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj11 2023-07-29T01:22:56.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj11 2023-07-29T01:22:56.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj11 2023-07-29T01:22:56.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:56.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:56.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.399 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.399 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.399 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.400 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.400 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj11 2023-07-29T01:22:56.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.400 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:xbcZQz3XaE81XFWG/f6GAO8hHLw= 2023-07-29T01:22:56.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xbcZQz3XaE81XFWG/f6GAO8hHLw=', 'Content-Length': '0'} 2023-07-29T01:22:56.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9d0ea91fe70462c-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj11 2023-07-29T01:22:56.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj11 2023-07-29T01:22:56.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj11 2023-07-29T01:22:56.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.403 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.404 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.404 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.404 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.404 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj11 2023-07-29T01:22:56.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.405 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:xbcZQz3XaE81XFWG/f6GAO8hHLw= 2023-07-29T01:22:56.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xbcZQz3XaE81XFWG/f6GAO8hHLw=', 'Content-Length': '0'} 2023-07-29T01:22:56.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000351cd01c98d37783-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.406 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2023-07-29T01:22:56.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj11 2023-07-29T01:22:56.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj11 2023-07-29T01:22:56.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj11 2023-07-29T01:22:56.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:56.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:56.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.408 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.409 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.409 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.409 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.409 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj11 2023-07-29T01:22:56.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.410 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:ZcdF4BmKXTP0r+kDurGZmjcQlQ4= 2023-07-29T01:22:56.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZcdF4BmKXTP0r+kDurGZmjcQlQ4=', 'Content-Length': '0'} 2023-07-29T01:22:56.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047d262a2eb4b3c62-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj11 2023-07-29T01:22:56.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj11 2023-07-29T01:22:56.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj11 2023-07-29T01:22:56.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.413 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.413 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.413 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.414 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.414 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj11 2023-07-29T01:22:56.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.414 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:ZcdF4BmKXTP0r+kDurGZmjcQlQ4= 2023-07-29T01:22:56.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ZcdF4BmKXTP0r+kDurGZmjcQlQ4=', 'Content-Length': '0'} 2023-07-29T01:22:56.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0e2b1652251bf6a-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/?prefix=obj11 2023-07-29T01:22:56.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/?prefix=obj11 2023-07-29T01:22:56.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/?prefix=obj11 2023-07-29T01:22:56.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:56.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:56.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.418 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.418 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.418 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.419 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.419 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.419 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU= 2023-07-29T01:22:56.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU=', 'Content-Length': '0'} 2023-07-29T01:22:56.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000097f987e89eb1b75-0064c46427-2068-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-64obj111000falseobj112023-07-29T00:57:44.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:56.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/?prefix=obj11 2023-07-29T01:22:56.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/?prefix=obj11 2023-07-29T01:22:56.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/?prefix=obj11 2023-07-29T01:22:56.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.423 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.423 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.424 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.424 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.424 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.425 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU= 2023-07-29T01:22:56.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU=', 'Content-Length': '0'} 2023-07-29T01:22:56.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5bc05c8bf53554c-0064c46427-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-64obj111000falseobj112023-07-29T00:57:44.671Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:56.426 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2023-07-29T01:22:56.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj22 2023-07-29T01:22:56.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj22 2023-07-29T01:22:56.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj22?versionId=null 2023-07-29T01:22:56.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj22?versionId=null 2023-07-29T01:22:56.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj22?versionId=null 2023-07-29T01:22:56.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:56.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:56.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.429 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.429 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.429 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.429 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.430 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj22?versionId=null 2023-07-29T01:22:56.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.430 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:eKjgE+uYwh346RSIkLIvDkS6txo= 2023-07-29T01:22:56.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eKjgE+uYwh346RSIkLIvDkS6txo=', 'Content-Length': '0'} 2023-07-29T01:22:56.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5f8c48de68042e4-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj22 2023-07-29T01:22:56.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj22 2023-07-29T01:22:56.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj22?versionId=null 2023-07-29T01:22:56.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj22?versionId=null 2023-07-29T01:22:56.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj22?versionId=null 2023-07-29T01:22:56.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.434 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.434 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.434 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.435 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.435 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj22?versionId=null 2023-07-29T01:22:56.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.435 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:eKjgE+uYwh346RSIkLIvDkS6txo= 2023-07-29T01:22:56.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eKjgE+uYwh346RSIkLIvDkS6txo=', 'Content-Length': '0'} 2023-07-29T01:22:56.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006150b1776c24dd75-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj22 2023-07-29T01:22:56.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj22 2023-07-29T01:22:56.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj22?versionId=null 2023-07-29T01:22:56.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj22?versionId=null 2023-07-29T01:22:56.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj22?versionId=null 2023-07-29T01:22:56.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:56.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:56.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.439 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.439 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.439 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.440 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.440 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj22?versionId=null 2023-07-29T01:22:56.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.440 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:ym1jHBlbq73Lp2t+iw2fZs6r758= 2023-07-29T01:22:56.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ym1jHBlbq73Lp2t+iw2fZs6r758=', 'Content-Length': '0'} 2023-07-29T01:22:56.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a48c3948d1db5e6d-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj22 2023-07-29T01:22:56.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj22 2023-07-29T01:22:56.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj22?versionId=null 2023-07-29T01:22:56.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj22?versionId=null 2023-07-29T01:22:56.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj22?versionId=null 2023-07-29T01:22:56.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.444 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.444 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.444 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.445 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.445 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj22?versionId=null 2023-07-29T01:22:56.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.445 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:ym1jHBlbq73Lp2t+iw2fZs6r758= 2023-07-29T01:22:56.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ym1jHBlbq73Lp2t+iw2fZs6r758=', 'Content-Length': '0'} 2023-07-29T01:22:56.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4149a6cd8fdd489-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.446 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2023-07-29T01:22:56.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj22 2023-07-29T01:22:56.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj22 2023-07-29T01:22:56.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj22?versionId=null 2023-07-29T01:22:56.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj22?versionId=null 2023-07-29T01:22:56.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj22?versionId=null 2023-07-29T01:22:56.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:56.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:56.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.449 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.449 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.450 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.450 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.450 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj22?versionId=null 2023-07-29T01:22:56.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.451 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:eKjgE+uYwh346RSIkLIvDkS6txo= 2023-07-29T01:22:56.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eKjgE+uYwh346RSIkLIvDkS6txo=', 'Content-Length': '0'} 2023-07-29T01:22:56.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2e10562bad0b0d3-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj22 2023-07-29T01:22:56.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj22 2023-07-29T01:22:56.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj22?versionId=null 2023-07-29T01:22:56.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj22?versionId=null 2023-07-29T01:22:56.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj22?versionId=null 2023-07-29T01:22:56.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.454 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.455 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.455 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.455 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.455 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj22?versionId=null 2023-07-29T01:22:56.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.456 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:eKjgE+uYwh346RSIkLIvDkS6txo= 2023-07-29T01:22:56.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eKjgE+uYwh346RSIkLIvDkS6txo=', 'Content-Length': '0'} 2023-07-29T01:22:56.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056948ca90aec8341-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/?prefix=obj22 2023-07-29T01:22:56.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/?prefix=obj22 2023-07-29T01:22:56.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/?prefix=obj22 2023-07-29T01:22:56.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:56.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:56.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.460 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.460 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.460 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.460 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.460 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.461 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU= 2023-07-29T01:22:56.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU=', 'Content-Length': '0'} 2023-07-29T01:22:56.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d4a8463394aa5f6-0064c46427-2068-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-64obj221000falseobj222023-07-29T00:57:44.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:56.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/?prefix=obj22 2023-07-29T01:22:56.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/?prefix=obj22 2023-07-29T01:22:56.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/?prefix=obj22 2023-07-29T01:22:56.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.465 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.465 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.465 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.466 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.466 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.466 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU= 2023-07-29T01:22:56.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU=', 'Content-Length': '0'} 2023-07-29T01:22:56.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029897b22dbc12f05-0064c46427-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-64obj221000falseobj222023-07-29T00:57:44.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:56.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj22 2023-07-29T01:22:56.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj22 2023-07-29T01:22:56.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj22 2023-07-29T01:22:56.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:56.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:56.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.470 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.470 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.470 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.471 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.471 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj22 2023-07-29T01:22:56.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.471 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:9oEB6tVZd9hKASyKVyRGrr8bEVg= 2023-07-29T01:22:56.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9oEB6tVZd9hKASyKVyRGrr8bEVg=', 'Content-Length': '0'} 2023-07-29T01:22:56.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054d229d9902bc2f8-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj22 2023-07-29T01:22:56.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj22 2023-07-29T01:22:56.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj22 2023-07-29T01:22:56.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.474 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.475 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.475 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.475 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.475 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj22 2023-07-29T01:22:56.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.476 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:9oEB6tVZd9hKASyKVyRGrr8bEVg= 2023-07-29T01:22:56.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9oEB6tVZd9hKASyKVyRGrr8bEVg=', 'Content-Length': '0'} 2023-07-29T01:22:56.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b65c7d1b0ac3c30c-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.477 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2023-07-29T01:22:56.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj22 2023-07-29T01:22:56.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj22 2023-07-29T01:22:56.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj22 2023-07-29T01:22:56.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:56.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:56.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.479 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.479 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.480 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.480 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.480 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj22 2023-07-29T01:22:56.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.481 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:de4C/37PRLn6jRnBQQ8A/Kn77rE= 2023-07-29T01:22:56.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:de4C/37PRLn6jRnBQQ8A/Kn77rE=', 'Content-Length': '0'} 2023-07-29T01:22:56.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebfb7102e186f80f-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj22 2023-07-29T01:22:56.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj22 2023-07-29T01:22:56.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj22 2023-07-29T01:22:56.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.484 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.484 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.484 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.485 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.485 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj22 2023-07-29T01:22:56.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.485 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:de4C/37PRLn6jRnBQQ8A/Kn77rE= 2023-07-29T01:22:56.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:de4C/37PRLn6jRnBQQ8A/Kn77rE=', 'Content-Length': '0'} 2023-07-29T01:22:56.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b99c45709eda0e2-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/?prefix=obj22 2023-07-29T01:22:56.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/?prefix=obj22 2023-07-29T01:22:56.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/?prefix=obj22 2023-07-29T01:22:56.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:56.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:56.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.489 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.489 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.490 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.490 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.490 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.490 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU= 2023-07-29T01:22:56.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU=', 'Content-Length': '0'} 2023-07-29T01:22:56.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053fe0d05c81b327c-0064c46427-2068-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-64obj221000falseobj222023-07-29T00:57:44.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:56.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/?prefix=obj22 2023-07-29T01:22:56.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/?prefix=obj22 2023-07-29T01:22:56.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/?prefix=obj22 2023-07-29T01:22:56.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.494 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.495 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.495 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.495 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.495 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.496 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU= 2023-07-29T01:22:56.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU=', 'Content-Length': '0'} 2023-07-29T01:22:56.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d28115dbe7834e01-0064c46427-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-64obj221000falseobj222023-07-29T00:57:44.690Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:56.497 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2023-07-29T01:22:56.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj33 2023-07-29T01:22:56.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj33 2023-07-29T01:22:56.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj33?versionId=null 2023-07-29T01:22:56.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj33?versionId=null 2023-07-29T01:22:56.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj33?versionId=null 2023-07-29T01:22:56.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:56.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:56.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.500 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.500 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.500 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.501 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.501 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj33?versionId=null 2023-07-29T01:22:56.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.501 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:98vId0VCVxPhdyj1T0JLJHIQvgc= 2023-07-29T01:22:56.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:98vId0VCVxPhdyj1T0JLJHIQvgc=', 'Content-Length': '0'} 2023-07-29T01:22:56.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1f2e34aba87d866-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj33 2023-07-29T01:22:56.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj33 2023-07-29T01:22:56.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj33?versionId=null 2023-07-29T01:22:56.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj33?versionId=null 2023-07-29T01:22:56.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj33?versionId=null 2023-07-29T01:22:56.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.505 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.505 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.506 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.506 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.506 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj33?versionId=null 2023-07-29T01:22:56.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.506 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:98vId0VCVxPhdyj1T0JLJHIQvgc= 2023-07-29T01:22:56.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:98vId0VCVxPhdyj1T0JLJHIQvgc=', 'Content-Length': '0'} 2023-07-29T01:22:56.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081cc019de9ce60b4-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj33 2023-07-29T01:22:56.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj33 2023-07-29T01:22:56.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj33?versionId=null 2023-07-29T01:22:56.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj33?versionId=null 2023-07-29T01:22:56.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj33?versionId=null 2023-07-29T01:22:56.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:56.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:56.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.510 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.510 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.511 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.511 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.511 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj33?versionId=null 2023-07-29T01:22:56.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.512 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:rKXpeKBN4BlFWZ/78Y+K2c1eyT0= 2023-07-29T01:22:56.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rKXpeKBN4BlFWZ/78Y+K2c1eyT0=', 'Content-Length': '0'} 2023-07-29T01:22:56.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c26dcd1663b2e247-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj33 2023-07-29T01:22:56.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj33 2023-07-29T01:22:56.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj33?versionId=null 2023-07-29T01:22:56.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj33?versionId=null 2023-07-29T01:22:56.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj33?versionId=null 2023-07-29T01:22:56.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.515 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.515 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.516 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.516 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.516 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj33?versionId=null 2023-07-29T01:22:56.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.517 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:rKXpeKBN4BlFWZ/78Y+K2c1eyT0= 2023-07-29T01:22:56.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rKXpeKBN4BlFWZ/78Y+K2c1eyT0=', 'Content-Length': '0'} 2023-07-29T01:22:56.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec75d978a85d4b03-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.517 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2023-07-29T01:22:56.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj33 2023-07-29T01:22:56.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj33 2023-07-29T01:22:56.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj33?versionId=null 2023-07-29T01:22:56.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj33?versionId=null 2023-07-29T01:22:56.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj33?versionId=null 2023-07-29T01:22:56.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:56.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:56.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.521 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.521 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.521 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.521 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.522 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj33?versionId=null 2023-07-29T01:22:56.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.522 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:98vId0VCVxPhdyj1T0JLJHIQvgc= 2023-07-29T01:22:56.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:98vId0VCVxPhdyj1T0JLJHIQvgc=', 'Content-Length': '0'} 2023-07-29T01:22:56.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006532b30c32489583-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj33 2023-07-29T01:22:56.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj33 2023-07-29T01:22:56.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj33?versionId=null 2023-07-29T01:22:56.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj33?versionId=null 2023-07-29T01:22:56.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj33?versionId=null 2023-07-29T01:22:56.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.526 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.526 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.526 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.526 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.527 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj33?versionId=null 2023-07-29T01:22:56.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.527 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:98vId0VCVxPhdyj1T0JLJHIQvgc= 2023-07-29T01:22:56.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:98vId0VCVxPhdyj1T0JLJHIQvgc=', 'Content-Length': '0'} 2023-07-29T01:22:56.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000329de7456efee6c-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/?prefix=obj33 2023-07-29T01:22:56.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/?prefix=obj33 2023-07-29T01:22:56.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/?prefix=obj33 2023-07-29T01:22:56.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:56.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:56.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.531 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.531 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.531 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.532 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.532 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.532 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU= 2023-07-29T01:22:56.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU=', 'Content-Length': '0'} 2023-07-29T01:22:56.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004dec7ea68b2d2627-0064c46427-2068-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-64obj331000falseobj332023-07-29T00:57:44.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:56.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/?prefix=obj33 2023-07-29T01:22:56.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/?prefix=obj33 2023-07-29T01:22:56.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/?prefix=obj33 2023-07-29T01:22:56.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.536 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.536 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.537 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.537 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.537 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.538 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU= 2023-07-29T01:22:56.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU=', 'Content-Length': '0'} 2023-07-29T01:22:56.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098b9fce2d22daad1-0064c46427-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-64obj331000falseobj332023-07-29T00:57:44.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:56.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj33 2023-07-29T01:22:56.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj33 2023-07-29T01:22:56.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj33 2023-07-29T01:22:56.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:56.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:56.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.541 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.541 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.542 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.542 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.542 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj33 2023-07-29T01:22:56.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.542 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:Giw8a+QOERvRUc02958X1eZ2PX4= 2023-07-29T01:22:56.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Giw8a+QOERvRUc02958X1eZ2PX4=', 'Content-Length': '0'} 2023-07-29T01:22:56.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd7ee6e536d8fe67-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj33 2023-07-29T01:22:56.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj33 2023-07-29T01:22:56.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj33 2023-07-29T01:22:56.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.546 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.546 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.546 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.546 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.547 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj33 2023-07-29T01:22:56.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.547 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:Giw8a+QOERvRUc02958X1eZ2PX4= 2023-07-29T01:22:56.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Giw8a+QOERvRUc02958X1eZ2PX4=', 'Content-Length': '0'} 2023-07-29T01:22:56.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009a3748a6de2db12-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.548 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2023-07-29T01:22:56.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj33 2023-07-29T01:22:56.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj33 2023-07-29T01:22:56.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj33 2023-07-29T01:22:56.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:56.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:56.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.551 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.551 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.551 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.551 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.551 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj33 2023-07-29T01:22:56.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.552 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:Q1P0LI9D6/vNWj/Q0w/jRhPBzTo= 2023-07-29T01:22:56.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q1P0LI9D6/vNWj/Q0w/jRhPBzTo=', 'Content-Length': '0'} 2023-07-29T01:22:56.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d871261a775e374-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj33 2023-07-29T01:22:56.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj33 2023-07-29T01:22:56.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj33 2023-07-29T01:22:56.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.555 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.555 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.556 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.556 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.556 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj33 2023-07-29T01:22:56.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.557 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:Q1P0LI9D6/vNWj/Q0w/jRhPBzTo= 2023-07-29T01:22:56.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Q1P0LI9D6/vNWj/Q0w/jRhPBzTo=', 'Content-Length': '0'} 2023-07-29T01:22:56.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008aa6b863648e9106-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/?prefix=obj33 2023-07-29T01:22:56.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/?prefix=obj33 2023-07-29T01:22:56.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/?prefix=obj33 2023-07-29T01:22:56.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:56.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:56.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.560 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.561 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.561 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.561 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.561 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.562 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU= 2023-07-29T01:22:56.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU=', 'Content-Length': '0'} 2023-07-29T01:22:56.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c185c4a3644d971b-0064c46427-2068-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-64obj331000falseobj332023-07-29T00:57:44.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:56.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/?prefix=obj33 2023-07-29T01:22:56.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/?prefix=obj33 2023-07-29T01:22:56.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/?prefix=obj33 2023-07-29T01:22:56.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.566 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.566 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.566 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.566 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.567 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.567 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU= 2023-07-29T01:22:56.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU=', 'Content-Length': '0'} 2023-07-29T01:22:56.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6346cbb7ecf2f60-0064c46427-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-64obj331000falseobj332023-07-29T00:57:44.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:56.568 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2023-07-29T01:22:56.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj44 2023-07-29T01:22:56.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj44 2023-07-29T01:22:56.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj44?versionId=null 2023-07-29T01:22:56.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj44?versionId=null 2023-07-29T01:22:56.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj44?versionId=null 2023-07-29T01:22:56.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:56.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:56.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.571 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.571 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.572 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.572 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.572 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj44?versionId=null 2023-07-29T01:22:56.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.573 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:eZlcpkHP4tYuMnSlNLK1W6c7aGw= 2023-07-29T01:22:56.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eZlcpkHP4tYuMnSlNLK1W6c7aGw=', 'Content-Length': '0'} 2023-07-29T01:22:56.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee342e170eaede1e-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj44 2023-07-29T01:22:56.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj44 2023-07-29T01:22:56.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj44?versionId=null 2023-07-29T01:22:56.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj44?versionId=null 2023-07-29T01:22:56.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj44?versionId=null 2023-07-29T01:22:56.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.576 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.577 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.577 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.577 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.577 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj44?versionId=null 2023-07-29T01:22:56.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.578 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:eZlcpkHP4tYuMnSlNLK1W6c7aGw= 2023-07-29T01:22:56.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eZlcpkHP4tYuMnSlNLK1W6c7aGw=', 'Content-Length': '0'} 2023-07-29T01:22:56.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d515e633cd249217-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj44 2023-07-29T01:22:56.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj44 2023-07-29T01:22:56.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj44?versionId=null 2023-07-29T01:22:56.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj44?versionId=null 2023-07-29T01:22:56.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj44?versionId=null 2023-07-29T01:22:56.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:56.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:56.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.581 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.582 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.582 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.582 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.582 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj44?versionId=null 2023-07-29T01:22:56.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.583 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:joK7DBT9QGpmoeowI/S8mICmzd0= 2023-07-29T01:22:56.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:joK7DBT9QGpmoeowI/S8mICmzd0=', 'Content-Length': '0'} 2023-07-29T01:22:56.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b55f4e9232c47514-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj44 2023-07-29T01:22:56.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj44 2023-07-29T01:22:56.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj44?versionId=null 2023-07-29T01:22:56.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj44?versionId=null 2023-07-29T01:22:56.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj44?versionId=null 2023-07-29T01:22:56.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.586 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.587 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.587 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.587 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.587 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj44?versionId=null 2023-07-29T01:22:56.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.588 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:joK7DBT9QGpmoeowI/S8mICmzd0= 2023-07-29T01:22:56.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:joK7DBT9QGpmoeowI/S8mICmzd0=', 'Content-Length': '0'} 2023-07-29T01:22:56.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052779aa98ab8ccae-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.589 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2023-07-29T01:22:56.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj44 2023-07-29T01:22:56.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj44 2023-07-29T01:22:56.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj44?versionId=null 2023-07-29T01:22:56.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj44?versionId=null 2023-07-29T01:22:56.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj44?versionId=null 2023-07-29T01:22:56.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:56.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:56.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.592 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.592 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.592 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.592 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.593 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj44?versionId=null 2023-07-29T01:22:56.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.593 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:eZlcpkHP4tYuMnSlNLK1W6c7aGw= 2023-07-29T01:22:56.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eZlcpkHP4tYuMnSlNLK1W6c7aGw=', 'Content-Length': '0'} 2023-07-29T01:22:56.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a63a8421fea0e09-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj44 2023-07-29T01:22:56.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj44 2023-07-29T01:22:56.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj44?versionId=null 2023-07-29T01:22:56.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj44?versionId=null 2023-07-29T01:22:56.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj44?versionId=null 2023-07-29T01:22:56.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.597 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.597 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.597 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.598 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.598 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj44?versionId=null 2023-07-29T01:22:56.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.598 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:eZlcpkHP4tYuMnSlNLK1W6c7aGw= 2023-07-29T01:22:56.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:eZlcpkHP4tYuMnSlNLK1W6c7aGw=', 'Content-Length': '0'} 2023-07-29T01:22:56.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f86644e622e92c5-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/?prefix=obj44 2023-07-29T01:22:56.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/?prefix=obj44 2023-07-29T01:22:56.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/?prefix=obj44 2023-07-29T01:22:56.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:56.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:56.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.602 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.602 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.602 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.603 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.603 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.603 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU= 2023-07-29T01:22:56.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU=', 'Content-Length': '0'} 2023-07-29T01:22:56.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002eabddfdf047d897-0064c46427-2068-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-64obj441000falseobj442023-07-29T00:57:44.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:56.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/?prefix=obj44 2023-07-29T01:22:56.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/?prefix=obj44 2023-07-29T01:22:56.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/?prefix=obj44 2023-07-29T01:22:56.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.607 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.607 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.608 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.608 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.608 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.609 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU= 2023-07-29T01:22:56.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU=', 'Content-Length': '0'} 2023-07-29T01:22:56.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ccb7eb5f4aa21cb-0064c46427-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-64obj441000falseobj442023-07-29T00:57:44.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:56.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj44 2023-07-29T01:22:56.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj44 2023-07-29T01:22:56.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj44 2023-07-29T01:22:56.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:56.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:56.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.612 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.612 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.613 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.613 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.613 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj44 2023-07-29T01:22:56.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.614 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:bhtA/bVfmGvDybE6jG9HOdc8ckU= 2023-07-29T01:22:56.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bhtA/bVfmGvDybE6jG9HOdc8ckU=', 'Content-Length': '0'} 2023-07-29T01:22:56.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cc5eebc5a62399f-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj44 2023-07-29T01:22:56.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj44 2023-07-29T01:22:56.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj44 2023-07-29T01:22:56.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.617 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.617 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.617 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.617 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.618 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj44 2023-07-29T01:22:56.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.618 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:bhtA/bVfmGvDybE6jG9HOdc8ckU= 2023-07-29T01:22:56.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bhtA/bVfmGvDybE6jG9HOdc8ckU=', 'Content-Length': '0'} 2023-07-29T01:22:56.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd0e1fdc6066e0f1-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.619 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2023-07-29T01:22:56.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj44 2023-07-29T01:22:56.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj44 2023-07-29T01:22:56.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj44 2023-07-29T01:22:56.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:56.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:56.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.622 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.622 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.622 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.622 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.623 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj44 2023-07-29T01:22:56.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.623 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:4rm99mIbKH5BAR7UyBF8agtr4zQ= 2023-07-29T01:22:56.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4rm99mIbKH5BAR7UyBF8agtr4zQ=', 'Content-Length': '0'} 2023-07-29T01:22:56.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b5319676c06b2b0-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj44 2023-07-29T01:22:56.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj44 2023-07-29T01:22:56.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj44 2023-07-29T01:22:56.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.626 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.626 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.627 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.627 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.627 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj44 2023-07-29T01:22:56.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.628 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:4rm99mIbKH5BAR7UyBF8agtr4zQ= 2023-07-29T01:22:56.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4rm99mIbKH5BAR7UyBF8agtr4zQ=', 'Content-Length': '0'} 2023-07-29T01:22:56.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001beddfddcc51ae78-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/?prefix=obj44 2023-07-29T01:22:56.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/?prefix=obj44 2023-07-29T01:22:56.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/?prefix=obj44 2023-07-29T01:22:56.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:56.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:56.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.631 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.631 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.632 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.632 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.632 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.633 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU= 2023-07-29T01:22:56.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU=', 'Content-Length': '0'} 2023-07-29T01:22:56.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbfaeff3bf570d22-0064c46427-2068-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-64obj441000falseobj442023-07-29T00:57:44.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:56.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/?prefix=obj44 2023-07-29T01:22:56.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/?prefix=obj44 2023-07-29T01:22:56.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/?prefix=obj44 2023-07-29T01:22:56.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.637 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.637 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.637 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.637 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.638 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.638 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU= 2023-07-29T01:22:56.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:J6pglBP2+hxciN0DAqqKBAdWKFU=', 'Content-Length': '0'} 2023-07-29T01:22:56.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f29e8a5742a51f4-0064c46427-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-64obj441000falseobj442023-07-29T00:57:44.769Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:56.639 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2023-07-29T01:22:56.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj55 2023-07-29T01:22:56.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj55 2023-07-29T01:22:56.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj55?versionId=null 2023-07-29T01:22:56.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj55?versionId=null 2023-07-29T01:22:56.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj55?versionId=null 2023-07-29T01:22:56.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:56.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:56.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.642 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.642 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.643 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.643 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.643 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj55?versionId=null 2023-07-29T01:22:56.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.644 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:Rv51aK3Fw8oxOxbfoDeK2A00grA= 2023-07-29T01:22:56.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Rv51aK3Fw8oxOxbfoDeK2A00grA=', 'Content-Length': '0'} 2023-07-29T01:22:56.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e15294ade3f77be8-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj55 2023-07-29T01:22:56.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj55 2023-07-29T01:22:56.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj55?versionId=null 2023-07-29T01:22:56.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj55?versionId=null 2023-07-29T01:22:56.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj55?versionId=null 2023-07-29T01:22:56.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.647 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.648 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.648 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.648 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.648 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj55?versionId=null 2023-07-29T01:22:56.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.649 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:Rv51aK3Fw8oxOxbfoDeK2A00grA= 2023-07-29T01:22:56.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Rv51aK3Fw8oxOxbfoDeK2A00grA=', 'Content-Length': '0'} 2023-07-29T01:22:56.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a64cf36af2d0d5b3-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj55 2023-07-29T01:22:56.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj55 2023-07-29T01:22:56.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj55?versionId=null 2023-07-29T01:22:56.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj55?versionId=null 2023-07-29T01:22:56.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj55?versionId=null 2023-07-29T01:22:56.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:56.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:56.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.652 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.653 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.653 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.653 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:15 GMT 2023-07-29T01:22:56.653 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj55?versionId=null 2023-07-29T01:22:56.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.654 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:labkhHXzNZFlaEsk2bPbC78lbk0= 2023-07-29T01:22:56.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:15 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:labkhHXzNZFlaEsk2bPbC78lbk0=', 'Content-Length': '0'} 2023-07-29T01:22:56.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e60fca35950d9215-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj55 2023-07-29T01:22:56.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj55 2023-07-29T01:22:56.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj55?versionId=null 2023-07-29T01:22:56.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj55?versionId=null 2023-07-29T01:22:56.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj55?versionId=null 2023-07-29T01:22:56.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.657 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.658 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.658 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.658 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:16 GMT 2023-07-29T01:22:56.658 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj55?versionId=null 2023-07-29T01:22:56.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.659 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:b8iBa8YetOPg6zrV7fvouc7nXkA= 2023-07-29T01:22:56.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:b8iBa8YetOPg6zrV7fvouc7nXkA=', 'Content-Length': '0'} 2023-07-29T01:22:56.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000271eb0a08341db51-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.660 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2023-07-29T01:22:56.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj55 2023-07-29T01:22:56.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj55 2023-07-29T01:22:56.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj55?versionId=null 2023-07-29T01:22:56.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj55?versionId=null 2023-07-29T01:22:56.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj55?versionId=null 2023-07-29T01:22:56.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:56.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:56.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.663 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.663 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.663 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.663 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:16 GMT 2023-07-29T01:22:56.664 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj55?versionId=null 2023-07-29T01:22:56.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.664 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:w9m+PMoRaOk3/jG4Z85BxF6kmiA= 2023-07-29T01:22:56.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w9m+PMoRaOk3/jG4Z85BxF6kmiA=', 'Content-Length': '0'} 2023-07-29T01:22:56.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf78df1a331ddec0-0064c46427-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj55 2023-07-29T01:22:56.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj55 2023-07-29T01:22:56.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj55?versionId=null 2023-07-29T01:22:56.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj55?versionId=null 2023-07-29T01:22:56.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj55?versionId=null 2023-07-29T01:22:56.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.668 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.668 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.668 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.669 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:16 GMT 2023-07-29T01:22:56.669 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj55?versionId=null 2023-07-29T01:22:56.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.669 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:w9m+PMoRaOk3/jG4Z85BxF6kmiA= 2023-07-29T01:22:56.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:w9m+PMoRaOk3/jG4Z85BxF6kmiA=', 'Content-Length': '0'} 2023-07-29T01:22:56.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028145327fc6e2df8-0064c46427-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/?prefix=obj55 2023-07-29T01:22:56.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/?prefix=obj55 2023-07-29T01:22:56.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/?prefix=obj55 2023-07-29T01:22:56.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:56.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:56.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.673 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.673 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.673 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.674 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:16 GMT 2023-07-29T01:22:56.674 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.674 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:rHVKG8gNW6ZA3ljBfJt53iz2WJc= 2023-07-29T01:22:56.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rHVKG8gNW6ZA3ljBfJt53iz2WJc=', 'Content-Length': '0'} 2023-07-29T01:22:56.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058961bd4eba36c08-0064c46427-2068-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:15 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-64obj551000falseobj552023-07-29T00:57:44.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:56.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/?prefix=obj55 2023-07-29T01:22:56.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/?prefix=obj55 2023-07-29T01:22:56.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/?prefix=obj55 2023-07-29T01:22:56.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.678 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.679 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.679 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.679 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:16 GMT 2023-07-29T01:22:56.679 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.680 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:rHVKG8gNW6ZA3ljBfJt53iz2WJc= 2023-07-29T01:22:56.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rHVKG8gNW6ZA3ljBfJt53iz2WJc=', 'Content-Length': '0'} 2023-07-29T01:22:56.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b361a1d2d021998-0064c46428-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-64obj551000falseobj552023-07-29T00:57:44.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:56.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj55 2023-07-29T01:22:56.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj55 2023-07-29T01:22:56.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj55 2023-07-29T01:22:56.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:56.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:56.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.683 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.683 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.684 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.684 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:16 GMT 2023-07-29T01:22:56.684 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj55 2023-07-29T01:22:56.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.685 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:fZ0+QhfGouvlodXzkysnYK9YlYI= 2023-07-29T01:22:56.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fZ0+QhfGouvlodXzkysnYK9YlYI=', 'Content-Length': '0'} 2023-07-29T01:22:56.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000449179b8c357e5c2-0064c46428-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj55 2023-07-29T01:22:56.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj55 2023-07-29T01:22:56.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj55 2023-07-29T01:22:56.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.688 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.688 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.688 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.688 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:16 GMT 2023-07-29T01:22:56.689 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj55 2023-07-29T01:22:56.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.689 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:fZ0+QhfGouvlodXzkysnYK9YlYI= 2023-07-29T01:22:56.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fZ0+QhfGouvlodXzkysnYK9YlYI=', 'Content-Length': '0'} 2023-07-29T01:22:56.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b07df3e8f83b7e93-0064c46428-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.690 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2023-07-29T01:22:56.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj55 2023-07-29T01:22:56.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj55 2023-07-29T01:22:56.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj55 2023-07-29T01:22:56.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:56.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:56.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.693 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.693 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.693 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.693 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:16 GMT 2023-07-29T01:22:56.694 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj55 2023-07-29T01:22:56.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.694 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:pbRFhGVXA+fSYmiWAXLRX7x3Te4= 2023-07-29T01:22:56.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pbRFhGVXA+fSYmiWAXLRX7x3Te4=', 'Content-Length': '0'} 2023-07-29T01:22:56.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-64:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.5729.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000787a3d4960cec3-0064c46428-2068-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj55 2023-07-29T01:22:56.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj55 2023-07-29T01:22:56.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj55 2023-07-29T01:22:56.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.697 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.698 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.698 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.698 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:16 GMT 2023-07-29T01:22:56.698 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj55 2023-07-29T01:22:56.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.699 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:pbRFhGVXA+fSYmiWAXLRX7x3Te4= 2023-07-29T01:22:56.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:pbRFhGVXA+fSYmiWAXLRX7x3Te4=', 'Content-Length': '0'} 2023-07-29T01:22:56.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 00:57:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e97a594b1912f24d-0064c46428-165f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 00:58:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/?prefix=obj55 2023-07-29T01:22:56.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/?prefix=obj55 2023-07-29T01:22:56.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/?prefix=obj55 2023-07-29T01:22:56.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:56.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:56.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.702 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.703 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.703 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.703 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:16 GMT 2023-07-29T01:22:56.703 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.704 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:rHVKG8gNW6ZA3ljBfJt53iz2WJc= 2023-07-29T01:22:56.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rHVKG8gNW6ZA3ljBfJt53iz2WJc=', 'Content-Length': '0'} 2023-07-29T01:22:56.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a014e2f8f633424-0064c46428-2068-a2'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-64obj551000falseobj552023-07-29T00:57:44.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:56.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/?prefix=obj55 2023-07-29T01:22:56.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/?prefix=obj55 2023-07-29T01:22:56.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:56.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/?prefix=obj55 2023-07-29T01:22:56.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.708 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:56.708 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.708 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.708 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:16 GMT 2023-07-29T01:22:56.709 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.709 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:rHVKG8gNW6ZA3ljBfJt53iz2WJc= 2023-07-29T01:22:56.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:16 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rHVKG8gNW6ZA3ljBfJt53iz2WJc=', 'Content-Length': '0'} 2023-07-29T01:22:56.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086afef46f64ed538-0064c46428-165f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 00:58:16 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hiiqos-64obj551000falseobj552023-07-29T00:57:44.787Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-07-29T01:22:56.710 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=hiiqos-64 zones={a2, a1} 2023-07-29T01:22:56.710 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: DEBUG: waiting for process to exit 2023-07-29T01:22:56.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: INFO: waiting for 300 2023-07-29T01:22:56.711 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi101.stdout: INFO: 2023-07-29T00:58:16.097+0000 7f2e5bb77640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 145401) UID: 0 2023-07-29T01:22:56.711 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi101.stdout: INFO: 2023-07-29T00:58:16.097+0000 7f2e61524880 -1 shutting down 2023-07-29T01:22:56.711 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~0s 2023-07-29T01:22:56.711 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi139.stdout: INFO: 2023-07-29T00:58:21.969+0000 7f5f2671b640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T01:22:56.712 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Stopped 2023-07-29T01:22:56.712 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-64', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:56.712 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-64', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:56.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-64 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:22:56.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:56.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:56.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: tenant: 2023-07-29T01:22:56.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: bucket name: hiiqos-64 2023-07-29T01:22:56.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: total entries: 5 2023-07-29T01:22:56.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/ 2023-07-29T01:22:56.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.716 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.716 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.717 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.717 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:22 GMT 2023-07-29T01:22:56.717 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.718 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:viGS3AtqUOvuzqqrwcqs6CUp3eE= 2023-07-29T01:22:56.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:viGS3AtqUOvuzqqrwcqs6CUp3eE=', 'Content-Length': '0'} 2023-07-29T01:22:56.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000040bb43c709d1b584-0064c4642e-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj17 2023-07-29T01:22:56.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj17 2023-07-29T01:22:56.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj17 2023-07-29T01:22:56.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.721 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.721 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.721 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.722 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:22 GMT 2023-07-29T01:22:56.722 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj17 2023-07-29T01:22:56.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.722 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:W/xWD3XWVoAlkSY++0mf6/umbdw= 2023-07-29T01:22:56.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:W/xWD3XWVoAlkSY++0mf6/umbdw='} 2023-07-29T01:22:56.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009fa9b7f28032b2e9-0064c4642e-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/ 2023-07-29T01:22:56.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.726 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.726 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.726 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.726 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:22 GMT 2023-07-29T01:22:56.726 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.727 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:viGS3AtqUOvuzqqrwcqs6CUp3eE= 2023-07-29T01:22:56.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:viGS3AtqUOvuzqqrwcqs6CUp3eE=', 'Content-Length': '0'} 2023-07-29T01:22:56.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cf631aa8294d11c6-0064c4642e-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj34 2023-07-29T01:22:56.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj34 2023-07-29T01:22:56.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj34 2023-07-29T01:22:56.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.730 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.730 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.731 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.731 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:22 GMT 2023-07-29T01:22:56.731 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj34 2023-07-29T01:22:56.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.732 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:V4R47cW6lqgWVopUio+u0R34IMI= 2023-07-29T01:22:56.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:V4R47cW6lqgWVopUio+u0R34IMI='} 2023-07-29T01:22:56.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005db99e6913b92dbb-0064c4642e-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/ 2023-07-29T01:22:56.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.735 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.735 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.735 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.735 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:22 GMT 2023-07-29T01:22:56.736 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.736 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:viGS3AtqUOvuzqqrwcqs6CUp3eE= 2023-07-29T01:22:56.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:viGS3AtqUOvuzqqrwcqs6CUp3eE=', 'Content-Length': '0'} 2023-07-29T01:22:56.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ed024b9caf61c9c-0064c4642e-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj51 2023-07-29T01:22:56.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj51 2023-07-29T01:22:56.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj51 2023-07-29T01:22:56.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.739 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.740 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.740 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.740 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:22 GMT 2023-07-29T01:22:56.740 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj51 2023-07-29T01:22:56.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.741 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:n1DRIfknmEiJem+yZSBoyswxdk4= 2023-07-29T01:22:56.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:n1DRIfknmEiJem+yZSBoyswxdk4='} 2023-07-29T01:22:56.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e2bf925e934a6589-0064c4642e-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/ 2023-07-29T01:22:56.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.744 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.744 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.744 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.745 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:22 GMT 2023-07-29T01:22:56.745 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.745 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:viGS3AtqUOvuzqqrwcqs6CUp3eE= 2023-07-29T01:22:56.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:viGS3AtqUOvuzqqrwcqs6CUp3eE=', 'Content-Length': '0'} 2023-07-29T01:22:56.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008bc598a6043d6ce5-0064c4642e-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj68 2023-07-29T01:22:56.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj68 2023-07-29T01:22:56.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj68 2023-07-29T01:22:56.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.749 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.749 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.749 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.749 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:22 GMT 2023-07-29T01:22:56.750 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj68 2023-07-29T01:22:56.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.750 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:TMJ7K8KV+po0mbn+pbo/DaSbDsA= 2023-07-29T01:22:56.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TMJ7K8KV+po0mbn+pbo/DaSbDsA='} 2023-07-29T01:22:56.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dfba0626dcef323f-0064c4642e-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/ 2023-07-29T01:22:56.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.753 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.754 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.754 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.754 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:22 GMT 2023-07-29T01:22:56.754 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.755 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:viGS3AtqUOvuzqqrwcqs6CUp3eE= 2023-07-29T01:22:56.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:viGS3AtqUOvuzqqrwcqs6CUp3eE=', 'Content-Length': '0'} 2023-07-29T01:22:56.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f8dcbd8be4d13fdf-0064c4642e-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj85 2023-07-29T01:22:56.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj85 2023-07-29T01:22:56.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj85 2023-07-29T01:22:56.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.758 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.758 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.758 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.759 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:22 GMT 2023-07-29T01:22:56.759 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj85 2023-07-29T01:22:56.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.759 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:t8Z1mEsEbUKr3SAhBcm+AoGnfBU= 2023-07-29T01:22:56.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:t8Z1mEsEbUKr3SAhBcm+AoGnfBU='} 2023-07-29T01:22:56.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000005389d987b02de96-0064c4642e-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.760 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-64', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:56.760 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-64', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:56.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-64 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:22:56.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:56.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:56.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: tenant: 2023-07-29T01:22:56.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: bucket name: hiiqos-64 2023-07-29T01:22:56.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: total entries: 10 2023-07-29T01:22:56.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/ 2023-07-29T01:22:56.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.764 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.765 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.765 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.765 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:22 GMT 2023-07-29T01:22:56.765 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.766 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:viGS3AtqUOvuzqqrwcqs6CUp3eE= 2023-07-29T01:22:56.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:viGS3AtqUOvuzqqrwcqs6CUp3eE=', 'Content-Length': '0'} 2023-07-29T01:22:56.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d57fe3862868c237-0064c4642e-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj19 2023-07-29T01:22:56.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj19 2023-07-29T01:22:56.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj19 2023-07-29T01:22:56.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.769 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.769 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.770 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.770 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:22 GMT 2023-07-29T01:22:56.770 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj19 2023-07-29T01:22:56.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.771 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:rUt6UCIxlrKz4QZn2V3QPPv7loc= 2023-07-29T01:22:56.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rUt6UCIxlrKz4QZn2V3QPPv7loc='} 2023-07-29T01:22:56.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001b8946c3b4e51e8d-0064c4642e-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/ 2023-07-29T01:22:56.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.774 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.774 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.774 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.775 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:22 GMT 2023-07-29T01:22:56.775 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.775 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:viGS3AtqUOvuzqqrwcqs6CUp3eE= 2023-07-29T01:22:56.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:viGS3AtqUOvuzqqrwcqs6CUp3eE=', 'Content-Length': '0'} 2023-07-29T01:22:56.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b237c8948fc6bc6b-0064c4642e-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj38 2023-07-29T01:22:56.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj38 2023-07-29T01:22:56.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj38 2023-07-29T01:22:56.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.778 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.779 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.779 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.779 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:22 GMT 2023-07-29T01:22:56.779 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj38 2023-07-29T01:22:56.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.780 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:gyhcN0+TBy/uimINIG1NNxLHxk8= 2023-07-29T01:22:56.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:gyhcN0+TBy/uimINIG1NNxLHxk8='} 2023-07-29T01:22:56.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005fbad17e4535b2db-0064c4642e-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/ 2023-07-29T01:22:56.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.783 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.783 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.784 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.784 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:22 GMT 2023-07-29T01:22:56.784 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.784 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:viGS3AtqUOvuzqqrwcqs6CUp3eE= 2023-07-29T01:22:56.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:viGS3AtqUOvuzqqrwcqs6CUp3eE=', 'Content-Length': '0'} 2023-07-29T01:22:56.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f116d5b65ff9f3af-0064c4642e-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj57 2023-07-29T01:22:56.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj57 2023-07-29T01:22:56.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj57 2023-07-29T01:22:56.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.788 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.788 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.788 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.788 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:22 GMT 2023-07-29T01:22:56.789 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj57 2023-07-29T01:22:56.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.789 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:8kUys6wYE/CA4JIZRs/80Qv97Po= 2023-07-29T01:22:56.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:8kUys6wYE/CA4JIZRs/80Qv97Po='} 2023-07-29T01:22:56.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003015312dbe4b3444-0064c4642e-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/ 2023-07-29T01:22:56.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.792 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.793 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.793 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.793 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:22 GMT 2023-07-29T01:22:56.793 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.794 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:viGS3AtqUOvuzqqrwcqs6CUp3eE= 2023-07-29T01:22:56.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:viGS3AtqUOvuzqqrwcqs6CUp3eE=', 'Content-Length': '0'} 2023-07-29T01:22:56.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c2907760aadc3b74-0064c4642e-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj76 2023-07-29T01:22:56.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj76 2023-07-29T01:22:56.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj76 2023-07-29T01:22:56.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.797 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.797 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.797 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.798 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:22 GMT 2023-07-29T01:22:56.798 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj76 2023-07-29T01:22:56.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.798 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:kmHC/RENItvlH+6U0bdM0rU2U8I= 2023-07-29T01:22:56.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kmHC/RENItvlH+6U0bdM0rU2U8I='} 2023-07-29T01:22:56.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000087ff0937d1726794-0064c4642e-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/ 2023-07-29T01:22:56.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.802 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.802 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.802 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.802 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:22 GMT 2023-07-29T01:22:56.802 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.803 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:viGS3AtqUOvuzqqrwcqs6CUp3eE= 2023-07-29T01:22:56.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:viGS3AtqUOvuzqqrwcqs6CUp3eE=', 'Content-Length': '0'} 2023-07-29T01:22:56.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000094cd01086acc0a21-0064c4642e-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj95 2023-07-29T01:22:56.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj95 2023-07-29T01:22:56.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj95 2023-07-29T01:22:56.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.806 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.806 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.807 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.807 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:22 GMT 2023-07-29T01:22:56.807 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj95 2023-07-29T01:22:56.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.808 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:bjdmDoPjNNTRuwu4fyXCybAwOaQ= 2023-07-29T01:22:56.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:22 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:bjdmDoPjNNTRuwu4fyXCybAwOaQ='} 2023-07-29T01:22:56.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d3c13157921b737c-0064c4642e-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.808 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-64', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:56.809 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-64', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:56.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-64 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:22:56.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:56.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:56.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: tenant: 2023-07-29T01:22:56.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: bucket name: hiiqos-64 2023-07-29T01:22:56.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: total entries: 15 2023-07-29T01:22:56.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/ 2023-07-29T01:22:56.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.813 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.813 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.813 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.813 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:23 GMT 2023-07-29T01:22:56.814 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.814 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY= 2023-07-29T01:22:56.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY=', 'Content-Length': '0'} 2023-07-29T01:22:56.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fb951661839030e0-0064c4642e-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj23 2023-07-29T01:22:56.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj23 2023-07-29T01:22:56.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj23 2023-07-29T01:22:56.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.817 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.818 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.818 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.818 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:23 GMT 2023-07-29T01:22:56.818 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj23 2023-07-29T01:22:56.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.819 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:EvCk7cAWZ8zDYz4uFP7DBNStlWM= 2023-07-29T01:22:56.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:EvCk7cAWZ8zDYz4uFP7DBNStlWM='} 2023-07-29T01:22:56.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000075dd1f3338e42087-0064c4642e-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/ 2023-07-29T01:22:56.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.822 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.822 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.822 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.823 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:23 GMT 2023-07-29T01:22:56.823 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.823 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY= 2023-07-29T01:22:56.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY=', 'Content-Length': '0'} 2023-07-29T01:22:56.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000071b2878f38fe6ea-0064c4642e-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj46 2023-07-29T01:22:56.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj46 2023-07-29T01:22:56.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj46 2023-07-29T01:22:56.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.827 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.827 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.827 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.827 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:23 GMT 2023-07-29T01:22:56.827 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj46 2023-07-29T01:22:56.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.828 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:fIHtw415vulG6mkYz3py2ENacQM= 2023-07-29T01:22:56.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:fIHtw415vulG6mkYz3py2ENacQM='} 2023-07-29T01:22:56.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009b1cd9d619d22627-0064c4642e-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:22 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/ 2023-07-29T01:22:56.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.831 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.831 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.832 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.832 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:23 GMT 2023-07-29T01:22:56.832 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.833 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY= 2023-07-29T01:22:56.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY=', 'Content-Length': '0'} 2023-07-29T01:22:56.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ce2130b191864bbb-0064c4642f-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj69 2023-07-29T01:22:56.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj69 2023-07-29T01:22:56.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj69 2023-07-29T01:22:56.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.836 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.836 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.836 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.836 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:23 GMT 2023-07-29T01:22:56.837 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj69 2023-07-29T01:22:56.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.837 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:2ZXUZSJXvoXKjA4Zy/7ehKQmbh8= 2023-07-29T01:22:56.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2ZXUZSJXvoXKjA4Zy/7ehKQmbh8='} 2023-07-29T01:22:56.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000071168e943b7caf05-0064c4642f-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/ 2023-07-29T01:22:56.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.840 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.841 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.841 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.841 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:23 GMT 2023-07-29T01:22:56.841 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.842 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY= 2023-07-29T01:22:56.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY=', 'Content-Length': '0'} 2023-07-29T01:22:56.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000285c4cdee6050228-0064c4642f-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj92 2023-07-29T01:22:56.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj92 2023-07-29T01:22:56.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj92 2023-07-29T01:22:56.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.845 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.845 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.845 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.846 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:23 GMT 2023-07-29T01:22:56.846 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj92 2023-07-29T01:22:56.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.846 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:VZ7PhQmiNX9PGrRJ9Wz71ETfCeo= 2023-07-29T01:22:56.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VZ7PhQmiNX9PGrRJ9Wz71ETfCeo='} 2023-07-29T01:22:56.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006cd978a43f9a834c-0064c4642f-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/ 2023-07-29T01:22:56.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.850 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.850 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.850 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.850 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:23 GMT 2023-07-29T01:22:56.850 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.851 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY= 2023-07-29T01:22:56.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY=', 'Content-Length': '0'} 2023-07-29T01:22:56.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a991308a4fd16102-0064c4642f-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj115 2023-07-29T01:22:56.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj115 2023-07-29T01:22:56.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj115 2023-07-29T01:22:56.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.854 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.854 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.855 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.855 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:23 GMT 2023-07-29T01:22:56.855 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj115 2023-07-29T01:22:56.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.856 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:yS0i+IBJBvz964tNQXTcwP14/U8= 2023-07-29T01:22:56.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yS0i+IBJBvz964tNQXTcwP14/U8='} 2023-07-29T01:22:56.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000077907eaf809a301d-0064c4642f-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.856 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-64', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:56.857 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-64', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:56.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-64 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:22:56.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:56.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:56.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: tenant: 2023-07-29T01:22:56.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: bucket name: hiiqos-64 2023-07-29T01:22:56.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: total entries: 20 2023-07-29T01:22:56.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/ 2023-07-29T01:22:56.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.861 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.861 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.861 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.861 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:23 GMT 2023-07-29T01:22:56.862 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.862 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY= 2023-07-29T01:22:56.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY=', 'Content-Length': '0'} 2023-07-29T01:22:56.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000338f6a70caed4fd9-0064c4642f-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj29 2023-07-29T01:22:56.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj29 2023-07-29T01:22:56.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj29 2023-07-29T01:22:56.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.865 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.866 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.866 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.866 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:23 GMT 2023-07-29T01:22:56.866 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj29 2023-07-29T01:22:56.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.867 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:20LMFnIN5bzJ4sH3Ax3NRmptNFM= 2023-07-29T01:22:56.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:20LMFnIN5bzJ4sH3Ax3NRmptNFM='} 2023-07-29T01:22:56.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000594ad3a219ee7503-0064c4642f-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/ 2023-07-29T01:22:56.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.870 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.870 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.870 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.871 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:23 GMT 2023-07-29T01:22:56.871 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.871 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY= 2023-07-29T01:22:56.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY=', 'Content-Length': '0'} 2023-07-29T01:22:56.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000136f6ca122cc1f1f-0064c4642f-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj58 2023-07-29T01:22:56.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj58 2023-07-29T01:22:56.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj58 2023-07-29T01:22:56.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.875 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.875 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.875 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.875 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:23 GMT 2023-07-29T01:22:56.875 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj58 2023-07-29T01:22:56.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.876 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:L/LtnCmK4c4ltqCCZk1J9GhsxOM= 2023-07-29T01:22:56.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:L/LtnCmK4c4ltqCCZk1J9GhsxOM='} 2023-07-29T01:22:56.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b343bf92fca05e8f-0064c4642f-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/ 2023-07-29T01:22:56.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.879 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.879 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.880 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.880 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:23 GMT 2023-07-29T01:22:56.880 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.881 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY= 2023-07-29T01:22:56.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY=', 'Content-Length': '0'} 2023-07-29T01:22:56.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c60fa29b40de807-0064c4642f-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj87 2023-07-29T01:22:56.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj87 2023-07-29T01:22:56.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj87 2023-07-29T01:22:56.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.884 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.884 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.884 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.884 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:23 GMT 2023-07-29T01:22:56.885 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj87 2023-07-29T01:22:56.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.885 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH://xrAfWCQs1JKtug7LLY9W7MfXk= 2023-07-29T01:22:56.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH://xrAfWCQs1JKtug7LLY9W7MfXk='} 2023-07-29T01:22:56.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000062f5ae65d3440796-0064c4642f-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/ 2023-07-29T01:22:56.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.888 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.889 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.889 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.889 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:23 GMT 2023-07-29T01:22:56.889 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.890 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY= 2023-07-29T01:22:56.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY=', 'Content-Length': '0'} 2023-07-29T01:22:56.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001edfc5576eab331c-0064c4642f-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj116 2023-07-29T01:22:56.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj116 2023-07-29T01:22:56.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj116 2023-07-29T01:22:56.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.893 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.893 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.894 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.894 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:23 GMT 2023-07-29T01:22:56.894 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj116 2023-07-29T01:22:56.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.894 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:BiIVMOn/8XnCRhiWxoDMPn9cpvs= 2023-07-29T01:22:56.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BiIVMOn/8XnCRhiWxoDMPn9cpvs='} 2023-07-29T01:22:56.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e8ce936601278e6b-0064c4642f-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/ 2023-07-29T01:22:56.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.898 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.898 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.898 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.898 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:23 GMT 2023-07-29T01:22:56.899 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.899 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY= 2023-07-29T01:22:56.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY=', 'Content-Length': '0'} 2023-07-29T01:22:56.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000071dc2dbb150da616-0064c4642f-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj145 2023-07-29T01:22:56.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj145 2023-07-29T01:22:56.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj145 2023-07-29T01:22:56.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.902 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.903 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.903 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.903 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:23 GMT 2023-07-29T01:22:56.903 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj145 2023-07-29T01:22:56.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.904 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:rJ4rCCqu75csmkLGWwu1Xj4pHuA= 2023-07-29T01:22:56.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:rJ4rCCqu75csmkLGWwu1Xj4pHuA='} 2023-07-29T01:22:56.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f7ab446e3b8a239f-0064c4642f-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.904 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-64', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:56.905 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-64', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:56.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-64 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:22:56.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:56.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:56.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: tenant: 2023-07-29T01:22:56.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: bucket name: hiiqos-64 2023-07-29T01:22:56.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: total entries: 25 2023-07-29T01:22:56.906 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 has been restored 2023-07-29T01:22:56.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/ 2023-07-29T01:22:56.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.909 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.909 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.909 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.910 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:23 GMT 2023-07-29T01:22:56.910 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.910 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY= 2023-07-29T01:22:56.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY=', 'Content-Length': '0'} 2023-07-29T01:22:56.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb96a4de6fece522-0064c4642f-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj31 2023-07-29T01:22:56.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj31 2023-07-29T01:22:56.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj31 2023-07-29T01:22:56.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.914 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.914 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.914 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.914 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:23 GMT 2023-07-29T01:22:56.915 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj31 2023-07-29T01:22:56.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.915 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:ADX7jo0PAtVZeTQyf2WWGyliYw8= 2023-07-29T01:22:56.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ADX7jo0PAtVZeTQyf2WWGyliYw8='} 2023-07-29T01:22:56.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c3ee9d01bd548065-0064c4642f-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/ 2023-07-29T01:22:56.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.918 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.919 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.919 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.919 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:23 GMT 2023-07-29T01:22:56.919 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.920 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY= 2023-07-29T01:22:56.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY=', 'Content-Length': '0'} 2023-07-29T01:22:56.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004492d81d770addf6-0064c4642f-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj62 2023-07-29T01:22:56.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj62 2023-07-29T01:22:56.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj62 2023-07-29T01:22:56.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.923 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.923 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.923 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.924 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:23 GMT 2023-07-29T01:22:56.924 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj62 2023-07-29T01:22:56.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.924 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:Ben/UU9dw8COBVcXtB6GkMwraxA= 2023-07-29T01:22:56.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:Ben/UU9dw8COBVcXtB6GkMwraxA='} 2023-07-29T01:22:56.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005807066d8a53450a-0064c4642f-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/ 2023-07-29T01:22:56.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.928 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.928 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.928 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.928 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:23 GMT 2023-07-29T01:22:56.928 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.929 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY= 2023-07-29T01:22:56.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY=', 'Content-Length': '0'} 2023-07-29T01:22:56.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001e802c9167dbb77e-0064c4642f-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj93 2023-07-29T01:22:56.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj93 2023-07-29T01:22:56.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj93 2023-07-29T01:22:56.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.932 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.932 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.933 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.933 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:23 GMT 2023-07-29T01:22:56.933 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj93 2023-07-29T01:22:56.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.934 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:tEyhZuyW0BSFmigNxxRTyl51LKo= 2023-07-29T01:22:56.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tEyhZuyW0BSFmigNxxRTyl51LKo='} 2023-07-29T01:22:56.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000de634436149d0fdb-0064c4642f-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/ 2023-07-29T01:22:56.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.937 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.937 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.937 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.938 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:23 GMT 2023-07-29T01:22:56.938 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.938 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY= 2023-07-29T01:22:56.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY=', 'Content-Length': '0'} 2023-07-29T01:22:56.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e17fa169fe81a014-0064c4642f-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj124 2023-07-29T01:22:56.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj124 2023-07-29T01:22:56.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj124 2023-07-29T01:22:56.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.942 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.942 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.942 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.942 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:23 GMT 2023-07-29T01:22:56.942 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj124 2023-07-29T01:22:56.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.943 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:x16d5Y7vIzwi1MoCpWG1YnD0qJY= 2023-07-29T01:22:56.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:x16d5Y7vIzwi1MoCpWG1YnD0qJY='} 2023-07-29T01:22:56.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008560f975adb190d1-0064c4642f-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/ 2023-07-29T01:22:56.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.946 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.946 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.947 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.947 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:23 GMT 2023-07-29T01:22:56.947 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.948 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY= 2023-07-29T01:22:56.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:IheF2VURc+Q6N4ItJhTq95EcKqY=', 'Content-Length': '0'} 2023-07-29T01:22:56.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f05d22c2a3b95c68-0064c4642f-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj155 2023-07-29T01:22:56.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj155 2023-07-29T01:22:56.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj155 2023-07-29T01:22:56.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.951 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.951 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.951 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.951 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:23 GMT 2023-07-29T01:22:56.952 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj155 2023-07-29T01:22:56.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.952 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:UPuNe24LCHpb6JxiDYiIERwX/xc= 2023-07-29T01:22:56.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:23 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:UPuNe24LCHpb6JxiDYiIERwX/xc='} 2023-07-29T01:22:56.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000033bc912772c9e448-0064c4642f-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.953 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-64', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:56.953 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hiiqos-64', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:56.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hiiqos-64 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:22:56.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:56.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:56.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: tenant: 2023-07-29T01:22:56.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: bucket name: hiiqos-64 2023-07-29T01:22:56.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: total entries: 30 2023-07-29T01:22:56.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/ 2023-07-29T01:22:56.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.957 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.958 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.958 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.958 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:24 GMT 2023-07-29T01:22:56.958 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.959 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:QFflsEIpX1mG+hR5r/ygIAaBax4= 2023-07-29T01:22:56.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QFflsEIpX1mG+hR5r/ygIAaBax4=', 'Content-Length': '0'} 2023-07-29T01:22:56.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005ae0e84edbe4f7a6-0064c4642f-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj37 2023-07-29T01:22:56.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj37 2023-07-29T01:22:56.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj37 2023-07-29T01:22:56.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.962 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.962 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.962 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.963 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:24 GMT 2023-07-29T01:22:56.963 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj37 2023-07-29T01:22:56.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.963 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:VOH4YMi/PTl+sjx/EwL8mxVBnfE= 2023-07-29T01:22:56.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:VOH4YMi/PTl+sjx/EwL8mxVBnfE='} 2023-07-29T01:22:56.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006f52a9297a76dced-0064c4642f-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:23 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/ 2023-07-29T01:22:56.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.967 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.967 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.967 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.967 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:24 GMT 2023-07-29T01:22:56.967 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.968 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:QFflsEIpX1mG+hR5r/ygIAaBax4= 2023-07-29T01:22:56.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QFflsEIpX1mG+hR5r/ygIAaBax4=', 'Content-Length': '0'} 2023-07-29T01:22:56.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000661304cd3cf5fb33-0064c46430-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj74 2023-07-29T01:22:56.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj74 2023-07-29T01:22:56.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj74 2023-07-29T01:22:56.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.971 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.971 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.972 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.972 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:24 GMT 2023-07-29T01:22:56.972 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj74 2023-07-29T01:22:56.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.973 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:2WdMDkMbMXJUHTeS1G25M9pNwBE= 2023-07-29T01:22:56.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:2WdMDkMbMXJUHTeS1G25M9pNwBE='} 2023-07-29T01:22:56.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006f0c4ed465e74113-0064c46430-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/ 2023-07-29T01:22:56.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.976 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.976 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.976 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.976 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:24 GMT 2023-07-29T01:22:56.977 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.977 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:QFflsEIpX1mG+hR5r/ygIAaBax4= 2023-07-29T01:22:56.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QFflsEIpX1mG+hR5r/ygIAaBax4=', 'Content-Length': '0'} 2023-07-29T01:22:56.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007bb9058b7153c3f9-0064c46430-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj111 2023-07-29T01:22:56.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj111 2023-07-29T01:22:56.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj111 2023-07-29T01:22:56.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.980 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.981 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.981 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.981 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:24 GMT 2023-07-29T01:22:56.981 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj111 2023-07-29T01:22:56.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.982 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:0DDFFPR8hY5rbxdfLIniRerLJ+o= 2023-07-29T01:22:56.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:0DDFFPR8hY5rbxdfLIniRerLJ+o='} 2023-07-29T01:22:56.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000062f6c1f1e094520f-0064c46430-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/ 2023-07-29T01:22:56.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.985 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.985 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.986 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.986 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:24 GMT 2023-07-29T01:22:56.986 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.986 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:QFflsEIpX1mG+hR5r/ygIAaBax4= 2023-07-29T01:22:56.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QFflsEIpX1mG+hR5r/ygIAaBax4=', 'Content-Length': '0'} 2023-07-29T01:22:56.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000003e958dd09ed0c8b-0064c46430-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj148 2023-07-29T01:22:56.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj148 2023-07-29T01:22:56.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj148 2023-07-29T01:22:56.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.990 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.990 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.990 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:56.990 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:24 GMT 2023-07-29T01:22:56.991 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj148 2023-07-29T01:22:56.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.991 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:QKuNJNpD64X5Z61YBXus+gu34EQ= 2023-07-29T01:22:56.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QKuNJNpD64X5Z61YBXus+gu34EQ='} 2023-07-29T01:22:56.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000012383b1a8a7ff97f-0064c46430-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/ 2023-07-29T01:22:56.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/ 2023-07-29T01:22:56.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:56.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/ 2023-07-29T01:22:56.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:56.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.994 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:56.995 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.995 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:56.995 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:24 GMT 2023-07-29T01:22:56.995 INFO:tasks.rgw_multisite_tests:/hiiqos-64/ 2023-07-29T01:22:56.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:56.996 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:QFflsEIpX1mG+hR5r/ygIAaBax4= 2023-07-29T01:22:56.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 00:58:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:QFflsEIpX1mG+hR5r/ygIAaBax4=', 'Content-Length': '0'} 2023-07-29T01:22:56.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000d8ab57464f360b7-0064c46430-165f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 00:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:56.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-64/obj185 2023-07-29T01:22:56.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-64/obj185 2023-07-29T01:22:56.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:56.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-64/obj185 2023-07-29T01:22:56.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:56.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-07-29T01:22:56.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:56.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:56.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:56.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:56.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:56.999 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:56.999 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-07-29T01:22:56.999 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:57.000 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 00:58:24 GMT 2023-07-29T01:22:57.000 INFO:tasks.rgw_multisite_tests:/hiiqos-64/obj185 2023-07-29T01:22:57.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:57.000 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:1oP1h/rF3Y5Mwf0ltytXmhB6KMI= 2023-07-29T01:22:57.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 00:58:24 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:1oP1h/rF3Y5Mwf0ltytXmhB6KMI='} 2023-07-29T01:22:57.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002c9df75fbb40112a-0064c46430-165f-a1'), ('Date', 'Sat, 29 Jul 2023 00:58:24 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:57.001 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:57.001 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:57.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:22:57.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:57.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:57.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:58:24.223+0000 7fe304010880 5 RGW-SYNC:data: waiting on data sync status lease 2023-07-29T01:22:57.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:58:24.224+0000 7fe304010880 5 RGW-SYNC:data: acquired data sync status lease 2023-07-29T01:22:57.003 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Restarting daemon 2023-07-29T01:22:57.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-07-29T01:22:57.003 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Started 2023-07-29T01:22:57.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > curl http://smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:57.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2023-07-29T01:22:57.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: Dload Upload Total Spent Left Speed 2023-07-29T01:22:57.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-07-29T01:22:57.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: curl: (7) Failed to connect to smithi101.front.sepia.ceph.com port 8001: Connection refused 2023-07-29T01:22:57.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 7 2023-07-29T01:22:57.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > curl http://smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:57.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2023-07-29T01:22:57.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: Dload Upload Total Spent Left Speed 2023-07-29T01:22:57.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-07-29T01:22:57.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: anonymous 2023-07-29T01:22:57.006 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=hiiqos-64 2023-07-29T01:22:57.006 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T01:22:57.006 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hiiqos-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-07-29T01:22:57.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hiiqos-64 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-07-29T01:22:57.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:57.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:57.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:58:25.416+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T01:22:57.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:58:30.416+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T01:22:57.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:58:35.417+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T01:22:57.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:58:40.418+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T01:22:57.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:58:45.419+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T01:22:57.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:58:50.419+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T01:22:57.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:58:55.420+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T01:22:57.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:59:00.420+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T01:22:57.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:59:05.421+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T01:22:57.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:59:10.422+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T01:22:57.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:59:15.422+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T01:22:57.010 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.1.smithi139.stdout: INFO: 2023-07-29T00:59:17.886+0000 7f5731a6a640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T01:22:57.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:59:20.423+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T01:22:57.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:59:25.423+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T01:22:57.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:59:30.424+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T01:22:57.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:59:35.424+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T01:22:57.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:59:40.425+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T01:22:57.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:59:45.426+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T01:22:57.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:59:50.427+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T01:22:57.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T00:59:55.427+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T01:22:57.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:00:00.428+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T01:22:57.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:00:05.428+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T01:22:57.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:00:10.429+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T01:22:57.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:00:15.430+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T01:22:57.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:00:20.430+0000 7fa62e326880 1 waiting to reach latest gen 6, on 0.. 2023-07-29T01:22:57.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:00:25.431+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:57.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:00:30.431+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:57.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:00:35.432+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:57.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:00:40.433+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:57.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:00:45.433+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:57.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:00:50.434+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:57.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:00:55.434+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:57.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:01:00.435+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:57.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:01:05.435+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:57.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:01:10.435+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:57.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:01:15.436+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:57.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:01:20.437+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:57.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:01:25.437+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:57.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:01:30.438+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:57.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:01:35.438+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:57.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:01:40.439+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:57.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:01:45.440+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:57.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:01:50.440+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:57.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:01:55.441+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:57.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:02:00.442+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:57.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:02:05.442+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:57.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:02:10.442+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:57.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:02:15.443+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:57.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:02:20.444+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:57.019 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi101.stdout: INFO: 2023-07-29T01:02:24.700+0000 7f0081a5e640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T01:22:57.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:02:25.444+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:57.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:02:30.445+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:57.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:02:35.445+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:57.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:02:40.446+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:57.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:02:45.448+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:57.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:02:50.448+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:57.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:02:55.448+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:57.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:03:00.449+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:57.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:03:05.449+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:57.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:03:10.450+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:57.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:03:15.451+0000 7fa62e326880 1 waiting to reach latest gen 6, on 2.. 2023-07-29T01:22:57.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:03:20.452+0000 7fa62e326880 -1 bucket checkpoint timed out waiting to reach latest generation 6 2023-07-29T01:22:57.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:03:20.452+0000 7fa62e326880 0 bucket sync checkpoint failed: (110) Connection timed out 2023-07-29T01:22:57.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:03:20.452+0000 7fa62e326880 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-07-29T01:22:57.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2023-07-29T01:22:57.023 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-07-29T01:22:57.023 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:57.023 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-07-29T01:22:57.023 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_flow_symmetrical_zonegroup_all: 2023-07-29T01:22:57.024 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-07-29T01:22:57.025 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-07-29T01:22:57.025 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_61140164d2579c30ffff3970615e0f82e40c6470/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-07-29T01:22:57.025 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-07-29T01:22:57.025 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_0484097b0572e96f42cf6402cbe0ac8dcb046577/qa/../src/test/rgw/rgw_multi/tests.py", line 2177, in test_sync_flow_symmetrical_zonegroup_all 2023-07-29T01:22:57.026 INFO:tasks.rgw_multisite_tests: check_object_exists(bucket, objnames[0], content) 2023-07-29T01:22:57.026 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_0484097b0572e96f42cf6402cbe0ac8dcb046577/qa/../src/test/rgw/rgw_multi/tests.py", line 2041, in check_object_exists 2023-07-29T01:22:57.026 INFO:tasks.rgw_multisite_tests: assert_not_equal(k, None) 2023-07-29T01:22:57.026 INFO:tasks.rgw_multisite_tests:AssertionError: None == None 2023-07-29T01:22:57.027 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-07-29T01:22:57.027 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.027 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:57.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:57.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:57.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T01:22:57.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: [ 2023-07-29T01:22:57.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "last_update": "2023-07-29T00:40:17.836402Z" 2023-07-29T01:22:57.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "last_update": "2023-07-29T01:04:39.844113Z" 2023-07-29T01:22:57.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:57.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "last_update": "0.000000" 2023-07-29T01:22:57.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "last_update": "2023-07-29T00:40:12.811406Z" 2023-07-29T01:22:57.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:57.032 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.033 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-07-29T01:22:57.033 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.033 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:57.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:57.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:57.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:04:51.464+0000 7feea0922880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:04:51.466+0000 7feea0922880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:57.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:57.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:57.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:57.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:57.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:57.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 1, 2023-07-29T01:22:57.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:57.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:57.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:57.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "0.000000", 2023-07-29T01:22:57.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:57.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:57.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:57.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 1, 2023-07-29T01:22:57.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 1 2023-07-29T01:22:57.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:04:51Z" 2023-07-29T01:22:57.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.050 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-07-29T01:22:57.050 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:57.050 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:57.050 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:57.051 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:57.051 INFO:tasks.rgw_multisite_tests: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.051 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.051 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.052 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:57.052 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.052 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:57.052 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.052 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.053 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.053 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.053 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-07-29T01:22:57.053 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.054 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.054 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.054 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.054 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.055 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.055 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:57.055 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.055 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.055 INFO:tasks.rgw_multisite_tests: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.056 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.056 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.056 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.056 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.057 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.057 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.057 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.057 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.058 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:57.058 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.058 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.058 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-07-29T01:22:57.058 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.059 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.059 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.059 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-07-29T01:22:57.059 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.060 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.060 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.060 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.060 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:57.061 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.061 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.061 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.061 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.062 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.062 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.062 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.062 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.062 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.063 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.063 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:57.063 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.063 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:57.064 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-07-29T01:22:57.064 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-07-29T01:22:57.064 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.064 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:04:51Z" 2023-07-29T01:22:57.065 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:57.065 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:57.065 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:22:57.065 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.065 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.066 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-07-29T01:22:57.066 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:57.066 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:57.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:22:57.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:57.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:57.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: {"groups":[]} 2023-07-29T01:22:57.067 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:57.068 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:57.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:22:57.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:57.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:57.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-07-29T01:22:57.069 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:57.069 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:57.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:22:57.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:57.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:57.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2023-07-29T01:22:57.070 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:57.070 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:57.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:22:57.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:57.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:57.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-07-29T01:22:57.072 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:57.072 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:57.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:22:57.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:57.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:57.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-07-29T01:22:57.073 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:57.073 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:57.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:22:57.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:57.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:57.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "epoch": 15, 2023-07-29T01:22:57.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "predecessor_uuid": "a0987484-ce37-4520-a564-54b7ea8c5251", 2023-07-29T01:22:57.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": [], 2023-07-29T01:22:57.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period_map": { 2023-07-29T01:22:57.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "zonegroups": [ 2023-07-29T01:22:57.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "id": "a", 2023-07-29T01:22:57.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "name": "a", 2023-07-29T01:22:57.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "api_name": "", 2023-07-29T01:22:57.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "is_master": true, 2023-07-29T01:22:57.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "endpoints": [ 2023-07-29T01:22:57.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-29T01:22:57.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ], 2023-07-29T01:22:57.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "hostnames": [], 2023-07-29T01:22:57.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "hostnames_s3website": [], 2023-07-29T01:22:57.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:22:57.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "zones": [ 2023-07-29T01:22:57.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "id": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:22:57.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "name": "a1", 2023-07-29T01:22:57.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "endpoints": [ 2023-07-29T01:22:57.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-29T01:22:57.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ], 2023-07-29T01:22:57.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "log_meta": false, 2023-07-29T01:22:57.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "log_data": true, 2023-07-29T01:22:57.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "bucket_index_max_shards": 11, 2023-07-29T01:22:57.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "read_only": false, 2023-07-29T01:22:57.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "tier_type": "", 2023-07-29T01:22:57.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_from_all": true, 2023-07-29T01:22:57.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_from": [], 2023-07-29T01:22:57.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "redirect_zone": "", 2023-07-29T01:22:57.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "supported_features": [ 2023-07-29T01:22:57.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "compress-encrypted", 2023-07-29T01:22:57.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "resharding" 2023-07-29T01:22:57.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:57.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "id": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-29T01:22:57.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "name": "a2", 2023-07-29T01:22:57.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "endpoints": [ 2023-07-29T01:22:57.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "http://smithi101.front.sepia.ceph.com:8001" 2023-07-29T01:22:57.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ], 2023-07-29T01:22:57.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "log_meta": false, 2023-07-29T01:22:57.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "log_data": true, 2023-07-29T01:22:57.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "bucket_index_max_shards": 11, 2023-07-29T01:22:57.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "read_only": false, 2023-07-29T01:22:57.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "tier_type": "", 2023-07-29T01:22:57.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_from_all": true, 2023-07-29T01:22:57.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_from": [], 2023-07-29T01:22:57.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "redirect_zone": "", 2023-07-29T01:22:57.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "supported_features": [ 2023-07-29T01:22:57.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "compress-encrypted", 2023-07-29T01:22:57.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "resharding" 2023-07-29T01:22:57.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:57.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ], 2023-07-29T01:22:57.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "placement_targets": [ 2023-07-29T01:22:57.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "name": "default-placement", 2023-07-29T01:22:57.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "tags": [], 2023-07-29T01:22:57.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "storage_classes": [ 2023-07-29T01:22:57.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "STANDARD" 2023-07-29T01:22:57.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:57.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ], 2023-07-29T01:22:57.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "default_placement": "default-placement", 2023-07-29T01:22:57.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T01:22:57.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_policy": { 2023-07-29T01:22:57.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "groups": [ 2023-07-29T01:22:57.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "id": "sync-group", 2023-07-29T01:22:57.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "data_flow": { 2023-07-29T01:22:57.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "symmetrical": [ 2023-07-29T01:22:57.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "id": "sync-flow1", 2023-07-29T01:22:57.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "zones": [ 2023-07-29T01:22:57.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:22:57.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "59d1bb30-3b1c-476f-9111-052c1ea0b711" 2023-07-29T01:22:57.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:57.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:57.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pipes": [ 2023-07-29T01:22:57.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "id": "sync-pipe", 2023-07-29T01:22:57.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "source": { 2023-07-29T01:22:57.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "bucket": "*", 2023-07-29T01:22:57.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "zones": [ 2023-07-29T01:22:57.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:22:57.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "59d1bb30-3b1c-476f-9111-052c1ea0b711" 2023-07-29T01:22:57.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:57.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "dest": { 2023-07-29T01:22:57.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "bucket": "*", 2023-07-29T01:22:57.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "zones": [ 2023-07-29T01:22:57.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:22:57.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "59d1bb30-3b1c-476f-9111-052c1ea0b711" 2023-07-29T01:22:57.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:57.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "params": { 2023-07-29T01:22:57.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "source": { 2023-07-29T01:22:57.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "filter": { 2023-07-29T01:22:57.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "tags": [] 2023-07-29T01:22:57.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "dest": {}, 2023-07-29T01:22:57.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "priority": 0, 2023-07-29T01:22:57.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "mode": "system", 2023-07-29T01:22:57.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "user": "" 2023-07-29T01:22:57.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ], 2023-07-29T01:22:57.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "enabled" 2023-07-29T01:22:57.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:57.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "enabled_features": [ 2023-07-29T01:22:57.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "resharding" 2023-07-29T01:22:57.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:57.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "id": "b", 2023-07-29T01:22:57.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "name": "b", 2023-07-29T01:22:57.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "api_name": "", 2023-07-29T01:22:57.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "is_master": false, 2023-07-29T01:22:57.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "endpoints": [ 2023-07-29T01:22:57.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-29T01:22:57.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ], 2023-07-29T01:22:57.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "hostnames": [], 2023-07-29T01:22:57.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "hostnames_s3website": [], 2023-07-29T01:22:57.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "master_zone": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T01:22:57.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "zones": [ 2023-07-29T01:22:57.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "id": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T01:22:57.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "name": "b1", 2023-07-29T01:22:57.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "endpoints": [ 2023-07-29T01:22:57.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-29T01:22:57.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ], 2023-07-29T01:22:57.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "log_meta": false, 2023-07-29T01:22:57.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "log_data": true, 2023-07-29T01:22:57.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "bucket_index_max_shards": 11, 2023-07-29T01:22:57.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "read_only": false, 2023-07-29T01:22:57.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "tier_type": "", 2023-07-29T01:22:57.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_from_all": true, 2023-07-29T01:22:57.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_from": [], 2023-07-29T01:22:57.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "redirect_zone": "", 2023-07-29T01:22:57.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "supported_features": [ 2023-07-29T01:22:57.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "compress-encrypted", 2023-07-29T01:22:57.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "resharding" 2023-07-29T01:22:57.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:57.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "id": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-29T01:22:57.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "name": "b2", 2023-07-29T01:22:57.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "endpoints": [ 2023-07-29T01:22:57.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "http://smithi139.front.sepia.ceph.com:8001" 2023-07-29T01:22:57.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ], 2023-07-29T01:22:57.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "log_meta": false, 2023-07-29T01:22:57.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "log_data": true, 2023-07-29T01:22:57.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "bucket_index_max_shards": 11, 2023-07-29T01:22:57.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "read_only": false, 2023-07-29T01:22:57.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "tier_type": "", 2023-07-29T01:22:57.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_from_all": true, 2023-07-29T01:22:57.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_from": [], 2023-07-29T01:22:57.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "redirect_zone": "", 2023-07-29T01:22:57.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "supported_features": [ 2023-07-29T01:22:57.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "compress-encrypted", 2023-07-29T01:22:57.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "resharding" 2023-07-29T01:22:57.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:57.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ], 2023-07-29T01:22:57.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "placement_targets": [ 2023-07-29T01:22:57.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "name": "default-placement", 2023-07-29T01:22:57.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "tags": [], 2023-07-29T01:22:57.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "storage_classes": [ 2023-07-29T01:22:57.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "STANDARD" 2023-07-29T01:22:57.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:57.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ], 2023-07-29T01:22:57.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "default_placement": "default-placement", 2023-07-29T01:22:57.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T01:22:57.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_policy": { 2023-07-29T01:22:57.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "groups": [] 2023-07-29T01:22:57.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "enabled_features": [ 2023-07-29T01:22:57.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "resharding" 2023-07-29T01:22:57.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:57.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ], 2023-07-29T01:22:57.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "short_zone_ids": [ 2023-07-29T01:22:57.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:22:57.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": 2896287231 2023-07-29T01:22:57.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-29T01:22:57.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": 815606810 2023-07-29T01:22:57.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T01:22:57.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": 2009171363 2023-07-29T01:22:57.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-29T01:22:57.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": 3105190295 2023-07-29T01:22:57.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:57.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "master_zonegroup": "a", 2023-07-29T01:22:57.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:22:57.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period_config": { 2023-07-29T01:22:57.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "bucket_quota": { 2023-07-29T01:22:57.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "enabled": false, 2023-07-29T01:22:57.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "check_on_raw": false, 2023-07-29T01:22:57.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "max_size": -1, 2023-07-29T01:22:57.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "max_size_kb": 0, 2023-07-29T01:22:57.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "max_objects": -1 2023-07-29T01:22:57.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "user_quota": { 2023-07-29T01:22:57.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "enabled": false, 2023-07-29T01:22:57.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "check_on_raw": false, 2023-07-29T01:22:57.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "max_size": -1, 2023-07-29T01:22:57.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "max_size_kb": 0, 2023-07-29T01:22:57.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "max_objects": -1 2023-07-29T01:22:57.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "user_ratelimit": { 2023-07-29T01:22:57.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "max_read_ops": 0, 2023-07-29T01:22:57.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "max_write_ops": 0, 2023-07-29T01:22:57.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "max_read_bytes": 0, 2023-07-29T01:22:57.128 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "max_write_bytes": 0, 2023-07-29T01:22:57.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "enabled": false 2023-07-29T01:22:57.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "bucket_ratelimit": { 2023-07-29T01:22:57.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "max_read_ops": 0, 2023-07-29T01:22:57.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "max_write_ops": 0, 2023-07-29T01:22:57.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "max_read_bytes": 0, 2023-07-29T01:22:57.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "max_write_bytes": 0, 2023-07-29T01:22:57.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "enabled": false 2023-07-29T01:22:57.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "anonymous_ratelimit": { 2023-07-29T01:22:57.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "max_read_ops": 0, 2023-07-29T01:22:57.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "max_write_ops": 0, 2023-07-29T01:22:57.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "max_read_bytes": 0, 2023-07-29T01:22:57.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "max_write_bytes": 0, 2023-07-29T01:22:57.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "enabled": false 2023-07-29T01:22:57.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T01:22:57.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_name": "test-realm", 2023-07-29T01:22:57.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.134 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:57.134 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:57.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:22:57.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:57.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:57.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-07-29T01:22:57.135 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=hiiqos-66 2023-07-29T01:22:57.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-66/ 2023-07-29T01:22:57.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-66/ 2023-07-29T01:22:57.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:57.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-66/ 2023-07-29T01:22:57.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:57.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:57.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:57.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:57.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:57.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-07-29T01:22:57.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:57.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:57.138 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:57.138 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:57.138 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:57.139 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 01:04:52 GMT 2023-07-29T01:22:57.139 INFO:tasks.rgw_multisite_tests:/hiiqos-66/ 2023-07-29T01:22:57.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:57.139 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:yHATXwtMvF1zxWEuR4NG9pfhIOU= 2023-07-29T01:22:57.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:04:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yHATXwtMvF1zxWEuR4NG9pfhIOU=', 'Content-Length': '0'} 2023-07-29T01:22:57.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000000aa8bf2aee9a9003-0064c465b4-2348-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:04:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:57.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-66/ 2023-07-29T01:22:57.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-66/ 2023-07-29T01:22:57.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:57.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-66/ 2023-07-29T01:22:57.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:57.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:57.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:57.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:57.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:57.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:57.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:57.143 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:57.143 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:57.143 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:57.143 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 01:04:52 GMT 2023-07-29T01:22:57.144 INFO:tasks.rgw_multisite_tests:/hiiqos-66/ 2023-07-29T01:22:57.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:57.144 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:d3d3momYAX1cK6JVhu/iBqD5YxI= 2023-07-29T01:22:57.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:04:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:d3d3momYAX1cK6JVhu/iBqD5YxI=', 'Content-Length': '0'} 2023-07-29T01:22:57.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000516e7d981436700f-0064c465b4-2348-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:04:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:57.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-66/obj1 2023-07-29T01:22:57.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-66/obj1 2023-07-29T01:22:57.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:57.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-66/obj1 2023-07-29T01:22:57.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:57.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T01:22:57.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:57.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:57.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:57.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:57.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:57.147 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:57.147 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-07-29T01:22:57.148 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:57.148 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 01:04:52 GMT 2023-07-29T01:22:57.148 INFO:tasks.rgw_multisite_tests:/hiiqos-66/obj1 2023-07-29T01:22:57.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:57.149 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:BElmvig117lTNquBaWWmcijIFnw= 2023-07-29T01:22:57.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 01:04:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:BElmvig117lTNquBaWWmcijIFnw='} 2023-07-29T01:22:57.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000033db44b3b18ab085-0064c465b4-2348-a1'), ('Date', 'Sat, 29 Jul 2023 01:04:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:57.149 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a2 name=hiiqos-67 2023-07-29T01:22:57.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-67/ 2023-07-29T01:22:57.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-67/ 2023-07-29T01:22:57.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:57.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-67/ 2023-07-29T01:22:57.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:57.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:57.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:57.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:57.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:57.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-29T01:22:57.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:57.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:57.152 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:57.153 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:57.153 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:57.153 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 01:04:52 GMT 2023-07-29T01:22:57.153 INFO:tasks.rgw_multisite_tests:/hiiqos-67/ 2023-07-29T01:22:57.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:57.154 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:D4th+czaQ0xOyr1/BRs9I0S1jS0= 2023-07-29T01:22:57.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:04:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:D4th+czaQ0xOyr1/BRs9I0S1jS0=', 'Content-Length': '0'} 2023-07-29T01:22:57.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000003fa8b7f2bce749d6-0064c465b4-233c-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:04:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:57.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-67/ 2023-07-29T01:22:57.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-67/ 2023-07-29T01:22:57.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:57.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-67/ 2023-07-29T01:22:57.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:57.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:57.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:57.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:57.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:57.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:57.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:57.157 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:57.157 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:57.157 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:57.158 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 01:04:52 GMT 2023-07-29T01:22:57.158 INFO:tasks.rgw_multisite_tests:/hiiqos-67/ 2023-07-29T01:22:57.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:57.158 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:KUbMrRZqlHA50kakeUand6rzPE4= 2023-07-29T01:22:57.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:04:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:KUbMrRZqlHA50kakeUand6rzPE4=', 'Content-Length': '0'} 2023-07-29T01:22:57.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000059dbbb3c6a5577c1-0064c465b4-233c-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:04:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:57.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-67/obj2 2023-07-29T01:22:57.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-67/obj2 2023-07-29T01:22:57.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:57.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-67/obj2 2023-07-29T01:22:57.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:57.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T01:22:57.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:57.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:57.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:57.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:57.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:57.162 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:57.162 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-07-29T01:22:57.162 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:57.162 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 01:04:52 GMT 2023-07-29T01:22:57.163 INFO:tasks.rgw_multisite_tests:/hiiqos-67/obj2 2023-07-29T01:22:57.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:57.163 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:6VjbmdJZ4ckcf7JNRkgFcwNrViQ= 2023-07-29T01:22:57.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 01:04:52 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:6VjbmdJZ4ckcf7JNRkgFcwNrViQ='} 2023-07-29T01:22:57.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007eb3a8db0f58ec28-0064c465b4-233c-a2'), ('Date', 'Sat, 29 Jul 2023 01:04:52 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:57.164 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.164 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:57.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:57.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:57.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T01:22:57.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: [ 2023-07-29T01:22:57.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "last_update": "2023-07-29T00:40:17.836402Z" 2023-07-29T01:22:57.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690592692.567652_1462.1", 2023-07-29T01:22:57.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "last_update": "2023-07-29T01:04:52.567652Z" 2023-07-29T01:22:57.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:57.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "last_update": "0.000000" 2023-07-29T01:22:57.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "last_update": "2023-07-29T00:40:12.811406Z" 2023-07-29T01:22:57.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:57.169 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.169 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-07-29T01:22:57.170 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.170 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:57.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:57.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:57.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:04:52.836+0000 7fac1df35880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:04:52.837+0000 7fac1df35880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.171 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:57.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:57.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:57.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:57.172 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:57.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:57.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 1, 2023-07-29T01:22:57.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:57.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:57.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:57.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "0.000000", 2023-07-29T01:22:57.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:57.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.183 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:57.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:57.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 1, 2023-07-29T01:22:57.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 1 2023-07-29T01:22:57.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:04:52Z" 2023-07-29T01:22:57.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.186 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-07-29T01:22:57.186 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:57.187 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:57.187 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:57.187 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:57.187 INFO:tasks.rgw_multisite_tests: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.188 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.188 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.188 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:57.188 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.189 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:57.189 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.189 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.189 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.189 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.190 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-07-29T01:22:57.190 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.190 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.190 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.191 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.191 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.191 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.191 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:57.192 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.192 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.192 INFO:tasks.rgw_multisite_tests: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.192 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.192 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.193 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.193 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.193 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.193 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.194 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.194 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.194 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:57.194 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.195 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.195 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-07-29T01:22:57.195 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.195 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.195 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.196 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-07-29T01:22:57.196 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.196 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.196 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.197 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.197 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:57.197 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.197 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.198 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.198 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.198 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.198 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.198 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.199 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.199 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.199 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.199 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:57.200 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.200 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:57.200 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-07-29T01:22:57.200 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-07-29T01:22:57.201 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.201 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:04:52Z" 2023-07-29T01:22:57.201 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:57.201 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:57.201 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:22:57.202 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.202 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.202 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:22:57.202 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.203 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:57.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:57.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:57.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:04:57.954+0000 7f3f1b29d880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:04:57.955+0000 7f3f1b29d880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:57.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:57.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:57.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:57.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:57.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:57.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 1, 2023-07-29T01:22:57.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:57.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:57.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:57.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "0.000000", 2023-07-29T01:22:57.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:57.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:57.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:57.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 1, 2023-07-29T01:22:57.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 1 2023-07-29T01:22:57.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:04:57Z" 2023-07-29T01:22:57.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.219 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-07-29T01:22:57.219 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:57.219 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:57.220 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:57.220 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:57.220 INFO:tasks.rgw_multisite_tests: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.220 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.221 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.221 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:57.221 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.221 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:57.222 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.222 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.222 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.222 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.222 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-07-29T01:22:57.223 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.223 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.223 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.223 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.224 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.224 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.224 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:57.224 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.225 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.225 INFO:tasks.rgw_multisite_tests: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.225 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.225 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.225 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.226 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.226 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.226 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.226 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.227 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.227 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:57.227 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.227 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.228 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-07-29T01:22:57.228 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.228 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.228 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.228 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-07-29T01:22:57.229 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.229 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.229 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.229 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.230 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:57.230 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.230 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.230 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.231 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.231 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.231 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.231 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.232 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.232 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.232 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.232 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:57.232 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.233 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:57.233 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-07-29T01:22:57.233 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-07-29T01:22:57.233 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.234 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:04:57Z" 2023-07-29T01:22:57.235 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:57.236 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:57.236 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:22:57.236 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.236 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.237 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:22:57.237 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.237 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:57.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:57.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:57.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:05:03.073+0000 7ff8ebe9d880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:05:03.074+0000 7ff8ebe9d880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:57.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:57.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:57.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:57.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:57.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:57.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 1, 2023-07-29T01:22:57.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:57.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:57.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:57.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "0.000000", 2023-07-29T01:22:57.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:57.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:57.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:57.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 1, 2023-07-29T01:22:57.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 1 2023-07-29T01:22:57.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:05:03Z" 2023-07-29T01:22:57.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.253 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-07-29T01:22:57.254 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:57.254 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:57.254 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:57.254 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:57.255 INFO:tasks.rgw_multisite_tests: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.255 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.255 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.255 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:57.256 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.256 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:57.256 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.256 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.256 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.257 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.257 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-07-29T01:22:57.257 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.257 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.258 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.258 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.258 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.258 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.258 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:57.259 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.259 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.259 INFO:tasks.rgw_multisite_tests: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.259 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.260 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.260 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.260 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.260 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.261 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.261 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.261 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.261 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:57.261 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.262 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.262 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-07-29T01:22:57.262 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.262 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.263 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.263 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-07-29T01:22:57.263 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.263 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.264 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.264 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.264 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:57.264 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.264 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.265 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.265 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.265 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.265 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.266 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.266 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.266 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.266 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.267 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:57.267 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.267 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:57.267 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-07-29T01:22:57.267 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-07-29T01:22:57.268 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.268 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:05:03Z" 2023-07-29T01:22:57.268 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:57.268 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:57.269 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:22:57.269 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.269 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.269 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:22:57.270 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.270 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:57.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:57.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:57.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:05:08.189+0000 7f4188213880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:05:08.190+0000 7f4188213880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:57.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:57.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:57.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:57.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:57.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:57.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 1, 2023-07-29T01:22:57.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:57.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:57.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:57.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "0.000000", 2023-07-29T01:22:57.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:57.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:57.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:57.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 1, 2023-07-29T01:22:57.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 1 2023-07-29T01:22:57.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:05:08Z" 2023-07-29T01:22:57.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.286 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-07-29T01:22:57.286 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:57.287 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:57.287 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:57.287 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:57.287 INFO:tasks.rgw_multisite_tests: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.288 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.288 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.288 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:57.288 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.289 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:57.289 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.289 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.289 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.289 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.290 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-07-29T01:22:57.290 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.290 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.290 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.291 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.291 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.291 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.291 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:57.292 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.292 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.292 INFO:tasks.rgw_multisite_tests: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.292 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.292 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.293 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.293 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.293 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.293 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.294 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.294 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.294 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:57.294 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.295 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.295 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-07-29T01:22:57.295 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.295 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.295 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.296 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-07-29T01:22:57.296 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.296 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.296 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.297 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.297 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:57.297 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.297 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.297 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.298 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.298 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.298 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.298 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.299 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.299 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.299 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.299 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:57.300 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.300 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:57.300 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-07-29T01:22:57.300 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-07-29T01:22:57.300 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.301 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:05:08Z" 2023-07-29T01:22:57.301 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:57.301 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:57.301 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:22:57.302 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.302 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.302 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:22:57.302 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.303 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:57.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:57.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:57.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:05:13.305+0000 7efdc0216880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:05:13.306+0000 7efdc0216880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:57.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:57.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:57.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:57.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:57.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:57.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 1, 2023-07-29T01:22:57.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:57.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:57.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:57.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "0.000000", 2023-07-29T01:22:57.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:57.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:57.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:57.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 1, 2023-07-29T01:22:57.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 1 2023-07-29T01:22:57.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:05:13Z" 2023-07-29T01:22:57.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.319 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-07-29T01:22:57.320 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:57.320 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:57.320 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:57.320 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:57.320 INFO:tasks.rgw_multisite_tests: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.321 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.321 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.321 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:57.321 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.322 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:57.322 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.322 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.322 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.323 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.323 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-07-29T01:22:57.323 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.323 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.323 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.324 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.324 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.324 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.324 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:57.325 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.325 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.325 INFO:tasks.rgw_multisite_tests: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.325 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.326 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.326 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.326 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.326 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.327 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.327 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.327 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.327 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:57.327 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.328 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.328 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-07-29T01:22:57.328 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.328 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.329 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.329 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-07-29T01:22:57.329 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.329 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.330 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.330 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.330 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:57.330 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.331 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.331 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.331 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.331 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.331 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.332 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.332 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.332 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.332 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.333 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:57.333 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.333 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:57.333 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-07-29T01:22:57.334 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-07-29T01:22:57.334 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.334 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:05:13Z" 2023-07-29T01:22:57.334 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:57.335 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:57.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:22:57.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:22:57.336 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.336 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:57.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:57.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:57.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:05:18.419+0000 7fde5d3af880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:05:18.420+0000 7fde5d3af880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:57.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:57.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:57.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:57.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:57.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:57.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 1, 2023-07-29T01:22:57.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:57.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:57.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:57.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "0.000000", 2023-07-29T01:22:57.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:57.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:57.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:57.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 1, 2023-07-29T01:22:57.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 1 2023-07-29T01:22:57.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:05:18Z" 2023-07-29T01:22:57.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.352 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-07-29T01:22:57.353 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:57.353 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:57.353 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:57.353 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:57.354 INFO:tasks.rgw_multisite_tests: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.354 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.354 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.354 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:57.355 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.355 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:57.355 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.355 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.355 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.356 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.356 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-07-29T01:22:57.356 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.356 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.357 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.357 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.357 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.357 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.358 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:57.358 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.358 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.358 INFO:tasks.rgw_multisite_tests: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.358 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.359 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.359 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.359 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.359 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.360 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.360 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.360 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.360 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:57.361 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.361 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.361 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-07-29T01:22:57.361 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.361 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.362 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.362 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-07-29T01:22:57.362 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.362 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.363 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.363 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.363 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:57.363 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.364 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.364 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.364 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.364 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.364 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.365 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.365 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.365 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.365 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.366 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:57.366 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.366 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:57.366 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-07-29T01:22:57.367 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-07-29T01:22:57.367 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.367 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:05:18Z" 2023-07-29T01:22:57.367 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:57.367 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:57.368 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:22:57.368 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.368 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.368 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:22:57.369 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.369 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:57.369 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi101.stdout: INFO: 2023-07-29T01:05:23.486+0000 7f0081a5e640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-07-29T01:22:57.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:57.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:57.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:05:23.555+0000 7f55e3013880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:05:23.556+0000 7f55e3013880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:57.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:57.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:57.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:57.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:57.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:57.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 1, 2023-07-29T01:22:57.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:57.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:57.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:57.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "0.000000", 2023-07-29T01:22:57.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:57.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:57.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:57.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 1, 2023-07-29T01:22:57.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 1 2023-07-29T01:22:57.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:05:23Z" 2023-07-29T01:22:57.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-07-29T01:22:57.386 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:57.386 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:57.386 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:57.387 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:57.387 INFO:tasks.rgw_multisite_tests: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.387 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.387 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.387 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:57.388 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.388 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:57.388 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.388 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.389 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.389 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.389 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-07-29T01:22:57.389 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.390 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.390 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.390 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.390 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.390 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.391 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:57.391 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.391 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.391 INFO:tasks.rgw_multisite_tests: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.392 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.392 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.392 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.392 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.393 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.393 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.393 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.393 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.393 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:57.394 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.394 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.394 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-07-29T01:22:57.394 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.395 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.395 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.395 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-07-29T01:22:57.395 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.395 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.396 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.396 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.396 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:57.396 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.397 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.397 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.397 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.397 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.398 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.398 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.398 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.398 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.398 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.399 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:57.399 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.399 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:57.399 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-07-29T01:22:57.400 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-07-29T01:22:57.400 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.400 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:05:23Z" 2023-07-29T01:22:57.400 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:57.401 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:57.401 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:22:57.401 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.401 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.401 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:22:57.402 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.402 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:57.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:57.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:57.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:05:28.673+0000 7f95e7826880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:05:28.674+0000 7f95e7826880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:57.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:57.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:57.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:57.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:57.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:57.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 1, 2023-07-29T01:22:57.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:57.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:57.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:57.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "0.000000", 2023-07-29T01:22:57.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:57.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:57.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:57.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 1, 2023-07-29T01:22:57.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 1 2023-07-29T01:22:57.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:05:28Z" 2023-07-29T01:22:57.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-07-29T01:22:57.419 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:57.419 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:57.419 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:57.419 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:57.419 INFO:tasks.rgw_multisite_tests: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.420 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.420 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.420 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:57.420 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.421 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:57.421 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.421 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.421 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.422 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.422 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-07-29T01:22:57.422 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.422 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.423 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.423 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.423 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.423 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.423 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:57.424 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.424 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.424 INFO:tasks.rgw_multisite_tests: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.424 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.425 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.425 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.425 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.425 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.426 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.426 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.426 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.426 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:57.427 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.427 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.427 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-07-29T01:22:57.427 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.427 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.428 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.428 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-07-29T01:22:57.428 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.428 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.429 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.429 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.429 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:57.429 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.430 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.430 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.430 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.430 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.430 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.431 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.431 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.431 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.431 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.432 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:57.432 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.432 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:57.432 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-07-29T01:22:57.433 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-07-29T01:22:57.433 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.433 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:05:28Z" 2023-07-29T01:22:57.433 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:57.433 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:57.434 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:22:57.434 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.434 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.434 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:22:57.435 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.435 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:57.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:57.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:57.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:05:33.795+0000 7f33b11a2880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:05:33.796+0000 7f33b11a2880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:57.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:57.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:57.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:57.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:57.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:57.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 1, 2023-07-29T01:22:57.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:57.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:57.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:57.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "0.000000", 2023-07-29T01:22:57.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:57.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:57.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:57.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 1, 2023-07-29T01:22:57.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 1 2023-07-29T01:22:57.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:05:33Z" 2023-07-29T01:22:57.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.451 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-07-29T01:22:57.452 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:57.452 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:57.452 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:57.452 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:57.452 INFO:tasks.rgw_multisite_tests: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.453 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.453 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.453 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:57.453 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.454 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:57.454 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.454 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.454 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.455 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.455 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-07-29T01:22:57.455 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.455 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.455 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.456 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.456 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.456 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.456 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:57.457 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.457 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.457 INFO:tasks.rgw_multisite_tests: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.457 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.458 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.458 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.458 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.458 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.458 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.459 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.459 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.459 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:57.459 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.460 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.460 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-07-29T01:22:57.460 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.460 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.461 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.461 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-07-29T01:22:57.461 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.461 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.461 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.462 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.462 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:57.462 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.462 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.463 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.463 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.463 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.463 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.463 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.464 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.464 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.464 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.464 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:57.465 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.465 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:57.465 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-07-29T01:22:57.465 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-07-29T01:22:57.466 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.466 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:05:33Z" 2023-07-29T01:22:57.466 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:57.466 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:57.466 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:22:57.467 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.467 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.467 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:22:57.467 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.468 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:57.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:57.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:57.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:05:38.910+0000 7fef91126880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:05:38.911+0000 7fef91126880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:57.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:57.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:57.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:57.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:57.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:57.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 1, 2023-07-29T01:22:57.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:57.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:57.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:57.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "0.000000", 2023-07-29T01:22:57.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:57.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:57.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:57.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 1, 2023-07-29T01:22:57.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 1 2023-07-29T01:22:57.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:05:38Z" 2023-07-29T01:22:57.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.484 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-07-29T01:22:57.484 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:57.485 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:57.485 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:57.485 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:57.485 INFO:tasks.rgw_multisite_tests: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.485 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.486 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.486 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:57.486 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.486 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:57.487 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.487 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.487 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.487 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.487 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-07-29T01:22:57.488 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.488 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.488 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.488 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.489 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.489 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.489 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:57.489 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.490 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.490 INFO:tasks.rgw_multisite_tests: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.490 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.490 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.491 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.491 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.491 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.491 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.491 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.492 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.492 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:57.492 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.492 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.493 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-07-29T01:22:57.493 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.493 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.493 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.494 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-07-29T01:22:57.494 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.494 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.494 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.494 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.495 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:57.495 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.495 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.495 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.496 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.496 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.496 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.496 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.497 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.497 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.497 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.497 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:57.497 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.498 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:57.498 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-07-29T01:22:57.498 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-07-29T01:22:57.498 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.499 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:05:38Z" 2023-07-29T01:22:57.499 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:57.499 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:57.499 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:22:57.500 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.500 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.500 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:22:57.500 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.500 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:57.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:57.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:57.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:05:44.024+0000 7f96da00f880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:05:44.025+0000 7f96da00f880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:57.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:57.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:57.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:57.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:57.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:57.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 1, 2023-07-29T01:22:57.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:57.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:57.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:57.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "0.000000", 2023-07-29T01:22:57.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:57.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:57.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:57.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 1, 2023-07-29T01:22:57.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 1 2023-07-29T01:22:57.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:05:44Z" 2023-07-29T01:22:57.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-07-29T01:22:57.517 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:57.517 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:57.518 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:57.518 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:57.518 INFO:tasks.rgw_multisite_tests: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.518 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.519 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.519 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:57.519 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.519 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:57.519 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.520 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.520 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.520 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.520 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-07-29T01:22:57.521 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.521 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.521 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.521 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.522 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.522 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.522 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:57.522 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.522 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.523 INFO:tasks.rgw_multisite_tests: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.523 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.523 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.523 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.524 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.524 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.524 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.524 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.525 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.525 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:57.525 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.525 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.526 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-07-29T01:22:57.526 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.526 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.526 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.526 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-07-29T01:22:57.527 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.527 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.527 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.527 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.528 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:57.528 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.528 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.528 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.529 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.529 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.529 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.529 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.529 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.530 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.530 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.530 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:57.530 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.531 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:57.531 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-07-29T01:22:57.531 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-07-29T01:22:57.531 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.532 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:05:44Z" 2023-07-29T01:22:57.532 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:57.532 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:57.532 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:22:57.532 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.533 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.533 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:22:57.533 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.533 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:57.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:57.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:57.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:05:49.142+0000 7ff9cb11f880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:05:49.143+0000 7ff9cb11f880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:57.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:57.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:57.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:57.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:57.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:57.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 1, 2023-07-29T01:22:57.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:57.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:57.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:57.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "0.000000", 2023-07-29T01:22:57.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:57.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:57.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:57.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 1, 2023-07-29T01:22:57.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 1 2023-07-29T01:22:57.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:05:49Z" 2023-07-29T01:22:57.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.550 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-07-29T01:22:57.550 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:57.550 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:57.551 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:57.551 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:57.551 INFO:tasks.rgw_multisite_tests: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.551 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.552 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.552 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:57.552 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.552 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:57.552 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.553 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.553 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.553 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.553 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-07-29T01:22:57.554 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.554 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.554 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.554 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.555 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.555 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.555 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:57.555 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.555 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.556 INFO:tasks.rgw_multisite_tests: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.556 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.556 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.556 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.557 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.557 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.557 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.557 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.558 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.558 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:57.558 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.558 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.558 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-07-29T01:22:57.559 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.559 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.559 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.559 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-07-29T01:22:57.560 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.560 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.560 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.560 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.561 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:57.561 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.561 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.561 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.562 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.562 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.562 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.562 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.562 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.563 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.563 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.563 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:57.563 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.564 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:57.564 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-07-29T01:22:57.564 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-07-29T01:22:57.564 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.564 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:05:49Z" 2023-07-29T01:22:57.565 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:57.565 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:57.565 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:22:57.565 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.566 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.566 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:22:57.566 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.566 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:57.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:57.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:57.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:05:54.260+0000 7fc354321880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:05:54.261+0000 7fc354321880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:57.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:57.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:57.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:57.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:57.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:57.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 1, 2023-07-29T01:22:57.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:57.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:57.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:57.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "0.000000", 2023-07-29T01:22:57.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:57.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:57.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:57.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 1, 2023-07-29T01:22:57.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 1 2023-07-29T01:22:57.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:05:54Z" 2023-07-29T01:22:57.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-07-29T01:22:57.583 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:57.583 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:57.584 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:57.584 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:57.584 INFO:tasks.rgw_multisite_tests: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.584 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.584 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.585 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:57.585 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.585 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:57.585 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.586 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.586 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.586 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.586 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-07-29T01:22:57.587 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.587 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.587 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.587 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.587 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.588 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.588 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:57.588 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.588 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.589 INFO:tasks.rgw_multisite_tests: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.589 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.589 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.589 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.590 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.590 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.590 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.590 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.590 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.591 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:57.591 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.591 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.591 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-07-29T01:22:57.592 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.592 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.592 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.592 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-07-29T01:22:57.593 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.593 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.593 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.593 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.593 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:57.594 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.594 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.594 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.594 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.595 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.595 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.595 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.595 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.596 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.596 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.596 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:57.597 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.597 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:57.598 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-07-29T01:22:57.598 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-07-29T01:22:57.599 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.599 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:05:54Z" 2023-07-29T01:22:57.600 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:57.600 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:57.601 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:22:57.601 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.602 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.602 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:22:57.602 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.602 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:57.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:57.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:57.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:05:59.403+0000 7f1e99c67880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:05:59.404+0000 7f1e99c67880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:57.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:57.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:57.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:57.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:57.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:57.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 1, 2023-07-29T01:22:57.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:57.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:57.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:57.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "0.000000", 2023-07-29T01:22:57.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:57.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:57.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:57.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 1, 2023-07-29T01:22:57.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 1 2023-07-29T01:22:57.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:05:59Z" 2023-07-29T01:22:57.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.620 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-07-29T01:22:57.621 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:57.621 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:57.621 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:57.622 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:57.622 INFO:tasks.rgw_multisite_tests: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.623 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.623 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.623 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:57.623 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.624 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:57.624 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.624 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.624 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.625 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.625 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-07-29T01:22:57.625 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.626 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.626 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.626 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.626 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.627 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.627 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:57.627 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.627 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.628 INFO:tasks.rgw_multisite_tests: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.628 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.628 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.629 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.629 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.629 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.629 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.630 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.630 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.630 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:57.631 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.631 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.631 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-07-29T01:22:57.631 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.632 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.632 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.633 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-07-29T01:22:57.633 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.633 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.634 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.634 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.634 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:57.634 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.634 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.635 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.635 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.635 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.635 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.636 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.636 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.636 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.636 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.637 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:57.637 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.637 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:57.637 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-07-29T01:22:57.638 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-07-29T01:22:57.638 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.638 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:05:59Z" 2023-07-29T01:22:57.638 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:57.638 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:57.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:22:57.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.639 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:22:57.640 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.640 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:57.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:57.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:57.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:06:04.530+0000 7f4072f35880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:06:04.531+0000 7f4072f35880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:57.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:57.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:57.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:57.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:57.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:57.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 1, 2023-07-29T01:22:57.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:57.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:57.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:57.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "0.000000", 2023-07-29T01:22:57.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:57.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:57.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:57.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 1, 2023-07-29T01:22:57.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 1 2023-07-29T01:22:57.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:06:04Z" 2023-07-29T01:22:57.664 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.664 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-07-29T01:22:57.665 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:57.665 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:57.666 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:57.666 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:57.666 INFO:tasks.rgw_multisite_tests: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.666 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.667 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.667 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:57.667 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.667 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:57.668 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.668 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.668 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.669 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.669 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-07-29T01:22:57.669 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.669 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.670 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.670 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.670 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.670 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.670 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:57.671 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.671 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.671 INFO:tasks.rgw_multisite_tests: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.671 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.672 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.672 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.672 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.672 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.673 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.673 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.673 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.673 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:57.674 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.674 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.674 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-07-29T01:22:57.674 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.675 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.675 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.675 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-07-29T01:22:57.676 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.676 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.677 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.677 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.678 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:57.678 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.679 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.679 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.680 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.681 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.681 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.682 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.682 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.682 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.683 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.683 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:57.683 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.683 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:57.684 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-07-29T01:22:57.684 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-07-29T01:22:57.684 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.685 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:06:04Z" 2023-07-29T01:22:57.685 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:57.685 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:57.685 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:22:57.685 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.686 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.686 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:22:57.686 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.687 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:57.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:57.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:57.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:06:09.646+0000 7ffba9216880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:06:09.647+0000 7ffba9216880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:57.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:57.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:57.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:57.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:57.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:57.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 1, 2023-07-29T01:22:57.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.695 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:57.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:57.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:57.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "0.000000", 2023-07-29T01:22:57.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:57.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:57.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:57.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 1, 2023-07-29T01:22:57.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 1 2023-07-29T01:22:57.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:06:09Z" 2023-07-29T01:22:57.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.709 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-07-29T01:22:57.709 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:57.709 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:57.709 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:57.710 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:57.710 INFO:tasks.rgw_multisite_tests: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.710 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.711 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.711 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:57.712 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.712 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:57.712 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.712 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.713 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.713 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.713 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-07-29T01:22:57.713 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.714 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.714 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.714 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.714 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.715 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.715 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:57.715 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.715 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.716 INFO:tasks.rgw_multisite_tests: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.716 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.717 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.718 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.718 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.719 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.719 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.720 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.721 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.721 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:57.722 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.722 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.723 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-07-29T01:22:57.723 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.724 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.724 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.724 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-07-29T01:22:57.725 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.725 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.726 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.726 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.727 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:57.727 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.727 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.728 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.728 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.728 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.729 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.729 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.729 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.730 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.730 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.731 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:57.731 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.732 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:57.732 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-07-29T01:22:57.733 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-07-29T01:22:57.733 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.733 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:06:09Z" 2023-07-29T01:22:57.734 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:57.734 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:57.734 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:22:57.735 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.735 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.735 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:22:57.735 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.735 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:57.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:57.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:57.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:06:14.761+0000 7fc70a410880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:06:14.762+0000 7fc70a410880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:57.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:57.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:57.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:57.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:57.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:57.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 1, 2023-07-29T01:22:57.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:57.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:57.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:57.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "0.000000", 2023-07-29T01:22:57.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:57.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:57.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:57.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 1, 2023-07-29T01:22:57.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 1 2023-07-29T01:22:57.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:06:14Z" 2023-07-29T01:22:57.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.755 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-07-29T01:22:57.755 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:57.756 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:57.756 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:57.756 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:57.757 INFO:tasks.rgw_multisite_tests: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.757 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.757 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.757 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:57.757 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.758 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:57.758 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.759 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.759 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.759 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.759 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-07-29T01:22:57.760 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.760 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.761 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.761 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.761 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.761 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.762 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:57.762 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.762 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.763 INFO:tasks.rgw_multisite_tests: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.763 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.763 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.764 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.764 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.765 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.765 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.765 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.765 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.766 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:57.766 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.766 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.766 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-07-29T01:22:57.767 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.767 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.767 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.767 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-07-29T01:22:57.768 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.768 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.768 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.769 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.769 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:57.769 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.769 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.770 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.770 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.771 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.771 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.771 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.772 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.772 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.773 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.773 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:57.773 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.773 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:57.774 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-07-29T01:22:57.774 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-07-29T01:22:57.775 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.775 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:06:14Z" 2023-07-29T01:22:57.776 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:57.776 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:57.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:22:57.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.778 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.779 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:22:57.779 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.780 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:57.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:57.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:57.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:06:19.879+0000 7f1b18611880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:06:19.880+0000 7f1b18611880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:57.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:57.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:57.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:57.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:57.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:57.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 1, 2023-07-29T01:22:57.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:57.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.791 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:57.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:57.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "0.000000", 2023-07-29T01:22:57.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:57.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:57.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:57.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 1, 2023-07-29T01:22:57.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 1 2023-07-29T01:22:57.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:06:19Z" 2023-07-29T01:22:57.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.803 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-07-29T01:22:57.804 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:57.804 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:57.805 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:57.805 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:57.806 INFO:tasks.rgw_multisite_tests: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.807 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.807 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.808 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:57.808 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.809 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:57.809 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.809 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.810 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.810 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.810 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-07-29T01:22:57.811 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.811 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.811 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.811 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.812 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.812 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.813 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:57.813 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.813 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.814 INFO:tasks.rgw_multisite_tests: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.814 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.814 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.815 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.815 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.815 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.815 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.816 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.816 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.816 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:57.816 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.817 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.817 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-07-29T01:22:57.817 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.817 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.818 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.818 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-07-29T01:22:57.818 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.818 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.819 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.819 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.819 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:57.819 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.819 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.820 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.820 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.820 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.820 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.821 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.821 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.821 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.822 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.822 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:57.822 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.823 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:57.823 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-07-29T01:22:57.823 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-07-29T01:22:57.824 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.824 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:06:19Z" 2023-07-29T01:22:57.824 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:57.825 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:57.825 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:22:57.825 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.825 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.826 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:22:57.826 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.826 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:57.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:57.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:57.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:06:24.992+0000 7f6a39934880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:06:24.993+0000 7f6a39934880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:57.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:57.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:57.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:57.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:57.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:57.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 1, 2023-07-29T01:22:57.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:57.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:57.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:57.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "0.000000", 2023-07-29T01:22:57.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:57.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:57.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:57.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 1, 2023-07-29T01:22:57.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 1 2023-07-29T01:22:57.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:06:24Z" 2023-07-29T01:22:57.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.850 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-07-29T01:22:57.850 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:57.850 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:57.850 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:57.851 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:57.851 INFO:tasks.rgw_multisite_tests: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.851 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.852 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.852 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:57.852 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.852 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:57.853 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.853 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.853 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.853 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.854 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-07-29T01:22:57.854 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.854 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.855 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.855 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.855 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.855 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.856 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:57.856 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.856 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.857 INFO:tasks.rgw_multisite_tests: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.857 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.857 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.857 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.858 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.858 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.858 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.859 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.859 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.860 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:57.860 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.861 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.861 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-07-29T01:22:57.861 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.861 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.862 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.862 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-07-29T01:22:57.862 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.862 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.862 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.863 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.863 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:57.864 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.864 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.864 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.864 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.864 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.865 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.865 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.865 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.865 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.866 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.866 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:57.866 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.866 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:57.867 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-07-29T01:22:57.867 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-07-29T01:22:57.867 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.867 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:06:24Z" 2023-07-29T01:22:57.868 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:57.868 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:57.868 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:22:57.868 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.869 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.869 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:22:57.869 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.869 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:57.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:57.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:57.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:06:30.115+0000 7fac2372a880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:06:30.116+0000 7fac2372a880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:57.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:57.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:57.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:57.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:57.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:57.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 1, 2023-07-29T01:22:57.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:57.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:57.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:57.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "0.000000", 2023-07-29T01:22:57.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:57.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:57.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:57.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 1, 2023-07-29T01:22:57.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 1 2023-07-29T01:22:57.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:06:30Z" 2023-07-29T01:22:57.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.886 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-07-29T01:22:57.886 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:57.886 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:57.886 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:57.887 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:57.887 INFO:tasks.rgw_multisite_tests: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.887 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.887 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.888 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:57.888 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.888 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:57.888 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.889 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.889 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.889 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.889 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-07-29T01:22:57.889 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.890 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.890 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.890 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.890 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.891 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.891 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:57.891 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.891 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.892 INFO:tasks.rgw_multisite_tests: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.892 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.892 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.892 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.892 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.893 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.893 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.893 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.893 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.894 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:57.894 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.894 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.894 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-07-29T01:22:57.895 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.895 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.895 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.895 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-07-29T01:22:57.895 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.896 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.896 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.896 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.896 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:57.897 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.897 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.897 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.897 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.898 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.898 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.898 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.898 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.898 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.899 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.899 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:57.899 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.899 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:57.900 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-07-29T01:22:57.900 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-07-29T01:22:57.900 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.900 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:06:30Z" 2023-07-29T01:22:57.901 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:57.901 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:57.901 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:22:57.901 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.902 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.902 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:22:57.902 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.902 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:57.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:57.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:57.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:06:35.229+0000 7f42857a2880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:06:35.230+0000 7f42857a2880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:57.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:57.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:57.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:57.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:57.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:57.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 1, 2023-07-29T01:22:57.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:57.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:57.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:57.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "0.000000", 2023-07-29T01:22:57.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:57.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:57.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:57.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 1, 2023-07-29T01:22:57.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 1 2023-07-29T01:22:57.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:06:35Z" 2023-07-29T01:22:57.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.919 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-07-29T01:22:57.919 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:57.919 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:57.919 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:57.920 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:57.920 INFO:tasks.rgw_multisite_tests: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.920 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.920 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.921 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:57.921 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.921 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:57.921 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.922 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.922 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.922 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.922 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-07-29T01:22:57.923 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.923 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.923 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.923 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.923 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.924 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.924 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:57.924 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.924 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.925 INFO:tasks.rgw_multisite_tests: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.925 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.925 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.925 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.926 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.926 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.926 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.926 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.926 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.927 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:57.927 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.927 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.927 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-07-29T01:22:57.928 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.928 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.928 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.928 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-07-29T01:22:57.929 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.929 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.929 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.929 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.930 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:57.930 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.930 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.930 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.930 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.931 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.931 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.931 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.931 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.932 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.932 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.932 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:57.932 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.933 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:57.933 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-07-29T01:22:57.933 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-07-29T01:22:57.933 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.933 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:06:35Z" 2023-07-29T01:22:57.934 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:57.934 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:57.934 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:22:57.934 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.935 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.935 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:22:57.935 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.935 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:57.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:57.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:57.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:06:40.349+0000 7f866a328880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:06:40.350+0000 7f866a328880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:57.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:57.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:57.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:57.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:57.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:57.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 1, 2023-07-29T01:22:57.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:57.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:57.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:57.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "0.000000", 2023-07-29T01:22:57.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:57.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:57.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:57.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 1, 2023-07-29T01:22:57.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 1 2023-07-29T01:22:57.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:06:40Z" 2023-07-29T01:22:57.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.952 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-07-29T01:22:57.953 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:57.953 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:57.953 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:57.953 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:57.954 INFO:tasks.rgw_multisite_tests: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.954 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.954 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.954 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:57.955 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.955 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:57.955 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.955 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.956 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.956 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.956 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-07-29T01:22:57.956 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.956 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.957 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.957 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.957 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.957 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.958 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:57.958 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.958 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.958 INFO:tasks.rgw_multisite_tests: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.959 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.959 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.959 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.959 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.959 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.960 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.960 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.960 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.960 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:57.961 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.961 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.961 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-07-29T01:22:57.961 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.962 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.962 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.962 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-07-29T01:22:57.962 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.962 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.963 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.963 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.963 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:57.963 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.964 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.964 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.964 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.964 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.965 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.965 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.965 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.965 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.965 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.966 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:57.966 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.966 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:57.966 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-07-29T01:22:57.967 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-07-29T01:22:57.967 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.967 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:06:40Z" 2023-07-29T01:22:57.967 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:57.968 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:57.968 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:22:57.968 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.968 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:57.969 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:22:57.969 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.969 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:57.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:57.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:57.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:57.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:06:45.466+0000 7fe45b210880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:06:45.467+0000 7fe45b210880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:57.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:57.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:57.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:57.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:57.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:57.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:57.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 1, 2023-07-29T01:22:57.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:57.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:57.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:57.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "0.000000", 2023-07-29T01:22:57.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:57.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:57.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:57.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:57.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:57.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:57.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:57.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:57.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:57.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:57.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 1, 2023-07-29T01:22:57.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 1 2023-07-29T01:22:57.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:57.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:06:45Z" 2023-07-29T01:22:57.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:57.986 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-07-29T01:22:57.986 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:57.986 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:57.986 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:57.986 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:57.987 INFO:tasks.rgw_multisite_tests: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:57.987 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.987 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.987 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:57.988 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.988 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:57.988 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.988 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.989 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:57.989 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.989 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-07-29T01:22:57.989 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.989 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:57.990 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.990 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.990 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.990 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.991 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:57.991 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.991 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.991 INFO:tasks.rgw_multisite_tests: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:57.992 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.992 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.992 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.992 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:57.993 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.993 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.993 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.993 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.993 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:57.994 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.994 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.994 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-07-29T01:22:57.994 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.995 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.995 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.995 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-07-29T01:22:57.995 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.996 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.996 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.996 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:57.996 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:57.997 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:57.997 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:57.997 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:57.997 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:57.998 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:57.998 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:57.998 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:57.998 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:57.998 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.999 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:57.999 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:57.999 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:57.999 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:58.000 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-07-29T01:22:58.000 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-07-29T01:22:58.000 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.000 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:06:45Z" 2023-07-29T01:22:58.001 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:58.001 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.001 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:22:58.001 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:58.002 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:58.002 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:22:58.002 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.002 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:58.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:58.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:58.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:06:50.582+0000 7f9043ac0880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:58.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:06:50.583+0000 7f9043ac0880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:58.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:58.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:58.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:58.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:58.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:58.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:58.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:58.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:58.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:58.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 1, 2023-07-29T01:22:58.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:58.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:58.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:58.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:58.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:58.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:58.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:58.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:58.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "0.000000", 2023-07-29T01:22:58.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:58.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:58.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:58.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:58.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:58.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 1, 2023-07-29T01:22:58.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 1 2023-07-29T01:22:58.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:06:50Z" 2023-07-29T01:22:58.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.019 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-07-29T01:22:58.019 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:58.019 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:58.019 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:58.020 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:58.020 INFO:tasks.rgw_multisite_tests: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:58.020 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.020 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.021 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:58.021 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.021 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:58.021 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.022 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:58.022 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:58.022 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.022 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-07-29T01:22:58.023 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.023 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:58.023 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.023 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.023 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.024 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.024 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:58.024 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.024 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:58.025 INFO:tasks.rgw_multisite_tests: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:58.025 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.025 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.025 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.026 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:58.026 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.026 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.026 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.026 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.027 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:58.027 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.027 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:58.027 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-07-29T01:22:58.028 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.028 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.028 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.028 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-07-29T01:22:58.029 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.029 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.029 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.029 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.029 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:58.030 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.030 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:58.030 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:58.030 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.031 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.031 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.031 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:58.031 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.032 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.032 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.032 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:58.032 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.032 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:58.033 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-07-29T01:22:58.033 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-07-29T01:22:58.033 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.033 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:06:50Z" 2023-07-29T01:22:58.034 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:58.034 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.034 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:22:58.034 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:58.035 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:58.035 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:22:58.035 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi101.stdout: INFO: 2023-07-29T01:06:53.495+0000 7f5a3155a640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-07-29T01:22:58.035 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.035 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:58.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:58.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:58.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:06:55.697+0000 7f0cdec67880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:58.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:06:55.698+0000 7f0cdec67880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:58.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:58.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:58.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:58.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:58.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:58.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:58.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:58.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:58.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:58.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 1, 2023-07-29T01:22:58.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:58.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:58.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:58.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:58.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:58.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:58.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:58.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:58.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "0.000000", 2023-07-29T01:22:58.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:58.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:58.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:58.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:58.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:58.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 1, 2023-07-29T01:22:58.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 1 2023-07-29T01:22:58.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:06:55Z" 2023-07-29T01:22:58.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.052 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-07-29T01:22:58.052 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:58.053 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:58.053 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:58.053 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:58.053 INFO:tasks.rgw_multisite_tests: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:58.054 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.054 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.054 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:58.054 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.055 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:58.055 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.055 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:58.055 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:58.055 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.056 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-07-29T01:22:58.056 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.056 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:58.056 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.057 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.057 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.057 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.057 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:58.058 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.058 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:58.058 INFO:tasks.rgw_multisite_tests: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:58.058 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.058 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.059 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.059 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:58.059 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.059 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.060 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.060 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.060 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:58.060 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.061 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:58.061 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-07-29T01:22:58.061 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.061 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.061 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.062 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-07-29T01:22:58.062 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.062 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.062 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.063 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.063 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:58.063 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.063 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:58.064 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:58.064 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.064 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.064 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.064 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:58.065 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.065 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.065 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.065 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:58.066 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.066 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:58.066 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-07-29T01:22:58.066 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-07-29T01:22:58.067 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.067 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:06:55Z" 2023-07-29T01:22:58.067 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:58.067 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.067 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:22:58.068 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:58.068 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:58.068 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:22:58.068 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.069 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:58.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:58.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:58.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:07:00.813+0000 7f7c29610880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:58.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:07:00.814+0000 7f7c29610880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:58.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:58.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:58.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:58.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:58.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:58.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:58.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:58.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:58.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:58.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 1, 2023-07-29T01:22:58.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:58.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:58.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:58.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:58.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:58.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:58.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:58.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:58.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "0.000000", 2023-07-29T01:22:58.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:58.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:58.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:58.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:58.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:58.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 1, 2023-07-29T01:22:58.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 1 2023-07-29T01:22:58.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:07:00Z" 2023-07-29T01:22:58.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.085 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-07-29T01:22:58.086 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:58.086 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:58.086 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:58.086 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:58.086 INFO:tasks.rgw_multisite_tests: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:58.087 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.087 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.087 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:58.087 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.088 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:58.088 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.088 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:58.088 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:58.089 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.089 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-07-29T01:22:58.089 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.089 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:58.089 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.090 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.090 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.090 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.090 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:58.091 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.091 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:58.091 INFO:tasks.rgw_multisite_tests: "marker": "1_1690592679.844113_1377.1", 2023-07-29T01:22:58.091 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.091 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.092 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.092 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:04:39.844113Z", 2023-07-29T01:22:58.092 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.092 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.093 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.093 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.093 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:58.093 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.094 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:58.094 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-07-29T01:22:58.094 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.094 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.095 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.095 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-07-29T01:22:58.095 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.095 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.095 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.096 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.096 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:58.096 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.096 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:58.097 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:58.097 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.097 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.097 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.098 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:58.098 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.098 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.098 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.099 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:58.099 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.099 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:58.099 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-07-29T01:22:58.099 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-07-29T01:22:58.100 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.100 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:07:00Z" 2023-07-29T01:22:58.100 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:58.100 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.101 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592679.844113_1377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:39.844113Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:22:58.101 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:58.101 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592679.844113_1377.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:58.101 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1690592692.567652_1462.1 target=1_1690592679.844113_1377.1 2023-07-29T01:22:58.102 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.102 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:58.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:58.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:58.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:07:05.928+0000 7fb7e0124880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:58.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:07:05.929+0000 7fb7e0124880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:58.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:58.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:58.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:58.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:58.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:58.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:58.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:58.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:58.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:58.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 1, 2023-07-29T01:22:58.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:58.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:58.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:58.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690592692.567652_1462.1", 2023-07-29T01:22:58.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:04:52.567652Z", 2023-07-29T01:22:58.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:58.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:58.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:58.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "0.000000", 2023-07-29T01:22:58.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:58.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:58.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:58.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:58.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:58.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 1, 2023-07-29T01:22:58.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 1 2023-07-29T01:22:58.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:07:05Z" 2023-07-29T01:22:58.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.118 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-07-29T01:22:58.119 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:58.119 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:58.119 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:58.119 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:58.119 INFO:tasks.rgw_multisite_tests: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:58.120 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.120 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.120 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:58.120 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.121 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:58.121 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.121 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:58.121 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:58.122 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.122 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-07-29T01:22:58.122 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.122 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:58.122 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.123 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.123 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.123 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.123 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:58.124 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.124 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:58.124 INFO:tasks.rgw_multisite_tests: "marker": "1_1690592692.567652_1462.1", 2023-07-29T01:22:58.124 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.125 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.125 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.125 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:04:52.567652Z", 2023-07-29T01:22:58.125 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.125 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.126 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.126 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.126 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:58.126 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.127 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:58.127 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-07-29T01:22:58.127 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.127 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.128 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.128 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-07-29T01:22:58.128 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.128 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.128 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.129 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.129 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:58.129 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.129 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:58.130 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:58.130 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.130 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.130 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.131 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:58.131 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.131 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.131 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.131 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:58.132 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.132 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:58.132 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-07-29T01:22:58.132 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-07-29T01:22:58.133 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.133 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:07:05Z" 2023-07-29T01:22:58.133 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:58.133 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.134 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690592692.567652_1462.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:04:52.567652Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:22:58.134 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:58.134 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690592692.567652_1462.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:58.134 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-07-29T01:22:58.134 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.135 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:58.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:58.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:58.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: [ 2023-07-29T01:22:58.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000051393", 2023-07-29T01:22:58.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "last_update": "2023-07-29T01:04:52.529981Z" 2023-07-29T01:22:58.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000044739", 2023-07-29T01:22:58.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "last_update": "2023-07-29T00:58:23.964578Z" 2023-07-29T01:22:58.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.138 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000045520", 2023-07-29T01:22:58.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "last_update": "2023-07-29T01:06:23.586689Z" 2023-07-29T01:22:58.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.139 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:22:58.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "last_update": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:22:58.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.140 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.140 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: data markers for zone=a1 markers={0: '00000000000000000000:00000000000000051393', 1: '00000000000000000000:00000000000000044739', 2: '00000000000000000000:00000000000000045520', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:22:58.141 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting data checkpoint for target_zone=a2 source_zone=a1 2023-07-29T01:22:58.141 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.141 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:58.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:58.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:58.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:58.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:58.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:58.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:58.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "instance_id": 7555809319554727162 2023-07-29T01:22:58.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:58.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:58.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000051393", 2023-07-29T01:22:58.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 55, 2023-07-29T01:22:58.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:04:52.529981Z" 2023-07-29T01:22:58.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:58.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000044739", 2023-07-29T01:22:58.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 9, 2023-07-29T01:22:58.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:58:23.964578Z" 2023-07-29T01:22:58.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:58.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000045520", 2023-07-29T01:22:58.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.150 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:06:23.586689Z" 2023-07-29T01:22:58.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:58.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:22:58.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:22:58.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:58.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 64, 2023-07-29T01:22:58.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 64 2023-07-29T01:22:58.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:07:06Z" 2023-07-29T01:22:58.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.156 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-07-29T01:22:58.156 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:58.156 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:58.157 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:58.157 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:58.157 INFO:tasks.rgw_multisite_tests: "instance_id": 7555809319554727162 2023-07-29T01:22:58.157 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.157 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:58.158 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.158 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:58.158 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.158 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.159 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000051393", 2023-07-29T01:22:58.159 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.159 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-07-29T01:22:58.159 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.160 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:04:52.529981Z" 2023-07-29T01:22:58.160 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.160 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.160 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.160 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:58.161 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.161 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.161 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000044739", 2023-07-29T01:22:58.161 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.162 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-07-29T01:22:58.162 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.162 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:58:23.964578Z" 2023-07-29T01:22:58.162 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.163 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.163 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.163 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:58.163 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.163 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.164 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045520", 2023-07-29T01:22:58.164 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.164 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.164 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.165 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:06:23.586689Z" 2023-07-29T01:22:58.165 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.165 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.165 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.166 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:58.166 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.166 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.166 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031493", 2023-07-29T01:22:58.166 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.167 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.167 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.167 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:58:25.189790Z" 2023-07-29T01:22:58.167 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.168 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.168 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:58.168 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.168 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:58.169 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-07-29T01:22:58.169 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-07-29T01:22:58.169 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.169 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:07:06Z" 2023-07-29T01:22:58.169 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:58.170 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.170 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051393', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:04:52.529981Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000044739', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T00:58:23.964578Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045520', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:06:23.586689Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:58:25.189790Z'}}] 2023-07-29T01:22:58.170 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000051393', 1: '00000000000000000000:00000000000000044739', 2: '00000000000000000000:00000000000000045520', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:22:58.170 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000051393', 1: '00000000000000000000:00000000000000044739', 2: '00000000000000000000:00000000000000045520', 3: '00000000000000000000:00000000000000031493'} 2023-07-29T01:22:58.171 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finished data checkpoint for target_zone=a2 source_zone=a1 2023-07-29T01:22:58.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-66/ 2023-07-29T01:22:58.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-66/ 2023-07-29T01:22:58.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:58.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-66/ 2023-07-29T01:22:58.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:58.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:58.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:58.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:58.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:58.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-29T01:22:58.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:58.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:58.174 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:58.174 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.174 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.174 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 01:07:06 GMT 2023-07-29T01:22:58.175 INFO:tasks.rgw_multisite_tests:/hiiqos-66/ 2023-07-29T01:22:58.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:58.175 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:K26aV3Uo69IWzbElfPvaDaz0Gpk= 2023-07-29T01:22:58.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:07:06 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:K26aV3Uo69IWzbElfPvaDaz0Gpk=', 'Content-Length': '0'} 2023-07-29T01:22:58.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006389ac04b8d3b524-0064c4663f-233c-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:58.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-66/obj1 2023-07-29T01:22:58.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-66/obj1 2023-07-29T01:22:58.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:58.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-66/obj1 2023-07-29T01:22:58.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:58.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:58.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:58.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:58.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:58.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:58.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:58.178 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:58.179 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.179 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.179 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 01:07:11 GMT 2023-07-29T01:22:58.179 INFO:tasks.rgw_multisite_tests:/hiiqos-66/obj1 2023-07-29T01:22:58.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:58.180 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:TMGyP0SD4cuVaZw+5fJv43Vtp4c= 2023-07-29T01:22:58.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:07:11 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TMGyP0SD4cuVaZw+5fJv43Vtp4c=', 'Content-Length': '0'} 2023-07-29T01:22:58.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000c8116c313f69b6de-0064c4663f-233c-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 01:07:11 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:58.180 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-07-29T01:22:58.181 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.181 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-07-29T01:22:58.182 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_multiple_buckets_to_single: 2023-07-29T01:22:58.182 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-07-29T01:22:58.183 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-07-29T01:22:58.183 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_61140164d2579c30ffff3970615e0f82e40c6470/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-07-29T01:22:58.184 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-07-29T01:22:58.184 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_0484097b0572e96f42cf6402cbe0ac8dcb046577/qa/../src/test/rgw/rgw_multi/tests.py", line 2697, in test_sync_multiple_buckets_to_single 2023-07-29T01:22:58.184 INFO:tasks.rgw_multisite_tests: check_objects_exist(bucket, objnamesB, content) 2023-07-29T01:22:58.184 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_0484097b0572e96f42cf6402cbe0ac8dcb046577/qa/../src/test/rgw/rgw_multi/tests.py", line 2047, in check_objects_exist 2023-07-29T01:22:58.185 INFO:tasks.rgw_multisite_tests: check_object_exists(bucket, objname, content) 2023-07-29T01:22:58.185 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_0484097b0572e96f42cf6402cbe0ac8dcb046577/qa/../src/test/rgw/rgw_multi/tests.py", line 2041, in check_object_exists 2023-07-29T01:22:58.185 INFO:tasks.rgw_multisite_tests: assert_not_equal(k, None) 2023-07-29T01:22:58.185 INFO:tasks.rgw_multisite_tests:AssertionError: None == None 2023-07-29T01:22:58.186 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-07-29T01:22:58.186 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.186 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:58.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:58.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:58.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T01:22:58.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: [ 2023-07-29T01:22:58.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:58.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "last_update": "2023-07-29T00:40:17.836402Z" 2023-07-29T01:22:58.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690593413.808092_1766.1", 2023-07-29T01:22:58.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "last_update": "2023-07-29T01:16:53.808092Z" 2023-07-29T01:22:58.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:58.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "last_update": "0.000000" 2023-07-29T01:22:58.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:58.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "last_update": "2023-07-29T00:40:12.811406Z" 2023-07-29T01:22:58.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.191 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1690591217.836402_294.1', 1: '1_1690593413.808092_1766.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:58.192 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-07-29T01:22:58.192 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.192 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:58.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:58.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:58.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:16:54.089+0000 7fdd10122880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:58.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:16:54.090+0000 7fdd10122880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:58.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:58.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:58.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:58.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:58.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:58.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:58.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:58.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:58.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:58.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 1, 2023-07-29T01:22:58.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:58.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:58.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:58.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690593387.549974_1756.1", 2023-07-29T01:22:58.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:16:27.549974Z", 2023-07-29T01:22:58.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:58.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:58.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:58.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "0.000000", 2023-07-29T01:22:58.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.203 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:58.204 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:58.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:58.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:58.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:58.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 1, 2023-07-29T01:22:58.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 1 2023-07-29T01:22:58.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:16:54Z" 2023-07-29T01:22:58.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.209 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-07-29T01:22:58.209 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:58.209 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:58.209 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:58.209 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:58.210 INFO:tasks.rgw_multisite_tests: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:58.210 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.210 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.210 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:58.211 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.211 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:58.211 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.211 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:58.211 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:58.212 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.212 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-07-29T01:22:58.212 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.212 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:58.213 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.213 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.213 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.213 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.214 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:58.214 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.214 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:58.214 INFO:tasks.rgw_multisite_tests: "marker": "1_1690593387.549974_1756.1", 2023-07-29T01:22:58.214 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.215 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.215 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.215 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:16:27.549974Z", 2023-07-29T01:22:58.215 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.216 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.216 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.216 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.216 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:58.217 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.217 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:58.217 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-07-29T01:22:58.217 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.217 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.218 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.218 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-07-29T01:22:58.218 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.218 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.219 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.219 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.219 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:58.219 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.220 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:58.220 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:58.220 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.220 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.220 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.221 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:58.221 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.221 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.221 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.222 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:58.222 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.222 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:58.222 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-07-29T01:22:58.223 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-07-29T01:22:58.223 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.223 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:16:54Z" 2023-07-29T01:22:58.223 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:58.223 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.224 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593387.549974_1756.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:27.549974Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:22:58.224 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593413.808092_1766.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:58.224 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593387.549974_1756.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:58.224 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1690593413.808092_1766.1 target=1_1690593387.549974_1756.1 2023-07-29T01:22:58.225 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.225 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:58.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:58.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:58.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:16:59.204+0000 7f9377a13880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:58.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:16:59.205+0000 7f9377a13880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:58.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:58.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:58.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:58.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:58.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:58.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:58.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:58.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:58.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:58.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 1, 2023-07-29T01:22:58.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:58.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:58.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:58.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690593387.549974_1756.1", 2023-07-29T01:22:58.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:16:27.549974Z", 2023-07-29T01:22:58.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:58.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:58.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:58.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "0.000000", 2023-07-29T01:22:58.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:58.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:58.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:58.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:58.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:58.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 1, 2023-07-29T01:22:58.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 1 2023-07-29T01:22:58.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:16:59Z" 2023-07-29T01:22:58.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.242 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-07-29T01:22:58.242 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:58.243 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:58.243 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:58.243 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:58.243 INFO:tasks.rgw_multisite_tests: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:58.243 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.244 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.244 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:58.244 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.244 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:58.245 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.245 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:58.245 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:58.245 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.246 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-07-29T01:22:58.246 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.246 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:58.246 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.246 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.247 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.247 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.247 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:58.247 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.248 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:58.248 INFO:tasks.rgw_multisite_tests: "marker": "1_1690593387.549974_1756.1", 2023-07-29T01:22:58.248 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.248 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.249 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.249 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:16:27.549974Z", 2023-07-29T01:22:58.249 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.249 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.249 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.250 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.250 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:58.250 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.250 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:58.251 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-07-29T01:22:58.251 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.251 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.251 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.251 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-07-29T01:22:58.252 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.252 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.252 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.252 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.253 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:58.253 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.253 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:58.253 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:58.254 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.254 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.254 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.254 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:58.254 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.255 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.255 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.255 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:58.255 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.256 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:58.256 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-07-29T01:22:58.256 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-07-29T01:22:58.256 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.257 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:16:59Z" 2023-07-29T01:22:58.257 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:58.257 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.257 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593387.549974_1756.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:27.549974Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:22:58.257 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593413.808092_1766.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:58.258 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593387.549974_1756.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:58.258 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1690593413.808092_1766.1 target=1_1690593387.549974_1756.1 2023-07-29T01:22:58.258 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.258 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:58.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:58.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:58.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:17:04.316+0000 7f5776fa2880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:58.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:17:04.317+0000 7f5776fa2880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:58.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:58.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:58.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:58.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:58.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:58.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:58.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:58.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:58.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:58.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 1, 2023-07-29T01:22:58.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:58.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:58.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:58.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690593387.549974_1756.1", 2023-07-29T01:22:58.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:16:27.549974Z", 2023-07-29T01:22:58.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:58.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:58.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:58.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "0.000000", 2023-07-29T01:22:58.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:58.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:58.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:58.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:58.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:58.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 1, 2023-07-29T01:22:58.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 1 2023-07-29T01:22:58.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:17:04Z" 2023-07-29T01:22:58.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.275 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-07-29T01:22:58.275 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:58.275 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:58.275 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:58.276 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:58.276 INFO:tasks.rgw_multisite_tests: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:58.276 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.276 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.277 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:58.277 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.277 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:58.277 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.277 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:58.278 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:58.278 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.278 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-07-29T01:22:58.278 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.279 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:58.279 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.279 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.279 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.279 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.280 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:58.280 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.280 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:58.280 INFO:tasks.rgw_multisite_tests: "marker": "1_1690593387.549974_1756.1", 2023-07-29T01:22:58.281 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.281 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.281 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.281 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:16:27.549974Z", 2023-07-29T01:22:58.282 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.282 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.282 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.282 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.282 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:58.283 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.283 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:58.283 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-07-29T01:22:58.283 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.284 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.284 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.284 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-07-29T01:22:58.284 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.285 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.285 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.285 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.285 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:58.286 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.286 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:58.286 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:58.286 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.286 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.287 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.287 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:58.287 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.287 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.288 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.288 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:58.288 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.288 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:58.289 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-07-29T01:22:58.289 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-07-29T01:22:58.289 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.289 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:17:04Z" 2023-07-29T01:22:58.289 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:58.290 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593387.549974_1756.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:27.549974Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:22:58.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593413.808092_1766.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:58.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593387.549974_1756.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:58.291 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1690593413.808092_1766.1 target=1_1690593387.549974_1756.1 2023-07-29T01:22:58.291 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.291 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:58.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:58.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:58.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:17:09.438+0000 7f64397a2880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:58.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:17:09.439+0000 7f64397a2880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:58.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:58.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:58.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:58.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:58.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:58.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:58.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:58.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:58.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:58.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 1, 2023-07-29T01:22:58.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:58.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:58.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:58.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690593413.808092_1766.1", 2023-07-29T01:22:58.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:16:53.808092Z", 2023-07-29T01:22:58.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:58.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:58.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:58.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "0.000000", 2023-07-29T01:22:58.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:58.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:58.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:58.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:58.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:58.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 1, 2023-07-29T01:22:58.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 1 2023-07-29T01:22:58.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:17:09Z" 2023-07-29T01:22:58.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.307 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-07-29T01:22:58.308 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:58.308 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:58.308 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:58.308 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:58.309 INFO:tasks.rgw_multisite_tests: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:58.309 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.309 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.309 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:58.309 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.310 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:58.310 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.310 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:58.310 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:58.311 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.311 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-07-29T01:22:58.311 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.311 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:58.311 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.312 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.312 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.312 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.312 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:58.313 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.313 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:58.313 INFO:tasks.rgw_multisite_tests: "marker": "1_1690593413.808092_1766.1", 2023-07-29T01:22:58.313 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.314 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.314 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.314 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:16:53.808092Z", 2023-07-29T01:22:58.314 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.314 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.315 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.315 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.315 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:58.315 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.316 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:58.316 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-07-29T01:22:58.316 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.316 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.317 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.317 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-07-29T01:22:58.317 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.317 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.317 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.318 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.318 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:58.318 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.318 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:58.319 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:58.319 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.319 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.319 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.319 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:58.320 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.320 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.320 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.320 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:58.321 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.321 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:58.321 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-07-29T01:22:58.321 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-07-29T01:22:58.322 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.322 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:17:09Z" 2023-07-29T01:22:58.322 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:58.322 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.322 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593413.808092_1766.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:53.808092Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:22:58.323 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593413.808092_1766.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:58.323 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593413.808092_1766.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:58.323 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-07-29T01:22:58.323 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:58.324 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:58.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:22:58.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:58.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:58.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: {"groups":[]} 2023-07-29T01:22:58.325 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=hiiqos-72 2023-07-29T01:22:58.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-72/ 2023-07-29T01:22:58.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-72/ 2023-07-29T01:22:58.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:58.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-72/ 2023-07-29T01:22:58.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:58.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:58.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:58.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:58.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:58.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:58.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:58.328 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:58.328 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.328 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.328 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 01:17:09 GMT 2023-07-29T01:22:58.328 INFO:tasks.rgw_multisite_tests:/hiiqos-72/ 2023-07-29T01:22:58.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:58.329 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:kTbiUbMJLVHbqwDaRKD2l6tgZ2E= 2023-07-29T01:22:58.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:17:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kTbiUbMJLVHbqwDaRKD2l6tgZ2E=', 'Content-Length': '0'} 2023-07-29T01:22:58.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000017341a735e991956-0064c46895-283f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:17:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:58.330 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=hiiqos-73 2023-07-29T01:22:58.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-73/ 2023-07-29T01:22:58.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-73/ 2023-07-29T01:22:58.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:58.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-73/ 2023-07-29T01:22:58.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:58.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:58.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:58.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:58.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:58.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:58.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:58.332 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:58.333 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.333 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.333 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 01:17:09 GMT 2023-07-29T01:22:58.333 INFO:tasks.rgw_multisite_tests:/hiiqos-73/ 2023-07-29T01:22:58.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:58.334 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:yW1vTzYWyxWX1+O1ylWlKWLA2e4= 2023-07-29T01:22:58.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:17:09 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:yW1vTzYWyxWX1+O1ylWlKWLA2e4=', 'Content-Length': '0'} 2023-07-29T01:22:58.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000223736c1a4fb7ed3-0064c46895-283f-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:17:09 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:58.334 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.335 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:58.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:58.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:58.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T01:22:58.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: [ 2023-07-29T01:22:58.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:58.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "last_update": "2023-07-29T00:40:17.836402Z" 2023-07-29T01:22:58.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690593429.606201_1781.1", 2023-07-29T01:22:58.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "last_update": "2023-07-29T01:17:09.606201Z" 2023-07-29T01:22:58.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:58.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "last_update": "0.000000" 2023-07-29T01:22:58.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:58.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "last_update": "2023-07-29T00:40:12.811406Z" 2023-07-29T01:22:58.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.340 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1690591217.836402_294.1', 1: '1_1690593429.606201_1781.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:58.340 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-07-29T01:22:58.340 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.341 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:58.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:58.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:58.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:17:09.785+0000 7f02fe926880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:58.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:17:09.786+0000 7f02fe926880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:58.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:58.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:58.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:58.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:58.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:58.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:58.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:58.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:58.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:58.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 1, 2023-07-29T01:22:58.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:58.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:58.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:58.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690593429.606201_1781.1", 2023-07-29T01:22:58.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:17:09.606201Z", 2023-07-29T01:22:58.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:58.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:58.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:58.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "0.000000", 2023-07-29T01:22:58.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:58.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:58.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:58.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:58.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:58.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 1, 2023-07-29T01:22:58.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 1 2023-07-29T01:22:58.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:17:09Z" 2023-07-29T01:22:58.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.357 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-07-29T01:22:58.357 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:58.357 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:58.358 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:58.358 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:58.358 INFO:tasks.rgw_multisite_tests: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:58.358 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.359 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.359 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:58.359 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.359 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:58.359 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.360 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:58.360 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:58.360 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.360 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-07-29T01:22:58.361 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.361 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:58.361 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.361 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.362 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.362 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.362 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:58.362 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.362 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:58.363 INFO:tasks.rgw_multisite_tests: "marker": "1_1690593429.606201_1781.1", 2023-07-29T01:22:58.363 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.363 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.363 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.364 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:17:09.606201Z", 2023-07-29T01:22:58.364 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.364 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.364 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.365 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.365 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:58.365 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.365 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:58.365 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-07-29T01:22:58.366 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.366 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.366 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.366 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-07-29T01:22:58.367 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.367 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.367 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.367 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.368 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:58.368 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.368 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:58.368 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:58.368 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.369 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.369 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.369 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:58.369 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.370 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.370 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.370 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:58.370 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.371 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:58.371 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-07-29T01:22:58.371 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-07-29T01:22:58.371 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.371 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:17:09Z" 2023-07-29T01:22:58.372 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:58.372 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.372 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593429.606201_1781.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:09.606201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:22:58.372 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593429.606201_1781.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:58.373 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593429.606201_1781.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:58.373 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-07-29T01:22:58.373 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:58.373 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:58.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:22:58.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:58.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:58.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-07-29T01:22:58.375 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:58.375 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:58.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:22:58.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:58.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:58.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"allowed"}]} 2023-07-29T01:22:58.376 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hiiqos-72', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hiiqos-72', '--dest-bucket=hiiqos-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:58.376 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hiiqos-72', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hiiqos-72', '--dest-bucket=hiiqos-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:58.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-hiiqos-72 --source-zones=a1 --dest-zones=a2 --source-bucket=hiiqos-72 --dest-bucket=hiiqos-73 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:22:58.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:58.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:58.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-hiiqos-72","source":{"bucket":"hiiqos-72","zones":["a1"]},"dest":{"bucket":"hiiqos-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-07-29T01:22:58.377 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hiiqos-73', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hiiqos-73', '--dest-bucket=hiiqos-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:58.378 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hiiqos-73', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hiiqos-73', '--dest-bucket=hiiqos-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:58.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-hiiqos-73 --source-zones=a1 --dest-zones=a2 --source-bucket=hiiqos-73 --dest-bucket=hiiqos-73 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:22:58.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:58.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:58.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-hiiqos-72","source":{"bucket":"hiiqos-72","zones":["a1"]},"dest":{"bucket":"hiiqos-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hiiqos-73","source":{"bucket":"hiiqos-73","zones":["a1"]},"dest":{"bucket":"hiiqos-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-07-29T01:22:58.379 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:58.379 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:58.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:22:58.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:58.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:58.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-hiiqos-72","source":{"bucket":"hiiqos-72","zones":["a1"]},"dest":{"bucket":"hiiqos-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hiiqos-73","source":{"bucket":"hiiqos-73","zones":["a1"]},"dest":{"bucket":"hiiqos-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-07-29T01:22:58.380 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:58.381 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:58.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:22:58.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:58.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:58.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:58.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "epoch": 20, 2023-07-29T01:22:58.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "predecessor_uuid": "a0987484-ce37-4520-a564-54b7ea8c5251", 2023-07-29T01:22:58.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": [], 2023-07-29T01:22:58.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period_map": { 2023-07-29T01:22:58.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "id": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:58.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "zonegroups": [ 2023-07-29T01:22:58.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "id": "a", 2023-07-29T01:22:58.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "name": "a", 2023-07-29T01:22:58.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "api_name": "", 2023-07-29T01:22:58.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "is_master": true, 2023-07-29T01:22:58.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "endpoints": [ 2023-07-29T01:22:58.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-29T01:22:58.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ], 2023-07-29T01:22:58.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "hostnames": [], 2023-07-29T01:22:58.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "hostnames_s3website": [], 2023-07-29T01:22:58.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:22:58.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "zones": [ 2023-07-29T01:22:58.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "id": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:22:58.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "name": "a1", 2023-07-29T01:22:58.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "endpoints": [ 2023-07-29T01:22:58.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "http://smithi101.front.sepia.ceph.com:8000" 2023-07-29T01:22:58.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ], 2023-07-29T01:22:58.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "log_meta": false, 2023-07-29T01:22:58.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "log_data": true, 2023-07-29T01:22:58.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "bucket_index_max_shards": 11, 2023-07-29T01:22:58.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "read_only": false, 2023-07-29T01:22:58.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "tier_type": "", 2023-07-29T01:22:58.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_from_all": true, 2023-07-29T01:22:58.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_from": [], 2023-07-29T01:22:58.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "redirect_zone": "", 2023-07-29T01:22:58.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "supported_features": [ 2023-07-29T01:22:58.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "compress-encrypted", 2023-07-29T01:22:58.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "resharding" 2023-07-29T01:22:58.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "id": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-29T01:22:58.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "name": "a2", 2023-07-29T01:22:58.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "endpoints": [ 2023-07-29T01:22:58.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "http://smithi101.front.sepia.ceph.com:8001" 2023-07-29T01:22:58.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ], 2023-07-29T01:22:58.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "log_meta": false, 2023-07-29T01:22:58.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "log_data": true, 2023-07-29T01:22:58.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "bucket_index_max_shards": 11, 2023-07-29T01:22:58.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "read_only": false, 2023-07-29T01:22:58.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "tier_type": "", 2023-07-29T01:22:58.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_from_all": true, 2023-07-29T01:22:58.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_from": [], 2023-07-29T01:22:58.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "redirect_zone": "", 2023-07-29T01:22:58.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "supported_features": [ 2023-07-29T01:22:58.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "compress-encrypted", 2023-07-29T01:22:58.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "resharding" 2023-07-29T01:22:58.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ], 2023-07-29T01:22:58.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "placement_targets": [ 2023-07-29T01:22:58.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "name": "default-placement", 2023-07-29T01:22:58.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "tags": [], 2023-07-29T01:22:58.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "storage_classes": [ 2023-07-29T01:22:58.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "STANDARD" 2023-07-29T01:22:58.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ], 2023-07-29T01:22:58.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "default_placement": "default-placement", 2023-07-29T01:22:58.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T01:22:58.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_policy": { 2023-07-29T01:22:58.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "groups": [ 2023-07-29T01:22:58.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "id": "sync-group", 2023-07-29T01:22:58.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "data_flow": { 2023-07-29T01:22:58.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "directional": [ 2023-07-29T01:22:58.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "source_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:22:58.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "dest_zone": "59d1bb30-3b1c-476f-9111-052c1ea0b711" 2023-07-29T01:22:58.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pipes": [ 2023-07-29T01:22:58.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "id": "sync-pipe-hiiqos-72", 2023-07-29T01:22:58.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "source": { 2023-07-29T01:22:58.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "bucket": "hiiqos-72", 2023-07-29T01:22:58.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "zones": [ 2023-07-29T01:22:58.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433" 2023-07-29T01:22:58.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "dest": { 2023-07-29T01:22:58.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "bucket": "hiiqos-73", 2023-07-29T01:22:58.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "zones": [ 2023-07-29T01:22:58.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "59d1bb30-3b1c-476f-9111-052c1ea0b711" 2023-07-29T01:22:58.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "params": { 2023-07-29T01:22:58.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "source": { 2023-07-29T01:22:58.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "filter": { 2023-07-29T01:22:58.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "tags": [] 2023-07-29T01:22:58.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "dest": {}, 2023-07-29T01:22:58.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "priority": 0, 2023-07-29T01:22:58.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "mode": "system", 2023-07-29T01:22:58.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "user": "" 2023-07-29T01:22:58.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "id": "sync-pipe-hiiqos-73", 2023-07-29T01:22:58.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "source": { 2023-07-29T01:22:58.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "bucket": "hiiqos-73", 2023-07-29T01:22:58.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "zones": [ 2023-07-29T01:22:58.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433" 2023-07-29T01:22:58.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "dest": { 2023-07-29T01:22:58.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "bucket": "hiiqos-73", 2023-07-29T01:22:58.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "zones": [ 2023-07-29T01:22:58.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "59d1bb30-3b1c-476f-9111-052c1ea0b711" 2023-07-29T01:22:58.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "params": { 2023-07-29T01:22:58.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "source": { 2023-07-29T01:22:58.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "filter": { 2023-07-29T01:22:58.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "tags": [] 2023-07-29T01:22:58.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "dest": {}, 2023-07-29T01:22:58.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "priority": 0, 2023-07-29T01:22:58.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "mode": "system", 2023-07-29T01:22:58.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "user": "" 2023-07-29T01:22:58.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ], 2023-07-29T01:22:58.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "enabled" 2023-07-29T01:22:58.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "enabled_features": [ 2023-07-29T01:22:58.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "resharding" 2023-07-29T01:22:58.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "id": "b", 2023-07-29T01:22:58.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "name": "b", 2023-07-29T01:22:58.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "api_name": "", 2023-07-29T01:22:58.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "is_master": false, 2023-07-29T01:22:58.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "endpoints": [ 2023-07-29T01:22:58.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-29T01:22:58.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ], 2023-07-29T01:22:58.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "hostnames": [], 2023-07-29T01:22:58.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "hostnames_s3website": [], 2023-07-29T01:22:58.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "master_zone": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T01:22:58.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "zones": [ 2023-07-29T01:22:58.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "id": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T01:22:58.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "name": "b1", 2023-07-29T01:22:58.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "endpoints": [ 2023-07-29T01:22:58.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "http://smithi139.front.sepia.ceph.com:8000" 2023-07-29T01:22:58.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ], 2023-07-29T01:22:58.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "log_meta": false, 2023-07-29T01:22:58.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "log_data": true, 2023-07-29T01:22:58.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "bucket_index_max_shards": 11, 2023-07-29T01:22:58.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "read_only": false, 2023-07-29T01:22:58.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "tier_type": "", 2023-07-29T01:22:58.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_from_all": true, 2023-07-29T01:22:58.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_from": [], 2023-07-29T01:22:58.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "redirect_zone": "", 2023-07-29T01:22:58.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "supported_features": [ 2023-07-29T01:22:58.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "compress-encrypted", 2023-07-29T01:22:58.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "resharding" 2023-07-29T01:22:58.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "id": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-29T01:22:58.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "name": "b2", 2023-07-29T01:22:58.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "endpoints": [ 2023-07-29T01:22:58.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "http://smithi139.front.sepia.ceph.com:8001" 2023-07-29T01:22:58.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ], 2023-07-29T01:22:58.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "log_meta": false, 2023-07-29T01:22:58.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "log_data": true, 2023-07-29T01:22:58.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "bucket_index_max_shards": 11, 2023-07-29T01:22:58.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "read_only": false, 2023-07-29T01:22:58.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "tier_type": "", 2023-07-29T01:22:58.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_from_all": true, 2023-07-29T01:22:58.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_from": [], 2023-07-29T01:22:58.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "redirect_zone": "", 2023-07-29T01:22:58.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "supported_features": [ 2023-07-29T01:22:58.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "compress-encrypted", 2023-07-29T01:22:58.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "resharding" 2023-07-29T01:22:58.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ], 2023-07-29T01:22:58.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "placement_targets": [ 2023-07-29T01:22:58.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "name": "default-placement", 2023-07-29T01:22:58.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "tags": [], 2023-07-29T01:22:58.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "storage_classes": [ 2023-07-29T01:22:58.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "STANDARD" 2023-07-29T01:22:58.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ], 2023-07-29T01:22:58.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "default_placement": "default-placement", 2023-07-29T01:22:58.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T01:22:58.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_policy": { 2023-07-29T01:22:58.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "groups": [] 2023-07-29T01:22:58.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "enabled_features": [ 2023-07-29T01:22:58.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "resharding" 2023-07-29T01:22:58.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ], 2023-07-29T01:22:58.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "short_zone_ids": [ 2023-07-29T01:22:58.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:22:58.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": 2896287231 2023-07-29T01:22:58.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": "59d1bb30-3b1c-476f-9111-052c1ea0b711", 2023-07-29T01:22:58.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": 815606810 2023-07-29T01:22:58.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": "61176ab8-057a-43ac-9dd0-9e7d5d38f6e5", 2023-07-29T01:22:58.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": 2009171363 2023-07-29T01:22:58.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": "9d60ade7-31f2-415d-9bf3-633ac867407d", 2023-07-29T01:22:58.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": 3105190295 2023-07-29T01:22:58.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "master_zonegroup": "a", 2023-07-29T01:22:58.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "master_zone": "3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433", 2023-07-29T01:22:58.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period_config": { 2023-07-29T01:22:58.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "bucket_quota": { 2023-07-29T01:22:58.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "enabled": false, 2023-07-29T01:22:58.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "check_on_raw": false, 2023-07-29T01:22:58.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "max_size": -1, 2023-07-29T01:22:58.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "max_size_kb": 0, 2023-07-29T01:22:58.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "max_objects": -1 2023-07-29T01:22:58.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "user_quota": { 2023-07-29T01:22:58.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "enabled": false, 2023-07-29T01:22:58.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "check_on_raw": false, 2023-07-29T01:22:58.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "max_size": -1, 2023-07-29T01:22:58.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "max_size_kb": 0, 2023-07-29T01:22:58.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "max_objects": -1 2023-07-29T01:22:58.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "user_ratelimit": { 2023-07-29T01:22:58.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "max_read_ops": 0, 2023-07-29T01:22:58.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "max_write_ops": 0, 2023-07-29T01:22:58.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "max_read_bytes": 0, 2023-07-29T01:22:58.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "max_write_bytes": 0, 2023-07-29T01:22:58.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "enabled": false 2023-07-29T01:22:58.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "bucket_ratelimit": { 2023-07-29T01:22:58.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "max_read_ops": 0, 2023-07-29T01:22:58.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "max_write_ops": 0, 2023-07-29T01:22:58.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "max_read_bytes": 0, 2023-07-29T01:22:58.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "max_write_bytes": 0, 2023-07-29T01:22:58.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "enabled": false 2023-07-29T01:22:58.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "anonymous_ratelimit": { 2023-07-29T01:22:58.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "max_read_ops": 0, 2023-07-29T01:22:58.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "max_write_ops": 0, 2023-07-29T01:22:58.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "max_read_bytes": 0, 2023-07-29T01:22:58.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "max_write_bytes": 0, 2023-07-29T01:22:58.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "enabled": false 2023-07-29T01:22:58.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_id": "72b68b73-6728-4b14-854a-54780b6610bf", 2023-07-29T01:22:58.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_name": "test-realm", 2023-07-29T01:22:58.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.445 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:58.446 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-07-29T01:22:58.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-07-29T01:22:58.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:58.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:58.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-hiiqos-72","source":{"bucket":"hiiqos-72","zones":["a1"]},"dest":{"bucket":"hiiqos-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hiiqos-73","source":{"bucket":"hiiqos-73","zones":["a1"]},"dest":{"bucket":"hiiqos-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-07-29T01:22:58.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-72/ 2023-07-29T01:22:58.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-72/ 2023-07-29T01:22:58.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:58.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-72/ 2023-07-29T01:22:58.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:58.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:58.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:58.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:58.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:58.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:58.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:58.449 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:58.450 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.450 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.450 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 01:17:10 GMT 2023-07-29T01:22:58.450 INFO:tasks.rgw_multisite_tests:/hiiqos-72/ 2023-07-29T01:22:58.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:58.451 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:+naYur/uScys9ZKHwDh3kM3J6sc= 2023-07-29T01:22:58.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:17:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+naYur/uScys9ZKHwDh3kM3J6sc=', 'Content-Length': '0'} 2023-07-29T01:22:58.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000066c24a9b20589373-0064c46896-2a77-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:17:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:58.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-72/obj1 2023-07-29T01:22:58.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-72/obj1 2023-07-29T01:22:58.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:58.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-72/obj1 2023-07-29T01:22:58.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:58.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T01:22:58.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:58.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:58.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:58.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:58.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:58.454 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:58.454 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-07-29T01:22:58.455 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:58.455 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 01:17:10 GMT 2023-07-29T01:22:58.455 INFO:tasks.rgw_multisite_tests:/hiiqos-72/obj1 2023-07-29T01:22:58.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:58.455 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:xjKkdr9R7OYDWtVRnszZoNMeQjg= 2023-07-29T01:22:58.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 01:17:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:xjKkdr9R7OYDWtVRnszZoNMeQjg='} 2023-07-29T01:22:58.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a24a85ff0c1eec5e-0064c46896-2a77-a1'), ('Date', 'Sat, 29 Jul 2023 01:17:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:58.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-72/ 2023-07-29T01:22:58.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-72/ 2023-07-29T01:22:58.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:58.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-72/ 2023-07-29T01:22:58.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:58.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:58.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:58.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:58.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:58.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:58.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:58.459 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:58.459 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.459 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.459 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 01:17:10 GMT 2023-07-29T01:22:58.460 INFO:tasks.rgw_multisite_tests:/hiiqos-72/ 2023-07-29T01:22:58.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:58.460 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:+naYur/uScys9ZKHwDh3kM3J6sc= 2023-07-29T01:22:58.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:17:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+naYur/uScys9ZKHwDh3kM3J6sc=', 'Content-Length': '0'} 2023-07-29T01:22:58.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002aac6f8792eb42bc-0064c46896-2a77-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:17:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:58.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-72/obj2 2023-07-29T01:22:58.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-72/obj2 2023-07-29T01:22:58.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:58.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-72/obj2 2023-07-29T01:22:58.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:58.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T01:22:58.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:58.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:58.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:58.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:58.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:58.463 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:58.464 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-07-29T01:22:58.464 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:58.464 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 01:17:10 GMT 2023-07-29T01:22:58.464 INFO:tasks.rgw_multisite_tests:/hiiqos-72/obj2 2023-07-29T01:22:58.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:58.465 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:4kDvtFWQlxVLkvnaDakrDgDEq+E= 2023-07-29T01:22:58.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 01:17:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4kDvtFWQlxVLkvnaDakrDgDEq+E='} 2023-07-29T01:22:58.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004a67b85995526949-0064c46896-2a77-a1'), ('Date', 'Sat, 29 Jul 2023 01:17:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:58.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-73/ 2023-07-29T01:22:58.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-73/ 2023-07-29T01:22:58.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:58.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-73/ 2023-07-29T01:22:58.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:58.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:58.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:58.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:58.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:58.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:58.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:58.468 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:58.468 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.468 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.469 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 01:17:10 GMT 2023-07-29T01:22:58.469 INFO:tasks.rgw_multisite_tests:/hiiqos-73/ 2023-07-29T01:22:58.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:58.469 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:9juOUUKQ1t7JuIifojmap4PQkPI= 2023-07-29T01:22:58.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:17:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9juOUUKQ1t7JuIifojmap4PQkPI=', 'Content-Length': '0'} 2023-07-29T01:22:58.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000065b0b6d2d2a440e5-0064c46896-2a77-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:17:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:58.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-73/obj3 2023-07-29T01:22:58.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-73/obj3 2023-07-29T01:22:58.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:58.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-73/obj3 2023-07-29T01:22:58.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:58.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T01:22:58.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:58.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:58.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:58.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:58.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:58.473 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:58.473 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-07-29T01:22:58.473 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:58.473 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 01:17:10 GMT 2023-07-29T01:22:58.473 INFO:tasks.rgw_multisite_tests:/hiiqos-73/obj3 2023-07-29T01:22:58.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:58.474 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:ebaUGY7mWF+CIwMGkUkvcvcfpSo= 2023-07-29T01:22:58.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 01:17:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:ebaUGY7mWF+CIwMGkUkvcvcfpSo='} 2023-07-29T01:22:58.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000e49c94f63582fc9-0064c46896-2a77-a1'), ('Date', 'Sat, 29 Jul 2023 01:17:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:58.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-73/ 2023-07-29T01:22:58.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-73/ 2023-07-29T01:22:58.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:58.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-73/ 2023-07-29T01:22:58.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:58.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:58.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:58.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:58.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:58.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:58.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:58.477 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:58.477 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.478 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.478 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 01:17:10 GMT 2023-07-29T01:22:58.478 INFO:tasks.rgw_multisite_tests:/hiiqos-73/ 2023-07-29T01:22:58.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:58.478 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:9juOUUKQ1t7JuIifojmap4PQkPI= 2023-07-29T01:22:58.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:17:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:9juOUUKQ1t7JuIifojmap4PQkPI=', 'Content-Length': '0'} 2023-07-29T01:22:58.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000055819dfe0b5203b9-0064c46896-2a77-a1'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:17:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:58.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-73/obj4 2023-07-29T01:22:58.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-73/obj4 2023-07-29T01:22:58.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-07-29T01:22:58.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-73/obj4 2023-07-29T01:22:58.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:58.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-07-29T01:22:58.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8000 2023-07-29T01:22:58.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-07-29T01:22:58.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:58.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:58.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:58.482 INFO:tasks.rgw_multisite_tests:PUT 2023-07-29T01:22:58.482 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-07-29T01:22:58.482 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-07-29T01:22:58.482 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 01:17:10 GMT 2023-07-29T01:22:58.483 INFO:tasks.rgw_multisite_tests:/hiiqos-73/obj4 2023-07-29T01:22:58.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:58.483 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:FzT8DMZ8Vjxcn9vt2stunqRlq6I= 2023-07-29T01:22:58.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Sat, 29 Jul 2023 01:17:10 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:FzT8DMZ8Vjxcn9vt2stunqRlq6I='} 2023-07-29T01:22:58.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eec8dec4d012fb03-0064c46896-2a77-a1'), ('Date', 'Sat, 29 Jul 2023 01:17:10 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:58.484 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.484 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:58.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:58.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:58.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: No --period given, using current period=2fd8ad79-2326-4bb2-9f6c-9f77e2acc853 2023-07-29T01:22:58.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: [ 2023-07-29T01:22:58.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:58.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "last_update": "2023-07-29T00:40:17.836402Z" 2023-07-29T01:22:58.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690593429.606201_1781.1", 2023-07-29T01:22:58.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "last_update": "2023-07-29T01:17:09.606201Z" 2023-07-29T01:22:58.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:58.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "last_update": "0.000000" 2023-07-29T01:22:58.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:58.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "last_update": "2023-07-29T00:40:12.811406Z" 2023-07-29T01:22:58.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.489 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1690591217.836402_294.1', 1: '1_1690593429.606201_1781.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:58.490 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-07-29T01:22:58.490 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.490 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:58.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:58.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:58.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:17:11.117+0000 7fbe2b867880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:58.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: 2023-07-29T01:17:11.118+0000 7fbe2b867880 20 RGW-SYNC:meta: read sync status 2023-07-29T01:22:58.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:58.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:58.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:58.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:58.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:58.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:58.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:58.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:58.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:58.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 1, 2023-07-29T01:22:58.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:58.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:58.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:58.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690593429.606201_1781.1", 2023-07-29T01:22:58.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:17:09.606201Z", 2023-07-29T01:22:58.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:58.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:58.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "", 2023-07-29T01:22:58.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "0.000000", 2023-07-29T01:22:58.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:58.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "state": 1, 2023-07-29T01:22:58.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:58.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:58.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "realm_epoch": 2 2023-07-29T01:22:58.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:58.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 1, 2023-07-29T01:22:58.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 1 2023-07-29T01:22:58.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:17:11Z" 2023-07-29T01:22:58.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.506 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-07-29T01:22:58.507 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:58.507 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:58.507 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:58.507 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:58.508 INFO:tasks.rgw_multisite_tests: "period": "2fd8ad79-2326-4bb2-9f6c-9f77e2acc853", 2023-07-29T01:22:58.508 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.508 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.508 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:58.508 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.509 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:58.509 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.509 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:58.509 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591217.836402_294.1", 2023-07-29T01:22:58.510 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.510 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-07-29T01:22:58.510 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.510 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:17.836402Z", 2023-07-29T01:22:58.511 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.511 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.511 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.511 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.511 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:58.512 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.512 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:58.512 INFO:tasks.rgw_multisite_tests: "marker": "1_1690593429.606201_1781.1", 2023-07-29T01:22:58.512 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.513 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.513 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.513 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:17:09.606201Z", 2023-07-29T01:22:58.513 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.513 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.514 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.514 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.514 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:58.514 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.515 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:58.515 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-07-29T01:22:58.515 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.515 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.516 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.516 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-07-29T01:22:58.516 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.516 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.516 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.517 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.517 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:58.517 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.517 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-07-29T01:22:58.518 INFO:tasks.rgw_multisite_tests: "marker": "1_1690591212.811406_223.1", 2023-07-29T01:22:58.518 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.518 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.518 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.518 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T00:40:12.811406Z", 2023-07-29T01:22:58.519 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-07-29T01:22:58.519 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.519 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.519 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:58.520 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.520 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:58.520 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-07-29T01:22:58.520 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-07-29T01:22:58.521 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.521 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:17:11Z" 2023-07-29T01:22:58.521 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:58.521 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.521 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1690591217.836402_294.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-07-29T00:40:17.836402Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1690593429.606201_1781.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:09.606201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1690591212.811406_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T00:40:12.811406Z', 'realm_epoch': 2}}] 2023-07-29T01:22:58.522 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1690591217.836402_294.1', 1: '1_1690593429.606201_1781.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:58.522 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1690591217.836402_294.1', 1: '1_1690593429.606201_1781.1', 2: '', 3: '1_1690591212.811406_223.1'} 2023-07-29T01:22:58.522 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-07-29T01:22:58.522 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.523 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:58.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:58.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:58.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: [ 2023-07-29T01:22:58.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000052514", 2023-07-29T01:22:58.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "last_update": "2023-07-29T01:17:10.772704Z" 2023-07-29T01:22:58.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:58.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "last_update": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:58.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000046080", 2023-07-29T01:22:58.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "last_update": "2023-07-29T01:17:10.709533Z" 2023-07-29T01:22:58.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000031774", 2023-07-29T01:22:58.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "last_update": "2023-07-29T01:17:10.812152Z" 2023-07-29T01:22:58.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.528 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: data markers for zone=a1 markers={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:22:58.528 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting data checkpoint for target_zone=a2 source_zone=a1 2023-07-29T01:22:58.528 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.528 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:58.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:58.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:58.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:58.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:58.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:58.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:58.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "instance_id": 7555809319554727162 2023-07-29T01:22:58.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:58.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:58.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:22:58.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 55, 2023-07-29T01:22:58.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:22:58.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:58.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:58.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 9, 2023-07-29T01:22:58.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:58.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:58.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:22:58.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:22:58.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:58.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:22:58.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:22:58.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:58.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 64, 2023-07-29T01:22:58.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 64 2023-07-29T01:22:58.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:17:11Z" 2023-07-29T01:22:58.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.543 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-07-29T01:22:58.543 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:58.544 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:58.544 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:58.544 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:58.544 INFO:tasks.rgw_multisite_tests: "instance_id": 7555809319554727162 2023-07-29T01:22:58.545 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.545 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:58.545 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.545 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:58.545 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.546 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.546 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:22:58.546 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.546 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-07-29T01:22:58.547 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.547 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:22:58.547 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.547 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.548 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.548 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:58.548 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.548 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.548 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:58.549 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.549 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-07-29T01:22:58.549 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.549 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:58.550 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.550 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.550 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.550 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:58.551 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.551 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.551 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:22:58.551 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.551 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.552 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.552 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:22:58.552 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.552 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.553 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.553 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:58.553 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.553 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.554 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:22:58.554 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.554 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.554 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.554 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:22:58.555 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.555 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.555 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:58.555 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.556 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:58.556 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-07-29T01:22:58.556 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-07-29T01:22:58.556 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.557 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:17:11Z" 2023-07-29T01:22:58.557 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:58.557 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.557 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052233', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.371644Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045940', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.390653Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031634', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640507Z'}}] 2023-07-29T01:22:58.557 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:22:58.558 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031634'} 2023-07-29T01:22:58.558 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052233, shard=2 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045940, shard=3 master=00000000000000000000:00000000000000031774 target=00000000000000000000:00000000000000031634 2023-07-29T01:22:58.558 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.558 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:58.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:58.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:58.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:58.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:58.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:58.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:58.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "instance_id": 7555809319554727162 2023-07-29T01:22:58.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:58.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:58.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:22:58.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 55, 2023-07-29T01:22:58.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:22:58.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:58.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:58.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 9, 2023-07-29T01:22:58.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:58.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:58.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:22:58.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:22:58.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:58.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:22:58.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:22:58.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:58.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 64, 2023-07-29T01:22:58.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 64 2023-07-29T01:22:58.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:17:16Z" 2023-07-29T01:22:58.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.573 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-07-29T01:22:58.573 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:58.574 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:58.574 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:58.574 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:58.574 INFO:tasks.rgw_multisite_tests: "instance_id": 7555809319554727162 2023-07-29T01:22:58.575 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.575 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:58.575 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.575 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:58.576 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.576 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.576 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:22:58.576 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.576 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-07-29T01:22:58.577 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.577 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:22:58.577 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.577 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.578 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.578 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:58.578 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.578 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.579 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:58.579 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.579 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-07-29T01:22:58.579 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.579 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:58.580 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.580 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.580 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.580 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:58.581 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.581 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.581 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:22:58.581 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.582 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.582 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.582 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:22:58.582 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.582 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.583 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.583 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:58.583 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.583 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.584 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:22:58.584 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.584 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.584 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.584 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:22:58.585 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.585 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.585 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:58.585 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.586 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:58.586 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-07-29T01:22:58.586 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-07-29T01:22:58.586 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.586 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:17:16Z" 2023-07-29T01:22:58.587 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:58.587 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.587 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052233', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.371644Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045940', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.390653Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031634', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640507Z'}}] 2023-07-29T01:22:58.587 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:22:58.588 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031634'} 2023-07-29T01:22:58.588 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052233, shard=2 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045940, shard=3 master=00000000000000000000:00000000000000031774 target=00000000000000000000:00000000000000031634 2023-07-29T01:22:58.588 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.588 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:58.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:58.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:58.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:58.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:58.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:58.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:58.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "instance_id": 7555809319554727162 2023-07-29T01:22:58.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:58.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:58.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:22:58.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 55, 2023-07-29T01:22:58.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:22:58.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:58.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:58.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 9, 2023-07-29T01:22:58.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:58.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:58.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:22:58.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:22:58.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:58.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:22:58.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.600 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:22:58.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:58.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 64, 2023-07-29T01:22:58.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 64 2023-07-29T01:22:58.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:17:21Z" 2023-07-29T01:22:58.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.603 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-07-29T01:22:58.603 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:58.604 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:58.604 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:58.604 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:58.604 INFO:tasks.rgw_multisite_tests: "instance_id": 7555809319554727162 2023-07-29T01:22:58.605 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.605 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:58.605 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.605 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:58.606 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.606 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.606 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:22:58.606 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.606 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-07-29T01:22:58.607 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.607 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:22:58.607 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.607 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.608 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.608 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:58.608 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.608 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.608 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:58.609 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.609 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-07-29T01:22:58.609 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.609 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:58.610 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.610 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.610 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.610 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:58.611 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.611 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.611 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:22:58.611 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.611 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.612 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.612 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:22:58.612 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.612 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.613 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.613 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:58.613 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.613 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.614 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:22:58.614 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.614 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.614 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.614 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:22:58.615 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.615 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.615 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:58.615 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.616 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:58.616 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-07-29T01:22:58.616 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-07-29T01:22:58.616 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.617 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:17:21Z" 2023-07-29T01:22:58.617 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:58.617 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.617 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052233', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.371644Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045940', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.390653Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031634', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640507Z'}}] 2023-07-29T01:22:58.617 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:22:58.618 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031634'} 2023-07-29T01:22:58.618 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052233, shard=2 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045940, shard=3 master=00000000000000000000:00000000000000031774 target=00000000000000000000:00000000000000031634 2023-07-29T01:22:58.618 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.618 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:58.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:58.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:58.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:58.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:58.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:58.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:58.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "instance_id": 7555809319554727162 2023-07-29T01:22:58.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:58.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:58.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:22:58.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 55, 2023-07-29T01:22:58.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:22:58.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:58.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:58.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 9, 2023-07-29T01:22:58.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:58.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:58.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:22:58.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:22:58.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:58.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:22:58.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:22:58.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.631 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:58.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 64, 2023-07-29T01:22:58.632 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 64 2023-07-29T01:22:58.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:17:26Z" 2023-07-29T01:22:58.633 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.633 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-07-29T01:22:58.633 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:58.634 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:58.634 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:58.634 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:58.634 INFO:tasks.rgw_multisite_tests: "instance_id": 7555809319554727162 2023-07-29T01:22:58.635 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.635 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:58.635 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.635 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:58.636 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.636 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.636 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:22:58.636 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.636 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-07-29T01:22:58.637 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.637 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:22:58.637 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.637 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.638 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.638 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:58.638 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.638 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.639 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:58.639 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.639 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-07-29T01:22:58.639 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.640 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:58.640 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.640 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.640 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.641 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:58.641 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.641 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.641 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:22:58.641 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.642 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.642 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.642 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:22:58.642 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.643 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.643 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.643 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:58.643 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.644 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.644 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:22:58.644 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.644 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.644 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.645 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:22:58.645 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.645 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.645 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:58.646 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.646 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:58.646 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-07-29T01:22:58.646 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-07-29T01:22:58.647 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.647 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:17:26Z" 2023-07-29T01:22:58.647 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:58.647 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.647 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052233', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.371644Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045940', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.390653Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031634', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640507Z'}}] 2023-07-29T01:22:58.648 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:22:58.648 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031634'} 2023-07-29T01:22:58.648 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052233, shard=2 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045940, shard=3 master=00000000000000000000:00000000000000031774 target=00000000000000000000:00000000000000031634 2023-07-29T01:22:58.648 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.649 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:58.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:58.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:58.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:58.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:58.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:58.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:58.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "instance_id": 7555809319554727162 2023-07-29T01:22:58.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:58.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:58.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:22:58.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 55, 2023-07-29T01:22:58.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:22:58.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:58.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:58.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 9, 2023-07-29T01:22:58.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:58.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:58.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:22:58.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:22:58.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:58.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:22:58.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:22:58.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:58.662 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 64, 2023-07-29T01:22:58.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 64 2023-07-29T01:22:58.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:17:31Z" 2023-07-29T01:22:58.663 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.663 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-07-29T01:22:58.664 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:58.664 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:58.664 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:58.664 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:58.665 INFO:tasks.rgw_multisite_tests: "instance_id": 7555809319554727162 2023-07-29T01:22:58.665 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.665 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:58.665 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.666 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:58.666 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.666 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.666 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:22:58.666 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.667 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-07-29T01:22:58.667 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.667 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:22:58.667 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.668 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.668 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.668 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:58.668 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.668 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.669 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:58.669 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.669 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-07-29T01:22:58.669 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.670 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:58.670 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.670 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.670 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.671 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:58.671 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.671 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.671 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:22:58.671 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.672 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.672 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.672 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:22:58.672 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.673 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.673 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.673 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:58.673 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.674 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.674 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:22:58.674 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.674 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.674 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.675 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:22:58.675 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.675 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.675 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:58.676 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.676 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:58.676 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-07-29T01:22:58.676 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-07-29T01:22:58.677 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.677 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:17:31Z" 2023-07-29T01:22:58.677 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:58.677 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.677 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052233', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.371644Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045940', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.390653Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031634', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640507Z'}}] 2023-07-29T01:22:58.678 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:22:58.678 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031634'} 2023-07-29T01:22:58.678 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052233, shard=2 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045940, shard=3 master=00000000000000000000:00000000000000031774 target=00000000000000000000:00000000000000031634 2023-07-29T01:22:58.678 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.679 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:58.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:58.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:58.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:58.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:58.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:58.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:58.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "instance_id": 7555809319554727162 2023-07-29T01:22:58.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:58.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:58.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:22:58.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 55, 2023-07-29T01:22:58.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:22:58.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:58.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:58.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 9, 2023-07-29T01:22:58.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:58.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:58.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:22:58.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:22:58.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:58.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:22:58.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:22:58.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:58.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 64, 2023-07-29T01:22:58.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 64 2023-07-29T01:22:58.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:17:36Z" 2023-07-29T01:22:58.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.693 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-07-29T01:22:58.694 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:58.694 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:58.694 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:58.694 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:58.695 INFO:tasks.rgw_multisite_tests: "instance_id": 7555809319554727162 2023-07-29T01:22:58.695 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.695 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:58.695 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.695 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:58.696 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.696 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.696 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:22:58.696 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.697 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-07-29T01:22:58.697 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.697 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:22:58.697 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.698 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.698 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.698 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:58.698 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.698 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.699 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:58.699 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.699 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-07-29T01:22:58.699 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.700 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:58.700 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.700 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.700 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.701 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:58.701 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.701 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.701 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:22:58.701 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.702 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.702 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.702 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:22:58.702 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.703 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.703 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.703 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:58.703 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.704 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.704 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:22:58.704 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.704 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.704 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.705 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:22:58.705 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.705 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.705 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:58.706 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.706 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:58.706 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-07-29T01:22:58.706 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-07-29T01:22:58.707 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.707 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:17:36Z" 2023-07-29T01:22:58.707 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:58.707 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.707 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052233', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.371644Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045940', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.390653Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031634', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640507Z'}}] 2023-07-29T01:22:58.708 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:22:58.708 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031634'} 2023-07-29T01:22:58.708 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052233, shard=2 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045940, shard=3 master=00000000000000000000:00000000000000031774 target=00000000000000000000:00000000000000031634 2023-07-29T01:22:58.708 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.709 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:58.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:58.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:58.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:58.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:58.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:58.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:58.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "instance_id": 7555809319554727162 2023-07-29T01:22:58.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:58.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:58.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:22:58.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 55, 2023-07-29T01:22:58.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:22:58.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:58.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:58.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 9, 2023-07-29T01:22:58.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:58.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:58.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:22:58.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:22:58.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:58.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:22:58.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:22:58.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:58.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 64, 2023-07-29T01:22:58.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 64 2023-07-29T01:22:58.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:17:42Z" 2023-07-29T01:22:58.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.723 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-07-29T01:22:58.724 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:58.724 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:58.724 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:58.724 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:58.725 INFO:tasks.rgw_multisite_tests: "instance_id": 7555809319554727162 2023-07-29T01:22:58.725 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.725 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:58.725 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.725 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:58.726 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.726 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.726 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:22:58.726 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.727 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-07-29T01:22:58.727 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.727 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:22:58.727 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.728 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.728 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.728 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:58.728 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.728 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.729 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:58.729 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.729 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-07-29T01:22:58.729 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.730 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:58.730 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.730 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.730 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.731 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:58.731 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.731 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.731 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:22:58.731 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.732 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.732 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.732 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:22:58.732 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.733 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.733 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.733 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:58.733 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.734 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.734 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:22:58.734 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.734 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.734 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.735 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:22:58.735 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.735 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.735 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:58.736 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.736 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:58.736 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-07-29T01:22:58.736 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-07-29T01:22:58.737 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.737 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:17:42Z" 2023-07-29T01:22:58.737 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:58.737 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.737 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052233', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.371644Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045940', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.390653Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031634', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640507Z'}}] 2023-07-29T01:22:58.738 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:22:58.738 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031634'} 2023-07-29T01:22:58.738 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052233, shard=2 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045940, shard=3 master=00000000000000000000:00000000000000031774 target=00000000000000000000:00000000000000031634 2023-07-29T01:22:58.738 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.739 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:58.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:58.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:58.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:58.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:58.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:58.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:58.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "instance_id": 7555809319554727162 2023-07-29T01:22:58.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:58.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:58.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:22:58.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 55, 2023-07-29T01:22:58.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:22:58.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:58.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:58.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 9, 2023-07-29T01:22:58.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:58.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:58.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:22:58.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:22:58.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:58.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:22:58.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:22:58.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:58.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 64, 2023-07-29T01:22:58.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 64 2023-07-29T01:22:58.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:17:47Z" 2023-07-29T01:22:58.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.753 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-07-29T01:22:58.754 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:58.754 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:58.754 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:58.754 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:58.755 INFO:tasks.rgw_multisite_tests: "instance_id": 7555809319554727162 2023-07-29T01:22:58.755 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.755 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:58.755 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.755 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:58.756 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.756 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.756 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:22:58.756 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.757 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-07-29T01:22:58.757 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.757 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:22:58.757 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.758 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.758 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.758 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:58.758 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.758 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.759 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:58.759 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.759 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-07-29T01:22:58.759 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.760 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:58.760 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.760 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.760 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.760 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:58.761 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.761 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.761 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:22:58.761 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.762 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.762 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.762 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:22:58.762 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.763 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.763 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.763 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:58.763 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.763 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.764 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:22:58.764 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.764 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.764 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.765 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:22:58.765 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.765 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.765 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:58.766 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.766 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:58.766 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-07-29T01:22:58.766 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-07-29T01:22:58.766 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.767 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:17:47Z" 2023-07-29T01:22:58.767 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:58.767 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.767 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052233', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.371644Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045940', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.390653Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031634', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640507Z'}}] 2023-07-29T01:22:58.768 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:22:58.768 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031634'} 2023-07-29T01:22:58.768 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052233, shard=2 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045940, shard=3 master=00000000000000000000:00000000000000031774 target=00000000000000000000:00000000000000031634 2023-07-29T01:22:58.768 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.769 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:58.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:58.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:58.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:58.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:58.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:58.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:58.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "instance_id": 7555809319554727162 2023-07-29T01:22:58.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:58.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:58.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:22:58.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 55, 2023-07-29T01:22:58.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:22:58.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:58.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:58.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 9, 2023-07-29T01:22:58.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:58.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:58.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:22:58.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:22:58.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:58.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:22:58.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:22:58.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:58.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 64, 2023-07-29T01:22:58.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 64 2023-07-29T01:22:58.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:17:52Z" 2023-07-29T01:22:58.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.783 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-07-29T01:22:58.784 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:58.784 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:58.784 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:58.784 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:58.785 INFO:tasks.rgw_multisite_tests: "instance_id": 7555809319554727162 2023-07-29T01:22:58.785 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.785 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:58.785 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.785 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:58.786 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.786 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.786 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:22:58.786 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.787 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-07-29T01:22:58.787 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.787 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:22:58.787 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.788 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.788 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.788 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:58.788 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.789 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.789 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:58.789 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.789 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-07-29T01:22:58.789 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.790 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:58.790 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.790 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.790 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.791 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:58.791 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.791 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.791 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:22:58.792 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.792 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.792 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.792 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:22:58.792 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.793 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.793 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.793 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:58.793 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.794 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.794 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:22:58.794 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.794 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.795 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.795 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:22:58.795 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.795 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.795 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:58.796 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.796 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:58.796 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-07-29T01:22:58.796 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-07-29T01:22:58.797 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.797 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:17:52Z" 2023-07-29T01:22:58.797 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:58.797 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.798 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052233', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.371644Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045940', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.390653Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031634', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640507Z'}}] 2023-07-29T01:22:58.798 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:22:58.798 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031634'} 2023-07-29T01:22:58.798 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052233, shard=2 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045940, shard=3 master=00000000000000000000:00000000000000031774 target=00000000000000000000:00000000000000031634 2023-07-29T01:22:58.798 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.799 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:58.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:58.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:58.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:58.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:58.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:58.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:58.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "instance_id": 7555809319554727162 2023-07-29T01:22:58.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:58.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:58.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.802 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:22:58.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 55, 2023-07-29T01:22:58.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:22:58.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:58.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:58.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 9, 2023-07-29T01:22:58.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:58.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:58.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:22:58.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:22:58.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:58.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:22:58.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:22:58.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:58.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 64, 2023-07-29T01:22:58.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 64 2023-07-29T01:22:58.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:17:57Z" 2023-07-29T01:22:58.813 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.814 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-07-29T01:22:58.814 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:58.814 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:58.814 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:58.814 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:58.815 INFO:tasks.rgw_multisite_tests: "instance_id": 7555809319554727162 2023-07-29T01:22:58.815 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.815 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:58.815 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.816 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:58.816 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.816 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.816 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:22:58.817 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.817 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-07-29T01:22:58.817 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.817 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:22:58.818 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.818 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.818 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.818 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:58.818 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.819 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.819 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:58.819 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.819 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-07-29T01:22:58.820 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.820 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:58.820 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.820 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.821 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.821 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:58.821 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.821 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.821 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:22:58.822 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.822 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.822 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.822 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:22:58.823 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.823 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.823 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.823 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:58.824 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.824 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.824 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:22:58.824 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.825 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.825 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.825 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:22:58.825 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.825 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.826 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:58.826 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.826 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:58.826 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-07-29T01:22:58.827 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-07-29T01:22:58.827 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.827 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:17:57Z" 2023-07-29T01:22:58.827 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:58.828 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.828 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052233', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.371644Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045940', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.390653Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031634', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640507Z'}}] 2023-07-29T01:22:58.828 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:22:58.828 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031634'} 2023-07-29T01:22:58.829 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052233, shard=2 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045940, shard=3 master=00000000000000000000:00000000000000031774 target=00000000000000000000:00000000000000031634 2023-07-29T01:22:58.829 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.829 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:58.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:58.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:58.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:58.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:58.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:58.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:58.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "instance_id": 7555809319554727162 2023-07-29T01:22:58.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:58.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:58.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:22:58.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 55, 2023-07-29T01:22:58.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:22:58.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:58.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:58.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 9, 2023-07-29T01:22:58.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:58.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:58.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:22:58.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:22:58.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:58.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:22:58.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:22:58.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:58.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 64, 2023-07-29T01:22:58.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 64 2023-07-29T01:22:58.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:18:02Z" 2023-07-29T01:22:58.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-07-29T01:22:58.844 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:58.844 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:58.844 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:58.845 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:58.845 INFO:tasks.rgw_multisite_tests: "instance_id": 7555809319554727162 2023-07-29T01:22:58.845 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.845 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:58.846 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.846 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:58.846 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.846 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.846 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:22:58.847 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.847 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-07-29T01:22:58.847 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.847 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:22:58.848 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.848 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.848 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.848 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:58.849 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.849 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.849 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:58.849 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.849 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-07-29T01:22:58.850 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.850 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:58.850 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.850 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.851 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.851 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:58.851 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.851 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.852 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:22:58.852 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.852 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.852 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.852 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:22:58.853 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.853 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.853 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.853 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:58.854 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.854 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.854 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:22:58.854 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.855 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.855 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.855 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:22:58.855 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.855 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.856 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:58.856 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.856 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:58.856 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-07-29T01:22:58.857 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-07-29T01:22:58.857 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.857 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:18:02Z" 2023-07-29T01:22:58.857 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:58.858 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.858 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052233', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.371644Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045940', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.390653Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031634', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640507Z'}}] 2023-07-29T01:22:58.858 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:22:58.858 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031634'} 2023-07-29T01:22:58.858 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052233, shard=2 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045940, shard=3 master=00000000000000000000:00000000000000031774 target=00000000000000000000:00000000000000031634 2023-07-29T01:22:58.859 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.859 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:58.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:58.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:58.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:58.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:58.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:58.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:58.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "instance_id": 7555809319554727162 2023-07-29T01:22:58.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:58.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:58.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:22:58.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 55, 2023-07-29T01:22:58.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:22:58.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:58.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:58.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 9, 2023-07-29T01:22:58.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:58.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:58.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:22:58.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:22:58.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:58.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:22:58.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:22:58.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:58.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 64, 2023-07-29T01:22:58.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 64 2023-07-29T01:22:58.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:18:07Z" 2023-07-29T01:22:58.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.874 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-07-29T01:22:58.874 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:58.874 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:58.874 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:58.875 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:58.875 INFO:tasks.rgw_multisite_tests: "instance_id": 7555809319554727162 2023-07-29T01:22:58.875 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.875 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:58.876 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.876 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:58.876 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.876 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.877 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:22:58.877 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.877 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-07-29T01:22:58.877 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.877 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:22:58.878 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.878 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.878 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.878 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:58.879 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.879 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.879 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:58.879 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.880 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-07-29T01:22:58.880 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.880 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:58.880 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.880 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.881 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.881 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:58.881 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.881 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.882 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:22:58.882 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.882 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.882 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.883 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:22:58.883 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.883 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.883 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.883 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:58.884 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.884 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.884 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:22:58.884 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.885 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.885 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.885 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:22:58.885 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.886 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.886 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:58.886 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.886 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:58.886 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-07-29T01:22:58.887 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-07-29T01:22:58.887 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.887 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:18:07Z" 2023-07-29T01:22:58.887 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:58.888 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.888 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052233', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.371644Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045940', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.390653Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031634', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640507Z'}}] 2023-07-29T01:22:58.888 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:22:58.888 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031634'} 2023-07-29T01:22:58.889 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052233, shard=2 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045940, shard=3 master=00000000000000000000:00000000000000031774 target=00000000000000000000:00000000000000031634 2023-07-29T01:22:58.889 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.889 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:58.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:58.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:58.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:58.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:58.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:58.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:58.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "instance_id": 7555809319554727162 2023-07-29T01:22:58.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:58.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:58.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:22:58.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 55, 2023-07-29T01:22:58.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:22:58.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:58.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:58.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 9, 2023-07-29T01:22:58.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:58.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:58.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:22:58.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:22:58.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:58.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:22:58.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:22:58.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:58.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 64, 2023-07-29T01:22:58.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 64 2023-07-29T01:22:58.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:18:12Z" 2023-07-29T01:22:58.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.904 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-07-29T01:22:58.904 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:58.904 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:58.905 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:58.905 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:58.905 INFO:tasks.rgw_multisite_tests: "instance_id": 7555809319554727162 2023-07-29T01:22:58.905 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.905 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:58.906 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.906 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:58.906 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.906 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.907 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:22:58.907 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.907 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-07-29T01:22:58.907 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.908 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:22:58.908 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.908 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.908 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.908 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:58.909 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.909 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.909 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:58.909 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.910 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-07-29T01:22:58.910 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.910 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:58.910 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.910 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.911 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.911 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:58.911 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.911 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.912 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:22:58.912 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.912 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.912 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.913 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:22:58.913 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.913 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.913 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.914 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:58.914 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.914 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.914 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:22:58.914 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.915 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.915 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.915 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:22:58.915 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.916 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.916 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:58.916 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.916 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:58.917 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-07-29T01:22:58.917 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-07-29T01:22:58.917 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.917 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:18:12Z" 2023-07-29T01:22:58.918 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:58.918 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.918 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052233', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.371644Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045940', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.390653Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031634', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640507Z'}}] 2023-07-29T01:22:58.918 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:22:58.919 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031634'} 2023-07-29T01:22:58.919 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052233, shard=2 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045940, shard=3 master=00000000000000000000:00000000000000031774 target=00000000000000000000:00000000000000031634 2023-07-29T01:22:58.919 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.919 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:58.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:58.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:58.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:58.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:58.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:58.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:58.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "instance_id": 7555809319554727162 2023-07-29T01:22:58.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:58.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:58.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:22:58.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 55, 2023-07-29T01:22:58.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:22:58.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:58.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:58.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 9, 2023-07-29T01:22:58.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:58.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:58.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:22:58.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:22:58.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:58.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:22:58.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:22:58.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:58.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 64, 2023-07-29T01:22:58.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 64 2023-07-29T01:22:58.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:18:17Z" 2023-07-29T01:22:58.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.934 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-07-29T01:22:58.934 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:58.935 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:58.935 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:58.935 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:58.935 INFO:tasks.rgw_multisite_tests: "instance_id": 7555809319554727162 2023-07-29T01:22:58.935 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.936 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:58.936 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.936 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:58.936 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.937 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.937 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:22:58.937 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.937 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-07-29T01:22:58.938 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.938 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:22:58.938 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.938 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.938 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.939 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:58.939 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.939 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.939 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:58.940 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.940 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-07-29T01:22:58.940 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.940 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:58.941 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.941 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.941 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.941 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:58.942 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.942 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.942 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:22:58.942 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.942 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.943 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.943 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:22:58.943 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.943 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.944 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.944 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:58.944 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.944 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.945 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:22:58.945 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.945 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.945 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.945 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:22:58.946 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.946 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.946 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:58.946 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.947 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:58.947 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-07-29T01:22:58.947 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-07-29T01:22:58.947 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.947 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:18:17Z" 2023-07-29T01:22:58.948 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:58.948 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.948 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052233', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.371644Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045940', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.390653Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031634', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640507Z'}}] 2023-07-29T01:22:58.948 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:22:58.949 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031634'} 2023-07-29T01:22:58.949 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052233, shard=2 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045940, shard=3 master=00000000000000000000:00000000000000031774 target=00000000000000000000:00000000000000031634 2023-07-29T01:22:58.949 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.950 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:58.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:58.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:58.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:58.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:58.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:58.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:58.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "instance_id": 7555809319554727162 2023-07-29T01:22:58.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:58.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:58.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:22:58.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 55, 2023-07-29T01:22:58.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:22:58.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:58.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:58.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 9, 2023-07-29T01:22:58.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:58.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:58.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:22:58.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:22:58.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:58.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:22:58.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:22:58.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:58.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 64, 2023-07-29T01:22:58.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 64 2023-07-29T01:22:58.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:18:22Z" 2023-07-29T01:22:58.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.964 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-07-29T01:22:58.965 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:58.965 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:58.965 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:58.965 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:58.966 INFO:tasks.rgw_multisite_tests: "instance_id": 7555809319554727162 2023-07-29T01:22:58.966 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.966 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:58.966 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.966 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:58.967 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.967 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.967 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:22:58.967 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.968 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-07-29T01:22:58.968 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.968 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:22:58.968 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.969 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.969 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.969 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:58.969 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.969 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.970 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:58.970 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.970 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-07-29T01:22:58.970 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.971 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:58.971 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.971 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.971 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.972 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:58.972 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.972 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.972 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:22:58.972 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.973 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.973 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.973 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:22:58.973 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.974 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.974 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.974 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:58.974 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.975 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.975 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:22:58.975 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.975 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:58.975 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.976 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:22:58.976 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.976 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.976 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:58.977 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.977 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:58.977 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-07-29T01:22:58.977 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-07-29T01:22:58.978 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.978 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:18:22Z" 2023-07-29T01:22:58.978 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:58.978 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:58.978 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052233', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.371644Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045940', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.390653Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031634', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640507Z'}}] 2023-07-29T01:22:58.979 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:22:58.979 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031634'} 2023-07-29T01:22:58.979 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052233, shard=2 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045940, shard=3 master=00000000000000000000:00000000000000031774 target=00000000000000000000:00000000000000031634 2023-07-29T01:22:58.979 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.980 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:58.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:58.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:58.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:58.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:58.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:58.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:58.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:58.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "instance_id": 7555809319554727162 2023-07-29T01:22:58.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:58.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:58.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:22:58.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 55, 2023-07-29T01:22:58.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:22:58.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:58.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:58.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 9, 2023-07-29T01:22:58.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:58.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:58.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:22:58.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:22:58.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:58.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:58.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:58.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:58.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:22:58.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:58.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:58.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:58.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:22:58.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:58.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:58.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 64, 2023-07-29T01:22:58.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 64 2023-07-29T01:22:58.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:58.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:18:28Z" 2023-07-29T01:22:58.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:58.995 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-07-29T01:22:58.995 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:58.995 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:58.995 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:58.995 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:58.996 INFO:tasks.rgw_multisite_tests: "instance_id": 7555809319554727162 2023-07-29T01:22:58.996 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.996 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:58.996 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.997 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:58.997 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:58.997 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:58.997 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:22:58.998 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:58.998 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-07-29T01:22:58.998 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:58.998 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:22:58.998 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:58.999 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:58.999 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:58.999 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:58.999 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:59.000 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:59.000 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:59.000 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:59.000 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-07-29T01:22:59.001 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:59.001 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:59.001 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:59.001 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:59.001 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:59.002 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:59.002 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:59.002 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:59.002 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:22:59.003 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:59.003 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:59.003 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:59.003 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:22:59.003 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:59.004 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:59.004 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:59.004 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:59.004 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:59.005 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:59.005 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:22:59.005 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:59.005 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:59.006 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:59.006 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:22:59.006 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:59.006 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:59.006 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:59.007 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:59.007 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:59.007 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-07-29T01:22:59.007 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-07-29T01:22:59.008 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:59.008 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:18:28Z" 2023-07-29T01:22:59.008 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:59.008 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:59.009 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052233', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.371644Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045940', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.390653Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031634', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640507Z'}}] 2023-07-29T01:22:59.009 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:22:59.009 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031634'} 2023-07-29T01:22:59.009 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052233, shard=2 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045940, shard=3 master=00000000000000000000:00000000000000031774 target=00000000000000000000:00000000000000031634 2023-07-29T01:22:59.010 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:59.010 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:59.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:59.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:59.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:59.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:59.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:59.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:59.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:59.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:59.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "instance_id": 7555809319554727162 2023-07-29T01:22:59.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:59.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:59.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:59.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:59.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:59.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:59.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:22:59.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:59.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 55, 2023-07-29T01:22:59.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:59.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:22:59.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:59.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:59.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:59.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:59.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:59.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:59.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:59.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:59.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 9, 2023-07-29T01:22:59.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:59.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:59.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:59.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:59.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:59.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:59.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:59.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:59.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:22:59.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:59.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:59.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:59.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:22:59.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:59.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:59.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:59.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:59.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:59.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:59.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:22:59.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:59.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:59.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:59.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:22:59.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:59.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:59.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:59.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:59.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:59.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 64, 2023-07-29T01:22:59.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 64 2023-07-29T01:22:59.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:59.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:18:33Z" 2023-07-29T01:22:59.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:59.025 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-07-29T01:22:59.025 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:59.025 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:59.026 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:59.026 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:59.026 INFO:tasks.rgw_multisite_tests: "instance_id": 7555809319554727162 2023-07-29T01:22:59.026 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:59.027 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:59.027 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:59.027 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:59.027 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:59.028 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:59.028 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:22:59.028 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:59.028 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-07-29T01:22:59.029 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:59.029 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:22:59.029 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:59.029 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:59.029 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:59.030 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:59.030 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:59.030 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:59.030 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:59.031 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:59.031 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-07-29T01:22:59.031 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:59.031 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:59.032 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:59.032 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:59.032 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:59.032 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:59.033 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:59.033 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:59.033 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:22:59.033 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:59.033 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:59.034 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:59.034 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:22:59.034 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:59.034 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:59.035 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:59.035 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:59.035 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:59.035 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:59.036 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:22:59.036 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:59.036 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:59.036 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:59.036 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:22:59.037 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:59.037 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:59.037 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:59.037 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:59.038 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:59.038 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-07-29T01:22:59.038 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-07-29T01:22:59.038 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:59.039 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:18:33Z" 2023-07-29T01:22:59.039 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:59.039 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:59.039 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052233', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.371644Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045940', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.390653Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031634', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640507Z'}}] 2023-07-29T01:22:59.039 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:22:59.040 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031634'} 2023-07-29T01:22:59.040 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052233, shard=2 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045940, shard=3 master=00000000000000000000:00000000000000031774 target=00000000000000000000:00000000000000031634 2023-07-29T01:22:59.040 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:59.040 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:59.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:59.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:59.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:59.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:59.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:59.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:59.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:59.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:59.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "instance_id": 7555809319554727162 2023-07-29T01:22:59.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:59.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:59.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:59.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:59.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:59.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:59.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:22:59.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:59.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 55, 2023-07-29T01:22:59.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:59.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:22:59.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:59.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:59.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:59.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:59.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:59.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:59.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:59.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:59.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 9, 2023-07-29T01:22:59.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:59.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:59.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:59.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:59.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:59.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:59.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:59.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:59.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:22:59.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:59.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:59.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:59.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:22:59.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:59.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:59.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:59.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:59.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:59.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:59.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:22:59.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:59.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:59.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:59.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:22:59.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:59.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:59.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:59.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:59.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:59.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 64, 2023-07-29T01:22:59.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 64 2023-07-29T01:22:59.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:59.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:18:38Z" 2023-07-29T01:22:59.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:59.055 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-07-29T01:22:59.056 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:59.056 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:59.056 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:59.056 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:59.057 INFO:tasks.rgw_multisite_tests: "instance_id": 7555809319554727162 2023-07-29T01:22:59.057 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:59.057 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:59.057 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:59.057 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:59.058 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:59.058 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:59.058 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052233", 2023-07-29T01:22:59.058 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:59.059 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-07-29T01:22:59.059 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:59.059 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:15:45.371644Z" 2023-07-29T01:22:59.059 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:59.060 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:59.060 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:59.060 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:59.060 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:59.060 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:59.061 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:59.061 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:59.061 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-07-29T01:22:59.061 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:59.062 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:59.062 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:59.062 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:59.062 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:59.063 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:59.063 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:59.063 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:59.063 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045940", 2023-07-29T01:22:59.063 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:59.064 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:59.064 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:59.064 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:15:45.390653Z" 2023-07-29T01:22:59.064 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:59.065 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:59.065 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:59.065 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:59.065 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:59.066 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:59.066 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031634", 2023-07-29T01:22:59.066 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:59.066 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:59.067 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:59.067 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:16:32.640507Z" 2023-07-29T01:22:59.067 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:59.067 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:59.068 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:59.068 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:59.068 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:59.068 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-07-29T01:22:59.068 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-07-29T01:22:59.069 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:59.069 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:18:38Z" 2023-07-29T01:22:59.069 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:59.069 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:59.070 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052233', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.371644Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045940', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:15:45.390653Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031634', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640507Z'}}] 2023-07-29T01:22:59.070 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:22:59.070 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000052233', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000045940', 3: '00000000000000000000:00000000000000031634'} 2023-07-29T01:22:59.070 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052514 target=00000000000000000000:00000000000000052233, shard=2 master=00000000000000000000:00000000000000046080 target=00000000000000000000:00000000000000045940, shard=3 master=00000000000000000000:00000000000000031774 target=00000000000000000000:00000000000000031634 2023-07-29T01:22:59.071 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:59.071 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-07-29T01:22:59.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-07-29T01:22:59.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setuser ceph since I am not root 2023-07-29T01:22:59.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-07-29T01:22:59.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:59.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "sync_status": { 2023-07-29T01:22:59.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "info": { 2023-07-29T01:22:59.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "sync", 2023-07-29T01:22:59.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "num_shards": 4, 2023-07-29T01:22:59.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "instance_id": 7555809319554727162 2023-07-29T01:22:59.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:59.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "markers": [ 2023-07-29T01:22:59.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:59.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 0, 2023-07-29T01:22:59.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:59.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:59.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000052514", 2023-07-29T01:22:59.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:59.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 55, 2023-07-29T01:22:59.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:59.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:17:10.772704Z" 2023-07-29T01:22:59.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:59.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:59.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:59.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 1, 2023-07-29T01:22:59.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:59.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:59.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:59.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:59.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 9, 2023-07-29T01:22:59.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:59.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:59.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:59.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:59.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:59.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 2, 2023-07-29T01:22:59.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:59.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:59.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000046080", 2023-07-29T01:22:59.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:59.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:59.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:59.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:17:10.709533Z" 2023-07-29T01:22:59.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:59.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:59.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: { 2023-07-29T01:22:59.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "key": 3, 2023-07-29T01:22:59.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "val": { 2023-07-29T01:22:59.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "status": "incremental-sync", 2023-07-29T01:22:59.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "marker": "00000000000000000000:00000000000000031774", 2023-07-29T01:22:59.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "next_step_marker": "", 2023-07-29T01:22:59.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total_entries": 0, 2023-07-29T01:22:59.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "pos": 0, 2023-07-29T01:22:59.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "timestamp": "2023-07-29T01:17:10.812152Z" 2023-07-29T01:22:59.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:59.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:59.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: ] 2023-07-29T01:22:59.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:59.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "full_sync": { 2023-07-29T01:22:59.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "total": 64, 2023-07-29T01:22:59.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "complete": 64 2023-07-29T01:22:59.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: }, 2023-07-29T01:22:59.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: "current_time": "2023-07-29T01:18:43Z" 2023-07-29T01:22:59.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi101.stdout: INFO: } 2023-07-29T01:22:59.086 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-07-29T01:22:59.086 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-07-29T01:22:59.086 INFO:tasks.rgw_multisite_tests: "info": { 2023-07-29T01:22:59.086 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-07-29T01:22:59.087 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-07-29T01:22:59.087 INFO:tasks.rgw_multisite_tests: "instance_id": 7555809319554727162 2023-07-29T01:22:59.087 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:59.087 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-07-29T01:22:59.088 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:59.088 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-07-29T01:22:59.088 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:59.088 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:59.089 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000052514", 2023-07-29T01:22:59.089 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:59.089 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-07-29T01:22:59.089 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:59.090 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:17:10.772704Z" 2023-07-29T01:22:59.090 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:59.090 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:59.090 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:59.090 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-07-29T01:22:59.091 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:59.091 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:59.091 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045021", 2023-07-29T01:22:59.091 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:59.092 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-07-29T01:22:59.092 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:59.092 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:16:32.640326Z" 2023-07-29T01:22:59.092 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:59.093 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:59.093 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:59.093 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-07-29T01:22:59.093 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:59.093 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:59.094 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046080", 2023-07-29T01:22:59.094 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:59.094 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:59.094 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:59.095 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:17:10.709533Z" 2023-07-29T01:22:59.095 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:59.095 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:59.095 INFO:tasks.rgw_multisite_tests: { 2023-07-29T01:22:59.096 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-07-29T01:22:59.096 INFO:tasks.rgw_multisite_tests: "val": { 2023-07-29T01:22:59.096 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-07-29T01:22:59.096 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031774", 2023-07-29T01:22:59.096 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-07-29T01:22:59.097 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-07-29T01:22:59.097 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-07-29T01:22:59.097 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-07-29T01:17:10.812152Z" 2023-07-29T01:22:59.097 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:59.098 INFO:tasks.rgw_multisite_tests: } 2023-07-29T01:22:59.098 INFO:tasks.rgw_multisite_tests: ] 2023-07-29T01:22:59.098 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:59.098 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-07-29T01:22:59.099 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-07-29T01:22:59.099 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-07-29T01:22:59.099 INFO:tasks.rgw_multisite_tests: }, 2023-07-29T01:22:59.099 INFO:tasks.rgw_multisite_tests: "current_time": "2023-07-29T01:18:43Z" 2023-07-29T01:22:59.100 INFO:tasks.rgw_multisite_tests:} 2023-07-29T01:22:59.100 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:59.100 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052514', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.772704Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045021', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-07-29T01:16:32.640326Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046080', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.709533Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031774', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-07-29T01:17:10.812152Z'}}] 2023-07-29T01:22:59.100 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:22:59.100 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000052514', 1: '00000000000000000000:00000000000000045021', 2: '00000000000000000000:00000000000000046080', 3: '00000000000000000000:00000000000000031774'} 2023-07-29T01:22:59.101 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finished data checkpoint for target_zone=a2 source_zone=a1 2023-07-29T01:22:59.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-72/ 2023-07-29T01:22:59.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-72/ 2023-07-29T01:22:59.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:59.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-72/ 2023-07-29T01:22:59.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:59.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:59.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:59.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:59.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:59.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-07-29T01:22:59.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:59.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:59.104 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:59.104 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:59.104 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:59.104 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 01:18:43 GMT 2023-07-29T01:22:59.105 INFO:tasks.rgw_multisite_tests:/hiiqos-72/ 2023-07-29T01:22:59.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:59.105 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:cQh7ieXoBBfnWv31LydzJ+nRYyM= 2023-07-29T01:22:59.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:18:43 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:cQh7ieXoBBfnWv31LydzJ+nRYyM=', 'Content-Length': '0'} 2023-07-29T01:22:59.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c7d1c1cb18d04321-0064c468f3-2a75-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:18:43 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:59.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-72/obj1 2023-07-29T01:22:59.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-72/obj1 2023-07-29T01:22:59.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:59.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-72/obj1 2023-07-29T01:22:59.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:59.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:59.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:59.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:59.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:59.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:59.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:59.108 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:59.109 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:59.109 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:59.109 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 01:18:43 GMT 2023-07-29T01:22:59.109 INFO:tasks.rgw_multisite_tests:/hiiqos-72/obj1 2023-07-29T01:22:59.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:59.110 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:erveqlHrB7vT3BJZyJzhXuFnctU= 2023-07-29T01:22:59.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:18:43 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:erveqlHrB7vT3BJZyJzhXuFnctU=', 'Content-Length': '0'} 2023-07-29T01:22:59.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000eaa6f090c15f3e18-0064c468f3-2a75-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 01:18:43 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:59.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-72/obj2 2023-07-29T01:22:59.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-72/obj2 2023-07-29T01:22:59.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:59.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-72/obj2 2023-07-29T01:22:59.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:59.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:59.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:59.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:59.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:59.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:59.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:59.113 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:59.113 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:59.114 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:59.114 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 01:18:43 GMT 2023-07-29T01:22:59.114 INFO:tasks.rgw_multisite_tests:/hiiqos-72/obj2 2023-07-29T01:22:59.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:59.115 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:kv8IPJblGjTzXBZV8xllcLYcQPY= 2023-07-29T01:22:59.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:18:43 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:kv8IPJblGjTzXBZV8xllcLYcQPY=', 'Content-Length': '0'} 2023-07-29T01:22:59.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000453e87c0838c0cc4-0064c468f3-2a75-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 01:18:43 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:59.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-72/obj3 2023-07-29T01:22:59.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-72/obj3 2023-07-29T01:22:59.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:59.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-72/obj3 2023-07-29T01:22:59.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:59.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:59.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:59.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:59.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:59.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:59.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:59.118 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:59.118 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:59.118 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:59.119 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 01:18:43 GMT 2023-07-29T01:22:59.119 INFO:tasks.rgw_multisite_tests:/hiiqos-72/obj3 2023-07-29T01:22:59.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:59.119 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:LaqP0NQRyw18k5PVWxByol1NRf0= 2023-07-29T01:22:59.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:18:43 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:LaqP0NQRyw18k5PVWxByol1NRf0=', 'Content-Length': '0'} 2023-07-29T01:22:59.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000772da370cdd5d427-0064c468f3-2a75-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 01:18:43 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:59.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-72/obj4 2023-07-29T01:22:59.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-72/obj4 2023-07-29T01:22:59.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:59.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-72/obj4 2023-07-29T01:22:59.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:59.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:59.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:59.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:59.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:59.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:59.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:59.123 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:59.123 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:59.123 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:59.123 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 01:18:43 GMT 2023-07-29T01:22:59.124 INFO:tasks.rgw_multisite_tests:/hiiqos-72/obj4 2023-07-29T01:22:59.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:59.124 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:4L5Xggp/26nGQ9ma25xX8cnLTBs= 2023-07-29T01:22:59.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:18:43 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:4L5Xggp/26nGQ9ma25xX8cnLTBs=', 'Content-Length': '0'} 2023-07-29T01:22:59.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000a78b17d69b6a608c-0064c468f3-2a75-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 01:18:43 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:59.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-73/ 2023-07-29T01:22:59.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-73/ 2023-07-29T01:22:59.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:59.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-73/ 2023-07-29T01:22:59.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:59.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:59.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:59.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:59.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:59.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:59.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:59.127 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:59.128 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:59.128 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:59.128 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 01:18:43 GMT 2023-07-29T01:22:59.128 INFO:tasks.rgw_multisite_tests:/hiiqos-73/ 2023-07-29T01:22:59.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:59.129 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:dQh/WZ12xx8FsMcxhY9obKwGpeM= 2023-07-29T01:22:59.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:18:43 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:dQh/WZ12xx8FsMcxhY9obKwGpeM=', 'Content-Length': '0'} 2023-07-29T01:22:59.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000075b1c92f7f593732-0064c468f3-2a75-a2'), ('Content-Length', '0'), ('Date', 'Sat, 29 Jul 2023 01:18:43 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:59.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-73/obj1 2023-07-29T01:22:59.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-73/obj1 2023-07-29T01:22:59.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:59.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-73/obj1 2023-07-29T01:22:59.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:59.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:59.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:59.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:59.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:59.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:59.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:59.132 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:59.132 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:59.132 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:59.133 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 01:18:43 GMT 2023-07-29T01:22:59.133 INFO:tasks.rgw_multisite_tests:/hiiqos-73/obj1 2023-07-29T01:22:59.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:59.133 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:+JfyKUYMFv5UnO+OkWO0JD3LdcY= 2023-07-29T01:22:59.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:18:43 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:+JfyKUYMFv5UnO+OkWO0JD3LdcY=', 'Content-Length': '0'} 2023-07-29T01:22:59.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:17:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-72:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.10310.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d90eba63c7a67c0a-0064c468f3-2a75-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:18:43 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:59.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-73/obj1 2023-07-29T01:22:59.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-73/obj1 2023-07-29T01:22:59.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:59.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-73/obj1 2023-07-29T01:22:59.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:59.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:59.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:59.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:59.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:59.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:59.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:59.137 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:59.137 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:59.137 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:59.137 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 01:18:43 GMT 2023-07-29T01:22:59.138 INFO:tasks.rgw_multisite_tests:/hiiqos-73/obj1 2023-07-29T01:22:59.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:59.138 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:lD+wLDtxIAn/0EjwgSP4Jo/h+AU= 2023-07-29T01:22:59.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:18:43 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:lD+wLDtxIAn/0EjwgSP4Jo/h+AU=', 'Content-Length': '0'} 2023-07-29T01:22:59.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:17:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-72:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.10310.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a83a7c3d8e7e1c06-0064c468f3-2a75-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:18:43 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:59.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-73/obj2 2023-07-29T01:22:59.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-73/obj2 2023-07-29T01:22:59.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:59.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-73/obj2 2023-07-29T01:22:59.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:59.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:59.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:59.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:59.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:59.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:59.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:59.141 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:59.142 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:59.142 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:59.142 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 01:18:43 GMT 2023-07-29T01:22:59.142 INFO:tasks.rgw_multisite_tests:/hiiqos-73/obj2 2023-07-29T01:22:59.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:59.143 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:g23/qhuyRA2ew1JJ3al9lPbvglQ= 2023-07-29T01:22:59.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:18:43 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:g23/qhuyRA2ew1JJ3al9lPbvglQ=', 'Content-Length': '0'} 2023-07-29T01:22:59.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:17:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-72:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.10310.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fbe3f7e3f7c3286a-0064c468f3-2a75-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:18:43 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:59.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-73/obj2 2023-07-29T01:22:59.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-73/obj2 2023-07-29T01:22:59.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:59.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-73/obj2 2023-07-29T01:22:59.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:59.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:59.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:59.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:59.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:59.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:59.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:59.146 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:59.146 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:59.146 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:59.147 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 01:18:43 GMT 2023-07-29T01:22:59.147 INFO:tasks.rgw_multisite_tests:/hiiqos-73/obj2 2023-07-29T01:22:59.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:59.147 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:tgDbr808w6oWIMnnH0BY+hkxe64= 2023-07-29T01:22:59.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:18:43 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:tgDbr808w6oWIMnnH0BY+hkxe64=', 'Content-Length': '0'} 2023-07-29T01:22:59.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:17:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-72:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.10310.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d1f0a2ecd3064b3-0064c468f3-2a75-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:18:43 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:59.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-73/obj3 2023-07-29T01:22:59.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-73/obj3 2023-07-29T01:22:59.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:59.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-73/obj3 2023-07-29T01:22:59.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:59.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:59.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:59.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:59.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:59.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:59.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:59.151 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:59.151 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:59.151 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:59.151 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 01:18:43 GMT 2023-07-29T01:22:59.152 INFO:tasks.rgw_multisite_tests:/hiiqos-73/obj3 2023-07-29T01:22:59.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:59.152 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:feq6f+rFATV2ZQl1R4aqdmEp5vk= 2023-07-29T01:22:59.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:18:43 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:feq6f+rFATV2ZQl1R4aqdmEp5vk=', 'Content-Length': '0'} 2023-07-29T01:22:59.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:17:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-73:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.10310.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007ae6c0519b1f5403-0064c468f3-2a75-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:18:43 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:59.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-73/obj3 2023-07-29T01:22:59.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-73/obj3 2023-07-29T01:22:59.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-07-29T01:22:59.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-73/obj3 2023-07-29T01:22:59.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:59.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:59.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:59.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:59.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:59.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:59.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:59.155 INFO:tasks.rgw_multisite_tests:GET 2023-07-29T01:22:59.156 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:59.156 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:59.156 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 01:18:43 GMT 2023-07-29T01:22:59.156 INFO:tasks.rgw_multisite_tests:/hiiqos-73/obj3 2023-07-29T01:22:59.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:59.157 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:zp93/oJOVsvol1FhZ1q8NR4RExA= 2023-07-29T01:22:59.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:18:43 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:zp93/oJOVsvol1FhZ1q8NR4RExA=', 'Content-Length': '0'} 2023-07-29T01:22:59.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Sat, 29 Jul 2023 01:17:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433:hiiqos-73:3e3b19d0-fc9f-43a0-bfa4-8f7cf31d6433.10310.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000092c0dff41e5cfa9a-0064c468f3-2a75-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Sat, 29 Jul 2023 01:18:43 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:59.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hiiqos-73/obj4 2023-07-29T01:22:59.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hiiqos-73/obj4 2023-07-29T01:22:59.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-07-29T01:22:59.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hiiqos-73/obj4 2023-07-29T01:22:59.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-07-29T01:22:59.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-07-29T01:22:59.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi101.front.sepia.ceph.com:8001 2023-07-29T01:22:59.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-07-29T01:22:59.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-07-29T01:22:59.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-07-29T01:22:59.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-07-29T01:22:59.160 INFO:tasks.rgw_multisite_tests:HEAD 2023-07-29T01:22:59.160 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:59.160 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:59.161 INFO:tasks.rgw_multisite_tests:Sat, 29 Jul 2023 01:18:43 GMT 2023-07-29T01:22:59.161 INFO:tasks.rgw_multisite_tests:/hiiqos-73/obj4 2023-07-29T01:22:59.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-07-29T01:22:59.161 INFO:tasks.rgw_multisite_tests:AWS S7W2J6K8BFBCG65TLASH:TcIGBx61tmdBKXVduhC0myt0g5k= 2023-07-29T01:22:59.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.0 Linux/5.4.0-137-generic', 'Date': 'Sat, 29 Jul 2023 01:18:43 GMT', 'Authorization': 'AWS S7W2J6K8BFBCG65TLASH:TcIGBx61tmdBKXVduhC0myt0g5k=', 'Content-Length': '0'} 2023-07-29T01:22:59.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000f9e0bb51bad4ee26-0064c468f3-2a75-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Sat, 29 Jul 2023 01:18:43 GMT'), ('Connection', 'Keep-Alive')] 2023-07-29T01:22:59.162 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-07-29T01:22:59.162 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:59.163 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-07-29T01:22:59.163 INFO:tasks.rgw_multisite_tests:Ran 47 tests in 6035.865s 2023-07-29T01:22:59.163 INFO:tasks.rgw_multisite_tests: 2023-07-29T01:22:59.164 INFO:tasks.rgw_multisite_tests:FAILED (SKIP=4, failures=5) 2023-07-29T01:22:59.166 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_61140164d2579c30ffff3970615e0f82e40c6470/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_61140164d2579c30ffff3970615e0f82e40c6470/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/github.com_ceph_ceph-c_0484097b0572e96f42cf6402cbe0ac8dcb046577/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2023-07-29T01:22:59.389 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=a323684034f64c66a939be8a5b47490d Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_61140164d2579c30ffff3970615e0f82e40c6470/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_61140164d2579c30ffff3970615e0f82e40c6470/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/github.com_ceph_ceph-c_0484097b0572e96f42cf6402cbe0ac8dcb046577/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2023-07-29T01:22:59.391 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite-tests 2023-07-29T01:22:59.406 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite 2023-07-29T01:22:59.415 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2023-07-29T01:22:59.423 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2023-07-29T01:22:59.424 INFO:teuthology.orchestra.run:waiting for 300 2023-07-29T01:23:04.512 INFO:tasks.rgw.c1.client.0.smithi101.stdout:2023-07-29T01:22:59.327+0000 7f00a029b640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm (PID: 88861) UID: 0 2023-07-29T01:23:04.513 INFO:tasks.rgw.c1.client.0.smithi101.stdout:2023-07-29T01:22:59.327+0000 7f00a660b880 -1 shutting down 2023-07-29T01:23:05.114 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2023-07-29T01:23:10.512 INFO:tasks.rgw.c1.client.0:Stopped 2023-07-29T01:23:10.513 DEBUG:teuthology.orchestra.run.smithi101:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock 2023-07-29T01:23:10.531 DEBUG:teuthology.orchestra.run.smithi101:> sudo rm -f /etc/ceph/vault-root-token 2023-07-29T01:23:10.604 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-07-29T01:23:10.605 INFO:teuthology.orchestra.run:waiting for 300 2023-07-29T01:23:10.646 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T01:23:10.548+0000 7f5a3cd71640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 149288) UID: 0 2023-07-29T01:23:10.646 INFO:tasks.rgw.c1.client.1.smithi101.stdout:2023-07-29T01:23:10.548+0000 7f5a43212880 -1 shutting down 2023-07-29T01:23:10.718 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2023-07-29T01:23:16.222 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-07-29T01:23:16.707 INFO:tasks.rgw.c1.client.1:Stopped 2023-07-29T01:23:16.707 DEBUG:teuthology.orchestra.run.smithi101:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock 2023-07-29T01:23:16.725 DEBUG:teuthology.orchestra.run.smithi101:> sudo rm -f /etc/ceph/vault-root-token 2023-07-29T01:23:16.797 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2023-07-29T01:23:16.797 INFO:teuthology.orchestra.run:waiting for 300 2023-07-29T01:23:16.799 INFO:tasks.rgw.c2.client.0.smithi139.stdout:2023-07-29T01:23:16.700+0000 7f5f44757640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm (PID: 88750) UID: 0 2023-07-29T01:23:16.799 INFO:tasks.rgw.c2.client.0.smithi139.stdout:2023-07-29T01:23:16.700+0000 7f5f4a834880 -1 shutting down 2023-07-29T01:23:21.726 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-07-29T01:23:22.228 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2023-07-29T01:23:22.900 INFO:tasks.rgw.c2.client.0:Stopped 2023-07-29T01:23:22.900 DEBUG:teuthology.orchestra.run.smithi139:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock 2023-07-29T01:23:22.943 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f /etc/ceph/vault-root-token 2023-07-29T01:23:23.076 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2023-07-29T01:23:23.076 INFO:teuthology.orchestra.run:waiting for 300 2023-07-29T01:23:23.077 INFO:tasks.rgw.c2.client.1.smithi139.stdout:2023-07-29T01:23:22.979+0000 7f574f2a5640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm (PID: 89991) UID: 0 2023-07-29T01:23:23.078 INFO:tasks.rgw.c2.client.1.smithi139.stdout:2023-07-29T01:23:22.979+0000 7f5754b2c880 -1 shutting down 2023-07-29T01:23:27.734 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2023-07-29T01:23:27.734 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2023-07-29T01:23:29.178 INFO:tasks.rgw.c2.client.1:Stopped 2023-07-29T01:23:29.179 DEBUG:teuthology.orchestra.run.smithi139:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock 2023-07-29T01:23:29.199 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f /etc/ceph/vault-root-token 2023-07-29T01:23:29.271 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-07-29T01:23:29.284 DEBUG:teuthology.orchestra.run.smithi139:> sudo ceph --cluster c2 config set global mon_health_to_clog false 2023-07-29T01:23:29.761 INFO:teuthology.misc:Shutting down mds daemons... 2023-07-29T01:23:29.761 INFO:teuthology.misc:Shutting down osd daemons... 2023-07-29T01:23:29.762 DEBUG:tasks.ceph.c2.osd.0:waiting for process to exit 2023-07-29T01:23:29.762 INFO:teuthology.orchestra.run:waiting for 300 2023-07-29T01:23:29.810 INFO:tasks.ceph.c2.osd.0:Stopped 2023-07-29T01:23:29.811 DEBUG:tasks.ceph.c2.osd.1:waiting for process to exit 2023-07-29T01:23:29.811 INFO:teuthology.orchestra.run:waiting for 300 2023-07-29T01:23:29.881 INFO:tasks.ceph.c2.osd.1:Stopped 2023-07-29T01:23:29.882 DEBUG:tasks.ceph.c2.osd.2:waiting for process to exit 2023-07-29T01:23:29.882 INFO:teuthology.orchestra.run:waiting for 300 2023-07-29T01:23:29.952 INFO:tasks.ceph.c2.osd.2:Stopped 2023-07-29T01:23:29.953 INFO:teuthology.misc:Shutting down mgr daemons... 2023-07-29T01:23:29.953 DEBUG:tasks.ceph.c2.mgr.x:waiting for process to exit 2023-07-29T01:23:29.954 INFO:teuthology.orchestra.run:waiting for 300 2023-07-29T01:23:29.977 INFO:tasks.ceph.c2.mgr.x:Stopped 2023-07-29T01:23:29.978 INFO:teuthology.misc:Shutting down mon daemons... 2023-07-29T01:23:29.978 DEBUG:tasks.ceph.c2.mon.a:waiting for process to exit 2023-07-29T01:23:29.979 INFO:teuthology.orchestra.run:waiting for 300 2023-07-29T01:23:30.037 INFO:tasks.ceph.c2.mon.a:Stopped 2023-07-29T01:23:30.037 INFO:tasks.ceph:Checking cluster log for badness... 2023-07-29T01:23:30.038 DEBUG:teuthology.orchestra.run.smithi139:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-07-29T01:23:30.155 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-0 on ubuntu@smithi139.front.sepia.ceph.com 2023-07-29T01:23:30.156 DEBUG:teuthology.orchestra.run.smithi139:> sync && sudo umount -f /var/lib/ceph/osd/c2-0 2023-07-29T01:23:30.684 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-1 on ubuntu@smithi139.front.sepia.ceph.com 2023-07-29T01:23:30.684 DEBUG:teuthology.orchestra.run.smithi139:> sync && sudo umount -f /var/lib/ceph/osd/c2-1 2023-07-29T01:23:30.817 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-2 on ubuntu@smithi139.front.sepia.ceph.com 2023-07-29T01:23:30.818 DEBUG:teuthology.orchestra.run.smithi139:> sync && sudo umount -f /var/lib/ceph/osd/c2-2 2023-07-29T01:23:30.924 INFO:tasks.ceph:Archiving mon data... 2023-07-29T01:23:30.926 DEBUG:teuthology.misc:Transferring archived files from smithi139:/var/lib/ceph/mon/c2-a to /home/teuthworker/archive/yuriw-2023-07-28_23:12:07-rgw-reef-release-distro-default-smithi/7355843/data/c2.mon.a.tgz 2023-07-29T01:23:30.926 DEBUG:teuthology.orchestra.run.smithi139:> mktemp 2023-07-29T01:23:30.950 INFO:teuthology.orchestra.run.smithi139.stdout:/tmp/tmp.h50ybnwFOK 2023-07-29T01:23:30.951 DEBUG:teuthology.orchestra.run.smithi139:> sudo tar cz -f - -C /var/lib/ceph/mon/c2-a -- . > /tmp/tmp.h50ybnwFOK 2023-07-29T01:23:31.134 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod 0666 /tmp/tmp.h50ybnwFOK 2023-07-29T01:23:31.229 DEBUG:teuthology.orchestra.remote:smithi139:/tmp/tmp.h50ybnwFOK is 426KB 2023-07-29T01:23:31.303 DEBUG:teuthology.orchestra.run.smithi139:> rm -fr /tmp/tmp.h50ybnwFOK 2023-07-29T01:23:31.320 INFO:tasks.ceph:Cleaning ceph cluster... 2023-07-29T01:23:31.320 DEBUG:teuthology.orchestra.run.smithi101:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2023-07-29T01:23:31.323 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2023-07-29T01:23:31.386 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-07-29T01:23:31.401 DEBUG:teuthology.orchestra.run.smithi101:> sudo ceph --cluster c1 config set global mon_health_to_clog false 2023-07-29T01:23:31.703 INFO:teuthology.misc:Shutting down mds daemons... 2023-07-29T01:23:31.704 INFO:teuthology.misc:Shutting down osd daemons... 2023-07-29T01:23:31.704 DEBUG:tasks.ceph.c1.osd.0:waiting for process to exit 2023-07-29T01:23:31.705 INFO:teuthology.orchestra.run:waiting for 300 2023-07-29T01:23:31.750 INFO:tasks.ceph.c1.osd.0:Stopped 2023-07-29T01:23:31.751 DEBUG:tasks.ceph.c1.osd.1:waiting for process to exit 2023-07-29T01:23:31.751 INFO:teuthology.orchestra.run:waiting for 300 2023-07-29T01:23:31.797 INFO:tasks.ceph.c1.osd.1:Stopped 2023-07-29T01:23:31.798 DEBUG:tasks.ceph.c1.osd.2:waiting for process to exit 2023-07-29T01:23:31.798 INFO:teuthology.orchestra.run:waiting for 300 2023-07-29T01:23:31.846 INFO:tasks.ceph.c1.osd.2:Stopped 2023-07-29T01:23:31.846 INFO:teuthology.misc:Shutting down mgr daemons... 2023-07-29T01:23:31.847 DEBUG:tasks.ceph.c1.mgr.x:waiting for process to exit 2023-07-29T01:23:31.847 INFO:teuthology.orchestra.run:waiting for 300 2023-07-29T01:23:31.871 INFO:tasks.ceph.c1.mgr.x:Stopped 2023-07-29T01:23:31.871 INFO:teuthology.misc:Shutting down mon daemons... 2023-07-29T01:23:31.871 DEBUG:tasks.ceph.c1.mon.a:waiting for process to exit 2023-07-29T01:23:31.872 INFO:teuthology.orchestra.run:waiting for 300 2023-07-29T01:23:31.931 INFO:tasks.ceph.c1.mon.a:Stopped 2023-07-29T01:23:31.932 INFO:tasks.ceph:Checking cluster log for badness... 2023-07-29T01:23:31.932 DEBUG:teuthology.orchestra.run.smithi101:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-07-29T01:23:32.001 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-0 on ubuntu@smithi101.front.sepia.ceph.com 2023-07-29T01:23:32.002 DEBUG:teuthology.orchestra.run.smithi101:> sync && sudo umount -f /var/lib/ceph/osd/c1-0 2023-07-29T01:23:32.327 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-1 on ubuntu@smithi101.front.sepia.ceph.com 2023-07-29T01:23:32.327 DEBUG:teuthology.orchestra.run.smithi101:> sync && sudo umount -f /var/lib/ceph/osd/c1-1 2023-07-29T01:23:32.441 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-2 on ubuntu@smithi101.front.sepia.ceph.com 2023-07-29T01:23:32.442 DEBUG:teuthology.orchestra.run.smithi101:> sync && sudo umount -f /var/lib/ceph/osd/c1-2 2023-07-29T01:23:32.498 INFO:tasks.ceph:Archiving mon data... 2023-07-29T01:23:32.499 DEBUG:teuthology.misc:Transferring archived files from smithi101:/var/lib/ceph/mon/c1-a to /home/teuthworker/archive/yuriw-2023-07-28_23:12:07-rgw-reef-release-distro-default-smithi/7355843/data/c1.mon.a.tgz 2023-07-29T01:23:32.500 DEBUG:teuthology.orchestra.run.smithi101:> mktemp 2023-07-29T01:23:32.515 INFO:teuthology.orchestra.run.smithi101.stdout:/tmp/tmp.ykgCQASJth 2023-07-29T01:23:32.516 DEBUG:teuthology.orchestra.run.smithi101:> sudo tar cz -f - -C /var/lib/ceph/mon/c1-a -- . > /tmp/tmp.ykgCQASJth 2023-07-29T01:23:32.711 DEBUG:teuthology.orchestra.run.smithi101:> sudo chmod 0666 /tmp/tmp.ykgCQASJth 2023-07-29T01:23:32.736 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2023-07-29T01:23:32.796 DEBUG:teuthology.orchestra.remote:smithi101:/tmp/tmp.ykgCQASJth is 408KB 2023-07-29T01:23:32.891 DEBUG:teuthology.orchestra.run.smithi101:> rm -fr /tmp/tmp.ykgCQASJth 2023-07-29T01:23:32.906 INFO:tasks.ceph:Cleaning ceph cluster... 2023-07-29T01:23:32.907 DEBUG:teuthology.orchestra.run.smithi101:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2023-07-29T01:23:32.948 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2023-07-29T01:23:32.980 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-07-29T01:23:32.981 DEBUG:teuthology.orchestra.run.smithi101:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-07-29T01:23:33.015 DEBUG:teuthology.orchestra.run.smithi139:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-07-29T01:23:33.041 INFO:teuthology.orchestra.run.smithi101.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-07-29T01:23:33.054 INFO:teuthology.orchestra.run.smithi139.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-07-29T01:23:33.058 INFO:tasks.ceph:Archiving crash dumps... 2023-07-29T01:23:33.060 DEBUG:teuthology.misc:Transferring archived files from smithi101:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-07-28_23:12:07-rgw-reef-release-distro-default-smithi/7355843/remote/smithi101/crash 2023-07-29T01:23:33.062 DEBUG:teuthology.orchestra.run.smithi101:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-07-29T01:23:33.112 DEBUG:teuthology.misc:Transferring archived files from smithi139:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-07-28_23:12:07-rgw-reef-release-distro-default-smithi/7355843/remote/smithi139/crash 2023-07-29T01:23:33.113 DEBUG:teuthology.orchestra.run.smithi139:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-07-29T01:23:33.143 INFO:tasks.ceph:Compressing logs... 2023-07-29T01:23:33.144 DEBUG:teuthology.orchestra.run.smithi101:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-29T01:23:33.158 DEBUG:teuthology.orchestra.run.smithi139:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-29T01:36:20.807 INFO:tasks.ceph:Archiving logs... 2023-07-29T01:36:20.808 DEBUG:teuthology.misc:Transferring archived files from smithi101:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-07-28_23:12:07-rgw-reef-release-distro-default-smithi/7355843/remote/smithi101/log 2023-07-29T01:36:20.809 DEBUG:teuthology.orchestra.run.smithi101:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-29T01:36:34.750 DEBUG:teuthology.misc:Transferring archived files from smithi139:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-07-28_23:12:07-rgw-reef-release-distro-default-smithi/7355843/remote/smithi139/log 2023-07-29T01:36:34.751 DEBUG:teuthology.orchestra.run.smithi139:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-29T01:37:41.692 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-07-29T01:37:41.701 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-07-29T01:37:41.702 DEBUG:teuthology.orchestra.run.smithi101:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-29T01:37:41.704 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-29T01:37:51.827 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-07-29T01:37:51.847 INFO:teuthology.task.clock:Checking final clock skew... 2023-07-29T01:37:51.848 DEBUG:teuthology.orchestra.run.smithi101:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-29T01:37:51.851 DEBUG:teuthology.orchestra.run.smithi139:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-29T01:37:51.865 INFO:teuthology.orchestra.run.smithi101.stderr:bash: line 1: ntpq: command not found 2023-07-29T01:37:51.868 INFO:teuthology.orchestra.run.smithi139.stderr:bash: line 1: ntpq: command not found 2023-07-29T01:37:51.870 INFO:teuthology.orchestra.run.smithi101.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-29T01:37:51.870 INFO:teuthology.orchestra.run.smithi101.stdout:=============================================================================== 2023-07-29T01:37:51.871 INFO:teuthology.orchestra.run.smithi101.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 36 -124us[ -124us] +/- 84ms 2023-07-29T01:37:51.871 INFO:teuthology.orchestra.run.smithi101.stdout:^* hv02.front.sepia.ceph.com 3 6 377 37 -1523us[-1506us] +/- 64ms 2023-07-29T01:37:51.872 INFO:teuthology.orchestra.run.smithi101.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 34 +1721us[+1721us] +/- 68ms 2023-07-29T01:37:51.872 INFO:teuthology.orchestra.run.smithi101.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-07-29T01:37:52.036 INFO:teuthology.orchestra.run.smithi139.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-29T01:37:52.037 INFO:teuthology.orchestra.run.smithi139.stdout:=============================================================================== 2023-07-29T01:37:52.037 INFO:teuthology.orchestra.run.smithi139.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 39 -128us[ -145us] +/- 84ms 2023-07-29T01:37:52.037 INFO:teuthology.orchestra.run.smithi139.stdout:^* hv02.front.sepia.ceph.com 3 6 377 38 -1512us[-1529us] +/- 64ms 2023-07-29T01:37:52.037 INFO:teuthology.orchestra.run.smithi139.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 40 +1723us[+1706us] +/- 68ms 2023-07-29T01:37:52.038 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-07-29T01:37:52.045 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-07-29T01:37:52.055 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-07-29T01:37:52.056 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-07-29T01:37:52.066 DEBUG:teuthology.orchestra.run.smithi101:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-29T01:37:52.166 DEBUG:teuthology.orchestra.run.smithi139:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-29T01:37:52.307 DEBUG:teuthology.orchestra.run.smithi101:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-29T01:37:52.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-29T01:37:52.341 DEBUG:teuthology.orchestra.run.smithi139:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-29T01:37:52.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-29T01:37:52.376 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-07-29T01:37:52.386 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-07-29T01:37:52.449 INFO:teuthology.task.internal:Duration was 7891.723289 seconds 2023-07-29T01:37:52.449 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-07-29T01:37:52.462 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-07-29T01:37:52.463 DEBUG:teuthology.orchestra.run.smithi101:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-29T01:37:52.465 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-29T01:37:52.508 INFO:teuthology.orchestra.run.smithi101.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-29T01:37:52.546 INFO:teuthology.orchestra.run.smithi139.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-29T01:37:53.027 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-07-29T01:37:53.028 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi101.front.sepia.ceph.com 2023-07-29T01:37:53.028 DEBUG:teuthology.orchestra.run.smithi101:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-29T01:37:53.049 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi139.front.sepia.ceph.com 2023-07-29T01:37:53.049 DEBUG:teuthology.orchestra.run.smithi139:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-29T01:37:53.076 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-07-29T01:37:53.077 DEBUG:teuthology.orchestra.run.smithi101:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-29T01:37:53.092 DEBUG:teuthology.orchestra.run.smithi139:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-29T01:37:53.146 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-07-29T01:37:53.147 DEBUG:teuthology.orchestra.run.smithi101:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-29T01:37:53.163 DEBUG:teuthology.orchestra.run.smithi139:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-29T01:37:53.802 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-07-29T01:37:53.813 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-07-29T01:37:53.814 DEBUG:teuthology.orchestra.run.smithi101:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-29T01:37:53.840 DEBUG:teuthology.orchestra.run.smithi139:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-29T01:37:53.881 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-07-29T01:37:53.892 DEBUG:teuthology.orchestra.run.smithi101:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-29T01:37:53.895 DEBUG:teuthology.orchestra.run.smithi139:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-29T01:37:53.917 INFO:teuthology.orchestra.run.smithi101.stdout:kernel.core_pattern = core 2023-07-29T01:37:53.954 INFO:teuthology.orchestra.run.smithi139.stdout:kernel.core_pattern = core 2023-07-29T01:37:53.979 DEBUG:teuthology.orchestra.run.smithi101:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-29T01:37:54.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-29T01:37:54.006 DEBUG:teuthology.orchestra.run.smithi139:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-29T01:37:54.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-29T01:37:54.033 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-07-29T01:37:54.046 INFO:teuthology.task.internal:Transferring archived files... 2023-07-29T01:37:54.048 DEBUG:teuthology.misc:Transferring archived files from smithi101:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-07-28_23:12:07-rgw-reef-release-distro-default-smithi/7355843/remote/smithi101 2023-07-29T01:37:54.048 DEBUG:teuthology.orchestra.run.smithi101:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-29T01:37:54.093 DEBUG:teuthology.misc:Transferring archived files from smithi139:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-07-28_23:12:07-rgw-reef-release-distro-default-smithi/7355843/remote/smithi139 2023-07-29T01:37:54.093 DEBUG:teuthology.orchestra.run.smithi139:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-29T01:37:54.136 INFO:teuthology.task.internal:Removing archive directory... 2023-07-29T01:37:54.137 DEBUG:teuthology.orchestra.run.smithi101:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-29T01:37:54.139 DEBUG:teuthology.orchestra.run.smithi139:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-29T01:37:54.178 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-07-29T01:37:54.191 INFO:teuthology.task.internal:Not uploading archives. 2023-07-29T01:37:54.192 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-07-29T01:37:54.202 INFO:teuthology.task.internal:Tidying up after the test... 2023-07-29T01:37:54.203 DEBUG:teuthology.orchestra.run.smithi101:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-29T01:37:54.205 DEBUG:teuthology.orchestra.run.smithi139:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-29T01:37:54.217 INFO:teuthology.orchestra.run.smithi101.stdout: 263646 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 29 01:37 /home/ubuntu/cephtest 2023-07-29T01:37:54.232 INFO:teuthology.orchestra.run.smithi139.stdout: 263648 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 29 01:37 /home/ubuntu/cephtest 2023-07-29T01:37:54.234 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-07-29T01:37:54.245 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-07-29T01:37:54.266 INFO:teuthology.nuke:Checking targets against current locks 2023-07-29T01:37:54.297 DEBUG:teuthology.nuke:shortname: smithi101 2023-07-29T01:37:54.298 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-29T01:37:54.323 DEBUG:teuthology.nuke:shortname: smithi139 2023-07-29T01:37:54.323 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-29T01:37:54.327 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi101.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-28_23:12:07-rgw-reef-release-distro-default-smithi/7355843', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-28 23:17:01.281838', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDciPeiA+F8OU2n9arsBRdxjittPCt9x6Dl/s+Hfxtf1oRgJdXHBnQWafNP5NjVmYg7hnf5zZnMl1HsH2pMMc+8='} 2023-07-29T01:37:54.352 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi139.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-28_23:12:07-rgw-reef-release-distro-default-smithi/7355843', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-28 23:17:01.280354', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGcp7KnNK6sxlDspWEyXozyB+lgk0NC9jK/uC8FVE4JLB64hFISTyFr+uS0NN4bYfyCUHn21kkjENMNOmGNcFI4='} 2023-07-29T01:37:54.384 INFO:teuthology.orchestra.console.smithi101:Power off 2023-07-29T01:37:54.386 DEBUG:teuthology.orchestra.console.smithi101:pexpect command: ipmitool -H smithi101.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-29T01:37:54.411 DEBUG:teuthology.orchestra.console.smithi101:power off output: Chassis Power Control: Down/Off 2023-07-29T01:37:54.412 DEBUG:teuthology.orchestra.console.smithi101:pexpect command: ipmitool -H smithi101.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-29T01:37:54.431 INFO:teuthology.orchestra.console.smithi139:Power off 2023-07-29T01:37:54.433 DEBUG:teuthology.orchestra.console.smithi139:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-29T01:37:54.447 DEBUG:teuthology.orchestra.console.smithi101:check power output: Chassis Power is on 2023-07-29T01:37:54.459 DEBUG:teuthology.orchestra.console.smithi139:power off output: Chassis Power Control: Down/Off 2023-07-29T01:37:54.460 DEBUG:teuthology.orchestra.console.smithi139:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-29T01:37:54.486 DEBUG:teuthology.orchestra.console.smithi139:check power output: Chassis Power is on 2023-07-29T01:37:58.449 DEBUG:teuthology.orchestra.console.smithi101:pexpect command: ipmitool -H smithi101.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-29T01:37:58.488 DEBUG:teuthology.orchestra.console.smithi139:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-29T01:37:58.563 DEBUG:teuthology.orchestra.console.smithi101:check power output: Chassis Power is on 2023-07-29T01:37:58.601 DEBUG:teuthology.orchestra.console.smithi139:check power output: Chassis Power is on 2023-07-29T01:38:02.566 DEBUG:teuthology.orchestra.console.smithi101:pexpect command: ipmitool -H smithi101.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-29T01:38:02.603 DEBUG:teuthology.orchestra.console.smithi139:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-29T01:38:03.679 DEBUG:teuthology.orchestra.console.smithi101:check power output: 2023-07-29T01:38:03.715 DEBUG:teuthology.orchestra.console.smithi139:check power output: 2023-07-29T01:38:07.680 DEBUG:teuthology.orchestra.console.smithi101:pexpect command: ipmitool -H smithi101.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-29T01:38:07.717 DEBUG:teuthology.orchestra.console.smithi139:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-29T01:38:07.830 DEBUG:teuthology.orchestra.console.smithi139:check power output: Chassis Power is off 2023-07-29T01:38:07.931 INFO:teuthology.orchestra.console.smithi139:Power off completed 2023-07-29T01:38:08.792 DEBUG:teuthology.orchestra.console.smithi101:check power output: 2023-07-29T01:38:12.794 DEBUG:teuthology.orchestra.console.smithi101:pexpect command: ipmitool -H smithi101.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-29T01:38:13.908 DEBUG:teuthology.orchestra.console.smithi101:check power output: 2023-07-29T01:38:17.911 DEBUG:teuthology.orchestra.console.smithi101:pexpect command: ipmitool -H smithi101.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-29T01:38:19.023 DEBUG:teuthology.orchestra.console.smithi101:check power output: 2023-07-29T01:38:23.024 DEBUG:teuthology.orchestra.console.smithi101:pexpect command: ipmitool -H smithi101.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-29T01:38:23.136 DEBUG:teuthology.orchestra.console.smithi101:check power output: Chassis Power is off 2023-07-29T01:38:23.237 INFO:teuthology.orchestra.console.smithi101:Power off completed 2023-07-29T01:38:23.341 INFO:teuthology.run:Summary data: description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_latest} tasks/test_multi} duration: 7891.7232892513275 failure_reason: rgw multisite test failures flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=a323684034f64c66a939be8a5b47490d status: fail success: false 2023-07-29T01:38:23.342 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-29T01:38:28.467 INFO:teuthology.run:FAIL